Keep manuscript processing and generation local-first.
Books, edits, voices and generated media remain in a self-hosted storage boundary.Alternative not selected: Making a hosted model or cloud storage service mandatory.
Local-first AI audiobook workflows
Whispbook
A self-hosted React and FastAPI audiobook studio, paired with research tooling for schema-constrained speaker attribution and local LoRA evaluation.
Context and problem
Turning a manuscript into editable, chaptered audio involves document cleanup, voice control, long-running generation, subtitles and character attribution without sending the working corpus to a hosted black box.
Ownership and contribution
Chakib owns the product workflow and backend/frontend implementation, and built the separate attribution research path for controlled context, structured tool outputs, local LoRA adapters and evaluation reports.
Responsibilities
- Built the React writing desk and FastAPI storage/job APIs.
- Integrated local/open TTS engines, preview, generation and subtitle packaging.
- Defined structured attribution/export schemas and candidate-aware context preparation.
- Built MLX-LM and Unsloth LoRA asset generation plus checkpoint evaluation-report tooling.
Architecture and data flow
The product path converts local documents into editable chapters, snapshots book state and TTS settings, then generates chapter audio, subtitles and an M4B package. The research path prepares speaker candidates and surrounding evidence for schema-constrained attribution and evaluates local adapters separately.
- Local selectable-text document
- Chapter and paragraph workspace
- Planner/coordinator workflow with bounded context
- Speaker attribution / voice mapping
- Local TTS generation jobs
- Chapter audio + VTT/SRT + M4B export
Engineering decisions
Separate the shipping studio from model-attribution experiments.
The runnable product can stay reliable while attribution datasets, adapters and evaluations evolve independently.Alternative not selected: Wiring an unvalidated model directly into the production annotation path.
Reliability, security and evaluation
- Structured JSON Schemas validate attribution examples, tool-call outputs and annotation exports.
- Frontend and backend have independent CI checks; a mock TTS path supports deterministic smoke tests.
- The public studio currently does not wire a separate model-based annotation provider; attribution remains a separate research workflow.
Outcome and status
Active self-hosted studio with document import, local TTS, chapter/subtitle export and CI. Speaker-attribution training and evaluation are maintained as a separate experimental track.
Technology groups
Product
React, TypeScript, FastAPI, Docker
Media
Kokoro, Chatterbox, ffmpeg, M4B, VTT, SRT
Applied AI
FunctionGemma, LoRA, MLX-LM, Unsloth, JSON Schema