Transcription pipeline
worker/README.md says how to run a worker. This page records what the pipeline does and what it measured, so capacity and quality decisions have numbers.
Stages
Section titled “Stages”| Stage | Tool | Where | What it produces |
|---|---|---|---|
| Enqueue | admin panel التفريغ الآلي | API | one transcription-jobs row per eligible material |
| Claim | POST /api/pipeline/claim |
worker → API | up to N jobs, atomically; stale running jobs handed back first |
| Download | urllib |
worker | the mp3 from files.kalelm.com in a temp dir |
| ASR | cohere-transcribe (CohereLabs 2B Arabic/English, Silero VAD, segment timing) |
worker GPU | cues with start times, ~13.8 words each |
| Post-edit (optional) | qwen3:8b via Ollama, worker/postedit.py |
worker GPU | spelling, hamza, ta-marbuta, punctuation fixes; lines that change word content are rejected |
| Tashkeel | CAMeL Tools MLE, worker/lite.py |
worker CPU | hamza, ta-marbuta, alef-maqsura restored; shadda and tanween kept; vowels dropped |
| Upload | POST /api/pipeline/complete |
worker → API | transcript created (source: cohere), material flagged, Meilisearch updated by the hooks |
A result is written to results/job-<id>.json before upload and retried three times; a job that fails three claims stays failed until re-queued from the panel.
Eligibility
Section titled “Eligibility”Published, has audioUrl, no youtubeId (videos have caption tracks already), language ar or en, no transcript yet. On 2026-09-10: 30,974 materials, 12,696 known audio hours across the two thirds with a recorded duration, so roughly 19,000 hours in all, averaging 36 minutes each.
Measurements
Section titled “Measurements”All on the 59-minute fatawa episode unless noted; Apple M1 Max GPU via Metal.
| What | Result |
|---|---|
| ASR, default model | 3 min 09 s total, of which model load 1 min 30 s first time, 11 s cached; 18.8× real time |
| ASR + word-level alignment | 3 min 34 s; every word aligned by CTC, zero fallbacks; word boundaries land on the right audio 14 of 15 times, start edges ~80 ms tight |
| Full pipeline with post-edit, short clips | 9.6× to 14.6× real time |
| ASR on CPU (M1 Max) | 0.4× real time — impractical for the backlog |
| Post-edit change rate | 3.8 % of words on a 20-chunk sample; ~3 s per 60 words |
| Hamza/ta-marbuta restoration | 90.3 % correct on 1,344 reference words; the top miss is أن vs إن |
| Shadda / tanween recovery | 71 % / 55 % against a noisy reference |
| Tashkeel ASR fine-tune (rejected) | 35 min 39 s, 3× tokens, 48.9 % word divergence |
Sizing the backlog
Section titled “Sizing the backlog”| Where | Throughput | 19,000 hours takes |
|---|---|---|
| This Mac, full pipeline | ~10× | ~7 weeks |
| This Mac, no post-edit | ~19× | ~4 weeks |
| Rented RTX 3090/4090, no post-edit | 50–100× | ~1 week; $20–60 at spot prices |
| Rented GPU, with post-edit as built | ~12× | 1,600 GPU-hours; run post-edit 4-way parallel or skip it for bulk |
After the backlog, new content is a few recordings a day; the panel’s local worker button or a Mac mini covers it.
Operating it
Section titled “Operating it”- Panel shows backlog, queue, running jobs with stage and heartbeat age, last failures with a re-queue button, pause, post-edit toggle, and a start/stop for a worker on the backend host (
PIPELINE_WORKER_PYTHONmust be set there). - Pause stops workers taking new jobs; they finish what they hold.
- Stale running jobs (no heartbeat for 30 min) return to the queue on the next claim.
- Never overwrites: if a transcript appeared by another route while a job ran, the job is marked
skipped. - Stats per job (audio seconds, RTFx, stage timings, post-edit counts) are stored on the job row.