System map

Machines

Where	What runs there	Reach it
Cloudflare	DNS, TLS, page cache, rate limit for *.mohanad.xyz	dashboard
Hangar box 188.245.169.96 (admin.mohanad.xyz)	kalelm (Astro site) and kalelm-api (Payload + Next admin/API) as Hangar app sites	Hangar API with a token; ssh root
Search VPS 49.12.78.95	Caddy; Meilisearch 1.53.1 (25 GB, 4 indexes); Ollama + bge-m3 (query embeddings); Postgres 17 (the archive DB, TLS)	ssh root; /srv/kalelm
This Mac	dev backend (3001), local Postgres + Meilisearch in Docker, the transcription worker (GPU)	—

Public URLs (test): https://kalelm.mohanad.xyz, https://kalelm-api.mohanad.xyz/admin. Internal: search.49.12.78.95.sslip.io (Meilisearch), embed.49.12.78.95.sslip.io (Ollama), 49.12.78.95:5432 (Postgres) — all reachable only from the Hangar server.

Flows

A page view. Browser → Cloudflare (cache: pages 10 min fresh, 1 h stale) → Hangar/Caddy → kalelm (Astro SSR) → kalelm-api REST → Postgres on the search VPS. Facets and the browse tree are cached in the API for 5 min; the sitemap is built hourly.

A search. Site → POST /api/search with mode keyword | hybrid | semantic. Keyword hits the caption_segments index. Hybrid/semantic embed the query through Ollama on the VPS and search caption_sentences with vectors. If the vector path fails, or SEARCH_VECTORS=off is set, the API answers with keyword results and degraded: <mode>.

A transcription. Admin panel /admin/transcription enqueues eligible materials (published, audio-only, no transcript) into transcription-jobs. A worker (any machine, outbound HTTPS only, CRON_SECRET bearer) claims a job → downloads the mp3 → Cohere ASR → optional LLM post-edit (Ollama qwen3:8b) → hamza/shadda restore (CAMeL, CPU) → uploads cues. The API creates the transcript; its hooks flag the material and push it into Meilisearch.

Screens

The transcription control panel at /admin/transcription: backlog, queue, workers, failures, the pause switch and the local worker card.

The queue itself as a Payload collection: one row per material with status, stage, worker and attempts.

The admin home: the archive dashboard above the collection list.

The public search page in hybrid mode, results grouped by material with cue-level marks.

An archive listing page, served from the Cloudflare cache after the first visitor.

The site home page on the test domain.

Where things live (repo: backend/)

Thing	Path
Queue collection, settings global	src/collections/TranscriptionJobs.ts, src/globals/PipelineSettings.ts
Pipeline API (stats, control, claim, heartbeat, complete)	src/endpoints/pipeline.ts
Control panel	src/components/TranscriptionDashboard/
Local worker start/stop	src/lib/pipelineWorker.ts
Search API + fallback + switch	src/endpoints/search.ts
Meilisearch sync hooks	src/search/sync.ts
Endpoint cache	src/utilities/ttlCache.ts
Worker (Python)	worker/ (README.md has VPS/Mac setup)
Search VPS config	infra/search-vps/ (compose, Caddyfile, top-queries.sql)
Deploy to Hangar	scripts/deploy-hangar.sh backend|frontend|docs — docs publish to https://kalelm-docs.mohanad.xyz
Reindex one material	scripts/reindex-material.ts <id>
The gate	pnpm verify (73 checks)

Secrets and env

Never in git. Backend env is held by Hangar (GET/PUT /api/v1/sites/kalelm-api/env): DATABASE_URL (VPS Postgres, sslmode=no-verify for the self-signed cert), PAYLOAD_SECRET, CRON_SECRET (also the worker’s bearer), MEILI_HOST, MEILI_MASTER_KEY, OLLAMA_URL, NEXT_PUBLIC_SERVER_URL, FRONTEND_URL, optional SEARCH_VECTORS=off, PIPELINE_WORKER_PYTHON. VPS secrets: /srv/kalelm/.env (MEILI_MASTER_KEY, POSTGRES_PASSWORD). Tokens for Hangar, Cloudflare and Neon are yours; rotate the ones pasted into chat.

Runbook

Do	How
Deploy backend / frontend	HANGAR_TOKEN=… scripts/deploy-hangar.sh backend (or frontend)
Change backend env	PUT …/sites/kalelm-api/env with the full set (it replaces); app restarts
Start/stop a worker on the backend host	panel → عامل على هذا الخادم; elsewhere: systemd/launchd unit in worker/README.md
Pause all workers	panel → إيقاف مؤقت
Turn semantic search off/on	set/remove SEARCH_VECTORS=off in Hangar env
Rebuild search indexes	pnpm migrate:wp index (keyword), sentences / passages (vectors) with MEILI_HOST pointing at the target; or copy data.ms between same-version instances
Fix one material’s search entry	scripts/reindex-material.ts <id>
See what hits the database	on the VPS: docker compose exec -T postgres psql -U kalelm -d kalelm < top-queries.sql
See traffic	Cloudflare dashboard → mohanad.xyz → Analytics
Prove the repo is sound	pnpm verify

Known limits (test deployment)

• Search VPS has 3.7 GB RAM for a 25 GB search database: keyword fast, semantic ~1.5 s.
• Hostnames use sslip.io and the test domain; robots.txt disallows all crawlers until launch (launch block inside the file).
• Bot Fight Mode still needs one toggle in Cloudflare → Security → Bots.
• ESLint config is broken repo-wide; tsc and pnpm verify are the checks that work.
• Backend has one commit and ~230 uncommitted files; frontend and infra are not in git at all.

Maximizing organization — the plan

1. Commit everything, now. One repository kalelm with backend/, frontend/, worker/, infra/, docs/. Until then the only copy of most of the code is this laptop.
2. One way to deploy each thing, checked in: scripts/deploy-hangar.sh, infra/search-vps/README.md. Nothing deployed by hand-typed commands again.
3. Config in files, secrets in one place. Every non-secret setting in git; secrets in Hangar env / /srv/kalelm/.env only, with .env.example files listing every key. A password manager entry per token.
4. Docs next to code, one page per audience: this file (map + runbook), worker/README.md (operators of workers), infra/search-vps/README.md (the box). Update the page in the same PR as the change.
5. Real names. Custom domains via Cloudflare instead of sslip.io; search./embed./db. hostnames on your domain.
6. The gate stays the definition of done. pnpm verify before every deploy; add a check with every bug fix (the pipeline section is the pattern).
7. Observe three things weekly: Cloudflare analytics, top-queries.sql, the transcription panel’s failures list.