Skip to content
Kalelm

Learned the hard way

Each of these was found by breaking something. None is derivable from the code.

Deploying

Section titled “Deploying”

  • Hangar env PUT replaces the whole set. GET it, merge, then PUT. A partial PUT deletes every other key and the app restarts without them.

  • Cloudflare cuts uploads at 100 s with a 524. Hangar usually finishes the deploy anyway. Check GET /sites/<site>/releases before re-uploading.

  • The backend bundle must include .next/static and public. Without them the admin renders blank with 404s on every chunk. scripts/deploy-hangar.sh does the copying.

  • Bundles are built on a Mac for an x86_64 host. package.json → pnpm.supportedArchitectures pulls the Linux binaries (sharp, swc); the deploy script strips the Darwin ones. The frontend gets npm ci --os=linux --cpu=x64 in a separate directory, because installing Linux binaries in place removes the Mac build tools.

  • next build needs no database. Dummy DATABASE_URL and PAYLOAD_SECRET at build time; the real ones come from Hangar env at runtime.

  • Never build in the working tree, and never edit a running shell script. A production build in .next deleted the dev server’s output from under it (every route 500); builds now go to .next-build. And bash reads scripts incrementally, so editing deploy-hangar.sh while it ran produced a syntax error mid-deploy.

  • Uploads through Cloudflare die at 100 s. The deploy script now rsyncs the bundle to the Hangar box (resumable, retried) and hands it to Hangar locally with --resolve admin.mohanad.xyz:443:127.0.0.1.

  • A CASE of string literals cannot be assigned to an enum column. Cast it: (case … end)::enum_transcription_jobs_status. The reclaim query 500’d every claim until it was.

Database

Section titled “Database”
  • Neon free tier caps egress at 5 GB/month. Crawlers on the sitemap and listing pages burned it in a day. Postgres now lives on the search VPS; infra/search-vps/top-queries.sql shows who is hitting it.
  • Self-signed Postgres cert ⇒ sslmode=no-verify. This node-postgres treats sslmode=require as full verification and refuses the connection.
  • payload migrate:create diffs against a stale drizzle snapshot (the last generated migration’s JSON), not the live database. It proposed dropping tables. Hand-scope generated migrations; see src/migrations/20260909_043541_pipeline.ts.
  • A hook that calls the Local API without req opens a second transaction. Creating a transcript deadlocked on the material’s foreign-key lock, and the cue reindex read an uncommitted row and indexed nothing. Pass req through every read and write inside hooks (src/search/sync.ts).
  • materials_texts and materials_rels scans equal to material reads means a query walking all materials with relations. The sitemap did that per request; it is cached hourly now.
Section titled “Search”
  • Meilisearch only opens a database written by the same version. Pin the image tag (v1.53.1) in infra/search-vps/docker-compose.yml; a mismatch crash-loops with exit 0 and no useful log.
  • A truncated vector index crashes Meilisearch on every semantic query, taking keyword search down for seconds each time. SEARCH_VECTORS=off in Hangar env serves keyword results for all modes while an index is being moved or rebuilt.
  • Moving data.ms is faster than re-embedding: 25 GB copied versus ~10 GPU-hours plus a day of vector-graph building on 2 vCPUs. Copy with local Meilisearch stopped for the final pass, verify with md5sum before swapping.
  • The Docker default log driver copies container stdout to disk. Streaming a 23 GB tar through docker run filled the Docker VM and took local Postgres down. Use --log-driver none for streams.
  • Busybox tail -c +N fails past 2 GiB with “Bad address”. Use dd with a block offset.
  • ssh inside a while read loop eats the loop’s stdin. Use ssh -n. Two scripts silently processed one file each because of this.
  • A shell that mentions a process name in its own command line matches pkill -f for that name. Use a self-excluding pattern like 'wor[k]er.py'.
  • Never truncate a remote file based on a size you did not actually read. A failed stat over ssh became “0” and wiped 5 GB. The copy script now refuses to act on a non-numeric size.

Worker and pipeline

Section titled “Worker and pipeline”
  • Cohere ASR on CPU is 0.4× real time; on Apple GPU ~19×; the whole pipeline with post-edit ~10×. Backlog is ~19,000 hours of audio. A rented consumer GPU does it in about a week.
  • The tashkeel ASR fine-tune breaks down on 26-second segments, three times the tokens and 14 retries. Use the default model plus the CAMeL hamza/shadda pass in worker/lite.py.
  • The LLM post-edit rewrites occasionally. worker/postedit.py keeps the original line when word content changes; it only ever accepts spelling, hamza, ta-marbuta and punctuation edits.
  • Ollama on this Mac has two servers: localhost resolves to an empty IPv6 one; models live on 127.0.0.1:11434.

Repo hygiene

Section titled “Repo hygiene”
  • pnpm verify is the gate and a Stop hook runs it. It needs local Postgres, the dev backend on 3001, the frontend on 4322, Meilisearch on 7701 and Ollama. If local Meilisearch is stopped the gate fails for that reason alone.
  • ESLint’s config crashes on every file. tsc --noEmit and the gate are the working checks.
  • docs/ is excluded from the backend tsconfig; Astro’s virtual modules would otherwise fail the type check.