Backups and restore

What exists and how it is protected

Data	Where it lives	Protection
The archive database (materials, transcripts, everything in the admin)	Postgres on the search VPS	nightly dump at 03:00 UTC, copied to the Hangar box; 14 days kept on both
Search indexes	Meilisearch on the search VPS	rebuildable from Postgres (pnpm migrate:wp index, then sentences and passages); also a copy on the owner’s Mac
Audio and video	files.kalelm.com	outside this project — confirm its own backup
Deployed releases	Hangar keeps the last 10 per site with rollback	plus the repository, once committed
Worker results in flight	results/job-<id>.json on the worker until the API accepts them	the queue re-runs a lost job

The backup script is infra/search-vps/backup.sh, installed at /srv/kalelm/backup.sh on the search VPS, run by root’s crontab. It refuses to keep a dump under 10 MB (the real one is ~74 MB) and logs to /var/log/kalelm-backup.log. First run: 2026-09-10.

Restore the database

From the Hangar box’s copy into the search VPS (replace the date):

scp [email protected]:/srv/backups/kalelm/kalelm-2026-09-10.dump /tmp/
cd /srv/kalelm
docker compose exec -T postgres psql -U kalelm -d postgres -c "drop database kalelm; create database kalelm;"
docker cp /tmp/kalelm-2026-09-10.dump kalelm-postgres-1:/tmp/
docker compose exec -T postgres pg_restore -U kalelm -d kalelm --no-owner --no-acl /tmp/kalelm-2026-09-10.dump
docker compose exec -T postgres psql -U kalelm -d kalelm -c "create extension if not exists pg_stat_statements;"

Then restart the backend on Hangar (POST /api/v1/sites/kalelm-api/app/restart) so its pool reconnects, and check pnpm verify data from a machine that can reach the database.

Into a local dev database: pg_restore --no-owner --no-acl -d "$DATABASE_URL" kalelm-2026-09-10.dump.

Restore search

If the Meilisearch volume is lost, rebuild from Postgres. Keyword first, it is minutes:

MEILI_HOST=https://search.49.12.78.95.sslip.io MEILI_MASTER_KEY=… pnpm migrate:wp index

Vectors take hours of GPU; run sentences then passages the same way from a machine with Ollama and bge-m3, or copy data.ms from the Mac’s instance if it is the same Meilisearch version (see the move notes in Learned the hard way). Set SEARCH_VECTORS=off in Hangar env until the vector indexes are complete, or semantic queries will crash the instance.

Restore an app

Hangar: POST /api/v1/sites/<site>/rollback {"release": "<id>"} or {} for the previous release. Releases are immutable, so this is instant.

What to test, and when

• Restore last night’s dump into a scratch database once a quarter and run pnpm verify data against it.
• Check /var/log/kalelm-backup.log when anything about the database changes.