backup-agent/docs/superpowers/specs/2026-07-25-backup-recovery-system-design.md

Backup & Recovery System Design

Date: 2026-07-25

Goal

Reliable, automated backup of /home/srv/files/content (including MariaDB running in Docker) to a local Borg repo + offsite Scaleway S3 mirror, plus a documented, scriptable recovery path so downtime is short and predictable.

Current State

• old/borg.sh, old/maria.sh — scripts currently live on the server.
• borg-backup.sh, dump_db.sh (repo root) — improved replacements, not yet deployed. Add: encrypted repo (repokey-blake2 via BORG_PASSCOMMAND), flock-based locking, preflight checks, container health polling, atomic dump staging/swap, non-transactional table warnings, integrity checks (borg check, daily archive-only / weekly --verify-data), retry/timeout wrapping, log rotation.
• No restore tooling and no scheduling exist yet.

dump_db.sh is kept as-is. borg-backup.sh had one functional change made during this design: it no longer stops the MariaDB container. dump_db.sh already produces a transactionally-consistent dump via --single-transaction, so a live logical dump is safe without stopping anything (this only covers InnoDB tables — the script already warns if it finds non-InnoDB tables). The remaining risk was borg also archiving MariaDB's raw on-disk data directory while it's being written to; that's resolved by excluding it from the archive entirely (drop a .nobackup marker file in it — borg-backup.sh already passes --exclude-if-present .nobackup), so only the logical dump ever gets backed up, and restore only ever needs the dump anyway. Net effect: daily backups run with zero MariaDB downtime. This exclusion step needs to be added to the one-time setup, documented in the runbook.

Architecture

cron (daily) → borg-backup.sh
                 ├─ dump_db.sh (mysqldump per-DB, atomic staging swap;
                 │              container stays up throughout)
                 ├─ borg create (encrypted, zstd) → local repo
                 │              (raw MariaDB data dir excluded via .nobackup;
                 │               only the logical dump is archived)
                 ├─ borg prune + compact
                 ├─ borg check (daily fast / weekly full --verify-data)
                 └─ rclone sync → Scaleway S3 (offsite mirror)

restore.sh (manual, run on demand during an incident)
  mode=full   : fresh/broken server → extract full content dir from archive
                → restore all DBs → start container
  mode=db     : single database → restore one .sql from a chosen archive
  mode=file   : single file/dir → borg extract path, no DB involved

Components

All at repo root, deployed to the server alongside the existing scripts.

borg-backup.sh (existing, one change: no longer stops MariaDB)

Orchestrates the daily backup: dump → archive → prune → compact → check → offsite sync. The container is never stopped — the dump step alone produces a consistent backup, and the raw data directory is excluded from the archive. Already has locking, timeouts, health-check pings, and cleanup-on-exit logic (cleanup now only intervenes if the container happens to be down for an unrelated reason, restarting it defensively).

dump_db.sh (existing, unchanged)

Runs inside borg-backup.sh step 1. Dumps each non-system database plus users/grants to a staging dir, verifies each dump file, then atomically swaps staging into place. Deployed at /home/srv/files/content/mariadb/dump_db.sh.

restore.sh (new)

restore.sh full  [--archive NAME] [--force]
restore.sh db    <db_name> [--archive NAME]
restore.sh file  <path-within-target> [--archive NAME] [--dest DIR]
restore.sh --dry-run <above args>
restore.sh --list-archives

• Defaults --archive to the most recent archive in the repo.
• full: refuses to run if $TARGET is non-empty unless --force is passed (protects against overwriting a working system by accident). Extracts the full archive over $TARGET, then restores every .sql dump found under mariadb/dump/ using root DB credentials, then starts the mariadb container and waits for it to become healthy.
• db <name>: extracts only mariadb/dump/<name>.sql from the archive, prompts for the database name to be typed again as confirmation (it's destructive — drops/recreates the DB), restores it with root credentials.
• file <path>: borg extract of a single path from the archive into --dest (default: a scratch dir under /tmp), no DB or container involvement — safe, non-destructive.
• --dry-run: prints exactly what would run (archive chosen, paths extracted, commands) without touching anything.
• Same logging conventions as borg-backup.sh (/var/log/borg/restore-*.log), non-zero exit on any failure, no partial/silent restores — a failed step aborts before touching the "good" copy where possible.
• Uses the same BORG_PASSCOMMAND / passphrase file as backups.
• Restore DB operations authenticate as root via MYSQL_ROOT_PASSWORD or a root password file (mirrors how dump_db.sh resolves credentials), since restore needs full privileges (CREATE/DROP) that the limited backup user used for dumping does not have.

RUNBOOK.md (new)

Single operational document covering:

1. One-time setup (run once, by hand, not scripted): borg init --encryption=repokey-blake2, create /root/.borg-passphrase (chmod 600), create the backup MariaDB user with the grants dump_db.sh needs (SELECT, LOCK TABLES, SHOW VIEW, TRIGGER, PROCESS, RELOAD) and its password file /root/.mariadb-backup.pw, drop a .nobackup marker file into MariaDB's Docker volume data directory (excludes raw DB files from the archive — only the logical dump under mariadb/dump/ gets backed up), configure rclone for the scaleway remote, set the real healthcheck URL.
2. Deploying the scripts: where each file goes, permissions, chmod +x.
3. Scheduling: cron entry (daily, off-peak hours) invoking borg-backup.sh, plus how to check the last run (/var/log/borg/, healthcheck dashboard).
4. Day-2 operations: listing archives, checking repo size/health, rotating the passphrase, what to do if the lockfile is stale.
5. Recovery procedures — one clearly-numbered walkthrough per scenario (full disaster, single DB, single file), written so someone unfamiliar with the internals can follow it step-by-step under pressure, including exact restore.sh invocations and what to verify afterward (container healthy, app responds, row counts sane).
6. Restore drill cadence: recommend a quarterly test restore into a scratch location to confirm backups are actually usable.

Error Handling & Safety

• restore.sh never overwrites live data without an explicit --force (full) or typed confirmation (db).
• All destructive steps are logged before execution.
• --dry-run available for every mode.
• Failures abort immediately (set -euo pipefail), matching the existing scripts' style; partial state is called out explicitly in the log.

Testing

No automated test framework — these are ops scripts against real infrastructure. Verification consists of:

• restore.sh --dry-run runs against the real repo to sanity-check archive selection and command construction.
• Quarterly real restore drill into a scratch directory (documented in the runbook), confirming dumps are complete and importable.

Out of Scope

• No changes to dump_db.sh, and no changes to borg-backup.sh beyond removing the DB stop/start (see Components above).
• No monitoring/alerting beyond the existing healthcheck ping.
• No multi-server / multi-target generalization — this is specific to /home/srv/files/content.