Files

T

kbeandClaude Sonnet 5 592ef45bab fix: start mariadb before restoring DBs in cmd_full, add lock/preflight

Final review of the restore.sh branch found cmd_full restored every
database via `docker exec` before starting the container, which fails
immediately in the exact scenario full restore exists for (a freshly
rebuilt, stopped container). Reorders to extract -> start container ->
restore DBs.

Also, while touching cmd_full:
- Extract directly into place (cd / && borg extract) instead of staging
  a full copy under /tmp then cp -a'ing it into $TARGET - halves disk
  usage and restore time.
- Replace `rm -rf "$TARGET"/*` with `find "$TARGET" -mindepth 1 -delete`
  so dotfiles don't survive a --force wipe.
- Add acquire_lock() (shares borg-backup.sh's lockfile so a restore and
  the nightly backup cron can't run concurrently) and preflight()
  (passphrase file readable, repo reachable) before any real work in
  full/db/file.

Test isolation: mock borg/docker/mysql/mariadb consistently via a
BASH_ENV shim (previously only db-mode's test worked around PATH
shadowing by a real docker binary; every mocked test needed it, and a
missing `flock` mock broke everything once acquire_lock was added,
since flock(1) doesn't exist on macOS). Tests also isolate LOCKFILE and
BORG_PASSPHRASE_FILE to throwaway paths.

RUNBOOK.md: fix the quarterly drill command (borg extract has no
--destination flag, and needs `borg list --short` for a bare archive
name), reword the full-restore --force comment which read backwards,
and document the MYSQL_ROOT_PASSWORD/RESTORE_LOGDIR env overrides and
where restore logs land.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

2026-07-25 19:42:56 +02:00

7.2 KiB

Raw Blame History

Backup & Recovery Runbook

Covers borg-backup.sh (daily backup), dump_db.sh (MariaDB logical dumps, invoked by the backup script), and restore.sh (recovery).

1. One-Time Setup

Run once, by hand, on the server:

Initialize the encrypted Borg repo:

mkdir -p /home/srv/files/backups
borg init --encryption=repokey-blake2 /home/srv/files/backups/borg-2025

Create the passphrase file used by both backup and restore:

echo 'your-strong-passphrase' > /root/.borg-passphrase
chmod 600 /root/.borg-passphrase

Create the MariaDB backup user used by dump_db.sh (read-only, no stop/lock of the server required thanks to --single-transaction):

CREATE USER 'backup'@'%' IDENTIFIED BY 'a-strong-password';
GRANT SELECT, LOCK TABLES, SHOW VIEW, TRIGGER, PROCESS, RELOAD ON *.* TO 'backup'@'%';

echo 'a-strong-password' > /root/.mariadb-backup.pw
chmod 600 /root/.mariadb-backup.pw

Create the root password file used only by restore.sh (restore needs CREATE/DROP privileges the backup user does not have):
```
echo 'the-mariadb-root-password' > /root/.mariadb-root.pw
chmod 600 /root/.mariadb-root.pw
```
Exclude MariaDB's raw data directory from the archive. Find the directory bind-mounted into the container as its datadir and drop a marker file in it:
```
touch /home/srv/files/content/mariadb/data/.nobackup
```
This is what allows backups to run with the container up: only the logical dump under mariadb/dump/ is ever archived or restored from.

Configure the scaleway rclone remote:

rclone config
# create a remote named "scaleway", type S3, matching your Scaleway
# Object Storage credentials and region

Set a real healthcheck URL in borg-backup.sh (HEALTHCHECK_URL=), for example from https://healthchecks.io.

2. Deploying the Scripts

Copy borg-backup.sh, dump_db.sh, and restore.sh to the server (e.g. /opt/backup-agent/), and dump_db.sh additionally to /home/srv/files/content/mariadb/dump_db.sh (this exact path is what borg-backup.sh invokes). Make all three executable:

chmod +x /opt/backup-agent/borg-backup.sh /opt/backup-agent/restore.sh
chmod +x /home/srv/files/content/mariadb/dump_db.sh

3. Scheduling

Add a cron entry to run the backup daily, off-peak:

# /etc/cron.d/borg-backup
30 2 * * * root /opt/backup-agent/borg-backup.sh >> /var/log/borg/cron.log 2>&1

Check the last run:

ls -lt /var/log/borg/backup-*.log | head -1   # latest log file
tail -50 /var/log/borg/backup-*.log           # inspect it

Or watch the healthcheck dashboard configured in step 1.7 — a missed or failed run pages/alerts there.

4. Day-2 Operations

List archives:

./restore.sh --list-archives

Check repo size and health:

BORG_PASSCOMMAND="cat /root/.borg-passphrase" borg info /home/srv/files/backups/borg-2025

Rotate the passphrase (creates a new key, re-encrypts nothing — old archives still need the old passphrase to read, so keep both until fully migrated):

BORG_PASSCOMMAND="cat /root/.borg-passphrase" borg key change-passphrase /home/srv/files/backups/borg-2025

Stale lockfile (backup or restore aborted mid-run and left the repo locked):

BORG_PASSCOMMAND="cat /root/.borg-passphrase" borg break-lock /home/srv/files/backups/borg-2025

5. Recovery Procedures

All restore.sh commands accept --dry-run to preview exactly what would happen without touching anything, and --archive NAME to target a specific archive instead of the latest (see archive names via --list-archives). Root DB credentials come from MYSQL_ROOT_PASSWORD in the environment if set, otherwise from /root/.mariadb-root.pw — set whichever is more convenient for how you're invoking it. Restore logs go to /var/log/borg/restore-*.log (the RESTORE_LOGDIR environment variable overrides the directory, mainly useful for testing). full and db share borg-backup.sh's lockfile, so a restore refuses to start while the nightly backup is mid-run (and vice versa) rather than racing it.

5.1 Full disaster recovery (new or wiped server)

Use when the whole server/container is gone and you're rebuilding from scratch.

# 1. Reinstall borg, docker, and the mariadb container image/compose file
#    (not covered by restore.sh - this is infra provisioning).
# 2. Restore the passphrase file (from your password manager / secondary
#    backup - it is NOT stored in the repo it protects) to
#    /root/.borg-passphrase, and the root DB password to
#    /root/.mariadb-root.pw.
# 3. Preview:
./restore.sh full --dry-run
# 4. Run for real. --force is only required if /home/srv/files/content
#    already has data in it (e.g. a stale mount); omit it on a genuinely
#    empty/fresh server:
./restore.sh full --force

This extracts the full content tree from the archive, starts the mariadb container and waits for it to report healthy, then restores every database dump (users/grants first) — the container must be running before any of the dump restores, which is why it starts first.

Verify afterward:

docker ps shows mariadb running and healthy.
The application responds normally.
Spot-check row counts on a couple of tables against what you'd expect.

5.2 Single database restore

Use when one database got corrupted or someone ran a bad migration/query against it — this drops and recreates that database.

./restore.sh db shopdb --dry-run     # preview
./restore.sh db shopdb                # prompts: type "shopdb" to confirm

Non-interactive (e.g. scripted from a monitoring alert): add --yes to skip the typed confirmation.

Verify afterward: connect to the database and check the tables/row counts you expect.

5.3 Single file/directory restore

Use for accidental deletion of a file, or to inspect an old version — this never touches the running database or container.

./restore.sh file path/relative/to/content/some-file.txt --dest /tmp/recovered

The final location of the recovered item is printed at the end (it lands under /tmp/recovered/home/srv/files/content/... — Borg preserves the absolute path it was archived with).

6. Restore Drill Cadence

Quarterly, run a real full restore into a scratch directory (not /home/srv/files/content) to confirm backups are actually usable:

borg extract always extracts into the current directory (there's no --destination flag — this is why restore.sh itself cds into the destination before extracting), and the archive name must come from borg list --short (plain borg list prints a formatted line, not a bare name), so:

mkdir -p /tmp/restore-drill && cd /tmp/restore-drill
export BORG_REPO=/home/srv/files/backups/borg-2025
export BORG_PASSCOMMAND="cat /root/.borg-passphrase"
LATEST=$(borg list --short | tail -1)
borg extract --lock-wait 600 "::$LATEST"

Confirm the dump files under mariadb/dump/ are present, non-empty, and importable (mysql -u root -p < mariadb/dump/somedb.sql against a throwaway MariaDB container). Log the drill date and outcome somewhere durable (e.g. the healthcheck dashboard's notes, or a team wiki page).

7.2 KiB Raw Blame History