fix: start mariadb before restoring DBs in cmd_full, add lock/preflight

Final review of the restore.sh branch found cmd_full restored every
database via `docker exec` before starting the container, which fails
immediately in the exact scenario full restore exists for (a freshly
rebuilt, stopped container). Reorders to extract -> start container ->
restore DBs.

Also, while touching cmd_full:
- Extract directly into place (cd / && borg extract) instead of staging
  a full copy under /tmp then cp -a'ing it into $TARGET - halves disk
  usage and restore time.
- Replace `rm -rf "$TARGET"/*` with `find "$TARGET" -mindepth 1 -delete`
  so dotfiles don't survive a --force wipe.
- Add acquire_lock() (shares borg-backup.sh's lockfile so a restore and
  the nightly backup cron can't run concurrently) and preflight()
  (passphrase file readable, repo reachable) before any real work in
  full/db/file.

Test isolation: mock borg/docker/mysql/mariadb consistently via a
BASH_ENV shim (previously only db-mode's test worked around PATH
shadowing by a real docker binary; every mocked test needed it, and a
missing `flock` mock broke everything once acquire_lock was added,
since flock(1) doesn't exist on macOS). Tests also isolate LOCKFILE and
BORG_PASSPHRASE_FILE to throwaway paths.

RUNBOOK.md: fix the quarterly drill command (borg extract has no
--destination flag, and needs `borg list --short` for a bare archive
name), reword the full-restore --force comment which read backwards,
and document the MYSQL_ROOT_PASSWORD/RESTORE_LOGDIR env overrides and
where restore logs land.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
kbe
2026-07-25 19:42:56 +02:00
co-authored by Claude Sonnet 5
parent 8d7a603615
commit 592ef45bab
4 changed files with 138 additions and 46 deletions
+26 -9
View File
@@ -115,7 +115,14 @@ BORG_PASSCOMMAND="cat /root/.borg-passphrase" borg break-lock /home/srv/files/ba
All `restore.sh` commands accept `--dry-run` to preview exactly what would
happen without touching anything, and `--archive NAME` to target a
specific archive instead of the latest (see archive names via
`--list-archives`).
`--list-archives`). Root DB credentials come from `MYSQL_ROOT_PASSWORD` in
the environment if set, otherwise from `/root/.mariadb-root.pw` — set
whichever is more convenient for how you're invoking it. Restore logs go
to `/var/log/borg/restore-*.log` (the `RESTORE_LOGDIR` environment
variable overrides the directory, mainly useful for testing). `full` and
`db` share `borg-backup.sh`'s lockfile, so a restore refuses to start
while the nightly backup is mid-run (and vice versa) rather than racing
it.
### 5.1 Full disaster recovery (new or wiped server)
@@ -131,13 +138,16 @@ scratch.
# /root/.mariadb-root.pw.
# 3. Preview:
./restore.sh full --dry-run
# 4. Run for real (refuses if /home/srv/files/content is non-empty):
# 4. Run for real. --force is only required if /home/srv/files/content
# already has data in it (e.g. a stale mount); omit it on a genuinely
# empty/fresh server:
./restore.sh full --force
```
This extracts the full content tree from the archive, restores every
database dump (users/grants first), and starts the `mariadb` container,
waiting for it to report healthy.
This extracts the full content tree from the archive, starts the
`mariadb` container and waits for it to report healthy, then restores
every database dump (users/grants first) — the container must be running
before any of the dump restores, which is why it starts first.
**Verify afterward:**
- `docker ps` shows `mariadb` running and healthy.
@@ -178,11 +188,18 @@ absolute path it was archived with).
Quarterly, run a real `full` restore into a scratch directory (not
`/home/srv/files/content`) to confirm backups are actually usable:
`borg extract` always extracts into the current directory (there's no
`--destination` flag — this is why `restore.sh` itself `cd`s into the
destination before extracting), and the archive name must come from
`borg list --short` (plain `borg list` prints a formatted line, not a bare
name), so:
```bash
mkdir -p /tmp/restore-drill
BORG_PASSCOMMAND="cat /root/.borg-passphrase" \
borg extract --lock-wait 600 /home/srv/files/backups/borg-2025::$(./restore.sh --list-archives | tail -1) \
--destination /tmp/restore-drill # (or adapt restore.sh's TARGET for a one-off dry run into scratch)
mkdir -p /tmp/restore-drill && cd /tmp/restore-drill
export BORG_REPO=/home/srv/files/backups/borg-2025
export BORG_PASSCOMMAND="cat /root/.borg-passphrase"
LATEST=$(borg list --short | tail -1)
borg extract --lock-wait 600 "::$LATEST"
```
Confirm the dump files under `mariadb/dump/` are present, non-empty, and