fix: start mariadb before restoring DBs in cmd_full, add lock/preflight
Final review of the restore.sh branch found cmd_full restored every database via `docker exec` before starting the container, which fails immediately in the exact scenario full restore exists for (a freshly rebuilt, stopped container). Reorders to extract -> start container -> restore DBs. Also, while touching cmd_full: - Extract directly into place (cd / && borg extract) instead of staging a full copy under /tmp then cp -a'ing it into $TARGET - halves disk usage and restore time. - Replace `rm -rf "$TARGET"/*` with `find "$TARGET" -mindepth 1 -delete` so dotfiles don't survive a --force wipe. - Add acquire_lock() (shares borg-backup.sh's lockfile so a restore and the nightly backup cron can't run concurrently) and preflight() (passphrase file readable, repo reachable) before any real work in full/db/file. Test isolation: mock borg/docker/mysql/mariadb consistently via a BASH_ENV shim (previously only db-mode's test worked around PATH shadowing by a real docker binary; every mocked test needed it, and a missing `flock` mock broke everything once acquire_lock was added, since flock(1) doesn't exist on macOS). Tests also isolate LOCKFILE and BORG_PASSPHRASE_FILE to throwaway paths. RUNBOOK.md: fix the quarterly drill command (borg extract has no --destination flag, and needs `borg list --short` for a bare archive name), reword the full-restore --force comment which read backwards, and document the MYSQL_ROOT_PASSWORD/RESTORE_LOGDIR env overrides and where restore logs land. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
+26
-9
@@ -115,7 +115,14 @@ BORG_PASSCOMMAND="cat /root/.borg-passphrase" borg break-lock /home/srv/files/ba
|
||||
All `restore.sh` commands accept `--dry-run` to preview exactly what would
|
||||
happen without touching anything, and `--archive NAME` to target a
|
||||
specific archive instead of the latest (see archive names via
|
||||
`--list-archives`).
|
||||
`--list-archives`). Root DB credentials come from `MYSQL_ROOT_PASSWORD` in
|
||||
the environment if set, otherwise from `/root/.mariadb-root.pw` — set
|
||||
whichever is more convenient for how you're invoking it. Restore logs go
|
||||
to `/var/log/borg/restore-*.log` (the `RESTORE_LOGDIR` environment
|
||||
variable overrides the directory, mainly useful for testing). `full` and
|
||||
`db` share `borg-backup.sh`'s lockfile, so a restore refuses to start
|
||||
while the nightly backup is mid-run (and vice versa) rather than racing
|
||||
it.
|
||||
|
||||
### 5.1 Full disaster recovery (new or wiped server)
|
||||
|
||||
@@ -131,13 +138,16 @@ scratch.
|
||||
# /root/.mariadb-root.pw.
|
||||
# 3. Preview:
|
||||
./restore.sh full --dry-run
|
||||
# 4. Run for real (refuses if /home/srv/files/content is non-empty):
|
||||
# 4. Run for real. --force is only required if /home/srv/files/content
|
||||
# already has data in it (e.g. a stale mount); omit it on a genuinely
|
||||
# empty/fresh server:
|
||||
./restore.sh full --force
|
||||
```
|
||||
|
||||
This extracts the full content tree from the archive, restores every
|
||||
database dump (users/grants first), and starts the `mariadb` container,
|
||||
waiting for it to report healthy.
|
||||
This extracts the full content tree from the archive, starts the
|
||||
`mariadb` container and waits for it to report healthy, then restores
|
||||
every database dump (users/grants first) — the container must be running
|
||||
before any of the dump restores, which is why it starts first.
|
||||
|
||||
**Verify afterward:**
|
||||
- `docker ps` shows `mariadb` running and healthy.
|
||||
@@ -178,11 +188,18 @@ absolute path it was archived with).
|
||||
Quarterly, run a real `full` restore into a scratch directory (not
|
||||
`/home/srv/files/content`) to confirm backups are actually usable:
|
||||
|
||||
`borg extract` always extracts into the current directory (there's no
|
||||
`--destination` flag — this is why `restore.sh` itself `cd`s into the
|
||||
destination before extracting), and the archive name must come from
|
||||
`borg list --short` (plain `borg list` prints a formatted line, not a bare
|
||||
name), so:
|
||||
|
||||
```bash
|
||||
mkdir -p /tmp/restore-drill
|
||||
BORG_PASSCOMMAND="cat /root/.borg-passphrase" \
|
||||
borg extract --lock-wait 600 /home/srv/files/backups/borg-2025::$(./restore.sh --list-archives | tail -1) \
|
||||
--destination /tmp/restore-drill # (or adapt restore.sh's TARGET for a one-off dry run into scratch)
|
||||
mkdir -p /tmp/restore-drill && cd /tmp/restore-drill
|
||||
export BORG_REPO=/home/srv/files/backups/borg-2025
|
||||
export BORG_PASSCOMMAND="cat /root/.borg-passphrase"
|
||||
LATEST=$(borg list --short | tail -1)
|
||||
borg extract --lock-wait 600 "::$LATEST"
|
||||
```
|
||||
|
||||
Confirm the dump files under `mariadb/dump/` are present, non-empty, and
|
||||
|
||||
Reference in New Issue
Block a user