kbe
d6f30c506e
Existing borg-backup.sh/dump_db.sh, old/ reference scripts, and the brainstormed design spec for restore tooling + runbook.
1045 lines
33 KiB
Markdown
1045 lines
33 KiB
Markdown
# Restore Tooling & Runbook Implementation Plan
|
|
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
|
|
**Goal:** Build `restore.sh` (full / single-database / single-file recovery) and `RUNBOOK.md` so the backup system built around `borg-backup.sh` and `dump_db.sh` has a documented, scriptable, fast recovery path.
|
|
|
|
**Architecture:** `restore.sh` is a standalone bash script mirroring the config/logging conventions of `borg-backup.sh`. It offers three subcommands (`full`, `db <name>`, `file <path>`) plus `--list-archives`, all reading from the same encrypted Borg repo. Destructive modes require explicit confirmation (`--force` for `full`, typed database name or `--yes` for `db`). `RUNBOOK.md` documents one-time setup, deployment, scheduling, day-2 ops, and step-by-step recovery for all three scenarios.
|
|
|
|
**Tech Stack:** bash, borg, docker, mysql/mariadb client, rclone (setup only, not used by restore.sh).
|
|
|
|
## Global Constraints
|
|
|
|
- Target directory: `/home/srv/files/content` (`TARGET`)
|
|
- Repo: `/home/srv/files/backups/borg-2025` (`NAME=borg-2025`)
|
|
- DB container name: `mariadb`
|
|
- Borg passphrase file: `/root/.borg-passphrase`, chmod 600, read via `BORG_PASSCOMMAND`
|
|
- Backup DB user `backup`, password file `/root/.mariadb-backup.pw` (used by `dump_db.sh`, unrelated to restore)
|
|
- Dump directory inside target: `mariadb/dump`, one `.sql` file per database, plus `00-users-and-grants.sql`
|
|
- MariaDB is never stopped for backups; its raw data directory is excluded from the archive via a `.nobackup` marker file — restores are always driven from the logical dump, never from raw archived DB files
|
|
- `restore.sh` must never silently overwrite live data: `full` refuses to run against a non-empty target without `--force`; `db` requires the database name to be typed back for confirmation unless `--yes` is passed
|
|
- Restore DB operations authenticate as `root` (env `MYSQL_ROOT_PASSWORD` or a password file, default `/root/.mariadb-root.pw`) since restore needs privileges the `backup` user does not have
|
|
- Logs go to `/var/log/borg/restore-*.log` (override via `RESTORE_LOGDIR` for testing), same naming convention as `borg-backup.sh`
|
|
|
|
---
|
|
|
|
## File Structure
|
|
|
|
- Create: `restore.sh` — the recovery script (repo root, alongside `borg-backup.sh`)
|
|
- Create: `RUNBOOK.md` — operational documentation (repo root)
|
|
- Create: `tests/test_restore.sh` — bash test script exercising `restore.sh` against mocked `borg`/`docker`/`mysql`/`mariadb` binaries; grows across tasks
|
|
- Create: `tests/lib/setup_mocks.sh` — shared helper that builds the mock binaries into a temp `$PATH` directory
|
|
|
|
## Task 1: `restore.sh` scaffold — config, logging, arg parsing, usage
|
|
|
|
**Files:**
|
|
- Create: `restore.sh`
|
|
- Create: `tests/test_restore.sh`
|
|
|
|
**Interfaces:**
|
|
- Produces: `log(msg)`, `step(msg)`, `die(msg)` (exits 1), `run_cmd(cmd...)`, `usage()`, `parse_common_flags(args...)` (sets globals `ARCHIVE_OVERRIDE`, `FORCE`, `YES`, `DRY_RUN`, `DEST`), `main(args...)`. Config globals: `NAME`, `REPO`, `TARGET`, `DB_CONTAINER`, `DB_START_TIMEOUT`, `BORG_PASSPHRASE_FILE`, `ROOT_PASSWORD_FILE`, `DUMP_SUBDIR`, `ARCHIVE_TARGET_PATH`, `LOGDIR`, `LOGFILE`.
|
|
|
|
- [ ] **Step 1: Write `restore.sh` scaffold**
|
|
|
|
```bash
|
|
#!/bin/bash
|
|
# =============================================================================
|
|
# Restore tooling for the borg-backup.sh / dump_db.sh backup system.
|
|
# Modes:
|
|
# restore.sh full [--archive NAME] [--force] [--dry-run]
|
|
# restore.sh db <db_name> [--archive NAME] [--yes] [--dry-run]
|
|
# restore.sh file <path-within-target> [--archive NAME] [--dest DIR] [--dry-run]
|
|
# restore.sh --list-archives
|
|
# =============================================================================
|
|
|
|
set -euo pipefail
|
|
|
|
export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$PATH"
|
|
umask 077
|
|
|
|
# ========================= CONFIGURATION =========================
|
|
|
|
NAME="borg-2025"
|
|
REPO="/home/srv/files/backups/$NAME"
|
|
TARGET="/home/srv/files/content"
|
|
ARCHIVE_TARGET_PATH="${TARGET#/}"
|
|
|
|
DB_CONTAINER="mariadb"
|
|
DB_START_TIMEOUT=180
|
|
|
|
BORG_PASSPHRASE_FILE="${BORG_PASSPHRASE_FILE:-/root/.borg-passphrase}"
|
|
ROOT_PASSWORD_FILE="${ROOT_PASSWORD_FILE:-/root/.mariadb-root.pw}"
|
|
DUMP_SUBDIR="mariadb/dump"
|
|
|
|
LOGDIR="${RESTORE_LOGDIR:-/var/log/borg}"
|
|
mkdir -p "$LOGDIR" 2>/dev/null || LOGDIR="/tmp"
|
|
LOGFILE="$LOGDIR/restore-$(date +%Y-%m-%d-%H%M%S).log"
|
|
|
|
export BORG_REPO="$REPO"
|
|
export BORG_PASSCOMMAND="cat $BORG_PASSPHRASE_FILE"
|
|
|
|
# =================================================================
|
|
|
|
log() {
|
|
local line
|
|
line="[$(date '+%F %T')] $*"
|
|
echo "$line"
|
|
echo "$line" >> "$LOGFILE" 2>/dev/null || true
|
|
}
|
|
|
|
step() { echo; echo "=== $* ==="; }
|
|
|
|
die() { log "ERROR: $*"; exit 1; }
|
|
|
|
run_cmd() {
|
|
log "[RUN] $*"
|
|
"$@"
|
|
}
|
|
|
|
usage() {
|
|
cat <<'EOF'
|
|
Usage:
|
|
restore.sh full [--archive NAME] [--force] [--dry-run]
|
|
restore.sh db <db_name> [--archive NAME] [--yes] [--dry-run]
|
|
restore.sh file <path-within-target> [--archive NAME] [--dest DIR] [--dry-run]
|
|
restore.sh --list-archives
|
|
restore.sh -h | --help
|
|
EOF
|
|
}
|
|
|
|
DRY_RUN=false
|
|
FORCE=false
|
|
YES=false
|
|
ARCHIVE_OVERRIDE=""
|
|
DEST=""
|
|
|
|
parse_common_flags() {
|
|
while [[ $# -gt 0 ]]; do
|
|
case "$1" in
|
|
--archive) ARCHIVE_OVERRIDE="$2"; shift 2 ;;
|
|
--force) FORCE=true; shift ;;
|
|
--yes) YES=true; shift ;;
|
|
--dry-run) DRY_RUN=true; shift ;;
|
|
--dest) DEST="$2"; shift 2 ;;
|
|
*) die "unknown flag: $1" ;;
|
|
esac
|
|
done
|
|
}
|
|
|
|
main() {
|
|
local cmd="${1:-}"
|
|
case "$cmd" in
|
|
"")
|
|
usage
|
|
exit 1
|
|
;;
|
|
-h|--help)
|
|
usage
|
|
exit 0
|
|
;;
|
|
--list-archives)
|
|
shift
|
|
parse_common_flags "$@"
|
|
list_archives
|
|
;;
|
|
full)
|
|
shift
|
|
parse_common_flags "$@"
|
|
cmd_full
|
|
;;
|
|
db)
|
|
shift
|
|
local dbname="${1:-}"
|
|
[[ -n "$dbname" ]] || die "db: missing <db_name>"
|
|
shift
|
|
parse_common_flags "$@"
|
|
cmd_db "$dbname"
|
|
;;
|
|
file)
|
|
shift
|
|
local relpath="${1:-}"
|
|
[[ -n "$relpath" ]] || die "file: missing <path>"
|
|
shift
|
|
parse_common_flags "$@"
|
|
cmd_file "$relpath"
|
|
;;
|
|
*)
|
|
usage
|
|
die "unknown command: $cmd"
|
|
;;
|
|
esac
|
|
}
|
|
|
|
main "$@"
|
|
```
|
|
|
|
- [ ] **Step 2: Make it executable and syntax-check**
|
|
|
|
Run: `chmod +x restore.sh && bash -n restore.sh`
|
|
Expected: no output, exit 0.
|
|
|
|
- [ ] **Step 3: Write `tests/test_restore.sh`**
|
|
|
|
```bash
|
|
#!/bin/bash
|
|
set -uo pipefail
|
|
|
|
HERE="$(cd "$(dirname "$0")" && pwd)"
|
|
RESTORE="$HERE/../restore.sh"
|
|
FAILURES=0
|
|
|
|
assert_eq() {
|
|
local expected="$1" actual="$2" msg="$3"
|
|
if [[ "$expected" != "$actual" ]]; then
|
|
echo "FAIL: $msg (expected '$expected', got '$actual')"
|
|
FAILURES=$((FAILURES + 1))
|
|
else
|
|
echo "PASS: $msg"
|
|
fi
|
|
}
|
|
|
|
assert_contains() {
|
|
local haystack="$1" needle="$2" msg="$3"
|
|
if [[ "$haystack" != *"$needle"* ]]; then
|
|
echo "FAIL: $msg (expected to contain '$needle', got '$haystack')"
|
|
FAILURES=$((FAILURES + 1))
|
|
else
|
|
echo "PASS: $msg"
|
|
fi
|
|
}
|
|
|
|
test_help_exits_zero() {
|
|
local out rc
|
|
out="$(bash "$RESTORE" -h)"
|
|
rc=$?
|
|
assert_eq "0" "$rc" "help exits 0"
|
|
assert_contains "$out" "Usage:" "help prints usage"
|
|
}
|
|
|
|
test_no_args_exits_one() {
|
|
local rc
|
|
bash "$RESTORE" >/dev/null 2>&1
|
|
rc=$?
|
|
assert_eq "1" "$rc" "no args exits 1"
|
|
}
|
|
|
|
test_unknown_command_exits_one() {
|
|
local rc
|
|
bash "$RESTORE" bogus >/dev/null 2>&1
|
|
rc=$?
|
|
assert_eq "1" "$rc" "unknown command exits 1"
|
|
}
|
|
|
|
test_help_exits_zero
|
|
test_no_args_exits_one
|
|
test_unknown_command_exits_one
|
|
|
|
echo "-----"
|
|
if [[ "$FAILURES" -gt 0 ]]; then
|
|
echo "$FAILURES failure(s)"
|
|
exit 1
|
|
fi
|
|
echo "All tests passed"
|
|
```
|
|
|
|
- [ ] **Step 4: Run the tests**
|
|
|
|
Run: `chmod +x tests/test_restore.sh && RESTORE_LOGDIR=/tmp/restore-test-logs bash tests/test_restore.sh`
|
|
Expected: three `PASS:` lines, then `All tests passed`, exit 0.
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add restore.sh tests/test_restore.sh
|
|
git commit -m "feat: add restore.sh scaffold with arg parsing and usage"
|
|
```
|
|
|
|
---
|
|
|
|
## Task 2: Archive listing and resolution
|
|
|
|
**Files:**
|
|
- Modify: `restore.sh` (add functions before `main()`)
|
|
- Modify: `tests/test_restore.sh`
|
|
- Create: `tests/lib/setup_mocks.sh`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `die()`, `log()` from Task 1.
|
|
- Produces: `resolve_archive()` (sets global `RESOLVED_ARCHIVE`, honors `ARCHIVE_OVERRIDE`), `list_archives()`.
|
|
|
|
- [ ] **Step 1: Write `tests/lib/setup_mocks.sh`**
|
|
|
|
```bash
|
|
#!/bin/bash
|
|
# Builds fake borg/docker/mysql/mariadb executables into $1 so restore.sh
|
|
# can be exercised without real infrastructure. Each mock logs its
|
|
# invocation to $1/mock.log.
|
|
|
|
setup_mock_bin() {
|
|
local dir="$1"
|
|
mkdir -p "$dir"
|
|
local mocklog="$dir/mock.log"
|
|
: > "$mocklog"
|
|
|
|
cat > "$dir/borg" <<EOF
|
|
#!/bin/bash
|
|
echo "borg \$*" >> "$mocklog"
|
|
case "\$1" in
|
|
list)
|
|
echo "host-2026-01-01T00-00-00"
|
|
echo "host-2026-06-01T00-00-00"
|
|
;;
|
|
extract)
|
|
shift
|
|
paths=()
|
|
for a in "\$@"; do
|
|
case "\$a" in
|
|
-*) ;;
|
|
*::*) ;;
|
|
*) paths+=("\$a") ;;
|
|
esac
|
|
done
|
|
for p in "\${paths[@]}"; do
|
|
if [[ "\$p" == *.* ]]; then
|
|
mkdir -p "\$(dirname "\$p")"
|
|
printf 'mock-content\n' > "\$p"
|
|
else
|
|
mkdir -p "\$p"
|
|
touch "\$p/RESTORED_MARKER"
|
|
fi
|
|
done
|
|
;;
|
|
*) ;;
|
|
esac
|
|
exit 0
|
|
EOF
|
|
|
|
cat > "$dir/docker" <<EOF
|
|
#!/bin/bash
|
|
echo "docker \$*" >> "$mocklog"
|
|
case "\$1" in
|
|
inspect)
|
|
if [[ "\$*" == *"State.Running"* ]]; then
|
|
echo true
|
|
else
|
|
echo healthy
|
|
fi
|
|
;;
|
|
start|stop)
|
|
exit 0
|
|
;;
|
|
exec)
|
|
case "\$*" in
|
|
*"command -v mariadb"*) exit 0 ;;
|
|
*) cat >/dev/null 2>&1 || true; exit 0 ;;
|
|
esac
|
|
;;
|
|
*) ;;
|
|
esac
|
|
exit 0
|
|
EOF
|
|
|
|
cat > "$dir/mysql" <<EOF
|
|
#!/bin/bash
|
|
echo "mysql \$*" >> "$mocklog"
|
|
cat >/dev/null 2>&1 || true
|
|
exit 0
|
|
EOF
|
|
cp "$dir/mysql" "$dir/mariadb"
|
|
|
|
chmod +x "$dir/borg" "$dir/docker" "$dir/mysql" "$dir/mariadb"
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 2: Add `resolve_archive()` and `list_archives()` to `restore.sh`**
|
|
|
|
Insert directly above the `usage()` function:
|
|
|
|
```bash
|
|
resolve_archive() {
|
|
if [[ -n "$ARCHIVE_OVERRIDE" ]]; then
|
|
RESOLVED_ARCHIVE="$ARCHIVE_OVERRIDE"
|
|
return 0
|
|
fi
|
|
RESOLVED_ARCHIVE="$(borg list --short --lock-wait 60 2>/dev/null | tail -n1)"
|
|
[[ -n "$RESOLVED_ARCHIVE" ]] || die "no archives found in repo $REPO"
|
|
}
|
|
|
|
list_archives() {
|
|
borg list --lock-wait 60
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 3: Syntax-check**
|
|
|
|
Run: `bash -n restore.sh`
|
|
Expected: no output, exit 0.
|
|
|
|
- [ ] **Step 4: Add a test for `--list-archives`**
|
|
|
|
Append to `tests/test_restore.sh`, just above the `test_help_exits_zero` calls at the bottom:
|
|
|
|
```bash
|
|
source "$HERE/lib/setup_mocks.sh"
|
|
|
|
test_list_archives() {
|
|
local mockdir out
|
|
mockdir="$(mktemp -d)"
|
|
setup_mock_bin "$mockdir"
|
|
out="$(PATH="$mockdir:$PATH" bash "$RESTORE" --list-archives)"
|
|
assert_contains "$out" "host-2026-01-01T00-00-00" "list-archives shows first archive"
|
|
assert_contains "$out" "host-2026-06-01T00-00-00" "list-archives shows latest archive"
|
|
rm -rf "$mockdir"
|
|
}
|
|
```
|
|
|
|
And add the call `test_list_archives` next to the other `test_*` invocations (before the `echo "-----"` line).
|
|
|
|
- [ ] **Step 5: Run the tests**
|
|
|
|
Run: `RESTORE_LOGDIR=/tmp/restore-test-logs bash tests/test_restore.sh`
|
|
Expected: all `PASS:` lines including the two new ones, `All tests passed`, exit 0.
|
|
|
|
- [ ] **Step 6: Commit**
|
|
|
|
```bash
|
|
git add restore.sh tests/test_restore.sh tests/lib/setup_mocks.sh
|
|
git commit -m "feat: add archive listing and resolution to restore.sh"
|
|
```
|
|
|
|
---
|
|
|
|
## Task 3: `file` mode (non-destructive single path restore)
|
|
|
|
**Files:**
|
|
- Modify: `restore.sh`
|
|
- Modify: `tests/test_restore.sh`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `resolve_archive()`, `RESOLVED_ARCHIVE`, `ARCHIVE_TARGET_PATH`, `REPO`, `run_cmd()`, `log()`, `step()`, global `DRY_RUN`/`DEST` from Tasks 1-2.
|
|
- Produces: `extract_path(rel_path, dest_dir, archive)` (echoes the final absolute path of the extracted item), `cmd_file(rel_path)`.
|
|
|
|
- [ ] **Step 1: Add `extract_path()` and `cmd_file()` to `restore.sh`**
|
|
|
|
Insert directly above `usage()`, after `list_archives()`:
|
|
|
|
```bash
|
|
extract_path() {
|
|
local rel_path="$1" dest_dir="$2" archive="$3"
|
|
mkdir -p "$dest_dir"
|
|
( cd "$dest_dir" && run_cmd borg extract --lock-wait 600 "${REPO}::${archive}" "${ARCHIVE_TARGET_PATH}/${rel_path}" )
|
|
echo "${dest_dir%/}/${ARCHIVE_TARGET_PATH}/${rel_path}"
|
|
}
|
|
|
|
cmd_file() {
|
|
local rel_path="$1" dest
|
|
resolve_archive
|
|
dest="${DEST:-/tmp/restore-file-$$}"
|
|
step "Restoring '$rel_path' from archive $RESOLVED_ARCHIVE into $dest"
|
|
if [[ "$DRY_RUN" == true ]]; then
|
|
log "[DRY-RUN] would extract ${ARCHIVE_TARGET_PATH}/${rel_path} from ${REPO}::${RESOLVED_ARCHIVE} into $dest"
|
|
return 0
|
|
fi
|
|
local final
|
|
final="$(extract_path "$rel_path" "$dest" "$RESOLVED_ARCHIVE")"
|
|
log "Restored file available at: $final"
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 2: Syntax-check**
|
|
|
|
Run: `bash -n restore.sh`
|
|
Expected: no output, exit 0.
|
|
|
|
- [ ] **Step 3: Add tests for `file` mode**
|
|
|
|
Append to `tests/test_restore.sh` (near `test_list_archives`):
|
|
|
|
```bash
|
|
test_file_restore_extracts_to_dest() {
|
|
local mockdir dest out final
|
|
mockdir="$(mktemp -d)"
|
|
dest="$(mktemp -d)"
|
|
setup_mock_bin "$mockdir"
|
|
out="$(PATH="$mockdir:$PATH" bash "$RESTORE" file photos/img.jpg --dest "$dest")"
|
|
final="$dest/home/srv/files/content/photos/img.jpg"
|
|
assert_contains "$out" "Restored file available at: $final" "file mode reports final path"
|
|
if [[ -f "$final" ]]; then
|
|
echo "PASS: extracted file exists on disk"
|
|
else
|
|
echo "FAIL: extracted file missing at $final"
|
|
FAILURES=$((FAILURES + 1))
|
|
fi
|
|
rm -rf "$mockdir" "$dest"
|
|
}
|
|
|
|
test_file_restore_dry_run_makes_no_borg_call() {
|
|
local mockdir dest
|
|
mockdir="$(mktemp -d)"
|
|
dest="$(mktemp -d)"
|
|
setup_mock_bin "$mockdir"
|
|
PATH="$mockdir:$PATH" bash "$RESTORE" file photos/img.jpg --dest "$dest" --dry-run >/dev/null
|
|
if [[ -s "$mockdir/mock.log" ]] && grep -q "^borg extract" "$mockdir/mock.log"; then
|
|
echo "FAIL: dry-run invoked borg extract"
|
|
FAILURES=$((FAILURES + 1))
|
|
else
|
|
echo "PASS: dry-run made no borg extract call"
|
|
fi
|
|
rm -rf "$mockdir" "$dest"
|
|
}
|
|
```
|
|
|
|
Add `test_file_restore_extracts_to_dest` and `test_file_restore_dry_run_makes_no_borg_call` to the list of test invocations.
|
|
|
|
- [ ] **Step 4: Run the tests**
|
|
|
|
Run: `RESTORE_LOGDIR=/tmp/restore-test-logs bash tests/test_restore.sh`
|
|
Expected: all tests pass, including the two new ones.
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add restore.sh tests/test_restore.sh
|
|
git commit -m "feat: add non-destructive file-restore mode to restore.sh"
|
|
```
|
|
|
|
---
|
|
|
|
## Task 4: `db` mode (single-database restore with confirmation)
|
|
|
|
**Files:**
|
|
- Modify: `restore.sh`
|
|
- Modify: `tests/test_restore.sh`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `resolve_archive()`, `RESOLVED_ARCHIVE`, `extract_path()`, `DUMP_SUBDIR`, `DB_CONTAINER`, `ROOT_PASSWORD_FILE`, global `YES`/`DRY_RUN` from Tasks 1-3.
|
|
- Produces: `detect_client()` (sets `CLIENT_BIN`), `get_root_creds()` (sets `DB_PASS`), `confirm_or_abort(dbname)`, `restore_single_db(sqlfile, dbname)`, `cmd_db(dbname)`.
|
|
|
|
- [ ] **Step 1: Add the db-mode functions to `restore.sh`**
|
|
|
|
Insert directly above `usage()`, after `cmd_file()`:
|
|
|
|
```bash
|
|
detect_client() {
|
|
if docker exec "$DB_CONTAINER" sh -c 'command -v mariadb' >/dev/null 2>&1; then
|
|
CLIENT_BIN="mariadb"
|
|
else
|
|
CLIENT_BIN="mysql"
|
|
fi
|
|
}
|
|
|
|
get_root_creds() {
|
|
if [[ -n "${MYSQL_ROOT_PASSWORD:-}" ]]; then
|
|
DB_PASS="$MYSQL_ROOT_PASSWORD"
|
|
elif [[ -r "$ROOT_PASSWORD_FILE" ]]; then
|
|
DB_PASS="$(< "$ROOT_PASSWORD_FILE")"
|
|
else
|
|
die "no root DB password: set MYSQL_ROOT_PASSWORD or create $ROOT_PASSWORD_FILE (chmod 600)"
|
|
fi
|
|
}
|
|
|
|
confirm_or_abort() {
|
|
local dbname="$1" typed
|
|
[[ "$YES" == true ]] && return 0
|
|
echo "This will DROP/overwrite database '$dbname'. Type the database name to confirm:"
|
|
read -r typed
|
|
[[ "$typed" == "$dbname" ]] || die "confirmation did not match '$dbname' - aborting"
|
|
}
|
|
|
|
restore_single_db() {
|
|
local sqlfile="$1" dbname="$2"
|
|
[[ -s "$sqlfile" ]] || die "dump file missing or empty: $sqlfile"
|
|
run_cmd docker exec -e MYSQL_PWD="$DB_PASS" "$DB_CONTAINER" \
|
|
"$CLIENT_BIN" -u root -e "CREATE DATABASE IF NOT EXISTS \`$dbname\`;"
|
|
log "[RUN] docker exec -i ... $CLIENT_BIN -u root $dbname < $sqlfile"
|
|
docker exec -i -e MYSQL_PWD="$DB_PASS" "$DB_CONTAINER" \
|
|
"$CLIENT_BIN" -u root "$dbname" < "$sqlfile"
|
|
}
|
|
|
|
cmd_db() {
|
|
local dbname="$1" dumpfile dest
|
|
resolve_archive
|
|
step "Restoring database '$dbname' from archive $RESOLVED_ARCHIVE"
|
|
if [[ "$DRY_RUN" == true ]]; then
|
|
log "[DRY-RUN] would extract ${ARCHIVE_TARGET_PATH}/${DUMP_SUBDIR}/${dbname}.sql from ${REPO}::${RESOLVED_ARCHIVE}"
|
|
log "[DRY-RUN] would DROP/recreate database '$dbname' and import the dump using root credentials"
|
|
return 0
|
|
fi
|
|
confirm_or_abort "$dbname"
|
|
dest="/tmp/restore-db-$$"
|
|
dumpfile="$(extract_path "${DUMP_SUBDIR}/${dbname}.sql" "$dest" "$RESOLVED_ARCHIVE")"
|
|
detect_client
|
|
get_root_creds
|
|
restore_single_db "$dumpfile" "$dbname"
|
|
log "Database '$dbname' restored from $dumpfile"
|
|
rm -rf "$dest"
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 2: Syntax-check**
|
|
|
|
Run: `bash -n restore.sh`
|
|
Expected: no output, exit 0.
|
|
|
|
- [ ] **Step 3: Add tests for `db` mode**
|
|
|
|
Append to `tests/test_restore.sh`:
|
|
|
|
```bash
|
|
test_db_restore_with_yes_runs_full_sequence() {
|
|
local mockdir out
|
|
mockdir="$(mktemp -d)"
|
|
setup_mock_bin "$mockdir"
|
|
out="$(PATH="$mockdir:$PATH" ROOT_PASSWORD_FILE="$mockdir/rootpw" bash -c '
|
|
echo "rootpass" > "'"$mockdir"'/rootpw"
|
|
bash "'"$RESTORE"'" db shopdb --yes
|
|
')"
|
|
assert_contains "$out" "Database 'shopdb' restored from" "db mode reports success"
|
|
if grep -q "CREATE DATABASE IF NOT EXISTS" "$mockdir/mock.log"; then
|
|
echo "PASS: db mode issued CREATE DATABASE"
|
|
else
|
|
echo "FAIL: db mode did not issue CREATE DATABASE"
|
|
FAILURES=$((FAILURES + 1))
|
|
fi
|
|
rm -rf "$mockdir"
|
|
}
|
|
|
|
test_db_restore_dry_run_skips_confirmation_and_calls() {
|
|
local mockdir out
|
|
mockdir="$(mktemp -d)"
|
|
setup_mock_bin "$mockdir"
|
|
out="$(PATH="$mockdir:$PATH" bash "$RESTORE" db shopdb --dry-run </dev/null)"
|
|
assert_contains "$out" "DRY-RUN" "db dry-run prints DRY-RUN plan"
|
|
if [[ -s "$mockdir/mock.log" ]]; then
|
|
echo "FAIL: dry-run invoked a mock binary"
|
|
FAILURES=$((FAILURES + 1))
|
|
else
|
|
echo "PASS: dry-run made no external calls"
|
|
fi
|
|
rm -rf "$mockdir"
|
|
}
|
|
|
|
test_db_restore_aborts_on_wrong_confirmation() {
|
|
local mockdir rc
|
|
mockdir="$(mktemp -d)"
|
|
setup_mock_bin "$mockdir"
|
|
set +e
|
|
echo "wrongname" | PATH="$mockdir:$PATH" bash "$RESTORE" db shopdb >/dev/null 2>&1
|
|
rc=$?
|
|
set -e
|
|
assert_eq "1" "$rc" "db mode aborts on mismatched confirmation"
|
|
rm -rf "$mockdir"
|
|
}
|
|
```
|
|
|
|
Add all three test names to the invocation list.
|
|
|
|
- [ ] **Step 4: Run the tests**
|
|
|
|
Run: `RESTORE_LOGDIR=/tmp/restore-test-logs bash tests/test_restore.sh`
|
|
Expected: all tests pass, including the three new ones.
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add restore.sh tests/test_restore.sh
|
|
git commit -m "feat: add single-database restore mode to restore.sh"
|
|
```
|
|
|
|
---
|
|
|
|
## Task 5: `full` mode (disaster recovery) and shared container helpers
|
|
|
|
**Files:**
|
|
- Modify: `restore.sh`
|
|
- Modify: `tests/test_restore.sh`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `resolve_archive()`, `detect_client()`, `get_root_creds()`, `restore_single_db()`, `ARCHIVE_TARGET_PATH`, `TARGET`, `DUMP_SUBDIR`, `DB_CONTAINER`, `DB_START_TIMEOUT`, global `FORCE`/`DRY_RUN` from Tasks 1-4.
|
|
- Produces: `container_running(name)`, `container_health(name)`, `wait_for_container(name)`, `start_db()`, `cmd_full()`.
|
|
|
|
- [ ] **Step 1: Add container helpers and `cmd_full()` to `restore.sh`**
|
|
|
|
Insert directly above `usage()`, after `cmd_db()`:
|
|
|
|
```bash
|
|
container_running() {
|
|
[[ "$(docker inspect -f '{{.State.Running}}' "$1" 2>/dev/null || echo false)" == "true" ]]
|
|
}
|
|
|
|
container_health() {
|
|
docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' \
|
|
"$1" 2>/dev/null || echo unknown
|
|
}
|
|
|
|
wait_for_container() {
|
|
local name="$1" deadline=$((SECONDS + DB_START_TIMEOUT)) health
|
|
while (( SECONDS < deadline )); do
|
|
if container_running "$name"; then
|
|
health="$(container_health "$name")"
|
|
case "$health" in
|
|
healthy|none) log "$name is up (health: $health)"; return 0 ;;
|
|
unhealthy) log "WARNING: $name reports unhealthy" ;;
|
|
esac
|
|
fi
|
|
sleep 3
|
|
done
|
|
return 1
|
|
}
|
|
|
|
start_db() {
|
|
docker start "$DB_CONTAINER" >/dev/null 2>&1 || true
|
|
wait_for_container "$DB_CONTAINER"
|
|
}
|
|
|
|
cmd_full() {
|
|
local staging dumpdir f dbname
|
|
resolve_archive
|
|
step "Full restore from archive $RESOLVED_ARCHIVE into $TARGET"
|
|
|
|
if [[ -d "$TARGET" ]] && [[ -n "$(ls -A "$TARGET" 2>/dev/null)" ]] && [[ "$FORCE" != true ]]; then
|
|
die "$TARGET is not empty - pass --force to overwrite (existing data will be replaced)"
|
|
fi
|
|
|
|
if [[ "$DRY_RUN" == true ]]; then
|
|
log "[DRY-RUN] would extract full ${ARCHIVE_TARGET_PATH} tree from ${REPO}::${RESOLVED_ARCHIVE} into $TARGET"
|
|
log "[DRY-RUN] would restore every *.sql dump under ${DUMP_SUBDIR}/ using root credentials"
|
|
log "[DRY-RUN] would start $DB_CONTAINER and wait for it to become healthy"
|
|
return 0
|
|
fi
|
|
|
|
staging="/tmp/restore-full-$$"
|
|
mkdir -p "$staging"
|
|
( cd "$staging" && run_cmd borg extract --lock-wait 600 "${REPO}::${RESOLVED_ARCHIVE}" "${ARCHIVE_TARGET_PATH}" )
|
|
|
|
mkdir -p "$(dirname "$TARGET")"
|
|
rm -rf "${TARGET:?}"/* 2>/dev/null || true
|
|
mkdir -p "$TARGET"
|
|
run_cmd cp -a "${staging}/${ARCHIVE_TARGET_PATH}/." "$TARGET/"
|
|
rm -rf "$staging"
|
|
|
|
detect_client
|
|
get_root_creds
|
|
|
|
dumpdir="${TARGET}/${DUMP_SUBDIR}"
|
|
[[ -d "$dumpdir" ]] || die "no dump directory found after extract: $dumpdir"
|
|
|
|
if [[ -f "${dumpdir}/00-users-and-grants.sql" ]]; then
|
|
step "Restoring users and grants"
|
|
docker exec -i -e MYSQL_PWD="$DB_PASS" "$DB_CONTAINER" \
|
|
"$CLIENT_BIN" -u root < "${dumpdir}/00-users-and-grants.sql"
|
|
fi
|
|
|
|
for f in "$dumpdir"/*.sql; do
|
|
[[ -e "$f" ]] || continue
|
|
dbname="$(basename "$f" .sql)"
|
|
[[ "$dbname" == "00-users-and-grants" ]] && continue
|
|
restore_single_db "$f" "$dbname"
|
|
done
|
|
|
|
step "Starting $DB_CONTAINER"
|
|
start_db || die "CRITICAL: $DB_CONTAINER did not come up after restore"
|
|
|
|
log "Full restore complete from archive $RESOLVED_ARCHIVE"
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 2: Syntax-check**
|
|
|
|
Run: `bash -n restore.sh`
|
|
Expected: no output, exit 0.
|
|
|
|
- [ ] **Step 3: Add tests for `full` mode**
|
|
|
|
Append to `tests/test_restore.sh`:
|
|
|
|
```bash
|
|
test_full_restore_refuses_nonempty_target_without_force() {
|
|
local mockdir target rc
|
|
mockdir="$(mktemp -d)"
|
|
target="$(mktemp -d)"
|
|
touch "$target/existing-file"
|
|
setup_mock_bin "$mockdir"
|
|
set +e
|
|
PATH="$mockdir:$PATH" TARGET_OVERRIDE=1 bash -c '
|
|
sed "s#^TARGET=\"/home/srv/files/content\"#TARGET=\"'"$target"'\"#; s#^ARCHIVE_TARGET_PATH=.*#ARCHIVE_TARGET_PATH=\"\${TARGET#/}\"#" "'"$RESTORE"'" > "'"$mockdir"'/restore_patched.sh"
|
|
bash "'"$mockdir"'/restore_patched.sh" full
|
|
' >/dev/null 2>&1
|
|
rc=$?
|
|
set -e
|
|
assert_eq "1" "$rc" "full mode refuses non-empty target without --force"
|
|
rm -rf "$mockdir" "$target"
|
|
}
|
|
|
|
test_full_restore_dry_run_makes_no_calls() {
|
|
local mockdir target out
|
|
mockdir="$(mktemp -d)"
|
|
target="$(mktemp -d)"
|
|
setup_mock_bin "$mockdir"
|
|
out="$(PATH="$mockdir:$PATH" bash -c '
|
|
sed "s#^TARGET=\"/home/srv/files/content\"#TARGET=\"'"$target"'\"#; s#^ARCHIVE_TARGET_PATH=.*#ARCHIVE_TARGET_PATH=\"\${TARGET#/}\"#" "'"$RESTORE"'" > "'"$mockdir"'/restore_patched.sh"
|
|
bash "'"$mockdir"'/restore_patched.sh" full --dry-run
|
|
')"
|
|
assert_contains "$out" "DRY-RUN" "full dry-run prints DRY-RUN plan"
|
|
if [[ -s "$mockdir/mock.log" ]]; then
|
|
echo "FAIL: full dry-run invoked a mock binary"
|
|
FAILURES=$((FAILURES + 1))
|
|
else
|
|
echo "PASS: full dry-run made no external calls"
|
|
fi
|
|
rm -rf "$mockdir" "$target"
|
|
}
|
|
```
|
|
|
|
Add `test_full_restore_refuses_nonempty_target_without_force` and `test_full_restore_dry_run_makes_no_calls` to the invocation list.
|
|
|
|
- [ ] **Step 4: Run the tests**
|
|
|
|
Run: `RESTORE_LOGDIR=/tmp/restore-test-logs bash tests/test_restore.sh`
|
|
Expected: all tests pass, including the two new ones.
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add restore.sh tests/test_restore.sh
|
|
git commit -m "feat: add full disaster-recovery mode to restore.sh"
|
|
```
|
|
|
|
---
|
|
|
|
## Task 6: `RUNBOOK.md`
|
|
|
|
**Files:**
|
|
- Create: `RUNBOOK.md`
|
|
|
|
**Interfaces:**
|
|
- Consumes: final `restore.sh` CLI surface from Tasks 1-5 (`full`, `db <name>`, `file <path>`, `--list-archives`, flags `--archive`/`--force`/`--yes`/`--dry-run`/`--dest`), and `borg-backup.sh`'s config (`NAME=borg-2025`, `REPO=/home/srv/files/backups/borg-2025`, `TARGET=/home/srv/files/content`, `BORG_PASSPHRASE_FILE=/root/.borg-passphrase`).
|
|
|
|
- [ ] **Step 1: Write `RUNBOOK.md`**
|
|
|
|
```markdown
|
|
# Backup & Recovery Runbook
|
|
|
|
Covers `borg-backup.sh` (daily backup), `dump_db.sh` (MariaDB logical
|
|
dumps, invoked by the backup script), and `restore.sh` (recovery).
|
|
|
|
## 1. One-Time Setup
|
|
|
|
Run once, by hand, on the server:
|
|
|
|
1. Initialize the encrypted Borg repo:
|
|
```bash
|
|
mkdir -p /home/srv/files/backups
|
|
borg init --encryption=repokey-blake2 /home/srv/files/backups/borg-2025
|
|
```
|
|
2. Create the passphrase file used by both backup and restore:
|
|
```bash
|
|
echo 'your-strong-passphrase' > /root/.borg-passphrase
|
|
chmod 600 /root/.borg-passphrase
|
|
```
|
|
3. Create the MariaDB `backup` user used by `dump_db.sh` (read-only, no
|
|
stop/lock of the server required thanks to `--single-transaction`):
|
|
```sql
|
|
CREATE USER 'backup'@'%' IDENTIFIED BY 'a-strong-password';
|
|
GRANT SELECT, LOCK TABLES, SHOW VIEW, TRIGGER, PROCESS, RELOAD ON *.* TO 'backup'@'%';
|
|
```
|
|
```bash
|
|
echo 'a-strong-password' > /root/.mariadb-backup.pw
|
|
chmod 600 /root/.mariadb-backup.pw
|
|
```
|
|
4. Create the root password file used only by `restore.sh` (restore needs
|
|
CREATE/DROP privileges the `backup` user does not have):
|
|
```bash
|
|
echo 'the-mariadb-root-password' > /root/.mariadb-root.pw
|
|
chmod 600 /root/.mariadb-root.pw
|
|
```
|
|
5. Exclude MariaDB's raw data directory from the archive. Find the
|
|
directory bind-mounted into the container as its datadir and drop a
|
|
marker file in it:
|
|
```bash
|
|
touch /home/srv/files/content/mariadb/data/.nobackup
|
|
```
|
|
This is what allows backups to run with the container up: only the
|
|
logical dump under `mariadb/dump/` is ever archived or restored from.
|
|
6. Configure the `scaleway` rclone remote:
|
|
```bash
|
|
rclone config
|
|
# create a remote named "scaleway", type S3, matching your Scaleway
|
|
# Object Storage credentials and region
|
|
```
|
|
7. Set a real healthcheck URL in `borg-backup.sh` (`HEALTHCHECK_URL=`), for
|
|
example from https://healthchecks.io.
|
|
|
|
## 2. Deploying the Scripts
|
|
|
|
Copy `borg-backup.sh`, `dump_db.sh`, and `restore.sh` to the server (e.g.
|
|
`/opt/backup-agent/`), and `dump_db.sh` additionally to
|
|
`/home/srv/files/content/mariadb/dump_db.sh` (this exact path is what
|
|
`borg-backup.sh` invokes). Make all three executable:
|
|
|
|
```bash
|
|
chmod +x /opt/backup-agent/borg-backup.sh /opt/backup-agent/restore.sh
|
|
chmod +x /home/srv/files/content/mariadb/dump_db.sh
|
|
```
|
|
|
|
## 3. Scheduling
|
|
|
|
Add a cron entry to run the backup daily, off-peak:
|
|
|
|
```
|
|
# /etc/cron.d/borg-backup
|
|
30 2 * * * root /opt/backup-agent/borg-backup.sh >> /var/log/borg/cron.log 2>&1
|
|
```
|
|
|
|
Check the last run:
|
|
|
|
```bash
|
|
ls -lt /var/log/borg/backup-*.log | head -1 # latest log file
|
|
tail -50 /var/log/borg/backup-*.log # inspect it
|
|
```
|
|
|
|
Or watch the healthcheck dashboard configured in step 1.7 — a missed or
|
|
failed run pages/alerts there.
|
|
|
|
## 4. Day-2 Operations
|
|
|
|
List archives:
|
|
|
|
```bash
|
|
./restore.sh --list-archives
|
|
```
|
|
|
|
Check repo size and health:
|
|
|
|
```bash
|
|
BORG_PASSCOMMAND="cat /root/.borg-passphrase" borg info /home/srv/files/backups/borg-2025
|
|
```
|
|
|
|
Rotate the passphrase (creates a new key, re-encrypts nothing — old
|
|
archives still need the old passphrase to read, so keep both until fully
|
|
migrated):
|
|
|
|
```bash
|
|
BORG_PASSCOMMAND="cat /root/.borg-passphrase" borg key change-passphrase /home/srv/files/backups/borg-2025
|
|
```
|
|
|
|
Stale lockfile (backup or restore aborted mid-run and left the repo
|
|
locked):
|
|
|
|
```bash
|
|
BORG_PASSCOMMAND="cat /root/.borg-passphrase" borg break-lock /home/srv/files/backups/borg-2025
|
|
```
|
|
|
|
## 5. Recovery Procedures
|
|
|
|
All `restore.sh` commands accept `--dry-run` to preview exactly what would
|
|
happen without touching anything, and `--archive NAME` to target a
|
|
specific archive instead of the latest (see archive names via
|
|
`--list-archives`).
|
|
|
|
### 5.1 Full disaster recovery (new or wiped server)
|
|
|
|
Use when the whole server/container is gone and you're rebuilding from
|
|
scratch.
|
|
|
|
```bash
|
|
# 1. Reinstall borg, docker, and the mariadb container image/compose file
|
|
# (not covered by restore.sh - this is infra provisioning).
|
|
# 2. Restore the passphrase file (from your password manager / secondary
|
|
# backup - it is NOT stored in the repo it protects) to
|
|
# /root/.borg-passphrase, and the root DB password to
|
|
# /root/.mariadb-root.pw.
|
|
# 3. Preview:
|
|
./restore.sh full --dry-run
|
|
# 4. Run for real (refuses if /home/srv/files/content is non-empty):
|
|
./restore.sh full --force
|
|
```
|
|
|
|
This extracts the full content tree from the archive, restores every
|
|
database dump (users/grants first), and starts the `mariadb` container,
|
|
waiting for it to report healthy.
|
|
|
|
**Verify afterward:**
|
|
- `docker ps` shows `mariadb` running and healthy.
|
|
- The application responds normally.
|
|
- Spot-check row counts on a couple of tables against what you'd expect.
|
|
|
|
### 5.2 Single database restore
|
|
|
|
Use when one database got corrupted or someone ran a bad migration/query
|
|
against it — this **drops and recreates** that database.
|
|
|
|
```bash
|
|
./restore.sh db shopdb --dry-run # preview
|
|
./restore.sh db shopdb # prompts: type "shopdb" to confirm
|
|
```
|
|
|
|
Non-interactive (e.g. scripted from a monitoring alert): add `--yes` to
|
|
skip the typed confirmation.
|
|
|
|
**Verify afterward:** connect to the database and check the tables/row
|
|
counts you expect.
|
|
|
|
### 5.3 Single file/directory restore
|
|
|
|
Use for accidental deletion of a file, or to inspect an old version — this
|
|
never touches the running database or container.
|
|
|
|
```bash
|
|
./restore.sh file path/relative/to/content/some-file.txt --dest /tmp/recovered
|
|
```
|
|
|
|
The final location of the recovered item is printed at the end (it lands
|
|
under `/tmp/recovered/home/srv/files/content/...` — Borg preserves the
|
|
absolute path it was archived with).
|
|
|
|
## 6. Restore Drill Cadence
|
|
|
|
Quarterly, run a real `full` restore into a scratch directory (not
|
|
`/home/srv/files/content`) to confirm backups are actually usable:
|
|
|
|
```bash
|
|
mkdir -p /tmp/restore-drill
|
|
BORG_PASSCOMMAND="cat /root/.borg-passphrase" \
|
|
borg extract --lock-wait 600 /home/srv/files/backups/borg-2025::$(./restore.sh --list-archives | tail -1) \
|
|
--destination /tmp/restore-drill # (or adapt restore.sh's TARGET for a one-off dry run into scratch)
|
|
```
|
|
|
|
Confirm the dump files under `mariadb/dump/` are present, non-empty, and
|
|
importable (`mysql -u root -p < mariadb/dump/somedb.sql` against a
|
|
throwaway MariaDB container). Log the drill date and outcome somewhere
|
|
durable (e.g. the healthcheck dashboard's notes, or a team wiki page).
|
|
```
|
|
|
|
- [ ] **Step 2: Cross-check the runbook against the script's actual CLI**
|
|
|
|
Run: `grep -oE 'restore\.sh [a-z-]+' RUNBOOK.md | sort -u`
|
|
Expected output includes `restore.sh --list-archives`, `restore.sh db`,
|
|
`restore.sh file`, `restore.sh full` — confirming every implemented
|
|
subcommand is documented. Run `./restore.sh -h` and confirm no flag or
|
|
subcommand appears there that's missing from `RUNBOOK.md`.
|
|
|
|
- [ ] **Step 3: Commit**
|
|
|
|
```bash
|
|
git add RUNBOOK.md
|
|
git commit -m "docs: add operational runbook for backup and recovery"
|
|
```
|
|
|
|
---
|
|
|
|
## Self-Review Notes
|
|
|
|
- **Spec coverage:** one-time setup (Task 6 §1), deployment (§2), cron
|
|
scheduling (§3), day-2 ops (§4), all three recovery scenarios (§5.1-5.3),
|
|
quarterly drill (§6) — all present. `restore.sh` covers `full`/`db`/`file`
|
|
with `--force`/typed-confirmation/`--yes` safety guards and `--dry-run`
|
|
for every mode, per the approved design.
|
|
- **No changes** were made to `dump_db.sh`; `borg-backup.sh`'s only change
|
|
(removing the DB stop/start) was already implemented and committed
|
|
before this plan.
|
|
- **Type/name consistency:** `RESOLVED_ARCHIVE`, `CLIENT_BIN`, `DB_PASS`,
|
|
`ARCHIVE_TARGET_PATH`, `DUMP_SUBDIR` are defined once (Tasks 1-2) and
|
|
reused with the same names through Task 5; `restore_single_db()` is
|
|
defined in Task 4 and reused unchanged by `cmd_full()` in Task 5.
|