Files

T

kbe d6f30c506e chore: initial import of backup scripts and design docs

Existing borg-backup.sh/dump_db.sh, old/ reference scripts, and the
brainstormed design spec for restore tooling + runbook.

2026-07-25 18:51:38 +02:00

33 KiB

Raw Blame History

Restore Tooling & Runbook Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Build restore.sh (full / single-database / single-file recovery) and RUNBOOK.md so the backup system built around borg-backup.sh and dump_db.sh has a documented, scriptable, fast recovery path.

Architecture: restore.sh is a standalone bash script mirroring the config/logging conventions of borg-backup.sh. It offers three subcommands (full, db <name>, file <path>) plus --list-archives, all reading from the same encrypted Borg repo. Destructive modes require explicit confirmation (--force for full, typed database name or --yes for db). RUNBOOK.md documents one-time setup, deployment, scheduling, day-2 ops, and step-by-step recovery for all three scenarios.

Tech Stack: bash, borg, docker, mysql/mariadb client, rclone (setup only, not used by restore.sh).

Global Constraints

Target directory: /home/srv/files/content (TARGET)
Repo: /home/srv/files/backups/borg-2025 (NAME=borg-2025)
DB container name: mariadb
Borg passphrase file: /root/.borg-passphrase, chmod 600, read via BORG_PASSCOMMAND
Backup DB user backup, password file /root/.mariadb-backup.pw (used by dump_db.sh, unrelated to restore)
Dump directory inside target: mariadb/dump, one .sql file per database, plus 00-users-and-grants.sql
MariaDB is never stopped for backups; its raw data directory is excluded from the archive via a .nobackup marker file — restores are always driven from the logical dump, never from raw archived DB files
restore.sh must never silently overwrite live data: full refuses to run against a non-empty target without --force; db requires the database name to be typed back for confirmation unless --yes is passed
Restore DB operations authenticate as root (env MYSQL_ROOT_PASSWORD or a password file, default /root/.mariadb-root.pw) since restore needs privileges the backup user does not have
Logs go to /var/log/borg/restore-*.log (override via RESTORE_LOGDIR for testing), same naming convention as borg-backup.sh

File Structure

Create: restore.sh — the recovery script (repo root, alongside borg-backup.sh)
Create: RUNBOOK.md — operational documentation (repo root)
Create: tests/test_restore.sh — bash test script exercising restore.sh against mocked borg/docker/mysql/mariadb binaries; grows across tasks
Create: tests/lib/setup_mocks.sh — shared helper that builds the mock binaries into a temp $PATH directory

Task 1: `restore.sh` scaffold — config, logging, arg parsing, usage

Files:

Create: restore.sh
Create: tests/test_restore.sh

Interfaces:

Produces: log(msg), step(msg), die(msg) (exits 1), run_cmd(cmd...), usage(), parse_common_flags(args...) (sets globals ARCHIVE_OVERRIDE, FORCE, YES, DRY_RUN, DEST), main(args...). Config globals: NAME, REPO, TARGET, DB_CONTAINER, DB_START_TIMEOUT, BORG_PASSPHRASE_FILE, ROOT_PASSWORD_FILE, DUMP_SUBDIR, ARCHIVE_TARGET_PATH, LOGDIR, LOGFILE.
Step 1: Write restore.sh scaffold

#!/bin/bash
# =============================================================================
# Restore tooling for the borg-backup.sh / dump_db.sh backup system.
# Modes:
#   restore.sh full  [--archive NAME] [--force] [--dry-run]
#   restore.sh db    <db_name> [--archive NAME] [--yes] [--dry-run]
#   restore.sh file  <path-within-target> [--archive NAME] [--dest DIR] [--dry-run]
#   restore.sh --list-archives
# =============================================================================

set -euo pipefail

export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$PATH"
umask 077

# ========================= CONFIGURATION =========================

NAME="borg-2025"
REPO="/home/srv/files/backups/$NAME"
TARGET="/home/srv/files/content"
ARCHIVE_TARGET_PATH="${TARGET#/}"

DB_CONTAINER="mariadb"
DB_START_TIMEOUT=180

BORG_PASSPHRASE_FILE="${BORG_PASSPHRASE_FILE:-/root/.borg-passphrase}"
ROOT_PASSWORD_FILE="${ROOT_PASSWORD_FILE:-/root/.mariadb-root.pw}"
DUMP_SUBDIR="mariadb/dump"

LOGDIR="${RESTORE_LOGDIR:-/var/log/borg}"
mkdir -p "$LOGDIR" 2>/dev/null || LOGDIR="/tmp"
LOGFILE="$LOGDIR/restore-$(date +%Y-%m-%d-%H%M%S).log"

export BORG_REPO="$REPO"
export BORG_PASSCOMMAND="cat $BORG_PASSPHRASE_FILE"

# =================================================================

log() {
    local line
    line="[$(date '+%F %T')] $*"
    echo "$line"
    echo "$line" >> "$LOGFILE" 2>/dev/null || true
}

step() { echo; echo "=== $* ==="; }

die() { log "ERROR: $*"; exit 1; }

run_cmd() {
    log "[RUN] $*"
    "$@"
}

usage() {
    cat <<'EOF'
Usage:
  restore.sh full  [--archive NAME] [--force] [--dry-run]
  restore.sh db    <db_name> [--archive NAME] [--yes] [--dry-run]
  restore.sh file  <path-within-target> [--archive NAME] [--dest DIR] [--dry-run]
  restore.sh --list-archives
  restore.sh -h | --help
EOF
}

DRY_RUN=false
FORCE=false
YES=false
ARCHIVE_OVERRIDE=""
DEST=""

parse_common_flags() {
    while [[ $# -gt 0 ]]; do
        case "$1" in
            --archive) ARCHIVE_OVERRIDE="$2"; shift 2 ;;
            --force) FORCE=true; shift ;;
            --yes) YES=true; shift ;;
            --dry-run) DRY_RUN=true; shift ;;
            --dest) DEST="$2"; shift 2 ;;
            *) die "unknown flag: $1" ;;
        esac
    done
}

main() {
    local cmd="${1:-}"
    case "$cmd" in
        "")
            usage
            exit 1
            ;;
        -h|--help)
            usage
            exit 0
            ;;
        --list-archives)
            shift
            parse_common_flags "$@"
            list_archives
            ;;
        full)
            shift
            parse_common_flags "$@"
            cmd_full
            ;;
        db)
            shift
            local dbname="${1:-}"
            [[ -n "$dbname" ]] || die "db: missing <db_name>"
            shift
            parse_common_flags "$@"
            cmd_db "$dbname"
            ;;
        file)
            shift
            local relpath="${1:-}"
            [[ -n "$relpath" ]] || die "file: missing <path>"
            shift
            parse_common_flags "$@"
            cmd_file "$relpath"
            ;;
        *)
            usage
            die "unknown command: $cmd"
            ;;
    esac
}

main "$@"

Step 2: Make it executable and syntax-check

Run: chmod +x restore.sh && bash -n restore.sh Expected: no output, exit 0.

Step 3: Write tests/test_restore.sh

#!/bin/bash
set -uo pipefail

HERE="$(cd "$(dirname "$0")" && pwd)"
RESTORE="$HERE/../restore.sh"
FAILURES=0

assert_eq() {
    local expected="$1" actual="$2" msg="$3"
    if [[ "$expected" != "$actual" ]]; then
        echo "FAIL: $msg (expected '$expected', got '$actual')"
        FAILURES=$((FAILURES + 1))
    else
        echo "PASS: $msg"
    fi
}

assert_contains() {
    local haystack="$1" needle="$2" msg="$3"
    if [[ "$haystack" != *"$needle"* ]]; then
        echo "FAIL: $msg (expected to contain '$needle', got '$haystack')"
        FAILURES=$((FAILURES + 1))
    else
        echo "PASS: $msg"
    fi
}

test_help_exits_zero() {
    local out rc
    out="$(bash "$RESTORE" -h)"
    rc=$?
    assert_eq "0" "$rc" "help exits 0"
    assert_contains "$out" "Usage:" "help prints usage"
}

test_no_args_exits_one() {
    local rc
    bash "$RESTORE" >/dev/null 2>&1
    rc=$?
    assert_eq "1" "$rc" "no args exits 1"
}

test_unknown_command_exits_one() {
    local rc
    bash "$RESTORE" bogus >/dev/null 2>&1
    rc=$?
    assert_eq "1" "$rc" "unknown command exits 1"
}

test_help_exits_zero
test_no_args_exits_one
test_unknown_command_exits_one

echo "-----"
if [[ "$FAILURES" -gt 0 ]]; then
    echo "$FAILURES failure(s)"
    exit 1
fi
echo "All tests passed"

Step 4: Run the tests

Run: chmod +x tests/test_restore.sh && RESTORE_LOGDIR=/tmp/restore-test-logs bash tests/test_restore.sh Expected: three PASS: lines, then All tests passed, exit 0.

Step 5: Commit

git add restore.sh tests/test_restore.sh
git commit -m "feat: add restore.sh scaffold with arg parsing and usage"

Task 2: Archive listing and resolution

Files:

Modify: restore.sh (add functions before main())
Modify: tests/test_restore.sh
Create: tests/lib/setup_mocks.sh

Interfaces:

Consumes: die(), log() from Task 1.
Produces: resolve_archive() (sets global RESOLVED_ARCHIVE, honors ARCHIVE_OVERRIDE), list_archives().
Step 1: Write tests/lib/setup_mocks.sh

#!/bin/bash
# Builds fake borg/docker/mysql/mariadb executables into $1 so restore.sh
# can be exercised without real infrastructure. Each mock logs its
# invocation to $1/mock.log.

setup_mock_bin() {
    local dir="$1"
    mkdir -p "$dir"
    local mocklog="$dir/mock.log"
    : > "$mocklog"

    cat > "$dir/borg" <<EOF
#!/bin/bash
echo "borg \$*" >> "$mocklog"
case "\$1" in
    list)
        echo "host-2026-01-01T00-00-00"
        echo "host-2026-06-01T00-00-00"
        ;;
    extract)
        shift
        paths=()
        for a in "\$@"; do
            case "\$a" in
                -*) ;;
                *::*) ;;
                *) paths+=("\$a") ;;
            esac
        done
        for p in "\${paths[@]}"; do
            if [[ "\$p" == *.* ]]; then
                mkdir -p "\$(dirname "\$p")"
                printf 'mock-content\n' > "\$p"
            else
                mkdir -p "\$p"
                touch "\$p/RESTORED_MARKER"
            fi
        done
        ;;
    *) ;;
esac
exit 0
EOF

    cat > "$dir/docker" <<EOF
#!/bin/bash
echo "docker \$*" >> "$mocklog"
case "\$1" in
    inspect)
        if [[ "\$*" == *"State.Running"* ]]; then
            echo true
        else
            echo healthy
        fi
        ;;
    start|stop)
        exit 0
        ;;
    exec)
        case "\$*" in
            *"command -v mariadb"*) exit 0 ;;
            *) cat >/dev/null 2>&1 || true; exit 0 ;;
        esac
        ;;
    *) ;;
esac
exit 0
EOF

    cat > "$dir/mysql" <<EOF
#!/bin/bash
echo "mysql \$*" >> "$mocklog"
cat >/dev/null 2>&1 || true
exit 0
EOF
    cp "$dir/mysql" "$dir/mariadb"

    chmod +x "$dir/borg" "$dir/docker" "$dir/mysql" "$dir/mariadb"
}

Step 2: Add resolve_archive() and list_archives() to restore.sh

Insert directly above the usage() function:

resolve_archive() {
    if [[ -n "$ARCHIVE_OVERRIDE" ]]; then
        RESOLVED_ARCHIVE="$ARCHIVE_OVERRIDE"
        return 0
    fi
    RESOLVED_ARCHIVE="$(borg list --short --lock-wait 60 2>/dev/null | tail -n1)"
    [[ -n "$RESOLVED_ARCHIVE" ]] || die "no archives found in repo $REPO"
}

list_archives() {
    borg list --lock-wait 60
}

Step 3: Syntax-check

Run: bash -n restore.sh Expected: no output, exit 0.

Step 4: Add a test for --list-archives

Append to tests/test_restore.sh, just above the test_help_exits_zero calls at the bottom:

source "$HERE/lib/setup_mocks.sh"

test_list_archives() {
    local mockdir out
    mockdir="$(mktemp -d)"
    setup_mock_bin "$mockdir"
    out="$(PATH="$mockdir:$PATH" bash "$RESTORE" --list-archives)"
    assert_contains "$out" "host-2026-01-01T00-00-00" "list-archives shows first archive"
    assert_contains "$out" "host-2026-06-01T00-00-00" "list-archives shows latest archive"
    rm -rf "$mockdir"
}

And add the call test_list_archives next to the other test_* invocations (before the echo "-----" line).

Step 5: Run the tests

Run: RESTORE_LOGDIR=/tmp/restore-test-logs bash tests/test_restore.sh Expected: all PASS: lines including the two new ones, All tests passed, exit 0.

Step 6: Commit

git add restore.sh tests/test_restore.sh tests/lib/setup_mocks.sh
git commit -m "feat: add archive listing and resolution to restore.sh"

Task 3: `file` mode (non-destructive single path restore)

Files:

Modify: restore.sh
Modify: tests/test_restore.sh

Interfaces:

Consumes: resolve_archive(), RESOLVED_ARCHIVE, ARCHIVE_TARGET_PATH, REPO, run_cmd(), log(), step(), global DRY_RUN/DEST from Tasks 1-2.
Produces: extract_path(rel_path, dest_dir, archive) (echoes the final absolute path of the extracted item), cmd_file(rel_path).
Step 1: Add extract_path() and cmd_file() to restore.sh

Insert directly above usage(), after list_archives():

extract_path() {
    local rel_path="$1" dest_dir="$2" archive="$3"
    mkdir -p "$dest_dir"
    ( cd "$dest_dir" && run_cmd borg extract --lock-wait 600 "${REPO}::${archive}" "${ARCHIVE_TARGET_PATH}/${rel_path}" )
    echo "${dest_dir%/}/${ARCHIVE_TARGET_PATH}/${rel_path}"
}

cmd_file() {
    local rel_path="$1" dest
    resolve_archive
    dest="${DEST:-/tmp/restore-file-$$}"
    step "Restoring '$rel_path' from archive $RESOLVED_ARCHIVE into $dest"
    if [[ "$DRY_RUN" == true ]]; then
        log "[DRY-RUN] would extract ${ARCHIVE_TARGET_PATH}/${rel_path} from ${REPO}::${RESOLVED_ARCHIVE} into $dest"
        return 0
    fi
    local final
    final="$(extract_path "$rel_path" "$dest" "$RESOLVED_ARCHIVE")"
    log "Restored file available at: $final"
}

Step 2: Syntax-check

Run: bash -n restore.sh Expected: no output, exit 0.

Step 3: Add tests for file mode

Append to tests/test_restore.sh (near test_list_archives):

test_file_restore_extracts_to_dest() {
    local mockdir dest out final
    mockdir="$(mktemp -d)"
    dest="$(mktemp -d)"
    setup_mock_bin "$mockdir"
    out="$(PATH="$mockdir:$PATH" bash "$RESTORE" file photos/img.jpg --dest "$dest")"
    final="$dest/home/srv/files/content/photos/img.jpg"
    assert_contains "$out" "Restored file available at: $final" "file mode reports final path"
    if [[ -f "$final" ]]; then
        echo "PASS: extracted file exists on disk"
    else
        echo "FAIL: extracted file missing at $final"
        FAILURES=$((FAILURES + 1))
    fi
    rm -rf "$mockdir" "$dest"
}

test_file_restore_dry_run_makes_no_borg_call() {
    local mockdir dest
    mockdir="$(mktemp -d)"
    dest="$(mktemp -d)"
    setup_mock_bin "$mockdir"
    PATH="$mockdir:$PATH" bash "$RESTORE" file photos/img.jpg --dest "$dest" --dry-run >/dev/null
    if [[ -s "$mockdir/mock.log" ]] && grep -q "^borg extract" "$mockdir/mock.log"; then
        echo "FAIL: dry-run invoked borg extract"
        FAILURES=$((FAILURES + 1))
    else
        echo "PASS: dry-run made no borg extract call"
    fi
    rm -rf "$mockdir" "$dest"
}

Add test_file_restore_extracts_to_dest and test_file_restore_dry_run_makes_no_borg_call to the list of test invocations.

Step 4: Run the tests

Run: RESTORE_LOGDIR=/tmp/restore-test-logs bash tests/test_restore.sh Expected: all tests pass, including the two new ones.

Step 5: Commit

git add restore.sh tests/test_restore.sh
git commit -m "feat: add non-destructive file-restore mode to restore.sh"

Task 4: `db` mode (single-database restore with confirmation)

Files:

Modify: restore.sh
Modify: tests/test_restore.sh

Interfaces:

Consumes: resolve_archive(), RESOLVED_ARCHIVE, extract_path(), DUMP_SUBDIR, DB_CONTAINER, ROOT_PASSWORD_FILE, global YES/DRY_RUN from Tasks 1-3.
Produces: detect_client() (sets CLIENT_BIN), get_root_creds() (sets DB_PASS), confirm_or_abort(dbname), restore_single_db(sqlfile, dbname), cmd_db(dbname).
Step 1: Add the db-mode functions to restore.sh

Insert directly above usage(), after cmd_file():

detect_client() {
    if docker exec "$DB_CONTAINER" sh -c 'command -v mariadb' >/dev/null 2>&1; then
        CLIENT_BIN="mariadb"
    else
        CLIENT_BIN="mysql"
    fi
}

get_root_creds() {
    if [[ -n "${MYSQL_ROOT_PASSWORD:-}" ]]; then
        DB_PASS="$MYSQL_ROOT_PASSWORD"
    elif [[ -r "$ROOT_PASSWORD_FILE" ]]; then
        DB_PASS="$(< "$ROOT_PASSWORD_FILE")"
    else
        die "no root DB password: set MYSQL_ROOT_PASSWORD or create $ROOT_PASSWORD_FILE (chmod 600)"
    fi
}

confirm_or_abort() {
    local dbname="$1" typed
    [[ "$YES" == true ]] && return 0
    echo "This will DROP/overwrite database '$dbname'. Type the database name to confirm:"
    read -r typed
    [[ "$typed" == "$dbname" ]] || die "confirmation did not match '$dbname' - aborting"
}

restore_single_db() {
    local sqlfile="$1" dbname="$2"
    [[ -s "$sqlfile" ]] || die "dump file missing or empty: $sqlfile"
    run_cmd docker exec -e MYSQL_PWD="$DB_PASS" "$DB_CONTAINER" \
        "$CLIENT_BIN" -u root -e "CREATE DATABASE IF NOT EXISTS \`$dbname\`;"
    log "[RUN] docker exec -i ... $CLIENT_BIN -u root $dbname < $sqlfile"
    docker exec -i -e MYSQL_PWD="$DB_PASS" "$DB_CONTAINER" \
        "$CLIENT_BIN" -u root "$dbname" < "$sqlfile"
}

cmd_db() {
    local dbname="$1" dumpfile dest
    resolve_archive
    step "Restoring database '$dbname' from archive $RESOLVED_ARCHIVE"
    if [[ "$DRY_RUN" == true ]]; then
        log "[DRY-RUN] would extract ${ARCHIVE_TARGET_PATH}/${DUMP_SUBDIR}/${dbname}.sql from ${REPO}::${RESOLVED_ARCHIVE}"
        log "[DRY-RUN] would DROP/recreate database '$dbname' and import the dump using root credentials"
        return 0
    fi
    confirm_or_abort "$dbname"
    dest="/tmp/restore-db-$$"
    dumpfile="$(extract_path "${DUMP_SUBDIR}/${dbname}.sql" "$dest" "$RESOLVED_ARCHIVE")"
    detect_client
    get_root_creds
    restore_single_db "$dumpfile" "$dbname"
    log "Database '$dbname' restored from $dumpfile"
    rm -rf "$dest"
}

Step 2: Syntax-check

Run: bash -n restore.sh Expected: no output, exit 0.

Step 3: Add tests for db mode

Append to tests/test_restore.sh:

test_db_restore_with_yes_runs_full_sequence() {
    local mockdir out
    mockdir="$(mktemp -d)"
    setup_mock_bin "$mockdir"
    out="$(PATH="$mockdir:$PATH" ROOT_PASSWORD_FILE="$mockdir/rootpw" bash -c '
        echo "rootpass" > "'"$mockdir"'/rootpw"
        bash "'"$RESTORE"'" db shopdb --yes
    ')"
    assert_contains "$out" "Database 'shopdb' restored from" "db mode reports success"
    if grep -q "CREATE DATABASE IF NOT EXISTS" "$mockdir/mock.log"; then
        echo "PASS: db mode issued CREATE DATABASE"
    else
        echo "FAIL: db mode did not issue CREATE DATABASE"
        FAILURES=$((FAILURES + 1))
    fi
    rm -rf "$mockdir"
}

test_db_restore_dry_run_skips_confirmation_and_calls() {
    local mockdir out
    mockdir="$(mktemp -d)"
    setup_mock_bin "$mockdir"
    out="$(PATH="$mockdir:$PATH" bash "$RESTORE" db shopdb --dry-run </dev/null)"
    assert_contains "$out" "DRY-RUN" "db dry-run prints DRY-RUN plan"
    if [[ -s "$mockdir/mock.log" ]]; then
        echo "FAIL: dry-run invoked a mock binary"
        FAILURES=$((FAILURES + 1))
    else
        echo "PASS: dry-run made no external calls"
    fi
    rm -rf "$mockdir"
}

test_db_restore_aborts_on_wrong_confirmation() {
    local mockdir rc
    mockdir="$(mktemp -d)"
    setup_mock_bin "$mockdir"
    set +e
    echo "wrongname" | PATH="$mockdir:$PATH" bash "$RESTORE" db shopdb >/dev/null 2>&1
    rc=$?
    set -e
    assert_eq "1" "$rc" "db mode aborts on mismatched confirmation"
    rm -rf "$mockdir"
}

Add all three test names to the invocation list.

Step 4: Run the tests

Run: RESTORE_LOGDIR=/tmp/restore-test-logs bash tests/test_restore.sh Expected: all tests pass, including the three new ones.

Step 5: Commit

git add restore.sh tests/test_restore.sh
git commit -m "feat: add single-database restore mode to restore.sh"

Task 5: `full` mode (disaster recovery) and shared container helpers

Files:

Modify: restore.sh
Modify: tests/test_restore.sh

Interfaces:

Consumes: resolve_archive(), detect_client(), get_root_creds(), restore_single_db(), ARCHIVE_TARGET_PATH, TARGET, DUMP_SUBDIR, DB_CONTAINER, DB_START_TIMEOUT, global FORCE/DRY_RUN from Tasks 1-4.
Produces: container_running(name), container_health(name), wait_for_container(name), start_db(), cmd_full().
Step 1: Add container helpers and cmd_full() to restore.sh

Insert directly above usage(), after cmd_db():

container_running() {
    [[ "$(docker inspect -f '{{.State.Running}}' "$1" 2>/dev/null || echo false)" == "true" ]]
}

container_health() {
    docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' \
        "$1" 2>/dev/null || echo unknown
}

wait_for_container() {
    local name="$1" deadline=$((SECONDS + DB_START_TIMEOUT)) health
    while (( SECONDS < deadline )); do
        if container_running "$name"; then
            health="$(container_health "$name")"
            case "$health" in
                healthy|none) log "$name is up (health: $health)"; return 0 ;;
                unhealthy)    log "WARNING: $name reports unhealthy" ;;
            esac
        fi
        sleep 3
    done
    return 1
}

start_db() {
    docker start "$DB_CONTAINER" >/dev/null 2>&1 || true
    wait_for_container "$DB_CONTAINER"
}

cmd_full() {
    local staging dumpdir f dbname
    resolve_archive
    step "Full restore from archive $RESOLVED_ARCHIVE into $TARGET"

    if [[ -d "$TARGET" ]] && [[ -n "$(ls -A "$TARGET" 2>/dev/null)" ]] && [[ "$FORCE" != true ]]; then
        die "$TARGET is not empty - pass --force to overwrite (existing data will be replaced)"
    fi

    if [[ "$DRY_RUN" == true ]]; then
        log "[DRY-RUN] would extract full ${ARCHIVE_TARGET_PATH} tree from ${REPO}::${RESOLVED_ARCHIVE} into $TARGET"
        log "[DRY-RUN] would restore every *.sql dump under ${DUMP_SUBDIR}/ using root credentials"
        log "[DRY-RUN] would start $DB_CONTAINER and wait for it to become healthy"
        return 0
    fi

    staging="/tmp/restore-full-$$"
    mkdir -p "$staging"
    ( cd "$staging" && run_cmd borg extract --lock-wait 600 "${REPO}::${RESOLVED_ARCHIVE}" "${ARCHIVE_TARGET_PATH}" )

    mkdir -p "$(dirname "$TARGET")"
    rm -rf "${TARGET:?}"/* 2>/dev/null || true
    mkdir -p "$TARGET"
    run_cmd cp -a "${staging}/${ARCHIVE_TARGET_PATH}/." "$TARGET/"
    rm -rf "$staging"

    detect_client
    get_root_creds

    dumpdir="${TARGET}/${DUMP_SUBDIR}"
    [[ -d "$dumpdir" ]] || die "no dump directory found after extract: $dumpdir"

    if [[ -f "${dumpdir}/00-users-and-grants.sql" ]]; then
        step "Restoring users and grants"
        docker exec -i -e MYSQL_PWD="$DB_PASS" "$DB_CONTAINER" \
            "$CLIENT_BIN" -u root < "${dumpdir}/00-users-and-grants.sql"
    fi

    for f in "$dumpdir"/*.sql; do
        [[ -e "$f" ]] || continue
        dbname="$(basename "$f" .sql)"
        [[ "$dbname" == "00-users-and-grants" ]] && continue
        restore_single_db "$f" "$dbname"
    done

    step "Starting $DB_CONTAINER"
    start_db || die "CRITICAL: $DB_CONTAINER did not come up after restore"

    log "Full restore complete from archive $RESOLVED_ARCHIVE"
}

Step 2: Syntax-check

Run: bash -n restore.sh Expected: no output, exit 0.

Step 3: Add tests for full mode

Append to tests/test_restore.sh:

test_full_restore_refuses_nonempty_target_without_force() {
    local mockdir target rc
    mockdir="$(mktemp -d)"
    target="$(mktemp -d)"
    touch "$target/existing-file"
    setup_mock_bin "$mockdir"
    set +e
    PATH="$mockdir:$PATH" TARGET_OVERRIDE=1 bash -c '
        sed "s#^TARGET=\"/home/srv/files/content\"#TARGET=\"'"$target"'\"#; s#^ARCHIVE_TARGET_PATH=.*#ARCHIVE_TARGET_PATH=\"\${TARGET#/}\"#" "'"$RESTORE"'" > "'"$mockdir"'/restore_patched.sh"
        bash "'"$mockdir"'/restore_patched.sh" full
    ' >/dev/null 2>&1
    rc=$?
    set -e
    assert_eq "1" "$rc" "full mode refuses non-empty target without --force"
    rm -rf "$mockdir" "$target"
}

test_full_restore_dry_run_makes_no_calls() {
    local mockdir target out
    mockdir="$(mktemp -d)"
    target="$(mktemp -d)"
    setup_mock_bin "$mockdir"
    out="$(PATH="$mockdir:$PATH" bash -c '
        sed "s#^TARGET=\"/home/srv/files/content\"#TARGET=\"'"$target"'\"#; s#^ARCHIVE_TARGET_PATH=.*#ARCHIVE_TARGET_PATH=\"\${TARGET#/}\"#" "'"$RESTORE"'" > "'"$mockdir"'/restore_patched.sh"
        bash "'"$mockdir"'/restore_patched.sh" full --dry-run
    ')"
    assert_contains "$out" "DRY-RUN" "full dry-run prints DRY-RUN plan"
    if [[ -s "$mockdir/mock.log" ]]; then
        echo "FAIL: full dry-run invoked a mock binary"
        FAILURES=$((FAILURES + 1))
    else
        echo "PASS: full dry-run made no external calls"
    fi
    rm -rf "$mockdir" "$target"
}

Add test_full_restore_refuses_nonempty_target_without_force and test_full_restore_dry_run_makes_no_calls to the invocation list.

Step 4: Run the tests

Run: RESTORE_LOGDIR=/tmp/restore-test-logs bash tests/test_restore.sh Expected: all tests pass, including the two new ones.

Step 5: Commit

git add restore.sh tests/test_restore.sh
git commit -m "feat: add full disaster-recovery mode to restore.sh"

Task 6: `RUNBOOK.md`

Files:

Create: RUNBOOK.md

Interfaces:

Consumes: final restore.sh CLI surface from Tasks 1-5 (full, db <name>, file <path>, --list-archives, flags --archive/--force/--yes/--dry-run/--dest), and borg-backup.sh's config (NAME=borg-2025, REPO=/home/srv/files/backups/borg-2025, TARGET=/home/srv/files/content, BORG_PASSPHRASE_FILE=/root/.borg-passphrase).
Step 1: Write RUNBOOK.md

# Backup & Recovery Runbook

Covers `borg-backup.sh` (daily backup), `dump_db.sh` (MariaDB logical
dumps, invoked by the backup script), and `restore.sh` (recovery).

## 1. One-Time Setup

Run once, by hand, on the server:

1. Initialize the encrypted Borg repo:
   ```bash
   mkdir -p /home/srv/files/backups
   borg init --encryption=repokey-blake2 /home/srv/files/backups/borg-2025

Create the passphrase file used by both backup and restore:

echo 'your-strong-passphrase' > /root/.borg-passphrase
chmod 600 /root/.borg-passphrase

Create the MariaDB backup user used by dump_db.sh (read-only, no stop/lock of the server required thanks to --single-transaction):

CREATE USER 'backup'@'%' IDENTIFIED BY 'a-strong-password';
GRANT SELECT, LOCK TABLES, SHOW VIEW, TRIGGER, PROCESS, RELOAD ON *.* TO 'backup'@'%';

echo 'a-strong-password' > /root/.mariadb-backup.pw
chmod 600 /root/.mariadb-backup.pw

Create the root password file used only by restore.sh (restore needs CREATE/DROP privileges the backup user does not have):
```
echo 'the-mariadb-root-password' > /root/.mariadb-root.pw
chmod 600 /root/.mariadb-root.pw
```
Exclude MariaDB's raw data directory from the archive. Find the directory bind-mounted into the container as its datadir and drop a marker file in it:
```
touch /home/srv/files/content/mariadb/data/.nobackup
```
This is what allows backups to run with the container up: only the logical dump under mariadb/dump/ is ever archived or restored from.

Configure the scaleway rclone remote:

rclone config
# create a remote named "scaleway", type S3, matching your Scaleway
# Object Storage credentials and region

Set a real healthcheck URL in borg-backup.sh (HEALTHCHECK_URL=), for example from https://healthchecks.io.

2. Deploying the Scripts

Copy borg-backup.sh, dump_db.sh, and restore.sh to the server (e.g. /opt/backup-agent/), and dump_db.sh additionally to /home/srv/files/content/mariadb/dump_db.sh (this exact path is what borg-backup.sh invokes). Make all three executable:

chmod +x /opt/backup-agent/borg-backup.sh /opt/backup-agent/restore.sh
chmod +x /home/srv/files/content/mariadb/dump_db.sh

3. Scheduling

Add a cron entry to run the backup daily, off-peak:

# /etc/cron.d/borg-backup
30 2 * * * root /opt/backup-agent/borg-backup.sh >> /var/log/borg/cron.log 2>&1

Check the last run:

ls -lt /var/log/borg/backup-*.log | head -1   # latest log file
tail -50 /var/log/borg/backup-*.log           # inspect it

Or watch the healthcheck dashboard configured in step 1.7 — a missed or failed run pages/alerts there.

4. Day-2 Operations

List archives:

./restore.sh --list-archives

Check repo size and health:

BORG_PASSCOMMAND="cat /root/.borg-passphrase" borg info /home/srv/files/backups/borg-2025

Rotate the passphrase (creates a new key, re-encrypts nothing — old archives still need the old passphrase to read, so keep both until fully migrated):

BORG_PASSCOMMAND="cat /root/.borg-passphrase" borg key change-passphrase /home/srv/files/backups/borg-2025

Stale lockfile (backup or restore aborted mid-run and left the repo locked):

BORG_PASSCOMMAND="cat /root/.borg-passphrase" borg break-lock /home/srv/files/backups/borg-2025

5. Recovery Procedures

All restore.sh commands accept --dry-run to preview exactly what would happen without touching anything, and --archive NAME to target a specific archive instead of the latest (see archive names via --list-archives).

5.1 Full disaster recovery (new or wiped server)

Use when the whole server/container is gone and you're rebuilding from scratch.

# 1. Reinstall borg, docker, and the mariadb container image/compose file
#    (not covered by restore.sh - this is infra provisioning).
# 2. Restore the passphrase file (from your password manager / secondary
#    backup - it is NOT stored in the repo it protects) to
#    /root/.borg-passphrase, and the root DB password to
#    /root/.mariadb-root.pw.
# 3. Preview:
./restore.sh full --dry-run
# 4. Run for real (refuses if /home/srv/files/content is non-empty):
./restore.sh full --force

This extracts the full content tree from the archive, restores every database dump (users/grants first), and starts the mariadb container, waiting for it to report healthy.

Verify afterward:

docker ps shows mariadb running and healthy.
The application responds normally.
Spot-check row counts on a couple of tables against what you'd expect.

5.2 Single database restore

Use when one database got corrupted or someone ran a bad migration/query against it — this drops and recreates that database.

./restore.sh db shopdb --dry-run     # preview
./restore.sh db shopdb                # prompts: type "shopdb" to confirm

Non-interactive (e.g. scripted from a monitoring alert): add --yes to skip the typed confirmation.

Verify afterward: connect to the database and check the tables/row counts you expect.

5.3 Single file/directory restore

Use for accidental deletion of a file, or to inspect an old version — this never touches the running database or container.

./restore.sh file path/relative/to/content/some-file.txt --dest /tmp/recovered

The final location of the recovered item is printed at the end (it lands under /tmp/recovered/home/srv/files/content/... — Borg preserves the absolute path it was archived with).

6. Restore Drill Cadence

Quarterly, run a real full restore into a scratch directory (not /home/srv/files/content) to confirm backups are actually usable:

mkdir -p /tmp/restore-drill
BORG_PASSCOMMAND="cat /root/.borg-passphrase" \
  borg extract --lock-wait 600 /home/srv/files/backups/borg-2025::$(./restore.sh --list-archives | tail -1) \
  --destination /tmp/restore-drill  # (or adapt restore.sh's TARGET for a one-off dry run into scratch)

Confirm the dump files under mariadb/dump/ are present, non-empty, and importable (mysql -u root -p < mariadb/dump/somedb.sql against a throwaway MariaDB container). Log the drill date and outcome somewhere durable (e.g. the healthcheck dashboard's notes, or a team wiki page).


- [ ] **Step 2: Cross-check the runbook against the script's actual CLI**

Run: `grep -oE 'restore\.sh [a-z-]+' RUNBOOK.md | sort -u`
Expected output includes `restore.sh --list-archives`, `restore.sh db`,
`restore.sh file`, `restore.sh full` — confirming every implemented
subcommand is documented. Run `./restore.sh -h` and confirm no flag or
subcommand appears there that's missing from `RUNBOOK.md`.

- [ ] **Step 3: Commit**

```bash
git add RUNBOOK.md
git commit -m "docs: add operational runbook for backup and recovery"

Self-Review Notes

Spec coverage: one-time setup (Task 6 §1), deployment (§2), cron scheduling (§3), day-2 ops (§4), all three recovery scenarios (§5.1-5.3), quarterly drill (§6) — all present. restore.sh covers full/db/file with --force/typed-confirmation/--yes safety guards and --dry-run for every mode, per the approved design.
No changes were made to dump_db.sh; borg-backup.sh's only change (removing the DB stop/start) was already implemented and committed before this plan.
Type/name consistency: RESOLVED_ARCHIVE, CLIENT_BIN, DB_PASS, ARCHIVE_TARGET_PATH, DUMP_SUBDIR are defined once (Tasks 1-2) and reused with the same names through Task 5; restore_single_db() is defined in Task 4 and reused unchanged by cmd_full() in Task 5.

33 KiB Raw Blame History

Restore Tooling & Runbook Implementation Plan

Global Constraints

File Structure

Task 1: restore.sh scaffold — config, logging, arg parsing, usage

Task 2: Archive listing and resolution

Task 3: file mode (non-destructive single path restore)

Task 4: db mode (single-database restore with confirmation)

Task 5: full mode (disaster recovery) and shared container helpers

Task 6: RUNBOOK.md

2. Deploying the Scripts

3. Scheduling

4. Day-2 Operations

5. Recovery Procedures

5.1 Full disaster recovery (new or wiped server)

5.2 Single database restore

5.3 Single file/directory restore

6. Restore Drill Cadence

Self-Review Notes

33 KiB

Raw Blame History

Task 1: `restore.sh` scaffold — config, logging, arg parsing, usage

Task 3: `file` mode (non-destructive single path restore)

Task 4: `db` mode (single-database restore with confirmation)

Task 5: `full` mode (disaster recovery) and shared container helpers

Task 6: `RUNBOOK.md`