# Restore Tooling & Runbook Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Build `restore.sh` (full / single-database / single-file recovery) and `RUNBOOK.md` so the backup system built around `borg-backup.sh` and `dump_db.sh` has a documented, scriptable, fast recovery path.

**Architecture:** `restore.sh` is a standalone bash script mirroring the config/logging conventions of `borg-backup.sh`. It offers three subcommands (`full`, `db <name>`, `file <path>`) plus `--list-archives`, all reading from the same encrypted Borg repo. Destructive modes require explicit confirmation (`--force` for `full`, typed database name or `--yes` for `db`). `RUNBOOK.md` documents one-time setup, deployment, scheduling, day-2 ops, and step-by-step recovery for all three scenarios.

**Tech Stack:** bash, borg, docker, mysql/mariadb client, rclone (setup only, not used by restore.sh).

## Global Constraints

- Target directory: `/home/srv/files/content` (`TARGET`)
- Repo: `/home/srv/files/backups/borg-2025` (`NAME=borg-2025`)
- DB container name: `mariadb`
- Borg passphrase file: `/root/.borg-passphrase`, chmod 600, read via `BORG_PASSCOMMAND`
- Backup DB user `backup`, password file `/root/.mariadb-backup.pw` (used by `dump_db.sh`, unrelated to restore)
- Dump directory inside target: `mariadb/dump`, one `.sql` file per database, plus `00-users-and-grants.sql`
- MariaDB is never stopped for backups; its raw data directory is excluded from the archive via a `.nobackup` marker file — restores are always driven from the logical dump, never from raw archived DB files
- `restore.sh` must never silently overwrite live data: `full` refuses to run against a non-empty target without `--force`; `db` requires the database name to be typed back for confirmation unless `--yes` is passed
- Restore DB operations authenticate as `root` (env `MYSQL_ROOT_PASSWORD` or a password file, default `/root/.mariadb-root.pw`) since restore needs privileges the `backup` user does not have
- Logs go to `/var/log/borg/restore-*.log` (override via `RESTORE_LOGDIR` for testing), same naming convention as `borg-backup.sh`

---

## File Structure

- Create: `restore.sh` — the recovery script (repo root, alongside `borg-backup.sh`)
- Create: `RUNBOOK.md` — operational documentation (repo root)
- Create: `tests/test_restore.sh` — bash test script exercising `restore.sh` against mocked `borg`/`docker`/`mysql`/`mariadb` binaries; grows across tasks
- Create: `tests/lib/setup_mocks.sh` — shared helper that builds the mock binaries into a temp `$PATH` directory

## Task 1: `restore.sh` scaffold — config, logging, arg parsing, usage

**Files:**
- Create: `restore.sh`
- Create: `tests/test_restore.sh`

**Interfaces:**
- Produces: `log(msg)`, `step(msg)`, `die(msg)` (exits 1), `run_cmd(cmd...)`, `usage()`, `parse_common_flags(args...)` (sets globals `ARCHIVE_OVERRIDE`, `FORCE`, `YES`, `DRY_RUN`, `DEST`), `main(args...)`. Config globals: `NAME`, `REPO`, `TARGET`, `DB_CONTAINER`, `DB_START_TIMEOUT`, `BORG_PASSPHRASE_FILE`, `ROOT_PASSWORD_FILE`, `DUMP_SUBDIR`, `ARCHIVE_TARGET_PATH`, `LOGDIR`, `LOGFILE`.

- [ ] **Step 1: Write `restore.sh` scaffold**

```bash
#!/bin/bash
# =============================================================================
# Restore tooling for the borg-backup.sh / dump_db.sh backup system.
# Modes:
#   restore.sh full  [--archive NAME] [--force] [--dry-run]
#   restore.sh db    <db_name> [--archive NAME] [--yes] [--dry-run]
#   restore.sh file  <path-within-target> [--archive NAME] [--dest DIR] [--dry-run]
#   restore.sh --list-archives
# =============================================================================

set -euo pipefail

export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$PATH"
umask 077

# ========================= CONFIGURATION =========================

NAME="borg-2025"
REPO="/home/srv/files/backups/$NAME"
TARGET="/home/srv/files/content"
ARCHIVE_TARGET_PATH="${TARGET#/}"

DB_CONTAINER="mariadb"
DB_START_TIMEOUT=180

BORG_PASSPHRASE_FILE="${BORG_PASSPHRASE_FILE:-/root/.borg-passphrase}"
ROOT_PASSWORD_FILE="${ROOT_PASSWORD_FILE:-/root/.mariadb-root.pw}"
DUMP_SUBDIR="mariadb/dump"

LOGDIR="${RESTORE_LOGDIR:-/var/log/borg}"
mkdir -p "$LOGDIR" 2>/dev/null || LOGDIR="/tmp"
LOGFILE="$LOGDIR/restore-$(date +%Y-%m-%d-%H%M%S).log"

export BORG_REPO="$REPO"
export BORG_PASSCOMMAND="cat $BORG_PASSPHRASE_FILE"

# =================================================================

log() {
    local line
    line="[$(date '+%F %T')] $*"
    echo "$line"
    echo "$line" >> "$LOGFILE" 2>/dev/null || true
}

step() { echo; echo "=== $* ==="; }

die() { log "ERROR: $*"; exit 1; }

run_cmd() {
    log "[RUN] $*"
    "$@"
}

usage() {
    cat <<'EOF'
Usage:
  restore.sh full  [--archive NAME] [--force] [--dry-run]
  restore.sh db    <db_name> [--archive NAME] [--yes] [--dry-run]
  restore.sh file  <path-within-target> [--archive NAME] [--dest DIR] [--dry-run]
  restore.sh --list-archives
  restore.sh -h | --help
EOF
}

DRY_RUN=false
FORCE=false
YES=false
ARCHIVE_OVERRIDE=""
DEST=""

parse_common_flags() {
    while [[ $# -gt 0 ]]; do
        case "$1" in
            --archive) ARCHIVE_OVERRIDE="$2"; shift 2 ;;
            --force) FORCE=true; shift ;;
            --yes) YES=true; shift ;;
            --dry-run) DRY_RUN=true; shift ;;
            --dest) DEST="$2"; shift 2 ;;
            *) die "unknown flag: $1" ;;
        esac
    done
}

main() {
    local cmd="${1:-}"
    case "$cmd" in
        "")
            usage
            exit 1
            ;;
        -h|--help)
            usage
            exit 0
            ;;
        --list-archives)
            shift
            parse_common_flags "$@"
            list_archives
            ;;
        full)
            shift
            parse_common_flags "$@"
            cmd_full
            ;;
        db)
            shift
            local dbname="${1:-}"
            [[ -n "$dbname" ]] || die "db: missing <db_name>"
            shift
            parse_common_flags "$@"
            cmd_db "$dbname"
            ;;
        file)
            shift
            local relpath="${1:-}"
            [[ -n "$relpath" ]] || die "file: missing <path>"
            shift
            parse_common_flags "$@"
            cmd_file "$relpath"
            ;;
        *)
            usage
            die "unknown command: $cmd"
            ;;
    esac
}

main "$@"
```

- [ ] **Step 2: Make it executable and syntax-check**

Run: `chmod +x restore.sh && bash -n restore.sh`
Expected: no output, exit 0.

- [ ] **Step 3: Write `tests/test_restore.sh`**

```bash
#!/bin/bash
set -uo pipefail

HERE="$(cd "$(dirname "$0")" && pwd)"
RESTORE="$HERE/../restore.sh"
FAILURES=0

assert_eq() {
    local expected="$1" actual="$2" msg="$3"
    if [[ "$expected" != "$actual" ]]; then
        echo "FAIL: $msg (expected '$expected', got '$actual')"
        FAILURES=$((FAILURES + 1))
    else
        echo "PASS: $msg"
    fi
}

assert_contains() {
    local haystack="$1" needle="$2" msg="$3"
    if [[ "$haystack" != *"$needle"* ]]; then
        echo "FAIL: $msg (expected to contain '$needle', got '$haystack')"
        FAILURES=$((FAILURES + 1))
    else
        echo "PASS: $msg"
    fi
}

test_help_exits_zero() {
    local out rc
    out="$(bash "$RESTORE" -h)"
    rc=$?
    assert_eq "0" "$rc" "help exits 0"
    assert_contains "$out" "Usage:" "help prints usage"
}

test_no_args_exits_one() {
    local rc
    bash "$RESTORE" >/dev/null 2>&1
    rc=$?
    assert_eq "1" "$rc" "no args exits 1"
}

test_unknown_command_exits_one() {
    local rc
    bash "$RESTORE" bogus >/dev/null 2>&1
    rc=$?
    assert_eq "1" "$rc" "unknown command exits 1"
}

test_help_exits_zero
test_no_args_exits_one
test_unknown_command_exits_one

echo "-----"
if [[ "$FAILURES" -gt 0 ]]; then
    echo "$FAILURES failure(s)"
    exit 1
fi
echo "All tests passed"
```

- [ ] **Step 4: Run the tests**

Run: `chmod +x tests/test_restore.sh && RESTORE_LOGDIR=/tmp/restore-test-logs bash tests/test_restore.sh`
Expected: three `PASS:` lines, then `All tests passed`, exit 0.

- [ ] **Step 5: Commit**

```bash
git add restore.sh tests/test_restore.sh
git commit -m "feat: add restore.sh scaffold with arg parsing and usage"
```

---

## Task 2: Archive listing and resolution

**Files:**
- Modify: `restore.sh` (add functions before `main()`)
- Modify: `tests/test_restore.sh`
- Create: `tests/lib/setup_mocks.sh`

**Interfaces:**
- Consumes: `die()`, `log()` from Task 1.
- Produces: `resolve_archive()` (sets global `RESOLVED_ARCHIVE`, honors `ARCHIVE_OVERRIDE`), `list_archives()`.

- [ ] **Step 1: Write `tests/lib/setup_mocks.sh`**

```bash
#!/bin/bash
# Builds fake borg/docker/mysql/mariadb executables into $1 so restore.sh
# can be exercised without real infrastructure. Each mock logs its
# invocation to $1/mock.log.

setup_mock_bin() {
    local dir="$1"
    mkdir -p "$dir"
    local mocklog="$dir/mock.log"
    : > "$mocklog"

    cat > "$dir/borg" <<EOF
#!/bin/bash
echo "borg \$*" >> "$mocklog"
case "\$1" in
    list)
        echo "host-2026-01-01T00-00-00"
        echo "host-2026-06-01T00-00-00"
        ;;
    extract)
        shift
        paths=()
        for a in "\$@"; do
            case "\$a" in
                -*) ;;
                *::*) ;;
                *) paths+=("\$a") ;;
            esac
        done
        for p in "\${paths[@]}"; do
            if [[ "\$p" == *.* ]]; then
                mkdir -p "\$(dirname "\$p")"
                printf 'mock-content\n' > "\$p"
            else
                mkdir -p "\$p"
                touch "\$p/RESTORED_MARKER"
            fi
        done
        ;;
    *) ;;
esac
exit 0
EOF

    cat > "$dir/docker" <<EOF
#!/bin/bash
echo "docker \$*" >> "$mocklog"
case "\$1" in
    inspect)
        if [[ "\$*" == *"State.Running"* ]]; then
            echo true
        else
            echo healthy
        fi
        ;;
    start|stop)
        exit 0
        ;;
    exec)
        case "\$*" in
            *"command -v mariadb"*) exit 0 ;;
            *) cat >/dev/null 2>&1 || true; exit 0 ;;
        esac
        ;;
    *) ;;
esac
exit 0
EOF

    cat > "$dir/mysql" <<EOF
#!/bin/bash
echo "mysql \$*" >> "$mocklog"
cat >/dev/null 2>&1 || true
exit 0
EOF
    cp "$dir/mysql" "$dir/mariadb"

    chmod +x "$dir/borg" "$dir/docker" "$dir/mysql" "$dir/mariadb"
}
```

- [ ] **Step 2: Add `resolve_archive()` and `list_archives()` to `restore.sh`**

Insert directly above the `usage()` function:

```bash
resolve_archive() {
    if [[ -n "$ARCHIVE_OVERRIDE" ]]; then
        RESOLVED_ARCHIVE="$ARCHIVE_OVERRIDE"
        return 0
    fi
    RESOLVED_ARCHIVE="$(borg list --short --lock-wait 60 2>/dev/null | tail -n1)"
    [[ -n "$RESOLVED_ARCHIVE" ]] || die "no archives found in repo $REPO"
}

list_archives() {
    borg list --lock-wait 60
}
```

- [ ] **Step 3: Syntax-check**

Run: `bash -n restore.sh`
Expected: no output, exit 0.

- [ ] **Step 4: Add a test for `--list-archives`**

Append to `tests/test_restore.sh`, just above the `test_help_exits_zero` calls at the bottom:

```bash
source "$HERE/lib/setup_mocks.sh"

test_list_archives() {
    local mockdir out
    mockdir="$(mktemp -d)"
    setup_mock_bin "$mockdir"
    out="$(PATH="$mockdir:$PATH" bash "$RESTORE" --list-archives)"
    assert_contains "$out" "host-2026-01-01T00-00-00" "list-archives shows first archive"
    assert_contains "$out" "host-2026-06-01T00-00-00" "list-archives shows latest archive"
    rm -rf "$mockdir"
}
```

And add the call `test_list_archives` next to the other `test_*` invocations (before the `echo "-----"` line).

- [ ] **Step 5: Run the tests**

Run: `RESTORE_LOGDIR=/tmp/restore-test-logs bash tests/test_restore.sh`
Expected: all `PASS:` lines including the two new ones, `All tests passed`, exit 0.

- [ ] **Step 6: Commit**

```bash
git add restore.sh tests/test_restore.sh tests/lib/setup_mocks.sh
git commit -m "feat: add archive listing and resolution to restore.sh"
```

---

## Task 3: `file` mode (non-destructive single path restore)

**Files:**
- Modify: `restore.sh`
- Modify: `tests/test_restore.sh`

**Interfaces:**
- Consumes: `resolve_archive()`, `RESOLVED_ARCHIVE`, `ARCHIVE_TARGET_PATH`, `REPO`, `run_cmd()`, `log()`, `step()`, global `DRY_RUN`/`DEST` from Tasks 1-2.
- Produces: `extract_path(rel_path, dest_dir, archive)` (echoes the final absolute path of the extracted item), `cmd_file(rel_path)`.

- [ ] **Step 1: Add `extract_path()` and `cmd_file()` to `restore.sh`**

Insert directly above `usage()`, after `list_archives()`:

```bash
extract_path() {
    local rel_path="$1" dest_dir="$2" archive="$3"
    mkdir -p "$dest_dir"
    ( cd "$dest_dir" && run_cmd borg extract --lock-wait 600 "${REPO}::${archive}" "${ARCHIVE_TARGET_PATH}/${rel_path}" )
    echo "${dest_dir%/}/${ARCHIVE_TARGET_PATH}/${rel_path}"
}

cmd_file() {
    local rel_path="$1" dest
    resolve_archive
    dest="${DEST:-/tmp/restore-file-$$}"
    step "Restoring '$rel_path' from archive $RESOLVED_ARCHIVE into $dest"
    if [[ "$DRY_RUN" == true ]]; then
        log "[DRY-RUN] would extract ${ARCHIVE_TARGET_PATH}/${rel_path} from ${REPO}::${RESOLVED_ARCHIVE} into $dest"
        return 0
    fi
    local final
    final="$(extract_path "$rel_path" "$dest" "$RESOLVED_ARCHIVE")"
    log "Restored file available at: $final"
}
```

- [ ] **Step 2: Syntax-check**

Run: `bash -n restore.sh`
Expected: no output, exit 0.

- [ ] **Step 3: Add tests for `file` mode**

Append to `tests/test_restore.sh` (near `test_list_archives`):

```bash
test_file_restore_extracts_to_dest() {
    local mockdir dest out final
    mockdir="$(mktemp -d)"
    dest="$(mktemp -d)"
    setup_mock_bin "$mockdir"
    out="$(PATH="$mockdir:$PATH" bash "$RESTORE" file photos/img.jpg --dest "$dest")"
    final="$dest/home/srv/files/content/photos/img.jpg"
    assert_contains "$out" "Restored file available at: $final" "file mode reports final path"
    if [[ -f "$final" ]]; then
        echo "PASS: extracted file exists on disk"
    else
        echo "FAIL: extracted file missing at $final"
        FAILURES=$((FAILURES + 1))
    fi
    rm -rf "$mockdir" "$dest"
}

test_file_restore_dry_run_makes_no_borg_call() {
    local mockdir dest
    mockdir="$(mktemp -d)"
    dest="$(mktemp -d)"
    setup_mock_bin "$mockdir"
    PATH="$mockdir:$PATH" bash "$RESTORE" file photos/img.jpg --dest "$dest" --dry-run >/dev/null
    if [[ -s "$mockdir/mock.log" ]] && grep -q "^borg extract" "$mockdir/mock.log"; then
        echo "FAIL: dry-run invoked borg extract"
        FAILURES=$((FAILURES + 1))
    else
        echo "PASS: dry-run made no borg extract call"
    fi
    rm -rf "$mockdir" "$dest"
}
```

Add `test_file_restore_extracts_to_dest` and `test_file_restore_dry_run_makes_no_borg_call` to the list of test invocations.

- [ ] **Step 4: Run the tests**

Run: `RESTORE_LOGDIR=/tmp/restore-test-logs bash tests/test_restore.sh`
Expected: all tests pass, including the two new ones.

- [ ] **Step 5: Commit**

```bash
git add restore.sh tests/test_restore.sh
git commit -m "feat: add non-destructive file-restore mode to restore.sh"
```

---

## Task 4: `db` mode (single-database restore with confirmation)

**Files:**
- Modify: `restore.sh`
- Modify: `tests/test_restore.sh`

**Interfaces:**
- Consumes: `resolve_archive()`, `RESOLVED_ARCHIVE`, `extract_path()`, `DUMP_SUBDIR`, `DB_CONTAINER`, `ROOT_PASSWORD_FILE`, global `YES`/`DRY_RUN` from Tasks 1-3.
- Produces: `detect_client()` (sets `CLIENT_BIN`), `get_root_creds()` (sets `DB_PASS`), `confirm_or_abort(dbname)`, `restore_single_db(sqlfile, dbname)`, `cmd_db(dbname)`.

- [ ] **Step 1: Add the db-mode functions to `restore.sh`**

Insert directly above `usage()`, after `cmd_file()`:

```bash
detect_client() {
    if docker exec "$DB_CONTAINER" sh -c 'command -v mariadb' >/dev/null 2>&1; then
        CLIENT_BIN="mariadb"
    else
        CLIENT_BIN="mysql"
    fi
}

get_root_creds() {
    if [[ -n "${MYSQL_ROOT_PASSWORD:-}" ]]; then
        DB_PASS="$MYSQL_ROOT_PASSWORD"
    elif [[ -r "$ROOT_PASSWORD_FILE" ]]; then
        DB_PASS="$(< "$ROOT_PASSWORD_FILE")"
    else
        die "no root DB password: set MYSQL_ROOT_PASSWORD or create $ROOT_PASSWORD_FILE (chmod 600)"
    fi
}

confirm_or_abort() {
    local dbname="$1" typed
    [[ "$YES" == true ]] && return 0
    echo "This will DROP/overwrite database '$dbname'. Type the database name to confirm:"
    read -r typed
    [[ "$typed" == "$dbname" ]] || die "confirmation did not match '$dbname' - aborting"
}

restore_single_db() {
    local sqlfile="$1" dbname="$2"
    [[ -s "$sqlfile" ]] || die "dump file missing or empty: $sqlfile"
    run_cmd docker exec -e MYSQL_PWD="$DB_PASS" "$DB_CONTAINER" \
        "$CLIENT_BIN" -u root -e "CREATE DATABASE IF NOT EXISTS \`$dbname\`;"
    log "[RUN] docker exec -i ... $CLIENT_BIN -u root $dbname < $sqlfile"
    docker exec -i -e MYSQL_PWD="$DB_PASS" "$DB_CONTAINER" \
        "$CLIENT_BIN" -u root "$dbname" < "$sqlfile"
}

cmd_db() {
    local dbname="$1" dumpfile dest
    resolve_archive
    step "Restoring database '$dbname' from archive $RESOLVED_ARCHIVE"
    if [[ "$DRY_RUN" == true ]]; then
        log "[DRY-RUN] would extract ${ARCHIVE_TARGET_PATH}/${DUMP_SUBDIR}/${dbname}.sql from ${REPO}::${RESOLVED_ARCHIVE}"
        log "[DRY-RUN] would DROP/recreate database '$dbname' and import the dump using root credentials"
        return 0
    fi
    confirm_or_abort "$dbname"
    dest="/tmp/restore-db-$$"
    dumpfile="$(extract_path "${DUMP_SUBDIR}/${dbname}.sql" "$dest" "$RESOLVED_ARCHIVE")"
    detect_client
    get_root_creds
    restore_single_db "$dumpfile" "$dbname"
    log "Database '$dbname' restored from $dumpfile"
    rm -rf "$dest"
}
```

- [ ] **Step 2: Syntax-check**

Run: `bash -n restore.sh`
Expected: no output, exit 0.

- [ ] **Step 3: Add tests for `db` mode**

Append to `tests/test_restore.sh`:

```bash
test_db_restore_with_yes_runs_full_sequence() {
    local mockdir out
    mockdir="$(mktemp -d)"
    setup_mock_bin "$mockdir"
    out="$(PATH="$mockdir:$PATH" ROOT_PASSWORD_FILE="$mockdir/rootpw" bash -c '
        echo "rootpass" > "'"$mockdir"'/rootpw"
        bash "'"$RESTORE"'" db shopdb --yes
    ')"
    assert_contains "$out" "Database 'shopdb' restored from" "db mode reports success"
    if grep -q "CREATE DATABASE IF NOT EXISTS" "$mockdir/mock.log"; then
        echo "PASS: db mode issued CREATE DATABASE"
    else
        echo "FAIL: db mode did not issue CREATE DATABASE"
        FAILURES=$((FAILURES + 1))
    fi
    rm -rf "$mockdir"
}

test_db_restore_dry_run_skips_confirmation_and_calls() {
    local mockdir out
    mockdir="$(mktemp -d)"
    setup_mock_bin "$mockdir"
    out="$(PATH="$mockdir:$PATH" bash "$RESTORE" db shopdb --dry-run </dev/null)"
    assert_contains "$out" "DRY-RUN" "db dry-run prints DRY-RUN plan"
    if [[ -s "$mockdir/mock.log" ]]; then
        echo "FAIL: dry-run invoked a mock binary"
        FAILURES=$((FAILURES + 1))
    else
        echo "PASS: dry-run made no external calls"
    fi
    rm -rf "$mockdir"
}

test_db_restore_aborts_on_wrong_confirmation() {
    local mockdir rc
    mockdir="$(mktemp -d)"
    setup_mock_bin "$mockdir"
    set +e
    echo "wrongname" | PATH="$mockdir:$PATH" bash "$RESTORE" db shopdb >/dev/null 2>&1
    rc=$?
    set -e
    assert_eq "1" "$rc" "db mode aborts on mismatched confirmation"
    rm -rf "$mockdir"
}
```

Add all three test names to the invocation list.

- [ ] **Step 4: Run the tests**

Run: `RESTORE_LOGDIR=/tmp/restore-test-logs bash tests/test_restore.sh`
Expected: all tests pass, including the three new ones.

- [ ] **Step 5: Commit**

```bash
git add restore.sh tests/test_restore.sh
git commit -m "feat: add single-database restore mode to restore.sh"
```

---

## Task 5: `full` mode (disaster recovery) and shared container helpers

**Files:**
- Modify: `restore.sh`
- Modify: `tests/test_restore.sh`

**Interfaces:**
- Consumes: `resolve_archive()`, `detect_client()`, `get_root_creds()`, `restore_single_db()`, `ARCHIVE_TARGET_PATH`, `TARGET`, `DUMP_SUBDIR`, `DB_CONTAINER`, `DB_START_TIMEOUT`, global `FORCE`/`DRY_RUN` from Tasks 1-4.
- Produces: `container_running(name)`, `container_health(name)`, `wait_for_container(name)`, `start_db()`, `cmd_full()`.

- [ ] **Step 1: Add container helpers and `cmd_full()` to `restore.sh`**

Insert directly above `usage()`, after `cmd_db()`:

```bash
container_running() {
    [[ "$(docker inspect -f '{{.State.Running}}' "$1" 2>/dev/null || echo false)" == "true" ]]
}

container_health() {
    docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' \
        "$1" 2>/dev/null || echo unknown
}

wait_for_container() {
    local name="$1" deadline=$((SECONDS + DB_START_TIMEOUT)) health
    while (( SECONDS < deadline )); do
        if container_running "$name"; then
            health="$(container_health "$name")"
            case "$health" in
                healthy|none) log "$name is up (health: $health)"; return 0 ;;
                unhealthy)    log "WARNING: $name reports unhealthy" ;;
            esac
        fi
        sleep 3
    done
    return 1
}

start_db() {
    docker start "$DB_CONTAINER" >/dev/null 2>&1 || true
    wait_for_container "$DB_CONTAINER"
}

cmd_full() {
    local staging dumpdir f dbname
    resolve_archive
    step "Full restore from archive $RESOLVED_ARCHIVE into $TARGET"

    if [[ -d "$TARGET" ]] && [[ -n "$(ls -A "$TARGET" 2>/dev/null)" ]] && [[ "$FORCE" != true ]]; then
        die "$TARGET is not empty - pass --force to overwrite (existing data will be replaced)"
    fi

    if [[ "$DRY_RUN" == true ]]; then
        log "[DRY-RUN] would extract full ${ARCHIVE_TARGET_PATH} tree from ${REPO}::${RESOLVED_ARCHIVE} into $TARGET"
        log "[DRY-RUN] would restore every *.sql dump under ${DUMP_SUBDIR}/ using root credentials"
        log "[DRY-RUN] would start $DB_CONTAINER and wait for it to become healthy"
        return 0
    fi

    staging="/tmp/restore-full-$$"
    mkdir -p "$staging"
    ( cd "$staging" && run_cmd borg extract --lock-wait 600 "${REPO}::${RESOLVED_ARCHIVE}" "${ARCHIVE_TARGET_PATH}" )

    mkdir -p "$(dirname "$TARGET")"
    rm -rf "${TARGET:?}"/* 2>/dev/null || true
    mkdir -p "$TARGET"
    run_cmd cp -a "${staging}/${ARCHIVE_TARGET_PATH}/." "$TARGET/"
    rm -rf "$staging"

    detect_client
    get_root_creds

    dumpdir="${TARGET}/${DUMP_SUBDIR}"
    [[ -d "$dumpdir" ]] || die "no dump directory found after extract: $dumpdir"

    if [[ -f "${dumpdir}/00-users-and-grants.sql" ]]; then
        step "Restoring users and grants"
        docker exec -i -e MYSQL_PWD="$DB_PASS" "$DB_CONTAINER" \
            "$CLIENT_BIN" -u root < "${dumpdir}/00-users-and-grants.sql"
    fi

    for f in "$dumpdir"/*.sql; do
        [[ -e "$f" ]] || continue
        dbname="$(basename "$f" .sql)"
        [[ "$dbname" == "00-users-and-grants" ]] && continue
        restore_single_db "$f" "$dbname"
    done

    step "Starting $DB_CONTAINER"
    start_db || die "CRITICAL: $DB_CONTAINER did not come up after restore"

    log "Full restore complete from archive $RESOLVED_ARCHIVE"
}
```

- [ ] **Step 2: Syntax-check**

Run: `bash -n restore.sh`
Expected: no output, exit 0.

- [ ] **Step 3: Add tests for `full` mode**

Append to `tests/test_restore.sh`:

```bash
test_full_restore_refuses_nonempty_target_without_force() {
    local mockdir target rc
    mockdir="$(mktemp -d)"
    target="$(mktemp -d)"
    touch "$target/existing-file"
    setup_mock_bin "$mockdir"
    set +e
    PATH="$mockdir:$PATH" TARGET_OVERRIDE=1 bash -c '
        sed "s#^TARGET=\"/home/srv/files/content\"#TARGET=\"'"$target"'\"#; s#^ARCHIVE_TARGET_PATH=.*#ARCHIVE_TARGET_PATH=\"\${TARGET#/}\"#" "'"$RESTORE"'" > "'"$mockdir"'/restore_patched.sh"
        bash "'"$mockdir"'/restore_patched.sh" full
    ' >/dev/null 2>&1
    rc=$?
    set -e
    assert_eq "1" "$rc" "full mode refuses non-empty target without --force"
    rm -rf "$mockdir" "$target"
}

test_full_restore_dry_run_makes_no_calls() {
    local mockdir target out
    mockdir="$(mktemp -d)"
    target="$(mktemp -d)"
    setup_mock_bin "$mockdir"
    out="$(PATH="$mockdir:$PATH" bash -c '
        sed "s#^TARGET=\"/home/srv/files/content\"#TARGET=\"'"$target"'\"#; s#^ARCHIVE_TARGET_PATH=.*#ARCHIVE_TARGET_PATH=\"\${TARGET#/}\"#" "'"$RESTORE"'" > "'"$mockdir"'/restore_patched.sh"
        bash "'"$mockdir"'/restore_patched.sh" full --dry-run
    ')"
    assert_contains "$out" "DRY-RUN" "full dry-run prints DRY-RUN plan"
    if [[ -s "$mockdir/mock.log" ]]; then
        echo "FAIL: full dry-run invoked a mock binary"
        FAILURES=$((FAILURES + 1))
    else
        echo "PASS: full dry-run made no external calls"
    fi
    rm -rf "$mockdir" "$target"
}
```

Add `test_full_restore_refuses_nonempty_target_without_force` and `test_full_restore_dry_run_makes_no_calls` to the invocation list.

- [ ] **Step 4: Run the tests**

Run: `RESTORE_LOGDIR=/tmp/restore-test-logs bash tests/test_restore.sh`
Expected: all tests pass, including the two new ones.

- [ ] **Step 5: Commit**

```bash
git add restore.sh tests/test_restore.sh
git commit -m "feat: add full disaster-recovery mode to restore.sh"
```

---

## Task 6: `RUNBOOK.md`

**Files:**
- Create: `RUNBOOK.md`

**Interfaces:**
- Consumes: final `restore.sh` CLI surface from Tasks 1-5 (`full`, `db <name>`, `file <path>`, `--list-archives`, flags `--archive`/`--force`/`--yes`/`--dry-run`/`--dest`), and `borg-backup.sh`'s config (`NAME=borg-2025`, `REPO=/home/srv/files/backups/borg-2025`, `TARGET=/home/srv/files/content`, `BORG_PASSPHRASE_FILE=/root/.borg-passphrase`).

- [ ] **Step 1: Write `RUNBOOK.md`**

```markdown
# Backup & Recovery Runbook

Covers `borg-backup.sh` (daily backup), `dump_db.sh` (MariaDB logical
dumps, invoked by the backup script), and `restore.sh` (recovery).

## 1. One-Time Setup

Run once, by hand, on the server:

1. Initialize the encrypted Borg repo:
   ```bash
   mkdir -p /home/srv/files/backups
   borg init --encryption=repokey-blake2 /home/srv/files/backups/borg-2025
   ```
2. Create the passphrase file used by both backup and restore:
   ```bash
   echo 'your-strong-passphrase' > /root/.borg-passphrase
   chmod 600 /root/.borg-passphrase
   ```
3. Create the MariaDB `backup` user used by `dump_db.sh` (read-only, no
   stop/lock of the server required thanks to `--single-transaction`):
   ```sql
   CREATE USER 'backup'@'%' IDENTIFIED BY 'a-strong-password';
   GRANT SELECT, LOCK TABLES, SHOW VIEW, TRIGGER, PROCESS, RELOAD ON *.* TO 'backup'@'%';
   ```
   ```bash
   echo 'a-strong-password' > /root/.mariadb-backup.pw
   chmod 600 /root/.mariadb-backup.pw
   ```
4. Create the root password file used only by `restore.sh` (restore needs
   CREATE/DROP privileges the `backup` user does not have):
   ```bash
   echo 'the-mariadb-root-password' > /root/.mariadb-root.pw
   chmod 600 /root/.mariadb-root.pw
   ```
5. Exclude MariaDB's raw data directory from the archive. Find the
   directory bind-mounted into the container as its datadir and drop a
   marker file in it:
   ```bash
   touch /home/srv/files/content/mariadb/data/.nobackup
   ```
   This is what allows backups to run with the container up: only the
   logical dump under `mariadb/dump/` is ever archived or restored from.
6. Configure the `scaleway` rclone remote:
   ```bash
   rclone config
   # create a remote named "scaleway", type S3, matching your Scaleway
   # Object Storage credentials and region
   ```
7. Set a real healthcheck URL in `borg-backup.sh` (`HEALTHCHECK_URL=`), for
   example from https://healthchecks.io.

## 2. Deploying the Scripts

Copy `borg-backup.sh`, `dump_db.sh`, and `restore.sh` to the server (e.g.
`/opt/backup-agent/`), and `dump_db.sh` additionally to
`/home/srv/files/content/mariadb/dump_db.sh` (this exact path is what
`borg-backup.sh` invokes). Make all three executable:

```bash
chmod +x /opt/backup-agent/borg-backup.sh /opt/backup-agent/restore.sh
chmod +x /home/srv/files/content/mariadb/dump_db.sh
```

## 3. Scheduling

Add a cron entry to run the backup daily, off-peak:

```
# /etc/cron.d/borg-backup
30 2 * * * root /opt/backup-agent/borg-backup.sh >> /var/log/borg/cron.log 2>&1
```

Check the last run:

```bash
ls -lt /var/log/borg/backup-*.log | head -1   # latest log file
tail -50 /var/log/borg/backup-*.log           # inspect it
```

Or watch the healthcheck dashboard configured in step 1.7 — a missed or
failed run pages/alerts there.

## 4. Day-2 Operations

List archives:

```bash
./restore.sh --list-archives
```

Check repo size and health:

```bash
BORG_PASSCOMMAND="cat /root/.borg-passphrase" borg info /home/srv/files/backups/borg-2025
```

Rotate the passphrase (creates a new key, re-encrypts nothing — old
archives still need the old passphrase to read, so keep both until fully
migrated):

```bash
BORG_PASSCOMMAND="cat /root/.borg-passphrase" borg key change-passphrase /home/srv/files/backups/borg-2025
```

Stale lockfile (backup or restore aborted mid-run and left the repo
locked):

```bash
BORG_PASSCOMMAND="cat /root/.borg-passphrase" borg break-lock /home/srv/files/backups/borg-2025
```

## 5. Recovery Procedures

All `restore.sh` commands accept `--dry-run` to preview exactly what would
happen without touching anything, and `--archive NAME` to target a
specific archive instead of the latest (see archive names via
`--list-archives`).

### 5.1 Full disaster recovery (new or wiped server)

Use when the whole server/container is gone and you're rebuilding from
scratch.

```bash
# 1. Reinstall borg, docker, and the mariadb container image/compose file
#    (not covered by restore.sh - this is infra provisioning).
# 2. Restore the passphrase file (from your password manager / secondary
#    backup - it is NOT stored in the repo it protects) to
#    /root/.borg-passphrase, and the root DB password to
#    /root/.mariadb-root.pw.
# 3. Preview:
./restore.sh full --dry-run
# 4. Run for real (refuses if /home/srv/files/content is non-empty):
./restore.sh full --force
```

This extracts the full content tree from the archive, restores every
database dump (users/grants first), and starts the `mariadb` container,
waiting for it to report healthy.

**Verify afterward:**
- `docker ps` shows `mariadb` running and healthy.
- The application responds normally.
- Spot-check row counts on a couple of tables against what you'd expect.

### 5.2 Single database restore

Use when one database got corrupted or someone ran a bad migration/query
against it — this **drops and recreates** that database.

```bash
./restore.sh db shopdb --dry-run     # preview
./restore.sh db shopdb                # prompts: type "shopdb" to confirm
```

Non-interactive (e.g. scripted from a monitoring alert): add `--yes` to
skip the typed confirmation.

**Verify afterward:** connect to the database and check the tables/row
counts you expect.

### 5.3 Single file/directory restore

Use for accidental deletion of a file, or to inspect an old version — this
never touches the running database or container.

```bash
./restore.sh file path/relative/to/content/some-file.txt --dest /tmp/recovered
```

The final location of the recovered item is printed at the end (it lands
under `/tmp/recovered/home/srv/files/content/...` — Borg preserves the
absolute path it was archived with).

## 6. Restore Drill Cadence

Quarterly, run a real `full` restore into a scratch directory (not
`/home/srv/files/content`) to confirm backups are actually usable:

```bash
mkdir -p /tmp/restore-drill
BORG_PASSCOMMAND="cat /root/.borg-passphrase" \
  borg extract --lock-wait 600 /home/srv/files/backups/borg-2025::$(./restore.sh --list-archives | tail -1) \
  --destination /tmp/restore-drill  # (or adapt restore.sh's TARGET for a one-off dry run into scratch)
```

Confirm the dump files under `mariadb/dump/` are present, non-empty, and
importable (`mysql -u root -p < mariadb/dump/somedb.sql` against a
throwaway MariaDB container). Log the drill date and outcome somewhere
durable (e.g. the healthcheck dashboard's notes, or a team wiki page).
```

- [ ] **Step 2: Cross-check the runbook against the script's actual CLI**

Run: `grep -oE 'restore\.sh [a-z-]+' RUNBOOK.md | sort -u`
Expected output includes `restore.sh --list-archives`, `restore.sh db`,
`restore.sh file`, `restore.sh full` — confirming every implemented
subcommand is documented. Run `./restore.sh -h` and confirm no flag or
subcommand appears there that's missing from `RUNBOOK.md`.

- [ ] **Step 3: Commit**

```bash
git add RUNBOOK.md
git commit -m "docs: add operational runbook for backup and recovery"
```

---

## Self-Review Notes

- **Spec coverage:** one-time setup (Task 6 §1), deployment (§2), cron
  scheduling (§3), day-2 ops (§4), all three recovery scenarios (§5.1-5.3),
  quarterly drill (§6) — all present. `restore.sh` covers `full`/`db`/`file`
  with `--force`/typed-confirmation/`--yes` safety guards and `--dry-run`
  for every mode, per the approved design.
- **No changes** were made to `dump_db.sh`; `borg-backup.sh`'s only change
  (removing the DB stop/start) was already implemented and committed
  before this plan.
- **Type/name consistency:** `RESOLVED_ARCHIVE`, `CLIENT_BIN`, `DB_PASS`,
  `ARCHIVE_TARGET_PATH`, `DUMP_SUBDIR` are defined once (Tasks 1-2) and
  reused with the same names through Task 5; `restore_single_db()` is
  defined in Task 4 and reused unchanged by `cmd_full()` in Task 5.