docs: add operational runbook for backup and recovery
This commit is contained in:
+191
@@ -0,0 +1,191 @@
|
||||
# Backup & Recovery Runbook
|
||||
|
||||
Covers `borg-backup.sh` (daily backup), `dump_db.sh` (MariaDB logical
|
||||
dumps, invoked by the backup script), and `restore.sh` (recovery).
|
||||
|
||||
## 1. One-Time Setup
|
||||
|
||||
Run once, by hand, on the server:
|
||||
|
||||
1. Initialize the encrypted Borg repo:
|
||||
```bash
|
||||
mkdir -p /home/srv/files/backups
|
||||
borg init --encryption=repokey-blake2 /home/srv/files/backups/borg-2025
|
||||
```
|
||||
2. Create the passphrase file used by both backup and restore:
|
||||
```bash
|
||||
echo 'your-strong-passphrase' > /root/.borg-passphrase
|
||||
chmod 600 /root/.borg-passphrase
|
||||
```
|
||||
3. Create the MariaDB `backup` user used by `dump_db.sh` (read-only, no
|
||||
stop/lock of the server required thanks to `--single-transaction`):
|
||||
```sql
|
||||
CREATE USER 'backup'@'%' IDENTIFIED BY 'a-strong-password';
|
||||
GRANT SELECT, LOCK TABLES, SHOW VIEW, TRIGGER, PROCESS, RELOAD ON *.* TO 'backup'@'%';
|
||||
```
|
||||
```bash
|
||||
echo 'a-strong-password' > /root/.mariadb-backup.pw
|
||||
chmod 600 /root/.mariadb-backup.pw
|
||||
```
|
||||
4. Create the root password file used only by `restore.sh` (restore needs
|
||||
CREATE/DROP privileges the `backup` user does not have):
|
||||
```bash
|
||||
echo 'the-mariadb-root-password' > /root/.mariadb-root.pw
|
||||
chmod 600 /root/.mariadb-root.pw
|
||||
```
|
||||
5. Exclude MariaDB's raw data directory from the archive. Find the
|
||||
directory bind-mounted into the container as its datadir and drop a
|
||||
marker file in it:
|
||||
```bash
|
||||
touch /home/srv/files/content/mariadb/data/.nobackup
|
||||
```
|
||||
This is what allows backups to run with the container up: only the
|
||||
logical dump under `mariadb/dump/` is ever archived or restored from.
|
||||
6. Configure the `scaleway` rclone remote:
|
||||
```bash
|
||||
rclone config
|
||||
# create a remote named "scaleway", type S3, matching your Scaleway
|
||||
# Object Storage credentials and region
|
||||
```
|
||||
7. Set a real healthcheck URL in `borg-backup.sh` (`HEALTHCHECK_URL=`), for
|
||||
example from https://healthchecks.io.
|
||||
|
||||
## 2. Deploying the Scripts
|
||||
|
||||
Copy `borg-backup.sh`, `dump_db.sh`, and `restore.sh` to the server (e.g.
|
||||
`/opt/backup-agent/`), and `dump_db.sh` additionally to
|
||||
`/home/srv/files/content/mariadb/dump_db.sh` (this exact path is what
|
||||
`borg-backup.sh` invokes). Make all three executable:
|
||||
|
||||
```bash
|
||||
chmod +x /opt/backup-agent/borg-backup.sh /opt/backup-agent/restore.sh
|
||||
chmod +x /home/srv/files/content/mariadb/dump_db.sh
|
||||
```
|
||||
|
||||
## 3. Scheduling
|
||||
|
||||
Add a cron entry to run the backup daily, off-peak:
|
||||
|
||||
```
|
||||
# /etc/cron.d/borg-backup
|
||||
30 2 * * * root /opt/backup-agent/borg-backup.sh >> /var/log/borg/cron.log 2>&1
|
||||
```
|
||||
|
||||
Check the last run:
|
||||
|
||||
```bash
|
||||
ls -lt /var/log/borg/backup-*.log | head -1 # latest log file
|
||||
tail -50 /var/log/borg/backup-*.log # inspect it
|
||||
```
|
||||
|
||||
Or watch the healthcheck dashboard configured in step 1.7 — a missed or
|
||||
failed run pages/alerts there.
|
||||
|
||||
## 4. Day-2 Operations
|
||||
|
||||
List archives:
|
||||
|
||||
```bash
|
||||
./restore.sh --list-archives
|
||||
```
|
||||
|
||||
Check repo size and health:
|
||||
|
||||
```bash
|
||||
BORG_PASSCOMMAND="cat /root/.borg-passphrase" borg info /home/srv/files/backups/borg-2025
|
||||
```
|
||||
|
||||
Rotate the passphrase (creates a new key, re-encrypts nothing — old
|
||||
archives still need the old passphrase to read, so keep both until fully
|
||||
migrated):
|
||||
|
||||
```bash
|
||||
BORG_PASSCOMMAND="cat /root/.borg-passphrase" borg key change-passphrase /home/srv/files/backups/borg-2025
|
||||
```
|
||||
|
||||
Stale lockfile (backup or restore aborted mid-run and left the repo
|
||||
locked):
|
||||
|
||||
```bash
|
||||
BORG_PASSCOMMAND="cat /root/.borg-passphrase" borg break-lock /home/srv/files/backups/borg-2025
|
||||
```
|
||||
|
||||
## 5. Recovery Procedures
|
||||
|
||||
All `restore.sh` commands accept `--dry-run` to preview exactly what would
|
||||
happen without touching anything, and `--archive NAME` to target a
|
||||
specific archive instead of the latest (see archive names via
|
||||
`--list-archives`).
|
||||
|
||||
### 5.1 Full disaster recovery (new or wiped server)
|
||||
|
||||
Use when the whole server/container is gone and you're rebuilding from
|
||||
scratch.
|
||||
|
||||
```bash
|
||||
# 1. Reinstall borg, docker, and the mariadb container image/compose file
|
||||
# (not covered by restore.sh - this is infra provisioning).
|
||||
# 2. Restore the passphrase file (from your password manager / secondary
|
||||
# backup - it is NOT stored in the repo it protects) to
|
||||
# /root/.borg-passphrase, and the root DB password to
|
||||
# /root/.mariadb-root.pw.
|
||||
# 3. Preview:
|
||||
./restore.sh full --dry-run
|
||||
# 4. Run for real (refuses if /home/srv/files/content is non-empty):
|
||||
./restore.sh full --force
|
||||
```
|
||||
|
||||
This extracts the full content tree from the archive, restores every
|
||||
database dump (users/grants first), and starts the `mariadb` container,
|
||||
waiting for it to report healthy.
|
||||
|
||||
**Verify afterward:**
|
||||
- `docker ps` shows `mariadb` running and healthy.
|
||||
- The application responds normally.
|
||||
- Spot-check row counts on a couple of tables against what you'd expect.
|
||||
|
||||
### 5.2 Single database restore
|
||||
|
||||
Use when one database got corrupted or someone ran a bad migration/query
|
||||
against it — this **drops and recreates** that database.
|
||||
|
||||
```bash
|
||||
./restore.sh db shopdb --dry-run # preview
|
||||
./restore.sh db shopdb # prompts: type "shopdb" to confirm
|
||||
```
|
||||
|
||||
Non-interactive (e.g. scripted from a monitoring alert): add `--yes` to
|
||||
skip the typed confirmation.
|
||||
|
||||
**Verify afterward:** connect to the database and check the tables/row
|
||||
counts you expect.
|
||||
|
||||
### 5.3 Single file/directory restore
|
||||
|
||||
Use for accidental deletion of a file, or to inspect an old version — this
|
||||
never touches the running database or container.
|
||||
|
||||
```bash
|
||||
./restore.sh file path/relative/to/content/some-file.txt --dest /tmp/recovered
|
||||
```
|
||||
|
||||
The final location of the recovered item is printed at the end (it lands
|
||||
under `/tmp/recovered/home/srv/files/content/...` — Borg preserves the
|
||||
absolute path it was archived with).
|
||||
|
||||
## 6. Restore Drill Cadence
|
||||
|
||||
Quarterly, run a real `full` restore into a scratch directory (not
|
||||
`/home/srv/files/content`) to confirm backups are actually usable:
|
||||
|
||||
```bash
|
||||
mkdir -p /tmp/restore-drill
|
||||
BORG_PASSCOMMAND="cat /root/.borg-passphrase" \
|
||||
borg extract --lock-wait 600 /home/srv/files/backups/borg-2025::$(./restore.sh --list-archives | tail -1) \
|
||||
--destination /tmp/restore-drill # (or adapt restore.sh's TARGET for a one-off dry run into scratch)
|
||||
```
|
||||
|
||||
Confirm the dump files under `mariadb/dump/` are present, non-empty, and
|
||||
importable (`mysql -u root -p < mariadb/dump/somedb.sql` against a
|
||||
throwaway MariaDB container). Log the drill date and outcome somewhere
|
||||
durable (e.g. the healthcheck dashboard's notes, or a team wiki page).
|
||||
Reference in New Issue
Block a user