diff --git a/README.md b/README.md index 0e16e53..2bb6e23 100644 --- a/README.md +++ b/README.md @@ -263,6 +263,168 @@ databases written by `database` targets (`archive-{uuid}.db`). Mount this as a persistent volume to preserve data across container restarts. +## Backup, Restore, and Upgrades + +### What to back up + +All persistent state lives in `DATA_DIR` (`/var/lib/webhooker` by +default). Back up that directory in full — the state is spread across +several files whose names depend on your data, so copying the directory +is both the simplest and the only complete rule: + +- `webhooker.db` — one per install. Settings (including the session + encryption key), users, API keys, webhooks, entrypoints, targets. +- `events-{webhook_uuid}.db` — **one per webhook**. Events, deliveries, + delivery results. +- `archive-{webhook_uuid}.db` — **one per webhook that has a `database` + target**. Archived events. Keyed on the webhook UUID, not the target + UUID: a webhook with several `database` targets still has exactly one + archive file. + +`{webhook_uuid}` is the webhook's UUID primary key in its canonical +36-character hyphenated form, so a real filename looks like +`events-3f2a1c9e-....db`. Nothing else is written to `DATA_DIR`, and no +`-wal` or `-shm` files are produced (see below); a transient +`{name}.db-journal` may exist beside a database while a write is in +flight and is not part of the backup set. + +Configuration is **not** in `DATA_DIR` — it comes from the environment +and from a `.env` file read out of the process working directory. Back +that up with your deployment config, separately. + +### A hot copy is not safe + +No `journal_mode` pragma is ever issued on any database webhooker opens, +so all of them run on SQLite's default rollback journal. There is no +WAL. The main and event databases are also held open for the entire +process lifetime — `WebhookDBManager` caches event database handles and +closes them only on webhook deletion or shutdown — so "it looked idle" +is not a guarantee that nothing was mid-transaction. + +That means `cp`, `rsync`, `tar` or a filesystem snapshot taken against a +running instance can capture a database mid-transaction and yield a file +that is corrupt or missing state the journal would have rolled back. Use +one of the two procedures below instead. + +**Stop, copy, start.** The simplest, needs no extra tooling, and the +only one that gives a single point in time across every file: + +```bash +docker stop webhooker +cp -a /path/to/data /path/to/backup-$(date -u +%Y%m%dT%H%M%SZ) +docker start webhooker +``` + +**SQLite online backup.** No downtime, one file at a time: + +```bash +for db in /path/to/data/*.db; do + sqlite3 "$db" ".backup '/path/to/backup/$(basename "$db")'" +done +``` + +`.backup` takes the proper locks and writes a consistent file. Two +caveats. First, the runtime image is `alpine:3.21` with only +`ca-certificates` added — the `sqlite3` CLI is **not** in it, so run +this on the host against the volume path, or from a throwaway container +that mounts the volume. Second, each file is captured at its own +instant, so a webhook created or an event delivered between two files +being copied lands in one and not the other. If you need the whole set +coherent as of a single moment, stop the service. + +Archive databases are the one exception the service is built for: the +archive writer closes its handle after each write (debounced to at most +one reopen per second), so an operator can move `archive-{uuid}.db` +away for offline retention while the service runs, and it is recreated +on the next write (see +[Database Architecture](#database-architecture)). That is a +move-the-file-away workflow, not a substitute for the backup procedures +above. + +### Restore + +1. Stop the service. + +2. Restore the **whole set together**: `webhooker.db` *and* every + `events-*.db` *and* every `archive-*.db`. A partial restore fails + quietly rather than loudly. Every database is opened `mode=rwc`, so a + missing `events-{uuid}.db` is **created empty** on first access + instead of erroring — the webhook comes back with its configuration + intact and its entire event history silently gone. Event databases + restored without `webhooker.db` are simply orphaned; nothing + references their UUIDs. + +3. Do not carry `*.db-journal` files into the restore. Backups taken by + either procedure above are self-consistent and do not need one. + +4. **Fix ownership.** The container runs as the non-root `webhooker` + user, UID 1000 / GID 1000. Restored files must be owned by (or + writable by) that UID, and so must the directory itself — SQLite + creates the rollback journal beside the database, so a writable file + inside a directory it cannot write is not enough: + + ```bash + chown -R 1000:1000 /path/to/data + ``` + + Restoring as `root` on the host and forgetting this step is the + usual way a restore fails. + +5. Start the service. `AutoMigrate` runs against each restored database + as it is opened. + +### Upgrades + +**Back up before every upgrade.** Every start runs GORM `AutoMigrate` +unconditionally against whatever files it finds: + +- the main database on connect — `Setting`, `User`, `APIKey`, `Webhook`, + `Entrypoint`, `Target` +- each event database when it is lazily opened — `Event`, `Delivery`, + `DeliveryResult` +- each archive database on every open and reopen + +There is no schema version table, no migration ledger, and no down +migrations. Nothing in the files records which version wrote them, and +no code path undoes a migration. + +Upgrade procedure: + +1. Stop the service. +2. Back up `DATA_DIR` using one of the procedures above. +3. Pull the new image and start it. +4. Confirm `database migrations completed` in the logs before putting + traffic back on it. + +**Downgrade is unsupported.** Once a newer binary has migrated the files +there is no way to move them back. `AutoMigrate` is additive — it adds +tables, columns and indexes and never drops or rewrites them — so an +older binary will generally open migrated files and appear to work while +writing against a schema it does not know about. The failure mode is +silent divergence, not a startup error. The only supported way back to +an older version is restoring the pre-upgrade backup, which discards +everything received since that backup was taken. + +### Backups contain secrets + +Treat a backup with the same care as the credentials inside it. Encrypt +backups at rest and restrict who can read them. + +- `events-{uuid}.db` and `archive-{uuid}.db` hold the **full payload + body and headers** of every event as received, including whatever the + sending service put in them — tokens, signatures, personal data. +- Until + [issue #206](https://git.eeqj.de/sneak/webhooker/issues/206) is fixed, + the event databases **also contain target credentials**: a GORM + association upsert on the delivery and retry write path copies + `targets` rows, `config` included, into the per-webhook database. For + a Slack target the `webhookUrl` *is* the bearer credential, and an + `http` target's URL can embed userinfo. Handing someone an + `events-*.db` today hands them live delivery destinations. +- `webhooker.db` stores target config **unencrypted**, tracked at + [issue #212](https://git.eeqj.de/sneak/webhooker/issues/212), next to + the session encryption key and the Argon2id password hashes. + ## Entrypoints This repository adheres to the