Document backup, restore and upgrade procedures (closes #210) (#214)

README.md

@@ -263,6 +263,168 @@ databases written by `database` targets (`archive-{uuid}.db`). Mount
 this as a persistent volume to preserve data across container
 restarts.
 
+## Backup, Restore, and Upgrades
+
+### What to back up
+
+All persistent state lives in `DATA_DIR` (`/var/lib/webhooker` by
+default). Back up that directory in full — the state is spread across
+several files whose names depend on your data, so copying the directory
+is both the simplest and the only complete rule:
+
+- `webhooker.db` — one per install. Settings (including the session
+  encryption key), users, API keys, webhooks, entrypoints, targets.
+- `events-{webhook_uuid}.db` — **one per webhook**. Events, deliveries,
+  delivery results.
+- `archive-{webhook_uuid}.db` — **one per webhook that has a `database`
+  target**. Archived events. Keyed on the webhook UUID, not the target
+  UUID: a webhook with several `database` targets still has exactly one
+  archive file.
+
+`{webhook_uuid}` is the webhook's UUID primary key in its canonical
+36-character hyphenated form, so a real filename looks like
+`events-3f2a1c9e-....db`. Nothing else is written to `DATA_DIR`, and no
+`-wal` or `-shm` files are produced (see below); a transient
+`{name}.db-journal` may exist beside a database while a write is in
+flight and is not part of the backup set.
+
+Configuration is **not** in `DATA_DIR` — it comes from the environment
+and from a `.env` file read out of the process working directory. Back
+that up with your deployment config, separately.
+
+### A hot copy is not safe
+
+No `journal_mode` pragma is ever issued on any database webhooker opens,
+so all of them run on SQLite's default rollback journal. There is no
+WAL. The main and event databases are also held open for the entire
+process lifetime — `WebhookDBManager` caches event database handles and
+closes them only on webhook deletion or shutdown — so "it looked idle"
+is not a guarantee that nothing was mid-transaction.
+
+That means `cp`, `rsync`, `tar` or a filesystem snapshot taken against a
+running instance can capture a database mid-transaction and yield a file
+that is corrupt or missing state the journal would have rolled back. Use
+one of the two procedures below instead.
+
+**Stop, copy, start.** The simplest, needs no extra tooling, and the
+only one that gives a single point in time across every file:
+
+```bash
+docker stop webhooker
+cp -a /path/to/data /path/to/backup-$(date -u +%Y%m%dT%H%M%SZ)
+docker start webhooker
+```
+
+**SQLite online backup.** No downtime, one file at a time:
+
+```bash
+for db in /path/to/data/*.db; do
+    sqlite3 "$db" ".backup '/path/to/backup/$(basename "$db")'"
+done
+```
+
+`.backup` takes the proper locks and writes a consistent file. Two
+caveats. First, the runtime image is `alpine:3.21` with only
+`ca-certificates` added — the `sqlite3` CLI is **not** in it, so run
+this on the host against the volume path, or from a throwaway container
+that mounts the volume. Second, each file is captured at its own
+instant, so a webhook created or an event delivered between two files
+being copied lands in one and not the other. If you need the whole set
+coherent as of a single moment, stop the service.
+
+Archive databases are the one exception the service is built for: the
+archive writer closes its handle after each write (debounced to at most
+one reopen per second), so an operator can move `archive-{uuid}.db`
+away for offline retention while the service runs, and it is recreated
+on the next write (see
+[Database Architecture](#database-architecture)). That is a
+move-the-file-away workflow, not a substitute for the backup procedures
+above.
+
+### Restore
+
+1. Stop the service.
+
+2. Restore the **whole set together**: `webhooker.db` *and* every
+   `events-*.db` *and* every `archive-*.db`. A partial restore fails
+   quietly rather than loudly. Every database is opened `mode=rwc`, so a
+   missing `events-{uuid}.db` is **created empty** on first access
+   instead of erroring — the webhook comes back with its configuration
+   intact and its entire event history silently gone. Event databases
+   restored without `webhooker.db` are simply orphaned; nothing
+   references their UUIDs.
+
+3. Do not carry `*.db-journal` files into the restore. Backups taken by
+   either procedure above are self-consistent and do not need one.
+
+4. **Fix ownership.** The container runs as the non-root `webhooker`
+   user, UID 1000 / GID 1000. Restored files must be owned by (or
+   writable by) that UID, and so must the directory itself — SQLite
+   creates the rollback journal beside the database, so a writable file
+   inside a directory it cannot write is not enough:
+
+    ```bash
+    chown -R 1000:1000 /path/to/data
+    ```
+
+    Restoring as `root` on the host and forgetting this step is the
+    usual way a restore fails.
+
+5. Start the service. `AutoMigrate` runs against each restored database
+   as it is opened.
+
+### Upgrades
+
+**Back up before every upgrade.** Every start runs GORM `AutoMigrate`
+unconditionally against whatever files it finds:
+
+- the main database on connect — `Setting`, `User`, `APIKey`, `Webhook`,
+  `Entrypoint`, `Target`
+- each event database when it is lazily opened — `Event`, `Delivery`,
+  `DeliveryResult`
+- each archive database on every open and reopen
+
+There is no schema version table, no migration ledger, and no down
+migrations. Nothing in the files records which version wrote them, and
+no code path undoes a migration.
+
+Upgrade procedure:
+
+1. Stop the service.
+2. Back up `DATA_DIR` using one of the procedures above.
+3. Pull the new image and start it.
+4. Confirm `database migrations completed` in the logs before putting
+   traffic back on it.
+
+**Downgrade is unsupported.** Once a newer binary has migrated the files
+there is no way to move them back. `AutoMigrate` is additive — it adds
+tables, columns and indexes and never drops or rewrites them — so an
+older binary will generally open migrated files and appear to work while
+writing against a schema it does not know about. The failure mode is
+silent divergence, not a startup error. The only supported way back to
+an older version is restoring the pre-upgrade backup, which discards
+everything received since that backup was taken.
+
+### Backups contain secrets
+
+Treat a backup with the same care as the credentials inside it. Encrypt
+backups at rest and restrict who can read them.
+
+- `events-{uuid}.db` and `archive-{uuid}.db` hold the **full payload
+  body and headers** of every event as received, including whatever the
+  sending service put in them — tokens, signatures, personal data.
+- Until
+  [issue #206](https://git.eeqj.de/sneak/webhooker/issues/206) is fixed,
+  the event databases **also contain target credentials**: a GORM
+  association upsert on the delivery and retry write path copies
+  `targets` rows, `config` included, into the per-webhook database. For
+  a Slack target the `webhookUrl` *is* the bearer credential, and an
+  `http` target's URL can embed userinfo. Handing someone an
+  `events-*.db` today hands them live delivery destinations.
+- `webhooker.db` stores target config **unencrypted**, tracked at
+  [issue #212](https://git.eeqj.de/sneak/webhooker/issues/212), next to
+  the session encryption key and the Argon2id password hashes.
+
 ## Entrypoints
 
 This repository adheres to the