Document backup, restore and upgrade procedures (closes #210)
All checks were successful
check / check (push) Successful in 10s

The README had nothing on any of the three, leaving an operator with
no answer to which files to back up, whether a hot copy is safe, how
to restore, or what a newer image does to their data.

Adds a "Backup, Restore, and Upgrades" section covering:

- the backup set: webhooker.db, one events-{uuid}.db per webhook, one
  archive-{uuid}.db per webhook with a database target, all under
  DATA_DIR, plus the .env that is not under it
- why a hot copy is unsafe (no journal_mode pragma is issued on any
  DSN, so every database runs the default rollback journal, and the
  main and event handles stay open for the process lifetime), with
  stop-copy-start and sqlite3 .backup as the safe procedures
- restore: the whole set together, since a missing events DB is
  created empty rather than erroring, and chown to UID 1000 because
  SQLite needs to write the journal beside the database
- upgrade: AutoMigrate runs unconditionally on every start against
  all three database kinds, there is no schema version table and no
  down migrations, so back up first and downgrade is unsupported
- that event and archive databases hold full payload bodies and
  headers, that event databases also carry target credentials until
  #206 is fixed, and that target config in webhooker.db is
  unencrypted, tracked in #212

Docs only; no code, config, script or build changes.
This commit is contained in:
clawbot
2026-08-20 04:01:49 +00:00
committed by sneak
parent ea51612446
commit 0f2ac5a3c8

162
README.md
View File

@@ -263,6 +263,168 @@ databases written by `database` targets (`archive-{uuid}.db`). Mount
this as a persistent volume to preserve data across container
restarts.
## Backup, Restore, and Upgrades
### What to back up
All persistent state lives in `DATA_DIR` (`/var/lib/webhooker` by
default). Back up that directory in full — the state is spread across
several files whose names depend on your data, so copying the directory
is both the simplest and the only complete rule:
- `webhooker.db` — one per install. Settings (including the session
encryption key), users, API keys, webhooks, entrypoints, targets.
- `events-{webhook_uuid}.db`**one per webhook**. Events, deliveries,
delivery results.
- `archive-{webhook_uuid}.db` — **one per webhook that has a `database`
target**. Archived events. Keyed on the webhook UUID, not the target
UUID: a webhook with several `database` targets still has exactly one
archive file.
`{webhook_uuid}` is the webhook's UUID primary key in its canonical
36-character hyphenated form, so a real filename looks like
`events-3f2a1c9e-....db`. Nothing else is written to `DATA_DIR`, and no
`-wal` or `-shm` files are produced (see below); a transient
`{name}.db-journal` may exist beside a database while a write is in
flight and is not part of the backup set.
Configuration is **not** in `DATA_DIR` — it comes from the environment
and from a `.env` file read out of the process working directory. Back
that up with your deployment config, separately.
### A hot copy is not safe
No `journal_mode` pragma is ever issued on any database webhooker opens,
so all of them run on SQLite's default rollback journal. There is no
WAL. The main and event databases are also held open for the entire
process lifetime — `WebhookDBManager` caches event database handles and
closes them only on webhook deletion or shutdown — so "it looked idle"
is not a guarantee that nothing was mid-transaction.
That means `cp`, `rsync`, `tar` or a filesystem snapshot taken against a
running instance can capture a database mid-transaction and yield a file
that is corrupt or missing state the journal would have rolled back. Use
one of the two procedures below instead.
**Stop, copy, start.** The simplest, needs no extra tooling, and the
only one that gives a single point in time across every file:
```bash
docker stop webhooker
cp -a /path/to/data /path/to/backup-$(date -u +%Y%m%dT%H%M%SZ)
docker start webhooker
```
**SQLite online backup.** No downtime, one file at a time:
```bash
for db in /path/to/data/*.db; do
sqlite3 "$db" ".backup '/path/to/backup/$(basename "$db")'"
done
```
`.backup` takes the proper locks and writes a consistent file. Two
caveats. First, the runtime image is `alpine:3.21` with only
`ca-certificates` added — the `sqlite3` CLI is **not** in it, so run
this on the host against the volume path, or from a throwaway container
that mounts the volume. Second, each file is captured at its own
instant, so a webhook created or an event delivered between two files
being copied lands in one and not the other. If you need the whole set
coherent as of a single moment, stop the service.
Archive databases are the one exception the service is built for: the
archive writer closes its handle after each write (debounced to at most
one reopen per second), so an operator can move `archive-{uuid}.db`
away for offline retention while the service runs, and it is recreated
on the next write (see
[Database Architecture](#database-architecture)). That is a
move-the-file-away workflow, not a substitute for the backup procedures
above.
### Restore
1. Stop the service.
2. Restore the **whole set together**: `webhooker.db` *and* every
`events-*.db` *and* every `archive-*.db`. A partial restore fails
quietly rather than loudly. Every database is opened `mode=rwc`, so a
missing `events-{uuid}.db` is **created empty** on first access
instead of erroring — the webhook comes back with its configuration
intact and its entire event history silently gone. Event databases
restored without `webhooker.db` are simply orphaned; nothing
references their UUIDs.
3. Do not carry `*.db-journal` files into the restore. Backups taken by
either procedure above are self-consistent and do not need one.
4. **Fix ownership.** The container runs as the non-root `webhooker`
user, UID 1000 / GID 1000. Restored files must be owned by (or
writable by) that UID, and so must the directory itself — SQLite
creates the rollback journal beside the database, so a writable file
inside a directory it cannot write is not enough:
```bash
chown -R 1000:1000 /path/to/data
```
Restoring as `root` on the host and forgetting this step is the
usual way a restore fails.
5. Start the service. `AutoMigrate` runs against each restored database
as it is opened.
### Upgrades
**Back up before every upgrade.** Every start runs GORM `AutoMigrate`
unconditionally against whatever files it finds:
- the main database on connect — `Setting`, `User`, `APIKey`, `Webhook`,
`Entrypoint`, `Target`
- each event database when it is lazily opened — `Event`, `Delivery`,
`DeliveryResult`
- each archive database on every open and reopen
There is no schema version table, no migration ledger, and no down
migrations. Nothing in the files records which version wrote them, and
no code path undoes a migration.
Upgrade procedure:
1. Stop the service.
2. Back up `DATA_DIR` using one of the procedures above.
3. Pull the new image and start it.
4. Confirm `database migrations completed` in the logs before putting
traffic back on it.
**Downgrade is unsupported.** Once a newer binary has migrated the files
there is no way to move them back. `AutoMigrate` is additive — it adds
tables, columns and indexes and never drops or rewrites them — so an
older binary will generally open migrated files and appear to work while
writing against a schema it does not know about. The failure mode is
silent divergence, not a startup error. The only supported way back to
an older version is restoring the pre-upgrade backup, which discards
everything received since that backup was taken.
### Backups contain secrets
Treat a backup with the same care as the credentials inside it. Encrypt
backups at rest and restrict who can read them.
- `events-{uuid}.db` and `archive-{uuid}.db` hold the **full payload
body and headers** of every event as received, including whatever the
sending service put in them — tokens, signatures, personal data.
- Until
[issue #206](https://git.eeqj.de/sneak/webhooker/issues/206) is fixed,
the event databases **also contain target credentials**: a GORM
association upsert on the delivery and retry write path copies
`targets` rows, `config` included, into the per-webhook database. For
a Slack target the `webhookUrl` *is* the bearer credential, and an
`http` target's URL can embed userinfo. Handing someone an
`events-*.db` today hands them live delivery destinations.
- `webhooker.db` stores target config **unencrypted**, tracked at
[issue #212](https://git.eeqj.de/sneak/webhooker/issues/212), next to
the session encryption key and the Argon2id password hashes.
## Entrypoints
This repository adheres to the