Document backup, restore and upgrade procedures (closes #210)
All checks were successful
check / check (push) Successful in 10s
All checks were successful
check / check (push) Successful in 10s
The README had nothing on any of the three, leaving an operator with
no answer to which files to back up, whether a hot copy is safe, how
to restore, or what a newer image does to their data.
Adds a "Backup, Restore, and Upgrades" section covering:
- the backup set: webhooker.db, one events-{uuid}.db per webhook, one
archive-{uuid}.db per webhook with a database target, all under
DATA_DIR, plus the .env that is not under it
- why a hot copy is unsafe (no journal_mode pragma is issued on any
DSN, so every database runs the default rollback journal, and the
main and event handles stay open for the process lifetime), with
stop-copy-start and sqlite3 .backup as the safe procedures
- restore: the whole set together, since a missing events DB is
created empty rather than erroring, and chown to UID 1000 because
SQLite needs to write the journal beside the database
- upgrade: AutoMigrate runs unconditionally on every start against
all three database kinds, there is no schema version table and no
down migrations, so back up first and downgrade is unsupported
- that event and archive databases hold full payload bodies and
headers, that event databases also carry target credentials until
#206 is fixed, and that target config in webhooker.db is
unencrypted, tracked in #212
Docs only; no code, config, script or build changes.
This commit is contained in:
162
README.md
162
README.md
@@ -263,6 +263,168 @@ databases written by `database` targets (`archive-{uuid}.db`). Mount
|
||||
this as a persistent volume to preserve data across container
|
||||
restarts.
|
||||
|
||||
## Backup, Restore, and Upgrades
|
||||
|
||||
### What to back up
|
||||
|
||||
All persistent state lives in `DATA_DIR` (`/var/lib/webhooker` by
|
||||
default). Back up that directory in full — the state is spread across
|
||||
several files whose names depend on your data, so copying the directory
|
||||
is both the simplest and the only complete rule:
|
||||
|
||||
- `webhooker.db` — one per install. Settings (including the session
|
||||
encryption key), users, API keys, webhooks, entrypoints, targets.
|
||||
- `events-{webhook_uuid}.db` — **one per webhook**. Events, deliveries,
|
||||
delivery results.
|
||||
- `archive-{webhook_uuid}.db` — **one per webhook that has a `database`
|
||||
target**. Archived events. Keyed on the webhook UUID, not the target
|
||||
UUID: a webhook with several `database` targets still has exactly one
|
||||
archive file.
|
||||
|
||||
`{webhook_uuid}` is the webhook's UUID primary key in its canonical
|
||||
36-character hyphenated form, so a real filename looks like
|
||||
`events-3f2a1c9e-....db`. Nothing else is written to `DATA_DIR`, and no
|
||||
`-wal` or `-shm` files are produced (see below); a transient
|
||||
`{name}.db-journal` may exist beside a database while a write is in
|
||||
flight and is not part of the backup set.
|
||||
|
||||
Configuration is **not** in `DATA_DIR` — it comes from the environment
|
||||
and from a `.env` file read out of the process working directory. Back
|
||||
that up with your deployment config, separately.
|
||||
|
||||
### A hot copy is not safe
|
||||
|
||||
No `journal_mode` pragma is ever issued on any database webhooker opens,
|
||||
so all of them run on SQLite's default rollback journal. There is no
|
||||
WAL. The main and event databases are also held open for the entire
|
||||
process lifetime — `WebhookDBManager` caches event database handles and
|
||||
closes them only on webhook deletion or shutdown — so "it looked idle"
|
||||
is not a guarantee that nothing was mid-transaction.
|
||||
|
||||
That means `cp`, `rsync`, `tar` or a filesystem snapshot taken against a
|
||||
running instance can capture a database mid-transaction and yield a file
|
||||
that is corrupt or missing state the journal would have rolled back. Use
|
||||
one of the two procedures below instead.
|
||||
|
||||
**Stop, copy, start.** The simplest, needs no extra tooling, and the
|
||||
only one that gives a single point in time across every file:
|
||||
|
||||
```bash
|
||||
docker stop webhooker
|
||||
cp -a /path/to/data /path/to/backup-$(date -u +%Y%m%dT%H%M%SZ)
|
||||
docker start webhooker
|
||||
```
|
||||
|
||||
**SQLite online backup.** No downtime, one file at a time:
|
||||
|
||||
```bash
|
||||
for db in /path/to/data/*.db; do
|
||||
sqlite3 "$db" ".backup '/path/to/backup/$(basename "$db")'"
|
||||
done
|
||||
```
|
||||
|
||||
`.backup` takes the proper locks and writes a consistent file. Two
|
||||
caveats. First, the runtime image is `alpine:3.21` with only
|
||||
`ca-certificates` added — the `sqlite3` CLI is **not** in it, so run
|
||||
this on the host against the volume path, or from a throwaway container
|
||||
that mounts the volume. Second, each file is captured at its own
|
||||
instant, so a webhook created or an event delivered between two files
|
||||
being copied lands in one and not the other. If you need the whole set
|
||||
coherent as of a single moment, stop the service.
|
||||
|
||||
Archive databases are the one exception the service is built for: the
|
||||
archive writer closes its handle after each write (debounced to at most
|
||||
one reopen per second), so an operator can move `archive-{uuid}.db`
|
||||
away for offline retention while the service runs, and it is recreated
|
||||
on the next write (see
|
||||
[Database Architecture](#database-architecture)). That is a
|
||||
move-the-file-away workflow, not a substitute for the backup procedures
|
||||
above.
|
||||
|
||||
### Restore
|
||||
|
||||
1. Stop the service.
|
||||
|
||||
2. Restore the **whole set together**: `webhooker.db` *and* every
|
||||
`events-*.db` *and* every `archive-*.db`. A partial restore fails
|
||||
quietly rather than loudly. Every database is opened `mode=rwc`, so a
|
||||
missing `events-{uuid}.db` is **created empty** on first access
|
||||
instead of erroring — the webhook comes back with its configuration
|
||||
intact and its entire event history silently gone. Event databases
|
||||
restored without `webhooker.db` are simply orphaned; nothing
|
||||
references their UUIDs.
|
||||
|
||||
3. Do not carry `*.db-journal` files into the restore. Backups taken by
|
||||
either procedure above are self-consistent and do not need one.
|
||||
|
||||
4. **Fix ownership.** The container runs as the non-root `webhooker`
|
||||
user, UID 1000 / GID 1000. Restored files must be owned by (or
|
||||
writable by) that UID, and so must the directory itself — SQLite
|
||||
creates the rollback journal beside the database, so a writable file
|
||||
inside a directory it cannot write is not enough:
|
||||
|
||||
```bash
|
||||
chown -R 1000:1000 /path/to/data
|
||||
```
|
||||
|
||||
Restoring as `root` on the host and forgetting this step is the
|
||||
usual way a restore fails.
|
||||
|
||||
5. Start the service. `AutoMigrate` runs against each restored database
|
||||
as it is opened.
|
||||
|
||||
### Upgrades
|
||||
|
||||
**Back up before every upgrade.** Every start runs GORM `AutoMigrate`
|
||||
unconditionally against whatever files it finds:
|
||||
|
||||
- the main database on connect — `Setting`, `User`, `APIKey`, `Webhook`,
|
||||
`Entrypoint`, `Target`
|
||||
- each event database when it is lazily opened — `Event`, `Delivery`,
|
||||
`DeliveryResult`
|
||||
- each archive database on every open and reopen
|
||||
|
||||
There is no schema version table, no migration ledger, and no down
|
||||
migrations. Nothing in the files records which version wrote them, and
|
||||
no code path undoes a migration.
|
||||
|
||||
Upgrade procedure:
|
||||
|
||||
1. Stop the service.
|
||||
2. Back up `DATA_DIR` using one of the procedures above.
|
||||
3. Pull the new image and start it.
|
||||
4. Confirm `database migrations completed` in the logs before putting
|
||||
traffic back on it.
|
||||
|
||||
**Downgrade is unsupported.** Once a newer binary has migrated the files
|
||||
there is no way to move them back. `AutoMigrate` is additive — it adds
|
||||
tables, columns and indexes and never drops or rewrites them — so an
|
||||
older binary will generally open migrated files and appear to work while
|
||||
writing against a schema it does not know about. The failure mode is
|
||||
silent divergence, not a startup error. The only supported way back to
|
||||
an older version is restoring the pre-upgrade backup, which discards
|
||||
everything received since that backup was taken.
|
||||
|
||||
### Backups contain secrets
|
||||
|
||||
Treat a backup with the same care as the credentials inside it. Encrypt
|
||||
backups at rest and restrict who can read them.
|
||||
|
||||
- `events-{uuid}.db` and `archive-{uuid}.db` hold the **full payload
|
||||
body and headers** of every event as received, including whatever the
|
||||
sending service put in them — tokens, signatures, personal data.
|
||||
- Until
|
||||
[issue #206](https://git.eeqj.de/sneak/webhooker/issues/206) is fixed,
|
||||
the event databases **also contain target credentials**: a GORM
|
||||
association upsert on the delivery and retry write path copies
|
||||
`targets` rows, `config` included, into the per-webhook database. For
|
||||
a Slack target the `webhookUrl` *is* the bearer credential, and an
|
||||
`http` target's URL can embed userinfo. Handing someone an
|
||||
`events-*.db` today hands them live delivery destinations.
|
||||
- `webhooker.db` stores target config **unencrypted**, tracked at
|
||||
[issue #212](https://git.eeqj.de/sneak/webhooker/issues/212), next to
|
||||
the session encryption key and the Argon2id password hashes.
|
||||
|
||||
## Entrypoints
|
||||
|
||||
This repository adheres to the
|
||||
|
||||
Reference in New Issue
Block a user