Make SQLite durable under concurrent readers and stop re-delivering stranded webhooks (closes #256)
All checks were successful
check / check (push) Successful in 3m33s
All checks were successful
check / check (push) Successful in 3m33s
An operator running `sqlite3 <db> .dump` against their own per-webhook database wedged it: inbound webhooks rejected with HTTP 500, delivered webhooks stranded at `pending`, and every one of them POSTed a second time on the next restart while the event log recorded a single attempt. Durability. Every SQLite file — main, per-webhook, and archive — now opens through one path, `internal/database/sqlite_open.go`, in WAL journal mode with a 10-second busy timeout, `BEGIN IMMEDIATE` transactions, and a bounded connection pool. WAL is what stops a reader blocking writers at all. `_txlock=immediate` is what stops a `COMMIT` failing while its transaction stays open on a pooled connection, which is how four `database is locked` errors became 593 `cannot start a transaction within a transaction`. `cache=shared` is gone, because under it an in-process conflict is SQLITE_LOCKED, which the busy handler does not retry. The busy timeout is applied before journal_mode: the driver runs DSN pragmas in order on every new connection, and `PRAGMA journal_mode` takes a lock, so the reverse order leaves the one pragma that can block uncovered by the handler meant to cover it. Eligibility. `internal/delivery/inflight.go` holds the set of deliveries the engine owns — taken when a task is queued, when a target schedules a retry, and by every recovery path before it re-dispatches; dropped when the worker that ran the task returns. Recovery and both sweep arms re-dispatch only what the set does not hold. Nothing decides that from a row's age: a delivery waiting in a 10000-deep channel is arbitrarily old and perfectly healthy, and reasoning from age re-sends it. `takeForRedispatch` is the single gate every re-dispatch goes through — ownership first, then a conditional update confirming the row is still in the status the batch read. Bookkeeping. `recordResult` and `updateDeliveryStatus` return their errors instead of logging and dropping them, and a caller whose bookkeeping write failed writes nothing at all: the delivery keeps whichever non-terminal status it already held, and the sweeps recover it. Every recovery path — pending and retrying alike — first settles any delivery that already holds a successful `DeliveryResult` rather than sending it again. Recovery continues each delivery's own attempt numbering instead of restarting at 1. The sweep gains a `pending`-with-age-bound arm, so a stranded delivery no longer waits for a restart. Docs. WAL produces `-wal`/`-shm` sidecars, so the backup and restore procedures in README.md are corrected against measurement: both documented procedures were re-run against a live instance, a `-wal` left by a crash carries data the `.db` alone does not, and an archive file normally holds its rows in a `-wal` rather than in the `.db`.
This commit is contained in:
105
README.md
105
README.md
@@ -566,9 +566,25 @@ is both the simplest and the only complete rule:
|
||||
`events-3f2a1c9e-....db`. The only other file is `webhooker.lock`, the
|
||||
always-empty [single-instance lock](#single-instance-lock); it holds no
|
||||
state and is not part of the backup set — a copied one is stale and
|
||||
blocks nothing. No `-wal` or `-shm` files are produced (see below); a
|
||||
transient `{name}.db-journal` may exist beside a database while a write
|
||||
is in flight and is not part of the backup set either.
|
||||
blocks nothing.
|
||||
|
||||
**`-wal` and `-shm` sidecars.** Every database runs in WAL journal mode,
|
||||
so while the service is running each `{name}.db` has a `{name}.db-wal`
|
||||
and a `{name}.db-shm` beside it. **`-wal` is part of the database, not a
|
||||
scratch file**: it holds committed transactions that are not yet in the
|
||||
`.db`, so a copy of the `.db` without its `-wal` is missing data and may
|
||||
have no readable schema at all. `-shm` is regenerable, but there is no
|
||||
reason to separate the two — copy the directory and you have them.
|
||||
|
||||
A clean shutdown closes `webhooker.db` and every `events-*.db`, which
|
||||
checkpoints and removes their sidecars; a killed or crashed instance
|
||||
leaves them, and they must be carried with the `.db`. **Archive
|
||||
databases are different**: their handle is not closed at shutdown, so
|
||||
`archive-*.db-wal` and `-shm` normally survive a clean stop and the
|
||||
`-wal` can hold every row the archive has. Measured on a stopped
|
||||
instance: `archive-….db` 4096 bytes with no table, its `-wal` 157 KB
|
||||
holding all 8 archived events. Copying `DATA_DIR` in full is what makes
|
||||
this a non-issue; copying `.db` files out of it by name is not.
|
||||
|
||||
Configuration is **not** in `DATA_DIR` — it comes from the environment
|
||||
and from a `.env` file read out of the process working directory. Back
|
||||
@@ -576,17 +592,19 @@ that up with your deployment config, separately.
|
||||
|
||||
### A hot copy is not safe
|
||||
|
||||
No `journal_mode` pragma is ever issued on any database webhooker opens,
|
||||
so all of them run on SQLite's default rollback journal. There is no
|
||||
WAL. The main and event databases are also held open for the entire
|
||||
process lifetime — `WebhookDBManager` caches event database handles and
|
||||
closes them only on webhook deletion or shutdown — so "it looked idle"
|
||||
is not a guarantee that nothing was mid-transaction.
|
||||
Every database webhooker opens runs in WAL journal mode. The main and
|
||||
event databases are also held open for the entire process lifetime —
|
||||
`WebhookDBManager` caches event database handles and closes them only on
|
||||
webhook deletion or shutdown — so "it looked idle" is not a guarantee
|
||||
that nothing was mid-transaction.
|
||||
|
||||
That means `cp`, `rsync`, `tar` or a filesystem snapshot taken against a
|
||||
running instance can capture a database mid-transaction and yield a file
|
||||
that is corrupt or missing state the journal would have rolled back. Use
|
||||
one of the two procedures below instead.
|
||||
running instance can capture a database and its `-wal` at two different
|
||||
instants and yield a file that is corrupt or missing state. Copying a
|
||||
`.db` on its own is worse and fails loudly: recently written pages,
|
||||
including the schema itself on a young database, live in the `-wal`, so
|
||||
the copy reads back as an empty or table-less database. Use one of the
|
||||
two procedures below instead.
|
||||
|
||||
**Stop, copy, start.** The simplest, needs no extra tooling, and the
|
||||
only one that gives a single point in time across every file:
|
||||
@@ -605,8 +623,9 @@ for db in /path/to/data/*.db; do
|
||||
done
|
||||
```
|
||||
|
||||
`.backup` takes the proper locks and writes a consistent file. Two
|
||||
caveats. First, the runtime image is `alpine:3.21` with only
|
||||
`.backup` reads through the WAL and writes a single consistent file with
|
||||
no sidecars of its own, so the destination is complete as it stands.
|
||||
Two caveats. First, the runtime image is `alpine:3.21` with only
|
||||
`ca-certificates` added — the `sqlite3` CLI is **not** in it, so run
|
||||
this on the host against the volume path, or from a throwaway container
|
||||
that mounts the volume. Second, each file is captured at its own
|
||||
@@ -614,15 +633,37 @@ instant, so a webhook created or an event delivered between two files
|
||||
being copied lands in one and not the other. If you need the whole set
|
||||
coherent as of a single moment, stop the service.
|
||||
|
||||
Note that `sqlite3 <db> .dump` is **not** one of these procedures: it is
|
||||
an export, it holds a read transaction open for as long as it runs, and
|
||||
it pins the WAL against checkpointing for that whole time. It is safe to
|
||||
run — it does not block ingestion — but back up with `.backup` or a
|
||||
stopped copy.
|
||||
|
||||
Archive databases are the one exception the service is built for: the
|
||||
archive writer closes its handle after each write (debounced to at most
|
||||
one reopen per second), so an operator can move `archive-{uuid}.db`
|
||||
away for offline retention while the service runs, and it is recreated
|
||||
on the next write (see
|
||||
[Database Architecture](#database-architecture)). That is a
|
||||
archive writer closes and reopens its handle around writes (debounced
|
||||
to at most one reopen per second), so an operator can move
|
||||
`archive-{uuid}.db` away for offline retention while the service runs,
|
||||
and it is recreated on the next write. See
|
||||
[Database Architecture](#database-architecture). That is a
|
||||
move-the-file-away workflow, not a substitute for the backup procedures
|
||||
above.
|
||||
|
||||
**Move the sidecars with it.** Under WAL that workflow is no longer a
|
||||
single file, and the common case is the dangerous one. The reopen
|
||||
happens on the *next* write after the debounce window elapses, so after
|
||||
the last write of a burst nothing checkpoints: measured, 20 s after ten
|
||||
events the `archive-….db` was 4096 bytes — a header, no table — with
|
||||
all ten rows sitting in a 189 KB `-wal`. Copying the `.db` alone at that
|
||||
moment yields a file that opens with `no such table: archived_events`.
|
||||
The file becomes self-contained again when the handle closes, which
|
||||
happens on the next write past the debounce window, when the connection
|
||||
pool retires the idle connection (about a minute after the last write),
|
||||
or at the idle archive sweep — measured, the same file was a complete
|
||||
20 KB `.db` with no sidecars about a minute after its last write.
|
||||
Shutdown is **not** on that list: the archive handle is not closed when
|
||||
the service stops. So either move `archive-{uuid}.db` together with any
|
||||
`-wal`/`-shm` beside it, or wait until there are none.
|
||||
|
||||
### Restore
|
||||
|
||||
1. Stop the service.
|
||||
@@ -636,14 +677,22 @@ above.
|
||||
restored without `webhooker.db` are simply orphaned; nothing
|
||||
references their UUIDs.
|
||||
|
||||
3. Do not carry `*.db-journal` files into the restore. Backups taken by
|
||||
either procedure above are self-consistent and do not need one.
|
||||
3. Carry any `*.db-wal` and `*.db-shm` files that are in the backup.
|
||||
They are part of the database, and dropping a `-wal` silently
|
||||
discards every transaction it still holds. An `.backup` set will not
|
||||
contain any: it writes a single consolidated file per database. A
|
||||
stop-and-copy set has none for `webhooker.db` or the `events-*.db`,
|
||||
because a clean stop closes those and checkpoints their sidecars
|
||||
away — but it will normally have them for `archive-*.db`, whose
|
||||
handle stays open across shutdown, and those carry the archive's
|
||||
rows. A copy salvaged from a crashed instance has them for
|
||||
everything, and needs all of them.
|
||||
|
||||
4. **Fix ownership.** The container runs as the non-root `webhooker`
|
||||
user, UID 1000 / GID 1000. Restored files must be owned by (or
|
||||
writable by) that UID, and so must the directory itself — SQLite
|
||||
creates the rollback journal beside the database, so a writable file
|
||||
inside a directory it cannot write is not enough:
|
||||
creates the `-wal` and `-shm` sidecars beside the database, so a
|
||||
writable file inside a directory it cannot write is not enough:
|
||||
|
||||
```bash
|
||||
chown -R 1000:1000 /path/to/data
|
||||
@@ -1391,10 +1440,12 @@ This separation provides:
|
||||
only, or disables cleanup entirely when set to `0` (retain forever).
|
||||
- **Performance** — each webhook's database has its own page cache and
|
||||
its own lock, so concurrent event ingestion across webhooks won't
|
||||
contend. No write-ahead log is involved: both DSNs are
|
||||
`file:{path}?cache=shared&mode=rwc` and no `journal_mode` pragma is
|
||||
ever issued, so every database runs on SQLite's default rollback
|
||||
journal.
|
||||
contend. Every database — main, per-webhook, and archive — is opened
|
||||
through one code path (`internal/database/sqlite_open.go`) in WAL
|
||||
journal mode, with a 10-second busy timeout, `BEGIN IMMEDIATE`
|
||||
transactions, and a bounded connection pool. Under WAL a reader never
|
||||
blocks a writer, so an operator reading a database does not stall
|
||||
event ingestion into it.
|
||||
|
||||
The **database target type** builds on this architecture to provide
|
||||
long-term archiving, separate from the per-webhook event database (which
|
||||
|
||||
Reference in New Issue
Block a user