Make SQLite durable under concurrent readers and stop re-delivering stranded webhooks (closes #256)
All checks were successful
check / check (push) Successful in 3m33s
All checks were successful
check / check (push) Successful in 3m33s
An operator running `sqlite3 <db> .dump` against their own per-webhook database wedged it: inbound webhooks rejected with HTTP 500, delivered webhooks stranded at `pending`, and every one of them POSTed a second time on the next restart while the event log recorded a single attempt. Durability. Every SQLite file — main, per-webhook, and archive — now opens through one path, `internal/database/sqlite_open.go`, in WAL journal mode with a 10-second busy timeout, `BEGIN IMMEDIATE` transactions, and a bounded connection pool. WAL is what stops a reader blocking writers at all. `_txlock=immediate` is what stops a `COMMIT` failing while its transaction stays open on a pooled connection, which is how four `database is locked` errors became 593 `cannot start a transaction within a transaction`. `cache=shared` is gone, because under it an in-process conflict is SQLITE_LOCKED, which the busy handler does not retry. The busy timeout is applied before journal_mode: the driver runs DSN pragmas in order on every new connection, and `PRAGMA journal_mode` takes a lock, so the reverse order leaves the one pragma that can block uncovered by the handler meant to cover it. Eligibility. `internal/delivery/inflight.go` holds the set of deliveries the engine owns — taken when a task is queued, when a target schedules a retry, and by every recovery path before it re-dispatches; dropped when the worker that ran the task returns. Recovery and both sweep arms re-dispatch only what the set does not hold. Nothing decides that from a row's age: a delivery waiting in a 10000-deep channel is arbitrarily old and perfectly healthy, and reasoning from age re-sends it. `takeForRedispatch` is the single gate every re-dispatch goes through — ownership first, then a conditional update confirming the row is still in the status the batch read. Bookkeeping. `recordResult` and `updateDeliveryStatus` return their errors instead of logging and dropping them, and a caller whose bookkeeping write failed writes nothing at all: the delivery keeps whichever non-terminal status it already held, and the sweeps recover it. Every recovery path — pending and retrying alike — first settles any delivery that already holds a successful `DeliveryResult` rather than sending it again. Recovery continues each delivery's own attempt numbering instead of restarting at 1. The sweep gains a `pending`-with-age-bound arm, so a stranded delivery no longer waits for a restart. Docs. WAL produces `-wal`/`-shm` sidecars, so the backup and restore procedures in README.md are corrected against measurement: both documented procedures were re-run against a live instance, a `-wal` left by a crash carries data the `.db` alone does not, and an archive file normally holds its rows in a `-wal` rather than in the `.db`.
This commit is contained in:
@@ -143,10 +143,10 @@ func (h *Handlers) resubmitEvent(
|
||||
}
|
||||
|
||||
// Read before the write transaction is opened. The body can be up
|
||||
// to the 1 MB ingest cap, and holding a read of it inside the
|
||||
// transaction would extend how long the per-webhook database is
|
||||
// locked against the receiver, which runs these files in
|
||||
// SQLite's default journal mode rather than WAL.
|
||||
// to the 1 MB ingest cap, and every transaction on these files
|
||||
// takes the write lock at BEGIN (_txlock=immediate, see
|
||||
// internal/database/sqlite_open.go), so reading inside it would
|
||||
// hold that lock against the receiver for the length of the read.
|
||||
src, found, err := loadResubmitSource(
|
||||
webhookDB, webhook.ID, eventID.String(),
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user