Make SQLite durable under concurrent readers and stop re-delivering stranded webhooks (closes #256)
All checks were successful
check / check (push) Successful in 3m3s
All checks were successful
check / check (push) Successful in 3m3s
An operator running `sqlite3 <db> .dump` against their own per-webhook database wedged it: inbound webhooks rejected with HTTP 500, delivered webhooks stranded at `pending`, and every one of them POSTed a second time on the next restart while the event log recorded a single attempt. Durability. Every SQLite file — main, per-webhook, and archive — now opens through one path, `internal/database/sqlite_open.go`, in WAL journal mode with a 10-second busy timeout, `BEGIN IMMEDIATE` transactions, and a bounded connection pool. WAL is what stops a reader blocking writers at all. `_txlock=immediate` is what stops a `COMMIT` failing while its transaction stays open on a pooled connection, which is how four `database is locked` errors became 593 `cannot start a transaction within a transaction`. `cache=shared` is gone, because under it an in-process conflict is SQLITE_LOCKED, which the busy handler does not retry. The busy timeout is applied before journal_mode: the driver runs DSN pragmas in order on every new connection, and `PRAGMA journal_mode` takes a lock, so the reverse order leaves the one pragma that can block uncovered by the handler meant to cover it. Eligibility. `internal/delivery/inflight.go` holds the set of deliveries the engine owns — taken when a task is queued, when a target schedules a retry, and by every recovery path before it re-dispatches; dropped when the worker that ran the task returns. Recovery and both sweep arms re-dispatch only what the set does not hold. Nothing decides that from a row's age: a delivery waiting in a 10000-deep channel is arbitrarily old and perfectly healthy, and reasoning from age re-sends it. `takeForRedispatch` is the single gate every re-dispatch goes through — ownership first, then a conditional update confirming the row is still in the status the batch read. Bookkeeping. `recordResult` and `updateDeliveryStatus` return their errors instead of logging and dropping them, and a caller whose bookkeeping write failed writes nothing at all: the delivery keeps whichever non-terminal status it already held, and the sweeps recover it. Every recovery path — pending and retrying alike — first settles any delivery that already holds a successful `DeliveryResult` rather than sending it again. Recovery continues each delivery's own attempt numbering instead of restarting at 1. The sweep gains a `pending`-with-age-bound arm, so a stranded delivery no longer waits for a restart. Docs. WAL produces `-wal`/`-shm` sidecars, so the backup and restore procedures in README.md are corrected against measurement: both documented procedures were re-run against a live instance, a `-wal` left by a crash carries data the `.db` alone does not, and an archive file normally holds its rows in a `-wal` rather than in the `.db`.
This commit is contained in:
@@ -33,6 +33,11 @@ const (
|
||||
// response is written against this number, so a test has to
|
||||
// be able to name it.
|
||||
ExportMaxBodyLog = maxBodyLog
|
||||
|
||||
// ExportPendingSweepMinAge is how long a delivery must sit at
|
||||
// pending before the sweep treats it as stranded. A test has to
|
||||
// name it to age a row past the bound.
|
||||
ExportPendingSweepMinAge = pendingSweepMinAge
|
||||
)
|
||||
|
||||
// ExportIsBlockedIP exposes isBlockedIP for testing.
|
||||
@@ -286,6 +291,26 @@ func (e *Engine) ExportWedgeWorker(release <-chan struct{}) {
|
||||
})
|
||||
}
|
||||
|
||||
// ExportInflightHeld reports how many deliveries the engine currently
|
||||
// owns, so a test can prove ownership is released rather than leaked.
|
||||
func (e *Engine) ExportInflightHeld() int {
|
||||
return e.inflight.held()
|
||||
}
|
||||
|
||||
// ExportRetainDelivery takes the first reference on a delivery, as the
|
||||
// queueing side does. It lets a test put a delivery into the state a
|
||||
// worker or a full channel would, without running the pool.
|
||||
func (e *Engine) ExportRetainDelivery(deliveryID string) bool {
|
||||
return e.inflight.retainIdle(deliveryID)
|
||||
}
|
||||
|
||||
// ExportRecoverRetryingDeliveries exposes recoverRetryingDeliveries.
|
||||
func (e *Engine) ExportRecoverRetryingDeliveries(
|
||||
webhookDB *gorm.DB, webhookID string,
|
||||
) {
|
||||
e.recoverRetryingDeliveries(webhookDB, webhookID)
|
||||
}
|
||||
|
||||
// ExportDeliveryCh returns the delivery channel.
|
||||
func (e *Engine) ExportDeliveryCh() chan Task {
|
||||
return e.deliveryCh
|
||||
|
||||
Reference in New Issue
Block a user