43f72e0fd8
All checks were successful
check / check (push) Successful in 3m33s
An operator running `sqlite3 <db> .dump` against their own per-webhook database wedged it: inbound webhooks rejected with HTTP 500, delivered webhooks stranded at `pending`, and every one of them POSTed a second time on the next restart while the event log recorded a single attempt. Durability. Every SQLite file — main, per-webhook, and archive — now opens through one path, `internal/database/sqlite_open.go`, in WAL journal mode with a 10-second busy timeout, `BEGIN IMMEDIATE` transactions, and a bounded connection pool. WAL is what stops a reader blocking writers at all. `_txlock=immediate` is what stops a `COMMIT` failing while its transaction stays open on a pooled connection, which is how four `database is locked` errors became 593 `cannot start a transaction within a transaction`. `cache=shared` is gone, because under it an in-process conflict is SQLITE_LOCKED, which the busy handler does not retry. The busy timeout is applied before journal_mode: the driver runs DSN pragmas in order on every new connection, and `PRAGMA journal_mode` takes a lock, so the reverse order leaves the one pragma that can block uncovered by the handler meant to cover it. Eligibility. `internal/delivery/inflight.go` holds the set of deliveries the engine owns — taken when a task is queued, when a target schedules a retry, and by every recovery path before it re-dispatches; dropped when the worker that ran the task returns. Recovery and both sweep arms re-dispatch only what the set does not hold. Nothing decides that from a row's age: a delivery waiting in a 10000-deep channel is arbitrarily old and perfectly healthy, and reasoning from age re-sends it. `takeForRedispatch` is the single gate every re-dispatch goes through — ownership first, then a conditional update confirming the row is still in the status the batch read. Bookkeeping. `recordResult` and `updateDeliveryStatus` return their errors instead of logging and dropping them, and a caller whose bookkeeping write failed writes nothing at all: the delivery keeps whichever non-terminal status it already held, and the sweeps recover it. Every recovery path — pending and retrying alike — first settles any delivery that already holds a successful `DeliveryResult` rather than sending it again. Recovery continues each delivery's own attempt numbering instead of restarting at 1. The sweep gains a `pending`-with-age-bound arm, so a stranded delivery no longer waits for a restart. Docs. WAL produces `-wal`/`-shm` sidecars, so the backup and restore procedures in README.md are corrected against measurement: both documented procedures were re-run against a live instance, a `-wal` left by a crash carries data the `.db` alone does not, and an archive file normally holds its rows in a `-wal` rather than in the `.db`.
111 lines
3.6 KiB
Go
111 lines
3.6 KiB
Go
package delivery
|
|
|
|
import "sync"
|
|
|
|
// inflightSet records which deliveries the engine currently owns.
|
|
//
|
|
// A delivery is owned from the moment a task for it is handed to a
|
|
// channel or to a retry timer until the engine has no further plan for
|
|
// it in memory. Restart recovery and both arms of the periodic sweep
|
|
// re-dispatch only deliveries the set does not hold, which is what
|
|
// makes them exact rather than a guess about how long a row has sat at
|
|
// pending.
|
|
//
|
|
// This replaces reasoning from timestamps. A delivery's row says
|
|
// pending from creation until its outcome is written, which covers
|
|
// four different situations — never dispatched, waiting in a channel,
|
|
// being attempted right now, and genuinely stranded — and no column
|
|
// distinguishes them. Only the engine knows which, and it knows
|
|
// exactly. `deliveryChannelSize` is 10000 against 10 workers, so a
|
|
// perfectly healthy delivery can wait far longer than any age bound
|
|
// worth setting before its attempt even begins; an age bound alone
|
|
// re-sends it. See
|
|
// https://git.eeqj.de/sneak/webhooker/issues/256.
|
|
//
|
|
// In-memory state is sufficient because a data directory admits one
|
|
// process: internal/datadir takes an flock on it at startup and a
|
|
// second instance refuses to run. Deliveries owned by a process that
|
|
// died are not in any successor's set, and restart recovery is what
|
|
// picks those up.
|
|
//
|
|
// References are counted rather than held as a plain set because
|
|
// ownership outlives the worker that took it. A target that schedules
|
|
// a retry from inside Deliver adds a reference while the worker still
|
|
// holds one, so the delivery stays owned across the gap between the
|
|
// worker returning and the timer firing — the window in which a sweep
|
|
// would otherwise find the row at retrying and send it again.
|
|
//
|
|
// The zero value is ready to use, and the Engine holds one by value.
|
|
// That is deliberate: an engine built by a constructor that forgot to
|
|
// initialise this would not refuse to re-dispatch anything, and the
|
|
// symptom would be duplicate deliveries rather than a failure anybody
|
|
// notices.
|
|
type inflightSet struct {
|
|
mu sync.Mutex
|
|
ids map[string]int
|
|
}
|
|
|
|
// retain adds a reference to a delivery the caller already knows the
|
|
// engine owns, so that ownership survives the current holder letting
|
|
// go. It cannot fail.
|
|
func (s *inflightSet) retain(deliveryID string) {
|
|
s.mu.Lock()
|
|
defer s.mu.Unlock()
|
|
|
|
if s.ids == nil {
|
|
s.ids = make(map[string]int)
|
|
}
|
|
|
|
s.ids[deliveryID]++
|
|
}
|
|
|
|
// retainIdle takes the first reference on a delivery, and reports
|
|
// whether it got it. It fails when the engine already owns the
|
|
// delivery, which is what makes two claimants — restart recovery and
|
|
// the sweep run concurrently, or two sweep arms — mutually exclusive
|
|
// rather than merely atomic.
|
|
func (s *inflightSet) retainIdle(deliveryID string) bool {
|
|
s.mu.Lock()
|
|
defer s.mu.Unlock()
|
|
|
|
if s.ids[deliveryID] > 0 {
|
|
return false
|
|
}
|
|
|
|
if s.ids == nil {
|
|
s.ids = make(map[string]int)
|
|
}
|
|
|
|
s.ids[deliveryID] = 1
|
|
|
|
return true
|
|
}
|
|
|
|
// release drops one reference. The delivery becomes eligible for
|
|
// re-dispatch again once the last one goes.
|
|
func (s *inflightSet) release(deliveryID string) {
|
|
s.mu.Lock()
|
|
defer s.mu.Unlock()
|
|
|
|
n := s.ids[deliveryID] - 1
|
|
if n <= 0 {
|
|
delete(s.ids, deliveryID)
|
|
|
|
return
|
|
}
|
|
|
|
s.ids[deliveryID] = n
|
|
}
|
|
|
|
// held reports how many deliveries the engine currently owns. It
|
|
// exists so a test can assert that ownership is released rather than
|
|
// leaked: a reference that is never dropped hides its delivery from
|
|
// every sweep for the life of the process, which is the one way this
|
|
// mechanism can fail silently.
|
|
func (s *inflightSet) held() int {
|
|
s.mu.Lock()
|
|
defer s.mu.Unlock()
|
|
|
|
return len(s.ids)
|
|
}
|