Make SQLite durable under concurrent readers and stop re-delivering stranded webhooks (closes #256)

An operator running `sqlite3 <db> .dump` against their own per-webhook
database wedged it: 60 of 60 inbound webhooks rejected with HTTP 500,
206 delivered webhooks stranded at `pending`, and every one of them
POSTed a second time on the next restart while the event log recorded a
single attempt.

Durability. Every SQLite file — main, per-webhook, and archive — now
opens through one path, `internal/database/sqlite_open.go`, in WAL
journal mode with a 10-second busy timeout, `BEGIN IMMEDIATE`
transactions, and a bounded connection pool. WAL is what stops a
reader blocking writers at all. `_txlock=immediate` is what stops a
`COMMIT` failing while its transaction stays open on a pooled
connection, which is how four `database is locked` errors became 593
`cannot start a transaction within a transaction`: a deferred
transaction that upgrades to a write lock mid-flight gets SQLITE_BUSY
without the busy handler being consulted. `cache=shared` is gone,
because under it an in-process conflict is SQLITE_LOCKED, which the
busy handler does not retry.

Delivery. `recordResult` and `updateDeliveryStatus` return their
errors instead of logging and dropping them, and a caller whose
bookkeeping write failed writes nothing at all — the delivery keeps
whichever non-terminal status it already held, and both sweeps recover
it. Recovery and the sweep now reconcile before re-sending: a pending
delivery that already holds a successful `DeliveryResult` is marked
delivered rather than sent again, which is the state that did not
previously exist. A delivery handed back out is claimed by
compare-and-set so successive sweeps cannot send it repeatedly, and it
continues its own attempt numbering instead of restarting at 1. The
sweep gains a `pending`-with-age-bound arm, so a stranded delivery no
longer waits for a restart.

Docs. WAL produces `-wal`/`-shm` sidecars, so the backup and restore
procedures in README.md are corrected: both documented procedures were
re-run against a live instance, and a `-wal` left by a crash carries
data the `.db` alone does not.

Verified by reproducing the failure on unmodified `next` first — 6
targets, 60 events at 5/s, a concurrent `.dump` reader — which gave 38
HTTP 500s and 112 duplicate POSTs at the sinks across a restart. Both
arms of the matched pair now show 0 inbound 500s, 0 engine write
errors, and 0 new requests at the sinks after a restart, counted by
payload.

README.md

@@ -566,9 +566,18 @@ is both the simplest and the only complete rule:
 `events-3f2a1c9e-....db`. The only other file is `webhooker.lock`, the
 always-empty [single-instance lock](#single-instance-lock); it holds no
 state and is not part of the backup set — a copied one is stale and
-blocks nothing. No `-wal` or `-shm` files are produced (see below); a
+blocks nothing.
-transient `{name}.db-journal` may exist beside a database while a write
+
-is in flight and is not part of the backup set either.
+**`-wal` and `-shm` sidecars.** Every database runs in WAL journal mode,
+so while the service is running each `{name}.db` has a `{name}.db-wal`
+and a `{name}.db-shm` beside it. **`-wal` is part of the database, not a
+scratch file**: it holds committed transactions that are not yet in the
+`.db`, so a copy of the `.db` without its `-wal` is missing data and may
+have no readable schema at all. A clean shutdown checkpoints and removes
+both sidecars, so a stopped deployment has none; a killed or crashed one
+leaves them, and they must be carried with the `.db`. `-shm` is
+regenerable, but there is no reason to separate the two — copy the
+directory.
 
 Configuration is **not** in `DATA_DIR` — it comes from the environment
 and from a `.env` file read out of the process working directory. Back
@@ -576,17 +585,19 @@ that up with your deployment config, separately.
 
 ### A hot copy is not safe
 
-No `journal_mode` pragma is ever issued on any database webhooker opens,
+Every database webhooker opens runs in WAL journal mode. The main and
-so all of them run on SQLite's default rollback journal. There is no
+event databases are also held open for the entire process lifetime —
-WAL. The main and event databases are also held open for the entire
+`WebhookDBManager` caches event database handles and closes them only on
-process lifetime — `WebhookDBManager` caches event database handles and
+webhook deletion or shutdown — so "it looked idle" is not a guarantee
-closes them only on webhook deletion or shutdown — so "it looked idle"
+that nothing was mid-transaction.
-is not a guarantee that nothing was mid-transaction.
 
 That means `cp`, `rsync`, `tar` or a filesystem snapshot taken against a
-running instance can capture a database mid-transaction and yield a file
+running instance can capture a database and its `-wal` at two different
-that is corrupt or missing state the journal would have rolled back. Use
+instants and yield a file that is corrupt or missing state. Copying a
-one of the two procedures below instead.
+`.db` on its own is worse and fails loudly: recently written pages,
+including the schema itself on a young database, live in the `-wal`, so
+the copy reads back as an empty or table-less database. Use one of the
+two procedures below instead.
 
 **Stop, copy, start.** The simplest, needs no extra tooling, and the
 only one that gives a single point in time across every file:
@@ -605,8 +616,9 @@ for db in /path/to/data/*.db; do
 done
 ```
 
-`.backup` takes the proper locks and writes a consistent file. Two
+`.backup` reads through the WAL and writes a single consistent file with
-caveats. First, the runtime image is `alpine:3.21` with only
+no sidecars of its own, so the destination is complete as it stands.
+Two caveats. First, the runtime image is `alpine:3.21` with only
 `ca-certificates` added — the `sqlite3` CLI is **not** in it, so run
 this on the host against the volume path, or from a throwaway container
 that mounts the volume. Second, each file is captured at its own
@@ -614,12 +626,21 @@ instant, so a webhook created or an event delivered between two files
 being copied lands in one and not the other. If you need the whole set
 coherent as of a single moment, stop the service.
 
+Note that `sqlite3 <db> .dump` is **not** one of these procedures: it is
+an export, it holds a read transaction open for as long as it runs, and
+it pins the WAL against checkpointing for that whole time. It is safe to
+run — it does not block ingestion — but back up with `.backup` or a
+stopped copy.
+
 Archive databases are the one exception the service is built for: the
 archive writer closes its handle after each write (debounced to at most
 one reopen per second), so an operator can move `archive-{uuid}.db`
 away for offline retention while the service runs, and it is recreated
-on the next write (see
+on the next write. Closing the handle checkpoints and removes that
-[Database Architecture](#database-architecture)). That is a
+file's sidecars, so what is left to move is a single self-contained
+`.db` — but if a `-wal` is there, a write is in flight, and it has to
+move with it. See
+[Database Architecture](#database-architecture). That is a
 move-the-file-away workflow, not a substitute for the backup procedures
 above.
 
@@ -636,14 +657,18 @@ above.
    restored without `webhooker.db` are simply orphaned; nothing
    references their UUIDs.
 
-3. Do not carry `*.db-journal` files into the restore. Backups taken by
+3. Carry any `*.db-wal` and `*.db-shm` files that are in the backup.
-   either procedure above are self-consistent and do not need one.
+   They are part of the database, and dropping a `-wal` silently
+   discards every transaction it still holds. Backups taken by either
+   procedure above will not contain them — `.backup` writes a single
+   consolidated file, and a clean stop checkpoints the sidecars away —
+   but a copy salvaged from a crashed instance will, and it needs them.
 
 4. **Fix ownership.** The container runs as the non-root `webhooker`
    user, UID 1000 / GID 1000. Restored files must be owned by (or
    writable by) that UID, and so must the directory itself — SQLite
-   creates the rollback journal beside the database, so a writable file
+   creates the `-wal` and `-shm` sidecars beside the database, so a
-   inside a directory it cannot write is not enough:
+   writable file inside a directory it cannot write is not enough:
 
     ```bash
     chown -R 1000:1000 /path/to/data
@@ -1391,10 +1416,12 @@ This separation provides:
   only, or disables cleanup entirely when set to `0` (retain forever).
 - **Performance** — each webhook's database has its own page cache and
   its own lock, so concurrent event ingestion across webhooks won't
-  contend. No write-ahead log is involved: both DSNs are
+  contend. Every database — main, per-webhook, and archive — is opened
-  `file:{path}?cache=shared&mode=rwc` and no `journal_mode` pragma is
+  through one code path (`internal/database/sqlite_open.go`) in WAL
-  ever issued, so every database runs on SQLite's default rollback
+  journal mode, with a 10-second busy timeout, `BEGIN IMMEDIATE`
-  journal.
+  transactions, and a bounded connection pool. Under WAL a reader never
+  blocks a writer, so an operator reading a database does not stall
+  event ingestion into it.
 
 The **database target type** builds on this architecture to provide
 long-term archiving, separate from the per-webhook event database (which

internal/database/database.go

@@ -4,7 +4,6 @@ package database
 import (
 	"context"
 	"crypto/rand"
-	"database/sql"
 	"encoding/base64"
 	"errors"
 	"fmt"
@@ -16,7 +15,6 @@ import (
 	"go.uber.org/fx"
 	"gorm.io/driver/sqlite"
 	"gorm.io/gorm"
-	_ "modernc.org/sqlite" // Pure Go SQLite driver
 	"sneak.berlin/go/webhooker/internal/banner"
 	"sneak.berlin/go/webhooker/internal/config"
 	"sneak.berlin/go/webhooker/internal/gormlog"
@@ -198,13 +196,11 @@ func (d *Database) connectTo(dataDir string) error {
 
 	// Construct the main application database path inside DATA_DIR.
 	dbPath := filepath.Join(dataDir, MainDBFileName)
-	dbURL := fmt.Sprintf(
-		"file:%s?cache=shared&mode=rwc",
-		dbPath,
-	)
 
-	// Open the database with the pure Go SQLite driver
+	// Opened through OpenSQLite so this handle carries the same WAL
-	sqlDB, err := sql.Open("sqlite", dbURL)
+	// journaling, busy timeout, immediate-transaction locking, and pool
+	// bounds as every other database file. See sqlite_open.go.
+	sqlDB, err := OpenSQLite(dbPath, SQLiteModeCreate)
 	if err != nil {
 		d.log.Error(
 			"failed to open database",

internal/database/sqlite_open.go

@@ -0,0 +1,147 @@
+package database
+
+import (
+	"database/sql"
+	"fmt"
+	"net/url"
+	"time"
+
+	_ "modernc.org/sqlite" // Pure Go SQLite driver
+)
+
+// Every SQLite file this service opens — the main database, the
+// per-webhook event databases, and the archive databases — is opened
+// through OpenSQLite, so the durability settings below are properties
+// of the service rather than of one call site.
+//
+// modernc.org/sqlite installs no busy handler and issues no pragmas of
+// its own: it executes only the pragmas named in explicit `_pragma=`
+// DSN parameters, and gorm.io/driver/sqlite adds none when it is
+// handed an existing *sql.DB. Every setting therefore has to be
+// spelled out here or it is simply not in effect.
+// SQLite URI open modes.
+const (
+	// SQLiteModeCreate creates the database file when it is missing.
+	SQLiteModeCreate = "rwc"
+
+	// SQLiteModeExisting requires the file to exist already.
+	SQLiteModeExisting = "rw"
+)
+
+const (
+	// SQLiteBusyTimeout is how long SQLite retries a lock conflict
+	// before returning SQLITE_BUSY.
+	//
+	// Under WAL a reader never blocks a writer, so the only conflict
+	// left is writer against writer: this process's delivery workers
+	// against each other, or against another process holding the write
+	// lock. Those clear in milliseconds. Ten seconds is far above that
+	// and still well inside the receiver's request budget, so an
+	// inbound webhook waits rather than being rejected with a 500.
+	SQLiteBusyTimeout = 10 * time.Second
+
+	// sqliteMaxOpenConns bounds the connection pool for one database
+	// file.
+	//
+	// The pool needs a bound at all because database/sql cannot detect
+	// a connection left mid-transaction: modernc.org/sqlite implements
+	// neither driver.Validator nor driver.SessionResetter, so a
+	// connection whose COMMIT failed is returned to the pool with its
+	// transaction still open and handed out again indefinitely. That is
+	// what turned four `database is locked` errors into 593
+	// `cannot start a transaction within a transaction` in
+	// https://git.eeqj.de/sneak/webhooker/issues/256.
+	//
+	// Four is above the one writer SQLite allows at a time, so reads
+	// still proceed while a write is in flight, and low enough that
+	// contention is resolved by the busy handler rather than by piling
+	// up connections against a lock only one of them can hold.
+	sqliteMaxOpenConns = 4
+
+	// sqliteMaxIdleConns keeps the pool warm without holding every
+	// connection open through an idle period.
+	sqliteMaxIdleConns = 2
+
+	// sqliteConnMaxLifetime and sqliteConnMaxIdleTime retire pooled
+	// connections on a schedule. With _txlock=immediate a failed
+	// COMMIT should no longer be reachable, but these bound the damage
+	// if one happens anyway: a poisoned connection is closed and
+	// replaced within the lifetime instead of wedging the file until
+	// the process restarts.
+	sqliteConnMaxLifetime = 5 * time.Minute
+	sqliteConnMaxIdleTime = time.Minute
+)
+
+// SQLiteDSN builds the connection string for one database file.
+//
+// mode is the SQLite URI open mode: "rwc" to create the file when it
+// is missing, "rw" to require that it already exists.
+//
+// Three settings carry the fix for
+// https://git.eeqj.de/sneak/webhooker/issues/256 and none of them is
+// optional:
+//
+//   - journal_mode=WAL, so a reader — an operator running
+//     `sqlite3 <db> .dump` over their own data — takes a snapshot
+//     instead of blocking every writer behind it.
+//
+//   - busy_timeout, so a writer that does meet a lock waits for it.
+//     Without one SQLite gives up immediately; nothing above it
+//     retries.
+//
+//   - _txlock=immediate, so every transaction takes the write lock at
+//     BEGIN. A deferred transaction acquires it lazily on its first
+//     write, and that upgrade returns SQLITE_BUSY *without* consulting
+//     the busy handler, because SQLite cannot block a transaction that
+//     may already hold a read snapshot. Such a COMMIT then fails while
+//     the transaction stays open on the connection. A busy timeout
+//     alone does not prevent this; BEGIN IMMEDIATE does, by putting
+//     the wait somewhere the handler applies.
+//
+// Note what is absent: `cache=shared`. Under a shared cache an
+// in-process conflict is reported as SQLITE_LOCKED rather than
+// SQLITE_BUSY, and the busy handler does not retry SQLITE_LOCKED — so
+// leaving it in would have defeated the busy timeout for exactly the
+// contention this service generates. Dropping it is part of the fix,
+// not housekeeping.
+//
+// synchronous is deliberately left at SQLite's default of FULL: this
+// is a webhook receiver whose one promise is that an event it answered
+// 200 for is durable.
+func SQLiteDSN(path, mode string) string {
+	q := url.Values{}
+	q.Set("mode", mode)
+	q.Set("_txlock", "immediate")
+	q.Add("_pragma", "journal_mode(WAL)")
+	q.Add(
+		"_pragma",
+		fmt.Sprintf(
+			"busy_timeout(%d)",
+			SQLiteBusyTimeout.Milliseconds(),
+		),
+	)
+
+	return "file:" + path + "?" + q.Encode()
+}
+
+// OpenSQLite opens the SQLite file at path with the service's
+// durability settings and pool bounds applied. mode is the SQLite URI
+// open mode ("rwc" or "rw").
+//
+// The handle is returned rather than a *gorm.DB because the callers
+// wrap it in gorm themselves with their own logger.
+func OpenSQLite(path, mode string) (*sql.DB, error) {
+	sqlDB, err := sql.Open("sqlite", SQLiteDSN(path, mode))
+	if err != nil {
+		return nil, fmt.Errorf(
+			"opening sqlite database %s: %w", path, err,
+		)
+	}
+
+	sqlDB.SetMaxOpenConns(sqliteMaxOpenConns)
+	sqlDB.SetMaxIdleConns(sqliteMaxIdleConns)
+	sqlDB.SetConnMaxLifetime(sqliteConnMaxLifetime)
+	sqlDB.SetConnMaxIdleTime(sqliteConnMaxIdleTime)
+
+	return sqlDB, nil
+}

internal/database/sqlite_open_test.go

@@ -0,0 +1,167 @@
+package database_test
+
+import (
+	"context"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"github.com/google/uuid"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"gorm.io/gorm"
+	"sneak.berlin/go/webhooker/internal/database"
+)
+
+// livePragma reads a pragma off a live handle. Reading the DSN back
+// would prove only that the string was built; these tests assert that
+// SQLite actually applied it.
+func livePragma(t *testing.T, db *gorm.DB, name string) string {
+	t.Helper()
+
+	var v string
+
+	row := db.Raw("pragma " + name).Row()
+	require.NoError(t, row.Scan(&v))
+
+	return v
+}
+
+func TestSQLiteDSNCarriesTheDurabilitySettings(t *testing.T) {
+	t.Parallel()
+
+	dsn := database.SQLiteDSN(
+		"/var/lib/webhooker/webhooker.db",
+		database.SQLiteModeCreate,
+	)
+
+	assert.Contains(t, dsn, "journal_mode%28WAL%29")
+	assert.Contains(t, dsn, "busy_timeout%2810000%29")
+	assert.Contains(t, dsn, "_txlock=immediate")
+	assert.Contains(t, dsn, "mode=rwc")
+
+	// cache=shared turns an in-process conflict into SQLITE_LOCKED,
+	// which the busy handler does not retry. It must never come back.
+	// See https://git.eeqj.de/sneak/webhooker/issues/256.
+	assert.NotContains(t, strings.ToLower(dsn), "cache=shared")
+}
+
+// TestPerWebhookDBAppliesPragmasOnALiveHandle is the check the issue
+// asks for by name: the settings are confirmed by querying the running
+// database, not by inspecting the connection string.
+func TestPerWebhookDBAppliesPragmasOnALiveHandle(t *testing.T) {
+	t.Parallel()
+
+	mgr, lc := setupTestWebhookDBManager(t)
+	ctx := context.Background()
+	require.NoError(t, lc.Start(ctx))
+
+	defer func() { require.NoError(t, lc.Stop(ctx)) }()
+
+	webhookID := uuid.New().String()
+
+	db, err := mgr.GetDB(webhookID)
+	require.NoError(t, err)
+
+	assert.Equal(
+		t, "wal",
+		strings.ToLower(livePragma(t, db, "journal_mode")),
+	)
+	assert.Equal(
+		t, "10000", livePragma(t, db, "busy_timeout"),
+	)
+}
+
+func TestMainDBAppliesPragmasOnALiveHandle(t *testing.T) {
+	t.Parallel()
+
+	ctx := context.Background()
+	dir := t.TempDir()
+
+	sqlDB, err := database.OpenSQLite(
+		filepath.Join(dir, database.MainDBFileName),
+		database.SQLiteModeCreate,
+	)
+	require.NoError(t, err)
+
+	defer func() { require.NoError(t, sqlDB.Close()) }()
+
+	var journal string
+
+	require.NoError(t, sqlDB.
+		QueryRowContext(ctx, "pragma journal_mode").
+		Scan(&journal))
+	assert.Equal(t, "wal", strings.ToLower(journal))
+
+	var busy string
+
+	require.NoError(t, sqlDB.
+		QueryRowContext(ctx, "pragma busy_timeout").
+		Scan(&busy))
+	assert.Equal(t, "10000", busy)
+}
+
+// TestConcurrentReaderDoesNotBlockWrites is the unit-scale form of the
+// reproduction in
+// https://git.eeqj.de/sneak/webhooker/issues/256: an operator's
+// long-held read of their own data used to make every concurrent write
+// fail. Under WAL the reader takes a snapshot and the writes proceed.
+func TestConcurrentReaderDoesNotBlockWrites(t *testing.T) {
+	t.Parallel()
+
+	mgr, lc := setupTestWebhookDBManager(t)
+	ctx := context.Background()
+	require.NoError(t, lc.Start(ctx))
+
+	defer func() { require.NoError(t, lc.Stop(ctx)) }()
+
+	webhookID := uuid.New().String()
+
+	db, err := mgr.GetDB(webhookID)
+	require.NoError(t, err)
+
+	// A second handle on the same file, holding a read transaction
+	// open across every write below — what `sqlite3 <db> .dump` is.
+	readerSQL, err := database.OpenSQLite(
+		mgr.DBPath(webhookID), database.SQLiteModeExisting,
+	)
+	require.NoError(t, err)
+
+	defer func() { require.NoError(t, readerSQL.Close()) }()
+
+	readerConn, err := readerSQL.Conn(ctx)
+	require.NoError(t, err)
+
+	defer func() { require.NoError(t, readerConn.Close()) }()
+
+	_, err = readerConn.ExecContext(ctx, "begin deferred")
+	require.NoError(t, err)
+
+	_, err = readerConn.ExecContext(
+		ctx, "select count(*) from events",
+	)
+	require.NoError(t, err)
+
+	for range 25 {
+		err = db.Transaction(func(tx *gorm.DB) error {
+			return tx.Create(&database.Event{
+				WebhookID:    webhookID,
+				EntrypointID: uuid.New().String(),
+				Method:       "POST",
+				Body:         "{}",
+			}).Error
+		})
+		require.NoError(t, err)
+	}
+
+	_, err = readerConn.ExecContext(ctx, "commit")
+	require.NoError(t, err)
+
+	var count int64
+
+	require.NoError(
+		t,
+		db.Model(&database.Event{}).Count(&count).Error,
+	)
+	assert.Equal(t, int64(25), count)
+}

internal/database/webhook_db_manager.go

@@ -2,7 +2,6 @@ package database
 
 import (
 	"context"
-	"database/sql"
 	"errors"
 	"fmt"
 	"log/slog"
@@ -234,12 +233,11 @@ func (m *WebhookDBManager) openDB(
 	webhookID string,
 ) (*gorm.DB, error) {
 	path := m.dbPath(webhookID)
-	dbURL := fmt.Sprintf(
-		"file:%s?cache=shared&mode=rwc",
-		path,
-	)
 
-	sqlDB, err := sql.Open("sqlite", dbURL)
+	// See sqlite_open.go: WAL, a busy timeout, immediate-transaction
+	// locking, and a bounded pool, all of which this file needs most —
+	// it is the one every delivery worker writes to concurrently.
+	sqlDB, err := OpenSQLite(path, SQLiteModeCreate)
 	if err != nil {
 		return nil, fmt.Errorf(
 			"opening webhook database %s: %w",

internal/delivery/engine.go

@@ -41,6 +41,26 @@ const (
 	// sweep runs.
 	retrySweepInterval = 60 * time.Second
 
+	// pendingSweepMinAge is how long a delivery must have sat at
+	// pending before the sweep treats it as stranded rather than as
+	// in flight.
+	//
+	// A delivery is pending from the moment it is created until its
+	// outcome is written, which includes the whole time a worker
+	// spends on it, so the bound has to clear the longest a live
+	// attempt can take: httpClientTimeout plus queueing behind the
+	// other deliveries in front of it. Five minutes is far above
+	// that, and still recovers a stranded delivery in minutes rather
+	// than at the next restart.
+	pendingSweepMinAge = 5 * time.Minute
+
+	// pendingSweepBatch bounds how many stranded pending deliveries
+	// one sweep of one webhook re-dispatches. The sweep runs every
+	// retrySweepInterval, so a larger backlog drains across
+	// successive sweeps instead of arriving as one burst against a
+	// database that was already struggling to accept writes.
+	pendingSweepBatch = 500
+
 	// MaxInlineBodySize is the maximum event body size that
 	// will be carried inline in a Task through the channel.
 	// Bodies at or above this size are left nil and fetched
@@ -634,11 +654,120 @@ func (e *Engine) recoverPendingDeliveries(
 		"count", len(deliveries),
 	)
 
+	e.recoverPendingBatch(
+		ctx, webhookDB, webhookID, deliveries,
+	)
+}
+
+// recoverPendingBatch settles every delivery in the batch that was
+// already delivered, and re-dispatches only the rest. Both the
+// restart-time recovery and the periodic sweep go through it, so a
+// pending delivery is treated the same however it was found.
+func (e *Engine) recoverPendingBatch(
+	ctx context.Context,
+	webhookDB *gorm.DB,
+	webhookID string,
+	deliveries []database.Delivery,
+) {
 	targetMap := e.loadTargetMap(deliveries)
 
-	e.sendRecoveredDeliveries(
+	settled := e.reconcileDelivered(
-		ctx, deliveries, webhookID, targetMap,
+		webhookDB, webhookID, deliveries, targetMap,
 	)
+
+	e.sendRecoveredDeliveries(
+		ctx, webhookDB, deliveries, webhookID,
+		targetMap, settled,
+	)
+}
+
+// reconcileDelivered finds the deliveries in a pending batch that
+// already have a successful DeliveryResult, marks them delivered, and
+// returns their ids so the caller does not send them a second time.
+//
+// This is the state the engine previously had no way to represent. A
+// delivery is left pending by a failed bookkeeping write, and that
+// covers two different histories: nothing was ever sent, or the send
+// reached the receiver and only the status write failed. Re-sending
+// was the sole option, so every stranded row produced a duplicate at
+// the receiver and an event log that recorded one attempt for two
+// POSTs. A successful result row distinguishes them: it is written
+// before the status, so its presence means the wire I/O happened and
+// was recorded, and all that is missing is the status.
+//
+// Deliveries whose result row itself never landed are not in the
+// returned set and are re-sent, recorded as the further attempt they
+// are. That is honest at-least-once delivery rather than a silent
+// duplicate.
+func (e *Engine) reconcileDelivered(
+	webhookDB *gorm.DB,
+	webhookID string,
+	deliveries []database.Delivery,
+	targetMap map[string]database.Target,
+) map[string]struct{} {
+	settled := make(map[string]struct{})
+
+	if len(deliveries) == 0 {
+		return settled
+	}
+
+	ids := make([]string, 0, len(deliveries))
+	for i := range deliveries {
+		ids = append(ids, deliveries[i].ID)
+	}
+
+	var deliveredIDs []string
+
+	err := webhookDB.
+		Model(&database.DeliveryResult{}).
+		Where(
+			"delivery_id IN ? AND success = ?", ids, true,
+		).
+		Distinct().
+		Pluck("delivery_id", &deliveredIDs).Error
+	if err != nil {
+		// Every delivery stays out of the settled set, so the batch
+		// is re-sent exactly as it was before this check existed.
+		// That is the safe direction: a duplicate delivery beats
+		// declaring a delivery successful on a query that failed.
+		e.log.Error(
+			"failed to query successful delivery results; "+
+				"pending deliveries will be re-sent",
+			"webhook_id", webhookID,
+			"error", err,
+		)
+
+		return settled
+	}
+
+	for _, id := range deliveredIDs {
+		settled[id] = struct{}{}
+	}
+
+	if len(settled) == 0 {
+		return settled
+	}
+
+	e.log.Info(
+		"settling pending deliveries that already succeeded",
+		"webhook_id", webhookID,
+		"count", len(settled),
+	)
+
+	for i := range deliveries {
+		if _, ok := settled[deliveries[i].ID]; !ok {
+			continue
+		}
+
+		e.settleStatus(
+			webhookDB,
+			&deliveries[i],
+			targetMap[deliveries[i].TargetID].Type,
+			database.DeliveryStatusDelivered,
+		)
+	}
+
+	return settled
 }
 
 func (e *Engine) retrySweep(ctx context.Context) {
@@ -733,6 +862,65 @@ func (e *Engine) sweepWebhookRetries(
 			webhookDB, webhookID, &retrying[i],
 		)
 	}
+
+	e.sweepWebhookPending(ctx, webhookDB, webhookID)
+}
+
+// sweepWebhookPending recovers deliveries stranded at pending.
+//
+// A delivery is created pending and leaves that state only when its
+// outcome is written, so a pending row older than the age bound is one
+// whose bookkeeping write failed — the state that used to sit there
+// until a restart, and then produce a duplicate at the receiver. The
+// sweep gives it the same reconcile-then-dispatch treatment restart
+// recovery gets, so it costs a minute rather than an operator
+// noticing.
+//
+// The age bound is what keeps the sweep off deliveries the workers
+// still hold: a delivery in flight is pending too, and re-dispatching
+// one would race the worker that owns it. It is measured on updated_at
+// rather than created_at because claimPending stamps that column when
+// a delivery is handed out, which is what stops the next sweep, a
+// minute later, from sending the same delivery again while the first
+// attempt is still running.
+func (e *Engine) sweepWebhookPending(
+	ctx context.Context,
+	webhookDB *gorm.DB,
+	webhookID string,
+) {
+	var pending []database.Delivery
+
+	err := webhookDB.
+		Where(
+			"status = ? AND updated_at < ?",
+			database.DeliveryStatusPending,
+			time.Now().Add(-pendingSweepMinAge),
+		).
+		Preload("Event").
+		Limit(pendingSweepBatch).
+		Find(&pending).Error
+	if err != nil {
+		e.log.Error(
+			"retry sweep: "+
+				"failed to query pending deliveries",
+			"webhook_id", webhookID,
+			"error", err,
+		)
+
+		return
+	}
+
+	if len(pending) == 0 {
+		return
+	}
+
+	e.log.Info(
+		"retry sweep: recovering stranded pending deliveries",
+		"webhook_id", webhookID,
+		"count", len(pending),
+	)
+
+	e.recoverPendingBatch(ctx, webhookDB, webhookID, pending)
 }
 
 // sweepSingleRetry re-enqueues an orphaned retrying delivery
@@ -839,7 +1027,7 @@ func (e *Engine) failUnretryableRetry(
 		target.Type,
 	)
 
-	e.recordResult(
+	err := e.recordResult(
 		webhookDB,
 		d,
 		e.countAttempts(webhookDB, d.ID)+1,
@@ -849,6 +1037,11 @@ func (e *Engine) failUnretryableRetry(
 		reason,
 		0,
 	)
+	if err != nil {
+		e.bookkeepingFailed(d, err)
+
+		return
+	}
 
 	// The type is passed rather than assigned onto d: the delivery
 	// is loaded here without its target relation, and populating
@@ -856,7 +1049,7 @@ func (e *Engine) failUnretryableRetry(
 	// whole target row — plaintext config, which for a slack target
 	// is the credential — into the per-webhook event database. See
 	// https://git.eeqj.de/sneak/webhooker/issues/206.
-	e.updateDeliveryStatus(
+	e.settleStatus(
 		webhookDB, d, target.Type,
 		database.DeliveryStatusFailed,
 	)
@@ -878,7 +1071,7 @@ func (e *Engine) processDelivery(
 			"type", d.Target.Type,
 		)
 
-		e.updateDeliveryStatus(
+		e.settleStatus(
 			webhookDB, d, d.Target.Type,
 			database.DeliveryStatusFailed,
 		)
@@ -910,6 +1103,14 @@ func (e *Engine) observeAttempt(
 // recordResult persists a DeliveryResult row describing a
 // single attempt. It is a cross-target helper the targets
 // call.
+//
+// It returns its error rather than swallowing it. A DeliveryResult
+// row is the only record that an attempt happened at all, so a
+// caller that ignored a failed write would go on to mark the
+// delivery delivered — leaving the event log claiming one attempt
+// for a receiver that got two. Every caller must instead stop
+// advancing the delivery's status and let it stay in the
+// non-terminal state it already holds; see bookkeepingFailed.
 func (e *Engine) recordResult(
 	webhookDB *gorm.DB,
 	d *database.Delivery,
@@ -918,7 +1119,7 @@ func (e *Engine) recordResult(
 	statusCode int,
 	respBody, errMsg string,
 	durationMs int64,
-) {
+) error {
 	result := &database.DeliveryResult{
 		DeliveryID:   d.ID,
 		AttemptNum:   attemptNum,
@@ -931,12 +1132,44 @@ func (e *Engine) recordResult(
 
 	err := webhookDB.Create(result).Error
 	if err != nil {
-		e.log.Error(
+		return fmt.Errorf(
-			"failed to record delivery result",
+			"recording delivery result for %s: %w", d.ID, err,
-			"delivery_id", d.ID,
-			"error", err,
 		)
 	}
+
+	return nil
+}
+
+// bookkeepingFailed reports that a delivery's own record of what
+// happened could not be written, and deliberately writes nothing in
+// response.
+//
+// Leaving the row alone is the whole point. A delivery is created
+// pending and only ever leaves that state through
+// updateDeliveryStatus, so a delivery whose bookkeeping write failed
+// is still pending or retrying — the two non-terminal states, per
+// DeliveryStatus.Terminal — and both are swept and recovered. Writing
+// anything here would need the very database that just refused a
+// write, and would be one more thing to fail; not writing cannot.
+//
+// The cost is honest at-least-once behaviour: a send that reached the
+// receiver but whose result row did not land is attempted again, and
+// recorded as the further attempt it is. What no longer happens is the
+// silent duplicate — a second POST the event log denies ever
+// occurred. Where the result row *did* land and only the status write
+// failed, reconcileDelivered settles the row without re-sending.
+func (e *Engine) bookkeepingFailed(
+	d *database.Delivery, err error,
+) {
+	e.log.Error(
+		"delivery bookkeeping write failed; leaving delivery "+
+			"in a recoverable state",
+		"delivery_id", d.ID,
+		"event_id", d.EventID,
+		"target_id", d.TargetID,
+		"status", d.Status,
+		"error", err,
+	)
 }
 
 // updateDeliveryStatus persists a new status for a delivery.
@@ -952,26 +1185,45 @@ func (e *Engine) recordResult(
 //
 // The counter moves only after the row is written, so a transition
 // the database rejected is not claimed as an outcome that happened.
+// For the same reason the error is returned rather than logged and
+// dropped: a delivery whose status write failed has not reached that
+// status, and its caller must not act as though it had.
 func (e *Engine) updateDeliveryStatus(
 	webhookDB *gorm.DB,
 	d *database.Delivery,
 	targetType database.TargetType,
 	status database.DeliveryStatus,
-) {
+) error {
 	err := webhookDB.Model(d).
 		Update("status", status).Error
 	if err != nil {
-		e.log.Error(
+		return fmt.Errorf(
-			"failed to update delivery status",
+			"updating delivery %s to status %s: %w",
-			"delivery_id", d.ID,
+			d.ID, status, err,
-			"status", status,
-			"error", err,
 		)
-
-		return
 	}
 
 	e.mtr.DeliveryStatusChanged(targetType, status)
+
+	return nil
+}
+
+// settleStatus moves a delivery to its outcome status and reports a
+// failed write through bookkeepingFailed, which leaves the row
+// recoverable. It exists so the target call sites read as one
+// statement rather than four lines of identical error handling.
+func (e *Engine) settleStatus(
+	webhookDB *gorm.DB,
+	d *database.Delivery,
+	targetType database.TargetType,
+	status database.DeliveryStatus,
+) {
+	err := e.updateDeliveryStatus(
+		webhookDB, d, targetType, status,
+	)
+	if err != nil {
+		e.bookkeepingFailed(d, err)
+	}
 }
 
 func truncate(s string, maxLen int) string {
@@ -1065,6 +1317,95 @@ func (e *Engine) countAttempts(
 	return int(resultCount)
 }
 
+// claimPending takes ownership of a pending delivery before it is
+// re-dispatched, and reports whether the claim succeeded.
+//
+// The claim is a compare-and-set on the status: it takes effect only
+// while the delivery is still pending, so a worker that settled the
+// delivery between the query and here wins and nothing is re-sent.
+// Stamping updated_at is the claim itself — the sweep selects on that
+// column, so a delivery handed out now is out of the sweep's reach for
+// a further pendingSweepMinAge, rather than being sent again on every
+// sweep for as long as the attempt takes.
+//
+// A claim that cannot be written means the database is refusing
+// writes, which is the condition that stranded this delivery in the
+// first place. Not sending is then the right answer: the attempt
+// could not be recorded either, and an unrecordable send is exactly
+// the duplicate this issue is about.
+func (e *Engine) claimPending(
+	webhookDB *gorm.DB, d *database.Delivery,
+) bool {
+	res := webhookDB.
+		Model(&database.Delivery{}).
+		Where(
+			"id = ? AND status = ?",
+			d.ID, database.DeliveryStatusPending,
+		).
+		UpdateColumn("updated_at", time.Now())
+	if res.Error != nil {
+		e.log.Error(
+			"failed to claim pending delivery for recovery; "+
+				"leaving it for the next sweep",
+			"delivery_id", d.ID,
+			"error", res.Error,
+		)
+
+		return false
+	}
+
+	return res.RowsAffected == 1
+}
+
+// countAttemptsBatch counts the recorded attempts of every delivery
+// in a batch with one grouped query, keyed by delivery id. Deliveries
+// with no attempts are simply absent from the result, which reads back
+// as the zero this caller wants.
+//
+// One query rather than one per delivery: this runs on the recovery
+// path, which is a burst of writes against a database that has just
+// been under enough contention to strand these rows in the first
+// place. See https://git.eeqj.de/sneak/webhooker/issues/256.
+func (e *Engine) countAttemptsBatch(
+	webhookDB *gorm.DB, deliveries []database.Delivery,
+) map[string]int {
+	counts := make(map[string]int, len(deliveries))
+
+	if len(deliveries) == 0 {
+		return counts
+	}
+
+	ids := make([]string, 0, len(deliveries))
+	for i := range deliveries {
+		ids = append(ids, deliveries[i].ID)
+	}
+
+	// One delivery id per recorded attempt, tallied here rather than
+	// grouped in SQL: internal/gormlog forbids (*gorm.DB).Scan, which
+	// a GROUP BY into a struct would need, and an attempt row per
+	// delivery is bounded by the target's MaxRetries.
+	var attemptIDs []string
+
+	err := webhookDB.
+		Model(&database.DeliveryResult{}).
+		Where("delivery_id IN ?", ids).
+		Pluck("delivery_id", &attemptIDs).Error
+	if err != nil {
+		e.log.Error(
+			"failed to count delivery attempts for recovery",
+			"error", err,
+		)
+
+		return counts
+	}
+
+	for _, id := range attemptIDs {
+		counts[id]++
+	}
+
+	return counts
+}
+
 func (e *Engine) loadEvent(
 	webhookDB *gorm.DB, eventID string,
 ) (database.Event, error) {
@@ -1168,12 +1509,24 @@ func (e *Engine) loadTargetMap(
 	return targetMap
 }
 
+// sendRecoveredDeliveries re-dispatches pending deliveries, skipping
+// the ids in settled — those already reached their receiver and have
+// been marked delivered by reconcileDelivered.
 func (e *Engine) sendRecoveredDeliveries(
 	ctx context.Context,
+	webhookDB *gorm.DB,
 	deliveries []database.Delivery,
 	webhookID string,
 	targetMap map[string]database.Target,
+	settled map[string]struct{},
 ) {
+	// The attempt number continues each delivery's own history
+	// rather than restarting at 1. A recovered delivery may already
+	// have recorded attempts, and numbering the next one 1 again
+	// both collides in the event log and hands the retry path a
+	// backoff computed from the wrong attempt.
+	attempts := e.countAttemptsBatch(webhookDB, deliveries)
+
 	for i := range deliveries {
 		select {
 		case <-ctx.Done():
@@ -1181,6 +1534,10 @@ func (e *Engine) sendRecoveredDeliveries(
 		default:
 		}
 
+		if _, ok := settled[deliveries[i].ID]; ok {
+			continue
+		}
+
 		target, ok := targetMap[deliveries[i].TargetID]
 		if !ok {
 			e.log.Error(
@@ -1192,9 +1549,14 @@ func (e *Engine) sendRecoveredDeliveries(
 			continue
 		}
 
+		if !e.claimPending(webhookDB, &deliveries[i]) {
+			continue
+		}
+
 		task := buildRecoveryTask(
 			&deliveries[i], webhookID,
-			&deliveries[i].Event, &target, 1,
+			&deliveries[i].Event, &target,
+			attempts[deliveries[i].ID]+1,
 		)
 
 		select {

internal/delivery/engine_integration_test.go

@@ -2,7 +2,6 @@ package delivery_test
 
 import (
 	"context"
-	"database/sql"
 	"encoding/json"
 	"fmt"
 	"io"
@@ -70,11 +69,12 @@ func iMainDB(t *testing.T) *gorm.DB {
 		t.TempDir(), "main-test.db",
 	)
 
-	dsn := fmt.Sprintf(
+	// Opened the way the service opens the main database, so these
-		"file:%s?cache=shared&mode=rwc", dbPath,
+	// tests cannot pass against journal and locking settings
+	// production does not use.
+	sqlDB, err := database.OpenSQLite(
+		dbPath, database.SQLiteModeCreate,
 	)
-
-	sqlDB, err := sql.Open("sqlite", dsn)
 	require.NoError(t, err)
 
 	t.Cleanup(func() { _ = sqlDB.Close() })

internal/delivery/engine_test.go

@@ -3,7 +3,6 @@ package delivery_test
 import (
 	"bytes"
 	"context"
-	"database/sql"
 	"encoding/json"
 	"fmt"
 	"log/slog"
@@ -37,11 +36,12 @@ func testWebhookDB(t *testing.T) *gorm.DB {
 		t.TempDir(), "events-test.db",
 	)
 
-	dsn := fmt.Sprintf(
+	// Opened the way the service opens a per-webhook database, so
-		"file:%s?cache=shared&mode=rwc", dbPath,
+	// these tests cannot pass against journal and locking settings
+	// production does not use.
+	sqlDB, err := database.OpenSQLite(
+		dbPath, database.SQLiteModeCreate,
 	)
-
-	sqlDB, err := sql.Open("sqlite", dsn)
 	require.NoError(t, err)
 
 	t.Cleanup(func() { _ = sqlDB.Close() })

internal/delivery/export_test.go

@@ -33,6 +33,11 @@ const (
 	// response is written against this number, so a test has to
 	// be able to name it.
 	ExportMaxBodyLog = maxBodyLog
+
+	// ExportPendingSweepMinAge is how long a delivery must sit at
+	// pending before the sweep treats it as stranded. A test has to
+	// name it to age a row past the bound.
+	ExportPendingSweepMinAge = pendingSweepMinAge
 )
 
 // ExportIsBlockedIP exposes isBlockedIP for testing.

internal/delivery/queue_depth_gormlog_test.go

@@ -3,8 +3,6 @@ package delivery_test
 import (
 	"bytes"
 	"context"
-	"database/sql"
-	"fmt"
 	"log/slog"
 	"net/http"
 	"path/filepath"
@@ -54,12 +52,10 @@ func (q *qdSyncBuf) String() string {
 func qdMainDB(t *testing.T, log *slog.Logger) *gorm.DB {
 	t.Helper()
 
-	dsn := fmt.Sprintf(
+	sqlDB, err := database.OpenSQLite(
-		"file:%s?cache=shared&mode=rwc",
 		filepath.Join(t.TempDir(), "main-gormlog.db"),
+		database.SQLiteModeCreate,
 	)
-
-	sqlDB, err := sql.Open("sqlite", dsn)
 	require.NoError(t, err)
 
 	t.Cleanup(func() { _ = sqlDB.Close() })

internal/delivery/recovery_durability_test.go

@@ -0,0 +1,411 @@
+package delivery_test
+
+import (
+	"context"
+	"net/http"
+	"net/http/httptest"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"github.com/google/uuid"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"gorm.io/gorm"
+	"sneak.berlin/go/webhooker/internal/database"
+	"sneak.berlin/go/webhooker/internal/delivery"
+)
+
+// These tests cover the delivery half of
+// https://git.eeqj.de/sneak/webhooker/issues/256: a delivery that
+// reached its receiver but whose bookkeeping write failed used to be
+// left at pending and re-sent on the next restart, giving the receiver
+// a second copy while the event log recorded one attempt.
+
+// rSeedResult records a DeliveryResult against a delivery, standing in
+// for the attempt row the send path writes before the status.
+func rSeedResult(
+	t *testing.T,
+	db *gorm.DB,
+	deliveryID string,
+	attemptNum int,
+	success bool,
+) {
+	t.Helper()
+
+	require.NoError(t, db.Create(&database.DeliveryResult{
+		DeliveryID: deliveryID,
+		AttemptNum: attemptNum,
+		Success:    success,
+	}).Error)
+}
+
+// rAgePending backdates a delivery past the sweep's age bound, which is
+// what separates a stranded delivery from one a worker still holds.
+func rAgePending(
+	t *testing.T, db *gorm.DB, deliveryID string,
+) {
+	t.Helper()
+
+	old := time.Now().Add(
+		-2 * delivery.ExportPendingSweepMinAge,
+	)
+
+	require.NoError(t, db.Model(&database.Delivery{}).
+		Where("id = ?", deliveryID).
+		UpdateColumn("updated_at", old).Error)
+}
+
+func TestRecoverySkipsPendingWithSuccessfulResult(
+	t *testing.T,
+) {
+	t.Parallel()
+
+	s := newISetup(t)
+	targetID := uuid.New().String()
+
+	iCreateTarget(t, s.MainDB, targetID,
+		s.WebhookID, "already-delivered",
+		database.TargetTypeLog, "", 0,
+	)
+
+	event := iSeedEvent(
+		t, s.WebhookDB, s.WebhookID, `{"delivered":true}`,
+	)
+
+	// The delivery whose send succeeded and whose result row landed:
+	// only the status write failed, so it sits at pending.
+	done := iSeedDelivery(
+		t, s.WebhookDB, event.ID, targetID,
+		database.DeliveryStatusPending,
+	)
+	rSeedResult(t, s.WebhookDB, done.ID, 1, true)
+
+	// A delivery that was genuinely never attempted.
+	fresh := iSeedDelivery(
+		t, s.WebhookDB, event.ID, targetID,
+		database.DeliveryStatusPending,
+	)
+
+	s.Engine.ExportRecoverPendingDeliveries(
+		context.Background(), s.WebhookDB, s.WebhookID,
+	)
+
+	select {
+	case task := <-s.Engine.ExportDeliveryCh():
+		assert.Equal(
+			t, fresh.ID, task.DeliveryID,
+			"only the unattempted delivery may be re-sent",
+		)
+	case <-time.After(2 * time.Second):
+		t.Fatal("expected the unattempted delivery")
+	}
+
+	select {
+	case task := <-s.Engine.ExportDeliveryCh():
+		t.Fatalf(
+			"re-sent an already delivered delivery: %s",
+			task.DeliveryID,
+		)
+	case <-time.After(200 * time.Millisecond):
+	}
+
+	// It is settled rather than merely skipped: leaving it pending
+	// would strand it again on the next sweep.
+	iAssertStatus(
+		t, s.WebhookDB, done.ID,
+		database.DeliveryStatusDelivered,
+	)
+}
+
+// TestRecoveryContinuesTheAttemptNumbering pins the audit trail: a
+// recovered delivery that already recorded two attempts is re-sent as
+// attempt three, not as attempt one again.
+func TestRecoveryContinuesTheAttemptNumbering(t *testing.T) {
+	t.Parallel()
+
+	s := newISetup(t)
+	targetID := uuid.New().String()
+
+	iCreateTarget(t, s.MainDB, targetID,
+		s.WebhookID, "numbering",
+		database.TargetTypeLog, "", 0,
+	)
+
+	event := iSeedEvent(
+		t, s.WebhookDB, s.WebhookID, `{"numbering":true}`,
+	)
+
+	d := iSeedDelivery(
+		t, s.WebhookDB, event.ID, targetID,
+		database.DeliveryStatusPending,
+	)
+
+	rSeedResult(t, s.WebhookDB, d.ID, 1, false)
+	rSeedResult(t, s.WebhookDB, d.ID, 2, false)
+
+	s.Engine.ExportRecoverPendingDeliveries(
+		context.Background(), s.WebhookDB, s.WebhookID,
+	)
+
+	select {
+	case task := <-s.Engine.ExportDeliveryCh():
+		assert.Equal(t, d.ID, task.DeliveryID)
+		assert.Equal(t, 3, task.AttemptNum)
+	case <-time.After(2 * time.Second):
+		t.Fatal("expected the delivery to be recovered")
+	}
+}
+
+// TestSweepRecoversStrandedPending is the half that removes the
+// restart requirement: a delivery left at pending is picked up by the
+// periodic sweep.
+func TestSweepRecoversStrandedPending(t *testing.T) {
+	t.Parallel()
+
+	s := newISetup(t)
+	targetID := uuid.New().String()
+
+	iCreateTarget(t, s.MainDB, targetID,
+		s.WebhookID, "stranded",
+		database.TargetTypeLog, "", 0,
+	)
+
+	require.NoError(t, s.MainDB.Create(&database.Webhook{
+		BaseModel: database.BaseModel{ID: s.WebhookID},
+		UserID:    uuid.New().String(),
+		Name:      "stranded",
+	}).Error)
+
+	event := iSeedEvent(
+		t, s.WebhookDB, s.WebhookID, `{"stranded":true}`,
+	)
+
+	stranded := iSeedDelivery(
+		t, s.WebhookDB, event.ID, targetID,
+		database.DeliveryStatusPending,
+	)
+	rAgePending(t, s.WebhookDB, stranded.ID)
+
+	// A delivery a worker may still be holding: young, and therefore
+	// none of the sweep's business.
+	inFlight := iSeedDelivery(
+		t, s.WebhookDB, event.ID, targetID,
+		database.DeliveryStatusPending,
+	)
+
+	s.Engine.ExportSweepWebhookRetries(
+		context.Background(), s.WebhookID,
+	)
+
+	select {
+	case task := <-s.Engine.ExportDeliveryCh():
+		assert.Equal(t, stranded.ID, task.DeliveryID)
+	case <-time.After(2 * time.Second):
+		t.Fatal("expected the stranded delivery")
+	}
+
+	select {
+	case task := <-s.Engine.ExportDeliveryCh():
+		t.Fatalf(
+			"swept an in-flight delivery: %s",
+			task.DeliveryID,
+		)
+	case <-time.After(200 * time.Millisecond):
+	}
+
+	iAssertStatus(
+		t, s.WebhookDB, inFlight.ID,
+		database.DeliveryStatusPending,
+	)
+}
+
+// TestSweepClaimsAStrandedDeliveryOnlyOnce guards the repeat the sweep
+// would otherwise be: the row stays pending for as long as the attempt
+// runs, and a sweep a minute later must not send it a second time.
+func TestSweepClaimsAStrandedDeliveryOnlyOnce(t *testing.T) {
+	t.Parallel()
+
+	s := newISetup(t)
+	targetID := uuid.New().String()
+
+	iCreateTarget(t, s.MainDB, targetID,
+		s.WebhookID, "claimed",
+		database.TargetTypeLog, "", 0,
+	)
+
+	require.NoError(t, s.MainDB.Create(&database.Webhook{
+		BaseModel: database.BaseModel{ID: s.WebhookID},
+		UserID:    uuid.New().String(),
+		Name:      "claimed",
+	}).Error)
+
+	event := iSeedEvent(
+		t, s.WebhookDB, s.WebhookID, `{"claimed":true}`,
+	)
+
+	d := iSeedDelivery(
+		t, s.WebhookDB, event.ID, targetID,
+		database.DeliveryStatusPending,
+	)
+	rAgePending(t, s.WebhookDB, d.ID)
+
+	ctx := context.Background()
+
+	s.Engine.ExportSweepWebhookRetries(ctx, s.WebhookID)
+
+	select {
+	case task := <-s.Engine.ExportDeliveryCh():
+		assert.Equal(t, d.ID, task.DeliveryID)
+	case <-time.After(2 * time.Second):
+		t.Fatal("expected the stranded delivery")
+	}
+
+	// The delivery is still pending — nothing has run it yet — but
+	// the claim must keep the next sweep off it.
+	iAssertStatus(
+		t, s.WebhookDB, d.ID,
+		database.DeliveryStatusPending,
+	)
+
+	s.Engine.ExportSweepWebhookRetries(ctx, s.WebhookID)
+
+	select {
+	case task := <-s.Engine.ExportDeliveryCh():
+		t.Fatalf(
+			"sent a claimed delivery again: %s",
+			task.DeliveryID,
+		)
+	case <-time.After(200 * time.Millisecond):
+	}
+}
+
+// TestSweepSettlesStrandedPendingWithoutResending is the sweep's own
+// version of the reconcile: a stranded delivery holding a successful
+// result is settled where it stands, and the receiver hears nothing.
+func TestSweepSettlesStrandedPendingWithoutResending(
+	t *testing.T,
+) {
+	t.Parallel()
+
+	s := newISetup(t)
+	targetID := uuid.New().String()
+
+	iCreateTarget(t, s.MainDB, targetID,
+		s.WebhookID, "settled",
+		database.TargetTypeLog, "", 0,
+	)
+
+	require.NoError(t, s.MainDB.Create(&database.Webhook{
+		BaseModel: database.BaseModel{ID: s.WebhookID},
+		UserID:    uuid.New().String(),
+		Name:      "settled",
+	}).Error)
+
+	event := iSeedEvent(
+		t, s.WebhookDB, s.WebhookID, `{"settled":true}`,
+	)
+
+	d := iSeedDelivery(
+		t, s.WebhookDB, event.ID, targetID,
+		database.DeliveryStatusPending,
+	)
+	rSeedResult(t, s.WebhookDB, d.ID, 1, true)
+	rAgePending(t, s.WebhookDB, d.ID)
+
+	s.Engine.ExportSweepWebhookRetries(
+		context.Background(), s.WebhookID,
+	)
+
+	select {
+	case task := <-s.Engine.ExportDeliveryCh():
+		t.Fatalf(
+			"re-sent a delivery that already succeeded: %s",
+			task.DeliveryID,
+		)
+	case <-time.After(200 * time.Millisecond):
+	}
+
+	iAssertStatus(
+		t, s.WebhookDB, d.ID,
+		database.DeliveryStatusDelivered,
+	)
+
+	var attempts int64
+
+	require.NoError(t, s.WebhookDB.
+		Model(&database.DeliveryResult{}).
+		Where("delivery_id = ?", d.ID).
+		Count(&attempts).Error)
+	assert.Equal(
+		t, int64(1), attempts,
+		"settling must not invent an attempt",
+	)
+}
+
+// TestFailedResultWriteLeavesDeliveryRecoverable is the rule the
+// targets now follow: a bookkeeping write that fails must not advance
+// the status, because pending and retrying are the states the sweeps
+// recover and delivered is a claim the database refused to record.
+func TestFailedResultWriteLeavesDeliveryRecoverable(
+	t *testing.T,
+) {
+	t.Parallel()
+
+	s := newISetup(t)
+	targetID := uuid.New().String()
+
+	var hits atomic.Int64
+
+	ts := httptest.NewServer(http.HandlerFunc(
+		func(w http.ResponseWriter, _ *http.Request) {
+			hits.Add(1)
+			w.WriteHeader(http.StatusOK)
+		},
+	))
+	defer ts.Close()
+
+	event := iSeedEvent(
+		t, s.WebhookDB, s.WebhookID, `{"unwritable":true}`,
+	)
+
+	d := iSeedDelivery(
+		t, s.WebhookDB, event.ID, targetID,
+		database.DeliveryStatusPending,
+	)
+
+	// Drop the table the attempt row goes in, so the send succeeds
+	// and only the bookkeeping write fails.
+	require.NoError(
+		t,
+		s.WebhookDB.Exec("drop table delivery_results").Error,
+	)
+
+	full := &database.Delivery{
+		EventID:  event.ID,
+		TargetID: targetID,
+		Status:   database.DeliveryStatusPending,
+		Event:    event,
+		Target: database.Target{
+			Name:   "unwritable",
+			Type:   database.TargetTypeHTTP,
+			Config: iHTTPConfig(ts.URL),
+		},
+	}
+	full.ID = d.ID
+
+	s.Engine.ExportDeliverHTTP(
+		context.Background(), s.WebhookDB, full,
+		&delivery.Task{DeliveryID: d.ID, AttemptNum: 1},
+	)
+
+	assert.Equal(
+		t, int64(1), hits.Load(),
+		"the send itself must still happen",
+	)
+
+	iAssertStatus(
+		t, s.WebhookDB, d.ID,
+		database.DeliveryStatusPending,
+	)
+}

internal/delivery/target_database.go

@@ -58,12 +58,17 @@ func (t *databaseTarget) Deliver(
 			"error", err,
 		)
 
-		t.eng.recordResult(
+		recErr := t.eng.recordResult(
 			webhookDB, d, 1, false, 0, "",
 			err.Error(), elapsed.Milliseconds(),
 		)
+		if recErr != nil {
+			t.eng.bookkeepingFailed(d, recErr)
 
-		t.eng.updateDeliveryStatus(
+			return
+		}
+
+		t.eng.settleStatus(
 			webhookDB, d, d.Target.Type,
 			database.DeliveryStatusFailed,
 		)
@@ -71,12 +76,17 @@ func (t *databaseTarget) Deliver(
 		return
 	}
 
-	t.eng.recordResult(
+	recErr := t.eng.recordResult(
 		webhookDB, d, 1, true, 0, "", "",
 		elapsed.Milliseconds(),
 	)
+	if recErr != nil {
+		t.eng.bookkeepingFailed(d, recErr)
 
-	t.eng.updateDeliveryStatus(
+		return
+	}
+
+	t.eng.settleStatus(
 		webhookDB, d, d.Target.Type,
 		database.DeliveryStatusDelivered,
 	)

internal/delivery/target_database_archive.go

@@ -1,7 +1,6 @@
 package delivery
 
 import (
-	"database/sql"
 	"encoding/json"
 	"errors"
 	"fmt"
@@ -12,6 +11,7 @@ import (
 
 	"gorm.io/driver/sqlite"
 	"gorm.io/gorm"
+	"sneak.berlin/go/webhooker/internal/database"
 	"sneak.berlin/go/webhooker/internal/gormlog"
 )
 
@@ -30,13 +30,13 @@ const (
 	// path: open the archive file, creating it if missing, so a
 	// first write (or a write after the operator moved the file
 	// away) recreates it.
-	archiveModeCreate = "rwc"
+	archiveModeCreate = database.SQLiteModeCreate
 
 	// archiveModeExisting is the SQLite URI mode used by the idle
 	// sweep: open read-write but never create. A sweep must never
 	// conjure an empty archive file for a webhook that has a
 	// database target but has never received an event.
-	archiveModeExisting = "rw"
+	archiveModeExisting = database.SQLiteModeExisting
 )
 
 var (
@@ -273,9 +273,11 @@ func (w *archiveWriter) open(expiry time.Duration) error {
 func (w *archiveWriter) openMode(
 	mode string, expiry time.Duration,
 ) error {
-	dbURL := fmt.Sprintf("file:%s?mode=%s", w.path, mode)
+	// Opened through database.OpenSQLite so an archive file carries
-
+	// the same WAL journaling, busy timeout, immediate-transaction
-	sqlDB, err := sql.Open("sqlite", dbURL)
+	// locking, and pool bounds as every other database file. See
+	// internal/database/sqlite_open.go.
+	sqlDB, err := database.OpenSQLite(w.path, mode)
 	if err != nil {
 		return fmt.Errorf(
 			"opening archive database %s: %w", w.path, err,

internal/delivery/target_http.go

@@ -77,14 +77,19 @@ func (c *httpCore) fireAndForget(
 ) {
 	c.eng.observeAttempt(d.Target.Type, res.elapsed())
 
-	c.eng.recordResult(
+	err := c.eng.recordResult(
 		webhookDB, d, 1, res.success,
 		res.statusCode, res.respBody, res.errMsg,
 		res.duration,
 	)
+	if err != nil {
+		c.eng.bookkeepingFailed(d, err)
+
+		return
+	}
 
 	if res.success {
-		c.eng.updateDeliveryStatus(
+		c.eng.settleStatus(
 			webhookDB, d, d.Target.Type,
 			database.DeliveryStatusDelivered,
 		)
@@ -92,7 +97,7 @@ func (c *httpCore) fireAndForget(
 		return
 	}
 
-	c.eng.updateDeliveryStatus(
+	c.eng.settleStatus(
 		webhookDB, d, d.Target.Type,
 		database.DeliveryStatusFailed,
 	)
@@ -122,16 +127,25 @@ func (c *httpCore) withRetry(
 
 	c.eng.observeAttempt(d.Target.Type, res.elapsed())
 
-	c.eng.recordResult(
+	err := c.eng.recordResult(
 		webhookDB, d, attemptNum, res.success,
 		res.statusCode, res.respBody, res.errMsg,
 		res.duration,
 	)
+	if err != nil {
+		// The breaker still learns the outcome: it describes the
+		// target's health, which is unaffected by this database's.
+		c.recordCircuitOutcome(cb, res.success)
+
+		c.eng.bookkeepingFailed(d, err)
+
+		return
+	}
 
 	if res.success {
 		cb.RecordSuccess()
 
-		c.eng.updateDeliveryStatus(
+		c.eng.settleStatus(
 			webhookDB, d, d.Target.Type,
 			database.DeliveryStatusDelivered,
 		)
@@ -146,6 +160,20 @@ func (c *httpCore) withRetry(
 	)
 }
 
+// recordCircuitOutcome feeds one attempt's outcome to the target's
+// circuit breaker.
+func (c *httpCore) recordCircuitOutcome(
+	cb *CircuitBreaker, success bool,
+) {
+	if success {
+		cb.RecordSuccess()
+
+		return
+	}
+
+	cb.RecordFailure()
+}
+
 func (c *httpCore) circuitBreakerBlock(
 	webhookDB *gorm.DB,
 	d *database.Delivery,
@@ -169,7 +197,7 @@ func (c *httpCore) circuitBreakerBlock(
 		"cooldown_remaining", remaining,
 	)
 
-	c.eng.updateDeliveryStatus(
+	c.eng.settleStatus(
 		webhookDB, d, d.Target.Type,
 		database.DeliveryStatusRetrying,
 	)
@@ -189,7 +217,7 @@ func (c *httpCore) handleRetry(
 	attemptNum int,
 ) {
 	if attemptNum >= maxRetries {
-		c.eng.updateDeliveryStatus(
+		c.eng.settleStatus(
 			webhookDB, d, d.Target.Type,
 			database.DeliveryStatusFailed,
 		)
@@ -197,7 +225,7 @@ func (c *httpCore) handleRetry(
 		return
 	}
 
-	c.eng.updateDeliveryStatus(
+	c.eng.settleStatus(
 		webhookDB, d, d.Target.Type,
 		database.DeliveryStatusRetrying,
 	)
@@ -332,12 +360,17 @@ func (t *httpTarget) Deliver(
 			"error", err,
 		)
 
-		t.eng.recordResult(
+		recErr := t.eng.recordResult(
 			webhookDB, d, task.AttemptNum,
 			false, 0, "", err.Error(), 0,
 		)
+		if recErr != nil {
+			t.eng.bookkeepingFailed(d, recErr)
 
-		t.eng.updateDeliveryStatus(
+			return
+		}
+
+		t.eng.settleStatus(
 			webhookDB, d, d.Target.Type,
 			database.DeliveryStatusFailed,
 		)

internal/delivery/target_log.go

@@ -55,12 +55,17 @@ func (t *logTarget) Deliver(
 
 	t.eng.observeAttempt(d.Target.Type, elapsed)
 
-	t.eng.recordResult(
+	err := t.eng.recordResult(
 		webhookDB, d, 1, true, 0, "", "",
 		elapsed.Milliseconds(),
 	)
+	if err != nil {
+		t.eng.bookkeepingFailed(d, err)
 
-	t.eng.updateDeliveryStatus(
+		return
+	}
+
+	t.eng.settleStatus(
 		webhookDB, d, d.Target.Type,
 		database.DeliveryStatusDelivered,
 	)

internal/delivery/target_slack.go

@@ -95,12 +95,17 @@ func (t *slackTarget) failConfig(
 	d *database.Delivery,
 	err error,
 ) {
-	t.eng.recordResult(
+	recErr := t.eng.recordResult(
 		webhookDB, d, 1,
 		false, 0, "", err.Error(), 0,
 	)
+	if recErr != nil {
+		t.eng.bookkeepingFailed(d, recErr)
 
-	t.eng.updateDeliveryStatus(
+		return
+	}
+
+	t.eng.settleStatus(
 		webhookDB, d, d.Target.Type,
 		database.DeliveryStatusFailed,
 	)

internal/handlers/event_body.go

@@ -92,11 +92,10 @@ func (h *Handlers) HandleEventBodyDownload() http.HandlerFunc {
 // once per range.
 //
 // One consequence is worth keeping in view: the read finishes
-// before the client is written to, so no read lock is held for
+// before the client is written to, so nothing is held open for
-// the length of a slow download. These per-webhook databases
+// the length of a slow download. Under WAL a read no longer
-// run in SQLite's default journal mode rather than WAL, so a
+// blocks the receiver, but it does pin the WAL against
-// lock held that long would block the receiver from recording
+// checkpointing, and a download can last minutes.
-// new events.
 func (h *Handlers) serveEventBody(
 	w http.ResponseWriter,
 	r *http.Request,

internal/handlers/event_resubmit.go

@@ -143,10 +143,10 @@ func (h *Handlers) resubmitEvent(
 	}
 
 	// Read before the write transaction is opened. The body can be up
-	// to the 1 MB ingest cap, and holding a read of it inside the
+	// to the 1 MB ingest cap, and every transaction on these files
-	// transaction would extend how long the per-webhook database is
+	// takes the write lock at BEGIN (_txlock=immediate, see
-	// locked against the receiver, which runs these files in
+	// internal/database/sqlite_open.go), so reading inside it would
-	// SQLite's default journal mode rather than WAL.
+	// hold that lock against the receiver for the length of the read.
 	src, found, err := loadResubmitSource(
 		webhookDB, webhook.ID, eventID.String(),
 	)