027f0898e7
All checks were successful
check / check (push) Successful in 3m38s
An operator running `sqlite3 <db> .dump` against their own per-webhook database wedged it: 60 of 60 inbound webhooks rejected with HTTP 500, 206 delivered webhooks stranded at `pending`, and every one of them POSTed a second time on the next restart while the event log recorded a single attempt. Durability. Every SQLite file — main, per-webhook, and archive — now opens through one path, `internal/database/sqlite_open.go`, in WAL journal mode with a 10-second busy timeout, `BEGIN IMMEDIATE` transactions, and a bounded connection pool. WAL is what stops a reader blocking writers at all. `_txlock=immediate` is what stops a `COMMIT` failing while its transaction stays open on a pooled connection, which is how four `database is locked` errors became 593 `cannot start a transaction within a transaction`: a deferred transaction that upgrades to a write lock mid-flight gets SQLITE_BUSY without the busy handler being consulted. `cache=shared` is gone, because under it an in-process conflict is SQLITE_LOCKED, which the busy handler does not retry. Delivery. `recordResult` and `updateDeliveryStatus` return their errors instead of logging and dropping them, and a caller whose bookkeeping write failed writes nothing at all — the delivery keeps whichever non-terminal status it already held, and both sweeps recover it. Recovery and the sweep now reconcile before re-sending: a pending delivery that already holds a successful `DeliveryResult` is marked delivered rather than sent again, which is the state that did not previously exist. A delivery handed back out is claimed by compare-and-set so successive sweeps cannot send it repeatedly, and it continues its own attempt numbering instead of restarting at 1. The sweep gains a `pending`-with-age-bound arm, so a stranded delivery no longer waits for a restart. Docs. WAL produces `-wal`/`-shm` sidecars, so the backup and restore procedures in README.md are corrected: both documented procedures were re-run against a live instance, and a `-wal` left by a crash carries data the `.db` alone does not. Verified by reproducing the failure on unmodified `next` first — 6 targets, 60 events at 5/s, a concurrent `.dump` reader — which gave 38 HTTP 500s and 112 duplicate POSTs at the sinks across a restart. Both arms of the matched pair now show 0 inbound 500s, 0 engine write errors, and 0 new requests at the sinks after a restart, counted by payload.
73 lines
1.9 KiB
Go
73 lines
1.9 KiB
Go
package delivery
|
|
|
|
import (
|
|
"context"
|
|
"time"
|
|
|
|
"gorm.io/gorm"
|
|
"sneak.berlin/go/webhooker/internal/database"
|
|
)
|
|
|
|
// logTarget is a fire-and-forget target that logs the entire
|
|
// inbound webhook — the full request body and headers, plus
|
|
// the method, content type, and the webhook and entrypoint
|
|
// ids — then records a single successful attempt.
|
|
//
|
|
// This is the one log call in the service that deliberately writes
|
|
// unbounded client-chosen bytes, so it is the one exception to the
|
|
// per-field budgets in internal/logfield and to the ceiling stated on
|
|
// middleware.MaxAccessLogLineBytes. Capping here would defeat the
|
|
// target: emitting the payload IS the delivery. It costs nothing by
|
|
// default — an authenticated operator has to create a target of this
|
|
// type on a specific webhook before a single line is written — and the
|
|
// bytes it writes are bounded per event by maxWebhookBodySize (1 MB).
|
|
// An operator who adds one is choosing to spend log volume on the
|
|
// payloads that webhook receives.
|
|
type logTarget struct {
|
|
eng *Engine
|
|
}
|
|
|
|
// Deliver implements Target.
|
|
func (t *logTarget) Deliver(
|
|
_ context.Context,
|
|
webhookDB *gorm.DB,
|
|
d *database.Delivery,
|
|
_ *Task,
|
|
_ Scheduler,
|
|
) {
|
|
start := time.Now()
|
|
|
|
t.eng.log.Info(
|
|
"webhook event delivered to log target",
|
|
"delivery_id", d.ID,
|
|
"event_id", d.EventID,
|
|
"target_id", d.TargetID,
|
|
"target_name", d.Target.Name,
|
|
"webhook_id", d.Event.WebhookID,
|
|
"entrypoint_id", d.Event.EntrypointID,
|
|
"method", d.Event.Method,
|
|
"content_type", d.Event.ContentType,
|
|
"headers", d.Event.Headers,
|
|
"body", d.Event.Body,
|
|
)
|
|
|
|
elapsed := time.Since(start)
|
|
|
|
t.eng.observeAttempt(d.Target.Type, elapsed)
|
|
|
|
err := t.eng.recordResult(
|
|
webhookDB, d, 1, true, 0, "", "",
|
|
elapsed.Milliseconds(),
|
|
)
|
|
if err != nil {
|
|
t.eng.bookkeepingFailed(d, err)
|
|
|
|
return
|
|
}
|
|
|
|
t.eng.settleStatus(
|
|
webhookDB, d, d.Target.Type,
|
|
database.DeliveryStatusDelivered,
|
|
)
|
|
}
|