Files
webhooker/internal/handlers/event_resubmit.go
clawbot 027f0898e7
All checks were successful
check / check (push) Successful in 3m38s
Make SQLite durable under concurrent readers and stop re-delivering stranded webhooks (closes #256)
An operator running `sqlite3 <db> .dump` against their own per-webhook
database wedged it: 60 of 60 inbound webhooks rejected with HTTP 500,
206 delivered webhooks stranded at `pending`, and every one of them
POSTed a second time on the next restart while the event log recorded a
single attempt.

Durability. Every SQLite file — main, per-webhook, and archive — now
opens through one path, `internal/database/sqlite_open.go`, in WAL
journal mode with a 10-second busy timeout, `BEGIN IMMEDIATE`
transactions, and a bounded connection pool. WAL is what stops a
reader blocking writers at all. `_txlock=immediate` is what stops a
`COMMIT` failing while its transaction stays open on a pooled
connection, which is how four `database is locked` errors became 593
`cannot start a transaction within a transaction`: a deferred
transaction that upgrades to a write lock mid-flight gets SQLITE_BUSY
without the busy handler being consulted. `cache=shared` is gone,
because under it an in-process conflict is SQLITE_LOCKED, which the
busy handler does not retry.

Delivery. `recordResult` and `updateDeliveryStatus` return their
errors instead of logging and dropping them, and a caller whose
bookkeeping write failed writes nothing at all — the delivery keeps
whichever non-terminal status it already held, and both sweeps recover
it. Recovery and the sweep now reconcile before re-sending: a pending
delivery that already holds a successful `DeliveryResult` is marked
delivered rather than sent again, which is the state that did not
previously exist. A delivery handed back out is claimed by
compare-and-set so successive sweeps cannot send it repeatedly, and it
continues its own attempt numbering instead of restarting at 1. The
sweep gains a `pending`-with-age-bound arm, so a stranded delivery no
longer waits for a restart.

Docs. WAL produces `-wal`/`-shm` sidecars, so the backup and restore
procedures in README.md are corrected: both documented procedures were
re-run against a live instance, and a `-wal` left by a crash carries
data the `.db` alone does not.

Verified by reproducing the failure on unmodified `next` first — 6
targets, 60 events at 5/s, a concurrent `.dump` reader — which gave 38
HTTP 500s and 112 duplicate POSTs at the sinks across a restart. Both
arms of the matched pair now show 0 inbound 500s, 0 engine write
errors, and 0 new requests at the sinks after a restart, counted by
payload.
2026-08-23 23:40:03 +00:00

278 lines
8.1 KiB
Go

package handlers
import (
"errors"
"net/http"
"strconv"
"github.com/go-chi/chi"
"github.com/google/uuid"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// resubmitOutcomeParam is the query parameter the resubmit POST
// redirects with and the event log page reads its banner from.
const resubmitOutcomeParam = "resubmit"
// resubmitOutcomeCode is the outcome of a resubmit POST. The redirect
// carries one of these fixed codes rather than a message, so nothing a
// client submits can reach the rendered page through it.
type resubmitOutcomeCode string
const (
// resubmitQueued reports that a new event was stored and its
// deliveries handed to the delivery engine.
resubmitQueued resubmitOutcomeCode = "queued"
// resubmitNoTargets reports a source with no active targets. The
// new event is stored either way, exactly as a received event
// with no targets is.
resubmitNoTargets resubmitOutcomeCode = "no-targets"
)
// resubmitOutcome returns the banner the event log page shows for an
// outcome code, and whether the resubmit was queued. An unrecognised
// code yields no banner.
func resubmitOutcome(code string) (string, bool) {
switch resubmitOutcomeCode(code) {
case resubmitQueued:
return "Resubmitted: a new event was created from the stored " +
"one and queued to every active target.", true
case resubmitNoTargets:
return "Resubmitted: a new event was created, but this " +
"source has no active targets, so nothing was queued.",
true
default:
return "", false
}
}
// resubmitSource is the stored event a resubmit copies. Its body is
// read as bytes rather than as a string so the copy is byte-identical
// to what was received, whatever the payload's encoding.
type resubmitSource struct {
ID string
EntrypointID string
Method string
Headers string
ContentType string
Body []byte
}
// resubmitColumns is the projection resubmitSource is loaded through.
// The cast to blob is what makes the driver hand back the stored bytes
// rather than a string conversion, the same reason eventBodyQuery
// casts.
const resubmitColumns = "id, entrypoint_id, method, headers, " +
"content_type, cast(body as blob) AS body"
// HandleEventResubmit re-injects a stored event as a new undelivered
// event.
//
// This is the testing counterpart to per-delivery replay, and the two
// select targets differently on purpose. A replay re-sends ONE
// finished delivery to ITS OWN target, which is recovery. A resubmit
// stores a NEW event copied from the stored one and fans it out to the
// webhook's currently ACTIVE targets, resolved fresh by the query the
// receiver uses — so a target created after the original event arrived
// receives it, which is what makes capturing real traffic and firing
// it at a backend under development possible. The original event's
// deliveries have no bearing on where the copy goes.
//
// Nothing about the original delivery is re-sent: what is re-injected
// is the stored EVENT. The response bodies and headers the original
// deliveries received stay where they are.
//
// Inbound signature verification is deliberately not re-run. There is
// no inbound signature to check on a copy the operator submits; the
// route is authenticated and CSRF-protected as an operator action.
//
// Resubmitting the same event repeatedly is supported and is the point
// of the feature, so replay's in-flight refusal is deliberately not
// applied here. The route's rate limit is what bounds a held-down
// button.
func (h *Handlers) HandleEventResubmit() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
webhook, ok := h.ownedWebhook(w, r)
if !ok {
return
}
// The body size cap is enforced by the MaxBodySize
// middleware, which runs before CSRF parses the form.
err := r.ParseForm()
if err != nil {
http.Error(w, "Bad request", http.StatusBadRequest)
return
}
h.resubmitEvent(w, r, webhook)
}
}
// resubmitEvent performs the resubmit for a webhook the caller has
// already established the session's user owns.
func (h *Handlers) resubmitEvent(
w http.ResponseWriter,
r *http.Request,
webhook database.Webhook,
) {
// Parsing the id before use keeps a malformed id out of the SQL
// and makes the value the query sees come from uuid's own fixed
// alphabet rather than from the request.
eventID, err := uuid.Parse(chi.URLParam(r, "eventID"))
if err != nil {
http.NotFound(w, r)
return
}
if !h.dbMgr.DBExists(webhook.ID) {
http.NotFound(w, r)
return
}
webhookDB, err := h.dbMgr.GetDB(webhook.ID)
if err != nil {
h.serverError(w, "failed to get webhook database", err)
return
}
// Read before the write transaction is opened. The body can be up
// to the 1 MB ingest cap, and every transaction on these files
// takes the write lock at BEGIN (_txlock=immediate, see
// internal/database/sqlite_open.go), so reading inside it would
// hold that lock against the receiver for the length of the read.
src, found, err := loadResubmitSource(
webhookDB, webhook.ID, eventID.String(),
)
if err != nil {
h.serverError(w, "failed to load event to resubmit", err)
return
}
// A miss is a 404 whether the event was reaped, belongs to
// another webhook, or never existed.
if !found {
http.NotFound(w, r)
return
}
h.queueResubmit(w, r, webhook, src)
}
// loadResubmitSource reads the stored event a resubmit copies, and
// whether it exists within the webhook.
//
// The webhook_id predicate is currently redundant against the
// per-webhook database files — a sibling webhook's event is not in the
// database being queried at all — and is there so the scoping survives
// any future change that puts more than one webhook's events in one
// file. Going through Model applies GORM's soft-delete scope, which is
// what stops a reaped event being resubmitted.
func loadResubmitSource(
webhookDB *gorm.DB,
webhookID, eventID string,
) (resubmitSource, bool, error) {
var src resubmitSource
err := webhookDB.Model(&database.Event{}).
Select(resubmitColumns).
Where("id = ? AND webhook_id = ?", eventID, webhookID).
First(&src).Error
if errors.Is(err, gorm.ErrRecordNotFound) {
return src, false, nil
}
if err != nil {
return src, false, err
}
return src, true, nil
}
// queueResubmit stores the copy and fans it out to the webhook's
// active targets.
func (h *Handlers) queueResubmit(
w http.ResponseWriter,
r *http.Request,
webhook database.Webhook,
src resubmitSource,
) {
// The receiver's own query, run now: an active target created
// after the original event arrived is included, and an
// inactive one is skipped rather than refused.
targets, err := h.loadActiveTargets(webhook.ID)
if err != nil {
h.serverError(w, "failed to query targets", err)
return
}
event, tasks, err := h.createAndFanOut(
eventSource{
WebhookID: webhook.ID,
EntrypointID: src.EntrypointID,
Method: src.Method,
HeadersJSON: src.Headers,
ContentType: src.ContentType,
Body: src.Body,
ResubmittedFromID: &src.ID,
},
targets,
)
if err != nil {
h.serverError(w, "failed to store resubmitted event", err)
return
}
h.mtr.EventResubmitted()
h.log.Info(
"event resubmitted",
"webhook_id", webhook.ID,
"event_id", event.ID,
"resubmitted_from_id", src.ID,
"target_count", len(tasks),
)
code := resubmitQueued
if len(tasks) == 0 {
code = resubmitNoTargets
}
h.finishResubmit(w, r, webhook, code)
}
// finishResubmit redirects back to the event log the resubmit was
// triggered from, carrying the outcome code the page turns into a
// banner and the page number the form submitted.
func (h *Handlers) finishResubmit(
w http.ResponseWriter,
r *http.Request,
webhook database.Webhook,
code resubmitOutcomeCode,
) {
dest := "/source/" + webhook.ID + "/logs?" +
resubmitOutcomeParam + "=" + string(code)
// The page is read from the form rather than the query string:
// this is a POST, and its query string is what logs and Referer
// headers record.
if page := pageOrFirst(
r.PostFormValue("page"),
); page > 1 {
dest += "&page=" + strconv.Itoa(page)
}
http.Redirect(w, r, dest, http.StatusSeeOther)
}