All checks were successful
check / check (push) Successful in 3m33s
An operator running `sqlite3 <db> .dump` against their own per-webhook database wedged it: inbound webhooks rejected with HTTP 500, delivered webhooks stranded at `pending`, and every one of them POSTed a second time on the next restart while the event log recorded a single attempt. Durability. Every SQLite file — main, per-webhook, and archive — now opens through one path, `internal/database/sqlite_open.go`, in WAL journal mode with a 10-second busy timeout, `BEGIN IMMEDIATE` transactions, and a bounded connection pool. WAL is what stops a reader blocking writers at all. `_txlock=immediate` is what stops a `COMMIT` failing while its transaction stays open on a pooled connection, which is how four `database is locked` errors became 593 `cannot start a transaction within a transaction`. `cache=shared` is gone, because under it an in-process conflict is SQLITE_LOCKED, which the busy handler does not retry. The busy timeout is applied before journal_mode: the driver runs DSN pragmas in order on every new connection, and `PRAGMA journal_mode` takes a lock, so the reverse order leaves the one pragma that can block uncovered by the handler meant to cover it. Eligibility. `internal/delivery/inflight.go` holds the set of deliveries the engine owns — taken when a task is queued, when a target schedules a retry, and by every recovery path before it re-dispatches; dropped when the worker that ran the task returns. Recovery and both sweep arms re-dispatch only what the set does not hold. Nothing decides that from a row's age: a delivery waiting in a 10000-deep channel is arbitrarily old and perfectly healthy, and reasoning from age re-sends it. `takeForRedispatch` is the single gate every re-dispatch goes through — ownership first, then a conditional update confirming the row is still in the status the batch read. Bookkeeping. `recordResult` and `updateDeliveryStatus` return their errors instead of logging and dropping them, and a caller whose bookkeeping write failed writes nothing at all: the delivery keeps whichever non-terminal status it already held, and the sweeps recover it. Every recovery path — pending and retrying alike — first settles any delivery that already holds a successful `DeliveryResult` rather than sending it again. Recovery continues each delivery's own attempt numbering instead of restarting at 1. The sweep gains a `pending`-with-age-bound arm, so a stranded delivery no longer waits for a restart. Docs. WAL produces `-wal`/`-shm` sidecars, so the backup and restore procedures in README.md are corrected against measurement: both documented procedures were re-run against a live instance, a `-wal` left by a crash carries data the `.db` alone does not, and an archive file normally holds its rows in a `-wal` rather than in the `.db`.
278 lines
8.1 KiB
Go
278 lines
8.1 KiB
Go
package handlers
|
|
|
|
import (
|
|
"errors"
|
|
"net/http"
|
|
"strconv"
|
|
|
|
"github.com/go-chi/chi"
|
|
"github.com/google/uuid"
|
|
"gorm.io/gorm"
|
|
"sneak.berlin/go/webhooker/internal/database"
|
|
)
|
|
|
|
// resubmitOutcomeParam is the query parameter the resubmit POST
|
|
// redirects with and the event log page reads its banner from.
|
|
const resubmitOutcomeParam = "resubmit"
|
|
|
|
// resubmitOutcomeCode is the outcome of a resubmit POST. The redirect
|
|
// carries one of these fixed codes rather than a message, so nothing a
|
|
// client submits can reach the rendered page through it.
|
|
type resubmitOutcomeCode string
|
|
|
|
const (
|
|
// resubmitQueued reports that a new event was stored and its
|
|
// deliveries handed to the delivery engine.
|
|
resubmitQueued resubmitOutcomeCode = "queued"
|
|
|
|
// resubmitNoTargets reports a source with no active targets. The
|
|
// new event is stored either way, exactly as a received event
|
|
// with no targets is.
|
|
resubmitNoTargets resubmitOutcomeCode = "no-targets"
|
|
)
|
|
|
|
// resubmitOutcome returns the banner the event log page shows for an
|
|
// outcome code, and whether the resubmit was queued. An unrecognised
|
|
// code yields no banner.
|
|
func resubmitOutcome(code string) (string, bool) {
|
|
switch resubmitOutcomeCode(code) {
|
|
case resubmitQueued:
|
|
return "Resubmitted: a new event was created from the stored " +
|
|
"one and queued to every active target.", true
|
|
case resubmitNoTargets:
|
|
return "Resubmitted: a new event was created, but this " +
|
|
"source has no active targets, so nothing was queued.",
|
|
true
|
|
default:
|
|
return "", false
|
|
}
|
|
}
|
|
|
|
// resubmitSource is the stored event a resubmit copies. Its body is
|
|
// read as bytes rather than as a string so the copy is byte-identical
|
|
// to what was received, whatever the payload's encoding.
|
|
type resubmitSource struct {
|
|
ID string
|
|
EntrypointID string
|
|
Method string
|
|
Headers string
|
|
ContentType string
|
|
Body []byte
|
|
}
|
|
|
|
// resubmitColumns is the projection resubmitSource is loaded through.
|
|
// The cast to blob is what makes the driver hand back the stored bytes
|
|
// rather than a string conversion, the same reason eventBodyQuery
|
|
// casts.
|
|
const resubmitColumns = "id, entrypoint_id, method, headers, " +
|
|
"content_type, cast(body as blob) AS body"
|
|
|
|
// HandleEventResubmit re-injects a stored event as a new undelivered
|
|
// event.
|
|
//
|
|
// This is the testing counterpart to per-delivery replay, and the two
|
|
// select targets differently on purpose. A replay re-sends ONE
|
|
// finished delivery to ITS OWN target, which is recovery. A resubmit
|
|
// stores a NEW event copied from the stored one and fans it out to the
|
|
// webhook's currently ACTIVE targets, resolved fresh by the query the
|
|
// receiver uses — so a target created after the original event arrived
|
|
// receives it, which is what makes capturing real traffic and firing
|
|
// it at a backend under development possible. The original event's
|
|
// deliveries have no bearing on where the copy goes.
|
|
//
|
|
// Nothing about the original delivery is re-sent: what is re-injected
|
|
// is the stored EVENT. The response bodies and headers the original
|
|
// deliveries received stay where they are.
|
|
//
|
|
// Inbound signature verification is deliberately not re-run. There is
|
|
// no inbound signature to check on a copy the operator submits; the
|
|
// route is authenticated and CSRF-protected as an operator action.
|
|
//
|
|
// Resubmitting the same event repeatedly is supported and is the point
|
|
// of the feature, so replay's in-flight refusal is deliberately not
|
|
// applied here. The route's rate limit is what bounds a held-down
|
|
// button.
|
|
func (h *Handlers) HandleEventResubmit() http.HandlerFunc {
|
|
return func(w http.ResponseWriter, r *http.Request) {
|
|
webhook, ok := h.ownedWebhook(w, r)
|
|
if !ok {
|
|
return
|
|
}
|
|
|
|
// The body size cap is enforced by the MaxBodySize
|
|
// middleware, which runs before CSRF parses the form.
|
|
err := r.ParseForm()
|
|
if err != nil {
|
|
http.Error(w, "Bad request", http.StatusBadRequest)
|
|
|
|
return
|
|
}
|
|
|
|
h.resubmitEvent(w, r, webhook)
|
|
}
|
|
}
|
|
|
|
// resubmitEvent performs the resubmit for a webhook the caller has
|
|
// already established the session's user owns.
|
|
func (h *Handlers) resubmitEvent(
|
|
w http.ResponseWriter,
|
|
r *http.Request,
|
|
webhook database.Webhook,
|
|
) {
|
|
// Parsing the id before use keeps a malformed id out of the SQL
|
|
// and makes the value the query sees come from uuid's own fixed
|
|
// alphabet rather than from the request.
|
|
eventID, err := uuid.Parse(chi.URLParam(r, "eventID"))
|
|
if err != nil {
|
|
http.NotFound(w, r)
|
|
|
|
return
|
|
}
|
|
|
|
if !h.dbMgr.DBExists(webhook.ID) {
|
|
http.NotFound(w, r)
|
|
|
|
return
|
|
}
|
|
|
|
webhookDB, err := h.dbMgr.GetDB(webhook.ID)
|
|
if err != nil {
|
|
h.serverError(w, "failed to get webhook database", err)
|
|
|
|
return
|
|
}
|
|
|
|
// Read before the write transaction is opened. The body can be up
|
|
// to the 1 MB ingest cap, and every transaction on these files
|
|
// takes the write lock at BEGIN (_txlock=immediate, see
|
|
// internal/database/sqlite_open.go), so reading inside it would
|
|
// hold that lock against the receiver for the length of the read.
|
|
src, found, err := loadResubmitSource(
|
|
webhookDB, webhook.ID, eventID.String(),
|
|
)
|
|
if err != nil {
|
|
h.serverError(w, "failed to load event to resubmit", err)
|
|
|
|
return
|
|
}
|
|
|
|
// A miss is a 404 whether the event was reaped, belongs to
|
|
// another webhook, or never existed.
|
|
if !found {
|
|
http.NotFound(w, r)
|
|
|
|
return
|
|
}
|
|
|
|
h.queueResubmit(w, r, webhook, src)
|
|
}
|
|
|
|
// loadResubmitSource reads the stored event a resubmit copies, and
|
|
// whether it exists within the webhook.
|
|
//
|
|
// The webhook_id predicate is currently redundant against the
|
|
// per-webhook database files — a sibling webhook's event is not in the
|
|
// database being queried at all — and is there so the scoping survives
|
|
// any future change that puts more than one webhook's events in one
|
|
// file. Going through Model applies GORM's soft-delete scope, which is
|
|
// what stops a reaped event being resubmitted.
|
|
func loadResubmitSource(
|
|
webhookDB *gorm.DB,
|
|
webhookID, eventID string,
|
|
) (resubmitSource, bool, error) {
|
|
var src resubmitSource
|
|
|
|
err := webhookDB.Model(&database.Event{}).
|
|
Select(resubmitColumns).
|
|
Where("id = ? AND webhook_id = ?", eventID, webhookID).
|
|
First(&src).Error
|
|
|
|
if errors.Is(err, gorm.ErrRecordNotFound) {
|
|
return src, false, nil
|
|
}
|
|
|
|
if err != nil {
|
|
return src, false, err
|
|
}
|
|
|
|
return src, true, nil
|
|
}
|
|
|
|
// queueResubmit stores the copy and fans it out to the webhook's
|
|
// active targets.
|
|
func (h *Handlers) queueResubmit(
|
|
w http.ResponseWriter,
|
|
r *http.Request,
|
|
webhook database.Webhook,
|
|
src resubmitSource,
|
|
) {
|
|
// The receiver's own query, run now: an active target created
|
|
// after the original event arrived is included, and an
|
|
// inactive one is skipped rather than refused.
|
|
targets, err := h.loadActiveTargets(webhook.ID)
|
|
if err != nil {
|
|
h.serverError(w, "failed to query targets", err)
|
|
|
|
return
|
|
}
|
|
|
|
event, tasks, err := h.createAndFanOut(
|
|
eventSource{
|
|
WebhookID: webhook.ID,
|
|
EntrypointID: src.EntrypointID,
|
|
Method: src.Method,
|
|
HeadersJSON: src.Headers,
|
|
ContentType: src.ContentType,
|
|
Body: src.Body,
|
|
ResubmittedFromID: &src.ID,
|
|
},
|
|
targets,
|
|
)
|
|
if err != nil {
|
|
h.serverError(w, "failed to store resubmitted event", err)
|
|
|
|
return
|
|
}
|
|
|
|
h.mtr.EventResubmitted()
|
|
|
|
h.log.Info(
|
|
"event resubmitted",
|
|
"webhook_id", webhook.ID,
|
|
"event_id", event.ID,
|
|
"resubmitted_from_id", src.ID,
|
|
"target_count", len(tasks),
|
|
)
|
|
|
|
code := resubmitQueued
|
|
if len(tasks) == 0 {
|
|
code = resubmitNoTargets
|
|
}
|
|
|
|
h.finishResubmit(w, r, webhook, code)
|
|
}
|
|
|
|
// finishResubmit redirects back to the event log the resubmit was
|
|
// triggered from, carrying the outcome code the page turns into a
|
|
// banner and the page number the form submitted.
|
|
func (h *Handlers) finishResubmit(
|
|
w http.ResponseWriter,
|
|
r *http.Request,
|
|
webhook database.Webhook,
|
|
code resubmitOutcomeCode,
|
|
) {
|
|
dest := "/source/" + webhook.ID + "/logs?" +
|
|
resubmitOutcomeParam + "=" + string(code)
|
|
|
|
// The page is read from the form rather than the query string:
|
|
// this is a POST, and its query string is what logs and Referer
|
|
// headers record.
|
|
if page := pageOrFirst(
|
|
r.PostFormValue("page"),
|
|
); page > 1 {
|
|
dest += "&page=" + strconv.Itoa(page)
|
|
}
|
|
|
|
http.Redirect(w, r, dest, http.StatusSeeOther)
|
|
}
|