Add per-delivery replay to the event log (closes #203)
All checks were successful
check / check (push) Successful in 3m3s

A delivery that exhausted max_retries was failed forever. The event
body is durably stored, so the only way to get it delivered was to
download it and re-POST by hand.

The event log now offers a Replay action on any finished delivery.
Replay creates a NEW pending delivery for the same event and target
and hands it to the delivery engine through the same Notifier the
receiver uses, so it is retried, SSRF-guarded and circuit-broken
exactly as a first attempt. The original delivery's status,
timestamps and recorded attempts are never touched, and what is
re-sent is the stored event body, not the response the original
attempt received.

The target is read as it stands now, including soft-deleted rows so
that a deleted target refuses the replay with a message on the page
instead of erroring or delivering from stale configuration. A
deactivated target and a target id that names nothing refuse the same
way, as does a replay of a delivery the engine has not finished.

Two bounds on replay storms: the route carries a per-client POST rate
limit of 30 per minute, and the handler refuses a replay while an
earlier one for the same event and target is still pending or
retrying.

One new metric, webhooker_delivery_replays_total, on the existing
target_type label. A replay is a real delivery and moves the attempt,
outcome and duration series like any other; this counter is what
separates it from ordinary traffic without adding a dimension to
every existing series.

The delivery row is written with associations omitted and with
neither Event nor Target populated, so no target row reaches the
per-webhook event database.
This commit is contained in:
clawbot
2026-08-20 05:45:04 +00:00
committed by sneak
parent 9969694a47
commit ea6639ff31
12 changed files with 1240 additions and 14 deletions

View File

@@ -24,9 +24,32 @@ import (
"sneak.berlin/go/webhooker/internal/session"
)
type noopNotifier struct{}
// recordingNotifier is a delivery.Notifier that records the tasks it
// was handed, so a test can prove a handler queued the delivery it
// claims to have queued — and, on the refusal paths, that it queued
// nothing.
type recordingNotifier struct {
mu sync.Mutex
tasks []delivery.Task
}
func (n *noopNotifier) Notify([]delivery.Task) {}
func (n *recordingNotifier) Notify(tasks []delivery.Task) {
n.mu.Lock()
defer n.mu.Unlock()
n.tasks = append(n.tasks, tasks...)
}
// Tasks returns a copy of the recorded tasks.
func (n *recordingNotifier) Tasks() []delivery.Task {
n.mu.Lock()
defer n.mu.Unlock()
out := make([]delivery.Task, len(n.tasks))
copy(out, n.tasks)
return out
}
// recordingEvictor is a delivery.WebhookEvictor that records
// the webhook ids it was asked to evict, so a test can prove
@@ -74,8 +97,11 @@ func newTestApp(
database.NewWebhookDBManager,
healthcheck.New,
session.New,
func() delivery.Notifier {
return &noopNotifier{}
func() *recordingNotifier {
return &recordingNotifier{}
},
func(n *recordingNotifier) delivery.Notifier {
return n
},
func() *recordingEvictor {
return &recordingEvictor{}