Add per-delivery replay to the event log (closes #203)
Some checks failed
check / check (push) Failing after 2m17s

A delivery that exhausted max_retries was failed forever. The event
body is durably stored, so the only way to get it delivered was to
download it and re-POST by hand.

The event log now offers a Replay action on any finished delivery.
Replay creates a NEW pending delivery for the same event and target
and hands it to the delivery engine through the same Notifier the
receiver uses, so it is retried, SSRF-guarded and circuit-broken
exactly as a first attempt. The original delivery's status,
timestamps and recorded attempts are never touched, and what is
re-sent is the stored event body, not the response the original
attempt received.

The target is read as it stands now, including soft-deleted rows so
that a deleted target refuses the replay with a message on the page
instead of erroring or delivering from stale configuration. A
deactivated target and a target id that names nothing refuse the same
way, as does a replay of a delivery the engine has not finished.

Two bounds on replay storms: the route carries a per-client POST rate
limit of 30 per minute, and the handler refuses a replay while an
earlier one for the same event and target is still pending or
retrying.

One new metric, webhooker_delivery_replays_total, on the existing
target_type label. A replay is a real delivery and moves the attempt,
outcome and duration series like any other; this counter is what
separates it from ordinary traffic without adding a dimension to
every existing series.

The delivery row is written with associations omitted and with
neither Event nor Target populated, so no target row reaches the
per-webhook event database.
This commit is contained in:
clawbot
2026-08-20 05:45:04 +00:00
parent aba02bc509
commit c1fce4326c
12 changed files with 1240 additions and 14 deletions

View File

@@ -34,6 +34,16 @@ const (
// password change rate limit.
passwordChangeRateInterval = 1 * time.Minute
// replayRateLimit is the maximum number of delivery replays one
// client may queue per interval. Each replay puts a delivery on
// the engine's queue, so without a ceiling one operator holding
// the button down — or scripting it — queues unbounded outbound
// work. It sits far above any rate a person clicks at.
replayRateLimit = 30
// replayRateInterval is the time window for the replay limit.
replayRateInterval = 1 * time.Minute
// receiverRateInterval is the time window for the webhook
// receiver rate limit. The configured limit is expressed in
// requests per minute.
@@ -290,6 +300,21 @@ func (m *Middleware) PasswordChangeRateLimit() func(http.Handler) http.Handler {
)
}
// ReplayRateLimit returns middleware that enforces per-IP rate
// limiting on delivery replays.
//
// Like the password-change limit it is spent on arrival, which is safe
// for the same reason: RequireAuth runs ahead of it, so only a request
// already carrying a valid session can reach the bucket.
func (m *Middleware) ReplayRateLimit() func(http.Handler) http.Handler {
return m.postRateLimit(
replayRateLimit,
replayRateInterval,
"delivery replay rate limit exceeded",
"Too many replays. Please try again later.",
)
}
// postRateLimit builds middleware that enforces a per-IP rate
// limit on POST requests only; all other methods pass through
// unaffected. Requests over the limit receive a 429 with the