Add per-delivery replay to the event log (closes #203)
Some checks failed
check / check (push) Failing after 2m17s

A delivery that exhausted max_retries was failed forever. The event
body is durably stored, so the only way to get it delivered was to
download it and re-POST by hand.

The event log now offers a Replay action on any finished delivery.
Replay creates a NEW pending delivery for the same event and target
and hands it to the delivery engine through the same Notifier the
receiver uses, so it is retried, SSRF-guarded and circuit-broken
exactly as a first attempt. The original delivery's status,
timestamps and recorded attempts are never touched, and what is
re-sent is the stored event body, not the response the original
attempt received.

The target is read as it stands now, including soft-deleted rows so
that a deleted target refuses the replay with a message on the page
instead of erroring or delivering from stale configuration. A
deactivated target and a target id that names nothing refuse the same
way, as does a replay of a delivery the engine has not finished.

Two bounds on replay storms: the route carries a per-client POST rate
limit of 30 per minute, and the handler refuses a replay while an
earlier one for the same event and target is still pending or
retrying.

One new metric, webhooker_delivery_replays_total, on the existing
target_type label. A replay is a real delivery and moves the attempt,
outcome and duration series like any other; this counter is what
separates it from ordinary traffic without adding a dimension to
every existing series.

The delivery row is written with associations omitted and with
neither Event nor Target populated, so no target row reaches the
per-webhook event database.
This commit is contained in:
clawbot
2026-08-20 05:45:04 +00:00
parent aba02bc509
commit c1fce4326c
12 changed files with 1240 additions and 14 deletions

View File

@@ -11,6 +11,23 @@ const (
DeliveryStatusRetrying DeliveryStatus = "retrying"
)
// Terminal reports whether a delivery in this status has finished, so
// the delivery engine will make no further attempt of its own.
//
// It is what decides which deliveries the event log offers to replay:
// a pending or retrying delivery is still the engine's, and replaying
// one would race it.
func (s DeliveryStatus) Terminal() bool {
switch s {
case DeliveryStatusDelivered, DeliveryStatusFailed:
return true
case DeliveryStatusPending, DeliveryStatusRetrying:
return false
default:
return false
}
}
// Delivery represents a delivery attempt for an event to a target
type Delivery struct {
BaseModel