Add per-delivery replay to the event log (closes #203)
Some checks failed
check / check (push) Failing after 2m17s
Some checks failed
check / check (push) Failing after 2m17s
A delivery that exhausted max_retries was failed forever. The event body is durably stored, so the only way to get it delivered was to download it and re-POST by hand. The event log now offers a Replay action on any finished delivery. Replay creates a NEW pending delivery for the same event and target and hands it to the delivery engine through the same Notifier the receiver uses, so it is retried, SSRF-guarded and circuit-broken exactly as a first attempt. The original delivery's status, timestamps and recorded attempts are never touched, and what is re-sent is the stored event body, not the response the original attempt received. The target is read as it stands now, including soft-deleted rows so that a deleted target refuses the replay with a message on the page instead of erroring or delivering from stale configuration. A deactivated target and a target id that names nothing refuse the same way, as does a replay of a delivery the engine has not finished. Two bounds on replay storms: the route carries a per-client POST rate limit of 30 per minute, and the handler refuses a replay while an earlier one for the same event and target is still pending or retrying. One new metric, webhooker_delivery_replays_total, on the existing target_type label. A replay is a real delivery and moves the attempt, outcome and duration series like any other; this counter is what separates it from ordinary traffic without adding a dimension to every existing series. The delivery row is written with associations omitted and with neither Event nor Target populated, so no target row reaches the per-webhook event database.
This commit is contained in:
@@ -11,6 +11,23 @@ const (
|
||||
DeliveryStatusRetrying DeliveryStatus = "retrying"
|
||||
)
|
||||
|
||||
// Terminal reports whether a delivery in this status has finished, so
|
||||
// the delivery engine will make no further attempt of its own.
|
||||
//
|
||||
// It is what decides which deliveries the event log offers to replay:
|
||||
// a pending or retrying delivery is still the engine's, and replaying
|
||||
// one would race it.
|
||||
func (s DeliveryStatus) Terminal() bool {
|
||||
switch s {
|
||||
case DeliveryStatusDelivered, DeliveryStatusFailed:
|
||||
return true
|
||||
case DeliveryStatusPending, DeliveryStatusRetrying:
|
||||
return false
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
// Delivery represents a delivery attempt for an event to a target
|
||||
type Delivery struct {
|
||||
BaseModel
|
||||
|
||||
Reference in New Issue
Block a user