Add per-delivery replay to the event log (closes #203)
All checks were successful
check / check (push) Successful in 3m3s
All checks were successful
check / check (push) Successful in 3m3s
A delivery that exhausted max_retries was failed forever. The event body is durably stored, so the only way to get it delivered was to download it and re-POST by hand. The event log now offers a Replay action on any finished delivery. Replay creates a NEW pending delivery for the same event and target and hands it to the delivery engine through the same Notifier the receiver uses, so it is retried, SSRF-guarded and circuit-broken exactly as a first attempt. The original delivery's status, timestamps and recorded attempts are never touched, and what is re-sent is the stored event body, not the response the original attempt received. The target is read as it stands now, including soft-deleted rows so that a deleted target refuses the replay with a message on the page instead of erroring or delivering from stale configuration. A deactivated target and a target id that names nothing refuse the same way, as does a replay of a delivery the engine has not finished. Two bounds on replay storms: the route carries a per-client POST rate limit of 30 per minute, and the handler refuses a replay while an earlier one for the same event and target is still pending or retrying. One new metric, webhooker_delivery_replays_total, on the existing target_type label. A replay is a real delivery and moves the attempt, outcome and duration series like any other; this counter is what separates it from ordinary traffic without adding a dimension to every existing series. The delivery row is written with associations omitted and with neither Event nor Target populated, so no target row reaches the per-webhook event database.
This commit is contained in:
@@ -34,6 +34,16 @@ const (
|
||||
// password change rate limit.
|
||||
passwordChangeRateInterval = 1 * time.Minute
|
||||
|
||||
// replayRateLimit is the maximum number of delivery replays one
|
||||
// client may queue per interval. Each replay puts a delivery on
|
||||
// the engine's queue, so without a ceiling one operator holding
|
||||
// the button down — or scripting it — queues unbounded outbound
|
||||
// work. It sits far above any rate a person clicks at.
|
||||
replayRateLimit = 30
|
||||
|
||||
// replayRateInterval is the time window for the replay limit.
|
||||
replayRateInterval = 1 * time.Minute
|
||||
|
||||
// receiverRateInterval is the time window for the webhook
|
||||
// receiver rate limit. The configured limit is expressed in
|
||||
// requests per minute.
|
||||
@@ -290,6 +300,21 @@ func (m *Middleware) PasswordChangeRateLimit() func(http.Handler) http.Handler {
|
||||
)
|
||||
}
|
||||
|
||||
// ReplayRateLimit returns middleware that enforces per-IP rate
|
||||
// limiting on delivery replays.
|
||||
//
|
||||
// Like the password-change limit it is spent on arrival, which is safe
|
||||
// for the same reason: RequireAuth runs ahead of it, so only a request
|
||||
// already carrying a valid session can reach the bucket.
|
||||
func (m *Middleware) ReplayRateLimit() func(http.Handler) http.Handler {
|
||||
return m.postRateLimit(
|
||||
replayRateLimit,
|
||||
replayRateInterval,
|
||||
"delivery replay rate limit exceeded",
|
||||
"Too many replays. Please try again later.",
|
||||
)
|
||||
}
|
||||
|
||||
// postRateLimit builds middleware that enforces a per-IP rate
|
||||
// limit on POST requests only; all other methods pass through
|
||||
// unaffected. Requests over the limit receive a 429 with the
|
||||
|
||||
Reference in New Issue
Block a user