Add per-delivery replay to the event log (closes #203)
Some checks failed
check / check (push) Failing after 2m17s
Some checks failed
check / check (push) Failing after 2m17s
A delivery that exhausted max_retries was failed forever. The event body is durably stored, so the only way to get it delivered was to download it and re-POST by hand. The event log now offers a Replay action on any finished delivery. Replay creates a NEW pending delivery for the same event and target and hands it to the delivery engine through the same Notifier the receiver uses, so it is retried, SSRF-guarded and circuit-broken exactly as a first attempt. The original delivery's status, timestamps and recorded attempts are never touched, and what is re-sent is the stored event body, not the response the original attempt received. The target is read as it stands now, including soft-deleted rows so that a deleted target refuses the replay with a message on the page instead of erroring or delivering from stale configuration. A deactivated target and a target id that names nothing refuse the same way, as does a replay of a delivery the engine has not finished. Two bounds on replay storms: the route carries a per-client POST rate limit of 30 per minute, and the handler refuses a replay while an earlier one for the same event and target is still pending or retrying. One new metric, webhooker_delivery_replays_total, on the existing target_type label. A replay is a real delivery and moves the attempt, outcome and duration series like any other; this counter is what separates it from ordinary traffic without adding a dimension to every existing series. The delivery row is written with associations omitted and with neither Event nor Target populated, so no target row reaches the per-webhook event database.
This commit is contained in:
20
README.md
20
README.md
@@ -904,6 +904,23 @@ DeliveryResults.
|
||||
succeeded).
|
||||
- **`failed`** — All retry attempts exhausted without success.
|
||||
|
||||
**Replay.** A `delivered` or `failed` delivery is finished as far as
|
||||
the engine is concerned, but the event is still stored, so the event
|
||||
log offers a per-delivery **Replay** action for it. Replay creates a
|
||||
NEW `pending` delivery for the same event and target and hands it to
|
||||
the engine on the ordinary path — same retries, same SSRF guard, same
|
||||
circuit breaker as a first attempt. It never touches the delivery it
|
||||
repeats: that row's status, timestamps and recorded attempts stand as
|
||||
the record of what happened.
|
||||
|
||||
What is re-sent is the stored event body, against the target's
|
||||
configuration **as it stands now** — the point of a replay is to
|
||||
deliver where the destination has since been fixed. A target that has
|
||||
been deleted or deactivated therefore refuses the replay with a
|
||||
message on the event log rather than delivering from stale
|
||||
configuration, and a replay is refused while an earlier one for the
|
||||
same event and target is still pending or retrying.
|
||||
|
||||
#### DeliveryResult
|
||||
|
||||
The result of a single delivery attempt. Every attempt (including
|
||||
@@ -1231,6 +1248,7 @@ arriving and being stored, they are just not getting anywhere.
|
||||
| `webhooker_deliveries_succeeded_total` | counter | Deliveries that reached `delivered` |
|
||||
| `webhooker_deliveries_failed_total` | counter | Deliveries that failed terminally and will not be retried |
|
||||
| `webhooker_delivery_retries_total` | counter | Deliveries put back into `retrying` |
|
||||
| `webhooker_delivery_replays_total` | counter | Deliveries an operator replayed from the event log. A replay runs the ordinary engine path, so it also moves the attempt, outcome and duration series; this is the only one that separates it from ordinary traffic |
|
||||
| `webhooker_delivery_duration_seconds` | histogram | Wall time of a single dispatched delivery attempt, the same duration the attempt's `DeliveryResult` records |
|
||||
| `webhooker_deliveries_pending` | gauge | Deliveries currently in `pending` |
|
||||
| `webhooker_deliveries_retrying` | gauge | Deliveries currently in `retrying` |
|
||||
@@ -1828,6 +1846,7 @@ abuse limit later; they are tracked as future work.
|
||||
| `POST` | `/source/{id}/edit` | Edit webhook submission |
|
||||
| `POST` | `/source/{id}/delete` | Delete webhook |
|
||||
| `GET` | `/source/{id}/logs` | Webhook event logs |
|
||||
| `POST` | `/source/{id}/deliveries/{deliveryID}/replay` | Replay a finished delivery: creates a new delivery for the same event against the target's current configuration (30 per minute per bucket, then `429`) |
|
||||
| `POST` | `/source/{id}/entrypoints` | Add entrypoint to webhook |
|
||||
| `POST` | `/source/{id}/entrypoints/{entrypointID}/delete` | Delete an entrypoint |
|
||||
| `POST` | `/source/{id}/entrypoints/{entrypointID}/toggle` | Enable or disable an entrypoint |
|
||||
@@ -1912,6 +1931,7 @@ webhooker/
|
||||
│ ├── handlers/
|
||||
│ │ ├── handlers.go # Base handler struct, JSON helpers, template rendering
|
||||
│ │ ├── auth.go # Login, logout handlers
|
||||
│ │ ├── delivery_replay.go # Per-delivery replay: new delivery, current target config
|
||||
│ │ ├── event_log_view.go # Event log projection, byte-capped in SQL
|
||||
│ │ ├── healthcheck.go # Health check handler
|
||||
│ │ ├── index.go # Index page handler
|
||||
|
||||
Reference in New Issue
Block a user