Leave a delivery cut off by a stop for the next start (closes #527)
check / check (push) Waiting to run
check / check (push) Waiting to run
Stopping the engine cancels the context every http and slack request is sent with. An attempt cut off that way was recorded as failed, which for a target with no retries, or on a last retry, ended the delivery failed and nothing sent it again. The shared retry core now records nothing for a failed attempt when the engine's context is cancelled: no result, no circuit breaker outcome, no status change, no count in the attempt metrics. The delivery stays pending or retrying and startup recovery sends it again. A target's own timeout is still a failed attempt. The README says what happens to an attempt in flight at a stop and no longer claims every attempt is recorded. Model: opus-5-5
This commit was merged in pull request #532.
This commit is contained in:
@@ -1317,20 +1317,22 @@ webhooker solves this by acting as a durable intermediary:
|
||||
nothing is lost if downstream targets are unavailable.
|
||||
|
||||
2. **Guaranteed delivery** — Events are queued for delivery to each configured
|
||||
target. Failed deliveries are retried with configurable backoff. Every
|
||||
delivery attempt is logged with status codes, response bodies, and timing.
|
||||
target. Failed deliveries are retried with configurable backoff. Delivery
|
||||
attempts are logged with status codes, response bodies, and timing.
|
||||
|
||||
**That guarantee is at-least-once, not exactly-once.** When a send reaches
|
||||
its target but the write recording that outcome fails, the delivery is
|
||||
deliberately left in a recoverable state rather than marked done — losing a
|
||||
delivery is the worse failure — so the pending sweep picks it up about
|
||||
fifteen minutes later, or the next restart does, and the target receives a
|
||||
payload it already got. webhooker adds no delivery identifier of its own to
|
||||
an outbound request, so **make your receiver idempotent** against whatever
|
||||
the payload itself carries.
|
||||
payload it already got. A send still in flight when the service stops is
|
||||
left the same way: its attempt is not recorded, and the next start sends it
|
||||
again (see Shutdown). webhooker adds no delivery identifier of its own to an
|
||||
outbound request, so **make your receiver idempotent** against whatever the
|
||||
payload itself carries.
|
||||
|
||||
3. **Observability** — Full request/response logging for every webhook received
|
||||
and every delivery attempted. Prometheus metrics expose volume, latency, and
|
||||
and for delivery attempts. Prometheus metrics expose volume, latency, and
|
||||
error rates. The web UI provides real-time visibility into event flow.
|
||||
|
||||
4. **Fan-out** — A single incoming webhook can be delivered to multiple targets
|
||||
@@ -1768,7 +1770,7 @@ is what bounds it.
|
||||
#### DeliveryResult
|
||||
|
||||
The result of a single delivery attempt. Every attempt (including retries) is
|
||||
individually logged for full observability.
|
||||
individually logged, except one cut off by a stop (see Shutdown).
|
||||
|
||||
| Field | Type | Description |
|
||||
| --------------- | ------- | ----------------------------------------- |
|
||||
@@ -2206,12 +2208,16 @@ This means:
|
||||
buffer rather than spawning more goroutines.
|
||||
- **Independent results** — each worker records its own delivery result in the
|
||||
per-webhook database without coordination.
|
||||
- **Graceful shutdown** — cancel the context, workers finish their current task
|
||||
and exit. The stop hook waits for the pool via `lifecycle.WaitForShutdown`,
|
||||
which bounds that wait by fx's stop timeout rather than blocking forever on a
|
||||
wedged worker. On timeout it logs at `ERROR` and returns an error, and the
|
||||
goroutines that did not finish are still running — an unclean shutdown is
|
||||
reported rather than hidden.
|
||||
- **Graceful shutdown** — the stop hook cancels the workers' context. A worker
|
||||
may still take tasks already queued before it exits. An `http` or `slack`
|
||||
request in flight is cut off, and one started after the cancel fails at once.
|
||||
Neither is recorded or counted in the metrics, and the next start sends the
|
||||
delivery again (see Shutdown). A `database` or `log` task runs as normal. The
|
||||
stop hook waits for the pool via `lifecycle.WaitForShutdown`, which bounds
|
||||
that wait by fx's stop timeout rather than blocking forever on a wedged
|
||||
worker. On timeout it logs at `ERROR` and returns an error, and the goroutines
|
||||
that did not finish are still running — an unclean shutdown is reported rather
|
||||
than hidden.
|
||||
|
||||
**Recovery paths:**
|
||||
|
||||
@@ -2317,7 +2323,7 @@ getting anywhere.
|
||||
| Metric | Type | Meaning |
|
||||
| -------------------------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `webhooker_events_received_total` | counter | Events received and durably stored. Compare against the delivery counters on one dashboard |
|
||||
| `webhooker_delivery_attempts_total` | counter | Delivery attempts actually dispatched to a target. A delivery a circuit breaker refused is not one: it is counted as a retry instead, but only when the refusal moves it into `retrying` |
|
||||
| `webhooker_delivery_attempts_total` | counter | Delivery attempts actually dispatched to a target. A delivery a circuit breaker refused is not one: it is counted as a retry instead, but only when the refusal moves it into `retrying`. An attempt cut off by a stop is not counted (see Shutdown) |
|
||||
| `webhooker_deliveries_succeeded_total` | counter | Deliveries that reached `delivered` |
|
||||
| `webhooker_deliveries_failed_total` | counter | Deliveries that failed terminally and will not be retried |
|
||||
| `webhooker_delivery_retries_total` | counter | Deliveries put back into `retrying` |
|
||||
@@ -3284,7 +3290,8 @@ stop-hook log:
|
||||
3. `server` — the HTTP drain, bounded by `server.ShutdownTimeout` (**3
|
||||
seconds**) and by what the hooks before it left, then a Sentry flush if
|
||||
`SENTRY_DSN` is set
|
||||
4. `delivery.Engine` — waits for its workers, then closes the archive databases
|
||||
4. `delivery.Engine` — cuts off any `http` or `slack` request in flight, waits
|
||||
for its workers, then closes the archive databases
|
||||
5. `healthcheck`
|
||||
6. `WebhookDBManager`
|
||||
7. the database close
|
||||
@@ -3295,6 +3302,12 @@ by the stop context, so a wedge there consumes the 5 seconds before the HTTP
|
||||
server hook is ever entered. The hooks after the server are microsecond-scale in
|
||||
normal operation.
|
||||
|
||||
The delivery engine cancels a delivery request in flight. That attempt is not
|
||||
recorded or counted in the metrics, and the delivery keeps its `pending` or
|
||||
`retrying` status, so startup recovery sends it again on the next start (see
|
||||
Recovery paths under Bounded Worker Pool). If the receiver already had it, it
|
||||
receives it twice: the at-least-once duplicate described under Rationale.
|
||||
|
||||
The HTTP drain budget is deliberately **shorter** than the sequence budget. Were
|
||||
the two equal, a drain that used its whole budget would exhaust the sequence
|
||||
budget at the instant it finished, and every later hook — the delivery engine,
|
||||
|
||||
Reference in New Issue
Block a user