Expose delivery metrics on /metrics (closes #209)
All checks were successful
check / check (push) Successful in 3m10s
All checks were successful
check / check (push) Successful in 3m10s
/metrics carried only the inbound HTTP surface, so a destination failing for an hour, a growing retry backlog and a stuck-open circuit breaker were all invisible: the receive side stays healthy in each case because it is. New internal/metrics registers, on the existing default registry that the go-http-metrics recorder and the promhttp handler already share: - webhooker_events_received_total - webhooker_delivery_attempts_total - webhooker_deliveries_succeeded_total - webhooker_deliveries_failed_total - webhooker_delivery_retries_total - webhooker_delivery_duration_seconds - webhooker_deliveries_pending / _retrying - webhooker_circuit_breakers_open The route mounting is untouched. Every delivery metric carries one label, target_type, whose domain is the four target-type constants; anything outside it collapses to "unknown" so no series can be minted from a UUID. Target ids, event ids and entrypoint ids are deliberately not labels. An attempt is counted, and its duration observed, only where one was actually dispatched — the target's own result path, which is also where the DeliveryResult is written. A delivery an open circuit breaker refuses sends nothing and records no result row; counting it would climb the attempts counter with no traffic behind it and pull the duration quantiles down for as long as the breaker stayed open, moving the metric the wrong way during the outage it exists to reveal. The log and database targets now time their own work, so their result rows carry a real duration too. The outcome counters move after the status row is written rather than before, so a transition the database rejected is never reported as an outcome that happened. The queue-depth gauges are counted out of the per-webhook databases by a 30s sampler rather than tracked as deltas, which would need seeding at startup and would drift on any transition that failed to persist. They publish an "unknown" series from registration: deliveries queued against a target that has since been deleted resolve to the empty type and are folded there, because a backlog behind a deleted target is precisely the one nobody is watching. The open-breaker gauge is recounted from the target's breaker registry on every state change. The orphaned-retry terminal path takes the target type as an argument rather than attaching the loaded target to the delivery. That path loads the delivery without its target relation on purpose: a populated Delivery.Target makes GORM's SaveBeforeAssociations upsert the whole target row on the status UPDATE, writing the plaintext target config — the credential, for a slack target — into the per-webhook events database. A test asserts that path leaves the targets table empty.
This commit is contained in:
49
README.md
49
README.md
@@ -1134,6 +1134,52 @@ delivery as `retrying` and schedules a retry timer for after the
|
||||
remaining cooldown period. This ensures no deliveries are lost — they're
|
||||
just delayed until the target is healthy again.
|
||||
|
||||
### Metrics
|
||||
|
||||
`/metrics` serves one Prometheus registry behind basic auth (see
|
||||
[Infrastructure Endpoints](#infrastructure-endpoints)). Alongside the
|
||||
inbound HTTP metrics recorded by the middleware, it exposes the
|
||||
delivery pipeline — the part of the service that can be failing while
|
||||
the receive side looks perfectly healthy, because it is: events are
|
||||
arriving and being stored, they are just not getting anywhere.
|
||||
|
||||
| Metric | Type | Meaning |
|
||||
| ------ | ---- | ------- |
|
||||
| `webhooker_events_received_total` | counter | Events received and durably stored. Compare against the delivery counters on one dashboard |
|
||||
| `webhooker_delivery_attempts_total` | counter | Delivery attempts actually dispatched to a target. A delivery an open circuit breaker refused is not one: it is counted as a retry instead |
|
||||
| `webhooker_deliveries_succeeded_total` | counter | Deliveries that reached `delivered` |
|
||||
| `webhooker_deliveries_failed_total` | counter | Deliveries that failed terminally and will not be retried |
|
||||
| `webhooker_delivery_retries_total` | counter | Deliveries put back into `retrying` |
|
||||
| `webhooker_delivery_duration_seconds` | histogram | Wall time of a single dispatched delivery attempt, the same duration the attempt's `DeliveryResult` records |
|
||||
| `webhooker_deliveries_pending` | gauge | Deliveries currently in `pending` |
|
||||
| `webhooker_deliveries_retrying` | gauge | Deliveries currently in `retrying` |
|
||||
| `webhooker_circuit_breakers_open` | gauge | Circuit breakers currently open |
|
||||
|
||||
Every delivery metric carries exactly one label, `target_type`, and
|
||||
cardinality is the whole reason for that restriction. A target type is
|
||||
one of four compile-time constants, so the label domain is bounded by
|
||||
construction; a value outside that set collapses to `unknown` rather
|
||||
than minting a series of its own. Target ids, event ids and entrypoint
|
||||
ids are deliberately not labels: they are UUIDs minted per operator
|
||||
action or per inbound request, a series is never reclaimed once it
|
||||
exists, and labelling by any of them would make `/metrics` a memory
|
||||
leak that grows with traffic.
|
||||
|
||||
The two queue-depth gauges are counted out of the databases by a
|
||||
sampler that runs every 30 seconds for as long as the delivery engine
|
||||
does, rather than tracked as deltas alongside the status transitions: a
|
||||
delta would have to be seeded at startup from rows a previous process
|
||||
wrote, and would drift permanently on any transition that failed to
|
||||
persist.
|
||||
|
||||
Those two gauges also publish an `unknown` series, from startup rather
|
||||
than on first occurrence. Deliveries queued against a target that has
|
||||
since been deleted are counted there: that backlog is the one nobody is
|
||||
watching, so it is the one that must not silently vanish from the
|
||||
gauge. The outcome counters move only after the status change has been
|
||||
written, so a transition the database rejected is never reported as an
|
||||
outcome that happened.
|
||||
|
||||
### Rate Limiting
|
||||
|
||||
Global blanket rate limiting middleware (e.g., a per-IP throttle shared
|
||||
@@ -1759,6 +1805,7 @@ webhooker/
|
||||
│ │ ├── target_log.go # Log target (stdout)
|
||||
│ │ ├── target_config_view.go # Masked target config for templates
|
||||
│ │ ├── archive_sweeper.go # Periodic pruning of idle archives
|
||||
│ │ ├── queue_depth.go # Periodic sampler behind the queue-depth gauges
|
||||
│ │ ├── url_mask.go # Strips credentials from *url.Error
|
||||
│ │ └── ssrf.go # SSRF prevention (IP validation, safe HTTP transport)
|
||||
│ ├── handlers/
|
||||
@@ -1776,6 +1823,8 @@ webhooker/
|
||||
│ │ └── lifecycle.go # Shared stop-hook waiter, bounded by the stop context
|
||||
│ ├── logger/
|
||||
│ │ └── logger.go # slog setup with TTY detection
|
||||
│ ├── metrics/
|
||||
│ │ └── metrics.go # Delivery Prometheus collectors, labelled by target type
|
||||
│ ├── middleware/
|
||||
│ │ ├── middleware.go # Logging, CORS, Auth, Metrics, MetricsAuth, SecurityHeaders, MaxBodySize
|
||||
│ │ ├── csrf.go # CSRF protection middleware (gorilla/csrf)
|
||||
|
||||
Reference in New Issue
Block a user