Expose delivery metrics on /metrics (closes #209)
All checks were successful
check / check (push) Successful in 4m13s

/metrics carried only the inbound HTTP surface, so a destination
failing for an hour, a growing retry backlog and a stuck-open circuit
breaker were all invisible: the receive side stays healthy in each
case because it is.

New internal/metrics registers, on the existing default registry that
the go-http-metrics recorder and the promhttp handler already share:

- webhooker_events_received_total
- webhooker_delivery_attempts_total
- webhooker_deliveries_succeeded_total
- webhooker_deliveries_failed_total
- webhooker_delivery_retries_total
- webhooker_delivery_duration_seconds
- webhooker_deliveries_pending / _retrying
- webhooker_circuit_breakers_open

The route mounting is untouched.

Every delivery metric carries one label, target_type, whose domain is
the four target-type constants; anything outside it collapses to
"unknown" so no series can be minted from a UUID. Target ids, event
ids and entrypoint ids are deliberately not labels.

Instrumentation sits at the points every target type already passes
through: processDelivery for the attempt counter and the duration
histogram, updateDeliveryStatus for the outcome counters. The
queue-depth gauges are counted out of the per-webhook databases by a
30s sampler rather than tracked as deltas, which would need seeding at
startup and would drift on any transition that failed to persist. The
open-breaker gauge is recounted from the target's breaker registry on
every state change.
This commit is contained in:
2026-08-20 04:19:12 +00:00
parent 10c8dd2331
commit b8c8b75e04
11 changed files with 1219 additions and 2 deletions

View File

@@ -107,6 +107,11 @@ func (c *httpCore) withRetry(
return
}
// Allow may have moved the breaker to half-open, and the
// attempt below may open or close it, so the gauge is
// republished on every exit from here.
defer c.publishCircuitState(d.Target.Type)
attemptNum := task.AttemptNum
res := attempt()
@@ -146,6 +151,8 @@ func (c *httpCore) circuitBreakerBlock(
return false
}
defer c.publishCircuitState(d.Target.Type)
remaining := cb.CooldownRemaining()
c.eng.log.Info(
@@ -215,6 +222,28 @@ func (c *httpCore) getCircuitBreaker(
return cb
}
// publishCircuitState recounts this core's open breakers and
// publishes the gauge. Each core holds the breakers of exactly one
// target type, so the recount is over that type's targets alone.
// Counting rather than adjusting a delta keeps the gauge honest
// however a breaker changed state.
func (c *httpCore) publishCircuitState(
targetType database.TargetType,
) {
open := 0
c.circuitBreakers.Range(func(_, val any) bool {
cb, ok := val.(*CircuitBreaker)
if ok && cb.State() == CircuitOpen {
open++
}
return true
})
c.eng.mx.SetCircuitBreakersOpen(targetType, open)
}
// remainingBackoff returns how long remains of the backoff
// window for the last attempt of a recovered retrying
// delivery. It implements rescheduler.