Expose delivery metrics on /metrics (closes #209)
All checks were successful
check / check (push) Successful in 4m13s
All checks were successful
check / check (push) Successful in 4m13s
/metrics carried only the inbound HTTP surface, so a destination failing for an hour, a growing retry backlog and a stuck-open circuit breaker were all invisible: the receive side stays healthy in each case because it is. New internal/metrics registers, on the existing default registry that the go-http-metrics recorder and the promhttp handler already share: - webhooker_events_received_total - webhooker_delivery_attempts_total - webhooker_deliveries_succeeded_total - webhooker_deliveries_failed_total - webhooker_delivery_retries_total - webhooker_delivery_duration_seconds - webhooker_deliveries_pending / _retrying - webhooker_circuit_breakers_open The route mounting is untouched. Every delivery metric carries one label, target_type, whose domain is the four target-type constants; anything outside it collapses to "unknown" so no series can be minted from a UUID. Target ids, event ids and entrypoint ids are deliberately not labels. Instrumentation sits at the points every target type already passes through: processDelivery for the attempt counter and the duration histogram, updateDeliveryStatus for the outcome counters. The queue-depth gauges are counted out of the per-webhook databases by a 30s sampler rather than tracked as deltas, which would need seeding at startup and would drift on any transition that failed to persist. The open-breaker gauge is recounted from the target's breaker registry on every state change.
This commit is contained in:
@@ -107,6 +107,11 @@ func (c *httpCore) withRetry(
|
||||
return
|
||||
}
|
||||
|
||||
// Allow may have moved the breaker to half-open, and the
|
||||
// attempt below may open or close it, so the gauge is
|
||||
// republished on every exit from here.
|
||||
defer c.publishCircuitState(d.Target.Type)
|
||||
|
||||
attemptNum := task.AttemptNum
|
||||
|
||||
res := attempt()
|
||||
@@ -146,6 +151,8 @@ func (c *httpCore) circuitBreakerBlock(
|
||||
return false
|
||||
}
|
||||
|
||||
defer c.publishCircuitState(d.Target.Type)
|
||||
|
||||
remaining := cb.CooldownRemaining()
|
||||
|
||||
c.eng.log.Info(
|
||||
@@ -215,6 +222,28 @@ func (c *httpCore) getCircuitBreaker(
|
||||
return cb
|
||||
}
|
||||
|
||||
// publishCircuitState recounts this core's open breakers and
|
||||
// publishes the gauge. Each core holds the breakers of exactly one
|
||||
// target type, so the recount is over that type's targets alone.
|
||||
// Counting rather than adjusting a delta keeps the gauge honest
|
||||
// however a breaker changed state.
|
||||
func (c *httpCore) publishCircuitState(
|
||||
targetType database.TargetType,
|
||||
) {
|
||||
open := 0
|
||||
|
||||
c.circuitBreakers.Range(func(_, val any) bool {
|
||||
cb, ok := val.(*CircuitBreaker)
|
||||
if ok && cb.State() == CircuitOpen {
|
||||
open++
|
||||
}
|
||||
|
||||
return true
|
||||
})
|
||||
|
||||
c.eng.mx.SetCircuitBreakersOpen(targetType, open)
|
||||
}
|
||||
|
||||
// remainingBackoff returns how long remains of the backoff
|
||||
// window for the last attempt of a recovered retrying
|
||||
// delivery. It implements rescheduler.
|
||||
|
||||
Reference in New Issue
Block a user