Expose delivery metrics on /metrics (closes #209)
All checks were successful
check / check (push) Successful in 4m13s
All checks were successful
check / check (push) Successful in 4m13s
/metrics carried only the inbound HTTP surface, so a destination failing for an hour, a growing retry backlog and a stuck-open circuit breaker were all invisible: the receive side stays healthy in each case because it is. New internal/metrics registers, on the existing default registry that the go-http-metrics recorder and the promhttp handler already share: - webhooker_events_received_total - webhooker_delivery_attempts_total - webhooker_deliveries_succeeded_total - webhooker_deliveries_failed_total - webhooker_delivery_retries_total - webhooker_delivery_duration_seconds - webhooker_deliveries_pending / _retrying - webhooker_circuit_breakers_open The route mounting is untouched. Every delivery metric carries one label, target_type, whose domain is the four target-type constants; anything outside it collapses to "unknown" so no series can be minted from a UUID. Target ids, event ids and entrypoint ids are deliberately not labels. Instrumentation sits at the points every target type already passes through: processDelivery for the attempt counter and the duration histogram, updateDeliveryStatus for the outcome counters. The queue-depth gauges are counted out of the per-webhook databases by a 30s sampler rather than tracked as deltas, which would need seeding at startup and would drift on any transition that failed to persist. The open-breaker gauge is recounted from the target's breaker registry on every state change.
This commit is contained in:
@@ -18,6 +18,7 @@ import (
|
||||
"sneak.berlin/go/webhooker/internal/globals"
|
||||
"sneak.berlin/go/webhooker/internal/healthcheck"
|
||||
"sneak.berlin/go/webhooker/internal/logger"
|
||||
"sneak.berlin/go/webhooker/internal/metrics"
|
||||
"sneak.berlin/go/webhooker/internal/middleware"
|
||||
"sneak.berlin/go/webhooker/internal/session"
|
||||
"sneak.berlin/go/webhooker/templates"
|
||||
@@ -73,6 +74,7 @@ type Handlers struct {
|
||||
mw *middleware.Middleware
|
||||
notifier delivery.Notifier
|
||||
evictor delivery.WebhookEvictor
|
||||
mx *metrics.Set
|
||||
templates map[string]*template.Template
|
||||
|
||||
// dummyVerifications counts the equivalent-cost verifications
|
||||
@@ -114,6 +116,7 @@ func New(
|
||||
s.mw = params.Middleware
|
||||
s.notifier = params.Notifier
|
||||
s.evictor = params.Evictor
|
||||
s.mx = metrics.Default()
|
||||
|
||||
// Parse all page templates once at startup
|
||||
s.templates = map[string]*template.Template{
|
||||
|
||||
@@ -217,6 +217,11 @@ func (h *Handlers) createAndDeliverEvent(
|
||||
return
|
||||
}
|
||||
|
||||
// Counted here, after the commit: an event is received once it
|
||||
// is durably stored, which is what the delivery counters are
|
||||
// compared against on a dashboard.
|
||||
h.mx.EventReceived()
|
||||
|
||||
h.finishWebhookResponse(w, event, entrypoint, tasks)
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user