Expose delivery metrics on /metrics (closes #209)
All checks were successful
check / check (push) Successful in 4m13s

/metrics carried only the inbound HTTP surface, so a destination
failing for an hour, a growing retry backlog and a stuck-open circuit
breaker were all invisible: the receive side stays healthy in each
case because it is.

New internal/metrics registers, on the existing default registry that
the go-http-metrics recorder and the promhttp handler already share:

- webhooker_events_received_total
- webhooker_delivery_attempts_total
- webhooker_deliveries_succeeded_total
- webhooker_deliveries_failed_total
- webhooker_delivery_retries_total
- webhooker_delivery_duration_seconds
- webhooker_deliveries_pending / _retrying
- webhooker_circuit_breakers_open

The route mounting is untouched.

Every delivery metric carries one label, target_type, whose domain is
the four target-type constants; anything outside it collapses to
"unknown" so no series can be minted from a UUID. Target ids, event
ids and entrypoint ids are deliberately not labels.

Instrumentation sits at the points every target type already passes
through: processDelivery for the attempt counter and the duration
histogram, updateDeliveryStatus for the outcome counters. The
queue-depth gauges are counted out of the per-webhook databases by a
30s sampler rather than tracked as deltas, which would need seeding at
startup and would drift on any transition that failed to persist. The
open-breaker gauge is recounted from the target's breaker registry on
every state change.
This commit is contained in:
2026-08-20 04:19:12 +00:00
parent 10c8dd2331
commit b8c8b75e04
11 changed files with 1219 additions and 2 deletions

View File

@@ -10,6 +10,7 @@ import (
"go.uber.org/fx"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/metrics"
)
// ErrExportArchiveWriterEvicted exposes the sentinel returned by
@@ -253,6 +254,7 @@ func NewTestEngine(
deliveryCh: make(chan Task, deliveryChannelSize),
retryCh: make(chan Task, retryChannelSize),
workers: workers,
mx: metrics.Default(),
}
e.initTargets(client)
@@ -267,6 +269,7 @@ func NewTestEngineSmallRetry(
e := &Engine{
log: log,
retryCh: make(chan Task, 1),
mx: metrics.Default(),
}
e.initTargets(nil)
@@ -289,12 +292,25 @@ func NewTestEngineWithDB(
deliveryCh: make(chan Task, deliveryChannelSize),
retryCh: make(chan Task, retryChannelSize),
workers: workers,
mx: metrics.Default(),
}
e.initTargets(client)
return e
}
// ExportSetMetrics substitutes the engine's metric set, so a test can
// assert on collectors registered on a private registry instead of
// the process-wide ones every other test is also moving.
func (e *Engine) ExportSetMetrics(mx *metrics.Set) {
e.mx = mx
}
// ExportSampleQueueDepths runs one queue depth sample synchronously.
func (e *Engine) ExportSampleQueueDepths(ctx context.Context) {
e.sampleQueueDepths(ctx)
}
// NewTestCircuitBreaker creates a CircuitBreaker with
// custom settings for testing.
func NewTestCircuitBreaker(