Expose delivery metrics on /metrics (closes #209)
All checks were successful
check / check (push) Successful in 4m13s
All checks were successful
check / check (push) Successful in 4m13s
/metrics carried only the inbound HTTP surface, so a destination failing for an hour, a growing retry backlog and a stuck-open circuit breaker were all invisible: the receive side stays healthy in each case because it is. New internal/metrics registers, on the existing default registry that the go-http-metrics recorder and the promhttp handler already share: - webhooker_events_received_total - webhooker_delivery_attempts_total - webhooker_deliveries_succeeded_total - webhooker_deliveries_failed_total - webhooker_delivery_retries_total - webhooker_delivery_duration_seconds - webhooker_deliveries_pending / _retrying - webhooker_circuit_breakers_open The route mounting is untouched. Every delivery metric carries one label, target_type, whose domain is the four target-type constants; anything outside it collapses to "unknown" so no series can be minted from a UUID. Target ids, event ids and entrypoint ids are deliberately not labels. Instrumentation sits at the points every target type already passes through: processDelivery for the attempt counter and the duration histogram, updateDeliveryStatus for the outcome counters. The queue-depth gauges are counted out of the per-webhook databases by a 30s sampler rather than tracked as deltas, which would need seeding at startup and would drift on any transition that failed to persist. The open-breaker gauge is recounted from the target's breaker registry on every state change.
This commit is contained in:
@@ -10,6 +10,7 @@ import (
|
||||
"go.uber.org/fx"
|
||||
"gorm.io/gorm"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/metrics"
|
||||
)
|
||||
|
||||
// ErrExportArchiveWriterEvicted exposes the sentinel returned by
|
||||
@@ -253,6 +254,7 @@ func NewTestEngine(
|
||||
deliveryCh: make(chan Task, deliveryChannelSize),
|
||||
retryCh: make(chan Task, retryChannelSize),
|
||||
workers: workers,
|
||||
mx: metrics.Default(),
|
||||
}
|
||||
e.initTargets(client)
|
||||
|
||||
@@ -267,6 +269,7 @@ func NewTestEngineSmallRetry(
|
||||
e := &Engine{
|
||||
log: log,
|
||||
retryCh: make(chan Task, 1),
|
||||
mx: metrics.Default(),
|
||||
}
|
||||
e.initTargets(nil)
|
||||
|
||||
@@ -289,12 +292,25 @@ func NewTestEngineWithDB(
|
||||
deliveryCh: make(chan Task, deliveryChannelSize),
|
||||
retryCh: make(chan Task, retryChannelSize),
|
||||
workers: workers,
|
||||
mx: metrics.Default(),
|
||||
}
|
||||
e.initTargets(client)
|
||||
|
||||
return e
|
||||
}
|
||||
|
||||
// ExportSetMetrics substitutes the engine's metric set, so a test can
|
||||
// assert on collectors registered on a private registry instead of
|
||||
// the process-wide ones every other test is also moving.
|
||||
func (e *Engine) ExportSetMetrics(mx *metrics.Set) {
|
||||
e.mx = mx
|
||||
}
|
||||
|
||||
// ExportSampleQueueDepths runs one queue depth sample synchronously.
|
||||
func (e *Engine) ExportSampleQueueDepths(ctx context.Context) {
|
||||
e.sampleQueueDepths(ctx)
|
||||
}
|
||||
|
||||
// NewTestCircuitBreaker creates a CircuitBreaker with
|
||||
// custom settings for testing.
|
||||
func NewTestCircuitBreaker(
|
||||
|
||||
Reference in New Issue
Block a user