No delivery metrics: /metrics cannot tell you whether the product is doing its job #209

New Issue

clawbot · 2026-08-20T05:48:28+02:00

clawbot commented

2026-08-20 05:48:28 +02:00

/metrics exposes only http_request_duration_seconds, http_requests_inflight and http_response_size_bytes — the inbound HTTP surface. There is nothing about delivery, which is the entire purpose of the service.

You cannot alert on a destination that has been failing for an hour, on a retry backlog that is growing, or on a circuit breaker that is stuck open. The receive side looks perfectly healthy in all three cases, because it is: events are arriving and being stored. They are just not getting anywhere.

Definition of done, all labelled by target type (and by target id only if cardinality is bounded — do not label by event or entrypoint):

counter: deliveries attempted, succeeded, failed-terminal, retried
histogram: delivery duration
gauge: deliveries currently in pending and in retrying
gauge: circuit breakers currently open
counter: events received, so receive and deliver can be compared on one dashboard
metrics registered through the existing registry and served on the existing authenticated /metrics route
a test asserts the counters move across a delivery that succeeds and one that exhausts its retries

`/metrics` exposes only `http_request_duration_seconds`, `http_requests_inflight` and `http_response_size_bytes` — the inbound HTTP surface. There is nothing about delivery, which is the entire purpose of the service. You cannot alert on a destination that has been failing for an hour, on a retry backlog that is growing, or on a circuit breaker that is stuck open. The receive side looks perfectly healthy in all three cases, because it is: events are arriving and being stored. They are just not getting anywhere. Definition of done, all labelled by target type (and by target id only if cardinality is bounded — do not label by event or entrypoint): - counter: deliveries attempted, succeeded, failed-terminal, retried - histogram: delivery duration - gauge: deliveries currently in `pending` and in `retrying` - gauge: circuit breakers currently open - counter: events received, so receive and deliver can be compared on one dashboard - metrics registered through the existing registry and served on the existing authenticated `/metrics` route - a test asserts the counters move across a delivery that succeeds and one that exhausts its retries

clawbot added this to the 1.0.0 milestone 2026-08-20 05:48:28 +02:00

clawbot referenced this issue

2026-08-20 05:49:42 +02:00

1.0/mvp #33

clawbot referenced this issue from a commit

2026-08-20 06:26:45 +02:00

Expose delivery metrics on /metrics (closes #209)

clawbot referenced a pull request that will close this issue

2026-08-20 06:27:17 +02:00

Expose delivery metrics on /metrics (closes #209) #224

clawbot referenced this issue

2026-08-20 06:29:20 +02:00