Label HTTP metrics with the chi route pattern (closes #254)
All checks were successful
check / check (push) Successful in 3m1s

The metrics recorder labelled its `handler` dimension with the
concrete request path, so every distinct /webhook/<uuid> minted a
permanent label set that nothing ever evicted. Measured on this
branch's parent: a scrape went from 106 series and 12 KB to 78,132
series and 10.7 MB after 3,000 unauthenticated POSTs to invented
entrypoint UUIDs, and stayed there. It also published those UUIDs --
the receiver's only credential -- verbatim in the scrape.

The label now comes from chi's route pattern. It cannot be supplied
as go-http-metrics' handler id: the recorder is global middleware, so
it is entered before chi has matched anything, and the library fixes
the id up front. What the library does pass through unchanged is the
request context, on every recorder call, and the duration and size
observations happen after the wrapped handler returns -- the same
point accessLogURL already reads the pattern from. So a recorder
decorator rewrites the id there instead. std.Handler and its
response-writer interceptor are untouched, so status and size capture
are unchanged.

Recording after the whole chain returns is what makes this hold for
requests the route-level receiver limiter rejects, which were the
majority of the leaked series: chi has matched the route before the
limiter runs, so a 429 carries the pattern like any other response.

A path matching no route carries the existing unmatchedRoute
sentinel, the same fixed value the access log uses.

http_requests_inflight cannot carry a pattern -- it is incremented
before routing and decremented after, so a route-derived label would
increment one series and decrement another and leave the gauge
permanently wrong. It gets a fixed aggregate label instead.

Verified against a live instance: 6,000 distinct receiver paths and
250 unmatched paths across two floods left the scrape at 206 series
and 21 KB, flat between floods, with no UUID anywhere in the output
and all 4,800 429s on the single pattern.
This commit is contained in:
2026-08-23 23:19:15 +00:00
parent 89f3b984d2
commit 6e26885349
4 changed files with 598 additions and 18 deletions

View File

@@ -13,9 +13,6 @@ import (
"github.com/go-chi/chi"
"github.com/go-chi/chi/middleware"
"github.com/go-chi/cors"
metrics "github.com/slok/go-http-metrics/metrics/prometheus"
ghmm "github.com/slok/go-http-metrics/middleware"
"github.com/slok/go-http-metrics/middleware/std"
"go.uber.org/fx"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/globals"
@@ -29,10 +26,15 @@ const (
// preflight response can be cached.
corsMaxAge = 300
// unmatchedRoute is logged in the access log's url field when a
// redirected or rejected request matched no route pattern at
// all. Every byte of such a path is client-chosen, so none of it
// is logged.
// unmatchedRoute stands in for a request that matched no route
// pattern at all. Every byte of such a path is client-chosen, so
// none of it is kept.
//
// It is the access log's url field on a redirected or rejected
// request, and it is the metrics `handler` label on the same
// request; see metrics.go. Both surfaces are written once per
// request from a path the client picks, so both have to collapse
// the unmatched case into one fixed value.
unmatchedRoute = "(unmatched)"
// redactedQuery stands in for the query string on the access log
@@ -438,17 +440,6 @@ func (s *Middleware) RequireAuth() func(http.Handler) http.Handler {
}
}
// Metrics returns middleware that records Prometheus HTTP metrics.
func (s *Middleware) Metrics() func(http.Handler) http.Handler {
mdlw := ghmm.New(ghmm.Config{
Recorder: metrics.NewRecorder(metrics.Config{}),
})
return func(next http.Handler) http.Handler {
return std.Handler("", mdlw, next)
}
}
// MetricsAuth returns middleware that protects metrics endpoints
// with basic auth.
func (s *Middleware) MetricsAuth() func(http.Handler) http.Handler {