All checks were successful
check / check (push) Successful in 3m1s
The metrics recorder labelled its `handler` dimension with the concrete request path, so every distinct /webhook/<uuid> minted a permanent label set that nothing ever evicted. Measured on this branch's parent: a scrape went from 106 series and 12 KB to 78,132 series and 10.7 MB after 3,000 unauthenticated POSTs to invented entrypoint UUIDs, and stayed there. It also published those UUIDs -- the receiver's only credential -- verbatim in the scrape. The label now comes from chi's route pattern. It cannot be supplied as go-http-metrics' handler id: the recorder is global middleware, so it is entered before chi has matched anything, and the library fixes the id up front. What the library does pass through unchanged is the request context, on every recorder call, and the duration and size observations happen after the wrapped handler returns -- the same point accessLogURL already reads the pattern from. So a recorder decorator rewrites the id there instead. std.Handler and its response-writer interceptor are untouched, so status and size capture are unchanged. Recording after the whole chain returns is what makes this hold for requests the route-level receiver limiter rejects, which were the majority of the leaked series: chi has matched the route before the limiter runs, so a 429 carries the pattern like any other response. A path matching no route carries the existing unmatchedRoute sentinel, the same fixed value the access log uses. http_requests_inflight cannot carry a pattern -- it is incremented before routing and decremented after, so a route-derived label would increment one series and decrement another and leave the gauge permanently wrong. It gets a fixed aggregate label instead. Verified against a live instance: 6,000 distinct receiver paths and 250 unmatched paths across two floods left the scrape at 206 series and 21 KB, flat between floods, with no UUID anywhere in the output and all 4,800 429s on the single pattern.
136 lines
4.9 KiB
Go
136 lines
4.9 KiB
Go
package middleware
|
|
|
|
import (
|
|
"context"
|
|
"net/http"
|
|
"time"
|
|
|
|
"github.com/go-chi/chi"
|
|
httpmetrics "github.com/slok/go-http-metrics/metrics"
|
|
prommetrics "github.com/slok/go-http-metrics/metrics/prometheus"
|
|
ghmm "github.com/slok/go-http-metrics/middleware"
|
|
"github.com/slok/go-http-metrics/middleware/std"
|
|
)
|
|
|
|
// inflightHandler is the fixed `handler` label on
|
|
// http_requests_inflight, the one HTTP metric here that cannot carry
|
|
// a route pattern.
|
|
//
|
|
// The gauge is incremented before the wrapped handler runs and
|
|
// decremented after it returns, and the pattern only exists between
|
|
// those two moments. Deriving the label from the route would
|
|
// therefore increment one series and decrement another, leaving every
|
|
// pattern permanently off by the number of requests it served — a
|
|
// broken gauge, on top of the per-path cardinality this file exists
|
|
// to remove. So the gauge is deliberately aggregate: one series,
|
|
// counting the requests in flight across the whole service.
|
|
const inflightHandler = "(all)"
|
|
|
|
// routePatternID is the `handler` label for a request: the chi route
|
|
// pattern, never the concrete path.
|
|
//
|
|
// The pattern is what bounds the label's domain to the routes the
|
|
// service registers. The path does not bound it at all — every byte
|
|
// after /webhook/ is client-chosen, so labelling by path lets any
|
|
// unauthenticated client mint permanent series at will, and publishes
|
|
// the entrypoint UUID (the receiver's only credential) in the scrape
|
|
// while doing it.
|
|
//
|
|
// chi populates the route context during routeHTTP, so this is only
|
|
// valid once routing has run. Every caller below is on the recording
|
|
// side of the middleware, which go-http-metrics defers until after
|
|
// the wrapped handler returns.
|
|
func routePatternID(ctx context.Context) string {
|
|
if rc := chi.RouteContext(ctx); rc != nil {
|
|
if pattern := rc.RoutePattern(); pattern != "" {
|
|
return pattern
|
|
}
|
|
}
|
|
|
|
return unmatchedRoute
|
|
}
|
|
|
|
// routePatternRecorder wraps a go-http-metrics recorder and replaces
|
|
// the handler id on every observation with the request's route
|
|
// pattern.
|
|
//
|
|
// This is the seam that makes the pattern usable at all. The metrics
|
|
// middleware is global (see Server.setupGlobalMiddleware), so it is
|
|
// entered before chi has matched anything, and go-http-metrics fixes
|
|
// its handler id up front — passing the pattern in as that id is not
|
|
// possible, and leaving the id empty makes the library substitute the
|
|
// raw URL path, which is the defect. What the library does hand over
|
|
// is the request context, unchanged, on each recorder call; that
|
|
// context carries the same *chi.Context pointer routing mutates in
|
|
// place, and the duration and size calls happen after the wrapped
|
|
// handler has returned. Reading the pattern there is what the access
|
|
// log already does in accessLogURL.
|
|
//
|
|
// Recording after the whole chain returns is also what makes this
|
|
// hold for requests the route-level receiver rate limiter rejects.
|
|
// Those never reach a handler, but chi has already matched the route
|
|
// by the time the limiter runs, so their 429s land on the pattern
|
|
// like any other response.
|
|
type routePatternRecorder struct {
|
|
inner httpmetrics.Recorder
|
|
}
|
|
|
|
func (r routePatternRecorder) ObserveHTTPRequestDuration(
|
|
ctx context.Context,
|
|
props httpmetrics.HTTPReqProperties,
|
|
duration time.Duration,
|
|
) {
|
|
props.ID = routePatternID(ctx)
|
|
r.inner.ObserveHTTPRequestDuration(ctx, props, duration)
|
|
}
|
|
|
|
func (r routePatternRecorder) ObserveHTTPResponseSize(
|
|
ctx context.Context,
|
|
props httpmetrics.HTTPReqProperties,
|
|
sizeBytes int64,
|
|
) {
|
|
props.ID = routePatternID(ctx)
|
|
r.inner.ObserveHTTPResponseSize(ctx, props, sizeBytes)
|
|
}
|
|
|
|
func (r routePatternRecorder) AddInflightRequests(
|
|
ctx context.Context,
|
|
props httpmetrics.HTTPProperties,
|
|
quantity int,
|
|
) {
|
|
props.ID = inflightHandler
|
|
r.inner.AddInflightRequests(ctx, props, quantity)
|
|
}
|
|
|
|
var _ httpmetrics.Recorder = routePatternRecorder{}
|
|
|
|
// Metrics returns middleware that records Prometheus HTTP metrics on
|
|
// the default registry, which is the one the /metrics route gathers.
|
|
func (s *Middleware) Metrics() func(http.Handler) http.Handler {
|
|
return metricsMiddleware(
|
|
prommetrics.NewRecorder(prommetrics.Config{}),
|
|
)
|
|
}
|
|
|
|
// metricsMiddleware builds the recording middleware against a given
|
|
// recorder, so tests can gather from a registry of their own instead
|
|
// of the process-wide default.
|
|
func metricsMiddleware(
|
|
rec httpmetrics.Recorder,
|
|
) func(http.Handler) http.Handler {
|
|
mdlw := ghmm.New(ghmm.Config{
|
|
Recorder: routePatternRecorder{inner: rec},
|
|
})
|
|
|
|
return func(next http.Handler) http.Handler {
|
|
// The handler id is unmatchedRoute rather than "" so that
|
|
// the client-chosen URL path never enters the metrics
|
|
// pipeline at all: an empty id is the library's signal to
|
|
// substitute it. routePatternRecorder overwrites this value
|
|
// on every observation, so it is reachable only if that
|
|
// decorator is removed — in which case the metrics collapse
|
|
// to one series instead of leaking again.
|
|
return std.Handler(unmatchedRoute, mdlw, next)
|
|
}
|
|
}
|