Label HTTP metrics with the chi route pattern (closes #254)
All checks were successful
check / check (push) Successful in 3m1s
All checks were successful
check / check (push) Successful in 3m1s
The metrics recorder labelled its `handler` dimension with the concrete request path, so every distinct /webhook/<uuid> minted a permanent label set that nothing ever evicted. Measured on this branch's parent: a scrape went from 106 series and 12 KB to 78,132 series and 10.7 MB after 3,000 unauthenticated POSTs to invented entrypoint UUIDs, and stayed there. It also published those UUIDs -- the receiver's only credential -- verbatim in the scrape. The label now comes from chi's route pattern. It cannot be supplied as go-http-metrics' handler id: the recorder is global middleware, so it is entered before chi has matched anything, and the library fixes the id up front. What the library does pass through unchanged is the request context, on every recorder call, and the duration and size observations happen after the wrapped handler returns -- the same point accessLogURL already reads the pattern from. So a recorder decorator rewrites the id there instead. std.Handler and its response-writer interceptor are untouched, so status and size capture are unchanged. Recording after the whole chain returns is what makes this hold for requests the route-level receiver limiter rejects, which were the majority of the leaked series: chi has matched the route before the limiter runs, so a 429 carries the pattern like any other response. A path matching no route carries the existing unmatchedRoute sentinel, the same fixed value the access log uses. http_requests_inflight cannot carry a pattern -- it is incremented before routing and decremented after, so a route-derived label would increment one series and decrement another and leave the gauge permanently wrong. It gets a fixed aggregate label instead. Verified against a live instance: 6,000 distinct receiver paths and 250 unmatched paths across two floods left the scrape at 206 series and 21 KB, flat between floods, with no UUID anywhere in the output and all 4,800 429s on the single pattern.
This commit is contained in:
@@ -13,9 +13,6 @@ import (
|
||||
"github.com/go-chi/chi"
|
||||
"github.com/go-chi/chi/middleware"
|
||||
"github.com/go-chi/cors"
|
||||
metrics "github.com/slok/go-http-metrics/metrics/prometheus"
|
||||
ghmm "github.com/slok/go-http-metrics/middleware"
|
||||
"github.com/slok/go-http-metrics/middleware/std"
|
||||
"go.uber.org/fx"
|
||||
"sneak.berlin/go/webhooker/internal/config"
|
||||
"sneak.berlin/go/webhooker/internal/globals"
|
||||
@@ -29,10 +26,15 @@ const (
|
||||
// preflight response can be cached.
|
||||
corsMaxAge = 300
|
||||
|
||||
// unmatchedRoute is logged in the access log's url field when a
|
||||
// redirected or rejected request matched no route pattern at
|
||||
// all. Every byte of such a path is client-chosen, so none of it
|
||||
// is logged.
|
||||
// unmatchedRoute stands in for a request that matched no route
|
||||
// pattern at all. Every byte of such a path is client-chosen, so
|
||||
// none of it is kept.
|
||||
//
|
||||
// It is the access log's url field on a redirected or rejected
|
||||
// request, and it is the metrics `handler` label on the same
|
||||
// request; see metrics.go. Both surfaces are written once per
|
||||
// request from a path the client picks, so both have to collapse
|
||||
// the unmatched case into one fixed value.
|
||||
unmatchedRoute = "(unmatched)"
|
||||
|
||||
// redactedQuery stands in for the query string on the access log
|
||||
@@ -438,17 +440,6 @@ func (s *Middleware) RequireAuth() func(http.Handler) http.Handler {
|
||||
}
|
||||
}
|
||||
|
||||
// Metrics returns middleware that records Prometheus HTTP metrics.
|
||||
func (s *Middleware) Metrics() func(http.Handler) http.Handler {
|
||||
mdlw := ghmm.New(ghmm.Config{
|
||||
Recorder: metrics.NewRecorder(metrics.Config{}),
|
||||
})
|
||||
|
||||
return func(next http.Handler) http.Handler {
|
||||
return std.Handler("", mdlw, next)
|
||||
}
|
||||
}
|
||||
|
||||
// MetricsAuth returns middleware that protects metrics endpoints
|
||||
// with basic auth.
|
||||
func (s *Middleware) MetricsAuth() func(http.Handler) http.Handler {
|
||||
|
||||
Reference in New Issue
Block a user