All checks were successful
check / check (push) Successful in 3m6s
The `method` label was recorded as `r.Method` verbatim. net/http
accepts any RFC 9110 token as a method and passes it through, so the
label was bounded at nothing: 300 requests to `/` carrying random
12-character method tokens took a live scrape from 81 lines to 7,606,
unauthenticated and on a route with no rate limiter. This is the same
remote memory-exhaustion vector the handler label carried, reached
through a second dimension.
The fix goes in the same recorder seam that bounds the handler label,
which is renamed to reflect that it now bounds both. A method the
router can route is kept verbatim, so real methods stay
distinguishable; anything else carries the existing `(unmatched)`
sentinel, deliberately the same spelling rather than a second one for
the same idea. The bound is ten values.
The retained set is restated against the net/http constants because
chi's own methodMap is unexported. A token outside it can only ever
produce chi's 405, so folding those together loses nothing a scrape
could have used.
README's Metrics section gains the inbound HTTP metrics, the bound on
each of their labels, and the aggregate
`http_requests_inflight{handler="(all)"}` semantics.
183 lines
6.8 KiB
Go
183 lines
6.8 KiB
Go
package middleware
|
|
|
|
import (
|
|
"context"
|
|
"net/http"
|
|
"time"
|
|
|
|
"github.com/go-chi/chi"
|
|
httpmetrics "github.com/slok/go-http-metrics/metrics"
|
|
prommetrics "github.com/slok/go-http-metrics/metrics/prometheus"
|
|
ghmm "github.com/slok/go-http-metrics/middleware"
|
|
"github.com/slok/go-http-metrics/middleware/std"
|
|
)
|
|
|
|
// inflightHandler is the fixed `handler` label on
|
|
// http_requests_inflight, the one HTTP metric here that cannot carry
|
|
// a route pattern.
|
|
//
|
|
// The gauge is incremented before the wrapped handler runs and
|
|
// decremented after it returns, and the pattern only exists between
|
|
// those two moments. Deriving the label from the route would
|
|
// therefore increment one series and decrement another, leaving every
|
|
// pattern permanently off by the number of requests it served — a
|
|
// broken gauge, on top of the per-path cardinality this file exists
|
|
// to remove. So the gauge is deliberately aggregate: one series,
|
|
// counting the requests in flight across the whole service.
|
|
const inflightHandler = "(all)"
|
|
|
|
// unmatchedMethod is the `method` label for a request whose method
|
|
// the router can never route.
|
|
//
|
|
// It is deliberately the same sentinel as unmatchedRoute rather than
|
|
// a spelling of its own: both stand for a client-chosen token that
|
|
// matched nothing this service registers, and giving one idea two
|
|
// spellings would read in a scrape as two different unmatched states.
|
|
const unmatchedMethod = unmatchedRoute
|
|
|
|
// routePatternID is the `handler` label for a request: the chi route
|
|
// pattern, never the concrete path.
|
|
//
|
|
// The pattern is what bounds the label's domain to the routes the
|
|
// service registers. The path does not bound it at all — every byte
|
|
// after /webhook/ is client-chosen, so labelling by path lets any
|
|
// unauthenticated client mint permanent series at will, and publishes
|
|
// the entrypoint UUID (the receiver's only credential) in the scrape
|
|
// while doing it.
|
|
//
|
|
// chi populates the route context during routeHTTP, so this is only
|
|
// valid once routing has run. Every caller below is on the recording
|
|
// side of the middleware, which go-http-metrics defers until after
|
|
// the wrapped handler returns.
|
|
func routePatternID(ctx context.Context) string {
|
|
if rc := chi.RouteContext(ctx); rc != nil {
|
|
if pattern := rc.RoutePattern(); pattern != "" {
|
|
return pattern
|
|
}
|
|
}
|
|
|
|
return unmatchedRoute
|
|
}
|
|
|
|
// methodID is the `method` label for a request: the request method
|
|
// when the router can route it, and the unmatched sentinel otherwise.
|
|
//
|
|
// net/http accepts any RFC 9110 token as a method and hands it
|
|
// through verbatim, so the raw method is client-chosen bytes and
|
|
// bounds the label at nothing — the same unauthenticated
|
|
// series-minting the handler label carried, reached through a second
|
|
// dimension. What bounds it is the set chi's router will match a
|
|
// route for: its methodMap, which is unexported, so it is restated
|
|
// here against the net/http constants it is built from. A token
|
|
// outside that set can only ever produce chi's 405, so folding every
|
|
// one of them onto a single series loses no information a scrape
|
|
// could have used, while the nine methods that can reach a handler
|
|
// stay distinguishable.
|
|
//
|
|
// chi.RegisterMethod would extend the router's set at runtime; this
|
|
// service never calls it, and a caller that started to would have to
|
|
// extend this switch with it.
|
|
func methodID(method string) string {
|
|
switch method {
|
|
case http.MethodConnect,
|
|
http.MethodDelete,
|
|
http.MethodGet,
|
|
http.MethodHead,
|
|
http.MethodOptions,
|
|
http.MethodPatch,
|
|
http.MethodPost,
|
|
http.MethodPut,
|
|
http.MethodTrace:
|
|
return method
|
|
default:
|
|
return unmatchedMethod
|
|
}
|
|
}
|
|
|
|
// boundedLabelRecorder wraps a go-http-metrics recorder and replaces
|
|
// the request-controlled labels on every observation with bounded
|
|
// ones: the handler id becomes the request's route pattern, and the
|
|
// method becomes one the router can route.
|
|
//
|
|
// This is the seam that makes the pattern usable at all. The metrics
|
|
// middleware is global (see Server.setupGlobalMiddleware), so it is
|
|
// entered before chi has matched anything, and go-http-metrics fixes
|
|
// its handler id up front — passing the pattern in as that id is not
|
|
// possible, and leaving the id empty makes the library substitute the
|
|
// raw URL path, which is the defect. What the library does hand over
|
|
// is the request context, unchanged, on each recorder call; that
|
|
// context carries the same *chi.Context pointer routing mutates in
|
|
// place, and the duration and size calls happen after the wrapped
|
|
// handler has returned. Reading the pattern there is what the access
|
|
// log already does in accessLogURL.
|
|
//
|
|
// Recording after the whole chain returns is also what makes this
|
|
// hold for requests the route-level receiver rate limiter rejects.
|
|
// Those never reach a handler, but chi has already matched the route
|
|
// by the time the limiter runs, so their 429s land on the pattern
|
|
// like any other response.
|
|
type boundedLabelRecorder struct {
|
|
inner httpmetrics.Recorder
|
|
}
|
|
|
|
func (r boundedLabelRecorder) ObserveHTTPRequestDuration(
|
|
ctx context.Context,
|
|
props httpmetrics.HTTPReqProperties,
|
|
duration time.Duration,
|
|
) {
|
|
props.ID = routePatternID(ctx)
|
|
props.Method = methodID(props.Method)
|
|
r.inner.ObserveHTTPRequestDuration(ctx, props, duration)
|
|
}
|
|
|
|
func (r boundedLabelRecorder) ObserveHTTPResponseSize(
|
|
ctx context.Context,
|
|
props httpmetrics.HTTPReqProperties,
|
|
sizeBytes int64,
|
|
) {
|
|
props.ID = routePatternID(ctx)
|
|
props.Method = methodID(props.Method)
|
|
r.inner.ObserveHTTPResponseSize(ctx, props, sizeBytes)
|
|
}
|
|
|
|
func (r boundedLabelRecorder) AddInflightRequests(
|
|
ctx context.Context,
|
|
props httpmetrics.HTTPProperties,
|
|
quantity int,
|
|
) {
|
|
props.ID = inflightHandler
|
|
r.inner.AddInflightRequests(ctx, props, quantity)
|
|
}
|
|
|
|
var _ httpmetrics.Recorder = boundedLabelRecorder{}
|
|
|
|
// Metrics returns middleware that records Prometheus HTTP metrics on
|
|
// the default registry, which is the one the /metrics route gathers.
|
|
func (s *Middleware) Metrics() func(http.Handler) http.Handler {
|
|
return metricsMiddleware(
|
|
prommetrics.NewRecorder(prommetrics.Config{}),
|
|
)
|
|
}
|
|
|
|
// metricsMiddleware builds the recording middleware against a given
|
|
// recorder, so tests can gather from a registry of their own instead
|
|
// of the process-wide default.
|
|
func metricsMiddleware(
|
|
rec httpmetrics.Recorder,
|
|
) func(http.Handler) http.Handler {
|
|
mdlw := ghmm.New(ghmm.Config{
|
|
Recorder: boundedLabelRecorder{inner: rec},
|
|
})
|
|
|
|
return func(next http.Handler) http.Handler {
|
|
// The handler id is unmatchedRoute rather than "" so that
|
|
// the client-chosen URL path never enters the metrics
|
|
// pipeline at all: an empty id is the library's signal to
|
|
// substitute it. boundedLabelRecorder overwrites this value
|
|
// on every observation, so it is reachable only if that
|
|
// decorator is removed — in which case the metrics collapse
|
|
// to one series instead of leaking again.
|
|
return std.Handler(unmatchedRoute, mdlw, next)
|
|
}
|
|
}
|