Bound the receiver rate limit per client IP across /webhook/* (closes #139)

The receiver limiter keyed buckets on (client IP, request path). The
route pattern /webhook/{uuid} matches any single segment, so a client
that invented a fresh path per request minted a fresh bucket per
request and never refilled one: its aggregate rate against the only
unauthenticated, internet-exposed endpoint was unbounded, and every
one of those requests reached an entrypoint lookup before it 404ed.

Put a second limiter in front of it, keyed on the client IP alone and
covering the whole route at ten times the configured per-entrypoint
limit (1200/min by default). The per-entrypoint limit is unchanged and
still wanted; it just bounds nothing in aggregate on its own. Ten
entrypoints' worth of headroom lets one sender address drive several
entrypoints at full rate while still capping what one address costs
the receiver. The multiplication saturates rather than wrapping, since
nothing bounds RECEIVER_RATE_LIMIT from above and a negative limit
would reject every request.

The aggregate limiter is the outer one, so it counts the requests the
per-entrypoint limiter rejects; a test pins that order by exhausting
one path against the inner limit and then requiring a request to an
unused path to be rejected.

Move the handler's INFO line for an incoming webhook below the
entrypoint lookup. The UUID is attacker-controlled path text, so
logging it first let a client write an INFO line per invented path; a
miss is already logged at DEBUG and the request is already in the
access log.

Give the aggregate limiter its own 429 handler that logs at DEBUG and
without the path, rather than the shared one that logs at WARN with
it. Its rejections are one line per request of the very flood it
exists to bound, so the shared handler would have let a client write
its own text into the operator's log at an alerting level once per
request. What this limiter bounds is the database work an invented
path costs; the access log still records every request once at INFO,
and the README now says so instead of claiming DEBUG-only logging.

README.md

@@ -93,6 +93,7 @@ TTY detection, and security headers are always applied.
 | `METRICS_USERNAME`      | Basic auth username for `/metrics`  | `""`     |
 | `METRICS_PASSWORD`      | Basic auth password for `/metrics`  | `""`     |
 | `SENTRY_DSN`            | Sentry error reporting DSN          | `""`     |
+| `RETENTION_SWEEP_INTERVAL` | Retention reaper period (Go duration, must be positive) | `1h` |
 | `SESSION_IDLE_TIMEOUT`  | Idle session timeout (Go duration)  | `24h`    |
 | `RECEIVER_RATE_LIMIT`   | Receiver requests/minute per IP per entrypoint (10x that per IP across the route) | `120` |
 | `TRUSTED_PROXIES`       | CIDRs whose forwarded headers are trusted | `""` (none) |
@@ -173,8 +174,12 @@ its value and refuses to start, rather than silently running with a
 substituted default. `PORT=eighty`, `DEBUG=ture`, and
 `RETENTION_SWEEP_INTERVAL=1 hour` all abort startup. `PORT` must
 additionally be a number in the range 1–65535,
-`RECEIVER_RATE_LIMIT` must be at least 1, and every entry in
+`RECEIVER_RATE_LIMIT` must be at least 1,
-`TRUSTED_PROXIES` must be a CIDR block or a bare IP address.
+`RETENTION_SWEEP_INTERVAL` must be greater than zero (it is a ticker
+period, so `0s` or a negative value would crash the reaper after
+startup), and every entry in `TRUSTED_PROXIES` must be a CIDR block or
+a bare IP address. `SESSION_IDLE_TIMEOUT` is the exception: a
+non-positive value there means idle expiry is disabled, not invalid.
 
 Boolean variables (`DEBUG`, `MAINTENANCE_MODE`) accept exactly the
 spellings Go's `strconv.ParseBool` accepts — `1`, `t`, `T`, `TRUE`,
@@ -861,9 +866,15 @@ requests still costs an entrypoint lookup before it 404s. The aggregate
 limit leaves room for one address to drive several entrypoints at their
 full rate, and it is not configurable separately.
 
-Requests to a `/webhook/` path that names no entrypoint are logged at
+What that aggregate limit bounds is the database work an invented path
-`DEBUG` only, since the path is attacker-controlled; the request itself
+costs, not the number of log lines it produces. The path is
-still appears in the access log.
+attacker-controlled, so nothing on this route writes it to the log
+above `DEBUG`: a path that names no entrypoint is recorded by the
+handler at `DEBUG`, and the aggregate limiter logs its rejections at
+`DEBUG` and without the path. Every request is still recorded once by
+the access log, at `INFO`, with its full URL, whether it was served or
+rejected — so a flood of invented paths still writes one `INFO` line
+per request.
 
 Every limiter here — receiver, login, and password change — identifies
 the client the same way, through one shared key function: the
@@ -873,7 +884,14 @@ instead. See [Trusted proxies](#trusted-proxies). Deployed without that
 variable set, a client behind a reverse proxy shares one bucket with
 every other client behind the same proxy, which is the safe direction
 to be wrong in: set `TRUSTED_PROXIES` to the proxy's address to get
-per-client limits back.
+per-client limits back. That shared bucket matters more for the
+aggregate limit than for the per-entrypoint one: with `TRUSTED_PROXIES`
+unset behind the reverse proxy a production deployment is required to
+run behind, every request keys on the proxy, so the aggregate limit
+becomes a service-wide ceiling of 1200 requests per minute across all
+senders and all entrypoints, where the per-entrypoint limit's capacity
+still grows with the number of entrypoints. Any deployment with more
+than a handful of busy entrypoints must set `TRUSTED_PROXIES`.
 
 Finer-grained per-webhook rate limits (configured in the web UI and
 enforced in the webhook handler) can layer on top of this env-level

internal/config/config.go

@@ -92,6 +92,7 @@ type Config struct {
 	SentryDSN       string
 
 	// RetentionSweepInterval is how often the retention reaper runs.
+	// Always positive: it becomes a time.NewTicker period.
 	RetentionSweepInterval time.Duration
 
 	// SessionIdleTimeout is the sliding inactivity window after
@@ -235,6 +236,34 @@ func envDuration(
 	return d, nil
 }
 
+// envPositiveDuration returns the value of the named environment
+// variable parsed as a Go duration that must be greater than zero.
+// Returns defaultValue if not set. A set value that is unparseable or
+// non-positive is a hard error naming the key and the bad value.
+//
+// This is for durations that reach time.NewTicker, which panics on a
+// non-positive period, in a goroutine started after startup has
+// already reported success. It is deliberately not used for durations
+// where non-positive means "disabled" (SESSION_IDLE_TIMEOUT).
+func envPositiveDuration(
+	key string,
+	defaultValue time.Duration,
+) (time.Duration, error) {
+	d, err := envDuration(key, defaultValue)
+	if err != nil {
+		return 0, err
+	}
+
+	if d <= 0 {
+		return 0, fmt.Errorf(
+			"%w: %s must be greater than zero, got %s",
+			ErrNonPositiveValue, key, d,
+		)
+	}
+
+	return d, nil
+}
+
 // parseCIDR parses one trusted-proxy list entry, which may be a
 // CIDR block ("10.0.0.0/8") or a bare address ("10.0.0.1", treated
 // as a single-host block).
@@ -346,7 +375,7 @@ func loadFromEnv() (*Config, error) {
 		return nil, err
 	}
 
-	retentionSweepInterval, err := envDuration(
+	retentionSweepInterval, err := envPositiveDuration(
 		"RETENTION_SWEEP_INTERVAL",
 		defaultRetentionSweepInterval,
 	)
@@ -354,6 +383,8 @@ func loadFromEnv() (*Config, error) {
 		return nil, err
 	}
 
+	// Non-positive is "disabled" here, not invalid, so this stays on
+	// envDuration.
 	sessionIdleTimeout, err := envDuration(
 		"SESSION_IDLE_TIMEOUT",
 		defaultSessionIdleTimeout,

internal/config/config_test.go

@@ -139,7 +139,11 @@ func TestRetentionSweepInterval(t *testing.T) {
 		set         bool
 		value       string
 		expectError bool
-		expected    time.Duration
+		// sentinel, when set, must be wrapped by the startup
+		// error; every error case must additionally name the
+		// variable in its message.
+		sentinel error
+		expected time.Duration
 	}{
 		{
 			name:     caseUnsetUsesDefault,
@@ -158,6 +162,24 @@ func TestRetentionSweepInterval(t *testing.T) {
 			value:       "not-a-duration",
 			expectError: true,
 		},
+		{
+			// A non-positive period panics the ticker in the
+			// reaper and archive-sweeper goroutines, long after
+			// startup has reported success, so it has to fail
+			// here instead.
+			name:        "zero fails startup",
+			set:         true,
+			value:       "0s",
+			expectError: true,
+			sentinel:    config.ErrNonPositiveValue,
+		},
+		{
+			name:        "negative fails startup",
+			set:         true,
+			value:       "-1h",
+			expectError: true,
+			sentinel:    config.ErrNonPositiveValue,
+		},
 	}
 
 	for _, tt := range tests {
@@ -175,7 +197,9 @@ func TestRetentionSweepInterval(t *testing.T) {
 			}
 
 			if tt.expectError {
-				expectStartupError(t)
+				expectStartupErrorFor(
+					t, "RETENTION_SWEEP_INTERVAL", tt.sentinel,
+				)
 			} else {
 				testRetentionSweepIntervalSuccess(t, tt.expected)
 			}
@@ -281,6 +305,22 @@ func TestSessionIdleTimeout(t *testing.T) {
 			value:       "not-a-duration",
 			expectError: true,
 		},
+		{
+			// Non-positive is "idle expiry disabled" for this
+			// variable, not a configuration error: unlike
+			// RETENTION_SWEEP_INTERVAL it never becomes a ticker
+			// period.
+			name:     "zero disables idle expiry",
+			set:      true,
+			value:    "0s",
+			expected: 0,
+		},
+		{
+			name:     "negative disables idle expiry",
+			set:      true,
+			value:    "-1h",
+			expected: -time.Hour,
+		},
 	}
 
 	for _, tt := range tests {

internal/middleware/ratelimit.go

@@ -183,6 +183,31 @@ func (m *Middleware) tooManyRequests(
 	}
 }
 
+// floodTooManyRequests returns the 429 handler for a limiter whose
+// rejections are themselves the flood: it logs at DEBUG and without
+// the path, then answers with responseMessage.
+//
+// The aggregate receiver limiter trips exactly when one address is
+// sending faster than the receiver wants to serve, so its rejection
+// log is one line per request of that flood. At WARN with "path" that
+// hands a client a way to write its own text into the operator's log,
+// at a level that trips alerting, once per request — the log-volume
+// problem this limiter exists to bound. DEBUG is off in production by
+// default, so a flood costs nothing here; the path is dropped so that
+// turning DEBUG on to diagnose one does not restore the problem.
+//
+// This limiter bounds the database work an invented path costs, not
+// the number of log lines it produces: the access log in
+// middleware.go still records every request, served or rejected.
+func (m *Middleware) floodTooManyRequests(
+	logMessage, responseMessage string,
+) http.HandlerFunc {
+	return func(w http.ResponseWriter, _ *http.Request) {
+		m.log.Debug(logMessage)
+		http.Error(w, responseMessage, http.StatusTooManyRequests)
+	}
+}
+
 // LoginRateLimit returns middleware that enforces per-IP rate
 // limiting on login attempts using go-chi/httprate. Only POST
 // requests are rate-limited; GET requests (rendering the login
@@ -287,7 +312,7 @@ func (m *Middleware) ReceiverRateLimit() func(http.Handler) http.Handler {
 		receiverAggregateLimit(m.params.Config.ReceiverRateLimit),
 		receiverRateInterval,
 		httprate.WithKeyFuncs(m.rateLimitKey),
-		httprate.WithLimitHandler(m.tooManyRequests(
+		httprate.WithLimitHandler(m.floodTooManyRequests(
 			"webhook receiver aggregate rate limit exceeded",
 			"Too many requests. Please slow down.",
 		)),

internal/middleware/ratelimit_test.go

@@ -723,6 +723,58 @@ func TestReceiverRateLimit_LimitsAggregateAcrossInventedPaths(
 	)
 }
 
+// TestReceiverRateLimit_RejectedRequestsCountTowardAggregate pins the
+// order the two limiters are chained in. The aggregate limiter has to
+// be the outer one, so that it counts requests the per-entrypoint
+// limiter rejects: those requests still arrive, and the aggregate
+// limit exists to bound what one address can make the receiver do.
+//
+// One path is hammered past the per-entrypoint limit, which alone
+// would leave the aggregate budget almost untouched; then a path the
+// client has never used must be rejected, which only the aggregate
+// limiter can do. Swap the two limiters and that last request is
+// served, because the rejected ones never reached the aggregate
+// limiter to be counted.
+func TestReceiverRateLimit_RejectedRequestsCountTowardAggregate(
+	t *testing.T,
+) {
+	t.Parallel()
+
+	const (
+		limit = 3
+		ip    = "6.6.6.8:1234"
+	)
+
+	aggregate := limit * middleware.ReceiverAggregateMultiplierConst
+
+	handler := receiverLimitedHandler(t, limit)
+
+	// Spend the whole aggregate budget on one path. Only the first
+	// limit requests are served; the rest are rejected by the
+	// per-entrypoint limiter but still count against the aggregate.
+	for i := range aggregate {
+		w := receiverPost(handler, ip, "/webhook/exhausted")
+
+		want := http.StatusTooManyRequests
+		if i < limit {
+			want = http.StatusOK
+		}
+
+		assert.Equal(
+			t, want, w.Code,
+			"request %d to the exhausted path", i,
+		)
+	}
+
+	w := receiverPost(handler, ip, "/webhook/never-used")
+	assert.Equal(
+		t, http.StatusTooManyRequests, w.Code,
+		"requests rejected per entrypoint must still count "+
+			"toward the aggregate limit, so the aggregate "+
+			"limiter has to run first",
+	)
+}
+
 // TestReceiverAggregateLimit_SaturatesOnOverflow covers the derived
 // aggregate limit for a configured per-entrypoint limit large enough
 // that multiplying it would wrap negative, which httprate would read