Compare commits
4 Commits
99968231ad
...
74a57ad237
| Author | SHA1 | Date | |
|---|---|---|---|
| 74a57ad237 | |||
| bef9986542 | |||
| c3b6623be1 | |||
| 39064a3d6c |
500
README.md
500
README.md
@@ -11,9 +11,14 @@ with retry support, logging, and observability. Category: infrastructure
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Go 1.26+
|
||||
- golangci-lint v2.11+
|
||||
- Docker (for containerized deployment)
|
||||
- Go 1.26.1+ (the version in `go.mod`)
|
||||
- golangci-lint v2.12.2 (the version pinned in `script/bootstrap` and
|
||||
in the `Dockerfile`'s lint stage; `make bootstrap` installs it)
|
||||
- Docker (for containerized deployment, and for the lint and test
|
||||
stages of the CI gate)
|
||||
- `curl`, used by `script/fetch-assets` to download the third-party
|
||||
browser assets, which are not committed (`make bootstrap` installs
|
||||
it if missing)
|
||||
|
||||
### Quick Start
|
||||
|
||||
@@ -22,14 +27,18 @@ with retry support, logging, and observability. Category: infrastructure
|
||||
git clone https://git.eeqj.de/sneak/webhooker.git
|
||||
cd webhooker
|
||||
|
||||
# Install Go dependencies
|
||||
make deps
|
||||
# Install Go dependencies, the pinned linter, and the third-party
|
||||
# browser assets. `make deps` alone is not enough: it only runs
|
||||
# go mod download/tidy, and the checks below need the fetched assets.
|
||||
make bootstrap
|
||||
|
||||
# Run all checks (format, lint, test, build)
|
||||
# Run all checks (test, lint, format check)
|
||||
make check
|
||||
|
||||
# Run in development mode (uses SQLite in current directory)
|
||||
make dev
|
||||
# Run in development mode. DATA_DIR defaults to /var/lib/webhooker in
|
||||
# every environment, so set it (in .env or the shell) to a writable
|
||||
# directory when running from a clone.
|
||||
DATA_DIR=./data make dev
|
||||
|
||||
# Build Docker image
|
||||
make docker
|
||||
@@ -42,13 +51,18 @@ make bootstrap # Install all dependencies (idempotent)
|
||||
make setup # Bootstrap + install git pre-commit hook
|
||||
make assets # Fetch + verify third-party browser assets
|
||||
make fmt # Format code (gofmt + goimports)
|
||||
make fmt-check # Fail if gofmt would change anything (writes nothing)
|
||||
make lint # Run golangci-lint
|
||||
make test # Run tests with race detection
|
||||
make check # test + lint + fmt-check (CI gate)
|
||||
make build # Build binary to bin/webhooker
|
||||
make run # build, then run ./bin/webhooker
|
||||
make dev # go run ./cmd/webhooker
|
||||
make deps # go mod download + go mod tidy
|
||||
make docker # Build Docker image
|
||||
make hooks # Install git pre-commit hook that runs script/precommit
|
||||
make css # Regenerate static/css/tailwind.css (needs tailwindcss)
|
||||
make clean # Remove bin/
|
||||
```
|
||||
|
||||
### Configuration
|
||||
@@ -90,7 +104,7 @@ TTY detection, and security headers are always applied.
|
||||
| `PORT` | HTTP listen port | `8080` |
|
||||
| `DATA_DIR` | Directory for all SQLite databases | `/var/lib/webhooker` |
|
||||
| `DEBUG` | Enable debug logging | `false` |
|
||||
| `MAINTENANCE_MODE` | Serve the maintenance page | `false` |
|
||||
| `MAINTENANCE_MODE` | Report `maintenanceMode: true` in the healthcheck JSON. It does not change how any request is served — no maintenance page exists | `false` |
|
||||
| `METRICS_USERNAME` | Basic auth username for `/metrics` | `""` |
|
||||
| `METRICS_PASSWORD` | Basic auth password for `/metrics` | `""` |
|
||||
| `SENTRY_DSN` | Sentry error reporting DSN | `""` |
|
||||
@@ -130,8 +144,13 @@ sustained trickle re-locks them immediately.
|
||||
|
||||
The remedy is to set `TRUSTED_PROXIES` to your reverse proxy's
|
||||
address, which restores per-client buckets. webhooker logs a warning
|
||||
at startup when `WEBHOOKER_ENVIRONMENT=prod` and `TRUSTED_PROXIES` is
|
||||
empty. See [Rate Limiting](#rate-limiting) for what each limit shares.
|
||||
at startup whenever `TRUSTED_PROXIES` is empty, in every environment —
|
||||
not only when `WEBHOOKER_ENVIRONMENT=prod`, because that variable
|
||||
defaults to `dev` and an operator who never set it is precisely the
|
||||
one at risk. The warning is informational when nothing proxies to the
|
||||
process: with no proxy in front, the peer address is the client's own
|
||||
and the buckets are already per-client. See
|
||||
[Rate Limiting](#rate-limiting) for what each limit shares.
|
||||
|
||||
`X-Real-IP` and `True-Client-IP` are **never** read, from any peer.
|
||||
Reverse proxies append to `X-Forwarded-For` but forward other client
|
||||
@@ -163,6 +182,8 @@ Two operator requirements follow:
|
||||
makes all three limits, including the unauthenticated webhook
|
||||
receiver, silently bypassable by every client in the block.
|
||||
|
||||
#### Sessions
|
||||
|
||||
Sessions are bounded by two independent clocks, and end at whichever
|
||||
one runs out first:
|
||||
|
||||
@@ -232,17 +253,20 @@ docker run -d \
|
||||
The container runs as a non-root user (`webhooker`, UID 1000), exposes
|
||||
port 8080, and includes a health check against
|
||||
`/.well-known/healthcheck`. The `/var/lib/webhooker` volume holds all
|
||||
SQLite databases: the main application database (`webhooker.db`) and
|
||||
the per-webhook event databases (`events-{uuid}.db`). Mount this as a
|
||||
persistent volume to preserve data across container restarts.
|
||||
SQLite databases: the main application database (`webhooker.db`), the
|
||||
per-webhook event databases (`events-{uuid}.db`), and any archive
|
||||
databases written by `database` targets (`archive-{uuid}.db`). Mount
|
||||
this as a persistent volume to preserve data across container
|
||||
restarts.
|
||||
|
||||
## Entrypoints
|
||||
|
||||
This repository adheres to the
|
||||
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
||||
standard: normalized scripts in `script/` are the entrypoints for the
|
||||
development workflow, and the Makefile targets are thin shims that call
|
||||
them. We provide:
|
||||
development workflow. Ten of the Makefile's sixteen targets are thin
|
||||
shims that call them; `build`, `run`, `dev`, `deps`, `clean` and `css`
|
||||
are inline commands with no script behind them. We provide:
|
||||
|
||||
- `script/bootstrap` — install all dependencies (idempotent)
|
||||
- `script/setup` — make a fresh clone ready for development
|
||||
@@ -354,7 +378,11 @@ It uses:
|
||||
- **[gorilla/csrf](https://github.com/gorilla/csrf)** for CSRF
|
||||
protection (cookie-based double-submit tokens)
|
||||
- **[go-chi/httprate](https://github.com/go-chi/httprate)** for
|
||||
per-IP login rate limiting (sliding window counter)
|
||||
sliding-window rate limiting of the login, password-change and
|
||||
webhook receiver endpoints. The bucket is per client IP only when
|
||||
`TRUSTED_PROXIES` names the reverse proxy; unset, every client
|
||||
behind that proxy shares one bucket per limit (see
|
||||
[Rate Limiting](#rate-limiting))
|
||||
- **[Prometheus](https://prometheus.io)** for metrics, served at
|
||||
`/metrics` behind basic auth
|
||||
- **[Sentry](https://sentry.io)** for optional error reporting
|
||||
@@ -372,7 +400,7 @@ The codebase uses consistent naming throughout (rename completed in
|
||||
|
||||
### Data Model
|
||||
|
||||
webhooker's data model has eight entities organized into two tiers: the
|
||||
webhooker's data model has nine entities organized into two tiers: the
|
||||
**application tier** (user and webhook configuration) and the **event
|
||||
tier** (event ingestion, delivery, and logging).
|
||||
|
||||
@@ -464,9 +492,25 @@ days (`database.RetentionForeverDays`). The retention reaper recognises
|
||||
that sentinel and skips the webhook entirely, and the web UI displays
|
||||
such a webhook's retention as "forever" rather than as a day count.
|
||||
|
||||
A *finite* retention is capped at `database.MaxFiniteRetentionDays`
|
||||
(106751 days, about 292 years), and a larger one is rejected with a
|
||||
400. The cap is not arbitrary: the reaper computes its cutoff as a
|
||||
Submitted `retention_days` values therefore fall into three bands, not
|
||||
two:
|
||||
|
||||
- `1` up to `database.MaxFiniteRetentionDays` (106751 days, about 292
|
||||
years) is accepted as a finite retention.
|
||||
- Above that ceiling but below the retain-forever sentinel of 365000
|
||||
(`database.RetentionForeverDays`) is rejected with a 400. This is the
|
||||
band the cap exists for.
|
||||
- `0`, and `365000` or above, are accepted and mean retain forever,
|
||||
collapsing to the sentinel — `0` in `Webhook.BeforeSave`, the large
|
||||
values in `parseRetentionDays`. The large values are not out of
|
||||
range: the edit form pre-fills the sentinel for a retain-forever
|
||||
webhook, so submitting that form back unchanged has to keep meaning
|
||||
"forever".
|
||||
|
||||
A negative value is in none of the three: `parseRetentionDays` rejects
|
||||
it with a 400 before `BeforeSave` ever sees it.
|
||||
|
||||
The cap is not arbitrary: the reaper computes its cutoff as a
|
||||
`time.Duration`, an int64 nanosecond count, and a longer period
|
||||
overflows it. An overflowed cutoff lands in the future, where it
|
||||
matches every row, so the sweep would delete every event the webhook
|
||||
@@ -484,7 +528,7 @@ the full request and creates an Event.
|
||||
| -------------- | ------- | ----------- |
|
||||
| `id` | UUID | Primary key |
|
||||
| `webhook_id` | UUID | Foreign key → Webhook |
|
||||
| `path` | string | Unique URL path (UUID-based, e.g. `/webhook/{uuid}`) |
|
||||
| `path` | string | Unique bare UUID, generated at creation. The `/webhook/` prefix is route only and is not stored: the receiver matches this column against the raw `{uuid}` path segment |
|
||||
| `description` | string | Optional description |
|
||||
| `active` | boolean | Whether this entrypoint accepts events (default: true) |
|
||||
|
||||
@@ -508,8 +552,8 @@ events should be forwarded.
|
||||
| `type` | TargetType | One of: `http`, `slack`, `database`, `log` |
|
||||
| `active` | boolean | Whether deliveries are enabled (default: true) |
|
||||
| `config` | JSON text | Type-specific configuration |
|
||||
| `max_retries` | integer | Maximum retry attempts for HTTP targets (0 = fire-and-forget, >0 = retries with backoff) |
|
||||
| `max_queue_size` | integer | Maximum queued deliveries (for HTTP targets with retries) |
|
||||
| `max_retries` | integer | Maximum retry attempts for `http` and `slack` targets (0 = fire-and-forget, >0 = retries with backoff and a circuit breaker). Ignored by `database` and `log` targets |
|
||||
| `max_queue_size` | integer | Stored and shown on the target's detail view, but not enforced anywhere yet: nothing in the delivery engine consults it. Queue depth is set by the two fixed 10,000-entry channels |
|
||||
|
||||
**Relations:** Belongs to Webhook. Has many Deliveries.
|
||||
|
||||
@@ -522,6 +566,11 @@ events should be forwarded.
|
||||
greater than 0, failed deliveries are retried with exponential backoff
|
||||
up to `max_retries` attempts, protected by a per-target circuit
|
||||
breaker.
|
||||
- **`slack`** — Post the event as a formatted message to a
|
||||
Slack-compatible incoming webhook URL (`webhookUrl` in `config`). It
|
||||
is built on the same HTTP core as `http` and honours `max_retries`
|
||||
identically, circuit breaker included. See the Slack target section
|
||||
under "Per-Webhook Event Databases" for the message format.
|
||||
- **`database`** — Archive the full event as a row into a separate
|
||||
per-webhook archive database (`archive-{webhookID}.db`) for long-term
|
||||
retention, with an optional creation-validated expiry (default: keep
|
||||
@@ -558,7 +607,7 @@ data for auditing and for the planned replay capability.
|
||||
| `id` | UUID | Primary key |
|
||||
| `webhook_id` | UUID | Foreign key → Webhook |
|
||||
| `entrypoint_id` | UUID | Foreign key → Entrypoint |
|
||||
| `method` | string | HTTP method (POST, PUT, etc.) |
|
||||
| `method` | string | HTTP method of the captured request. Always `POST`: the receiver answers every other method with 405 before an Event is created |
|
||||
| `headers` | JSON | Complete request headers |
|
||||
| `body` | text | Raw request body |
|
||||
| `content_type` | string | Content-Type header value |
|
||||
@@ -613,7 +662,9 @@ retries) is individually logged for full observability.
|
||||
|
||||
#### Common Fields
|
||||
|
||||
All entities include these fields from `BaseModel`:
|
||||
Every entity except `Setting` includes these fields from `BaseModel`.
|
||||
`Setting` is a bare key-value row with no `id`, no timestamps and no
|
||||
soft delete:
|
||||
|
||||
| Field | Type | Description |
|
||||
| ------------ | --------- | ----------- |
|
||||
@@ -659,7 +710,7 @@ handles connection pooling, lazy opening, migrations, and cleanup.
|
||||
This separation provides:
|
||||
|
||||
- **Isolation** — a high-volume webhook won't cause lock contention or
|
||||
WAL bloat affecting the main application or other webhooks.
|
||||
journal growth affecting the main application or other webhooks.
|
||||
- **Independent lifecycle** — event databases can be independently
|
||||
backed up, archived, rotated, or size-limited without impacting the
|
||||
application.
|
||||
@@ -669,9 +720,12 @@ This separation provides:
|
||||
- **Per-webhook retention** — the `retention_days` field on each webhook
|
||||
controls automatic cleanup of old events in that webhook's database
|
||||
only, or disables cleanup entirely when set to `0` (retain forever).
|
||||
- **Performance** — each webhook's database has its own WAL, its own
|
||||
page cache, and its own lock, so concurrent event ingestion across
|
||||
webhooks won't contend.
|
||||
- **Performance** — each webhook's database has its own page cache and
|
||||
its own lock, so concurrent event ingestion across webhooks won't
|
||||
contend. No write-ahead log is involved: both DSNs are
|
||||
`file:{path}?cache=shared&mode=rwc` and no `journal_mode` pragma is
|
||||
ever issued, so every database runs on SQLite's default rollback
|
||||
journal.
|
||||
|
||||
The **database target type** builds on this architecture to provide
|
||||
long-term archiving, separate from the per-webhook event database (which
|
||||
@@ -727,8 +781,9 @@ and other compatible services). Each message includes event metadata
|
||||
pretty-printed in a code block. JSON payloads are automatically
|
||||
formatted with indentation for readability; non-JSON payloads are shown
|
||||
as raw text. Large payloads are truncated to keep messages reasonable.
|
||||
Config stores `webhook_url` — the Slack/Mattermost incoming webhook
|
||||
endpoint.
|
||||
Config stores `webhookUrl` — the Slack/Mattermost incoming webhook
|
||||
endpoint. That is the JSON key; the error text for a missing one reads
|
||||
`webhook_url is required`, which is the message, not the key.
|
||||
|
||||
The database uses the
|
||||
[modernc.org/sqlite](https://pkg.go.dev/modernc.org/sqlite) driver at
|
||||
@@ -750,8 +805,9 @@ External Service
|
||||
1. Look up Entrypoint by UUID
|
||||
2. Capture full request as Event
|
||||
3. Create Delivery records for each active Target
|
||||
4. Build self-contained DeliveryTask structs
|
||||
(target config + event data inline for ≤16KB)
|
||||
4. Build self-contained delivery.Task structs
|
||||
(target config + event data inline for
|
||||
bodies < 16 KiB)
|
||||
5. Notify Engine via channel (no DB read needed)
|
||||
│
|
||||
▼
|
||||
@@ -786,7 +842,7 @@ at any time, preventing goroutine explosions regardless of queue depth.
|
||||
a delivery channel (new tasks from the webhook handler) and a retry
|
||||
channel (tasks from backoff timers). Both are buffered to 10,000.
|
||||
- **Fan-out via channel, not goroutines:** When an event arrives with
|
||||
multiple targets, each `DeliveryTask` is sent to the delivery channel.
|
||||
multiple targets, each `delivery.Task` is sent to the delivery channel.
|
||||
Workers pick them up and process them — no goroutine-per-target.
|
||||
- **Worker goroutines:** A fixed number of worker goroutines select from
|
||||
both channels. Each worker processes one task at a time, then picks up
|
||||
@@ -810,7 +866,12 @@ This means:
|
||||
- **Independent results** — each worker records its own delivery result
|
||||
in the per-webhook database without coordination.
|
||||
- **Graceful shutdown** — cancel the context, workers finish their
|
||||
current task and exit. `WaitGroup.Wait()` ensures clean shutdown.
|
||||
current task and exit. The stop hook waits for the pool via
|
||||
`lifecycle.WaitForShutdown`, which bounds that wait by fx's stop
|
||||
timeout rather than blocking forever on a wedged worker. On timeout
|
||||
it logs at `ERROR` and returns an error, and the goroutines that
|
||||
did not finish are still running — an unclean shutdown is reported
|
||||
rather than hidden.
|
||||
|
||||
**Recovery paths:**
|
||||
|
||||
@@ -838,12 +899,13 @@ remains stored in the per-webhook event database, there is no way to
|
||||
redeliver it: manual redelivery is planned, not implemented (see
|
||||
[TODO.md](TODO.md)).
|
||||
|
||||
### Circuit Breaker (HTTP Targets with Retries)
|
||||
### Circuit Breaker (HTTP and Slack Targets with Retries)
|
||||
|
||||
HTTP targets with `max_retries` > 0 are protected by a **per-target circuit breaker** that
|
||||
prevents hammering a down target with repeated failed delivery attempts.
|
||||
The circuit breaker is in-memory only and resets on restart (which is
|
||||
fine — startup recovery rescans the database anyway).
|
||||
`http` and `slack` targets with `max_retries` > 0 are protected by a
|
||||
**per-target circuit breaker** that prevents hammering a down target
|
||||
with repeated failed delivery attempts. The circuit breaker is
|
||||
in-memory only and resets on restart (which is fine — startup recovery
|
||||
rescans the database anyway).
|
||||
|
||||
**States:**
|
||||
|
||||
@@ -879,10 +941,12 @@ fine — startup recovery rescans the database anyway).
|
||||
- **Failure threshold:** 5 consecutive failures before opening
|
||||
- **Cooldown:** 30 seconds in open state before probing
|
||||
|
||||
**Scope:** Circuit breakers only apply to **HTTP targets with
|
||||
`max_retries` > 0**. Fire-and-forget HTTP targets (`max_retries` == 0),
|
||||
Slack targets, database targets (local operations), and log
|
||||
targets (stdout) do not use circuit breakers.
|
||||
**Scope:** Circuit breakers apply to **`http` and `slack` targets with
|
||||
`max_retries` > 0**. The Slack target is built on the same HTTP core
|
||||
and hands its own `max_retries` to the same retry path, so it gets a
|
||||
breaker with the same 5-failure / 30-second defaults. Fire-and-forget
|
||||
targets of either type (`max_retries` == 0), database targets (local
|
||||
operations), and log targets (stdout) do not use circuit breakers.
|
||||
|
||||
When a circuit is open and a new delivery arrives, the engine marks the
|
||||
delivery as `retrying` and schedules a retry timer for after the
|
||||
@@ -898,9 +962,11 @@ unpredictable rates, and blanket limits shared with other routes would
|
||||
cause legitimate deliveries to be dropped.
|
||||
|
||||
The receiver instead has its own dedicated abuse limit, scoped to the
|
||||
`/webhook/{uuid}` route only and keyed per client IP per entrypoint: one
|
||||
misbehaving sender is throttled without affecting other senders of the
|
||||
same entrypoint or the same sender's other entrypoints. The limit is
|
||||
`/webhook/{uuid}` route only and keyed per client IP per request path
|
||||
(`httprate.KeyByEndpoint`): one misbehaving sender is throttled without
|
||||
affecting other senders of the same entrypoint or the same sender's
|
||||
other entrypoints. Keying on the path rather than on the entrypoint
|
||||
matters — see the aggregate limit below. The limit is
|
||||
`RECEIVER_RATE_LIMIT` requests per minute (default 120, generous for
|
||||
legitimate webhook senders). Requests over the limit receive HTTP 429
|
||||
with a `Retry-After` header. A set-but-invalid `RECEIVER_RATE_LIMIT`
|
||||
@@ -928,14 +994,83 @@ requests and has the rest of its aggregate budget rejected there, so
|
||||
the aggregate limit is what bounds those `WARN` lines — to under ten
|
||||
times `RECEIVER_RATE_LIMIT` per minute per client IP, 1080 at the
|
||||
defaults, where before it there was no bound at all. The access log is
|
||||
bounded by neither limit: every request is recorded once at `INFO` with
|
||||
its full URL, served or rejected alike.
|
||||
bounded by neither limit: every request is recorded once at `INFO`,
|
||||
served or rejected alike.
|
||||
|
||||
What the access log does bound is the _content_ of those lines. A 3xx
|
||||
or 4xx response logs the chi route pattern — `/webhook/{uuid}`,
|
||||
`/user/{username}//`, or the literal `(unmatched)` when the request hit
|
||||
no route at all — in place of the concrete URL. Those are the outcomes
|
||||
an unauthenticated client can drive for free: 404 and 429 on any
|
||||
invented receiver path, a login redirect on any invented profile path.
|
||||
Logging the URL there would let a flood write text of its own choosing,
|
||||
at a length of its own choosing, into the log. 2xx and 5xx responses
|
||||
keep the concrete path — a success resolved against a static route or
|
||||
against the operator's own data (on the receiver, against a stored
|
||||
entrypoint UUID), and a 5xx is a bug in this service, where the exact
|
||||
path is the evidence and no client can provoke one at will.
|
||||
|
||||
The query string is never logged; it is replaced by the fixed marker
|
||||
`?(redacted)`. It is client-chosen on every route, and
|
||||
`/.well-known/healthcheck` and `/s/*` answer 200 to anyone with no rate
|
||||
limiter in front of them, so a query on a fixed 200 URL would otherwise
|
||||
buy the same amplification as an invented path. Nothing debuggable is
|
||||
lost: `page`, on the authenticated pagination links, is the only query
|
||||
parameter this service reads.
|
||||
|
||||
The remaining client-supplied fields are truncated rather than dropped,
|
||||
each to a fixed budget: 512 bytes for `url`, `useragent` and `referer`,
|
||||
128 for `request_id` (chi passes an inbound `X-Request-Id` header
|
||||
through), and 32 for `method`. A truncated `User-Agent` is still worth
|
||||
reading; an absent one is not. A cut value ends in `[truncated]`, which
|
||||
is charged on top of the budget rather than inside it.
|
||||
|
||||
Each budget is spent in _encoded_ bytes, not in the bytes the client
|
||||
sent. Every rune is charged what the wider of the two log handlers
|
||||
emits for it: two bytes for a quotation mark, a backslash or a tab; six
|
||||
for a non-printable rune below U+10000; ten for one at or above it,
|
||||
which the text handler spells `\UXXXXXXXX`. Go's header parser accepts
|
||||
all of them in a header value, so a budget counted raw would buy a
|
||||
field several times its nominal size — and the line, not the header, is
|
||||
what an operator has to store. Plain ASCII encodes one byte for one, so
|
||||
a real browser's `User-Agent` still fits whole; a value built out of
|
||||
escapes keeps a proportionally shorter prefix, which is the right
|
||||
trade.
|
||||
|
||||
Net: **one `INFO` line per request, of at most 2,560 bytes.** That
|
||||
ceiling is arithmetic, not an observation: 3 × (512 + 11) for `url`,
|
||||
`useragent` and `referer`, plus 128 + 11 for `request_id`, plus 32 + 11
|
||||
for `method`, plus a 336-byte fixed portion (the field names, the
|
||||
punctuation, both timestamps at their longest, an IPv6 `remoteIP` with
|
||||
a zone, the status and the latency) — 2,087 bytes, stated at 2,560 so
|
||||
the figure has headroom. `internal/middleware/accesslog_test.go`
|
||||
asserts it against 8 KB of client-chosen text in the path, in the
|
||||
query, and in each of `User-Agent`, `Referer` and `X-Request-Id`,
|
||||
including cases built from the characters the handlers escape, and
|
||||
against the widest line the service can be made to write: a 5xx that
|
||||
keeps its concrete path while all three header fields are also at their
|
||||
budget. Every case runs through both handlers `internal/logger` can
|
||||
select — the JSON one and the text one it installs on a tty — since the
|
||||
two do not escape alike and the ceiling is quoted unqualified. Measured
|
||||
over a real connection, the widest line is 1,972 bytes.
|
||||
|
||||
Multiply that ceiling by the request rate to size log storage. Note
|
||||
that the rate is not bounded by the limits above on every route:
|
||||
`/.well-known/healthcheck` and `/s/*` sit behind no limiter, so there
|
||||
the multiplier is whatever the deployment will serve.
|
||||
|
||||
Every limiter here — receiver, login, and password change — identifies
|
||||
the client the same way, through one shared key function: the
|
||||
connection's own address, unless the peer is listed in
|
||||
`TRUSTED_PROXIES`, in which case the forwarded client address is used
|
||||
instead. See [Trusted proxies](#trusted-proxies). Deployed without that
|
||||
instead. That address becomes a bucket by family: IPv4 keys on the full
|
||||
address, IPv6 on its `/64` prefix. A routed `/64` is the normal
|
||||
residential and mobile IPv6 allocation, so keying IPv6 per address would
|
||||
let one subscriber rotate source addresses and mint a fresh bucket per
|
||||
request, evading these limits at the network layer without spoofing
|
||||
anything; the cost is that distinct clients inside one `/64` share a
|
||||
bucket. IPv4-mapped addresses (`::ffff:1.2.3.4`) key as the IPv4 address
|
||||
they carry. See [Trusted proxies](#trusted-proxies). Deployed without that
|
||||
variable set, a client behind a reverse proxy shares one bucket with
|
||||
every other client behind the same proxy. Set `TRUSTED_PROXIES` to the
|
||||
proxy's address to get per-client limits back. What the shared bucket
|
||||
@@ -958,8 +1093,8 @@ opposite directions:
|
||||
login bucket full, and the operator's own login returns HTTP 429 for
|
||||
as long as that trickle continues. A restart clears the in-memory
|
||||
buckets and a resumed trickle re-locks them. Production deployments
|
||||
must set `TRUSTED_PROXIES`; webhooker warns at startup when it is
|
||||
empty in `prod`.
|
||||
must set `TRUSTED_PROXIES`; webhooker warns at startup whenever it is
|
||||
empty, in any environment.
|
||||
|
||||
Finer-grained per-webhook rate limits (configured in the web UI and
|
||||
enforced in the webhook handler) can layer on top of this env-level
|
||||
@@ -971,17 +1106,17 @@ abuse limit later; they are tracked as future work.
|
||||
|
||||
| Method | Path | Description |
|
||||
| ------ | --------------------------- | ----------- |
|
||||
| `GET` | `/` | Root redirect (authenticated → `/sources`, unauthenticated → `/pages/login`) |
|
||||
| `GET` | `/.well-known/healthcheck` | Health check (JSON: status, uptime, version) |
|
||||
| `GET` | `/s/*` | Static file serving (embedded CSS, JS) |
|
||||
| `ANY` | `/webhook/{uuid}` | Webhook receiver endpoint (accepts all methods) |
|
||||
| `GET` | `/` | Root redirect, 303 (authenticated → `/sources`, unauthenticated → `/pages/login`) |
|
||||
| `GET` | `/.well-known/healthcheck` | Health check (JSON: `status`, `now`, `uptimeSeconds`, `uptimeHuman`, `version`, `appname`, `maintenanceMode`) |
|
||||
| any | `/s/*` | Static file serving (embedded CSS, JS). Mounted for every method, not just `GET`/`HEAD`: chi's `Mount` registers all methods and `http.FileServer` special-cases only `HEAD` (by omitting the body), so a `POST` or `DELETE` to an asset is answered `200` with the file. Pinned by `TestStaticServesEveryMethod` |
|
||||
| `POST` | `/webhook/{uuid}` | Webhook receiver endpoint. `POST` only — every other method is answered `405 Method Not Allowed` with `Allow: POST`. Rate limited (see [Rate Limiting](#rate-limiting)) |
|
||||
|
||||
#### Authentication Endpoints
|
||||
|
||||
| Method | Path | Description |
|
||||
| ------ | --------------- | ----------- |
|
||||
| `GET` | `/pages/login` | Login page |
|
||||
| `POST` | `/pages/login` | Login form submission |
|
||||
| `GET` | `/pages/login` | Login page (not rate limited; the limiter applies to POST only) |
|
||||
| `POST` | `/pages/login` | Login form submission (5 per minute per bucket, then 429) |
|
||||
| `POST` | `/pages/logout` | Logout (destroys session) |
|
||||
|
||||
#### Authenticated Endpoints
|
||||
@@ -989,6 +1124,7 @@ abuse limit later; they are tracked as future work.
|
||||
| Method | Path | Description |
|
||||
| ------ | ------------------------ | ----------- |
|
||||
| `GET` | `/user/{username}` | User profile page |
|
||||
| `POST` | `/user/{username}/password` | Change the user's password (5 per minute per bucket, then 429) |
|
||||
| `GET` | `/sources` | List user's webhooks |
|
||||
| `GET` | `/sources/new` | Create webhook form |
|
||||
| `POST` | `/sources/new` | Create webhook submission |
|
||||
@@ -998,13 +1134,17 @@ abuse limit later; they are tracked as future work.
|
||||
| `POST` | `/source/{id}/delete` | Delete webhook |
|
||||
| `GET` | `/source/{id}/logs` | Webhook event logs |
|
||||
| `POST` | `/source/{id}/entrypoints` | Add entrypoint to webhook |
|
||||
| `POST` | `/source/{id}/entrypoints/{entrypointID}/delete` | Delete an entrypoint |
|
||||
| `POST` | `/source/{id}/entrypoints/{entrypointID}/toggle` | Enable or disable an entrypoint |
|
||||
| `POST` | `/source/{id}/targets` | Add target to webhook |
|
||||
| `POST` | `/source/{id}/targets/{targetID}/delete` | Delete a target |
|
||||
| `POST` | `/source/{id}/targets/{targetID}/toggle` | Enable or disable a target |
|
||||
|
||||
#### Infrastructure Endpoints
|
||||
|
||||
| Method | Path | Description |
|
||||
| ------ | ---------- | ----------- |
|
||||
| `GET` | `/metrics` | Prometheus metrics (requires basic auth) |
|
||||
| `GET` | `/metrics` | Prometheus metrics, behind basic auth. The route is registered only when `METRICS_USERNAME` is set; otherwise it does not exist and returns 404 |
|
||||
|
||||
#### API (Planned)
|
||||
|
||||
@@ -1018,8 +1158,10 @@ abuse limit later; they are tracked as future work.
|
||||
| `GET` | `/api/v1/webhooks/{id}/events` | List events for webhook |
|
||||
| `POST` | `/api/v1/events/{id}/redeliver`| Redeliver an event |
|
||||
|
||||
API authentication will use API keys passed via `Authorization: Bearer
|
||||
<key>` header.
|
||||
None of these exist yet. `/api/v1` is mounted with no routes, so every
|
||||
path under it returns 404 today. API authentication will use API keys
|
||||
passed via `Authorization: Bearer <key>` header; no Bearer middleware
|
||||
is implemented either.
|
||||
|
||||
### Package Layout
|
||||
|
||||
@@ -1047,16 +1189,28 @@ webhooker/
|
||||
│ │ ├── model_delivery_result.go # DeliveryResult entity (per-webhook DB)
|
||||
│ │ ├── model_apikey.go # APIKey entity
|
||||
│ │ ├── password.go # Argon2id hashing and verification
|
||||
│ │ ├── retention.go # Retention reaper (per-webhook event expiry)
|
||||
│ │ ├── testing.go # NewTestDatabase: wrapper for tests, no fx lifecycle
|
||||
│ │ └── webhook_db_manager.go # Per-webhook DB lifecycle manager
|
||||
│ ├── globals/
|
||||
│ │ └── globals.go # Build-time variables (appname, version, arch)
|
||||
│ ├── delivery/
|
||||
│ │ ├── engine.go # Event-driven delivery engine (channel + timer based)
|
||||
│ │ ├── circuit_breaker.go # Per-target circuit breaker for HTTP targets with retries
|
||||
│ │ ├── circuit_breaker.go # Per-target circuit breaker for http/slack targets with retries
|
||||
│ │ ├── target.go # Target interface, Task, Scheduler
|
||||
│ │ ├── target_http.go # HTTP target (retries, circuit breaker)
|
||||
│ │ ├── target_slack.go # Slack/Mattermost incoming-webhook target
|
||||
│ │ ├── target_database.go # Database archive target
|
||||
│ │ ├── target_database_archive.go # Archive file lifecycle and pruning
|
||||
│ │ ├── target_log.go # Log target (stdout)
|
||||
│ │ ├── target_config_view.go # Masked target config for templates
|
||||
│ │ ├── archive_sweeper.go # Periodic pruning of idle archives
|
||||
│ │ ├── url_mask.go # Strips credentials from *url.Error
|
||||
│ │ └── ssrf.go # SSRF prevention (IP validation, safe HTTP transport)
|
||||
│ ├── handlers/
|
||||
│ │ ├── handlers.go # Base handler struct, JSON helpers, template rendering
|
||||
│ │ ├── auth.go # Login, logout handlers
|
||||
│ │ ├── event_log_view.go # Event log projection, byte-capped in SQL
|
||||
│ │ ├── healthcheck.go # Health check handler
|
||||
│ │ ├── index.go # Index page handler
|
||||
│ │ ├── profile.go # User profile handler
|
||||
@@ -1064,25 +1218,34 @@ webhooker/
|
||||
│ │ └── webhook.go # Webhook receiver handler
|
||||
│ ├── healthcheck/
|
||||
│ │ └── healthcheck.go # Health check service (uptime, version)
|
||||
│ ├── lifecycle/
|
||||
│ │ └── lifecycle.go # Shared stop-hook waiter, bounded by the stop context
|
||||
│ ├── logger/
|
||||
│ │ └── logger.go # slog setup with TTY detection
|
||||
│ ├── middleware/
|
||||
│ │ ├── middleware.go # Logging, CORS, Auth, Metrics, MetricsAuth, SecurityHeaders, MaxBodySize
|
||||
│ │ ├── csrf.go # CSRF protection middleware (gorilla/csrf)
|
||||
│ │ └── ratelimit.go # Per-IP rate limiting middleware (go-chi/httprate)
|
||||
│ │ ├── ratelimit.go # Per-IP rate limiting middleware (go-chi/httprate)
|
||||
│ │ └── testing.go # NewForTest: Middleware without the fx lifecycle
|
||||
│ ├── server/
|
||||
│ │ ├── server.go # Server struct, fx lifecycle, signal handling
|
||||
│ │ ├── http.go # HTTP server setup with timeouts
|
||||
│ │ └── routes.go # All route definitions
|
||||
│ └── session/
|
||||
│ └── session.go # Cookie-based session management
|
||||
│ ├── session.go # Cookie-based session management
|
||||
│ └── testing.go # NewForTest: Session without the fx lifecycle
|
||||
├── static/
|
||||
│ ├── static.go # //go:embed directive
|
||||
│ ├── css/style.css # Custom stylesheet (system font stack, card effects, layout)
|
||||
│ └── js/app.js # Client-side JavaScript (minimal bootstrap)
|
||||
├── templates/ # Go HTML templates (base, index, login, etc.)
|
||||
├── Dockerfile # Multi-stage: lint, build+test, then Alpine runtime
|
||||
├── Makefile # fmt, lint, test, check, build, docker targets
|
||||
│ ├── css/input.css # Tailwind input, source for tailwind.css (make css)
|
||||
│ ├── css/tailwind.css # Generated stylesheet the pages load
|
||||
│ ├── css/style.css # Older hand-written stylesheet, no longer loaded
|
||||
│ ├── js/app.js # Progressive-enhancement copy-to-clipboard
|
||||
│ ├── js/alpine.min.js # Alpine.js, fetched by script/fetch-assets, not committed
|
||||
│ └── vendor.sha256 # Pinned hashes the fetched assets are verified against
|
||||
├── templates/ # Go HTML templates (base, login, sources, etc.)
|
||||
├── script/ # Scripts to Rule Them All entrypoints
|
||||
├── Dockerfile # Three stages: lint, test+build, Alpine runtime
|
||||
├── Makefile # 10 of 16 targets shim script/; 6 are inline
|
||||
├── go.mod / go.sum
|
||||
└── .golangci.yml # Linter configuration
|
||||
```
|
||||
@@ -1098,21 +1261,27 @@ Components are wired via Uber fx in this order:
|
||||
user seed
|
||||
5. `database.NewWebhookDBManager` — Per-webhook event database
|
||||
lifecycle manager
|
||||
6. `healthcheck.New` — Health check service
|
||||
7. `session.New` — Cookie-based session manager (key from database)
|
||||
8. `handlers.New` — HTTP handlers
|
||||
9. `middleware.New` — HTTP middleware
|
||||
10. `delivery.New` — Event-driven delivery engine
|
||||
11. `delivery.Engine` → `handlers.DeliveryNotifier` — interface bridge
|
||||
12. `server.New` — HTTP server and router
|
||||
6. `database.NewRetentionReaper` — Per-webhook event retention sweep
|
||||
7. `healthcheck.New` — Health check service
|
||||
8. `session.New` — Cookie-based session manager (key from database)
|
||||
9. `handlers.New` — HTTP handlers
|
||||
10. `middleware.New` — HTTP middleware
|
||||
11. `delivery.New` — Event-driven delivery engine
|
||||
12. `delivery.NewArchiveSweeper` — Periodic pruning of idle archives
|
||||
13. `delivery.Engine` → `delivery.Notifier` — interface bridge
|
||||
14. `delivery.Engine` → `delivery.WebhookEvictor` — interface bridge so
|
||||
deleting a webhook releases its archive writer
|
||||
15. `server.New` — HTTP server and router
|
||||
|
||||
The server starts via `fx.Invoke(func(*server.Server, *delivery.Engine)
|
||||
{})` which triggers the fx lifecycle hooks in dependency order. The
|
||||
`DeliveryNotifier` interface allows the webhook handler to send
|
||||
self-contained `DeliveryTask` slices to the engine without a direct
|
||||
The server starts via `fx.Invoke(func(*server.Server, *delivery.Engine,
|
||||
*database.RetentionReaper, *delivery.ArchiveSweeper) {})`, which
|
||||
triggers the fx lifecycle hooks in dependency order. The
|
||||
`delivery.Notifier` interface allows the webhook handler to send
|
||||
self-contained `delivery.Task` slices to the engine without a direct
|
||||
package dependency. Each task carries all target config and event data
|
||||
inline (for bodies ≤16KB), so the engine can deliver without reading
|
||||
from any database — it only writes to record results.
|
||||
inline (for bodies under 16 KiB, `delivery.MaxInlineBodySize`), so the
|
||||
engine can deliver without reading from any database — it only writes
|
||||
to record results.
|
||||
|
||||
### Middleware Stack
|
||||
|
||||
@@ -1138,16 +1307,32 @@ CSRF middleware in every one of those route groups, because
|
||||
gorilla/csrf parses the form; if the cap were installed after it, form
|
||||
parsing would run under net/http's 10 MB default and the 1 MB limit
|
||||
would never apply. A request that declares a `Content-Length` over the
|
||||
limit is answered with `413 Request Entity Too Large` before any other
|
||||
middleware or handler runs; a chunked request, or one that lies about
|
||||
its length, is hard-capped by `http.MaxBytesReader` and fails
|
||||
downstream at form-parse time.
|
||||
limit is answered with `413 Request Entity Too Large` without its body
|
||||
being read and without reaching CSRF, the route group's remaining
|
||||
middleware, or the handler. It is not rejected before *any* other
|
||||
middleware, though: the global entries listed above all run first, so
|
||||
such a request is still logged and given the security headers — and
|
||||
counted in the metrics, on a deployment where `METRICS_USERNAME` is
|
||||
set and the Metrics middleware is therefore registered at all. The
|
||||
rejection itself is logged at `WARN` with the method, path and
|
||||
declared length. A chunked request, or
|
||||
one that lies about its length, is hard-capped by
|
||||
`http.MaxBytesReader` and fails downstream at form-parse time.
|
||||
|
||||
Those same four route groups then apply **CSRF** and **NoCache**
|
||||
(`Cache-Control: no-store`, `Pragma: no-cache`), and every group except
|
||||
`/pages` applies **RequireAuth**. The rate limiters are per-route
|
||||
rather than global: **LoginRateLimit** on `/pages/login`,
|
||||
**PasswordChangeRateLimit** on `/user/{username}/password`, and
|
||||
**ReceiverRateLimit** on `/webhook/{uuid}`.
|
||||
|
||||
### Authentication
|
||||
|
||||
- **Web UI:** Cookie-based sessions using gorilla/sessions with
|
||||
encrypted cookies. Sessions are configured with HttpOnly, SameSite
|
||||
Lax, and Secure (in production). Session lifetime is 7 days.
|
||||
Lax, and Secure (in production). Absolute session lifetime is 7 days,
|
||||
with a sliding idle timeout on top of it (see
|
||||
[Sessions](#sessions)).
|
||||
- **API (planned):** API key authentication via `Authorization: Bearer`
|
||||
header. API keys are stored per-user with usage tracking
|
||||
(`last_used_at`).
|
||||
@@ -1179,33 +1364,125 @@ downstream at form-parse time.
|
||||
IPs before connecting, preventing DNS rebinding attacks)
|
||||
- **Login rate limiting** via [go-chi/httprate](https://github.com/go-chi/httprate):
|
||||
sliding-window rate limiter on the login endpoint, 5 POST attempts
|
||||
per minute per bucket, to slow brute-force attacks. The bucket is per
|
||||
client IP only when `TRUSTED_PROXIES` names the reverse proxy;
|
||||
unset, every client shares one bucket and the login becomes remotely
|
||||
deniable (see [Rate Limiting](#rate-limiting))
|
||||
per minute per bucket, to slow brute-force attacks. GET requests to
|
||||
the login page are not limited. The password-change endpoint carries
|
||||
the same 5-per-minute limit. The bucket is per client IP only when
|
||||
`TRUSTED_PROXIES` names the reverse proxy; unset, every client
|
||||
shares one bucket and the login becomes remotely deniable (see
|
||||
[Rate Limiting](#rate-limiting)). webhooker warns at startup
|
||||
whenever `TRUSTED_PROXIES` is empty
|
||||
- Prometheus metrics behind basic auth
|
||||
- Static assets embedded in binary (no filesystem access needed at
|
||||
runtime)
|
||||
- Container runs as non-root user (UID 1000)
|
||||
- GORM soft deletes on all entities (data preserved for audit)
|
||||
- GORM soft deletes on every entity that carries `BaseModel`, which is
|
||||
all of them but `Setting` (data preserved for audit)
|
||||
|
||||
### Shutdown
|
||||
|
||||
On SIGINT or SIGTERM, fx runs the registered stop hooks in reverse
|
||||
dependency order under a **5 second budget** (`fx.StopTimeout` in
|
||||
`cmd/webhooker/main.go`). That budget covers the whole sequence, not
|
||||
each hook. The order, read off the fx stop-hook log:
|
||||
|
||||
1. `ArchiveSweeper`
|
||||
2. `RetentionReaper`
|
||||
3. `server` — the HTTP drain, bounded separately by
|
||||
`server.ShutdownTimeout` (**3 seconds**), then a Sentry flush if
|
||||
`SENTRY_DSN` is set
|
||||
4. `delivery.Engine`
|
||||
5. `healthcheck`
|
||||
6. `WebhookDBManager`
|
||||
7. the database close
|
||||
|
||||
The two components that can realistically hold the budget run
|
||||
first: a retention sweep or an archive prune caught mid-tick each
|
||||
waits on its `WaitGroup` bounded by the stop context, so a wedge
|
||||
there consumes the 5 seconds before the HTTP server hook is ever
|
||||
entered. The hooks after the server are microsecond-scale in normal
|
||||
operation.
|
||||
|
||||
The HTTP drain budget is deliberately **shorter** than the sequence
|
||||
budget. Were the two equal, a drain that used its whole budget would
|
||||
exhaust the sequence budget at the instant it finished, and every
|
||||
later hook — the delivery engine, the healthcheck, the webhook DB
|
||||
manager and the database close — would be skipped in exactly the
|
||||
case where the drain mattered. 3 seconds leaves 2 seconds
|
||||
(`server.TailHookReserve`) for the tail, which is far more than the
|
||||
microseconds it needs.
|
||||
|
||||
That reserve belongs to the tail hooks, not to the server hook, and
|
||||
the Sentry flush is what could take it: it runs after the drain
|
||||
**inside the same hook**, and `sentry.Flush` takes a bare duration
|
||||
and honours no context, so an unreachable Sentry endpoint would add
|
||||
its own timeout on top of a full-length drain and consume the whole
|
||||
sequence budget by itself. It is therefore clamped to whatever is
|
||||
left on the stop context minus the reserve, and skipped when that
|
||||
leaves too little to be worth attempting — so a full-length drain
|
||||
means Sentry events are dropped rather than the database close being
|
||||
skipped.
|
||||
|
||||
This does not make the database close unconditional: a wedged
|
||||
`ArchiveSweeper` or `RetentionReaper` still runs first and can
|
||||
consume the whole budget on its own.
|
||||
|
||||
The value is chosen to sit inside the container stop grace period.
|
||||
Docker's default `docker stop` grace is 10 seconds and the Dockerfile
|
||||
sets no `STOPSIGNAL` or grace override, so the process must be gone
|
||||
before that. fx's own default is 15 seconds, which is past the grace:
|
||||
the container would be SIGKILLed (exit 137) before the bound could
|
||||
fire, and nothing that depends on it — including the
|
||||
`shutdown timed out, goroutines still running` error log that tells
|
||||
an operator a component is wedged — would ever be reached.
|
||||
|
||||
Two operational consequences follow from bounding the sequence:
|
||||
|
||||
- **A wedged component aborts the rest of the shutdown.** fx checks
|
||||
the stop context before each remaining hook and returns outright
|
||||
once it has expired, skipping the hooks it has not reached. If the
|
||||
first-stopped component consumes the whole budget, the later hooks
|
||||
never run — **the database close among them**. SQLite is crash-safe,
|
||||
so this is not corruption, but it is not a clean close either.
|
||||
- **Lowering the grace below 5 seconds reintroduces the silent
|
||||
truncation.** `docker stop --time`, Compose's `stop_grace_period`,
|
||||
or Kubernetes' `terminationGracePeriodSeconds` set under 5 seconds
|
||||
put SIGKILL back in front of the bound, and the process dies with
|
||||
no shutdown diagnostics at all. Keep the deployment's grace above
|
||||
the stop timeout.
|
||||
|
||||
### Docker
|
||||
|
||||
The Dockerfile uses a multi-stage build:
|
||||
The Dockerfile uses a three-stage build. Each stage is pinned by
|
||||
digest, and the two check stages are separate images so the linter's
|
||||
version is fixed independently of the compiler's:
|
||||
|
||||
1. **Builder stage** (Debian-based `golang:1.24`) — installs
|
||||
golangci-lint, downloads dependencies, copies source, runs `make
|
||||
check` (format verification, linting, tests, compilation).
|
||||
2. **Runtime stage** (`alpine:3.21`) — copies the binary, creates the
|
||||
`/var/lib/webhooker` directory for all SQLite databases, runs as
|
||||
non-root user, exposes port 8080, includes a health check.
|
||||
1. **Lint stage** (`golangci/golangci-lint:v2.12.2`, Debian-based) —
|
||||
installs `make`, downloads dependencies, copies the source, and runs
|
||||
`make fmt-check` then `make lint`.
|
||||
2. **Builder stage** (`golang:1.26.1-bookworm`) — depends on the lint
|
||||
stage passing (it copies a file from it), runs `script/fetch-assets`
|
||||
to download and verify the third-party browser assets, then runs
|
||||
`make test` and `make build`, and finally rebuilds the binary with
|
||||
`CGO_ENABLED=1` and static linking so it runs on musl.
|
||||
3. **Runtime stage** (`alpine:3.21`) — copies the static binary,
|
||||
creates the `/var/lib/webhooker` directory for all SQLite databases,
|
||||
runs as the non-root `webhooker` user (UID 1000), exposes port 8080,
|
||||
and includes a health check against `/.well-known/healthcheck`.
|
||||
|
||||
The builder uses Debian rather than Alpine because GORM's SQLite
|
||||
dialect pulls in CGO-dependent headers at compile time. The runtime
|
||||
binary is statically linked and runs on Alpine.
|
||||
Both check stages use Debian rather than Alpine because
|
||||
`gorm.io/driver/sqlite` pulls in `mattn/go-sqlite3`, which needs CGO
|
||||
and does not compile against musl. Only the final binary is statically
|
||||
linked, which is what lets it run on the Alpine runtime image.
|
||||
|
||||
`docker build .` is the CI gate — if it passes, the code is formatted,
|
||||
linted, tested, and compiled.
|
||||
`script/cibuild` — `docker build .` — is the CI gate: the four check
|
||||
targets run inside the image, so a build that succeeds is a repo that
|
||||
is formatted, linted, tested and compiled. Only `script/cibuild` and
|
||||
`script/docker` involve Docker. `script/lint`, and therefore
|
||||
`make lint` and `make check`, run whatever `golangci-lint` is on the
|
||||
host, which can be a different version from the pinned one — so the
|
||||
container is the authoritative lint result
|
||||
([issue #109](https://git.eeqj.de/sneak/webhooker/issues/109) tracks
|
||||
routing local linting through it as well).
|
||||
|
||||
#### CI gate honesty
|
||||
|
||||
@@ -1221,16 +1498,17 @@ the hash of the last commit that touched the build context, so:
|
||||
`make fmt-check`, `make lint`, `make test`, and `make build`. A run
|
||||
that reports success ran them.
|
||||
- A docs-only commit leaves the fingerprint unchanged — `.dockerignore`
|
||||
excludes `*.md` and `LICENSE` from the context anyway — so the image
|
||||
replays from cache and costs seconds.
|
||||
excludes `*.md`, `LICENSE` and `.editorconfig` from the context
|
||||
anyway — so the image replays from cache and costs seconds.
|
||||
|
||||
The module download layer sits above `COPY . .` and stays cached either
|
||||
way.
|
||||
|
||||
The workflow's first step covers a second way the gate lied: Gitea
|
||||
cancels an in-flight run when a newer commit lands on the same branch
|
||||
and records that cancellation as a `failure` status, marking a commit
|
||||
red that was never tested. Cancellation is unconditional server-side for
|
||||
A separate workflow step, run before the fingerprint is written, covers
|
||||
a second way the gate lied: Gitea cancels an in-flight run when a newer
|
||||
commit lands on the same branch and records that cancellation as a
|
||||
`failure` status, marking a commit red that was never tested.
|
||||
Cancellation is unconditional server-side for
|
||||
push events, so the superseding run rewrites the exact
|
||||
`Has been cancelled` status to `skipped`. Genuine failures are never
|
||||
touched.
|
||||
|
||||
22
TODO.md
22
TODO.md
@@ -25,13 +25,17 @@ password change flow (#65), policy compliance (#6), pinned lint tooling
|
||||
(#55), and fail-loud configuration parsing (#80).
|
||||
|
||||
`next` holds the completed 1.0.0 milestone: every issue in it is closed,
|
||||
and it is verified green both by CI and by cache-defeated container
|
||||
runs. The two were only made to mean the same thing this cycle — before
|
||||
#119, a warm layer cache let the gate report success without executing
|
||||
anything, and replayed the previous build's console log so the lie
|
||||
looked like a real run. Note: TODO.md was deliberately deleted from this
|
||||
repo in f9a9569 (2026-03-01, #6); its content was folded into the README
|
||||
TODO section, which this draft reconstructs as of 2026-07-06.
|
||||
and it is verified green by cache-defeated container runs
|
||||
(`docker build --no-cache-filter=lint --no-cache-filter=builder`). The
|
||||
CI status is not independently claimed here: a superseded run is
|
||||
recorded as `skipped` and still rolls up green, so a commit status on
|
||||
`next` does not by itself evidence an executed check (#152). Before
|
||||
#119, a warm layer cache also let the gate report success without
|
||||
executing anything, and replayed the previous build's console log so
|
||||
the lie looked like a real run. Note: `TODO.md` was deliberately
|
||||
deleted from this repo in f9a9569 (2026-03-01, #6); its content was
|
||||
folded into the README TODO section, which this draft reconstructs as
|
||||
of 2026-07-06.
|
||||
|
||||
# Next Step
|
||||
|
||||
@@ -190,7 +194,9 @@ rate-limit keys should bucket by `/64`).
|
||||
- OpenAPI specification
|
||||
- Analytics dashboard: success rates, response times, volume
|
||||
- A remember-me option at login
|
||||
- Password change and reset flow
|
||||
- Password reset flow for a forgotten password. The authenticated
|
||||
password *change* flow already landed on `main` (#65); reset does not
|
||||
exist
|
||||
- Later, nice to have
|
||||
- email delivery target type
|
||||
- SNS and S3 delivery targets
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"go.uber.org/fx"
|
||||
"sneak.berlin/go/webhooker/internal/config"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
@@ -15,6 +17,33 @@ import (
|
||||
"sneak.berlin/go/webhooker/internal/session"
|
||||
)
|
||||
|
||||
// stopTimeout bounds the whole fx stop sequence, not each hook.
|
||||
//
|
||||
// fx defaults to 15s, which is longer than Docker's 10s default
|
||||
// stop grace: the container would be SIGKILLed before the bound
|
||||
// could fire, so nothing bounded by it would ever be observed.
|
||||
// 5s leaves headroom inside that grace for signal delivery and
|
||||
// process exit; the observed wedge case already exits at ~5.3s,
|
||||
// so a larger bound would trade a rare skipped database close for
|
||||
// a more common hard kill.
|
||||
//
|
||||
// The server's stop hook must fit inside it with room to spare: a
|
||||
// hook that used the whole budget would exhaust it at that instant,
|
||||
// and fx would skip every hook after the server — the delivery
|
||||
// engine, the healthcheck, the webhook DB manager and the database
|
||||
// close. That hook is the 3s HTTP drain plus the Sentry flush that
|
||||
// follows it in the same hook, so the flush is clamped to the stop
|
||||
// context's remaining time less server.TailHookReserve rather than
|
||||
// running for its own fixed 2s; the reserve is what the tail hooks
|
||||
// live on, and they are microsecond-scale in normal operation.
|
||||
// TestStopTimeout_LeavesHeadroomForTailHooks pins the arithmetic
|
||||
// across every drain length.
|
||||
//
|
||||
// This does not make the database close unconditional: the
|
||||
// ArchiveSweeper and RetentionReaper hooks run before the server
|
||||
// and can still consume the whole budget on their own.
|
||||
const stopTimeout = 5 * time.Second
|
||||
|
||||
// Build-time variables set via -ldflags.
|
||||
//
|
||||
//nolint:gochecknoglobals // Build-time variables injected by the linker.
|
||||
@@ -27,7 +56,14 @@ func main() {
|
||||
globals.Appname = appname
|
||||
globals.Version = version
|
||||
|
||||
fx.New(
|
||||
newApp().Run()
|
||||
}
|
||||
|
||||
// newApp builds the application graph. It is separate from main so
|
||||
// a test can assert the options it carries.
|
||||
func newApp() *fx.App {
|
||||
return fx.New(
|
||||
fx.StopTimeout(stopTimeout),
|
||||
fx.Provide(
|
||||
globals.New,
|
||||
logger.New,
|
||||
@@ -60,5 +96,5 @@ func main() {
|
||||
) {
|
||||
},
|
||||
),
|
||||
).Run()
|
||||
)
|
||||
}
|
||||
|
||||
75
cmd/webhooker/main_test.go
Normal file
75
cmd/webhooker/main_test.go
Normal file
@@ -0,0 +1,75 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/webhooker/internal/server"
|
||||
)
|
||||
|
||||
// dockerStopGrace is Docker's default `docker stop` grace period.
|
||||
// The Dockerfile sets no STOPSIGNAL or grace override, so this is
|
||||
// the deadline the container is actually held to, and the fx stop
|
||||
// timeout has to fit inside it with room for signal delivery and
|
||||
// process exit.
|
||||
const dockerStopGrace = 10 * time.Second
|
||||
|
||||
// TestNewApp_StopTimeout pins the fx stop timeout. Without the
|
||||
// explicit fx.StopTimeout option the app reads fx's 15s
|
||||
// DefaultTimeout, which exceeds dockerStopGrace: the container is
|
||||
// SIGKILLed before the bound fires and every shutdown hook bounded
|
||||
// by it — including the operator-facing timeout log — becomes
|
||||
// unreachable in the image this repo produces.
|
||||
//
|
||||
// fx.New applies options before it executes invokes, so the timeout
|
||||
// is set whether or not the graph itself can be constructed here.
|
||||
func TestNewApp_StopTimeout(t *testing.T) {
|
||||
t.Setenv("DATA_DIR", t.TempDir())
|
||||
|
||||
got := newApp().StopTimeout()
|
||||
|
||||
require.Equal(t, stopTimeout, got)
|
||||
require.Less(t, got, dockerStopGrace)
|
||||
}
|
||||
|
||||
// tailHeadroom is the slack the fx stop budget must keep beyond the
|
||||
// server stop hook. The hooks that run after the server — the
|
||||
// delivery engine, the healthcheck, the webhook DB manager and the
|
||||
// database close — are microsecond-scale in normal operation, so
|
||||
// this is generous for them.
|
||||
const tailHeadroom = 2 * time.Second
|
||||
|
||||
// TestStopTimeout_LeavesHeadroomForTailHooks pins the relationship
|
||||
// between the server's stop hook and the fx stop budget. fx bounds
|
||||
// the whole stop sequence, and returns without running its
|
||||
// remaining hooks once the stop context has expired. If the hook
|
||||
// could use the entire budget, every later hook — the database close
|
||||
// included — would be skipped in exactly the case where the drain
|
||||
// mattered.
|
||||
//
|
||||
// The hook is not just the HTTP drain: a Sentry flush follows it in
|
||||
// the same hook, and sentry.Flush honours no context, so both halves
|
||||
// have to be counted. The sweep walks every drain length the hook
|
||||
// can produce, since a shorter drain leaves the flush more room and
|
||||
// the worst case is not necessarily at either extreme.
|
||||
//
|
||||
// Shrinking either budget, or unbounding the flush again, must fail
|
||||
// here rather than silently recreating a hook that swallows the
|
||||
// whole sequence.
|
||||
func TestStopTimeout_LeavesHeadroomForTailHooks(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
require.Less(t, server.ShutdownTimeout, stopTimeout)
|
||||
|
||||
const step = 10 * time.Millisecond
|
||||
|
||||
for drain := time.Duration(0); drain <= server.ShutdownTimeout; drain += step {
|
||||
hook := drain + server.SentryFlushBudget(stopTimeout-drain)
|
||||
|
||||
require.LessOrEqual(
|
||||
t, hook+tailHeadroom, stopTimeout,
|
||||
"a %s drain leaves the tail hooks short", drain,
|
||||
)
|
||||
}
|
||||
}
|
||||
@@ -422,33 +422,43 @@ func loadFromEnv() (*Config, error) {
|
||||
}, nil
|
||||
}
|
||||
|
||||
// warnSharedRateLimitBucket logs a startup warning when a production
|
||||
// deployment leaves TRUSTED_PROXIES empty.
|
||||
// warnSharedRateLimitBucket logs a startup warning whenever
|
||||
// TRUSTED_PROXIES is empty, in any environment.
|
||||
//
|
||||
// With no trusted proxies every rate limiter keys on the connecting
|
||||
// peer's address. A production deployment is required to run behind a
|
||||
// TLS-terminating reverse proxy, and the peer is then that proxy for
|
||||
// every request, so all clients share one bucket per limiter. The
|
||||
// peer's address. Whether that is harmless or dangerous depends on
|
||||
// what is in front of the process, which this code cannot observe:
|
||||
// with nothing in front, the peer is the client and the limits are
|
||||
// per-client as intended; behind a reverse proxy the peer is the proxy
|
||||
// for every request, so all clients share one bucket per limiter. The
|
||||
// login limiter's bucket is the dangerous one: any remote client can
|
||||
// keep it full, which denies the only administrative login to
|
||||
// everyone until the process restarts.
|
||||
// keep it full, which denies the only administrative login to everyone
|
||||
// until the process restarts.
|
||||
//
|
||||
// The warning is deliberately not gated on WEBHOOKER_ENVIRONMENT. That
|
||||
// variable defaults to dev, so gating on it would silence the warning
|
||||
// for exactly the operator who forgot to configure the deployment —
|
||||
// the case it exists to catch.
|
||||
//
|
||||
// The default of trusting nobody is deliberate — trusting forwarded
|
||||
// headers from arbitrary peers lets any client choose its own bucket —
|
||||
// so this warns rather than failing startup or changing the key.
|
||||
func (c *Config) warnSharedRateLimitBucket(log *slog.Logger) {
|
||||
if !c.IsProd() || len(c.TrustedProxies) > 0 {
|
||||
if len(c.TrustedProxies) > 0 {
|
||||
return
|
||||
}
|
||||
|
||||
log.Warn(
|
||||
"TRUSTED_PROXIES is empty: rate limits key on the "+
|
||||
"connecting peer, so behind the reverse proxy a "+
|
||||
"production deployment runs behind, every client "+
|
||||
"shares one bucket per limit. Any remote client can "+
|
||||
"then keep the login limit full and deny the admin "+
|
||||
"login, the only administrative path, until restart. "+
|
||||
"Set TRUSTED_PROXIES to your reverse proxy's address.",
|
||||
"TRUSTED_PROXIES is empty: every rate limit keys on the "+
|
||||
"connecting peer's address. With nothing proxying to "+
|
||||
"this process that is the client itself and the limits "+
|
||||
"are per-client as intended. Behind a reverse proxy the "+
|
||||
"peer is the proxy on every request, so all clients "+
|
||||
"share one bucket per limit and any remote client can "+
|
||||
"keep the login limit full, denying the admin login — "+
|
||||
"the only administrative path — until restart. If "+
|
||||
"anything proxies to this process, set TRUSTED_PROXIES "+
|
||||
"to its address.",
|
||||
"environment", c.Environment,
|
||||
"trustedProxies", len(c.TrustedProxies),
|
||||
)
|
||||
@@ -491,6 +501,10 @@ func New(lc fx.Lifecycle, params ConfigParams) (*Config, error) {
|
||||
"maintenanceMode", s.MaintenanceMode,
|
||||
"dataDir", s.DataDir,
|
||||
"retentionSweepInterval", s.RetentionSweepInterval.String(),
|
||||
// Logged because a perfectly valid non-positive value here
|
||||
// disables idle expiry entirely, and that is worth showing
|
||||
// back to the operator.
|
||||
"sessionIdleTimeout", s.SessionIdleTimeout.String(),
|
||||
"receiverRateLimit", s.ReceiverRateLimit,
|
||||
"trustedProxies", len(s.TrustedProxies),
|
||||
"hasSentryDSN", s.SentryDSN != "",
|
||||
|
||||
@@ -628,10 +628,12 @@ func testTrustedProxiesSuccess(
|
||||
}
|
||||
|
||||
// TestSharedRateLimitBucketWarning covers the startup warning that
|
||||
// tells an operator their production deployment shares one rate-limit
|
||||
// bucket between every client, which makes the admin login remotely
|
||||
// deniable. It must fire when TRUSTED_PROXIES is empty in production
|
||||
// and stay quiet otherwise.
|
||||
// tells an operator a deployment behind a reverse proxy shares one
|
||||
// rate-limit bucket between every client, which makes the admin login
|
||||
// remotely deniable. It must fire whenever TRUSTED_PROXIES is empty,
|
||||
// in any environment: WEBHOOKER_ENVIRONMENT defaults to dev, so gating
|
||||
// on it would silence the warning for exactly the operator who never
|
||||
// configured the deployment. It stays quiet once proxies are named.
|
||||
func TestSharedRateLimitBucketWarning(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
@@ -651,11 +653,18 @@ func TestSharedRateLimitBucketWarning(t *testing.T) {
|
||||
expectWarning: false,
|
||||
},
|
||||
{
|
||||
// Development is not required to run behind a
|
||||
// reverse proxy, so the shared bucket the warning
|
||||
// describes is not the expected shape there.
|
||||
name: "dev without trusted proxies is quiet",
|
||||
// The default environment. An internet-exposed
|
||||
// deployment whose operator never set
|
||||
// WEBHOOKER_ENVIRONMENT lands here and has exactly
|
||||
// the exposure the warning announces.
|
||||
name: "dev without trusted proxies warns",
|
||||
environment: config.EnvironmentDev,
|
||||
expectWarning: true,
|
||||
},
|
||||
{
|
||||
name: "dev with trusted proxies is quiet",
|
||||
environment: config.EnvironmentDev,
|
||||
trustedProxies: cidrPrivateV4,
|
||||
expectWarning: false,
|
||||
},
|
||||
}
|
||||
@@ -697,8 +706,14 @@ func TestSharedRateLimitBucketWarning(t *testing.T) {
|
||||
|
||||
assert.Contains(t, logged, `"level":"WARN"`)
|
||||
assert.Contains(t, logged, "TRUSTED_PROXIES")
|
||||
assert.Contains(t, logged, "shares one bucket")
|
||||
assert.Contains(t, logged, "deny the admin login")
|
||||
assert.Contains(t, logged, "share one bucket")
|
||||
assert.Contains(t, logged, "denying the admin login")
|
||||
// The text must stay accurate for a developer with
|
||||
// nothing in front of the process, where an empty
|
||||
// list costs nothing.
|
||||
assert.Contains(
|
||||
t, logged, "nothing proxying to this process",
|
||||
)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
21
internal/lifecycle/export_test.go
Normal file
21
internal/lifecycle/export_test.go
Normal file
@@ -0,0 +1,21 @@
|
||||
package lifecycle
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
)
|
||||
|
||||
// WaitDone exposes waitDone to the external test package. Only the
|
||||
// unexported waiter can be handed a channel that is already closed
|
||||
// before the call, which is the state the preamble exists for;
|
||||
// through WaitForShutdown the waiter goroutine may or may not have
|
||||
// closed the channel yet, so the case is not reachable
|
||||
// deterministically from outside.
|
||||
func WaitDone(
|
||||
ctx context.Context,
|
||||
log *slog.Logger,
|
||||
component string,
|
||||
done <-chan struct{},
|
||||
) error {
|
||||
return waitDone(ctx, log, component, done)
|
||||
}
|
||||
@@ -38,6 +38,29 @@ func WaitForShutdown(
|
||||
wg.Wait()
|
||||
}()
|
||||
|
||||
return waitDone(ctx, log, component, done)
|
||||
}
|
||||
|
||||
// waitDone waits for done to close, bounded by ctx.
|
||||
//
|
||||
// The non-blocking preamble is load-bearing. When the component has
|
||||
// already drained and ctx has already expired, both cases of the
|
||||
// bounded select are ready and Go picks between them uniformly at
|
||||
// random, so a clean shutdown would be reported as a timeout about
|
||||
// half the time. Draining wins: the goroutines are gone, and there
|
||||
// is nothing left for the operator to act on.
|
||||
func waitDone(
|
||||
ctx context.Context,
|
||||
log *slog.Logger,
|
||||
component string,
|
||||
done <-chan struct{},
|
||||
) error {
|
||||
select {
|
||||
case <-done:
|
||||
return nil
|
||||
default:
|
||||
}
|
||||
|
||||
select {
|
||||
case <-done:
|
||||
return nil
|
||||
|
||||
@@ -37,6 +37,57 @@ func TestWaitForShutdown_DrainedGroup(t *testing.T) {
|
||||
)
|
||||
}
|
||||
|
||||
// racePasses is how many times the both-cases-ready race is run.
|
||||
// Without the preamble each pass is an independent coin flip, so
|
||||
// the probability of the whole loop passing by luck is 2^-N: at
|
||||
// this N the test is deterministic in practice, and it involves no
|
||||
// wall-clock waiting at all.
|
||||
const racePasses = 1000
|
||||
|
||||
// TestWaitDone_DrainedBeforeExpiredContext covers the case where a
|
||||
// component drained cleanly but the stop context had already
|
||||
// expired. Both select cases are ready, and Go chooses among ready
|
||||
// cases uniformly at random, so the drained case must be settled by
|
||||
// the preamble before the bounded select ever runs.
|
||||
func TestWaitDone_DrainedBeforeExpiredContext(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
done := make(chan struct{})
|
||||
close(done)
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
|
||||
for pass := range racePasses {
|
||||
require.NoErrorf(
|
||||
t,
|
||||
lifecycle.WaitDone(
|
||||
ctx, discardLogger(), "test component", done,
|
||||
),
|
||||
"pass %d reported a timeout for a drained component",
|
||||
pass,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// TestWaitDone_ExpiredContext pins the other side of the preamble:
|
||||
// an expired context with a component that has not drained is still
|
||||
// a timeout.
|
||||
func TestWaitDone_ExpiredContext(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
|
||||
err := lifecycle.WaitDone(
|
||||
ctx, discardLogger(), "test component",
|
||||
make(chan struct{}),
|
||||
)
|
||||
|
||||
require.ErrorIs(t, err, context.Canceled)
|
||||
require.ErrorContains(t, err, "test component")
|
||||
}
|
||||
|
||||
func TestWaitForShutdown_ContextExpires(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
658
internal/middleware/accesslog_test.go
Normal file
658
internal/middleware/accesslog_test.go
Normal file
@@ -0,0 +1,658 @@
|
||||
package middleware_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/go-chi/chi"
|
||||
chimw "github.com/go-chi/chi/middleware"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/webhooker/internal/config"
|
||||
"sneak.berlin/go/webhooker/internal/middleware"
|
||||
)
|
||||
|
||||
// floodRequests is the number of distinct invented paths each flood
|
||||
// test drives through the access log.
|
||||
const floodRequests = 64
|
||||
|
||||
// attackerMarker is embedded in every invented path. No access log
|
||||
// line for a redirected or rejected request may contain it.
|
||||
const attackerMarker = "QQATTACKERTEXTQQ"
|
||||
|
||||
// maxLineBytes bounds a single access log line whose client-supplied
|
||||
// fields are of ordinary size. Well above what the fixed fields need,
|
||||
// well below the length of the oversized input the amplification tests
|
||||
// send.
|
||||
const maxLineBytes = 1024
|
||||
|
||||
// maxCappedLineBytes bounds a single access log line when every
|
||||
// client-supplied field arrives oversized and is truncated to its
|
||||
// budget. This is the number the README quotes as the per-line cost an
|
||||
// operator sizes log storage against, and it is a bound on the
|
||||
// ENCODED line, which is what the operator's disk holds.
|
||||
const maxCappedLineBytes = 2560
|
||||
|
||||
// oversizedSegmentBytes is the length of the single attacker-chosen
|
||||
// path segment, query string or header used to show line size does not
|
||||
// track input size.
|
||||
const oversizedSegmentBytes = 8192
|
||||
|
||||
// tailMarker is placed at the END of an oversized header value, so its
|
||||
// absence from the log proves the value was truncated rather than
|
||||
// merely being short.
|
||||
const tailMarker = "QQTRUNCATEDTAILQQ"
|
||||
|
||||
// These mirror the middleware's own budgets, which are unexported.
|
||||
// They are duplicated rather than exported so that widening a budget
|
||||
// in the middleware has to be restated here deliberately.
|
||||
const (
|
||||
maxFieldBytes = 512
|
||||
maxRequestIDBytes = 128
|
||||
maxMethodBytes = 32
|
||||
truncationSuffix = "[truncated]"
|
||||
unmatchedRouteLiteral = "(unmatched)"
|
||||
)
|
||||
|
||||
// capturingMiddleware returns a Middleware whose logger writes JSON
|
||||
// lines into the returned buffer, so the access log can be asserted
|
||||
// on directly.
|
||||
func capturingMiddleware(t *testing.T) (*middleware.Middleware, *bytes.Buffer) {
|
||||
t.Helper()
|
||||
|
||||
buf := new(bytes.Buffer)
|
||||
log := slog.New(slog.NewJSONHandler(
|
||||
buf,
|
||||
&slog.HandlerOptions{Level: slog.LevelInfo},
|
||||
))
|
||||
|
||||
cfg := &config.Config{Environment: config.EnvironmentDev}
|
||||
|
||||
return middleware.NewForTest(log, cfg, nil), buf
|
||||
}
|
||||
|
||||
// capturingTextMiddleware is capturingMiddleware for the other handler
|
||||
// internal/logger can select: slog's text handler, which
|
||||
// internal/logger/logger.go installs when stderr is a tty. It escapes
|
||||
// differently from the JSON one, so the line bound has to be asserted
|
||||
// against both.
|
||||
func capturingTextMiddleware(
|
||||
t *testing.T,
|
||||
) (*middleware.Middleware, *bytes.Buffer) {
|
||||
t.Helper()
|
||||
|
||||
buf := new(bytes.Buffer)
|
||||
log := slog.New(slog.NewTextHandler(
|
||||
buf,
|
||||
&slog.HandlerOptions{Level: slog.LevelInfo},
|
||||
))
|
||||
|
||||
cfg := &config.Config{Environment: config.EnvironmentDev}
|
||||
|
||||
return middleware.NewForTest(log, cfg, nil), buf
|
||||
}
|
||||
|
||||
// accessLogRouter mirrors the production route shapes that an
|
||||
// unauthenticated client can reach: the public receiver, the
|
||||
// authenticated profile route (which redirects to login rather than
|
||||
// rejecting outright), the health check (which answers 200 to anyone,
|
||||
// behind no rate limiter at all), and a plain static route.
|
||||
func accessLogRouter(m *middleware.Middleware) *chi.Mux {
|
||||
router := chi.NewRouter()
|
||||
// Production registers RequestID ahead of Logging, and chi's
|
||||
// RequestID passes an inbound X-Request-Id header straight
|
||||
// through, so the request_id field is client-supplied too.
|
||||
router.Use(chimw.RequestID)
|
||||
router.Use(m.Logging())
|
||||
|
||||
router.Get(
|
||||
"/.well-known/healthcheck",
|
||||
func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(http.StatusOK)
|
||||
},
|
||||
)
|
||||
|
||||
router.HandleFunc(
|
||||
"/webhook/{uuid}",
|
||||
func(w http.ResponseWriter, r *http.Request) {
|
||||
// Stands in for the real handler: an unknown entrypoint
|
||||
// UUID 404s, a known one succeeds.
|
||||
if chi.URLParam(r, "uuid") != "known" {
|
||||
http.Error(w, "not found", http.StatusNotFound)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
w.WriteHeader(http.StatusOK)
|
||||
},
|
||||
)
|
||||
|
||||
router.Route("/user/{username}", func(r chi.Router) {
|
||||
r.Get("/", func(w http.ResponseWriter, r *http.Request) {
|
||||
http.Redirect(
|
||||
w, r, "/pages/login", http.StatusSeeOther,
|
||||
)
|
||||
})
|
||||
})
|
||||
|
||||
boom := func(w http.ResponseWriter, _ *http.Request) {
|
||||
http.Error(w, "boom", http.StatusInternalServerError)
|
||||
}
|
||||
|
||||
router.Get("/boom", boom)
|
||||
// The 5xx branch keeps the concrete path, so it needs a route that
|
||||
// answers 500 to a path of the client's choosing: that is where the
|
||||
// url field and the header fields are both at their budget on the
|
||||
// same line.
|
||||
router.Get("/boom/*", boom)
|
||||
|
||||
return router
|
||||
}
|
||||
|
||||
// accessLogEntries decodes the captured buffer into one map per
|
||||
// logged line, holding every line to maxLineBytes.
|
||||
func accessLogEntries(
|
||||
t *testing.T,
|
||||
buf *bytes.Buffer,
|
||||
) []map[string]any {
|
||||
t.Helper()
|
||||
|
||||
return accessLogEntriesWithin(t, buf, maxLineBytes)
|
||||
}
|
||||
|
||||
// accessLogEntriesWithin decodes the captured buffer into one map per
|
||||
// logged line, holding every line to bound bytes.
|
||||
func accessLogEntriesWithin(
|
||||
t *testing.T,
|
||||
buf *bytes.Buffer,
|
||||
bound int,
|
||||
) []map[string]any {
|
||||
t.Helper()
|
||||
|
||||
var entries []map[string]any
|
||||
|
||||
for line := range strings.SplitSeq(
|
||||
strings.TrimSpace(buf.String()), "\n",
|
||||
) {
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
require.LessOrEqual(
|
||||
t, len(line), bound,
|
||||
"access log line exceeded its bound",
|
||||
)
|
||||
|
||||
var entry map[string]any
|
||||
|
||||
require.NoError(t, json.Unmarshal([]byte(line), &entry))
|
||||
|
||||
entries = append(entries, entry)
|
||||
}
|
||||
|
||||
return entries
|
||||
}
|
||||
|
||||
// get drives one GET through the router.
|
||||
func get(t *testing.T, router *chi.Mux, target string) int {
|
||||
t.Helper()
|
||||
|
||||
return getWithHeaders(t, router, target, nil)
|
||||
}
|
||||
|
||||
// getWithHeaders drives one GET through the router with the supplied
|
||||
// request headers set.
|
||||
func getWithHeaders(
|
||||
t *testing.T,
|
||||
router *chi.Mux,
|
||||
target string,
|
||||
headers map[string]string,
|
||||
) int {
|
||||
t.Helper()
|
||||
|
||||
req := httptest.NewRequestWithContext(
|
||||
context.Background(), http.MethodGet, target, nil,
|
||||
)
|
||||
|
||||
for name, value := range headers {
|
||||
req.Header.Set(name, value)
|
||||
}
|
||||
|
||||
w := httptest.NewRecorder()
|
||||
router.ServeHTTP(w, req)
|
||||
|
||||
return w.Code
|
||||
}
|
||||
|
||||
// assertFloodIsBounded drives floodRequests distinct invented paths
|
||||
// built by pathFor and asserts every logged line names wantURL, that
|
||||
// none carries the invented text, and that the line count is exactly
|
||||
// one per request.
|
||||
func assertFloodIsBounded(
|
||||
t *testing.T,
|
||||
pathFor func(i int) string,
|
||||
wantStatus int,
|
||||
wantURL string,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
m, buf := capturingMiddleware(t)
|
||||
router := accessLogRouter(m)
|
||||
|
||||
for i := range floodRequests {
|
||||
assert.Equal(t, wantStatus, get(t, router, pathFor(i)))
|
||||
}
|
||||
|
||||
assert.NotContains(
|
||||
t, buf.String(), attackerMarker,
|
||||
"access log carried attacker-chosen path text",
|
||||
)
|
||||
|
||||
entries := accessLogEntries(t, buf)
|
||||
require.Len(t, entries, floodRequests)
|
||||
|
||||
for _, entry := range entries {
|
||||
assert.Equal(t, wantURL, entry["url"])
|
||||
assert.InDelta(
|
||||
t, float64(wantStatus), entry["status"], 0,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAccessLog_InventedReceiverPathsLogRoutePattern(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
assertFloodIsBounded(
|
||||
t,
|
||||
func(i int) string {
|
||||
return "/webhook/" + attackerMarker +
|
||||
strings.Repeat("x", i) + "?q=" + attackerMarker
|
||||
},
|
||||
http.StatusNotFound,
|
||||
"/webhook/{uuid}",
|
||||
)
|
||||
}
|
||||
|
||||
func TestAccessLog_InventedProfilePathsLogRoutePattern(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// The login redirect is a 3xx, not a 4xx, but it is just as free
|
||||
// for an unauthenticated client to drive with invented input.
|
||||
// The doubled slash is what chi's RoutePattern yields for a
|
||||
// mounted subrouter's index route.
|
||||
assertFloodIsBounded(
|
||||
t,
|
||||
func(i int) string {
|
||||
return "/user/" + attackerMarker +
|
||||
strings.Repeat("x", i) + "/"
|
||||
},
|
||||
http.StatusSeeOther,
|
||||
"/user/{username}//",
|
||||
)
|
||||
}
|
||||
|
||||
func TestAccessLog_UnroutablePathsLogFixedLiteral(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
assertFloodIsBounded(
|
||||
t,
|
||||
func(i int) string {
|
||||
return "/" + attackerMarker + strings.Repeat("x", i)
|
||||
},
|
||||
http.StatusNotFound,
|
||||
"(unmatched)",
|
||||
)
|
||||
}
|
||||
|
||||
// oversizedValue builds an 8 KB header value out of repetitions of ch,
|
||||
// with the tail marker at its end.
|
||||
//
|
||||
// The leading 'x' is load-bearing for tab: net/textproto strips leading
|
||||
// and trailing whitespace from a header value, so a value that were
|
||||
// nothing but tabs would arrive empty over a real connection and the
|
||||
// case would prove nothing.
|
||||
func oversizedValue(ch string) string {
|
||||
return "x" + strings.Repeat(ch, oversizedSegmentBytes) + tailMarker
|
||||
}
|
||||
|
||||
// oversizedHeaders fills every client-supplied header the access log
|
||||
// reads with the same value.
|
||||
func oversizedHeaders(value string) map[string]string {
|
||||
return map[string]string{
|
||||
"User-Agent": value,
|
||||
"Referer": value,
|
||||
"X-Request-Id": value,
|
||||
}
|
||||
}
|
||||
|
||||
// sizeCase is one way of pointing 8 KB of client-chosen text at the
|
||||
// access log.
|
||||
type sizeCase struct {
|
||||
target string
|
||||
headers map[string]string
|
||||
wantStatus int
|
||||
wantURL string
|
||||
bound int
|
||||
}
|
||||
|
||||
// lineSizeCases enumerates every part of a request that reaches the
|
||||
// access log, at 8 KB apiece.
|
||||
func lineSizeCases() map[string]sizeCase {
|
||||
cases := map[string]sizeCase{
|
||||
"oversized path segment": {
|
||||
target: "/webhook/" + attackerMarker +
|
||||
strings.Repeat("x", oversizedSegmentBytes),
|
||||
wantStatus: http.StatusNotFound,
|
||||
wantURL: "/webhook/{uuid}",
|
||||
bound: maxLineBytes,
|
||||
},
|
||||
// /.well-known/healthcheck answers 200 to anyone and has no
|
||||
// rate limiter in front of it, so an oversized query appended
|
||||
// to it would otherwise buy the same amplification as an
|
||||
// invented 404 path, unauthenticated and unthrottled.
|
||||
"oversized query on an unauthenticated 200": {
|
||||
target: "/.well-known/healthcheck?q=" + attackerMarker +
|
||||
strings.Repeat("x", oversizedSegmentBytes),
|
||||
wantStatus: http.StatusOK,
|
||||
wantURL: "/.well-known/healthcheck?(redacted)",
|
||||
bound: maxLineBytes,
|
||||
},
|
||||
// These reach the line on every request, including one whose
|
||||
// url field is correctly redacted.
|
||||
"oversized headers": {
|
||||
target: "/" + attackerMarker,
|
||||
headers: oversizedHeaders(oversizedValue("h")),
|
||||
wantStatus: http.StatusNotFound,
|
||||
wantURL: unmatchedRouteLiteral,
|
||||
bound: maxCappedLineBytes,
|
||||
},
|
||||
}
|
||||
|
||||
// The url field on a 5xx keeps the concrete path, so it reaches its
|
||||
// own budget on the same line as the three header fields. That is
|
||||
// the widest line the service can be made to write.
|
||||
longPath := "/boom/" + strings.Repeat("x", oversizedSegmentBytes)
|
||||
wantLongURL := longPath[:maxFieldBytes] + truncationSuffix
|
||||
|
||||
// escapeChars are the runes Go's header parser accepts in a header
|
||||
// value and the log handler then escapes, coming out wider than
|
||||
// they went in. A budget counted in raw bytes lets any of them buy
|
||||
// a field several times its nominal size, so every one of them
|
||||
// gets a case.
|
||||
//
|
||||
// The astral one is the case the JSON handler alone does not
|
||||
// reach: U+1000C is unassigned, so it is non-printable, and
|
||||
// strconv.Quote spells a non-printable rune at or above U+10000
|
||||
// as a ten-byte \UXXXXXXXX. The JSON handler passes it through as
|
||||
// its four UTF-8 bytes, so only the text-handler shape of this
|
||||
// test holds the ten-byte charge honest.
|
||||
escapeChars := map[string]string{
|
||||
"quote": `"`,
|
||||
"backslash": `\`,
|
||||
"tab": "\t",
|
||||
"astral": "\U0001000C",
|
||||
}
|
||||
|
||||
for kind, char := range escapeChars {
|
||||
fill := oversizedValue(char)
|
||||
|
||||
cases["oversized "+kind+" headers"] = sizeCase{
|
||||
target: "/" + attackerMarker,
|
||||
headers: oversizedHeaders(fill),
|
||||
wantStatus: http.StatusNotFound,
|
||||
wantURL: unmatchedRouteLiteral,
|
||||
bound: maxCappedLineBytes,
|
||||
}
|
||||
|
||||
cases["oversized "+kind+" headers with a 5xx concrete url"] =
|
||||
sizeCase{
|
||||
target: longPath,
|
||||
headers: oversizedHeaders(fill),
|
||||
wantStatus: http.StatusInternalServerError,
|
||||
wantURL: wantLongURL,
|
||||
bound: maxCappedLineBytes,
|
||||
}
|
||||
}
|
||||
|
||||
return cases
|
||||
}
|
||||
|
||||
// TestAccessLog_LineSizeDoesNotTrackInputSize drives 8 KB of
|
||||
// client-chosen text at the access log through each part of the
|
||||
// request that reaches it, and holds the resulting line to a fixed
|
||||
// bound in every case.
|
||||
//
|
||||
// The bound is on the ENCODED line, so the cases built out of
|
||||
// characters the handler escapes are the ones that matter: a budget
|
||||
// spent in raw bytes passes every plain-ASCII case here and still
|
||||
// writes a line half again as long as the stated ceiling.
|
||||
func TestAccessLog_LineSizeDoesNotTrackInputSize(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
require.Equal(
|
||||
t, middleware.MaxAccessLogLineBytes, maxCappedLineBytes,
|
||||
"the README quotes this ceiling and the middleware derives "+
|
||||
"it; they have to agree",
|
||||
)
|
||||
|
||||
for name, tc := range lineSizeCases() {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
m, buf := capturingMiddleware(t)
|
||||
router := accessLogRouter(m)
|
||||
|
||||
assert.Equal(
|
||||
t,
|
||||
tc.wantStatus,
|
||||
getWithHeaders(t, router, tc.target, tc.headers),
|
||||
)
|
||||
|
||||
// accessLogEntriesWithin enforces the bound, which is
|
||||
// orders of magnitude smaller than the input just sent.
|
||||
entries := accessLogEntriesWithin(t, buf, tc.bound)
|
||||
require.Len(t, entries, 1)
|
||||
assert.Equal(t, tc.wantURL, entries[0]["url"])
|
||||
|
||||
// The markers sit at the far end of the client-chosen
|
||||
// text, so their absence is what proves the redaction and
|
||||
// the truncation actually ran.
|
||||
assert.NotContains(
|
||||
t, buf.String(), attackerMarker,
|
||||
"access log carried attacker-chosen text",
|
||||
)
|
||||
assert.NotContains(
|
||||
t, buf.String(), tailMarker,
|
||||
"access log carried an untruncated client field",
|
||||
)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestAccessLog_LineSizeDoesNotTrackInputSizeOnTheTextHandler runs the
|
||||
// same cases through slog's text handler, which internal/logger
|
||||
// selects on a tty.
|
||||
//
|
||||
// MaxAccessLogLineBytes is quoted to operators unqualified, so it has
|
||||
// to hold for whichever handler is installed — and the two do not
|
||||
// escape alike. The astral case is the one that separates them: the
|
||||
// JSON handler emits U+1000C as its four UTF-8 bytes, while
|
||||
// strconv.Quote spells it \U0001000C at ten. Charging six for it, as
|
||||
// this code did, put a real 2,676-byte line on the wire here while
|
||||
// every JSON case stayed comfortably inside the bound.
|
||||
//
|
||||
// Only the size bound is asserted; the url field's contents are the
|
||||
// JSON shape's business above.
|
||||
func TestAccessLog_LineSizeDoesNotTrackInputSizeOnTheTextHandler(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
for name, tc := range lineSizeCases() {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
m, buf := capturingTextMiddleware(t)
|
||||
router := accessLogRouter(m)
|
||||
|
||||
assert.Equal(
|
||||
t,
|
||||
tc.wantStatus,
|
||||
getWithHeaders(t, router, tc.target, tc.headers),
|
||||
)
|
||||
|
||||
line := strings.TrimSpace(buf.String())
|
||||
|
||||
require.NotEmpty(t, line)
|
||||
assert.NotContains(
|
||||
t, line, "\n", "expected exactly one log line",
|
||||
)
|
||||
require.LessOrEqual(
|
||||
t, len(line), tc.bound,
|
||||
"access log line exceeded its bound",
|
||||
)
|
||||
assert.Contains(t, line, "url=")
|
||||
assert.NotContains(
|
||||
t, line, attackerMarker,
|
||||
"access log carried attacker-chosen text",
|
||||
)
|
||||
assert.NotContains(
|
||||
t, line, tailMarker,
|
||||
"access log carried an untruncated client field",
|
||||
)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestAccessLog_OversizedMethodIsTruncated covers the last term in the
|
||||
// MaxAccessLogLineBytes arithmetic that the size cases above cannot
|
||||
// reach: Go accepts any RFC 7230 token as a method, and getWithHeaders
|
||||
// only ever sends GET.
|
||||
func TestAccessLog_OversizedMethodIsTruncated(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
m, buf := capturingMiddleware(t)
|
||||
router := accessLogRouter(m)
|
||||
|
||||
method := strings.Repeat("M", oversizedSegmentBytes) + attackerMarker
|
||||
req := httptest.NewRequestWithContext(
|
||||
context.Background(), method, "/"+attackerMarker, nil,
|
||||
)
|
||||
rec := httptest.NewRecorder()
|
||||
router.ServeHTTP(rec, req)
|
||||
|
||||
entries := accessLogEntriesWithin(t, buf, maxLineBytes)
|
||||
require.Len(t, entries, 1)
|
||||
assert.Equal(
|
||||
t,
|
||||
strings.Repeat("M", maxMethodBytes)+truncationSuffix,
|
||||
entries[0]["method"],
|
||||
)
|
||||
assert.NotContains(
|
||||
t, buf.String(), attackerMarker,
|
||||
"access log carried attacker-chosen text",
|
||||
)
|
||||
}
|
||||
|
||||
// TestAccessLog_OversizedHeadersKeepATruncatedPrefix checks the other
|
||||
// half of the header cap: the fields are cut, not dropped, so a
|
||||
// truncated User-Agent is still worth reading.
|
||||
func TestAccessLog_OversizedHeadersKeepATruncatedPrefix(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
m, buf := capturingMiddleware(t)
|
||||
router := accessLogRouter(m)
|
||||
|
||||
assert.Equal(
|
||||
t,
|
||||
http.StatusNotFound,
|
||||
getWithHeaders(
|
||||
t, router, "/nope",
|
||||
oversizedHeaders(oversizedValue("h")),
|
||||
),
|
||||
)
|
||||
|
||||
entries := accessLogEntriesWithin(t, buf, maxCappedLineBytes)
|
||||
require.Len(t, entries, 1)
|
||||
|
||||
for key, budget := range map[string]int{
|
||||
"useragent": maxFieldBytes,
|
||||
"referer": maxFieldBytes,
|
||||
"request_id": maxRequestIDBytes,
|
||||
} {
|
||||
value, ok := entries[0][key].(string)
|
||||
require.True(t, ok, key)
|
||||
assert.LessOrEqual(
|
||||
t, len(value), budget+len(truncationSuffix), key,
|
||||
)
|
||||
assert.Contains(t, value, truncationSuffix, key)
|
||||
assert.Contains(t, value, "hhhh", key)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAccessLog_SuccessKeepsConcretePathAndRedactsQuery(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
m, buf := capturingMiddleware(t)
|
||||
router := accessLogRouter(m)
|
||||
|
||||
assert.Equal(
|
||||
t, http.StatusOK, get(t, router, "/webhook/known?src=ci"),
|
||||
)
|
||||
|
||||
// The path resolved against a stored entrypoint, so it stays. The
|
||||
// query never does: see TestAccessLog_UnauthenticatedSuccess...
|
||||
entries := accessLogEntries(t, buf)
|
||||
require.Len(t, entries, 1)
|
||||
assert.Equal(t, "/webhook/known?(redacted)", entries[0]["url"])
|
||||
assert.NotContains(t, buf.String(), "src=ci")
|
||||
}
|
||||
|
||||
func TestAccessLog_ServerErrorKeepsConcreteURL(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
m, buf := capturingMiddleware(t)
|
||||
router := accessLogRouter(m)
|
||||
|
||||
assert.Equal(
|
||||
t, http.StatusInternalServerError, get(t, router, "/boom"),
|
||||
)
|
||||
|
||||
entries := accessLogEntries(t, buf)
|
||||
require.Len(t, entries, 1)
|
||||
assert.Equal(t, "/boom", entries[0]["url"])
|
||||
}
|
||||
|
||||
func TestAccessLog_RetainsEveryOtherField(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
m, buf := capturingMiddleware(t)
|
||||
router := accessLogRouter(m)
|
||||
|
||||
assert.Equal(
|
||||
t,
|
||||
http.StatusNotFound,
|
||||
get(t, router, "/webhook/"+attackerMarker),
|
||||
)
|
||||
|
||||
entries := accessLogEntries(t, buf)
|
||||
require.Len(t, entries, 1)
|
||||
|
||||
for _, key := range []string{
|
||||
"request_start", "method", "url", "useragent", "request_id",
|
||||
"referer", "proto", "remoteIP", "status", "latency_ms",
|
||||
} {
|
||||
assert.Contains(t, entries[0], key)
|
||||
}
|
||||
|
||||
assert.Equal(t, http.MethodGet, entries[0]["method"])
|
||||
assert.Equal(t, "HTTP/1.1", entries[0]["proto"])
|
||||
}
|
||||
@@ -6,9 +6,13 @@ import (
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
|
||||
basicauth "github.com/99designs/basicauth-go"
|
||||
"github.com/go-chi/chi"
|
||||
"github.com/go-chi/chi/middleware"
|
||||
"github.com/go-chi/cors"
|
||||
metrics "github.com/slok/go-http-metrics/metrics/prometheus"
|
||||
@@ -25,6 +29,75 @@ const (
|
||||
// corsMaxAge is the maximum time (in seconds) that a
|
||||
// preflight response can be cached.
|
||||
corsMaxAge = 300
|
||||
|
||||
// unmatchedRoute is logged in the access log's url field when a
|
||||
// redirected or rejected request matched no route pattern at
|
||||
// all. Every byte of such a path is client-chosen, so none of it
|
||||
// is logged.
|
||||
unmatchedRoute = "(unmatched)"
|
||||
|
||||
// redactedQuery stands in for the query string on the access log
|
||||
// branches that keep the concrete URL. The query is client-chosen
|
||||
// on every route, including the ones that answer an
|
||||
// unauthenticated 200, so logging it verbatim would let a client
|
||||
// pick the size of the line it writes.
|
||||
redactedQuery = "?(redacted)"
|
||||
|
||||
// maxLogFieldBytes bounds each access log field whose value the
|
||||
// client supplies outright: the URL, the User-Agent and the
|
||||
// Referer. The budget is spent in ENCODED bytes (see
|
||||
// truncateLogField), so 512 still holds a real browser's User-Agent
|
||||
// whole — those are plain ASCII, which encodes one byte for one —
|
||||
// while a value built from characters the encoder escapes keeps a
|
||||
// shorter prefix. That is the intended trade: 500 quotation marks
|
||||
// are not a debugging asset.
|
||||
maxLogFieldBytes = 512
|
||||
|
||||
// maxLogRequestIDBytes bounds the request id, which is also
|
||||
// client-supplied: chi's RequestID middleware passes an inbound
|
||||
// X-Request-Id header through verbatim. Its generated form is an
|
||||
// order of magnitude shorter than this.
|
||||
maxLogRequestIDBytes = 128
|
||||
|
||||
// maxLogMethodBytes bounds the method. Go accepts any RFC 7230
|
||||
// token there, bounded only by the header size limit, so it is
|
||||
// client-chosen text like the rest. The longest registered method
|
||||
// is half this.
|
||||
maxLogMethodBytes = 32
|
||||
|
||||
// truncationMarker is appended to any field the access log cut, so
|
||||
// a short value and a truncated one cannot be confused. It is
|
||||
// charged on top of the budget, not inside it.
|
||||
truncationMarker = "[truncated]"
|
||||
|
||||
// MaxAccessLogLineBytes is the ceiling on one JSON access log line,
|
||||
// and the number an operator multiplies by the request rate to size
|
||||
// log storage. It is not an observation of a sample: it is the sum
|
||||
// of the budgets above, each of which truncateLogField enforces in
|
||||
// ENCODED bytes, plus the part of the line no client can influence.
|
||||
//
|
||||
// url, useragent, referer 3*(512+11) = 1569
|
||||
// request_id 128+11 = 139
|
||||
// method 32+11 = 43
|
||||
// fixed portion = 336
|
||||
// ----
|
||||
// 2087
|
||||
//
|
||||
// The fixed portion is the JSON punctuation, the field names, the
|
||||
// level and the message, both timestamps at their longest, an IPv6
|
||||
// remoteIP with a zone, a three-digit status and a full-width int64
|
||||
// latency. Stated at 2560 so the figure carries headroom rather
|
||||
// than sitting on the arithmetic.
|
||||
//
|
||||
// The tty text handler in internal/logger is covered by the same
|
||||
// figure. encodedLogFieldBytes charges every rune at least what
|
||||
// the wider of the two handlers emits for it — including the ten
|
||||
// bytes strconv.Quote spends on a non-printable rune at or above
|
||||
// U+10000, which is four more than the JSON handler ever spends —
|
||||
// so each budget bounds the encoded field under either handler.
|
||||
// The text handler's fixed portion is 286, the smaller of the two,
|
||||
// which puts its worst case at 2037.
|
||||
MaxAccessLogLineBytes = 2560
|
||||
)
|
||||
|
||||
//nolint:revive // MiddlewareParams is a standard fx naming convention.
|
||||
@@ -94,6 +167,178 @@ func (lrw *loggingResponseWriter) WriteHeader(code int) {
|
||||
lrw.ResponseWriter.WriteHeader(code)
|
||||
}
|
||||
|
||||
// encodedLogFieldBytes is what r costs on the line once the log
|
||||
// handler has escaped it, taking the worse of the two handlers
|
||||
// internal/logger configures.
|
||||
//
|
||||
// slog's JSON handler escapes quote, backslash, newline, carriage
|
||||
// return and tab to two bytes each, and every other C0 control plus
|
||||
// LINE SEPARATOR and PARAGRAPH SEPARATOR to a six-byte \u escape; it
|
||||
// passes every other rune through as its own UTF-8. Its text handler
|
||||
// quotes with strconv.Quote, which spells a non-printable rune below
|
||||
// U+10000 as \uXXXX but one at or above U+10000 as \UXXXXXXXX — ten
|
||||
// bytes, not six. The text handler is therefore the worse of the two
|
||||
// for every non-printable rune, and by four bytes apiece for the
|
||||
// 955,086 unassigned, private-use and format code points on planes 1
|
||||
// to 16.
|
||||
//
|
||||
// Charging ten there is what makes MaxAccessLogLineBytes hold for the
|
||||
// tty handler as well: U+1000C encodes as F0 90 80 8C, every byte
|
||||
// >= 0x80, which httpguts.ValidHeaderFieldValue accepts and
|
||||
// net/textproto does not strip, so a header can be filled with them.
|
||||
//
|
||||
// Both handlers pass printable runes through as their own UTF-8, so
|
||||
// unicode.IsPrint separates the escaped cases from the plain ones for
|
||||
// either handler.
|
||||
func encodedLogFieldBytes(r rune) int {
|
||||
const (
|
||||
// A backslash and the character itself.
|
||||
shortEscapeBytes = 2
|
||||
// \uXXXX, which is also the width of \u00XX.
|
||||
escapedRuneBytes = 6
|
||||
// \UXXXXXXXX, strconv.Quote's spelling of a non-printable
|
||||
// rune outside the basic multilingual plane.
|
||||
escapedAstralRuneBytes = 10
|
||||
// The first code point strconv.Quote spells with \U.
|
||||
firstAstralRune = 0x10000
|
||||
)
|
||||
|
||||
switch {
|
||||
case r == '"' || r == '\\' || r == '\n' || r == '\r' || r == '\t':
|
||||
return shortEscapeBytes
|
||||
case !unicode.IsPrint(r) && r >= firstAstralRune:
|
||||
return escapedAstralRuneBytes
|
||||
case !unicode.IsPrint(r):
|
||||
return escapedRuneBytes
|
||||
default:
|
||||
return utf8.RuneLen(r)
|
||||
}
|
||||
}
|
||||
|
||||
// truncateLogField caps s at maxBytes of ENCODED output, marking the
|
||||
// value when it cuts.
|
||||
//
|
||||
// Budgeting raw bytes would not bound the line. Escaping only ever
|
||||
// grows a value, so a raw budget spent on characters the encoder
|
||||
// escapes buys a field several times its nominal size — and the line
|
||||
// is the thing an operator is told to multiply by their request rate.
|
||||
// Charging each rune what it will actually cost is what makes
|
||||
// MaxAccessLogLineBytes true rather than merely larger. The visible
|
||||
// consequence is that an escape-heavy value keeps a shorter prefix
|
||||
// than a plain one, which is the correct trade.
|
||||
//
|
||||
// The result is always valid UTF-8. A cut on a byte boundary can split
|
||||
// a multi-byte rune, and a header can carry bytes that were never
|
||||
// valid UTF-8 to begin with; both are dropped rather than kept, since
|
||||
// an encoder would otherwise spend six bytes replacing each one.
|
||||
func truncateLogField(s string, maxBytes int) string {
|
||||
// No rune encodes to fewer bytes than it occupies, so nothing past
|
||||
// maxBytes raw can fit the budget. Slicing first bounds the scan
|
||||
// below to the budget rather than to the size of the header the
|
||||
// client sent.
|
||||
window, cut := s, false
|
||||
if len(window) > maxBytes {
|
||||
window, cut = window[:maxBytes], true
|
||||
}
|
||||
|
||||
var (
|
||||
kept strings.Builder
|
||||
spent int
|
||||
)
|
||||
|
||||
for i := 0; i < len(window); {
|
||||
r, size := utf8.DecodeRuneInString(window[i:])
|
||||
if r == utf8.RuneError && size == 1 {
|
||||
i += size
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
cost := encodedLogFieldBytes(r)
|
||||
if spent+cost > maxBytes {
|
||||
cut = true
|
||||
|
||||
break
|
||||
}
|
||||
|
||||
spent += cost
|
||||
|
||||
kept.WriteString(window[i : i+size])
|
||||
|
||||
i += size
|
||||
}
|
||||
|
||||
if !cut {
|
||||
return kept.String()
|
||||
}
|
||||
|
||||
return kept.String() + truncationMarker
|
||||
}
|
||||
|
||||
// concreteLogURL renders the request's own URL for the access log
|
||||
// branches that keep it, with the query string replaced by a fixed
|
||||
// marker.
|
||||
//
|
||||
// The path on those branches is bounded by the service's routes or by
|
||||
// the operator's data — a 2xx on the receiver means the UUID named a
|
||||
// stored entrypoint, a 2xx under /s means the file is in the embedded
|
||||
// tree. The query is not bounded by anything: /.well-known/healthcheck
|
||||
// and /s/* take no authentication and sit behind no rate limiter, and
|
||||
// /pages/login behind only the login limiter, so any of them will
|
||||
// answer 200 to a URL carrying an arbitrary number of arbitrary bytes
|
||||
// after the '?'. Keeping the path and dropping the query is what makes
|
||||
// this branch as bounded as the pattern branches below.
|
||||
//
|
||||
// Nothing debuggable is lost. One route in the service reads a query
|
||||
// parameter at all — `page`, on the authenticated pagination links in
|
||||
// internal/handlers/source_management.go — and the alternatives that
|
||||
// would preserve more (a key count, a key allowlist) all require
|
||||
// parsing an attacker-sized query on every request, which is work an
|
||||
// unauthenticated client would then be choosing for us.
|
||||
func concreteLogURL(r *http.Request) string {
|
||||
path := r.URL.EscapedPath()
|
||||
|
||||
if r.URL.RawQuery == "" && !r.URL.ForceQuery {
|
||||
return path
|
||||
}
|
||||
|
||||
return path + redactedQuery
|
||||
}
|
||||
|
||||
// accessLogURL returns the value for the access log's url field.
|
||||
//
|
||||
// 2xx and 5xx responses get the concrete path (see concreteLogURL). A
|
||||
// success resolved against a static route or against the operator's
|
||||
// own data — on the receiver, a 2xx means the UUID named a stored
|
||||
// entrypoint — and a server error is our own bug, where the exact URL
|
||||
// is the primary evidence and which no client can provoke at will.
|
||||
//
|
||||
// 3xx and 4xx responses get the chi route pattern instead. Those are
|
||||
// the outcomes an unauthenticated client drives for free: 404 or 429
|
||||
// on any invented /webhook/ path, 303 to the login page on any
|
||||
// invented /user/ path. Logging the concrete URL there lets a flood
|
||||
// write attacker-chosen text, of attacker-chosen length, into the
|
||||
// operator's log at one line per request. The pattern comes from the
|
||||
// router's own table, so it is bounded by the service's routes while
|
||||
// still naming which class of request was rejected.
|
||||
//
|
||||
// The pattern is only populated once routing has run, so this must be
|
||||
// called after the handler returns, not before.
|
||||
func accessLogURL(r *http.Request, status int) string {
|
||||
if status < http.StatusMultipleChoices ||
|
||||
status >= http.StatusInternalServerError {
|
||||
return concreteLogURL(r)
|
||||
}
|
||||
|
||||
if rc := chi.RouteContext(r.Context()); rc != nil {
|
||||
if pattern := rc.RoutePattern(); pattern != "" {
|
||||
return pattern
|
||||
}
|
||||
}
|
||||
|
||||
return unmatchedRoute
|
||||
}
|
||||
|
||||
// Logging returns middleware that logs each HTTP request with
|
||||
// timing and metadata.
|
||||
func (s *Middleware) Logging() func(http.Handler) http.Handler {
|
||||
@@ -118,13 +363,27 @@ func (s *Middleware) Logging() func(http.Handler) http.Handler {
|
||||
}
|
||||
}
|
||||
|
||||
// Every field below that a client can influence is
|
||||
// truncated to a fixed budget, so the size of this
|
||||
// line does not track the size of the request.
|
||||
s.log.Info("http request",
|
||||
"request_start", start,
|
||||
"method", r.Method,
|
||||
"url", r.URL.String(),
|
||||
"useragent", r.UserAgent(),
|
||||
"request_id", requestID,
|
||||
"referer", r.Referer(),
|
||||
"method", truncateLogField(
|
||||
r.Method, maxLogMethodBytes,
|
||||
),
|
||||
"url", truncateLogField(
|
||||
accessLogURL(r, lrw.statusCode),
|
||||
maxLogFieldBytes,
|
||||
),
|
||||
"useragent", truncateLogField(
|
||||
r.UserAgent(), maxLogFieldBytes,
|
||||
),
|
||||
"request_id", truncateLogField(
|
||||
requestID, maxLogRequestIDBytes,
|
||||
),
|
||||
"referer", truncateLogField(
|
||||
r.Referer(), maxLogFieldBytes,
|
||||
),
|
||||
"proto", r.Proto,
|
||||
"remoteIP", ipFromHostPort(r.RemoteAddr),
|
||||
"status", lrw.statusCode,
|
||||
|
||||
@@ -48,6 +48,12 @@ const (
|
||||
// bound every request pays a walk proportional to whatever the
|
||||
// client sent.
|
||||
maxForwardedHops = 64
|
||||
|
||||
// ipv6BucketBits is the prefix length IPv6 clients are bucketed
|
||||
// on. A routed /64 is the normal residential and mobile
|
||||
// allocation, so it is the unit an attacker gets addresses in
|
||||
// and therefore the unit worth limiting.
|
||||
ipv6BucketBits = 64
|
||||
)
|
||||
|
||||
// normalizeAddr strips the IPv4-in-IPv6 wrapper and any zone from
|
||||
@@ -56,6 +62,40 @@ func normalizeAddr(addr netip.Addr) netip.Addr {
|
||||
return addr.Unmap().WithZone("")
|
||||
}
|
||||
|
||||
// bucketKey is the rate-limit bucket identity of a client address.
|
||||
// IPv4 keys on the full address; IPv6 keys on its /64 prefix,
|
||||
// because keying IPv6 per /128 lets one ordinary subscriber rotate
|
||||
// source addresses inside its own routed /64 and mint a fresh bucket
|
||||
// per request — evading every limiter here at the network layer,
|
||||
// with no spoofing and nothing to detect.
|
||||
//
|
||||
// An IPv4-mapped address (::ffff:1.2.3.4) is keyed as the IPv4
|
||||
// address it carries, never masked to a /64: mapped form all shares
|
||||
// the ::ffff:0:0/96 prefix, so masking would collapse every IPv4
|
||||
// client reaching a proxy that emits it into one bucket. Callers
|
||||
// pass addresses through normalizeAddr, which already unmaps; the
|
||||
// unmap here keeps the property true of the key function itself.
|
||||
//
|
||||
// The two families cannot collide: an IPv4 key is a bare dotted
|
||||
// quad, and an IPv6 key always carries a "/64" suffix.
|
||||
func bucketKey(addr netip.Addr) string {
|
||||
addr = addr.Unmap()
|
||||
|
||||
if addr.Is4() {
|
||||
return addr.String()
|
||||
}
|
||||
|
||||
// Prefix errors only on a negative bit count, on over 32 bits
|
||||
// for an IPv4 address, or on over 128 for IPv6. The count here
|
||||
// is the constant 64 and the IPv4 case returned above, so the
|
||||
// error is unreachable. (The zero Addr does not error either: it
|
||||
// yields the zero Prefix. Neither call site can produce one,
|
||||
// since both parse the address first.)
|
||||
prefix, _ := addr.Prefix(ipv6BucketBits)
|
||||
|
||||
return prefix.String()
|
||||
}
|
||||
|
||||
// isTrustedProxy reports whether addr belongs to a network the
|
||||
// operator listed in TRUSTED_PROXIES. The list is empty by default,
|
||||
// so by default nothing is trusted.
|
||||
@@ -143,6 +183,9 @@ func (m *Middleware) forwardedClientAddr(
|
||||
// another client's bucket, by picking an X-Forwarded-For value —
|
||||
// which makes every limit here decorative against a deliberate
|
||||
// attacker.
|
||||
//
|
||||
// The address that identifies the client is then reduced to a bucket
|
||||
// by bucketKey: full address for IPv4, /64 prefix for IPv6.
|
||||
func (m *Middleware) rateLimitKey(r *http.Request) (string, error) {
|
||||
return m.clientKey(r), nil
|
||||
}
|
||||
@@ -152,23 +195,25 @@ func (m *Middleware) clientKey(r *http.Request) string {
|
||||
peer, err := netip.ParseAddr(ipFromHostPort(r.RemoteAddr))
|
||||
if err != nil {
|
||||
// Not an address we can reason about; key on the raw
|
||||
// value, the most specific identity left. On a
|
||||
// Unix-socket listener every peer carries the same
|
||||
// RemoteAddr and so shares one bucket, which is the
|
||||
// fail-closed direction.
|
||||
// value, the most specific identity left. Distinct
|
||||
// RemoteAddr values stay in distinct buckets, so this
|
||||
// path cannot silently collapse unrelated clients
|
||||
// together. On a Unix-socket listener every peer
|
||||
// carries the same RemoteAddr and so shares one bucket,
|
||||
// which is the fail-closed direction.
|
||||
return r.RemoteAddr
|
||||
}
|
||||
|
||||
peer = normalizeAddr(peer)
|
||||
if !m.isTrustedProxy(peer) {
|
||||
return peer.String()
|
||||
return bucketKey(peer)
|
||||
}
|
||||
|
||||
if addr, ok := m.forwardedClientAddr(r); ok {
|
||||
return addr.String()
|
||||
return bucketKey(addr)
|
||||
}
|
||||
|
||||
return peer.String()
|
||||
return bucketKey(peer)
|
||||
}
|
||||
|
||||
// tooManyRequests returns the 429 handler used by the login,
|
||||
|
||||
@@ -15,6 +15,7 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/webhooker/internal/config"
|
||||
"sneak.berlin/go/webhooker/internal/middleware"
|
||||
)
|
||||
@@ -370,6 +371,30 @@ const (
|
||||
headerXFF = "X-Forwarded-For"
|
||||
headerReal = "X-Real-IP"
|
||||
headerTrue = "True-Client-IP"
|
||||
|
||||
// clientIPv4 is the sample IPv4 client address these tests key
|
||||
// on, both directly and in IPv4-mapped form. clientIPv4Alt is
|
||||
// its neighbour, used to show the two do not share a bucket.
|
||||
clientIPv4 = "198.51.100.7"
|
||||
clientIPv4Alt = "198.51.100.8"
|
||||
|
||||
// clientIPv6 and clientIPv6Same are two addresses inside one
|
||||
// routed /64, so both must key on clientBucketV6.
|
||||
// clientIPv6Other is a different allocation and must key on
|
||||
// clientOtherBucketV6.
|
||||
clientIPv6 = "2001:db8:1:2:3:4:5:6"
|
||||
clientIPv6Same = "2001:db8:1:2:aaaa:bbbb:cccc:dddd"
|
||||
clientIPv6Other = "2001:db8:1:3::1"
|
||||
clientBucketV6 = "2001:db8:1:2::/64"
|
||||
clientOtherBucketV6 = "2001:db8:1:3::/64"
|
||||
|
||||
// trustedProxyCIDR is the proxy network the forwarded-path
|
||||
// tests configure, and trustedPeer an address inside it. A
|
||||
// production deployment is required to run behind a reverse
|
||||
// proxy with TRUSTED_PROXIES set, so this is the shape the
|
||||
// bucketing has to hold in.
|
||||
trustedProxyCIDR = "10.0.0.0/8"
|
||||
trustedPeer = "10.0.0.1:44444"
|
||||
)
|
||||
|
||||
// assertSharedBucket drives the login limiter from peer with the
|
||||
@@ -458,8 +483,8 @@ func TestRateLimitKey_SingleValuedHeadersIgnoredFromTrustedPeer(
|
||||
t.Parallel()
|
||||
|
||||
assertSharedBucket(
|
||||
t, trustedProxies("10.0.0.0/8"),
|
||||
"10.0.0.1:44444",
|
||||
t, trustedProxies(trustedProxyCIDR),
|
||||
trustedPeer,
|
||||
func(i int) map[string]string {
|
||||
return map[string]string{
|
||||
header: fmt.Sprintf(
|
||||
@@ -495,8 +520,8 @@ func TestRateLimitKey_MalformedRightmostHopFallsBackToPeer(
|
||||
t.Parallel()
|
||||
|
||||
assertSharedBucket(
|
||||
t, trustedProxies("10.0.0.0/8"),
|
||||
"10.0.0.1:44444",
|
||||
t, trustedProxies(trustedProxyCIDR),
|
||||
trustedPeer,
|
||||
func(i int) map[string]string {
|
||||
return map[string]string{
|
||||
headerXFF: fmt.Sprintf(
|
||||
@@ -522,13 +547,13 @@ func TestRateLimitKey_ForwardedHonouredFromTrustedPeer(
|
||||
t.Parallel()
|
||||
|
||||
m := rateLimitMiddleware(t, &config.Config{
|
||||
TrustedProxies: trustedProxies("10.0.0.0/8"),
|
||||
TrustedProxies: trustedProxies(trustedProxyCIDR),
|
||||
})
|
||||
handler := m.LoginRateLimit()(okHandler())
|
||||
|
||||
const peer = "10.0.0.1:44444"
|
||||
const peer = trustedPeer
|
||||
|
||||
first := map[string]string{headerXFF: "198.51.100.7"}
|
||||
first := map[string]string{headerXFF: clientIPv4}
|
||||
|
||||
for range middleware.LoginRateLimitConst {
|
||||
postWithHeaders(handler, peer, loginPath, first)
|
||||
@@ -542,7 +567,7 @@ func TestRateLimitKey_ForwardedHonouredFromTrustedPeer(
|
||||
|
||||
w = postWithHeaders(
|
||||
handler, peer, loginPath,
|
||||
map[string]string{headerXFF: "198.51.100.8"},
|
||||
map[string]string{headerXFF: clientIPv4Alt},
|
||||
)
|
||||
assert.Equal(
|
||||
t, http.StatusOK, w.Code,
|
||||
@@ -559,7 +584,7 @@ func TestRateLimitKey_ChainWalkSkipsClientPrepended(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
assertSharedBucket(
|
||||
t, trustedProxies("10.0.0.0/8"), "10.0.0.1:44444",
|
||||
t, trustedProxies(trustedProxyCIDR), trustedPeer,
|
||||
func(i int) map[string]string {
|
||||
return map[string]string{
|
||||
headerXFF: fmt.Sprintf(
|
||||
@@ -594,7 +619,7 @@ func TestRateLimitKey_LongChainCapsWalkAndFallsBackToPeer(
|
||||
start := time.Now()
|
||||
|
||||
assertSharedBucket(
|
||||
t, trustedProxies("10.0.0.0/8"), "10.0.0.1:44444",
|
||||
t, trustedProxies(trustedProxyCIDR), trustedPeer,
|
||||
func(i int) map[string]string {
|
||||
return map[string]string{
|
||||
headerXFF: fmt.Sprintf("9.9.9.%d%s", i+1, padding),
|
||||
@@ -633,13 +658,13 @@ func TestRateLimitKey_LongChainAllocationIsBounded(t *testing.T) {
|
||||
)
|
||||
|
||||
m := rateLimitMiddleware(t, &config.Config{
|
||||
TrustedProxies: trustedProxies("10.0.0.0/8"),
|
||||
TrustedProxies: trustedProxies(trustedProxyCIDR),
|
||||
})
|
||||
|
||||
req := httptest.NewRequestWithContext(
|
||||
context.Background(), http.MethodPost, loginPath, nil,
|
||||
)
|
||||
req.RemoteAddr = "10.0.0.1:44444"
|
||||
req.RemoteAddr = trustedPeer
|
||||
req.Header.Set(
|
||||
headerXFF, "9.9.9.9"+strings.Repeat(", 10.0.0.2", hops),
|
||||
)
|
||||
@@ -835,3 +860,369 @@ func TestReceiverRateLimit_IgnoresForwardedFromUntrustedPeer(
|
||||
"not mint a fresh receiver bucket",
|
||||
)
|
||||
}
|
||||
|
||||
// clientKeyFor returns the bucket key m computes for a request whose
|
||||
// direct peer is remoteAddr and which carries no forwarded headers.
|
||||
func clientKeyFor(
|
||||
t *testing.T, m *middleware.Middleware, remoteAddr string,
|
||||
) string {
|
||||
t.Helper()
|
||||
|
||||
req := httptest.NewRequestWithContext(
|
||||
context.Background(), http.MethodPost, loginPath, nil,
|
||||
)
|
||||
req.RemoteAddr = remoteAddr
|
||||
|
||||
return middleware.ClientKeyForTest(m, req)
|
||||
}
|
||||
|
||||
// TestRateLimitKey_IPv6BucketsByPrefix pins the key function's
|
||||
// address-family behaviour. IPv6 clients must bucket by /64 — a
|
||||
// routed /64 is the normal residential and mobile allocation, so
|
||||
// per-/128 keying lets one subscriber rotate source addresses and
|
||||
// mint a fresh bucket per request — while IPv4 keeps keying on the
|
||||
// full address and IPv4-mapped form is keyed as the IPv4 address it
|
||||
// carries.
|
||||
func TestRateLimitKey_IPv6BucketsByPrefix(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
m := rateLimitMiddleware(t, &config.Config{})
|
||||
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
peer string
|
||||
want string
|
||||
about string
|
||||
}{{
|
||||
name: "ipv6",
|
||||
peer: "[" + clientIPv6 + "]:44444",
|
||||
want: clientBucketV6,
|
||||
about: "an IPv6 peer must key on its /64",
|
||||
}, {
|
||||
name: "ipv6-other-in-same-64",
|
||||
peer: "[" + clientIPv6Same + "]:1",
|
||||
want: clientBucketV6,
|
||||
about: "another address in the same /64 must key the same",
|
||||
}, {
|
||||
name: "ipv6-different-64",
|
||||
peer: "[" + clientIPv6Other + "]:44444",
|
||||
want: clientOtherBucketV6,
|
||||
about: "a different /64 must key differently",
|
||||
}, {
|
||||
name: "ipv4",
|
||||
peer: clientIPv4 + ":44444",
|
||||
want: clientIPv4,
|
||||
about: "IPv4 must keep keying on the full address",
|
||||
}, {
|
||||
name: "ipv4-neighbour",
|
||||
peer: clientIPv4Alt + ":44444",
|
||||
want: clientIPv4Alt,
|
||||
about: "adjacent IPv4 addresses must not share a bucket",
|
||||
}, {
|
||||
name: "ipv4-mapped",
|
||||
peer: "[::ffff:" + clientIPv4 + "]:44444",
|
||||
want: clientIPv4,
|
||||
about: "IPv4-mapped form must key as the IPv4 address, " +
|
||||
"not be masked to a /64: mapped addresses all share " +
|
||||
"::ffff:0:0/96, so masking would collapse every IPv4 " +
|
||||
"client behind a mapping proxy into one bucket",
|
||||
}} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
assert.Equal(
|
||||
t, tc.want, clientKeyFor(t, m, tc.peer), tc.about,
|
||||
)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRateLimitKey_FamiliesDoNotCollide pins the structure the
|
||||
// no-collision property rests on, rather than one sample pair: every
|
||||
// IPv4 key is a bare address and every IPv6 key is a /64 in CIDR
|
||||
// form, so the two name spaces are disjoint by shape. Dropping the
|
||||
// masking strips the suffix that guarantees it, which is why this
|
||||
// asserts the form of each key and not just that two of them differ.
|
||||
func TestRateLimitKey_FamiliesDoNotCollide(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// Restated here rather than imported from the package under
|
||||
// test, so that changing the production bucket width fails this
|
||||
// test instead of silently moving with it.
|
||||
const wantBits = 64
|
||||
|
||||
m := rateLimitMiddleware(t, &config.Config{})
|
||||
|
||||
v4Keys := map[string]bool{}
|
||||
|
||||
for _, peer := range []string{
|
||||
clientIPv4 + ":44444",
|
||||
clientIPv4Alt + ":44444",
|
||||
"[::ffff:" + clientIPv4 + "]:44444",
|
||||
} {
|
||||
key := clientKeyFor(t, m, peer)
|
||||
|
||||
addr, err := netip.ParseAddr(key)
|
||||
require.NoError(
|
||||
t, err, "%s: an IPv4 key must be a bare address", peer,
|
||||
)
|
||||
assert.True(
|
||||
t, addr.Is4(),
|
||||
"%s: an IPv4 key must be a dotted quad, got %q", peer, key,
|
||||
)
|
||||
|
||||
v4Keys[key] = true
|
||||
}
|
||||
|
||||
for _, peer := range []string{
|
||||
"[" + clientIPv6 + "]:44444",
|
||||
"[" + clientIPv6Same + "]:44444",
|
||||
"[" + clientIPv6Other + "]:44444",
|
||||
"[2001:db8::" + clientIPv4 + "]:44444",
|
||||
} {
|
||||
key := clientKeyFor(t, m, peer)
|
||||
|
||||
prefix, err := netip.ParsePrefix(key)
|
||||
require.NoError(
|
||||
t, err, "%s: an IPv6 key must be a CIDR prefix", peer,
|
||||
)
|
||||
assert.Equal(
|
||||
t, wantBits, prefix.Bits(),
|
||||
"%s: an IPv6 key must name a /64", peer,
|
||||
)
|
||||
assert.False(
|
||||
t, v4Keys[key],
|
||||
"%s: an IPv6 key must never equal an IPv4 key", peer,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRateLimitKey_UnparseablePeerKeepsDistinctBuckets covers the
|
||||
// fallback path. A RemoteAddr that is not an address must not panic,
|
||||
// and must not drop unrelated clients into one shared bucket by
|
||||
// accident: the raw value is the most specific identity left, so
|
||||
// distinct values stay in distinct buckets.
|
||||
func TestRateLimitKey_UnparseablePeerKeepsDistinctBuckets(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
m := rateLimitMiddleware(t, &config.Config{})
|
||||
|
||||
first := clientKeyFor(t, m, "not-an-address")
|
||||
second := clientKeyFor(t, m, "also-not-an-address:1234")
|
||||
|
||||
assert.NotEmpty(t, first)
|
||||
assert.NotEqual(
|
||||
t, first, second,
|
||||
"unparseable peers must not collapse into one bucket",
|
||||
)
|
||||
}
|
||||
|
||||
// TestLoginRateLimit_IPv6SharesBucketWithinSlash64 is the behavioural
|
||||
// half, and the regression test for the bypass itself: a client that
|
||||
// rotates source addresses inside its own routed /64 must stay in one
|
||||
// bucket. Reverting the masking makes this test fail, because each
|
||||
// rotated address would mint a fresh bucket and nothing would be
|
||||
// rejected.
|
||||
func TestLoginRateLimit_IPv6SharesBucketWithinSlash64(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
m := rateLimitMiddleware(t, &config.Config{})
|
||||
handler := m.LoginRateLimit()(okHandler())
|
||||
|
||||
for i := range middleware.LoginRateLimitConst {
|
||||
w := postWithHeaders(
|
||||
handler,
|
||||
fmt.Sprintf("[2001:db8:1:2::%d]:44444", i+1),
|
||||
loginPath, nil,
|
||||
)
|
||||
assert.Equal(
|
||||
t, http.StatusOK, w.Code, "request %d should pass", i,
|
||||
)
|
||||
}
|
||||
|
||||
w := postWithHeaders(
|
||||
handler, "[2001:db8:1:2::ffff]:44444", loginPath, nil,
|
||||
)
|
||||
assert.Equal(
|
||||
t, http.StatusTooManyRequests, w.Code,
|
||||
"rotating source addresses inside one routed /64 must not "+
|
||||
"mint fresh buckets",
|
||||
)
|
||||
}
|
||||
|
||||
// TestLoginRateLimit_IPv6IndependentAcrossSlash64 is the other side
|
||||
// of the trade: bucketing by /64 must not merge separate allocations,
|
||||
// so a client in a different /64 keeps its own limit.
|
||||
func TestLoginRateLimit_IPv6IndependentAcrossSlash64(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
m := rateLimitMiddleware(t, &config.Config{})
|
||||
handler := m.LoginRateLimit()(okHandler())
|
||||
|
||||
for range middleware.LoginRateLimitConst + 1 {
|
||||
postWithHeaders(
|
||||
handler, "[2001:db8:1:2::1]:44444", loginPath, nil,
|
||||
)
|
||||
}
|
||||
|
||||
w := postWithHeaders(
|
||||
handler, "[2001:db8:1:3::1]:44444", loginPath, nil,
|
||||
)
|
||||
assert.Equal(
|
||||
t, http.StatusOK, w.Code,
|
||||
"a different /64 must have its own bucket",
|
||||
)
|
||||
}
|
||||
|
||||
// TestLoginRateLimit_IPv4IndependentPerAddress guards against the
|
||||
// masking leaking into IPv4: two addresses one apart must still hold
|
||||
// separate buckets.
|
||||
func TestLoginRateLimit_IPv4IndependentPerAddress(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
m := rateLimitMiddleware(t, &config.Config{})
|
||||
handler := m.LoginRateLimit()(okHandler())
|
||||
|
||||
for range middleware.LoginRateLimitConst + 1 {
|
||||
postWithHeaders(
|
||||
handler, clientIPv4+":44444", loginPath, nil,
|
||||
)
|
||||
}
|
||||
|
||||
w := postWithHeaders(
|
||||
handler, clientIPv4Alt+":44444", loginPath, nil,
|
||||
)
|
||||
assert.Equal(
|
||||
t, http.StatusOK, w.Code,
|
||||
"a second IPv4 address must have its own bucket",
|
||||
)
|
||||
}
|
||||
|
||||
// forwardedKeyFor returns the bucket key m computes for a request
|
||||
// that arrives from trustedPeer — a configured trusted proxy — and
|
||||
// names forwarded as its client in X-Forwarded-For. That is the
|
||||
// production path: a deployment is required to run behind a reverse
|
||||
// proxy with TRUSTED_PROXIES set, so the forwarded address, not the
|
||||
// peer, is what the limiters bucket on there.
|
||||
func forwardedKeyFor(
|
||||
t *testing.T, m *middleware.Middleware, forwarded string,
|
||||
) string {
|
||||
t.Helper()
|
||||
|
||||
req := httptest.NewRequestWithContext(
|
||||
context.Background(), http.MethodPost, loginPath, nil,
|
||||
)
|
||||
req.RemoteAddr = trustedPeer
|
||||
req.Header.Set(headerXFF, forwarded)
|
||||
|
||||
return middleware.ClientKeyForTest(m, req)
|
||||
}
|
||||
|
||||
// TestRateLimitKey_ForwardedIPv6BucketsByPrefix pins the /64
|
||||
// bucketing on the trusted-proxy branch. The direct-peer tests above
|
||||
// cannot reach it, so without this the masking could be reverted for
|
||||
// forwarded clients alone — the only shape a production deployment
|
||||
// runs in — and the rest of the suite would stay green.
|
||||
func TestRateLimitKey_ForwardedIPv6BucketsByPrefix(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
m := rateLimitMiddleware(t, &config.Config{
|
||||
TrustedProxies: trustedProxies(trustedProxyCIDR),
|
||||
})
|
||||
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
forwarded string
|
||||
want string
|
||||
about string
|
||||
}{{
|
||||
name: "ipv6",
|
||||
forwarded: clientIPv6,
|
||||
want: clientBucketV6,
|
||||
about: "a forwarded IPv6 client must key on its /64",
|
||||
}, {
|
||||
name: "ipv6-other-in-same-64",
|
||||
forwarded: clientIPv6Same,
|
||||
want: clientBucketV6,
|
||||
about: "another forwarded address in the same /64 must " +
|
||||
"key the same",
|
||||
}, {
|
||||
name: "ipv6-different-64",
|
||||
forwarded: clientIPv6Other,
|
||||
want: clientOtherBucketV6,
|
||||
about: "a forwarded address in another /64 must differ",
|
||||
}, {
|
||||
name: "ipv4",
|
||||
forwarded: clientIPv4,
|
||||
want: clientIPv4,
|
||||
about: "a forwarded IPv4 client must key on the address",
|
||||
}, {
|
||||
name: "ipv4-mapped",
|
||||
forwarded: "::ffff:" + clientIPv4,
|
||||
want: clientIPv4,
|
||||
about: "a proxy that forwards IPv4-mapped form must key as " +
|
||||
"the IPv4 address it carries, not be masked to a /64: " +
|
||||
"mapped addresses all share ::ffff:0:0/96",
|
||||
}} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
assert.Equal(
|
||||
t, tc.want,
|
||||
forwardedKeyFor(t, m, tc.forwarded), tc.about,
|
||||
)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestLoginRateLimit_ForwardedIPv6SharesBucketWithinSlash64 is the
|
||||
// behavioural half on the production path: behind a trusted proxy, a
|
||||
// client rotating source addresses inside its own routed /64 must
|
||||
// stay in one bucket.
|
||||
func TestLoginRateLimit_ForwardedIPv6SharesBucketWithinSlash64(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
assertSharedBucket(
|
||||
t, trustedProxies(trustedProxyCIDR), trustedPeer,
|
||||
func(i int) map[string]string {
|
||||
return map[string]string{
|
||||
headerXFF: fmt.Sprintf("2001:db8:1:2::%d", i+1),
|
||||
}
|
||||
},
|
||||
"rotating forwarded source addresses inside one routed /64 "+
|
||||
"must not mint fresh buckets",
|
||||
)
|
||||
}
|
||||
|
||||
// TestLoginRateLimit_ForwardedIPv6IndependentAcrossSlash64 is the
|
||||
// other side of that trade on the same path: bucketing by /64 must
|
||||
// not merge two allocations reaching the proxy.
|
||||
func TestLoginRateLimit_ForwardedIPv6IndependentAcrossSlash64(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
m := rateLimitMiddleware(t, &config.Config{
|
||||
TrustedProxies: trustedProxies(trustedProxyCIDR),
|
||||
})
|
||||
handler := m.LoginRateLimit()(okHandler())
|
||||
|
||||
spent := map[string]string{headerXFF: clientIPv6}
|
||||
for range middleware.LoginRateLimitConst + 1 {
|
||||
postWithHeaders(handler, trustedPeer, loginPath, spent)
|
||||
}
|
||||
|
||||
w := postWithHeaders(
|
||||
handler, trustedPeer, loginPath,
|
||||
map[string]string{headerXFF: clientIPv6Other},
|
||||
)
|
||||
assert.Equal(
|
||||
t, http.StatusOK, w.Code,
|
||||
"a forwarded client in a different /64 must have its own "+
|
||||
"bucket",
|
||||
)
|
||||
}
|
||||
|
||||
@@ -24,6 +24,7 @@ import (
|
||||
"sneak.berlin/go/webhooker/internal/middleware"
|
||||
"sneak.berlin/go/webhooker/internal/server"
|
||||
"sneak.berlin/go/webhooker/internal/session"
|
||||
"sneak.berlin/go/webhooker/static"
|
||||
)
|
||||
|
||||
// csrfCookieName is the cookie gorilla/csrf issues when it runs. Its
|
||||
@@ -246,6 +247,56 @@ func (e *testEnv) storedHash(t *testing.T, username string) string {
|
||||
return user.Password
|
||||
}
|
||||
|
||||
// --- /s static group ---
|
||||
|
||||
// TestStaticServesEveryMethod pins what the static mount actually
|
||||
// answers. chi's Mount registers the handler for all methods and
|
||||
// http.FileServer only special-cases HEAD (by suppressing the body),
|
||||
// so a POST or a DELETE to an asset is served the file rather than
|
||||
// refused. The README documents this; the test is what keeps the two
|
||||
// from drifting.
|
||||
func TestStaticServesEveryMethod(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := newTestEnv(t)
|
||||
|
||||
body, err := static.Static.ReadFile("js/app.js")
|
||||
require.NoError(t, err)
|
||||
require.NotEmpty(t, body)
|
||||
|
||||
for _, method := range []string{
|
||||
http.MethodGet,
|
||||
http.MethodHead,
|
||||
http.MethodPost,
|
||||
http.MethodPut,
|
||||
http.MethodDelete,
|
||||
} {
|
||||
t.Run(method, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
req := httptest.NewRequestWithContext(
|
||||
context.Background(), method,
|
||||
"/s/js/app.js", nil,
|
||||
)
|
||||
w := httptest.NewRecorder()
|
||||
env.router.ServeHTTP(w, req)
|
||||
|
||||
assert.Equal(t, http.StatusOK, w.Code,
|
||||
"static mount answers every method")
|
||||
|
||||
if method == http.MethodHead {
|
||||
assert.Empty(t, w.Body.Bytes(),
|
||||
"HEAD must not carry a body")
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
assert.Equal(t, body, w.Body.Bytes(),
|
||||
"the asset itself is returned")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// --- /pages group ---
|
||||
|
||||
// TestPagesLogin_OversizeBody_RejectedBeforeCSRF proves the cap runs
|
||||
|
||||
@@ -24,15 +24,48 @@ import (
|
||||
)
|
||||
|
||||
const (
|
||||
// shutdownTimeout is the maximum time to wait for the HTTP
|
||||
// ShutdownTimeout is the maximum time to wait for the HTTP
|
||||
// server to finish in-flight requests during shutdown.
|
||||
shutdownTimeout = 5 * time.Second
|
||||
//
|
||||
// It must stay strictly below the fx stop timeout in
|
||||
// cmd/webhooker, which bounds the whole stop sequence: a drain
|
||||
// that used the entire sequence budget would leave nothing for
|
||||
// the hooks that run after the server, including the database
|
||||
// close. It is exported so that relationship can be tested.
|
||||
ShutdownTimeout = 3 * time.Second
|
||||
|
||||
// sentryFlushTimeout is the maximum time to wait for Sentry
|
||||
// to flush pending events during shutdown.
|
||||
// TailHookReserve is the share of the fx stop budget this hook
|
||||
// refuses to spend, leaving it for the hooks that run after the
|
||||
// server: the delivery engine, the healthcheck, the webhook DB
|
||||
// manager and the database close.
|
||||
TailHookReserve = 2 * time.Second
|
||||
|
||||
// sentryFlushTimeout is the longest wait for Sentry to flush
|
||||
// pending events during shutdown, before the remaining stop
|
||||
// budget is taken into account.
|
||||
sentryFlushTimeout = 2 * time.Second
|
||||
|
||||
// minSentryFlush is the shortest flush worth attempting. Below
|
||||
// it the remaining budget goes to the tail hooks instead.
|
||||
minSentryFlush = 250 * time.Millisecond
|
||||
)
|
||||
|
||||
// SentryFlushBudget reports how long the Sentry flush may run when
|
||||
// remaining is the time left on the fx stop context after the HTTP
|
||||
// drain. sentry.Flush takes a bare duration and honours no context,
|
||||
// so this clamp is the only thing keeping a stalled flush from
|
||||
// spending the tail hooks' share of the budget on top of a
|
||||
// full-length drain. TailHookReserve is held back, and anything
|
||||
// under minSentryFlush is skipped rather than attempted uselessly.
|
||||
func SentryFlushBudget(remaining time.Duration) time.Duration {
|
||||
budget := min(remaining-TailHookReserve, sentryFlushTimeout)
|
||||
if budget < minSentryFlush {
|
||||
return 0
|
||||
}
|
||||
|
||||
return budget
|
||||
}
|
||||
|
||||
//nolint:revive // ServerParams is a standard fx naming convention.
|
||||
type ServerParams struct {
|
||||
fx.In
|
||||
@@ -164,7 +197,7 @@ func (s *Server) cleanShutdown(ctx context.Context) {
|
||||
s.exitCode = 0
|
||||
|
||||
ctxShutdown, shutdownCancel := context.WithTimeout(
|
||||
ctx, shutdownTimeout,
|
||||
ctx, ShutdownTimeout,
|
||||
)
|
||||
defer shutdownCancel()
|
||||
|
||||
@@ -178,10 +211,31 @@ func (s *Server) cleanShutdown(ctx context.Context) {
|
||||
s.cleanupForExit()
|
||||
|
||||
if s.sentryEnabled {
|
||||
sentry.Flush(sentryFlushTimeout)
|
||||
s.flushSentry(ctx)
|
||||
}
|
||||
}
|
||||
|
||||
// flushSentry drains Sentry's queue inside what is left of the fx
|
||||
// stop budget. A context carrying no deadline — a caller outside the
|
||||
// fx lifecycle — gets the full timeout.
|
||||
func (s *Server) flushSentry(ctx context.Context) {
|
||||
flush := sentryFlushTimeout
|
||||
|
||||
if deadline, ok := ctx.Deadline(); ok {
|
||||
flush = SentryFlushBudget(time.Until(deadline))
|
||||
}
|
||||
|
||||
if flush <= 0 {
|
||||
s.log.Warn(
|
||||
"skipping sentry flush, stop budget exhausted",
|
||||
)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
sentry.Flush(flush)
|
||||
}
|
||||
|
||||
func (s *Server) configure() {
|
||||
// identify ourselves in the logs
|
||||
s.params.Logger.Identify()
|
||||
|
||||
59
internal/server/shutdown_test.go
Normal file
59
internal/server/shutdown_test.go
Normal file
@@ -0,0 +1,59 @@
|
||||
package server_test
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/webhooker/internal/server"
|
||||
)
|
||||
|
||||
// TestSentryFlushBudget covers the clamp that keeps the Sentry flush
|
||||
// from spending the tail hooks' share of the fx stop budget.
|
||||
// sentry.Flush ignores the stop context, so without the clamp a
|
||||
// stalled flush adds its whole timeout on top of the HTTP drain.
|
||||
func TestSentryFlushBudget(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
remaining time.Duration
|
||||
want time.Duration
|
||||
}{
|
||||
{
|
||||
name: "full drain leaves only the reserve",
|
||||
remaining: server.TailHookReserve,
|
||||
want: 0,
|
||||
},
|
||||
{
|
||||
name: "expired budget",
|
||||
remaining: -time.Second,
|
||||
want: 0,
|
||||
},
|
||||
{
|
||||
name: "sliver above the reserve is not worth it",
|
||||
remaining: server.TailHookReserve + 10*time.Millisecond,
|
||||
want: 0,
|
||||
},
|
||||
{
|
||||
name: "partial flush when some room is left",
|
||||
remaining: server.TailHookReserve + time.Second,
|
||||
want: time.Second,
|
||||
},
|
||||
{
|
||||
name: "capped at the nominal timeout",
|
||||
remaining: time.Hour,
|
||||
want: 2 * time.Second,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
require.Equal(
|
||||
t, tt.want, server.SentryFlushBudget(tt.remaining),
|
||||
)
|
||||
})
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user