Verify login credentials before spending rate-limit budget (closes #150)
All checks were successful
check / check (push) Successful in 2m46s

In the shipped default, any stranger denied the operator the only
administrative path at 5 requests per minute: TRUSTED_PROXIES is empty,
the README requires a reverse proxy, so every login POST shared one
bucket keyed on the proxy.

Credentials are now verified first and only a FAILED attempt spends
budget, so a correct password is never throttled. Failures are counted
per (client bucket, submitted username), bounded. Concurrent Argon2id
verifications are capped at two, and the queue for them at 16 — because
verifying first lets an attacker force a 64 MB hash per request, and
bounding the wait alone bounds nothing.

The issue's own recommendation was insufficient and is rejected here:
keying by username stops an attacker locking out a DIFFERENT account,
but this is a single-admin product with a predictable bootstrap
username, so flooding the operator's own name still locks them out.

This is speculative — it implements a corrected recommendation ahead of
the owner's ruling so the decision can be made by merging or reverting.
Three things are disclosed rather than glossed: online guessing rises
from 5/min to roughly 27/s, because the 429 is a label on the response
and not a gate in front of the hash; the residual exposure is a loss of
login AVAILABILITY, not latency, and a determined flood still denies
login while it runs, at ~400x the cost and clearing the moment it
stops; and the endpoint should be provisioned for ~400 MB resident, not
the 203 MB of live commitment it itemises.

Independently reviewed four times. Reviewers disproved the suspected
FIFO starvation by measurement, then caught two successive memory
bounds the code did not have — the second by parking waiters and
reading the heap rather than checking the arithmetic.
This commit was merged in pull request #171.
This commit is contained in:
2026-08-18 01:55:41 +02:00
parent 992b3c68f5
commit 977fe87588
20 changed files with 2010 additions and 139 deletions

188
README.md
View File

@@ -113,7 +113,7 @@ TTY detection, and security headers are always applied.
| `RETENTION_SWEEP_INTERVAL` | How often the retention reaper and archive sweeper run (Go duration, must be positive) | `1h` |
| `SESSION_IDLE_TIMEOUT` | Idle session timeout (Go duration) | `24h` |
| `RECEIVER_RATE_LIMIT` | Receiver requests/minute per IP per entrypoint (10x that per IP across the route) | `120` |
| `TRUSTED_PROXIES` | CIDRs whose forwarded headers are trusted (unset: all clients behind a proxy share one rate-limit bucket) | `""` (none) |
| `TRUSTED_PROXIES` | CIDRs whose forwarded headers are trusted (unset: all clients behind a proxy share one rate-limit bucket; a correct login password is never throttled either way) | `""` (none) |
#### Trusted proxies
@@ -136,13 +136,15 @@ That default is safe against forged headers, but leaving it unset in
production has a cost you must know about. Production runs behind a
TLS-terminating reverse proxy, so with `TRUSTED_PROXIES` unset every
request keys on the proxy's own address and all clients share a single
bucket per limit. For the login and password-change limits that is a
denial of service anyone can perform: a steady five POSTs per minute
from any address on the internet keeps the shared login bucket full,
and the operator's own login then returns HTTP 429 for as long as the
trickle continues. There is no second administrative path and no
bypass. Restarting the service clears the in-memory buckets, but a
sustained trickle re-locks them immediately.
bucket per limit. The receiver limits become service-wide ceilings,
and the login endpoint's failure counting collapses onto one key, so a
stranger's wrong passwords throttle every other client's wrong
passwords.
What it cannot do is lock the operator out. The login endpoint
verifies credentials **before** it consults any limit and charges only
failures, so a correct password is never throttled no matter how full
the bucket is. See [Rate Limiting](#rate-limiting).
The remedy is to set `TRUSTED_PROXIES` to your reverse proxy's
address, which restores per-client buckets. webhooker logs a warning
@@ -382,11 +384,12 @@ It uses:
- **[gorilla/csrf](https://github.com/gorilla/csrf)** for CSRF
protection (cookie-based double-submit tokens)
- **[go-chi/httprate](https://github.com/go-chi/httprate)** for
sliding-window rate limiting of the login, password-change and
webhook receiver endpoints. The bucket is per client IP only when
sliding-window rate limiting of the password-change and webhook
receiver endpoints. The bucket is per client IP only when
`TRUSTED_PROXIES` names the reverse proxy; unset, every client
behind that proxy shares one bucket per limit (see
[Rate Limiting](#rate-limiting))
behind that proxy shares one bucket per limit. The login endpoint
counts failed attempts itself instead, so that a correct password is
never throttled (see [Rate Limiting](#rate-limiting))
- **[Prometheus](https://prometheus.io)** for metrics, served at
`/metrics` behind basic auth
- **[Sentry](https://sentry.io)** for optional error reporting
@@ -1091,14 +1094,117 @@ opposite directions:
and all entrypoints, where the per-entrypoint limit's capacity still
grows with the number of entrypoints. Any deployment with more than a
handful of busy entrypoints must set `TRUSTED_PROXIES`.
- For the **login and password-change** limits it costs availability of
the only administrative path, which is not safe at all. Five POSTs
per minute from any address on the internet keeps the single shared
login bucket full, and the operator's own login returns HTTP 429 for
as long as that trickle continues. A restart clears the in-memory
buckets and a resumed trickle re-locks them. Production deployments
must set `TRUSTED_PROXIES`; webhooker warns at startup whenever it is
empty, in any environment.
- For the **login and password-change** limits it costs precision, not
availability. Login failures from every client land in one counter,
so a stranger's wrong passwords make the operator's own wrong
passwords answer `429` sooner; the operator's _correct_ password is
never affected, because it is never counted. Production deployments
should still set `TRUSTED_PROXIES`; webhooker warns at startup
whenever it is empty, in any environment.
#### The login endpoint
The login `POST` is the one endpoint with no pre-emptive limiter in
front of it, and that is deliberate. A limiter that spends budget on
arrival is a lockout in this deployment shape: sharing one bucket, a
stranger sending five POSTs a minute — about 0.08 requests per second,
from anywhere — keeps it permanently full, and the operator has no
second administrative path. So the handler inverts the order:
1. **Credentials are verified first, and only a failed attempt spends
budget.** A correct password is never rate-limited, whatever the
counters hold. This is what guarantees the admin UI stays
reachable.
2. **Failures are counted per (client bucket, submitted username)**,
five per minute, after which further _failures_ from that pair are
answered `429` with a `Retry-After`. That `429` is a label on the
response, not a gate in front of the work: the credential check has
already run by the time the counter is consulted, so a throttled
client's guess is still evaluated. See the guessing rate below. A
successful login clears the counter, so mistyping a few times and
then getting it right leaves you unthrottled. Because the submitted
username is attacker-controlled, at most 1024 username counters and
1024 fallback address counters are tracked; past the first cap
failures fall back to the address counter, and past both they are
answered as throttled without being recorded. Total tracked state
is under half a megabyte and does not grow with the number of
usernames an attacker invents.
3. **Concurrent password verifications are capped at two, and the
queue for them at 16.** Verifying before counting means every login
request costs an Argon2id hash, and Argon2id here is 64 MB per
hash — two slots is a 128 MB ceiling on password hashing. Every
endpoint that hashes a password takes a slot, including the
password-change endpoint, which holds one across both the
verification and the new hash. A request that waits five seconds
without getting a slot is answered `503 Service Unavailable` and no
hash is computed for it. The wait alone does not bound memory, only
how long one request holds some, so the number of waiters is capped
as well. Size the queue from what a parked waiter actually retains,
not from the 1 MB body cap: that caps the raw body read, while the
body-cap, CSRF and form-parsing middleware all run before the
guard, so a waiter holds its parsed form plus its request header
block for the whole wait. Measured on the pinned Go 1.26.1
toolchain, as the heap delta with 64 waiters parked in the handler,
an ordinary two-field login form retains ~0 MB, a 1 MB urlencoded
body at Go's 10,000-parameter parse cap retains 2.82 MB (3.09 MB
with `%41` escapes), and the ~0.9 MB of headers the 1 MB header cap
allows takes it to **4.18 MB** — the retained parse and the headers
dominate, not the raw body. So the cap is 16 waiters: 16 x 4.18 MB
is about 67 MB of committed queue memory, and two slots drain a
full 16-deep queue in roughly 0.6 s, far inside the five-second
deadline. A request arriving past the cap is shed with `503`
immediately instead of joining the queue. **Peak commitment for the
endpoint is therefore about 203 MB**: 128 MB of Argon2id, plus the
18 requests holding a parsed form — 16 queued and the 2 being
hashed — at about 75 MB. That 203 MB is _live_ commitment, not
resident size: the Go collector lets the heap reach roughly twice
the live set before collecting, with transient parse garbage on top
of it. The independent review of this endpoint fired 18 adversarial
requests at an idle guard and measured a peak `HeapAlloc` of
392 MB. **Provision on the order of 400 MB**, not for the 203 MB
itemised here and not for the hashing budget alone.
An unknown username is verified against a dummy hash rather than
rejected early, so a nonexistent account costs the same time as a real
one and the response cannot be used to enumerate usernames.
**This raises online guessing throughput by about 300x, and that is
the trade.** Because the credential check always precedes the counter,
what bounds online brute force is the semaphore, not the failure
counter. Two slots at the cost of one Argon2id verification is on the
order of **27 guesses per second, about 2.3 million per day**, against
5 per minute under the pre-emptive limiter this replaced. Treat that
figure as a lower bound rather than a ceiling: it was measured with
Go's race detector enabled, so real hardware verifies faster and
guesses faster. Choose the admin password to survive millions of
online guesses per day — a long random passphrase, not a memorable
one. Rate-limiting `POST /pages/login` at the reverse proxy, where the
real client address is visible, is the way to put a cheaper bound back
on top.
The residual exposure is a bounded, self-clearing loss of login
**availability** — not merely of latency. A flood can keep both
verification slots busy, and a request that neither gets a slot within
five seconds nor finds room in the queue is answered `503`. Above
roughly 27 requests per second the operator is not served slowly, it
is shed: its chance per attempt is about the ratio of service rate to
flood rate, so at 400 requests per second it is roughly one attempt in
fourteen. A sufficiently determined flood still denies login for as
long as it runs.
What changed is the price and the aftermath. Denying login used to
cost an attacker 0.08 requests per second from anywhere; it now costs
30 or more sustained, about 400 times as much. Nothing accumulates
while the flood runs, nothing needs resetting when it stops, and the
operator's correct password succeeds on the first attempt afterwards.
Restarting the service is **not** a remedy: a restart clears the
failure counters, which are not what is saturated, and the flood
re-fills both verification slots on its first two requests. The
remedies are to block the source at the reverse proxy, or to
rate-limit `POST /pages/login` there — the one place a limit can be
applied without reintroducing the lockout, because the proxy sees the
real client address. Setting `TRUSTED_PROXIES` does not stop the
saturation, but it makes the source visible in the failure logs.
Finer-grained per-webhook rate limits (configured in the web UI and
enforced in the webhook handler) can layer on top of this env-level
@@ -1119,8 +1225,8 @@ abuse limit later; they are tracked as future work.
| Method | Path | Description |
| ------ | --------------- | ----------- |
| `GET` | `/pages/login` | Login page (not rate limited; the limiter applies to POST only) |
| `POST` | `/pages/login` | Login form submission (5 per minute per bucket, then 429) |
| `GET` | `/pages/login` | Login page (not rate limited) |
| `POST` | `/pages/login` | Login form submission. Credentials are verified before any limit is consulted, so a correct password is never throttled; 5 FAILED attempts per minute per bucket per submitted username, then `429`. `503` if no verification slot frees up within 5s, or immediately if 16 requests are already queued for one (see [Rate Limiting](#rate-limiting)) |
| `POST` | `/pages/logout` | Logout (destroys session) |
#### Authenticated Endpoints
@@ -1128,7 +1234,7 @@ abuse limit later; they are tracked as future work.
| Method | Path | Description |
| ------ | ------------------------ | ----------- |
| `GET` | `/user/{username}` | User profile page |
| `POST` | `/user/{username}/password` | Change the user's password (5 per minute per bucket, then 429) |
| `POST` | `/user/{username}/password` | Change the user's password (5 per minute per bucket, then `429`; `503` if no verification slot frees up within 5s, or immediately if 16 requests are already queued for one) |
| `GET` | `/sources` | List user's webhooks |
| `GET` | `/sources/new` | Create webhook form |
| `POST` | `/sources/new` | Create webhook submission |
@@ -1230,6 +1336,7 @@ webhooker/
│ │ ├── middleware.go # Logging, CORS, Auth, Metrics, MetricsAuth, SecurityHeaders, MaxBodySize
│ │ ├── csrf.go # CSRF protection middleware (gorilla/csrf)
│ │ ├── ratelimit.go # Per-IP rate limiting middleware (go-chi/httprate)
│ │ ├── loginguard.go # Login failure counters and the Argon2id verification semaphore
│ │ └── testing.go # NewForTest: Middleware without the fx lifecycle
│ ├── server/
│ │ ├── server.go # Server struct, fx lifecycle, signal handling
@@ -1327,9 +1434,11 @@ one that lies about its length, is hard-capped by
Those same four route groups then apply **CSRF** and **NoCache**
(`Cache-Control: no-store`, `Pragma: no-cache`), and every group except
`/pages` applies **RequireAuth**. The rate limiters are per-route
rather than global: **LoginRateLimit** on `/pages/login`,
**PasswordChangeRateLimit** on `/user/{username}/password`, and
**ReceiverRateLimit** on `/webhook/{uuid}`.
rather than global: **PasswordChangeRateLimit** on
`/user/{username}/password` and **ReceiverRateLimit** on
`/webhook/{uuid}`. There is deliberately none on `/pages/login` — that
endpoint counts failures inside the handler, after the credential
check, see [The login endpoint](#the-login-endpoint).
### Authentication
@@ -1367,14 +1476,27 @@ rather than global: **LoginRateLimit** on `/pages/login`,
both at target creation time (URL validation) and at delivery time
(custom HTTP transport with SSRF-safe dialer that validates resolved
IPs before connecting, preventing DNS rebinding attacks)
- **Login rate limiting** via [go-chi/httprate](https://github.com/go-chi/httprate):
sliding-window rate limiter on the login endpoint, 5 POST attempts
per minute per bucket, to slow brute-force attacks. GET requests to
the login page are not limited. The password-change endpoint carries
the same 5-per-minute limit. The bucket is per client IP only when
- **Login limiting is inverted, deliberately.** The login `POST` has
no pre-emptive rate limiter in front of it. Credentials are
verified first and only a _failed_ attempt spends budget, so a
correct password is never throttled and no flood of wrong ones can
deny the operator the only administrative path. Failures are
counted per (bucket, submitted username), five per minute, after
which further failures are answered `429` with a `Retry-After`.
What bounds brute force is not that counter but the cap of two
concurrent Argon2id verifications: a throttled client's guess is
still evaluated, so roughly 27 guesses a second get through and the
admin password has to carry that load (see
[The login endpoint](#the-login-endpoint)). `GET` requests to the
login page are not limited
- **Password-change rate limiting** via [go-chi/httprate](https://github.com/go-chi/httprate):
sliding-window rate limiter, 5 POST attempts per minute per bucket.
It runs behind session auth, so only a client already holding a
valid session reaches it, and an operator throttled out of changing
a password can still log in. The bucket is per client IP only when
`TRUSTED_PROXIES` names the reverse proxy; unset, every client
shares one bucket and the login becomes remotely deniable (see
[Rate Limiting](#rate-limiting)). webhooker warns at startup
shares one bucket, which costs precision rather than availability
(see [Rate Limiting](#rate-limiting)). webhooker warns at startup
whenever `TRUSTED_PROXIES` is empty
- Prometheus metrics behind basic auth
- Static assets embedded in binary (no filesystem access needed at