Verify login credentials before spending rate-limit budget (closes #150)
All checks were successful
check / check (push) Successful in 2m46s
All checks were successful
check / check (push) Successful in 2m46s
In the shipped default, any stranger denied the operator the only administrative path at 5 requests per minute: TRUSTED_PROXIES is empty, the README requires a reverse proxy, so every login POST shared one bucket keyed on the proxy. Credentials are now verified first and only a FAILED attempt spends budget, so a correct password is never throttled. Failures are counted per (client bucket, submitted username), bounded. Concurrent Argon2id verifications are capped at two, and the queue for them at 16 — because verifying first lets an attacker force a 64 MB hash per request, and bounding the wait alone bounds nothing. The issue's own recommendation was insufficient and is rejected here: keying by username stops an attacker locking out a DIFFERENT account, but this is a single-admin product with a predictable bootstrap username, so flooding the operator's own name still locks them out. This is speculative — it implements a corrected recommendation ahead of the owner's ruling so the decision can be made by merging or reverting. Three things are disclosed rather than glossed: online guessing rises from 5/min to roughly 27/s, because the 429 is a label on the response and not a gate in front of the hash; the residual exposure is a loss of login AVAILABILITY, not latency, and a determined flood still denies login while it runs, at ~400x the cost and clearing the moment it stops; and the endpoint should be provisioned for ~400 MB resident, not the 203 MB of live commitment it itemises. Independently reviewed four times. Reviewers disproved the suspected FIFO starvation by measurement, then caught two successive memory bounds the code did not have — the second by parking waiters and reading the heap rather than checking the arithmetic.
This commit was merged in pull request #171.
This commit is contained in:
188
README.md
188
README.md
@@ -113,7 +113,7 @@ TTY detection, and security headers are always applied.
|
||||
| `RETENTION_SWEEP_INTERVAL` | How often the retention reaper and archive sweeper run (Go duration, must be positive) | `1h` |
|
||||
| `SESSION_IDLE_TIMEOUT` | Idle session timeout (Go duration) | `24h` |
|
||||
| `RECEIVER_RATE_LIMIT` | Receiver requests/minute per IP per entrypoint (10x that per IP across the route) | `120` |
|
||||
| `TRUSTED_PROXIES` | CIDRs whose forwarded headers are trusted (unset: all clients behind a proxy share one rate-limit bucket) | `""` (none) |
|
||||
| `TRUSTED_PROXIES` | CIDRs whose forwarded headers are trusted (unset: all clients behind a proxy share one rate-limit bucket; a correct login password is never throttled either way) | `""` (none) |
|
||||
|
||||
#### Trusted proxies
|
||||
|
||||
@@ -136,13 +136,15 @@ That default is safe against forged headers, but leaving it unset in
|
||||
production has a cost you must know about. Production runs behind a
|
||||
TLS-terminating reverse proxy, so with `TRUSTED_PROXIES` unset every
|
||||
request keys on the proxy's own address and all clients share a single
|
||||
bucket per limit. For the login and password-change limits that is a
|
||||
denial of service anyone can perform: a steady five POSTs per minute
|
||||
from any address on the internet keeps the shared login bucket full,
|
||||
and the operator's own login then returns HTTP 429 for as long as the
|
||||
trickle continues. There is no second administrative path and no
|
||||
bypass. Restarting the service clears the in-memory buckets, but a
|
||||
sustained trickle re-locks them immediately.
|
||||
bucket per limit. The receiver limits become service-wide ceilings,
|
||||
and the login endpoint's failure counting collapses onto one key, so a
|
||||
stranger's wrong passwords throttle every other client's wrong
|
||||
passwords.
|
||||
|
||||
What it cannot do is lock the operator out. The login endpoint
|
||||
verifies credentials **before** it consults any limit and charges only
|
||||
failures, so a correct password is never throttled no matter how full
|
||||
the bucket is. See [Rate Limiting](#rate-limiting).
|
||||
|
||||
The remedy is to set `TRUSTED_PROXIES` to your reverse proxy's
|
||||
address, which restores per-client buckets. webhooker logs a warning
|
||||
@@ -382,11 +384,12 @@ It uses:
|
||||
- **[gorilla/csrf](https://github.com/gorilla/csrf)** for CSRF
|
||||
protection (cookie-based double-submit tokens)
|
||||
- **[go-chi/httprate](https://github.com/go-chi/httprate)** for
|
||||
sliding-window rate limiting of the login, password-change and
|
||||
webhook receiver endpoints. The bucket is per client IP only when
|
||||
sliding-window rate limiting of the password-change and webhook
|
||||
receiver endpoints. The bucket is per client IP only when
|
||||
`TRUSTED_PROXIES` names the reverse proxy; unset, every client
|
||||
behind that proxy shares one bucket per limit (see
|
||||
[Rate Limiting](#rate-limiting))
|
||||
behind that proxy shares one bucket per limit. The login endpoint
|
||||
counts failed attempts itself instead, so that a correct password is
|
||||
never throttled (see [Rate Limiting](#rate-limiting))
|
||||
- **[Prometheus](https://prometheus.io)** for metrics, served at
|
||||
`/metrics` behind basic auth
|
||||
- **[Sentry](https://sentry.io)** for optional error reporting
|
||||
@@ -1091,14 +1094,117 @@ opposite directions:
|
||||
and all entrypoints, where the per-entrypoint limit's capacity still
|
||||
grows with the number of entrypoints. Any deployment with more than a
|
||||
handful of busy entrypoints must set `TRUSTED_PROXIES`.
|
||||
- For the **login and password-change** limits it costs availability of
|
||||
the only administrative path, which is not safe at all. Five POSTs
|
||||
per minute from any address on the internet keeps the single shared
|
||||
login bucket full, and the operator's own login returns HTTP 429 for
|
||||
as long as that trickle continues. A restart clears the in-memory
|
||||
buckets and a resumed trickle re-locks them. Production deployments
|
||||
must set `TRUSTED_PROXIES`; webhooker warns at startup whenever it is
|
||||
empty, in any environment.
|
||||
- For the **login and password-change** limits it costs precision, not
|
||||
availability. Login failures from every client land in one counter,
|
||||
so a stranger's wrong passwords make the operator's own wrong
|
||||
passwords answer `429` sooner; the operator's _correct_ password is
|
||||
never affected, because it is never counted. Production deployments
|
||||
should still set `TRUSTED_PROXIES`; webhooker warns at startup
|
||||
whenever it is empty, in any environment.
|
||||
|
||||
#### The login endpoint
|
||||
|
||||
The login `POST` is the one endpoint with no pre-emptive limiter in
|
||||
front of it, and that is deliberate. A limiter that spends budget on
|
||||
arrival is a lockout in this deployment shape: sharing one bucket, a
|
||||
stranger sending five POSTs a minute — about 0.08 requests per second,
|
||||
from anywhere — keeps it permanently full, and the operator has no
|
||||
second administrative path. So the handler inverts the order:
|
||||
|
||||
1. **Credentials are verified first, and only a failed attempt spends
|
||||
budget.** A correct password is never rate-limited, whatever the
|
||||
counters hold. This is what guarantees the admin UI stays
|
||||
reachable.
|
||||
2. **Failures are counted per (client bucket, submitted username)**,
|
||||
five per minute, after which further _failures_ from that pair are
|
||||
answered `429` with a `Retry-After`. That `429` is a label on the
|
||||
response, not a gate in front of the work: the credential check has
|
||||
already run by the time the counter is consulted, so a throttled
|
||||
client's guess is still evaluated. See the guessing rate below. A
|
||||
successful login clears the counter, so mistyping a few times and
|
||||
then getting it right leaves you unthrottled. Because the submitted
|
||||
username is attacker-controlled, at most 1024 username counters and
|
||||
1024 fallback address counters are tracked; past the first cap
|
||||
failures fall back to the address counter, and past both they are
|
||||
answered as throttled without being recorded. Total tracked state
|
||||
is under half a megabyte and does not grow with the number of
|
||||
usernames an attacker invents.
|
||||
3. **Concurrent password verifications are capped at two, and the
|
||||
queue for them at 16.** Verifying before counting means every login
|
||||
request costs an Argon2id hash, and Argon2id here is 64 MB per
|
||||
hash — two slots is a 128 MB ceiling on password hashing. Every
|
||||
endpoint that hashes a password takes a slot, including the
|
||||
password-change endpoint, which holds one across both the
|
||||
verification and the new hash. A request that waits five seconds
|
||||
without getting a slot is answered `503 Service Unavailable` and no
|
||||
hash is computed for it. The wait alone does not bound memory, only
|
||||
how long one request holds some, so the number of waiters is capped
|
||||
as well. Size the queue from what a parked waiter actually retains,
|
||||
not from the 1 MB body cap: that caps the raw body read, while the
|
||||
body-cap, CSRF and form-parsing middleware all run before the
|
||||
guard, so a waiter holds its parsed form plus its request header
|
||||
block for the whole wait. Measured on the pinned Go 1.26.1
|
||||
toolchain, as the heap delta with 64 waiters parked in the handler,
|
||||
an ordinary two-field login form retains ~0 MB, a 1 MB urlencoded
|
||||
body at Go's 10,000-parameter parse cap retains 2.82 MB (3.09 MB
|
||||
with `%41` escapes), and the ~0.9 MB of headers the 1 MB header cap
|
||||
allows takes it to **4.18 MB** — the retained parse and the headers
|
||||
dominate, not the raw body. So the cap is 16 waiters: 16 x 4.18 MB
|
||||
is about 67 MB of committed queue memory, and two slots drain a
|
||||
full 16-deep queue in roughly 0.6 s, far inside the five-second
|
||||
deadline. A request arriving past the cap is shed with `503`
|
||||
immediately instead of joining the queue. **Peak commitment for the
|
||||
endpoint is therefore about 203 MB**: 128 MB of Argon2id, plus the
|
||||
18 requests holding a parsed form — 16 queued and the 2 being
|
||||
hashed — at about 75 MB. That 203 MB is _live_ commitment, not
|
||||
resident size: the Go collector lets the heap reach roughly twice
|
||||
the live set before collecting, with transient parse garbage on top
|
||||
of it. The independent review of this endpoint fired 18 adversarial
|
||||
requests at an idle guard and measured a peak `HeapAlloc` of
|
||||
392 MB. **Provision on the order of 400 MB**, not for the 203 MB
|
||||
itemised here and not for the hashing budget alone.
|
||||
|
||||
An unknown username is verified against a dummy hash rather than
|
||||
rejected early, so a nonexistent account costs the same time as a real
|
||||
one and the response cannot be used to enumerate usernames.
|
||||
|
||||
**This raises online guessing throughput by about 300x, and that is
|
||||
the trade.** Because the credential check always precedes the counter,
|
||||
what bounds online brute force is the semaphore, not the failure
|
||||
counter. Two slots at the cost of one Argon2id verification is on the
|
||||
order of **27 guesses per second, about 2.3 million per day**, against
|
||||
5 per minute under the pre-emptive limiter this replaced. Treat that
|
||||
figure as a lower bound rather than a ceiling: it was measured with
|
||||
Go's race detector enabled, so real hardware verifies faster and
|
||||
guesses faster. Choose the admin password to survive millions of
|
||||
online guesses per day — a long random passphrase, not a memorable
|
||||
one. Rate-limiting `POST /pages/login` at the reverse proxy, where the
|
||||
real client address is visible, is the way to put a cheaper bound back
|
||||
on top.
|
||||
|
||||
The residual exposure is a bounded, self-clearing loss of login
|
||||
**availability** — not merely of latency. A flood can keep both
|
||||
verification slots busy, and a request that neither gets a slot within
|
||||
five seconds nor finds room in the queue is answered `503`. Above
|
||||
roughly 27 requests per second the operator is not served slowly, it
|
||||
is shed: its chance per attempt is about the ratio of service rate to
|
||||
flood rate, so at 400 requests per second it is roughly one attempt in
|
||||
fourteen. A sufficiently determined flood still denies login for as
|
||||
long as it runs.
|
||||
|
||||
What changed is the price and the aftermath. Denying login used to
|
||||
cost an attacker 0.08 requests per second from anywhere; it now costs
|
||||
30 or more sustained, about 400 times as much. Nothing accumulates
|
||||
while the flood runs, nothing needs resetting when it stops, and the
|
||||
operator's correct password succeeds on the first attempt afterwards.
|
||||
Restarting the service is **not** a remedy: a restart clears the
|
||||
failure counters, which are not what is saturated, and the flood
|
||||
re-fills both verification slots on its first two requests. The
|
||||
remedies are to block the source at the reverse proxy, or to
|
||||
rate-limit `POST /pages/login` there — the one place a limit can be
|
||||
applied without reintroducing the lockout, because the proxy sees the
|
||||
real client address. Setting `TRUSTED_PROXIES` does not stop the
|
||||
saturation, but it makes the source visible in the failure logs.
|
||||
|
||||
Finer-grained per-webhook rate limits (configured in the web UI and
|
||||
enforced in the webhook handler) can layer on top of this env-level
|
||||
@@ -1119,8 +1225,8 @@ abuse limit later; they are tracked as future work.
|
||||
|
||||
| Method | Path | Description |
|
||||
| ------ | --------------- | ----------- |
|
||||
| `GET` | `/pages/login` | Login page (not rate limited; the limiter applies to POST only) |
|
||||
| `POST` | `/pages/login` | Login form submission (5 per minute per bucket, then 429) |
|
||||
| `GET` | `/pages/login` | Login page (not rate limited) |
|
||||
| `POST` | `/pages/login` | Login form submission. Credentials are verified before any limit is consulted, so a correct password is never throttled; 5 FAILED attempts per minute per bucket per submitted username, then `429`. `503` if no verification slot frees up within 5s, or immediately if 16 requests are already queued for one (see [Rate Limiting](#rate-limiting)) |
|
||||
| `POST` | `/pages/logout` | Logout (destroys session) |
|
||||
|
||||
#### Authenticated Endpoints
|
||||
@@ -1128,7 +1234,7 @@ abuse limit later; they are tracked as future work.
|
||||
| Method | Path | Description |
|
||||
| ------ | ------------------------ | ----------- |
|
||||
| `GET` | `/user/{username}` | User profile page |
|
||||
| `POST` | `/user/{username}/password` | Change the user's password (5 per minute per bucket, then 429) |
|
||||
| `POST` | `/user/{username}/password` | Change the user's password (5 per minute per bucket, then `429`; `503` if no verification slot frees up within 5s, or immediately if 16 requests are already queued for one) |
|
||||
| `GET` | `/sources` | List user's webhooks |
|
||||
| `GET` | `/sources/new` | Create webhook form |
|
||||
| `POST` | `/sources/new` | Create webhook submission |
|
||||
@@ -1230,6 +1336,7 @@ webhooker/
|
||||
│ │ ├── middleware.go # Logging, CORS, Auth, Metrics, MetricsAuth, SecurityHeaders, MaxBodySize
|
||||
│ │ ├── csrf.go # CSRF protection middleware (gorilla/csrf)
|
||||
│ │ ├── ratelimit.go # Per-IP rate limiting middleware (go-chi/httprate)
|
||||
│ │ ├── loginguard.go # Login failure counters and the Argon2id verification semaphore
|
||||
│ │ └── testing.go # NewForTest: Middleware without the fx lifecycle
|
||||
│ ├── server/
|
||||
│ │ ├── server.go # Server struct, fx lifecycle, signal handling
|
||||
@@ -1327,9 +1434,11 @@ one that lies about its length, is hard-capped by
|
||||
Those same four route groups then apply **CSRF** and **NoCache**
|
||||
(`Cache-Control: no-store`, `Pragma: no-cache`), and every group except
|
||||
`/pages` applies **RequireAuth**. The rate limiters are per-route
|
||||
rather than global: **LoginRateLimit** on `/pages/login`,
|
||||
**PasswordChangeRateLimit** on `/user/{username}/password`, and
|
||||
**ReceiverRateLimit** on `/webhook/{uuid}`.
|
||||
rather than global: **PasswordChangeRateLimit** on
|
||||
`/user/{username}/password` and **ReceiverRateLimit** on
|
||||
`/webhook/{uuid}`. There is deliberately none on `/pages/login` — that
|
||||
endpoint counts failures inside the handler, after the credential
|
||||
check, see [The login endpoint](#the-login-endpoint).
|
||||
|
||||
### Authentication
|
||||
|
||||
@@ -1367,14 +1476,27 @@ rather than global: **LoginRateLimit** on `/pages/login`,
|
||||
both at target creation time (URL validation) and at delivery time
|
||||
(custom HTTP transport with SSRF-safe dialer that validates resolved
|
||||
IPs before connecting, preventing DNS rebinding attacks)
|
||||
- **Login rate limiting** via [go-chi/httprate](https://github.com/go-chi/httprate):
|
||||
sliding-window rate limiter on the login endpoint, 5 POST attempts
|
||||
per minute per bucket, to slow brute-force attacks. GET requests to
|
||||
the login page are not limited. The password-change endpoint carries
|
||||
the same 5-per-minute limit. The bucket is per client IP only when
|
||||
- **Login limiting is inverted, deliberately.** The login `POST` has
|
||||
no pre-emptive rate limiter in front of it. Credentials are
|
||||
verified first and only a _failed_ attempt spends budget, so a
|
||||
correct password is never throttled and no flood of wrong ones can
|
||||
deny the operator the only administrative path. Failures are
|
||||
counted per (bucket, submitted username), five per minute, after
|
||||
which further failures are answered `429` with a `Retry-After`.
|
||||
What bounds brute force is not that counter but the cap of two
|
||||
concurrent Argon2id verifications: a throttled client's guess is
|
||||
still evaluated, so roughly 27 guesses a second get through and the
|
||||
admin password has to carry that load (see
|
||||
[The login endpoint](#the-login-endpoint)). `GET` requests to the
|
||||
login page are not limited
|
||||
- **Password-change rate limiting** via [go-chi/httprate](https://github.com/go-chi/httprate):
|
||||
sliding-window rate limiter, 5 POST attempts per minute per bucket.
|
||||
It runs behind session auth, so only a client already holding a
|
||||
valid session reaches it, and an operator throttled out of changing
|
||||
a password can still log in. The bucket is per client IP only when
|
||||
`TRUSTED_PROXIES` names the reverse proxy; unset, every client
|
||||
shares one bucket and the login becomes remotely deniable (see
|
||||
[Rate Limiting](#rate-limiting)). webhooker warns at startup
|
||||
shares one bucket, which costs precision rather than availability
|
||||
(see [Rate Limiting](#rate-limiting)). webhooker warns at startup
|
||||
whenever `TRUSTED_PROXIES` is empty
|
||||
- Prometheus metrics behind basic auth
|
||||
- Static assets embedded in binary (no filesystem access needed at
|
||||
|
||||
Reference in New Issue
Block a user