sneak/webhooker
1
0
Fork 0
Code Issues 34 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Verify login credentials before spending rate-limit budget (closes #150)
All checks were successful
check / check (push) Successful in 2m45s
Details

Browse Source 
With TRUSTED_PROXIES empty behind the reverse proxy production is
required to run behind, every login POST keyed on the proxy's address
and shared one 5/minute bucket. A stranger sending five POSTs a
minute -- 0.08 requests per second, from anywhere -- kept that bucket
permanently full, and the operator's own correct password was answered
429 indefinitely with no second administrative path.

The login POST no longer has a pre-emptive limiter. The handler
verifies credentials first and spends budget only on a FAILED attempt,
so a correct password is never throttled whatever the counters hold.
Three things follow, and are implemented together because the first is
unsafe without the other two:

- Failures are counted per (client bucket, submitted username), five
  per minute, after which further failures get 429 with a Retry-After.
  A successful login clears the counter, so mistyping and then
  succeeding does not leave the operator throttled.
- Both key sets are capped at 1024 entries. The submitted username is
  attacker-controlled, so past the first cap failures fall back to a
  counter keyed on the client alone, and past both caps a failure is
  answered as throttled without being recorded. Tracked state stays
  under half a megabyte and does not grow with invented usernames.
- Concurrent Argon2id verifications are capped at two, a 128 MB
  ceiling at 64 MB per hash, and the queue for those slots is capped
  at 16 waiters. Every password-hashing endpoint takes a slot,
  including the password-change endpoint, which holds one across both
  its hashes. A request that waits five seconds without a slot is
  answered 503, and one that arrives with the queue already full is
  shed with 503 immediately rather than joining it. Bounding the wait
  alone would not bound memory, and the queue depth is sized from what
  a parked waiter measurably retains rather than from the 1 MB body
  cap, which bounds only the raw body read. The body-cap, CSRF and
  form-parsing middleware all run before the guard, so a waiter holds
  its parsed form plus its request header block for the whole wait.
  Measured on the pinned go1.26.1 toolchain as the HeapAlloc delta
  across two GCs with 64 waiters parked in the handler: an ordinary
  two-field login form retains ~0 MB, a 1 MB urlencoded body at Go's
  10,000-parameter parse cap retains 2.82 MB (3.09 MB with %41
  escapes), and the ~0.9 MB of headers the 1 MB header cap allows
  takes it to 4.18 MB. The retained parse and the header block
  dominate, not the raw body. So 16 waiters: 16 x 4.18 MB is about
  67 MB of committed queue memory, and two slots drain a full 16-deep
  queue in about 0.6 s, far inside the deadline. Peak commitment for
  the endpoint is about 203 MB — 128 MB of Argon2id plus the 18
  requests holding a parsed form, 16 queued and the 2 being hashed, at
  about 75 MB. That 203 MB is live commitment, not resident size: the
  Go collector lets the heap reach roughly twice the live set before
  collecting, and the review measured a peak HeapAlloc of 392 MB under
  18 adversarial requests, so the README says to provision on the
  order of 400 MB.

An unknown username is verified against a dummy hash instead of
returning early, so a nonexistent account costs the same time as a
real one and the response cannot be used to enumerate usernames.

The password-change limiter is unchanged: RequireAuth runs ahead of
it, so only a request already carrying a valid session reaches its
bucket.

Two consequences are documented rather than fixed, because they follow
from the shape the issue asks for. Online guessing throughput rises
from 5 a minute to roughly 27 a second, about 2.3 million a day: the
credential check always precedes the counter, so the 429 is a label on
the response rather than a gate in front of the hash, and what bounds
brute force is the semaphore. And under a sustained flood the residual
exposure is a loss of login availability, not merely of latency --
above about 27 requests a second most attempts are shed with 503, so a
determined flood still denies login for as long as it runs. It costs
roughly 400x more to run, nothing accumulates, and the first attempt
after it stops succeeds. Restarting the service does not help: the
counters a restart clears are not what is saturated.

Also adds the missing test for the third bucketKey call site, where
the peer is a trusted proxy but the forwarded chain names no client.
Every existing test of that fallback uses an IPv4 proxy, where
bucketKey is the identity function, so dropping the /64 masking there
left the suite green.

README and the TRUSTED_PROXIES startup warning updated: a shared
bucket now costs precision, not the availability of the admin path.
This commit is contained in:
Jeffrey Paul
2026-08-17 22:17:45 +00:00
committed by clawbot
parent 992b3c68f5
commit 6fce522016
20 changed files with 2010 additions and 139 deletions
Download Patch File Download Diff File Expand all files Collapse all files

188
README.md
View File

@@ -113,7 +113,7 @@ TTY detection, and security headers are always applied.
| `RETENTION_SWEEP_INTERVAL` | How often the retention reaper and archive sweeper run (Go duration, must be positive) | `1h` |
| `SESSION_IDLE_TIMEOUT` | Idle session timeout (Go duration) | `24h` |
| `RECEIVER_RATE_LIMIT` | Receiver requests/minute per IP per entrypoint (10x that per IP across the route) | `120` |
| `TRUSTED_PROXIES` | CIDRs whose forwarded headers are trusted (unset: all clients behind a proxy share one rate-limit bucket) | `""` (none) |
| `TRUSTED_PROXIES` | CIDRs whose forwarded headers are trusted (unset: all clients behind a proxy share one rate-limit bucket; a correct login password is never throttled either way) | `""` (none) |
#### Trusted proxies
@@ -136,13 +136,15 @@ That default is safe against forged headers, but leaving it unset in
production has a cost you must know about. Production runs behind a
TLS-terminating reverse proxy, so with `TRUSTED_PROXIES` unset every
request keys on the proxy's own address and all clients share a single
bucket per limit. For the login and password-change limits that is a
denial of service anyone can perform: a steady five POSTs per minute
from any address on the internet keeps the shared login bucket full,
and the operator's own login then returns HTTP 429 for as long as the
trickle continues. There is no second administrative path and no
bypass. Restarting the service clears the in-memory buckets, but a
sustained trickle re-locks them immediately.
bucket per limit. The receiver limits become service-wide ceilings,
and the login endpoint's failure counting collapses onto one key, so a
stranger's wrong passwords throttle every other client's wrong
passwords.
What it cannot do is lock the operator out. The login endpoint
verifies credentials **before** it consults any limit and charges only
failures, so a correct password is never throttled no matter how full
the bucket is. See [Rate Limiting](#rate-limiting).
The remedy is to set `TRUSTED_PROXIES` to your reverse proxy's
address, which restores per-client buckets. webhooker logs a warning
@@ -382,11 +384,12 @@ It uses:
- **[gorilla/csrf](https://github.com/gorilla/csrf)** for CSRF
protection (cookie-based double-submit tokens)
- **[go-chi/httprate](https://github.com/go-chi/httprate)** for
sliding-window rate limiting of the login, password-change and
webhook receiver endpoints. The bucket is per client IP only when
sliding-window rate limiting of the password-change and webhook
receiver endpoints. The bucket is per client IP only when
`TRUSTED_PROXIES` names the reverse proxy; unset, every client
behind that proxy shares one bucket per limit (see
[Rate Limiting](#rate-limiting))
behind that proxy shares one bucket per limit. The login endpoint
counts failed attempts itself instead, so that a correct password is
never throttled (see [Rate Limiting](#rate-limiting))
- **[Prometheus](https://prometheus.io)** for metrics, served at
`/metrics` behind basic auth
- **[Sentry](https://sentry.io)** for optional error reporting
@@ -1091,14 +1094,117 @@ opposite directions:
and all entrypoints, where the per-entrypoint limit's capacity still
grows with the number of entrypoints. Any deployment with more than a
handful of busy entrypoints must set `TRUSTED_PROXIES`.
- For the **login and password-change** limits it costs availability of
the only administrative path, which is not safe at all. Five POSTs
per minute from any address on the internet keeps the single shared
login bucket full, and the operator's own login returns HTTP 429 for
as long as that trickle continues. A restart clears the in-memory
buckets and a resumed trickle re-locks them. Production deployments
must set `TRUSTED_PROXIES`; webhooker warns at startup whenever it is
empty, in any environment.
- For the **login and password-change** limits it costs precision, not
availability. Login failures from every client land in one counter,
so a stranger's wrong passwords make the operator's own wrong
passwords answer `429` sooner; the operator's _correct_ password is
never affected, because it is never counted. Production deployments
should still set `TRUSTED_PROXIES`; webhooker warns at startup
whenever it is empty, in any environment.
#### The login endpoint
The login `POST` is the one endpoint with no pre-emptive limiter in
front of it, and that is deliberate. A limiter that spends budget on
arrival is a lockout in this deployment shape: sharing one bucket, a
stranger sending five POSTs a minute — about 0.08 requests per second,
from anywhere — keeps it permanently full, and the operator has no
second administrative path. So the handler inverts the order:
1. **Credentials are verified first, and only a failed attempt spends
budget.** A correct password is never rate-limited, whatever the
counters hold. This is what guarantees the admin UI stays
reachable.
2. **Failures are counted per (client bucket, submitted username)**,
five per minute, after which further _failures_ from that pair are
answered `429` with a `Retry-After`. That `429` is a label on the
response, not a gate in front of the work: the credential check has
already run by the time the counter is consulted, so a throttled
client's guess is still evaluated. See the guessing rate below. A
successful login clears the counter, so mistyping a few times and
then getting it right leaves you unthrottled. Because the submitted
username is attacker-controlled, at most 1024 username counters and
1024 fallback address counters are tracked; past the first cap
failures fall back to the address counter, and past both they are
answered as throttled without being recorded. Total tracked state
is under half a megabyte and does not grow with the number of
usernames an attacker invents.
3. **Concurrent password verifications are capped at two, and the
queue for them at 16.** Verifying before counting means every login
request costs an Argon2id hash, and Argon2id here is 64 MB per
hash — two slots is a 128 MB ceiling on password hashing. Every
endpoint that hashes a password takes a slot, including the
password-change endpoint, which holds one across both the
verification and the new hash. A request that waits five seconds
without getting a slot is answered `503 Service Unavailable` and no
hash is computed for it. The wait alone does not bound memory, only
how long one request holds some, so the number of waiters is capped
as well. Size the queue from what a parked waiter actually retains,
not from the 1 MB body cap: that caps the raw body read, while the
body-cap, CSRF and form-parsing middleware all run before the
guard, so a waiter holds its parsed form plus its request header
block for the whole wait. Measured on the pinned Go 1.26.1
toolchain, as the heap delta with 64 waiters parked in the handler,
an ordinary two-field login form retains ~0 MB, a 1 MB urlencoded
body at Go's 10,000-parameter parse cap retains 2.82 MB (3.09 MB
with `%41` escapes), and the ~0.9 MB of headers the 1 MB header cap
allows takes it to **4.18 MB** — the retained parse and the headers
dominate, not the raw body. So the cap is 16 waiters: 16 x 4.18 MB
is about 67 MB of committed queue memory, and two slots drain a
full 16-deep queue in roughly 0.6 s, far inside the five-second
deadline. A request arriving past the cap is shed with `503`
immediately instead of joining the queue. **Peak commitment for the
endpoint is therefore about 203 MB**: 128 MB of Argon2id, plus the
18 requests holding a parsed form — 16 queued and the 2 being
hashed — at about 75 MB. That 203 MB is _live_ commitment, not
resident size: the Go collector lets the heap reach roughly twice
the live set before collecting, with transient parse garbage on top
of it. The independent review of this endpoint fired 18 adversarial
requests at an idle guard and measured a peak `HeapAlloc` of
392 MB. **Provision on the order of 400 MB**, not for the 203 MB
itemised here and not for the hashing budget alone.
An unknown username is verified against a dummy hash rather than
rejected early, so a nonexistent account costs the same time as a real
one and the response cannot be used to enumerate usernames.
**This raises online guessing throughput by about 300x, and that is
the trade.** Because the credential check always precedes the counter,
what bounds online brute force is the semaphore, not the failure
counter. Two slots at the cost of one Argon2id verification is on the
order of **27 guesses per second, about 2.3 million per day**, against
5 per minute under the pre-emptive limiter this replaced. Treat that
figure as a lower bound rather than a ceiling: it was measured with
Go's race detector enabled, so real hardware verifies faster and
guesses faster. Choose the admin password to survive millions of
online guesses per day — a long random passphrase, not a memorable
one. Rate-limiting `POST /pages/login` at the reverse proxy, where the
real client address is visible, is the way to put a cheaper bound back
on top.
The residual exposure is a bounded, self-clearing loss of login
**availability** — not merely of latency. A flood can keep both
verification slots busy, and a request that neither gets a slot within
five seconds nor finds room in the queue is answered `503`. Above
roughly 27 requests per second the operator is not served slowly, it
is shed: its chance per attempt is about the ratio of service rate to
flood rate, so at 400 requests per second it is roughly one attempt in
fourteen. A sufficiently determined flood still denies login for as
long as it runs.
What changed is the price and the aftermath. Denying login used to
cost an attacker 0.08 requests per second from anywhere; it now costs
30 or more sustained, about 400 times as much. Nothing accumulates
while the flood runs, nothing needs resetting when it stops, and the
operator's correct password succeeds on the first attempt afterwards.
Restarting the service is **not** a remedy: a restart clears the
failure counters, which are not what is saturated, and the flood
re-fills both verification slots on its first two requests. The
remedies are to block the source at the reverse proxy, or to
rate-limit `POST /pages/login` there — the one place a limit can be
applied without reintroducing the lockout, because the proxy sees the
real client address. Setting `TRUSTED_PROXIES` does not stop the
saturation, but it makes the source visible in the failure logs.
Finer-grained per-webhook rate limits (configured in the web UI and
enforced in the webhook handler) can layer on top of this env-level
@@ -1119,8 +1225,8 @@ abuse limit later; they are tracked as future work.
| Method | Path | Description |
| ------ | --------------- | ----------- |
| `GET` | `/pages/login` | Login page (not rate limited; the limiter applies to POST only) |
| `POST` | `/pages/login` | Login form submission (5 per minute per bucket, then 429) |
| `GET` | `/pages/login` | Login page (not rate limited) |
| `POST` | `/pages/login` | Login form submission. Credentials are verified before any limit is consulted, so a correct password is never throttled; 5 FAILED attempts per minute per bucket per submitted username, then `429`. `503` if no verification slot frees up within 5s, or immediately if 16 requests are already queued for one (see [Rate Limiting](#rate-limiting)) |
| `POST` | `/pages/logout` | Logout (destroys session) |
#### Authenticated Endpoints
@@ -1128,7 +1234,7 @@ abuse limit later; they are tracked as future work.
| Method | Path | Description |
| ------ | ------------------------ | ----------- |
| `GET` | `/user/{username}` | User profile page |
| `POST` | `/user/{username}/password` | Change the user's password (5 per minute per bucket, then 429) |
| `POST` | `/user/{username}/password` | Change the user's password (5 per minute per bucket, then `429`; `503` if no verification slot frees up within 5s, or immediately if 16 requests are already queued for one) |
| `GET` | `/sources` | List user's webhooks |
| `GET` | `/sources/new` | Create webhook form |
| `POST` | `/sources/new` | Create webhook submission |
@@ -1230,6 +1336,7 @@ webhooker/
│ │ ├── middleware.go # Logging, CORS, Auth, Metrics, MetricsAuth, SecurityHeaders, MaxBodySize
│ │ ├── csrf.go # CSRF protection middleware (gorilla/csrf)
│ │ ├── ratelimit.go # Per-IP rate limiting middleware (go-chi/httprate)
│ │ ├── loginguard.go # Login failure counters and the Argon2id verification semaphore
│ │ └── testing.go # NewForTest: Middleware without the fx lifecycle
│ ├── server/
│ │ ├── server.go # Server struct, fx lifecycle, signal handling
@@ -1327,9 +1434,11 @@ one that lies about its length, is hard-capped by
Those same four route groups then apply **CSRF** and **NoCache**
(`Cache-Control: no-store`, `Pragma: no-cache`), and every group except
`/pages` applies **RequireAuth**. The rate limiters are per-route
rather than global: **LoginRateLimit** on `/pages/login`,
**PasswordChangeRateLimit** on `/user/{username}/password`, and
**ReceiverRateLimit** on `/webhook/{uuid}`.
rather than global: **PasswordChangeRateLimit** on
`/user/{username}/password` and **ReceiverRateLimit** on
`/webhook/{uuid}`. There is deliberately none on `/pages/login` — that
endpoint counts failures inside the handler, after the credential
check, see [The login endpoint](#the-login-endpoint).
### Authentication
@@ -1367,14 +1476,27 @@ rather than global: **LoginRateLimit** on `/pages/login`,
both at target creation time (URL validation) and at delivery time
(custom HTTP transport with SSRF-safe dialer that validates resolved
IPs before connecting, preventing DNS rebinding attacks)
- **Login rate limiting** via [go-chi/httprate](https://github.com/go-chi/httprate):
sliding-window rate limiter on the login endpoint, 5 POST attempts
per minute per bucket, to slow brute-force attacks. GET requests to
the login page are not limited. The password-change endpoint carries
the same 5-per-minute limit. The bucket is per client IP only when
- **Login limiting is inverted, deliberately.** The login `POST` has
no pre-emptive rate limiter in front of it. Credentials are
verified first and only a _failed_ attempt spends budget, so a
correct password is never throttled and no flood of wrong ones can
deny the operator the only administrative path. Failures are
counted per (bucket, submitted username), five per minute, after
which further failures are answered `429` with a `Retry-After`.
What bounds brute force is not that counter but the cap of two
concurrent Argon2id verifications: a throttled client's guess is
still evaluated, so roughly 27 guesses a second get through and the
admin password has to carry that load (see
[The login endpoint](#the-login-endpoint)). `GET` requests to the
login page are not limited
- **Password-change rate limiting** via [go-chi/httprate](https://github.com/go-chi/httprate):
sliding-window rate limiter, 5 POST attempts per minute per bucket.
It runs behind session auth, so only a client already holding a
valid session reaches it, and an operator throttled out of changing
a password can still log in. The bucket is per client IP only when
`TRUSTED_PROXIES` names the reverse proxy; unset, every client
shares one bucket and the login becomes remotely deniable (see
[Rate Limiting](#rate-limiting)). webhooker warns at startup
shares one bucket, which costs precision rather than availability
(see [Rate Limiting](#rate-limiting)). webhooker warns at startup
whenever `TRUSTED_PROXIES` is empty
- Prometheus metrics behind basic auth
- Static assets embedded in binary (no filesystem access needed at