With TRUSTED_PROXIES empty behind the reverse proxy production is
required to run behind, every login POST keyed on the proxy's address
and shared one 5/minute bucket. A stranger sending five POSTs a
minute -- 0.08 requests per second, from anywhere -- kept that bucket
permanently full, and the operator's own correct password was answered
429 indefinitely with no second administrative path.
The login POST no longer has a pre-emptive limiter. The handler
verifies credentials first and spends budget only on a FAILED attempt,
so a correct password is never throttled whatever the counters hold.
Three things follow, and are implemented together because the first is
unsafe without the other two:
- Failures are counted per (client bucket, submitted username), five
per minute, after which further failures get 429 with a Retry-After.
A successful login clears the counter, so mistyping and then
succeeding does not leave the operator throttled.
- Both key sets are capped at 1024 entries. The submitted username is
attacker-controlled, so past the first cap failures fall back to a
counter keyed on the client alone, and past both caps a failure is
answered as throttled without being recorded. Tracked state stays
under half a megabyte and does not grow with invented usernames.
- Concurrent Argon2id verifications are capped at two, a 128 MB
ceiling at 64 MB per hash, and the queue for those slots is capped
at 16 waiters. Every password-hashing endpoint takes a slot,
including the password-change endpoint, which holds one across both
its hashes. A request that waits five seconds without a slot is
answered 503, and one that arrives with the queue already full is
shed with 503 immediately rather than joining it. Bounding the wait
alone would not bound memory, and the queue depth is sized from what
a parked waiter measurably retains rather than from the 1 MB body
cap, which bounds only the raw body read. The body-cap, CSRF and
form-parsing middleware all run before the guard, so a waiter holds
its parsed form plus its request header block for the whole wait.
Measured on the pinned go1.26.1 toolchain as the HeapAlloc delta
across two GCs with 64 waiters parked in the handler: an ordinary
two-field login form retains ~0 MB, a 1 MB urlencoded body at Go's
10,000-parameter parse cap retains 2.82 MB (3.09 MB with %41
escapes), and the ~0.9 MB of headers the 1 MB header cap allows
takes it to 4.18 MB. The retained parse and the header block
dominate, not the raw body. So 16 waiters: 16 x 4.18 MB is about
67 MB of committed queue memory, and two slots drain a full 16-deep
queue in about 0.6 s, far inside the deadline. Peak commitment for
the endpoint is about 203 MB — 128 MB of Argon2id plus the 18
requests holding a parsed form, 16 queued and the 2 being hashed, at
about 75 MB. That 203 MB is live commitment, not resident size: the
Go collector lets the heap reach roughly twice the live set before
collecting, and the review measured a peak HeapAlloc of 392 MB under
18 adversarial requests, so the README says to provision on the
order of 400 MB.
An unknown username is verified against a dummy hash instead of
returning early, so a nonexistent account costs the same time as a
real one and the response cannot be used to enumerate usernames.
The password-change limiter is unchanged: RequireAuth runs ahead of
it, so only a request already carrying a valid session reaches its
bucket.
Two consequences are documented rather than fixed, because they follow
from the shape the issue asks for. Online guessing throughput rises
from 5 a minute to roughly 27 a second, about 2.3 million a day: the
credential check always precedes the counter, so the 429 is a label on
the response rather than a gate in front of the hash, and what bounds
brute force is the semaphore. And under a sustained flood the residual
exposure is a loss of login availability, not merely of latency --
above about 27 requests a second most attempts are shed with 503, so a
determined flood still denies login for as long as it runs. It costs
roughly 400x more to run, nothing accumulates, and the first attempt
after it stops succeeds. Restarting the service does not help: the
counters a restart clears are not what is saturated.
Also adds the missing test for the third bucketKey call site, where
the peer is a trusted proxy but the forwarded chain names no client.
Every existing test of that fallback uses an IPv4 proxy, where
bucketKey is the identity function, so dropping the /64 masking there
left the suite green.
README and the TRUSTED_PROXIES startup warning updated: a shared
bucket now costs precision, not the availability of the admin path.
The 8 KB render cap from #135 left storage untouched but no route served
the rest, so a body over the cap was reachable only with filesystem
access to the SQLite files — in a product whose purpose is storing
webhooks so they can be inspected.
GET /source/{sourceID}/logs/{eventID}/body serves the whole body to the
webhook's owner, as application/octet-stream with an attachment
disposition and nosniff. Those are a security control, not formatting:
the bytes come from the public receiver and are handed back inside the
operator's authenticated origin, and the existing CSP would not stop a
stored HTML payload executing there. The truncation marker links to it
only when a body was actually cut.
Accepted deviation, documented rather than glossed: #157's definition of
done asks the route to stream from the row. It buffers whole instead,
because database/sql exposes no incremental handle on a SQLite BLOB and
substr range reads re-materialise the entire column per call — an
earlier revision chunked at 64 KiB and was 11-15x slower for a worse
bound. Three independent reviewers confirmed no streaming path exists.
Independently reviewed three times. Two earlier revisions each asserted
a memory bound the code did not have; the final reviewer measured
2.057x at the ingest cap and pinned the two overlapping allocations from
source — the driver's column buffer and database/sql's convertAssign
clone — confirming the stated "roughly two bodies, and 2x is a floor
not a ceiling" is now accurate, since SQLite's own materialisation sits
outside the Go heap.
Publishing this README would have shipped false statements about the
product. Corrects the eight items on the issue plus everything a full
sweep turned up: the Slack circuit-breaker scope, a nonexistent WAL, the
wrong config key for slack targets, six undocumented routes, the
conditional /metrics registration, wrong retention bands, wrong shutdown
mechanism, and a Quick Start that led a new contributor into a red
build.
The lockout warning now fires whenever TRUSTED_PROXIES is empty rather
than only in production, since the variable it was gated on defaults to
dev. Rate-limit keying, the limits and the TRUSTED_PROXIES default are
untouched — those belong to #150.
What /s/* actually serves was settled empirically rather than by
reading: all five of GET/HEAD/POST/PUT/DELETE return 200, pinned by
TestStaticServesEveryMethod. Restricting it is filed separately.
Independently reviewed after three prior rounds. The reviewer
re-derived all fifteen claim-table rows against the code, including
every row a previous revision had marked "correct, left alone" and got
wrong, and found zero false; then verified every route method-by-method,
all twelve environment variables, all nine entity tables, and the
package tree against git ls-files. The Quick Start was confirmed by
running it in a fresh clone.
check / check (push) Superseded by a newer commit; never tested
CSRF ran before MaxBodySize, so the CSRF middleware parsed the form body
before any cap applied and an oversized request was read in full before
being rejected. MaxBodySize is now the first middleware in all four route
groups that parse forms, ahead of CSRF and RequireAuth.
An oversize request therefore gets 413 without the handler running and
without state changing, including the password-change route.
Note the ordering trade: an unauthenticated client now receives 413 rather
than an auth redirect on /user/{username}/password.
