Trust the RFC 1918 ranges as proxies when TRUSTED_PROXIES is unset (closes #333)
check / check (push) Successful in 4m43s
check / check (push) Successful in 4m43s
Unset or empty, TRUSTED_PROXIES now defaults to 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16, so a reverse proxy reaching webhooker from one of those ranges gets per-client rate-limit buckets with nothing set. A set value replaces the default entirely; an unparseable one still fails startup. The startup warning for an empty list is gone. The README gives the default, one rule (if any client can reach webhooker, or the proxy in front of it, from an RFC 1918 source address, set the list to the proxy's address alone), and where to find that address: the remoteIP field of the http request log line. Loopback is not in the default. Model: opus-5-5
This commit was merged in pull request #336.
This commit is contained in:
@@ -142,7 +142,7 @@ TTY detection, and security headers are always applied.
|
||||
| `RETENTION_SWEEP_INTERVAL` | How often the retention reaper and archive sweeper run (Go duration, must be positive) | `1h` |
|
||||
| `SESSION_IDLE_TIMEOUT` | Idle session timeout (Go duration) | `24h` |
|
||||
| `RECEIVER_RATE_LIMIT` | Receiver requests/minute per IP per entrypoint (10x that per IP across the route) | `120` |
|
||||
| `TRUSTED_PROXIES` | CIDRs whose forwarded headers are trusted (unset: all clients behind a proxy share one rate-limit bucket; a correct login password is never throttled either way) | `""` (none) |
|
||||
| `TRUSTED_PROXIES` | CIDRs whose forwarded headers are trusted. A set value replaces the default. If any client can reach webhooker, or the proxy in front of it, from an RFC 1918 source address, set it to the proxy's address alone. See [Trusted proxies](#trusted-proxies) | `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16` (RFC 1918) |
|
||||
| `ALLOWED_EGRESS_CIDRS` | CIDRs that delivery targets may reach despite the SSRF blocklist. Read [Allowing egress to your own network](#allowing-egress-to-your-own-network) before setting it | `""` (none) |
|
||||
|
||||
#### Allowing egress to your own network
|
||||
@@ -389,41 +389,37 @@ unlocked.
|
||||
`TRUSTED_PROXIES` is a comma-separated list of CIDR blocks (a bare
|
||||
address such as `192.168.1.7` is accepted and treated as a single
|
||||
host), for example `192.168.1.7, 2001:db8::5`. It decides whose
|
||||
`X-Forwarded-For` header the rate limiters believe, so it should name
|
||||
the addresses of your reverse proxies and nothing else.
|
||||
`X-Forwarded-For` header the rate limiters believe, so it should cover
|
||||
the addresses of your reverse proxies.
|
||||
|
||||
`X-Forwarded-For` is honoured **only** when the connecting peer is
|
||||
inside one of these blocks; for every other peer the client identity is
|
||||
the connection's own address and the header is ignored. The default is
|
||||
the empty list, which trusts nobody — anything else would let any
|
||||
client pick its own rate limit bucket, minting a fresh one per request
|
||||
or draining someone else's. Set it to the address of your reverse
|
||||
proxy, and to nothing wider. A set but unparseable value aborts
|
||||
startup.
|
||||
the connection's own address and the header is ignored. Unset (or
|
||||
empty), the list is the RFC 1918 private ranges: `10.0.0.0/8`,
|
||||
`172.16.0.0/12` and `192.168.0.0/16`. A set value replaces the default
|
||||
entirely. A set but unparseable value aborts startup.
|
||||
|
||||
That default is safe against forged headers, but leaving it unset in
|
||||
production has a cost you must know about. Production runs behind a
|
||||
TLS-terminating reverse proxy, so with `TRUSTED_PROXIES` unset every
|
||||
request keys on the proxy's own address and all clients share a single
|
||||
bucket per limit. The receiver limits become service-wide ceilings,
|
||||
and the login endpoint's failure counting collapses onto one key, so a
|
||||
stranger's wrong passwords throttle every other client's wrong
|
||||
passwords.
|
||||
If any client can reach webhooker, or the proxy in front of it, from an
|
||||
RFC 1918 source address (directly, or through anything that can
|
||||
rewrite source addresses, such as NAT or a published container port),
|
||||
set `TRUSTED_PROXIES` to the proxy's address alone, or every rate
|
||||
limit, the webhook receiver's included, can be bypassed by those
|
||||
clients. The address to set is the `remoteIP` field of the
|
||||
`http request` log line for a request that came through the proxy.
|
||||
|
||||
Behind a proxy the list does not cover, every request keys on the
|
||||
proxy's own address and all clients share a single bucket per limit.
|
||||
The receiver limits become service-wide ceilings, and the login
|
||||
endpoint's failure counting collapses onto one key, so a stranger's
|
||||
wrong passwords throttle every other client's wrong passwords. Set
|
||||
`TRUSTED_PROXIES` to that proxy's address to restore per-client
|
||||
buckets.
|
||||
|
||||
What it cannot do is lock the operator out. The login endpoint
|
||||
verifies credentials **before** it consults any limit and charges only
|
||||
failures, so a correct password is never throttled no matter how full
|
||||
the bucket is. See [Rate Limiting](#rate-limiting).
|
||||
|
||||
The remedy is to set `TRUSTED_PROXIES` to your reverse proxy's
|
||||
address, which restores per-client buckets. webhooker logs a warning
|
||||
at startup whenever `TRUSTED_PROXIES` is empty, in every environment,
|
||||
because behind a proxy every client shares one bucket in `dev` and
|
||||
`prod` alike. The warning is informational when nothing proxies to the
|
||||
process: with no proxy in front, the peer address is the client's own
|
||||
and the buckets are already per-client. See
|
||||
[Rate Limiting](#rate-limiting) for what each limit shares.
|
||||
|
||||
`X-Real-IP` and `True-Client-IP` are **never** read, from any peer.
|
||||
Reverse proxies append to `X-Forwarded-For` but forward other client
|
||||
headers verbatim, so a single-valued header is client-controlled even
|
||||
@@ -439,20 +435,10 @@ instead, since past such an entry the chain is not the shape assumed
|
||||
here. The peer address is likewise used when the header is absent or
|
||||
every hop in it is a trusted proxy.
|
||||
|
||||
Two operator requirements follow:
|
||||
|
||||
- Your proxy must **append** the peer address to `X-Forwarded-For`
|
||||
(nginx `$proxy_add_x_forwarded_for`, HAProxy `option forwardfor`,
|
||||
Caddy and AWS ALB by default), and must append a bare address with
|
||||
no port.
|
||||
- List proxy hosts **only**. Any address inside `TRUSTED_PROXIES`
|
||||
chooses its own rate-limit key: its `X-Forwarded-For` is walked, so
|
||||
it can name a different address on every request to get a fresh
|
||||
bucket each time, or name another client's address to drain that
|
||||
client's bucket. Never list a block that also covers clients — a
|
||||
broad `10.0.0.0/8` on a network where clients live in the same range
|
||||
makes all three limits, including the unauthenticated webhook
|
||||
receiver, silently bypassable by every client in the block.
|
||||
Your proxy must therefore **append** the peer address to
|
||||
`X-Forwarded-For` (nginx `$proxy_add_x_forwarded_for`, HAProxy
|
||||
`option forwardfor`, Caddy and AWS ALB by default), and must append a
|
||||
bare address with no port.
|
||||
|
||||
#### Sessions
|
||||
|
||||
@@ -751,10 +737,15 @@ repository's `Dockerfile` and runs it. The app needs:
|
||||
- **Volume:** one host directory mounted at `/var/lib/webhooker`.
|
||||
- **Environment variables:**
|
||||
- `WEBHOOKER_ENVIRONMENT=prod`
|
||||
- `TRUSTED_PROXIES`: your reverse proxy's address on that Docker
|
||||
network. The `remoteIP` field of the `http request` log line for a
|
||||
request that came through the proxy shows it; the health check's
|
||||
own lines show `::1`. See [Trusted proxies](#trusted-proxies).
|
||||
- `TRUSTED_PROXIES`: unset, it is the RFC 1918 ranges. Set it to
|
||||
your reverse proxy's address alone if that address is outside
|
||||
those ranges, or if any client can reach webhooker, or the proxy,
|
||||
from an RFC 1918 source address (directly, or through anything
|
||||
that can rewrite source addresses, such as NAT or a published
|
||||
container port). The `remoteIP` field of the `http request` log
|
||||
line for a request that came through the proxy shows that
|
||||
address; the health check's own lines show `::1`. See
|
||||
[Trusted proxies](#trusted-proxies).
|
||||
- Leave `BIND_ADDRESS` and `DATA_DIR` unset: the image sets
|
||||
`BIND_ADDRESS` to `0.0.0.0`, and `DATA_DIR` defaults to
|
||||
`/var/lib/webhooker`.
|
||||
@@ -817,12 +808,16 @@ reports.
|
||||
behind a proxy means the `X-Forwarded-Proto` header. The block below
|
||||
sets it; without it every request is read as plaintext and cookies
|
||||
ship without `Secure`. See [Configuration](#configuration).
|
||||
3. **Set `TRUSTED_PROXIES` to the proxy's address.** Unset, every rate
|
||||
limiter keys on the connecting peer, which behind a proxy is the
|
||||
proxy on every request: all clients collapse into one global bucket
|
||||
per limit and the receiver's per-IP limits become service-wide
|
||||
ceilings. See [Trusted proxies](#trusted-proxies). List the proxy
|
||||
and nothing else.
|
||||
3. **Make sure `TRUSTED_PROXIES` covers the proxy's address.** For a
|
||||
proxy it does not cover, every rate limiter keys on the proxy, so
|
||||
all clients share one bucket per limit. Unset, the list is the RFC
|
||||
1918 ranges, which do not cover a proxy that reaches the binary
|
||||
itself over loopback (the binary bound to `127.0.0.1`). With the
|
||||
image, the address to check is the `remoteIP` field of the
|
||||
`http request` log line for a request that came through the proxy.
|
||||
If any client can reach webhooker, or the proxy, from an RFC 1918
|
||||
source address, set the list to the proxy's address alone. See
|
||||
[Trusted proxies](#trusted-proxies).
|
||||
4. **Send `Host` as `$http_host`, not `$host`.** `$host` strips the
|
||||
port. webhooker's Origin/Referer check compares against the host it
|
||||
was given, so on any port other than 443 `$host` makes every form
|
||||
@@ -1375,10 +1370,11 @@ It uses:
|
||||
- **[go-chi/httprate](https://github.com/go-chi/httprate)** for
|
||||
sliding-window rate limiting of the password-change and webhook
|
||||
receiver endpoints. The bucket is per client IP only when
|
||||
`TRUSTED_PROXIES` names the reverse proxy; unset, every client
|
||||
behind that proxy shares one bucket per limit. The login endpoint
|
||||
counts failed attempts itself instead, so that a correct password is
|
||||
never throttled (see [Rate Limiting](#rate-limiting))
|
||||
`TRUSTED_PROXIES` covers the reverse proxy (by default it covers the
|
||||
RFC 1918 private ranges); otherwise every client behind that proxy
|
||||
shares one bucket per limit. The login endpoint counts failed
|
||||
attempts itself instead, so that a correct password is never
|
||||
throttled (see [Rate Limiting](#rate-limiting))
|
||||
- **[Prometheus](https://prometheus.io)** for metrics, served at
|
||||
`/metrics` behind basic auth
|
||||
- **[Sentry](https://sentry.io)** for optional error reporting
|
||||
@@ -2549,47 +2545,44 @@ the tree is checked out: four checkouts have reported 3,959, 3,961,
|
||||
client-supplied field was cut, and that the shipped chain's stack
|
||||
arrived uncut — never the numbers.
|
||||
|
||||
Every limiter here — receiver, login, and password change — identifies
|
||||
the client the same way, through one shared key function: the
|
||||
connection's own address, unless the peer is listed in
|
||||
`TRUSTED_PROXIES`, in which case the forwarded client address is used
|
||||
instead. That address becomes a bucket by family: IPv4 keys on the full
|
||||
address, IPv6 on its `/64` prefix. A routed `/64` is the normal
|
||||
Every limiter here — receiver, login, password change, delivery replay
|
||||
and event resubmit — identifies the client the same way, through one
|
||||
shared key function: the connection's own address, unless the peer is
|
||||
inside `TRUSTED_PROXIES`, in which case the forwarded client address is
|
||||
used instead. That address becomes a bucket by family: IPv4 keys on
|
||||
the full address, IPv6 on its `/64` prefix. A routed `/64` is the normal
|
||||
residential and mobile IPv6 allocation, so keying IPv6 per address would
|
||||
let one subscriber rotate source addresses and mint a fresh bucket per
|
||||
request, evading these limits at the network layer without spoofing
|
||||
anything; the cost is that distinct clients inside one `/64` share a
|
||||
bucket. IPv4-mapped addresses (`::ffff:1.2.3.4`) key as the IPv4 address
|
||||
they carry. See [Trusted proxies](#trusted-proxies). Deployed without that
|
||||
variable set, a client behind a reverse proxy shares one bucket with
|
||||
every other client behind the same proxy. Set `TRUSTED_PROXIES` to the
|
||||
proxy's address to get per-client limits back. What the shared bucket
|
||||
they carry. See [Trusted proxies](#trusted-proxies). When that variable
|
||||
does not cover the reverse proxy, a client behind it shares one bucket
|
||||
with every other client behind the same proxy. Set `TRUSTED_PROXIES` to
|
||||
the proxy's address to get per-client limits back. What the shared bucket
|
||||
costs is not the same for every limiter, and the two cases pull in
|
||||
opposite directions:
|
||||
|
||||
- For the **receiver** limits it costs throughput, which is the safe
|
||||
direction to be wrong in: sharing can only make a limit bind sooner,
|
||||
never let a sender past it. It matters more for the aggregate limit
|
||||
than for the per-entrypoint one: with `TRUSTED_PROXIES` unset behind
|
||||
the reverse proxy a production deployment is required to run behind,
|
||||
every request keys on the proxy, so the aggregate limit becomes a
|
||||
service-wide ceiling of 1200 requests per minute across all senders
|
||||
and all entrypoints, where the per-entrypoint limit's capacity still
|
||||
grows with the number of entrypoints. Any deployment with more than a
|
||||
handful of busy entrypoints must set `TRUSTED_PROXIES`.
|
||||
than for the per-entrypoint one: with every request keyed on the
|
||||
proxy, the aggregate limit becomes a service-wide ceiling of 1200
|
||||
requests per minute across all senders and all entrypoints, where the
|
||||
per-entrypoint limit's capacity still grows with the number of
|
||||
entrypoints.
|
||||
- For the **login and password-change** limits it costs precision, not
|
||||
availability. Login failures from every client land in one counter,
|
||||
so a stranger's wrong passwords make the operator's own wrong
|
||||
passwords answer `429` sooner; the operator's _correct_ password is
|
||||
never affected, because it is never counted. Production deployments
|
||||
should still set `TRUSTED_PROXIES`; webhooker warns at startup
|
||||
whenever it is empty, in any environment.
|
||||
never affected, because it is never counted.
|
||||
|
||||
#### The login endpoint
|
||||
|
||||
The login `POST` is the one endpoint with no pre-emptive limiter in
|
||||
front of it, and that is deliberate. A limiter that spends budget on
|
||||
arrival is a lockout in this deployment shape: sharing one bucket, a
|
||||
arrival is a lockout wherever clients share one bucket, as they do
|
||||
behind a reverse proxy that `TRUSTED_PROXIES` does not cover: a
|
||||
stranger sending five POSTs a minute — about 0.08 requests per second,
|
||||
from anywhere — keeps it permanently full, and the operator has no
|
||||
second administrative path. So the handler inverts the order:
|
||||
@@ -2686,8 +2679,10 @@ re-fills both verification slots on its first two requests. The
|
||||
remedies are to block the source at the reverse proxy, or to
|
||||
rate-limit `POST /pages/login` there — the one place a limit can be
|
||||
applied without reintroducing the lockout, because the proxy sees the
|
||||
real client address. Setting `TRUSTED_PROXIES` does not stop the
|
||||
saturation, but it makes the source visible in the failure logs.
|
||||
real client address. `TRUSTED_PROXIES` does not stop the saturation.
|
||||
The flood's source is in the proxy's access log: webhooker's own logs
|
||||
record the proxy's address, not the client's (see
|
||||
[Deployment behind a reverse proxy](#deployment-behind-a-reverse-proxy)).
|
||||
|
||||
Finer-grained per-webhook rate limits (configured in the web UI and
|
||||
enforced in the webhook handler) can layer on top of this env-level
|
||||
@@ -3045,10 +3040,9 @@ check, see [The login endpoint](#the-login-endpoint).
|
||||
It runs behind session auth, so only a client already holding a
|
||||
valid session reaches it, and an operator throttled out of changing
|
||||
a password can still log in. The bucket is per client IP only when
|
||||
`TRUSTED_PROXIES` names the reverse proxy; unset, every client
|
||||
`TRUSTED_PROXIES` covers the reverse proxy; otherwise every client
|
||||
shares one bucket, which costs precision rather than availability
|
||||
(see [Rate Limiting](#rate-limiting)). webhooker warns at startup
|
||||
whenever `TRUSTED_PROXIES` is empty
|
||||
(see [Rate Limiting](#rate-limiting))
|
||||
- Prometheus metrics behind basic auth
|
||||
- Static assets embedded in binary (no filesystem access needed at
|
||||
runtime)
|
||||
|
||||
Reference in New Issue
Block a user