Compare commits

2 Commits
Author SHA1 Message Date
clawbot 39e847f09e Ban the netblock of a client that breaks a rate limit, in memory (closes #18)
check / check (push) Successful in 3m27s
A request over a rate limit is refused with SWWAF_BAN_RESPONSE and bans
the client's netblock: an hour at first, three times the last ban when
broken again within a day of its end, permanent past seven days. The
ban ledger in internal/bans is checked after the static lists and
before the lookup, and the requests it refuses are not counted. A ban
resets the client's counters and carries notes holding the request
that broke the limit, as SPEC.md now says. At most SWWAF_MAX_BANS are
held. SWWAF_BAN_RESPONSE also answers SWWAF_DENY_NETS and the country
lists.

Judgement call: the six ban settings cannot be off.
Judgement call: a permanent ban's ban_expires is "permanent".

Model: opus-5-5
2026-10-06 02:30:02 +00:00
clawbot 50df9ee36e Re-vendor the canonical files from sneak/prompts at dd4027b (closes #65)
check / check (push) Successful in 4m1s
The vendored files are fetched from sneak/prompts commit dd4027b. This
repository's own entries come after the canonical content, at the end of
each file: /bin in .dockerignore, the Go lines of .gitignore and [*.go]
in .editorconfig; the test-support deny list has no entries of its own.
The lint phase moves to golangci-lint v2.14.0. The build stage now takes
the version from git describe on the .git the build context carries,
unless VERSION is passed, and fails when .git is present but no version
comes out. The test phase drops -count=1, which the policy says it does
not need, and keeps its tmpfs build cache. One test calls Header.Get
with X-Real-IP, as canonicalheader asks.

Model: opus-5-5
2026-10-06 04:13:35 +02:00
28 changed files with 1693 additions and 195 deletions
+20 -6
View File
@@ -13,9 +13,21 @@
# `/myapp`, never `**/myapp`, which also matches `cmd/myapp/` and # `/myapp`, never `**/myapp`, which also matches `cmd/myapp/` and
# deletes the package directory from the context. # deletes the package directory from the context.
# Excluding .git means `git describe` cannot run in any build stage and # .git is sent without its config. Without a VERSION build argument the
# fails quietly there; pass the version in with --build-arg VERSION. # stage that compiles runs `git describe --tags --always` on .git, which
.git # does not need .git/config; that file can hold a credential, such as a
# password in a remote URL or the token the CI checkout step stores there.
# Each submodule keeps a config with the same exposure in its git directory
# under .git/modules/, nested again for a submodule's own submodules, or in
# its own .git directory when it keeps one.
# KNOWN GAP: a submodule whose name has a `config` segment (`config`,
# `deploy/config`, `config/lib`) loses its whole git directory, because
# `**/.git/modules/**/config` also matches that segment's directory
# under .git/modules/. Go's version stamping then fails the build;
# nothing leaks. Name such a submodule without that segment:
# `git submodule add --name`.
**/.git/config
**/.git/modules/**/config
# Agent scratch: one full checkout of the repo per in-flight agent. # Agent scratch: one full checkout of the repo per in-flight agent.
# Anchored because it occurs once where agents run at the repo root. # Anchored because it occurs once where agents run at the repo root.
@@ -39,14 +51,13 @@
**/[iI][dD]_[rR][sS][aA] **/[iI][dD]_[rR][sS][aA]
**/[iI][dD]_[dD][sS][aA] **/[iI][dD]_[dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA] **/[iI][dD]_[eE][cC][dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
**/[iI][dD]_[eE][dD]25519 **/[iI][dD]_[eE][dD]25519
**/[iI][dD]_[eE][dD]25519_[sS][kK]
# Dependencies: restored inside the image, never copied in. # Dependencies: restored inside the image, never copied in.
**/node_modules **/node_modules
# The binary `make build` writes on the host; the image builds its own.
/bin
# OS metadata. # OS metadata.
**/.DS_Store **/.DS_Store
**/Thumbs.db **/Thumbs.db
@@ -59,3 +70,6 @@
**/.idea **/.idea
**/.vscode **/.vscode
**/*.sublime-* **/*.sublime-*
# The binary `make build` writes on the host; the image builds its own.
/bin
+25 -5
View File
@@ -20,11 +20,31 @@ Thumbs.db
# Node # Node
node_modules/ node_modules/
# Environment / secrets # Secrets. Unanchored like every entry above, so each matches at every
.env # depth. Matching is case-sensitive on Linux, so names use character
.env.* # ranges rather than a lowercase form that misses `Server.Key`.
*.pem
*.key # Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Only the templates `example.env` and `sample.env` are
# re-included below. A repository that commits any other template adds
# its own negation after these lines, for example `!.env.example`.
*.[eE][nN][vV]
.[eE][nN][vV].*
.[eE][nN][vV][rR][cC]
!example.env
!sample.env
# Private keys and the bundles carrying them.
*.[pP][eE][mM]
*.[kK][eE][yY]
*.[pP]12
*.[pP][fF][xX]
[iI][dD]_[rR][sS][aA]
[iI][dD]_[dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
[iI][dD]_[eE][dD]25519
[iI][dD]_[eE][dD]25519_[sS][kK]
# Go: the binary `make build` writes, test binaries, profiles and logs # Go: the binary `make build` writes, test binaries, profiles and logs
/bin/ /bin/
+1
View File
@@ -17,6 +17,7 @@ linters:
disable: disable:
# Genuinely incompatible with project patterns # Genuinely incompatible with project patterns
- exhaustruct # Requires all struct fields - exhaustruct # Requires all struct fields
- exhaustruct_v5 # Requires all struct fields (successor to exhaustruct)
- godot # Requires comments to end with periods - godot # Requires comments to end with periods
- wrapcheck # Too verbose for internal packages - wrapcheck # Too verbose for internal packages
- varnamelen # Short names like db, id are idiomatic Go - varnamelen # Short names like db, id are idiomatic Go
+21 -8
View File
@@ -2,8 +2,8 @@
# lint` or `script/lint`, which are themselves a docker build and would # lint` or `script/lint`, which are themselves a docker build and would
# recurse into a daemon that does not exist in a build step. # recurse into a daemon that does not exist in a build step.
# #
# golangci/golangci-lint v2.12.2 (built with go1.26.2), 2026-05-06 # golangci/golangci-lint v2.14.0 (built with go1.27.0), 2026-09-24
FROM golangci/golangci-lint@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240 AS lint FROM golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS lint
WORKDIR /src WORKDIR /src
@@ -32,9 +32,9 @@ COPY . .
# Go's build cache is kept on a tmpfs, out of the image: nothing uses it # Go's build cache is kept on a tmpfs, out of the image: nothing uses it
# after this step, and writing it into the image takes seconds. # after this step, and writing it into the image takes seconds.
RUN --mount=type=tmpfs,target=/root/.cache/go-build \ RUN --mount=type=tmpfs,target=/root/.cache/go-build \
go test -count=1 -timeout 90s -race -cover ./... || \ go test -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \ { echo "--- Rerunning with -v for details ---"; \
go test -count=1 -timeout 90s -race -v ./...; exit 1; } go test -timeout 90s -race -v ./...; exit 1; }
# Build stage. Nothing is wanted from the two phases above; the copies # Build stage. Nothing is wanted from the two phases above; the copies
# are what make BuildKit build them first, so the image, which needs this # are what make BuildKit build them first, so the image, which needs this
@@ -46,6 +46,10 @@ FROM golang@sha256:3b77fc618ec235a1ab412de7737f120dd507c57e8d87de4cbb7994fb94275
COPY --from=lint /src/go.sum /dev/null COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null COPY --from=test /src/go.sum /dev/null
# This image has git. A tar-stream context keeps the sender's file
# owners, which git refuses.
RUN git config --system --add safe.directory /src
WORKDIR /src WORKDIR /src
COPY go.mod go.sum ./ COPY go.mod go.sum ./
@@ -53,10 +57,19 @@ RUN go mod download
COPY . . COPY . .
# The version is computed on the host and passed in, because # The VERSION build arg when one is given, otherwise
# .dockerignore excludes .git. # `git describe --tags --always` on the .git in the build context. With
ARG VERSION=dev # .git present, a version that is still empty, dev or unknown fails the
RUN CGO_ENABLED=0 go build -trimpath \ # build: git is missing or could not read the checkout.
ARG VERSION
RUN VERSION="${VERSION:-$(git describe --tags --always)}"; \
if [ -e .git ]; then \
case "$VERSION" in ""|dev|unknown) \
echo "version is '$VERSION' although .git is present" >&2; \
exit 1 ;; \
esac; \
fi; \
CGO_ENABLED=0 go build -trimpath \
-ldflags="-s -w -X main.Version=${VERSION}" \ -ldflags="-s -w -X main.Version=${VERSION}" \
-o /usr/local/bin/smallwebwaf ./cmd/smallwebwaf -o /usr/local/bin/smallwebwaf ./cmd/smallwebwaf
+108 -64
View File
@@ -13,15 +13,16 @@ JSON log line for every request.
Status: the first two milestones are built Status: the first two milestones are built
(https://git.eeqj.de/sneak/smallwebwaf/issues/13 and (https://git.eeqj.de/sneak/smallwebwaf/issues/13 and
https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are the static lists, https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are the static lists
which come next in the build order. `smallwebwaf` passes each request to the app and the bans that broken rate limits lead to, which come next in the build
and the app's answer back, unchanged, within its timeouts and size limits, works order. `smallwebwaf` passes each request to the app and the app's answer back,
out each client's address, refuses a client that sends too many requests, comes unchanged, within its timeouts and size limits, works out each client's address,
from a country you refuse or from a network you refuse, lets the networks you bans a client that sends too many requests, refuses a client that comes from a
choose through, and writes a JSON log line for every request. It comes as the country you refuse or from a network you refuse, lets the networks you choose
image the app's own image is built on. The rest of the design comes after that, through, and writes a JSON log line for every request. It comes as the image the
in the order of the build order in [`SPEC.md`](SPEC.md). The survey of existing app's own image is built on. The rest of the design comes after that, in the
tools that led to the design is in [`EVALUATION.md`](EVALUATION.md). order of the build order in [`SPEC.md`](SPEC.md). The survey of existing tools
that led to the design is in [`EVALUATION.md`](EVALUATION.md).
## Getting started ## Getting started
@@ -69,31 +70,48 @@ and `make run` builds and runs it, listening on port 8080 in front of an app at
waiting for the app to take what it had. Once the response has started, a waiting for the app to take what it had. Once the response has started, a
limit can only cut the connection. limit can only cut the connection.
- Counts each client's requests over a minute, an hour and a day. A request that - Counts each client's requests over a minute, an hour and a day. A request that
takes the client over one of the rate limits below is refused with `429` takes the client over one of the rate limits below is refused with
before anything reaches the app, and so is each request after it until the `SWWAF_BAN_RESPONSE`, `403` by default, before anything reaches the app, and
client is back under every limit. A client is one IPv4 address, or one IPv6 bans the client. A client is one IPv4 address, or one IPv6 /64, since one
/64, since one abuser usually holds a whole /64. Refused requests count too, abuser usually holds a whole /64. Each window is counted in two fixed buckets,
so a client that keeps sending too fast stays refused until it slows down. the earlier one weighted by how much of it the window still covers. At most
Each window is counted in two fixed buckets, the earlier one weighted by how 20,000 clients are kept, the least recently seen dropped first, and only in
much of it the window still covers. At most 20,000 clients are kept, the least memory: a restart starts every client afresh.
recently seen dropped first, and only in memory: a restart starts every client - Bans a client that breaks a rate limit, as "Bans" in [`SPEC.md`](SPEC.md)
afresh. describes: the first ban lasts an hour, and a limit broken again within a day
- Refuses a request from a country you refuse with `403`, as soon as the of a ban ending bans for three times as long as that ban, so 1, 3, 9, 27 and
client's country is known and before its body is read; such a request is not 81 hours; a ban that would last longer than seven days is permanent instead. A
counted for the rate limits. While one of the country lists below is set, each ban covers the client's netblock: its IPv4 address, or the netblock around it
client's country is looked up through GeoJS (see "Country and AS number that `SWWAF_BAN_SCOPE_V4_PREFIX` sets, or its IPv6 /64. While it lasts, every
lookup" below); with neither set, no visitor's address leaves the host. A request from the netblock is refused with `SWWAF_BAN_RESPONSE` after the
client on a private, loopback or link-local address has no country and is static lists and before the country lists, so the client is not looked up, and
is not counted for the rate limits. A ban sets the client's counters back to
zero. Each ban carries notes for deciding whether to lift it: the limit, its
window and the requests counted in it, the request that broke it, the client's
country when it was looked up, how many requests the ban has refused, and how
many bans the netblock had before. At most `SWWAF_MAX_BANS` bans are kept,
past, active and permanent; past that, the earliest ban of the netblock that
has gone longest without a request is dropped first. Bans and their notes are
kept in memory only, so a restart lifts every ban, and nothing shows them yet:
`bans.json`, which shows them and lets you lift a ban, comes with the state
files (https://git.eeqj.de/sneak/smallwebwaf/issues/17).
- Refuses a request from a country you refuse with `SWWAF_BAN_RESPONSE`, as soon
as the client's country is known and before its body is read; such a request
is not counted for the rate limits. While one of the country lists below is
set, each client's country is looked up through GeoJS (see "Country and AS
number lookup" below); with neither set, no visitor's address leaves the host.
A client on a private, loopback or link-local address has no country and is
never looked up: `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses it unless it is never looked up: `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses it unless it is
in `SWWAF_ALLOW_NETS`, and `SWWAF_DENIED_COUNTRIES` does not refuse it. in `SWWAF_ALLOW_NETS`, and `SWWAF_DENIED_COUNTRIES` does not refuse it.
- Checks the client's own address against the static lists, the three netblock - Checks the client's own address against the static lists, the three netblock
settings below, before anything else, its country included. A client in settings below, before anything else, its country included. A client in
`SWWAF_ALLOW_NETS` skips the country lists and the rate limits, and is not `SWWAF_ALLOW_NETS` skips bans, the country lists and the rate limits, and is
looked up; the timeouts and size limits still apply. A client in not looked up; the timeouts and size limits still apply. A client in
`SWWAF_DENY_NETS` is refused with `403` before its body is read, and the `SWWAF_DENY_NETS` is refused with `SWWAF_BAN_RESPONSE` before its body is
request is not counted for the rate limits; an address in `SWWAF_ALLOW_NETS` read, and the request is not counted for the rate limits; an address in
too is let through. A client in `SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither `SWWAF_ALLOW_NETS` too is let through. A client in
counted nor refused by the rate limits; the country lists still apply to it. `SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither counted nor refused by the rate
limits; the country lists and bans still apply to it.
- Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any - Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any
check and without asking the app, for the image's health check. check and without asking the app, for the image's health check.
- Writes a line in the request log for each request (see "Request log" below). - Writes a line in the request log for each request (see "Request log" below).
@@ -121,8 +139,9 @@ it, and the effective settings are logged at start.
to send its whole answer, from the end of the request to the last byte. to send its whole answer, from the end of the request to the last byte.
- `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body. - `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body.
- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body. - `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body.
- `SWWAF_ALLOW_NETS` (default empty): netblocks whose clients skip the country - `SWWAF_ALLOW_NETS` (default empty): netblocks whose clients skip bans, the
lists and the rate limits, such as your monitoring or your own networks. country lists and the rate limits, such as your monitoring or your own
networks.
- `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients the - `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients the
rate limits do not apply to, such as a machine that talks to the app all day. rate limits do not apply to, such as a machine that talks to the app all day.
- `SWWAF_DENY_NETS` (default empty): netblocks whose clients are always refused. - `SWWAF_DENY_NETS` (default empty): netblocks whose clients are always refused.
@@ -137,6 +156,22 @@ it, and the effective settings are logged at start.
countries whose clients get through, for example `us,de`. A client whose countries whose clients get through, for example `us,de`. A client whose
country cannot be found is refused too, so that new clients are not let in country cannot be found is refused too, so that new clients are not let in
whenever GeoJS stops answering. whenever GeoJS stops answering.
- `SWWAF_BAN_RESPONSE` (default `403`): how a refused client is answered, one
that is banned, breaks a rate limit, is in `SWWAF_DENY_NETS` or comes from a
refused country: `403`, `429`, or `close` to close the connection without an
answer. Behind traefik, `close` does not leave the client unanswered: traefik
answers `502`, as it does whenever its backend drops a connection.
- `SWWAF_LIMIT_BAN_DURATION` (default `1h`): the ban for a first broken rate
limit.
- `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): a rate limit broken again
within this time after a ban ended bans for three times as long as that ban.
- `SWWAF_MAX_BAN_DURATION` (default `7d`): a ban that would be longer is
permanent instead.
- `SWWAF_MAX_BANS` (default `5000`): the most bans kept, past, active and
permanent.
- `SWWAF_BAN_SCOPE_V4_PREFIX` (default `32`): the length of the netblock around
an IPv4 client that a ban covers, such as `24` to ban the surrounding /24. An
IPv6 ban covers the client's /64.
Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are
bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024 bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024
@@ -145,7 +180,7 @@ and a bare address stands for itself alone. Countries are the two-letter codes
ISO 3166-1 assigns today, and `xk` for Kosovo, in either case (`de` and `DE` are ISO 3166-1 assigns today, and `xk` for Kosovo, in either case (`de` and `DE` are
the same); any other code, such as `nk` (North Korea is `kp`) or the withdrawn the same); any other code, such as `nk` (North Korea is `kp`) or the withdrawn
`su`, stops the start, and so does a code on both country lists. `off` switches `su`, stops the start, and so does a code on both country lists. `off` switches
a timeout, a size limit or a rate limit off. a timeout, a size limit or a rate limit off; the ban settings cannot be off.
Several limits are fixed rather than settings. The request line and headers may Several limits are fixed rather than settings. The request line and headers may
take up to 32 KiB, above which the answer is `431` and nothing reaches the app. take up to 32 KiB, above which the answer is `431` and nothing reaches the app.
@@ -167,23 +202,27 @@ refused ones included:
- `time` is when the request arrived, in UTC. `peer_ip` is the TCP peer, - `time` is when the request arrived, in UTC. `peer_ip` is the TCP peer,
normally traefik. `path` and `query` are as the client sent them. normally traefik. `path` and `query` are as the client sent them.
- `country` is the client's country as GeoJS places it, and empty when it is not - `country` is the client's country as GeoJS places it. It is empty with neither
known: with neither country list set, for a client in `SWWAF_ALLOW_NETS` or country list set, for a client in `SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS`, for
`SWWAF_DENY_NETS`, for a client on a private, loopback or link-local address, a client on a private, loopback or link-local address, when GeoJS cannot place
and when GeoJS cannot place the client or has not answered in time. the client or has not answered in time, and for a request refused because a
ban covers its client, even when the client's country is known.
- `status` is what the client was sent, `0` if nothing was; `upstream_status` is - `status` is what the client was sent, `0` if nothing was; `upstream_status` is
what the app answered, and is left out when the app did not answer. what the app answered, and is left out when the app did not answer.
- `request_bytes` and `response_bytes` count body bytes. - `request_bytes` and `response_bytes` count body bytes.
- `action` is `forward` for a request passed to the app, `denied` for one - `action` is `forward` for a request passed to the app, `denied` for one
refused because its client is in `SWWAF_DENY_NETS`, `country_denied` for one refused because its client is in `SWWAF_DENY_NETS`, `banned` for one refused
refused for its client's country, `rate_limited` for one refused for a rate because a ban covers its client, `country_denied` for one refused for its
limit, `too_large` for a request or response over its size limit, `timed_out` client's country, `rate_limited` for one that broke a rate limit and banned
for one that ran out of time, `upstream_error` when the app could not be its client, `too_large` for a request or response over its size limit,
reached or its answer broke off, and `admin` for one `smallwebwaf` answered at `timed_out` for one that ran out of time, `upstream_error` when the app could
its own endpoint. not be reached or its answer broke off, and `admin` for one `smallwebwaf`
- `limit_hit` is there for a request refused for a rate limit, and names the answered at its own endpoint.
- `limit_hit` is there for a request that broke a rate limit, and names the
window whose limit it went over: `minute`, `hour` or `day`, the shortest if it window whose limit it went over: `minute`, `hour` or `day`, the shortest if it
went over several. went over several. `offence` is then `limit`.
- `ban_expires` is there for a request that made a ban or was refused under one,
and gives when the ban ends, in the same form as `time`, or `permanent`.
- `aborted` is there, and true, when the client went away early. - `aborted` is there, and true, when the client went away early.
- `duration_total` and `duration_upstream_total` are in milliseconds. - `duration_total` and `duration_upstream_total` are in milliseconds.
@@ -300,8 +339,8 @@ goes through the candidates one by one.
nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks
up the change. Nothing is read from disk while serving a request. The files up the change. Nothing is read from disk while serving a request. The files
come in milestone 3 or later (see the build order in [`SPEC.md`](SPEC.md)); come in milestone 3 or later (see the build order in [`SPEC.md`](SPEC.md));
until then the rate counters and the GeoJS answers are kept in memory only, until then the rate counters, the bans and the GeoJS answers are kept in
and a restart loses them. memory only, and a restart loses them.
- Health checks, the metrics, and listing, adding and lifting bans or asking why - Health checks, the metrics, and listing, adding and lifting bans or asking why
a given address was refused, all on the one port every request uses: under a given address was refused, all on the one port every request uses: under
`/_smallwebwaf/` on the app's own address, through traefik like any other `/_smallwebwaf/` on the app's own address, through traefik like any other
@@ -418,17 +457,18 @@ the metrics, failure behaviour and the build order.
So far `smallwebwaf` looks up only the country, only through GeoJS, and only So far `smallwebwaf` looks up only the country, only through GeoJS, and only
while `SWWAF_DENIED_COUNTRIES` or `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` is set: while `SWWAF_DENIED_COUNTRIES` or `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` is set:
then the address of every new visitor outside `SWWAF_ALLOW_NETS` and then the address of every new visitor is sent to GeoJS, except a visitor in
`SWWAF_DENY_NETS` is sent to GeoJS, and with neither set, none is. An IPv6 `SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS` and one refused because a ban covers its
visitor is asked about by the first address of its /64. A new visitor waits at netblock, and with neither set, none is. An IPv6 visitor is asked about by the
most a second for its answer, and without one counts as coming from an unknown first address of its /64. A new visitor waits at most a second for its answer,
country until the answer arrives. The addresses waiting are asked about and without one counts as coming from an unknown country until the answer
together, up to 200 in one request, one request at a time; at most 10,000 arrives. The addresses waiting are asked about together, up to 200 in one
visitors wait, and one more counts as coming from an unknown country until there request, one request at a time; at most 10,000 visitors wait, and one more
is room. While GeoJS fails, visitors with a kept answer are unaffected and new counts as coming from an unknown country until there is room. While GeoJS fails,
ones count as coming from an unknown country. GeoJS is then left alone for a visitors with a kept answer are unaffected and new ones count as coming from an
second, twice as long after each further failure up to five minutes, and asked unknown country. GeoJS is then left alone for a second, twice as long after each
again by the next request that needs it. further failure up to five minutes, and asked again by the next request that
needs it.
In the full design, `smallwebwaf` looks up the AS number and country of every In the full design, `smallwebwaf` looks up the AS number and country of every
client, for the request log, the metrics and the ban notes, and for the country client, for the request log, the metrics and the ban notes, and for the country
@@ -479,8 +519,10 @@ addresses are never sent to GeoJS.
standard library's `httputil.ReverseProxy` within the timeouts and size standard library's `httputil.ReverseProxy` within the timeouts and size
limits, and writes the request's log line. Its `check` method is where a limits, and writes the request's log line. Its `check` method is where a
request is refused before anything reaches the app: for `SWWAF_DENY_NETS`, for request is refused before anything reaches the app: for `SWWAF_DENY_NETS`, for
the country lists, for a rate limit, and for an announced body over the size a ban, for the country lists, for a rate limit, which bans the client, and for
limit. an announced body over the size limit.
- `internal/bans`: the ban ledger: each netblock's bans with their notes, how
long a new ban lasts, and which ban is dropped when `SWWAF_MAX_BANS` are held.
- `internal/lookup`: looks up each client's country through GeoJS, and keeps the - `internal/lookup`: looks up each client's country through GeoJS, and keeps the
answers. answers.
- `internal/ratelimit`: counts each client's requests and tells when one takes - `internal/ratelimit`: counts each client's requests and tells when one takes
@@ -494,8 +536,9 @@ addresses are never sent to GeoJS.
checks. checks.
Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the
table of clients to 20,000 and the GeoJS answers to 100,000, dropping the least table of clients to 20,000, the GeoJS answers to 100,000 and the banned
recently seen. The country codes are the list in `internal/config/config.go`. netblocks to `SWWAF_MAX_BANS`, dropping the least recently seen. The country
codes are the list in `internal/config/config.go`.
## Entrypoints ## Entrypoints
@@ -535,8 +578,9 @@ so that they run in minimal containers.
## TODO ## TODO
- The rest of milestone 3, after the static lists, and the rest of the design, - The rest of milestone 3, after the bans that broken rate limits lead to, and
in the order of the build order in [`SPEC.md`](SPEC.md). the rest of the design, in the order of the build order in
[`SPEC.md`](SPEC.md).
## Documents ## Documents
+118 -42
View File
@@ -1,6 +1,6 @@
--- ---
title: Repository Policies title: Repository Policies
last_modified: 2026-09-08 last_modified: 2026-10-04
--- ---
This document covers repository structure, tooling, and workflow standards. Code This document covers repository structure, tooling, and workflow standards. Code
@@ -104,10 +104,14 @@ style conventions are in separate documents:
`lint` phase and a `test` phase, with the final stage depending on both so the `lint` phase and a `test` phase, with the final stage depending on both so the
image cannot be built unless they pass. For non-server repos the final stage image cannot be built unless they pass. For non-server repos the final stage
brings up a development environment; for server repos it is the runtime image. brings up a development environment; for server repos it is the runtime image.
Dockerfiles install development prerequisites by running `script/bootstrap` The gate phases and the build stage start from their pinned base images and
rather than duplicating installs inline; COPY `script/` and the dependency install what those images lack either inline, as the canonical Go `Dockerfile`
manifests (`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before below does for `git`, or by running `script/bootstrap`, as the `prompts`
running it. repo's own `Dockerfile` does for its yarn packages. The development
environment stage installs development prerequisites by running
`script/bootstrap` rather than duplicating its installs inline. A stage that
runs `script/bootstrap` COPYs `script/` and the dependency manifests
(`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before running it.
- **Linting and testing run in Docker, as phases of the `Dockerfile`.** There is - **Linting and testing run in Docker, as phases of the `Dockerfile`.** There is
no separate lint file. `script/lint` and `script/test` each build one phase no separate lint file. `script/lint` and `script/test` each build one phase
@@ -156,11 +160,14 @@ style conventions are in separate documents:
not evidence that anything ran: a sub-second build reporting success is a not evidence that anything ran: a sub-second build reporting success is a
cache hit, not a result. Never invalidate by pruning — `docker builder prune` cache hit, not a result. Never invalidate by pruning — `docker builder prune`
and friends destroy a build cache shared with every other build on the host. and friends destroy a build cache shared with every other build on the host.
When a check is added or changed, prove it works by planting a defect it must
catch and watching the run fail on it, then revert the defect. A green run
alone shows neither that the check ran nor that it covers what it should.
- **The gate phases are separate stages, and the build stage depends on both.** - **The gate phases are separate stages, and the build stage depends on both.**
The lint phase is based on the `golangci/golangci-lint` image (pinned by The lint phase is based on the `golangci/golangci-lint` image (pinned by
hash), so lint failures surface in seconds rather than after a full compile, hash), so lint failures surface in seconds rather than after a full compile,
and the test phase is based on the Go image. The canonical Go repo and the test phase is based on the Debian Go image. The canonical Go repo
`Dockerfile`: `Dockerfile`:
```dockerfile ```dockerfile
@@ -173,8 +180,9 @@ style conventions are in separate documents:
COPY . . COPY . .
RUN golangci-lint run --config .golangci.yml ./... RUN golangci-lint run --config .golangci.yml ./...
# Test phase # Test phase. -race needs cgo and so a C compiler, which the Debian Go
# golang:1.x-alpine, YYYY-MM-DD # image ships and the alpine one does not.
# golang:1.x, YYYY-MM-DD
FROM golang@sha256:... AS test FROM golang@sha256:... AS test
WORKDIR /src WORKDIR /src
COPY go.mod go.sum ./ COPY go.mod go.sum ./
@@ -191,13 +199,27 @@ style conventions are in separate documents:
FROM golang@sha256:... AS builder FROM golang@sha256:... AS builder
COPY --from=lint /src/go.sum /dev/null COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null COPY --from=test /src/go.sum /dev/null
RUN apk add --no-cache git
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
WORKDIR /src WORKDIR /src
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
COPY . . COPY . .
ARG VERSION=dev # The VERSION build arg when one is given, otherwise
RUN CGO_ENABLED=0 go build -trimpath \ # `git describe --tags --always` on the .git in the build context. With
# .git present, a version that is still empty, dev or unknown fails the
# build: git is missing or could not read the checkout.
ARG VERSION
RUN VERSION="${VERSION:-$(git describe --tags --always)}"; \
if [ -e .git ]; then \
case "$VERSION" in ""|dev|unknown) \
echo "version is '$VERSION' although .git is present" >&2; \
exit 1 ;; \
esac; \
fi; \
CGO_ENABLED=0 go build -trimpath \
-ldflags="-s -w -X main.Version=${VERSION}" \ -ldflags="-s -w -X main.Version=${VERSION}" \
-o /app ./cmd/app/ -o /app ./cmd/app/
@@ -221,10 +243,41 @@ style conventions are in separate documents:
(e.g. a web frontend compiled in a separate stage), the lint phase must (e.g. a web frontend compiled in a separate stage), the lint phase must
create placeholder files so the embed directives resolve. Example: create placeholder files so the embed directives resolve. Example:
`RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`. `RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
- If the project requires CGO or system libraries for linting (e.g. - If the project requires CGO or system libraries for linting, install them
`vips-dev`), install them in the lint phase with `apk add`. in the lint phase. The `golangci/golangci-lint` image is Debian-based and
- `ARG VERSION=dev` is declared in the stage that compiles and supplied by has no `apk`, so install with `apt-get` under the Debian package name
`script/docker` and `script/cibuild`; no stage may call `git describe`. (`libvips-dev`, where alpine says `vips-dev`), and delete the package
lists in the same `RUN`, so the layer does not keep them:
```dockerfile
RUN apt-get update \
&& apt-get install -y --no-install-recommends libvips-dev \
&& rm -rf /var/lib/apt/lists/*
```
- `.dockerignore` lets `.git` into the build context. It keeps out every git
`config` at any depth (`**/.git/config`, `**/.git/modules/**/config`): the
repository's own, each submodule's under `.git/modules/`, and that of a
submodule keeping its own `.git` directory. `git describe` does not need
them, and each can hold a credential: a password in a remote URL, or the
token the CI checkout step stores there. A submodule whose name has a
`config` segment (`config`, `deploy/config`, `config/lib`) loses its whole
git directory to `**/.git/modules/**/config`, and Go's version stamping
then fails the build: give it a name without that segment
(`git submodule add --name`). The stage that compiles has `git` (the
Debian Go image has it; an alpine one needs `apk add --no-cache git`) and
takes the version from the `VERSION` build argument when one is given,
otherwise from `git describe --tags --always`. That gives the tag on a
tagged commit; on a later commit, the tag, the number of commits since it
and the short commit (`v1.2.3-4-gabc1234`); and the short commit when no
tag is reachable. The stage that compiles also marks its working directory
safe for git (`git config --system --add safe.directory /src`): a context
sent as a tar stream keeps the sender's file owners, and git refuses a
checkout owned by another user, so the version would come out empty.
`ARG VERSION` has no default, and the build fails if the context carries
`.git` and the version still comes out empty, `dev` or `unknown`. A plain
`docker build .` with no build arguments must succeed; a Dockerfile that
refuses an empty build argument drops that refusal and keeps the argument.
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that - Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
runs `script/cibuild` on push, and checks out the repo as its only other step. runs `script/cibuild` on push, and checks out the repo as its only other step.
@@ -233,7 +286,12 @@ style conventions are in separate documents:
carry the same guarantee, because its gate phases may come from the cache. The carry the same guarantee, because its gate phases may come from the cache. The
image build is uncached and so runs the gate phases a second time. That is the image build is uncached and so runs the gate phases a second time. That is the
price of the rule above, and it is worth paying: the image that ships is built price of the rule above, and it is worth paying: the image that ships is built
from a run of its own gates rather than from a cache entry. from a run of its own gates rather than from a cache entry. A separate
workflow limited to `main` by a `branches` list under `on: push` cannot be
checked by review: to try a change to it, add the feature branch to that list
and push, then remove the branch from the list again before merging. Keep any
job in it that publishes behind `if: github.ref_name == 'main'`, so the run
from the feature branch publishes nothing.
- Use platform-standard formatters: `black` for Python, `prettier` for - Use platform-standard formatters: `black` for Python, `prettier` for
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
@@ -286,17 +344,19 @@ style conventions are in separate documents:
``` ```
`-count=1` is required on both invocations: it defeats Go's test _result_ `-count=1` is required on both invocations: it defeats Go's test _result_
cache, so the target cannot report a pass it did not earn, and the rerun cache, so neither run can report a stored pass in place of running the
reproduces a failure instead of replaying it. It leaves the build cache tests. It leaves the build cache alone, so it costs the runtime of the suite
alone, so it costs the runtime of the suite and no recompilation. and no recompilation.
Note that this is a second, independent cache, stacked below the Docker That cache is Go's own, separate from Docker's layer cache. Go stores a
layer cache that [issue #26](https://git.eeqj.de/sneak/prompts/issues/26) passing result in its cache directory (`GOCACHE`), and when the same tests
addresses. `CHECK_EPOCH` guarantees the `RUN make test` _step_ re-executes; run again on unchanged code it prints that result, marked `(cached)`,
it does not guarantee `go test` inside that step does any work, because the without running them. That matters on a developer's machine, where this
`GOCACHE` baked into earlier image layers survives into the re-executed target runs and the directory lasts from one run to the next. The `test`
step. They are two separate defects requiring two separate fixes, and a fix phase of the `Dockerfile` needs no `-count=1`: its base image holds no
for one must not be recorded as covering the other. result for this repo's tests and nothing before its `go test` step runs a
test, so there is nothing to replay. `--no-cache` (above) is what makes that
step run on an unchanged tree.
Python example: Python example:
@@ -340,7 +400,7 @@ style conventions are in separate documents:
— which is more dangerous than a short file with no secret patterns at all, — which is more dangerous than a short file with no secret patterns at all,
because it reads as solved and stops anyone looking. Give every because it reads as solved and stops anyone looking. Give every
depth-independent pattern the `**/` prefix and leave only genuinely depth-independent pattern the `**/` prefix and leave only genuinely
root-anchored entries unprefixed: `.git`, and the repo's own host-built root-anchored entries unprefixed: `.claude`, and the repo's own host-built
binary, written `/myapp` and never `**/myapp`, which would also match binary, written `/myapp` and never `**/myapp`, which would also match
`cmd/myapp/` and delete the package directory from the context. Matching is `cmd/myapp/` and delete the package directory from the context. Matching is
case-sensitive, and an ALL-CAPS twin per pattern still misses `Server.Key`, so case-sensitive, and an ALL-CAPS twin per pattern still misses `Server.Key`, so
@@ -365,12 +425,13 @@ style conventions are in separate documents:
directory, so a repo running agents in subdirectories still ships directory, so a repo running agents in subdirectories still ships
`services/api/.claude/` and must add its own anchored entry there. `services/api/.claude/` and must add its own anchored entry there.
- **Excluding `.git` means `git describe` cannot run inside any build stage, and - **A plain `docker build .` of a clone stamps the version that
it fails quietly there.** In a build stage there is no repository, so `git describe --tags --always` gives**, derived from the `.git` in the build
`git describe` writes nothing to stdout, `-X main.Version=` comes out empty, context as the canonical `Dockerfile` above shows. Without its failure check,
the binary reports no version at all, and the build still exits 0. Compute the a missing `git` or an unreadable checkout would leave `-X main.Version=` empty
version on the host and thread it in as a build arg. `script/docker` and and the build would still exit 0. `script/docker` and `script/cibuild` pass
`script/cibuild` do this, byte-identically across repos: the version they compute on the host; it takes precedence. They do this
byte-identically across repos:
```sh ```sh
# Own line: a failing command substitution inside an argument does not # Own line: a failing command substitution inside an argument does not
@@ -387,7 +448,7 @@ style conventions are in separate documents:
fallback is applied — a live check that fires on a build from an export with fallback is applied — a live check that fires on a build from an export with
no `.git` and on a repository with no commits yet. Do not fold it into the no `.git` and on a repository with no commits yet. Do not fold it into the
substitution as `|| echo unknown`, which makes the guard unreachable. The substitution as `|| echo unknown`, which makes the guard unreachable. The
Dockerfile's side is `ARG VERSION=dev` in the stage that compiles, declared Dockerfile's side is `ARG VERSION` in the stage that compiles, declared
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
the scripts stay byte-identical. One consequence for CI: the standard the scripts stay byte-identical. One consequence for CI: the standard
@@ -426,12 +487,18 @@ style conventions are in separate documents:
`test-support` depguard rule, where a repo names its own test-support packages `test-support` depguard rule, where a repo names its own test-support packages
by full import path. A repo adds entries there and changes nothing else, and a by full import path. A repo adds entries there and changes nothing else, and a
re-vendor carries its entries forward. The canonical golangci-lint version is re-vendor carries its entries forward. The canonical golangci-lint version is
v2.12.2 (released 2026-05-06), pinned as the digest of the lint phase's base v2.14.0 (released 2026-09-24), pinned as the digest of the lint phase's base
image image
(`golangci/golangci-lint@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240`, (`golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f`,
which reports `2.12.2 built with go1.26.2 from c0d3ddc9`). That digest is the which reports `2.14.0 built with go1.27.0 from 114493f9`). A module's `go`
only pin, since no repo installs golangci-lint on the host: bumping the directive must not name a newer Go minor version than the one golangci-lint
version means changing it and nothing else. was built with, or golangci-lint refuses to lint it: this release lints
`go 1.27.1` but not `go 1.28`. That digest is the only pin, since no repo
installs golangci-lint on the host. A repo sets the lint phase digest to the
one named here and re-vendors `.golangci.yml` in the same commit, whichever of
the two prompted the change: the canonical copy can name linters that an older
golangci-lint rejects, and a newer golangci-lint can add linters that
`default: all` switches on until the canonical copy disables them.
- **`script/bootstrap` installs a pinned tool by comparing versions, never by - **`script/bootstrap` installs a pinned tool by comparing versions, never by
testing presence.** An `if ! command -v <tool>; then install; fi` guard tests testing presence.** An `if ! command -v <tool>; then install; fi` guard tests
@@ -455,6 +522,11 @@ style conventions are in separate documents:
Keep it POSIX sh: no arrays, no `[[`, no `grep -P`. Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.
A Go tool a repo needs on the host is installed with `go install` pinned to
a commit hash (`go install <package>@<commit hash>`). It is never tracked as
a `go.mod` tool dependency or through a `tools.go` file, either of which
pulls the tool's own dependencies into the repo's `go.mod` and `go.sum`.
- When pinning images or packages by hash, add a comment above the reference - When pinning images or packages by hash, add a comment above the reference
with the version and date (YYYY-MM-DD). with the version and date (YYYY-MM-DD).
@@ -567,10 +639,10 @@ style conventions are in separate documents:
settings. settings.
- Avoid putting files in the repo root unless necessary. Root should contain - Avoid putting files in the repo root unless necessary. Root should contain
only project-level config files (`README.md`, `Makefile`, `Dockerfile`, only project-level config files (`README.md`, `AGENTS.md`, `Makefile`,
`LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`, and `Dockerfile`, `LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`,
language-specific config). Everything else goes in a subdirectory. Canonical and language-specific config). Everything else goes in a subdirectory.
subdirectory names: Canonical subdirectory names:
- `bin/` — executable scripts and tools - `bin/` — executable scripts and tools
- `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose - `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose
body is a single call into `internal/` or `pkg/`, no project logic in body is a single call into `internal/` or `pkg/`, no project logic in
@@ -601,3 +673,7 @@ style conventions are in separate documents:
- Go: `go.mod`, `go.sum`, `.golangci.yml` - Go: `go.mod`, `go.sum`, `.golangci.yml`
- JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore` - JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore`
- Python: `pyproject.toml` - Python: `pyproject.toml`
- Guidance for coding agents lives in one `AGENTS.md` at the repository root. It
is never committed under a file or directory named after one agent tool, such
as `CLAUDE.md` or `.claude/`, and never split into separate memory files.
+10 -10
View File
@@ -945,9 +945,9 @@ and the running `smallwebwaf` takes the edit in.
- what was broken: the rule ids and target that matched, or the limit, its - what was broken: the rule ids and target that matched, or the limit, its
window, the count reached and the client's limit percentage with what set window, the count reached and the client's limit percentage with what set
it; and any reputation sources that listed the client; it; and any reputation sources that listed the client;
- the requests that caused the ban, up to the last ten: time, method, host, - the request that caused the ban, the one that broke the limit or carried
path with its query string, status and user agent, each text cut to 256 the clear sign of attack: time, method, host, path with its query string,
bytes; status and user agent, each text cut to 256 bytes;
- how many requests counted toward the ban, and the time span over which - how many requests counted toward the ban, and the time span over which
they came; they came;
- the netblock's total requests since it was first seen, and the requests - the netblock's total requests since it was first seen, and the requests
@@ -958,13 +958,13 @@ and the running `smallwebwaf` takes the edit in.
the table is full, so on a public service the file grows to the default the table is full, so on a public service the file grows to the default
`SWWAF_MAX_TRACKED_CLIENTS` of 20,000, about 20 MiB. Written every 15 `SWWAF_MAX_TRACKED_CLIENTS` of 20,000, about 20 MiB. Written every 15
minutes, that is under 2 GiB of disk writes a day. minutes, that is under 2 GiB of disk writes a day.
- `bans.json` takes about 2 KiB per ban and at most about 8 KiB, since the - `bans.json` takes about 1.2 KiB per ban and at most about 2.5 KiB, since
texts in the notes are cut short. At the default `SWWAF_MAX_BANS` of 5,000 the notes hold one request and their texts are cut short. At the default
it is about 10 MiB, and never more than about 40 MiB, plus whatever bans `SWWAF_MAX_BANS` of 5,000 it is about 6 MiB, and never more than about 12
an admin made. It is written when a ban is made, lifted or made permanent, MiB, plus whatever bans an admin made. It is written when a ban is made,
at most once every 10 seconds, and otherwise with the 15-minute write, so lifted or made permanent, at most once every 10 seconds, and otherwise
its writes follow the bans made: with a full file, a hundred new bans a with the 15-minute write, so its writes follow the bans made: with a full
day come to about 1 GiB of disk writes. file, a hundred new bans a day come to about 600 MiB of disk writes.
- `lookups.json` takes about 150 bytes per answer, about 15 MiB when full. - `lookups.json` takes about 150 bytes per answer, about 15 MiB when full.
Written every 15 minutes, that is under 1.5 GiB of disk writes a day. Written every 15 minutes, that is under 1.5 GiB of disk writes a day.
- `reputation.json` and `alerts.json` are usually a few MiB or less. - `reputation.json` and `alerts.json` are usually a few MiB or less.
+255
View File
@@ -0,0 +1,255 @@
// Package bans is the ban ledger: the bans smallwebwaf makes on the
// netblocks of clients that break a rate limit, with their notes, as the
// "Bans" section of SPEC.md describes. The bans are kept in memory only.
package bans
import (
"net/netip"
"slices"
"strings"
"sync"
"time"
"github.com/hashicorp/golang-lru/v2/simplelru"
)
// repeatFactor is how many times as long as the netblock's last ban a ban
// for a limit broken again within the repeat window lasts.
const repeatFactor = 3
// maxTextBytes is how much of each text in a ban's notes is kept.
const maxTextBytes = 256
// Rules are how long a ban for a broken limit lasts, and how many bans
// are held.
type Rules struct {
// LimitBanDuration is how long a first ban lasts.
LimitBanDuration time.Duration
// LimitBanRepeatWindow is how soon after the netblock's last ban
// ended a broken limit counts as a repeat, which bans for
// repeatFactor times as long as that ban.
LimitBanRepeatWindow time.Duration
// MaxBanDuration is the longest ban; a ban that would be longer is
// permanent instead.
MaxBanDuration time.Duration
// MaxBans is the most bans held, at least one. Past it, the earliest
// ban of the netblock that has gone longest without a request is
// dropped.
MaxBans int
}
// Ban is a ban on a netblock for a broken limit, the only kind of ban
// smallwebwaf makes so far.
type Ban struct {
Netblock netip.Prefix
Start time.Time
// Expires is when the ban ends, zero for a permanent ban.
Expires time.Time
Notes Notes
}
// Permanent reports whether the ban never runs out.
func (b Ban) Permanent() bool {
return b.Expires.IsZero()
}
// ActiveAt reports whether the ban refuses requests at now.
func (b Ban) ActiveAt(now time.Time) bool {
return b.Permanent() || now.Before(b.Expires)
}
// Notes are what an admin needs to decide whether to lift a ban.
type Notes struct {
// Country is the client's country, when it was looked up.
Country string
// Limit, Window and Count are the limit that was broken, its window,
// "minute", "hour" or "day", and the count reached: the client's
// requests in the window, the one that broke the limit included.
// These are the requests that counted toward the ban, and the window
// is the time over which they came.
Limit int64
Window string
Count float64
// Request is the request that broke the limit.
Request Request
// Refused is how many requests the ban has refused so far.
Refused int64
// EarlierBans is how many bans the netblock had before this one.
EarlierBans int
}
// Request is a request in a ban's notes. Each text is cut to 256 bytes.
type Request struct {
Time time.Time
Method string
Host string
// Path is the path with its query string.
Path string
// Status is what the client was sent, 0 if nothing was.
Status int
UserAgent string
}
// Ledger holds the bans. It is safe for concurrent use.
type Ledger struct {
rules Rules
mu sync.Mutex
// netblocks holds each banned netblock's bans, oldest first. Each
// request from a netblock makes it the most recently seen.
netblocks *simplelru.LRU[netip.Prefix, *[]Ban]
// held is how many bans netblocks holds, at most rules.MaxBans.
held int
}
// New returns a Ledger with no ban yet.
func New(rules Rules) *Ledger {
// Every netblock held has a ban, so there are never more netblocks
// than rules.MaxBans, and the LRU never drops one itself.
netblocks, err := simplelru.NewLRU[netip.Prefix, *[]Ban](rules.MaxBans, nil)
if err != nil {
panic(err) // NewLRU fails only for a size below one
}
return &Ledger{rules: rules, netblocks: netblocks}
}
// Check is called for each request from netblock, at now. It reports
// whether a ban on netblock is active, and returns that ban, with the
// request counted among those it refused.
func (l *Ledger) Check(netblock netip.Prefix, now time.Time) (Ban, bool) {
l.mu.Lock()
defer l.mu.Unlock()
bans, found := l.netblocks.Get(netblock)
if !found {
return Ban{}, false
}
// A ban is made only once the one before has ended, so only the last
// can be active.
last := &(*bans)[len(*bans)-1]
if !last.ActiveAt(now) {
return Ban{}, false
}
last.Notes.Refused++
return *last, true
}
// BanForLimit bans netblock at now for a broken limit, with notes, and
// returns the ban. A first ban lasts LimitBanDuration. A ban made within
// LimitBanRepeatWindow after the netblock's last ban ended lasts
// repeatFactor times as long as that one. A ban that would be longer
// than MaxBanDuration is permanent instead. If a ban on netblock is still
// active, as when two of its requests break a limit at once, that ban is
// returned and no other is made. The ledger fills in the notes' Refused
// and EarlierBans itself.
func (l *Ledger) BanForLimit(netblock netip.Prefix, now time.Time, notes Notes) Ban {
l.mu.Lock()
defer l.mu.Unlock()
var last *Ban
bans, found := l.netblocks.Get(netblock)
if found {
last = &(*bans)[len(*bans)-1]
if last.ActiveAt(now) {
return *last
}
notes.EarlierBans = last.Notes.EarlierBans + 1
}
notes.Request = notes.Request.cut()
ban := Ban{
Netblock: netblock,
Start: now,
Expires: l.expiry(last, now),
Notes: notes,
}
if l.held == l.rules.MaxBans {
l.dropOne()
}
// dropOne can have dropped netblock's last ban, and netblock with it.
bans, found = l.netblocks.Peek(netblock)
if !found {
bans = &[]Ban{}
l.netblocks.Add(netblock, bans)
}
*bans = append(*bans, ban)
l.held++
return ban
}
// Bans returns the bans held on netblock, oldest first. It is not a
// request from netblock, and leaves when it was last seen unchanged.
func (l *Ledger) Bans(netblock netip.Prefix) []Ban {
l.mu.Lock()
defer l.mu.Unlock()
bans, found := l.netblocks.Peek(netblock)
if !found {
return nil
}
return slices.Clone(*bans)
}
// expiry returns when a ban for a broken limit made at now ends, or zero
// when it is permanent. last is the netblock's last ban, which has ended,
// or nil when it has none.
func (l *Ledger) expiry(last *Ban, now time.Time) time.Time {
length := l.rules.LimitBanDuration
if last != nil && now.Sub(last.Expires) <= l.rules.LimitBanRepeatWindow {
lastLength := last.Expires.Sub(last.Start)
// This is repeatFactor * lastLength > MaxBanDuration, written so
// that it cannot overflow.
if lastLength > l.rules.MaxBanDuration/repeatFactor {
return time.Time{}
}
length = repeatFactor * lastLength
}
if length > l.rules.MaxBanDuration {
return time.Time{}
}
return now.Add(length)
}
// dropOne drops the earliest ban of the netblock that has gone longest
// without a request, and the netblock with it if that was its only ban.
func (l *Ledger) dropOne() {
netblock, bans, _ := l.netblocks.GetOldest()
if len(*bans) == 1 {
l.netblocks.Remove(netblock)
} else {
*bans = slices.Delete(*bans, 0, 1)
}
l.held--
}
// cut returns r with each text cut to maxTextBytes and copied, so that
// the notes do not keep the rest of the request in memory.
func (r Request) cut() Request {
r.Method = cutText(r.Method)
r.Host = cutText(r.Host)
r.Path = cutText(r.Path)
r.UserAgent = cutText(r.UserAgent)
return r
}
// cutText returns a copy of the first maxTextBytes of text.
func cutText(text string) string {
return strings.Clone(text[:min(len(text), maxTextBytes)])
}
+247
View File
@@ -0,0 +1,247 @@
package bans_test
import (
"net/netip"
"strings"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
)
const day = 24 * time.Hour
func TestRepeatsTripleUntilPermanent(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultRules())
netblock := netip.MustParsePrefix("203.0.113.9/32")
now := midnight()
// Each ban is followed by another as soon as it ends: 1, 3, 9, 27 and
// 81 hours.
for i, hours := range []int{1, 3, 9, 27, 81} {
ban := ledger.BanForLimit(netblock, now, bans.Notes{})
length := time.Duration(hours) * time.Hour
if !ban.Expires.Equal(now.Add(length)) || ban.Notes.EarlierBans != i {
t.Fatalf("ban %d lasts %s with %d earlier bans, want %d hours and %d",
i+1, ban.Expires.Sub(now), ban.Notes.EarlierBans, hours, i)
}
now = ban.Expires
}
// The sixth would last 243 hours, more than seven days: it is
// permanent, and never ends.
ban := ledger.BanForLimit(netblock, now, bans.Notes{})
if !ban.Permanent() {
t.Fatalf("sixth ban ends at %s, want a permanent one", ban.Expires)
}
_, banned := ledger.Check(netblock, now.Add(100*365*day))
if !banned {
t.Error("a permanent ban ended")
}
}
func TestRepeatWindowRunsOut(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
// gap is the time between the end of the first ban and the second.
gap time.Duration
want time.Duration
}{
{"broken again as the window ends", day, 3 * time.Hour},
{"broken again after the window", day + time.Nanosecond, time.Hour},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultRules())
netblock := netip.MustParsePrefix("203.0.113.9/32")
first := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
second := ledger.BanForLimit(netblock, first.Expires.Add(tc.gap), bans.Notes{})
if second.Expires.Sub(second.Start) != tc.want || second.Notes.EarlierBans != 1 {
t.Errorf("second ban lasts %s with %d earlier bans, want %s and 1",
second.Expires.Sub(second.Start), second.Notes.EarlierBans, tc.want)
}
})
}
}
func TestFirstBanLongerThanTheMaximumIsPermanent(t *testing.T) {
t.Parallel()
rules := defaultRules()
rules.LimitBanDuration = rules.MaxBanDuration + time.Hour
ledger := bans.New(rules)
ban := ledger.BanForLimit(netip.MustParsePrefix("203.0.113.9/32"), midnight(),
bans.Notes{})
if !ban.Permanent() {
t.Errorf("first ban ends at %s, want a permanent one", ban.Expires)
}
}
func TestLongestBanSetFarOffDoesNotOverflow(t *testing.T) {
t.Parallel()
// With bans of up to 100,000 days, the 14th ban in a row, of 3^13
// hours, is within the maximum, and three times as long would not fit
// in a time.Duration. The 15th is permanent.
rules := defaultRules()
rules.MaxBanDuration = 100000 * day
ledger := bans.New(rules)
netblock := netip.MustParsePrefix("203.0.113.9/32")
now := midnight()
for i := range 14 {
ban := ledger.BanForLimit(netblock, now, bans.Notes{})
if !ban.Expires.After(ban.Start) {
t.Fatalf("ban %d starts at %s and ends at %s", i+1, ban.Start, ban.Expires)
}
now = ban.Expires
}
ban := ledger.BanForLimit(netblock, now, bans.Notes{})
if !ban.Permanent() {
t.Errorf("15th ban ends at %s, want a permanent one", ban.Expires)
}
}
func TestBrokenLimitDuringABanMakesNoOther(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultRules())
netblock := netip.MustParsePrefix("203.0.113.9/32")
first := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
again := ledger.BanForLimit(netblock, midnight().Add(time.Minute), bans.Notes{})
if again != first || len(ledger.Bans(netblock)) != 1 {
t.Errorf("a limit broken during a ban gave %+v and %d bans, want %+v and 1",
again, len(ledger.Bans(netblock)), first)
}
}
func TestCheckRefusesWhileTheBanLastsAndCountsTheRefusals(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultRules())
netblock := netip.MustParsePrefix("203.0.113.9/32")
ban := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
for range 3 {
got, banned := ledger.Check(netblock, ban.Expires.Add(-time.Nanosecond))
if !banned || got.Start != ban.Start {
t.Fatalf("check during the ban gives %+v and %t", got, banned)
}
}
_, banned := ledger.Check(netip.MustParsePrefix("203.0.113.10/32"), midnight())
if banned {
t.Error("another netblock is banned")
}
_, banned = ledger.Check(netblock, ban.Expires)
if banned {
t.Error("the ban did not end")
}
refused := ledger.Bans(netblock)[0].Notes.Refused
if refused != 3 {
t.Errorf("the notes count %d refused requests, want 3", refused)
}
}
func TestMaxBansDropsTheEarliestBanOfTheNetblockSeenLongestAgo(t *testing.T) {
t.Parallel()
rules := defaultRules()
rules.MaxBans = 3
ledger := bans.New(rules)
a := netip.MustParsePrefix("203.0.113.1/32")
b := netip.MustParsePrefix("203.0.113.2/32")
c := netip.MustParsePrefix("203.0.113.3/32")
d := netip.MustParsePrefix("2001:db8::/64")
now := midnight()
first := ledger.BanForLimit(a, now, bans.Notes{})
ledger.BanForLimit(b, now, bans.Notes{})
ledger.BanForLimit(c, now, bans.Notes{})
// A request from a makes b the netblock seen longest ago, and its ban
// goes to make room for d's.
ledger.Check(a, now)
ledger.BanForLimit(d, now, bans.Notes{})
wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 0, c: 1, d: 1})
// a is banned again once its ban has ended; c, seen longest ago, goes.
ledger.BanForLimit(a, first.Expires, bans.Notes{})
wantBans(t, ledger, map[netip.Prefix]int{a: 2, c: 0, d: 1})
// With d seen since, a is seen longest ago, and its earlier ban goes
// first.
ledger.Check(d, first.Expires)
ledger.BanForLimit(b, first.Expires, bans.Notes{})
wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 1, d: 1})
if !ledger.Bans(a)[0].Start.Equal(first.Expires) {
t.Errorf("a kept its ban of %s, want the later one", ledger.Bans(a)[0].Start)
}
}
func TestRequestTextsAreCutTo256Bytes(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultRules())
netblock := netip.MustParsePrefix("203.0.113.9/32")
long := strings.Repeat("a", 300)
request := bans.Request{
Time: midnight(), Method: long, Host: long, Path: long, Status: 403, UserAgent: long,
}
ban := ledger.BanForLimit(netblock, midnight(), bans.Notes{Request: request})
cut := long[:256]
want := bans.Request{
Time: midnight(), Method: cut, Host: cut, Path: cut, Status: 403, UserAgent: cut,
}
if ban.Notes.Request != want || ledger.Bans(netblock)[0].Notes.Request != want {
t.Errorf("the notes keep %+v, want each text cut to 256 bytes", ban.Notes.Request)
}
}
// defaultRules are the rules at the settings' defaults.
func defaultRules() bans.Rules {
return bans.Rules{
LimitBanDuration: time.Hour,
LimitBanRepeatWindow: day,
MaxBanDuration: 7 * day,
MaxBans: 5000,
}
}
// midnight is when the tests' first bans are made.
func midnight() time.Time {
return time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)
}
// wantBans checks how many bans the ledger holds on each netblock.
func wantBans(t *testing.T, ledger *bans.Ledger, want map[netip.Prefix]int) {
t.Helper()
for netblock, count := range want {
got := len(ledger.Bans(netblock))
if got != count {
t.Errorf("%s has %d bans, want %d", netblock, got, count)
}
}
}
+114
View File
@@ -9,6 +9,7 @@ import (
"log/slog" "log/slog"
"math" "math"
"net" "net"
"net/http"
"net/netip" "net/netip"
"net/url" "net/url"
"slices" "slices"
@@ -67,6 +68,25 @@ type Config struct {
// capitals, as GeoJS gives them. // capitals, as GeoJS gives them.
DeniedCountries []string DeniedCountries []string
ExclusivelyAllowedCountries []string ExclusivelyAllowedCountries []string
// BanResponse is the status a refused client is answered with, 403
// or 429, or 0 to close the connection without an answer
// (SWWAF_BAN_RESPONSE). It answers a banned client, a request that
// breaks a rate limit, SWWAF_DENY_NETS and the country lists.
BanResponse int
// LimitBanDuration is the ban for a first broken rate limit
// (SWWAF_LIMIT_BAN_DURATION). A limit broken again within
// LimitBanRepeatWindow after the last ban ended bans for three times
// as long as that ban (SWWAF_LIMIT_BAN_REPEAT_WINDOW), and a ban that
// would be longer than MaxBanDuration is permanent instead
// (SWWAF_MAX_BAN_DURATION). None of them can be off.
LimitBanDuration time.Duration
LimitBanRepeatWindow time.Duration
MaxBanDuration time.Duration
// MaxBans is the most bans held (SWWAF_MAX_BANS).
MaxBans int
// BanScopeV4Prefix is the length of the netblock around an IPv4
// client that a ban covers (SWWAF_BAN_SCOPE_V4_PREFIX).
BanScopeV4Prefix int
// settings are the values read, as given or by default, for the // settings are the values read, as given or by default, for the
// log line at start. // log line at start.
@@ -82,6 +102,7 @@ const (
kibibyte = 1 << 10 kibibyte = 1 << 10
mebibyte = 1 << 20 mebibyte = 1 << 20
gibibyte = 1 << 30 gibibyte = 1 << 30
ipv4Bits = 32
) )
var ( var (
@@ -103,6 +124,13 @@ var (
errNotCountry = errors.New( errNotCountry = errors.New(
"is not a two-letter country code such as de or kp") "is not a two-letter country code such as de or kp")
errOnBothLists = errors.New("is in SWWAF_DENIED_COUNTRIES too") errOnBothLists = errors.New("is in SWWAF_DENIED_COUNTRIES too")
errNotDurationAboveZero = errors.New(
"is not a duration above zero, such as 1h or 7d")
errNotNumberAboveZero = errors.New(
"is not a whole number above zero, such as 5000")
errNotBanResponse = errors.New("is not 403, 429 or close")
errNotV4Prefix = errors.New(
"is not the length of an IPv4 netblock, from 0 to 32, such as 24")
) )
// FromEnvironment reads the settings with lookupEnv, normally // FromEnvironment reads the settings with lookupEnv, normally
@@ -129,6 +157,12 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
DeniedCountries: env.countries("SWWAF_DENIED_COUNTRIES", ""), DeniedCountries: env.countries("SWWAF_DENIED_COUNTRIES", ""),
ExclusivelyAllowedCountries: env.countries( ExclusivelyAllowedCountries: env.countries(
"SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES", ""), "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES", ""),
BanResponse: env.banResponse("SWWAF_BAN_RESPONSE", "403"),
LimitBanDuration: env.durationNotOff("SWWAF_LIMIT_BAN_DURATION", "1h"),
LimitBanRepeatWindow: env.durationNotOff("SWWAF_LIMIT_BAN_REPEAT_WINDOW", "24h"),
MaxBanDuration: env.durationNotOff("SWWAF_MAX_BAN_DURATION", "7d"),
MaxBans: env.numberNotOff("SWWAF_MAX_BANS", "5000"),
BanScopeV4Prefix: env.v4Prefix("SWWAF_BAN_SCOPE_V4_PREFIX", "32"),
} }
for _, country := range cfg.ExclusivelyAllowedCountries { for _, country := range cfg.ExclusivelyAllowedCountries {
@@ -241,6 +275,40 @@ func (e *environment) countries(name, defaultValue string) []string {
return countries return countries
} }
// durationNotOff reads a setting that is a duration and, unlike a
// timeout, cannot be off.
func (e *environment) durationNotOff(name, defaultValue string) time.Duration {
duration, err := parseDurationNotOff(e.value(name, defaultValue))
e.check(name, err)
return duration
}
// numberNotOff reads a setting that is a whole number above zero, which
// cannot be off.
func (e *environment) numberNotOff(name, defaultValue string) int {
number, err := parseNumberNotOff(e.value(name, defaultValue))
e.check(name, err)
return number
}
// banResponse reads a setting that is how a refused client is answered.
func (e *environment) banResponse(name, defaultValue string) int {
status, err := parseBanResponse(e.value(name, defaultValue))
e.check(name, err)
return status
}
// v4Prefix reads a setting that is the length of an IPv4 netblock.
func (e *environment) v4Prefix(name, defaultValue string) int {
length, err := parseV4Prefix(e.value(name, defaultValue))
e.check(name, err)
return length
}
// parseDuration reads a duration in Go's syntax, such as 90s or 15m, a // parseDuration reads a duration in Go's syntax, such as 90s or 15m, a
// whole number of days such as 7d, or off. // whole number of days such as 7d, or off.
func parseDuration(value string) (time.Duration, error) { func parseDuration(value string) (time.Duration, error) {
@@ -329,6 +397,52 @@ func parseCount(value string) (int64, error) {
return n, nil return n, nil
} }
// parseDurationNotOff reads a duration above zero, as parseDuration does,
// but not off.
func parseDurationNotOff(value string) (time.Duration, error) {
duration, err := parseDuration(value)
if err != nil || duration == 0 {
return 0, fmt.Errorf("%q %w", value, errNotDurationAboveZero)
}
return duration, nil
}
// parseNumberNotOff reads a whole number above zero.
func parseNumberNotOff(value string) (int, error) {
n, err := strconv.Atoi(value)
if err != nil || n <= 0 {
return 0, fmt.Errorf("%q %w", value, errNotNumberAboveZero)
}
return n, nil
}
// parseBanResponse reads how a refused client is answered: 403, 429, or
// close, which is 0.
func parseBanResponse(value string) (int, error) {
switch value {
case "403":
return http.StatusForbidden, nil
case "429":
return http.StatusTooManyRequests, nil
case "close":
return 0, nil
default:
return 0, fmt.Errorf("%q %w", value, errNotBanResponse)
}
}
// parseV4Prefix reads the length of an IPv4 netblock, from 0 to 32.
func parseV4Prefix(value string) (int, error) {
n, err := strconv.Atoi(value)
if err != nil || n < 0 || n > ipv4Bits {
return 0, fmt.Errorf("%q %w", value, errNotV4Prefix)
}
return n, nil
}
// parseList splits a comma-separated list and trims the spaces around // parseList splits a comma-separated list and trims the spaces around
// each item. An empty value is an empty list. // each item. An empty value is an empty list.
func parseList(value string) ([]string, error) { func parseList(value string) ([]string, error) {
+61
View File
@@ -33,6 +33,12 @@ const (
rateLimitPerDay = "SWWAF_RATE_LIMIT_PER_DAY" rateLimitPerDay = "SWWAF_RATE_LIMIT_PER_DAY"
deniedCountries = "SWWAF_DENIED_COUNTRIES" deniedCountries = "SWWAF_DENIED_COUNTRIES"
allowedCountries = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES" allowedCountries = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES"
banResponse = "SWWAF_BAN_RESPONSE"
limitBanDuration = "SWWAF_LIMIT_BAN_DURATION"
limitBanRepeatWindow = "SWWAF_LIMIT_BAN_REPEAT_WINDOW"
maxBanDuration = "SWWAF_MAX_BAN_DURATION"
maxBans = "SWWAF_MAX_BANS"
banScopeV4Prefix = "SWWAF_BAN_SCOPE_V4_PREFIX"
) )
// off switches a timeout, a size limit or a rate limit off. // off switches a timeout, a size limit or a rate limit off.
@@ -76,6 +82,12 @@ func TestDefaults(t *testing.T) {
RateLimitPerMinute: 1000, RateLimitPerMinute: 1000,
RateLimitPerHour: 10000, RateLimitPerHour: 10000,
RateLimitPerDay: 50000, RateLimitPerDay: 50000,
BanResponse: 403,
LimitBanDuration: time.Hour,
LimitBanRepeatWindow: 24 * time.Hour,
MaxBanDuration: 7 * 24 * time.Hour,
MaxBans: 5000,
BanScopeV4Prefix: 32,
}) })
if cfg.UpstreamURL.String() != "http://127.0.0.1:8081" { if cfg.UpstreamURL.String() != "http://127.0.0.1:8081" {
@@ -112,6 +124,12 @@ func TestValuesAsSet(t *testing.T) {
rateLimitPerDay: "6000", rateLimitPerDay: "6000",
deniedCountries: "cn, RU,kp,Xk", deniedCountries: "cn, RU,kp,Xk",
allowedCountries: "de", allowedCountries: "de",
banResponse: "429",
limitBanDuration: "15m",
limitBanRepeatWindow: "2d",
maxBanDuration: "30d",
maxBans: "100",
banScopeV4Prefix: "24",
}) })
wantSettings(t, cfg, config.Config{ wantSettings(t, cfg, config.Config{
@@ -125,6 +143,12 @@ func TestValuesAsSet(t *testing.T) {
RateLimitPerMinute: 60, RateLimitPerMinute: 60,
RateLimitPerHour: 600, RateLimitPerHour: 600,
RateLimitPerDay: 6000, RateLimitPerDay: 6000,
BanResponse: 429,
LimitBanDuration: 15 * time.Minute,
LimitBanRepeatWindow: 48 * time.Hour,
MaxBanDuration: 30 * 24 * time.Hour,
MaxBans: 100,
BanScopeV4Prefix: 24,
}) })
if cfg.UpstreamURL.String() != "https://app.internal:8443/" { if cfg.UpstreamURL.String() != "https://app.internal:8443/" {
@@ -188,6 +212,15 @@ func TestRateLimitsOff(t *testing.T) {
} }
} }
func TestBanResponseCloseIsZero(t *testing.T) {
t.Parallel()
cfg := fromEnvironment(t, environment{banResponse: "close"})
if cfg.BanResponse != 0 {
t.Errorf("close read as %d, want 0", cfg.BanResponse)
}
}
func TestTrustedProxiesSetButEmptyTrustNothing(t *testing.T) { func TestTrustedProxiesSetButEmptyTrustNothing(t *testing.T) {
t.Parallel() t.Parallel()
@@ -250,6 +283,12 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
{allowedCountries, "uk"}, {allowedCountries, "uk"},
{allowedCountries, "zz"}, {allowedCountries, "zz"},
{allowedCountries, "de,germany"}, {allowedCountries, "de,germany"},
{banResponse, "404"}, {banResponse, "drop"}, {banResponse, ""},
{limitBanDuration, off}, {limitBanDuration, "0s"}, {limitBanDuration, "1"},
{limitBanRepeatWindow, off}, {limitBanRepeatWindow, "-1h"},
{maxBanDuration, off}, {maxBanDuration, "1w"},
{maxBans, off}, {maxBans, "0"}, {maxBans, "5K"},
{banScopeV4Prefix, "33"}, {banScopeV4Prefix, "-1"}, {banScopeV4Prefix, "/24"},
} { } {
t.Run(tc.name+"="+tc.value, func(t *testing.T) { t.Run(tc.name+"="+tc.value, func(t *testing.T) {
t.Parallel() t.Parallel()
@@ -302,6 +341,12 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
rateLimitPerDay: "50000", rateLimitPerDay: "50000",
deniedCountries: "", deniedCountries: "",
allowedCountries: "", allowedCountries: "",
banResponse: "403",
limitBanDuration: "1h",
limitBanRepeatWindow: "24h",
maxBanDuration: "7d",
maxBans: "5000",
banScopeV4Prefix: "32",
} }
if !maps.Equal(line.Settings, want) { if !maps.Equal(line.Settings, want) {
t.Errorf("logged settings\n%v\nwant\n%v", line.Settings, want) t.Errorf("logged settings\n%v\nwant\n%v", line.Settings, want)
@@ -324,6 +369,22 @@ func wantSettings(t *testing.T, got *config.Config, want config.Config) {
got.RateLimitPerDay != want.RateLimitPerDay { got.RateLimitPerDay != want.RateLimitPerDay {
t.Errorf("settings\n%+v\nwant\n%+v", got, want) t.Errorf("settings\n%+v\nwant\n%+v", got, want)
} }
wantBanSettings(t, got, want)
}
// wantBanSettings checks the settings for bans.
func wantBanSettings(t *testing.T, got *config.Config, want config.Config) {
t.Helper()
if got.BanResponse != want.BanResponse ||
got.LimitBanDuration != want.LimitBanDuration ||
got.LimitBanRepeatWindow != want.LimitBanRepeatWindow ||
got.MaxBanDuration != want.MaxBanDuration ||
got.MaxBans != want.MaxBans ||
got.BanScopeV4Prefix != want.BanScopeV4Prefix {
t.Errorf("ban settings\n%+v\nwant\n%+v", got, want)
}
} }
// wantNetblocks checks a list of netblocks. // wantNetblocks checks a list of netblocks.
+82
View File
@@ -0,0 +1,82 @@
package proxy
import (
"net/netip"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// banResponse is a refusal answered with SWWAF_BAN_RESPONSE, and logged
// with action.
func (rq *request) banResponse(action string) *refusal {
return &refusal{status: rq.h.config.BanResponse, action: action}
}
// banned reports whether a ban on the client's netblock refuses the
// request at now, and notes for the log line when that ban ends.
func (rq *request) banned(now time.Time) bool {
ban, banned := rq.h.ledger.Check(rq.netblock(), now)
if banned {
rq.line.BanExpires = banExpires(ban)
}
return banned
}
// limitBroken counts the request for the rate limits at now, and reports
// whether it takes the client over one. Such a request bans the client's
// netblock, and sets the client's counters back to zero.
func (rq *request) limitBroken(now time.Time) bool {
group := clientGroup(rq.client)
hit, over := rq.h.limiter.Count(group, now)
if !over {
return false
}
ban := rq.h.ledger.BanForLimit(rq.netblock(), now, bans.Notes{
Country: rq.line.Country,
Limit: hit.Limit,
Window: hit.Window,
Count: hit.Requests,
Request: bans.Request{
Time: now,
Method: rq.in.Method,
Host: rq.in.Host,
Path: rq.in.URL.RequestURI(),
Status: rq.h.config.BanResponse,
UserAgent: rq.in.UserAgent(),
},
})
rq.h.limiter.Reset(group)
rq.line.LimitHit = hit.Window
rq.line.Offence = requestlog.OffenceLimit
rq.line.BanExpires = banExpires(ban)
return true
}
// netblock is the netblock a ban on the client covers: its IPv4 address,
// widened to SWWAF_BAN_SCOPE_V4_PREFIX, or the IPv6 group clientGroup
// counts it in.
func (rq *request) netblock() netip.Prefix {
addr := rq.client.Unmap()
if addr.Is4() {
return netip.PrefixFrom(addr, rq.h.config.BanScopeV4Prefix).Masked()
}
return clientGroup(addr)
}
// banExpires is when ban ends, as the log line gives it: a time, or
// permanent.
func banExpires(ban bans.Ban) string {
if ban.Permanent() {
return "permanent"
}
return requestlog.FormatTime(ban.Expires)
}
+434
View File
@@ -0,0 +1,434 @@
package proxy_test
import (
"bufio"
"errors"
"io"
"maps"
"net/http"
"net/netip"
"slices"
"sync"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/proxy"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
const (
// otherClient is a client next to client.
otherClient = "203.0.113.10"
// userAgent is the user agent of every request a sender sends.
userAgent = "ban-test/1.0"
// permanent is the log line's ban_expires for a permanent ban.
permanent = "permanent"
)
func TestBrokenLimitBansTheClient(t *testing.T) {
t.Parallel()
s, clk, _ := startWithClock(t, "", map[string]string{rateLimitPerMinute: "1"})
expires := requestlog.FormatTime(clk.Now().Add(time.Hour))
// The request over the limit of one a minute is refused, and bans the
// client for an hour, the default.
s.get(client, http.StatusOK, requestlog.ActionForward)
line := s.get(client, http.StatusForbidden, requestlog.ActionRateLimited)
if line.LimitHit != minute || line.Offence != requestlog.OffenceLimit ||
line.BanExpires != expires {
t.Errorf("log line has limit_hit %q, offence %q and ban_expires %q, "+
"want minute, limit and %s", line.LimitHit, line.Offence, line.BanExpires,
expires)
}
// Every request while the ban lasts is refused.
clk.advance(time.Hour - time.Second)
line = s.get(client, http.StatusForbidden, requestlog.ActionBanned)
if line.BanExpires != expires || line.Offence != "" || line.LimitHit != "" {
t.Errorf("log line has ban_expires %q, offence %q and limit_hit %q, "+
"want %s and neither of the others", line.BanExpires, line.Offence,
line.LimitHit, expires)
}
// Once it ends, the client is let through.
clk.advance(time.Second)
s.get(client, http.StatusOK, requestlog.ActionForward)
}
func TestBanLengthsFollowTheSettings(t *testing.T) {
t.Parallel()
s, clk, _ := startWithClock(t, "", map[string]string{
rateLimitPerMinute: "1",
limitBanDuration: "10m",
limitBanRepeatWindow: "1h",
maxBanDuration: "1h",
})
// breakLimit has client go over the limit of one a minute, and
// returns when the ban that makes ends.
breakLimit := func() string {
s.get(client, http.StatusOK, requestlog.ActionForward)
return s.get(client, http.StatusForbidden, requestlog.ActionRateLimited).BanExpires
}
wantExpires := func(got string, length time.Duration) {
t.Helper()
want := requestlog.FormatTime(clk.Now().Add(length))
if got != want {
t.Errorf("ban ends at %s, want %s", got, want)
}
}
// A first ban lasts SWWAF_LIMIT_BAN_DURATION; one within
// SWWAF_LIMIT_BAN_REPEAT_WINDOW after it ended, three times as long.
wantExpires(breakLimit(), 10*time.Minute)
clk.advance(10*time.Minute + time.Hour)
wantExpires(breakLimit(), 30*time.Minute)
// Later than that, SWWAF_LIMIT_BAN_DURATION again.
clk.advance(30*time.Minute + time.Hour + time.Second)
wantExpires(breakLimit(), 10*time.Minute)
clk.advance(10 * time.Minute)
wantExpires(breakLimit(), 30*time.Minute)
// 90 minutes would be longer than SWWAF_MAX_BAN_DURATION: the ban is
// permanent.
clk.advance(30 * time.Minute)
got := breakLimit()
if got != permanent {
t.Errorf("ban ends at %s, want a permanent one", got)
}
clk.advance(365 * 24 * time.Hour)
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
}
func TestBanIsNotCountedAndResetsTheCounters(t *testing.T) {
t.Parallel()
s, clk, server := startWithClock(t, "", map[string]string{rateLimitPerDay: "2"})
// The third request in a day is over the limit of two, and bans the
// client for an hour.
s.get(client, http.StatusOK, requestlog.ActionForward)
s.get(client, http.StatusOK, requestlog.ActionForward)
s.get(client, http.StatusForbidden, requestlog.ActionRateLimited)
for range 3 {
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
}
// Later the same day the client has its whole allowance again: the
// ban set its counters back to zero, and the requests it refused were
// not counted for the rate limits, only in its notes.
clk.advance(time.Hour)
s.get(client, http.StatusOK, requestlog.ActionForward)
s.get(client, http.StatusOK, requestlog.ActionForward)
s.get(client, http.StatusForbidden, requestlog.ActionRateLimited)
banned := proxy.LedgerOf(server).Bans(netip.MustParsePrefix(client + "/32"))
if len(banned) != 2 || banned[0].Notes.Refused != 3 {
t.Errorf("bans %+v, want two, the first with 3 requests refused", banned)
}
}
func TestBanCoversTheClientsNetblock(t *testing.T) {
t.Parallel()
// In the IPv4 cases, client breaks the limit; these two are next to it.
const (
allowed = "203.0.113.60" // in SWWAF_ALLOW_NETS
exempt = "203.0.113.50" // in SWWAF_RATE_LIMIT_EXEMPT_NETS
)
for _, tc := range []struct {
name string
env map[string]string
breaker string // the client that breaks the limit
refused []string
let []string // let through
}{
{
"an IPv4 address, by default", nil, client,
nil, []string{otherClient, exempt},
},
{
"the IPv4 netblock SWWAF_BAN_SCOPE_V4_PREFIX sets",
map[string]string{banScopeV4Prefix: "24"}, client,
[]string{otherClient, exempt}, []string{"203.0.112.9", allowed},
},
{
"an IPv6 /64", nil, "2001:db8:5::1",
[]string{"2001:db8:5::ffff:1"}, []string{"2001:db8:5:1::1"},
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
env := map[string]string{
rateLimitPerMinute: "1",
allowNets: allowed,
rateLimitExemptNets: exempt,
}
maps.Copy(env, tc.env)
s, _, _ := startWithClock(t, "", env)
s.get(tc.breaker, http.StatusOK, requestlog.ActionForward)
s.get(tc.breaker, http.StatusForbidden, requestlog.ActionRateLimited)
for _, sent := range tc.refused {
s.get(sent, http.StatusForbidden, requestlog.ActionBanned)
}
for _, sent := range tc.let {
s.get(sent, http.StatusOK, requestlog.ActionForward)
}
})
}
}
func TestBannedClientIsRefusedBeforeItsCountryIsLookedUp(t *testing.T) {
t.Parallel()
geojsURL, asked := startGeoJS(t)
s, _, _ := startWithClock(t, geojsURL, map[string]string{
rateLimitPerMinute: "1",
banScopeV4Prefix: "24",
deniedCountries: "kp",
})
// fromDE's ban covers otherClient, which is refused unasked about.
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited)
line := s.get(otherClient, http.StatusForbidden, requestlog.ActionBanned)
if line.Country != "" {
t.Errorf("log line has country %q, want none", line.Country)
}
if !slices.Equal(asked(), []string{fromDE}) {
t.Errorf("GeoJS was asked about %v, want %s alone", asked(), fromDE)
}
}
func TestBanResponseAnswersEveryRefusalButTheSizeLimits(t *testing.T) {
t.Parallel()
const denied = "192.0.2.50" // in SWWAF_DENY_NETS
for _, tc := range []struct {
setting string // "" leaves SWWAF_BAN_RESPONSE at its default
status int // 0 is the connection closed without an answer
}{
{"", http.StatusForbidden},
{"403", http.StatusForbidden},
{"429", http.StatusTooManyRequests},
{"close", 0},
} {
t.Run(banResponse+"="+tc.setting, func(t *testing.T) {
t.Parallel()
geojsURL, _ := startGeoJS(t)
env := map[string]string{
rateLimitPerMinute: "1",
denyNets: denied,
deniedCountries: "kp",
}
if tc.setting != "" {
env[banResponse] = tc.setting
}
s, _, _ := startWithClock(t, geojsURL, env)
s.get(denied, tc.status, requestlog.ActionDenied)
s.get(fromKP, tc.status, requestlog.ActionCountryDenied)
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
s.get(fromDE, tc.status, requestlog.ActionRateLimited)
s.get(fromDE, tc.status, requestlog.ActionBanned)
})
}
}
func TestBanNotes(t *testing.T) {
t.Parallel()
geojsURL, _ := startGeoJS(t)
s, clk, server := startWithClock(t, geojsURL, map[string]string{
rateLimitPerMinute: "1",
deniedCountries: "kp",
})
start := clk.Now()
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
s.request(fromDE, "/repo/commits?page=2",
http.StatusForbidden, requestlog.ActionRateLimited)
s.get(fromDE, http.StatusForbidden, requestlog.ActionBanned)
s.get(fromDE, http.StatusForbidden, requestlog.ActionBanned)
netblock := netip.MustParsePrefix(fromDE + "/32")
want := bans.Ban{
Netblock: netblock,
Start: start,
Expires: start.Add(time.Hour),
Notes: bans.Notes{
Country: "DE",
Limit: 1,
Window: minute,
Count: 2,
Request: bans.Request{
Time: start,
Method: http.MethodGet,
Host: appHost,
Path: "/repo/commits?page=2",
Status: http.StatusForbidden,
UserAgent: userAgent,
},
Refused: 2,
EarlierBans: 0,
},
}
ledger := proxy.LedgerOf(server)
got := ledger.Bans(netblock)
if len(got) != 1 || got[0] != want {
t.Fatalf("bans\n%+v\nwant\n%+v", got, want)
}
// The next ban counts this one among the earlier.
clk.advance(time.Hour)
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited)
got = ledger.Bans(netblock)
if len(got) != 2 || got[1].Notes.EarlierBans != 1 {
t.Errorf("bans %+v, want two, the second with one earlier ban", got)
}
}
func TestMaxBansDropsTheBanOfTheNetblockSeenLongestAgo(t *testing.T) {
t.Parallel()
s, _, _ := startWithClock(t, "", map[string]string{
rateLimitPerMinute: "1",
maxBans: "1",
})
// One ban is held, so otherClient's ban drops client's.
s.get(client, http.StatusOK, requestlog.ActionForward)
s.get(client, http.StatusForbidden, requestlog.ActionRateLimited)
s.get(otherClient, http.StatusOK, requestlog.ActionForward)
s.get(otherClient, http.StatusForbidden, requestlog.ActionRateLimited)
s.get(client, http.StatusOK, requestlog.ActionForward)
s.get(otherClient, http.StatusForbidden, requestlog.ActionBanned)
}
// clock is the time a test sets, by which smallwebwaf counts requests and
// makes bans.
type clock struct {
mu sync.Mutex
now time.Time
}
// Now tells the time.
func (c *clock) Now() time.Time {
c.mu.Lock()
defer c.mu.Unlock()
return c.now
}
// advance moves the clock on by d.
func (c *clock) advance(d time.Duration) {
c.mu.Lock()
defer c.mu.Unlock()
c.now = c.now.Add(d)
}
// startWithClock starts smallwebwaf in front of an app that answers 200,
// with the settings in env on top of trusting localhost's
// X-Forwarded-For, clients' countries looked up at geojsURL, and a clock
// set to midnight, the start of a bucket in every window.
func startWithClock(
t *testing.T, geojsURL string, env map[string]string,
) (*sender, *clock, *http.Server) {
t.Helper()
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
clk := &clock{now: time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)}
settings := map[string]string{trustedProxies: trustLocalhost}
maps.Copy(settings, env)
addr, out, server := startProxyWithClock(t, app.URL, geojsURL, clk.Now, settings)
return &sender{t: t, addr: addr, out: out}, clk, server
}
// sender sends requests to smallwebwaf one after another, each on a
// connection of its own, and checks each one's answer and log line. They
// must be the only requests smallwebwaf is sent, since the log lines are
// matched to them in order.
type sender struct {
t *testing.T
addr string
out *output
sent int
}
// get sends a GET request for / from the client at from.
func (s *sender) get(from string, status int, action string) logLine {
s.t.Helper()
return s.request(from, "/", status, action)
}
// request sends a GET request for path from the client at from, as
// X-Forwarded-For names it, and checks that its answer and its log line
// have status, 0 for the connection closed without an answer, and that
// the line has action. It returns the log line.
func (s *sender) request(from, path string, status int, action string) logLine {
s.t.Helper()
conn := dial(s.t, s.addr)
send(s.t, conn, "GET "+path+" HTTP/1.1\r\nHost: "+appHost+
"\r\nUser-Agent: "+userAgent+"\r\n"+forwardedFor+": "+from+"\r\n\r\n")
err := conn.SetReadDeadline(time.Now().Add(waitLimit))
if err != nil {
s.t.Fatalf("set read deadline: %v", err)
}
got := 0
res, err := http.ReadResponse(bufio.NewReader(conn), nil)
switch {
case err == nil:
got = readAnswer(res).status
case !errors.Is(err, io.ErrUnexpectedEOF):
s.t.Fatalf("read response: %v", err)
}
_ = conn.Close()
if got != status {
s.t.Errorf("request %d, from %s: status %d, want %d", s.sent+1, from, got,
status)
}
line := s.out.requestLines(s.t, s.sent+1)[s.sent]
s.sent++
wantLine(s.t, line, status, action)
return line
}
+1 -1
View File
@@ -139,7 +139,7 @@ func requestWithHeaders(
ForwardedFor: r.Header.Get(forwardedFor), ForwardedFor: r.Header.Get(forwardedFor),
ForwardedHost: r.Header.Get("X-Forwarded-Host"), ForwardedHost: r.Header.Get("X-Forwarded-Host"),
ForwardedProto: r.Header.Get("X-Forwarded-Proto"), ForwardedProto: r.Header.Get("X-Forwarded-Proto"),
RealIP: r.Header.Get("X-Real-Ip"), RealIP: r.Header.Get("X-Real-IP"),
}) })
}) })
addr, out := startProxy(t, app.URL, env) addr, out := startProxy(t, app.URL, env)
+15
View File
@@ -0,0 +1,15 @@
package proxy
import (
"net/http"
"sneak.berlin/go/smallwebwaf/internal/bans"
)
// LedgerOf returns the ban ledger of a server New returned, so that the
// tests can read the bans' notes.
func LedgerOf(server *http.Server) *bans.Ledger {
h, _ := server.Handler.(*handler)
return h.ledger
}
+13
View File
@@ -10,6 +10,7 @@ import (
"net/http" "net/http"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/lookup" "sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
@@ -49,6 +50,9 @@ type Params struct {
// GeoJSURL is where clients' countries are looked up, normally // GeoJSURL is where clients' countries are looked up, normally
// lookup.URL. GeoJS is asked only while a country list is set. // lookup.URL. GeoJS is asked only while a country list is set.
GeoJSURL string GeoJSURL string
// Now tells the time by which requests are counted for the rate
// limits and bans are made and run out, normally time.Now.
Now func() time.Time
} }
// New returns the server smallwebwaf runs: each request it reads passes // New returns the server smallwebwaf runs: each request it reads passes
@@ -67,11 +71,18 @@ func New(params Params) *http.Server {
processLog: params.ProcessLog, processLog: params.ProcessLog,
errorLog: errorLog, errorLog: errorLog,
transport: newTransport(), transport: newTransport(),
now: params.Now,
limiter: ratelimit.New(ratelimit.Limits{ limiter: ratelimit.New(ratelimit.Limits{
PerMinute: params.Config.RateLimitPerMinute, PerMinute: params.Config.RateLimitPerMinute,
PerHour: params.Config.RateLimitPerHour, PerHour: params.Config.RateLimitPerHour,
PerDay: params.Config.RateLimitPerDay, PerDay: params.Config.RateLimitPerDay,
}), }),
ledger: bans.New(bans.Rules{
LimitBanDuration: params.Config.LimitBanDuration,
LimitBanRepeatWindow: params.Config.LimitBanRepeatWindow,
MaxBanDuration: params.Config.MaxBanDuration,
MaxBans: params.Config.MaxBans,
}),
geojs: lookup.New(lookup.Params{ geojs: lookup.New(lookup.Params{
URL: params.GeoJSURL, URL: params.GeoJSURL,
Now: time.Now, Now: time.Now,
@@ -93,7 +104,9 @@ type handler struct {
processLog *slog.Logger processLog *slog.Logger
errorLog *log.Logger errorLog *log.Logger
transport http.RoundTripper transport http.RoundTripper
now func() time.Time
limiter *ratelimit.Limiter limiter *ratelimit.Limiter
ledger *bans.Ledger
geojs *lookup.GeoJS geojs *lookup.GeoJS
} }
+22 -1
View File
@@ -55,8 +55,15 @@ const (
rateLimitExemptNets = "SWWAF_RATE_LIMIT_EXEMPT_NETS" rateLimitExemptNets = "SWWAF_RATE_LIMIT_EXEMPT_NETS"
denyNets = "SWWAF_DENY_NETS" denyNets = "SWWAF_DENY_NETS"
rateLimitPerMinute = "SWWAF_RATE_LIMIT_PER_MINUTE" rateLimitPerMinute = "SWWAF_RATE_LIMIT_PER_MINUTE"
rateLimitPerDay = "SWWAF_RATE_LIMIT_PER_DAY"
deniedCountries = "SWWAF_DENIED_COUNTRIES" deniedCountries = "SWWAF_DENIED_COUNTRIES"
allowedCountries = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES" allowedCountries = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES"
banResponse = "SWWAF_BAN_RESPONSE"
limitBanDuration = "SWWAF_LIMIT_BAN_DURATION"
limitBanRepeatWindow = "SWWAF_LIMIT_BAN_REPEAT_WINDOW"
maxBanDuration = "SWWAF_MAX_BAN_DURATION"
maxBans = "SWWAF_MAX_BANS"
banScopeV4Prefix = "SWWAF_BAN_SCOPE_V4_PREFIX"
) )
// output collects what smallwebwaf writes on stdout. // output collects what smallwebwaf writes on stdout.
@@ -181,6 +188,19 @@ func startProxyWithGeoJS(
) (string, *output) { ) (string, *output) {
t.Helper() t.Helper()
addr, out, _ := startProxyWithClock(t, appURL, geojsURL, time.Now, env)
return addr, out
}
// startProxyWithClock is startProxyWithGeoJS with requests counted and
// bans made by the time now tells, and returns the server as well.
func startProxyWithClock(
t *testing.T, appURL, geojsURL string, now func() time.Time,
env map[string]string,
) (string, *output, *http.Server) {
t.Helper()
settings := map[string]string{"SWWAF_UPSTREAM_URL": appURL} settings := map[string]string{"SWWAF_UPSTREAM_URL": appURL}
maps.Copy(settings, env) maps.Copy(settings, env)
@@ -199,6 +219,7 @@ func startProxyWithGeoJS(
RequestLog: out, RequestLog: out,
ProcessLog: requestlog.NewProcessLogger(out), ProcessLog: requestlog.NewProcessLogger(out),
GeoJSURL: geojsURL, GeoJSURL: geojsURL,
Now: now,
}) })
listener, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", localhost+":0") listener, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", localhost+":0")
@@ -214,7 +235,7 @@ func startProxyWithGeoJS(
_ = server.Close() _ = server.Close()
}) })
return listener.Addr().String(), out return listener.Addr().String(), out, server
} }
// newClient returns an HTTP client that sends requests as they are made, // newClient returns an HTTP client that sends requests as they are made,
+12 -8
View File
@@ -8,7 +8,11 @@ import (
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
func TestRateLimitRefusesWith429BeforeTheApp(t *testing.T) { // minute is the window of SWWAF_RATE_LIMIT_PER_MINUTE, as a log line's
// limit_hit names it.
const minute = "minute"
func TestRateLimitRefusesBeforeTheApp(t *testing.T) {
t.Parallel() t.Parallel()
var calls atomic.Int32 var calls atomic.Int32
@@ -24,19 +28,19 @@ func TestRateLimitRefusesWith429BeforeTheApp(t *testing.T) {
const otherClient = "203.0.113.10" const otherClient = "203.0.113.10"
// With a limit of one request a minute, a client's second request is // With a limit of one request a minute, a client's second request is
// refused. A client is one IPv4 address, or one IPv6 /64; an IPv4 // refused, with 403 by default. A client is one IPv4 address, or one
// address in IPv6 form is that IPv4 address. // IPv6 /64; an IPv4 address in IPv6 form is that IPv4 address.
requests := []struct { requests := []struct {
client string // as X-Forwarded-For names it client string // as X-Forwarded-For names it
logged string // as the log line's client_ip names it logged string // as the log line's client_ip names it
want int want int
}{ }{
{client, client, http.StatusOK}, {client, client, http.StatusOK},
{client, client, http.StatusTooManyRequests}, {client, client, http.StatusForbidden},
{otherClient, otherClient, http.StatusOK}, {otherClient, otherClient, http.StatusOK},
{"::ffff:" + otherClient, otherClient, http.StatusTooManyRequests}, {"::ffff:" + otherClient, otherClient, http.StatusForbidden},
{"2001:db8::1", "2001:db8::1", http.StatusOK}, {"2001:db8::1", "2001:db8::1", http.StatusOK},
{"2001:db8::8000:0:0:1", "2001:db8::8000:0:0:1", http.StatusTooManyRequests}, {"2001:db8::8000:0:0:1", "2001:db8::8000:0:0:1", http.StatusForbidden},
{"2001:db8:0:1::1", "2001:db8:0:1::1", http.StatusOK}, {"2001:db8:0:1::1", "2001:db8:0:1::1", http.StatusOK},
} }
@@ -53,9 +57,9 @@ func TestRateLimitRefusesWith429BeforeTheApp(t *testing.T) {
if sent.want == http.StatusOK { if sent.want == http.StatusOK {
wantLine(t, line, http.StatusOK, requestlog.ActionForward) wantLine(t, line, http.StatusOK, requestlog.ActionForward)
} else { } else {
wantLine(t, line, http.StatusTooManyRequests, requestlog.ActionRateLimited) wantLine(t, line, http.StatusForbidden, requestlog.ActionRateLimited)
if line.LimitHit != "minute" { if line.LimitHit != minute {
t.Errorf("log line has limit_hit %q, want minute", line.LimitHit) t.Errorf("log line has limit_hit %q, want minute", line.LimitHit)
} }
} }
+24 -22
View File
@@ -21,7 +21,8 @@ const flushAfterEachWrite time.Duration = -1
// refusal is smallwebwaf refusing a request, or refusing to go on with it: // refusal is smallwebwaf refusing a request, or refusing to go on with it:
// the status the client is answered if the response has not started yet, // the status the client is answered if the response has not started yet,
// and the action the log line names. // 0 to close the connection without an answer, and the action the log
// line names.
type refusal struct { type refusal struct {
status int status int
action string action string
@@ -105,39 +106,33 @@ func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request {
// is known, before its body is read or anything reaches the app. It // is known, before its body is read or anything reaches the app. It
// returns nil to let the request through. A client in SWWAF_ALLOW_NETS // returns nil to let the request through. A client in SWWAF_ALLOW_NETS
// skips every check but the size limit. For any other client, // skips every check but the size limit. For any other client,
// SWWAF_DENY_NETS comes first, so that a client it refuses is not looked // SWWAF_DENY_NETS comes first, then a ban on its netblock, so that a
// up, and then the country lists; a request either refuses is not counted // client either refuses is not looked up, and then the country lists; a
// for the rate limits. Then come the rate limits, unless the client is in // request any of them refuses is not counted for the rate limits. Then
// come the rate limits, unless the client is in
// SWWAF_RATE_LIMIT_EXEMPT_NETS, so that every other request is counted, // SWWAF_RATE_LIMIT_EXEMPT_NETS, so that every other request is counted,
// one refused for its size too. ctx is the request's own context. // one refused for its size too. Every refusal but the size limit's is
// answered with SWWAF_BAN_RESPONSE. ctx is the request's own context.
func (rq *request) check(ctx context.Context) *refusal { func (rq *request) check(ctx context.Context) *refusal {
cfg := rq.h.config cfg := rq.h.config
allowed := isInside(rq.client, cfg.AllowNets) allowed := isInside(rq.client, cfg.AllowNets)
exempt := isInside(rq.client, cfg.RateLimitExemptNets)
now := rq.h.now()
if !allowed && isInside(rq.client, cfg.DenyNets) { if !allowed && isInside(rq.client, cfg.DenyNets) {
return &refusal{ return rq.banResponse(requestlog.ActionDenied)
status: http.StatusForbidden,
action: requestlog.ActionDenied,
} }
if !allowed && rq.banned(now) {
return rq.banResponse(requestlog.ActionBanned)
} }
if !allowed && rq.countryDenied(ctx) { if !allowed && rq.countryDenied(ctx) {
return &refusal{ return rq.banResponse(requestlog.ActionCountryDenied)
status: http.StatusForbidden,
action: requestlog.ActionCountryDenied,
}
} }
if !allowed && !isInside(rq.client, cfg.RateLimitExemptNets) { if !allowed && !exempt && rq.limitBroken(now) {
limitHit := rq.h.limiter.Count(clientGroup(rq.client), rq.start) return rq.banResponse(requestlog.ActionRateLimited)
if limitHit != "" {
rq.line.LimitHit = limitHit
return &refusal{
status: http.StatusTooManyRequests,
action: requestlog.ActionRateLimited,
}
}
} }
maxBytes := cfg.RequestMaxBytes maxBytes := cfg.RequestMaxBytes
@@ -250,6 +245,13 @@ func (rq *request) answer(r refusal) {
return // too late to answer: the connection can only be cut return // too late to answer: the connection can only be cut
} }
if r.status == 0 {
// SWWAF_BAN_RESPONSE is close. This panic has Go's server close
// the connection without an answer, and log nothing; the log line
// is still written as the handler returns.
panic(http.ErrAbortHandler)
}
// A client found too slow is read no more; any other may go on // A client found too slow is read no more; any other may go on
// sending until its time is up, so that Go's server can read the // sending until its time is up, so that Go's server can read the
// rest of the body and end the request cleanly. // rest of the body and end the request cleanly.
+3 -3
View File
@@ -78,7 +78,7 @@ func TestRequestFromAllowNetsIsNotCounted(t *testing.T) {
{listedAddr, http.StatusOK, requestlog.ActionForward}, {listedAddr, http.StatusOK, requestlog.ActionForward},
{listedAddr, http.StatusOK, requestlog.ActionForward}, {listedAddr, http.StatusOK, requestlog.ActionForward},
{unlistedAddr, http.StatusOK, requestlog.ActionForward}, {unlistedAddr, http.StatusOK, requestlog.ActionForward},
{unlistedAddr, http.StatusTooManyRequests, requestlog.ActionRateLimited}, {unlistedAddr, http.StatusForbidden, requestlog.ActionRateLimited},
}) })
} }
@@ -133,7 +133,7 @@ func TestRequestRefusedByDenyNetsIsNotCounted(t *testing.T) {
{listedAddr, http.StatusForbidden, requestlog.ActionDenied}, {listedAddr, http.StatusForbidden, requestlog.ActionDenied},
{listedAddr, http.StatusForbidden, requestlog.ActionDenied}, {listedAddr, http.StatusForbidden, requestlog.ActionDenied},
{unlistedAddr, http.StatusOK, requestlog.ActionForward}, {unlistedAddr, http.StatusOK, requestlog.ActionForward},
{unlistedAddr, http.StatusTooManyRequests, requestlog.ActionRateLimited}, {unlistedAddr, http.StatusForbidden, requestlog.ActionRateLimited},
}) })
} }
@@ -156,7 +156,7 @@ func TestRateLimitExemptNetsAreNeitherCountedNorRefused(t *testing.T) {
{listedAddr, http.StatusOK, requestlog.ActionForward}, {listedAddr, http.StatusOK, requestlog.ActionForward},
{listedAddr, http.StatusOK, requestlog.ActionForward}, {listedAddr, http.StatusOK, requestlog.ActionForward},
{unlistedAddr, http.StatusOK, requestlog.ActionForward}, {unlistedAddr, http.StatusOK, requestlog.ActionForward},
{unlistedAddr, http.StatusTooManyRequests, requestlog.ActionRateLimited}, {unlistedAddr, http.StatusForbidden, requestlog.ActionRateLimited},
{fromKP, http.StatusForbidden, requestlog.ActionCountryDenied}, {fromKP, http.StatusForbidden, requestlog.ActionCountryDenied},
}) })
} }
+27 -8
View File
@@ -54,11 +54,22 @@ func New(limits Limits) *Limiter {
} }
} }
// Hit is a request that takes a client over a rate limit.
type Hit struct {
// Window is "minute", "hour" or "day".
Window string
// Limit is the window's limit.
Limit int64
// Requests is the client's requests counted in the window, this one
// included.
Requests float64
}
// Count counts a request from client at now, in every window, whether or // Count counts a request from client at now, in every window, whether or
// not it is refused. It returns the window whose limit the request takes // not it is refused. It reports whether the request takes the client over
// the client over, "minute", "hour" or "day", the shortest if it is over // a limit, and the window whose limit it goes over, the shortest if it is
// several, or "" if it is within every limit. // over several.
func (l *Limiter) Count(client netip.Prefix, now time.Time) string { func (l *Limiter) Count(client netip.Prefix, now time.Time) (Hit, bool) {
l.mu.Lock() l.mu.Lock()
defer l.mu.Unlock() defer l.mu.Unlock()
@@ -68,16 +79,24 @@ func (l *Limiter) Count(client netip.Prefix, now time.Time) string {
l.clients.Add(client, counts) l.clients.Add(client, counts)
} }
limitHit := "" var hit Hit
for i, w := range l.windows { for i, w := range l.windows {
requests := counts[i].add(now, w.length) requests := counts[i].add(now, w.length)
if limitHit == "" && w.limit > 0 && requests > float64(w.limit) { if hit.Window == "" && w.limit > 0 && requests > float64(w.limit) {
limitHit = w.name hit = Hit{Window: w.name, Limit: w.limit, Requests: requests}
} }
} }
return limitHit return hit, hit.Window != ""
}
// Reset sets client's counts in every window back to zero.
func (l *Limiter) Reset(client netip.Prefix) {
l.mu.Lock()
defer l.mu.Unlock()
l.clients.Remove(client)
} }
// window is a length of time over which requests are counted, and the // window is a length of time over which requests are counted, and the
+49 -3
View File
@@ -54,6 +54,52 @@ func TestEachWindowRefusesAtItsLimitAndLetsTheClientBack(t *testing.T) {
} }
} }
func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit, PerHour: limit})
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
for range limit {
_, over := limiter.Count(client, start)
if over {
t.Fatal("a request within the limit is over it")
}
}
// Over both limits; the minute's is named, with the four requests.
hit, over := limiter.Count(client, start)
want := ratelimit.Hit{Window: minute, Limit: limit, Requests: limit + 1}
if !over || hit != want {
t.Errorf("request over the limit gives %+v and %t, want %+v and true",
hit, over, want)
}
}
func TestResetSetsTheCountsBackToZero(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit, PerDay: limit})
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
for range limit {
wantCount(t, limiter, client, start, "")
}
wantCount(t, limiter, client, start, minute)
limiter.Reset(client)
// At the same moment, the client has its whole allowance again.
for range limit {
wantCount(t, limiter, client, start, "")
}
wantCount(t, limiter, client, start, minute)
}
func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) { func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) {
t.Parallel() t.Parallel()
@@ -192,9 +238,9 @@ func wantCount(
) { ) {
t.Helper() t.Helper()
got := limiter.Count(client, now) hit, _ := limiter.Count(client, now)
if got != want { if hit.Window != want {
t.Errorf("request from %s at %s is over %q, want %q", t.Errorf("request from %s at %s is over %q, want %q",
client, now.Format(time.RFC3339), got, want) client, now.Format(time.RFC3339), hit.Window, want)
} }
} }
+12 -1
View File
@@ -24,8 +24,10 @@ const (
// for, or whose answer could not be passed on. // for, or whose answer could not be passed on.
ActionUpstreamError = "upstream_error" ActionUpstreamError = "upstream_error"
// ActionRateLimited is a request refused because it took its client // ActionRateLimited is a request refused because it took its client
// over a rate limit, or came while the client was over one. // over a rate limit, which bans the client.
ActionRateLimited = "rate_limited" ActionRateLimited = "rate_limited"
// ActionBanned is a request refused because a ban covers its client.
ActionBanned = "banned"
// ActionDenied is a request refused because its client is in // ActionDenied is a request refused because its client is in
// SWWAF_DENY_NETS. // SWWAF_DENY_NETS.
ActionDenied = "denied" ActionDenied = "denied"
@@ -36,6 +38,10 @@ const (
ActionAdmin = "admin" ActionAdmin = "admin"
) )
// OffenceLimit is the offence a request line names for a request that
// broke a rate limit.
const OffenceLimit = "limit"
// timeLayout is RFC 3339 with milliseconds. // timeLayout is RFC 3339 with milliseconds.
const timeLayout = "2006-01-02T15:04:05.000Z07:00" const timeLayout = "2006-01-02T15:04:05.000Z07:00"
@@ -64,6 +70,11 @@ type Line struct {
// LimitHit is the window whose rate limit the request went over: // LimitHit is the window whose rate limit the request went over:
// minute, hour or day. // minute, hour or day.
LimitHit string `json:"limit_hit,omitempty"` LimitHit string `json:"limit_hit,omitempty"`
// Offence is the offence the request was held as, OffenceLimit.
Offence string `json:"offence,omitempty"`
// BanExpires is when the ban the request made, or was refused under,
// ends: a time, or "permanent".
BanExpires string `json:"ban_expires,omitempty"`
// Aborted is true when the client went away early. // Aborted is true when the client went away early.
Aborted bool `json:"aborted,omitempty"` Aborted bool `json:"aborted,omitempty"`
// DurationTotal and DurationUpstreamTotal are in milliseconds. // DurationTotal and DurationUpstreamTotal are in milliseconds.
+2 -1
View File
@@ -50,7 +50,8 @@ func TestWriteWritesOneJSONLineMarkedRequest(t *testing.T) {
} }
unset := []string{ unset := []string{
"upstream_status", "limit_hit", "aborted", "duration_upstream_total", "upstream_status", "limit_hit", "offence", "ban_expires", "aborted",
"duration_upstream_total",
} }
for _, name := range unset { for _, name := range unset {
_, present := fields[name] _, present := fields[name]
+1
View File
@@ -80,6 +80,7 @@ func Run(ctx context.Context, params Params) int {
RequestLog: params.Stdout, RequestLog: params.Stdout,
ProcessLog: processLog, ProcessLog: processLog,
GeoJSURL: lookup.URL, GeoJSURL: lookup.URL,
Now: time.Now,
}) })
processLog.Info("starting", processLog.Info("starting",
+6
View File
@@ -194,6 +194,12 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL string) {
"SWWAF_RATE_LIMIT_PER_DAY": "50000", "SWWAF_RATE_LIMIT_PER_DAY": "50000",
"SWWAF_DENIED_COUNTRIES": "", "SWWAF_DENIED_COUNTRIES": "",
"SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES": "", "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES": "",
"SWWAF_BAN_RESPONSE": "403",
"SWWAF_LIMIT_BAN_DURATION": "1h",
"SWWAF_LIMIT_BAN_REPEAT_WINDOW": "24h",
"SWWAF_MAX_BAN_DURATION": "7d",
"SWWAF_MAX_BANS": "5000",
"SWWAF_BAN_SCOPE_V4_PREFIX": "32",
} }
for name, value := range want { for name, value := range want {
+2 -3
View File
@@ -16,9 +16,8 @@ main() {
"$SCRIPT_DIR/check" "$SCRIPT_DIR/check"
# Own line: a failing command substitution inside an argument does # Own line: a failing command substitution inside an argument does
# not trip `set -e`, so the inline form degrades silently to an # not trip `set -e`, so the inline form degrades silently to an
# empty constant. VERSION is computed here because .dockerignore # empty constant. The VERSION build argument takes precedence over
# excludes .git, so `git describe` in a build stage yields an empty # the version a build stage derives from the .git in the context.
# version without failing.
version="$(git describe --tags --always --dirty 2>/dev/null || true)" version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown" [ -n "$version" ] || version="unknown"
docker build --no-cache \ docker build --no-cache \
+2 -3
View File
@@ -12,9 +12,8 @@ main() {
cd "$ROOT" cd "$ROOT"
# Own line: a failing command substitution inside an argument does # Own line: a failing command substitution inside an argument does
# not trip `set -e`, so the inline form degrades silently to an # not trip `set -e`, so the inline form degrades silently to an
# empty constant. VERSION is computed here because .dockerignore # empty constant. The VERSION build argument takes precedence over
# excludes .git, so `git describe` in a build stage yields an empty # the version a build stage derives from the .git in the context.
# version without failing.
version="$(git describe --tags --always --dirty 2>/dev/null || true)" version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown" [ -n "$version" ] || version="unknown"
docker build --no-cache \ docker build --no-cache \