Ban the netblock of a client that breaks a rate limit, in memory (closes #18)
check / check (push) Successful in 4m11s

A request over a rate limit is refused with SWWAF_BAN_RESPONSE and bans
the client's netblock: an hour at first, three times the last ban when
broken again within a day of its end, permanent past seven days. The
ban ledger in internal/bans is checked after the static lists and
before the lookup, and the requests it refuses are not counted. A ban
resets the client's counters and carries notes holding the request
that broke the limit, as SPEC.md now says. At most SWWAF_MAX_BANS are
held. SWWAF_BAN_RESPONSE also answers SWWAF_DENY_NETS and the country
lists.

Judgement call: the six ban settings cannot be off.
Judgement call: a permanent ban's ban_expires is "permanent".

Model: opus-5-5
This commit is contained in:
2026-10-06 03:12:48 +00:00
parent 0f85c9ae07
commit cdaa2a0aca
20 changed files with 1522 additions and 127 deletions
+110 -67
View File
@@ -13,16 +13,17 @@ JSON log line for every request.
Status: the first two milestones are built Status: the first two milestones are built
(https://git.eeqj.de/sneak/smallwebwaf/issues/13 and (https://git.eeqj.de/sneak/smallwebwaf/issues/13 and
https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are two parts of https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are three parts of
milestone 3: the static lists, which come next in the build order, and the milestone 3: the static lists and the bans that broken rate limits lead to,
header size and the idle time as settings, which come last in it. `smallwebwaf` which come next in the build order, and the header size and the idle time as
passes each request to the app and the app's answer back, unchanged, within its settings, which come last in it. `smallwebwaf` passes each request to the app
timeouts and size limits, works out each client's address, refuses a client that and the app's answer back, unchanged, within its timeouts and size limits, works
sends too many requests, comes from a country you refuse or from a network you out each client's address, bans a client that sends too many requests, refuses a
refuse, lets the networks you choose through, and writes a JSON log line for client that comes from a country you refuse or from a network you refuse, lets
every request. It comes as the image the app's own image is built on. The rest the networks you choose through, and writes a JSON log line for every request.
of the design comes after that, in the order of the build order in It comes as the image the app's own image is built on. The rest of the design
[`SPEC.md`](SPEC.md). The survey of existing tools that led to the design is in comes after that, in the order of the build order in [`SPEC.md`](SPEC.md). The
survey of existing tools that led to the design is in
[`EVALUATION.md`](EVALUATION.md). [`EVALUATION.md`](EVALUATION.md).
## Getting started ## Getting started
@@ -71,31 +72,48 @@ and `make run` builds and runs it, listening on port 8080 in front of an app at
to take what it had. Once the response has started, a limit can only cut the to take what it had. Once the response has started, a limit can only cut the
connection. connection.
- Counts each client's requests over a minute, an hour and a day. A request that - Counts each client's requests over a minute, an hour and a day. A request that
takes the client over one of the rate limits below is refused with `429` takes the client over one of the rate limits below is refused with
before anything reaches the app, and so is each request after it until the `SWWAF_BAN_RESPONSE`, `403` by default, before anything reaches the app, and
client is back under every limit. A client is one IPv4 address, or one IPv6 bans the client. A client is one IPv4 address, or one IPv6 /64, since one
/64, since one abuser usually holds a whole /64. Refused requests count too, abuser usually holds a whole /64. Each window is counted in two fixed buckets,
so a client that keeps sending too fast stays refused until it slows down. the earlier one weighted by how much of it the window still covers. At most
Each window is counted in two fixed buckets, the earlier one weighted by how 20,000 clients are kept, the least recently seen dropped first, and only in
much of it the window still covers. At most 20,000 clients are kept, the least memory: a restart starts every client afresh.
recently seen dropped first, and only in memory: a restart starts every client - Bans a client that breaks a rate limit, as "Bans" in [`SPEC.md`](SPEC.md)
afresh. describes: the first ban lasts an hour, and a limit broken again within a day
- Refuses a request from a country you refuse with `403`, as soon as the of a ban ending bans for three times as long as that ban, so 1, 3, 9, 27 and
client's country is known and before its body is read; such a request is not 81 hours; a ban that would last longer than seven days is permanent instead. A
counted for the rate limits. While one of the country lists below is set, each ban covers the client's netblock: its IPv4 address, or the netblock around it
client's country is looked up through GeoJS (see "Country and AS number that `SWWAF_BAN_SCOPE_V4_PREFIX` sets, or its IPv6 /64. While it lasts, every
lookup" below); with neither set, no visitor's address leaves the host. A request from the netblock is refused with `SWWAF_BAN_RESPONSE` after the
client on a private, loopback or link-local address has no country and is static lists and before the country lists, so the client is not looked up, and
is not counted for the rate limits. A ban sets the client's counters back to
zero. Each ban carries notes for deciding whether to lift it: the limit, its
window and the requests counted in it, the request that broke it, the client's
country when it was looked up, how many requests the ban has refused, and how
many bans the netblock had before. At most `SWWAF_MAX_BANS` bans are kept,
past, active and permanent; past that, the earliest ban of the netblock that
has gone longest without a request is dropped first. Bans and their notes are
kept in memory only, so a restart lifts every ban, and nothing shows them yet:
`bans.json`, which shows them and lets you lift a ban, comes with the state
files (https://git.eeqj.de/sneak/smallwebwaf/issues/17).
- Refuses a request from a country you refuse with `SWWAF_BAN_RESPONSE`, as soon
as the client's country is known and before its body is read; such a request
is not counted for the rate limits. While one of the country lists below is
set, each client's country is looked up through GeoJS (see "Country and AS
number lookup" below); with neither set, no visitor's address leaves the host.
A client on a private, loopback or link-local address has no country and is
never looked up: `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses it unless it is never looked up: `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses it unless it is
in `SWWAF_ALLOW_NETS`, and `SWWAF_DENIED_COUNTRIES` does not refuse it. in `SWWAF_ALLOW_NETS`, and `SWWAF_DENIED_COUNTRIES` does not refuse it.
- Checks the client's own address against the static lists, the three netblock - Checks the client's own address against the static lists, the three netblock
settings below, before anything else, its country included. A client in settings below, before anything else, its country included. A client in
`SWWAF_ALLOW_NETS` skips the country lists and the rate limits, and is not `SWWAF_ALLOW_NETS` skips bans, the country lists and the rate limits, and is
looked up; the timeouts and size limits still apply. A client in not looked up; the timeouts and size limits still apply. A client in
`SWWAF_DENY_NETS` is refused with `403` before its body is read, and the `SWWAF_DENY_NETS` is refused with `SWWAF_BAN_RESPONSE` before its body is
request is not counted for the rate limits; an address in `SWWAF_ALLOW_NETS` read, and the request is not counted for the rate limits; an address in
too is let through. A client in `SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither `SWWAF_ALLOW_NETS` too is let through. A client in
counted nor refused by the rate limits; the country lists still apply to it. `SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither counted nor refused by the rate
limits; the country lists and bans still apply to it.
- Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any - Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any
check and without asking the app, for the image's health check. check and without asking the app, for the image's health check.
- Writes a line in the request log for each request (see "Request log" below). - Writes a line in the request log for each request (see "Request log" below).
@@ -133,8 +151,9 @@ it, and the effective settings are logged at start.
to send its whole answer, from the end of the request to the last byte. to send its whole answer, from the end of the request to the last byte.
- `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body. - `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body.
- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body. - `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body.
- `SWWAF_ALLOW_NETS` (default empty): netblocks whose clients skip the country - `SWWAF_ALLOW_NETS` (default empty): netblocks whose clients skip bans, the
lists and the rate limits, such as your monitoring or your own networks. country lists and the rate limits, such as your monitoring or your own
networks.
- `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients the - `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients the
rate limits do not apply to, such as a machine that talks to the app all day. rate limits do not apply to, such as a machine that talks to the app all day.
- `SWWAF_DENY_NETS` (default empty): netblocks whose clients are always refused. - `SWWAF_DENY_NETS` (default empty): netblocks whose clients are always refused.
@@ -149,6 +168,22 @@ it, and the effective settings are logged at start.
countries whose clients get through, for example `us,de`. A client whose countries whose clients get through, for example `us,de`. A client whose
country cannot be found is refused too, so that new clients are not let in country cannot be found is refused too, so that new clients are not let in
whenever GeoJS stops answering. whenever GeoJS stops answering.
- `SWWAF_BAN_RESPONSE` (default `403`): how a refused client is answered, one
that is banned, breaks a rate limit, is in `SWWAF_DENY_NETS` or comes from a
refused country: `403`, `429`, or `close` to close the connection without an
answer. Behind traefik, `close` does not leave the client unanswered: traefik
answers `502`, as it does whenever its backend drops a connection.
- `SWWAF_LIMIT_BAN_DURATION` (default `1h`): the ban for a first broken rate
limit.
- `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): a rate limit broken again
within this time after a ban ended bans for three times as long as that ban.
- `SWWAF_MAX_BAN_DURATION` (default `7d`): a ban that would be longer is
permanent instead.
- `SWWAF_MAX_BANS` (default `5000`): the most bans kept, past, active and
permanent.
- `SWWAF_BAN_SCOPE_V4_PREFIX` (default `32`): the length of the netblock around
an IPv4 client that a ban covers, such as `24` to ban the surrounding /24. An
IPv6 ban covers the client's /64.
Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are
bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024 bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024
@@ -157,8 +192,8 @@ and a bare address stands for itself alone. Countries are the two-letter codes
ISO 3166-1 assigns today, and `xk` for Kosovo, in either case (`de` and `DE` are ISO 3166-1 assigns today, and `xk` for Kosovo, in either case (`de` and `DE` are
the same); any other code, such as `nk` (North Korea is `kp`) or the withdrawn the same); any other code, such as `nk` (North Korea is `kp`) or the withdrawn
`su`, stops the start, and so does a code on both country lists. `off` switches `su`, stops the start, and so does a code on both country lists. `off` switches
a timeout, a size limit or a rate limit off; only a timeout, a size limit or a rate limit off;
`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES` cannot be off. `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES` and the ban settings cannot be off.
Several limits are fixed rather than settings. At most 20,000 clients are kept Several limits are fixed rather than settings. At most 20,000 clients are kept
for the rate limits, and an IPv6 client is counted by its /64. A new client for the rate limits, and an IPv6 client is counted by its /64. A new client
@@ -176,23 +211,27 @@ refused ones included:
- `time` is when the request arrived, in UTC. `peer_ip` is the TCP peer, - `time` is when the request arrived, in UTC. `peer_ip` is the TCP peer,
normally traefik. `path` and `query` are as the client sent them. normally traefik. `path` and `query` are as the client sent them.
- `country` is the client's country as GeoJS places it, and empty when it is not - `country` is the client's country as GeoJS places it. It is empty with neither
known: with neither country list set, for a client in `SWWAF_ALLOW_NETS` or country list set, for a client in `SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS`, for
`SWWAF_DENY_NETS`, for a client on a private, loopback or link-local address, a client on a private, loopback or link-local address, when GeoJS cannot place
and when GeoJS cannot place the client or has not answered in time. the client or has not answered in time, and for a request refused because a
ban covers its client, even when the client's country is known.
- `status` is what the client was sent, `0` if nothing was; `upstream_status` is - `status` is what the client was sent, `0` if nothing was; `upstream_status` is
what the app answered, and is left out when the app did not answer. what the app answered, and is left out when the app did not answer.
- `request_bytes` and `response_bytes` count body bytes. - `request_bytes` and `response_bytes` count body bytes.
- `action` is `forward` for a request passed to the app, `denied` for one - `action` is `forward` for a request passed to the app, `denied` for one
refused because its client is in `SWWAF_DENY_NETS`, `country_denied` for one refused because its client is in `SWWAF_DENY_NETS`, `banned` for one refused
refused for its client's country, `rate_limited` for one refused for a rate because a ban covers its client, `country_denied` for one refused for its
limit, `too_large` for a request or response over its size limit, `timed_out` client's country, `rate_limited` for one that broke a rate limit and banned
for one that ran out of time, `upstream_error` when the app could not be its client, `too_large` for a request or response over its size limit,
reached or its answer broke off, and `admin` for one `smallwebwaf` answered at `timed_out` for one that ran out of time, `upstream_error` when the app could
its own endpoint. not be reached or its answer broke off, and `admin` for one `smallwebwaf`
- `limit_hit` is there for a request refused for a rate limit, and names the answered at its own endpoint.
- `limit_hit` is there for a request that broke a rate limit, and names the
window whose limit it went over: `minute`, `hour` or `day`, the shortest if it window whose limit it went over: `minute`, `hour` or `day`, the shortest if it
went over several. went over several. `offence` is then `limit`.
- `ban_expires` is there for a request that made a ban or was refused under one,
and gives when the ban ends, in the same form as `time`, or `permanent`.
- `aborted` is there, and true, when the client went away early. - `aborted` is there, and true, when the client went away early.
- `duration_total` and `duration_upstream_total` are in milliseconds. - `duration_total` and `duration_upstream_total` are in milliseconds.
@@ -309,8 +348,8 @@ goes through the candidates one by one.
nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks
up the change. Nothing is read from disk while serving a request. The files up the change. Nothing is read from disk while serving a request. The files
come in milestone 3 or later (see the build order in [`SPEC.md`](SPEC.md)); come in milestone 3 or later (see the build order in [`SPEC.md`](SPEC.md));
until then the rate counters and the GeoJS answers are kept in memory only, until then the rate counters, the bans and the GeoJS answers are kept in
and a restart loses them. memory only, and a restart loses them.
- Health checks, the metrics, and listing, adding and lifting bans or asking why - Health checks, the metrics, and listing, adding and lifting bans or asking why
a given address was refused, all on the one port every request uses: under a given address was refused, all on the one port every request uses: under
`/_smallwebwaf/` on the app's own address, through traefik like any other `/_smallwebwaf/` on the app's own address, through traefik like any other
@@ -427,17 +466,18 @@ the metrics, failure behaviour and the build order.
So far `smallwebwaf` looks up only the country, only through GeoJS, and only So far `smallwebwaf` looks up only the country, only through GeoJS, and only
while `SWWAF_DENIED_COUNTRIES` or `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` is set: while `SWWAF_DENIED_COUNTRIES` or `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` is set:
then the address of every new visitor outside `SWWAF_ALLOW_NETS` and then the address of every new visitor is sent to GeoJS, except a visitor in
`SWWAF_DENY_NETS` is sent to GeoJS, and with neither set, none is. An IPv6 `SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS` and one refused because a ban covers its
visitor is asked about by the first address of its /64. A new visitor waits at netblock, and with neither set, none is. An IPv6 visitor is asked about by the
most a second for its answer, and without one counts as coming from an unknown first address of its /64. A new visitor waits at most a second for its answer,
country until the answer arrives. The addresses waiting are asked about and without one counts as coming from an unknown country until the answer
together, up to 200 in one request, one request at a time; at most 10,000 arrives. The addresses waiting are asked about together, up to 200 in one
visitors wait, and one more counts as coming from an unknown country until there request, one request at a time; at most 10,000 visitors wait, and one more
is room. While GeoJS fails, visitors with a kept answer are unaffected and new counts as coming from an unknown country until there is room. While GeoJS fails,
ones count as coming from an unknown country. GeoJS is then left alone for a visitors with a kept answer are unaffected and new ones count as coming from an
second, twice as long after each further failure up to five minutes, and asked unknown country. GeoJS is then left alone for a second, twice as long after each
again by the next request that needs it. further failure up to five minutes, and asked again by the next request that
needs it.
In the full design, `smallwebwaf` looks up the AS number and country of every In the full design, `smallwebwaf` looks up the AS number and country of every
client, for the request log, the metrics and the ban notes, and for the country client, for the request log, the metrics and the ban notes, and for the country
@@ -488,8 +528,10 @@ addresses are never sent to GeoJS.
standard library's `httputil.ReverseProxy` within the timeouts and size standard library's `httputil.ReverseProxy` within the timeouts and size
limits, and writes the request's log line. Its `check` method is where a limits, and writes the request's log line. Its `check` method is where a
request is refused before anything reaches the app: for `SWWAF_DENY_NETS`, for request is refused before anything reaches the app: for `SWWAF_DENY_NETS`, for
the country lists, for a rate limit, and for an announced body over the size a ban, for the country lists, for a rate limit, which bans the client, and for
limit. an announced body over the size limit.
- `internal/bans`: the ban ledger: each netblock's bans with their notes, how
long a new ban lasts, and which ban is dropped when `SWWAF_MAX_BANS` are held.
- `internal/lookup`: looks up each client's country through GeoJS, and keeps the - `internal/lookup`: looks up each client's country through GeoJS, and keeps the
answers. answers.
- `internal/ratelimit`: counts each client's requests and tells when one takes - `internal/ratelimit`: counts each client's requests and tells when one takes
@@ -503,8 +545,9 @@ addresses are never sent to GeoJS.
checks. checks.
Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the
table of clients to 20,000 and the GeoJS answers to 100,000, dropping the least table of clients to 20,000, the GeoJS answers to 100,000 and the banned
recently seen. The country codes are the list in `internal/config/config.go`. netblocks to `SWWAF_MAX_BANS`, dropping the least recently seen. The country
codes are the list in `internal/config/config.go`.
## Entrypoints ## Entrypoints
@@ -544,9 +587,9 @@ so that they run in minimal containers.
## TODO ## TODO
- The rest of milestone 3, from the bans that broken request limits lead to - The rest of milestone 3, after the bans that broken rate limits lead to and up
through the metrics endpoint, and the rest of the design, in the order of the to the metrics endpoint, and the rest of the design, in the order of the build
build order in [`SPEC.md`](SPEC.md). order in [`SPEC.md`](SPEC.md).
## Documents ## Documents
+10 -10
View File
@@ -948,9 +948,9 @@ and the running `smallwebwaf` takes the edit in.
- what was broken: the rule ids and target that matched, or the limit, its - what was broken: the rule ids and target that matched, or the limit, its
window, the count reached and the client's limit percentage with what set window, the count reached and the client's limit percentage with what set
it; and any reputation sources that listed the client; it; and any reputation sources that listed the client;
- the requests that caused the ban, up to the last ten: time, method, host, - the request that caused the ban, the one that broke the limit or carried
path with its query string, status and user agent, each text cut to 256 the clear sign of attack: time, method, host, path with its query string,
bytes; status and user agent, each text cut to 256 bytes;
- how many requests counted toward the ban, and the time span over which - how many requests counted toward the ban, and the time span over which
they came; they came;
- the netblock's total requests since it was first seen, and the requests - the netblock's total requests since it was first seen, and the requests
@@ -961,13 +961,13 @@ and the running `smallwebwaf` takes the edit in.
the table is full, so on a public service the file grows to the default the table is full, so on a public service the file grows to the default
`SWWAF_MAX_TRACKED_CLIENTS` of 20,000, about 20 MiB. Written every 15 `SWWAF_MAX_TRACKED_CLIENTS` of 20,000, about 20 MiB. Written every 15
minutes, that is under 2 GiB of disk writes a day. minutes, that is under 2 GiB of disk writes a day.
- `bans.json` takes about 2 KiB per ban and at most about 8 KiB, since the - `bans.json` takes about 1.2 KiB per ban and at most about 2.5 KiB, since
texts in the notes are cut short. At the default `SWWAF_MAX_BANS` of 5,000 the notes hold one request and their texts are cut short. At the default
it is about 10 MiB, and never more than about 40 MiB, plus whatever bans `SWWAF_MAX_BANS` of 5,000 it is about 6 MiB, and never more than about 12
an admin made. It is written when a ban is made, lifted or made permanent, MiB, plus whatever bans an admin made. It is written when a ban is made,
at most once every 10 seconds, and otherwise with the 15-minute write, so lifted or made permanent, at most once every 10 seconds, and otherwise
its writes follow the bans made: with a full file, a hundred new bans a with the 15-minute write, so its writes follow the bans made: with a full
day come to about 1 GiB of disk writes. file, a hundred new bans a day come to about 600 MiB of disk writes.
- `lookups.json` takes about 150 bytes per answer, about 15 MiB when full. - `lookups.json` takes about 150 bytes per answer, about 15 MiB when full.
Written every 15 minutes, that is under 1.5 GiB of disk writes a day. Written every 15 minutes, that is under 1.5 GiB of disk writes a day.
- `reputation.json` and `alerts.json` are usually a few MiB or less. - `reputation.json` and `alerts.json` are usually a few MiB or less.
+255
View File
@@ -0,0 +1,255 @@
// Package bans is the ban ledger: the bans smallwebwaf makes on the
// netblocks of clients that break a rate limit, with their notes, as the
// "Bans" section of SPEC.md describes. The bans are kept in memory only.
package bans
import (
"net/netip"
"slices"
"strings"
"sync"
"time"
"github.com/hashicorp/golang-lru/v2/simplelru"
)
// repeatFactor is how many times as long as the netblock's last ban a ban
// for a limit broken again within the repeat window lasts.
const repeatFactor = 3
// maxTextBytes is how much of each text in a ban's notes is kept.
const maxTextBytes = 256
// Rules are how long a ban for a broken limit lasts, and how many bans
// are held.
type Rules struct {
// LimitBanDuration is how long a first ban lasts.
LimitBanDuration time.Duration
// LimitBanRepeatWindow is how soon after the netblock's last ban
// ended a broken limit counts as a repeat, which bans for
// repeatFactor times as long as that ban.
LimitBanRepeatWindow time.Duration
// MaxBanDuration is the longest ban; a ban that would be longer is
// permanent instead.
MaxBanDuration time.Duration
// MaxBans is the most bans held, at least one. Past it, the earliest
// ban of the netblock that has gone longest without a request is
// dropped.
MaxBans int
}
// Ban is a ban on a netblock for a broken limit, the only kind of ban
// smallwebwaf makes so far.
type Ban struct {
Netblock netip.Prefix
Start time.Time
// Expires is when the ban ends, zero for a permanent ban.
Expires time.Time
Notes Notes
}
// Permanent reports whether the ban never runs out.
func (b Ban) Permanent() bool {
return b.Expires.IsZero()
}
// ActiveAt reports whether the ban refuses requests at now.
func (b Ban) ActiveAt(now time.Time) bool {
return b.Permanent() || now.Before(b.Expires)
}
// Notes are what an admin needs to decide whether to lift a ban.
type Notes struct {
// Country is the client's country, when it was looked up.
Country string
// Limit, Window and Count are the limit that was broken, its window,
// "minute", "hour" or "day", and the count reached: the client's
// requests in the window, the one that broke the limit included.
// These are the requests that counted toward the ban, and the window
// is the time over which they came.
Limit int64
Window string
Count float64
// Request is the request that broke the limit.
Request Request
// Refused is how many requests the ban has refused so far.
Refused int64
// EarlierBans is how many bans the netblock had before this one.
EarlierBans int
}
// Request is a request in a ban's notes. Each text is cut to 256 bytes.
type Request struct {
Time time.Time
Method string
Host string
// Path is the path with its query string.
Path string
// Status is what the client was sent, 0 if nothing was.
Status int
UserAgent string
}
// Ledger holds the bans. It is safe for concurrent use.
type Ledger struct {
rules Rules
mu sync.Mutex
// netblocks holds each banned netblock's bans, oldest first. Each
// request from a netblock makes it the most recently seen.
netblocks *simplelru.LRU[netip.Prefix, *[]Ban]
// held is how many bans netblocks holds, at most rules.MaxBans.
held int
}
// New returns a Ledger with no ban yet.
func New(rules Rules) *Ledger {
// Every netblock held has a ban, so there are never more netblocks
// than rules.MaxBans, and the LRU never drops one itself.
netblocks, err := simplelru.NewLRU[netip.Prefix, *[]Ban](rules.MaxBans, nil)
if err != nil {
panic(err) // NewLRU fails only for a size below one
}
return &Ledger{rules: rules, netblocks: netblocks}
}
// Check is called for each request from netblock, at now. It reports
// whether a ban on netblock is active, and returns that ban, with the
// request counted among those it refused.
func (l *Ledger) Check(netblock netip.Prefix, now time.Time) (Ban, bool) {
l.mu.Lock()
defer l.mu.Unlock()
bans, found := l.netblocks.Get(netblock)
if !found {
return Ban{}, false
}
// A ban is made only once the one before has ended, so only the last
// can be active.
last := &(*bans)[len(*bans)-1]
if !last.ActiveAt(now) {
return Ban{}, false
}
last.Notes.Refused++
return *last, true
}
// BanForLimit bans netblock at now for a broken limit, with notes, and
// returns the ban. A first ban lasts LimitBanDuration. A ban made within
// LimitBanRepeatWindow after the netblock's last ban ended lasts
// repeatFactor times as long as that one. A ban that would be longer
// than MaxBanDuration is permanent instead. If a ban on netblock is still
// active, as when two of its requests break a limit at once, that ban is
// returned and no other is made. The ledger fills in the notes' Refused
// and EarlierBans itself.
func (l *Ledger) BanForLimit(netblock netip.Prefix, now time.Time, notes Notes) Ban {
l.mu.Lock()
defer l.mu.Unlock()
var last *Ban
bans, found := l.netblocks.Get(netblock)
if found {
last = &(*bans)[len(*bans)-1]
if last.ActiveAt(now) {
return *last
}
notes.EarlierBans = last.Notes.EarlierBans + 1
}
notes.Request = notes.Request.cut()
ban := Ban{
Netblock: netblock,
Start: now,
Expires: l.expiry(last, now),
Notes: notes,
}
if l.held == l.rules.MaxBans {
l.dropOne()
}
// dropOne can have dropped netblock's last ban, and netblock with it.
bans, found = l.netblocks.Peek(netblock)
if !found {
bans = &[]Ban{}
l.netblocks.Add(netblock, bans)
}
*bans = append(*bans, ban)
l.held++
return ban
}
// Bans returns the bans held on netblock, oldest first. It is not a
// request from netblock, and leaves when it was last seen unchanged.
func (l *Ledger) Bans(netblock netip.Prefix) []Ban {
l.mu.Lock()
defer l.mu.Unlock()
bans, found := l.netblocks.Peek(netblock)
if !found {
return nil
}
return slices.Clone(*bans)
}
// expiry returns when a ban for a broken limit made at now ends, or zero
// when it is permanent. last is the netblock's last ban, which has ended,
// or nil when it has none.
func (l *Ledger) expiry(last *Ban, now time.Time) time.Time {
length := l.rules.LimitBanDuration
if last != nil && now.Sub(last.Expires) <= l.rules.LimitBanRepeatWindow {
lastLength := last.Expires.Sub(last.Start)
// This is repeatFactor * lastLength > MaxBanDuration, written so
// that it cannot overflow.
if lastLength > l.rules.MaxBanDuration/repeatFactor {
return time.Time{}
}
length = repeatFactor * lastLength
}
if length > l.rules.MaxBanDuration {
return time.Time{}
}
return now.Add(length)
}
// dropOne drops the earliest ban of the netblock that has gone longest
// without a request, and the netblock with it if that was its only ban.
func (l *Ledger) dropOne() {
netblock, bans, _ := l.netblocks.GetOldest()
if len(*bans) == 1 {
l.netblocks.Remove(netblock)
} else {
*bans = slices.Delete(*bans, 0, 1)
}
l.held--
}
// cut returns r with each text cut to maxTextBytes and copied, so that
// the notes do not keep the rest of the request in memory.
func (r Request) cut() Request {
r.Method = cutText(r.Method)
r.Host = cutText(r.Host)
r.Path = cutText(r.Path)
r.UserAgent = cutText(r.UserAgent)
return r
}
// cutText returns a copy of the first maxTextBytes of text.
func cutText(text string) string {
return strings.Clone(text[:min(len(text), maxTextBytes)])
}
+267
View File
@@ -0,0 +1,267 @@
package bans_test
import (
"net/netip"
"strings"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
)
const day = 24 * time.Hour
func TestRepeatsTripleUntilPermanent(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultRules())
netblock := netip.MustParsePrefix("203.0.113.9/32")
now := midnight()
// Each ban is followed by another as soon as it ends: 1, 3, 9, 27 and
// 81 hours.
for i, hours := range []int{1, 3, 9, 27, 81} {
ban := ledger.BanForLimit(netblock, now, bans.Notes{})
length := time.Duration(hours) * time.Hour
if !ban.Expires.Equal(now.Add(length)) || ban.Notes.EarlierBans != i {
t.Fatalf("ban %d lasts %s with %d earlier bans, want %d hours and %d",
i+1, ban.Expires.Sub(now), ban.Notes.EarlierBans, hours, i)
}
now = ban.Expires
}
// The sixth would last 243 hours, more than seven days: it is
// permanent, and never ends.
ban := ledger.BanForLimit(netblock, now, bans.Notes{})
if !ban.Permanent() {
t.Fatalf("sixth ban ends at %s, want a permanent one", ban.Expires)
}
_, banned := ledger.Check(netblock, now.Add(100*365*day))
if !banned {
t.Error("a permanent ban ended")
}
}
func TestRepeatWindowRunsOut(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
// gap is the time between the end of the first ban and the second.
gap time.Duration
want time.Duration
}{
{"broken again as the window ends", day, 3 * time.Hour},
{"broken again after the window", day + time.Nanosecond, time.Hour},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultRules())
netblock := netip.MustParsePrefix("203.0.113.9/32")
first := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
second := ledger.BanForLimit(netblock, first.Expires.Add(tc.gap), bans.Notes{})
if second.Expires.Sub(second.Start) != tc.want || second.Notes.EarlierBans != 1 {
t.Errorf("second ban lasts %s with %d earlier bans, want %s and 1",
second.Expires.Sub(second.Start), second.Notes.EarlierBans, tc.want)
}
})
}
}
func TestFirstBanLongerThanTheMaximumIsPermanent(t *testing.T) {
t.Parallel()
rules := defaultRules()
rules.LimitBanDuration = rules.MaxBanDuration + time.Hour
ledger := bans.New(rules)
ban := ledger.BanForLimit(netip.MustParsePrefix("203.0.113.9/32"), midnight(),
bans.Notes{})
if !ban.Permanent() {
t.Errorf("first ban ends at %s, want a permanent one", ban.Expires)
}
}
func TestLongestBanSetFarOffDoesNotOverflow(t *testing.T) {
t.Parallel()
// With bans of up to 100,000 days, the 14th ban in a row, of 3^13
// hours, is within the maximum, and three times as long would not fit
// in a time.Duration. The 15th is permanent.
rules := defaultRules()
rules.MaxBanDuration = 100000 * day
ledger := bans.New(rules)
netblock := netip.MustParsePrefix("203.0.113.9/32")
now := midnight()
for i := range 14 {
ban := ledger.BanForLimit(netblock, now, bans.Notes{})
if !ban.Expires.After(ban.Start) {
t.Fatalf("ban %d starts at %s and ends at %s", i+1, ban.Start, ban.Expires)
}
now = ban.Expires
}
ban := ledger.BanForLimit(netblock, now, bans.Notes{})
if !ban.Permanent() {
t.Errorf("15th ban ends at %s, want a permanent one", ban.Expires)
}
}
func TestBrokenLimitDuringABanMakesNoOther(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultRules())
netblock := netip.MustParsePrefix("203.0.113.9/32")
first := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
again := ledger.BanForLimit(netblock, midnight().Add(time.Minute), bans.Notes{})
if again != first || len(ledger.Bans(netblock)) != 1 {
t.Errorf("a limit broken during a ban gave %+v and %d bans, want %+v and 1",
again, len(ledger.Bans(netblock)), first)
}
}
func TestCheckRefusesWhileTheBanLastsAndCountsTheRefusals(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultRules())
netblock := netip.MustParsePrefix("203.0.113.9/32")
ban := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
for range 3 {
got, banned := ledger.Check(netblock, ban.Expires.Add(-time.Nanosecond))
if !banned || got.Start != ban.Start {
t.Fatalf("check during the ban gives %+v and %t", got, banned)
}
}
_, banned := ledger.Check(netip.MustParsePrefix("203.0.113.10/32"), midnight())
if banned {
t.Error("another netblock is banned")
}
_, banned = ledger.Check(netblock, ban.Expires)
if banned {
t.Error("the ban did not end")
}
refused := ledger.Bans(netblock)[0].Notes.Refused
if refused != 3 {
t.Errorf("the notes count %d refused requests, want 3", refused)
}
}
func TestMaxBansDropsTheEarliestBanOfTheNetblockSeenLongestAgo(t *testing.T) {
t.Parallel()
rules := defaultRules()
rules.MaxBans = 3
ledger := bans.New(rules)
a := netip.MustParsePrefix("203.0.113.1/32")
b := netip.MustParsePrefix("203.0.113.2/32")
c := netip.MustParsePrefix("203.0.113.3/32")
d := netip.MustParsePrefix("2001:db8::/64")
now := midnight()
first := ledger.BanForLimit(a, now, bans.Notes{})
ledger.BanForLimit(b, now, bans.Notes{})
ledger.BanForLimit(c, now, bans.Notes{})
// A request from a makes b the netblock seen longest ago, and its ban
// goes to make room for d's.
ledger.Check(a, now)
ledger.BanForLimit(d, now, bans.Notes{})
wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 0, c: 1, d: 1})
// a is banned again once its ban has ended; c, seen longest ago, goes.
ledger.BanForLimit(a, first.Expires, bans.Notes{})
wantBans(t, ledger, map[netip.Prefix]int{a: 2, c: 0, d: 1})
// With d seen since, a is seen longest ago, and its earlier ban goes
// first.
ledger.Check(d, first.Expires)
ledger.BanForLimit(b, first.Expires, bans.Notes{})
wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 1, d: 1})
if !ledger.Bans(a)[0].Start.Equal(first.Expires) {
t.Errorf("a kept its ban of %s, want the later one", ledger.Bans(a)[0].Start)
}
}
func TestFullLedgerDropsTheEarlierBanOfTheNetblockBannedAgain(t *testing.T) {
t.Parallel()
// With room for one ban, the netblock's ended ban goes to make room for
// its new one, whose notes still count it.
rules := defaultRules()
rules.MaxBans = 1
ledger := bans.New(rules)
netblock := netip.MustParsePrefix("203.0.113.9/32")
first := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
second := ledger.BanForLimit(netblock, first.Expires, bans.Notes{})
held := ledger.Bans(netblock)
if len(held) != 1 || held[0] != second || held[0].Notes.EarlierBans != 1 {
t.Errorf("the ledger holds %+v, want only the second ban, with 1 earlier ban",
held)
}
}
func TestRequestTextsAreCutTo256Bytes(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultRules())
netblock := netip.MustParsePrefix("203.0.113.9/32")
long := strings.Repeat("a", 300)
request := bans.Request{
Time: midnight(), Method: long, Host: long, Path: long, Status: 403, UserAgent: long,
}
ban := ledger.BanForLimit(netblock, midnight(), bans.Notes{Request: request})
cut := long[:256]
want := bans.Request{
Time: midnight(), Method: cut, Host: cut, Path: cut, Status: 403, UserAgent: cut,
}
if ban.Notes.Request != want || ledger.Bans(netblock)[0].Notes.Request != want {
t.Errorf("the notes keep %+v, want each text cut to 256 bytes", ban.Notes.Request)
}
}
// defaultRules are the rules at the settings' defaults.
func defaultRules() bans.Rules {
return bans.Rules{
LimitBanDuration: time.Hour,
LimitBanRepeatWindow: day,
MaxBanDuration: 7 * day,
MaxBans: 5000,
}
}
// midnight is when the tests' first bans are made.
func midnight() time.Time {
return time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)
}
// wantBans checks how many bans the ledger holds on each netblock.
func wantBans(t *testing.T, ledger *bans.Ledger, want map[netip.Prefix]int) {
t.Helper()
for netblock, count := range want {
got := len(ledger.Bans(netblock))
if got != count {
t.Errorf("%s has %d bans, want %d", netblock, got, count)
}
}
}
+116 -2
View File
@@ -9,6 +9,7 @@ import (
"log/slog" "log/slog"
"math" "math"
"net" "net"
"net/http"
"net/netip" "net/netip"
"net/url" "net/url"
"slices" "slices"
@@ -74,6 +75,25 @@ type Config struct {
// capitals, as GeoJS gives them. // capitals, as GeoJS gives them.
DeniedCountries []string DeniedCountries []string
ExclusivelyAllowedCountries []string ExclusivelyAllowedCountries []string
// BanResponse is the status a refused client is answered with, 403
// or 429, or 0 to close the connection without an answer
// (SWWAF_BAN_RESPONSE). It answers a banned client, a request that
// breaks a rate limit, SWWAF_DENY_NETS and the country lists.
BanResponse int
// LimitBanDuration is the ban for a first broken rate limit
// (SWWAF_LIMIT_BAN_DURATION). A limit broken again within
// LimitBanRepeatWindow after the last ban ended bans for three times
// as long as that ban (SWWAF_LIMIT_BAN_REPEAT_WINDOW), and a ban that
// would be longer than MaxBanDuration is permanent instead
// (SWWAF_MAX_BAN_DURATION). None of them can be off.
LimitBanDuration time.Duration
LimitBanRepeatWindow time.Duration
MaxBanDuration time.Duration
// MaxBans is the most bans held (SWWAF_MAX_BANS).
MaxBans int
// BanScopeV4Prefix is the length of the netblock around an IPv4
// client that a ban covers (SWWAF_BAN_SCOPE_V4_PREFIX).
BanScopeV4Prefix int
// settings are the values read, as given or by default, for the // settings are the values read, as given or by default, for the
// log line at start. // log line at start.
@@ -89,6 +109,7 @@ const (
kibibyte = 1 << 10 kibibyte = 1 << 10
mebibyte = 1 << 20 mebibyte = 1 << 20
gibibyte = 1 << 30 gibibyte = 1 << 30
ipv4Bits = 32
) )
var ( var (
@@ -109,8 +130,15 @@ var (
"such as http://127.0.0.1:8081") "such as http://127.0.0.1:8081")
errNotCountry = errors.New( errNotCountry = errors.New(
"is not a two-letter country code such as de or kp") "is not a two-letter country code such as de or kp")
errOnBothLists = errors.New("is in SWWAF_DENIED_COUNTRIES too") errOnBothLists = errors.New("is in SWWAF_DENIED_COUNTRIES too")
errNotOver4K = errors.New("is not a size of more than 4K, such as 32K") errNotOver4K = errors.New("is not a size of more than 4K, such as 32K")
errNotDurationAboveZero = errors.New(
"is not a duration above zero, such as 1h or 7d")
errNotNumberAboveZero = errors.New(
"is not a whole number above zero, such as 5000")
errNotBanResponse = errors.New("is not 403, 429 or close")
errNotV4Prefix = errors.New(
"is not the length of an IPv4 netblock, from 0 to 32, such as 24")
) )
// FromEnvironment reads the settings with lookupEnv, normally // FromEnvironment reads the settings with lookupEnv, normally
@@ -140,6 +168,12 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
DeniedCountries: env.countries("SWWAF_DENIED_COUNTRIES", ""), DeniedCountries: env.countries("SWWAF_DENIED_COUNTRIES", ""),
ExclusivelyAllowedCountries: env.countries( ExclusivelyAllowedCountries: env.countries(
"SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES", ""), "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES", ""),
BanResponse: env.banResponse("SWWAF_BAN_RESPONSE", "403"),
LimitBanDuration: env.durationNotOff("SWWAF_LIMIT_BAN_DURATION", "1h"),
LimitBanRepeatWindow: env.durationNotOff("SWWAF_LIMIT_BAN_REPEAT_WINDOW", "24h"),
MaxBanDuration: env.durationNotOff("SWWAF_MAX_BAN_DURATION", "7d"),
MaxBans: env.numberNotOff("SWWAF_MAX_BANS", "5000"),
BanScopeV4Prefix: env.v4Prefix("SWWAF_BAN_SCOPE_V4_PREFIX", "32"),
} }
for _, country := range cfg.ExclusivelyAllowedCountries { for _, country := range cfg.ExclusivelyAllowedCountries {
@@ -261,6 +295,40 @@ func (e *environment) countries(name, defaultValue string) []string {
return countries return countries
} }
// durationNotOff reads a setting that is a duration and, unlike a
// timeout, cannot be off.
func (e *environment) durationNotOff(name, defaultValue string) time.Duration {
duration, err := parseDurationNotOff(e.value(name, defaultValue))
e.check(name, err)
return duration
}
// numberNotOff reads a setting that is a whole number above zero, which
// cannot be off.
func (e *environment) numberNotOff(name, defaultValue string) int {
number, err := parseNumberNotOff(e.value(name, defaultValue))
e.check(name, err)
return number
}
// banResponse reads a setting that is how a refused client is answered.
func (e *environment) banResponse(name, defaultValue string) int {
status, err := parseBanResponse(e.value(name, defaultValue))
e.check(name, err)
return status
}
// v4Prefix reads a setting that is the length of an IPv4 netblock.
func (e *environment) v4Prefix(name, defaultValue string) int {
length, err := parseV4Prefix(e.value(name, defaultValue))
e.check(name, err)
return length
}
// parseDuration reads a duration in Go's syntax, such as 90s or 15m, a // parseDuration reads a duration in Go's syntax, such as 90s or 15m, a
// whole number of days such as 7d, or off. // whole number of days such as 7d, or off.
func parseDuration(value string) (time.Duration, error) { func parseDuration(value string) (time.Duration, error) {
@@ -362,6 +430,52 @@ func parseCount(value string) (int64, error) {
return n, nil return n, nil
} }
// parseDurationNotOff reads a duration above zero, as parseDuration does,
// but not off.
func parseDurationNotOff(value string) (time.Duration, error) {
duration, err := parseDuration(value)
if err != nil || duration == 0 {
return 0, fmt.Errorf("%q %w", value, errNotDurationAboveZero)
}
return duration, nil
}
// parseNumberNotOff reads a whole number above zero.
func parseNumberNotOff(value string) (int, error) {
n, err := strconv.Atoi(value)
if err != nil || n <= 0 {
return 0, fmt.Errorf("%q %w", value, errNotNumberAboveZero)
}
return n, nil
}
// parseBanResponse reads how a refused client is answered: 403, 429, or
// close, which is 0.
func parseBanResponse(value string) (int, error) {
switch value {
case "403":
return http.StatusForbidden, nil
case "429":
return http.StatusTooManyRequests, nil
case "close":
return 0, nil
default:
return 0, fmt.Errorf("%q %w", value, errNotBanResponse)
}
}
// parseV4Prefix reads the length of an IPv4 netblock, from 0 to 32.
func parseV4Prefix(value string) (int, error) {
n, err := strconv.Atoi(value)
if err != nil || n < 0 || n > ipv4Bits {
return 0, fmt.Errorf("%q %w", value, errNotV4Prefix)
}
return n, nil
}
// parseList splits a comma-separated list and trims the spaces around // parseList splits a comma-separated list and trims the spaces around
// each item. An empty value is an empty list. // each item. An empty value is an empty list.
func parseList(value string) ([]string, error) { func parseList(value string) ([]string, error) {
+61
View File
@@ -35,6 +35,12 @@ const (
rateLimitPerDay = "SWWAF_RATE_LIMIT_PER_DAY" rateLimitPerDay = "SWWAF_RATE_LIMIT_PER_DAY"
deniedCountries = "SWWAF_DENIED_COUNTRIES" deniedCountries = "SWWAF_DENIED_COUNTRIES"
allowedCountries = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES" allowedCountries = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES"
banResponse = "SWWAF_BAN_RESPONSE"
limitBanDuration = "SWWAF_LIMIT_BAN_DURATION"
limitBanRepeatWindow = "SWWAF_LIMIT_BAN_REPEAT_WINDOW"
maxBanDuration = "SWWAF_MAX_BAN_DURATION"
maxBans = "SWWAF_MAX_BANS"
banScopeV4Prefix = "SWWAF_BAN_SCOPE_V4_PREFIX"
) )
// off switches a timeout, a size limit or a rate limit off. // off switches a timeout, a size limit or a rate limit off.
@@ -80,6 +86,12 @@ func TestDefaults(t *testing.T) {
RateLimitPerMinute: 1000, RateLimitPerMinute: 1000,
RateLimitPerHour: 10000, RateLimitPerHour: 10000,
RateLimitPerDay: 50000, RateLimitPerDay: 50000,
BanResponse: 403,
LimitBanDuration: time.Hour,
LimitBanRepeatWindow: 24 * time.Hour,
MaxBanDuration: 7 * 24 * time.Hour,
MaxBans: 5000,
BanScopeV4Prefix: 32,
}) })
if cfg.UpstreamURL.String() != "http://127.0.0.1:8081" { if cfg.UpstreamURL.String() != "http://127.0.0.1:8081" {
@@ -118,6 +130,12 @@ func TestValuesAsSet(t *testing.T) {
rateLimitPerDay: "6000", rateLimitPerDay: "6000",
deniedCountries: "cn, RU,kp,Xk", deniedCountries: "cn, RU,kp,Xk",
allowedCountries: "de", allowedCountries: "de",
banResponse: "429",
limitBanDuration: "15m",
limitBanRepeatWindow: "2d",
maxBanDuration: "30d",
maxBans: "100",
banScopeV4Prefix: "24",
}) })
wantSettings(t, cfg, config.Config{ wantSettings(t, cfg, config.Config{
@@ -133,6 +151,12 @@ func TestValuesAsSet(t *testing.T) {
RateLimitPerMinute: 60, RateLimitPerMinute: 60,
RateLimitPerHour: 600, RateLimitPerHour: 600,
RateLimitPerDay: 6000, RateLimitPerDay: 6000,
BanResponse: 429,
LimitBanDuration: 15 * time.Minute,
LimitBanRepeatWindow: 48 * time.Hour,
MaxBanDuration: 30 * 24 * time.Hour,
MaxBans: 100,
BanScopeV4Prefix: 24,
}) })
if cfg.UpstreamURL.String() != "https://app.internal:8443/" { if cfg.UpstreamURL.String() != "https://app.internal:8443/" {
@@ -226,6 +250,15 @@ func TestRateLimitsOff(t *testing.T) {
} }
} }
func TestBanResponseCloseIsZero(t *testing.T) {
t.Parallel()
cfg := fromEnvironment(t, environment{banResponse: "close"})
if cfg.BanResponse != 0 {
t.Errorf("close read as %d, want 0", cfg.BanResponse)
}
}
func TestTrustedProxiesSetButEmptyTrustNothing(t *testing.T) { func TestTrustedProxiesSetButEmptyTrustNothing(t *testing.T) {
t.Parallel() t.Parallel()
@@ -290,6 +323,12 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
{allowedCountries, "uk"}, {allowedCountries, "uk"},
{allowedCountries, "zz"}, {allowedCountries, "zz"},
{allowedCountries, "de,germany"}, {allowedCountries, "de,germany"},
{banResponse, "404"}, {banResponse, "drop"}, {banResponse, ""},
{limitBanDuration, off}, {limitBanDuration, "0s"}, {limitBanDuration, "1"},
{limitBanRepeatWindow, off}, {limitBanRepeatWindow, "-1h"},
{maxBanDuration, off}, {maxBanDuration, "1w"},
{maxBans, off}, {maxBans, "0"}, {maxBans, "5K"},
{banScopeV4Prefix, "33"}, {banScopeV4Prefix, "-1"}, {banScopeV4Prefix, "/24"},
} { } {
t.Run(tc.name+"="+tc.value, func(t *testing.T) { t.Run(tc.name+"="+tc.value, func(t *testing.T) {
t.Parallel() t.Parallel()
@@ -344,6 +383,12 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
rateLimitPerDay: "50000", rateLimitPerDay: "50000",
deniedCountries: "", deniedCountries: "",
allowedCountries: "", allowedCountries: "",
banResponse: "403",
limitBanDuration: "1h",
limitBanRepeatWindow: "24h",
maxBanDuration: "7d",
maxBans: "5000",
banScopeV4Prefix: "32",
} }
if !maps.Equal(line.Settings, want) { if !maps.Equal(line.Settings, want) {
t.Errorf("logged settings\n%v\nwant\n%v", line.Settings, want) t.Errorf("logged settings\n%v\nwant\n%v", line.Settings, want)
@@ -368,6 +413,22 @@ func wantSettings(t *testing.T, got *config.Config, want config.Config) {
got.RateLimitPerDay != want.RateLimitPerDay { got.RateLimitPerDay != want.RateLimitPerDay {
t.Errorf("settings\n%+v\nwant\n%+v", got, want) t.Errorf("settings\n%+v\nwant\n%+v", got, want)
} }
wantBanSettings(t, got, want)
}
// wantBanSettings checks the settings for bans.
func wantBanSettings(t *testing.T, got *config.Config, want config.Config) {
t.Helper()
if got.BanResponse != want.BanResponse ||
got.LimitBanDuration != want.LimitBanDuration ||
got.LimitBanRepeatWindow != want.LimitBanRepeatWindow ||
got.MaxBanDuration != want.MaxBanDuration ||
got.MaxBans != want.MaxBans ||
got.BanScopeV4Prefix != want.BanScopeV4Prefix {
t.Errorf("ban settings\n%+v\nwant\n%+v", got, want)
}
} }
// wantNetblocks checks a list of netblocks. // wantNetblocks checks a list of netblocks.
+82
View File
@@ -0,0 +1,82 @@
package proxy
import (
"net/netip"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// banResponse is a refusal answered with SWWAF_BAN_RESPONSE, and logged
// with action.
func (rq *request) banResponse(action string) *refusal {
return &refusal{status: rq.h.config.BanResponse, action: action}
}
// banned reports whether a ban on the client's netblock refuses the
// request at now, and notes for the log line when that ban ends.
func (rq *request) banned(now time.Time) bool {
ban, banned := rq.h.ledger.Check(rq.netblock(), now)
if banned {
rq.line.BanExpires = banExpires(ban)
}
return banned
}
// limitBroken counts the request for the rate limits at now, and reports
// whether it takes the client over one. Such a request bans the client's
// netblock, and sets the client's counters back to zero.
func (rq *request) limitBroken(now time.Time) bool {
group := clientGroup(rq.client)
hit, over := rq.h.limiter.Count(group, now)
if !over {
return false
}
ban := rq.h.ledger.BanForLimit(rq.netblock(), now, bans.Notes{
Country: rq.line.Country,
Limit: hit.Limit,
Window: hit.Window,
Count: hit.Requests,
Request: bans.Request{
Time: now,
Method: rq.in.Method,
Host: rq.in.Host,
Path: rq.in.URL.RequestURI(),
Status: rq.h.config.BanResponse,
UserAgent: rq.in.UserAgent(),
},
})
rq.h.limiter.Reset(group)
rq.line.LimitHit = hit.Window
rq.line.Offence = requestlog.OffenceLimit
rq.line.BanExpires = banExpires(ban)
return true
}
// netblock is the netblock a ban on the client covers: its IPv4 address,
// widened to SWWAF_BAN_SCOPE_V4_PREFIX, or the IPv6 group clientGroup
// counts it in.
func (rq *request) netblock() netip.Prefix {
addr := rq.client.Unmap()
if addr.Is4() {
return netip.PrefixFrom(addr, rq.h.config.BanScopeV4Prefix).Masked()
}
return clientGroup(addr)
}
// banExpires is when ban ends, as the log line gives it: a time, or
// permanent.
func banExpires(ban bans.Ban) string {
if ban.Permanent() {
return "permanent"
}
return requestlog.FormatTime(ban.Expires)
}
+434
View File
@@ -0,0 +1,434 @@
package proxy_test
import (
"bufio"
"errors"
"io"
"maps"
"net/http"
"net/netip"
"slices"
"sync"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/proxy"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
const (
// otherClient is a client next to client.
otherClient = "203.0.113.10"
// userAgent is the user agent of every request a sender sends.
userAgent = "ban-test/1.0"
// permanent is the log line's ban_expires for a permanent ban.
permanent = "permanent"
)
func TestBrokenLimitBansTheClient(t *testing.T) {
t.Parallel()
s, clk, _ := startWithClock(t, "", map[string]string{rateLimitPerMinute: "1"})
expires := requestlog.FormatTime(clk.Now().Add(time.Hour))
// The request over the limit of one a minute is refused, and bans the
// client for an hour, the default.
s.get(client, http.StatusOK, requestlog.ActionForward)
line := s.get(client, http.StatusForbidden, requestlog.ActionRateLimited)
if line.LimitHit != minute || line.Offence != requestlog.OffenceLimit ||
line.BanExpires != expires {
t.Errorf("log line has limit_hit %q, offence %q and ban_expires %q, "+
"want minute, limit and %s", line.LimitHit, line.Offence, line.BanExpires,
expires)
}
// Every request while the ban lasts is refused.
clk.advance(time.Hour - time.Second)
line = s.get(client, http.StatusForbidden, requestlog.ActionBanned)
if line.BanExpires != expires || line.Offence != "" || line.LimitHit != "" {
t.Errorf("log line has ban_expires %q, offence %q and limit_hit %q, "+
"want %s and neither of the others", line.BanExpires, line.Offence,
line.LimitHit, expires)
}
// Once it ends, the client is let through.
clk.advance(time.Second)
s.get(client, http.StatusOK, requestlog.ActionForward)
}
func TestBanLengthsFollowTheSettings(t *testing.T) {
t.Parallel()
s, clk, _ := startWithClock(t, "", map[string]string{
rateLimitPerMinute: "1",
limitBanDuration: "10m",
limitBanRepeatWindow: "1h",
maxBanDuration: "1h",
})
// breakLimit has client go over the limit of one a minute, and
// returns when the ban that makes ends.
breakLimit := func() string {
s.get(client, http.StatusOK, requestlog.ActionForward)
return s.get(client, http.StatusForbidden, requestlog.ActionRateLimited).BanExpires
}
wantExpires := func(got string, length time.Duration) {
t.Helper()
want := requestlog.FormatTime(clk.Now().Add(length))
if got != want {
t.Errorf("ban ends at %s, want %s", got, want)
}
}
// A first ban lasts SWWAF_LIMIT_BAN_DURATION; one within
// SWWAF_LIMIT_BAN_REPEAT_WINDOW after it ended, three times as long.
wantExpires(breakLimit(), 10*time.Minute)
clk.advance(10*time.Minute + time.Hour)
wantExpires(breakLimit(), 30*time.Minute)
// Later than that, SWWAF_LIMIT_BAN_DURATION again.
clk.advance(30*time.Minute + time.Hour + time.Second)
wantExpires(breakLimit(), 10*time.Minute)
clk.advance(10 * time.Minute)
wantExpires(breakLimit(), 30*time.Minute)
// 90 minutes would be longer than SWWAF_MAX_BAN_DURATION: the ban is
// permanent.
clk.advance(30 * time.Minute)
got := breakLimit()
if got != permanent {
t.Errorf("ban ends at %s, want a permanent one", got)
}
clk.advance(365 * 24 * time.Hour)
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
}
func TestBanIsNotCountedAndResetsTheCounters(t *testing.T) {
t.Parallel()
s, clk, server := startWithClock(t, "", map[string]string{rateLimitPerDay: "2"})
// The third request in a day is over the limit of two, and bans the
// client for an hour.
s.get(client, http.StatusOK, requestlog.ActionForward)
s.get(client, http.StatusOK, requestlog.ActionForward)
s.get(client, http.StatusForbidden, requestlog.ActionRateLimited)
for range 3 {
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
}
// Later the same day the client has its whole allowance again: the
// ban set its counters back to zero, and the requests it refused were
// not counted for the rate limits, only in its notes.
clk.advance(time.Hour)
s.get(client, http.StatusOK, requestlog.ActionForward)
s.get(client, http.StatusOK, requestlog.ActionForward)
s.get(client, http.StatusForbidden, requestlog.ActionRateLimited)
banned := proxy.LedgerOf(server).Bans(netip.MustParsePrefix(client + "/32"))
if len(banned) != 2 || banned[0].Notes.Refused != 3 {
t.Errorf("bans %+v, want two, the first with 3 requests refused", banned)
}
}
func TestBanCoversTheClientsNetblock(t *testing.T) {
t.Parallel()
// In the IPv4 cases, client breaks the limit; these two are next to it.
const (
allowed = "203.0.113.60" // in SWWAF_ALLOW_NETS
exempt = "203.0.113.50" // in SWWAF_RATE_LIMIT_EXEMPT_NETS
)
for _, tc := range []struct {
name string
env map[string]string
breaker string // the client that breaks the limit
refused []string
let []string // let through
}{
{
"an IPv4 address, by default", nil, client,
nil, []string{otherClient, exempt},
},
{
"the IPv4 netblock SWWAF_BAN_SCOPE_V4_PREFIX sets",
map[string]string{banScopeV4Prefix: "24"}, client,
[]string{otherClient, exempt}, []string{"203.0.112.9", allowed},
},
{
"an IPv6 /64", nil, "2001:db8:5::1",
[]string{"2001:db8:5::ffff:1"}, []string{"2001:db8:5:1::1"},
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
env := map[string]string{
rateLimitPerMinute: "1",
allowNets: allowed,
rateLimitExemptNets: exempt,
}
maps.Copy(env, tc.env)
s, _, _ := startWithClock(t, "", env)
s.get(tc.breaker, http.StatusOK, requestlog.ActionForward)
s.get(tc.breaker, http.StatusForbidden, requestlog.ActionRateLimited)
for _, sent := range tc.refused {
s.get(sent, http.StatusForbidden, requestlog.ActionBanned)
}
for _, sent := range tc.let {
s.get(sent, http.StatusOK, requestlog.ActionForward)
}
})
}
}
func TestBannedClientIsRefusedBeforeItsCountryIsLookedUp(t *testing.T) {
t.Parallel()
geojsURL, asked := startGeoJS(t)
s, _, _ := startWithClock(t, geojsURL, map[string]string{
rateLimitPerMinute: "1",
banScopeV4Prefix: "24",
deniedCountries: "kp",
})
// fromDE's ban covers otherClient, which is refused unasked about.
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited)
line := s.get(otherClient, http.StatusForbidden, requestlog.ActionBanned)
if line.Country != "" {
t.Errorf("log line has country %q, want none", line.Country)
}
if !slices.Equal(asked(), []string{fromDE}) {
t.Errorf("GeoJS was asked about %v, want %s alone", asked(), fromDE)
}
}
func TestBanResponseAnswersEveryRefusalButTheSizeLimits(t *testing.T) {
t.Parallel()
const denied = "192.0.2.50" // in SWWAF_DENY_NETS
for _, tc := range []struct {
setting string // "" leaves SWWAF_BAN_RESPONSE at its default
status int // 0 is the connection closed without an answer
}{
{"", http.StatusForbidden},
{"403", http.StatusForbidden},
{"429", http.StatusTooManyRequests},
{"close", 0},
} {
t.Run(banResponse+"="+tc.setting, func(t *testing.T) {
t.Parallel()
geojsURL, _ := startGeoJS(t)
env := map[string]string{
rateLimitPerMinute: "1",
denyNets: denied,
deniedCountries: "kp",
}
if tc.setting != "" {
env[banResponse] = tc.setting
}
s, _, _ := startWithClock(t, geojsURL, env)
s.get(denied, tc.status, requestlog.ActionDenied)
s.get(fromKP, tc.status, requestlog.ActionCountryDenied)
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
s.get(fromDE, tc.status, requestlog.ActionRateLimited)
s.get(fromDE, tc.status, requestlog.ActionBanned)
})
}
}
func TestBanNotes(t *testing.T) {
t.Parallel()
geojsURL, _ := startGeoJS(t)
s, clk, server := startWithClock(t, geojsURL, map[string]string{
rateLimitPerMinute: "1",
deniedCountries: "kp",
})
start := clk.Now()
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
s.request(fromDE, "/repo/commits?page=2",
http.StatusForbidden, requestlog.ActionRateLimited)
s.get(fromDE, http.StatusForbidden, requestlog.ActionBanned)
s.get(fromDE, http.StatusForbidden, requestlog.ActionBanned)
netblock := netip.MustParsePrefix(fromDE + "/32")
want := bans.Ban{
Netblock: netblock,
Start: start,
Expires: start.Add(time.Hour),
Notes: bans.Notes{
Country: "DE",
Limit: 1,
Window: minute,
Count: 2,
Request: bans.Request{
Time: start,
Method: http.MethodGet,
Host: appHost,
Path: "/repo/commits?page=2",
Status: http.StatusForbidden,
UserAgent: userAgent,
},
Refused: 2,
EarlierBans: 0,
},
}
ledger := proxy.LedgerOf(server)
got := ledger.Bans(netblock)
if len(got) != 1 || got[0] != want {
t.Fatalf("bans\n%+v\nwant\n%+v", got, want)
}
// The next ban counts this one among the earlier.
clk.advance(time.Hour)
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited)
got = ledger.Bans(netblock)
if len(got) != 2 || got[1].Notes.EarlierBans != 1 {
t.Errorf("bans %+v, want two, the second with one earlier ban", got)
}
}
func TestMaxBansDropsTheBanOfTheNetblockSeenLongestAgo(t *testing.T) {
t.Parallel()
s, _, _ := startWithClock(t, "", map[string]string{
rateLimitPerMinute: "1",
maxBans: "1",
})
// One ban is held, so otherClient's ban drops client's.
s.get(client, http.StatusOK, requestlog.ActionForward)
s.get(client, http.StatusForbidden, requestlog.ActionRateLimited)
s.get(otherClient, http.StatusOK, requestlog.ActionForward)
s.get(otherClient, http.StatusForbidden, requestlog.ActionRateLimited)
s.get(client, http.StatusOK, requestlog.ActionForward)
s.get(otherClient, http.StatusForbidden, requestlog.ActionBanned)
}
// clock is the time a test sets, by which smallwebwaf counts requests and
// makes bans.
type clock struct {
mu sync.Mutex
now time.Time
}
// Now tells the time.
func (c *clock) Now() time.Time {
c.mu.Lock()
defer c.mu.Unlock()
return c.now
}
// advance moves the clock on by d.
func (c *clock) advance(d time.Duration) {
c.mu.Lock()
defer c.mu.Unlock()
c.now = c.now.Add(d)
}
// startWithClock starts smallwebwaf in front of an app that answers 200,
// with the settings in env on top of trusting localhost's
// X-Forwarded-For, clients' countries looked up at geojsURL, and a clock
// set to midnight, the start of a bucket in every window.
func startWithClock(
t *testing.T, geojsURL string, env map[string]string,
) (*sender, *clock, *http.Server) {
t.Helper()
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
clk := &clock{now: time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)}
settings := map[string]string{trustedProxies: trustLocalhost}
maps.Copy(settings, env)
addr, out, server := startProxyWithClock(t, app.URL, geojsURL, clk.Now, settings)
return &sender{t: t, addr: addr, out: out}, clk, server
}
// sender sends requests to smallwebwaf one after another, each on a
// connection of its own, and checks each one's answer and log line. They
// must be the only requests smallwebwaf is sent, since the log lines are
// matched to them in order.
type sender struct {
t *testing.T
addr string
out *output
sent int
}
// get sends a GET request for / from the client at from.
func (s *sender) get(from string, status int, action string) logLine {
s.t.Helper()
return s.request(from, "/", status, action)
}
// request sends a GET request for path from the client at from, as
// X-Forwarded-For names it, and checks that its answer and its log line
// have status, 0 for the connection closed without an answer, and that
// the line has action. It returns the log line.
func (s *sender) request(from, path string, status int, action string) logLine {
s.t.Helper()
conn := dial(s.t, s.addr)
send(s.t, conn, "GET "+path+" HTTP/1.1\r\nHost: "+appHost+
"\r\nUser-Agent: "+userAgent+"\r\n"+forwardedFor+": "+from+"\r\n\r\n")
err := conn.SetReadDeadline(time.Now().Add(waitLimit))
if err != nil {
s.t.Fatalf("set read deadline: %v", err)
}
got := 0
res, err := http.ReadResponse(bufio.NewReader(conn), nil)
switch {
case err == nil:
got = readAnswer(res).status
case !errors.Is(err, io.ErrUnexpectedEOF):
s.t.Fatalf("read response: %v", err)
}
_ = conn.Close()
if got != status {
s.t.Errorf("request %d, from %s: status %d, want %d", s.sent+1, from, got,
status)
}
line := s.out.requestLines(s.t, s.sent+1)[s.sent]
s.sent++
wantLine(s.t, line, status, action)
return line
}
+15
View File
@@ -0,0 +1,15 @@
package proxy
import (
"net/http"
"sneak.berlin/go/smallwebwaf/internal/bans"
)
// LedgerOf returns the ban ledger of a server New returned, so that the
// tests can read the bans' notes.
func LedgerOf(server *http.Server) *bans.Ledger {
h, _ := server.Handler.(*handler)
return h.ledger
}
+13
View File
@@ -10,6 +10,7 @@ import (
"net/http" "net/http"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/lookup" "sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
@@ -36,6 +37,9 @@ type Params struct {
// GeoJSURL is where clients' countries are looked up, normally // GeoJSURL is where clients' countries are looked up, normally
// lookup.URL. GeoJS is asked only while a country list is set. // lookup.URL. GeoJS is asked only while a country list is set.
GeoJSURL string GeoJSURL string
// Now tells the time by which requests are counted for the rate
// limits and bans are made and run out, normally time.Now.
Now func() time.Time
} }
// New returns the server smallwebwaf runs: each request it reads passes // New returns the server smallwebwaf runs: each request it reads passes
@@ -55,11 +59,18 @@ func New(params Params) *http.Server {
processLog: params.ProcessLog, processLog: params.ProcessLog,
errorLog: errorLog, errorLog: errorLog,
transport: newTransport(), transport: newTransport(),
now: params.Now,
limiter: ratelimit.New(ratelimit.Limits{ limiter: ratelimit.New(ratelimit.Limits{
PerMinute: params.Config.RateLimitPerMinute, PerMinute: params.Config.RateLimitPerMinute,
PerHour: params.Config.RateLimitPerHour, PerHour: params.Config.RateLimitPerHour,
PerDay: params.Config.RateLimitPerDay, PerDay: params.Config.RateLimitPerDay,
}), }),
ledger: bans.New(bans.Rules{
LimitBanDuration: params.Config.LimitBanDuration,
LimitBanRepeatWindow: params.Config.LimitBanRepeatWindow,
MaxBanDuration: params.Config.MaxBanDuration,
MaxBans: params.Config.MaxBans,
}),
geojs: lookup.New(lookup.Params{ geojs: lookup.New(lookup.Params{
URL: params.GeoJSURL, URL: params.GeoJSURL,
Now: time.Now, Now: time.Now,
@@ -85,7 +96,9 @@ type handler struct {
processLog *slog.Logger processLog *slog.Logger
errorLog *log.Logger errorLog *log.Logger
transport http.RoundTripper transport http.RoundTripper
now func() time.Time
limiter *ratelimit.Limiter limiter *ratelimit.Limiter
ledger *bans.Ledger
geojs *lookup.GeoJS geojs *lookup.GeoJS
} }
+22 -1
View File
@@ -57,8 +57,15 @@ const (
rateLimitExemptNets = "SWWAF_RATE_LIMIT_EXEMPT_NETS" rateLimitExemptNets = "SWWAF_RATE_LIMIT_EXEMPT_NETS"
denyNets = "SWWAF_DENY_NETS" denyNets = "SWWAF_DENY_NETS"
rateLimitPerMinute = "SWWAF_RATE_LIMIT_PER_MINUTE" rateLimitPerMinute = "SWWAF_RATE_LIMIT_PER_MINUTE"
rateLimitPerDay = "SWWAF_RATE_LIMIT_PER_DAY"
deniedCountries = "SWWAF_DENIED_COUNTRIES" deniedCountries = "SWWAF_DENIED_COUNTRIES"
allowedCountries = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES" allowedCountries = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES"
banResponse = "SWWAF_BAN_RESPONSE"
limitBanDuration = "SWWAF_LIMIT_BAN_DURATION"
limitBanRepeatWindow = "SWWAF_LIMIT_BAN_REPEAT_WINDOW"
maxBanDuration = "SWWAF_MAX_BAN_DURATION"
maxBans = "SWWAF_MAX_BANS"
banScopeV4Prefix = "SWWAF_BAN_SCOPE_V4_PREFIX"
) )
// output collects what smallwebwaf writes on stdout. // output collects what smallwebwaf writes on stdout.
@@ -183,6 +190,19 @@ func startProxyWithGeoJS(
) (string, *output) { ) (string, *output) {
t.Helper() t.Helper()
addr, out, _ := startProxyWithClock(t, appURL, geojsURL, time.Now, env)
return addr, out
}
// startProxyWithClock is startProxyWithGeoJS with requests counted and
// bans made by the time now tells, and returns the server as well.
func startProxyWithClock(
t *testing.T, appURL, geojsURL string, now func() time.Time,
env map[string]string,
) (string, *output, *http.Server) {
t.Helper()
settings := map[string]string{"SWWAF_UPSTREAM_URL": appURL} settings := map[string]string{"SWWAF_UPSTREAM_URL": appURL}
maps.Copy(settings, env) maps.Copy(settings, env)
@@ -201,6 +221,7 @@ func startProxyWithGeoJS(
RequestLog: out, RequestLog: out,
ProcessLog: requestlog.NewProcessLogger(out), ProcessLog: requestlog.NewProcessLogger(out),
GeoJSURL: geojsURL, GeoJSURL: geojsURL,
Now: now,
}) })
listener, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", localhost+":0") listener, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", localhost+":0")
@@ -216,7 +237,7 @@ func startProxyWithGeoJS(
_ = server.Close() _ = server.Close()
}) })
return listener.Addr().String(), out return listener.Addr().String(), out, server
} }
// newClient returns an HTTP client that sends requests as they are made, // newClient returns an HTTP client that sends requests as they are made,
+12 -8
View File
@@ -8,7 +8,11 @@ import (
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
func TestRateLimitRefusesWith429BeforeTheApp(t *testing.T) { // minute is the window of SWWAF_RATE_LIMIT_PER_MINUTE, as a log line's
// limit_hit names it.
const minute = "minute"
func TestRateLimitRefusesBeforeTheApp(t *testing.T) {
t.Parallel() t.Parallel()
var calls atomic.Int32 var calls atomic.Int32
@@ -24,19 +28,19 @@ func TestRateLimitRefusesWith429BeforeTheApp(t *testing.T) {
const otherClient = "203.0.113.10" const otherClient = "203.0.113.10"
// With a limit of one request a minute, a client's second request is // With a limit of one request a minute, a client's second request is
// refused. A client is one IPv4 address, or one IPv6 /64; an IPv4 // refused, with 403 by default. A client is one IPv4 address, or one
// address in IPv6 form is that IPv4 address. // IPv6 /64; an IPv4 address in IPv6 form is that IPv4 address.
requests := []struct { requests := []struct {
client string // as X-Forwarded-For names it client string // as X-Forwarded-For names it
logged string // as the log line's client_ip names it logged string // as the log line's client_ip names it
want int want int
}{ }{
{client, client, http.StatusOK}, {client, client, http.StatusOK},
{client, client, http.StatusTooManyRequests}, {client, client, http.StatusForbidden},
{otherClient, otherClient, http.StatusOK}, {otherClient, otherClient, http.StatusOK},
{"::ffff:" + otherClient, otherClient, http.StatusTooManyRequests}, {"::ffff:" + otherClient, otherClient, http.StatusForbidden},
{"2001:db8::1", "2001:db8::1", http.StatusOK}, {"2001:db8::1", "2001:db8::1", http.StatusOK},
{"2001:db8::8000:0:0:1", "2001:db8::8000:0:0:1", http.StatusTooManyRequests}, {"2001:db8::8000:0:0:1", "2001:db8::8000:0:0:1", http.StatusForbidden},
{"2001:db8:0:1::1", "2001:db8:0:1::1", http.StatusOK}, {"2001:db8:0:1::1", "2001:db8:0:1::1", http.StatusOK},
} }
@@ -53,9 +57,9 @@ func TestRateLimitRefusesWith429BeforeTheApp(t *testing.T) {
if sent.want == http.StatusOK { if sent.want == http.StatusOK {
wantLine(t, line, http.StatusOK, requestlog.ActionForward) wantLine(t, line, http.StatusOK, requestlog.ActionForward)
} else { } else {
wantLine(t, line, http.StatusTooManyRequests, requestlog.ActionRateLimited) wantLine(t, line, http.StatusForbidden, requestlog.ActionRateLimited)
if line.LimitHit != "minute" { if line.LimitHit != minute {
t.Errorf("log line has limit_hit %q, want minute", line.LimitHit) t.Errorf("log line has limit_hit %q, want minute", line.LimitHit)
} }
} }
+25 -23
View File
@@ -21,7 +21,8 @@ const flushAfterEachWrite time.Duration = -1
// refusal is smallwebwaf refusing a request, or refusing to go on with it: // refusal is smallwebwaf refusing a request, or refusing to go on with it:
// the status the client is answered if the response has not started yet, // the status the client is answered if the response has not started yet,
// and the action the log line names. // 0 to close the connection without an answer, and the action the log
// line names.
type refusal struct { type refusal struct {
status int status int
action string action string
@@ -105,39 +106,33 @@ func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request {
// is known, before its body is read or anything reaches the app. It // is known, before its body is read or anything reaches the app. It
// returns nil to let the request through. A client in SWWAF_ALLOW_NETS // returns nil to let the request through. A client in SWWAF_ALLOW_NETS
// skips every check but the size limit. For any other client, // skips every check but the size limit. For any other client,
// SWWAF_DENY_NETS comes first, so that a client it refuses is not looked // SWWAF_DENY_NETS comes first, then a ban on its netblock, so that a
// up, and then the country lists; a request either refuses is not counted // client either refuses is not looked up, and then the country lists; a
// for the rate limits. Then come the rate limits, unless the client is in // request any of them refuses is not counted for the rate limits. Then
// come the rate limits, unless the client is in
// SWWAF_RATE_LIMIT_EXEMPT_NETS, so that every other request is counted, // SWWAF_RATE_LIMIT_EXEMPT_NETS, so that every other request is counted,
// one refused for its size too. ctx is the request's own context. // one refused for its size too. Every refusal but the size limit's is
// answered with SWWAF_BAN_RESPONSE. ctx is the request's own context.
func (rq *request) check(ctx context.Context) *refusal { func (rq *request) check(ctx context.Context) *refusal {
cfg := rq.h.config cfg := rq.h.config
allowed := isInside(rq.client, cfg.AllowNets) allowed := isInside(rq.client, cfg.AllowNets)
exempt := isInside(rq.client, cfg.RateLimitExemptNets)
now := rq.h.now()
if !allowed && isInside(rq.client, cfg.DenyNets) { if !allowed && isInside(rq.client, cfg.DenyNets) {
return &refusal{ return rq.banResponse(requestlog.ActionDenied)
status: http.StatusForbidden, }
action: requestlog.ActionDenied,
} if !allowed && rq.banned(now) {
return rq.banResponse(requestlog.ActionBanned)
} }
if !allowed && rq.countryDenied(ctx) { if !allowed && rq.countryDenied(ctx) {
return &refusal{ return rq.banResponse(requestlog.ActionCountryDenied)
status: http.StatusForbidden,
action: requestlog.ActionCountryDenied,
}
} }
if !allowed && !isInside(rq.client, cfg.RateLimitExemptNets) { if !allowed && !exempt && rq.limitBroken(now) {
limitHit := rq.h.limiter.Count(clientGroup(rq.client), rq.start) return rq.banResponse(requestlog.ActionRateLimited)
if limitHit != "" {
rq.line.LimitHit = limitHit
return &refusal{
status: http.StatusTooManyRequests,
action: requestlog.ActionRateLimited,
}
}
} }
maxBytes := cfg.RequestMaxBytes maxBytes := cfg.RequestMaxBytes
@@ -250,6 +245,13 @@ func (rq *request) answer(r refusal) {
return // too late to answer: the connection can only be cut return // too late to answer: the connection can only be cut
} }
if r.status == 0 {
// SWWAF_BAN_RESPONSE is close. This panic has Go's server close
// the connection without an answer, and log nothing; the log line
// is still written as the handler returns.
panic(http.ErrAbortHandler)
}
// A client found too slow is read no more; any other may go on // A client found too slow is read no more; any other may go on
// sending until its time is up, so that Go's server can read the // sending until its time is up, so that Go's server can read the
// rest of the body and end the request cleanly. // rest of the body and end the request cleanly.
+3 -3
View File
@@ -78,7 +78,7 @@ func TestRequestFromAllowNetsIsNotCounted(t *testing.T) {
{listedAddr, http.StatusOK, requestlog.ActionForward}, {listedAddr, http.StatusOK, requestlog.ActionForward},
{listedAddr, http.StatusOK, requestlog.ActionForward}, {listedAddr, http.StatusOK, requestlog.ActionForward},
{unlistedAddr, http.StatusOK, requestlog.ActionForward}, {unlistedAddr, http.StatusOK, requestlog.ActionForward},
{unlistedAddr, http.StatusTooManyRequests, requestlog.ActionRateLimited}, {unlistedAddr, http.StatusForbidden, requestlog.ActionRateLimited},
}) })
} }
@@ -133,7 +133,7 @@ func TestRequestRefusedByDenyNetsIsNotCounted(t *testing.T) {
{listedAddr, http.StatusForbidden, requestlog.ActionDenied}, {listedAddr, http.StatusForbidden, requestlog.ActionDenied},
{listedAddr, http.StatusForbidden, requestlog.ActionDenied}, {listedAddr, http.StatusForbidden, requestlog.ActionDenied},
{unlistedAddr, http.StatusOK, requestlog.ActionForward}, {unlistedAddr, http.StatusOK, requestlog.ActionForward},
{unlistedAddr, http.StatusTooManyRequests, requestlog.ActionRateLimited}, {unlistedAddr, http.StatusForbidden, requestlog.ActionRateLimited},
}) })
} }
@@ -156,7 +156,7 @@ func TestRateLimitExemptNetsAreNeitherCountedNorRefused(t *testing.T) {
{listedAddr, http.StatusOK, requestlog.ActionForward}, {listedAddr, http.StatusOK, requestlog.ActionForward},
{listedAddr, http.StatusOK, requestlog.ActionForward}, {listedAddr, http.StatusOK, requestlog.ActionForward},
{unlistedAddr, http.StatusOK, requestlog.ActionForward}, {unlistedAddr, http.StatusOK, requestlog.ActionForward},
{unlistedAddr, http.StatusTooManyRequests, requestlog.ActionRateLimited}, {unlistedAddr, http.StatusForbidden, requestlog.ActionRateLimited},
{fromKP, http.StatusForbidden, requestlog.ActionCountryDenied}, {fromKP, http.StatusForbidden, requestlog.ActionCountryDenied},
}) })
} }
+27 -8
View File
@@ -54,11 +54,22 @@ func New(limits Limits) *Limiter {
} }
} }
// Hit is a request that takes a client over a rate limit.
type Hit struct {
// Window is "minute", "hour" or "day".
Window string
// Limit is the window's limit.
Limit int64
// Requests is the client's requests counted in the window, this one
// included.
Requests float64
}
// Count counts a request from client at now, in every window, whether or // Count counts a request from client at now, in every window, whether or
// not it is refused. It returns the window whose limit the request takes // not it is refused. It reports whether the request takes the client over
// the client over, "minute", "hour" or "day", the shortest if it is over // a limit, and the window whose limit it goes over, the shortest if it is
// several, or "" if it is within every limit. // over several.
func (l *Limiter) Count(client netip.Prefix, now time.Time) string { func (l *Limiter) Count(client netip.Prefix, now time.Time) (Hit, bool) {
l.mu.Lock() l.mu.Lock()
defer l.mu.Unlock() defer l.mu.Unlock()
@@ -68,16 +79,24 @@ func (l *Limiter) Count(client netip.Prefix, now time.Time) string {
l.clients.Add(client, counts) l.clients.Add(client, counts)
} }
limitHit := "" var hit Hit
for i, w := range l.windows { for i, w := range l.windows {
requests := counts[i].add(now, w.length) requests := counts[i].add(now, w.length)
if limitHit == "" && w.limit > 0 && requests > float64(w.limit) { if hit.Window == "" && w.limit > 0 && requests > float64(w.limit) {
limitHit = w.name hit = Hit{Window: w.name, Limit: w.limit, Requests: requests}
} }
} }
return limitHit return hit, hit.Window != ""
}
// Reset sets client's counts in every window back to zero.
func (l *Limiter) Reset(client netip.Prefix) {
l.mu.Lock()
defer l.mu.Unlock()
l.clients.Remove(client)
} }
// window is a length of time over which requests are counted, and the // window is a length of time over which requests are counted, and the
+49 -3
View File
@@ -54,6 +54,52 @@ func TestEachWindowRefusesAtItsLimitAndLetsTheClientBack(t *testing.T) {
} }
} }
func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit, PerHour: limit})
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
for range limit {
_, over := limiter.Count(client, start)
if over {
t.Fatal("a request within the limit is over it")
}
}
// Over both limits; the minute's is named, with the four requests.
hit, over := limiter.Count(client, start)
want := ratelimit.Hit{Window: minute, Limit: limit, Requests: limit + 1}
if !over || hit != want {
t.Errorf("request over the limit gives %+v and %t, want %+v and true",
hit, over, want)
}
}
func TestResetSetsTheCountsBackToZero(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit, PerDay: limit})
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
for range limit {
wantCount(t, limiter, client, start, "")
}
wantCount(t, limiter, client, start, minute)
limiter.Reset(client)
// At the same moment, the client has its whole allowance again.
for range limit {
wantCount(t, limiter, client, start, "")
}
wantCount(t, limiter, client, start, minute)
}
func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) { func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) {
t.Parallel() t.Parallel()
@@ -192,9 +238,9 @@ func wantCount(
) { ) {
t.Helper() t.Helper()
got := limiter.Count(client, now) hit, _ := limiter.Count(client, now)
if got != want { if hit.Window != want {
t.Errorf("request from %s at %s is over %q, want %q", t.Errorf("request from %s at %s is over %q, want %q",
client, now.Format(time.RFC3339), got, want) client, now.Format(time.RFC3339), hit.Window, want)
} }
} }
+12 -1
View File
@@ -24,8 +24,10 @@ const (
// for, or whose answer could not be passed on. // for, or whose answer could not be passed on.
ActionUpstreamError = "upstream_error" ActionUpstreamError = "upstream_error"
// ActionRateLimited is a request refused because it took its client // ActionRateLimited is a request refused because it took its client
// over a rate limit, or came while the client was over one. // over a rate limit, which bans the client.
ActionRateLimited = "rate_limited" ActionRateLimited = "rate_limited"
// ActionBanned is a request refused because a ban covers its client.
ActionBanned = "banned"
// ActionDenied is a request refused because its client is in // ActionDenied is a request refused because its client is in
// SWWAF_DENY_NETS. // SWWAF_DENY_NETS.
ActionDenied = "denied" ActionDenied = "denied"
@@ -36,6 +38,10 @@ const (
ActionAdmin = "admin" ActionAdmin = "admin"
) )
// OffenceLimit is the offence a request line names for a request that
// broke a rate limit.
const OffenceLimit = "limit"
// timeLayout is RFC 3339 with milliseconds. // timeLayout is RFC 3339 with milliseconds.
const timeLayout = "2006-01-02T15:04:05.000Z07:00" const timeLayout = "2006-01-02T15:04:05.000Z07:00"
@@ -64,6 +70,11 @@ type Line struct {
// LimitHit is the window whose rate limit the request went over: // LimitHit is the window whose rate limit the request went over:
// minute, hour or day. // minute, hour or day.
LimitHit string `json:"limit_hit,omitempty"` LimitHit string `json:"limit_hit,omitempty"`
// Offence is the offence the request was held as, OffenceLimit.
Offence string `json:"offence,omitempty"`
// BanExpires is when the ban the request made, or was refused under,
// ends: a time, or "permanent".
BanExpires string `json:"ban_expires,omitempty"`
// Aborted is true when the client went away early. // Aborted is true when the client went away early.
Aborted bool `json:"aborted,omitempty"` Aborted bool `json:"aborted,omitempty"`
// DurationTotal and DurationUpstreamTotal are in milliseconds. // DurationTotal and DurationUpstreamTotal are in milliseconds.
+2 -1
View File
@@ -50,7 +50,8 @@ func TestWriteWritesOneJSONLineMarkedRequest(t *testing.T) {
} }
unset := []string{ unset := []string{
"upstream_status", "limit_hit", "aborted", "duration_upstream_total", "upstream_status", "limit_hit", "offence", "ban_expires", "aborted",
"duration_upstream_total",
} }
for _, name := range unset { for _, name := range unset {
_, present := fields[name] _, present := fields[name]
+1
View File
@@ -80,6 +80,7 @@ func Run(ctx context.Context, params Params) int {
RequestLog: params.Stdout, RequestLog: params.Stdout,
ProcessLog: processLog, ProcessLog: processLog,
GeoJSURL: lookup.URL, GeoJSURL: lookup.URL,
Now: time.Now,
}) })
processLog.Info("starting", processLog.Info("starting",
+6
View File
@@ -196,6 +196,12 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL string) {
"SWWAF_RATE_LIMIT_PER_DAY": "50000", "SWWAF_RATE_LIMIT_PER_DAY": "50000",
"SWWAF_DENIED_COUNTRIES": "", "SWWAF_DENIED_COUNTRIES": "",
"SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES": "", "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES": "",
"SWWAF_BAN_RESPONSE": "403",
"SWWAF_LIMIT_BAN_DURATION": "1h",
"SWWAF_LIMIT_BAN_REPEAT_WINDOW": "24h",
"SWWAF_MAX_BAN_DURATION": "7d",
"SWWAF_MAX_BANS": "5000",
"SWWAF_BAN_SCOPE_V4_PREFIX": "32",
} }
for name, value := range want { for name, value := range want {