Ban the netblock of a client that breaks a rate limit, in memory (closes #18)
check / check (push) Successful in 4m11s

A request over a rate limit is refused with SWWAF_BAN_RESPONSE and bans
the client's netblock: an hour at first, three times the last ban when
broken again within a day of its end, permanent past seven days. The
ban ledger in internal/bans is checked after the static lists and
before the lookup, and the requests it refuses are not counted. A ban
resets the client's counters and carries notes holding the request
that broke the limit, as SPEC.md now says. At most SWWAF_MAX_BANS are
held. SWWAF_BAN_RESPONSE also answers SWWAF_DENY_NETS and the country
lists.

Judgement call: the six ban settings cannot be off.
Judgement call: a permanent ban's ban_expires is "permanent".

Model: opus-5-5
This commit is contained in:
2026-10-06 03:12:48 +00:00
parent 0f85c9ae07
commit cdaa2a0aca
20 changed files with 1522 additions and 127 deletions
+110 -67
View File
@@ -13,16 +13,17 @@ JSON log line for every request.
Status: the first two milestones are built
(https://git.eeqj.de/sneak/smallwebwaf/issues/13 and
https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are two parts of
milestone 3: the static lists, which come next in the build order, and the
header size and the idle time as settings, which come last in it. `smallwebwaf`
passes each request to the app and the app's answer back, unchanged, within its
timeouts and size limits, works out each client's address, refuses a client that
sends too many requests, comes from a country you refuse or from a network you
refuse, lets the networks you choose through, and writes a JSON log line for
every request. It comes as the image the app's own image is built on. The rest
of the design comes after that, in the order of the build order in
[`SPEC.md`](SPEC.md). The survey of existing tools that led to the design is in
https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are three parts of
milestone 3: the static lists and the bans that broken rate limits lead to,
which come next in the build order, and the header size and the idle time as
settings, which come last in it. `smallwebwaf` passes each request to the app
and the app's answer back, unchanged, within its timeouts and size limits, works
out each client's address, bans a client that sends too many requests, refuses a
client that comes from a country you refuse or from a network you refuse, lets
the networks you choose through, and writes a JSON log line for every request.
It comes as the image the app's own image is built on. The rest of the design
comes after that, in the order of the build order in [`SPEC.md`](SPEC.md). The
survey of existing tools that led to the design is in
[`EVALUATION.md`](EVALUATION.md).
## Getting started
@@ -71,31 +72,48 @@ and `make run` builds and runs it, listening on port 8080 in front of an app at
to take what it had. Once the response has started, a limit can only cut the
connection.
- Counts each client's requests over a minute, an hour and a day. A request that
takes the client over one of the rate limits below is refused with `429`
before anything reaches the app, and so is each request after it until the
client is back under every limit. A client is one IPv4 address, or one IPv6
/64, since one abuser usually holds a whole /64. Refused requests count too,
so a client that keeps sending too fast stays refused until it slows down.
Each window is counted in two fixed buckets, the earlier one weighted by how
much of it the window still covers. At most 20,000 clients are kept, the least
recently seen dropped first, and only in memory: a restart starts every client
afresh.
- Refuses a request from a country you refuse with `403`, as soon as the
client's country is known and before its body is read; such a request is not
counted for the rate limits. While one of the country lists below is set, each
client's country is looked up through GeoJS (see "Country and AS number
lookup" below); with neither set, no visitor's address leaves the host. A
client on a private, loopback or link-local address has no country and is
takes the client over one of the rate limits below is refused with
`SWWAF_BAN_RESPONSE`, `403` by default, before anything reaches the app, and
bans the client. A client is one IPv4 address, or one IPv6 /64, since one
abuser usually holds a whole /64. Each window is counted in two fixed buckets,
the earlier one weighted by how much of it the window still covers. At most
20,000 clients are kept, the least recently seen dropped first, and only in
memory: a restart starts every client afresh.
- Bans a client that breaks a rate limit, as "Bans" in [`SPEC.md`](SPEC.md)
describes: the first ban lasts an hour, and a limit broken again within a day
of a ban ending bans for three times as long as that ban, so 1, 3, 9, 27 and
81 hours; a ban that would last longer than seven days is permanent instead. A
ban covers the client's netblock: its IPv4 address, or the netblock around it
that `SWWAF_BAN_SCOPE_V4_PREFIX` sets, or its IPv6 /64. While it lasts, every
request from the netblock is refused with `SWWAF_BAN_RESPONSE` after the
static lists and before the country lists, so the client is not looked up, and
is not counted for the rate limits. A ban sets the client's counters back to
zero. Each ban carries notes for deciding whether to lift it: the limit, its
window and the requests counted in it, the request that broke it, the client's
country when it was looked up, how many requests the ban has refused, and how
many bans the netblock had before. At most `SWWAF_MAX_BANS` bans are kept,
past, active and permanent; past that, the earliest ban of the netblock that
has gone longest without a request is dropped first. Bans and their notes are
kept in memory only, so a restart lifts every ban, and nothing shows them yet:
`bans.json`, which shows them and lets you lift a ban, comes with the state
files (https://git.eeqj.de/sneak/smallwebwaf/issues/17).
- Refuses a request from a country you refuse with `SWWAF_BAN_RESPONSE`, as soon
as the client's country is known and before its body is read; such a request
is not counted for the rate limits. While one of the country lists below is
set, each client's country is looked up through GeoJS (see "Country and AS
number lookup" below); with neither set, no visitor's address leaves the host.
A client on a private, loopback or link-local address has no country and is
never looked up: `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses it unless it is
in `SWWAF_ALLOW_NETS`, and `SWWAF_DENIED_COUNTRIES` does not refuse it.
- Checks the client's own address against the static lists, the three netblock
settings below, before anything else, its country included. A client in
`SWWAF_ALLOW_NETS` skips the country lists and the rate limits, and is not
looked up; the timeouts and size limits still apply. A client in
`SWWAF_DENY_NETS` is refused with `403` before its body is read, and the
request is not counted for the rate limits; an address in `SWWAF_ALLOW_NETS`
too is let through. A client in `SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither
counted nor refused by the rate limits; the country lists still apply to it.
`SWWAF_ALLOW_NETS` skips bans, the country lists and the rate limits, and is
not looked up; the timeouts and size limits still apply. A client in
`SWWAF_DENY_NETS` is refused with `SWWAF_BAN_RESPONSE` before its body is
read, and the request is not counted for the rate limits; an address in
`SWWAF_ALLOW_NETS` too is let through. A client in
`SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither counted nor refused by the rate
limits; the country lists and bans still apply to it.
- Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any
check and without asking the app, for the image's health check.
- Writes a line in the request log for each request (see "Request log" below).
@@ -133,8 +151,9 @@ it, and the effective settings are logged at start.
to send its whole answer, from the end of the request to the last byte.
- `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body.
- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body.
- `SWWAF_ALLOW_NETS` (default empty): netblocks whose clients skip the country
lists and the rate limits, such as your monitoring or your own networks.
- `SWWAF_ALLOW_NETS` (default empty): netblocks whose clients skip bans, the
country lists and the rate limits, such as your monitoring or your own
networks.
- `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients the
rate limits do not apply to, such as a machine that talks to the app all day.
- `SWWAF_DENY_NETS` (default empty): netblocks whose clients are always refused.
@@ -149,6 +168,22 @@ it, and the effective settings are logged at start.
countries whose clients get through, for example `us,de`. A client whose
country cannot be found is refused too, so that new clients are not let in
whenever GeoJS stops answering.
- `SWWAF_BAN_RESPONSE` (default `403`): how a refused client is answered, one
that is banned, breaks a rate limit, is in `SWWAF_DENY_NETS` or comes from a
refused country: `403`, `429`, or `close` to close the connection without an
answer. Behind traefik, `close` does not leave the client unanswered: traefik
answers `502`, as it does whenever its backend drops a connection.
- `SWWAF_LIMIT_BAN_DURATION` (default `1h`): the ban for a first broken rate
limit.
- `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): a rate limit broken again
within this time after a ban ended bans for three times as long as that ban.
- `SWWAF_MAX_BAN_DURATION` (default `7d`): a ban that would be longer is
permanent instead.
- `SWWAF_MAX_BANS` (default `5000`): the most bans kept, past, active and
permanent.
- `SWWAF_BAN_SCOPE_V4_PREFIX` (default `32`): the length of the netblock around
an IPv4 client that a ban covers, such as `24` to ban the surrounding /24. An
IPv6 ban covers the client's /64.
Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are
bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024
@@ -157,8 +192,8 @@ and a bare address stands for itself alone. Countries are the two-letter codes
ISO 3166-1 assigns today, and `xk` for Kosovo, in either case (`de` and `DE` are
the same); any other code, such as `nk` (North Korea is `kp`) or the withdrawn
`su`, stops the start, and so does a code on both country lists. `off` switches
a timeout, a size limit or a rate limit off; only
`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES` cannot be off.
a timeout, a size limit or a rate limit off;
`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES` and the ban settings cannot be off.
Several limits are fixed rather than settings. At most 20,000 clients are kept
for the rate limits, and an IPv6 client is counted by its /64. A new client
@@ -176,23 +211,27 @@ refused ones included:
- `time` is when the request arrived, in UTC. `peer_ip` is the TCP peer,
normally traefik. `path` and `query` are as the client sent them.
- `country` is the client's country as GeoJS places it, and empty when it is not
known: with neither country list set, for a client in `SWWAF_ALLOW_NETS` or
`SWWAF_DENY_NETS`, for a client on a private, loopback or link-local address,
and when GeoJS cannot place the client or has not answered in time.
- `country` is the client's country as GeoJS places it. It is empty with neither
country list set, for a client in `SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS`, for
a client on a private, loopback or link-local address, when GeoJS cannot place
the client or has not answered in time, and for a request refused because a
ban covers its client, even when the client's country is known.
- `status` is what the client was sent, `0` if nothing was; `upstream_status` is
what the app answered, and is left out when the app did not answer.
- `request_bytes` and `response_bytes` count body bytes.
- `action` is `forward` for a request passed to the app, `denied` for one
refused because its client is in `SWWAF_DENY_NETS`, `country_denied` for one
refused for its client's country, `rate_limited` for one refused for a rate
limit, `too_large` for a request or response over its size limit, `timed_out`
for one that ran out of time, `upstream_error` when the app could not be
reached or its answer broke off, and `admin` for one `smallwebwaf` answered at
its own endpoint.
- `limit_hit` is there for a request refused for a rate limit, and names the
refused because its client is in `SWWAF_DENY_NETS`, `banned` for one refused
because a ban covers its client, `country_denied` for one refused for its
client's country, `rate_limited` for one that broke a rate limit and banned
its client, `too_large` for a request or response over its size limit,
`timed_out` for one that ran out of time, `upstream_error` when the app could
not be reached or its answer broke off, and `admin` for one `smallwebwaf`
answered at its own endpoint.
- `limit_hit` is there for a request that broke a rate limit, and names the
window whose limit it went over: `minute`, `hour` or `day`, the shortest if it
went over several.
went over several. `offence` is then `limit`.
- `ban_expires` is there for a request that made a ban or was refused under one,
and gives when the ban ends, in the same form as `time`, or `permanent`.
- `aborted` is there, and true, when the client went away early.
- `duration_total` and `duration_upstream_total` are in milliseconds.
@@ -309,8 +348,8 @@ goes through the candidates one by one.
nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks
up the change. Nothing is read from disk while serving a request. The files
come in milestone 3 or later (see the build order in [`SPEC.md`](SPEC.md));
until then the rate counters and the GeoJS answers are kept in memory only,
and a restart loses them.
until then the rate counters, the bans and the GeoJS answers are kept in
memory only, and a restart loses them.
- Health checks, the metrics, and listing, adding and lifting bans or asking why
a given address was refused, all on the one port every request uses: under
`/_smallwebwaf/` on the app's own address, through traefik like any other
@@ -427,17 +466,18 @@ the metrics, failure behaviour and the build order.
So far `smallwebwaf` looks up only the country, only through GeoJS, and only
while `SWWAF_DENIED_COUNTRIES` or `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` is set:
then the address of every new visitor outside `SWWAF_ALLOW_NETS` and
`SWWAF_DENY_NETS` is sent to GeoJS, and with neither set, none is. An IPv6
visitor is asked about by the first address of its /64. A new visitor waits at
most a second for its answer, and without one counts as coming from an unknown
country until the answer arrives. The addresses waiting are asked about
together, up to 200 in one request, one request at a time; at most 10,000
visitors wait, and one more counts as coming from an unknown country until there
is room. While GeoJS fails, visitors with a kept answer are unaffected and new
ones count as coming from an unknown country. GeoJS is then left alone for a
second, twice as long after each further failure up to five minutes, and asked
again by the next request that needs it.
then the address of every new visitor is sent to GeoJS, except a visitor in
`SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS` and one refused because a ban covers its
netblock, and with neither set, none is. An IPv6 visitor is asked about by the
first address of its /64. A new visitor waits at most a second for its answer,
and without one counts as coming from an unknown country until the answer
arrives. The addresses waiting are asked about together, up to 200 in one
request, one request at a time; at most 10,000 visitors wait, and one more
counts as coming from an unknown country until there is room. While GeoJS fails,
visitors with a kept answer are unaffected and new ones count as coming from an
unknown country. GeoJS is then left alone for a second, twice as long after each
further failure up to five minutes, and asked again by the next request that
needs it.
In the full design, `smallwebwaf` looks up the AS number and country of every
client, for the request log, the metrics and the ban notes, and for the country
@@ -488,8 +528,10 @@ addresses are never sent to GeoJS.
standard library's `httputil.ReverseProxy` within the timeouts and size
limits, and writes the request's log line. Its `check` method is where a
request is refused before anything reaches the app: for `SWWAF_DENY_NETS`, for
the country lists, for a rate limit, and for an announced body over the size
limit.
a ban, for the country lists, for a rate limit, which bans the client, and for
an announced body over the size limit.
- `internal/bans`: the ban ledger: each netblock's bans with their notes, how
long a new ban lasts, and which ban is dropped when `SWWAF_MAX_BANS` are held.
- `internal/lookup`: looks up each client's country through GeoJS, and keeps the
answers.
- `internal/ratelimit`: counts each client's requests and tells when one takes
@@ -503,8 +545,9 @@ addresses are never sent to GeoJS.
checks.
Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the
table of clients to 20,000 and the GeoJS answers to 100,000, dropping the least
recently seen. The country codes are the list in `internal/config/config.go`.
table of clients to 20,000, the GeoJS answers to 100,000 and the banned
netblocks to `SWWAF_MAX_BANS`, dropping the least recently seen. The country
codes are the list in `internal/config/config.go`.
## Entrypoints
@@ -544,9 +587,9 @@ so that they run in minimal containers.
## TODO
- The rest of milestone 3, from the bans that broken request limits lead to
through the metrics endpoint, and the rest of the design, in the order of the
build order in [`SPEC.md`](SPEC.md).
- The rest of milestone 3, after the bans that broken rate limits lead to and up
to the metrics endpoint, and the rest of the design, in the order of the build
order in [`SPEC.md`](SPEC.md).
## Documents
+10 -10
View File
@@ -948,9 +948,9 @@ and the running `smallwebwaf` takes the edit in.
- what was broken: the rule ids and target that matched, or the limit, its
window, the count reached and the client's limit percentage with what set
it; and any reputation sources that listed the client;
- the requests that caused the ban, up to the last ten: time, method, host,
path with its query string, status and user agent, each text cut to 256
bytes;
- the request that caused the ban, the one that broke the limit or carried
the clear sign of attack: time, method, host, path with its query string,
status and user agent, each text cut to 256 bytes;
- how many requests counted toward the ban, and the time span over which
they came;
- the netblock's total requests since it was first seen, and the requests
@@ -961,13 +961,13 @@ and the running `smallwebwaf` takes the edit in.
the table is full, so on a public service the file grows to the default
`SWWAF_MAX_TRACKED_CLIENTS` of 20,000, about 20 MiB. Written every 15
minutes, that is under 2 GiB of disk writes a day.
- `bans.json` takes about 2 KiB per ban and at most about 8 KiB, since the
texts in the notes are cut short. At the default `SWWAF_MAX_BANS` of 5,000
it is about 10 MiB, and never more than about 40 MiB, plus whatever bans
an admin made. It is written when a ban is made, lifted or made permanent,
at most once every 10 seconds, and otherwise with the 15-minute write, so
its writes follow the bans made: with a full file, a hundred new bans a
day come to about 1 GiB of disk writes.
- `bans.json` takes about 1.2 KiB per ban and at most about 2.5 KiB, since
the notes hold one request and their texts are cut short. At the default
`SWWAF_MAX_BANS` of 5,000 it is about 6 MiB, and never more than about 12
MiB, plus whatever bans an admin made. It is written when a ban is made,
lifted or made permanent, at most once every 10 seconds, and otherwise
with the 15-minute write, so its writes follow the bans made: with a full
file, a hundred new bans a day come to about 600 MiB of disk writes.
- `lookups.json` takes about 150 bytes per answer, about 15 MiB when full.
Written every 15 minutes, that is under 1.5 GiB of disk writes a day.
- `reputation.json` and `alerts.json` are usually a few MiB or less.
+255
View File
@@ -0,0 +1,255 @@
// Package bans is the ban ledger: the bans smallwebwaf makes on the
// netblocks of clients that break a rate limit, with their notes, as the
// "Bans" section of SPEC.md describes. The bans are kept in memory only.
package bans
import (
"net/netip"
"slices"
"strings"
"sync"
"time"
"github.com/hashicorp/golang-lru/v2/simplelru"
)
// repeatFactor is how many times as long as the netblock's last ban a ban
// for a limit broken again within the repeat window lasts.
const repeatFactor = 3
// maxTextBytes is how much of each text in a ban's notes is kept.
const maxTextBytes = 256
// Rules are how long a ban for a broken limit lasts, and how many bans
// are held.
type Rules struct {
// LimitBanDuration is how long a first ban lasts.
LimitBanDuration time.Duration
// LimitBanRepeatWindow is how soon after the netblock's last ban
// ended a broken limit counts as a repeat, which bans for
// repeatFactor times as long as that ban.
LimitBanRepeatWindow time.Duration
// MaxBanDuration is the longest ban; a ban that would be longer is
// permanent instead.
MaxBanDuration time.Duration
// MaxBans is the most bans held, at least one. Past it, the earliest
// ban of the netblock that has gone longest without a request is
// dropped.
MaxBans int
}
// Ban is a ban on a netblock for a broken limit, the only kind of ban
// smallwebwaf makes so far.
type Ban struct {
Netblock netip.Prefix
Start time.Time
// Expires is when the ban ends, zero for a permanent ban.
Expires time.Time
Notes Notes
}
// Permanent reports whether the ban never runs out.
func (b Ban) Permanent() bool {
return b.Expires.IsZero()
}
// ActiveAt reports whether the ban refuses requests at now.
func (b Ban) ActiveAt(now time.Time) bool {
return b.Permanent() || now.Before(b.Expires)
}
// Notes are what an admin needs to decide whether to lift a ban.
type Notes struct {
// Country is the client's country, when it was looked up.
Country string
// Limit, Window and Count are the limit that was broken, its window,
// "minute", "hour" or "day", and the count reached: the client's
// requests in the window, the one that broke the limit included.
// These are the requests that counted toward the ban, and the window
// is the time over which they came.
Limit int64
Window string
Count float64
// Request is the request that broke the limit.
Request Request
// Refused is how many requests the ban has refused so far.
Refused int64
// EarlierBans is how many bans the netblock had before this one.
EarlierBans int
}
// Request is a request in a ban's notes. Each text is cut to 256 bytes.
type Request struct {
Time time.Time
Method string
Host string
// Path is the path with its query string.
Path string
// Status is what the client was sent, 0 if nothing was.
Status int
UserAgent string
}
// Ledger holds the bans. It is safe for concurrent use.
type Ledger struct {
rules Rules
mu sync.Mutex
// netblocks holds each banned netblock's bans, oldest first. Each
// request from a netblock makes it the most recently seen.
netblocks *simplelru.LRU[netip.Prefix, *[]Ban]
// held is how many bans netblocks holds, at most rules.MaxBans.
held int
}
// New returns a Ledger with no ban yet.
func New(rules Rules) *Ledger {
// Every netblock held has a ban, so there are never more netblocks
// than rules.MaxBans, and the LRU never drops one itself.
netblocks, err := simplelru.NewLRU[netip.Prefix, *[]Ban](rules.MaxBans, nil)
if err != nil {
panic(err) // NewLRU fails only for a size below one
}
return &Ledger{rules: rules, netblocks: netblocks}
}
// Check is called for each request from netblock, at now. It reports
// whether a ban on netblock is active, and returns that ban, with the
// request counted among those it refused.
func (l *Ledger) Check(netblock netip.Prefix, now time.Time) (Ban, bool) {
l.mu.Lock()
defer l.mu.Unlock()
bans, found := l.netblocks.Get(netblock)
if !found {
return Ban{}, false
}
// A ban is made only once the one before has ended, so only the last
// can be active.
last := &(*bans)[len(*bans)-1]
if !last.ActiveAt(now) {
return Ban{}, false
}
last.Notes.Refused++
return *last, true
}
// BanForLimit bans netblock at now for a broken limit, with notes, and
// returns the ban. A first ban lasts LimitBanDuration. A ban made within
// LimitBanRepeatWindow after the netblock's last ban ended lasts
// repeatFactor times as long as that one. A ban that would be longer
// than MaxBanDuration is permanent instead. If a ban on netblock is still
// active, as when two of its requests break a limit at once, that ban is
// returned and no other is made. The ledger fills in the notes' Refused
// and EarlierBans itself.
func (l *Ledger) BanForLimit(netblock netip.Prefix, now time.Time, notes Notes) Ban {
l.mu.Lock()
defer l.mu.Unlock()
var last *Ban
bans, found := l.netblocks.Get(netblock)
if found {
last = &(*bans)[len(*bans)-1]
if last.ActiveAt(now) {
return *last
}
notes.EarlierBans = last.Notes.EarlierBans + 1
}
notes.Request = notes.Request.cut()
ban := Ban{
Netblock: netblock,
Start: now,
Expires: l.expiry(last, now),
Notes: notes,
}
if l.held == l.rules.MaxBans {
l.dropOne()
}
// dropOne can have dropped netblock's last ban, and netblock with it.
bans, found = l.netblocks.Peek(netblock)
if !found {
bans = &[]Ban{}
l.netblocks.Add(netblock, bans)
}
*bans = append(*bans, ban)
l.held++
return ban
}
// Bans returns the bans held on netblock, oldest first. It is not a
// request from netblock, and leaves when it was last seen unchanged.
func (l *Ledger) Bans(netblock netip.Prefix) []Ban {
l.mu.Lock()
defer l.mu.Unlock()
bans, found := l.netblocks.Peek(netblock)
if !found {
return nil
}
return slices.Clone(*bans)
}
// expiry returns when a ban for a broken limit made at now ends, or zero
// when it is permanent. last is the netblock's last ban, which has ended,
// or nil when it has none.
func (l *Ledger) expiry(last *Ban, now time.Time) time.Time {
length := l.rules.LimitBanDuration
if last != nil && now.Sub(last.Expires) <= l.rules.LimitBanRepeatWindow {
lastLength := last.Expires.Sub(last.Start)
// This is repeatFactor * lastLength > MaxBanDuration, written so
// that it cannot overflow.
if lastLength > l.rules.MaxBanDuration/repeatFactor {
return time.Time{}
}
length = repeatFactor * lastLength
}
if length > l.rules.MaxBanDuration {
return time.Time{}
}
return now.Add(length)
}
// dropOne drops the earliest ban of the netblock that has gone longest
// without a request, and the netblock with it if that was its only ban.
func (l *Ledger) dropOne() {
netblock, bans, _ := l.netblocks.GetOldest()
if len(*bans) == 1 {
l.netblocks.Remove(netblock)
} else {
*bans = slices.Delete(*bans, 0, 1)
}
l.held--
}
// cut returns r with each text cut to maxTextBytes and copied, so that
// the notes do not keep the rest of the request in memory.
func (r Request) cut() Request {
r.Method = cutText(r.Method)
r.Host = cutText(r.Host)
r.Path = cutText(r.Path)
r.UserAgent = cutText(r.UserAgent)
return r
}
// cutText returns a copy of the first maxTextBytes of text.
func cutText(text string) string {
return strings.Clone(text[:min(len(text), maxTextBytes)])
}
+267
View File
@@ -0,0 +1,267 @@
package bans_test
import (
"net/netip"
"strings"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
)
const day = 24 * time.Hour
func TestRepeatsTripleUntilPermanent(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultRules())
netblock := netip.MustParsePrefix("203.0.113.9/32")
now := midnight()
// Each ban is followed by another as soon as it ends: 1, 3, 9, 27 and
// 81 hours.
for i, hours := range []int{1, 3, 9, 27, 81} {
ban := ledger.BanForLimit(netblock, now, bans.Notes{})
length := time.Duration(hours) * time.Hour
if !ban.Expires.Equal(now.Add(length)) || ban.Notes.EarlierBans != i {
t.Fatalf("ban %d lasts %s with %d earlier bans, want %d hours and %d",
i+1, ban.Expires.Sub(now), ban.Notes.EarlierBans, hours, i)
}
now = ban.Expires
}
// The sixth would last 243 hours, more than seven days: it is
// permanent, and never ends.
ban := ledger.BanForLimit(netblock, now, bans.Notes{})
if !ban.Permanent() {
t.Fatalf("sixth ban ends at %s, want a permanent one", ban.Expires)
}
_, banned := ledger.Check(netblock, now.Add(100*365*day))
if !banned {
t.Error("a permanent ban ended")
}
}
func TestRepeatWindowRunsOut(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
// gap is the time between the end of the first ban and the second.
gap time.Duration
want time.Duration
}{
{"broken again as the window ends", day, 3 * time.Hour},
{"broken again after the window", day + time.Nanosecond, time.Hour},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultRules())
netblock := netip.MustParsePrefix("203.0.113.9/32")
first := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
second := ledger.BanForLimit(netblock, first.Expires.Add(tc.gap), bans.Notes{})
if second.Expires.Sub(second.Start) != tc.want || second.Notes.EarlierBans != 1 {
t.Errorf("second ban lasts %s with %d earlier bans, want %s and 1",
second.Expires.Sub(second.Start), second.Notes.EarlierBans, tc.want)
}
})
}
}
func TestFirstBanLongerThanTheMaximumIsPermanent(t *testing.T) {
t.Parallel()
rules := defaultRules()
rules.LimitBanDuration = rules.MaxBanDuration + time.Hour
ledger := bans.New(rules)
ban := ledger.BanForLimit(netip.MustParsePrefix("203.0.113.9/32"), midnight(),
bans.Notes{})
if !ban.Permanent() {
t.Errorf("first ban ends at %s, want a permanent one", ban.Expires)
}
}
func TestLongestBanSetFarOffDoesNotOverflow(t *testing.T) {
t.Parallel()
// With bans of up to 100,000 days, the 14th ban in a row, of 3^13
// hours, is within the maximum, and three times as long would not fit
// in a time.Duration. The 15th is permanent.
rules := defaultRules()
rules.MaxBanDuration = 100000 * day
ledger := bans.New(rules)
netblock := netip.MustParsePrefix("203.0.113.9/32")
now := midnight()
for i := range 14 {
ban := ledger.BanForLimit(netblock, now, bans.Notes{})
if !ban.Expires.After(ban.Start) {
t.Fatalf("ban %d starts at %s and ends at %s", i+1, ban.Start, ban.Expires)
}
now = ban.Expires
}
ban := ledger.BanForLimit(netblock, now, bans.Notes{})
if !ban.Permanent() {
t.Errorf("15th ban ends at %s, want a permanent one", ban.Expires)
}
}
func TestBrokenLimitDuringABanMakesNoOther(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultRules())
netblock := netip.MustParsePrefix("203.0.113.9/32")
first := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
again := ledger.BanForLimit(netblock, midnight().Add(time.Minute), bans.Notes{})
if again != first || len(ledger.Bans(netblock)) != 1 {
t.Errorf("a limit broken during a ban gave %+v and %d bans, want %+v and 1",
again, len(ledger.Bans(netblock)), first)
}
}
func TestCheckRefusesWhileTheBanLastsAndCountsTheRefusals(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultRules())
netblock := netip.MustParsePrefix("203.0.113.9/32")
ban := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
for range 3 {
got, banned := ledger.Check(netblock, ban.Expires.Add(-time.Nanosecond))
if !banned || got.Start != ban.Start {
t.Fatalf("check during the ban gives %+v and %t", got, banned)
}
}
_, banned := ledger.Check(netip.MustParsePrefix("203.0.113.10/32"), midnight())
if banned {
t.Error("another netblock is banned")
}
_, banned = ledger.Check(netblock, ban.Expires)
if banned {
t.Error("the ban did not end")
}
refused := ledger.Bans(netblock)[0].Notes.Refused
if refused != 3 {
t.Errorf("the notes count %d refused requests, want 3", refused)
}
}
func TestMaxBansDropsTheEarliestBanOfTheNetblockSeenLongestAgo(t *testing.T) {
t.Parallel()
rules := defaultRules()
rules.MaxBans = 3
ledger := bans.New(rules)
a := netip.MustParsePrefix("203.0.113.1/32")
b := netip.MustParsePrefix("203.0.113.2/32")
c := netip.MustParsePrefix("203.0.113.3/32")
d := netip.MustParsePrefix("2001:db8::/64")
now := midnight()
first := ledger.BanForLimit(a, now, bans.Notes{})
ledger.BanForLimit(b, now, bans.Notes{})
ledger.BanForLimit(c, now, bans.Notes{})
// A request from a makes b the netblock seen longest ago, and its ban
// goes to make room for d's.
ledger.Check(a, now)
ledger.BanForLimit(d, now, bans.Notes{})
wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 0, c: 1, d: 1})
// a is banned again once its ban has ended; c, seen longest ago, goes.
ledger.BanForLimit(a, first.Expires, bans.Notes{})
wantBans(t, ledger, map[netip.Prefix]int{a: 2, c: 0, d: 1})
// With d seen since, a is seen longest ago, and its earlier ban goes
// first.
ledger.Check(d, first.Expires)
ledger.BanForLimit(b, first.Expires, bans.Notes{})
wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 1, d: 1})
if !ledger.Bans(a)[0].Start.Equal(first.Expires) {
t.Errorf("a kept its ban of %s, want the later one", ledger.Bans(a)[0].Start)
}
}
func TestFullLedgerDropsTheEarlierBanOfTheNetblockBannedAgain(t *testing.T) {
t.Parallel()
// With room for one ban, the netblock's ended ban goes to make room for
// its new one, whose notes still count it.
rules := defaultRules()
rules.MaxBans = 1
ledger := bans.New(rules)
netblock := netip.MustParsePrefix("203.0.113.9/32")
first := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
second := ledger.BanForLimit(netblock, first.Expires, bans.Notes{})
held := ledger.Bans(netblock)
if len(held) != 1 || held[0] != second || held[0].Notes.EarlierBans != 1 {
t.Errorf("the ledger holds %+v, want only the second ban, with 1 earlier ban",
held)
}
}
func TestRequestTextsAreCutTo256Bytes(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultRules())
netblock := netip.MustParsePrefix("203.0.113.9/32")
long := strings.Repeat("a", 300)
request := bans.Request{
Time: midnight(), Method: long, Host: long, Path: long, Status: 403, UserAgent: long,
}
ban := ledger.BanForLimit(netblock, midnight(), bans.Notes{Request: request})
cut := long[:256]
want := bans.Request{
Time: midnight(), Method: cut, Host: cut, Path: cut, Status: 403, UserAgent: cut,
}
if ban.Notes.Request != want || ledger.Bans(netblock)[0].Notes.Request != want {
t.Errorf("the notes keep %+v, want each text cut to 256 bytes", ban.Notes.Request)
}
}
// defaultRules are the rules at the settings' defaults.
func defaultRules() bans.Rules {
return bans.Rules{
LimitBanDuration: time.Hour,
LimitBanRepeatWindow: day,
MaxBanDuration: 7 * day,
MaxBans: 5000,
}
}
// midnight is when the tests' first bans are made.
func midnight() time.Time {
return time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)
}
// wantBans checks how many bans the ledger holds on each netblock.
func wantBans(t *testing.T, ledger *bans.Ledger, want map[netip.Prefix]int) {
t.Helper()
for netblock, count := range want {
got := len(ledger.Bans(netblock))
if got != count {
t.Errorf("%s has %d bans, want %d", netblock, got, count)
}
}
}
+114
View File
@@ -9,6 +9,7 @@ import (
"log/slog"
"math"
"net"
"net/http"
"net/netip"
"net/url"
"slices"
@@ -74,6 +75,25 @@ type Config struct {
// capitals, as GeoJS gives them.
DeniedCountries []string
ExclusivelyAllowedCountries []string
// BanResponse is the status a refused client is answered with, 403
// or 429, or 0 to close the connection without an answer
// (SWWAF_BAN_RESPONSE). It answers a banned client, a request that
// breaks a rate limit, SWWAF_DENY_NETS and the country lists.
BanResponse int
// LimitBanDuration is the ban for a first broken rate limit
// (SWWAF_LIMIT_BAN_DURATION). A limit broken again within
// LimitBanRepeatWindow after the last ban ended bans for three times
// as long as that ban (SWWAF_LIMIT_BAN_REPEAT_WINDOW), and a ban that
// would be longer than MaxBanDuration is permanent instead
// (SWWAF_MAX_BAN_DURATION). None of them can be off.
LimitBanDuration time.Duration
LimitBanRepeatWindow time.Duration
MaxBanDuration time.Duration
// MaxBans is the most bans held (SWWAF_MAX_BANS).
MaxBans int
// BanScopeV4Prefix is the length of the netblock around an IPv4
// client that a ban covers (SWWAF_BAN_SCOPE_V4_PREFIX).
BanScopeV4Prefix int
// settings are the values read, as given or by default, for the
// log line at start.
@@ -89,6 +109,7 @@ const (
kibibyte = 1 << 10
mebibyte = 1 << 20
gibibyte = 1 << 30
ipv4Bits = 32
)
var (
@@ -111,6 +132,13 @@ var (
"is not a two-letter country code such as de or kp")
errOnBothLists = errors.New("is in SWWAF_DENIED_COUNTRIES too")
errNotOver4K = errors.New("is not a size of more than 4K, such as 32K")
errNotDurationAboveZero = errors.New(
"is not a duration above zero, such as 1h or 7d")
errNotNumberAboveZero = errors.New(
"is not a whole number above zero, such as 5000")
errNotBanResponse = errors.New("is not 403, 429 or close")
errNotV4Prefix = errors.New(
"is not the length of an IPv4 netblock, from 0 to 32, such as 24")
)
// FromEnvironment reads the settings with lookupEnv, normally
@@ -140,6 +168,12 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
DeniedCountries: env.countries("SWWAF_DENIED_COUNTRIES", ""),
ExclusivelyAllowedCountries: env.countries(
"SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES", ""),
BanResponse: env.banResponse("SWWAF_BAN_RESPONSE", "403"),
LimitBanDuration: env.durationNotOff("SWWAF_LIMIT_BAN_DURATION", "1h"),
LimitBanRepeatWindow: env.durationNotOff("SWWAF_LIMIT_BAN_REPEAT_WINDOW", "24h"),
MaxBanDuration: env.durationNotOff("SWWAF_MAX_BAN_DURATION", "7d"),
MaxBans: env.numberNotOff("SWWAF_MAX_BANS", "5000"),
BanScopeV4Prefix: env.v4Prefix("SWWAF_BAN_SCOPE_V4_PREFIX", "32"),
}
for _, country := range cfg.ExclusivelyAllowedCountries {
@@ -261,6 +295,40 @@ func (e *environment) countries(name, defaultValue string) []string {
return countries
}
// durationNotOff reads a setting that is a duration and, unlike a
// timeout, cannot be off.
func (e *environment) durationNotOff(name, defaultValue string) time.Duration {
duration, err := parseDurationNotOff(e.value(name, defaultValue))
e.check(name, err)
return duration
}
// numberNotOff reads a setting that is a whole number above zero, which
// cannot be off.
func (e *environment) numberNotOff(name, defaultValue string) int {
number, err := parseNumberNotOff(e.value(name, defaultValue))
e.check(name, err)
return number
}
// banResponse reads a setting that is how a refused client is answered.
func (e *environment) banResponse(name, defaultValue string) int {
status, err := parseBanResponse(e.value(name, defaultValue))
e.check(name, err)
return status
}
// v4Prefix reads a setting that is the length of an IPv4 netblock.
func (e *environment) v4Prefix(name, defaultValue string) int {
length, err := parseV4Prefix(e.value(name, defaultValue))
e.check(name, err)
return length
}
// parseDuration reads a duration in Go's syntax, such as 90s or 15m, a
// whole number of days such as 7d, or off.
func parseDuration(value string) (time.Duration, error) {
@@ -362,6 +430,52 @@ func parseCount(value string) (int64, error) {
return n, nil
}
// parseDurationNotOff reads a duration above zero, as parseDuration does,
// but not off.
func parseDurationNotOff(value string) (time.Duration, error) {
duration, err := parseDuration(value)
if err != nil || duration == 0 {
return 0, fmt.Errorf("%q %w", value, errNotDurationAboveZero)
}
return duration, nil
}
// parseNumberNotOff reads a whole number above zero.
func parseNumberNotOff(value string) (int, error) {
n, err := strconv.Atoi(value)
if err != nil || n <= 0 {
return 0, fmt.Errorf("%q %w", value, errNotNumberAboveZero)
}
return n, nil
}
// parseBanResponse reads how a refused client is answered: 403, 429, or
// close, which is 0.
func parseBanResponse(value string) (int, error) {
switch value {
case "403":
return http.StatusForbidden, nil
case "429":
return http.StatusTooManyRequests, nil
case "close":
return 0, nil
default:
return 0, fmt.Errorf("%q %w", value, errNotBanResponse)
}
}
// parseV4Prefix reads the length of an IPv4 netblock, from 0 to 32.
func parseV4Prefix(value string) (int, error) {
n, err := strconv.Atoi(value)
if err != nil || n < 0 || n > ipv4Bits {
return 0, fmt.Errorf("%q %w", value, errNotV4Prefix)
}
return n, nil
}
// parseList splits a comma-separated list and trims the spaces around
// each item. An empty value is an empty list.
func parseList(value string) ([]string, error) {
+61
View File
@@ -35,6 +35,12 @@ const (
rateLimitPerDay = "SWWAF_RATE_LIMIT_PER_DAY"
deniedCountries = "SWWAF_DENIED_COUNTRIES"
allowedCountries = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES"
banResponse = "SWWAF_BAN_RESPONSE"
limitBanDuration = "SWWAF_LIMIT_BAN_DURATION"
limitBanRepeatWindow = "SWWAF_LIMIT_BAN_REPEAT_WINDOW"
maxBanDuration = "SWWAF_MAX_BAN_DURATION"
maxBans = "SWWAF_MAX_BANS"
banScopeV4Prefix = "SWWAF_BAN_SCOPE_V4_PREFIX"
)
// off switches a timeout, a size limit or a rate limit off.
@@ -80,6 +86,12 @@ func TestDefaults(t *testing.T) {
RateLimitPerMinute: 1000,
RateLimitPerHour: 10000,
RateLimitPerDay: 50000,
BanResponse: 403,
LimitBanDuration: time.Hour,
LimitBanRepeatWindow: 24 * time.Hour,
MaxBanDuration: 7 * 24 * time.Hour,
MaxBans: 5000,
BanScopeV4Prefix: 32,
})
if cfg.UpstreamURL.String() != "http://127.0.0.1:8081" {
@@ -118,6 +130,12 @@ func TestValuesAsSet(t *testing.T) {
rateLimitPerDay: "6000",
deniedCountries: "cn, RU,kp,Xk",
allowedCountries: "de",
banResponse: "429",
limitBanDuration: "15m",
limitBanRepeatWindow: "2d",
maxBanDuration: "30d",
maxBans: "100",
banScopeV4Prefix: "24",
})
wantSettings(t, cfg, config.Config{
@@ -133,6 +151,12 @@ func TestValuesAsSet(t *testing.T) {
RateLimitPerMinute: 60,
RateLimitPerHour: 600,
RateLimitPerDay: 6000,
BanResponse: 429,
LimitBanDuration: 15 * time.Minute,
LimitBanRepeatWindow: 48 * time.Hour,
MaxBanDuration: 30 * 24 * time.Hour,
MaxBans: 100,
BanScopeV4Prefix: 24,
})
if cfg.UpstreamURL.String() != "https://app.internal:8443/" {
@@ -226,6 +250,15 @@ func TestRateLimitsOff(t *testing.T) {
}
}
func TestBanResponseCloseIsZero(t *testing.T) {
t.Parallel()
cfg := fromEnvironment(t, environment{banResponse: "close"})
if cfg.BanResponse != 0 {
t.Errorf("close read as %d, want 0", cfg.BanResponse)
}
}
func TestTrustedProxiesSetButEmptyTrustNothing(t *testing.T) {
t.Parallel()
@@ -290,6 +323,12 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
{allowedCountries, "uk"},
{allowedCountries, "zz"},
{allowedCountries, "de,germany"},
{banResponse, "404"}, {banResponse, "drop"}, {banResponse, ""},
{limitBanDuration, off}, {limitBanDuration, "0s"}, {limitBanDuration, "1"},
{limitBanRepeatWindow, off}, {limitBanRepeatWindow, "-1h"},
{maxBanDuration, off}, {maxBanDuration, "1w"},
{maxBans, off}, {maxBans, "0"}, {maxBans, "5K"},
{banScopeV4Prefix, "33"}, {banScopeV4Prefix, "-1"}, {banScopeV4Prefix, "/24"},
} {
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
t.Parallel()
@@ -344,6 +383,12 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
rateLimitPerDay: "50000",
deniedCountries: "",
allowedCountries: "",
banResponse: "403",
limitBanDuration: "1h",
limitBanRepeatWindow: "24h",
maxBanDuration: "7d",
maxBans: "5000",
banScopeV4Prefix: "32",
}
if !maps.Equal(line.Settings, want) {
t.Errorf("logged settings\n%v\nwant\n%v", line.Settings, want)
@@ -368,6 +413,22 @@ func wantSettings(t *testing.T, got *config.Config, want config.Config) {
got.RateLimitPerDay != want.RateLimitPerDay {
t.Errorf("settings\n%+v\nwant\n%+v", got, want)
}
wantBanSettings(t, got, want)
}
// wantBanSettings checks the settings for bans.
func wantBanSettings(t *testing.T, got *config.Config, want config.Config) {
t.Helper()
if got.BanResponse != want.BanResponse ||
got.LimitBanDuration != want.LimitBanDuration ||
got.LimitBanRepeatWindow != want.LimitBanRepeatWindow ||
got.MaxBanDuration != want.MaxBanDuration ||
got.MaxBans != want.MaxBans ||
got.BanScopeV4Prefix != want.BanScopeV4Prefix {
t.Errorf("ban settings\n%+v\nwant\n%+v", got, want)
}
}
// wantNetblocks checks a list of netblocks.
+82
View File
@@ -0,0 +1,82 @@
package proxy
import (
"net/netip"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// banResponse is a refusal answered with SWWAF_BAN_RESPONSE, and logged
// with action.
func (rq *request) banResponse(action string) *refusal {
return &refusal{status: rq.h.config.BanResponse, action: action}
}
// banned reports whether a ban on the client's netblock refuses the
// request at now, and notes for the log line when that ban ends.
func (rq *request) banned(now time.Time) bool {
ban, banned := rq.h.ledger.Check(rq.netblock(), now)
if banned {
rq.line.BanExpires = banExpires(ban)
}
return banned
}
// limitBroken counts the request for the rate limits at now, and reports
// whether it takes the client over one. Such a request bans the client's
// netblock, and sets the client's counters back to zero.
func (rq *request) limitBroken(now time.Time) bool {
group := clientGroup(rq.client)
hit, over := rq.h.limiter.Count(group, now)
if !over {
return false
}
ban := rq.h.ledger.BanForLimit(rq.netblock(), now, bans.Notes{
Country: rq.line.Country,
Limit: hit.Limit,
Window: hit.Window,
Count: hit.Requests,
Request: bans.Request{
Time: now,
Method: rq.in.Method,
Host: rq.in.Host,
Path: rq.in.URL.RequestURI(),
Status: rq.h.config.BanResponse,
UserAgent: rq.in.UserAgent(),
},
})
rq.h.limiter.Reset(group)
rq.line.LimitHit = hit.Window
rq.line.Offence = requestlog.OffenceLimit
rq.line.BanExpires = banExpires(ban)
return true
}
// netblock is the netblock a ban on the client covers: its IPv4 address,
// widened to SWWAF_BAN_SCOPE_V4_PREFIX, or the IPv6 group clientGroup
// counts it in.
func (rq *request) netblock() netip.Prefix {
addr := rq.client.Unmap()
if addr.Is4() {
return netip.PrefixFrom(addr, rq.h.config.BanScopeV4Prefix).Masked()
}
return clientGroup(addr)
}
// banExpires is when ban ends, as the log line gives it: a time, or
// permanent.
func banExpires(ban bans.Ban) string {
if ban.Permanent() {
return "permanent"
}
return requestlog.FormatTime(ban.Expires)
}
+434
View File
@@ -0,0 +1,434 @@
package proxy_test
import (
"bufio"
"errors"
"io"
"maps"
"net/http"
"net/netip"
"slices"
"sync"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/proxy"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
const (
// otherClient is a client next to client.
otherClient = "203.0.113.10"
// userAgent is the user agent of every request a sender sends.
userAgent = "ban-test/1.0"
// permanent is the log line's ban_expires for a permanent ban.
permanent = "permanent"
)
func TestBrokenLimitBansTheClient(t *testing.T) {
t.Parallel()
s, clk, _ := startWithClock(t, "", map[string]string{rateLimitPerMinute: "1"})
expires := requestlog.FormatTime(clk.Now().Add(time.Hour))
// The request over the limit of one a minute is refused, and bans the
// client for an hour, the default.
s.get(client, http.StatusOK, requestlog.ActionForward)
line := s.get(client, http.StatusForbidden, requestlog.ActionRateLimited)
if line.LimitHit != minute || line.Offence != requestlog.OffenceLimit ||
line.BanExpires != expires {
t.Errorf("log line has limit_hit %q, offence %q and ban_expires %q, "+
"want minute, limit and %s", line.LimitHit, line.Offence, line.BanExpires,
expires)
}
// Every request while the ban lasts is refused.
clk.advance(time.Hour - time.Second)
line = s.get(client, http.StatusForbidden, requestlog.ActionBanned)
if line.BanExpires != expires || line.Offence != "" || line.LimitHit != "" {
t.Errorf("log line has ban_expires %q, offence %q and limit_hit %q, "+
"want %s and neither of the others", line.BanExpires, line.Offence,
line.LimitHit, expires)
}
// Once it ends, the client is let through.
clk.advance(time.Second)
s.get(client, http.StatusOK, requestlog.ActionForward)
}
func TestBanLengthsFollowTheSettings(t *testing.T) {
t.Parallel()
s, clk, _ := startWithClock(t, "", map[string]string{
rateLimitPerMinute: "1",
limitBanDuration: "10m",
limitBanRepeatWindow: "1h",
maxBanDuration: "1h",
})
// breakLimit has client go over the limit of one a minute, and
// returns when the ban that makes ends.
breakLimit := func() string {
s.get(client, http.StatusOK, requestlog.ActionForward)
return s.get(client, http.StatusForbidden, requestlog.ActionRateLimited).BanExpires
}
wantExpires := func(got string, length time.Duration) {
t.Helper()
want := requestlog.FormatTime(clk.Now().Add(length))
if got != want {
t.Errorf("ban ends at %s, want %s", got, want)
}
}
// A first ban lasts SWWAF_LIMIT_BAN_DURATION; one within
// SWWAF_LIMIT_BAN_REPEAT_WINDOW after it ended, three times as long.
wantExpires(breakLimit(), 10*time.Minute)
clk.advance(10*time.Minute + time.Hour)
wantExpires(breakLimit(), 30*time.Minute)
// Later than that, SWWAF_LIMIT_BAN_DURATION again.
clk.advance(30*time.Minute + time.Hour + time.Second)
wantExpires(breakLimit(), 10*time.Minute)
clk.advance(10 * time.Minute)
wantExpires(breakLimit(), 30*time.Minute)
// 90 minutes would be longer than SWWAF_MAX_BAN_DURATION: the ban is
// permanent.
clk.advance(30 * time.Minute)
got := breakLimit()
if got != permanent {
t.Errorf("ban ends at %s, want a permanent one", got)
}
clk.advance(365 * 24 * time.Hour)
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
}
func TestBanIsNotCountedAndResetsTheCounters(t *testing.T) {
t.Parallel()
s, clk, server := startWithClock(t, "", map[string]string{rateLimitPerDay: "2"})
// The third request in a day is over the limit of two, and bans the
// client for an hour.
s.get(client, http.StatusOK, requestlog.ActionForward)
s.get(client, http.StatusOK, requestlog.ActionForward)
s.get(client, http.StatusForbidden, requestlog.ActionRateLimited)
for range 3 {
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
}
// Later the same day the client has its whole allowance again: the
// ban set its counters back to zero, and the requests it refused were
// not counted for the rate limits, only in its notes.
clk.advance(time.Hour)
s.get(client, http.StatusOK, requestlog.ActionForward)
s.get(client, http.StatusOK, requestlog.ActionForward)
s.get(client, http.StatusForbidden, requestlog.ActionRateLimited)
banned := proxy.LedgerOf(server).Bans(netip.MustParsePrefix(client + "/32"))
if len(banned) != 2 || banned[0].Notes.Refused != 3 {
t.Errorf("bans %+v, want two, the first with 3 requests refused", banned)
}
}
func TestBanCoversTheClientsNetblock(t *testing.T) {
t.Parallel()
// In the IPv4 cases, client breaks the limit; these two are next to it.
const (
allowed = "203.0.113.60" // in SWWAF_ALLOW_NETS
exempt = "203.0.113.50" // in SWWAF_RATE_LIMIT_EXEMPT_NETS
)
for _, tc := range []struct {
name string
env map[string]string
breaker string // the client that breaks the limit
refused []string
let []string // let through
}{
{
"an IPv4 address, by default", nil, client,
nil, []string{otherClient, exempt},
},
{
"the IPv4 netblock SWWAF_BAN_SCOPE_V4_PREFIX sets",
map[string]string{banScopeV4Prefix: "24"}, client,
[]string{otherClient, exempt}, []string{"203.0.112.9", allowed},
},
{
"an IPv6 /64", nil, "2001:db8:5::1",
[]string{"2001:db8:5::ffff:1"}, []string{"2001:db8:5:1::1"},
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
env := map[string]string{
rateLimitPerMinute: "1",
allowNets: allowed,
rateLimitExemptNets: exempt,
}
maps.Copy(env, tc.env)
s, _, _ := startWithClock(t, "", env)
s.get(tc.breaker, http.StatusOK, requestlog.ActionForward)
s.get(tc.breaker, http.StatusForbidden, requestlog.ActionRateLimited)
for _, sent := range tc.refused {
s.get(sent, http.StatusForbidden, requestlog.ActionBanned)
}
for _, sent := range tc.let {
s.get(sent, http.StatusOK, requestlog.ActionForward)
}
})
}
}
func TestBannedClientIsRefusedBeforeItsCountryIsLookedUp(t *testing.T) {
t.Parallel()
geojsURL, asked := startGeoJS(t)
s, _, _ := startWithClock(t, geojsURL, map[string]string{
rateLimitPerMinute: "1",
banScopeV4Prefix: "24",
deniedCountries: "kp",
})
// fromDE's ban covers otherClient, which is refused unasked about.
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited)
line := s.get(otherClient, http.StatusForbidden, requestlog.ActionBanned)
if line.Country != "" {
t.Errorf("log line has country %q, want none", line.Country)
}
if !slices.Equal(asked(), []string{fromDE}) {
t.Errorf("GeoJS was asked about %v, want %s alone", asked(), fromDE)
}
}
func TestBanResponseAnswersEveryRefusalButTheSizeLimits(t *testing.T) {
t.Parallel()
const denied = "192.0.2.50" // in SWWAF_DENY_NETS
for _, tc := range []struct {
setting string // "" leaves SWWAF_BAN_RESPONSE at its default
status int // 0 is the connection closed without an answer
}{
{"", http.StatusForbidden},
{"403", http.StatusForbidden},
{"429", http.StatusTooManyRequests},
{"close", 0},
} {
t.Run(banResponse+"="+tc.setting, func(t *testing.T) {
t.Parallel()
geojsURL, _ := startGeoJS(t)
env := map[string]string{
rateLimitPerMinute: "1",
denyNets: denied,
deniedCountries: "kp",
}
if tc.setting != "" {
env[banResponse] = tc.setting
}
s, _, _ := startWithClock(t, geojsURL, env)
s.get(denied, tc.status, requestlog.ActionDenied)
s.get(fromKP, tc.status, requestlog.ActionCountryDenied)
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
s.get(fromDE, tc.status, requestlog.ActionRateLimited)
s.get(fromDE, tc.status, requestlog.ActionBanned)
})
}
}
func TestBanNotes(t *testing.T) {
t.Parallel()
geojsURL, _ := startGeoJS(t)
s, clk, server := startWithClock(t, geojsURL, map[string]string{
rateLimitPerMinute: "1",
deniedCountries: "kp",
})
start := clk.Now()
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
s.request(fromDE, "/repo/commits?page=2",
http.StatusForbidden, requestlog.ActionRateLimited)
s.get(fromDE, http.StatusForbidden, requestlog.ActionBanned)
s.get(fromDE, http.StatusForbidden, requestlog.ActionBanned)
netblock := netip.MustParsePrefix(fromDE + "/32")
want := bans.Ban{
Netblock: netblock,
Start: start,
Expires: start.Add(time.Hour),
Notes: bans.Notes{
Country: "DE",
Limit: 1,
Window: minute,
Count: 2,
Request: bans.Request{
Time: start,
Method: http.MethodGet,
Host: appHost,
Path: "/repo/commits?page=2",
Status: http.StatusForbidden,
UserAgent: userAgent,
},
Refused: 2,
EarlierBans: 0,
},
}
ledger := proxy.LedgerOf(server)
got := ledger.Bans(netblock)
if len(got) != 1 || got[0] != want {
t.Fatalf("bans\n%+v\nwant\n%+v", got, want)
}
// The next ban counts this one among the earlier.
clk.advance(time.Hour)
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited)
got = ledger.Bans(netblock)
if len(got) != 2 || got[1].Notes.EarlierBans != 1 {
t.Errorf("bans %+v, want two, the second with one earlier ban", got)
}
}
func TestMaxBansDropsTheBanOfTheNetblockSeenLongestAgo(t *testing.T) {
t.Parallel()
s, _, _ := startWithClock(t, "", map[string]string{
rateLimitPerMinute: "1",
maxBans: "1",
})
// One ban is held, so otherClient's ban drops client's.
s.get(client, http.StatusOK, requestlog.ActionForward)
s.get(client, http.StatusForbidden, requestlog.ActionRateLimited)
s.get(otherClient, http.StatusOK, requestlog.ActionForward)
s.get(otherClient, http.StatusForbidden, requestlog.ActionRateLimited)
s.get(client, http.StatusOK, requestlog.ActionForward)
s.get(otherClient, http.StatusForbidden, requestlog.ActionBanned)
}
// clock is the time a test sets, by which smallwebwaf counts requests and
// makes bans.
type clock struct {
mu sync.Mutex
now time.Time
}
// Now tells the time.
func (c *clock) Now() time.Time {
c.mu.Lock()
defer c.mu.Unlock()
return c.now
}
// advance moves the clock on by d.
func (c *clock) advance(d time.Duration) {
c.mu.Lock()
defer c.mu.Unlock()
c.now = c.now.Add(d)
}
// startWithClock starts smallwebwaf in front of an app that answers 200,
// with the settings in env on top of trusting localhost's
// X-Forwarded-For, clients' countries looked up at geojsURL, and a clock
// set to midnight, the start of a bucket in every window.
func startWithClock(
t *testing.T, geojsURL string, env map[string]string,
) (*sender, *clock, *http.Server) {
t.Helper()
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
clk := &clock{now: time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)}
settings := map[string]string{trustedProxies: trustLocalhost}
maps.Copy(settings, env)
addr, out, server := startProxyWithClock(t, app.URL, geojsURL, clk.Now, settings)
return &sender{t: t, addr: addr, out: out}, clk, server
}
// sender sends requests to smallwebwaf one after another, each on a
// connection of its own, and checks each one's answer and log line. They
// must be the only requests smallwebwaf is sent, since the log lines are
// matched to them in order.
type sender struct {
t *testing.T
addr string
out *output
sent int
}
// get sends a GET request for / from the client at from.
func (s *sender) get(from string, status int, action string) logLine {
s.t.Helper()
return s.request(from, "/", status, action)
}
// request sends a GET request for path from the client at from, as
// X-Forwarded-For names it, and checks that its answer and its log line
// have status, 0 for the connection closed without an answer, and that
// the line has action. It returns the log line.
func (s *sender) request(from, path string, status int, action string) logLine {
s.t.Helper()
conn := dial(s.t, s.addr)
send(s.t, conn, "GET "+path+" HTTP/1.1\r\nHost: "+appHost+
"\r\nUser-Agent: "+userAgent+"\r\n"+forwardedFor+": "+from+"\r\n\r\n")
err := conn.SetReadDeadline(time.Now().Add(waitLimit))
if err != nil {
s.t.Fatalf("set read deadline: %v", err)
}
got := 0
res, err := http.ReadResponse(bufio.NewReader(conn), nil)
switch {
case err == nil:
got = readAnswer(res).status
case !errors.Is(err, io.ErrUnexpectedEOF):
s.t.Fatalf("read response: %v", err)
}
_ = conn.Close()
if got != status {
s.t.Errorf("request %d, from %s: status %d, want %d", s.sent+1, from, got,
status)
}
line := s.out.requestLines(s.t, s.sent+1)[s.sent]
s.sent++
wantLine(s.t, line, status, action)
return line
}
+15
View File
@@ -0,0 +1,15 @@
package proxy
import (
"net/http"
"sneak.berlin/go/smallwebwaf/internal/bans"
)
// LedgerOf returns the ban ledger of a server New returned, so that the
// tests can read the bans' notes.
func LedgerOf(server *http.Server) *bans.Ledger {
h, _ := server.Handler.(*handler)
return h.ledger
}
+13
View File
@@ -10,6 +10,7 @@ import (
"net/http"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
@@ -36,6 +37,9 @@ type Params struct {
// GeoJSURL is where clients' countries are looked up, normally
// lookup.URL. GeoJS is asked only while a country list is set.
GeoJSURL string
// Now tells the time by which requests are counted for the rate
// limits and bans are made and run out, normally time.Now.
Now func() time.Time
}
// New returns the server smallwebwaf runs: each request it reads passes
@@ -55,11 +59,18 @@ func New(params Params) *http.Server {
processLog: params.ProcessLog,
errorLog: errorLog,
transport: newTransport(),
now: params.Now,
limiter: ratelimit.New(ratelimit.Limits{
PerMinute: params.Config.RateLimitPerMinute,
PerHour: params.Config.RateLimitPerHour,
PerDay: params.Config.RateLimitPerDay,
}),
ledger: bans.New(bans.Rules{
LimitBanDuration: params.Config.LimitBanDuration,
LimitBanRepeatWindow: params.Config.LimitBanRepeatWindow,
MaxBanDuration: params.Config.MaxBanDuration,
MaxBans: params.Config.MaxBans,
}),
geojs: lookup.New(lookup.Params{
URL: params.GeoJSURL,
Now: time.Now,
@@ -85,7 +96,9 @@ type handler struct {
processLog *slog.Logger
errorLog *log.Logger
transport http.RoundTripper
now func() time.Time
limiter *ratelimit.Limiter
ledger *bans.Ledger
geojs *lookup.GeoJS
}
+22 -1
View File
@@ -57,8 +57,15 @@ const (
rateLimitExemptNets = "SWWAF_RATE_LIMIT_EXEMPT_NETS"
denyNets = "SWWAF_DENY_NETS"
rateLimitPerMinute = "SWWAF_RATE_LIMIT_PER_MINUTE"
rateLimitPerDay = "SWWAF_RATE_LIMIT_PER_DAY"
deniedCountries = "SWWAF_DENIED_COUNTRIES"
allowedCountries = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES"
banResponse = "SWWAF_BAN_RESPONSE"
limitBanDuration = "SWWAF_LIMIT_BAN_DURATION"
limitBanRepeatWindow = "SWWAF_LIMIT_BAN_REPEAT_WINDOW"
maxBanDuration = "SWWAF_MAX_BAN_DURATION"
maxBans = "SWWAF_MAX_BANS"
banScopeV4Prefix = "SWWAF_BAN_SCOPE_V4_PREFIX"
)
// output collects what smallwebwaf writes on stdout.
@@ -183,6 +190,19 @@ func startProxyWithGeoJS(
) (string, *output) {
t.Helper()
addr, out, _ := startProxyWithClock(t, appURL, geojsURL, time.Now, env)
return addr, out
}
// startProxyWithClock is startProxyWithGeoJS with requests counted and
// bans made by the time now tells, and returns the server as well.
func startProxyWithClock(
t *testing.T, appURL, geojsURL string, now func() time.Time,
env map[string]string,
) (string, *output, *http.Server) {
t.Helper()
settings := map[string]string{"SWWAF_UPSTREAM_URL": appURL}
maps.Copy(settings, env)
@@ -201,6 +221,7 @@ func startProxyWithGeoJS(
RequestLog: out,
ProcessLog: requestlog.NewProcessLogger(out),
GeoJSURL: geojsURL,
Now: now,
})
listener, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", localhost+":0")
@@ -216,7 +237,7 @@ func startProxyWithGeoJS(
_ = server.Close()
})
return listener.Addr().String(), out
return listener.Addr().String(), out, server
}
// newClient returns an HTTP client that sends requests as they are made,
+12 -8
View File
@@ -8,7 +8,11 @@ import (
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
func TestRateLimitRefusesWith429BeforeTheApp(t *testing.T) {
// minute is the window of SWWAF_RATE_LIMIT_PER_MINUTE, as a log line's
// limit_hit names it.
const minute = "minute"
func TestRateLimitRefusesBeforeTheApp(t *testing.T) {
t.Parallel()
var calls atomic.Int32
@@ -24,19 +28,19 @@ func TestRateLimitRefusesWith429BeforeTheApp(t *testing.T) {
const otherClient = "203.0.113.10"
// With a limit of one request a minute, a client's second request is
// refused. A client is one IPv4 address, or one IPv6 /64; an IPv4
// address in IPv6 form is that IPv4 address.
// refused, with 403 by default. A client is one IPv4 address, or one
// IPv6 /64; an IPv4 address in IPv6 form is that IPv4 address.
requests := []struct {
client string // as X-Forwarded-For names it
logged string // as the log line's client_ip names it
want int
}{
{client, client, http.StatusOK},
{client, client, http.StatusTooManyRequests},
{client, client, http.StatusForbidden},
{otherClient, otherClient, http.StatusOK},
{"::ffff:" + otherClient, otherClient, http.StatusTooManyRequests},
{"::ffff:" + otherClient, otherClient, http.StatusForbidden},
{"2001:db8::1", "2001:db8::1", http.StatusOK},
{"2001:db8::8000:0:0:1", "2001:db8::8000:0:0:1", http.StatusTooManyRequests},
{"2001:db8::8000:0:0:1", "2001:db8::8000:0:0:1", http.StatusForbidden},
{"2001:db8:0:1::1", "2001:db8:0:1::1", http.StatusOK},
}
@@ -53,9 +57,9 @@ func TestRateLimitRefusesWith429BeforeTheApp(t *testing.T) {
if sent.want == http.StatusOK {
wantLine(t, line, http.StatusOK, requestlog.ActionForward)
} else {
wantLine(t, line, http.StatusTooManyRequests, requestlog.ActionRateLimited)
wantLine(t, line, http.StatusForbidden, requestlog.ActionRateLimited)
if line.LimitHit != "minute" {
if line.LimitHit != minute {
t.Errorf("log line has limit_hit %q, want minute", line.LimitHit)
}
}
+24 -22
View File
@@ -21,7 +21,8 @@ const flushAfterEachWrite time.Duration = -1
// refusal is smallwebwaf refusing a request, or refusing to go on with it:
// the status the client is answered if the response has not started yet,
// and the action the log line names.
// 0 to close the connection without an answer, and the action the log
// line names.
type refusal struct {
status int
action string
@@ -105,39 +106,33 @@ func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request {
// is known, before its body is read or anything reaches the app. It
// returns nil to let the request through. A client in SWWAF_ALLOW_NETS
// skips every check but the size limit. For any other client,
// SWWAF_DENY_NETS comes first, so that a client it refuses is not looked
// up, and then the country lists; a request either refuses is not counted
// for the rate limits. Then come the rate limits, unless the client is in
// SWWAF_DENY_NETS comes first, then a ban on its netblock, so that a
// client either refuses is not looked up, and then the country lists; a
// request any of them refuses is not counted for the rate limits. Then
// come the rate limits, unless the client is in
// SWWAF_RATE_LIMIT_EXEMPT_NETS, so that every other request is counted,
// one refused for its size too. ctx is the request's own context.
// one refused for its size too. Every refusal but the size limit's is
// answered with SWWAF_BAN_RESPONSE. ctx is the request's own context.
func (rq *request) check(ctx context.Context) *refusal {
cfg := rq.h.config
allowed := isInside(rq.client, cfg.AllowNets)
exempt := isInside(rq.client, cfg.RateLimitExemptNets)
now := rq.h.now()
if !allowed && isInside(rq.client, cfg.DenyNets) {
return &refusal{
status: http.StatusForbidden,
action: requestlog.ActionDenied,
return rq.banResponse(requestlog.ActionDenied)
}
if !allowed && rq.banned(now) {
return rq.banResponse(requestlog.ActionBanned)
}
if !allowed && rq.countryDenied(ctx) {
return &refusal{
status: http.StatusForbidden,
action: requestlog.ActionCountryDenied,
}
return rq.banResponse(requestlog.ActionCountryDenied)
}
if !allowed && !isInside(rq.client, cfg.RateLimitExemptNets) {
limitHit := rq.h.limiter.Count(clientGroup(rq.client), rq.start)
if limitHit != "" {
rq.line.LimitHit = limitHit
return &refusal{
status: http.StatusTooManyRequests,
action: requestlog.ActionRateLimited,
}
}
if !allowed && !exempt && rq.limitBroken(now) {
return rq.banResponse(requestlog.ActionRateLimited)
}
maxBytes := cfg.RequestMaxBytes
@@ -250,6 +245,13 @@ func (rq *request) answer(r refusal) {
return // too late to answer: the connection can only be cut
}
if r.status == 0 {
// SWWAF_BAN_RESPONSE is close. This panic has Go's server close
// the connection without an answer, and log nothing; the log line
// is still written as the handler returns.
panic(http.ErrAbortHandler)
}
// A client found too slow is read no more; any other may go on
// sending until its time is up, so that Go's server can read the
// rest of the body and end the request cleanly.
+3 -3
View File
@@ -78,7 +78,7 @@ func TestRequestFromAllowNetsIsNotCounted(t *testing.T) {
{listedAddr, http.StatusOK, requestlog.ActionForward},
{listedAddr, http.StatusOK, requestlog.ActionForward},
{unlistedAddr, http.StatusOK, requestlog.ActionForward},
{unlistedAddr, http.StatusTooManyRequests, requestlog.ActionRateLimited},
{unlistedAddr, http.StatusForbidden, requestlog.ActionRateLimited},
})
}
@@ -133,7 +133,7 @@ func TestRequestRefusedByDenyNetsIsNotCounted(t *testing.T) {
{listedAddr, http.StatusForbidden, requestlog.ActionDenied},
{listedAddr, http.StatusForbidden, requestlog.ActionDenied},
{unlistedAddr, http.StatusOK, requestlog.ActionForward},
{unlistedAddr, http.StatusTooManyRequests, requestlog.ActionRateLimited},
{unlistedAddr, http.StatusForbidden, requestlog.ActionRateLimited},
})
}
@@ -156,7 +156,7 @@ func TestRateLimitExemptNetsAreNeitherCountedNorRefused(t *testing.T) {
{listedAddr, http.StatusOK, requestlog.ActionForward},
{listedAddr, http.StatusOK, requestlog.ActionForward},
{unlistedAddr, http.StatusOK, requestlog.ActionForward},
{unlistedAddr, http.StatusTooManyRequests, requestlog.ActionRateLimited},
{unlistedAddr, http.StatusForbidden, requestlog.ActionRateLimited},
{fromKP, http.StatusForbidden, requestlog.ActionCountryDenied},
})
}
+27 -8
View File
@@ -54,11 +54,22 @@ func New(limits Limits) *Limiter {
}
}
// Hit is a request that takes a client over a rate limit.
type Hit struct {
// Window is "minute", "hour" or "day".
Window string
// Limit is the window's limit.
Limit int64
// Requests is the client's requests counted in the window, this one
// included.
Requests float64
}
// Count counts a request from client at now, in every window, whether or
// not it is refused. It returns the window whose limit the request takes
// the client over, "minute", "hour" or "day", the shortest if it is over
// several, or "" if it is within every limit.
func (l *Limiter) Count(client netip.Prefix, now time.Time) string {
// not it is refused. It reports whether the request takes the client over
// a limit, and the window whose limit it goes over, the shortest if it is
// over several.
func (l *Limiter) Count(client netip.Prefix, now time.Time) (Hit, bool) {
l.mu.Lock()
defer l.mu.Unlock()
@@ -68,16 +79,24 @@ func (l *Limiter) Count(client netip.Prefix, now time.Time) string {
l.clients.Add(client, counts)
}
limitHit := ""
var hit Hit
for i, w := range l.windows {
requests := counts[i].add(now, w.length)
if limitHit == "" && w.limit > 0 && requests > float64(w.limit) {
limitHit = w.name
if hit.Window == "" && w.limit > 0 && requests > float64(w.limit) {
hit = Hit{Window: w.name, Limit: w.limit, Requests: requests}
}
}
return limitHit
return hit, hit.Window != ""
}
// Reset sets client's counts in every window back to zero.
func (l *Limiter) Reset(client netip.Prefix) {
l.mu.Lock()
defer l.mu.Unlock()
l.clients.Remove(client)
}
// window is a length of time over which requests are counted, and the
+49 -3
View File
@@ -54,6 +54,52 @@ func TestEachWindowRefusesAtItsLimitAndLetsTheClientBack(t *testing.T) {
}
}
func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit, PerHour: limit})
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
for range limit {
_, over := limiter.Count(client, start)
if over {
t.Fatal("a request within the limit is over it")
}
}
// Over both limits; the minute's is named, with the four requests.
hit, over := limiter.Count(client, start)
want := ratelimit.Hit{Window: minute, Limit: limit, Requests: limit + 1}
if !over || hit != want {
t.Errorf("request over the limit gives %+v and %t, want %+v and true",
hit, over, want)
}
}
func TestResetSetsTheCountsBackToZero(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit, PerDay: limit})
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
for range limit {
wantCount(t, limiter, client, start, "")
}
wantCount(t, limiter, client, start, minute)
limiter.Reset(client)
// At the same moment, the client has its whole allowance again.
for range limit {
wantCount(t, limiter, client, start, "")
}
wantCount(t, limiter, client, start, minute)
}
func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) {
t.Parallel()
@@ -192,9 +238,9 @@ func wantCount(
) {
t.Helper()
got := limiter.Count(client, now)
if got != want {
hit, _ := limiter.Count(client, now)
if hit.Window != want {
t.Errorf("request from %s at %s is over %q, want %q",
client, now.Format(time.RFC3339), got, want)
client, now.Format(time.RFC3339), hit.Window, want)
}
}
+12 -1
View File
@@ -24,8 +24,10 @@ const (
// for, or whose answer could not be passed on.
ActionUpstreamError = "upstream_error"
// ActionRateLimited is a request refused because it took its client
// over a rate limit, or came while the client was over one.
// over a rate limit, which bans the client.
ActionRateLimited = "rate_limited"
// ActionBanned is a request refused because a ban covers its client.
ActionBanned = "banned"
// ActionDenied is a request refused because its client is in
// SWWAF_DENY_NETS.
ActionDenied = "denied"
@@ -36,6 +38,10 @@ const (
ActionAdmin = "admin"
)
// OffenceLimit is the offence a request line names for a request that
// broke a rate limit.
const OffenceLimit = "limit"
// timeLayout is RFC 3339 with milliseconds.
const timeLayout = "2006-01-02T15:04:05.000Z07:00"
@@ -64,6 +70,11 @@ type Line struct {
// LimitHit is the window whose rate limit the request went over:
// minute, hour or day.
LimitHit string `json:"limit_hit,omitempty"`
// Offence is the offence the request was held as, OffenceLimit.
Offence string `json:"offence,omitempty"`
// BanExpires is when the ban the request made, or was refused under,
// ends: a time, or "permanent".
BanExpires string `json:"ban_expires,omitempty"`
// Aborted is true when the client went away early.
Aborted bool `json:"aborted,omitempty"`
// DurationTotal and DurationUpstreamTotal are in milliseconds.
+2 -1
View File
@@ -50,7 +50,8 @@ func TestWriteWritesOneJSONLineMarkedRequest(t *testing.T) {
}
unset := []string{
"upstream_status", "limit_hit", "aborted", "duration_upstream_total",
"upstream_status", "limit_hit", "offence", "ban_expires", "aborted",
"duration_upstream_total",
}
for _, name := range unset {
_, present := fields[name]
+1
View File
@@ -80,6 +80,7 @@ func Run(ctx context.Context, params Params) int {
RequestLog: params.Stdout,
ProcessLog: processLog,
GeoJSURL: lookup.URL,
Now: time.Now,
})
processLog.Info("starting",
+6
View File
@@ -196,6 +196,12 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL string) {
"SWWAF_RATE_LIMIT_PER_DAY": "50000",
"SWWAF_DENIED_COUNTRIES": "",
"SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES": "",
"SWWAF_BAN_RESPONSE": "403",
"SWWAF_LIMIT_BAN_DURATION": "1h",
"SWWAF_LIMIT_BAN_REPEAT_WINDOW": "24h",
"SWWAF_MAX_BAN_DURATION": "7d",
"SWWAF_MAX_BANS": "5000",
"SWWAF_BAN_SCOPE_V4_PREFIX": "32",
}
for name, value := range want {