Trap paths, and the error burst banning a client refused too often (closes #115)
check / check (push) Waiting to run

SWWAF_TRAP_PATHS: a request whose path, as a path rule sees it, is one
of them is a clear sign of attack, banned as a ban rule's match is; the
ban's notes give its trap_path. Checked after the rate limits, before the
rule files.

SWWAF_ERROR_BURST_THRESHOLD (default 30, or off): more refusals in a
minute after a block or ban rule or a trap path, or for a missing or
wrong token, ban the client as a broken limit does. Counted in
clients.json's minute_refusals; limit_hit error_burst, notes kind
refusals.

A token refusal is now the offence token_refused, and
smallwebwaf_offences_total counts every kind the history does.

Judgement call: the threshold is not lowered by a client's limit percentage.

Model: opus-5-5
This commit is contained in:
2026-10-08 03:57:37 +00:00
parent 5f3fb48809
commit 9ebf2a9e7e
21 changed files with 1208 additions and 318 deletions
+250 -187
View File
@@ -31,29 +31,32 @@ you name by URL, which it fetches and keeps, with a file of AS numbers'
percentages fetched the same way, the DNS blocklists (DNSBL zones), which it percentages fetched the same way, the DNS blocklists (DNSBL zones), which it
asks about each client in the background, AbuseIPDB, which it asks in the asks about each client in the background, AbuseIPDB, which it asks in the
background about each client that has committed an offence, and the decision background about each client that has committed an offence, and the decision
list of a CrowdSec engine you run, which it fetches and keeps as a blocklist. list of a CrowdSec engine you run, which it fetches and keeps as a blocklist. So
`smallwebwaf` passes each request to the app and the app's answer back, are two parts of the last stage, which need no Core Rule Set: the trap paths and
unchanged, within its timeouts and size limits, works out each client's address, the error burst. `smallwebwaf` passes each request to the app and the app's
looks up its AS number and country unless you switch that off, bans a client answer back, unchanged, within its timeouts and size limits, works out each
that sends too many requests or too many bytes, not counting those for the paths client's address, looks up its AS number and country unless you switch that off,
you choose, with lower limits for the clients of the AS numbers and countries bans a client that sends too many requests or too many bytes, not counting those
you list, refuses a client that comes from a country you refuse or from a for the paths you choose, with lower limits for the clients of the AS numbers
network you refuse, refuses, limits or only notes a client a blocklist or a and countries you list, refuses a client that comes from a country you refuse or
DNSBL zone you name lists, or AbuseIPDB scores at or over the score you set, from a network you refuse, refuses, limits or only notes a client a blocklist or
a DNSBL zone you name lists, or AbuseIPDB scores at or over the score you set,
bans a client your CrowdSec engine's decision list lists until that decision bans a client your CrowdSec engine's decision list lists until that decision
ends, lets the networks you choose through, checks each request against the rule ends, lets the networks you choose through, checks each request against the rule
files and bans a client whose request is a clear sign of attack, keeps its bans, files and the trap paths you name and bans a client whose request is a clear
each client's counters and history, GeoJS's answers, the last good copy of each sign of attack, bans a client it refuses again and again within a minute for a
list it fetches, the DNSBL zones' verdicts, AbuseIPDB's scores and the AbuseIPDB rule, a trap path or a missing or wrong token, keeps its bans, each client's
checks spent today in JSON files across restarts, takes in your edits of those counters and history, GeoJS's answers, the last good copy of each list it
files, such as a ban you make, keep or lift, and of the rule files while it fetches, the DNSBL zones' verdicts, AbuseIPDB's scores and the AbuseIPDB checks
runs, writes a JSON log line for every request, sends its log lines to a syslog spent today in JSON files across restarts, takes in your edits of those files,
server too if you name one, sends an alert to a webhook, to Slack and to ntfy, such as a ban you make, keep or lift, and of the rule files while it runs,
each if you name one, for each ban it makes or makes permanent, for traffic over writes a JSON log line for every request, sends its log lines to a syslog server
an anomaly threshold you set, for a client a blocklist, the CrowdSec decision too if you name one, sends an alert to a webhook, to Slack and to ntfy, each if
list, a DNSBL zone or AbuseIPDB lists, for GeoJS failing, a list it cannot you name one, for each ban it makes or makes permanent, for traffic over an
fetch, a DNSBL zone or AbuseIPDB that fails or refuses a query and the day's anomaly threshold you set, for a client a blocklist, the CrowdSec decision list,
AbuseIPDB checks used up, for a rule file or state file with an error and for a a DNSBL zone or AbuseIPDB lists, for GeoJS failing, a list it cannot fetch, a
DNSBL zone or AbuseIPDB that fails or refuses a query and the day's AbuseIPDB
checks used up, for a rule file or state file with an error and for a
replacement of the lookup database it cannot read, serves Prometheus metrics to replacement of the lookup database it cannot read, serves Prometheus metrics to
a scraper that holds the metrics token, lets an admin who holds the admin token a scraper that holds the metrics token, lets an admin who holds the admin token
list, add and lift bans and ask what it knows of a client, and in `observe` mode list, add and lift bans and ask what it knows of a client, and in `observe` mode
@@ -152,37 +155,47 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
The log line of each request the rate limits count gives its client's The log line of each request the rate limits count gives its client's
percentages below 100 and the settings that gave them, and so do the notes of percentages below 100 and the settings that gave them, and so do the notes of
a ban for a lowered limit, and its alert. a ban for a lowered limit, and its alert.
- Bans a client that breaks a rate limit or a byte limit, as "Bans" in - Bans a client that breaks a rate limit, a byte limit or the error burst, as
[`SPEC.md`](SPEC.md) describes: the first ban lasts an hour, and a limit "Bans" in [`SPEC.md`](SPEC.md) describes: the first ban lasts an hour, and a
broken again within a day of a ban ending bans for three times as long as that limit broken again within a day of a ban ending bans for three times as long
ban, so 1, 3, 9, 27 and 81 hours; a ban that would last longer than seven days as that ban, so 1, 3, 9, 27 and 81 hours; a ban that would last longer than
is permanent instead. A ban covers the client's netblock: its IPv4 address, or seven days is permanent instead. A ban covers the client's netblock: its IPv4
the netblock around it that `SWWAF_BAN_SCOPE_V4_PREFIX` sets, or its IPv6 address, or the netblock around it that `SWWAF_BAN_SCOPE_V4_PREFIX` sets, or
group. While it lasts, every request from the netblock is refused with its IPv6 group. While it lasts, every request from the netblock is refused
`SWWAF_BAN_RESPONSE` after the static lists and before the country lists, so with `SWWAF_BAN_RESPONSE` after the static lists and before the country lists,
the client is not looked up, and is not counted for the rate limits. A ban so the client is not looked up, and is not counted for the rate limits. A ban
sets the client's counters back to zero. Each ban carries notes for deciding sets the client's counters back to zero. Each ban carries notes for deciding
whether to lift it: the limit, whether it is on requests or bytes, its window whether to lift it: the limit, whether it is on requests, bytes or refusals,
and the requests or bytes counted in it, the client's percentage of that kind its window and the requests, bytes or refusals counted in it, the client's
of limit and the setting that gave it when a biased threshold lowered the percentage of that kind of limit and the setting that gave it when a biased
limit, the request that broke it, the client's AS number, AS name and country threshold lowered the limit, the request that broke it, the client's AS
once they are looked up, the blocklists, the CrowdSec decision list, DNSBL number, AS name and country once they are looked up, the blocklists, the
zones and AbuseIPDB, with its score, that listed the client when the ban was CrowdSec decision list, DNSBL zones and AbuseIPDB, with its score, that listed
made, the netblock's requests since it was first seen, how many of them the the client when the ban was made, the netblock's requests since it was first
ban has refused, and how many bans the netblock had before, for a broken seen, how many of them the ban has refused, and how many bans the netblock had
limit, for a clear sign of attack, by an admin and for CrowdSec's decision. At before, for a broken limit, for a clear sign of attack, by an admin and for
most `SWWAF_MAX_BANS` bans `smallwebwaf` made are kept, past, active and CrowdSec's decision. At most `SWWAF_MAX_BANS` bans `smallwebwaf` made are
permanent; past that, the earliest such ban of the netblock that has gone kept, past, active and permanent; past that, the earliest such ban of the
longest without a request is dropped first. The bans whose cause is `admin`, netblock that has gone longest without a request is dropped first. The bans
those you make or keep, are kept besides, and never dropped. `bans.json` shows whose cause is `admin`, those you make or keep, are kept besides, and never
the bans and their notes, a restart lifts none, and you make, keep or lift a dropped. `bans.json` shows the bans and their notes, a restart lifts none, and
ban by editing it (see "State files" below). you make, keep or lift a ban by editing it (see "State files" below).
- Checks each request against the rules of the rule files (see "Rule files" - Checks each request against the rules of the rule files (see "Rule files"
below) after the rate limits, and before its body is read. A `log` rule that below) after the rate limits, and before its body is read. A `log` rule that
matches is noted in the log line; a `block` rule refuses the request with matches is noted in the log line; a `block` rule refuses the request with
`403`, and bans no one; a `ban` rule refuses it with `SWWAF_BAN_RESPONSE` and `403`, and bans no one for it, though the refusal counts toward the error
bans the client's netblock for a clear sign of attack. Matching stops at the burst; a `ban` rule refuses it with `SWWAF_BAN_RESPONSE` and bans the client's
first rule that refuses. A client in `SWWAF_ALLOW_NETS` is not checked. netblock for a clear sign of attack. Matching stops at the first rule that
refuses. A client in `SWWAF_ALLOW_NETS` is not checked.
- Checks each request against the trap paths `SWWAF_TRAP_PATHS` names, after the
rate limits and before the rule files, which it does not need. A request for
one is refused with `SWWAF_BAN_RESPONSE` and bans the client's netblock for a
clear sign of attack, as a `ban` rule's match does. A trap path is matched
against the path a `path` rule sees: as the client sent it, before any
decoding and without the query, the whole of it, character for character.
`/wp-login.php` matches `/wp-login.php?redirect_to=x`, but not
`/wp-login.php/`, `/WP-LOGIN.PHP`, `/blog/wp-login.php` or `/%77p-login.php`.
A client in `SWWAF_ALLOW_NETS` is not checked.
- Bans a client for a clear sign of attack, as "Bans" in [`SPEC.md`](SPEC.md) - Bans a client for a clear sign of attack, as "Bans" in [`SPEC.md`](SPEC.md)
describes: the first such ban lasts `SWWAF_ATTACK_BAN_DURATION`, seven days by describes: the first such ban lasts `SWWAF_ATTACK_BAN_DURATION`, seven days by
default, and any request from the netblock while it lasts makes it permanent. default, and any request from the netblock while it lasts makes it permanent.
@@ -190,8 +203,22 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
sign of attack bans it permanently at once. Such a ban covers the same sign of attack bans it permanently at once. Such a ban covers the same
netblock as a ban for a broken limit, does not set the client's counters back netblock as a ban for a broken limit, does not set the client's counters back
to zero, and does not make the netblock's next ban for a broken limit longer. to zero, and does not make the netblock's next ban for a broken limit longer.
Its notes give the id and the target of the rule that matched in place of the Its notes give the id and the target of the rule that matched, or the trap
limit. path asked for, in place of the limit.
- Bans a client that `smallwebwaf` refused more than
`SWWAF_ERROR_BURST_THRESHOLD` times within a minute, 30 by default, after a
match of a `block` or `ban` rule or a trap path, or for a missing or wrong
token at one of its own endpoints, as a broken rate limit bans it. Each such
refusal is counted once it has been answered, in two buckets of a minute, as
the rate limits count requests. The refusal that takes the client over the
threshold breaks the error burst; it is answered as any other such refusal is,
and the client's next request is refused under the ban. The app's own answers,
such as its `401` and `404`, are not counted. The threshold is the same for
every client, whatever percentage of the rate limits a biased threshold or a
reputation source gives it. A client in `SWWAF_ALLOW_NETS` is not counted, and
one in `SWWAF_RATE_LIMIT_EXEMPT_NETS` is. The ban's notes give `refusals` as
what the limit is on, the threshold as the limit, and the refusals counted in
the minute.
- Looks up the AS number and country of every client through GeoJS, or in the - Looks up the AS number and country of every client through GeoJS, or in the
IPinfo Lite database file while `SWWAF_LOOKUP_SOURCE` is `file`, after the IPinfo Lite database file while `SWWAF_LOOKUP_SOURCE` is `file`, after the
static lists and bans, unless `SWWAF_LOOKUP_SOURCE` is `off` (see "Country and static lists and bans, unless `SWWAF_LOOKUP_SOURCE` is `off` (see "Country and
@@ -242,44 +269,50 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
- Checks the client with AbuseIPDB while `SWWAF_ABUSEIPDB_KEY` is set, after the - Checks the client with AbuseIPDB while `SWWAF_ABUSEIPDB_KEY` is set, after the
DNSBL zones and before the rate limits (see "AbuseIPDB" below), by the scores DNSBL zones and before the rate limits (see "AbuseIPDB" below), by the scores
it keeps. Only a client whose history counts an offence is checked, so far one it keeps. Only a client whose history counts an offence is checked, so far one
that has broken a rate limit or a byte limit, matched a ban rule, or had a that has broken a rate limit, a byte limit or the error burst, matched a ban
request refused by a block rule, and only in the background, so that no rule, asked for a trap path, or had a request refused by a block rule or for a
request waits for AbuseIPDB. A score at or over `SWWAF_ABUSEIPDB_MIN_SCORE` is missing or wrong token, and only in the background, so that no request waits
a hit, and `SWWAF_REPUTATION_ACTION` does with its client what it does with for AbuseIPDB. A score at or over `SWWAF_ABUSEIPDB_MIN_SCORE` is a hit, and
one a DNSBL zone's verdict lists. The request's log line names AbuseIPDB, and `SWWAF_REPUTATION_ACTION` does with its client what it does with one a DNSBL
it raises an alert. A client a blocklist or a DNSBL zone refuses, or the zone's verdict lists. The request's log line names AbuseIPDB, and it raises an
CrowdSec decision list bans, is not checked. alert. A client a blocklist or a DNSBL zone refuses, or the CrowdSec decision
list bans, is not checked.
- Checks the client's own address against the static lists, the three netblock - Checks the client's own address against the static lists, the three netblock
settings below, before anything else, its lookup included. A client in settings below, before anything else, its lookup included. A client in
`SWWAF_ALLOW_NETS` skips bans, the country lists, the blocklists, the CrowdSec `SWWAF_ALLOW_NETS` skips bans, the country lists, the blocklists, the CrowdSec
decision list, the DNSBL zones, AbuseIPDB, the rate limits, the byte limits decision list, the DNSBL zones, AbuseIPDB, the rate limits, the byte limits,
and the rule files, and is not looked up; the timeouts and size limits still the trap paths, the rule files and the error burst, and is not looked up; the
apply. A client in `SWWAF_DENY_NETS` is refused with `SWWAF_BAN_RESPONSE` timeouts and size limits still apply. A client in `SWWAF_DENY_NETS` is refused
before its body is read, and the request is not counted for the rate limits; with `SWWAF_BAN_RESPONSE` before its body is read, and the request is not
an address in `SWWAF_ALLOW_NETS` too is let through. A client in counted for the rate limits; an address in `SWWAF_ALLOW_NETS` too is let
`SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither counted nor refused by the rate through. A client in `SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither counted nor
limits, and has no bytes counted by the byte limits; the country lists, the refused by the rate limits, and has no bytes counted by the byte limits; the
rule files and bans still apply to it. country lists, the trap paths, the rule files, the error burst and bans still
apply to it.
- In `observe` mode, with `SWWAF_MODE=observe`, refuses none of the requests - In `observe` mode, with `SWWAF_MODE=observe`, refuses none of the requests
that `SWWAF_DENY_NETS`, a ban, the country lists, a blocklist, the CrowdSec that `SWWAF_DENY_NETS`, a ban, the country lists, a blocklist, the CrowdSec
decision list, a DNSBL zone's verdict, AbuseIPDB's score, a rate limit or a decision list, a DNSBL zone's verdict, AbuseIPDB's score, a rate limit, a trap
rule would refuse: it passes them to the app, and their log lines name what path or a rule would refuse: it passes them to the app, and their log lines
`enforce` mode would have done (see `would_action` in "Request log" below). name what `enforce` mode would have done (see `would_action` in "Request log"
The checks run, and requests and bytes are counted, as in `enforce` mode, with below). The checks run, and requests, bytes and refusals are counted, as in
three differences: neither a broken rate limit or byte limit, a `ban` rule nor `enforce` mode, with three differences: neither a broken rate limit, byte
the CrowdSec decision list makes a ban; a broken limit does not set the limit or error burst, a `ban` rule, a trap path nor the CrowdSec decision list
client's counters back to zero, so each request over a rate limit is logged as makes a ban; a broken limit does not set the client's counters back to zero,
one that would be refused, and each whose bytes keep the client over a byte so each request over a rate limit is logged as one that would be refused, each
limit as breaking it; and a request under a ban does not make it permanent. As whose bytes keep the client over a byte limit as breaking it, and each refusal
in `enforce` mode, the bytes counted are only those of the requests `enforce` that keeps it over the error burst as breaking that; and a request under a ban
mode would have passed to the app. A ban it would have made, or made does not make it permanent. As in `enforce` mode, the bytes counted are only
permanent, raises the alert `enforce` mode would have raised, marked as what those of the requests `enforce` mode would have passed to the app, and the
would have happened (see "Alerts" below). The bans in `bans.json` are kept, refusals counted for the error burst only those it would have made: a request
and refuse requests again when `smallwebwaf` next runs in `enforce` mode, as it would have refused before it reached an endpoint is not counted for its
long as they last. The timeouts and size limits still apply, since they token. A ban it would have made, or made permanent, raises the alert `enforce`
protect `smallwebwaf` and the app themselves, and a request for one of mode would have raised, marked as what would have happened (see "Alerts"
`smallwebwaf`'s own endpoints without its token is still answered `401`. It is below). The bans in `bans.json` are kept, and refuse requests again when
for trying a configuration before enforcing it. `smallwebwaf` next runs in `enforce` mode, as long as they last. The timeouts
and size limits still apply, since they protect `smallwebwaf` and the app
themselves, and a request for one of `smallwebwaf`'s own endpoints without its
token is still answered `401`. It is for trying a configuration before
enforcing it.
- Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any - Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any
check and without asking the app, for the image's health check. check and without asking the app, for the image's health check.
- Answers `GET /_smallwebwaf/metrics` with its metrics (see "Metrics" below) for - Answers `GET /_smallwebwaf/metrics` with its metrics (see "Metrics" below) for
@@ -385,8 +418,8 @@ effective settings are logged at start, unless `SWWAF_LOG_LEVEL` is `warn` or
- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body. - `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body.
- `SWWAF_ALLOW_NETS` (default empty): netblocks whose clients skip bans, the - `SWWAF_ALLOW_NETS` (default empty): netblocks whose clients skip bans, the
country lists, the blocklists, the CrowdSec decision list, the DNSBL zones, country lists, the blocklists, the CrowdSec decision list, the DNSBL zones,
AbuseIPDB, the rate limits, the byte limits and the rule files, such as your AbuseIPDB, the rate limits, the byte limits, the trap paths, the rule files
monitoring or your own networks. and the error burst, such as your monitoring or your own networks.
- `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients the - `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients the
rate limits and the byte limits do not apply to, such as a machine that talks rate limits and the byte limits do not apply to, such as a machine that talks
to the app all day. to the app all day.
@@ -530,21 +563,21 @@ effective settings are logged at start, unless `SWWAF_LOG_LEVEL` is `warn` or
- `SWWAF_REPUTATION_TIMEOUT` (default `2s`): how long a query to a zone, or a - `SWWAF_REPUTATION_TIMEOUT` (default `2s`): how long a query to a zone, or a
check with AbuseIPDB, may take before it fails. check with AbuseIPDB, may take before it fails.
- `SWWAF_BAN_RESPONSE` (default `403`): how a refused client is answered, one - `SWWAF_BAN_RESPONSE` (default `403`): how a refused client is answered, one
that is banned, breaks a rate limit, matches a `ban` rule, is in that is banned, breaks a rate limit, matches a `ban` rule, asks for a trap
`SWWAF_DENY_NETS`, comes from a refused country, is in a blocklist while path, is in `SWWAF_DENY_NETS`, comes from a refused country, is in a blocklist
`SWWAF_BLOCKLIST_ACTION` is `deny` or is listed by a DNSBL zone or AbuseIPDB while `SWWAF_BLOCKLIST_ACTION` is `deny` or is listed by a DNSBL zone or
while `SWWAF_REPUTATION_ACTION` is `deny`: `403`, `429`, or `close` to close AbuseIPDB while `SWWAF_REPUTATION_ACTION` is `deny`: `403`, `429`, or `close`
the connection without an answer. Behind traefik, `close` does not leave the to close the connection without an answer. Behind traefik, `close` does not
client unanswered: traefik answers `502`, as it does whenever its backend leave the client unanswered: traefik answers `502`, as it does whenever its
drops a connection. A `block` rule always answers `403`. backend drops a connection. A `block` rule always answers `403`.
- `SWWAF_LIMIT_BAN_DURATION` (default `1h`): the ban for a first broken rate - `SWWAF_LIMIT_BAN_DURATION` (default `1h`): the ban for a first broken rate
limit or byte limit. limit, byte limit or error burst.
- `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): a rate limit or byte limit - `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): a rate limit, byte limit or
broken again within this time after a ban ended, other than one for a clear error burst broken again within this time after a ban ended, other than one
sign of attack or for CrowdSec's decision, bans for three times as long as for a clear sign of attack or for CrowdSec's decision, bans for three times as
that ban. long as that ban.
- `SWWAF_MAX_BAN_DURATION` (default `7d`): a ban for a broken rate limit or byte - `SWWAF_MAX_BAN_DURATION` (default `7d`): a ban for a broken rate limit, byte
limit that would be longer is permanent instead. limit or error burst that would be longer is permanent instead.
- `SWWAF_ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of - `SWWAF_ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of
attack. attack.
- `SWWAF_MAX_BANS` (default `5000`): the most bans `smallwebwaf` made that are - `SWWAF_MAX_BANS` (default `5000`): the most bans `smallwebwaf` made that are
@@ -588,6 +621,18 @@ effective settings are logged at start, unless `SWWAF_LOG_LEVEL` is `warn` or
rule files. A directory that does not exist stops the start. rule files. A directory that does not exist stops the start.
- `SWWAF_RULES_ENABLED` (default `true`): `false` reads no rule file, and checks - `SWWAF_RULES_ENABLED` (default `true`): `false` reads no rule file, and checks
no request against one. no request against one.
- `SWWAF_TRAP_PATHS` (default empty): paths the app never serves and only
scanners ask for, such as `/wp-login.php,/xmlrpc.php` in front of gitea; a
request for one bans its client for a clear sign of attack, as a `ban` rule
does, with or without rule files (see "What it does so far" above). A path
that does not start with `/`, or holds a `?`, which no path a `path` rule sees
holds, stops the start.
- `SWWAF_ERROR_BURST_THRESHOLD` (default `30`): the most requests of a client in
a minute that `smallwebwaf` may refuse after a rule file match or a trap path,
or for a missing or wrong token; one more breaks the error burst, and bans the
client as a broken rate limit does (see "What it does so far" above). A client
trying one probe or token after another is refused many times a minute, a
person rarely more than a few times. `off` switches it off.
- `SWWAF_LOG_REMOTE_URL` (default unset): a syslog server that every line on - `SWWAF_LOG_REMOTE_URL` (default unset): a syslog server that every line on
stdout is also sent to, as `syslog+udp://`, `syslog+tcp://` or `syslog+tls://` stdout is also sent to, as `syslog+udp://`, `syslog+tcp://` or `syslog+tls://`
with a host and a port, such as `syslog+tls://logs.example:6514`. Unset or with a host and a port, such as `syslog+tls://logs.example:6514`. Unset or
@@ -671,18 +716,19 @@ effective settings are logged at start, unless `SWWAF_LOG_LEVEL` is `warn` or
Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are
bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024 bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024
bytes). Rate limits and the anomaly thresholds on requests are whole numbers of bytes). Rate limits, `SWWAF_ERROR_BURST_THRESHOLD` and the anomaly thresholds on
requests, and byte limits and the anomaly thresholds on bytes are sizes. requests are whole numbers of requests, and byte limits and the anomaly
Netblocks are in CIDR form, and a bare address stands for itself alone. thresholds on bytes are sizes. Netblocks are in CIDR form, and a bare address
Countries are the two-letter codes ISO 3166-1 assigns today, and `xk` for stands for itself alone. Countries are the two-letter codes ISO 3166-1 assigns
Kosovo, in either case (`de` and `DE` are the same); any other code, such as today, and `xk` for Kosovo, in either case (`de` and `DE` are the same); any
`nk` (North Korea is `kp`) or the withdrawn `su`, stops the start, and so does a other code, such as `nk` (North Korea is `kp`) or the withdrawn `su`, stops the
code on both country lists. AS numbers are `AS` and the number, in either case. start, and so does a code on both country lists. AS numbers are `AS` and the
Percentages are whole numbers from 0 to 100, and an entry of a list of them is number, in either case. Percentages are whole numbers from 0 to 100, and an
an AS number or a country, `:` and a percentage; an AS number or a country entry of a list of them is an AS number or a country, `:` and a percentage; an
listed twice in one of them stops the start. `off` switches a timeout, a size AS number or a country listed twice in one of them stops the start. `off`
limit, a rate limit, a byte limit, an anomaly threshold, `SWWAF_ALERT_COOLDOWN` switches a timeout, a size limit, a rate limit, a byte limit, an anomaly
or `SWWAF_ALERT_MAX_PER_HOUR` off; `SWWAF_IPV6_GROUP_PREFIX`, threshold, `SWWAF_ERROR_BURST_THRESHOLD`, `SWWAF_ALERT_COOLDOWN` or
`SWWAF_ALERT_MAX_PER_HOUR` off; `SWWAF_IPV6_GROUP_PREFIX`,
`SWWAF_MAX_TRACKED_CLIENTS`, `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, `SWWAF_MAX_TRACKED_CLIENTS`, `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`,
`SWWAF_LOOKUP_TIMEOUT`, `SWWAF_UNKNOWN_LIMIT_PERCENT`, `SWWAF_LOOKUP_TIMEOUT`, `SWWAF_UNKNOWN_LIMIT_PERCENT`,
`SWWAF_BLOCKLIST_REFRESH`, `SWWAF_ABUSEIPDB_MIN_SCORE`, `SWWAF_BLOCKLIST_REFRESH`, `SWWAF_ABUSEIPDB_MIN_SCORE`,
@@ -781,20 +827,20 @@ which every line has.
refused because its client is in `SWWAF_DENY_NETS`, in a blocklist while refused because its client is in `SWWAF_DENY_NETS`, in a blocklist while
`SWWAF_BLOCKLIST_ACTION` is `deny`, or listed by a DNSBL zone or AbuseIPDB `SWWAF_BLOCKLIST_ACTION` is `deny`, or listed by a DNSBL zone or AbuseIPDB
while `SWWAF_REPUTATION_ACTION` is `deny`, `banned` for one refused because a while `SWWAF_REPUTATION_ACTION` is `deny`, `banned` for one refused because a
ban covers its client, or because it matched a `ban` rule or the CrowdSec ban covers its client, or because it matched a `ban` rule, asked for a trap
decision list lists its client, either of which bans its client, path or the CrowdSec decision list lists its client, each of which bans its
`country_denied` for one refused for its client's country, `rate_limited` for client, `country_denied` for one refused for its client's country,
one that broke a rate limit and banned its client, `rule_blocked` for one a `rate_limited` for one that broke a rate limit and banned its client,
`block` rule refused, `too_large` for a request or response over its size `rule_blocked` for one a `block` rule refused, `too_large` for a request or
limit, `timed_out` for one that ran out of time, `upstream_error` when the app response over its size limit, `timed_out` for one that ran out of time,
could not be reached or its answer broke off, and `admin` for one `upstream_error` when the app could not be reached or its answer broke off,
`smallwebwaf` answered at its own endpoint. and `admin` for one `smallwebwaf` answered at its own endpoint.
- `would_action` is there in `observe` mode for a request that - `would_action` is there in `observe` mode for a request that
`SWWAF_DENY_NETS`, a ban, the country lists, a blocklist, the CrowdSec `SWWAF_DENY_NETS`, a ban, the country lists, a blocklist, the CrowdSec
decision list, a DNSBL zone's verdict, AbuseIPDB's score, a rate limit or a decision list, a DNSBL zone's verdict, AbuseIPDB's score, a rate limit, a trap
rule would have refused in `enforce` mode, and names the action that refusal path or a rule would have refused in `enforce` mode, and names the action that
would have had: `denied`, `banned`, `country_denied`, `rate_limited` or refusal would have had: `denied`, `banned`, `country_denied`, `rate_limited`
`rule_blocked`. `action` then names what was done: `forward` for a request or `rule_blocked`. `action` then names what was done: `forward` for a request
passed to the app, and another action, such as `too_large`, for one a size or passed to the app, and another action, such as `too_large`, for one a size or
time limit refused. time limit refused.
- `limit_percent` is there for a request the rate limits count whose client a - `limit_percent` is there for a request the rate limits count whose client a
@@ -822,13 +868,15 @@ which every line has.
counted before it. counted before it.
- `rule_ids` is there for a request that matched rules of the rule files, and - `rule_ids` is there for a request that matched rules of the rule files, and
lists their ids in the order they matched, up to the one that refused it. lists their ids in the order they matched, up to the one that refused it.
- `limit_hit` is there for a request that broke a rate limit, or whose bytes - `limit_hit` is there for a request that broke a rate limit or the error burst,
broke a byte limit, and names the window whose limit it went over as `counts` or whose bytes broke a byte limit, and names the window whose limit it went
names it: `minute`, `hour` or `day` for a rate limit, and `minute_bytes`, over as `counts` names it: `minute`, `hour` or `day` for a rate limit, and
`hour_bytes` or `day_bytes` for a byte limit, the shortest if it went over `minute_bytes`, `hour_bytes` or `day_bytes` for a byte limit, the shortest if
several. `offence` is then `limit`. A request whose bytes broke a byte limit it went over several; or `error_burst` for the error burst. `offence` is then
is not refused: its `action` is what it would have been otherwise, such as `limit`. A request whose bytes broke a byte limit is not refused: its `action`
`forward`. is what it would have been otherwise, such as `forward`. Nor is one that broke
the error burst refused for that: its `action` is that of the refusal counted,
such as `rule_blocked` or `admin`.
- `reputation` is there for a request whose client a blocklist, the CrowdSec - `reputation` is there for a request whose client a blocklist, the CrowdSec
decision list or a DNSBL zone's verdict lists, or whose AbuseIPDB score is a decision list or a DNSBL zone's verdict lists, or whose AbuseIPDB score is a
hit, and gives the URLs of the blocklists that list it, in the order hit, and gives the URLs of the blocklists that list it, in the order
@@ -1101,33 +1149,42 @@ which are listed by scope first, and the copies of the lists, listed by URL,
with times in UTC. with times in UTC.
- `bans.json`: every ban with its notes, indented to be read. A permanent ban's - `bans.json`: every ban with its notes, indented to be read. A permanent ban's
`expires` is `null`. A ban's `cause` is `limit` for a broken rate limit or `expires` is `null`. A ban's `cause` is `limit` for a broken rate limit, byte
byte limit, `attack` for a clear sign of attack or `crowdsec` for a client the limit or error burst, `attack` for a clear sign of attack or `crowdsec` for a
CrowdSec decision list lists, for a ban `smallwebwaf` made, and `admin` for client the CrowdSec decision list lists, for a ban `smallwebwaf` made, and
one you made or keep. Its `reason` is a short text: for a ban `smallwebwaf` `admin` for one you made or keep. Its `reason` is a short text: for a ban
made, the limit broken, such as `requests per minute over the limit of 1000` `smallwebwaf` made, the limit broken, such as
or `bytes per hour over the limit of 21474836480`, the rule that matched, such `requests per minute over the limit of 1000`,
as `matched the rule env-file`, or the scenario that made CrowdSec's decision, `bytes per hour over the limit of 21474836480` or
such as `CrowdSec's decision for crowdsecurity/ssh-bf`; for yours, what you `refusals per minute over the limit of 30`, the rule that matched, such as
wrote. Its `lifted` is when you lifted it, and is left out until you do. The `matched the rule env-file`, the trap path asked for, such as
`kind` in the notes of a ban for a broken limit is `requests` or `bytes`, what `asked for the trap path /wp-login.php`, or the scenario that made CrowdSec's
the limit is on. For a limit a biased threshold lowered, the reason and the decision, such as `CrowdSec's decision for crowdsecurity/ssh-bf`; for yours,
notes' `limit` give the lowered limit, and the notes' `limit_percent` and what you wrote. Its `lifted` is when you lifted it, and is left out until you
`limit_percent_setting` the client's percentage of that kind of limit and the do. The `kind` in the notes of a ban for a broken limit is `requests`, `bytes`
setting that gave it. The notes' `reputation` gives each blocklist, the or `refusals`, for the error burst, what the limit is on. The notes of a ban
CrowdSec decision list, each DNSBL zone or AbuseIPDB that listed the client for a clear sign of attack give the `rule_id` and the `target` of the rule
when the ban was made, as its `source`, named and ordered as in the request that matched, or the `trap_path` asked for. For a limit a biased threshold
log's `reputation`, with AbuseIPDB's `score` of the client. It is left out lowered, the reason and the notes' `limit` give the lowered limit, and the
when none did, and the example below shows it. notes' `limit_percent` and `limit_percent_setting` the client's percentage of
that kind of limit and the setting that gave it. The notes' `reputation` gives
each blocklist, the CrowdSec decision list, each DNSBL zone or AbuseIPDB that
listed the client when the ban was made, as its `source`, named and ordered as
in the request log's `reputation`, with AbuseIPDB's `score` of the client. It
is left out when none did, and the example below shows it.
- `clients.json`: each client's two buckets of requests in the minute, the hour - `clients.json`: each client's two buckets of requests in the minute, the hour
and the day, its two buckets of bytes in each, `minute_bytes`, `hour_bytes` and the day, its two buckets of bytes in each, `minute_bytes`, `hour_bytes`
and `day_bytes`, and its history: when it was first and last seen, its AS and `day_bytes`, its two buckets of refusals in the minute, which the error
number, AS name and country as last looked up and when the lookup gave them, burst counts, `minute_refusals`, and its history: when it was first and last
its requests, how many were forwarded and how many refused (one `smallwebwaf` seen, its AS number, AS name and country as last looked up and when the lookup
answered at its own endpoints is neither, unless it was refused with `401` for gave them, its requests, how many were forwarded and how many refused (one
a missing or wrong token), the body bytes in each direction, its responses by `smallwebwaf` answered at its own endpoints is neither, unless it was refused
status class and its offences by kind. Each client is on a line of its own, so with `401` for a missing or wrong token), the body bytes in each direction,
`grep` shows everything about one. its responses by status class and its offences by kind: `limit` for a broken
rate limit, byte limit or error burst, `attack` for a clear sign of attack,
`rule_blocked` for a request a `block` rule refused and `token_refused` for
one refused for a missing or wrong token. Each client is on a line of its own,
so `grep` shows everything about one.
- `lookups.json`: GeoJS's answers, one to a line, each with the client's AS - `lookups.json`: GeoJS's answers, one to a line, each with the client's AS
number, AS name and country, when GeoJS gave it and when it was last used. number, AS name and country, when GeoJS gave it and when it was last used.
- `reputation.json`: each list fetched from a URL (see "Blocklists" and - `reputation.json`: each list fetched from a URL (see "Blocklists" and
@@ -1243,19 +1300,19 @@ and the line and column where Go's JSON decoder gives them; so does a state
directory `smallwebwaf` cannot write. So does an entry without a field it needs, directory `smallwebwaf` cannot write. So does an entry without a field it needs,
named with the entry's place in the file: a ban's `netblock`, `start` or named with the entry's place in the file: a ban's `netblock`, `start` or
`expires`, which is `null` for a permanent ban; a client's `client`, or the `expires`, which is `null` for a permanent ban; a client's `client`, or the
`start` of a window in which it has requests or bytes; an answer's `client`, `start` of a window in which it has requests, bytes or refusals; an answer's
`country`, which is `""` for a client GeoJS cannot place, or `answered`; a `client`, `country`, which is `""` for a client GeoJS cannot place, or
cooldown's `event` or `sent`; an alert waiting's `event` or `time`; an anomaly `answered`; a cooldown's `event` or `sent`; an alert waiting's `event` or
counter's `netblock`, unless it counts an AS number or the whole service, its `time`; an anomaly counter's `netblock`, unless it counts an AS number or the
`asn`, for an AS number, its `name`, for a named netblock, or the `start` of a whole service, its `asn`, for an AS number, its `name`, for a named netblock, or
window in which it has requests or bytes; a list's `url`, `fetched` or `lines`, the `start` of a window in which it has requests or bytes; a list's `url`,
which is `[]` for an empty list; a verdict's `zone`, `client`, `listed`, which `fetched` or `lines`, which is `[]` for an empty list; a verdict's `zone`,
is `false` for a client the zone does not list, or `fetched`. So does a ban `client`, `listed`, which is `false` for a client the zone does not list, or
whose `cause` is not `limit`, `attack`, `admin` or `crowdsec`, alerts waiting `fetched`. So does a ban whose `cause` is not `limit`, `attack`, `admin` or
for a destination that is not `webhook`, `slack` or `ntfy`, an anomaly counter `crowdsec`, alerts waiting for a destination that is not `webhook`, `slack` or
whose `scope` is not `client`, `net`, `asn`, `total` or `watch`, and a copy of a `ntfy`, an anomaly counter whose `scope` is not `client`, `net`, `asn`, `total`
list with a line that would make its fetch fail. An answer's `asn` or `as_name` or `watch`, and a copy of a list with a line that would make its fetch fail. An
left out reads as empty. answer's `asn` or `as_name` left out reads as empty.
While it runs, `smallwebwaf` watches `SWWAF_STATE_DIR` and takes in your edit of While it runs, `smallwebwaf` watches `SWWAF_STATE_DIR` and takes in your edit of
a state file as soon as you save it: what the file then holds replaces what a state file as soon as you save it: what the file then holds replaces what
@@ -1406,9 +1463,11 @@ scraped, and keeps this one as `exported_instance` unless the scrape sets
from then on, as histograms; `smallwebwaf_requests_in_flight`: the requests from then on, as histograms; `smallwebwaf_requests_in_flight`: the requests
under way. under way.
- `smallwebwaf_rate_limit_hits_total` by `window`, `minute`, `hour` or `day`, - `smallwebwaf_rate_limit_hits_total` by `window`, `minute`, `hour` or `day`,
and `kind`, `requests` for a rate limit or `bytes` for a byte limit, and `kind`, `requests` for a rate limit, `bytes` for a byte limit or
`refusals` for the error burst, whose window is `minute`,
`smallwebwaf_size_and_time_limit_hits_total` by `limit`, the setting whose `smallwebwaf_size_and_time_limit_hits_total` by `limit`, the setting whose
limit was passed, `smallwebwaf_offences_total` by `kind`, and limit was passed, `smallwebwaf_offences_total` by `kind`, `limit`, `attack`,
`rule_blocked` or `token_refused`, as `clients.json` counts them, and
`smallwebwaf_bans_made_total` by `cause`, `limit`, `attack`, `admin` or `smallwebwaf_bans_made_total` by `cause`, `limit`, `attack`, `admin` or
`crowdsec`, `admin` for the bans you add through `POST /_smallwebwaf/bans`, `crowdsec`, `admin` for the bans you add through `POST /_smallwebwaf/bans`,
and those whose `cause` is `admin` that you add to `bans.json` while and those whose `cause` is `admin` that you add to `bans.json` while
@@ -1521,7 +1580,8 @@ or lift is written to `bans.json` `SWWAF_STATE_WRITE_DELAY` later. Refusals, and
the answers to requests that cannot be read, are plain text. the answers to requests that cannot be read, are plain text.
A request without the token, or with another, such as the metrics token, is A request without the token, or with another, such as the metrics token, is
answered `401`, in `observe` mode too. While the token is unset, each of these answered `401`, in `observe` mode too, and counts toward the error burst, as one
for the metrics without theirs does. While the token is unset, each of these
answers `404`, as does any request under `/_smallwebwaf/` that is not for one of answers `404`, as does any request under `/_smallwebwaf/` that is not for one of
its endpoints. Like the metrics, these requests go through every check any other its endpoints. Like the metrics, these requests go through every check any other
request goes through, and are answered where another would be passed to the app: request goes through, and are answered where another would be passed to the app:
@@ -1679,13 +1739,14 @@ For each request `smallwebwaf`:
- picks the client's limit percentage from those; - picks the client's limit percentage from those;
- checks the minute, hour and day request counters against the limits, and bans - checks the minute, hour and day request counters against the limits, and bans
the client if it breaks one; the client if it breaks one;
- checks the request against the rule files and the Core Rule Set, and bans the - checks the request against the trap paths, the rule files and the Core Rule
client at once for a clear sign of attack; Set, and bans the client at once for a clear sign of attack;
- forwards it to the app and streams the response back, within the size and time - forwards it to the app and streams the response back, within the size and time
limits; limits;
- counts the bytes and any refusal by the rule files or the Core Rule Set, bans - counts the bytes and any refusal by the trap paths, the rule files or the Core
the client if it broke a limit, updates its history and the anomaly counters, Rule Set or for a missing or wrong token, bans the client if it broke a limit,
sends any alerts that are due, and writes the log line. updates its history and the anomaly counters, sends any alerts that are due,
and writes the log line.
A minimal deployment is the app's own Dockerfile, built on the `smallwebwaf` A minimal deployment is the app's own Dockerfile, built on the `smallwebwaf`
image, with no setting. That image is built on Ubuntu 26.04 LTS, the newest image, with no setting. That image is built on Ubuntu 26.04 LTS, the newest
@@ -2054,14 +2115,16 @@ alerts, the metrics nor `reputation.json` hold it. Given as a file, with
request is refused before anything reaches the app: for `SWWAF_DENY_NETS`, for request is refused before anything reaches the app: for `SWWAF_DENY_NETS`, for
a ban, for the country lists, for a blocklist, for the CrowdSec decision list, a ban, for the country lists, for a blocklist, for the CrowdSec decision list,
which bans the client, for a DNSBL zone's verdict, for AbuseIPDB's score, for which bans the client, for a DNSBL zone's verdict, for AbuseIPDB's score, for
a rate limit, which bans the client, for a `block` or `ban` rule, the latter a rate limit, which bans the client, for a trap path, which bans the client,
banning the client, and for an announced body over the size limit; in for a `block` or `ban` rule, the latter banning the client, and for an
`observe` mode, only for the size limit, with what it would have refused for announced body over the size limit; in `observe` mode, only for the size
noted in the log line. A request under `/_smallwebwaf/` that `check` lets limit, with what it would have refused for noted in the log line. A request
through is answered by `answerAdmin` instead of reaching the app. Once the under `/_smallwebwaf/` that `check` lets through is answered by `answerAdmin`
answer to a request passed to the app has ended, `countBytes` counts its bytes instead of reaching the app. Once the answer to a request passed to the app
for the byte limits, and once any request but the health check has ended, has ended, `countBytes` counts its bytes for the byte limits, and once any
`countAnomalies` counts it for the anomaly thresholds. request but the health check has ended, `countRefusal` counts it for the error
burst if it was refused after a rule file match or a trap path or for its
token, and `countAnomalies` counts it for the anomaly thresholds.
- `internal/metrics`: the metrics, counted as the other parts tell it what - `internal/metrics`: the metrics, counted as the other parts tell it what
happened, and served in the Prometheus text format. happened, and served in the Prometheus text format.
- `internal/bans`: the ban ledger: each netblock's bans with their notes, how - `internal/bans`: the ban ledger: each netblock's bans with their notes, how
@@ -2083,9 +2146,9 @@ alerts, the metrics nor `reputation.json` hold it. Given as a file, with
tells which zones' verdicts list an address; and checks clients with AbuseIPDB tells which zones' verdicts list an address; and checks clients with AbuseIPDB
in the background, keeps their scores and the checks spent today, and tells in the background, keeps their scores and the checks spent today, and tells
whether a client's score is a hit. whether a client's score is a hit.
- `internal/ratelimit`: the table of clients: counts each client's requests and - `internal/ratelimit`: the table of clients: counts each client's requests,
bytes, tells when they take it over a rate limit or a byte limit, and keeps bytes and refusals, tells when they take it over a rate limit, a byte limit or
each client's history. the error burst, and keeps each client's history.
- `internal/anomaly`: the anomaly counters: counts each request and its bytes - `internal/anomaly`: the anomaly counters: counts each request and its bytes
per client, per netblock around a client, per AS number, for the whole service per client, per netblock around a client, per AS number, for the whole service
and per named netblock, in the buckets `internal/ratelimit` counts in, and and per named netblock, in the buckets `internal/ratelimit` counts in, and
+18 -10
View File
@@ -1,6 +1,7 @@
// Package bans is the ban ledger: the bans smallwebwaf makes on the // Package bans is the ban ledger: the bans smallwebwaf makes on the
// netblocks of clients that break a rate limit or a byte limit, show a // netblocks of clients that break a rate limit, a byte limit or the error
// clear sign of attack or are listed by the CrowdSec decision list, and // burst, show a clear sign of attack or are listed by the CrowdSec
// decision list, and
// those an admin makes, with their notes, as the "Bans" section of SPEC.md // those an admin makes, with their notes, as the "Bans" section of SPEC.md
// describes. The bans are kept in memory, and written to bans.json and // describes. The bans are kept in memory, and written to bans.json and
// read from it by the state package. // read from it by the state package.
@@ -103,10 +104,11 @@ type Notes struct {
ASName string `json:"as_name"` ASName string `json:"as_name"`
Country string `json:"country"` Country string `json:"country"`
// Kind, Limit, Window and Count are, for a ban for a broken limit, // Kind, Limit, Window and Count are, for a ban for a broken limit,
// what the limit was on, "requests" for a rate limit or "bytes" for a // what the limit was on, "requests" for a rate limit, "bytes" for a
// byte limit, the limit that was broken, its window, "minute", "hour" // byte limit or "refusals" for the error burst, the limit that was
// or "day", and the count reached: the client's requests, or bytes, in // broken, its window, "minute", "hour" or "day", and the count reached:
// the window, those of the request that broke the limit included. // the client's requests, bytes or refusals in the window, those of the
// request that broke the limit included.
// These are what counted toward the ban, and the window is the time // These are what counted toward the ban, and the window is the time
// over which they came. // over which they came.
Kind string `json:"kind,omitempty"` Kind string `json:"kind,omitempty"`
@@ -120,9 +122,11 @@ type Notes struct {
LimitPercent *int64 `json:"limit_percent,omitempty"` LimitPercent *int64 `json:"limit_percent,omitempty"`
LimitPercentSetting string `json:"limit_percent_setting,omitempty"` LimitPercentSetting string `json:"limit_percent_setting,omitempty"`
// RuleID and Target are, for a ban for a clear sign of attack, the id // RuleID and Target are, for a ban for a clear sign of attack, the id
// of the rule file rule that matched, and its target. // of the rule file rule that matched, and its target; TrapPath is, for
RuleID string `json:"rule_id,omitempty"` // one for a request for a path in SWWAF_TRAP_PATHS, that path.
Target string `json:"target,omitempty"` RuleID string `json:"rule_id,omitempty"`
Target string `json:"target,omitempty"`
TrapPath string `json:"trap_path,omitempty"`
// Reputation is the reputation sources that listed the client when // Reputation is the reputation sources that listed the client when
// the request that caused the ban was made, in the order the request // the request that caused the ban was made, in the order the request
// log's reputation names them. It is left out when none did. // log's reputation names them. It is left out when none did.
@@ -311,7 +315,7 @@ func (l *Ledger) WouldBanForLimit(
// notes, and returns the ban, and whether it made it, as BanForLimit // notes, and returns the ban, and whether it made it, as BanForLimit
// does. A first ban lasts AttackBanDuration; once the netblock has had // does. A first ban lasts AttackBanDuration; once the netblock has had
// one that was not lifted, the next is permanent. Its reason is "matched // one that was not lifted, the next is permanent. Its reason is "matched
// the rule <RuleID>". // the rule <RuleID>", or "asked for the trap path <TrapPath>".
func (l *Ledger) BanForAttack( func (l *Ledger) BanForAttack(
netblock netip.Prefix, now time.Time, notes Notes, netblock netip.Prefix, now time.Time, notes Notes,
) (Ban, bool) { ) (Ban, bool) {
@@ -382,6 +386,10 @@ func limitReason(notes Notes) string {
// attackReason is the reason of a ban for a clear sign of attack, with // attackReason is the reason of a ban for a clear sign of attack, with
// notes. // notes.
func attackReason(notes Notes) string { func attackReason(notes Notes) string {
if notes.TrapPath != "" {
return "asked for the trap path " + notes.TrapPath
}
return "matched the rule " + notes.RuleID return "matched the rule " + notes.RuleID
} }
+39
View File
@@ -239,6 +239,14 @@ type Config struct {
// unless RulesEnabled is false (SWWAF_RULES_ENABLED). // unless RulesEnabled is false (SWWAF_RULES_ENABLED).
RulesDir string RulesDir string
RulesEnabled bool RulesEnabled bool
// TrapPaths are the paths a request for which is a clear sign of
// attack (SWWAF_TRAP_PATHS), each starting with / and without a ?.
TrapPaths []string
// ErrorBurstThreshold is the most requests of a client within a minute
// that smallwebwaf may refuse after a rule file match or for a missing
// or wrong token; one more breaks a limit
// (SWWAF_ERROR_BURST_THRESHOLD). 0 is off.
ErrorBurstThreshold int64
// LogRemoteURL is where every line on stdout is also sent // LogRemoteURL is where every line on stdout is also sent
// (SWWAF_LOG_REMOTE_URL), nil while it is unset and nothing is sent. // (SWWAF_LOG_REMOTE_URL), nil while it is unset and nothing is sent.
// LogRemoteTLSCAs are the certificates a syslog+tls endpoint's // LogRemoteTLSCAs are the certificates a syslog+tls endpoint's
@@ -378,6 +386,8 @@ var (
errNotBytesCount = errors.New("is not response, request or both") errNotBytesCount = errors.New("is not response, request or both")
errNotPathPrefix = errors.New( errNotPathPrefix = errors.New(
"is not a path prefix starting with /, such as /assets/") "is not a path prefix starting with /, such as /assets/")
errNotTrapPath = errors.New(
"is not a path starting with / and without a ?, such as /wp-login.php")
errNotBoolean = errors.New("is not true or false") errNotBoolean = errors.New("is not true or false")
errNotLogRemoteURL = errors.New( errNotLogRemoteURL = errors.New(
"is not syslog+udp, syslog+tcp or syslog+tls with a host and a port, " + "is not syslog+udp, syslog+tcp or syslog+tls with a host and a port, " +
@@ -505,6 +515,8 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
MetricsTopN: env.numberNotOff("SWWAF_METRICS_TOP_N", "50"), MetricsTopN: env.numberNotOff("SWWAF_METRICS_TOP_N", "50"),
RulesDir: env.value("SWWAF_RULES_DIR", "/etc/smallwebwaf/rules.d"), RulesDir: env.value("SWWAF_RULES_DIR", "/etc/smallwebwaf/rules.d"),
RulesEnabled: env.boolean("SWWAF_RULES_ENABLED", "true"), RulesEnabled: env.boolean("SWWAF_RULES_ENABLED", "true"),
TrapPaths: env.trapPaths("SWWAF_TRAP_PATHS"),
ErrorBurstThreshold: env.count("SWWAF_ERROR_BURST_THRESHOLD", "30"),
LogRemoteURL: env.logRemoteURL("SWWAF_LOG_REMOTE_URL"), LogRemoteURL: env.logRemoteURL("SWWAF_LOG_REMOTE_URL"),
LogRemoteTLSCAs: env.certificates("SWWAF_LOG_REMOTE_TLS_CA_FILE"), LogRemoteTLSCAs: env.certificates("SWWAF_LOG_REMOTE_TLS_CA_FILE"),
LogRemoteBuffer: env.numberNotOff("SWWAF_LOG_REMOTE_BUFFER", "10000"), LogRemoteBuffer: env.numberNotOff("SWWAF_LOG_REMOTE_BUFFER", "10000"),
@@ -744,6 +756,15 @@ func (e *environment) pathPrefixes(name, defaultValue string) []string {
return prefixes return prefixes
} }
// trapPaths reads the setting that is the list of trap paths. It is empty
// by default.
func (e *environment) trapPaths(name string) []string {
paths, err := parseTrapPaths(e.value(name, ""))
e.check(name, err)
return paths
}
// countries reads a setting that is a list of countries. // countries reads a setting that is a list of countries.
func (e *environment) countries(name, defaultValue string) []string { func (e *environment) countries(name, defaultValue string) []string {
countries, err := parseCountries(e.value(name, defaultValue)) countries, err := parseCountries(e.value(name, defaultValue))
@@ -1536,6 +1557,24 @@ func parsePathPrefixes(value string) ([]string, error) {
return prefixes, nil return prefixes, nil
} }
// parseTrapPaths reads a comma-separated list of trap paths. Each is
// matched against a request's path as a path rule is, without the query,
// so a path that does not start with / or holds a ? would never match.
func parseTrapPaths(value string) ([]string, error) {
paths, err := parseList(value)
if err != nil {
return nil, err
}
for _, path := range paths {
if !strings.HasPrefix(path, "/") || strings.Contains(path, "?") {
return nil, fmt.Errorf("%q %w", path, errNotTrapPath)
}
}
return paths, nil
}
// countryCodes are the two-letter codes ISO 3166-1 assigns today, and XK, // countryCodes are the two-letter codes ISO 3166-1 assigns today, and XK,
// the code in common use for Kosovo. golang.org/x/text/language cannot // the code in common use for Kosovo. golang.org/x/text/language cannot
// check them: it also takes withdrawn codes such as su, and reserved ones // check them: it also takes withdrawn codes such as su, and reserved ones
+60
View File
@@ -91,6 +91,8 @@ const (
logLevel = "SWWAF_LOG_LEVEL" logLevel = "SWWAF_LOG_LEVEL"
rulesDir = "SWWAF_RULES_DIR" rulesDir = "SWWAF_RULES_DIR"
rulesEnabled = "SWWAF_RULES_ENABLED" rulesEnabled = "SWWAF_RULES_ENABLED"
trapPaths = "SWWAF_TRAP_PATHS"
errorBurstThreshold = "SWWAF_ERROR_BURST_THRESHOLD"
logRemoteURL = "SWWAF_LOG_REMOTE_URL" logRemoteURL = "SWWAF_LOG_REMOTE_URL"
logRemoteTLSCAFile = "SWWAF_LOG_REMOTE_TLS_CA_FILE" logRemoteTLSCAFile = "SWWAF_LOG_REMOTE_TLS_CA_FILE"
logRemoteBuffer = "SWWAF_LOG_REMOTE_BUFFER" logRemoteBuffer = "SWWAF_LOG_REMOTE_BUFFER"
@@ -495,6 +497,62 @@ func TestPathPrefixNotStartingWithSlashStopsTheStart(t *testing.T) {
} }
} }
func TestTrapPathsAndErrorBurstThreshold(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
env environment
paths []string
threshold int64
}{
{environment{}, []string{}, 30},
{
environment{trapPaths: "/wp-login.php, /xmlrpc.php", errorBurstThreshold: "5"},
[]string{"/wp-login.php", "/xmlrpc.php"}, 5,
},
{environment{errorBurstThreshold: off}, []string{}, 0},
} {
cfg := fromEnvironment(t, tc.env)
if !slices.Equal(cfg.TrapPaths, tc.paths) ||
cfg.ErrorBurstThreshold != tc.threshold {
t.Errorf("%v gave %v and %d, want %v and %d", tc.env, cfg.TrapPaths,
cfg.ErrorBurstThreshold, tc.paths, tc.threshold)
}
}
}
func TestInvalidTrapPathOrErrorBurstThresholdStopsTheStart(t *testing.T) {
t.Parallel()
const notTrapPath = " is not a path starting with / and without a ?, " +
"such as /wp-login.php"
for _, tc := range []struct{ name, value, want string }{
{trapPaths, "/wp-login.php,xmlrpc.php", `"xmlrpc.php"` + notTrapPath},
{trapPaths, "/xmlrpc.php?rsd", `"/xmlrpc.php?rsd"` + notTrapPath},
{
trapPaths, "/wp-login.php,,/xmlrpc.php",
`"/wp-login.php,,/xmlrpc.php" has an empty item in its list`,
},
{errorBurstThreshold, "0", `"0" must be more than zero, or off`},
{
errorBurstThreshold, "30/min",
`"30/min" is not a whole number of requests such as 1000, or off`,
},
} {
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
t.Parallel()
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
want := tc.name + ": " + tc.want
if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want)
}
})
}
}
func TestInstanceNameAndLoggedHeadersAsSet(t *testing.T) { func TestInstanceNameAndLoggedHeadersAsSet(t *testing.T) {
t.Parallel() t.Parallel()
@@ -2233,6 +2291,8 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
logLevel: "info", logLevel: "info",
rulesDir: "/etc/smallwebwaf/rules.d", rulesDir: "/etc/smallwebwaf/rules.d",
rulesEnabled: "true", rulesEnabled: "true",
trapPaths: "",
errorBurstThreshold: "30",
logRemoteURL: "", logRemoteURL: "",
logRemoteTLSCAFile: "", logRemoteTLSCAFile: "",
logRemoteBuffer: "10000", logRemoteBuffer: "10000",
+24 -19
View File
@@ -6,7 +6,6 @@ package metrics
import ( import (
"net/http" "net/http"
"strconv" "strconv"
"strings"
"time" "time"
"github.com/prometheus/client_golang/prometheus" "github.com/prometheus/client_golang/prometheus"
@@ -94,8 +93,8 @@ func New(topN int, instanceName string) *Metrics {
Help: "How long requests passed to the app took, from then to their end.", Help: "How long requests passed to the app took, from then to their end.",
}), }),
rateLimitHits: counterVec("smallwebwaf_rate_limit_hits_total", rateLimitHits: counterVec("smallwebwaf_rate_limit_hits_total",
"Requests that broke a rate limit or a byte limit, by its window and "+ "Requests that broke a rate limit, a byte limit or the error burst, by "+
"its kind, requests or bytes.", "its window and its kind, requests, bytes or refusals.",
[]string{"window", "kind"}), []string{"window", "kind"}),
sizeAndTimeLimitHits: counterVec("smallwebwaf_size_and_time_limit_hits_total", sizeAndTimeLimitHits: counterVec("smallwebwaf_size_and_time_limit_hits_total",
"Requests that passed a size or time limit, by its setting.", "Requests that passed a size or time limit, by its setting.",
@@ -407,26 +406,10 @@ func (m *Metrics) RequestEnded(
m.upstreamDuration.Observe(upstreamDuration.Seconds()) m.upstreamDuration.Observe(upstreamDuration.Seconds())
} }
if line.LimitHit != "" {
// The log line names a byte limit's window with _bytes after it.
window, isBytes := strings.CutSuffix(line.LimitHit, "_bytes")
kind := ratelimit.KindRequests
if isBytes {
kind = ratelimit.KindBytes
}
m.rateLimitHits.WithLabelValues(window, kind).Inc()
}
if limit != "" { if limit != "" {
m.sizeAndTimeLimitHits.WithLabelValues(limit).Inc() m.sizeAndTimeLimitHits.WithLabelValues(limit).Inc()
} }
if line.Offence != "" {
m.offences.WithLabelValues(line.Offence).Inc()
}
if line.Country != "" { if line.Country != "" {
m.countries.add(line.Country, line) m.countries.add(line.Country, line)
} }
@@ -436,6 +419,28 @@ func (m *Metrics) RequestEnded(
} }
} }
// LimitHit counts a request that broke a rate limit, a byte limit or the
// error burst, by the window and the kind of hit.
func (m *Metrics) LimitHit(hit ratelimit.Hit) {
m.rateLimitHits.WithLabelValues(hit.Window, hit.Kind).Inc()
}
// Offences counts the offences of r, a request that has ended, as its
// client's history counts them, by kind, named as clients.json names
// them.
func (m *Metrics) Offences(r ratelimit.Request) {
for kind, committed := range map[string]bool{
"limit": r.BrokeLimit,
"attack": r.Attack,
"rule_blocked": r.RuleBlocked,
"token_refused": r.TokenRefused,
} {
if committed {
m.offences.WithLabelValues(kind).Inc()
}
}
}
// RuleMatched counts a request that matched the rule id, whose action is // RuleMatched counts a request that matched the rule id, whose action is
// action. // action.
func (m *Metrics) RuleMatched(id, action string) { func (m *Metrics) RuleMatched(id, action string) {
+4 -2
View File
@@ -45,8 +45,9 @@ var (
// /_smallwebwaf/, once it has passed the checks. Each endpoint needs a // /_smallwebwaf/, once it has passed the checks. Each endpoint needs a
// token, sent as Authorization: Bearer <token>: the metrics // token, sent as Authorization: Bearer <token>: the metrics
// SWWAF_METRICS_TOKEN, the others SWWAF_ADMIN_TOKEN. A request without // SWWAF_METRICS_TOKEN, the others SWWAF_ADMIN_TOKEN. A request without
// it is refused with 401. An endpoint whose token is unset answers 404, // it is refused with 401, which counts toward the error burst. An
// as any other request under /_smallwebwaf/ does. // endpoint whose token is unset answers 404, as any other request under
// /_smallwebwaf/ does.
func (rq *request) answerAdmin() { func (rq *request) answerAdmin() {
rq.line.Action = requestlog.ActionAdmin rq.line.Action = requestlog.ActionAdmin
rq.startClientResponseTimeout() rq.startClientResponseTimeout()
@@ -57,6 +58,7 @@ func (rq *request) answerAdmin() {
case token == "": case token == "":
http.Error(rq.out, http.StatusText(http.StatusNotFound), http.StatusNotFound) http.Error(rq.out, http.StatusText(http.StatusNotFound), http.StatusNotFound)
case !hasToken(rq.in, token): case !hasToken(rq.in, token):
rq.tokenRefused = true
rq.out.Header().Set("WWW-Authenticate", "Bearer") rq.out.Header().Set("WWW-Authenticate", "Bearer")
rq.answer(refusal{ rq.answer(refusal{
status: http.StatusUnauthorized, status: http.StatusUnauthorized,
+76 -30
View File
@@ -1,6 +1,7 @@
package proxy package proxy
import ( import (
"net/http"
"net/netip" "net/netip"
"time" "time"
@@ -9,7 +10,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation" "sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
) )
// banResponse is a refusal answered with SWWAF_BAN_RESPONSE, and logged // banResponse is a refusal answered with SWWAF_BAN_RESPONSE, and logged
@@ -83,6 +83,48 @@ func (rq *request) countBytes() {
} }
} }
// countRefusal counts the request for the error burst once it has been
// answered, if smallwebwaf refused it after a rule file match or a trap
// path, or for a missing or wrong token, and in observe mode if enforce
// mode would have: more than SWWAF_ERROR_BURST_THRESHOLD such refusals of
// the client within a minute break a limit. A client in SWWAF_ALLOW_NETS,
// which the checks skip, is not counted, and nothing is while the
// threshold is off.
func (rq *request) countRefusal() {
cfg := rq.h.config
if cfg.ErrorBurstThreshold == 0 {
return
}
// In observe mode, a request that enforce mode would have refused
// before it reached the endpoint has had no token refused there.
tokenRefused := rq.tokenRefused && rq.line.WouldAction == "" &&
!isInside(rq.client, cfg.AllowNets)
if !rq.attack && !rq.ruleBlocked && !tokenRefused {
return
}
now := rq.h.now()
hit, over := rq.h.limiter.CountRefusal(rq.h.clientGroup(rq.client), now,
cfg.ErrorBurstThreshold)
if !over {
return
}
// What the client was sent, or in observe mode would have been.
status := rq.out.status
switch rq.line.WouldAction {
case requestlog.ActionRuleBlocked:
status = http.StatusForbidden
case requestlog.ActionBanned:
status = cfg.BanResponse
}
rq.banForLimit(now, hit, status)
}
// countedBytes returns the request's bytes, once it has ended, as the // countedBytes returns the request's bytes, once it has ended, as the
// byte limits and the anomaly thresholds count them: the response's body // byte limits and the anomaly thresholds count them: the response's body
// bytes, the request's, or both, as SWWAF_BYTES_COUNT says. For an // bytes, the request's, or both, as SWWAF_BYTES_COUNT says. For an
@@ -107,20 +149,27 @@ func (rq *request) countedBytes() int64 {
} }
// banForLimit bans the client's netblock at now for a broken limit, the // banForLimit bans the client's netblock at now for a broken limit, the
// one hit names, and notes the offence for the log line. status is what // one hit names, notes the offence for the log line and counts the hit in
// the client was sent, or is sent: SWWAF_BAN_RESPONSE for a request over // the metrics. status is what the client was sent, or is sent:
// a rate limit, the app's answer for one whose bytes broke a byte limit. // SWWAF_BAN_RESPONSE for a request over a rate limit, the app's answer for
// The ban's notes give the client's limit percentage for that kind of // one whose bytes broke a byte limit, the refusal for one that broke the
// limit. The ban sets the client's counters back to zero. In observe mode // error burst. The ban's notes give the client's limit percentage for a
// it makes no ban and sets nothing back, and raises the alert for the ban // rate limit or a byte limit; the error burst is not lowered. The ban sets
// it would have made, if that alert would be sent. // the client's counters back to zero. In observe mode it makes no ban and
// sets nothing back, and raises the alert for the ban it would have made,
// if that alert would be sent.
func (rq *request) banForLimit(now time.Time, hit ratelimit.Hit, status int) { func (rq *request) banForLimit(now time.Time, hit ratelimit.Hit, status int) {
rq.line.LimitHit = hit.Window switch hit.Kind {
if hit.Kind == ratelimit.KindBytes { case ratelimit.KindBytes:
rq.line.LimitHit += "_bytes" // as counts names the byte totals rq.line.LimitHit = hit.Window + "_bytes" // as counts names the byte totals
case ratelimit.KindRefusals:
rq.line.LimitHit = requestlog.LimitHitErrorBurst
default:
rq.line.LimitHit = hit.Window
} }
rq.line.Offence = requestlog.OffenceLimit rq.line.Offence = requestlog.OffenceLimit
rq.h.metrics.LimitHit(hit)
netblock := rq.h.netblock(rq.client) netblock := rq.h.netblock(rq.client)
if rq.h.config.Observe && !rq.wouldAlertBan(netblock, now, bans.CauseLimit) { if rq.h.config.Observe && !rq.wouldAlertBan(netblock, now, bans.CauseLimit) {
@@ -140,13 +189,13 @@ func (rq *request) banForLimit(now time.Time, hit ratelimit.Hit, status int) {
Requests: rq.netblockRequests(netblock), Requests: rq.netblockRequests(netblock),
} }
percent := rq.limitPercent switch hit.Kind {
if hit.Kind == ratelimit.KindBytes { case ratelimit.KindRequests:
percent = rq.bytesPercent notes.LimitPercent, notes.LimitPercentSetting = rq.limitPercent.logged()
case ratelimit.KindBytes:
notes.LimitPercent, notes.LimitPercentSetting = rq.bytesPercent.logged()
} }
notes.LimitPercent, notes.LimitPercentSetting = percent.logged()
if rq.h.config.Observe { if rq.h.config.Observe {
ban, wouldBan := rq.h.ledger.WouldBanForLimit(netblock, now, notes) ban, wouldBan := rq.h.ledger.WouldBanForLimit(netblock, now, notes)
if wouldBan { if wouldBan {
@@ -166,25 +215,22 @@ func (rq *request) banForLimit(now time.Time, hit ratelimit.Hit, status int) {
} }
// banForAttack bans the client's netblock at now for a clear sign of // banForAttack bans the client's netblock at now for a clear sign of
// attack, the match of rule, a ban rule. In observe mode it makes no ban, // attack, which notes name: the ban rule that matched, or the trap path
// and raises the alert for the ban it would have made, if that alert // asked for. It fills in the rest of the notes. In observe mode it makes
// would be sent. // no ban, and raises the alert for the ban it would have made, if that
func (rq *request) banForAttack(now time.Time, rule rules.Rule) { // alert would be sent.
func (rq *request) banForAttack(now time.Time, notes bans.Notes) {
netblock := rq.h.netblock(rq.client) netblock := rq.h.netblock(rq.client)
if rq.h.config.Observe && !rq.wouldAlertBan(netblock, now, bans.CauseAttack) { if rq.h.config.Observe && !rq.wouldAlertBan(netblock, now, bans.CauseAttack) {
return return
} }
notes := bans.Notes{ notes.ASN = rq.line.ASN
ASN: rq.line.ASN, notes.ASName = rq.line.ASName
ASName: rq.line.ASName, notes.Country = rq.line.Country
Country: rq.line.Country, notes.Reputation = rq.reputation
RuleID: rule.ID, notes.Request = rq.noted(now, rq.h.config.BanResponse)
Target: rule.Target, notes.Requests = rq.netblockRequests(netblock)
Reputation: rq.reputation,
Request: rq.noted(now, rq.h.config.BanResponse),
Requests: rq.netblockRequests(netblock),
}
if rq.h.config.Observe { if rq.h.config.Observe {
ban, wouldBan := rq.h.ledger.WouldBanForAttack(netblock, now, notes) ban, wouldBan := rq.h.ledger.WouldBanForAttack(netblock, now, notes)
+396
View File
@@ -0,0 +1,396 @@
package proxy_test
import (
"maps"
"net/http"
"net/netip"
"reflect"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/proxy"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
const errorBurstThreshold = "SWWAF_ERROR_BURST_THRESHOLD"
// refused is a request the tests here send, which smallwebwaf refuses
// after a rule file match, or for a missing or wrong token.
type refused int
const (
// blockRule is a request testRules' block rule refuses with 403.
blockRule refused = iota
// banRule is one its ban rule refuses with 403, and bans the client
// for.
banRule
// noMetricsToken is one for the metrics without a token, and
// wrongAdminToken one for the bans with the metrics token, each
// refused with 401.
noMetricsToken
wrongAdminToken
)
// send sends r from the client at from, checks its answer and log line as
// sender.request does, and returns the line.
func (r refused) send(s *sender, from string) logLine {
s.t.Helper()
switch r {
case blockRule:
return s.request(from, blockedPath, http.StatusForbidden,
requestlog.ActionRuleBlocked)
case banRule:
return s.request(from, probePath, http.StatusForbidden, requestlog.ActionBanned)
case noMetricsToken:
return s.request(from, proxy.MetricsPath, http.StatusUnauthorized,
requestlog.ActionAdmin)
case wrongAdminToken:
line, _ := s.requestWithHeader(from, proxy.BansPath, "Authorization: "+bearer,
http.StatusUnauthorized, requestlog.ActionAdmin)
return line
}
s.t.Fatalf("no request for the refusal %d", r)
return logLine{}
}
// startForErrorBurst is startWithClock with testRules, both tokens and
// SWWAF_ERROR_BURST_THRESHOLD at threshold, and the settings in env.
func startForErrorBurst(
t *testing.T, threshold string, env map[string]string,
) (*sender, *clock, *proxy.Server) {
t.Helper()
settings := map[string]string{
errorBurstThreshold: threshold,
rulesDir: writeRules(t, testRules),
adminToken: adminSecret,
metricsToken: token,
}
maps.Copy(settings, env)
return startWithClock(t, "", settings)
}
func TestErrorBurstBreaksAtOneOverTheThreshold(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
// refusals are four, one over the threshold of three.
refusals []refused
}{
{"block rule", []refused{blockRule, blockRule, blockRule, blockRule}},
{
"missing or wrong token",
[]refused{noMetricsToken, wrongAdminToken, noMetricsToken, wrongAdminToken},
},
{
"a mix ending in a ban rule",
[]refused{blockRule, noMetricsToken, blockRule, banRule},
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
s, _, _ := startForErrorBurst(t, "3", nil)
// Three refusals break nothing, and the app's answers between
// them are not counted.
for i, r := range tc.refusals[:3] {
line := r.send(s, client)
if line.LimitHit != "" || line.Offence != "" {
t.Errorf("refusal %d: log line has limit_hit %q and offence %q, "+
"want none", i+1, line.LimitHit, line.Offence)
}
s.get(client, http.StatusOK, requestlog.ActionForward)
}
// The fourth is answered as the others were, breaks the error
// burst, and bans the client.
line := tc.refusals[3].send(s, client)
if line.LimitHit != requestlog.LimitHitErrorBurst ||
line.Offence != requestlog.OffenceLimit {
t.Errorf("log line has limit_hit %q and offence %q, want error_burst "+
"and limit", line.LimitHit, line.Offence)
}
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
})
}
}
func TestErrorBurstBanNotesHistoryAndMetrics(t *testing.T) {
t.Parallel()
const scraper = "192.0.2.200"
s, clk, server := startForErrorBurst(t, "2", nil)
start := clk.Now()
blockRule.send(s, client)
wrongAdminToken.send(s, client)
line := blockRule.send(s, client)
expires := start.Add(time.Hour)
if line.BanExpires != requestlog.FormatTime(expires) {
t.Errorf("log line has ban_expires %q, want an hour on", line.BanExpires)
}
netblock := netip.MustParsePrefix(client + "/32")
want := bans.Ban{
Netblock: netblock,
Start: start,
Expires: expires,
Cause: bans.CauseLimit,
Reason: "refusals per minute over the limit of 2",
Notes: bans.Notes{
Kind: ratelimit.KindRefusals,
Limit: 2,
Window: minute,
Count: 3,
Request: bans.Request{
Time: start,
Method: http.MethodGet,
Host: appHost,
Path: blockedPath,
Status: http.StatusForbidden,
UserAgent: userAgent,
},
Requests: 3,
},
}
got := server.Ledger.Bans(netblock)
if len(got) != 1 || !reflect.DeepEqual(got[0], want) {
t.Fatalf("bans\n%+v\nwant\n%+v", got, want)
}
wantOffences := ratelimit.Offences{Limit: 1, RuleBlocked: 2, TokenRefused: 1}
if offences := historyOf(t, server, client).Offences; offences != wantOffences {
t.Errorf("history counts the offences %+v, want %+v", offences, wantOffences)
}
metrics := s.scrape(scraper)
wantMetric(t, metrics, `smallwebwaf_rate_limit_hits_total{instance="app",`+
`kind="refusals",window="minute"}`, 1)
wantMetric(t, metrics, `smallwebwaf_offences_total{instance="app",kind="limit"}`, 1)
wantMetric(t, metrics,
`smallwebwaf_offences_total{instance="app",kind="rule_blocked"}`, 2)
wantMetric(t, metrics,
`smallwebwaf_offences_total{instance="app",kind="token_refused"}`, 1)
wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="limit",instance="app"}`, 1)
}
func TestErrorBurstIsNotLoweredForAClientWithLowerLimits(t *testing.T) {
t.Parallel()
geojsURL, _ := startGeoJS(t)
s, _, server := startWithClock(t, geojsURL, map[string]string{
errorBurstThreshold: "2",
rulesDir: writeRules(t, testRules),
countryLimitPercent: countryDEHalf,
})
// Half of the threshold would be one, which the second refusal is over.
for range 2 {
line := blockRule.send(s, fromDE)
if line.LimitHit != "" {
t.Errorf("log line has limit_hit %q, want none", line.LimitHit)
}
}
blockRule.send(s, fromDE)
got := server.Ledger.Bans(netip.MustParsePrefix(fromDE + "/32"))
if len(got) != 1 || got[0].Notes.Limit != 2 || got[0].Notes.LimitPercent != nil {
t.Errorf("bans %+v, want one for the limit of 2, without a limit percentage", got)
}
}
func TestErrorBurstDoesNotCountTheAppsAnswers(t *testing.T) {
t.Parallel()
statuses := map[string]int{
"/missing": http.StatusNotFound,
"/private": http.StatusUnauthorized,
"/forbidden": http.StatusForbidden,
}
s, _, _, queue := startAppWithAlerts(t, func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(statuses[r.URL.Path])
}, map[string]string{errorBurstThreshold: "1", rulesDir: writeRules(t, testRules)})
for range 2 {
for path, status := range statuses {
s.request(client, path, status, requestlog.ActionForward)
}
}
// The first refusal is one, not over the threshold.
line := blockRule.send(s, client)
if line.LimitHit != "" {
t.Errorf("log line has limit_hit %q, want none", line.LimitHit)
}
// No ban was made, nor its alert raised.
s.request(client, "/missing", http.StatusNotFound, requestlog.ActionForward)
wantAlerts(t, queue)
}
func TestErrorBurstOffOrAtItsDefault(t *testing.T) {
t.Parallel()
const off = "off"
for _, tc := range []struct {
threshold string
// broken is whether the 31st refusal breaks the error burst.
broken bool
}{
{"", true},
{off, false},
} {
t.Run(errorBurstThreshold+"="+tc.threshold, func(t *testing.T) {
t.Parallel()
env := map[string]string{rulesDir: writeRules(t, testRules)}
if tc.threshold != "" {
env[errorBurstThreshold] = tc.threshold
}
s, _, _ := startWithClock(t, "", env)
var line logLine
for range 31 {
line = blockRule.send(s, client)
}
if broken := line.LimitHit == requestlog.LimitHitErrorBurst; broken != tc.broken {
t.Errorf("the 31st refusal broke the error burst: %t, want %t",
broken, tc.broken)
}
})
}
}
func TestErrorBurstCountsEachClientTheChecksApplyTo(t *testing.T) {
t.Parallel()
const (
allowed = "192.0.2.60" // in SWWAF_ALLOW_NETS
exempt = "192.0.2.50" // in SWWAF_RATE_LIMIT_EXEMPT_NETS
)
s, _, _ := startForErrorBurst(t, "1", map[string]string{
allowNets: allowed, rateLimitExemptNets: exempt,
})
// A client in SWWAF_ALLOW_NETS still needs the token, but is not
// counted.
for range 3 {
line := noMetricsToken.send(s, allowed)
if line.LimitHit != "" {
t.Errorf("log line has limit_hit %q, want none", line.LimitHit)
}
}
// One the rate limits do not apply to is.
noMetricsToken.send(s, exempt)
line := wrongAdminToken.send(s, exempt)
if line.LimitHit != requestlog.LimitHitErrorBurst {
t.Errorf("log line has limit_hit %q, want error_burst", line.LimitHit)
}
s.get(exempt, http.StatusForbidden, requestlog.ActionBanned)
}
func TestErrorBurstBanSetsTheRefusalsBackToZero(t *testing.T) {
t.Parallel()
s, clk, _ := startForErrorBurst(t, "1", map[string]string{limitBanDuration: "1s"})
blockRule.send(s, client)
blockRule.send(s, client)
// Within the same minute, once the ban has ended, the next refusal is
// the first again.
clk.advance(time.Second)
line := blockRule.send(s, client)
if line.LimitHit != "" {
t.Errorf("log line has limit_hit %q, want none", line.LimitHit)
}
}
func TestObserveModeLogsAndAlertsTheErrorBurst(t *testing.T) {
t.Parallel()
s, clk, server, queue := startWithAlerts(t, map[string]string{
mode: observe,
errorBurstThreshold: "1",
rulesDir: writeRules(t, testRules),
adminToken: adminSecret,
})
start := clk.Now()
held := bans.Ban{
Netblock: netip.MustParsePrefix(otherClient + "/32"),
Start: start,
Expires: start.Add(time.Hour),
Cause: bans.CauseAdmin,
}
server.Ledger.Load([]bans.Ban{held})
// Under a ban, enforce mode would have refused these before the
// endpoint, so their tokens are not counted.
for range 2 {
line := wrongAdminToken.send(s, otherClient)
wantWouldAction(t, line, requestlog.ActionBanned)
if line.LimitHit != "" {
t.Errorf("log line has limit_hit %q, want none", line.LimitHit)
}
}
// The block rule's refusal, which enforce mode would have answered 403,
// is the second of the client's, and would have banned it.
wrongAdminToken.send(s, client)
line := s.request(client, blockedPath, http.StatusOK, requestlog.ActionForward)
wantWouldAction(t, line, requestlog.ActionRuleBlocked)
if line.LimitHit != requestlog.LimitHitErrorBurst || line.BanExpires != "" {
t.Errorf("log line has limit_hit %q and ban_expires %q, want error_burst "+
"and none", line.LimitHit, line.BanExpires)
}
if got := server.Ledger.Snapshot(); len(got) != 1 || !reflect.DeepEqual(got[0], held) {
t.Errorf("bans %+v, want only the one held", got)
}
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
if len(waiting) != 1 {
t.Fatalf("%d alerts wait, want 1: %+v", len(waiting), waiting)
}
notes, _ := waiting[0].Detail["notes"].(bans.Notes)
if notes.Kind != ratelimit.KindRefusals || notes.Count != 2 ||
notes.Request.Status != http.StatusForbidden {
t.Errorf("the alert's notes are %+v, want two refusals, the last answered 403",
notes)
}
alert := banAlert(alerts.EventBan, start, client, bans.Ban{
Netblock: netip.MustParsePrefix(client + "/32"), Cause: bans.CauseLimit,
Reason: "refusals per minute over the limit of 1", Notes: notes,
}, requestlog.FormatTime(start.Add(time.Hour)))
alert.Detail["mode"] = observe
wantAlerts(t, queue, alert)
}
+4 -1
View File
@@ -282,9 +282,12 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
return return
} }
// Once the request has ended, before its log line is written. // Once the request has ended, before its log line is written. The
// last deferred runs first: countRefusal before addToHistory, so that
// a broken error burst is in the client's history.
defer rq.addToHistory() defer rq.addToHistory()
defer rq.countAnomalies() defer rq.countAnomalies()
defer rq.countRefusal()
refused := rq.check(r.Context()) refused := rq.check(r.Context())
rq.checked = time.Now() rq.checked = time.Now()
+19 -6
View File
@@ -701,10 +701,14 @@ func TestIPv6ClientCostsOneAbuseIPDBCheckWhicheverOfItsAddressesSends(t *testing
wantAbuseIPDBChecks(t, server, 1) wantAbuseIPDBChecks(t, server, 1)
} }
// probePath is the path the ban rule of testRules, probe, matches. // probePath is the path the ban rule of testRules, probe, matches, and
const probePath = "/.env" // blockedPath the one its block rule, blocked, matches.
const (
probePath = "/.env"
blockedPath = "/blocked"
)
func TestClientARuleRefusedIsCheckedWithAbuseIPDBAtItsNextRequest(t *testing.T) { func TestClientRefusedForAnOffenceIsCheckedWithAbuseIPDBAtItsNextRequest(t *testing.T) {
t.Parallel() t.Parallel()
for _, tc := range []struct { for _, tc := range []struct {
@@ -718,13 +722,21 @@ func TestClientARuleRefusedIsCheckedWithAbuseIPDBAtItsNextRequest(t *testing.T)
want ratelimit.Offences want ratelimit.Offences
}{ }{
{ {
"a block rule", "/blocked", http.StatusForbidden, requestlog.ActionRuleBlocked, "a block rule", blockedPath, http.StatusForbidden, requestlog.ActionRuleBlocked,
ratelimit.Offences{RuleBlocked: 1}, ratelimit.Offences{RuleBlocked: 1},
}, },
{ {
"a ban rule", probePath, http.StatusForbidden, requestlog.ActionBanned, "a ban rule", probePath, http.StatusForbidden, requestlog.ActionBanned,
ratelimit.Offences{Attack: 1}, ratelimit.Offences{Attack: 1},
}, },
{
"a trap path", "/xmlrpc.php", http.StatusForbidden, requestlog.ActionBanned,
ratelimit.Offences{Attack: 1},
},
{
"a missing token", proxy.MetricsPath, http.StatusUnauthorized,
requestlog.ActionAdmin, ratelimit.Offences{TokenRefused: 1},
},
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
t.Parallel() t.Parallel()
@@ -732,6 +744,7 @@ func TestClientARuleRefusedIsCheckedWithAbuseIPDBAtItsNextRequest(t *testing.T)
s, clk, server := startWithClock(t, "", map[string]string{ s, clk, server := startWithClock(t, "", map[string]string{
abuseIPDBKey: accountKey, reputationAction: actionLog, abuseIPDBKey: accountKey, reputationAction: actionLog,
rulesDir: writeRules(t, testRules), attackBanDuration: "1h", rulesDir: writeRules(t, testRules), attackBanDuration: "1h",
trapPaths: trapPathList, metricsToken: token,
}) })
s.request(client, tc.path, tc.status, tc.action) s.request(client, tc.path, tc.status, tc.action)
@@ -741,8 +754,8 @@ func TestClientARuleRefusedIsCheckedWithAbuseIPDBAtItsNextRequest(t *testing.T)
t.Errorf("history counts the offences %+v, want %+v", got, tc.want) t.Errorf("history counts the offences %+v, want %+v", got, tc.want)
} }
// Its next request, once a ban rule's ban has ended, has it // Its next request, once any ban for a clear sign of attack
// checked. // has ended, has it checked.
clk.advance(time.Hour) clk.advance(time.Hour)
s.get(client, http.StatusOK, requestlog.ActionForward) s.get(client, http.StatusOK, requestlog.ActionForward)
wantAbuseIPDBChecks(t, server, 1) wantAbuseIPDBChecks(t, server, 1)
+23 -14
View File
@@ -64,10 +64,11 @@ type request struct {
// limits and for the byte limits. // limits and for the byte limits.
counted bool counted bool
limitPercent, bytesPercent percentage limitPercent, bytesPercent percentage
// attack is true for a request that matched a ban rule, and // attack is true for a request that matched a ban rule or asked for a
// ruleBlocked for one a block rule refused, each an offence its // trap path, ruleBlocked for one a block rule refused, and
// client's history counts. // tokenRefused for one refused for a missing or wrong token, each an
attack, ruleBlocked bool // offence its client's history counts.
attack, ruleBlocked, tokenRefused bool
// blocklisted is true once a blocklist is found to list the client, // blocklisted is true once a blocklist is found to list the client,
// dnsblListed once a DNSBL zone's verdict is, and abuseIPDBHit once // dnsblListed once a DNSBL zone's verdict is, and abuseIPDBHit once
// AbuseIPDB's score of it is a hit. // AbuseIPDB's score of it is a hit.
@@ -232,9 +233,9 @@ func (rq *request) check(ctx context.Context) *refusal {
// rate limits, unless the client is in SWWAF_RATE_LIMIT_EXEMPT_NETS or the // rate limits, unless the client is in SWWAF_RATE_LIMIT_EXEMPT_NETS or the
// request's path is exempt under SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that // request's path is exempt under SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that
// every other request is counted, each of them by the client's limit // every other request is counted, each of them by the client's limit
// percentages, and last the rule files. A request exempt from the rate // percentages, then SWWAF_TRAP_PATHS, and last the rule files. A request
// limits is exempt from the byte limits too. ctx is the request's own // exempt from the rate limits is exempt from the byte limits too. ctx is
// context. // the request's own context.
func (rq *request) checkClient(ctx context.Context) string { func (rq *request) checkClient(ctx context.Context) string {
cfg := rq.h.config cfg := rq.h.config
if isInside(rq.client, cfg.AllowNets) { if isInside(rq.client, cfg.AllowNets) {
@@ -281,6 +282,10 @@ func (rq *request) checkClient(ctx context.Context) string {
return requestlog.ActionRateLimited return requestlog.ActionRateLimited
} }
if rq.trapPath(now) {
return requestlog.ActionBanned
}
return rq.checkRules(now) return rq.checkRules(now)
} }
@@ -541,14 +546,14 @@ func timing(start, end time.Time) *float64 {
} }
// addToHistory adds the request, which has ended, to its client's // addToHistory adds the request, which has ended, to its client's
// history, and then the lookup's answer about the client, as // history, and counts its offences in the metrics, and then the lookup's
// answerAtTheEnd gives it, to that history and to the notes of the bans // answer about the client, as answerAtTheEnd gives it, to that history and
// on its netblock: an answer may have come before either was there, and // to the notes of the bans on its netblock: an answer may have come
// one from GeoJS that comes later is added when it comes. // before either was there, and one from GeoJS that comes later is added
// when it comes.
func (rq *request) addToHistory() { func (rq *request) addToHistory() {
forwarded := !rq.upstreamStart.IsZero() forwarded := !rq.upstreamStart.IsZero()
request := ratelimit.Request{
rq.h.limiter.AddToHistory(rq.h.clientGroup(rq.client), rq.h.now(), ratelimit.Request{
Forwarded: forwarded, Forwarded: forwarded,
Refused: !forwarded && rq.refused.Load() != nil, Refused: !forwarded && rq.refused.Load() != nil,
Status: rq.out.status, Status: rq.out.status,
@@ -557,7 +562,11 @@ func (rq *request) addToHistory() {
BrokeLimit: rq.line.Offence == requestlog.OffenceLimit, BrokeLimit: rq.line.Offence == requestlog.OffenceLimit,
Attack: rq.attack, Attack: rq.attack,
RuleBlocked: rq.ruleBlocked, RuleBlocked: rq.ruleBlocked,
}) TokenRefused: rq.tokenRefused,
}
rq.h.limiter.AddToHistory(rq.h.clientGroup(rq.client), rq.h.now(), request)
rq.h.metrics.Offences(request)
answer, found := rq.answerAtTheEnd() answer, found := rq.answerAtTheEnd()
if found { if found {
+3
View File
@@ -204,6 +204,9 @@ func TestMetricsCountRuleMatchesAndBansForAnAttack(t *testing.T) {
wantMetric(t, metrics, `smallwebwaf_rules_loaded{instance="app"}`, 2) wantMetric(t, metrics, `smallwebwaf_rules_loaded{instance="app"}`, 2)
wantMetric(t, metrics, `smallwebwaf_requests_total{action="rule_blocked",`+ wantMetric(t, metrics, `smallwebwaf_requests_total{action="rule_blocked",`+
`instance="app",status_class="4xx"}`, 1) `instance="app",status_class="4xx"}`, 1)
wantMetric(t, metrics,
`smallwebwaf_offences_total{instance="app",kind="rule_blocked"}`, 1)
wantMetric(t, metrics, `smallwebwaf_offences_total{instance="app",kind="attack"}`, 1)
wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="attack",instance="app"}`, 1) wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="attack",instance="app"}`, 1)
wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="limit",instance="app"}`, 0) wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="limit",instance="app"}`, 0)
wantMetric(t, metrics, `smallwebwaf_permanent_bans{instance="app"}`, 1) wantMetric(t, metrics, `smallwebwaf_permanent_bans{instance="app"}`, 1)
+20 -1
View File
@@ -1,12 +1,31 @@
package proxy package proxy
import ( import (
"slices"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules" "sneak.berlin/go/smallwebwaf/internal/rules"
) )
// trapPath reports whether the request asks for a path in
// SWWAF_TRAP_PATHS: its path as a path rule sees it, before any decoding
// and without the query, is one of them. Such a request is a clear sign of
// attack, as a ban rule's match is: it bans the client's netblock, or in
// observe mode raises the alert for the ban it would have made.
func (rq *request) trapPath(now time.Time) bool {
path := rules.Path(rq.in)
if !slices.Contains(rq.h.config.TrapPaths, path) {
return false
}
rq.attack = true
rq.banForAttack(now, bans.Notes{TrapPath: path})
return true
}
// checkRules checks the request against the rules of the rule files at // checkRules checks the request against the rules of the rule files at
// now, notes the ids of those it matches in the log line, and returns the // now, notes the ids of those it matches in the log line, and returns the
// action of the rule that refuses it, ActionRuleBlocked for a block rule // action of the rule that refuses it, ActionRuleBlocked for a block rule
@@ -34,7 +53,7 @@ func (rq *request) checkRules(now time.Time) string {
return requestlog.ActionRuleBlocked return requestlog.ActionRuleBlocked
case rules.ActionBan: case rules.ActionBan:
rq.attack = true rq.attack = true
rq.banForAttack(now, last) rq.banForAttack(now, bans.Notes{RuleID: last.ID, Target: last.Target})
return requestlog.ActionBanned return requestlog.ActionBanned
default: default:
+115
View File
@@ -0,0 +1,115 @@
package proxy_test
import (
"net/http"
"net/netip"
"reflect"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// trapPaths is the setting's name, and trapPathList what the tests set it
// to.
const (
trapPaths = "SWWAF_TRAP_PATHS"
trapPathList = "/wp-login.php,/xmlrpc.php"
)
func TestTrapPathBansAsABanRuleDoes(t *testing.T) {
t.Parallel()
const allowed = "192.0.2.60" // in SWWAF_ALLOW_NETS
// A block rule for the same path: the trap path comes first.
s, clk, server := startWithClock(t, "", map[string]string{
trapPaths: trapPathList,
rulesDir: writeRules(t, `wp path block ^/wp-login\.php$`),
allowNets: allowed,
banResponse: "429",
})
start := clk.Now()
// Only the path itself, as the client sent it, is a trap path.
for _, path := range []string{
"/wp-login.php/", "/WP-LOGIN.PHP", "/blog/xmlrpc.php", "/%77p-login.php",
} {
s.request(otherClient, path, http.StatusOK, requestlog.ActionForward)
}
// A client in SWWAF_ALLOW_NETS is not checked.
s.request(allowed, "/xmlrpc.php", http.StatusOK, requestlog.ActionForward)
// The query is not part of the path.
line := s.request(client, "/wp-login.php?redirect_to=x", http.StatusTooManyRequests,
requestlog.ActionBanned)
wantRuleIDs(t, line)
if line.BanExpires != requestlog.FormatTime(start.Add(7*24*time.Hour)) {
t.Errorf("log line has ban_expires %q, want seven days on", line.BanExpires)
}
netblock := netip.MustParsePrefix(client + "/32")
want := bans.Ban{
Netblock: netblock,
Start: start,
Expires: start.Add(7 * 24 * time.Hour),
Cause: bans.CauseAttack,
Reason: "asked for the trap path /wp-login.php",
Notes: bans.Notes{
TrapPath: "/wp-login.php",
Request: bans.Request{
Time: start,
Method: http.MethodGet,
Host: appHost,
Path: "/wp-login.php?redirect_to=x",
Status: http.StatusTooManyRequests,
UserAgent: userAgent,
},
Requests: 1,
},
}
got := server.Ledger.Bans(netblock)
if len(got) != 1 || !reflect.DeepEqual(got[0], want) {
t.Fatalf("bans\n%+v\nwant\n%+v", got, want)
}
// The next request is refused under the ban, and makes it permanent.
line = s.get(client, http.StatusTooManyRequests, requestlog.ActionBanned)
if line.BanExpires != permanent {
t.Errorf("log line has ban_expires %q, want permanent", line.BanExpires)
}
}
func TestTrapPathsNeedNoRuleFiles(t *testing.T) {
t.Parallel()
s, _, _ := startWithClock(t, "", map[string]string{
trapPaths: trapPathList,
"SWWAF_RULES_ENABLED": "false",
})
s.request(client, "/xmlrpc.php", http.StatusForbidden, requestlog.ActionBanned)
}
func TestObserveModeLogsWhatATrapPathWouldDo(t *testing.T) {
t.Parallel()
s, _, server := startWithClock(t, "", map[string]string{
trapPaths: trapPathList,
mode: observe,
})
line := s.request(client, "/xmlrpc.php", http.StatusOK, requestlog.ActionForward)
wantWouldAction(t, line, requestlog.ActionBanned)
// No ban was made.
s.get(client, http.StatusOK, requestlog.ActionForward)
if got := server.Ledger.Snapshot(); len(got) != 0 {
t.Errorf("bans %+v, want none", got)
}
}
+65 -28
View File
@@ -25,6 +25,9 @@ const (
KindRequests = "requests" KindRequests = "requests"
// KindBytes is a byte limit, on a client's bytes. // KindBytes is a byte limit, on a client's bytes.
KindBytes = "bytes" KindBytes = "bytes"
// KindRefusals is the error burst, on a client's requests smallwebwaf
// refused after a rule file match or for a missing or wrong token.
KindRefusals = "refusals"
) )
// Limits are the most requests a client may make in a minute, an hour and // Limits are the most requests a client may make in a minute, an hour and
@@ -50,18 +53,20 @@ type Limiter struct {
} }
// Client is a client in the table, as clients.json holds it: its buckets // Client is a client in the table, as clients.json holds it: its buckets
// of requests and of bytes in each window, and its history. // of requests and of bytes in each window, its buckets of refusals in the
// minute, which the error burst counts, and its history.
// //
//nolint:tagliatelle // the state files use snake_case, as the request log does //nolint:tagliatelle // the state files use snake_case, as the request log does
type Client struct { type Client struct {
Client netip.Prefix `json:"client"` Client netip.Prefix `json:"client"`
Minute Buckets `json:"minute"` Minute Buckets `json:"minute"`
Hour Buckets `json:"hour"` Hour Buckets `json:"hour"`
Day Buckets `json:"day"` Day Buckets `json:"day"`
MinuteBytes Buckets `json:"minute_bytes"` MinuteBytes Buckets `json:"minute_bytes"`
HourBytes Buckets `json:"hour_bytes"` HourBytes Buckets `json:"hour_bytes"`
DayBytes Buckets `json:"day_bytes"` DayBytes Buckets `json:"day_bytes"`
History History `json:"history"` MinuteRefusals Buckets `json:"minute_refusals"`
History History `json:"history"`
} }
// Buckets are a client's two buckets in one window: the requests, or the // Buckets are a client's two buckets in one window: the requests, or the
@@ -116,12 +121,15 @@ type Responses struct {
// //
//nolint:tagliatelle // the state files use snake_case, as the request log does //nolint:tagliatelle // the state files use snake_case, as the request log does
type Offences struct { type Offences struct {
// Limit is its requests that broke a rate limit or a byte limit, // Limit is its requests that broke a rate limit, a byte limit or the
// Attack those that matched a ban rule, a clear sign of attack, and // error burst, Attack those that were a clear sign of attack, a match
// RuleBlocked those a block rule refused. // of a ban rule or a request for a trap path, RuleBlocked those a block
Limit int64 `json:"limit"` // rule refused, and TokenRefused those refused for a missing or wrong
Attack int64 `json:"attack"` // token.
RuleBlocked int64 `json:"rule_blocked"` Limit int64 `json:"limit"`
Attack int64 `json:"attack"`
RuleBlocked int64 `json:"rule_blocked"`
TokenRefused int64 `json:"token_refused"`
} }
// Request is what a client's history keeps of one of its requests. // Request is what a client's history keeps of one of its requests.
@@ -138,12 +146,14 @@ type Request struct {
// and of its response. // and of its response.
RequestBytes int64 RequestBytes int64
ResponseBytes int64 ResponseBytes int64
// BrokeLimit is true for a request that broke a rate limit or a byte // BrokeLimit is true for a request that broke a rate limit, a byte
// limit, Attack for one that matched a ban rule, and RuleBlocked for // limit or the error burst, Attack for one that matched a ban rule or
// one a block rule refused. // asked for a trap path, RuleBlocked for one a block rule refused, and
BrokeLimit bool // TokenRefused for one refused for a missing or wrong token.
Attack bool BrokeLimit bool
RuleBlocked bool Attack bool
RuleBlocked bool
TokenRefused bool
} }
// New returns a Limiter for limits, with no client counted yet, whose // New returns a Limiter for limits, with no client counted yet, whose
@@ -175,17 +185,18 @@ func New(limits Limits, maxClients int) *Limiter {
} }
} }
// Hit is a request that takes a client over a rate limit, or whose bytes // Hit is a request that takes a client over a rate limit or the error
// take it over a byte limit. // burst, or whose bytes take it over a byte limit.
type Hit struct { type Hit struct {
// Kind is KindRequests for a rate limit, KindBytes for a byte limit. // Kind is KindRequests for a rate limit, KindBytes for a byte limit,
// KindRefusals for the error burst.
Kind string Kind string
// Window is "minute", "hour" or "day". // Window is "minute", "hour" or "day".
Window string Window string
// Limit is the window's limit, as the client's percentage of it. // Limit is the window's limit, as the client's percentage of it.
Limit int64 Limit int64
// Count is the client's requests, or bytes, counted in the window, // Count is the client's requests, bytes or refusals counted in the
// this request's included. // window, this request's included.
Count float64 Count float64
} }
@@ -224,8 +235,25 @@ func (l *Limiter) CountBytes(
return l.count(client, now, 0, bytes, percent) return l.count(client, now, 0, bytes, percent)
} }
// Reset sets client's counts of requests and of bytes in every window // CountRefusal counts a request from client at now that smallwebwaf
// back to zero. Its history keeps its totals. // refused after a rule file match or for a missing or wrong token, and
// reports whether the client's refusals in the minute that ends at now,
// this one included, are more than threshold, which breaks the error
// burst, and the hit.
func (l *Limiter) CountRefusal(
client netip.Prefix, now time.Time, threshold int64,
) (Hit, bool) {
l.mu.Lock()
defer l.mu.Unlock()
count := l.get(client).MinuteRefusals.Add(now, time.Minute, 1)
hit := Hit{Kind: KindRefusals, Window: "minute", Limit: threshold, Count: count}
return hit, count > float64(threshold)
}
// Reset sets client's counts of requests, of bytes and of refusals in
// every window back to zero. Its history keeps its totals.
func (l *Limiter) Reset(client netip.Prefix) { func (l *Limiter) Reset(client netip.Prefix) {
l.mu.Lock() l.mu.Lock()
defer l.mu.Unlock() defer l.mu.Unlock()
@@ -234,6 +262,7 @@ func (l *Limiter) Reset(client netip.Prefix) {
if seen { if seen {
c.Minute, c.Hour, c.Day = Buckets{}, Buckets{}, Buckets{} c.Minute, c.Hour, c.Day = Buckets{}, Buckets{}, Buckets{}
c.MinuteBytes, c.HourBytes, c.DayBytes = Buckets{}, Buckets{}, Buckets{} c.MinuteBytes, c.HourBytes, c.DayBytes = Buckets{}, Buckets{}, Buckets{}
c.MinuteRefusals = Buckets{}
} }
} }
@@ -274,6 +303,10 @@ func (l *Limiter) AddToHistory(client netip.Prefix, now time.Time, r Request) {
if r.RuleBlocked { if r.RuleBlocked {
h.Offences.RuleBlocked++ h.Offences.RuleBlocked++
} }
if r.TokenRefused {
h.Offences.TokenRefused++
}
} }
// AddLookup gives client's history its AS number, AS name and country, as // AddLookup gives client's history its AS number, AS name and country, as
@@ -383,6 +416,10 @@ func (l *Limiter) Load(clients []Client, now time.Time) {
} }
} }
if c.MinuteRefusals.Passed(now, time.Minute) {
c.MinuteRefusals = Buckets{}
}
l.clients.Add(c.Client, &c) l.clients.Add(c.Client, &c)
} }
} }
+40
View File
@@ -248,6 +248,46 @@ func TestResetSetsTheBytesBackToZero(t *testing.T) {
wantBytesCount(t, limiter, client, start, 1000, "") wantBytesCount(t, limiter, client, start, 1000, "")
} }
func TestRefusalsOverTheThresholdInAMinuteBreakTheErrorBurst(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
for range limit {
if _, over := limiter.CountRefusal(client, start, limit); over {
t.Fatalf("a refusal within the threshold of %d broke the error burst", limit)
}
}
hit, over := limiter.CountRefusal(client, start, limit)
want := ratelimit.Hit{
Kind: ratelimit.KindRefusals, Window: minute, Limit: limit, Count: limit + 1,
}
if !over || hit != want {
t.Errorf("one over the threshold broke it: %t, with %+v; want %+v", over, hit,
want)
}
// Half a minute into the next, half of those four still count, 2, and
// this one: 3, within the threshold.
hit, over = limiter.CountRefusal(client, start.Add(time.Minute+time.Minute/2), limit)
if over || hit.Count != 3 {
t.Errorf("half a minute on, %v refusals broke it: %t; want 3, false",
hit.Count, over)
}
// A ban sets them back to zero.
limiter.Reset(client)
hit, _ = limiter.CountRefusal(client, start.Add(time.Minute+time.Minute/2), limit)
if hit.Count != 1 {
t.Errorf("after a reset, %v refusals, want 1", hit.Count)
}
}
func TestCountGivesTheRequestsInEachWindow(t *testing.T) { func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
t.Parallel() t.Parallel()
+13 -6
View File
@@ -65,6 +65,7 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
limiter := ratelimit.New(ratelimit.Limits{}, tableSize) limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
limiter.Count(client, start, whole) limiter.Count(client, start, whole)
limiter.CountBytes(client, start, 5, whole) limiter.CountBytes(client, start, 5, whole)
limiter.CountRefusal(client, start, limit)
limiter.AddToHistory(client, start, ratelimit.Request{Forwarded: true}) limiter.AddToHistory(client, start, ratelimit.Request{Forwarded: true})
loaded := func(now time.Time) ratelimit.Client { loaded := func(now time.Time) ratelimit.Client {
@@ -76,9 +77,9 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
return after.Snapshot()[0] return after.Snapshot()[0]
} }
// Two minutes on, the window that ends then covers neither of the // Two minutes on, the window that ends then covers none of the
// minute's buckets, of requests and of bytes, which are emptied; the // minute's buckets, of requests, of bytes and of refusals, which are
// hour's and the day's stay, and so does the history. // emptied; the hour's and the day's stay, and so does the history.
got := loaded(start.Add(2 * time.Minute)) got := loaded(start.Add(2 * time.Minute))
if got.Minute != (ratelimit.Buckets{}) || got.Hour.Current != 1 || if got.Minute != (ratelimit.Buckets{}) || got.Hour.Current != 1 ||
got.Day.Current != 1 || got.History.Requests != 1 { got.Day.Current != 1 || got.History.Requests != 1 {
@@ -91,11 +92,17 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
got.MinuteBytes, got.HourBytes, got.DayBytes) got.MinuteBytes, got.HourBytes, got.DayBytes)
} }
if got.MinuteRefusals != (ratelimit.Buckets{}) {
t.Errorf("loaded two minutes on with buckets of refusals %+v",
got.MinuteRefusals)
}
// A moment before, the window still covers some of the earlier one. // A moment before, the window still covers some of the earlier one.
got = loaded(start.Add(2*time.Minute - time.Nanosecond)) got = loaded(start.Add(2*time.Minute - time.Nanosecond))
if got.Minute.Current != 1 || got.MinuteBytes.Current != 5 { if got.Minute.Current != 1 || got.MinuteBytes.Current != 5 ||
t.Errorf("loaded just under two minutes on with minute buckets %+v and %+v", got.MinuteRefusals.Current != 1 {
got.Minute, got.MinuteBytes) t.Errorf("loaded just under two minutes on with minute buckets %+v, %+v "+
"and %+v", got.Minute, got.MinuteBytes, got.MinuteRefusals)
} }
} }
+11 -4
View File
@@ -29,8 +29,9 @@ const (
// over a rate limit, which bans the client. // over a rate limit, which bans the client.
ActionRateLimited = "rate_limited" ActionRateLimited = "rate_limited"
// ActionBanned is a request refused because a ban covers its client, // ActionBanned is a request refused because a ban covers its client,
// or because it matched a ban rule or the CrowdSec decision list lists // or because it matched a ban rule, asked for a trap path or the
// its client, either of which bans the client. // CrowdSec decision list lists its client, each of which bans the
// client.
ActionBanned = "banned" ActionBanned = "banned"
// ActionRuleBlocked is a request refused because it matched a block // ActionRuleBlocked is a request refused because it matched a block
// rule. // rule.
@@ -48,9 +49,14 @@ const (
) )
// OffenceLimit is the offence a request line names for a request that // OffenceLimit is the offence a request line names for a request that
// broke a rate limit, or whose bytes broke a byte limit. // broke a rate limit or the error burst, or whose bytes broke a byte
// limit.
const OffenceLimit = "limit" const OffenceLimit = "limit"
// LimitHitErrorBurst is the limit_hit a request line names for a request
// that broke the error burst.
const LimitHitErrorBurst = "error_burst"
// timeLayout is RFC 3339 with milliseconds. // timeLayout is RFC 3339 with milliseconds.
const timeLayout = "2006-01-02T15:04:05.000Z07:00" const timeLayout = "2006-01-02T15:04:05.000Z07:00"
@@ -138,7 +144,8 @@ type Line struct {
RuleIDs []string `json:"rule_ids,omitempty"` RuleIDs []string `json:"rule_ids,omitempty"`
// LimitHit is the window whose limit the request went over, named as // LimitHit is the window whose limit the request went over, named as
// Counts names its count: minute, hour or day for a rate limit, and // Counts names its count: minute, hour or day for a rate limit, and
// minute_bytes, hour_bytes or day_bytes for a byte limit. // minute_bytes, hour_bytes or day_bytes for a byte limit; or
// LimitHitErrorBurst for the error burst.
LimitHit string `json:"limit_hit,omitempty"` LimitHit string `json:"limit_hit,omitempty"`
// Reputation are the URLs of the blocklists that list the client, then // Reputation are the URLs of the blocklists that list the client, then
// that of the CrowdSec decision list when it does, then the DNSBL zones // that of the CrowdSec decision list when it does, then the DNSBL zones
+13 -6
View File
@@ -396,16 +396,23 @@ func (rule Rule) matches(r *http.Request) bool {
return rule.regex.MatchString(value(rule.Target, r)) return rule.regex.MatchString(value(rule.Target, r))
} }
// Path returns r's path as the client sent it, before any decoding or
// re-encoding, up to the first ?: what a path rule is matched against.
func Path(r *http.Request) string {
path, _, _ := strings.Cut(pathAndQuery(r), "?")
return path
}
// value returns what a rule with target, other than uri, is matched // value returns what a rule with target, other than uri, is matched
// against in r: the path and the query as the client sent them, before // against in r: the path, as Path gives it, and the query as the client
// any decoding or re-encoding, split at the first ?, and a header's values // sent it, before any decoding or re-encoding, after the first ?, and a
// joined by ", ", as HTTP joins those of a header sent more than once. // header's values joined by ", ", as HTTP joins those of a header sent
// more than once.
func value(target string, r *http.Request) string { func value(target string, r *http.Request) string {
switch target { switch target {
case "path": case "path":
path, _, _ := strings.Cut(pathAndQuery(r), "?") return Path(r)
return path
case "query": case "query":
_, query, _ := strings.Cut(pathAndQuery(r), "?") _, query, _ := strings.Cut(pathAndQuery(r), "?")
+6 -4
View File
@@ -679,8 +679,8 @@ func (f *bansFile) check(data []byte) error {
} }
// check refuses a client without its address, which would count nobody's // check refuses a client without its address, which would count nobody's
// requests, or with requests or bytes in a window but no start, which // requests, or with requests, bytes or refusals in a window but no start,
// would drop them and give the client a fresh allowance. // which would drop them and give the client a fresh allowance.
func (f *clientsFile) check([]byte) error { func (f *clientsFile) check([]byte) error {
for i, client := range f.Clients { for i, client := range f.Clients {
switch { switch {
@@ -698,6 +698,8 @@ func (f *clientsFile) check([]byte) error {
return missing(i, "hour_bytes.start") return missing(i, "hour_bytes.start")
case countsWithoutStart(client.DayBytes): case countsWithoutStart(client.DayBytes):
return missing(i, "day_bytes.start") return missing(i, "day_bytes.start")
case countsWithoutStart(client.MinuteRefusals):
return missing(i, "minute_refusals.start")
} }
} }
@@ -898,8 +900,8 @@ func missingFromCounter(counter anomaly.Counter) string {
} }
} }
// countsWithoutStart reports whether b holds requests, or bytes, but no // countsWithoutStart reports whether b holds requests, bytes or refusals
// start, which places them in time. // but no start, which places them in time.
func countsWithoutStart(b ratelimit.Buckets) bool { func countsWithoutStart(b ratelimit.Buckets) bool {
return b.Start.IsZero() && (b.Current != 0 || b.Previous != 0) return b.Start.IsZero() && (b.Current != 0 || b.Previous != 0)
} }
+9
View File
@@ -639,6 +639,15 @@ func TestEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
} }
} }
func TestClientWithRefusalsInTheMinuteWithoutTheirStartStopsTheStart(t *testing.T) {
t.Parallel()
wantRefused(t, clientsJSON,
`{"version": 1, "clients": [{"client": "203.0.113.9/32", `+
`"minute_refusals": {"current": 2}}]}`,
`: entry 1 has no "minute_refusals.start"`)
}
func TestReputationJSONEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) { func TestReputationJSONEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
t.Parallel() t.Parallel()