sneak/smallwebwaf
1
0
Fork 0
Code Issues 47 Pull Requests 1 Actions 29 Packages Projects Releases Wiki Activity

Trap paths, and the error burst banning a client refused too often #118

Merged
clawbot merged 1 commits from issue-115-trap-paths-error-burst into next 2026-10-08 06:44:56 +02:00
Conversation 1 Commits 1 Files Changed 21
+1208 -318
21 changed files with 1208 additions and 318 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+250 -187
View File
@@ -31,29 +31,32 @@ you name by URL, which it fetches and keeps, with a file of AS numbers'
percentages fetched the same way, the DNS blocklists (DNSBL zones), which it
asks about each client in the background, AbuseIPDB, which it asks in the
background about each client that has committed an offence, and the decision
list of a CrowdSec engine you run, which it fetches and keeps as a blocklist.
`smallwebwaf` passes each request to the app and the app's answer back,
unchanged, within its timeouts and size limits, works out each client's address,
looks up its AS number and country unless you switch that off, bans a client
that sends too many requests or too many bytes, not counting those for the paths
you choose, with lower limits for the clients of the AS numbers and countries
you list, refuses a client that comes from a country you refuse or from a
network you refuse, refuses, limits or only notes a client a blocklist or a
DNSBL zone you name lists, or AbuseIPDB scores at or over the score you set,
list of a CrowdSec engine you run, which it fetches and keeps as a blocklist. So
are two parts of the last stage, which need no Core Rule Set: the trap paths and
the error burst. `smallwebwaf` passes each request to the app and the app's
answer back, unchanged, within its timeouts and size limits, works out each
client's address, looks up its AS number and country unless you switch that off,
bans a client that sends too many requests or too many bytes, not counting those
for the paths you choose, with lower limits for the clients of the AS numbers
and countries you list, refuses a client that comes from a country you refuse or
from a network you refuse, refuses, limits or only notes a client a blocklist or
a DNSBL zone you name lists, or AbuseIPDB scores at or over the score you set,
bans a client your CrowdSec engine's decision list lists until that decision
ends, lets the networks you choose through, checks each request against the rule
files and bans a client whose request is a clear sign of attack, keeps its bans,
each client's counters and history, GeoJS's answers, the last good copy of each
list it fetches, the DNSBL zones' verdicts, AbuseIPDB's scores and the AbuseIPDB
checks spent today in JSON files across restarts, takes in your edits of those
files, such as a ban you make, keep or lift, and of the rule files while it
runs, writes a JSON log line for every request, sends its log lines to a syslog
server too if you name one, sends an alert to a webhook, to Slack and to ntfy,
each if you name one, for each ban it makes or makes permanent, for traffic over
an anomaly threshold you set, for a client a blocklist, the CrowdSec decision
list, a DNSBL zone or AbuseIPDB lists, for GeoJS failing, a list it cannot
fetch, a DNSBL zone or AbuseIPDB that fails or refuses a query and the day's
AbuseIPDB checks used up, for a rule file or state file with an error and for a
files and the trap paths you name and bans a client whose request is a clear
sign of attack, bans a client it refuses again and again within a minute for a
rule, a trap path or a missing or wrong token, keeps its bans, each client's
counters and history, GeoJS's answers, the last good copy of each list it
fetches, the DNSBL zones' verdicts, AbuseIPDB's scores and the AbuseIPDB checks
spent today in JSON files across restarts, takes in your edits of those files,
such as a ban you make, keep or lift, and of the rule files while it runs,
writes a JSON log line for every request, sends its log lines to a syslog server
too if you name one, sends an alert to a webhook, to Slack and to ntfy, each if
you name one, for each ban it makes or makes permanent, for traffic over an
anomaly threshold you set, for a client a blocklist, the CrowdSec decision list,
a DNSBL zone or AbuseIPDB lists, for GeoJS failing, a list it cannot fetch, a
DNSBL zone or AbuseIPDB that fails or refuses a query and the day's AbuseIPDB
checks used up, for a rule file or state file with an error and for a
replacement of the lookup database it cannot read, serves Prometheus metrics to
a scraper that holds the metrics token, lets an admin who holds the admin token
list, add and lift bans and ask what it knows of a client, and in `observe` mode
@@ -152,37 +155,47 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
The log line of each request the rate limits count gives its client's
percentages below 100 and the settings that gave them, and so do the notes of
a ban for a lowered limit, and its alert.
- Bans a client that breaks a rate limit or a byte limit, as "Bans" in
[`SPEC.md`](SPEC.md) describes: the first ban lasts an hour, and a limit
broken again within a day of a ban ending bans for three times as long as that
ban, so 1, 3, 9, 27 and 81 hours; a ban that would last longer than seven days
is permanent instead. A ban covers the client's netblock: its IPv4 address, or
the netblock around it that `SWWAF_BAN_SCOPE_V4_PREFIX` sets, or its IPv6
group. While it lasts, every request from the netblock is refused with
`SWWAF_BAN_RESPONSE` after the static lists and before the country lists, so
the client is not looked up, and is not counted for the rate limits. A ban
- Bans a client that breaks a rate limit, a byte limit or the error burst, as
"Bans" in [`SPEC.md`](SPEC.md) describes: the first ban lasts an hour, and a
limit broken again within a day of a ban ending bans for three times as long
as that ban, so 1, 3, 9, 27 and 81 hours; a ban that would last longer than
seven days is permanent instead. A ban covers the client's netblock: its IPv4
address, or the netblock around it that `SWWAF_BAN_SCOPE_V4_PREFIX` sets, or
its IPv6 group. While it lasts, every request from the netblock is refused
with `SWWAF_BAN_RESPONSE` after the static lists and before the country lists,
so the client is not looked up, and is not counted for the rate limits. A ban
sets the client's counters back to zero. Each ban carries notes for deciding
whether to lift it: the limit, whether it is on requests or bytes, its window
and the requests or bytes counted in it, the client's percentage of that kind
of limit and the setting that gave it when a biased threshold lowered the
limit, the request that broke it, the client's AS number, AS name and country
once they are looked up, the blocklists, the CrowdSec decision list, DNSBL
zones and AbuseIPDB, with its score, that listed the client when the ban was
made, the netblock's requests since it was first seen, how many of them the
ban has refused, and how many bans the netblock had before, for a broken
limit, for a clear sign of attack, by an admin and for CrowdSec's decision. At
most `SWWAF_MAX_BANS` bans `smallwebwaf` made are kept, past, active and
permanent; past that, the earliest such ban of the netblock that has gone
longest without a request is dropped first. The bans whose cause is `admin`,
those you make or keep, are kept besides, and never dropped. `bans.json` shows
the bans and their notes, a restart lifts none, and you make, keep or lift a
ban by editing it (see "State files" below).
whether to lift it: the limit, whether it is on requests, bytes or refusals,
its window and the requests, bytes or refusals counted in it, the client's
percentage of that kind of limit and the setting that gave it when a biased
threshold lowered the limit, the request that broke it, the client's AS
number, AS name and country once they are looked up, the blocklists, the
CrowdSec decision list, DNSBL zones and AbuseIPDB, with its score, that listed
the client when the ban was made, the netblock's requests since it was first
seen, how many of them the ban has refused, and how many bans the netblock had
before, for a broken limit, for a clear sign of attack, by an admin and for
CrowdSec's decision. At most `SWWAF_MAX_BANS` bans `smallwebwaf` made are
kept, past, active and permanent; past that, the earliest such ban of the
netblock that has gone longest without a request is dropped first. The bans
whose cause is `admin`, those you make or keep, are kept besides, and never
dropped. `bans.json` shows the bans and their notes, a restart lifts none, and
you make, keep or lift a ban by editing it (see "State files" below).
- Checks each request against the rules of the rule files (see "Rule files"
below) after the rate limits, and before its body is read. A `log` rule that
matches is noted in the log line; a `block` rule refuses the request with
`403`, and bans no one; a `ban` rule refuses it with `SWWAF_BAN_RESPONSE` and
bans the client's netblock for a clear sign of attack. Matching stops at the
first rule that refuses. A client in `SWWAF_ALLOW_NETS` is not checked.
`403`, and bans no one for it, though the refusal counts toward the error
burst; a `ban` rule refuses it with `SWWAF_BAN_RESPONSE` and bans the client's
netblock for a clear sign of attack. Matching stops at the first rule that
refuses. A client in `SWWAF_ALLOW_NETS` is not checked.
- Checks each request against the trap paths `SWWAF_TRAP_PATHS` names, after the
rate limits and before the rule files, which it does not need. A request for
one is refused with `SWWAF_BAN_RESPONSE` and bans the client's netblock for a
clear sign of attack, as a `ban` rule's match does. A trap path is matched
against the path a `path` rule sees: as the client sent it, before any
decoding and without the query, the whole of it, character for character.
`/wp-login.php` matches `/wp-login.php?redirect_to=x`, but not
`/wp-login.php/`, `/WP-LOGIN.PHP`, `/blog/wp-login.php` or `/%77p-login.php`.
A client in `SWWAF_ALLOW_NETS` is not checked.
- Bans a client for a clear sign of attack, as "Bans" in [`SPEC.md`](SPEC.md)
describes: the first such ban lasts `SWWAF_ATTACK_BAN_DURATION`, seven days by
default, and any request from the netblock while it lasts makes it permanent.
@@ -190,8 +203,22 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
sign of attack bans it permanently at once. Such a ban covers the same
netblock as a ban for a broken limit, does not set the client's counters back
to zero, and does not make the netblock's next ban for a broken limit longer.
Its notes give the id and the target of the rule that matched in place of the
limit.
Its notes give the id and the target of the rule that matched, or the trap
path asked for, in place of the limit.
- Bans a client that `smallwebwaf` refused more than
`SWWAF_ERROR_BURST_THRESHOLD` times within a minute, 30 by default, after a
match of a `block` or `ban` rule or a trap path, or for a missing or wrong
token at one of its own endpoints, as a broken rate limit bans it. Each such
refusal is counted once it has been answered, in two buckets of a minute, as
the rate limits count requests. The refusal that takes the client over the
threshold breaks the error burst; it is answered as any other such refusal is,
and the client's next request is refused under the ban. The app's own answers,
such as its `401` and `404`, are not counted. The threshold is the same for
every client, whatever percentage of the rate limits a biased threshold or a
reputation source gives it. A client in `SWWAF_ALLOW_NETS` is not counted, and
one in `SWWAF_RATE_LIMIT_EXEMPT_NETS` is. The ban's notes give `refusals` as
what the limit is on, the threshold as the limit, and the refusals counted in
the minute.
- Looks up the AS number and country of every client through GeoJS, or in the
IPinfo Lite database file while `SWWAF_LOOKUP_SOURCE` is `file`, after the
static lists and bans, unless `SWWAF_LOOKUP_SOURCE` is `off` (see "Country and
@@ -242,44 +269,50 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
- Checks the client with AbuseIPDB while `SWWAF_ABUSEIPDB_KEY` is set, after the
DNSBL zones and before the rate limits (see "AbuseIPDB" below), by the scores
it keeps. Only a client whose history counts an offence is checked, so far one
that has broken a rate limit or a byte limit, matched a ban rule, or had a
request refused by a block rule, and only in the background, so that no
request waits for AbuseIPDB. A score at or over `SWWAF_ABUSEIPDB_MIN_SCORE` is
a hit, and `SWWAF_REPUTATION_ACTION` does with its client what it does with
one a DNSBL zone's verdict lists. The request's log line names AbuseIPDB, and
it raises an alert. A client a blocklist or a DNSBL zone refuses, or the
CrowdSec decision list bans, is not checked.
that has broken a rate limit, a byte limit or the error burst, matched a ban
rule, asked for a trap path, or had a request refused by a block rule or for a
missing or wrong token, and only in the background, so that no request waits
for AbuseIPDB. A score at or over `SWWAF_ABUSEIPDB_MIN_SCORE` is a hit, and
`SWWAF_REPUTATION_ACTION` does with its client what it does with one a DNSBL
zone's verdict lists. The request's log line names AbuseIPDB, and it raises an
alert. A client a blocklist or a DNSBL zone refuses, or the CrowdSec decision
list bans, is not checked.
- Checks the client's own address against the static lists, the three netblock
settings below, before anything else, its lookup included. A client in
`SWWAF_ALLOW_NETS` skips bans, the country lists, the blocklists, the CrowdSec
decision list, the DNSBL zones, AbuseIPDB, the rate limits, the byte limits
and the rule files, and is not looked up; the timeouts and size limits still
apply. A client in `SWWAF_DENY_NETS` is refused with `SWWAF_BAN_RESPONSE`
before its body is read, and the request is not counted for the rate limits;
an address in `SWWAF_ALLOW_NETS` too is let through. A client in
`SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither counted nor refused by the rate
limits, and has no bytes counted by the byte limits; the country lists, the
rule files and bans still apply to it.
decision list, the DNSBL zones, AbuseIPDB, the rate limits, the byte limits,
the trap paths, the rule files and the error burst, and is not looked up; the
timeouts and size limits still apply. A client in `SWWAF_DENY_NETS` is refused
with `SWWAF_BAN_RESPONSE` before its body is read, and the request is not
counted for the rate limits; an address in `SWWAF_ALLOW_NETS` too is let
through. A client in `SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither counted nor
refused by the rate limits, and has no bytes counted by the byte limits; the
country lists, the trap paths, the rule files, the error burst and bans still
apply to it.
- In `observe` mode, with `SWWAF_MODE=observe`, refuses none of the requests
that `SWWAF_DENY_NETS`, a ban, the country lists, a blocklist, the CrowdSec
decision list, a DNSBL zone's verdict, AbuseIPDB's score, a rate limit or a
rule would refuse: it passes them to the app, and their log lines name what
`enforce` mode would have done (see `would_action` in "Request log" below).
The checks run, and requests and bytes are counted, as in `enforce` mode, with
three differences: neither a broken rate limit or byte limit, a `ban` rule nor
the CrowdSec decision list makes a ban; a broken limit does not set the
client's counters back to zero, so each request over a rate limit is logged as
one that would be refused, and each whose bytes keep the client over a byte
limit as breaking it; and a request under a ban does not make it permanent. As
in `enforce` mode, the bytes counted are only those of the requests `enforce`
mode would have passed to the app. A ban it would have made, or made
permanent, raises the alert `enforce` mode would have raised, marked as what
would have happened (see "Alerts" below). The bans in `bans.json` are kept,
and refuse requests again when `smallwebwaf` next runs in `enforce` mode, as
long as they last. The timeouts and size limits still apply, since they
protect `smallwebwaf` and the app themselves, and a request for one of
`smallwebwaf`'s own endpoints without its token is still answered `401`. It is
for trying a configuration before enforcing it.
decision list, a DNSBL zone's verdict, AbuseIPDB's score, a rate limit, a trap
path or a rule would refuse: it passes them to the app, and their log lines
name what `enforce` mode would have done (see `would_action` in "Request log"
below). The checks run, and requests, bytes and refusals are counted, as in
`enforce` mode, with three differences: neither a broken rate limit, byte
limit or error burst, a `ban` rule, a trap path nor the CrowdSec decision list
makes a ban; a broken limit does not set the client's counters back to zero,
so each request over a rate limit is logged as one that would be refused, each
whose bytes keep the client over a byte limit as breaking it, and each refusal
that keeps it over the error burst as breaking that; and a request under a ban
does not make it permanent. As in `enforce` mode, the bytes counted are only
those of the requests `enforce` mode would have passed to the app, and the
refusals counted for the error burst only those it would have made: a request
it would have refused before it reached an endpoint is not counted for its
token. A ban it would have made, or made permanent, raises the alert `enforce`
mode would have raised, marked as what would have happened (see "Alerts"
below). The bans in `bans.json` are kept, and refuse requests again when
`smallwebwaf` next runs in `enforce` mode, as long as they last. The timeouts
and size limits still apply, since they protect `smallwebwaf` and the app
themselves, and a request for one of `smallwebwaf`'s own endpoints without its
token is still answered `401`. It is for trying a configuration before
enforcing it.
- Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any
check and without asking the app, for the image's health check.
- Answers `GET /_smallwebwaf/metrics` with its metrics (see "Metrics" below) for
@@ -385,8 +418,8 @@ effective settings are logged at start, unless `SWWAF_LOG_LEVEL` is `warn` or
- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body.
- `SWWAF_ALLOW_NETS` (default empty): netblocks whose clients skip bans, the
country lists, the blocklists, the CrowdSec decision list, the DNSBL zones,
AbuseIPDB, the rate limits, the byte limits and the rule files, such as your
monitoring or your own networks.
AbuseIPDB, the rate limits, the byte limits, the trap paths, the rule files
and the error burst, such as your monitoring or your own networks.
- `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients the
rate limits and the byte limits do not apply to, such as a machine that talks
to the app all day.
@@ -530,21 +563,21 @@ effective settings are logged at start, unless `SWWAF_LOG_LEVEL` is `warn` or
- `SWWAF_REPUTATION_TIMEOUT` (default `2s`): how long a query to a zone, or a
check with AbuseIPDB, may take before it fails.
- `SWWAF_BAN_RESPONSE` (default `403`): how a refused client is answered, one
that is banned, breaks a rate limit, matches a `ban` rule, is in
`SWWAF_DENY_NETS`, comes from a refused country, is in a blocklist while
`SWWAF_BLOCKLIST_ACTION` is `deny` or is listed by a DNSBL zone or AbuseIPDB
while `SWWAF_REPUTATION_ACTION` is `deny`: `403`, `429`, or `close` to close
the connection without an answer. Behind traefik, `close` does not leave the
client unanswered: traefik answers `502`, as it does whenever its backend
drops a connection. A `block` rule always answers `403`.
that is banned, breaks a rate limit, matches a `ban` rule, asks for a trap
path, is in `SWWAF_DENY_NETS`, comes from a refused country, is in a blocklist
while `SWWAF_BLOCKLIST_ACTION` is `deny` or is listed by a DNSBL zone or
AbuseIPDB while `SWWAF_REPUTATION_ACTION` is `deny`: `403`, `429`, or `close`
to close the connection without an answer. Behind traefik, `close` does not
leave the client unanswered: traefik answers `502`, as it does whenever its
backend drops a connection. A `block` rule always answers `403`.
- `SWWAF_LIMIT_BAN_DURATION` (default `1h`): the ban for a first broken rate
limit or byte limit.
- `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): a rate limit or byte limit
broken again within this time after a ban ended, other than one for a clear
sign of attack or for CrowdSec's decision, bans for three times as long as
that ban.
- `SWWAF_MAX_BAN_DURATION` (default `7d`): a ban for a broken rate limit or byte
limit that would be longer is permanent instead.
limit, byte limit or error burst.
- `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): a rate limit, byte limit or
error burst broken again within this time after a ban ended, other than one
for a clear sign of attack or for CrowdSec's decision, bans for three times as
long as that ban.
- `SWWAF_MAX_BAN_DURATION` (default `7d`): a ban for a broken rate limit, byte
limit or error burst that would be longer is permanent instead.
- `SWWAF_ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of
attack.
- `SWWAF_MAX_BANS` (default `5000`): the most bans `smallwebwaf` made that are
@@ -588,6 +621,18 @@ effective settings are logged at start, unless `SWWAF_LOG_LEVEL` is `warn` or
rule files. A directory that does not exist stops the start.
- `SWWAF_RULES_ENABLED` (default `true`): `false` reads no rule file, and checks
no request against one.
- `SWWAF_TRAP_PATHS` (default empty): paths the app never serves and only
scanners ask for, such as `/wp-login.php,/xmlrpc.php` in front of gitea; a
request for one bans its client for a clear sign of attack, as a `ban` rule
does, with or without rule files (see "What it does so far" above). A path
that does not start with `/`, or holds a `?`, which no path a `path` rule sees
holds, stops the start.
- `SWWAF_ERROR_BURST_THRESHOLD` (default `30`): the most requests of a client in
a minute that `smallwebwaf` may refuse after a rule file match or a trap path,
or for a missing or wrong token; one more breaks the error burst, and bans the
client as a broken rate limit does (see "What it does so far" above). A client
trying one probe or token after another is refused many times a minute, a
person rarely more than a few times. `off` switches it off.
- `SWWAF_LOG_REMOTE_URL` (default unset): a syslog server that every line on
stdout is also sent to, as `syslog+udp://`, `syslog+tcp://` or `syslog+tls://`
with a host and a port, such as `syslog+tls://logs.example:6514`. Unset or
@@ -671,18 +716,19 @@ effective settings are logged at start, unless `SWWAF_LOG_LEVEL` is `warn` or
Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are
bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024
bytes). Rate limits and the anomaly thresholds on requests are whole numbers of
requests, and byte limits and the anomaly thresholds on bytes are sizes.
Netblocks are in CIDR form, and a bare address stands for itself alone.
Countries are the two-letter codes ISO 3166-1 assigns today, and `xk` for
Kosovo, in either case (`de` and `DE` are the same); any other code, such as
`nk` (North Korea is `kp`) or the withdrawn `su`, stops the start, and so does a
code on both country lists. AS numbers are `AS` and the number, in either case.
Percentages are whole numbers from 0 to 100, and an entry of a list of them is
an AS number or a country, `:` and a percentage; an AS number or a country
listed twice in one of them stops the start. `off` switches a timeout, a size
limit, a rate limit, a byte limit, an anomaly threshold, `SWWAF_ALERT_COOLDOWN`
or `SWWAF_ALERT_MAX_PER_HOUR` off; `SWWAF_IPV6_GROUP_PREFIX`,
bytes). Rate limits, `SWWAF_ERROR_BURST_THRESHOLD` and the anomaly thresholds on
requests are whole numbers of requests, and byte limits and the anomaly
thresholds on bytes are sizes. Netblocks are in CIDR form, and a bare address
stands for itself alone. Countries are the two-letter codes ISO 3166-1 assigns
today, and `xk` for Kosovo, in either case (`de` and `DE` are the same); any
other code, such as `nk` (North Korea is `kp`) or the withdrawn `su`, stops the
start, and so does a code on both country lists. AS numbers are `AS` and the
number, in either case. Percentages are whole numbers from 0 to 100, and an
entry of a list of them is an AS number or a country, `:` and a percentage; an
AS number or a country listed twice in one of them stops the start. `off`
switches a timeout, a size limit, a rate limit, a byte limit, an anomaly
threshold, `SWWAF_ERROR_BURST_THRESHOLD`, `SWWAF_ALERT_COOLDOWN` or
`SWWAF_ALERT_MAX_PER_HOUR` off; `SWWAF_IPV6_GROUP_PREFIX`,
`SWWAF_MAX_TRACKED_CLIENTS`, `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`,
`SWWAF_LOOKUP_TIMEOUT`, `SWWAF_UNKNOWN_LIMIT_PERCENT`,
`SWWAF_BLOCKLIST_REFRESH`, `SWWAF_ABUSEIPDB_MIN_SCORE`,
@@ -781,20 +827,20 @@ which every line has.
refused because its client is in `SWWAF_DENY_NETS`, in a blocklist while
`SWWAF_BLOCKLIST_ACTION` is `deny`, or listed by a DNSBL zone or AbuseIPDB
while `SWWAF_REPUTATION_ACTION` is `deny`, `banned` for one refused because a
ban covers its client, or because it matched a `ban` rule or the CrowdSec
decision list lists its client, either of which bans its client,
`country_denied` for one refused for its client's country, `rate_limited` for
one that broke a rate limit and banned its client, `rule_blocked` for one a
`block` rule refused, `too_large` for a request or response over its size
limit, `timed_out` for one that ran out of time, `upstream_error` when the app
could not be reached or its answer broke off, and `admin` for one
`smallwebwaf` answered at its own endpoint.
ban covers its client, or because it matched a `ban` rule, asked for a trap
path or the CrowdSec decision list lists its client, each of which bans its
client, `country_denied` for one refused for its client's country,
`rate_limited` for one that broke a rate limit and banned its client,
`rule_blocked` for one a `block` rule refused, `too_large` for a request or
response over its size limit, `timed_out` for one that ran out of time,
`upstream_error` when the app could not be reached or its answer broke off,
and `admin` for one `smallwebwaf` answered at its own endpoint.
- `would_action` is there in `observe` mode for a request that
`SWWAF_DENY_NETS`, a ban, the country lists, a blocklist, the CrowdSec
decision list, a DNSBL zone's verdict, AbuseIPDB's score, a rate limit or a
rule would have refused in `enforce` mode, and names the action that refusal
would have had: `denied`, `banned`, `country_denied`, `rate_limited` or
`rule_blocked`. `action` then names what was done: `forward` for a request
decision list, a DNSBL zone's verdict, AbuseIPDB's score, a rate limit, a trap
path or a rule would have refused in `enforce` mode, and names the action that
refusal would have had: `denied`, `banned`, `country_denied`, `rate_limited`
or `rule_blocked`. `action` then names what was done: `forward` for a request
passed to the app, and another action, such as `too_large`, for one a size or
time limit refused.
- `limit_percent` is there for a request the rate limits count whose client a
@@ -822,13 +868,15 @@ which every line has.
counted before it.
- `rule_ids` is there for a request that matched rules of the rule files, and
lists their ids in the order they matched, up to the one that refused it.
- `limit_hit` is there for a request that broke a rate limit, or whose bytes
broke a byte limit, and names the window whose limit it went over as `counts`
names it: `minute`, `hour` or `day` for a rate limit, and `minute_bytes`,
`hour_bytes` or `day_bytes` for a byte limit, the shortest if it went over
several. `offence` is then `limit`. A request whose bytes broke a byte limit
is not refused: its `action` is what it would have been otherwise, such as
`forward`.
- `limit_hit` is there for a request that broke a rate limit or the error burst,
or whose bytes broke a byte limit, and names the window whose limit it went
over as `counts` names it: `minute`, `hour` or `day` for a rate limit, and
`minute_bytes`, `hour_bytes` or `day_bytes` for a byte limit, the shortest if
it went over several; or `error_burst` for the error burst. `offence` is then
`limit`. A request whose bytes broke a byte limit is not refused: its `action`
is what it would have been otherwise, such as `forward`. Nor is one that broke
the error burst refused for that: its `action` is that of the refusal counted,
such as `rule_blocked` or `admin`.
- `reputation` is there for a request whose client a blocklist, the CrowdSec
decision list or a DNSBL zone's verdict lists, or whose AbuseIPDB score is a
hit, and gives the URLs of the blocklists that list it, in the order
@@ -1101,33 +1149,42 @@ which are listed by scope first, and the copies of the lists, listed by URL,
with times in UTC.
- `bans.json`: every ban with its notes, indented to be read. A permanent ban's
`expires` is `null`. A ban's `cause` is `limit` for a broken rate limit or
byte limit, `attack` for a clear sign of attack or `crowdsec` for a client the
CrowdSec decision list lists, for a ban `smallwebwaf` made, and `admin` for
one you made or keep. Its `reason` is a short text: for a ban `smallwebwaf`
made, the limit broken, such as `requests per minute over the limit of 1000`
or `bytes per hour over the limit of 21474836480`, the rule that matched, such
as `matched the rule env-file`, or the scenario that made CrowdSec's decision,
such as `CrowdSec's decision for crowdsecurity/ssh-bf`; for yours, what you
wrote. Its `lifted` is when you lifted it, and is left out until you do. The
`kind` in the notes of a ban for a broken limit is `requests` or `bytes`, what
the limit is on. For a limit a biased threshold lowered, the reason and the
notes' `limit` give the lowered limit, and the notes' `limit_percent` and
`limit_percent_setting` the client's percentage of that kind of limit and the
setting that gave it. The notes' `reputation` gives each blocklist, the
CrowdSec decision list, each DNSBL zone or AbuseIPDB that listed the client
when the ban was made, as its `source`, named and ordered as in the request
log's `reputation`, with AbuseIPDB's `score` of the client. It is left out
when none did, and the example below shows it.
`expires` is `null`. A ban's `cause` is `limit` for a broken rate limit, byte
limit or error burst, `attack` for a clear sign of attack or `crowdsec` for a
client the CrowdSec decision list lists, for a ban `smallwebwaf` made, and
`admin` for one you made or keep. Its `reason` is a short text: for a ban
`smallwebwaf` made, the limit broken, such as
`requests per minute over the limit of 1000`,
`bytes per hour over the limit of 21474836480` or
`refusals per minute over the limit of 30`, the rule that matched, such as
`matched the rule env-file`, the trap path asked for, such as
`asked for the trap path /wp-login.php`, or the scenario that made CrowdSec's
decision, such as `CrowdSec's decision for crowdsecurity/ssh-bf`; for yours,
what you wrote. Its `lifted` is when you lifted it, and is left out until you
do. The `kind` in the notes of a ban for a broken limit is `requests`, `bytes`
or `refusals`, for the error burst, what the limit is on. The notes of a ban
for a clear sign of attack give the `rule_id` and the `target` of the rule
that matched, or the `trap_path` asked for. For a limit a biased threshold
lowered, the reason and the notes' `limit` give the lowered limit, and the
notes' `limit_percent` and `limit_percent_setting` the client's percentage of
that kind of limit and the setting that gave it. The notes' `reputation` gives
each blocklist, the CrowdSec decision list, each DNSBL zone or AbuseIPDB that
listed the client when the ban was made, as its `source`, named and ordered as
in the request log's `reputation`, with AbuseIPDB's `score` of the client. It
is left out when none did, and the example below shows it.
- `clients.json`: each client's two buckets of requests in the minute, the hour
and the day, its two buckets of bytes in each, `minute_bytes`, `hour_bytes`
and `day_bytes`, and its history: when it was first and last seen, its AS
number, AS name and country as last looked up and when the lookup gave them,
its requests, how many were forwarded and how many refused (one `smallwebwaf`
answered at its own endpoints is neither, unless it was refused with `401` for
a missing or wrong token), the body bytes in each direction, its responses by
status class and its offences by kind. Each client is on a line of its own, so
`grep` shows everything about one.
and `day_bytes`, its two buckets of refusals in the minute, which the error
burst counts, `minute_refusals`, and its history: when it was first and last
seen, its AS number, AS name and country as last looked up and when the lookup
gave them, its requests, how many were forwarded and how many refused (one
`smallwebwaf` answered at its own endpoints is neither, unless it was refused
with `401` for a missing or wrong token), the body bytes in each direction,
its responses by status class and its offences by kind: `limit` for a broken
rate limit, byte limit or error burst, `attack` for a clear sign of attack,
`rule_blocked` for a request a `block` rule refused and `token_refused` for
one refused for a missing or wrong token. Each client is on a line of its own,
so `grep` shows everything about one.
- `lookups.json`: GeoJS's answers, one to a line, each with the client's AS
number, AS name and country, when GeoJS gave it and when it was last used.
- `reputation.json`: each list fetched from a URL (see "Blocklists" and
@@ -1243,19 +1300,19 @@ and the line and column where Go's JSON decoder gives them; so does a state
directory `smallwebwaf` cannot write. So does an entry without a field it needs,
named with the entry's place in the file: a ban's `netblock`, `start` or
`expires`, which is `null` for a permanent ban; a client's `client`, or the
`start` of a window in which it has requests or bytes; an answer's `client`,
`country`, which is `""` for a client GeoJS cannot place, or `answered`; a
cooldown's `event` or `sent`; an alert waiting's `event` or `time`; an anomaly
counter's `netblock`, unless it counts an AS number or the whole service, its
`asn`, for an AS number, its `name`, for a named netblock, or the `start` of a
window in which it has requests or bytes; a list's `url`, `fetched` or `lines`,
which is `[]` for an empty list; a verdict's `zone`, `client`, `listed`, which
is `false` for a client the zone does not list, or `fetched`. So does a ban
whose `cause` is not `limit`, `attack`, `admin` or `crowdsec`, alerts waiting
for a destination that is not `webhook`, `slack` or `ntfy`, an anomaly counter
whose `scope` is not `client`, `net`, `asn`, `total` or `watch`, and a copy of a
list with a line that would make its fetch fail. An answer's `asn` or `as_name`
left out reads as empty.
`start` of a window in which it has requests, bytes or refusals; an answer's
`client`, `country`, which is `""` for a client GeoJS cannot place, or
`answered`; a cooldown's `event` or `sent`; an alert waiting's `event` or
`time`; an anomaly counter's `netblock`, unless it counts an AS number or the
whole service, its `asn`, for an AS number, its `name`, for a named netblock, or
the `start` of a window in which it has requests or bytes; a list's `url`,
`fetched` or `lines`, which is `[]` for an empty list; a verdict's `zone`,
`client`, `listed`, which is `false` for a client the zone does not list, or
`fetched`. So does a ban whose `cause` is not `limit`, `attack`, `admin` or
`crowdsec`, alerts waiting for a destination that is not `webhook`, `slack` or
`ntfy`, an anomaly counter whose `scope` is not `client`, `net`, `asn`, `total`
or `watch`, and a copy of a list with a line that would make its fetch fail. An
answer's `asn` or `as_name` left out reads as empty.
While it runs, `smallwebwaf` watches `SWWAF_STATE_DIR` and takes in your edit of
a state file as soon as you save it: what the file then holds replaces what
@@ -1406,9 +1463,11 @@ scraped, and keeps this one as `exported_instance` unless the scrape sets
from then on, as histograms; `smallwebwaf_requests_in_flight`: the requests
under way.
- `smallwebwaf_rate_limit_hits_total` by `window`, `minute`, `hour` or `day`,
and `kind`, `requests` for a rate limit or `bytes` for a byte limit,
and `kind`, `requests` for a rate limit, `bytes` for a byte limit or
`refusals` for the error burst, whose window is `minute`,
`smallwebwaf_size_and_time_limit_hits_total` by `limit`, the setting whose
limit was passed, `smallwebwaf_offences_total` by `kind`, and
limit was passed, `smallwebwaf_offences_total` by `kind`, `limit`, `attack`,
`rule_blocked` or `token_refused`, as `clients.json` counts them, and
`smallwebwaf_bans_made_total` by `cause`, `limit`, `attack`, `admin` or
`crowdsec`, `admin` for the bans you add through `POST /_smallwebwaf/bans`,
and those whose `cause` is `admin` that you add to `bans.json` while
@@ -1521,7 +1580,8 @@ or lift is written to `bans.json` `SWWAF_STATE_WRITE_DELAY` later. Refusals, and
the answers to requests that cannot be read, are plain text.
A request without the token, or with another, such as the metrics token, is
answered `401`, in `observe` mode too. While the token is unset, each of these
answered `401`, in `observe` mode too, and counts toward the error burst, as one
for the metrics without theirs does. While the token is unset, each of these
answers `404`, as does any request under `/_smallwebwaf/` that is not for one of
its endpoints. Like the metrics, these requests go through every check any other
request goes through, and are answered where another would be passed to the app:
@@ -1679,13 +1739,14 @@ For each request `smallwebwaf`:
- picks the client's limit percentage from those;
- checks the minute, hour and day request counters against the limits, and bans
the client if it breaks one;
- checks the request against the rule files and the Core Rule Set, and bans the
client at once for a clear sign of attack;
- checks the request against the trap paths, the rule files and the Core Rule
Set, and bans the client at once for a clear sign of attack;
- forwards it to the app and streams the response back, within the size and time
limits;
- counts the bytes and any refusal by the rule files or the Core Rule Set, bans
the client if it broke a limit, updates its history and the anomaly counters,
sends any alerts that are due, and writes the log line.
- counts the bytes and any refusal by the trap paths, the rule files or the Core
Rule Set or for a missing or wrong token, bans the client if it broke a limit,
updates its history and the anomaly counters, sends any alerts that are due,
and writes the log line.
A minimal deployment is the app's own Dockerfile, built on the `smallwebwaf`
image, with no setting. That image is built on Ubuntu 26.04 LTS, the newest
@@ -2054,14 +2115,16 @@ alerts, the metrics nor `reputation.json` hold it. Given as a file, with
request is refused before anything reaches the app: for `SWWAF_DENY_NETS`, for
a ban, for the country lists, for a blocklist, for the CrowdSec decision list,
which bans the client, for a DNSBL zone's verdict, for AbuseIPDB's score, for
a rate limit, which bans the client, for a `block` or `ban` rule, the latter
banning the client, and for an announced body over the size limit; in
`observe` mode, only for the size limit, with what it would have refused for
noted in the log line. A request under `/_smallwebwaf/` that `check` lets
through is answered by `answerAdmin` instead of reaching the app. Once the
answer to a request passed to the app has ended, `countBytes` counts its bytes
for the byte limits, and once any request but the health check has ended,
`countAnomalies` counts it for the anomaly thresholds.
a rate limit, which bans the client, for a trap path, which bans the client,
for a `block` or `ban` rule, the latter banning the client, and for an
announced body over the size limit; in `observe` mode, only for the size
limit, with what it would have refused for noted in the log line. A request
under `/_smallwebwaf/` that `check` lets through is answered by `answerAdmin`
instead of reaching the app. Once the answer to a request passed to the app
has ended, `countBytes` counts its bytes for the byte limits, and once any
request but the health check has ended, `countRefusal` counts it for the error
burst if it was refused after a rule file match or a trap path or for its
token, and `countAnomalies` counts it for the anomaly thresholds.
- `internal/metrics`: the metrics, counted as the other parts tell it what
happened, and served in the Prometheus text format.
- `internal/bans`: the ban ledger: each netblock's bans with their notes, how
@@ -2083,9 +2146,9 @@ alerts, the metrics nor `reputation.json` hold it. Given as a file, with
tells which zones' verdicts list an address; and checks clients with AbuseIPDB
in the background, keeps their scores and the checks spent today, and tells
whether a client's score is a hit.
- `internal/ratelimit`: the table of clients: counts each client's requests and
bytes, tells when they take it over a rate limit or a byte limit, and keeps
each client's history.
- `internal/ratelimit`: the table of clients: counts each client's requests,
bytes and refusals, tells when they take it over a rate limit, a byte limit or
the error burst, and keeps each client's history.
- `internal/anomaly`: the anomaly counters: counts each request and its bytes
per client, per netblock around a client, per AS number, for the whole service
and per named netblock, in the buckets `internal/ratelimit` counts in, and
internal/bans/bans.go
+18 -10
View File
@@ -1,6 +1,7 @@
// Package bans is the ban ledger: the bans smallwebwaf makes on the
// netblocks of clients that break a rate limit or a byte limit, show a
// clear sign of attack or are listed by the CrowdSec decision list, and
// netblocks of clients that break a rate limit, a byte limit or the error
// burst, show a clear sign of attack or are listed by the CrowdSec
// decision list, and
// those an admin makes, with their notes, as the "Bans" section of SPEC.md
// describes. The bans are kept in memory, and written to bans.json and
// read from it by the state package.
@@ -103,10 +104,11 @@ type Notes struct {
ASName string `json:"as_name"`
Country string `json:"country"`
// Kind, Limit, Window and Count are, for a ban for a broken limit,
// what the limit was on, "requests" for a rate limit or "bytes" for a
// byte limit, the limit that was broken, its window, "minute", "hour"
// or "day", and the count reached: the client's requests, or bytes, in
// the window, those of the request that broke the limit included.
// what the limit was on, "requests" for a rate limit, "bytes" for a
// byte limit or "refusals" for the error burst, the limit that was
// broken, its window, "minute", "hour" or "day", and the count reached:
// the client's requests, bytes or refusals in the window, those of the
// request that broke the limit included.
// These are what counted toward the ban, and the window is the time
// over which they came.
Kind string `json:"kind,omitempty"`
@@ -120,9 +122,11 @@ type Notes struct {
LimitPercent *int64 `json:"limit_percent,omitempty"`
LimitPercentSetting string `json:"limit_percent_setting,omitempty"`
// RuleID and Target are, for a ban for a clear sign of attack, the id
// of the rule file rule that matched, and its target.
RuleID string `json:"rule_id,omitempty"`
Target string `json:"target,omitempty"`
// of the rule file rule that matched, and its target; TrapPath is, for
// one for a request for a path in SWWAF_TRAP_PATHS, that path.
RuleID string `json:"rule_id,omitempty"`
Target string `json:"target,omitempty"`
TrapPath string `json:"trap_path,omitempty"`
// Reputation is the reputation sources that listed the client when
// the request that caused the ban was made, in the order the request
// log's reputation names them. It is left out when none did.
@@ -311,7 +315,7 @@ func (l *Ledger) WouldBanForLimit(
// notes, and returns the ban, and whether it made it, as BanForLimit
// does. A first ban lasts AttackBanDuration; once the netblock has had
// one that was not lifted, the next is permanent. Its reason is "matched
// the rule <RuleID>".
// the rule <RuleID>", or "asked for the trap path <TrapPath>".
func (l *Ledger) BanForAttack(
netblock netip.Prefix, now time.Time, notes Notes,
) (Ban, bool) {
@@ -382,6 +386,10 @@ func limitReason(notes Notes) string {
// attackReason is the reason of a ban for a clear sign of attack, with
// notes.
func attackReason(notes Notes) string {
if notes.TrapPath != "" {
return "asked for the trap path " + notes.TrapPath
}
return "matched the rule " + notes.RuleID
}
internal/config/config.go
+39
View File
@@ -239,6 +239,14 @@ type Config struct {
// unless RulesEnabled is false (SWWAF_RULES_ENABLED).
RulesDir string
RulesEnabled bool
// TrapPaths are the paths a request for which is a clear sign of
// attack (SWWAF_TRAP_PATHS), each starting with / and without a ?.
TrapPaths []string
// ErrorBurstThreshold is the most requests of a client within a minute
// that smallwebwaf may refuse after a rule file match or for a missing
// or wrong token; one more breaks a limit
// (SWWAF_ERROR_BURST_THRESHOLD). 0 is off.
ErrorBurstThreshold int64
// LogRemoteURL is where every line on stdout is also sent
// (SWWAF_LOG_REMOTE_URL), nil while it is unset and nothing is sent.
// LogRemoteTLSCAs are the certificates a syslog+tls endpoint's
@@ -378,6 +386,8 @@ var (
errNotBytesCount = errors.New("is not response, request or both")
errNotPathPrefix = errors.New(
"is not a path prefix starting with /, such as /assets/")
errNotTrapPath = errors.New(
"is not a path starting with / and without a ?, such as /wp-login.php")
errNotBoolean = errors.New("is not true or false")
errNotLogRemoteURL = errors.New(
"is not syslog+udp, syslog+tcp or syslog+tls with a host and a port, " +
@@ -505,6 +515,8 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
MetricsTopN: env.numberNotOff("SWWAF_METRICS_TOP_N", "50"),
RulesDir: env.value("SWWAF_RULES_DIR", "/etc/smallwebwaf/rules.d"),
RulesEnabled: env.boolean("SWWAF_RULES_ENABLED", "true"),
TrapPaths: env.trapPaths("SWWAF_TRAP_PATHS"),
ErrorBurstThreshold: env.count("SWWAF_ERROR_BURST_THRESHOLD", "30"),
LogRemoteURL: env.logRemoteURL("SWWAF_LOG_REMOTE_URL"),
LogRemoteTLSCAs: env.certificates("SWWAF_LOG_REMOTE_TLS_CA_FILE"),
LogRemoteBuffer: env.numberNotOff("SWWAF_LOG_REMOTE_BUFFER", "10000"),
@@ -744,6 +756,15 @@ func (e *environment) pathPrefixes(name, defaultValue string) []string {
return prefixes
}
// trapPaths reads the setting that is the list of trap paths. It is empty
// by default.
func (e *environment) trapPaths(name string) []string {
paths, err := parseTrapPaths(e.value(name, ""))
e.check(name, err)
return paths
}
// countries reads a setting that is a list of countries.
func (e *environment) countries(name, defaultValue string) []string {
countries, err := parseCountries(e.value(name, defaultValue))
@@ -1536,6 +1557,24 @@ func parsePathPrefixes(value string) ([]string, error) {
return prefixes, nil
}
// parseTrapPaths reads a comma-separated list of trap paths. Each is
// matched against a request's path as a path rule is, without the query,
// so a path that does not start with / or holds a ? would never match.
func parseTrapPaths(value string) ([]string, error) {
paths, err := parseList(value)
if err != nil {
return nil, err
}
for _, path := range paths {
if !strings.HasPrefix(path, "/") || strings.Contains(path, "?") {
return nil, fmt.Errorf("%q %w", path, errNotTrapPath)
}
}
return paths, nil
}
// countryCodes are the two-letter codes ISO 3166-1 assigns today, and XK,
// the code in common use for Kosovo. golang.org/x/text/language cannot
// check them: it also takes withdrawn codes such as su, and reserved ones
internal/config/config_test.go
+60
View File
@@ -91,6 +91,8 @@ const (
logLevel = "SWWAF_LOG_LEVEL"
rulesDir = "SWWAF_RULES_DIR"
rulesEnabled = "SWWAF_RULES_ENABLED"
trapPaths = "SWWAF_TRAP_PATHS"
errorBurstThreshold = "SWWAF_ERROR_BURST_THRESHOLD"
logRemoteURL = "SWWAF_LOG_REMOTE_URL"
logRemoteTLSCAFile = "SWWAF_LOG_REMOTE_TLS_CA_FILE"
logRemoteBuffer = "SWWAF_LOG_REMOTE_BUFFER"
@@ -495,6 +497,62 @@ func TestPathPrefixNotStartingWithSlashStopsTheStart(t *testing.T) {
}
}
func TestTrapPathsAndErrorBurstThreshold(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
env environment
paths []string
threshold int64
}{
{environment{}, []string{}, 30},
{
environment{trapPaths: "/wp-login.php, /xmlrpc.php", errorBurstThreshold: "5"},
[]string{"/wp-login.php", "/xmlrpc.php"}, 5,
},
{environment{errorBurstThreshold: off}, []string{}, 0},
} {
cfg := fromEnvironment(t, tc.env)
if !slices.Equal(cfg.TrapPaths, tc.paths) ||
cfg.ErrorBurstThreshold != tc.threshold {
t.Errorf("%v gave %v and %d, want %v and %d", tc.env, cfg.TrapPaths,
cfg.ErrorBurstThreshold, tc.paths, tc.threshold)
}
}
}
func TestInvalidTrapPathOrErrorBurstThresholdStopsTheStart(t *testing.T) {
t.Parallel()
const notTrapPath = " is not a path starting with / and without a ?, " +
"such as /wp-login.php"
for _, tc := range []struct{ name, value, want string }{
{trapPaths, "/wp-login.php,xmlrpc.php", `"xmlrpc.php"` + notTrapPath},
{trapPaths, "/xmlrpc.php?rsd", `"/xmlrpc.php?rsd"` + notTrapPath},
{
trapPaths, "/wp-login.php,,/xmlrpc.php",
`"/wp-login.php,,/xmlrpc.php" has an empty item in its list`,
},
{errorBurstThreshold, "0", `"0" must be more than zero, or off`},
{
errorBurstThreshold, "30/min",
`"30/min" is not a whole number of requests such as 1000, or off`,
},
} {
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
t.Parallel()
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
want := tc.name + ": " + tc.want
if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want)
}
})
}
}
func TestInstanceNameAndLoggedHeadersAsSet(t *testing.T) {
t.Parallel()
@@ -2233,6 +2291,8 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
logLevel: "info",
rulesDir: "/etc/smallwebwaf/rules.d",
rulesEnabled: "true",
trapPaths: "",
errorBurstThreshold: "30",
logRemoteURL: "",
logRemoteTLSCAFile: "",
logRemoteBuffer: "10000",
internal/metrics/metrics.go
+24 -19
View File
@@ -6,7 +6,6 @@ package metrics
import (
"net/http"
"strconv"
"strings"
"time"
"github.com/prometheus/client_golang/prometheus"
@@ -94,8 +93,8 @@ func New(topN int, instanceName string) *Metrics {
Help: "How long requests passed to the app took, from then to their end.",
}),
rateLimitHits: counterVec("smallwebwaf_rate_limit_hits_total",
"Requests that broke a rate limit or a byte limit, by its window and "+
"its kind, requests or bytes.",
"Requests that broke a rate limit, a byte limit or the error burst, by "+
"its window and its kind, requests, bytes or refusals.",
[]string{"window", "kind"}),
sizeAndTimeLimitHits: counterVec("smallwebwaf_size_and_time_limit_hits_total",
"Requests that passed a size or time limit, by its setting.",
@@ -407,26 +406,10 @@ func (m *Metrics) RequestEnded(
m.upstreamDuration.Observe(upstreamDuration.Seconds())
}
if line.LimitHit != "" {
// The log line names a byte limit's window with _bytes after it.
window, isBytes := strings.CutSuffix(line.LimitHit, "_bytes")
kind := ratelimit.KindRequests
if isBytes {
kind = ratelimit.KindBytes
}
m.rateLimitHits.WithLabelValues(window, kind).Inc()
}
if limit != "" {
m.sizeAndTimeLimitHits.WithLabelValues(limit).Inc()
}
if line.Offence != "" {
m.offences.WithLabelValues(line.Offence).Inc()
}
if line.Country != "" {
m.countries.add(line.Country, line)
}
@@ -436,6 +419,28 @@ func (m *Metrics) RequestEnded(
}
}
// LimitHit counts a request that broke a rate limit, a byte limit or the
// error burst, by the window and the kind of hit.
func (m *Metrics) LimitHit(hit ratelimit.Hit) {
m.rateLimitHits.WithLabelValues(hit.Window, hit.Kind).Inc()
}
// Offences counts the offences of r, a request that has ended, as its
// client's history counts them, by kind, named as clients.json names
// them.
func (m *Metrics) Offences(r ratelimit.Request) {
for kind, committed := range map[string]bool{
"limit": r.BrokeLimit,
"attack": r.Attack,
"rule_blocked": r.RuleBlocked,
"token_refused": r.TokenRefused,
} {
if committed {
m.offences.WithLabelValues(kind).Inc()
}
}
}
// RuleMatched counts a request that matched the rule id, whose action is
// action.
func (m *Metrics) RuleMatched(id, action string) {
internal/proxy/admin.go
+4 -2
View File
@@ -45,8 +45,9 @@ var (
// /_smallwebwaf/, once it has passed the checks. Each endpoint needs a
// token, sent as Authorization: Bearer <token>: the metrics
// SWWAF_METRICS_TOKEN, the others SWWAF_ADMIN_TOKEN. A request without
// it is refused with 401. An endpoint whose token is unset answers 404,
// as any other request under /_smallwebwaf/ does.
// it is refused with 401, which counts toward the error burst. An
// endpoint whose token is unset answers 404, as any other request under
// /_smallwebwaf/ does.
func (rq *request) answerAdmin() {
rq.line.Action = requestlog.ActionAdmin
rq.startClientResponseTimeout()
@@ -57,6 +58,7 @@ func (rq *request) answerAdmin() {
case token == "":
http.Error(rq.out, http.StatusText(http.StatusNotFound), http.StatusNotFound)
case !hasToken(rq.in, token):
rq.tokenRefused = true
rq.out.Header().Set("WWW-Authenticate", "Bearer")
rq.answer(refusal{
status: http.StatusUnauthorized,
internal/proxy/bans.go
+76 -30
View File
@@ -1,6 +1,7 @@
package proxy
import (
"net/http"
"net/netip"
"time"
@@ -9,7 +10,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
)
// banResponse is a refusal answered with SWWAF_BAN_RESPONSE, and logged
@@ -83,6 +83,48 @@ func (rq *request) countBytes() {
}
}
// countRefusal counts the request for the error burst once it has been
// answered, if smallwebwaf refused it after a rule file match or a trap
// path, or for a missing or wrong token, and in observe mode if enforce
// mode would have: more than SWWAF_ERROR_BURST_THRESHOLD such refusals of
// the client within a minute break a limit. A client in SWWAF_ALLOW_NETS,
// which the checks skip, is not counted, and nothing is while the
// threshold is off.
func (rq *request) countRefusal() {
cfg := rq.h.config
if cfg.ErrorBurstThreshold == 0 {
return
}
// In observe mode, a request that enforce mode would have refused
// before it reached the endpoint has had no token refused there.
tokenRefused := rq.tokenRefused && rq.line.WouldAction == "" &&
!isInside(rq.client, cfg.AllowNets)
if !rq.attack && !rq.ruleBlocked && !tokenRefused {
return
}
now := rq.h.now()
hit, over := rq.h.limiter.CountRefusal(rq.h.clientGroup(rq.client), now,
cfg.ErrorBurstThreshold)
if !over {
return
}
// What the client was sent, or in observe mode would have been.
status := rq.out.status
switch rq.line.WouldAction {
case requestlog.ActionRuleBlocked:
status = http.StatusForbidden
case requestlog.ActionBanned:
status = cfg.BanResponse
}
rq.banForLimit(now, hit, status)
}
// countedBytes returns the request's bytes, once it has ended, as the
// byte limits and the anomaly thresholds count them: the response's body
// bytes, the request's, or both, as SWWAF_BYTES_COUNT says. For an
@@ -107,20 +149,27 @@ func (rq *request) countedBytes() int64 {
}
// banForLimit bans the client's netblock at now for a broken limit, the
// one hit names, and notes the offence for the log line. status is what
// the client was sent, or is sent: SWWAF_BAN_RESPONSE for a request over
// a rate limit, the app's answer for one whose bytes broke a byte limit.
// The ban's notes give the client's limit percentage for that kind of
// limit. The ban sets the client's counters back to zero. In observe mode
// it makes no ban and sets nothing back, and raises the alert for the ban
// it would have made, if that alert would be sent.
// one hit names, notes the offence for the log line and counts the hit in
// the metrics. status is what the client was sent, or is sent:
// SWWAF_BAN_RESPONSE for a request over a rate limit, the app's answer for
// one whose bytes broke a byte limit, the refusal for one that broke the
// error burst. The ban's notes give the client's limit percentage for a
// rate limit or a byte limit; the error burst is not lowered. The ban sets
// the client's counters back to zero. In observe mode it makes no ban and
// sets nothing back, and raises the alert for the ban it would have made,
// if that alert would be sent.
func (rq *request) banForLimit(now time.Time, hit ratelimit.Hit, status int) {
rq.line.LimitHit = hit.Window
if hit.Kind == ratelimit.KindBytes {
rq.line.LimitHit += "_bytes" // as counts names the byte totals
switch hit.Kind {
case ratelimit.KindBytes:
rq.line.LimitHit = hit.Window + "_bytes" // as counts names the byte totals
case ratelimit.KindRefusals:
rq.line.LimitHit = requestlog.LimitHitErrorBurst
default:
rq.line.LimitHit = hit.Window
}
rq.line.Offence = requestlog.OffenceLimit
rq.h.metrics.LimitHit(hit)
netblock := rq.h.netblock(rq.client)
if rq.h.config.Observe && !rq.wouldAlertBan(netblock, now, bans.CauseLimit) {
@@ -140,13 +189,13 @@ func (rq *request) banForLimit(now time.Time, hit ratelimit.Hit, status int) {
Requests: rq.netblockRequests(netblock),
}
percent := rq.limitPercent
if hit.Kind == ratelimit.KindBytes {
percent = rq.bytesPercent
switch hit.Kind {
case ratelimit.KindRequests:
notes.LimitPercent, notes.LimitPercentSetting = rq.limitPercent.logged()
case ratelimit.KindBytes:
notes.LimitPercent, notes.LimitPercentSetting = rq.bytesPercent.logged()
}
notes.LimitPercent, notes.LimitPercentSetting = percent.logged()
if rq.h.config.Observe {
ban, wouldBan := rq.h.ledger.WouldBanForLimit(netblock, now, notes)
if wouldBan {
@@ -166,25 +215,22 @@ func (rq *request) banForLimit(now time.Time, hit ratelimit.Hit, status int) {
}
// banForAttack bans the client's netblock at now for a clear sign of
// attack, the match of rule, a ban rule. In observe mode it makes no ban,
// and raises the alert for the ban it would have made, if that alert
// would be sent.
func (rq *request) banForAttack(now time.Time, rule rules.Rule) {
// attack, which notes name: the ban rule that matched, or the trap path
// asked for. It fills in the rest of the notes. In observe mode it makes
// no ban, and raises the alert for the ban it would have made, if that
// alert would be sent.
func (rq *request) banForAttack(now time.Time, notes bans.Notes) {
netblock := rq.h.netblock(rq.client)
if rq.h.config.Observe && !rq.wouldAlertBan(netblock, now, bans.CauseAttack) {
return
}
notes := bans.Notes{
ASN: rq.line.ASN,
ASName: rq.line.ASName,
Country: rq.line.Country,
RuleID: rule.ID,
Target: rule.Target,
Reputation: rq.reputation,
Request: rq.noted(now, rq.h.config.BanResponse),
Requests: rq.netblockRequests(netblock),
}
notes.ASN = rq.line.ASN
notes.ASName = rq.line.ASName
notes.Country = rq.line.Country
notes.Reputation = rq.reputation
notes.Request = rq.noted(now, rq.h.config.BanResponse)
notes.Requests = rq.netblockRequests(netblock)
if rq.h.config.Observe {
ban, wouldBan := rq.h.ledger.WouldBanForAttack(netblock, now, notes)
internal/proxy/errorburst_test.go
+396
View File
@@ -0,0 +1,396 @@
package proxy_test
import (
"maps"
"net/http"
"net/netip"
"reflect"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/proxy"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
const errorBurstThreshold = "SWWAF_ERROR_BURST_THRESHOLD"
// refused is a request the tests here send, which smallwebwaf refuses
// after a rule file match, or for a missing or wrong token.
type refused int
const (
// blockRule is a request testRules' block rule refuses with 403.
blockRule refused = iota
// banRule is one its ban rule refuses with 403, and bans the client
// for.
banRule
// noMetricsToken is one for the metrics without a token, and
// wrongAdminToken one for the bans with the metrics token, each
// refused with 401.
noMetricsToken
wrongAdminToken
)
// send sends r from the client at from, checks its answer and log line as
// sender.request does, and returns the line.
func (r refused) send(s *sender, from string) logLine {
s.t.Helper()
switch r {
case blockRule:
return s.request(from, blockedPath, http.StatusForbidden,
requestlog.ActionRuleBlocked)
case banRule:
return s.request(from, probePath, http.StatusForbidden, requestlog.ActionBanned)
case noMetricsToken:
return s.request(from, proxy.MetricsPath, http.StatusUnauthorized,
requestlog.ActionAdmin)
case wrongAdminToken:
line, _ := s.requestWithHeader(from, proxy.BansPath, "Authorization: "+bearer,
http.StatusUnauthorized, requestlog.ActionAdmin)
return line
}
s.t.Fatalf("no request for the refusal %d", r)
return logLine{}
}
// startForErrorBurst is startWithClock with testRules, both tokens and
// SWWAF_ERROR_BURST_THRESHOLD at threshold, and the settings in env.
func startForErrorBurst(
t *testing.T, threshold string, env map[string]string,
) (*sender, *clock, *proxy.Server) {
t.Helper()
settings := map[string]string{
errorBurstThreshold: threshold,
rulesDir: writeRules(t, testRules),
adminToken: adminSecret,
metricsToken: token,
}
maps.Copy(settings, env)
return startWithClock(t, "", settings)
}
func TestErrorBurstBreaksAtOneOverTheThreshold(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
// refusals are four, one over the threshold of three.
refusals []refused
}{
{"block rule", []refused{blockRule, blockRule, blockRule, blockRule}},
{
"missing or wrong token",
[]refused{noMetricsToken, wrongAdminToken, noMetricsToken, wrongAdminToken},
},
{
"a mix ending in a ban rule",
[]refused{blockRule, noMetricsToken, blockRule, banRule},
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
s, _, _ := startForErrorBurst(t, "3", nil)
// Three refusals break nothing, and the app's answers between
// them are not counted.
for i, r := range tc.refusals[:3] {
line := r.send(s, client)
if line.LimitHit != "" || line.Offence != "" {
t.Errorf("refusal %d: log line has limit_hit %q and offence %q, "+
"want none", i+1, line.LimitHit, line.Offence)
}
s.get(client, http.StatusOK, requestlog.ActionForward)
}
// The fourth is answered as the others were, breaks the error
// burst, and bans the client.
line := tc.refusals[3].send(s, client)
if line.LimitHit != requestlog.LimitHitErrorBurst ||
line.Offence != requestlog.OffenceLimit {
t.Errorf("log line has limit_hit %q and offence %q, want error_burst "+
"and limit", line.LimitHit, line.Offence)
}
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
})
}
}
func TestErrorBurstBanNotesHistoryAndMetrics(t *testing.T) {
t.Parallel()
const scraper = "192.0.2.200"
s, clk, server := startForErrorBurst(t, "2", nil)
start := clk.Now()
blockRule.send(s, client)
wrongAdminToken.send(s, client)
line := blockRule.send(s, client)
expires := start.Add(time.Hour)
if line.BanExpires != requestlog.FormatTime(expires) {
t.Errorf("log line has ban_expires %q, want an hour on", line.BanExpires)
}
netblock := netip.MustParsePrefix(client + "/32")
want := bans.Ban{
Netblock: netblock,
Start: start,
Expires: expires,
Cause: bans.CauseLimit,
Reason: "refusals per minute over the limit of 2",
Notes: bans.Notes{
Kind: ratelimit.KindRefusals,
Limit: 2,
Window: minute,
Count: 3,
Request: bans.Request{
Time: start,
Method: http.MethodGet,
Host: appHost,
Path: blockedPath,
Status: http.StatusForbidden,
UserAgent: userAgent,
},
Requests: 3,
},
}
got := server.Ledger.Bans(netblock)
if len(got) != 1 || !reflect.DeepEqual(got[0], want) {
t.Fatalf("bans\n%+v\nwant\n%+v", got, want)
}
wantOffences := ratelimit.Offences{Limit: 1, RuleBlocked: 2, TokenRefused: 1}
if offences := historyOf(t, server, client).Offences; offences != wantOffences {
t.Errorf("history counts the offences %+v, want %+v", offences, wantOffences)
}
metrics := s.scrape(scraper)
wantMetric(t, metrics, `smallwebwaf_rate_limit_hits_total{instance="app",`+
`kind="refusals",window="minute"}`, 1)
wantMetric(t, metrics, `smallwebwaf_offences_total{instance="app",kind="limit"}`, 1)
wantMetric(t, metrics,
`smallwebwaf_offences_total{instance="app",kind="rule_blocked"}`, 2)
wantMetric(t, metrics,
`smallwebwaf_offences_total{instance="app",kind="token_refused"}`, 1)
wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="limit",instance="app"}`, 1)
}
func TestErrorBurstIsNotLoweredForAClientWithLowerLimits(t *testing.T) {
t.Parallel()
geojsURL, _ := startGeoJS(t)
s, _, server := startWithClock(t, geojsURL, map[string]string{
errorBurstThreshold: "2",
rulesDir: writeRules(t, testRules),
countryLimitPercent: countryDEHalf,
})
// Half of the threshold would be one, which the second refusal is over.
for range 2 {
line := blockRule.send(s, fromDE)
if line.LimitHit != "" {
t.Errorf("log line has limit_hit %q, want none", line.LimitHit)
}
}
blockRule.send(s, fromDE)
got := server.Ledger.Bans(netip.MustParsePrefix(fromDE + "/32"))
if len(got) != 1 || got[0].Notes.Limit != 2 || got[0].Notes.LimitPercent != nil {
t.Errorf("bans %+v, want one for the limit of 2, without a limit percentage", got)
}
}
func TestErrorBurstDoesNotCountTheAppsAnswers(t *testing.T) {
t.Parallel()
statuses := map[string]int{
"/missing": http.StatusNotFound,
"/private": http.StatusUnauthorized,
"/forbidden": http.StatusForbidden,
}
s, _, _, queue := startAppWithAlerts(t, func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(statuses[r.URL.Path])
}, map[string]string{errorBurstThreshold: "1", rulesDir: writeRules(t, testRules)})
for range 2 {
for path, status := range statuses {
s.request(client, path, status, requestlog.ActionForward)
}
}
// The first refusal is one, not over the threshold.
line := blockRule.send(s, client)
if line.LimitHit != "" {
t.Errorf("log line has limit_hit %q, want none", line.LimitHit)
}
// No ban was made, nor its alert raised.
s.request(client, "/missing", http.StatusNotFound, requestlog.ActionForward)
wantAlerts(t, queue)
}
func TestErrorBurstOffOrAtItsDefault(t *testing.T) {
t.Parallel()
const off = "off"
for _, tc := range []struct {
threshold string
// broken is whether the 31st refusal breaks the error burst.
broken bool
}{
{"", true},
{off, false},
} {
t.Run(errorBurstThreshold+"="+tc.threshold, func(t *testing.T) {
t.Parallel()
env := map[string]string{rulesDir: writeRules(t, testRules)}
if tc.threshold != "" {
env[errorBurstThreshold] = tc.threshold
}
s, _, _ := startWithClock(t, "", env)
var line logLine
for range 31 {
line = blockRule.send(s, client)
}
if broken := line.LimitHit == requestlog.LimitHitErrorBurst; broken != tc.broken {
t.Errorf("the 31st refusal broke the error burst: %t, want %t",
broken, tc.broken)
}
})
}
}
func TestErrorBurstCountsEachClientTheChecksApplyTo(t *testing.T) {
t.Parallel()
const (
allowed = "192.0.2.60" // in SWWAF_ALLOW_NETS
exempt = "192.0.2.50" // in SWWAF_RATE_LIMIT_EXEMPT_NETS
)
s, _, _ := startForErrorBurst(t, "1", map[string]string{
allowNets: allowed, rateLimitExemptNets: exempt,
})
// A client in SWWAF_ALLOW_NETS still needs the token, but is not
// counted.
for range 3 {
line := noMetricsToken.send(s, allowed)
if line.LimitHit != "" {
t.Errorf("log line has limit_hit %q, want none", line.LimitHit)
}
}
// One the rate limits do not apply to is.
noMetricsToken.send(s, exempt)
line := wrongAdminToken.send(s, exempt)
if line.LimitHit != requestlog.LimitHitErrorBurst {
t.Errorf("log line has limit_hit %q, want error_burst", line.LimitHit)
}
s.get(exempt, http.StatusForbidden, requestlog.ActionBanned)
}
func TestErrorBurstBanSetsTheRefusalsBackToZero(t *testing.T) {
t.Parallel()
s, clk, _ := startForErrorBurst(t, "1", map[string]string{limitBanDuration: "1s"})
blockRule.send(s, client)
blockRule.send(s, client)
// Within the same minute, once the ban has ended, the next refusal is
// the first again.
clk.advance(time.Second)
line := blockRule.send(s, client)
if line.LimitHit != "" {
t.Errorf("log line has limit_hit %q, want none", line.LimitHit)
}
}
func TestObserveModeLogsAndAlertsTheErrorBurst(t *testing.T) {
t.Parallel()
s, clk, server, queue := startWithAlerts(t, map[string]string{
mode: observe,
errorBurstThreshold: "1",
rulesDir: writeRules(t, testRules),
adminToken: adminSecret,
})
start := clk.Now()
held := bans.Ban{
Netblock: netip.MustParsePrefix(otherClient + "/32"),
Start: start,
Expires: start.Add(time.Hour),
Cause: bans.CauseAdmin,
}
server.Ledger.Load([]bans.Ban{held})
// Under a ban, enforce mode would have refused these before the
// endpoint, so their tokens are not counted.
for range 2 {
line := wrongAdminToken.send(s, otherClient)
wantWouldAction(t, line, requestlog.ActionBanned)
if line.LimitHit != "" {
t.Errorf("log line has limit_hit %q, want none", line.LimitHit)
}
}
// The block rule's refusal, which enforce mode would have answered 403,
// is the second of the client's, and would have banned it.
wrongAdminToken.send(s, client)
line := s.request(client, blockedPath, http.StatusOK, requestlog.ActionForward)
wantWouldAction(t, line, requestlog.ActionRuleBlocked)
if line.LimitHit != requestlog.LimitHitErrorBurst || line.BanExpires != "" {
t.Errorf("log line has limit_hit %q and ban_expires %q, want error_burst "+
"and none", line.LimitHit, line.BanExpires)
}
if got := server.Ledger.Snapshot(); len(got) != 1 || !reflect.DeepEqual(got[0], held) {
t.Errorf("bans %+v, want only the one held", got)
}
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
if len(waiting) != 1 {
t.Fatalf("%d alerts wait, want 1: %+v", len(waiting), waiting)
}
notes, _ := waiting[0].Detail["notes"].(bans.Notes)
if notes.Kind != ratelimit.KindRefusals || notes.Count != 2 ||
notes.Request.Status != http.StatusForbidden {
t.Errorf("the alert's notes are %+v, want two refusals, the last answered 403",
notes)
}
alert := banAlert(alerts.EventBan, start, client, bans.Ban{
Netblock: netip.MustParsePrefix(client + "/32"), Cause: bans.CauseLimit,
Reason: "refusals per minute over the limit of 1", Notes: notes,
}, requestlog.FormatTime(start.Add(time.Hour)))
alert.Detail["mode"] = observe
wantAlerts(t, queue, alert)
}
internal/proxy/proxy.go
+4 -1
View File
@@ -282,9 +282,12 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
return
}
// Once the request has ended, before its log line is written.
// Once the request has ended, before its log line is written. The
// last deferred runs first: countRefusal before addToHistory, so that
// a broken error burst is in the client's history.
defer rq.addToHistory()
defer rq.countAnomalies()
defer rq.countRefusal()
refused := rq.check(r.Context())
rq.checked = time.Now()
internal/proxy/reputation_test.go
+19 -6
View File
@@ -701,10 +701,14 @@ func TestIPv6ClientCostsOneAbuseIPDBCheckWhicheverOfItsAddressesSends(t *testing
wantAbuseIPDBChecks(t, server, 1)
}
// probePath is the path the ban rule of testRules, probe, matches.
const probePath = "/.env"
// probePath is the path the ban rule of testRules, probe, matches, and
// blockedPath the one its block rule, blocked, matches.
const (
probePath = "/.env"
blockedPath = "/blocked"
)
func TestClientARuleRefusedIsCheckedWithAbuseIPDBAtItsNextRequest(t *testing.T) {
func TestClientRefusedForAnOffenceIsCheckedWithAbuseIPDBAtItsNextRequest(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
@@ -718,13 +722,21 @@ func TestClientARuleRefusedIsCheckedWithAbuseIPDBAtItsNextRequest(t *testing.T)
want ratelimit.Offences
}{
{
"a block rule", "/blocked", http.StatusForbidden, requestlog.ActionRuleBlocked,
"a block rule", blockedPath, http.StatusForbidden, requestlog.ActionRuleBlocked,
ratelimit.Offences{RuleBlocked: 1},
},
{
"a ban rule", probePath, http.StatusForbidden, requestlog.ActionBanned,
ratelimit.Offences{Attack: 1},
},
{
"a trap path", "/xmlrpc.php", http.StatusForbidden, requestlog.ActionBanned,
ratelimit.Offences{Attack: 1},
},
{
"a missing token", proxy.MetricsPath, http.StatusUnauthorized,
requestlog.ActionAdmin, ratelimit.Offences{TokenRefused: 1},
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
@@ -732,6 +744,7 @@ func TestClientARuleRefusedIsCheckedWithAbuseIPDBAtItsNextRequest(t *testing.T)
s, clk, server := startWithClock(t, "", map[string]string{
abuseIPDBKey: accountKey, reputationAction: actionLog,
rulesDir: writeRules(t, testRules), attackBanDuration: "1h",
trapPaths: trapPathList, metricsToken: token,
})
s.request(client, tc.path, tc.status, tc.action)
@@ -741,8 +754,8 @@ func TestClientARuleRefusedIsCheckedWithAbuseIPDBAtItsNextRequest(t *testing.T)
t.Errorf("history counts the offences %+v, want %+v", got, tc.want)
}
// Its next request, once a ban rule's ban has ended, has it
// checked.
// Its next request, once any ban for a clear sign of attack
// has ended, has it checked.
clk.advance(time.Hour)
s.get(client, http.StatusOK, requestlog.ActionForward)
wantAbuseIPDBChecks(t, server, 1)
internal/proxy/request.go
+23 -14
View File
@@ -64,10 +64,11 @@ type request struct {
// limits and for the byte limits.
counted bool
limitPercent, bytesPercent percentage
// attack is true for a request that matched a ban rule, and
// ruleBlocked for one a block rule refused, each an offence its
// client's history counts.
attack, ruleBlocked bool
// attack is true for a request that matched a ban rule or asked for a
// trap path, ruleBlocked for one a block rule refused, and
// tokenRefused for one refused for a missing or wrong token, each an
// offence its client's history counts.
attack, ruleBlocked, tokenRefused bool
// blocklisted is true once a blocklist is found to list the client,
// dnsblListed once a DNSBL zone's verdict is, and abuseIPDBHit once
// AbuseIPDB's score of it is a hit.
@@ -232,9 +233,9 @@ func (rq *request) check(ctx context.Context) *refusal {
// rate limits, unless the client is in SWWAF_RATE_LIMIT_EXEMPT_NETS or the
// request's path is exempt under SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that
// every other request is counted, each of them by the client's limit
// percentages, and last the rule files. A request exempt from the rate
// limits is exempt from the byte limits too. ctx is the request's own
// context.
// percentages, then SWWAF_TRAP_PATHS, and last the rule files. A request
// exempt from the rate limits is exempt from the byte limits too. ctx is
// the request's own context.
func (rq *request) checkClient(ctx context.Context) string {
cfg := rq.h.config
if isInside(rq.client, cfg.AllowNets) {
@@ -281,6 +282,10 @@ func (rq *request) checkClient(ctx context.Context) string {
return requestlog.ActionRateLimited
}
if rq.trapPath(now) {
return requestlog.ActionBanned
}
return rq.checkRules(now)
}
@@ -541,14 +546,14 @@ func timing(start, end time.Time) *float64 {
}
// addToHistory adds the request, which has ended, to its client's
// history, and then the lookup's answer about the client, as
// answerAtTheEnd gives it, to that history and to the notes of the bans
// on its netblock: an answer may have come before either was there, and
// one from GeoJS that comes later is added when it comes.
// history, and counts its offences in the metrics, and then the lookup's
// answer about the client, as answerAtTheEnd gives it, to that history and
// to the notes of the bans on its netblock: an answer may have come
// before either was there, and one from GeoJS that comes later is added
// when it comes.
func (rq *request) addToHistory() {
forwarded := !rq.upstreamStart.IsZero()
rq.h.limiter.AddToHistory(rq.h.clientGroup(rq.client), rq.h.now(), ratelimit.Request{
request := ratelimit.Request{
Forwarded: forwarded,
Refused: !forwarded && rq.refused.Load() != nil,
Status: rq.out.status,
@@ -557,7 +562,11 @@ func (rq *request) addToHistory() {
BrokeLimit: rq.line.Offence == requestlog.OffenceLimit,
Attack: rq.attack,
RuleBlocked: rq.ruleBlocked,
})
TokenRefused: rq.tokenRefused,
}
rq.h.limiter.AddToHistory(rq.h.clientGroup(rq.client), rq.h.now(), request)
rq.h.metrics.Offences(request)
answer, found := rq.answerAtTheEnd()
if found {
internal/proxy/rulefiles_test.go
+3
View File
@@ -204,6 +204,9 @@ func TestMetricsCountRuleMatchesAndBansForAnAttack(t *testing.T) {
wantMetric(t, metrics, `smallwebwaf_rules_loaded{instance="app"}`, 2)
wantMetric(t, metrics, `smallwebwaf_requests_total{action="rule_blocked",`+
`instance="app",status_class="4xx"}`, 1)
wantMetric(t, metrics,
`smallwebwaf_offences_total{instance="app",kind="rule_blocked"}`, 1)
wantMetric(t, metrics, `smallwebwaf_offences_total{instance="app",kind="attack"}`, 1)
wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="attack",instance="app"}`, 1)
wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="limit",instance="app"}`, 0)
wantMetric(t, metrics, `smallwebwaf_permanent_bans{instance="app"}`, 1)
internal/proxy/rules.go
+20 -1
View File
@@ -1,12 +1,31 @@
package proxy
import (
"slices"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
)
// trapPath reports whether the request asks for a path in
// SWWAF_TRAP_PATHS: its path as a path rule sees it, before any decoding
// and without the query, is one of them. Such a request is a clear sign of
// attack, as a ban rule's match is: it bans the client's netblock, or in
// observe mode raises the alert for the ban it would have made.
func (rq *request) trapPath(now time.Time) bool {
path := rules.Path(rq.in)
if !slices.Contains(rq.h.config.TrapPaths, path) {
return false
}
rq.attack = true
rq.banForAttack(now, bans.Notes{TrapPath: path})
return true
}
// checkRules checks the request against the rules of the rule files at
// now, notes the ids of those it matches in the log line, and returns the
// action of the rule that refuses it, ActionRuleBlocked for a block rule
@@ -34,7 +53,7 @@ func (rq *request) checkRules(now time.Time) string {
return requestlog.ActionRuleBlocked
case rules.ActionBan:
rq.attack = true
rq.banForAttack(now, last)
rq.banForAttack(now, bans.Notes{RuleID: last.ID, Target: last.Target})
return requestlog.ActionBanned
default:
internal/proxy/trappaths_test.go
+115
View File
@@ -0,0 +1,115 @@
package proxy_test
import (
"net/http"
"net/netip"
"reflect"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// trapPaths is the setting's name, and trapPathList what the tests set it
// to.
const (
trapPaths = "SWWAF_TRAP_PATHS"
trapPathList = "/wp-login.php,/xmlrpc.php"
)
func TestTrapPathBansAsABanRuleDoes(t *testing.T) {
t.Parallel()
const allowed = "192.0.2.60" // in SWWAF_ALLOW_NETS
// A block rule for the same path: the trap path comes first.
s, clk, server := startWithClock(t, "", map[string]string{
trapPaths: trapPathList,
rulesDir: writeRules(t, `wp path block ^/wp-login\.php$`),
allowNets: allowed,
banResponse: "429",
})
start := clk.Now()
// Only the path itself, as the client sent it, is a trap path.
for _, path := range []string{
"/wp-login.php/", "/WP-LOGIN.PHP", "/blog/xmlrpc.php", "/%77p-login.php",
} {
s.request(otherClient, path, http.StatusOK, requestlog.ActionForward)
}
// A client in SWWAF_ALLOW_NETS is not checked.
s.request(allowed, "/xmlrpc.php", http.StatusOK, requestlog.ActionForward)
// The query is not part of the path.
line := s.request(client, "/wp-login.php?redirect_to=x", http.StatusTooManyRequests,
requestlog.ActionBanned)
wantRuleIDs(t, line)
if line.BanExpires != requestlog.FormatTime(start.Add(7*24*time.Hour)) {
t.Errorf("log line has ban_expires %q, want seven days on", line.BanExpires)
}
netblock := netip.MustParsePrefix(client + "/32")
want := bans.Ban{
Netblock: netblock,
Start: start,
Expires: start.Add(7 * 24 * time.Hour),
Cause: bans.CauseAttack,
Reason: "asked for the trap path /wp-login.php",
Notes: bans.Notes{
TrapPath: "/wp-login.php",
Request: bans.Request{
Time: start,
Method: http.MethodGet,
Host: appHost,
Path: "/wp-login.php?redirect_to=x",
Status: http.StatusTooManyRequests,
UserAgent: userAgent,
},
Requests: 1,
},
}
got := server.Ledger.Bans(netblock)
if len(got) != 1 || !reflect.DeepEqual(got[0], want) {
t.Fatalf("bans\n%+v\nwant\n%+v", got, want)
}
// The next request is refused under the ban, and makes it permanent.
line = s.get(client, http.StatusTooManyRequests, requestlog.ActionBanned)
if line.BanExpires != permanent {
t.Errorf("log line has ban_expires %q, want permanent", line.BanExpires)
}
}
func TestTrapPathsNeedNoRuleFiles(t *testing.T) {
t.Parallel()
s, _, _ := startWithClock(t, "", map[string]string{
trapPaths: trapPathList,
"SWWAF_RULES_ENABLED": "false",
})
s.request(client, "/xmlrpc.php", http.StatusForbidden, requestlog.ActionBanned)
}
func TestObserveModeLogsWhatATrapPathWouldDo(t *testing.T) {
t.Parallel()
s, _, server := startWithClock(t, "", map[string]string{
trapPaths: trapPathList,
mode: observe,
})
line := s.request(client, "/xmlrpc.php", http.StatusOK, requestlog.ActionForward)
wantWouldAction(t, line, requestlog.ActionBanned)
// No ban was made.
s.get(client, http.StatusOK, requestlog.ActionForward)
if got := server.Ledger.Snapshot(); len(got) != 0 {
t.Errorf("bans %+v, want none", got)
}
}
internal/ratelimit/ratelimit.go
+65 -28
View File
@@ -25,6 +25,9 @@ const (
KindRequests = "requests"
// KindBytes is a byte limit, on a client's bytes.
KindBytes = "bytes"
// KindRefusals is the error burst, on a client's requests smallwebwaf
// refused after a rule file match or for a missing or wrong token.
KindRefusals = "refusals"
)
// Limits are the most requests a client may make in a minute, an hour and
@@ -50,18 +53,20 @@ type Limiter struct {
}
// Client is a client in the table, as clients.json holds it: its buckets
// of requests and of bytes in each window, and its history.
// of requests and of bytes in each window, its buckets of refusals in the
// minute, which the error burst counts, and its history.
//
//nolint:tagliatelle // the state files use snake_case, as the request log does
type Client struct {
Client netip.Prefix `json:"client"`
Minute Buckets `json:"minute"`
Hour Buckets `json:"hour"`
Day Buckets `json:"day"`
MinuteBytes Buckets `json:"minute_bytes"`
HourBytes Buckets `json:"hour_bytes"`
DayBytes Buckets `json:"day_bytes"`
History History `json:"history"`
Client netip.Prefix `json:"client"`
Minute Buckets `json:"minute"`
Hour Buckets `json:"hour"`
Day Buckets `json:"day"`
MinuteBytes Buckets `json:"minute_bytes"`
HourBytes Buckets `json:"hour_bytes"`
DayBytes Buckets `json:"day_bytes"`
MinuteRefusals Buckets `json:"minute_refusals"`
History History `json:"history"`
}
// Buckets are a client's two buckets in one window: the requests, or the
@@ -116,12 +121,15 @@ type Responses struct {
//
//nolint:tagliatelle // the state files use snake_case, as the request log does
type Offences struct {
// Limit is its requests that broke a rate limit or a byte limit,
// Attack those that matched a ban rule, a clear sign of attack, and
// RuleBlocked those a block rule refused.
Limit int64 `json:"limit"`
Attack int64 `json:"attack"`
RuleBlocked int64 `json:"rule_blocked"`
// Limit is its requests that broke a rate limit, a byte limit or the
// error burst, Attack those that were a clear sign of attack, a match
// of a ban rule or a request for a trap path, RuleBlocked those a block
// rule refused, and TokenRefused those refused for a missing or wrong
// token.
Limit int64 `json:"limit"`
Attack int64 `json:"attack"`
RuleBlocked int64 `json:"rule_blocked"`
TokenRefused int64 `json:"token_refused"`
}
// Request is what a client's history keeps of one of its requests.
@@ -138,12 +146,14 @@ type Request struct {
// and of its response.
RequestBytes int64
ResponseBytes int64
// BrokeLimit is true for a request that broke a rate limit or a byte
// limit, Attack for one that matched a ban rule, and RuleBlocked for
// one a block rule refused.
BrokeLimit bool
Attack bool
RuleBlocked bool
// BrokeLimit is true for a request that broke a rate limit, a byte
// limit or the error burst, Attack for one that matched a ban rule or
// asked for a trap path, RuleBlocked for one a block rule refused, and
// TokenRefused for one refused for a missing or wrong token.
BrokeLimit bool
Attack bool
RuleBlocked bool
TokenRefused bool
}
// New returns a Limiter for limits, with no client counted yet, whose
@@ -175,17 +185,18 @@ func New(limits Limits, maxClients int) *Limiter {
}
}
// Hit is a request that takes a client over a rate limit, or whose bytes
// take it over a byte limit.
// Hit is a request that takes a client over a rate limit or the error
// burst, or whose bytes take it over a byte limit.
type Hit struct {
// Kind is KindRequests for a rate limit, KindBytes for a byte limit.
// Kind is KindRequests for a rate limit, KindBytes for a byte limit,
// KindRefusals for the error burst.
Kind string
// Window is "minute", "hour" or "day".
Window string
// Limit is the window's limit, as the client's percentage of it.
Limit int64
// Count is the client's requests, or bytes, counted in the window,
// this request's included.
// Count is the client's requests, bytes or refusals counted in the
// window, this request's included.
Count float64
}
@@ -224,8 +235,25 @@ func (l *Limiter) CountBytes(
return l.count(client, now, 0, bytes, percent)
}
// Reset sets client's counts of requests and of bytes in every window
// back to zero. Its history keeps its totals.
// CountRefusal counts a request from client at now that smallwebwaf
// refused after a rule file match or for a missing or wrong token, and
// reports whether the client's refusals in the minute that ends at now,
// this one included, are more than threshold, which breaks the error
// burst, and the hit.
func (l *Limiter) CountRefusal(
client netip.Prefix, now time.Time, threshold int64,
) (Hit, bool) {
l.mu.Lock()
defer l.mu.Unlock()
count := l.get(client).MinuteRefusals.Add(now, time.Minute, 1)
hit := Hit{Kind: KindRefusals, Window: "minute", Limit: threshold, Count: count}
return hit, count > float64(threshold)
}
// Reset sets client's counts of requests, of bytes and of refusals in
// every window back to zero. Its history keeps its totals.
func (l *Limiter) Reset(client netip.Prefix) {
l.mu.Lock()
defer l.mu.Unlock()
@@ -234,6 +262,7 @@ func (l *Limiter) Reset(client netip.Prefix) {
if seen {
c.Minute, c.Hour, c.Day = Buckets{}, Buckets{}, Buckets{}
c.MinuteBytes, c.HourBytes, c.DayBytes = Buckets{}, Buckets{}, Buckets{}
c.MinuteRefusals = Buckets{}
}
}
@@ -274,6 +303,10 @@ func (l *Limiter) AddToHistory(client netip.Prefix, now time.Time, r Request) {
if r.RuleBlocked {
h.Offences.RuleBlocked++
}
if r.TokenRefused {
h.Offences.TokenRefused++
}
}
// AddLookup gives client's history its AS number, AS name and country, as
@@ -383,6 +416,10 @@ func (l *Limiter) Load(clients []Client, now time.Time) {
}
}
if c.MinuteRefusals.Passed(now, time.Minute) {
c.MinuteRefusals = Buckets{}
}
l.clients.Add(c.Client, &c)
}
}
internal/ratelimit/ratelimit_test.go
+40
View File
@@ -248,6 +248,46 @@ func TestResetSetsTheBytesBackToZero(t *testing.T) {
wantBytesCount(t, limiter, client, start, 1000, "")
}
func TestRefusalsOverTheThresholdInAMinuteBreakTheErrorBurst(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
for range limit {
if _, over := limiter.CountRefusal(client, start, limit); over {
t.Fatalf("a refusal within the threshold of %d broke the error burst", limit)
}
}
hit, over := limiter.CountRefusal(client, start, limit)
want := ratelimit.Hit{
Kind: ratelimit.KindRefusals, Window: minute, Limit: limit, Count: limit + 1,
}
if !over || hit != want {
t.Errorf("one over the threshold broke it: %t, with %+v; want %+v", over, hit,
want)
}
// Half a minute into the next, half of those four still count, 2, and
// this one: 3, within the threshold.
hit, over = limiter.CountRefusal(client, start.Add(time.Minute+time.Minute/2), limit)
if over || hit.Count != 3 {
t.Errorf("half a minute on, %v refusals broke it: %t; want 3, false",
hit.Count, over)
}
// A ban sets them back to zero.
limiter.Reset(client)
hit, _ = limiter.CountRefusal(client, start.Add(time.Minute+time.Minute/2), limit)
if hit.Count != 1 {
t.Errorf("after a reset, %v refusals, want 1", hit.Count)
}
}
func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
t.Parallel()
internal/ratelimit/snapshot_test.go
+13 -6
View File
@@ -65,6 +65,7 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
limiter.Count(client, start, whole)
limiter.CountBytes(client, start, 5, whole)
limiter.CountRefusal(client, start, limit)
limiter.AddToHistory(client, start, ratelimit.Request{Forwarded: true})
loaded := func(now time.Time) ratelimit.Client {
@@ -76,9 +77,9 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
return after.Snapshot()[0]
}
// Two minutes on, the window that ends then covers neither of the
// minute's buckets, of requests and of bytes, which are emptied; the
// hour's and the day's stay, and so does the history.
// Two minutes on, the window that ends then covers none of the
// minute's buckets, of requests, of bytes and of refusals, which are
// emptied; the hour's and the day's stay, and so does the history.
got := loaded(start.Add(2 * time.Minute))
if got.Minute != (ratelimit.Buckets{}) || got.Hour.Current != 1 ||
got.Day.Current != 1 || got.History.Requests != 1 {
@@ -91,11 +92,17 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
got.MinuteBytes, got.HourBytes, got.DayBytes)
}
if got.MinuteRefusals != (ratelimit.Buckets{}) {
t.Errorf("loaded two minutes on with buckets of refusals %+v",
got.MinuteRefusals)
}
// A moment before, the window still covers some of the earlier one.
got = loaded(start.Add(2*time.Minute - time.Nanosecond))
if got.Minute.Current != 1 || got.MinuteBytes.Current != 5 {
t.Errorf("loaded just under two minutes on with minute buckets %+v and %+v",
got.Minute, got.MinuteBytes)
if got.Minute.Current != 1 || got.MinuteBytes.Current != 5 ||
got.MinuteRefusals.Current != 1 {
t.Errorf("loaded just under two minutes on with minute buckets %+v, %+v "+
"and %+v", got.Minute, got.MinuteBytes, got.MinuteRefusals)
}
}
internal/requestlog/requestlog.go
+11 -4
View File
@@ -29,8 +29,9 @@ const (
// over a rate limit, which bans the client.
ActionRateLimited = "rate_limited"
// ActionBanned is a request refused because a ban covers its client,
// or because it matched a ban rule or the CrowdSec decision list lists
// its client, either of which bans the client.
// or because it matched a ban rule, asked for a trap path or the
// CrowdSec decision list lists its client, each of which bans the
// client.
ActionBanned = "banned"
// ActionRuleBlocked is a request refused because it matched a block
// rule.
@@ -48,9 +49,14 @@ const (
)
// OffenceLimit is the offence a request line names for a request that
// broke a rate limit, or whose bytes broke a byte limit.
// broke a rate limit or the error burst, or whose bytes broke a byte
// limit.
const OffenceLimit = "limit"
// LimitHitErrorBurst is the limit_hit a request line names for a request
// that broke the error burst.
const LimitHitErrorBurst = "error_burst"
// timeLayout is RFC 3339 with milliseconds.
const timeLayout = "2006-01-02T15:04:05.000Z07:00"
@@ -138,7 +144,8 @@ type Line struct {
RuleIDs []string `json:"rule_ids,omitempty"`
// LimitHit is the window whose limit the request went over, named as
// Counts names its count: minute, hour or day for a rate limit, and
// minute_bytes, hour_bytes or day_bytes for a byte limit.
// minute_bytes, hour_bytes or day_bytes for a byte limit; or
// LimitHitErrorBurst for the error burst.
LimitHit string `json:"limit_hit,omitempty"`
// Reputation are the URLs of the blocklists that list the client, then
// that of the CrowdSec decision list when it does, then the DNSBL zones
internal/rules/rules.go
+13 -6
View File
@@ -396,16 +396,23 @@ func (rule Rule) matches(r *http.Request) bool {
return rule.regex.MatchString(value(rule.Target, r))
}
// Path returns r's path as the client sent it, before any decoding or
// re-encoding, up to the first ?: what a path rule is matched against.
func Path(r *http.Request) string {
path, _, _ := strings.Cut(pathAndQuery(r), "?")
return path
}
// value returns what a rule with target, other than uri, is matched
// against in r: the path and the query as the client sent them, before
// any decoding or re-encoding, split at the first ?, and a header's values
// joined by ", ", as HTTP joins those of a header sent more than once.
// against in r: the path, as Path gives it, and the query as the client
// sent it, before any decoding or re-encoding, after the first ?, and a
// header's values joined by ", ", as HTTP joins those of a header sent
// more than once.
func value(target string, r *http.Request) string {
switch target {
case "path":
path, _, _ := strings.Cut(pathAndQuery(r), "?")
return path
return Path(r)
case "query":
_, query, _ := strings.Cut(pathAndQuery(r), "?")
internal/state/state.go
+6 -4
View File
@@ -679,8 +679,8 @@ func (f *bansFile) check(data []byte) error {
}
// check refuses a client without its address, which would count nobody's
// requests, or with requests or bytes in a window but no start, which
// would drop them and give the client a fresh allowance.
// requests, or with requests, bytes or refusals in a window but no start,
// which would drop them and give the client a fresh allowance.
func (f *clientsFile) check([]byte) error {
for i, client := range f.Clients {
switch {
@@ -698,6 +698,8 @@ func (f *clientsFile) check([]byte) error {
return missing(i, "hour_bytes.start")
case countsWithoutStart(client.DayBytes):
return missing(i, "day_bytes.start")
case countsWithoutStart(client.MinuteRefusals):
return missing(i, "minute_refusals.start")
}
}
@@ -898,8 +900,8 @@ func missingFromCounter(counter anomaly.Counter) string {
}
}
// countsWithoutStart reports whether b holds requests, or bytes, but no
// start, which places them in time.
// countsWithoutStart reports whether b holds requests, bytes or refusals
// but no start, which places them in time.
func countsWithoutStart(b ratelimit.Buckets) bool {
return b.Start.IsZero() && (b.Current != 0 || b.Previous != 0)
}
internal/state/state_test.go
+9
View File
@@ -639,6 +639,15 @@ func TestEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
}
}
func TestClientWithRefusalsInTheMinuteWithoutTheirStartStopsTheStart(t *testing.T) {
t.Parallel()
wantRefused(t, clientsJSON,
`{"version": 1, "clients": [{"client": "203.0.113.9/32", `+
`"minute_refusals": {"current": 2}}]}`,
`: entry 1 has no "minute_refusals.start"`)
}
func TestReputationJSONEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
t.Parallel()