waf: ban the clients on AbuseIPDB's blacklist (closes #411)
check / check (push) Successful in 8m16s

While SWW_WAF_ABUSEIPDB_KEY is set, waf fetches AbuseIPDB's blacklist at
the start and every SWW_WAF_ABUSEIPDB_BLACKLIST_REFRESH_INTERVAL_MINS
(60), cut at SWW_WAF_ABUSEIPDB_THRESHOLD (90, from 25 to 100). Each
listed client's netblock gets a ban of SWW_WAF_ABUSEIPDB_BAN_DAYS (30)
with the new cause abuseipdb. The blacklist itself is not kept.

These bans do not count toward SWW_WAF_MAX_BANS, so a 10,000-address
blacklist cannot push out the bans for limits and attacks. An ended
abuseipdb ban is forgotten at each fetch and each read of bans.json. A
netblock with an active ban is left alone, and so is one whose abuseipdb
ban an admin lifted, until that ban would have ended. Observe mode makes
no ban.

Model: opus-5-5
This commit was merged in pull request #416.
This commit is contained in:
clawbot committed 2026-11-19 12:42:41 +01:00
1 parent 01e6b5d47e
commit 809d2664d7
22 files changed
+1338 -222

No files matched your search

+45 -17
View File
@@ -94,7 +94,8 @@ never as a time.
to count and report failures.
- `internal/lookup`: looks clients up in the lookup database.
- `internal/reputation`: fetches the lists, asks the DNS blocklists and
AbuseIPDB in the background, and keeps their answers.
AbuseIPDB in the background, and keeps their answers. It also fetches
AbuseIPDB's blacklist, which `internal/waf` bans.
- `internal/routewatch`: asks a routewatch instance about clients in the
background, and keeps its answers.
- `internal/anomaly`: the anomaly counters and their alerts.
@@ -240,13 +241,15 @@ apply.
load.
- Memory is bounded. When the table of clients holds
`SWW_WAF_MAX_TRACKED_CLIENTS`, the least recently seen client is dropped
first, with its history. The bans are bounded by `SWW_WAF_MAX_BANS` (see
"Bans"). The GeoJS answers, the reputation verdicts and scores, the DNS
queries under way and the anomaly counters have fixed bounds, listed under
"Fixed limits" in `docs/settings.md`. Past its bound, the GeoJS answer or
anomaly counter used longest ago goes first, and the verdict or score fetched
longest ago. Past the bound on DNS queries under way, a zone is asked at the
client's next request.
first, with its history. The bans for broken limits and clear signs of attack
are bounded by `SWW_WAF_MAX_BANS` (see "Bans"). Those for AbuseIPDB's
blacklist are bounded by the clients it listed over the last
`SWW_WAF_ABUSEIPDB_BAN_DAYS` days (see "Reputation"). The GeoJS answers, the
reputation verdicts and scores, the DNS queries under way and the anomaly
counters have fixed bounds, listed under "Fixed limits" in `docs/settings.md`.
Past its bound, the GeoJS answer or anomaly counter used longest ago goes
first, and the verdict or score fetched longest ago. Past the bound on DNS
queries under way, a zone is asked at the client's next request.
## Bans
@@ -301,6 +304,9 @@ to a ban, each by its own rule:
Once it ends, only new requests can break a limit again. The client's
history keeps its totals.
A client on AbuseIPDB's blacklist is banned before it sends a request, with the
cause `abuseipdb` (see "Reputation").
A Core Rule Set match refuses only the request it matched, with `403` and
without a ban, because the Core Rule Set has false positives. A `block` rule
does the same. Those refusals count toward the refusal limit, so a client that
@@ -327,11 +333,12 @@ does not count toward a longer ban; it refuses nothing. Deleting its entry from
A permanent ban does not run out. It ends when an admin lifts it, or when `waf`
drops it from a full table, as below. When `waf` holds `SWW_WAF_MAX_BANS` bans
of its own, it drops the earliest ban of one netblock: the netblock that has
gone longest without a request. A past, active or permanent ban is dropped
alike. A scanner whose ban was dropped that way is refused and banned again by
its next probe. A ban whose cause is `admin` is never dropped and does not count
toward `SWW_WAF_MAX_BANS`, so such bans can never fill the table.
for broken limits and clear signs of attack, it drops the earliest such ban of
one netblock: the netblock that has gone longest without a request. A past,
active or permanent ban is dropped alike. A scanner whose ban was dropped that
way is refused and banned again by its next probe. Bans whose cause is `admin`
or `abuseipdb` do not count toward `SWW_WAF_MAX_BANS` and are never dropped to
make room, so they can never push out the others.
A request refused under a ban that runs out is logged with the action
`temporarily_banned`. Under a permanent ban the action is `permanently_banned`.
@@ -2038,6 +2045,26 @@ host, sharing one address, would often fetch too close together.
- AbuseIPDB answers other than `200`;
- the answer gives no score;
- no answer comes within `SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS`.
- While `SWW_WAF_ABUSEIPDB_KEY` is set, AbuseIPDB's blacklist is fetched at the
start and every `SWW_WAF_ABUSEIPDB_BLACKLIST_REFRESH_INTERVAL_MINS` minutes
(default `60`). It lists the clients AbuseIPDB scores at
`SWW_WAF_ABUSEIPDB_THRESHOLD` (default `90`) or more. The blacklist itself is
not kept.
- Each client on it is banned for `SWW_WAF_ABUSEIPDB_BAN_DAYS` days (default
`30`), with the cause `abuseipdb`. The ban covers the client's netblock,
as any ban does.
- A netblock that already has an active ban is left as it is. So is one
whose `abuseipdb` ban an admin lifted, until that ban would have ended.
- An `abuseipdb` ban that has ended is forgotten at the next of these:
- a fetch of the blacklist in `enforce` mode;
- a read of `bans.json`, at the start or after an admin's edit.
- In `observe` mode the blacklist makes no ban.
- The threshold takes 25 to 100. It should not be set below 50 to 75, since
lower scores list ordinary visitors. AbuseIPDB serves free accounts the
blacklist at the threshold 100, whatever is asked for.
- AbuseIPDB limits the fetches of the blacklist a day: 5 for a free account.
A fetch over the limit fails, and is logged and alerted. The default
refresh fetches it 24 times a day.
- `SWW_WAF_DNSBL_ABUSEIPDB_ACTION`, for DNS blocklist and AbuseIPDB hits, is
`limit:25` by default. Such verdicts are less certain than a blocklist, so a
listed client gets a quarter of every request and byte limit rather than a
@@ -2052,9 +2079,10 @@ host, sharing one address, would often fetch too close together.
- Alerts go to a webhook as JSON, and to Slack and ntfy as formatted messages.
`docs/alerts.md` lists the events, the JSON object and the messages.
`SWW_WAF_ALERT_EVENTS` chooses which of the seven event types are sent; the
hourly `summary` is sent whatever it says. The bans an admin makes raise no
alert. In `observe` mode, a request that would have made a ban still raises
the ban's alert, with `mode` set to `observe` in its `detail`.
hourly `summary` is sent whatever it says. The bans an admin makes, and the
bans for AbuseIPDB's blacklist, raise no alert. In `observe` mode, a request
that would have made a ban still raises the ban's alert, with `mode` set to
`observe` in its `detail`.
- A repeat is an alert with the same event, and the same netblock, source, file
or anomaly scope, as one sent less than `SWW_WAF_ALERT_COOLDOWN` before. An
anomaly scope is one of the things counted under "Anomaly thresholds". A
@@ -2202,7 +2230,7 @@ lists the files, what each holds, and their sizes.
these stop the process with a message naming the file and position:
- a file that does not parse, or has an unknown `version`;
- an entry without a field it needs;
- a ban whose cause is not one of the four;
- a ban whose cause is not `limit`, `attack`, `admin` or `abuseipdb`;
- a ban whose `expires` is not after its `start`.
Starting with empty state would silently forgive every repeat offender.