sneak/smallwebwaf
1
0
Fork 0
Code Issues 44 Pull Requests 1 Actions 48 Packages Projects Releases Wiki Activity

Compare commits

base: sneak/smallwebwaf:next
Branches Tags
sneak/smallwebwaf:main sneak/smallwebwaf:next
..
compare: sneak/smallwebwaf:acde5bde0745648da96d5181348a721d0cdb08df
Branches Tags
sneak/smallwebwaf:next sneak/smallwebwaf:main
1 Commits
next .. acde5bde07
Author SHA1 Message Date
clawbot acde5bde07 Anomaly thresholds: alerts for unusual traffic, nothing refused (closes #101)
check / check (push) Waiting to run
Details 
SWWAF_ANOMALY_CLIENT_*, _NET_*, _ASN_*, _TOTAL_* and SWWAF_WATCH_* with
SWWAF_WATCH_NETS: requests and bytes per minute and per hour, each off by
default; with all off, nothing is counted. Otherwise every request but the
health check is counted, allow-listed and exempt ones included; a count
over its threshold raises an anomaly alert, with a cooldown per scope. At
most 20,000 counters, kept in alerts.json. A per-AS-number threshold with
lookups off, or a malformed SWWAF_WATCH_NETS, stops the start. A cooldown
that has run out is dropped as the hour ends, whatever it held back; the
hour's summary gives its repeats.

Judgement call: refused requests are counted too.
Judgement call: per-client counters are kept in alerts.json, which SPEC.md does not list.
Judgement call: a request counts for an AS number only if the lookup answered before it ended.

Model: opus-5-5
 2026-10-07 14:04:15 +00:00
41 changed files with 431 additions and 6861 deletions
Expand all files Collapse all files
README.md
+197 -558
View File
@@ -26,39 +26,29 @@ Slack and ntfy, and remote log sending. So is the stage after that: the AS
number and country of every client, looked up through GeoJS or in the IPinfo number and country of every client, looked up through GeoJS or in the IPinfo
Lite database file, the byte limits, the biased thresholds, lower limits for the Lite database file, the byte limits, the biased thresholds, lower limits for the
AS numbers and countries you list, and the anomaly thresholds, alerts for AS numbers and countries you list, and the anomaly thresholds, alerts for
unusual traffic that refuse nothing. So are the first three parts of the stage unusual traffic that refuse nothing. `smallwebwaf` passes each request to the
after that: the blocklists you name by URL, which it fetches and keeps, with a app and the app's answer back, unchanged, within its timeouts and size limits,
file of AS numbers' percentages fetched the same way, the DNS blocklists (DNSBL works out each client's address, looks up its AS number and country unless you
zones), which it asks about each client in the background, and AbuseIPDB, which switch that off, bans a client that sends too many requests or too many bytes,
it asks in the background about each client that has committed an offence. not counting those for the paths you choose, with lower limits for the clients
`smallwebwaf` passes each request to the app and the app's answer back, of the AS numbers and countries you list, refuses a client that comes from a
unchanged, within its timeouts and size limits, works out each client's address, country you refuse or from a network you refuse, lets the networks you choose
looks up its AS number and country unless you switch that off, bans a client through, checks each request against the rule files and bans a client whose
that sends too many requests or too many bytes, not counting those for the paths request is a clear sign of attack, keeps its bans, each client's counters and
you choose, with lower limits for the clients of the AS numbers and countries history, and GeoJS's answers in JSON files across restarts, takes in your edits
you list, refuses a client that comes from a country you refuse or from a of those files, such as a ban you make, keep or lift, and of the rule files
network you refuse, refuses, limits or only notes a client a blocklist or a while it runs, writes a JSON log line for every request, sends its log lines to
DNSBL zone you name lists, or AbuseIPDB scores at or over the score you set, a syslog server too if you name one, sends an alert to a webhook, to Slack and
lets the networks you choose through, checks each request against the rule files to ntfy, each if you name one, for each ban it makes or makes permanent, for
and bans a client whose request is a clear sign of attack, keeps its bans, each traffic over an anomaly threshold you set, for GeoJS failing, for a rule file or
client's counters and history, GeoJS's answers, the last good copy of each list state file with an error and for a replacement of the lookup database it cannot
it fetches, the DNSBL zones' verdicts, AbuseIPDB's scores and the AbuseIPDB read, serves Prometheus metrics to a scraper that holds the metrics token, lets
checks spent today in JSON files across restarts, takes in your edits of those an admin who holds the admin token list, add and lift bans and ask what it knows
files, such as a ban you make, keep or lift, and of the rule files while it of a client, and in `observe` mode passes on the requests it would refuse,
runs, writes a JSON log line for every request, sends its log lines to a syslog logging what it would have done with them. It comes as the image the app's own
server too if you name one, sends an alert to a webhook, to Slack and to ntfy, image is built on. The rest of the design comes after that, in the order of the
each if you name one, for each ban it makes or makes permanent, for traffic over build order in [`SPEC.md`](SPEC.md). The survey of existing tools that led to
an anomaly threshold you set, for a client a blocklist, a DNSBL zone or the design is in [`EVALUATION.md`](EVALUATION.md).
AbuseIPDB lists, for GeoJS failing, a list it cannot fetch, a DNSBL zone or
AbuseIPDB that fails or refuses a query and the day's AbuseIPDB checks used up,
for a rule file or state file with an error and for a replacement of the lookup
database it cannot read, serves Prometheus metrics to a scraper that holds the
metrics token, lets an admin who holds the admin token list, add and lift bans
and ask what it knows of a client, and in `observe` mode passes on the requests
it would refuse, logging what it would have done with them. It comes as the
image the app's own image is built on. The rest of the design comes after that,
in the order of the build order in [`SPEC.md`](SPEC.md). The survey of existing
tools that led to the design is in [`EVALUATION.md`](EVALUATION.md).
## Getting started ## Getting started
@@ -116,12 +106,11 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
`SWWAF_RATE_LIMIT_EXEMPT_PATHS`, as that setting below describes, is neither `SWWAF_RATE_LIMIT_EXEMPT_PATHS`, as that setting below describes, is neither
counted nor refused by the rate limits; the static lists, bans, the country counted nor refused by the rate limits; the static lists, bans, the country
lists and the rule files still apply to it. A client is one IPv4 address, or lists and the rule files still apply to it. A client is one IPv4 address, or
one IPv6 group, the netblock of `SWWAF_IPV6_GROUP_PREFIX` its address is in, a one IPv6 /64, since one abuser usually holds a whole /64. Each window is
/64 by default, since one abuser usually holds a whole /64. Each window is
counted in two fixed buckets, the earlier one weighted by how much of it the counted in two fixed buckets, the earlier one weighted by how much of it the
window still covers. At most `SWWAF_MAX_TRACKED_CLIENTS` clients are kept, window still covers. At most 20,000 clients are kept, the least recently seen
20,000 by default, the least recently seen dropped first, with their history, dropped first, with their history, and a restart gives no client a fresh
and a restart gives no client a fresh allowance (see "State files" below). allowance (see "State files" below).
- Counts each client's bytes over a minute, an hour and a day, in the same way: - Counts each client's bytes over a minute, an hour and a day, in the same way:
once a request passed to the app has ended, the body bytes of its answer, of once a request passed to the app has ended, the body bytes of its answer, of
the request, or of both, as `SWWAF_BYTES_COUNT` says. For a WebSocket, or any the request, or of both, as `SWWAF_BYTES_COUNT` says. For a WebSocket, or any
@@ -135,27 +124,25 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
or `SWWAF_RATE_LIMIT_EXEMPT_NETS`, and a request for a path or `SWWAF_RATE_LIMIT_EXEMPT_NETS`, and a request for a path
`SWWAF_RATE_LIMIT_EXEMPT_PATHS` exempts. `SWWAF_RATE_LIMIT_EXEMPT_PATHS` exempts.
- Gives the clients of the AS numbers and countries the biased thresholds list, - Gives the clients of the AS numbers and countries the biased thresholds list,
`SWWAF_ASN_LIMIT_PERCENT`, the file `SWWAF_ASN_LIMIT_PERCENT_URL` names and `SWWAF_ASN_LIMIT_PERCENT` and `SWWAF_COUNTRY_LIMIT_PERCENT`, the percentage
`SWWAF_COUNTRY_LIMIT_PERCENT`, the percentage they give of every rate limit they give of every rate limit and byte limit, so that the same rules ban them
and byte limit, so that the same rules ban them after fewer requests, and, after fewer requests, and, while `SWWAF_UNKNOWN_LIMIT_PERCENT` is below 100,
while `SWWAF_UNKNOWN_LIMIT_PERCENT` is below 100, every client without a every client without a country that percentage. A client to which several
country that percentage. A client to which several apply gets the lowest. For apply gets the lowest. `SWWAF_ASN_BYTES_PERCENT` and
the byte limits, `SWWAF_ASN_BYTES_PERCENT` gives an AS number it lists a `SWWAF_COUNTRY_BYTES_PERCENT` give the AS numbers and countries they list a
percentage in place of those `SWWAF_ASN_LIMIT_PERCENT` and the file give it, percentage of the byte limits in place of the other two. Each client is
and `SWWAF_COUNTRY_BYTES_PERCENT` gives a country it lists one in place of the counted on its own, against its own lowered limits: no budget is shared by a
one `SWWAF_COUNTRY_LIMIT_PERCENT` gives it. Each client is counted on its own, whole AS number or country, which one abuser could use up and so lock out
against its own lowered limits: no budget is shared by a whole AS number or everyone else there. The log line of each request the rate limits count gives
country, which one abuser could use up and so lock out everyone else there. its client's percentages below 100 and the settings that gave them, and so do
The log line of each request the rate limits count gives its client's the notes of a ban for a lowered limit, and its alert.
percentages below 100 and the settings that gave them, and so do the notes of
a ban for a lowered limit, and its alert.
- Bans a client that breaks a rate limit or a byte limit, as "Bans" in - Bans a client that breaks a rate limit or a byte limit, as "Bans" in
[`SPEC.md`](SPEC.md) describes: the first ban lasts an hour, and a limit [`SPEC.md`](SPEC.md) describes: the first ban lasts an hour, and a limit
broken again within a day of a ban ending bans for three times as long as that broken again within a day of a ban ending bans for three times as long as that
ban, so 1, 3, 9, 27 and 81 hours; a ban that would last longer than seven days ban, so 1, 3, 9, 27 and 81 hours; a ban that would last longer than seven days
is permanent instead. A ban covers the client's netblock: its IPv4 address, or is permanent instead. A ban covers the client's netblock: its IPv4 address, or
the netblock around it that `SWWAF_BAN_SCOPE_V4_PREFIX` sets, or its IPv6 the netblock around it that `SWWAF_BAN_SCOPE_V4_PREFIX` sets, or its IPv6 /64.
group. While it lasts, every request from the netblock is refused with While it lasts, every request from the netblock is refused with
`SWWAF_BAN_RESPONSE` after the static lists and before the country lists, so `SWWAF_BAN_RESPONSE` after the static lists and before the country lists, so
the client is not looked up, and is not counted for the rate limits. A ban the client is not looked up, and is not counted for the rate limits. A ban
sets the client's counters back to zero. Each ban carries notes for deciding sets the client's counters back to zero. Each ban carries notes for deciding
@@ -204,68 +191,35 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
link-local address has no country and is never looked up: link-local address has no country and is never looked up:
`SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses it unless it is in `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses it unless it is in
`SWWAF_ALLOW_NETS`, and `SWWAF_DENIED_COUNTRIES` does not refuse it. `SWWAF_ALLOW_NETS`, and `SWWAF_DENIED_COUNTRIES` does not refuse it.
- Checks the client's own address against the blocklists `SWWAF_BLOCKLIST_URLS`
names, after the country lists and before the rate limits (see "Blocklists"
below). With `SWWAF_BLOCKLIST_ACTION` at `deny`, its default, a request from a
client a blocklist lists is refused with `SWWAF_BAN_RESPONSE` before its body
is read; it is not counted for the rate limits, and makes no ban. With
`limit:<percent>` the client gets that percentage of every rate limit and byte
limit, the lowest of its percentages applying, as for a biased threshold, and
with `log` nothing more is done. Whatever the action, the request's log line
names the lists, and each raises an alert.
- Checks the client's own address against the DNSBL zones `SWWAF_DNSBL_ZONES`
names, after the blocklists and before the rate limits (see "DNS blocklists"
below), by the verdicts it keeps. A zone is asked about a client in the
background, and no request waits for its answer: the request that has it
asked, and any other from the client before the answer comes, goes on as from
a client the zone does not list. With `SWWAF_REPUTATION_ACTION` at `limit:25`,
its default, a client a zone's verdict lists gets a quarter of every rate
limit and byte limit, the lowest of its percentages applying, as for a biased
threshold. With `deny`, its requests are refused with `SWWAF_BAN_RESPONSE`
before their bodies are read; they are not counted for the rate limits, and
make no ban. With `log`, nothing more is done. Whatever the action, the
request's log line names the zones, and each raises an alert. A client a
blocklist refuses is not checked.
- Checks the client with AbuseIPDB while `SWWAF_ABUSEIPDB_KEY` is set, after the
DNSBL zones and before the rate limits (see "AbuseIPDB" below), by the scores
it keeps. Only a client whose history counts an offence is checked, so far one
that has broken a rate limit or a byte limit, matched a ban rule, or had a
request refused by a block rule, and only in the background, so that no
request waits for AbuseIPDB. A score at or over `SWWAF_ABUSEIPDB_MIN_SCORE` is
a hit, and `SWWAF_REPUTATION_ACTION` does with its client what it does with
one a DNSBL zone's verdict lists. The request's log line names AbuseIPDB, and
it raises an alert. A client a blocklist or a DNSBL zone refuses is not
checked.
- Checks the client's own address against the static lists, the three netblock - Checks the client's own address against the static lists, the three netblock
settings below, before anything else, its lookup included. A client in settings below, before anything else, its lookup included. A client in
`SWWAF_ALLOW_NETS` skips bans, the country lists, the blocklists, the DNSBL `SWWAF_ALLOW_NETS` skips bans, the country lists, the rate limits, the byte
zones, AbuseIPDB, the rate limits, the byte limits and the rule files, and is limits and the rule files, and is not looked up; the timeouts and size limits
not looked up; the timeouts and size limits still apply. A client in still apply. A client in `SWWAF_DENY_NETS` is refused with
`SWWAF_DENY_NETS` is refused with `SWWAF_BAN_RESPONSE` before its body is `SWWAF_BAN_RESPONSE` before its body is read, and the request is not counted
read, and the request is not counted for the rate limits; an address in for the rate limits; an address in `SWWAF_ALLOW_NETS` too is let through. A
`SWWAF_ALLOW_NETS` too is let through. A client in client in `SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither counted nor refused by the
`SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither counted nor refused by the rate rate limits, and has no bytes counted by the byte limits; the country lists,
limits, and has no bytes counted by the byte limits; the country lists, the the rule files and bans still apply to it.
rule files and bans still apply to it.
- In `observe` mode, with `SWWAF_MODE=observe`, refuses none of the requests - In `observe` mode, with `SWWAF_MODE=observe`, refuses none of the requests
that `SWWAF_DENY_NETS`, a ban, the country lists, a blocklist, a DNSBL zone's that `SWWAF_DENY_NETS`, a ban, the country lists, a rate limit or a rule would
verdict, AbuseIPDB's score, a rate limit or a rule would refuse: it passes refuse: it passes them to the app, and their log lines name what `enforce`
them to the app, and their log lines name what `enforce` mode would have done mode would have done (see `would_action` in "Request log" below). The checks
(see `would_action` in "Request log" below). The checks run, and requests and run, and requests and bytes are counted, as in `enforce` mode, with three
bytes are counted, as in `enforce` mode, with three differences: neither a differences: neither a broken rate limit or byte limit nor a `ban` rule makes
broken rate limit or byte limit nor a `ban` rule makes a ban; a broken limit a ban; a broken limit does not set the client's counters back to zero, so each
does not set the client's counters back to zero, so each request over a rate request over a rate limit is logged as one that would be refused, and each
limit is logged as one that would be refused, and each whose bytes keep the whose bytes keep the client over a byte limit as breaking it; and a request
client over a byte limit as breaking it; and a request under a ban does not under a ban does not make it permanent. As in `enforce` mode, the bytes
make it permanent. As in `enforce` mode, the bytes counted are only those of counted are only those of the requests `enforce` mode would have passed to the
the requests `enforce` mode would have passed to the app. A ban it would have app. A ban it would have made, or made permanent, raises the alert `enforce`
made, or made permanent, raises the alert `enforce` mode would have raised, mode would have raised, marked as what would have happened (see "Alerts"
marked as what would have happened (see "Alerts" below). The bans in below). The bans in `bans.json` are kept, and refuse requests again when
`bans.json` are kept, and refuse requests again when `smallwebwaf` next runs `smallwebwaf` next runs in `enforce` mode, as long as they last. The timeouts
in `enforce` mode, as long as they last. The timeouts and size limits still and size limits still apply, since they protect `smallwebwaf` and the app
apply, since they protect `smallwebwaf` and the app themselves, and a request themselves, and a request for one of `smallwebwaf`'s own endpoints without its
for one of `smallwebwaf`'s own endpoints without its token is still answered token is still answered `401`. It is for trying a configuration before
`401`. It is for trying a configuration before enforcing it. enforcing it.
- Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any - Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any
check and without asking the app, for the image's health check. check and without asking the app, for the image's health check.
- Answers `GET /_smallwebwaf/metrics` with its metrics (see "Metrics" below) for - Answers `GET /_smallwebwaf/metrics` with its metrics (see "Metrics" below) for
@@ -285,15 +239,13 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
`SWWAF_LOG_REMOTE_URL` names one (see "Sending the log to a syslog server" `SWWAF_LOG_REMOTE_URL` names one (see "Sending the log to a syslog server"
below). below).
- Sends an alert for each ban it makes or makes permanent, for a count over an - Sends an alert for each ban it makes or makes permanent, for a count over an
anomaly threshold, for a client a blocklist, a DNSBL zone or AbuseIPDB lists, anomaly threshold, for GeoJS failing, for a rule file or state file with an
for GeoJS failing, a list it cannot fetch, a DNSBL zone or AbuseIPDB that error, and for a replacement of the lookup database it cannot read, holding
fails or refuses a query, and the day's AbuseIPDB checks used up, for a rule back repeats and, past an hourly limit, rolling the rest into one summary, to
file or state file with an error, and for a replacement of the lookup database each destination you name: as a JSON object to the webhook
it cannot read, holding back repeats and, past an hourly limit, rolling the `SWWAF_ALERT_WEBHOOK_URL` names, as a message to the Slack incoming webhook
rest into one summary, to each destination you name: as a JSON object to the `SWWAF_ALERT_SLACK_WEBHOOK_URL` names, and as a message to the ntfy topic
webhook `SWWAF_ALERT_WEBHOOK_URL` names, as a message to the Slack incoming `SWWAF_ALERT_NTFY_URL` names (see "Alerts" below).
webhook `SWWAF_ALERT_SLACK_WEBHOOK_URL` names, and as a message to the ntfy
topic `SWWAF_ALERT_NTFY_URL` names (see "Alerts" below).
- Counts requests and their bytes over a minute and an hour, per client, per - Counts requests and their bytes over a minute and an hour, per client, per
netblock around a client, per AS number, for the whole service and per named netblock around a client, per AS number, for the whole service and per named
netblock, and sends an `anomaly` alert for a count over the anomaly threshold netblock, and sends an `anomaly` alert for a count over the anomaly threshold
@@ -316,8 +268,7 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
Each setting is an environment variable, or a file one names (see "Settings Each setting is an environment variable, or a file one names (see "Settings
given as files" below), and each has a default, so none has to be set. A setting given as files" below), and each has a default, so none has to be set. A setting
that is set but invalid stops the start with a message naming it, and the that is set but invalid stops the start with a message naming it, and the
effective settings are logged at start, unless `SWWAF_LOG_LEVEL` is `warn` or effective settings are logged at start.
`error`, which hold that line back.
- `SWWAF_LISTEN_ADDR` (default `:8080`): where `smallwebwaf` listens. - `SWWAF_LISTEN_ADDR` (default `:8080`): where `smallwebwaf` listens.
- `SWWAF_UPSTREAM_URL` (default `http://127.0.0.1:8081`): the app, as `http` or - `SWWAF_UPSTREAM_URL` (default `http://127.0.0.1:8081`): the app, as `http` or
@@ -335,18 +286,6 @@ effective settings are logged at start, unless `SWWAF_LOG_LEVEL` is `warn` or
- `SWWAF_TRUSTED_PROXIES` (default `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`, - `SWWAF_TRUSTED_PROXIES` (default `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`,
the private address ranges): the netblocks whose `X-Forwarded-For` is the private address ranges): the netblocks whose `X-Forwarded-For` is
believed. A list given replaces the default; set but empty, it trusts nothing. believed. A list given replaces the default; set but empty, it trusts nothing.
- `SWWAF_IPV6_GROUP_PREFIX` (default `64`): the length of an IPv6 client's
group, the netblock that is one client, from 32 to 128, since a shorter one
would make one client of the customers of several providers. The rate limits,
the byte limits, bans, the table of clients, the lookups, AbuseIPDB's scores
and the anomaly thresholds per client all take an IPv6 client as its group,
and `client_group` gives it. After it changes, each IPv6 client starts afresh:
what was kept of it under its earlier group, its counts, history, GeoJS answer
and AbuseIPDB score, is not used for it, while each ban keeps refusing its
netblock until it ends.
- `SWWAF_MAX_TRACKED_CLIENTS` (default `20000`): the most clients kept in memory
and in `clients.json`, with their counters and history, a whole number above
zero. Past it, the least recently seen is dropped first.
- `SWWAF_CLIENT_REQUEST_TIMEOUT` (default `60s`): how long a client may take to - `SWWAF_CLIENT_REQUEST_TIMEOUT` (default `60s`): how long a client may take to
send its request line and headers, and then, from the end of the headers, its send its request line and headers, and then, from the end of the headers, its
body. body.
@@ -369,9 +308,8 @@ effective settings are logged at start, unless `SWWAF_LOG_LEVEL` is `warn` or
- `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body. - `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body.
- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body. - `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body.
- `SWWAF_ALLOW_NETS` (default empty): netblocks whose clients skip bans, the - `SWWAF_ALLOW_NETS` (default empty): netblocks whose clients skip bans, the
country lists, the blocklists, the DNSBL zones, AbuseIPDB, the rate limits, country lists, the rate limits, the byte limits and the rule files, such as
the byte limits and the rule files, such as your monitoring or your own your monitoring or your own networks.
networks.
- `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients the - `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients the
rate limits and the byte limits do not apply to, such as a machine that talks rate limits and the byte limits do not apply to, such as a machine that talks
to the app all day. to the app all day.
@@ -411,10 +349,9 @@ effective settings are logged at start, unless `SWWAF_LOG_LEVEL` is `warn` or
`SWWAF_LOOKUP_DB_PATH` names, or `off`, which looks up no client and sends no `SWWAF_LOOKUP_DB_PATH` names, or `off`, which looks up no client and sends no
address to GeoJS. With `off`, a country list that is not empty, address to GeoJS. With `off`, a country list that is not empty,
`SWWAF_ADD_LOOKUP_HEADERS` set to `true`, a biased threshold that lowers a `SWWAF_ADD_LOOKUP_HEADERS` set to `true`, a biased threshold that lowers a
limit, a list of them that is not empty, `SWWAF_ASN_LIMIT_PERCENT_URL` set or limit, a list of them that is not empty or `SWWAF_UNKNOWN_LIMIT_PERCENT` below
`SWWAF_UNKNOWN_LIMIT_PERCENT` below 100, or an anomaly threshold per AS number 100, or an anomaly threshold per AS number that is not `off`, stops the start,
that is not `off`, stops the start, with a message naming it and with a message naming it and `SWWAF_LOOKUP_SOURCE`.
`SWWAF_LOOKUP_SOURCE`.
- `SWWAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in its - `SWWAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in its
`.mmdb` form, for `SWWAF_LOOKUP_SOURCE=file`. `file` without it, or it with `.mmdb` form, for `SWWAF_LOOKUP_SOURCE=file`. `file` without it, or it with
any other `SWWAF_LOOKUP_SOURCE`, the default included, stops the start, with a any other `SWWAF_LOOKUP_SOURCE`, the default included, stops the start, with a
@@ -452,64 +389,11 @@ effective settings are logged at start, unless `SWWAF_LOG_LEVEL` is `warn` or
client without a country gets: one the lookup cannot place, one on a private, client without a country gets: one the lookup cannot place, one on a private,
loopback or link-local address, which is never looked up, and one whose answer loopback or link-local address, which is never looked up, and one whose answer
from GeoJS has not come in time. from GeoJS has not come in time.
- `SWWAF_ASN_LIMIT_PERCENT_URL` (default unset): the `http` or `https` URL of a
file of AS numbers, each with its percentage of every rate limit and byte
limit, one such as `AS14061:50` to a line, whose percentages count as those of
`SWWAF_ASN_LIMIT_PERCENT` do, so that one list can serve several instances.
For an AS number both give a percentage, the lower applies, and an AS number
the file lists twice gets the lower of its two. It is fetched and kept as a
blocklist is (see "Blocklists" below). A URL `SWWAF_BLOCKLIST_URLS` names too
stops the start.
- `SWWAF_BLOCKLIST_URLS` (default empty): the blocklists, as `http` or `https`
URLs without a user or a fragment, such as
`https://www.spamhaus.org/drop/drop.txt` (see "Blocklists" below). A URL
listed twice stops the start.
- `SWWAF_BLOCKLIST_REFRESH` (default `24h`): how long after a list was last
fetched, or a fetch of it failed, it is fetched again, for the blocklists and
`SWWAF_ASN_LIMIT_PERCENT_URL`. Less than `1h` stops the start: the Spamhaus
lists may be fetched no more than once an hour.
- `SWWAF_BLOCKLIST_ACTION` (default `deny`): what is done with a client a
blocklist lists: `deny` refuses its requests with `SWWAF_BAN_RESPONSE`,
`limit:<percent>`, such as `limit:25`, gives it that percentage of every rate
limit and byte limit, and `log` does nothing more than note the lists in the
log line and raise the alert.
- `SWWAF_DNSBL_ZONES` (default empty): the DNSBL zones each client is asked
about, such as `dnsbl.dronebl.org` (see "DNS blocklists" below). A zone is a
DNS name of at most 189 characters: labels of letters, digits and hyphens, of
up to 63 characters each, neither starting nor ending with a hyphen, joined by
dots. A Spamhaus zone is the name of its keyed query service, with the key in
it, such as `<key>.xbl.dq.spamhaus.net`, and is shown with its key masked (see
"DNS blocklists" below). A zone that is not such a name stops the start, as
does one listed twice, even with its letters in another case or with another
key. Do not name a zone meant for mail (see "DNS blocklists" below).
- `SWWAF_DNSBL_RESOLVER` (default unset): the resolver the zones are asked
through, an IP address with an optional port, 53 when none is given, such as
`192.0.2.53` or `[2001:db8::53]:5353`. Unset, it is the host's, as
`/etc/resolv.conf` names it. Several zones refuse queries that come through a
public resolver.
- `SWWAF_ABUSEIPDB_KEY` (default unset): the key of your AbuseIPDB account,
which clients are checked with (see "AbuseIPDB" below). While it is unset, no
client is checked. The settings logged at start show `********` in its place.
- `SWWAF_ABUSEIPDB_MIN_SCORE` (default `75`): the least abuse confidence score,
from 0 to 100, that is a hit.
- `SWWAF_ABUSEIPDB_DAILY_BUDGET` (default `900`): the most checks made in a day,
in UTC, a whole number above zero. AbuseIPDB's free accounts may make 1,000.
- `SWWAF_REPUTATION_ACTION` (default `limit:25`): what is done with a client a
zone's verdict lists, or whose AbuseIPDB score is a hit, as
`SWWAF_BLOCKLIST_ACTION` is for a blocklist: `deny`, `limit:<percent>` or
`log`. Such a verdict is less certain than a list such as DROP, so by default
its client gets a quarter of every rate limit and byte limit.
- `SWWAF_REPUTATION_CACHE_TTL` (default `24h`): how long a zone's verdict on a
client, or AbuseIPDB's score of it, is used after it was given.
- `SWWAF_REPUTATION_TIMEOUT` (default `2s`): how long a query to a zone, or a
check with AbuseIPDB, may take before it fails.
- `SWWAF_BAN_RESPONSE` (default `403`): how a refused client is answered, one - `SWWAF_BAN_RESPONSE` (default `403`): how a refused client is answered, one
that is banned, breaks a rate limit, matches a `ban` rule, is in that is banned, breaks a rate limit, matches a `ban` rule, is in
`SWWAF_DENY_NETS`, comes from a refused country, is in a blocklist while `SWWAF_DENY_NETS` or comes from a refused country: `403`, `429`, or `close` to
`SWWAF_BLOCKLIST_ACTION` is `deny` or is listed by a DNSBL zone or AbuseIPDB close the connection without an answer. Behind traefik, `close` does not leave
while `SWWAF_REPUTATION_ACTION` is `deny`: `403`, `429`, or `close` to close the client unanswered: traefik answers `502`, as it does whenever its backend
the connection without an answer. Behind traefik, `close` does not leave the
client unanswered: traefik answers `502`, as it does whenever its backend
drops a connection. A `block` rule always answers `403`. drops a connection. A `block` rule always answers `403`.
- `SWWAF_LIMIT_BAN_DURATION` (default `1h`): the ban for a first broken rate - `SWWAF_LIMIT_BAN_DURATION` (default `1h`): the ban for a first broken rate
limit or byte limit. limit or byte limit.
@@ -524,7 +408,7 @@ effective settings are logged at start, unless `SWWAF_LOG_LEVEL` is `warn` or
kept, past, active and permanent. The bans you make or keep are kept besides. kept, past, active and permanent. The bans you make or keep are kept besides.
- `SWWAF_BAN_SCOPE_V4_PREFIX` (default `32`): the length of the netblock around - `SWWAF_BAN_SCOPE_V4_PREFIX` (default `32`): the length of the netblock around
an IPv4 client that a ban covers, such as `24` to ban the surrounding /24. An an IPv4 client that a ban covers, such as `24` to ban the surrounding /24. An
IPv6 ban covers the client's group, as `SWWAF_IPV6_GROUP_PREFIX` sets it. IPv6 ban covers the client's /64.
- `SWWAF_STATE_DIR` (default `/var/lib/smallwebwaf`): the directory of the state - `SWWAF_STATE_DIR` (default `/var/lib/smallwebwaf`): the directory of the state
files, an absolute path. A directory `smallwebwaf` cannot write stops the files, an absolute path. A directory `smallwebwaf` cannot write stops the
start. start.
@@ -539,10 +423,6 @@ effective settings are logged at start, unless `SWWAF_LOG_LEVEL` is `warn` or
(see "Request log" below). An entry naming `Host` or `Transfer-Encoding` stops (see "Request log" below). An entry naming `Host` or `Transfer-Encoding` stops
the start, since Go's HTTP server takes both out of the request; the request's the start, since Go's HTTP server takes both out of the request; the request's
host is the field `host`. host is the field `host`.
- `SWWAF_LOG_LEVEL` (default `info`): the least severe of `smallwebwaf`'s own
messages that are written, on stdout and to the syslog server: `debug`,
`info`, `warn` or `error`. No message is at `debug` yet, so it writes what
`info` does. It holds back no line of the request log.
- `SWWAF_ADMIN_TOKEN` (default unset): the token an admin sends for the ban - `SWWAF_ADMIN_TOKEN` (default unset): the token an admin sends for the ban
endpoints and `/_smallwebwaf/clients/<ip>` (see "Admin endpoints" below), a endpoints and `/_smallwebwaf/clients/<ip>` (see "Admin endpoints" below), a
long random value. While it is unset they are off; one shorter than 32 long random value. While it is unset they are off; one shorter than 32
@@ -606,8 +486,8 @@ effective settings are logged at start, unless `SWWAF_LOG_LEVEL` is `warn` or
`SWWAF_INSTANCE_NAME`, which ntfy is sent in the title. `SWWAF_INSTANCE_NAME`, which ntfy is sent in the title.
- `SWWAF_ALERT_EVENTS` (default - `SWWAF_ALERT_EVENTS` (default
`ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure,file_error`): `ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure,file_error`):
the events alerts are sent for. `waf_block` comes with the Core Rule Set; the events alerts are sent for. `waf_block` and `reputation_hit` come with the
nothing raises it yet. features that raise them; nothing raises them yet.
- `SWWAF_ALERT_COOLDOWN` (default `15m`): how long a repeat of an alert is held - `SWWAF_ALERT_COOLDOWN` (default `15m`): how long a repeat of an alert is held
back (see "Alerts" below). back (see "Alerts" below).
- `SWWAF_ALERT_MAX_PER_HOUR` (default `60`): the most alerts sent in an hour; - `SWWAF_ALERT_MAX_PER_HOUR` (default `60`): the most alerts sent in an hour;
@@ -655,19 +535,15 @@ Percentages are whole numbers from 0 to 100, and an entry of a list of them is
an AS number or a country, `:` and a percentage; an AS number or a country an AS number or a country, `:` and a percentage; an AS number or a country
listed twice in one of them stops the start. `off` switches a timeout, a size listed twice in one of them stops the start. `off` switches a timeout, a size
limit, a rate limit, a byte limit, an anomaly threshold, `SWWAF_ALERT_COOLDOWN` limit, a rate limit, a byte limit, an anomaly threshold, `SWWAF_ALERT_COOLDOWN`
or `SWWAF_ALERT_MAX_PER_HOUR` off; `SWWAF_IPV6_GROUP_PREFIX`, or `SWWAF_ALERT_MAX_PER_HOUR` off; `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`,
`SWWAF_MAX_TRACKED_CLIENTS`, `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, `SWWAF_LOOKUP_TIMEOUT`, `SWWAF_UNKNOWN_LIMIT_PERCENT`, the ban settings, the
`SWWAF_LOOKUP_TIMEOUT`, `SWWAF_UNKNOWN_LIMIT_PERCENT`, state settings, `SWWAF_METRICS_TOP_N`, `SWWAF_LOG_REMOTE_BUFFER`,
`SWWAF_BLOCKLIST_REFRESH`, `SWWAF_ABUSEIPDB_MIN_SCORE`, `SWWAF_ANOMALY_NET_V4_PREFIX` and `SWWAF_ANOMALY_NET_V6_PREFIX` cannot be off.
`SWWAF_ABUSEIPDB_DAILY_BUDGET`, `SWWAF_REPUTATION_CACHE_TTL`,
`SWWAF_REPUTATION_TIMEOUT`, the ban settings, the state settings,
`SWWAF_METRICS_TOP_N`, `SWWAF_LOG_REMOTE_BUFFER`, `SWWAF_ANOMALY_NET_V4_PREFIX`
and `SWWAF_ANOMALY_NET_V6_PREFIX` cannot be off.
Several limits are fixed rather than settings. At most 100,000 answers from Several limits are fixed rather than settings. At most 20,000 clients are kept,
GeoJS are kept, for 7 days each, at most 20,000 anomaly counters, at most with their counters and history, and an IPv6 client is counted by its /64. At
100,000 verdicts of the DNSBL zones, with at most 1,000 queries to them under most 100,000 answers from GeoJS are kept, for 7 days each, and at most 20,000
way at once, and at most 100,000 scores of AbuseIPDB. anomaly counters.
### Settings given as files ### Settings given as files
@@ -724,7 +600,7 @@ which every line has.
- `peer_ip` is the TCP peer, normally traefik. `forwarded_for` is the - `peer_ip` is the TCP peer, normally traefik. `forwarded_for` is the
`X-Forwarded-For` header as received, several lines of it joined with `, `. `X-Forwarded-For` header as received, several lines of it joined with `, `.
`client_group` is the client as the rate limits count it: its IPv4 address as `client_group` is the client as the rate limits count it: its IPv4 address as
a /32, or its IPv6 group, as `SWWAF_IPV6_GROUP_PREFIX` sets it. a /32, or the /64 of its IPv6 address.
- `asn`, `as_name` and `country` are the client's AS number, such as `AS64496`, - `asn`, `as_name` and `country` are the client's AS number, such as `AS64496`,
the name of that AS, and its country, as GeoJS or the lookup database gives the name of that AS, and its country, as GeoJS or the lookup database gives
them. Each is empty when `SWWAF_LOOKUP_SOURCE` is `off`, for a client in them. Each is empty when `SWWAF_LOOKUP_SOURCE` is `off`, for a client in
@@ -750,30 +626,24 @@ which every line has.
app's, as passed on, or those of `smallwebwaf`'s own answer. app's, as passed on, or those of `smallwebwaf`'s own answer.
- `request_bytes` and `response_bytes` count body bytes. - `request_bytes` and `response_bytes` count body bytes.
- `action` is `forward` for a request passed to the app, `denied` for one - `action` is `forward` for a request passed to the app, `denied` for one
refused because its client is in `SWWAF_DENY_NETS`, in a blocklist while refused because its client is in `SWWAF_DENY_NETS`, `banned` for one refused
`SWWAF_BLOCKLIST_ACTION` is `deny`, or listed by a DNSBL zone or AbuseIPDB because a ban covers its client or because it matched a `ban` rule, which bans
while `SWWAF_REPUTATION_ACTION` is `deny`, `banned` for one refused because a its client, `country_denied` for one refused for its client's country,
ban covers its client or because it matched a `ban` rule, which bans its
client, `country_denied` for one refused for its client's country,
`rate_limited` for one that broke a rate limit and banned its client, `rate_limited` for one that broke a rate limit and banned its client,
`rule_blocked` for one a `block` rule refused, `too_large` for a request or `rule_blocked` for one a `block` rule refused, `too_large` for a request or
response over its size limit, `timed_out` for one that ran out of time, response over its size limit, `timed_out` for one that ran out of time,
`upstream_error` when the app could not be reached or its answer broke off, `upstream_error` when the app could not be reached or its answer broke off,
and `admin` for one `smallwebwaf` answered at its own endpoint. and `admin` for one `smallwebwaf` answered at its own endpoint.
- `would_action` is there in `observe` mode for a request that - `would_action` is there in `observe` mode for a request that
`SWWAF_DENY_NETS`, a ban, the country lists, a blocklist, a DNSBL zone's `SWWAF_DENY_NETS`, a ban, the country lists, a rate limit or a rule would have
verdict, AbuseIPDB's score, a rate limit or a rule would have refused in refused in `enforce` mode, and names the action that refusal would have had:
`enforce` mode, and names the action that refusal would have had: `denied`, `denied`, `banned`, `country_denied`, `rate_limited` or `rule_blocked`.
`banned`, `country_denied`, `rate_limited` or `rule_blocked`. `action` then `action` then names what was done: `forward` for a request passed to the app,
names what was done: `forward` for a request passed to the app, and another and another action, such as `too_large`, for one a size or time limit refused.
action, such as `too_large`, for one a size or time limit refused.
- `limit_percent` is there for a request the rate limits count whose client a - `limit_percent` is there for a request the rate limits count whose client a
biased threshold, `SWWAF_BLOCKLIST_ACTION` for a blocklist that lists it, or biased threshold gives less than the whole of the rate limits, and gives the
`SWWAF_REPUTATION_ACTION` for a DNSBL zone whose verdict lists it or an percentage it gets, with `limit_percent_setting` naming the setting that gave
AbuseIPDB score that is a hit, gives less than the whole of the rate limits, it, such as `SWWAF_ASN_LIMIT_PERCENT`. `bytes_percent` and
and gives the percentage it gets, with `limit_percent_setting` naming the
setting that gave it, such as `SWWAF_ASN_LIMIT_PERCENT`, or
`SWWAF_ASN_LIMIT_PERCENT_URL` for the file it names. `bytes_percent` and
`bytes_percent_setting` are the same for the byte limits. Each is left out `bytes_percent_setting` are the same for the byte limits. Each is left out
when the client gets the whole of those limits. when the client gets the whole of those limits.
- `counts` gives the client's requests in the minute, the hour and the day as - `counts` gives the client's requests in the minute, the hour and the day as
@@ -783,12 +653,12 @@ which every line has.
broke it. It is left out for a request the rate limits do not count: the broke it. It is left out for a request the rate limits do not count: the
health check, one from a client in `SWWAF_ALLOW_NETS` or health check, one from a client in `SWWAF_ALLOW_NETS` or
`SWWAF_RATE_LIMIT_EXEMPT_NETS`, one for a path that `SWWAF_RATE_LIMIT_EXEMPT_NETS`, one for a path that
`SWWAF_RATE_LIMIT_EXEMPT_PATHS` exempts, and one that `SWWAF_DENY_NETS`, a `SWWAF_RATE_LIMIT_EXEMPT_PATHS` exempts, and one that `SWWAF_DENY_NETS`, a ban
ban, the country lists, a blocklist, a DNSBL zone's verdict or AbuseIPDB's or the country lists refuse, or would refuse in `observe` mode. Its
score refuse, or would refuse in `observe` mode. Its `minute_bytes`, `minute_bytes`, `hour_bytes` and `day_bytes` give the client's bytes in each
`hour_bytes` and `day_bytes` give the client's bytes in each window as the window as the byte limits count them, in the same way: for a request whose
byte limits count them, in the same way: for a request whose bytes they count, bytes they count, with its own, once it has ended; for any other, those
with its own, once it has ended; for any other, those counted before it. counted before it.
- `rule_ids` is there for a request that matched rules of the rule files, and - `rule_ids` is there for a request that matched rules of the rule files, and
lists their ids in the order they matched, up to the one that refused it. lists their ids in the order they matched, up to the one that refused it.
- `limit_hit` is there for a request that broke a rate limit, or whose bytes - `limit_hit` is there for a request that broke a rate limit, or whose bytes
@@ -798,18 +668,6 @@ which every line has.
several. `offence` is then `limit`. A request whose bytes broke a byte limit several. `offence` is then `limit`. A request whose bytes broke a byte limit
is not refused: its `action` is what it would have been otherwise, such as is not refused: its `action` is what it would have been otherwise, such as
`forward`. `forward`.
- `reputation` is there for a request whose client a blocklist or a DNSBL zone's
verdict lists, or whose AbuseIPDB score is a hit, and gives the URLs of the
blocklists that list it, in the order `SWWAF_BLOCKLIST_URLS` names them, then
the zones whose verdict lists it, in the order `SWWAF_DNSBL_ZONES` names them,
then `abuseipdb`, whatever `SWWAF_BLOCKLIST_ACTION` and
`SWWAF_REPUTATION_ACTION` say. A zone whose verdict on the client has not come
yet, or was given `SWWAF_REPUTATION_CACHE_TTL` ago or more, is not named, nor
is AbuseIPDB for such a score. It is left out for a client the blocklists are
not checked for: one in `SWWAF_ALLOW_NETS`, and one `SWWAF_DENY_NETS`, a ban
or the country lists refuse first, or would in `observe` mode. The zones are
not checked either for a client a blocklist refuses, nor AbuseIPDB's score for
one a blocklist or a zone refuses, or would in `observe` mode.
- `ban_expires` is there for a request that made a ban or was refused under one, - `ban_expires` is there for a request that made a ban or was refused under one,
or in `observe` mode would have been refused under one, and gives when the ban or in `observe` mode would have been refused under one, and gives when the ban
ends, in the same form as `time`, or `permanent`. ends, in the same form as `time`, or `permanent`.
@@ -827,8 +685,7 @@ which every line has.
No body is logged, and no header but those above. `smallwebwaf`'s own messages No body is logged, and no header but those above. `smallwebwaf`'s own messages
(start, the settings, stop, errors) share the stream as JSON lines marked (start, the settings, stop, errors) share the stream as JSON lines marked
`"type":"process"`, each with `instance` as a request's line has it, and those `"type":"process"`, each with `instance` as a request's line has it.
less severe than `SWWAF_LOG_LEVEL` are not written.
Go's HTTP server, on which `smallwebwaf` is built, reads a request's line and Go's HTTP server, on which `smallwebwaf` is built, reads a request's line and
headers before `smallwebwaf` sees the request, and some requests end there, headers before `smallwebwaf` sees the request, and some requests end there,
@@ -880,15 +737,7 @@ it, as below. An alert is for one of these events, and is sent when
- `anomaly`: a count of requests or bytes over an anomaly threshold, raised by - `anomaly`: a count of requests or bytes over an anomaly threshold, raised by
each request that ends with the count over it, in `observe` mode as in each request that ends with the count over it, in `observe` mode as in
`enforce` mode. It refuses and bans nothing. `enforce` mode. It refuses and bans nothing.
- `reputation_hit`: a request whose client a blocklist or a DNSBL zone's verdict - `source_failure`: GeoJS failing or refusing `smallwebwaf`.
lists, or whose AbuseIPDB score is a hit, one alert for each blocklist and
each zone that lists it, and one for AbuseIPDB, whatever
`SWWAF_BLOCKLIST_ACTION` and `SWWAF_REPUTATION_ACTION` say, in `observe` mode
as in `enforce` mode.
- `source_failure`: GeoJS failing or refusing `smallwebwaf`, a fetch of a list
failing (see "Blocklists" below), a query to a DNSBL zone failing or refused
(see "DNS blocklists" below), or a check with AbuseIPDB failing or refused, or
the check that uses up the day's AbuseIPDB checks (see "AbuseIPDB" below).
- `file_error`: a rule file edited while it runs that has an error, an edit of a - `file_error`: a rule file edited while it runs that has an error, an edit of a
state file set aside as `<name>.bad`, a state file it could not write while state file set aside as `<name>.bad`, a state file it could not write while
running, or a replacement of the lookup database it could not read, which it running, or a replacement of the lookup database it could not read, which it
@@ -951,31 +800,25 @@ is sent on one line:
- `client` is the address of the client whose request raised the alert, and - `client` is the address of the client whose request raised the alert, and
`netblock` the netblock of the ban, or for an `anomaly`, the netblock counted: `netblock` the netblock of the ban, or for an `anomaly`, the netblock counted:
the client's own, the netblock around it or a named netblock, and none for an the client's own, the netblock around it or a named netblock, and none for an
AS number or the whole service, or for a `reputation_hit`, the client's own, AS number or the whole service; both are empty for `source_failure` and
as `client_group` gives it; both are empty for `source_failure` and
`file_error`. `asn`, `as_name` and `country` are, for a ban, the client's as `file_error`. `asn`, `as_name` and `country` are, for a ban, the client's as
the ban's notes give them when the alert is raised: empty, as in this alert, the ban's notes give them when the alert is raised: empty, as in this alert,
when GeoJS had not answered about the client by then; for an `anomaly`, the when GeoJS had not answered about the client by then; for an `anomaly`, the
client's as the lookup gave them by the time its request ended; for a client's as the lookup gave them by the time its request ended.
`reputation_hit`, the client's as its request's log line gives them.
- `reason` is a short sentence; for a ban, the ban's `reason` in `bans.json`; - `reason` is a short sentence; for a ban, the ban's `reason` in `bans.json`;
for an `anomaly`, what was counted over which threshold, such as for an `anomaly`, what was counted over which threshold, such as
`requests per minute of the netblock 203.0.113.0/24 over the threshold of 1000`; `requests per minute of the netblock 203.0.113.0/24 over the threshold of 1000`.
for a `reputation_hit`, `listed by a blocklist`, `listed by a DNSBL zone` or
`scored by AbuseIPDB at or over SWWAF_ABUSEIPDB_MIN_SCORE`.
- `detail` is what is particular to the event: for a ban, its `cause`, when it - `detail` is what is particular to the event: for a ban, its `cause`, when it
ends as `ban_expires`, in the form the request log gives it, and its `notes`, ends as `ban_expires`, in the form the request log gives it, and its `notes`,
as `bans.json` gives them; for an `anomaly`, the `scope`, `client`, `net`, as `bans.json` gives them; for an `anomaly`, the `scope`, `client`, `net`,
`asn`, `total` or `watch`, as the settings name them, the `asn` counted for `asn`, `total` or `watch`, as the settings name them, the `asn` counted for
`asn` and the `name` of the named netblock for `watch`, the `window`, `minute` `asn` and the `name` of the named netblock for `watch`, the `window`, `minute`
or `hour`, the `kind`, `requests` or `bytes`, the `count`, which is weighted or `hour`, the `kind`, `requests` or `bytes`, the `count`, which is weighted
as the rate limits weigh theirs, and the `threshold`; for a `reputation_hit`, as the rate limits weigh theirs, and the `threshold`; for `source_failure`,
the `source`, the URL of the blocklist, the zone or `abuseipdb`, and for the `source`, `geojs`, the `error`, and when GeoJS is asked again,
AbuseIPDB the `score`; for `source_failure`, the `source`, `geojs`, the URL of `asking_again_in`; for `file_error`, the `file`, which for an edit set aside
the list, the zone or `abuseipdb`, the `error`, and for GeoJS, when it is is the file it was renamed to, and the `error`, which for a file that does not
asked again, `asking_again_in`; for `file_error`, the `file`, which for an parse names where in it the error is.
edit set aside is the file it was renamed to, and the `error`, which for a
file that does not parse names where in it the error is.
- `suppressed_repeats` is how many repeats the cooldown held back before this - `suppressed_repeats` is how many repeats the cooldown held back before this
alert, and for a `summary`, those no other alert gives (see below). alert, and for a `summary`, those no other alert gives (see below).
@@ -1020,8 +863,7 @@ and Slack this JSON object, shown indented; it is sent on one line:
} }
``` ```
An alert for the same event as the last one sent, on the same netblock, and for An alert for the same event as the last one sent, on the same netblock, or for a
a `reputation_hit` about the same blocklist, zone or AbuseIPDB, or for a
`file_error` about the same file, or for a `source_failure` about the same `file_error` about the same file, or for a `source_failure` about the same
source, or for an `anomaly` in the same scope, with the same netblock, AS number source, or for an `anomaly` in the same scope, with the same netblock, AS number
or name, whatever its window and kind, less than `SWWAF_ALERT_COOLDOWN` after or name, whatever its window and kind, less than `SWWAF_ALERT_COOLDOWN` after
@@ -1055,12 +897,11 @@ restart the alerts waiting are sent, and the cooldowns go on.
## State files ## State files
`smallwebwaf` keeps its state in memory and a copy of it in five JSON files in `smallwebwaf` keeps its state in memory and a copy of it in four JSON files in
`SWWAF_STATE_DIR`, `/var/lib/smallwebwaf` by default, as "Persistent state" in `SWWAF_STATE_DIR`, `/var/lib/smallwebwaf` by default, as "Persistent state" in
[`SPEC.md`](SPEC.md) describes. Each has a top-level `version`, 1, and lists its [`SPEC.md`](SPEC.md) describes. Each has a top-level `version`, 1, and lists its
entries by client address, but for the alerts waiting, the anomaly counters, entries by client address, but for the alerts waiting, and the anomaly counters,
which are listed by scope first, and the copies of the lists, listed by URL, which are listed by scope first, with times in UTC.
with times in UTC.
- `bans.json`: every ban with its notes, indented to be read. A permanent ban's - `bans.json`: every ban with its notes, indented to be read. A permanent ban's
`expires` is `null`. A ban's `cause` is `limit` for a broken rate limit or `expires` is `null`. A ban's `cause` is `limit` for a broken rate limit or
@@ -1086,40 +927,25 @@ with times in UTC.
`grep` shows everything about one. `grep` shows everything about one.
- `lookups.json`: GeoJS's answers, one to a line, each with the client's AS - `lookups.json`: GeoJS's answers, one to a line, each with the client's AS
number, AS name and country, when GeoJS gave it and when it was last used. number, AS name and country, when GeoJS gave it and when it was last used.
- `reputation.json`: each list fetched from a URL (see "Blocklists" below),
indented to be read, under `lists`: its `url`, when it was last `tried`, the
fetch failed or not, and its last good copy: when that was `fetched`, and its
`lines`, as fetched, comment lines included, each on a line of its own, both
left out while no fetch of it has succeeded; and under `verdicts`, each
verdict of a DNSBL zone still in use (see "DNS blocklists" below): its `zone`,
the `client`'s address, whether the zone `listed` the client, and when the
zone gave it, `fetched`; and under `abuseipdb` (see "AbuseIPDB" below), the
`day`, in UTC, whose checks it counts, left out before the first, the checks
`spent` that day, and under `scores`, each score of AbuseIPDB still in use:
the `client`, its IPv4 address as a /32 or its IPv6 group, its `score`, and
when AbuseIPDB gave it, `fetched`. As the file is read, the lists the settings
no longer name, and the verdicts of the zones they no longer name, are
dropped.
- `alerts.json`: the state of the alerts (see "Alerts" above), indented to be - `alerts.json`: the state of the alerts (see "Alerts" above), indented to be
read: under `cooldowns`, for each event and netblock, with the `source` too read: under `cooldowns`, for each event and netblock, or event and `file` or
for a `reputation_hit`, or event and `file` or `source`, or for an `anomaly`, `source`, or for an `anomaly`, its `scope` with its `netblock`, `asn` or
its `scope` with its `netblock`, `asn` or `name`, or event alone, when the `name`, or event alone, when the last alert was sent, `sent`, and the repeats
last alert was sent, `sent`, and the repeats held back since, held back since, `suppressed_repeats`; under `hour`, the hour under way, from
`suppressed_repeats`; under `hour`, the hour under way, from its `start`, the its `start`, the alerts `sent` in it and those `held_back` for its summary, by
alerts `sent` in it and those `held_back` for its summary, by event; under event; under `waiting`, for each destination you name, `webhook`, `slack` or
`waiting`, for each destination you name, `webhook`, `slack` or `ntfy`, the `ntfy`, the alerts still waiting to be sent to it, the oldest first, each as
alerts still waiting to be sent to it, the oldest first, each as the webhook the webhook is sent it; and under `anomaly_counters`, each anomaly counter:
is sent it; and under `anomaly_counters`, each anomaly counter: its `scope`, its `scope`, as an `anomaly` alert names it, with the `netblock` of a client,
as an `anomaly` alert names it, with the `netblock` of a client, of a netblock of a netblock around a client or of a named netblock, the `asn` of an AS
around a client or of a named netblock, the `asn` of an AS number and the number and the `name` of a named netblock, and its two buckets of requests in
`name` of a named netblock, and its two buckets of requests in the minute and the minute and the hour, `minute` and `hour`, and of bytes, `minute_bytes` and
the hour, `minute` and `hour`, and of bytes, `minute_bytes` and `hour_bytes`, `hour_bytes`, each left out while it is empty. As an hour ends, the cooldowns
each left out while it is empty. As an hour ends, the cooldowns that have run that have run out are dropped, and the hour's summary gives the repeats they
out are dropped, and the hour's summary gives the repeats they held back. As held back. As the file is read, the alerts waiting for a destination you no
the file is read, the alerts waiting for a destination you no longer name are longer name are dropped. A file whose `waiting` is a list, as it was before
dropped. A file whose `waiting` is a list, as it was before alerts went to alerts went to Slack and ntfy too, stops the start: put the list under
Slack and ntfy too, stops the start: put the list under `"webhook"`, or remove `"webhook"`, or remove the file.
the file.
`bans.json` is written `SWWAF_STATE_WRITE_DELAY` after a ban is made, lifted `bans.json` is written `SWWAF_STATE_WRITE_DELAY` after a ban is made, lifted
through `DELETE /_smallwebwaf/bans/<client>`, or made permanent, with every such through `DELETE /_smallwebwaf/bans/<client>`, or made permanent, with every such
@@ -1131,32 +957,27 @@ whole. A write that fails is logged, raised as a `file_error` alert while
changed since the last write. changed since the last write.
At start the files are read back: each client keeps its counts, so a restart At start the files are read back: each client keeps its counts, so a restart
gives it no fresh allowance, each anomaly counter keeps its counts, each ban gives it no fresh allowance, each anomaly counter keeps its counts, and each ban
keeps refusing every client in its netblock until it ends, even after keeps refusing every client in its netblock until it ends, even after
`SWWAF_BAN_SCOPE_V4_PREFIX` has changed, and the copy of each list stays in use `SWWAF_BAN_SCOPE_V4_PREFIX` has changed. A netblock whose address has bits past
until a fetch of it succeeds. A netblock whose address has bits past its length, its length, such as `203.0.113.9/24`, is read as the netblock it is in,
such as `203.0.113.9/24`, is read as the netblock it is in, `203.0.113.0/24`. `203.0.113.0/24`. Buckets and answers whose time has passed are dropped, and so
Buckets and answers whose time has passed are dropped, and so is an anomaly is an anomaly counter left with no bucket. A missing file is empty state, as on
counter left with no bucket; a verdict past its time is neither used nor written a first start. A file that does not parse, or has another `version`, stops the
again. A missing file is empty state, as on a first start. A file that does not start with a message naming the file, and the line and column where Go's JSON
parse, or has another `version`, stops the start with a message naming the file, decoder gives them; so does a state directory `smallwebwaf` cannot write. So
and the line and column where Go's JSON decoder gives them; so does a state does an entry without a field it needs, named with the entry's place in the
directory `smallwebwaf` cannot write. So does an entry without a field it needs, file: a ban's `netblock`, `start` or `expires`, which is `null` for a permanent
named with the entry's place in the file: a ban's `netblock`, `start` or ban; a client's `client`, or the `start` of a window in which it has requests or
`expires`, which is `null` for a permanent ban; a client's `client`, or the bytes; an answer's `client`, `country`, which is `""` for a client GeoJS cannot
`start` of a window in which it has requests or bytes; an answer's `client`, place, or `answered`; a cooldown's `event` or `sent`; an alert waiting's `event`
`country`, which is `""` for a client GeoJS cannot place, or `answered`; a or `time`; an anomaly counter's `netblock`, unless it counts an AS number or the
cooldown's `event` or `sent`; an alert waiting's `event` or `time`; an anomaly whole service, its `asn`, for an AS number, its `name`, for a named netblock, or
counter's `netblock`, unless it counts an AS number or the whole service, its the `start` of a window in which it has requests or bytes. So does a ban whose
`asn`, for an AS number, its `name`, for a named netblock, or the `start` of a `cause` is not `limit`, `attack` or `admin`, alerts waiting for a destination
window in which it has requests or bytes; a list's `url`, `fetched` or `lines`, that is not `webhook`, `slack` or `ntfy`, and an anomaly counter whose `scope`
which is `[]` for an empty list; a verdict's `zone`, `client`, `listed`, which is not `client`, `net`, `asn`, `total` or `watch`. An answer's `asn` or
is `false` for a client the zone does not list, or `fetched`. So does a ban `as_name` left out reads as empty.
whose `cause` is not `limit`, `attack` or `admin`, alerts waiting for a
destination that is not `webhook`, `slack` or `ntfy`, an anomaly counter whose
`scope` is not `client`, `net`, `asn`, `total` or `watch`, and a copy of a list
with a line that would make its fetch fail. An answer's `asn` or `as_name` left
out reads as empty.
While it runs, `smallwebwaf` watches `SWWAF_STATE_DIR` and takes in your edit of While it runs, `smallwebwaf` watches `SWWAF_STATE_DIR` and takes in your edit of
a state file as soon as you save it: what the file then holds replaces what a state file as soon as you save it: what the file then holds replaces what
@@ -1166,14 +987,14 @@ writes a file it takes in any edit made since, so your edit is not overwritten;
a change `smallwebwaf` made after you opened the file, such as a new ban, is a change `smallwebwaf` made after you opened the file, such as a new ban, is
lost when you save over it. An edit that would stop the start, because it does lost when you save over it. An edit that would stop the start, because it does
not parse, has another `version`, leaves out a field an entry needs, gives a ban not parse, has another `version`, leaves out a field an entry needs, gives a ban
another `cause`, names another destination, gives an anomaly counter another another `cause`, names another destination or gives an anomaly counter another
`scope` or gives a list's copy a line that would make its fetch fail, does not `scope`, does not stop the running `smallwebwaf`: it keeps what it holds, and at
stop the running `smallwebwaf`: it keeps what it holds, and at the file's next the file's next write renames your file to `<name>.bad`, such as
write renames your file to `<name>.bad`, such as `bans.json.bad`, writes the `bans.json.bad`, writes the file again from memory, logs the file and where the
file again from memory, logs the file and where the error is, and raises a error is, and raises a `file_error` alert for it. It waits for that write
`file_error` alert for it. It waits for that write because an editor's file can because an editor's file can be read before the editor has finished writing it.
be read before the editor has finished writing it. Mend the `.bad` file and move Mend the `.bad` file and move it back. A file you remove is written again at its
it back. A file you remove is written again at its next write. next write.
To ban a netblock, add an entry to `bans.json` with its `netblock`, its `start` To ban a netblock, add an entry to `bans.json` with its `netblock`, its `start`
and its `expires`, `null` for a ban that never ends; its `reason` and its and its `expires`, `null` for a ban that never ends; its `reason` and its
@@ -1342,24 +1163,6 @@ scraped, and keeps this one as `exported_instance` unless the scrape sets
database in use was read; and database in use was read; and
`smallwebwaf_lookup_database_read_failures_total`: the replacements of it that `smallwebwaf_lookup_database_read_failures_total`: the replacements of it that
could not be read. could not be read.
- By `source`, the URL of each list `SWWAF_BLOCKLIST_URLS` or
`SWWAF_ASN_LIMIT_PERCENT_URL` names: `smallwebwaf_reputation_hits_total`: the
requests whose client the blocklist lists, a series that comes with the first;
`smallwebwaf_reputation_failures_total`: the fetches of the list that failed;
and `smallwebwaf_reputation_last_fetch_timestamp_seconds`: when the copy of it
in use was fetched, `0` while there is none.
- By `source`, each zone `SWWAF_DNSBL_ZONES` names:
`smallwebwaf_reputation_hits_total`: the requests whose client the zone's
verdict lists, a series that comes with the first;
`smallwebwaf_reputation_queries_total`: the queries made to the zone; and
`smallwebwaf_reputation_failures_total`: those that failed.
- While `SWWAF_ABUSEIPDB_KEY` is set, by `source`, `abuseipdb`:
`smallwebwaf_reputation_hits_total`: the requests whose client's AbuseIPDB
score is a hit, a series that comes with the first;
`smallwebwaf_reputation_queries_total`: the checks made, each of which spends
one of the day's budget; `smallwebwaf_reputation_failures_total`: those that
failed; and `smallwebwaf_reputation_daily_budget_remaining`: the checks the
day's `SWWAF_ABUSEIPDB_DAILY_BUDGET` has left.
- `smallwebwaf_tracked_clients`: the clients in the table of clients. - `smallwebwaf_tracked_clients`: the clients in the table of clients.
- `smallwebwaf_state_file_writes_total`, - `smallwebwaf_state_file_writes_total`,
`smallwebwaf_state_file_write_failures_total`, `smallwebwaf_state_file_write_failures_total`,
@@ -1398,7 +1201,7 @@ request that carries the token as `Authorization: Bearer <token>`:
`reason`. `netblock` is a netblock such as `203.0.113.0/24`, or a client's `reason`. `netblock` is a netblock such as `203.0.113.0/24`, or a client's
address, which bans the netblock a ban on that client covers: its IPv4 address, which bans the netblock a ban on that client covers: its IPv4
address, or the netblock around it that `SWWAF_BAN_SCOPE_V4_PREFIX` sets, or address, or the netblock around it that `SWWAF_BAN_SCOPE_V4_PREFIX` sets, or
its IPv6 group. `duration` is a duration such as `1h` or `7d`, or `permanent`. its IPv6 /64. `duration` is a duration such as `1h` or `7d`, or `permanent`.
The ban starts at once, its `cause` is `admin`, and it is made even while The ban starts at once, its `cause` is `admin`, and it is made even while
another ban on the netblock lasts. A body that is not such an object, has another ban on the netblock lasts. A body that is not such an object, has
another field, has anything but whitespace after the object, or is longer than another field, has anything but whitespace after the object, or is longer than
@@ -1551,10 +1354,9 @@ goes through the candidates one by one.
readable JSON files, written regularly and at every stop, so a restart loses readable JSON files, written regularly and at every stop, so a restart loses
nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks
up the change. Nothing is read from disk while serving a request. The files up the change. Nothing is read from disk while serving a request. The files
for the bans, the clients, the GeoJS answers, the copies of the lists, the for the bans, the clients, the GeoJS answers and the alerts are built, with an
verdicts of the DNSBL zones, AbuseIPDB's scores and the alerts are built, with edit taken in while running (see "State files" above); the others come with
an edit taken in while running (see "State files" above); the rest comes with their features.
its features.
- Health checks, the metrics, and listing, adding and lifting bans or asking why - Health checks, the metrics, and listing, adding and lifting bans or asking why
a given address was refused, all on the one port every request uses: under a given address was refused, all on the one port every request uses: under
`/_smallwebwaf/` on the app's own address, through traefik like any other `/_smallwebwaf/` on the app's own address, through traefik like any other
@@ -1574,7 +1376,7 @@ For each request `smallwebwaf`:
the deny list or currently banned; the deny list or currently banned;
- looks up its AS number and country, and refuses it if that country is denied, - looks up its AS number and country, and refuses it if that country is denied,
or is not among the only ones allowed; or is not among the only ones allowed;
- checks it against the blocklists, and for a cached reputation verdict; - checks for a cached reputation verdict;
- picks the client's limit percentage from those; - picks the client's limit percentage from those;
- checks the minute, hour and day request counters against the limits, and bans - checks the minute, hour and day request counters against the limits, and bans
the client if it breaks one; the client if it breaks one;
@@ -1679,7 +1481,7 @@ setting uses the answer, unless you set `SWWAF_LOOKUP_SOURCE=off`. The only
visitors it is not told about are those in `SWWAF_ALLOW_NETS` or visitors it is not told about are those in `SWWAF_ALLOW_NETS` or
`SWWAF_DENY_NETS`, those whose netblock a ban covers, and those on a private, `SWWAF_DENY_NETS`, those whose netblock a ban covers, and those on a private,
loopback or link-local address. An IPv6 visitor is asked about by the first loopback or link-local address. An IPv6 visitor is asked about by the first
address of its IPv6 group. Each answer is kept for seven days, in memory and in address of its /64. Each answer is kept for seven days, in memory and in
`lookups.json`, so that it survives a restart, and a visitor whose answer is `lookups.json`, so that it survives a restart, and a visitor whose answer is
kept is not asked about again. kept is not asked about again.
@@ -1731,160 +1533,6 @@ country: `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses it unless you list it in
`SWWAF_UNKNOWN_LIMIT_PERCENT` sets its limits. Such addresses are never sent to `SWWAF_UNKNOWN_LIMIT_PERCENT` sets its limits. Such addresses are never sent to
GeoJS. GeoJS.
## Blocklists
`SWWAF_BLOCKLIST_URLS` names blocklists: text files of addresses and netblocks,
one to a line, written as the Spamhaus DROP list,
`https://www.spamhaus.org/drop/drop.txt`, is. Anything after a `;` or a `#` on a
line is left out, and so is a line left blank. A bare address stands for itself
alone, as in the settings. An IPv4-mapped address or netblock, such as
`::ffff:192.0.2.0/120`, is read as the IPv4 one it stands for, here
`192.0.2.0/24`, since a client's IPv4 address is checked as IPv4; a mapped
netblock shorter than `/96` stands for none, and is not a netblock. None is
named by default: a list judges a client by what others saw it do, while the
defaults judge it by what it does to your service.
`smallwebwaf` fetches each list, and the file `SWWAF_ASN_LIMIT_PERCENT_URL`
names, `SWWAF_BLOCKLIST_REFRESH` after it last fetched it or tried to, 24 hours
by default and never less than one, one list after another. Since
`reputation.json` keeps when each list was last tried, the fetch failed or not,
even one cut off as `smallwebwaf` stopped, whose request the server may have
had, at start `smallwebwaf` fetches at once only a list it has never tried, and
one it last tried that long ago; any other waits its turn, so that restarts do
not fetch a list more often. A fetch fails when the server answers other than
`200`, when it does not finish within a minute, when the list is longer than 16
MiB, or when a line of it is not an address or a netblock, or for
`SWWAF_ASN_LIMIT_PERCENT_URL`, not an AS number, `:` and a percentage. The copy
fetched before then stays in use, and the failure is counted, logged and raised
as a `source_failure` alert; a fetch cut off as `smallwebwaf` stops is not a
failure. The last good copy of each list is kept whole, comment lines included,
in `reputation.json` (see "State files" above), so that a restart keeps it in
use too. Each list is named by its URL, in the request log, the alerts and the
metrics, so keep a secret out of it.
A client in `SWWAF_ALLOW_NETS` is not checked. Any other is checked by its own
address after the country lists, and `SWWAF_BLOCKLIST_ACTION` says what is done
with one a list lists, as "What it does so far" above describes. `deny` suits
lists of networks that send nothing legitimate, such as DROP; `limit:<percent>`
suits lists of addresses shared with ordinary visitors, such as those of Tor's
exits.
The Spamhaus DROP list is The Spamhaus Project's, https://www.spamhaus.org. Its
terms, on its DROP page, ask that a product using it credit The Spamhaus Project
and keep the list's date and copyright lines with the data, which the copy in
`reputation.json` does; a service that uses DROP through `smallwebwaf` uses data
from The Spamhaus Project, and should say so. They also ask that it be fetched
automatically no more than once an hour, once a day being more than enough in
most cases, and Spamhaus may block an address that fetches it more often. Each
`smallwebwaf` fetches its own copy, on its own schedule, so on a host where
several apps run it, sharing one address, their fetches can come less than an
hour apart whatever `SWWAF_BLOCKLIST_REFRESH` is: each fetches a list again that
long after its own last try, so those first started within the same hour, with
the same refresh, keep fetching within the same hour.
## DNS blocklists
`SWWAF_DNSBL_ZONES` names DNS blocklists, DNSBL zones, which list an address by
answering a query for a name made from it. None is named by default, for the
reason none of the blocklists is. `smallwebwaf` asks each zone about a client's
own address in the background, the first time it sees the client: the request
goes on at once, as from a client the zone does not list, and so does every
request from it until the zone has answered. The name asked about is the one RFC
5782 gives: the four numbers of an IPv4 address in reverse order, so that
`192.0.2.99` is asked about in `dnsbl.dronebl.org` as
`99.2.0.192.dnsbl.dronebl.org`, or the 32 hex digits of an IPv6 address in
reverse order, each followed by a dot. An IPv6 client is asked about by its own
address, not by its IPv6 group.
A zone that answers that the name does not exist, or has no address, does not
list the client, and one that answers with an address in `127.0.0.0/8` lists it.
Any other answer gives no verdict, and is a failure: an address in
`127.255.255.0/24`, with which Spamhaus refuses a query, such as one sent
through a public resolver or one past its limit; an address outside
`127.0.0.0/8`, such as a resolver gives that answers even for names that do not
exist; an error the resolver answers with, such as a refusal; and no answer
within `SWWAF_REPUTATION_TIMEOUT`. A failure is counted, logged and raised as a
`source_failure` alert, held back as a repeat within `SWWAF_ALERT_COOLDOWN`, and
the zone is not asked again for a minute, so that a zone that refuses queries is
not asked on every request.
Each verdict, the zone listing the client or not, is kept, in memory and in
`reputation.json` (see "State files" above), so that a restart keeps it, and is
used for `SWWAF_REPUTATION_CACHE_TTL` after the zone gave it, 24 hours by
default. The client's first request after that has the zone asked again, and
until it answers, no verdict of the zone applies to the client. At most 100,000
verdicts are kept, the one fetched longest ago dropped first, and at most 1,000
queries are under way at once; past that, a zone is asked about a client at the
client's next request.
A client in `SWWAF_ALLOW_NETS` is not checked, nor is one a blocklist refuses.
`SWWAF_REPUTATION_ACTION` says what is done with one a zone's verdict lists, as
"What it does so far" above describes.
Do not name a zone meant for mail, such as one of residential and dynamic
address ranges, which ordinary visitors come from, or one that includes such a
list, as Spamhaus's `zen` does: it would refuse them, or lower their limits.
Spamhaus's zones answer through its keyed query service, named with the key in
it, such as `<key>.xbl.dq.spamhaus.net`. The key of a zone under
`dq.spamhaus.net` is its first label, and `smallwebwaf` shows `********` in its
place wherever it names the zone, as `********.xbl.dq.spamhaus.net`: in the
settings logged at start, an error that stops the start, its own messages, the
request log, the alerts and the metrics. Only `reputation.json` keeps the zone
with its key. A key in the name of any other zone is shown as given. Several
zones refuse queries that come through a public resolver; `SWWAF_DNSBL_RESOLVER`
names another resolver to ask through.
## AbuseIPDB
While `SWWAF_ABUSEIPDB_KEY` holds the key of an AbuseIPDB account, `smallwebwaf`
asks AbuseIPDB's check endpoint, `https://api.abuseipdb.com/api/v2/check`, for
the abuse confidence score of a client's own address, from 0 to 100. It is unset
by default, for the reason no blocklist is named, and since AbuseIPDB needs an
account. An IPv6 client, an IPv6 group, is checked by the address of the request
that has it checked, and its score is used for the whole group, whichever of its
addresses sends, so that one client costs at most one check every
`SWWAF_REPUTATION_CACHE_TTL`.
Only a client whose history counts an offence is checked, so that the checks are
spent on suspects: so far, one that has broken a rate limit or a byte limit,
matched a ban rule, or had a request refused by a block rule. A client dropped
from the table of clients loses its history, and with it its offences. A client
is checked in the background, at its first request after its offence that
reaches the check: no request waits, a request refused under its ban is not
checked, and the request that has it checked, and any other from it before the
answer comes, goes on as from a client without a score.
A score at or over `SWWAF_ABUSEIPDB_MIN_SCORE`, 75 by default, is a hit, and
`SWWAF_REPUTATION_ACTION` says what is done with its client, as for a DNSBL
zone's verdict (see "What it does so far" above). Each score, a hit or not, is
kept, in memory and in `reputation.json` (see "State files" above), and used for
`SWWAF_REPUTATION_CACHE_TTL` after AbuseIPDB gave it, 24 hours by default. The
client's first request after that has it checked again, if its history still
counts an offence. At most 100,000 scores are kept, the one fetched longest ago
dropped first.
At most `SWWAF_ABUSEIPDB_DAILY_BUDGET` checks are made in a day, 900 by default,
below the 1,000 of AbuseIPDB's free accounts. The day is counted in UTC, from
00:00. Each check sent spends one of them, whatever AbuseIPDB answers, and the
checks spent today are kept in `reputation.json`, so that a restart does not
make the budget whole again. The check that spends the last of the day's budget
is logged and raised as a `source_failure` alert; from then until the day ends,
no client is checked, and a client without a score counts as one AbuseIPDB does
not list.
A check fails when AbuseIPDB answers other than `200`, such as `429` past its
own daily limit or `401` for a wrong key, when its answer gives no
`abuseConfidenceScore`, and when it does not answer within
`SWWAF_REPUTATION_TIMEOUT`. A failure gives no score, and is counted, logged and
raised as a `source_failure` alert, held back as a repeat within
`SWWAF_ALERT_COOLDOWN`, and no client is checked for a minute after it.
The key is sent to AbuseIPDB in the `Key` header, and nowhere else. The settings
logged at start show `********` in its place, and neither the log, the alerts,
the metrics nor `reputation.json` hold it. Given as a file, with
`SWWAF_ABUSEIPDB_KEY_FILE`, it can be kept out of the app's reach (see "Settings
given as files" above).
## How the code is laid out ## How the code is laid out
- `cmd/smallwebwaf`: the binary, which only calls `internal/smallwebwaf`. - `cmd/smallwebwaf`: the binary, which only calls `internal/smallwebwaf`.
@@ -1898,15 +1546,15 @@ given as files" above).
standard library's `httputil.ReverseProxy` within the timeouts and size standard library's `httputil.ReverseProxy` within the timeouts and size
limits, and writes the request's log line. Its `check` method is where a limits, and writes the request's log line. Its `check` method is where a
request is refused before anything reaches the app: for `SWWAF_DENY_NETS`, for request is refused before anything reaches the app: for `SWWAF_DENY_NETS`, for
a ban, for the country lists, for a blocklist, for a DNSBL zone's verdict, for a ban, for the country lists, for a rate limit, which bans the client, for a
AbuseIPDB's score, for a rate limit, which bans the client, for a `block` or `block` or `ban` rule, the latter banning the client, and for an announced
`ban` rule, the latter banning the client, and for an announced body over the body over the size limit; in `observe` mode, only for the size limit, with
size limit; in `observe` mode, only for the size limit, with what it would what it would have refused for noted in the log line. A request under
have refused for noted in the log line. A request under `/_smallwebwaf/` that `/_smallwebwaf/` that `check` lets through is answered by `answerAdmin`
`check` lets through is answered by `answerAdmin` instead of reaching the app. instead of reaching the app. Once the answer to a request passed to the app
Once the answer to a request passed to the app has ended, `countBytes` counts has ended, `countBytes` counts its bytes for the byte limits, and once any
its bytes for the byte limits, and once any request but the health check has request but the health check has ended, `countAnomalies` counts it for the
ended, `countAnomalies` counts it for the anomaly thresholds. anomaly thresholds.
- `internal/metrics`: the metrics, counted as the other parts tell it what - `internal/metrics`: the metrics, counted as the other parts tell it what
happened, and served in the Prometheus text format. happened, and served in the Prometheus text format.
- `internal/bans`: the ban ledger: each netblock's bans with their notes, how - `internal/bans`: the ban ledger: each netblock's bans with their notes, how
@@ -1919,14 +1567,6 @@ given as files" above).
client's history and to the notes of its bans; or in the lookup database, client's history and to the notes of its bans; or in the lookup database,
which it reads again when the file is replaced. `internal/lookup/lookuptest` which it reads again when the file is replaced. `internal/lookup/lookuptest`
writes lookup databases for the tests. writes lookup databases for the tests.
- `internal/reputation`: fetches the blocklists and the file
`SWWAF_ASN_LIMIT_PERCENT_URL` names as they are due, keeps the last good copy
of each, and tells which blocklists list an address and what percentage the
file gives an AS number; asks the DNSBL zones about clients in the background,
through the standard library's resolver, keeps their verdicts, and tells which
zones' verdicts list an address; and checks clients with AbuseIPDB in the
background, keeps their scores and the checks spent today, and tells whether a
client's score is a hit.
- `internal/ratelimit`: the table of clients: counts each client's requests and - `internal/ratelimit`: the table of clients: counts each client's requests and
bytes, tells when they take it over a rate limit or a byte limit, and keeps bytes, tells when they take it over a rate limit or a byte limit, and keeps
each client's history. each client's history.
@@ -1953,17 +1593,16 @@ given as files" above).
checks. checks.
Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the
table of clients to `SWWAF_MAX_TRACKED_CLIENTS` and the GeoJS answers to table of clients to 20,000 and the GeoJS answers to 100,000, dropping the least
100,000, dropping the least recently seen, the DNSBL zones' verdicts and recently seen, the anomaly counters to 20,000, dropping the one counted least
AbuseIPDB's scores to 100,000 each, dropping the one fetched longest ago, the recently, and the banned netblocks in the order they were last seen, from which
anomaly counters to 20,000, dropping the one counted least recently, and the the ledger picks the ban to drop past `SWWAF_MAX_BANS`, and
banned netblocks in the order they were last seen, from which the ledger picks `github.com/prometheus/client_golang` keeps the metrics and serves them, and
the ban to drop past `SWWAF_MAX_BANS`, and `github.com/prometheus/client_golang` `github.com/fsnotify/fsnotify` tells `smallwebwaf` when a state file or a rule
keeps the metrics and serves them, and `github.com/fsnotify/fsnotify` tells file is saved, or the lookup database replaced, and
`smallwebwaf` when a state file or a rule file is saved, or the lookup database `github.com/oschwald/maxminddb-golang/v2` reads the lookup database, which the
replaced, and `github.com/oschwald/maxminddb-golang/v2` reads the lookup tests write with `github.com/maxmind/mmdbwriter`. The country codes are the list
database, which the tests write with `github.com/maxmind/mmdbwriter`. The in `internal/config/config.go`.
country codes are the list in `internal/config/config.go`.
## Entrypoints ## Entrypoints
internal/alerts/alerts.go
+4 -7
View File
@@ -44,14 +44,11 @@ const (
// EventAnomaly is a count of requests or bytes over an anomaly // EventAnomaly is a count of requests or bytes over an anomaly
// threshold. // threshold.
EventAnomaly = "anomaly" EventAnomaly = "anomaly"
// EventWAFBlock comes with the Core Rule Set; nothing raises it yet. // EventWAFBlock and EventReputationHit come with the Core Rule Set and
EventWAFBlock = "waf_block" // the reputation sources; nothing raises them yet.
// EventReputationHit is a request whose client a blocklist or a DNSBL EventWAFBlock = "waf_block"
// zone lists, or whose AbuseIPDB score is a hit.
EventReputationHit = "reputation_hit" EventReputationHit = "reputation_hit"
// EventSourceFailure is GeoJS failing or refusing smallwebwaf, a fetch // EventSourceFailure is GeoJS failing or refusing smallwebwaf.
// of a list failing, a query to a DNSBL zone or a check with AbuseIPDB
// failing or refused, or the day's AbuseIPDB checks used up.
EventSourceFailure = "source_failure" EventSourceFailure = "source_failure"
// EventFileError is a rule file or state file edited while smallwebwaf // EventFileError is a rule file or state file edited while smallwebwaf
// runs that does not parse, a replacement of the lookup database that // runs that does not parse, a replacement of the lookup database that
internal/anomaly/anomaly.go
+2 -4
View File
@@ -27,8 +27,7 @@ const maxCounters = 20000
// The scopes, what a counter counts, as the settings, alerts.json and the // The scopes, what a counter counts, as the settings, alerts.json and the
// alerts name them. // alerts name them.
const ( const (
// ScopeClient is one client: an IPv4 address, or an IPv6 netblock of // ScopeClient is one client: an IPv4 address, or an IPv6 /64.
// SWWAF_IPV6_GROUP_PREFIX.
ScopeClient = "client" ScopeClient = "client"
// ScopeNet is the netblock around a client, SWWAF_ANOMALY_NET_V4_PREFIX // ScopeNet is the netblock around a client, SWWAF_ANOMALY_NET_V4_PREFIX
// or SWWAF_ANOMALY_NET_V6_PREFIX long. // or SWWAF_ANOMALY_NET_V6_PREFIX long.
@@ -104,8 +103,7 @@ type Counter struct {
// Request is a request that has ended, as the counters count it. // Request is a request that has ended, as the counters count it.
type Request struct { type Request struct {
// Client is the client's address, and ClientGroup the client it is // Client is the client's address, and ClientGroup the client it is
// counted as: its IPv4 address, or the IPv6 netblock of // counted as: its IPv4 address, or its IPv6 /64.
// SWWAF_IPV6_GROUP_PREFIX its address is in.
Client netip.Addr Client netip.Addr
ClientGroup netip.Prefix ClientGroup netip.Prefix
// ASN, ASName and Country are the client's as looked up, each "" when // ASN, ASName and Country are the client's as looked up, each "" when
internal/config/config.go
+31 -380
View File
@@ -48,12 +48,6 @@ type Config struct {
// TrustedProxies are the netblocks whose X-Forwarded-For is // TrustedProxies are the netblocks whose X-Forwarded-For is
// believed (SWWAF_TRUSTED_PROXIES). // believed (SWWAF_TRUSTED_PROXIES).
TrustedProxies []netip.Prefix TrustedProxies []netip.Prefix
// IPv6GroupPrefix is the length of the IPv6 netblock that is one client
// (SWWAF_IPV6_GROUP_PREFIX), from 32 to 128.
IPv6GroupPrefix int
// MaxTrackedClients is the most clients the table of clients holds, in
// memory and in clients.json (SWWAF_MAX_TRACKED_CLIENTS).
MaxTrackedClients int
// ClientRequestTimeout bounds reading the whole request from the // ClientRequestTimeout bounds reading the whole request from the
// client (SWWAF_CLIENT_REQUEST_TIMEOUT). // client (SWWAF_CLIENT_REQUEST_TIMEOUT).
ClientRequestTimeout time.Duration ClientRequestTimeout time.Duration
@@ -137,50 +131,12 @@ type Config struct {
// percentages from 0 to 100, by AS number, written as AS64496, or by // percentages from 0 to 100, by AS number, written as AS64496, or by
// country, a two-letter code in capitals, as the lookup gives them. // country, a two-letter code in capitals, as the lookup gives them.
// UnknownLimitPercent is the percentage of every limit a client without // UnknownLimitPercent is the percentage of every limit a client without
// a country gets (SWWAF_UNKNOWN_LIMIT_PERCENT). ASNLimitPercentURL is // a country gets (SWWAF_UNKNOWN_LIMIT_PERCENT).
// where a file of AS:percent lines is fetched from, whose percentages
// count as those of ASNLimitPercent do (SWWAF_ASN_LIMIT_PERCENT_URL), ""
// while it is unset.
ASNLimitPercent map[string]int64 ASNLimitPercent map[string]int64
CountryLimitPercent map[string]int64 CountryLimitPercent map[string]int64
ASNBytesPercent map[string]int64 ASNBytesPercent map[string]int64
CountryBytesPercent map[string]int64 CountryBytesPercent map[string]int64
UnknownLimitPercent int64 UnknownLimitPercent int64
ASNLimitPercentURL string
// BlocklistURLs are where the blocklists are fetched from
// (SWWAF_BLOCKLIST_URLS). Each list, and ASNLimitPercentURL's, is
// fetched again BlocklistRefresh after it was last fetched or tried
// (SWWAF_BLOCKLIST_REFRESH), which is never less than an hour.
// BlocklistAction is what is done with a client a blocklist lists
// (SWWAF_BLOCKLIST_ACTION): deny, limit or log; for limit,
// BlocklistLimitPercent is the percentage of every limit it gets.
BlocklistURLs []string
BlocklistRefresh time.Duration
BlocklistAction string
BlocklistLimitPercent int64
// DNSBLZones are the DNSBL zones clients are asked about
// (SWWAF_DNSBL_ZONES), through DNSBLResolver (SWWAF_DNSBL_RESOLVER), or
// the host's resolver while that is the zero AddrPort.
// AbuseIPDBKey is the key of the AbuseIPDB account clients are checked
// with (SWWAF_ABUSEIPDB_KEY), "" while it is unset and none is. A score
// of AbuseIPDBMinScore or more is a hit (SWWAF_ABUSEIPDB_MIN_SCORE), and
// at most AbuseIPDBDailyBudget checks are made a day
// (SWWAF_ABUSEIPDB_DAILY_BUDGET).
// ReputationAction is what is done with a client a zone's verdict lists,
// or whose score is a hit (SWWAF_REPUTATION_ACTION): deny, limit or log;
// for limit, ReputationLimitPercent is the percentage of every limit it
// gets. A verdict or a score is used for ReputationCacheTTL after it was
// fetched (SWWAF_REPUTATION_CACHE_TTL), and a query or a check may take
// ReputationTimeout (SWWAF_REPUTATION_TIMEOUT). Neither can be off.
DNSBLZones []string
DNSBLResolver netip.AddrPort
AbuseIPDBKey string
AbuseIPDBMinScore int64
AbuseIPDBDailyBudget int
ReputationAction string
ReputationLimitPercent int64
ReputationCacheTTL time.Duration
ReputationTimeout time.Duration
// BanResponse is the status a refused client is answered with, 403 // BanResponse is the status a refused client is answered with, 403
// or 429, or 0 to close the connection without an answer // or 429, or 0 to close the connection without an answer
// (SWWAF_BAN_RESPONSE). It answers a banned client, a request that // (SWWAF_BAN_RESPONSE). It answers a banned client, a request that
@@ -215,9 +171,6 @@ type Config struct {
// LogRequestHeaders are the request headers whose values the request // LogRequestHeaders are the request headers whose values the request
// log gives, in lower case (SWWAF_LOG_REQUEST_HEADERS). // log gives, in lower case (SWWAF_LOG_REQUEST_HEADERS).
LogRequestHeaders []string LogRequestHeaders []string
// LogLevel is the least severe of the process's own messages that are
// written (SWWAF_LOG_LEVEL). It holds back no request log line.
LogLevel slog.Level
// AdminToken is the bearer token an admin sends for the ban endpoints // AdminToken is the bearer token an admin sends for the ban endpoints
// and /_smallwebwaf/clients/<ip> (SWWAF_ADMIN_TOKEN), "" while it is // and /_smallwebwaf/clients/<ip> (SWWAF_ADMIN_TOKEN), "" while it is
// unset and they are off. // unset and they are off.
@@ -306,10 +259,6 @@ const (
gibibyte = 1 << 30 gibibyte = 1 << 30
ipv4Bits = 32 ipv4Bits = 32
ipv6Bits = 128 ipv6Bits = 128
// minIPv6GroupPrefix is the shortest SWWAF_IPV6_GROUP_PREFIX, the
// netblock a provider is usually given: a shorter one would make one
// client of the customers of several providers.
minIPv6GroupPrefix = 32
// minTokenLength is the fewest characters a token may have. // minTokenLength is the fewest characters a token may have.
minTokenLength = 32 minTokenLength = 32
// masked is what the log shows for a token that is set, and in place of // masked is what the log shows for a token that is set, and in place of
@@ -359,9 +308,6 @@ var (
"is not the length of an IPv4 netblock, from 0 to 32, such as 24") "is not the length of an IPv4 netblock, from 0 to 32, such as 24")
errNotV6Prefix = errors.New( errNotV6Prefix = errors.New(
"is not the length of an IPv6 netblock, from 0 to 128, such as 48") "is not the length of an IPv6 netblock, from 0 to 128, such as 48")
errNotIPv6GroupPrefix = errors.New(
"is not the length of an IPv6 netblock, from 32 to 128, such as 64")
errNotLogLevel = errors.New("is not debug, info, warn or error")
errNotNamedNetblock = errors.New( errNotNamedNetblock = errors.New(
"is not a name, = and a netblock, such as office=203.0.113.0/24") "is not a name, = and a netblock, such as office=203.0.113.0/24")
errNotAbsolutePath = errors.New( errNotAbsolutePath = errors.New(
@@ -398,18 +344,6 @@ var (
"is not a code, : and a percentage, such as AS64496:50 or cn:25") "is not a code, : and a percentage, such as AS64496:50 or cn:25")
errNotPercent = errors.New("is not a percentage, a whole number from 0 to 100") errNotPercent = errors.New("is not a percentage, a whole number from 0 to 100")
errListedTwice = errors.New("is listed twice") errListedTwice = errors.New("is listed twice")
errNotListURL = errors.New(
"is not an http or https URL without a user or a fragment, " +
"such as https://www.spamhaus.org/drop/drop.txt")
errInBlocklistURLs = errors.New("is in SWWAF_BLOCKLIST_URLS too")
errNotAnHourOrMore = errors.New("is not a duration of 1h or more, such as 24h")
errNotAction = errors.New(
"is not deny, limit:<percent> such as limit:25, or log")
errNotZone = errors.New("is not a DNS zone such as dnsbl.dronebl.org")
errZoneTooLong = errors.New("is longer than 189 characters, too long for the " +
"names IPv6 clients are asked about by")
errNotResolver = errors.New("is not an IP address with an optional port, " +
"such as 192.0.2.53 or [2001:db8::53]:5353")
) )
// FromEnvironment reads the settings with lookupEnv, normally // FromEnvironment reads the settings with lookupEnv, normally
@@ -427,8 +361,6 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
InstanceName: env.instanceName(), InstanceName: env.instanceName(),
Observe: env.observe("SWWAF_MODE", "enforce"), Observe: env.observe("SWWAF_MODE", "enforce"),
TrustedProxies: env.netblocks("SWWAF_TRUSTED_PROXIES", privateRanges), TrustedProxies: env.netblocks("SWWAF_TRUSTED_PROXIES", privateRanges),
IPv6GroupPrefix: env.ipv6GroupPrefix("SWWAF_IPV6_GROUP_PREFIX", "64"),
MaxTrackedClients: env.numberNotOff("SWWAF_MAX_TRACKED_CLIENTS", "20000"),
ClientRequestTimeout: env.duration("SWWAF_CLIENT_REQUEST_TIMEOUT", "60s"), ClientRequestTimeout: env.duration("SWWAF_CLIENT_REQUEST_TIMEOUT", "60s"),
ClientRequestHeaderMaxBytes: env.headerSize( ClientRequestHeaderMaxBytes: env.headerSize(
"SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES", "32K"), "SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES", "32K"),
@@ -456,21 +388,11 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
DeniedCountries: env.countries("SWWAF_DENIED_COUNTRIES", ""), DeniedCountries: env.countries("SWWAF_DENIED_COUNTRIES", ""),
ExclusivelyAllowedCountries: env.countries( ExclusivelyAllowedCountries: env.countries(
"SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES", ""), "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES", ""),
ASNLimitPercent: env.percents("SWWAF_ASN_LIMIT_PERCENT", ParseASN), ASNLimitPercent: env.percents("SWWAF_ASN_LIMIT_PERCENT", parseASN),
CountryLimitPercent: env.percents("SWWAF_COUNTRY_LIMIT_PERCENT", parseCountry), CountryLimitPercent: env.percents("SWWAF_COUNTRY_LIMIT_PERCENT", parseCountry),
ASNBytesPercent: env.percents("SWWAF_ASN_BYTES_PERCENT", ParseASN), ASNBytesPercent: env.percents("SWWAF_ASN_BYTES_PERCENT", parseASN),
CountryBytesPercent: env.percents("SWWAF_COUNTRY_BYTES_PERCENT", parseCountry), CountryBytesPercent: env.percents("SWWAF_COUNTRY_BYTES_PERCENT", parseCountry),
UnknownLimitPercent: env.percent("SWWAF_UNKNOWN_LIMIT_PERCENT", "100"), UnknownLimitPercent: env.percent("SWWAF_UNKNOWN_LIMIT_PERCENT", "100"),
ASNLimitPercentURL: env.listURL("SWWAF_ASN_LIMIT_PERCENT_URL"),
BlocklistURLs: env.listURLs("SWWAF_BLOCKLIST_URLS"),
BlocklistRefresh: env.refresh("SWWAF_BLOCKLIST_REFRESH", "24h"),
DNSBLZones: env.zones("SWWAF_DNSBL_ZONES"),
DNSBLResolver: env.resolver("SWWAF_DNSBL_RESOLVER"),
AbuseIPDBKey: env.secret("SWWAF_ABUSEIPDB_KEY"),
AbuseIPDBMinScore: env.percent("SWWAF_ABUSEIPDB_MIN_SCORE", "75"),
AbuseIPDBDailyBudget: env.numberNotOff("SWWAF_ABUSEIPDB_DAILY_BUDGET", "900"),
ReputationCacheTTL: env.durationNotOff("SWWAF_REPUTATION_CACHE_TTL", "24h"),
ReputationTimeout: env.durationNotOff("SWWAF_REPUTATION_TIMEOUT", "2s"),
BanResponse: env.banResponse("SWWAF_BAN_RESPONSE", "403"), BanResponse: env.banResponse("SWWAF_BAN_RESPONSE", "403"),
LimitBanDuration: env.durationNotOff("SWWAF_LIMIT_BAN_DURATION", "1h"), LimitBanDuration: env.durationNotOff("SWWAF_LIMIT_BAN_DURATION", "1h"),
LimitBanRepeatWindow: env.durationNotOff("SWWAF_LIMIT_BAN_REPEAT_WINDOW", "24h"), LimitBanRepeatWindow: env.durationNotOff("SWWAF_LIMIT_BAN_REPEAT_WINDOW", "24h"),
@@ -483,7 +405,6 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
StateCounterInterval: env.durationNotOff("SWWAF_STATE_COUNTER_INTERVAL", "15m"), StateCounterInterval: env.durationNotOff("SWWAF_STATE_COUNTER_INTERVAL", "15m"),
LogRequestHeaders: env.headerNames("SWWAF_LOG_REQUEST_HEADERS", LogRequestHeaders: env.headerNames("SWWAF_LOG_REQUEST_HEADERS",
"accept,accept-language,accept-encoding,content-type,origin,range"), "accept,accept-language,accept-encoding,content-type,origin,range"),
LogLevel: env.logLevel("SWWAF_LOG_LEVEL", "info"),
AdminToken: env.token("SWWAF_ADMIN_TOKEN"), AdminToken: env.token("SWWAF_ADMIN_TOKEN"),
MetricsToken: env.token("SWWAF_METRICS_TOKEN"), MetricsToken: env.token("SWWAF_METRICS_TOKEN"),
MetricsTopN: env.numberNotOff("SWWAF_METRICS_TOP_N", "50"), MetricsTopN: env.numberNotOff("SWWAF_METRICS_TOP_N", "50"),
@@ -514,15 +435,10 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
cfg.LogRemoteAppName = env.appName("SWWAF_LOG_REMOTE_APP_NAME", cfg.LogRemoteAppName = env.appName("SWWAF_LOG_REMOTE_APP_NAME",
cfg.InstanceName, cfg.LogRemoteURL != nil) cfg.InstanceName, cfg.LogRemoteURL != nil)
cfg.BlocklistAction, cfg.BlocklistLimitPercent = env.action(
"SWWAF_BLOCKLIST_ACTION", "deny")
cfg.ReputationAction, cfg.ReputationLimitPercent = env.action(
"SWWAF_REPUTATION_ACTION", "limit:25")
env.checkInstanceNameForNtfy(cfg.InstanceName, cfg.AlertNtfyURL != nil) env.checkInstanceNameForNtfy(cfg.InstanceName, cfg.AlertNtfyURL != nil)
env.checkLookupDBPath(cfg) env.checkLookupDBPath(cfg)
env.checkCountriesAndLookups(cfg) env.checkCountriesAndLookups(cfg)
env.checkASNLimitPercentURL(cfg)
if env.err != nil { if env.err != nil {
return nil, env.err return nil, env.err
@@ -748,93 +664,12 @@ func (e *environment) percents(
// percent reads a setting that is a percentage, from 0 to 100. // percent reads a setting that is a percentage, from 0 to 100.
func (e *environment) percent(name, defaultValue string) int64 { func (e *environment) percent(name, defaultValue string) int64 {
percent, err := ParsePercent(e.value(name, defaultValue)) percent, err := parsePercent(e.value(name, defaultValue))
e.check(name, err) e.check(name, err)
return percent return percent
} }
// listURL reads a setting that is the URL a list is fetched from, "" while
// it is unset or empty.
func (e *environment) listURL(name string) string {
value := e.value(name, "")
if value != "" && !isListURL(value) {
e.check(name, fmt.Errorf("%q %w", value, errNotListURL))
}
return value
}
// listURLs reads a setting that is a list of the URLs lists are fetched
// from. It is empty by default.
func (e *environment) listURLs(name string) []string {
urls, err := parseListURLs(e.value(name, ""))
e.check(name, err)
return urls
}
// refresh reads the setting that is how long after a list was last
// fetched or tried it is fetched again: a duration of an hour or more,
// since the Spamhaus lists may be fetched no more often, which cannot be
// off.
func (e *environment) refresh(name, defaultValue string) time.Duration {
value := e.value(name, defaultValue)
duration, err := parseDuration(value)
if err != nil || duration < time.Hour {
e.check(name, fmt.Errorf("%q %w", value, errNotAnHourOrMore))
}
return duration
}
// action reads a setting that is what is done with a client a blocklist
// or a DNSBL zone lists: deny, log, or limit:<percent>, which it returns
// as limit and the percentage.
func (e *environment) action(name, defaultValue string) (string, int64) {
value := e.value(name, defaultValue)
if value == "deny" || value == "log" {
return value, 0
}
percentText, isLimit := strings.CutPrefix(value, "limit:")
percent, err := ParsePercent(percentText)
if !isLimit || err != nil {
e.check(name, fmt.Errorf("%q %w", value, errNotAction))
}
return "limit", percent
}
// zones reads the setting that is the list of DNSBL zones. It is empty by
// default. The log shows each zone with its key masked, as MaskZoneKey
// masks it.
func (e *environment) zones(name string) []string {
value, _ := e.lookup(name)
zones, err := parseZones(value)
e.check(name, err)
logged := make([]string, len(zones))
for i, zone := range zones {
logged[i] = MaskZoneKey(zone)
}
e.settings = append(e.settings, slog.String(name, strings.Join(logged, ",")))
return zones
}
// resolver reads the setting that is the resolver the DNSBL zones are
// asked through, the zero AddrPort while it is unset or empty.
func (e *environment) resolver(name string) netip.AddrPort {
resolver, err := parseResolver(e.value(name, ""))
e.check(name, err)
return resolver
}
// lookupSource reads the setting that is where clients are looked up: // lookupSource reads the setting that is where clients are looked up:
// geojs, file, or off. // geojs, file, or off.
func (e *environment) lookupSource(name, defaultValue string) string { func (e *environment) lookupSource(name, defaultValue string) string {
@@ -863,9 +698,8 @@ func (e *environment) checkLookupDBPath(cfg *Config) {
// checkCountriesAndLookups refuses a country on both country lists, and, // checkCountriesAndLookups refuses a country on both country lists, and,
// while SWWAF_LOOKUP_SOURCE is off, each setting that needs clients looked // while SWWAF_LOOKUP_SOURCE is off, each setting that needs clients looked
// up: the country lists, SWWAF_ADD_LOOKUP_HEADERS, the biased thresholds, // up: the country lists, SWWAF_ADD_LOOKUP_HEADERS, the biased thresholds,
// SWWAF_ASN_LIMIT_PERCENT_URL among them, of which // of which SWWAF_UNKNOWN_LIMIT_PERCENT needs them only below 100, where it
// SWWAF_UNKNOWN_LIMIT_PERCENT needs them only below 100, where it lowers a // lowers a limit, and the anomaly thresholds per AS number.
// limit, and the anomaly thresholds per AS number.
func (e *environment) checkCountriesAndLookups(cfg *Config) { func (e *environment) checkCountriesAndLookups(cfg *Config) {
for _, country := range cfg.ExclusivelyAllowedCountries { for _, country := range cfg.ExclusivelyAllowedCountries {
if slices.Contains(cfg.DeniedCountries, country) { if slices.Contains(cfg.DeniedCountries, country) {
@@ -890,7 +724,6 @@ func (e *environment) checkCountriesAndLookups(cfg *Config) {
{"SWWAF_ASN_BYTES_PERCENT", len(cfg.ASNBytesPercent) > 0}, {"SWWAF_ASN_BYTES_PERCENT", len(cfg.ASNBytesPercent) > 0},
{"SWWAF_COUNTRY_BYTES_PERCENT", len(cfg.CountryBytesPercent) > 0}, {"SWWAF_COUNTRY_BYTES_PERCENT", len(cfg.CountryBytesPercent) > 0},
{"SWWAF_UNKNOWN_LIMIT_PERCENT", cfg.UnknownLimitPercent < 100}, {"SWWAF_UNKNOWN_LIMIT_PERCENT", cfg.UnknownLimitPercent < 100},
{"SWWAF_ASN_LIMIT_PERCENT_URL", cfg.ASNLimitPercentURL != ""},
{"SWWAF_ANOMALY_ASN_REQUESTS_PER_MINUTE", cfg.AnomalyASN.RequestsPerMinute > 0}, {"SWWAF_ANOMALY_ASN_REQUESTS_PER_MINUTE", cfg.AnomalyASN.RequestsPerMinute > 0},
{"SWWAF_ANOMALY_ASN_REQUESTS_PER_HOUR", cfg.AnomalyASN.RequestsPerHour > 0}, {"SWWAF_ANOMALY_ASN_REQUESTS_PER_HOUR", cfg.AnomalyASN.RequestsPerHour > 0},
{"SWWAF_ANOMALY_ASN_BYTES_PER_MINUTE", cfg.AnomalyASN.BytesPerMinute > 0}, {"SWWAF_ANOMALY_ASN_BYTES_PER_MINUTE", cfg.AnomalyASN.BytesPerMinute > 0},
@@ -903,15 +736,6 @@ func (e *environment) checkCountriesAndLookups(cfg *Config) {
} }
} }
// checkASNLimitPercentURL refuses SWWAF_ASN_LIMIT_PERCENT_URL naming a
// blocklist too: the file at a URL is fetched as one list or the other.
func (e *environment) checkASNLimitPercentURL(cfg *Config) {
if slices.Contains(cfg.BlocklistURLs, cfg.ASNLimitPercentURL) {
e.check("SWWAF_ASN_LIMIT_PERCENT_URL",
fmt.Errorf("%q %w", cfg.ASNLimitPercentURL, errInBlocklistURLs))
}
}
// headerNames reads a setting that is a list of header names, and // headerNames reads a setting that is a list of header names, and
// returns them in lower case. // returns them in lower case.
func (e *environment) headerNames(name, defaultValue string) []string { func (e *environment) headerNames(name, defaultValue string) []string {
@@ -963,37 +787,6 @@ func (e *environment) v6Prefix(name, defaultValue string) int {
return length return length
} }
// ipv6GroupPrefix reads the setting that is the length of the IPv6
// netblock that is one client, from minIPv6GroupPrefix to 128.
func (e *environment) ipv6GroupPrefix(name, defaultValue string) int {
value := e.value(name, defaultValue)
length, err := strconv.Atoi(value)
if err != nil || length < minIPv6GroupPrefix || length > ipv6Bits {
e.check(name, fmt.Errorf("%q %w", value, errNotIPv6GroupPrefix))
}
return length
}
// logLevel reads the setting that is the least severe of the process's
// own messages that are written: debug, info, warn or error.
func (e *environment) logLevel(name, defaultValue string) slog.Level {
value := e.value(name, defaultValue)
level, known := map[string]slog.Level{
"debug": slog.LevelDebug,
"info": slog.LevelInfo,
"warn": slog.LevelWarn,
"error": slog.LevelError,
}[value]
if !known {
e.check(name, fmt.Errorf("%q %w", value, errNotLogLevel))
}
return level
}
// thresholds reads the four anomaly thresholds whose settings' names // thresholds reads the four anomaly thresholds whose settings' names
// start with prefix: requests and bytes per minute and per hour. Each is // start with prefix: requests and bytes per minute and per hour. Each is
// off by default. // off by default.
@@ -1171,10 +964,10 @@ func (e *environment) webhookHeaders(name string) http.Header {
} }
// secret reads a setting that is a secret another service gave, such as // secret reads a setting that is a secret another service gave, such as
// an ntfy token or an AbuseIPDB key, "" while it is unset. It is sent in // an ntfy token, "" while it is unset. It is sent in a header, which
// a header, which cannot hold a control character, so one in it is an // cannot hold a control character, so one in it is an error. The log
// error. The log shows ******** in place of a value that is not empty, and // shows ******** in place of a value that is not empty, and an error
// an error shows none of it. // shows none of it.
func (e *environment) secret(name string) string { func (e *environment) secret(name string) string {
value, _ := e.lookup(name) value, _ := e.lookup(name)
@@ -1394,7 +1187,7 @@ func parseNetblocks(value string) ([]netip.Prefix, error) {
netblocks := make([]netip.Prefix, 0, len(items)) netblocks := make([]netip.Prefix, 0, len(items))
for _, item := range items { for _, item := range items {
netblock, err := ParseNetblock(item) netblock, err := parseNetblock(item)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -1405,10 +1198,9 @@ func parseNetblocks(value string) ([]netip.Prefix, error) {
return netblocks, nil return netblocks, nil
} }
// ParseNetblock reads a netblock in CIDR form, such as 10.0.0.0/8. A bare // parseNetblock reads a netblock in CIDR form, such as 10.0.0.0/8. A bare
// address is a netblock of that address alone, a /32 or a /128. A // address is a netblock of that address alone, a /32 or a /128.
// blocklist's lines are read with it too. func parseNetblock(value string) (netip.Prefix, error) {
func ParseNetblock(value string) (netip.Prefix, error) {
if strings.Contains(value, "/") { if strings.Contains(value, "/") {
netblock, err := netip.ParsePrefix(value) netblock, err := netip.ParsePrefix(value)
if err != nil { if err != nil {
@@ -1445,7 +1237,7 @@ func parseNamedNetblocks(value string) ([]anomaly.NamedNetblock, error) {
return nil, fmt.Errorf("%q %w", item, errNotNamedNetblock) return nil, fmt.Errorf("%q %w", item, errNotNamedNetblock)
} }
netblock, err := ParseNetblock(strings.TrimSpace(netblockText)) netblock, err := parseNetblock(strings.TrimSpace(netblockText))
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -1545,11 +1337,10 @@ func parseCountry(value string) (string, error) {
return country, nil return country, nil
} }
// ParseASN reads an AS number such as AS64496, in either case, and // parseASN reads an AS number such as AS64496, in either case, and
// returns it as the lookup gives it: AS and the number, in capitals and // returns it as the lookup gives it: AS and the number, in capitals and
// without leading zeros. The file SWWAF_ASN_LIMIT_PERCENT_URL names is // without leading zeros.
// read with it too. func parseASN(value string) (string, error) {
func ParseASN(value string) (string, error) {
digits, hasAS := strings.CutPrefix(strings.ToUpper(value), "AS") digits, hasAS := strings.CutPrefix(strings.ToUpper(value), "AS")
number, err := strconv.ParseUint(digits, 10, 32) number, err := strconv.ParseUint(digits, 10, 32)
@@ -1585,7 +1376,7 @@ func parsePercents(
return nil, err return nil, err
} }
percent, err := ParsePercent(percentText) percent, err := parsePercent(percentText)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -1600,9 +1391,8 @@ func parsePercents(
return percents, nil return percents, nil
} }
// ParsePercent reads a percentage, a whole number from 0 to 100. The file // parsePercent reads a percentage, a whole number from 0 to 100.
// SWWAF_ASN_LIMIT_PERCENT_URL names is read with it too. func parsePercent(value string) (int64, error) {
func ParsePercent(value string) (int64, error) {
percent, err := strconv.ParseInt(value, 10, 64) percent, err := strconv.ParseInt(value, 10, 64)
if err != nil || percent < 0 || percent > 100 { if err != nil || percent < 0 || percent > 100 {
return 0, fmt.Errorf("%q %w", value, errNotPercent) return 0, fmt.Errorf("%q %w", value, errNotPercent)
@@ -1758,7 +1548,16 @@ func parseWebhookURL(value string) (*url.URL, string, error) {
} }
webhook, err := url.Parse(value) webhook, err := url.Parse(value)
if err != nil || !isHTTPURL(webhook) { if err != nil {
return nil, "", errNotWebhookURL
}
port, err := strconv.ParseUint(webhook.Port(), 10, 16)
valid := (webhook.Scheme == "http" || webhook.Scheme == "https") &&
webhook.Hostname() != "" && (webhook.Port() == "" || (err == nil && port != 0)) &&
webhook.User == nil && webhook.Opaque == "" && webhook.Fragment == ""
if !valid {
return nil, "", errNotWebhookURL return nil, "", errNotWebhookURL
} }
@@ -1770,154 +1569,6 @@ func parseWebhookURL(value string) (*url.URL, string, error) {
return webhook, logged, nil return webhook, logged, nil
} }
// isHTTPURL reports whether u is http or https, with a host, and an
// optional port from 1 to 65535, path and query, without a user or a
// fragment.
func isHTTPURL(u *url.URL) bool {
port, err := strconv.ParseUint(u.Port(), 10, 16)
return (u.Scheme == "http" || u.Scheme == "https") && u.Hostname() != "" &&
(u.Port() == "" || (err == nil && port != 0)) &&
u.User == nil && u.Opaque == "" && u.Fragment == ""
}
// isListURL reports whether value is a URL a list can be fetched from, as
// isHTTPURL says.
func isListURL(value string) bool {
u, err := url.Parse(value)
return err == nil && isHTTPURL(u)
}
// parseListURLs reads a comma-separated list of the URLs lists are fetched
// from. A URL listed twice is an error: it would be fetched twice as
// often.
func parseListURLs(value string) ([]string, error) {
urls, err := parseList(value)
if err != nil {
return nil, err
}
for i, listURL := range urls {
if !isListURL(listURL) {
return nil, fmt.Errorf("%q %w", listURL, errNotListURL)
}
if slices.Contains(urls[:i], listURL) {
return nil, fmt.Errorf("%q %w", listURL, errListedTwice)
}
}
return urls, nil
}
const (
// maxZoneLength is the most characters a DNSBL zone may have: 253, the
// most a DNS name may have, less the 64 that come before the zone in
// the name an IPv6 client is asked about by, its 32 hex digits each
// followed by a dot.
maxZoneLength = 189
// maxLabelLength is the most characters a label of a DNS name may have.
maxLabelLength = 63
// labelChars are the characters a label of a DNS zone may hold.
labelChars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-"
// dnsPort is the port a resolver is asked on when SWWAF_DNSBL_RESOLVER
// gives none.
dnsPort = 53
)
// parseZones reads a comma-separated list of DNSBL zones, each a DNS name
// such as dnsbl.dronebl.org: labels separated by dots, each of 1 to 63
// letters, digits and hyphens, neither starting nor ending with a hyphen,
// and at most maxZoneLength characters in all. Go's resolver takes any
// other name for one that does not exist, so that the zone would list no
// client. A zone listed twice is an error, whatever the case of its
// letters, which DNS names ignore, and whatever its key, since
// MaskZoneKey shows two keys of one zone alike. An error shows a zone as
// MaskZoneKey does.
func parseZones(value string) ([]string, error) {
zones, err := parseList(value)
if err != nil {
// parseList's error, for an empty item, shows the whole value, keys
// included.
return nil, errEmptyItem
}
for i, zone := range zones {
shown := MaskZoneKey(zone)
listedBefore := slices.ContainsFunc(zones[:i], func(earlier string) bool {
return strings.EqualFold(MaskZoneKey(earlier), shown)
})
switch {
case len(zone) > maxZoneLength:
return nil, fmt.Errorf("%q %w", shown, errZoneTooLong)
case !isZone(zone):
return nil, fmt.Errorf("%q %w", shown, errNotZone)
case listedBefore:
return nil, fmt.Errorf("%q %w", shown, errListedTwice)
}
}
return zones, nil
}
// MaskZoneKey returns zone with ******** in place of its key, if it is a
// zone of Spamhaus's keyed query service, a name under dq.spamhaus.net,
// such as <key>.xbl.dq.spamhaus.net, whose first label is the key. Any
// other zone it returns as it is. A zone is shown so wherever it leaves
// the process: in the log, the alerts and the metrics.
func MaskZoneKey(zone string) string {
// DNS names ignore case, and a name may be written with a dot at its
// end.
name := strings.TrimSuffix(strings.ToLower(zone), ".")
if !strings.HasSuffix(name, ".dq.spamhaus.net") {
return zone
}
_, rest, _ := strings.Cut(zone, ".")
return masked + "." + rest
}
// isZone reports whether each label of zone is as parseZones takes it.
func isZone(zone string) bool {
for label := range strings.SplitSeq(zone, ".") {
badChar := strings.ContainsFunc(label, func(char rune) bool {
return !strings.ContainsRune(labelChars, char)
})
if label == "" || len(label) > maxLabelLength || badChar ||
strings.HasPrefix(label, "-") || strings.HasSuffix(label, "-") {
return false
}
}
return true
}
// parseResolver reads the resolver the DNSBL zones are asked through: an
// IP address with a port from 1 to 65535, such as 192.0.2.53:5353 or
// [2001:db8::53]:5353, or without one, such as 192.0.2.53 or 2001:db8::53,
// for port 53. An empty value is none, the zero AddrPort.
func parseResolver(value string) (netip.AddrPort, error) {
if value == "" {
return netip.AddrPort{}, nil
}
resolver, err := netip.ParseAddrPort(value)
if err == nil && resolver.Port() != 0 {
return resolver, nil
}
addr, err := netip.ParseAddr(value)
if err != nil {
return netip.AddrPort{}, fmt.Errorf("%q %w", value, errNotResolver)
}
return netip.AddrPortFrom(addr, dnsPort), nil
}
// parseWebhookHeaders reads a comma-separated list of headers, each its // parseWebhookHeaders reads a comma-separated list of headers, each its
// name, :, and its value, and returns them, and how the log shows them, // name, :, and its value, and returns them, and how the log shows them,
// with each value as ********. An error names the item by its place in // with each value as ********. An error names the item by its place in
internal/config/config_test.go
+2 -471
View File
@@ -27,8 +27,6 @@ const (
upstreamURL = "SWWAF_UPSTREAM_URL" upstreamURL = "SWWAF_UPSTREAM_URL"
mode = "SWWAF_MODE" mode = "SWWAF_MODE"
trustedProxies = "SWWAF_TRUSTED_PROXIES" trustedProxies = "SWWAF_TRUSTED_PROXIES"
ipv6GroupPrefix = "SWWAF_IPV6_GROUP_PREFIX"
maxTrackedClients = "SWWAF_MAX_TRACKED_CLIENTS"
clientRequestTimeout = "SWWAF_CLIENT_REQUEST_TIMEOUT" clientRequestTimeout = "SWWAF_CLIENT_REQUEST_TIMEOUT"
clientHeaderMaxBytes = "SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES" clientHeaderMaxBytes = "SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES"
clientIdleTimeout = "SWWAF_CLIENT_IDLE_TIMEOUT" clientIdleTimeout = "SWWAF_CLIENT_IDLE_TIMEOUT"
@@ -59,18 +57,6 @@ const (
asnBytesPercent = "SWWAF_ASN_BYTES_PERCENT" asnBytesPercent = "SWWAF_ASN_BYTES_PERCENT"
countryBytesPercent = "SWWAF_COUNTRY_BYTES_PERCENT" countryBytesPercent = "SWWAF_COUNTRY_BYTES_PERCENT"
unknownLimitPercent = "SWWAF_UNKNOWN_LIMIT_PERCENT" unknownLimitPercent = "SWWAF_UNKNOWN_LIMIT_PERCENT"
asnLimitPercentURL = "SWWAF_ASN_LIMIT_PERCENT_URL"
blocklistURLs = "SWWAF_BLOCKLIST_URLS"
blocklistRefresh = "SWWAF_BLOCKLIST_REFRESH"
blocklistAction = "SWWAF_BLOCKLIST_ACTION"
dnsblZones = "SWWAF_DNSBL_ZONES"
dnsblResolver = "SWWAF_DNSBL_RESOLVER"
abuseIPDBKey = "SWWAF_ABUSEIPDB_KEY"
abuseIPDBMinScore = "SWWAF_ABUSEIPDB_MIN_SCORE"
abuseIPDBDailyBudget = "SWWAF_ABUSEIPDB_DAILY_BUDGET"
reputationAction = "SWWAF_REPUTATION_ACTION"
reputationCacheTTL = "SWWAF_REPUTATION_CACHE_TTL"
reputationTimeout = "SWWAF_REPUTATION_TIMEOUT"
banResponse = "SWWAF_BAN_RESPONSE" banResponse = "SWWAF_BAN_RESPONSE"
limitBanDuration = "SWWAF_LIMIT_BAN_DURATION" limitBanDuration = "SWWAF_LIMIT_BAN_DURATION"
limitBanRepeatWindow = "SWWAF_LIMIT_BAN_REPEAT_WINDOW" limitBanRepeatWindow = "SWWAF_LIMIT_BAN_REPEAT_WINDOW"
@@ -86,7 +72,6 @@ const (
metricsTopN = "SWWAF_METRICS_TOP_N" metricsTopN = "SWWAF_METRICS_TOP_N"
instanceName = "SWWAF_INSTANCE_NAME" instanceName = "SWWAF_INSTANCE_NAME"
logRequestHeaders = "SWWAF_LOG_REQUEST_HEADERS" logRequestHeaders = "SWWAF_LOG_REQUEST_HEADERS"
logLevel = "SWWAF_LOG_LEVEL"
rulesDir = "SWWAF_RULES_DIR" rulesDir = "SWWAF_RULES_DIR"
rulesEnabled = "SWWAF_RULES_ENABLED" rulesEnabled = "SWWAF_RULES_ENABLED"
logRemoteURL = "SWWAF_LOG_REMOTE_URL" logRemoteURL = "SWWAF_LOG_REMOTE_URL"
@@ -163,9 +148,6 @@ const (
defaultAlertCooldown = "15m" defaultAlertCooldown = "15m"
) )
// defaultReputationCacheTTL is the default of SWWAF_REPUTATION_CACHE_TTL.
const defaultReputationCacheTTL = "24h"
// defaultLogRequestHeaders is the default of SWWAF_LOG_REQUEST_HEADERS. // defaultLogRequestHeaders is the default of SWWAF_LOG_REQUEST_HEADERS.
const defaultLogRequestHeaders = "accept,accept-language,accept-encoding," + const defaultLogRequestHeaders = "accept,accept-language,accept-encoding," +
"content-type,origin,range" "content-type,origin,range"
@@ -402,57 +384,6 @@ func TestValuesAsSet(t *testing.T) {
wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries, "DE") wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries, "DE")
} }
func TestIPv6GroupPrefixMaxTrackedClientsAndLogLevel(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
env environment
prefix, clients int
level slog.Level
}{
{environment{}, 64, 20000, slog.LevelInfo},
{
environment{ipv6GroupPrefix: "48", maxTrackedClients: "500", logLevel: "warn"},
48, 500, slog.LevelWarn,
},
} {
cfg := fromEnvironment(t, tc.env)
if cfg.IPv6GroupPrefix != tc.prefix || cfg.MaxTrackedClients != tc.clients ||
cfg.LogLevel != tc.level {
t.Errorf("%v gave %d, %d and %v, want %d, %d and %v", tc.env,
cfg.IPv6GroupPrefix, cfg.MaxTrackedClients, cfg.LogLevel,
tc.prefix, tc.clients, tc.level)
}
}
}
func TestIPv6GroupPrefixFrom32To128(t *testing.T) {
t.Parallel()
for _, length := range []int{32, 128} {
cfg := fromEnvironment(t, environment{ipv6GroupPrefix: strconv.Itoa(length)})
if cfg.IPv6GroupPrefix != length {
t.Errorf("%s=%d gave %d", ipv6GroupPrefix, length, cfg.IPv6GroupPrefix)
}
}
}
func TestEachLogLevel(t *testing.T) {
t.Parallel()
for value, want := range map[string]slog.Level{
"debug": slog.LevelDebug,
"info": slog.LevelInfo,
"warn": slog.LevelWarn,
"error": slog.LevelError,
} {
cfg := fromEnvironment(t, environment{logLevel: value})
if cfg.LogLevel != want {
t.Errorf("%s=%s gave %v, want %v", logLevel, value, cfg.LogLevel, want)
}
}
}
func TestByteLimitSettingsAsSet(t *testing.T) { func TestByteLimitSettingsAsSet(t *testing.T) {
t.Parallel() t.Parallel()
@@ -1022,7 +953,6 @@ func TestSettingNeedingLookupsStopsTheStartWhileTheyAreOff(t *testing.T) {
deniedCountries: "kp", deniedCountries: "kp",
allowedCountries: "de", allowedCountries: "de",
addLookupHeaders: enabled, addLookupHeaders: enabled,
asnLimitPercentURL: asnURL,
asnLimitPercent: "AS64496:50", asnLimitPercent: "AS64496:50",
countryLimitPercent: "cn:25", countryLimitPercent: "cn:25",
asnBytesPercent: "AS64496:50", asnBytesPercent: "AS64496:50",
@@ -1048,13 +978,11 @@ func TestSettingNeedingLookupsStopsTheStartWhileTheyAreOff(t *testing.T) {
// Set empty, the lists need nothing looked up, and nor does // Set empty, the lists need nothing looked up, and nor does
// SWWAF_UNKNOWN_LIMIT_PERCENT at 100, which lowers no limit, an anomaly // SWWAF_UNKNOWN_LIMIT_PERCENT at 100, which lowers no limit, an anomaly
// threshold per AS number that is off, any other anomaly threshold, or // threshold per AS number that is off, or any other anomaly threshold.
// a blocklist.
env := environment{ env := environment{
lookupSource: off, deniedCountries: "", allowedCountries: "", lookupSource: off, deniedCountries: "", allowedCountries: "",
asnLimitPercent: "", countryLimitPercent: "", asnBytesPercent: "", asnLimitPercent: "", countryLimitPercent: "", asnBytesPercent: "",
countryBytesPercent: "", unknownLimitPercent: "100", asnLimitPercentURL: "", countryBytesPercent: "", unknownLimitPercent: "100",
blocklistURLs: dropURL,
} }
for _, name := range anomalyThresholds() { for _, name := range anomalyThresholds() {
env[name] = "1000" env[name] = "1000"
@@ -1277,382 +1205,6 @@ func TestInvalidBiasedThresholdStopsTheStartSayingWhatIsWrong(t *testing.T) {
} }
} }
// dropURL and torURL are blocklists, and asnURL a file of AS:percent
// lines.
const (
dropURL = "https://www.spamhaus.org/drop/drop.txt"
torURL = "https://lists.example/tor-exits.txt"
asnURL = "https://lists.example/asn.txt"
)
// The actions of SWWAF_BLOCKLIST_ACTION, as Config gives them.
const (
actionDeny = "deny"
actionLimit = "limit"
actionLog = "log"
)
func TestReputationSettingsAsSet(t *testing.T) {
t.Parallel()
cfg := fromEnvironment(t, environment{})
if len(cfg.BlocklistURLs) != 0 || cfg.BlocklistRefresh != 24*time.Hour ||
cfg.BlocklistAction != actionDeny || cfg.ASNLimitPercentURL != "" {
t.Errorf("%s, %s, %s and %s gave %v, %s, %s and %q by default, "+
"want none, 24h, deny and none", blocklistURLs, blocklistRefresh,
blocklistAction, asnLimitPercentURL, cfg.BlocklistURLs, cfg.BlocklistRefresh,
cfg.BlocklistAction, cfg.ASNLimitPercentURL)
}
for _, tc := range []struct {
value, action string
percent int64
}{
{actionDeny, actionDeny, 0},
{actionLog, actionLog, 0},
{"limit:25", actionLimit, 25},
{"limit:0", actionLimit, 0},
} {
// An hour, the shortest refresh allowed.
cfg := fromEnvironment(t, environment{
blocklistURLs: dropURL + ", " + torURL, blocklistRefresh: "1h",
blocklistAction: tc.value, asnLimitPercentURL: asnURL,
})
if !slices.Equal(cfg.BlocklistURLs, []string{dropURL, torURL}) ||
cfg.BlocklistRefresh != time.Hour || cfg.BlocklistAction != tc.action ||
cfg.BlocklistLimitPercent != tc.percent || cfg.ASNLimitPercentURL != asnURL {
t.Errorf("%s=%s gave %v, %s, %s, %d and %s", blocklistAction, tc.value,
cfg.BlocklistURLs, cfg.BlocklistRefresh, cfg.BlocklistAction,
cfg.BlocklistLimitPercent, cfg.ASNLimitPercentURL)
}
}
}
func TestInvalidReputationSettingStopsTheStartSayingWhatIsWrong(t *testing.T) {
t.Parallel()
const (
notURL = " is not an http or https URL without a user or a fragment, " +
"such as https://www.spamhaus.org/drop/drop.txt"
notAnHour = " is not a duration of 1h or more, such as 24h"
notAction = " is not deny, limit:<percent> such as limit:25, or log"
)
for _, tc := range []struct{ name, value, want string }{
{
blocklistURLs, "ftp://lists.example/drop.txt",
`"ftp://lists.example/drop.txt"` + notURL,
},
{blocklistURLs, "lists.example/drop.txt", `"lists.example/drop.txt"` + notURL},
{
blocklistURLs, "https://me:secret@lists.example/drop.txt",
`"https://me:secret@lists.example/drop.txt"` + notURL,
},
{
blocklistURLs, dropURL + "," + torURL + "," + dropURL,
`"` + dropURL + `" is listed twice`,
},
{asnLimitPercentURL, asnURL + "#top", `"` + asnURL + `#top"` + notURL},
{blocklistRefresh, "59m", `"59m"` + notAnHour},
{blocklistRefresh, off, `"off"` + notAnHour},
{blocklistRefresh, "a day", `"a day"` + notAnHour},
{blocklistAction, "block", `"block"` + notAction},
{blocklistAction, actionLimit, `"limit"` + notAction},
{blocklistAction, "limit:101", `"limit:101"` + notAction},
} {
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
t.Parallel()
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
want := tc.name + ": " + tc.want
if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want)
}
})
}
}
func TestASNLimitPercentURLThatIsABlocklistStopsTheStart(t *testing.T) {
t.Parallel()
_, err := config.FromEnvironment(environment{
blocklistURLs: dropURL + "," + asnURL, asnLimitPercentURL: asnURL,
}.lookupEnv)
want := asnLimitPercentURL + `: "` + asnURL + `" is in SWWAF_BLOCKLIST_URLS too`
if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want)
}
}
// dronebl is a DNSBL zone, and spamhaus one of Spamhaus's, a name
// containing spamhausKey, the key of its keyed query service, which the
// log shows as spamhausMasked.
const (
dronebl = "dnsbl.dronebl.org"
spamhausKey = "abcdefghijklmnopqrstuvwxyz"
spamhaus = spamhausKey + ".xbl.dq.spamhaus.net"
spamhausMasked = "********.xbl.dq.spamhaus.net"
)
func TestDNSBLSettingsAsSet(t *testing.T) {
t.Parallel()
cfg := fromEnvironment(t, environment{})
if len(cfg.DNSBLZones) != 0 || cfg.DNSBLResolver.IsValid() ||
cfg.ReputationAction != actionLimit || cfg.ReputationLimitPercent != 25 ||
cfg.ReputationCacheTTL != 24*time.Hour || cfg.ReputationTimeout != 2*time.Second {
t.Errorf("by default, the zones %v, the resolver %s, the action %s:%d, the TTL %s "+
"and the timeout %s, want no zone, no resolver, limit:25, 24h and 2s",
cfg.DNSBLZones,
cfg.DNSBLResolver, cfg.ReputationAction, cfg.ReputationLimitPercent,
cfg.ReputationCacheTTL, cfg.ReputationTimeout)
}
// The longest zone allowed, of 189 characters, with labels of 63, the
// longest allowed.
longest := strings.Repeat("a", 63) + "." + strings.Repeat("b", 63) + "." +
strings.Repeat("c", 61)
for _, tc := range []struct {
zones, resolver, action string
// want are the zones, resolver, action and percent Config gives.
want []string
wantResolver string
wantAction string
wantPercent int64
}{
{
dronebl + ", " + spamhaus, "192.0.2.53", actionDeny,
[]string{dronebl, spamhaus}, "192.0.2.53:53", actionDeny, 0,
},
{
longest, "192.0.2.53:5353", actionLog,
[]string{longest}, "192.0.2.53:5353", actionLog, 0,
},
{
"Dnsbl-1.Example", "2001:db8::53", "limit:10",
[]string{"Dnsbl-1.Example"}, "[2001:db8::53]:53", actionLimit, 10,
},
{
dronebl, "[2001:db8::53]:5353", "limit:0",
[]string{dronebl}, "[2001:db8::53]:5353", actionLimit, 0,
},
} {
cfg := fromEnvironment(t, environment{
dnsblZones: tc.zones, dnsblResolver: tc.resolver, reputationAction: tc.action,
reputationCacheTTL: "12h", reputationTimeout: "3s",
})
if !slices.Equal(cfg.DNSBLZones, tc.want) ||
cfg.DNSBLResolver.String() != tc.wantResolver ||
cfg.ReputationAction != tc.wantAction ||
cfg.ReputationLimitPercent != tc.wantPercent ||
cfg.ReputationCacheTTL != 12*time.Hour ||
cfg.ReputationTimeout != 3*time.Second {
t.Errorf("%s=%s, %s=%s and %s=%s gave %v, %s, %s:%d, %s and %s", dnsblZones,
tc.zones, dnsblResolver, tc.resolver, reputationAction, tc.action,
cfg.DNSBLZones, cfg.DNSBLResolver, cfg.ReputationAction,
cfg.ReputationLimitPercent, cfg.ReputationCacheTTL, cfg.ReputationTimeout)
}
}
}
func TestInvalidDNSBLSettingStopsTheStartSayingWhatIsWrong(t *testing.T) {
t.Parallel()
const (
notZone = " is not a DNS zone such as dnsbl.dronebl.org"
notResolver = " is not an IP address with an optional port, such as 192.0.2.53 " +
"or [2001:db8::53]:5353"
notAboveZero = " is not a duration above zero, such as 1h or 7d"
)
label64 := strings.Repeat("a", 64) + ".example"
tooLong := strings.Repeat("a", 63) + "." + strings.Repeat("b", 63) + "." +
strings.Repeat("c", 62)
for _, tc := range []struct{ name, value, want string }{
{dnsblZones, "dnsbl..example", `"dnsbl..example"` + notZone},
{dnsblZones, "dnsbl.example.", `"dnsbl.example."` + notZone},
{dnsblZones, "-dnsbl.example", `"-dnsbl.example"` + notZone},
{dnsblZones, "dnsbl-.example", `"dnsbl-.example"` + notZone},
{dnsblZones, "dns_bl.example", `"dns_bl.example"` + notZone},
{dnsblZones, label64, `"` + label64 + `"` + notZone},
{
dnsblZones, tooLong,
`"` + tooLong + `" is longer than 189 characters, too long for the names ` +
`IPv6 clients are asked about by`,
},
{
dnsblZones, dronebl + "," + spamhaus + "," + dronebl,
`"` + dronebl + `" is listed twice`,
},
// DNS names ignore case.
{dnsblZones, "dnsbl.example,DNSBL.example", `"DNSBL.example" is listed twice`},
{dnsblResolver, "resolver.example", `"resolver.example"` + notResolver},
{dnsblResolver, "192.0.2.53:0", `"192.0.2.53:0"` + notResolver},
{dnsblResolver, "192.0.2.53:65536", `"192.0.2.53:65536"` + notResolver},
{dnsblResolver, "[2001:db8::53]", `"[2001:db8::53]"` + notResolver},
{
reputationAction, "ban",
`"ban" is not deny, limit:<percent> such as limit:25, or log`,
},
{reputationCacheTTL, off, `"off"` + notAboveZero},
{reputationTimeout, "0s", `"0s"` + notAboveZero},
} {
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
t.Parallel()
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
want := tc.name + ": " + tc.want
if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want)
}
})
}
}
func TestMaskZoneKeyMasksTheFirstLabelOfAZoneUnderDqSpamhausNet(t *testing.T) {
t.Parallel()
for zone, want := range map[string]string{
spamhaus: spamhausMasked,
spamhaus + ".": spamhausMasked + ".",
"KEY.ZEN.DQ.SPAMHAUS.NET": "********.ZEN.DQ.SPAMHAUS.NET",
dronebl: dronebl,
"dq.spamhaus.net": "dq.spamhaus.net",
spamhaus + ".example": spamhaus + ".example",
} {
if got := config.MaskZoneKey(zone); got != want {
t.Errorf("MaskZoneKey(%q) is %q, want %q", zone, got, want)
}
}
}
func TestDNSBLZoneKeyIsLoggedMaskedAndNeverShown(t *testing.T) {
t.Parallel()
cfg := fromEnvironment(t, environment{dnsblZones: dronebl + ", " + spamhaus})
var out bytes.Buffer
slog.New(slog.NewJSONHandler(&out, nil)).Info("starting", "settings", cfg)
logged := out.String()
if strings.Contains(logged, spamhausKey) ||
!strings.Contains(logged, `"`+dnsblZones+`":"`+dronebl+","+spamhausMasked+`"`) {
t.Errorf("the zones are not logged with the key masked: %s", logged)
}
// Nor does an error that stops the start show a key, in any case.
const (
notZone = " is not a DNS zone such as dnsbl.dronebl.org"
otherKey = "zyxwvutsrqponmlkjihgfedcba"
otherZone = otherKey + ".xbl.dq.spamhaus.net"
)
// 205 characters, 187 with the key masked.
labels := strings.Repeat("a", 63) + "." + strings.Repeat("b", 63) + "." +
strings.Repeat("c", 30) + ".xbl.dq.spamhaus.net"
for _, tc := range []struct{ value, want string }{
{spamhaus + ".", `"` + spamhausMasked + `."` + notZone},
{spamhausKey + "_.xbl.dq.spamhaus.net", `"` + spamhausMasked + `"` + notZone},
{
spamhausKey + "." + labels,
`"********.` + labels + `" is longer than 189 characters, too long ` +
`for the names IPv6 clients are asked about by`,
},
{
spamhaus + "," + strings.ToUpper(spamhaus),
`"********.XBL.DQ.SPAMHAUS.NET" is listed twice`,
},
{spamhaus + "," + otherZone, `"` + spamhausMasked + `" is listed twice`},
{spamhaus + ",,", "has an empty item in its list"},
} {
_, err := config.FromEnvironment(environment{dnsblZones: tc.value}.lookupEnv)
want := dnsblZones + ": " + tc.want
if err == nil || err.Error() != want {
t.Errorf("%s=%s gave the error %v, want %s", dnsblZones, tc.value, err, want)
}
}
}
func TestAbuseIPDBSettingsAsSet(t *testing.T) {
t.Parallel()
cfg := fromEnvironment(t, environment{})
if cfg.AbuseIPDBKey != "" || cfg.AbuseIPDBMinScore != 75 ||
cfg.AbuseIPDBDailyBudget != 900 {
t.Errorf("by default, the key %q, the minimum score %d and the daily budget %d, "+
"want none, 75 and 900", cfg.AbuseIPDBKey, cfg.AbuseIPDBMinScore,
cfg.AbuseIPDBDailyBudget)
}
cfg = fromEnvironment(t, environment{
abuseIPDBKey: token, abuseIPDBMinScore: "0", abuseIPDBDailyBudget: "1",
})
if cfg.AbuseIPDBKey != token || cfg.AbuseIPDBMinScore != 0 ||
cfg.AbuseIPDBDailyBudget != 1 {
t.Errorf("set, the key %q, the minimum score %d and the daily budget %d, "+
"want %s, 0 and 1", cfg.AbuseIPDBKey, cfg.AbuseIPDBMinScore,
cfg.AbuseIPDBDailyBudget, token)
}
}
func TestInvalidAbuseIPDBSettingStopsTheStartSayingWhatIsWrong(t *testing.T) {
t.Parallel()
const (
notScore = " is not a percentage, a whole number from 0 to 100"
notBudget = " is not a whole number above zero, such as 5000"
)
for _, tc := range []struct{ name, value, want string }{
{abuseIPDBMinScore, "101", `"101"` + notScore},
{abuseIPDBMinScore, off, `"off"` + notScore},
{abuseIPDBDailyBudget, "0", `"0"` + notBudget},
{abuseIPDBDailyBudget, off, `"off"` + notBudget},
// The key itself is never shown.
{
abuseIPDBKey, token + "\r",
"holds a control character, such as the carriage return of a Windows line end",
},
} {
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
t.Parallel()
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
want := tc.name + ": " + tc.want
if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want)
}
})
}
}
func TestAbuseIPDBKeyIsLoggedMasked(t *testing.T) {
t.Parallel()
cfg := fromEnvironment(t, environment{abuseIPDBKey: token})
var out bytes.Buffer
slog.New(slog.NewJSONHandler(&out, nil)).Info("starting", "settings", cfg)
logged := out.String()
if strings.Contains(logged, token) ||
!strings.Contains(logged, `"`+abuseIPDBKey+`":"********"`) {
t.Errorf("the key is not logged masked: %s", logged)
}
}
func TestSizesAndOff(t *testing.T) { func TestSizesAndOff(t *testing.T) {
t.Parallel() t.Parallel()
@@ -1752,10 +1304,6 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
{trustedProxies, "traefik"}, {trustedProxies, "traefik"},
{trustedProxies, "10.0.0.0/8,,192.168.0.0/16"}, {trustedProxies, "10.0.0.0/8,,192.168.0.0/16"},
{trustedProxies, "fe80::1%eth0"}, {trustedProxies, "fe80::1%eth0"},
{ipv6GroupPrefix, "31"}, {ipv6GroupPrefix, "129"}, {ipv6GroupPrefix, "/64"},
{ipv6GroupPrefix, off}, {ipv6GroupPrefix, ""},
{maxTrackedClients, "0"}, {maxTrackedClients, "-1"}, {maxTrackedClients, off},
{maxTrackedClients, "20K"},
{allowNets, "192.0.2.0/24,monitoring"}, {allowNets, "192.0.2.0/24,monitoring"},
{rateLimitExemptNets, "2001:db8::/129"}, {rateLimitExemptNets, "2001:db8::/129"},
{denyNets, "198.51.100.0/24,"}, {denyNets, "198.51.100.0/24,"},
@@ -1809,7 +1357,6 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
{logRequestHeaders, "accept language"}, {logRequestHeaders, "x-foo:"}, {logRequestHeaders, "accept language"}, {logRequestHeaders, "x-foo:"},
{logRequestHeaders, "host"}, {logRequestHeaders, "accept,Host"}, {logRequestHeaders, "host"}, {logRequestHeaders, "accept,Host"},
{logRequestHeaders, "transfer-encoding"}, {logRequestHeaders, "TRANSFER-ENCODING"}, {logRequestHeaders, "transfer-encoding"}, {logRequestHeaders, "TRANSFER-ENCODING"},
{logLevel, "INFO"}, {logLevel, "warning"}, {logLevel, "trace"}, {logLevel, ""},
{rulesEnabled, "yes"}, {rulesEnabled, "True"}, {rulesEnabled, "yes"}, {rulesEnabled, "True"},
}) })
} }
@@ -2040,7 +1587,6 @@ func writeFile(t *testing.T, contents string) string {
return path return path
} }
//nolint:funlen // one line for each setting, a list that grows with them
func TestLogsEachSettingWithItsValue(t *testing.T) { func TestLogsEachSettingWithItsValue(t *testing.T) {
t.Parallel() t.Parallel()
@@ -2052,8 +1598,6 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
upstreamURL: "http://127.0.0.1:8081", upstreamURL: "http://127.0.0.1:8081",
mode: "enforce", mode: "enforce",
trustedProxies: "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16", trustedProxies: "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16",
ipv6GroupPrefix: "64",
maxTrackedClients: "20000",
clientRequestTimeout: "45s", clientRequestTimeout: "45s",
clientHeaderMaxBytes: "32K", clientHeaderMaxBytes: "32K",
clientIdleTimeout: "120s", clientIdleTimeout: "120s",
@@ -2084,18 +1628,6 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
asnBytesPercent: "", asnBytesPercent: "",
countryBytesPercent: "", countryBytesPercent: "",
unknownLimitPercent: "100", unknownLimitPercent: "100",
asnLimitPercentURL: "",
blocklistURLs: "",
blocklistRefresh: "24h",
blocklistAction: actionDeny,
dnsblZones: "",
dnsblResolver: "",
abuseIPDBKey: "",
abuseIPDBMinScore: "75",
abuseIPDBDailyBudget: "900",
reputationAction: "limit:25",
reputationCacheTTL: defaultReputationCacheTTL,
reputationTimeout: "2s",
banResponse: "403", banResponse: "403",
limitBanDuration: "1h", limitBanDuration: "1h",
limitBanRepeatWindow: "24h", limitBanRepeatWindow: "24h",
@@ -2111,7 +1643,6 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
metricsTopN: "50", metricsTopN: "50",
instanceName: hostname, instanceName: hostname,
logRequestHeaders: defaultLogRequestHeaders, logRequestHeaders: defaultLogRequestHeaders,
logLevel: "info",
rulesDir: "/etc/smallwebwaf/rules.d", rulesDir: "/etc/smallwebwaf/rules.d",
rulesEnabled: "true", rulesEnabled: "true",
logRemoteURL: "", logRemoteURL: "",
internal/lookup/lookup.go
+1 -2
View File
@@ -176,8 +176,7 @@ func New(params Params) *GeoJS {
// when it ends. // when it ends.
// //
// GeoJS is asked about the client's first address, which is the client's // GeoJS is asked about the client's first address, which is the client's
// own address for IPv4, and an address in the same place for an IPv6 // own address for IPv4, and an address in the same place for an IPv6 /64.
// netblock.
func (g *GeoJS) LookUp(ctx context.Context, client netip.Prefix) Answer { func (g *GeoJS) LookUp(ctx context.Context, client netip.Prefix) Answer {
answer, asked := g.answerOrWait(ctx, client) answer, asked := g.answerOrWait(ctx, client)
if asked == nil { if asked == nil {
internal/metrics/metrics.go
+4 -108
View File
@@ -14,10 +14,8 @@ import (
"github.com/prometheus/client_golang/prometheus/promhttp" "github.com/prometheus/client_golang/prometheus/promhttp"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/remotelog" "sneak.berlin/go/smallwebwaf/internal/remotelog"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules" "sneak.berlin/go/smallwebwaf/internal/rules"
) )
@@ -37,12 +35,10 @@ type Metrics struct {
rateLimitHits *prometheus.CounterVec rateLimitHits *prometheus.CounterVec
sizeAndTimeLimitHits *prometheus.CounterVec sizeAndTimeLimitHits *prometheus.CounterVec
offences *prometheus.CounterVec offences *prometheus.CounterVec
// ruleMatches are made by AddRules, and reputationHits by // ruleMatches are made by AddRules.
// AddReputation. ruleMatches *prometheus.CounterVec
ruleMatches *prometheus.CounterVec countries *busiest
reputationHits *prometheus.CounterVec asns *busiest
countries *busiest
asns *busiest
// GeoJSRequests are the requests to GeoJS, and GeoJSFailures those // GeoJSRequests are the requests to GeoJS, and GeoJSFailures those
// that failed. GeoJSUnanswered are the requests that needed their // that failed. GeoJSUnanswered are the requests that needed their
@@ -257,80 +253,6 @@ func (m *Metrics) AddLookupFile(lastRead func() time.Time, readFailures func() i
) )
} }
// sourceLabel is the label of the reputation metrics: a list's URL, a
// DNSBL zone, its key masked, or abuseipdb.
const sourceLabel = "source"
// AddReputation adds the metrics of the lists fetched from URLs and of the
// DNSBL zones, by source, each list's URL or each zone, its key masked as
// config.MaskZoneKey masks it: the requests whose client a blocklist, a
// zone's verdict or AbuseIPDB's score lists, which ReputationHit counts,
// and, read from lists and dnsbl as the metrics are asked for, for a list,
// the fetches that failed and when the copy in use was fetched, and for a
// zone, the queries made and those that failed. It is called once, before
// ReputationHit.
func (m *Metrics) AddReputation(lists *reputation.Lists, dnsbl *reputation.DNSBL) {
m.reputationHits = counterVec("smallwebwaf_reputation_hits_total",
"Requests whose client a blocklist, a DNSBL zone or AbuseIPDB lists, by "+
"the blocklist's URL, the zone, or abuseipdb.",
[]string{sourceLabel})
m.registry.MustRegister(m.reputationHits)
for _, zone := range dnsbl.Zones() {
source := prometheus.Labels{sourceLabel: config.MaskZoneKey(zone)}
m.addReputationQueries(source, func() int { return dnsbl.Queries(zone) })
m.addReputationFailures(source, func() int { return dnsbl.Failures(zone) })
}
for _, listURL := range lists.URLs() {
source := prometheus.Labels{sourceLabel: listURL}
m.addReputationFailures(source, func() int { return lists.Failures(listURL) })
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "smallwebwaf_reputation_last_fetch_timestamp_seconds",
Help: "When the copy of the list in use was fetched, in seconds since " +
"1970, or 0 while there is none.",
ConstLabels: source,
}, func() float64 {
fetched := lists.Fetched(listURL)
if fetched.IsZero() {
return 0
}
return float64(fetched.Unix())
}),
)
}
}
// AddAbuseIPDB adds the metrics of AbuseIPDB, with the source abuseipdb,
// read from abuseIPDB as the metrics are asked for: the checks made, those
// that failed, and how many checks the day's budget has left. It is
// called once, after AddReputation, while SWWAF_ABUSEIPDB_KEY is set.
func (m *Metrics) AddAbuseIPDB(abuseIPDB *reputation.AbuseIPDB) {
source := prometheus.Labels{sourceLabel: reputation.AbuseIPDBSource}
m.addReputationQueries(source, abuseIPDB.Checked)
m.addReputationFailures(source, abuseIPDB.Failures)
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "smallwebwaf_reputation_daily_budget_remaining",
Help: "Checks of the day's SWWAF_ABUSEIPDB_DAILY_BUDGET not yet spent.",
ConstLabels: source,
}, func() float64 {
return float64(abuseIPDB.BudgetLeft())
}),
)
}
// ReputationHit counts a request whose client source lists: a blocklist,
// by its URL, a DNSBL zone, its key masked, or AbuseIPDB, abuseipdb.
func (m *Metrics) ReputationHit(source string) {
m.reputationHits.WithLabelValues(source).Inc()
}
// AddAlerts adds the metrics of the alerts sent to each destination set, // AddAlerts adds the metrics of the alerts sent to each destination set,
// read from queue as the metrics are asked for, by destination: the // read from queue as the metrics are asked for, by destination: the
// alerts sent, the requests to the destination that failed, the alerts // alerts sent, the requests to the destination that failed, the alerts
@@ -470,32 +392,6 @@ func (m *Metrics) StateFileEditSetAside(name string) {
m.stateFileEditsSetAside.WithLabelValues(name).Inc() m.stateFileEditsSetAside.WithLabelValues(name).Inc()
} }
// addReputationQueries adds the counter of the queries to source, a DNSBL
// zone, or of the checks of clients with AbuseIPDB, which count tells.
func (m *Metrics) addReputationQueries(source prometheus.Labels, count func() int) {
m.registry.MustRegister(prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "smallwebwaf_reputation_queries_total",
Help: "Queries to the DNSBL zone, or checks of clients with AbuseIPDB.",
ConstLabels: source,
}, func() float64 {
return float64(count())
}))
}
// addReputationFailures adds the counter of the fetches of source, a
// list, the queries to it, a DNSBL zone, or the checks with it, AbuseIPDB,
// that failed, which count tells.
func (m *Metrics) addReputationFailures(source prometheus.Labels, count func() int) {
m.registry.MustRegister(prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "smallwebwaf_reputation_failures_total",
Help: "Fetches of the list, queries to the DNSBL zone, or checks with " +
"AbuseIPDB, that failed.",
ConstLabels: source,
}, func() float64 {
return float64(count())
}))
}
// statusClass returns the class of status, such as 2xx, or none when no // statusClass returns the class of status, such as 2xx, or none when no
// status was sent. // status was sent.
func statusClass(status int) string { func statusClass(status int) string {
internal/proxy/admin.go
+1 -1
View File
@@ -282,7 +282,7 @@ func (rq *request) showClient() {
answer := clientAnswer{Bans: state.BanEntries(rq.h.ledger.Covering(addr))} answer := clientAnswer{Bans: state.BanEntries(rq.h.ledger.Covering(addr))}
client, seen := rq.h.limiter.Client(rq.h.clientGroup(addr)) client, seen := rq.h.limiter.Client(clientGroup(addr))
if seen { if seen {
answer.Client = &client answer.Client = &client
} }
internal/proxy/bans.go
+4 -4
View File
@@ -45,7 +45,7 @@ func (rq *request) banned(now time.Time) bool {
// the client over a rate limit, as its limit percentage lowers it, which // the client over a rate limit, as its limit percentage lowers it, which
// breaks it. // breaks it.
func (rq *request) limitBroken(now time.Time) bool { func (rq *request) limitBroken(now time.Time) bool {
counts, hit, over := rq.h.limiter.Count(rq.h.clientGroup(rq.client), now, counts, hit, over := rq.h.limiter.Count(clientGroup(rq.client), now,
rq.limitPercent.percent) rq.limitPercent.percent)
rq.line.Counts = counts rq.line.Counts = counts
@@ -71,7 +71,7 @@ func (rq *request) countBytes() {
now := rq.h.now() now := rq.h.now()
counts, hit, over := rq.h.limiter.CountBytes(rq.h.clientGroup(rq.client), now, counts, hit, over := rq.h.limiter.CountBytes(clientGroup(rq.client), now,
rq.countedBytes(), rq.bytesPercent.percent) rq.countedBytes(), rq.bytesPercent.percent)
rq.line.Counts.MinuteBytes = counts.MinuteBytes rq.line.Counts.MinuteBytes = counts.MinuteBytes
rq.line.Counts.HourBytes = counts.HourBytes rq.line.Counts.HourBytes = counts.HourBytes
@@ -155,7 +155,7 @@ func (rq *request) banForLimit(now time.Time, hit ratelimit.Hit, status int) {
} }
ban, made := rq.h.ledger.BanForLimit(netblock, now, notes) ban, made := rq.h.ledger.BanForLimit(netblock, now, notes)
rq.h.limiter.Reset(rq.h.clientGroup(rq.client)) rq.h.limiter.Reset(clientGroup(rq.client))
rq.line.BanExpires = banExpires(ban) rq.line.BanExpires = banExpires(ban)
if made { if made {
@@ -276,7 +276,7 @@ func (h *handler) netblock(client netip.Addr) netip.Prefix {
return netip.PrefixFrom(addr, h.config.BanScopeV4Prefix).Masked() return netip.PrefixFrom(addr, h.config.BanScopeV4Prefix).Masked()
} }
return h.clientGroup(addr) return clientGroup(addr)
} }
// banExpires is when ban ends, as the log line gives it: a time, or // banExpires is when ban ends, as the log line gives it: a time, or
internal/proxy/bans_test.go
+1 -6
View File
@@ -165,14 +165,9 @@ func TestBanCoversTheClientsNetblock(t *testing.T) {
[]string{otherClient, exempt}, []string{"203.0.112.9", allowed}, []string{otherClient, exempt}, []string{"203.0.112.9", allowed},
}, },
{ {
"an IPv6 /64, by default", nil, "2001:db8:5::1", "an IPv6 /64", nil, "2001:db8:5::1",
[]string{"2001:db8:5::ffff:1"}, []string{"2001:db8:5:1::1"}, []string{"2001:db8:5::ffff:1"}, []string{"2001:db8:5:1::1"},
}, },
{
"the IPv6 netblock SWWAF_IPV6_GROUP_PREFIX sets",
map[string]string{ipv6GroupPrefix: "48"}, "2001:db8:7::1",
[]string{"2001:db8:7:ffff::1"}, []string{"2001:db8:8::1"},
},
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
t.Parallel() t.Parallel()
internal/proxy/biased.go
+18 -41
View File
@@ -17,56 +17,33 @@ type percentage struct {
} }
// biasedThresholdsSet reports whether a biased threshold can lower a // biasedThresholdsSet reports whether a biased threshold can lower a
// client's limits: one of its lists is not empty, // client's limits: one of its lists is not empty, or
// SWWAF_UNKNOWN_LIMIT_PERCENT is below 100, or SWWAF_ASN_LIMIT_PERCENT_URL // SWWAF_UNKNOWN_LIMIT_PERCENT is below 100. The client's lookup is then
// is set. The client's lookup is then needed before its request goes on. // needed before its request goes on.
func biasedThresholdsSet(cfg *config.Config) bool { func biasedThresholdsSet(cfg *config.Config) bool {
return len(cfg.ASNLimitPercent) > 0 || len(cfg.CountryLimitPercent) > 0 || return len(cfg.ASNLimitPercent) > 0 || len(cfg.CountryLimitPercent) > 0 ||
len(cfg.ASNBytesPercent) > 0 || len(cfg.CountryBytesPercent) > 0 || len(cfg.ASNBytesPercent) > 0 || len(cfg.CountryBytesPercent) > 0 ||
cfg.UnknownLimitPercent < whole || cfg.ASNLimitPercentURL != "" cfg.UnknownLimitPercent < whole
} }
// limitPercentages returns the client's limit percentages, for the rate // limitPercentages returns a client's limit percentages, for the rate
// limits and for the byte limits, by its AS number and country as looked // limits and for the byte limits, by its AS number and country as looked
// up, each "" when unknown, and the blocklists, DNSBL zones and AbuseIPDB // up, each "" when unknown. Each is the lowest of those the settings give
// that list it. Each is the lowest of those the settings give it, the // it, the first of them in the order below when several are lowest: the
// first of them in the order below when several are lowest: the // percentage SWWAF_ASN_LIMIT_PERCENT gives its AS number, the one
// percentage SWWAF_ASN_LIMIT_PERCENT gives its AS number, the one the file // SWWAF_COUNTRY_LIMIT_PERCENT gives its country, and, for a client
// SWWAF_ASN_LIMIT_PERCENT_URL names gives it, the one // without a country, SWWAF_UNKNOWN_LIMIT_PERCENT. For the byte limits,
// SWWAF_COUNTRY_LIMIT_PERCENT gives its country, for a client without a // SWWAF_ASN_BYTES_PERCENT and SWWAF_COUNTRY_BYTES_PERCENT take the place
// country, SWWAF_UNKNOWN_LIMIT_PERCENT, for a client a blocklist lists, // of the first two for an AS number or a country they list.
// the percentage of SWWAF_BLOCKLIST_ACTION while it is limit, and for a func limitPercentages(
// client a DNSBL zone's verdict lists, or whose AbuseIPDB score is a hit, cfg *config.Config, asn, country string,
// the percentage of SWWAF_REPUTATION_ACTION while it is limit. For the ) (percentage, percentage) {
// byte limits, SWWAF_ASN_BYTES_PERCENT and SWWAF_COUNTRY_BYTES_PERCENT
// take the place of the first three for an AS number or a country they
// list.
func (rq *request) limitPercentages() (percentage, percentage) {
cfg := rq.h.config
asn, country := rq.line.ASN, rq.line.Country
unknown := percentage{percent: whole} unknown := percentage{percent: whole}
if country == "" { if country == "" {
unknown = percentage{cfg.UnknownLimitPercent, "SWWAF_UNKNOWN_LIMIT_PERCENT"} unknown = percentage{cfg.UnknownLimitPercent, "SWWAF_UNKNOWN_LIMIT_PERCENT"}
} }
fetched := percentage{percent: whole} asnRequests := given(cfg.ASNLimitPercent, asn, "SWWAF_ASN_LIMIT_PERCENT")
if percent, listed := rq.h.lists.ASNLimitPercent(asn); listed {
fetched = percentage{percent, "SWWAF_ASN_LIMIT_PERCENT_URL"}
}
blocklisted := percentage{percent: whole}
if rq.blocklisted && cfg.BlocklistAction == "limit" {
blocklisted = percentage{cfg.BlocklistLimitPercent, "SWWAF_BLOCKLIST_ACTION"}
}
reputationListed := percentage{percent: whole}
if (rq.dnsblListed || rq.abuseIPDBHit) && cfg.ReputationAction == "limit" {
reputationListed = percentage{cfg.ReputationLimitPercent, "SWWAF_REPUTATION_ACTION"}
}
asnRequests := lowest(given(cfg.ASNLimitPercent, asn, "SWWAF_ASN_LIMIT_PERCENT"),
fetched)
countryRequests := given(cfg.CountryLimitPercent, country, countryRequests := given(cfg.CountryLimitPercent, country,
"SWWAF_COUNTRY_LIMIT_PERCENT") "SWWAF_COUNTRY_LIMIT_PERCENT")
@@ -79,8 +56,8 @@ func (rq *request) limitPercentages() (percentage, percentage) {
countryBytes = given(cfg.CountryBytesPercent, country, "SWWAF_COUNTRY_BYTES_PERCENT") countryBytes = given(cfg.CountryBytesPercent, country, "SWWAF_COUNTRY_BYTES_PERCENT")
} }
return lowest(asnRequests, countryRequests, unknown, blocklisted, reputationListed), return lowest(asnRequests, countryRequests, unknown),
lowest(asnBytes, countryBytes, unknown, blocklisted, reputationListed) lowest(asnBytes, countryBytes, unknown)
} }
// given returns the percentage percents, the setting named setting, gives // given returns the percentage percents, the setting named setting, gives
internal/proxy/biased_test.go
-1
View File
@@ -310,7 +310,6 @@ func TestRequestWaitsForItsLookupWhileABiasedThresholdIsSet(t *testing.T) {
{asnBytesPercent, asnDEHalf, true}, {asnBytesPercent, asnDEHalf, true},
{countryBytesPercent, countryDEHalf, true}, {countryBytesPercent, countryDEHalf, true},
{unknownLimitPercent, "99", true}, {unknownLimitPercent, "99", true},
{asnLimitPercentURL, asnURL, true},
// At 100, its default, it lowers no limit. // At 100, its default, it lowers no limit.
{unknownLimitPercent, "100", false}, {unknownLimitPercent, "100", false},
} { } {
internal/proxy/client.go
+7 -5
View File
@@ -76,14 +76,16 @@ func scheme(r *http.Request, peerTrusted bool) string {
return proto return proto
} }
// ipv6GroupPrefix is the length of the IPv6 netblock that is one client.
const ipv6GroupPrefix = 64
// clientGroup is the client a request is counted toward: its IPv4 // clientGroup is the client a request is counted toward: its IPv4
// address, or its IPv6 group, the netblock its IPv6 address is in of the // address, or the /64 its IPv6 address is in, since one abuser usually
// length SWWAF_IPV6_GROUP_PREFIX sets, a /64 by default, since one abuser // holds a whole /64. An IPv4 address in IPv6 form counts as IPv4.
// usually holds a whole /64. An IPv4 address in IPv6 form counts as IPv4. func clientGroup(addr netip.Addr) netip.Prefix {
func (h *handler) clientGroup(addr netip.Addr) netip.Prefix {
addr = addr.Unmap() addr = addr.Unmap()
if addr.Is6() { if addr.Is6() {
return netip.PrefixFrom(addr, h.config.IPv6GroupPrefix).Masked() return netip.PrefixFrom(addr, ipv6GroupPrefix).Masked()
} }
return netip.PrefixFrom(addr, addr.BitLen()) return netip.PrefixFrom(addr, addr.BitLen())
internal/proxy/history_test.go
-18
View File
@@ -56,24 +56,6 @@ func TestHistoryKeepsEachRequestOfTheClient(t *testing.T) {
} }
} }
func TestTableOfClientsHoldsAtMostMaxTrackedClients(t *testing.T) {
t.Parallel()
s, _, server := startWithClock(t, "", map[string]string{maxTrackedClients: "2"})
// The third client drops the least recently seen, the first, with its
// history.
for _, from := range []string{"192.0.2.1", "192.0.2.2", "192.0.2.3"} {
s.get(from, http.StatusOK, requestlog.ActionForward)
}
_, held := server.Limiter.Client(netip.MustParsePrefix("192.0.2.1/32"))
if server.Limiter.Len() != 2 || held {
t.Errorf("the table holds %d clients, the first among them: %t; want 2, "+
"without it", server.Limiter.Len(), held)
}
}
func TestHistoryCountsTheBodiesEachWay(t *testing.T) { func TestHistoryCountsTheBodiesEachWay(t *testing.T) {
t.Parallel() t.Parallel()
internal/proxy/lookup.go
+2 -2
View File
@@ -31,9 +31,9 @@ func (rq *request) lookUp(ctx context.Context) {
} }
if rq.h.config.LookupSource == "file" { if rq.h.config.LookupSource == "file" {
rq.lookupAnswer = rq.h.lookupFile.LookUp(rq.h.clientGroup(rq.client)) rq.lookupAnswer = rq.h.lookupFile.LookUp(clientGroup(rq.client))
} else { } else {
rq.lookupAnswer = rq.h.geojs.LookUp(ctx, rq.h.clientGroup(rq.client)) rq.lookupAnswer = rq.h.geojs.LookUp(ctx, clientGroup(rq.client))
} }
rq.lookedUp = true rq.lookedUp = true
internal/proxy/lookup_test.go
-1
View File
@@ -349,7 +349,6 @@ const unansweredGeoJSURL = "unanswered://geojs/v1/ip/geo.json"
func TestMain(m *testing.M) { func TestMain(m *testing.M) {
transport, _ := http.DefaultTransport.(*http.Transport) transport, _ := http.DefaultTransport.(*http.Transport)
transport.RegisterProtocol("unanswered", unansweredGeoJS{}) transport.RegisterProtocol("unanswered", unansweredGeoJS{})
transport.RegisterProtocol("abuseipdb", abuseIPDBStandIn{})
m.Run() m.Run()
} }
internal/proxy/passthrough_test.go
+1 -20
View File
@@ -320,7 +320,7 @@ func TestServerHasTheDefaultLimits(t *testing.T) {
server := proxy.New(proxy.Params{ server := proxy.New(proxy.Params{
Config: cfg, Config: cfg,
RequestLog: io.Discard, RequestLog: io.Discard,
ProcessLog: requestlog.NewProcessLogger(io.Discard, cfg.InstanceName, cfg.LogLevel), ProcessLog: requestlog.NewProcessLogger(io.Discard, cfg.InstanceName),
}) })
if server.Addr != ":8080" || server.MaxHeaderBytes != 28<<10 || if server.Addr != ":8080" || server.MaxHeaderBytes != 28<<10 ||
@@ -399,25 +399,6 @@ func TestAnswers502WhenTheAppCannotBeReached(t *testing.T) {
} }
} }
func TestLogLevelHoldsBackNoRequestLine(t *testing.T) {
t.Parallel()
// At error the warning that the request to the app failed is held back,
// and is written before the answer is.
addr, out := startProxy(t, "http://"+localhost+":1", map[string]string{
"SWWAF_LOG_LEVEL": "error",
})
wantStatus(t, get(t, addr, "/"), http.StatusBadGateway)
wantLine(t, out.requestLine(t), http.StatusBadGateway, requestlog.ActionUpstreamError)
for _, line := range out.lines(t) {
if line["type"] == "process" {
t.Errorf("process line %v, want none at error", line)
}
}
}
func TestLogsAnAnswerThatBrokeOff(t *testing.T) { func TestLogsAnAnswerThatBrokeOff(t *testing.T) {
t.Parallel() t.Parallel()
internal/proxy/proxy.go
+4 -59
View File
@@ -18,7 +18,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/lookup" "sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/metrics" "sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules" "sneak.berlin/go/smallwebwaf/internal/rules"
) )
@@ -59,9 +58,6 @@ type Params struct {
// GeoJSURL is where clients' AS numbers and countries are looked up // GeoJSURL is where clients' AS numbers and countries are looked up
// while SWWAF_LOOKUP_SOURCE is geojs, normally lookup.URL. // while SWWAF_LOOKUP_SOURCE is geojs, normally lookup.URL.
GeoJSURL string GeoJSURL string
// AbuseIPDBURL is where clients are checked with AbuseIPDB while
// SWWAF_ABUSEIPDB_KEY is set, normally reputation.AbuseIPDBURL.
AbuseIPDBURL string
// LookupFile is the lookup database they are looked up in while // LookupFile is the lookup database they are looked up in while
// SWWAF_LOOKUP_SOURCE is file, and nil otherwise. // SWWAF_LOOKUP_SOURCE is file, and nil otherwise.
LookupFile *lookup.File LookupFile *lookup.File
@@ -73,18 +69,14 @@ type Params struct {
// against. // against.
Rules *rules.Files Rules *rules.Files
// Alerts receive the alert for each ban the proxy makes or makes // Alerts receive the alert for each ban the proxy makes or makes
// permanent, for each count over an anomaly threshold, for each request // permanent, for each count over an anomaly threshold, and for GeoJS
// whose client a blocklist, a DNSBL zone or AbuseIPDB lists, and for // failing.
// GeoJS failing, a fetch of a list failing, a query to a DNSBL zone or
// a check with AbuseIPDB failing, or the day's AbuseIPDB checks used up.
Alerts *alerts.Queue Alerts *alerts.Queue
} }
// Server is the server smallwebwaf runs, with the parts of the proxy // Server is the server smallwebwaf runs, with the parts of the proxy
// whose state the state files keep, the lookup database, nil unless // whose state the state files keep, the lookup database, nil unless
// SWWAF_LOOKUP_SOURCE is file, the lists fetched from URLs, which its Run // SWWAF_LOOKUP_SOURCE is file, and the metrics.
// fetches, the DNSBL zones' verdicts, AbuseIPDB's scores and checks
// spent, and the metrics.
type Server struct { type Server struct {
*http.Server *http.Server
@@ -93,9 +85,6 @@ type Server struct {
GeoJS *lookup.GeoJS GeoJS *lookup.GeoJS
Anomalies *anomaly.Counters Anomalies *anomaly.Counters
LookupFile *lookup.File LookupFile *lookup.File
Lists *reputation.Lists
DNSBL *reputation.DNSBL
AbuseIPDB *reputation.AbuseIPDB
Metrics *metrics.Metrics Metrics *metrics.Metrics
} }
@@ -108,7 +97,6 @@ type Server struct {
func New(params Params) *Server { func New(params Params) *Server {
errorLog := slog.NewLogLogger(params.ProcessLog.Handler(), slog.LevelWarn) errorLog := slog.NewLogLogger(params.ProcessLog.Handler(), slog.LevelWarn)
m := metrics.New(params.Config.MetricsTopN, params.Config.InstanceName) m := metrics.New(params.Config.MetricsTopN, params.Config.InstanceName)
lists, dnsbl, abuseIPDB := newReputation(params, m)
h := &handler{ h := &handler{
config: params.Config, config: params.Config,
requestLog: params.RequestLog, requestLog: params.RequestLog,
@@ -124,7 +112,7 @@ func New(params Params) *Server {
BytesPerMinute: params.Config.BytesLimitPerMinute, BytesPerMinute: params.Config.BytesLimitPerMinute,
BytesPerHour: params.Config.BytesLimitPerHour, BytesPerHour: params.Config.BytesLimitPerHour,
BytesPerDay: params.Config.BytesLimitPerDay, BytesPerDay: params.Config.BytesLimitPerDay,
}, params.Config.MaxTrackedClients), }),
ledger: bans.New(bans.Rules{ ledger: bans.New(bans.Rules{
LimitBanDuration: params.Config.LimitBanDuration, LimitBanDuration: params.Config.LimitBanDuration,
LimitBanRepeatWindow: params.Config.LimitBanRepeatWindow, LimitBanRepeatWindow: params.Config.LimitBanRepeatWindow,
@@ -144,9 +132,6 @@ func New(params Params) *Server {
Alerts: params.Alerts, Alerts: params.Alerts,
}), }),
lookupFile: params.LookupFile, lookupFile: params.LookupFile,
lists: lists,
dnsbl: dnsbl,
abuseIPDB: abuseIPDB,
rules: params.Rules, rules: params.Rules,
alerts: params.Alerts, alerts: params.Alerts,
} }
@@ -185,47 +170,10 @@ func New(params Params) *Server {
GeoJS: h.geojs, GeoJS: h.geojs,
Anomalies: h.anomalies, Anomalies: h.anomalies,
LookupFile: h.lookupFile, LookupFile: h.lookupFile,
Lists: h.lists,
DNSBL: h.dnsbl,
AbuseIPDB: h.abuseIPDB,
Metrics: m, Metrics: m,
} }
} }
// newReputation returns the lists fetched from URLs, the DNSBL zones'
// verdicts and AbuseIPDB's scores, as the settings in params name them,
// with none fetched, asked for or checked yet, and adds their metrics to
// m, AbuseIPDB's while SWWAF_ABUSEIPDB_KEY is set.
func newReputation(
params Params, m *metrics.Metrics,
) (*reputation.Lists, *reputation.DNSBL, *reputation.AbuseIPDB) {
cfg := params.Config
lists := reputation.New(reputation.Params{
BlocklistURLs: cfg.BlocklistURLs, Refresh: cfg.BlocklistRefresh,
ASNLimitPercentURL: cfg.ASNLimitPercentURL, Now: params.Now,
ProcessLog: params.ProcessLog, Alerts: params.Alerts,
})
dnsbl := reputation.NewDNSBL(reputation.DNSBLParams{
Zones: cfg.DNSBLZones, Resolver: cfg.DNSBLResolver, CacheTTL: cfg.ReputationCacheTTL,
Timeout: cfg.ReputationTimeout, Now: params.Now, ProcessLog: params.ProcessLog,
Alerts: params.Alerts,
})
abuseIPDB := reputation.NewAbuseIPDB(reputation.AbuseIPDBParams{
URL: params.AbuseIPDBURL, Key: cfg.AbuseIPDBKey, MinScore: cfg.AbuseIPDBMinScore,
DailyBudget: cfg.AbuseIPDBDailyBudget, CacheTTL: cfg.ReputationCacheTTL,
Timeout: cfg.ReputationTimeout, Now: params.Now, ProcessLog: params.ProcessLog,
Alerts: params.Alerts,
})
m.AddReputation(lists, dnsbl)
if cfg.AbuseIPDBKey != "" {
m.AddAbuseIPDB(abuseIPDB)
}
return lists, dnsbl, abuseIPDB
}
// handler is the proxy. It holds what every request shares; what belongs // handler is the proxy. It holds what every request shares; what belongs
// to one request is in a request. // to one request is in a request.
type handler struct { type handler struct {
@@ -241,9 +189,6 @@ type handler struct {
geojs *lookup.GeoJS geojs *lookup.GeoJS
anomalies *anomaly.Counters anomalies *anomaly.Counters
lookupFile *lookup.File lookupFile *lookup.File
lists *reputation.Lists
dnsbl *reputation.DNSBL
abuseIPDB *reputation.AbuseIPDB
rules *rules.Files rules *rules.Files
alerts *alerts.Queue alerts *alerts.Queue
} }
internal/proxy/proxy_test.go
+10 -14
View File
@@ -61,8 +61,6 @@ const (
requestMaxBytes = "SWWAF_REQUEST_MAX_BYTES" requestMaxBytes = "SWWAF_REQUEST_MAX_BYTES"
responseMaxBytes = "SWWAF_RESPONSE_MAX_BYTES" responseMaxBytes = "SWWAF_RESPONSE_MAX_BYTES"
trustedProxies = "SWWAF_TRUSTED_PROXIES" trustedProxies = "SWWAF_TRUSTED_PROXIES"
ipv6GroupPrefix = "SWWAF_IPV6_GROUP_PREFIX"
maxTrackedClients = "SWWAF_MAX_TRACKED_CLIENTS"
allowNets = "SWWAF_ALLOW_NETS" allowNets = "SWWAF_ALLOW_NETS"
rateLimitExemptNets = "SWWAF_RATE_LIMIT_EXEMPT_NETS" rateLimitExemptNets = "SWWAF_RATE_LIMIT_EXEMPT_NETS"
denyNets = "SWWAF_DENY_NETS" denyNets = "SWWAF_DENY_NETS"
@@ -270,8 +268,7 @@ func startProxyWithAlerts(
// queue: they wait in it, for the test to look at. With no geojsURL, there // queue: they wait in it, for the test to look at. With no geojsURL, there
// is no stand-in for GeoJS to look clients up at, and SWWAF_LOOKUP_SOURCE // is no stand-in for GeoJS to look clients up at, and SWWAF_LOOKUP_SOURCE
// is off unless env sets it. While it is file, the lookup database // is off unless env sets it. While it is file, the lookup database
// SWWAF_LOOKUP_DB_PATH names is read. Clients are checked with AbuseIPDB // SWWAF_LOOKUP_DB_PATH names is read.
// at abuseIPDBURL while env sets SWWAF_ABUSEIPDB_KEY.
func newProxy( func newProxy(
t *testing.T, appURL, geojsURL string, now func() time.Time, t *testing.T, appURL, geojsURL string, now func() time.Time,
env map[string]string, env map[string]string,
@@ -297,7 +294,7 @@ func newProxy(
} }
out := &output{} out := &output{}
processLog := requestlog.NewProcessLogger(out, cfg.InstanceName, cfg.LogLevel) processLog := requestlog.NewProcessLogger(out, cfg.InstanceName)
ruleFiles, err := rules.Load(rules.Params{ ruleFiles, err := rules.Load(rules.Params{
Dir: cfg.RulesDir, Enabled: cfg.RulesEnabled, ProcessLog: processLog, Dir: cfg.RulesDir, Enabled: cfg.RulesEnabled, ProcessLog: processLog,
@@ -328,15 +325,14 @@ func newProxy(
} }
server := proxy.New(proxy.Params{ server := proxy.New(proxy.Params{
Config: cfg, Config: cfg,
RequestLog: out, RequestLog: out,
ProcessLog: processLog, ProcessLog: processLog,
GeoJSURL: geojsURL, GeoJSURL: geojsURL,
AbuseIPDBURL: abuseIPDBURL, LookupFile: lookupFile,
LookupFile: lookupFile, Now: now,
Now: now, Rules: ruleFiles,
Rules: ruleFiles, Alerts: alertQueue,
Alerts: alertQueue,
}) })
return server, out, alertQueue return server, out, alertQueue
internal/proxy/ratelimits_test.go
-43
View File
@@ -72,49 +72,6 @@ func TestRateLimitRefusesBeforeTheApp(t *testing.T) {
} }
} }
func TestIPv6GroupPrefixSetsTheClientTheLimitsCount(t *testing.T) {
t.Parallel()
// With SWWAF_IPV6_GROUP_PREFIX at 48, the first two addresses, in two
// /64s of one /48, are one client, and the second's request breaks the
// limit; the third, in the next /48, is another client.
const (
first = "2001:db8:9::1"
second = "2001:db8:9:1::1"
other = "2001:db8:a::1"
)
for _, tc := range []struct {
setting, value string
// status and action are those of the request that breaks the
// limit: a rate limit refuses it, a byte limit passes it on.
status int
action string
}{
{rateLimitPerMinute, "1", http.StatusForbidden, requestlog.ActionRateLimited},
{bytesLimitPerMinute, byteLimit, http.StatusOK, requestlog.ActionForward},
} {
t.Run(tc.setting, func(t *testing.T) {
t.Parallel()
s, _ := startWithAnswers(t, map[string]string{
ipv6GroupPrefix: "48", tc.setting: tc.value,
})
s.get(first, http.StatusOK, requestlog.ActionForward)
line := s.get(second, tc.status, tc.action)
if line.ClientGroup != "2001:db8:9::/48" ||
line.Offence != requestlog.OffenceLimit {
t.Errorf("log line has client_group %q and offence %q, "+
"want 2001:db8:9::/48 and limit", line.ClientGroup, line.Offence)
}
s.get(other, http.StatusOK, requestlog.ActionForward)
})
}
}
func TestRateLimitExemptPathsAreNeitherCountedNorRefused(t *testing.T) { func TestRateLimitExemptPathsAreNeitherCountedNorRefused(t *testing.T) {
t.Parallel() t.Parallel()
internal/proxy/reputation.go
-98
View File
@@ -1,98 +0,0 @@
package proxy
import (
"context"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation"
)
// deny is the SWWAF_BLOCKLIST_ACTION and the SWWAF_REPUTATION_ACTION that
// refuses the requests of a client a source lists.
const deny = "deny"
// blocklistDenied notes the blocklists that list the client, as
// noteListed does, and reports whether SWWAF_BLOCKLIST_ACTION, being deny,
// refuses the request. Being limit, it lowers the client's limits instead
// (see limitPercentages), and being log, it does nothing more.
func (rq *request) blocklistDenied() bool {
listedBy := rq.h.lists.ListedBy(rq.client)
rq.blocklisted = len(listedBy) > 0
rq.noteListed(listedBy, "listed by a blocklist")
return rq.blocklisted && rq.h.config.BlocklistAction == deny
}
// dnsblDenied notes the DNSBL zones whose verdict lists the client, as
// noteListed does, and reports whether SWWAF_REPUTATION_ACTION, being
// deny, refuses the request. Being limit, it lowers the client's limits
// instead (see limitPercentages), and being log, it does nothing more. A
// zone without a verdict on the client is asked about it in the
// background, and the request does not wait for the answer. ctx is the
// request's own context.
func (rq *request) dnsblDenied(ctx context.Context) bool {
listedBy := rq.h.dnsbl.ListedBy(ctx, rq.client)
rq.dnsblListed = len(listedBy) > 0
rq.noteListed(listedBy, "listed by a DNSBL zone")
return rq.dnsblListed && rq.h.config.ReputationAction == deny
}
// abuseIPDBDenied notes AbuseIPDB, as noteHit does, with the score, when
// its score of the client is a hit, and reports whether
// SWWAF_REPUTATION_ACTION, being deny, refuses the request, as dnsblDenied
// does for a zone. While SWWAF_ABUSEIPDB_KEY is unset it does nothing. A
// client without a score is checked in the background, by the request's
// address, if its history counts an offence, and the request does not
// wait for the answer. The score is then used for each address of the
// client. ctx is the request's own context.
func (rq *request) abuseIPDBDenied(ctx context.Context) bool {
if rq.h.config.AbuseIPDBKey == "" {
return false
}
client := rq.h.clientGroup(rq.client)
held, _ := rq.h.limiter.Client(client)
offender := held.History.Offences != ratelimit.Offences{}
score, hit := rq.h.abuseIPDB.Hit(ctx, client, rq.client, offender)
if !hit {
return false
}
rq.abuseIPDBHit = true
rq.noteHit(reputation.AbuseIPDBSource, "scored by AbuseIPDB at or over "+
"SWWAF_ABUSEIPDB_MIN_SCORE", map[string]any{
"source": reputation.AbuseIPDBSource, "score": score,
})
return rq.h.config.ReputationAction == deny
}
// noteListed notes each of sources, the URLs of the blocklists or the
// DNSBL zones, their keys masked, that list the client, as noteHit does,
// with reason, and the source in the alert's detail.
func (rq *request) noteListed(sources []string, reason string) {
for _, source := range sources {
rq.noteHit(source, reason, map[string]any{"source": source})
}
}
// noteHit adds source, which lists the client, to the log line's
// reputation, counts it in the metrics, and raises a reputation_hit alert
// with reason and detail.
func (rq *request) noteHit(source, reason string, detail map[string]any) {
rq.line.Reputation = append(rq.line.Reputation, source)
rq.h.metrics.ReputationHit(source)
rq.h.alerts.Raise(alerts.Alert{
Event: alerts.EventReputationHit,
Client: rq.client,
Netblock: rq.h.clientGroup(rq.client),
ASN: rq.line.ASN,
ASName: rq.line.ASName,
Country: rq.line.Country,
Reason: reason,
Detail: detail,
})
}
internal/proxy/reputation_test.go
-960
View File
@@ -1,960 +0,0 @@
package proxy_test
import (
"fmt"
"io"
"maps"
"net/http"
"net/netip"
"slices"
"strconv"
"strings"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/proxy"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// The reputation settings.
const (
blocklistURLs = "SWWAF_BLOCKLIST_URLS"
blocklistAction = "SWWAF_BLOCKLIST_ACTION"
asnLimitPercentURL = "SWWAF_ASN_LIMIT_PERCENT_URL"
)
// The actions of SWWAF_BLOCKLIST_ACTION and SWWAF_REPUTATION_ACTION:
// limitHalf gives a listed client half of every limit, and limitQuarter a
// quarter.
const (
actionDeny = "deny"
actionLog = "log"
limitHalf = "limit:50"
limitQuarter = "limit:25"
)
// The lists these tests name, which are never fetched: each test puts in
// the copies it needs, as reputation.json would at start.
const (
dropURL = "https://lists.example/drop.txt"
torURL = "https://lists.example/tor.txt"
asnURL = "https://lists.example/asn.txt"
)
func TestEachBlocklistActionForAListedAddressAndAListedNetblock(t *testing.T) {
t.Parallel()
forward, denied := requestlog.ActionForward, requestlog.ActionDenied
for _, tc := range []struct {
action string
// statuses and actions are those of a listed client's three
// requests, and percent their limit_percent, as percentText gives it.
statuses []int
actions []string
percent string
}{
{
actionDeny, []int{http.StatusForbidden, http.StatusForbidden, http.StatusForbidden},
[]string{denied, denied, denied}, none,
},
{
// Half of 4 requests a minute: the third breaks the limit.
limitHalf, []int{http.StatusOK, http.StatusOK, http.StatusForbidden},
[]string{forward, forward, requestlog.ActionRateLimited},
"50 from " + blocklistAction,
},
{
actionLog, []int{http.StatusOK, http.StatusOK, http.StatusOK},
[]string{forward, forward, forward}, none,
},
} {
t.Run(tc.action, func(t *testing.T) {
t.Parallel()
s, server, _ := startWithLookups(t, map[string]string{
rateLimitPerMinute: fourAMinute, blocklistURLs: dropURL,
blocklistAction: tc.action,
})
// fromDE is listed as an address, and fromKP in a netblock.
loadLists(t, server, map[string][]string{
dropURL: {"; DROP", fromDE, "198.51.100.0/24 ; SBL1"},
})
for _, from := range []string{fromDE, fromKP} {
for i := range 3 {
line := s.get(from, tc.statuses[i], tc.actions[i])
wantReputation(t, line, dropURL)
wantPercent(t, "limit_percent", line.LimitPercent,
line.LimitPercentSetting, tc.percent)
// A request refused for the list is not counted.
counted := line.fields["counts"] != nil
if counted != (tc.actions[i] != denied) {
t.Errorf("request from %s counted %t, logged %s", from, counted,
tc.actions[i])
}
}
}
// A client no list lists has the whole limit.
for range 3 {
line := s.get(unplaced, http.StatusOK, forward)
wantReputation(t, line)
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
none)
}
// A refusal for the list makes no ban.
if held := server.Ledger.Snapshot(); tc.action == actionDeny && len(held) != 0 {
t.Errorf("bans %+v, want none", held)
}
})
}
}
func TestBlocklistsComeAfterTheCountryListsAndSkipAllowNets(t *testing.T) {
t.Parallel()
s, server, queue := startWithLookups(t, map[string]string{
blocklistURLs: dropURL, deniedCountries: "kp", allowNets: fromDE,
})
loadLists(t, server, map[string][]string{dropURL: {fromDE, fromKP}})
// fromKP's country refuses it before the list is looked at, and fromDE,
// in SWWAF_ALLOW_NETS, is not checked at all: neither is noted, nor
// alerted.
wantReputation(t, s.get(fromKP, http.StatusForbidden, requestlog.ActionCountryDenied))
wantReputation(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward))
wantAlerts(t, queue)
}
func TestObserveModeForwardsAClientABlocklistDeniesAndAlertsIt(t *testing.T) {
t.Parallel()
s, server, queue := startWithLookups(t, map[string]string{
blocklistURLs: dropURL, mode: observe,
})
loadLists(t, server, map[string][]string{dropURL: {fromDE}})
line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
wantWouldAction(t, line, requestlog.ActionDenied)
wantReputation(t, line, dropURL)
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
if len(waiting) != 1 || waiting[0].Event != alerts.EventReputationHit {
t.Errorf("alerts waiting %+v, want a reputation_hit alert", waiting)
}
}
func TestBlocklistLimitTakesPartInTheLowestPercentageOfEveryLimit(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
action, asnPercent string
// want is the upload's limit_percent and bytes_percent, as
// percentText gives them, and limitHit its limit_hit.
want, limitHit string
}{
{limitHalf, asnDEQuarter, "25 from " + asnLimitPercent, minuteBytes},
{limitQuarter, asnDEHalf, "25 from " + blocklistAction, minuteBytes},
// The AS number's, the first of two alike.
{limitQuarter, asnDEQuarter, "25 from " + asnLimitPercent, minuteBytes},
{actionLog, asnDE + ":100", none, ""},
} {
t.Run(tc.action+" "+tc.asnPercent, func(t *testing.T) {
t.Parallel()
s, server, _ := startWithLookups(t, map[string]string{
bytesLimitPerMinute: twoUploads, blocklistURLs: dropURL,
blocklistAction: tc.action, asnLimitPercent: tc.asnPercent,
})
loadLists(t, server, map[string][]string{dropURL: {fromDE}})
// The upload's 100 bytes are over 49, a quarter of 199, and 99,
// half of it, and within 199.
line := s.uploadFrom(fromDE)
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
tc.want)
wantPercent(t, "bytes_percent", line.BytesPercent, line.BytesPercentSetting,
tc.want)
if line.LimitHit != tc.limitHit {
t.Errorf("log line has limit_hit %q, want %q", line.LimitHit, tc.limitHit)
}
})
}
}
func TestASNLimitPercentFileCountsAsTheSettingDoesTheLowerWinning(t *testing.T) {
t.Parallel()
const (
fromURL = "25 from " + asnLimitPercentURL
fromSetting = "25 from " + asnLimitPercent
)
for _, tc := range []struct {
name string
env map[string]string
file string
// limitPercent and bytesPercent are the upload's, as percentText
// gives them.
limitPercent, bytesPercent string
}{
{"the file's alone", nil, asnDEQuarter, fromURL, fromURL},
{
"the file's, lower than the setting's",
map[string]string{asnLimitPercent: asnDEHalf}, asnDEQuarter, fromURL, fromURL,
},
{
"the setting's, lower than the file's",
map[string]string{asnLimitPercent: asnDEQuarter}, asnDEHalf,
fromSetting, fromSetting,
},
{
"the setting's, the first of two alike",
map[string]string{asnLimitPercent: asnDEQuarter}, asnDEQuarter,
fromSetting, fromSetting,
},
{"none, for an AS number the file does not list", nil, asnKP + ":25", none, none},
{
"SWWAF_ASN_BYTES_PERCENT's in place of the file's for the byte limits",
map[string]string{asnBytesPercent: asnDE + ":100"}, asnDEQuarter, fromURL, none,
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
env := map[string]string{asnLimitPercentURL: asnURL}
maps.Copy(env, tc.env)
s, server, _ := startWithLookups(t, env)
loadLists(t, server, map[string][]string{asnURL: {"# by AS number", tc.file}})
line := s.uploadFrom(fromDE)
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
tc.limitPercent)
wantPercent(t, "bytes_percent", line.BytesPercent, line.BytesPercentSetting,
tc.bytesPercent)
})
}
}
func TestEachBlocklistThatListsAClientRaisesAnAlertOncePerCooldownAndIsCounted(
t *testing.T,
) {
t.Parallel()
const emptyURL = "https://lists.example/empty.txt"
s, clk, server, queue := startWithLookupsAndClock(t, map[string]string{
blocklistURLs: dropURL + "," + torURL + "," + emptyURL,
blocklistAction: actionLog,
metricsToken: token,
})
loadLists(t, server, map[string][]string{dropURL: {fromDE}, torURL: {fromDE}})
// The second request's alerts are repeats, which the cooldown holds
// back.
for range 2 {
wantReputation(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward),
dropURL, torURL)
}
hit := func(source string) alerts.Alert {
return alerts.Alert{
Instance: alertInstance,
Time: clk.Now(),
Event: alerts.EventReputationHit,
Client: netip.MustParseAddr(fromDE),
Netblock: netip.MustParsePrefix(fromDE + "/32"),
ASN: asnDE,
ASName: asNameDE,
Country: "DE",
Reason: "listed by a blocklist",
Detail: map[string]any{"source": source},
}
}
wantAlerts(t, queue, hit(dropURL), hit(torURL))
if queue.Suppressed() != 2 {
t.Errorf("%d alerts held back, want the second request's 2", queue.Suppressed())
}
// Each list's hits, none of its fetches failed, and when its copy was
// fetched, 0 for the one without.
metrics := s.scrape(unplaced)
fetched := float64(listsFetched().Unix())
for listURL, want := range map[string]struct{ hits, fetched float64 }{
dropURL: {2, fetched}, torURL: {2, fetched}, emptyURL: {0, 0},
} {
labels := `{instance="` + alertInstance + `",source="` + listURL + `"}`
if want.hits == 0 {
wantNoSeries(t, metrics, "smallwebwaf_reputation_hits_total"+labels)
} else {
wantMetric(t, metrics, "smallwebwaf_reputation_hits_total"+labels, want.hits)
}
wantMetric(t, metrics, "smallwebwaf_reputation_failures_total"+labels, 0)
wantMetric(t, metrics, "smallwebwaf_reputation_last_fetch_timestamp_seconds"+labels,
want.fetched)
}
}
// The DNSBL settings.
const (
dnsblZones = "SWWAF_DNSBL_ZONES"
dnsblResolver = "SWWAF_DNSBL_RESOLVER"
reputationAction = "SWWAF_REPUTATION_ACTION"
)
// The DNSBL zones these tests name, which are never asked about the
// clients the tests send requests from: each test puts in the verdicts it
// needs, as reputation.json would at start. A query a test does start is
// sent to noResolver, where nothing listens, so that none leaves the host.
const (
dnsblZone = "dnsbl.example"
otherZone = "other.example"
noResolver = "127.0.0.1:9"
)
func TestEachReputationActionForAClientADNSBLZoneLists(t *testing.T) {
t.Parallel()
forward, denied := requestlog.ActionForward, requestlog.ActionDenied
for _, tc := range []struct {
action string
// statuses and actions are those of a listed client's three
// requests, and percent their limit_percent, as percentText gives it.
statuses []int
actions []string
percent string
}{
{
actionDeny, []int{http.StatusForbidden, http.StatusForbidden, http.StatusForbidden},
[]string{denied, denied, denied}, none,
},
{
// Half of 4 requests a minute: the third breaks the limit.
limitHalf, []int{http.StatusOK, http.StatusOK, http.StatusForbidden},
[]string{forward, forward, requestlog.ActionRateLimited},
"50 from " + reputationAction,
},
{
actionLog, []int{http.StatusOK, http.StatusOK, http.StatusOK},
[]string{forward, forward, forward}, none,
},
} {
t.Run(tc.action, func(t *testing.T) {
t.Parallel()
s, server, _ := startWithLookups(t, map[string]string{
rateLimitPerMinute: fourAMinute, dnsblZones: dnsblZone + "," + otherZone,
dnsblResolver: noResolver, reputationAction: tc.action,
})
listedBy := map[string][]string{
fromDE: {dnsblZone, otherZone}, fromKP: {otherZone}, unplaced: nil,
}
loadVerdicts(server, listedBy)
for _, from := range []string{fromDE, fromKP} {
for i := range 3 {
line := s.get(from, tc.statuses[i], tc.actions[i])
// In the order SWWAF_DNSBL_ZONES names them.
wantReputation(t, line, listedBy[from]...)
wantPercent(t, "limit_percent", line.LimitPercent,
line.LimitPercentSetting, tc.percent)
// A request refused for the verdict is not counted.
counted := line.fields["counts"] != nil
if counted != (tc.actions[i] != denied) {
t.Errorf("request from %s counted %t, logged %s", from, counted,
tc.actions[i])
}
}
}
// A client no zone lists has the whole limit.
for range 3 {
line := s.get(unplaced, http.StatusOK, forward)
wantReputation(t, line)
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
none)
}
// A refusal for the verdict makes no ban, and every client had its
// verdicts, so no zone was asked.
if held := server.Ledger.Snapshot(); tc.action == actionDeny && len(held) != 0 {
t.Errorf("bans %+v, want none", held)
}
if queries := server.DNSBL.Queries(dnsblZone); queries != 0 {
t.Errorf("%d queries, want none", queries)
}
})
}
}
func TestDNSBLZonesComeAfterTheBlocklistsAndSkipAllowNets(t *testing.T) {
t.Parallel()
s, server, queue := startWithLookups(t, map[string]string{
blocklistURLs: dropURL, dnsblZones: dnsblZone, dnsblResolver: noResolver,
reputationAction: actionDeny, allowNets: fromDE,
})
loadLists(t, server, map[string][]string{dropURL: {fromKP}})
loadVerdicts(server, map[string][]string{fromKP: {dnsblZone}, fromDE: {dnsblZone}})
// The blocklist refuses fromKP before its verdict is looked at, and
// fromDE, in SWWAF_ALLOW_NETS, is not checked at all: neither is noted
// for the zone, nor alerted, nor asked about.
wantReputation(t, s.get(fromKP, http.StatusForbidden, requestlog.ActionDenied),
dropURL)
wantReputation(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward))
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
if len(waiting) != 1 || waiting[0].Detail["source"] != dropURL {
t.Errorf("alerts waiting %+v, want the blocklist's reputation_hit alone", waiting)
}
if queries := server.DNSBL.Queries(dnsblZone); queries != 0 {
t.Errorf("%d queries, want none", queries)
}
}
func TestObserveModeForwardsAClientADNSBLZoneDeniesAndAlertsIt(t *testing.T) {
t.Parallel()
s, server, queue := startWithLookups(t, map[string]string{
dnsblZones: dnsblZone, dnsblResolver: noResolver, reputationAction: actionDeny,
mode: observe,
})
loadVerdicts(server, map[string][]string{fromDE: {dnsblZone}})
line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
wantWouldAction(t, line, requestlog.ActionDenied)
wantReputation(t, line, dnsblZone)
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
if len(waiting) != 1 || waiting[0].Event != alerts.EventReputationHit {
t.Errorf("alerts waiting %+v, want a reputation_hit alert", waiting)
}
}
func TestReputationLimitTakesPartInTheLowestPercentageOfEveryLimit(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
blocklistAction, reputationAction string
// want is the request's limit_percent and bytes_percent, as
// percentText gives them.
want string
}{
{limitHalf, limitQuarter, "25 from " + reputationAction},
{limitQuarter, limitHalf, "25 from " + blocklistAction},
// The blocklist's, the first of two alike.
{limitQuarter, limitQuarter, "25 from " + blocklistAction},
{actionLog, limitQuarter, "25 from " + reputationAction},
{actionLog, actionLog, none},
} {
t.Run(tc.blocklistAction+" "+tc.reputationAction, func(t *testing.T) {
t.Parallel()
s, server, _ := startWithLookups(t, map[string]string{
blocklistURLs: dropURL, blocklistAction: tc.blocklistAction,
dnsblZones: dnsblZone, dnsblResolver: noResolver,
reputationAction: tc.reputationAction,
})
loadLists(t, server, map[string][]string{dropURL: {fromDE}})
loadVerdicts(server, map[string][]string{fromDE: {dnsblZone}})
// Named by the blocklist, then by the zone.
line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
wantReputation(t, line, dropURL, dnsblZone)
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
tc.want)
wantPercent(t, "bytes_percent", line.BytesPercent, line.BytesPercentSetting,
tc.want)
})
}
}
func TestEachZoneThatListsAClientRaisesAnAlertOncePerCooldownAndIsCounted(
t *testing.T,
) {
t.Parallel()
s, clk, server, queue := startWithLookupsAndClock(t, map[string]string{
dnsblZones: dnsblZone + "," + otherZone, dnsblResolver: noResolver,
reputationAction: actionLog, metricsToken: token,
})
loadVerdicts(server, map[string][]string{
fromDE: {dnsblZone, otherZone}, unplaced: nil,
})
// The second request's alerts are repeats, which the cooldown holds
// back.
for range 2 {
wantReputation(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward),
dnsblZone, otherZone)
}
hit := func(zone string) alerts.Alert {
return alerts.Alert{
Instance: alertInstance,
Time: clk.Now(),
Event: alerts.EventReputationHit,
Client: netip.MustParseAddr(fromDE),
Netblock: netip.MustParsePrefix(fromDE + "/32"),
ASN: asnDE,
ASName: asNameDE,
Country: "DE",
Reason: "listed by a DNSBL zone",
Detail: map[string]any{"source": zone},
}
}
wantAlerts(t, queue, hit(dnsblZone), hit(otherZone))
if queue.Suppressed() != 2 {
t.Errorf("%d alerts held back, want the second request's 2", queue.Suppressed())
}
// Each zone's hits, and its queries and their failures, none, since
// every client had its verdicts.
metrics := s.scrape(unplaced)
for _, zone := range []string{dnsblZone, otherZone} {
labels := `{instance="` + alertInstance + `",source="` + zone + `"}`
wantMetric(t, metrics, "smallwebwaf_reputation_hits_total"+labels, 2)
wantMetric(t, metrics, "smallwebwaf_reputation_queries_total"+labels, 0)
wantMetric(t, metrics, "smallwebwaf_reputation_failures_total"+labels, 0)
}
}
func TestZoneKeyIsMaskedInTheLogTheAlertAndTheMetrics(t *testing.T) {
t.Parallel()
const (
key = "abcdefghijklmnopqrstuvwxyz"
keyed = key + ".xbl.dq.spamhaus.net"
masked = "********.xbl.dq.spamhaus.net"
)
s, server, queue := startWithLookups(t, map[string]string{
dnsblZones: keyed, dnsblResolver: noResolver, reputationAction: actionLog,
metricsToken: token,
})
loadVerdicts(server, map[string][]string{fromDE: {keyed}, unplaced: nil})
wantReputation(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward), masked)
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
if len(waiting) != 1 || waiting[0].Detail["source"] != masked {
t.Errorf("alerts waiting %+v, want a reputation_hit alert from %s", waiting,
masked)
}
metrics := s.scrape(unplaced)
wantMetric(t, metrics, `smallwebwaf_reputation_hits_total{instance="`+
alertInstance+`",source="`+masked+`"}`, 1)
for name, shown := range map[string]string{
"the log": s.out.text(), "the metrics": metrics,
} {
if strings.Contains(shown, key) {
t.Errorf("%s shows the key:\n%s", name, shown)
}
}
}
func TestRequestFromAClientWithoutAVerdictHasTheZoneAskedAboutIt(t *testing.T) {
t.Parallel()
s, server, _ := startWithLookups(t, map[string]string{
dnsblZones: dnsblZone + "," + otherZone, dnsblResolver: noResolver,
})
server.DNSBL.Load([]reputation.Verdict{{
Zone: otherZone, Client: netip.MustParseAddr(fromDE), Listed: true,
Fetched: verdictsFetched(),
}})
// The verdict of the other zone is used, and dnsbl.example, which has
// none, is asked about the client in the background, once: the second
// request finds the query under way, or the zone left alone after it
// failed, since nothing answers at noResolver.
for range 2 {
wantReputation(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward), otherZone)
}
if queries := server.DNSBL.Queries(dnsblZone); queries != 1 {
t.Errorf("%d queries to %s, want 1", queries, dnsblZone)
}
if queries := server.DNSBL.Queries(otherZone); queries != 0 {
t.Errorf("%d queries to %s, want none", queries, otherZone)
}
}
// The AbuseIPDB settings, and accountKey, the key the tests set.
const (
abuseIPDBKey = "SWWAF_ABUSEIPDB_KEY"
accountKey = "abuseipdb-key-0123456789abcdef"
)
// abuseipdb is how the request log, the alerts and the metrics name
// AbuseIPDB.
const abuseipdb = reputation.AbuseIPDBSource
// abuseIPDBURL is where newProxy has clients checked with AbuseIPDB: at
// abuseIPDBStandIn, which TestMain registers with Go's default transport,
// through which AbuseIPDB is asked.
const abuseIPDBURL = "abuseipdb://stand-in/api/v2/check"
// abuseIPDBStandIn is a stand-in for AbuseIPDB that gives every client the
// score 100, at once and without the network.
type abuseIPDBStandIn struct{}
// RoundTrip answers req with the score 100.
func (abuseIPDBStandIn) RoundTrip(req *http.Request) (*http.Response, error) {
return &http.Response{
StatusCode: http.StatusOK,
Status: "200 OK",
Header: http.Header{},
Body: io.NopCloser(strings.NewReader(`{"data":{"abuseConfidenceScore":100}}`)),
Request: req,
}, nil
}
func TestOnlyAClientThatHasCommittedAnOffenceIsCheckedWithAbuseIPDB(t *testing.T) {
t.Parallel()
forward := requestlog.ActionForward
s, clk, server, _ := startWithLookupsAndClock(t, map[string]string{
abuseIPDBKey: accountKey, rateLimitPerMinute: "2", reputationAction: actionLog,
})
// Neither fromDE, until it breaks a rate limit, nor fromKP, which never
// does, is checked, nor fromDE under the ban that makes.
s.get(fromDE, http.StatusOK, forward)
s.get(fromDE, http.StatusOK, forward)
s.get(fromKP, http.StatusOK, forward)
s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited)
s.get(fromDE, http.StatusForbidden, requestlog.ActionBanned)
wantAbuseIPDBChecks(t, server, 0)
// Once the ban has ended, fromDE's first request has it checked in the
// background, and goes on without its score, which its next request
// finds.
clk.advance(time.Hour)
wantReputation(t, s.get(fromDE, http.StatusOK, forward))
wantAbuseIPDBChecks(t, server, 1)
waitUntil(func() bool { return len(server.AbuseIPDB.Snapshot().Scores) == 1 })
wantReputation(t, s.get(fromDE, http.StatusOK, forward), abuseipdb)
s.get(fromKP, http.StatusOK, forward)
wantAbuseIPDBChecks(t, server, 1)
}
func TestIPv6ClientCostsOneAbuseIPDBCheckWhicheverOfItsAddressesSends(t *testing.T) {
t.Parallel()
forward := requestlog.ActionForward
// 15 addresses of 2001:db8:1:2::/64, one client, each in a part of it
// of its own.
var addresses []string
for i := 1; i < 16; i++ {
addresses = append(addresses, fmt.Sprintf("2001:db8:1:2:%x::9", i<<12))
}
s, clk, server := startWithClock(t, "", map[string]string{
abuseIPDBKey: accountKey, reputationAction: actionLog,
rateLimitPerMinute: strconv.Itoa(len(addresses)),
})
// The client breaks the rate limit from its first address, which bans
// it for an hour.
for range addresses {
s.get(addresses[0], http.StatusOK, forward)
}
s.get(addresses[0], http.StatusForbidden, requestlog.ActionRateLimited)
clk.advance(time.Hour)
// Once the ban has ended, which set its counters back to zero, a
// request from each of its addresses has it checked once.
for _, address := range addresses {
s.get(address, http.StatusOK, forward)
}
wantAbuseIPDBChecks(t, server, 1)
}
// probePath is the path the ban rule of testRules, probe, matches.
const probePath = "/.env"
func TestClientARuleRefusedIsCheckedWithAbuseIPDBAtItsNextRequest(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
// path is what the client asks for, status and action what that
// request is answered and logged with, and want the offences its
// history then counts.
path string
status int
action string
want ratelimit.Offences
}{
{
"a block rule", "/blocked", http.StatusForbidden, requestlog.ActionRuleBlocked,
ratelimit.Offences{RuleBlocked: 1},
},
{
"a ban rule", probePath, http.StatusForbidden, requestlog.ActionBanned,
ratelimit.Offences{Attack: 1},
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
s, clk, server := startWithClock(t, "", map[string]string{
abuseIPDBKey: accountKey, reputationAction: actionLog,
rulesDir: writeRules(t, testRules), attackBanDuration: "1h",
})
s.request(client, tc.path, tc.status, tc.action)
wantAbuseIPDBChecks(t, server, 0)
if got := historyOf(t, server, client).Offences; got != tc.want {
t.Errorf("history counts the offences %+v, want %+v", got, tc.want)
}
// Its next request, once a ban rule's ban has ended, has it
// checked.
clk.advance(time.Hour)
s.get(client, http.StatusOK, requestlog.ActionForward)
wantAbuseIPDBChecks(t, server, 1)
})
}
}
func TestEachReputationActionForAClientAbuseIPDBScoresAtOrOverTheMinimum(t *testing.T) {
t.Parallel()
forward, denied := requestlog.ActionForward, requestlog.ActionDenied
for _, tc := range []struct {
action string
// statuses and actions are those of fromDE's three requests, and
// percent their limit_percent, as percentText gives it.
statuses []int
actions []string
percent string
}{
{
actionDeny, []int{http.StatusForbidden, http.StatusForbidden, http.StatusForbidden},
[]string{denied, denied, denied}, none,
},
{
// Half of 4 requests a minute: the third breaks the limit.
limitHalf, []int{http.StatusOK, http.StatusOK, http.StatusForbidden},
[]string{forward, forward, requestlog.ActionRateLimited},
"50 from " + reputationAction,
},
{
actionLog, []int{http.StatusOK, http.StatusOK, http.StatusOK},
[]string{forward, forward, forward}, none,
},
} {
t.Run(tc.action, func(t *testing.T) {
t.Parallel()
s, server, _ := startWithLookups(t, map[string]string{
rateLimitPerMinute: fourAMinute, abuseIPDBKey: accountKey,
reputationAction: tc.action,
})
// At SWWAF_ABUSEIPDB_MIN_SCORE, 75 by default, and just under it.
loadScores(server, map[string]int64{fromDE: 75, fromKP: 74})
for i := range 3 {
line := s.get(fromDE, tc.statuses[i], tc.actions[i])
wantReputation(t, line, abuseipdb)
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
tc.percent)
// A request refused for the score is not counted.
counted := line.fields["counts"] != nil
if counted != (tc.actions[i] != denied) {
t.Errorf("request counted %t, logged %s", counted, tc.actions[i])
}
}
// fromKP's score is no hit, and it has the whole limit.
for range 3 {
line := s.get(fromKP, http.StatusOK, forward)
wantReputation(t, line)
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
none)
}
// A refusal for the score makes no ban.
if held := server.Ledger.Snapshot(); tc.action == actionDeny && len(held) != 0 {
t.Errorf("bans %+v, want none", held)
}
})
}
}
func TestAbuseIPDBHitRaisesAnAlertWithTheScoreOncePerCooldownAndIsCounted(
t *testing.T,
) {
t.Parallel()
s, server, queue := startWithLookups(t, map[string]string{
abuseIPDBKey: accountKey, reputationAction: actionLog, metricsToken: token,
})
loadScores(server, map[string]int64{fromDE: 90})
// The second request's alert is a repeat, which the cooldown holds back.
for range 2 {
wantReputation(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward),
abuseipdb)
}
// The alert is made as a DNSBL zone's is, with the score besides.
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
if len(waiting) != 1 || waiting[0].Event != alerts.EventReputationHit ||
waiting[0].Reason != "scored by AbuseIPDB at or over SWWAF_ABUSEIPDB_MIN_SCORE" ||
waiting[0].Detail["source"] != abuseipdb || waiting[0].Detail["score"] != int64(90) ||
queue.Suppressed() != 1 {
t.Errorf("alerts waiting %+v, %d held back, want AbuseIPDB's reputation_hit "+
"with the score 90, and 1", waiting, queue.Suppressed())
}
// The hits, and the checks, none, since no client committed an
// offence, so that the whole budget is left.
metrics := s.scrape(unplaced)
labels := `{instance="` + alertInstance + `",source="` + abuseipdb + `"}`
wantMetric(t, metrics, "smallwebwaf_reputation_hits_total"+labels, 2)
wantMetric(t, metrics, "smallwebwaf_reputation_queries_total"+labels, 0)
wantMetric(t, metrics, "smallwebwaf_reputation_failures_total"+labels, 0)
wantMetric(t, metrics, "smallwebwaf_reputation_daily_budget_remaining"+labels, 900)
}
func TestWithoutAnAbuseIPDBKeyNoClientIsCheckedNorAScoreUsed(t *testing.T) {
t.Parallel()
forward := requestlog.ActionForward
s, clk, server, _ := startWithLookupsAndClock(t, map[string]string{
rateLimitPerMinute: "1", metricsToken: token,
})
loadScores(server, map[string]int64{fromDE: 100})
// fromDE's score is not used, and once it has committed an offence it
// is not checked either.
wantReputation(t, s.get(fromDE, http.StatusOK, forward))
s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited)
clk.advance(time.Hour)
wantReputation(t, s.get(fromDE, http.StatusOK, forward))
wantAbuseIPDBChecks(t, server, 0)
wantNoSeries(t, s.scrape(unplaced), `smallwebwaf_reputation_daily_budget_remaining{`+
`instance="`+alertInstance+`",source="`+abuseipdb+`"}`)
}
// listsFetched is when loadLists has the copies fetched.
func listsFetched() time.Time {
return time.Date(2026, 10, 5, 0, 0, 0, 0, time.UTC)
}
// verdictsFetched is when loadVerdicts has the verdicts fetched: half a
// day before the time the tests' clock is set to, so that they are in use
// until half a day later.
func verdictsFetched() time.Time {
return time.Date(2026, 10, 5, 12, 0, 0, 0, time.UTC)
}
// loadVerdicts puts into server's DNSBL, for each client listedBy names,
// a verdict of each zone SWWAF_DNSBL_ZONES names, fetched at
// verdictsFetched, as reputation.json would at start: one that lists the
// client from each zone listedBy gives for it, and one that does not from
// each other zone.
func loadVerdicts(server *proxy.Server, listedBy map[string][]string) {
verdicts := make([]reputation.Verdict, 0, len(listedBy)*len(server.DNSBL.Zones()))
for client, zones := range listedBy {
for _, zone := range server.DNSBL.Zones() {
verdicts = append(verdicts, reputation.Verdict{
Zone: zone, Client: netip.MustParseAddr(client),
Listed: slices.Contains(zones, zone), Fetched: verdictsFetched(),
})
}
}
server.DNSBL.Load(verdicts)
}
// loadScores puts into server's AbuseIPDB the score scores gives each
// client, an IPv4 address, fetched at verdictsFetched, as reputation.json
// would at start.
func loadScores(server *proxy.Server, scores map[string]int64) {
kept := make([]reputation.Score, 0, len(scores))
for client, score := range scores {
kept = append(kept, reputation.Score{
Client: netip.MustParsePrefix(client + "/32"), Score: score,
Fetched: verdictsFetched(),
})
}
server.AbuseIPDB.Load(reputation.Checks{Scores: kept})
}
// wantAbuseIPDBChecks checks how many clients server has checked with
// AbuseIPDB.
func wantAbuseIPDBChecks(t *testing.T, server *proxy.Server, want int) {
t.Helper()
if got := server.AbuseIPDB.Checked(); got != want {
t.Errorf("%d clients checked with AbuseIPDB, want %d", got, want)
}
}
// loadLists puts copies of lists into server's lists, by URL, each with
// its lines, fetched at listsFetched, as reputation.json would at start.
func loadLists(t *testing.T, server *proxy.Server, copies map[string][]string) {
t.Helper()
lists := make([]reputation.List, 0, len(copies))
for listURL, lines := range copies {
lists = append(lists, reputation.List{
URL: listURL, Fetched: listsFetched(), Lines: lines,
})
}
err := server.Lists.Load(lists)
if err != nil {
t.Fatalf("load the lists: %v", err)
}
}
// wantReputation checks the URLs of the blocklists the log line names in
// its reputation.
func wantReputation(t *testing.T, line logLine, want ...string) {
t.Helper()
if !slices.Equal(line.Reputation, want) {
t.Errorf("log line has reputation %v, want %v", line.Reputation, want)
}
}
internal/proxy/request.go
+9 -28
View File
@@ -63,15 +63,7 @@ type request struct {
// limits and for the byte limits. // limits and for the byte limits.
counted bool counted bool
limitPercent, bytesPercent percentage limitPercent, bytesPercent percentage
// attack is true for a request that matched a ban rule, and start time.Time
// ruleBlocked for one a block rule refused, each an offence its
// client's history counts.
attack, ruleBlocked bool
// blocklisted is true once a blocklist is found to list the client,
// dnsblListed once a DNSBL zone's verdict is, and abuseIPDBHit once
// AbuseIPDB's score of it is a hit.
blocklisted, dnsblListed, abuseIPDBHit bool
start time.Time
// checked is when the checks were done, and upstreamStart when the // checked is when the checks were done, and upstreamStart when the
// request was handed to the app. // request was handed to the app.
checked time.Time checked time.Time
@@ -140,7 +132,7 @@ func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request {
RequestID: requestID(r, peerTrusted), RequestID: requestID(r, peerTrusted),
PeerIP: peer.String(), PeerIP: peer.String(),
ForwardedFor: strings.Join(forwardedFor, ", "), ForwardedFor: strings.Join(forwardedFor, ", "),
ClientGroup: h.clientGroup(client).String(), ClientGroup: clientGroup(client).String(),
ContentType: r.Header.Get("Content-Type"), ContentType: r.Header.Get("Content-Type"),
RequestHeaders: requestHeaders(r, h.config.LogRequestHeaders), RequestHeaders: requestHeaders(r, h.config.LogRequestHeaders),
HasAuthorization: len(r.Header.Values("Authorization")) > 0, HasAuthorization: len(r.Header.Values("Authorization")) > 0,
@@ -221,10 +213,9 @@ func (rq *request) check(ctx context.Context) *refusal {
// client in SWWAF_ALLOW_NETS skips them, and is not looked up. For any // client in SWWAF_ALLOW_NETS skips them, and is not looked up. For any
// other client, SWWAF_DENY_NETS comes first, then a ban on its netblock, // other client, SWWAF_DENY_NETS comes first, then a ban on its netblock,
// so that a client either refuses is not looked up, then the lookup of // so that a client either refuses is not looked up, then the lookup of
// its AS number and country, then the country lists, then the blocklists, // its AS number and country, and then the country lists; a request any of
// then the DNSBL zones' verdicts, and then AbuseIPDB's score; a request // them refuses is not counted for the rate limits. Then come the rate
// any of them refuses is not counted for the rate limits. Then come the // limits, unless the client is in SWWAF_RATE_LIMIT_EXEMPT_NETS or the
// rate limits, unless the client is in SWWAF_RATE_LIMIT_EXEMPT_NETS or the
// request's path is exempt under SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that // request's path is exempt under SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that
// every other request is counted, each of them by the client's limit // every other request is counted, each of them by the client's limit
// percentages, and last the rule files. A request exempt from the rate // percentages, and last the rule files. A request exempt from the rate
@@ -252,18 +243,10 @@ func (rq *request) checkClient(ctx context.Context) string {
return requestlog.ActionCountryDenied return requestlog.ActionCountryDenied
} }
if rq.blocklistDenied() {
return requestlog.ActionDenied
}
if rq.dnsblDenied(ctx) || rq.abuseIPDBDenied(ctx) {
return requestlog.ActionDenied
}
rq.counted = !isInside(rq.client, cfg.RateLimitExemptNets) && rq.counted = !isInside(rq.client, cfg.RateLimitExemptNets) &&
!pathExempt(rq.in.URL, cfg.RateLimitExemptPaths) !pathExempt(rq.in.URL, cfg.RateLimitExemptPaths)
if rq.counted { if rq.counted {
rq.limitPercent, rq.bytesPercent = rq.limitPercentages() rq.limitPercent, rq.bytesPercent = limitPercentages(cfg, rq.line.ASN, rq.line.Country)
rq.line.LimitPercent, rq.line.LimitPercentSetting = rq.limitPercent.logged() rq.line.LimitPercent, rq.line.LimitPercentSetting = rq.limitPercent.logged()
rq.line.BytesPercent, rq.line.BytesPercentSetting = rq.bytesPercent.logged() rq.line.BytesPercent, rq.line.BytesPercentSetting = rq.bytesPercent.logged()
} }
@@ -539,15 +522,13 @@ func timing(start, end time.Time) *float64 {
func (rq *request) addToHistory() { func (rq *request) addToHistory() {
forwarded := !rq.upstreamStart.IsZero() forwarded := !rq.upstreamStart.IsZero()
rq.h.limiter.AddToHistory(rq.h.clientGroup(rq.client), rq.h.now(), ratelimit.Request{ rq.h.limiter.AddToHistory(clientGroup(rq.client), rq.h.now(), ratelimit.Request{
Forwarded: forwarded, Forwarded: forwarded,
Refused: !forwarded && rq.refused.Load() != nil, Refused: !forwarded && rq.refused.Load() != nil,
Status: rq.out.status, Status: rq.out.status,
RequestBytes: rq.requestBytes(), RequestBytes: rq.requestBytes(),
ResponseBytes: rq.out.bytes, ResponseBytes: rq.out.bytes,
BrokeLimit: rq.line.Offence == requestlog.OffenceLimit, BrokeLimit: rq.line.Offence == requestlog.OffenceLimit,
Attack: rq.attack,
RuleBlocked: rq.ruleBlocked,
}) })
answer, found := rq.answerAtTheEnd() answer, found := rq.answerAtTheEnd()
@@ -572,7 +553,7 @@ func (rq *request) countAnomalies() {
rq.h.anomalies.Count(rq.h.now(), anomaly.Request{ rq.h.anomalies.Count(rq.h.now(), anomaly.Request{
Client: rq.client, Client: rq.client,
ClientGroup: rq.h.clientGroup(rq.client), ClientGroup: clientGroup(rq.client),
ASN: answer.ASN, ASN: answer.ASN,
ASName: answer.ASName, ASName: answer.ASName,
Country: answer.Country, Country: answer.Country,
@@ -601,7 +582,7 @@ func (rq *request) answerAtTheEnd() (lookup.Answer, bool) {
return rq.lookupAnswer, true return rq.lookupAnswer, true
} }
return rq.h.geojs.Kept(rq.h.clientGroup(rq.client)) return rq.h.geojs.Kept(clientGroup(rq.client))
} }
// requestBytes is how many bytes of the request's body have been read. // requestBytes is how many bytes of the request's body have been read.
internal/proxy/rules.go
+1 -5
View File
@@ -12,8 +12,7 @@ import (
// action of the rule that refuses it, ActionRuleBlocked for a block rule // action of the rule that refuses it, ActionRuleBlocked for a block rule
// and ActionBanned for a ban rule, or "" when none does. A ban rule bans // and ActionBanned for a ban rule, or "" when none does. A ban rule bans
// the client's netblock for a clear sign of attack, or in observe mode // the client's netblock for a clear sign of attack, or in observe mode
// raises the alert for the ban it would have made. Either rule's match // raises the alert for the ban it would have made.
// is noted as an offence, for the client's history.
func (rq *request) checkRules(now time.Time) string { func (rq *request) checkRules(now time.Time) string {
matched := rq.h.rules.Match(rq.in) matched := rq.h.rules.Match(rq.in)
@@ -29,11 +28,8 @@ func (rq *request) checkRules(now time.Time) string {
// Only the last rule matched can refuse the request. // Only the last rule matched can refuse the request.
switch last := matched[len(matched)-1]; last.Action { switch last := matched[len(matched)-1]; last.Action {
case rules.ActionBlock: case rules.ActionBlock:
rq.ruleBlocked = true
return requestlog.ActionRuleBlocked return requestlog.ActionRuleBlocked
case rules.ActionBan: case rules.ActionBan:
rq.attack = true
rq.banForAttack(now, last) rq.banForAttack(now, last)
return requestlog.ActionBanned return requestlog.ActionBanned
internal/ratelimit/history_test.go
+4 -4
View File
@@ -11,7 +11,7 @@ import (
func TestHistoryKeepsEveryRequest(t *testing.T) { func TestHistoryKeepsEveryRequest(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize) limiter := ratelimit.New(ratelimit.Limits{})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -53,7 +53,7 @@ func TestHistoryKeepsEveryRequest(t *testing.T) {
func TestLookupReachesTheHistoryOfAClientInTheTable(t *testing.T) { func TestLookupReachesTheHistoryOfAClientInTheTable(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize) limiter := ratelimit.New(ratelimit.Limits{})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
other := netip.MustParsePrefix("198.51.100.7/32") other := netip.MustParsePrefix("198.51.100.7/32")
start := midnight() start := midnight()
@@ -90,7 +90,7 @@ func TestLookupReachesTheHistoryOfAClientInTheTable(t *testing.T) {
func TestResetKeepsTheHistory(t *testing.T) { func TestResetKeepsTheHistory(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit}, tableSize) limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -109,7 +109,7 @@ func TestResetKeepsTheHistory(t *testing.T) {
func TestRequestsAddsUpTheClientsInsideTheNetblock(t *testing.T) { func TestRequestsAddsUpTheClientsInsideTheNetblock(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize) limiter := ratelimit.New(ratelimit.Limits{})
for client, requests := range map[string]int{ for client, requests := range map[string]int{
"198.51.100.9/32": 2, "198.51.100.9/32": 2,
internal/ratelimit/ratelimit.go
+13 -29
View File
@@ -2,9 +2,8 @@
// and bytes counted over a minute, an hour and a day, as the "Counting // and bytes counted over a minute, an hour and a day, as the "Counting
// method" section of SPEC.md describes, which tell when a request takes // method" section of SPEC.md describes, which tell when a request takes
// the client over a rate limit or a byte limit, and each client's history // the client over a rate limit or a byte limit, and each client's history
// since it was first seen. At most SWWAF_MAX_TRACKED_CLIENTS clients are // since it was first seen. At most 20,000 clients are kept, in memory, and
// kept, in memory, and written to clients.json and read from it by the // written to clients.json and read from it by the state package.
// state package.
package ratelimit package ratelimit
import ( import (
@@ -17,6 +16,11 @@ import (
"github.com/hashicorp/golang-lru/v2/simplelru" "github.com/hashicorp/golang-lru/v2/simplelru"
) )
// maxClients is how many clients are kept. Past it, the least recently
// seen client is dropped, with its history, and starts afresh if it comes
// back.
const maxClients = 20000
const day = 24 * time.Hour const day = 24 * time.Hour
// The kinds of limits, as the metrics name them. // The kinds of limits, as the metrics name them.
@@ -113,15 +117,9 @@ type Responses struct {
} }
// Offences are a client's offences, by kind. // Offences are a client's offences, by kind.
//
//nolint:tagliatelle // the state files use snake_case, as the request log does
type Offences struct { type Offences struct {
// Limit is its requests that broke a rate limit or a byte limit, // Limit is its requests that broke a rate limit or a byte limit.
// Attack those that matched a ban rule, a clear sign of attack, and Limit int64 `json:"limit"`
// RuleBlocked those a block rule refused.
Limit int64 `json:"limit"`
Attack int64 `json:"attack"`
RuleBlocked int64 `json:"rule_blocked"`
} }
// Request is what a client's history keeps of one of its requests. // Request is what a client's history keeps of one of its requests.
@@ -139,18 +137,12 @@ type Request struct {
RequestBytes int64 RequestBytes int64
ResponseBytes int64 ResponseBytes int64
// BrokeLimit is true for a request that broke a rate limit or a byte // BrokeLimit is true for a request that broke a rate limit or a byte
// limit, Attack for one that matched a ban rule, and RuleBlocked for // limit.
// one a block rule refused. BrokeLimit bool
BrokeLimit bool
Attack bool
RuleBlocked bool
} }
// New returns a Limiter for limits, with no client counted yet, whose // New returns a Limiter for limits, with no client counted yet.
// table holds at most maxClients clients (SWWAF_MAX_TRACKED_CLIENTS). Past func New(limits Limits) *Limiter {
// it, the least recently seen client is dropped, with its history, and
// starts afresh if it comes back.
func New(limits Limits, maxClients int) *Limiter {
clients, err := simplelru.NewLRU[netip.Prefix, *Client](maxClients, nil) clients, err := simplelru.NewLRU[netip.Prefix, *Client](maxClients, nil)
if err != nil { if err != nil {
panic(err) // NewLRU fails only for a size below one panic(err) // NewLRU fails only for a size below one
@@ -266,14 +258,6 @@ func (l *Limiter) AddToHistory(client netip.Prefix, now time.Time, r Request) {
if r.BrokeLimit { if r.BrokeLimit {
h.Offences.Limit++ h.Offences.Limit++
} }
if r.Attack {
h.Offences.Attack++
}
if r.RuleBlocked {
h.Offences.RuleBlocked++
}
} }
// AddLookup gives client's history its AS number, AS name and country, as // AddLookup gives client's history its AS number, AS name and country, as
internal/ratelimit/ratelimit_test.go
+17 -30
View File
@@ -12,10 +12,6 @@ import (
// limit is the limit the tests set. // limit is the limit the tests set.
const limit = 3 const limit = 3
// tableSize is the most clients the tests' tables hold, the default of
// SWWAF_MAX_TRACKED_CLIENTS.
const tableSize = 20000
// whole is the percentage of each limit a client gets when nothing lowers // whole is the percentage of each limit a client gets when nothing lowers
// its limits. // its limits.
const whole = 100 const whole = 100
@@ -41,7 +37,7 @@ func TestEachWindowRefusesAtItsLimitAndLetsTheClientBack(t *testing.T) {
t.Run(tc.window, func(t *testing.T) { t.Run(tc.window, func(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(tc.limits, tableSize) limiter := ratelimit.New(tc.limits)
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
quarter := tc.length / 4 quarter := tc.length / 4
@@ -66,7 +62,7 @@ func TestEachWindowRefusesAtItsLimitAndLetsTheClientBack(t *testing.T) {
func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) { func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit, PerHour: limit}, tableSize) limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit, PerHour: limit})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -92,8 +88,7 @@ func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
func TestClientGetsItsPercentageOfEachLimitRoundedDown(t *testing.T) { func TestClientGetsItsPercentageOfEachLimitRoundedDown(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerMinute: 5, BytesPerDay: math.MaxInt64}, limiter := ratelimit.New(ratelimit.Limits{PerMinute: 5, BytesPerDay: math.MaxInt64})
tableSize)
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -124,8 +119,7 @@ func TestZeroPercentIsAZeroAllowanceAndALimitOffStaysOff(t *testing.T) {
t.Parallel() t.Parallel()
// Only the hour has limits: the minute's and the day's are off. // Only the hour has limits: the minute's and the day's are off.
limiter := ratelimit.New(ratelimit.Limits{PerHour: limit, BytesPerHour: 1000}, limiter := ratelimit.New(ratelimit.Limits{PerHour: limit, BytesPerHour: 1000})
tableSize)
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -162,7 +156,7 @@ func TestEachByteLimitIsBrokenByTheBytesCounted(t *testing.T) {
t.Run(tc.window, func(t *testing.T) { t.Run(tc.window, func(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(tc.limits, tableSize) limiter := ratelimit.New(tc.limits)
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
// 600 bytes are within the limit, 600 more over it. // 600 bytes are within the limit, 600 more over it.
@@ -186,8 +180,7 @@ func TestEachByteLimitIsBrokenByTheBytesCounted(t *testing.T) {
func TestALimitIsBrokenOnlyByWhatIsAddedToIt(t *testing.T) { func TestALimitIsBrokenOnlyByWhatIsAddedToIt(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerMinute: 2, BytesPerMinute: 1000}, limiter := ratelimit.New(ratelimit.Limits{PerMinute: 2, BytesPerMinute: 1000})
tableSize)
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
other := netip.MustParsePrefix("203.0.113.10/32") other := netip.MustParsePrefix("203.0.113.10/32")
start := midnight() start := midnight()
@@ -211,7 +204,7 @@ func TestALimitIsBrokenOnlyByWhatIsAddedToIt(t *testing.T) {
func TestCountGivesTheBytesInEachWindow(t *testing.T) { func TestCountGivesTheBytesInEachWindow(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize) limiter := ratelimit.New(ratelimit.Limits{})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -237,7 +230,7 @@ func TestCountGivesTheBytesInEachWindow(t *testing.T) {
func TestResetSetsTheBytesBackToZero(t *testing.T) { func TestResetSetsTheBytesBackToZero(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{BytesPerDay: 1000}, tableSize) limiter := ratelimit.New(ratelimit.Limits{BytesPerDay: 1000})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -251,7 +244,7 @@ func TestResetSetsTheBytesBackToZero(t *testing.T) {
func TestCountGivesTheRequestsInEachWindow(t *testing.T) { func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize) limiter := ratelimit.New(ratelimit.Limits{})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -274,7 +267,7 @@ func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
func TestResetSetsTheCountsBackToZero(t *testing.T) { func TestResetSetsTheCountsBackToZero(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit, PerDay: limit}, tableSize) limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit, PerDay: limit})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -296,7 +289,7 @@ func TestResetSetsTheCountsBackToZero(t *testing.T) {
func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) { func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize) limiter := ratelimit.New(ratelimit.Limits{PerHour: limit})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -315,8 +308,7 @@ func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) {
func TestRefusedRequestsCount(t *testing.T) { func TestRefusedRequestsCount(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit, PerHour: 2 * limit}, limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit, PerHour: 2 * limit})
tableSize)
refused := netip.MustParsePrefix("203.0.113.9/32") refused := netip.MustParsePrefix("203.0.113.9/32")
within := netip.MustParsePrefix("203.0.113.10/32") within := netip.MustParsePrefix("203.0.113.10/32")
start := midnight() start := midnight()
@@ -349,7 +341,7 @@ func TestRefusedRequestsCount(t *testing.T) {
func TestRequestCountedLateGoesInTheBucketUnderWay(t *testing.T) { func TestRequestCountedLateGoesInTheBucketUnderWay(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit}, tableSize) limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -365,7 +357,7 @@ func TestRequestCountedLateGoesInTheBucketUnderWay(t *testing.T) {
func TestClockSetBackStartsTheBucketsAfresh(t *testing.T) { func TestClockSetBackStartsTheBucketsAfresh(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize) limiter := ratelimit.New(ratelimit.Limits{PerHour: limit})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -388,12 +380,12 @@ func TestClockSetBackStartsTheBucketsAfresh(t *testing.T) {
wantCount(t, limiter, client, setBack, hour) wantCount(t, limiter, client, setBack, hour)
} }
func TestKeepsAtMostMaxClientsDroppingTheLeastRecentlySeen(t *testing.T) { func TestKeepsAtMost20000ClientsDroppingTheLeastRecentlySeen(t *testing.T) {
t.Parallel() t.Parallel()
const maxClients = 3 const maxClients = 20000
limiter := ratelimit.New(ratelimit.Limits{PerMinute: 1}, maxClients) limiter := ratelimit.New(ratelimit.Limits{PerMinute: 1})
now := midnight() now := midnight()
clients := make([]netip.Prefix, maxClients+1) clients := make([]netip.Prefix, maxClients+1)
@@ -415,11 +407,6 @@ func TestKeepsAtMostMaxClientsDroppingTheLeastRecentlySeen(t *testing.T) {
// One client more drops the least recently seen, the second, which // One client more drops the least recently seen, the second, which
// starts afresh, while the first is kept. // starts afresh, while the first is kept.
wantCount(t, limiter, clients[maxClients], now, "") wantCount(t, limiter, clients[maxClients], now, "")
if limiter.Len() != maxClients {
t.Errorf("the table holds %d clients, want %d", limiter.Len(), maxClients)
}
wantCount(t, limiter, clients[1], now, "") wantCount(t, limiter, clients[1], now, "")
wantCount(t, limiter, clients[0], now, minute) wantCount(t, limiter, clients[0], now, minute)
} }
internal/ratelimit/snapshot_test.go
+7 -7
View File
@@ -14,7 +14,7 @@ func TestSnapshotListsTheClientsByAddress(t *testing.T) {
want := []string{"192.0.2.1/32", "203.0.113.9/32", "203.0.113.10/32", "2001:db8::/64"} want := []string{"192.0.2.1/32", "203.0.113.9/32", "203.0.113.10/32", "2001:db8::/64"}
limiter := ratelimit.New(ratelimit.Limits{}, tableSize) limiter := ratelimit.New(ratelimit.Limits{})
for _, i := range []int{2, 3, 0, 1} { for _, i := range []int{2, 3, 0, 1} {
limiter.Count(netip.MustParsePrefix(want[i]), midnight(), whole) limiter.Count(netip.MustParsePrefix(want[i]), midnight(), whole)
} }
@@ -43,7 +43,7 @@ func TestLoadedCountsCarryOn(t *testing.T) {
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
before := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize) before := ratelimit.New(ratelimit.Limits{PerHour: limit})
for range limit { for range limit {
wantCount(t, before, client, start, "") wantCount(t, before, client, start, "")
} }
@@ -51,7 +51,7 @@ func TestLoadedCountsCarryOn(t *testing.T) {
// Loaded into a new limiter, as across a restart, the client has no // Loaded into a new limiter, as across a restart, the client has no
// fresh allowance. // fresh allowance.
later := start.Add(time.Minute) later := start.Add(time.Minute)
after := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize) after := ratelimit.New(ratelimit.Limits{PerHour: limit})
after.Load(before.Snapshot(), later) after.Load(before.Snapshot(), later)
wantCount(t, after, client, later, hour) wantCount(t, after, client, later, hour)
} }
@@ -62,7 +62,7 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize) limiter := ratelimit.New(ratelimit.Limits{})
limiter.Count(client, start, whole) limiter.Count(client, start, whole)
limiter.CountBytes(client, start, 5, whole) limiter.CountBytes(client, start, 5, whole)
limiter.AddToHistory(client, start, ratelimit.Request{Forwarded: true}) limiter.AddToHistory(client, start, ratelimit.Request{Forwarded: true})
@@ -70,7 +70,7 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
loaded := func(now time.Time) ratelimit.Client { loaded := func(now time.Time) ratelimit.Client {
t.Helper() t.Helper()
after := ratelimit.New(ratelimit.Limits{}, tableSize) after := ratelimit.New(ratelimit.Limits{})
after.Load(limiter.Snapshot(), now) after.Load(limiter.Snapshot(), now)
return after.Snapshot()[0] return after.Snapshot()[0]
@@ -102,7 +102,7 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
func TestLoadDropsTheLeastRecentlySeenFirst(t *testing.T) { func TestLoadDropsTheLeastRecentlySeenFirst(t *testing.T) {
t.Parallel() t.Parallel()
const maxClients = 3 const maxClients = 20000
// clients.json lists the clients by address. Here each was last seen // clients.json lists the clients by address. Here each was last seen
// a second before the one listed before it, so the last listed is the // a second before the one listed before it, so the last listed is the
@@ -116,7 +116,7 @@ func TestLoadDropsTheLeastRecentlySeenFirst(t *testing.T) {
addr = addr.Next() addr = addr.Next()
} }
limiter := ratelimit.New(ratelimit.Limits{}, maxClients) limiter := ratelimit.New(ratelimit.Limits{})
limiter.Load(clients, midnight()) limiter.Load(clients, midnight())
got := limiter.Snapshot() got := limiter.Snapshot()
internal/reputation/abuseipdb.go
-328
View File
@@ -1,328 +0,0 @@
package reputation
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"net/http"
"net/netip"
"net/url"
"slices"
"sync"
"time"
"github.com/hashicorp/golang-lru/v2/simplelru"
"sneak.berlin/go/smallwebwaf/internal/alerts"
)
const (
// AbuseIPDBURL is where clients are checked: the check endpoint of
// AbuseIPDB's API.
AbuseIPDBURL = "https://api.abuseipdb.com/api/v2/check"
// AbuseIPDBSource is how the request log, the alerts and the metrics
// name AbuseIPDB.
AbuseIPDBSource = "abuseipdb"
// maxAnswerBytes is the most of an answer of AbuseIPDB that is read.
maxAnswerBytes = 64 << 10
// day is the length of the day the checks are counted in, in UTC.
day = 24 * time.Hour
)
var (
errNoScore = errors.New("the answer gives no abuseConfidenceScore")
errBudgetUsedUp = errors.New(
"checks spent; none is made until the day ends at 00:00 UTC")
)
// Score is what AbuseIPDB said about a client, as reputation.json holds
// it: the client, an IPv4 address or an IPv6 group, its abuse confidence
// score, from 0 to 100, and when AbuseIPDB answered.
type Score struct {
Client netip.Prefix `json:"client"`
Score int64 `json:"score"`
Fetched time.Time `json:"fetched"`
}
// Checks are what reputation.json keeps of the checks of clients with
// AbuseIPDB: the day, in UTC, of the checks Spent counts, zero before the
// first, and the scores still in use.
type Checks struct {
Day time.Time `json:"day,omitzero"`
Spent int `json:"spent"`
Scores []Score `json:"scores"`
}
// AbuseIPDBParams are what NewAbuseIPDB needs.
type AbuseIPDBParams struct {
// URL is where clients are checked, normally AbuseIPDBURL, with Key,
// the account's key (SWWAF_ABUSEIPDB_KEY).
URL string
Key string
// MinScore is the least score that is a hit (SWWAF_ABUSEIPDB_MIN_SCORE),
// and DailyBudget the most checks made in a day, in UTC
// (SWWAF_ABUSEIPDB_DAILY_BUDGET).
MinScore int64
DailyBudget int
// CacheTTL is how long a score is used after it was fetched
// (SWWAF_REPUTATION_CACHE_TTL), and Timeout how long a check may take
// (SWWAF_REPUTATION_TIMEOUT).
CacheTTL time.Duration
Timeout time.Duration
// Now tells the time, normally time.Now in UTC.
Now func() time.Time
// ProcessLog receives each check that fails, and why, and the day's
// budget used up.
ProcessLog *slog.Logger
// Alerts receive a source_failure alert for each.
Alerts *alerts.Queue
}
// AbuseIPDB checks clients with AbuseIPDB, in the background, and keeps
// their scores. It is safe for concurrent use.
type AbuseIPDB struct {
params AbuseIPDBParams
httpClient *http.Client
mu sync.Mutex
// scores are by client. Each is added as it is fetched and never moved
// up, so that the one fetched longest ago is the first dropped.
scores *simplelru.LRU[netip.Prefix, Score]
// checking are the clients whose check is under way.
checking map[netip.Prefix]bool
// day is the day, in UTC, of the checks spent counts.
day time.Time
spent int
// checks and failures count the checks made and those that failed,
// and retryAt is when a client may be checked again after the last
// check failed.
checks int
failures int
retryAt time.Time
}
// NewAbuseIPDB returns an AbuseIPDB with no score yet, and no check spent.
func NewAbuseIPDB(params AbuseIPDBParams) *AbuseIPDB {
scores, err := simplelru.NewLRU[netip.Prefix, Score](maxVerdicts, nil)
if err != nil {
panic(err) // NewLRU fails only for a size below one
}
return &AbuseIPDB{
params: params,
httpClient: &http.Client{},
scores: scores,
checking: map[netip.Prefix]bool{},
}
}
// Hit returns AbuseIPDB's score of client, an IPv4 address or an IPv6
// group, and whether it is a hit: MinScore or more. A score is used until
// CacheTTL has passed since it was fetched, whichever of the client's
// addresses its request comes from. A client without one is checked in
// the background, by addr, the address its request came from, if
// offender, if it has committed an offence, unless its check is under
// way, a check failed less than failureDelay ago, or the day's checks
// have used up DailyBudget; Hit never waits for a check. The check that
// uses the budget up is logged and raised as a source_failure alert. ctx
// is the context of the client's request, and a check goes on after the
// request ends.
func (a *AbuseIPDB) Hit(
ctx context.Context, client netip.Prefix, addr netip.Addr, offender bool,
) (int64, bool) {
a.mu.Lock()
now := a.params.Now()
kept, found := a.scores.Peek(client)
if found && now.Sub(kept.Fetched) < a.params.CacheTTL {
a.mu.Unlock()
return kept.Score, kept.Score >= a.params.MinScore
}
if today := now.Truncate(day); !a.day.Equal(today) {
a.day, a.spent = today, 0
}
check := offender && !a.checking[client] && !now.Before(a.retryAt) &&
a.spent < a.params.DailyBudget
if check {
a.checking[client] = true
a.checks++
a.spent++
go a.check(context.WithoutCancel(ctx), client, addr)
}
usedUp := check && a.spent == a.params.DailyBudget
a.mu.Unlock()
if usedUp {
a.alert("the daily budget of AbuseIPDB checks is used up",
fmt.Errorf("%d %w", a.params.DailyBudget, errBudgetUsedUp))
}
return 0, false
}
// Checked returns how many checks were made.
func (a *AbuseIPDB) Checked() int {
a.mu.Lock()
defer a.mu.Unlock()
return a.checks
}
// Failures returns how many checks failed.
func (a *AbuseIPDB) Failures() int {
a.mu.Lock()
defer a.mu.Unlock()
return a.failures
}
// BudgetLeft returns how many checks the day's budget has left.
func (a *AbuseIPDB) BudgetLeft() int {
a.mu.Lock()
defer a.mu.Unlock()
if !a.day.Equal(a.params.Now().Truncate(day)) {
return a.params.DailyBudget
}
return max(a.params.DailyBudget-a.spent, 0)
}
// Snapshot returns the checks spent and every score still in use, sorted
// by client, as reputation.json keeps them.
func (a *AbuseIPDB) Snapshot() Checks {
a.mu.Lock()
now := a.params.Now()
checks := Checks{Day: a.day, Spent: a.spent, Scores: make([]Score, 0, a.scores.Len())}
for _, kept := range a.scores.Values() {
if now.Sub(kept.Fetched) < a.params.CacheTTL {
checks.Scores = append(checks.Scores, kept)
}
}
a.mu.Unlock()
slices.SortFunc(checks.Scores, func(x, y Score) int {
return x.Client.Compare(y.Client)
})
return checks
}
// Load keeps checks, read from reputation.json, in place of those it
// keeps, but for the scores past maxVerdicts, those fetched longest ago.
// One fetched CacheTTL ago or more is neither used nor written, as for any
// score.
func (a *AbuseIPDB) Load(checks Checks) {
scores := slices.Clone(checks.Scores)
slices.SortStableFunc(scores, func(x, y Score) int {
return x.Fetched.Compare(y.Fetched)
})
a.mu.Lock()
defer a.mu.Unlock()
a.day, a.spent = checks.Day, checks.Spent
a.scores.Purge()
for _, kept := range scores {
a.scores.Add(kept.Client, kept)
}
}
// check checks client with AbuseIPDB by addr, one of its addresses, keeps
// the score as client's, and notes the check as no longer under way. A
// check that fails gives no score: it is counted, logged and raised as a
// source_failure alert, and no client is checked for failureDelay.
func (a *AbuseIPDB) check(ctx context.Context, client netip.Prefix, addr netip.Addr) {
score, err := a.ask(ctx, addr)
now := a.params.Now()
a.mu.Lock()
delete(a.checking, client)
if err == nil {
a.scores.Add(client, Score{Client: client, Score: score, Fetched: now})
} else {
a.failures++
a.retryAt = now.Add(failureDelay)
}
a.mu.Unlock()
if err != nil {
a.alert("checking a client with AbuseIPDB failed", err)
}
}
// ask asks AbuseIPDB for addr's abuse confidence score, sending the key
// in the header Key. An answer other than 200, one that gives no score,
// and none within Timeout, fail.
func (a *AbuseIPDB) ask(ctx context.Context, addr netip.Addr) (int64, error) {
ctx, cancel := context.WithTimeout(ctx, a.params.Timeout)
defer cancel()
query := url.Values{"ipAddress": {addr.String()}}
req, err := http.NewRequestWithContext(ctx, http.MethodGet,
a.params.URL+"?"+query.Encode(), http.NoBody)
if err != nil {
return 0, fmt.Errorf("make the request: %w", err)
}
req.Header.Set("Key", a.params.Key)
req.Header.Set("Accept", "application/json")
res, err := a.httpClient.Do(req)
if err != nil {
// Do's error names the URL, which holds the client's address, which
// is not to be logged: only what went wrong is kept.
return 0, fmt.Errorf("check the client: %w", errors.Unwrap(err))
}
defer func() {
_ = res.Body.Close()
}()
if res.StatusCode != http.StatusOK {
return 0, fmt.Errorf("%w %s", errStatus, res.Status)
}
var answer struct {
Data struct {
AbuseConfidenceScore *int64 `json:"abuseConfidenceScore"`
} `json:"data"`
}
err = json.NewDecoder(io.LimitReader(res.Body, maxAnswerBytes)).Decode(&answer)
if err != nil {
return 0, fmt.Errorf("read the answer: %w", err)
}
if answer.Data.AbuseConfidenceScore == nil {
return 0, errNoScore
}
return *answer.Data.AbuseConfidenceScore, nil
}
// alert raises a source_failure alert from AbuseIPDB with reason and err,
// and logs them.
func (a *AbuseIPDB) alert(reason string, err error) {
// Raised before it is logged, so that the alert is there once the log
// line is.
raiseFailure(a.params.Alerts, reason, AbuseIPDBSource, err)
a.params.ProcessLog.Warn(reason, "source", AbuseIPDBSource, "error", err.Error())
}
internal/reputation/abuseipdb_test.go
-663
View File
@@ -1,663 +0,0 @@
package reputation_test
import (
"bytes"
"encoding/json"
"fmt"
"io"
"log/slog"
"net/http"
"net/http/httptest"
"net/netip"
"reflect"
"slices"
"strings"
"sync"
"testing"
"testing/synctest"
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/reputation"
)
// The tests of AbuseIPDB run in synctest bubbles, as those of the lists
// do, and AbuseIPDB is a stand-in reached without the network, for the
// same reason. A bubble's clock starts at midnight UTC, as a day the
// checks are counted in starts.
const (
// key is the account's key the tests give, the only one the stand-in
// takes.
key = "abuseipdb-key-0123456789abcdef"
// suspect and other are clients that have committed an offence.
suspect = "203.0.113.9"
other = "2001:db8::9"
)
func TestOnlyAnOffenderWithoutAScoreIsChecked(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
abuseIPDB := &abuseIPDBStandIn{scores: map[string]int64{suspect: 100}}
checker := newAbuseIPDB(abuseIPDB, abuseIPDBParams())
// A client that has committed no offence is not checked.
wantScore(t, checker, suspect, false, 0, false)
synctest.Wait()
wantChecked(t, abuseIPDB)
// An offender is, and from then on its score is used, whether or not
// it is an offender.
wantScore(t, checker, suspect, true, 0, false)
synctest.Wait()
wantScore(t, checker, suspect, true, 100, true)
wantScore(t, checker, suspect, false, 100, true)
synctest.Wait()
wantChecked(t, abuseIPDB, suspect)
})
}
func TestIPv6ClientIsCheckedOnceAndItsScoreUsedForEachOfItsAddresses(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
// 15 addresses of 2001:db8:1:2::/64, one client, each in a part of
// it of its own.
var addresses []string
for i := 1; i < 16; i++ {
addresses = append(addresses, fmt.Sprintf("2001:db8:1:2:%x::9", i<<12))
}
abuseIPDB := &abuseIPDBStandIn{scores: map[string]int64{addresses[0]: 100}}
checker := newAbuseIPDB(abuseIPDB, abuseIPDBParams())
// A request from each has the client checked once, by the first.
for _, address := range addresses {
hitFrom(t, checker, address, true)
}
synctest.Wait()
wantChecked(t, abuseIPDB, addresses[0])
// Its score is the whole client's.
for _, address := range addresses {
wantScore(t, checker, address, true, 100, true)
}
synctest.Wait()
wantChecked(t, abuseIPDB, addresses[0])
})
}
func TestScoreAtOrOverTheMinimumIsAHit(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
scores := map[string]int64{"192.0.2.74": 74, "192.0.2.75": 75, "192.0.2.100": 100}
p := abuseIPDBParams()
p.MinScore = 75
checker := newAbuseIPDB(&abuseIPDBStandIn{scores: scores}, p)
for client := range scores {
hitFrom(t, checker, client, true)
}
synctest.Wait()
for client, score := range scores {
wantScore(t, checker, client, true, score, score >= 75)
}
})
}
func TestScoreUsedUntilTheCacheTTLHasPassedSinceItWasFetched(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
abuseIPDB := &abuseIPDBStandIn{scores: map[string]int64{suspect: 100}}
checker := newAbuseIPDB(abuseIPDB, abuseIPDBParams())
hitFrom(t, checker, suspect, true)
synctest.Wait()
// AbuseIPDB gives another score from now on, but the one kept is
// used, and the client is not checked again, until the TTL has
// passed.
abuseIPDB.setScore(suspect, 80)
time.Sleep(cacheTTL - time.Nanosecond)
wantScore(t, checker, suspect, true, 100, true)
synctest.Wait()
wantChecked(t, abuseIPDB, suspect)
// Then it is not used, and the client is checked again.
time.Sleep(time.Nanosecond)
wantScore(t, checker, suspect, true, 0, false)
synctest.Wait()
wantScore(t, checker, suspect, true, 80, true)
wantChecked(t, abuseIPDB, suspect, suspect)
})
}
func TestDailyBudgetKeptAcrossARestartAndWholeAgainAsTheDayEnds(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
var log bytes.Buffer
queue := newQueue()
p := abuseIPDBParams()
p.DailyBudget = 3
p.Alerts = queue
p.ProcessLog = slog.New(slog.NewJSONHandler(&log, nil))
abuseIPDB := &abuseIPDBStandIn{scores: map[string]int64{suspect: 100}}
checker := newAbuseIPDB(abuseIPDB, p)
// At noon, the first three offenders spend the budget, and the
// fourth, unchecked, is not.
time.Sleep(12 * time.Hour)
const unchecked = "192.0.2.4"
clients := []string{suspect, "192.0.2.2", "192.0.2.3", unchecked}
for _, client := range clients {
hitFrom(t, checker, client, true)
}
synctest.Wait()
wantChecked(t, abuseIPDB, clients[:3]...)
wantBudgetLeft(t, checker, 0)
// The check that used the budget up raised the alert, and logged it.
const usedUp = "the daily budget of AbuseIPDB checks is used up"
wantFailureAlert(t, queue, time.Now(), usedUp,
"3 checks spent; none is made until the day ends at 00:00 UTC", 0)
if !strings.Contains(log.String(), `"msg":"`+usedUp+`"`) {
t.Errorf("logged\n%s\nwant the budget used up", log.String())
}
// Restarted with what reputation.json keeps, it uses the scores, and
// checks no client until the day ends.
restarted := &abuseIPDBStandIn{}
again := newAbuseIPDB(restarted, p)
again.Load(checker.Snapshot())
wantScore(t, again, suspect, true, 100, true)
wantBudgetLeft(t, again, 0)
time.Sleep(12*time.Hour - time.Nanosecond)
wantScore(t, again, unchecked, true, 0, false)
synctest.Wait()
wantChecked(t, restarted)
// At midnight the budget is whole again.
time.Sleep(time.Nanosecond)
wantBudgetLeft(t, again, 3)
wantScore(t, again, unchecked, true, 0, false)
synctest.Wait()
wantChecked(t, restarted, unchecked)
wantBudgetLeft(t, again, 2)
})
}
func TestFailedCheckGivesNoScoreAndNoClientIsCheckedForAMinute(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
// key is the key sent, status and body what AbuseIPDB answers with,
// and error the failure.
key, body string
status int
error string
}{
{
"a refusal, past AbuseIPDB's own limit", key,
`{"errors":[{"detail":"Daily rate limit of 1000 requests exceeded"}]}`,
http.StatusTooManyRequests, "the server answered 429 Too Many Requests",
},
{
"a refusal of a wrong key", "wrong-key-0123456789abcdef", "", 0,
"the server answered 401 Unauthorized",
},
{
"a server failure", key, "", http.StatusInternalServerError,
"the server answered 500 Internal Server Error",
},
{
"an answer without a score", key, `{"data":{"ipAddress":"` + suspect + `"}}`,
http.StatusOK, "the answer gives no abuseConfidenceScore",
},
{
"an answer that is not JSON", key, "<html>", http.StatusOK,
"read the answer: invalid character '<' looking for beginning of value",
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
var log bytes.Buffer
queue := newQueue()
p := abuseIPDBParams()
p.Key = tc.key
p.Alerts = queue
p.ProcessLog = slog.New(slog.NewJSONHandler(&log, nil))
abuseIPDB := &abuseIPDBStandIn{status: tc.status, body: tc.body}
checker := newAbuseIPDB(abuseIPDB, p)
// The failure gives no score, and no client is checked within a
// minute of it.
wantScore(t, checker, suspect, true, 0, false)
synctest.Wait()
time.Sleep(time.Minute - time.Nanosecond)
wantScore(t, checker, other, true, 0, false)
synctest.Wait()
wantChecked(t, abuseIPDB, suspect)
wantFailures(t, checker, 1)
time.Sleep(time.Nanosecond)
wantScore(t, checker, other, true, 0, false)
synctest.Wait()
wantChecked(t, abuseIPDB, suspect, other)
wantFailures(t, checker, 2)
if scores := checker.Snapshot().Scores; len(scores) != 0 {
t.Errorf("scores %+v, want none", scores)
}
// One alert for the first failure; the cooldown holds back the
// second.
wantFailureAlert(t, queue, time.Now().Add(-time.Minute),
"checking a client with AbuseIPDB failed", tc.error, 1)
if !strings.Contains(log.String(), `"msg":"checking a client with `+
`AbuseIPDB failed","source":"abuseipdb","error":"`+tc.error) {
t.Errorf("logged\n%s\nwant the failures", log.String())
}
})
})
}
}
func TestCheckNotAnsweredWithinTheTimeoutFailsAndHitNeverWaits(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
queue := newQueue()
p := abuseIPDBParams()
p.Alerts = queue
abuseIPDB := &abuseIPDBStandIn{hanging: true}
checker := newAbuseIPDB(abuseIPDB, p)
began := time.Now()
// The second, while the first's check is under way, starts none.
wantScore(t, checker, suspect, true, 0, false)
wantScore(t, checker, suspect, true, 0, false)
if waited := time.Since(began); waited != 0 {
t.Errorf("waited %s for the check, want no wait", waited)
}
time.Sleep(timeout - time.Nanosecond)
synctest.Wait()
wantChecked(t, abuseIPDB, suspect)
wantFailures(t, checker, 0)
time.Sleep(time.Nanosecond)
synctest.Wait()
wantFailures(t, checker, 1)
wantFailureAlert(t, queue, time.Now(), "checking a client with AbuseIPDB failed",
"check the client: context deadline exceeded", 0)
})
}
func TestKeyIsSentInTheKeyHeaderAndNeverShown(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
var log bytes.Buffer
queue := newQueue()
p := abuseIPDBParams()
p.Alerts = queue
p.ProcessLog = slog.New(slog.NewJSONHandler(&log, nil))
abuseIPDB := &abuseIPDBStandIn{scores: map[string]int64{suspect: 100}}
checker := newAbuseIPDB(abuseIPDB, p)
m := metrics.New(1, "app")
m.AddReputation(reputation.New(params()), reputation.NewDNSBL(dnsblParams()))
m.AddAbuseIPDB(checker)
// One check that AbuseIPDB answers, and one it refuses with an answer
// that names the key.
wantScore(t, checker, suspect, true, 0, false)
synctest.Wait()
wantScore(t, checker, suspect, true, 100, true)
abuseIPDB.answerWith(http.StatusUnauthorized, `{"errors":[{"detail":"`+key+`"}]}`)
wantScore(t, checker, other, true, 0, false)
synctest.Wait()
wantFailures(t, checker, 1)
abuseIPDB.mu.Lock()
sent := slices.Clone(abuseIPDB.keys)
abuseIPDB.mu.Unlock()
if !slices.Equal(sent, []string{key, key}) {
t.Errorf("checks sent the keys %v, want %s twice", sent, key)
}
alerted, err := json.Marshal(waiting(queue))
if err != nil {
t.Fatalf("encode the alerts: %v", err)
}
kept, err := json.Marshal(checker.Snapshot())
if err != nil {
t.Fatalf("encode the checks: %v", err)
}
for name, shown := range map[string]string{
"the log": log.String(), "the alerts": string(alerted),
"the metrics": scrapeMetrics(t, m), "reputation.json": string(kept),
} {
if strings.Contains(shown, key) {
t.Errorf("%s shows the key:\n%s", name, shown)
}
}
})
}
func TestMetricsCountTheChecksTheFailuresAndTheBudgetLeft(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
abuseIPDB := &abuseIPDBStandIn{}
p := abuseIPDBParams()
p.DailyBudget = 5
checker := newAbuseIPDB(abuseIPDB, p)
m := metrics.New(1, "app")
m.AddReputation(reputation.New(params()), reputation.NewDNSBL(dnsblParams()))
m.AddAbuseIPDB(checker)
// One check that AbuseIPDB answers, and one that fails.
hitFrom(t, checker, suspect, true)
synctest.Wait()
abuseIPDB.answerWith(http.StatusInternalServerError, "")
hitFrom(t, checker, other, true)
synctest.Wait()
scraped := scrapeMetrics(t, m)
for series, want := range map[string]string{
"queries_total": "2",
"failures_total": "1",
"daily_budget_remaining": "3",
} {
line := "\nsmallwebwaf_reputation_" + series +
`{instance="app",source="abuseipdb"} ` + want + "\n"
if !strings.Contains(scraped, line) {
t.Errorf("metrics\n%s\nwant%s", scraped, line)
}
}
})
}
func TestScoreFetchedATTLAgoIsNeitherUsedNorKept(t *testing.T) {
t.Parallel()
now := time.Date(2026, 10, 7, 0, 0, 0, 0, time.UTC)
p := abuseIPDBParams()
p.Now = func() time.Time { return now }
checker := reputation.NewAbuseIPDB(p)
// The last score still in use, and one, of other's /64, fetched a TTL
// ago.
inUse := reputation.Score{
Client: netip.MustParsePrefix(suspect + "/32"), Score: 100,
Fetched: now.Add(-cacheTTL + time.Nanosecond),
}
stale := reputation.Score{
Client: netip.MustParsePrefix("2001:db8::/64"), Score: 100,
Fetched: now.Add(-cacheTTL),
}
checker.Load(reputation.Checks{Scores: []reputation.Score{stale, inUse}})
wantScore(t, checker, suspect, false, 100, true)
wantScore(t, checker, other, false, 0, false)
got := checker.Snapshot().Scores
if !reflect.DeepEqual(got, []reputation.Score{inUse}) {
t.Errorf("scores %+v, want only %+v", got, inUse)
}
}
func TestAtMost100000ScoresKeptTheOneFetchedLongestAgoDroppedFirst(t *testing.T) {
t.Parallel()
now := time.Date(2026, 10, 7, 0, 0, 0, 0, time.UTC)
p := abuseIPDBParams()
p.Now = func() time.Time { return now }
checker := reputation.NewAbuseIPDB(p)
// 100,001 scores, listed by client, as reputation.json lists them, each
// fetched a millisecond before the one before it: the last is one too
// many.
const count = 100001
scores := make([]reputation.Score, 0, count)
addr := netip.MustParseAddr("198.18.0.0")
for i := range count {
scores = append(scores, reputation.Score{
Client: netip.PrefixFrom(addr, 32),
Fetched: now.Add(-time.Duration(i) * time.Millisecond),
})
addr = addr.Next()
}
checker.Load(reputation.Checks{Scores: scores})
got := checker.Snapshot().Scores
if len(got) != count-1 || !slices.Contains(got, scores[0]) ||
slices.Contains(got, scores[count-1]) {
t.Errorf("%d scores kept, want all but the one fetched longest ago", len(got))
}
}
// abuseIPDBStandIn is a stand-in for AbuseIPDB. It answers a check sent
// with key by the client's score, as scores gives it, 0 for a client it
// does not give; a check sent with another key with 401; and, while
// status is not 0, every check with status and body; and while hanging,
// none at all. It notes each client checked, and the key sent.
type abuseIPDBStandIn struct {
mu sync.Mutex
scores map[string]int64
status int
body string
hanging bool
checked []string
keys []string
}
// RoundTrip has the stand-in answer req, in place of the network. A check
// abandoned before the stand-in answers fails, as over the network.
func (s *abuseIPDBStandIn) RoundTrip(req *http.Request) (*http.Response, error) {
client := req.URL.Query().Get("ipAddress")
sent := req.Header.Get("Key")
s.mu.Lock()
s.checked = append(s.checked, client)
s.keys = append(s.keys, sent)
score := s.scores[client]
status, body, hanging := s.status, s.body, s.hanging
s.mu.Unlock()
switch {
case hanging:
<-req.Context().Done()
return nil, req.Context().Err()
case sent != key:
status = http.StatusUnauthorized
case status == 0:
status = http.StatusOK
body = fmt.Sprintf(`{"data":{"ipAddress":%q,"abuseConfidenceScore":%d}}`, client,
score)
}
return &http.Response{
StatusCode: status,
Status: fmt.Sprintf("%d %s", status, http.StatusText(status)),
Header: http.Header{},
Body: io.NopCloser(strings.NewReader(body)),
Request: req,
}, nil
}
// setScore has the stand-in give client score.
func (s *abuseIPDBStandIn) setScore(client string, score int64) {
s.mu.Lock()
defer s.mu.Unlock()
s.scores[client] = score
}
// answerWith has the stand-in answer every check with status and body.
func (s *abuseIPDBStandIn) answerWith(status int, body string) {
s.mu.Lock()
defer s.mu.Unlock()
s.status, s.body = status, body
}
// abuseIPDBParams returns the AbuseIPDBParams of the tests: key, a minimum
// score of 75, a daily budget of 900, and the cache TTL and timeout of the
// DNSBL tests, by the bubble's clock, with alerts to a queue that sends
// none.
func abuseIPDBParams() reputation.AbuseIPDBParams {
return reputation.AbuseIPDBParams{
URL: "https://abuseipdb.example/api/v2/check",
Key: key,
MinScore: 75,
DailyBudget: 900,
CacheTTL: cacheTTL,
Timeout: timeout,
Now: time.Now,
ProcessLog: slog.New(slog.DiscardHandler),
Alerts: newQueue(),
}
}
// newAbuseIPDB returns the AbuseIPDB of p, checking clients with
// abuseIPDB.
func newAbuseIPDB(
abuseIPDB *abuseIPDBStandIn, p reputation.AbuseIPDBParams,
) *reputation.AbuseIPDB {
checker := reputation.NewAbuseIPDB(p)
checker.SetTransport(abuseIPDB)
return checker
}
// wantScore checks the score checker gives client, and whether it is a
// hit, as a request from client finds them, offender or not.
func wantScore(
t *testing.T, checker *reputation.AbuseIPDB, client string, offender bool,
score int64, hit bool,
) {
t.Helper()
gotScore, gotHit := hitFrom(t, checker, client, offender)
if gotScore != score || gotHit != hit {
t.Errorf("%s has the score %d, a hit %t, want %d, %t", client, gotScore, gotHit,
score, hit)
}
}
// hitFrom is checker's Hit for a request from address, offender or not.
// Its client is address for an IPv4 address, and its /64 for an IPv6 one,
// as smallwebwaf counts clients.
func hitFrom(
t *testing.T, checker *reputation.AbuseIPDB, address string, offender bool,
) (int64, bool) {
t.Helper()
addr := netip.MustParseAddr(address)
client := netip.PrefixFrom(addr, addr.BitLen())
if addr.Is6() {
client = netip.PrefixFrom(addr, 64).Masked()
}
return checker.Hit(t.Context(), client, addr, offender)
}
// wantChecked checks the clients the stand-in was asked about, in any
// order.
func wantChecked(t *testing.T, abuseIPDB *abuseIPDBStandIn, want ...string) {
t.Helper()
abuseIPDB.mu.Lock()
got := slices.Sorted(slices.Values(abuseIPDB.checked))
abuseIPDB.mu.Unlock()
want = slices.Sorted(slices.Values(want))
if !slices.Equal(got, want) {
t.Errorf("checked %v, want %v", got, want)
}
}
// wantFailures checks how many checks failed.
func wantFailures(t *testing.T, checker *reputation.AbuseIPDB, want int) {
t.Helper()
if got := checker.Failures(); got != want {
t.Errorf("%d checks failed, want %d", got, want)
}
}
// wantFailureAlert checks that the one alert waiting in queue is a
// source_failure alert from AbuseIPDB, raised at raised, with reason and
// the error failure, and that the cooldown has held back held repeats of
// it.
func wantFailureAlert(
t *testing.T, queue *alerts.Queue, raised time.Time, reason, failure string,
held int64,
) {
t.Helper()
got := waiting(queue)
if len(got) != 1 || !got[0].Time.Equal(raised) ||
got[0].Event != alerts.EventSourceFailure || got[0].Reason != reason ||
got[0].Detail["source"] != reputation.AbuseIPDBSource ||
got[0].Detail["error"] != failure || queue.Suppressed() != held {
t.Errorf("alerts waiting %+v, %d held back, want only AbuseIPDB's %q with %q, "+
"and %d", got, queue.Suppressed(), reason, failure, held)
}
}
// wantBudgetLeft checks how many checks the day's budget has left.
func wantBudgetLeft(t *testing.T, checker *reputation.AbuseIPDB, want int) {
t.Helper()
if got := checker.BudgetLeft(); got != want {
t.Errorf("%d checks left, want %d", got, want)
}
}
// scrapeMetrics returns the metrics m serves.
func scrapeMetrics(t *testing.T, m *metrics.Metrics) string {
t.Helper()
scraped := httptest.NewRecorder()
m.ServeHTTP(scraped, httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/",
http.NoBody))
return scraped.Body.String()
}
internal/reputation/dnsbl.go
-332
View File
@@ -1,332 +0,0 @@
package reputation
import (
"cmp"
"context"
"encoding/hex"
"errors"
"fmt"
"log/slog"
"net"
"net/netip"
"slices"
"strings"
"sync"
"time"
"github.com/hashicorp/golang-lru/v2/simplelru"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/config"
)
const (
// maxVerdicts is how many verdicts of the DNSBL zones are kept, and how
// many scores of AbuseIPDB. Past it, the one fetched longest ago is
// dropped.
maxVerdicts = 100000
// maxQueries is how many queries may be under way at once. Past it, a
// zone is not asked about a client until the client's next request, so
// that a swarm of new addresses cannot fill the memory.
maxQueries = 1000
// failureDelay is how long a zone is not asked again after a query to
// it fails, and no client is checked with AbuseIPDB after a check
// fails, so that a source refusing them is not asked on every request.
failureDelay = time.Minute
)
var (
errAsk = errors.New("ask the zone")
errRefused = errors.New("the zone refused the query")
errNotListing = errors.New("the answer is outside 127.0.0.0/8")
)
// Verdict is what a zone said about a client, as reputation.json holds
// it: the zone, the client's address, whether the zone lists it, and when
// the zone answered.
type Verdict struct {
Zone string `json:"zone"`
Client netip.Addr `json:"client"`
Listed bool `json:"listed"`
Fetched time.Time `json:"fetched"`
}
// DNSBLParams are what NewDNSBL needs.
type DNSBLParams struct {
// Zones are the DNSBL zones clients are asked about in
// (SWWAF_DNSBL_ZONES).
Zones []string
// Resolver is the resolver they are asked through
// (SWWAF_DNSBL_RESOLVER), or, while it is the zero AddrPort, the
// host's, as /etc/resolv.conf names it.
Resolver netip.AddrPort
// CacheTTL is how long a verdict is used after it was fetched
// (SWWAF_REPUTATION_CACHE_TTL), and Timeout how long a query may take
// (SWWAF_REPUTATION_TIMEOUT).
CacheTTL time.Duration
Timeout time.Duration
// Now tells the time, normally time.Now in UTC.
Now func() time.Time
// ProcessLog receives each query that fails, and why.
ProcessLog *slog.Logger
// Alerts receive a source_failure alert for each query that fails.
Alerts *alerts.Queue
}
// DNSBL asks the DNSBL zones about clients, in the background, and keeps
// their verdicts. It is safe for concurrent use.
type DNSBL struct {
params DNSBLParams
resolver *net.Resolver
mu sync.Mutex
// verdicts are by query. Each is added as it is fetched and never moved
// up, so that the one fetched longest ago is the first dropped.
verdicts *simplelru.LRU[query, Verdict]
// asking are the queries under way.
asking map[query]bool
// queries and failures count, by zone, the queries made and those that
// failed, and retryAt is when a zone whose last query failed may be
// asked again.
queries map[string]int
failures map[string]int
retryAt map[string]time.Time
}
// query is a client's address, to ask a zone about.
type query struct {
zone string
client netip.Addr
}
// NewDNSBL returns a DNSBL with no verdict yet.
func NewDNSBL(params DNSBLParams) *DNSBL {
verdicts, err := simplelru.NewLRU[query, Verdict](maxVerdicts, nil)
if err != nil {
panic(err) // NewLRU fails only for a size below one
}
resolver := &net.Resolver{}
if params.Resolver.IsValid() {
// Dial is used by Go's own resolver alone.
resolver.PreferGo = true
resolver.Dial = func(ctx context.Context, network, _ string) (net.Conn, error) {
var dialer net.Dialer
return dialer.DialContext(ctx, network, params.Resolver.String())
}
}
return &DNSBL{
params: params,
resolver: resolver,
verdicts: verdicts,
asking: map[query]bool{},
queries: map[string]int{},
failures: map[string]int{},
retryAt: map[string]time.Time{},
}
}
// Zones returns the zones, in the order SWWAF_DNSBL_ZONES names them.
func (d *DNSBL) Zones() []string {
return slices.Clone(d.params.Zones)
}
// ListedBy returns the zones whose verdict on addr, a client's address,
// lists it, in the order SWWAF_DNSBL_ZONES names them, each with its key
// masked, as config.MaskZoneKey masks it, since they go to the request
// log, the alerts and the metrics. A verdict is used until CacheTTL has
// passed since it was fetched. Each zone without one is asked about addr
// in the background, unless a query about addr to it is under way, the
// zone is left alone after a failure, or maxQueries are under way;
// ListedBy never waits for a query. ctx is the context of the client's
// request, and a query goes on after the request ends.
func (d *DNSBL) ListedBy(ctx context.Context, addr netip.Addr) []string {
d.mu.Lock()
defer d.mu.Unlock()
now := d.params.Now()
var listedBy []string
for _, zone := range d.params.Zones {
q := query{zone: zone, client: addr}
kept, found := d.verdicts.Peek(q)
switch {
case found && now.Sub(kept.Fetched) < d.params.CacheTTL:
if kept.Listed {
listedBy = append(listedBy, config.MaskZoneKey(zone))
}
case !d.asking[q] && !now.Before(d.retryAt[zone]) && len(d.asking) < maxQueries:
d.asking[q] = true
d.queries[zone]++
go d.ask(context.WithoutCancel(ctx), q)
}
}
return listedBy
}
// Queries returns how many queries were made to zone.
func (d *DNSBL) Queries(zone string) int {
d.mu.Lock()
defer d.mu.Unlock()
return d.queries[zone]
}
// Failures returns how many queries to zone failed.
func (d *DNSBL) Failures(zone string) int {
d.mu.Lock()
defer d.mu.Unlock()
return d.failures[zone]
}
// Snapshot returns every verdict still in use, sorted by client, then by
// zone, as reputation.json lists them.
func (d *DNSBL) Snapshot() []Verdict {
d.mu.Lock()
now := d.params.Now()
verdicts := make([]Verdict, 0, d.verdicts.Len())
for _, kept := range d.verdicts.Values() {
if now.Sub(kept.Fetched) < d.params.CacheTTL {
verdicts = append(verdicts, kept)
}
}
d.mu.Unlock()
slices.SortFunc(verdicts, func(a, b Verdict) int {
return cmp.Or(a.Client.Compare(b.Client), strings.Compare(a.Zone, b.Zone))
})
return verdicts
}
// Load keeps verdicts, read from reputation.json, in place of those it
// keeps, but for those of a zone SWWAF_DNSBL_ZONES does not name, and,
// past maxVerdicts, those fetched longest ago. One fetched CacheTTL ago or
// more is neither used nor written, as for any verdict.
func (d *DNSBL) Load(verdicts []Verdict) {
verdicts = slices.Clone(verdicts)
slices.SortStableFunc(verdicts, func(a, b Verdict) int {
return a.Fetched.Compare(b.Fetched)
})
d.mu.Lock()
defer d.mu.Unlock()
d.verdicts.Purge()
for _, kept := range verdicts {
if slices.Contains(d.params.Zones, kept.Zone) {
d.verdicts.Add(query{zone: kept.Zone, client: kept.Client}, kept)
}
}
}
// ask asks q's zone about q's client, keeps the verdict, and notes the
// query as no longer under way. A query that fails gives no verdict: it
// is counted, logged and raised as a source_failure alert, which show the
// zone with its key masked, and the zone is not asked again for
// failureDelay.
func (d *DNSBL) ask(ctx context.Context, q query) {
listed, err := d.lookUp(ctx, q)
now := d.params.Now()
d.mu.Lock()
delete(d.asking, q)
if err == nil {
d.verdicts.Add(q, Verdict{
Zone: q.zone, Client: q.client, Listed: listed, Fetched: now,
})
} else {
d.failures[q.zone]++
d.retryAt[q.zone] = now.Add(failureDelay)
}
d.mu.Unlock()
if err != nil {
const failed = "asking a DNSBL zone failed"
shown := config.MaskZoneKey(q.zone)
// Raised before it is logged, so that the alert is there once the
// log line is.
raiseFailure(d.params.Alerts, failed, shown, err)
d.params.ProcessLog.Warn(failed, "zone", shown, "error", err.Error())
}
}
// lookUp asks q's zone about q's client through the resolver, and returns
// whether the zone lists it, as readAnswer reads the answer. No such name
// is a client the zone does not list. A query not answered within Timeout
// fails.
func (d *DNSBL) lookUp(ctx context.Context, q query) (bool, error) {
ctx, cancel := context.WithTimeout(ctx, d.params.Timeout)
defer cancel()
answer, err := d.resolver.LookupNetIP(ctx, "ip4", queryName(q.zone, q.client))
var dnsErr *net.DNSError
switch {
case err == nil:
return readAnswer(answer)
case errors.As(err, &dnsErr) && dnsErr.IsNotFound:
return false, nil
case errors.As(err, &dnsErr):
// The error names the name asked about, which holds the client's
// address, which is not to be logged: only what went wrong is kept.
return false, fmt.Errorf("%w: %s", errAsk, dnsErr.Err)
default:
return false, fmt.Errorf("%w: %w", errAsk, err)
}
}
// queryName returns the name a zone is asked about addr by, as RFC 5782
// builds it: the four numbers of an IPv4 address, or the 32 hex digits of
// an IPv6 address, in reverse order, each followed by a dot, then the zone
// and a dot, which makes it a full name, to which the resolver adds no
// search domain of /etc/resolv.conf.
func queryName(zone string, addr netip.Addr) string {
parts := strings.Split(addr.String(), ".")
if addr.Is6() {
parts = strings.Split(hex.EncodeToString(addr.AsSlice()), "")
}
slices.Reverse(parts)
return strings.Join(parts, ".") + "." + zone + "."
}
// readAnswer reads the addresses a zone answered with. An address in
// 127.0.0.0/8 lists the client, as RFC 5782 has zones answer, but one in
// 127.255.255.0/24 is how Spamhaus refuses a query, such as one sent
// through a public resolver or one past its limit, and is a failure. So is
// an address outside 127.0.0.0/8, such as a resolver gives that answers
// even for names that do not exist.
func readAnswer(answer []netip.Addr) (bool, error) {
listing := netip.MustParsePrefix("127.0.0.0/8")
refusal := netip.MustParsePrefix("127.255.255.0/24")
for _, addr := range answer {
switch {
case refusal.Contains(addr):
return false, fmt.Errorf("%w: %s", errRefused, addr)
case !listing.Contains(addr):
return false, fmt.Errorf("%w: %s", errNotListing, addr)
}
}
return len(answer) > 0, nil
}
internal/reputation/dnsbl_test.go
-737
View File
@@ -1,737 +0,0 @@
package reputation_test
import (
"bytes"
"context"
"encoding/binary"
"errors"
"io"
"log/slog"
"net"
"net/http"
"net/http/httptest"
"net/netip"
"reflect"
"slices"
"strings"
"sync"
"testing"
"testing/synctest"
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/reputation"
)
// The tests of the DNSBL zones run in synctest bubbles, as those of the
// lists do, and the resolver the zones are asked through is a stand-in
// reached through an in-memory connection, net.Pipe's, for the same
// reason. They run one at a time, none in parallel with another test of
// this package: Go's resolver counts the queries under way in one
// sync.WaitGroup for the whole process, and the process fails when
// queries from two bubbles, or from a bubble and from outside one, are
// under way at once. TestMain has the resolver make its configuration,
// which it makes on its first query, outside every bubble, since the
// configuration holds a channel, which the bubble it was made in would
// keep to itself.
const (
// zone and otherZone are the DNSBL zones the tests name.
zone = "dnsbl.example"
otherZone = "other.example"
// cacheTTL is the tests' SWWAF_REPUTATION_CACHE_TTL, and timeout their
// SWWAF_REPUTATION_TIMEOUT: a second, the least time /etc/resolv.conf
// can have Go's resolver wait for one server, so that it is the
// DNSBL's own timeout that ends a query, whatever that file says.
cacheTTL = 24 * time.Hour
timeout = time.Second
// listed and unlisted are clients zone is asked about by the names
// listedName and unlistedName, and most tests have zone list the first
// alone, by answering with listing.
listed = "192.0.2.99"
unlisted = "192.0.2.100"
listedName = "99.2.0.192." + zone + "."
unlistedName = "100.2.0.192." + zone + "."
listing = "127.0.0.2"
)
// The DNS response codes the stand-in answers with, besides no error.
const (
serverFailure = 2
noSuchName = 3
refused = 5
)
var errNoNetwork = errors.New("the test dials nothing")
func TestMain(m *testing.M) {
// A query that fails at once, as nothing is dialled for it.
resolver := &net.Resolver{
PreferGo: true,
Dial: func(context.Context, string, string) (net.Conn, error) {
return nil, errNoNetwork
},
}
_, _ = resolver.LookupNetIP(context.Background(), "ip4", "warm-up.invalid.")
m.Run()
}
//nolint:paralleltest // one at a time, as the comment at the top of this file says
func TestZonesListOrNotClientsByTheirIPv4AndIPv6Addresses(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
// The addresses of the examples of RFC 5782, and the names it
// gives for them.
const (
v4 = "192.0.2.99"
v6 = "2001:db8:1:2:3:4:567:89ab"
// v6Name is the hex digits of v6, in reverse order.
v6Name = "b.a.9.8.7.6.5.0.4.0.0.0.3.0.0.0.2.0.0.0.1.0.0.0.8.b.d.0.1.0.0.2."
)
resolver := &resolverStandIn{answers: map[string]answer{
"99.2.0.192." + zone + ".": {addrs: []string{listing}},
v6Name + otherZone + ".": {addrs: []string{"127.0.0.4", "127.0.0.10"}},
"99.2.0.192." + otherZone + ".": {},
}}
dnsbl := newDNSBL(resolver, dnsblParams(zone, otherZone))
// Neither client has a verdict yet, so neither is listed, and each
// zone is asked about each.
wantZones(t, dnsbl, v4)
wantZones(t, dnsbl, v6)
synctest.Wait()
wantZones(t, dnsbl, v4, zone)
wantZones(t, dnsbl, v6, otherZone)
wantAsked(t, resolver,
"99.2.0.192."+zone+".", "99.2.0.192."+otherZone+".",
v6Name+zone+".", v6Name+otherZone+".")
})
}
//nolint:paralleltest // one at a time, as the comment at the top of this file says
func TestListedByNeverWaitsForAQuery(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
dnsbl := newDNSBL(&resolverStandIn{hanging: true}, dnsblParams(zone))
began := time.Now()
// The second, while the first's query is under way, starts none.
wantZones(t, dnsbl, listed)
wantZones(t, dnsbl, listed)
if waited := time.Since(began); waited != 0 {
t.Errorf("waited %s for the query, want no wait", waited)
}
synctest.Wait()
wantQueries(t, dnsbl, 1, 0)
waitForTheResolver()
})
}
//nolint:paralleltest // one at a time, as the comment at the top of this file says
func TestVerdictUsedUntilTheCacheTTLHasPassedSinceItWasFetched(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
resolver := &resolverStandIn{answers: map[string]answer{
listedName: {addrs: []string{listing}},
}}
dnsbl := newDNSBL(resolver, dnsblParams(zone))
wantZones(t, dnsbl, listed)
wantZones(t, dnsbl, unlisted)
synctest.Wait()
// The zone lists the other client from now on, but the verdicts
// kept are used, and the zone is not asked again, until the TTL
// has passed.
resolver.set(listedName, answer{rcode: noSuchName})
resolver.set(unlistedName, answer{addrs: []string{listing}})
time.Sleep(cacheTTL - time.Nanosecond)
wantZones(t, dnsbl, listed, zone)
wantZones(t, dnsbl, unlisted)
synctest.Wait()
wantQueries(t, dnsbl, 2, 0)
// Then neither verdict is used, and both clients are asked about
// again.
time.Sleep(time.Nanosecond)
wantZones(t, dnsbl, listed)
wantZones(t, dnsbl, unlisted)
synctest.Wait()
wantQueries(t, dnsbl, 4, 0)
wantZones(t, dnsbl, listed)
wantZones(t, dnsbl, unlisted, zone)
})
}
//nolint:paralleltest // one at a time, as the comment at the top of this file says
func TestQueryNotAnsweredWithinTheTimeoutFails(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
queue := newQueue()
p := dnsblParams(zone)
p.Alerts = queue
dnsbl := newDNSBL(&resolverStandIn{hanging: true}, p)
wantZones(t, dnsbl, listed)
time.Sleep(timeout - time.Nanosecond)
synctest.Wait()
wantQueries(t, dnsbl, 1, 0)
time.Sleep(time.Nanosecond)
synctest.Wait()
wantQueries(t, dnsbl, 1, 1)
if got := waiting(queue); len(got) != 1 ||
got[0].Detail["error"] != "ask the zone: i/o timeout" {
t.Errorf("alerts waiting %+v, want the timeout's", got)
}
if verdicts := dnsbl.Snapshot(); len(verdicts) != 0 {
t.Errorf("verdicts %+v, want none", verdicts)
}
waitForTheResolver()
})
}
//nolint:paralleltest // one at a time, as the comment at the top of this file says
func TestZoneThatFailsOrRefusesGivesNoVerdictAndIsLeftAloneForAMinute(t *testing.T) {
for _, tc := range []struct {
name string
answer answer
error string
}{
{
"a server failure", answer{rcode: serverFailure},
"ask the zone: server misbehaving",
},
{"a refusal", answer{rcode: refused}, "ask the zone: server misbehaving"},
{
"an answer in 127.255.255.0/24, with which Spamhaus refuses a query",
answer{addrs: []string{"127.255.255.254"}},
"the zone refused the query: 127.255.255.254",
},
{
"an answer outside 127.0.0.0/8, as for a name that does not exist",
answer{addrs: []string{"192.0.2.1"}},
"the answer is outside 127.0.0.0/8: 192.0.2.1",
},
} {
//nolint:paralleltest // one at a time, as the comment at the top of this file says
t.Run(tc.name, func(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
var log bytes.Buffer
queue := newQueue()
p := dnsblParams(zone)
p.Alerts = queue
p.ProcessLog = slog.New(slog.NewJSONHandler(&log, nil))
dnsbl := newDNSBL(&resolverStandIn{answers: map[string]answer{
listedName: tc.answer,
}}, p)
// The failure gives no verdict, and the zone is not asked
// again within a minute of it.
wantZones(t, dnsbl, listed)
synctest.Wait()
time.Sleep(time.Minute - time.Nanosecond)
wantZones(t, dnsbl, listed)
synctest.Wait()
wantQueries(t, dnsbl, 1, 1)
time.Sleep(time.Nanosecond)
wantZones(t, dnsbl, listed)
synctest.Wait()
wantQueries(t, dnsbl, 2, 2)
if verdicts := dnsbl.Snapshot(); len(verdicts) != 0 {
t.Errorf("verdicts %+v, want none", verdicts)
}
// One alert for the first failure; the cooldown holds back
// the second.
wantAlert(t, queue, alerts.Alert{
Time: time.Now().Add(-time.Minute),
Event: alerts.EventSourceFailure,
Reason: "asking a DNSBL zone failed",
Detail: map[string]any{"source": zone, "error": tc.error},
})
if !strings.Contains(log.String(), `"msg":"asking a DNSBL zone failed",`+
`"zone":"`+zone+`","error":"`+tc.error) {
t.Errorf("logged\n%s\nwant the failures", log.String())
}
})
})
}
}
//nolint:paralleltest // one at a time, as the comment at the top of this file says
func TestAtMost1000QueriesUnderWay(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
dnsbl := newDNSBL(&resolverStandIn{hanging: true}, dnsblParams(zone))
client := netip.MustParseAddr("198.18.0.0")
for range 1001 {
dnsbl.ListedBy(t.Context(), client)
client = client.Next()
}
synctest.Wait()
wantQueries(t, dnsbl, 1000, 0)
waitForTheResolver()
})
}
//nolint:paralleltest // one at a time, as the comment at the top of this file says
func TestMetricsCountEachZonesQueriesAndThoseThatFailed(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
resolver := &resolverStandIn{answers: map[string]answer{
listedName: {addrs: []string{listing}},
"99.2.0.192." + otherZone + ".": {rcode: serverFailure},
}}
dnsbl := newDNSBL(resolver, dnsblParams(zone, otherZone))
m := metrics.New(1, "app")
m.AddReputation(reputation.New(params()), dnsbl)
wantZones(t, dnsbl, listed)
synctest.Wait()
scraped := httptest.NewRecorder()
m.ServeHTTP(scraped, httptest.NewRequestWithContext(t.Context(), http.MethodGet,
"/", http.NoBody))
for series, want := range map[string]string{
"queries_total" + `{instance="app",source="` + zone + `"}`: "1",
"failures_total" + `{instance="app",source="` + zone + `"}`: "0",
"queries_total" + `{instance="app",source="` + otherZone + `"}`: "1",
"failures_total" + `{instance="app",source="` + otherZone + `"}`: "1",
} {
line := "\nsmallwebwaf_reputation_" + series + " " + want + "\n"
if !strings.Contains(scraped.Body.String(), line) {
t.Errorf("metrics\n%s\nwant%s", scraped.Body.String(), line)
}
}
})
}
//nolint:paralleltest // one at a time, as the comment at the top of this file says
func TestZoneKeyIsMaskedInTheVerdictsTheFailuresAndTheMetrics(t *testing.T) {
const (
key = "abcdefghijklmnopqrstuvwxyz"
keyed = key + ".xbl.dq.spamhaus.net"
masked = "********.xbl.dq.spamhaus.net"
)
synctest.Test(t, func(t *testing.T) {
var log bytes.Buffer
queue := newQueue()
p := dnsblParams(keyed)
p.Alerts = queue
p.ProcessLog = slog.New(slog.NewJSONHandler(&log, nil))
dnsbl := newDNSBL(&resolverStandIn{answers: map[string]answer{
"99.2.0.192." + keyed + ".": {addrs: []string{listing}},
"100.2.0.192." + keyed + ".": {rcode: serverFailure},
}}, p)
m := metrics.New(1, "app")
m.AddReputation(reputation.New(params()), dnsbl)
// Both clients are asked about before either answer comes, so that
// the failure does not keep the zone from the other query.
wantZones(t, dnsbl, listed)
wantZones(t, dnsbl, unlisted)
synctest.Wait()
wantZones(t, dnsbl, listed, masked)
if got := waiting(queue); len(got) != 1 || got[0].Detail["source"] != masked {
t.Errorf("alerts waiting %+v, want the failure's, from %s", got, masked)
}
scraped := httptest.NewRecorder()
m.ServeHTTP(scraped, httptest.NewRequestWithContext(t.Context(), http.MethodGet,
"/", http.NoBody))
for name, shown := range map[string]string{
"the log": log.String(), "the metrics": scraped.Body.String(),
} {
if strings.Contains(shown, key) || !strings.Contains(shown, masked) {
t.Errorf("%s shows the key, or does not name the zone:\n%s", name, shown)
}
}
})
}
//nolint:paralleltest // one at a time, as the comment at the top of this file says
func TestVerdictsKeptAcrossARestart(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
resolver := &resolverStandIn{answers: map[string]answer{
listedName: {addrs: []string{listing}},
}}
dnsbl := newDNSBL(resolver, dnsblParams(zone))
fetched := time.Now()
wantZones(t, dnsbl, unlisted)
wantZones(t, dnsbl, listed)
synctest.Wait()
kept := dnsbl.Snapshot()
want := []reputation.Verdict{
{Zone: zone, Client: netip.MustParseAddr(listed), Listed: true, Fetched: fetched},
{Zone: zone, Client: netip.MustParseAddr(unlisted), Fetched: fetched},
}
if !reflect.DeepEqual(kept, want) {
t.Errorf("verdicts %+v, want %+v", kept, want)
}
// Restarted an hour later with what reputation.json keeps, it uses
// the verdicts, and asks the zone nothing, until the TTL has passed
// since they were fetched.
time.Sleep(time.Hour)
restarted := &resolverStandIn{}
again := newDNSBL(restarted, dnsblParams(zone))
again.Load(kept)
wantZones(t, again, listed, zone)
wantZones(t, again, unlisted)
synctest.Wait()
wantAsked(t, restarted)
time.Sleep(cacheTTL - time.Hour)
wantZones(t, again, listed)
synctest.Wait()
wantAsked(t, restarted, listedName)
})
}
func TestNeitherAVerdictOfAZoneNotNamedNorOnePastItsTTLIsKept(t *testing.T) {
t.Parallel()
now := time.Date(2026, 10, 7, 0, 0, 0, 0, time.UTC)
p := dnsblParams(zone)
p.Now = func() time.Time { return now }
dnsbl := reputation.NewDNSBL(p)
// The last verdict still in use, one fetched a TTL ago, and one of a
// zone SWWAF_DNSBL_ZONES does not name.
inUse := reputation.Verdict{
Zone: zone, Client: netip.MustParseAddr(listed), Listed: true,
Fetched: now.Add(-cacheTTL + time.Nanosecond),
}
stale := reputation.Verdict{
Zone: zone, Client: netip.MustParseAddr(unlisted), Fetched: now.Add(-cacheTTL),
}
notNamed := reputation.Verdict{
Zone: otherZone, Client: netip.MustParseAddr(listed), Listed: true, Fetched: now,
}
dnsbl.Load([]reputation.Verdict{notNamed, stale, inUse})
if got := dnsbl.Snapshot(); !reflect.DeepEqual(got, []reputation.Verdict{inUse}) {
t.Errorf("verdicts %+v, want only %+v", got, inUse)
}
}
func TestAtMost100000VerdictsKeptTheOneFetchedLongestAgoDroppedFirst(t *testing.T) {
t.Parallel()
now := time.Date(2026, 10, 7, 0, 0, 0, 0, time.UTC)
p := dnsblParams(zone)
p.Now = func() time.Time { return now }
dnsbl := reputation.NewDNSBL(p)
// 100,001 verdicts, listed by client, as reputation.json lists them,
// each fetched a millisecond before the one before it: the last is one
// too many.
const count = 100001
verdicts := make([]reputation.Verdict, 0, count)
client := netip.MustParseAddr("198.18.0.0")
for i := range count {
verdicts = append(verdicts, reputation.Verdict{
Zone: zone, Client: client, Fetched: now.Add(-time.Duration(i) * time.Millisecond),
})
client = client.Next()
}
dnsbl.Load(verdicts)
got := dnsbl.Snapshot()
if len(got) != count-1 || !slices.Contains(got, verdicts[0]) ||
slices.Contains(got, verdicts[count-1]) {
t.Errorf("%d verdicts kept, want all but the one fetched longest ago", len(got))
}
}
//nolint:paralleltest // one at a time, as the comment at the top of this file says
func TestQueriesGoToTheResolverSWWAFDNSBLResolverNames(t *testing.T) {
resolver := &resolverStandIn{answers: map[string]answer{
listedName: {addrs: []string{listing}},
}}
conn, err := (&net.ListenConfig{}).ListenPacket(t.Context(), "udp", "127.0.0.1:0")
if err != nil {
t.Fatalf("listen: %v", err)
}
served := make(chan struct{})
go func() {
resolver.serveUDP(conn)
close(served)
}()
t.Cleanup(func() {
_ = conn.Close()
<-served
})
p := dnsblParams(zone)
p.Resolver = netip.MustParseAddrPort(conn.LocalAddr().String())
// On the real clock: the stand-in answers at once, so only a test
// process held up for a whole minute would see the query fail.
p.Timeout = time.Minute
isListed, err := reputation.NewDNSBL(p).LookUp(zone, netip.MustParseAddr(listed))
if err != nil || !isListed {
t.Errorf("listed %t (%v), want true", isListed, err)
}
wantAsked(t, resolver, listedName)
}
// resolverStandIn is a stand-in for the resolver the zones are asked
// through. It answers each query by the name asked about, as answers
// gives, with no such name for a name answers does not give, and not at
// all while hanging. It notes each name asked about.
type resolverStandIn struct {
mu sync.Mutex
answers map[string]answer
hanging bool
names []string
}
// answer is how the stand-in answers a name: with an A record of each of
// addrs, or with the response code rcode, unless it is 0, for no error.
type answer struct {
addrs []string
rcode uint16
}
// What the stand-in reads of a query, and writes in its reply.
const (
// headerLength is the length of a DNS message's header, which the
// question follows: its id, its flags, and how many questions,
// answers and other records it holds, two bytes each.
headerLength = 12
// typeAndClass is the length of the type and the class that end a
// question, after its name.
typeAndClass = 4
// replyFlags mark a reply to a query that asked for recursion, which
// is available, with no error. The response code goes in their last
// four bits.
replyFlags = 0x8180
// maxMessage is the longest query read over UDP.
maxMessage = 1232
)
// set has the stand-in answer name with given.
func (s *resolverStandIn) set(name string, given answer) {
s.mu.Lock()
defer s.mu.Unlock()
s.answers[name] = given
}
// dial connects Go's resolver to the stand-in through an in-memory
// connection, on which it sends each query, and reads each reply, after
// its length, as over TCP.
func (s *resolverStandIn) dial(context.Context, string, string) (net.Conn, error) {
client, server := net.Pipe()
go s.serve(server)
return client, nil
}
// serve answers the queries that come on conn until the resolver closes
// it.
func (s *resolverStandIn) serve(conn net.Conn) {
defer func() {
_ = conn.Close()
}()
for {
var length [2]byte
_, err := io.ReadFull(conn, length[:])
if err != nil {
return
}
message := make([]byte, binary.BigEndian.Uint16(length[:]))
_, err = io.ReadFull(conn, message)
if err != nil {
return
}
reply, answered := s.reply(message)
if !answered {
continue // the resolver gives up, and closes conn
}
//nolint:gosec // a reply of a few dozen bytes
_, err = conn.Write(append(binary.BigEndian.AppendUint16(nil, uint16(len(reply))),
reply...))
if err != nil {
return
}
}
}
// serveUDP answers the queries that come on conn, each in a datagram, as
// a resolver does, until conn is closed.
func (s *resolverStandIn) serveUDP(conn net.PacketConn) {
message := make([]byte, maxMessage)
for {
n, from, err := conn.ReadFrom(message)
if err != nil {
return
}
reply, answered := s.reply(message[:n])
if answered {
_, _ = conn.WriteTo(reply, from)
}
}
}
// reply returns the stand-in's reply to message, a query, and false for
// none, while it hangs. It notes the name asked about.
func (s *resolverStandIn) reply(message []byte) ([]byte, bool) {
// The name is labels, each after its length, ended by a length of 0.
var labels []string
end := headerLength
for message[end] != 0 {
length := int(message[end])
labels = append(labels, string(message[end+1:end+1+length]))
end += 1 + length
}
end += 1 + typeAndClass
name := strings.Join(labels, ".") + "."
s.mu.Lock()
s.names = append(s.names, name)
given, found := s.answers[name]
hanging := s.hanging
s.mu.Unlock()
if hanging {
return nil, false
}
if !found {
given = answer{rcode: noSuchName}
}
// The query's id, the flags, one question, the answers, and no other
// records, then the question, as asked.
reply := slices.Clone(message[:2])
reply = binary.BigEndian.AppendUint16(reply, replyFlags|given.rcode)
reply = binary.BigEndian.AppendUint16(reply, 1)
//nolint:gosec // a handful of answers
reply = binary.BigEndian.AppendUint16(reply, uint16(len(given.addrs)))
reply = append(reply, 0, 0, 0, 0)
reply = append(reply, message[headerLength:end]...)
// An A record starts with the name asked about, by a pointer to it in
// the question, then its type, A, its class, IN, how long it may be
// kept, 60 seconds, and the length of its address, 4 bytes.
record := []byte{0xc0, headerLength, 0, 1, 0, 1, 0, 0, 0, 60, 0, 4}
for _, addr := range given.addrs {
reply = append(reply, record...)
reply = append(reply, netip.MustParseAddr(addr).AsSlice()...)
}
return reply, true
}
// dnsblParams returns the DNSBLParams of zones, with the tests' cache TTL
// and timeout, by the bubble's clock, with alerts to a queue that sends
// none.
func dnsblParams(zones ...string) reputation.DNSBLParams {
return reputation.DNSBLParams{
Zones: zones,
CacheTTL: cacheTTL,
Timeout: timeout,
Now: time.Now,
ProcessLog: slog.New(slog.DiscardHandler),
Alerts: newQueue(),
}
}
// waitForTheResolver waits, on the bubble's clock, an hour, until Go's
// resolver has given up on every stand-in that does not answer: it waits
// for a server as long as /etc/resolv.conf has it wait, a few seconds,
// even after the query was given up, and a bubble cannot end before it.
func waitForTheResolver() {
time.Sleep(time.Hour)
}
// newDNSBL returns the DNSBL of p, asking resolver.
func newDNSBL(resolver *resolverStandIn, p reputation.DNSBLParams) *reputation.DNSBL {
dnsbl := reputation.NewDNSBL(p)
dnsbl.SetDial(resolver.dial)
return dnsbl
}
// wantZones checks the zones whose verdict dnsbl says lists client, as a
// request from client finds them.
func wantZones(t *testing.T, dnsbl *reputation.DNSBL, client string, want ...string) {
t.Helper()
got := dnsbl.ListedBy(t.Context(), netip.MustParseAddr(client))
if !slices.Equal(got, want) {
t.Errorf("%s is listed by %v, want %v", client, got, want)
}
}
// wantQueries checks how many queries dnsbl made to zone, and how many of
// them failed.
func wantQueries(t *testing.T, dnsbl *reputation.DNSBL, queries, failures int) {
t.Helper()
if dnsbl.Queries(zone) != queries || dnsbl.Failures(zone) != failures {
t.Errorf("%d queries and %d failures, want %d and %d", dnsbl.Queries(zone),
dnsbl.Failures(zone), queries, failures)
}
}
// wantAsked checks the names the stand-in was asked about, in any order.
func wantAsked(t *testing.T, resolver *resolverStandIn, want ...string) {
t.Helper()
resolver.mu.Lock()
got := slices.Sorted(slices.Values(resolver.names))
resolver.mu.Unlock()
slices.Sort(want)
if !slices.Equal(got, want) {
t.Errorf("asked about %v, want %v", got, want)
}
}
internal/reputation/export_test.go
-33
View File
@@ -1,33 +0,0 @@
package reputation
import (
"context"
"net"
"net/http"
"net/netip"
)
// SetTransport has l's fetches go through transport instead of the
// network.
func (l *Lists) SetTransport(transport http.RoundTripper) {
l.httpClient.Transport = transport
}
// SetTransport has a's checks go through transport instead of the
// network.
func (a *AbuseIPDB) SetTransport(transport http.RoundTripper) {
a.httpClient.Transport = transport
}
// SetDial has d's queries go through dial instead of the network.
func (d *DNSBL) SetDial(
dial func(ctx context.Context, network, address string) (net.Conn, error),
) {
d.resolver = &net.Resolver{PreferGo: true, Dial: dial}
}
// LookUp asks zone about addr at once, as a query in the background does,
// and returns whether zone lists addr.
func (d *DNSBL) LookUp(zone string, addr netip.Addr) (bool, error) {
return d.lookUp(context.Background(), query{zone: zone, client: addr})
}
internal/reputation/reputation.go
-509
View File
@@ -1,509 +0,0 @@
// Package reputation fetches the lists the settings name by URL: the
// blocklists of SWWAF_BLOCKLIST_URLS, and the file of AS:percent lines
// SWWAF_ASN_LIMIT_PERCENT_URL names. It keeps the last good copy of each,
// whole, comment lines included, which is used while a fetch fails, and
// when each was last tried. It also asks the DNSBL zones of
// SWWAF_DNSBL_ZONES about clients, and keeps their verdicts, and checks
// clients with AbuseIPDB, and keeps their scores and the checks spent
// today. The state package writes all of these to reputation.json and
// reads them from it, so that a restart keeps them too.
package reputation
import (
"context"
"errors"
"fmt"
"io"
"log/slog"
"net/http"
"net/netip"
"slices"
"strings"
"sync"
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/config"
)
const (
// maxListBytes is the most of a list that is read. A longer one is a
// failure, so that a wrong URL cannot fill the memory.
maxListBytes = 16 << 20
// fetchTimeout bounds one fetch of a list.
fetchTimeout = time.Minute
// mappedBits is the length of ::ffff:0.0.0.0/96, the netblock of every
// IPv4-mapped address.
mappedBits = 96
)
var (
errStatus = errors.New("the server answered")
errTooLong = errors.New("the list is longer than 16 MiB")
errNotNetblock = errors.New("is not an address or a netblock, such as 192.0.2.0/24")
errNotASNPercent = errors.New(
"is not an AS number, : and a percentage, such as AS64496:50")
)
// List is a list as reputation.json holds it: the URL it is fetched from,
// when it was last tried, the fetch failed or not, and its last good copy:
// when that was fetched, and its lines, as fetched, comment lines
// included, both left out while no fetch of it has succeeded.
type List struct {
URL string `json:"url"`
Tried time.Time `json:"tried"`
Fetched time.Time `json:"fetched,omitzero"`
Lines []string `json:"lines,omitzero"`
}
// Params are what New needs.
type Params struct {
// BlocklistURLs are the blocklists (SWWAF_BLOCKLIST_URLS), and
// ASNLimitPercentURL the file of AS:percent lines
// (SWWAF_ASN_LIMIT_PERCENT_URL), "" while it is unset.
BlocklistURLs []string
ASNLimitPercentURL string
// Refresh is how long after a list was last fetched or tried it is
// fetched again (SWWAF_BLOCKLIST_REFRESH).
Refresh time.Duration
// Now tells the time, normally time.Now in UTC.
Now func() time.Time
// ProcessLog receives each fetch of a list, and why one failed.
ProcessLog *slog.Logger
// Alerts receive a source_failure alert for each fetch that fails.
Alerts *alerts.Queue
}
// Lists are the lists Params names, each with its last good copy. They
// are safe for concurrent use.
type Lists struct {
params Params
httpClient *http.Client
mu sync.Mutex
// lists are by URL, one for each URL Params names.
lists map[string]*list
}
// list is one list: what reputation.json keeps of it, its last try, zero
// before the first, and its last good copy, what that copy says, and how
// many fetches of it failed.
type list struct {
kept List
entries entries
failures int
}
// entries are what the lines of a copy say: for a blocklist, the netblocks
// it names, with the lengths among them, and for the file of AS:percent
// lines, the percentage it gives each AS number.
type entries struct {
netblocks map[netip.Prefix]bool
lengths []int
percents map[string]int64
}
// New returns the lists, without a copy of any yet.
func New(params Params) *Lists {
l := &Lists{params: params, httpClient: &http.Client{}, lists: map[string]*list{}}
for _, listURL := range l.URLs() {
l.lists[listURL] = &list{kept: List{URL: listURL}}
}
return l
}
// URLs returns the URL of every list: the blocklists' in the order
// SWWAF_BLOCKLIST_URLS names them, then SWWAF_ASN_LIMIT_PERCENT_URL.
func (l *Lists) URLs() []string {
urls := slices.Clone(l.params.BlocklistURLs)
if l.params.ASNLimitPercentURL != "" {
urls = append(urls, l.params.ASNLimitPercentURL)
}
return urls
}
// ListedBy returns the URLs of the blocklists whose copy lists addr, in
// the order SWWAF_BLOCKLIST_URLS names them.
func (l *Lists) ListedBy(addr netip.Addr) []string {
l.mu.Lock()
defer l.mu.Unlock()
var listedBy []string
for _, listURL := range l.params.BlocklistURLs {
if l.lists[listURL].entries.contain(addr) {
listedBy = append(listedBy, listURL)
}
}
return listedBy
}
// ASNLimitPercent returns the percentage the copy of the file of
// AS:percent lines gives asn, and whether it lists asn.
func (l *Lists) ASNLimitPercent(asn string) (int64, bool) {
if l.params.ASNLimitPercentURL == "" {
return 0, false
}
l.mu.Lock()
defer l.mu.Unlock()
percent, listed := l.lists[l.params.ASNLimitPercentURL].entries.percents[asn]
return percent, listed
}
// Fetched returns when the copy in use of the list at listURL was
// fetched, or zero while there is none.
func (l *Lists) Fetched(listURL string) time.Time {
l.mu.Lock()
defer l.mu.Unlock()
return l.lists[listURL].kept.Fetched
}
// Failures returns how many fetches of the list at listURL failed.
func (l *Lists) Failures(listURL string) int {
l.mu.Lock()
defer l.mu.Unlock()
return l.lists[listURL].failures
}
// Run fetches each list once Refresh has passed since it was last fetched
// or tried, the later of the two, until ctx is done. A list never tried is
// fetched at once, and so is one whose last try or copy, read from
// reputation.json, is that old.
func (l *Lists) Run(ctx context.Context) {
if len(l.lists) == 0 {
return
}
for ctx.Err() == nil {
next := l.fetchDue(ctx)
timer := time.NewTimer(next.Sub(l.params.Now()))
select {
case <-ctx.Done():
case <-timer.C:
}
timer.Stop()
}
}
// Snapshot returns each list that has been tried, with its copy, if it
// has one, sorted by URL, as reputation.json lists them.
func (l *Lists) Snapshot() []List {
l.mu.Lock()
tried := make([]List, 0, len(l.lists))
for _, held := range l.lists {
if !held.kept.Tried.IsZero() {
tried = append(tried, held.kept)
}
}
l.mu.Unlock()
slices.SortFunc(tried, func(a, b List) int {
return strings.Compare(a.URL, b.URL)
})
return tried
}
// Load puts lists, read from reputation.json, in place of the last tries
// and copies held. A list Params does not name is dropped. A copy with a
// line that parse refuses is an error, and then nothing changes.
func (l *Lists) Load(lists []List) error {
found := make(map[string]entries, len(lists))
for _, kept := range lists {
if _, named := l.lists[kept.URL]; !named {
continue
}
read, err := l.parse(kept.URL, kept.Lines)
if err != nil {
return fmt.Errorf("the copy of %s: %w", kept.URL, err)
}
found[kept.URL] = read
}
l.mu.Lock()
defer l.mu.Unlock()
for listURL, held := range l.lists {
held.kept, held.entries = List{URL: listURL}, entries{}
}
for _, kept := range lists {
read, named := found[kept.URL]
if named {
l.lists[kept.URL].kept, l.lists[kept.URL].entries = kept, read
}
}
return nil
}
// fetchDue fetches each list that is due, one after another, and returns
// when the next is due. Once ctx has ended, it starts none, since a fetch
// cut off is noted as a try.
func (l *Lists) fetchDue(ctx context.Context) time.Time {
var next time.Time
for _, listURL := range l.URLs() {
due := l.due(listURL)
if ctx.Err() == nil && !l.params.Now().Before(due) {
l.fetch(ctx, listURL)
due = l.due(listURL)
}
if next.IsZero() || due.Before(next) {
next = due
}
}
return next
}
// due returns when the list at listURL is to be fetched: Refresh after it
// was last fetched or tried, the later of the two.
func (l *Lists) due(listURL string) time.Time {
l.mu.Lock()
defer l.mu.Unlock()
held := l.lists[listURL]
last := held.kept.Fetched
if held.kept.Tried.After(last) {
last = held.kept.Tried
}
return last.Add(l.params.Refresh)
}
// fetch fetches the list at listURL, and notes the try. A good copy takes
// the place of the one held. A failure leaves that in use, and is counted,
// logged and raised as a source_failure alert. A fetch cut off as ctx
// ends, as smallwebwaf stops, is no failure, but is still noted as a try,
// so that a restart waits for it: the server may have had its request.
func (l *Lists) fetch(ctx context.Context, listURL string) {
lines, err := l.get(ctx, listURL)
var found entries
if err == nil {
found, err = l.parse(listURL, lines)
}
cutOff := err != nil && ctx.Err() != nil
now := l.params.Now()
l.mu.Lock()
held := l.lists[listURL]
held.kept.Tried = now
if err == nil {
held.kept.Fetched, held.kept.Lines = now, lines
held.entries = found
} else if !cutOff {
held.failures++
}
l.mu.Unlock()
if cutOff {
return
}
if err != nil {
const failed = "fetching a list failed"
// Raised before it is logged, so that the alert is there once the
// log line is.
raiseFailure(l.params.Alerts, failed, listURL, err)
l.params.ProcessLog.Warn(failed, "url", listURL, "error", err.Error())
return
}
l.params.ProcessLog.Info("fetched a list", "url", listURL, "lines", len(lines))
}
// raiseFailure raises a source_failure alert into queue, with reason, and
// in its detail the source that failed, a list's URL, a zone with its key
// masked or abuseipdb, and err.
func raiseFailure(queue *alerts.Queue, reason, source string, err error) {
queue.Raise(alerts.Alert{
Event: alerts.EventSourceFailure,
Reason: reason,
Detail: map[string]any{"source": source, "error": err.Error()},
})
}
// get fetches the list at listURL, and returns its lines. An answer other
// than 200, or a list longer than maxListBytes, is a failure.
func (l *Lists) get(ctx context.Context, listURL string) ([]string, error) {
ctx, cancel := context.WithTimeout(ctx, fetchTimeout)
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, listURL, http.NoBody)
if err != nil {
return nil, fmt.Errorf("make the request: %w", err)
}
res, err := l.httpClient.Do(req)
if err != nil {
// Do's error names the URL, which the log line and the alert name
// already: only what went wrong is kept.
return nil, fmt.Errorf("fetch the list: %w", errors.Unwrap(err))
}
defer func() {
_ = res.Body.Close()
}()
if res.StatusCode != http.StatusOK {
return nil, fmt.Errorf("%w %s", errStatus, res.Status)
}
body, err := io.ReadAll(io.LimitReader(res.Body, maxListBytes+1))
if err != nil {
return nil, fmt.Errorf("read the list: %w", err)
}
if len(body) > maxListBytes {
return nil, errTooLong
}
lines := []string{}
for line := range strings.Lines(string(body)) {
lines = append(lines, strings.TrimSuffix(line, "\n"))
}
return lines, nil
}
// parse reads the lines of the list at listURL: those of a blocklist, or
// of the file of AS:percent lines. Anything after a ; or a # on a line is
// left out, and so is a line left blank. Any other line that does not read
// is an error naming it by its number.
func (l *Lists) parse(listURL string, lines []string) (entries, error) {
if listURL == l.params.ASNLimitPercentURL {
return parsePercents(lines)
}
return parseNetblocks(lines)
}
// parseNetblocks reads a blocklist's lines, each an address or a netblock
// as the settings take them.
func parseNetblocks(lines []string) (entries, error) {
found := entries{netblocks: map[netip.Prefix]bool{}}
for i, line := range lines {
text := withoutComment(line)
if text == "" {
continue
}
netblock, ok := parseNetblock(text)
if !ok {
return entries{}, fmt.Errorf("line %d %w", i+1, errNotNetblock)
}
found.netblocks[netblock] = true
if !slices.Contains(found.lengths, netblock.Bits()) {
found.lengths = append(found.lengths, netblock.Bits())
}
}
return found, nil
}
// parseNetblock reads text, a line of a blocklist, and reports whether it
// is an address or a netblock as the settings take them. A client's IPv4
// address is checked as IPv4, never IPv4-mapped, so an IPv4-mapped line,
// such as ::ffff:192.0.2.0/120, is read as the IPv4 address or netblock it
// stands for, 192.0.2.0/24, and a mapped netblock shorter than /96, which
// stands for none, is refused.
func parseNetblock(text string) (netip.Prefix, bool) {
netblock, err := config.ParseNetblock(text)
if err != nil {
return netip.Prefix{}, false
}
// The address as written: ParseNetblock's has the bits past the
// netblock's length cleared, the ::ffff among them below /96.
written, _, _ := strings.Cut(text, "/")
if addr, _ := netip.ParseAddr(written); !addr.Is4In6() {
return netblock, true
}
if netblock.Bits() < mappedBits {
return netip.Prefix{}, false
}
return netip.PrefixFrom(netblock.Addr().Unmap(), netblock.Bits()-mappedBits), true
}
// parsePercents reads the lines of the file of AS:percent lines, each an
// AS number, : and a percentage, as SWWAF_ASN_LIMIT_PERCENT takes them. An
// AS number listed more than once gets the lowest of its percentages.
func parsePercents(lines []string) (entries, error) {
found := entries{percents: map[string]int64{}}
for i, line := range lines {
text := withoutComment(line)
if text == "" {
continue
}
asnText, percentText, _ := strings.Cut(text, ":")
asn, asnErr := config.ParseASN(asnText)
percent, percentErr := config.ParsePercent(percentText)
if asnErr != nil || percentErr != nil {
return entries{}, fmt.Errorf("line %d %w", i+1, errNotASNPercent)
}
earlier, listed := found.percents[asn]
if !listed || percent < earlier {
found.percents[asn] = percent
}
}
return found, nil
}
// withoutComment returns line without anything after a ; or a #, and
// without the spaces around what is left.
func withoutComment(line string) string {
text, _, _ := strings.Cut(line, ";")
text, _, _ = strings.Cut(text, "#")
return strings.TrimSpace(text)
}
// contain reports whether the netblocks of a blocklist's copy hold addr:
// whether addr, cut to one of their lengths, is one of them.
func (e entries) contain(addr netip.Addr) bool {
for _, length := range e.lengths {
netblock, err := addr.Prefix(length)
if err == nil && e.netblocks[netblock] {
return true
}
}
return false
}
internal/reputation/reputation_test.go
-610
View File
@@ -1,610 +0,0 @@
package reputation_test
import (
"bytes"
"context"
"fmt"
"io"
"log/slog"
"net/http"
"net/netip"
"net/url"
"reflect"
"slices"
"strings"
"sync"
"testing"
"testing/synctest"
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/reputation"
)
// The tests run in a synctest bubble, where the time package runs on a
// clock of the test's own: time.Sleep moves it on at once, and
// synctest.Wait returns once Run waits for the next list to be due, so
// that every fetch due by then has been made. The stand-in for the
// servers the lists are fetched from answers without the network, since a
// fetch waiting on the network would keep that clock from moving on.
const (
// dropURL and torURL are the blocklists, and asnURL the file of
// AS:percent lines.
dropURL = "https://lists.example/drop.txt"
torURL = "https://lists.example/tor.txt"
asnURL = "https://lists.example/asn.txt"
// refresh is the tests' SWWAF_BLOCKLIST_REFRESH, and cooldown their
// SWWAF_ALERT_COOLDOWN, longer than it.
refresh = 24 * time.Hour
cooldown = 48 * time.Hour
// drop is a blocklist as the Spamhaus DROP list is written, with an
// address and a netblock in each of its comments, which list nothing.
drop = "; Spamhaus DROP List 2026/10/07 - (c) 2026 The Spamhaus Project SLL\n" +
"; Last-Modified: Wed, 07 Oct 2026 00:00:00 GMT ; 192.0.2.1\n" +
"# 198.51.100.0/24\n" +
"\n" +
"203.0.113.0/24 ; SBL1\n" +
" 192.0.2.9 # one address\n" +
"2001:db8:1::/48 ; SBL2\n"
)
func TestListedAddressesAndNetblocksWithTheCommentsLeftOut(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
servers := &standIn{lists: map[string]string{dropURL: drop}}
lists := start(t, servers, params(dropURL))
for addr, want := range map[string][]string{
"203.0.113.0": {dropURL},
"203.0.113.255": {dropURL},
"192.0.2.9": {dropURL},
"2001:db8:1::7": {dropURL},
"203.0.114.0": nil,
"192.0.2.8": nil,
"192.0.2.1": nil,
"198.51.100.7": nil,
"2001:db8:2::7": nil,
} {
wantListedBy(t, lists, addr, want...)
}
})
}
func TestIPv4MappedLineListsTheIPv4AddressOrNetblockItStandsFor(t *testing.T) {
t.Parallel()
now := time.Now()
lists := reputation.New(params(dropURL))
err := lists.Load([]reputation.List{{
URL: dropURL, Tried: now, Fetched: now,
Lines: []string{"::ffff:192.0.2.9", "::ffff:203.0.113.0/120"},
}})
if err != nil {
t.Fatalf("load: %v", err)
}
for addr, want := range map[string][]string{
"192.0.2.9": {dropURL},
"203.0.113.255": {dropURL},
"192.0.2.8": nil,
"203.0.114.0": nil,
} {
wantListedBy(t, lists, addr, want...)
}
// A mapped netblock shorter than /96 stands for no IPv4 one.
err = lists.Load([]reputation.List{{
URL: dropURL, Tried: now, Fetched: now, Lines: []string{"::ffff:198.51.100.0/88"},
}})
const want = "the copy of " + dropURL +
": line 1 is not an address or a netblock, such as 192.0.2.0/24"
if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want)
}
}
func TestClientIsListedByEachBlocklistThatListsIt(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
servers := &standIn{lists: map[string]string{
dropURL: "203.0.113.0/24\n", torURL: "203.0.113.9\n",
}}
lists := start(t, servers, params(torURL, dropURL))
// In the order SWWAF_BLOCKLIST_URLS names them.
wantListedBy(t, lists, "203.0.113.9", torURL, dropURL)
wantListedBy(t, lists, "203.0.113.8", dropURL)
})
}
func TestListFetchedAgainOnceRefreshHasPassed(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
servers := &standIn{lists: map[string]string{dropURL: "203.0.113.9\n"}}
lists := start(t, servers, params(dropURL))
began := time.Now()
wantFetches(t, servers, 1)
servers.set(dropURL, "203.0.113.10\n")
time.Sleep(refresh - time.Nanosecond)
wantFetches(t, servers, 1)
wantListedBy(t, lists, "203.0.113.9", dropURL)
time.Sleep(time.Nanosecond)
wantFetches(t, servers, 2)
wantListedBy(t, lists, "203.0.113.9")
wantListedBy(t, lists, "203.0.113.10", dropURL)
if fetched := lists.Fetched(dropURL); !fetched.Equal(began.Add(refresh)) {
t.Errorf("the copy in use was fetched at %s, want %s", fetched,
began.Add(refresh))
}
})
}
func TestFailedFetchKeepsTheLastGoodCopyAndAlertsOncePerCooldown(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
// fail has the stand-in answer the fetches after the first so that
// they fail with error.
fail func(servers *standIn)
error string
}{
{
"an answer other than 200",
func(servers *standIn) { servers.set(dropURL, "") },
"the server answered 503 Service Unavailable",
},
{
"a line that does not read",
func(servers *standIn) { servers.set(dropURL, "203.0.113.10\n<html>\n") },
"line 2 is not an address or a netblock, such as 192.0.2.0/24",
},
{
"a list longer than 16 MiB",
func(servers *standIn) {
servers.set(dropURL, strings.Repeat("#\n", 8<<20+1))
},
"the list is longer than 16 MiB",
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
var log bytes.Buffer
servers := &standIn{lists: map[string]string{dropURL: drop}}
queue := newQueue()
p := params(dropURL)
p.ProcessLog = slog.New(slog.NewJSONHandler(&log, nil))
p.Alerts = queue
lists := start(t, servers, p)
kept := lists.Snapshot()
tc.fail(servers)
// Each failure is tried again once refresh has passed since it.
for range 2 {
time.Sleep(refresh)
synctest.Wait()
}
wantFetches(t, servers, 3)
wantListedBy(t, lists, "203.0.113.9", dropURL)
want := kept[0]
want.Tried = time.Now()
if got := lists.Snapshot(); !reflect.DeepEqual(got, []reputation.List{want}) {
t.Errorf("lists %+v, want the first copy, last tried now, %+v", got, want)
}
if lists.Failures(dropURL) != 2 {
t.Errorf("%d failures, want 2", lists.Failures(dropURL))
}
// One alert for the first failure; the cooldown holds back the
// second.
wantAlert(t, queue, alerts.Alert{
Time: time.Now().Add(-refresh),
Event: alerts.EventSourceFailure,
Reason: "fetching a list failed",
Detail: map[string]any{"source": dropURL, "error": tc.error},
})
if !strings.Contains(log.String(), `"msg":"fetching a list failed",`+
`"url":"`+dropURL+`","error":"`+tc.error) {
t.Errorf("logged\n%s\nwant the failures", log.String())
}
})
})
}
}
func TestFetchNotDoneWithinAMinuteFails(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
servers := &standIn{lists: map[string]string{}, hanging: true}
lists := start(t, servers, params(dropURL))
time.Sleep(time.Minute - time.Nanosecond)
synctest.Wait()
if lists.Failures(dropURL) != 0 {
t.Errorf("%d failures before a minute, want none", lists.Failures(dropURL))
}
time.Sleep(time.Nanosecond)
synctest.Wait()
if lists.Failures(dropURL) != 1 {
t.Errorf("%d failures after a minute, want 1", lists.Failures(dropURL))
}
})
}
func TestKeptCopyIsFetchedAgainOnceRefreshHasPassedSinceItWasFetched(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
servers := &standIn{lists: map[string]string{
dropURL: "203.0.113.10\n", torURL: "198.51.100.10\n",
}}
lists := reputation.New(params(dropURL, torURL))
lists.SetTransport(servers)
// drop.txt was fetched an hour ago, and tor.txt a refresh ago, as
// reputation.json says at start.
err := lists.Load([]reputation.List{
{URL: dropURL, Fetched: time.Now().Add(-time.Hour), Lines: []string{"203.0.113.9"}},
{URL: torURL, Fetched: time.Now().Add(-refresh), Lines: []string{"198.51.100.9"}},
})
if err != nil {
t.Fatalf("load: %v", err)
}
run(t, lists)
wantFetches(t, servers, 1)
wantListedBy(t, lists, "203.0.113.9", dropURL)
wantListedBy(t, lists, "198.51.100.10", torURL)
time.Sleep(refresh - time.Hour - time.Nanosecond)
wantFetches(t, servers, 1)
time.Sleep(time.Nanosecond)
wantFetches(t, servers, 2)
wantListedBy(t, lists, "203.0.113.10", dropURL)
})
}
func TestRestartWaitsRefreshAfterTheLastTryEvenOneThatFailed(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
// drop.txt is fetched, and a refresh later the fetch downloads it
// whole but fails on a line that does not read.
servers := &standIn{lists: map[string]string{dropURL: "198.51.100.1\n"}}
lists := start(t, servers, params(dropURL))
servers.set(dropURL, "198.51.100.2\n<html>\n")
time.Sleep(refresh)
wantFetches(t, servers, 2)
// Restarted with what reputation.json keeps, it waits a refresh
// after the failed try, as it does while it runs.
restarted := &standIn{lists: map[string]string{dropURL: "198.51.100.2\n"}}
again := reputation.New(params(dropURL))
again.SetTransport(restarted)
err := again.Load(lists.Snapshot())
if err != nil {
t.Fatalf("load: %v", err)
}
run(t, again)
wantFetches(t, restarted, 0)
wantListedBy(t, again, "198.51.100.1", dropURL)
time.Sleep(refresh - time.Nanosecond)
wantFetches(t, restarted, 0)
time.Sleep(time.Nanosecond)
wantFetches(t, restarted, 1)
wantListedBy(t, again, "198.51.100.2", dropURL)
})
}
func TestFetchCutOffAsItStopsIsNoFailureButARestartWaitsForIt(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
// Stopped 30 seconds into the fetch of drop.txt, before tor.txt's.
servers := &standIn{lists: map[string]string{}, hanging: true}
queue := newQueue()
p := params(dropURL, torURL)
p.Alerts = queue
lists := reputation.New(p)
lists.SetTransport(servers)
ctx, stop := context.WithCancel(t.Context())
stopped := make(chan struct{})
go func() {
lists.Run(ctx)
close(stopped)
}()
time.Sleep(30 * time.Second)
stop()
<-stopped
wantFetches(t, servers, 1)
if lists.Failures(dropURL) != 0 || len(waiting(queue)) != 0 {
t.Errorf("%d failures and alerts %+v, want none", lists.Failures(dropURL),
waiting(queue))
}
// Restarted an hour later with what reputation.json keeps, it fetches
// tor.txt, never tried, at once, and drop.txt a refresh after its
// cut-off try.
time.Sleep(time.Hour)
restarted := &standIn{lists: map[string]string{
dropURL: "203.0.113.7\n", torURL: "198.51.100.7\n",
}}
again := reputation.New(params(dropURL, torURL))
again.SetTransport(restarted)
err := again.Load(lists.Snapshot())
if err != nil {
t.Fatalf("load: %v", err)
}
run(t, again)
wantFetches(t, restarted, 1)
wantListedBy(t, again, "198.51.100.7", torURL)
time.Sleep(refresh - time.Hour - time.Nanosecond)
wantFetches(t, restarted, 1)
time.Sleep(time.Nanosecond)
wantFetches(t, restarted, 2)
wantListedBy(t, again, "203.0.113.7", dropURL)
})
}
func TestASNLimitPercentFileGivesEachASNumberItsLowestPercentage(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
servers := &standIn{lists: map[string]string{
asnURL: "# hosting networks\nAS14061:50 ; DigitalOcean\nas16276:25\n\n" +
"AS14061:10\nAS14061:30\n",
}}
p := params()
p.ASNLimitPercentURL = asnURL
lists := start(t, servers, p)
for asn, want := range map[string]int64{"AS14061": 10, "AS16276": 25} {
percent, listed := lists.ASNLimitPercent(asn)
if !listed || percent != want {
t.Errorf("%s has %d (listed %t), want %d", asn, percent, listed, want)
}
}
if _, listed := lists.ASNLimitPercent("AS64496"); listed {
t.Error("AS64496 is listed")
}
// A line that does not read fails the fetch.
servers.set(asnURL, "AS14061:50\nAS16276\n")
time.Sleep(refresh)
synctest.Wait()
if lists.Failures(asnURL) != 1 {
t.Errorf("%d failures, want 1", lists.Failures(asnURL))
}
})
}
func TestLoadDropsCopiesOfListsNotNamedAndRefusesOnesThatDoNotRead(t *testing.T) {
t.Parallel()
fetched := time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)
kept := reputation.List{
URL: dropURL, Tried: fetched, Fetched: fetched, Lines: []string{"203.0.113.9"},
}
lists := reputation.New(params(dropURL))
err := lists.Load([]reputation.List{kept, {
URL: torURL, Tried: fetched, Fetched: fetched, Lines: []string{"198.51.100.9"},
}})
if err != nil {
t.Fatalf("load: %v", err)
}
if got := lists.Snapshot(); !reflect.DeepEqual(got, []reputation.List{kept}) {
t.Errorf("copies %+v, want only %+v", got, kept)
}
err = lists.Load([]reputation.List{{URL: dropURL, Fetched: fetched, Lines: []string{
"; DROP", "203.0.113.300",
}}})
const want = "the copy of " + dropURL +
": line 2 is not an address or a netblock, such as 192.0.2.0/24"
if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want)
}
if got := lists.Snapshot(); !reflect.DeepEqual(got, []reputation.List{kept}) {
t.Errorf("copies %+v after the error, want %+v still", got, kept)
}
}
// standIn is a stand-in for the servers the lists are fetched from. It
// notes the URL of each fetch.
type standIn struct {
mu sync.Mutex
// lists are what it answers with, by URL; it answers a URL it has no
// list for with 503, and none at all while hanging.
lists map[string]string
hanging bool
fetches []string
}
// RoundTrip has the stand-in answer req, in place of the network. A fetch
// abandoned before the stand-in answers fails, as over the network.
func (s *standIn) RoundTrip(req *http.Request) (*http.Response, error) {
s.mu.Lock()
s.fetches = append(s.fetches, req.URL.String())
list, found := s.lists[req.URL.String()]
hanging := s.hanging
s.mu.Unlock()
if hanging {
<-req.Context().Done()
return nil, req.Context().Err()
}
status := http.StatusOK
if !found {
status = http.StatusServiceUnavailable
}
return &http.Response{
StatusCode: status,
Status: fmt.Sprintf("%d %s", status, http.StatusText(status)),
Header: http.Header{},
Body: io.NopCloser(strings.NewReader(list)),
Request: req,
}, nil
}
// set has the stand-in answer listURL with list, or with 503 for "".
func (s *standIn) set(listURL, list string) {
s.mu.Lock()
defer s.mu.Unlock()
if list == "" {
delete(s.lists, listURL)
return
}
s.lists[listURL] = list
}
// params returns the Params of the blocklists at urls, refreshed every
// refresh, by the bubble's clock, with alerts to a queue that sends none.
func params(urls ...string) reputation.Params {
return reputation.Params{
BlocklistURLs: urls,
Refresh: refresh,
Now: time.Now,
ProcessLog: slog.New(slog.DiscardHandler),
Alerts: newQueue(),
}
}
// newQueue returns a queue of alerts to a webhook that is never sent
// them, with a cooldown of cooldown.
func newQueue() *alerts.Queue {
return alerts.New(alerts.Params{
WebhookURL: &url.URL{Scheme: "https", Host: "alerts.example"},
Events: alerts.Events(),
Cooldown: cooldown,
Now: time.Now,
})
}
// start returns the lists of p, fetched through servers by Run, which runs
// until the test ends, once Run has fetched those due at start.
func start(t *testing.T, servers *standIn, p reputation.Params) *reputation.Lists {
t.Helper()
lists := reputation.New(p)
lists.SetTransport(servers)
run(t, lists)
return lists
}
// run runs lists' Run until the test ends, and waits until it has fetched
// the lists due.
func run(t *testing.T, lists *reputation.Lists) {
t.Helper()
ctx, stop := context.WithCancel(t.Context())
stopped := make(chan struct{})
go func() {
lists.Run(ctx)
close(stopped)
}()
t.Cleanup(func() {
stop()
<-stopped
})
synctest.Wait()
}
// wantFetches waits until Run has made the fetches due, and checks how
// many the servers have had.
func wantFetches(t *testing.T, servers *standIn, want int) {
t.Helper()
synctest.Wait()
servers.mu.Lock()
got := len(servers.fetches)
servers.mu.Unlock()
if got != want {
t.Errorf("%d fetches, want %d", got, want)
}
}
// wantListedBy checks the URLs of the blocklists lists says list addr.
func wantListedBy(t *testing.T, lists *reputation.Lists, addr string, want ...string) {
t.Helper()
got := lists.ListedBy(netip.MustParseAddr(addr))
if !slices.Equal(got, want) {
t.Errorf("%s is listed by %v, want %v", addr, got, want)
}
}
// waiting returns the alerts waiting in queue.
func waiting(queue *alerts.Queue) []alerts.Alert {
return queue.Snapshot().Waiting[alerts.DestinationWebhook]
}
// wantAlert checks that want is the one alert waiting in queue, and that
// the cooldown has held back one repeat of it.
func wantAlert(t *testing.T, queue *alerts.Queue, want alerts.Alert) {
t.Helper()
got := waiting(queue)
if len(got) != 1 || !reflect.DeepEqual(got[0], want) || queue.Suppressed() != 1 {
t.Errorf("alerts waiting %+v, %d held back, want only %+v and 1", got,
queue.Suppressed(), want)
}
}
internal/requestlog/requestlog.go
+2 -11
View File
@@ -35,9 +35,7 @@ const (
// rule. // rule.
ActionRuleBlocked = "rule_blocked" ActionRuleBlocked = "rule_blocked"
// ActionDenied is a request refused because its client is in // ActionDenied is a request refused because its client is in
// SWWAF_DENY_NETS, in a blocklist while SWWAF_BLOCKLIST_ACTION is deny, // SWWAF_DENY_NETS.
// or listed by a DNSBL zone, or scored a hit by AbuseIPDB, while
// SWWAF_REPUTATION_ACTION is deny.
ActionDenied = "denied" ActionDenied = "denied"
// ActionCountryDenied is a request refused for its client's country. // ActionCountryDenied is a request refused for its client's country.
ActionCountryDenied = "country_denied" ActionCountryDenied = "country_denied"
@@ -139,10 +137,6 @@ type Line struct {
// Counts names its count: minute, hour or day for a rate limit, and // Counts names its count: minute, hour or day for a rate limit, and
// minute_bytes, hour_bytes or day_bytes for a byte limit. // minute_bytes, hour_bytes or day_bytes for a byte limit.
LimitHit string `json:"limit_hit,omitempty"` LimitHit string `json:"limit_hit,omitempty"`
// Reputation are the URLs of the blocklists that list the client, then
// the DNSBL zones whose verdict lists it, their keys masked, then
// abuseipdb when its score is a hit.
Reputation []string `json:"reputation,omitempty"`
// Offence is the offence the request was held as, OffenceLimit. // Offence is the offence the request was held as, OffenceLimit.
Offence string `json:"offence,omitempty"` Offence string `json:"offence,omitempty"`
// BanExpires is when the ban the request made, or was refused under, // BanExpires is when the ban the request made, or was refused under,
@@ -193,11 +187,8 @@ func Milliseconds(d time.Duration) float64 {
// NewProcessLogger returns the logger for the process's own messages: // NewProcessLogger returns the logger for the process's own messages:
// JSON lines on w, marked "type":"process", with the time in the same form // JSON lines on w, marked "type":"process", with the time in the same form
// as a request line's, and instanceName, SWWAF_INSTANCE_NAME, as instance. // as a request line's, and instanceName, SWWAF_INSTANCE_NAME, as instance.
// It writes only the messages at level, SWWAF_LOG_LEVEL, or more severe; func NewProcessLogger(w io.Writer, instanceName string) *slog.Logger {
// the request lines Write writes are never held back.
func NewProcessLogger(w io.Writer, instanceName string, level slog.Level) *slog.Logger {
handler := slog.NewJSONHandler(w, &slog.HandlerOptions{ handler := slog.NewJSONHandler(w, &slog.HandlerOptions{
Level: level,
ReplaceAttr: func(groups []string, attr slog.Attr) slog.Attr { ReplaceAttr: func(groups []string, attr slog.Attr) slog.Attr {
if attr.Key == slog.TimeKey && len(groups) == 0 { if attr.Key == slog.TimeKey && len(groups) == 0 {
return slog.String(slog.TimeKey, FormatTime(attr.Value.Time())) return slog.String(slog.TimeKey, FormatTime(attr.Value.Time()))
internal/requestlog/requestlog_test.go
+1 -48
View File
@@ -3,8 +3,6 @@ package requestlog_test
import ( import (
"bytes" "bytes"
"encoding/json" "encoding/json"
"log/slog"
"slices"
"strings" "strings"
"testing" "testing"
"time" "time"
@@ -72,8 +70,7 @@ func TestProcessLinesAreMarkedProcessAndGiveTheInstance(t *testing.T) {
var out bytes.Buffer var out bytes.Buffer
requestlog.NewProcessLogger(&out, "fsn1app1/gitea", slog.LevelInfo).Info("starting", requestlog.NewProcessLogger(&out, "fsn1app1/gitea").Info("starting", "version", "v1")
"version", "v1")
var fields map[string]any var fields map[string]any
@@ -97,47 +94,3 @@ func TestProcessLinesAreMarkedProcessAndGiveTheInstance(t *testing.T) {
t.Errorf("process line time %q, want now in UTC with milliseconds", timeText) t.Errorf("process line time %q, want now in UTC with milliseconds", timeText)
} }
} }
func TestProcessLoggerWritesTheMessagesAtItsLevelOrMoreSevere(t *testing.T) {
t.Parallel()
levels := []slog.Level{
slog.LevelDebug, slog.LevelInfo, slog.LevelWarn, slog.LevelError,
}
for i, level := range levels {
t.Run(level.String(), func(t *testing.T) {
t.Parallel()
var out bytes.Buffer
processLog := requestlog.NewProcessLogger(&out, "fsn1app1/gitea", level)
for _, at := range levels {
processLog.Log(t.Context(), at, "message")
}
var got, want []string
for line := range strings.Lines(out.String()) {
var fields struct {
Level string `json:"level"`
}
err := json.Unmarshal([]byte(line), &fields)
if err != nil {
t.Fatalf("decode %q: %v", line, err)
}
got = append(got, fields.Level)
}
for _, written := range levels[i:] {
want = append(want, written.String())
}
if !slices.Equal(got, want) {
t.Errorf("lines at %v, want %v", got, want)
}
})
}
}
internal/smallwebwaf/smallwebwaf.go
+12 -25
View File
@@ -21,7 +21,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/lookup" "sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/proxy" "sneak.berlin/go/smallwebwaf/internal/proxy"
"sneak.berlin/go/smallwebwaf/internal/remotelog" "sneak.berlin/go/smallwebwaf/internal/remotelog"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules" "sneak.berlin/go/smallwebwaf/internal/rules"
"sneak.berlin/go/smallwebwaf/internal/state" "sneak.berlin/go/smallwebwaf/internal/state"
@@ -70,10 +69,8 @@ func Main(version string) int {
// state files, then serves requests until ctx is done. It returns the // state files, then serves requests until ctx is done. It returns the
// process's exit status, 1 when smallwebwaf cannot start. // process's exit status, 1 when smallwebwaf cannot start.
func Run(ctx context.Context, params Params) int { func Run(ctx context.Context, params Params) int {
// Until the settings are read, the one message is an invalid setting's
// error, which every SWWAF_LOG_LEVEL lets through.
processLog := requestlog.NewProcessLogger(params.Stdout, processLog := requestlog.NewProcessLogger(params.Stdout,
config.InstanceName(params.LookupEnv), slog.LevelError) config.InstanceName(params.LookupEnv))
cfg, err := config.FromEnvironment(params.LookupEnv) cfg, err := config.FromEnvironment(params.LookupEnv)
if err != nil { if err != nil {
@@ -91,11 +88,8 @@ func Run(ctx context.Context, params Params) int {
if cfg.LogRemoteURL != nil { if cfg.LogRemoteURL != nil {
remote = newRemoteLogSender(cfg) remote = newRemoteLogSender(cfg)
stdout = io.MultiWriter(params.Stdout, remote) stdout = io.MultiWriter(params.Stdout, remote)
} processLog = requestlog.NewProcessLogger(stdout, cfg.InstanceName)
processLog = requestlog.NewProcessLogger(stdout, cfg.InstanceName, cfg.LogLevel)
if remote != nil {
stopSending := startSending(ctx, remote, processLog) stopSending := startSending(ctx, remote, processLog)
defer stopSending() defer stopSending()
} }
@@ -175,15 +169,14 @@ func newServer(
} }
server := proxy.New(proxy.Params{ server := proxy.New(proxy.Params{
Config: cfg, Config: cfg,
RequestLog: stdout, RequestLog: stdout,
ProcessLog: processLog, ProcessLog: processLog,
GeoJSURL: lookup.URL, GeoJSURL: lookup.URL,
AbuseIPDBURL: reputation.AbuseIPDBURL, LookupFile: lookupFile,
LookupFile: lookupFile, Now: now,
Now: now, Rules: ruleFiles,
Rules: ruleFiles, Alerts: alertQueue,
Alerts: alertQueue,
}) })
server.Metrics.AddAlerts(alertQueue) server.Metrics.AddAlerts(alertQueue)
@@ -207,9 +200,6 @@ func loadStateFiles(
Ledger: server.Ledger, Ledger: server.Ledger,
Limiter: server.Limiter, Limiter: server.Limiter,
GeoJS: server.GeoJS, GeoJS: server.GeoJS,
Lists: server.Lists,
DNSBL: server.DNSBL,
AbuseIPDB: server.AbuseIPDB,
Alerts: alertQueue, Alerts: alertQueue,
Anomalies: server.Anomalies, Anomalies: server.Anomalies,
Now: now, Now: now,
@@ -277,9 +267,8 @@ func startSending(
// serve serves requests on listener, writes the state files as they are // serve serves requests on listener, writes the state files as they are
// due, takes in an admin's edits of them, reads the rule files again as // due, takes in an admin's edits of them, reads the rule files again as
// they change, and the lookup database when it is replaced, fetches the // they change, and the lookup database when it is replaced, and sends the
// lists the settings name by URL as they are due, and sends the alerts, // alerts, until ctx is done. Then it gives the requests in progress
// until ctx is done. Then it gives the requests in progress
// shutdownTimeout to finish, and writes every state file, alerts.json with // shutdownTimeout to finish, and writes every state file, alerts.json with
// the alerts still waiting. // the alerts still waiting.
func serve( func serve(
@@ -304,7 +293,6 @@ func serve(
server.LookupFile.Watch(writing) server.LookupFile.Watch(writing)
} }
}) })
listsFetched := inBackground(func() { server.Lists.Run(writing) })
alertsSent := inBackground(func() { alertQueue.Run(writing) }) alertsSent := inBackground(func() { alertQueue.Run(writing) })
select { select {
@@ -348,7 +336,6 @@ func serve(
<-watched <-watched
<-rulesWatched <-rulesWatched
<-lookupFileWatched <-lookupFileWatched
<-listsFetched
<-alertsSent <-alertsSent
err = files.WriteAll() err = files.WriteAll()
internal/smallwebwaf/smallwebwaf_test.go
+2 -156
View File
@@ -38,7 +38,6 @@ const (
stateWriteDelay = "SWWAF_STATE_WRITE_DELAY" stateWriteDelay = "SWWAF_STATE_WRITE_DELAY"
stateCounterInterval = "SWWAF_STATE_COUNTER_INTERVAL" stateCounterInterval = "SWWAF_STATE_COUNTER_INTERVAL"
rateLimitPerDay = "SWWAF_RATE_LIMIT_PER_DAY" rateLimitPerDay = "SWWAF_RATE_LIMIT_PER_DAY"
rateLimitExemptNets = "SWWAF_RATE_LIMIT_EXEMPT_NETS"
rulesDir = "SWWAF_RULES_DIR" rulesDir = "SWWAF_RULES_DIR"
lookupSource = "SWWAF_LOOKUP_SOURCE" lookupSource = "SWWAF_LOOKUP_SOURCE"
lookupDBPath = "SWWAF_LOOKUP_DB_PATH" lookupDBPath = "SWWAF_LOOKUP_DB_PATH"
@@ -249,54 +248,6 @@ func TestServesUntilToldToStop(t *testing.T) {
out.line(t, "msg", "stopped") out.line(t, "msg", "stopped")
} }
func TestLogLevelHoldsBackTheLessSevereProcessLines(t *testing.T) {
t.Parallel()
// A list that cannot be fetched has a warning written once smallwebwaf
// serves, after its starting line.
lists := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusServiceUnavailable)
}))
t.Cleanup(lists.Close)
ctx, stop := context.WithCancel(t.Context())
out := &output{}
exited := make(chan int, 1)
go func() {
exited <- run(ctx, map[string]string{
listenAddr: localhost + ":0",
stateDir: t.TempDir(),
rulesDir: t.TempDir(),
"SWWAF_BLOCKLIST_URLS": lists.URL + "/tor.txt",
"SWWAF_LOG_LEVEL": "warn",
}, out)
}()
out.line(t, "msg", "fetching a list failed")
stop()
select {
case status := <-exited:
if status != 0 {
t.Fatalf("exit status %d, want 0; output:\n%s", status, out.text())
}
case <-time.After(waitLimit):
t.Fatal("still running after being told to stop")
}
// Not one of the info lines from the start to the stop.
for line := range strings.Lines(out.text()) {
var fields map[string]any
err := json.Unmarshal([]byte(line), &fields)
if err != nil || fields["level"] == "INFO" {
t.Errorf("line %q (%v), want none at info", line, err)
}
}
}
func TestEveryLogLineAndMetricCarriesTheInstanceName(t *testing.T) { func TestEveryLogLineAndMetricCarriesTheInstanceName(t *testing.T) {
t.Parallel() t.Parallel()
@@ -571,7 +522,7 @@ func TestLookupDatabaseReplacedWhileRunningTakesEffect(t *testing.T) {
// The requests sent until a replacement takes effect, and those // The requests sent until a replacement takes effect, and those
// for the metrics, must not break a rate limit, whose ban would // for the metrics, must not break a rate limit, whose ban would
// refuse them too. // refuse them too.
rateLimitExemptNets: placed + "," + localhost, "SWWAF_RATE_LIMIT_EXEMPT_NETS": placed + "," + localhost,
} }
began := time.Now() began := time.Now()
// Each replacement is written beside the file and renamed over it, as // Each replacement is written beside the file and renamed over it, as
@@ -622,111 +573,6 @@ func TestLookupDatabaseReplacedWhileRunningTakesEffect(t *testing.T) {
} }
} }
func TestBlocklistTriesAndCopiesKeptInReputationJSONAcrossRestarts(t *testing.T) {
t.Parallel()
const (
token = "0123456789abcdef0123456789abcdef"
torPath = "/tor.txt"
dropPath = "/drop.txt"
// copyright is the DROP list's date and copyright line.
copyright = "; Spamhaus DROP List 2026/10/07 - (c) 2026 The Spamhaus Project SLL"
)
failing, torFetches := new(atomic.Bool), new(atomic.Int32)
lists := map[string]string{
torPath: "198.51.100.0/24\n",
dropPath: copyright + "\n" + placed + " ; SBL1\n",
}
server := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == torPath {
torFetches.Add(1)
}
if failing.Load() {
w.WriteHeader(http.StatusServiceUnavailable)
return
}
_, _ = io.WriteString(w, lists[r.URL.Path])
}))
t.Cleanup(server.Close)
torURL, dropURL := server.URL+torPath, server.URL+dropPath
dir := t.TempDir()
env := map[string]string{
listenAddr: localhost + ":0",
upstreamURL: startApp(t),
stateDir: dir,
rulesDir: t.TempDir(),
trustedProxies: localhost + "/32",
metricsToken: token,
instanceName: instance,
"SWWAF_BLOCKLIST_URLS": torURL,
// The requests sent until the list takes effect, and those for the
// metrics, must not break a rate limit, whose ban would refuse them
// too.
rateLimitExemptNets: placed + "," + localhost,
}
failures := `smallwebwaf_reputation_failures_total{instance="fsn1app1/gitea",` +
`source="` + torURL + `"} `
// tor.txt cannot be fetched at first, which is counted.
failing.Store(true)
runUntilStopped(t, env, func(url string) {
metricsWith(t, url+"_smallwebwaf/metrics", token, failures+"1")
})
// Restarted with drop.txt named after it and the server answering, tor.txt
// waits SWWAF_BLOCKLIST_REFRESH after its failed try, kept in reputation.json,
// while drop.txt, never tried, is fetched at once. Lists are fetched in the
// order named, so once drop.txt refuses the client, tor.txt has had its turn.
failing.Store(false)
env["SWWAF_BLOCKLIST_URLS"] = torURL + "," + dropURL
out := runUntilStopped(t, env, func(url string) {
for statusFrom(t, url, placed) != http.StatusForbidden {
time.Sleep(pollInterval)
}
})
wantDeniedByList(t, out.line(t, "action", "denied"), dropURL)
if fetches := torFetches.Load(); fetches != 1 {
t.Errorf("tor.txt fetched %d times, want once, before the restart", fetches)
}
// After another restart, with the server failing, the copy of drop.txt
// kept in reputation.json, its copyright line included, refuses the
// client from the first request.
failing.Store(true)
out = runUntilStopped(t, env, func(url string) {
wantStatus(t, url, placed, http.StatusForbidden)
})
wantDeniedByList(t, out.line(t, "type", "request"), dropURL)
path := filepath.Join(dir, "reputation.json")
kept, err := os.ReadFile(path) //nolint:gosec // a file in the test's directory
if err != nil || !strings.Contains(string(kept), `"`+copyright+`"`) {
t.Errorf("reputation.json holds\n%s\nwant the copy with %q (%v)", kept, copyright,
err)
}
}
// wantDeniedByList checks that the request log line is of a request the
// blocklist at listURL refused.
func wantDeniedByList(t *testing.T, line map[string]any, listURL string) {
t.Helper()
reputation, _ := line["reputation"].([]any)
if line["action"] != "denied" || len(reputation) != 1 || reputation[0] != listURL {
t.Errorf("request log line %v, want one denied for %s", line, listURL)
}
}
func TestLookupDatabaseThatCannotBeReadStopsTheStart(t *testing.T) { func TestLookupDatabaseThatCannotBeReadStopsTheStart(t *testing.T) {
t.Parallel() t.Parallel()
@@ -1168,7 +1014,7 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL, dir string) {
"SWWAF_REQUEST_MAX_BYTES": "100M", "SWWAF_REQUEST_MAX_BYTES": "100M",
"SWWAF_RESPONSE_MAX_BYTES": "5G", "SWWAF_RESPONSE_MAX_BYTES": "5G",
"SWWAF_ALLOW_NETS": "", "SWWAF_ALLOW_NETS": "",
rateLimitExemptNets: "", "SWWAF_RATE_LIMIT_EXEMPT_NETS": "",
"SWWAF_DENY_NETS": "", "SWWAF_DENY_NETS": "",
"SWWAF_RATE_LIMIT_PER_MINUTE": "1000", "SWWAF_RATE_LIMIT_PER_MINUTE": "1000",
"SWWAF_RATE_LIMIT_PER_HOUR": "10000", "SWWAF_RATE_LIMIT_PER_HOUR": "10000",
internal/state/state.go
+54 -190
View File
@@ -1,11 +1,9 @@
// Package state keeps smallwebwaf's state in JSON files in // Package state keeps smallwebwaf's state in JSON files in
// SWWAF_STATE_DIR, as the "Persistent state" section of SPEC.md describes: // SWWAF_STATE_DIR, as the "Persistent state" section of SPEC.md describes:
// bans.json holds the bans, clients.json each client's counters and // bans.json holds the bans, clients.json each client's counters and
// history, lookups.json GeoJS's answers, reputation.json the last try and // history, lookups.json GeoJS's answers, and alerts.json the cooldowns,
// last good copy of each list fetched from a URL, the DNSBL zones' // the hour under way, the alerts waiting for each destination and the
// verdicts, and AbuseIPDB's scores and checks spent, and alerts.json the // anomaly counters. Load
// cooldowns, the hour under way, the alerts waiting for each destination
// and the anomaly counters. Load
// reads them at start, Watch takes in an admin's edit of one while // reads them at start, Watch takes in an admin's edit of one while
// smallwebwaf runs, and Run and WriteAll write them. The disk is read and // smallwebwaf runs, and Run and WriteAll write them. The disk is read and
// written outside the parts' locks, which are held only to take a // written outside the parts' locks, which are held only to take a
@@ -37,7 +35,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/lookup" "sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/metrics" "sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation"
) )
// version is the version of the files' format, the only one read. // version is the version of the files' format, the only one read.
@@ -49,11 +46,10 @@ const fileMode = 0o600
// The state files' names. // The state files' names.
const ( const (
bansJSON = "bans.json" bansJSON = "bans.json"
clientsJSON = "clients.json" clientsJSON = "clients.json"
lookupsJSON = "lookups.json" lookupsJSON = "lookups.json"
reputationJSON = "reputation.json" alertsJSON = "alerts.json"
alertsJSON = "alerts.json"
) )
var ( var (
@@ -77,15 +73,12 @@ type Params struct {
// is (SWWAF_STATE_COUNTER_INTERVAL). // is (SWWAF_STATE_COUNTER_INTERVAL).
WriteDelay time.Duration WriteDelay time.Duration
CounterInterval time.Duration CounterInterval time.Duration
// Ledger, Limiter, GeoJS, Lists, DNSBL, AbuseIPDB, Alerts and Anomalies // Ledger, Limiter, GeoJS, Alerts and Anomalies hold the state. Alerts
// hold the state. Alerts also receive a file_error alert for an edit set // also receive a file_error alert for an edit set aside, and for a
// aside, and for a write that fails while smallwebwaf runs. // write that fails while smallwebwaf runs.
Ledger *bans.Ledger Ledger *bans.Ledger
Limiter *ratelimit.Limiter Limiter *ratelimit.Limiter
GeoJS *lookup.GeoJS GeoJS *lookup.GeoJS
Lists *reputation.Lists
DNSBL *reputation.DNSBL
AbuseIPDB *reputation.AbuseIPDB
Alerts *alerts.Queue Alerts *alerts.Queue
Anomalies *anomaly.Counters Anomalies *anomaly.Counters
// Now tells the time by which the counters' buckets run out, normally // Now tells the time by which the counters' buckets run out, normally
@@ -145,15 +138,6 @@ type lookupsFile struct {
Lookups []lookup.Answer `json:"lookups"` Lookups []lookup.Answer `json:"lookups"`
} }
// reputationFile is reputation.json, indented for an admin to read and
// edit, so that each line of a list's copy is on a line of its own.
type reputationFile struct {
Version int `json:"version"`
Lists []reputation.List `json:"lists"`
Verdicts []reputation.Verdict `json:"verdicts"`
AbuseIPDB reputation.Checks `json:"abuseipdb"`
}
// alertsFile is alerts.json, indented for an admin to read and edit. // alertsFile is alerts.json, indented for an admin to read and edit.
// //
//nolint:tagliatelle // the state files use snake_case, as the request log does //nolint:tagliatelle // the state files use snake_case, as the request log does
@@ -175,10 +159,10 @@ type stateFile interface {
} }
// Load checks that files can be written in Dir, and reads the state files // Load checks that files can be written in Dir, and reads the state files
// in it into the parts of Params that hold the state. A missing file is // in it into the ledger, the limiter and GeoJS. A missing file is empty
// empty state, as on a first start. A file that does not parse, has an // state, as on a first start. A file that does not parse, has an unknown
// unknown version, or has an entry without a field it needs, is an error // version, or has an entry without a field it needs, is an error that
// that names the file and, where the JSON decoder tells it, the line and // names the file and, where the JSON decoder tells it, the line and
// column, or else the entry. // column, or else the entry.
func Load(params Params) (*Files, error) { func Load(params Params) (*Files, error) {
err := checkWritable(params.Dir) err := checkWritable(params.Dir)
@@ -191,17 +175,16 @@ func Load(params Params) (*Files, error) {
bansRead, bansErr := f.read(bansJSON) bansRead, bansErr := f.read(bansJSON)
clientsRead, clientsErr := f.read(clientsJSON) clientsRead, clientsErr := f.read(clientsJSON)
lookupsRead, lookupsErr := f.read(lookupsJSON) lookupsRead, lookupsErr := f.read(lookupsJSON)
reputationRead, reputationErr := f.read(reputationJSON)
alertsRead, alertsErr := f.read(alertsJSON) alertsRead, alertsErr := f.read(alertsJSON)
err = errors.Join(bansErr, clientsErr, lookupsErr, reputationErr, alertsErr) err = errors.Join(bansErr, clientsErr, lookupsErr, alertsErr)
if err != nil { if err != nil {
return nil, err return nil, err
} }
params.ProcessLog.Info("read the state files", "directory", params.Dir, params.ProcessLog.Info("read the state files", "directory", params.Dir,
"bans", bansRead, "clients", clientsRead, "lookups", lookupsRead, "bans", bansRead, "clients", clientsRead, "lookups", lookupsRead,
"lists", reputationRead, "alerts_waiting", alertsRead) "alerts_waiting", alertsRead)
return f, nil return f, nil
} }
@@ -230,9 +213,7 @@ func (f *Files) Run(ctx context.Context) {
f.logFailure(bansJSON, f.writeFile(bansJSON)) f.logFailure(bansJSON, f.writeFile(bansJSON))
case <-interval.C: case <-interval.C:
for _, name := range []string{ for _, name := range []string{bansJSON, clientsJSON, lookupsJSON, alertsJSON} {
bansJSON, clientsJSON, lookupsJSON, reputationJSON, alertsJSON,
} {
f.logFailure(name, f.writeFile(name)) f.logFailure(name, f.writeFile(name))
} }
} }
@@ -243,7 +224,7 @@ func (f *Files) Run(ctx context.Context) {
// fails does not keep the others from being written. // fails does not keep the others from being written.
func (f *Files) WriteAll() error { func (f *Files) WriteAll() error {
return errors.Join(f.writeFile(bansJSON), f.writeFile(clientsJSON), return errors.Join(f.writeFile(bansJSON), f.writeFile(clientsJSON),
f.writeFile(lookupsJSON), f.writeFile(reputationJSON), f.writeFile(alertsJSON)) f.writeFile(lookupsJSON), f.writeFile(alertsJSON))
} }
// Watch watches Dir until ctx is done, and takes in an admin's edit of a // Watch watches Dir until ctx is done, and takes in an admin's edit of a
@@ -278,7 +259,7 @@ func (f *Files) Watch(ctx context.Context) {
return return
case event := <-watcher.Events: case event := <-watcher.Events:
switch name := filepath.Base(event.Name); name { switch name := filepath.Base(event.Name); name {
case bansJSON, clientsJSON, lookupsJSON, reputationJSON, alertsJSON: case bansJSON, clientsJSON, lookupsJSON, alertsJSON:
f.fileChanged(name) f.fileChanged(name)
} }
case err = <-watcher.Errors: case err = <-watcher.Errors:
@@ -424,29 +405,33 @@ func (f *Files) takeIn(name string, data []byte, edit bool) (int, error) {
f.params.GeoJS.Load(file.Lookups) f.params.GeoJS.Load(file.Lookups)
entries = len(file.Lookups) entries = len(file.Lookups)
case reputationJSON: case alertsJSON:
var file reputationFile // waiting was a list, of the alerts waiting for the webhook, before
// alerts went to Slack and ntfy too.
var written struct {
Waiting json.RawMessage `json:"waiting"`
}
if json.Unmarshal(data, &written) == nil &&
bytes.HasPrefix(written.Waiting, []byte("[")) {
return 0, fmt.Errorf("%s: %w", path, errWaitingList)
}
var file alertsFile
err := parse(path, data, &file) err := parse(path, data, &file)
if err != nil { if err != nil {
return 0, err return 0, err
} }
err = f.params.Lists.Load(file.Lists) f.params.Alerts.Load(alerts.State{
if err != nil { Cooldowns: file.Cooldowns, Hour: file.Hour, Waiting: file.Waiting,
return 0, fmt.Errorf("%s: %w", path, err) })
} f.params.Anomalies.Load(file.AnomalyCounters, f.params.Now())
f.params.DNSBL.Load(file.Verdicts) for _, waiting := range file.Waiting {
f.params.AbuseIPDB.Load(file.AbuseIPDB) entries += len(waiting)
entries = len(file.Lists)
case alertsJSON:
waiting, err := f.takeInAlerts(path, data)
if err != nil {
return 0, err
} }
entries = waiting
} }
f.sums[name] = sha256.Sum256(data) f.sums[name] = sha256.Sum256(data)
@@ -454,41 +439,6 @@ func (f *Files) takeIn(name string, data []byte, edit bool) (int, error) {
return entries, nil return entries, nil
} }
// takeInAlerts parses data, what alerts.json, at path, holds, puts it
// into the alerts and the anomaly counters, in place of what they held,
// and returns how many alerts wait in it, as takeIn describes.
func (f *Files) takeInAlerts(path string, data []byte) (int, error) {
// waiting was a list, of the alerts waiting for the webhook, before
// alerts went to Slack and ntfy too.
var written struct {
Waiting json.RawMessage `json:"waiting"`
}
if json.Unmarshal(data, &written) == nil &&
bytes.HasPrefix(written.Waiting, []byte("[")) {
return 0, fmt.Errorf("%s: %w", path, errWaitingList)
}
var file alertsFile
err := parse(path, data, &file)
if err != nil {
return 0, err
}
f.params.Alerts.Load(alerts.State{
Cooldowns: file.Cooldowns, Hour: file.Hour, Waiting: file.Waiting,
})
f.params.Anomalies.Load(file.AnomalyCounters, f.params.Now())
entries := 0
for _, waiting := range file.Waiting {
entries += len(waiting)
}
return entries, nil
}
// writeFile writes the state file name from what smallwebwaf holds. An // writeFile writes the state file name from what smallwebwaf holds. An
// edit made since smallwebwaf last read or wrote the file is taken in // edit made since smallwebwaf last read or wrote the file is taken in
// first, so that it is not overwritten, or set aside if it does not // first, so that it is not overwritten, or set aside if it does not
@@ -564,37 +514,32 @@ func (f *Files) setAside(name string, parseErr error) error {
func (f *Files) encode(name string) ([]byte, error) { func (f *Files) encode(name string) ([]byte, error) {
switch name { switch name {
case bansJSON: case bansJSON:
return encodeIndented(bansFile{ file := bansFile{Version: version, Bans: BanEntries(f.params.Ledger.Snapshot())}
Version: version, Bans: BanEntries(f.params.Ledger.Snapshot()),
}) data, err := json.MarshalIndent(file, "", " ")
if err != nil {
return nil, err
}
return append(data, '\n'), nil
case clientsJSON: case clientsJSON:
return encodeOnePerLine("clients", f.params.Limiter.Snapshot()) return encodeOnePerLine("clients", f.params.Limiter.Snapshot())
case lookupsJSON: case lookupsJSON:
return encodeOnePerLine("lookups", f.params.GeoJS.Snapshot()) return encodeOnePerLine("lookups", f.params.GeoJS.Snapshot())
case reputationJSON:
return encodeIndented(reputationFile{
Version: version, Lists: f.params.Lists.Snapshot(),
Verdicts: f.params.DNSBL.Snapshot(), AbuseIPDB: f.params.AbuseIPDB.Snapshot(),
})
default: // alerts.json default: // alerts.json
held := f.params.Alerts.Snapshot() held := f.params.Alerts.Snapshot()
file := alertsFile{
return encodeIndented(alertsFile{
Version: version, Cooldowns: held.Cooldowns, Hour: held.Hour, Version: version, Cooldowns: held.Cooldowns, Hour: held.Hour,
Waiting: held.Waiting, AnomalyCounters: f.params.Anomalies.Snapshot(), Waiting: held.Waiting, AnomalyCounters: f.params.Anomalies.Snapshot(),
}) }
}
}
// encodeIndented encodes file, a state file's struct, indented for an data, err := json.MarshalIndent(file, "", " ")
// admin to read and edit. if err != nil {
func encodeIndented(file any) ([]byte, error) { return nil, err
data, err := json.MarshalIndent(file, "", " ") }
if err != nil {
return nil, err
}
return append(data, '\n'), nil return append(data, '\n'), nil
}
} }
// BanEntries returns held as bans.json lists them, an empty list for // BanEntries returns held as bans.json lists them, an empty list for
@@ -734,87 +679,6 @@ func (f *lookupsFile) check(data []byte) error {
return nil return nil
} }
// check refuses a list without its URL, which would name no list, or the
// time it was last tried, which would have it fetched at once, and a copy
// of it without the time it was fetched, or without its lines, which hold
// the list. It refuses a verdict without its zone or its client, which
// would be about no one, whether the zone lists the client, or the time
// it was fetched, which would drop it, and so an AbuseIPDB score without
// its client, the score, or the time it was fetched. A verdict's listed is
// false for a client the zone does not list, and a score can be 0, which
// the structs cannot tell from a missing one, so each is read again as
// written.
func (f *reputationFile) check(data []byte) error {
for i, kept := range f.Lists {
switch {
case kept.URL == "":
return missing(i, "url")
case kept.Tried.IsZero():
return missing(i, "tried")
case kept.Fetched.IsZero() && kept.Lines != nil:
return missing(i, "fetched")
case kept.Lines == nil && !kept.Fetched.IsZero():
return missing(i, "lines")
}
}
var written struct {
Verdicts []struct {
Listed *bool `json:"listed"`
} `json:"verdicts"`
}
err := json.Unmarshal(data, &written)
if err != nil {
return err
}
for i, verdict := range f.Verdicts {
switch {
case verdict.Zone == "":
return fmt.Errorf("verdicts %w", missing(i, "zone"))
case !verdict.Client.IsValid():
return fmt.Errorf("verdicts %w", missing(i, "client"))
case written.Verdicts[i].Listed == nil:
return fmt.Errorf("verdicts %w", missing(i, "listed"))
case verdict.Fetched.IsZero():
return fmt.Errorf("verdicts %w", missing(i, "fetched"))
}
}
return checkScores(f.AbuseIPDB.Scores, data)
}
// checkScores refuses an AbuseIPDB score, of scores, read from data, as
// reputationFile's check describes.
func checkScores(scores []reputation.Score, data []byte) error {
var written struct {
AbuseIPDB struct {
Scores []struct {
Score *int64 `json:"score"`
} `json:"scores"`
} `json:"abuseipdb"`
}
err := json.Unmarshal(data, &written)
if err != nil {
return err
}
for i, kept := range scores {
switch {
case !kept.Client.IsValid():
return fmt.Errorf("abuseipdb scores %w", missing(i, "client"))
case written.AbuseIPDB.Scores[i].Score == nil:
return fmt.Errorf("abuseipdb scores %w", missing(i, "score"))
case kept.Fetched.IsZero():
return fmt.Errorf("abuseipdb scores %w", missing(i, "fetched"))
}
}
return nil
}
// check refuses a cooldown without its event or when its alert was sent, // check refuses a cooldown without its event or when its alert was sent,
// which would hold back no repeat, alerts waiting for a destination with // which would hold back no repeat, alerts waiting for a destination with
// another name than webhook, slack or ntfy, most likely misspelt, an // another name than webhook, slack or ntfy, most likely misspelt, an
internal/state/state_test.go
+20 -313
View File
@@ -27,22 +27,15 @@ import (
"sneak.berlin/go/smallwebwaf/internal/lookup" "sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/metrics" "sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/state" "sneak.berlin/go/smallwebwaf/internal/state"
) )
const ( const (
// The state files. // The state files.
bansJSON = "bans.json" bansJSON = "bans.json"
clientsJSON = "clients.json" clientsJSON = "clients.json"
lookupsJSON = "lookups.json" lookupsJSON = "lookups.json"
reputationJSON = "reputation.json" alertsJSON = "alerts.json"
alertsJSON = "alerts.json"
// blocklistURL and torURL are the blocklists the tests' lists name, and
// dnsblZone the DNSBL zone of the tests' verdicts.
blocklistURL = "https://lists.example/drop.txt"
torURL = "https://lists.example/tor.txt"
dnsblZone = "dnsbl.example"
// The AS number and AS name the tests' clients are looked up in. // The AS number and AS name the tests' clients are looked up in.
asn = "AS64496" asn = "AS64496"
asName = "Example Net" asName = "Example Net"
@@ -211,61 +204,6 @@ const filledAlertsJSON = `{
} }
` `
// filledReputationJSON is reputation.json holding the blocklists' last
// tries and the copy of one, with its comment line, two verdicts of a
// DNSBL zone, and the AbuseIPDB checks spent today with two scores, as
// fill puts them in.
const filledReputationJSON = `{
"version": 1,
"lists": [
{
"url": "https://lists.example/drop.txt",
"tried": "2026-10-06T00:00:00Z",
"fetched": "2026-10-05T23:00:00Z",
"lines": [
"; Spamhaus DROP List 2026/10/05 - (c) 2026 The Spamhaus Project SLL",
"203.0.113.0/24 ; SBL1",
"2001:db8::/32 ; SBL2"
]
},
{
"url": "https://lists.example/tor.txt",
"tried": "2026-10-06T00:00:00Z"
}
],
"verdicts": [
{
"zone": "dnsbl.example",
"client": "203.0.113.9",
"listed": true,
"fetched": "2026-10-05T23:00:00Z"
},
{
"zone": "dnsbl.example",
"client": "2001:db8::1",
"listed": false,
"fetched": "2026-10-05T22:00:00Z"
}
],
"abuseipdb": {
"day": "2026-10-06T00:00:00Z",
"spent": 3,
"scores": [
{
"client": "203.0.113.9/32",
"score": 100,
"fetched": "2026-10-05T23:00:00Z"
},
{
"client": "2001:db8::/64",
"score": 0,
"fetched": "2026-10-05T22:00:00Z"
}
]
}
}
`
func TestFilesWrittenAndReadBack(t *testing.T) { func TestFilesWrittenAndReadBack(t *testing.T) {
t.Parallel() t.Parallel()
@@ -292,18 +230,6 @@ func TestFilesWrittenAndReadBack(t *testing.T) {
wantEqual(t, clientsJSON, after.Limiter.Snapshot(), before.Limiter.Snapshot()) wantEqual(t, clientsJSON, after.Limiter.Snapshot(), before.Limiter.Snapshot())
wantEqual(t, lookupsJSON, after.GeoJS.Snapshot(), before.GeoJS.Snapshot()) wantEqual(t, lookupsJSON, after.GeoJS.Snapshot(), before.GeoJS.Snapshot())
if got, want := after.Lists.Snapshot(), before.Lists.Snapshot(); !reflect.DeepEqual(
got, want) {
t.Errorf("%s read back\n%+v\nwant\n%+v", reputationJSON, got, want)
}
wantEqual(t, reputationJSON, after.DNSBL.Snapshot(), before.DNSBL.Snapshot())
checks, wantChecks := after.AbuseIPDB.Snapshot(), before.AbuseIPDB.Snapshot()
if !reflect.DeepEqual(checks, wantChecks) {
t.Errorf("%s read back\n%+v\nwant\n%+v", reputationJSON, checks, wantChecks)
}
if got, want := after.Alerts.Snapshot(), before.Alerts.Snapshot(); !reflect.DeepEqual( if got, want := after.Alerts.Snapshot(), before.Alerts.Snapshot(); !reflect.DeepEqual(
got, want) { got, want) {
t.Errorf("%s read back\n%+v\nwant\n%+v", alertsJSON, got, want) t.Errorf("%s read back\n%+v\nwant\n%+v", alertsJSON, got, want)
@@ -312,32 +238,12 @@ func TestFilesWrittenAndReadBack(t *testing.T) {
wantEqual(t, alertsJSON, after.Anomalies.Snapshot(), before.Anomalies.Snapshot()) wantEqual(t, alertsJSON, after.Anomalies.Snapshot(), before.Anomalies.Snapshot())
// Each one-per-line file lists its entries by client, and nothing // Each one-per-line file lists its entries by client, and nothing
// but the five files is left in the directory. // but the four files is left in the directory.
wantEntries(t, filepath.Join(dir, clientsJSON), "clients", wantEntries(t, filepath.Join(dir, clientsJSON), "clients",
"192.0.2.1/32", "203.0.113.9/32", "2001:db8::/64") "192.0.2.1/32", "203.0.113.9/32", "2001:db8::/64")
wantEntries(t, filepath.Join(dir, lookupsJSON), "lookups", wantEntries(t, filepath.Join(dir, lookupsJSON), "lookups",
"192.0.2.1/32", "203.0.113.9/32") "192.0.2.1/32", "203.0.113.9/32")
wantFiles(t, dir, alertsJSON, bansJSON, clientsJSON, lookupsJSON, reputationJSON) wantFiles(t, dir, alertsJSON, bansJSON, clientsJSON, lookupsJSON)
}
func TestReputationJSONKeepsEachCopyWholeOneLineToALine(t *testing.T) {
t.Parallel()
dir := t.TempDir()
params := newParams(dir)
fill(params)
files := load(t, params)
err := files.WriteAll()
if err != nil {
t.Fatalf("write: %v", err)
}
got := readFile(t, filepath.Join(dir, reputationJSON))
if got != filledReputationJSON {
t.Errorf("reputation.json\n%s\nwant\n%s", got, filledReputationJSON)
}
} }
func TestAlertsJSONIsIndentedWithTheCooldownsTheHourAndTheAlertsWaiting(t *testing.T) { func TestAlertsJSONIsIndentedWithTheCooldownsTheHourAndTheAlertsWaiting(t *testing.T) {
@@ -461,13 +367,9 @@ func TestMissingFilesAreEmptyState(t *testing.T) {
load(t, params) load(t, params)
held := params.Alerts.Snapshot() held := params.Alerts.Snapshot()
checks := params.AbuseIPDB.Snapshot()
if len(params.Ledger.Snapshot()) != 0 || len(params.Limiter.Snapshot()) != 0 || if len(params.Ledger.Snapshot()) != 0 || len(params.Limiter.Snapshot()) != 0 ||
len(params.GeoJS.Snapshot()) != 0 || len(params.Lists.Snapshot()) != 0 || len(params.GeoJS.Snapshot()) != 0 || len(held.Cooldowns) != 0 ||
len(params.DNSBL.Snapshot()) != 0 || len(checks.Scores) != 0 || checks.Spent != 0 || len(held.Waiting[alerts.DestinationWebhook]) != 0 || held.Hour.Sent != 0 {
len(held.Cooldowns) != 0 || len(held.Waiting[alerts.DestinationWebhook]) != 0 ||
held.Hour.Sent != 0 {
t.Error("state from no files") t.Error("state from no files")
} }
} }
@@ -526,14 +428,6 @@ func TestFileThatDoesNotParseStopsTheStart(t *testing.T) {
`: anomaly_counters entry 2's scope "nett" is not client, net, asn, total ` + `: anomaly_counters entry 2's scope "nett" is not client, net, asn, total ` +
`or watch`, `or watch`,
}, },
{
"a copy of a list with a line that does not read", reputationJSON,
`{"version": 1, "lists": [{"url": "` + blocklistURL + `", ` +
`"tried": "2026-10-06T00:00:00Z", "fetched": "2026-10-06T00:00:00Z", ` +
`"lines": ["; DROP", "203.0.113.300"]}]}`,
`: the copy of ` + blocklistURL + `: line 2 is not an address or a netblock, ` +
`such as 192.0.2.0/24`,
},
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
t.Parallel() t.Parallel()
@@ -629,121 +523,6 @@ func TestEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
} }
} }
func TestReputationJSONEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
t.Parallel()
const (
drop = `"url": "` + blocklistURL + `", `
tried = `"tried": "2026-10-06T00:00:00Z", `
fetched = `"fetched": "2026-10-06T00:00:00Z"`
// verdictZone, verdictClient and listed start a verdict, which
// fetched ends.
verdictZone = `"zone": "` + dnsblZone + `", `
verdictClient = `"client": "198.51.100.7", `
listed = `"listed": false, `
)
for _, tc := range []struct {
name, content string
// want is what the error says after the file's path.
want string
}{
{
"a list without its URL",
`{"version": 1, "lists": [{` + tried + fetched + `, "lines": []}]}`,
`: entry 1 has no "url"`,
},
{
"a list without the time it was last tried",
`{"version": 1, "lists": [{` + drop + fetched + `, "lines": []}]}`,
`: entry 1 has no "tried"`,
},
{
"a copy of a list without the time it was fetched",
`{"version": 1, "lists": [{` + drop + tried + `"lines": []}]}`,
`: entry 1 has no "fetched"`,
},
{
// An empty list has no lines, which is not having none.
"a copy of a list without its lines",
`{"version": 1, "lists": [{` + drop + tried + fetched + `, "lines": []}, ` +
`{"url": "` + torURL + `", ` + tried + fetched + `}]}`,
`: entry 2 has no "lines"`,
},
{
"a verdict without its zone",
`{"version": 1, "verdicts": [{` + verdictClient + listed + fetched + `}]}`,
`: verdicts entry 1 has no "zone"`,
},
{
"a verdict without its client",
`{"version": 1, "verdicts": [{` + verdictZone + listed + fetched + `}]}`,
`: verdicts entry 1 has no "client"`,
},
{
// A client the zone does not list has a listed of false, which is
// not having none.
"a verdict without whether the zone lists the client",
`{"version": 1, "verdicts": [{` + verdictZone + verdictClient + listed +
fetched + `}, {` + verdictZone + verdictClient + fetched + `}]}`,
`: verdicts entry 2 has no "listed"`,
},
{
"a verdict without the time it was fetched",
`{"version": 1, "verdicts": [{` + verdictZone + verdictClient +
`"listed": true}]}`,
`: verdicts entry 1 has no "fetched"`,
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
wantRefused(t, reputationJSON, tc.content, tc.want)
})
}
}
func TestReputationJSONScoreWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
t.Parallel()
// scores opens the list of AbuseIPDB scores, and ends closes it; client,
// score and fetched make a score.
const (
scores = `{"version": 1, "abuseipdb": {"scores": [`
client = `"client": "198.51.100.7/32", `
score = `"score": 0, `
fetched = `"fetched": "2026-10-06T00:00:00Z"`
ends = `}]}}`
)
for _, tc := range []struct {
name, content string
// want is what the error says after the file's path.
want string
}{
{
"without its client", scores + `{` + score + fetched + ends,
`: abuseipdb scores entry 1 has no "client"`,
},
{
// A score of 0 is not having none.
"without the score",
scores + `{` + client + score + fetched + `}, {` + client + fetched + ends,
`: abuseipdb scores entry 2 has no "score"`,
},
{
"without the time it was fetched", scores + `{` + client + `"score": 100` + ends,
`: abuseipdb scores entry 1 has no "fetched"`,
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
wantRefused(t, reputationJSON, tc.content, tc.want)
})
}
}
func TestAlertsJSONEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) { func TestAlertsJSONEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
t.Parallel() t.Parallel()
@@ -847,9 +626,7 @@ func TestBanWithAnotherCauseStopsTheStart(t *testing.T) {
func TestUnknownVersionStopsTheStart(t *testing.T) { func TestUnknownVersionStopsTheStart(t *testing.T) {
t.Parallel() t.Parallel()
for _, file := range []string{ for _, file := range []string{bansJSON, clientsJSON, lookupsJSON, alertsJSON} {
bansJSON, clientsJSON, lookupsJSON, reputationJSON, alertsJSON,
} {
for _, content := range []string{`{"version": 2}`, `{}`} { for _, content := range []string{`{"version": 2}`, `{}`} {
t.Run(file+" "+content, func(t *testing.T) { t.Run(file+" "+content, func(t *testing.T) {
t.Parallel() t.Parallel()
@@ -951,9 +728,8 @@ func TestEveryFileWrittenEveryCounterInterval(t *testing.T) {
time.Sleep(time.Nanosecond) time.Sleep(time.Nanosecond)
synctest.Wait() synctest.Wait()
wantFiles(t, dir, alertsJSON, bansJSON, clientsJSON, lookupsJSON, reputationJSON) wantFiles(t, dir, alertsJSON, bansJSON, clientsJSON, lookupsJSON)
removeFiles(t, dir, alertsJSON, bansJSON, clientsJSON, lookupsJSON, removeFiles(t, dir, alertsJSON, bansJSON, clientsJSON, lookupsJSON)
reputationJSON)
} }
}) })
} }
@@ -1172,7 +948,7 @@ func TestFileThatCannotBeReadIsNotWrittenOver(t *testing.T) {
t.Errorf("bans.json is now %v (%v), want the socket", info, err) t.Errorf("bans.json is now %v (%v), want the socket", info, err)
} }
wantFiles(t, dir, alertsJSON, bansJSON, clientsJSON, lookupsJSON, reputationJSON) wantFiles(t, dir, alertsJSON, bansJSON, clientsJSON, lookupsJSON)
wantWriteFailed(t, params, bansJSON) wantWriteFailed(t, params, bansJSON)
} }
@@ -1245,32 +1021,6 @@ func TestEditOfEachFileTakenIn(t *testing.T) {
wantEqual(t, lookupsJSON, params.GeoJS.Snapshot(), wantEqual(t, lookupsJSON, params.GeoJS.Snapshot(),
[]lookup.Answer{{Client: client, Country: "FR", Answered: midnight()}}) []lookup.Answer{{Client: client, Country: "FR", Answered: midnight()}})
edit(t, dir, reputationJSON, `{"version": 1, "lists": [{"url": "`+blocklistURL+`", `+
`"tried": "2026-10-06T00:00:00Z", "fetched": "2026-10-06T00:00:00Z", `+
`"lines": ["198.51.100.7"]}], "verdicts": [{"zone": "`+dnsblZone+`", `+
`"client": "198.51.100.7", "listed": true, "fetched": "2026-10-06T00:00:00Z"}], `+
`"abuseipdb": {"day": "2026-10-06T00:00:00Z", "spent": 9, "scores": [`+
`{"client": "198.51.100.7/32", "score": 80, "fetched": "2026-10-06T00:00:00Z"}]}}`)
wantTakenIn(t, lines, dir, reputationJSON)
listedBy := params.Lists.ListedBy(client.Addr())
if !slices.Equal(listedBy, []string{blocklistURL}) {
t.Errorf("%s taken in lists %s on %v, want on the blocklist", reputationJSON,
client.Addr(), listedBy)
}
wantEqual(t, reputationJSON, params.DNSBL.Snapshot(), []reputation.Verdict{{
Zone: dnsblZone, Client: client.Addr(), Listed: true, Fetched: midnight(),
}})
checks := reputation.Checks{
Day: midnight(), Spent: 9,
Scores: []reputation.Score{{Client: client, Score: 80, Fetched: midnight()}},
}
if got := params.AbuseIPDB.Snapshot(); !reflect.DeepEqual(got, checks) {
t.Errorf("%s taken in as\n%+v\nwant\n%+v", reputationJSON, got, checks)
}
// A netblock with bits past its length is read as the netblock it is // A netblock with bits past its length is read as the netblock it is
// in. // in.
edit(t, dir, alertsJSON, `{"version": 1, "cooldowns": [{"event": "ban", `+ edit(t, dir, alertsJSON, `{"version": 1, "cooldowns": [{"event": "ban", `+
@@ -1498,7 +1248,7 @@ func TestBrokenEditSetAsideAtTheNextWrite(t *testing.T) {
edit(t, dir, bansJSON, broken) edit(t, dir, bansJSON, broken)
edit(t, dir, clientsJSON, `{"version": 1, "clients": []}`) edit(t, dir, clientsJSON, `{"version": 1, "clients": []}`)
wantTakenIn(t, lines, dir, clientsJSON) wantTakenIn(t, lines, dir, clientsJSON)
wantFiles(t, dir, alertsJSON, bansJSON, clientsJSON, lookupsJSON, reputationJSON) wantFiles(t, dir, alertsJSON, bansJSON, clientsJSON, lookupsJSON)
// The next write sets it aside, logged with where the error is, and // The next write sets it aside, logged with where the error is, and
// writes bans.json again from what smallwebwaf still holds. // writes bans.json again from what smallwebwaf still holds.
@@ -1522,8 +1272,7 @@ func TestBrokenEditSetAsideAtTheNextWrite(t *testing.T) {
t.Errorf("alerts waiting %+v, want a file_error alert for %s", waiting, path+".bad") t.Errorf("alerts waiting %+v, want a file_error alert for %s", waiting, path+".bad")
} }
wantFiles(t, dir, alertsJSON, bansJSON, bansJSON+".bad", clientsJSON, lookupsJSON, wantFiles(t, dir, alertsJSON, bansJSON, bansJSON+".bad", clientsJSON, lookupsJSON)
reputationJSON)
if got := readFile(t, path+".bad"); got != broken { if got := readFile(t, path+".bad"); got != broken {
t.Errorf("bans.json.bad holds\n%s\nwant the edit", got) t.Errorf("bans.json.bad holds\n%s\nwant the edit", got)
@@ -1652,10 +1401,9 @@ func midnight() time.Time {
} }
// newParams returns Params for the state files in dir, with parts that // newParams returns Params for the state files in dir, with parts that
// hold nothing yet. GeoJS is never asked, the lists, two blocklists, are // hold nothing yet. GeoJS is never asked, and the alerts, at most two an
// never fetched, and the alerts, at most two an hour, are // hour, are never sent. The anomaly counters count the scopes fill
// never sent. The anomaly counters count the scopes fill counts, with // counts, with thresholds fill does not reach.
// thresholds fill does not reach.
func newParams(dir string) state.Params { func newParams(dir string) state.Params {
discard := slog.New(slog.DiscardHandler) discard := slog.New(slog.DiscardHandler)
m := metrics.New(1, "app") m := metrics.New(1, "app")
@@ -1679,22 +1427,10 @@ func newParams(dir string) state.Params {
AttackBanDuration: 7 * 24 * time.Hour, AttackBanDuration: 7 * 24 * time.Hour,
MaxBans: 5000, MaxBans: 5000,
}), }),
Limiter: ratelimit.New(ratelimit.Limits{}, 20000), Limiter: ratelimit.New(ratelimit.Limits{}),
GeoJS: lookup.New(lookup.Params{ GeoJS: lookup.New(lookup.Params{
Now: midnight, ProcessLog: discard, Metrics: m, Now: midnight, ProcessLog: discard, Metrics: m,
}), }),
Lists: reputation.New(reputation.Params{
BlocklistURLs: []string{blocklistURL, torURL}, Refresh: 24 * time.Hour,
Now: midnight, ProcessLog: discard, Alerts: queue,
}),
DNSBL: reputation.NewDNSBL(reputation.DNSBLParams{
Zones: []string{dnsblZone}, CacheTTL: 24 * time.Hour, Timeout: time.Second,
Now: midnight, ProcessLog: discard, Alerts: queue,
}),
AbuseIPDB: reputation.NewAbuseIPDB(reputation.AbuseIPDBParams{
MinScore: 75, DailyBudget: 900, CacheTTL: 24 * time.Hour, Timeout: time.Second,
Now: midnight, ProcessLog: discard, Alerts: queue,
}),
Alerts: queue, Alerts: queue,
Anomalies: anomaly.New(anomaly.Params{ Anomalies: anomaly.New(anomaly.Params{
Net: anomaly.Thresholds{RequestsPerMinute: 1000}, Net: anomaly.Thresholds{RequestsPerMinute: 1000},
@@ -1719,11 +1455,8 @@ func office() netip.Prefix {
// fill puts a permanent ban an admin made, a ban for a broken limit and // fill puts a permanent ban an admin made, a ban for a broken limit and
// one for a clear sign of attack, clients with counts and histories, // one for a clear sign of attack, clients with counts and histories,
// GeoJS answers, the blocklists' last tries and the copy of one, two // GeoJS answers, and alerts and anomaly counters, as filledAlertsJSON
// verdicts of a DNSBL zone, and the AbuseIPDB checks spent today with two // holds them, into the parts of params.
// scores, as filledReputationJSON holds them, and alerts
// and anomaly counters, as filledAlertsJSON holds them, into the parts of
// params.
func fill(params state.Params) { func fill(params state.Params) {
now := midnight() now := midnight()
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
@@ -1756,32 +1489,6 @@ func fill(params state.Params) {
}, },
}) })
// The copy of drop.txt was fetched an hour ago, and the fetches of it
// and of tor.txt tried since failed.
err := params.Lists.Load([]reputation.List{{
URL: blocklistURL, Tried: now, Fetched: now.Add(-time.Hour),
Lines: []string{
"; Spamhaus DROP List 2026/10/05 - (c) 2026 The Spamhaus Project SLL",
"203.0.113.0/24 ; SBL1",
"2001:db8::/32 ; SBL2",
},
}, {URL: torURL, Tried: now}})
if err != nil {
panic(err) // the copy reads
}
params.DNSBL.Load([]reputation.Verdict{
{
Zone: dnsblZone, Client: netip.MustParseAddr("2001:db8::1"),
Fetched: now.Add(-2 * time.Hour),
},
{Zone: dnsblZone, Client: client.Addr(), Listed: true, Fetched: now.Add(-time.Hour)},
})
params.AbuseIPDB.Load(reputation.Checks{Day: now, Spent: 3, Scores: []reputation.Score{
{Client: netip.MustParsePrefix("2001:db8::/64"), Fetched: now.Add(-2 * time.Hour)},
{Client: client, Score: 100, Fetched: now.Add(-time.Hour)},
}})
// An alert waiting, a repeat of it the cooldown holds back, another // An alert waiting, a repeat of it the cooldown holds back, another
// alert waiting, and one past the two an hour, for the hour's summary. // alert waiting, and one past the two an hour, for the hour's summary.
ban := alerts.Alert{ ban := alerts.Alert{