DNS blocklists asked in the background, verdicts kept #110
@@ -26,33 +26,36 @@ Slack and ntfy, and remote log sending. So is the stage after that: the AS
|
|||||||
number and country of every client, looked up through GeoJS or in the IPinfo
|
number and country of every client, looked up through GeoJS or in the IPinfo
|
||||||
Lite database file, the byte limits, the biased thresholds, lower limits for the
|
Lite database file, the byte limits, the biased thresholds, lower limits for the
|
||||||
AS numbers and countries you list, and the anomaly thresholds, alerts for
|
AS numbers and countries you list, and the anomaly thresholds, alerts for
|
||||||
unusual traffic that refuse nothing. So is the first part of the stage after
|
unusual traffic that refuse nothing. So are the first two parts of the stage
|
||||||
that: the blocklists you name by URL, which it fetches and keeps, and a file of
|
after that: the blocklists you name by URL, which it fetches and keeps, with a
|
||||||
AS numbers' percentages, fetched the same way. `smallwebwaf` passes each request
|
file of AS numbers' percentages fetched the same way, and the DNS blocklists
|
||||||
to the app and the app's answer back, unchanged, within its timeouts and size
|
(DNSBL zones), which it asks about each client in the background. `smallwebwaf`
|
||||||
limits, works out each client's address, looks up its AS number and country
|
passes each request to the app and the app's answer back, unchanged, within its
|
||||||
unless you switch that off, bans a client that sends too many requests or too
|
timeouts and size limits, works out each client's address, looks up its AS
|
||||||
many bytes, not counting those for the paths you choose, with lower limits for
|
number and country unless you switch that off, bans a client that sends too many
|
||||||
the clients of the AS numbers and countries you list, refuses a client that
|
requests or too many bytes, not counting those for the paths you choose, with
|
||||||
comes from a country you refuse or from a network you refuse, refuses, limits or
|
lower limits for the clients of the AS numbers and countries you list, refuses a
|
||||||
only notes a client a blocklist you name lists, lets the networks you choose
|
client that comes from a country you refuse or from a network you refuse,
|
||||||
through, checks each request against the rule files and bans a client whose
|
refuses, limits or only notes a client a blocklist or a DNSBL zone you name
|
||||||
request is a clear sign of attack, keeps its bans, each client's counters and
|
lists, lets the networks you choose through, checks each request against the
|
||||||
history, GeoJS's answers and the last good copy of each list it fetches in JSON
|
rule files and bans a client whose request is a clear sign of attack, keeps its
|
||||||
files across restarts, takes in your edits of those files, such as a ban you
|
bans, each client's counters and history, GeoJS's answers, the last good copy of
|
||||||
make, keep or lift, and of the rule files while it runs, writes a JSON log line
|
each list it fetches and the DNSBL zones' verdicts in JSON files across
|
||||||
for every request, sends its log lines to a syslog server too if you name one,
|
restarts, takes in your edits of those files, such as a ban you make, keep or
|
||||||
sends an alert to a webhook, to Slack and to ntfy, each if you name one, for
|
lift, and of the rule files while it runs, writes a JSON log line for every
|
||||||
each ban it makes or makes permanent, for traffic over an anomaly threshold you
|
request, sends its log lines to a syslog server too if you name one, sends an
|
||||||
set, for a client a blocklist lists, for GeoJS failing or a list it cannot
|
alert to a webhook, to Slack and to ntfy, each if you name one, for each ban it
|
||||||
fetch, for a rule file or state file with an error and for a replacement of the
|
makes or makes permanent, for traffic over an anomaly threshold you set, for a
|
||||||
lookup database it cannot read, serves Prometheus metrics to a scraper that
|
client a blocklist or a DNSBL zone lists, for GeoJS failing, a list it cannot
|
||||||
holds the metrics token, lets an admin who holds the admin token list, add and
|
fetch or a DNSBL zone that fails or refuses a query, for a rule file or state
|
||||||
lift bans and ask what it knows of a client, and in `observe` mode passes on the
|
file with an error and for a replacement of the lookup database it cannot read,
|
||||||
requests it would refuse, logging what it would have done with them. It comes as
|
serves Prometheus metrics to a scraper that holds the metrics token, lets an
|
||||||
the image the app's own image is built on. The rest of the design comes after
|
admin who holds the admin token list, add and lift bans and ask what it knows of
|
||||||
that, in the order of the build order in [`SPEC.md`](SPEC.md). The survey of
|
a client, and in `observe` mode passes on the requests it would refuse, logging
|
||||||
existing tools that led to the design is in [`EVALUATION.md`](EVALUATION.md).
|
what it would have done with them. It comes as the image the app's own image is
|
||||||
|
built on. The rest of the design comes after that, in the order of the build
|
||||||
|
order in [`SPEC.md`](SPEC.md). The survey of existing tools that led to the
|
||||||
|
design is in [`EVALUATION.md`](EVALUATION.md).
|
||||||
|
|
||||||
## Getting started
|
## Getting started
|
||||||
|
|
||||||
@@ -206,35 +209,48 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
|
|||||||
limit, the lowest of its percentages applying, as for a biased threshold, and
|
limit, the lowest of its percentages applying, as for a biased threshold, and
|
||||||
with `log` nothing more is done. Whatever the action, the request's log line
|
with `log` nothing more is done. Whatever the action, the request's log line
|
||||||
names the lists, and each raises an alert.
|
names the lists, and each raises an alert.
|
||||||
|
- Checks the client's own address against the DNSBL zones `SWWAF_DNSBL_ZONES`
|
||||||
|
names, after the blocklists and before the rate limits (see "DNS blocklists"
|
||||||
|
below), by the verdicts it keeps. A zone is asked about a client in the
|
||||||
|
background, and no request waits for its answer: the request that has it
|
||||||
|
asked, and any other from the client before the answer comes, goes on as from
|
||||||
|
a client the zone does not list. With `SWWAF_REPUTATION_ACTION` at `limit:25`,
|
||||||
|
its default, a client a zone's verdict lists gets a quarter of every rate
|
||||||
|
limit and byte limit, the lowest of its percentages applying, as for a biased
|
||||||
|
threshold. With `deny`, its requests are refused with `SWWAF_BAN_RESPONSE`
|
||||||
|
before their bodies are read; they are not counted for the rate limits, and
|
||||||
|
make no ban. With `log`, nothing more is done. Whatever the action, the
|
||||||
|
request's log line names the zones, and each raises an alert. A client a
|
||||||
|
blocklist refuses is not checked.
|
||||||
- Checks the client's own address against the static lists, the three netblock
|
- Checks the client's own address against the static lists, the three netblock
|
||||||
settings below, before anything else, its lookup included. A client in
|
settings below, before anything else, its lookup included. A client in
|
||||||
`SWWAF_ALLOW_NETS` skips bans, the country lists, the blocklists, the rate
|
`SWWAF_ALLOW_NETS` skips bans, the country lists, the blocklists, the DNSBL
|
||||||
limits, the byte limits and the rule files, and is not looked up; the timeouts
|
zones, the rate limits, the byte limits and the rule files, and is not looked
|
||||||
and size limits still apply. A client in `SWWAF_DENY_NETS` is refused with
|
up; the timeouts and size limits still apply. A client in `SWWAF_DENY_NETS` is
|
||||||
`SWWAF_BAN_RESPONSE` before its body is read, and the request is not counted
|
refused with `SWWAF_BAN_RESPONSE` before its body is read, and the request is
|
||||||
for the rate limits; an address in `SWWAF_ALLOW_NETS` too is let through. A
|
not counted for the rate limits; an address in `SWWAF_ALLOW_NETS` too is let
|
||||||
client in `SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither counted nor refused by the
|
through. A client in `SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither counted nor
|
||||||
rate limits, and has no bytes counted by the byte limits; the country lists,
|
refused by the rate limits, and has no bytes counted by the byte limits; the
|
||||||
the rule files and bans still apply to it.
|
country lists, the rule files and bans still apply to it.
|
||||||
- In `observe` mode, with `SWWAF_MODE=observe`, refuses none of the requests
|
- In `observe` mode, with `SWWAF_MODE=observe`, refuses none of the requests
|
||||||
that `SWWAF_DENY_NETS`, a ban, the country lists, a blocklist, a rate limit or
|
that `SWWAF_DENY_NETS`, a ban, the country lists, a blocklist, a DNSBL zone's
|
||||||
a rule would refuse: it passes them to the app, and their log lines name what
|
verdict, a rate limit or a rule would refuse: it passes them to the app, and
|
||||||
`enforce` mode would have done (see `would_action` in "Request log" below).
|
their log lines name what `enforce` mode would have done (see `would_action`
|
||||||
The checks run, and requests and bytes are counted, as in `enforce` mode, with
|
in "Request log" below). The checks run, and requests and bytes are counted,
|
||||||
three differences: neither a broken rate limit or byte limit nor a `ban` rule
|
as in `enforce` mode, with three differences: neither a broken rate limit or
|
||||||
makes a ban; a broken limit does not set the client's counters back to zero,
|
byte limit nor a `ban` rule makes a ban; a broken limit does not set the
|
||||||
so each request over a rate limit is logged as one that would be refused, and
|
client's counters back to zero, so each request over a rate limit is logged as
|
||||||
each whose bytes keep the client over a byte limit as breaking it; and a
|
one that would be refused, and each whose bytes keep the client over a byte
|
||||||
request under a ban does not make it permanent. As in `enforce` mode, the
|
limit as breaking it; and a request under a ban does not make it permanent. As
|
||||||
bytes counted are only those of the requests `enforce` mode would have passed
|
in `enforce` mode, the bytes counted are only those of the requests `enforce`
|
||||||
to the app. A ban it would have made, or made permanent, raises the alert
|
mode would have passed to the app. A ban it would have made, or made
|
||||||
`enforce` mode would have raised, marked as what would have happened (see
|
permanent, raises the alert `enforce` mode would have raised, marked as what
|
||||||
"Alerts" below). The bans in `bans.json` are kept, and refuse requests again
|
would have happened (see "Alerts" below). The bans in `bans.json` are kept,
|
||||||
when `smallwebwaf` next runs in `enforce` mode, as long as they last. The
|
and refuse requests again when `smallwebwaf` next runs in `enforce` mode, as
|
||||||
timeouts and size limits still apply, since they protect `smallwebwaf` and the
|
long as they last. The timeouts and size limits still apply, since they
|
||||||
app themselves, and a request for one of `smallwebwaf`'s own endpoints without
|
protect `smallwebwaf` and the app themselves, and a request for one of
|
||||||
its token is still answered `401`. It is for trying a configuration before
|
`smallwebwaf`'s own endpoints without its token is still answered `401`. It is
|
||||||
enforcing it.
|
for trying a configuration before enforcing it.
|
||||||
- Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any
|
- Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any
|
||||||
check and without asking the app, for the image's health check.
|
check and without asking the app, for the image's health check.
|
||||||
- Answers `GET /_smallwebwaf/metrics` with its metrics (see "Metrics" below) for
|
- Answers `GET /_smallwebwaf/metrics` with its metrics (see "Metrics" below) for
|
||||||
@@ -254,14 +270,14 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
|
|||||||
`SWWAF_LOG_REMOTE_URL` names one (see "Sending the log to a syslog server"
|
`SWWAF_LOG_REMOTE_URL` names one (see "Sending the log to a syslog server"
|
||||||
below).
|
below).
|
||||||
- Sends an alert for each ban it makes or makes permanent, for a count over an
|
- Sends an alert for each ban it makes or makes permanent, for a count over an
|
||||||
anomaly threshold, for a client a blocklist lists, for GeoJS failing or a list
|
anomaly threshold, for a client a blocklist or a DNSBL zone lists, for GeoJS
|
||||||
it cannot fetch, for a rule file or state file with an error, and for a
|
failing, a list it cannot fetch or a DNSBL zone that fails or refuses a query,
|
||||||
replacement of the lookup database it cannot read, holding back repeats and,
|
for a rule file or state file with an error, and for a replacement of the
|
||||||
past an hourly limit, rolling the rest into one summary, to each destination
|
lookup database it cannot read, holding back repeats and, past an hourly
|
||||||
you name: as a JSON object to the webhook `SWWAF_ALERT_WEBHOOK_URL` names, as
|
limit, rolling the rest into one summary, to each destination you name: as a
|
||||||
a message to the Slack incoming webhook `SWWAF_ALERT_SLACK_WEBHOOK_URL` names,
|
JSON object to the webhook `SWWAF_ALERT_WEBHOOK_URL` names, as a message to
|
||||||
and as a message to the ntfy topic `SWWAF_ALERT_NTFY_URL` names (see "Alerts"
|
the Slack incoming webhook `SWWAF_ALERT_SLACK_WEBHOOK_URL` names, and as a
|
||||||
below).
|
message to the ntfy topic `SWWAF_ALERT_NTFY_URL` names (see "Alerts" below).
|
||||||
- Counts requests and their bytes over a minute and an hour, per client, per
|
- Counts requests and their bytes over a minute and an hour, per client, per
|
||||||
netblock around a client, per AS number, for the whole service and per named
|
netblock around a client, per AS number, for the whole service and per named
|
||||||
netblock, and sends an `anomaly` alert for a count over the anomaly threshold
|
netblock, and sends an `anomaly` alert for a count over the anomaly threshold
|
||||||
@@ -324,8 +340,8 @@ effective settings are logged at start.
|
|||||||
- `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body.
|
- `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body.
|
||||||
- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body.
|
- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body.
|
||||||
- `SWWAF_ALLOW_NETS` (default empty): netblocks whose clients skip bans, the
|
- `SWWAF_ALLOW_NETS` (default empty): netblocks whose clients skip bans, the
|
||||||
country lists, the blocklists, the rate limits, the byte limits and the rule
|
country lists, the blocklists, the DNSBL zones, the rate limits, the byte
|
||||||
files, such as your monitoring or your own networks.
|
limits and the rule files, such as your monitoring or your own networks.
|
||||||
- `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients the
|
- `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients the
|
||||||
rate limits and the byte limits do not apply to, such as a machine that talks
|
rate limits and the byte limits do not apply to, such as a machine that talks
|
||||||
to the app all day.
|
to the app all day.
|
||||||
@@ -427,10 +443,34 @@ effective settings are logged at start.
|
|||||||
`limit:<percent>`, such as `limit:25`, gives it that percentage of every rate
|
`limit:<percent>`, such as `limit:25`, gives it that percentage of every rate
|
||||||
limit and byte limit, and `log` does nothing more than note the lists in the
|
limit and byte limit, and `log` does nothing more than note the lists in the
|
||||||
log line and raise the alert.
|
log line and raise the alert.
|
||||||
|
- `SWWAF_DNSBL_ZONES` (default empty): the DNSBL zones each client is asked
|
||||||
|
about, such as `dnsbl.dronebl.org` (see "DNS blocklists" below). A zone is a
|
||||||
|
DNS name of at most 189 characters: labels of letters, digits and hyphens, of
|
||||||
|
up to 63 characters each, neither starting nor ending with a hyphen, joined by
|
||||||
|
dots. A Spamhaus zone is the name of its keyed query service, with the key in
|
||||||
|
it, such as `<key>.xbl.dq.spamhaus.net`, and is shown with its key masked (see
|
||||||
|
"DNS blocklists" below). A zone that is not such a name stops the start, as
|
||||||
|
does one listed twice, even with its letters in another case or with another
|
||||||
|
key. Do not name a zone meant for mail (see "DNS blocklists" below).
|
||||||
|
- `SWWAF_DNSBL_RESOLVER` (default unset): the resolver the zones are asked
|
||||||
|
through, an IP address with an optional port, 53 when none is given, such as
|
||||||
|
`192.0.2.53` or `[2001:db8::53]:5353`. Unset, it is the host's, as
|
||||||
|
`/etc/resolv.conf` names it. Several zones refuse queries that come through a
|
||||||
|
public resolver.
|
||||||
|
- `SWWAF_REPUTATION_ACTION` (default `limit:25`): what is done with a client a
|
||||||
|
zone's verdict lists, as `SWWAF_BLOCKLIST_ACTION` is for a blocklist: `deny`,
|
||||||
|
`limit:<percent>` or `log`. A zone's verdict is less certain than a list such
|
||||||
|
as DROP, so by default a client it lists gets a quarter of every rate limit
|
||||||
|
and byte limit.
|
||||||
|
- `SWWAF_REPUTATION_CACHE_TTL` (default `24h`): how long a zone's verdict on a
|
||||||
|
client is used after the zone gave it.
|
||||||
|
- `SWWAF_REPUTATION_TIMEOUT` (default `2s`): how long a query to a zone may take
|
||||||
|
before it fails.
|
||||||
- `SWWAF_BAN_RESPONSE` (default `403`): how a refused client is answered, one
|
- `SWWAF_BAN_RESPONSE` (default `403`): how a refused client is answered, one
|
||||||
that is banned, breaks a rate limit, matches a `ban` rule, is in
|
that is banned, breaks a rate limit, matches a `ban` rule, is in
|
||||||
`SWWAF_DENY_NETS`, comes from a refused country or is in a blocklist while
|
`SWWAF_DENY_NETS`, comes from a refused country, is in a blocklist while
|
||||||
`SWWAF_BLOCKLIST_ACTION` is `deny`: `403`, `429`, or `close` to close the
|
`SWWAF_BLOCKLIST_ACTION` is `deny` or is listed by a DNSBL zone while
|
||||||
|
`SWWAF_REPUTATION_ACTION` is `deny`: `403`, `429`, or `close` to close the
|
||||||
connection without an answer. Behind traefik, `close` does not leave the
|
connection without an answer. Behind traefik, `close` does not leave the
|
||||||
client unanswered: traefik answers `502`, as it does whenever its backend
|
client unanswered: traefik answers `502`, as it does whenever its backend
|
||||||
drops a connection. A `block` rule always answers `403`.
|
drops a connection. A `block` rule always answers `403`.
|
||||||
@@ -576,14 +616,16 @@ listed twice in one of them stops the start. `off` switches a timeout, a size
|
|||||||
limit, a rate limit, a byte limit, an anomaly threshold, `SWWAF_ALERT_COOLDOWN`
|
limit, a rate limit, a byte limit, an anomaly threshold, `SWWAF_ALERT_COOLDOWN`
|
||||||
or `SWWAF_ALERT_MAX_PER_HOUR` off; `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`,
|
or `SWWAF_ALERT_MAX_PER_HOUR` off; `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`,
|
||||||
`SWWAF_LOOKUP_TIMEOUT`, `SWWAF_UNKNOWN_LIMIT_PERCENT`,
|
`SWWAF_LOOKUP_TIMEOUT`, `SWWAF_UNKNOWN_LIMIT_PERCENT`,
|
||||||
`SWWAF_BLOCKLIST_REFRESH`, the ban settings, the state settings,
|
`SWWAF_BLOCKLIST_REFRESH`, `SWWAF_REPUTATION_CACHE_TTL`,
|
||||||
|
`SWWAF_REPUTATION_TIMEOUT`, the ban settings, the state settings,
|
||||||
`SWWAF_METRICS_TOP_N`, `SWWAF_LOG_REMOTE_BUFFER`, `SWWAF_ANOMALY_NET_V4_PREFIX`
|
`SWWAF_METRICS_TOP_N`, `SWWAF_LOG_REMOTE_BUFFER`, `SWWAF_ANOMALY_NET_V4_PREFIX`
|
||||||
and `SWWAF_ANOMALY_NET_V6_PREFIX` cannot be off.
|
and `SWWAF_ANOMALY_NET_V6_PREFIX` cannot be off.
|
||||||
|
|
||||||
Several limits are fixed rather than settings. At most 20,000 clients are kept,
|
Several limits are fixed rather than settings. At most 20,000 clients are kept,
|
||||||
with their counters and history, and an IPv6 client is counted by its /64. At
|
with their counters and history, and an IPv6 client is counted by its /64. At
|
||||||
most 100,000 answers from GeoJS are kept, for 7 days each, and at most 20,000
|
most 100,000 answers from GeoJS are kept, for 7 days each, at most 20,000
|
||||||
anomaly counters.
|
anomaly counters, and at most 100,000 verdicts of the DNSBL zones, with at most
|
||||||
|
1,000 queries to them under way at once.
|
||||||
|
|
||||||
### Settings given as files
|
### Settings given as files
|
||||||
|
|
||||||
@@ -666,8 +708,9 @@ which every line has.
|
|||||||
app's, as passed on, or those of `smallwebwaf`'s own answer.
|
app's, as passed on, or those of `smallwebwaf`'s own answer.
|
||||||
- `request_bytes` and `response_bytes` count body bytes.
|
- `request_bytes` and `response_bytes` count body bytes.
|
||||||
- `action` is `forward` for a request passed to the app, `denied` for one
|
- `action` is `forward` for a request passed to the app, `denied` for one
|
||||||
refused because its client is in `SWWAF_DENY_NETS`, or in a blocklist while
|
refused because its client is in `SWWAF_DENY_NETS`, in a blocklist while
|
||||||
`SWWAF_BLOCKLIST_ACTION` is `deny`, `banned` for one refused because a ban
|
`SWWAF_BLOCKLIST_ACTION` is `deny`, or listed by a DNSBL zone while
|
||||||
|
`SWWAF_REPUTATION_ACTION` is `deny`, `banned` for one refused because a ban
|
||||||
covers its client or because it matched a `ban` rule, which bans its client,
|
covers its client or because it matched a `ban` rule, which bans its client,
|
||||||
`country_denied` for one refused for its client's country, `rate_limited` for
|
`country_denied` for one refused for its client's country, `rate_limited` for
|
||||||
one that broke a rate limit and banned its client, `rule_blocked` for one a
|
one that broke a rate limit and banned its client, `rule_blocked` for one a
|
||||||
@@ -676,16 +719,17 @@ which every line has.
|
|||||||
could not be reached or its answer broke off, and `admin` for one
|
could not be reached or its answer broke off, and `admin` for one
|
||||||
`smallwebwaf` answered at its own endpoint.
|
`smallwebwaf` answered at its own endpoint.
|
||||||
- `would_action` is there in `observe` mode for a request that
|
- `would_action` is there in `observe` mode for a request that
|
||||||
`SWWAF_DENY_NETS`, a ban, the country lists, a blocklist, a rate limit or a
|
`SWWAF_DENY_NETS`, a ban, the country lists, a blocklist, a DNSBL zone's
|
||||||
rule would have refused in `enforce` mode, and names the action that refusal
|
verdict, a rate limit or a rule would have refused in `enforce` mode, and
|
||||||
would have had: `denied`, `banned`, `country_denied`, `rate_limited` or
|
names the action that refusal would have had: `denied`, `banned`,
|
||||||
`rule_blocked`. `action` then names what was done: `forward` for a request
|
`country_denied`, `rate_limited` or `rule_blocked`. `action` then names what
|
||||||
passed to the app, and another action, such as `too_large`, for one a size or
|
was done: `forward` for a request passed to the app, and another action, such
|
||||||
time limit refused.
|
as `too_large`, for one a size or time limit refused.
|
||||||
- `limit_percent` is there for a request the rate limits count whose client a
|
- `limit_percent` is there for a request the rate limits count whose client a
|
||||||
biased threshold, or `SWWAF_BLOCKLIST_ACTION` for a blocklist that lists it,
|
biased threshold, `SWWAF_BLOCKLIST_ACTION` for a blocklist that lists it, or
|
||||||
gives less than the whole of the rate limits, and gives the percentage it
|
`SWWAF_REPUTATION_ACTION` for a DNSBL zone whose verdict lists it, gives less
|
||||||
gets, with `limit_percent_setting` naming the setting that gave it, such as
|
than the whole of the rate limits, and gives the percentage it gets, with
|
||||||
|
`limit_percent_setting` naming the setting that gave it, such as
|
||||||
`SWWAF_ASN_LIMIT_PERCENT`, or `SWWAF_ASN_LIMIT_PERCENT_URL` for the file it
|
`SWWAF_ASN_LIMIT_PERCENT`, or `SWWAF_ASN_LIMIT_PERCENT_URL` for the file it
|
||||||
names. `bytes_percent` and `bytes_percent_setting` are the same for the byte
|
names. `bytes_percent` and `bytes_percent_setting` are the same for the byte
|
||||||
limits. Each is left out when the client gets the whole of those limits.
|
limits. Each is left out when the client gets the whole of those limits.
|
||||||
@@ -697,11 +741,11 @@ which every line has.
|
|||||||
health check, one from a client in `SWWAF_ALLOW_NETS` or
|
health check, one from a client in `SWWAF_ALLOW_NETS` or
|
||||||
`SWWAF_RATE_LIMIT_EXEMPT_NETS`, one for a path that
|
`SWWAF_RATE_LIMIT_EXEMPT_NETS`, one for a path that
|
||||||
`SWWAF_RATE_LIMIT_EXEMPT_PATHS` exempts, and one that `SWWAF_DENY_NETS`, a
|
`SWWAF_RATE_LIMIT_EXEMPT_PATHS` exempts, and one that `SWWAF_DENY_NETS`, a
|
||||||
ban, the country lists or a blocklist refuse, or would refuse in `observe`
|
ban, the country lists, a blocklist or a DNSBL zone's verdict refuse, or would
|
||||||
mode. Its `minute_bytes`, `hour_bytes` and `day_bytes` give the client's bytes
|
refuse in `observe` mode. Its `minute_bytes`, `hour_bytes` and `day_bytes`
|
||||||
in each window as the byte limits count them, in the same way: for a request
|
give the client's bytes in each window as the byte limits count them, in the
|
||||||
whose bytes they count, with its own, once it has ended; for any other, those
|
same way: for a request whose bytes they count, with its own, once it has
|
||||||
counted before it.
|
ended; for any other, those counted before it.
|
||||||
- `rule_ids` is there for a request that matched rules of the rule files, and
|
- `rule_ids` is there for a request that matched rules of the rule files, and
|
||||||
lists their ids in the order they matched, up to the one that refused it.
|
lists their ids in the order they matched, up to the one that refused it.
|
||||||
- `limit_hit` is there for a request that broke a rate limit, or whose bytes
|
- `limit_hit` is there for a request that broke a rate limit, or whose bytes
|
||||||
@@ -711,12 +755,16 @@ which every line has.
|
|||||||
several. `offence` is then `limit`. A request whose bytes broke a byte limit
|
several. `offence` is then `limit`. A request whose bytes broke a byte limit
|
||||||
is not refused: its `action` is what it would have been otherwise, such as
|
is not refused: its `action` is what it would have been otherwise, such as
|
||||||
`forward`.
|
`forward`.
|
||||||
- `reputation` is there for a request whose client a blocklist lists, and gives
|
- `reputation` is there for a request whose client a blocklist or a DNSBL zone's
|
||||||
the URLs of the blocklists that list it, in the order `SWWAF_BLOCKLIST_URLS`
|
verdict lists, and gives the URLs of the blocklists that list it, in the order
|
||||||
names them, whatever `SWWAF_BLOCKLIST_ACTION` says. It is left out for a
|
`SWWAF_BLOCKLIST_URLS` names them, then the zones whose verdict lists it, in
|
||||||
client the blocklists are not checked for: one in `SWWAF_ALLOW_NETS`, and one
|
the order `SWWAF_DNSBL_ZONES` names them, whatever `SWWAF_BLOCKLIST_ACTION`
|
||||||
`SWWAF_DENY_NETS`, a ban or the country lists refuse first, or would in
|
and `SWWAF_REPUTATION_ACTION` say. A zone whose verdict on the client has not
|
||||||
`observe` mode.
|
come yet, or was given `SWWAF_REPUTATION_CACHE_TTL` ago or more, is not named.
|
||||||
|
It is left out for a client the blocklists are not checked for: one in
|
||||||
|
`SWWAF_ALLOW_NETS`, and one `SWWAF_DENY_NETS`, a ban or the country lists
|
||||||
|
refuse first, or would in `observe` mode. The zones are not checked either for
|
||||||
|
a client a blocklist refuses, or would in `observe` mode.
|
||||||
- `ban_expires` is there for a request that made a ban or was refused under one,
|
- `ban_expires` is there for a request that made a ban or was refused under one,
|
||||||
or in `observe` mode would have been refused under one, and gives when the ban
|
or in `observe` mode would have been refused under one, and gives when the ban
|
||||||
ends, in the same form as `time`, or `permanent`.
|
ends, in the same form as `time`, or `permanent`.
|
||||||
@@ -786,11 +834,13 @@ it, as below. An alert is for one of these events, and is sent when
|
|||||||
- `anomaly`: a count of requests or bytes over an anomaly threshold, raised by
|
- `anomaly`: a count of requests or bytes over an anomaly threshold, raised by
|
||||||
each request that ends with the count over it, in `observe` mode as in
|
each request that ends with the count over it, in `observe` mode as in
|
||||||
`enforce` mode. It refuses and bans nothing.
|
`enforce` mode. It refuses and bans nothing.
|
||||||
- `reputation_hit`: a request whose client a blocklist lists, one alert for each
|
- `reputation_hit`: a request whose client a blocklist or a DNSBL zone's verdict
|
||||||
blocklist that lists it, whatever `SWWAF_BLOCKLIST_ACTION` says, in `observe`
|
lists, one alert for each blocklist and each zone that lists it, whatever
|
||||||
mode as in `enforce` mode.
|
`SWWAF_BLOCKLIST_ACTION` and `SWWAF_REPUTATION_ACTION` say, in `observe` mode
|
||||||
- `source_failure`: GeoJS failing or refusing `smallwebwaf`, or a fetch of a
|
as in `enforce` mode.
|
||||||
list failing (see "Blocklists" below).
|
- `source_failure`: GeoJS failing or refusing `smallwebwaf`, a fetch of a list
|
||||||
|
failing (see "Blocklists" below), or a query to a DNSBL zone failing or
|
||||||
|
refused (see "DNS blocklists" below).
|
||||||
- `file_error`: a rule file edited while it runs that has an error, an edit of a
|
- `file_error`: a rule file edited while it runs that has an error, an edit of a
|
||||||
state file set aside as `<name>.bad`, a state file it could not write while
|
state file set aside as `<name>.bad`, a state file it could not write while
|
||||||
running, or a replacement of the lookup database it could not read, which it
|
running, or a replacement of the lookup database it could not read, which it
|
||||||
@@ -863,7 +913,7 @@ is sent on one line:
|
|||||||
- `reason` is a short sentence; for a ban, the ban's `reason` in `bans.json`;
|
- `reason` is a short sentence; for a ban, the ban's `reason` in `bans.json`;
|
||||||
for an `anomaly`, what was counted over which threshold, such as
|
for an `anomaly`, what was counted over which threshold, such as
|
||||||
`requests per minute of the netblock 203.0.113.0/24 over the threshold of 1000`;
|
`requests per minute of the netblock 203.0.113.0/24 over the threshold of 1000`;
|
||||||
for a `reputation_hit`, `listed by a blocklist`.
|
for a `reputation_hit`, `listed by a blocklist` or `listed by a DNSBL zone`.
|
||||||
- `detail` is what is particular to the event: for a ban, its `cause`, when it
|
- `detail` is what is particular to the event: for a ban, its `cause`, when it
|
||||||
ends as `ban_expires`, in the form the request log gives it, and its `notes`,
|
ends as `ban_expires`, in the form the request log gives it, and its `notes`,
|
||||||
as `bans.json` gives them; for an `anomaly`, the `scope`, `client`, `net`,
|
as `bans.json` gives them; for an `anomaly`, the `scope`, `client`, `net`,
|
||||||
@@ -871,11 +921,11 @@ is sent on one line:
|
|||||||
`asn` and the `name` of the named netblock for `watch`, the `window`, `minute`
|
`asn` and the `name` of the named netblock for `watch`, the `window`, `minute`
|
||||||
or `hour`, the `kind`, `requests` or `bytes`, the `count`, which is weighted
|
or `hour`, the `kind`, `requests` or `bytes`, the `count`, which is weighted
|
||||||
as the rate limits weigh theirs, and the `threshold`; for a `reputation_hit`,
|
as the rate limits weigh theirs, and the `threshold`; for a `reputation_hit`,
|
||||||
the `source`, the URL of the blocklist; for `source_failure`, the `source`,
|
the `source`, the URL of the blocklist or the zone; for `source_failure`, the
|
||||||
`geojs` or the URL of the list, the `error`, and for GeoJS, when it is asked
|
`source`, `geojs`, the URL of the list or the zone, the `error`, and for
|
||||||
again, `asking_again_in`; for `file_error`, the `file`, which for an edit set
|
GeoJS, when it is asked again, `asking_again_in`; for `file_error`, the
|
||||||
aside is the file it was renamed to, and the `error`, which for a file that
|
`file`, which for an edit set aside is the file it was renamed to, and the
|
||||||
does not parse names where in it the error is.
|
`error`, which for a file that does not parse names where in it the error is.
|
||||||
- `suppressed_repeats` is how many repeats the cooldown held back before this
|
- `suppressed_repeats` is how many repeats the cooldown held back before this
|
||||||
alert, and for a `summary`, those no other alert gives (see below).
|
alert, and for a `summary`, those no other alert gives (see below).
|
||||||
|
|
||||||
@@ -921,11 +971,11 @@ and Slack this JSON object, shown indented; it is sent on one line:
|
|||||||
```
|
```
|
||||||
|
|
||||||
An alert for the same event as the last one sent, on the same netblock, and for
|
An alert for the same event as the last one sent, on the same netblock, and for
|
||||||
a `reputation_hit` about the same blocklist, or for a `file_error` about the
|
a `reputation_hit` about the same blocklist or zone, or for a `file_error` about
|
||||||
same file, or for a `source_failure` about the same source, or for an `anomaly`
|
the same file, or for a `source_failure` about the same source, or for an
|
||||||
in the same scope, with the same netblock, AS number or name, whatever its
|
`anomaly` in the same scope, with the same netblock, AS number or name, whatever
|
||||||
window and kind, less than `SWWAF_ALERT_COOLDOWN` after it, is a repeat: it is
|
its window and kind, less than `SWWAF_ALERT_COOLDOWN` after it, is a repeat: it
|
||||||
held back and counted, and the next alert sent for them gives that count as
|
is held back and counted, and the next alert sent for them gives that count as
|
||||||
`suppressed_repeats`. As each hour of the clock, in UTC, ends, the cooldowns
|
`suppressed_repeats`. As each hour of the clock, in UTC, ends, the cooldowns
|
||||||
that have run out are dropped, and the repeats they held back, which no alert
|
that have run out are dropped, and the repeats they held back, which no alert
|
||||||
sent since has given, go in that hour's summary.
|
sent since has given, go in that hour's summary.
|
||||||
@@ -990,8 +1040,11 @@ with times in UTC.
|
|||||||
indented to be read, under `lists`: its `url`, when it was last `tried`, the
|
indented to be read, under `lists`: its `url`, when it was last `tried`, the
|
||||||
fetch failed or not, and its last good copy: when that was `fetched`, and its
|
fetch failed or not, and its last good copy: when that was `fetched`, and its
|
||||||
`lines`, as fetched, comment lines included, each on a line of its own, both
|
`lines`, as fetched, comment lines included, each on a line of its own, both
|
||||||
left out while no fetch of it has succeeded. As the file is read, the lists
|
left out while no fetch of it has succeeded; and under `verdicts`, each
|
||||||
the settings no longer name are dropped.
|
verdict of a DNSBL zone still in use (see "DNS blocklists" below): its `zone`,
|
||||||
|
the `client`'s address, whether the zone `listed` the client, and when the
|
||||||
|
zone gave it, `fetched`. As the file is read, the lists the settings no longer
|
||||||
|
name, and the verdicts of the zones they no longer name, are dropped.
|
||||||
- `alerts.json`: the state of the alerts (see "Alerts" above), indented to be
|
- `alerts.json`: the state of the alerts (see "Alerts" above), indented to be
|
||||||
read: under `cooldowns`, for each event and netblock, with the `source` too
|
read: under `cooldowns`, for each event and netblock, with the `source` too
|
||||||
for a `reputation_hit`, or event and `file` or `source`, or for an `anomaly`,
|
for a `reputation_hit`, or event and `file` or `source`, or for an `anomaly`,
|
||||||
@@ -1029,24 +1082,26 @@ keeps refusing every client in its netblock until it ends, even after
|
|||||||
until a fetch of it succeeds. A netblock whose address has bits past its length,
|
until a fetch of it succeeds. A netblock whose address has bits past its length,
|
||||||
such as `203.0.113.9/24`, is read as the netblock it is in, `203.0.113.0/24`.
|
such as `203.0.113.9/24`, is read as the netblock it is in, `203.0.113.0/24`.
|
||||||
Buckets and answers whose time has passed are dropped, and so is an anomaly
|
Buckets and answers whose time has passed are dropped, and so is an anomaly
|
||||||
counter left with no bucket. A missing file is empty state, as on a first start.
|
counter left with no bucket; a verdict past its time is neither used nor written
|
||||||
A file that does not parse, or has another `version`, stops the start with a
|
again. A missing file is empty state, as on a first start. A file that does not
|
||||||
message naming the file, and the line and column where Go's JSON decoder gives
|
parse, or has another `version`, stops the start with a message naming the file,
|
||||||
them; so does a state directory `smallwebwaf` cannot write. So does an entry
|
and the line and column where Go's JSON decoder gives them; so does a state
|
||||||
without a field it needs, named with the entry's place in the file: a ban's
|
directory `smallwebwaf` cannot write. So does an entry without a field it needs,
|
||||||
`netblock`, `start` or `expires`, which is `null` for a permanent ban; a
|
named with the entry's place in the file: a ban's `netblock`, `start` or
|
||||||
client's `client`, or the `start` of a window in which it has requests or bytes;
|
`expires`, which is `null` for a permanent ban; a client's `client`, or the
|
||||||
an answer's `client`, `country`, which is `""` for a client GeoJS cannot place,
|
`start` of a window in which it has requests or bytes; an answer's `client`,
|
||||||
or `answered`; a cooldown's `event` or `sent`; an alert waiting's `event` or
|
`country`, which is `""` for a client GeoJS cannot place, or `answered`; a
|
||||||
`time`; an anomaly counter's `netblock`, unless it counts an AS number or the
|
cooldown's `event` or `sent`; an alert waiting's `event` or `time`; an anomaly
|
||||||
whole service, its `asn`, for an AS number, its `name`, for a named netblock, or
|
counter's `netblock`, unless it counts an AS number or the whole service, its
|
||||||
the `start` of a window in which it has requests or bytes; a list's `url`,
|
`asn`, for an AS number, its `name`, for a named netblock, or the `start` of a
|
||||||
`fetched` or `lines`, which is `[]` for an empty list. So does a ban whose
|
window in which it has requests or bytes; a list's `url`, `fetched` or `lines`,
|
||||||
`cause` is not `limit`, `attack` or `admin`, alerts waiting for a destination
|
which is `[]` for an empty list; a verdict's `zone`, `client`, `listed`, which
|
||||||
that is not `webhook`, `slack` or `ntfy`, an anomaly counter whose `scope` is
|
is `false` for a client the zone does not list, or `fetched`. So does a ban
|
||||||
not `client`, `net`, `asn`, `total` or `watch`, and a copy of a list with a line
|
whose `cause` is not `limit`, `attack` or `admin`, alerts waiting for a
|
||||||
that would make its fetch fail. An answer's `asn` or `as_name` left out reads as
|
destination that is not `webhook`, `slack` or `ntfy`, an anomaly counter whose
|
||||||
empty.
|
`scope` is not `client`, `net`, `asn`, `total` or `watch`, and a copy of a list
|
||||||
|
with a line that would make its fetch fail. An answer's `asn` or `as_name` left
|
||||||
|
out reads as empty.
|
||||||
|
|
||||||
While it runs, `smallwebwaf` watches `SWWAF_STATE_DIR` and takes in your edit of
|
While it runs, `smallwebwaf` watches `SWWAF_STATE_DIR` and takes in your edit of
|
||||||
a state file as soon as you save it: what the file then holds replaces what
|
a state file as soon as you save it: what the file then holds replaces what
|
||||||
@@ -1238,6 +1293,11 @@ scraped, and keeps this one as `exported_instance` unless the scrape sets
|
|||||||
`smallwebwaf_reputation_failures_total`: the fetches of the list that failed;
|
`smallwebwaf_reputation_failures_total`: the fetches of the list that failed;
|
||||||
and `smallwebwaf_reputation_last_fetch_timestamp_seconds`: when the copy of it
|
and `smallwebwaf_reputation_last_fetch_timestamp_seconds`: when the copy of it
|
||||||
in use was fetched, `0` while there is none.
|
in use was fetched, `0` while there is none.
|
||||||
|
- By `source`, each zone `SWWAF_DNSBL_ZONES` names:
|
||||||
|
`smallwebwaf_reputation_hits_total`: the requests whose client the zone's
|
||||||
|
verdict lists, a series that comes with the first;
|
||||||
|
`smallwebwaf_reputation_queries_total`: the queries made to the zone; and
|
||||||
|
`smallwebwaf_reputation_failures_total`: those that failed.
|
||||||
- `smallwebwaf_tracked_clients`: the clients in the table of clients.
|
- `smallwebwaf_tracked_clients`: the clients in the table of clients.
|
||||||
- `smallwebwaf_state_file_writes_total`,
|
- `smallwebwaf_state_file_writes_total`,
|
||||||
`smallwebwaf_state_file_write_failures_total`,
|
`smallwebwaf_state_file_write_failures_total`,
|
||||||
@@ -1429,9 +1489,9 @@ goes through the candidates one by one.
|
|||||||
readable JSON files, written regularly and at every stop, so a restart loses
|
readable JSON files, written regularly and at every stop, so a restart loses
|
||||||
nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks
|
nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks
|
||||||
up the change. Nothing is read from disk while serving a request. The files
|
up the change. Nothing is read from disk while serving a request. The files
|
||||||
for the bans, the clients, the GeoJS answers, the copies of the lists and the
|
for the bans, the clients, the GeoJS answers, the copies of the lists, the
|
||||||
alerts are built, with an edit taken in while running (see "State files"
|
verdicts of the DNSBL zones and the alerts are built, with an edit taken in
|
||||||
above); the rest comes with its features.
|
while running (see "State files" above); the rest comes with its features.
|
||||||
- Health checks, the metrics, and listing, adding and lifting bans or asking why
|
- Health checks, the metrics, and listing, adding and lifting bans or asking why
|
||||||
a given address was refused, all on the one port every request uses: under
|
a given address was refused, all on the one port every request uses: under
|
||||||
`/_smallwebwaf/` on the app's own address, through traefik like any other
|
`/_smallwebwaf/` on the app's own address, through traefik like any other
|
||||||
@@ -1659,6 +1719,58 @@ hour apart whatever `SWWAF_BLOCKLIST_REFRESH` is: each fetches a list again that
|
|||||||
long after its own last try, so those first started within the same hour, with
|
long after its own last try, so those first started within the same hour, with
|
||||||
the same refresh, keep fetching within the same hour.
|
the same refresh, keep fetching within the same hour.
|
||||||
|
|
||||||
|
## DNS blocklists
|
||||||
|
|
||||||
|
`SWWAF_DNSBL_ZONES` names DNS blocklists, DNSBL zones, which list an address by
|
||||||
|
answering a query for a name made from it. None is named by default, for the
|
||||||
|
reason none of the blocklists is. `smallwebwaf` asks each zone about a client's
|
||||||
|
own address in the background, the first time it sees the client: the request
|
||||||
|
goes on at once, as from a client the zone does not list, and so does every
|
||||||
|
request from it until the zone has answered. The name asked about is the one RFC
|
||||||
|
5782 gives: the four numbers of an IPv4 address in reverse order, so that
|
||||||
|
`192.0.2.99` is asked about in `dnsbl.dronebl.org` as
|
||||||
|
`99.2.0.192.dnsbl.dronebl.org`, or the 32 hex digits of an IPv6 address in
|
||||||
|
reverse order, each followed by a dot. An IPv6 client is asked about by its own
|
||||||
|
address, not by its /64.
|
||||||
|
|
||||||
|
A zone that answers that the name does not exist, or has no address, does not
|
||||||
|
list the client, and one that answers with an address in `127.0.0.0/8` lists it.
|
||||||
|
Any other answer gives no verdict, and is a failure: an address in
|
||||||
|
`127.255.255.0/24`, with which Spamhaus refuses a query, such as one sent
|
||||||
|
through a public resolver or one past its limit; an address outside
|
||||||
|
`127.0.0.0/8`, such as a resolver gives that answers even for names that do not
|
||||||
|
exist; an error the resolver answers with, such as a refusal; and no answer
|
||||||
|
within `SWWAF_REPUTATION_TIMEOUT`. A failure is counted, logged and raised as a
|
||||||
|
`source_failure` alert, held back as a repeat within `SWWAF_ALERT_COOLDOWN`, and
|
||||||
|
the zone is not asked again for a minute, so that a zone that refuses queries is
|
||||||
|
not asked on every request.
|
||||||
|
|
||||||
|
Each verdict, the zone listing the client or not, is kept, in memory and in
|
||||||
|
`reputation.json` (see "State files" above), so that a restart keeps it, and is
|
||||||
|
used for `SWWAF_REPUTATION_CACHE_TTL` after the zone gave it, 24 hours by
|
||||||
|
default. The client's first request after that has the zone asked again, and
|
||||||
|
until it answers, no verdict of the zone applies to the client. At most 100,000
|
||||||
|
verdicts are kept, the one fetched longest ago dropped first, and at most 1,000
|
||||||
|
queries are under way at once; past that, a zone is asked about a client at the
|
||||||
|
client's next request.
|
||||||
|
|
||||||
|
A client in `SWWAF_ALLOW_NETS` is not checked, nor is one a blocklist refuses.
|
||||||
|
`SWWAF_REPUTATION_ACTION` says what is done with one a zone's verdict lists, as
|
||||||
|
"What it does so far" above describes.
|
||||||
|
|
||||||
|
Do not name a zone meant for mail, such as one of residential and dynamic
|
||||||
|
address ranges, which ordinary visitors come from, or one that includes such a
|
||||||
|
list, as Spamhaus's `zen` does: it would refuse them, or lower their limits.
|
||||||
|
Spamhaus's zones answer through its keyed query service, named with the key in
|
||||||
|
it, such as `<key>.xbl.dq.spamhaus.net`. The key of a zone under
|
||||||
|
`dq.spamhaus.net` is its first label, and `smallwebwaf` shows `********` in its
|
||||||
|
place wherever it names the zone, as `********.xbl.dq.spamhaus.net`: in the
|
||||||
|
settings logged at start, an error that stops the start, its own messages, the
|
||||||
|
request log, the alerts and the metrics. Only `reputation.json` keeps the zone
|
||||||
|
with its key. A key in the name of any other zone is shown as given. Several
|
||||||
|
zones refuse queries that come through a public resolver; `SWWAF_DNSBL_RESOLVER`
|
||||||
|
names another resolver to ask through.
|
||||||
|
|
||||||
## How the code is laid out
|
## How the code is laid out
|
||||||
|
|
||||||
- `cmd/smallwebwaf`: the binary, which only calls `internal/smallwebwaf`.
|
- `cmd/smallwebwaf`: the binary, which only calls `internal/smallwebwaf`.
|
||||||
@@ -1672,15 +1784,15 @@ the same refresh, keep fetching within the same hour.
|
|||||||
standard library's `httputil.ReverseProxy` within the timeouts and size
|
standard library's `httputil.ReverseProxy` within the timeouts and size
|
||||||
limits, and writes the request's log line. Its `check` method is where a
|
limits, and writes the request's log line. Its `check` method is where a
|
||||||
request is refused before anything reaches the app: for `SWWAF_DENY_NETS`, for
|
request is refused before anything reaches the app: for `SWWAF_DENY_NETS`, for
|
||||||
a ban, for the country lists, for a blocklist, for a rate limit, which bans
|
a ban, for the country lists, for a blocklist, for a DNSBL zone's verdict, for
|
||||||
the client, for a `block` or `ban` rule, the latter banning the client, and
|
a rate limit, which bans the client, for a `block` or `ban` rule, the latter
|
||||||
for an announced body over the size limit; in `observe` mode, only for the
|
banning the client, and for an announced body over the size limit; in
|
||||||
size limit, with what it would have refused for noted in the log line. A
|
`observe` mode, only for the size limit, with what it would have refused for
|
||||||
request under `/_smallwebwaf/` that `check` lets through is answered by
|
noted in the log line. A request under `/_smallwebwaf/` that `check` lets
|
||||||
`answerAdmin` instead of reaching the app. Once the answer to a request passed
|
through is answered by `answerAdmin` instead of reaching the app. Once the
|
||||||
to the app has ended, `countBytes` counts its bytes for the byte limits, and
|
answer to a request passed to the app has ended, `countBytes` counts its bytes
|
||||||
once any request but the health check has ended, `countAnomalies` counts it
|
for the byte limits, and once any request but the health check has ended,
|
||||||
for the anomaly thresholds.
|
`countAnomalies` counts it for the anomaly thresholds.
|
||||||
- `internal/metrics`: the metrics, counted as the other parts tell it what
|
- `internal/metrics`: the metrics, counted as the other parts tell it what
|
||||||
happened, and served in the Prometheus text format.
|
happened, and served in the Prometheus text format.
|
||||||
- `internal/bans`: the ban ledger: each netblock's bans with their notes, how
|
- `internal/bans`: the ban ledger: each netblock's bans with their notes, how
|
||||||
@@ -1696,7 +1808,9 @@ the same refresh, keep fetching within the same hour.
|
|||||||
- `internal/reputation`: fetches the blocklists and the file
|
- `internal/reputation`: fetches the blocklists and the file
|
||||||
`SWWAF_ASN_LIMIT_PERCENT_URL` names as they are due, keeps the last good copy
|
`SWWAF_ASN_LIMIT_PERCENT_URL` names as they are due, keeps the last good copy
|
||||||
of each, and tells which blocklists list an address and what percentage the
|
of each, and tells which blocklists list an address and what percentage the
|
||||||
file gives an AS number.
|
file gives an AS number; and asks the DNSBL zones about clients in the
|
||||||
|
background, through the standard library's resolver, keeps their verdicts, and
|
||||||
|
tells which zones' verdicts list an address.
|
||||||
- `internal/ratelimit`: the table of clients: counts each client's requests and
|
- `internal/ratelimit`: the table of clients: counts each client's requests and
|
||||||
bytes, tells when they take it over a rate limit or a byte limit, and keeps
|
bytes, tells when they take it over a rate limit or a byte limit, and keeps
|
||||||
each client's history.
|
each client's history.
|
||||||
@@ -1724,7 +1838,8 @@ the same refresh, keep fetching within the same hour.
|
|||||||
|
|
||||||
Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the
|
Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the
|
||||||
table of clients to 20,000 and the GeoJS answers to 100,000, dropping the least
|
table of clients to 20,000 and the GeoJS answers to 100,000, dropping the least
|
||||||
recently seen, the anomaly counters to 20,000, dropping the one counted least
|
recently seen, the DNSBL zones' verdicts to 100,000, dropping the one fetched
|
||||||
|
longest ago, the anomaly counters to 20,000, dropping the one counted least
|
||||||
recently, and the banned netblocks in the order they were last seen, from which
|
recently, and the banned netblocks in the order they were last seen, from which
|
||||||
the ledger picks the ban to drop past `SWWAF_MAX_BANS`, and
|
the ledger picks the ban to drop past `SWWAF_MAX_BANS`, and
|
||||||
`github.com/prometheus/client_golang` keeps the metrics and serves them, and
|
`github.com/prometheus/client_golang` keeps the metrics and serves them, and
|
||||||
|
|||||||
@@ -46,10 +46,11 @@ const (
|
|||||||
EventAnomaly = "anomaly"
|
EventAnomaly = "anomaly"
|
||||||
// EventWAFBlock comes with the Core Rule Set; nothing raises it yet.
|
// EventWAFBlock comes with the Core Rule Set; nothing raises it yet.
|
||||||
EventWAFBlock = "waf_block"
|
EventWAFBlock = "waf_block"
|
||||||
// EventReputationHit is a request whose client a blocklist lists.
|
// EventReputationHit is a request whose client a blocklist or a DNSBL
|
||||||
|
// zone lists.
|
||||||
EventReputationHit = "reputation_hit"
|
EventReputationHit = "reputation_hit"
|
||||||
// EventSourceFailure is GeoJS failing or refusing smallwebwaf, or a
|
// EventSourceFailure is GeoJS failing or refusing smallwebwaf, a fetch
|
||||||
// fetch of a list failing.
|
// of a list failing, or a query to a DNSBL zone failing or refused.
|
||||||
EventSourceFailure = "source_failure"
|
EventSourceFailure = "source_failure"
|
||||||
// EventFileError is a rule file or state file edited while smallwebwaf
|
// EventFileError is a rule file or state file edited while smallwebwaf
|
||||||
// runs that does not parse, a replacement of the lookup database that
|
// runs that does not parse, a replacement of the lookup database that
|
||||||
|
|||||||
+163
-3
@@ -152,6 +152,21 @@ type Config struct {
|
|||||||
BlocklistRefresh time.Duration
|
BlocklistRefresh time.Duration
|
||||||
BlocklistAction string
|
BlocklistAction string
|
||||||
BlocklistLimitPercent int64
|
BlocklistLimitPercent int64
|
||||||
|
// DNSBLZones are the DNSBL zones clients are asked about
|
||||||
|
// (SWWAF_DNSBL_ZONES), through DNSBLResolver (SWWAF_DNSBL_RESOLVER), or
|
||||||
|
// the host's resolver while that is the zero AddrPort.
|
||||||
|
// ReputationAction is what is done with a client a zone's verdict lists
|
||||||
|
// (SWWAF_REPUTATION_ACTION): deny, limit or log; for limit,
|
||||||
|
// ReputationLimitPercent is the percentage of every limit it gets. A
|
||||||
|
// verdict is used for ReputationCacheTTL after it was fetched
|
||||||
|
// (SWWAF_REPUTATION_CACHE_TTL), and a query may take ReputationTimeout
|
||||||
|
// (SWWAF_REPUTATION_TIMEOUT). Neither can be off.
|
||||||
|
DNSBLZones []string
|
||||||
|
DNSBLResolver netip.AddrPort
|
||||||
|
ReputationAction string
|
||||||
|
ReputationLimitPercent int64
|
||||||
|
ReputationCacheTTL time.Duration
|
||||||
|
ReputationTimeout time.Duration
|
||||||
// BanResponse is the status a refused client is answered with, 403
|
// BanResponse is the status a refused client is answered with, 403
|
||||||
// or 429, or 0 to close the connection without an answer
|
// or 429, or 0 to close the connection without an answer
|
||||||
// (SWWAF_BAN_RESPONSE). It answers a banned client, a request that
|
// (SWWAF_BAN_RESPONSE). It answers a banned client, a request that
|
||||||
@@ -366,6 +381,11 @@ var (
|
|||||||
errNotAnHourOrMore = errors.New("is not a duration of 1h or more, such as 24h")
|
errNotAnHourOrMore = errors.New("is not a duration of 1h or more, such as 24h")
|
||||||
errNotAction = errors.New(
|
errNotAction = errors.New(
|
||||||
"is not deny, limit:<percent> such as limit:25, or log")
|
"is not deny, limit:<percent> such as limit:25, or log")
|
||||||
|
errNotZone = errors.New("is not a DNS zone such as dnsbl.dronebl.org")
|
||||||
|
errZoneTooLong = errors.New("is longer than 189 characters, too long for the " +
|
||||||
|
"names IPv6 clients are asked about by")
|
||||||
|
errNotResolver = errors.New("is not an IP address with an optional port, " +
|
||||||
|
"such as 192.0.2.53 or [2001:db8::53]:5353")
|
||||||
)
|
)
|
||||||
|
|
||||||
// FromEnvironment reads the settings with lookupEnv, normally
|
// FromEnvironment reads the settings with lookupEnv, normally
|
||||||
@@ -418,6 +438,10 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
|
|||||||
ASNLimitPercentURL: env.listURL("SWWAF_ASN_LIMIT_PERCENT_URL"),
|
ASNLimitPercentURL: env.listURL("SWWAF_ASN_LIMIT_PERCENT_URL"),
|
||||||
BlocklistURLs: env.listURLs("SWWAF_BLOCKLIST_URLS"),
|
BlocklistURLs: env.listURLs("SWWAF_BLOCKLIST_URLS"),
|
||||||
BlocklistRefresh: env.refresh("SWWAF_BLOCKLIST_REFRESH", "24h"),
|
BlocklistRefresh: env.refresh("SWWAF_BLOCKLIST_REFRESH", "24h"),
|
||||||
|
DNSBLZones: env.zones("SWWAF_DNSBL_ZONES"),
|
||||||
|
DNSBLResolver: env.resolver("SWWAF_DNSBL_RESOLVER"),
|
||||||
|
ReputationCacheTTL: env.durationNotOff("SWWAF_REPUTATION_CACHE_TTL", "24h"),
|
||||||
|
ReputationTimeout: env.durationNotOff("SWWAF_REPUTATION_TIMEOUT", "2s"),
|
||||||
BanResponse: env.banResponse("SWWAF_BAN_RESPONSE", "403"),
|
BanResponse: env.banResponse("SWWAF_BAN_RESPONSE", "403"),
|
||||||
LimitBanDuration: env.durationNotOff("SWWAF_LIMIT_BAN_DURATION", "1h"),
|
LimitBanDuration: env.durationNotOff("SWWAF_LIMIT_BAN_DURATION", "1h"),
|
||||||
LimitBanRepeatWindow: env.durationNotOff("SWWAF_LIMIT_BAN_REPEAT_WINDOW", "24h"),
|
LimitBanRepeatWindow: env.durationNotOff("SWWAF_LIMIT_BAN_REPEAT_WINDOW", "24h"),
|
||||||
@@ -462,6 +486,8 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
|
|||||||
cfg.InstanceName, cfg.LogRemoteURL != nil)
|
cfg.InstanceName, cfg.LogRemoteURL != nil)
|
||||||
cfg.BlocklistAction, cfg.BlocklistLimitPercent = env.action(
|
cfg.BlocklistAction, cfg.BlocklistLimitPercent = env.action(
|
||||||
"SWWAF_BLOCKLIST_ACTION", "deny")
|
"SWWAF_BLOCKLIST_ACTION", "deny")
|
||||||
|
cfg.ReputationAction, cfg.ReputationLimitPercent = env.action(
|
||||||
|
"SWWAF_REPUTATION_ACTION", "limit:25")
|
||||||
|
|
||||||
env.checkInstanceNameForNtfy(cfg.InstanceName, cfg.AlertNtfyURL != nil)
|
env.checkInstanceNameForNtfy(cfg.InstanceName, cfg.AlertNtfyURL != nil)
|
||||||
env.checkLookupDBPath(cfg)
|
env.checkLookupDBPath(cfg)
|
||||||
@@ -733,9 +759,9 @@ func (e *environment) refresh(name, defaultValue string) time.Duration {
|
|||||||
return duration
|
return duration
|
||||||
}
|
}
|
||||||
|
|
||||||
// action reads a setting that is what is done with a client a list names:
|
// action reads a setting that is what is done with a client a blocklist
|
||||||
// deny, log, or limit:<percent>, which it returns as limit and the
|
// or a DNSBL zone lists: deny, log, or limit:<percent>, which it returns
|
||||||
// percentage.
|
// as limit and the percentage.
|
||||||
func (e *environment) action(name, defaultValue string) (string, int64) {
|
func (e *environment) action(name, defaultValue string) (string, int64) {
|
||||||
value := e.value(name, defaultValue)
|
value := e.value(name, defaultValue)
|
||||||
if value == "deny" || value == "log" {
|
if value == "deny" || value == "log" {
|
||||||
@@ -752,6 +778,33 @@ func (e *environment) action(name, defaultValue string) (string, int64) {
|
|||||||
return "limit", percent
|
return "limit", percent
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// zones reads the setting that is the list of DNSBL zones. It is empty by
|
||||||
|
// default. The log shows each zone with its key masked, as MaskZoneKey
|
||||||
|
// masks it.
|
||||||
|
func (e *environment) zones(name string) []string {
|
||||||
|
value, _ := e.lookup(name)
|
||||||
|
zones, err := parseZones(value)
|
||||||
|
e.check(name, err)
|
||||||
|
|
||||||
|
logged := make([]string, len(zones))
|
||||||
|
for i, zone := range zones {
|
||||||
|
logged[i] = MaskZoneKey(zone)
|
||||||
|
}
|
||||||
|
|
||||||
|
e.settings = append(e.settings, slog.String(name, strings.Join(logged, ",")))
|
||||||
|
|
||||||
|
return zones
|
||||||
|
}
|
||||||
|
|
||||||
|
// resolver reads the setting that is the resolver the DNSBL zones are
|
||||||
|
// asked through, the zero AddrPort while it is unset or empty.
|
||||||
|
func (e *environment) resolver(name string) netip.AddrPort {
|
||||||
|
resolver, err := parseResolver(e.value(name, ""))
|
||||||
|
e.check(name, err)
|
||||||
|
|
||||||
|
return resolver
|
||||||
|
}
|
||||||
|
|
||||||
// lookupSource reads the setting that is where clients are looked up:
|
// lookupSource reads the setting that is where clients are looked up:
|
||||||
// geojs, file, or off.
|
// geojs, file, or off.
|
||||||
func (e *environment) lookupSource(name, defaultValue string) string {
|
func (e *environment) lookupSource(name, defaultValue string) string {
|
||||||
@@ -1697,6 +1750,113 @@ func parseListURLs(value string) ([]string, error) {
|
|||||||
return urls, nil
|
return urls, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const (
|
||||||
|
// maxZoneLength is the most characters a DNSBL zone may have: 253, the
|
||||||
|
// most a DNS name may have, less the 64 that come before the zone in
|
||||||
|
// the name an IPv6 client is asked about by, its 32 hex digits each
|
||||||
|
// followed by a dot.
|
||||||
|
maxZoneLength = 189
|
||||||
|
// maxLabelLength is the most characters a label of a DNS name may have.
|
||||||
|
maxLabelLength = 63
|
||||||
|
// labelChars are the characters a label of a DNS zone may hold.
|
||||||
|
labelChars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-"
|
||||||
|
// dnsPort is the port a resolver is asked on when SWWAF_DNSBL_RESOLVER
|
||||||
|
// gives none.
|
||||||
|
dnsPort = 53
|
||||||
|
)
|
||||||
|
|
||||||
|
// parseZones reads a comma-separated list of DNSBL zones, each a DNS name
|
||||||
|
// such as dnsbl.dronebl.org: labels separated by dots, each of 1 to 63
|
||||||
|
// letters, digits and hyphens, neither starting nor ending with a hyphen,
|
||||||
|
// and at most maxZoneLength characters in all. Go's resolver takes any
|
||||||
|
// other name for one that does not exist, so that the zone would list no
|
||||||
|
// client. A zone listed twice is an error, whatever the case of its
|
||||||
|
// letters, which DNS names ignore, and whatever its key, since
|
||||||
|
// MaskZoneKey shows two keys of one zone alike. An error shows a zone as
|
||||||
|
// MaskZoneKey does.
|
||||||
|
func parseZones(value string) ([]string, error) {
|
||||||
|
zones, err := parseList(value)
|
||||||
|
if err != nil {
|
||||||
|
// parseList's error, for an empty item, shows the whole value, keys
|
||||||
|
// included.
|
||||||
|
return nil, errEmptyItem
|
||||||
|
}
|
||||||
|
|
||||||
|
for i, zone := range zones {
|
||||||
|
shown := MaskZoneKey(zone)
|
||||||
|
listedBefore := slices.ContainsFunc(zones[:i], func(earlier string) bool {
|
||||||
|
return strings.EqualFold(MaskZoneKey(earlier), shown)
|
||||||
|
})
|
||||||
|
|
||||||
|
switch {
|
||||||
|
case len(zone) > maxZoneLength:
|
||||||
|
return nil, fmt.Errorf("%q %w", shown, errZoneTooLong)
|
||||||
|
case !isZone(zone):
|
||||||
|
return nil, fmt.Errorf("%q %w", shown, errNotZone)
|
||||||
|
case listedBefore:
|
||||||
|
return nil, fmt.Errorf("%q %w", shown, errListedTwice)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return zones, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// MaskZoneKey returns zone with ******** in place of its key, if it is a
|
||||||
|
// zone of Spamhaus's keyed query service, a name under dq.spamhaus.net,
|
||||||
|
// such as <key>.xbl.dq.spamhaus.net, whose first label is the key. Any
|
||||||
|
// other zone it returns as it is. A zone is shown so wherever it leaves
|
||||||
|
// the process: in the log, the alerts and the metrics.
|
||||||
|
func MaskZoneKey(zone string) string {
|
||||||
|
// DNS names ignore case, and a name may be written with a dot at its
|
||||||
|
// end.
|
||||||
|
name := strings.TrimSuffix(strings.ToLower(zone), ".")
|
||||||
|
if !strings.HasSuffix(name, ".dq.spamhaus.net") {
|
||||||
|
return zone
|
||||||
|
}
|
||||||
|
|
||||||
|
_, rest, _ := strings.Cut(zone, ".")
|
||||||
|
|
||||||
|
return masked + "." + rest
|
||||||
|
}
|
||||||
|
|
||||||
|
// isZone reports whether each label of zone is as parseZones takes it.
|
||||||
|
func isZone(zone string) bool {
|
||||||
|
for label := range strings.SplitSeq(zone, ".") {
|
||||||
|
badChar := strings.ContainsFunc(label, func(char rune) bool {
|
||||||
|
return !strings.ContainsRune(labelChars, char)
|
||||||
|
})
|
||||||
|
|
||||||
|
if label == "" || len(label) > maxLabelLength || badChar ||
|
||||||
|
strings.HasPrefix(label, "-") || strings.HasSuffix(label, "-") {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseResolver reads the resolver the DNSBL zones are asked through: an
|
||||||
|
// IP address with a port from 1 to 65535, such as 192.0.2.53:5353 or
|
||||||
|
// [2001:db8::53]:5353, or without one, such as 192.0.2.53 or 2001:db8::53,
|
||||||
|
// for port 53. An empty value is none, the zero AddrPort.
|
||||||
|
func parseResolver(value string) (netip.AddrPort, error) {
|
||||||
|
if value == "" {
|
||||||
|
return netip.AddrPort{}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
resolver, err := netip.ParseAddrPort(value)
|
||||||
|
if err == nil && resolver.Port() != 0 {
|
||||||
|
return resolver, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
addr, err := netip.ParseAddr(value)
|
||||||
|
if err != nil {
|
||||||
|
return netip.AddrPort{}, fmt.Errorf("%q %w", value, errNotResolver)
|
||||||
|
}
|
||||||
|
|
||||||
|
return netip.AddrPortFrom(addr, dnsPort), nil
|
||||||
|
}
|
||||||
|
|
||||||
// parseWebhookHeaders reads a comma-separated list of headers, each its
|
// parseWebhookHeaders reads a comma-separated list of headers, each its
|
||||||
// name, :, and its value, and returns them, and how the log shows them,
|
// name, :, and its value, and returns them, and how the log shows them,
|
||||||
// with each value as ********. An error names the item by its place in
|
// with each value as ********. An error names the item by its place in
|
||||||
|
|||||||
@@ -61,6 +61,11 @@ const (
|
|||||||
blocklistURLs = "SWWAF_BLOCKLIST_URLS"
|
blocklistURLs = "SWWAF_BLOCKLIST_URLS"
|
||||||
blocklistRefresh = "SWWAF_BLOCKLIST_REFRESH"
|
blocklistRefresh = "SWWAF_BLOCKLIST_REFRESH"
|
||||||
blocklistAction = "SWWAF_BLOCKLIST_ACTION"
|
blocklistAction = "SWWAF_BLOCKLIST_ACTION"
|
||||||
|
dnsblZones = "SWWAF_DNSBL_ZONES"
|
||||||
|
dnsblResolver = "SWWAF_DNSBL_RESOLVER"
|
||||||
|
reputationAction = "SWWAF_REPUTATION_ACTION"
|
||||||
|
reputationCacheTTL = "SWWAF_REPUTATION_CACHE_TTL"
|
||||||
|
reputationTimeout = "SWWAF_REPUTATION_TIMEOUT"
|
||||||
banResponse = "SWWAF_BAN_RESPONSE"
|
banResponse = "SWWAF_BAN_RESPONSE"
|
||||||
limitBanDuration = "SWWAF_LIMIT_BAN_DURATION"
|
limitBanDuration = "SWWAF_LIMIT_BAN_DURATION"
|
||||||
limitBanRepeatWindow = "SWWAF_LIMIT_BAN_REPEAT_WINDOW"
|
limitBanRepeatWindow = "SWWAF_LIMIT_BAN_REPEAT_WINDOW"
|
||||||
@@ -152,6 +157,9 @@ const (
|
|||||||
defaultAlertCooldown = "15m"
|
defaultAlertCooldown = "15m"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// defaultReputationCacheTTL is the default of SWWAF_REPUTATION_CACHE_TTL.
|
||||||
|
const defaultReputationCacheTTL = "24h"
|
||||||
|
|
||||||
// defaultLogRequestHeaders is the default of SWWAF_LOG_REQUEST_HEADERS.
|
// defaultLogRequestHeaders is the default of SWWAF_LOG_REQUEST_HEADERS.
|
||||||
const defaultLogRequestHeaders = "accept,accept-language,accept-encoding," +
|
const defaultLogRequestHeaders = "accept,accept-language,accept-encoding," +
|
||||||
"content-type,origin,range"
|
"content-type,origin,range"
|
||||||
@@ -1322,6 +1330,202 @@ func TestASNLimitPercentURLThatIsABlocklistStopsTheStart(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// dronebl is a DNSBL zone, and spamhaus one of Spamhaus's, a name
|
||||||
|
// containing spamhausKey, the key of its keyed query service, which the
|
||||||
|
// log shows as spamhausMasked.
|
||||||
|
const (
|
||||||
|
dronebl = "dnsbl.dronebl.org"
|
||||||
|
spamhausKey = "abcdefghijklmnopqrstuvwxyz"
|
||||||
|
spamhaus = spamhausKey + ".xbl.dq.spamhaus.net"
|
||||||
|
spamhausMasked = "********.xbl.dq.spamhaus.net"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestDNSBLSettingsAsSet(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
cfg := fromEnvironment(t, environment{})
|
||||||
|
if len(cfg.DNSBLZones) != 0 || cfg.DNSBLResolver.IsValid() ||
|
||||||
|
cfg.ReputationAction != actionLimit || cfg.ReputationLimitPercent != 25 ||
|
||||||
|
cfg.ReputationCacheTTL != 24*time.Hour || cfg.ReputationTimeout != 2*time.Second {
|
||||||
|
t.Errorf("by default, the zones %v, the resolver %s, the action %s:%d, the TTL %s "+
|
||||||
|
"and the timeout %s, want no zone, no resolver, limit:25, 24h and 2s",
|
||||||
|
cfg.DNSBLZones,
|
||||||
|
cfg.DNSBLResolver, cfg.ReputationAction, cfg.ReputationLimitPercent,
|
||||||
|
cfg.ReputationCacheTTL, cfg.ReputationTimeout)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The longest zone allowed, of 189 characters, with labels of 63, the
|
||||||
|
// longest allowed.
|
||||||
|
longest := strings.Repeat("a", 63) + "." + strings.Repeat("b", 63) + "." +
|
||||||
|
strings.Repeat("c", 61)
|
||||||
|
|
||||||
|
for _, tc := range []struct {
|
||||||
|
zones, resolver, action string
|
||||||
|
// want are the zones, resolver, action and percent Config gives.
|
||||||
|
want []string
|
||||||
|
wantResolver string
|
||||||
|
wantAction string
|
||||||
|
wantPercent int64
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
dronebl + ", " + spamhaus, "192.0.2.53", actionDeny,
|
||||||
|
[]string{dronebl, spamhaus}, "192.0.2.53:53", actionDeny, 0,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
longest, "192.0.2.53:5353", actionLog,
|
||||||
|
[]string{longest}, "192.0.2.53:5353", actionLog, 0,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Dnsbl-1.Example", "2001:db8::53", "limit:10",
|
||||||
|
[]string{"Dnsbl-1.Example"}, "[2001:db8::53]:53", actionLimit, 10,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
dronebl, "[2001:db8::53]:5353", "limit:0",
|
||||||
|
[]string{dronebl}, "[2001:db8::53]:5353", actionLimit, 0,
|
||||||
|
},
|
||||||
|
} {
|
||||||
|
cfg := fromEnvironment(t, environment{
|
||||||
|
dnsblZones: tc.zones, dnsblResolver: tc.resolver, reputationAction: tc.action,
|
||||||
|
reputationCacheTTL: "12h", reputationTimeout: "3s",
|
||||||
|
})
|
||||||
|
|
||||||
|
if !slices.Equal(cfg.DNSBLZones, tc.want) ||
|
||||||
|
cfg.DNSBLResolver.String() != tc.wantResolver ||
|
||||||
|
cfg.ReputationAction != tc.wantAction ||
|
||||||
|
cfg.ReputationLimitPercent != tc.wantPercent ||
|
||||||
|
cfg.ReputationCacheTTL != 12*time.Hour ||
|
||||||
|
cfg.ReputationTimeout != 3*time.Second {
|
||||||
|
t.Errorf("%s=%s, %s=%s and %s=%s gave %v, %s, %s:%d, %s and %s", dnsblZones,
|
||||||
|
tc.zones, dnsblResolver, tc.resolver, reputationAction, tc.action,
|
||||||
|
cfg.DNSBLZones, cfg.DNSBLResolver, cfg.ReputationAction,
|
||||||
|
cfg.ReputationLimitPercent, cfg.ReputationCacheTTL, cfg.ReputationTimeout)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestInvalidDNSBLSettingStopsTheStartSayingWhatIsWrong(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const (
|
||||||
|
notZone = " is not a DNS zone such as dnsbl.dronebl.org"
|
||||||
|
notResolver = " is not an IP address with an optional port, such as 192.0.2.53 " +
|
||||||
|
"or [2001:db8::53]:5353"
|
||||||
|
notAboveZero = " is not a duration above zero, such as 1h or 7d"
|
||||||
|
)
|
||||||
|
|
||||||
|
label64 := strings.Repeat("a", 64) + ".example"
|
||||||
|
tooLong := strings.Repeat("a", 63) + "." + strings.Repeat("b", 63) + "." +
|
||||||
|
strings.Repeat("c", 62)
|
||||||
|
|
||||||
|
for _, tc := range []struct{ name, value, want string }{
|
||||||
|
{dnsblZones, "dnsbl..example", `"dnsbl..example"` + notZone},
|
||||||
|
{dnsblZones, "dnsbl.example.", `"dnsbl.example."` + notZone},
|
||||||
|
{dnsblZones, "-dnsbl.example", `"-dnsbl.example"` + notZone},
|
||||||
|
{dnsblZones, "dnsbl-.example", `"dnsbl-.example"` + notZone},
|
||||||
|
{dnsblZones, "dns_bl.example", `"dns_bl.example"` + notZone},
|
||||||
|
{dnsblZones, label64, `"` + label64 + `"` + notZone},
|
||||||
|
{
|
||||||
|
dnsblZones, tooLong,
|
||||||
|
`"` + tooLong + `" is longer than 189 characters, too long for the names ` +
|
||||||
|
`IPv6 clients are asked about by`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
dnsblZones, dronebl + "," + spamhaus + "," + dronebl,
|
||||||
|
`"` + dronebl + `" is listed twice`,
|
||||||
|
},
|
||||||
|
// DNS names ignore case.
|
||||||
|
{dnsblZones, "dnsbl.example,DNSBL.example", `"DNSBL.example" is listed twice`},
|
||||||
|
{dnsblResolver, "resolver.example", `"resolver.example"` + notResolver},
|
||||||
|
{dnsblResolver, "192.0.2.53:0", `"192.0.2.53:0"` + notResolver},
|
||||||
|
{dnsblResolver, "192.0.2.53:65536", `"192.0.2.53:65536"` + notResolver},
|
||||||
|
{dnsblResolver, "[2001:db8::53]", `"[2001:db8::53]"` + notResolver},
|
||||||
|
{
|
||||||
|
reputationAction, "ban",
|
||||||
|
`"ban" is not deny, limit:<percent> such as limit:25, or log`,
|
||||||
|
},
|
||||||
|
{reputationCacheTTL, off, `"off"` + notAboveZero},
|
||||||
|
{reputationTimeout, "0s", `"0s"` + notAboveZero},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
|
||||||
|
|
||||||
|
want := tc.name + ": " + tc.want
|
||||||
|
if err == nil || err.Error() != want {
|
||||||
|
t.Errorf("error %v, want %s", err, want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMaskZoneKeyMasksTheFirstLabelOfAZoneUnderDqSpamhausNet(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
for zone, want := range map[string]string{
|
||||||
|
spamhaus: spamhausMasked,
|
||||||
|
spamhaus + ".": spamhausMasked + ".",
|
||||||
|
"KEY.ZEN.DQ.SPAMHAUS.NET": "********.ZEN.DQ.SPAMHAUS.NET",
|
||||||
|
dronebl: dronebl,
|
||||||
|
"dq.spamhaus.net": "dq.spamhaus.net",
|
||||||
|
spamhaus + ".example": spamhaus + ".example",
|
||||||
|
} {
|
||||||
|
if got := config.MaskZoneKey(zone); got != want {
|
||||||
|
t.Errorf("MaskZoneKey(%q) is %q, want %q", zone, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDNSBLZoneKeyIsLoggedMaskedAndNeverShown(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
cfg := fromEnvironment(t, environment{dnsblZones: dronebl + ", " + spamhaus})
|
||||||
|
|
||||||
|
var out bytes.Buffer
|
||||||
|
|
||||||
|
slog.New(slog.NewJSONHandler(&out, nil)).Info("starting", "settings", cfg)
|
||||||
|
|
||||||
|
logged := out.String()
|
||||||
|
if strings.Contains(logged, spamhausKey) ||
|
||||||
|
!strings.Contains(logged, `"`+dnsblZones+`":"`+dronebl+","+spamhausMasked+`"`) {
|
||||||
|
t.Errorf("the zones are not logged with the key masked: %s", logged)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Nor does an error that stops the start show a key, in any case.
|
||||||
|
const (
|
||||||
|
notZone = " is not a DNS zone such as dnsbl.dronebl.org"
|
||||||
|
otherKey = "zyxwvutsrqponmlkjihgfedcba"
|
||||||
|
otherZone = otherKey + ".xbl.dq.spamhaus.net"
|
||||||
|
)
|
||||||
|
|
||||||
|
// 205 characters, 187 with the key masked.
|
||||||
|
labels := strings.Repeat("a", 63) + "." + strings.Repeat("b", 63) + "." +
|
||||||
|
strings.Repeat("c", 30) + ".xbl.dq.spamhaus.net"
|
||||||
|
|
||||||
|
for _, tc := range []struct{ value, want string }{
|
||||||
|
{spamhaus + ".", `"` + spamhausMasked + `."` + notZone},
|
||||||
|
{spamhausKey + "_.xbl.dq.spamhaus.net", `"` + spamhausMasked + `"` + notZone},
|
||||||
|
{
|
||||||
|
spamhausKey + "." + labels,
|
||||||
|
`"********.` + labels + `" is longer than 189 characters, too long ` +
|
||||||
|
`for the names IPv6 clients are asked about by`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
spamhaus + "," + strings.ToUpper(spamhaus),
|
||||||
|
`"********.XBL.DQ.SPAMHAUS.NET" is listed twice`,
|
||||||
|
},
|
||||||
|
{spamhaus + "," + otherZone, `"` + spamhausMasked + `" is listed twice`},
|
||||||
|
{spamhaus + ",,", "has an empty item in its list"},
|
||||||
|
} {
|
||||||
|
_, err := config.FromEnvironment(environment{dnsblZones: tc.value}.lookupEnv)
|
||||||
|
|
||||||
|
want := dnsblZones + ": " + tc.want
|
||||||
|
if err == nil || err.Error() != want {
|
||||||
|
t.Errorf("%s=%s gave the error %v, want %s", dnsblZones, tc.value, err, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestSizesAndOff(t *testing.T) {
|
func TestSizesAndOff(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -1704,6 +1908,7 @@ func writeFile(t *testing.T, contents string) string {
|
|||||||
return path
|
return path
|
||||||
}
|
}
|
||||||
|
|
||||||
|
//nolint:funlen // one line for each setting, a list that grows with them
|
||||||
func TestLogsEachSettingWithItsValue(t *testing.T) {
|
func TestLogsEachSettingWithItsValue(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -1749,6 +1954,11 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
|||||||
blocklistURLs: "",
|
blocklistURLs: "",
|
||||||
blocklistRefresh: "24h",
|
blocklistRefresh: "24h",
|
||||||
blocklistAction: actionDeny,
|
blocklistAction: actionDeny,
|
||||||
|
dnsblZones: "",
|
||||||
|
dnsblResolver: "",
|
||||||
|
reputationAction: "limit:25",
|
||||||
|
reputationCacheTTL: defaultReputationCacheTTL,
|
||||||
|
reputationTimeout: "2s",
|
||||||
banResponse: "403",
|
banResponse: "403",
|
||||||
limitBanDuration: "1h",
|
limitBanDuration: "1h",
|
||||||
limitBanRepeatWindow: "24h",
|
limitBanRepeatWindow: "24h",
|
||||||
|
|||||||
+42
-12
@@ -14,6 +14,7 @@ import (
|
|||||||
"github.com/prometheus/client_golang/prometheus/promhttp"
|
"github.com/prometheus/client_golang/prometheus/promhttp"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/remotelog"
|
"sneak.berlin/go/smallwebwaf/internal/remotelog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
@@ -256,24 +257,53 @@ func (m *Metrics) AddLookupFile(lastRead func() time.Time, readFailures func() i
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
// AddReputation adds the metrics of the lists fetched from URLs, by
|
// AddReputation adds the metrics of the lists fetched from URLs and of the
|
||||||
// source, each list's URL: the requests whose client a blocklist lists,
|
// DNSBL zones, by source, each list's URL or each zone, its key masked as
|
||||||
// which ReputationHit counts, and, read from lists as the metrics are
|
// config.MaskZoneKey masks it: the requests whose client a blocklist or a
|
||||||
// asked for, the fetches that failed and when the copy in use was fetched.
|
// zone's verdict lists, which ReputationHit counts, and, read from lists
|
||||||
// It is called once, before ReputationHit.
|
// and dnsbl as the metrics are asked for, for a list, the fetches that
|
||||||
func (m *Metrics) AddReputation(lists *reputation.Lists) {
|
// failed and when the copy in use was fetched, and for a zone, the queries
|
||||||
|
// made and those that failed. It is called once, before ReputationHit.
|
||||||
|
func (m *Metrics) AddReputation(lists *reputation.Lists, dnsbl *reputation.DNSBL) {
|
||||||
|
const (
|
||||||
|
sourceLabel = "source"
|
||||||
|
failuresHelp = "Fetches of the list, or queries to the DNSBL zone, that failed."
|
||||||
|
)
|
||||||
|
|
||||||
m.reputationHits = counterVec("smallwebwaf_reputation_hits_total",
|
m.reputationHits = counterVec("smallwebwaf_reputation_hits_total",
|
||||||
"Requests whose client a blocklist lists, by the blocklist's URL.",
|
"Requests whose client a blocklist or a DNSBL zone lists, by the "+
|
||||||
[]string{"source"})
|
"blocklist's URL or the zone.",
|
||||||
|
[]string{sourceLabel})
|
||||||
m.registry.MustRegister(m.reputationHits)
|
m.registry.MustRegister(m.reputationHits)
|
||||||
|
|
||||||
|
for _, zone := range dnsbl.Zones() {
|
||||||
|
source := prometheus.Labels{sourceLabel: config.MaskZoneKey(zone)}
|
||||||
|
|
||||||
|
m.registry.MustRegister(
|
||||||
|
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
||||||
|
Name: "smallwebwaf_reputation_queries_total",
|
||||||
|
Help: "Queries to the DNSBL zone.",
|
||||||
|
ConstLabels: source,
|
||||||
|
}, func() float64 {
|
||||||
|
return float64(dnsbl.Queries(zone))
|
||||||
|
}),
|
||||||
|
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
||||||
|
Name: "smallwebwaf_reputation_failures_total",
|
||||||
|
Help: failuresHelp,
|
||||||
|
ConstLabels: source,
|
||||||
|
}, func() float64 {
|
||||||
|
return float64(dnsbl.Failures(zone))
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
for _, listURL := range lists.URLs() {
|
for _, listURL := range lists.URLs() {
|
||||||
source := prometheus.Labels{"source": listURL}
|
source := prometheus.Labels{sourceLabel: listURL}
|
||||||
|
|
||||||
m.registry.MustRegister(
|
m.registry.MustRegister(
|
||||||
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
||||||
Name: "smallwebwaf_reputation_failures_total",
|
Name: "smallwebwaf_reputation_failures_total",
|
||||||
Help: "Fetches of the list that failed.",
|
Help: failuresHelp,
|
||||||
ConstLabels: source,
|
ConstLabels: source,
|
||||||
}, func() float64 {
|
}, func() float64 {
|
||||||
return float64(lists.Failures(listURL))
|
return float64(lists.Failures(listURL))
|
||||||
@@ -295,8 +325,8 @@ func (m *Metrics) AddReputation(lists *reputation.Lists) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// ReputationHit counts a request whose client the blocklist at source, its
|
// ReputationHit counts a request whose client source lists: a blocklist,
|
||||||
// URL, lists.
|
// by its URL, or a DNSBL zone, its key masked.
|
||||||
func (m *Metrics) ReputationHit(source string) {
|
func (m *Metrics) ReputationHit(source string) {
|
||||||
m.reputationHits.WithLabelValues(source).Inc()
|
m.reputationHits.WithLabelValues(source).Inc()
|
||||||
}
|
}
|
||||||
|
|||||||
+22
-15
@@ -28,16 +28,18 @@ func biasedThresholdsSet(cfg *config.Config) bool {
|
|||||||
|
|
||||||
// limitPercentages returns the client's limit percentages, for the rate
|
// limitPercentages returns the client's limit percentages, for the rate
|
||||||
// limits and for the byte limits, by its AS number and country as looked
|
// limits and for the byte limits, by its AS number and country as looked
|
||||||
// up, each "" when unknown, and the blocklists that list it. Each is the
|
// up, each "" when unknown, and the blocklists and DNSBL zones that list
|
||||||
// lowest of those the settings give it, the first of them in the order
|
// it. Each is the lowest of those the settings give it, the first of them
|
||||||
// below when several are lowest: the percentage SWWAF_ASN_LIMIT_PERCENT
|
// in the order below when several are lowest: the percentage
|
||||||
// gives its AS number, the one the file SWWAF_ASN_LIMIT_PERCENT_URL names
|
// SWWAF_ASN_LIMIT_PERCENT gives its AS number, the one the file
|
||||||
// gives it, the one SWWAF_COUNTRY_LIMIT_PERCENT gives its country, for a
|
// SWWAF_ASN_LIMIT_PERCENT_URL names gives it, the one
|
||||||
// client without a country, SWWAF_UNKNOWN_LIMIT_PERCENT, and for a client
|
// SWWAF_COUNTRY_LIMIT_PERCENT gives its country, for a client without a
|
||||||
// a blocklist lists, the percentage of SWWAF_BLOCKLIST_ACTION while it is
|
// country, SWWAF_UNKNOWN_LIMIT_PERCENT, for a client a blocklist lists,
|
||||||
// limit. For the byte limits, SWWAF_ASN_BYTES_PERCENT and
|
// the percentage of SWWAF_BLOCKLIST_ACTION while it is limit, and for a
|
||||||
// SWWAF_COUNTRY_BYTES_PERCENT take the place of the first three for an AS
|
// client a DNSBL zone's verdict lists, the percentage of
|
||||||
// number or a country they list.
|
// SWWAF_REPUTATION_ACTION while it is limit. For the byte limits,
|
||||||
|
// SWWAF_ASN_BYTES_PERCENT and SWWAF_COUNTRY_BYTES_PERCENT take the place
|
||||||
|
// of the first three for an AS number or a country they list.
|
||||||
func (rq *request) limitPercentages() (percentage, percentage) {
|
func (rq *request) limitPercentages() (percentage, percentage) {
|
||||||
cfg := rq.h.config
|
cfg := rq.h.config
|
||||||
asn, country := rq.line.ASN, rq.line.Country
|
asn, country := rq.line.ASN, rq.line.Country
|
||||||
@@ -52,9 +54,14 @@ func (rq *request) limitPercentages() (percentage, percentage) {
|
|||||||
fetched = percentage{percent, "SWWAF_ASN_LIMIT_PERCENT_URL"}
|
fetched = percentage{percent, "SWWAF_ASN_LIMIT_PERCENT_URL"}
|
||||||
}
|
}
|
||||||
|
|
||||||
listed := percentage{percent: whole}
|
blocklisted := percentage{percent: whole}
|
||||||
if len(rq.line.Reputation) > 0 && cfg.BlocklistAction == "limit" {
|
if rq.blocklisted && cfg.BlocklistAction == "limit" {
|
||||||
listed = percentage{cfg.BlocklistLimitPercent, "SWWAF_BLOCKLIST_ACTION"}
|
blocklisted = percentage{cfg.BlocklistLimitPercent, "SWWAF_BLOCKLIST_ACTION"}
|
||||||
|
}
|
||||||
|
|
||||||
|
dnsblListed := percentage{percent: whole}
|
||||||
|
if rq.dnsblListed && cfg.ReputationAction == "limit" {
|
||||||
|
dnsblListed = percentage{cfg.ReputationLimitPercent, "SWWAF_REPUTATION_ACTION"}
|
||||||
}
|
}
|
||||||
|
|
||||||
asnRequests := lowest(given(cfg.ASNLimitPercent, asn, "SWWAF_ASN_LIMIT_PERCENT"),
|
asnRequests := lowest(given(cfg.ASNLimitPercent, asn, "SWWAF_ASN_LIMIT_PERCENT"),
|
||||||
@@ -71,8 +78,8 @@ func (rq *request) limitPercentages() (percentage, percentage) {
|
|||||||
countryBytes = given(cfg.CountryBytesPercent, country, "SWWAF_COUNTRY_BYTES_PERCENT")
|
countryBytes = given(cfg.CountryBytesPercent, country, "SWWAF_COUNTRY_BYTES_PERCENT")
|
||||||
}
|
}
|
||||||
|
|
||||||
return lowest(asnRequests, countryRequests, unknown, listed),
|
return lowest(asnRequests, countryRequests, unknown, blocklisted, dnsblListed),
|
||||||
lowest(asnBytes, countryBytes, unknown, listed)
|
lowest(asnBytes, countryBytes, unknown, blocklisted, dnsblListed)
|
||||||
}
|
}
|
||||||
|
|
||||||
// given returns the percentage percents, the setting named setting, gives
|
// given returns the percentage percents, the setting named setting, gives
|
||||||
|
|||||||
+31
-11
@@ -71,15 +71,15 @@ type Params struct {
|
|||||||
Rules *rules.Files
|
Rules *rules.Files
|
||||||
// Alerts receive the alert for each ban the proxy makes or makes
|
// Alerts receive the alert for each ban the proxy makes or makes
|
||||||
// permanent, for each count over an anomaly threshold, for each request
|
// permanent, for each count over an anomaly threshold, for each request
|
||||||
// whose client a blocklist lists, and for GeoJS failing or a fetch of a
|
// whose client a blocklist or a DNSBL zone lists, and for GeoJS failing,
|
||||||
// list failing.
|
// a fetch of a list failing or a query to a DNSBL zone failing.
|
||||||
Alerts *alerts.Queue
|
Alerts *alerts.Queue
|
||||||
}
|
}
|
||||||
|
|
||||||
// Server is the server smallwebwaf runs, with the parts of the proxy
|
// Server is the server smallwebwaf runs, with the parts of the proxy
|
||||||
// whose state the state files keep, the lookup database, nil unless
|
// whose state the state files keep, the lookup database, nil unless
|
||||||
// SWWAF_LOOKUP_SOURCE is file, the lists fetched from URLs, which its Run
|
// SWWAF_LOOKUP_SOURCE is file, the lists fetched from URLs, which its Run
|
||||||
// fetches, and the metrics.
|
// fetches, the DNSBL zones' verdicts, and the metrics.
|
||||||
type Server struct {
|
type Server struct {
|
||||||
*http.Server
|
*http.Server
|
||||||
|
|
||||||
@@ -89,6 +89,7 @@ type Server struct {
|
|||||||
Anomalies *anomaly.Counters
|
Anomalies *anomaly.Counters
|
||||||
LookupFile *lookup.File
|
LookupFile *lookup.File
|
||||||
Lists *reputation.Lists
|
Lists *reputation.Lists
|
||||||
|
DNSBL *reputation.DNSBL
|
||||||
Metrics *metrics.Metrics
|
Metrics *metrics.Metrics
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -101,6 +102,7 @@ type Server struct {
|
|||||||
func New(params Params) *Server {
|
func New(params Params) *Server {
|
||||||
errorLog := slog.NewLogLogger(params.ProcessLog.Handler(), slog.LevelWarn)
|
errorLog := slog.NewLogLogger(params.ProcessLog.Handler(), slog.LevelWarn)
|
||||||
m := metrics.New(params.Config.MetricsTopN, params.Config.InstanceName)
|
m := metrics.New(params.Config.MetricsTopN, params.Config.InstanceName)
|
||||||
|
lists, dnsbl := newReputation(params)
|
||||||
h := &handler{
|
h := &handler{
|
||||||
config: params.Config,
|
config: params.Config,
|
||||||
requestLog: params.RequestLog,
|
requestLog: params.RequestLog,
|
||||||
@@ -136,13 +138,10 @@ func New(params Params) *Server {
|
|||||||
Alerts: params.Alerts,
|
Alerts: params.Alerts,
|
||||||
}),
|
}),
|
||||||
lookupFile: params.LookupFile,
|
lookupFile: params.LookupFile,
|
||||||
lists: reputation.New(reputation.Params{
|
lists: lists,
|
||||||
BlocklistURLs: params.Config.BlocklistURLs, Refresh: params.Config.BlocklistRefresh,
|
dnsbl: dnsbl,
|
||||||
ASNLimitPercentURL: params.Config.ASNLimitPercentURL, Now: params.Now,
|
rules: params.Rules,
|
||||||
ProcessLog: params.ProcessLog, Alerts: params.Alerts,
|
alerts: params.Alerts,
|
||||||
}),
|
|
||||||
rules: params.Rules,
|
|
||||||
alerts: params.Alerts,
|
|
||||||
}
|
}
|
||||||
h.geojs = lookup.New(lookup.Params{
|
h.geojs = lookup.New(lookup.Params{
|
||||||
URL: params.GeoJSURL,
|
URL: params.GeoJSURL,
|
||||||
@@ -160,7 +159,7 @@ func New(params Params) *Server {
|
|||||||
})
|
})
|
||||||
m.AddBansAndClients(h.ledger, h.limiter, params.Now)
|
m.AddBansAndClients(h.ledger, h.limiter, params.Now)
|
||||||
m.AddRules(params.Rules)
|
m.AddRules(params.Rules)
|
||||||
m.AddReputation(h.lists)
|
m.AddReputation(h.lists, h.dnsbl)
|
||||||
|
|
||||||
return &Server{
|
return &Server{
|
||||||
Server: &http.Server{
|
Server: &http.Server{
|
||||||
@@ -181,10 +180,30 @@ func New(params Params) *Server {
|
|||||||
Anomalies: h.anomalies,
|
Anomalies: h.anomalies,
|
||||||
LookupFile: h.lookupFile,
|
LookupFile: h.lookupFile,
|
||||||
Lists: h.lists,
|
Lists: h.lists,
|
||||||
|
DNSBL: h.dnsbl,
|
||||||
Metrics: m,
|
Metrics: m,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// newReputation returns the lists fetched from URLs and the DNSBL zones'
|
||||||
|
// verdicts, as the settings in params name them, with none fetched or
|
||||||
|
// asked for yet.
|
||||||
|
func newReputation(params Params) (*reputation.Lists, *reputation.DNSBL) {
|
||||||
|
cfg := params.Config
|
||||||
|
lists := reputation.New(reputation.Params{
|
||||||
|
BlocklistURLs: cfg.BlocklistURLs, Refresh: cfg.BlocklistRefresh,
|
||||||
|
ASNLimitPercentURL: cfg.ASNLimitPercentURL, Now: params.Now,
|
||||||
|
ProcessLog: params.ProcessLog, Alerts: params.Alerts,
|
||||||
|
})
|
||||||
|
dnsbl := reputation.NewDNSBL(reputation.DNSBLParams{
|
||||||
|
Zones: cfg.DNSBLZones, Resolver: cfg.DNSBLResolver, CacheTTL: cfg.ReputationCacheTTL,
|
||||||
|
Timeout: cfg.ReputationTimeout, Now: params.Now, ProcessLog: params.ProcessLog,
|
||||||
|
Alerts: params.Alerts,
|
||||||
|
})
|
||||||
|
|
||||||
|
return lists, dnsbl
|
||||||
|
}
|
||||||
|
|
||||||
// handler is the proxy. It holds what every request shares; what belongs
|
// handler is the proxy. It holds what every request shares; what belongs
|
||||||
// to one request is in a request.
|
// to one request is in a request.
|
||||||
type handler struct {
|
type handler struct {
|
||||||
@@ -201,6 +220,7 @@ type handler struct {
|
|||||||
anomalies *anomaly.Counters
|
anomalies *anomaly.Counters
|
||||||
lookupFile *lookup.File
|
lookupFile *lookup.File
|
||||||
lists *reputation.Lists
|
lists *reputation.Lists
|
||||||
|
dnsbl *reputation.DNSBL
|
||||||
rules *rules.Files
|
rules *rules.Files
|
||||||
alerts *alerts.Queue
|
alerts *alerts.Queue
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,21 +1,47 @@
|
|||||||
package proxy
|
package proxy
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"context"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
)
|
)
|
||||||
|
|
||||||
// blocklistDenied notes in the log line the URLs of the blocklists that
|
// blocklistDenied notes the blocklists that list the client, as
|
||||||
// list the client, counts each of them in the metrics and raises a
|
// noteListed does, and reports whether SWWAF_BLOCKLIST_ACTION, being deny,
|
||||||
// reputation_hit alert for it, and reports whether SWWAF_BLOCKLIST_ACTION,
|
// refuses the request. Being limit, it lowers the client's limits instead
|
||||||
// being deny, refuses the request. Being limit, it lowers the client's
|
// (see limitPercentages), and being log, it does nothing more.
|
||||||
// limits instead (see limitPercentages), and being log, it does nothing
|
|
||||||
// more.
|
|
||||||
func (rq *request) blocklistDenied() bool {
|
func (rq *request) blocklistDenied() bool {
|
||||||
listedBy := rq.h.lists.ListedBy(rq.client)
|
listedBy := rq.h.lists.ListedBy(rq.client)
|
||||||
rq.line.Reputation = listedBy
|
rq.blocklisted = len(listedBy) > 0
|
||||||
|
rq.noteListed(listedBy, "listed by a blocklist")
|
||||||
|
|
||||||
for _, listURL := range listedBy {
|
return rq.blocklisted && rq.h.config.BlocklistAction == "deny"
|
||||||
rq.h.metrics.ReputationHit(listURL)
|
}
|
||||||
|
|
||||||
|
// dnsblDenied notes the DNSBL zones whose verdict lists the client, as
|
||||||
|
// noteListed does, and reports whether SWWAF_REPUTATION_ACTION, being
|
||||||
|
// deny, refuses the request. Being limit, it lowers the client's limits
|
||||||
|
// instead (see limitPercentages), and being log, it does nothing more. A
|
||||||
|
// zone without a verdict on the client is asked about it in the
|
||||||
|
// background, and the request does not wait for the answer. ctx is the
|
||||||
|
// request's own context.
|
||||||
|
func (rq *request) dnsblDenied(ctx context.Context) bool {
|
||||||
|
listedBy := rq.h.dnsbl.ListedBy(ctx, rq.client)
|
||||||
|
rq.dnsblListed = len(listedBy) > 0
|
||||||
|
rq.noteListed(listedBy, "listed by a DNSBL zone")
|
||||||
|
|
||||||
|
return rq.dnsblListed && rq.h.config.ReputationAction == "deny"
|
||||||
|
}
|
||||||
|
|
||||||
|
// noteListed adds sources, the URLs of the blocklists or the DNSBL zones,
|
||||||
|
// their keys masked, that list the client, to the log line's reputation,
|
||||||
|
// counts each of them in the metrics, and raises a reputation_hit alert,
|
||||||
|
// with reason, for each.
|
||||||
|
func (rq *request) noteListed(sources []string, reason string) {
|
||||||
|
rq.line.Reputation = append(rq.line.Reputation, sources...)
|
||||||
|
|
||||||
|
for _, source := range sources {
|
||||||
|
rq.h.metrics.ReputationHit(source)
|
||||||
rq.h.alerts.Raise(alerts.Alert{
|
rq.h.alerts.Raise(alerts.Alert{
|
||||||
Event: alerts.EventReputationHit,
|
Event: alerts.EventReputationHit,
|
||||||
Client: rq.client,
|
Client: rq.client,
|
||||||
@@ -23,10 +49,8 @@ func (rq *request) blocklistDenied() bool {
|
|||||||
ASN: rq.line.ASN,
|
ASN: rq.line.ASN,
|
||||||
ASName: rq.line.ASName,
|
ASName: rq.line.ASName,
|
||||||
Country: rq.line.Country,
|
Country: rq.line.Country,
|
||||||
Reason: "listed by a blocklist",
|
Reason: reason,
|
||||||
Detail: map[string]any{"source": listURL},
|
Detail: map[string]any{"source": source},
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
return len(listedBy) > 0 && rq.h.config.BlocklistAction == "deny"
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"slices"
|
"slices"
|
||||||
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -21,11 +22,14 @@ const (
|
|||||||
asnLimitPercentURL = "SWWAF_ASN_LIMIT_PERCENT_URL"
|
asnLimitPercentURL = "SWWAF_ASN_LIMIT_PERCENT_URL"
|
||||||
)
|
)
|
||||||
|
|
||||||
// The actions of SWWAF_BLOCKLIST_ACTION but limit, which has a
|
// The actions of SWWAF_BLOCKLIST_ACTION and SWWAF_REPUTATION_ACTION:
|
||||||
// percentage.
|
// limitHalf gives a listed client half of every limit, and limitQuarter a
|
||||||
|
// quarter.
|
||||||
const (
|
const (
|
||||||
actionDeny = "deny"
|
actionDeny = "deny"
|
||||||
actionLog = "log"
|
actionLog = "log"
|
||||||
|
limitHalf = "limit:50"
|
||||||
|
limitQuarter = "limit:25"
|
||||||
)
|
)
|
||||||
|
|
||||||
// The lists these tests name, which are never fetched: each test puts in
|
// The lists these tests name, which are never fetched: each test puts in
|
||||||
@@ -55,7 +59,7 @@ func TestEachBlocklistActionForAListedAddressAndAListedNetblock(t *testing.T) {
|
|||||||
},
|
},
|
||||||
{
|
{
|
||||||
// Half of 4 requests a minute: the third breaks the limit.
|
// Half of 4 requests a minute: the third breaks the limit.
|
||||||
"limit:50", []int{http.StatusOK, http.StatusOK, http.StatusForbidden},
|
limitHalf, []int{http.StatusOK, http.StatusOK, http.StatusForbidden},
|
||||||
[]string{forward, forward, requestlog.ActionRateLimited},
|
[]string{forward, forward, requestlog.ActionRateLimited},
|
||||||
"50 from " + blocklistAction,
|
"50 from " + blocklistAction,
|
||||||
},
|
},
|
||||||
@@ -151,10 +155,10 @@ func TestBlocklistLimitTakesPartInTheLowestPercentageOfEveryLimit(t *testing.T)
|
|||||||
// percentText gives them, and limitHit its limit_hit.
|
// percentText gives them, and limitHit its limit_hit.
|
||||||
want, limitHit string
|
want, limitHit string
|
||||||
}{
|
}{
|
||||||
{"limit:50", asnDEQuarter, "25 from " + asnLimitPercent, minuteBytes},
|
{limitHalf, asnDEQuarter, "25 from " + asnLimitPercent, minuteBytes},
|
||||||
{"limit:25", asnDEHalf, "25 from " + blocklistAction, minuteBytes},
|
{limitQuarter, asnDEHalf, "25 from " + blocklistAction, minuteBytes},
|
||||||
// The AS number's, the first of two alike.
|
// The AS number's, the first of two alike.
|
||||||
{"limit:25", asnDEQuarter, "25 from " + asnLimitPercent, minuteBytes},
|
{limitQuarter, asnDEQuarter, "25 from " + asnLimitPercent, minuteBytes},
|
||||||
{actionLog, asnDE + ":100", none, ""},
|
{actionLog, asnDE + ":100", none, ""},
|
||||||
} {
|
} {
|
||||||
t.Run(tc.action+" "+tc.asnPercent, func(t *testing.T) {
|
t.Run(tc.action+" "+tc.asnPercent, func(t *testing.T) {
|
||||||
@@ -299,11 +303,334 @@ func TestEachBlocklistThatListsAClientRaisesAnAlertOncePerCooldownAndIsCounted(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The DNSBL settings.
|
||||||
|
const (
|
||||||
|
dnsblZones = "SWWAF_DNSBL_ZONES"
|
||||||
|
dnsblResolver = "SWWAF_DNSBL_RESOLVER"
|
||||||
|
reputationAction = "SWWAF_REPUTATION_ACTION"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The DNSBL zones these tests name, which are never asked about the
|
||||||
|
// clients the tests send requests from: each test puts in the verdicts it
|
||||||
|
// needs, as reputation.json would at start. A query a test does start is
|
||||||
|
// sent to noResolver, where nothing listens, so that none leaves the host.
|
||||||
|
const (
|
||||||
|
dnsblZone = "dnsbl.example"
|
||||||
|
otherZone = "other.example"
|
||||||
|
noResolver = "127.0.0.1:9"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestEachReputationActionForAClientADNSBLZoneLists(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
forward, denied := requestlog.ActionForward, requestlog.ActionDenied
|
||||||
|
|
||||||
|
for _, tc := range []struct {
|
||||||
|
action string
|
||||||
|
// statuses and actions are those of a listed client's three
|
||||||
|
// requests, and percent their limit_percent, as percentText gives it.
|
||||||
|
statuses []int
|
||||||
|
actions []string
|
||||||
|
percent string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
actionDeny, []int{http.StatusForbidden, http.StatusForbidden, http.StatusForbidden},
|
||||||
|
[]string{denied, denied, denied}, none,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
// Half of 4 requests a minute: the third breaks the limit.
|
||||||
|
limitHalf, []int{http.StatusOK, http.StatusOK, http.StatusForbidden},
|
||||||
|
[]string{forward, forward, requestlog.ActionRateLimited},
|
||||||
|
"50 from " + reputationAction,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
actionLog, []int{http.StatusOK, http.StatusOK, http.StatusOK},
|
||||||
|
[]string{forward, forward, forward}, none,
|
||||||
|
},
|
||||||
|
} {
|
||||||
|
t.Run(tc.action, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
s, server, _ := startWithLookups(t, map[string]string{
|
||||||
|
rateLimitPerMinute: fourAMinute, dnsblZones: dnsblZone + "," + otherZone,
|
||||||
|
dnsblResolver: noResolver, reputationAction: tc.action,
|
||||||
|
})
|
||||||
|
listedBy := map[string][]string{
|
||||||
|
fromDE: {dnsblZone, otherZone}, fromKP: {otherZone}, unplaced: nil,
|
||||||
|
}
|
||||||
|
loadVerdicts(server, listedBy)
|
||||||
|
|
||||||
|
for _, from := range []string{fromDE, fromKP} {
|
||||||
|
for i := range 3 {
|
||||||
|
line := s.get(from, tc.statuses[i], tc.actions[i])
|
||||||
|
// In the order SWWAF_DNSBL_ZONES names them.
|
||||||
|
wantReputation(t, line, listedBy[from]...)
|
||||||
|
wantPercent(t, "limit_percent", line.LimitPercent,
|
||||||
|
line.LimitPercentSetting, tc.percent)
|
||||||
|
|
||||||
|
// A request refused for the verdict is not counted.
|
||||||
|
counted := line.fields["counts"] != nil
|
||||||
|
if counted != (tc.actions[i] != denied) {
|
||||||
|
t.Errorf("request from %s counted %t, logged %s", from, counted,
|
||||||
|
tc.actions[i])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A client no zone lists has the whole limit.
|
||||||
|
for range 3 {
|
||||||
|
line := s.get(unplaced, http.StatusOK, forward)
|
||||||
|
wantReputation(t, line)
|
||||||
|
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||||
|
none)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A refusal for the verdict makes no ban, and every client had its
|
||||||
|
// verdicts, so no zone was asked.
|
||||||
|
if held := server.Ledger.Snapshot(); tc.action == actionDeny && len(held) != 0 {
|
||||||
|
t.Errorf("bans %+v, want none", held)
|
||||||
|
}
|
||||||
|
|
||||||
|
if queries := server.DNSBL.Queries(dnsblZone); queries != 0 {
|
||||||
|
t.Errorf("%d queries, want none", queries)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDNSBLZonesComeAfterTheBlocklistsAndSkipAllowNets(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
s, server, queue := startWithLookups(t, map[string]string{
|
||||||
|
blocklistURLs: dropURL, dnsblZones: dnsblZone, dnsblResolver: noResolver,
|
||||||
|
reputationAction: actionDeny, allowNets: fromDE,
|
||||||
|
})
|
||||||
|
loadLists(t, server, map[string][]string{dropURL: {fromKP}})
|
||||||
|
loadVerdicts(server, map[string][]string{fromKP: {dnsblZone}, fromDE: {dnsblZone}})
|
||||||
|
|
||||||
|
// The blocklist refuses fromKP before its verdict is looked at, and
|
||||||
|
// fromDE, in SWWAF_ALLOW_NETS, is not checked at all: neither is noted
|
||||||
|
// for the zone, nor alerted, nor asked about.
|
||||||
|
wantReputation(t, s.get(fromKP, http.StatusForbidden, requestlog.ActionDenied),
|
||||||
|
dropURL)
|
||||||
|
wantReputation(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward))
|
||||||
|
|
||||||
|
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
|
||||||
|
if len(waiting) != 1 || waiting[0].Detail["source"] != dropURL {
|
||||||
|
t.Errorf("alerts waiting %+v, want the blocklist's reputation_hit alone", waiting)
|
||||||
|
}
|
||||||
|
|
||||||
|
if queries := server.DNSBL.Queries(dnsblZone); queries != 0 {
|
||||||
|
t.Errorf("%d queries, want none", queries)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestObserveModeForwardsAClientADNSBLZoneDeniesAndAlertsIt(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
s, server, queue := startWithLookups(t, map[string]string{
|
||||||
|
dnsblZones: dnsblZone, dnsblResolver: noResolver, reputationAction: actionDeny,
|
||||||
|
mode: observe,
|
||||||
|
})
|
||||||
|
loadVerdicts(server, map[string][]string{fromDE: {dnsblZone}})
|
||||||
|
|
||||||
|
line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||||
|
wantWouldAction(t, line, requestlog.ActionDenied)
|
||||||
|
wantReputation(t, line, dnsblZone)
|
||||||
|
|
||||||
|
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
|
||||||
|
if len(waiting) != 1 || waiting[0].Event != alerts.EventReputationHit {
|
||||||
|
t.Errorf("alerts waiting %+v, want a reputation_hit alert", waiting)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReputationLimitTakesPartInTheLowestPercentageOfEveryLimit(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
for _, tc := range []struct {
|
||||||
|
blocklistAction, reputationAction string
|
||||||
|
// want is the request's limit_percent and bytes_percent, as
|
||||||
|
// percentText gives them.
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{limitHalf, limitQuarter, "25 from " + reputationAction},
|
||||||
|
{limitQuarter, limitHalf, "25 from " + blocklistAction},
|
||||||
|
// The blocklist's, the first of two alike.
|
||||||
|
{limitQuarter, limitQuarter, "25 from " + blocklistAction},
|
||||||
|
{actionLog, limitQuarter, "25 from " + reputationAction},
|
||||||
|
{actionLog, actionLog, none},
|
||||||
|
} {
|
||||||
|
t.Run(tc.blocklistAction+" "+tc.reputationAction, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
s, server, _ := startWithLookups(t, map[string]string{
|
||||||
|
blocklistURLs: dropURL, blocklistAction: tc.blocklistAction,
|
||||||
|
dnsblZones: dnsblZone, dnsblResolver: noResolver,
|
||||||
|
reputationAction: tc.reputationAction,
|
||||||
|
})
|
||||||
|
loadLists(t, server, map[string][]string{dropURL: {fromDE}})
|
||||||
|
loadVerdicts(server, map[string][]string{fromDE: {dnsblZone}})
|
||||||
|
|
||||||
|
// Named by the blocklist, then by the zone.
|
||||||
|
line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||||
|
wantReputation(t, line, dropURL, dnsblZone)
|
||||||
|
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||||
|
tc.want)
|
||||||
|
wantPercent(t, "bytes_percent", line.BytesPercent, line.BytesPercentSetting,
|
||||||
|
tc.want)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEachZoneThatListsAClientRaisesAnAlertOncePerCooldownAndIsCounted(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
s, clk, server, queue := startWithLookupsAndClock(t, map[string]string{
|
||||||
|
dnsblZones: dnsblZone + "," + otherZone, dnsblResolver: noResolver,
|
||||||
|
reputationAction: actionLog, metricsToken: token,
|
||||||
|
})
|
||||||
|
loadVerdicts(server, map[string][]string{
|
||||||
|
fromDE: {dnsblZone, otherZone}, unplaced: nil,
|
||||||
|
})
|
||||||
|
|
||||||
|
// The second request's alerts are repeats, which the cooldown holds
|
||||||
|
// back.
|
||||||
|
for range 2 {
|
||||||
|
wantReputation(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward),
|
||||||
|
dnsblZone, otherZone)
|
||||||
|
}
|
||||||
|
|
||||||
|
hit := func(zone string) alerts.Alert {
|
||||||
|
return alerts.Alert{
|
||||||
|
Instance: alertInstance,
|
||||||
|
Time: clk.Now(),
|
||||||
|
Event: alerts.EventReputationHit,
|
||||||
|
Client: netip.MustParseAddr(fromDE),
|
||||||
|
Netblock: netip.MustParsePrefix(fromDE + "/32"),
|
||||||
|
ASN: asnDE,
|
||||||
|
ASName: asNameDE,
|
||||||
|
Country: "DE",
|
||||||
|
Reason: "listed by a DNSBL zone",
|
||||||
|
Detail: map[string]any{"source": zone},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
wantAlerts(t, queue, hit(dnsblZone), hit(otherZone))
|
||||||
|
|
||||||
|
if queue.Suppressed() != 2 {
|
||||||
|
t.Errorf("%d alerts held back, want the second request's 2", queue.Suppressed())
|
||||||
|
}
|
||||||
|
|
||||||
|
// Each zone's hits, and its queries and their failures, none, since
|
||||||
|
// every client had its verdicts.
|
||||||
|
metrics := s.scrape(unplaced)
|
||||||
|
|
||||||
|
for _, zone := range []string{dnsblZone, otherZone} {
|
||||||
|
labels := `{instance="` + alertInstance + `",source="` + zone + `"}`
|
||||||
|
|
||||||
|
wantMetric(t, metrics, "smallwebwaf_reputation_hits_total"+labels, 2)
|
||||||
|
wantMetric(t, metrics, "smallwebwaf_reputation_queries_total"+labels, 0)
|
||||||
|
wantMetric(t, metrics, "smallwebwaf_reputation_failures_total"+labels, 0)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestZoneKeyIsMaskedInTheLogTheAlertAndTheMetrics(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const (
|
||||||
|
key = "abcdefghijklmnopqrstuvwxyz"
|
||||||
|
keyed = key + ".xbl.dq.spamhaus.net"
|
||||||
|
masked = "********.xbl.dq.spamhaus.net"
|
||||||
|
)
|
||||||
|
|
||||||
|
s, server, queue := startWithLookups(t, map[string]string{
|
||||||
|
dnsblZones: keyed, dnsblResolver: noResolver, reputationAction: actionLog,
|
||||||
|
metricsToken: token,
|
||||||
|
})
|
||||||
|
loadVerdicts(server, map[string][]string{fromDE: {keyed}, unplaced: nil})
|
||||||
|
|
||||||
|
wantReputation(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward), masked)
|
||||||
|
|
||||||
|
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
|
||||||
|
if len(waiting) != 1 || waiting[0].Detail["source"] != masked {
|
||||||
|
t.Errorf("alerts waiting %+v, want a reputation_hit alert from %s", waiting,
|
||||||
|
masked)
|
||||||
|
}
|
||||||
|
|
||||||
|
metrics := s.scrape(unplaced)
|
||||||
|
wantMetric(t, metrics, `smallwebwaf_reputation_hits_total{instance="`+
|
||||||
|
alertInstance+`",source="`+masked+`"}`, 1)
|
||||||
|
|
||||||
|
for name, shown := range map[string]string{
|
||||||
|
"the log": s.out.text(), "the metrics": metrics,
|
||||||
|
} {
|
||||||
|
if strings.Contains(shown, key) {
|
||||||
|
t.Errorf("%s shows the key:\n%s", name, shown)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRequestFromAClientWithoutAVerdictHasTheZoneAskedAboutIt(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
s, server, _ := startWithLookups(t, map[string]string{
|
||||||
|
dnsblZones: dnsblZone + "," + otherZone, dnsblResolver: noResolver,
|
||||||
|
})
|
||||||
|
server.DNSBL.Load([]reputation.Verdict{{
|
||||||
|
Zone: otherZone, Client: netip.MustParseAddr(fromDE), Listed: true,
|
||||||
|
Fetched: verdictsFetched(),
|
||||||
|
}})
|
||||||
|
|
||||||
|
// The verdict of the other zone is used, and dnsbl.example, which has
|
||||||
|
// none, is asked about the client in the background, once: the second
|
||||||
|
// request finds the query under way, or the zone left alone after it
|
||||||
|
// failed, since nothing answers at noResolver.
|
||||||
|
for range 2 {
|
||||||
|
wantReputation(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward), otherZone)
|
||||||
|
}
|
||||||
|
|
||||||
|
if queries := server.DNSBL.Queries(dnsblZone); queries != 1 {
|
||||||
|
t.Errorf("%d queries to %s, want 1", queries, dnsblZone)
|
||||||
|
}
|
||||||
|
|
||||||
|
if queries := server.DNSBL.Queries(otherZone); queries != 0 {
|
||||||
|
t.Errorf("%d queries to %s, want none", queries, otherZone)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// listsFetched is when loadLists has the copies fetched.
|
// listsFetched is when loadLists has the copies fetched.
|
||||||
func listsFetched() time.Time {
|
func listsFetched() time.Time {
|
||||||
return time.Date(2026, 10, 5, 0, 0, 0, 0, time.UTC)
|
return time.Date(2026, 10, 5, 0, 0, 0, 0, time.UTC)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// verdictsFetched is when loadVerdicts has the verdicts fetched: half a
|
||||||
|
// day before the time the tests' clock is set to, so that they are in use
|
||||||
|
// until half a day later.
|
||||||
|
func verdictsFetched() time.Time {
|
||||||
|
return time.Date(2026, 10, 5, 12, 0, 0, 0, time.UTC)
|
||||||
|
}
|
||||||
|
|
||||||
|
// loadVerdicts puts into server's DNSBL, for each client listedBy names,
|
||||||
|
// a verdict of each zone SWWAF_DNSBL_ZONES names, fetched at
|
||||||
|
// verdictsFetched, as reputation.json would at start: one that lists the
|
||||||
|
// client from each zone listedBy gives for it, and one that does not from
|
||||||
|
// each other zone.
|
||||||
|
func loadVerdicts(server *proxy.Server, listedBy map[string][]string) {
|
||||||
|
verdicts := make([]reputation.Verdict, 0, len(listedBy)*len(server.DNSBL.Zones()))
|
||||||
|
|
||||||
|
for client, zones := range listedBy {
|
||||||
|
for _, zone := range server.DNSBL.Zones() {
|
||||||
|
verdicts = append(verdicts, reputation.Verdict{
|
||||||
|
Zone: zone, Client: netip.MustParseAddr(client),
|
||||||
|
Listed: slices.Contains(zones, zone), Fetched: verdictsFetched(),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
server.DNSBL.Load(verdicts)
|
||||||
|
}
|
||||||
|
|
||||||
// loadLists puts copies of lists into server's lists, by URL, each with
|
// loadLists puts copies of lists into server's lists, by URL, each with
|
||||||
// its lines, fetched at listsFetched, as reputation.json would at start.
|
// its lines, fetched at listsFetched, as reputation.json would at start.
|
||||||
func loadLists(t *testing.T, server *proxy.Server, copies map[string][]string) {
|
func loadLists(t *testing.T, server *proxy.Server, copies map[string][]string) {
|
||||||
|
|||||||
@@ -63,7 +63,10 @@ type request struct {
|
|||||||
// limits and for the byte limits.
|
// limits and for the byte limits.
|
||||||
counted bool
|
counted bool
|
||||||
limitPercent, bytesPercent percentage
|
limitPercent, bytesPercent percentage
|
||||||
start time.Time
|
// blocklisted is true once a blocklist is found to list the client,
|
||||||
|
// and dnsblListed once a DNSBL zone's verdict is.
|
||||||
|
blocklisted, dnsblListed bool
|
||||||
|
start time.Time
|
||||||
// checked is when the checks were done, and upstreamStart when the
|
// checked is when the checks were done, and upstreamStart when the
|
||||||
// request was handed to the app.
|
// request was handed to the app.
|
||||||
checked time.Time
|
checked time.Time
|
||||||
@@ -213,14 +216,14 @@ func (rq *request) check(ctx context.Context) *refusal {
|
|||||||
// client in SWWAF_ALLOW_NETS skips them, and is not looked up. For any
|
// client in SWWAF_ALLOW_NETS skips them, and is not looked up. For any
|
||||||
// other client, SWWAF_DENY_NETS comes first, then a ban on its netblock,
|
// other client, SWWAF_DENY_NETS comes first, then a ban on its netblock,
|
||||||
// so that a client either refuses is not looked up, then the lookup of
|
// so that a client either refuses is not looked up, then the lookup of
|
||||||
// its AS number and country, then the country lists, and then the
|
// its AS number and country, then the country lists, then the blocklists,
|
||||||
// blocklists; a request any of them refuses is not counted for the rate
|
// and then the DNSBL zones' verdicts; a request any of them refuses is not
|
||||||
// limits. Then come the rate limits, unless the client is in
|
// counted for the rate limits. Then come the rate limits, unless the
|
||||||
// SWWAF_RATE_LIMIT_EXEMPT_NETS or the request's path is exempt under
|
// client is in SWWAF_RATE_LIMIT_EXEMPT_NETS or the request's path is
|
||||||
// SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that every other request is counted,
|
// exempt under SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that every other request
|
||||||
// each of them by the client's limit percentages, and last the rule
|
// is counted, each of them by the client's limit percentages, and last the
|
||||||
// files. A request exempt from the rate limits is exempt from the byte
|
// rule files. A request exempt from the rate limits is exempt from the
|
||||||
// limits too. ctx is the request's own context.
|
// byte limits too. ctx is the request's own context.
|
||||||
func (rq *request) checkClient(ctx context.Context) string {
|
func (rq *request) checkClient(ctx context.Context) string {
|
||||||
cfg := rq.h.config
|
cfg := rq.h.config
|
||||||
if isInside(rq.client, cfg.AllowNets) {
|
if isInside(rq.client, cfg.AllowNets) {
|
||||||
@@ -247,6 +250,10 @@ func (rq *request) checkClient(ctx context.Context) string {
|
|||||||
return requestlog.ActionDenied
|
return requestlog.ActionDenied
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if rq.dnsblDenied(ctx) {
|
||||||
|
return requestlog.ActionDenied
|
||||||
|
}
|
||||||
|
|
||||||
rq.counted = !isInside(rq.client, cfg.RateLimitExemptNets) &&
|
rq.counted = !isInside(rq.client, cfg.RateLimitExemptNets) &&
|
||||||
!pathExempt(rq.in.URL, cfg.RateLimitExemptPaths)
|
!pathExempt(rq.in.URL, cfg.RateLimitExemptPaths)
|
||||||
if rq.counted {
|
if rq.counted {
|
||||||
|
|||||||
@@ -0,0 +1,335 @@
|
|||||||
|
package reputation
|
||||||
|
|
||||||
|
import (
|
||||||
|
"cmp"
|
||||||
|
"context"
|
||||||
|
"encoding/hex"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"log/slog"
|
||||||
|
"net"
|
||||||
|
"net/netip"
|
||||||
|
"slices"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/hashicorp/golang-lru/v2/simplelru"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
// maxVerdicts is how many verdicts are kept. Past it, the one fetched
|
||||||
|
// longest ago is dropped.
|
||||||
|
maxVerdicts = 100000
|
||||||
|
// maxQueries is how many queries may be under way at once. Past it, a
|
||||||
|
// zone is not asked about a client until the client's next request, so
|
||||||
|
// that a swarm of new addresses cannot fill the memory.
|
||||||
|
maxQueries = 1000
|
||||||
|
// failureDelay is how long a zone is not asked again after a query to
|
||||||
|
// it fails, so that a zone refusing queries is not asked on every
|
||||||
|
// request.
|
||||||
|
failureDelay = time.Minute
|
||||||
|
)
|
||||||
|
|
||||||
|
var (
|
||||||
|
errAsk = errors.New("ask the zone")
|
||||||
|
errRefused = errors.New("the zone refused the query")
|
||||||
|
errNotListing = errors.New("the answer is outside 127.0.0.0/8")
|
||||||
|
)
|
||||||
|
|
||||||
|
// Verdict is what a zone said about a client, as reputation.json holds
|
||||||
|
// it: the zone, the client's address, whether the zone lists it, and when
|
||||||
|
// the zone answered.
|
||||||
|
type Verdict struct {
|
||||||
|
Zone string `json:"zone"`
|
||||||
|
Client netip.Addr `json:"client"`
|
||||||
|
Listed bool `json:"listed"`
|
||||||
|
Fetched time.Time `json:"fetched"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// DNSBLParams are what NewDNSBL needs.
|
||||||
|
type DNSBLParams struct {
|
||||||
|
// Zones are the DNSBL zones clients are asked about in
|
||||||
|
// (SWWAF_DNSBL_ZONES).
|
||||||
|
Zones []string
|
||||||
|
// Resolver is the resolver they are asked through
|
||||||
|
// (SWWAF_DNSBL_RESOLVER), or, while it is the zero AddrPort, the
|
||||||
|
// host's, as /etc/resolv.conf names it.
|
||||||
|
Resolver netip.AddrPort
|
||||||
|
// CacheTTL is how long a verdict is used after it was fetched
|
||||||
|
// (SWWAF_REPUTATION_CACHE_TTL), and Timeout how long a query may take
|
||||||
|
// (SWWAF_REPUTATION_TIMEOUT).
|
||||||
|
CacheTTL time.Duration
|
||||||
|
Timeout time.Duration
|
||||||
|
// Now tells the time, normally time.Now in UTC.
|
||||||
|
Now func() time.Time
|
||||||
|
// ProcessLog receives each query that fails, and why.
|
||||||
|
ProcessLog *slog.Logger
|
||||||
|
// Alerts receive a source_failure alert for each query that fails.
|
||||||
|
Alerts *alerts.Queue
|
||||||
|
}
|
||||||
|
|
||||||
|
// DNSBL asks the DNSBL zones about clients, in the background, and keeps
|
||||||
|
// their verdicts. It is safe for concurrent use.
|
||||||
|
type DNSBL struct {
|
||||||
|
params DNSBLParams
|
||||||
|
resolver *net.Resolver
|
||||||
|
|
||||||
|
mu sync.Mutex
|
||||||
|
// verdicts are by query. Each is added as it is fetched and never moved
|
||||||
|
// up, so that the one fetched longest ago is the first dropped.
|
||||||
|
verdicts *simplelru.LRU[query, Verdict]
|
||||||
|
// asking are the queries under way.
|
||||||
|
asking map[query]bool
|
||||||
|
// queries and failures count, by zone, the queries made and those that
|
||||||
|
// failed, and retryAt is when a zone whose last query failed may be
|
||||||
|
// asked again.
|
||||||
|
queries map[string]int
|
||||||
|
failures map[string]int
|
||||||
|
retryAt map[string]time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
// query is a client's address, to ask a zone about.
|
||||||
|
type query struct {
|
||||||
|
zone string
|
||||||
|
client netip.Addr
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewDNSBL returns a DNSBL with no verdict yet.
|
||||||
|
func NewDNSBL(params DNSBLParams) *DNSBL {
|
||||||
|
verdicts, err := simplelru.NewLRU[query, Verdict](maxVerdicts, nil)
|
||||||
|
if err != nil {
|
||||||
|
panic(err) // NewLRU fails only for a size below one
|
||||||
|
}
|
||||||
|
|
||||||
|
resolver := &net.Resolver{}
|
||||||
|
if params.Resolver.IsValid() {
|
||||||
|
// Dial is used by Go's own resolver alone.
|
||||||
|
resolver.PreferGo = true
|
||||||
|
resolver.Dial = func(ctx context.Context, network, _ string) (net.Conn, error) {
|
||||||
|
var dialer net.Dialer
|
||||||
|
|
||||||
|
return dialer.DialContext(ctx, network, params.Resolver.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return &DNSBL{
|
||||||
|
params: params,
|
||||||
|
resolver: resolver,
|
||||||
|
verdicts: verdicts,
|
||||||
|
asking: map[query]bool{},
|
||||||
|
queries: map[string]int{},
|
||||||
|
failures: map[string]int{},
|
||||||
|
retryAt: map[string]time.Time{},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Zones returns the zones, in the order SWWAF_DNSBL_ZONES names them.
|
||||||
|
func (d *DNSBL) Zones() []string {
|
||||||
|
return slices.Clone(d.params.Zones)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ListedBy returns the zones whose verdict on addr, a client's address,
|
||||||
|
// lists it, in the order SWWAF_DNSBL_ZONES names them, each with its key
|
||||||
|
// masked, as config.MaskZoneKey masks it, since they go to the request
|
||||||
|
// log, the alerts and the metrics. A verdict is used until CacheTTL has
|
||||||
|
// passed since it was fetched. Each zone without one is asked about addr
|
||||||
|
// in the background, unless a query about addr to it is under way, the
|
||||||
|
// zone is left alone after a failure, or maxQueries are under way;
|
||||||
|
// ListedBy never waits for a query. ctx is the context of the client's
|
||||||
|
// request, and a query goes on after the request ends.
|
||||||
|
func (d *DNSBL) ListedBy(ctx context.Context, addr netip.Addr) []string {
|
||||||
|
d.mu.Lock()
|
||||||
|
defer d.mu.Unlock()
|
||||||
|
|
||||||
|
now := d.params.Now()
|
||||||
|
|
||||||
|
var listedBy []string
|
||||||
|
|
||||||
|
for _, zone := range d.params.Zones {
|
||||||
|
q := query{zone: zone, client: addr}
|
||||||
|
|
||||||
|
kept, found := d.verdicts.Peek(q)
|
||||||
|
|
||||||
|
switch {
|
||||||
|
case found && now.Sub(kept.Fetched) < d.params.CacheTTL:
|
||||||
|
if kept.Listed {
|
||||||
|
listedBy = append(listedBy, config.MaskZoneKey(zone))
|
||||||
|
}
|
||||||
|
case !d.asking[q] && !now.Before(d.retryAt[zone]) && len(d.asking) < maxQueries:
|
||||||
|
d.asking[q] = true
|
||||||
|
d.queries[zone]++
|
||||||
|
|
||||||
|
go d.ask(context.WithoutCancel(ctx), q)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return listedBy
|
||||||
|
}
|
||||||
|
|
||||||
|
// Queries returns how many queries were made to zone.
|
||||||
|
func (d *DNSBL) Queries(zone string) int {
|
||||||
|
d.mu.Lock()
|
||||||
|
defer d.mu.Unlock()
|
||||||
|
|
||||||
|
return d.queries[zone]
|
||||||
|
}
|
||||||
|
|
||||||
|
// Failures returns how many queries to zone failed.
|
||||||
|
func (d *DNSBL) Failures(zone string) int {
|
||||||
|
d.mu.Lock()
|
||||||
|
defer d.mu.Unlock()
|
||||||
|
|
||||||
|
return d.failures[zone]
|
||||||
|
}
|
||||||
|
|
||||||
|
// Snapshot returns every verdict still in use, sorted by client, then by
|
||||||
|
// zone, as reputation.json lists them.
|
||||||
|
func (d *DNSBL) Snapshot() []Verdict {
|
||||||
|
d.mu.Lock()
|
||||||
|
|
||||||
|
now := d.params.Now()
|
||||||
|
verdicts := make([]Verdict, 0, d.verdicts.Len())
|
||||||
|
|
||||||
|
for _, kept := range d.verdicts.Values() {
|
||||||
|
if now.Sub(kept.Fetched) < d.params.CacheTTL {
|
||||||
|
verdicts = append(verdicts, kept)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
d.mu.Unlock()
|
||||||
|
|
||||||
|
slices.SortFunc(verdicts, func(a, b Verdict) int {
|
||||||
|
return cmp.Or(a.Client.Compare(b.Client), strings.Compare(a.Zone, b.Zone))
|
||||||
|
})
|
||||||
|
|
||||||
|
return verdicts
|
||||||
|
}
|
||||||
|
|
||||||
|
// Load keeps verdicts, read from reputation.json, in place of those it
|
||||||
|
// keeps, but for those of a zone SWWAF_DNSBL_ZONES does not name, and,
|
||||||
|
// past maxVerdicts, those fetched longest ago. One fetched CacheTTL ago or
|
||||||
|
// more is neither used nor written, as for any verdict.
|
||||||
|
func (d *DNSBL) Load(verdicts []Verdict) {
|
||||||
|
verdicts = slices.Clone(verdicts)
|
||||||
|
slices.SortStableFunc(verdicts, func(a, b Verdict) int {
|
||||||
|
return a.Fetched.Compare(b.Fetched)
|
||||||
|
})
|
||||||
|
|
||||||
|
d.mu.Lock()
|
||||||
|
defer d.mu.Unlock()
|
||||||
|
|
||||||
|
d.verdicts.Purge()
|
||||||
|
|
||||||
|
for _, kept := range verdicts {
|
||||||
|
if slices.Contains(d.params.Zones, kept.Zone) {
|
||||||
|
d.verdicts.Add(query{zone: kept.Zone, client: kept.Client}, kept)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ask asks q's zone about q's client, keeps the verdict, and notes the
|
||||||
|
// query as no longer under way. A query that fails gives no verdict: it
|
||||||
|
// is counted, logged and raised as a source_failure alert, which show the
|
||||||
|
// zone with its key masked, and the zone is not asked again for
|
||||||
|
// failureDelay.
|
||||||
|
func (d *DNSBL) ask(ctx context.Context, q query) {
|
||||||
|
listed, err := d.lookUp(ctx, q)
|
||||||
|
now := d.params.Now()
|
||||||
|
|
||||||
|
d.mu.Lock()
|
||||||
|
|
||||||
|
delete(d.asking, q)
|
||||||
|
|
||||||
|
if err == nil {
|
||||||
|
d.verdicts.Add(q, Verdict{
|
||||||
|
Zone: q.zone, Client: q.client, Listed: listed, Fetched: now,
|
||||||
|
})
|
||||||
|
} else {
|
||||||
|
d.failures[q.zone]++
|
||||||
|
d.retryAt[q.zone] = now.Add(failureDelay)
|
||||||
|
}
|
||||||
|
|
||||||
|
d.mu.Unlock()
|
||||||
|
|
||||||
|
if err != nil {
|
||||||
|
const failed = "asking a DNSBL zone failed"
|
||||||
|
|
||||||
|
shown := config.MaskZoneKey(q.zone)
|
||||||
|
|
||||||
|
// Raised before it is logged, so that the alert is there once the
|
||||||
|
// log line is.
|
||||||
|
d.params.Alerts.Raise(alerts.Alert{
|
||||||
|
Event: alerts.EventSourceFailure,
|
||||||
|
Reason: failed,
|
||||||
|
Detail: map[string]any{"source": shown, "error": err.Error()},
|
||||||
|
})
|
||||||
|
d.params.ProcessLog.Warn(failed, "zone", shown, "error", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// lookUp asks q's zone about q's client through the resolver, and returns
|
||||||
|
// whether the zone lists it, as readAnswer reads the answer. No such name
|
||||||
|
// is a client the zone does not list. A query not answered within Timeout
|
||||||
|
// fails.
|
||||||
|
func (d *DNSBL) lookUp(ctx context.Context, q query) (bool, error) {
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, d.params.Timeout)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
answer, err := d.resolver.LookupNetIP(ctx, "ip4", queryName(q.zone, q.client))
|
||||||
|
|
||||||
|
var dnsErr *net.DNSError
|
||||||
|
|
||||||
|
switch {
|
||||||
|
case err == nil:
|
||||||
|
return readAnswer(answer)
|
||||||
|
case errors.As(err, &dnsErr) && dnsErr.IsNotFound:
|
||||||
|
return false, nil
|
||||||
|
case errors.As(err, &dnsErr):
|
||||||
|
// The error names the name asked about, which holds the client's
|
||||||
|
// address, which is not to be logged: only what went wrong is kept.
|
||||||
|
return false, fmt.Errorf("%w: %s", errAsk, dnsErr.Err)
|
||||||
|
default:
|
||||||
|
return false, fmt.Errorf("%w: %w", errAsk, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// queryName returns the name a zone is asked about addr by, as RFC 5782
|
||||||
|
// builds it: the four numbers of an IPv4 address, or the 32 hex digits of
|
||||||
|
// an IPv6 address, in reverse order, each followed by a dot, then the zone
|
||||||
|
// and a dot, which makes it a full name, to which the resolver adds no
|
||||||
|
// search domain of /etc/resolv.conf.
|
||||||
|
func queryName(zone string, addr netip.Addr) string {
|
||||||
|
parts := strings.Split(addr.String(), ".")
|
||||||
|
if addr.Is6() {
|
||||||
|
parts = strings.Split(hex.EncodeToString(addr.AsSlice()), "")
|
||||||
|
}
|
||||||
|
|
||||||
|
slices.Reverse(parts)
|
||||||
|
|
||||||
|
return strings.Join(parts, ".") + "." + zone + "."
|
||||||
|
}
|
||||||
|
|
||||||
|
// readAnswer reads the addresses a zone answered with. An address in
|
||||||
|
// 127.0.0.0/8 lists the client, as RFC 5782 has zones answer, but one in
|
||||||
|
// 127.255.255.0/24 is how Spamhaus refuses a query, such as one sent
|
||||||
|
// through a public resolver or one past its limit, and is a failure. So is
|
||||||
|
// an address outside 127.0.0.0/8, such as a resolver gives that answers
|
||||||
|
// even for names that do not exist.
|
||||||
|
func readAnswer(answer []netip.Addr) (bool, error) {
|
||||||
|
listing := netip.MustParsePrefix("127.0.0.0/8")
|
||||||
|
refusal := netip.MustParsePrefix("127.255.255.0/24")
|
||||||
|
|
||||||
|
for _, addr := range answer {
|
||||||
|
switch {
|
||||||
|
case refusal.Contains(addr):
|
||||||
|
return false, fmt.Errorf("%w: %s", errRefused, addr)
|
||||||
|
case !listing.Contains(addr):
|
||||||
|
return false, fmt.Errorf("%w: %s", errNotListing, addr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return len(answer) > 0, nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,737 @@
|
|||||||
|
package reputation_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"encoding/binary"
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
"log/slog"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"net/netip"
|
||||||
|
"reflect"
|
||||||
|
"slices"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"testing/synctest"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The tests of the DNSBL zones run in synctest bubbles, as those of the
|
||||||
|
// lists do, and the resolver the zones are asked through is a stand-in
|
||||||
|
// reached through an in-memory connection, net.Pipe's, for the same
|
||||||
|
// reason. They run one at a time, none in parallel with another test of
|
||||||
|
// this package: Go's resolver counts the queries under way in one
|
||||||
|
// sync.WaitGroup for the whole process, and the process fails when
|
||||||
|
// queries from two bubbles, or from a bubble and from outside one, are
|
||||||
|
// under way at once. TestMain has the resolver make its configuration,
|
||||||
|
// which it makes on its first query, outside every bubble, since the
|
||||||
|
// configuration holds a channel, which the bubble it was made in would
|
||||||
|
// keep to itself.
|
||||||
|
|
||||||
|
const (
|
||||||
|
// zone and otherZone are the DNSBL zones the tests name.
|
||||||
|
zone = "dnsbl.example"
|
||||||
|
otherZone = "other.example"
|
||||||
|
// cacheTTL is the tests' SWWAF_REPUTATION_CACHE_TTL, and timeout their
|
||||||
|
// SWWAF_REPUTATION_TIMEOUT: a second, the least time /etc/resolv.conf
|
||||||
|
// can have Go's resolver wait for one server, so that it is the
|
||||||
|
// DNSBL's own timeout that ends a query, whatever that file says.
|
||||||
|
cacheTTL = 24 * time.Hour
|
||||||
|
timeout = time.Second
|
||||||
|
// listed and unlisted are clients zone is asked about by the names
|
||||||
|
// listedName and unlistedName, and most tests have zone list the first
|
||||||
|
// alone, by answering with listing.
|
||||||
|
listed = "192.0.2.99"
|
||||||
|
unlisted = "192.0.2.100"
|
||||||
|
listedName = "99.2.0.192." + zone + "."
|
||||||
|
unlistedName = "100.2.0.192." + zone + "."
|
||||||
|
listing = "127.0.0.2"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The DNS response codes the stand-in answers with, besides no error.
|
||||||
|
const (
|
||||||
|
serverFailure = 2
|
||||||
|
noSuchName = 3
|
||||||
|
refused = 5
|
||||||
|
)
|
||||||
|
|
||||||
|
var errNoNetwork = errors.New("the test dials nothing")
|
||||||
|
|
||||||
|
func TestMain(m *testing.M) {
|
||||||
|
// A query that fails at once, as nothing is dialled for it.
|
||||||
|
resolver := &net.Resolver{
|
||||||
|
PreferGo: true,
|
||||||
|
Dial: func(context.Context, string, string) (net.Conn, error) {
|
||||||
|
return nil, errNoNetwork
|
||||||
|
},
|
||||||
|
}
|
||||||
|
_, _ = resolver.LookupNetIP(context.Background(), "ip4", "warm-up.invalid.")
|
||||||
|
|
||||||
|
m.Run()
|
||||||
|
}
|
||||||
|
|
||||||
|
//nolint:paralleltest // one at a time, as the comment at the top of this file says
|
||||||
|
func TestZonesListOrNotClientsByTheirIPv4AndIPv6Addresses(t *testing.T) {
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
// The addresses of the examples of RFC 5782, and the names it
|
||||||
|
// gives for them.
|
||||||
|
const (
|
||||||
|
v4 = "192.0.2.99"
|
||||||
|
v6 = "2001:db8:1:2:3:4:567:89ab"
|
||||||
|
// v6Name is the hex digits of v6, in reverse order.
|
||||||
|
v6Name = "b.a.9.8.7.6.5.0.4.0.0.0.3.0.0.0.2.0.0.0.1.0.0.0.8.b.d.0.1.0.0.2."
|
||||||
|
)
|
||||||
|
|
||||||
|
resolver := &resolverStandIn{answers: map[string]answer{
|
||||||
|
"99.2.0.192." + zone + ".": {addrs: []string{listing}},
|
||||||
|
v6Name + otherZone + ".": {addrs: []string{"127.0.0.4", "127.0.0.10"}},
|
||||||
|
"99.2.0.192." + otherZone + ".": {},
|
||||||
|
}}
|
||||||
|
dnsbl := newDNSBL(resolver, dnsblParams(zone, otherZone))
|
||||||
|
|
||||||
|
// Neither client has a verdict yet, so neither is listed, and each
|
||||||
|
// zone is asked about each.
|
||||||
|
wantZones(t, dnsbl, v4)
|
||||||
|
wantZones(t, dnsbl, v6)
|
||||||
|
synctest.Wait()
|
||||||
|
|
||||||
|
wantZones(t, dnsbl, v4, zone)
|
||||||
|
wantZones(t, dnsbl, v6, otherZone)
|
||||||
|
wantAsked(t, resolver,
|
||||||
|
"99.2.0.192."+zone+".", "99.2.0.192."+otherZone+".",
|
||||||
|
v6Name+zone+".", v6Name+otherZone+".")
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
//nolint:paralleltest // one at a time, as the comment at the top of this file says
|
||||||
|
func TestListedByNeverWaitsForAQuery(t *testing.T) {
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
dnsbl := newDNSBL(&resolverStandIn{hanging: true}, dnsblParams(zone))
|
||||||
|
began := time.Now()
|
||||||
|
|
||||||
|
// The second, while the first's query is under way, starts none.
|
||||||
|
wantZones(t, dnsbl, listed)
|
||||||
|
wantZones(t, dnsbl, listed)
|
||||||
|
|
||||||
|
if waited := time.Since(began); waited != 0 {
|
||||||
|
t.Errorf("waited %s for the query, want no wait", waited)
|
||||||
|
}
|
||||||
|
|
||||||
|
synctest.Wait()
|
||||||
|
wantQueries(t, dnsbl, 1, 0)
|
||||||
|
waitForTheResolver()
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
//nolint:paralleltest // one at a time, as the comment at the top of this file says
|
||||||
|
func TestVerdictUsedUntilTheCacheTTLHasPassedSinceItWasFetched(t *testing.T) {
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
resolver := &resolverStandIn{answers: map[string]answer{
|
||||||
|
listedName: {addrs: []string{listing}},
|
||||||
|
}}
|
||||||
|
dnsbl := newDNSBL(resolver, dnsblParams(zone))
|
||||||
|
|
||||||
|
wantZones(t, dnsbl, listed)
|
||||||
|
wantZones(t, dnsbl, unlisted)
|
||||||
|
synctest.Wait()
|
||||||
|
|
||||||
|
// The zone lists the other client from now on, but the verdicts
|
||||||
|
// kept are used, and the zone is not asked again, until the TTL
|
||||||
|
// has passed.
|
||||||
|
resolver.set(listedName, answer{rcode: noSuchName})
|
||||||
|
resolver.set(unlistedName, answer{addrs: []string{listing}})
|
||||||
|
time.Sleep(cacheTTL - time.Nanosecond)
|
||||||
|
|
||||||
|
wantZones(t, dnsbl, listed, zone)
|
||||||
|
wantZones(t, dnsbl, unlisted)
|
||||||
|
synctest.Wait()
|
||||||
|
wantQueries(t, dnsbl, 2, 0)
|
||||||
|
|
||||||
|
// Then neither verdict is used, and both clients are asked about
|
||||||
|
// again.
|
||||||
|
time.Sleep(time.Nanosecond)
|
||||||
|
wantZones(t, dnsbl, listed)
|
||||||
|
wantZones(t, dnsbl, unlisted)
|
||||||
|
synctest.Wait()
|
||||||
|
|
||||||
|
wantQueries(t, dnsbl, 4, 0)
|
||||||
|
wantZones(t, dnsbl, listed)
|
||||||
|
wantZones(t, dnsbl, unlisted, zone)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
//nolint:paralleltest // one at a time, as the comment at the top of this file says
|
||||||
|
func TestQueryNotAnsweredWithinTheTimeoutFails(t *testing.T) {
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
queue := newQueue()
|
||||||
|
p := dnsblParams(zone)
|
||||||
|
p.Alerts = queue
|
||||||
|
dnsbl := newDNSBL(&resolverStandIn{hanging: true}, p)
|
||||||
|
|
||||||
|
wantZones(t, dnsbl, listed)
|
||||||
|
time.Sleep(timeout - time.Nanosecond)
|
||||||
|
synctest.Wait()
|
||||||
|
wantQueries(t, dnsbl, 1, 0)
|
||||||
|
|
||||||
|
time.Sleep(time.Nanosecond)
|
||||||
|
synctest.Wait()
|
||||||
|
wantQueries(t, dnsbl, 1, 1)
|
||||||
|
|
||||||
|
if got := waiting(queue); len(got) != 1 ||
|
||||||
|
got[0].Detail["error"] != "ask the zone: i/o timeout" {
|
||||||
|
t.Errorf("alerts waiting %+v, want the timeout's", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
if verdicts := dnsbl.Snapshot(); len(verdicts) != 0 {
|
||||||
|
t.Errorf("verdicts %+v, want none", verdicts)
|
||||||
|
}
|
||||||
|
|
||||||
|
waitForTheResolver()
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
//nolint:paralleltest // one at a time, as the comment at the top of this file says
|
||||||
|
func TestZoneThatFailsOrRefusesGivesNoVerdictAndIsLeftAloneForAMinute(t *testing.T) {
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
answer answer
|
||||||
|
error string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
"a server failure", answer{rcode: serverFailure},
|
||||||
|
"ask the zone: server misbehaving",
|
||||||
|
},
|
||||||
|
{"a refusal", answer{rcode: refused}, "ask the zone: server misbehaving"},
|
||||||
|
{
|
||||||
|
"an answer in 127.255.255.0/24, with which Spamhaus refuses a query",
|
||||||
|
answer{addrs: []string{"127.255.255.254"}},
|
||||||
|
"the zone refused the query: 127.255.255.254",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"an answer outside 127.0.0.0/8, as for a name that does not exist",
|
||||||
|
answer{addrs: []string{"192.0.2.1"}},
|
||||||
|
"the answer is outside 127.0.0.0/8: 192.0.2.1",
|
||||||
|
},
|
||||||
|
} {
|
||||||
|
//nolint:paralleltest // one at a time, as the comment at the top of this file says
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
var log bytes.Buffer
|
||||||
|
|
||||||
|
queue := newQueue()
|
||||||
|
p := dnsblParams(zone)
|
||||||
|
p.Alerts = queue
|
||||||
|
p.ProcessLog = slog.New(slog.NewJSONHandler(&log, nil))
|
||||||
|
dnsbl := newDNSBL(&resolverStandIn{answers: map[string]answer{
|
||||||
|
listedName: tc.answer,
|
||||||
|
}}, p)
|
||||||
|
|
||||||
|
// The failure gives no verdict, and the zone is not asked
|
||||||
|
// again within a minute of it.
|
||||||
|
wantZones(t, dnsbl, listed)
|
||||||
|
synctest.Wait()
|
||||||
|
time.Sleep(time.Minute - time.Nanosecond)
|
||||||
|
wantZones(t, dnsbl, listed)
|
||||||
|
synctest.Wait()
|
||||||
|
wantQueries(t, dnsbl, 1, 1)
|
||||||
|
|
||||||
|
time.Sleep(time.Nanosecond)
|
||||||
|
wantZones(t, dnsbl, listed)
|
||||||
|
synctest.Wait()
|
||||||
|
wantQueries(t, dnsbl, 2, 2)
|
||||||
|
|
||||||
|
if verdicts := dnsbl.Snapshot(); len(verdicts) != 0 {
|
||||||
|
t.Errorf("verdicts %+v, want none", verdicts)
|
||||||
|
}
|
||||||
|
|
||||||
|
// One alert for the first failure; the cooldown holds back
|
||||||
|
// the second.
|
||||||
|
wantAlert(t, queue, alerts.Alert{
|
||||||
|
Time: time.Now().Add(-time.Minute),
|
||||||
|
Event: alerts.EventSourceFailure,
|
||||||
|
Reason: "asking a DNSBL zone failed",
|
||||||
|
Detail: map[string]any{"source": zone, "error": tc.error},
|
||||||
|
})
|
||||||
|
|
||||||
|
if !strings.Contains(log.String(), `"msg":"asking a DNSBL zone failed",`+
|
||||||
|
`"zone":"`+zone+`","error":"`+tc.error) {
|
||||||
|
t.Errorf("logged\n%s\nwant the failures", log.String())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
//nolint:paralleltest // one at a time, as the comment at the top of this file says
|
||||||
|
func TestAtMost1000QueriesUnderWay(t *testing.T) {
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
dnsbl := newDNSBL(&resolverStandIn{hanging: true}, dnsblParams(zone))
|
||||||
|
|
||||||
|
client := netip.MustParseAddr("198.18.0.0")
|
||||||
|
for range 1001 {
|
||||||
|
dnsbl.ListedBy(t.Context(), client)
|
||||||
|
client = client.Next()
|
||||||
|
}
|
||||||
|
|
||||||
|
synctest.Wait()
|
||||||
|
wantQueries(t, dnsbl, 1000, 0)
|
||||||
|
waitForTheResolver()
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
//nolint:paralleltest // one at a time, as the comment at the top of this file says
|
||||||
|
func TestMetricsCountEachZonesQueriesAndThoseThatFailed(t *testing.T) {
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
resolver := &resolverStandIn{answers: map[string]answer{
|
||||||
|
listedName: {addrs: []string{listing}},
|
||||||
|
"99.2.0.192." + otherZone + ".": {rcode: serverFailure},
|
||||||
|
}}
|
||||||
|
dnsbl := newDNSBL(resolver, dnsblParams(zone, otherZone))
|
||||||
|
m := metrics.New(1, "app")
|
||||||
|
m.AddReputation(reputation.New(params()), dnsbl)
|
||||||
|
|
||||||
|
wantZones(t, dnsbl, listed)
|
||||||
|
synctest.Wait()
|
||||||
|
|
||||||
|
scraped := httptest.NewRecorder()
|
||||||
|
m.ServeHTTP(scraped, httptest.NewRequestWithContext(t.Context(), http.MethodGet,
|
||||||
|
"/", http.NoBody))
|
||||||
|
|
||||||
|
for series, want := range map[string]string{
|
||||||
|
"queries_total" + `{instance="app",source="` + zone + `"}`: "1",
|
||||||
|
"failures_total" + `{instance="app",source="` + zone + `"}`: "0",
|
||||||
|
"queries_total" + `{instance="app",source="` + otherZone + `"}`: "1",
|
||||||
|
"failures_total" + `{instance="app",source="` + otherZone + `"}`: "1",
|
||||||
|
} {
|
||||||
|
line := "\nsmallwebwaf_reputation_" + series + " " + want + "\n"
|
||||||
|
if !strings.Contains(scraped.Body.String(), line) {
|
||||||
|
t.Errorf("metrics\n%s\nwant%s", scraped.Body.String(), line)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
//nolint:paralleltest // one at a time, as the comment at the top of this file says
|
||||||
|
func TestZoneKeyIsMaskedInTheVerdictsTheFailuresAndTheMetrics(t *testing.T) {
|
||||||
|
const (
|
||||||
|
key = "abcdefghijklmnopqrstuvwxyz"
|
||||||
|
keyed = key + ".xbl.dq.spamhaus.net"
|
||||||
|
masked = "********.xbl.dq.spamhaus.net"
|
||||||
|
)
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
var log bytes.Buffer
|
||||||
|
|
||||||
|
queue := newQueue()
|
||||||
|
p := dnsblParams(keyed)
|
||||||
|
p.Alerts = queue
|
||||||
|
p.ProcessLog = slog.New(slog.NewJSONHandler(&log, nil))
|
||||||
|
dnsbl := newDNSBL(&resolverStandIn{answers: map[string]answer{
|
||||||
|
"99.2.0.192." + keyed + ".": {addrs: []string{listing}},
|
||||||
|
"100.2.0.192." + keyed + ".": {rcode: serverFailure},
|
||||||
|
}}, p)
|
||||||
|
m := metrics.New(1, "app")
|
||||||
|
m.AddReputation(reputation.New(params()), dnsbl)
|
||||||
|
|
||||||
|
// Both clients are asked about before either answer comes, so that
|
||||||
|
// the failure does not keep the zone from the other query.
|
||||||
|
wantZones(t, dnsbl, listed)
|
||||||
|
wantZones(t, dnsbl, unlisted)
|
||||||
|
synctest.Wait()
|
||||||
|
wantZones(t, dnsbl, listed, masked)
|
||||||
|
|
||||||
|
if got := waiting(queue); len(got) != 1 || got[0].Detail["source"] != masked {
|
||||||
|
t.Errorf("alerts waiting %+v, want the failure's, from %s", got, masked)
|
||||||
|
}
|
||||||
|
|
||||||
|
scraped := httptest.NewRecorder()
|
||||||
|
m.ServeHTTP(scraped, httptest.NewRequestWithContext(t.Context(), http.MethodGet,
|
||||||
|
"/", http.NoBody))
|
||||||
|
|
||||||
|
for name, shown := range map[string]string{
|
||||||
|
"the log": log.String(), "the metrics": scraped.Body.String(),
|
||||||
|
} {
|
||||||
|
if strings.Contains(shown, key) || !strings.Contains(shown, masked) {
|
||||||
|
t.Errorf("%s shows the key, or does not name the zone:\n%s", name, shown)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
//nolint:paralleltest // one at a time, as the comment at the top of this file says
|
||||||
|
func TestVerdictsKeptAcrossARestart(t *testing.T) {
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
resolver := &resolverStandIn{answers: map[string]answer{
|
||||||
|
listedName: {addrs: []string{listing}},
|
||||||
|
}}
|
||||||
|
dnsbl := newDNSBL(resolver, dnsblParams(zone))
|
||||||
|
fetched := time.Now()
|
||||||
|
|
||||||
|
wantZones(t, dnsbl, unlisted)
|
||||||
|
wantZones(t, dnsbl, listed)
|
||||||
|
synctest.Wait()
|
||||||
|
|
||||||
|
kept := dnsbl.Snapshot()
|
||||||
|
|
||||||
|
want := []reputation.Verdict{
|
||||||
|
{Zone: zone, Client: netip.MustParseAddr(listed), Listed: true, Fetched: fetched},
|
||||||
|
{Zone: zone, Client: netip.MustParseAddr(unlisted), Fetched: fetched},
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(kept, want) {
|
||||||
|
t.Errorf("verdicts %+v, want %+v", kept, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Restarted an hour later with what reputation.json keeps, it uses
|
||||||
|
// the verdicts, and asks the zone nothing, until the TTL has passed
|
||||||
|
// since they were fetched.
|
||||||
|
time.Sleep(time.Hour)
|
||||||
|
|
||||||
|
restarted := &resolverStandIn{}
|
||||||
|
again := newDNSBL(restarted, dnsblParams(zone))
|
||||||
|
again.Load(kept)
|
||||||
|
|
||||||
|
wantZones(t, again, listed, zone)
|
||||||
|
wantZones(t, again, unlisted)
|
||||||
|
synctest.Wait()
|
||||||
|
wantAsked(t, restarted)
|
||||||
|
|
||||||
|
time.Sleep(cacheTTL - time.Hour)
|
||||||
|
wantZones(t, again, listed)
|
||||||
|
synctest.Wait()
|
||||||
|
wantAsked(t, restarted, listedName)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNeitherAVerdictOfAZoneNotNamedNorOnePastItsTTLIsKept(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
now := time.Date(2026, 10, 7, 0, 0, 0, 0, time.UTC)
|
||||||
|
p := dnsblParams(zone)
|
||||||
|
p.Now = func() time.Time { return now }
|
||||||
|
dnsbl := reputation.NewDNSBL(p)
|
||||||
|
|
||||||
|
// The last verdict still in use, one fetched a TTL ago, and one of a
|
||||||
|
// zone SWWAF_DNSBL_ZONES does not name.
|
||||||
|
inUse := reputation.Verdict{
|
||||||
|
Zone: zone, Client: netip.MustParseAddr(listed), Listed: true,
|
||||||
|
Fetched: now.Add(-cacheTTL + time.Nanosecond),
|
||||||
|
}
|
||||||
|
stale := reputation.Verdict{
|
||||||
|
Zone: zone, Client: netip.MustParseAddr(unlisted), Fetched: now.Add(-cacheTTL),
|
||||||
|
}
|
||||||
|
notNamed := reputation.Verdict{
|
||||||
|
Zone: otherZone, Client: netip.MustParseAddr(listed), Listed: true, Fetched: now,
|
||||||
|
}
|
||||||
|
|
||||||
|
dnsbl.Load([]reputation.Verdict{notNamed, stale, inUse})
|
||||||
|
|
||||||
|
if got := dnsbl.Snapshot(); !reflect.DeepEqual(got, []reputation.Verdict{inUse}) {
|
||||||
|
t.Errorf("verdicts %+v, want only %+v", got, inUse)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAtMost100000VerdictsKeptTheOneFetchedLongestAgoDroppedFirst(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
now := time.Date(2026, 10, 7, 0, 0, 0, 0, time.UTC)
|
||||||
|
p := dnsblParams(zone)
|
||||||
|
p.Now = func() time.Time { return now }
|
||||||
|
dnsbl := reputation.NewDNSBL(p)
|
||||||
|
|
||||||
|
// 100,001 verdicts, listed by client, as reputation.json lists them,
|
||||||
|
// each fetched a millisecond before the one before it: the last is one
|
||||||
|
// too many.
|
||||||
|
const count = 100001
|
||||||
|
|
||||||
|
verdicts := make([]reputation.Verdict, 0, count)
|
||||||
|
|
||||||
|
client := netip.MustParseAddr("198.18.0.0")
|
||||||
|
for i := range count {
|
||||||
|
verdicts = append(verdicts, reputation.Verdict{
|
||||||
|
Zone: zone, Client: client, Fetched: now.Add(-time.Duration(i) * time.Millisecond),
|
||||||
|
})
|
||||||
|
client = client.Next()
|
||||||
|
}
|
||||||
|
|
||||||
|
dnsbl.Load(verdicts)
|
||||||
|
|
||||||
|
got := dnsbl.Snapshot()
|
||||||
|
if len(got) != count-1 || !slices.Contains(got, verdicts[0]) ||
|
||||||
|
slices.Contains(got, verdicts[count-1]) {
|
||||||
|
t.Errorf("%d verdicts kept, want all but the one fetched longest ago", len(got))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
//nolint:paralleltest // one at a time, as the comment at the top of this file says
|
||||||
|
func TestQueriesGoToTheResolverSWWAFDNSBLResolverNames(t *testing.T) {
|
||||||
|
resolver := &resolverStandIn{answers: map[string]answer{
|
||||||
|
listedName: {addrs: []string{listing}},
|
||||||
|
}}
|
||||||
|
|
||||||
|
conn, err := (&net.ListenConfig{}).ListenPacket(t.Context(), "udp", "127.0.0.1:0")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("listen: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
served := make(chan struct{})
|
||||||
|
|
||||||
|
go func() {
|
||||||
|
resolver.serveUDP(conn)
|
||||||
|
close(served)
|
||||||
|
}()
|
||||||
|
|
||||||
|
t.Cleanup(func() {
|
||||||
|
_ = conn.Close()
|
||||||
|
|
||||||
|
<-served
|
||||||
|
})
|
||||||
|
|
||||||
|
p := dnsblParams(zone)
|
||||||
|
p.Resolver = netip.MustParseAddrPort(conn.LocalAddr().String())
|
||||||
|
// On the real clock: the stand-in answers at once, so only a test
|
||||||
|
// process held up for a whole minute would see the query fail.
|
||||||
|
p.Timeout = time.Minute
|
||||||
|
|
||||||
|
isListed, err := reputation.NewDNSBL(p).LookUp(zone, netip.MustParseAddr(listed))
|
||||||
|
if err != nil || !isListed {
|
||||||
|
t.Errorf("listed %t (%v), want true", isListed, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
wantAsked(t, resolver, listedName)
|
||||||
|
}
|
||||||
|
|
||||||
|
// resolverStandIn is a stand-in for the resolver the zones are asked
|
||||||
|
// through. It answers each query by the name asked about, as answers
|
||||||
|
// gives, with no such name for a name answers does not give, and not at
|
||||||
|
// all while hanging. It notes each name asked about.
|
||||||
|
type resolverStandIn struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
answers map[string]answer
|
||||||
|
hanging bool
|
||||||
|
names []string
|
||||||
|
}
|
||||||
|
|
||||||
|
// answer is how the stand-in answers a name: with an A record of each of
|
||||||
|
// addrs, or with the response code rcode, unless it is 0, for no error.
|
||||||
|
type answer struct {
|
||||||
|
addrs []string
|
||||||
|
rcode uint16
|
||||||
|
}
|
||||||
|
|
||||||
|
// What the stand-in reads of a query, and writes in its reply.
|
||||||
|
const (
|
||||||
|
// headerLength is the length of a DNS message's header, which the
|
||||||
|
// question follows: its id, its flags, and how many questions,
|
||||||
|
// answers and other records it holds, two bytes each.
|
||||||
|
headerLength = 12
|
||||||
|
// typeAndClass is the length of the type and the class that end a
|
||||||
|
// question, after its name.
|
||||||
|
typeAndClass = 4
|
||||||
|
// replyFlags mark a reply to a query that asked for recursion, which
|
||||||
|
// is available, with no error. The response code goes in their last
|
||||||
|
// four bits.
|
||||||
|
replyFlags = 0x8180
|
||||||
|
// maxMessage is the longest query read over UDP.
|
||||||
|
maxMessage = 1232
|
||||||
|
)
|
||||||
|
|
||||||
|
// set has the stand-in answer name with given.
|
||||||
|
func (s *resolverStandIn) set(name string, given answer) {
|
||||||
|
s.mu.Lock()
|
||||||
|
defer s.mu.Unlock()
|
||||||
|
|
||||||
|
s.answers[name] = given
|
||||||
|
}
|
||||||
|
|
||||||
|
// dial connects Go's resolver to the stand-in through an in-memory
|
||||||
|
// connection, on which it sends each query, and reads each reply, after
|
||||||
|
// its length, as over TCP.
|
||||||
|
func (s *resolverStandIn) dial(context.Context, string, string) (net.Conn, error) {
|
||||||
|
client, server := net.Pipe()
|
||||||
|
|
||||||
|
go s.serve(server)
|
||||||
|
|
||||||
|
return client, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// serve answers the queries that come on conn until the resolver closes
|
||||||
|
// it.
|
||||||
|
func (s *resolverStandIn) serve(conn net.Conn) {
|
||||||
|
defer func() {
|
||||||
|
_ = conn.Close()
|
||||||
|
}()
|
||||||
|
|
||||||
|
for {
|
||||||
|
var length [2]byte
|
||||||
|
|
||||||
|
_, err := io.ReadFull(conn, length[:])
|
||||||
|
if err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
message := make([]byte, binary.BigEndian.Uint16(length[:]))
|
||||||
|
|
||||||
|
_, err = io.ReadFull(conn, message)
|
||||||
|
if err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
reply, answered := s.reply(message)
|
||||||
|
if !answered {
|
||||||
|
continue // the resolver gives up, and closes conn
|
||||||
|
}
|
||||||
|
|
||||||
|
//nolint:gosec // a reply of a few dozen bytes
|
||||||
|
_, err = conn.Write(append(binary.BigEndian.AppendUint16(nil, uint16(len(reply))),
|
||||||
|
reply...))
|
||||||
|
if err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// serveUDP answers the queries that come on conn, each in a datagram, as
|
||||||
|
// a resolver does, until conn is closed.
|
||||||
|
func (s *resolverStandIn) serveUDP(conn net.PacketConn) {
|
||||||
|
message := make([]byte, maxMessage)
|
||||||
|
|
||||||
|
for {
|
||||||
|
n, from, err := conn.ReadFrom(message)
|
||||||
|
if err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
reply, answered := s.reply(message[:n])
|
||||||
|
if answered {
|
||||||
|
_, _ = conn.WriteTo(reply, from)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// reply returns the stand-in's reply to message, a query, and false for
|
||||||
|
// none, while it hangs. It notes the name asked about.
|
||||||
|
func (s *resolverStandIn) reply(message []byte) ([]byte, bool) {
|
||||||
|
// The name is labels, each after its length, ended by a length of 0.
|
||||||
|
var labels []string
|
||||||
|
|
||||||
|
end := headerLength
|
||||||
|
for message[end] != 0 {
|
||||||
|
length := int(message[end])
|
||||||
|
labels = append(labels, string(message[end+1:end+1+length]))
|
||||||
|
end += 1 + length
|
||||||
|
}
|
||||||
|
|
||||||
|
end += 1 + typeAndClass
|
||||||
|
name := strings.Join(labels, ".") + "."
|
||||||
|
|
||||||
|
s.mu.Lock()
|
||||||
|
s.names = append(s.names, name)
|
||||||
|
given, found := s.answers[name]
|
||||||
|
hanging := s.hanging
|
||||||
|
s.mu.Unlock()
|
||||||
|
|
||||||
|
if hanging {
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
|
||||||
|
if !found {
|
||||||
|
given = answer{rcode: noSuchName}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The query's id, the flags, one question, the answers, and no other
|
||||||
|
// records, then the question, as asked.
|
||||||
|
reply := slices.Clone(message[:2])
|
||||||
|
reply = binary.BigEndian.AppendUint16(reply, replyFlags|given.rcode)
|
||||||
|
reply = binary.BigEndian.AppendUint16(reply, 1)
|
||||||
|
//nolint:gosec // a handful of answers
|
||||||
|
reply = binary.BigEndian.AppendUint16(reply, uint16(len(given.addrs)))
|
||||||
|
reply = append(reply, 0, 0, 0, 0)
|
||||||
|
reply = append(reply, message[headerLength:end]...)
|
||||||
|
|
||||||
|
// An A record starts with the name asked about, by a pointer to it in
|
||||||
|
// the question, then its type, A, its class, IN, how long it may be
|
||||||
|
// kept, 60 seconds, and the length of its address, 4 bytes.
|
||||||
|
record := []byte{0xc0, headerLength, 0, 1, 0, 1, 0, 0, 0, 60, 0, 4}
|
||||||
|
|
||||||
|
for _, addr := range given.addrs {
|
||||||
|
reply = append(reply, record...)
|
||||||
|
reply = append(reply, netip.MustParseAddr(addr).AsSlice()...)
|
||||||
|
}
|
||||||
|
|
||||||
|
return reply, true
|
||||||
|
}
|
||||||
|
|
||||||
|
// dnsblParams returns the DNSBLParams of zones, with the tests' cache TTL
|
||||||
|
// and timeout, by the bubble's clock, with alerts to a queue that sends
|
||||||
|
// none.
|
||||||
|
func dnsblParams(zones ...string) reputation.DNSBLParams {
|
||||||
|
return reputation.DNSBLParams{
|
||||||
|
Zones: zones,
|
||||||
|
CacheTTL: cacheTTL,
|
||||||
|
Timeout: timeout,
|
||||||
|
Now: time.Now,
|
||||||
|
ProcessLog: slog.New(slog.DiscardHandler),
|
||||||
|
Alerts: newQueue(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// waitForTheResolver waits, on the bubble's clock, an hour, until Go's
|
||||||
|
// resolver has given up on every stand-in that does not answer: it waits
|
||||||
|
// for a server as long as /etc/resolv.conf has it wait, a few seconds,
|
||||||
|
// even after the query was given up, and a bubble cannot end before it.
|
||||||
|
func waitForTheResolver() {
|
||||||
|
time.Sleep(time.Hour)
|
||||||
|
}
|
||||||
|
|
||||||
|
// newDNSBL returns the DNSBL of p, asking resolver.
|
||||||
|
func newDNSBL(resolver *resolverStandIn, p reputation.DNSBLParams) *reputation.DNSBL {
|
||||||
|
dnsbl := reputation.NewDNSBL(p)
|
||||||
|
dnsbl.SetDial(resolver.dial)
|
||||||
|
|
||||||
|
return dnsbl
|
||||||
|
}
|
||||||
|
|
||||||
|
// wantZones checks the zones whose verdict dnsbl says lists client, as a
|
||||||
|
// request from client finds them.
|
||||||
|
func wantZones(t *testing.T, dnsbl *reputation.DNSBL, client string, want ...string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
got := dnsbl.ListedBy(t.Context(), netip.MustParseAddr(client))
|
||||||
|
if !slices.Equal(got, want) {
|
||||||
|
t.Errorf("%s is listed by %v, want %v", client, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// wantQueries checks how many queries dnsbl made to zone, and how many of
|
||||||
|
// them failed.
|
||||||
|
func wantQueries(t *testing.T, dnsbl *reputation.DNSBL, queries, failures int) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
if dnsbl.Queries(zone) != queries || dnsbl.Failures(zone) != failures {
|
||||||
|
t.Errorf("%d queries and %d failures, want %d and %d", dnsbl.Queries(zone),
|
||||||
|
dnsbl.Failures(zone), queries, failures)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// wantAsked checks the names the stand-in was asked about, in any order.
|
||||||
|
func wantAsked(t *testing.T, resolver *resolverStandIn, want ...string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
resolver.mu.Lock()
|
||||||
|
got := slices.Sorted(slices.Values(resolver.names))
|
||||||
|
resolver.mu.Unlock()
|
||||||
|
|
||||||
|
slices.Sort(want)
|
||||||
|
|
||||||
|
if !slices.Equal(got, want) {
|
||||||
|
t.Errorf("asked about %v, want %v", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,9 +1,27 @@
|
|||||||
package reputation
|
package reputation
|
||||||
|
|
||||||
import "net/http"
|
import (
|
||||||
|
"context"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
"net/netip"
|
||||||
|
)
|
||||||
|
|
||||||
// SetTransport has l's fetches go through transport instead of the
|
// SetTransport has l's fetches go through transport instead of the
|
||||||
// network.
|
// network.
|
||||||
func (l *Lists) SetTransport(transport http.RoundTripper) {
|
func (l *Lists) SetTransport(transport http.RoundTripper) {
|
||||||
l.httpClient.Transport = transport
|
l.httpClient.Transport = transport
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// SetDial has d's queries go through dial instead of the network.
|
||||||
|
func (d *DNSBL) SetDial(
|
||||||
|
dial func(ctx context.Context, network, address string) (net.Conn, error),
|
||||||
|
) {
|
||||||
|
d.resolver = &net.Resolver{PreferGo: true, Dial: dial}
|
||||||
|
}
|
||||||
|
|
||||||
|
// LookUp asks zone about addr at once, as a query in the background does,
|
||||||
|
// and returns whether zone lists addr.
|
||||||
|
func (d *DNSBL) LookUp(zone string, addr netip.Addr) (bool, error) {
|
||||||
|
return d.lookUp(context.Background(), query{zone: zone, client: addr})
|
||||||
|
}
|
||||||
|
|||||||
@@ -2,8 +2,10 @@
|
|||||||
// blocklists of SWWAF_BLOCKLIST_URLS, and the file of AS:percent lines
|
// blocklists of SWWAF_BLOCKLIST_URLS, and the file of AS:percent lines
|
||||||
// SWWAF_ASN_LIMIT_PERCENT_URL names. It keeps the last good copy of each,
|
// SWWAF_ASN_LIMIT_PERCENT_URL names. It keeps the last good copy of each,
|
||||||
// whole, comment lines included, which is used while a fetch fails, and
|
// whole, comment lines included, which is used while a fetch fails, and
|
||||||
// when each was last tried, which the state package writes to
|
// when each was last tried. It also asks the DNSBL zones of
|
||||||
// reputation.json and reads from it, so that a restart keeps them too.
|
// SWWAF_DNSBL_ZONES about clients, and keeps their verdicts. The state
|
||||||
|
// package writes all of these to reputation.json and reads them from it,
|
||||||
|
// so that a restart keeps them too.
|
||||||
package reputation
|
package reputation
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
|||||||
@@ -35,8 +35,8 @@ const (
|
|||||||
// rule.
|
// rule.
|
||||||
ActionRuleBlocked = "rule_blocked"
|
ActionRuleBlocked = "rule_blocked"
|
||||||
// ActionDenied is a request refused because its client is in
|
// ActionDenied is a request refused because its client is in
|
||||||
// SWWAF_DENY_NETS, or in a blocklist while SWWAF_BLOCKLIST_ACTION is
|
// SWWAF_DENY_NETS, in a blocklist while SWWAF_BLOCKLIST_ACTION is deny,
|
||||||
// deny.
|
// or listed by a DNSBL zone while SWWAF_REPUTATION_ACTION is deny.
|
||||||
ActionDenied = "denied"
|
ActionDenied = "denied"
|
||||||
// ActionCountryDenied is a request refused for its client's country.
|
// ActionCountryDenied is a request refused for its client's country.
|
||||||
ActionCountryDenied = "country_denied"
|
ActionCountryDenied = "country_denied"
|
||||||
@@ -138,7 +138,8 @@ type Line struct {
|
|||||||
// Counts names its count: minute, hour or day for a rate limit, and
|
// Counts names its count: minute, hour or day for a rate limit, and
|
||||||
// minute_bytes, hour_bytes or day_bytes for a byte limit.
|
// minute_bytes, hour_bytes or day_bytes for a byte limit.
|
||||||
LimitHit string `json:"limit_hit,omitempty"`
|
LimitHit string `json:"limit_hit,omitempty"`
|
||||||
// Reputation are the URLs of the blocklists that list the client.
|
// Reputation are the URLs of the blocklists that list the client, then
|
||||||
|
// the DNSBL zones whose verdict lists it, their keys masked.
|
||||||
Reputation []string `json:"reputation,omitempty"`
|
Reputation []string `json:"reputation,omitempty"`
|
||||||
// Offence is the offence the request was held as, OffenceLimit.
|
// Offence is the offence the request was held as, OffenceLimit.
|
||||||
Offence string `json:"offence,omitempty"`
|
Offence string `json:"offence,omitempty"`
|
||||||
|
|||||||
@@ -201,6 +201,7 @@ func loadStateFiles(
|
|||||||
Limiter: server.Limiter,
|
Limiter: server.Limiter,
|
||||||
GeoJS: server.GeoJS,
|
GeoJS: server.GeoJS,
|
||||||
Lists: server.Lists,
|
Lists: server.Lists,
|
||||||
|
DNSBL: server.DNSBL,
|
||||||
Alerts: alertQueue,
|
Alerts: alertQueue,
|
||||||
Anomalies: server.Anomalies,
|
Anomalies: server.Anomalies,
|
||||||
Now: now,
|
Now: now,
|
||||||
|
|||||||
+41
-8
@@ -2,7 +2,8 @@
|
|||||||
// SWWAF_STATE_DIR, as the "Persistent state" section of SPEC.md describes:
|
// SWWAF_STATE_DIR, as the "Persistent state" section of SPEC.md describes:
|
||||||
// bans.json holds the bans, clients.json each client's counters and
|
// bans.json holds the bans, clients.json each client's counters and
|
||||||
// history, lookups.json GeoJS's answers, reputation.json the last try and
|
// history, lookups.json GeoJS's answers, reputation.json the last try and
|
||||||
// last good copy of each list fetched from a URL, and alerts.json the
|
// last good copy of each list fetched from a URL and the DNSBL zones'
|
||||||
|
// verdicts, and alerts.json the
|
||||||
// cooldowns, the hour under way, the alerts waiting for each destination
|
// cooldowns, the hour under way, the alerts waiting for each destination
|
||||||
// and the anomaly counters. Load
|
// and the anomaly counters. Load
|
||||||
// reads them at start, Watch takes in an admin's edit of one while
|
// reads them at start, Watch takes in an admin's edit of one while
|
||||||
@@ -76,13 +77,14 @@ type Params struct {
|
|||||||
// is (SWWAF_STATE_COUNTER_INTERVAL).
|
// is (SWWAF_STATE_COUNTER_INTERVAL).
|
||||||
WriteDelay time.Duration
|
WriteDelay time.Duration
|
||||||
CounterInterval time.Duration
|
CounterInterval time.Duration
|
||||||
// Ledger, Limiter, GeoJS, Lists, Alerts and Anomalies hold the state.
|
// Ledger, Limiter, GeoJS, Lists, DNSBL, Alerts and Anomalies hold the
|
||||||
// Alerts also receive a file_error alert for an edit set aside, and for
|
// state. Alerts also receive a file_error alert for an edit set aside,
|
||||||
// a write that fails while smallwebwaf runs.
|
// and for a write that fails while smallwebwaf runs.
|
||||||
Ledger *bans.Ledger
|
Ledger *bans.Ledger
|
||||||
Limiter *ratelimit.Limiter
|
Limiter *ratelimit.Limiter
|
||||||
GeoJS *lookup.GeoJS
|
GeoJS *lookup.GeoJS
|
||||||
Lists *reputation.Lists
|
Lists *reputation.Lists
|
||||||
|
DNSBL *reputation.DNSBL
|
||||||
Alerts *alerts.Queue
|
Alerts *alerts.Queue
|
||||||
Anomalies *anomaly.Counters
|
Anomalies *anomaly.Counters
|
||||||
// Now tells the time by which the counters' buckets run out, normally
|
// Now tells the time by which the counters' buckets run out, normally
|
||||||
@@ -145,8 +147,9 @@ type lookupsFile struct {
|
|||||||
// reputationFile is reputation.json, indented for an admin to read and
|
// reputationFile is reputation.json, indented for an admin to read and
|
||||||
// edit, so that each line of a list's copy is on a line of its own.
|
// edit, so that each line of a list's copy is on a line of its own.
|
||||||
type reputationFile struct {
|
type reputationFile struct {
|
||||||
Version int `json:"version"`
|
Version int `json:"version"`
|
||||||
Lists []reputation.List `json:"lists"`
|
Lists []reputation.List `json:"lists"`
|
||||||
|
Verdicts []reputation.Verdict `json:"verdicts"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// alertsFile is alerts.json, indented for an admin to read and edit.
|
// alertsFile is alerts.json, indented for an admin to read and edit.
|
||||||
@@ -432,6 +435,7 @@ func (f *Files) takeIn(name string, data []byte, edit bool) (int, error) {
|
|||||||
return 0, fmt.Errorf("%s: %w", path, err)
|
return 0, fmt.Errorf("%s: %w", path, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
f.params.DNSBL.Load(file.Verdicts)
|
||||||
entries = len(file.Lists)
|
entries = len(file.Lists)
|
||||||
case alertsJSON:
|
case alertsJSON:
|
||||||
waiting, err := f.takeInAlerts(path, data)
|
waiting, err := f.takeInAlerts(path, data)
|
||||||
@@ -567,6 +571,7 @@ func (f *Files) encode(name string) ([]byte, error) {
|
|||||||
case reputationJSON:
|
case reputationJSON:
|
||||||
return encodeIndented(reputationFile{
|
return encodeIndented(reputationFile{
|
||||||
Version: version, Lists: f.params.Lists.Snapshot(),
|
Version: version, Lists: f.params.Lists.Snapshot(),
|
||||||
|
Verdicts: f.params.DNSBL.Snapshot(),
|
||||||
})
|
})
|
||||||
default: // alerts.json
|
default: // alerts.json
|
||||||
held := f.params.Alerts.Snapshot()
|
held := f.params.Alerts.Snapshot()
|
||||||
@@ -729,8 +734,12 @@ func (f *lookupsFile) check(data []byte) error {
|
|||||||
// check refuses a list without its URL, which would name no list, or the
|
// check refuses a list without its URL, which would name no list, or the
|
||||||
// time it was last tried, which would have it fetched at once, and a copy
|
// time it was last tried, which would have it fetched at once, and a copy
|
||||||
// of it without the time it was fetched, or without its lines, which hold
|
// of it without the time it was fetched, or without its lines, which hold
|
||||||
// the list.
|
// the list. It refuses a verdict without its zone or its client, which
|
||||||
func (f *reputationFile) check([]byte) error {
|
// would be about no one, whether the zone lists the client, or the time
|
||||||
|
// it was fetched, which would drop it. A verdict's listed is false for a
|
||||||
|
// client the zone does not list, which Verdicts cannot tell from a
|
||||||
|
// missing one, so each listed is read again as written.
|
||||||
|
func (f *reputationFile) check(data []byte) error {
|
||||||
for i, kept := range f.Lists {
|
for i, kept := range f.Lists {
|
||||||
switch {
|
switch {
|
||||||
case kept.URL == "":
|
case kept.URL == "":
|
||||||
@@ -744,6 +753,30 @@ func (f *reputationFile) check([]byte) error {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
var written struct {
|
||||||
|
Verdicts []struct {
|
||||||
|
Listed *bool `json:"listed"`
|
||||||
|
} `json:"verdicts"`
|
||||||
|
}
|
||||||
|
|
||||||
|
err := json.Unmarshal(data, &written)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
for i, verdict := range f.Verdicts {
|
||||||
|
switch {
|
||||||
|
case verdict.Zone == "":
|
||||||
|
return fmt.Errorf("verdicts %w", missing(i, "zone"))
|
||||||
|
case !verdict.Client.IsValid():
|
||||||
|
return fmt.Errorf("verdicts %w", missing(i, "client"))
|
||||||
|
case written.Verdicts[i].Listed == nil:
|
||||||
|
return fmt.Errorf("verdicts %w", missing(i, "listed"))
|
||||||
|
case verdict.Fetched.IsZero():
|
||||||
|
return fmt.Errorf("verdicts %w", missing(i, "fetched"))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -38,9 +38,11 @@ const (
|
|||||||
lookupsJSON = "lookups.json"
|
lookupsJSON = "lookups.json"
|
||||||
reputationJSON = "reputation.json"
|
reputationJSON = "reputation.json"
|
||||||
alertsJSON = "alerts.json"
|
alertsJSON = "alerts.json"
|
||||||
// blocklistURL and torURL are the blocklists the tests' lists name.
|
// blocklistURL and torURL are the blocklists the tests' lists name, and
|
||||||
|
// dnsblZone the DNSBL zone of the tests' verdicts.
|
||||||
blocklistURL = "https://lists.example/drop.txt"
|
blocklistURL = "https://lists.example/drop.txt"
|
||||||
torURL = "https://lists.example/tor.txt"
|
torURL = "https://lists.example/tor.txt"
|
||||||
|
dnsblZone = "dnsbl.example"
|
||||||
// The AS number and AS name the tests' clients are looked up in.
|
// The AS number and AS name the tests' clients are looked up in.
|
||||||
asn = "AS64496"
|
asn = "AS64496"
|
||||||
asName = "Example Net"
|
asName = "Example Net"
|
||||||
@@ -210,7 +212,8 @@ const filledAlertsJSON = `{
|
|||||||
`
|
`
|
||||||
|
|
||||||
// filledReputationJSON is reputation.json holding the blocklists' last
|
// filledReputationJSON is reputation.json holding the blocklists' last
|
||||||
// tries and the copy of one, with its comment line, as fill puts them in.
|
// tries and the copy of one, with its comment line, and two verdicts of a
|
||||||
|
// DNSBL zone, as fill puts them in.
|
||||||
const filledReputationJSON = `{
|
const filledReputationJSON = `{
|
||||||
"version": 1,
|
"version": 1,
|
||||||
"lists": [
|
"lists": [
|
||||||
@@ -228,6 +231,20 @@ const filledReputationJSON = `{
|
|||||||
"url": "https://lists.example/tor.txt",
|
"url": "https://lists.example/tor.txt",
|
||||||
"tried": "2026-10-06T00:00:00Z"
|
"tried": "2026-10-06T00:00:00Z"
|
||||||
}
|
}
|
||||||
|
],
|
||||||
|
"verdicts": [
|
||||||
|
{
|
||||||
|
"zone": "dnsbl.example",
|
||||||
|
"client": "203.0.113.9",
|
||||||
|
"listed": true,
|
||||||
|
"fetched": "2026-10-05T23:00:00Z"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"zone": "dnsbl.example",
|
||||||
|
"client": "2001:db8::1",
|
||||||
|
"listed": false,
|
||||||
|
"fetched": "2026-10-05T22:00:00Z"
|
||||||
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
`
|
`
|
||||||
@@ -263,6 +280,8 @@ func TestFilesWrittenAndReadBack(t *testing.T) {
|
|||||||
t.Errorf("%s read back\n%+v\nwant\n%+v", reputationJSON, got, want)
|
t.Errorf("%s read back\n%+v\nwant\n%+v", reputationJSON, got, want)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
wantEqual(t, reputationJSON, after.DNSBL.Snapshot(), before.DNSBL.Snapshot())
|
||||||
|
|
||||||
if got, want := after.Alerts.Snapshot(), before.Alerts.Snapshot(); !reflect.DeepEqual(
|
if got, want := after.Alerts.Snapshot(), before.Alerts.Snapshot(); !reflect.DeepEqual(
|
||||||
got, want) {
|
got, want) {
|
||||||
t.Errorf("%s read back\n%+v\nwant\n%+v", alertsJSON, got, want)
|
t.Errorf("%s read back\n%+v\nwant\n%+v", alertsJSON, got, want)
|
||||||
@@ -422,8 +441,8 @@ func TestMissingFilesAreEmptyState(t *testing.T) {
|
|||||||
held := params.Alerts.Snapshot()
|
held := params.Alerts.Snapshot()
|
||||||
if len(params.Ledger.Snapshot()) != 0 || len(params.Limiter.Snapshot()) != 0 ||
|
if len(params.Ledger.Snapshot()) != 0 || len(params.Limiter.Snapshot()) != 0 ||
|
||||||
len(params.GeoJS.Snapshot()) != 0 || len(params.Lists.Snapshot()) != 0 ||
|
len(params.GeoJS.Snapshot()) != 0 || len(params.Lists.Snapshot()) != 0 ||
|
||||||
len(held.Cooldowns) != 0 || len(held.Waiting[alerts.DestinationWebhook]) != 0 ||
|
len(params.DNSBL.Snapshot()) != 0 || len(held.Cooldowns) != 0 ||
|
||||||
held.Hour.Sent != 0 {
|
len(held.Waiting[alerts.DestinationWebhook]) != 0 || held.Hour.Sent != 0 {
|
||||||
t.Error("state from no files")
|
t.Error("state from no files")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -592,6 +611,11 @@ func TestReputationJSONEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
|
|||||||
drop = `"url": "` + blocklistURL + `", `
|
drop = `"url": "` + blocklistURL + `", `
|
||||||
tried = `"tried": "2026-10-06T00:00:00Z", `
|
tried = `"tried": "2026-10-06T00:00:00Z", `
|
||||||
fetched = `"fetched": "2026-10-06T00:00:00Z"`
|
fetched = `"fetched": "2026-10-06T00:00:00Z"`
|
||||||
|
// verdictZone, verdictClient and listed start a verdict, which
|
||||||
|
// fetched ends.
|
||||||
|
verdictZone = `"zone": "` + dnsblZone + `", `
|
||||||
|
verdictClient = `"client": "198.51.100.7", `
|
||||||
|
listed = `"listed": false, `
|
||||||
)
|
)
|
||||||
|
|
||||||
for _, tc := range []struct {
|
for _, tc := range []struct {
|
||||||
@@ -621,6 +645,30 @@ func TestReputationJSONEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
|
|||||||
`{"url": "` + torURL + `", ` + tried + fetched + `}]}`,
|
`{"url": "` + torURL + `", ` + tried + fetched + `}]}`,
|
||||||
`: entry 2 has no "lines"`,
|
`: entry 2 has no "lines"`,
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"a verdict without its zone",
|
||||||
|
`{"version": 1, "verdicts": [{` + verdictClient + listed + fetched + `}]}`,
|
||||||
|
`: verdicts entry 1 has no "zone"`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"a verdict without its client",
|
||||||
|
`{"version": 1, "verdicts": [{` + verdictZone + listed + fetched + `}]}`,
|
||||||
|
`: verdicts entry 1 has no "client"`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
// A client the zone does not list has a listed of false, which is
|
||||||
|
// not having none.
|
||||||
|
"a verdict without whether the zone lists the client",
|
||||||
|
`{"version": 1, "verdicts": [{` + verdictZone + verdictClient + listed +
|
||||||
|
fetched + `}, {` + verdictZone + verdictClient + fetched + `}]}`,
|
||||||
|
`: verdicts entry 2 has no "listed"`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"a verdict without the time it was fetched",
|
||||||
|
`{"version": 1, "verdicts": [{` + verdictZone + verdictClient +
|
||||||
|
`"listed": true}]}`,
|
||||||
|
`: verdicts entry 1 has no "fetched"`,
|
||||||
|
},
|
||||||
} {
|
} {
|
||||||
t.Run(tc.name, func(t *testing.T) {
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
@@ -1133,7 +1181,8 @@ func TestEditOfEachFileTakenIn(t *testing.T) {
|
|||||||
|
|
||||||
edit(t, dir, reputationJSON, `{"version": 1, "lists": [{"url": "`+blocklistURL+`", `+
|
edit(t, dir, reputationJSON, `{"version": 1, "lists": [{"url": "`+blocklistURL+`", `+
|
||||||
`"tried": "2026-10-06T00:00:00Z", "fetched": "2026-10-06T00:00:00Z", `+
|
`"tried": "2026-10-06T00:00:00Z", "fetched": "2026-10-06T00:00:00Z", `+
|
||||||
`"lines": ["198.51.100.7"]}]}`)
|
`"lines": ["198.51.100.7"]}], "verdicts": [{"zone": "`+dnsblZone+`", `+
|
||||||
|
`"client": "198.51.100.7", "listed": true, "fetched": "2026-10-06T00:00:00Z"}]}`)
|
||||||
wantTakenIn(t, lines, dir, reputationJSON)
|
wantTakenIn(t, lines, dir, reputationJSON)
|
||||||
|
|
||||||
listedBy := params.Lists.ListedBy(client.Addr())
|
listedBy := params.Lists.ListedBy(client.Addr())
|
||||||
@@ -1142,6 +1191,10 @@ func TestEditOfEachFileTakenIn(t *testing.T) {
|
|||||||
client.Addr(), listedBy)
|
client.Addr(), listedBy)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
wantEqual(t, reputationJSON, params.DNSBL.Snapshot(), []reputation.Verdict{{
|
||||||
|
Zone: dnsblZone, Client: client.Addr(), Listed: true, Fetched: midnight(),
|
||||||
|
}})
|
||||||
|
|
||||||
// A netblock with bits past its length is read as the netblock it is
|
// A netblock with bits past its length is read as the netblock it is
|
||||||
// in.
|
// in.
|
||||||
edit(t, dir, alertsJSON, `{"version": 1, "cooldowns": [{"event": "ban", `+
|
edit(t, dir, alertsJSON, `{"version": 1, "cooldowns": [{"event": "ban", `+
|
||||||
@@ -1558,6 +1611,10 @@ func newParams(dir string) state.Params {
|
|||||||
BlocklistURLs: []string{blocklistURL, torURL}, Refresh: 24 * time.Hour,
|
BlocklistURLs: []string{blocklistURL, torURL}, Refresh: 24 * time.Hour,
|
||||||
Now: midnight, ProcessLog: discard, Alerts: queue,
|
Now: midnight, ProcessLog: discard, Alerts: queue,
|
||||||
}),
|
}),
|
||||||
|
DNSBL: reputation.NewDNSBL(reputation.DNSBLParams{
|
||||||
|
Zones: []string{dnsblZone}, CacheTTL: 24 * time.Hour, Timeout: time.Second,
|
||||||
|
Now: midnight, ProcessLog: discard, Alerts: queue,
|
||||||
|
}),
|
||||||
Alerts: queue,
|
Alerts: queue,
|
||||||
Anomalies: anomaly.New(anomaly.Params{
|
Anomalies: anomaly.New(anomaly.Params{
|
||||||
Net: anomaly.Thresholds{RequestsPerMinute: 1000},
|
Net: anomaly.Thresholds{RequestsPerMinute: 1000},
|
||||||
@@ -1582,9 +1639,10 @@ func office() netip.Prefix {
|
|||||||
|
|
||||||
// fill puts a permanent ban an admin made, a ban for a broken limit and
|
// fill puts a permanent ban an admin made, a ban for a broken limit and
|
||||||
// one for a clear sign of attack, clients with counts and histories,
|
// one for a clear sign of attack, clients with counts and histories,
|
||||||
// GeoJS answers, the blocklists' last tries and the copy of one, as
|
// GeoJS answers, the blocklists' last tries and the copy of one, and two
|
||||||
// filledReputationJSON holds them, and alerts and anomaly counters, as
|
// verdicts of a DNSBL zone, as filledReputationJSON holds them, and alerts
|
||||||
// filledAlertsJSON holds them, into the parts of params.
|
// and anomaly counters, as filledAlertsJSON holds them, into the parts of
|
||||||
|
// params.
|
||||||
func fill(params state.Params) {
|
func fill(params state.Params) {
|
||||||
now := midnight()
|
now := midnight()
|
||||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
@@ -1631,6 +1689,14 @@ func fill(params state.Params) {
|
|||||||
panic(err) // the copy reads
|
panic(err) // the copy reads
|
||||||
}
|
}
|
||||||
|
|
||||||
|
params.DNSBL.Load([]reputation.Verdict{
|
||||||
|
{
|
||||||
|
Zone: dnsblZone, Client: netip.MustParseAddr("2001:db8::1"),
|
||||||
|
Fetched: now.Add(-2 * time.Hour),
|
||||||
|
},
|
||||||
|
{Zone: dnsblZone, Client: client.Addr(), Listed: true, Fetched: now.Add(-time.Hour)},
|
||||||
|
})
|
||||||
|
|
||||||
// An alert waiting, a repeat of it the cooldown holds back, another
|
// An alert waiting, a repeat of it the cooldown holds back, another
|
||||||
// alert waiting, and one past the two an hour, for the hour's summary.
|
// alert waiting, and one past the two an hour, for the hour's summary.
|
||||||
ban := alerts.Alert{
|
ban := alerts.Alert{
|
||||||
|
|||||||
Reference in New Issue
Block a user