Blocklists and an AS percentage file fetched by URL (closes #29)

SWWAF_BLOCKLIST_URLS names lists of addresses and netblocks, fetched every
SWWAF_BLOCKLIST_REFRESH (24h, never under 1h); an IPv4-mapped line stands
for its IPv4 address or netblock. reputation.json keeps each list's last
try, failed or not, even one cut off by a stop, which a restart waits on
as a running instance does, and its last good copy, whole, used while a
fetch fails. SWWAF_BLOCKLIST_ACTION denies, limits or only logs a listed
client; the log line names the lists, each raises reputation_hit, and a
failed fetch raises source_failure. SWWAF_ASN_LIMIT_PERCENT_URL is fetched
the same way and counts as SWWAF_ASN_LIMIT_PERCENT does, the lower winning.

Judgement call: a failed fetch is retried after the refresh, not sooner.
Not done: ban notes do not name the lists yet.

Model: opus-5-5

README.md

@@ -22,30 +22,37 @@ fields, which come a little later, and the metrics endpoint and the header size
 and the idle time as settings, which come last in it. So are the four parts of
 the stage after it: the rule files, with the bans for a clear sign of attack,
 the other admin endpoints, alerts to all three destinations, a JSON webhook,
-Slack and ntfy, and remote log sending. So are two parts of the stage after
-that: the AS number and country of every client, looked up through GeoJS or in
-the IPinfo Lite database file, and the byte limits. `smallwebwaf` passes each
-request to the app and the app's answer back, unchanged, within its timeouts and
-size limits, works out each client's address, looks up its AS number and country
+Slack and ntfy, and remote log sending. So is the stage after that: the AS
+number and country of every client, looked up through GeoJS or in the IPinfo
+Lite database file, the byte limits, the biased thresholds, lower limits for the
+AS numbers and countries you list, and the anomaly thresholds, alerts for
+unusual traffic that refuse nothing. So is the first part of the stage after
+that: the blocklists you name by URL, which it fetches and keeps, and a file of
+AS numbers' percentages, fetched the same way. `smallwebwaf` passes each request
+to the app and the app's answer back, unchanged, within its timeouts and size
+limits, works out each client's address, looks up its AS number and country
 unless you switch that off, bans a client that sends too many requests or too
-many bytes, not counting those for the paths you choose, refuses a client that
-comes from a country you refuse or from a network you refuse, lets the networks
-you choose through, checks each request against the rule files and bans a client
-whose request is a clear sign of attack, keeps its bans, each client's counters
-and history, and GeoJS's answers in JSON files across restarts, takes in your
-edits of those files, such as a ban you make, keep or lift, and of the rule
-files while it runs, writes a JSON log line for every request, sends its log
-lines to a syslog server too if you name one, sends an alert to a webhook, to
-Slack and to ntfy, each if you name one, for each ban it makes or makes
-permanent, for GeoJS failing, for a rule file or state file with an error and
-for a replacement of the lookup database it cannot read, serves Prometheus
-metrics to a scraper that holds the metrics token, lets an admin who holds the
-admin token list, add and lift bans and ask what it knows of a client, and in
-`observe` mode passes on the requests it would refuse, logging what it would
-have done with them. It comes as the image the app's own image is built on. The
-rest of the design comes after that, in the order of the build order in
-[`SPEC.md`](SPEC.md). The survey of existing tools that led to the design is in
-[`EVALUATION.md`](EVALUATION.md).
+many bytes, not counting those for the paths you choose, with lower limits for
+the clients of the AS numbers and countries you list, refuses a client that
+comes from a country you refuse or from a network you refuse, refuses, limits or
+only notes a client a blocklist you name lists, lets the networks you choose
+through, checks each request against the rule files and bans a client whose
+request is a clear sign of attack, keeps its bans, each client's counters and
+history, GeoJS's answers and the last good copy of each list it fetches in JSON
+files across restarts, takes in your edits of those files, such as a ban you
+make, keep or lift, and of the rule files while it runs, writes a JSON log line
+for every request, sends its log lines to a syslog server too if you name one,
+sends an alert to a webhook, to Slack and to ntfy, each if you name one, for
+each ban it makes or makes permanent, for traffic over an anomaly threshold you
+set, for a client a blocklist lists, for GeoJS failing or a list it cannot
+fetch, for a rule file or state file with an error and for a replacement of the
+lookup database it cannot read, serves Prometheus metrics to a scraper that
+holds the metrics token, lets an admin who holds the admin token list, add and
+lift bans and ask what it knows of a client, and in `observe` mode passes on the
+requests it would refuse, logging what it would have done with them. It comes as
+the image the app's own image is built on. The rest of the design comes after
+that, in the order of the build order in [`SPEC.md`](SPEC.md). The survey of
+existing tools that led to the design is in [`EVALUATION.md`](EVALUATION.md).
 
 ## Getting started
 
@@ -120,6 +127,21 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
   They leave out what the rate limits leave out: a client in `SWWAF_ALLOW_NETS`
   or `SWWAF_RATE_LIMIT_EXEMPT_NETS`, and a request for a path
   `SWWAF_RATE_LIMIT_EXEMPT_PATHS` exempts.
+- Gives the clients of the AS numbers and countries the biased thresholds list,
+  `SWWAF_ASN_LIMIT_PERCENT`, the file `SWWAF_ASN_LIMIT_PERCENT_URL` names and
+  `SWWAF_COUNTRY_LIMIT_PERCENT`, the percentage they give of every rate limit
+  and byte limit, so that the same rules ban them after fewer requests, and,
+  while `SWWAF_UNKNOWN_LIMIT_PERCENT` is below 100, every client without a
+  country that percentage. A client to which several apply gets the lowest. For
+  the byte limits, `SWWAF_ASN_BYTES_PERCENT` gives an AS number it lists a
+  percentage in place of those `SWWAF_ASN_LIMIT_PERCENT` and the file give it,
+  and `SWWAF_COUNTRY_BYTES_PERCENT` gives a country it lists one in place of the
+  one `SWWAF_COUNTRY_LIMIT_PERCENT` gives it. Each client is counted on its own,
+  against its own lowered limits: no budget is shared by a whole AS number or
+  country, which one abuser could use up and so lock out everyone else there.
+  The log line of each request the rate limits count gives its client's
+  percentages below 100 and the settings that gave them, and so do the notes of
+  a ban for a lowered limit, and its alert.
 - Bans a client that breaks a rate limit or a byte limit, as "Bans" in
   [`SPEC.md`](SPEC.md) describes: the first ban lasts an hour, and a limit
   broken again within a day of a ban ending bans for three times as long as that
@@ -131,17 +153,18 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
   the client is not looked up, and is not counted for the rate limits. A ban
   sets the client's counters back to zero. Each ban carries notes for deciding
   whether to lift it: the limit, whether it is on requests or bytes, its window
-  and the requests or bytes counted in it, the request that broke it, the
-  client's AS number, AS name and country once they are looked up, the
-  netblock's requests since it was first seen, how many of them the ban has
-  refused, and how many bans the netblock had before, for a broken limit, for a
-  clear sign of attack and by an admin. At most `SWWAF_MAX_BANS` bans
-  `smallwebwaf` made are kept, past, active and permanent; past that, the
-  earliest such ban of the netblock that has gone longest without a request is
-  dropped first. The bans whose cause is `admin`, those you make or keep, are
-  kept besides, and never dropped. `bans.json` shows the bans and their notes, a
-  restart lifts none, and you make, keep or lift a ban by editing it (see "State
-  files" below).
+  and the requests or bytes counted in it, the client's percentage of that kind
+  of limit and the setting that gave it when a biased threshold lowered the
+  limit, the request that broke it, the client's AS number, AS name and country
+  once they are looked up, the netblock's requests since it was first seen, how
+  many of them the ban has refused, and how many bans the netblock had before,
+  for a broken limit, for a clear sign of attack and by an admin. At most
+  `SWWAF_MAX_BANS` bans `smallwebwaf` made are kept, past, active and permanent;
+  past that, the earliest such ban of the netblock that has gone longest without
+  a request is dropped first. The bans whose cause is `admin`, those you make or
+  keep, are kept besides, and never dropped. `bans.json` shows the bans and
+  their notes, a restart lifts none, and you make, keep or lift a ban by editing
+  it (see "State files" below).
 - Checks each request against the rules of the rule files (see "Rule files"
   below) after the rate limits, and before its body is read. A `log` rule that
   matches is noted in the log line; a `block` rule refuses the request with
@@ -161,46 +184,56 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
   IPinfo Lite database file while `SWWAF_LOOKUP_SOURCE` is `file`, after the
   static lists and bans, unless `SWWAF_LOOKUP_SOURCE` is `off` (see "Country and
   AS number lookup" below), for the request log, the client's history, the notes
-  of its bans, their alerts and the metrics. The file answers at once. With
-  GeoJS, a request waits for its client's first answer only while a setting acts
-  on it, a country list or `SWWAF_ADD_LOOKUP_HEADERS`. Otherwise it goes on at
-  once, and the answer reaches the client's history and the notes of its bans
-  when it comes, but not the log lines of the requests that went on without it,
-  nor the alerts already raised for those bans.
+  of its bans, their alerts, the metrics and the anomaly thresholds per AS
+  number. The file answers at once. With GeoJS, a request waits for its client's
+  first answer only while a setting acts on it, a country list,
+  `SWWAF_ADD_LOOKUP_HEADERS` or a biased threshold that lowers a limit.
+  Otherwise it goes on at once, and the answer reaches the client's history and
+  the notes of its bans when it comes, but not the log lines of the requests
+  that went on without it, nor the alerts already raised for those bans.
 - Refuses a request from a country you refuse with `SWWAF_BAN_RESPONSE`, as soon
   as the client's country is known and before its body is read; such a request
   is not counted for the rate limits. A client on a private, loopback or
   link-local address has no country and is never looked up:
   `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses it unless it is in
   `SWWAF_ALLOW_NETS`, and `SWWAF_DENIED_COUNTRIES` does not refuse it.
+- Checks the client's own address against the blocklists `SWWAF_BLOCKLIST_URLS`
+  names, after the country lists and before the rate limits (see "Blocklists"
+  below). With `SWWAF_BLOCKLIST_ACTION` at `deny`, its default, a request from a
+  client a blocklist lists is refused with `SWWAF_BAN_RESPONSE` before its body
+  is read; it is not counted for the rate limits, and makes no ban. With
+  `limit:<percent>` the client gets that percentage of every rate limit and byte
+  limit, the lowest of its percentages applying, as for a biased threshold, and
+  with `log` nothing more is done. Whatever the action, the request's log line
+  names the lists, and each raises an alert.
 - Checks the client's own address against the static lists, the three netblock
   settings below, before anything else, its lookup included. A client in
-  `SWWAF_ALLOW_NETS` skips bans, the country lists, the rate limits, the byte
-  limits and the rule files, and is not looked up; the timeouts and size limits
-  still apply. A client in `SWWAF_DENY_NETS` is refused with
+  `SWWAF_ALLOW_NETS` skips bans, the country lists, the blocklists, the rate
+  limits, the byte limits and the rule files, and is not looked up; the timeouts
+  and size limits still apply. A client in `SWWAF_DENY_NETS` is refused with
   `SWWAF_BAN_RESPONSE` before its body is read, and the request is not counted
   for the rate limits; an address in `SWWAF_ALLOW_NETS` too is let through. A
   client in `SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither counted nor refused by the
   rate limits, and has no bytes counted by the byte limits; the country lists,
   the rule files and bans still apply to it.
 - In `observe` mode, with `SWWAF_MODE=observe`, refuses none of the requests
-  that `SWWAF_DENY_NETS`, a ban, the country lists, a rate limit or a rule would
-  refuse: it passes them to the app, and their log lines name what `enforce`
-  mode would have done (see `would_action` in "Request log" below). The checks
-  run, and requests and bytes are counted, as in `enforce` mode, with three
-  differences: neither a broken rate limit or byte limit nor a `ban` rule makes
-  a ban; a broken limit does not set the client's counters back to zero, so each
-  request over a rate limit is logged as one that would be refused, and each
-  whose bytes keep the client over a byte limit as breaking it; and a request
-  under a ban does not make it permanent. As in `enforce` mode, the bytes
-  counted are only those of the requests `enforce` mode would have passed to the
-  app. A ban it would have made, or made permanent, raises the alert `enforce`
-  mode would have raised, marked as what would have happened (see "Alerts"
-  below). The bans in `bans.json` are kept, and refuse requests again when
-  `smallwebwaf` next runs in `enforce` mode, as long as they last. The timeouts
-  and size limits still apply, since they protect `smallwebwaf` and the app
-  themselves, and a request for one of `smallwebwaf`'s own endpoints without its
-  token is still answered `401`. It is for trying a configuration before
+  that `SWWAF_DENY_NETS`, a ban, the country lists, a blocklist, a rate limit or
+  a rule would refuse: it passes them to the app, and their log lines name what
+  `enforce` mode would have done (see `would_action` in "Request log" below).
+  The checks run, and requests and bytes are counted, as in `enforce` mode, with
+  three differences: neither a broken rate limit or byte limit nor a `ban` rule
+  makes a ban; a broken limit does not set the client's counters back to zero,
+  so each request over a rate limit is logged as one that would be refused, and
+  each whose bytes keep the client over a byte limit as breaking it; and a
+  request under a ban does not make it permanent. As in `enforce` mode, the
+  bytes counted are only those of the requests `enforce` mode would have passed
+  to the app. A ban it would have made, or made permanent, raises the alert
+  `enforce` mode would have raised, marked as what would have happened (see
+  "Alerts" below). The bans in `bans.json` are kept, and refuse requests again
+  when `smallwebwaf` next runs in `enforce` mode, as long as they last. The
+  timeouts and size limits still apply, since they protect `smallwebwaf` and the
+  app themselves, and a request for one of `smallwebwaf`'s own endpoints without
+  its token is still answered `401`. It is for trying a configuration before
   enforcing it.
 - Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any
   check and without asking the app, for the image's health check.
@@ -220,13 +253,31 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
 - Sends every line it writes on stdout to a syslog server as well, while
   `SWWAF_LOG_REMOTE_URL` names one (see "Sending the log to a syslog server"
   below).
-- Sends an alert for each ban it makes or makes permanent, for GeoJS failing,
-  for a rule file or state file with an error, and for a replacement of the
-  lookup database it cannot read, holding back repeats and, past an hourly
-  limit, rolling the rest into one summary, to each destination you name: as a
-  JSON object to the webhook `SWWAF_ALERT_WEBHOOK_URL` names, as a message to
-  the Slack incoming webhook `SWWAF_ALERT_SLACK_WEBHOOK_URL` names, and as a
-  message to the ntfy topic `SWWAF_ALERT_NTFY_URL` names (see "Alerts" below).
+- Sends an alert for each ban it makes or makes permanent, for a count over an
+  anomaly threshold, for a client a blocklist lists, for GeoJS failing or a list
+  it cannot fetch, for a rule file or state file with an error, and for a
+  replacement of the lookup database it cannot read, holding back repeats and,
+  past an hourly limit, rolling the rest into one summary, to each destination
+  you name: as a JSON object to the webhook `SWWAF_ALERT_WEBHOOK_URL` names, as
+  a message to the Slack incoming webhook `SWWAF_ALERT_SLACK_WEBHOOK_URL` names,
+  and as a message to the ntfy topic `SWWAF_ALERT_NTFY_URL` names (see "Alerts"
+  below).
+- Counts requests and their bytes over a minute and an hour, per client, per
+  netblock around a client, per AS number, for the whole service and per named
+  netblock, and sends an `anomaly` alert for a count over the anomaly threshold
+  you set for it (see the anomaly thresholds below). These thresholds only
+  alert: they refuse and ban nothing. A scope whose four thresholds are all off
+  is not counted, and within a scope only the counts whose threshold is set are
+  counted. Every request but the health check is counted, whatever is done with
+  it: one that is refused, one from a client in `SWWAF_ALLOW_NETS` or
+  `SWWAF_RATE_LIMIT_EXEMPT_NETS`, and one for a path in
+  `SWWAF_RATE_LIMIT_EXEMPT_PATHS`, with its body bytes once it has ended, those
+  of the answer, of the request or both, as `SWWAF_BYTES_COUNT` says. A request
+  is counted for its client's AS number only if the lookup has given that by the
+  time the request ends: no request waits for it, and a client in
+  `SWWAF_ALLOW_NETS`, which is not looked up, counts for no AS number. At most
+  20,000 counters are kept, the one counted least recently dropped first, and
+  `alerts.json` keeps them across a restart (see "State files" below).
 
 ## Settings
 
@@ -273,8 +324,8 @@ effective settings are logged at start.
 - `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body.
 - `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body.
 - `SWWAF_ALLOW_NETS` (default empty): netblocks whose clients skip bans, the
-  country lists, the rate limits, the byte limits and the rule files, such as
-  your monitoring or your own networks.
+  country lists, the blocklists, the rate limits, the byte limits and the rule
+  files, such as your monitoring or your own networks.
 - `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients the
   rate limits and the byte limits do not apply to, such as a machine that talks
   to the app all day.
@@ -312,9 +363,12 @@ effective settings are logged at start.
   country are looked up: `geojs`, the GeoJS web service, which is then told the
   address of every new visitor, `file`, the IPinfo Lite database file
   `SWWAF_LOOKUP_DB_PATH` names, or `off`, which looks up no client and sends no
-  address to GeoJS. With `off`, a country list that is not empty, or
-  `SWWAF_ADD_LOOKUP_HEADERS` set to `true`, stops the start, with a message
-  naming it and `SWWAF_LOOKUP_SOURCE`.
+  address to GeoJS. With `off`, a country list that is not empty,
+  `SWWAF_ADD_LOOKUP_HEADERS` set to `true`, a biased threshold that lowers a
+  limit, a list of them that is not empty, `SWWAF_ASN_LIMIT_PERCENT_URL` set or
+  `SWWAF_UNKNOWN_LIMIT_PERCENT` below 100, or an anomaly threshold per AS number
+  that is not `off`, stops the start, with a message naming it and
+  `SWWAF_LOOKUP_SOURCE`.
 - `SWWAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in its
   `.mmdb` form, for `SWWAF_LOOKUP_SOURCE=file`. `file` without it, or it with
   any other `SWWAF_LOOKUP_SOURCE`, the default included, stops the start, with a
@@ -334,11 +388,51 @@ effective settings are logged at start.
   countries whose clients get through, for example `us,de`. A client whose
   country cannot be found is refused too, so that new clients are not let in
   whenever GeoJS stops answering.
+- `SWWAF_ASN_LIMIT_PERCENT` (default empty): AS numbers, each with the
+  percentage of every rate limit and byte limit its clients get, such as
+  `AS14061:50,AS16276:50,AS45102:25`. A lowered limit is rounded down to a whole
+  number: half of 1000 requests a minute is 500, and half of 5 is 2. `0` is a
+  zero allowance: the client's first request breaks a limit, and bans it.
+- `SWWAF_COUNTRY_LIMIT_PERCENT` (default empty): the same by country, such as
+  `cn:25,ru:50`.
+- `SWWAF_ASN_BYTES_PERCENT` and `SWWAF_COUNTRY_BYTES_PERCENT` (default empty):
+  the same for the byte limits alone. For an AS number or a country one of them
+  lists, its percentage takes the place, for the byte limits, of the one
+  `SWWAF_ASN_LIMIT_PERCENT` or `SWWAF_COUNTRY_LIMIT_PERCENT` gives, so that
+  `SWWAF_ASN_LIMIT_PERCENT=AS14061:50` with
+  `SWWAF_ASN_BYTES_PERCENT=AS14061:100` halves that AS number's rate limits and
+  leaves its byte limits whole.
+- `SWWAF_UNKNOWN_LIMIT_PERCENT` (default `100`): the percentage of every limit a
+  client without a country gets: one the lookup cannot place, one on a private,
+  loopback or link-local address, which is never looked up, and one whose answer
+  from GeoJS has not come in time.
+- `SWWAF_ASN_LIMIT_PERCENT_URL` (default unset): the `http` or `https` URL of a
+  file of AS numbers, each with its percentage of every rate limit and byte
+  limit, one such as `AS14061:50` to a line, whose percentages count as those of
+  `SWWAF_ASN_LIMIT_PERCENT` do, so that one list can serve several instances.
+  For an AS number both give a percentage, the lower applies, and an AS number
+  the file lists twice gets the lower of its two. It is fetched and kept as a
+  blocklist is (see "Blocklists" below). A URL `SWWAF_BLOCKLIST_URLS` names too
+  stops the start.
+- `SWWAF_BLOCKLIST_URLS` (default empty): the blocklists, as `http` or `https`
+  URLs without a user or a fragment, such as
+  `https://www.spamhaus.org/drop/drop.txt` (see "Blocklists" below). A URL
+  listed twice stops the start.
+- `SWWAF_BLOCKLIST_REFRESH` (default `24h`): how long after a list was last
+  fetched, or a fetch of it failed, it is fetched again, for the blocklists and
+  `SWWAF_ASN_LIMIT_PERCENT_URL`. Less than `1h` stops the start: the Spamhaus
+  lists may be fetched no more than once an hour.
+- `SWWAF_BLOCKLIST_ACTION` (default `deny`): what is done with a client a
+  blocklist lists: `deny` refuses its requests with `SWWAF_BAN_RESPONSE`,
+  `limit:<percent>`, such as `limit:25`, gives it that percentage of every rate
+  limit and byte limit, and `log` does nothing more than note the lists in the
+  log line and raise the alert.
 - `SWWAF_BAN_RESPONSE` (default `403`): how a refused client is answered, one
   that is banned, breaks a rate limit, matches a `ban` rule, is in
-  `SWWAF_DENY_NETS` or comes from a refused country: `403`, `429`, or `close` to
-  close the connection without an answer. Behind traefik, `close` does not leave
-  the client unanswered: traefik answers `502`, as it does whenever its backend
+  `SWWAF_DENY_NETS`, comes from a refused country or is in a blocklist while
+  `SWWAF_BLOCKLIST_ACTION` is `deny`: `403`, `429`, or `close` to close the
+  connection without an answer. Behind traefik, `close` does not leave the
+  client unanswered: traefik answers `502`, as it does whenever its backend
   drops a connection. A `block` rule always answers `403`.
 - `SWWAF_LIMIT_BAN_DURATION` (default `1h`): the ban for a first broken rate
   limit or byte limit.
@@ -431,29 +525,65 @@ effective settings are logged at start.
   `SWWAF_INSTANCE_NAME`, which ntfy is sent in the title.
 - `SWWAF_ALERT_EVENTS` (default
   `ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure,file_error`):
-  the events alerts are sent for. `waf_block`, `anomaly` and `reputation_hit`
-  come with the features that raise them; nothing raises them yet.
+  the events alerts are sent for. `waf_block` comes with the Core Rule Set;
+  nothing raises it yet.
 - `SWWAF_ALERT_COOLDOWN` (default `15m`): how long a repeat of an alert is held
   back (see "Alerts" below).
 - `SWWAF_ALERT_MAX_PER_HOUR` (default `60`): the most alerts sent in an hour;
   the rest of the hour's alerts are rolled into one summary.
+- `SWWAF_ANOMALY_CLIENT_REQUESTS_PER_MINUTE`,
+  `SWWAF_ANOMALY_CLIENT_REQUESTS_PER_HOUR`,
+  `SWWAF_ANOMALY_CLIENT_BYTES_PER_MINUTE` and
+  `SWWAF_ANOMALY_CLIENT_BYTES_PER_HOUR` (default `off`): the anomaly thresholds
+  per client, the most requests and the most bytes a client may have counted in
+  a minute and in an hour before an `anomaly` alert is sent for it. They refuse
+  and ban nothing. Each scope below has the same four thresholds, their names
+  ending in `REQUESTS_PER_MINUTE`, `REQUESTS_PER_HOUR`, `BYTES_PER_MINUTE` and
+  `BYTES_PER_HOUR`, and each is `off` by default, since what is unusual depends
+  on each service's normal traffic, which the metrics show.
+- `SWWAF_ANOMALY_NET_REQUESTS_PER_MINUTE`, `..._PER_HOUR`,
+  `SWWAF_ANOMALY_NET_BYTES_PER_MINUTE` and `..._PER_HOUR` (default `off`): the
+  anomaly thresholds per netblock around a client, which is
+  `SWWAF_ANOMALY_NET_V4_PREFIX` (default `24`) long, from 0 to 32, for an IPv4
+  client, and `SWWAF_ANOMALY_NET_V6_PREFIX` (default `48`) long, from 0 to 128,
+  for an IPv6 one.
+- `SWWAF_ANOMALY_ASN_REQUESTS_PER_MINUTE`, `..._PER_HOUR`,
+  `SWWAF_ANOMALY_ASN_BYTES_PER_MINUTE` and `..._PER_HOUR` (default `off`): the
+  anomaly thresholds per AS number.
+- `SWWAF_ANOMALY_TOTAL_REQUESTS_PER_MINUTE`, `..._PER_HOUR`,
+  `SWWAF_ANOMALY_TOTAL_BYTES_PER_MINUTE` and `..._PER_HOUR` (default `off`): the
+  anomaly thresholds for the whole service.
+- `SWWAF_WATCH_NETS` (default empty): named netblocks, each a name, `=` and a
+  netblock, such as `office=203.0.113.0/24,scraper-x=198.51.100.0/22`. An item
+  without `=`, without a name or without a valid netblock, or a name listed
+  twice, stops the start. `SWWAF_WATCH_REQUESTS_PER_MINUTE`, `..._PER_HOUR`,
+  `SWWAF_WATCH_BYTES_PER_MINUTE` and `..._PER_HOUR` (default `off`) are the
+  anomaly thresholds of each named netblock as a whole, which counts every
+  client inside it; a client inside several is counted in each.
 
 Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are
 bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024
-bytes). Rate limits are whole numbers of requests, and byte limits are sizes.
+bytes). Rate limits and the anomaly thresholds on requests are whole numbers of
+requests, and byte limits and the anomaly thresholds on bytes are sizes.
 Netblocks are in CIDR form, and a bare address stands for itself alone.
 Countries are the two-letter codes ISO 3166-1 assigns today, and `xk` for
 Kosovo, in either case (`de` and `DE` are the same); any other code, such as
 `nk` (North Korea is `kp`) or the withdrawn `su`, stops the start, and so does a
-code on both country lists. `off` switches a timeout, a size limit, a rate
-limit, a byte limit, `SWWAF_ALERT_COOLDOWN` or `SWWAF_ALERT_MAX_PER_HOUR` off;
-`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, `SWWAF_LOOKUP_TIMEOUT`, the ban
-settings, the state settings, `SWWAF_METRICS_TOP_N` and
-`SWWAF_LOG_REMOTE_BUFFER` cannot be off.
+code on both country lists. AS numbers are `AS` and the number, in either case.
+Percentages are whole numbers from 0 to 100, and an entry of a list of them is
+an AS number or a country, `:` and a percentage; an AS number or a country
+listed twice in one of them stops the start. `off` switches a timeout, a size
+limit, a rate limit, a byte limit, an anomaly threshold, `SWWAF_ALERT_COOLDOWN`
+or `SWWAF_ALERT_MAX_PER_HOUR` off; `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`,
+`SWWAF_LOOKUP_TIMEOUT`, `SWWAF_UNKNOWN_LIMIT_PERCENT`,
+`SWWAF_BLOCKLIST_REFRESH`, the ban settings, the state settings,
+`SWWAF_METRICS_TOP_N`, `SWWAF_LOG_REMOTE_BUFFER`, `SWWAF_ANOMALY_NET_V4_PREFIX`
+and `SWWAF_ANOMALY_NET_V6_PREFIX` cannot be off.
 
 Several limits are fixed rather than settings. At most 20,000 clients are kept,
 with their counters and history, and an IPv6 client is counted by its /64. At
-most 100,000 answers from GeoJS are kept, for 7 days each.
+most 100,000 answers from GeoJS are kept, for 7 days each, and at most 20,000
+anomaly counters.
 
 ### Settings given as files
 
@@ -536,20 +666,29 @@ which every line has.
   app's, as passed on, or those of `smallwebwaf`'s own answer.
 - `request_bytes` and `response_bytes` count body bytes.
 - `action` is `forward` for a request passed to the app, `denied` for one
-  refused because its client is in `SWWAF_DENY_NETS`, `banned` for one refused
-  because a ban covers its client or because it matched a `ban` rule, which bans
-  its client, `country_denied` for one refused for its client's country,
-  `rate_limited` for one that broke a rate limit and banned its client,
-  `rule_blocked` for one a `block` rule refused, `too_large` for a request or
-  response over its size limit, `timed_out` for one that ran out of time,
-  `upstream_error` when the app could not be reached or its answer broke off,
-  and `admin` for one `smallwebwaf` answered at its own endpoint.
+  refused because its client is in `SWWAF_DENY_NETS`, or in a blocklist while
+  `SWWAF_BLOCKLIST_ACTION` is `deny`, `banned` for one refused because a ban
+  covers its client or because it matched a `ban` rule, which bans its client,
+  `country_denied` for one refused for its client's country, `rate_limited` for
+  one that broke a rate limit and banned its client, `rule_blocked` for one a
+  `block` rule refused, `too_large` for a request or response over its size
+  limit, `timed_out` for one that ran out of time, `upstream_error` when the app
+  could not be reached or its answer broke off, and `admin` for one
+  `smallwebwaf` answered at its own endpoint.
 - `would_action` is there in `observe` mode for a request that
-  `SWWAF_DENY_NETS`, a ban, the country lists, a rate limit or a rule would have
-  refused in `enforce` mode, and names the action that refusal would have had:
-  `denied`, `banned`, `country_denied`, `rate_limited` or `rule_blocked`.
-  `action` then names what was done: `forward` for a request passed to the app,
-  and another action, such as `too_large`, for one a size or time limit refused.
+  `SWWAF_DENY_NETS`, a ban, the country lists, a blocklist, a rate limit or a
+  rule would have refused in `enforce` mode, and names the action that refusal
+  would have had: `denied`, `banned`, `country_denied`, `rate_limited` or
+  `rule_blocked`. `action` then names what was done: `forward` for a request
+  passed to the app, and another action, such as `too_large`, for one a size or
+  time limit refused.
+- `limit_percent` is there for a request the rate limits count whose client a
+  biased threshold, or `SWWAF_BLOCKLIST_ACTION` for a blocklist that lists it,
+  gives less than the whole of the rate limits, and gives the percentage it
+  gets, with `limit_percent_setting` naming the setting that gave it, such as
+  `SWWAF_ASN_LIMIT_PERCENT`, or `SWWAF_ASN_LIMIT_PERCENT_URL` for the file it
+  names. `bytes_percent` and `bytes_percent_setting` are the same for the byte
+  limits. Each is left out when the client gets the whole of those limits.
 - `counts` gives the client's requests in the minute, the hour and the day as
   the rate limits count them, this request included: in each window, those in
   the bucket under way and a share of those in the bucket before, so a count can
@@ -557,11 +696,11 @@ which every line has.
   broke it. It is left out for a request the rate limits do not count: the
   health check, one from a client in `SWWAF_ALLOW_NETS` or
   `SWWAF_RATE_LIMIT_EXEMPT_NETS`, one for a path that
-  `SWWAF_RATE_LIMIT_EXEMPT_PATHS` exempts, and one that `SWWAF_DENY_NETS`, a ban
-  or the country lists refuse, or would refuse in `observe` mode. Its
-  `minute_bytes`, `hour_bytes` and `day_bytes` give the client's bytes in each
-  window as the byte limits count them, in the same way: for a request whose
-  bytes they count, with its own, once it has ended; for any other, those
+  `SWWAF_RATE_LIMIT_EXEMPT_PATHS` exempts, and one that `SWWAF_DENY_NETS`, a
+  ban, the country lists or a blocklist refuse, or would refuse in `observe`
+  mode. Its `minute_bytes`, `hour_bytes` and `day_bytes` give the client's bytes
+  in each window as the byte limits count them, in the same way: for a request
+  whose bytes they count, with its own, once it has ended; for any other, those
   counted before it.
 - `rule_ids` is there for a request that matched rules of the rule files, and
   lists their ids in the order they matched, up to the one that refused it.
@@ -572,6 +711,12 @@ which every line has.
   several. `offence` is then `limit`. A request whose bytes broke a byte limit
   is not refused: its `action` is what it would have been otherwise, such as
   `forward`.
+- `reputation` is there for a request whose client a blocklist lists, and gives
+  the URLs of the blocklists that list it, in the order `SWWAF_BLOCKLIST_URLS`
+  names them, whatever `SWWAF_BLOCKLIST_ACTION` says. It is left out for a
+  client the blocklists are not checked for: one in `SWWAF_ALLOW_NETS`, and one
+  `SWWAF_DENY_NETS`, a ban or the country lists refuse first, or would in
+  `observe` mode.
 - `ban_expires` is there for a request that made a ban or was refused under one,
   or in `observe` mode would have been refused under one, and gives when the ban
   ends, in the same form as `time`, or `permanent`.
@@ -638,7 +783,14 @@ it, as below. An alert is for one of these events, and is sent when
   clear sign of attack.
 - `permanent_ban`: a permanent ban it makes, or a ban for a clear sign of attack
   that a request made permanent.
-- `source_failure`: GeoJS failing or refusing `smallwebwaf`.
+- `anomaly`: a count of requests or bytes over an anomaly threshold, raised by
+  each request that ends with the count over it, in `observe` mode as in
+  `enforce` mode. It refuses and bans nothing.
+- `reputation_hit`: a request whose client a blocklist lists, one alert for each
+  blocklist that lists it, whatever `SWWAF_BLOCKLIST_ACTION` says, in `observe`
+  mode as in `enforce` mode.
+- `source_failure`: GeoJS failing or refusing `smallwebwaf`, or a fetch of a
+  list failing (see "Blocklists" below).
 - `file_error`: a rule file edited while it runs that has an error, an edit of a
   state file set aside as `<name>.bad`, a state file it could not write while
   running, or a replacement of the lookup database it could not read, which it
@@ -699,19 +851,33 @@ is sent on one line:
 - `instance` is `SWWAF_INSTANCE_NAME`, and `time` when the alert was raised, in
   UTC.
 - `client` is the address of the client whose request raised the alert, and
-  `netblock` the netblock of the ban; both are empty for `source_failure` and
+  `netblock` the netblock of the ban, or for an `anomaly`, the netblock counted:
+  the client's own, the netblock around it or a named netblock, and none for an
+  AS number or the whole service, or for a `reputation_hit`, the client's own,
+  as `client_group` gives it; both are empty for `source_failure` and
   `file_error`. `asn`, `as_name` and `country` are, for a ban, the client's as
   the ban's notes give them when the alert is raised: empty, as in this alert,
-  when GeoJS had not answered about the client by then.
-- `reason` is a short sentence; for a ban, the ban's `reason` in `bans.json`.
+  when GeoJS had not answered about the client by then; for an `anomaly`, the
+  client's as the lookup gave them by the time its request ended; for a
+  `reputation_hit`, the client's as its request's log line gives them.
+- `reason` is a short sentence; for a ban, the ban's `reason` in `bans.json`;
+  for an `anomaly`, what was counted over which threshold, such as
+  `requests per minute of the netblock 203.0.113.0/24 over the threshold of 1000`;
+  for a `reputation_hit`, `listed by a blocklist`.
 - `detail` is what is particular to the event: for a ban, its `cause`, when it
   ends as `ban_expires`, in the form the request log gives it, and its `notes`,
-  as `bans.json` gives them; for `source_failure`, the `source`, `geojs`, the
-  `error`, and when GeoJS is asked again, `asking_again_in`; for `file_error`,
-  the `file`, which for an edit set aside is the file it was renamed to, and the
-  `error`, which for a file that does not parse names where in it the error is.
+  as `bans.json` gives them; for an `anomaly`, the `scope`, `client`, `net`,
+  `asn`, `total` or `watch`, as the settings name them, the `asn` counted for
+  `asn` and the `name` of the named netblock for `watch`, the `window`, `minute`
+  or `hour`, the `kind`, `requests` or `bytes`, the `count`, which is weighted
+  as the rate limits weigh theirs, and the `threshold`; for a `reputation_hit`,
+  the `source`, the URL of the blocklist; for `source_failure`, the `source`,
+  `geojs` or the URL of the list, the `error`, and for GeoJS, when it is asked
+  again, `asking_again_in`; for `file_error`, the `file`, which for an edit set
+  aside is the file it was renamed to, and the `error`, which for a file that
+  does not parse names where in it the error is.
 - `suppressed_repeats` is how many repeats the cooldown held back before this
-  alert.
+  alert, and for a `summary`, those no other alert gives (see below).
 
 Slack and ntfy are each sent the alert as a message: a title, the instance and
 the event, such as `fsn1app1/gitea: ban`, and a text, the `reason`, then a line
@@ -754,19 +920,24 @@ and Slack this JSON object, shown indented; it is sent on one line:
 }
 ```
 
-An alert for the same event as the last one sent, on the same netblock, or for a
-`file_error` about the same file, or for a `source_failure` about the same
-source, less than `SWWAF_ALERT_COOLDOWN` after it, is a repeat: it is held back
-and counted, and the next alert sent for them gives that count as
-`suppressed_repeats`.
+An alert for the same event as the last one sent, on the same netblock, and for
+a `reputation_hit` about the same blocklist, or for a `file_error` about the
+same file, or for a `source_failure` about the same source, or for an `anomaly`
+in the same scope, with the same netblock, AS number or name, whatever its
+window and kind, less than `SWWAF_ALERT_COOLDOWN` after it, is a repeat: it is
+held back and counted, and the next alert sent for them gives that count as
+`suppressed_repeats`. As each hour of the clock, in UTC, ends, the cooldowns
+that have run out are dropped, and the repeats they held back, which no alert
+sent since has given, go in that hour's summary.
 
-Past `SWWAF_ALERT_MAX_PER_HOUR` alerts in an hour of the clock, in UTC, the
-hour's other alerts are held back and counted by event. Once the hour has ended,
-one alert sums them up: its `event` is `summary`, its `reason` says how many
-were held back, and its `detail` gives the `hour` as when it started, the
-`count`, and the count for each event, as `events`. An alert held back this way
-starts no cooldown, and the repeats held back before it are given by the next
-alert sent for the same event and netblock, file or source.
+Past `SWWAF_ALERT_MAX_PER_HOUR` alerts in an hour, the hour's other alerts are
+held back and counted by event. An alert held back this way starts no cooldown.
+Once the hour has ended, one alert sums up the alerts held back and the repeats
+of the cooldowns dropped: its `event` is `summary`, its `reason` says how many
+of each were held back, its `detail` gives the `hour` as when it started, the
+`count` of alerts held back, and the count for each event, as `events`, and its
+`suppressed_repeats` gives the repeats. An hour with neither ends without a
+summary.
 
 Each destination has a queue of its own, of at most 1000 alerts, from which they
 are sent to it one at a time, the oldest first, so a destination that is slow or
@@ -784,10 +955,12 @@ restart the alerts waiting are sent, and the cooldowns go on.
 
 ## State files
 
-`smallwebwaf` keeps its state in memory and a copy of it in four JSON files in
+`smallwebwaf` keeps its state in memory and a copy of it in five JSON files in
 `SWWAF_STATE_DIR`, `/var/lib/smallwebwaf` by default, as "Persistent state" in
 [`SPEC.md`](SPEC.md) describes. Each has a top-level `version`, 1, and lists its
-entries by client address, but for the alerts waiting, with times in UTC.
+entries by client address, but for the alerts waiting, the anomaly counters,
+which are listed by scope first, and the copies of the lists, listed by URL,
+with times in UTC.
 
 - `bans.json`: every ban with its notes, indented to be read. A permanent ban's
   `expires` is `null`. A ban's `cause` is `limit` for a broken rate limit or
@@ -798,7 +971,10 @@ entries by client address, but for the alerts waiting, with times in UTC.
   `bytes per hour over the limit of 21474836480`, or the rule that matched, such
   as `matched the rule env-file`; for yours, what you wrote. Its `lifted` is
   when you lifted it, and is left out until you do. The `kind` in the notes of a
-  ban for a broken limit is `requests` or `bytes`, what the limit is on.
+  ban for a broken limit is `requests` or `bytes`, what the limit is on. For a
+  limit a biased threshold lowered, the reason and the notes' `limit` give the
+  lowered limit, and the notes' `limit_percent` and `limit_percent_setting` the
+  client's percentage of that kind of limit and the setting that gave it.
 - `clients.json`: each client's two buckets of requests in the minute, the hour
   and the day, its two buckets of bytes in each, `minute_bytes`, `hour_bytes`
   and `day_bytes`, and its history: when it was first and last seen, its AS
@@ -810,18 +986,32 @@ entries by client address, but for the alerts waiting, with times in UTC.
   `grep` shows everything about one.
 - `lookups.json`: GeoJS's answers, one to a line, each with the client's AS
   number, AS name and country, when GeoJS gave it and when it was last used.
+- `reputation.json`: each list fetched from a URL (see "Blocklists" below),
+  indented to be read, under `lists`: its `url`, when it was last `tried`, the
+  fetch failed or not, and its last good copy: when that was `fetched`, and its
+  `lines`, as fetched, comment lines included, each on a line of its own, both
+  left out while no fetch of it has succeeded. As the file is read, the lists
+  the settings no longer name are dropped.
 - `alerts.json`: the state of the alerts (see "Alerts" above), indented to be
-  read: under `cooldowns`, for each event and netblock, or event and `file` or
-  `source`, or event alone, when the last alert was sent, `sent`, and the
-  repeats held back since, `suppressed_repeats`; under `hour`, the hour under
-  way, from its `start`, the alerts `sent` in it and those `held_back` for its
-  summary, by event; and under `waiting`, for each destination you name,
-  `webhook`, `slack` or `ntfy`, the alerts still waiting to be sent to it, the
-  oldest first, each as the webhook is sent it. As an hour ends, the cooldowns
-  that have run out with no repeat held back are dropped. As the file is read,
-  the alerts waiting for a destination you no longer name are dropped. A file
-  whose `waiting` is a list, as it was before alerts went to Slack and ntfy too,
-  stops the start: put the list under `"webhook"`, or remove the file.
+  read: under `cooldowns`, for each event and netblock, with the `source` too
+  for a `reputation_hit`, or event and `file` or `source`, or for an `anomaly`,
+  its `scope` with its `netblock`, `asn` or `name`, or event alone, when the
+  last alert was sent, `sent`, and the repeats held back since,
+  `suppressed_repeats`; under `hour`, the hour under way, from its `start`, the
+  alerts `sent` in it and those `held_back` for its summary, by event; under
+  `waiting`, for each destination you name, `webhook`, `slack` or `ntfy`, the
+  alerts still waiting to be sent to it, the oldest first, each as the webhook
+  is sent it; and under `anomaly_counters`, each anomaly counter: its `scope`,
+  as an `anomaly` alert names it, with the `netblock` of a client, of a netblock
+  around a client or of a named netblock, the `asn` of an AS number and the
+  `name` of a named netblock, and its two buckets of requests in the minute and
+  the hour, `minute` and `hour`, and of bytes, `minute_bytes` and `hour_bytes`,
+  each left out while it is empty. As an hour ends, the cooldowns that have run
+  out are dropped, and the hour's summary gives the repeats they held back. As
+  the file is read, the alerts waiting for a destination you no longer name are
+  dropped. A file whose `waiting` is a list, as it was before alerts went to
+  Slack and ntfy too, stops the start: put the list under `"webhook"`, or remove
+  the file.
 
 `bans.json` is written `SWWAF_STATE_WRITE_DELAY` after a ban is made, lifted
 through `DELETE /_smallwebwaf/bans/<client>`, or made permanent, with every such
@@ -833,12 +1023,14 @@ whole. A write that fails is logged, raised as a `file_error` alert while
 changed since the last write.
 
 At start the files are read back: each client keeps its counts, so a restart
-gives it no fresh allowance, and each ban keeps refusing every client in its
-netblock until it ends, even after `SWWAF_BAN_SCOPE_V4_PREFIX` has changed. A
-netblock whose address has bits past its length, such as `203.0.113.9/24`, is
-read as the netblock it is in, `203.0.113.0/24`. Buckets and answers whose time
-has passed are dropped. A missing file is empty state, as on a first start. A
-file that does not parse, or has another `version`, stops the start with a
+gives it no fresh allowance, each anomaly counter keeps its counts, each ban
+keeps refusing every client in its netblock until it ends, even after
+`SWWAF_BAN_SCOPE_V4_PREFIX` has changed, and the copy of each list stays in use
+until a fetch of it succeeds. A netblock whose address has bits past its length,
+such as `203.0.113.9/24`, is read as the netblock it is in, `203.0.113.0/24`.
+Buckets and answers whose time has passed are dropped, and so is an anomaly
+counter left with no bucket. A missing file is empty state, as on a first start.
+A file that does not parse, or has another `version`, stops the start with a
 message naming the file, and the line and column where Go's JSON decoder gives
 them; so does a state directory `smallwebwaf` cannot write. So does an entry
 without a field it needs, named with the entry's place in the file: a ban's
@@ -846,9 +1038,15 @@ without a field it needs, named with the entry's place in the file: a ban's
 client's `client`, or the `start` of a window in which it has requests or bytes;
 an answer's `client`, `country`, which is `""` for a client GeoJS cannot place,
 or `answered`; a cooldown's `event` or `sent`; an alert waiting's `event` or
-`time`. So does a ban whose `cause` is not `limit`, `attack` or `admin`, and
-alerts waiting for a destination that is not `webhook`, `slack` or `ntfy`. An
-answer's `asn` or `as_name` left out reads as empty.
+`time`; an anomaly counter's `netblock`, unless it counts an AS number or the
+whole service, its `asn`, for an AS number, its `name`, for a named netblock, or
+the `start` of a window in which it has requests or bytes; a list's `url`,
+`fetched` or `lines`, which is `[]` for an empty list. So does a ban whose
+`cause` is not `limit`, `attack` or `admin`, alerts waiting for a destination
+that is not `webhook`, `slack` or `ntfy`, an anomaly counter whose `scope` is
+not `client`, `net`, `asn`, `total` or `watch`, and a copy of a list with a line
+that would make its fetch fail. An answer's `asn` or `as_name` left out reads as
+empty.
 
 While it runs, `smallwebwaf` watches `SWWAF_STATE_DIR` and takes in your edit of
 a state file as soon as you save it: what the file then holds replaces what
@@ -858,13 +1056,14 @@ writes a file it takes in any edit made since, so your edit is not overwritten;
 a change `smallwebwaf` made after you opened the file, such as a new ban, is
 lost when you save over it. An edit that would stop the start, because it does
 not parse, has another `version`, leaves out a field an entry needs, gives a ban
-another `cause` or names another destination, does not stop the running
-`smallwebwaf`: it keeps what it holds, and at the file's next write renames your
-file to `<name>.bad`, such as `bans.json.bad`, writes the file again from
-memory, logs the file and where the error is, and raises a `file_error` alert
-for it. It waits for that write because an editor's file can be read before the
-editor has finished writing it. Mend the `.bad` file and move it back. A file
-you remove is written again at its next write.
+another `cause`, names another destination, gives an anomaly counter another
+`scope` or gives a list's copy a line that would make its fetch fail, does not
+stop the running `smallwebwaf`: it keeps what it holds, and at the file's next
+write renames your file to `<name>.bad`, such as `bans.json.bad`, writes the
+file again from memory, logs the file and where the error is, and raises a
+`file_error` alert for it. It waits for that write because an editor's file can
+be read before the editor has finished writing it. Mend the `.bad` file and move
+it back. A file you remove is written again at its next write.
 
 To ban a netblock, add an entry to `bans.json` with its `netblock`, its `start`
 and its `expires`, `null` for a ban that never ends; its `reason` and its
@@ -1025,14 +1224,20 @@ scraped, and keeps this one as `exported_instance` unless the scrape sets
 - `smallwebwaf_geojs_requests_total`: the requests to GeoJS;
   `smallwebwaf_geojs_failures_total`: those that failed, an answer that leaves
   out an address asked about included; and `smallwebwaf_geojs_unanswered_total`:
-  the requests that needed their client's answer, for a country list or
-  `SWWAF_ADD_LOOKUP_HEADERS`, and went on without it because GeoJS had not given
-  it in time.
+  the requests that needed their client's answer, for a country list,
+  `SWWAF_ADD_LOOKUP_HEADERS` or a biased threshold, and went on without it
+  because GeoJS had not given it in time.
 - While `SWWAF_LOOKUP_SOURCE` is `file`,
   `smallwebwaf_lookup_database_last_read_timestamp_seconds`: when the lookup
   database in use was read; and
   `smallwebwaf_lookup_database_read_failures_total`: the replacements of it that
   could not be read.
+- By `source`, the URL of each list `SWWAF_BLOCKLIST_URLS` or
+  `SWWAF_ASN_LIMIT_PERCENT_URL` names: `smallwebwaf_reputation_hits_total`: the
+  requests whose client the blocklist lists, a series that comes with the first;
+  `smallwebwaf_reputation_failures_total`: the fetches of the list that failed;
+  and `smallwebwaf_reputation_last_fetch_timestamp_seconds`: when the copy of it
+  in use was fetched, `0` while there is none.
 - `smallwebwaf_tracked_clients`: the clients in the table of clients.
 - `smallwebwaf_state_file_writes_total`,
   `smallwebwaf_state_file_write_failures_total`,
@@ -1224,9 +1429,9 @@ goes through the candidates one by one.
   readable JSON files, written regularly and at every stop, so a restart loses
   nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks
   up the change. Nothing is read from disk while serving a request. The files
-  for the bans, the clients, the GeoJS answers and the alerts are built, with an
-  edit taken in while running (see "State files" above); the others come with
-  their features.
+  for the bans, the clients, the GeoJS answers, the copies of the lists and the
+  alerts are built, with an edit taken in while running (see "State files"
+  above); the rest comes with its features.
 - Health checks, the metrics, and listing, adding and lifting bans or asking why
   a given address was refused, all on the one port every request uses: under
   `/_smallwebwaf/` on the app's own address, through traefik like any other
@@ -1246,7 +1451,7 @@ For each request `smallwebwaf`:
   the deny list or currently banned;
 - looks up its AS number and country, and refuses it if that country is denied,
   or is not among the only ones allowed;
-- checks for a cached reputation verdict;
+- checks it against the blocklists, and for a cached reputation verdict;
 - picks the client's limit percentage from those;
 - checks the minute, hour and day request counters against the limits, and bans
   the client if it breaks one;
@@ -1255,8 +1460,8 @@ For each request `smallwebwaf`:
 - forwards it to the app and streams the response back, within the size and time
   limits;
 - counts the bytes and any refusal by the rule files or the Core Rule Set, bans
-  the client if it broke a limit, updates its history, sends any alerts that are
-  due, and writes the log line.
+  the client if it broke a limit, updates its history and the anomaly counters,
+  sends any alerts that are due, and writes the log line.
 
 A minimal deployment is the app's own Dockerfile, built on the `smallwebwaf`
 image, with no setting. That image is built on Ubuntu 26.04 LTS, the newest
@@ -1345,30 +1550,32 @@ the metrics, failure behaviour and the build order.
 `smallwebwaf` looks up the AS number and country of every client through GeoJS,
 a free web service that needs no account and no file, for the request log, the
 client's history, the notes of its bans, their alerts and the metrics, and for
-the country lists when you set them. This means that GeoJS is told the address
-of every new visitor, whether or not a setting uses the answer, unless you set
-`SWWAF_LOOKUP_SOURCE=off`. The only visitors it is not told about are those in
-`SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS`, those whose netblock a ban covers, and
-those on a private, loopback or link-local address. An IPv6 visitor is asked
-about by the first address of its /64. Each answer is kept for seven days, in
-memory and in `lookups.json`, so that it survives a restart, and a visitor whose
-answer is kept is not asked about again.
+the country lists and the anomaly thresholds per AS number when you set them.
+This means that GeoJS is told the address of every new visitor, whether or not a
+setting uses the answer, unless you set `SWWAF_LOOKUP_SOURCE=off`. The only
+visitors it is not told about are those in `SWWAF_ALLOW_NETS` or
+`SWWAF_DENY_NETS`, those whose netblock a ban covers, and those on a private,
+loopback or link-local address. An IPv6 visitor is asked about by the first
+address of its /64. Each answer is kept for seven days, in memory and in
+`lookups.json`, so that it survives a restart, and a visitor whose answer is
+kept is not asked about again.
 
 A request waits for its client's first answer only while a setting acts on it
-before the request goes on: a country list, or `SWWAF_ADD_LOOKUP_HEADERS`. A new
-visitor then waits up to `SWWAF_LOOKUP_TIMEOUT`, a second by default, and
-without an answer counts as coming from an unknown country until the answer
-arrives. Otherwise no request waits: it goes on at once and is logged without
-the answer, which reaches the client's history and the notes of its bans when it
-comes. The addresses waiting are asked about together, up to 200 in one request,
-one request at a time; at most 10,000 visitors wait, and one more is not asked
-about until there is room, counting meanwhile as coming from an unknown country.
-GeoJS publishes no rate limit but may block a caller it thinks asks too much.
-While GeoJS fails, visitors with a kept answer are unaffected and new ones count
-as coming from an unknown country, which `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES`
-refuses. GeoJS is then left alone for a second, twice as long after each further
-failure up to five minutes, and asked again by the next request from a visitor
-without an answer.
+before the request goes on: a country list, `SWWAF_ADD_LOOKUP_HEADERS`, or a
+biased threshold that lowers a limit. A new visitor then waits up to
+`SWWAF_LOOKUP_TIMEOUT`, a second by default, and without an answer counts as
+coming from an unknown country until the answer arrives. Otherwise no request
+waits: it goes on at once and is logged without the answer, which reaches the
+client's history and the notes of its bans when it comes. The addresses waiting
+are asked about together, up to 200 in one request, one request at a time; at
+most 10,000 visitors wait, and one more is not asked about until there is room,
+counting meanwhile as coming from an unknown country. GeoJS publishes no rate
+limit but may block a caller it thinks asks too much. While GeoJS fails,
+visitors with a kept answer are unaffected and new ones count as coming from an
+unknown country, which `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses, and whose
+limits `SWWAF_UNKNOWN_LIMIT_PERCENT` sets. GeoJS is then left alone for a
+second, twice as long after each further failure up to five minutes, and asked
+again by the next request from a visitor without an answer.
 
 To keep your visitors' addresses on your own host, set
 `SWWAF_LOOKUP_SOURCE=off`, or use the database file instead of GeoJS:
@@ -1397,8 +1604,60 @@ service that uses the database through `smallwebwaf` should carry that link.
 Neither source can place a private address, so a client on one, such as a
 visitor on your local network, another container or your monitoring, has no
 country: `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses it unless you list it in
-`SWWAF_ALLOW_NETS`, and `SWWAF_DENIED_COUNTRIES` does not refuse it. Such
-addresses are never sent to GeoJS.
+`SWWAF_ALLOW_NETS`, `SWWAF_DENIED_COUNTRIES` does not refuse it, and
+`SWWAF_UNKNOWN_LIMIT_PERCENT` sets its limits. Such addresses are never sent to
+GeoJS.
+
+## Blocklists
+
+`SWWAF_BLOCKLIST_URLS` names blocklists: text files of addresses and netblocks,
+one to a line, written as the Spamhaus DROP list,
+`https://www.spamhaus.org/drop/drop.txt`, is. Anything after a `;` or a `#` on a
+line is left out, and so is a line left blank. A bare address stands for itself
+alone, as in the settings. An IPv4-mapped address or netblock, such as
+`::ffff:192.0.2.0/120`, is read as the IPv4 one it stands for, here
+`192.0.2.0/24`, since a client's IPv4 address is checked as IPv4; a mapped
+netblock shorter than `/96` stands for none, and is not a netblock. None is
+named by default: a list judges a client by what others saw it do, while the
+defaults judge it by what it does to your service.
+
+`smallwebwaf` fetches each list, and the file `SWWAF_ASN_LIMIT_PERCENT_URL`
+names, `SWWAF_BLOCKLIST_REFRESH` after it last fetched it or tried to, 24 hours
+by default and never less than one, one list after another. Since
+`reputation.json` keeps when each list was last tried, the fetch failed or not,
+even one cut off as `smallwebwaf` stopped, whose request the server may have
+had, at start `smallwebwaf` fetches at once only a list it has never tried, and
+one it last tried that long ago; any other waits its turn, so that restarts do
+not fetch a list more often. A fetch fails when the server answers other than
+`200`, when it does not finish within a minute, when the list is longer than 16
+MiB, or when a line of it is not an address or a netblock, or for
+`SWWAF_ASN_LIMIT_PERCENT_URL`, not an AS number, `:` and a percentage. The copy
+fetched before then stays in use, and the failure is counted, logged and raised
+as a `source_failure` alert; a fetch cut off as `smallwebwaf` stops is not a
+failure. The last good copy of each list is kept whole, comment lines included,
+in `reputation.json` (see "State files" above), so that a restart keeps it in
+use too. Each list is named by its URL, in the request log, the alerts and the
+metrics, so keep a secret out of it.
+
+A client in `SWWAF_ALLOW_NETS` is not checked. Any other is checked by its own
+address after the country lists, and `SWWAF_BLOCKLIST_ACTION` says what is done
+with one a list lists, as "What it does so far" above describes. `deny` suits
+lists of networks that send nothing legitimate, such as DROP; `limit:<percent>`
+suits lists of addresses shared with ordinary visitors, such as those of Tor's
+exits.
+
+The Spamhaus DROP list is The Spamhaus Project's, https://www.spamhaus.org. Its
+terms, on its DROP page, ask that a product using it credit The Spamhaus Project
+and keep the list's date and copyright lines with the data, which the copy in
+`reputation.json` does; a service that uses DROP through `smallwebwaf` uses data
+from The Spamhaus Project, and should say so. They also ask that it be fetched
+automatically no more than once an hour, once a day being more than enough in
+most cases, and Spamhaus may block an address that fetches it more often. Each
+`smallwebwaf` fetches its own copy, on its own schedule, so on a host where
+several apps run it, sharing one address, their fetches can come less than an
+hour apart whatever `SWWAF_BLOCKLIST_REFRESH` is: each fetches a list again that
+long after its own last try, so those first started within the same hour, with
+the same refresh, keep fetching within the same hour.
 
 ## How the code is laid out
 
@@ -1413,13 +1672,15 @@ addresses are never sent to GeoJS.
   standard library's `httputil.ReverseProxy` within the timeouts and size
   limits, and writes the request's log line. Its `check` method is where a
   request is refused before anything reaches the app: for `SWWAF_DENY_NETS`, for
-  a ban, for the country lists, for a rate limit, which bans the client, for a
-  `block` or `ban` rule, the latter banning the client, and for an announced
-  body over the size limit; in `observe` mode, only for the size limit, with
-  what it would have refused for noted in the log line. A request under
-  `/_smallwebwaf/` that `check` lets through is answered by `answerAdmin`
-  instead of reaching the app. Once the answer to a request passed to the app
-  has ended, `countBytes` counts its bytes for the byte limits.
+  a ban, for the country lists, for a blocklist, for a rate limit, which bans
+  the client, for a `block` or `ban` rule, the latter banning the client, and
+  for an announced body over the size limit; in `observe` mode, only for the
+  size limit, with what it would have refused for noted in the log line. A
+  request under `/_smallwebwaf/` that `check` lets through is answered by
+  `answerAdmin` instead of reaching the app. Once the answer to a request passed
+  to the app has ended, `countBytes` counts its bytes for the byte limits, and
+  once any request but the health check has ended, `countAnomalies` counts it
+  for the anomaly thresholds.
 - `internal/metrics`: the metrics, counted as the other parts tell it what
   happened, and served in the Prometheus text format.
 - `internal/bans`: the ban ledger: each netblock's bans with their notes, how
@@ -1432,9 +1693,17 @@ addresses are never sent to GeoJS.
   client's history and to the notes of its bans; or in the lookup database,
   which it reads again when the file is replaced. `internal/lookup/lookuptest`
   writes lookup databases for the tests.
+- `internal/reputation`: fetches the blocklists and the file
+  `SWWAF_ASN_LIMIT_PERCENT_URL` names as they are due, keeps the last good copy
+  of each, and tells which blocklists list an address and what percentage the
+  file gives an AS number.
 - `internal/ratelimit`: the table of clients: counts each client's requests and
   bytes, tells when they take it over a rate limit or a byte limit, and keeps
   each client's history.
+- `internal/anomaly`: the anomaly counters: counts each request and its bytes
+  per client, per netblock around a client, per AS number, for the whole service
+  and per named netblock, in the buckets `internal/ratelimit` counts in, and
+  raises an `anomaly` alert for a count over its threshold.
 - `internal/state`: reads the state files at start, takes in an admin's edit of
   one while running, and writes them when they are due and at the stop.
 - `internal/requestlog`: the lines on stdout: the request log line and the
@@ -1455,8 +1724,9 @@ addresses are never sent to GeoJS.
 
 Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the
 table of clients to 20,000 and the GeoJS answers to 100,000, dropping the least
-recently seen, and the banned netblocks in the order they were last seen, from
-which the ledger picks the ban to drop past `SWWAF_MAX_BANS`, and
+recently seen, the anomaly counters to 20,000, dropping the one counted least
+recently, and the banned netblocks in the order they were last seen, from which
+the ledger picks the ban to drop past `SWWAF_MAX_BANS`, and
 `github.com/prometheus/client_golang` keeps the metrics and serves them, and
 `github.com/fsnotify/fsnotify` tells `smallwebwaf` when a state file or a rule
 file is saved, or the lookup database replaced, and

internal/alerts/alerts.go

@@ -1,5 +1,6 @@
-// Package alerts sends alerts on bans, on a source that fails and on a
-// file with an error to each destination set: to the webhook
+// Package alerts sends alerts on bans, on traffic over an anomaly
+// threshold, on a source that fails and on a file with an error to each
+// destination set: to the webhook
 // SWWAF_ALERT_WEBHOOK_URL names, each as one JSON object, as the "Alert
 // webhook schema" section of SPEC.md describes, to the Slack incoming
 // webhook SWWAF_ALERT_SLACK_WEBHOOK_URL names, as a message, and to the
@@ -40,20 +41,25 @@ const (
 	// EventPermanentBan is a permanent ban smallwebwaf made, or a ban it
 	// made permanent.
 	EventPermanentBan = "permanent_ban"
-	// EventWAFBlock, EventAnomaly and EventReputationHit come with the
-	// Core Rule Set, the anomaly thresholds and the reputation sources;
-	// nothing raises them yet.
-	EventWAFBlock      = "waf_block"
-	EventAnomaly       = "anomaly"
+	// EventAnomaly is a count of requests or bytes over an anomaly
+	// threshold.
+	EventAnomaly = "anomaly"
+	// EventWAFBlock comes with the Core Rule Set; nothing raises it yet.
+	EventWAFBlock = "waf_block"
+	// EventReputationHit is a request whose client a blocklist lists.
 	EventReputationHit = "reputation_hit"
-	// EventSourceFailure is GeoJS failing or refusing smallwebwaf.
+	// EventSourceFailure is GeoJS failing or refusing smallwebwaf, or a
+	// fetch of a list failing.
 	EventSourceFailure = "source_failure"
 	// EventFileError is a rule file or state file edited while smallwebwaf
 	// runs that does not parse, a replacement of the lookup database that
 	// cannot be read, or a state file that cannot be written.
 	EventFileError = "file_error"
-	// EventSummary is the summary of the alerts an hour held back past
-	// SWWAF_ALERT_MAX_PER_HOUR. SWWAF_ALERT_EVENTS does not name it.
+	// EventSummary is the summary sent as an hour ends: of the alerts held
+	// back in it past SWWAF_ALERT_MAX_PER_HOUR, and of the repeats held
+	// back by the cooldowns dropped as it ends, which no alert let through
+	// has given. It is sent with SWWAF_ALERT_MAX_PER_HOUR off too, for
+	// those repeats. SWWAF_ALERT_EVENTS does not name it.
 	EventSummary = "summary"
 )
 
@@ -153,18 +159,22 @@ type Alert struct {
 	ASName   string       `json:"as_name"`
 	Country  string       `json:"country"`
 	// Reason is a short sentence, and Detail what is particular to the
-	// event: for a file_error, its "file", and for a source_failure, its
-	// "source", which the cooldown tells repeats by.
+	// event: for a file_error, its "file", for a source_failure, its
+	// "source", and for an anomaly, its "scope", with the "asn" or the
+	// "name" of some scopes, which the cooldown tells repeats by.
 	Reason string         `json:"reason"`
 	Detail map[string]any `json:"detail"`
 	// SuppressedRepeats is how many repeats of the alert the cooldown
-	// held back since the last one let through.
+	// held back since the last one let through. For a summary, it is how
+	// many the cooldowns dropped as the hour ended had held back that no
+	// alert let through gave.
 	SuppressedRepeats int `json:"suppressed_repeats"`
 }
 
-// Cooldown is, for an event on a netblock, or about a file or a source,
-// when the last alert let through was raised, and how many repeats the
-// cooldown has held back since, as alerts.json holds it.
+// Cooldown is, for an event on a netblock, about a file or a source, or
+// for an anomaly in a scope, when the last alert let through was raised,
+// and how many repeats the cooldown has held back since, as alerts.json
+// holds it.
 //
 //nolint:tagliatelle // the state files use snake_case, as the request log does
 type Cooldown struct {
@@ -172,6 +182,9 @@ type Cooldown struct {
 	Netblock          netip.Prefix `json:"netblock"`
 	File              string       `json:"file,omitempty"`
 	Source            string       `json:"source,omitempty"`
+	Scope             string       `json:"scope,omitempty"`
+	ASN               string       `json:"asn,omitempty"`
+	Name              string       `json:"name,omitempty"`
 	Sent              time.Time    `json:"sent"`
 	SuppressedRepeats int          `json:"suppressed_repeats"`
 }
@@ -214,7 +227,7 @@ type Queue struct {
 
 	mu sync.Mutex
 	// cooldowns are the alerts last let through, by event and netblock,
-	// file or source.
+	// file, source or scope.
 	cooldowns map[cooldownKey]*Cooldown
 	hour      Hour
 
@@ -248,21 +261,28 @@ type destination struct {
 }
 
 // cooldownKey is what makes an alert a repeat of another: the same event
-// on the same netblock, and about the same file or source, as its detail
-// names them. Each is empty for an alert without one.
+// on the same netblock, and about the same file or source, or in the same
+// scope with the same AS number or name, as its detail names them. Each
+// is empty for an alert without one.
 type cooldownKey struct {
 	event    string
 	netblock netip.Prefix
 	file     string
 	source   string
+	scope    string
+	asn      string
+	name     string
 }
 
 // cooldownKeyOf returns what makes another alert a repeat of alert.
 func cooldownKeyOf(alert *Alert) cooldownKey {
 	file, _ := alert.Detail["file"].(string)
 	source, _ := alert.Detail["source"].(string)
+	scope, _ := alert.Detail["scope"].(string)
+	asn, _ := alert.Detail["asn"].(string)
+	name, _ := alert.Detail["name"].(string)
 
-	return cooldownKey{alert.Event, alert.Netblock, file, source}
+	return cooldownKey{alert.Event, alert.Netblock, file, source, scope, asn, name}
 }
 
 // New returns a Queue with no alert yet.
@@ -295,14 +315,14 @@ func New(params Params) *Queue {
 // unless no destination is set or SWWAF_ALERT_EVENTS leaves its event
 // out. It gives alert the instance and the time. An alert that repeats
 // the last one let through less than Cooldown before is held back and
-// counted, and the next one let through gives that count. Past MaxPerHour
-// alerts let through in the hour under way, by the clock, an alert is
-// held back for that hour's summary instead, which is sent once the hour
-// has ended; it starts no cooldown, and the repeats held back before it
-// are given by the next alert let through. Raise never waits: an alert
-// let through joins the queue of each destination, from which Run sends
-// it, and with queueSize alerts waiting for a destination, the oldest is
-// dropped.
+// counted. The next one let through gives that count, unless an hour of
+// the clock ends first after the cooldown has run out: the cooldown is
+// then dropped, and that hour's summary gives the count. Past MaxPerHour
+// alerts let through in the hour under way, an alert is held back for
+// that hour's summary instead, which is sent once the hour has ended; it
+// starts no cooldown. Raise never waits: an alert let through joins the
+// queue of each destination, from which Run sends it, and with queueSize
+// alerts waiting for a destination, the oldest is dropped.
 func (q *Queue) Raise(alert Alert) {
 	if len(q.destinations) == 0 || !slices.Contains(q.params.Events, alert.Event) {
 		return
@@ -425,7 +445,8 @@ func (q *Queue) Suppressed() int64 {
 }
 
 // Snapshot returns the queue's state, as alerts.json holds it, with the
-// cooldowns sorted by netblock, then by event, file and source.
+// cooldowns sorted by netblock, then by event, file, source, scope, AS
+// number and name.
 func (q *Queue) Snapshot() State {
 	q.mu.Lock()
 	defer q.mu.Unlock()
@@ -443,7 +464,9 @@ func (q *Queue) Snapshot() State {
 
 	slices.SortFunc(state.Cooldowns, func(a, b Cooldown) int {
 		return cmp.Or(a.Netblock.Compare(b.Netblock), cmp.Compare(a.Event, b.Event),
-			cmp.Compare(a.File, b.File), cmp.Compare(a.Source, b.Source))
+			cmp.Compare(a.File, b.File), cmp.Compare(a.Source, b.Source),
+			cmp.Compare(a.Scope, b.Scope), cmp.Compare(a.ASN, b.ASN),
+			cmp.Compare(a.Name, b.Name))
 	})
 
 	for _, d := range q.destinations {
@@ -466,7 +489,10 @@ func (q *Queue) Load(state State) {
 
 	for _, cooldown := range state.Cooldowns {
 		cooldown.Netblock = cooldown.Netblock.Masked()
-		key := cooldownKey{cooldown.Event, cooldown.Netblock, cooldown.File, cooldown.Source}
+		key := cooldownKey{
+			cooldown.Event, cooldown.Netblock, cooldown.File, cooldown.Source,
+			cooldown.Scope, cooldown.ASN, cooldown.Name,
+		}
 		q.cooldowns[key] = &cooldown
 	}
 
@@ -538,46 +564,64 @@ func (q *Queue) startCooldown(alert *Alert, now time.Time) {
 
 	q.cooldowns[key] = &Cooldown{
 		Event: alert.Event, Netblock: alert.Netblock, File: key.file, Source: key.source,
-		Sent: now,
+		Scope: key.scope, ASN: key.asn, Name: key.name, Sent: now,
 	}
 }
 
-// endHour ends the hour under way, if now is past it: it queues that
-// hour's summary when alerts were held back in it past MaxPerHour, and
-// forgets the cooldowns that have run out with no repeat held back, which
-// no alert needs any more.
+// endHour ends the hour under way, if now is past it. It drops the
+// cooldowns that have run out, whatever repeats they held back, so that
+// they do not pile up, and queues that hour's summary when alerts were
+// held back in it past MaxPerHour, or when a cooldown dropped had held
+// back repeats, which no alert let through has given: the summary gives
+// them.
 func (q *Queue) endHour(now time.Time) {
 	start := now.Truncate(time.Hour)
 	if !start.After(q.hour.Start) {
 		return
 	}
 
+	repeats := 0
+
+	for key, cooldown := range q.cooldowns {
+		if now.Sub(cooldown.Sent) >= q.params.Cooldown {
+			repeats += cooldown.SuppressedRepeats
+
+			delete(q.cooldowns, key)
+		}
+	}
+
 	heldBack := 0
 	for _, count := range q.hour.HeldBack {
 		heldBack += count
 	}
 
+	var reasons []string
+
 	if heldBack > 0 {
+		reasons = append(reasons, fmt.Sprintf("%d alerts held back in the hour from %s, "+
+			"past the %d an hour SWWAF_ALERT_MAX_PER_HOUR allows", heldBack,
+			q.hour.Start.Format(time.RFC3339), q.params.MaxPerHour))
+	}
+
+	if repeats > 0 {
+		reasons = append(reasons, fmt.Sprintf("%d repeats held back by "+
+			"SWWAF_ALERT_COOLDOWN that no later alert gives", repeats))
+	}
+
+	if len(reasons) > 0 {
 		q.queue(&Alert{
 			Instance: q.params.Instance,
 			Time:     now,
 			Event:    EventSummary,
-			Reason: fmt.Sprintf("%d alerts held back in the hour from %s, past the %d "+
-				"an hour SWWAF_ALERT_MAX_PER_HOUR allows", heldBack,
-				q.hour.Start.Format(time.RFC3339), q.params.MaxPerHour),
+			Reason:   strings.Join(reasons, "; "),
 			Detail: map[string]any{
 				"hour": q.hour.Start, "count": heldBack, "events": q.hour.HeldBack,
 			},
+			SuppressedRepeats: repeats,
 		})
 	}
 
 	q.hour = Hour{Start: start, HeldBack: map[string]int{}}
-
-	for key, cooldown := range q.cooldowns {
-		if now.Sub(cooldown.Sent) >= q.params.Cooldown && cooldown.SuppressedRepeats == 0 {
-			delete(q.cooldowns, key)
-		}
-	}
 }
 
 // queue adds alert to the alerts waiting for each destination.

internal/alerts/alerts_test.go

@@ -381,7 +381,7 @@ func TestAlertsPastTheHourlyLimitAreRolledIntoOneSummary(t *testing.T) {
 	})
 }
 
-func TestRepeatsBeforeAnAlertPastTheHourlyLimitAreGivenByTheNextSent(t *testing.T) {
+func TestRepeatsBeforeAnAlertPastTheHourlyLimitAreGivenByTheSummary(t *testing.T) {
 	t.Parallel()
 
 	synctest.Test(t, func(t *testing.T) {
@@ -401,8 +401,8 @@ func TestRepeatsBeforeAnAlertPastTheHourlyLimitAreGivenByTheNextSent(t *testing.
 		time.Sleep(cooldown)
 		raise()
 
-		// The next hour's first alert gives the two repeats, and the summary
-		// the alert past the limit.
+		// The summary gives the alert past the limit and the two repeats,
+		// and the next hour's first alert none.
 		time.Sleep(time.Hour - cooldown)
 		synctest.Wait()
 		raise()
@@ -413,11 +413,14 @@ func TestRepeatsBeforeAnAlertPastTheHourlyLimitAreGivenByTheNextSent(t *testing.
 		got := webhook.received()
 		if len(got) == 3 {
 			detail, _ := got[1].alert["detail"].(map[string]any)
-			repeats := got[2].alert["suppressed_repeats"]
+			summaryRepeats := got[1].alert["suppressed_repeats"]
+			lastRepeats := got[2].alert["suppressed_repeats"]
 
-			if detail["count"] != float64(1) || repeats != float64(2) {
-				t.Errorf("the summary counts %v alerts, and the last alert gives %v "+
-					"repeats, want 1 and 2", detail["count"], repeats)
+			if detail["count"] != float64(1) || summaryRepeats != float64(2) ||
+				lastRepeats != float64(0) {
+				t.Errorf("the summary counts %v alerts and %v repeats, and the last "+
+					"alert gives %v repeats, want 1, 2 and 0", detail["count"],
+					summaryRepeats, lastRepeats)
 			}
 		}
 
@@ -425,6 +428,70 @@ func TestRepeatsBeforeAnAlertPastTheHourlyLimitAreGivenByTheNextSent(t *testing.
 	})
 }
 
+func TestCooldownsThatHaveRunOutAreDroppedAndTheirRepeatsSummedUp(t *testing.T) {
+	t.Parallel()
+
+	for name, maxPerHour := range map[string]int{"limit off": 0, "limit set": 60} {
+		t.Run(name, func(t *testing.T) {
+			t.Parallel()
+
+			synctest.Test(t, func(t *testing.T) {
+				params := newParams()
+				params.MaxPerHour = maxPerHour
+				webhook, q := start(t, params)
+
+				// For four hours, a netblock of its own each minute is over an
+				// anomaly threshold twice: an alert, and a repeat the cooldown
+				// holds back.
+				netblocks := 0
+
+				for range 4 {
+					for range 60 {
+						anomaly := alerts.Alert{
+							Event: alerts.EventAnomaly, Netblock: netblock(netblocks),
+							Detail: map[string]any{"scope": "net"},
+						}
+						q.Raise(anomaly)
+						q.Raise(anomaly)
+
+						netblocks++
+
+						time.Sleep(time.Minute)
+					}
+
+					// As the hour ends, only the cooldowns started less than the
+					// cooldown before are kept, in memory and for alerts.json.
+					synctest.Wait()
+
+					kept := len(q.Snapshot().Cooldowns)
+					if kept > int(cooldown/time.Minute) {
+						t.Errorf("after %d netblocks, %d cooldowns are kept, want at most %d",
+							netblocks, kept, int(cooldown/time.Minute))
+					}
+				}
+
+				// An hour on, every cooldown has been dropped, and the summaries
+				// have given every repeat.
+				time.Sleep(time.Hour)
+				synctest.Wait()
+
+				repeats := 0.0
+
+				for _, request := range webhook.received() {
+					count, _ := request.alert["suppressed_repeats"].(float64)
+					repeats += count
+				}
+
+				kept := len(q.Snapshot().Cooldowns)
+				if kept != 0 || repeats != float64(netblocks) {
+					t.Errorf("%d cooldowns are kept and the webhook was given %v repeats, "+
+						"want 0 and %d", kept, repeats, netblocks)
+				}
+			})
+		})
+	}
+}
+
 func TestFailedRequestIsSentAgainWithBackoff(t *testing.T) {
 	t.Parallel()
 
@@ -635,7 +702,8 @@ func TestStateLoadedIntoANewQueueCarriesOn(t *testing.T) {
 		after.Load(roundTrip(t, before.Snapshot()))
 
 		// The new queue sends the alert waiting, holds back the repeat as
-		// the cooldown still runs, and sends the summary of the hour.
+		// the cooldown still runs, and sends the summary of the hour, which
+		// gives both repeats, as the cooldown has run out.
 		after.Raise(alerts.Alert{Event: alerts.EventBan, Netblock: netblock(1)})
 		synctest.Wait()
 		wantEvents(t, webhook, alerts.EventBan)
@@ -644,18 +712,12 @@ func TestStateLoadedIntoANewQueueCarriesOn(t *testing.T) {
 		synctest.Wait()
 		wantEvents(t, webhook, alerts.EventBan, alerts.EventSummary)
 
-		detail, _ := webhook.received()[1].alert["detail"].(map[string]any)
-		if detail["count"] != float64(1) {
-			t.Errorf("the summary counts %v alerts, want 1", detail["count"])
-		}
+		summary := webhook.received()[1].alert
+		detail, _ := summary["detail"].(map[string]any)
 
-		// The cooldown has run out, and the next one gives both repeats.
-		after.Raise(alerts.Alert{Event: alerts.EventBan, Netblock: netblock(1)})
-		synctest.Wait()
-
-		got := webhook.received()
-		if repeats := got[len(got)-1].alert["suppressed_repeats"]; repeats != float64(2) {
-			t.Errorf("the last alert gives %v repeats, want 2", repeats)
+		if detail["count"] != float64(1) || summary["suppressed_repeats"] != float64(2) {
+			t.Errorf("the summary counts %v alerts and %v repeats, want 1 and 2",
+				detail["count"], summary["suppressed_repeats"])
 		}
 	})
 }
@@ -701,8 +763,8 @@ func TestSlackAndNtfyAreSentTheSummaryAndTheRepeatsHeldBack(t *testing.T) {
 		ban := alerts.Alert{Event: alerts.EventBan, Netblock: netblock(1), Reason: "a ban"}
 
 		// The hour's one alert, a repeat of it the cooldown holds back, and
-		// an alert past the limit; once the hour has ended, its summary, and
-		// the next alert, which gives the repeat.
+		// an alert past the limit; once the hour has ended, its summary,
+		// which gives the repeat, and the next alert, which gives none.
 		q.Raise(ban)
 		q.Raise(ban)
 		q.Raise(alerts.Alert{Event: alerts.EventFileError, Reason: "a file error"})
@@ -720,14 +782,14 @@ func TestSlackAndNtfyAreSentTheSummaryAndTheRepeatsHeldBack(t *testing.T) {
 		}
 
 		const summary = "1 alerts held back in the hour from 2000-01-01T00:00:00Z, " +
-			"past the 1 an hour SWWAF_ALERT_MAX_PER_HOUR allows"
+			"past the 1 an hour SWWAF_ALERT_MAX_PER_HOUR allows; 1 repeats held back " +
+			"by SWWAF_ALERT_COOLDOWN that no later alert gives\nsuppressed repeats: 1"
 
 		wantSlackMessage(t, slack[1], "*"+instance+": summary*\n"+summary)
 		wantNtfyMessage(t, ntfy[1], instance+": summary", "default bar_chart", summary)
-		wantSlackMessage(t, slack[2],
-			"*"+instance+": ban*\na ban\nnetblock: 203.0.113.1/32\nsuppressed repeats: 1")
+		wantSlackMessage(t, slack[2], "*"+instance+": ban*\na ban\nnetblock: 203.0.113.1/32")
 		wantNtfyMessage(t, ntfy[2], instance+": ban", "default no_entry",
-			"a ban\nnetblock: 203.0.113.1/32\nsuppressed repeats: 1")
+			"a ban\nnetblock: 203.0.113.1/32")
 	})
 }
 

internal/anomaly/anomaly.go

@@ -0,0 +1,406 @@
+// Package anomaly counts requests and bytes over a minute and an hour, per
+// client, per surrounding netblock, per AS number, for the whole service
+// and per named netblock, and raises an anomaly alert for a count over its
+// threshold, as "Anomaly thresholds" under "Configuration surface" in
+// SPEC.md describes. It refuses and bans nothing. At most 20,000 counters
+// are kept, in memory, and written to alerts.json and read from it by the
+// state package.
+package anomaly
+
+import (
+	"cmp"
+	"fmt"
+	"net/netip"
+	"slices"
+	"sync"
+	"time"
+
+	"github.com/hashicorp/golang-lru/v2/simplelru"
+	"sneak.berlin/go/smallwebwaf/internal/alerts"
+	"sneak.berlin/go/smallwebwaf/internal/ratelimit"
+)
+
+// maxCounters is how many counters are kept. Past it, the counter counted
+// least recently is dropped, and starts afresh if it is counted again.
+const maxCounters = 20000
+
+// The scopes, what a counter counts, as the settings, alerts.json and the
+// alerts name them.
+const (
+	// ScopeClient is one client: an IPv4 address, or an IPv6 /64.
+	ScopeClient = "client"
+	// ScopeNet is the netblock around a client, SWWAF_ANOMALY_NET_V4_PREFIX
+	// or SWWAF_ANOMALY_NET_V6_PREFIX long.
+	ScopeNet = "net"
+	// ScopeASN is an AS number.
+	ScopeASN = "asn"
+	// ScopeTotal is the whole service.
+	ScopeTotal = "total"
+	// ScopeWatch is a named netblock of SWWAF_WATCH_NETS.
+	ScopeWatch = "watch"
+)
+
+// Scopes returns every scope.
+func Scopes() []string {
+	return []string{ScopeClient, ScopeNet, ScopeASN, ScopeTotal, ScopeWatch}
+}
+
+// The windows a counter counts in, as the alerts name them.
+const (
+	minute = "minute"
+	hour   = "hour"
+)
+
+// Thresholds are the most requests and the most bytes a scope may have
+// counted in a minute and in an hour before an alert is raised. Zero is
+// off.
+type Thresholds struct {
+	RequestsPerMinute int64
+	RequestsPerHour   int64
+	BytesPerMinute    int64
+	BytesPerHour      int64
+}
+
+// NamedNetblock is a netblock SWWAF_WATCH_NETS names.
+type NamedNetblock struct {
+	Name     string
+	Netblock netip.Prefix
+}
+
+// Params are what New needs.
+type Params struct {
+	// The thresholds of each scope: SWWAF_ANOMALY_CLIENT_*,
+	// SWWAF_ANOMALY_NET_*, SWWAF_ANOMALY_ASN_*, SWWAF_ANOMALY_TOTAL_* and
+	// SWWAF_WATCH_*.
+	Client, Net, ASN, Total, Watch Thresholds
+	// NetV4Prefix and NetV6Prefix are the lengths of the netblock around a
+	// client (SWWAF_ANOMALY_NET_V4_PREFIX and SWWAF_ANOMALY_NET_V6_PREFIX).
+	NetV4Prefix, NetV6Prefix int
+	// NamedNetblocks are SWWAF_WATCH_NETS.
+	NamedNetblocks []NamedNetblock
+	// Alerts receive the anomaly alerts.
+	Alerts *alerts.Queue
+}
+
+// Counter is one scope's counts, as alerts.json holds them: the scope,
+// with the netblock, the AS number or the name that tells it from the
+// others in that scope, and its two buckets of requests and of bytes in
+// the minute and in the hour. A bucket whose threshold is off counts
+// nothing, and is left out.
+//
+//nolint:tagliatelle // the state files use snake_case, as the request log does
+type Counter struct {
+	Scope       string            `json:"scope"`
+	Netblock    netip.Prefix      `json:"netblock,omitzero"`
+	ASN         string            `json:"asn,omitempty"`
+	Name        string            `json:"name,omitempty"`
+	Minute      ratelimit.Buckets `json:"minute,omitzero"`
+	Hour        ratelimit.Buckets `json:"hour,omitzero"`
+	MinuteBytes ratelimit.Buckets `json:"minute_bytes,omitzero"`
+	HourBytes   ratelimit.Buckets `json:"hour_bytes,omitzero"`
+}
+
+// Request is a request that has ended, as the counters count it.
+type Request struct {
+	// Client is the client's address, and ClientGroup the client it is
+	// counted as: its IPv4 address, or its IPv6 /64.
+	Client      netip.Addr
+	ClientGroup netip.Prefix
+	// ASN, ASName and Country are the client's as looked up, each "" when
+	// unknown.
+	ASN, ASName, Country string
+	// Bytes are the request's bytes, as SWWAF_BYTES_COUNT counts them.
+	Bytes int64
+}
+
+// Counters counts each request in the scopes it is in. It is safe for
+// concurrent use.
+type Counters struct {
+	params Params
+
+	mu       sync.Mutex
+	counters *simplelru.LRU[key, *Counter]
+}
+
+// key is what tells a counter from the others: its scope, with its
+// netblock, AS number or name.
+type key struct {
+	scope    string
+	netblock netip.Prefix
+	asn      string
+	name     string
+}
+
+// New returns Counters for params, with nothing counted yet.
+func New(params Params) *Counters {
+	counters, err := simplelru.NewLRU[key, *Counter](maxCounters, nil)
+	if err != nil {
+		panic(err) // NewLRU fails only for a size below one
+	}
+
+	return &Counters{params: params, counters: counters}
+}
+
+// Count counts r, a request that has ended, at now, in each scope it is
+// in whose thresholds are not all off: its client, the netblock around
+// it, its AS number once known, the whole service, and each named
+// netblock it is in. Only the counts whose threshold is set are counted.
+// For each scope whose count is over a threshold, it raises an anomaly
+// alert, for the first such count in the order requests and bytes in the
+// minute, then in the hour; the alert queue's cooldown holds back the
+// repeats. Nothing is refused or banned.
+func (c *Counters) Count(now time.Time, r Request) {
+	var raised []alerts.Alert
+
+	c.mu.Lock()
+
+	for _, scope := range c.scopesOf(r) {
+		counter, found := c.counters.Get(scope.key)
+		if !found {
+			counter = scope.key.counter()
+			c.counters.Add(scope.key, counter)
+		}
+
+		over, passed := counter.add(now, r.Bytes, scope.thresholds)
+		if passed {
+			raised = append(raised, alertFor(r, scope.key, over))
+		}
+	}
+
+	c.mu.Unlock()
+
+	for _, alert := range raised {
+		c.params.Alerts.Raise(alert)
+	}
+}
+
+// Snapshot returns every counter, sorted by scope, then by netblock, AS
+// number and name, as alerts.json lists them.
+func (c *Counters) Snapshot() []Counter {
+	c.mu.Lock()
+
+	counters := make([]Counter, 0, c.counters.Len())
+	for _, counter := range c.counters.Values() {
+		counters = append(counters, *counter)
+	}
+
+	c.mu.Unlock()
+
+	slices.SortFunc(counters, func(a, b Counter) int {
+		return cmp.Or(cmp.Compare(a.Scope, b.Scope), a.Netblock.Compare(b.Netblock),
+			cmp.Compare(a.ASN, b.ASN), cmp.Compare(a.Name, b.Name))
+	})
+
+	return counters
+}
+
+// Load puts counters, read from alerts.json, in place of those held, in
+// the order they were last counted, as the starts of their buckets tell,
+// so that the one counted least recently is dropped first. Each netblock
+// is masked to its length, so that 203.0.113.9/24 is 203.0.113.0/24.
+// Buckets whose time has passed at now are emptied, and a counter left
+// with every bucket empty is dropped.
+func (c *Counters) Load(counters []Counter, now time.Time) {
+	counters = slices.Clone(counters)
+	slices.SortStableFunc(counters, func(a, b Counter) int {
+		return a.lastStart().Compare(b.lastStart())
+	})
+
+	c.mu.Lock()
+	defer c.mu.Unlock()
+
+	c.counters.Purge()
+
+	for _, counter := range counters {
+		counter.Netblock = counter.Netblock.Masked()
+		empty := true
+
+		for _, count := range counter.counts() {
+			if count.buckets.Passed(now, count.length) {
+				*count.buckets = ratelimit.Buckets{}
+			}
+
+			empty = empty && *count.buckets == ratelimit.Buckets{}
+		}
+
+		if !empty {
+			c.counters.Add(counter.key(), &counter)
+		}
+	}
+}
+
+// scope is a scope a request is counted in, and its thresholds.
+type scope struct {
+	key        key
+	thresholds Thresholds
+}
+
+// scopesOf returns the scopes r is in whose thresholds are not all off.
+func (c *Counters) scopesOf(r Request) []scope {
+	p := c.params
+	client := r.Client.Unmap()
+
+	all := []scope{
+		{key{scope: ScopeClient, netblock: r.ClientGroup}, p.Client},
+		{key{scope: ScopeNet, netblock: c.netAround(client)}, p.Net},
+		{key{scope: ScopeTotal}, p.Total},
+	}
+
+	if r.ASN != "" {
+		all = append(all, scope{key{scope: ScopeASN, asn: r.ASN}, p.ASN})
+	}
+
+	for _, named := range p.NamedNetblocks {
+		if named.Netblock.Contains(client) {
+			all = append(all, scope{
+				key{scope: ScopeWatch, netblock: named.Netblock, name: named.Name}, p.Watch,
+			})
+		}
+	}
+
+	return slices.DeleteFunc(all, func(s scope) bool {
+		return s.thresholds == Thresholds{}
+	})
+}
+
+// netAround returns the netblock around client that ScopeNet counts it
+// in: NetV4Prefix or NetV6Prefix long.
+func (c *Counters) netAround(client netip.Addr) netip.Prefix {
+	length := c.params.NetV6Prefix
+	if client.Is4() {
+		length = c.params.NetV4Prefix
+	}
+
+	return netip.PrefixFrom(client, length).Masked()
+}
+
+// overThreshold is a count over its threshold: what it counts, requests or
+// bytes, its window, the count and the threshold.
+type overThreshold struct {
+	kind, window string
+	count        float64
+	threshold    int64
+}
+
+// add counts a request of bytes at now in each of c's counts whose
+// threshold, in thresholds, is set, and returns the first count over its
+// threshold, and whether there is one.
+func (c *Counter) add(
+	now time.Time, bytes int64, thresholds Thresholds,
+) (overThreshold, bool) {
+	// In the order of counts.
+	inOrder := [4]int64{
+		thresholds.RequestsPerMinute, thresholds.BytesPerMinute,
+		thresholds.RequestsPerHour, thresholds.BytesPerHour,
+	}
+
+	var (
+		first  overThreshold
+		passed bool
+	)
+
+	for i, count := range c.counts() {
+		threshold := inOrder[i]
+		if threshold == 0 {
+			continue
+		}
+
+		n := int64(1)
+		if count.kind == ratelimit.KindBytes {
+			n = bytes
+		}
+
+		counted := count.buckets.Add(now, count.length, n)
+		if !passed && counted > float64(threshold) {
+			first = overThreshold{count.kind, count.window, counted, threshold}
+			passed = true
+		}
+	}
+
+	return first, passed
+}
+
+// bucketCount is one of a counter's four counts: requests or bytes, in a
+// window of length, and the buckets they are counted in.
+type bucketCount struct {
+	kind, window string
+	length       time.Duration
+	buckets      *ratelimit.Buckets
+}
+
+// counts returns c's counts: requests and bytes in the minute, then in
+// the hour.
+func (c *Counter) counts() [4]bucketCount {
+	return [4]bucketCount{
+		{ratelimit.KindRequests, minute, time.Minute, &c.Minute},
+		{ratelimit.KindBytes, minute, time.Minute, &c.MinuteBytes},
+		{ratelimit.KindRequests, hour, time.Hour, &c.Hour},
+		{ratelimit.KindBytes, hour, time.Hour, &c.HourBytes},
+	}
+}
+
+// lastStart returns the start of c's latest bucket, which tells, to the
+// minute or to the hour, when c was last counted.
+func (c *Counter) lastStart() time.Time {
+	var latest time.Time
+
+	for _, count := range c.counts() {
+		if count.buckets.Start.After(latest) {
+			latest = count.buckets.Start
+		}
+	}
+
+	return latest
+}
+
+// key returns what tells c from the other counters.
+func (c *Counter) key() key {
+	return key{scope: c.Scope, netblock: c.Netblock, asn: c.ASN, name: c.Name}
+}
+
+// counter returns a counter for k, with nothing counted yet.
+func (k key) counter() *Counter {
+	return &Counter{Scope: k.scope, Netblock: k.netblock, ASN: k.asn, Name: k.name}
+}
+
+// alertFor returns the anomaly alert for o, a count over its threshold in
+// the scope k, which r took over it. It gives r's client, with its AS
+// number, AS name and country, and the netblock counted, of a client, the
+// netblock around it or a named netblock. Its detail gives the scope, the
+// AS number or the name of a scope that has one, the window, what is
+// counted, the count and the threshold.
+func alertFor(r Request, k key, o overThreshold) alerts.Alert {
+	detail := map[string]any{
+		"scope": k.scope, "window": o.window, "kind": o.kind, "count": o.count,
+		"threshold": o.threshold,
+	}
+
+	var counted string
+
+	switch k.scope {
+	case ScopeClient:
+		counted = "the client " + k.netblock.String()
+	case ScopeNet:
+		counted = "the netblock " + k.netblock.String()
+	case ScopeASN:
+		counted = k.asn
+		detail["asn"] = k.asn
+	case ScopeTotal:
+		counted = "the whole service"
+	default: // watch
+		counted = "the named netblock " + k.name + ", " + k.netblock.String()
+		detail["name"] = k.name
+	}
+
+	return alerts.Alert{
+		Event:    alerts.EventAnomaly,
+		Client:   r.Client,
+		Netblock: k.netblock,
+		ASN:      r.ASN,
+		ASName:   r.ASName,
+		Country:  r.Country,
+		Reason: fmt.Sprintf("%s per %s of %s over the threshold of %d", o.kind, o.window,
+			counted, o.threshold),
+		Detail: detail,
+	}
+}

internal/anomaly/anomaly_test.go

@@ -0,0 +1,238 @@
+package anomaly_test
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/netip"
+	"net/url"
+	"reflect"
+	"slices"
+	"testing"
+	"time"
+
+	"sneak.berlin/go/smallwebwaf/internal/alerts"
+	"sneak.berlin/go/smallwebwaf/internal/anomaly"
+	"sneak.berlin/go/smallwebwaf/internal/ratelimit"
+)
+
+// maxCounters is how many counters are kept.
+const maxCounters = 20000
+
+func TestEachScopeHasACooldownOfItsOwn(t *testing.T) {
+	t.Parallel()
+
+	queue := newQueue()
+	office := netip.MustParsePrefix("203.0.113.0/24")
+	overAtTheSecond := anomaly.Thresholds{RequestsPerMinute: 1}
+	counters := anomaly.New(anomaly.Params{
+		Client: overAtTheSecond, Net: overAtTheSecond, ASN: overAtTheSecond,
+		Total: overAtTheSecond, Watch: overAtTheSecond,
+		// The netblock around a client is the client's own, and two names
+		// name one netblock.
+		NetV4Prefix: 32,
+		NamedNetblocks: []anomaly.NamedNetblock{
+			{Name: "office", Netblock: office}, {Name: "hq", Netblock: office},
+		},
+		Alerts: queue,
+	})
+
+	// The first client's second request is over the threshold in the six
+	// scopes it is in. The other client's two are both over it in the whole
+	// service and in each named netblock, three repeats each, and its
+	// second is over it in the scopes of its own, its client, its netblock
+	// and its AS number, which are no repeats.
+	for _, r := range []anomaly.Request{
+		{Client: netip.MustParseAddr("203.0.113.9"), ASN: "AS64496"},
+		{Client: netip.MustParseAddr("203.0.113.10"), ASN: "AS64511"},
+	} {
+		r.ClientGroup = netip.PrefixFrom(r.Client, 32)
+
+		for range 2 {
+			counters.Count(midnight(), r)
+		}
+	}
+
+	waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
+	if len(waiting) != 9 || queue.Suppressed() != 6 {
+		t.Fatalf("%d alerts wait and %d are held back, want 9 and 6: %+v",
+			len(waiting), queue.Suppressed(), waiting)
+	}
+
+	// alerts.json keeps each scope's cooldown: each alert raised again
+	// after a restart is a repeat.
+	data, err := json.Marshal(queue.Snapshot())
+	if err != nil {
+		t.Fatalf("encode: %v", err)
+	}
+
+	var read alerts.State
+
+	err = json.Unmarshal(data, &read)
+	if err != nil {
+		t.Fatalf("decode: %v", err)
+	}
+
+	after := newQueue()
+	after.Load(read)
+
+	for _, alert := range read.Waiting[alerts.DestinationWebhook] {
+		after.Raise(alert)
+	}
+
+	if after.Suppressed() != 9 {
+		t.Errorf("after loading, %d alerts are held back, want 9", after.Suppressed())
+	}
+}
+
+func TestKeepsAtMost20000CountersDroppingTheLeastRecentlyCounted(t *testing.T) {
+	t.Parallel()
+
+	counters := newCounters(anomaly.Params{
+		Client: anomaly.Thresholds{RequestsPerMinute: 1000},
+	})
+
+	for i := range maxCounters {
+		counters.Count(midnight(), request(i))
+	}
+
+	// Counted again, the first client is the most recently counted, and
+	// the second is dropped for a new one.
+	counters.Count(midnight(), request(0))
+	counters.Count(midnight(), request(maxCounters))
+
+	got := counters.Snapshot()
+	if len(got) != maxCounters || !holds(got, 0) || holds(got, 1) ||
+		!holds(got, maxCounters) {
+		t.Errorf("%d counters, holding the first client %v, the second %v and the "+
+			"new one %v, want %d, the first and the new one", len(got), holds(got, 0),
+			holds(got, 1), holds(got, maxCounters), maxCounters)
+	}
+}
+
+func TestLoadEmptiesBucketsWhoseTimeHasPassedAndDropsEmptyCounters(t *testing.T) {
+	t.Parallel()
+
+	counters := newCounters(anomaly.Params{
+		Net:         anomaly.Thresholds{RequestsPerMinute: 1000, RequestsPerHour: 1000},
+		Total:       anomaly.Thresholds{RequestsPerMinute: 1000},
+		NetV4Prefix: 24,
+	})
+	halfAnHourOn := midnight().Add(30 * time.Minute)
+
+	// Half an hour on, the hour's buckets count still, and the minute's
+	// do not.
+	counters.Load([]anomaly.Counter{
+		{
+			Scope:    anomaly.ScopeNet,
+			Netblock: netip.MustParsePrefix("203.0.113.9/24"),
+			Minute:   ratelimit.Buckets{Start: midnight(), Current: 5},
+			Hour:     ratelimit.Buckets{Start: midnight(), Current: 7},
+		},
+		{
+			Scope:  anomaly.ScopeTotal,
+			Minute: ratelimit.Buckets{Start: midnight(), Current: 1},
+		},
+	}, halfAnHourOn)
+
+	// The whole service's counter, left empty, is dropped, and the
+	// netblock read is masked to its length.
+	netblock := anomaly.Counter{
+		Scope:    anomaly.ScopeNet,
+		Netblock: netip.MustParsePrefix("203.0.113.0/24"),
+		Hour:     ratelimit.Buckets{Start: midnight(), Current: 7},
+	}
+	if got, want := counters.Snapshot(), []anomaly.Counter{netblock}; !reflect.DeepEqual(
+		got, want) {
+		t.Errorf("counters read\n%+v\nwant\n%+v", got, want)
+	}
+
+	// A request from the netblock is counted with the requests read.
+	counters.Count(halfAnHourOn, anomaly.Request{
+		Client:      netip.MustParseAddr("203.0.113.9"),
+		ClientGroup: netip.MustParsePrefix("203.0.113.9/32"),
+	})
+
+	netblock.Minute = ratelimit.Buckets{Start: halfAnHourOn, Current: 1}
+	netblock.Hour.Current = 8
+	want := []anomaly.Counter{netblock, {
+		Scope:  anomaly.ScopeTotal,
+		Minute: ratelimit.Buckets{Start: halfAnHourOn, Current: 1},
+	}}
+
+	if got := counters.Snapshot(); !reflect.DeepEqual(got, want) {
+		t.Errorf("counters after a request\n%+v\nwant\n%+v", got, want)
+	}
+}
+
+func TestLoadDropsTheLeastRecentlyCountedFirst(t *testing.T) {
+	t.Parallel()
+
+	counters := newCounters(anomaly.Params{
+		Client: anomaly.Thresholds{RequestsPerMinute: 1000},
+	})
+	now := midnight().Add(time.Minute)
+
+	// The second half of the file was counted in the minute before the
+	// first half.
+	read := make([]anomaly.Counter, 0, maxCounters)
+	for i := range maxCounters {
+		start := now
+		if i >= maxCounters/2 {
+			start = midnight()
+		}
+
+		read = append(read, anomaly.Counter{
+			Scope: anomaly.ScopeClient, Netblock: request(i).ClientGroup,
+			Minute: ratelimit.Buckets{Start: start, Current: 1},
+		})
+	}
+
+	counters.Load(read, now)
+	counters.Count(now, request(maxCounters))
+
+	got := counters.Snapshot()
+	if !holds(got, 0) || holds(got, maxCounters/2) {
+		t.Errorf("holding the first client of the file %v, and the first counted in "+
+			"the minute before %v, want only the first", holds(got, 0),
+			holds(got, maxCounters/2))
+	}
+}
+
+// midnight is the time of the tests' requests.
+func midnight() time.Time {
+	return time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)
+}
+
+// newCounters returns Counters for params, whose alerts go nowhere.
+func newCounters(params anomaly.Params) *anomaly.Counters {
+	params.Alerts = alerts.New(alerts.Params{})
+
+	return anomaly.New(params)
+}
+
+// newQueue returns a queue of alerts to a webhook, with the default
+// cooldown, which keeps them waiting, since it is never run.
+func newQueue() *alerts.Queue {
+	return alerts.New(alerts.Params{
+		WebhookURL: &url.URL{Scheme: "https", Host: "alerts.example"},
+		Events:     alerts.Events(),
+		Cooldown:   15 * time.Minute,
+		MaxPerHour: 60,
+		Now:        midnight,
+	})
+}
+
+// request returns a request from client number i, an address in
+// 10.0.0.0/8.
+func request(i int) anomaly.Request {
+	client := netip.MustParseAddr(fmt.Sprintf("10.%d.%d.%d", i>>16, i>>8&255, i&255))
+
+	return anomaly.Request{Client: client, ClientGroup: netip.PrefixFrom(client, 32)}
+}
+
+// holds reports whether counters hold the counter of client number i.
+func holds(counters []anomaly.Counter, i int) bool {
+	return slices.ContainsFunc(counters, func(counter anomaly.Counter) bool {
+		return counter.Netblock == request(i).ClientGroup
+	})
+}

internal/bans/bans.go

@@ -108,6 +108,12 @@ type Notes struct {
 	Limit  int64   `json:"limit,omitempty"`
 	Window string  `json:"window,omitempty"`
 	Count  float64 `json:"count,omitempty"`
+	// LimitPercent and LimitPercentSetting are, for a ban for a limit a
+	// biased threshold lowered, the client's percentage of that kind of
+	// limit, of which Limit is the result, and the setting that gave it.
+	// Both are left out for a limit that was not lowered.
+	LimitPercent        *int64 `json:"limit_percent,omitempty"`
+	LimitPercentSetting string `json:"limit_percent_setting,omitempty"`
 	// RuleID and Target are, for a ban for a clear sign of attack, the id
 	// of the rule file rule that matched, and its target.
 	RuleID string `json:"rule_id,omitempty"`

internal/config/config.go

@@ -24,6 +24,7 @@ import (
 	"unicode/utf8"
 
 	"sneak.berlin/go/smallwebwaf/internal/alerts"
+	"sneak.berlin/go/smallwebwaf/internal/anomaly"
 	"sneak.berlin/go/smallwebwaf/internal/remotelog"
 )
 
@@ -120,6 +121,37 @@ type Config struct {
 	// capitals, as GeoJS gives them.
 	DeniedCountries             []string
 	ExclusivelyAllowedCountries []string
+	// The biased thresholds. ASNLimitPercent and CountryLimitPercent give
+	// the clients of the AS numbers and the countries they list that
+	// percentage of every rate limit and byte limit
+	// (SWWAF_ASN_LIMIT_PERCENT and SWWAF_COUNTRY_LIMIT_PERCENT).
+	// ASNBytesPercent and CountryBytesPercent give those they list a
+	// percentage of the byte limits in place of that one
+	// (SWWAF_ASN_BYTES_PERCENT and SWWAF_COUNTRY_BYTES_PERCENT). Each holds
+	// percentages from 0 to 100, by AS number, written as AS64496, or by
+	// country, a two-letter code in capitals, as the lookup gives them.
+	// UnknownLimitPercent is the percentage of every limit a client without
+	// a country gets (SWWAF_UNKNOWN_LIMIT_PERCENT). ASNLimitPercentURL is
+	// where a file of AS:percent lines is fetched from, whose percentages
+	// count as those of ASNLimitPercent do (SWWAF_ASN_LIMIT_PERCENT_URL), ""
+	// while it is unset.
+	ASNLimitPercent     map[string]int64
+	CountryLimitPercent map[string]int64
+	ASNBytesPercent     map[string]int64
+	CountryBytesPercent map[string]int64
+	UnknownLimitPercent int64
+	ASNLimitPercentURL  string
+	// BlocklistURLs are where the blocklists are fetched from
+	// (SWWAF_BLOCKLIST_URLS). Each list, and ASNLimitPercentURL's, is
+	// fetched again BlocklistRefresh after it was last fetched or tried
+	// (SWWAF_BLOCKLIST_REFRESH), which is never less than an hour.
+	// BlocklistAction is what is done with a client a blocklist lists
+	// (SWWAF_BLOCKLIST_ACTION): deny, limit or log; for limit,
+	// BlocklistLimitPercent is the percentage of every limit it gets.
+	BlocklistURLs         []string
+	BlocklistRefresh      time.Duration
+	BlocklistAction       string
+	BlocklistLimitPercent int64
 	// BanResponse is the status a refused client is answered with, 403
 	// or 429, or 0 to close the connection without an answer
 	// (SWWAF_BAN_RESPONSE). It answers a banned client, a request that
@@ -204,6 +236,23 @@ type Config struct {
 	AlertEvents          []string
 	AlertCooldown        time.Duration
 	AlertMaxPerHour      int
+	// The anomaly thresholds, which only raise alerts: the most requests
+	// and bytes a minute and an hour per client (SWWAF_ANOMALY_CLIENT_*),
+	// per netblock around a client (SWWAF_ANOMALY_NET_*), per AS number
+	// (SWWAF_ANOMALY_ASN_*), for the whole service (SWWAF_ANOMALY_TOTAL_*)
+	// and per named netblock (SWWAF_WATCH_*), each 0 while it is off.
+	// AnomalyNetV4Prefix and AnomalyNetV6Prefix are the lengths of the
+	// netblock around a client (SWWAF_ANOMALY_NET_V4_PREFIX and
+	// SWWAF_ANOMALY_NET_V6_PREFIX), and WatchNets the named netblocks
+	// (SWWAF_WATCH_NETS).
+	AnomalyClient      anomaly.Thresholds
+	AnomalyNet         anomaly.Thresholds
+	AnomalyASN         anomaly.Thresholds
+	AnomalyTotal       anomaly.Thresholds
+	AnomalyWatch       anomaly.Thresholds
+	AnomalyNetV4Prefix int
+	AnomalyNetV6Prefix int
+	WatchNets          []anomaly.NamedNetblock
 
 	// settings are the values read, as given or by default, and the
 	// files they were read from, for the log line at start.
@@ -224,6 +273,7 @@ const (
 	mebibyte = 1 << 20
 	gibibyte = 1 << 30
 	ipv4Bits = 32
+	ipv6Bits = 128
 	// minTokenLength is the fewest characters a token may have.
 	minTokenLength = 32
 	// masked is what the log shows for a token that is set, and in place of
@@ -271,6 +321,10 @@ var (
 	errNotBanResponse = errors.New("is not 403, 429 or close")
 	errNotV4Prefix    = errors.New(
 		"is not the length of an IPv4 netblock, from 0 to 32, such as 24")
+	errNotV6Prefix = errors.New(
+		"is not the length of an IPv6 netblock, from 0 to 128, such as 48")
+	errNotNamedNetblock = errors.New(
+		"is not a name, = and a netblock, such as office=203.0.113.0/24")
 	errNotAbsolutePath = errors.New(
 		"is not an absolute path, such as /var/lib/smallwebwaf")
 	errShortToken    = errors.New("is shorter than 32 characters")
@@ -300,6 +354,18 @@ var (
 			"source_failure or file_error")
 	errNotNumberOrOff = errors.New("is not a whole number above zero, such as 60, or off")
 	errNotUTF8        = errors.New("is not valid UTF-8")
+	errNotASN         = errors.New("is not an AS number such as AS64496")
+	errNotPercentItem = errors.New(
+		"is not a code, : and a percentage, such as AS64496:50 or cn:25")
+	errNotPercent  = errors.New("is not a percentage, a whole number from 0 to 100")
+	errListedTwice = errors.New("is listed twice")
+	errNotListURL  = errors.New(
+		"is not an http or https URL without a user or a fragment, " +
+			"such as https://www.spamhaus.org/drop/drop.txt")
+	errInBlocklistURLs = errors.New("is in SWWAF_BLOCKLIST_URLS too")
+	errNotAnHourOrMore = errors.New("is not a duration of 1h or more, such as 24h")
+	errNotAction       = errors.New(
+		"is not deny, limit:<percent> such as limit:25, or log")
 )
 
 // FromEnvironment reads the settings with lookupEnv, normally
@@ -307,6 +373,8 @@ var (
 // named by the setting's name with _FILE added names the file, which is
 // read now (see lookup). A setting that is not set takes its default. A
 // setting that is set but invalid is an error that names it.
+//
+//nolint:funlen // one line for each setting, a list that grows with them
 func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
 	env := &environment{lookupEnv: lookupEnv}
 	cfg := &Config{
@@ -342,6 +410,14 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
 		DeniedCountries:         env.countries("SWWAF_DENIED_COUNTRIES", ""),
 		ExclusivelyAllowedCountries: env.countries(
 			"SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES", ""),
+		ASNLimitPercent:      env.percents("SWWAF_ASN_LIMIT_PERCENT", ParseASN),
+		CountryLimitPercent:  env.percents("SWWAF_COUNTRY_LIMIT_PERCENT", parseCountry),
+		ASNBytesPercent:      env.percents("SWWAF_ASN_BYTES_PERCENT", ParseASN),
+		CountryBytesPercent:  env.percents("SWWAF_COUNTRY_BYTES_PERCENT", parseCountry),
+		UnknownLimitPercent:  env.percent("SWWAF_UNKNOWN_LIMIT_PERCENT", "100"),
+		ASNLimitPercentURL:   env.listURL("SWWAF_ASN_LIMIT_PERCENT_URL"),
+		BlocklistURLs:        env.listURLs("SWWAF_BLOCKLIST_URLS"),
+		BlocklistRefresh:     env.refresh("SWWAF_BLOCKLIST_REFRESH", "24h"),
 		BanResponse:          env.banResponse("SWWAF_BAN_RESPONSE", "403"),
 		LimitBanDuration:     env.durationNotOff("SWWAF_LIMIT_BAN_DURATION", "1h"),
 		LimitBanRepeatWindow: env.durationNotOff("SWWAF_LIMIT_BAN_REPEAT_WINDOW", "24h"),
@@ -370,16 +446,27 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
 		AlertNtfyToken:       env.secret("SWWAF_ALERT_NTFY_TOKEN"),
 		AlertEvents: env.alertEvents("SWWAF_ALERT_EVENTS",
 			strings.Join(alerts.Events(), ",")),
-		AlertCooldown:   env.duration("SWWAF_ALERT_COOLDOWN", "15m"),
-		AlertMaxPerHour: env.numberOrOff("SWWAF_ALERT_MAX_PER_HOUR", "60"),
+		AlertCooldown:      env.duration("SWWAF_ALERT_COOLDOWN", "15m"),
+		AlertMaxPerHour:    env.numberOrOff("SWWAF_ALERT_MAX_PER_HOUR", "60"),
+		AnomalyClient:      env.thresholds("SWWAF_ANOMALY_CLIENT_"),
+		AnomalyNet:         env.thresholds("SWWAF_ANOMALY_NET_"),
+		AnomalyASN:         env.thresholds("SWWAF_ANOMALY_ASN_"),
+		AnomalyTotal:       env.thresholds("SWWAF_ANOMALY_TOTAL_"),
+		AnomalyWatch:       env.thresholds("SWWAF_WATCH_"),
+		AnomalyNetV4Prefix: env.v4Prefix("SWWAF_ANOMALY_NET_V4_PREFIX", "24"),
+		AnomalyNetV6Prefix: env.v6Prefix("SWWAF_ANOMALY_NET_V6_PREFIX", "48"),
+		WatchNets:          env.namedNetblocks("SWWAF_WATCH_NETS"),
 	}
 
 	cfg.LogRemoteAppName = env.appName("SWWAF_LOG_REMOTE_APP_NAME",
 		cfg.InstanceName, cfg.LogRemoteURL != nil)
+	cfg.BlocklistAction, cfg.BlocklistLimitPercent = env.action(
+		"SWWAF_BLOCKLIST_ACTION", "deny")
 
 	env.checkInstanceNameForNtfy(cfg.InstanceName, cfg.AlertNtfyURL != nil)
 	env.checkLookupDBPath(cfg)
 	env.checkCountriesAndLookups(cfg)
+	env.checkASNLimitPercentURL(cfg)
 
 	if env.err != nil {
 		return nil, env.err
@@ -592,6 +679,79 @@ func (e *environment) countries(name, defaultValue string) []string {
 	return countries
 }
 
+// percents reads a setting that is a list of AS numbers or countries,
+// which parseCode reads, each with a percentage. It is empty by default.
+func (e *environment) percents(
+	name string, parseCode func(string) (string, error),
+) map[string]int64 {
+	percents, err := parsePercents(e.value(name, ""), parseCode)
+	e.check(name, err)
+
+	return percents
+}
+
+// percent reads a setting that is a percentage, from 0 to 100.
+func (e *environment) percent(name, defaultValue string) int64 {
+	percent, err := ParsePercent(e.value(name, defaultValue))
+	e.check(name, err)
+
+	return percent
+}
+
+// listURL reads a setting that is the URL a list is fetched from, "" while
+// it is unset or empty.
+func (e *environment) listURL(name string) string {
+	value := e.value(name, "")
+	if value != "" && !isListURL(value) {
+		e.check(name, fmt.Errorf("%q %w", value, errNotListURL))
+	}
+
+	return value
+}
+
+// listURLs reads a setting that is a list of the URLs lists are fetched
+// from. It is empty by default.
+func (e *environment) listURLs(name string) []string {
+	urls, err := parseListURLs(e.value(name, ""))
+	e.check(name, err)
+
+	return urls
+}
+
+// refresh reads the setting that is how long after a list was last
+// fetched or tried it is fetched again: a duration of an hour or more,
+// since the Spamhaus lists may be fetched no more often, which cannot be
+// off.
+func (e *environment) refresh(name, defaultValue string) time.Duration {
+	value := e.value(name, defaultValue)
+
+	duration, err := parseDuration(value)
+	if err != nil || duration < time.Hour {
+		e.check(name, fmt.Errorf("%q %w", value, errNotAnHourOrMore))
+	}
+
+	return duration
+}
+
+// action reads a setting that is what is done with a client a list names:
+// deny, log, or limit:<percent>, which it returns as limit and the
+// percentage.
+func (e *environment) action(name, defaultValue string) (string, int64) {
+	value := e.value(name, defaultValue)
+	if value == "deny" || value == "log" {
+		return value, 0
+	}
+
+	percentText, isLimit := strings.CutPrefix(value, "limit:")
+
+	percent, err := ParsePercent(percentText)
+	if !isLimit || err != nil {
+		e.check(name, fmt.Errorf("%q %w", value, errNotAction))
+	}
+
+	return "limit", percent
+}
+
 // lookupSource reads the setting that is where clients are looked up:
 // geojs, file, or off.
 func (e *environment) lookupSource(name, defaultValue string) string {
@@ -619,7 +779,10 @@ func (e *environment) checkLookupDBPath(cfg *Config) {
 
 // checkCountriesAndLookups refuses a country on both country lists, and,
 // while SWWAF_LOOKUP_SOURCE is off, each setting that needs clients looked
-// up: the country lists and SWWAF_ADD_LOOKUP_HEADERS.
+// up: the country lists, SWWAF_ADD_LOOKUP_HEADERS, the biased thresholds,
+// SWWAF_ASN_LIMIT_PERCENT_URL among them, of which
+// SWWAF_UNKNOWN_LIMIT_PERCENT needs them only below 100, where it lowers a
+// limit, and the anomaly thresholds per AS number.
 func (e *environment) checkCountriesAndLookups(cfg *Config) {
 	for _, country := range cfg.ExclusivelyAllowedCountries {
 		if slices.Contains(cfg.DeniedCountries, country) {
@@ -639,6 +802,16 @@ func (e *environment) checkCountriesAndLookups(cfg *Config) {
 		{"SWWAF_DENIED_COUNTRIES", len(cfg.DeniedCountries) > 0},
 		{"SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES", len(cfg.ExclusivelyAllowedCountries) > 0},
 		{"SWWAF_ADD_LOOKUP_HEADERS", cfg.AddLookupHeaders},
+		{"SWWAF_ASN_LIMIT_PERCENT", len(cfg.ASNLimitPercent) > 0},
+		{"SWWAF_COUNTRY_LIMIT_PERCENT", len(cfg.CountryLimitPercent) > 0},
+		{"SWWAF_ASN_BYTES_PERCENT", len(cfg.ASNBytesPercent) > 0},
+		{"SWWAF_COUNTRY_BYTES_PERCENT", len(cfg.CountryBytesPercent) > 0},
+		{"SWWAF_UNKNOWN_LIMIT_PERCENT", cfg.UnknownLimitPercent < 100},
+		{"SWWAF_ASN_LIMIT_PERCENT_URL", cfg.ASNLimitPercentURL != ""},
+		{"SWWAF_ANOMALY_ASN_REQUESTS_PER_MINUTE", cfg.AnomalyASN.RequestsPerMinute > 0},
+		{"SWWAF_ANOMALY_ASN_REQUESTS_PER_HOUR", cfg.AnomalyASN.RequestsPerHour > 0},
+		{"SWWAF_ANOMALY_ASN_BYTES_PER_MINUTE", cfg.AnomalyASN.BytesPerMinute > 0},
+		{"SWWAF_ANOMALY_ASN_BYTES_PER_HOUR", cfg.AnomalyASN.BytesPerHour > 0},
 	} {
 		if setting.set {
 			e.check(setting.name, fmt.Errorf("is set while SWWAF_LOOKUP_SOURCE is off; %w",
@@ -647,6 +820,15 @@ func (e *environment) checkCountriesAndLookups(cfg *Config) {
 	}
 }
 
+// checkASNLimitPercentURL refuses SWWAF_ASN_LIMIT_PERCENT_URL naming a
+// blocklist too: the file at a URL is fetched as one list or the other.
+func (e *environment) checkASNLimitPercentURL(cfg *Config) {
+	if slices.Contains(cfg.BlocklistURLs, cfg.ASNLimitPercentURL) {
+		e.check("SWWAF_ASN_LIMIT_PERCENT_URL",
+			fmt.Errorf("%q %w", cfg.ASNLimitPercentURL, errInBlocklistURLs))
+	}
+}
+
 // headerNames reads a setting that is a list of header names, and
 // returns them in lower case.
 func (e *environment) headerNames(name, defaultValue string) []string {
@@ -690,6 +872,35 @@ func (e *environment) v4Prefix(name, defaultValue string) int {
 	return length
 }
 
+// v6Prefix reads a setting that is the length of an IPv6 netblock.
+func (e *environment) v6Prefix(name, defaultValue string) int {
+	length, err := parseV6Prefix(e.value(name, defaultValue))
+	e.check(name, err)
+
+	return length
+}
+
+// thresholds reads the four anomaly thresholds whose settings' names
+// start with prefix: requests and bytes per minute and per hour. Each is
+// off by default.
+func (e *environment) thresholds(prefix string) anomaly.Thresholds {
+	return anomaly.Thresholds{
+		RequestsPerMinute: e.count(prefix+"REQUESTS_PER_MINUTE", off),
+		RequestsPerHour:   e.count(prefix+"REQUESTS_PER_HOUR", off),
+		BytesPerMinute:    e.size(prefix+"BYTES_PER_MINUTE", off),
+		BytesPerHour:      e.size(prefix+"BYTES_PER_HOUR", off),
+	}
+}
+
+// namedNetblocks reads a setting that is a list of named netblocks. It is
+// empty by default.
+func (e *environment) namedNetblocks(name string) []anomaly.NamedNetblock {
+	named, err := parseNamedNetblocks(e.value(name, ""))
+	e.check(name, err)
+
+	return named
+}
+
 // absolutePath reads a setting that is an absolute path.
 func (e *environment) absolutePath(name, defaultValue string) string {
 	path := e.value(name, defaultValue)
@@ -1031,6 +1242,16 @@ func parseV4Prefix(value string) (int, error) {
 	return n, nil
 }
 
+// parseV6Prefix reads the length of an IPv6 netblock, from 0 to 128.
+func parseV6Prefix(value string) (int, error) {
+	n, err := strconv.Atoi(value)
+	if err != nil || n < 0 || n > ipv6Bits {
+		return 0, fmt.Errorf("%q %w", value, errNotV6Prefix)
+	}
+
+	return n, nil
+}
+
 // parseList splits a comma-separated list and trims the spaces around
 // each item. An empty value is an empty list.
 func parseList(value string) ([]string, error) {
@@ -1059,7 +1280,7 @@ func parseNetblocks(value string) ([]netip.Prefix, error) {
 	netblocks := make([]netip.Prefix, 0, len(items))
 
 	for _, item := range items {
-		netblock, err := parseNetblock(item)
+		netblock, err := ParseNetblock(item)
 		if err != nil {
 			return nil, err
 		}
@@ -1070,9 +1291,10 @@ func parseNetblocks(value string) ([]netip.Prefix, error) {
 	return netblocks, nil
 }
 
-// parseNetblock reads a netblock in CIDR form, such as 10.0.0.0/8. A bare
-// address is a netblock of that address alone, a /32 or a /128.
-func parseNetblock(value string) (netip.Prefix, error) {
+// ParseNetblock reads a netblock in CIDR form, such as 10.0.0.0/8. A bare
+// address is a netblock of that address alone, a /32 or a /128. A
+// blocklist's lines are read with it too.
+func ParseNetblock(value string) (netip.Prefix, error) {
 	if strings.Contains(value, "/") {
 		netblock, err := netip.ParsePrefix(value)
 		if err != nil {
@@ -1090,6 +1312,42 @@ func parseNetblock(value string) (netip.Prefix, error) {
 	return netip.PrefixFrom(addr, addr.BitLen()), nil
 }
 
+// parseNamedNetblocks reads a comma-separated list of named netblocks,
+// each a name, = and a netblock, such as office=203.0.113.0/24. An empty
+// value is an empty list. A name listed twice is an error.
+func parseNamedNetblocks(value string) ([]anomaly.NamedNetblock, error) {
+	items, err := parseList(value)
+	if err != nil {
+		return nil, err
+	}
+
+	named := make([]anomaly.NamedNetblock, 0, len(items))
+
+	for _, item := range items {
+		name, netblockText, found := strings.Cut(item, "=")
+		name = strings.TrimSpace(name)
+
+		if !found || name == "" {
+			return nil, fmt.Errorf("%q %w", item, errNotNamedNetblock)
+		}
+
+		netblock, err := ParseNetblock(strings.TrimSpace(netblockText))
+		if err != nil {
+			return nil, err
+		}
+
+		if slices.ContainsFunc(named, func(n anomaly.NamedNetblock) bool {
+			return n.Name == name
+		}) {
+			return nil, fmt.Errorf("%q %w", name, errListedTwice)
+		}
+
+		named = append(named, anomaly.NamedNetblock{Name: name, Netblock: netblock})
+	}
+
+	return named, nil
+}
+
 // parsePathPrefixes reads a comma-separated list of path prefixes, each
 // starting with /.
 func parsePathPrefixes(value string) ([]string, error) {
@@ -1148,13 +1406,12 @@ func parseCountries(value string) ([]string, error) {
 		return nil, err
 	}
 
-	known := strings.Fields(countryCodes)
 	countries := make([]string, 0, len(items))
 
 	for _, item := range items {
-		country := strings.ToUpper(item)
-		if !slices.Contains(known, country) {
-			return nil, fmt.Errorf("%q %w", item, errNotCountry)
+		country, err := parseCountry(item)
+		if err != nil {
+			return nil, err
 		}
 
 		countries = append(countries, country)
@@ -1163,6 +1420,83 @@ func parseCountries(value string) ([]string, error) {
 	return countries, nil
 }
 
+// parseCountry reads a country code in either case, and returns it in
+// capitals.
+func parseCountry(value string) (string, error) {
+	country := strings.ToUpper(value)
+	if !slices.Contains(strings.Fields(countryCodes), country) {
+		return "", fmt.Errorf("%q %w", value, errNotCountry)
+	}
+
+	return country, nil
+}
+
+// ParseASN reads an AS number such as AS64496, in either case, and
+// returns it as the lookup gives it: AS and the number, in capitals and
+// without leading zeros. The file SWWAF_ASN_LIMIT_PERCENT_URL names is
+// read with it too.
+func ParseASN(value string) (string, error) {
+	digits, hasAS := strings.CutPrefix(strings.ToUpper(value), "AS")
+
+	number, err := strconv.ParseUint(digits, 10, 32)
+	if !hasAS || err != nil {
+		return "", fmt.Errorf("%q %w", value, errNotASN)
+	}
+
+	return "AS" + strconv.FormatUint(number, 10), nil
+}
+
+// parsePercents reads a comma-separated list of items, each an AS number
+// or a country, which parseCode reads, then : and a percentage, such as
+// AS64496:50 or cn:25, and returns each one's percentage. An empty value
+// is an empty list. An AS number or country listed twice is an error.
+func parsePercents(
+	value string, parseCode func(string) (string, error),
+) (map[string]int64, error) {
+	items, err := parseList(value)
+	if err != nil {
+		return nil, err
+	}
+
+	percents := make(map[string]int64, len(items))
+
+	for _, item := range items {
+		codeText, percentText, found := strings.Cut(item, ":")
+		if !found {
+			return nil, fmt.Errorf("%q %w", item, errNotPercentItem)
+		}
+
+		code, err := parseCode(codeText)
+		if err != nil {
+			return nil, err
+		}
+
+		percent, err := ParsePercent(percentText)
+		if err != nil {
+			return nil, err
+		}
+
+		if _, listed := percents[code]; listed {
+			return nil, fmt.Errorf("%q %w", codeText, errListedTwice)
+		}
+
+		percents[code] = percent
+	}
+
+	return percents, nil
+}
+
+// ParsePercent reads a percentage, a whole number from 0 to 100. The file
+// SWWAF_ASN_LIMIT_PERCENT_URL names is read with it too.
+func ParsePercent(value string) (int64, error) {
+	percent, err := strconv.ParseInt(value, 10, 64)
+	if err != nil || percent < 0 || percent > 100 {
+		return 0, fmt.Errorf("%q %w", value, errNotPercent)
+	}
+
+	return percent, nil
+}
+
 // headerNameChars are the characters RFC 9110 allows in a header name:
 // letters, digits and these marks.
 const headerNameChars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz" +
@@ -1310,16 +1644,7 @@ func parseWebhookURL(value string) (*url.URL, string, error) {
 	}
 
 	webhook, err := url.Parse(value)
-	if err != nil {
-		return nil, "", errNotWebhookURL
-	}
-
-	port, err := strconv.ParseUint(webhook.Port(), 10, 16)
-
-	valid := (webhook.Scheme == "http" || webhook.Scheme == "https") &&
-		webhook.Hostname() != "" && (webhook.Port() == "" || (err == nil && port != 0)) &&
-		webhook.User == nil && webhook.Opaque == "" && webhook.Fragment == ""
-	if !valid {
+	if err != nil || !isHTTPURL(webhook) {
 		return nil, "", errNotWebhookURL
 	}
 
@@ -1331,6 +1656,47 @@ func parseWebhookURL(value string) (*url.URL, string, error) {
 	return webhook, logged, nil
 }
 
+// isHTTPURL reports whether u is http or https, with a host, and an
+// optional port from 1 to 65535, path and query, without a user or a
+// fragment.
+func isHTTPURL(u *url.URL) bool {
+	port, err := strconv.ParseUint(u.Port(), 10, 16)
+
+	return (u.Scheme == "http" || u.Scheme == "https") && u.Hostname() != "" &&
+		(u.Port() == "" || (err == nil && port != 0)) &&
+		u.User == nil && u.Opaque == "" && u.Fragment == ""
+}
+
+// isListURL reports whether value is a URL a list can be fetched from, as
+// isHTTPURL says.
+func isListURL(value string) bool {
+	u, err := url.Parse(value)
+
+	return err == nil && isHTTPURL(u)
+}
+
+// parseListURLs reads a comma-separated list of the URLs lists are fetched
+// from. A URL listed twice is an error: it would be fetched twice as
+// often.
+func parseListURLs(value string) ([]string, error) {
+	urls, err := parseList(value)
+	if err != nil {
+		return nil, err
+	}
+
+	for i, listURL := range urls {
+		if !isListURL(listURL) {
+			return nil, fmt.Errorf("%q %w", listURL, errNotListURL)
+		}
+
+		if slices.Contains(urls[:i], listURL) {
+			return nil, fmt.Errorf("%q %w", listURL, errListedTwice)
+		}
+	}
+
+	return urls, nil
+}
+
 // parseWebhookHeaders reads a comma-separated list of headers, each its
 // name, :, and its value, and returns them, and how the log shows them,
 // with each value as ********. An error names the item by its place in

internal/config/config_test.go

@@ -12,10 +12,12 @@ import (
 	"path/filepath"
 	"reflect"
 	"slices"
+	"strconv"
 	"strings"
 	"testing"
 	"time"
 
+	"sneak.berlin/go/smallwebwaf/internal/anomaly"
 	"sneak.berlin/go/smallwebwaf/internal/config"
 )
 
@@ -50,6 +52,15 @@ const (
 	addLookupHeaders        = "SWWAF_ADD_LOOKUP_HEADERS"
 	deniedCountries         = "SWWAF_DENIED_COUNTRIES"
 	allowedCountries        = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES"
+	asnLimitPercent         = "SWWAF_ASN_LIMIT_PERCENT"
+	countryLimitPercent     = "SWWAF_COUNTRY_LIMIT_PERCENT"
+	asnBytesPercent         = "SWWAF_ASN_BYTES_PERCENT"
+	countryBytesPercent     = "SWWAF_COUNTRY_BYTES_PERCENT"
+	unknownLimitPercent     = "SWWAF_UNKNOWN_LIMIT_PERCENT"
+	asnLimitPercentURL      = "SWWAF_ASN_LIMIT_PERCENT_URL"
+	blocklistURLs           = "SWWAF_BLOCKLIST_URLS"
+	blocklistRefresh        = "SWWAF_BLOCKLIST_REFRESH"
+	blocklistAction         = "SWWAF_BLOCKLIST_ACTION"
 	banResponse             = "SWWAF_BAN_RESPONSE"
 	limitBanDuration        = "SWWAF_LIMIT_BAN_DURATION"
 	limitBanRepeatWindow    = "SWWAF_LIMIT_BAN_REPEAT_WINDOW"
@@ -80,8 +91,59 @@ const (
 	alertEvents             = "SWWAF_ALERT_EVENTS"
 	alertCooldown           = "SWWAF_ALERT_COOLDOWN"
 	alertMaxPerHour         = "SWWAF_ALERT_MAX_PER_HOUR"
+	anomalyNetV4Prefix      = "SWWAF_ANOMALY_NET_V4_PREFIX"
+	anomalyNetV6Prefix      = "SWWAF_ANOMALY_NET_V6_PREFIX"
+	watchNets               = "SWWAF_WATCH_NETS"
 )
 
+// The anomaly thresholds: each of the prefixes below, which name a scope,
+// followed by each of the four ends.
+const (
+	anomalyClient = "SWWAF_ANOMALY_CLIENT_"
+	anomalyNet    = "SWWAF_ANOMALY_NET_"
+	anomalyASN    = "SWWAF_ANOMALY_ASN_"
+	anomalyTotal  = "SWWAF_ANOMALY_TOTAL_"
+	watch         = "SWWAF_WATCH_"
+
+	requestsPerMinute = "REQUESTS_PER_MINUTE"
+	requestsPerHour   = "REQUESTS_PER_HOUR"
+	bytesPerMinute    = "BYTES_PER_MINUTE"
+	bytesPerHour      = "BYTES_PER_HOUR"
+)
+
+// anomalyScopes returns the prefixes of the anomaly thresholds, one for
+// each scope.
+func anomalyScopes() []string {
+	return []string{anomalyClient, anomalyNet, anomalyASN, anomalyTotal, watch}
+}
+
+// anomalyThresholds returns the names of the twenty anomaly thresholds.
+func anomalyThresholds() []string {
+	ends := []string{requestsPerMinute, requestsPerHour, bytesPerMinute, bytesPerHour}
+	names := make([]string, 0, len(anomalyScopes())*len(ends))
+
+	for _, scope := range anomalyScopes() {
+		for _, end := range ends {
+			names = append(names, scope+end)
+		}
+	}
+
+	return names
+}
+
+// loggedAnomalyDefaults returns the anomaly settings as the settings
+// logged at start give them by default.
+func loggedAnomalyDefaults() map[string]string {
+	logged := map[string]string{
+		anomalyNetV4Prefix: "24", anomalyNetV6Prefix: "48", watchNets: "",
+	}
+	for _, name := range anomalyThresholds() {
+		logged[name] = off
+	}
+
+	return logged
+}
+
 // defaultAlertEvents is the default of SWWAF_ALERT_EVENTS, and
 // defaultAlertCooldown that of SWWAF_ALERT_COOLDOWN.
 const (
@@ -892,9 +954,19 @@ func TestSettingNeedingLookupsStopsTheStartWhileTheyAreOff(t *testing.T) {
 	t.Parallel()
 
 	for name, value := range map[string]string{
-		deniedCountries:  "kp",
-		allowedCountries: "de",
-		addLookupHeaders: enabled,
+		deniedCountries:                "kp",
+		allowedCountries:               "de",
+		addLookupHeaders:               enabled,
+		asnLimitPercentURL:             asnURL,
+		asnLimitPercent:                "AS64496:50",
+		countryLimitPercent:            "cn:25",
+		asnBytesPercent:                "AS64496:50",
+		countryBytesPercent:            "cn:25",
+		unknownLimitPercent:            "99",
+		anomalyASN + requestsPerMinute: "1000",
+		anomalyASN + requestsPerHour:   "10000",
+		anomalyASN + bytesPerMinute:    "1G",
+		anomalyASN + bytesPerHour:      "10G",
 	} {
 		t.Run(name, func(t *testing.T) {
 			t.Parallel()
@@ -909,10 +981,345 @@ func TestSettingNeedingLookupsStopsTheStartWhileTheyAreOff(t *testing.T) {
 		})
 	}
 
-	// Set empty, the country lists need nothing looked up.
-	fromEnvironment(t, environment{
+	// Set empty, the lists need nothing looked up, and nor does
+	// SWWAF_UNKNOWN_LIMIT_PERCENT at 100, which lowers no limit, an anomaly
+	// threshold per AS number that is off, any other anomaly threshold, or
+	// a blocklist.
+	env := environment{
 		lookupSource: off, deniedCountries: "", allowedCountries: "",
+		asnLimitPercent: "", countryLimitPercent: "", asnBytesPercent: "",
+		countryBytesPercent: "", unknownLimitPercent: "100", asnLimitPercentURL: "",
+		blocklistURLs: dropURL,
+	}
+	for _, name := range anomalyThresholds() {
+		env[name] = "1000"
+		if strings.HasPrefix(name, anomalyASN) {
+			env[name] = off
+		}
+	}
+
+	fromEnvironment(t, env)
+}
+
+func TestAnomalySettingsDefaults(t *testing.T) {
+	t.Parallel()
+
+	cfg := fromEnvironment(t, environment{})
+	wantAllOff(t, cfg)
+
+	if cfg.AnomalyNetV4Prefix != 24 || cfg.AnomalyNetV6Prefix != 48 ||
+		len(cfg.WatchNets) != 0 {
+		t.Errorf("%s, %s and %s gave %d, %d and %v, want 24, 48 and none",
+			anomalyNetV4Prefix, anomalyNetV6Prefix, watchNets, cfg.AnomalyNetV4Prefix,
+			cfg.AnomalyNetV6Prefix, cfg.WatchNets)
+	}
+}
+
+func TestAnomalySettingsAsSet(t *testing.T) {
+	t.Parallel()
+
+	// Each threshold of a scope its own value; bytes are sizes.
+	env := environment{
+		anomalyNetV4Prefix: "16",
+		anomalyNetV6Prefix: "56",
+		// Spaces around a name or a netblock, and a bare address.
+		watchNets: "office = 203.0.113.0/24, scraper-x=198.51.100.7,v6=2001:db8::/32",
+	}
+	want := map[string]anomaly.Thresholds{}
+
+	for i, scope := range anomalyScopes() {
+		n := int64(i + 1)
+		env[scope+requestsPerMinute] = strconv.FormatInt(n, 10)
+		env[scope+requestsPerHour] = strconv.FormatInt(10*n, 10)
+		env[scope+bytesPerMinute] = strconv.FormatInt(n, 10) + "K"
+		env[scope+bytesPerHour] = strconv.FormatInt(n, 10) + "G"
+		want[scope] = anomaly.Thresholds{
+			RequestsPerMinute: n, RequestsPerHour: 10 * n,
+			BytesPerMinute: n << 10, BytesPerHour: n << 30,
+		}
+	}
+
+	cfg := fromEnvironment(t, env)
+
+	if got := thresholdsByScope(cfg); !maps.Equal(got, want) {
+		t.Errorf("thresholds by scope\n%+v\nwant\n%+v", got, want)
+	}
+
+	wantNamed := []anomaly.NamedNetblock{
+		{Name: "office", Netblock: netip.MustParsePrefix("203.0.113.0/24")},
+		{Name: "scraper-x", Netblock: netip.MustParsePrefix("198.51.100.7/32")},
+		{Name: "v6", Netblock: netip.MustParsePrefix("2001:db8::/32")},
+	}
+	if cfg.AnomalyNetV4Prefix != 16 || cfg.AnomalyNetV6Prefix != 56 ||
+		!slices.Equal(cfg.WatchNets, wantNamed) {
+		t.Errorf("%s, %s and %s gave %d, %d and %v, want 16, 56 and %v",
+			anomalyNetV4Prefix, anomalyNetV6Prefix, watchNets, cfg.AnomalyNetV4Prefix,
+			cfg.AnomalyNetV6Prefix, cfg.WatchNets, wantNamed)
+	}
+
+	// off switches each threshold off.
+	for _, name := range anomalyThresholds() {
+		env[name] = off
+	}
+
+	wantAllOff(t, fromEnvironment(t, env))
+}
+
+// thresholdsByScope returns cfg's anomaly thresholds, each by the prefix
+// of its scope's settings.
+func thresholdsByScope(cfg *config.Config) map[string]anomaly.Thresholds {
+	return map[string]anomaly.Thresholds{
+		anomalyClient: cfg.AnomalyClient, anomalyNet: cfg.AnomalyNet,
+		anomalyASN: cfg.AnomalyASN, anomalyTotal: cfg.AnomalyTotal,
+		watch: cfg.AnomalyWatch,
+	}
+}
+
+// wantAllOff checks that every anomaly threshold of cfg is off.
+func wantAllOff(t *testing.T, cfg *config.Config) {
+	t.Helper()
+
+	for scope, thresholds := range thresholdsByScope(cfg) {
+		if thresholds != (anomaly.Thresholds{}) {
+			t.Errorf("%s* gave %+v, want every one off", scope, thresholds)
+		}
+	}
+}
+
+func TestInvalidAnomalySettingStopsTheStartSayingWhatIsWrong(t *testing.T) {
+	t.Parallel()
+
+	const (
+		notCount        = " is not a whole number of requests such as 1000, or off"
+		notSize         = " is not a size such as 512K, 100M or 5G, or off"
+		notPositive     = " must be more than zero, or off"
+		notNamed        = " is not a name, = and a netblock, such as office=203.0.113.0/24"
+		notNetblock     = " is not a netblock such as 10.0.0.0/8, or an address"
+		notV4Prefix     = " is not the length of an IPv4 netblock, from 0 to 32, such as 24"
+		notV6Prefix     = " is not the length of an IPv6 netblock, from 0 to 128, such as 48"
+		officeNetblock  = "office=203.0.113.0/24"
+		scraperNetblock = "scraper=198.51.100.0/24"
+	)
+
+	for _, tc := range []struct{ name, value, want string }{
+		{anomalyClient + requestsPerMinute, "1K", `"1K"` + notCount},
+		{anomalyNet + requestsPerHour, "0", `"0"` + notPositive},
+		{anomalyTotal + bytesPerMinute, "1T", `"1T"` + notSize},
+		{watch + bytesPerHour, "-1G", `"-1G"` + notPositive},
+		{anomalyNetV4Prefix, "33", `"33"` + notV4Prefix},
+		{anomalyNetV4Prefix, off, `"off"` + notV4Prefix},
+		{anomalyNetV6Prefix, "129", `"129"` + notV6Prefix},
+		{anomalyNetV6Prefix, "/48", `"/48"` + notV6Prefix},
+		{watchNets, "office", `"office"` + notNamed},
+		{watchNets, "=203.0.113.0/24", `"=203.0.113.0/24"` + notNamed},
+		{watchNets, "office=203.0.113.300/24", `"203.0.113.300/24"` + notNetblock},
+		{watchNets, officeNetblock + ",", `"` + officeNetblock + `," has an empty item ` +
+			`in its list`},
+		{
+			watchNets, officeNetblock + "," + scraperNetblock + ",office=192.0.2.0/24",
+			`"office" is listed twice`,
+		},
+	} {
+		t.Run(tc.name+"="+tc.value, func(t *testing.T) {
+			t.Parallel()
+
+			_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
+
+			want := tc.name + ": " + tc.want
+			if err == nil || err.Error() != want {
+				t.Errorf("error %v, want %s", err, want)
+			}
+		})
+	}
+}
+
+func TestBiasedThresholdsAsSet(t *testing.T) {
+	t.Parallel()
+
+	cfg := fromEnvironment(t, environment{})
+	if len(cfg.ASNLimitPercent) != 0 || len(cfg.CountryLimitPercent) != 0 ||
+		len(cfg.ASNBytesPercent) != 0 || len(cfg.CountryBytesPercent) != 0 ||
+		cfg.UnknownLimitPercent != 100 {
+		t.Errorf("biased thresholds %v, %v, %v, %v and %d by default, "+
+			"want four empty lists and 100", cfg.ASNLimitPercent, cfg.CountryLimitPercent,
+			cfg.ASNBytesPercent, cfg.CountryBytesPercent, cfg.UnknownLimitPercent)
+	}
+
+	// AS numbers and countries in either case, an AS number with leading
+	// zeros, 0 and 100.
+	cfg = fromEnvironment(t, environment{
+		asnLimitPercent:     "AS14061:50, as16276:0,AS045102:100",
+		countryLimitPercent: "cn:25,RU:50",
+		asnBytesPercent:     "as16276:75",
+		countryBytesPercent: "ru:10",
+		unknownLimitPercent: "0",
 	})
+
+	for name, tc := range map[string]struct{ got, want map[string]int64 }{
+		asnLimitPercent: {
+			cfg.ASNLimitPercent,
+			map[string]int64{"AS14061": 50, "AS16276": 0, "AS45102": 100},
+		},
+		countryLimitPercent: {cfg.CountryLimitPercent, map[string]int64{"CN": 25, "RU": 50}},
+		asnBytesPercent:     {cfg.ASNBytesPercent, map[string]int64{"AS16276": 75}},
+		countryBytesPercent: {cfg.CountryBytesPercent, map[string]int64{"RU": 10}},
+	} {
+		if !maps.Equal(tc.got, tc.want) {
+			t.Errorf("%s gave %v, want %v", name, tc.got, tc.want)
+		}
+	}
+
+	if cfg.UnknownLimitPercent != 0 {
+		t.Errorf("%s gave %d, want 0", unknownLimitPercent, cfg.UnknownLimitPercent)
+	}
+}
+
+func TestInvalidBiasedThresholdStopsTheStartSayingWhatIsWrong(t *testing.T) {
+	t.Parallel()
+
+	const (
+		notASN     = " is not an AS number such as AS64496"
+		notItem    = " is not a code, : and a percentage, such as AS64496:50 or cn:25"
+		notPercent = " is not a percentage, a whole number from 0 to 100"
+	)
+
+	for _, tc := range []struct{ name, value, want string }{
+		{asnLimitPercent, "14061:50", `"14061"` + notASN},
+		{asnLimitPercent, "AS4294967296:50", `"AS4294967296"` + notASN},
+		{asnLimitPercent, "AS14061", `"AS14061"` + notItem},
+		{asnLimitPercent, "AS14061:101", `"101"` + notPercent},
+		{asnLimitPercent, "AS14061:50,as14061:25", `"as14061" is listed twice`},
+		{
+			countryLimitPercent, "nk:25",
+			`"nk" is not a two-letter country code such as de or kp`,
+		},
+		{countryLimitPercent, "cn:25,CN:50", `"CN" is listed twice`},
+		{asnBytesPercent, "AS14061:-1", `"-1"` + notPercent},
+		{countryBytesPercent, "cn:50%", `"50%"` + notPercent},
+		{unknownLimitPercent, "101", `"101"` + notPercent},
+		{unknownLimitPercent, off, `"off"` + notPercent},
+	} {
+		t.Run(tc.name+"="+tc.value, func(t *testing.T) {
+			t.Parallel()
+
+			_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
+
+			want := tc.name + ": " + tc.want
+			if err == nil || err.Error() != want {
+				t.Errorf("error %v, want %s", err, want)
+			}
+		})
+	}
+}
+
+// dropURL and torURL are blocklists, and asnURL a file of AS:percent
+// lines.
+const (
+	dropURL = "https://www.spamhaus.org/drop/drop.txt"
+	torURL  = "https://lists.example/tor-exits.txt"
+	asnURL  = "https://lists.example/asn.txt"
+)
+
+// The actions of SWWAF_BLOCKLIST_ACTION, as Config gives them.
+const (
+	actionDeny  = "deny"
+	actionLimit = "limit"
+	actionLog   = "log"
+)
+
+func TestReputationSettingsAsSet(t *testing.T) {
+	t.Parallel()
+
+	cfg := fromEnvironment(t, environment{})
+	if len(cfg.BlocklistURLs) != 0 || cfg.BlocklistRefresh != 24*time.Hour ||
+		cfg.BlocklistAction != actionDeny || cfg.ASNLimitPercentURL != "" {
+		t.Errorf("%s, %s, %s and %s gave %v, %s, %s and %q by default, "+
+			"want none, 24h, deny and none", blocklistURLs, blocklistRefresh,
+			blocklistAction, asnLimitPercentURL, cfg.BlocklistURLs, cfg.BlocklistRefresh,
+			cfg.BlocklistAction, cfg.ASNLimitPercentURL)
+	}
+
+	for _, tc := range []struct {
+		value, action string
+		percent       int64
+	}{
+		{actionDeny, actionDeny, 0},
+		{actionLog, actionLog, 0},
+		{"limit:25", actionLimit, 25},
+		{"limit:0", actionLimit, 0},
+	} {
+		// An hour, the shortest refresh allowed.
+		cfg := fromEnvironment(t, environment{
+			blocklistURLs: dropURL + ", " + torURL, blocklistRefresh: "1h",
+			blocklistAction: tc.value, asnLimitPercentURL: asnURL,
+		})
+
+		if !slices.Equal(cfg.BlocklistURLs, []string{dropURL, torURL}) ||
+			cfg.BlocklistRefresh != time.Hour || cfg.BlocklistAction != tc.action ||
+			cfg.BlocklistLimitPercent != tc.percent || cfg.ASNLimitPercentURL != asnURL {
+			t.Errorf("%s=%s gave %v, %s, %s, %d and %s", blocklistAction, tc.value,
+				cfg.BlocklistURLs, cfg.BlocklistRefresh, cfg.BlocklistAction,
+				cfg.BlocklistLimitPercent, cfg.ASNLimitPercentURL)
+		}
+	}
+}
+
+func TestInvalidReputationSettingStopsTheStartSayingWhatIsWrong(t *testing.T) {
+	t.Parallel()
+
+	const (
+		notURL = " is not an http or https URL without a user or a fragment, " +
+			"such as https://www.spamhaus.org/drop/drop.txt"
+		notAnHour = " is not a duration of 1h or more, such as 24h"
+		notAction = " is not deny, limit:<percent> such as limit:25, or log"
+	)
+
+	for _, tc := range []struct{ name, value, want string }{
+		{
+			blocklistURLs, "ftp://lists.example/drop.txt",
+			`"ftp://lists.example/drop.txt"` + notURL,
+		},
+		{blocklistURLs, "lists.example/drop.txt", `"lists.example/drop.txt"` + notURL},
+		{
+			blocklistURLs, "https://me:secret@lists.example/drop.txt",
+			`"https://me:secret@lists.example/drop.txt"` + notURL,
+		},
+		{
+			blocklistURLs, dropURL + "," + torURL + "," + dropURL,
+			`"` + dropURL + `" is listed twice`,
+		},
+		{asnLimitPercentURL, asnURL + "#top", `"` + asnURL + `#top"` + notURL},
+		{blocklistRefresh, "59m", `"59m"` + notAnHour},
+		{blocklistRefresh, off, `"off"` + notAnHour},
+		{blocklistRefresh, "a day", `"a day"` + notAnHour},
+		{blocklistAction, "block", `"block"` + notAction},
+		{blocklistAction, actionLimit, `"limit"` + notAction},
+		{blocklistAction, "limit:101", `"limit:101"` + notAction},
+	} {
+		t.Run(tc.name+"="+tc.value, func(t *testing.T) {
+			t.Parallel()
+
+			_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
+
+			want := tc.name + ": " + tc.want
+			if err == nil || err.Error() != want {
+				t.Errorf("error %v, want %s", err, want)
+			}
+		})
+	}
+}
+
+func TestASNLimitPercentURLThatIsABlocklistStopsTheStart(t *testing.T) {
+	t.Parallel()
+
+	_, err := config.FromEnvironment(environment{
+		blocklistURLs: dropURL + "," + asnURL, asnLimitPercentURL: asnURL,
+	}.lookupEnv)
+
+	want := asnLimitPercentURL + `: "` + asnURL + `" is in SWWAF_BLOCKLIST_URLS too`
+	if err == nil || err.Error() != want {
+		t.Errorf("error %v, want %s", err, want)
+	}
 }
 
 func TestSizesAndOff(t *testing.T) {
@@ -1054,6 +1461,14 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
 		{allowedCountries, "uk"},
 		{allowedCountries, "zz"},
 		{allowedCountries, "de,germany"},
+		{asnLimitPercent, "AS14061:50,,AS16276:50"}, {asnLimitPercent, "ASX:50"},
+		{asnLimitPercent, "AS14061:"}, {asnLimitPercent, "AS14061 :50"},
+		{asnLimitPercent, "AS14061:1.5"}, {asnLimitPercent, "AS-1:50"},
+		{countryLimitPercent, "cn"}, {countryLimitPercent, "cn:"},
+		{countryLimitPercent, "cn:25:50"}, {countryLimitPercent, "china:25"},
+		{asnBytesPercent, "AS14061:101"}, {countryBytesPercent, "su:50"},
+		{unknownLimitPercent, ""}, {unknownLimitPercent, "-1"},
+		{unknownLimitPercent, "50%"},
 		{metricsTopN, off}, {metricsTopN, "0"}, {metricsTopN, "-1"},
 		{logRequestHeaders, "accept,,origin"}, {logRequestHeaders, "accept;origin"},
 		{logRequestHeaders, "accept language"}, {logRequestHeaders, "x-foo:"},
@@ -1325,6 +1740,15 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
 		addLookupHeaders:        "false",
 		deniedCountries:         "",
 		allowedCountries:        "",
+		asnLimitPercent:         "",
+		countryLimitPercent:     "",
+		asnBytesPercent:         "",
+		countryBytesPercent:     "",
+		unknownLimitPercent:     "100",
+		asnLimitPercentURL:      "",
+		blocklistURLs:           "",
+		blocklistRefresh:        "24h",
+		blocklistAction:         actionDeny,
 		banResponse:             "403",
 		limitBanDuration:        "1h",
 		limitBanRepeatWindow:    "24h",
@@ -1356,6 +1780,8 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
 		alertCooldown:           defaultAlertCooldown,
 		alertMaxPerHour:         "60",
 	}
+	maps.Copy(want, loggedAnomalyDefaults())
+
 	if got := loggedSettings(t, cfg); !maps.Equal(got, want) {
 		t.Errorf("logged settings\n%v\nwant\n%v", got, want)
 	}

internal/metrics/metrics.go

@@ -16,6 +16,7 @@ import (
 	"sneak.berlin/go/smallwebwaf/internal/bans"
 	"sneak.berlin/go/smallwebwaf/internal/ratelimit"
 	"sneak.berlin/go/smallwebwaf/internal/remotelog"
+	"sneak.berlin/go/smallwebwaf/internal/reputation"
 	"sneak.berlin/go/smallwebwaf/internal/requestlog"
 	"sneak.berlin/go/smallwebwaf/internal/rules"
 )
@@ -35,10 +36,12 @@ type Metrics struct {
 	rateLimitHits        *prometheus.CounterVec
 	sizeAndTimeLimitHits *prometheus.CounterVec
 	offences             *prometheus.CounterVec
-	// ruleMatches are made by AddRules.
-	ruleMatches *prometheus.CounterVec
-	countries   *busiest
-	asns        *busiest
+	// ruleMatches are made by AddRules, and reputationHits by
+	// AddReputation.
+	ruleMatches    *prometheus.CounterVec
+	reputationHits *prometheus.CounterVec
+	countries      *busiest
+	asns           *busiest
 
 	// GeoJSRequests are the requests to GeoJS, and GeoJSFailures those
 	// that failed. GeoJSUnanswered are the requests that needed their
@@ -253,6 +256,51 @@ func (m *Metrics) AddLookupFile(lastRead func() time.Time, readFailures func() i
 	)
 }
 
+// AddReputation adds the metrics of the lists fetched from URLs, by
+// source, each list's URL: the requests whose client a blocklist lists,
+// which ReputationHit counts, and, read from lists as the metrics are
+// asked for, the fetches that failed and when the copy in use was fetched.
+// It is called once, before ReputationHit.
+func (m *Metrics) AddReputation(lists *reputation.Lists) {
+	m.reputationHits = counterVec("smallwebwaf_reputation_hits_total",
+		"Requests whose client a blocklist lists, by the blocklist's URL.",
+		[]string{"source"})
+	m.registry.MustRegister(m.reputationHits)
+
+	for _, listURL := range lists.URLs() {
+		source := prometheus.Labels{"source": listURL}
+
+		m.registry.MustRegister(
+			prometheus.NewCounterFunc(prometheus.CounterOpts{
+				Name:        "smallwebwaf_reputation_failures_total",
+				Help:        "Fetches of the list that failed.",
+				ConstLabels: source,
+			}, func() float64 {
+				return float64(lists.Failures(listURL))
+			}),
+			prometheus.NewGaugeFunc(prometheus.GaugeOpts{
+				Name: "smallwebwaf_reputation_last_fetch_timestamp_seconds",
+				Help: "When the copy of the list in use was fetched, in seconds since " +
+					"1970, or 0 while there is none.",
+				ConstLabels: source,
+			}, func() float64 {
+				fetched := lists.Fetched(listURL)
+				if fetched.IsZero() {
+					return 0
+				}
+
+				return float64(fetched.Unix())
+			}),
+		)
+	}
+}
+
+// ReputationHit counts a request whose client the blocklist at source, its
+// URL, lists.
+func (m *Metrics) ReputationHit(source string) {
+	m.reputationHits.WithLabelValues(source).Inc()
+}
+
 // AddAlerts adds the metrics of the alerts sent to each destination set,
 // read from queue as the metrics are asked for, by destination: the
 // alerts sent, the requests to the destination that failed, the alerts

internal/proxy/anomaly_internal_test.go

@@ -0,0 +1,30 @@
+package proxy
+
+import (
+	"net/netip"
+	"testing"
+
+	"sneak.berlin/go/smallwebwaf/internal/config"
+)
+
+func TestWithEveryAnomalyThresholdOffARequestIsNotCounted(t *testing.T) {
+	t.Parallel()
+
+	// A request from a client looked up through GeoJS, with every anomaly
+	// threshold off. Its handler has neither GeoJS's answers nor the
+	// anomaly counters, nor a clock, and the request no response: reading
+	// any of them to count the request panics.
+	rq := &request{
+		h:        &handler{config: &config.Config{LookupSource: "geojs"}},
+		client:   netip.MustParseAddr("203.0.113.9"),
+		lookedUp: true,
+	}
+
+	defer func() {
+		if r := recover(); r != nil {
+			t.Errorf("counting the request did work, with every threshold off: %v", r)
+		}
+	}()
+
+	rq.countAnomalies()
+}

internal/proxy/anomaly_test.go

@@ -0,0 +1,377 @@
+package proxy_test
+
+import (
+	"maps"
+	"net/http"
+	"net/netip"
+	"reflect"
+	"slices"
+	"strconv"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"sneak.berlin/go/smallwebwaf/internal/alerts"
+	"sneak.berlin/go/smallwebwaf/internal/anomaly"
+	"sneak.berlin/go/smallwebwaf/internal/proxy"
+	"sneak.berlin/go/smallwebwaf/internal/ratelimit"
+	"sneak.berlin/go/smallwebwaf/internal/requestlog"
+)
+
+// The anomaly thresholds: the prefix of a scope followed by the end of a
+// count.
+const (
+	anomalyClient = "SWWAF_ANOMALY_CLIENT_"
+	anomalyNet    = "SWWAF_ANOMALY_NET_"
+	anomalyASN    = "SWWAF_ANOMALY_ASN_"
+	anomalyTotal  = "SWWAF_ANOMALY_TOTAL_"
+	anomalyWatch  = "SWWAF_WATCH_"
+
+	requestsPerMinute = "REQUESTS_PER_MINUTE"
+	requestsPerHour   = "REQUESTS_PER_HOUR"
+	bytesPerMinute    = "BYTES_PER_MINUTE"
+	bytesPerHour      = "BYTES_PER_HOUR"
+)
+
+// The other anomaly settings.
+const (
+	anomalyNetV4Prefix = "SWWAF_ANOMALY_NET_V4_PREFIX"
+	anomalyNetV6Prefix = "SWWAF_ANOMALY_NET_V6_PREFIX"
+	watchNets          = "SWWAF_WATCH_NETS"
+)
+
+const (
+	// clientsNet is the netblock around client at the default length, and
+	// office a named netblock of the same.
+	clientsNet = "203.0.113.0/24"
+	office     = "office=" + clientsNet
+	// aLot is a threshold no test reaches.
+	aLot = "1000"
+	// hour is the window an alert names for a threshold per hour.
+	hour = "hour"
+)
+
+func TestEachScopeAndWindowOverItsThresholdAlertsOncePerCooldown(t *testing.T) {
+	t.Parallel()
+
+	for _, scope := range []struct {
+		prefix, scope string
+		// netblock is the alert's, and counted what its reason names. extra
+		// is what its detail gives besides what every anomaly alert's does.
+		netblock netip.Prefix
+		counted  string
+		extra    map[string]any
+	}{
+		{
+			anomalyClient, anomaly.ScopeClient, netip.MustParsePrefix(client + "/32"),
+			"the client " + client + "/32", nil,
+		},
+		{
+			anomalyNet, anomaly.ScopeNet, netip.MustParsePrefix(clientsNet),
+			"the netblock " + clientsNet, nil,
+		},
+		{anomalyASN, anomaly.ScopeASN, netip.Prefix{}, asnDE, map[string]any{"asn": asnDE}},
+		{anomalyTotal, anomaly.ScopeTotal, netip.Prefix{}, "the whole service", nil},
+		{
+			anomalyWatch, anomaly.ScopeWatch, netip.MustParsePrefix(clientsNet),
+			"the named netblock office, " + clientsNet, map[string]any{"name": "office"},
+		},
+	} {
+		for _, threshold := range []struct {
+			end, kind, window string
+			// value is the threshold, which the third upload of 100 bytes
+			// takes the count over, to count.
+			value int64
+			count float64
+		}{
+			{requestsPerMinute, ratelimit.KindRequests, minute, 2, 3},
+			{requestsPerHour, ratelimit.KindRequests, hour, 2, 3},
+			{bytesPerMinute, ratelimit.KindBytes, minute, 250, 300},
+			{bytesPerHour, ratelimit.KindBytes, hour, 250, 300},
+		} {
+			setting := scope.prefix + threshold.end
+			value := strconv.FormatInt(threshold.value, 10)
+
+			t.Run(setting, func(t *testing.T) {
+				t.Parallel()
+
+				s, clk, server, queue := startWithLookupsAndClock(t, map[string]string{
+					setting: value, watchNets: office,
+				})
+				start := clk.Now()
+
+				// The third upload takes the count over the threshold, and the
+				// fourth, within the cooldown, is held back. Each is passed to
+				// the app.
+				for range 4 {
+					s.uploadFrom(client)
+				}
+
+				detail := map[string]any{
+					"scope": scope.scope, "window": threshold.window, "kind": threshold.kind,
+					"count": threshold.count, "threshold": threshold.value,
+				}
+				maps.Copy(detail, scope.extra)
+
+				wantAlerts(t, queue, alerts.Alert{
+					Instance: alertInstance,
+					Time:     start,
+					Event:    alerts.EventAnomaly,
+					Client:   netip.MustParseAddr(client),
+					Netblock: scope.netblock,
+					ASN:      asnDE,
+					ASName:   asNameDE,
+					Country:  "DE",
+					Reason: threshold.kind + " per " + threshold.window + " of " +
+						scope.counted + " over the threshold of " + value,
+					Detail: detail,
+				})
+
+				wantAlertedAgainOnceTheCooldownHasRunOut(t, s, clk, queue)
+
+				if held := server.Ledger.Snapshot(); len(held) != 0 {
+					t.Errorf("the ledger holds %+v, want no ban", held)
+				}
+			})
+		}
+	}
+}
+
+// wantAlertedAgainOnceTheCooldownHasRunOut checks that, once the cooldown
+// has run out after a first alert, which held back one repeat, the next
+// count over the threshold, at the latest three uploads from client on,
+// raises another alert, giving that repeat.
+func wantAlertedAgainOnceTheCooldownHasRunOut(
+	t *testing.T, s *sender, clk *clock, queue *alerts.Queue,
+) {
+	t.Helper()
+
+	clk.advance(15 * time.Minute)
+
+	for range 3 {
+		s.uploadFrom(client)
+	}
+
+	waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
+	if len(waiting) != 2 || !waiting[1].Time.Equal(clk.Now()) ||
+		waiting[1].SuppressedRepeats != 1 {
+		t.Errorf("alerts wait %+v, want the first and another, with 1 repeat", waiting)
+	}
+}
+
+func TestEveryRequestIsCountedWhateverIsDoneWithIt(t *testing.T) {
+	t.Parallel()
+
+	const (
+		allowed = "192.0.2.7"  // in SWWAF_ALLOW_NETS
+		exempt  = "192.0.2.10" // in SWWAF_RATE_LIMIT_EXEMPT_NETS
+		denied  = "192.0.2.20" // in SWWAF_DENY_NETS
+	)
+
+	s, _, _, queue := startAppWithAlerts(t, readAndAnswer, map[string]string{
+		anomalyClient + requestsPerMinute: "2",
+		allowNets:                         allowed,
+		rateLimitExemptNets:               exempt,
+		rateLimitExemptPaths:              "/static/",
+		denyNets:                          denied,
+	})
+
+	// The third request of each takes its client's count over the threshold
+	// of 2.
+	for _, sent := range []struct {
+		from, path string
+		status     int
+		action     string
+	}{
+		{allowed, "/", http.StatusOK, requestlog.ActionForward},
+		{exempt, "/", http.StatusOK, requestlog.ActionForward},
+		{client, "/static/app.js", http.StatusOK, requestlog.ActionForward},
+		{denied, "/", http.StatusForbidden, requestlog.ActionDenied},
+	} {
+		for range 3 {
+			s.request(sent.from, sent.path, sent.status, sent.action)
+		}
+	}
+
+	waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
+
+	got := make([]string, 0, len(waiting))
+	for _, alert := range waiting {
+		got = append(got, alert.Client.String())
+	}
+
+	if want := []string{allowed, exempt, client, denied}; !slices.Equal(got, want) {
+		t.Errorf("alerts for the clients %v, want %v", got, want)
+	}
+}
+
+func TestThresholdsOffCountNothingAndAlertNothing(t *testing.T) {
+	t.Parallel()
+
+	// With every threshold off, nothing is counted.
+	s, server, queue := startWithLookups(t, map[string]string{watchNets: office})
+
+	for range 5 {
+		s.uploadFrom(client)
+	}
+
+	if counters := server.Anomalies.Snapshot(); len(counters) != 0 {
+		t.Errorf("counters %+v, want none", counters)
+	}
+
+	wantAlerts(t, queue)
+
+	// With one set, its count alone is counted, in its scope alone.
+	s, clk, server, queue := startWithLookupsAndClock(t, map[string]string{
+		anomalyNet + requestsPerMinute: aLot, watchNets: office,
+	})
+
+	for range 5 {
+		s.uploadFrom(client)
+	}
+
+	want := []anomaly.Counter{{
+		Scope:    anomaly.ScopeNet,
+		Netblock: netip.MustParsePrefix(clientsNet),
+		Minute:   ratelimit.Buckets{Start: clk.Now(), Current: 5},
+	}}
+	if got := server.Anomalies.Snapshot(); !reflect.DeepEqual(got, want) {
+		t.Errorf("counters\n%+v\nwant\n%+v", got, want)
+	}
+
+	wantAlerts(t, queue)
+}
+
+func TestNetblockAroundAClientIsAsLongAsTheSettingsSay(t *testing.T) {
+	t.Parallel()
+
+	for _, tc := range []struct {
+		name string
+		env  map[string]string
+		// Each client of sent sends one request, and want gives the
+		// netblocks they are counted in, each with its requests.
+		sent []string
+		want map[string]int64
+	}{
+		{
+			"by default", nil,
+			[]string{client, "203.0.113.200", "192.0.2.7", ipv6Client, "2001:db8:0:ffff::1"},
+			map[string]int64{clientsNet: 2, "192.0.2.0/24": 1, "2001:db8::/48": 2},
+		},
+		{
+			"as set", map[string]string{anomalyNetV4Prefix: "16", anomalyNetV6Prefix: "32"},
+			[]string{client, "203.0.200.1", ipv6Client, "2001:db8:ffff::1"},
+			map[string]int64{"203.0.0.0/16": 2, "2001:db8::/32": 2},
+		},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+
+			env := map[string]string{anomalyNet + requestsPerMinute: aLot}
+			maps.Copy(env, tc.env)
+
+			s, _, server, _ := startAppWithAlerts(t, readAndAnswer, env)
+
+			for _, from := range tc.sent {
+				s.get(from, http.StatusOK, requestlog.ActionForward)
+			}
+
+			got := map[string]int64{}
+			for _, counter := range server.Anomalies.Snapshot() {
+				got[counter.Netblock.String()] = counter.Minute.Current
+			}
+
+			if !maps.Equal(got, tc.want) {
+				t.Errorf("requests by netblock %v, want %v", got, tc.want)
+			}
+		})
+	}
+}
+
+func TestClientIsCountedForItsASNumberOnceTheLookupGivesOne(t *testing.T) {
+	t.Parallel()
+
+	s, server, _ := startWithLookups(t, map[string]string{
+		anomalyASN + requestsPerMinute: aLot,
+	})
+
+	// The lookup database does not hold unplaced.
+	for _, from := range []string{fromDE, fromDE, fromKP, noCountry, unplaced} {
+		s.uploadFrom(from)
+	}
+
+	got := map[string]int64{}
+	for _, counter := range server.Anomalies.Snapshot() {
+		got[counter.ASN] = counter.Minute.Current
+	}
+
+	if want := map[string]int64{asnDE: 2, asnKP: 1, "AS64500": 1}; !maps.Equal(got, want) {
+		t.Errorf("requests by AS number %v, want %v", got, want)
+	}
+}
+
+func TestRequestCountsForTheASNumberGeoJSGivesBeforeItEnds(t *testing.T) {
+	t.Parallel()
+
+	// The stand-in for GeoJS answers only once released, which the app
+	// does as it answers the request, and then waits until the answer is
+	// kept.
+	geojsURL, _, release := startHeldGeoJS(t)
+
+	var server atomic.Pointer[proxy.Server]
+
+	app := startApp(t, func(http.ResponseWriter, *http.Request) {
+		release()
+		waitUntil(func() bool {
+			_, kept := server.Load().GeoJS.Kept(netip.MustParsePrefix(fromDE + "/32"))
+
+			return kept
+		})
+	})
+	clk := &clock{now: time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)}
+	addr, out, started := startProxyWithClock(t, app.URL, geojsURL, clk.Now,
+		map[string]string{
+			trustedProxies:                 trustLocalhost,
+			lookupTimeout:                  "1h",
+			anomalyASN + requestsPerMinute: aLot,
+		})
+	server.Store(started)
+
+	// The request went on without the answer, and is counted for the AS
+	// number it gives.
+	s := &sender{t: t, addr: addr, out: out}
+	if line := s.get(fromDE, http.StatusOK, requestlog.ActionForward); line.ASN != "" {
+		t.Errorf("log line has AS number %q, want none: the request waited", line.ASN)
+	}
+
+	want := []anomaly.Counter{{
+		Scope: anomaly.ScopeASN, ASN: asnDE,
+		Minute: ratelimit.Buckets{Start: clk.Now(), Current: 1},
+	}}
+	if got := started.Anomalies.Snapshot(); !reflect.DeepEqual(got, want) {
+		t.Errorf("counters\n%+v\nwant\n%+v", got, want)
+	}
+}
+
+func TestEachNamedNetblockCountsTheClientsInIt(t *testing.T) {
+	t.Parallel()
+
+	s, _, server, _ := startAppWithAlerts(t, readAndAnswer, map[string]string{
+		anomalyWatch + requestsPerMinute: aLot,
+		watchNets:                        office + ",wide=203.0.0.0/16,other=198.51.100.0/25",
+	})
+
+	// client is in office and in wide.
+	for _, from := range []string{client, "203.0.200.1", "192.0.2.7"} {
+		s.get(from, http.StatusOK, requestlog.ActionForward)
+	}
+
+	got := map[string]int64{}
+	for _, counter := range server.Anomalies.Snapshot() {
+		got[counter.Name] = counter.Minute.Current
+	}
+
+	if want := map[string]int64{"office": 1, "wide": 2}; !maps.Equal(got, want) {
+		t.Errorf("requests by named netblock %v, want %v", got, want)
+	}
+}

internal/proxy/bans.go

@@ -42,9 +42,11 @@ func (rq *request) banned(now time.Time) bool {
 
 // limitBroken counts the request for the rate limits at now, notes the
 // client's counts for the log line, and reports whether the request takes
-// the client over a rate limit, which breaks it.
+// the client over a rate limit, as its limit percentage lowers it, which
+// breaks it.
 func (rq *request) limitBroken(now time.Time) bool {
-	counts, hit, over := rq.h.limiter.Count(clientGroup(rq.client), now)
+	counts, hit, over := rq.h.limiter.Count(clientGroup(rq.client), now,
+		rq.limitPercent.percent)
 	rq.line.Counts = counts
 
 	if over {
@@ -54,42 +56,23 @@ func (rq *request) limitBroken(now time.Time) bool {
 	return over
 }
 
-// countBytes counts the request's bytes for the byte limits, once its
-// response has ended, and notes the client's byte totals for the log line;
-// its requests stay there as the rate limits counted them. The bytes are
-// the response's body bytes, the request's, or both, as SWWAF_BYTES_COUNT
-// says; for an upgraded connection, such as a WebSocket, which has closed
-// by then, what it carried from the app counts with the response's and
-// what it carried from the client with the request's. Only a request
-// passed to the app has them counted, and only one the rate limits
-// counted; in observe mode, not one that enforce mode would have refused.
-// Bytes that take the client over a byte limit break it; the response was
-// passed on whole.
+// countBytes counts the request's bytes, as countedBytes gives them, for
+// the byte limits, once its response has ended, and notes the client's
+// byte totals for the log line; its requests stay there as the rate limits
+// counted them. Only a request passed to the app has them counted, and
+// only one the rate limits counted; in observe mode, not one that enforce
+// mode would have refused. Bytes that take the client over a byte limit,
+// as its limit percentage for the byte limits lowers it, break it; the
+// response was passed on whole.
 func (rq *request) countBytes() {
 	if !rq.counted || rq.line.WouldAction != "" {
 		return
 	}
 
-	response, request := rq.out.bytes, rq.requestBytes()
-	if rq.upgraded != nil {
-		response += rq.upgraded.fromApp.Load()
-		request += rq.upgraded.toApp.Load()
-	}
-
-	var bytes int64
-
-	switch rq.h.config.BytesCount {
-	case "response":
-		bytes = response
-	case "request":
-		bytes = request
-	default: // both
-		bytes = response + request
-	}
-
 	now := rq.h.now()
 
-	counts, hit, over := rq.h.limiter.CountBytes(clientGroup(rq.client), now, bytes)
+	counts, hit, over := rq.h.limiter.CountBytes(clientGroup(rq.client), now,
+		rq.countedBytes(), rq.bytesPercent.percent)
 	rq.line.Counts.MinuteBytes = counts.MinuteBytes
 	rq.line.Counts.HourBytes = counts.HourBytes
 	rq.line.Counts.DayBytes = counts.DayBytes
@@ -99,13 +82,37 @@ func (rq *request) countBytes() {
 	}
 }
 
+// countedBytes returns the request's bytes, once it has ended, as the
+// byte limits and the anomaly thresholds count them: the response's body
+// bytes, the request's, or both, as SWWAF_BYTES_COUNT says. For an
+// upgraded connection, such as a WebSocket, which has closed by then, what
+// it carried from the app counts with the response's and what it carried
+// from the client with the request's.
+func (rq *request) countedBytes() int64 {
+	response, request := rq.out.bytes, rq.requestBytes()
+	if rq.upgraded != nil {
+		response += rq.upgraded.fromApp.Load()
+		request += rq.upgraded.toApp.Load()
+	}
+
+	switch rq.h.config.BytesCount {
+	case "response":
+		return response
+	case "request":
+		return request
+	default: // both
+		return response + request
+	}
+}
+
 // banForLimit bans the client's netblock at now for a broken limit, the
 // one hit names, and notes the offence for the log line. status is what
 // the client was sent, or is sent: SWWAF_BAN_RESPONSE for a request over
 // a rate limit, the app's answer for one whose bytes broke a byte limit.
-// The ban sets the client's counters back to zero. In observe mode it
-// makes no ban and sets nothing back, and raises the alert for the ban it
-// would have made, if that alert would be sent.
+// The ban's notes give the client's limit percentage for that kind of
+// limit. The ban sets the client's counters back to zero. In observe mode
+// it makes no ban and sets nothing back, and raises the alert for the ban
+// it would have made, if that alert would be sent.
 func (rq *request) banForLimit(now time.Time, hit ratelimit.Hit, status int) {
 	rq.line.LimitHit = hit.Window
 	if hit.Kind == ratelimit.KindBytes {
@@ -131,6 +138,13 @@ func (rq *request) banForLimit(now time.Time, hit ratelimit.Hit, status int) {
 		Requests: rq.netblockRequests(netblock),
 	}
 
+	percent := rq.limitPercent
+	if hit.Kind == ratelimit.KindBytes {
+		percent = rq.bytesPercent
+	}
+
+	notes.LimitPercent, notes.LimitPercentSetting = percent.logged()
+
 	if rq.h.config.Observe {
 		ban, wouldBan := rq.h.ledger.WouldBanForLimit(netblock, now, notes)
 		if wouldBan {

internal/proxy/biased.go

@@ -0,0 +1,111 @@
+package proxy
+
+import (
+	"sneak.berlin/go/smallwebwaf/internal/config"
+)
+
+// whole is the percentage of each limit a client gets when no biased
+// threshold lowers its limits.
+const whole = 100
+
+// percentage is a client's limit percentage for the rate limits or for
+// the byte limits, as the biased thresholds give it, and the setting that
+// gave it: "" with whole when none lowers that kind of limit.
+type percentage struct {
+	percent int64
+	setting string
+}
+
+// biasedThresholdsSet reports whether a biased threshold can lower a
+// client's limits: one of its lists is not empty,
+// SWWAF_UNKNOWN_LIMIT_PERCENT is below 100, or SWWAF_ASN_LIMIT_PERCENT_URL
+// is set. The client's lookup is then needed before its request goes on.
+func biasedThresholdsSet(cfg *config.Config) bool {
+	return len(cfg.ASNLimitPercent) > 0 || len(cfg.CountryLimitPercent) > 0 ||
+		len(cfg.ASNBytesPercent) > 0 || len(cfg.CountryBytesPercent) > 0 ||
+		cfg.UnknownLimitPercent < whole || cfg.ASNLimitPercentURL != ""
+}
+
+// limitPercentages returns the client's limit percentages, for the rate
+// limits and for the byte limits, by its AS number and country as looked
+// up, each "" when unknown, and the blocklists that list it. Each is the
+// lowest of those the settings give it, the first of them in the order
+// below when several are lowest: the percentage SWWAF_ASN_LIMIT_PERCENT
+// gives its AS number, the one the file SWWAF_ASN_LIMIT_PERCENT_URL names
+// gives it, the one SWWAF_COUNTRY_LIMIT_PERCENT gives its country, for a
+// client without a country, SWWAF_UNKNOWN_LIMIT_PERCENT, and for a client
+// a blocklist lists, the percentage of SWWAF_BLOCKLIST_ACTION while it is
+// limit. For the byte limits, SWWAF_ASN_BYTES_PERCENT and
+// SWWAF_COUNTRY_BYTES_PERCENT take the place of the first three for an AS
+// number or a country they list.
+func (rq *request) limitPercentages() (percentage, percentage) {
+	cfg := rq.h.config
+	asn, country := rq.line.ASN, rq.line.Country
+
+	unknown := percentage{percent: whole}
+	if country == "" {
+		unknown = percentage{cfg.UnknownLimitPercent, "SWWAF_UNKNOWN_LIMIT_PERCENT"}
+	}
+
+	fetched := percentage{percent: whole}
+	if percent, listed := rq.h.lists.ASNLimitPercent(asn); listed {
+		fetched = percentage{percent, "SWWAF_ASN_LIMIT_PERCENT_URL"}
+	}
+
+	listed := percentage{percent: whole}
+	if len(rq.line.Reputation) > 0 && cfg.BlocklistAction == "limit" {
+		listed = percentage{cfg.BlocklistLimitPercent, "SWWAF_BLOCKLIST_ACTION"}
+	}
+
+	asnRequests := lowest(given(cfg.ASNLimitPercent, asn, "SWWAF_ASN_LIMIT_PERCENT"),
+		fetched)
+	countryRequests := given(cfg.CountryLimitPercent, country,
+		"SWWAF_COUNTRY_LIMIT_PERCENT")
+
+	asnBytes, countryBytes := asnRequests, countryRequests
+	if _, listed := cfg.ASNBytesPercent[asn]; listed {
+		asnBytes = given(cfg.ASNBytesPercent, asn, "SWWAF_ASN_BYTES_PERCENT")
+	}
+
+	if _, listed := cfg.CountryBytesPercent[country]; listed {
+		countryBytes = given(cfg.CountryBytesPercent, country, "SWWAF_COUNTRY_BYTES_PERCENT")
+	}
+
+	return lowest(asnRequests, countryRequests, unknown, listed),
+		lowest(asnBytes, countryBytes, unknown, listed)
+}
+
+// given returns the percentage percents, the setting named setting, gives
+// code, an AS number or a country, or whole when it does not list code.
+func given(percents map[string]int64, code, setting string) percentage {
+	percent, listed := percents[code]
+	if !listed {
+		return percentage{percent: whole}
+	}
+
+	return percentage{percent, setting}
+}
+
+// lowest returns the lowest of percentages below whole, the first of them
+// when several are lowest, or whole when none is below it.
+func lowest(percentages ...percentage) percentage {
+	low := percentage{percent: whole}
+
+	for _, p := range percentages {
+		if p.percent < low.percent {
+			low = p
+		}
+	}
+
+	return low
+}
+
+// logged returns p as the log line and the notes of a ban give it: its
+// percent and setting, or nil and "" for whole, which they leave out.
+func (p percentage) logged() (*int64, string) {
+	if p.percent == whole {
+		return nil, ""
+	}
+
+	return &p.percent, p.setting
+}

internal/proxy/biased_test.go

@@ -0,0 +1,504 @@
+package proxy_test
+
+import (
+	"fmt"
+	"io"
+	"maps"
+	"net"
+	"net/http"
+	"net/http/httptest"
+	"net/netip"
+	"path/filepath"
+	"strings"
+	"testing"
+	"testing/synctest"
+	"time"
+
+	"sneak.berlin/go/smallwebwaf/internal/alerts"
+	"sneak.berlin/go/smallwebwaf/internal/bans"
+	"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
+	"sneak.berlin/go/smallwebwaf/internal/proxy"
+	"sneak.berlin/go/smallwebwaf/internal/requestlog"
+)
+
+// The biased thresholds.
+const (
+	asnLimitPercent     = "SWWAF_ASN_LIMIT_PERCENT"
+	countryLimitPercent = "SWWAF_COUNTRY_LIMIT_PERCENT"
+	asnBytesPercent     = "SWWAF_ASN_BYTES_PERCENT"
+	countryBytesPercent = "SWWAF_COUNTRY_BYTES_PERCENT"
+	unknownLimitPercent = "SWWAF_UNKNOWN_LIMIT_PERCENT"
+)
+
+const (
+	// asnDEHalf and countryDEHalf give fromDE's AS number and its country
+	// half of every limit, and asnDEQuarter gives its AS number a quarter.
+	asnDEHalf     = asnDE + ":50"
+	asnDEQuarter  = asnDE + ":25"
+	countryDEHalf = "de:50"
+	// noCountry is in an AS of its own, AS64500, and in no country.
+	noCountry = "192.0.2.80"
+	// fourAMinute is the rate limit these tests set: half of it is 2
+	// requests a minute, a quarter of it 1.
+	fourAMinute = "4"
+	// twoUploads is the byte limit these tests set: 199 bytes, which an
+	// upload, a request with a body and its answer, 100 bytes, is within,
+	// and half of which, 99 bytes, it is over.
+	twoUploads = "199"
+	// none is how percentText gives a percentage left out.
+	none = "none"
+)
+
+func TestEachBiasedThresholdLowersTheRateLimits(t *testing.T) {
+	t.Parallel()
+
+	for _, tc := range []struct {
+		setting, value, from string
+	}{
+		{asnLimitPercent, asnDEHalf, fromDE},
+		{countryLimitPercent, countryDEHalf, fromDE},
+		{unknownLimitPercent, "50", unplaced},
+	} {
+		t.Run(tc.setting, func(t *testing.T) {
+			t.Parallel()
+
+			s, _, _ := startWithLookups(t, map[string]string{
+				rateLimitPerMinute: fourAMinute, tc.setting: tc.value,
+			})
+
+			// Half of 4 requests a minute: the third breaks the limit.
+			for _, sent := range []struct {
+				status int
+				action string
+			}{
+				{http.StatusOK, requestlog.ActionForward},
+				{http.StatusOK, requestlog.ActionForward},
+				{http.StatusForbidden, requestlog.ActionRateLimited},
+			} {
+				line := s.get(tc.from, sent.status, sent.action)
+				wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
+					"50 from "+tc.setting)
+			}
+
+			// fromKP, which no setting lists, has the whole limit.
+			for range 3 {
+				line := s.get(fromKP, http.StatusOK, requestlog.ActionForward)
+				wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
+					none)
+			}
+		})
+	}
+}
+
+func TestEachBiasedThresholdLowersTheByteLimits(t *testing.T) {
+	t.Parallel()
+
+	// The AS numbers and countries are given in either case.
+	for _, tc := range []struct {
+		setting, value, from string
+	}{
+		{asnLimitPercent, asnDEHalf, fromDE},
+		{countryLimitPercent, "DE:50", fromDE},
+		{unknownLimitPercent, "50", unplaced},
+		{asnBytesPercent, "as64496:50", fromDE},
+		{countryBytesPercent, countryDEHalf, fromDE},
+	} {
+		t.Run(tc.setting, func(t *testing.T) {
+			t.Parallel()
+
+			s, _, _ := startWithLookups(t, map[string]string{
+				bytesLimitPerMinute: twoUploads, tc.setting: tc.value,
+			})
+
+			// The upload's 100 bytes are over half of 199, 99.
+			line := s.uploadFrom(tc.from)
+			if line.LimitHit != minuteBytes {
+				t.Errorf("log line has limit_hit %q, want %s", line.LimitHit, minuteBytes)
+			}
+
+			wantPercent(t, "bytes_percent", line.BytesPercent, line.BytesPercentSetting,
+				"50 from "+tc.setting)
+
+			// fromKP, which no setting lists, has the whole limit.
+			line = s.uploadFrom(fromKP)
+			if line.LimitHit != "" {
+				t.Errorf("log line for %s has limit_hit %q, want none", fromKP, line.LimitHit)
+			}
+
+			wantPercent(t, "bytes_percent", line.BytesPercent, line.BytesPercentSetting, none)
+		})
+	}
+}
+
+func TestBytesPercentSettingsTakeThePlaceOfTheOthersForByteLimits(t *testing.T) {
+	t.Parallel()
+
+	for _, tc := range []struct {
+		name string
+		env  map[string]string
+		// limitPercent and bytesPercent are the log line's, as percentText
+		// gives them, and limitHit is its limit_hit.
+		limitPercent, bytesPercent, limitHit string
+	}{
+		{
+			"lowering the byte limits alone",
+			map[string]string{asnBytesPercent: asnDEHalf},
+			none, "50 from " + asnBytesPercent, minuteBytes,
+		},
+		{
+			"lowering the byte limits alone, by country",
+			map[string]string{countryBytesPercent: countryDEHalf},
+			none, "50 from " + countryBytesPercent, minuteBytes,
+		},
+		{
+			"raising the byte limits back",
+			map[string]string{asnLimitPercent: asnDEHalf, asnBytesPercent: asnDE + ":100"},
+			"50 from " + asnLimitPercent, none, "",
+		},
+		{
+			"raising the byte limits back, by country",
+			map[string]string{countryLimitPercent: countryDEHalf, countryBytesPercent: "de:100"},
+			"50 from " + countryLimitPercent, none, "",
+		},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+
+			env := map[string]string{bytesLimitPerMinute: twoUploads}
+			maps.Copy(env, tc.env)
+			s, _, _ := startWithLookups(t, env)
+
+			// The upload's 100 bytes are over 99, half of 199, and within 199.
+			line := s.uploadFrom(fromDE)
+			if line.LimitHit != tc.limitHit {
+				t.Errorf("log line has limit_hit %q, want %q", line.LimitHit, tc.limitHit)
+			}
+
+			wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
+				tc.limitPercent)
+			wantPercent(t, "bytes_percent", line.BytesPercent, line.BytesPercentSetting,
+				tc.bytesPercent)
+		})
+	}
+}
+
+func TestZeroPercentIsAZeroAllowance(t *testing.T) {
+	t.Parallel()
+
+	s, _, _ := startWithLookups(t, map[string]string{asnLimitPercent: asnDE + ":0"})
+
+	// The first request breaks the limit, and bans the client; the log line
+	// gives the 0.
+	line := s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited)
+	if line.fields["limit_percent"] != float64(0) ||
+		line.fields["limit_percent_setting"] != asnLimitPercent {
+		t.Errorf("log line has limit_percent %v from %v, want 0 from %s",
+			line.fields["limit_percent"], line.fields["limit_percent_setting"],
+			asnLimitPercent)
+	}
+
+	s.get(fromDE, http.StatusForbidden, requestlog.ActionBanned)
+}
+
+func TestLowestPercentageApplies(t *testing.T) {
+	t.Parallel()
+
+	for _, tc := range []struct {
+		name string
+		env  map[string]string
+		from string
+		// want is the log line's limit_percent, as percentText gives it.
+		want string
+	}{
+		{
+			"the country's",
+			map[string]string{asnLimitPercent: asnDEHalf, countryLimitPercent: "de:25"},
+			fromDE, "25 from " + countryLimitPercent,
+		},
+		{
+			"the AS number's",
+			map[string]string{asnLimitPercent: asnDEQuarter, countryLimitPercent: countryDEHalf},
+			fromDE, "25 from " + asnLimitPercent,
+		},
+		{
+			"the AS number's, the first of two alike",
+			map[string]string{asnLimitPercent: asnDEQuarter, countryLimitPercent: "de:25"},
+			fromDE, "25 from " + asnLimitPercent,
+		},
+		{
+			"that for a client without a country",
+			map[string]string{asnLimitPercent: "AS64500:50", unknownLimitPercent: "25"},
+			noCountry, "25 from " + unknownLimitPercent,
+		},
+		{
+			// SWWAF_UNKNOWN_LIMIT_PERCENT is left at its default, 100.
+			"the AS number's, for a client without a country",
+			map[string]string{asnLimitPercent: "AS64500:25"},
+			noCountry, "25 from " + asnLimitPercent,
+		},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+
+			env := map[string]string{rateLimitPerMinute: fourAMinute}
+			maps.Copy(env, tc.env)
+			s, _, _ := startWithLookups(t, env)
+
+			// A quarter of 4 requests a minute: the second breaks the limit.
+			s.get(tc.from, http.StatusOK, requestlog.ActionForward)
+
+			line := s.get(tc.from, http.StatusForbidden, requestlog.ActionRateLimited)
+			wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
+				tc.want)
+		})
+	}
+}
+
+func TestUnknownLimitPercentGivesEveryClientWithoutACountryItsPercentage(t *testing.T) {
+	t.Parallel()
+
+	s, _, _ := startWithLookups(t, map[string]string{
+		rateLimitPerMinute: fourAMinute, unknownLimitPercent: "50",
+	})
+
+	// One the lookup database does not hold, and one on a private address,
+	// which is never looked up: the third request of each breaks half of 4.
+	for _, from := range []string{unplaced, "10.0.0.8"} {
+		s.get(from, http.StatusOK, requestlog.ActionForward)
+		s.get(from, http.StatusOK, requestlog.ActionForward)
+		s.get(from, http.StatusForbidden, requestlog.ActionRateLimited)
+	}
+
+	// One in a country has the whole limit.
+	for range 3 {
+		s.get(fromDE, http.StatusOK, requestlog.ActionForward)
+	}
+}
+
+func TestClientWithoutAnAnswerInTimeHasTheUnknownLimitPercent(t *testing.T) {
+	t.Parallel()
+
+	// In a synctest bubble, as TestRequestWaitsAsLongAsTheLookupTimeoutSays
+	// says, with a GeoJS that never answers.
+	synctest.Test(t, func(t *testing.T) {
+		server, out, _ := newProxy(t, "http://app.invalid", unansweredGeoJSURL,
+			time.Now, map[string]string{unknownLimitPercent: "0"})
+
+		// Once the second the request waits for its answer is up, the client
+		// counts as without a country, and its zero allowance refuses the
+		// request before it reaches the app.
+		serveFromDE(t, server, http.MethodGet, http.NoBody)
+
+		line := out.requestLine(t)
+		wantLine(t, line, http.StatusForbidden, requestlog.ActionRateLimited)
+		wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
+			"0 from "+unknownLimitPercent)
+	})
+}
+
+func TestRequestWaitsForItsLookupWhileABiasedThresholdIsSet(t *testing.T) {
+	t.Parallel()
+
+	const timeout = 3 * time.Second
+
+	for _, tc := range []struct {
+		setting, value string
+		waits          bool
+	}{
+		{asnLimitPercent, asnDEHalf, true},
+		{countryLimitPercent, countryDEHalf, true},
+		{asnBytesPercent, asnDEHalf, true},
+		{countryBytesPercent, countryDEHalf, true},
+		{unknownLimitPercent, "99", true},
+		{asnLimitPercentURL, asnURL, true},
+		// At 100, its default, it lowers no limit.
+		{unknownLimitPercent, "100", false},
+	} {
+		t.Run(tc.setting+"="+tc.value, func(t *testing.T) {
+			t.Parallel()
+
+			// In a synctest bubble, as TestRequestWaitsAsLongAsTheLookupTimeoutSays
+			// says, with a GeoJS that never answers.
+			synctest.Test(t, func(t *testing.T) {
+				// The request's body is over SWWAF_REQUEST_MAX_BYTES, so that it
+				// is refused after the checks, and never reaches the app.
+				server, out, _ := newProxy(t, "http://app.invalid", unansweredGeoJSURL,
+					time.Now, map[string]string{
+						lookupTimeout: timeout.String(), requestMaxBytes: "1",
+						tc.setting: tc.value,
+					})
+				began := time.Now()
+
+				serveFromDE(t, server, http.MethodPost, strings.NewReader("ab"))
+
+				want := time.Duration(0)
+				if tc.waits {
+					want = timeout
+				}
+
+				if waited := time.Since(began); waited != want {
+					t.Errorf("the request waited %s for its answer, want %s", waited, want)
+				}
+
+				wantLine(t, out.requestLine(t), http.StatusRequestEntityTooLarge,
+					requestlog.ActionTooLarge)
+
+				// The bubble's clock stops once this function returns, so the
+				// request to GeoJS, which a request that did not wait leaves
+				// under way, has to be abandoned before then.
+				time.Sleep(timeout)
+			})
+		})
+	}
+}
+
+func TestBanForALoweredLimitGivesThePercentageInItsNotesAndItsAlert(t *testing.T) {
+	t.Parallel()
+
+	for _, tc := range []struct {
+		name string
+		env  map[string]string
+		// before is how many uploads come before the one that breaks a
+		// limit, which is answered with status and logged with action.
+		before int
+		status int
+		action string
+		// reason and want are the ban's reason, and its notes' limit
+		// percentage, as percentText gives it.
+		reason, want string
+	}{
+		{
+			// A quarter of 12 requests a minute is 3: the fourth breaks it.
+			"a rate limit",
+			map[string]string{rateLimitPerMinute: "12", asnLimitPercent: asnDEQuarter},
+			3, http.StatusForbidden, requestlog.ActionRateLimited,
+			"requests per minute over the limit of 3", "25 from " + asnLimitPercent,
+		},
+		{
+			// The byte limits' percentage, not the rate limits'.
+			"a byte limit",
+			map[string]string{
+				bytesLimitPerMinute: twoUploads, asnLimitPercent: asnDEQuarter,
+				asnBytesPercent: asnDEHalf,
+			},
+			0, http.StatusOK, requestlog.ActionForward,
+			"bytes per minute over the limit of 99", "50 from " + asnBytesPercent,
+		},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+
+			s, server, queue := startWithLookups(t, tc.env)
+
+			for range tc.before {
+				s.uploadFrom(fromDE)
+			}
+
+			s.requestWithBody(http.MethodPost, fromDE, "/", uploadHeader, uploadBody,
+				tc.status, tc.action)
+
+			held := server.Ledger.Bans(netip.MustParsePrefix(fromDE + "/32"))
+			if len(held) != 1 {
+				t.Fatalf("bans %+v, want one", held)
+			}
+
+			notes := held[0].Notes
+			if held[0].Reason != tc.reason {
+				t.Errorf("the ban's reason is %q, want %q", held[0].Reason, tc.reason)
+			}
+
+			wantPercent(t, "the notes' limit_percent", notes.LimitPercent,
+				notes.LimitPercentSetting, tc.want)
+
+			waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
+			if len(waiting) != 1 {
+				t.Fatalf("%d alerts wait, want the ban's alone: %+v", len(waiting), waiting)
+			}
+
+			alerted, _ := waiting[0].Detail["notes"].(bans.Notes)
+			wantPercent(t, "the alert's notes' limit_percent", alerted.LimitPercent,
+				alerted.LimitPercentSetting, tc.want)
+		})
+	}
+}
+
+// startWithLookups is startWithLookupsAndClock for a test that needs no
+// clock.
+func startWithLookups(
+	t *testing.T, env map[string]string,
+) (*sender, *proxy.Server, *alerts.Queue) {
+	t.Helper()
+
+	s, _, server, queue := startWithLookupsAndClock(t, env)
+
+	return s, server, queue
+}
+
+// startWithLookupsAndClock is startAppWithAlerts in front of
+// readAndAnswer, with the settings in env on top of clients looked up in a
+// lookup database, which places fromDE and fromKP in the AS numbers and
+// countries the stand-in for GeoJS gives them, noCountry in AS64500 and no
+// country, and no other address.
+func startWithLookupsAndClock(
+	t *testing.T, env map[string]string,
+) (*sender, *clock, *proxy.Server, *alerts.Queue) {
+	t.Helper()
+
+	path := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb")
+	lookuptest.Write(t, path, map[string]lookuptest.Network{
+		fromDE + "/32":    {ASN: asnDE, ASName: asNameDE, Country: "DE"},
+		fromKP + "/32":    {ASN: asnKP, ASName: asNameKP, Country: "KP"},
+		noCountry + "/32": {ASN: "AS64500", ASName: "Nowhere Net"},
+	})
+
+	settings := map[string]string{lookupSource: fileSource, lookupDBPath: path}
+	maps.Copy(settings, env)
+
+	return startAppWithAlerts(t, readAndAnswer, settings)
+}
+
+// uploadFrom is upload from the client at from.
+func (s *sender) uploadFrom(from string) logLine {
+	s.t.Helper()
+
+	line, _ := s.requestWithBody(http.MethodPost, from, "/", uploadHeader, uploadBody,
+		http.StatusOK, requestlog.ActionForward)
+
+	return line
+}
+
+// serveFromDE hands a request from fromDE with method and body straight to
+// server's handler, without the network, and returns once it is answered.
+func serveFromDE(t *testing.T, server *proxy.Server, method string, body io.Reader) {
+	t.Helper()
+
+	req := httptest.NewRequestWithContext(t.Context(), method, "/", body)
+	req.RemoteAddr = net.JoinHostPort(fromDE, "1234")
+
+	server.Handler.ServeHTTP(httptest.NewRecorder(), req)
+}
+
+// wantPercent checks a limit percentage that a log line or a ban's notes
+// give, what, and the setting that gave it, against want, as percentText
+// gives them.
+func wantPercent(t *testing.T, what string, percent *int64, setting, want string) {
+	t.Helper()
+
+	if got := percentText(percent, setting); got != want {
+		t.Errorf("%s is %s, want %s", what, got, want)
+	}
+}
+
+// percentText gives a limit percentage and the setting that gave it as
+// text, such as "50 from SWWAF_ASN_LIMIT_PERCENT", or none when both are
+// left out.
+func percentText(percent *int64, setting string) string {
+	switch {
+	case percent == nil && setting == "":
+		return none
+	case percent == nil:
+		return "none from " + setting
+	default:
+		return fmt.Sprintf("%d from %s", *percent, setting)
+	}
+}

internal/proxy/bytelimits_test.go

@@ -3,6 +3,7 @@ package proxy_test
 import (
 	"bufio"
 	"io"
+	"net"
 	"net/http"
 	"net/netip"
 	"strconv"
@@ -166,6 +167,39 @@ func TestWebSocketBytesAreCountedOnceItCloses(t *testing.T) {
 	}
 }
 
+func TestWebSocketPassesTheAnswerAfterTheClientStopsSending(t *testing.T) {
+	t.Parallel()
+
+	app := startApp(t, echoOnceTheClientStops)
+	addr, out := startProxy(t, app.URL,
+		map[string]string{trustedProxies: trustLocalhost})
+	s := &sender{t: t, addr: addr, out: out}
+
+	conn, reader := s.openWebSocket()
+	send(t, conn, uploadBody)
+
+	// The client closes its sending side and waits for the answer, which the
+	// app sends only once it has seen the client stop. smallwebwaf passes the
+	// close on to the app through CloseWrite on upgradedConn; without that,
+	// it closes both connections, and the answer is lost.
+	tcp, ok := conn.(*net.TCPConn)
+	if !ok {
+		t.Fatalf("connection is a %T, want a *net.TCPConn", conn)
+	}
+
+	err := tcp.CloseWrite()
+	if err != nil {
+		t.Fatalf("close the sending side: %v", err)
+	}
+
+	got, err := io.ReadAll(reader)
+	if err != nil || string(got) != uploadBody {
+		t.Errorf("got %q (%v), want %q", got, err, uploadBody)
+	}
+
+	s.closeWebSocket(conn)
+}
+
 func TestBytesCountSaysWhichBytesCount(t *testing.T) {
 	t.Parallel()
 
@@ -432,12 +466,52 @@ func answerAfterUpgrade(w http.ResponseWriter, _ *http.Request) {
 	}
 }
 
+// echoOnceTheClientStops is an app that switches protocols, as for a
+// WebSocket, reads what the client sends until the client stops sending,
+// and then sends it all back.
+func echoOnceTheClientStops(w http.ResponseWriter, _ *http.Request) {
+	conn, buffered, err := http.NewResponseController(w).Hijack()
+	if err != nil {
+		return
+	}
+
+	defer func() {
+		_ = conn.Close()
+	}()
+
+	_, _ = buffered.WriteString("HTTP/1.1 101 Switching Protocols\r\n" +
+		"Connection: Upgrade\r\nUpgrade: websocket\r\n\r\n")
+	_ = buffered.Flush()
+
+	received, _ := io.ReadAll(buffered)
+	_, _ = buffered.Write(received)
+	_ = buffered.Flush()
+}
+
 // webSocket opens a WebSocket from client to answerAfterUpgrade, sends a
 // line of bodyBytes on it, reads the answer, and closes it. It checks the
 // answer, and the log line as request does, and returns the log line.
 func (s *sender) webSocket() logLine {
 	s.t.Helper()
 
+	conn, reader := s.openWebSocket()
+	send(s.t, conn, strings.Repeat("u", bodyBytes-1)+"\n")
+
+	got, err := reader.ReadString('\n')
+	if err != nil || len(got) != answerBytes {
+		s.t.Errorf("got %d bytes (%v), want %d", len(got), err, answerBytes)
+	}
+
+	return s.closeWebSocket(conn)
+}
+
+// openWebSocket sends a request from client to switch protocols, as for a
+// WebSocket, and checks that the app switches. It returns the connection,
+// on which reading fails once waitLimit has passed, and a reader of what
+// the app sends on it.
+func (s *sender) openWebSocket() (net.Conn, *bufio.Reader) {
+	s.t.Helper()
+
 	conn := dial(s.t, s.addr)
 	send(s.t, conn, "GET /socket HTTP/1.1\r\nHost: "+appHost+"\r\n"+forwardedFor+
 		": "+client+"\r\nConnection: Upgrade\r\nUpgrade: websocket\r\n\r\n")
@@ -460,12 +534,13 @@ func (s *sender) webSocket() logLine {
 		s.t.Fatalf("status %d, want %d", res.StatusCode, http.StatusSwitchingProtocols)
 	}
 
-	send(s.t, conn, strings.Repeat("u", bodyBytes-1)+"\n")
+	return conn, reader
+}
 
-	got, err := reader.ReadString('\n')
-	if err != nil || len(got) != answerBytes {
-		s.t.Errorf("got %d bytes (%v), want %d", len(got), err, answerBytes)
-	}
+// closeWebSocket closes conn, a WebSocket openWebSocket opened, checks its
+// log line as request does, and returns it.
+func (s *sender) closeWebSocket(conn net.Conn) logLine {
+	s.t.Helper()
 
 	_ = conn.Close()
 

internal/proxy/lookup.go

@@ -22,8 +22,9 @@ const (
 // database or through GeoJS, and notes them for the log line, unless
 // SWWAF_LOOKUP_SOURCE is off or the client is on a private, loopback or
 // link-local address, which no lookup can place. The lookup database
-// answers at once. With GeoJS, while a setting needs the answer, a new
-// client's request waits for it. ctx is the request's own context.
+// answers at once. With GeoJS, while a setting needs the answer, such as a
+// country list or a biased threshold, a new client's request waits for it.
+// ctx is the request's own context.
 func (rq *request) lookUp(ctx context.Context) {
 	if rq.h.config.LookupSource == "off" || !canBePlaced(rq.client) {
 		return

internal/proxy/lookup_test.go

@@ -22,6 +22,10 @@ import (
 // and country.
 type asnAndCountry struct{ asn, asName, country string }
 
+// fileSource is the SWWAF_LOOKUP_SOURCE that looks clients up in the
+// lookup database.
+const fileSource = "file"
+
 func TestEveryClientIsLookedUpWithoutWaitingWhileNoSettingNeedsIt(t *testing.T) {
 	t.Parallel()
 
@@ -178,7 +182,7 @@ func TestClientsAreLookedUpInTheLookupDatabaseAndGeoJSIsNotAsked(t *testing.T) {
 		fromKP + "/32": {ASN: asnKP, ASName: asNameKP, Country: "KP"},
 	})
 	s, clk, server := startWithClock(t, geojsURL, map[string]string{
-		lookupSource:       "file",
+		lookupSource:       fileSource,
 		lookupDBPath:       path,
 		allowedCountries:   "DE",
 		rateLimitPerMinute: "1",

internal/proxy/proxy.go

@@ -12,11 +12,13 @@ import (
 	"time"
 
 	"sneak.berlin/go/smallwebwaf/internal/alerts"
+	"sneak.berlin/go/smallwebwaf/internal/anomaly"
 	"sneak.berlin/go/smallwebwaf/internal/bans"
 	"sneak.berlin/go/smallwebwaf/internal/config"
 	"sneak.berlin/go/smallwebwaf/internal/lookup"
 	"sneak.berlin/go/smallwebwaf/internal/metrics"
 	"sneak.berlin/go/smallwebwaf/internal/ratelimit"
+	"sneak.berlin/go/smallwebwaf/internal/reputation"
 	"sneak.berlin/go/smallwebwaf/internal/requestlog"
 	"sneak.berlin/go/smallwebwaf/internal/rules"
 )
@@ -68,20 +70,25 @@ type Params struct {
 	// against.
 	Rules *rules.Files
 	// Alerts receive the alert for each ban the proxy makes or makes
-	// permanent, and for GeoJS failing.
+	// permanent, for each count over an anomaly threshold, for each request
+	// whose client a blocklist lists, and for GeoJS failing or a fetch of a
+	// list failing.
 	Alerts *alerts.Queue
 }
 
 // Server is the server smallwebwaf runs, with the parts of the proxy
 // whose state the state files keep, the lookup database, nil unless
-// SWWAF_LOOKUP_SOURCE is file, and the metrics.
+// SWWAF_LOOKUP_SOURCE is file, the lists fetched from URLs, which its Run
+// fetches, and the metrics.
 type Server struct {
 	*http.Server
 
 	Ledger     *bans.Ledger
 	Limiter    *ratelimit.Limiter
 	GeoJS      *lookup.GeoJS
+	Anomalies  *anomaly.Counters
 	LookupFile *lookup.File
+	Lists      *reputation.Lists
 	Metrics    *metrics.Metrics
 }
 
@@ -117,18 +124,34 @@ func New(params Params) *Server {
 			AttackBanDuration:    params.Config.AttackBanDuration,
 			MaxBans:              params.Config.MaxBans,
 		}),
+		anomalies: anomaly.New(anomaly.Params{
+			Client:         params.Config.AnomalyClient,
+			Net:            params.Config.AnomalyNet,
+			ASN:            params.Config.AnomalyASN,
+			Total:          params.Config.AnomalyTotal,
+			Watch:          params.Config.AnomalyWatch,
+			NetV4Prefix:    params.Config.AnomalyNetV4Prefix,
+			NetV6Prefix:    params.Config.AnomalyNetV6Prefix,
+			NamedNetblocks: params.Config.WatchNets,
+			Alerts:         params.Alerts,
+		}),
 		lookupFile: params.LookupFile,
-		rules:      params.Rules,
-		alerts:     params.Alerts,
+		lists: reputation.New(reputation.Params{
+			BlocklistURLs: params.Config.BlocklistURLs, Refresh: params.Config.BlocklistRefresh,
+			ASNLimitPercentURL: params.Config.ASNLimitPercentURL, Now: params.Now,
+			ProcessLog: params.ProcessLog, Alerts: params.Alerts,
+		}),
+		rules:  params.Rules,
+		alerts: params.Alerts,
 	}
 	h.geojs = lookup.New(lookup.Params{
 		URL:     params.GeoJSURL,
 		Timeout: params.Config.LookupTimeout,
-		// The country lists and the headers act on the answer before the
-		// request goes on.
+		// The country lists, the headers and the biased thresholds act on
+		// the answer before the request goes on.
 		Wait: len(params.Config.DeniedCountries) > 0 ||
 			len(params.Config.ExclusivelyAllowedCountries) > 0 ||
-			params.Config.AddLookupHeaders,
+			params.Config.AddLookupHeaders || biasedThresholdsSet(params.Config),
 		Answered:   h.addLookup,
 		Now:        params.Now,
 		ProcessLog: params.ProcessLog,
@@ -137,6 +160,7 @@ func New(params Params) *Server {
 	})
 	m.AddBansAndClients(h.ledger, h.limiter, params.Now)
 	m.AddRules(params.Rules)
+	m.AddReputation(h.lists)
 
 	return &Server{
 		Server: &http.Server{
@@ -154,7 +178,9 @@ func New(params Params) *Server {
 		Ledger:     h.ledger,
 		Limiter:    h.limiter,
 		GeoJS:      h.geojs,
+		Anomalies:  h.anomalies,
 		LookupFile: h.lookupFile,
+		Lists:      h.lists,
 		Metrics:    m,
 	}
 }
@@ -172,7 +198,9 @@ type handler struct {
 	limiter    *ratelimit.Limiter
 	ledger     *bans.Ledger
 	geojs      *lookup.GeoJS
+	anomalies  *anomaly.Counters
 	lookupFile *lookup.File
+	lists      *reputation.Lists
 	rules      *rules.Files
 	alerts     *alerts.Queue
 }
@@ -211,6 +239,7 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 
 	// Once the request has ended, before its log line is written.
 	defer rq.addToHistory()
+	defer rq.countAnomalies()
 
 	refused := rq.check(r.Context())
 	rq.checked = time.Now()

internal/proxy/proxy_test.go

@@ -315,7 +315,7 @@ func newProxy(
 
 	var lookupFile *lookup.File
 
-	if cfg.LookupSource == "file" {
+	if cfg.LookupSource == fileSource {
 		lookupFile, err = lookup.OpenFile(lookup.FileParams{
 			Path: cfg.LookupDBPath, Now: now, ProcessLog: processLog, Alerts: alertQueue,
 		})

internal/proxy/reputation.go

@@ -0,0 +1,32 @@
+package proxy
+
+import (
+	"sneak.berlin/go/smallwebwaf/internal/alerts"
+)
+
+// blocklistDenied notes in the log line the URLs of the blocklists that
+// list the client, counts each of them in the metrics and raises a
+// reputation_hit alert for it, and reports whether SWWAF_BLOCKLIST_ACTION,
+// being deny, refuses the request. Being limit, it lowers the client's
+// limits instead (see limitPercentages), and being log, it does nothing
+// more.
+func (rq *request) blocklistDenied() bool {
+	listedBy := rq.h.lists.ListedBy(rq.client)
+	rq.line.Reputation = listedBy
+
+	for _, listURL := range listedBy {
+		rq.h.metrics.ReputationHit(listURL)
+		rq.h.alerts.Raise(alerts.Alert{
+			Event:    alerts.EventReputationHit,
+			Client:   rq.client,
+			Netblock: clientGroup(rq.client),
+			ASN:      rq.line.ASN,
+			ASName:   rq.line.ASName,
+			Country:  rq.line.Country,
+			Reason:   "listed by a blocklist",
+			Detail:   map[string]any{"source": listURL},
+		})
+	}
+
+	return len(listedBy) > 0 && rq.h.config.BlocklistAction == "deny"
+}

internal/proxy/reputation_test.go

@@ -0,0 +1,333 @@
+package proxy_test
+
+import (
+	"maps"
+	"net/http"
+	"net/netip"
+	"slices"
+	"testing"
+	"time"
+
+	"sneak.berlin/go/smallwebwaf/internal/alerts"
+	"sneak.berlin/go/smallwebwaf/internal/proxy"
+	"sneak.berlin/go/smallwebwaf/internal/reputation"
+	"sneak.berlin/go/smallwebwaf/internal/requestlog"
+)
+
+// The reputation settings.
+const (
+	blocklistURLs      = "SWWAF_BLOCKLIST_URLS"
+	blocklistAction    = "SWWAF_BLOCKLIST_ACTION"
+	asnLimitPercentURL = "SWWAF_ASN_LIMIT_PERCENT_URL"
+)
+
+// The actions of SWWAF_BLOCKLIST_ACTION but limit, which has a
+// percentage.
+const (
+	actionDeny = "deny"
+	actionLog  = "log"
+)
+
+// The lists these tests name, which are never fetched: each test puts in
+// the copies it needs, as reputation.json would at start.
+const (
+	dropURL = "https://lists.example/drop.txt"
+	torURL  = "https://lists.example/tor.txt"
+	asnURL  = "https://lists.example/asn.txt"
+)
+
+func TestEachBlocklistActionForAListedAddressAndAListedNetblock(t *testing.T) {
+	t.Parallel()
+
+	forward, denied := requestlog.ActionForward, requestlog.ActionDenied
+
+	for _, tc := range []struct {
+		action string
+		// statuses and actions are those of a listed client's three
+		// requests, and percent their limit_percent, as percentText gives it.
+		statuses []int
+		actions  []string
+		percent  string
+	}{
+		{
+			actionDeny, []int{http.StatusForbidden, http.StatusForbidden, http.StatusForbidden},
+			[]string{denied, denied, denied}, none,
+		},
+		{
+			// Half of 4 requests a minute: the third breaks the limit.
+			"limit:50", []int{http.StatusOK, http.StatusOK, http.StatusForbidden},
+			[]string{forward, forward, requestlog.ActionRateLimited},
+			"50 from " + blocklistAction,
+		},
+		{
+			actionLog, []int{http.StatusOK, http.StatusOK, http.StatusOK},
+			[]string{forward, forward, forward}, none,
+		},
+	} {
+		t.Run(tc.action, func(t *testing.T) {
+			t.Parallel()
+
+			s, server, _ := startWithLookups(t, map[string]string{
+				rateLimitPerMinute: fourAMinute, blocklistURLs: dropURL,
+				blocklistAction: tc.action,
+			})
+			// fromDE is listed as an address, and fromKP in a netblock.
+			loadLists(t, server, map[string][]string{
+				dropURL: {"; DROP", fromDE, "198.51.100.0/24 ; SBL1"},
+			})
+
+			for _, from := range []string{fromDE, fromKP} {
+				for i := range 3 {
+					line := s.get(from, tc.statuses[i], tc.actions[i])
+					wantReputation(t, line, dropURL)
+					wantPercent(t, "limit_percent", line.LimitPercent,
+						line.LimitPercentSetting, tc.percent)
+
+					// A request refused for the list is not counted.
+					counted := line.fields["counts"] != nil
+					if counted != (tc.actions[i] != denied) {
+						t.Errorf("request from %s counted %t, logged %s", from, counted,
+							tc.actions[i])
+					}
+				}
+			}
+
+			// A client no list lists has the whole limit.
+			for range 3 {
+				line := s.get(unplaced, http.StatusOK, forward)
+				wantReputation(t, line)
+				wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
+					none)
+			}
+
+			// A refusal for the list makes no ban.
+			if held := server.Ledger.Snapshot(); tc.action == actionDeny && len(held) != 0 {
+				t.Errorf("bans %+v, want none", held)
+			}
+		})
+	}
+}
+
+func TestBlocklistsComeAfterTheCountryListsAndSkipAllowNets(t *testing.T) {
+	t.Parallel()
+
+	s, server, queue := startWithLookups(t, map[string]string{
+		blocklistURLs: dropURL, deniedCountries: "kp", allowNets: fromDE,
+	})
+	loadLists(t, server, map[string][]string{dropURL: {fromDE, fromKP}})
+
+	// fromKP's country refuses it before the list is looked at, and fromDE,
+	// in SWWAF_ALLOW_NETS, is not checked at all: neither is noted, nor
+	// alerted.
+	wantReputation(t, s.get(fromKP, http.StatusForbidden, requestlog.ActionCountryDenied))
+	wantReputation(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward))
+	wantAlerts(t, queue)
+}
+
+func TestObserveModeForwardsAClientABlocklistDeniesAndAlertsIt(t *testing.T) {
+	t.Parallel()
+
+	s, server, queue := startWithLookups(t, map[string]string{
+		blocklistURLs: dropURL, mode: observe,
+	})
+	loadLists(t, server, map[string][]string{dropURL: {fromDE}})
+
+	line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
+	wantWouldAction(t, line, requestlog.ActionDenied)
+	wantReputation(t, line, dropURL)
+
+	waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
+	if len(waiting) != 1 || waiting[0].Event != alerts.EventReputationHit {
+		t.Errorf("alerts waiting %+v, want a reputation_hit alert", waiting)
+	}
+}
+
+func TestBlocklistLimitTakesPartInTheLowestPercentageOfEveryLimit(t *testing.T) {
+	t.Parallel()
+
+	for _, tc := range []struct {
+		action, asnPercent string
+		// want is the upload's limit_percent and bytes_percent, as
+		// percentText gives them, and limitHit its limit_hit.
+		want, limitHit string
+	}{
+		{"limit:50", asnDEQuarter, "25 from " + asnLimitPercent, minuteBytes},
+		{"limit:25", asnDEHalf, "25 from " + blocklistAction, minuteBytes},
+		// The AS number's, the first of two alike.
+		{"limit:25", asnDEQuarter, "25 from " + asnLimitPercent, minuteBytes},
+		{actionLog, asnDE + ":100", none, ""},
+	} {
+		t.Run(tc.action+" "+tc.asnPercent, func(t *testing.T) {
+			t.Parallel()
+
+			s, server, _ := startWithLookups(t, map[string]string{
+				bytesLimitPerMinute: twoUploads, blocklistURLs: dropURL,
+				blocklistAction: tc.action, asnLimitPercent: tc.asnPercent,
+			})
+			loadLists(t, server, map[string][]string{dropURL: {fromDE}})
+
+			// The upload's 100 bytes are over 49, a quarter of 199, and 99,
+			// half of it, and within 199.
+			line := s.uploadFrom(fromDE)
+			wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
+				tc.want)
+			wantPercent(t, "bytes_percent", line.BytesPercent, line.BytesPercentSetting,
+				tc.want)
+
+			if line.LimitHit != tc.limitHit {
+				t.Errorf("log line has limit_hit %q, want %q", line.LimitHit, tc.limitHit)
+			}
+		})
+	}
+}
+
+func TestASNLimitPercentFileCountsAsTheSettingDoesTheLowerWinning(t *testing.T) {
+	t.Parallel()
+
+	const (
+		fromURL     = "25 from " + asnLimitPercentURL
+		fromSetting = "25 from " + asnLimitPercent
+	)
+
+	for _, tc := range []struct {
+		name string
+		env  map[string]string
+		file string
+		// limitPercent and bytesPercent are the upload's, as percentText
+		// gives them.
+		limitPercent, bytesPercent string
+	}{
+		{"the file's alone", nil, asnDEQuarter, fromURL, fromURL},
+		{
+			"the file's, lower than the setting's",
+			map[string]string{asnLimitPercent: asnDEHalf}, asnDEQuarter, fromURL, fromURL,
+		},
+		{
+			"the setting's, lower than the file's",
+			map[string]string{asnLimitPercent: asnDEQuarter}, asnDEHalf,
+			fromSetting, fromSetting,
+		},
+		{
+			"the setting's, the first of two alike",
+			map[string]string{asnLimitPercent: asnDEQuarter}, asnDEQuarter,
+			fromSetting, fromSetting,
+		},
+		{"none, for an AS number the file does not list", nil, asnKP + ":25", none, none},
+		{
+			"SWWAF_ASN_BYTES_PERCENT's in place of the file's for the byte limits",
+			map[string]string{asnBytesPercent: asnDE + ":100"}, asnDEQuarter, fromURL, none,
+		},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+
+			env := map[string]string{asnLimitPercentURL: asnURL}
+			maps.Copy(env, tc.env)
+
+			s, server, _ := startWithLookups(t, env)
+			loadLists(t, server, map[string][]string{asnURL: {"# by AS number", tc.file}})
+
+			line := s.uploadFrom(fromDE)
+			wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
+				tc.limitPercent)
+			wantPercent(t, "bytes_percent", line.BytesPercent, line.BytesPercentSetting,
+				tc.bytesPercent)
+		})
+	}
+}
+
+func TestEachBlocklistThatListsAClientRaisesAnAlertOncePerCooldownAndIsCounted(
+	t *testing.T,
+) {
+	t.Parallel()
+
+	const emptyURL = "https://lists.example/empty.txt"
+
+	s, clk, server, queue := startWithLookupsAndClock(t, map[string]string{
+		blocklistURLs:   dropURL + "," + torURL + "," + emptyURL,
+		blocklistAction: actionLog,
+		metricsToken:    token,
+	})
+	loadLists(t, server, map[string][]string{dropURL: {fromDE}, torURL: {fromDE}})
+
+	// The second request's alerts are repeats, which the cooldown holds
+	// back.
+	for range 2 {
+		wantReputation(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward),
+			dropURL, torURL)
+	}
+
+	hit := func(source string) alerts.Alert {
+		return alerts.Alert{
+			Instance: alertInstance,
+			Time:     clk.Now(),
+			Event:    alerts.EventReputationHit,
+			Client:   netip.MustParseAddr(fromDE),
+			Netblock: netip.MustParsePrefix(fromDE + "/32"),
+			ASN:      asnDE,
+			ASName:   asNameDE,
+			Country:  "DE",
+			Reason:   "listed by a blocklist",
+			Detail:   map[string]any{"source": source},
+		}
+	}
+	wantAlerts(t, queue, hit(dropURL), hit(torURL))
+
+	if queue.Suppressed() != 2 {
+		t.Errorf("%d alerts held back, want the second request's 2", queue.Suppressed())
+	}
+
+	// Each list's hits, none of its fetches failed, and when its copy was
+	// fetched, 0 for the one without.
+	metrics := s.scrape(unplaced)
+	fetched := float64(listsFetched().Unix())
+
+	for listURL, want := range map[string]struct{ hits, fetched float64 }{
+		dropURL: {2, fetched}, torURL: {2, fetched}, emptyURL: {0, 0},
+	} {
+		labels := `{instance="` + alertInstance + `",source="` + listURL + `"}`
+
+		if want.hits == 0 {
+			wantNoSeries(t, metrics, "smallwebwaf_reputation_hits_total"+labels)
+		} else {
+			wantMetric(t, metrics, "smallwebwaf_reputation_hits_total"+labels, want.hits)
+		}
+
+		wantMetric(t, metrics, "smallwebwaf_reputation_failures_total"+labels, 0)
+		wantMetric(t, metrics, "smallwebwaf_reputation_last_fetch_timestamp_seconds"+labels,
+			want.fetched)
+	}
+}
+
+// listsFetched is when loadLists has the copies fetched.
+func listsFetched() time.Time {
+	return time.Date(2026, 10, 5, 0, 0, 0, 0, time.UTC)
+}
+
+// loadLists puts copies of lists into server's lists, by URL, each with
+// its lines, fetched at listsFetched, as reputation.json would at start.
+func loadLists(t *testing.T, server *proxy.Server, copies map[string][]string) {
+	t.Helper()
+
+	lists := make([]reputation.List, 0, len(copies))
+	for listURL, lines := range copies {
+		lists = append(lists, reputation.List{
+			URL: listURL, Fetched: listsFetched(), Lines: lines,
+		})
+	}
+
+	err := server.Lists.Load(lists)
+	if err != nil {
+		t.Fatalf("load the lists: %v", err)
+	}
+}
+
+// wantReputation checks the URLs of the blocklists the log line names in
+// its reputation.
+func wantReputation(t *testing.T, line logLine, want ...string) {
+	t.Helper()
+
+	if !slices.Equal(line.Reputation, want) {
+		t.Errorf("log line has reputation %v, want %v", line.Reputation, want)
+	}
+}

internal/proxy/request.go

@@ -16,6 +16,8 @@ import (
 	"sync/atomic"
 	"time"
 
+	"sneak.berlin/go/smallwebwaf/internal/anomaly"
+	"sneak.berlin/go/smallwebwaf/internal/config"
 	"sneak.berlin/go/smallwebwaf/internal/lookup"
 	"sneak.berlin/go/smallwebwaf/internal/ratelimit"
 	"sneak.berlin/go/smallwebwaf/internal/requestlog"
@@ -56,9 +58,12 @@ type request struct {
 	lookedUp     bool
 	lookupAnswer lookup.Answer
 	// counted is true for a request the rate limits counted, whose bytes
-	// the byte limits count once it has ended.
-	counted bool
-	start   time.Time
+	// the byte limits count once it has ended. limitPercent and
+	// bytesPercent are then its client's limit percentages for the rate
+	// limits and for the byte limits.
+	counted                    bool
+	limitPercent, bytesPercent percentage
+	start                      time.Time
 	// checked is when the checks were done, and upstreamStart when the
 	// request was handed to the app.
 	checked       time.Time
@@ -208,13 +213,14 @@ func (rq *request) check(ctx context.Context) *refusal {
 // client in SWWAF_ALLOW_NETS skips them, and is not looked up. For any
 // other client, SWWAF_DENY_NETS comes first, then a ban on its netblock,
 // so that a client either refuses is not looked up, then the lookup of
-// its AS number and country, and then the country lists; a request any of
-// them refuses is not counted for the rate limits. Then come the rate
-// limits, unless the client is in SWWAF_RATE_LIMIT_EXEMPT_NETS or the
-// request's path is exempt under SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that
-// every other request is counted, and last the rule files. A request
-// exempt from the rate limits is exempt from the byte limits too. ctx is
-// the request's own context.
+// its AS number and country, then the country lists, and then the
+// blocklists; a request any of them refuses is not counted for the rate
+// limits. Then come the rate limits, unless the client is in
+// SWWAF_RATE_LIMIT_EXEMPT_NETS or the request's path is exempt under
+// SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that every other request is counted,
+// each of them by the client's limit percentages, and last the rule
+// files. A request exempt from the rate limits is exempt from the byte
+// limits too. ctx is the request's own context.
 func (rq *request) checkClient(ctx context.Context) string {
 	cfg := rq.h.config
 	if isInside(rq.client, cfg.AllowNets) {
@@ -237,8 +243,18 @@ func (rq *request) checkClient(ctx context.Context) string {
 		return requestlog.ActionCountryDenied
 	}
 
+	if rq.blocklistDenied() {
+		return requestlog.ActionDenied
+	}
+
 	rq.counted = !isInside(rq.client, cfg.RateLimitExemptNets) &&
 		!pathExempt(rq.in.URL, cfg.RateLimitExemptPaths)
+	if rq.counted {
+		rq.limitPercent, rq.bytesPercent = rq.limitPercentages()
+		rq.line.LimitPercent, rq.line.LimitPercentSetting = rq.limitPercent.logged()
+		rq.line.BytesPercent, rq.line.BytesPercentSetting = rq.bytesPercent.logged()
+	}
+
 	if rq.counted && rq.limitBroken(now) {
 		return requestlog.ActionRateLimited
 	}
@@ -503,16 +519,14 @@ func timing(start, end time.Time) *float64 {
 }
 
 // addToHistory adds the request, which has ended, to its client's
-// history, and then, for a client that was looked up, the lookup
-// database's answer about it, or the answer from GeoJS kept about it, to
-// that history and to the notes of the bans on its netblock: an answer
-// may have come before either was there, and one from GeoJS that comes
-// later is added when it comes.
+// history, and then the lookup's answer about the client, as
+// answerAtTheEnd gives it, to that history and to the notes of the bans
+// on its netblock: an answer may have come before either was there, and
+// one from GeoJS that comes later is added when it comes.
 func (rq *request) addToHistory() {
 	forwarded := !rq.upstreamStart.IsZero()
-	group := clientGroup(rq.client)
 
-	rq.h.limiter.AddToHistory(group, rq.h.now(), ratelimit.Request{
+	rq.h.limiter.AddToHistory(clientGroup(rq.client), rq.h.now(), ratelimit.Request{
 		Forwarded:     forwarded,
 		Refused:       !forwarded && rq.refused.Load() != nil,
 		Status:        rq.out.status,
@@ -521,23 +535,60 @@ func (rq *request) addToHistory() {
 		BrokeLimit:    rq.line.Offence == requestlog.OffenceLimit,
 	})
 
-	if !rq.lookedUp {
-		return
-	}
-
-	// The lookup database's answer was there at once.
-	if rq.h.config.LookupSource == "file" {
-		rq.h.addLookup(rq.lookupAnswer)
-
-		return
-	}
-
-	answer, kept := rq.h.geojs.Kept(group)
-	if kept {
+	answer, found := rq.answerAtTheEnd()
+	if found {
 		rq.h.addLookup(answer)
 	}
 }
 
+// countAnomalies counts the request, which has ended, and its bytes, as
+// countedBytes gives them, for the anomaly thresholds, whatever was done
+// with it: a request refused, one from a client in SWWAF_ALLOW_NETS or
+// SWWAF_RATE_LIMIT_EXEMPT_NETS, and one for a path in
+// SWWAF_RATE_LIMIT_EXEMPT_PATHS are counted too. It is counted for its
+// client's AS number when answerAtTheEnd gives one. With every anomaly
+// threshold off, the default, it does nothing.
+func (rq *request) countAnomalies() {
+	if !anomalyThresholdsSet(rq.h.config) {
+		return
+	}
+
+	answer, _ := rq.answerAtTheEnd()
+
+	rq.h.anomalies.Count(rq.h.now(), anomaly.Request{
+		Client:      rq.client,
+		ClientGroup: clientGroup(rq.client),
+		ASN:         answer.ASN,
+		ASName:      answer.ASName,
+		Country:     answer.Country,
+		Bytes:       rq.countedBytes(),
+	})
+}
+
+// anomalyThresholdsSet reports whether any anomaly threshold is set.
+func anomalyThresholdsSet(cfg *config.Config) bool {
+	off := anomaly.Thresholds{}
+
+	return cfg.AnomalyClient != off || cfg.AnomalyNet != off || cfg.AnomalyASN != off ||
+		cfg.AnomalyTotal != off || cfg.AnomalyWatch != off
+}
+
+// answerAtTheEnd returns, for a client that was looked up, the lookup's
+// answer about it as the request ends, and whether there is one: the
+// lookup database's, which was there at once, or the one GeoJS has given
+// by then, which a request does not wait for unless a setting needs it.
+func (rq *request) answerAtTheEnd() (lookup.Answer, bool) {
+	if !rq.lookedUp {
+		return lookup.Answer{}, false
+	}
+
+	if rq.h.config.LookupSource == "file" {
+		return rq.lookupAnswer, true
+	}
+
+	return rq.h.geojs.Kept(clientGroup(rq.client))
+}
+
 // requestBytes is how many bytes of the request's body have been read.
 func (rq *request) requestBytes() int64 {
 	if rq.body == nil {

internal/ratelimit/ratelimit.go

@@ -174,7 +174,7 @@ type Hit struct {
 	Kind string
 	// Window is "minute", "hour" or "day".
 	Window string
-	// Limit is the window's limit.
+	// Limit is the window's limit, as the client's percentage of it.
 	Limit int64
 	// Count is the client's requests, or bytes, counted in the window,
 	// this request's included.
@@ -196,21 +196,24 @@ type Counts struct {
 
 // Count counts a request from client at now, in every window, whether or
 // not it is refused, and returns the client's counts in each window. It
-// reports whether the request takes the client over a rate limit, and the
-// hit: the window whose limit it goes over, the shortest if it is over
-// several.
-func (l *Limiter) Count(client netip.Prefix, now time.Time) (Counts, Hit, bool) {
-	return l.count(client, now, 1, 0)
+// reports whether the request takes the client over a rate limit, of
+// which the client gets the percentage percent, rounded down, and the hit:
+// the window whose limit it goes over, the shortest if it is over
+// several. A limit that is off stays off.
+func (l *Limiter) Count(
+	client netip.Prefix, now time.Time, percent int64,
+) (Counts, Hit, bool) {
+	return l.count(client, now, 1, 0, percent)
 }
 
 // CountBytes counts bytes, those of a request from client that has ended,
 // at now, in every window, and returns the client's counts in each window.
-// It reports whether the bytes take the client over a byte limit, and the
-// hit, as Count does.
+// It reports whether the bytes take the client over a byte limit, of which
+// the client gets the percentage percent, and the hit, as Count does.
 func (l *Limiter) CountBytes(
-	client netip.Prefix, now time.Time, bytes int64,
+	client netip.Prefix, now time.Time, bytes, percent int64,
 ) (Counts, Hit, bool) {
-	return l.count(client, now, 0, bytes)
+	return l.count(client, now, 0, bytes, percent)
 }
 
 // Reset sets client's counts of requests and of bytes in every window
@@ -358,9 +361,7 @@ func (l *Limiter) Load(clients []Client, now time.Time) {
 	for _, c := range clients {
 		for i, w := range l.windows {
 			for _, b := range []*Buckets{c.buckets()[i], c.byteBuckets()[i]} {
-				// The window that ends at now covers neither bucket once it
-				// begins after the bucket under way has ended.
-				if !now.Add(-w.length).Before(b.Start.Add(w.length)) {
+				if b.Passed(now, w.length) {
 					*b = Buckets{}
 				}
 			}
@@ -373,9 +374,10 @@ func (l *Limiter) Load(clients []Client, now time.Time) {
 // count adds requests and bytes from client at now to its buckets in
 // every window, and returns its counts. A limit is broken only by what is
 // added to it, so that a request whose bytes are counted after another of
-// the client's requests broke a rate limit does not break it too.
+// the client's requests broke a rate limit does not break it too. The
+// client gets the percentage percent of each limit.
 func (l *Limiter) count(
-	client netip.Prefix, now time.Time, requests, bytes int64,
+	client netip.Prefix, now time.Time, requests, bytes, percent int64,
 ) (Counts, Hit, bool) {
 	l.mu.Lock()
 	defer l.mu.Unlock()
@@ -389,18 +391,19 @@ func (l *Limiter) count(
 	)
 
 	for i, w := range l.windows {
-		requestCounts[i] = requestBuckets[i].add(now, w.length, requests)
-		byteCounts[i] = byteBuckets[i].add(now, w.length, bytes)
+		requestCounts[i] = requestBuckets[i].Add(now, w.length, requests)
+		byteCounts[i] = byteBuckets[i].Add(now, w.length, bytes)
+		limit, byteLimit := percentOf(w.limit, percent), percentOf(w.byteLimit, percent)
 
 		switch {
 		case hit.Window != "":
-		case requests > 0 && w.limit > 0 && requestCounts[i] > float64(w.limit):
+		case requests > 0 && w.limit > 0 && requestCounts[i] > float64(limit):
 			hit = Hit{
-				Kind: KindRequests, Window: w.name, Limit: w.limit, Count: requestCounts[i],
+				Kind: KindRequests, Window: w.name, Limit: limit, Count: requestCounts[i],
 			}
-		case bytes > 0 && w.byteLimit > 0 && byteCounts[i] > float64(w.byteLimit):
+		case bytes > 0 && w.byteLimit > 0 && byteCounts[i] > float64(byteLimit):
 			hit = Hit{
-				Kind: KindBytes, Window: w.name, Limit: w.byteLimit, Count: byteCounts[i],
+				Kind: KindBytes, Window: w.name, Limit: byteLimit, Count: byteCounts[i],
 			}
 		}
 	}
@@ -446,18 +449,27 @@ type window struct {
 	byteLimit int64
 }
 
-// add counts n requests, or n bytes, at now in a window of length, and
-// returns the client's count in the window that ends at now: what is in
-// the bucket under way, and what is in the bucket before it weighted by
-// how much of that bucket the window still covers. With n zero it counts
-// nothing, and returns the count.
+// percentOf returns the percentage percent of limit, rounded down. It is
+// written as limit's hundreds times percent, plus the rest's share, since
+// limit*percent can overflow for a byte limit.
+func percentOf(limit, percent int64) int64 {
+	const hundred = 100
+
+	return limit/hundred*percent + limit%hundred*percent/hundred
+}
+
+// Add counts n requests, or n bytes, at now in a window of length, and
+// returns the count in the window that ends at now: what is in the bucket
+// under way, and what is in the bucket before it weighted by how much of
+// that bucket the window still covers. With n zero it counts nothing, and
+// returns the count. The anomaly counters count in Buckets too.
 //
 // Concurrent requests can be counted out of order, so now can be a moment
 // before the bucket under way began; such a request is counted in that
 // bucket. A request dated more than a second before it means the clock
 // was set back, and the buckets start afresh: otherwise the bucket before
 // would keep its full weight until the clock caught up.
-func (b *Buckets) add(now time.Time, length time.Duration, n int64) float64 {
+func (b *Buckets) Add(now time.Time, length time.Duration, n int64) float64 {
 	if now.Before(b.Start.Add(-time.Second)) {
 		*b = Buckets{}
 	}
@@ -482,6 +494,14 @@ func (b *Buckets) add(now time.Time, length time.Duration, n int64) float64 {
 	return float64(b.Previous)*covered + float64(b.Current)
 }
 
+// Passed reports whether the window of length that ends at now covers
+// neither of b's buckets: it begins after the bucket under way has ended.
+// What they hold then counts no more, and a state file read at now drops
+// it.
+func (b *Buckets) Passed(now time.Time, length time.Duration) bool {
+	return !now.Add(-length).Before(b.Start.Add(length))
+}
+
 // add counts a response with status in its class. A status of 0, for
 // nothing sent, is not a response.
 func (r *Responses) add(status int) {

internal/ratelimit/ratelimit_test.go

@@ -1,6 +1,7 @@
 package ratelimit_test
 
 import (
+	"math"
 	"net/netip"
 	"testing"
 	"time"
@@ -11,6 +12,10 @@ import (
 // limit is the limit the tests set.
 const limit = 3
 
+// whole is the percentage of each limit a client gets when nothing lowers
+// its limits.
+const whole = 100
+
 // The windows, as Count names them.
 const (
 	minute = "minute"
@@ -62,14 +67,14 @@ func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
 	start := midnight()
 
 	for range limit {
-		_, _, over := limiter.Count(client, start)
+		_, _, over := limiter.Count(client, start, whole)
 		if over {
 			t.Fatal("a request within the limit is over it")
 		}
 	}
 
 	// Over both limits; the minute's is named, with the four requests.
-	_, hit, over := limiter.Count(client, start)
+	_, hit, over := limiter.Count(client, start, whole)
 
 	want := ratelimit.Hit{
 		Kind: ratelimit.KindRequests, Window: minute, Limit: limit, Count: limit + 1,
@@ -80,6 +85,61 @@ func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
 	}
 }
 
+func TestClientGetsItsPercentageOfEachLimitRoundedDown(t *testing.T) {
+	t.Parallel()
+
+	limiter := ratelimit.New(ratelimit.Limits{PerMinute: 5, BytesPerDay: math.MaxInt64})
+	client := netip.MustParsePrefix("203.0.113.9/32")
+	start := midnight()
+
+	// Half of 5 requests is 2.5, rounded down to 2: the third is over.
+	for range 2 {
+		_, _, over := limiter.Count(client, start, 50)
+		if over {
+			t.Fatal("a request within half the limit is over it")
+		}
+	}
+
+	_, hit, over := limiter.Count(client, start, 50)
+
+	want := ratelimit.Hit{Kind: ratelimit.KindRequests, Window: minute, Limit: 2, Count: 3}
+	if !over || hit != want {
+		t.Errorf("the third request gives %+v and %t, want %+v and true", hit, over, want)
+	}
+
+	// Half of the largest byte limit is still far above a TiB: working it
+	// out does not overflow.
+	_, hit, over = limiter.CountBytes(client, start, 1<<40, 50)
+	if over {
+		t.Errorf("a TiB is over half the largest byte limit: %+v", hit)
+	}
+}
+
+func TestZeroPercentIsAZeroAllowanceAndALimitOffStaysOff(t *testing.T) {
+	t.Parallel()
+
+	// Only the hour has limits: the minute's and the day's are off.
+	limiter := ratelimit.New(ratelimit.Limits{PerHour: limit, BytesPerHour: 1000})
+	client := netip.MustParsePrefix("203.0.113.9/32")
+	start := midnight()
+
+	// At 0 percent, the first request and the first byte are over the
+	// hour's limits, which are 0; the minute's, which are off, stay off.
+	_, hit, _ := limiter.Count(client, start, 0)
+
+	want := ratelimit.Hit{Kind: ratelimit.KindRequests, Window: hour, Limit: 0, Count: 1}
+	if hit != want {
+		t.Errorf("the first request gives %+v, want %+v", hit, want)
+	}
+
+	_, hit, _ = limiter.CountBytes(client, start, 1, 0)
+
+	want = ratelimit.Hit{Kind: ratelimit.KindBytes, Window: hour, Limit: 0, Count: 1}
+	if hit != want {
+		t.Errorf("the first byte gives %+v, want %+v", hit, want)
+	}
+}
+
 func TestEachByteLimitIsBrokenByTheBytesCounted(t *testing.T) {
 	t.Parallel()
 
@@ -100,12 +160,12 @@ func TestEachByteLimitIsBrokenByTheBytesCounted(t *testing.T) {
 			client := netip.MustParsePrefix("203.0.113.9/32")
 
 			// 600 bytes are within the limit, 600 more over it.
-			_, _, over := limiter.CountBytes(client, midnight(), 600)
+			_, _, over := limiter.CountBytes(client, midnight(), 600, whole)
 			if over {
 				t.Fatal("600 bytes are over the limit of 1000")
 			}
 
-			_, hit, over := limiter.CountBytes(client, midnight(), 600)
+			_, hit, over := limiter.CountBytes(client, midnight(), 600, whole)
 
 			want := ratelimit.Hit{
 				Kind: ratelimit.KindBytes, Window: tc.window, Limit: byteLimit, Count: 1200,
@@ -148,16 +208,16 @@ func TestCountGivesTheBytesInEachWindow(t *testing.T) {
 	client := netip.MustParsePrefix("203.0.113.9/32")
 	start := midnight()
 
-	limiter.CountBytes(client, start, 300)
+	limiter.CountBytes(client, start, 300, whole)
 
 	// A quarter into the next hour, the minute has only these 100 bytes.
 	// The hour still covers three quarters of the bucket before, whose 300
 	// bytes count 225, and these: 325. The day covers all 400.
 	later := start.Add(time.Hour + time.Hour/4)
-	limiter.CountBytes(client, later, 100)
+	limiter.CountBytes(client, later, 100, whole)
 
 	// A request's counts give the bytes counted so far too.
-	counts, _, _ := limiter.Count(client, later)
+	counts, _, _ := limiter.Count(client, later, whole)
 
 	want := ratelimit.Counts{
 		Minute: 1, Hour: 1, Day: 1, MinuteBytes: 100, HourBytes: 325, DayBytes: 400,
@@ -189,14 +249,14 @@ func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
 	start := midnight()
 
 	for range 3 {
-		limiter.Count(client, start)
+		limiter.Count(client, start, whole)
 	}
 
 	// A quarter into the next hour, the minute has only this request. The
 	// hour still covers three quarters of the bucket before, with its three
 	// requests, which count 2.25, and this one: 3.25. The day covers all
 	// four.
-	counts, _, _ := limiter.Count(client, start.Add(time.Hour+time.Hour/4))
+	counts, _, _ := limiter.Count(client, start.Add(time.Hour+time.Hour/4), whole)
 
 	want := ratelimit.Counts{Minute: 1, Hour: 3.25, Day: 4}
 	if counts != want {
@@ -364,7 +424,7 @@ func wantCount(
 ) {
 	t.Helper()
 
-	_, hit, _ := limiter.Count(client, now)
+	_, hit, _ := limiter.Count(client, now, whole)
 	if hit.Window != want {
 		t.Errorf("request from %s at %s is over %q, want %q",
 			client, now.Format(time.RFC3339), hit.Window, want)
@@ -379,7 +439,7 @@ func wantBytesCount(
 ) {
 	t.Helper()
 
-	_, hit, _ := limiter.CountBytes(client, now, bytes)
+	_, hit, _ := limiter.CountBytes(client, now, bytes, whole)
 	if hit.Kind != want {
 		t.Errorf("%d bytes from %s at %s break a limit on %q, want %q",
 			bytes, client, now.Format(time.RFC3339), hit.Kind, want)

internal/ratelimit/snapshot_test.go

@@ -16,7 +16,7 @@ func TestSnapshotListsTheClientsByAddress(t *testing.T) {
 
 	limiter := ratelimit.New(ratelimit.Limits{})
 	for _, i := range []int{2, 3, 0, 1} {
-		limiter.Count(netip.MustParsePrefix(want[i]), midnight())
+		limiter.Count(netip.MustParsePrefix(want[i]), midnight(), whole)
 	}
 
 	snapshot := limiter.Snapshot()
@@ -63,8 +63,8 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
 	start := midnight()
 
 	limiter := ratelimit.New(ratelimit.Limits{})
-	limiter.Count(client, start)
-	limiter.CountBytes(client, start, 5)
+	limiter.Count(client, start, whole)
+	limiter.CountBytes(client, start, 5, whole)
 	limiter.AddToHistory(client, start, ratelimit.Request{Forwarded: true})
 
 	loaded := func(now time.Time) ratelimit.Client {

internal/reputation/export_test.go

@@ -0,0 +1,9 @@
+package reputation
+
+import "net/http"
+
+// SetTransport has l's fetches go through transport instead of the
+// network.
+func (l *Lists) SetTransport(transport http.RoundTripper) {
+	l.httpClient.Transport = transport
+}

internal/reputation/reputation.go

@@ -0,0 +1,499 @@
+// Package reputation fetches the lists the settings name by URL: the
+// blocklists of SWWAF_BLOCKLIST_URLS, and the file of AS:percent lines
+// SWWAF_ASN_LIMIT_PERCENT_URL names. It keeps the last good copy of each,
+// whole, comment lines included, which is used while a fetch fails, and
+// when each was last tried, which the state package writes to
+// reputation.json and reads from it, so that a restart keeps them too.
+package reputation
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"log/slog"
+	"net/http"
+	"net/netip"
+	"slices"
+	"strings"
+	"sync"
+	"time"
+
+	"sneak.berlin/go/smallwebwaf/internal/alerts"
+	"sneak.berlin/go/smallwebwaf/internal/config"
+)
+
+const (
+	// maxListBytes is the most of a list that is read. A longer one is a
+	// failure, so that a wrong URL cannot fill the memory.
+	maxListBytes = 16 << 20
+	// fetchTimeout bounds one fetch of a list.
+	fetchTimeout = time.Minute
+	// mappedBits is the length of ::ffff:0.0.0.0/96, the netblock of every
+	// IPv4-mapped address.
+	mappedBits = 96
+)
+
+var (
+	errStatus        = errors.New("the server answered")
+	errTooLong       = errors.New("the list is longer than 16 MiB")
+	errNotNetblock   = errors.New("is not an address or a netblock, such as 192.0.2.0/24")
+	errNotASNPercent = errors.New(
+		"is not an AS number, : and a percentage, such as AS64496:50")
+)
+
+// List is a list as reputation.json holds it: the URL it is fetched from,
+// when it was last tried, the fetch failed or not, and its last good copy:
+// when that was fetched, and its lines, as fetched, comment lines
+// included, both left out while no fetch of it has succeeded.
+type List struct {
+	URL     string    `json:"url"`
+	Tried   time.Time `json:"tried"`
+	Fetched time.Time `json:"fetched,omitzero"`
+	Lines   []string  `json:"lines,omitzero"`
+}
+
+// Params are what New needs.
+type Params struct {
+	// BlocklistURLs are the blocklists (SWWAF_BLOCKLIST_URLS), and
+	// ASNLimitPercentURL the file of AS:percent lines
+	// (SWWAF_ASN_LIMIT_PERCENT_URL), "" while it is unset.
+	BlocklistURLs      []string
+	ASNLimitPercentURL string
+	// Refresh is how long after a list was last fetched or tried it is
+	// fetched again (SWWAF_BLOCKLIST_REFRESH).
+	Refresh time.Duration
+	// Now tells the time, normally time.Now in UTC.
+	Now func() time.Time
+	// ProcessLog receives each fetch of a list, and why one failed.
+	ProcessLog *slog.Logger
+	// Alerts receive a source_failure alert for each fetch that fails.
+	Alerts *alerts.Queue
+}
+
+// Lists are the lists Params names, each with its last good copy. They
+// are safe for concurrent use.
+type Lists struct {
+	params     Params
+	httpClient *http.Client
+
+	mu sync.Mutex
+	// lists are by URL, one for each URL Params names.
+	lists map[string]*list
+}
+
+// list is one list: what reputation.json keeps of it, its last try, zero
+// before the first, and its last good copy, what that copy says, and how
+// many fetches of it failed.
+type list struct {
+	kept     List
+	entries  entries
+	failures int
+}
+
+// entries are what the lines of a copy say: for a blocklist, the netblocks
+// it names, with the lengths among them, and for the file of AS:percent
+// lines, the percentage it gives each AS number.
+type entries struct {
+	netblocks map[netip.Prefix]bool
+	lengths   []int
+	percents  map[string]int64
+}
+
+// New returns the lists, without a copy of any yet.
+func New(params Params) *Lists {
+	l := &Lists{params: params, httpClient: &http.Client{}, lists: map[string]*list{}}
+
+	for _, listURL := range l.URLs() {
+		l.lists[listURL] = &list{kept: List{URL: listURL}}
+	}
+
+	return l
+}
+
+// URLs returns the URL of every list: the blocklists' in the order
+// SWWAF_BLOCKLIST_URLS names them, then SWWAF_ASN_LIMIT_PERCENT_URL.
+func (l *Lists) URLs() []string {
+	urls := slices.Clone(l.params.BlocklistURLs)
+	if l.params.ASNLimitPercentURL != "" {
+		urls = append(urls, l.params.ASNLimitPercentURL)
+	}
+
+	return urls
+}
+
+// ListedBy returns the URLs of the blocklists whose copy lists addr, in
+// the order SWWAF_BLOCKLIST_URLS names them.
+func (l *Lists) ListedBy(addr netip.Addr) []string {
+	l.mu.Lock()
+	defer l.mu.Unlock()
+
+	var listedBy []string
+
+	for _, listURL := range l.params.BlocklistURLs {
+		if l.lists[listURL].entries.contain(addr) {
+			listedBy = append(listedBy, listURL)
+		}
+	}
+
+	return listedBy
+}
+
+// ASNLimitPercent returns the percentage the copy of the file of
+// AS:percent lines gives asn, and whether it lists asn.
+func (l *Lists) ASNLimitPercent(asn string) (int64, bool) {
+	if l.params.ASNLimitPercentURL == "" {
+		return 0, false
+	}
+
+	l.mu.Lock()
+	defer l.mu.Unlock()
+
+	percent, listed := l.lists[l.params.ASNLimitPercentURL].entries.percents[asn]
+
+	return percent, listed
+}
+
+// Fetched returns when the copy in use of the list at listURL was
+// fetched, or zero while there is none.
+func (l *Lists) Fetched(listURL string) time.Time {
+	l.mu.Lock()
+	defer l.mu.Unlock()
+
+	return l.lists[listURL].kept.Fetched
+}
+
+// Failures returns how many fetches of the list at listURL failed.
+func (l *Lists) Failures(listURL string) int {
+	l.mu.Lock()
+	defer l.mu.Unlock()
+
+	return l.lists[listURL].failures
+}
+
+// Run fetches each list once Refresh has passed since it was last fetched
+// or tried, the later of the two, until ctx is done. A list never tried is
+// fetched at once, and so is one whose last try or copy, read from
+// reputation.json, is that old.
+func (l *Lists) Run(ctx context.Context) {
+	if len(l.lists) == 0 {
+		return
+	}
+
+	for ctx.Err() == nil {
+		next := l.fetchDue(ctx)
+		timer := time.NewTimer(next.Sub(l.params.Now()))
+
+		select {
+		case <-ctx.Done():
+		case <-timer.C:
+		}
+
+		timer.Stop()
+	}
+}
+
+// Snapshot returns each list that has been tried, with its copy, if it
+// has one, sorted by URL, as reputation.json lists them.
+func (l *Lists) Snapshot() []List {
+	l.mu.Lock()
+
+	tried := make([]List, 0, len(l.lists))
+
+	for _, held := range l.lists {
+		if !held.kept.Tried.IsZero() {
+			tried = append(tried, held.kept)
+		}
+	}
+
+	l.mu.Unlock()
+
+	slices.SortFunc(tried, func(a, b List) int {
+		return strings.Compare(a.URL, b.URL)
+	})
+
+	return tried
+}
+
+// Load puts lists, read from reputation.json, in place of the last tries
+// and copies held. A list Params does not name is dropped. A copy with a
+// line that parse refuses is an error, and then nothing changes.
+func (l *Lists) Load(lists []List) error {
+	found := make(map[string]entries, len(lists))
+
+	for _, kept := range lists {
+		if _, named := l.lists[kept.URL]; !named {
+			continue
+		}
+
+		read, err := l.parse(kept.URL, kept.Lines)
+		if err != nil {
+			return fmt.Errorf("the copy of %s: %w", kept.URL, err)
+		}
+
+		found[kept.URL] = read
+	}
+
+	l.mu.Lock()
+	defer l.mu.Unlock()
+
+	for listURL, held := range l.lists {
+		held.kept, held.entries = List{URL: listURL}, entries{}
+	}
+
+	for _, kept := range lists {
+		read, named := found[kept.URL]
+		if named {
+			l.lists[kept.URL].kept, l.lists[kept.URL].entries = kept, read
+		}
+	}
+
+	return nil
+}
+
+// fetchDue fetches each list that is due, one after another, and returns
+// when the next is due. Once ctx has ended, it starts none, since a fetch
+// cut off is noted as a try.
+func (l *Lists) fetchDue(ctx context.Context) time.Time {
+	var next time.Time
+
+	for _, listURL := range l.URLs() {
+		due := l.due(listURL)
+		if ctx.Err() == nil && !l.params.Now().Before(due) {
+			l.fetch(ctx, listURL)
+			due = l.due(listURL)
+		}
+
+		if next.IsZero() || due.Before(next) {
+			next = due
+		}
+	}
+
+	return next
+}
+
+// due returns when the list at listURL is to be fetched: Refresh after it
+// was last fetched or tried, the later of the two.
+func (l *Lists) due(listURL string) time.Time {
+	l.mu.Lock()
+	defer l.mu.Unlock()
+
+	held := l.lists[listURL]
+
+	last := held.kept.Fetched
+	if held.kept.Tried.After(last) {
+		last = held.kept.Tried
+	}
+
+	return last.Add(l.params.Refresh)
+}
+
+// fetch fetches the list at listURL, and notes the try. A good copy takes
+// the place of the one held. A failure leaves that in use, and is counted,
+// logged and raised as a source_failure alert. A fetch cut off as ctx
+// ends, as smallwebwaf stops, is no failure, but is still noted as a try,
+// so that a restart waits for it: the server may have had its request.
+func (l *Lists) fetch(ctx context.Context, listURL string) {
+	lines, err := l.get(ctx, listURL)
+
+	var found entries
+	if err == nil {
+		found, err = l.parse(listURL, lines)
+	}
+
+	cutOff := err != nil && ctx.Err() != nil
+	now := l.params.Now()
+
+	l.mu.Lock()
+
+	held := l.lists[listURL]
+	held.kept.Tried = now
+
+	if err == nil {
+		held.kept.Fetched, held.kept.Lines = now, lines
+		held.entries = found
+	} else if !cutOff {
+		held.failures++
+	}
+
+	l.mu.Unlock()
+
+	if cutOff {
+		return
+	}
+
+	if err != nil {
+		const failed = "fetching a list failed"
+
+		// Raised before it is logged, so that the alert is there once the
+		// log line is.
+		l.params.Alerts.Raise(alerts.Alert{
+			Event:  alerts.EventSourceFailure,
+			Reason: failed,
+			Detail: map[string]any{"source": listURL, "error": err.Error()},
+		})
+		l.params.ProcessLog.Warn(failed, "url", listURL, "error", err.Error())
+
+		return
+	}
+
+	l.params.ProcessLog.Info("fetched a list", "url", listURL, "lines", len(lines))
+}
+
+// get fetches the list at listURL, and returns its lines. An answer other
+// than 200, or a list longer than maxListBytes, is a failure.
+func (l *Lists) get(ctx context.Context, listURL string) ([]string, error) {
+	ctx, cancel := context.WithTimeout(ctx, fetchTimeout)
+	defer cancel()
+
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, listURL, http.NoBody)
+	if err != nil {
+		return nil, fmt.Errorf("make the request: %w", err)
+	}
+
+	res, err := l.httpClient.Do(req)
+	if err != nil {
+		// Do's error names the URL, which the log line and the alert name
+		// already: only what went wrong is kept.
+		return nil, fmt.Errorf("fetch the list: %w", errors.Unwrap(err))
+	}
+
+	defer func() {
+		_ = res.Body.Close()
+	}()
+
+	if res.StatusCode != http.StatusOK {
+		return nil, fmt.Errorf("%w %s", errStatus, res.Status)
+	}
+
+	body, err := io.ReadAll(io.LimitReader(res.Body, maxListBytes+1))
+	if err != nil {
+		return nil, fmt.Errorf("read the list: %w", err)
+	}
+
+	if len(body) > maxListBytes {
+		return nil, errTooLong
+	}
+
+	lines := []string{}
+	for line := range strings.Lines(string(body)) {
+		lines = append(lines, strings.TrimSuffix(line, "\n"))
+	}
+
+	return lines, nil
+}
+
+// parse reads the lines of the list at listURL: those of a blocklist, or
+// of the file of AS:percent lines. Anything after a ; or a # on a line is
+// left out, and so is a line left blank. Any other line that does not read
+// is an error naming it by its number.
+func (l *Lists) parse(listURL string, lines []string) (entries, error) {
+	if listURL == l.params.ASNLimitPercentURL {
+		return parsePercents(lines)
+	}
+
+	return parseNetblocks(lines)
+}
+
+// parseNetblocks reads a blocklist's lines, each an address or a netblock
+// as the settings take them.
+func parseNetblocks(lines []string) (entries, error) {
+	found := entries{netblocks: map[netip.Prefix]bool{}}
+
+	for i, line := range lines {
+		text := withoutComment(line)
+		if text == "" {
+			continue
+		}
+
+		netblock, ok := parseNetblock(text)
+		if !ok {
+			return entries{}, fmt.Errorf("line %d %w", i+1, errNotNetblock)
+		}
+
+		found.netblocks[netblock] = true
+
+		if !slices.Contains(found.lengths, netblock.Bits()) {
+			found.lengths = append(found.lengths, netblock.Bits())
+		}
+	}
+
+	return found, nil
+}
+
+// parseNetblock reads text, a line of a blocklist, and reports whether it
+// is an address or a netblock as the settings take them. A client's IPv4
+// address is checked as IPv4, never IPv4-mapped, so an IPv4-mapped line,
+// such as ::ffff:192.0.2.0/120, is read as the IPv4 address or netblock it
+// stands for, 192.0.2.0/24, and a mapped netblock shorter than /96, which
+// stands for none, is refused.
+func parseNetblock(text string) (netip.Prefix, bool) {
+	netblock, err := config.ParseNetblock(text)
+	if err != nil {
+		return netip.Prefix{}, false
+	}
+
+	// The address as written: ParseNetblock's has the bits past the
+	// netblock's length cleared, the ::ffff among them below /96.
+	written, _, _ := strings.Cut(text, "/")
+	if addr, _ := netip.ParseAddr(written); !addr.Is4In6() {
+		return netblock, true
+	}
+
+	if netblock.Bits() < mappedBits {
+		return netip.Prefix{}, false
+	}
+
+	return netip.PrefixFrom(netblock.Addr().Unmap(), netblock.Bits()-mappedBits), true
+}
+
+// parsePercents reads the lines of the file of AS:percent lines, each an
+// AS number, : and a percentage, as SWWAF_ASN_LIMIT_PERCENT takes them. An
+// AS number listed more than once gets the lowest of its percentages.
+func parsePercents(lines []string) (entries, error) {
+	found := entries{percents: map[string]int64{}}
+
+	for i, line := range lines {
+		text := withoutComment(line)
+		if text == "" {
+			continue
+		}
+
+		asnText, percentText, _ := strings.Cut(text, ":")
+		asn, asnErr := config.ParseASN(asnText)
+
+		percent, percentErr := config.ParsePercent(percentText)
+		if asnErr != nil || percentErr != nil {
+			return entries{}, fmt.Errorf("line %d %w", i+1, errNotASNPercent)
+		}
+
+		earlier, listed := found.percents[asn]
+		if !listed || percent < earlier {
+			found.percents[asn] = percent
+		}
+	}
+
+	return found, nil
+}
+
+// withoutComment returns line without anything after a ; or a #, and
+// without the spaces around what is left.
+func withoutComment(line string) string {
+	text, _, _ := strings.Cut(line, ";")
+	text, _, _ = strings.Cut(text, "#")
+
+	return strings.TrimSpace(text)
+}
+
+// contain reports whether the netblocks of a blocklist's copy hold addr:
+// whether addr, cut to one of their lengths, is one of them.
+func (e entries) contain(addr netip.Addr) bool {
+	for _, length := range e.lengths {
+		netblock, err := addr.Prefix(length)
+		if err == nil && e.netblocks[netblock] {
+			return true
+		}
+	}
+
+	return false
+}

internal/reputation/reputation_test.go

@@ -0,0 +1,610 @@
+package reputation_test
+
+import (
+	"bytes"
+	"context"
+	"fmt"
+	"io"
+	"log/slog"
+	"net/http"
+	"net/netip"
+	"net/url"
+	"reflect"
+	"slices"
+	"strings"
+	"sync"
+	"testing"
+	"testing/synctest"
+	"time"
+
+	"sneak.berlin/go/smallwebwaf/internal/alerts"
+	"sneak.berlin/go/smallwebwaf/internal/reputation"
+)
+
+// The tests run in a synctest bubble, where the time package runs on a
+// clock of the test's own: time.Sleep moves it on at once, and
+// synctest.Wait returns once Run waits for the next list to be due, so
+// that every fetch due by then has been made. The stand-in for the
+// servers the lists are fetched from answers without the network, since a
+// fetch waiting on the network would keep that clock from moving on.
+
+const (
+	// dropURL and torURL are the blocklists, and asnURL the file of
+	// AS:percent lines.
+	dropURL = "https://lists.example/drop.txt"
+	torURL  = "https://lists.example/tor.txt"
+	asnURL  = "https://lists.example/asn.txt"
+	// refresh is the tests' SWWAF_BLOCKLIST_REFRESH, and cooldown their
+	// SWWAF_ALERT_COOLDOWN, longer than it.
+	refresh  = 24 * time.Hour
+	cooldown = 48 * time.Hour
+	// drop is a blocklist as the Spamhaus DROP list is written, with an
+	// address and a netblock in each of its comments, which list nothing.
+	drop = "; Spamhaus DROP List 2026/10/07 - (c) 2026 The Spamhaus Project SLL\n" +
+		"; Last-Modified: Wed, 07 Oct 2026 00:00:00 GMT ; 192.0.2.1\n" +
+		"# 198.51.100.0/24\n" +
+		"\n" +
+		"203.0.113.0/24 ; SBL1\n" +
+		"  192.0.2.9  # one address\n" +
+		"2001:db8:1::/48 ; SBL2\n"
+)
+
+func TestListedAddressesAndNetblocksWithTheCommentsLeftOut(t *testing.T) {
+	t.Parallel()
+
+	synctest.Test(t, func(t *testing.T) {
+		servers := &standIn{lists: map[string]string{dropURL: drop}}
+		lists := start(t, servers, params(dropURL))
+
+		for addr, want := range map[string][]string{
+			"203.0.113.0":   {dropURL},
+			"203.0.113.255": {dropURL},
+			"192.0.2.9":     {dropURL},
+			"2001:db8:1::7": {dropURL},
+			"203.0.114.0":   nil,
+			"192.0.2.8":     nil,
+			"192.0.2.1":     nil,
+			"198.51.100.7":  nil,
+			"2001:db8:2::7": nil,
+		} {
+			wantListedBy(t, lists, addr, want...)
+		}
+	})
+}
+
+func TestIPv4MappedLineListsTheIPv4AddressOrNetblockItStandsFor(t *testing.T) {
+	t.Parallel()
+
+	now := time.Now()
+	lists := reputation.New(params(dropURL))
+
+	err := lists.Load([]reputation.List{{
+		URL: dropURL, Tried: now, Fetched: now,
+		Lines: []string{"::ffff:192.0.2.9", "::ffff:203.0.113.0/120"},
+	}})
+	if err != nil {
+		t.Fatalf("load: %v", err)
+	}
+
+	for addr, want := range map[string][]string{
+		"192.0.2.9":     {dropURL},
+		"203.0.113.255": {dropURL},
+		"192.0.2.8":     nil,
+		"203.0.114.0":   nil,
+	} {
+		wantListedBy(t, lists, addr, want...)
+	}
+
+	// A mapped netblock shorter than /96 stands for no IPv4 one.
+	err = lists.Load([]reputation.List{{
+		URL: dropURL, Tried: now, Fetched: now, Lines: []string{"::ffff:198.51.100.0/88"},
+	}})
+
+	const want = "the copy of " + dropURL +
+		": line 1 is not an address or a netblock, such as 192.0.2.0/24"
+	if err == nil || err.Error() != want {
+		t.Errorf("error %v, want %s", err, want)
+	}
+}
+
+func TestClientIsListedByEachBlocklistThatListsIt(t *testing.T) {
+	t.Parallel()
+
+	synctest.Test(t, func(t *testing.T) {
+		servers := &standIn{lists: map[string]string{
+			dropURL: "203.0.113.0/24\n", torURL: "203.0.113.9\n",
+		}}
+		lists := start(t, servers, params(torURL, dropURL))
+
+		// In the order SWWAF_BLOCKLIST_URLS names them.
+		wantListedBy(t, lists, "203.0.113.9", torURL, dropURL)
+		wantListedBy(t, lists, "203.0.113.8", dropURL)
+	})
+}
+
+func TestListFetchedAgainOnceRefreshHasPassed(t *testing.T) {
+	t.Parallel()
+
+	synctest.Test(t, func(t *testing.T) {
+		servers := &standIn{lists: map[string]string{dropURL: "203.0.113.9\n"}}
+		lists := start(t, servers, params(dropURL))
+		began := time.Now()
+
+		wantFetches(t, servers, 1)
+
+		servers.set(dropURL, "203.0.113.10\n")
+		time.Sleep(refresh - time.Nanosecond)
+		wantFetches(t, servers, 1)
+		wantListedBy(t, lists, "203.0.113.9", dropURL)
+
+		time.Sleep(time.Nanosecond)
+		wantFetches(t, servers, 2)
+		wantListedBy(t, lists, "203.0.113.9")
+		wantListedBy(t, lists, "203.0.113.10", dropURL)
+
+		if fetched := lists.Fetched(dropURL); !fetched.Equal(began.Add(refresh)) {
+			t.Errorf("the copy in use was fetched at %s, want %s", fetched,
+				began.Add(refresh))
+		}
+	})
+}
+
+func TestFailedFetchKeepsTheLastGoodCopyAndAlertsOncePerCooldown(t *testing.T) {
+	t.Parallel()
+
+	for _, tc := range []struct {
+		name string
+		// fail has the stand-in answer the fetches after the first so that
+		// they fail with error.
+		fail  func(servers *standIn)
+		error string
+	}{
+		{
+			"an answer other than 200",
+			func(servers *standIn) { servers.set(dropURL, "") },
+			"the server answered 503 Service Unavailable",
+		},
+		{
+			"a line that does not read",
+			func(servers *standIn) { servers.set(dropURL, "203.0.113.10\n<html>\n") },
+			"line 2 is not an address or a netblock, such as 192.0.2.0/24",
+		},
+		{
+			"a list longer than 16 MiB",
+			func(servers *standIn) {
+				servers.set(dropURL, strings.Repeat("#\n", 8<<20+1))
+			},
+			"the list is longer than 16 MiB",
+		},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+
+			synctest.Test(t, func(t *testing.T) {
+				var log bytes.Buffer
+
+				servers := &standIn{lists: map[string]string{dropURL: drop}}
+				queue := newQueue()
+				p := params(dropURL)
+				p.ProcessLog = slog.New(slog.NewJSONHandler(&log, nil))
+				p.Alerts = queue
+				lists := start(t, servers, p)
+				kept := lists.Snapshot()
+
+				tc.fail(servers)
+
+				// Each failure is tried again once refresh has passed since it.
+				for range 2 {
+					time.Sleep(refresh)
+					synctest.Wait()
+				}
+
+				wantFetches(t, servers, 3)
+				wantListedBy(t, lists, "203.0.113.9", dropURL)
+
+				want := kept[0]
+				want.Tried = time.Now()
+
+				if got := lists.Snapshot(); !reflect.DeepEqual(got, []reputation.List{want}) {
+					t.Errorf("lists %+v, want the first copy, last tried now, %+v", got, want)
+				}
+
+				if lists.Failures(dropURL) != 2 {
+					t.Errorf("%d failures, want 2", lists.Failures(dropURL))
+				}
+
+				// One alert for the first failure; the cooldown holds back the
+				// second.
+				wantAlert(t, queue, alerts.Alert{
+					Time:   time.Now().Add(-refresh),
+					Event:  alerts.EventSourceFailure,
+					Reason: "fetching a list failed",
+					Detail: map[string]any{"source": dropURL, "error": tc.error},
+				})
+
+				if !strings.Contains(log.String(), `"msg":"fetching a list failed",`+
+					`"url":"`+dropURL+`","error":"`+tc.error) {
+					t.Errorf("logged\n%s\nwant the failures", log.String())
+				}
+			})
+		})
+	}
+}
+
+func TestFetchNotDoneWithinAMinuteFails(t *testing.T) {
+	t.Parallel()
+
+	synctest.Test(t, func(t *testing.T) {
+		servers := &standIn{lists: map[string]string{}, hanging: true}
+		lists := start(t, servers, params(dropURL))
+
+		time.Sleep(time.Minute - time.Nanosecond)
+		synctest.Wait()
+
+		if lists.Failures(dropURL) != 0 {
+			t.Errorf("%d failures before a minute, want none", lists.Failures(dropURL))
+		}
+
+		time.Sleep(time.Nanosecond)
+		synctest.Wait()
+
+		if lists.Failures(dropURL) != 1 {
+			t.Errorf("%d failures after a minute, want 1", lists.Failures(dropURL))
+		}
+	})
+}
+
+func TestKeptCopyIsFetchedAgainOnceRefreshHasPassedSinceItWasFetched(t *testing.T) {
+	t.Parallel()
+
+	synctest.Test(t, func(t *testing.T) {
+		servers := &standIn{lists: map[string]string{
+			dropURL: "203.0.113.10\n", torURL: "198.51.100.10\n",
+		}}
+		lists := reputation.New(params(dropURL, torURL))
+		lists.SetTransport(servers)
+
+		// drop.txt was fetched an hour ago, and tor.txt a refresh ago, as
+		// reputation.json says at start.
+		err := lists.Load([]reputation.List{
+			{URL: dropURL, Fetched: time.Now().Add(-time.Hour), Lines: []string{"203.0.113.9"}},
+			{URL: torURL, Fetched: time.Now().Add(-refresh), Lines: []string{"198.51.100.9"}},
+		})
+		if err != nil {
+			t.Fatalf("load: %v", err)
+		}
+
+		run(t, lists)
+
+		wantFetches(t, servers, 1)
+		wantListedBy(t, lists, "203.0.113.9", dropURL)
+		wantListedBy(t, lists, "198.51.100.10", torURL)
+
+		time.Sleep(refresh - time.Hour - time.Nanosecond)
+		wantFetches(t, servers, 1)
+
+		time.Sleep(time.Nanosecond)
+		wantFetches(t, servers, 2)
+		wantListedBy(t, lists, "203.0.113.10", dropURL)
+	})
+}
+
+func TestRestartWaitsRefreshAfterTheLastTryEvenOneThatFailed(t *testing.T) {
+	t.Parallel()
+
+	synctest.Test(t, func(t *testing.T) {
+		// drop.txt is fetched, and a refresh later the fetch downloads it
+		// whole but fails on a line that does not read.
+		servers := &standIn{lists: map[string]string{dropURL: "198.51.100.1\n"}}
+		lists := start(t, servers, params(dropURL))
+
+		servers.set(dropURL, "198.51.100.2\n<html>\n")
+		time.Sleep(refresh)
+		wantFetches(t, servers, 2)
+
+		// Restarted with what reputation.json keeps, it waits a refresh
+		// after the failed try, as it does while it runs.
+		restarted := &standIn{lists: map[string]string{dropURL: "198.51.100.2\n"}}
+		again := reputation.New(params(dropURL))
+		again.SetTransport(restarted)
+
+		err := again.Load(lists.Snapshot())
+		if err != nil {
+			t.Fatalf("load: %v", err)
+		}
+
+		run(t, again)
+		wantFetches(t, restarted, 0)
+		wantListedBy(t, again, "198.51.100.1", dropURL)
+
+		time.Sleep(refresh - time.Nanosecond)
+		wantFetches(t, restarted, 0)
+
+		time.Sleep(time.Nanosecond)
+		wantFetches(t, restarted, 1)
+		wantListedBy(t, again, "198.51.100.2", dropURL)
+	})
+}
+
+func TestFetchCutOffAsItStopsIsNoFailureButARestartWaitsForIt(t *testing.T) {
+	t.Parallel()
+
+	synctest.Test(t, func(t *testing.T) {
+		// Stopped 30 seconds into the fetch of drop.txt, before tor.txt's.
+		servers := &standIn{lists: map[string]string{}, hanging: true}
+		queue := newQueue()
+		p := params(dropURL, torURL)
+		p.Alerts = queue
+
+		lists := reputation.New(p)
+		lists.SetTransport(servers)
+
+		ctx, stop := context.WithCancel(t.Context())
+		stopped := make(chan struct{})
+
+		go func() {
+			lists.Run(ctx)
+			close(stopped)
+		}()
+
+		time.Sleep(30 * time.Second)
+		stop()
+		<-stopped
+
+		wantFetches(t, servers, 1)
+
+		if lists.Failures(dropURL) != 0 || len(waiting(queue)) != 0 {
+			t.Errorf("%d failures and alerts %+v, want none", lists.Failures(dropURL),
+				waiting(queue))
+		}
+
+		// Restarted an hour later with what reputation.json keeps, it fetches
+		// tor.txt, never tried, at once, and drop.txt a refresh after its
+		// cut-off try.
+		time.Sleep(time.Hour)
+
+		restarted := &standIn{lists: map[string]string{
+			dropURL: "203.0.113.7\n", torURL: "198.51.100.7\n",
+		}}
+		again := reputation.New(params(dropURL, torURL))
+		again.SetTransport(restarted)
+
+		err := again.Load(lists.Snapshot())
+		if err != nil {
+			t.Fatalf("load: %v", err)
+		}
+
+		run(t, again)
+		wantFetches(t, restarted, 1)
+		wantListedBy(t, again, "198.51.100.7", torURL)
+
+		time.Sleep(refresh - time.Hour - time.Nanosecond)
+		wantFetches(t, restarted, 1)
+
+		time.Sleep(time.Nanosecond)
+		wantFetches(t, restarted, 2)
+		wantListedBy(t, again, "203.0.113.7", dropURL)
+	})
+}
+
+func TestASNLimitPercentFileGivesEachASNumberItsLowestPercentage(t *testing.T) {
+	t.Parallel()
+
+	synctest.Test(t, func(t *testing.T) {
+		servers := &standIn{lists: map[string]string{
+			asnURL: "# hosting networks\nAS14061:50 ; DigitalOcean\nas16276:25\n\n" +
+				"AS14061:10\nAS14061:30\n",
+		}}
+		p := params()
+		p.ASNLimitPercentURL = asnURL
+		lists := start(t, servers, p)
+
+		for asn, want := range map[string]int64{"AS14061": 10, "AS16276": 25} {
+			percent, listed := lists.ASNLimitPercent(asn)
+			if !listed || percent != want {
+				t.Errorf("%s has %d (listed %t), want %d", asn, percent, listed, want)
+			}
+		}
+
+		if _, listed := lists.ASNLimitPercent("AS64496"); listed {
+			t.Error("AS64496 is listed")
+		}
+
+		// A line that does not read fails the fetch.
+		servers.set(asnURL, "AS14061:50\nAS16276\n")
+		time.Sleep(refresh)
+		synctest.Wait()
+
+		if lists.Failures(asnURL) != 1 {
+			t.Errorf("%d failures, want 1", lists.Failures(asnURL))
+		}
+	})
+}
+
+func TestLoadDropsCopiesOfListsNotNamedAndRefusesOnesThatDoNotRead(t *testing.T) {
+	t.Parallel()
+
+	fetched := time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)
+	kept := reputation.List{
+		URL: dropURL, Tried: fetched, Fetched: fetched, Lines: []string{"203.0.113.9"},
+	}
+	lists := reputation.New(params(dropURL))
+
+	err := lists.Load([]reputation.List{kept, {
+		URL: torURL, Tried: fetched, Fetched: fetched, Lines: []string{"198.51.100.9"},
+	}})
+	if err != nil {
+		t.Fatalf("load: %v", err)
+	}
+
+	if got := lists.Snapshot(); !reflect.DeepEqual(got, []reputation.List{kept}) {
+		t.Errorf("copies %+v, want only %+v", got, kept)
+	}
+
+	err = lists.Load([]reputation.List{{URL: dropURL, Fetched: fetched, Lines: []string{
+		"; DROP", "203.0.113.300",
+	}}})
+
+	const want = "the copy of " + dropURL +
+		": line 2 is not an address or a netblock, such as 192.0.2.0/24"
+	if err == nil || err.Error() != want {
+		t.Errorf("error %v, want %s", err, want)
+	}
+
+	if got := lists.Snapshot(); !reflect.DeepEqual(got, []reputation.List{kept}) {
+		t.Errorf("copies %+v after the error, want %+v still", got, kept)
+	}
+}
+
+// standIn is a stand-in for the servers the lists are fetched from. It
+// notes the URL of each fetch.
+type standIn struct {
+	mu sync.Mutex
+	// lists are what it answers with, by URL; it answers a URL it has no
+	// list for with 503, and none at all while hanging.
+	lists   map[string]string
+	hanging bool
+	fetches []string
+}
+
+// RoundTrip has the stand-in answer req, in place of the network. A fetch
+// abandoned before the stand-in answers fails, as over the network.
+func (s *standIn) RoundTrip(req *http.Request) (*http.Response, error) {
+	s.mu.Lock()
+	s.fetches = append(s.fetches, req.URL.String())
+	list, found := s.lists[req.URL.String()]
+	hanging := s.hanging
+	s.mu.Unlock()
+
+	if hanging {
+		<-req.Context().Done()
+
+		return nil, req.Context().Err()
+	}
+
+	status := http.StatusOK
+	if !found {
+		status = http.StatusServiceUnavailable
+	}
+
+	return &http.Response{
+		StatusCode: status,
+		Status:     fmt.Sprintf("%d %s", status, http.StatusText(status)),
+		Header:     http.Header{},
+		Body:       io.NopCloser(strings.NewReader(list)),
+		Request:    req,
+	}, nil
+}
+
+// set has the stand-in answer listURL with list, or with 503 for "".
+func (s *standIn) set(listURL, list string) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	if list == "" {
+		delete(s.lists, listURL)
+
+		return
+	}
+
+	s.lists[listURL] = list
+}
+
+// params returns the Params of the blocklists at urls, refreshed every
+// refresh, by the bubble's clock, with alerts to a queue that sends none.
+func params(urls ...string) reputation.Params {
+	return reputation.Params{
+		BlocklistURLs: urls,
+		Refresh:       refresh,
+		Now:           time.Now,
+		ProcessLog:    slog.New(slog.DiscardHandler),
+		Alerts:        newQueue(),
+	}
+}
+
+// newQueue returns a queue of alerts to a webhook that is never sent
+// them, with a cooldown of cooldown.
+func newQueue() *alerts.Queue {
+	return alerts.New(alerts.Params{
+		WebhookURL: &url.URL{Scheme: "https", Host: "alerts.example"},
+		Events:     alerts.Events(),
+		Cooldown:   cooldown,
+		Now:        time.Now,
+	})
+}
+
+// start returns the lists of p, fetched through servers by Run, which runs
+// until the test ends, once Run has fetched those due at start.
+func start(t *testing.T, servers *standIn, p reputation.Params) *reputation.Lists {
+	t.Helper()
+
+	lists := reputation.New(p)
+	lists.SetTransport(servers)
+	run(t, lists)
+
+	return lists
+}
+
+// run runs lists' Run until the test ends, and waits until it has fetched
+// the lists due.
+func run(t *testing.T, lists *reputation.Lists) {
+	t.Helper()
+
+	ctx, stop := context.WithCancel(t.Context())
+	stopped := make(chan struct{})
+
+	go func() {
+		lists.Run(ctx)
+		close(stopped)
+	}()
+
+	t.Cleanup(func() {
+		stop()
+		<-stopped
+	})
+
+	synctest.Wait()
+}
+
+// wantFetches waits until Run has made the fetches due, and checks how
+// many the servers have had.
+func wantFetches(t *testing.T, servers *standIn, want int) {
+	t.Helper()
+
+	synctest.Wait()
+
+	servers.mu.Lock()
+	got := len(servers.fetches)
+	servers.mu.Unlock()
+
+	if got != want {
+		t.Errorf("%d fetches, want %d", got, want)
+	}
+}
+
+// wantListedBy checks the URLs of the blocklists lists says list addr.
+func wantListedBy(t *testing.T, lists *reputation.Lists, addr string, want ...string) {
+	t.Helper()
+
+	got := lists.ListedBy(netip.MustParseAddr(addr))
+	if !slices.Equal(got, want) {
+		t.Errorf("%s is listed by %v, want %v", addr, got, want)
+	}
+}
+
+// waiting returns the alerts waiting in queue.
+func waiting(queue *alerts.Queue) []alerts.Alert {
+	return queue.Snapshot().Waiting[alerts.DestinationWebhook]
+}
+
+// wantAlert checks that want is the one alert waiting in queue, and that
+// the cooldown has held back one repeat of it.
+func wantAlert(t *testing.T, queue *alerts.Queue, want alerts.Alert) {
+	t.Helper()
+
+	got := waiting(queue)
+	if len(got) != 1 || !reflect.DeepEqual(got[0], want) || queue.Suppressed() != 1 {
+		t.Errorf("alerts waiting %+v, %d held back, want only %+v and 1", got,
+			queue.Suppressed(), want)
+	}
+}

internal/requestlog/requestlog.go

@@ -35,7 +35,8 @@ const (
 	// rule.
 	ActionRuleBlocked = "rule_blocked"
 	// ActionDenied is a request refused because its client is in
-	// SWWAF_DENY_NETS.
+	// SWWAF_DENY_NETS, or in a blocklist while SWWAF_BLOCKLIST_ACTION is
+	// deny.
 	ActionDenied = "denied"
 	// ActionCountryDenied is a request refused for its client's country.
 	ActionCountryDenied = "country_denied"
@@ -117,6 +118,15 @@ type Line struct {
 	// ActionBanned, ActionCountryDenied, ActionRateLimited or
 	// ActionRuleBlocked.
 	WouldAction string `json:"would_action,omitempty"`
+	// LimitPercent and LimitPercentSetting are, for a request the rate
+	// limits counted whose client a biased threshold gives a percentage of
+	// the rate limits below 100, that percentage and the setting that gave
+	// it. BytesPercent and BytesPercentSetting are the same for the byte
+	// limits.
+	LimitPercent        *int64 `json:"limit_percent,omitempty"`
+	LimitPercentSetting string `json:"limit_percent_setting,omitempty"`
+	BytesPercent        *int64 `json:"bytes_percent,omitempty"`
+	BytesPercentSetting string `json:"bytes_percent_setting,omitempty"`
 	// Counts are, for a request the rate limits counted, the client's
 	// requests as they counted them with this one, and its bytes as the
 	// byte limits counted them, with this request's once it has ended if
@@ -128,6 +138,8 @@ type Line struct {
 	// Counts names its count: minute, hour or day for a rate limit, and
 	// minute_bytes, hour_bytes or day_bytes for a byte limit.
 	LimitHit string `json:"limit_hit,omitempty"`
+	// Reputation are the URLs of the blocklists that list the client.
+	Reputation []string `json:"reputation,omitempty"`
 	// Offence is the offence the request was held as, OffenceLimit.
 	Offence string `json:"offence,omitempty"`
 	// BanExpires is when the ban the request made, or was refused under,

internal/smallwebwaf/smallwebwaf.go

@@ -200,7 +200,9 @@ func loadStateFiles(
 		Ledger:          server.Ledger,
 		Limiter:         server.Limiter,
 		GeoJS:           server.GeoJS,
+		Lists:           server.Lists,
 		Alerts:          alertQueue,
+		Anomalies:       server.Anomalies,
 		Now:             now,
 		ProcessLog:      processLog,
 		Metrics:         server.Metrics,
@@ -266,8 +268,9 @@ func startSending(
 
 // serve serves requests on listener, writes the state files as they are
 // due, takes in an admin's edits of them, reads the rule files again as
-// they change, and the lookup database when it is replaced, and sends the
-// alerts, until ctx is done. Then it gives the requests in progress
+// they change, and the lookup database when it is replaced, fetches the
+// lists the settings name by URL as they are due, and sends the alerts,
+// until ctx is done. Then it gives the requests in progress
 // shutdownTimeout to finish, and writes every state file, alerts.json with
 // the alerts still waiting.
 func serve(
@@ -292,6 +295,7 @@ func serve(
 			server.LookupFile.Watch(writing)
 		}
 	})
+	listsFetched := inBackground(func() { server.Lists.Run(writing) })
 	alertsSent := inBackground(func() { alertQueue.Run(writing) })
 
 	select {
@@ -335,6 +339,7 @@ func serve(
 	<-watched
 	<-rulesWatched
 	<-lookupFileWatched
+	<-listsFetched
 	<-alertsSent
 
 	err = files.WriteAll()

internal/smallwebwaf/smallwebwaf_test.go

@@ -38,6 +38,7 @@ const (
 	stateWriteDelay      = "SWWAF_STATE_WRITE_DELAY"
 	stateCounterInterval = "SWWAF_STATE_COUNTER_INTERVAL"
 	rateLimitPerDay      = "SWWAF_RATE_LIMIT_PER_DAY"
+	rateLimitExemptNets  = "SWWAF_RATE_LIMIT_EXEMPT_NETS"
 	rulesDir             = "SWWAF_RULES_DIR"
 	lookupSource         = "SWWAF_LOOKUP_SOURCE"
 	lookupDBPath         = "SWWAF_LOOKUP_DB_PATH"
@@ -522,7 +523,7 @@ func TestLookupDatabaseReplacedWhileRunningTakesEffect(t *testing.T) {
 		// The requests sent until a replacement takes effect, and those
 		// for the metrics, must not break a rate limit, whose ban would
 		// refuse them too.
-		"SWWAF_RATE_LIMIT_EXEMPT_NETS": placed + "," + localhost,
+		rateLimitExemptNets: placed + "," + localhost,
 	}
 	began := time.Now()
 	// Each replacement is written beside the file and renamed over it, as
@@ -573,6 +574,111 @@ func TestLookupDatabaseReplacedWhileRunningTakesEffect(t *testing.T) {
 	}
 }
 
+func TestBlocklistTriesAndCopiesKeptInReputationJSONAcrossRestarts(t *testing.T) {
+	t.Parallel()
+
+	const (
+		token    = "0123456789abcdef0123456789abcdef"
+		torPath  = "/tor.txt"
+		dropPath = "/drop.txt"
+		// copyright is the DROP list's date and copyright line.
+		copyright = "; Spamhaus DROP List 2026/10/07 - (c) 2026 The Spamhaus Project SLL"
+	)
+
+	failing, torFetches := new(atomic.Bool), new(atomic.Int32)
+	lists := map[string]string{
+		torPath:  "198.51.100.0/24\n",
+		dropPath: copyright + "\n" + placed + " ; SBL1\n",
+	}
+	server := httptest.NewServer(http.HandlerFunc(
+		func(w http.ResponseWriter, r *http.Request) {
+			if r.URL.Path == torPath {
+				torFetches.Add(1)
+			}
+
+			if failing.Load() {
+				w.WriteHeader(http.StatusServiceUnavailable)
+
+				return
+			}
+
+			_, _ = io.WriteString(w, lists[r.URL.Path])
+		}))
+	t.Cleanup(server.Close)
+
+	torURL, dropURL := server.URL+torPath, server.URL+dropPath
+	dir := t.TempDir()
+	env := map[string]string{
+		listenAddr:             localhost + ":0",
+		upstreamURL:            startApp(t),
+		stateDir:               dir,
+		rulesDir:               t.TempDir(),
+		trustedProxies:         localhost + "/32",
+		metricsToken:           token,
+		instanceName:           instance,
+		"SWWAF_BLOCKLIST_URLS": torURL,
+		// The requests sent until the list takes effect, and those for the
+		// metrics, must not break a rate limit, whose ban would refuse them
+		// too.
+		rateLimitExemptNets: placed + "," + localhost,
+	}
+	failures := `smallwebwaf_reputation_failures_total{instance="fsn1app1/gitea",` +
+		`source="` + torURL + `"} `
+
+	// tor.txt cannot be fetched at first, which is counted.
+	failing.Store(true)
+	runUntilStopped(t, env, func(url string) {
+		metricsWith(t, url+"_smallwebwaf/metrics", token, failures+"1")
+	})
+
+	// Restarted with drop.txt named after it and the server answering, tor.txt
+	// waits SWWAF_BLOCKLIST_REFRESH after its failed try, kept in reputation.json,
+	// while drop.txt, never tried, is fetched at once. Lists are fetched in the
+	// order named, so once drop.txt refuses the client, tor.txt has had its turn.
+	failing.Store(false)
+
+	env["SWWAF_BLOCKLIST_URLS"] = torURL + "," + dropURL
+	out := runUntilStopped(t, env, func(url string) {
+		for statusFrom(t, url, placed) != http.StatusForbidden {
+			time.Sleep(pollInterval)
+		}
+	})
+	wantDeniedByList(t, out.line(t, "action", "denied"), dropURL)
+
+	if fetches := torFetches.Load(); fetches != 1 {
+		t.Errorf("tor.txt fetched %d times, want once, before the restart", fetches)
+	}
+
+	// After another restart, with the server failing, the copy of drop.txt
+	// kept in reputation.json, its copyright line included, refuses the
+	// client from the first request.
+	failing.Store(true)
+
+	out = runUntilStopped(t, env, func(url string) {
+		wantStatus(t, url, placed, http.StatusForbidden)
+	})
+	wantDeniedByList(t, out.line(t, "type", "request"), dropURL)
+
+	path := filepath.Join(dir, "reputation.json")
+
+	kept, err := os.ReadFile(path) //nolint:gosec // a file in the test's directory
+	if err != nil || !strings.Contains(string(kept), `"`+copyright+`"`) {
+		t.Errorf("reputation.json holds\n%s\nwant the copy with %q (%v)", kept, copyright,
+			err)
+	}
+}
+
+// wantDeniedByList checks that the request log line is of a request the
+// blocklist at listURL refused.
+func wantDeniedByList(t *testing.T, line map[string]any, listURL string) {
+	t.Helper()
+
+	reputation, _ := line["reputation"].([]any)
+	if line["action"] != "denied" || len(reputation) != 1 || reputation[0] != listURL {
+		t.Errorf("request log line %v, want one denied for %s", line, listURL)
+	}
+}
+
 func TestLookupDatabaseThatCannotBeReadStopsTheStart(t *testing.T) {
 	t.Parallel()
 
@@ -1014,7 +1120,7 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL, dir string) {
 		"SWWAF_REQUEST_MAX_BYTES":               "100M",
 		"SWWAF_RESPONSE_MAX_BYTES":              "5G",
 		"SWWAF_ALLOW_NETS":                      "",
-		"SWWAF_RATE_LIMIT_EXEMPT_NETS":          "",
+		rateLimitExemptNets:                     "",
 		"SWWAF_DENY_NETS":                       "",
 		"SWWAF_RATE_LIMIT_PER_MINUTE":           "1000",
 		"SWWAF_RATE_LIMIT_PER_HOUR":             "10000",

internal/state/state.go

@@ -1,8 +1,10 @@
 // Package state keeps smallwebwaf's state in JSON files in
 // SWWAF_STATE_DIR, as the "Persistent state" section of SPEC.md describes:
 // bans.json holds the bans, clients.json each client's counters and
-// history, lookups.json GeoJS's answers, and alerts.json the cooldowns,
-// the hour under way and the alerts waiting for each destination. Load
+// history, lookups.json GeoJS's answers, reputation.json the last try and
+// last good copy of each list fetched from a URL, and alerts.json the
+// cooldowns, the hour under way, the alerts waiting for each destination
+// and the anomaly counters. Load
 // reads them at start, Watch takes in an admin's edit of one while
 // smallwebwaf runs, and Run and WriteAll write them. The disk is read and
 // written outside the parts' locks, which are held only to take a
@@ -29,10 +31,12 @@ import (
 
 	"github.com/fsnotify/fsnotify"
 	"sneak.berlin/go/smallwebwaf/internal/alerts"
+	"sneak.berlin/go/smallwebwaf/internal/anomaly"
 	"sneak.berlin/go/smallwebwaf/internal/bans"
 	"sneak.berlin/go/smallwebwaf/internal/lookup"
 	"sneak.berlin/go/smallwebwaf/internal/metrics"
 	"sneak.berlin/go/smallwebwaf/internal/ratelimit"
+	"sneak.berlin/go/smallwebwaf/internal/reputation"
 )
 
 // version is the version of the files' format, the only one read.
@@ -44,10 +48,11 @@ const fileMode = 0o600
 
 // The state files' names.
 const (
-	bansJSON    = "bans.json"
-	clientsJSON = "clients.json"
-	lookupsJSON = "lookups.json"
-	alertsJSON  = "alerts.json"
+	bansJSON       = "bans.json"
+	clientsJSON    = "clients.json"
+	lookupsJSON    = "lookups.json"
+	reputationJSON = "reputation.json"
+	alertsJSON     = "alerts.json"
 )
 
 var (
@@ -56,6 +61,7 @@ var (
 	errMissing     = errors.New("has no")
 	errCause       = errors.New("is not limit, attack or admin")
 	errDestination = errors.New("is not webhook, slack or ntfy")
+	errScope       = errors.New("is not client, net, asn, total or watch")
 	errWaitingList = errors.New(`waiting is a list, but now lists the alerts by ` +
 		`destination: put the list under "webhook", as "waiting": {"webhook": [...]}, ` +
 		`or remove the file`)
@@ -70,13 +76,15 @@ type Params struct {
 	// is (SWWAF_STATE_COUNTER_INTERVAL).
 	WriteDelay      time.Duration
 	CounterInterval time.Duration
-	// Ledger, Limiter, GeoJS and Alerts hold the state. Alerts also
-	// receive a file_error alert for an edit set aside, and for a write
-	// that fails while smallwebwaf runs.
-	Ledger  *bans.Ledger
-	Limiter *ratelimit.Limiter
-	GeoJS   *lookup.GeoJS
-	Alerts  *alerts.Queue
+	// Ledger, Limiter, GeoJS, Lists, Alerts and Anomalies hold the state.
+	// Alerts also receive a file_error alert for an edit set aside, and for
+	// a write that fails while smallwebwaf runs.
+	Ledger    *bans.Ledger
+	Limiter   *ratelimit.Limiter
+	GeoJS     *lookup.GeoJS
+	Lists     *reputation.Lists
+	Alerts    *alerts.Queue
+	Anomalies *anomaly.Counters
 	// Now tells the time by which the counters' buckets run out, normally
 	// time.Now in UTC.
 	Now func() time.Time
@@ -134,12 +142,22 @@ type lookupsFile struct {
 	Lookups []lookup.Answer `json:"lookups"`
 }
 
+// reputationFile is reputation.json, indented for an admin to read and
+// edit, so that each line of a list's copy is on a line of its own.
+type reputationFile struct {
+	Version int               `json:"version"`
+	Lists   []reputation.List `json:"lists"`
+}
+
 // alertsFile is alerts.json, indented for an admin to read and edit.
+//
+//nolint:tagliatelle // the state files use snake_case, as the request log does
 type alertsFile struct {
-	Version   int                       `json:"version"`
-	Cooldowns []alerts.Cooldown         `json:"cooldowns"`
-	Hour      alerts.Hour               `json:"hour"`
-	Waiting   map[string][]alerts.Alert `json:"waiting"`
+	Version         int                       `json:"version"`
+	Cooldowns       []alerts.Cooldown         `json:"cooldowns"`
+	Hour            alerts.Hour               `json:"hour"`
+	Waiting         map[string][]alerts.Alert `json:"waiting"`
+	AnomalyCounters []anomaly.Counter         `json:"anomaly_counters"`
 }
 
 // stateFile is the struct of a state file. Once the file is decoded, its
@@ -152,10 +170,10 @@ type stateFile interface {
 }
 
 // Load checks that files can be written in Dir, and reads the state files
-// in it into the ledger, the limiter and GeoJS. A missing file is empty
-// state, as on a first start. A file that does not parse, has an unknown
-// version, or has an entry without a field it needs, is an error that
-// names the file and, where the JSON decoder tells it, the line and
+// in it into the parts of Params that hold the state. A missing file is
+// empty state, as on a first start. A file that does not parse, has an
+// unknown version, or has an entry without a field it needs, is an error
+// that names the file and, where the JSON decoder tells it, the line and
 // column, or else the entry.
 func Load(params Params) (*Files, error) {
 	err := checkWritable(params.Dir)
@@ -168,16 +186,17 @@ func Load(params Params) (*Files, error) {
 	bansRead, bansErr := f.read(bansJSON)
 	clientsRead, clientsErr := f.read(clientsJSON)
 	lookupsRead, lookupsErr := f.read(lookupsJSON)
+	reputationRead, reputationErr := f.read(reputationJSON)
 	alertsRead, alertsErr := f.read(alertsJSON)
 
-	err = errors.Join(bansErr, clientsErr, lookupsErr, alertsErr)
+	err = errors.Join(bansErr, clientsErr, lookupsErr, reputationErr, alertsErr)
 	if err != nil {
 		return nil, err
 	}
 
 	params.ProcessLog.Info("read the state files", "directory", params.Dir,
 		"bans", bansRead, "clients", clientsRead, "lookups", lookupsRead,
-		"alerts_waiting", alertsRead)
+		"lists", reputationRead, "alerts_waiting", alertsRead)
 
 	return f, nil
 }
@@ -206,7 +225,9 @@ func (f *Files) Run(ctx context.Context) {
 
 			f.logFailure(bansJSON, f.writeFile(bansJSON))
 		case <-interval.C:
-			for _, name := range []string{bansJSON, clientsJSON, lookupsJSON, alertsJSON} {
+			for _, name := range []string{
+				bansJSON, clientsJSON, lookupsJSON, reputationJSON, alertsJSON,
+			} {
 				f.logFailure(name, f.writeFile(name))
 			}
 		}
@@ -217,7 +238,7 @@ func (f *Files) Run(ctx context.Context) {
 // fails does not keep the others from being written.
 func (f *Files) WriteAll() error {
 	return errors.Join(f.writeFile(bansJSON), f.writeFile(clientsJSON),
-		f.writeFile(lookupsJSON), f.writeFile(alertsJSON))
+		f.writeFile(lookupsJSON), f.writeFile(reputationJSON), f.writeFile(alertsJSON))
 }
 
 // Watch watches Dir until ctx is done, and takes in an admin's edit of a
@@ -252,7 +273,7 @@ func (f *Files) Watch(ctx context.Context) {
 			return
 		case event := <-watcher.Events:
 			switch name := filepath.Base(event.Name); name {
-			case bansJSON, clientsJSON, lookupsJSON, alertsJSON:
+			case bansJSON, clientsJSON, lookupsJSON, reputationJSON, alertsJSON:
 				f.fileChanged(name)
 			}
 		case err = <-watcher.Errors:
@@ -398,32 +419,27 @@ func (f *Files) takeIn(name string, data []byte, edit bool) (int, error) {
 
 		f.params.GeoJS.Load(file.Lookups)
 		entries = len(file.Lookups)
-	case alertsJSON:
-		// waiting was a list, of the alerts waiting for the webhook, before
-		// alerts went to Slack and ntfy too.
-		var written struct {
-			Waiting json.RawMessage `json:"waiting"`
-		}
-
-		if json.Unmarshal(data, &written) == nil &&
-			bytes.HasPrefix(written.Waiting, []byte("[")) {
-			return 0, fmt.Errorf("%s: %w", path, errWaitingList)
-		}
-
-		var file alertsFile
+	case reputationJSON:
+		var file reputationFile
 
 		err := parse(path, data, &file)
 		if err != nil {
 			return 0, err
 		}
 
-		f.params.Alerts.Load(alerts.State{
-			Cooldowns: file.Cooldowns, Hour: file.Hour, Waiting: file.Waiting,
-		})
-
-		for _, waiting := range file.Waiting {
-			entries += len(waiting)
+		err = f.params.Lists.Load(file.Lists)
+		if err != nil {
+			return 0, fmt.Errorf("%s: %w", path, err)
 		}
+
+		entries = len(file.Lists)
+	case alertsJSON:
+		waiting, err := f.takeInAlerts(path, data)
+		if err != nil {
+			return 0, err
+		}
+
+		entries = waiting
 	}
 
 	f.sums[name] = sha256.Sum256(data)
@@ -431,6 +447,41 @@ func (f *Files) takeIn(name string, data []byte, edit bool) (int, error) {
 	return entries, nil
 }
 
+// takeInAlerts parses data, what alerts.json, at path, holds, puts it
+// into the alerts and the anomaly counters, in place of what they held,
+// and returns how many alerts wait in it, as takeIn describes.
+func (f *Files) takeInAlerts(path string, data []byte) (int, error) {
+	// waiting was a list, of the alerts waiting for the webhook, before
+	// alerts went to Slack and ntfy too.
+	var written struct {
+		Waiting json.RawMessage `json:"waiting"`
+	}
+
+	if json.Unmarshal(data, &written) == nil &&
+		bytes.HasPrefix(written.Waiting, []byte("[")) {
+		return 0, fmt.Errorf("%s: %w", path, errWaitingList)
+	}
+
+	var file alertsFile
+
+	err := parse(path, data, &file)
+	if err != nil {
+		return 0, err
+	}
+
+	f.params.Alerts.Load(alerts.State{
+		Cooldowns: file.Cooldowns, Hour: file.Hour, Waiting: file.Waiting,
+	})
+	f.params.Anomalies.Load(file.AnomalyCounters, f.params.Now())
+
+	entries := 0
+	for _, waiting := range file.Waiting {
+		entries += len(waiting)
+	}
+
+	return entries, nil
+}
+
 // writeFile writes the state file name from what smallwebwaf holds. An
 // edit made since smallwebwaf last read or wrote the file is taken in
 // first, so that it is not overwritten, or set aside if it does not
@@ -506,34 +557,38 @@ func (f *Files) setAside(name string, parseErr error) error {
 func (f *Files) encode(name string) ([]byte, error) {
 	switch name {
 	case bansJSON:
-		file := bansFile{Version: version, Bans: BanEntries(f.params.Ledger.Snapshot())}
-
-		data, err := json.MarshalIndent(file, "", "    ")
-		if err != nil {
-			return nil, err
-		}
-
-		return append(data, '\n'), nil
+		return encodeIndented(bansFile{
+			Version: version, Bans: BanEntries(f.params.Ledger.Snapshot()),
+		})
 	case clientsJSON:
 		return encodeOnePerLine("clients", f.params.Limiter.Snapshot())
 	case lookupsJSON:
 		return encodeOnePerLine("lookups", f.params.GeoJS.Snapshot())
+	case reputationJSON:
+		return encodeIndented(reputationFile{
+			Version: version, Lists: f.params.Lists.Snapshot(),
+		})
 	default: // alerts.json
 		held := f.params.Alerts.Snapshot()
-		file := alertsFile{
+
+		return encodeIndented(alertsFile{
 			Version: version, Cooldowns: held.Cooldowns, Hour: held.Hour,
-			Waiting: held.Waiting,
-		}
-
-		data, err := json.MarshalIndent(file, "", "    ")
-		if err != nil {
-			return nil, err
-		}
-
-		return append(data, '\n'), nil
+			Waiting: held.Waiting, AnomalyCounters: f.params.Anomalies.Snapshot(),
+		})
 	}
 }
 
+// encodeIndented encodes file, a state file's struct, indented for an
+// admin to read and edit.
+func encodeIndented(file any) ([]byte, error) {
+	data, err := json.MarshalIndent(file, "", "    ")
+	if err != nil {
+		return nil, err
+	}
+
+	return append(data, '\n'), nil
+}
+
 // BanEntries returns held as bans.json lists them, an empty list for
 // none.
 func BanEntries(held []bans.Ban) []BanEntry {
@@ -671,10 +726,32 @@ func (f *lookupsFile) check(data []byte) error {
 	return nil
 }
 
+// check refuses a list without its URL, which would name no list, or the
+// time it was last tried, which would have it fetched at once, and a copy
+// of it without the time it was fetched, or without its lines, which hold
+// the list.
+func (f *reputationFile) check([]byte) error {
+	for i, kept := range f.Lists {
+		switch {
+		case kept.URL == "":
+			return missing(i, "url")
+		case kept.Tried.IsZero():
+			return missing(i, "tried")
+		case kept.Fetched.IsZero() && kept.Lines != nil:
+			return missing(i, "fetched")
+		case kept.Lines == nil && !kept.Fetched.IsZero():
+			return missing(i, "lines")
+		}
+	}
+
+	return nil
+}
+
 // check refuses a cooldown without its event or when its alert was sent,
 // which would hold back no repeat, alerts waiting for a destination with
-// another name than webhook, slack or ntfy, most likely misspelt, and an
-// alert waiting without its event or its time.
+// another name than webhook, slack or ntfy, most likely misspelt, an
+// alert waiting without its event or its time, and an anomaly counter as
+// checkAnomalyCounters does.
 func (f *alertsFile) check([]byte) error {
 	for i, cooldown := range f.Cooldowns {
 		switch {
@@ -700,9 +777,58 @@ func (f *alertsFile) check([]byte) error {
 		}
 	}
 
+	return checkAnomalyCounters(f.AnomalyCounters)
+}
+
+// checkAnomalyCounters refuses an anomaly counter whose scope is not
+// client, net, asn, total or watch, most likely misspelt, and one without
+// a field it needs, as missingFromCounter tells.
+func checkAnomalyCounters(counters []anomaly.Counter) error {
+	for i, counter := range counters {
+		if !slices.Contains(anomaly.Scopes(), counter.Scope) {
+			return fmt.Errorf("anomaly_counters entry %d's scope %q %w", i+1,
+				counter.Scope, errScope)
+		}
+
+		field := missingFromCounter(counter)
+		if field != "" {
+			return fmt.Errorf("anomaly_counters %w", missing(i, field))
+		}
+	}
+
 	return nil
 }
 
+// missingFromCounter returns the first field counter, an anomaly counter,
+// needs and has not, or "" when it has them all: what tells it from the
+// others in its scope, without which it would never be counted again, the
+// netblock of a client, net or watch counter, the AS number of an asn one
+// and the name of a watch one; and the start of a window in which it has
+// requests or bytes, without which they would be dropped.
+func missingFromCounter(counter anomaly.Counter) string {
+	scope := counter.Scope
+
+	switch {
+	case scope != anomaly.ScopeASN && scope != anomaly.ScopeTotal &&
+		!counter.Netblock.IsValid():
+		return "netblock"
+	case scope == anomaly.ScopeASN && counter.ASN == "":
+		return "asn"
+	case scope == anomaly.ScopeWatch && counter.Name == "":
+		return "name"
+	case countsWithoutStart(counter.Minute):
+		return "minute.start"
+	case countsWithoutStart(counter.Hour):
+		return "hour.start"
+	case countsWithoutStart(counter.MinuteBytes):
+		return "minute_bytes.start"
+	case countsWithoutStart(counter.HourBytes):
+		return "hour_bytes.start"
+	default:
+		return ""
+	}
+}
+
 // countsWithoutStart reports whether b holds requests, or bytes, but no
 // start, which places them in time.
 func countsWithoutStart(b ratelimit.Buckets) bool {

internal/state/state_test.go

@@ -22,19 +22,25 @@ import (
 	"time"
 
 	"sneak.berlin/go/smallwebwaf/internal/alerts"
+	"sneak.berlin/go/smallwebwaf/internal/anomaly"
 	"sneak.berlin/go/smallwebwaf/internal/bans"
 	"sneak.berlin/go/smallwebwaf/internal/lookup"
 	"sneak.berlin/go/smallwebwaf/internal/metrics"
 	"sneak.berlin/go/smallwebwaf/internal/ratelimit"
+	"sneak.berlin/go/smallwebwaf/internal/reputation"
 	"sneak.berlin/go/smallwebwaf/internal/state"
 )
 
 const (
 	// The state files.
-	bansJSON    = "bans.json"
-	clientsJSON = "clients.json"
-	lookupsJSON = "lookups.json"
-	alertsJSON  = "alerts.json"
+	bansJSON       = "bans.json"
+	clientsJSON    = "clients.json"
+	lookupsJSON    = "lookups.json"
+	reputationJSON = "reputation.json"
+	alertsJSON     = "alerts.json"
+	// blocklistURL and torURL are the blocklists the tests' lists name.
+	blocklistURL = "https://lists.example/drop.txt"
+	torURL       = "https://lists.example/tor.txt"
 	// The AS number and AS name the tests' clients are looked up in.
 	asn    = "AS64496"
 	asName = "Example Net"
@@ -45,6 +51,9 @@ const (
 	// maxLogLines is how many lines of the process log wait for a test to
 	// read them.
 	maxLogLines = 64
+	// whole is the percentage of each limit a client gets when nothing
+	// lowers its limits.
+	whole = 100
 )
 
 // permanentBansJSON is bans.json holding permanentBan.
@@ -153,7 +162,73 @@ const filledAlertsJSON = `{
                 "suppressed_repeats": 0
             }
         ]
-    }
+    },
+    "anomaly_counters": [
+        {
+            "scope": "asn",
+            "asn": "AS64496",
+            "hour_bytes": {
+                "start": "2026-10-06T00:00:00Z",
+                "current": 8,
+                "previous": 0
+            }
+        },
+        {
+            "scope": "net",
+            "netblock": "203.0.113.0/24",
+            "minute": {
+                "start": "2026-10-06T00:00:00Z",
+                "current": 1,
+                "previous": 0
+            }
+        },
+        {
+            "scope": "total",
+            "minute": {
+                "start": "2026-10-06T00:00:00Z",
+                "current": 1,
+                "previous": 0
+            },
+            "minute_bytes": {
+                "start": "2026-10-06T00:00:00Z",
+                "current": 8,
+                "previous": 0
+            }
+        },
+        {
+            "scope": "watch",
+            "netblock": "203.0.113.0/24",
+            "name": "office",
+            "hour": {
+                "start": "2026-10-06T00:00:00Z",
+                "current": 1,
+                "previous": 0
+            }
+        }
+    ]
+}
+`
+
+// filledReputationJSON is reputation.json holding the blocklists' last
+// tries and the copy of one, with its comment line, as fill puts them in.
+const filledReputationJSON = `{
+    "version": 1,
+    "lists": [
+        {
+            "url": "https://lists.example/drop.txt",
+            "tried": "2026-10-06T00:00:00Z",
+            "fetched": "2026-10-05T23:00:00Z",
+            "lines": [
+                "; Spamhaus DROP List 2026/10/05 - (c) 2026 The Spamhaus Project SLL",
+                "203.0.113.0/24 ; SBL1",
+                "2001:db8::/32 ; SBL2"
+            ]
+        },
+        {
+            "url": "https://lists.example/tor.txt",
+            "tried": "2026-10-06T00:00:00Z"
+        }
+    ]
 }
 `
 
@@ -183,18 +258,45 @@ func TestFilesWrittenAndReadBack(t *testing.T) {
 	wantEqual(t, clientsJSON, after.Limiter.Snapshot(), before.Limiter.Snapshot())
 	wantEqual(t, lookupsJSON, after.GeoJS.Snapshot(), before.GeoJS.Snapshot())
 
+	if got, want := after.Lists.Snapshot(), before.Lists.Snapshot(); !reflect.DeepEqual(
+		got, want) {
+		t.Errorf("%s read back\n%+v\nwant\n%+v", reputationJSON, got, want)
+	}
+
 	if got, want := after.Alerts.Snapshot(), before.Alerts.Snapshot(); !reflect.DeepEqual(
 		got, want) {
 		t.Errorf("%s read back\n%+v\nwant\n%+v", alertsJSON, got, want)
 	}
 
+	wantEqual(t, alertsJSON, after.Anomalies.Snapshot(), before.Anomalies.Snapshot())
+
 	// Each one-per-line file lists its entries by client, and nothing
-	// but the four files is left in the directory.
+	// but the five files is left in the directory.
 	wantEntries(t, filepath.Join(dir, clientsJSON), "clients",
 		"192.0.2.1/32", "203.0.113.9/32", "2001:db8::/64")
 	wantEntries(t, filepath.Join(dir, lookupsJSON), "lookups",
 		"192.0.2.1/32", "203.0.113.9/32")
-	wantFiles(t, dir, alertsJSON, bansJSON, clientsJSON, lookupsJSON)
+	wantFiles(t, dir, alertsJSON, bansJSON, clientsJSON, lookupsJSON, reputationJSON)
+}
+
+func TestReputationJSONKeepsEachCopyWholeOneLineToALine(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+	params := newParams(dir)
+	fill(params)
+
+	files := load(t, params)
+
+	err := files.WriteAll()
+	if err != nil {
+		t.Fatalf("write: %v", err)
+	}
+
+	got := readFile(t, filepath.Join(dir, reputationJSON))
+	if got != filledReputationJSON {
+		t.Errorf("reputation.json\n%s\nwant\n%s", got, filledReputationJSON)
+	}
 }
 
 func TestAlertsJSONIsIndentedWithTheCooldownsTheHourAndTheAlertsWaiting(t *testing.T) {
@@ -248,6 +350,49 @@ func TestSourceFailureCooldownKeptInAlertsJSONAcrossARestart(t *testing.T) {
 	}
 }
 
+func TestAnomalyCountersKeptInAlertsJSONAcrossARestart(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+	request := anomaly.Request{
+		Client:      netip.MustParseAddr("203.0.113.9"),
+		ClientGroup: netip.MustParsePrefix("203.0.113.9/32"),
+	}
+
+	// The whole service may have two requests a minute.
+	withThreshold := func() state.Params {
+		params := newParams(dir)
+		params.Anomalies = anomaly.New(anomaly.Params{
+			Total: anomaly.Thresholds{RequestsPerMinute: 2}, Alerts: params.Alerts,
+		})
+
+		return params
+	}
+
+	before := withThreshold()
+	files := load(t, before)
+
+	for range 2 {
+		before.Anomalies.Count(midnight(), request)
+	}
+
+	err := files.WriteAll()
+	if err != nil {
+		t.Fatalf("write: %v", err)
+	}
+
+	// After the restart, the third request in the minute is over it.
+	after := withThreshold()
+	load(t, after)
+	after.Anomalies.Count(midnight(), request)
+
+	waiting := after.Alerts.Snapshot().Waiting[alerts.DestinationWebhook]
+	if len(waiting) != 1 || waiting[0].Event != alerts.EventAnomaly ||
+		waiting[0].Detail["count"] != float64(3) {
+		t.Errorf("alerts wait %+v, want one for 3 requests", waiting)
+	}
+}
+
 func TestBansJSONIsIndentedWithNullForAPermanentBan(t *testing.T) {
 	t.Parallel()
 
@@ -276,8 +421,9 @@ func TestMissingFilesAreEmptyState(t *testing.T) {
 
 	held := params.Alerts.Snapshot()
 	if len(params.Ledger.Snapshot()) != 0 || len(params.Limiter.Snapshot()) != 0 ||
-		len(params.GeoJS.Snapshot()) != 0 || len(held.Cooldowns) != 0 ||
-		len(held.Waiting[alerts.DestinationWebhook]) != 0 || held.Hour.Sent != 0 {
+		len(params.GeoJS.Snapshot()) != 0 || len(params.Lists.Snapshot()) != 0 ||
+		len(held.Cooldowns) != 0 || len(held.Waiting[alerts.DestinationWebhook]) != 0 ||
+		held.Hour.Sent != 0 {
 		t.Error("state from no files")
 	}
 }
@@ -329,6 +475,21 @@ func TestFileThatDoesNotParseStopsTheStart(t *testing.T) {
 			`{"version": 1, "waiting": {"webhook": [], "slak": []}}`,
 			`: waiting "slak" is not webhook, slack or ntfy`,
 		},
+		{
+			"an anomaly counter of an unknown scope", alertsJSON,
+			`{"version": 1, "anomaly_counters": [{"scope": "total"}, ` +
+				`{"scope": "nett", "netblock": "203.0.113.0/24"}]}`,
+			`: anomaly_counters entry 2's scope "nett" is not client, net, asn, total ` +
+				`or watch`,
+		},
+		{
+			"a copy of a list with a line that does not read", reputationJSON,
+			`{"version": 1, "lists": [{"url": "` + blocklistURL + `", ` +
+				`"tried": "2026-10-06T00:00:00Z", "fetched": "2026-10-06T00:00:00Z", ` +
+				`"lines": ["; DROP", "203.0.113.300"]}]}`,
+			`: the copy of ` + blocklistURL + `: line 2 is not an address or a netblock, ` +
+				`such as 192.0.2.0/24`,
+		},
 	} {
 		t.Run(tc.name, func(t *testing.T) {
 			t.Parallel()
@@ -424,6 +585,51 @@ func TestEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
 	}
 }
 
+func TestReputationJSONEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
+	t.Parallel()
+
+	const (
+		drop    = `"url": "` + blocklistURL + `", `
+		tried   = `"tried": "2026-10-06T00:00:00Z", `
+		fetched = `"fetched": "2026-10-06T00:00:00Z"`
+	)
+
+	for _, tc := range []struct {
+		name, content string
+		// want is what the error says after the file's path.
+		want string
+	}{
+		{
+			"a list without its URL",
+			`{"version": 1, "lists": [{` + tried + fetched + `, "lines": []}]}`,
+			`: entry 1 has no "url"`,
+		},
+		{
+			"a list without the time it was last tried",
+			`{"version": 1, "lists": [{` + drop + fetched + `, "lines": []}]}`,
+			`: entry 1 has no "tried"`,
+		},
+		{
+			"a copy of a list without the time it was fetched",
+			`{"version": 1, "lists": [{` + drop + tried + `"lines": []}]}`,
+			`: entry 1 has no "fetched"`,
+		},
+		{
+			// An empty list has no lines, which is not having none.
+			"a copy of a list without its lines",
+			`{"version": 1, "lists": [{` + drop + tried + fetched + `, "lines": []}, ` +
+				`{"url": "` + torURL + `", ` + tried + fetched + `}]}`,
+			`: entry 2 has no "lines"`,
+		},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+
+			wantRefused(t, reputationJSON, tc.content, tc.want)
+		})
+	}
+}
+
 func TestAlertsJSONEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
 	t.Parallel()
 
@@ -452,6 +658,39 @@ func TestAlertsJSONEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
 			`{"version": 1, "waiting": {"slack": [{"event": "ban"}]}}`,
 			`: waiting slack entry 1 has no "time"`,
 		},
+		{
+			// The whole service's counter needs nothing to tell it apart.
+			"an anomaly counter of a netblock without it",
+			`{"version": 1, "anomaly_counters": [{"scope": "total"}, {"scope": "net"}]}`,
+			`: anomaly_counters entry 2 has no "netblock"`,
+		},
+		{
+			"an anomaly counter of a client without its netblock",
+			`{"version": 1, "anomaly_counters": [{"scope": "client"}]}`,
+			`: anomaly_counters entry 1 has no "netblock"`,
+		},
+		{
+			"an anomaly counter of an AS number without it",
+			`{"version": 1, "anomaly_counters": [{"scope": "asn"}]}`,
+			`: anomaly_counters entry 1 has no "asn"`,
+		},
+		{
+			"an anomaly counter of a named netblock without its name",
+			`{"version": 1, "anomaly_counters": [` +
+				`{"scope": "watch", "netblock": "203.0.113.0/24"}]}`,
+			`: anomaly_counters entry 1 has no "name"`,
+		},
+		{
+			"an anomaly counter of a named netblock without its netblock",
+			`{"version": 1, "anomaly_counters": [{"scope": "watch", "name": "office"}]}`,
+			`: anomaly_counters entry 1 has no "netblock"`,
+		},
+		{
+			"an anomaly counter with bytes in a window without its start",
+			`{"version": 1, "anomaly_counters": [` +
+				`{"scope": "total", "hour_bytes": {"current": 5}}]}`,
+			`: anomaly_counters entry 1 has no "hour_bytes.start"`,
+		},
 	} {
 		t.Run(tc.name, func(t *testing.T) {
 			t.Parallel()
@@ -494,7 +733,9 @@ func TestBanWithAnotherCauseStopsTheStart(t *testing.T) {
 func TestUnknownVersionStopsTheStart(t *testing.T) {
 	t.Parallel()
 
-	for _, file := range []string{bansJSON, clientsJSON, lookupsJSON, alertsJSON} {
+	for _, file := range []string{
+		bansJSON, clientsJSON, lookupsJSON, reputationJSON, alertsJSON,
+	} {
 		for _, content := range []string{`{"version": 2}`, `{}`} {
 			t.Run(file+" "+content, func(t *testing.T) {
 				t.Parallel()
@@ -596,8 +837,9 @@ func TestEveryFileWrittenEveryCounterInterval(t *testing.T) {
 
 			time.Sleep(time.Nanosecond)
 			synctest.Wait()
-			wantFiles(t, dir, alertsJSON, bansJSON, clientsJSON, lookupsJSON)
-			removeFiles(t, dir, alertsJSON, bansJSON, clientsJSON, lookupsJSON)
+			wantFiles(t, dir, alertsJSON, bansJSON, clientsJSON, lookupsJSON, reputationJSON)
+			removeFiles(t, dir, alertsJSON, bansJSON, clientsJSON, lookupsJSON,
+				reputationJSON)
 		}
 	})
 }
@@ -713,7 +955,7 @@ func TestFailedWriteLeavesTheFileAsItWas(t *testing.T) {
 
 	params.Ledger.BanForLimit(netip.MustParsePrefix("203.0.113.9/32"), midnight(),
 		bans.Notes{})
-	params.Limiter.Count(netip.MustParsePrefix("203.0.113.9/32"), midnight())
+	params.Limiter.Count(netip.MustParsePrefix("203.0.113.9/32"), midnight(), whole)
 
 	err = files.WriteAll()
 	if err == nil || !strings.Contains(err.Error(), bansJSON+".tmp") {
@@ -816,7 +1058,7 @@ func TestFileThatCannotBeReadIsNotWrittenOver(t *testing.T) {
 		t.Errorf("bans.json is now %v (%v), want the socket", info, err)
 	}
 
-	wantFiles(t, dir, alertsJSON, bansJSON, clientsJSON, lookupsJSON)
+	wantFiles(t, dir, alertsJSON, bansJSON, clientsJSON, lookupsJSON, reputationJSON)
 	wantWriteFailed(t, params, bansJSON)
 }
 
@@ -889,6 +1131,17 @@ func TestEditOfEachFileTakenIn(t *testing.T) {
 	wantEqual(t, lookupsJSON, params.GeoJS.Snapshot(),
 		[]lookup.Answer{{Client: client, Country: "FR", Answered: midnight()}})
 
+	edit(t, dir, reputationJSON, `{"version": 1, "lists": [{"url": "`+blocklistURL+`", `+
+		`"tried": "2026-10-06T00:00:00Z", "fetched": "2026-10-06T00:00:00Z", `+
+		`"lines": ["198.51.100.7"]}]}`)
+	wantTakenIn(t, lines, dir, reputationJSON)
+
+	listedBy := params.Lists.ListedBy(client.Addr())
+	if !slices.Equal(listedBy, []string{blocklistURL}) {
+		t.Errorf("%s taken in lists %s on %v, want on the blocklist", reputationJSON,
+			client.Addr(), listedBy)
+	}
+
 	// A netblock with bits past its length is read as the netblock it is
 	// in.
 	edit(t, dir, alertsJSON, `{"version": 1, "cooldowns": [{"event": "ban", `+
@@ -1116,7 +1369,7 @@ func TestBrokenEditSetAsideAtTheNextWrite(t *testing.T) {
 	edit(t, dir, bansJSON, broken)
 	edit(t, dir, clientsJSON, `{"version": 1, "clients": []}`)
 	wantTakenIn(t, lines, dir, clientsJSON)
-	wantFiles(t, dir, alertsJSON, bansJSON, clientsJSON, lookupsJSON)
+	wantFiles(t, dir, alertsJSON, bansJSON, clientsJSON, lookupsJSON, reputationJSON)
 
 	// The next write sets it aside, logged with where the error is, and
 	// writes bans.json again from what smallwebwaf still holds.
@@ -1140,7 +1393,8 @@ func TestBrokenEditSetAsideAtTheNextWrite(t *testing.T) {
 		t.Errorf("alerts waiting %+v, want a file_error alert for %s", waiting, path+".bad")
 	}
 
-	wantFiles(t, dir, alertsJSON, bansJSON, bansJSON+".bad", clientsJSON, lookupsJSON)
+	wantFiles(t, dir, alertsJSON, bansJSON, bansJSON+".bad", clientsJSON, lookupsJSON,
+		reputationJSON)
 
 	if got := readFile(t, path+".bad"); got != broken {
 		t.Errorf("bans.json.bad holds\n%s\nwant the edit", got)
@@ -1269,11 +1523,21 @@ func midnight() time.Time {
 }
 
 // newParams returns Params for the state files in dir, with parts that
-// hold nothing yet. GeoJS is never asked, and the alerts, at most two an
-// hour, are never sent.
+// hold nothing yet. GeoJS is never asked, the lists, two blocklists, are
+// never fetched, and the alerts, at most two an hour, are
+// never sent. The anomaly counters count the scopes fill counts, with
+// thresholds fill does not reach.
 func newParams(dir string) state.Params {
 	discard := slog.New(slog.DiscardHandler)
 	m := metrics.New(1, "app")
+	queue := alerts.New(alerts.Params{
+		WebhookURL: &url.URL{Scheme: "https", Host: "alerts.example"},
+		Events:     alerts.Events(),
+		Cooldown:   15 * time.Minute,
+		MaxPerHour: 2,
+		Instance:   "fsn1app1/gitea",
+		Now:        midnight,
+	})
 
 	return state.Params{
 		Dir:             dir,
@@ -1290,13 +1554,20 @@ func newParams(dir string) state.Params {
 		GeoJS: lookup.New(lookup.Params{
 			Now: midnight, ProcessLog: discard, Metrics: m,
 		}),
-		Alerts: alerts.New(alerts.Params{
-			WebhookURL: &url.URL{Scheme: "https", Host: "alerts.example"},
-			Events:     alerts.Events(),
-			Cooldown:   15 * time.Minute,
-			MaxPerHour: 2,
-			Instance:   "fsn1app1/gitea",
-			Now:        midnight,
+		Lists: reputation.New(reputation.Params{
+			BlocklistURLs: []string{blocklistURL, torURL}, Refresh: 24 * time.Hour,
+			Now: midnight, ProcessLog: discard, Alerts: queue,
+		}),
+		Alerts: queue,
+		Anomalies: anomaly.New(anomaly.Params{
+			Net:            anomaly.Thresholds{RequestsPerMinute: 1000},
+			ASN:            anomaly.Thresholds{BytesPerHour: 1 << 30},
+			Total:          anomaly.Thresholds{RequestsPerMinute: 1000, BytesPerMinute: 1 << 30},
+			Watch:          anomaly.Thresholds{RequestsPerHour: 1000},
+			NetV4Prefix:    24,
+			NetV6Prefix:    48,
+			NamedNetblocks: []anomaly.NamedNetblock{{Name: "office", Netblock: office()}},
+			Alerts:         queue,
 		}),
 		Now:        midnight,
 		ProcessLog: discard,
@@ -1304,10 +1575,16 @@ func newParams(dir string) state.Params {
 	}
 }
 
+// office is the named netblock of the anomaly counters of newParams.
+func office() netip.Prefix {
+	return netip.MustParsePrefix("203.0.113.0/24")
+}
+
 // fill puts a permanent ban an admin made, a ban for a broken limit and
 // one for a clear sign of attack, clients with counts and histories,
-// GeoJS answers, and alerts, as filledAlertsJSON holds them, into the
-// parts of params.
+// GeoJS answers, the blocklists' last tries and the copy of one, as
+// filledReputationJSON holds them, and alerts and anomaly counters, as
+// filledAlertsJSON holds them, into the parts of params.
 func fill(params state.Params) {
 	now := midnight()
 	client := netip.MustParsePrefix("203.0.113.9/32")
@@ -1320,10 +1597,10 @@ func fill(params state.Params) {
 		bans.Notes{RuleID: "env-file", Target: "path"})
 
 	for _, c := range []string{"2001:db8::/64", "203.0.113.9/32", "192.0.2.1/32"} {
-		params.Limiter.Count(netip.MustParsePrefix(c), now)
+		params.Limiter.Count(netip.MustParsePrefix(c), now, whole)
 	}
 
-	params.Limiter.CountBytes(client, now, 8)
+	params.Limiter.CountBytes(client, now, 8, whole)
 	params.Limiter.AddToHistory(client, now, ratelimit.Request{
 		Forwarded: true, Status: 200, RequestBytes: 3, ResponseBytes: 5,
 	})
@@ -1340,6 +1617,20 @@ func fill(params state.Params) {
 		},
 	})
 
+	// The copy of drop.txt was fetched an hour ago, and the fetches of it
+	// and of tor.txt tried since failed.
+	err := params.Lists.Load([]reputation.List{{
+		URL: blocklistURL, Tried: now, Fetched: now.Add(-time.Hour),
+		Lines: []string{
+			"; Spamhaus DROP List 2026/10/05 - (c) 2026 The Spamhaus Project SLL",
+			"203.0.113.0/24 ; SBL1",
+			"2001:db8::/32 ; SBL2",
+		},
+	}, {URL: torURL, Tried: now}})
+	if err != nil {
+		panic(err) // the copy reads
+	}
+
 	// An alert waiting, a repeat of it the cooldown holds back, another
 	// alert waiting, and one past the two an hour, for the hour's summary.
 	ban := alerts.Alert{
@@ -1358,6 +1649,10 @@ func fill(params state.Params) {
 	params.Alerts.Raise(alerts.Alert{
 		Event: alerts.EventSourceFailure, Reason: "asking GeoJS failed",
 	})
+
+	params.Anomalies.Count(now, anomaly.Request{
+		Client: client.Addr(), ClientGroup: client, ASN: asn, Bytes: 8,
+	})
 }
 
 // permanentBan is the ban permanentBansJSON holds.