Request tags, with refusing and limits per tag (closes #175)

Each request gets sorted tags from the lookup, new address lists, the
reputation sources and the rule matches, logged as `tags`. New settings:
`SWW_TAG_LISTS`, `SWW_TAG_DENY`, `SWW_TAG_LIMIT_PERCENT`. They replace
`SWW_ASN_LIMIT_PERCENT`, `SWW_COUNTRY_LIMIT_PERCENT` and
`SWW_DENIED_COUNTRIES`. New heuristics `tag_lists` and `tag_deny`; the
latter runs before the rate limits and again after the Core Rule Set.

Model: opus-5-5

SPEC.md

@@ -119,9 +119,9 @@ to them.
 
 ## What happens to a request
 
-`rproxy` runs steps 1, 2 and 14, and sends the report of step 15.
-`discriminator` runs steps 3 to 13, and does the counting of step 15. The first
-step that produces a final answer ends the checks. Step 15 runs for each request
+`rproxy` runs steps 1, 2 and 17, and sends the report of step 18.
+`discriminator` runs steps 3 to 16, and does the counting of step 18. The first
+step that produces a final answer ends the checks. Step 18 runs for each request
 `discriminator` answers with `allow`, `deny` or `close`. `rproxy`'s size and
 time limits apply to the whole exchange. Each step names the section that holds
 its rules. A step that names a heuristic in parentheses runs that heuristic. The
@@ -143,46 +143,54 @@ request log's `heuristic` field gives that name. `refusal_limit` and
     - An IPv6 client is its group: the netblock of `SWW_IPV6_GROUP_PREFIX`
       around its address, a /64 by default, since one abuser usually controls a
       whole /64. A client is therefore one IPv4 address or one IPv6 group.
-3. The static lists. A client in `SWW_ALLOW_NETS` skips steps 4 to 12, the byte
+3. The tag lists (`tag_lists`; see "Request tags"). Each list that holds the
+   client adds its tag. A client in `SWW_ALLOW_NETS` gets no tag.
+4. The static lists. A client in `SWW_ALLOW_NETS` skips steps 5 to 15, the byte
    limits and the error burst. A client in `SWW_DENY_NETS` is refused
    (`deny_nets`); a client in both lists is let through.
-4. Bans. A client under a ban is refused (`active_ban`; see "Bans").
-5. The lookup, unless `SWW_LOOKUP_SOURCE` is `off` (see "Lookup of AS number and
+5. Bans. A client under a ban is refused (`active_ban`; see "Bans").
+6. The lookup, unless `SWW_LOOKUP_SOURCE` is `off` (see "Lookup of AS number and
    country"). The request goes on at once. It waits for the first GeoJS answer,
    up to `SWW_LOOKUP_TIMEOUT`, only when one of these needs it:
-    - a country list;
+    - the country list;
     - a biased threshold;
+    - `SWW_TAG_DENY` or `SWW_TAG_LIMIT_PERCENT` naming an `asn=` or `country=`
+      tag;
     - `SWW_ADD_LOOKUP_HEADERS`.
 
     A request that ends before the answer comes is logged without it.
 
-6. The country lists (`countries`; see "Country lists").
-7. The reputation sources (see "Reputation"), in this order:
+7. The country list (`countries`; see "Country list").
+8. The reputation sources (see "Reputation"), in this order:
     - A client on a fetched blocklist gets `SWW_BLOCKLIST_ACTION`
       (`blocklists`).
     - A client on the CrowdSec decision list is banned (`crowdsec`).
     - A client listed by a DNS blocklist gets `SWW_REPUTATION_ACTION` (`dnsbl`).
     - A client scored at least `SWW_ABUSEIPDB_MIN_SCORE` by AbuseIPDB gets
       `SWW_REPUTATION_ACTION` (`abuseipdb`).
-8. The client's limit percentage (see "Biased thresholds").
-9. Rate limiting (`rate_limits`; see "Request rate limits, per client"). A
-   request over a limit times the percentage is refused and bans the client.
-10. The trap paths (`trap_paths`; see "Attack detection"). A request for one is
+9. The tag refusals (`tag_deny`; see "Request tags"). A request carrying a tag
+   in `SWW_TAG_DENY` is refused.
+10. The client's limit percentage (see "Biased thresholds").
+11. Rate limiting (`rate_limits`; see "Request rate limits, per client"). A
+    request over a limit times the percentage is refused and bans the client.
+12. The trap paths (`trap_paths`; see "Attack detection"). A request for one is
     refused and bans the client.
-11. The rule files (`rule_files`; see "Rule files"). Matching stops at the first
+13. The rule files (`rule_files`; see "Rule files"). Matching stops at the first
     rule that refuses.
-12. The header limits (`header_limits`; see "Answering in time"), and then the
+14. The header limits (`header_limits`; see "Answering in time"), and then the
     Core Rule Set (`core_rule_set`; see "Attack detection"). A Core Rule Set
     match is refused with `403` while `SWW_WAF_MODE` is `block`, and only logged
     and alerted while it is `detect`.
-13. A path under `/_sww/` is answered `404`, so that the app never sees that
+15. The tag refusals again (`tag_deny`), so that `exploitsignature` can refuse
+    too.
+16. A path under `/_sww/` is answered `404`, so that the app never sees that
     prefix.
-14. `rproxy` passes the request to `SWRP_UPSTREAM_URL`, streaming (see "Passing
+17. `rproxy` passes the request to `SWRP_UPSTREAM_URL`, streaming (see "Passing
     a request to the app"). With `SWW_ADD_LOOKUP_HEADERS`, `X-Client-ASN` and
     `X-Client-Country` are added for the app's own logs; a header whose value is
     unknown is left out. Whatever that setting says, any such header the client
     sent is removed.
-15. After the visitor's answer has ended, `rproxy` sends the report (see "The
+18. After the visitor's answer has ended, `rproxy` sends the report (see "The
     report"). `discriminator` then counts:
     - For a request it allowed, the response bytes, the request bytes or both
       are added to the byte counters, as `SWW_BYTES_COUNT` says. For a
@@ -285,7 +293,8 @@ no offence:
 
 - `SWW_DENY_NETS`;
 - a blocklist or reputation hit whose action is `deny`;
-- the country lists.
+- the country list;
+- `SWW_TAG_DENY`.
 
 Every ban carries notes with what an admin needs to decide whether to lift it,
 drawn from what `discriminator` already knows. No lookup is made for the notes
@@ -392,7 +401,7 @@ Conventions:
 
 ### Static lists
 
-- `SWW_ALLOW_NETS` skips the checks step 3 names, for monitoring or the owner's
+- `SWW_ALLOW_NETS` skips the checks step 4 names, for monitoring or the owner's
   own networks. `SWW_RATE_LIMIT_EXEMPT_NETS` bypasses the request and byte
   limits only; the error burst, attack detection and bans still apply.
 - Many people can share one address, as in an office, on a mobile carrier or
@@ -442,7 +451,7 @@ the same way.
 By default there is no minute limit, so a client can download in a burst at full
 speed. The hour limit defaults to 50 GiB and the day limit to 100 GiB. They
 allow several downloads of a few GiB each. A response's bytes are counted when
-it ends (see step 15).
+it ends (see step 18).
 
 ### Lookup of AS number and country
 
@@ -471,12 +480,12 @@ service that keeps its visitors' addresses on its own host.
   dropped from that table. The answer reaches the client's history and ban notes
   when it comes.
 - `SWW_LOOKUP_TIMEOUT` is how long a visitor's request waits for its client's
-  first answer, when a setting needs it (see step 5). It also bounds each
+  first answer, when a setting needs it (see step 6). It also bounds each
   request to GeoJS. A client whose answer has not come in time counts as unknown
   until it comes, and its later requests do not wait. Its address is asked about
   again in the background.
 - A client the lookup cannot place has an unknown country and AS number, which
-  the country lists and the biased thresholds treat as described there. Nothing
+  the country list and the biased thresholds treat as described there. Nothing
   else treats it differently.
 - GeoJS publishes no rate limit, but its terms forbid "an excessive amount of
   API requests", judged by GeoJS alone, and let it block a caller. While GeoJS
@@ -496,12 +505,12 @@ service that keeps its visitors' addresses on its own host.
   `file_error` alert and ignored. The database already loaded stays in use. A
   client whose address the file does not hold counts as unknown.
 
-### Country lists
+### Country list
 
-Both are empty by default: the defaults judge a client by what it does, not by
-where it comes from.
+`SWW_EXCLUSIVELY_ALLOWED_COUNTRIES` is empty by default: the defaults judge a
+client by what it does, not by where it comes from. A `country=` tag in
+`SWW_TAG_DENY` refuses one country (see "Request tags").
 
-- `SWW_DENIED_COUNTRIES` refuses every request from a listed country.
 - `SWW_EXCLUSIVELY_ALLOWED_COUNTRIES` lets through only clients placed in a
   listed country. Every request from any other country is refused. So is every
   request from a client the lookup cannot place, such as internal monitoring or
@@ -510,45 +519,84 @@ where it comes from.
   client in whenever GeoJS stops answering. The cost: while GeoJS fails, this
   list refuses every new client. No other failure of GeoJS, a reputation source
   or an alert destination comes this close to taking the service down.
-- Both may be set. `SWW_DENIED_COUNTRIES` then adds nothing, since the exclusive
-  list already refuses every other country. A code on both lists stops the start
+- A code in this list whose `country=` tag is in `SWW_TAG_DENY` stops the start
   with a message naming it.
 - A refused request is answered with `SWW_BAN_RESPONSE`, as soon as the client's
   address has been looked up. It is logged with the action `country_denied`, and
   counted in the metrics.
 
+### Request tags
+
+Each request gets tags from what `discriminator` knows of its client and what
+the checks found. The operator refuses or slows clients by tag. Tags appear in
+the log line as `tags`, sorted, with empty lists left out (see
+`docs/request-log.md`).
+
+Each tag and where it comes from:
+
+- `asn=<number>` and `country=<code>`, such as `asn=64496` and `country=DE`,
+  from the lookup. A client the lookup cannot place has neither. A request
+  refused by `deny_nets` or `active_ban` has neither, because those checks come
+  before the lookup.
+- One tag per list of `SWW_TAG_LISTS` that holds the client (`tag_lists`), the
+  tag being the list's name, such as `torexit` or `vpn=mullvad`. These lists are
+  fetched and kept like a blocklist. `discriminator` ships none of its own.
+- `crowdsec`, `dnsbl=<zone>` and `abuseipdb`, when that reputation source lists
+  the client, whatever the source's action. A zone with a key is named with the
+  key masked, as the `reputation` field gives it.
+- `exploitsignature`, when a rule file or the Core Rule Set matched the request.
+  For the Core Rule Set a match is a score at or over
+  `SWW_WAF_ANOMALY_THRESHOLD`.
+
+A client in `SWW_ALLOW_NETS` skips the checks and gets no tag. Every tag is in
+lower case, except a country code, which is in capitals. `SWW_TAG_DENY` and
+`SWW_TAG_LIMIT_PERCENT` accept tags in either case.
+
+Two settings act on tags:
+
+- `SWW_TAG_DENY` lists tags whose requests are refused with the action `denied`
+  (`tag_deny`). The check runs twice: just before the rate limits, and again
+  after the Core Rule Set, so that `exploitsignature` can refuse too.
+- `SWW_TAG_LIMIT_PERCENT` lists tags with a percentage from 1 to 99. Their
+  clients get that percentage of every request and byte limit. The lowest
+  percentage among a client's tags applies. `exploitsignature` cannot be listed,
+  since a request gets it only after the rate limits have counted it.
+
+The tag settings are empty by default, for the same reason as the country list.
+The lookup waits for GeoJS when either names an `asn=` or `country=` tag. A
+setting that names one stops the start while `SWW_LOOKUP_SOURCE` is `off`.
+
 ### Biased thresholds
 
-The biased thresholds lower the limits of clients in listed AS numbers and
-countries. The lists are empty by default, for the same reason as the country
-lists. The request log and the metrics show which AS numbers and countries a
-service's abuse comes from.
+The biased thresholds lower the limits of clients by AS number, country and tag.
+The lists are empty by default, for the same reason as the country list. The
+request log and the metrics show which AS numbers and countries a service's
+abuse comes from.
 
-- Each list gives AS numbers or countries a percentage, such as `AS14061:50`.
-  Their clients get that percentage of every request and byte limit
-  (`SWW_ASN_LIMIT_PERCENT`, `SWW_COUNTRY_LIMIT_PERCENT`). A lowered limit is
-  rounded down. `0` is a zero allowance: the client's first request breaks a
-  limit and bans it.
+- `SWW_TAG_LIMIT_PERCENT` gives a tag a percentage, such as `asn=14061:50` (see
+  "Request tags"). A lowered limit is rounded down.
+- `SWW_UNKNOWN_LIMIT_PERCENT` at `0` is a zero allowance: the client's first
+  request breaks a limit and bans it.
 - `SWW_ASN_BYTES_PERCENT` and `SWW_COUNTRY_BYTES_PERCENT` set the byte limits'
   percentage alone, when it should differ from the request limits'. For an AS
-  number they list, they replace its percentage from `SWW_ASN_LIMIT_PERCENT` and
-  `SWW_ASN_LIMIT_PERCENT_URL`; for a country, the one from
-  `SWW_COUNTRY_LIMIT_PERCENT`.
+  number they list, they replace its `asn=` tag's percentage and its entry in
+  `SWW_ASN_LIMIT_PERCENT_URL`; for a country, its `country=` tag's percentage.
 - `SWW_ASN_LIMIT_PERCENT_URL` names a text file of `AS:percent` lines, so one
   abuse-source list can be shared by every `discriminator` in the fleet. It is
-  fetched and kept like a blocklist (see "Reputation"). Where it and
-  `SWW_ASN_LIMIT_PERCENT` both give a percentage, the lower applies.
+  fetched and kept like a blocklist (see "Reputation"). Each entry counts like
+  an `asn=` tag in `SWW_TAG_LIMIT_PERCENT`. Where both give a percentage, the
+  lower applies.
 - A client's limit percentage is the lowest of those that apply to it: its AS
-  number's, its country's, or a reputation hit's `limit:<percent>`. A client
-  without a country gets `SWW_UNKNOWN_LIMIT_PERCENT` in place of a country's.
-  The percentage is 100 if none applies. Clients given a lower percentage are
-  banned after fewer requests, first for a while and then, for repeated abuse,
-  permanently. The log line of each counted request, and the notes of each ban,
-  say which setting lowered the limit.
+  number's, its country's, its tags', or a reputation hit's `limit:<percent>`. A
+  client without a country gets `SWW_UNKNOWN_LIMIT_PERCENT` in place of a
+  country's. The percentage is 100 if none applies. Clients given a lower
+  percentage are banned after fewer requests, first for a while and then, for
+  repeated abuse, permanently. The log line of each counted request, and the
+  notes of each ban, say which setting lowered the limit.
 - No budget is shared by a whole country or AS number. One abuser could use it
   up and lock out everyone else there: a denial of service nobody chose, and the
   one this tool exists to prevent. Refusing a whole country is left to the
-  operator's choice, through the country lists.
+  operator's choice, through `SWW_TAG_DENY` or the country list.
 
 ### Attack detection
 
@@ -903,7 +951,7 @@ left unset sends no alert. An alert needs a destination only the operator can
 supply, and what counts as unusual depends on each service's normal traffic,
 which the metrics show. They reveal a widely distributed scraper, thousands of
 addresses at low rates each, which the per-client limits do not see. A low
-`SWW_ASN_LIMIT_PERCENT` for its AS number is the response.
+percentage for its `asn=` tag in `SWW_TAG_LIMIT_PERCENT` is the response.
 
 - Requests and bytes are counted per minute and per hour, for each of:
     - a client;
@@ -1181,7 +1229,7 @@ the metrics.
 
 ### The answer
 
-`discriminator` runs steps 3 to 13 of "What happens to a request", in that
+`discriminator` runs steps 3 to 16 of "What happens to a request", in that
 order. Every answer but the last below carries `X-Sww-Disposition`.
 
 - `X-Sww-Disposition: allow`, status `200`, no body. `rproxy` passes the request
@@ -1220,13 +1268,13 @@ seconds, and `SWRP_DEFAULT_DISPOSITION` then decides. So at the defaults,
 and whatever its outside sources do.
 
 GeoJS is the only outside source a question can wait on. A question waits only
-when step 5 says so, and for at most `SWW_LOOKUP_TIMEOUT`, by default 800
+when step 6 says so, and for at most `SWW_LOOKUP_TIMEOUT`, by default 800
 milliseconds. A client without an answer by then counts as unknown. No question
 waits on these, which run in the background:
 
 - the DNS blocklists and AbuseIPDB;
-- fetching the blocklists, the CrowdSec decision list and the file
-  `SWW_ASN_LIMIT_PERCENT_URL` names.
+- fetching the blocklists, the tag lists, the CrowdSec decision list and the
+  file `SWW_ASN_LIMIT_PERCENT_URL` names.
 
 In `enforce` mode, a question does not wait for GeoJS when it is a request: