Lower limits for listed AS numbers and countries (closes #21)

SWWAF_ASN_LIMIT_PERCENT and SWWAF_COUNTRY_LIMIT_PERCENT give the clients
of the AS numbers and countries they list that percentage of every rate
and byte limit, rounded down; SWWAF_ASN_BYTES_PERCENT and
SWWAF_COUNTRY_BYTES_PERCENT take its place for the byte limits of those
they list; SWWAF_UNKNOWN_LIMIT_PERCENT (100) covers clients without a
country. The lowest applies. While one lowers a limit, a request waits
for its client's lookup, and SWWAF_LOOKUP_SOURCE=off stops the start. Log
lines give limit_percent and bytes_percent with their settings; ban
notes, and so alerts, give the broken limit's.

Judgement call: a client without a country is unknown, whatever its AS number.
Judgement call: bytes_percent and its setting are log fields SPEC does not name.
Rule suppressed: funlen on FromEnvironment, one line per setting.

Model: opus-5-5

README.md

@@ -22,13 +22,15 @@ fields, which come a little later, and the metrics endpoint and the header size
 and the idle time as settings, which come last in it. So are the four parts of
 the stage after it: the rule files, with the bans for a clear sign of attack,
 the other admin endpoints, alerts to all three destinations, a JSON webhook,
-Slack and ntfy, and remote log sending. So are two parts of the stage after
+Slack and ntfy, and remote log sending. So are three parts of the stage after
 that: the AS number and country of every client, looked up through GeoJS or in
-the IPinfo Lite database file, and the byte limits. `smallwebwaf` passes each
+the IPinfo Lite database file, the byte limits, and the biased thresholds, lower
+limits for the AS numbers and countries you list. `smallwebwaf` passes each
 request to the app and the app's answer back, unchanged, within its timeouts and
 size limits, works out each client's address, looks up its AS number and country
 unless you switch that off, bans a client that sends too many requests or too
-many bytes, not counting those for the paths you choose, refuses a client that
+many bytes, not counting those for the paths you choose, with lower limits for
+the clients of the AS numbers and countries you list, refuses a client that
 comes from a country you refuse or from a network you refuse, lets the networks
 you choose through, checks each request against the rule files and bans a client
 whose request is a clear sign of attack, keeps its bans, each client's counters
@@ -120,6 +122,19 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
   They leave out what the rate limits leave out: a client in `SWWAF_ALLOW_NETS`
   or `SWWAF_RATE_LIMIT_EXEMPT_NETS`, and a request for a path
   `SWWAF_RATE_LIMIT_EXEMPT_PATHS` exempts.
+- Gives the clients of the AS numbers and countries the biased thresholds list,
+  `SWWAF_ASN_LIMIT_PERCENT` and `SWWAF_COUNTRY_LIMIT_PERCENT`, the percentage
+  they give of every rate limit and byte limit, so that the same rules ban them
+  after fewer requests, and, while `SWWAF_UNKNOWN_LIMIT_PERCENT` is below 100,
+  every client without a country that percentage. A client to which several
+  apply gets the lowest. `SWWAF_ASN_BYTES_PERCENT` and
+  `SWWAF_COUNTRY_BYTES_PERCENT` give the AS numbers and countries they list a
+  percentage of the byte limits in place of the other two. Each client is
+  counted on its own, against its own lowered limits: no budget is shared by a
+  whole AS number or country, which one abuser could use up and so lock out
+  everyone else there. The log line of each request the rate limits count gives
+  its client's percentages below 100 and the settings that gave them, and so do
+  the notes of a ban for a lowered limit, and its alert.
 - Bans a client that breaks a rate limit or a byte limit, as "Bans" in
   [`SPEC.md`](SPEC.md) describes: the first ban lasts an hour, and a limit
   broken again within a day of a ban ending bans for three times as long as that
@@ -131,17 +146,18 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
   the client is not looked up, and is not counted for the rate limits. A ban
   sets the client's counters back to zero. Each ban carries notes for deciding
   whether to lift it: the limit, whether it is on requests or bytes, its window
-  and the requests or bytes counted in it, the request that broke it, the
-  client's AS number, AS name and country once they are looked up, the
-  netblock's requests since it was first seen, how many of them the ban has
-  refused, and how many bans the netblock had before, for a broken limit, for a
-  clear sign of attack and by an admin. At most `SWWAF_MAX_BANS` bans
-  `smallwebwaf` made are kept, past, active and permanent; past that, the
-  earliest such ban of the netblock that has gone longest without a request is
-  dropped first. The bans whose cause is `admin`, those you make or keep, are
-  kept besides, and never dropped. `bans.json` shows the bans and their notes, a
-  restart lifts none, and you make, keep or lift a ban by editing it (see "State
-  files" below).
+  and the requests or bytes counted in it, the client's percentage of that kind
+  of limit and the setting that gave it when a biased threshold lowered the
+  limit, the request that broke it, the client's AS number, AS name and country
+  once they are looked up, the netblock's requests since it was first seen, how
+  many of them the ban has refused, and how many bans the netblock had before,
+  for a broken limit, for a clear sign of attack and by an admin. At most
+  `SWWAF_MAX_BANS` bans `smallwebwaf` made are kept, past, active and permanent;
+  past that, the earliest such ban of the netblock that has gone longest without
+  a request is dropped first. The bans whose cause is `admin`, those you make or
+  keep, are kept besides, and never dropped. `bans.json` shows the bans and
+  their notes, a restart lifts none, and you make, keep or lift a ban by editing
+  it (see "State files" below).
 - Checks each request against the rules of the rule files (see "Rule files"
   below) after the rate limits, and before its body is read. A `log` rule that
   matches is noted in the log line; a `block` rule refuses the request with
@@ -163,10 +179,11 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
   AS number lookup" below), for the request log, the client's history, the notes
   of its bans, their alerts and the metrics. The file answers at once. With
   GeoJS, a request waits for its client's first answer only while a setting acts
-  on it, a country list or `SWWAF_ADD_LOOKUP_HEADERS`. Otherwise it goes on at
-  once, and the answer reaches the client's history and the notes of its bans
-  when it comes, but not the log lines of the requests that went on without it,
-  nor the alerts already raised for those bans.
+  on it, a country list, `SWWAF_ADD_LOOKUP_HEADERS` or a biased threshold that
+  lowers a limit. Otherwise it goes on at once, and the answer reaches the
+  client's history and the notes of its bans when it comes, but not the log
+  lines of the requests that went on without it, nor the alerts already raised
+  for those bans.
 - Refuses a request from a country you refuse with `SWWAF_BAN_RESPONSE`, as soon
   as the client's country is known and before its body is read; such a request
   is not counted for the rate limits. A client on a private, loopback or
@@ -312,9 +329,10 @@ effective settings are logged at start.
   country are looked up: `geojs`, the GeoJS web service, which is then told the
   address of every new visitor, `file`, the IPinfo Lite database file
   `SWWAF_LOOKUP_DB_PATH` names, or `off`, which looks up no client and sends no
-  address to GeoJS. With `off`, a country list that is not empty, or
-  `SWWAF_ADD_LOOKUP_HEADERS` set to `true`, stops the start, with a message
-  naming it and `SWWAF_LOOKUP_SOURCE`.
+  address to GeoJS. With `off`, a country list that is not empty,
+  `SWWAF_ADD_LOOKUP_HEADERS` set to `true`, or a biased threshold that lowers a
+  limit, a list of them that is not empty or `SWWAF_UNKNOWN_LIMIT_PERCENT` below
+  100, stops the start, with a message naming it and `SWWAF_LOOKUP_SOURCE`.
 - `SWWAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in its
   `.mmdb` form, for `SWWAF_LOOKUP_SOURCE=file`. `file` without it, or it with
   any other `SWWAF_LOOKUP_SOURCE`, the default included, stops the start, with a
@@ -334,6 +352,24 @@ effective settings are logged at start.
   countries whose clients get through, for example `us,de`. A client whose
   country cannot be found is refused too, so that new clients are not let in
   whenever GeoJS stops answering.
+- `SWWAF_ASN_LIMIT_PERCENT` (default empty): AS numbers, each with the
+  percentage of every rate limit and byte limit its clients get, such as
+  `AS14061:50,AS16276:50,AS45102:25`. A lowered limit is rounded down to a whole
+  number: half of 1000 requests a minute is 500, and half of 5 is 2. `0` is a
+  zero allowance: the client's first request breaks a limit, and bans it.
+- `SWWAF_COUNTRY_LIMIT_PERCENT` (default empty): the same by country, such as
+  `cn:25,ru:50`.
+- `SWWAF_ASN_BYTES_PERCENT` and `SWWAF_COUNTRY_BYTES_PERCENT` (default empty):
+  the same for the byte limits alone. For an AS number or a country one of them
+  lists, its percentage takes the place, for the byte limits, of the one
+  `SWWAF_ASN_LIMIT_PERCENT` or `SWWAF_COUNTRY_LIMIT_PERCENT` gives, so that
+  `SWWAF_ASN_LIMIT_PERCENT=AS14061:50` with
+  `SWWAF_ASN_BYTES_PERCENT=AS14061:100` halves that AS number's rate limits and
+  leaves its byte limits whole.
+- `SWWAF_UNKNOWN_LIMIT_PERCENT` (default `100`): the percentage of every limit a
+  client without a country gets: one the lookup cannot place, one on a private,
+  loopback or link-local address, which is never looked up, and one whose answer
+  from GeoJS has not come in time.
 - `SWWAF_BAN_RESPONSE` (default `403`): how a refused client is answered, one
   that is banned, breaks a rate limit, matches a `ban` rule, is in
   `SWWAF_DENY_NETS` or comes from a refused country: `403`, `429`, or `close` to
@@ -445,11 +481,15 @@ Netblocks are in CIDR form, and a bare address stands for itself alone.
 Countries are the two-letter codes ISO 3166-1 assigns today, and `xk` for
 Kosovo, in either case (`de` and `DE` are the same); any other code, such as
 `nk` (North Korea is `kp`) or the withdrawn `su`, stops the start, and so does a
-code on both country lists. `off` switches a timeout, a size limit, a rate
-limit, a byte limit, `SWWAF_ALERT_COOLDOWN` or `SWWAF_ALERT_MAX_PER_HOUR` off;
-`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, `SWWAF_LOOKUP_TIMEOUT`, the ban
-settings, the state settings, `SWWAF_METRICS_TOP_N` and
-`SWWAF_LOG_REMOTE_BUFFER` cannot be off.
+code on both country lists. AS numbers are `AS` and the number, in either case.
+Percentages are whole numbers from 0 to 100, and an entry of a list of them is
+an AS number or a country, `:` and a percentage; an AS number or a country
+listed twice in one of them stops the start. `off` switches a timeout, a size
+limit, a rate limit, a byte limit, `SWWAF_ALERT_COOLDOWN` or
+`SWWAF_ALERT_MAX_PER_HOUR` off; `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`,
+`SWWAF_LOOKUP_TIMEOUT`, `SWWAF_UNKNOWN_LIMIT_PERCENT`, the ban settings, the
+state settings, `SWWAF_METRICS_TOP_N` and `SWWAF_LOG_REMOTE_BUFFER` cannot be
+off.
 
 Several limits are fixed rather than settings. At most 20,000 clients are kept,
 with their counters and history, and an IPv6 client is counted by its /64. At
@@ -550,6 +590,12 @@ which every line has.
   `denied`, `banned`, `country_denied`, `rate_limited` or `rule_blocked`.
   `action` then names what was done: `forward` for a request passed to the app,
   and another action, such as `too_large`, for one a size or time limit refused.
+- `limit_percent` is there for a request the rate limits count whose client a
+  biased threshold gives less than the whole of the rate limits, and gives the
+  percentage it gets, with `limit_percent_setting` naming the setting that gave
+  it, such as `SWWAF_ASN_LIMIT_PERCENT`. `bytes_percent` and
+  `bytes_percent_setting` are the same for the byte limits. Each is left out
+  when the client gets the whole of those limits.
 - `counts` gives the client's requests in the minute, the hour and the day as
   the rate limits count them, this request included: in each window, those in
   the bucket under way and a share of those in the bucket before, so a count can
@@ -798,7 +844,10 @@ entries by client address, but for the alerts waiting, with times in UTC.
   `bytes per hour over the limit of 21474836480`, or the rule that matched, such
   as `matched the rule env-file`; for yours, what you wrote. Its `lifted` is
   when you lifted it, and is left out until you do. The `kind` in the notes of a
-  ban for a broken limit is `requests` or `bytes`, what the limit is on.
+  ban for a broken limit is `requests` or `bytes`, what the limit is on. For a
+  limit a biased threshold lowered, the reason and the notes' `limit` give the
+  lowered limit, and the notes' `limit_percent` and `limit_percent_setting` the
+  client's percentage of that kind of limit and the setting that gave it.
 - `clients.json`: each client's two buckets of requests in the minute, the hour
   and the day, its two buckets of bytes in each, `minute_bytes`, `hour_bytes`
   and `day_bytes`, and its history: when it was first and last seen, its AS
@@ -1025,9 +1074,9 @@ scraped, and keeps this one as `exported_instance` unless the scrape sets
 - `smallwebwaf_geojs_requests_total`: the requests to GeoJS;
   `smallwebwaf_geojs_failures_total`: those that failed, an answer that leaves
   out an address asked about included; and `smallwebwaf_geojs_unanswered_total`:
-  the requests that needed their client's answer, for a country list or
-  `SWWAF_ADD_LOOKUP_HEADERS`, and went on without it because GeoJS had not given
-  it in time.
+  the requests that needed their client's answer, for a country list,
+  `SWWAF_ADD_LOOKUP_HEADERS` or a biased threshold, and went on without it
+  because GeoJS had not given it in time.
 - While `SWWAF_LOOKUP_SOURCE` is `file`,
   `smallwebwaf_lookup_database_last_read_timestamp_seconds`: when the lookup
   database in use was read; and
@@ -1355,20 +1404,21 @@ memory and in `lookups.json`, so that it survives a restart, and a visitor whose
 answer is kept is not asked about again.
 
 A request waits for its client's first answer only while a setting acts on it
-before the request goes on: a country list, or `SWWAF_ADD_LOOKUP_HEADERS`. A new
-visitor then waits up to `SWWAF_LOOKUP_TIMEOUT`, a second by default, and
-without an answer counts as coming from an unknown country until the answer
-arrives. Otherwise no request waits: it goes on at once and is logged without
-the answer, which reaches the client's history and the notes of its bans when it
-comes. The addresses waiting are asked about together, up to 200 in one request,
-one request at a time; at most 10,000 visitors wait, and one more is not asked
-about until there is room, counting meanwhile as coming from an unknown country.
-GeoJS publishes no rate limit but may block a caller it thinks asks too much.
-While GeoJS fails, visitors with a kept answer are unaffected and new ones count
-as coming from an unknown country, which `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES`
-refuses. GeoJS is then left alone for a second, twice as long after each further
-failure up to five minutes, and asked again by the next request from a visitor
-without an answer.
+before the request goes on: a country list, `SWWAF_ADD_LOOKUP_HEADERS`, or a
+biased threshold that lowers a limit. A new visitor then waits up to
+`SWWAF_LOOKUP_TIMEOUT`, a second by default, and without an answer counts as
+coming from an unknown country until the answer arrives. Otherwise no request
+waits: it goes on at once and is logged without the answer, which reaches the
+client's history and the notes of its bans when it comes. The addresses waiting
+are asked about together, up to 200 in one request, one request at a time; at
+most 10,000 visitors wait, and one more is not asked about until there is room,
+counting meanwhile as coming from an unknown country. GeoJS publishes no rate
+limit but may block a caller it thinks asks too much. While GeoJS fails,
+visitors with a kept answer are unaffected and new ones count as coming from an
+unknown country, which `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses, and whose
+limits `SWWAF_UNKNOWN_LIMIT_PERCENT` sets. GeoJS is then left alone for a
+second, twice as long after each further failure up to five minutes, and asked
+again by the next request from a visitor without an answer.
 
 To keep your visitors' addresses on your own host, set
 `SWWAF_LOOKUP_SOURCE=off`, or use the database file instead of GeoJS:
@@ -1397,8 +1447,9 @@ service that uses the database through `smallwebwaf` should carry that link.
 Neither source can place a private address, so a client on one, such as a
 visitor on your local network, another container or your monitoring, has no
 country: `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses it unless you list it in
-`SWWAF_ALLOW_NETS`, and `SWWAF_DENIED_COUNTRIES` does not refuse it. Such
-addresses are never sent to GeoJS.
+`SWWAF_ALLOW_NETS`, `SWWAF_DENIED_COUNTRIES` does not refuse it, and
+`SWWAF_UNKNOWN_LIMIT_PERCENT` sets its limits. Such addresses are never sent to
+GeoJS.
 
 ## How the code is laid out