SPEC: requirements and reasons only, reference material in docs/

docs/settings.md is now the one settings reference; the spec keeps the
conventions, the off list and each group's requirements and reasons, and
links to it for the per-setting entries. The endpoint list, the ban
notes, the state file contents, the rule file format and the alert
object likewise live in docs/ and are linked from the spec. The package
list from the README's Design section joins "Architecture". The nine
docs files are byte-identical to those on the README branch. Package
comments that named "Alert webhook schema", "Request log" and "Metrics
endpoint" for their field lists now name the docs files.

Model: fable-5-1

docs/reputation.md

@@ -0,0 +1,113 @@
+# Reputation sources
+
+A reputation source judges a client by what others saw it do. None is on by
+default. The settings are under "Reputation sources" in
+[`settings.md`](settings.md); the reasons for the defaults are under
+"Reputation" in [`SPEC.md`](../SPEC.md). The sources are checked in this order,
+and a source the client is on refuses it, lowers its limits, or only notes it,
+as `SWWAF_BLOCKLIST_ACTION` and `SWWAF_REPUTATION_ACTION` say.
+
+## Blocklists
+
+`SWWAF_BLOCKLIST_URLS` lists text files of addresses and netblocks, one to a
+line, as the Spamhaus DROP list is written. Anything after a `;` or `#` is left
+out. An IPv4-mapped address such as `::ffff:192.0.2.0/120` is read as the IPv4
+netblock it stands for.
+
+Each list is fetched again `SWWAF_BLOCKLIST_REFRESH` after its last try, one
+list at a time. The file of `SWWAF_ASN_LIMIT_PERCENT_URL` is too.
+`reputation.json` keeps when each was last tried, so a restart does not fetch a
+list again before its time. A fetch fails when:
+
+- the server answers other than `200`;
+- the fetch does not finish within a minute;
+- the list is longer than 16 MiB;
+- a line is not an address or netblock, or, for the AS number file, not
+  `AS<number>:<percent>`.
+
+The copy fetched before stays in use, comment lines included. The failure is
+logged and raised as a `source_failure` alert. The log, the alerts, the ban
+notes and the metrics name each list by its URL, so keep a secret out of it.
+
+The Spamhaus DROP list is The Spamhaus Project's. Its terms, and what they mean
+for `SWWAF_BLOCKLIST_REFRESH` and for several instances on one host, are under
+"Reputation" in [`SPEC.md`](../SPEC.md). The copy in `reputation.json` keeps the
+list's date and copyright lines.
+
+## CrowdSec
+
+While `SWWAF_CROWDSEC_LAPI_URL` is set to the local API of a CrowdSec engine you
+run, `smallwebwaf` fetches its decision list every minute. It sends
+`SWWAF_CROWDSEC_LAPI_KEY` in the `X-Api-Key` header. The list is kept as a
+blocklist is kept. The decisions of the type `ban` on an address or a netblock
+are used. Any other, such as a captcha or a country, is left out. A decision
+lasts for the `duration` it had left when the engine answered.
+
+A request is refused when a decision still in force names its client. The
+client's netblock is banned until the decision ends, with the cause `crowdsec`.
+Only a client that sends a request is banned, so a long list does not fill
+`bans.json`. Such a ban is never made permanent, and does not lengthen the
+netblock's next ban for a broken limit. If you lift a ban while the decision is
+in force, the client's next request makes another. Delete the decision in
+CrowdSec first, such as with `cscli decisions delete --ip 203.0.113.9`, and lift
+the ban once the list has been fetched again. A fetch fails when:
+
+- the engine answers other than `200`, a redirect included;
+- the fetch does not finish within a minute;
+- the answer is longer than 16 MiB;
+- the answer is not a list of decisions.
+
+## DNS blocklists
+
+Each zone in `SWWAF_DNSBL_ZONES` is asked about a client's own address in the
+background, the first time the client is seen. The request goes on at once, and
+so does every request from the client until the zone has answered. The name
+asked about is the one RFC 5782 gives: the numbers of an IPv4 address reversed,
+or the 32 hex digits of an IPv6 address reversed. `192.0.2.99` is asked about as
+`99.2.0.192.dnsbl.dronebl.org`. An answer in `127.0.0.0/8` lists the client, but
+one in `127.255.255.0/24` is Spamhaus refusing the query. A name that does not
+exist does not list the client. These are failures:
+
+- an answer in `127.255.255.0/24`;
+- an answer outside `127.0.0.0/8`;
+- a resolver error;
+- no answer within `SWWAF_REPUTATION_TIMEOUT`.
+
+A failure is logged and alerted, and the zone is not asked again for a minute.
+Each verdict is kept for `SWWAF_REPUTATION_CACHE_TTL`, in memory and in
+`reputation.json`. Up to 100,000 verdicts are kept; past that, the one fetched
+longest ago goes first. At most 1,000 queries are under way at once.
+
+The key of a Spamhaus zone under `dq.spamhaus.net` is its first label.
+`smallwebwaf` shows `********` in its place wherever it names the zone. Only
+`reputation.json` keeps the zone with its key.
+
+## AbuseIPDB
+
+While `SWWAF_ABUSEIPDB_KEY` is set, `smallwebwaf` asks AbuseIPDB's check
+endpoint for the abuse confidence score of a client, from 0 to 100. Only a
+client that has committed an offence is checked. An offence is one of:
+
+- a broken limit;
+- a `ban` rule match or a trap path;
+- a refusal by a `block` rule, the Core Rule Set or a token check.
+
+The check runs in the background, at the client's first request after the
+offence that no ban refuses, so no request waits. The score applies to the
+client's later requests. For an IPv6 client, AbuseIPDB is asked about the
+address of the request that set off the check, and the score is used for the
+whole group. A score at or over `SWWAF_ABUSEIPDB_MIN_SCORE` is a hit, and
+`SWWAF_REPUTATION_ACTION` says what is done. Each score is kept for
+`SWWAF_REPUTATION_CACHE_TTL`, up to 100,000 scores.
+
+At most `SWWAF_ABUSEIPDB_DAILY_BUDGET` checks are made in a day, counted in UTC
+and kept in `reputation.json`. A restart does not make the budget whole again.
+The check that spends the last of the budget is logged and alerted. No client is
+checked until the day ends. A check fails when:
+
+- AbuseIPDB answers other than `200`;
+- the answer gives no score;
+- no answer comes within `SWWAF_REPUTATION_TIMEOUT`.
+
+A failure is logged and alerted, and no client is checked for a minute. The key
+is sent in the `Key` header and nowhere else, and is never logged.