README rewritten from scratch so a newcomer can follow it (closes #154, closes #155)

The old README opened with project history and one list of a hundred
settings. The new one leads with what smallwebwaf is, what it protects
against, how to build and deploy it, how it sits in front of the app,
what happens to a request, the settings most deployments touch, how to
check it works and how to develop it. The reference material, every
setting, bans, rule files, the Core Rule Set, the admin endpoints, the
request log, alerts, metrics, state files, lookups, reputation sources,
the image and gitea, moves to files under docs/, each linked from the
README. Every statement was checked against the code on next.

Model: fable-5-1

docs/reputation.md

@@ -0,0 +1,113 @@
+# Reputation sources
+
+A reputation source judges a client by what others saw it do. None is on by
+default. The settings are under "Reputation sources" in
+[`settings.md`](settings.md); the reasons for the defaults are under
+"Reputation" in [`SPEC.md`](../SPEC.md). The sources are checked in this order,
+and a source the client is on refuses it, lowers its limits, or only notes it,
+as `SWWAF_BLOCKLIST_ACTION` and `SWWAF_REPUTATION_ACTION` say.
+
+## Blocklists
+
+`SWWAF_BLOCKLIST_URLS` lists text files of addresses and netblocks, one to a
+line, as the Spamhaus DROP list is written. Anything after a `;` or `#` is left
+out. An IPv4-mapped address such as `::ffff:192.0.2.0/120` is read as the IPv4
+netblock it stands for.
+
+Each list is fetched again `SWWAF_BLOCKLIST_REFRESH` after its last try, one
+list at a time. The file of `SWWAF_ASN_LIMIT_PERCENT_URL` is too.
+`reputation.json` keeps when each was last tried, so a restart does not fetch a
+list again before its time. A fetch fails when:
+
+- the server answers other than `200`;
+- the fetch does not finish within a minute;
+- the list is longer than 16 MiB;
+- a line is not an address or netblock, or, for the AS number file, not
+  `AS<number>:<percent>`.
+
+The copy fetched before stays in use, comment lines included. The failure is
+logged and raised as a `source_failure` alert. The log, the alerts, the ban
+notes and the metrics name each list by its URL, so keep a secret out of it.
+
+The Spamhaus DROP list is The Spamhaus Project's. Its terms, and what they mean
+for `SWWAF_BLOCKLIST_REFRESH` and for several instances on one host, are under
+"Reputation" in [`SPEC.md`](../SPEC.md). The copy in `reputation.json` keeps the
+list's date and copyright lines.
+
+## CrowdSec
+
+While `SWWAF_CROWDSEC_LAPI_URL` is set to the local API of a CrowdSec engine you
+run, `smallwebwaf` fetches its decision list every minute. It sends
+`SWWAF_CROWDSEC_LAPI_KEY` in the `X-Api-Key` header. The list is kept as a
+blocklist is kept. The decisions of the type `ban` on an address or a netblock
+are used. Any other, such as a captcha or a country, is left out. A decision
+lasts for the `duration` it had left when the engine answered.
+
+A request is refused when a decision still in force names its client. The
+client's netblock is banned until the decision ends, with the cause `crowdsec`.
+Only a client that sends a request is banned, so a long list does not fill
+`bans.json`. Such a ban is never made permanent, and does not lengthen the
+netblock's next ban for a broken limit. If you lift a ban while the decision is
+in force, the client's next request makes another. Delete the decision in
+CrowdSec first, such as with `cscli decisions delete --ip 203.0.113.9`, and lift
+the ban once the list has been fetched again. A fetch fails when:
+
+- the engine answers other than `200`, a redirect included;
+- the fetch does not finish within a minute;
+- the answer is longer than 16 MiB;
+- the answer is not a list of decisions.
+
+## DNS blocklists
+
+Each zone in `SWWAF_DNSBL_ZONES` is asked about a client's own address in the
+background, the first time the client is seen. The request goes on at once, and
+so does every request from the client until the zone has answered. The name
+asked about is the one RFC 5782 gives: the numbers of an IPv4 address reversed,
+or the 32 hex digits of an IPv6 address reversed. `192.0.2.99` is asked about as
+`99.2.0.192.dnsbl.dronebl.org`. An answer in `127.0.0.0/8` lists the client, but
+one in `127.255.255.0/24` is Spamhaus refusing the query. A name that does not
+exist does not list the client. These are failures:
+
+- an answer in `127.255.255.0/24`;
+- an answer outside `127.0.0.0/8`;
+- a resolver error;
+- no answer within `SWWAF_REPUTATION_TIMEOUT`.
+
+A failure is logged and alerted, and the zone is not asked again for a minute.
+Each verdict is kept for `SWWAF_REPUTATION_CACHE_TTL`, in memory and in
+`reputation.json`. Up to 100,000 verdicts are kept; past that, the one fetched
+longest ago goes first. At most 1,000 queries are under way at once.
+
+The key of a Spamhaus zone under `dq.spamhaus.net` is its first label.
+`smallwebwaf` shows `********` in its place wherever it names the zone. Only
+`reputation.json` keeps the zone with its key.
+
+## AbuseIPDB
+
+While `SWWAF_ABUSEIPDB_KEY` is set, `smallwebwaf` asks AbuseIPDB's check
+endpoint for the abuse confidence score of a client, from 0 to 100. Only a
+client that has committed an offence is checked. An offence is one of:
+
+- a broken limit;
+- a `ban` rule match or a trap path;
+- a refusal by a `block` rule, the Core Rule Set or a token check.
+
+The check runs in the background, at the client's first request after the
+offence that no ban refuses, so no request waits. The score applies to the
+client's later requests. For an IPv6 client, AbuseIPDB is asked about the
+address of the request that set off the check, and the score is used for the
+whole group. A score at or over `SWWAF_ABUSEIPDB_MIN_SCORE` is a hit, and
+`SWWAF_REPUTATION_ACTION` says what is done. Each score is kept for
+`SWWAF_REPUTATION_CACHE_TTL`, up to 100,000 scores.
+
+At most `SWWAF_ABUSEIPDB_DAILY_BUDGET` checks are made in a day, counted in UTC
+and kept in `reputation.json`. A restart does not make the budget whole again.
+The check that spends the last of the budget is logged and alerted. No client is
+checked until the day ends. A check fails when:
+
+- AbuseIPDB answers other than `200`;
+- the answer gives no score;
+- no answer comes within `SWWAF_REPUTATION_TIMEOUT`.
+
+A failure is logged and alerted, and no client is checked for a minute. The key
+is sent in the `Key` header and nowhere else, and is never logged.