README rewritten from scratch so a newcomer can follow it (closes #154, closes #155)

The old README opened with project history and one list of a hundred
settings. The new one leads with what smallwebwaf is, what it protects
against, how to build and deploy it, what happens to a request, what a
refused visitor and the operator see, and how to check it works. The
settings follow, grouped by topic, each with its default and what a bad
value does. The rest is reference. Every statement was checked against
the code on next. Short sentences, bulleted lists in place of comma
chains, no tables, identifiers in backticks.

Model: fable-5-1

README.md

@@ -211,9 +211,11 @@ ends the handling. Nothing of a refused request reaches the app.
 
 Two terms recur below. A clear sign of attack is a request that matches a rule
 whose action is `ban`, or asks for a trap path (steps 9 and 10). An offence is a
-request that counts against its client in the client's history: a clear sign of
+request that counts against its client in its history. It is one of:
-attack, a broken limit, or a refusal by a rule, by the Core Rule Set or for a
+
-missing or wrong token.
+- a clear sign of attack;
+- a broken limit;
+- a refusal by a rule, by the Core Rule Set or for a missing or wrong token.
 
 1. `GET /_smallwebwaf/healthz` is answered `200` with `ok` at once, before any
    check and without asking the app.
@@ -353,8 +355,8 @@ configuration before enforcing it.
   `SWWAF_MODE=observe` and ask for `/.env`. The request reaches the app. Its log
   line has `"would_action":"banned"` and `"rule_ids":["env-file"]`. In `enforce`
   mode the same request bans your address for seven days, and your next request
-  for good. Lift the ban by editing `bans.json` or through the ban endpoint (see
+  makes the ban permanent. Lift the ban by editing `bans.json` or through the
-  "Bans" below).
+  ban endpoint (see "Bans" below).
 - With `SWWAF_METRICS_TOKEN` set,
   `curl -H "Authorization: Bearer $TOKEN" https://<site>/_smallwebwaf/metrics`
   answers with the metrics (see "Metrics" below).
@@ -417,8 +419,8 @@ The forms of values:
   `SWWAF_ALERT_NTFY_URL` is set, it may hold no control character. While
   `SWWAF_LOG_REMOTE_URL` is set and `SWWAF_LOG_REMOTE_APP_NAME` is not, it must
   be 1 to 48 printable ASCII characters without a space.
-- `SWWAF_MODE` (default `enforce`): `enforce`, or `observe` to refuse nothing
+- `SWWAF_MODE` (default `enforce`): `enforce`, or `observe` to ban no one and
-  and log what would have been refused (see "What happens to a request" above).
+  log what `enforce` would have done (see "What happens to a request" above).
 - `SWWAF_TRUSTED_PROXIES` (default `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`,
   the private address ranges): the netblocks whose `X-Forwarded-For` is
   believed, normally where traefik comes from. Set but empty, it trusts nothing:
@@ -550,8 +552,8 @@ ban.
 Clients of the AS numbers and countries listed here get a percentage of every
 rate limit and byte limit. The same rules then ban them after fewer requests. A
 lowered limit is rounded down. `0` is a zero allowance: the client's first
-request bans it. Each client is counted on its own. No budget is shared by a
+request bans it. Each client is counted on its own. An AS number or a country
-whole AS number or country, which one abuser could use up.
+has no shared budget, since one abuser could use it up for everyone in it.
 
 - `SWWAF_ASN_LIMIT_PERCENT` (default empty): a list of AS numbers, each with `:`
   and a percentage, such as `AS14061:50,AS16276:50,AS45102:25`. An AS number
@@ -622,8 +624,8 @@ do. The defaults judge it by what it does to your service.
 - `SWWAF_REPUTATION_ACTION` (default `limit:25`): what is done with a client on
   a DNS blocklist, or one whose AbuseIPDB score is a hit. `deny`,
   `limit:<percent>` or `log`, as for `SWWAF_BLOCKLIST_ACTION`. Such a verdict is
-  less certain than a list like DROP, so by default the client gets a quarter of
+  less certain than the Spamhaus DROP list, so by default the client gets a
-  every limit.
+  quarter of every limit.
 - `SWWAF_REPUTATION_CACHE_TTL` (default `24h`): how long a zone's verdict or
   AbuseIPDB's score is used. A duration; it cannot be `off`.
 - `SWWAF_REPUTATION_TIMEOUT` (default `2s`): how long a query to a zone or a
@@ -666,10 +668,11 @@ See "Blocklists", "DNS blocklists", "AbuseIPDB" and "CrowdSec" below.
       `.git/`, `.gitignore` or `package.json`.
 
     In front of a code forge these refuse git over HTTP, uploads and views of
-    ordinary files. The default rule file bans those probes at the site root
+    ordinary files. The default rule file still bans probes for version control
-    instead. A list given replaces the default, so include them. An id must be a
+    directories and secret files at the site root. A list given replaces the
-    whole number above zero. An id from 900000 to 900999 is refused: those are
+    default, so include them. An id must be a whole number above zero. An id
-    the rules that set the Core Rule Set up, and `smallwebwaf`'s own.
+    from 900000 to 900999 is refused: those are the rules that set the Core Rule
+    Set up, and `smallwebwaf`'s own.
 
 - `SWWAF_WAF_EXEMPT_PATHS` (default empty): path prefixes the Core Rule Set does
   not inspect, matched as `SWWAF_RATE_LIMIT_EXEMPT_PATHS` is.
@@ -794,7 +797,8 @@ thresholds per client are:
 - `SWWAF_ANOMALY_CLIENT_BYTES_PER_MINUTE`;
 - `SWWAF_ANOMALY_CLIENT_BYTES_PER_HOUR`.
 
-The other scopes have the same four names with `CLIENT` replaced:
+The other scopes' names have another prefix before the same four endings, such
+as `REQUESTS_PER_MINUTE`:
 
 - `SWWAF_ANOMALY_NET_...`: per netblock around a client. Two settings give the
   netblock's length:
@@ -843,6 +847,8 @@ docker run \
     ...
 ```
 
+### Fixed limits
+
 A few limits are fixed rather than settings:
 
 - at most 100,000 GeoJS answers are kept, for 7 days each;
@@ -853,9 +859,13 @@ A few limits are fixed rather than settings:
 ## Bans
 
 A ban refuses every request from a netblock until it ends, with
-`SWWAF_BAN_RESPONSE`. The netblock is the client's IPv4 address, or the netblock
+`SWWAF_BAN_RESPONSE`. The netblock is one of:
-around it, `SWWAF_BAN_SCOPE_V4_PREFIX` long, or the client's IPv6 group. A
+
-banned client is not looked up and not counted by rate limiting.
+- the client's IPv4 address;
+- the netblock around it, `SWWAF_BAN_SCOPE_V4_PREFIX` long;
+- the client's IPv6 group.
+
+A banned client is not looked up and not counted by rate limiting.
 
 `smallwebwaf` makes bans for three causes:
 
@@ -891,8 +901,8 @@ Each ban carries notes for deciding whether to lift it:
   refused;
 - how many earlier bans the netblock had, by cause.
 
-At most `SWWAF_MAX_BANS` bans `smallwebwaf` made are kept. Past that, one ban is
+`smallwebwaf` keeps at most `SWWAF_MAX_BANS` of the bans it made. Past that, one
-dropped: the earliest ban of the netblock that has gone longest without a
+ban is dropped: the earliest ban of the netblock that has gone longest without a
 request. Bans whose cause is `admin` are never dropped. Those are the bans you
 make, and the bans `smallwebwaf` made that you keep by setting their cause.
 
@@ -996,13 +1006,17 @@ scanner-agent  user_agent  ban     (?i)\b(sqlmap|nikto|nuclei|wpscan)\b
   takes time linear in the text it reads. It matches anywhere in the target
   unless anchored with `^` and `$`. `(?i)` at its front makes it ignore case.
 
-A line that is not a rule stops the start, with a message naming the file and
+These stop the start, with a message naming the file and line:
-line. So does a bad `header:` name, a rule for `Host` or `Transfer-Encoding`, a
+
-regex that does not compile, or an id used twice. While it runs, `smallwebwaf`
+- a line that is not a rule;
-watches the directory. It reads the files again once the directory has had no
+- a bad `header:` name, or a rule for `Host` or `Transfer-Encoding`;
-change for 2 seconds, so that a file still being written is read whole. If the
+- a regex that does not compile;
-files then hold such an error, the rules stay as they were. The log and a
+- an id used twice.
-`file_error` alert name the file and line.
+
+While it runs, `smallwebwaf` watches the directory. It reads the files again
+once the directory has had no change for 2 seconds, so that a file still being
+written is read whole. If the files then hold such an error, the rules stay as
+they were. The log and a `file_error` alert name the file and line.
 
 The image ships one file, `share/rules.d/00-default.rules`, installed as
 `/etc/smallwebwaf/rules.d/00-default.rules`. Its rules:
@@ -1057,14 +1071,18 @@ setting undoes them.
   address or `localhost` (rules 931100 and 934110). Git credential helpers ask
   to be sent back to such a URL.
 - Ten query parameters are not checked against three lists: system files
-  (930120), shell paths (932160) and command names (932260). They are `path`,
+  (930120), shell paths (932160) and command names (932260). They are:
-  `files`, `skip-to`, `sub_path`, `ref`, `sha`, `branch`, `workflow`,
+    - `path`, `files`, `skip-to` and `sub_path`;
-  `artifactName` and `redirect_to`. A file such as `.gitignore` or a branch such
+    - `ref`, `sha`, `branch` and `workflow`;
-  as `docker-build` then gets through there. So does an attack only those three
+    - `artifactName` and `redirect_to`.
-  rules catch, such as `|cat /etc/passwd`. Path traversal and SQL injection are
+
-  still refused there. Names are matched without regard to case. An app that
+    A file such as `.gitignore` or a branch such as `docker-build` then gets
-  uses one of these parameters as a file on the server, or passes it to a shell,
+    through there. So does an attack only those three rules catch, such as
-  has to check it itself.
+    `|cat /etc/passwd`. Path traversal and SQL injection are still refused
+    there. Names are matched without regard to case. An app that uses one of
+    these parameters as a file on the server, or passes it to a shell, has to
+    check it itself.
+
 - The cookies `gitea_flash` and `redirect_to` are not read. `Referer` is not
   checked for a Unix command given without arguments (932340), or for Java
   starting a process (944110).
@@ -1122,9 +1140,13 @@ refused ones included:
 {"type":"request","time":"2026-10-03T12:00:00.123Z","instance":"fsn1app1/gitea","client_ip":"203.0.113.9","method":"GET","scheme":"https","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","peer_ip":"172.18.0.2","forwarded_for":"203.0.113.9","client_group":"203.0.113.9/32","asn":"AS64496","as_name":"Example Net","country":"DE","request_headers":{"accept":"*/*"},"response_content_type":"text/html; charset=utf-8","upstream_status":200,"action":"forward","counts":{"minute":1,"hour":12,"day":40,"minute_bytes":5120,"hour_bytes":61440,"day_bytes":204800},"duration_total":3.217,"duration_checks":0.041,"duration_upstream_connect":0.052,"duration_upstream_first_byte":2.874,"duration_upstream_total":3.104}
 ```
 
-A field that does not apply to a request is left out. Every line has `type`, the
+A field that does not apply to a request is left out. Every line has:
-fields from `time` to `user_agent`, `request_id`, `peer_ip`, `client_group`,
+
-`asn`, `as_name`, `country`, `action` and `duration_total`.
+- `type`;
+- the fields from `time` to `user_agent`;
+- `request_id`, `peer_ip` and `client_group`;
+- `asn`, `as_name` and `country`;
+- `action` and `duration_total`.
 
 - `time` is when the request arrived, in UTC. `instance` is
   `SWWAF_INSTANCE_NAME`. `scheme` is the `X-Forwarded-Proto` a trusted proxy
@@ -1179,16 +1201,20 @@ fields from `time` to `user_agent`, `request_id`, `peer_ip`, `client_group`,
   limit; `minute_bytes`, `hour_bytes` or `day_bytes` for a byte limit; or
   `error_burst`. `offence` is then `limit`. A request whose bytes broke a limit
   was passed on whole, so its `action` stays `forward`.
-- `reputation` lists the sources the client is on: blocklist URLs, the CrowdSec
+- `reputation` lists the sources the client is on, in this order:
-  decision list's URL, DNS blocklist zones, then `abuseipdb`.
+    - blocklist URLs;
+    - the CrowdSec decision list's URL;
+    - DNS blocklist zones;
+    - `abuseipdb`.
 - `ban_expires` is when the client's ban ends, for a request that made one or
   was refused under one: a time, or `permanent`.
 - The timings are in milliseconds. `duration_total` runs from the request's
   headers being read to its log line. `duration_checks` runs to the end of the
   checks, and `duration_waf` is the part of that the Core Rule Set took. The
   three `duration_upstream_` timings run from the hand-over to the app:
-  `connect` until there was a connection, `first_byte` until the first byte of
+    - `connect`, until there was a connection;
-  the answer, and `total` until the end.
+    - `first_byte`, until the first byte of the answer;
+    - `total`, until the end.
 
 No body is logged, and no header but those above. `smallwebwaf`'s own messages
 (start, the settings, stop, errors) share the stream as JSON lines marked
@@ -1231,11 +1257,15 @@ its event is in `SWWAF_ALERT_EVENTS`:
 - `reputation_hit`: a request from a client on a blocklist, the CrowdSec
   decision list or a DNS blocklist, or with an AbuseIPDB score that is a hit.
   One alert per source.
-- `source_failure`: GeoJS failing, a list that cannot be fetched, a DNS
+- `source_failure`: one of:
-  blocklist or AbuseIPDB failing or refusing a query, or the day's AbuseIPDB
+    - GeoJS failing;
-  checks used up.
+    - a list that cannot be fetched;
-- `file_error`: a rule file or state file edit with an error, a state file that
+    - a DNS blocklist or AbuseIPDB failing or refusing a query;
-  cannot be written, or a replacement lookup database that cannot be read.
+    - the day's AbuseIPDB checks used up.
+- `file_error`: one of:
+    - a rule file or state file edit with an error;
+    - a state file that cannot be written;
+    - a replacement lookup database that cannot be read.
 
 The bans you make raise no alert. In `observe` mode, a request that would have
 made a ban still raises the ban's alert, with `mode` set to `observe` in its
@@ -1311,9 +1341,9 @@ line for each of these that the alert has:
 - the file, the source, the error and the mode from `detail`;
 - the suppressed repeats, if any.
 
-Slack is posted the title in bold and the text below it, with `&`, `<` and `>`
+Slack gets the title in bold and the text below it, with `&`, `<` and `>`
-escaped. ntfy is posted the text, with the title in `Title`, the token if set,
+escaped. ntfy gets the text, with the title in `Title`, the token if set, and a
-and a priority and a tag by event:
+priority and a tag by event:
 
 - `ban`: priority `default`, tag `no_entry`.
 - `permanent_ban`: priority `high`, tag `no_entry`.
@@ -1327,8 +1357,9 @@ A repeat is an alert for the same event on the same netblock, source, file or
 anomaly scope, less than `SWWAF_ALERT_COOLDOWN` after the last. It is held back
 and counted, and the next alert sent gives the count. Past
 `SWWAF_ALERT_MAX_PER_HOUR` alerts in an hour, the rest are held back and counted
-by event. Once the hour ends, one alert with the event `summary` sums them up,
+by event. Once the hour ends, one alert with the event `summary` gives those
-with the repeats of the cooldowns that ran out.
+counts. It also gives the repeats a cooldown held back that no later alert
+reported.
 
 Each destination has a queue of at most 1000 alerts, sent one at a time, so a
 slow destination holds up nothing. A `2xx` answer takes an alert. A `4xx` other
@@ -1418,9 +1449,11 @@ keeps it as `exported_instance` unless the scrape sets `honor_labels: true`.
     - its offences by kind: `limit`, `attack`, `rule_blocked`, `waf_blocked` and
       `token_refused`.
 - `lookups.json`: GeoJS's answers, one per line.
-- `reputation.json`: the last good copy of each list fetched from a URL, with
+- `reputation.json`:
-  when it was last tried; each DNS blocklist verdict still in use; and
+    - the last good copy of each list fetched from a URL, with when it was last
-  AbuseIPDB's scores and the checks spent today.
+      tried;
+    - each DNS blocklist verdict still in use;
+    - AbuseIPDB's scores, and the checks spent today.
 - `alerts.json`: the alert cooldowns, the hour under way, the alerts waiting for
   each destination, and the anomaly counters.
 
@@ -1463,8 +1496,8 @@ AS number and country. These addresses are never sent:
 - one under a ban;
 - a private, loopback or link-local address, which no source can place.
 
-An IPv6 visitor is asked about by the first address of its group. Each answer is
+For an IPv6 visitor, GeoJS is asked about the first address of its group. Each
-kept for seven days, in memory and in `lookups.json`.
+answer is kept for seven days, in memory and in `lookups.json`.
 
 A request waits for its client's first answer only when a setting needs the
 answer before the request goes on. It waits up to `SWWAF_LOOKUP_TIMEOUT`. The
@@ -1490,11 +1523,11 @@ address it does not hold counts as from an unknown country. `smallwebwaf` reads
 the whole file at start. It reads the file again once it has gone 2 seconds
 without a change after you replace it. A replacement it cannot read is logged
 and alerted, and the file read before stays in use. Mount the directory that
-holds the file rather than the file itself: many editors and download tools
+holds the file, not the file itself. Many editors and download tools save a file
-replace a file when they save it, and a container keeps seeing the old one when
+by replacing it, and a container keeps seeing the old file when only the file is
-the file alone is mounted. IPinfo releases the database under the Creative
+mounted. IPinfo releases the database under the Creative Commons
-Commons Attribution-ShareAlike 4.0 licence. It asks for a link to IPinfo on the
+Attribution-ShareAlike 4.0 licence. It asks for a link to IPinfo on the site
-site that uses it, such as "IP address data is powered by IPinfo".
+that uses it, such as "IP address data is powered by IPinfo".
 
 ## Blocklists
 
@@ -1515,8 +1548,8 @@ list again before its time. A fetch fails when:
   `AS<number>:<percent>`.
 
 The copy fetched before stays in use. The failure is logged and raised as a
-`source_failure` alert. Each list is named by its URL in the log, the alerts,
+`source_failure` alert. The log, the alerts, the ban notes and the metrics name
-the ban notes and the metrics, so keep a secret out of it.
+each list by its URL, so keep a secret out of it.
 
 `deny` suits lists of networks that send nothing legitimate, such as DROP.
 `limit:<percent>` suits lists of addresses shared with ordinary visitors, such
@@ -1563,18 +1596,23 @@ client that has committed an offence is checked. An offence is one of:
 
 The check runs in the background, at the client's first request after the
 offence that no ban refuses, so no request waits. The score applies to the
-client's later requests. An IPv6 group is checked by the address of the request
+client's later requests. For an IPv6 client, AbuseIPDB is asked about the
-that has it checked, and the score is used for the whole group. A score at or
+address of the request that set off the check, and the score is used for the
-over `SWWAF_ABUSEIPDB_MIN_SCORE` is a hit, and `SWWAF_REPUTATION_ACTION` says
+whole group. A score at or over `SWWAF_ABUSEIPDB_MIN_SCORE` is a hit, and
-what is done. Each score is kept for `SWWAF_REPUTATION_CACHE_TTL`.
+`SWWAF_REPUTATION_ACTION` says what is done. Each score is kept for
+`SWWAF_REPUTATION_CACHE_TTL`.
 
 At most `SWWAF_ABUSEIPDB_DAILY_BUDGET` checks are made in a day, counted in UTC
 and kept in `reputation.json`. A restart does not make the budget whole again.
 The check that spends the last of the budget is logged and alerted. No client is
-checked until the day ends. A check fails when AbuseIPDB answers other than
+checked until the day ends. A check fails when:
-`200`, gives no score, or does not answer within `SWWAF_REPUTATION_TIMEOUT`. A
+
-failure is logged and alerted, and no client is checked for a minute. The key is
+- AbuseIPDB answers other than `200`;
-sent in the `Key` header and nowhere else, and is never logged.
+- the answer gives no score;
+- no answer comes within `SWWAF_REPUTATION_TIMEOUT`.
+
+A failure is logged and alerted, and no client is checked for a minute. The key
+is sent in the `Key` header and nowhere else, and is never logged.
 
 ## CrowdSec
 
@@ -1589,13 +1627,15 @@ A request is refused when a decision still in force names its client. The
 client's netblock is banned until the decision ends, with the cause `crowdsec`.
 Only a client that sends a request is banned, so a long list does not fill
 `bans.json`. Such a ban is never made permanent, and does not lengthen the
-netblock's next ban for a broken limit. A ban you lift while the decision is in
+netblock's next ban for a broken limit. If you lift a ban while the decision is
-force is followed by another at the client's next request. Delete the decision
+in force, the client's next request makes another. Delete the decision in
-in CrowdSec first, such as with `cscli decisions delete --ip 203.0.113.9`, and
+CrowdSec first, such as with `cscli decisions delete --ip 203.0.113.9`, and lift
-lift the ban once the list has been fetched again. A fetch fails when the engine
+the ban once the list has been fetched again. A fetch fails when:
-answers other than `200`, a redirect included. It also fails when the fetch does
+
-not finish within a minute, the answer is longer than 16 MiB, or the answer is
+- the engine answers other than `200`, a redirect included;
-not a list of decisions.
+- the fetch does not finish within a minute;
+- the answer is longer than 16 MiB;
+- the answer is not a list of decisions.
 
 ## Rationale
 
@@ -1636,8 +1676,8 @@ candidates one by one.
   or `SIGINT`. Then it stops, writing the state files. Run as
   `smallwebwaf healthcheck`, it is the image's health check instead.
 - `internal/config`: reads the settings, the one place they are read.
-- `internal/proxy`: what happens to each request. It works out the client, runs
+- `internal/proxy`: what happens to each request. It works out the client and
-  the checks in `check`, and passes the request to the app with the standard
+  runs the checks in `check`. It passes the request to the app with the standard
   library's `httputil.ReverseProxy`, within the timeouts and size limits. It
   answers its own endpoints in `answerAdmin`. Once the request has ended, it
   counts bytes, refusals and anomalies, and writes the log line.
@@ -1683,13 +1723,16 @@ Besides the standard library, it uses:
 
 This repository adheres to the
 [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
-standard: the scripts in `script/` are the entrypoints for working on it, and
+standard. The scripts in `script/` are the entrypoints for working on it, and
 the `Makefile` targets are thin shims that call them. The scripts are POSIX sh.
 The tests and the linter run in Docker, as phases of the `Dockerfile`, and never
 on the host.
 
-- `script/bootstrap`: installs what the other scripts need on the host: `make`,
+- `script/bootstrap`: installs what the other scripts need on the host.
-  `git`, `curl`, Go for `gofmt`, node and yarn, and prettier. `make bootstrap`.
+  `make bootstrap`. They need:
+    - `make`, `git` and `curl`;
+    - Go, for `gofmt`;
+    - node, yarn and prettier.
 - `script/setup`: readies a fresh clone: `script/bootstrap`, then
   `script/install-precommit`. `make setup`.
 - `script/projectname`: prints `smallwebwaf`, which the other scripts tag their