README rewritten from scratch so a newcomer can follow it (closes #154, closes #155)

The old README opened with project history and one list of a hundred
settings. The new one leads with what smallwebwaf is, what it protects
against, how to build and deploy it, what happens to a request, the
settings most deployments touch, how to check it works, the entrypoints,
the rationale and the design, as REPO_POLICIES.md requires. The reference
lists, every setting, the rule file format, the endpoints, the request
log's fields, the alert events, the metrics, the state files, the image
and the gitea notes, move to files under docs/, each linked from the
README, with SPEC.md as the one place for behaviour. Every statement was
checked against the code on next.

Model: fable-5-1

README.md

@@ -150,8 +150,8 @@ request that counts against its client in its history. It is one of:
 - a broken limit;
 - a refusal by a rule, by the Core Rule Set or for a missing or wrong token.
 
-`smallwebwaf` handles each request in this order. Once a step refuses a request,
-the steps after it are skipped, apart from step 13. Nothing of a refused request
+`smallwebwaf` handles each request in this order. A request one step refuses
+skips the steps after it, apart from step 13. Nothing of a refused request
 reaches the app. "What happens to a request" in [`SPEC.md`](SPEC.md) gives each
 step in full.
 

docs/alerts.md

@@ -79,11 +79,10 @@ on one line:
 - `instance`: `SWWAF_INSTANCE_NAME`. `time`: when the alert was raised. `event`:
   one of the events above.
 - `client`: the address whose request raised the alert. Empty for
-  `source_failure`, `file_error` and `summary`.
-- `netblock`: empty for `source_failure`, `file_error` and `summary`; otherwise
-  one of:
+  `source_failure` and `file_error`.
+- `netblock`: empty for `source_failure` and `file_error`; otherwise one of:
     - the ban's netblock;
-    - the netblock an anomaly counted, empty in the `asn` and `total` scopes;
+    - the netblock an anomaly counted;
     - for the other events, the client as rate limiting counts it: an IPv4
       address as a /32, or an IPv6 group.
 - `asn`, `as_name` and `country`: the client's, as known when the alert was
@@ -98,8 +97,6 @@ on one line:
     - for a Core Rule Set match, the `rule_ids`, `score`, `method` and `path`;
     - for a source failure, the `source` and `error`;
     - for a file error, the `file` and `error`;
-    - for a summary, the `hour` it covers, the `count` of alerts held back, and
-      that count by event in `events`;
     - `mode`, when the request was not refused: `detect` or `observe`.
 - `suppressed_repeats`: how many repeats the cooldown held back before this
   alert.

docs/metrics.md

@@ -1,8 +1,8 @@
 # The metrics
 
-What `GET /_smallwebwaf/metrics` answers with. "Metrics endpoint" in
-[`SPEC.md`](../SPEC.md) describes the endpoint. Prometheus keeps the `instance`
-label as `exported_instance` unless the scrape sets `honor_labels: true`.
+The metrics `GET /_smallwebwaf/metrics` answers with, as "Metrics endpoint" in
+[`SPEC.md`](../SPEC.md) describes it. Prometheus keeps the `instance` label as
+`exported_instance` unless the scrape sets `honor_labels: true`.
 
 ## Traffic
 

docs/settings.md

@@ -91,11 +91,11 @@ that take `off`, the `_FILE` form, and what each group does and why.
 ## Static lists
 
 - `SWWAF_ALLOW_NETS` (default empty): netblocks you trust, such as your
-  monitoring (see step 3 in [`SPEC.md`](../SPEC.md)).
+  monitoring. Their clients are refused only for a size or time limit or a
+  missing or wrong token.
 - `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients are
   exempt from rate limiting and the byte limits.
-- `SWWAF_DENY_NETS` (default empty): netblocks whose clients are refused (see
-  step 3 in [`SPEC.md`](../SPEC.md)).
+- `SWWAF_DENY_NETS` (default empty): netblocks whose clients are always refused.
 
 ## Request rate limits, per client
 
@@ -136,13 +136,7 @@ that take `off`, the `_FILE` form, and what each group does and why.
 
 ## Lookup of AS number and country
 
-- `SWWAF_LOOKUP_SOURCE` (default `geojs`): `geojs`, `file` or `off`. `off` stops
-  the start while a setting that needs a lookup is set:
-    - the two country lists;
-    - `SWWAF_ADD_LOOKUP_HEADERS`;
-    - the four biased threshold lists and `SWWAF_ASN_LIMIT_PERCENT_URL`;
-    - `SWWAF_UNKNOWN_LIMIT_PERCENT` below `100`;
-    - an anomaly threshold per AS number.
+- `SWWAF_LOOKUP_SOURCE` (default `geojs`): `geojs`, `file` or `off`.
 - `SWWAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in its
   `.mmdb` form, for `SWWAF_LOOKUP_SOURCE=file`.
 - `SWWAF_LOOKUP_TIMEOUT` (default `1s`): how long a request waits for its
@@ -292,7 +286,7 @@ as `REQUESTS_PER_MINUTE`:
       client;
     - `SWWAF_ANOMALY_NET_V6_PREFIX` (default `48`, from 0 to 128) for an IPv6
       client.
-- `SWWAF_ANOMALY_ASN_...`: per AS number.
+- `SWWAF_ANOMALY_ASN_...`: per AS number. Needs a lookup source.
 - `SWWAF_ANOMALY_TOTAL_...`: the whole service.
 - `SWWAF_WATCH_...`: each netblock in `SWWAF_WATCH_NETS`, as a whole.
   `SWWAF_WATCH_NETS` (default empty) is a list of name, `=` and netblock, such

internal/alerts/alerts.go

@@ -1,8 +1,8 @@
 // Package alerts sends alerts on bans, on traffic over an anomaly
 // threshold, on a source that fails and on a file with an error to each
 // destination set: to the webhook
-// SWWAF_ALERT_WEBHOOK_URL names, each as one JSON object, as "The JSON
-// object" in docs/alerts.md describes, to the Slack incoming
+// SWWAF_ALERT_WEBHOOK_URL names, each as one JSON object, as the "Alert
+// webhook schema" section of SPEC.md describes, to the Slack incoming
 // webhook SWWAF_ALERT_SLACK_WEBHOOK_URL names, as a message, and to the
 // ntfy topic SWWAF_ALERT_NTFY_URL names. A repeat within
 // SWWAF_ALERT_COOLDOWN is held back, and so is an alert past
@@ -149,10 +149,11 @@ type Params struct {
 }
 
 // Alert is one alert, as the webhook is sent it and alerts.json holds it,
-// with the fields of "The JSON object" in docs/alerts.md. ASN, ASName and
-// Country are, for a ban, the client's as the ban's notes give them.
+// with the fields of the "Alert webhook schema" section of SPEC.md. ASN,
+// ASName and Country are, for a ban, the client's as the ban's notes give
+// them.
 //
-//nolint:tagliatelle // docs/alerts.md names the alert's fields in snake_case
+//nolint:tagliatelle // SPEC.md's alert webhook schema names its fields in snake_case
 type Alert struct {
 	Instance string       `json:"instance"`
 	Time     time.Time    `json:"time"`

internal/anomaly/anomaly.go

@@ -1,8 +1,8 @@
 // Package anomaly counts requests and bytes over a minute and an hour, per
 // client, per surrounding netblock, per AS number, for the whole service
 // and per named netblock, and raises an anomaly alert for a count over its
-// threshold, as "Anomaly thresholds" under "Settings" in SPEC.md
-// describes. It refuses and bans nothing. At most 20,000 counters
+// threshold, as "Anomaly thresholds" under "Configuration surface" in
+// SPEC.md describes. It refuses and bans nothing. At most 20,000 counters
 // are kept, in memory, and written to alerts.json and read from it by the
 // state package.
 package anomaly

internal/metrics/metrics.go

@@ -1,7 +1,6 @@
-// Package metrics keeps smallwebwaf's Prometheus metrics, as
-// docs/metrics.md lists them, and serves them in the Prometheus text
-// format, as the "Metrics endpoint" section of SPEC.md requires. No
-// metric carries a client's address.
+// Package metrics keeps smallwebwaf's Prometheus metrics, as the "Metrics
+// endpoint" section of SPEC.md lists them, and serves them in the
+// Prometheus text format. No metric carries a client's address.
 package metrics
 
 import (

internal/ratelimit/ratelimit.go

@@ -206,7 +206,7 @@ type Hit struct {
 // Counts are a client's requests and bytes in the minute, the hour and
 // the day that end at a request, that request's included.
 //
-//nolint:tagliatelle // docs/request-log.md names the fields in snake_case
+//nolint:tagliatelle // SPEC.md's request log names its fields in snake_case
 type Counts struct {
 	Minute      float64 `json:"minute"`
 	Hour        float64 `json:"hour"`

internal/requestlog/requestlog.go

@@ -64,10 +64,11 @@ const LimitHitErrorBurst = "error_burst"
 const timeLayout = "2006-01-02T15:04:05.000Z07:00"
 
 // Line is one request's line in the request log. The field names, and
-// their order, are those of docs/request-log.md. A field that may not
-// apply to a request is left out of its line when it does not.
+// their order, are those of the "Request log" section of SPEC.md. A field
+// that may not apply to a request is left out of its line when it does
+// not.
 //
-//nolint:tagliatelle // docs/request-log.md names the fields in snake_case
+//nolint:tagliatelle // SPEC.md's request log names its fields in snake_case
 type Line struct {
 	Type string `json:"type"`
 

internal/state/state.go

@@ -1,6 +1,5 @@
 // Package state keeps smallwebwaf's state in JSON files in
-// SWWAF_STATE_DIR, as the "Persistent state" section of SPEC.md requires
-// and docs/state-files.md lists them:
+// SWWAF_STATE_DIR, as the "Persistent state" section of SPEC.md describes:
 // bans.json holds the bans, clients.json each client's counters and
 // history, lookups.json GeoJS's answers, reputation.json the last try and
 // last good copy of each list fetched from a URL, the DNSBL zones'

internal/state/state_test.go

@@ -1019,8 +1019,8 @@ func TestEditJustBeforeAScheduledWriteSurvivesIt(t *testing.T) {
 		edit(t, dir, bansJSON, permanentBansJSON)
 
 		// The write takes the edit in first, and writes it back. The second
-		// ban, made after the admin opened the file, is lost, as the
-		// "Persistent state" section of SPEC.md says.
+		// ban, made after the admin opened the file, is lost, as "Edits
+		// while running" in SPEC.md says.
 		time.Sleep(time.Nanosecond)
 		synctest.Wait()
 

internal/waf/waf.go

@@ -1,8 +1,8 @@
 // Package waf runs the OWASP Core Rule Set 4.25.0, through Coraza, on the
 // method, the URL with its query and the headers of a request, and on its
 // body while SWWAF_WAF_BODY_LIMIT is set, with the six changes smallwebwaf
-// makes to it, as "Attack detection" under "Settings" in SPEC.md describes
-// them. It reads no response.
+// makes to it, as "Attack detection" under "Configuration surface" in
+// SPEC.md describes them. It reads no response.
 //
 // smallwebwaf writes only to its state directory, so Coraza is built with
 // its no_fs_access tag, as the Dockerfile and script/build build it: of a

script/gitea-requests

@@ -2,12 +2,11 @@
 # script/gitea-requests: build the image and run gitea 1.27.3 behind it,
 # with every setting of smallwebwaf at its default but SWWAF_UPSTREAM_URL,
 # which names gitea's container. Then send through it the gitea requests
-# that reading gitea's source found the Core Rule Set refuses, which
-# docs/gitea.md records, and a short sample of gitea's ordinary use:
-# signing in, pages of files and commits, a diff, the API, a push, clone
-# and fetch over HTTP, and two container image pushes. Last, print each
-# request smallwebwaf logged, with what it did with it and the ids of the
-# Core Rule Set's rules that matched it.
+# that reading gitea's source found the Core Rule Set refuses, and a short
+# sample of gitea's ordinary use: signing in, pages of files and commits,
+# a diff, the API, a push, clone and fetch over HTTP, and two container
+# image pushes. Last, print each request smallwebwaf logged, with what it
+# did with it and the ids of the Core Rule Set's rules that matched it.
 # Every request comes from one client, docker's gateway, so more than
 # SWWAF_ERROR_BURST_THRESHOLD, 30, refusals in a minute would ban it, and
 # the requests after would show as banned. The containers, the network,