Last pass: the sixth README review and the fifth SPEC review

The spec names every fault that stops the start, runs step 15 for every
request but the health check, and states each of the fourteen repeated
requirements once. docs/alerts.md says which alerts have an empty client
and netblock and what a summary's detail holds; docs/settings.md lists
the settings that need a lookup and points at step 3 for the two static
lists. The README and the spec now travel in one branch, so the README
never names a spec section that does not exist yet.

Model: fable-5-1

README.md

@@ -150,8 +150,8 @@ request that counts against its client in its history. It is one of:
 - a broken limit;
 - a refusal by a rule, by the Core Rule Set or for a missing or wrong token.
 
-`smallwebwaf` handles each request in this order. A request one step refuses
+`smallwebwaf` handles each request in this order. Once a step refuses a request,
-skips the steps after it, apart from step 13. Nothing of a refused request
+the steps after it are skipped, apart from step 13. Nothing of a refused request
 reaches the app. "What happens to a request" in [`SPEC.md`](SPEC.md) gives each
 step in full.
 

docs/alerts.md

@@ -79,10 +79,11 @@ on one line:
 - `instance`: `SWWAF_INSTANCE_NAME`. `time`: when the alert was raised. `event`:
   one of the events above.
 - `client`: the address whose request raised the alert. Empty for
-  `source_failure` and `file_error`.
+  `source_failure`, `file_error` and `summary`.
-- `netblock`: empty for `source_failure` and `file_error`; otherwise one of:
+- `netblock`: empty for `source_failure`, `file_error` and `summary`; otherwise
+  one of:
     - the ban's netblock;
-    - the netblock an anomaly counted;
+    - the netblock an anomaly counted, empty in the `asn` and `total` scopes;
     - for the other events, the client as rate limiting counts it: an IPv4
       address as a /32, or an IPv6 group.
 - `asn`, `as_name` and `country`: the client's, as known when the alert was
@@ -97,6 +98,8 @@ on one line:
     - for a Core Rule Set match, the `rule_ids`, `score`, `method` and `path`;
     - for a source failure, the `source` and `error`;
     - for a file error, the `file` and `error`;
+    - for a summary, the `hour` it covers, the `count` of alerts held back, and
+      that count by event in `events`;
     - `mode`, when the request was not refused: `detect` or `observe`.
 - `suppressed_repeats`: how many repeats the cooldown held back before this
   alert.

docs/metrics.md

@@ -1,8 +1,8 @@
 # The metrics
 
-The metrics `GET /_smallwebwaf/metrics` answers with, as "Metrics endpoint" in
+What `GET /_smallwebwaf/metrics` answers with. "Metrics endpoint" in
-[`SPEC.md`](../SPEC.md) describes it. Prometheus keeps the `instance` label as
+[`SPEC.md`](../SPEC.md) describes the endpoint. Prometheus keeps the `instance`
-`exported_instance` unless the scrape sets `honor_labels: true`.
+label as `exported_instance` unless the scrape sets `honor_labels: true`.
 
 ## Traffic
 

docs/settings.md

@@ -91,11 +91,11 @@ that take `off`, the `_FILE` form, and what each group does and why.
 ## Static lists
 
 - `SWWAF_ALLOW_NETS` (default empty): netblocks you trust, such as your
-  monitoring. Their clients are refused only for a size or time limit or a
+  monitoring (see step 3 in [`SPEC.md`](../SPEC.md)).
-  missing or wrong token.
 - `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients are
   exempt from rate limiting and the byte limits.
-- `SWWAF_DENY_NETS` (default empty): netblocks whose clients are always refused.
+- `SWWAF_DENY_NETS` (default empty): netblocks whose clients are refused (see
+  step 3 in [`SPEC.md`](../SPEC.md)).
 
 ## Request rate limits, per client
 
@@ -136,7 +136,13 @@ that take `off`, the `_FILE` form, and what each group does and why.
 
 ## Lookup of AS number and country
 
-- `SWWAF_LOOKUP_SOURCE` (default `geojs`): `geojs`, `file` or `off`.
+- `SWWAF_LOOKUP_SOURCE` (default `geojs`): `geojs`, `file` or `off`. `off` stops
+  the start while a setting that needs a lookup is set:
+    - the two country lists;
+    - `SWWAF_ADD_LOOKUP_HEADERS`;
+    - the four biased threshold lists and `SWWAF_ASN_LIMIT_PERCENT_URL`;
+    - `SWWAF_UNKNOWN_LIMIT_PERCENT` below `100`;
+    - an anomaly threshold per AS number.
 - `SWWAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in its
   `.mmdb` form, for `SWWAF_LOOKUP_SOURCE=file`.
 - `SWWAF_LOOKUP_TIMEOUT` (default `1s`): how long a request waits for its
@@ -286,7 +292,7 @@ as `REQUESTS_PER_MINUTE`:
       client;
     - `SWWAF_ANOMALY_NET_V6_PREFIX` (default `48`, from 0 to 128) for an IPv6
       client.
-- `SWWAF_ANOMALY_ASN_...`: per AS number. Needs a lookup source.
+- `SWWAF_ANOMALY_ASN_...`: per AS number.
 - `SWWAF_ANOMALY_TOTAL_...`: the whole service.
 - `SWWAF_WATCH_...`: each netblock in `SWWAF_WATCH_NETS`, as a whole.
   `SWWAF_WATCH_NETS` (default empty) is a list of name, `=` and netblock, such

internal/alerts/alerts.go

@@ -1,8 +1,8 @@
 // Package alerts sends alerts on bans, on traffic over an anomaly
 // threshold, on a source that fails and on a file with an error to each
 // destination set: to the webhook
-// SWWAF_ALERT_WEBHOOK_URL names, each as one JSON object, as the "Alert
+// SWWAF_ALERT_WEBHOOK_URL names, each as one JSON object, as "The JSON
-// webhook schema" section of SPEC.md describes, to the Slack incoming
+// object" in docs/alerts.md describes, to the Slack incoming
 // webhook SWWAF_ALERT_SLACK_WEBHOOK_URL names, as a message, and to the
 // ntfy topic SWWAF_ALERT_NTFY_URL names. A repeat within
 // SWWAF_ALERT_COOLDOWN is held back, and so is an alert past
@@ -149,11 +149,10 @@ type Params struct {
 }
 
 // Alert is one alert, as the webhook is sent it and alerts.json holds it,
-// with the fields of the "Alert webhook schema" section of SPEC.md. ASN,
+// with the fields of "The JSON object" in docs/alerts.md. ASN, ASName and
-// ASName and Country are, for a ban, the client's as the ban's notes give
+// Country are, for a ban, the client's as the ban's notes give them.
-// them.
 //
-//nolint:tagliatelle // SPEC.md's alert webhook schema names its fields in snake_case
+//nolint:tagliatelle // docs/alerts.md names the alert's fields in snake_case
 type Alert struct {
 	Instance string       `json:"instance"`
 	Time     time.Time    `json:"time"`

internal/anomaly/anomaly.go

@@ -1,8 +1,8 @@
 // Package anomaly counts requests and bytes over a minute and an hour, per
 // client, per surrounding netblock, per AS number, for the whole service
 // and per named netblock, and raises an anomaly alert for a count over its
-// threshold, as "Anomaly thresholds" under "Configuration surface" in
+// threshold, as "Anomaly thresholds" under "Settings" in SPEC.md
-// SPEC.md describes. It refuses and bans nothing. At most 20,000 counters
+// describes. It refuses and bans nothing. At most 20,000 counters
 // are kept, in memory, and written to alerts.json and read from it by the
 // state package.
 package anomaly

internal/metrics/metrics.go

@@ -1,6 +1,7 @@
-// Package metrics keeps smallwebwaf's Prometheus metrics, as the "Metrics
+// Package metrics keeps smallwebwaf's Prometheus metrics, as
-// endpoint" section of SPEC.md lists them, and serves them in the
+// docs/metrics.md lists them, and serves them in the Prometheus text
-// Prometheus text format. No metric carries a client's address.
+// format, as the "Metrics endpoint" section of SPEC.md requires. No
+// metric carries a client's address.
 package metrics
 
 import (

internal/ratelimit/ratelimit.go

@@ -206,7 +206,7 @@ type Hit struct {
 // Counts are a client's requests and bytes in the minute, the hour and
 // the day that end at a request, that request's included.
 //
-//nolint:tagliatelle // SPEC.md's request log names its fields in snake_case
+//nolint:tagliatelle // docs/request-log.md names the fields in snake_case
 type Counts struct {
 	Minute      float64 `json:"minute"`
 	Hour        float64 `json:"hour"`

internal/requestlog/requestlog.go

@@ -64,11 +64,10 @@ const LimitHitErrorBurst = "error_burst"
 const timeLayout = "2006-01-02T15:04:05.000Z07:00"
 
 // Line is one request's line in the request log. The field names, and
-// their order, are those of the "Request log" section of SPEC.md. A field
+// their order, are those of docs/request-log.md. A field that may not
-// that may not apply to a request is left out of its line when it does
+// apply to a request is left out of its line when it does not.
-// not.
 //
-//nolint:tagliatelle // SPEC.md's request log names its fields in snake_case
+//nolint:tagliatelle // docs/request-log.md names the fields in snake_case
 type Line struct {
 	Type string `json:"type"`
 

internal/state/state.go

@@ -1,5 +1,6 @@
 // Package state keeps smallwebwaf's state in JSON files in
-// SWWAF_STATE_DIR, as the "Persistent state" section of SPEC.md describes:
+// SWWAF_STATE_DIR, as the "Persistent state" section of SPEC.md requires
+// and docs/state-files.md lists them:
 // bans.json holds the bans, clients.json each client's counters and
 // history, lookups.json GeoJS's answers, reputation.json the last try and
 // last good copy of each list fetched from a URL, the DNSBL zones'

internal/state/state_test.go

@@ -1019,8 +1019,8 @@ func TestEditJustBeforeAScheduledWriteSurvivesIt(t *testing.T) {
 		edit(t, dir, bansJSON, permanentBansJSON)
 
 		// The write takes the edit in first, and writes it back. The second
-		// ban, made after the admin opened the file, is lost, as "Edits
+		// ban, made after the admin opened the file, is lost, as the
-		// while running" in SPEC.md says.
+		// "Persistent state" section of SPEC.md says.
 		time.Sleep(time.Nanosecond)
 		synctest.Wait()
 

internal/waf/waf.go

@@ -1,8 +1,8 @@
 // Package waf runs the OWASP Core Rule Set 4.25.0, through Coraza, on the
 // method, the URL with its query and the headers of a request, and on its
 // body while SWWAF_WAF_BODY_LIMIT is set, with the six changes smallwebwaf
-// makes to it, as "Attack detection" under "Configuration surface" in
+// makes to it, as "Attack detection" under "Settings" in SPEC.md describes
-// SPEC.md describes them. It reads no response.
+// them. It reads no response.
 //
 // smallwebwaf writes only to its state directory, so Coraza is built with
 // its no_fs_access tag, as the Dockerfile and script/build build it: of a

script/gitea-requests

@@ -2,11 +2,12 @@
 # script/gitea-requests: build the image and run gitea 1.27.3 behind it,
 # with every setting of smallwebwaf at its default but SWWAF_UPSTREAM_URL,
 # which names gitea's container. Then send through it the gitea requests
-# that reading gitea's source found the Core Rule Set refuses, and a short
+# that reading gitea's source found the Core Rule Set refuses, which
-# sample of gitea's ordinary use: signing in, pages of files and commits,
+# docs/gitea.md records, and a short sample of gitea's ordinary use:
-# a diff, the API, a push, clone and fetch over HTTP, and two container
+# signing in, pages of files and commits, a diff, the API, a push, clone
-# image pushes. Last, print each request smallwebwaf logged, with what it
+# and fetch over HTTP, and two container image pushes. Last, print each
-# did with it and the ids of the Core Rule Set's rules that matched it.
+# request smallwebwaf logged, with what it did with it and the ids of the
+# Core Rule Set's rules that matched it.
 # Every request comes from one client, docker's gateway, so more than
 # SWWAF_ERROR_BURST_THRESHOLD, 30, refusals in a minute would ban it, and
 # the requests after would show as banned. The containers, the network,