Merge pull request 'next into main' (#160) from next into main

Reviewed-on: #160

README.md

@@ -8,8 +8,7 @@ scanners and other abusers before they reach the app.
 A web application firewall reads each request on its way to a web app and
 refuses the ones that look like abuse. `smallwebwaf` is also a reverse proxy: it
 accepts requests on the app's behalf, passes them to the app, and passes the
-app's answers back. A request it does not refuse reaches the app as it came,
-apart from a few headers (see "Design" below).
+app's answers back.
 
 It needs no setting. Every setting has a default chosen for a service on the
 open internet. [`SPEC.md`](SPEC.md) says how `smallwebwaf` behaves and why. The
@@ -30,8 +29,8 @@ reference documents under [`docs/`](docs/) are listed under "Documents" below.
   of your own, or an entry in `SWWAF_TRAP_PATHS`, bans those the same way.
 - Attacks carried in a request's URL or headers, such as SQL injection and path
   traversal. The OWASP Core Rule Set is a published set of rules that recognise
-  common web attacks. Coraza, a Go engine, runs those rules (step 11 under "What
-  happens to a request" below). A request the rules take for an attack is
+  common web attacks. Coraza, a Go engine, runs those rules (see "Attack
+  detection" in [`SPEC.md`](SPEC.md)). A request the rules take for an attack is
   refused.
 - Abuse from networks and countries you do not want. You can:
     - refuse whole netblocks and countries;
@@ -142,89 +141,32 @@ while working on it, see "Entrypoints" below.
 
 ## What happens to a request
 
-Two terms recur below. A clear sign of attack is a request that matches a rule
-whose action is `ban`, or asks for a trap path (steps 9 and 10). An offence is a
-request that counts against its client in its history. It is one of:
+`smallwebwaf` handles each request in fifteen steps, in this order. Once a step
+refuses a request, the steps after it are skipped, apart from step 15, and
+nothing of the request reaches the app. "What happens to a request" in
+[`SPEC.md`](SPEC.md) gives each step in full, under the same numbers.
 
-- a clear sign of attack;
-- a broken limit;
-- a refusal by a rule, by the Core Rule Set or for a missing or wrong token.
+1. The health check, `GET /_smallwebwaf/healthz`, is answered at once.
+2. The client is identified.
+3. The static lists, `SWWAF_ALLOW_NETS` and `SWWAF_DENY_NETS`.
+4. The bans.
+5. The lookup of the client's AS number and country.
+6. The country lists.
+7. The reputation sources.
+8. The client's limit percentage.
+9. The rate limits.
+10. The trap paths.
+11. The rule files.
+12. The Core Rule Set.
+13. `smallwebwaf`'s own endpoints, under `/_smallwebwaf/`.
+14. The request is passed to the app.
+15. After the answer: the byte limits, the error burst and the client's history.
 
-`smallwebwaf` handles each request in this order. A request one step refuses
-skips the steps after it, apart from step 13. Nothing of a refused request
-reaches the app. "What happens to a request" in [`SPEC.md`](SPEC.md) gives each
-step in full.
-
-1. `GET /_smallwebwaf/healthz` is answered `200` with `ok` at once, before any
-   check and without asking the app.
-2. The client is worked out. A netblock is a range of addresses, written as an
-   address and a prefix length: `203.0.113.0/24`. `SWWAF_TRUSTED_PROXIES` lists
-   the netblocks traefik comes from, by default the private address ranges. For
-   a TCP peer inside them, `X-Forwarded-For` is read from the right, and the
-   first address outside those netblocks is the client. A peer outside them is
-   the client itself. An IPv6 client is its group: the whole netblock around its
-   address, a /64 by default; `SWWAF_IPV6_GROUP_PREFIX` sets its length.
-3. A client in `SWWAF_ALLOW_NETS` skips steps 4 to 11, the byte limits and the
-   error burst, but still needs a token at step 12. A client in
-   `SWWAF_DENY_NETS` is refused.
-4. A client under a ban is refused. If the ban was made for a clear sign of
-   attack, this request makes it permanent.
-5. The client's AS number and country are looked up, unless
-   `SWWAF_LOOKUP_SOURCE` is `off`. An AS number names the network an address
-   belongs to, such as `AS64496`.
-6. The country lists are checked. A client from a country in
-   `SWWAF_DENIED_COUNTRIES` is refused. When
-   `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` is set, a client from any other country
-   is refused, and so is a client without a country. Such a refusal is not an
-   offence.
-7. The reputation sources are checked: the blocklists, the CrowdSec decision
-   list, the DNS blocklists and AbuseIPDB. A client on the CrowdSec decision
-   list is banned until the decision ends. Any other source refuses the client,
-   lowers its limits or only notes it, as `SWWAF_BLOCKLIST_ACTION` and
-   `SWWAF_REPUTATION_ACTION` say. No request waits for a query.
-8. Rate limiting counts the request, unless the client or the path is exempt. A
-   request that takes the client over a limit is refused and bans the client.
-9. A request for a path in `SWWAF_TRAP_PATHS` is refused and bans the client, as
-   a clear sign of attack.
-10. The rule files are checked. A `log` rule notes the match. A `block` rule
-    refuses the request. A `ban` rule refuses it and bans the client, as a clear
-    sign of attack.
-11. The Core Rule Set inspects the request's method, URL and headers, and, once
-    `SWWAF_WAF_BODY_LIMIT` is set, form, JSON and XML bodies. A request it takes
-    for an attack is refused while `SWWAF_WAF_MODE` is `block`, the default, and
-    let through while it is `detect`. It bans no one by itself.
-12. A request under `/_smallwebwaf/` is answered here by `smallwebwaf`, after
-    its token is checked. Any other request is passed to the app. The app's
-    answer is streamed back within the size and time limits.
-13. Once the answer has ended, a request passed to the app has its bytes counted
-    for the byte limits. If a rule, a trap path, the Core Rule Set or a token
-    check refused the request, the refusal is counted for the error burst. The
-    client's history and the anomaly counters are updated. More than
-    `SWWAF_ERROR_BURST_THRESHOLD` refusals of one client in a minute ban it.
-
-In `observe` mode (`SWWAF_MODE=observe`), `smallwebwaf` makes no ban of its own.
-It refuses a request only for a size or time limit, or for a missing or wrong
-token. A request it would otherwise have refused is passed to the app, and its
-log line says what `enforce` mode would have done. Observe mode is for trying a
-configuration before enforcing it.
-
-What a refused visitor sees:
-
-- `SWWAF_BAN_RESPONSE`, by default `403` with the plain text `Forbidden`. It can
-  be `429` instead, or `close`, which closes the connection, so that traefik
-  answers `502`. It answers:
-    - a ban or a broken rate limit;
-    - a `ban` rule or a trap path;
-    - `SWWAF_DENY_NETS` and the country lists;
-    - a reputation source whose action is `deny`.
-- `403` for a `block` rule and the Core Rule Set.
-- For a size or time limit:
-    - `431` for headers over their limit;
-    - `413` for a body over its limit;
-    - `408` for a client too slow to send its body;
-    - `502` or `504` when the app passes a limit or cannot be reached.
-- `401` from an endpoint of `smallwebwaf`'s own, for a request without its
-  token.
+A refused visitor is answered with `SWWAF_BAN_RESPONSE`, `403` by default (see
+"Ban lengths, response and scope" in [`SPEC.md`](SPEC.md)). A `block` rule or a
+Core Rule Set match answers `403`. A missing or wrong token gets `401`. A
+request over a size or time limit gets the status for that limit, such as `413`
+for a body too large (see "Size and time limits" in [`SPEC.md`](SPEC.md)).
 
 ## Settings you are likely to touch
 
@@ -233,8 +175,11 @@ Each setting is an environment variable of the app's container, such as
 [`docs/settings.md`](docs/settings.md) lists every setting, its default and the
 values it takes. The ones most deployments touch:
 
-- `SWWAF_MODE=observe`: log what would have been refused, refuse almost nothing,
-  ban no one. For trying a configuration.
+- `SWWAF_MODE=observe`: `smallwebwaf` makes no ban of its own, and refuses a
+  request only for a size or time limit or a missing or wrong token. A request
+  it would have refused is passed to the app, and its log line says what
+  `enforce` mode would have done. For trying a configuration before enforcing
+  it.
 - `SWWAF_ALLOW_NETS`: netblocks you trust, such as your monitoring. Their
   clients are refused only for a size or time limit or a missing or wrong token.
 - `SWWAF_RATE_LIMIT_EXEMPT_NETS`: netblocks exempt from rate limiting and the
@@ -376,50 +321,32 @@ nixpkgs fixed at one release. An app's `Dockerfile` installs a package with
 [`docs/image.md`](docs/image.md) lists what the image holds and how each part is
 pinned.
 
-- The container's first process is `runsvinit`, which starts runit, a service
-  supervisor. runit runs the `run` script of each directory under
-  `/etc/service`, and runs it again a second after its program exits.
-  `smallwebwaf` runs as the `smallwebwaf` user (uid 65532) and the app as a user
-  of its own. So the app cannot touch the state files or the process of
-  `smallwebwaf`.
-- `smallwebwaf` listens on port 8080, on every address. traefik sends the app's
-  requests there. `smallwebwaf` passes them to the app on `127.0.0.1:8081`. The
-  app listens on `127.0.0.1` only, so nothing reaches it except through
-  `smallwebwaf`. The health check, the metrics and the ban management are on the
-  same port, under `/_smallwebwaf/`. A request for that prefix never reaches the
-  app.
-- The app sees what it would see from traefik directly: the same `Host`, the
-  same `X-Forwarded-Proto`, and `X-Forwarded-For` with traefik's address added
-  at the end. The app's TCP peer is `smallwebwaf` on `127.0.0.1`. So the app
-  must trust `127.0.0.1` and `::1` for forwarded headers, besides the private
-  address ranges. An app that keeps its usual trusted proxies sees every visitor
-  as `127.0.0.1`. Each request also carries its id in `X-Request-ID`. Any
-  `X-Client-ASN` or `X-Client-Country` header a client sent is removed.
-- Both processes start with the container's environment variables. That is why
-  every setting of `smallwebwaf` starts with `SWWAF_`. A token given as an
-  environment variable is readable by the app. A token given as a file that only
-  the `smallwebwaf` user can read is not (see "Settings given as files" in
+- runit, a service supervisor, runs `smallwebwaf` and the app side by side, each
+  as its own user, and starts either again when it exits. The app cannot touch
+  the state files or the process of `smallwebwaf`.
+- `smallwebwaf` listens on port 8080 and passes requests to the app on
+  `127.0.0.1:8081`. The app listens on `127.0.0.1` only, so nothing reaches it
+  except through `smallwebwaf`. `smallwebwaf`'s own endpoints are under
+  `/_smallwebwaf/`, on the same port.
+- The app sees the headers traefik sent, with traefik's address added to
+  `X-Forwarded-For`, from a TCP peer on `127.0.0.1`. So the app must trust
+  `127.0.0.1` and `::1` for forwarded headers, besides the private address
+  ranges.
+- Both processes start with the container's environment variables, so every
+  setting of `smallwebwaf` starts with `SWWAF_`, and the app can read a token
+  given as one. A token given as a file that only the `smallwebwaf` user can
+  read is out of its reach (see "Settings given as files" in
   [`docs/settings.md`](docs/settings.md)).
-- The state files live in `/var/lib/smallwebwaf`. Without a volume there, they
-  survive a restart of the container and are lost when a deploy replaces it. The
-  `run` script gives that directory and every file in it to the `smallwebwaf`
-  user at each start, so a host directory mounted there needs no change of
-  owner.
-- The image's health check passes while `smallwebwaf` answers
-  `GET /_smallwebwaf/healthz` and the app accepts connections. traefik sends a
-  container no requests until it is healthy.
-- When `smallwebwaf` or the app exits, runit starts it again a second later. The
-  other process keeps running. While either is down, the client gets `502`. A
-  `smallwebwaf` that stops at start, as on an invalid setting, is started again
-  every second until the cause is fixed, and logs its message each time.
-- `docker stop` has runit send `SIGTERM` to both processes at once.
-  `smallwebwaf` gives the requests in progress five seconds to finish and writes
-  its state files. To let them finish, the app must keep answering for five
+- The state files live in `/var/lib/smallwebwaf`. A volume mounted there keeps
+  them across a deploy.
+- `docker stop` stops both processes at once. `smallwebwaf` gives the requests
+  in progress five seconds to finish, so the app must keep answering for five
   seconds after `SIGTERM`.
 
-"Deployment" in [`SPEC.md`](SPEC.md) has the rest: moving `smallwebwaf` to
-another port, the health check's timing, and the forwarded headers.
-"Architecture" in [`SPEC.md`](SPEC.md) lists the packages and what each does.
+"Deployment" in [`SPEC.md`](SPEC.md) has the rest: the health check, what
+happens when a process stops, moving `smallwebwaf` to another port, and the
+forwarded headers. "Architecture" in [`SPEC.md`](SPEC.md) lists the packages and
+what each does.
 
 ## TODO
 

docs/alerts.md

@@ -79,10 +79,11 @@ on one line:
 - `instance`: `SWWAF_INSTANCE_NAME`. `time`: when the alert was raised. `event`:
   one of the events above.
 - `client`: the address whose request raised the alert. Empty for
-  `source_failure` and `file_error`.
-- `netblock`: empty for `source_failure` and `file_error`; otherwise one of:
+  `source_failure`, `file_error` and `summary`.
+- `netblock`: empty for `source_failure`, `file_error` and `summary`; otherwise
+  one of:
     - the ban's netblock;
-    - the netblock an anomaly counted;
+    - the netblock an anomaly counted, empty in the `asn` and `total` scopes;
     - for the other events, the client as rate limiting counts it: an IPv4
       address as a /32, or an IPv6 group.
 - `asn`, `as_name` and `country`: the client's, as known when the alert was
@@ -97,6 +98,8 @@ on one line:
     - for a Core Rule Set match, the `rule_ids`, `score`, `method` and `path`;
     - for a source failure, the `source` and `error`;
     - for a file error, the `file` and `error`;
+    - for a summary, the `hour` it covers, the `count` of alerts held back, and
+      that count by event in `events`;
     - `mode`, when the request was not refused: `detect` or `observe`.
 - `suppressed_repeats`: how many repeats the cooldown held back before this
   alert.

docs/metrics.md

@@ -1,8 +1,8 @@
 # The metrics
 
-The metrics `GET /_smallwebwaf/metrics` answers with, as "Metrics endpoint" in
-[`SPEC.md`](../SPEC.md) describes it. Prometheus keeps the `instance` label as
-`exported_instance` unless the scrape sets `honor_labels: true`.
+What `GET /_smallwebwaf/metrics` answers with. "Metrics endpoint" in
+[`SPEC.md`](../SPEC.md) describes the endpoint. Prometheus keeps the `instance`
+label as `exported_instance` unless the scrape sets `honor_labels: true`.
 
 ## Traffic
 

docs/settings.md

@@ -91,11 +91,11 @@ that take `off`, the `_FILE` form, and what each group does and why.
 ## Static lists
 
 - `SWWAF_ALLOW_NETS` (default empty): netblocks you trust, such as your
-  monitoring. Their clients are refused only for a size or time limit or a
-  missing or wrong token.
+  monitoring (see step 3 in [`SPEC.md`](../SPEC.md)).
 - `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients are
   exempt from rate limiting and the byte limits.
-- `SWWAF_DENY_NETS` (default empty): netblocks whose clients are always refused.
+- `SWWAF_DENY_NETS` (default empty): netblocks whose clients are refused (see
+  step 3 in [`SPEC.md`](../SPEC.md)).
 
 ## Request rate limits, per client
 
@@ -136,7 +136,13 @@ that take `off`, the `_FILE` form, and what each group does and why.
 
 ## Lookup of AS number and country
 
-- `SWWAF_LOOKUP_SOURCE` (default `geojs`): `geojs`, `file` or `off`.
+- `SWWAF_LOOKUP_SOURCE` (default `geojs`): `geojs`, `file` or `off`. `off` stops
+  the start while a setting that needs a lookup is set:
+    - the two country lists;
+    - `SWWAF_ADD_LOOKUP_HEADERS`;
+    - the four biased threshold lists and `SWWAF_ASN_LIMIT_PERCENT_URL`;
+    - `SWWAF_UNKNOWN_LIMIT_PERCENT` below `100`;
+    - an anomaly threshold per AS number.
 - `SWWAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in its
   `.mmdb` form, for `SWWAF_LOOKUP_SOURCE=file`.
 - `SWWAF_LOOKUP_TIMEOUT` (default `1s`): how long a request waits for its
@@ -190,8 +196,7 @@ that take `off`, the `_FILE` form, and what each group does and why.
 - `SWWAF_WAF_BODY_LIMIT` (default `off`): `off`, or how much of a request body
   the Core Rule Set reads, a size of at most `1G`.
 - `SWWAF_ERROR_BURST_THRESHOLD` (default `30`): the most refusals of one client
-  in a minute for a rule match, a trap path, a Core Rule Set match or a failed
-  token check. A count or `off`.
+  in a minute (see "Attack detection" in `SPEC.md`). A count or `off`.
 
 ## Ban lengths, response and scope
 
@@ -286,7 +291,7 @@ as `REQUESTS_PER_MINUTE`:
       client;
     - `SWWAF_ANOMALY_NET_V6_PREFIX` (default `48`, from 0 to 128) for an IPv6
       client.
-- `SWWAF_ANOMALY_ASN_...`: per AS number. Needs a lookup source.
+- `SWWAF_ANOMALY_ASN_...`: per AS number.
 - `SWWAF_ANOMALY_TOTAL_...`: the whole service.
 - `SWWAF_WATCH_...`: each netblock in `SWWAF_WATCH_NETS`, as a whole.
   `SWWAF_WATCH_NETS` (default empty) is a list of name, `=` and netblock, such

internal/alerts/alerts.go

@@ -1,8 +1,8 @@
 // Package alerts sends alerts on bans, on traffic over an anomaly
 // threshold, on a source that fails and on a file with an error to each
 // destination set: to the webhook
-// SWWAF_ALERT_WEBHOOK_URL names, each as one JSON object, as the "Alert
-// webhook schema" section of SPEC.md describes, to the Slack incoming
+// SWWAF_ALERT_WEBHOOK_URL names, each as one JSON object, as "The JSON
+// object" in docs/alerts.md describes, to the Slack incoming
 // webhook SWWAF_ALERT_SLACK_WEBHOOK_URL names, as a message, and to the
 // ntfy topic SWWAF_ALERT_NTFY_URL names. A repeat within
 // SWWAF_ALERT_COOLDOWN is held back, and so is an alert past
@@ -149,11 +149,10 @@ type Params struct {
 }
 
 // Alert is one alert, as the webhook is sent it and alerts.json holds it,
-// with the fields of the "Alert webhook schema" section of SPEC.md. ASN,
-// ASName and Country are, for a ban, the client's as the ban's notes give
-// them.
+// with the fields of "The JSON object" in docs/alerts.md. ASN, ASName and
+// Country are, for a ban, the client's as the ban's notes give them.
 //
-//nolint:tagliatelle // SPEC.md's alert webhook schema names its fields in snake_case
+//nolint:tagliatelle // docs/alerts.md names the alert's fields in snake_case
 type Alert struct {
 	Instance string       `json:"instance"`
 	Time     time.Time    `json:"time"`

internal/anomaly/anomaly.go

@@ -1,8 +1,8 @@
 // Package anomaly counts requests and bytes over a minute and an hour, per
 // client, per surrounding netblock, per AS number, for the whole service
 // and per named netblock, and raises an anomaly alert for a count over its
-// threshold, as "Anomaly thresholds" under "Configuration surface" in
-// SPEC.md describes. It refuses and bans nothing. At most 20,000 counters
+// threshold, as "Anomaly thresholds" under "Settings" in SPEC.md
+// describes. It refuses and bans nothing. At most 20,000 counters
 // are kept, in memory, and written to alerts.json and read from it by the
 // state package.
 package anomaly

internal/metrics/metrics.go

@@ -1,6 +1,7 @@
-// Package metrics keeps smallwebwaf's Prometheus metrics, as the "Metrics
-// endpoint" section of SPEC.md lists them, and serves them in the
-// Prometheus text format. No metric carries a client's address.
+// Package metrics keeps smallwebwaf's Prometheus metrics, as
+// docs/metrics.md lists them, and serves them in the Prometheus text
+// format, as the "Metrics endpoint" section of SPEC.md requires. No
+// metric carries a client's address.
 package metrics
 
 import (

internal/ratelimit/ratelimit.go

@@ -206,7 +206,7 @@ type Hit struct {
 // Counts are a client's requests and bytes in the minute, the hour and
 // the day that end at a request, that request's included.
 //
-//nolint:tagliatelle // SPEC.md's request log names its fields in snake_case
+//nolint:tagliatelle // docs/request-log.md names the fields in snake_case
 type Counts struct {
 	Minute      float64 `json:"minute"`
 	Hour        float64 `json:"hour"`

internal/requestlog/requestlog.go

@@ -64,11 +64,10 @@ const LimitHitErrorBurst = "error_burst"
 const timeLayout = "2006-01-02T15:04:05.000Z07:00"
 
 // Line is one request's line in the request log. The field names, and
-// their order, are those of the "Request log" section of SPEC.md. A field
-// that may not apply to a request is left out of its line when it does
-// not.
+// their order, are those of docs/request-log.md. A field that may not
+// apply to a request is left out of its line when it does not.
 //
-//nolint:tagliatelle // SPEC.md's request log names its fields in snake_case
+//nolint:tagliatelle // docs/request-log.md names the fields in snake_case
 type Line struct {
 	Type string `json:"type"`
 

internal/state/state.go

@@ -1,5 +1,6 @@
 // Package state keeps smallwebwaf's state in JSON files in
-// SWWAF_STATE_DIR, as the "Persistent state" section of SPEC.md describes:
+// SWWAF_STATE_DIR, as the "Persistent state" section of SPEC.md requires
+// and docs/state-files.md lists them:
 // bans.json holds the bans, clients.json each client's counters and
 // history, lookups.json GeoJS's answers, reputation.json the last try and
 // last good copy of each list fetched from a URL, the DNSBL zones'

internal/state/state_test.go

@@ -1019,8 +1019,8 @@ func TestEditJustBeforeAScheduledWriteSurvivesIt(t *testing.T) {
 		edit(t, dir, bansJSON, permanentBansJSON)
 
 		// The write takes the edit in first, and writes it back. The second
-		// ban, made after the admin opened the file, is lost, as "Edits
-		// while running" in SPEC.md says.
+		// ban, made after the admin opened the file, is lost, as the
+		// "Persistent state" section of SPEC.md says.
 		time.Sleep(time.Nanosecond)
 		synctest.Wait()
 

internal/waf/waf.go

@@ -1,8 +1,8 @@
 // Package waf runs the OWASP Core Rule Set 4.25.0, through Coraza, on the
 // method, the URL with its query and the headers of a request, and on its
 // body while SWWAF_WAF_BODY_LIMIT is set, with the six changes smallwebwaf
-// makes to it, as "Attack detection" under "Configuration surface" in
-// SPEC.md describes them. It reads no response.
+// makes to it, as "Attack detection" under "Settings" in SPEC.md describes
+// them. It reads no response.
 //
 // smallwebwaf writes only to its state directory, so Coraza is built with
 // its no_fs_access tag, as the Dockerfile and script/build build it: of a

script/gitea-requests

@@ -2,11 +2,12 @@
 # script/gitea-requests: build the image and run gitea 1.27.3 behind it,
 # with every setting of smallwebwaf at its default but SWWAF_UPSTREAM_URL,
 # which names gitea's container. Then send through it the gitea requests
-# that reading gitea's source found the Core Rule Set refuses, and a short
-# sample of gitea's ordinary use: signing in, pages of files and commits,
-# a diff, the API, a push, clone and fetch over HTTP, and two container
-# image pushes. Last, print each request smallwebwaf logged, with what it
-# did with it and the ids of the Core Rule Set's rules that matched it.
+# that reading gitea's source found the Core Rule Set refuses, which
+# docs/gitea.md records, and a short sample of gitea's ordinary use:
+# signing in, pages of files and commits, a diff, the API, a push, clone
+# and fetch over HTTP, and two container image pushes. Last, print each
+# request smallwebwaf logged, with what it did with it and the ids of the
+# Core Rule Set's rules that matched it.
 # Every request comes from one client, docker's gateway, so more than
 # SWWAF_ERROR_BURST_THRESHOLD, 30, refusals in a minute would ban it, and
 # the requests after would show as banned. The containers, the network,