Leave SWWAF_RATE_LIMIT_EXEMPT_PATHS out of the request rate limits (closes #77)

A request is neither counted nor refused by the request rate limits
when its path as sent, the path the app receives, not percent-decoded,
starts with one of the comma-separated prefixes in
SWWAF_RATE_LIMIT_EXEMPT_PATHS, so /%61ssets/x is not under /assets/. A
request whose decoded path contains .. or a backslash, or whose path as
sent holds an encoded slash, is never exempt, since an app may act on
it as a path outside every prefix, such as /assets/..%2Flogin as
/login. The static lists, bans and the country lists still apply, and
its log line has no counts. The setting is empty by default, and a
prefix that does not start with / stops the start. README.md documents
it.

Model: opus-5-5

README.md

@@ -13,19 +13,25 @@ JSON log line for every request.
 
 Status: the first two milestones are built
 (https://git.eeqj.de/sneak/smallwebwaf/issues/13 and
-https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are four parts of
-milestone 3: the static lists, the bans that broken rate limits lead to and the
-JSON state files, which come next in the build order, and the header size and
-the idle time as settings, which come last in it. `smallwebwaf` passes each
-request to the app and the app's answer back, unchanged, within its timeouts and
-size limits, works out each client's address, bans a client that sends too many
-requests, refuses a client that comes from a country you refuse or from a
-network you refuse, lets the networks you choose through, keeps its bans, each
-client's counters and history, and GeoJS's answers in JSON files across
-restarts, and writes a JSON log line for every request. It comes as the image
-the app's own image is built on. The rest of the design comes after that, in the
-order of the build order in [`SPEC.md`](SPEC.md). The survey of existing tools
-that led to the design is in [`EVALUATION.md`](EVALUATION.md).
+https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are eight parts of
+milestone 3: the static lists, the bans that broken rate limits lead to, the
+JSON state files with your edits taken in while it runs and the paths the rate
+limits do not count, which come next in the build order, `observe` mode and the
+rest of the request log's fields, which come a little later, and the metrics
+endpoint and the header size and the idle time as settings, which come last in
+it. `smallwebwaf` passes each request to the app and the app's answer back,
+unchanged, within its timeouts and size limits, works out each client's address,
+bans a client that sends too many requests, not counting those for the paths you
+choose, refuses a client that comes from a country you refuse or from a network
+you refuse, lets the networks you choose through, keeps its bans, each client's
+counters and history, and GeoJS's answers in JSON files across restarts, takes
+in your edits of those files while it runs, writes a JSON log line for every
+request, serves Prometheus metrics to a scraper that holds the metrics token,
+and in `observe` mode passes on the requests it would refuse, logging what it
+would have done with them. It comes as the image the app's own image is built
+on. The rest of the design comes after that, in the order of the build order in
+[`SPEC.md`](SPEC.md). The survey of existing tools that led to the design is in
+[`EVALUATION.md`](EVALUATION.md).
 
 ## Getting started
 
@@ -62,7 +68,9 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set.
   address outside `SWWAF_TRUSTED_PROXIES` is the client; if every address in it
   is inside, the leftmost is, and with no header the peer is. The app sees what
   it would see from traefik directly: the same `Host`, the same
-  `X-Forwarded-Proto`, and `X-Forwarded-For` with the peer added at the end.
+  `X-Forwarded-Proto`, and `X-Forwarded-For` with the peer added at the end. It
+  also gets the request's id in `X-Request-ID`, the same id as in the request's
+  log line (see `request_id` in "Request log" below).
 - Enforces the timeouts and the size limits below. A limit passed before the
   response has started gets `smallwebwaf`'s own answer: `408` for a client too
   slow to send its request, `413` for a request body that is too large, `504`
@@ -76,12 +84,15 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set.
 - Counts each client's requests over a minute, an hour and a day. A request that
   takes the client over one of the rate limits below is refused with
   `SWWAF_BAN_RESPONSE`, `403` by default, before anything reaches the app, and
-  bans the client. A client is one IPv4 address, or one IPv6 /64, since one
-  abuser usually holds a whole /64. Each window is counted in two fixed buckets,
-  the earlier one weighted by how much of it the window still covers. At most
-  20,000 clients are kept, the least recently seen dropped first, with their
-  history, and a restart gives no client a fresh allowance (see "State files"
-  below).
+  bans the client. A request whose path starts with one of
+  `SWWAF_RATE_LIMIT_EXEMPT_PATHS`, as that setting below describes, is neither
+  counted nor refused by the rate limits; the static lists, bans and the country
+  lists still apply to it. A client is one IPv4 address, or one IPv6 /64, since
+  one abuser usually holds a whole /64. Each window is counted in two fixed
+  buckets, the earlier one weighted by how much of it the window still covers.
+  At most 20,000 clients are kept, the least recently seen dropped first, with
+  their history, and a restart gives no client a fresh allowance (see "State
+  files" below).
 - Bans a client that breaks a rate limit, as "Bans" in [`SPEC.md`](SPEC.md)
   describes: the first ban lasts an hour, and a limit broken again within a day
   of a ban ending bans for three times as long as that ban, so 1, 3, 9, 27 and
@@ -97,9 +108,9 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set.
   seen, how many of them the ban has refused, and how many bans the netblock had
   before. At most `SWWAF_MAX_BANS` bans are kept, past, active and permanent;
   past that, the earliest ban of the netblock that has gone longest without a
-  request is dropped first. `bans.json` shows the bans and their notes, and a
-  restart lifts none (see "State files" below); lifting a ban by editing it
-  comes with https://git.eeqj.de/sneak/smallwebwaf/issues/68.
+  request is dropped first. `bans.json` shows the bans and their notes, a
+  restart lifts none, and you add or lift a ban by editing it (see "State files"
+  below).
 - Refuses a request from a country you refuse with `SWWAF_BAN_RESPONSE`, as soon
   as the client's country is known and before its body is read; such a request
   is not counted for the rate limits. While one of the country lists below is
@@ -117,8 +128,28 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set.
   `SWWAF_ALLOW_NETS` too is let through. A client in
   `SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither counted nor refused by the rate
   limits; the country lists and bans still apply to it.
+- In `observe` mode, with `SWWAF_MODE=observe`, refuses none of the requests
+  that `SWWAF_DENY_NETS`, a ban, the country lists or a rate limit would refuse:
+  it passes them to the app, and their log lines name what `enforce` mode would
+  have done (see `would_action` in "Request log" below). The checks run, and
+  requests are counted, as in `enforce` mode, but a broken rate limit makes no
+  ban and does not set the client's counters back to zero, so each request over
+  the limit is logged as one that would be refused. The bans in `bans.json` are
+  kept, and refuse requests again when `smallwebwaf` next runs in `enforce`
+  mode, as long as they last. The timeouts and size limits still apply, since
+  they protect `smallwebwaf` and the app themselves, and a request for the
+  metrics without the token is still answered `401`. It is for trying a
+  configuration before enforcing it.
 - Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any
   check and without asking the app, for the image's health check.
+- Answers `GET /_smallwebwaf/metrics` with its metrics (see "Metrics" below) for
+  a request that carries `SWWAF_METRICS_TOKEN` as
+  `Authorization: Bearer <token>`, and with `401` for one that does not. While
+  the token is unset the metrics answer `404`, as does any other request under
+  `/_smallwebwaf/`. Unlike the health check, such a request goes through every
+  check any other request goes through, and is answered where another would be
+  passed to the app: a banned client stays refused, and each counts toward the
+  client's rate limits. None of them reaches the app.
 - Writes a line in the request log for each request (see "Request log" below).
 
 ## Settings
@@ -130,6 +161,14 @@ it, and the effective settings are logged at start.
 - `SWWAF_LISTEN_ADDR` (default `:8080`): where `smallwebwaf` listens.
 - `SWWAF_UPSTREAM_URL` (default `http://127.0.0.1:8081`): the app, as `http` or
   `https`, a host and an optional port, and nothing more.
+- `SWWAF_INSTANCE_NAME` (default: the host's name, which docker sets to the
+  first 12 characters of the container's id unless the deployment names one):
+  the name each request log line gives as `instance`. Set it, for example to
+  `fsn1app1/gitea`, for a name that stays the same when a deploy replaces the
+  container, and that tells instances apart when several log to one place.
+- `SWWAF_MODE` (default `enforce`): `enforce`, or `observe` to pass on the
+  requests `smallwebwaf` would refuse and log what it would have done (see "What
+  it does so far" above).
 - `SWWAF_TRUSTED_PROXIES` (default `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`,
   the private address ranges): the netblocks whose `X-Forwarded-For` is
   believed. A list given replaces the default; set but empty, it trusts nothing.
@@ -165,6 +204,20 @@ it, and the effective settings are logged at start.
   requests a client may make in a minute, an hour and a day. The defaults are
   several times what one busy person produces, since a browser loading a heavy
   page makes a few hundred requests and several people often share one address.
+- `SWWAF_RATE_LIMIT_EXEMPT_PATHS` (default empty): path prefixes whose requests
+  the rate limits neither count nor refuse, such as `/assets/` for static
+  assets; each starts with `/`. A request whose path, percent-decoded, contains
+  `..` anywhere or a backslash, or whose path as sent holds an encoded slash
+  (`%2F` or `%2f`), is never exempt, since the app may act on it as a path
+  outside every prefix: `/assets/..%2Flogin` as `/login`. Any other request is
+  exempt when its path as sent, the path the app receives, before any query
+  string and not percent-decoded, starts with a prefix, character for character.
+  `/assets/` matches `/assets/app.js` and `/assets/`, but not `/assets`,
+  `/Assets/app.js`, `/%61ssets/app.js`, `/static/assets/app.js`,
+  `/static/../assets/app.js` or `/assets%2Fapp.js`. A character the client sends
+  percent-encoded, such as a space, is written percent-encoded in a prefix, as
+  in `/my%20files/`, and there are no wildcards: `*` is a character like any
+  other.
 - `SWWAF_DENIED_COUNTRIES` (default empty): countries whose clients are refused,
   for example `cn,ru,kp`.
 - `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` (default empty): when set, the only
@@ -194,6 +247,19 @@ it, and the effective settings are logged at start.
   `bans.json` is written, with every ban made in between.
 - `SWWAF_STATE_COUNTER_INTERVAL` (default `15m`): how often every state file is
   written.
+- `SWWAF_LOG_REQUEST_HEADERS` (default
+  `accept,accept-language,accept-encoding,content-type,origin,range`): the
+  request headers whose values the request log gives, in either case.
+  `Authorization`, `Cookie` and `Set-Cookie` are never logged, even when listed
+  (see "Request log" below). An entry naming `Host` or `Transfer-Encoding` stops
+  the start, since Go's HTTP server takes both out of the request; the request's
+  host is the field `host`.
+- `SWWAF_METRICS_TOKEN` (default unset): the token a scraper sends for the
+  metrics, a long random value. While it is unset the metrics are off; one
+  shorter than 32 characters stops the start. The settings logged at start show
+  `********` in its place.
+- `SWWAF_METRICS_TOP_N` (default `50`): how many countries get series of their
+  own in the metrics by country; the others are counted as `other`.
 
 Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are
 bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024
@@ -203,8 +269,8 @@ ISO 3166-1 assigns today, and `xk` for Kosovo, in either case (`de` and `DE` are
 the same); any other code, such as `nk` (North Korea is `kp`) or the withdrawn
 `su`, stops the start, and so does a code on both country lists. `off` switches
 a timeout, a size limit or a rate limit off;
-`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, the ban settings and the state settings
-cannot be off.
+`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, the ban settings, the state settings
+and `SWWAF_METRICS_TOP_N` cannot be off.
 
 Several limits are fixed rather than settings. At most 20,000 clients are kept,
 with their counters and history, and an IPv6 client is counted by its /64. A new
@@ -217,18 +283,42 @@ GeoJS are kept, for 7 days each.
 refused ones included:
 
 ```
-{"type":"request","time":"2026-10-03T12:00:00.123Z","client_ip":"203.0.113.9","peer_ip":"172.18.0.2","country":"DE","method":"GET","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"upstream_status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","action":"forward","duration_total":3.217,"duration_upstream_total":3.104}
+{"type":"request","time":"2026-10-03T12:00:00.123Z","instance":"fsn1app1/gitea","client_ip":"203.0.113.9","method":"GET","scheme":"https","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","peer_ip":"172.18.0.2","forwarded_for":"203.0.113.9","client_group":"203.0.113.9/32","country":"DE","request_headers":{"accept":"*/*"},"response_content_type":"text/html; charset=utf-8","upstream_status":200,"action":"forward","counts":{"minute":1,"hour":12,"day":40},"duration_total":3.217,"duration_checks":0.041,"duration_upstream_connect":0.052,"duration_upstream_first_byte":2.874,"duration_upstream_total":3.104}
 ```
 
-- `time` is when the request arrived, in UTC. `peer_ip` is the TCP peer,
-  normally traefik. `path` and `query` are as the client sent them.
+A field that does not apply to a request is left out of its line, apart from
+`type`, the fields from `time` to `user_agent`, `request_id`, `peer_ip`,
+`client_group`, `country`, `action` and `duration_total`, which every line has.
+
+- `time` is when the request arrived, in UTC. `instance` is
+  `SWWAF_INSTANCE_NAME`. `scheme` is the `X-Forwarded-Proto` a trusted proxy
+  sent, and otherwise `http`. `path` and `query` are as the client sent them.
+- `request_id` is the `X-Request-ID` a trusted proxy sent, or a new random one
+  of 26 letters and digits when it sent none, or when the peer is not a trusted
+  proxy. A request passed to the app takes it there in `X-Request-ID`.
+- `peer_ip` is the TCP peer, normally traefik. `forwarded_for` is the
+  `X-Forwarded-For` header as received, several lines of it joined with `, `.
+  `client_group` is the client as the rate limits count it: its IPv4 address as
+  a /32, or the /64 of its IPv6 address.
 - `country` is the client's country as GeoJS places it. It is empty with neither
   country list set, for a client in `SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS`, for
   a client on a private, loopback or link-local address, when GeoJS cannot place
-  the client or has not answered in time, and for a request refused because a
-  ban covers its client, even when the client's country is known.
+  the client or has not answered in time, and for a request whose client a ban
+  covers, even when the client's country is known.
+- `content_type` is the request's `Content-Type`, and `content_length` the
+  length the request announced for its body, which is left out for none or zero.
+- `request_headers` are the request's headers that `SWWAF_LOG_REQUEST_HEADERS`
+  names, by name in lower case, several lines of one joined with `, `.
+  `Authorization`, `Cookie` and `Set-Cookie` are never among them, whatever the
+  setting says: `has_authorization` and `has_cookie` are there instead, and
+  true, when the request has an `Authorization` or a `Cookie` header.
+- `websocket` is there, and true, when the app switched the connection to
+  another protocol, as it does for a WebSocket.
 - `status` is what the client was sent, `0` if nothing was; `upstream_status` is
   what the app answered, and is left out when the app did not answer.
+- `response_content_type`, `cache_control` and `location` are the
+  `Content-Type`, `Cache-Control` and `Location` headers of the answer: the
+  app's, as passed on, or those of `smallwebwaf`'s own answer.
 - `request_bytes` and `response_bytes` count body bytes.
 - `action` is `forward` for a request passed to the app, `denied` for one
   refused because its client is in `SWWAF_DENY_NETS`, `banned` for one refused
@@ -238,16 +328,42 @@ refused ones included:
   `timed_out` for one that ran out of time, `upstream_error` when the app could
   not be reached or its answer broke off, and `admin` for one `smallwebwaf`
   answered at its own endpoint.
+- `would_action` is there in `observe` mode for a request that
+  `SWWAF_DENY_NETS`, a ban, the country lists or a rate limit would have refused
+  in `enforce` mode, and names the action that refusal would have had: `denied`,
+  `banned`, `country_denied` or `rate_limited`. `action` then names what was
+  done: `forward` for a request passed to the app, and another action, such as
+  `too_large`, for one a size or time limit refused.
+- `counts` gives the client's requests in the minute, the hour and the day as
+  the rate limits count them, this request included: in each window, those in
+  the bucket under way and a share of those in the bucket before, so a count can
+  have a fraction. For a request that broke a limit, they are the counts that
+  broke it. It is left out for a request the rate limits do not count: the
+  health check, one from a client in `SWWAF_ALLOW_NETS` or
+  `SWWAF_RATE_LIMIT_EXEMPT_NETS`, one for a path that
+  `SWWAF_RATE_LIMIT_EXEMPT_PATHS` exempts, and one that `SWWAF_DENY_NETS`, a ban
+  or the country lists refuse, or would refuse in `observe` mode. The byte
+  totals come with the byte limits.
 - `limit_hit` is there for a request that broke a rate limit, and names the
   window whose limit it went over: `minute`, `hour` or `day`, the shortest if it
   went over several. `offence` is then `limit`.
 - `ban_expires` is there for a request that made a ban or was refused under one,
-  and gives when the ban ends, in the same form as `time`, or `permanent`.
+  or in `observe` mode would have been refused under one, and gives when the ban
+  ends, in the same form as `time`, or `permanent`.
 - `aborted` is there, and true, when the client went away early.
-- `duration_total` and `duration_upstream_total` are in milliseconds.
+- The timings are in milliseconds, to the microsecond. `duration_total` runs
+  from when the request's headers had been read to when its line is written, and
+  `duration_checks` over the same start to when the checks were done; the health
+  check runs none, and its line has no `duration_checks`.
+  `duration_upstream_connect`, `duration_upstream_first_byte` and
+  `duration_upstream_total` are there for a request passed to the app, and run
+  from when it was handed to the app: until there was a connection to it, new or
+  kept open from an earlier request, until the first byte of its answer arrived,
+  and until the end. The first two are left out when that never happened, as for
+  an app that cannot be reached.
 
-No body and no other header is logged. `smallwebwaf`'s own messages (start, the
-settings, stop, errors) share the stream as JSON lines marked
+No body is logged, and no header but those above. `smallwebwaf`'s own messages
+(start, the settings, stop, errors) share the stream as JSON lines marked
 `"type":"process"`.
 
 Go's HTTP server, on which `smallwebwaf` is built, reads a request's line and
@@ -268,10 +384,11 @@ entries by client address, with times in UTC.
   `expires` is `null`.
 - `clients.json`: each client's two buckets in the minute, the hour and the day,
   and its history: when it was first and last seen, its country as last looked
-  up and when, its requests, how many were forwarded and how many refused, the
-  body bytes in each direction, its responses by status class and its offences
-  by kind. Each client is on a line of its own, so `grep` shows everything about
-  one.
+  up and when, its requests, how many were forwarded and how many refused (one
+  `smallwebwaf` answered at its own endpoints is neither, unless it was refused
+  with `401` for a missing or wrong token), the body bytes in each direction,
+  its responses by status class and its offences by kind. Each client is on a
+  line of its own, so `grep` shows everything about one.
 - `lookups.json`: GeoJS's answers, one to a line, with when GeoJS gave each and
   when it was last used.
 
@@ -283,14 +400,101 @@ new one, whole. A write that fails is logged, and tried again at the next write.
 A hard kill loses what changed since the last write.
 
 At start the files are read back: each client keeps its counts, so a restart
-gives it no fresh allowance, and each ban keeps refusing until it ends. Buckets
-and answers whose time has passed are dropped. A missing file is empty state, as
-on a first start. A file that does not parse, or has another `version`, stops
-the start with a message naming the file, and the line and column where Go's
-JSON decoder gives them; so does a state directory `smallwebwaf` cannot write.
-An edit made while `smallwebwaf` runs is overwritten by its next write: taking
-it in comes with https://git.eeqj.de/sneak/smallwebwaf/issues/68. The AS number
-and AS name come with their lookup.
+gives it no fresh allowance, and each ban keeps refusing every client in its
+netblock until it ends, even after `SWWAF_BAN_SCOPE_V4_PREFIX` has changed. A
+netblock whose address has bits past its length, such as `203.0.113.9/24`, is
+read as the netblock it is in, `203.0.113.0/24`. Buckets and answers whose time
+has passed are dropped. A missing file is empty state, as on a first start. A
+file that does not parse, or has another `version`, stops the start with a
+message naming the file, and the line and column where Go's JSON decoder gives
+them; so does a state directory `smallwebwaf` cannot write. So does an entry
+without a field it needs, named with the entry's place in the file: a ban's
+`netblock`, `start` or `expires`, which is `null` for a permanent ban; a
+client's `client`, or the `start` of a window in which it has requests; an
+answer's `client`, `country`, which is `""` for a client GeoJS cannot place, or
+`answered`. The AS number and AS name come with their lookup.
+
+While it runs, `smallwebwaf` watches `SWWAF_STATE_DIR` and takes in your edit of
+a state file as soon as you save it: what the file then holds replaces what
+`smallwebwaf` held for it, as if read at start. It tells its own writes from
+yours by comparing the file with what it last read or wrote, and before it
+writes a file it takes in any edit made since, so your edit is not overwritten;
+a change `smallwebwaf` made after you opened the file, such as a new ban, is
+lost when you save over it. An edit that would stop the start, because it does
+not parse, has another `version` or leaves out a field an entry needs, does not
+stop the running `smallwebwaf`: it keeps what it holds, and at the file's next
+write renames your file to `<name>.bad`, such as `bans.json.bad`, writes the
+file again from memory, and logs the file and where the error is. It waits for
+that write because an editor's file can be read before the editor has finished
+writing it. Mend the `.bad` file and move it back. A file you remove is written
+again at its next write.
+
+To ban a netblock, add an entry to `bans.json` with its `netblock`, its `start`
+and its `expires`, `null` for a ban that never ends; its `notes` may be left
+out. This `bans.json` bans `203.0.113.0/24` for good:
+
+```json
+{
+    "version": 1,
+    "bans": [
+        {
+            "netblock": "203.0.113.0/24",
+            "start": "2026-10-06T12:00:00Z",
+            "expires": null
+        }
+    ]
+}
+```
+
+To lift a ban, delete its entry. `smallwebwaf` then forgets the ban, so it does
+not make the netblock's next ban longer.
+
+## Metrics
+
+`GET /_smallwebwaf/metrics` answers with the metrics in the Prometheus text
+format, for a scraper that sends `SWWAF_METRICS_TOKEN`, through traefik like any
+other request. No metric carries a client's address.
+
+- `smallwebwaf_requests_total`, `smallwebwaf_request_bytes_total` and
+  `smallwebwaf_response_bytes_total`: requests, and their body bytes each way,
+  by `status_class`, such as `2xx`, or `none` when nothing was sent, and by
+  `action`, as the request log names it.
+- `smallwebwaf_request_duration_seconds`: how long requests took, and
+  `smallwebwaf_upstream_duration_seconds`: how long those passed to the app took
+  from then on, as histograms; `smallwebwaf_requests_in_flight`: the requests
+  under way.
+- `smallwebwaf_rate_limit_hits_total` by `window`,
+  `smallwebwaf_size_and_time_limit_hits_total` by `limit`, the setting whose
+  limit was passed, `smallwebwaf_offences_total` by `kind`, and
+  `smallwebwaf_bans_made_total` by `cause`; `smallwebwaf_active_bans` and
+  `smallwebwaf_permanent_bans`.
+- `smallwebwaf_country_requests_total`,
+  `smallwebwaf_country_request_bytes_total`,
+  `smallwebwaf_country_response_bytes_total`, and
+  `smallwebwaf_country_list_refusals_total`, the requests the country lists
+  refused, by `country`, for the requests whose client's country is known. The
+  `SWWAF_METRICS_TOP_N` countries with the most requests since the start have
+  series of their own, and the others are counted as `other`. A country that
+  drops out of them loses its series, and its later requests count as `other`;
+  one that comes into them gets a series that counts from then on.
+- `smallwebwaf_geojs_requests_total`: the requests to GeoJS;
+  `smallwebwaf_geojs_failures_total`: those that failed, an answer that leaves
+  out an address asked about included; and `smallwebwaf_geojs_unanswered_total`:
+  the requests whose client counted as coming from an unknown country because
+  GeoJS had not answered about it in time.
+- `smallwebwaf_tracked_clients`: the clients in the table of clients.
+- `smallwebwaf_state_file_writes_total`,
+  `smallwebwaf_state_file_write_failures_total`,
+  `smallwebwaf_state_file_last_write_timestamp_seconds` and
+  `smallwebwaf_state_file_size_bytes`, by `file`; and, by `file` too,
+  `smallwebwaf_state_file_edits_taken_in_total`: your edits taken in, and
+  `smallwebwaf_state_file_edits_set_aside_total`: those renamed to `<name>.bad`
+  because they would stop the start.
+- Go's own `go_` metrics and the process's `process_` metrics.
+
+The requests Go's HTTP server ends before `smallwebwaf` sees them (see "Request
+log") are not counted. The metrics of the features still to come, such as the
+rule files, come with them.
 
 ## Why
 
@@ -393,9 +597,9 @@ goes through the candidates one by one.
   readable JSON files, written regularly and at every stop, so a restart loses
   nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks
   up the change. Nothing is read from disk while serving a request. The files
-  for the bans, the clients and the GeoJS answers are built (see "State files"
-  above); the others come with their features, and taking in an edit while
-  running comes with https://git.eeqj.de/sneak/smallwebwaf/issues/68.
+  for the bans, the clients and the GeoJS answers are built, with an edit taken
+  in while running (see "State files" above); the others come with their
+  features.
 - Health checks, the metrics, and listing, adding and lifting bans or asking why
   a given address was refused, all on the one port every request uses: under
   `/_smallwebwaf/` on the app's own address, through traefik like any other
@@ -514,17 +718,16 @@ the metrics, failure behaviour and the build order.
 So far `smallwebwaf` looks up only the country, only through GeoJS, and only
 while `SWWAF_DENIED_COUNTRIES` or `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` is set:
 then the address of every new visitor is sent to GeoJS, except a visitor in
-`SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS` and one refused because a ban covers its
-netblock, and with neither set, none is. An IPv6 visitor is asked about by the
-first address of its /64. A new visitor waits at most a second for its answer,
-and without one counts as coming from an unknown country until the answer
-arrives. The addresses waiting are asked about together, up to 200 in one
-request, one request at a time; at most 10,000 visitors wait, and one more
-counts as coming from an unknown country until there is room. While GeoJS fails,
-visitors with a kept answer are unaffected and new ones count as coming from an
-unknown country. GeoJS is then left alone for a second, twice as long after each
-further failure up to five minutes, and asked again by the next request that
-needs it.
+`SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS` and one whose netblock a ban covers, and
+with neither set, none is. An IPv6 visitor is asked about by the first address
+of its /64. A new visitor waits at most a second for its answer, and without one
+counts as coming from an unknown country until the answer arrives. The addresses
+waiting are asked about together, up to 200 in one request, one request at a
+time; at most 10,000 visitors wait, and one more counts as coming from an
+unknown country until there is room. While GeoJS fails, visitors with a kept
+answer are unaffected and new ones count as coming from an unknown country.
+GeoJS is then left alone for a second, twice as long after each further failure
+up to five minutes, and asked again by the next request that needs it.
 
 In the full design, `smallwebwaf` looks up the AS number and country of every
 client, for the request log, the metrics and the ban notes, and for the country
@@ -576,15 +779,20 @@ addresses are never sent to GeoJS.
   limits, and writes the request's log line. Its `check` method is where a
   request is refused before anything reaches the app: for `SWWAF_DENY_NETS`, for
   a ban, for the country lists, for a rate limit, which bans the client, and for
-  an announced body over the size limit.
+  an announced body over the size limit; in `observe` mode, only for the size
+  limit, with what it would have refused for noted in the log line. A request
+  under `/_smallwebwaf/` that `check` lets through is answered by `answerAdmin`
+  instead of reaching the app.
+- `internal/metrics`: the metrics, counted as the other parts tell it what
+  happened, and served in the Prometheus text format.
 - `internal/bans`: the ban ledger: each netblock's bans with their notes, how
   long a new ban lasts, and which ban is dropped when `SWWAF_MAX_BANS` are held.
 - `internal/lookup`: looks up each client's country through GeoJS, and keeps the
   answers.
 - `internal/ratelimit`: the table of clients: counts each client's requests,
   tells when one takes it over a rate limit, and keeps each client's history.
-- `internal/state`: reads the state files at start, and writes them when they
-  are due and at the stop.
+- `internal/state`: reads the state files at start, takes in an admin's edit of
+  one while running, and writes them when they are due and at the stop.
 - `internal/requestlog`: the lines on stdout: the request log line and the
   process's own messages.
 - `Dockerfile`: the lint and test phases, then the image, whose last stage
@@ -595,8 +803,10 @@ addresses are never sent to GeoJS.
 
 Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the
 table of clients to 20,000, the GeoJS answers to 100,000 and the banned
-netblocks to `SWWAF_MAX_BANS`, dropping the least recently seen. The country
-codes are the list in `internal/config/config.go`.
+netblocks to `SWWAF_MAX_BANS`, dropping the least recently seen, and
+`github.com/prometheus/client_golang` keeps the metrics and serves them, and
+`github.com/fsnotify/fsnotify` tells `smallwebwaf` when a state file is saved.
+The country codes are the list in `internal/config/config.go`.
 
 ## Entrypoints
 
@@ -639,9 +849,7 @@ so that they run in minimal containers.
 
 ## TODO
 
-- The rest of milestone 3, from taking in an admin's edits to the state files
-  (https://git.eeqj.de/sneak/smallwebwaf/issues/68) up to the metrics endpoint,
-  and the rest of the design, in the order of the build order in
+- The rest of the design, in the order of the build order in
   [`SPEC.md`](SPEC.md).
 
 ## Documents

go.mod

@@ -2,4 +2,20 @@ module sneak.berlin/go/smallwebwaf
 
 go 1.26.0
 
-require github.com/hashicorp/golang-lru/v2 v2.0.7
+require (
+	github.com/fsnotify/fsnotify v1.10.1
+	github.com/hashicorp/golang-lru/v2 v2.0.7
+	github.com/prometheus/client_golang v1.24.1
+)
+
+require (
+	github.com/beorn7/perks v1.0.1 // indirect
+	github.com/cespare/xxhash/v2 v2.3.0 // indirect
+	github.com/kylelemons/godebug v1.1.0 // indirect
+	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
+	github.com/prometheus/client_model v0.6.2 // indirect
+	github.com/prometheus/common v0.70.1 // indirect
+	github.com/prometheus/procfs v0.21.1 // indirect
+	golang.org/x/sys v0.47.0 // indirect
+	google.golang.org/protobuf v1.36.11 // indirect
+)

go.sum

@@ -1,2 +1,40 @@
+github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
+github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
+github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
+github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho=
+github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo=
+github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
+github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
 github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
 github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
+github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
+github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
+github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
+github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
+github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
+github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/prometheus/client_golang v1.24.1 h1:JnJkREXzWxUdCuPFpIWZiPispT9xVV59uiuyR2bPlnU=
+github.com/prometheus/client_golang v1.24.1/go.mod h1:F+oSRECHg4sse5ucfYpYDeIv/hu68Zo0uoHKetWnzcE=
+github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
+github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
+github.com/prometheus/common v0.70.1 h1:1HvjP4D5oL3t8RsPlwxA9onvvStjtIHYE5XuuwOi/PY=
+github.com/prometheus/common v0.70.1/go.mod h1:VdFUQDMZK3VLkurFUVhia6uys/0suUp86TJz5qbJRhc=
+github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+OJI=
+github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY=
+github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
+github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
+go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
+go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
+go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
+go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
+golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
+golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
+google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
+google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
+gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
+gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

internal/bans/bans.go

@@ -26,9 +26,9 @@ const maxTextBytes = 256
 type Rules struct {
 	// LimitBanDuration is how long a first ban lasts.
 	LimitBanDuration time.Duration
-	// LimitBanRepeatWindow is how soon after the netblock's last ban
-	// ended a broken limit counts as a repeat, which bans for
-	// repeatFactor times as long as that ban.
+	// LimitBanRepeatWindow is how soon after the end of the netblock's
+	// ban that ended last a broken limit counts as a repeat, which bans
+	// for repeatFactor times as long as that ban.
 	LimitBanRepeatWindow time.Duration
 	// MaxBanDuration is the longest ban; a ban that would be longer is
 	// permanent instead.
@@ -107,11 +107,19 @@ type Ledger struct {
 	changed chan struct{}
 
 	mu sync.Mutex
-	// netblocks holds each banned netblock's bans, oldest first. Each
-	// request from a netblock makes it the most recently seen.
+	// netblocks holds each banned netblock's bans, oldest first. Check and
+	// Find make each netblock they find the most recently seen.
 	netblocks *simplelru.LRU[netip.Prefix, *[]Ban]
 	// held is how many bans netblocks holds, at most rules.MaxBans.
 	held int
+	// made is how many bans BanForLimit has made since the start.
+	made int
+	// v4Lengths and v6Lengths are the lengths of the IPv4 and IPv6
+	// netblocks that have been banned. Check looks for a ban at each of
+	// them, so that a ban read from bans.json refuses every client in its
+	// netblock even when it was made with another SWWAF_BAN_SCOPE_V4_PREFIX,
+	// or another length of an IPv6 client's netblock.
+	v4Lengths, v6Lengths []int
 }
 
 // New returns a Ledger with no ban yet.
@@ -136,34 +144,55 @@ func (l *Ledger) Changed() <-chan struct{} {
 	return l.changed
 }
 
-// Check is called for each request from netblock, at now. It reports
-// whether a ban on netblock is active, and returns that ban, with the
-// request counted among those it refused.
-func (l *Ledger) Check(netblock netip.Prefix, now time.Time) (Ban, bool) {
+// Check is called for a request from client, at now. It reports whether
+// a ban on a netblock client is in is active, and returns that ban, with
+// the request counted among those it refused.
+func (l *Ledger) Check(client netip.Addr, now time.Time) (Ban, bool) {
 	l.mu.Lock()
 	defer l.mu.Unlock()
 
-	bans, found := l.netblocks.Get(netblock)
-	if !found {
+	ban := l.active(client, now)
+	if ban == nil {
 		return Ban{}, false
 	}
 
-	// A ban is made only once the one before has ended, so only the last
-	// can be active.
-	last := &(*bans)[len(*bans)-1]
-	if !last.ActiveAt(now) {
+	ban.Notes.Requests++
+	ban.Notes.Refused++
+
+	return *ban, true
+}
+
+// Find is Check without counting the request among those the ban
+// refused: in observe mode a ban refuses nothing.
+func (l *Ledger) Find(client netip.Addr, now time.Time) (Ban, bool) {
+	l.mu.Lock()
+	defer l.mu.Unlock()
+
+	ban := l.active(client, now)
+	if ban == nil {
 		return Ban{}, false
 	}
 
-	last.Notes.Requests++
-	last.Notes.Refused++
+	return *ban, true
+}
 
-	return *last, true
+// activeBan returns the ban in bans, a netblock's bans oldest first, that
+// is active at now, or nil when none is. If several are, it returns the
+// one that started last. Every ban is looked at, since a ban an admin adds
+// to bans.json can start before the netblock's others and outlast them.
+func activeBan(bans []Ban, now time.Time) *Ban {
+	for i := len(bans) - 1; i >= 0; i-- {
+		if bans[i].ActiveAt(now) {
+			return &bans[i]
+		}
+	}
+
+	return nil
 }
 
 // BanForLimit bans netblock at now for a broken limit, with notes, and
 // returns the ban. A first ban lasts LimitBanDuration. A ban made within
-// LimitBanRepeatWindow after the netblock's last ban ended lasts
+// LimitBanRepeatWindow after the netblock's ban that ended last lasts
 // repeatFactor times as long as that one. A ban that would be longer
 // than MaxBanDuration is permanent instead. If a ban on netblock is still
 // active, as when two of its requests break a limit at once, that ban is
@@ -177,12 +206,22 @@ func (l *Ledger) BanForLimit(netblock netip.Prefix, now time.Time, notes Notes)
 
 	bans, found := l.netblocks.Get(netblock)
 	if found {
-		last = &(*bans)[len(*bans)-1]
-		if last.ActiveAt(now) {
-			return *last
+		active := activeBan(*bans, now)
+		if active != nil {
+			return *active
 		}
 
-		notes.EarlierBans = last.Notes.EarlierBans + 1
+		// No ban is active, so each has an end. A ban an admin adds to
+		// bans.json can start after another and end before it, so the
+		// ban that ended last is looked for among them all.
+		ended := slices.MaxFunc(*bans, func(a, b Ban) int {
+			return a.Expires.Compare(b.Expires)
+		})
+		last = &ended
+
+		// The netblock's first ban held counts the bans it had before that
+		// one, since dropped to make room, and each ban held adds one.
+		notes.EarlierBans = (*bans)[0].Notes.EarlierBans + len(*bans)
 	}
 
 	notes.Request = notes.Request.cut()
@@ -193,6 +232,7 @@ func (l *Ledger) BanForLimit(netblock netip.Prefix, now time.Time, notes Notes)
 		Notes:    notes,
 	}
 	l.add(ban)
+	l.made++
 
 	select {
 	case l.changed <- struct{}{}:
@@ -216,6 +256,38 @@ func (l *Ledger) Bans(netblock netip.Prefix) []Ban {
 	return slices.Clone(*bans)
 }
 
+// Made returns how many bans the ledger has made since the start; bans
+// read from bans.json are not among them.
+func (l *Ledger) Made() int {
+	l.mu.Lock()
+	defer l.mu.Unlock()
+
+	return l.made
+}
+
+// Count returns how many of the bans held are active at now, and how many
+// are permanent.
+func (l *Ledger) Count(now time.Time) (int, int) {
+	l.mu.Lock()
+	defer l.mu.Unlock()
+
+	active, permanent := 0, 0
+
+	for _, bans := range l.netblocks.Values() {
+		for _, ban := range *bans {
+			if ban.ActiveAt(now) {
+				active++
+			}
+
+			if ban.Permanent() {
+				permanent++
+			}
+		}
+	}
+
+	return active, permanent
+}
+
 // Snapshot returns every ban held, sorted by netblock, and each
 // netblock's bans oldest first, as bans.json lists them.
 func (l *Ledger) Snapshot() []Ban {
@@ -234,26 +306,55 @@ func (l *Ledger) Snapshot() []Ban {
 	return held
 }
 
-// Load puts bans read from bans.json into a ledger that holds none yet,
-// in the order they started, so that a netblock whose last ban started
-// latest counts as the most recently seen. Each text in their notes is
-// cut to 256 bytes, and past MaxBans the earliest bans are dropped, as
-// when they are made.
+// Load puts bans read from bans.json into the ledger, in place of the
+// bans it holds, in the order they started, so that a netblock whose last
+// ban started latest counts as the most recently seen. Each netblock is
+// masked to its length, so that 203.0.113.9/24 is 203.0.113.0/24, and
+// each text in the notes is cut to 256 bytes. Past MaxBans the earliest
+// bans are dropped, as when they are made.
 func (l *Ledger) Load(bans []Ban) {
-	l.mu.Lock()
-	defer l.mu.Unlock()
-
 	bans = slices.Clone(bans)
 	slices.SortStableFunc(bans, func(a, b Ban) int {
 		return a.Start.Compare(b.Start)
 	})
 
+	l.mu.Lock()
+	defer l.mu.Unlock()
+
+	l.netblocks.Purge()
+	l.held = 0
+	l.v4Lengths, l.v6Lengths = nil, nil
+
 	for _, ban := range bans {
+		ban.Netblock = ban.Netblock.Masked()
 		ban.Notes.Request = ban.Notes.Request.cut()
 		l.add(ban)
 	}
 }
 
+// active returns the ban active at now on a netblock client is in, or
+// nil.
+func (l *Ledger) active(client netip.Addr, now time.Time) *Ban {
+	lengths := l.v6Lengths
+	if client.Is4() {
+		lengths = l.v4Lengths
+	}
+
+	for _, length := range lengths {
+		bans, found := l.netblocks.Get(netip.PrefixFrom(client, length).Masked())
+		if !found {
+			continue
+		}
+
+		ban := activeBan(*bans, now)
+		if ban != nil {
+			return ban
+		}
+	}
+
+	return nil
+}
+
 // add adds ban to its netblock's bans, after the last, and makes its
 // netblock the most recently seen. With MaxBans held, it drops one first.
 func (l *Ledger) add(ban Ban) {
@@ -271,11 +372,20 @@ func (l *Ledger) add(ban Ban) {
 
 	*bans = append(*bans, ban)
 	l.held++
+
+	lengths := &l.v6Lengths
+	if ban.Netblock.Addr().Is4() {
+		lengths = &l.v4Lengths
+	}
+
+	if !slices.Contains(*lengths, ban.Netblock.Bits()) {
+		*lengths = append(*lengths, ban.Netblock.Bits())
+	}
 }
 
 // expiry returns when a ban for a broken limit made at now ends, or zero
-// when it is permanent. last is the netblock's last ban, which has ended,
-// or nil when it has none.
+// when it is permanent. last is the netblock's ban that ended last, or nil
+// when it has none.
 func (l *Ledger) expiry(last *Ban, now time.Time) time.Time {
 	length := l.rules.LimitBanDuration
 

internal/bans/bans_test.go

@@ -39,7 +39,7 @@ func TestRepeatsTripleUntilPermanent(t *testing.T) {
 		t.Fatalf("sixth ban ends at %s, want a permanent one", ban.Expires)
 	}
 
-	_, banned := ledger.Check(netblock, now.Add(100*365*day))
+	_, banned := ledger.Check(netblock.Addr(), now.Add(100*365*day))
 	if !banned {
 		t.Error("a permanent ban ended")
 	}
@@ -138,18 +138,18 @@ func TestCheckRefusesWhileTheBanLastsAndCountsTheRefusals(t *testing.T) {
 	ban := ledger.BanForLimit(netblock, midnight(), bans.Notes{Requests: 5})
 
 	for range 3 {
-		got, banned := ledger.Check(netblock, ban.Expires.Add(-time.Nanosecond))
+		got, banned := ledger.Check(netblock.Addr(), ban.Expires.Add(-time.Nanosecond))
 		if !banned || got.Start != ban.Start {
 			t.Fatalf("check during the ban gives %+v and %t", got, banned)
 		}
 	}
 
-	_, banned := ledger.Check(netip.MustParsePrefix("203.0.113.10/32"), midnight())
+	_, banned := ledger.Check(netip.MustParseAddr("203.0.113.10"), midnight())
 	if banned {
 		t.Error("another netblock is banned")
 	}
 
-	_, banned = ledger.Check(netblock, ban.Expires)
+	_, banned = ledger.Check(netblock.Addr(), ban.Expires)
 	if banned {
 		t.Error("the ban did not end")
 	}
@@ -162,6 +162,28 @@ func TestCheckRefusesWhileTheBanLastsAndCountsTheRefusals(t *testing.T) {
 	}
 }
 
+func TestFindCountsNothing(t *testing.T) {
+	t.Parallel()
+
+	ledger := bans.New(defaultRules())
+	netblock := netip.MustParsePrefix("203.0.113.9/32")
+	ban := ledger.BanForLimit(netblock, midnight(), bans.Notes{Requests: 5})
+
+	got, banned := ledger.Find(netblock.Addr(), ban.Expires.Add(-time.Nanosecond))
+	if !banned || got != ban {
+		t.Errorf("find during the ban gives %+v and %t, want %+v", got, banned, ban)
+	}
+
+	_, banned = ledger.Find(netblock.Addr(), ban.Expires)
+	if banned {
+		t.Error("the ban did not end")
+	}
+
+	if notes := ledger.Bans(netblock)[0].Notes; notes != ban.Notes {
+		t.Errorf("the notes are %+v, want them unchanged, %+v", notes, ban.Notes)
+	}
+}
+
 func TestMaxBansDropsTheEarliestBanOfTheNetblockSeenLongestAgo(t *testing.T) {
 	t.Parallel()
 
@@ -180,7 +202,7 @@ func TestMaxBansDropsTheEarliestBanOfTheNetblockSeenLongestAgo(t *testing.T) {
 
 	// A request from a makes b the netblock seen longest ago, and its ban
 	// goes to make room for d's.
-	ledger.Check(a, now)
+	ledger.Check(a.Addr(), now)
 	ledger.BanForLimit(d, now, bans.Notes{})
 	wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 0, c: 1, d: 1})
 
@@ -190,7 +212,7 @@ func TestMaxBansDropsTheEarliestBanOfTheNetblockSeenLongestAgo(t *testing.T) {
 
 	// With d seen since, a is seen longest ago, and its earlier ban goes
 	// first.
-	ledger.Check(d, first.Expires)
+	ledger.Check(d.Addr(), first.Expires)
 	ledger.BanForLimit(b, first.Expires, bans.Notes{})
 	wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 1, d: 1})
 

internal/bans/snapshot_test.go

@@ -24,7 +24,7 @@ func TestChangedAfterABanIsMade(t *testing.T) {
 	// A limit broken during the ban makes no other, and a refusal changes
 	// only the counts in the notes, which wait for the interval's write.
 	ledger.BanForLimit(netblock, midnight().Add(time.Minute), bans.Notes{})
-	ledger.Check(netblock, midnight().Add(time.Minute))
+	ledger.Check(netblock.Addr(), midnight().Add(time.Minute))
 	wantChanged(t, ledger, false)
 
 	// Two bans before the value is read leave one.
@@ -76,7 +76,7 @@ func TestLoadedBansCarryOn(t *testing.T) {
 	after := bans.New(defaultRules())
 	after.Load(before.Snapshot())
 
-	_, banned := after.Check(netblock, ban.Expires.Add(-time.Second))
+	_, banned := after.Check(netblock.Addr(), ban.Expires.Add(-time.Second))
 	if !banned {
 		t.Error("the loaded ban does not refuse")
 	}
@@ -88,6 +88,117 @@ func TestLoadedBansCarryOn(t *testing.T) {
 	}
 }
 
+func TestLoadedBanRefusesEveryClientInItsNetblock(t *testing.T) {
+	t.Parallel()
+
+	// Two entries as an admin might write them, with addresses not masked
+	// to their lengths, the IPv6 one shorter than the /64 an IPv6 client's
+	// ban covers, beside a ban the ledger makes on one IPv4 address.
+	ledger := bans.New(defaultRules())
+	ledger.Load([]bans.Ban{
+		{Netblock: netip.MustParsePrefix("203.0.113.9/24"), Start: midnight()},
+		{Netblock: netip.MustParsePrefix("2001:db8::1/48"), Start: midnight()},
+	})
+	ledger.BanForLimit(netip.MustParsePrefix("198.51.100.7/32"), midnight(), bans.Notes{})
+
+	for client, want := range map[string]bool{
+		"203.0.113.0":     true,
+		"203.0.113.200":   true,
+		"203.0.114.1":     false,
+		"2001:db8:0:5::1": true,
+		"2001:db8:1::1":   false,
+		"198.51.100.7":    true,
+		"198.51.100.8":    false,
+	} {
+		_, banned := ledger.Check(netip.MustParseAddr(client), midnight())
+		if banned != want {
+			t.Errorf("%s is refused: %t, want %t", client, banned, want)
+		}
+	}
+
+	// The loaded netblocks are written back masked.
+	snapshot := ledger.Snapshot()
+
+	got := make([]string, 0, len(snapshot))
+	for _, ban := range snapshot {
+		got = append(got, ban.Netblock.String())
+	}
+
+	want := []string{"198.51.100.7/32", "203.0.113.0/24", "2001:db8::/48"}
+	if !slices.Equal(got, want) {
+		t.Errorf("the ledger holds bans on %v, want %v", got, want)
+	}
+}
+
+func TestPermanentBanStartedBeforeAnEndedOneRefuses(t *testing.T) {
+	t.Parallel()
+
+	// As when an admin adds a permanent ban to bans.json with a start
+	// before that of the netblock's ban that has ended.
+	netblock := netip.MustParsePrefix("203.0.113.0/24")
+	permanent := bans.Ban{Netblock: netblock, Start: midnight().Add(-time.Hour)}
+	ended := bans.Ban{
+		Netblock: netblock,
+		Start:    midnight(),
+		Expires:  midnight().Add(time.Hour),
+	}
+
+	ledger := bans.New(defaultRules())
+	ledger.Load([]bans.Ban{permanent, ended})
+
+	now := midnight().Add(2 * time.Hour)
+	client := netip.MustParseAddr("203.0.113.9")
+
+	ban, banned := ledger.Find(client, now)
+	if !banned || !ban.Permanent() {
+		t.Errorf("find gives %+v and %t, want the permanent ban", ban, banned)
+	}
+
+	ban, banned = ledger.Check(client, now)
+	if !banned || !ban.Permanent() {
+		t.Errorf("the client is refused: %t, under %+v, want under the permanent ban",
+			banned, ban)
+	}
+
+	// A limit broken now makes no shorter ban over the permanent one.
+	ban = ledger.BanForLimit(netblock, now, bans.Notes{})
+	if !ban.Permanent() || len(ledger.Bans(netblock)) != 2 {
+		t.Errorf("a broken limit returned %+v and left the netblock %d bans, "+
+			"want the permanent ban and 2", ban, len(ledger.Bans(netblock)))
+	}
+}
+
+func TestNextBanWorkedOutFromTheBanThatEndedLast(t *testing.T) {
+	t.Parallel()
+
+	// A 9-hour ban smallwebwaf made, the third in a row, and an admin's
+	// 1-hour ban added to bans.json over it, with no notes.
+	netblock := netip.MustParsePrefix("203.0.113.9/32")
+	nineHours := bans.Ban{
+		Netblock: netblock,
+		Start:    midnight(),
+		Expires:  midnight().Add(9 * time.Hour),
+		Notes:    bans.Notes{EarlierBans: 2},
+	}
+	admins := bans.Ban{
+		Netblock: netblock,
+		Start:    midnight().Add(time.Hour),
+		Expires:  midnight().Add(2 * time.Hour),
+	}
+
+	ledger := bans.New(defaultRules())
+	ledger.Load([]bans.Ban{nineHours, admins})
+
+	// Once both have ended, a limit broken within the repeat window bans
+	// for three times the 9 hours, and the notes count the two bans
+	// before the 9-hour one, it, and the admin's.
+	ban := ledger.BanForLimit(netblock, nineHours.Expires.Add(time.Hour), bans.Notes{})
+	if ban.Expires.Sub(ban.Start) != 27*time.Hour || ban.Notes.EarlierBans != 4 {
+		t.Errorf("the next ban lasts %s with %d earlier bans, want 27h and 4",
+			ban.Expires.Sub(ban.Start), ban.Notes.EarlierBans)
+	}
+}
+
 func TestLoadKeepsAtMostMaxBansDroppingTheEarliest(t *testing.T) {
 	t.Parallel()
 
@@ -109,6 +220,41 @@ func TestLoadKeepsAtMostMaxBansDroppingTheEarliest(t *testing.T) {
 	}
 }
 
+func TestLoadReplacesTheBansHeld(t *testing.T) {
+	t.Parallel()
+
+	// Room for three bans, so that the second load, were it added to the
+	// two bans held, would drop none of them to make room.
+	rules := defaultRules()
+	rules.MaxBans = 3
+	ledger := bans.New(rules)
+	kept := bans.Ban{Netblock: netip.MustParsePrefix("2001:db8::/64"), Start: midnight()}
+	ledger.Load([]bans.Ban{
+		{Netblock: netip.MustParsePrefix("203.0.113.0/24"), Start: midnight()},
+		kept,
+	})
+
+	// Loaded again without the first ban, as when an admin's edit of
+	// bans.json is taken in, that ban is lifted.
+	ledger.Load([]bans.Ban{kept})
+
+	_, banned := ledger.Check(netip.MustParseAddr("203.0.113.9"), midnight())
+	if banned {
+		t.Error("a ban left out of the second load still refuses")
+	}
+
+	// The ledger holds one ban, so it makes two more without dropping any.
+	first := ledger.BanForLimit(netip.MustParsePrefix("198.51.100.7/32"), midnight(),
+		bans.Notes{})
+	second := ledger.BanForLimit(netip.MustParsePrefix("198.51.100.8/32"), midnight(),
+		bans.Notes{})
+
+	want := []bans.Ban{first, second, kept}
+	if got := ledger.Snapshot(); !slices.Equal(got, want) {
+		t.Errorf("the ledger holds %+v, want %+v", got, want)
+	}
+}
+
 func TestLoadCutsTheTextsTo256Bytes(t *testing.T) {
 	t.Parallel()
 

internal/config/config.go

@@ -12,11 +12,13 @@ import (
 	"net/http"
 	"net/netip"
 	"net/url"
+	"os"
 	"path/filepath"
 	"slices"
 	"strconv"
 	"strings"
 	"time"
+	"unicode/utf8"
 )
 
 // Config is smallwebwaf's settings. A timeout, size or rate limit of zero
@@ -26,6 +28,15 @@ type Config struct {
 	ListenAddr string
 	// UpstreamURL is the app (SWWAF_UPSTREAM_URL).
 	UpstreamURL *url.URL
+	// InstanceName is the name each request log line gives as instance
+	// (SWWAF_INSTANCE_NAME), by default the host's name, which docker sets
+	// to the first 12 characters of the container's id.
+	InstanceName string
+	// Observe is true in observe mode, when SWWAF_MODE is observe rather
+	// than enforce: a request that SWWAF_DENY_NETS, a ban, the country
+	// lists or a rate limit would refuse is passed to the app instead, and
+	// no ban is made.
+	Observe bool
 	// TrustedProxies are the netblocks whose X-Forwarded-For is
 	// believed (SWWAF_TRUSTED_PROXIES).
 	TrustedProxies []netip.Prefix
@@ -69,6 +80,10 @@ type Config struct {
 	RateLimitPerMinute int64
 	RateLimitPerHour   int64
 	RateLimitPerDay    int64
+	// RateLimitExemptPaths are the path prefixes whose requests the rate
+	// limits neither count nor refuse (SWWAF_RATE_LIMIT_EXEMPT_PATHS).
+	// Each starts with /.
+	RateLimitExemptPaths []string
 	// DeniedCountries are the countries whose clients are refused
 	// (SWWAF_DENIED_COUNTRIES). ExclusivelyAllowedCountries, when not
 	// empty, are the only countries whose clients are let through
@@ -103,6 +118,15 @@ type Config struct {
 	StateDir             string
 	StateWriteDelay      time.Duration
 	StateCounterInterval time.Duration
+	// LogRequestHeaders are the request headers whose values the request
+	// log gives, in lower case (SWWAF_LOG_REQUEST_HEADERS).
+	LogRequestHeaders []string
+	// MetricsToken is the bearer token a scraper sends for the metrics
+	// (SWWAF_METRICS_TOKEN), "" while it is unset and the metrics are off.
+	// MetricsTopN is how many countries get series of their own in the
+	// metrics (SWWAF_METRICS_TOP_N).
+	MetricsToken string
+	MetricsTopN  int
 
 	// settings are the values read, as given or by default, for the
 	// log line at start.
@@ -119,6 +143,10 @@ const (
 	mebibyte = 1 << 20
 	gibibyte = 1 << 30
 	ipv4Bits = 32
+	// minTokenLength is the fewest characters a token may have.
+	minTokenLength = 32
+	// masked is what the log shows for a token that is set.
+	masked = "********"
 )
 
 var (
@@ -139,6 +167,11 @@ var (
 			"such as http://127.0.0.1:8081")
 	errNotCountry = errors.New(
 		"is not a two-letter country code such as de or kp")
+	errNotHeaderName = errors.New(
+		"is not a header name such as accept-language")
+	errHeaderTakenOut = errors.New(
+		"is taken out of every request by Go's HTTP server, so it can never " +
+			"be logged")
 	errOnBothLists          = errors.New("is in SWWAF_DENIED_COUNTRIES too")
 	errNotOver4K            = errors.New("is not a size of more than 4K, such as 32K")
 	errNotDurationAboveZero = errors.New(
@@ -150,6 +183,10 @@ var (
 		"is not the length of an IPv4 netblock, from 0 to 32, such as 24")
 	errNotAbsolutePath = errors.New(
 		"is not an absolute path, such as /var/lib/smallwebwaf")
+	errShortToken    = errors.New("is shorter than 32 characters")
+	errNotMode       = errors.New("is not enforce or observe")
+	errNotPathPrefix = errors.New(
+		"is not a path prefix starting with /, such as /assets/")
 )
 
 // FromEnvironment reads the settings with lookupEnv, normally
@@ -157,9 +194,12 @@ var (
 // that is set but invalid is an error that names it.
 func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
 	env := &environment{lookupEnv: lookupEnv}
+	hostname, _ := os.Hostname() // "" when the host has no name to give
 	cfg := &Config{
 		ListenAddr:           env.address("SWWAF_LISTEN_ADDR", ":8080"),
 		UpstreamURL:          env.appURL("SWWAF_UPSTREAM_URL", "http://127.0.0.1:8081"),
+		InstanceName:         env.value("SWWAF_INSTANCE_NAME", hostname),
+		Observe:              env.observe("SWWAF_MODE", "enforce"),
 		TrustedProxies:       env.netblocks("SWWAF_TRUSTED_PROXIES", privateRanges),
 		ClientRequestTimeout: env.duration("SWWAF_CLIENT_REQUEST_TIMEOUT", "60s"),
 		ClientRequestHeaderMaxBytes: env.headerSize(
@@ -176,6 +216,7 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
 		RateLimitPerMinute:      env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "1000"),
 		RateLimitPerHour:        env.count("SWWAF_RATE_LIMIT_PER_HOUR", "10000"),
 		RateLimitPerDay:         env.count("SWWAF_RATE_LIMIT_PER_DAY", "50000"),
+		RateLimitExemptPaths:    env.pathPrefixes("SWWAF_RATE_LIMIT_EXEMPT_PATHS", ""),
 		DeniedCountries:         env.countries("SWWAF_DENIED_COUNTRIES", ""),
 		ExclusivelyAllowedCountries: env.countries(
 			"SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES", ""),
@@ -188,6 +229,10 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
 		StateDir:             env.absolutePath("SWWAF_STATE_DIR", "/var/lib/smallwebwaf"),
 		StateWriteDelay:      env.durationNotOff("SWWAF_STATE_WRITE_DELAY", "10s"),
 		StateCounterInterval: env.durationNotOff("SWWAF_STATE_COUNTER_INTERVAL", "15m"),
+		LogRequestHeaders: env.headerNames("SWWAF_LOG_REQUEST_HEADERS",
+			"accept,accept-language,accept-encoding,content-type,origin,range"),
+		MetricsToken: env.token("SWWAF_METRICS_TOKEN"),
+		MetricsTopN:  env.numberNotOff("SWWAF_METRICS_TOP_N", "50"),
 	}
 
 	for _, country := range cfg.ExclusivelyAllowedCountries {
@@ -260,6 +305,17 @@ func (e *environment) appURL(name, defaultValue string) *url.URL {
 	return upstream
 }
 
+// observe reads the setting that is the mode, enforce or observe, and
+// reports whether it is observe.
+func (e *environment) observe(name, defaultValue string) bool {
+	mode := e.value(name, defaultValue)
+	if mode != "enforce" && mode != "observe" {
+		e.check(name, fmt.Errorf("%q %w", mode, errNotMode))
+	}
+
+	return mode == "observe"
+}
+
 // netblocks reads a setting that is a list of netblocks.
 func (e *environment) netblocks(name, defaultValue string) []netip.Prefix {
 	netblocks, err := parseNetblocks(e.value(name, defaultValue))
@@ -301,6 +357,14 @@ func (e *environment) count(name, defaultValue string) int64 {
 	return count
 }
 
+// pathPrefixes reads a setting that is a list of path prefixes.
+func (e *environment) pathPrefixes(name, defaultValue string) []string {
+	prefixes, err := parsePathPrefixes(e.value(name, defaultValue))
+	e.check(name, err)
+
+	return prefixes
+}
+
 // countries reads a setting that is a list of countries.
 func (e *environment) countries(name, defaultValue string) []string {
 	countries, err := parseCountries(e.value(name, defaultValue))
@@ -309,6 +373,15 @@ func (e *environment) countries(name, defaultValue string) []string {
 	return countries
 }
 
+// headerNames reads a setting that is a list of header names, and
+// returns them in lower case.
+func (e *environment) headerNames(name, defaultValue string) []string {
+	headers, err := parseHeaderNames(e.value(name, defaultValue))
+	e.check(name, err)
+
+	return headers
+}
+
 // durationNotOff reads a setting that is a duration and, unlike a
 // timeout, cannot be off.
 func (e *environment) durationNotOff(name, defaultValue string) time.Duration {
@@ -353,6 +426,26 @@ func (e *environment) absolutePath(name, defaultValue string) string {
 	return path
 }
 
+// token reads a setting that is a bearer token. Unset, it is "", which
+// switches off what it guards; set, it must be at least minTokenLength
+// characters. Neither the log nor an error shows its value.
+func (e *environment) token(name string) string {
+	value, set := e.lookupEnv(name)
+	if !set {
+		e.settings = append(e.settings, slog.String(name, ""))
+
+		return ""
+	}
+
+	e.settings = append(e.settings, slog.String(name, masked))
+
+	if utf8.RuneCountInString(value) < minTokenLength {
+		e.check(name, errShortToken)
+	}
+
+	return value
+}
+
 // parseDuration reads a duration in Go's syntax, such as 90s or 15m, a
 // whole number of days such as 7d, or off.
 func parseDuration(value string) (time.Duration, error) {
@@ -559,6 +652,23 @@ func parseNetblock(value string) (netip.Prefix, error) {
 	return netip.PrefixFrom(addr, addr.BitLen()), nil
 }
 
+// parsePathPrefixes reads a comma-separated list of path prefixes, each
+// starting with /.
+func parsePathPrefixes(value string) ([]string, error) {
+	prefixes, err := parseList(value)
+	if err != nil {
+		return nil, err
+	}
+
+	for _, prefix := range prefixes {
+		if !strings.HasPrefix(prefix, "/") {
+			return nil, fmt.Errorf("%q %w", prefix, errNotPathPrefix)
+		}
+	}
+
+	return prefixes, nil
+}
+
 // countryCodes are the two-letter codes ISO 3166-1 assigns today, and XK,
 // the code in common use for Kosovo. golang.org/x/text/language cannot
 // check them: it also takes withdrawn codes such as su, and reserved ones
@@ -615,6 +725,44 @@ func parseCountries(value string) ([]string, error) {
 	return countries, nil
 }
 
+// headerNameChars are the characters RFC 9110 allows in a header name:
+// letters, digits and these marks.
+const headerNameChars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz" +
+	"0123456789!#$%&'*+-.^_`|~"
+
+// parseHeaderNames reads a comma-separated list of header names in either
+// case, and returns them in lower case. Host and Transfer-Encoding are
+// refused: Go's HTTP server takes them out of the request's headers.
+func parseHeaderNames(value string) ([]string, error) {
+	items, err := parseList(value)
+	if err != nil {
+		return nil, err
+	}
+
+	headers := make([]string, 0, len(items))
+
+	for _, item := range items {
+		for _, char := range item {
+			if !strings.ContainsRune(headerNameChars, char) {
+				return nil, fmt.Errorf("%q %w", item, errNotHeaderName)
+			}
+		}
+
+		header := strings.ToLower(item)
+		switch header {
+		case "host":
+			return nil, fmt.Errorf("%q %w; the request's host is the field host",
+				item, errHeaderTakenOut)
+		case "transfer-encoding":
+			return nil, fmt.Errorf("%q %w", item, errHeaderTakenOut)
+		}
+
+		headers = append(headers, header)
+	}
+
+	return headers, nil
+}
+
 // parseListenAddr checks an address to listen on: an optional host and a
 // port number.
 func parseListenAddr(value string) (string, error) {

internal/config/config_test.go

@@ -6,6 +6,7 @@ import (
 	"log/slog"
 	"maps"
 	"net/netip"
+	"os"
 	"slices"
 	"strings"
 	"testing"
@@ -18,6 +19,7 @@ import (
 const (
 	listenAddr              = "SWWAF_LISTEN_ADDR"
 	upstreamURL             = "SWWAF_UPSTREAM_URL"
+	mode                    = "SWWAF_MODE"
 	trustedProxies          = "SWWAF_TRUSTED_PROXIES"
 	clientRequestTimeout    = "SWWAF_CLIENT_REQUEST_TIMEOUT"
 	clientHeaderMaxBytes    = "SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES"
@@ -33,6 +35,7 @@ const (
 	rateLimitPerMinute      = "SWWAF_RATE_LIMIT_PER_MINUTE"
 	rateLimitPerHour        = "SWWAF_RATE_LIMIT_PER_HOUR"
 	rateLimitPerDay         = "SWWAF_RATE_LIMIT_PER_DAY"
+	rateLimitExemptPaths    = "SWWAF_RATE_LIMIT_EXEMPT_PATHS"
 	deniedCountries         = "SWWAF_DENIED_COUNTRIES"
 	allowedCountries        = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES"
 	banResponse             = "SWWAF_BAN_RESPONSE"
@@ -44,8 +47,19 @@ const (
 	stateDir                = "SWWAF_STATE_DIR"
 	stateWriteDelay         = "SWWAF_STATE_WRITE_DELAY"
 	stateCounterInterval    = "SWWAF_STATE_COUNTER_INTERVAL"
+	metricsToken            = "SWWAF_METRICS_TOKEN" //nolint:gosec // the setting's name
+	metricsTopN             = "SWWAF_METRICS_TOP_N"
+	instanceName            = "SWWAF_INSTANCE_NAME"
+	logRequestHeaders       = "SWWAF_LOG_REQUEST_HEADERS"
 )
 
+// defaultLogRequestHeaders is the default of SWWAF_LOG_REQUEST_HEADERS.
+const defaultLogRequestHeaders = "accept,accept-language,accept-encoding," +
+	"content-type,origin,range"
+
+// token is a token of 32 characters, the shortest allowed.
+const token = "0123456789abcdef0123456789abcdef"
+
 // off switches a timeout, a size limit or a rate limit off.
 const off = "off"
 
@@ -78,6 +92,7 @@ func TestDefaults(t *testing.T) {
 
 	wantSettings(t, cfg, config.Config{
 		ListenAddr:                  ":8080",
+		Observe:                     false,
 		ClientRequestTimeout:        time.Minute,
 		ClientRequestHeaderMaxBytes: 32 << 10,
 		ClientIdleTimeout:           2 * time.Minute,
@@ -98,6 +113,8 @@ func TestDefaults(t *testing.T) {
 		StateDir:                    "/var/lib/smallwebwaf",
 		StateWriteDelay:             10 * time.Second,
 		StateCounterInterval:        15 * time.Minute,
+		MetricsToken:                "",
+		MetricsTopN:                 50,
 	})
 
 	if cfg.UpstreamURL.String() != "http://127.0.0.1:8081" {
@@ -111,6 +128,22 @@ func TestDefaults(t *testing.T) {
 	wantNetblocks(t, cfg.DenyNets)
 	wantCountries(t, deniedCountries, cfg.DeniedCountries)
 	wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries)
+
+	hostname, err := os.Hostname()
+	if err != nil || hostname == "" || cfg.InstanceName != hostname {
+		t.Errorf("%s is %q, want the host's name %q (%v)", instanceName,
+			cfg.InstanceName, hostname, err)
+	}
+
+	wantHeaders := strings.Split(defaultLogRequestHeaders, ",")
+	if !slices.Equal(cfg.LogRequestHeaders, wantHeaders) {
+		t.Errorf("%s gave %v, want %v", logRequestHeaders, cfg.LogRequestHeaders,
+			wantHeaders)
+	}
+
+	if len(cfg.RateLimitExemptPaths) != 0 {
+		t.Errorf("%s gave %v, want none", rateLimitExemptPaths, cfg.RateLimitExemptPaths)
+	}
 }
 
 func TestValuesAsSet(t *testing.T) {
@@ -119,6 +152,7 @@ func TestValuesAsSet(t *testing.T) {
 	cfg := fromEnvironment(t, environment{
 		listenAddr:              "127.0.0.1:9000",
 		upstreamURL:             "https://app.internal:8443/",
+		mode:                    "observe",
 		trustedProxies:          " 192.0.2.1, 10.1.2.3/8 ,2001:db8::/32",
 		clientRequestTimeout:    "90s",
 		clientHeaderMaxBytes:    "8K",
@@ -134,6 +168,7 @@ func TestValuesAsSet(t *testing.T) {
 		rateLimitPerMinute:      "60",
 		rateLimitPerHour:        "600",
 		rateLimitPerDay:         "6000",
+		rateLimitExemptPaths:    "/assets/, /favicon.ico",
 		deniedCountries:         "cn, RU,kp,Xk",
 		allowedCountries:        "de",
 		banResponse:             "429",
@@ -145,10 +180,13 @@ func TestValuesAsSet(t *testing.T) {
 		stateDir:                "/srv/waf-state",
 		stateWriteDelay:         "500ms",
 		stateCounterInterval:    "1h",
+		metricsToken:            token,
+		metricsTopN:             "10",
 	})
 
 	wantSettings(t, cfg, config.Config{
 		ListenAddr:                  "127.0.0.1:9000",
+		Observe:                     true,
 		ClientRequestTimeout:        90 * time.Second,
 		ClientRequestHeaderMaxBytes: 8 << 10,
 		ClientIdleTimeout:           5 * time.Minute,
@@ -169,6 +207,8 @@ func TestValuesAsSet(t *testing.T) {
 		StateDir:                    "/srv/waf-state",
 		StateWriteDelay:             500 * time.Millisecond,
 		StateCounterInterval:        time.Hour,
+		MetricsToken:                token,
+		MetricsTopN:                 10,
 	})
 
 	if cfg.UpstreamURL.String() != "https://app.internal:8443/" {
@@ -181,6 +221,39 @@ func TestValuesAsSet(t *testing.T) {
 	wantNetblocks(t, cfg.DenyNets, "198.51.100.0/24")
 	wantCountries(t, deniedCountries, cfg.DeniedCountries, "CN", "RU", "KP", "XK")
 	wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries, "DE")
+
+	if !slices.Equal(cfg.RateLimitExemptPaths, []string{"/assets/", "/favicon.ico"}) {
+		t.Errorf("%s gave %v, want /assets/ and /favicon.ico",
+			rateLimitExemptPaths, cfg.RateLimitExemptPaths)
+	}
+}
+
+func TestPathPrefixNotStartingWithSlashStopsTheStart(t *testing.T) {
+	t.Parallel()
+
+	_, err := config.FromEnvironment(
+		environment{rateLimitExemptPaths: "/favicon.ico,assets/"}.lookupEnv)
+
+	want := rateLimitExemptPaths + `: "assets/" is not a path prefix ` +
+		`starting with /, such as /assets/`
+	if err == nil || err.Error() != want {
+		t.Errorf("error %v, want %s", err, want)
+	}
+}
+
+func TestInstanceNameAndLoggedHeadersAsSet(t *testing.T) {
+	t.Parallel()
+
+	cfg := fromEnvironment(t, environment{
+		instanceName:      "fsn1app1/gitea",
+		logRequestHeaders: " Accept , X-Custom",
+	})
+
+	if cfg.InstanceName != "fsn1app1/gitea" ||
+		!slices.Equal(cfg.LogRequestHeaders, []string{"accept", "x-custom"}) {
+		t.Errorf("%s is %q and %s gives %v", instanceName, cfg.InstanceName,
+			logRequestHeaders, cfg.LogRequestHeaders)
+	}
 }
 
 func TestCodeOnBothCountryListsStopsTheStart(t *testing.T) {
@@ -284,9 +357,7 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
 	t.Parallel()
 
 	for _, tc := range []struct{ name, value string }{
-		{listenAddr, "8080"},
-		{listenAddr, ":http"},
-		{listenAddr, ":65536"},
+		{listenAddr, "8080"}, {listenAddr, ":http"}, {listenAddr, ":65536"},
 		{upstreamURL, "127.0.0.1:8081"},
 		{upstreamURL, "ftp://127.0.0.1:8081"},
 		{upstreamURL, "http://"},
@@ -296,6 +367,7 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
 		{upstreamURL, "http://127.0.0.1:8081/app"},
 		{upstreamURL, "http://127.0.0.1:8081/?a=1"},
 		{upstreamURL, "http://user:secret@127.0.0.1:8081"},
+		{mode, "Observe"}, {mode, "block"}, {mode, ""},
 		{trustedProxies, "10.0.0.0/33"},
 		{trustedProxies, "traefik"},
 		{trustedProxies, "10.0.0.0/8,,192.168.0.0/16"},
@@ -303,8 +375,7 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
 		{allowNets, "192.0.2.0/24,monitoring"},
 		{rateLimitExemptNets, "2001:db8::/129"},
 		{denyNets, "198.51.100.0/24,"},
-		{clientRequestTimeout, "60"},
-		{clientRequestTimeout, ""},
+		{clientRequestTimeout, "60"}, {clientRequestTimeout, ""},
 		{clientIdleTimeout, "0s"},
 		{clientIdleTimeout, "2 minutes"},
 		{clientResponseTimeout, "1y"},
@@ -321,8 +392,8 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
 		{rateLimitPerMinute, "1K"},
 		{rateLimitPerHour, "0"},
 		{rateLimitPerHour, "1.5"},
-		{rateLimitPerDay, "-1"},
-		{rateLimitPerDay, "lots"},
+		{rateLimitPerDay, "-1"}, {rateLimitPerDay, "lots"},
+		{rateLimitExemptPaths, "/assets/,,/static/"},
 		{deniedCountries, "nk"},
 		{deniedCountries, "kp,,ir"},
 		{deniedCountries, "prk"},
@@ -344,6 +415,11 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
 		{stateDir, ""}, {stateDir, "state"}, {stateDir, "./var/lib/smallwebwaf"},
 		{stateWriteDelay, off}, {stateWriteDelay, "0s"},
 		{stateCounterInterval, off}, {stateCounterInterval, "15"},
+		{metricsTopN, off}, {metricsTopN, "0"}, {metricsTopN, "-1"},
+		{logRequestHeaders, "accept,,origin"}, {logRequestHeaders, "accept;origin"},
+		{logRequestHeaders, "accept language"}, {logRequestHeaders, "x-foo:"},
+		{logRequestHeaders, "host"}, {logRequestHeaders, "accept,Host"},
+		{logRequestHeaders, "transfer-encoding"}, {logRequestHeaders, "TRANSFER-ENCODING"},
 	} {
 		t.Run(tc.name+"="+tc.value, func(t *testing.T) {
 			t.Parallel()
@@ -360,6 +436,60 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
 	}
 }
 
+func TestHostOrTransferEncodingStopsTheStart(t *testing.T) {
+	t.Parallel()
+
+	// Only Host's message points to the field host.
+	for value, want := range map[string]string{
+		"Host": `"Host" is taken out of every request by Go's HTTP server, ` +
+			"so it can never be logged; the request's host is the field host",
+		"transfer-encoding": `"transfer-encoding" is taken out of every ` +
+			"request by Go's HTTP server, so it can never be logged",
+	} {
+		t.Run(value, func(t *testing.T) {
+			t.Parallel()
+
+			_, err := config.FromEnvironment(environment{logRequestHeaders: value}.lookupEnv)
+			if err == nil || err.Error() != logRequestHeaders+": "+want {
+				t.Errorf("error %v, want %s: %s", err, logRequestHeaders, want)
+			}
+		})
+	}
+}
+
+func TestShortTokenStopsTheStartWithoutShowingIt(t *testing.T) {
+	t.Parallel()
+
+	// Characters are counted, not bytes: each é takes two.
+	for _, value := range []string{"", token[1:], strings.Repeat("é", 31)} {
+		t.Run(value, func(t *testing.T) {
+			t.Parallel()
+
+			_, err := config.FromEnvironment(environment{metricsToken: value}.lookupEnv)
+
+			want := metricsToken + ": is shorter than 32 characters"
+			if err == nil || err.Error() != want {
+				t.Errorf("error %v, want %s", err, want)
+			}
+		})
+	}
+}
+
+func TestTokenIsLoggedMasked(t *testing.T) {
+	t.Parallel()
+
+	cfg := fromEnvironment(t, environment{metricsToken: token})
+
+	var out bytes.Buffer
+
+	slog.New(slog.NewJSONHandler(&out, nil)).Info("starting", "settings", cfg)
+
+	if strings.Contains(out.String(), token) ||
+		!strings.Contains(out.String(), `"`+metricsToken+`":"********"`) {
+		t.Errorf("the token is not logged masked: %s", out.String())
+	}
+}
+
 func TestLogsEachSettingWithItsValue(t *testing.T) {
 	t.Parallel()
 
@@ -378,9 +508,12 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
 		t.Fatalf("decode %s: %v", out.Bytes(), err)
 	}
 
+	hostname, _ := os.Hostname()
+
 	want := map[string]string{
 		listenAddr:              ":8080",
 		upstreamURL:             "http://127.0.0.1:8081",
+		mode:                    "enforce",
 		trustedProxies:          "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16",
 		clientRequestTimeout:    "45s",
 		clientHeaderMaxBytes:    "32K",
@@ -396,6 +529,7 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
 		rateLimitPerMinute:      "1000",
 		rateLimitPerHour:        "10000",
 		rateLimitPerDay:         "50000",
+		rateLimitExemptPaths:    "",
 		deniedCountries:         "",
 		allowedCountries:        "",
 		banResponse:             "403",
@@ -407,6 +541,10 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
 		stateDir:                "/var/lib/smallwebwaf",
 		stateWriteDelay:         "10s",
 		stateCounterInterval:    "15m",
+		metricsToken:            "",
+		metricsTopN:             "50",
+		instanceName:            hostname,
+		logRequestHeaders:       defaultLogRequestHeaders,
 	}
 	if !maps.Equal(line.Settings, want) {
 		t.Errorf("logged settings\n%v\nwant\n%v", line.Settings, want)
@@ -418,6 +556,7 @@ func wantSettings(t *testing.T, got *config.Config, want config.Config) {
 	t.Helper()
 
 	if got.ListenAddr != want.ListenAddr ||
+		got.Observe != want.Observe ||
 		got.ClientRequestTimeout != want.ClientRequestTimeout ||
 		got.ClientRequestHeaderMaxBytes != want.ClientRequestHeaderMaxBytes ||
 		got.ClientIdleTimeout != want.ClientIdleTimeout ||
@@ -435,7 +574,8 @@ func wantSettings(t *testing.T, got *config.Config, want config.Config) {
 	wantBanSettings(t, got, want)
 }
 
-// wantBanSettings checks the settings for bans and the state files.
+// wantBanSettings checks the settings for bans, the state files and the
+// metrics.
 func wantBanSettings(t *testing.T, got *config.Config, want config.Config) {
 	t.Helper()
 
@@ -453,6 +593,11 @@ func wantBanSettings(t *testing.T, got *config.Config, want config.Config) {
 		got.StateCounterInterval != want.StateCounterInterval {
 		t.Errorf("state settings\n%+v\nwant\n%+v", got, want)
 	}
+
+	if got.MetricsToken != want.MetricsToken || got.MetricsTopN != want.MetricsTopN {
+		t.Errorf("metrics token %q and top %d, want %q and %d",
+			got.MetricsToken, got.MetricsTopN, want.MetricsToken, want.MetricsTopN)
+	}
 }
 
 // wantNetblocks checks a list of netblocks.

internal/lookup/export_test.go

@@ -0,0 +1,9 @@
+package lookup
+
+import "net/http"
+
+// SetTransport has g's requests to GeoJS go through transport instead of
+// the network.
+func (g *GeoJS) SetTransport(transport http.RoundTripper) {
+	g.httpClient.Transport = transport
+}

internal/lookup/lookup.go

@@ -19,6 +19,7 @@ import (
 	"time"
 
 	"github.com/hashicorp/golang-lru/v2/simplelru"
+	"sneak.berlin/go/smallwebwaf/internal/metrics"
 )
 
 // URL is GeoJS's country endpoint. Asked about several addresses at once,
@@ -64,6 +65,9 @@ type Params struct {
 	Now func() time.Time
 	// ProcessLog receives GeoJS's failures.
 	ProcessLog *slog.Logger
+	// Metrics count the requests to GeoJS, those that failed, and the
+	// clients that go without an answer.
+	Metrics *metrics.Metrics
 }
 
 // GeoJS looks up clients' countries through GeoJS. At most one request
@@ -73,6 +77,7 @@ type GeoJS struct {
 	url        string
 	now        func() time.Time
 	processLog *slog.Logger
+	metrics    *metrics.Metrics
 	// httpClient follows no redirect, so that visitors' addresses go to
 	// GeoJS alone: a redirect is a failure.
 	httpClient *http.Client
@@ -121,6 +126,7 @@ func New(params Params) *GeoJS {
 		url:        params.URL,
 		now:        params.Now,
 		processLog: params.ProcessLog,
+		metrics:    params.Metrics,
 		httpClient: &http.Client{
 			CheckRedirect: func(*http.Request, []*http.Request) error {
 				return http.ErrUseLastResponse
@@ -160,6 +166,9 @@ func (g *GeoJS) Country(ctx context.Context, client netip.Prefix) string {
 	defer g.mu.Unlock()
 
 	country, found := g.kept(client)
+	if !found {
+		g.metrics.GeoJSUnanswered.Inc()
+	}
 
 	w, waiting := g.waiting[client]
 	if !found && waiting {
@@ -188,19 +197,21 @@ func (g *GeoJS) Snapshot() []Answer {
 	return answers
 }
 
-// Load keeps answers read from lookups.json, in a GeoJS that keeps none
-// yet, in the order they were last used, so that the one used longest
+// Load keeps answers read from lookups.json, in place of the answers it
+// keeps, in the order they were last used, so that the one used longest
 // ago is dropped first. Answers GeoJS gave keepFor ago or more are
 // dropped.
 func (g *GeoJS) Load(answers []Answer) {
-	g.mu.Lock()
-	defer g.mu.Unlock()
-
 	answers = slices.Clone(answers)
 	slices.SortStableFunc(answers, func(a, b Answer) int {
 		return a.Used.Compare(b.Used)
 	})
 
+	g.mu.Lock()
+	defer g.mu.Unlock()
+
+	g.answers.Purge()
+
 	now := g.now()
 
 	for _, answer := range answers {
@@ -234,6 +245,8 @@ func (g *GeoJS) answerOrWait(
 	g.ask(ctx)
 
 	if w == nil {
+		g.metrics.GeoJSUnanswered.Inc()
+
 		return "", nil // too many clients wait already
 	}
 
@@ -243,6 +256,8 @@ func (g *GeoJS) answerOrWait(
 	}
 
 	if w.late {
+		g.metrics.GeoJSUnanswered.Inc()
+
 		return "", nil
 	}
 
@@ -355,6 +370,8 @@ func (g *GeoJS) keep(
 	}
 
 	if err != nil {
+		g.metrics.GeoJSFailures.Inc()
+
 		g.retryDelay = min(max(retryDelayFactor*g.retryDelay, firstRetryDelay),
 			maxRetryDelay)
 		g.retryAt = now.Add(g.retryDelay)
@@ -399,6 +416,8 @@ func (g *GeoJS) request(
 
 	req.URL.RawQuery = "ip=" + strings.Join(addrs, ",")
 
+	g.metrics.GeoJSRequests.Inc()
+
 	res, err := g.httpClient.Do(req)
 	if err != nil {
 		// Do's error names the URL, and so the visitors' addresses, which

internal/lookup/lookup_test.go

@@ -10,9 +10,12 @@ import (
 	"strings"
 	"sync"
 	"testing"
+	"testing/synctest"
 	"time"
 
+	"github.com/prometheus/client_golang/prometheus/testutil"
 	"sneak.berlin/go/smallwebwaf/internal/lookup"
+	"sneak.berlin/go/smallwebwaf/internal/metrics"
 )
 
 const (
@@ -26,80 +29,87 @@ const (
 	leftOut = "203.0.113.7"
 	// timeout is how long a new client waits for its answer.
 	timeout = time.Second
-	// waitLimit bounds how long a test waits for what should happen.
-	waitLimit = 10 * time.Second
-	// pollInterval is how often a test looks again.
-	pollInterval = 10 * time.Millisecond
 	// week is how long an answer is kept.
 	week = 7 * 24 * time.Hour
 )
 
+// The tests that have GeoJS asked run in a synctest bubble, where the time
+// package runs on a clock of the test's own: a wait lasts exactly as long
+// as it should, however slowly the test process runs, and synctest.Wait
+// returns once g has done all it can before time passes. The stand-in for
+// GeoJS answers without the network, since a request waiting on the
+// network would keep that clock from moving on.
+
 func TestKeptAnswerIsUsedFor7DaysThenAskedAgain(t *testing.T) {
 	t.Parallel()
 
-	geojs, clock, g := start(t)
-	placed := netip.MustParsePrefix("203.0.113.9/32")
-	notPlaced := netip.MustParsePrefix(unplaced + "/32")
+	synctest.Test(t, func(t *testing.T) {
+		geojs, clock, g := start()
+		placed := netip.MustParsePrefix("203.0.113.9/32")
+		notPlaced := netip.MustParsePrefix(unplaced + "/32")
 
-	wantCountry(t, g, placed, germany)
-	wantCountry(t, g, notPlaced, "")
-	wantRequests(t, geojs, 2)
+		wantCountry(t, g, placed, germany)
+		wantCountry(t, g, notPlaced, "")
+		wantRequests(t, geojs, 2)
 
-	// An answer without a country is kept too.
-	clock.advance(week - time.Second)
-	wantCountry(t, g, placed, germany)
-	wantCountry(t, g, notPlaced, "")
-	wantRequests(t, geojs, 2)
+		// An answer without a country is kept too.
+		clock.advance(week - time.Second)
+		wantCountry(t, g, placed, germany)
+		wantCountry(t, g, notPlaced, "")
+		wantRequests(t, geojs, 2)
 
-	clock.advance(time.Second)
-	wantCountry(t, g, placed, germany)
-	wantRequests(t, geojs, 3)
-	wantAsked(t, geojs, 2, "203.0.113.9")
+		clock.advance(time.Second)
+		wantCountry(t, g, placed, germany)
+		wantRequests(t, geojs, 3)
+		wantAsked(t, geojs, 2, "203.0.113.9")
+	})
 }
 
 func TestNewClientWaitsAtMostOneSecondThenCountsAsNotFound(t *testing.T) {
 	t.Parallel()
 
-	geojs, clock, g := start(t)
-	client := netip.MustParsePrefix("203.0.113.9/32")
+	synctest.Test(t, func(t *testing.T) {
+		geojs, clock, g := start()
+		client := netip.MustParsePrefix("203.0.113.9/32")
 
-	// The client comes while GeoJS is asked about an earlier client, which
-	// it answers most of a second later. It is then asked about the client
-	// and does not answer: that request is abandoned a second after it
-	// began, well after the client's wait is over.
-	geojs.set(answeringSlowly)
+		// The client comes while GeoJS is asked about an earlier client, which
+		// it answers most of a second later. It is then asked about the client
+		// and does not answer: that request is abandoned a second after it
+		// began, well after the client's wait is over.
+		geojs.set(answeringSlowly)
 
-	var earlier sync.WaitGroup
+		var earlier sync.WaitGroup
 
-	earlier.Go(func() { g.Country(t.Context(), netip.MustParsePrefix("203.0.113.1/32")) })
-	defer earlier.Wait()
+		earlier.Go(func() { g.Country(t.Context(), netip.MustParsePrefix("203.0.113.1/32")) })
+		defer earlier.Wait()
 
-	waitForRequests(t, geojs, 1)
-	geojs.set(hanging)
+		waitForRequests(t, geojs, 1)
+		geojs.set(hanging)
 
-	began := time.Now()
+		began := time.Now()
 
-	wantCountry(t, g, client, "")
+		wantCountry(t, g, client, "")
 
-	took := time.Since(began)
-	if took < timeout || took > timeout+timeout/2 {
-		t.Errorf("waited %s for the answer, want %s", took, timeout)
-	}
+		took := time.Since(began)
+		if took != timeout {
+			t.Errorf("waited %s for the answer, want %s", took, timeout)
+		}
 
-	// Its next request does not wait.
-	began = time.Now()
+		// Its next request does not wait.
+		began = time.Now()
 
-	wantCountry(t, g, client, "")
+		wantCountry(t, g, client, "")
 
-	took = time.Since(began)
-	if took > timeout/2 {
-		t.Errorf("waited %s again, want no wait", took)
-	}
+		took = time.Since(began)
+		if took != 0 {
+			t.Errorf("waited %s again, want no wait", took)
+		}
 
-	// Once GeoJS answers, the client is asked about again in the
-	// background, and has its country.
-	geojs.set(answering)
-	waitForCountry(t, g, clock, client, germany)
+		// Once GeoJS answers, the client is asked about again in the
+		// background, and has its country.
+		geojs.set(answering)
+		waitForCountry(t, g, clock, client, germany)
+	})
 }
 
 func TestAddressLeftOutOfAnAnswerIsAskedAboutAgain(t *testing.T) {
@@ -118,33 +128,35 @@ func TestAddressLeftOutOfAnAnswerIsAskedAboutAgain(t *testing.T) {
 		t.Run(tc.name, func(t *testing.T) {
 			t.Parallel()
 
-			geojs, clock, g := start(t)
-			other := netip.MustParsePrefix("203.0.113.1/32")
-			client := netip.MustParsePrefix(leftOut + "/32")
+			synctest.Test(t, func(t *testing.T) {
+				geojs, clock, g := start()
+				other := netip.MustParsePrefix("203.0.113.1/32")
+				client := netip.MustParsePrefix(leftOut + "/32")
 
-			// GeoJS fails, and is left alone for a second while the client
-			// comes too, so that the next request asks about both.
-			geojs.set(failing)
-			wantCountry(t, g, other, "")
-			wantCountry(t, g, client, "")
+				// GeoJS fails, and is left alone for a second while the client
+				// comes too, so that the next request asks about both.
+				geojs.set(failing)
+				wantCountry(t, g, other, "")
+				wantCountry(t, g, client, "")
 
-			geojs.set(tc.answers)
-			clock.advance(time.Second)
-			wantCountry(t, g, other, "")
-			waitForRequests(t, geojs, 2)
+				geojs.set(tc.answers)
+				clock.advance(time.Second)
+				wantCountry(t, g, other, "")
+				waitForRequests(t, geojs, 2)
 
-			// The answer counts as a failure, and the client is asked about
-			// again, with the other client only if the answer left it out too.
-			geojs.set(answering)
-			waitForCountry(t, g, clock, client, germany)
-			wantCountry(t, g, other, germany)
-			wantRequests(t, geojs, 3)
+				// The answer counts as a failure, and the client is asked about
+				// again, with the other client only if the answer left it out too.
+				geojs.set(answering)
+				waitForCountry(t, g, clock, client, germany)
+				wantCountry(t, g, other, germany)
+				wantRequests(t, geojs, 3)
 
-			if tc.named {
-				wantAsked(t, geojs, 2, leftOut)
-			} else {
-				wantAsked(t, geojs, 2, leftOut, "203.0.113.1")
-			}
+				if tc.named {
+					wantAsked(t, geojs, 2, leftOut)
+				} else {
+					wantAsked(t, geojs, 2, leftOut, "203.0.113.1")
+				}
+			})
 		})
 	}
 }
@@ -152,173 +164,225 @@ func TestAddressLeftOutOfAnAnswerIsAskedAboutAgain(t *testing.T) {
 func TestRedirectCountsAsFailure(t *testing.T) {
 	t.Parallel()
 
-	geojs, _, g := start(t)
-	geojs.set(redirecting)
+	synctest.Test(t, func(t *testing.T) {
+		geojs, _, g := start()
+		geojs.set(redirecting)
 
-	wantCountry(t, g, netip.MustParsePrefix("203.0.113.9/32"), "")
-	wantRequests(t, geojs, 1)
+		wantCountry(t, g, netip.MustParsePrefix("203.0.113.9/32"), "")
+		wantRequests(t, geojs, 1)
+	})
 }
 
 func TestCountryIsKeptInCapitals(t *testing.T) {
 	t.Parallel()
 
-	geojs, _, g := start(t)
-	geojs.set(answeringInLowerCase)
+	synctest.Test(t, func(t *testing.T) {
+		geojs, _, g := start()
+		geojs.set(answeringInLowerCase)
 
-	wantCountry(t, g, netip.MustParsePrefix("203.0.113.9/32"), germany)
+		wantCountry(t, g, netip.MustParsePrefix("203.0.113.9/32"), germany)
+	})
 }
 
 func TestFailureIsLoggedWithoutTheAddressesAskedAbout(t *testing.T) {
 	t.Parallel()
 
-	var log strings.Builder
+	synctest.Test(t, func(t *testing.T) {
+		var log strings.Builder
 
-	// Nothing listens on port 1, so asking GeoJS fails.
-	g := lookup.New(lookup.Params{
-		URL:        "http://127.0.0.1:1",
-		Now:        time.Now,
-		ProcessLog: slog.New(slog.NewTextHandler(&log, nil)),
+		// GeoJS does not answer, so the request to it is abandoned, and fails.
+		geojs := &standIn{answers: hanging}
+		g := lookup.New(lookup.Params{
+			URL:        lookup.URL,
+			Now:        time.Now,
+			ProcessLog: slog.New(slog.NewTextHandler(&log, nil)),
+			Metrics:    metrics.New(1),
+		})
+		g.SetTransport(geojs)
+
+		wantCountry(t, g, netip.MustParsePrefix("203.0.113.9/32"), "")
+		synctest.Wait()
+
+		logged := log.String()
+		if !strings.Contains(logged, "asking GeoJS failed") ||
+			strings.Contains(logged, "203.0.113.9") {
+			t.Errorf("logged %q, want the failure without the address asked about", logged)
+		}
 	})
-
-	wantCountry(t, g, netip.MustParsePrefix("203.0.113.9/32"), "")
-
-	logged := log.String()
-	if !strings.Contains(logged, "asking GeoJS failed") ||
-		strings.Contains(logged, "203.0.113.9") {
-		t.Errorf("logged %q, want the failure without the address asked about", logged)
-	}
 }
 
 func TestWaitingClientsAreAskedAboutInOneRequest(t *testing.T) {
 	t.Parallel()
 
-	geojs, clock, g := start(t)
+	synctest.Test(t, func(t *testing.T) {
+		geojs, clock, g := start()
 
-	// GeoJS fails, and is then left alone for a second, while three more
-	// clients come. An IPv6 client is a /64, and GeoJS is asked about its
-	// first address.
-	geojs.set(failing)
-	wantCountry(t, g, netip.MustParsePrefix("203.0.113.1/32"), "")
-	wantCountry(t, g, netip.MustParsePrefix("203.0.113.2/32"), "")
-	wantCountry(t, g, netip.MustParsePrefix("2001:db8:1:2::/64"), "")
-	wantRequests(t, geojs, 1)
+		// GeoJS fails, and is then left alone for a second, while three more
+		// clients come. An IPv6 client is a /64, and GeoJS is asked about its
+		// first address.
+		geojs.set(failing)
+		wantCountry(t, g, netip.MustParsePrefix("203.0.113.1/32"), "")
+		wantCountry(t, g, netip.MustParsePrefix("203.0.113.2/32"), "")
+		wantCountry(t, g, netip.MustParsePrefix("2001:db8:1:2::/64"), "")
+		wantRequests(t, geojs, 1)
 
-	geojs.set(answering)
-	clock.advance(time.Second)
-	wantCountry(t, g, netip.MustParsePrefix("203.0.113.3/32"), germany)
-	wantRequests(t, geojs, 2)
-	wantAsked(t, geojs, 1, "203.0.113.1", "203.0.113.2", "2001:db8:1:2::", "203.0.113.3")
+		geojs.set(answering)
+		clock.advance(time.Second)
+		wantCountry(t, g, netip.MustParsePrefix("203.0.113.3/32"), germany)
+		wantRequests(t, geojs, 2)
+		wantAsked(t, geojs, 1, "203.0.113.1", "203.0.113.2", "2001:db8:1:2::", "203.0.113.3")
+	})
 }
 
 func TestKeptAnswersUnaffectedWhileGeoJSFailsAndAskedAgainWithBackoff(t *testing.T) {
 	t.Parallel()
 
-	geojs, clock, g := start(t)
-	clients := newClients()
-	kept := clients()
+	synctest.Test(t, func(t *testing.T) {
+		geojs, clock, g := start()
+		clients := newClients()
+		kept := clients()
 
-	wantCountry(t, g, kept, germany)
-
-	geojs.set(failing)
-	wantCountry(t, g, kept, germany)
-	wantRequests(t, geojs, 1)
-
-	// Each failure leaves GeoJS alone twice as long as the one before, up
-	// to five minutes. New clients meanwhile count as not found, and the
-	// client with a kept answer still gets its country, without GeoJS being
-	// asked.
-	requests := 1
-
-	for _, delay := range []time.Duration{
-		time.Second, 2 * time.Second, 4 * time.Second, 8 * time.Second,
-		16 * time.Second, 32 * time.Second, 64 * time.Second, 128 * time.Second,
-		256 * time.Second, 5 * time.Minute, 5 * time.Minute,
-	} {
-		wantCountry(t, g, clients(), "")
-
-		requests++
-		wantRequests(t, geojs, requests)
-
-		clock.advance(delay - time.Millisecond)
-		wantCountry(t, g, clients(), "")
 		wantCountry(t, g, kept, germany)
-		wantRequests(t, geojs, requests)
 
-		clock.advance(time.Millisecond)
-	}
+		geojs.set(failing)
+		wantCountry(t, g, kept, germany)
+		wantRequests(t, geojs, 1)
 
-	// Once GeoJS answers again, it is asked about every client waiting.
-	geojs.set(answering)
-	wantCountry(t, g, clients(), germany)
-	wantRequests(t, geojs, requests+1)
+		// Each failure leaves GeoJS alone twice as long as the one before, up
+		// to five minutes. New clients meanwhile count as not found, and the
+		// client with a kept answer still gets its country, without GeoJS being
+		// asked.
+		requests := 1
 
-	asked := waitForRequests(t, geojs, requests+1)
-	if len(asked[requests]) != 23 {
-		t.Errorf("GeoJS was asked about %d clients, want 23", len(asked[requests]))
-	}
+		for _, delay := range []time.Duration{
+			time.Second, 2 * time.Second, 4 * time.Second, 8 * time.Second,
+			16 * time.Second, 32 * time.Second, 64 * time.Second, 128 * time.Second,
+			256 * time.Second, 5 * time.Minute, 5 * time.Minute,
+		} {
+			wantCountry(t, g, clients(), "")
+
+			requests++
+			wantRequests(t, geojs, requests)
+
+			clock.advance(delay - time.Millisecond)
+			wantCountry(t, g, clients(), "")
+			wantCountry(t, g, kept, germany)
+			wantRequests(t, geojs, requests)
+
+			clock.advance(time.Millisecond)
+		}
+
+		// Once GeoJS answers again, it is asked about every client waiting.
+		geojs.set(answering)
+		wantCountry(t, g, clients(), germany)
+		wantRequests(t, geojs, requests+1)
+
+		asked := waitForRequests(t, geojs, requests+1)
+		if len(asked[requests]) != 23 {
+			t.Errorf("GeoJS was asked about %d clients, want 23", len(asked[requests]))
+		}
+	})
 }
 
 func TestAtMost200AddressesInOneRequest(t *testing.T) {
 	t.Parallel()
 
-	geojs, clock, g := start(t)
-	clients := newClients()
-	first := clients()
+	synctest.Test(t, func(t *testing.T) {
+		geojs, clock, g := start()
+		clients := newClients()
+		first := clients()
 
-	// 201 clients wait while GeoJS is left alone after a failure.
-	geojs.set(failing)
-	wantCountry(t, g, first, "")
+		// 201 clients wait while GeoJS is left alone after a failure.
+		geojs.set(failing)
+		wantCountry(t, g, first, "")
 
-	for range 200 {
-		wantCountry(t, g, clients(), "")
-	}
+		for range 200 {
+			wantCountry(t, g, clients(), "")
+		}
 
-	// The first one's next request has GeoJS asked again.
-	geojs.set(answering)
-	clock.advance(time.Second)
-	wantCountry(t, g, first, "")
+		// The first one's next request has GeoJS asked again.
+		geojs.set(answering)
+		clock.advance(time.Second)
+		wantCountry(t, g, first, "")
 
-	asked := waitForRequests(t, geojs, 3)
-	if len(asked[1]) != 200 || len(asked[2]) != 1 {
-		t.Errorf("GeoJS was asked about %d and then %d clients, want 200 and 1",
-			len(asked[1]), len(asked[2]))
-	}
+		asked := waitForRequests(t, geojs, 3)
+		if len(asked[1]) != 200 || len(asked[2]) != 1 {
+			t.Errorf("GeoJS was asked about %d and then %d clients, want 200 and 1",
+				len(asked[1]), len(asked[2]))
+		}
+	})
 }
 
 func TestAtMost10000ClientsWait(t *testing.T) {
 	t.Parallel()
 
-	geojs, clock, g := start(t)
-	clients := newClients()
-	first := clients()
+	synctest.Test(t, func(t *testing.T) {
+		geojs, clock, g := start()
+		clients := newClients()
+		first := clients()
 
-	// 10,000 clients wait while GeoJS is left alone after a failure, and
-	// one more cannot join them.
-	geojs.set(failing)
-	wantCountry(t, g, first, "")
+		// 10,000 clients wait while GeoJS is left alone after a failure, and
+		// one more cannot join them.
+		geojs.set(failing)
+		wantCountry(t, g, first, "")
 
-	for range 9999 {
-		wantCountry(t, g, clients(), "")
-	}
-
-	extra := clients()
-	wantCountry(t, g, extra, "")
-
-	// The first one's next request has GeoJS asked about the 10,000, 200
-	// at a time, and not about the one more.
-	geojs.set(answering)
-	clock.advance(time.Second)
-	wantCountry(t, g, first, "")
-
-	asked := waitForRequests(t, geojs, 51)
-	for i, request := range asked {
-		if slices.Contains(request, extra.Addr().String()) {
-			t.Errorf("request %d asked about %s", i, extra.Addr())
+		for range 9999 {
+			wantCountry(t, g, clients(), "")
 		}
-	}
 
-	// With room among those waiting, it is asked about.
-	wantCountry(t, g, extra, germany)
+		extra := clients()
+		wantCountry(t, g, extra, "")
+
+		// The first one's next request has GeoJS asked about the 10,000, 200
+		// at a time, and not about the one more.
+		geojs.set(answering)
+		clock.advance(time.Second)
+		wantCountry(t, g, first, "")
+
+		asked := waitForRequests(t, geojs, 51)
+		for i, request := range asked {
+			if slices.Contains(request, extra.Addr().String()) {
+				t.Errorf("request %d asked about %s", i, extra.Addr())
+			}
+		}
+
+		// With room among those waiting, it is asked about.
+		wantCountry(t, g, extra, germany)
+	})
+}
+
+func TestClientsWithoutAnAnswerAreCounted(t *testing.T) {
+	t.Parallel()
+
+	synctest.Test(t, func(t *testing.T) {
+		m := metrics.New(1)
+		g := lookup.New(lookup.Params{
+			URL:        lookup.URL,
+			Now:        time.Now,
+			ProcessLog: slog.New(slog.DiscardHandler),
+			Metrics:    m,
+		})
+		g.SetTransport(&standIn{answers: failing})
+
+		clients := newClients()
+
+		// GeoJS fails, so the first client goes without an answer, and GeoJS
+		// is left alone for a second, which does not pass in this test.
+		wantCountry(t, g, clients(), "")
+		wantUnanswered(t, m, 1)
+
+		// Meanwhile each new client goes without one at once, while there is
+		// room for it among the 10,000 that may wait.
+		for range 9999 {
+			wantCountry(t, g, clients(), "")
+		}
+
+		wantUnanswered(t, m, 10000)
+
+		// One more, for which there is no room, goes without one too.
+		wantCountry(t, g, clients(), "")
+		wantUnanswered(t, m, 10001)
+	})
 }
 
 // How the stand-in for GeoJS answers.
@@ -337,13 +401,25 @@ const (
 // standIn is a stand-in for GeoJS. It notes the addresses each request
 // asks about.
 type standIn struct {
-	server *httptest.Server
-
 	mu       sync.Mutex
 	answers  int
 	requests [][]string
 }
 
+// RoundTrip has the stand-in answer req, in place of the network. A request
+// abandoned before the stand-in answers fails, as over the network.
+func (s *standIn) RoundTrip(req *http.Request) (*http.Response, error) {
+	answer := httptest.NewRecorder()
+	s.ServeHTTP(answer, req)
+
+	err := req.Context().Err()
+	if err != nil {
+		return nil, err
+	}
+
+	return answer.Result(), nil
+}
+
 // ServeHTTP answers a request about the addresses in its ip parameter.
 func (s *standIn) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 	addrs := strings.Split(r.URL.Query().Get("ip"), ",")
@@ -422,7 +498,8 @@ func (s *standIn) asked() [][]string {
 	return slices.Clone(s.requests)
 }
 
-// testClock is a clock the test sets.
+// testClock is a clock the test sets. GeoJS tells the time by it, while
+// waits run on the bubble's clock.
 type testClock struct {
 	mu  sync.Mutex
 	now time.Time
@@ -444,21 +521,18 @@ func (c *testClock) advance(d time.Duration) {
 	c.now = c.now.Add(d)
 }
 
-// start starts a stand-in for GeoJS that answers, and returns it, a
-// clock, and a GeoJS asking it by that clock.
-func start(t *testing.T) (*standIn, *testClock, *lookup.GeoJS) {
-	t.Helper()
-
+// start returns a stand-in for GeoJS that answers, a clock, and a GeoJS
+// asking the stand-in by that clock.
+func start() (*standIn, *testClock, *lookup.GeoJS) {
 	geojs := &standIn{}
-	geojs.server = httptest.NewServer(geojs)
-	t.Cleanup(geojs.server.Close)
-
 	clock := &testClock{now: time.Date(2026, 10, 4, 0, 0, 0, 0, time.UTC)}
 	g := lookup.New(lookup.Params{
-		URL:        geojs.server.URL,
+		URL:        lookup.URL,
 		Now:        clock.Now,
 		ProcessLog: slog.New(slog.DiscardHandler),
+		Metrics:    metrics.New(1),
 	})
+	g.SetTransport(geojs)
 
 	return geojs, clock, g
 }
@@ -513,41 +587,45 @@ func wantAsked(t *testing.T, geojs *standIn, i int, want ...string) {
 	}
 }
 
-// waitForRequests waits for GeoJS to have had count requests, and returns
-// the addresses each asked about.
+// wantUnanswered checks how many requests m counts as having gone without
+// an answer from GeoJS.
+func wantUnanswered(t *testing.T, m *metrics.Metrics, want float64) {
+	t.Helper()
+
+	got := testutil.ToFloat64(m.GeoJSUnanswered)
+	if got != want {
+		t.Errorf("%v requests went without an answer, want %v", got, want)
+	}
+}
+
+// waitForRequests waits until g has done all it can before time passes,
+// checks that GeoJS has had count requests, and returns the addresses each
+// asked about.
 func waitForRequests(t *testing.T, geojs *standIn, count int) [][]string {
 	t.Helper()
 
-	deadline := time.Now().Add(waitLimit)
-	for time.Now().Before(deadline) {
-		asked := geojs.asked()
-		if len(asked) >= count {
-			return asked
-		}
+	synctest.Wait()
 
-		time.Sleep(pollInterval)
+	asked := geojs.asked()
+	if len(asked) != count {
+		t.Fatalf("GeoJS had %d requests, want %d", len(asked), count)
 	}
 
-	t.Fatalf("fewer than %d requests to GeoJS after %s", count, waitLimit)
-
-	return nil
+	return asked
 }
 
-// waitForCountry waits for g to give client the country want, moving the
-// clock on a minute at a time, so that GeoJS is asked again after a
-// failure.
+// waitForCountry lets a request to GeoJS under way be abandoned, and moves
+// the clock on a minute, so that GeoJS may be asked again after a failure.
+// It then checks that client's next request does not wait but has it asked
+// about again in the background, after which g gives it the country want.
 func waitForCountry(
 	t *testing.T, g *lookup.GeoJS, clock *testClock, client netip.Prefix, want string,
 ) {
 	t.Helper()
 
-	deadline := time.Now().Add(waitLimit)
-	for g.Country(t.Context(), client) != want {
-		if time.Now().After(deadline) {
-			t.Fatalf("%s is not in %q after %s", client, want, waitLimit)
-		}
-
-		clock.advance(time.Minute)
-		time.Sleep(pollInterval)
-	}
+	time.Sleep(timeout)
+	clock.advance(time.Minute)
+	wantCountry(t, g, client, "")
+	synctest.Wait()
+	wantCountry(t, g, client, want)
 }

internal/lookup/snapshot_test.go

@@ -4,6 +4,7 @@ import (
 	"net/netip"
 	"slices"
 	"testing"
+	"testing/synctest"
 	"time"
 
 	"sneak.berlin/go/smallwebwaf/internal/lookup"
@@ -12,30 +13,32 @@ import (
 func TestSnapshotHoldsEachAnswerAndWhenItWasLastUsed(t *testing.T) {
 	t.Parallel()
 
-	_, clock, g := start(t)
-	placed := netip.MustParsePrefix("203.0.113.9/32")
-	notPlaced := netip.MustParsePrefix(unplaced + "/32")
-	asked := clock.Now()
+	synctest.Test(t, func(t *testing.T) {
+		_, clock, g := start()
+		placed := netip.MustParsePrefix("203.0.113.9/32")
+		notPlaced := netip.MustParsePrefix(unplaced + "/32")
+		asked := clock.Now()
 
-	wantCountry(t, g, placed, germany)
-	wantCountry(t, g, notPlaced, "")
+		wantCountry(t, g, placed, germany)
+		wantCountry(t, g, notPlaced, "")
 
-	clock.advance(time.Hour)
-	wantCountry(t, g, placed, germany)
+		clock.advance(time.Hour)
+		wantCountry(t, g, placed, germany)
 
-	want := []lookup.Answer{
-		{Client: notPlaced, Country: "", Answered: asked, Used: asked},
-		{Client: placed, Country: germany, Answered: asked, Used: asked.Add(time.Hour)},
-	}
-	if got := g.Snapshot(); !slices.Equal(got, want) {
-		t.Errorf("snapshot\n%+v\nwant\n%+v", got, want)
-	}
+		want := []lookup.Answer{
+			{Client: notPlaced, Country: "", Answered: asked, Used: asked},
+			{Client: placed, Country: germany, Answered: asked, Used: asked.Add(time.Hour)},
+		}
+		if got := g.Snapshot(); !slices.Equal(got, want) {
+			t.Errorf("snapshot\n%+v\nwant\n%+v", got, want)
+		}
+	})
 }
 
 func TestLoadedAnswersAreKeptFor7DaysFromWhenGeoJSGaveThem(t *testing.T) {
 	t.Parallel()
 
-	geojs, clock, g := start(t)
+	geojs, clock, g := start()
 	now := clock.Now()
 	kept := lookup.Answer{
 		Client:   netip.MustParsePrefix("203.0.113.9/32"),
@@ -65,7 +68,7 @@ func TestLoadDropsTheAnswerUsedLongestAgoFirst(t *testing.T) {
 
 	const maxAnswers = 100000
 
-	_, clock, g := start(t)
+	_, clock, g := start()
 	now := clock.Now()
 
 	// lookups.json lists the answers by client. Here each was last used a

internal/metrics/countries.go

@@ -0,0 +1,116 @@
+package metrics
+
+import (
+	"sync"
+
+	"github.com/prometheus/client_golang/prometheus"
+	"sneak.berlin/go/smallwebwaf/internal/requestlog"
+)
+
+// other is the label under which the countries outside the busiest are
+// counted.
+const other = "other"
+
+// countries are the metrics by the client's country, for requests whose
+// client's country is known. The topN busiest countries, by their requests
+// since the start, have series of their own, and the others are counted
+// under other, so that there are never more than topN + 1 series. A
+// country that drops out of the busiest loses its series, and its next
+// requests are counted under other; one that becomes one of them gets a
+// series that counts from then on. Each series therefore only ever goes
+// up.
+type countries struct {
+	topN int
+
+	requests      *prometheus.CounterVec
+	requestBytes  *prometheus.CounterVec
+	responseBytes *prometheus.CounterVec
+	// refused are the requests the country lists refused.
+	refused *prometheus.CounterVec
+
+	mu sync.Mutex
+	// seen is each country's requests since the start, by which the
+	// countries are ranked. GeoJS gives two-letter codes, so it holds at
+	// most a few hundred.
+	seen map[string]int64
+	// top are the countries with series of their own.
+	top map[string]bool
+}
+
+// newCountries returns the metrics by country, with series of their own
+// for the topN busiest countries.
+func newCountries(topN int) *countries {
+	byCountry := []string{"country"}
+
+	return &countries{
+		topN: topN,
+		requests: counterVec("smallwebwaf_country_requests_total",
+			"Requests, by the client's country.", byCountry),
+		requestBytes: counterVec("smallwebwaf_country_request_bytes_total",
+			"Request body bytes, by the client's country.", byCountry),
+		responseBytes: counterVec("smallwebwaf_country_response_bytes_total",
+			"Response body bytes, by the client's country.", byCountry),
+		refused: counterVec("smallwebwaf_country_list_refusals_total",
+			"Requests the country lists refused, by the client's country.",
+			byCountry),
+		seen: map[string]int64{},
+		top:  map[string]bool{},
+	}
+}
+
+// add counts a request from its log line, whose country is known.
+func (c *countries) add(line *requestlog.Line) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+
+	c.seen[line.Country]++
+
+	label := c.label(line.Country)
+	c.requests.WithLabelValues(label).Inc()
+	c.requestBytes.WithLabelValues(label).Add(float64(line.RequestBytes))
+	c.responseBytes.WithLabelValues(label).Add(float64(line.ResponseBytes))
+
+	if line.Action == requestlog.ActionCountryDenied {
+		c.refused.WithLabelValues(label).Inc()
+	}
+}
+
+// label returns the label a request from country is counted under: the
+// country while it is one of the busiest, other while it is not. A
+// country busier than the least busy of them takes its place, and that
+// country's series are dropped.
+func (c *countries) label(country string) string {
+	if c.top[country] {
+		return country
+	}
+
+	if len(c.top) < c.topN {
+		c.top[country] = true
+
+		return country
+	}
+
+	least := ""
+
+	for top := range c.top {
+		if least == "" || c.seen[top] < c.seen[least] {
+			least = top
+		}
+	}
+
+	if c.seen[country] <= c.seen[least] {
+		return other
+	}
+
+	delete(c.top, least)
+
+	for _, vec := range []*prometheus.CounterVec{
+		c.requests, c.requestBytes, c.responseBytes, c.refused,
+	} {
+		vec.DeleteLabelValues(least)
+	}
+
+	c.top[country] = true
+
+	return country
+}

internal/metrics/metrics.go

@@ -0,0 +1,277 @@
+// Package metrics keeps smallwebwaf's Prometheus metrics, as the "Metrics
+// endpoint" section of SPEC.md lists them, and serves them in the
+// Prometheus text format. No metric carries a client's address.
+package metrics
+
+import (
+	"net/http"
+	"strconv"
+	"time"
+
+	"github.com/prometheus/client_golang/prometheus"
+	"github.com/prometheus/client_golang/prometheus/collectors"
+	"github.com/prometheus/client_golang/prometheus/promhttp"
+	"sneak.berlin/go/smallwebwaf/internal/bans"
+	"sneak.berlin/go/smallwebwaf/internal/ratelimit"
+	"sneak.berlin/go/smallwebwaf/internal/requestlog"
+)
+
+// Metrics are smallwebwaf's metrics. They are safe for concurrent use.
+type Metrics struct {
+	registry *prometheus.Registry
+	handler  http.Handler
+
+	inFlight             prometheus.Gauge
+	requests             *prometheus.CounterVec
+	requestBytes         *prometheus.CounterVec
+	responseBytes        *prometheus.CounterVec
+	requestDuration      prometheus.Histogram
+	upstreamDuration     prometheus.Histogram
+	rateLimitHits        *prometheus.CounterVec
+	sizeAndTimeLimitHits *prometheus.CounterVec
+	offences             *prometheus.CounterVec
+	countries            *countries
+
+	// GeoJSRequests are the requests to GeoJS, and GeoJSFailures those
+	// that failed. GeoJSUnanswered are the requests whose client counted
+	// as coming from an unknown country because GeoJS had not answered
+	// about it in time.
+	GeoJSRequests   prometheus.Counter
+	GeoJSFailures   prometheus.Counter
+	GeoJSUnanswered prometheus.Counter
+
+	stateFileWrites        *prometheus.CounterVec
+	stateFileWriteFailures *prometheus.CounterVec
+	stateFileLastWrite     *prometheus.GaugeVec
+	stateFileSize          *prometheus.GaugeVec
+	stateFileEditsTakenIn  *prometheus.CounterVec
+	stateFileEditsSetAside *prometheus.CounterVec
+}
+
+// New returns the metrics, with the Go runtime's and the process's own.
+// topN is how many countries get series of their own
+// (SWWAF_METRICS_TOP_N).
+func New(topN int) *Metrics {
+	byStatus := []string{"status_class", "action"}
+	byFile := []string{"file"}
+
+	m := &Metrics{
+		registry: prometheus.NewRegistry(),
+		inFlight: prometheus.NewGauge(prometheus.GaugeOpts{
+			Name: "smallwebwaf_requests_in_flight",
+			Help: "Requests under way.",
+		}),
+		requests: counterVec("smallwebwaf_requests_total",
+			"Requests, by the class of their status and their action.", byStatus),
+		requestBytes: counterVec("smallwebwaf_request_bytes_total",
+			"Request body bytes, by the class of the status and the action.",
+			byStatus),
+		responseBytes: counterVec("smallwebwaf_response_bytes_total",
+			"Response body bytes, by the class of the status and the action.",
+			byStatus),
+		requestDuration: prometheus.NewHistogram(prometheus.HistogramOpts{
+			Name: "smallwebwaf_request_duration_seconds",
+			Help: "How long requests took, from their arrival to their end.",
+		}),
+		upstreamDuration: prometheus.NewHistogram(prometheus.HistogramOpts{
+			Name: "smallwebwaf_upstream_duration_seconds",
+			Help: "How long requests passed to the app took, from then to their end.",
+		}),
+		rateLimitHits: counterVec("smallwebwaf_rate_limit_hits_total",
+			"Requests that broke a rate limit, by its window.",
+			[]string{"window"}),
+		sizeAndTimeLimitHits: counterVec("smallwebwaf_size_and_time_limit_hits_total",
+			"Requests that passed a size or time limit, by its setting.",
+			[]string{"limit"}),
+		offences: counterVec("smallwebwaf_offences_total",
+			"Offences, by kind.", []string{"kind"}),
+		countries: newCountries(topN),
+		GeoJSRequests: prometheus.NewCounter(prometheus.CounterOpts{
+			Name: "smallwebwaf_geojs_requests_total",
+			Help: "Requests to GeoJS.",
+		}),
+		GeoJSFailures: prometheus.NewCounter(prometheus.CounterOpts{
+			Name: "smallwebwaf_geojs_failures_total",
+			Help: "Requests to GeoJS that failed.",
+		}),
+		GeoJSUnanswered: prometheus.NewCounter(prometheus.CounterOpts{
+			Name: "smallwebwaf_geojs_unanswered_total",
+			Help: "Requests whose client counted as coming from an unknown " +
+				"country because GeoJS had not answered about it in time.",
+		}),
+		stateFileWrites: counterVec("smallwebwaf_state_file_writes_total",
+			"Writes of each state file.", byFile),
+		stateFileWriteFailures: counterVec("smallwebwaf_state_file_write_failures_total",
+			"Writes of each state file that failed.", byFile),
+		stateFileLastWrite: gaugeVec("smallwebwaf_state_file_last_write_timestamp_seconds",
+			"When each state file was last written, in seconds since 1970.", byFile),
+		stateFileSize: gaugeVec("smallwebwaf_state_file_size_bytes",
+			"The size of each state file, as it was last written.", byFile),
+		stateFileEditsTakenIn: counterVec("smallwebwaf_state_file_edits_taken_in_total",
+			"Edits of each state file taken in while running.", byFile),
+		stateFileEditsSetAside: counterVec("smallwebwaf_state_file_edits_set_aside_total",
+			"Edits of each state file renamed to <name>.bad because they did not parse.",
+			byFile),
+	}
+
+	m.handler = promhttp.HandlerFor(m.registry, promhttp.HandlerOpts{})
+
+	m.registry.MustRegister(
+		collectors.NewGoCollector(),
+		collectors.NewProcessCollector(collectors.ProcessCollectorOpts{}),
+		m.inFlight, m.requests, m.requestBytes, m.responseBytes,
+		m.requestDuration, m.upstreamDuration,
+		m.rateLimitHits, m.sizeAndTimeLimitHits, m.offences,
+		m.countries.requests, m.countries.requestBytes, m.countries.responseBytes,
+		m.countries.refused,
+		m.GeoJSRequests, m.GeoJSFailures, m.GeoJSUnanswered,
+		m.stateFileWrites, m.stateFileWriteFailures,
+		m.stateFileLastWrite, m.stateFileSize,
+		m.stateFileEditsTakenIn, m.stateFileEditsSetAside,
+	)
+
+	return m
+}
+
+// AddBansAndClients adds the metrics read from the ledger and the table
+// of clients as the metrics are asked for: the bans made since the start,
+// the bans active and permanent at now, and the clients in the table.
+func (m *Metrics) AddBansAndClients(
+	ledger *bans.Ledger, limiter *ratelimit.Limiter, now func() time.Time,
+) {
+	m.registry.MustRegister(
+		// Every ban smallwebwaf makes so far is for a broken limit.
+		prometheus.NewCounterFunc(prometheus.CounterOpts{
+			Name:        "smallwebwaf_bans_made_total",
+			Help:        "Bans made, by cause.",
+			ConstLabels: prometheus.Labels{"cause": "limit"},
+		}, func() float64 {
+			return float64(ledger.Made())
+		}),
+		prometheus.NewGaugeFunc(prometheus.GaugeOpts{
+			Name: "smallwebwaf_active_bans",
+			Help: "Bans active now, the permanent ones included.",
+		}, func() float64 {
+			active, _ := ledger.Count(now())
+
+			return float64(active)
+		}),
+		prometheus.NewGaugeFunc(prometheus.GaugeOpts{
+			Name: "smallwebwaf_permanent_bans",
+			Help: "Permanent bans.",
+		}, func() float64 {
+			_, permanent := ledger.Count(now())
+
+			return float64(permanent)
+		}),
+		prometheus.NewGaugeFunc(prometheus.GaugeOpts{
+			Name: "smallwebwaf_tracked_clients",
+			Help: "Clients in the table of clients.",
+		}, func() float64 {
+			return float64(limiter.Len())
+		}),
+	)
+}
+
+// ServeHTTP answers with the metrics in the Prometheus text format.
+func (m *Metrics) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+	m.handler.ServeHTTP(w, r)
+}
+
+// RequestStarted counts a request as under way.
+func (m *Metrics) RequestStarted() {
+	m.inFlight.Inc()
+}
+
+// RequestEnded counts a request that has ended, from its log line. limit
+// is the setting whose size or time limit the request passed, "" if none.
+// duration is how long the request took, and upstreamDuration how long it
+// took from when it was passed to the app, zero if it was not.
+func (m *Metrics) RequestEnded(
+	line *requestlog.Line, limit string, duration, upstreamDuration time.Duration,
+) {
+	m.inFlight.Dec()
+
+	class := statusClass(line.Status)
+	m.requests.WithLabelValues(class, line.Action).Inc()
+	m.requestBytes.WithLabelValues(class, line.Action).Add(float64(line.RequestBytes))
+	m.responseBytes.WithLabelValues(class, line.Action).Add(float64(line.ResponseBytes))
+	m.requestDuration.Observe(duration.Seconds())
+
+	if upstreamDuration > 0 {
+		m.upstreamDuration.Observe(upstreamDuration.Seconds())
+	}
+
+	if line.LimitHit != "" {
+		m.rateLimitHits.WithLabelValues(line.LimitHit).Inc()
+	}
+
+	if limit != "" {
+		m.sizeAndTimeLimitHits.WithLabelValues(limit).Inc()
+	}
+
+	if line.Offence != "" {
+		m.offences.WithLabelValues(line.Offence).Inc()
+	}
+
+	if line.Country != "" {
+		m.countries.add(line)
+	}
+}
+
+// StateFileWritten counts a write of the state file name, of size bytes,
+// that ended with err.
+func (m *Metrics) StateFileWritten(name string, size int, err error) {
+	m.stateFileWrites.WithLabelValues(name).Inc()
+
+	// The series of failures is there from the first write, at zero until
+	// one fails.
+	failures := m.stateFileWriteFailures.WithLabelValues(name)
+
+	if err != nil {
+		failures.Inc()
+
+		return
+	}
+
+	m.stateFileLastWrite.WithLabelValues(name).SetToCurrentTime()
+	m.stateFileSize.WithLabelValues(name).Set(float64(size))
+}
+
+// StateFileEditTakenIn counts an admin's edit of the state file name
+// taken in while smallwebwaf runs.
+func (m *Metrics) StateFileEditTakenIn(name string) {
+	m.stateFileEditsTakenIn.WithLabelValues(name).Inc()
+}
+
+// StateFileEditSetAside counts an admin's edit of the state file name
+// renamed to name.bad because it did not parse.
+func (m *Metrics) StateFileEditSetAside(name string) {
+	m.stateFileEditsSetAside.WithLabelValues(name).Inc()
+}
+
+// statusClass returns the class of status, such as 2xx, or none when no
+// status was sent.
+func statusClass(status int) string {
+	if status == 0 {
+		return "none"
+	}
+
+	// A status's class is its hundreds: 404 is in 4xx.
+	const hundred = 100
+
+	return strconv.Itoa(status/hundred) + "xx"
+}
+
+// counterVec returns a counter named name, described by help, with a
+// series for each set of values of labels.
+func counterVec(name, help string, labels []string) *prometheus.CounterVec {
+	return prometheus.NewCounterVec(prometheus.CounterOpts{Name: name, Help: help},
+		labels)
+}
+
+// gaugeVec returns a gauge named name, described by help, with a series
+// for each set of values of labels.
+func gaugeVec(name, help string, labels []string) *prometheus.GaugeVec {
+	return prometheus.NewGaugeVec(prometheus.GaugeOpts{Name: name, Help: help}, labels)
+}

internal/proxy/admin.go

@@ -0,0 +1,43 @@
+package proxy
+
+import (
+	"crypto/subtle"
+	"net/http"
+	"strings"
+
+	"sneak.berlin/go/smallwebwaf/internal/requestlog"
+)
+
+// answerAdmin answers a request for smallwebwaf itself, under
+// /_smallwebwaf/, once it has passed the checks: GET MetricsPath with
+// SWWAF_METRICS_TOKEN gets the metrics, and without it is refused with
+// 401. Any other request gets 404, as the metrics do while
+// SWWAF_METRICS_TOKEN is unset.
+func (rq *request) answerAdmin() {
+	rq.line.Action = requestlog.ActionAdmin
+	rq.startClientResponseTimeout()
+
+	token := rq.h.config.MetricsToken
+
+	switch {
+	case token == "" || rq.in.Method != http.MethodGet || rq.in.URL.Path != MetricsPath:
+		http.Error(rq.out, http.StatusText(http.StatusNotFound), http.StatusNotFound)
+	case !hasToken(rq.in, token):
+		rq.out.Header().Set("WWW-Authenticate", "Bearer")
+		rq.answer(refusal{
+			status: http.StatusUnauthorized,
+			action: requestlog.ActionAdmin,
+		})
+	default:
+		rq.h.metrics.ServeHTTP(rq.out, rq.in)
+	}
+}
+
+// hasToken reports whether r carries token, as Authorization: Bearer
+// <token>.
+func hasToken(r *http.Request, token string) bool {
+	scheme, sent, _ := strings.Cut(r.Header.Get("Authorization"), " ")
+
+	return strings.EqualFold(scheme, "Bearer") &&
+		subtle.ConstantTimeCompare([]byte(sent), []byte(token)) == 1
+}

internal/proxy/bans.go

@@ -14,10 +14,15 @@ func (rq *request) banResponse(action string) *refusal {
 	return &refusal{status: rq.h.config.BanResponse, action: action}
 }
 
-// banned reports whether a ban on the client's netblock refuses the
+// banned reports whether a ban on a netblock the client is in covers the
 // request at now, and notes for the log line when that ban ends.
 func (rq *request) banned(now time.Time) bool {
-	ban, banned := rq.h.ledger.Check(rq.netblock(), now)
+	check := rq.h.ledger.Check
+	if rq.h.config.Observe {
+		check = rq.h.ledger.Find // in observe mode the ban refuses nothing
+	}
+
+	ban, banned := check(rq.client, now)
 	if banned {
 		rq.line.BanExpires = banExpires(ban)
 	}
@@ -25,17 +30,28 @@ func (rq *request) banned(now time.Time) bool {
 	return banned
 }
 
-// limitBroken counts the request for the rate limits at now, and reports
-// whether it takes the client over one. Such a request bans the client's
-// netblock, and sets the client's counters back to zero.
+// limitBroken counts the request for the rate limits at now, notes the
+// client's counts for the log line, and reports whether the request takes
+// the client over a limit. In enforce mode such a request bans the
+// client's netblock, and sets the client's counters back to zero; in
+// observe mode it does neither.
 func (rq *request) limitBroken(now time.Time) bool {
 	group := clientGroup(rq.client)
 
-	hit, over := rq.h.limiter.Count(group, now)
+	counts, hit, over := rq.h.limiter.Count(group, now)
+	rq.line.Counts = counts
+
 	if !over {
 		return false
 	}
 
+	rq.line.LimitHit = hit.Window
+	rq.line.Offence = requestlog.OffenceLimit
+
+	if rq.h.config.Observe {
+		return true
+	}
+
 	netblock := rq.netblock()
 	ban := rq.h.ledger.BanForLimit(netblock, now, bans.Notes{
 		Country: rq.line.Country,
@@ -54,9 +70,6 @@ func (rq *request) limitBroken(now time.Time) bool {
 		Requests: rq.h.limiter.Requests(netblock) + 1,
 	})
 	rq.h.limiter.Reset(group)
-
-	rq.line.LimitHit = hit.Window
-	rq.line.Offence = requestlog.OffenceLimit
 	rq.line.BanExpires = banExpires(ban)
 
 	return true

internal/proxy/bans_test.go

@@ -402,36 +402,54 @@ func (s *sender) get(from string, status int, action string) logLine {
 func (s *sender) request(from, path string, status int, action string) logLine {
 	s.t.Helper()
 
+	line, _ := s.requestWithHeader(from, path, "", status, action)
+
+	return line
+}
+
+// requestWithHeader is request with header, such as "Authorization:
+// Bearer x", added to the request unless it is "". It returns the body of
+// the answer too.
+func (s *sender) requestWithHeader(
+	from, path, header string, status int, action string,
+) (logLine, string) {
+	s.t.Helper()
+
+	if header != "" {
+		header += "\r\n"
+	}
+
 	conn := dial(s.t, s.addr)
 	send(s.t, conn, "GET "+path+" HTTP/1.1\r\nHost: "+appHost+
-		"\r\nUser-Agent: "+userAgent+"\r\n"+forwardedFor+": "+from+"\r\n\r\n")
+		"\r\nUser-Agent: "+userAgent+"\r\n"+forwardedFor+": "+from+"\r\n"+
+		header+"\r\n")
 
 	err := conn.SetReadDeadline(time.Now().Add(waitLimit))
 	if err != nil {
 		s.t.Fatalf("set read deadline: %v", err)
 	}
 
-	got := 0
+	var got answer
 
 	res, err := http.ReadResponse(bufio.NewReader(conn), nil)
 
 	switch {
 	case err == nil:
-		got = readAnswer(res).status
+		got = readAnswer(res)
 	case !errors.Is(err, io.ErrUnexpectedEOF):
 		s.t.Fatalf("read response: %v", err)
 	}
 
 	_ = conn.Close()
 
-	if got != status {
-		s.t.Errorf("request %d, from %s: status %d, want %d", s.sent+1, from, got,
-			status)
+	if got.status != status {
+		s.t.Errorf("request %d, from %s: status %d, want %d", s.sent+1, from,
+			got.status, status)
 	}
 
 	line := s.out.requestLines(s.t, s.sent+1)[s.sent]
 	s.sent++
 	wantLine(s.t, line, status, action)
 
-	return line
+	return line, string(got.body)
 }

internal/proxy/bodies.go

@@ -46,6 +46,7 @@ func (b *requestBody) Read(p []byte) (int, error) {
 		b.rq.refuse(refusal{
 			status: http.StatusRequestEntityTooLarge,
 			action: requestlog.ActionTooLarge,
+			limit:  "SWWAF_REQUEST_MAX_BYTES",
 		})
 	}
 
@@ -81,6 +82,7 @@ func (b *responseBody) Read(p []byte) (int, error) {
 		b.rq.refuse(refusal{
 			status: http.StatusBadGateway,
 			action: requestlog.ActionTooLarge,
+			limit:  "SWWAF_RESPONSE_MAX_BYTES",
 		})
 
 		return n, errResponseTooLarge

internal/proxy/client.go

@@ -1,6 +1,7 @@
 package proxy
 
 import (
+	"crypto/rand"
 	"net/http"
 	"net/netip"
 	"slices"
@@ -48,6 +49,33 @@ func clientAddress(
 	return client
 }
 
+// requestIDHeader carries the request's id, from traefik and to the app.
+const requestIDHeader = "X-Request-ID"
+
+// requestID is the request's id: the one a trusted proxy sent, or a new
+// random one. A peer outside the trusted proxies did not come through
+// traefik, so the id it sends is its own claim, and is replaced.
+func requestID(r *http.Request, peerTrusted bool) string {
+	id := r.Header.Get(requestIDHeader)
+	if !peerTrusted || id == "" {
+		id = rand.Text()
+	}
+
+	return id
+}
+
+// scheme is how the client reached traefik, as a trusted proxy says in
+// X-Forwarded-Proto, or otherwise http, the only scheme smallwebwaf
+// serves.
+func scheme(r *http.Request, peerTrusted bool) string {
+	proto := r.Header.Get("X-Forwarded-Proto")
+	if !peerTrusted || proto == "" {
+		return "http"
+	}
+
+	return proto
+}
+
 // ipv6GroupPrefix is the length of the IPv6 netblock that is one client.
 const ipv6GroupPrefix = 64
 

internal/proxy/client_test.go

@@ -14,10 +14,14 @@ const (
 	appHost = "app.example"
 	// client is the client's address, as a proxy names it.
 	client = "203.0.113.9"
-	// forwardedFor is the header that lists the client and its proxies.
-	forwardedFor = "X-Forwarded-For"
-	// secure is the scheme a client reached traefik with.
+	// forwardedFor is the header that lists the client and its proxies,
+	// and forwardedProto the one that gives the scheme the client used.
+	forwardedFor   = "X-Forwarded-For"
+	forwardedProto = "X-Forwarded-Proto"
+	// secure is the scheme a client reached traefik with, and plain the
+	// one smallwebwaf serves.
 	secure = "https"
+	plain  = "http"
 )
 
 // appHeaders is what the app tells about the headers it received.
@@ -65,13 +69,13 @@ func TestClientAddressAndForwardedHeaders(t *testing.T) {
 func clientAddressCases() []clientAddressCase {
 	trusted := map[string]string{trustedProxies: trustLocalhost}
 	forged := http.Header{
-		forwardedFor:        {client},
-		"X-Forwarded-Host":  {"forged.example"},
-		"X-Forwarded-Proto": {secure},
-		"X-Real-Ip":         {client},
+		forwardedFor:       {client},
+		"X-Forwarded-Host": {"forged.example"},
+		forwardedProto:     {secure},
+		"X-Real-Ip":        {client},
 	}
 	replaced := appHeaders{
-		ForwardedFor: localhost, ForwardedHost: appHost, ForwardedProto: "http",
+		ForwardedFor: localhost, ForwardedHost: appHost, ForwardedProto: plain,
 	}
 
 	return []clientAddressCase{{
@@ -87,10 +91,10 @@ func clientAddressCases() []clientAddressCase {
 			"outside the trusted proxies from the right",
 		env: trusted,
 		header: http.Header{
-			forwardedFor:        {"198.51.100.7, " + client + ", 10.0.0.2"},
-			"X-Forwarded-Host":  {appHost},
-			"X-Forwarded-Proto": {secure},
-			"X-Real-Ip":         {client},
+			forwardedFor:       {"198.51.100.7, " + client + ", 10.0.0.2"},
+			"X-Forwarded-Host": {appHost},
+			forwardedProto:     {secure},
+			"X-Real-Ip":        {client},
 		},
 		wantClient: client,
 		wantApp: appHeaders{
@@ -138,7 +142,7 @@ func requestWithHeaders(
 			Host:           r.Host,
 			ForwardedFor:   r.Header.Get(forwardedFor),
 			ForwardedHost:  r.Header.Get("X-Forwarded-Host"),
-			ForwardedProto: r.Header.Get("X-Forwarded-Proto"),
+			ForwardedProto: r.Header.Get(forwardedProto),
 			RealIP:         r.Header.Get("X-Real-IP"),
 		})
 	})

internal/proxy/health_test.go

@@ -21,14 +21,18 @@ func TestHealthEndpointIsAnsweredBeforeAnyCheck(t *testing.T) {
 	// the last one would have it refused.
 	addr, out := startProxy(t, app.URL, map[string]string{rateLimitPerMinute: "1"})
 
-	const healthChecks = 3
+	const (
+		healthChecks = 3
+		contentType  = "text/plain; charset=utf-8"
+	)
 
 	for range healthChecks {
 		got := get(t, addr, proxy.HealthPath)
 		wantStatus(t, got, http.StatusOK)
 
-		if string(got.body) != "ok\n" {
-			t.Errorf("health endpoint answered %q, want ok", got.body)
+		if string(got.body) != "ok\n" || got.header.Get("Content-Type") != contentType {
+			t.Errorf("health endpoint answered %q with Content-Type %q, want ok "+
+				"with %q", got.body, got.header.Get("Content-Type"), contentType)
 		}
 	}
 
@@ -37,6 +41,11 @@ func TestHealthEndpointIsAnsweredBeforeAnyCheck(t *testing.T) {
 	lines := out.requestLines(t, healthChecks+1)
 	for _, line := range lines[:healthChecks] {
 		wantLine(t, line, http.StatusOK, requestlog.ActionAdmin)
+
+		if line.ResponseContentType != contentType {
+			t.Errorf("health check's log line has response_content_type %q, "+
+				"want %q", line.ResponseContentType, contentType)
+		}
 	}
 
 	wantLine(t, lines[healthChecks], http.StatusOK, requestlog.ActionForward)

internal/proxy/history_test.go

@@ -86,6 +86,27 @@ func TestHealthEndpointIsNotInTheHistory(t *testing.T) {
 	}
 }
 
+func TestRequestForSmallwebwafIsRefusedOnlyWithoutTheToken(t *testing.T) {
+	t.Parallel()
+
+	app := startApp(t, func(http.ResponseWriter, *http.Request) {})
+	addr, out, server := startProxyWithClock(t, app.URL, "", time.Now,
+		map[string]string{metricsToken: token})
+
+	// The metrics and the 404 are neither forwarded nor refused; the 401
+	// is refused.
+	scrape(t, addr)
+	wantStatus(t, get(t, addr, "/_smallwebwaf/nothing"), http.StatusNotFound)
+	wantStatus(t, get(t, addr, proxy.MetricsPath), http.StatusUnauthorized)
+	out.requestLines(t, 3)
+
+	history := historyOf(t, server, localhost)
+	if history.Requests != 3 || history.Forwarded != 0 || history.Refused != 1 {
+		t.Errorf("history counts %d requests, %d forwarded and %d refused, "+
+			"want 3, 0 and 1", history.Requests, history.Forwarded, history.Refused)
+	}
+}
+
 // historyOf returns the history of the client at addr.
 func historyOf(t *testing.T, server *proxy.Server, addr string) ratelimit.History {
 	t.Helper()

internal/proxy/limits_test.go

@@ -55,6 +55,7 @@ func TestRequestBodyLimit(t *testing.T) {
 			})
 			addr, out := startProxy(t, app.URL, map[string]string{
 				requestMaxBytes: sizeLimitSetting,
+				metricsToken:    token,
 			})
 
 			var body io.Reader = bytes.NewReader(make([]byte, tc.size))
@@ -66,6 +67,13 @@ func TestRequestBodyLimit(t *testing.T) {
 				tc.want)
 			wantLine(t, out.requestLine(t), tc.want, tc.action)
 
+			hits := 0
+			if tc.action == requestlog.ActionTooLarge {
+				hits = 1
+			}
+
+			wantLimitHits(t, addr, requestMaxBytes, hits)
+
 			if tc.refusedBeforeApp && calls.Load() != 0 {
 				t.Errorf("the app was called %d times, want never", calls.Load())
 			}
@@ -106,6 +114,7 @@ func TestResponseBodyLimit(t *testing.T) {
 			})
 			addr, out := startProxy(t, app.URL, map[string]string{
 				responseMaxBytes: sizeLimitSetting,
+				metricsToken:     token,
 			})
 
 			got := get(t, addr, "/download")
@@ -123,6 +132,13 @@ func TestResponseBodyLimit(t *testing.T) {
 			if line.UpstreamStatus != http.StatusOK {
 				t.Errorf("log line has upstream_status %d", line.UpstreamStatus)
 			}
+
+			hits := 0
+			if tc.action == requestlog.ActionTooLarge {
+				hits = 1
+			}
+
+			wantLimitHits(t, addr, responseMaxBytes, hits)
 		})
 	}
 }

internal/proxy/metrics_test.go

@@ -0,0 +1,455 @@
+package proxy_test
+
+import (
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"net/netip"
+	"strconv"
+	"strings"
+	"sync"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"sneak.berlin/go/smallwebwaf/internal/lookup"
+	"sneak.berlin/go/smallwebwaf/internal/proxy"
+	"sneak.berlin/go/smallwebwaf/internal/requestlog"
+)
+
+const (
+	metricsToken = "SWWAF_METRICS_TOKEN" //nolint:gosec // the setting's name
+	metricsTopN  = "SWWAF_METRICS_TOP_N"
+	// token is the SWWAF_METRICS_TOKEN the tests set, and bearer how a
+	// request carries it.
+	token  = "0123456789abcdef0123456789abcdef"
+	bearer = "Bearer " + token
+)
+
+func TestMetricsAreOffWhileTheTokenIsUnset(t *testing.T) {
+	t.Parallel()
+
+	var calls atomic.Int32
+
+	app := startApp(t, func(http.ResponseWriter, *http.Request) {
+		calls.Add(1)
+	})
+	addr, out := startProxy(t, app.URL, nil)
+
+	// An empty token does not match the unset one either.
+	for i, authorization := range []string{bearer, "Bearer ", ""} {
+		req := newRequest(t, http.MethodGet, addr, proxy.MetricsPath, http.NoBody)
+		if authorization != "" {
+			req.Header.Set("Authorization", authorization)
+		}
+
+		wantStatus(t, do(t, req), http.StatusNotFound)
+		wantLine(t, out.requestLines(t, i+1)[i], http.StatusNotFound,
+			requestlog.ActionAdmin)
+	}
+
+	if calls.Load() != 0 {
+		t.Errorf("the app was called %d times, want never", calls.Load())
+	}
+}
+
+func TestMetricsNeedTheTokenAndOtherPathsAreNotFound(t *testing.T) {
+	t.Parallel()
+
+	var calls atomic.Int32
+
+	app := startApp(t, func(http.ResponseWriter, *http.Request) {
+		calls.Add(1)
+	})
+	addr, out := startProxy(t, app.URL, map[string]string{metricsToken: token})
+
+	for i, tc := range []struct {
+		method, path, authorization string
+		status                      int
+	}{
+		{http.MethodGet, proxy.MetricsPath, "", http.StatusUnauthorized},
+		{
+			http.MethodGet, proxy.MetricsPath, "Bearer " + strings.ToUpper(token),
+			http.StatusUnauthorized,
+		},
+		{http.MethodGet, proxy.MetricsPath, "Basic " + token, http.StatusUnauthorized},
+		{http.MethodGet, proxy.MetricsPath, bearer, http.StatusOK},
+		{http.MethodGet, proxy.MetricsPath, "bearer " + token, http.StatusOK},
+		{http.MethodPost, proxy.MetricsPath, bearer, http.StatusNotFound},
+		{http.MethodGet, proxy.MetricsPath + "/", bearer, http.StatusNotFound},
+		{http.MethodGet, "/_smallwebwaf/bans", bearer, http.StatusNotFound},
+		{http.MethodPost, proxy.HealthPath, "", http.StatusNotFound},
+	} {
+		req := newRequest(t, tc.method, addr, tc.path, http.NoBody)
+		if tc.authorization != "" {
+			req.Header.Set("Authorization", tc.authorization)
+		}
+
+		got := do(t, req)
+		wantStatus(t, got, tc.status)
+		wantLine(t, out.requestLines(t, i+1)[i], tc.status, requestlog.ActionAdmin)
+
+		if tc.status == http.StatusUnauthorized &&
+			got.header.Get("WWW-Authenticate") != "Bearer" {
+			t.Errorf("%q was answered without WWW-Authenticate: Bearer",
+				tc.authorization)
+		}
+
+		if tc.status == http.StatusOK &&
+			!strings.Contains(string(got.body), "# TYPE smallwebwaf_requests_total counter") {
+			t.Errorf("the metrics are\n%s", got.body)
+		}
+	}
+
+	if calls.Load() != 0 {
+		t.Errorf("the app was called %d times, want never", calls.Load())
+	}
+}
+
+func TestMetricsAreAskedForThroughTheChecks(t *testing.T) {
+	t.Parallel()
+
+	s, _, _ := startWithClock(t, "", map[string]string{
+		metricsToken:       token,
+		rateLimitPerMinute: "1",
+	})
+
+	// Asking for the metrics counts toward the client's limit of one
+	// request a minute, so its next request breaks it, and bans it. A
+	// banned client is refused the metrics too.
+	s.scrape(client)
+	s.get(client, http.StatusForbidden, requestlog.ActionRateLimited)
+	s.requestWithHeader(client, proxy.MetricsPath, "Authorization: "+bearer,
+		http.StatusForbidden, requestlog.ActionBanned)
+}
+
+func TestMetricsCountTheTraffic(t *testing.T) {
+	t.Parallel()
+
+	arrived, release := make(chan struct{}), make(chan struct{})
+	app := startApp(t, func(w http.ResponseWriter, r *http.Request) {
+		_, _ = io.Copy(io.Discard, r.Body)
+
+		if r.URL.Path == "/held" {
+			close(arrived)
+			<-release
+		}
+
+		_, _ = io.WriteString(w, "hello")
+	})
+	releaseApp := sync.OnceFunc(func() { close(release) })
+	t.Cleanup(releaseApp)
+
+	addr, out := startProxy(t, app.URL, map[string]string{metricsToken: token})
+
+	got := do(t, newRequest(t, http.MethodPost, addr, "/", strings.NewReader("abc")))
+	wantStatus(t, got, http.StatusOK)
+	wantStatus(t, get(t, addr, "/_smallwebwaf/nothing"), http.StatusNotFound)
+	out.requestLines(t, 2)
+
+	forward := `{action="forward",status_class="2xx"}`
+	notFound := `{action="admin",status_class="4xx"}`
+
+	// The request for the metrics is itself under way.
+	metrics := scrape(t, addr)
+	wantMetric(t, metrics, "smallwebwaf_requests_total"+forward, 1)
+	wantMetric(t, metrics, "smallwebwaf_requests_total"+notFound, 1)
+	wantMetric(t, metrics, "smallwebwaf_request_bytes_total"+forward, 3)
+	wantMetric(t, metrics, "smallwebwaf_response_bytes_total"+forward, 5)
+	wantMetric(t, metrics, "smallwebwaf_response_bytes_total"+notFound,
+		float64(len("Not Found\n")))
+	wantMetric(t, metrics, "smallwebwaf_request_duration_seconds_count", 2)
+	wantMetric(t, metrics, "smallwebwaf_upstream_duration_seconds_count", 1)
+	wantMetric(t, metrics, "smallwebwaf_requests_in_flight", 1)
+	metric(t, metrics, "go_goroutines")
+	metric(t, metrics, "process_start_time_seconds")
+
+	// A request the app holds is under way until it ends.
+	httpClient := newClient(t)
+	held := newRequest(t, http.MethodGet, addr, "/held", http.NoBody)
+	ended := make(chan error, 1)
+
+	go func() {
+		res, err := httpClient.Do(held)
+		if err == nil {
+			err = readAnswer(res).err
+		}
+
+		ended <- err
+	}()
+
+	<-arrived
+	wantMetric(t, scrape(t, addr), "smallwebwaf_requests_in_flight", 2)
+	releaseApp()
+
+	err := <-ended
+	if err != nil {
+		t.Fatalf("held request: %v", err)
+	}
+
+	out.requestLines(t, 5)
+	wantMetric(t, scrape(t, addr), "smallwebwaf_requests_in_flight", 1)
+}
+
+func TestMetricsCountLimitsAndBans(t *testing.T) {
+	t.Parallel()
+
+	const (
+		scraper = "192.0.2.200" // in SWWAF_RATE_LIMIT_EXEMPT_NETS
+		denied  = "192.0.2.50"  // in SWWAF_DENY_NETS
+	)
+
+	s, clk, _ := startWithClock(t, "", map[string]string{
+		metricsToken:        token,
+		rateLimitPerMinute:  "1",
+		rateLimitExemptNets: scraper,
+		denyNets:            denied,
+		banResponse:         "close",
+		limitBanDuration:    "1h",
+		maxBanDuration:      "2h",
+	})
+
+	// SWWAF_BAN_RESPONSE=close sends no status at all.
+	s.get(denied, 0, requestlog.ActionDenied)
+
+	// A first broken limit bans for an hour.
+	s.get(client, http.StatusOK, requestlog.ActionForward)
+	s.get(client, 0, requestlog.ActionRateLimited)
+
+	metrics := s.scrape(scraper)
+	wantMetric(t, metrics,
+		`smallwebwaf_requests_total{action="denied",status_class="none"}`, 1)
+	wantMetric(t, metrics, `smallwebwaf_rate_limit_hits_total{window="minute"}`, 1)
+	wantMetric(t, metrics, `smallwebwaf_offences_total{kind="limit"}`, 1)
+	wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="limit"}`, 1)
+	wantMetric(t, metrics, "smallwebwaf_active_bans", 1)
+	wantMetric(t, metrics, "smallwebwaf_permanent_bans", 0)
+
+	clk.advance(time.Hour)
+	wantMetric(t, s.scrape(scraper), "smallwebwaf_active_bans", 0)
+
+	// A limit broken again right after would ban for three hours, longer
+	// than SWWAF_MAX_BAN_DURATION, so the ban is permanent.
+	s.get(client, http.StatusOK, requestlog.ActionForward)
+	s.get(client, 0, requestlog.ActionRateLimited)
+
+	metrics = s.scrape(scraper)
+	wantMetric(t, metrics, `smallwebwaf_rate_limit_hits_total{window="minute"}`, 2)
+	wantMetric(t, metrics, `smallwebwaf_offences_total{kind="limit"}`, 2)
+	wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="limit"}`, 2)
+	wantMetric(t, metrics, "smallwebwaf_active_bans", 1)
+	wantMetric(t, metrics, "smallwebwaf_permanent_bans", 1)
+	// denied, client, and the scraper as of its earlier requests.
+	wantMetric(t, metrics, "smallwebwaf_tracked_clients", 3)
+}
+
+func TestMetricsByCountryKeepTheBusiestAndCountTheRestAsOther(t *testing.T) {
+	t.Parallel()
+
+	const fromFR = "198.51.100.20"
+
+	app := startApp(t, func(w http.ResponseWriter, r *http.Request) {
+		_, _ = io.Copy(io.Discard, r.Body)
+		_, _ = io.WriteString(w, "hello")
+	})
+	env := map[string]string{
+		trustedProxies:  trustLocalhost,
+		metricsToken:    token,
+		metricsTopN:     "2",
+		deniedCountries: "kp",
+	}
+	addr, out, server := startProxyWithClock(t, app.URL, "", time.Now, env)
+
+	// The answers are kept before the requests, so that none waits for
+	// GeoJS.
+	server.GeoJS.Load([]lookup.Answer{
+		keptAnswer(fromKP, "KP"), keptAnswer(fromDE, "DE"), keptAnswer(fromFR, "FR"),
+	})
+
+	lines := 0
+	send := func(from string, times, status int) {
+		t.Helper()
+
+		for range times {
+			req := newRequest(t, http.MethodPost, addr, "/", strings.NewReader("abc"))
+			req.Header.Set(forwardedFor, from)
+			wantStatus(t, do(t, req), status)
+
+			// Each is counted before the next is sent, so that the
+			// countries are ranked in the order sent.
+			lines++
+			out.requestLines(t, lines)
+		}
+	}
+
+	// With two countries of their own, the third is counted as other.
+	send(fromKP, 3, http.StatusForbidden)
+	send(fromDE, 2, http.StatusOK)
+	send(fromFR, 1, http.StatusOK)
+
+	metrics := scrape(t, addr)
+	lines++
+
+	wantMetric(t, metrics, `smallwebwaf_country_requests_total{country="KP"}`, 3)
+	wantMetric(t, metrics, `smallwebwaf_country_requests_total{country="DE"}`, 2)
+	wantMetric(t, metrics, `smallwebwaf_country_requests_total{country="other"}`, 1)
+	wantMetric(t, metrics, `smallwebwaf_country_list_refusals_total{country="KP"}`, 3)
+	wantMetric(t, metrics, `smallwebwaf_country_request_bytes_total{country="KP"}`, 0)
+	wantMetric(t, metrics, `smallwebwaf_country_request_bytes_total{country="DE"}`, 6)
+	wantMetric(t, metrics, `smallwebwaf_country_response_bytes_total{country="KP"}`,
+		float64(3*len("Forbidden\n")))
+	wantMetric(t, metrics, `smallwebwaf_country_response_bytes_total{country="other"}`,
+		float64(len("hello")))
+	wantNoSeries(t, metrics, `smallwebwaf_country_requests_total{country="FR"}`)
+
+	// Once FR is busier than DE, it takes DE's place: its series counts
+	// from then on, and DE's is gone.
+	send(fromFR, 3, http.StatusOK)
+
+	metrics = scrape(t, addr)
+	wantMetric(t, metrics, `smallwebwaf_country_requests_total{country="KP"}`, 3)
+	wantMetric(t, metrics, `smallwebwaf_country_requests_total{country="FR"}`, 2)
+	wantMetric(t, metrics, `smallwebwaf_country_requests_total{country="other"}`, 2)
+	wantNoSeries(t, metrics, `smallwebwaf_country_requests_total{country="DE"}`)
+	wantNoSeries(t, metrics, `smallwebwaf_country_request_bytes_total{country="DE"}`)
+}
+
+func TestMetricsCountGeoJSRequestsAndFailures(t *testing.T) {
+	t.Parallel()
+
+	geojs := httptest.NewServer(http.HandlerFunc(
+		func(w http.ResponseWriter, _ *http.Request) {
+			w.WriteHeader(http.StatusServiceUnavailable)
+		}))
+	t.Cleanup(geojs.Close)
+
+	app := startApp(t, func(http.ResponseWriter, *http.Request) {})
+	addr, _ := startProxyWithGeoJS(t, app.URL, geojs.URL, map[string]string{
+		trustedProxies:  trustLocalhost,
+		metricsToken:    token,
+		deniedCountries: "kp",
+	})
+
+	// GeoJS fails, so the client counts as coming from an unknown country,
+	// which SWWAF_DENIED_COUNTRIES does not refuse.
+	req := newRequest(t, http.MethodGet, addr, "/", http.NoBody)
+	req.Header.Set(forwardedFor, fromDE)
+	wantStatus(t, do(t, req), http.StatusOK)
+
+	// The client stops waiting for GeoJS after a second, so GeoJS's
+	// failure can come after its request has ended.
+	deadline := time.Now().Add(waitLimit)
+	metrics := scrape(t, addr)
+
+	for metric(t, metrics, "smallwebwaf_geojs_failures_total") == 0 &&
+		time.Now().Before(deadline) {
+		time.Sleep(pollInterval)
+
+		metrics = scrape(t, addr)
+	}
+
+	wantMetric(t, metrics, "smallwebwaf_geojs_requests_total", 1)
+	wantMetric(t, metrics, "smallwebwaf_geojs_failures_total", 1)
+	wantMetric(t, metrics, "smallwebwaf_geojs_unanswered_total", 1)
+}
+
+// keptAnswer returns GeoJS's answer that the client at addr is in
+// country, given now.
+func keptAnswer(addr, country string) lookup.Answer {
+	now := time.Now()
+
+	return lookup.Answer{
+		Client: netip.MustParsePrefix(addr + "/32"), Country: country,
+		Answered: now, Used: now,
+	}
+}
+
+// scrape asks smallwebwaf at addr for the metrics, with the token, and
+// returns them.
+func scrape(t *testing.T, addr string) string {
+	t.Helper()
+
+	req := newRequest(t, http.MethodGet, addr, proxy.MetricsPath, http.NoBody)
+	req.Header.Set("Authorization", bearer)
+
+	got := do(t, req)
+	if got.status != http.StatusOK {
+		t.Fatalf("the metrics were answered %d", got.status)
+	}
+
+	return string(got.body)
+}
+
+// scrape asks for the metrics, with the token, from the client at from,
+// and returns them.
+func (s *sender) scrape(from string) string {
+	s.t.Helper()
+
+	_, metrics := s.requestWithHeader(from, proxy.MetricsPath, "Authorization: "+bearer,
+		http.StatusOK, requestlog.ActionAdmin)
+
+	return metrics
+}
+
+// metric returns the value of series in metrics, which are in the
+// Prometheus text format. series is a name and its labels in the order of
+// their names, such as smallwebwaf_offences_total{kind="limit"}. It fails
+// the test if there is no such series.
+func metric(t *testing.T, metrics, series string) float64 {
+	t.Helper()
+
+	for line := range strings.Lines(metrics) {
+		value, found := strings.CutPrefix(strings.TrimSuffix(line, "\n"), series+" ")
+		if !found {
+			continue
+		}
+
+		number, err := strconv.ParseFloat(value, 64)
+		if err != nil {
+			t.Fatalf("%s has the value %q", series, value)
+		}
+
+		return number
+	}
+
+	t.Fatalf("no series %s in the metrics:\n%s", series, metrics)
+
+	return 0
+}
+
+// wantMetric checks the value of series in metrics, as metric reads it.
+func wantMetric(t *testing.T, metrics, series string, want float64) {
+	t.Helper()
+
+	got := metric(t, metrics, series)
+	if got != want {
+		t.Errorf("%s is %v, want %v", series, got, want)
+	}
+}
+
+// wantNoSeries checks that metrics have no series series.
+func wantNoSeries(t *testing.T, metrics, series string) {
+	t.Helper()
+
+	if strings.Contains(metrics, "\n"+series+" ") {
+		t.Errorf("there is a series %s", series)
+	}
+}
+
+// wantLimitHits checks that the metrics of smallwebwaf at addr count hits
+// requests that passed the size or time limit of the setting limit, with
+// no series for it when hits is 0.
+func wantLimitHits(t *testing.T, addr, limit string, hits int) {
+	t.Helper()
+
+	series := `smallwebwaf_size_and_time_limit_hits_total{limit="` + limit + `"}`
+	metrics := scrape(t, addr)
+
+	if hits == 0 {
+		wantNoSeries(t, metrics, series)
+
+		return
+	}
+
+	wantMetric(t, metrics, series, float64(hits))
+}

internal/proxy/observe_test.go

@@ -0,0 +1,202 @@
+package proxy_test
+
+import (
+	"bytes"
+	"io"
+	"net/http"
+	"net/netip"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"sneak.berlin/go/smallwebwaf/internal/bans"
+	"sneak.berlin/go/smallwebwaf/internal/proxy"
+	"sneak.berlin/go/smallwebwaf/internal/requestlog"
+)
+
+// observe is the value of SWWAF_MODE for observe mode.
+const observe = "observe"
+
+func TestObserveModeForwardsWhatEnforceModeRefuses(t *testing.T) {
+	t.Parallel()
+
+	const (
+		denied = "192.0.2.50" // in SWWAF_DENY_NETS
+		banned = otherClient  // under a ban read from bans.json
+	)
+
+	for _, tc := range []struct {
+		setting string // "" leaves SWWAF_MODE at its default
+		observe bool
+	}{
+		{"", false},
+		{"enforce", false},
+		{observe, true},
+	} {
+		t.Run(mode+"="+tc.setting, func(t *testing.T) {
+			t.Parallel()
+
+			geojsURL, _ := startGeoJS(t)
+			env := map[string]string{
+				rateLimitPerMinute: "1",
+				denyNets:           denied,
+				deniedCountries:    "kp",
+			}
+
+			if tc.setting != "" {
+				env[mode] = tc.setting
+			}
+
+			s, clk, server := startWithClock(t, geojsURL, env)
+			server.Ledger.Load([]bans.Ban{{
+				Netblock: netip.MustParsePrefix(banned + "/32"),
+				Start:    clk.Now(),
+				Expires:  clk.Now().Add(time.Hour),
+			}})
+
+			// fromDE's first request is within the limit of one a minute,
+			// and its second breaks it.
+			s.get(fromDE, http.StatusOK, requestlog.ActionForward)
+
+			for _, sent := range []struct{ from, refusal string }{
+				{denied, requestlog.ActionDenied},
+				{banned, requestlog.ActionBanned},
+				{fromKP, requestlog.ActionCountryDenied},
+				{fromDE, requestlog.ActionRateLimited},
+			} {
+				if !tc.observe {
+					line := s.get(sent.from, http.StatusForbidden, sent.refusal)
+					wantWouldAction(t, line, "")
+
+					continue
+				}
+
+				// Passed to the app, which answered it.
+				line := s.get(sent.from, http.StatusOK, requestlog.ActionForward)
+				wantWouldAction(t, line, sent.refusal)
+
+				if line.UpstreamStatus != http.StatusOK {
+					t.Errorf("log line has upstream_status %d, want 200",
+						line.UpstreamStatus)
+				}
+			}
+		})
+	}
+}
+
+func TestObserveModeMakesNoBanAndKeepsTheBansItHas(t *testing.T) {
+	t.Parallel()
+
+	s, clk, server := startWithClock(t, "", map[string]string{
+		mode:               observe,
+		rateLimitPerMinute: "1",
+	})
+	kept := bans.Ban{
+		Netblock: netip.MustParsePrefix(otherClient + "/32"),
+		Start:    clk.Now(),
+		Expires:  clk.Now().Add(time.Hour),
+	}
+	server.Ledger.Load([]bans.Ban{kept})
+
+	// No ban sets client's counters back to zero, so each request after
+	// the first breaks the limit of one a minute.
+	s.get(client, http.StatusOK, requestlog.ActionForward)
+
+	for range 2 {
+		line := s.get(client, http.StatusOK, requestlog.ActionForward)
+		wantWouldAction(t, line, requestlog.ActionRateLimited)
+
+		if line.LimitHit != minute || line.Offence != requestlog.OffenceLimit ||
+			line.BanExpires != "" {
+			t.Errorf("log line has limit_hit %q, offence %q and ban_expires %q, "+
+				"want minute, limit and none", line.LimitHit, line.Offence,
+				line.BanExpires)
+		}
+	}
+
+	// The ban read from bans.json refuses nothing, and so counts no
+	// refusal in its notes, but is kept.
+	line := s.get(otherClient, http.StatusOK, requestlog.ActionForward)
+	wantWouldAction(t, line, requestlog.ActionBanned)
+
+	if line.BanExpires != requestlog.FormatTime(kept.Expires) {
+		t.Errorf("log line has ban_expires %q, want %s", line.BanExpires,
+			requestlog.FormatTime(kept.Expires))
+	}
+
+	got := server.Ledger.Snapshot()
+	if len(got) != 1 || got[0] != kept {
+		t.Errorf("bans\n%+v\nwant only\n%+v", got, kept)
+	}
+}
+
+func TestObserveModeKeepsTheSizeLimitsAndTheToken(t *testing.T) {
+	t.Parallel()
+
+	const denied = "192.0.2.50" // in SWWAF_DENY_NETS
+
+	var calls atomic.Int32
+
+	app := startApp(t, func(w http.ResponseWriter, _ *http.Request) {
+		calls.Add(1)
+		answerWithSize(w, 2*sizeLimit, true)
+	})
+	addr, out := startProxy(t, app.URL, map[string]string{
+		mode:             observe,
+		trustedProxies:   trustLocalhost,
+		denyNets:         denied,
+		requestMaxBytes:  sizeLimitSetting,
+		responseMaxBytes: sizeLimitSetting,
+		metricsToken:     token,
+	})
+
+	// SWWAF_DENY_NETS would refuse each request; instead a size limit or
+	// the missing token does.
+	for i, tc := range []struct {
+		method, path string
+		body         io.Reader
+		status       int
+		action       string
+	}{
+		{
+			http.MethodPost, "/upload", bytes.NewReader(make([]byte, 2*sizeLimit)),
+			http.StatusRequestEntityTooLarge, requestlog.ActionTooLarge,
+		},
+		{
+			http.MethodGet, "/download", http.NoBody,
+			http.StatusBadGateway, requestlog.ActionTooLarge,
+		},
+		{
+			http.MethodGet, proxy.MetricsPath, http.NoBody,
+			http.StatusUnauthorized, requestlog.ActionAdmin,
+		},
+	} {
+		req := newRequest(t, tc.method, addr, tc.path, tc.body)
+		req.Header.Set(forwardedFor, denied)
+		wantStatus(t, do(t, req), tc.status)
+
+		line := out.requestLines(t, i+1)[i]
+		wantLine(t, line, tc.status, tc.action)
+		wantWouldAction(t, line, requestlog.ActionDenied)
+	}
+
+	// The upload was refused before it reached the app.
+	if calls.Load() != 1 {
+		t.Errorf("the app was called %d times, want once", calls.Load())
+	}
+}
+
+// wantWouldAction checks the request log line's would_action, and that a
+// line that should have none has no such field.
+func wantWouldAction(t *testing.T, line logLine, want string) {
+	t.Helper()
+
+	got, present := line.fields["would_action"]
+
+	switch {
+	case want == "" && present:
+		t.Errorf("log line has would_action %v, want none", got)
+	case want != "" && got != want:
+		t.Errorf("log line has would_action %v, want %s", got, want)
+	}
+}

internal/proxy/passthrough_test.go

@@ -6,6 +6,8 @@ import (
 	"errors"
 	"io"
 	"net/http"
+	"os"
+	"reflect"
 	"slices"
 	"strings"
 	"sync/atomic"
@@ -14,6 +16,7 @@ import (
 
 	"sneak.berlin/go/smallwebwaf/internal/config"
 	"sneak.berlin/go/smallwebwaf/internal/proxy"
+	"sneak.berlin/go/smallwebwaf/internal/ratelimit"
 	"sneak.berlin/go/smallwebwaf/internal/requestlog"
 )
 
@@ -115,27 +118,34 @@ func wantAnswer(t *testing.T, got answer, body []byte) {
 	}
 }
 
-// wantRequestFields checks the log line's fields about the request.
+// wantRequestFields checks the log line's fields about the request. Its
+// time, its id and its timings are checked only for being there.
 func wantRequestFields(t *testing.T, line logLine, host string, sent, received int) {
 	t.Helper()
 
-	want := requestlog.Line{
-		Type: "request", Time: line.Time, ClientIP: localhost, PeerIP: localhost,
-		Method: http.MethodPatch, Host: host, Path: rawPath, Query: rawQuery,
-		Protocol: "HTTP/1.1", Status: http.StatusTeapot,
-		UpstreamStatus: http.StatusTeapot, RequestBytes: int64(sent),
+	hostname, _ := os.Hostname()
+
+	want := withTimings(line, requestlog.Line{
+		Type: requestType, Time: line.Time, Instance: hostname,
+		ClientIP: localhost, Method: http.MethodPatch, Scheme: plain, Host: host,
+		Path: rawPath, Query: rawQuery, Protocol: protocol,
+		Status: http.StatusTeapot, RequestBytes: int64(sent),
 		ResponseBytes: int64(received), UserAgent: "test-agent",
-		Action: requestlog.ActionForward, DurationTotal: line.DurationTotal,
-		DurationUpstreamTotal: line.DurationUpstreamTotal,
-	}
-	if line.Line != want {
+		RequestID: line.RequestID, PeerIP: localhost, ClientGroup: localhost + "/32",
+		ContentLength: int64(sent), ResponseContentType: "text/plain; charset=utf-8",
+		UpstreamStatus: http.StatusTeapot, Action: requestlog.ActionForward,
+		Counts: ratelimit.Counts{Minute: 1, Hour: 1, Day: 1},
+	})
+	if !reflect.DeepEqual(line.Line, want) {
 		t.Errorf("log line\n%+v\nwant\n%+v", line.Line, want)
 	}
 
 	_, err := time.Parse(time.RFC3339, line.Time)
-	if err != nil || line.DurationTotal <= 0 || line.DurationUpstreamTotal <= 0 {
-		t.Errorf("log line has time %q and durations %v and %v",
-			line.Time, line.DurationTotal, line.DurationUpstreamTotal)
+	if err != nil || line.RequestID == "" || line.DurationTotal <= 0 ||
+		line.DurationUpstreamTotal == nil || *line.DurationUpstreamTotal <= 0 {
+		t.Errorf("log line has time %q, request_id %q and durations %v and %v",
+			line.Time, line.RequestID, line.DurationTotal,
+			line.fields["duration_upstream_total"])
 	}
 }
 
@@ -371,8 +381,13 @@ func TestAnswers502WhenTheAppCannotBeReached(t *testing.T) {
 	addr, out := startProxy(t, "http://"+localhost+":1", nil)
 
 	wantStatus(t, get(t, addr, "/"), http.StatusBadGateway)
-	wantLine(t, out.requestLine(t), http.StatusBadGateway,
-		requestlog.ActionUpstreamError)
+
+	line := out.requestLine(t)
+	wantLine(t, line, http.StatusBadGateway, requestlog.ActionUpstreamError)
+
+	// There never was a connection to the app, nor an answer from it.
+	wantTimings(t, line, "duration_total", "duration_checks",
+		"duration_upstream_total")
 
 	logged := slices.ContainsFunc(out.lines(t), func(line map[string]any) bool {
 		return line["type"] == "process" && line["msg"] == "request to the app failed"

internal/proxy/proxy.go

@@ -8,11 +8,13 @@ import (
 	"log"
 	"log/slog"
 	"net/http"
+	"strings"
 	"time"
 
 	"sneak.berlin/go/smallwebwaf/internal/bans"
 	"sneak.berlin/go/smallwebwaf/internal/config"
 	"sneak.berlin/go/smallwebwaf/internal/lookup"
+	"sneak.berlin/go/smallwebwaf/internal/metrics"
 	"sneak.berlin/go/smallwebwaf/internal/ratelimit"
 	"sneak.berlin/go/smallwebwaf/internal/requestlog"
 )
@@ -23,10 +25,18 @@ const (
 	appIdleConnTimeout = 90 * time.Second
 )
 
+// adminPrefix starts the path of every request for smallwebwaf itself,
+// which never reaches the app.
+const adminPrefix = "/_smallwebwaf/"
+
 // HealthPath is smallwebwaf's health endpoint, which the container's
 // health check asks.
 const HealthPath = "/_smallwebwaf/healthz"
 
+// MetricsPath is where the metrics are, for a request that carries
+// SWWAF_METRICS_TOKEN.
+const MetricsPath = "/_smallwebwaf/metrics"
+
 // Params are what New needs.
 type Params struct {
 	Config *config.Config
@@ -44,13 +54,14 @@ type Params struct {
 }
 
 // Server is the server smallwebwaf runs, with the parts of the proxy
-// whose state the state files keep.
+// whose state the state files keep, and the metrics.
 type Server struct {
 	*http.Server
 
 	Ledger  *bans.Ledger
 	Limiter *ratelimit.Limiter
 	GeoJS   *lookup.GeoJS
+	Metrics *metrics.Metrics
 }
 
 // New returns the server smallwebwaf runs: each request it reads passes
@@ -61,6 +72,7 @@ type Server struct {
 // applies the timeouts and size limits from then on.
 func New(params Params) *Server {
 	errorLog := slog.NewLogLogger(params.ProcessLog.Handler(), slog.LevelWarn)
+	m := metrics.New(params.Config.MetricsTopN)
 	h := &handler{
 		config:     params.Config,
 		requestLog: params.RequestLog,
@@ -68,6 +80,7 @@ func New(params Params) *Server {
 		errorLog:   errorLog,
 		transport:  newTransport(),
 		now:        params.Now,
+		metrics:    m,
 		limiter: ratelimit.New(ratelimit.Limits{
 			PerMinute: params.Config.RateLimitPerMinute,
 			PerHour:   params.Config.RateLimitPerHour,
@@ -83,8 +96,10 @@ func New(params Params) *Server {
 			URL:        params.GeoJSURL,
 			Now:        params.Now,
 			ProcessLog: params.ProcessLog,
+			Metrics:    m,
 		}),
 	}
+	m.AddBansAndClients(h.ledger, h.limiter, params.Now)
 
 	return &Server{
 		Server: &http.Server{
@@ -102,6 +117,7 @@ func New(params Params) *Server {
 		Ledger:  h.ledger,
 		Limiter: h.limiter,
 		GeoJS:   h.geojs,
+		Metrics: m,
 	}
 }
 
@@ -114,6 +130,7 @@ type handler struct {
 	errorLog   *log.Logger
 	transport  http.RoundTripper
 	now        func() time.Time
+	metrics    *metrics.Metrics
 	limiter    *ratelimit.Limiter
 	ledger     *bans.Ledger
 	geojs      *lookup.GeoJS
@@ -133,7 +150,8 @@ func newTransport() *http.Transport {
 
 // ServeHTTP handles one request: it works out the client, runs the
 // checks, passes the request to the app and the answer back within the
-// limits, and writes the request's log line.
+// limits, or answers it itself if it is for smallwebwaf, and writes the
+// request's log line.
 func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 	rq := h.newRequest(w, r)
 	defer rq.finish()
@@ -142,6 +160,9 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 	// a health checker is never refused. It does not ask the app.
 	if r.Method == http.MethodGet && r.URL.Path == HealthPath {
 		rq.line.Action = requestlog.ActionAdmin
+		// Set here rather than left to Go's server, which would set it only
+		// after the log line has taken the response's headers.
+		rq.out.Header().Set("Content-Type", "text/plain; charset=utf-8")
 		_, _ = io.WriteString(rq.out, "ok\n")
 
 		return
@@ -151,11 +172,21 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 	defer rq.addToHistory()
 
 	refused := rq.check(r.Context())
+	rq.checked = time.Now()
+
 	if refused != nil {
 		rq.answer(*refused)
 
 		return
 	}
 
+	// A request for smallwebwaf itself is answered where another would be
+	// passed to the app, so that it goes through every check first.
+	if strings.HasPrefix(r.URL.Path, adminPrefix) {
+		rq.answerAdmin()
+
+		return
+	}
+
 	rq.forward(r.Context())
 }

internal/proxy/proxy_test.go

@@ -35,6 +35,10 @@ const (
 	// localhost is where every test server listens, and so the address
 	// smallwebwaf sees each test's requests come from.
 	localhost = "127.0.0.1"
+	// requestType is the type that marks a request log line.
+	requestType = "request"
+	// protocol is the protocol of every test's requests.
+	protocol = "HTTP/1.1"
 )
 
 // shortTimeoutSetting is shortTimeout as a setting's value.
@@ -50,6 +54,7 @@ const (
 	clientResponseTimeout   = "SWWAF_CLIENT_RESPONSE_TIMEOUT"
 	upstreamRequestTimeout  = "SWWAF_UPSTREAM_REQUEST_TIMEOUT"
 	upstreamResponseTimeout = "SWWAF_UPSTREAM_RESPONSE_TIMEOUT"
+	mode                    = "SWWAF_MODE"
 	requestMaxBytes         = "SWWAF_REQUEST_MAX_BYTES"
 	responseMaxBytes        = "SWWAF_RESPONSE_MAX_BYTES"
 	trustedProxies          = "SWWAF_TRUSTED_PROXIES"
@@ -58,6 +63,7 @@ const (
 	denyNets                = "SWWAF_DENY_NETS"
 	rateLimitPerMinute      = "SWWAF_RATE_LIMIT_PER_MINUTE"
 	rateLimitPerDay         = "SWWAF_RATE_LIMIT_PER_DAY"
+	rateLimitExemptPaths    = "SWWAF_RATE_LIMIT_EXEMPT_PATHS"
 	deniedCountries         = "SWWAF_DENIED_COUNTRIES"
 	allowedCountries        = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES"
 	banResponse             = "SWWAF_BAN_RESPONSE"
@@ -66,6 +72,8 @@ const (
 	maxBanDuration          = "SWWAF_MAX_BAN_DURATION"
 	maxBans                 = "SWWAF_MAX_BANS"
 	banScopeV4Prefix        = "SWWAF_BAN_SCOPE_V4_PREFIX"
+	instanceName            = "SWWAF_INSTANCE_NAME"
+	logRequestHeaders       = "SWWAF_LOG_REQUEST_HEADERS"
 )
 
 // output collects what smallwebwaf writes on stdout.
@@ -82,6 +90,14 @@ func (o *output) Write(p []byte) (int, error) {
 	return o.buf.Write(p)
 }
 
+// text returns everything written so far.
+func (o *output) text() string {
+	o.mu.Lock()
+	defer o.mu.Unlock()
+
+	return o.buf.String()
+}
+
 // lines returns every line written so far, decoded.
 func (o *output) lines(t *testing.T) []map[string]any {
 	t.Helper()
@@ -121,7 +137,7 @@ func (o *output) requestLines(t *testing.T, count int) []logLine {
 		var found []logLine
 
 		for _, fields := range o.lines(t) {
-			if fields["type"] == "request" {
+			if fields["type"] == requestType {
 				found = append(found, decodeLine(t, fields))
 			}
 		}

internal/proxy/ratelimits_test.go

@@ -5,6 +5,8 @@ import (
 	"sync/atomic"
 	"testing"
 
+	"sneak.berlin/go/smallwebwaf/internal/lookup"
+	"sneak.berlin/go/smallwebwaf/internal/ratelimit"
 	"sneak.berlin/go/smallwebwaf/internal/requestlog"
 )
 
@@ -69,3 +71,89 @@ func TestRateLimitRefusesBeforeTheApp(t *testing.T) {
 		t.Errorf("the app was called %d times, want 4", calls.Load())
 	}
 }
+
+func TestRateLimitExemptPathsAreNeitherCountedNorRefused(t *testing.T) {
+	t.Parallel()
+
+	const denied = "192.0.2.50" // in SWWAF_DENY_NETS
+
+	s, _, server := startWithClock(t, "", map[string]string{
+		rateLimitPerMinute:   "1",
+		rateLimitExemptPaths: "/assets/,/favicon.ico",
+		denyNets:             denied,
+		deniedCountries:      "kp",
+	})
+
+	// The answers are kept before the requests, so that none waits for
+	// GeoJS.
+	server.GeoJS.Load([]lookup.Answer{
+		keptAnswer(client, "DE"), keptAnswer(fromKP, "KP"),
+	})
+
+	// With a limit of one request a minute, the requests for paths under a
+	// prefix are not counted, so client's first request for / is within
+	// the limit; and once client has reached it, they are not refused.
+	s.request(client, "/assets/app.js", http.StatusOK, requestlog.ActionForward)
+	s.request(client, "/favicon.ico?v=2", http.StatusOK, requestlog.ActionForward)
+	s.get(client, http.StatusOK, requestlog.ActionForward)
+
+	line := s.request(client, "/assets/app.js", http.StatusOK, requestlog.ActionForward)
+	if line.LimitHit != "" || line.Counts != (ratelimit.Counts{}) {
+		t.Errorf("log line has limit_hit %q and counts %+v, want neither",
+			line.LimitHit, line.Counts)
+	}
+
+	// A path outside every prefix is counted: /assets is not under
+	// /assets/, and breaks the limit.
+	s.request(client, "/assets", http.StatusForbidden, requestlog.ActionRateLimited)
+
+	// A ban, SWWAF_DENY_NETS and the country lists still refuse a path
+	// under a prefix.
+	s.request(client, "/assets/app.js", http.StatusForbidden, requestlog.ActionBanned)
+	s.request(denied, "/assets/app.js", http.StatusForbidden, requestlog.ActionDenied)
+	s.request(fromKP, "/assets/app.js",
+		http.StatusForbidden, requestlog.ActionCountryDenied)
+}
+
+func TestRateLimitCountsPathsThatAreNotExempt(t *testing.T) {
+	t.Parallel()
+
+	for _, sent := range []string{
+		// A prefix matches only at the start of the path.
+		"/static/assets/app.js",
+		// A prefix matches the path as sent: a router that matches the
+		// path as received does not take /%61ssets/x for a path under
+		// /assets/.
+		"/%61ssets/x",
+		// .. once percent-decoded: an app may act on these as /login, the
+		// last as a path under /sneak/app/ or as /assets/x.
+		"/assets/../login",
+		"/assets/%2e%2e/login",
+		"/assets/..%2Flogin",
+		"/assets/..;/login",
+		"/sneak/app/src/branch/main/..%2F..%2F..%2F..%2F..%2F..%2Fassets/x",
+		// Not under /assets/ as sent: Go's router takes /assets%2Fx for one
+		// path segment, not a path under /assets/.
+		"/assets%2Fx",
+		"/assets%2fx",
+		// Under /assets/ as sent, but holding an encoded slash, in either
+		// case, or a backslash: never exempt, whatever the prefix.
+		"/assets/x%2Fy",
+		"/assets/x%2fy",
+		`/assets/x\y`,
+	} {
+		t.Run(sent, func(t *testing.T) {
+			t.Parallel()
+
+			s, _, _ := startWithClock(t, "", map[string]string{
+				rateLimitPerMinute:   "1",
+				rateLimitExemptPaths: "/assets/",
+			})
+
+			// Counted, the second request breaks the limit of one request
+			// a minute.
+			s.request(client, sent, http.StatusOK, requestlog.ActionForward)
+			s.request(client, sent, http.StatusForbidden, requestlog.ActionRateLimited)
+		})
+	}
+}

internal/proxy/request.go

@@ -7,7 +7,10 @@ import (
 	"net/http/httptrace"
 	"net/http/httputil"
 	"net/netip"
+	"net/url"
 	"os"
+	"slices"
+	"strings"
 	"sync"
 	"sync/atomic"
 	"time"
@@ -22,11 +25,13 @@ const flushAfterEachWrite time.Duration = -1
 
 // refusal is smallwebwaf refusing a request, or refusing to go on with it:
 // the status the client is answered if the response has not started yet,
-// 0 to close the connection without an answer, and the action the log
-// line names.
+// 0 to close the connection without an answer, the action the log line
+// names, and the setting whose size or time limit the request passed, if
+// that is why.
 type refusal struct {
 	status int
 	action string
+	limit  string
 }
 
 // request is one request on its way through smallwebwaf, from the moment
@@ -44,7 +49,9 @@ type request struct {
 	peer        netip.Addr
 	peerTrusted bool
 	start       time.Time
-	// upstreamStart is when the request was handed to the app.
+	// checked is when the checks were done, and upstreamStart when the
+	// request was handed to the app.
+	checked       time.Time
 	upstreamStart time.Time
 	// cancel ends the request to the app.
 	cancel context.CancelFunc
@@ -54,24 +61,34 @@ type request struct {
 	complete bool
 
 	// mu guards what follows. The timeouts run on goroutines of their
-	// own, and the transport starts and stops them from its own; once
-	// timersStopped is set, none of them acts any more.
+	// own, and the transport starts and stops them, and notes the times
+	// below, from its own; once timersStopped is set, none of the timeouts
+	// acts any more.
 	mu                    sync.Mutex
 	timersStopped         bool
 	clientRequestTimer    *time.Timer
 	upstreamRequestTimer  *time.Timer
 	upstreamResponseTimer *time.Timer
-	// requestSent is when the app had been sent the whole request.
-	requestSent time.Time
+	// connected is when there was a connection to the app, requestSent
+	// when the app had been sent the whole request, and answerStarted
+	// when the first byte of its answer arrived.
+	connected     time.Time
+	requestSent   time.Time
+	answerStarted time.Time
 }
 
-// newRequest starts handling r: it notes the time and works out the
-// client.
+// newRequest starts handling r: it notes the time, counts the request as
+// under way, works out the client, and starts the log line with what is
+// known of the request.
 func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request {
+	h.metrics.RequestStarted()
+
 	start := time.Now()
 	peer := peerAddress(r)
 	trusted := h.config.TrustedProxies
-	client := clientAddress(peer, r.Header.Values("X-Forwarded-For"), trusted)
+	peerTrusted := isInside(peer, trusted)
+	forwardedFor := r.Header.Values("X-Forwarded-For")
+	client := clientAddress(peer, forwardedFor, trusted)
 
 	rq := &request{
 		h:           h,
@@ -80,22 +97,37 @@ func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request {
 		out:         &responseWriter{ResponseWriter: w},
 		client:      client,
 		peer:        peer,
-		peerTrusted: isInside(peer, trusted),
+		peerTrusted: peerTrusted,
 		start:       start,
 		line: requestlog.Line{
-			Time:      requestlog.FormatTime(start),
-			ClientIP:  client.String(),
-			PeerIP:    peer.String(),
-			Method:    r.Method,
-			Host:      r.Host,
-			Path:      r.URL.EscapedPath(),
-			Query:     r.URL.RawQuery,
-			Protocol:  r.Proto,
-			Referer:   r.Referer(),
-			UserAgent: r.UserAgent(),
-			Action:    requestlog.ActionForward,
+			Time:             requestlog.FormatTime(start),
+			Instance:         h.config.InstanceName,
+			ClientIP:         client.String(),
+			Method:           r.Method,
+			Scheme:           scheme(r, peerTrusted),
+			Host:             r.Host,
+			Path:             r.URL.EscapedPath(),
+			Query:            r.URL.RawQuery,
+			Protocol:         r.Proto,
+			Referer:          r.Referer(),
+			UserAgent:        r.UserAgent(),
+			RequestID:        requestID(r, peerTrusted),
+			PeerIP:           peer.String(),
+			ForwardedFor:     strings.Join(forwardedFor, ", "),
+			ClientGroup:      clientGroup(client).String(),
+			ContentType:      r.Header.Get("Content-Type"),
+			RequestHeaders:   requestHeaders(r, h.config.LogRequestHeaders),
+			HasAuthorization: len(r.Header.Values("Authorization")) > 0,
+			HasCookie:        len(r.Header.Values("Cookie")) > 0,
+			Action:           requestlog.ActionForward,
 		},
 	}
+
+	// A length of -1 is a body whose length was not announced.
+	if r.ContentLength > 0 {
+		rq.line.ContentLength = r.ContentLength
+	}
+
 	if r.Body != http.NoBody {
 		rq.body = &requestBody{body: limitBody(r.Body, h.config.RequestMaxBytes), rq: rq}
 	}
@@ -103,50 +135,121 @@ func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request {
 	return rq
 }
 
+// requestHeaders returns the headers of r that names lists, by name in
+// lower case, each with its values joined by ", ". Authorization, Cookie
+// and Set-Cookie are never among them, whatever names says.
+func requestHeaders(r *http.Request, names []string) map[string]string {
+	headers := map[string]string{}
+
+	for _, name := range names {
+		switch name {
+		case "authorization", "cookie", "set-cookie":
+			continue
+		}
+
+		values := r.Header.Values(name)
+		if len(values) > 0 {
+			headers[name] = strings.Join(values, ", ")
+		}
+	}
+
+	return headers
+}
+
 // check is the one place where a request can be refused once its client
 // is known, before its body is read or anything reaches the app. It
-// returns nil to let the request through. A client in SWWAF_ALLOW_NETS
-// skips every check but the size limit. For any other client,
-// SWWAF_DENY_NETS comes first, then a ban on its netblock, so that a
-// client either refuses is not looked up, and then the country lists; a
-// request any of them refuses is not counted for the rate limits. Then
-// come the rate limits, unless the client is in
-// SWWAF_RATE_LIMIT_EXEMPT_NETS, so that every other request is counted,
-// one refused for its size too. Every refusal but the size limit's is
-// answered with SWWAF_BAN_RESPONSE. ctx is the request's own context.
+// returns nil to let the request through. The checks of checkClient come
+// first, answered with SWWAF_BAN_RESPONSE, and then the size limit, so
+// that a request the rate limits count is counted even when it is
+// refused for its size. In observe mode a request checkClient refuses
+// goes on to the size limit like any other. ctx is the request's own
+// context.
 func (rq *request) check(ctx context.Context) *refusal {
-	cfg := rq.h.config
-	allowed := isInside(rq.client, cfg.AllowNets)
-	exempt := isInside(rq.client, cfg.RateLimitExemptNets)
-	now := rq.h.now()
+	action := rq.checkClient(ctx)
+	if action != "" {
+		if !rq.h.config.Observe {
+			return rq.banResponse(action)
+		}
 
-	if !allowed && isInside(rq.client, cfg.DenyNets) {
-		return rq.banResponse(requestlog.ActionDenied)
+		// The log line names what enforce mode would have done.
+		rq.line.WouldAction = action
 	}
 
-	if !allowed && rq.banned(now) {
-		return rq.banResponse(requestlog.ActionBanned)
-	}
-
-	if !allowed && rq.countryDenied(ctx) {
-		return rq.banResponse(requestlog.ActionCountryDenied)
-	}
-
-	if !allowed && !exempt && rq.limitBroken(now) {
-		return rq.banResponse(requestlog.ActionRateLimited)
-	}
-
-	maxBytes := cfg.RequestMaxBytes
+	maxBytes := rq.h.config.RequestMaxBytes
 	if maxBytes > 0 && rq.in.ContentLength > maxBytes {
 		return &refusal{
 			status: http.StatusRequestEntityTooLarge,
 			action: requestlog.ActionTooLarge,
+			limit:  "SWWAF_REQUEST_MAX_BYTES",
 		}
 	}
 
 	return nil
 }
 
+// checkClient runs the checks on the request's client, and returns the
+// action of the first that refuses the request, or "" when none does. A
+// client in SWWAF_ALLOW_NETS skips them. For any other client,
+// SWWAF_DENY_NETS comes first, then a ban on its netblock, so that a
+// client either refuses is not looked up, and then the country lists; a
+// request any of them refuses is not counted for the rate limits. Then
+// come the rate limits, unless the client is in
+// SWWAF_RATE_LIMIT_EXEMPT_NETS or the request's path is exempt under
+// SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that every other request is counted.
+// ctx is the request's own context.
+func (rq *request) checkClient(ctx context.Context) string {
+	cfg := rq.h.config
+	if isInside(rq.client, cfg.AllowNets) {
+		return ""
+	}
+
+	now := rq.h.now()
+
+	if isInside(rq.client, cfg.DenyNets) {
+		return requestlog.ActionDenied
+	}
+
+	if rq.banned(now) {
+		return requestlog.ActionBanned
+	}
+
+	if rq.countryDenied(ctx) {
+		return requestlog.ActionCountryDenied
+	}
+
+	exempt := isInside(rq.client, cfg.RateLimitExemptNets) ||
+		pathExempt(rq.in.URL, cfg.RateLimitExemptPaths)
+	if !exempt && rq.limitBroken(now) {
+		return requestlog.ActionRateLimited
+	}
+
+	return ""
+}
+
+// pathExempt reports whether the rate limits leave out a request for u
+// because of SWWAF_RATE_LIMIT_EXEMPT_PATHS: whether its path as sent, the
+// path the app receives, not percent-decoded, starts with one of
+// prefixes, so that /%61ssets/x is not under /assets/ for an app whose
+// router matches the path as received. A request whose decoded path
+// contains .. anywhere or a backslash, or whose path as sent holds an
+// encoded slash (%2F or %2f), never is, since an app may act on it as a
+// path outside every prefix: /assets/..%2Flogin as /login, or /assets%2Fx
+// as one path segment, as Go's router does.
+func pathExempt(u *url.URL, prefixes []string) bool {
+	decoded := u.Path
+	// EscapedPath is the path as the app receives it, not decoded.
+	sent := u.EscapedPath()
+
+	if strings.Contains(decoded, "..") || strings.Contains(decoded, `\`) ||
+		strings.Contains(strings.ToLower(sent), "%2f") {
+		return false
+	}
+
+	return slices.ContainsFunc(prefixes, func(prefix string) bool {
+		return strings.HasPrefix(sent, prefix)
+	})
+}
+
 // forward passes the request to the app and the app's answer back. ctx
 // is the request's own context.
 func (rq *request) forward(ctx context.Context) {
@@ -155,7 +258,9 @@ func (rq *request) forward(ctx context.Context) {
 
 	rq.cancel = cancel
 	ctx = httptrace.WithClientTrace(ctx, &httptrace.ClientTrace{
-		WroteRequest: rq.wroteRequest,
+		GotConn:              rq.gotConn,
+		WroteRequest:         rq.wroteRequest,
+		GotFirstResponseByte: rq.gotFirstResponseByte,
 	})
 
 	out := rq.in.WithContext(ctx)
@@ -178,7 +283,8 @@ func (rq *request) forward(ctx context.Context) {
 }
 
 // rewrite makes the request the app receives: the client's request,
-// unchanged, sent to SWWAF_UPSTREAM_URL, with the forwarded headers set.
+// unchanged, sent to SWWAF_UPSTREAM_URL, with the forwarded headers and
+// the request's id set.
 func (rq *request) rewrite(pr *httputil.ProxyRequest) {
 	upstream := rq.h.config.UpstreamURL
 	pr.Out.URL.Scheme = upstream.Scheme
@@ -187,6 +293,7 @@ func (rq *request) rewrite(pr *httputil.ProxyRequest) {
 	// the query as the client sent it.
 	pr.Out.URL.RawQuery = pr.In.URL.RawQuery
 	setForwardedHeaders(pr.In, pr.Out, rq.peer, rq.peerTrusted)
+	pr.Out.Header.Set(requestIDHeader, rq.line.RequestID)
 }
 
 // modifyResponse looks at the app's answer before ReverseProxy passes it
@@ -200,13 +307,18 @@ func (rq *request) modifyResponse(res *http.Response) error {
 		// connection it takes over, not through rq.out.
 		rq.stopTimers()
 		rq.out.status = res.StatusCode
+		rq.line.Websocket = true
 
 		return nil
 	}
 
 	maxBytes := rq.h.config.ResponseMaxBytes
 	if maxBytes > 0 && res.Body != http.NoBody && res.ContentLength > maxBytes {
-		rq.refuse(refusal{status: http.StatusBadGateway, action: requestlog.ActionTooLarge})
+		rq.refuse(refusal{
+			status: http.StatusBadGateway,
+			action: requestlog.ActionTooLarge,
+			limit:  "SWWAF_RESPONSE_MAX_BYTES",
+		})
 
 		return errResponseTooLarge
 	}
@@ -278,7 +390,8 @@ func (rq *request) refuse(r refusal) {
 	rq.cancel()
 }
 
-// finish ends the request's timeouts and writes its log line.
+// finish ends the request's timeouts, counts it in the metrics and writes
+// its log line.
 func (rq *request) finish() {
 	rq.stopTimers()
 
@@ -290,35 +403,69 @@ func (rq *request) finish() {
 	line := &rq.line
 	line.Status = rq.out.status
 	line.ResponseBytes = rq.out.bytes
+	header := rq.out.Header()
+	line.ResponseContentType = header.Get("Content-Type")
+	line.CacheControl = header.Get("Cache-Control")
+	line.Location = header.Get("Location")
 
 	if rq.body != nil {
 		line.RequestBytes = rq.body.bytes.Load()
 	}
 
+	// limit is the setting whose size or time limit the request passed.
+	var limit string
+
 	switch {
 	case refused != nil:
 		line.Action = refused.action
+		limit = refused.limit
 	case errors.Is(rq.out.err, os.ErrDeadlineExceeded):
 		// The client took longer than SWWAF_CLIENT_RESPONSE_TIMEOUT to
 		// take the response.
 		line.Action = requestlog.ActionTimedOut
+		limit = "SWWAF_CLIENT_RESPONSE_TIMEOUT"
 	case !rq.complete && (rq.out.err != nil || rq.in.Context().Err() != nil):
 		line.Aborted = true
 	}
 
 	now := time.Now()
-	line.DurationTotal = requestlog.Milliseconds(now.Sub(rq.start))
+	duration := now.Sub(rq.start)
+	line.DurationTotal = requestlog.Milliseconds(duration)
+	line.DurationChecks = timing(rq.start, rq.checked)
+
+	var upstreamDuration time.Duration
 
 	if !rq.upstreamStart.IsZero() {
-		line.DurationUpstreamTotal = requestlog.Milliseconds(now.Sub(rq.upstreamStart))
+		upstreamDuration = now.Sub(rq.upstreamStart)
+		line.DurationUpstreamTotal = new(requestlog.Milliseconds(upstreamDuration))
+
+		rq.mu.Lock()
+		line.DurationUpstreamConnect = timing(rq.upstreamStart, rq.connected)
+		line.DurationUpstreamFirstByte = timing(rq.upstreamStart, rq.answerStarted)
+		rq.mu.Unlock()
 	}
 
+	// Counted before the log line is written, so that the metrics count
+	// every request whose line is out.
+	rq.h.metrics.RequestEnded(line, limit, duration, upstreamDuration)
+
 	err := requestlog.Write(rq.h.requestLog, line)
 	if err != nil {
 		rq.h.processLog.Error("writing the request log failed", "error", err.Error())
 	}
 }
 
+// timing is the time from start to end in milliseconds, for one of the
+// log line's timings, or nil when end is zero: what it times never
+// happened.
+func timing(start, end time.Time) *float64 {
+	if end.IsZero() {
+		return nil
+	}
+
+	return new(requestlog.Milliseconds(end.Sub(start)))
+}
+
 // addToHistory adds the request, which has ended, to its client's
 // history.
 func (rq *request) addToHistory() {
@@ -327,9 +474,12 @@ func (rq *request) addToHistory() {
 		requestBytes = rq.body.bytes.Load()
 	}
 
+	forwarded := !rq.upstreamStart.IsZero()
+
 	rq.h.limiter.AddToHistory(clientGroup(rq.client), rq.h.now(), ratelimit.Request{
 		Country:       rq.line.Country,
-		Forwarded:     !rq.upstreamStart.IsZero(),
+		Forwarded:     forwarded,
+		Refused:       !forwarded && rq.refused.Load() != nil,
 		Status:        rq.out.status,
 		RequestBytes:  requestBytes,
 		ResponseBytes: rq.out.bytes,
@@ -369,21 +519,26 @@ func (rq *request) startRequestTimers() {
 
 	if rq.body != nil && rq.h.config.ClientRequestTimeout > 0 {
 		rq.clientRequestTimer = time.AfterFunc(
-			time.Until(rq.clientRequestDeadline()), rq.requestTimedOut)
+			time.Until(rq.clientRequestDeadline()), func() {
+				rq.requestTimedOut("SWWAF_CLIENT_REQUEST_TIMEOUT")
+			})
 	}
 
 	timeout := rq.h.config.UpstreamRequestTimeout
 	if timeout > 0 {
-		rq.upstreamRequestTimer = time.AfterFunc(timeout, rq.requestTimedOut)
+		rq.upstreamRequestTimer = time.AfterFunc(timeout, func() {
+			rq.requestTimedOut("SWWAF_UPSTREAM_REQUEST_TIMEOUT")
+		})
 	}
 }
 
-// requestTimedOut is called when a request timeout runs out while the
-// request is still on its way to the app. The answer names the side
-// smallwebwaf was waiting on at that moment: 408 when it was waiting for
-// the client to send more of its body, 504 when it was waiting for the
-// app to be reached or to take what it had.
-func (rq *request) requestTimedOut() {
+// requestTimedOut is called when limit, SWWAF_CLIENT_REQUEST_TIMEOUT or
+// SWWAF_UPSTREAM_REQUEST_TIMEOUT, runs out while the request is still on
+// its way to the app. The answer names the side smallwebwaf was waiting
+// on at that moment: 408 when it was waiting for the client to send more
+// of its body, 504 when it was waiting for the app to be reached or to
+// take what it had.
+func (rq *request) requestTimedOut(limit string) {
 	rq.mu.Lock()
 	defer rq.mu.Unlock()
 
@@ -395,6 +550,7 @@ func (rq *request) requestTimedOut() {
 		rq.refuse(refusal{
 			status: http.StatusGatewayTimeout,
 			action: requestlog.ActionTimedOut,
+			limit:  limit,
 		})
 
 		return
@@ -403,6 +559,7 @@ func (rq *request) requestTimedOut() {
 	rq.refuse(refusal{
 		status: http.StatusRequestTimeout,
 		action: requestlog.ActionTimedOut,
+		limit:  limit,
 	})
 	// The transport gives up on the app only once its Read of the
 	// client's body returns, so that Read is ended now. The lock keeps
@@ -418,6 +575,24 @@ func (rq *request) bodyReceived() {
 	stopTimer(rq.clientRequestTimer)
 }
 
+// gotConn is called once there is a connection to the app, a new one or
+// one kept open from an earlier request.
+func (rq *request) gotConn(httptrace.GotConnInfo) {
+	rq.mu.Lock()
+	defer rq.mu.Unlock()
+
+	rq.connected = time.Now()
+}
+
+// gotFirstResponseByte is called once the first byte of the app's answer
+// has arrived.
+func (rq *request) gotFirstResponseByte() {
+	rq.mu.Lock()
+	defer rq.mu.Unlock()
+
+	rq.answerStarted = time.Now()
+}
+
 // wroteRequest is called once the app has been sent the whole request:
 // the request timeouts end and SWWAF_UPSTREAM_RESPONSE_TIMEOUT starts.
 func (rq *request) wroteRequest(info httptrace.WroteRequestInfo) {
@@ -452,6 +627,7 @@ func (rq *request) responseTimedOut() {
 		rq.refuse(refusal{
 			status: http.StatusGatewayTimeout,
 			action: requestlog.ActionTimedOut,
+			limit:  "SWWAF_UPSTREAM_RESPONSE_TIMEOUT",
 		})
 	}
 }

internal/proxy/requestlog_test.go

@@ -0,0 +1,368 @@
+package proxy_test
+
+import (
+	"io"
+	"maps"
+	"math"
+	"net/http"
+	"reflect"
+	"slices"
+	"strings"
+	"testing"
+	"time"
+
+	"sneak.berlin/go/smallwebwaf/internal/proxy"
+	"sneak.berlin/go/smallwebwaf/internal/ratelimit"
+	"sneak.berlin/go/smallwebwaf/internal/requestlog"
+)
+
+const (
+	// requestIDHeader carries the request's id.
+	requestIDHeader = "X-Request-ID"
+	// instance is the SWWAF_INSTANCE_NAME a test sets.
+	instance = "fsn1app1/gitea"
+	// ipv6Client is a client on IPv6, and ipv6Group the netblock the rate
+	// limits count it as.
+	ipv6Client = "2001:db8::7"
+	ipv6Group  = "2001:db8::/64"
+)
+
+func TestLogLineHasEachFieldWhereItApplies(t *testing.T) {
+	t.Parallel()
+
+	received := make(chan string, 2) // the request ids the app received
+	app := startApp(t, func(w http.ResponseWriter, r *http.Request) {
+		received <- r.Header.Get(requestIDHeader)
+
+		_, _ = io.Copy(io.Discard, r.Body)
+
+		if r.URL.Path != "/full" {
+			w.WriteHeader(http.StatusNoContent)
+
+			return
+		}
+
+		w.Header().Set("Content-Type", "text/html")
+		w.Header().Set("Cache-Control", "no-store")
+		w.Header().Set("Location", "/elsewhere")
+		w.WriteHeader(http.StatusFound)
+		_, _ = io.WriteString(w, "moved")
+	})
+	addr, out := startProxy(t, app.URL, map[string]string{
+		trustedProxies:      trustLocalhost,
+		rateLimitExemptNets: localhost,
+		instanceName:        instance,
+		logRequestHeaders:   "Accept,x-custom,Authorization,cookie,SET-COOKIE",
+	})
+
+	// This request comes from ipv6Client through a trusted proxy, with a
+	// body and each header the log line looks at, and is answered with a
+	// redirect.
+	conn := dial(t, addr)
+	send(t, conn, "POST /full HTTP/1.1\r\nHost: "+appHost+"\r\n"+
+		forwardedFor+": 198.51.100.7, "+ipv6Client+"\r\n"+
+		forwardedProto+": "+secure+"\r\n"+requestIDHeader+": from-traefik\r\n"+
+		"Content-Type: application/x-www-form-urlencoded\r\nContent-Length: 3\r\n"+
+		"Accept: text/html\r\nX-Custom: one\r\nX-Custom: two\r\n"+
+		"Authorization: Bearer secret-token\r\nCookie: session=secret-cookie\r\n"+
+		"Set-Cookie: secret-set-cookie\r\n\r\na=b")
+	wantStatus(t, readResponse(t, conn), http.StatusFound)
+
+	// A request's log line can come after its answer: each is waited for
+	// before the next request, so that the lines are in order.
+	full := out.requestLines(t, 1)[0]
+
+	// This one comes from 127.0.0.1, which the rate limits do not count,
+	// with a body of 4 bytes whose length it does not announce, so that its
+	// request_bytes is not its content_length, and no header the log line
+	// looks at, and is answered with 204 and no header.
+	conn = dial(t, addr)
+	send(t, conn, "POST /bare HTTP/1.1\r\nHost: "+appHost+"\r\n"+
+		"Transfer-Encoding: chunked\r\n\r\n4\r\nbody\r\n0\r\n\r\n")
+	wantStatus(t, readResponse(t, conn), http.StatusNoContent)
+
+	bare := out.requestLines(t, 2)[1]
+
+	wantFullLine(t, full)
+	wantBareLine(t, bare)
+
+	for _, line := range []logLine{full, bare} {
+		got := <-received
+		if got != line.RequestID {
+			t.Errorf("the app received request id %q, the log line has %q",
+				got, line.RequestID)
+		}
+	}
+
+	if strings.Contains(out.text(), "secret") {
+		t.Errorf("a value of Authorization, Cookie or Set-Cookie is logged:\n%s",
+			out.text())
+	}
+}
+
+// wantFullLine checks the log line of the request with every header the
+// line looks at. Its timings are checked by TestTimingsAreInOrder.
+func wantFullLine(t *testing.T, line logLine) {
+	t.Helper()
+
+	headers := map[string]string{"accept": "text/html", "x-custom": "one, two"}
+
+	want := withTimings(line, requestlog.Line{
+		Type: requestType, Time: line.Time, Instance: instance,
+		ClientIP: ipv6Client, Method: http.MethodPost, Scheme: secure,
+		Host: appHost, Path: "/full", Protocol: protocol,
+		Status: http.StatusFound, RequestBytes: 3, ResponseBytes: 5,
+		RequestID: "from-traefik", PeerIP: localhost,
+		ForwardedFor: "198.51.100.7, " + ipv6Client, ClientGroup: ipv6Group,
+		ContentType: "application/x-www-form-urlencoded", ContentLength: 3,
+		RequestHeaders: headers, HasAuthorization: true, HasCookie: true,
+		ResponseContentType: "text/html", UpstreamStatus: http.StatusFound,
+		CacheControl: "no-store", Location: "/elsewhere",
+		Action: requestlog.ActionForward,
+		Counts: ratelimit.Counts{Minute: 1, Hour: 1, Day: 1},
+	})
+	if !reflect.DeepEqual(line.Line, want) {
+		t.Errorf("log line\n%+v\nwant\n%+v", line.Line, want)
+	}
+}
+
+// wantBareLine checks the log line of the request with none of them, and
+// that the fields that do not apply to it are left out.
+func wantBareLine(t *testing.T, line logLine) {
+	t.Helper()
+
+	want := withTimings(line, requestlog.Line{
+		Type: requestType, Time: line.Time, Instance: instance,
+		ClientIP: localhost, Method: http.MethodPost, Scheme: plain,
+		Host: appHost, Path: "/bare", Protocol: protocol,
+		Status: http.StatusNoContent, RequestBytes: 4, RequestID: line.RequestID,
+		PeerIP: localhost, ClientGroup: localhost + "/32",
+		UpstreamStatus: http.StatusNoContent, Action: requestlog.ActionForward,
+	})
+	if !reflect.DeepEqual(line.Line, want) || line.RequestID == "" {
+		t.Errorf("log line\n%+v\nwant\n%+v, with a request id", line.Line, want)
+	}
+
+	for _, name := range []string{
+		"forwarded_for", "content_type", "content_length", "request_headers",
+		"has_authorization", "has_cookie", "websocket", "response_content_type",
+		"cache_control", "location", "counts",
+	} {
+		_, present := line.fields[name]
+		if present {
+			t.Errorf("log line has %s, which does not apply", name)
+		}
+	}
+}
+
+// withTimings returns want with the timings of line.
+func withTimings(line logLine, want requestlog.Line) requestlog.Line {
+	want.DurationTotal = line.DurationTotal
+	want.DurationChecks = line.DurationChecks
+	want.DurationUpstreamConnect = line.DurationUpstreamConnect
+	want.DurationUpstreamFirstByte = line.DurationUpstreamFirstByte
+	want.DurationUpstreamTotal = line.DurationUpstreamTotal
+
+	return want
+}
+
+func TestHasAuthorizationAndHasCookieEachComeFromTheirOwnHeader(t *testing.T) {
+	t.Parallel()
+
+	const hasAuthorization, hasCookie = "has_authorization", "has_cookie"
+
+	for _, tc := range []struct{ header, field, other string }{
+		{"Authorization", hasAuthorization, hasCookie},
+		{"Cookie", hasCookie, hasAuthorization},
+	} {
+		t.Run("only "+tc.header, func(t *testing.T) {
+			t.Parallel()
+
+			app := startApp(t, func(http.ResponseWriter, *http.Request) {})
+			addr, out := startProxy(t, app.URL, nil)
+
+			req := newRequest(t, http.MethodGet, addr, "/", http.NoBody)
+			req.Header.Set(tc.header, "secret")
+			wantStatus(t, do(t, req), http.StatusOK)
+
+			line := out.requestLine(t)
+
+			_, otherPresent := line.fields[tc.other]
+			if line.fields[tc.field] != true || otherPresent {
+				t.Errorf("log line has %s %v and %s %v, want true and none",
+					tc.field, line.fields[tc.field], tc.other, line.fields[tc.other])
+			}
+		})
+	}
+}
+
+func TestRequestIDAndSchemeComeOnlyFromATrustedProxy(t *testing.T) {
+	t.Parallel()
+
+	const sentID = "from-traefik"
+
+	sent := http.Header{requestIDHeader: {sentID}, forwardedProto: {secure}}
+	trusted := map[string]string{trustedProxies: trustLocalhost}
+
+	for _, tc := range []struct {
+		name   string
+		env    map[string]string
+		header http.Header
+		// wantID is the request id logged, "" for a new one.
+		wantID, wantScheme string
+	}{
+		{"a trusted proxy's are kept", trusted, sent, sentID, secure},
+		{"without them, the id is new and the scheme http", trusted, nil, "", plain},
+		{"another peer's are replaced", nil, sent, "", plain},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+
+			received := make(chan string, 2)
+			app := startApp(t, func(_ http.ResponseWriter, r *http.Request) {
+				received <- r.Header.Get(requestIDHeader)
+			})
+			addr, out := startProxy(t, app.URL, tc.env)
+
+			// Two requests, so that two new ids can be told apart.
+			ids := make([]string, 0, 2)
+
+			for i := range 2 {
+				req := newRequest(t, http.MethodGet, addr, "/", http.NoBody)
+				maps.Copy(req.Header, tc.header)
+				wantStatus(t, do(t, req), http.StatusOK)
+
+				line := out.requestLines(t, i+1)[i]
+				ids = append(ids, line.RequestID)
+
+				got := <-received
+				if line.RequestID != got || line.Scheme != tc.wantScheme {
+					t.Errorf("log line has request_id %q and scheme %q, and the "+
+						"app received id %q; want the same id and scheme %q",
+						line.RequestID, line.Scheme, got, tc.wantScheme)
+				}
+			}
+
+			switch {
+			case tc.wantID != "" && (ids[0] != tc.wantID || ids[1] != tc.wantID):
+				t.Errorf("request ids %q, want %q", ids, tc.wantID)
+			case tc.wantID == "" && (slices.Contains(ids, sentID) ||
+				slices.Contains(ids, "") || ids[0] == ids[1]):
+				t.Errorf("request ids %q, want two new ones", ids)
+			}
+		})
+	}
+}
+
+func TestTimingsAreInOrder(t *testing.T) {
+	t.Parallel()
+
+	const denied = "192.0.2.50" // in SWWAF_DENY_NETS
+
+	app := startApp(t, func(w http.ResponseWriter, _ *http.Request) {
+		// The pauses set the times apart; a hold-up of the test only
+		// lengthens them.
+		time.Sleep(time.Millisecond)
+		w.WriteHeader(http.StatusOK)
+		_ = http.NewResponseController(w).Flush()
+
+		time.Sleep(time.Millisecond)
+
+		_, _ = io.WriteString(w, "done")
+	})
+	addr, out := startProxy(t, app.URL, map[string]string{
+		trustedProxies: trustLocalhost,
+		denyNets:       denied,
+	})
+
+	// Each log line is waited for before the next request, so that the
+	// lines are in order.
+	wantStatus(t, get(t, addr, "/"), http.StatusOK)
+	forwarded := out.requestLines(t, 1)[0]
+
+	req := newRequest(t, http.MethodGet, addr, "/", http.NoBody)
+	req.Header.Set(forwardedFor, denied)
+	wantStatus(t, do(t, req), http.StatusForbidden)
+	refused := out.requestLines(t, 2)[1]
+
+	wantStatus(t, get(t, addr, proxy.HealthPath), http.StatusOK)
+	health := out.requestLines(t, 3)[2]
+
+	// A request passed to the app has every timing; one refused, none of
+	// the app's; the health check, which runs no check, only the total.
+	wantTimings(t, forwarded, "duration_total", "duration_checks",
+		"duration_upstream_connect", "duration_upstream_first_byte",
+		"duration_upstream_total")
+	wantTimings(t, refused, "duration_total", "duration_checks")
+	wantTimings(t, health, "duration_total")
+
+	if t.Failed() {
+		return
+	}
+
+	// In whole microseconds, as they are logged, so that the sum below is
+	// exact.
+	total := microseconds(forwarded.DurationTotal)
+	checks := microseconds(*forwarded.DurationChecks)
+	connect := microseconds(*forwarded.DurationUpstreamConnect)
+	firstByte := microseconds(*forwarded.DurationUpstreamFirstByte)
+	upstream := microseconds(*forwarded.DurationUpstreamTotal)
+
+	// The checks end before the request is handed to the app, and the
+	// connection comes before the answer, which the app ends after a
+	// pause.
+	if checks+upstream > total || connect >= firstByte || firstByte >= upstream {
+		t.Errorf("timings in microseconds: total %d, checks %d, connect %d, "+
+			"first byte %d, upstream total %d", total, checks, connect, firstByte,
+			upstream)
+	}
+
+	if *refused.DurationChecks > refused.DurationTotal {
+		t.Errorf("refused request's checks took %v of %v milliseconds",
+			*refused.DurationChecks, refused.DurationTotal)
+	}
+}
+
+// wantTimings checks that the timings named are the only ones line has.
+func wantTimings(t *testing.T, line logLine, want ...string) {
+	t.Helper()
+
+	var got []string
+
+	for name := range line.fields {
+		if strings.HasPrefix(name, "duration_") {
+			got = append(got, name)
+		}
+	}
+
+	slices.Sort(got)
+	slices.Sort(want)
+
+	if !slices.Equal(got, want) {
+		t.Errorf("log line of %s has timings %v, want %v", line.Path, got, want)
+	}
+}
+
+// microseconds is a timing in whole microseconds.
+func microseconds(milliseconds float64) int64 {
+	return int64(math.Round(milliseconds * 1000))
+}
+
+func TestLogsAnUpgradedConnection(t *testing.T) {
+	t.Parallel()
+
+	app := startApp(t, echoAfterUpgrade)
+	addr, out := startProxy(t, app.URL, nil)
+
+	conn := dial(t, addr)
+	send(t, conn, "GET /socket HTTP/1.1\r\nHost: app\r\n"+
+		"Connection: Upgrade\r\nUpgrade: websocket\r\n\r\n")
+	wantStatus(t, readResponse(t, conn), http.StatusSwitchingProtocols)
+
+	_ = conn.Close()
+
+	line := out.requestLine(t)
+	if line.fields["websocket"] != true {
+		t.Errorf("log line has websocket %v, want true", line.fields["websocket"])
+	}
+}

internal/proxy/timeouts_test.go

@@ -28,7 +28,9 @@ func TestRequestTimeouts(t *testing.T) {
 
 	for _, tc := range []struct {
 		name string
-		env  map[string]string
+		// limit is the setting set to shortTimeout, which runs out; long
+		// is one set to longTimeoutSetting, which does not, or "".
+		limit, long string
 		// appTakesNothing has the app never read, while the client sends
 		// as fast as it can; otherwise the app reads, and the client
 		// stops sending halfway.
@@ -36,30 +38,26 @@ func TestRequestTimeouts(t *testing.T) {
 		want            int
 	}{
 		{
-			name: "client request timeout, waiting on the client",
-			env:  map[string]string{clientRequestTimeout: shortTimeoutSetting},
-			want: http.StatusRequestTimeout,
+			name:  "client request timeout, waiting on the client",
+			limit: clientRequestTimeout,
+			want:  http.StatusRequestTimeout,
 		},
 		{
-			name: "upstream request timeout, waiting on the client",
-			env: map[string]string{
-				upstreamRequestTimeout: shortTimeoutSetting,
-				clientRequestTimeout:   longTimeoutSetting,
-			},
-			want: http.StatusRequestTimeout,
+			name:  "upstream request timeout, waiting on the client",
+			limit: upstreamRequestTimeout,
+			long:  clientRequestTimeout,
+			want:  http.StatusRequestTimeout,
 		},
 		{
 			name:            "upstream request timeout, waiting on the app",
-			env:             map[string]string{upstreamRequestTimeout: shortTimeoutSetting},
+			limit:           upstreamRequestTimeout,
 			appTakesNothing: true,
 			want:            http.StatusGatewayTimeout,
 		},
 		{
-			name: "client request timeout, waiting on the app",
-			env: map[string]string{
-				clientRequestTimeout:   shortTimeoutSetting,
-				upstreamRequestTimeout: longTimeoutSetting,
-			},
+			name:            "client request timeout, waiting on the app",
+			limit:           clientRequestTimeout,
+			long:            upstreamRequestTimeout,
 			appTakesNothing: true,
 			want:            http.StatusGatewayTimeout,
 		},
@@ -84,7 +82,12 @@ func TestRequestTimeouts(t *testing.T) {
 				appURL, sendRequest = app.URL, sendPartOfBody
 			}
 
-			addr, out := startProxy(t, appURL, tc.env)
+			env := map[string]string{tc.limit: shortTimeoutSetting, metricsToken: token}
+			if tc.long != "" {
+				env[tc.long] = longTimeoutSetting
+			}
+
+			addr, out := startProxy(t, appURL, env)
 			start := time.Now()
 			got := readResponse(t, sendRequest(t, addr))
 			wantTimedOut(t, start)
@@ -105,6 +108,7 @@ func TestRequestTimeouts(t *testing.T) {
 
 			wantStatus(t, got, want)
 			wantLine(t, out.requestLine(t), want, requestlog.ActionTimedOut)
+			wantLimitHits(t, addr, tc.limit, 1)
 		})
 	}
 }
@@ -198,6 +202,7 @@ func TestAppTooSlowToAnswer(t *testing.T) {
 	})
 	addr, out := startProxy(t, app.URL, map[string]string{
 		upstreamResponseTimeout: shortTimeoutSetting,
+		metricsToken:            token,
 	})
 
 	start := time.Now()
@@ -213,6 +218,8 @@ func TestAppTooSlowToAnswer(t *testing.T) {
 		t.Errorf("log line has upstream_status %v for an app that never answered",
 			line.fields["upstream_status"])
 	}
+
+	wantLimitHits(t, addr, upstreamResponseTimeout, 1)
 }
 
 func TestAppTooSlowToFinishItsAnswer(t *testing.T) {
@@ -261,6 +268,7 @@ func TestClientTooSlowToTakeTheAnswer(t *testing.T) {
 	})
 	addr, out := startProxy(t, app.URL, map[string]string{
 		clientResponseTimeout: shortTimeoutSetting,
+		metricsToken:          token,
 	})
 
 	start := time.Now()
@@ -272,6 +280,7 @@ func TestClientTooSlowToTakeTheAnswer(t *testing.T) {
 	line := out.requestLine(t)
 	wantTimedOut(t, start)
 	wantLine(t, line, http.StatusOK, requestlog.ActionTimedOut)
+	wantLimitHits(t, addr, clientResponseTimeout, 1)
 }
 
 func TestClosesAnIdleConnection(t *testing.T) {

internal/ratelimit/history_test.go

@@ -19,26 +19,29 @@ func TestHistoryKeepsEveryRequest(t *testing.T) {
 		{Country: "DE", Forwarded: true, Status: 200, RequestBytes: 10, ResponseBytes: 100},
 		{Forwarded: true, Status: 101},
 		{Forwarded: true, Status: 304, RequestBytes: 5},
-		{Country: "FR", Status: 403, ResponseBytes: 10, BrokeLimit: true},
+		{Country: "FR", Refused: true, Status: 403, ResponseBytes: 10, BrokeLimit: true},
 		{Forwarded: true, Status: 502, ResponseBytes: 12},
 		// Closed without an answer: refused, and no response.
-		{Status: 0},
+		{Refused: true, Status: 0},
+		// Answered 404 at smallwebwaf's own endpoints: neither forwarded
+		// nor refused.
+		{Status: 404},
 	} {
 		limiter.AddToHistory(client, start.Add(time.Duration(i)*time.Minute), r)
 	}
 
 	want := ratelimit.History{
 		FirstSeen:     start,
-		LastSeen:      start.Add(5 * time.Minute),
+		LastSeen:      start.Add(6 * time.Minute),
 		Country:       "FR",
 		LookedUp:      start.Add(3 * time.Minute),
-		Requests:      6,
+		Requests:      7,
 		Forwarded:     4,
 		Refused:       2,
 		RequestBytes:  15,
 		ResponseBytes: 122,
 		Responses: ratelimit.Responses{
-			Status1xx: 1, Status2xx: 1, Status3xx: 1, Status4xx: 1, Status5xx: 1,
+			Status1xx: 1, Status2xx: 1, Status3xx: 1, Status4xx: 2, Status5xx: 1,
 		},
 		Offences: ratelimit.Offences{Limit: 1},
 	}

internal/ratelimit/ratelimit.go

@@ -71,7 +71,9 @@ type History struct {
 	Country  string    `json:"country,omitempty"`
 	LookedUp time.Time `json:"looked_up,omitzero"`
 	// Requests are all the client's requests: Forwarded those passed to
-	// the app, Refused those refused before anything reached it.
+	// the app, Refused those refused before anything reached it, a 401 at
+	// smallwebwaf's own endpoints included, and neither the others
+	// smallwebwaf answered there.
 	Requests  int64 `json:"requests"`
 	Forwarded int64 `json:"forwarded"`
 	Refused   int64 `json:"refused"`
@@ -103,9 +105,12 @@ type Offences struct {
 type Request struct {
 	// Country is the client's country, when the request looked it up.
 	Country string
-	// Forwarded is true for a request passed to the app, false for one
-	// refused before anything reached it.
+	// Forwarded is true for a request passed to the app, Refused for one
+	// refused before anything reached it, a 401 at smallwebwaf's own
+	// endpoints included. Both are false for any other request smallwebwaf
+	// answered there.
 	Forwarded bool
+	Refused   bool
 	// Status is what the client was sent, 0 if nothing was.
 	Status int
 	// RequestBytes and ResponseBytes are the body bytes of the request
@@ -144,26 +149,39 @@ type Hit struct {
 	Requests float64
 }
 
+// Counts are a client's requests in the minute, the hour and the day that
+// end at a request, that request included.
+type Counts struct {
+	Minute float64 `json:"minute"`
+	Hour   float64 `json:"hour"`
+	Day    float64 `json:"day"`
+}
+
 // Count counts a request from client at now, in every window, whether or
-// not it is refused. It reports whether the request takes the client over
-// a limit, and the window whose limit it goes over, the shortest if it is
-// over several.
-func (l *Limiter) Count(client netip.Prefix, now time.Time) (Hit, bool) {
+// not it is refused, and returns the client's requests in each window. It
+// reports whether the request takes the client over a limit, and the
+// window whose limit it goes over, the shortest if it is over several.
+func (l *Limiter) Count(client netip.Prefix, now time.Time) (Counts, Hit, bool) {
 	l.mu.Lock()
 	defer l.mu.Unlock()
 
-	var hit Hit
+	var (
+		requests [3]float64
+		hit      Hit
+	)
 
 	for i, b := range l.get(client).buckets() {
 		w := l.windows[i]
 
-		requests := b.add(now, w.length)
-		if hit.Window == "" && w.limit > 0 && requests > float64(w.limit) {
-			hit = Hit{Window: w.name, Limit: w.limit, Requests: requests}
+		requests[i] = b.add(now, w.length)
+		if hit.Window == "" && w.limit > 0 && requests[i] > float64(w.limit) {
+			hit = Hit{Window: w.name, Limit: w.limit, Requests: requests[i]}
 		}
 	}
 
-	return hit, hit.Window != ""
+	counts := Counts{Minute: requests[0], Hour: requests[1], Day: requests[2]}
+
+	return counts, hit, hit.Window != ""
 }
 
 // Reset sets client's counts in every window back to zero. Its history
@@ -199,7 +217,9 @@ func (l *Limiter) AddToHistory(client netip.Prefix, now time.Time, r Request) {
 	h.Requests++
 	if r.Forwarded {
 		h.Forwarded++
-	} else {
+	}
+
+	if r.Refused {
 		h.Refused++
 	}
 
@@ -235,6 +255,14 @@ func (l *Limiter) Requests(netblock netip.Prefix) int64 {
 	return requests
 }
 
+// Len returns how many clients are in the table.
+func (l *Limiter) Len() int {
+	l.mu.Lock()
+	defer l.mu.Unlock()
+
+	return l.clients.Len()
+}
+
 // Snapshot returns every client in the table, sorted by address, as
 // clients.json lists them.
 func (l *Limiter) Snapshot() []Client {
@@ -254,18 +282,21 @@ func (l *Limiter) Snapshot() []Client {
 	return clients
 }
 
-// Load puts clients read from clients.json into a table that holds none
-// yet, in the order they were last seen, so that the least recently seen
-// is dropped first. Buckets whose time has passed at now are emptied.
+// Load puts clients read from clients.json into the table, in place of
+// the clients it holds, in the order they were last seen, so that the
+// least recently seen is dropped first. Buckets whose time has passed at
+// now are emptied.
 func (l *Limiter) Load(clients []Client, now time.Time) {
-	l.mu.Lock()
-	defer l.mu.Unlock()
-
 	clients = slices.Clone(clients)
 	slices.SortStableFunc(clients, func(a, b Client) int {
 		return a.History.LastSeen.Compare(b.History.LastSeen)
 	})
 
+	l.mu.Lock()
+	defer l.mu.Unlock()
+
+	l.clients.Purge()
+
 	for _, c := range clients {
 		for i, b := range c.buckets() {
 			// The window that ends at now covers neither bucket once it

internal/ratelimit/ratelimit_test.go

@@ -62,14 +62,14 @@ func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
 	start := midnight()
 
 	for range limit {
-		_, over := limiter.Count(client, start)
+		_, _, over := limiter.Count(client, start)
 		if over {
 			t.Fatal("a request within the limit is over it")
 		}
 	}
 
 	// Over both limits; the minute's is named, with the four requests.
-	hit, over := limiter.Count(client, start)
+	_, hit, over := limiter.Count(client, start)
 
 	want := ratelimit.Hit{Window: minute, Limit: limit, Requests: limit + 1}
 	if !over || hit != want {
@@ -78,6 +78,29 @@ func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
 	}
 }
 
+func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
+	t.Parallel()
+
+	limiter := ratelimit.New(ratelimit.Limits{})
+	client := netip.MustParsePrefix("203.0.113.9/32")
+	start := midnight()
+
+	for range 3 {
+		limiter.Count(client, start)
+	}
+
+	// A quarter into the next hour, the minute has only this request. The
+	// hour still covers three quarters of the bucket before, with its three
+	// requests, which count 2.25, and this one: 3.25. The day covers all
+	// four.
+	counts, _, _ := limiter.Count(client, start.Add(time.Hour+time.Hour/4))
+
+	want := ratelimit.Counts{Minute: 1, Hour: 3.25, Day: 4}
+	if counts != want {
+		t.Errorf("counts %+v, want %+v", counts, want)
+	}
+}
+
 func TestResetSetsTheCountsBackToZero(t *testing.T) {
 	t.Parallel()
 
@@ -238,7 +261,7 @@ func wantCount(
 ) {
 	t.Helper()
 
-	hit, _ := limiter.Count(client, now)
+	_, hit, _ := limiter.Count(client, now)
 	if hit.Window != want {
 		t.Errorf("request from %s at %s is over %q, want %q",
 			client, now.Format(time.RFC3339), hit.Window, want)

internal/requestlog/requestlog.go

@@ -9,6 +9,8 @@ import (
 	"io"
 	"log/slog"
 	"time"
+
+	"sneak.berlin/go/smallwebwaf/internal/ratelimit"
 )
 
 // The action a request line names: what smallwebwaf did with the
@@ -45,28 +47,71 @@ const OffenceLimit = "limit"
 // timeLayout is RFC 3339 with milliseconds.
 const timeLayout = "2006-01-02T15:04:05.000Z07:00"
 
-// Line is one request's line in the request log. The field names are
-// those of the "Request log" section of SPEC.md.
+// Line is one request's line in the request log. The field names, and
+// their order, are those of the "Request log" section of SPEC.md. A field
+// that may not apply to a request is left out of its line when it does
+// not.
 //
 //nolint:tagliatelle // SPEC.md's request log names its fields in snake_case
 type Line struct {
-	Type           string `json:"type"`
-	Time           string `json:"time"`
-	ClientIP       string `json:"client_ip"`
-	PeerIP         string `json:"peer_ip"`
-	Country        string `json:"country"`
-	Method         string `json:"method"`
-	Host           string `json:"host"`
-	Path           string `json:"path"`
-	Query          string `json:"query"`
-	Protocol       string `json:"protocol"`
-	Status         int    `json:"status"`
-	UpstreamStatus int    `json:"upstream_status,omitempty"`
-	RequestBytes   int64  `json:"request_bytes"`
-	ResponseBytes  int64  `json:"response_bytes"`
-	Referer        string `json:"referer"`
-	UserAgent      string `json:"user_agent"`
-	Action         string `json:"action"`
+	Type string `json:"type"`
+
+	// The standard web log fields. Scheme is how the client reached
+	// smallwebwaf, or the trusted proxy in front of it.
+	Time          string `json:"time"`
+	Instance      string `json:"instance"`
+	ClientIP      string `json:"client_ip"`
+	Method        string `json:"method"`
+	Scheme        string `json:"scheme"`
+	Host          string `json:"host"`
+	Path          string `json:"path"`
+	Query         string `json:"query"`
+	Protocol      string `json:"protocol"`
+	Status        int    `json:"status"`
+	RequestBytes  int64  `json:"request_bytes"`
+	ResponseBytes int64  `json:"response_bytes"`
+	Referer       string `json:"referer"`
+	UserAgent     string `json:"user_agent"`
+
+	// Request detail. RequestID is the X-Request-ID a trusted proxy sent,
+	// or a new one, and is sent on to the app. ForwardedFor is the
+	// X-Forwarded-For header as received. ClientGroup is the netblock the
+	// client is counted as.
+	RequestID    string `json:"request_id"`
+	PeerIP       string `json:"peer_ip"`
+	ForwardedFor string `json:"forwarded_for,omitempty"`
+	ClientGroup  string `json:"client_group"`
+	Country      string `json:"country"`
+	ContentType  string `json:"content_type,omitempty"`
+	// ContentLength is the length of its body the request announced.
+	ContentLength int64 `json:"content_length,omitempty"`
+	// RequestHeaders are the headers SWWAF_LOG_REQUEST_HEADERS names that
+	// the request carried, by name in lower case.
+	RequestHeaders   map[string]string `json:"request_headers,omitempty"`
+	HasAuthorization bool              `json:"has_authorization,omitempty"`
+	HasCookie        bool              `json:"has_cookie,omitempty"`
+	// Websocket is true when the connection was upgraded, as for a
+	// WebSocket.
+	Websocket bool `json:"websocket,omitempty"`
+
+	// Response detail, from the headers of the answer: the app's, as
+	// passed on, or those of smallwebwaf's own. Aborted is true when the
+	// client went away early.
+	ResponseContentType string `json:"response_content_type,omitempty"`
+	UpstreamStatus      int    `json:"upstream_status,omitempty"`
+	CacheControl        string `json:"cache_control,omitempty"`
+	Location            string `json:"location,omitempty"`
+	Aborted             bool   `json:"aborted,omitempty"`
+
+	// The decision.
+	Action string `json:"action"`
+	// WouldAction is, in observe mode, the action enforce mode would have
+	// taken with a request it would have refused: ActionDenied,
+	// ActionBanned, ActionCountryDenied or ActionRateLimited.
+	WouldAction string `json:"would_action,omitempty"`
+	// Counts are the client's requests as the rate limits counted them
+	// with this one, for a request they counted.
+	Counts ratelimit.Counts `json:"counts,omitzero"`
 	// LimitHit is the window whose rate limit the request went over:
 	// minute, hour or day.
 	LimitHit string `json:"limit_hit,omitempty"`
@@ -75,11 +120,18 @@ type Line struct {
 	// BanExpires is when the ban the request made, or was refused under,
 	// ends: a time, or "permanent".
 	BanExpires string `json:"ban_expires,omitempty"`
-	// Aborted is true when the client went away early.
-	Aborted bool `json:"aborted,omitempty"`
-	// DurationTotal and DurationUpstreamTotal are in milliseconds.
-	DurationTotal         float64 `json:"duration_total"`
-	DurationUpstreamTotal float64 `json:"duration_upstream_total,omitempty"`
+
+	// The timings, in milliseconds. DurationChecks is the time until the
+	// checks were done. DurationUpstreamConnect, DurationUpstreamFirstByte
+	// and DurationUpstreamTotal run from when the request was handed to the
+	// app: until there was a connection to it, until the first byte of its
+	// answer arrived, and until the end. Each but DurationTotal is nil for
+	// a request that did not get that far.
+	DurationTotal             float64  `json:"duration_total"`
+	DurationChecks            *float64 `json:"duration_checks,omitempty"`
+	DurationUpstreamConnect   *float64 `json:"duration_upstream_connect,omitempty"`
+	DurationUpstreamFirstByte *float64 `json:"duration_upstream_first_byte,omitempty"`
+	DurationUpstreamTotal     *float64 `json:"duration_upstream_total,omitempty"`
 }
 
 // Write writes line to w as one JSON line marked "type":"request".

internal/requestlog/requestlog_test.go

@@ -50,7 +50,11 @@ func TestWriteWritesOneJSONLineMarkedRequest(t *testing.T) {
 	}
 
 	unset := []string{
-		"upstream_status", "limit_hit", "offence", "ban_expires", "aborted",
+		"forwarded_for", "content_type", "content_length", "request_headers",
+		"has_authorization", "has_cookie", "websocket", "response_content_type",
+		"upstream_status", "cache_control", "location", "aborted", "counts",
+		"limit_hit", "offence", "ban_expires", "duration_checks",
+		"duration_upstream_connect", "duration_upstream_first_byte",
 		"duration_upstream_total",
 	}
 	for _, name := range unset {

internal/smallwebwaf/smallwebwaf.go

@@ -89,6 +89,7 @@ func Run(ctx context.Context, params Params) int {
 		GeoJS:           server.GeoJS,
 		Now:             now,
 		ProcessLog:      processLog,
+		Metrics:         server.Metrics,
 	})
 	if err != nil {
 		processLog.Error("cannot use the state files", "error", err.Error())
@@ -112,9 +113,10 @@ func Run(ctx context.Context, params Params) int {
 	return serve(ctx, server.Server, listener, files, processLog)
 }
 
-// serve serves requests on listener, and writes the state files as they
-// are due, until ctx is done. Then it gives the requests in progress
-// shutdownTimeout to finish, and writes every state file.
+// serve serves requests on listener, writes the state files as they are
+// due, and takes in an admin's edits of them, until ctx is done. Then it
+// gives the requests in progress shutdownTimeout to finish, and writes
+// every state file.
 func serve(
 	ctx context.Context, server *http.Server, listener net.Listener,
 	files *state.Files, processLog *slog.Logger,
@@ -129,12 +131,18 @@ func serve(
 	defer stopWriting()
 
 	written := make(chan struct{})
+	watched := make(chan struct{})
 
 	go func() {
 		files.Run(writing)
 		close(written)
 	}()
 
+	go func() {
+		files.Watch(writing)
+		close(watched)
+	}()
+
 	select {
 	case err := <-served:
 		processLog.Error("serving failed", "error", err.Error())
@@ -164,9 +172,15 @@ func serve(
 		return 1
 	}
 
-	// Every request has ended, and so has Run's last write: the files
-	// written now hold everything, and nothing else writes them.
+	// Run and Watch have ended, so nothing else reads or writes the
+	// files. Every request has ended too, but for two kinds
+	// that Go's server does not wait for: one cut off because Shutdown
+	// timed out, and one whose connection switched protocols, such as a
+	// WebSocket. Such a request adds to its client's history only as it
+	// ends, which can be after this write, and then that request is
+	// missing from clients.json.
 	<-written
+	<-watched
 
 	err = files.WriteAll()
 	if err != nil {

internal/smallwebwaf/smallwebwaf_test.go

@@ -26,10 +26,14 @@ const (
 	// testVersion is the version the tests give smallwebwaf.
 	testVersion = "test"
 	// localhost is where the tests listen.
-	localhost   = "127.0.0.1"
-	listenAddr  = "SWWAF_LISTEN_ADDR"
-	upstreamURL = "SWWAF_UPSTREAM_URL"
-	stateDir    = "SWWAF_STATE_DIR"
+	localhost            = "127.0.0.1"
+	listenAddr           = "SWWAF_LISTEN_ADDR"
+	upstreamURL          = "SWWAF_UPSTREAM_URL"
+	trustedProxies       = "SWWAF_TRUSTED_PROXIES"
+	stateDir             = "SWWAF_STATE_DIR"
+	stateWriteDelay      = "SWWAF_STATE_WRITE_DELAY"
+	stateCounterInterval = "SWWAF_STATE_COUNTER_INTERVAL"
+	rateLimitPerDay      = "SWWAF_RATE_LIMIT_PER_DAY"
 	// greeting is what the tests' app answers.
 	greeting = "hello from the app"
 )
@@ -120,6 +124,28 @@ func TestInvalidSettingStopsTheStart(t *testing.T) {
 	}
 }
 
+func TestShortMetricsTokenStopsTheStartUnshown(t *testing.T) {
+	t.Parallel()
+
+	const token = "a-token-of-31-characters-at-all" //nolint:gosec // too short to use
+
+	out := &output{}
+
+	status := run(t.Context(), map[string]string{"SWWAF_METRICS_TOKEN": token}, out)
+	if status != 1 {
+		t.Errorf("exit status %d, want 1", status)
+	}
+
+	line := out.line(t, "msg", "invalid setting")
+	if line["error"] != "SWWAF_METRICS_TOKEN: is shorter than 32 characters" {
+		t.Errorf("start refused with %v", line)
+	}
+
+	if strings.Contains(out.text(), token) {
+		t.Errorf("the output shows the token:\n%s", out.text())
+	}
+}
+
 func TestAddressInUseStopsTheStart(t *testing.T) {
 	t.Parallel()
 
@@ -188,14 +214,14 @@ func TestStateKeptAcrossRestarts(t *testing.T) {
 	t.Parallel()
 
 	env := map[string]string{
-		listenAddr:                 localhost + ":0",
-		upstreamURL:                startApp(t),
-		stateDir:                   t.TempDir(),
-		"SWWAF_RATE_LIMIT_PER_DAY": "2",
+		listenAddr:      localhost + ":0",
+		upstreamURL:     startApp(t),
+		stateDir:        t.TempDir(),
+		rateLimitPerDay: "2",
 		// Neither comes due in the test: the files are written as
 		// smallwebwaf stops.
-		"SWWAF_STATE_WRITE_DELAY":      "1h",
-		"SWWAF_STATE_COUNTER_INTERVAL": "1h",
+		stateWriteDelay:      "1h",
+		stateCounterInterval: "1h",
 	}
 
 	// The two requests a day allows, and a stop.
@@ -218,6 +244,78 @@ func TestStateKeptAcrossRestarts(t *testing.T) {
 	out.line(t, "action", "banned")
 }
 
+func TestBanRefusesItsNetblockAfterARestartWithAnotherScope(t *testing.T) {
+	t.Parallel()
+
+	const scope = "SWWAF_BAN_SCOPE_V4_PREFIX"
+
+	env := map[string]string{
+		listenAddr:      localhost + ":0",
+		upstreamURL:     startApp(t),
+		stateDir:        t.TempDir(),
+		trustedProxies:  localhost + "/32",
+		rateLimitPerDay: "1",
+		scope:           "24",
+	}
+
+	// 203.0.113.9's second request breaks the day limit, and bans
+	// 203.0.113.0/24.
+	runUntilStopped(t, env, func(url string) {
+		wantStatus(t, url, "203.0.113.9", http.StatusOK)
+		wantStatus(t, url, "203.0.113.9", http.StatusForbidden)
+	})
+
+	// With each address a netblock of its own after a restart, that ban
+	// still refuses all of 203.0.113.0/24. 198.51.100.7 is banned alone.
+	env[scope] = "32"
+	runUntilStopped(t, env, func(url string) {
+		wantStatus(t, url, "203.0.113.200", http.StatusForbidden)
+		wantStatus(t, url, "203.0.114.1", http.StatusOK)
+		wantStatus(t, url, "198.51.100.7", http.StatusOK)
+		wantStatus(t, url, "198.51.100.7", http.StatusForbidden)
+	})
+
+	// With /24 netblocks again, that ban still refuses 198.51.100.7, and
+	// no other address.
+	env[scope] = "24"
+	runUntilStopped(t, env, func(url string) {
+		wantStatus(t, url, "198.51.100.7", http.StatusForbidden)
+		wantStatus(t, url, "198.51.100.8", http.StatusOK)
+	})
+}
+
+func TestBanAddedAndLiftedByEditingBansJSON(t *testing.T) {
+	t.Parallel()
+
+	const (
+		// bans.json as an admin writes it with a ban, permanent, on
+		// 203.0.113.0/24, and with none.
+		oneBan = `{"version": 1, "bans": [{"netblock": "203.0.113.0/24", ` +
+			`"start": "2026-10-06T00:00:00Z", "expires": null}]}`
+		noBan = `{"version": 1, "bans": []}`
+	)
+
+	dir := t.TempDir()
+	env := map[string]string{
+		listenAddr:     localhost + ":0",
+		upstreamURL:    startApp(t),
+		stateDir:       dir,
+		trustedProxies: localhost + "/32",
+		// No write comes due in the test, so only the watch on the
+		// directory can take the edits in.
+		stateWriteDelay:      "1h",
+		stateCounterInterval: "1h",
+	}
+
+	runUntilStopped(t, env, func(url string) {
+		path := filepath.Join(dir, "bans.json")
+
+		saveUntilAnswered(t, path, oneBan, url, "203.0.113.9", http.StatusForbidden)
+		wantStatus(t, url, "198.51.100.7", http.StatusOK)
+		saveUntilAnswered(t, path, noBan, url, "203.0.113.9", http.StatusOK)
+	})
+}
+
 func TestStateFileThatDoesNotParseStopsTheStart(t *testing.T) {
 	t.Parallel()
 
@@ -320,9 +418,10 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL, dir string) {
 		listenAddr:                              localhost + ":0",
 		upstreamURL:                             appURL,
 		stateDir:                                dir,
-		"SWWAF_STATE_WRITE_DELAY":               "10s",
-		"SWWAF_STATE_COUNTER_INTERVAL":          "15m",
-		"SWWAF_TRUSTED_PROXIES":                 "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16",
+		"SWWAF_MODE":                            "enforce",
+		stateWriteDelay:                         "10s",
+		stateCounterInterval:                    "15m",
+		trustedProxies:                          "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16",
 		"SWWAF_CLIENT_REQUEST_TIMEOUT":          "60s",
 		"SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES": "32K",
 		"SWWAF_CLIENT_IDLE_TIMEOUT":             "120s",
@@ -336,7 +435,8 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL, dir string) {
 		"SWWAF_DENY_NETS":                       "",
 		"SWWAF_RATE_LIMIT_PER_MINUTE":           "1000",
 		"SWWAF_RATE_LIMIT_PER_HOUR":             "10000",
-		"SWWAF_RATE_LIMIT_PER_DAY":              "50000",
+		rateLimitPerDay:                         "50000",
+		"SWWAF_RATE_LIMIT_EXEMPT_PATHS":         "",
 		"SWWAF_DENIED_COUNTRIES":                "",
 		"SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES":   "",
 		"SWWAF_BAN_RESPONSE":                    "403",
@@ -409,3 +509,63 @@ func wantRefused(t *testing.T, url string) {
 		t.Errorf("status %d, want %d", res.StatusCode, http.StatusForbidden)
 	}
 }
+
+// wantStatus checks that a request to url from the client at from, as
+// X-Forwarded-For names it, is answered with status.
+func wantStatus(t *testing.T, url, from string, status int) {
+	t.Helper()
+
+	got := statusFrom(t, url, from)
+	if got != status {
+		t.Errorf("request from %s: status %d, want %d", from, got, status)
+	}
+}
+
+// saveUntilAnswered writes content to the state file at path, as an
+// admin saves an edit of it, until a request to url from the client at
+// from is answered with status. The file is written again before each
+// request, since smallwebwaf may not watch its directory yet when it is
+// first written. It waits as long as that takes, so that a slow test
+// process cannot fail the test.
+func saveUntilAnswered(t *testing.T, path, content, url, from string, status int) {
+	t.Helper()
+
+	for {
+		err := os.WriteFile(path, []byte(content), 0o600)
+		if err != nil {
+			t.Fatalf("write %s: %v", path, err)
+		}
+
+		if statusFrom(t, url, from) == status {
+			return
+		}
+
+		time.Sleep(pollInterval)
+	}
+}
+
+// statusFrom returns the status a request to url from the client at
+// from, as X-Forwarded-For names it, is answered with.
+func statusFrom(t *testing.T, url, from string) int {
+	t.Helper()
+
+	req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, url,
+		http.NoBody)
+	if err != nil {
+		t.Fatalf("new request: %v", err)
+	}
+
+	req.Header.Set("X-Forwarded-For", from)
+
+	transport := &http.Transport{}
+	defer transport.CloseIdleConnections()
+
+	res, err := (&http.Client{Transport: transport}).Do(req)
+	if err != nil {
+		t.Fatalf("request: %v", err)
+	}
+
+	_ = res.Body.Close()
+
+	return res.StatusCode
+}

internal/state/state.go

@@ -1,14 +1,17 @@
 // Package state keeps smallwebwaf's state in JSON files in
 // SWWAF_STATE_DIR, as the "Persistent state" section of SPEC.md describes:
 // bans.json holds the bans, clients.json each client's counters and
-// history, and lookups.json GeoJS's answers. Load reads them at start, and
-// Run and WriteAll write them, each from a snapshot its part takes under
-// its own lock, so that no request waits on the disk.
+// history, and lookups.json GeoJS's answers. Load reads them at start,
+// Watch takes in an admin's edit of one while smallwebwaf runs, and Run
+// and WriteAll write them. The disk is read and written outside the
+// parts' locks, which are held only to take a snapshot or to put in what
+// a file holds, so that no request waits on the disk.
 package state
 
 import (
 	"bytes"
 	"context"
+	"crypto/sha256"
 	"encoding/json"
 	"errors"
 	"fmt"
@@ -17,10 +20,13 @@ import (
 	"net/netip"
 	"os"
 	"path/filepath"
+	"sync"
 	"time"
 
+	"github.com/fsnotify/fsnotify"
 	"sneak.berlin/go/smallwebwaf/internal/bans"
 	"sneak.berlin/go/smallwebwaf/internal/lookup"
+	"sneak.berlin/go/smallwebwaf/internal/metrics"
 	"sneak.berlin/go/smallwebwaf/internal/ratelimit"
 )
 
@@ -38,7 +44,11 @@ const (
 	lookupsJSON = "lookups.json"
 )
 
-var errVersion = errors.New("unknown version")
+var (
+	errVersion = errors.New("unknown version")
+	// errMissing is for an entry without a field it needs.
+	errMissing = errors.New("has no")
+)
 
 // Params are what Load needs.
 type Params struct {
@@ -56,13 +66,26 @@ type Params struct {
 	// Now tells the time by which the counters' buckets run out, normally
 	// time.Now in UTC.
 	Now func() time.Time
-	// ProcessLog receives what was read, and the writes that fail.
+	// ProcessLog receives what was read and taken in, the edits set aside,
+	// and the writes that fail.
 	ProcessLog *slog.Logger
+	// Metrics count each file's writes, and the edits taken in and set
+	// aside.
+	Metrics *metrics.Metrics
 }
 
 // Files are the state files of a running smallwebwaf.
 type Files struct {
 	params Params
+
+	// mu is held while a file is read for an edit, and while it is
+	// written, so that Watch and the writes take turns. No request takes
+	// it.
+	mu sync.Mutex
+	// sums are the SHA-256 sums of what each file held, by name, when
+	// smallwebwaf last read or wrote it. A file that holds anything else
+	// has been edited since.
+	sums map[string][sha256.Size]byte
 }
 
 // bansFile is bans.json, indented for an admin to read and edit.
@@ -92,52 +115,49 @@ type lookupsFile struct {
 	Lookups []lookup.Answer `json:"lookups"`
 }
 
+// stateFile is the struct of a state file. Once the file is decoded, its
+// check refuses the first entry without a field it needs, which would
+// otherwise be read as something the entry does not say. data is the
+// file, for a field that may be null or "" but not left out, which the
+// struct cannot tell apart.
+type stateFile interface {
+	check(data []byte) error
+}
+
 // Load checks that files can be written in Dir, and reads the state files
 // in it into the ledger, the limiter and GeoJS. A missing file is empty
-// state, as on a first start. A file that does not parse, or has an
-// unknown version, is an error that names the file and, where the JSON
-// decoder tells it, the line and column.
+// state, as on a first start. A file that does not parse, has an unknown
+// version, or has an entry without a field it needs, is an error that
+// names the file and, where the JSON decoder tells it, the line and
+// column, or else the entry.
 func Load(params Params) (*Files, error) {
 	err := checkWritable(params.Dir)
 	if err != nil {
 		return nil, fmt.Errorf("SWWAF_STATE_DIR cannot be written: %w", err)
 	}
 
-	var (
-		bansIn    bansFile
-		clientsIn clientsFile
-		lookupsIn lookupsFile
-	)
+	f := &Files{params: params, sums: map[string][sha256.Size]byte{}}
 
-	err = errors.Join(
-		read(params.Dir, bansJSON, &bansIn),
-		read(params.Dir, clientsJSON, &clientsIn),
-		read(params.Dir, lookupsJSON, &lookupsIn),
-	)
+	bansRead, bansErr := f.read(bansJSON)
+	clientsRead, clientsErr := f.read(clientsJSON)
+	lookupsRead, lookupsErr := f.read(lookupsJSON)
+
+	err = errors.Join(bansErr, clientsErr, lookupsErr)
 	if err != nil {
 		return nil, err
 	}
 
-	held := make([]bans.Ban, 0, len(bansIn.Bans))
-	for _, entry := range bansIn.Bans {
-		held = append(held, entry.ban())
-	}
-
-	params.Ledger.Load(held)
-	params.Limiter.Load(clientsIn.Clients, params.Now())
-	params.GeoJS.Load(lookupsIn.Lookups)
-
 	params.ProcessLog.Info("read the state files", "directory", params.Dir,
-		"bans", len(bansIn.Bans), "clients", len(clientsIn.Clients),
-		"lookups", len(lookupsIn.Lookups))
+		"bans", bansRead, "clients", clientsRead, "lookups", lookupsRead)
 
-	return &Files{params: params}, nil
+	return f, nil
 }
 
 // Run writes bans.json WriteDelay after a ban is made, with every ban
 // made in between, and every file every CounterInterval, until ctx is
 // done. A write that fails is logged, and the file is written again at
-// its next write.
+// its next write. Each write takes in an admin's edit of its file first,
+// as writeFile describes.
 func (f *Files) Run(ctx context.Context) {
 	interval := time.NewTicker(f.params.CounterInterval)
 	defer interval.Stop()
@@ -155,7 +175,7 @@ func (f *Files) Run(ctx context.Context) {
 		case <-bansDue:
 			bansDue = nil
 
-			f.logFailure(f.writeBans())
+			f.logFailure(f.writeFile(bansJSON))
 		case <-interval.C:
 			f.logFailure(f.WriteAll())
 		}
@@ -165,7 +185,50 @@ func (f *Files) Run(ctx context.Context) {
 // WriteAll writes every state file, as smallwebwaf stops. A file that
 // fails does not keep the others from being written.
 func (f *Files) WriteAll() error {
-	return errors.Join(f.writeBans(), f.writeClients(), f.writeLookups())
+	return errors.Join(f.writeFile(bansJSON), f.writeFile(clientsJSON),
+		f.writeFile(lookupsJSON))
+}
+
+// Watch watches Dir until ctx is done, and takes in an admin's edit of a
+// state file as soon as it is saved: what the file holds replaces what
+// smallwebwaf held for it. An edit that does not parse is left for the
+// file's next write, which sets it aside, since a file can be read while
+// an editor is still writing it. If Dir cannot be watched, that is
+// logged, and an edit is taken in only before its file is written.
+func (f *Files) Watch(ctx context.Context) {
+	watcher, err := fsnotify.NewWatcher()
+	if err == nil {
+		defer func() {
+			_ = watcher.Close()
+		}()
+
+		err = watcher.Add(f.params.Dir)
+	}
+
+	if err != nil {
+		f.params.ProcessLog.Error("cannot watch the state files for edits",
+			"error", err.Error())
+
+		return
+	}
+
+	f.params.ProcessLog.Info("watching the state files for edits",
+		"directory", f.params.Dir)
+
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case event := <-watcher.Events:
+			switch name := filepath.Base(event.Name); name {
+			case bansJSON, clientsJSON, lookupsJSON:
+				f.fileChanged(name)
+			}
+		case err = <-watcher.Errors:
+			f.params.ProcessLog.Warn("watching the state files failed",
+				"error", err.Error())
+		}
+	}
 }
 
 // logFailure logs a write that failed.
@@ -176,41 +239,207 @@ func (f *Files) logFailure(err error) {
 	}
 }
 
-// writeBans writes bans.json.
-func (f *Files) writeBans() error {
-	held := f.params.Ledger.Snapshot()
+// fileChanged takes in what the state file name holds, as Watch sees it
+// change, if that is an edit made since smallwebwaf last read or wrote
+// the file. A file that cannot be read or does not parse is left for its
+// next write.
+func (f *Files) fileChanged(name string) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
 
-	file := bansFile{Version: version, Bans: make([]banEntry, 0, len(held))}
-	for _, ban := range held {
-		file.Bans = append(file.Bans, newBanEntry(ban))
+	data, changed, err := f.readChanged(name)
+	if err != nil || !changed {
+		return
 	}
 
-	data, err := json.MarshalIndent(file, "", "    ")
-	if err != nil {
-		return fmt.Errorf("encode %s: %w", bansJSON, err)
-	}
-
-	return write(f.params.Dir, bansJSON, append(data, '\n'))
+	_ = f.takeInEdit(name, data)
 }
 
-// writeClients writes clients.json.
-func (f *Files) writeClients() error {
-	data, err := encodeOnePerLine("clients", f.params.Limiter.Snapshot())
+// takeInEdit takes in data, an edit of the state file name, as takeIn
+// does, and counts and logs it. Every edit taken in while smallwebwaf
+// runs, by Watch or by a write, is taken in here. An edit that does not
+// parse is neither counted nor logged, and takeIn's error returned.
+func (f *Files) takeInEdit(name string, data []byte) error {
+	_, err := f.takeIn(name, data)
 	if err != nil {
-		return fmt.Errorf("encode %s: %w", clientsJSON, err)
+		return err
 	}
 
-	return write(f.params.Dir, clientsJSON, data)
+	// Counted before it is logged, so that the count is there once the
+	// log line is.
+	f.params.Metrics.StateFileEditTakenIn(name)
+	f.params.ProcessLog.Info("took in an edit of a state file",
+		"file", filepath.Join(f.params.Dir, name))
+
+	return nil
 }
 
-// writeLookups writes lookups.json.
-func (f *Files) writeLookups() error {
-	data, err := encodeOnePerLine("lookups", f.params.GeoJS.Snapshot())
-	if err != nil {
-		return fmt.Errorf("encode %s: %w", lookupsJSON, err)
+// read takes in the state file name at start, and returns how many
+// entries it holds. A missing file holds none.
+func (f *Files) read(name string) (int, error) {
+	data, changed, err := f.readChanged(name)
+	if err != nil || !changed {
+		return 0, err
 	}
 
-	return write(f.params.Dir, lookupsJSON, data)
+	return f.takeIn(name, data)
+}
+
+// readChanged returns what the state file name holds, and whether that
+// has changed since smallwebwaf last read or wrote the file, as it has
+// for a file smallwebwaf never read or wrote. A missing file has not
+// changed: it is written again at its next write.
+func (f *Files) readChanged(name string) ([]byte, bool, error) {
+	path := filepath.Join(f.params.Dir, name)
+
+	data, err := os.ReadFile(path) //nolint:gosec // a state file, in SWWAF_STATE_DIR
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil, false, nil
+	}
+
+	if err != nil {
+		return nil, false, err
+	}
+
+	return data, sha256.Sum256(data) != f.sums[name], nil
+}
+
+// takeIn parses data, what the state file name holds, puts it into the
+// part that keeps that state, in place of what the part held, and returns
+// how many entries the file holds. An error names the file and, where the
+// JSON decoder tells it, the line and column, or else the entry.
+func (f *Files) takeIn(name string, data []byte) (int, error) {
+	path := filepath.Join(f.params.Dir, name)
+
+	var entries int
+
+	switch name {
+	case bansJSON:
+		var file bansFile
+
+		err := parse(path, data, &file)
+		if err != nil {
+			return 0, err
+		}
+
+		held := make([]bans.Ban, 0, len(file.Bans))
+		for _, entry := range file.Bans {
+			held = append(held, entry.ban())
+		}
+
+		f.params.Ledger.Load(held)
+		entries = len(held)
+	case clientsJSON:
+		var file clientsFile
+
+		err := parse(path, data, &file)
+		if err != nil {
+			return 0, err
+		}
+
+		f.params.Limiter.Load(file.Clients, f.params.Now())
+		entries = len(file.Clients)
+	case lookupsJSON:
+		var file lookupsFile
+
+		err := parse(path, data, &file)
+		if err != nil {
+			return 0, err
+		}
+
+		f.params.GeoJS.Load(file.Lookups)
+		entries = len(file.Lookups)
+	}
+
+	f.sums[name] = sha256.Sum256(data)
+
+	return entries, nil
+}
+
+// writeFile writes the state file name from what smallwebwaf holds. An
+// edit made since smallwebwaf last read or wrote the file is taken in
+// first, so that it is not overwritten, or set aside if it does not
+// parse. A file that cannot be read, or an edit that cannot be set
+// aside, is left as it is, and the write given up. Every write is counted
+// in the metrics, and one that fails or is given up as a failure.
+func (f *Files) writeFile(name string) error {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+
+	data, changed, err := f.readChanged(name)
+	if err == nil && changed {
+		err = f.takeInEdit(name, data)
+		if err != nil {
+			err = f.setAside(name, err)
+		}
+	}
+
+	if err == nil {
+		data, err = f.encode(name)
+		if err != nil {
+			err = fmt.Errorf("encode %s: %w", name, err)
+		}
+	}
+
+	if err == nil {
+		err = write(f.params.Dir, name, data)
+	}
+
+	if err == nil {
+		// The file holds data from here on, even if the directory sync
+		// fails, so that its next read does not take it for an admin's
+		// edit.
+		f.sums[name] = sha256.Sum256(data)
+		err = syncDirectory(f.params.Dir)
+	}
+
+	f.params.Metrics.StateFileWritten(name, len(data), err)
+
+	return err
+}
+
+// setAside renames the state file name, an edit that does not parse with
+// parseErr, to name.bad, for the admin to mend, and logs it with where in
+// the file the error is. If the rename fails, the edit is left as it is,
+// and the error returned is parseErr joined with the rename's.
+func (f *Files) setAside(name string, parseErr error) error {
+	path := filepath.Join(f.params.Dir, name)
+
+	err := os.Rename(path, path+".bad")
+	if err != nil {
+		return errors.Join(parseErr, err)
+	}
+
+	f.params.ProcessLog.Error("set aside an edit of a state file that does not parse",
+		"file", path+".bad", "error", parseErr.Error())
+	f.params.Metrics.StateFileEditSetAside(name)
+
+	return nil
+}
+
+// encode returns the state file name as smallwebwaf writes it, from a
+// snapshot of the part that keeps that state.
+func (f *Files) encode(name string) ([]byte, error) {
+	switch name {
+	case bansJSON:
+		held := f.params.Ledger.Snapshot()
+
+		file := bansFile{Version: version, Bans: make([]banEntry, 0, len(held))}
+		for _, ban := range held {
+			file.Bans = append(file.Bans, newBanEntry(ban))
+		}
+
+		data, err := json.MarshalIndent(file, "", "    ")
+		if err != nil {
+			return nil, err
+		}
+
+		return append(data, '\n'), nil
+	case clientsJSON:
+		return encodeOnePerLine("clients", f.params.Limiter.Snapshot())
+	default: // lookups.json
+		return encodeOnePerLine("lookups", f.params.GeoJS.Snapshot())
+	}
 }
 
 // newBanEntry returns ban as bans.json holds it.
@@ -233,6 +462,100 @@ func (e banEntry) ban() bans.Ban {
 	return ban
 }
 
+// check refuses a ban without a netblock, which would refuse every IPv6
+// client, a start, from which the length of the netblock's next ban is
+// worked out, or an expires, which would make it permanent. A permanent
+// ban's expires is null, which Bans cannot tell from a missing one, so
+// each expires is read again as written.
+func (f *bansFile) check(data []byte) error {
+	var written struct {
+		Bans []struct {
+			Expires json.RawMessage `json:"expires"`
+		} `json:"bans"`
+	}
+
+	err := json.Unmarshal(data, &written)
+	if err != nil {
+		return err
+	}
+
+	for i, entry := range f.Bans {
+		switch {
+		case !entry.Netblock.IsValid():
+			return missing(i, "netblock")
+		case entry.Start.IsZero():
+			return missing(i, "start")
+		case written.Bans[i].Expires == nil:
+			return missing(i, "expires")
+		}
+	}
+
+	return nil
+}
+
+// check refuses a client without its address, which would count nobody's
+// requests, or with requests in a window but no start, which would drop
+// them and give the client a fresh allowance.
+func (f *clientsFile) check([]byte) error {
+	for i, client := range f.Clients {
+		switch {
+		case !client.Client.IsValid():
+			return missing(i, "client")
+		case countsWithoutStart(client.Minute):
+			return missing(i, "minute.start")
+		case countsWithoutStart(client.Hour):
+			return missing(i, "hour.start")
+		case countsWithoutStart(client.Day):
+			return missing(i, "day.start")
+		}
+	}
+
+	return nil
+}
+
+// check refuses an answer without a client, which would answer for
+// nobody, a country, which would place the client nowhere, or the time
+// GeoJS gave it, which would drop it. "" is the country of a client
+// GeoJS cannot place, which Lookups cannot tell from a missing one, so
+// each country is read again as written.
+func (f *lookupsFile) check(data []byte) error {
+	var written struct {
+		Lookups []struct {
+			Country *string `json:"country"`
+		} `json:"lookups"`
+	}
+
+	err := json.Unmarshal(data, &written)
+	if err != nil {
+		return err
+	}
+
+	for i, answer := range f.Lookups {
+		switch {
+		case !answer.Client.IsValid():
+			return missing(i, "client")
+		case written.Lookups[i].Country == nil:
+			return missing(i, "country")
+		case answer.Answered.IsZero():
+			return missing(i, "answered")
+		}
+	}
+
+	return nil
+}
+
+// countsWithoutStart reports whether b holds requests but no start, which
+// places them in time.
+func countsWithoutStart(b ratelimit.Buckets) bool {
+	return b.Start.IsZero() && (b.Current != 0 || b.Previous != 0)
+}
+
+// missing returns the error for entry i, counted from 0, of a state file,
+// which has no field.
+func missing(i int, field string) error {
+	return fmt.Errorf("entry %d %w %q", i+1, errMissing, field)
+}
+
 // encodeOnePerLine encodes a state file whose entries, under key, are one
 // to a line, so that grep shows everything about one client.
 func encodeOnePerLine[E any](key string, entries []E) ([]byte, error) {
@@ -269,27 +592,16 @@ func checkWritable(dir string) error {
 	return errors.Join(file.Close(), os.Remove(file.Name()))
 }
 
-// read reads the state file name in dir into file, a pointer to that
-// file's struct. A missing file leaves file as it is.
-func read(dir, name string, file any) error {
-	path := filepath.Join(dir, name)
-
-	data, err := os.ReadFile(path) //nolint:gosec // a state file, in SWWAF_STATE_DIR
-	if errors.Is(err, fs.ErrNotExist) {
-		return nil
-	}
-
-	if err != nil {
-		return err
-	}
-
+// parse reads data, what the state file at path holds, into file, a
+// pointer to that file's struct, and checks its entries.
+func parse(path string, data []byte, file stateFile) error {
 	// The version is read first, so that a file of another version is
 	// refused for that, and not for an entry this version cannot read.
 	var header struct {
 		Version int `json:"version"`
 	}
 
-	err = json.Unmarshal(data, &header)
+	err := json.Unmarshal(data, &header)
 	if err == nil && header.Version != version {
 		err = fmt.Errorf("%w %d, where this smallwebwaf reads version %d",
 			errVersion, header.Version, version)
@@ -303,6 +615,10 @@ func read(dir, name string, file any) error {
 		err = decoder.Decode(file)
 	}
 
+	if err == nil {
+		err = file.check(data)
+	}
+
 	if err != nil {
 		return fmt.Errorf("%s%s: %w", path, position(data, err), err)
 	}
@@ -338,23 +654,25 @@ func position(data []byte, err error) string {
 // write writes data to the file name in dir so that a crash at any
 // moment leaves either the old file or the new one, whole: data goes to a
 // temporary file in the same directory, which is synced and renamed over
-// name, and then the directory is synced, so that the rename lasts.
+// name. syncDirectory must follow, so that the rename lasts.
 func write(dir, name string, data []byte) error {
 	path := filepath.Join(dir, name)
 	temporary := path + ".tmp"
 
 	err := writeSynced(temporary, data)
+	if err == nil {
+		err = os.Rename(temporary, path)
+	}
+
 	if err != nil {
 		_ = os.Remove(temporary)
-
-		return err
 	}
 
-	err = os.Rename(temporary, path)
-	if err != nil {
-		return err
-	}
+	return err
+}
 
+// syncDirectory syncs dir to the disk, so that a rename in it lasts.
+func syncDirectory(dir string) error {
 	directory, err := os.Open(dir) //nolint:gosec // SWWAF_STATE_DIR itself
 	if err != nil {
 		return err

internal/state/state_test.go

@@ -3,30 +3,41 @@ package state_test
 import (
 	"context"
 	"encoding/json"
+	"io/fs"
 	"log/slog"
+	"maps"
+	"net"
+	"net/http"
+	"net/http/httptest"
 	"net/netip"
 	"os"
 	"path/filepath"
 	"slices"
+	"strconv"
 	"strings"
 	"testing"
+	"testing/synctest"
 	"time"
 
 	"sneak.berlin/go/smallwebwaf/internal/bans"
 	"sneak.berlin/go/smallwebwaf/internal/lookup"
+	"sneak.berlin/go/smallwebwaf/internal/metrics"
 	"sneak.berlin/go/smallwebwaf/internal/ratelimit"
 	"sneak.berlin/go/smallwebwaf/internal/state"
 )
 
 const (
-	// waitLimit bounds how long a test waits for what should happen.
-	waitLimit = 10 * time.Second
-	// pollInterval is how often a test looks again.
-	pollInterval = 10 * time.Millisecond
 	// The state files.
 	bansJSON    = "bans.json"
 	clientsJSON = "clients.json"
 	lookupsJSON = "lookups.json"
+	// What the process log says once Watch watches the directory, and as
+	// it takes in an edit.
+	watching = "watching the state files for edits"
+	tookIn   = "took in an edit of a state file"
+	// maxLogLines is how many lines of the process log wait for a test to
+	// read them.
+	maxLogLines = 64
 )
 
 // permanentBansJSON is bans.json holding permanentBan.
@@ -167,18 +178,87 @@ func TestFileThatDoesNotParseStopsTheStart(t *testing.T) {
 		t.Run(tc.name, func(t *testing.T) {
 			t.Parallel()
 
-			dir := t.TempDir()
-			path := filepath.Join(dir, tc.file)
+			wantRefused(t, tc.file, tc.content, tc.want)
+		})
+	}
+}
 
-			err := os.WriteFile(path, []byte(tc.content), 0o600)
-			if err != nil {
-				t.Fatalf("write %s: %v", tc.file, err)
-			}
+func TestEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
+	t.Parallel()
 
-			_, err = state.Load(newParams(dir))
-			if err == nil || !strings.HasPrefix(err.Error(), path+tc.want) {
-				t.Errorf("error %v, want one starting %s%s", err, path, tc.want)
-			}
+	const (
+		// The other fields each entry needs.
+		ban    = `"start": "2026-10-06T00:00:00Z", "expires": null`
+		answer = `"answered": "2026-10-06T00:00:00Z"`
+
+		noNetblock = `: entry 1 has no "netblock"`
+	)
+
+	for _, tc := range []struct {
+		name, file, content string
+		// want is what the error says after the file's path.
+		want string
+	}{
+		{
+			"a ban without a netblock", bansJSON,
+			`{"version": 1, "bans": [{` + ban + `}]}`,
+			noNetblock,
+		},
+		{
+			"a ban whose netblock is null", bansJSON,
+			`{"version": 1, "bans": [{"netblock": null, ` + ban + `}]}`,
+			noNetblock,
+		},
+		{
+			"a ban whose netblock is empty", bansJSON,
+			`{"version": 1, "bans": [{"netblock": "", ` + ban + `}]}`,
+			noNetblock,
+		},
+		{
+			"a ban without a start", bansJSON,
+			`{"version": 1, "bans": [{"netblock": "203.0.113.9/32", "expires": null}]}`,
+			`: entry 1 has no "start"`,
+		},
+		{
+			// The first ban's expires is null, as a permanent ban's is.
+			"a ban without an expires", bansJSON,
+			`{"version": 1, "bans": [{"netblock": "203.0.113.9/32", ` + ban + `}, ` +
+				`{"netblock": "203.0.113.10/32", "start": "2026-10-06T00:00:00Z"}]}`,
+			`: entry 2 has no "expires"`,
+		},
+		{
+			"a client without its address", clientsJSON,
+			`{"version": 1, "clients": [{"history": {"requests": 3}}]}`,
+			`: entry 1 has no "client"`,
+		},
+		{
+			"a client with requests in a window without its start", clientsJSON,
+			`{"version": 1, "clients": [{"client": "203.0.113.9/32", ` +
+				`"hour": {"current": 3}}]}`,
+			`: entry 1 has no "hour.start"`,
+		},
+		{
+			"an answer without a client", lookupsJSON,
+			`{"version": 1, "lookups": [{"country": "DE", ` + answer + `}]}`,
+			`: entry 1 has no "client"`,
+		},
+		{
+			// A country of "" is a client GeoJS cannot place.
+			"an answer without a country", lookupsJSON,
+			`{"version": 1, "lookups": [{"client": "192.0.2.1/32", "country": "", ` +
+				answer + `}, {"client": "203.0.113.9/32", ` + answer + `}]}`,
+			`: entry 2 has no "country"`,
+		},
+		{
+			"an answer without the time GeoJS gave it", lookupsJSON,
+			`{"version": 1, "lookups": [{"client": "203.0.113.9/32", "country": "DE"}]}`,
+			`: entry 1 has no "answered"`,
+		},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+
+			wantRefused(t, tc.file, tc.content, tc.want)
 		})
 	}
 }
@@ -191,18 +271,7 @@ func TestUnknownVersionStopsTheStart(t *testing.T) {
 			t.Run(file+" "+content, func(t *testing.T) {
 				t.Parallel()
 
-				dir := t.TempDir()
-				path := filepath.Join(dir, file)
-
-				err := os.WriteFile(path, []byte(content), 0o600)
-				if err != nil {
-					t.Fatalf("write %s: %v", file, err)
-				}
-
-				_, err = state.Load(newParams(dir))
-				if err == nil || !strings.HasPrefix(err.Error(), path+": unknown version ") {
-					t.Errorf("error %v, want one naming %s and its version", err, path)
-				}
+				wantRefused(t, file, content, ": unknown version ")
 			})
 		}
 	}
@@ -231,41 +300,116 @@ func TestUnwritableDirectoryStopsTheStart(t *testing.T) {
 	}
 }
 
-func TestBanWrittenAfterTheWriteDelay(t *testing.T) {
+// The three tests below run Run in a synctest bubble, where time is a clock
+// of the test's own: time.Sleep moves it on at once, and synctest.Wait
+// returns once Run waits for its next write, so that every write due by
+// then is on disk.
+
+func TestBansWrittenOnceWriteDelayAfterABan(t *testing.T) {
 	t.Parallel()
 
-	dir := t.TempDir()
-	params := newParams(dir)
-	params.WriteDelay = 10 * time.Millisecond
-	run(t, load(t, params))
+	synctest.Test(t, func(t *testing.T) {
+		dir := t.TempDir()
+		params := newParams(dir)
+		params.WriteDelay = 10 * time.Second
+		run(t, load(t, params).Run)
 
-	ban := params.Ledger.BanForLimit(netip.MustParsePrefix("203.0.113.9/32"),
-		midnight(), bans.Notes{Limit: 1})
+		// A second ban, made while the first waits to be written, puts the
+		// write off no further, and is written with it.
+		first := params.Ledger.BanForLimit(netip.MustParsePrefix("203.0.113.9/32"),
+			midnight(), bans.Notes{})
 
-	waitForFile(t, filepath.Join(dir, bansJSON))
+		time.Sleep(5 * time.Second)
 
-	read := newParams(dir)
-	load(t, read)
+		second := params.Ledger.BanForLimit(netip.MustParsePrefix("203.0.113.10/32"),
+			midnight(), bans.Notes{})
 
-	if got := read.Ledger.Snapshot(); !slices.Equal(got, []bans.Ban{ban}) {
-		t.Errorf("bans.json holds %+v, want %+v", got, ban)
-	}
+		time.Sleep(5*time.Second - time.Nanosecond)
+		synctest.Wait()
+		wantFiles(t, dir)
 
-	// The other files wait for the interval, an hour away.
-	wantFiles(t, dir, bansJSON)
+		time.Sleep(time.Nanosecond)
+		synctest.Wait()
+		wantFiles(t, dir, bansJSON)
+
+		read := newParams(dir)
+		load(t, read)
+
+		want := []bans.Ban{first, second}
+		if got := read.Ledger.Snapshot(); !slices.Equal(got, want) {
+			t.Errorf("bans.json holds %+v, want %+v", got, want)
+		}
+
+		// That write was the only one: bans.json is not written again for
+		// the second ban. The other files wait for the interval, an hour
+		// away.
+		removeFiles(t, dir, bansJSON)
+		time.Sleep(params.WriteDelay)
+		synctest.Wait()
+		wantFiles(t, dir)
+	})
 }
 
 func TestEveryFileWrittenEveryCounterInterval(t *testing.T) {
 	t.Parallel()
 
-	dir := t.TempDir()
-	params := newParams(dir)
-	params.CounterInterval = 10 * time.Millisecond
-	run(t, load(t, params))
+	synctest.Test(t, func(t *testing.T) {
+		dir := t.TempDir()
+		params := newParams(dir)
+		params.CounterInterval = time.Minute
+		run(t, load(t, params).Run)
 
-	for _, file := range []string{bansJSON, clientsJSON, lookupsJSON} {
-		waitForFile(t, filepath.Join(dir, file))
-	}
+		// The files are removed once written, so that each interval shows
+		// them written again.
+		for range 3 {
+			time.Sleep(time.Minute - time.Nanosecond)
+			synctest.Wait()
+			wantFiles(t, dir)
+
+			time.Sleep(time.Nanosecond)
+			synctest.Wait()
+			wantFiles(t, dir, bansJSON, clientsJSON, lookupsJSON)
+			removeFiles(t, dir, bansJSON, clientsJSON, lookupsJSON)
+		}
+	})
+}
+
+func TestEditJustBeforeAScheduledWriteSurvivesIt(t *testing.T) {
+	t.Parallel()
+
+	synctest.Test(t, func(t *testing.T) {
+		dir := t.TempDir()
+		params := newParams(dir)
+		params.WriteDelay = 10 * time.Second
+		run(t, load(t, params).Run)
+
+		// A ban, and bans.json written with it.
+		params.Ledger.BanForLimit(netip.MustParsePrefix("203.0.113.9/32"),
+			midnight(), bans.Notes{})
+		time.Sleep(params.WriteDelay)
+		synctest.Wait()
+
+		// A second ban is to be written WriteDelay later. Just before
+		// then, an admin saves bans.json with the first ban lifted and
+		// another added.
+		params.Ledger.BanForLimit(netip.MustParsePrefix("203.0.113.10/32"),
+			midnight(), bans.Notes{})
+		time.Sleep(params.WriteDelay - time.Nanosecond)
+		synctest.Wait()
+		edit(t, dir, bansJSON, permanentBansJSON)
+
+		// The write takes the edit in first, and writes it back. The second
+		// ban, made after the admin opened the file, is lost, as "Edits
+		// while running" in SPEC.md says.
+		time.Sleep(time.Nanosecond)
+		synctest.Wait()
+
+		if got := readFile(t, filepath.Join(dir, bansJSON)); got != permanentBansJSON {
+			t.Errorf("bans.json holds\n%s\nwant the edit", got)
+		}
+
+		wantEqual(t, bansJSON, params.Ledger.Snapshot(), []bans.Ban{permanentBan()})
+	})
 }
 
 func TestFailedWriteLeavesTheFileAsItWas(t *testing.T) {
@@ -310,6 +454,414 @@ func TestFailedWriteLeavesTheFileAsItWas(t *testing.T) {
 	}
 }
 
+func TestWritesAreCountedInTheMetrics(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+	params := newParams(dir)
+	params.Ledger.Load([]bans.Ban{permanentBan()})
+	files := load(t, params)
+
+	err := files.WriteAll()
+	if err != nil {
+		t.Fatalf("write: %v", err)
+	}
+
+	const (
+		ofBans    = `{file="bans.json"}`
+		ofClients = `{file="clients.json"}`
+	)
+
+	got := scrape(t, params)
+	wantMetric(t, got, "smallwebwaf_state_file_writes_total"+ofBans, 1)
+	wantMetric(t, got, "smallwebwaf_state_file_writes_total"+ofClients, 1)
+	wantMetric(t, got, "smallwebwaf_state_file_write_failures_total"+ofBans, 0)
+	wantMetric(t, got, "smallwebwaf_state_file_size_bytes"+ofBans,
+		float64(len(permanentBansJSON)))
+
+	written := metric(t, got, "smallwebwaf_state_file_last_write_timestamp_seconds"+ofBans)
+	if written < float64(time.Now().Add(-time.Hour).Unix()) {
+		t.Errorf("bans.json was last written at %v, not by that write", written)
+	}
+
+	// A directory in the way of bans.json's temporary file fails its next
+	// write, which leaves its size as it was, although it has a ban more.
+	err = os.Mkdir(filepath.Join(dir, bansJSON+".tmp"), 0o700)
+	if err != nil {
+		t.Fatalf("mkdir: %v", err)
+	}
+
+	params.Ledger.BanForLimit(netip.MustParsePrefix("203.0.113.9/32"), midnight(),
+		bans.Notes{})
+
+	err = files.WriteAll()
+	if err == nil {
+		t.Fatal("the write did not fail")
+	}
+
+	got = scrape(t, params)
+	wantMetric(t, got, "smallwebwaf_state_file_writes_total"+ofBans, 2)
+	wantMetric(t, got, "smallwebwaf_state_file_write_failures_total"+ofBans, 1)
+	wantMetric(t, got, "smallwebwaf_state_file_write_failures_total"+ofClients, 0)
+	wantMetric(t, got, "smallwebwaf_state_file_size_bytes"+ofBans,
+		float64(len(permanentBansJSON)))
+}
+
+func TestFileThatCannotBeReadIsNotWrittenOver(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+	path := filepath.Join(dir, bansJSON)
+	params := newParams(dir)
+	files := load(t, params)
+
+	// bans.json is a socket, which cannot be opened as a file, even by
+	// root, as the tests run in Docker, but which a rename could replace.
+	// Whether it holds an edit cannot be told, so it is left as it is.
+	socket, err := (&net.ListenConfig{}).Listen(t.Context(), "unix", path)
+	if err != nil {
+		t.Fatalf("listen: %v", err)
+	}
+
+	defer func() {
+		_ = socket.Close()
+	}()
+
+	err = files.WriteAll()
+	if err == nil {
+		t.Error("writing with bans.json unreadable did not fail")
+	}
+
+	info, err := os.Lstat(path)
+	if err != nil || info.Mode().Type() != fs.ModeSocket {
+		t.Errorf("bans.json is now %v (%v), want the socket", info, err)
+	}
+
+	wantFiles(t, dir, bansJSON, clientsJSON, lookupsJSON)
+	wantWriteFailed(t, params, bansJSON)
+}
+
+func TestBrokenEditThatCannotBeSetAsideIsNotWrittenOver(t *testing.T) {
+	t.Parallel()
+
+	const broken = `{"version": 1, "bans": [`
+
+	dir := t.TempDir()
+	path := filepath.Join(dir, bansJSON)
+	params := newParams(dir)
+	files := load(t, params)
+
+	// A directory named bans.json.bad cannot be renamed over, so the
+	// broken edit cannot be set aside, and is left as it is.
+	edit(t, dir, bansJSON, broken)
+
+	err := os.Mkdir(path+".bad", 0o700)
+	if err != nil {
+		t.Fatalf("mkdir: %v", err)
+	}
+
+	err = files.WriteAll()
+	if err == nil {
+		t.Error("writing with bans.json.bad in the way did not fail")
+	}
+
+	if got := readFile(t, path); got != broken {
+		t.Errorf("bans.json holds\n%s\nwant the edit", got)
+	}
+
+	wantWriteFailed(t, params, bansJSON)
+}
+
+func TestEditOfEachFileTakenIn(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+	params := newParams(dir)
+	lines := logInto(&params)
+	fill(params)
+	files := load(t, params)
+
+	err := files.WriteAll()
+	if err != nil {
+		t.Fatalf("write: %v", err)
+	}
+
+	watch(t, files, lines)
+
+	// Each edit holds one entry, for a client the parts did not hold, and
+	// takes the place of everything the part held.
+	client := netip.MustParsePrefix("198.51.100.7/32")
+
+	edit(t, dir, bansJSON, `{"version": 1, "bans": [{"netblock": "198.51.100.7/32", `+
+		`"start": "2026-10-06T00:00:00Z", "expires": null}]}`)
+	wantTakenIn(t, lines, dir, bansJSON)
+	wantEqual(t, bansJSON, params.Ledger.Snapshot(),
+		[]bans.Ban{{Netblock: client, Start: midnight()}})
+
+	edit(t, dir, clientsJSON, `{"version": 1, "clients": [`+
+		`{"client": "198.51.100.7/32", "history": {"requests": 7}}]}`)
+	wantTakenIn(t, lines, dir, clientsJSON)
+	wantEqual(t, clientsJSON, params.Limiter.Snapshot(),
+		[]ratelimit.Client{{Client: client, History: ratelimit.History{Requests: 7}}})
+
+	edit(t, dir, lookupsJSON, `{"version": 1, "lookups": [{"client": "198.51.100.7/32", `+
+		`"country": "FR", "answered": "2026-10-06T00:00:00Z"}]}`)
+	wantTakenIn(t, lines, dir, lookupsJSON)
+	wantEqual(t, lookupsJSON, params.GeoJS.Snapshot(),
+		[]lookup.Answer{{Client: client, Country: "FR", Answered: midnight()}})
+}
+
+func TestOwnWritesAreNotTakenIn(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+	params := newParams(dir)
+	lines := logInto(&params)
+	fill(params)
+	files := load(t, params)
+	watch(t, files, lines)
+
+	// Every file is written while watched, and then lookups.json edited:
+	// the first edit taken in is that one.
+	err := files.WriteAll()
+	if err != nil {
+		t.Fatalf("write: %v", err)
+	}
+
+	edit(t, dir, lookupsJSON, `{"version": 1, "lookups": []}`)
+	wantTakenIn(t, lines, dir, lookupsJSON)
+}
+
+func TestFileRenamedOverAStateFileTakenIn(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+	path := filepath.Join(dir, bansJSON)
+	params := newParams(dir)
+	lines := logInto(&params)
+	files := load(t, params)
+
+	err := files.WriteAll()
+	if err != nil {
+		t.Fatalf("write: %v", err)
+	}
+
+	watch(t, files, lines)
+
+	// The admin mends bans.json.bad and moves it back, as editors that
+	// save by renaming do with a file of their own: nothing is written
+	// into bans.json itself. An edit of clients.json after it must be
+	// taken in second.
+	edit(t, dir, bansJSON+".bad", permanentBansJSON)
+
+	err = os.Rename(path+".bad", path)
+	if err != nil {
+		t.Fatalf("rename: %v", err)
+	}
+
+	edit(t, dir, clientsJSON, `{"version": 1, "clients": []}`)
+
+	wantTakenIn(t, lines, dir, bansJSON)
+	wantEqual(t, bansJSON, params.Ledger.Snapshot(), []bans.Ban{permanentBan()})
+}
+
+func TestBanAddedAndLiftedThroughBansJSON(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+	params := newParams(dir)
+	lines := logInto(&params)
+	watch(t, load(t, params), lines)
+
+	client := netip.MustParseAddr("203.0.113.9")
+
+	// An entry added, as an admin writes it, bans its netblock.
+	edit(t, dir, bansJSON, `{"version": 1, "bans": [{"netblock": "203.0.113.0/24", `+
+		`"start": "2026-10-06T00:00:00Z", "expires": null}]}`)
+	wantTakenIn(t, lines, dir, bansJSON)
+
+	_, banned := params.Ledger.Check(client, midnight())
+	if !banned {
+		t.Error("the ban added to bans.json does not refuse")
+	}
+
+	// The entry removed lifts the ban.
+	edit(t, dir, bansJSON, `{"version": 1, "bans": []}`)
+	wantTakenIn(t, lines, dir, bansJSON)
+
+	_, banned = params.Ledger.Check(client, midnight())
+	if banned {
+		t.Error("the ban removed from bans.json still refuses")
+	}
+}
+
+func TestBrokenEditSetAsideAtTheNextWrite(t *testing.T) {
+	t.Parallel()
+
+	// It ends a ban's entry with a comma.
+	const broken = "{\n    \"version\": 1,\n    \"bans\": [\n" +
+		"        {\"netblock\": \"203.0.113.9/32\",}\n    ]\n}\n"
+
+	dir := t.TempDir()
+	path := filepath.Join(dir, bansJSON)
+	params := newParams(dir)
+	lines := logInto(&params)
+	params.Ledger.Load([]bans.Ban{permanentBan()})
+	files := load(t, params)
+
+	err := files.WriteAll()
+	if err != nil {
+		t.Fatalf("write: %v", err)
+	}
+
+	watch(t, files, lines)
+
+	// While smallwebwaf runs, the broken edit is left as it is: an edit
+	// of clients.json, made after it and taken in, shows that it has been
+	// seen.
+	edit(t, dir, bansJSON, broken)
+	edit(t, dir, clientsJSON, `{"version": 1, "clients": []}`)
+	wantTakenIn(t, lines, dir, clientsJSON)
+	wantFiles(t, dir, bansJSON, clientsJSON, lookupsJSON)
+
+	// The next write sets it aside, logged with where the error is, and
+	// writes bans.json again from what smallwebwaf still holds.
+	err = files.WriteAll()
+	if err != nil {
+		t.Fatalf("write: %v", err)
+	}
+
+	line := lines.waitFor(t, "set aside an edit of a state file that does not parse")
+	message, _ := line["error"].(string)
+
+	if line["file"] != path+".bad" ||
+		!strings.HasPrefix(message, path+", line 4, column 39: ") {
+		t.Errorf("set aside with %v", line)
+	}
+
+	wantFiles(t, dir, bansJSON, bansJSON+".bad", clientsJSON, lookupsJSON)
+
+	if got := readFile(t, path+".bad"); got != broken {
+		t.Errorf("bans.json.bad holds\n%s\nwant the edit", got)
+	}
+
+	if got := readFile(t, path); got != permanentBansJSON {
+		t.Errorf("bans.json holds\n%s\nwant\n%s", got, permanentBansJSON)
+	}
+}
+
+func TestEditsTakenInAreCountedInTheMetrics(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+	params := newParams(dir)
+	lines := logInto(&params)
+	files := load(t, params)
+
+	// One edit is taken in by the write of its file, before Watch runs,
+	// and one by Watch.
+	edit(t, dir, bansJSON, `{"version": 1, "bans": []}`)
+
+	err := files.WriteAll()
+	if err != nil {
+		t.Fatalf("write: %v", err)
+	}
+
+	watch(t, files, lines)
+	edit(t, dir, bansJSON, permanentBansJSON)
+	wantTakenIn(t, lines, dir, bansJSON)
+
+	wantMetric(t, scrape(t, params),
+		`smallwebwaf_state_file_edits_taken_in_total{file="bans.json"}`, 2)
+}
+
+func TestEditTakenInByAWriteIsLoggedAsWatchLogsIt(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+	params := newParams(dir)
+	lines := logInto(&params)
+	files := load(t, params)
+
+	// An edit taken in by Watch, which is then stopped.
+	ctx, stop := context.WithCancel(t.Context())
+	stopped := make(chan struct{})
+
+	go func() {
+		files.Watch(ctx)
+		close(stopped)
+	}()
+
+	lines.waitFor(t, watching)
+	edit(t, dir, bansJSON, `{"version": 1, "bans": []}`)
+	byWatch := lines.waitFor(t, tookIn)
+
+	stop()
+	<-stopped
+
+	// An edit taken in by the write of its file. Nothing logs after the
+	// write, so the log is closed, and a write that does not log the edit
+	// fails the test at once instead of waiting for the line.
+	edit(t, dir, bansJSON, permanentBansJSON)
+
+	err := files.WriteAll()
+	if err != nil {
+		t.Fatalf("write: %v", err)
+	}
+
+	close(lines)
+
+	byWrite := lines.waitFor(t, tookIn)
+
+	// The two lines differ only in their time.
+	delete(byWatch, "time")
+	delete(byWrite, "time")
+
+	if !maps.Equal(byWrite, byWatch) {
+		t.Errorf("the write logged %v, where Watch logged %v", byWrite, byWatch)
+	}
+}
+
+func TestEditsSetAsideAreCountedInTheMetrics(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+	params := newParams(dir)
+	files := load(t, params)
+
+	edit(t, dir, bansJSON, `{"version": 1, "bans": [`)
+
+	err := files.WriteAll()
+	if err != nil {
+		t.Fatalf("write: %v", err)
+	}
+
+	wantMetric(t, scrape(t, params),
+		`smallwebwaf_state_file_edits_set_aside_total{file="bans.json"}`, 1)
+}
+
+func TestDirectoryThatCannotBeWatchedIsLogged(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+	params := newParams(dir)
+	lines := logInto(&params)
+	files := load(t, params)
+
+	err := os.Remove(dir)
+	if err != nil {
+		t.Fatalf("remove: %v", err)
+	}
+
+	// Watch returns at once.
+	files.Watch(t.Context())
+
+	line := lines.waitFor(t, "cannot watch the state files for edits")
+	if line["level"] != "ERROR" {
+		t.Errorf("logged as %v", line)
+	}
+}
+
 // midnight is the time of the tests' clock.
 func midnight() time.Time {
 	return time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)
@@ -319,6 +871,7 @@ func midnight() time.Time {
 // hold nothing yet. GeoJS is never asked.
 func newParams(dir string) state.Params {
 	discard := slog.New(slog.DiscardHandler)
+	m := metrics.New(1)
 
 	return state.Params{
 		Dir:             dir,
@@ -330,10 +883,13 @@ func newParams(dir string) state.Params {
 			MaxBanDuration:       7 * 24 * time.Hour,
 			MaxBans:              5000,
 		}),
-		Limiter:    ratelimit.New(ratelimit.Limits{}),
-		GeoJS:      lookup.New(lookup.Params{Now: midnight, ProcessLog: discard}),
+		Limiter: ratelimit.New(ratelimit.Limits{}),
+		GeoJS: lookup.New(lookup.Params{
+			Now: midnight, ProcessLog: discard, Metrics: m,
+		}),
 		Now:        midnight,
 		ProcessLog: discard,
+		Metrics:    m,
 	}
 }
 
@@ -400,15 +956,16 @@ func load(t *testing.T, params state.Params) *state.Files {
 	return files
 }
 
-// run runs files' writes until the test ends.
-func run(t *testing.T, files *state.Files) {
+// run runs task, the Run or the Watch of state files, until the test
+// ends.
+func run(t *testing.T, task func(context.Context)) {
 	t.Helper()
 
 	ctx, stop := context.WithCancel(t.Context())
 	stopped := make(chan struct{})
 
 	go func() {
-		files.Run(ctx)
+		task(ctx)
 		close(stopped)
 	}()
 
@@ -418,6 +975,80 @@ func run(t *testing.T, files *state.Files) {
 	})
 }
 
+// watch runs files' Watch until the test ends, and waits until it
+// watches the directory.
+func watch(t *testing.T, files *state.Files, lines processLog) {
+	t.Helper()
+
+	run(t, files.Watch)
+	lines.waitFor(t, watching)
+}
+
+// processLog receives the lines of a process log, each a JSON object, for
+// a test to wait for.
+type processLog chan string
+
+// logInto has params' process log write its lines into a new processLog,
+// and returns that.
+func logInto(params *state.Params) processLog {
+	lines := make(processLog, maxLogLines)
+	params.ProcessLog = slog.New(slog.NewJSONHandler(lines, nil))
+
+	return lines
+}
+
+// Write receives a line of the process log.
+func (l processLog) Write(line []byte) (int, error) {
+	l <- string(line)
+
+	return len(line), nil
+}
+
+// waitFor returns the next line of the process log whose message is msg,
+// passing over the lines before it, or nil if the log is closed first. It
+// waits as long as that takes, so that a slow test process cannot fail
+// the test.
+func (l processLog) waitFor(t *testing.T, msg string) map[string]any {
+	t.Helper()
+
+	for line := range l {
+		var fields map[string]any
+
+		err := json.Unmarshal([]byte(line), &fields)
+		if err != nil {
+			t.Fatalf("process log line %q is not JSON: %v", line, err)
+		}
+
+		if fields["msg"] == msg {
+			return fields
+		}
+	}
+
+	return nil
+}
+
+// wantTakenIn waits for the next edit taken in, and checks that it is of
+// the state file name in dir.
+func wantTakenIn(t *testing.T, lines processLog, dir, name string) {
+	t.Helper()
+
+	line := lines.waitFor(t, tookIn)
+	if line["file"] != filepath.Join(dir, name) {
+		t.Fatalf("took in %v, want an edit of %s", line, name)
+	}
+}
+
+// edit writes content to the state file name in dir, as an admin saves an
+// edit of it.
+func edit(t *testing.T, dir, name, content string) {
+	t.Helper()
+
+	err := os.WriteFile(filepath.Join(dir, name), []byte(content), 0o600)
+	if err != nil {
+		t.Fatalf("write %s: %v", name, err)
+	}
+}
+
 // wantEqual checks that the entries read back from file are those
 // written.
 func wantEqual[E comparable](t *testing.T, file string, got, want []E) {
@@ -440,21 +1071,36 @@ func readFile(t *testing.T, path string) string {
 	return string(data)
 }
 
-// waitForFile waits for the file at path to exist.
-func waitForFile(t *testing.T, path string) {
+// wantRefused writes content to the state file named file in a new
+// directory, and checks that Load refuses it with an error that is the
+// file's path and then starts with want.
+func wantRefused(t *testing.T, file, content, want string) {
 	t.Helper()
 
-	deadline := time.Now().Add(waitLimit)
-	for time.Now().Before(deadline) {
-		_, err := os.Stat(path)
-		if err == nil {
-			return
-		}
+	dir := t.TempDir()
+	path := filepath.Join(dir, file)
 
-		time.Sleep(pollInterval)
+	err := os.WriteFile(path, []byte(content), 0o600)
+	if err != nil {
+		t.Fatalf("write %s: %v", file, err)
 	}
 
-	t.Fatalf("no %s after %s", path, waitLimit)
+	_, err = state.Load(newParams(dir))
+	if err == nil || !strings.HasPrefix(err.Error(), path+want) {
+		t.Errorf("error %v, want one starting %s%s", err, path, want)
+	}
+}
+
+// removeFiles removes the named files from dir.
+func removeFiles(t *testing.T, dir string, names ...string) {
+	t.Helper()
+
+	for _, name := range names {
+		err := os.Remove(filepath.Join(dir, name))
+		if err != nil {
+			t.Fatalf("remove: %v", err)
+		}
+	}
 }
 
 // wantFiles checks the names of the files in dir.
@@ -506,3 +1152,66 @@ func wantEntries(t *testing.T, path, key string, want ...string) {
 		}
 	}
 }
+
+// scrape returns the metrics of params, in the Prometheus text format.
+func scrape(t *testing.T, params state.Params) string {
+	t.Helper()
+
+	recorder := httptest.NewRecorder()
+	params.Metrics.ServeHTTP(recorder,
+		httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", http.NoBody))
+
+	if recorder.Code != http.StatusOK {
+		t.Fatalf("the metrics were answered %d", recorder.Code)
+	}
+
+	return recorder.Body.String()
+}
+
+// metric returns the value of series in text, the metrics, such as
+// smallwebwaf_state_file_writes_total{file="bans.json"}, or fails the test
+// if there is no such series.
+func metric(t *testing.T, text, series string) float64 {
+	t.Helper()
+
+	for line := range strings.Lines(text) {
+		value, found := strings.CutPrefix(strings.TrimSuffix(line, "\n"), series+" ")
+		if !found {
+			continue
+		}
+
+		number, err := strconv.ParseFloat(value, 64)
+		if err != nil {
+			t.Fatalf("%s has the value %q", series, value)
+		}
+
+		return number
+	}
+
+	t.Fatalf("no series %s in the metrics:\n%s", series, text)
+
+	return 0
+}
+
+// wantWriteFailed checks that the metrics of params count one write of the
+// state file name, and that it failed.
+func wantWriteFailed(t *testing.T, params state.Params, name string) {
+	t.Helper()
+
+	got := scrape(t, params)
+	file := `{file="` + name + `"}`
+
+	wantMetric(t, got, "smallwebwaf_state_file_writes_total"+file, 1)
+	wantMetric(t, got, "smallwebwaf_state_file_write_failures_total"+file, 1)
+}
+
+// wantMetric checks the value of series in text, the metrics, as metric
+// reads it.
+func wantMetric(t *testing.T, text, series string, want float64) {
+	t.Helper()
+
+	got := metric(t, text, series)
+	if got != want {
+		t.Errorf("%s is %v, want %v", series, got, want)
+	}
+}

internal/state/write_internal_test.go

@@ -0,0 +1,35 @@
+package state
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+)
+
+// The test is on write itself: a state file is read before it is
+// written, and a directory in its place fails that read first.
+func TestFailedRenameLeavesNoTemporaryFile(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+
+	// A directory named bans.json cannot be renamed over.
+	err := os.Mkdir(filepath.Join(dir, bansJSON), 0o700)
+	if err != nil {
+		t.Fatalf("mkdir: %v", err)
+	}
+
+	err = write(dir, bansJSON, []byte("{}\n"))
+	if err == nil {
+		t.Error("writing over a directory did not fail")
+	}
+
+	entries, err := os.ReadDir(dir)
+	if err != nil {
+		t.Fatalf("read %s: %v", dir, err)
+	}
+
+	if len(entries) != 1 || entries[0].Name() != bansJSON {
+		t.Errorf("%s holds %v, want only bans.json", dir, entries)
+	}
+}