servewaf answers rproxy's question about each request and takes its report (closes #170)

`smallwebwaf servewaf` runs today's checks on the visitor's request each
question carries, answers rproxy, and passes nothing to an app. rproxy reports
every request servewaf answered. A report counts an allowed request's bytes and
writes servewaf's second log line. The table holds a copy of what a report
needs, never the request, so a report cannot reach a request being handled.

Judgement call: a question's body is read before any check; over 64 KiB gets 400.
Judgement call: a report's bytes and durations have ceilings, so they cannot overflow.
Judgement call: a `close` disposition for SWWAF_BAN_RESPONSE=close.

Model: opus-5-5

SPEC.md

@@ -3,7 +3,7 @@
 `smallwebwaf` is a web application firewall and reverse proxy for one app. It
 runs inside the app's container, between traefik and the app. This document says
 what it must do and why, each requirement once. Everything in it is built,
-except the last section, "Planned, not built". `README.md` says how to run it.
+except what the last section lists. `README.md` says how to run it.
 `EVALUATION.md` says why no existing tool was adopted. The reference documents
 under `docs/` give the settings, the formats and the lists. This document links
 to them.
@@ -47,14 +47,16 @@ to them.
     - `internal/smallwebwaf`: the process. It reads the settings, the rule
       files, the lookup database and the state files, listens, and serves until
       `SIGTERM` or `SIGINT`. Then it stops, writing the state files. Run as
-      `smallwebwaf healthcheck`, it is the image's health check instead.
+      `smallwebwaf servewaf`, it answers `rproxy` instead of passing requests to
+      an app (see "`servewaf`"). Run as `smallwebwaf healthcheck`, it is the
+      image's health check instead.
     - `internal/config`: reads the settings, the one place they are read.
     - `internal/proxy`: what happens to each request. It works out the client
       and runs the checks in `check`. It passes the request to the app with the
       standard library's `httputil.ReverseProxy`, within the timeouts and size
       limits, and answers its own endpoints in `answerAdmin`. Once the request
       has ended, it counts bytes, refusals and anomalies, and writes the log
-      line.
+      line. It also holds `servewaf`'s question, answer and report.
     - `internal/ratelimit`: the table of clients. It counts requests, bytes and
       refusals, and keeps each client's history.
     - `internal/bans`: the bans. It holds each netblock's bans with their notes,
@@ -1022,6 +1024,205 @@ answers.
   costs).
 - A ban made through the endpoints starts at once.
 
+## `servewaf`
+
+`smallwebwaf servewaf` runs the checks for `smallwebwaf rproxy`, a small reverse
+proxy in front of each app. `rproxy` asks `servewaf` about each visitor's
+request. `servewaf` answers `rproxy` and passes nothing to an app itself. One
+`servewaf` serves many sites.
+
+This section is the agreement between the two. The header names, the fields and
+the statuses below are exact. Each `SWWAF_` setting and check keeps the meaning
+this document gives it, except as this section says.
+
+`servewaf` starts as `smallwebwaf` with no command does. It reads the settings,
+the rule files, the lookup database and the state files, and listens at
+`SWWAF_LISTEN_ADDR`. It stops in the same orderly way, writing the state files.
+`docs/settings.md` lists the settings it ignores.
+
+### The question
+
+A request carrying `X-Smallwebwaf-Client` is `rproxy` asking about one visitor's
+request. It is the visitor's request as the visitor sent it:
+
+- the method;
+- the path and query as sent;
+- `Host`;
+- every header the visitor sent, including its own `X-Forwarded-For`;
+- the first 64 KiB of the body, or all of it when shorter.
+
+A body whose length the visitor did not announce comes chunked. Any other body
+comes with a `Content-Length` of the part it carries.
+
+`rproxy` sets these headers, replacing any the visitor sent, and adds no others:
+
+- `X-Smallwebwaf-Client`: the visitor's address, which `rproxy` has worked out.
+  `servewaf` takes it as the client, and groups an IPv6 address as "What happens
+  to a request" says.
+- `X-Smallwebwaf-Peer`: the address of `rproxy`'s own TCP peer, for the log
+  line's `peer_ip`.
+- `X-Smallwebwaf-Content-Length`: the body length the visitor announced. It is
+  absent when the visitor announced none.
+- `X-Request-ID`: the request's id, which `rproxy` makes fresh for every request
+  when it arrives. `servewaf` takes it as given and never makes one.
+- `X-Forwarded-Proto`: the scheme.
+
+Before any check, `servewaf` removes the three `X-Smallwebwaf-` headers. It also
+sets `Content-Length` to the length the visitor announced, or removes it. So the
+rule files and the Core Rule Set see the headers the visitor sent, with
+`rproxy`'s `X-Request-ID` and `X-Forwarded-Proto`.
+
+`servewaf` refuses a question with `400` when:
+
+- `X-Smallwebwaf-Client` is not an address, or is an address with a zone, such
+  as `2001:db8::1%eth0`;
+- `X-Request-ID` is absent, empty, over 128 bytes or not valid UTF-8;
+- `X-Request-ID` is the id of a question `servewaf` is still handling, or of a
+  request it holds (see "The report");
+- the body is over 64 KiB.
+
+For a refused question, `servewaf` writes a process warning. It runs no check,
+writes no request log line and counts nothing in the metrics.
+
+These do not refuse a question:
+
+- `X-Smallwebwaf-Peer` absent or not an address: the log line's `peer_ip` is
+  `invalid IP`.
+- `X-Smallwebwaf-Content-Length` not a whole number, or negative: it counts as
+  absent.
+- `X-Forwarded-Proto` absent: the log line's `scheme` is `http`.
+- One of the five headers above sent more than once: `servewaf` uses its first
+  value.
+
+`servewaf` reads the whole body before any check. A question whose body stops
+arriving gets no answer. `servewaf` closes the connection once the 60 seconds
+below run out, or as soon as `rproxy` closes its side. It writes a process
+warning, runs no check, writes no request log line and counts nothing in the
+metrics.
+
+### The answer
+
+`servewaf` runs steps 3 to 12 of "What happens to a request", in that order.
+Every answer but the last below carries `X-Smallwebwaf-Disposition`.
+
+- `X-Smallwebwaf-Disposition: allow`, status `200`, no body. `rproxy` passes the
+  request to the app. With `SWWAF_ADD_LOOKUP_HEADERS` the answer also carries
+  `X-Client-ASN` and `X-Client-Country`, which `rproxy` adds to the request it
+  passes on. A header whose value is unknown is left out.
+- `X-Smallwebwaf-Disposition: deny`, with the status, headers and body the
+  visitor is to get. `rproxy` gives them to the visitor, less this header. The
+  status is one of:
+    - `SWWAF_BAN_RESPONSE`, for a refusal;
+    - `403`, for a `block` rule or the Core Rule Set;
+    - `404`, for a path under `/_smallwebwaf/` that passes the checks, so that
+      the app never sees that prefix.
+- `X-Smallwebwaf-Disposition: close`, status `200`, no body. This takes the
+  place of a refusal while `SWWAF_BAN_RESPONSE` is `close`. `rproxy` closes the
+  visitor's connection without an answer.
+- No such header, status `400`, for a question `servewaf` refuses. `rproxy`
+  treats this answer as it treats no answer at all.
+
+In `observe` mode a request that `enforce` mode would refuse is answered
+`allow`, and its log line has `would_action`. A path under `/_smallwebwaf/` is
+still answered `404`.
+
+The Core Rule Set inspects the body a question carries only while
+`SWWAF_WAF_BODY_LIMIT` is set. It inspects at most 64 KiB less one byte. So it
+does not inspect a JSON or XML body of 64 KiB or more, which may have been cut
+short.
+
+### The report
+
+Once the visitor's answer has ended, `rproxy` sends `POST /_smallwebwaf/report`
+for each request `servewaf` answered with `allow`, `deny` or `close`. It sends
+none after a `400` or no answer. The report carries no `X-Smallwebwaf-Client`.
+Its body is one JSON object with all of these fields, named as the request log's
+fields of the same meaning:
+
+- `request_id`: the question's `X-Request-ID`;
+- `status`: what the visitor got, `0` if nothing, otherwise from `100` to `599`;
+- `request_bytes` and `response_bytes`: the body bytes each way, with what a
+  WebSocket carried each way, each a whole number from `0` to `1125899906842624`
+  (1 PiB);
+- `duration_total`: milliseconds from the request's headers being read to the
+  end of the answer, from `0` to `1000000000000` (about 31 years);
+- `duration_upstream_total`: milliseconds from the hand-over to the app to the
+  end of the answer, in the same range, and `0` for a request not passed to the
+  app.
+
+`servewaf` answers `400` and writes a process warning when the body:
+
+- is not one JSON object, such as `null`, or an object followed by other data;
+- lacks one of these fields, or gives it as `null`;
+- gives a value of the wrong type, or out of its range;
+- is over 1 MiB.
+
+It answers `204` to any other report, also for a request it does not hold, which
+it ignores. It ignores a field not listed here. `rproxy` does not act on the
+answer.
+
+For a request it holds, `servewaf` writes the request's last log line (see "The
+log"). For a request it allowed, it first:
+
+- counts the bytes as `SWWAF_BYTES_COUNT` says, toward the client's byte limits
+  at the percentages it gave that request;
+- bans a client that passes a byte limit;
+- adds the request with its status to the client's history, the anomaly counters
+  and the metrics.
+
+A denied request's counting is finished at its decision, so its report counts
+nothing.
+
+`servewaf` holds each answered request until its report comes, up to 10,000
+requests. Past that, the request answered longest ago is dropped. A request
+whose report never comes gets no last log line. If it was allowed, it is in
+neither the client's history nor the metrics, and its bytes are not counted.
+
+`servewaf` holds only the decision's log line and what counting needs. It never
+holds the request or response objects, and no `Authorization` or `Cookie`. A
+report reaches only what is held, so it can never reach a question still being
+handled. `servewaf` ignores a report naming such a question.
+
+### The log
+
+Each request gets three request log lines, joined by `request_id`:
+
+- `rproxy`'s short line;
+- `servewaf`'s line at its decision;
+- `servewaf`'s line when the report comes, with everything known: the decision
+  line's fields and the report's together.
+
+The repetition is intended. The field `servewaf_line` says which of `servewaf`'s
+two lines a line is. `docs/request-log.md` gives the fields.
+
+### `servewaf`'s own endpoints
+
+A request without `X-Smallwebwaf-Client` is for `servewaf` itself, at its own
+address on the private network. The internet must not be able to reach that
+address. Its endpoints are:
+
+- the health check, the metrics, and the ban and client endpoints, as
+  `docs/admin-endpoints.md` lists them, with the same tokens;
+- the report.
+
+Anything else gets `404`. No check runs on these requests, since their peers are
+`rproxy` and the admin's tools. The report needs no token, so anything on the
+private network can send one or ask a question.
+
+A request and its answer have 60 seconds together, from the request's headers.
+`servewaf` closes a connection idle for 120 seconds. It takes a request line and
+headers up to 1 MiB, Go's default, so that it never refuses a question for the
+size of a visitor's headers.
+
+### Sites
+
+One `servewaf` keeps one table of clients, counters and bans for every site that
+asks it. A client banned through one site is refused on all of them, and its
+requests on all of them count toward one set of limits. Log lines, alerts raised
+by a request and ban notes name the site by its host.
+
+`docs/metrics.md` says how its metrics count.
+
 ## Deployment
 
 `smallwebwaf` runs inside the app's own container: the app's Dockerfile builds
@@ -1187,9 +1388,9 @@ and `script/gitea-requests` checks those notes.
 
 ## Planned, not built
 
-These two points are specified as later work, and nothing in the code does them
-yet:
+These points are specified as later work, and nothing in the code does them yet:
 
+- Planned: `smallwebwaf rproxy`, the reverse proxy in front of each app.
 - Planned: sharing bans between the instances of several apps, by running one
   CrowdSec engine per host that every instance reports its bans to. It is
   reconsidered once real ban volumes are known. Until then each instance keeps