SPEC and README: rule set changes, error bursts, admin bans, GeoJS answers (closes #6)

Address the second review of the spec update. The Core Rule Set allows
PUT, PATCH and DELETE and the headers git and curl send, inspects only
bodies it can read, leaves responses alone, and by default switches off
the rules that refuse a code forge's ordinary files, uploads and git
traffic. The error burst counts only the sidecar's own refusals. Bans an
admin made are never dropped. A limit ban resets the client's counters.
GeoJS answers move to their own `lookups.json`. The spec now says which
address is the client when every forwarded address is trusted, that the
exclusive country list refuses private addresses, which are never sent to
GeoJS, what `close` gives behind traefik, and Spamhaus's terms for DROP.

Model: opus-5-5

README.md

@@ -79,7 +79,8 @@ goes through the candidates one by one.
       running;
     - the OWASP Core Rule Set, run by the Coraza engine, refusing the requests
       it flags;
-    - trap paths and bursts of error responses.
+    - trap paths, and a ban for a client that the rule files or the Core Rule
+      Set refuse again and again.
 - Bans:
     - a clear sign of attack, such as a probe for a `.env` file or a scanner's
       user agent, bans for seven days on the first request, and any further
@@ -102,8 +103,8 @@ goes through the candidates one by one.
   fields, the decision taken and why, AS number and country, and timings.
   Optionally also sent to a remote syslog server.
 - Prometheus metrics on their own port.
-- State (bans with their notes, each client's counters, history and lookup
+- State (bans with their notes, each client's counters and history, the GeoJS
-  answer, the reputation cache, the alerting state) held in memory and kept in
+  answers, the reputation cache, the alerting state) held in memory and kept in
   readable JSON files, written regularly and at every stop, so a restart loses
   nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks
   up the change. Nothing is read from disk while serving a request.
@@ -131,8 +132,9 @@ For each request `smallwebwaf`:
   client at once for a clear sign of attack;
 - forwards it to the app and streams the response back, within the size and time
   limits;
-- counts the bytes and any error response, bans the client if it broke a limit,
+- counts the bytes and any refusal by the rule files or the Core Rule Set, bans
-  updates its history, sends any alerts that are due, and writes the log line.
+  the client if it broke a limit, updates its history, sends any alerts that are
+  due, and writes the log line.
 
 A minimal deployment beside an app in docker-compose. `UPSTREAM_URL` is the only
 setting:
@@ -204,6 +206,11 @@ request. GeoJS publishes no rate limit but may block a caller it thinks asks too
 much; while it is not answering, new visitors count as coming from an unknown
 country, which `EXCLUSIVELY_ALLOWED_COUNTRIES` refuses.
 
+Neither source can place a private address, so a client on one, such as a
+visitor on your local network, another container or your monitoring, has no
+country: `EXCLUSIVELY_ALLOWED_COUNTRIES` refuses it unless you list it in
+`ALLOW_NETS`. Such addresses are never sent to GeoJS.
+
 ## Documents
 
 - [`SPEC.md`](SPEC.md): the design.

SPEC.md

@@ -63,7 +63,7 @@ directory of hand-editable text files.
     - Attack detection: regex rules read from a directory of plain text rule
       files and read again whenever the files change (see "Rule files"), Coraza
       with the OWASP Core Rule Set, and simple signals (requests for listed trap
-      paths, bursts of error responses).
+      paths, bursts of requests refused by the rule files or the Core Rule Set).
     - Ban ledger: active and past bans with their notes, and each client's
       history, held in memory and kept in JSON files that the sidecar watches
       for an admin's edits (see "Bans" and "Persistent state"). No database of
@@ -101,8 +101,13 @@ whole exchange.
 
 - Identify the client.
     - If the TCP peer is inside `TRUSTED_PROXIES`, walk `X-Forwarded-For` from
-      the right and take the first address not inside `TRUSTED_PROXIES`.
+      the right and take the first address not inside `TRUSTED_PROXIES`. If
-      Otherwise use the TCP peer address and ignore the header.
+      every address in the header is inside `TRUSTED_PROXIES`, take the
+      leftmost, so a visitor on a private network who comes through traefik is
+      known by its own address, not traefik's. If there is no header, as when
+      another container calls the sidecar directly, the TCP peer is the client.
+    - If the TCP peer is outside `TRUSTED_PROXIES`, it is the client, and the
+      header is ignored.
     - IPv6 clients are grouped by prefix (`IPV6_GROUP_PREFIX`, default 64) for
       counting and banning, because one abuser usually controls a whole /64. A
       client is therefore one IPv4 address or one IPv6 group, a /64 by default.
@@ -115,9 +120,10 @@ whole exchange.
   permanent (see "Bans").
 - Look up the AS number and country, unless `LOOKUP_SOURCE` is `off`. With
   GeoJS, a client whose answer is not yet kept waits for it, up to
-  `LOOKUP_TIMEOUT`. A client the lookup cannot place has an unknown country and
+  `LOOKUP_TIMEOUT`. A client the lookup cannot place, which includes every
-  AS number: the exclusive country list refuses it, the biased thresholds give
+  private, loopback and link-local address, has an unknown country and AS
-  it `UNKNOWN_LIMIT_PERCENT`, and nothing else treats it differently.
+  number: the exclusive country list refuses it, the biased thresholds give it
+  `UNKNOWN_LIMIT_PERCENT`, and nothing else treats it differently.
 - Country lists. A client whose country is in `DENIED_COUNTRIES`, or, when
   `EXCLUSIVELY_ALLOWED_COUNTRIES` is set, is not in it, is refused with
   `BAN_RESPONSE`. Every step up to here needs only the client's address, so the
@@ -139,20 +145,23 @@ whole exchange.
   in file name order then line order. Each rule that matches takes its action:
   only log, refuse with 403, or refuse and ban. Matching stops at the first rule
   that refuses or bans.
-- Core Rule Set inspection of the request (headers, URL, and body up to
+- Core Rule Set inspection of the request: its headers, its URL, and its body up
-  `WAF_BODY_LIMIT`). In `block` mode, the default, a match at or over the
+  to `WAF_BODY_LIMIT` when the body is form data, multipart, JSON or XML, the
-  anomaly threshold is refused with 403; in `detect` mode it is only logged and
+  kinds the Core Rule Set can read. Any other body, such as a git push or a
-  alerted.
+  container image layer, reaches the app uninspected, and so does a JSON or XML
+  body larger than `WAF_BODY_LIMIT`, which cannot be read in part. Responses are
+  not inspected. In `block` mode, the default, a match at or over the anomaly
+  threshold is refused with 403; in `detect` mode it is only logged and alerted.
 - Forward to `UPSTREAM_URL`, streaming. Add `X-Forwarded-For` and, if enabled,
   `X-Client-ASN` and `X-Client-Country` for the app's own logs.
 - After the response.
     - Add response bytes (and request body bytes) to the byte counters. A client
       whose byte total passes a byte limit times the percentage has broken that
       limit and is banned. The response in progress is not cut off.
-    - Count the client's requests answered with 403 or 404, whether the app gave
+    - Count the client's requests that the sidecar refused after a rule file or
-      that answer or the sidecar refused the request after a rule file or Core
+      Core Rule Set match; answers the app gave are not counted. More than
-      Rule Set match. More than `ERROR_BURST_THRESHOLD` of them within a minute
+      `ERROR_BURST_THRESHOLD` of them within a minute breaks a limit and bans
-      breaks a limit and bans the client.
+      the client.
     - Update the client's history and the anomaly counters, and evaluate alert
       thresholds.
 
@@ -167,20 +176,25 @@ whole exchange.
   shutdown and loaded again at start (see "Persistent state"), so a restart does
   not hand every client a fresh allowance. Buckets whose time has passed are
   discarded on load.
-- Memory is bounded by `MAX_TRACKED_CLIENTS` and `MAX_BANS`. When the table of
+- Memory is bounded by `MAX_TRACKED_CLIENTS`, `MAX_BANS` and the number of GeoJS
-  clients is full, the least recently seen client is dropped first, with its
+  answers kept (see "Configuration surface"). When the table of clients is full,
-  history and its lookup answer. When `MAX_BANS` bans are held, the ban that has
+  the least recently seen client is dropped first, with its history. When
-  gone longest without a request from its netblock is dropped first, whether it
+  `MAX_BANS` bans the sidecar made are held, the one that has gone longest
-  is past, active or permanent (see "Bans").
+  without a request from its netblock is dropped first, whether it is past,
+  active or permanent. Bans an admin made are never dropped, and there are as
+  many as the admin adds (see "Bans").
 
 ## Bans
 
 A ban refuses every request from a netblock, answering with `BAN_RESPONSE`
 (`403` by default), until the ban ends. A permanent ban does not run out: it
-ends only when an admin lifts it, or when a new ban is made while `MAX_BANS`
+ends when an admin lifts it or, if the sidecar made it, when a new ban is made
-bans are held and it is the ban that has gone longest without a request. A
+while `MAX_BANS` bans the sidecar made are held and it is the one that has gone
-scanner whose ban was dropped that way is refused and banned again by its next
+longest without a request. A scanner whose ban was dropped that way is refused
-probe.
+and banned again by its next probe. A ban an admin made, whose cause is `admin`,
+is never dropped and does not count toward `MAX_BANS`, so such bans can never
+fill the table, however many there are. An admin who wants to keep a ban the
+sidecar made sets its cause to `admin`.
 
 The netblock a ban covers is the client: its IPv4 address, or its IPv6 group of
 `IPV6_GROUP_PREFIX` (a /64 by default). `BAN_SCOPE_V4_PREFIX` can widen an IPv4
@@ -201,8 +215,8 @@ to a ban, each by its own rule:
     - Once that ban has run out, the netblock is served like any other, but its
       next clear sign of attack bans it permanently at once.
 - A broken limit: a request over a request limit, a response that takes the
-  client's byte total past a byte limit, or more error responses than
+  client's byte total past a byte limit, or more than `ERROR_BURST_THRESHOLD`
-  `ERROR_BURST_THRESHOLD` within a minute.
+  requests within a minute refused after a rule file or Core Rule Set match.
     - The first such ban, or one that comes more than `LIMIT_BAN_REPEAT_WINDOW`
       (default `24h`) after the last such ban ended, lasts `LIMIT_BAN_DURATION`
       (default `1h`).
@@ -210,13 +224,16 @@ to a ban, each by its own rule:
       long as the last one: one hour, then 3, 9, 27 and 81 hours.
     - A ban that would be longer than `MAX_BAN_DURATION` (default `7d`) is
       permanent instead; with the defaults, that is the sixth ban in a row.
+    - Each such ban sets the client's counters for every limit back to zero, so
+      once it ends only new requests can break a limit again; the client's
+      history keeps its totals.
 
 A Core Rule Set match refuses only the request it matched, without a ban,
 because the Core Rule Set has false positives; a `block` rule does the same.
-Those refusals count toward the error burst like any other 403, so a client that
+Those refusals count toward the error burst, so a client that keeps setting them
-keeps setting them off is banned under the second rule. Requests refused under a
+off is banned under the second rule. Requests refused under a ban are not
-ban are not counted against limits, but they are counted in the client's history
+counted against limits, but they are counted in the client's history and in the
-and in the ban's notes.
+ban's notes.
 
 Every ban carries notes with what an admin needs to decide whether to lift it
 (see "Persistent state"). An admin can add or lift a ban at any time by editing
@@ -283,8 +300,9 @@ The settings, by group:
     - `IPV6_GROUP_PREFIX` (default `64`).
     - `MAX_TRACKED_CLIENTS` (default `20000`): clients held in memory and in
       `clients.json` (see "Persistent state").
-    - `MAX_BANS` (default `5000`): bans held in memory and in `bans.json`, past,
+    - `MAX_BANS` (default `5000`): bans the sidecar made, past, active and
-      active and permanent (see "Bans").
+      permanent, held in memory and in `bans.json`. Bans an admin made are kept
+      besides these and never dropped (see "Bans").
 - Persistent state
     - `STATE_DIR` (default `/data`): the JSON state files. The image declares
       `/data` as a volume, so it is writable even when nothing is mounted; a
@@ -294,8 +312,8 @@ The settings, by group:
       permanent, `bans.json` is written this long later with every change made
       in between, so a burst of changes becomes one write.
     - `STATE_COUNTER_INTERVAL` (default `15m`): how often `clients.json`,
-      `reputation.json` and `alerts.json` are written, and `bans.json` when only
+      `lookups.json`, `reputation.json` and `alerts.json` are written, and
-      the counts in its notes changed.
+      `bans.json` when only the counts in its notes changed.
 - Logging
     - The request log on stdout is always on and has no switch.
     - `LOG_LEVEL` (default `info`): for the process's own messages (start-up,
@@ -400,10 +418,13 @@ The settings, by group:
       `country_code`, `asn` and `organization_name` (the AS name). A
       `country_code` of `null`, which GeoJS gives for some ranges, and an `asn`
       of `64512`, which it gives when it knows none, count as unknown. GeoJS is
-      told the address of every new visitor.
+      told the address of every new visitor, except private, loopback and
-    - Each GeoJS answer is kept for 7 days with the client's record, in memory
+      link-local addresses, which no source can place and which are never sent.
-      and in `clients.json`, so it survives a restart; after 7 days the client's
+    - Each GeoJS answer is kept for 7 days in memory and in `lookups.json`,
-      next request asks again. A client dropped from the table loses its answer.
+      apart from the table of clients, so it survives a restart and outlasts a
+      client dropped from that table; after 7 days the client's next request
+      asks again. Up to 100,000 answers are kept, about 15 MiB; when that many
+      are held, the answer used longest ago goes first.
     - `LOOKUP_TIMEOUT` (default `1s`): how long a client with no kept answer
       waits for one; GeoJS normally answers in a fraction of that. At most one
       request to GeoJS is under way at a time, the addresses that arrive
@@ -431,9 +452,12 @@ The settings, by group:
       listed country is refused.
     - `EXCLUSIVELY_ALLOWED_COUNTRIES`: for example `us,de`. Every request from
       any other country is refused, and so is every request from a client the
-      lookup cannot place, such as an address missing from the database or one
+      lookup cannot place: one whose address is missing from the database, one
-      GeoJS has not answered for in time. A list that let those through would
+      GeoJS has not answered for in time, and any client on a private address,
-      let every new client in whenever GeoJS stops answering.
+      such as a visitor on the local network, another container or internal
+      monitoring. Those that should reach the app go in `ALLOW_NETS`. A list
+      that let unplaced clients through would let every new client in whenever
+      GeoJS stops answering.
     - Both may be set. `DENIED_COUNTRIES` then adds nothing, since the exclusive
       list already refuses every other country, and a code on both lists stops
       the start with a message naming it.
@@ -477,27 +501,55 @@ The settings, by group:
     - `WAF_PARANOIA_LEVEL` (default `1`), `WAF_ANOMALY_THRESHOLD` (default `5`):
       the Core Rule Set's own two tuning values, at the Core Rule Set's own
       defaults.
-    - `WAF_DISABLED_RULES` (default `920420,920440`): rule ids to switch off
+    - The sidecar also changes the Core Rule Set in four ways that no setting
-      when an app trips a false positive. The two in the default refuse requests
+      undoes, since in front of gitea each would otherwise refuse ordinary
-      by content type and by file extension; in front of a code forge they would
+      requests:
-      refuse git's clone and push over HTTP and the display of source files such
+        - PUT, PATCH and DELETE are allowed methods besides GET, HEAD, POST and
-      as `.sh` or `.sql`. A list given replaces the default, so include them in
+          OPTIONS; APIs, container image pushes and package uploads use them.
-      it.
+          Other methods stay refused.
+        - The request headers `Content-Encoding` and `Expect` are allowed: git
+          compresses a fetch request over 1 KiB and says so in
+          `Content-Encoding`, and curl and git send `Expect` before some large
+          uploads. The other headers the Core Rule Set refuses, such as `Proxy`,
+          stay refused.
+        - Only a body the Core Rule Set can read is inspected (see "Data flow
+          for one request"). It would read any other body as form data, where
+          binary content such as a git push trips rules written for text.
+        - Responses are not inspected. A raw file from a repository, such as a
+          shell script, looks to the response rules like source code leaking
+          from the server.
+    - `WAF_DISABLED_RULES` (default
+      `920340,920420,920440,920640,930130,930140,932180`): rule ids to switch
+      off when an app trips a false positive. The default switches off the rules
+      that refuse a request for the type of its body, when it is missing or not
+      on the Core Rule Set's short list (920340, 920420, 920640); for its file
+      extension, such as `.sh` or `.sql` (920440); for a file or directory name
+      in its path, such as `.git/`, `.gitignore`, `Dockerfile`, `package.json`
+      or an editor's settings directory (930130, 930140); and for the name of an
+      uploaded file, such as `debug.log` or `config.yml` (932180). In front of a
+      code forge these refuse git over HTTP, container image and package
+      uploads, views of ordinary files in a repository, and logs attached to
+      issues. The default rule file still bans probes for `.env` and `.git` at
+      the site root (see "Rule files"). A list given replaces the default, so
+      include them in it.
     - `WAF_EXEMPT_PATHS`: path prefixes not inspected.
-    - `WAF_BODY_LIMIT` (default `128K`): bodies are inspected up to this size
+    - `WAF_BODY_LIMIT` (default `128K`): bodies the Core Rule Set reads are
-      and streamed beyond it without buffering, so large uploads and git pushes
+      inspected up to this size and streamed beyond it without buffering, so
-      are not held in memory.
+      large uploads are not held in memory.
     - `TRAP_PATHS`: paths the app never serves and only scanners ask for, for
       example `/wp-login.php,/xmlrpc.php` in front of gitea. A request for one
       is a clear sign of attack. This is the env-var short form of a `path` rule
       with the `ban` action, for deployments that mount no rule files.
-    - `ERROR_BURST_THRESHOLD` (default `30`): responses with status 403 or 404
+    - `ERROR_BURST_THRESHOLD` (default `30`): requests per client per minute
-      per client per minute; more than this breaks a limit (see "Bans").
+      that the sidecar refused after a rule file or Core Rule Set match; more
-      Scanners walking lists of paths cause hundreds a minute; a person rarely
+      than this breaks a limit (see "Bans"). A client trying one attack after
-      causes more than a few. 401 is not counted: git and container registry
+      another is refused many times a minute; a person rarely more than a few
-      clients send their first request without credentials and are answered 401
+      times. Answers the app gives are not counted, since they do not tell a
-      at the start of every push, every fetch from a private repository and
+      scanner from an ordinary client: in front of gitea, container image and
-      every image pull.
+      package clients are answered 404 by design, for each layer a push checks
+      and each package a lookup asks about, often hundreds of times a minute,
+      and git and registry clients are answered 401 at the start of every push,
+      every fetch from a private repository and every image pull.
 - Bans (R5), following the rules under "Bans"
     - `ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of
       attack.
@@ -509,21 +561,27 @@ The settings, by group:
       instead.
     - `BAN_RESPONSE` (default `403`): `403`, `429`, or `close` to drop the
       connection without an answer. `403` makes a ban easy to recognise when
-      debugging; `close` tells the client nothing.
+      debugging. Behind traefik, `close` does not leave the client unanswered:
+      traefik answers `502`, as it does whenever its backend drops a connection.
     - `BAN_SCOPE_V4_PREFIX` (default `32`): widen to for example `24` to ban the
       surrounding netblock.
 - Reputation (R6). No source is on by default. A DNS blocklist would be told the
   address of every visitor. AbuseIPDB and CrowdSec need an account or an engine
   of the operator's own. A downloaded list reveals nothing about visitors, but
-  each sidecar fetches its own copy, and the list most fit to be a default,
+  the list most fit to be a default, Spamhaus DROP, must not be downloaded
-  Spamhaus DROP, may be fetched at most once a day: a default would break that
+  automatically more than once an hour, and Spamhaus may block an address that
-  on any host that runs several sidecars.
+  downloads it too often. Each sidecar fetches its own copy, so on a host that
+  runs several sidecars, which share one address, downloads would often come
+  less than an hour apart.
     - `BLOCKLIST_URLS`: text files of addresses and netblocks, one per line;
       anything after a `;` or `#` on a line is ignored. For example the Spamhaus
-      DROP list, `https://www.spamhaus.org/drop/drop.txt`; Spamhaus asks that it
+      DROP list, `https://www.spamhaus.org/drop/drop.txt`. Spamhaus's terms for
-      be fetched at most once a day and that products using it credit The
+      it, on its DROP page: automated downloads must be at least one hour apart,
-      Spamhaus Project. Refreshed every `BLOCKLIST_REFRESH` (default `24h`); the
+      and once a day is more than enough in most cases; a product that uses it
-      last good copy is kept on failure and across restarts.
+      must credit The Spamhaus Project and keep the list's date and copyright
+      lines with the data. Refreshed every `BLOCKLIST_REFRESH` (default `24h`);
+      the last good copy is kept whole, comment lines included, on failure and
+      across restarts.
     - `BLOCKLIST_ACTION` (default `deny`): `deny`, `limit:<percent>`, or `log`.
       `deny` refuses every request from a listed address, which suits lists of
       networks that send nothing legitimate, such as DROP. `limit:<percent>`
@@ -691,17 +749,21 @@ files are meant for people as well: an admin can read or edit them at any time,
 and the running sidecar takes the edit in.
 
 - Files, each holding one kind of state:
-    - `bans.json`: active and past bans, up to `MAX_BANS`. Per entry: the
+    - `bans.json`: active and past bans, up to `MAX_BANS` that the sidecar made
-      netblock, start, expiry (`null` for permanent), what caused it (`attack`,
+      and every ban an admin made. Per entry: the netblock, start, expiry
-      `limit`, `admin` or `crowdsec`), a short reason, when it was lifted if an
+      (`null` for permanent), what caused it (`attack`, `limit`, `admin` or
-      admin lifted it, and a `notes` field (below).
+      `crowdsec`), a short reason, when it was lifted if an admin lifted it, and
+      a `notes` field (below).
     - `clients.json`: per client, its minute, hour and day counters and its
       history since it was first seen: first and last time seen, the AS number,
-      AS name and country last looked up and when (with GeoJS, the answer kept
+      AS name and country last looked up and when, total requests and bytes in
-      for 7 days), total requests and bytes in each direction, how many requests
+      each direction, how many requests were forwarded and how many refused,
-      were forwarded and how many refused, responses by status class, and
+      responses by status class, and offences by kind. Clients that were never
-      offences by kind. Clients that were never banned are kept too, so every
+      banned are kept too, so every client's history survives a restart;
-      client's history survives a restart; `GET /clients/<ip>` shows it.
+      `GET /clients/<ip>` shows it.
+    - `lookups.json`: GeoJS answers, one per client: the AS number, AS name and
+      country, when GeoJS was asked and when the answer was last used; up to
+      100,000, each for 7 days.
     - `reputation.json`: the last good copy of each list fetched from a URL,
       cached DNSBL and reputation API verdicts, each with the time it was
       fetched, and the AbuseIPDB checks spent today, so a restart does not reset
@@ -733,20 +795,22 @@ and the running sidecar takes the edit in.
       that is under 2 GiB of disk writes a day.
     - `bans.json` takes about 2 KiB per ban and at most about 8 KiB, since the
       texts in the notes are cut short. At the default `MAX_BANS` of 5,000 it is
-      about 10 MiB, and never more than about 40 MiB. It is written when a ban
+      about 10 MiB, and never more than about 40 MiB, plus whatever bans an
-      is made, lifted or made permanent, at most once every 10 seconds, and
+      admin made. It is written when a ban is made, lifted or made permanent, at
-      otherwise with the 15-minute write, so its writes follow the bans made:
+      most once every 10 seconds, and otherwise with the 15-minute write, so its
-      with a full file, a hundred new bans a day come to about 1 GiB of disk
+      writes follow the bans made: with a full file, a hundred new bans a day
-      writes.
+      come to about 1 GiB of disk writes.
+    - `lookups.json` takes about 150 bytes per answer, about 15 MiB when full.
+      Written every 15 minutes, that is under 1.5 GiB of disk writes a day.
     - `reputation.json` and `alerts.json` are usually a few MiB or less.
     - Writing a file of these sizes takes well under a second on an ordinary
       disk, in the background, so rewriting whole files needs nothing cleverer.
 - Format: indented JSON with a top-level `version` number, entries sorted by
   client address, times in RFC 3339 UTC, durations and sizes as plain numbers
   with the unit in the field name. The aim is that a person can open `bans.json`
-  in an editor, find an address, and remove or add an entry. `clients.json` puts
+  in an editor, find an address, and remove or add an entry. `clients.json` and
-  each client on one line instead, which halves its size and lets `grep` show
+  `lookups.json` put each entry on one line instead, which halves their size and
-  everything about one client.
+  lets `grep` show everything about one client.
 - Writing:
     - serialise from a snapshot taken under the lock, so requests are not held
       up while the file is written;
@@ -756,8 +820,8 @@ and the running sidecar takes the edit in.
     - `bans.json` is written `STATE_WRITE_DELAY` after a ban is made, lifted or
       made permanent. Changes that only update the counts in its notes wait for
       the write every `STATE_COUNTER_INTERVAL`, as all of `clients.json`,
-      `reputation.json` and `alerts.json` do. All four are written on orderly
+      `lookups.json`, `reputation.json` and `alerts.json` do. All five are
-      shutdown (SIGTERM);
+      written on orderly shutdown (SIGTERM);
     - before writing a file, the sidecar checks whether it changed on disk since
       the sidecar last read or wrote it; if it did, the sidecar takes that edit
       in first (below), so an admin's edit is never overwritten.
@@ -954,12 +1018,17 @@ networks:
       at the defaults, one larger than 100 MB or taking more than 60 seconds to
       upload is cut off, so a gitea that takes large pushes needs
       `CLIENT_REQUEST_MAX_BYTES`, `UPSTREAM_REQUEST_MAX_BYTES`,
-      `CLIENT_REQUEST_TIMEOUT` and `UPSTREAM_REQUEST_TIMEOUT` raised to fit.
+      `CLIENT_REQUEST_TIMEOUT` and `UPSTREAM_REQUEST_TIMEOUT` raised to fit. The
-      `WAF_BODY_LIMIT` keeps pack uploads out of memory.
+      Core Rule Set does not read a pack upload, which streams through without
-    - The default `WAF_DISABLED_RULES` lets git's clone and push over HTTP, and
+      being held in memory.
-      the display of source files, through the Core Rule Set. Pages where people
+    - With the sidecar's changes to the Core Rule Set and the default
-      post code (issues, pull requests, the web editor) can still trip other
+      `WAF_DISABLED_RULES` (see "Configuration surface", attack detection),
-      rules; such a match refuses only that request, and the request log names
+      git's clone, fetch and push over HTTP, the API, container images,
+      packages, and views of ordinary files pass the Core Rule Set. Requests
+      that carry text people write, such as issues, pull requests, searches, the
+      web editor and uploads, can still trip other rules, and so can a file
+      whose name the Core Rule Set takes for an attack, such as an `.xhtml`
+      page. Such a match refuses only that request, and the request log names
       the rule in `waf_rule_ids` for `WAF_DISABLED_RULES`.
     - Archive download and blame or history pages are what scrapers hammer;
       request limits do most of the work there.
@@ -1001,8 +1070,9 @@ networks:
 - Forged `X-Forwarded-For`: handled by believing the header only from a peer
   inside `TRUSTED_PROXIES` and by walking it from the right. The default trusts
   every private address, which in the intended deployment means traefik; where
-  other containers can reach the sidecar directly, setting `TRUSTED_PROXIES` to
+  other containers can reach the sidecar directly, or visitors on a private
-  traefik's own network closes that gap.
+  network reach traefik, they can name another address in the header, and
+  setting `TRUSTED_PROXIES` to traefik's own network closes that gap.
 - Many clients behind one address (mobile carriers, offices, Tor): they share
   its limits and its bans. The default limits sit several times above what one
   busy person produces, `RATE_LIMIT_EXEMPT_NETS` and `ALLOW_NETS` take known