Keep the bans, the clients and GeoJS's answers in state files (closes #17)

smallwebwaf now copies its state to bans.json, clients.json and
lookups.json in SWWAF_STATE_DIR, as "Persistent state" in SPEC.md
describes, and reads them back at start, so a restart lifts no ban and
gives no client a fresh allowance. Each client gains a history, and a
ban's notes count the netblock's requests. bans.json is written
SWWAF_STATE_WRITE_DELAY after a ban, and every file every
SWWAF_STATE_COUNTER_INTERVAL and at the stop. A ban read back is masked
to its netblock and refuses every client in it. A file that does not
parse, an unknown version, an entry without a field it needs, or an
unwritable directory stops the start.

Deviation: no AS number or name, and no ban cause, reason or lifting yet.

Model: opus-5-5

README.md

@@ -13,18 +13,19 @@ JSON log line for every request.
 
 Status: the first two milestones are built
 (https://git.eeqj.de/sneak/smallwebwaf/issues/13 and
-https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are three parts of
-milestone 3: the static lists and the bans that broken rate limits lead to,
-which come next in the build order, and the header size and the idle time as
-settings, which come last in it. `smallwebwaf` passes each request to the app
-and the app's answer back, unchanged, within its timeouts and size limits, works
-out each client's address, bans a client that sends too many requests, refuses a
-client that comes from a country you refuse or from a network you refuse, lets
-the networks you choose through, and writes a JSON log line for every request.
-It comes as the image the app's own image is built on. The rest of the design
-comes after that, in the order of the build order in [`SPEC.md`](SPEC.md). The
-survey of existing tools that led to the design is in
-[`EVALUATION.md`](EVALUATION.md).
+https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are four parts of
+milestone 3: the static lists, the bans that broken rate limits lead to and the
+JSON state files, which come next in the build order, and the header size and
+the idle time as settings, which come last in it. `smallwebwaf` passes each
+request to the app and the app's answer back, unchanged, within its timeouts and
+size limits, works out each client's address, bans a client that sends too many
+requests, refuses a client that comes from a country you refuse or from a
+network you refuse, lets the networks you choose through, keeps its bans, each
+client's counters and history, and GeoJS's answers in JSON files across
+restarts, and writes a JSON log line for every request. It comes as the image
+the app's own image is built on. The rest of the design comes after that, in the
+order of the build order in [`SPEC.md`](SPEC.md). The survey of existing tools
+that led to the design is in [`EVALUATION.md`](EVALUATION.md).
 
 ## Getting started
 
@@ -46,7 +47,8 @@ works.
 
 To work on the code, `make build` builds the binary alone, with Go installed,
 and `make run` builds and runs it, listening on port 8080 in front of an app at
-`SWWAF_UPSTREAM_URL`, by default `http://127.0.0.1:8081`.
+`SWWAF_UPSTREAM_URL`, by default `http://127.0.0.1:8081`, with its state files
+in `bin/state` unless `SWWAF_STATE_DIR` is set.
 
 ## What it does so far
 
@@ -77,8 +79,9 @@ and `make run` builds and runs it, listening on port 8080 in front of an app at
   bans the client. A client is one IPv4 address, or one IPv6 /64, since one
   abuser usually holds a whole /64. Each window is counted in two fixed buckets,
   the earlier one weighted by how much of it the window still covers. At most
-  20,000 clients are kept, the least recently seen dropped first, and only in
-  memory: a restart starts every client afresh.
+  20,000 clients are kept, the least recently seen dropped first, with their
+  history, and a restart gives no client a fresh allowance (see "State files"
+  below).
 - Bans a client that breaks a rate limit, as "Bans" in [`SPEC.md`](SPEC.md)
   describes: the first ban lasts an hour, and a limit broken again within a day
   of a ban ending bans for three times as long as that ban, so 1, 3, 9, 27 and
@@ -90,13 +93,13 @@ and `make run` builds and runs it, listening on port 8080 in front of an app at
   is not counted for the rate limits. A ban sets the client's counters back to
   zero. Each ban carries notes for deciding whether to lift it: the limit, its
   window and the requests counted in it, the request that broke it, the client's
-  country when it was looked up, how many requests the ban has refused, and how
-  many bans the netblock had before. At most `SWWAF_MAX_BANS` bans are kept,
-  past, active and permanent; past that, the earliest ban of the netblock that
-  has gone longest without a request is dropped first. Bans and their notes are
-  kept in memory only, so a restart lifts every ban, and nothing shows them yet:
-  `bans.json`, which shows them and lets you lift a ban, comes with the state
-  files (https://git.eeqj.de/sneak/smallwebwaf/issues/17).
+  country when it was looked up, the netblock's requests since it was first
+  seen, how many of them the ban has refused, and how many bans the netblock had
+  before. At most `SWWAF_MAX_BANS` bans are kept, past, active and permanent;
+  past that, the earliest ban of the netblock that has gone longest without a
+  request is dropped first. `bans.json` shows the bans and their notes, and a
+  restart lifts none (see "State files" below); lifting a ban by editing it
+  comes with https://git.eeqj.de/sneak/smallwebwaf/issues/68.
 - Refuses a request from a country you refuse with `SWWAF_BAN_RESPONSE`, as soon
   as the client's country is known and before its body is read; such a request
   is not counted for the rate limits. While one of the country lists below is
@@ -184,6 +187,13 @@ it, and the effective settings are logged at start.
 - `SWWAF_BAN_SCOPE_V4_PREFIX` (default `32`): the length of the netblock around
   an IPv4 client that a ban covers, such as `24` to ban the surrounding /24. An
   IPv6 ban covers the client's /64.
+- `SWWAF_STATE_DIR` (default `/var/lib/smallwebwaf`): the directory of the state
+  files, an absolute path. A directory `smallwebwaf` cannot write stops the
+  start.
+- `SWWAF_STATE_WRITE_DELAY` (default `10s`): how long after a ban is made
+  `bans.json` is written, with every ban made in between.
+- `SWWAF_STATE_COUNTER_INTERVAL` (default `15m`): how often every state file is
+  written.
 
 Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are
 bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024
@@ -193,12 +203,13 @@ ISO 3166-1 assigns today, and `xk` for Kosovo, in either case (`de` and `DE` are
 the same); any other code, such as `nk` (North Korea is `kp`) or the withdrawn
 `su`, stops the start, and so does a code on both country lists. `off` switches
 a timeout, a size limit or a rate limit off;
-`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES` and the ban settings cannot be off.
+`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, the ban settings and the state settings
+cannot be off.
 
-Several limits are fixed rather than settings. At most 20,000 clients are kept
-for the rate limits, and an IPv6 client is counted by its /64. A new client
-waits at most a second for its country, and at most 100,000 answers from GeoJS
-are kept, for 7 days each.
+Several limits are fixed rather than settings. At most 20,000 clients are kept,
+with their counters and history, and an IPv6 client is counted by its /64. A new
+client waits at most a second for its country, and at most 100,000 answers from
+GeoJS are kept, for 7 days each.
 
 ## Request log
 
@@ -246,6 +257,48 @@ which it answers `431`, headers slower than `SWWAF_CLIENT_REQUEST_TIMEOUT`,
 whose connection it closes without an answer, and requests it cannot read at
 all, which it answers itself, mostly with `400`.
 
+## State files
+
+`smallwebwaf` keeps its state in memory and a copy of it in three JSON files in
+`SWWAF_STATE_DIR`, `/var/lib/smallwebwaf` by default, as "Persistent state" in
+[`SPEC.md`](SPEC.md) describes. Each has a top-level `version`, 1, and lists its
+entries by client address, with times in UTC.
+
+- `bans.json`: every ban with its notes, indented to be read; a permanent ban's
+  `expires` is `null`.
+- `clients.json`: each client's two buckets in the minute, the hour and the day,
+  and its history: when it was first and last seen, its country as last looked
+  up and when, its requests, how many were forwarded and how many refused, the
+  body bytes in each direction, its responses by status class and its offences
+  by kind. Each client is on a line of its own, so `grep` shows everything about
+  one.
+- `lookups.json`: GeoJS's answers, one to a line, with when GeoJS gave each and
+  when it was last used.
+
+`bans.json` is written `SWWAF_STATE_WRITE_DELAY` after a ban is made, with every
+ban made in between, and every file every `SWWAF_STATE_COUNTER_INTERVAL` and
+when `smallwebwaf` stops. Each write goes to a temporary file in the same
+directory, which then replaces the file, so a crash leaves the old file or the
+new one, whole. A write that fails is logged, and tried again at the next write.
+A hard kill loses what changed since the last write.
+
+At start the files are read back: each client keeps its counts, so a restart
+gives it no fresh allowance, and each ban keeps refusing every client in its
+netblock until it ends, even after `SWWAF_BAN_SCOPE_V4_PREFIX` has changed. A
+netblock whose address has bits past its length, such as `203.0.113.9/24`, is
+read as the netblock it is in, `203.0.113.0/24`. Buckets and answers whose time
+has passed are dropped. A missing file is empty state, as on a first start. A
+file that does not parse, or has another `version`, stops the start with a
+message naming the file, and the line and column where Go's JSON decoder gives
+them; so does a state directory `smallwebwaf` cannot write. So does an entry
+without a field it needs, named with the entry's place in the file: a ban's
+`netblock`, `start` or `expires`, which is `null` for a permanent ban; a
+client's `client`, or the `start` of a window in which it has requests; an
+answer's `client`, `country`, which is `""` for a client GeoJS cannot place, or
+`answered`. An edit made while `smallwebwaf` runs is overwritten by its next
+write: taking it in comes with https://git.eeqj.de/sneak/smallwebwaf/issues/68.
+The AS number and AS name come with their lookup.
+
 ## Why
 
 Small self-hosted sites now receive a great deal of traffic nobody asked for:
@@ -347,9 +400,9 @@ goes through the candidates one by one.
   readable JSON files, written regularly and at every stop, so a restart loses
   nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks
   up the change. Nothing is read from disk while serving a request. The files
-  come in milestone 3 or later (see the build order in [`SPEC.md`](SPEC.md));
-  until then the rate counters, the bans and the GeoJS answers are kept in
-  memory only, and a restart loses them.
+  for the bans, the clients and the GeoJS answers are built (see "State files"
+  above); the others come with their features, and taking in an edit while
+  running comes with https://git.eeqj.de/sneak/smallwebwaf/issues/68.
 - Health checks, the metrics, and listing, adding and lifting bans or asking why
   a given address was refused, all on the one port every request uses: under
   `/_smallwebwaf/` on the app's own address, through traefik like any other
@@ -445,8 +498,9 @@ main "$@"
   the health check on `127.0.0.1`.
 - `smallwebwaf` keeps its state files in `/var/lib/smallwebwaf`. Mount a volume
   there to keep bans and client history when a deploy replaces the container;
-  without one, it still starts. The state files come in milestone 3 or later;
-  until then it writes nothing to disk and needs no volume.
+  without one, it still starts. At each start the `run` script of `smallwebwaf`
+  gives that directory and every file in it to the `smallwebwaf` user, so a host
+  directory mounted there needs no change of owner.
 - `docker stop` has runit stop both processes. `smallwebwaf` then stops taking
   requests and gives those in progress five seconds to finish.
 
@@ -484,11 +538,10 @@ client, for the request log, the metrics and the ban notes, and for the country
 lists and biased limits when you set them. It works with no setup: by default it
 asks the free GeoJS web service, which needs no account and no file. This means
 that, by default, the address of every new visitor is sent to GeoJS. Each answer
-is kept in memory for seven days, and many addresses are asked about in one
-request; writing the answers to disk, so that they survive a restart, comes in
-milestone 3 or later (see the build order in [`SPEC.md`](SPEC.md)). GeoJS
-publishes no rate limit but may block a caller it thinks asks too much; while it
-is not answering, new visitors count as coming from an unknown country, which
+is kept for seven days, in memory and in `lookups.json`, so that it survives a
+restart, and many addresses are asked about in one request. GeoJS publishes no
+rate limit but may block a caller it thinks asks too much; while it is not
+answering, new visitors count as coming from an unknown country, which
 `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses.
 
 To keep your visitors' addresses on your own host, set
@@ -519,9 +572,10 @@ addresses are never sent to GeoJS.
 ## How the code is laid out
 
 - `cmd/smallwebwaf`: the binary, which only calls `internal/smallwebwaf`.
-- `internal/smallwebwaf`: the process: it reads the settings, listens, serves
-  requests until `SIGTERM` or `SIGINT`, and stops. Run as
-  `smallwebwaf healthcheck`, it is the image's health check instead.
+- `internal/smallwebwaf`: the process: it reads the settings and the state
+  files, listens, serves requests until `SIGTERM` or `SIGINT`, and stops,
+  writing the state files. Run as `smallwebwaf healthcheck`, it is the image's
+  health check instead.
 - `internal/config`: reads the settings, the one place they are read.
 - `internal/proxy`: what happens to each request: it works out the client, runs
   the checks, passes the request to the app and the answer back with the
@@ -534,8 +588,10 @@ addresses are never sent to GeoJS.
   long a new ban lasts, and which ban is dropped when `SWWAF_MAX_BANS` are held.
 - `internal/lookup`: looks up each client's country through GeoJS, and keeps the
   answers.
-- `internal/ratelimit`: counts each client's requests and tells when one takes
-  it over a rate limit.
+- `internal/ratelimit`: the table of clients: counts each client's requests,
+  tells when one takes it over a rate limit, and keeps each client's history.
+- `internal/state`: reads the state files at start, and writes them when they
+  are due and at the stop.
 - `internal/requestlog`: the lines on stdout: the request log line and the
   process's own messages.
 - `Dockerfile`: the lint and test phases, then the image, whose last stage
@@ -576,20 +632,24 @@ so that they run in minimal containers.
 - `script/install-precommit`: installs that hook; `make hooks` runs it.
 - `script/build`: builds `bin/smallwebwaf` on the host, with Go installed, for
   working on the code by hand; `make build` runs it.
-- `script/run`: builds `bin/smallwebwaf` with `script/build` and runs it;
-  `make run` runs it.
+- `script/run`: builds `bin/smallwebwaf` with `script/build` and runs it, with
+  its state files in `bin/state` unless `SWWAF_STATE_DIR` is set; `make run`
+  runs it.
 - `script/example-app`: builds the image and, on it, the example app in
-  `deploy/example-app`, runs it, and checks that the health check passes, that a
-  request reaches the app through `smallwebwaf`, and that `sv stop` and
-  `docker stop` stop it in order; then removes the container and both images. It
-  needs network access, for nixpkgs' binary cache, and `script/check` does not
-  run it; `make example-app` does.
+  `deploy/example-app`, runs it with a volume for the state files, and checks
+  that the health check passes, that a request reaches the app through
+  `smallwebwaf`, that a second request in a minute bans the client, that
+  `sv stop` and `docker stop` stop it in order, and that a new container on the
+  same volume still refuses the banned client; then removes the containers, the
+  volume and both images. It needs network access, for nixpkgs' binary cache,
+  and `script/check` does not run it; `make example-app` does.
 
 ## TODO
 
-- The rest of milestone 3, after the bans that broken rate limits lead to and up
-  to the metrics endpoint, and the rest of the design, in the order of the build
-  order in [`SPEC.md`](SPEC.md).
+- The rest of milestone 3, from taking in an admin's edits to the state files
+  (https://git.eeqj.de/sneak/smallwebwaf/issues/68) up to the metrics endpoint,
+  and the rest of the design, in the order of the build order in
+  [`SPEC.md`](SPEC.md).
 
 ## Documents