Take in an admin's edits of the state files while running (closes #68)
check / check (push) Successful in 3m37s

smallwebwaf watches SWWAF_STATE_DIR with fsnotify and takes in a saved
edit of a state file in place of what it held. It knows its own writes
by the SHA-256 of what it last read or wrote; each write first takes in
an edit made since. An edit that does not parse is renamed to
<name>.bad at the next write. Each edit taken in or set aside is logged
and counted. Every ban on a netblock is checked, and the next ban is
worked out from the one that ended last. README.md says how to add and
lift a ban.

Judgement call: a broken edit is set aside at the next write, since an
editor's file can be read half written.

Model: opus-5-5
This commit is contained in:
2026-10-06 12:02:17 +00:00
parent cff385af41
commit c87bcca6e6
13 changed files with 1152 additions and 182 deletions
+69 -32
View File
@@ -15,20 +15,21 @@ Status: the first two milestones are built
(https://git.eeqj.de/sneak/smallwebwaf/issues/13 and
https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are six parts of
milestone 3: the static lists, the bans that broken rate limits lead to and the
JSON state files, which come next in the build order, `observe` mode, which
comes a little later, and the metrics endpoint and the header size and the idle
time as settings, which come last in it. `smallwebwaf` passes each request to
the app and the app's answer back, unchanged, within its timeouts and size
limits, works out each client's address, bans a client that sends too many
requests, refuses a client that comes from a country you refuse or from a
network you refuse, lets the networks you choose through, keeps its bans, each
client's counters and history, and GeoJS's answers in JSON files across
restarts, writes a JSON log line for every request, serves Prometheus metrics to
a scraper that holds the metrics token, and in `observe` mode passes on the
requests it would refuse, logging what it would have done with them. It comes as
the image the app's own image is built on. The rest of the design comes after
that, in the order of the build order in [`SPEC.md`](SPEC.md). The survey of
existing tools that led to the design is in [`EVALUATION.md`](EVALUATION.md).
JSON state files with your edits taken in while it runs, which come next in the
build order, `observe` mode, which comes a little later, and the metrics
endpoint and the header size and the idle time as settings, which come last in
it. `smallwebwaf` passes each request to the app and the app's answer back,
unchanged, within its timeouts and size limits, works out each client's address,
bans a client that sends too many requests, refuses a client that comes from a
country you refuse or from a network you refuse, lets the networks you choose
through, keeps its bans, each client's counters and history, and GeoJS's answers
in JSON files across restarts, takes in your edits of those files while it runs,
writes a JSON log line for every request, serves Prometheus metrics to a scraper
that holds the metrics token, and in `observe` mode passes on the requests it
would refuse, logging what it would have done with them. It comes as the image
the app's own image is built on. The rest of the design comes after that, in the
order of the build order in [`SPEC.md`](SPEC.md). The survey of existing tools
that led to the design is in [`EVALUATION.md`](EVALUATION.md).
## Getting started
@@ -100,9 +101,9 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set.
seen, how many of them the ban has refused, and how many bans the netblock had
before. At most `SWWAF_MAX_BANS` bans are kept, past, active and permanent;
past that, the earliest ban of the netblock that has gone longest without a
request is dropped first. `bans.json` shows the bans and their notes, and a
restart lifts none (see "State files" below); lifting a ban by editing it
comes with https://git.eeqj.de/sneak/smallwebwaf/issues/68.
request is dropped first. `bans.json` shows the bans and their notes, a
restart lifts none, and you add or lift a ban by editing it (see "State files"
below).
- Refuses a request from a country you refuse with `SWWAF_BAN_RESPONSE`, as soon
as the client's country is known and before its body is read; such a request
is not counted for the rate limits. While one of the country lists below is
@@ -335,9 +336,42 @@ without a field it needs, named with the entry's place in the file: a ban's
`netblock`, `start` or `expires`, which is `null` for a permanent ban; a
client's `client`, or the `start` of a window in which it has requests; an
answer's `client`, `country`, which is `""` for a client GeoJS cannot place, or
`answered`. An edit made while `smallwebwaf` runs is overwritten by its next
write: taking it in comes with https://git.eeqj.de/sneak/smallwebwaf/issues/68.
The AS number and AS name come with their lookup.
`answered`. The AS number and AS name come with their lookup.
While it runs, `smallwebwaf` watches `SWWAF_STATE_DIR` and takes in your edit of
a state file as soon as you save it: what the file then holds replaces what
`smallwebwaf` held for it, as if read at start. It tells its own writes from
yours by comparing the file with what it last read or wrote, and before it
writes a file it takes in any edit made since, so your edit is not overwritten;
a change `smallwebwaf` made after you opened the file, such as a new ban, is
lost when you save over it. An edit that would stop the start, because it does
not parse, has another `version` or leaves out a field an entry needs, does not
stop the running `smallwebwaf`: it keeps what it holds, and at the file's next
write renames your file to `<name>.bad`, such as `bans.json.bad`, writes the
file again from memory, and logs the file and where the error is. It waits for
that write because an editor's file can be read before the editor has finished
writing it. Mend the `.bad` file and move it back. A file you remove is written
again at its next write.
To ban a netblock, add an entry to `bans.json` with its `netblock`, its `start`
and its `expires`, `null` for a ban that never ends; its `notes` may be left
out. This `bans.json` bans `203.0.113.0/24` for good:
```json
{
"version": 1,
"bans": [
{
"netblock": "203.0.113.0/24",
"start": "2026-10-06T12:00:00Z",
"expires": null
}
]
}
```
To lift a ban, delete its entry. `smallwebwaf` then forgets the ban, so it does
not make the netblock's next ban longer.
## Metrics
@@ -376,7 +410,10 @@ other request. No metric carries a client's address.
- `smallwebwaf_state_file_writes_total`,
`smallwebwaf_state_file_write_failures_total`,
`smallwebwaf_state_file_last_write_timestamp_seconds` and
`smallwebwaf_state_file_size_bytes`, by `file`.
`smallwebwaf_state_file_size_bytes`, by `file`; and, by `file` too,
`smallwebwaf_state_file_edits_taken_in_total`: your edits taken in, and
`smallwebwaf_state_file_edits_set_aside_total`: those renamed to `<name>.bad`
because they would stop the start.
- Go's own `go_` metrics and the process's `process_` metrics.
The requests Go's HTTP server ends before `smallwebwaf` sees them (see "Request
@@ -484,9 +521,9 @@ goes through the candidates one by one.
readable JSON files, written regularly and at every stop, so a restart loses
nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks
up the change. Nothing is read from disk while serving a request. The files
for the bans, the clients and the GeoJS answers are built (see "State files"
above); the others come with their features, and taking in an edit while
running comes with https://git.eeqj.de/sneak/smallwebwaf/issues/68.
for the bans, the clients and the GeoJS answers are built, with an edit taken
in while running (see "State files" above); the others come with their
features.
- Health checks, the metrics, and listing, adding and lifting bans or asking why
a given address was refused, all on the one port every request uses: under
`/_smallwebwaf/` on the app's own address, through traefik like any other
@@ -678,8 +715,8 @@ addresses are never sent to GeoJS.
answers.
- `internal/ratelimit`: the table of clients: counts each client's requests,
tells when one takes it over a rate limit, and keeps each client's history.
- `internal/state`: reads the state files at start, and writes them when they
are due and at the stop.
- `internal/state`: reads the state files at start, takes in an admin's edit of
one while running, and writes them when they are due and at the stop.
- `internal/requestlog`: the lines on stdout: the request log line and the
process's own messages.
- `Dockerfile`: the lint and test phases, then the image, whose last stage
@@ -691,8 +728,9 @@ addresses are never sent to GeoJS.
Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the
table of clients to 20,000, the GeoJS answers to 100,000 and the banned
netblocks to `SWWAF_MAX_BANS`, dropping the least recently seen, and
`github.com/prometheus/client_golang` keeps the metrics and serves them. The
country codes are the list in `internal/config/config.go`.
`github.com/prometheus/client_golang` keeps the metrics and serves them, and
`github.com/fsnotify/fsnotify` tells `smallwebwaf` when a state file is saved.
The country codes are the list in `internal/config/config.go`.
## Entrypoints
@@ -735,10 +773,9 @@ so that they run in minimal containers.
## TODO
- The rest of milestone 3: taking in an admin's edits to the state files
(https://git.eeqj.de/sneak/smallwebwaf/issues/68), exemptions and the rest of
the request log's fields; then the rest of the design, in the order of the
build order in [`SPEC.md`](SPEC.md).
- The rest of milestone 3: exemptions and the rest of the request log's fields;
then the rest of the design, in the order of the build order in
[`SPEC.md`](SPEC.md).
## Documents