Take in an admin's edits of the state files while running (closes #68)
check / check (push) Successful in 3m37s
check / check (push) Successful in 3m37s
smallwebwaf watches SWWAF_STATE_DIR with fsnotify and takes in a saved edit of a state file in place of what it held. It knows its own writes by the SHA-256 of what it last read or wrote; each write first takes in an edit made since. An edit that does not parse is renamed to <name>.bad at the next write. Each edit taken in or set aside is logged and counted. Every ban on a netblock is checked, and the next ban is worked out from the one that ended last. README.md says how to add and lift a ban. Judgement call: a broken edit is set aside at the next write, since an editor's file can be read half written. Model: opus-5-5
This commit is contained in:
@@ -15,20 +15,21 @@ Status: the first two milestones are built
|
||||
(https://git.eeqj.de/sneak/smallwebwaf/issues/13 and
|
||||
https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are six parts of
|
||||
milestone 3: the static lists, the bans that broken rate limits lead to and the
|
||||
JSON state files, which come next in the build order, `observe` mode, which
|
||||
comes a little later, and the metrics endpoint and the header size and the idle
|
||||
time as settings, which come last in it. `smallwebwaf` passes each request to
|
||||
the app and the app's answer back, unchanged, within its timeouts and size
|
||||
limits, works out each client's address, bans a client that sends too many
|
||||
requests, refuses a client that comes from a country you refuse or from a
|
||||
network you refuse, lets the networks you choose through, keeps its bans, each
|
||||
client's counters and history, and GeoJS's answers in JSON files across
|
||||
restarts, writes a JSON log line for every request, serves Prometheus metrics to
|
||||
a scraper that holds the metrics token, and in `observe` mode passes on the
|
||||
requests it would refuse, logging what it would have done with them. It comes as
|
||||
the image the app's own image is built on. The rest of the design comes after
|
||||
that, in the order of the build order in [`SPEC.md`](SPEC.md). The survey of
|
||||
existing tools that led to the design is in [`EVALUATION.md`](EVALUATION.md).
|
||||
JSON state files with your edits taken in while it runs, which come next in the
|
||||
build order, `observe` mode, which comes a little later, and the metrics
|
||||
endpoint and the header size and the idle time as settings, which come last in
|
||||
it. `smallwebwaf` passes each request to the app and the app's answer back,
|
||||
unchanged, within its timeouts and size limits, works out each client's address,
|
||||
bans a client that sends too many requests, refuses a client that comes from a
|
||||
country you refuse or from a network you refuse, lets the networks you choose
|
||||
through, keeps its bans, each client's counters and history, and GeoJS's answers
|
||||
in JSON files across restarts, takes in your edits of those files while it runs,
|
||||
writes a JSON log line for every request, serves Prometheus metrics to a scraper
|
||||
that holds the metrics token, and in `observe` mode passes on the requests it
|
||||
would refuse, logging what it would have done with them. It comes as the image
|
||||
the app's own image is built on. The rest of the design comes after that, in the
|
||||
order of the build order in [`SPEC.md`](SPEC.md). The survey of existing tools
|
||||
that led to the design is in [`EVALUATION.md`](EVALUATION.md).
|
||||
|
||||
## Getting started
|
||||
|
||||
@@ -100,9 +101,9 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set.
|
||||
seen, how many of them the ban has refused, and how many bans the netblock had
|
||||
before. At most `SWWAF_MAX_BANS` bans are kept, past, active and permanent;
|
||||
past that, the earliest ban of the netblock that has gone longest without a
|
||||
request is dropped first. `bans.json` shows the bans and their notes, and a
|
||||
restart lifts none (see "State files" below); lifting a ban by editing it
|
||||
comes with https://git.eeqj.de/sneak/smallwebwaf/issues/68.
|
||||
request is dropped first. `bans.json` shows the bans and their notes, a
|
||||
restart lifts none, and you add or lift a ban by editing it (see "State files"
|
||||
below).
|
||||
- Refuses a request from a country you refuse with `SWWAF_BAN_RESPONSE`, as soon
|
||||
as the client's country is known and before its body is read; such a request
|
||||
is not counted for the rate limits. While one of the country lists below is
|
||||
@@ -335,9 +336,42 @@ without a field it needs, named with the entry's place in the file: a ban's
|
||||
`netblock`, `start` or `expires`, which is `null` for a permanent ban; a
|
||||
client's `client`, or the `start` of a window in which it has requests; an
|
||||
answer's `client`, `country`, which is `""` for a client GeoJS cannot place, or
|
||||
`answered`. An edit made while `smallwebwaf` runs is overwritten by its next
|
||||
write: taking it in comes with https://git.eeqj.de/sneak/smallwebwaf/issues/68.
|
||||
The AS number and AS name come with their lookup.
|
||||
`answered`. The AS number and AS name come with their lookup.
|
||||
|
||||
While it runs, `smallwebwaf` watches `SWWAF_STATE_DIR` and takes in your edit of
|
||||
a state file as soon as you save it: what the file then holds replaces what
|
||||
`smallwebwaf` held for it, as if read at start. It tells its own writes from
|
||||
yours by comparing the file with what it last read or wrote, and before it
|
||||
writes a file it takes in any edit made since, so your edit is not overwritten;
|
||||
a change `smallwebwaf` made after you opened the file, such as a new ban, is
|
||||
lost when you save over it. An edit that would stop the start, because it does
|
||||
not parse, has another `version` or leaves out a field an entry needs, does not
|
||||
stop the running `smallwebwaf`: it keeps what it holds, and at the file's next
|
||||
write renames your file to `<name>.bad`, such as `bans.json.bad`, writes the
|
||||
file again from memory, and logs the file and where the error is. It waits for
|
||||
that write because an editor's file can be read before the editor has finished
|
||||
writing it. Mend the `.bad` file and move it back. A file you remove is written
|
||||
again at its next write.
|
||||
|
||||
To ban a netblock, add an entry to `bans.json` with its `netblock`, its `start`
|
||||
and its `expires`, `null` for a ban that never ends; its `notes` may be left
|
||||
out. This `bans.json` bans `203.0.113.0/24` for good:
|
||||
|
||||
```json
|
||||
{
|
||||
"version": 1,
|
||||
"bans": [
|
||||
{
|
||||
"netblock": "203.0.113.0/24",
|
||||
"start": "2026-10-06T12:00:00Z",
|
||||
"expires": null
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
To lift a ban, delete its entry. `smallwebwaf` then forgets the ban, so it does
|
||||
not make the netblock's next ban longer.
|
||||
|
||||
## Metrics
|
||||
|
||||
@@ -376,7 +410,10 @@ other request. No metric carries a client's address.
|
||||
- `smallwebwaf_state_file_writes_total`,
|
||||
`smallwebwaf_state_file_write_failures_total`,
|
||||
`smallwebwaf_state_file_last_write_timestamp_seconds` and
|
||||
`smallwebwaf_state_file_size_bytes`, by `file`.
|
||||
`smallwebwaf_state_file_size_bytes`, by `file`; and, by `file` too,
|
||||
`smallwebwaf_state_file_edits_taken_in_total`: your edits taken in, and
|
||||
`smallwebwaf_state_file_edits_set_aside_total`: those renamed to `<name>.bad`
|
||||
because they would stop the start.
|
||||
- Go's own `go_` metrics and the process's `process_` metrics.
|
||||
|
||||
The requests Go's HTTP server ends before `smallwebwaf` sees them (see "Request
|
||||
@@ -484,9 +521,9 @@ goes through the candidates one by one.
|
||||
readable JSON files, written regularly and at every stop, so a restart loses
|
||||
nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks
|
||||
up the change. Nothing is read from disk while serving a request. The files
|
||||
for the bans, the clients and the GeoJS answers are built (see "State files"
|
||||
above); the others come with their features, and taking in an edit while
|
||||
running comes with https://git.eeqj.de/sneak/smallwebwaf/issues/68.
|
||||
for the bans, the clients and the GeoJS answers are built, with an edit taken
|
||||
in while running (see "State files" above); the others come with their
|
||||
features.
|
||||
- Health checks, the metrics, and listing, adding and lifting bans or asking why
|
||||
a given address was refused, all on the one port every request uses: under
|
||||
`/_smallwebwaf/` on the app's own address, through traefik like any other
|
||||
@@ -678,8 +715,8 @@ addresses are never sent to GeoJS.
|
||||
answers.
|
||||
- `internal/ratelimit`: the table of clients: counts each client's requests,
|
||||
tells when one takes it over a rate limit, and keeps each client's history.
|
||||
- `internal/state`: reads the state files at start, and writes them when they
|
||||
are due and at the stop.
|
||||
- `internal/state`: reads the state files at start, takes in an admin's edit of
|
||||
one while running, and writes them when they are due and at the stop.
|
||||
- `internal/requestlog`: the lines on stdout: the request log line and the
|
||||
process's own messages.
|
||||
- `Dockerfile`: the lint and test phases, then the image, whose last stage
|
||||
@@ -691,8 +728,9 @@ addresses are never sent to GeoJS.
|
||||
Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the
|
||||
table of clients to 20,000, the GeoJS answers to 100,000 and the banned
|
||||
netblocks to `SWWAF_MAX_BANS`, dropping the least recently seen, and
|
||||
`github.com/prometheus/client_golang` keeps the metrics and serves them. The
|
||||
country codes are the list in `internal/config/config.go`.
|
||||
`github.com/prometheus/client_golang` keeps the metrics and serves them, and
|
||||
`github.com/fsnotify/fsnotify` tells `smallwebwaf` when a state file is saved.
|
||||
The country codes are the list in `internal/config/config.go`.
|
||||
|
||||
## Entrypoints
|
||||
|
||||
@@ -735,10 +773,9 @@ so that they run in minimal containers.
|
||||
|
||||
## TODO
|
||||
|
||||
- The rest of milestone 3: taking in an admin's edits to the state files
|
||||
(https://git.eeqj.de/sneak/smallwebwaf/issues/68), exemptions and the rest of
|
||||
the request log's fields; then the rest of the design, in the order of the
|
||||
build order in [`SPEC.md`](SPEC.md).
|
||||
- The rest of milestone 3: exemptions and the rest of the request log's fields;
|
||||
then the rest of the design, in the order of the build order in
|
||||
[`SPEC.md`](SPEC.md).
|
||||
|
||||
## Documents
|
||||
|
||||
|
||||
Reference in New Issue
Block a user