Take in an admin's edits of the state files while running (closes #68)
check / check (push) Successful in 2m59s
check / check (push) Successful in 2m59s
smallwebwaf watches SWWAF_STATE_DIR with fsnotify and takes in a saved edit of a state file in place of what it held. It tells its own writes from an admin's by the SHA-256 of what it last read or wrote; each write first takes in an edit made since. An edit that does not parse is renamed to <name>.bad at the file's next write. Every ban on a netblock is checked, and the next ban is worked out from the one that ended last. Two metrics count the edits taken in and set aside. README.md says how to add and lift a ban. Judgement call: a broken edit is set aside at the next write, since an editor's file can be read half written. Model: opus-5-5
This commit is contained in:
@@ -15,14 +15,15 @@ Status: the first two milestones are built
|
||||
(https://git.eeqj.de/sneak/smallwebwaf/issues/13 and
|
||||
https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are five parts of
|
||||
milestone 3: the static lists, the bans that broken rate limits lead to and the
|
||||
JSON state files, which come next in the build order, and the metrics endpoint
|
||||
and the header size and the idle time as settings, which come last in it.
|
||||
`smallwebwaf` passes each request to the app and the app's answer back,
|
||||
unchanged, within its timeouts and size limits, works out each client's address,
|
||||
bans a client that sends too many requests, refuses a client that comes from a
|
||||
country you refuse or from a network you refuse, lets the networks you choose
|
||||
through, keeps its bans, each client's counters and history, and GeoJS's answers
|
||||
in JSON files across restarts, writes a JSON log line for every request, and
|
||||
JSON state files with your edits taken in while it runs, which come next in the
|
||||
build order, and the metrics endpoint and the header size and the idle time as
|
||||
settings, which come last in it. `smallwebwaf` passes each request to the app
|
||||
and the app's answer back, unchanged, within its timeouts and size limits, works
|
||||
out each client's address, bans a client that sends too many requests, refuses a
|
||||
client that comes from a country you refuse or from a network you refuse, lets
|
||||
the networks you choose through, keeps its bans, each client's counters and
|
||||
history, and GeoJS's answers in JSON files across restarts, takes in your edits
|
||||
of those files while it runs, writes a JSON log line for every request, and
|
||||
serves Prometheus metrics to a scraper that holds the metrics token. It comes as
|
||||
the image the app's own image is built on. The rest of the design comes after
|
||||
that, in the order of the build order in [`SPEC.md`](SPEC.md). The survey of
|
||||
@@ -98,9 +99,9 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set.
|
||||
seen, how many of them the ban has refused, and how many bans the netblock had
|
||||
before. At most `SWWAF_MAX_BANS` bans are kept, past, active and permanent;
|
||||
past that, the earliest ban of the netblock that has gone longest without a
|
||||
request is dropped first. `bans.json` shows the bans and their notes, and a
|
||||
restart lifts none (see "State files" below); lifting a ban by editing it
|
||||
comes with https://git.eeqj.de/sneak/smallwebwaf/issues/68.
|
||||
request is dropped first. `bans.json` shows the bans and their notes, a
|
||||
restart lifts none, and you add or lift a ban by editing it (see "State files"
|
||||
below).
|
||||
- Refuses a request from a country you refuse with `SWWAF_BAN_RESPONSE`, as soon
|
||||
as the client's country is known and before its body is read; such a request
|
||||
is not counted for the rate limits. While one of the country lists below is
|
||||
@@ -311,9 +312,42 @@ without a field it needs, named with the entry's place in the file: a ban's
|
||||
`netblock`, `start` or `expires`, which is `null` for a permanent ban; a
|
||||
client's `client`, or the `start` of a window in which it has requests; an
|
||||
answer's `client`, `country`, which is `""` for a client GeoJS cannot place, or
|
||||
`answered`. An edit made while `smallwebwaf` runs is overwritten by its next
|
||||
write: taking it in comes with https://git.eeqj.de/sneak/smallwebwaf/issues/68.
|
||||
The AS number and AS name come with their lookup.
|
||||
`answered`. The AS number and AS name come with their lookup.
|
||||
|
||||
While it runs, `smallwebwaf` watches `SWWAF_STATE_DIR` and takes in your edit of
|
||||
a state file as soon as you save it: what the file then holds replaces what
|
||||
`smallwebwaf` held for it, as if read at start. It tells its own writes from
|
||||
yours by comparing the file with what it last read or wrote, and before it
|
||||
writes a file it takes in any edit made since, so your edit is not overwritten;
|
||||
a change `smallwebwaf` made after you opened the file, such as a new ban, is
|
||||
lost when you save over it. An edit that would stop the start, because it does
|
||||
not parse, has another `version` or leaves out a field an entry needs, does not
|
||||
stop the running `smallwebwaf`: it keeps what it holds, and at the file's next
|
||||
write renames your file to `<name>.bad`, such as `bans.json.bad`, writes the
|
||||
file again from memory, and logs the file and where the error is. It waits for
|
||||
that write because an editor's file can be read before the editor has finished
|
||||
writing it. Mend the `.bad` file and move it back. A file you remove is written
|
||||
again at its next write.
|
||||
|
||||
To ban a netblock, add an entry to `bans.json` with its `netblock`, its `start`
|
||||
and its `expires`, `null` for a ban that never ends; its `notes` may be left
|
||||
out. This `bans.json` bans `203.0.113.0/24` for good:
|
||||
|
||||
```json
|
||||
{
|
||||
"version": 1,
|
||||
"bans": [
|
||||
{
|
||||
"netblock": "203.0.113.0/24",
|
||||
"start": "2026-10-06T12:00:00Z",
|
||||
"expires": null
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
To lift a ban, delete its entry. `smallwebwaf` then forgets the ban, so it does
|
||||
not make the netblock's next ban longer.
|
||||
|
||||
## Metrics
|
||||
|
||||
@@ -352,7 +386,10 @@ other request. No metric carries a client's address.
|
||||
- `smallwebwaf_state_file_writes_total`,
|
||||
`smallwebwaf_state_file_write_failures_total`,
|
||||
`smallwebwaf_state_file_last_write_timestamp_seconds` and
|
||||
`smallwebwaf_state_file_size_bytes`, by `file`.
|
||||
`smallwebwaf_state_file_size_bytes`, by `file`; and, by `file` too,
|
||||
`smallwebwaf_state_file_edits_taken_in_total`: your edits taken in, and
|
||||
`smallwebwaf_state_file_edits_set_aside_total`: those renamed to `<name>.bad`
|
||||
because they would stop the start.
|
||||
- Go's own `go_` metrics and the process's `process_` metrics.
|
||||
|
||||
The requests Go's HTTP server ends before `smallwebwaf` sees them (see "Request
|
||||
@@ -460,9 +497,9 @@ goes through the candidates one by one.
|
||||
readable JSON files, written regularly and at every stop, so a restart loses
|
||||
nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks
|
||||
up the change. Nothing is read from disk while serving a request. The files
|
||||
for the bans, the clients and the GeoJS answers are built (see "State files"
|
||||
above); the others come with their features, and taking in an edit while
|
||||
running comes with https://git.eeqj.de/sneak/smallwebwaf/issues/68.
|
||||
for the bans, the clients and the GeoJS answers are built, with an edit taken
|
||||
in while running (see "State files" above); the others come with their
|
||||
features.
|
||||
- Health checks, the metrics, and listing, adding and lifting bans or asking why
|
||||
a given address was refused, all on the one port every request uses: under
|
||||
`/_smallwebwaf/` on the app's own address, through traefik like any other
|
||||
@@ -653,8 +690,8 @@ addresses are never sent to GeoJS.
|
||||
answers.
|
||||
- `internal/ratelimit`: the table of clients: counts each client's requests,
|
||||
tells when one takes it over a rate limit, and keeps each client's history.
|
||||
- `internal/state`: reads the state files at start, and writes them when they
|
||||
are due and at the stop.
|
||||
- `internal/state`: reads the state files at start, takes in an admin's edit of
|
||||
one while running, and writes them when they are due and at the stop.
|
||||
- `internal/requestlog`: the lines on stdout: the request log line and the
|
||||
process's own messages.
|
||||
- `Dockerfile`: the lint and test phases, then the image, whose last stage
|
||||
@@ -666,8 +703,9 @@ addresses are never sent to GeoJS.
|
||||
Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the
|
||||
table of clients to 20,000, the GeoJS answers to 100,000 and the banned
|
||||
netblocks to `SWWAF_MAX_BANS`, dropping the least recently seen, and
|
||||
`github.com/prometheus/client_golang` keeps the metrics and serves them. The
|
||||
country codes are the list in `internal/config/config.go`.
|
||||
`github.com/prometheus/client_golang` keeps the metrics and serves them, and
|
||||
`github.com/fsnotify/fsnotify` tells `smallwebwaf` when a state file is saved.
|
||||
The country codes are the list in `internal/config/config.go`.
|
||||
|
||||
## Entrypoints
|
||||
|
||||
@@ -710,10 +748,9 @@ so that they run in minimal containers.
|
||||
|
||||
## TODO
|
||||
|
||||
- The rest of milestone 3, from taking in an admin's edits to the state files
|
||||
(https://git.eeqj.de/sneak/smallwebwaf/issues/68) up to the rest of the
|
||||
request log's fields, and the rest of the design, in the order of the build
|
||||
order in [`SPEC.md`](SPEC.md).
|
||||
- The rest of milestone 3, from exemptions up to the rest of the request log's
|
||||
fields, and the rest of the design, in the order of the build order in
|
||||
[`SPEC.md`](SPEC.md).
|
||||
|
||||
## Documents
|
||||
|
||||
|
||||
Reference in New Issue
Block a user