Take in an admin's edits of the state files while running (closes #68)
check / check (push) Successful in 3m28s
check / check (push) Successful in 3m28s
smallwebwaf watches SWWAF_STATE_DIR with fsnotify and takes in an edit of bans.json, clients.json or lookups.json as soon as it is saved, in place of what it held. It tells its own writes from an admin's by the SHA-256 of what it last read or wrote, and each write takes in an edit made since first. An edit that does not parse is renamed to <name>.bad at the file's next write, which writes the file again from memory and logs the file and where the error is. README.md says how to add and lift a ban. Judgement call: a broken edit is set aside at the file's next write, not when seen, since an editor's file can be read half written. Judgement call: a state file that cannot be read is not written over. Model: opus-5-5
This commit is contained in:
@@ -15,17 +15,18 @@ Status: the first two milestones are built
|
||||
(https://git.eeqj.de/sneak/smallwebwaf/issues/13 and
|
||||
https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are four parts of
|
||||
milestone 3: the static lists, the bans that broken rate limits lead to and the
|
||||
JSON state files, which come next in the build order, and the header size and
|
||||
the idle time as settings, which come last in it. `smallwebwaf` passes each
|
||||
request to the app and the app's answer back, unchanged, within its timeouts and
|
||||
size limits, works out each client's address, bans a client that sends too many
|
||||
requests, refuses a client that comes from a country you refuse or from a
|
||||
network you refuse, lets the networks you choose through, keeps its bans, each
|
||||
client's counters and history, and GeoJS's answers in JSON files across
|
||||
restarts, and writes a JSON log line for every request. It comes as the image
|
||||
the app's own image is built on. The rest of the design comes after that, in the
|
||||
order of the build order in [`SPEC.md`](SPEC.md). The survey of existing tools
|
||||
that led to the design is in [`EVALUATION.md`](EVALUATION.md).
|
||||
JSON state files with your edits taken in while it runs, which come next in the
|
||||
build order, and the header size and the idle time as settings, which come last
|
||||
in it. `smallwebwaf` passes each request to the app and the app's answer back,
|
||||
unchanged, within its timeouts and size limits, works out each client's address,
|
||||
bans a client that sends too many requests, refuses a client that comes from a
|
||||
country you refuse or from a network you refuse, lets the networks you choose
|
||||
through, keeps its bans, each client's counters and history, and GeoJS's answers
|
||||
in JSON files across restarts, takes in your edits of those files while it runs,
|
||||
and writes a JSON log line for every request. It comes as the image the app's
|
||||
own image is built on. The rest of the design comes after that, in the order of
|
||||
the build order in [`SPEC.md`](SPEC.md). The survey of existing tools that led
|
||||
to the design is in [`EVALUATION.md`](EVALUATION.md).
|
||||
|
||||
## Getting started
|
||||
|
||||
@@ -97,9 +98,9 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set.
|
||||
seen, how many of them the ban has refused, and how many bans the netblock had
|
||||
before. At most `SWWAF_MAX_BANS` bans are kept, past, active and permanent;
|
||||
past that, the earliest ban of the netblock that has gone longest without a
|
||||
request is dropped first. `bans.json` shows the bans and their notes, and a
|
||||
restart lifts none (see "State files" below); lifting a ban by editing it
|
||||
comes with https://git.eeqj.de/sneak/smallwebwaf/issues/68.
|
||||
request is dropped first. `bans.json` shows the bans and their notes, a
|
||||
restart lifts none, and you add or lift a ban by editing it (see "State files"
|
||||
below).
|
||||
- Refuses a request from a country you refuse with `SWWAF_BAN_RESPONSE`, as soon
|
||||
as the client's country is known and before its body is read; such a request
|
||||
is not counted for the rate limits. While one of the country lists below is
|
||||
@@ -295,9 +296,42 @@ without a field it needs, named with the entry's place in the file: a ban's
|
||||
`netblock`, `start` or `expires`, which is `null` for a permanent ban; a
|
||||
client's `client`, or the `start` of a window in which it has requests; an
|
||||
answer's `client`, `country`, which is `""` for a client GeoJS cannot place, or
|
||||
`answered`. An edit made while `smallwebwaf` runs is overwritten by its next
|
||||
write: taking it in comes with https://git.eeqj.de/sneak/smallwebwaf/issues/68.
|
||||
The AS number and AS name come with their lookup.
|
||||
`answered`. The AS number and AS name come with their lookup.
|
||||
|
||||
While it runs, `smallwebwaf` watches `SWWAF_STATE_DIR` and takes in your edit of
|
||||
a state file as soon as you save it: what the file then holds replaces what
|
||||
`smallwebwaf` held for it, as if read at start. It tells its own writes from
|
||||
yours by comparing the file with what it last read or wrote, and before it
|
||||
writes a file it takes in any edit made since, so your edit is not overwritten;
|
||||
a change `smallwebwaf` made after you opened the file, such as a new ban, is
|
||||
lost when you save over it. An edit that would stop the start, because it does
|
||||
not parse, has another `version` or leaves out a field an entry needs, does not
|
||||
stop the running `smallwebwaf`: it keeps what it holds, and at the file's next
|
||||
write renames your file to `<name>.bad`, such as `bans.json.bad`, writes the
|
||||
file again from memory, and logs the file and where the error is. It waits for
|
||||
that write because an editor's file can be read before the editor has finished
|
||||
writing it. Mend the `.bad` file and move it back. A file you remove is written
|
||||
again at its next write.
|
||||
|
||||
To ban a netblock, add an entry to `bans.json` with its `netblock`, its `start`
|
||||
and its `expires`, `null` for a ban that never ends; its `notes` may be left
|
||||
out. This `bans.json` bans `203.0.113.0/24` for good:
|
||||
|
||||
```json
|
||||
{
|
||||
"version": 1,
|
||||
"bans": [
|
||||
{
|
||||
"netblock": "203.0.113.0/24",
|
||||
"start": "2026-10-06T12:00:00Z",
|
||||
"expires": null
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
To lift a ban, delete its entry. `smallwebwaf` then forgets the ban, so it does
|
||||
not make the netblock's next ban longer.
|
||||
|
||||
## Why
|
||||
|
||||
@@ -400,9 +434,9 @@ goes through the candidates one by one.
|
||||
readable JSON files, written regularly and at every stop, so a restart loses
|
||||
nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks
|
||||
up the change. Nothing is read from disk while serving a request. The files
|
||||
for the bans, the clients and the GeoJS answers are built (see "State files"
|
||||
above); the others come with their features, and taking in an edit while
|
||||
running comes with https://git.eeqj.de/sneak/smallwebwaf/issues/68.
|
||||
for the bans, the clients and the GeoJS answers are built, with an edit taken
|
||||
in while running (see "State files" above); the others come with their
|
||||
features.
|
||||
- Health checks, the metrics, and listing, adding and lifting bans or asking why
|
||||
a given address was refused, all on the one port every request uses: under
|
||||
`/_smallwebwaf/` on the app's own address, through traefik like any other
|
||||
@@ -590,8 +624,8 @@ addresses are never sent to GeoJS.
|
||||
answers.
|
||||
- `internal/ratelimit`: the table of clients: counts each client's requests,
|
||||
tells when one takes it over a rate limit, and keeps each client's history.
|
||||
- `internal/state`: reads the state files at start, and writes them when they
|
||||
are due and at the stop.
|
||||
- `internal/state`: reads the state files at start, takes in an admin's edit of
|
||||
one while running, and writes them when they are due and at the stop.
|
||||
- `internal/requestlog`: the lines on stdout: the request log line and the
|
||||
process's own messages.
|
||||
- `Dockerfile`: the lint and test phases, then the image, whose last stage
|
||||
@@ -602,8 +636,9 @@ addresses are never sent to GeoJS.
|
||||
|
||||
Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the
|
||||
table of clients to 20,000, the GeoJS answers to 100,000 and the banned
|
||||
netblocks to `SWWAF_MAX_BANS`, dropping the least recently seen. The country
|
||||
codes are the list in `internal/config/config.go`.
|
||||
netblocks to `SWWAF_MAX_BANS`, dropping the least recently seen, and
|
||||
`github.com/fsnotify/fsnotify` tells `smallwebwaf` when a state file is saved.
|
||||
The country codes are the list in `internal/config/config.go`.
|
||||
|
||||
## Entrypoints
|
||||
|
||||
@@ -646,10 +681,8 @@ so that they run in minimal containers.
|
||||
|
||||
## TODO
|
||||
|
||||
- The rest of milestone 3, from taking in an admin's edits to the state files
|
||||
(https://git.eeqj.de/sneak/smallwebwaf/issues/68) up to the metrics endpoint,
|
||||
and the rest of the design, in the order of the build order in
|
||||
[`SPEC.md`](SPEC.md).
|
||||
- The rest of milestone 3, from exemptions up to the metrics endpoint, and the
|
||||
rest of the design, in the order of the build order in [`SPEC.md`](SPEC.md).
|
||||
|
||||
## Documents
|
||||
|
||||
|
||||
Reference in New Issue
Block a user