sneak
d712bb1496
check / check (push) Waiting to run
docs/settings.md is now the one settings reference; the spec keeps the conventions, the off list and each group's requirements and reasons, and links to it for the per-setting entries. The endpoint list, the ban notes, the state file contents, the rule file format and the alert object likewise live in docs/ and are linked from the spec. The package list from the README's Design section joins "Architecture". The nine docs files are byte-identical to those on the README branch. Package comments that named "Alert webhook schema", "Request log" and "Metrics endpoint" for their field lists now name the docs files. Model: fable-5-1
103 lines
4.9 KiB
Markdown
103 lines
4.9 KiB
Markdown
# State files
|
|
|
|
`smallwebwaf` keeps its state in memory, and a copy of it in five JSON files in
|
|
`SWWAF_STATE_DIR`. The files are meant for people as well: you can read or edit
|
|
them at any time, and the running `smallwebwaf` takes the edit in. The
|
|
requirements they meet are under "Persistent state" in [`SPEC.md`](../SPEC.md).
|
|
|
|
## The files
|
|
|
|
Each has a top-level `version` of 1.
|
|
|
|
- `bans.json`: active and past bans, up to `SWWAF_MAX_BANS` that `smallwebwaf`
|
|
made and every ban you made, indented to be read. Per entry:
|
|
- the netblock, start and expiry (`null` for permanent);
|
|
- the cause: `attack`, `limit`, `admin` or `crowdsec`;
|
|
- a short reason;
|
|
- when it was lifted, if you lifted it;
|
|
- the notes (see [`bans.md`](bans.md)).
|
|
- `clients.json`: per client, its minute, hour and day counters and its history
|
|
since it was first seen, one client per line, so `grep` shows everything about
|
|
one. The history holds:
|
|
- the first and last time seen;
|
|
- the AS number, AS name and country last looked up, and when;
|
|
- total requests, and how many were forwarded and how many refused;
|
|
- bytes in each direction;
|
|
- responses by status class;
|
|
- offences by kind: `limit`, `attack`, `rule_blocked`, `waf_blocked` and
|
|
`token_refused`.
|
|
|
|
Clients that were never banned are kept too, so every client's history
|
|
survives a restart.
|
|
|
|
- `lookups.json`: GeoJS answers, one per line: the AS number, AS name and
|
|
country, when GeoJS was asked and when the answer was last used.
|
|
- `reputation.json`:
|
|
- the last good copy of each list fetched from a URL, with when it was last
|
|
tried;
|
|
- each DNS blocklist verdict still in use;
|
|
- AbuseIPDB's scores, and the checks spent today.
|
|
- `alerts.json`:
|
|
- the anomaly counters;
|
|
- the alert cooldowns: for each event and netblock, source, file or thing
|
|
counted, when it was last sent and the count of repeats held back since;
|
|
- the alerts sent and held back in the current hour;
|
|
- the alerts waiting for each destination.
|
|
|
|
Times are RFC 3339 UTC. Durations and sizes are plain numbers, with the unit in
|
|
the field name. `bans.json`, `reputation.json` and `alerts.json` are indented;
|
|
`clients.json` and `lookups.json` put each entry on one line.
|
|
|
|
## Sizes
|
|
|
|
`clients.json` takes about 1 KiB per client, about 20 MiB at the default
|
|
`SWWAF_MAX_TRACKED_CLIENTS`. Written every 15 minutes, that is under 2 GiB of
|
|
disk writes a day. `bans.json` takes about 1.2 KiB per ban, at most about 2.5
|
|
KiB, so about 6 MiB at the default `SWWAF_MAX_BANS`. With a full file, a hundred
|
|
new bans a day come to about 600 MiB of writes. `lookups.json` takes about 150
|
|
bytes per answer, under 1.5 GiB of writes a day when full. The other two files
|
|
are usually a few MiB or less.
|
|
|
|
## Writing
|
|
|
|
`bans.json` is written `SWWAF_STATE_WRITE_DELAY` after a ban is made, lifted or
|
|
made permanent. Every file is written every `SWWAF_STATE_COUNTER_INTERVAL`, and
|
|
when `smallwebwaf` stops. Each write goes to a temporary file that then replaces
|
|
the file, so a crash leaves the old file or the new one, whole. Before writing a
|
|
file, `smallwebwaf` takes in any edit you made to it, so your edit is not
|
|
overwritten. A write that fails is logged, raised as a `file_error` alert, and
|
|
tried again at the next write. A hard kill loses what changed since the last
|
|
write:
|
|
|
|
- ban changes from the last `SWWAF_STATE_WRITE_DELAY` (10 seconds);
|
|
- everything else from the last `SWWAF_STATE_COUNTER_INTERVAL` (15 minutes):
|
|
- counts and history, and the counts in ban notes;
|
|
- GeoJS answers;
|
|
- reputation verdicts and the AbuseIPDB count;
|
|
- anomaly counters and alert cooldowns.
|
|
|
|
## Reading at start
|
|
|
|
The files are read back. Each client keeps its counts, so a restart gives no one
|
|
a fresh allowance. Each ban keeps refusing until it ends. Counter buckets and
|
|
cache entries whose time has passed are discarded. A missing file is empty
|
|
state. These stop the start, with a message naming the file and the place:
|
|
|
|
- a file that does not parse, or has another `version`;
|
|
- an entry without a field it needs;
|
|
- a ban whose `cause` is not `limit`, `attack`, `admin` or `crowdsec`;
|
|
- a ban whose `expires` is not after its `start`.
|
|
|
|
## Editing while it runs
|
|
|
|
`smallwebwaf` watches the directory and takes in your edit of a file as soon as
|
|
you save it. What the file then holds replaces what `smallwebwaf` held for it.
|
|
It tells its own writes from yours by comparing the file with what it last read
|
|
or wrote. A change it made after you opened the file, such as a new ban, is lost
|
|
when you save over it; most editors warn when a file changed on disk while it
|
|
was open. An edit with an error that would stop the start does not stop the
|
|
running `smallwebwaf`. It keeps what it holds. At the file's next write it
|
|
renames your file to `<name>.bad`, writes the file again from memory, and logs
|
|
and alerts the file and the place of the error. Mend the `.bad` file and move it
|
|
back. A file you remove is written again at its next write.
|