SPEC: the one place for behaviour and reasons; docs/ as lookup lists only
check / check (push) Waiting to run
check / check (push) Waiting to run
docs/core-rule-set.md, docs/lookup.md, docs/reputation.md and docs/bans.md restated the spec and are folded back into it and removed. The remaining docs files hold only lists: settings with their defaults and value forms, the request log's fields, the metrics, the endpoints, the state files and their contents, the rule file format and default file, the alert events and messages, the image's contents and pins, and the gitea notes. Every behaviour each of them held moves to the spec, said once. The settings groups have the same names in both files. Stale comments in internal/state and script/example-app point at the right places. Model: fable-5-1
This commit is contained in:
+90
-92
@@ -1,102 +1,100 @@
|
||||
# State files
|
||||
|
||||
`smallwebwaf` keeps its state in memory, and a copy of it in five JSON files in
|
||||
`SWWAF_STATE_DIR`. The files are meant for people as well: you can read or edit
|
||||
them at any time, and the running `smallwebwaf` takes the edit in. The
|
||||
requirements they meet are under "Persistent state" in [`SPEC.md`](../SPEC.md).
|
||||
|
||||
## The files
|
||||
|
||||
The five JSON files in `SWWAF_STATE_DIR`, and what each holds. "Persistent
|
||||
state" in [`SPEC.md`](../SPEC.md) says how they are written, read and edited.
|
||||
Each has a top-level `version` of 1.
|
||||
|
||||
- `bans.json`: active and past bans, up to `SWWAF_MAX_BANS` that `smallwebwaf`
|
||||
made and every ban you made, indented to be read. Per entry:
|
||||
- the netblock, start and expiry (`null` for permanent);
|
||||
- the cause: `attack`, `limit`, `admin` or `crowdsec`;
|
||||
- a short reason;
|
||||
- when it was lifted, if you lifted it;
|
||||
- the notes (see [`bans.md`](bans.md)).
|
||||
- `clients.json`: per client, its minute, hour and day counters and its history
|
||||
since it was first seen, one client per line, so `grep` shows everything about
|
||||
one. The history holds:
|
||||
- the first and last time seen;
|
||||
- the AS number, AS name and country last looked up, and when;
|
||||
- total requests, and how many were forwarded and how many refused;
|
||||
- bytes in each direction;
|
||||
- responses by status class;
|
||||
- offences by kind: `limit`, `attack`, `rule_blocked`, `waf_blocked` and
|
||||
`token_refused`.
|
||||
## `bans.json`
|
||||
|
||||
Clients that were never banned are kept too, so every client's history
|
||||
survives a restart.
|
||||
Active and past bans: every ban an admin made, and up to `SWWAF_MAX_BANS` of
|
||||
those `smallwebwaf` made. Per entry:
|
||||
|
||||
- `lookups.json`: GeoJS answers, one per line: the AS number, AS name and
|
||||
country, when GeoJS was asked and when the answer was last used.
|
||||
- `reputation.json`:
|
||||
- the last good copy of each list fetched from a URL, with when it was last
|
||||
tried;
|
||||
- each DNS blocklist verdict still in use;
|
||||
- AbuseIPDB's scores, and the checks spent today.
|
||||
- `alerts.json`:
|
||||
- the anomaly counters;
|
||||
- the alert cooldowns: for each event and netblock, source, file or thing
|
||||
counted, when it was last sent and the count of repeats held back since;
|
||||
- the alerts sent and held back in the current hour;
|
||||
- the alerts waiting for each destination.
|
||||
- `netblock`, `start` and `expires` (`null` for a permanent ban);
|
||||
- `cause`: `attack`, `limit`, `admin` or `crowdsec`;
|
||||
- `reason`: a short reason;
|
||||
- `lifted`: when an admin lifted it, if one did;
|
||||
- `notes`: the notes below.
|
||||
|
||||
Times are RFC 3339 UTC. Durations and sizes are plain numbers, with the unit in
|
||||
the field name. `bans.json`, `reputation.json` and `alerts.json` are indented;
|
||||
`clients.json` and `lookups.json` put each entry on one line.
|
||||
To ban a netblock, add an entry with its `netblock`, `start` and `expires`.
|
||||
`reason`, `notes` and `cause` may be left out; the cause is then `admin`:
|
||||
|
||||
```json
|
||||
{
|
||||
"version": 1,
|
||||
"bans": [
|
||||
{
|
||||
"netblock": "203.0.113.0/24",
|
||||
"start": "2026-10-06T12:00:00Z",
|
||||
"expires": null,
|
||||
"reason": "probes for logins"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
To lift a ban, add `"lifted"` with the time you lift it, such as
|
||||
`"lifted": "2026-10-06T13:00:00Z"`. To keep a ban `smallwebwaf` made, set its
|
||||
`cause` to `admin`. To forget a ban, delete its entry.
|
||||
|
||||
The notes:
|
||||
|
||||
- `asn`, `as_name` and `country`: the client's, added when the lookup answers
|
||||
(empty when lookups are off);
|
||||
- what was broken: the rule id and target that matched, or the trap path, or the
|
||||
limit, its window and the count reached;
|
||||
- the client's percentage of the limit, with the setting that gave it;
|
||||
- the reputation sources the client is on;
|
||||
- `request`: the request that caused the ban: time, method, host, path with its
|
||||
query string, status and user agent, each text cut to 256 bytes;
|
||||
- `requests` and `refused`: the netblock's requests since it was first seen, and
|
||||
how many this ban has refused so far, kept up to date while the ban lasts;
|
||||
- `earlier_bans`: how many earlier bans the netblock had, by cause.
|
||||
|
||||
## `clients.json`
|
||||
|
||||
Per client, one per line: its minute, hour and day counters, and its history
|
||||
since it was first seen:
|
||||
|
||||
- the first and last time seen;
|
||||
- the AS number, AS name and country last looked up, and when;
|
||||
- total requests, and how many were forwarded and how many refused;
|
||||
- bytes in each direction;
|
||||
- responses by status class;
|
||||
- offences by kind: `limit`, `attack`, `rule_blocked`, `waf_blocked` and
|
||||
`token_refused`.
|
||||
|
||||
## `lookups.json`
|
||||
|
||||
GeoJS answers, one per line:
|
||||
|
||||
- `client`;
|
||||
- `asn`, `as_name` and `country`;
|
||||
- `answered`: when GeoJS answered;
|
||||
- `used`: when the answer was last used.
|
||||
|
||||
## `reputation.json`
|
||||
|
||||
- the last good copy of each list fetched from a URL, with when it was last
|
||||
tried;
|
||||
- each DNS blocklist verdict still in use;
|
||||
- AbuseIPDB's scores, and the checks spent today.
|
||||
|
||||
## `alerts.json`
|
||||
|
||||
- the anomaly counters;
|
||||
- the alert cooldowns: for each event and netblock, source, file or anomaly
|
||||
scope, when it was last sent and the count of repeats held back since;
|
||||
- the alerts sent and held back in the current hour;
|
||||
- the alerts waiting for each destination.
|
||||
|
||||
## Sizes
|
||||
|
||||
`clients.json` takes about 1 KiB per client, about 20 MiB at the default
|
||||
`SWWAF_MAX_TRACKED_CLIENTS`. Written every 15 minutes, that is under 2 GiB of
|
||||
disk writes a day. `bans.json` takes about 1.2 KiB per ban, at most about 2.5
|
||||
KiB, so about 6 MiB at the default `SWWAF_MAX_BANS`. With a full file, a hundred
|
||||
new bans a day come to about 600 MiB of writes. `lookups.json` takes about 150
|
||||
bytes per answer, under 1.5 GiB of writes a day when full. The other two files
|
||||
are usually a few MiB or less.
|
||||
|
||||
## Writing
|
||||
|
||||
`bans.json` is written `SWWAF_STATE_WRITE_DELAY` after a ban is made, lifted or
|
||||
made permanent. Every file is written every `SWWAF_STATE_COUNTER_INTERVAL`, and
|
||||
when `smallwebwaf` stops. Each write goes to a temporary file that then replaces
|
||||
the file, so a crash leaves the old file or the new one, whole. Before writing a
|
||||
file, `smallwebwaf` takes in any edit you made to it, so your edit is not
|
||||
overwritten. A write that fails is logged, raised as a `file_error` alert, and
|
||||
tried again at the next write. A hard kill loses what changed since the last
|
||||
write:
|
||||
|
||||
- ban changes from the last `SWWAF_STATE_WRITE_DELAY` (10 seconds);
|
||||
- everything else from the last `SWWAF_STATE_COUNTER_INTERVAL` (15 minutes):
|
||||
- counts and history, and the counts in ban notes;
|
||||
- GeoJS answers;
|
||||
- reputation verdicts and the AbuseIPDB count;
|
||||
- anomaly counters and alert cooldowns.
|
||||
|
||||
## Reading at start
|
||||
|
||||
The files are read back. Each client keeps its counts, so a restart gives no one
|
||||
a fresh allowance. Each ban keeps refusing until it ends. Counter buckets and
|
||||
cache entries whose time has passed are discarded. A missing file is empty
|
||||
state. These stop the start, with a message naming the file and the place:
|
||||
|
||||
- a file that does not parse, or has another `version`;
|
||||
- an entry without a field it needs;
|
||||
- a ban whose `cause` is not `limit`, `attack`, `admin` or `crowdsec`;
|
||||
- a ban whose `expires` is not after its `start`.
|
||||
|
||||
## Editing while it runs
|
||||
|
||||
`smallwebwaf` watches the directory and takes in your edit of a file as soon as
|
||||
you save it. What the file then holds replaces what `smallwebwaf` held for it.
|
||||
It tells its own writes from yours by comparing the file with what it last read
|
||||
or wrote. A change it made after you opened the file, such as a new ban, is lost
|
||||
when you save over it; most editors warn when a file changed on disk while it
|
||||
was open. An edit with an error that would stop the start does not stop the
|
||||
running `smallwebwaf`. It keeps what it holds. At the file's next write it
|
||||
renames your file to `<name>.bad`, writes the file again from memory, and logs
|
||||
and alerts the file and the place of the error. Mend the `.bad` file and move it
|
||||
back. A file you remove is written again at its next write.
|
||||
- `clients.json`: about 1 KiB per client, about 20 MiB at the default
|
||||
`SWWAF_MAX_TRACKED_CLIENTS`. Written every 15 minutes, that is under 2 GiB of
|
||||
disk writes a day.
|
||||
- `bans.json`: about 1.2 KiB per ban, at most about 2.5 KiB, so about 6 MiB at
|
||||
the default `SWWAF_MAX_BANS`. With a full file, a hundred new bans a day come
|
||||
to about 600 MiB of writes.
|
||||
- `lookups.json`: about 150 bytes per answer, under 1.5 GiB of writes a day when
|
||||
full.
|
||||
- `reputation.json` and `alerts.json`: usually a few MiB or less.
|
||||
|
||||
Reference in New Issue
Block a user