smallwebwaf/docs/state-files.md

State files

smallwebwaf keeps its state in memory, and a copy of it in five JSON files in SWWAF_STATE_DIR. The files are meant for people as well: you can read or edit them at any time, and the running smallwebwaf takes the edit in. The requirements they meet are under "Persistent state" in SPEC.md.

The files

Each has a top-level version of 1.

• bans.json: active and past bans, up to SWWAF_MAX_BANS that smallwebwaf made and every ban you made, indented to be read. Per entry:

  • the netblock, start and expiry (null for permanent);
  • the cause: attack, limit, admin or crowdsec;
  • a short reason;
  • when it was lifted, if you lifted it;
  • the notes (see bans.md).

• clients.json: per client, its minute, hour and day counters and its history since it was first seen, one client per line, so grep shows everything about one. The history holds:

  • the first and last time seen;
  • the AS number, AS name and country last looked up, and when;
  • total requests, and how many were forwarded and how many refused;
  • bytes in each direction;
  • responses by status class;
  • offences by kind: limit, attack, rule_blocked, waf_blocked and token_refused.

  Clients that were never banned are kept too, so every client's history survives a restart.

• lookups.json: GeoJS answers, one per line: the AS number, AS name and country, when GeoJS was asked and when the answer was last used.

• reputation.json:

  • the last good copy of each list fetched from a URL, with when it was last tried;
  • each DNS blocklist verdict still in use;
  • AbuseIPDB's scores, and the checks spent today.

• alerts.json:

  • the anomaly counters;
  • the alert cooldowns: for each event and netblock, source, file or thing counted, when it was last sent and the count of repeats held back since;
  • the alerts sent and held back in the current hour;
  • the alerts waiting for each destination.

Times are RFC 3339 UTC. Durations and sizes are plain numbers, with the unit in the field name. bans.json, reputation.json and alerts.json are indented; clients.json and lookups.json put each entry on one line.

Sizes

clients.json takes about 1 KiB per client, about 20 MiB at the default SWWAF_MAX_TRACKED_CLIENTS. Written every 15 minutes, that is under 2 GiB of disk writes a day. bans.json takes about 1.2 KiB per ban, at most about 2.5 KiB, so about 6 MiB at the default SWWAF_MAX_BANS. With a full file, a hundred new bans a day come to about 600 MiB of writes. lookups.json takes about 150 bytes per answer, under 1.5 GiB of writes a day when full. The other two files are usually a few MiB or less.

Writing

bans.json is written SWWAF_STATE_WRITE_DELAY after a ban is made, lifted or made permanent. Every file is written every SWWAF_STATE_COUNTER_INTERVAL, and when smallwebwaf stops. Each write goes to a temporary file that then replaces the file, so a crash leaves the old file or the new one, whole. Before writing a file, smallwebwaf takes in any edit you made to it, so your edit is not overwritten. A write that fails is logged, raised as a file_error alert, and tried again at the next write. A hard kill loses what changed since the last write:

• ban changes from the last SWWAF_STATE_WRITE_DELAY (10 seconds);
• everything else from the last SWWAF_STATE_COUNTER_INTERVAL (15 minutes):
  • counts and history, and the counts in ban notes;
  • GeoJS answers;
  • reputation verdicts and the AbuseIPDB count;
  • anomaly counters and alert cooldowns.

Reading at start

The files are read back. Each client keeps its counts, so a restart gives no one a fresh allowance. Each ban keeps refusing until it ends. Counter buckets and cache entries whose time has passed are discarded. A missing file is empty state. These stop the start, with a message naming the file and the place:

• a file that does not parse, or has another version;
• an entry without a field it needs;
• a ban whose cause is not limit, attack, admin or crowdsec;
• a ban whose expires is not after its start.

Editing while it runs

smallwebwaf watches the directory and takes in your edit of a file as soon as you save it. What the file then holds replaces what smallwebwaf held for it. It tells its own writes from yours by comparing the file with what it last read or wrote. A change it made after you opened the file, such as a new ban, is lost when you save over it; most editors warn when a file changed on disk while it was open. An edit with an error that would stop the start does not stop the running smallwebwaf. It keeps what it holds. At the file's next write it renames your file to <name>.bad, writes the file again from memory, and logs and alerts the file and the place of the error. Mend the .bad file and move it back. A file you remove is written again at its next write.