Caches of outside APIs live in /var/cache/sww (closes #314)

The answers of GeoJS, the lists fetched by URL, the DNS blocklists,
AbuseIPDB and routewatch each go in a cache file in SWW_CACHE_DIR
(default /var/cache/sww): lookups.json, lists.json, dnsbl.json,
abuseipdb-scores.json and routewatch.json. reputation.json is gone. The
AbuseIPDB checks spent today are a count, not a cache, and stay in
SWW_STATE_DIR as abuseipdb-checks-spent.json.

A cache file never stops the start: one that does not parse is set
aside as an edit is, one that cannot be read is written over, and its
cache starts empty. An unwritable SWW_CACHE_DIR stops the start, as an
unwritable SWW_STATE_DIR does. Each directory is watched on its own.
The run script makes the cache directory and gives it to sww.

Model: opus-5-5

docs/alerts.md

@@ -23,7 +23,8 @@ back or dropped.
     - the day's AbuseIPDB checks used up.
 - `file_error`: one of:
     - a rule file or state file edit with an error;
-    - a state file that cannot be written;
+    - a cache file that does not parse, at start or after an edit;
+    - a state file or cache file that cannot be written;
     - a replacement lookup database that cannot be read.
 - `summary`: the hourly summary of alerts held back.
 

docs/metrics.md

@@ -58,9 +58,9 @@ By `source`: each list's URL, each zone, and `abuseipdb`:
 - `sww_reputation_last_fetch_timestamp_seconds`, for a list;
 - `sww_reputation_daily_budget_remaining`, for AbuseIPDB.
 
-## State files
+## State and cache files
 
-By `file`:
+By `file`, the name of a state or cache file:
 
 - `sww_state_file_writes_total`, `sww_state_file_write_failures_total`,
   `sww_state_file_last_write_timestamp_seconds` and `sww_state_file_size_bytes`;

docs/settings.md

@@ -50,14 +50,16 @@ the two (see "The image's command" below).
 - `SWW_METRICS_TOKEN` (default unset): the token for `/_sww/metrics`. At least
   32 characters.
 
-## State files
+## State and cache files
 
 - `SWW_STATE_DIR` (default `/var/lib/sww`): the directory of the state files, an
   absolute path that exists and can be written.
 - `SWW_STATE_WRITE_DELAY` (default `10s`): how long `bans.json` waits to be
   written after a ban is made, lifted or made permanent. A duration.
-- `SWW_STATE_COUNTER_INTERVAL` (default `15m`): how often every state file is
-  written. A duration.
+- `SWW_STATE_COUNTER_INTERVAL` (default `15m`): how often every state and cache
+  file is written. A duration.
+- `SWW_CACHE_DIR` (default `/var/cache/sww`): the directory of the cache files,
+  an absolute path that can be written. The image's `run` script makes it.
 
 ## Logging
 

docs/state-files.md

@@ -1,8 +1,23 @@
-# State files
+# State and cache files
 
-The five JSON files in `SWW_STATE_DIR`, and what each holds. "Persistent state"
-in [`SPEC.md`](../SPEC.md) says how they are written, read and edited. Each has
-a top-level `version` of 1.
+The JSON files `waf` keeps, and what each holds. "Persistent state" in
+[`SPEC.md`](../SPEC.md) says how they are written, read and edited. Each has a
+top-level `version` of 1.
+
+- The state files are in `SWW_STATE_DIR`, `/var/lib/sww` by default. They hold
+  what cannot be fetched again, so back them up:
+    - `bans.json`;
+    - `clients.json`;
+    - `abuseipdb-checks-spent.json`;
+    - `alerts.json`.
+- The cache files are in `SWW_CACHE_DIR`, `/var/cache/sww` by default. They hold
+  answers from outside services, which are asked again when a file is lost, so
+  the directory can be left out of backups:
+    - `lookups.json`;
+    - `lists.json`;
+    - `dnsbl.json`;
+    - `abuseipdb-scores.json`;
+    - `routewatch.json`.
 
 ## `bans.json`
 
@@ -67,21 +82,13 @@ since it was first seen:
   `blocked_by_core_rule_set`. A file may also hold `token_refused`, which is
   kept, and which nothing adds to.
 
-## `lookups.json`
+## `abuseipdb-checks-spent.json`
 
-GeoJS answers, one per line:
+The AbuseIPDB checks spent today, so that a restart does not spend the daily
+budget again:
 
-- `client`;
-- `asn`, `as_name` and `country`;
-- `answered`: when GeoJS answered;
-- `used`: when the answer was last used.
-
-## `reputation.json`
-
-- the last good copy of each list fetched from a URL, with when it was last
-  tried;
-- each DNS blocklist verdict still in use;
-- AbuseIPDB's scores, and the checks spent today.
+- `day`: the day, in UTC;
+- `spent`: the checks spent on it.
 
 ## `alerts.json`
 
@@ -91,6 +98,51 @@ GeoJS answers, one per line:
 - the alerts sent and held back in the current hour;
 - the alerts waiting for each destination.
 
+## `lookups.json`
+
+A cache file. GeoJS answers, one per line:
+
+- `client`;
+- `asn`, `as_name` and `country`;
+- `answered`: when GeoJS answered;
+- `used`: when the answer was last used.
+
+## `lists.json`
+
+A cache file. The last good copy of each list fetched from a URL, with when it
+was last tried:
+
+- `url`;
+- `tried`: when it was last fetched or tried;
+- `fetched` and `lines`: when the last good copy was fetched, and its lines,
+  comment lines included.
+
+## `dnsbl.json`
+
+A cache file. Each DNS blocklist verdict still in use, one per line:
+
+- `zone` and `client`;
+- `listed`: whether the zone lists the client;
+- `fetched`: when the zone answered.
+
+## `abuseipdb-scores.json`
+
+A cache file. AbuseIPDB's scores still in use, one per line:
+
+- `client`;
+- `score`: from 0 to 100;
+- `fetched`: when AbuseIPDB answered.
+
+## `routewatch.json`
+
+A cache file. routewatch's answers from the last hour, one per line:
+
+- `client`;
+- `netblock`: the netblock the client's address is announced in, empty for an
+  address without a live route;
+- `asn`: the number of the AS that originates the netblock, 0 for none;
+- `answered`: when routewatch answered.
+
 ## Sizes
 
 - `clients.json`: about 1 KiB per client, about 20 MiB at the default
@@ -102,4 +154,7 @@ GeoJS answers, one per line:
   more.
 - `lookups.json`: about 150 bytes per answer, under 1.5 GiB of writes a day when
   full.
-- `reputation.json` and `alerts.json`: usually a few MiB or less.
+- `routewatch.json`: about 120 bytes per answer, at most about 12 MiB.
+- `abuseipdb-checks-spent.json`: a few bytes.
+- `lists.json`, `dnsbl.json`, `abuseipdb-scores.json` and `alerts.json`: usually
+  a few MiB or less.