Alerts to a JSON webhook, with a cooldown and an hourly summary (closes #26)

SWWAF_ALERT_WEBHOOK_URL gets one JSON POST per alert, in SPEC.md's
schema, with SWWAF_ALERT_WEBHOOK_HEADERS: ban and permanent_ban, with
the ban's notes; source_failure for GeoJS; file_error for a rule or
state file edit that does not parse and a failed state write.
SWWAF_ALERT_EVENTS chooses, SWWAF_ALERT_COOLDOWN holds back repeats,
and past SWWAF_ALERT_MAX_PER_HOUR the hour ends in one summary. A
bounded queue, retried with backoff, holds up no request; alerts.json
keeps it, the cooldowns and the hour. The ledger now reports whether it
made a ban, or made one permanent.

Judgement call: the summary's event is summary, which SPEC.md omits.
Judgement call: admin bans and observe mode raise no alert.

Model: opus-5-5

README.md

@@ -19,9 +19,10 @@ ledger with the bans you make, keep and lift, the JSON state files with your
 edits taken in while it runs and the paths the rate limits do not count, which
 come next in the build order, `observe` mode and the rest of the request log's
 fields, which come a little later, and the metrics endpoint and the header size
-and the idle time as settings, which come last in it. So are three parts of the
+and the idle time as settings, which come last in it. So are four parts of the
 stage after it: the rule files, the first part, with the bans for a clear sign
-of attack, the other admin endpoints, the second, and remote log sending.
+of attack, the other admin endpoints, the second, alerts to a JSON webhook, the
+first of the three destinations alerts go to, and remote log sending.
 `smallwebwaf` passes each request to the app and the app's answer back,
 unchanged, within its timeouts and size limits, works out each client's address,
 bans a client that sends too many requests, not counting those for the paths you
@@ -31,14 +32,16 @@ the rule files and bans a client whose request is a clear sign of attack, keeps
 its bans, each client's counters and history, and GeoJS's answers in JSON files
 across restarts, takes in your edits of those files, such as a ban you make,
 keep or lift, and of the rule files while it runs, writes a JSON log line for
-every request, sends its log lines to a syslog server too if you name one,
-serves Prometheus metrics to a scraper that holds the metrics token, lets an
-admin who holds the admin token list, add and lift bans and ask what it knows of
-a client, and in `observe` mode passes on the requests it would refuse, logging
-what it would have done with them. It comes as the image the app's own image is
-built on. The rest of the design comes after that, in the order of the build
-order in [`SPEC.md`](SPEC.md). The survey of existing tools that led to the
-design is in [`EVALUATION.md`](EVALUATION.md).
+every request, sends its log lines to a syslog server too if you name one, sends
+an alert to a webhook you name for each ban it makes or makes permanent, for
+GeoJS failing and for a rule file or state file with an error, serves Prometheus
+metrics to a scraper that holds the metrics token, lets an admin who holds the
+admin token list, add and lift bans and ask what it knows of a client, and in
+`observe` mode passes on the requests it would refuse, logging what it would
+have done with them. It comes as the image the app's own image is built on. The
+rest of the design comes after that, in the order of the build order in
+[`SPEC.md`](SPEC.md). The survey of existing tools that led to the design is in
+[`EVALUATION.md`](EVALUATION.md).
 
 ## Getting started
 
@@ -185,6 +188,11 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
 - Sends every line it writes on stdout to a syslog server as well, while
   `SWWAF_LOG_REMOTE_URL` names one (see "Sending the log to a syslog server"
   below).
+- Sends an alert, as a JSON object, to the webhook `SWWAF_ALERT_WEBHOOK_URL`
+  names, while it names one, for each ban it makes or makes permanent, for GeoJS
+  failing, and for a rule file or state file with an error, holding back repeats
+  and, past an hourly limit, rolling the rest into one summary (see "Alerts"
+  below).
 
 ## Settings
 
@@ -327,6 +335,22 @@ effective settings are logged at start.
   lines are sent with, 1 to 48 printable ASCII characters without a space. While
   `SWWAF_LOG_REMOTE_URL` is set, an `SWWAF_INSTANCE_NAME` that is not such a
   name stops the start too, unless this setting gives one that is.
+- `SWWAF_ALERT_WEBHOOK_URL` (default unset): the webhook each alert is posted
+  to, an `http` or `https` URL without a user or a fragment, such as
+  `https://alerts.example/smallwebwaf` (see "Alerts" below). Unset or empty, no
+  alert is sent.
+- `SWWAF_ALERT_WEBHOOK_HEADERS` (default empty): headers sent with each alert,
+  such as one that authenticates it, as a list of a name, `:` and a value, such
+  as `Authorization:Bearer 0123456789abcdef`. A value cannot hold a comma. The
+  settings logged at start show `********` in place of each value.
+- `SWWAF_ALERT_EVENTS` (default
+  `ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure,file_error`):
+  the events alerts are sent for. `waf_block`, `anomaly` and `reputation_hit`
+  come with the features that raise them; nothing raises them yet.
+- `SWWAF_ALERT_COOLDOWN` (default `15m`): how long a repeat of an alert is held
+  back (see "Alerts" below).
+- `SWWAF_ALERT_MAX_PER_HOUR` (default `60`): the most alerts sent in an hour;
+  the rest of the hour's alerts are rolled into one summary.
 
 Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are
 bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024
@@ -335,9 +359,10 @@ and a bare address stands for itself alone. Countries are the two-letter codes
 ISO 3166-1 assigns today, and `xk` for Kosovo, in either case (`de` and `DE` are
 the same); any other code, such as `nk` (North Korea is `kp`) or the withdrawn
 `su`, stops the start, and so does a code on both country lists. `off` switches
-a timeout, a size limit or a rate limit off;
-`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, the ban settings, the state settings,
-`SWWAF_METRICS_TOP_N` and `SWWAF_LOG_REMOTE_BUFFER` cannot be off.
+a timeout, a size limit, a rate limit, `SWWAF_ALERT_COOLDOWN` or
+`SWWAF_ALERT_MAX_PER_HOUR` off; `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, the ban
+settings, the state settings, `SWWAF_METRICS_TOP_N` and
+`SWWAF_LOG_REMOTE_BUFFER` cannot be off.
 
 Several limits are fixed rather than settings. At most 20,000 clients are kept,
 with their counters and history, and an IPv6 client is counted by its /64. A new
@@ -501,12 +526,114 @@ As `smallwebwaf` stops, it sends the lines still waiting, on the connection open
 or a new one, for at most two seconds, and gives up the rest; stdout has carried
 them.
 
+## Alerts
+
+While `SWWAF_ALERT_WEBHOOK_URL` is set, `smallwebwaf` posts each alert to it as
+one JSON object, with `Content-Type: application/json` and the headers
+`SWWAF_ALERT_WEBHOOK_HEADERS` gives, as "Alert webhook schema" in
+[`SPEC.md`](SPEC.md) describes. An alert is for one of these events, and is sent
+when `SWWAF_ALERT_EVENTS` names its event:
+
+- `ban`: a ban `smallwebwaf` makes, for a broken rate limit or a clear sign of
+  attack.
+- `permanent_ban`: a permanent ban it makes, or a ban for a clear sign of attack
+  that a request made permanent.
+- `source_failure`: GeoJS failing or refusing `smallwebwaf`.
+- `file_error`: a rule file edited while it runs that has an error, an edit of a
+  state file set aside as `<name>.bad`, or a state file it could not write while
+  running.
+
+The bans you make, in `bans.json` or through the ban endpoints, raise no alert,
+and neither does what `observe` mode would have done. This is the alert for a
+ban for a broken rate limit, shown indented; it is sent on one line:
+
+```json
+{
+    "instance": "fsn1app1/gitea",
+    "time": "2026-10-06T12:00:00.123461Z",
+    "event": "ban",
+    "client": "203.0.113.9",
+    "netblock": "203.0.113.9/32",
+    "asn": "",
+    "as_name": "",
+    "country": "",
+    "reason": "requests per minute over the limit of 1000",
+    "detail": {
+        "ban_expires": "2026-10-06T13:00:00.123Z",
+        "cause": "limit",
+        "notes": {
+            "country": "",
+            "limit": 1000,
+            "window": "minute",
+            "count": 1001,
+            "request": {
+                "time": "2026-10-06T12:00:00.123456789Z",
+                "method": "GET",
+                "host": "app.example",
+                "path": "/owner/repo/commits/branch/main?page=812",
+                "status": 403,
+                "user_agent": "scraper/1.0"
+            },
+            "requests": 5210,
+            "refused": 0,
+            "earlier_bans": {
+                "limit": 0,
+                "attack": 0,
+                "admin": 0
+            }
+        }
+    },
+    "suppressed_repeats": 0
+}
+```
+
+- `instance` is `SWWAF_INSTANCE_NAME`, and `time` when the alert was raised, in
+  UTC.
+- `client` is the address of the client whose request raised the alert, and
+  `netblock` the netblock of the ban; both are empty for `source_failure` and
+  `file_error`. `asn` and `as_name` are empty until AS numbers are looked up,
+  and `country` is, for a ban, the client's country as the ban's notes give it.
+- `reason` is a short sentence; for a ban, the ban's `reason` in `bans.json`.
+- `detail` is what is particular to the event: for a ban, its `cause`, when it
+  ends as `ban_expires`, in the form the request log gives it, and its `notes`,
+  as `bans.json` gives them; for `source_failure`, the `source`, `geojs`, the
+  `error`, and when GeoJS is asked again, `asking_again_in`; for `file_error`,
+  the `error`, which names the file and where in it the error is, and for an
+  edit set aside, the `file` it was renamed to.
+- `suppressed_repeats` is how many repeats the cooldown held back before this
+  alert.
+
+An alert for the same event on the same netblock as the last one sent, or, for
+an event without a netblock, for the same event, less than
+`SWWAF_ALERT_COOLDOWN` after it, is a repeat: it is held back and counted, and
+the next alert sent for them gives that count as `suppressed_repeats`. So every
+`file_error` alert shares one cooldown, and so does every `source_failure`
+alert.
+
+Past `SWWAF_ALERT_MAX_PER_HOUR` alerts in an hour of the clock, in UTC, the
+hour's other alerts are held back and counted by event. Once the hour has ended,
+one alert sums them up: its `event` is `summary`, its `reason` says how many
+were held back, and its `detail` gives the `hour` as when it started, the
+`count`, and the count for each event, as `events`. An alert held back this way
+starts its cooldown as one sent does.
+
+The alerts wait in a queue of at most 1000, from which they are sent one at a
+time, the oldest first, so a webhook that is slow or down never holds up a
+request. The webhook takes an alert by answering with a 2xx status. Any other
+answer, a redirect included, a connection that fails, or no answer within 10
+seconds is a failure: it is logged, and the alert is sent again a second later,
+twice as long after each further failure in a row, up to a minute. With 1000
+alerts waiting, the oldest is dropped to make room for a new one. The cooldowns,
+the hour under way and the alerts still waiting are kept in `alerts.json` (see
+"State files" below), so that after a restart the alerts waiting are sent, and
+the cooldowns go on.
+
 ## State files
 
-`smallwebwaf` keeps its state in memory and a copy of it in three JSON files in
+`smallwebwaf` keeps its state in memory and a copy of it in four JSON files in
 `SWWAF_STATE_DIR`, `/var/lib/smallwebwaf` by default, as "Persistent state" in
 [`SPEC.md`](SPEC.md) describes. Each has a top-level `version`, 1, and lists its
-entries by client address, with times in UTC.
+entries by client address, but for the alerts waiting, with times in UTC.
 
 - `bans.json`: every ban with its notes, indented to be read. A permanent ban's
   `expires` is `null`. A ban's `cause` is `limit` for a broken rate limit or
@@ -525,14 +652,23 @@ entries by client address, with times in UTC.
   line of its own, so `grep` shows everything about one.
 - `lookups.json`: GeoJS's answers, one to a line, with when GeoJS gave each and
   when it was last used.
+- `alerts.json`: the state of the alerts (see "Alerts" above), indented to be
+  read: under `cooldowns`, for each event and netblock, or event alone, when the
+  last alert was sent, `sent`, and the repeats held back since,
+  `suppressed_repeats`; under `hour`, the hour under way, from its `start`, the
+  alerts `sent` in it and those `held_back` for its summary, by event; and under
+  `waiting`, the alerts still waiting to be sent, the oldest first, each as the
+  webhook is sent it. As an hour ends, the cooldowns that have run out with no
+  repeat held back are dropped.
 
 `bans.json` is written `SWWAF_STATE_WRITE_DELAY` after a ban is made, lifted
 through `DELETE /_smallwebwaf/bans/<client>`, or made permanent, with every such
 change in between, and every file every `SWWAF_STATE_COUNTER_INTERVAL` and when
 `smallwebwaf` stops. Each write goes to a temporary file in the same directory,
 which then replaces the file, so a crash leaves the old file or the new one,
-whole. A write that fails is logged, and tried again at the next write. A hard
-kill loses what changed since the last write.
+whole. A write that fails is logged, raised as a `file_error` alert while
+`smallwebwaf` runs, and tried again at the next write. A hard kill loses what
+changed since the last write.
 
 At start the files are read back: each client keeps its counts, so a restart
 gives it no fresh allowance, and each ban keeps refusing every client in its
@@ -547,8 +683,9 @@ without a field it needs, named with the entry's place in the file: a ban's
 `netblock`, `start` or `expires`, which is `null` for a permanent ban; a
 client's `client`, or the `start` of a window in which it has requests; an
 answer's `client`, `country`, which is `""` for a client GeoJS cannot place, or
-`answered`. So does a ban whose `cause` is not `limit`, `attack` or `admin`. The
-AS number and AS name come with their lookup.
+`answered`; a cooldown's `event` or `sent`; an alert waiting's `event` or
+`time`. So does a ban whose `cause` is not `limit`, `attack` or `admin`. The AS
+number and AS name come with their lookup.
 
 While it runs, `smallwebwaf` watches `SWWAF_STATE_DIR` and takes in your edit of
 a state file as soon as you save it: what the file then holds replaces what
@@ -560,10 +697,11 @@ lost when you save over it. An edit that would stop the start, because it does
 not parse, has another `version`, leaves out a field an entry needs or gives a
 ban another `cause`, does not stop the running `smallwebwaf`: it keeps what it
 holds, and at the file's next write renames your file to `<name>.bad`, such as
-`bans.json.bad`, writes the file again from memory, and logs the file and where
-the error is. It waits for that write because an editor's file can be read
-before the editor has finished writing it. Mend the `.bad` file and move it
-back. A file you remove is written again at its next write.
+`bans.json.bad`, writes the file again from memory, logs the file and where the
+error is, and raises a `file_error` alert for it. It waits for that write
+because an editor's file can be read before the editor has finished writing it.
+Mend the `.bad` file and move it back. A file you remove is written again at its
+next write.
 
 To ban a netblock, add an entry to `bans.json` with its `netblock`, its `start`
 and its `expires`, `null` for a ban that never ends; its `reason` and its
@@ -649,8 +787,8 @@ place, appended to or copied in with `scp` is read only once whole, unless its
 writing stops for longer. It also reads them 2 seconds after it starts watching,
 so that an edit saved while it started is not missed. If they then hold one of
 those errors, the rules stay as they were, the earlier version of the edited
-file included, the log names the file and the line, and the files are read again
-after the next change.
+file included, the log and a `file_error` alert name the file and the line, and
+the files are read again after the next change.
 
 The image ships one rule file, `share/rules.d/00-default.rules` here: rules that
 ban probes no real visitor sends, for secrets, version control directories,
@@ -727,6 +865,12 @@ other request. No metric carries a client's address.
   `smallwebwaf_remote_log_lines_dropped_total`: those dropped, from a full
   buffer or because their sending failed; and
   `smallwebwaf_remote_log_buffer_depth`: those waiting in the buffer.
+- While `SWWAF_ALERT_WEBHOOK_URL` is set, by `destination`, `webhook`:
+  `smallwebwaf_alerts_sent_total`: the alerts the webhook took;
+  `smallwebwaf_alerts_failed_total`: the requests to it that failed;
+  `smallwebwaf_alerts_suppressed_total`: the alerts held back, as repeats or for
+  an hour's summary; and `smallwebwaf_alerts_dropped_total`: those dropped from
+  a full queue.
 - Go's own `go_` metrics and the process's `process_` metrics.
 
 The requests Go's HTTP server ends before `smallwebwaf` sees them (see "Request
@@ -897,9 +1041,9 @@ goes through the candidates one by one.
   readable JSON files, written regularly and at every stop, so a restart loses
   nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks
   up the change. Nothing is read from disk while serving a request. The files
-  for the bans, the clients and the GeoJS answers are built, with an edit taken
-  in while running (see "State files" above); the others come with their
-  features.
+  for the bans, the clients, the GeoJS answers and the alerts are built, with an
+  edit taken in while running (see "State files" above); the others come with
+  their features.
 - Health checks, the metrics, and listing, adding and lifting bans or asking why
   a given address was refused, all on the one port every request uses: under
   `/_smallwebwaf/` on the app's own address, through traefik like any other
@@ -1103,6 +1247,9 @@ addresses are never sent to GeoJS.
   each as a syslog record, from a buffer of its own. It is written with the
   standard library alone, whose `log/syslog` writes only the older syslog
   format.
+- `internal/alerts`: takes the alerts the other parts raise, holds back repeats
+  and those past the hourly limit, and sends the others to
+  `SWWAF_ALERT_WEBHOOK_URL` from a queue of its own.
 - `Dockerfile`: the lint and test phases, then the image, whose last stage
   installs Ubuntu's packages, nixpkgs, `runsvinit` and `smallwebwaf`, with
   `share/smallwebwaf.run` as runit's `run` script for `smallwebwaf` and