Alerts to a JSON webhook, with a cooldown and an hourly summary (closes #26)

SWWAF_ALERT_WEBHOOK_URL gets one JSON POST per alert, in SPEC.md's
schema, with SWWAF_ALERT_WEBHOOK_HEADERS: ban and permanent_ban, with
the ban's notes, in observe mode too, marked mode observe and worked
out only when the alert would be sent; source_failure for GeoJS;
file_error for a rule or state file with an error. SWWAF_ALERT_EVENTS
chooses; SWWAF_ALERT_COOLDOWN holds back repeats by netblock, file or
source; past SWWAF_ALERT_MAX_PER_HOUR the hour ends in one summary. A
bounded queue, retried with backoff, holds up no request; a 4xx other
than 408 and 429 gives the alert up. alerts.json keeps the queue, the
cooldowns and the hour. Nothing shows the URL's path or query.

Judgement call: the summary's event is summary, which SPEC.md omits.
Judgement call: an admin's ban raises no alert.

Model: opus-5-5

README.md

@@ -19,9 +19,10 @@ ledger with the bans you make, keep and lift, the JSON state files with your
 edits taken in while it runs and the paths the rate limits do not count, which
 come next in the build order, `observe` mode and the rest of the request log's
 fields, which come a little later, and the metrics endpoint and the header size
-and the idle time as settings, which come last in it. So are three parts of the
+and the idle time as settings, which come last in it. So are four parts of the
 stage after it: the rule files, the first part, with the bans for a clear sign
-of attack, the other admin endpoints, the second, and remote log sending.
+of attack, the other admin endpoints, the second, alerts to a JSON webhook, the
+first of the three destinations alerts go to, and remote log sending.
 `smallwebwaf` passes each request to the app and the app's answer back,
 unchanged, within its timeouts and size limits, works out each client's address,
 bans a client that sends too many requests, not counting those for the paths you
@@ -31,14 +32,16 @@ the rule files and bans a client whose request is a clear sign of attack, keeps
 its bans, each client's counters and history, and GeoJS's answers in JSON files
 across restarts, takes in your edits of those files, such as a ban you make,
 keep or lift, and of the rule files while it runs, writes a JSON log line for
-every request, sends its log lines to a syslog server too if you name one,
-serves Prometheus metrics to a scraper that holds the metrics token, lets an
-admin who holds the admin token list, add and lift bans and ask what it knows of
-a client, and in `observe` mode passes on the requests it would refuse, logging
-what it would have done with them. It comes as the image the app's own image is
-built on. The rest of the design comes after that, in the order of the build
-order in [`SPEC.md`](SPEC.md). The survey of existing tools that led to the
-design is in [`EVALUATION.md`](EVALUATION.md).
+every request, sends its log lines to a syslog server too if you name one, sends
+an alert to a webhook you name for each ban it makes or makes permanent, for
+GeoJS failing and for a rule file or state file with an error, serves Prometheus
+metrics to a scraper that holds the metrics token, lets an admin who holds the
+admin token list, add and lift bans and ask what it knows of a client, and in
+`observe` mode passes on the requests it would refuse, logging what it would
+have done with them. It comes as the image the app's own image is built on. The
+rest of the design comes after that, in the order of the build order in
+[`SPEC.md`](SPEC.md). The survey of existing tools that led to the design is in
+[`EVALUATION.md`](EVALUATION.md).
 
 ## Getting started
 
@@ -161,7 +164,9 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
   neither a broken rate limit nor a `ban` rule makes a ban; a broken rate limit
   does not set the client's counters back to zero, so each request over the
   limit is logged as one that would be refused; and a request under a ban does
-  not make it permanent. The bans in `bans.json` are kept, and refuse requests
+  not make it permanent. A ban it would have made, or made permanent, raises the
+  alert `enforce` mode would have raised, marked as what would have happened
+  (see "Alerts" below). The bans in `bans.json` are kept, and refuse requests
   again when `smallwebwaf` next runs in `enforce` mode, as long as they last.
   The timeouts and size limits still apply, since they protect `smallwebwaf` and
   the app themselves, and a request for one of `smallwebwaf`'s own endpoints
@@ -185,6 +190,11 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
 - Sends every line it writes on stdout to a syslog server as well, while
   `SWWAF_LOG_REMOTE_URL` names one (see "Sending the log to a syslog server"
   below).
+- Sends an alert, as a JSON object, to the webhook `SWWAF_ALERT_WEBHOOK_URL`
+  names, while it names one, for each ban it makes or makes permanent, for GeoJS
+  failing, and for a rule file or state file with an error, holding back repeats
+  and, past an hourly limit, rolling the rest into one summary (see "Alerts"
+  below).
 
 ## Settings
 
@@ -327,6 +337,24 @@ effective settings are logged at start.
   lines are sent with, 1 to 48 printable ASCII characters without a space. While
   `SWWAF_LOG_REMOTE_URL` is set, an `SWWAF_INSTANCE_NAME` that is not such a
   name stops the start too, unless this setting gives one that is.
+- `SWWAF_ALERT_WEBHOOK_URL` (default unset): the webhook each alert is posted
+  to, an `http` or `https` URL without a user or a fragment, such as
+  `https://alerts.example/smallwebwaf` (see "Alerts" below). Unset or empty, no
+  alert is sent. Since many webhooks carry their secret in the path or the
+  query, the settings logged at start show `********` in place of them, and a
+  value that stops the start is not shown.
+- `SWWAF_ALERT_WEBHOOK_HEADERS` (default empty): headers sent with each alert,
+  such as one that authenticates it, as a list of a name, `:` and a value, such
+  as `Authorization:Bearer 0123456789abcdef`. A value cannot hold a comma. The
+  settings logged at start show `********` in place of each value.
+- `SWWAF_ALERT_EVENTS` (default
+  `ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure,file_error`):
+  the events alerts are sent for. `waf_block`, `anomaly` and `reputation_hit`
+  come with the features that raise them; nothing raises them yet.
+- `SWWAF_ALERT_COOLDOWN` (default `15m`): how long a repeat of an alert is held
+  back (see "Alerts" below).
+- `SWWAF_ALERT_MAX_PER_HOUR` (default `60`): the most alerts sent in an hour;
+  the rest of the hour's alerts are rolled into one summary.
 
 Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are
 bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024
@@ -335,9 +363,10 @@ and a bare address stands for itself alone. Countries are the two-letter codes
 ISO 3166-1 assigns today, and `xk` for Kosovo, in either case (`de` and `DE` are
 the same); any other code, such as `nk` (North Korea is `kp`) or the withdrawn
 `su`, stops the start, and so does a code on both country lists. `off` switches
-a timeout, a size limit or a rate limit off;
-`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, the ban settings, the state settings,
-`SWWAF_METRICS_TOP_N` and `SWWAF_LOG_REMOTE_BUFFER` cannot be off.
+a timeout, a size limit, a rate limit, `SWWAF_ALERT_COOLDOWN` or
+`SWWAF_ALERT_MAX_PER_HOUR` off; `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, the ban
+settings, the state settings, `SWWAF_METRICS_TOP_N` and
+`SWWAF_LOG_REMOTE_BUFFER` cannot be off.
 
 Several limits are fixed rather than settings. At most 20,000 clients are kept,
 with their counters and history, and an IPv6 client is counted by its /64. A new
@@ -501,12 +530,121 @@ As `smallwebwaf` stops, it sends the lines still waiting, on the connection open
 or a new one, for at most two seconds, and gives up the rest; stdout has carried
 them.
 
+## Alerts
+
+While `SWWAF_ALERT_WEBHOOK_URL` is set, `smallwebwaf` posts each alert to it as
+one JSON object, with `Content-Type: application/json` and the headers
+`SWWAF_ALERT_WEBHOOK_HEADERS` gives, as "Alert webhook schema" in
+[`SPEC.md`](SPEC.md) describes. An alert is for one of these events, and is sent
+when `SWWAF_ALERT_EVENTS` names its event:
+
+- `ban`: a ban `smallwebwaf` makes, for a broken rate limit or a clear sign of
+  attack.
+- `permanent_ban`: a permanent ban it makes, or a ban for a clear sign of attack
+  that a request made permanent.
+- `source_failure`: GeoJS failing or refusing `smallwebwaf`.
+- `file_error`: a rule file edited while it runs that has an error, an edit of a
+  state file set aside as `<name>.bad`, or a state file it could not write while
+  running.
+
+The bans you make, in `bans.json` or through the ban endpoints, raise no alert.
+In `observe` mode, a request that would have made a ban, or made one permanent,
+raises the alert `enforce` mode would have raised, for the ban as it would have
+been, with `mode`, `observe`, in its `detail`: no ban was made, or made
+permanent. A request that would have made a ban whose alert would be held back,
+by the cooldown or past `SWWAF_ALERT_MAX_PER_HOUR`, raises none, and is not
+counted. This is the alert for a ban for a broken rate limit, shown indented; it
+is sent on one line:
+
+```json
+{
+    "instance": "fsn1app1/gitea",
+    "time": "2026-10-06T12:00:00.123461Z",
+    "event": "ban",
+    "client": "203.0.113.9",
+    "netblock": "203.0.113.9/32",
+    "asn": "",
+    "as_name": "",
+    "country": "",
+    "reason": "requests per minute over the limit of 1000",
+    "detail": {
+        "ban_expires": "2026-10-06T13:00:00.123Z",
+        "cause": "limit",
+        "notes": {
+            "country": "",
+            "limit": 1000,
+            "window": "minute",
+            "count": 1001,
+            "request": {
+                "time": "2026-10-06T12:00:00.123456789Z",
+                "method": "GET",
+                "host": "app.example",
+                "path": "/owner/repo/commits/branch/main?page=812",
+                "status": 403,
+                "user_agent": "scraper/1.0"
+            },
+            "requests": 5210,
+            "refused": 0,
+            "earlier_bans": {
+                "limit": 0,
+                "attack": 0,
+                "admin": 0
+            }
+        }
+    },
+    "suppressed_repeats": 0
+}
+```
+
+- `instance` is `SWWAF_INSTANCE_NAME`, and `time` when the alert was raised, in
+  UTC.
+- `client` is the address of the client whose request raised the alert, and
+  `netblock` the netblock of the ban; both are empty for `source_failure` and
+  `file_error`. `asn` and `as_name` are empty until AS numbers are looked up,
+  and `country` is, for a ban, the client's country as the ban's notes give it.
+- `reason` is a short sentence; for a ban, the ban's `reason` in `bans.json`.
+- `detail` is what is particular to the event: for a ban, its `cause`, when it
+  ends as `ban_expires`, in the form the request log gives it, and its `notes`,
+  as `bans.json` gives them; for `source_failure`, the `source`, `geojs`, the
+  `error`, and when GeoJS is asked again, `asking_again_in`; for `file_error`,
+  the `file`, which for an edit set aside is the file it was renamed to, and the
+  `error`, which for a file that does not parse names where in it the error is.
+- `suppressed_repeats` is how many repeats the cooldown held back before this
+  alert.
+
+An alert for the same event as the last one sent, on the same netblock, or for a
+`file_error` about the same file, or for a `source_failure` about the same
+source, less than `SWWAF_ALERT_COOLDOWN` after it, is a repeat: it is held back
+and counted, and the next alert sent for them gives that count as
+`suppressed_repeats`.
+
+Past `SWWAF_ALERT_MAX_PER_HOUR` alerts in an hour of the clock, in UTC, the
+hour's other alerts are held back and counted by event. Once the hour has ended,
+one alert sums them up: its `event` is `summary`, its `reason` says how many
+were held back, and its `detail` gives the `hour` as when it started, the
+`count`, and the count for each event, as `events`. An alert held back this way
+starts no cooldown, and the repeats held back before it are given by the next
+alert sent for the same event and netblock, file or source.
+
+The alerts wait in a queue of at most 1000, from which they are sent one at a
+time, the oldest first, so a webhook that is slow or down never holds up a
+request. The webhook takes an alert by answering with a 2xx status, and refuses
+it with a 4xx status other than `408` and `429`: a refused alert is logged,
+counted as dropped, and given up, so that the next is sent. Any other answer, a
+redirect included, a connection that fails, or no answer within 10 seconds is a
+failure: it is logged, without the webhook's URL, and the alert is sent again a
+second later, twice as long after each further failure in a row, up to a minute.
+With 1000 alerts waiting, the oldest is dropped to make room for a new one. The
+cooldowns, the hour under way and the alerts still waiting are kept in
+`alerts.json` (see "State files" below), so that after a restart the alerts
+waiting are sent, and the cooldowns go on.
+
 ## State files
 
-`smallwebwaf` keeps its state in memory and a copy of it in three JSON files in
+`smallwebwaf` keeps its state in memory and a copy of it in four JSON files in
 `SWWAF_STATE_DIR`, `/var/lib/smallwebwaf` by default, as "Persistent state" in
 [`SPEC.md`](SPEC.md) describes. Each has a top-level `version`, 1, and lists its
-entries by client address, with times in UTC.
+entries by client address, but for the alerts waiting, with times in UTC.
 
 - `bans.json`: every ban with its notes, indented to be read. A permanent ban's
   `expires` is `null`. A ban's `cause` is `limit` for a broken rate limit or
@@ -525,14 +663,23 @@ entries by client address, with times in UTC.
   line of its own, so `grep` shows everything about one.
 - `lookups.json`: GeoJS's answers, one to a line, with when GeoJS gave each and
   when it was last used.
+- `alerts.json`: the state of the alerts (see "Alerts" above), indented to be
+  read: under `cooldowns`, for each event and netblock, or event and `file` or
+  `source`, or event alone, when the last alert was sent, `sent`, and the
+  repeats held back since, `suppressed_repeats`; under `hour`, the hour under
+  way, from its `start`, the alerts `sent` in it and those `held_back` for its
+  summary, by event; and under `waiting`, the alerts still waiting to be sent,
+  the oldest first, each as the webhook is sent it. As an hour ends, the
+  cooldowns that have run out with no repeat held back are dropped.
 
 `bans.json` is written `SWWAF_STATE_WRITE_DELAY` after a ban is made, lifted
 through `DELETE /_smallwebwaf/bans/<client>`, or made permanent, with every such
 change in between, and every file every `SWWAF_STATE_COUNTER_INTERVAL` and when
 `smallwebwaf` stops. Each write goes to a temporary file in the same directory,
 which then replaces the file, so a crash leaves the old file or the new one,
-whole. A write that fails is logged, and tried again at the next write. A hard
-kill loses what changed since the last write.
+whole. A write that fails is logged, raised as a `file_error` alert while
+`smallwebwaf` runs, and tried again at the next write. A hard kill loses what
+changed since the last write.
 
 At start the files are read back: each client keeps its counts, so a restart
 gives it no fresh allowance, and each ban keeps refusing every client in its
@@ -547,8 +694,9 @@ without a field it needs, named with the entry's place in the file: a ban's
 `netblock`, `start` or `expires`, which is `null` for a permanent ban; a
 client's `client`, or the `start` of a window in which it has requests; an
 answer's `client`, `country`, which is `""` for a client GeoJS cannot place, or
-`answered`. So does a ban whose `cause` is not `limit`, `attack` or `admin`. The
-AS number and AS name come with their lookup.
+`answered`; a cooldown's `event` or `sent`; an alert waiting's `event` or
+`time`. So does a ban whose `cause` is not `limit`, `attack` or `admin`. The AS
+number and AS name come with their lookup.
 
 While it runs, `smallwebwaf` watches `SWWAF_STATE_DIR` and takes in your edit of
 a state file as soon as you save it: what the file then holds replaces what
@@ -560,10 +708,11 @@ lost when you save over it. An edit that would stop the start, because it does
 not parse, has another `version`, leaves out a field an entry needs or gives a
 ban another `cause`, does not stop the running `smallwebwaf`: it keeps what it
 holds, and at the file's next write renames your file to `<name>.bad`, such as
-`bans.json.bad`, writes the file again from memory, and logs the file and where
-the error is. It waits for that write because an editor's file can be read
-before the editor has finished writing it. Mend the `.bad` file and move it
-back. A file you remove is written again at its next write.
+`bans.json.bad`, writes the file again from memory, logs the file and where the
+error is, and raises a `file_error` alert for it. It waits for that write
+because an editor's file can be read before the editor has finished writing it.
+Mend the `.bad` file and move it back. A file you remove is written again at its
+next write.
 
 To ban a netblock, add an entry to `bans.json` with its `netblock`, its `start`
 and its `expires`, `null` for a ban that never ends; its `reason` and its
@@ -649,8 +798,8 @@ place, appended to or copied in with `scp` is read only once whole, unless its
 writing stops for longer. It also reads them 2 seconds after it starts watching,
 so that an edit saved while it started is not missed. If they then hold one of
 those errors, the rules stay as they were, the earlier version of the edited
-file included, the log names the file and the line, and the files are read again
-after the next change.
+file included, the log and a `file_error` alert name the file and the line, and
+the files are read again after the next change.
 
 The image ships one rule file, `share/rules.d/00-default.rules` here: rules that
 ban probes no real visitor sends, for secrets, version control directories,
@@ -727,6 +876,12 @@ other request. No metric carries a client's address.
   `smallwebwaf_remote_log_lines_dropped_total`: those dropped, from a full
   buffer or because their sending failed; and
   `smallwebwaf_remote_log_buffer_depth`: those waiting in the buffer.
+- While `SWWAF_ALERT_WEBHOOK_URL` is set, by `destination`, `webhook`:
+  `smallwebwaf_alerts_sent_total`: the alerts the webhook took;
+  `smallwebwaf_alerts_failed_total`: the requests to it that failed;
+  `smallwebwaf_alerts_suppressed_total`: the alerts held back, as repeats or for
+  an hour's summary; and `smallwebwaf_alerts_dropped_total`: those dropped from
+  a full queue, or given up as the webhook refused them.
 - Go's own `go_` metrics and the process's `process_` metrics.
 
 The requests Go's HTTP server ends before `smallwebwaf` sees them (see "Request
@@ -897,9 +1052,9 @@ goes through the candidates one by one.
   readable JSON files, written regularly and at every stop, so a restart loses
   nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks
   up the change. Nothing is read from disk while serving a request. The files
-  for the bans, the clients and the GeoJS answers are built, with an edit taken
-  in while running (see "State files" above); the others come with their
-  features.
+  for the bans, the clients, the GeoJS answers and the alerts are built, with an
+  edit taken in while running (see "State files" above); the others come with
+  their features.
 - Health checks, the metrics, and listing, adding and lifting bans or asking why
   a given address was refused, all on the one port every request uses: under
   `/_smallwebwaf/` on the app's own address, through traefik like any other
@@ -1103,6 +1258,9 @@ addresses are never sent to GeoJS.
   each as a syslog record, from a buffer of its own. It is written with the
   standard library alone, whose `log/syslog` writes only the older syslog
   format.
+- `internal/alerts`: takes the alerts the other parts raise, holds back repeats
+  and those past the hourly limit, and sends the others to
+  `SWWAF_ALERT_WEBHOOK_URL` from a queue of its own.
 - `Dockerfile`: the lint and test phases, then the image, whose last stage
   installs Ubuntu's packages, nixpkgs, `runsvinit` and `smallwebwaf`, with
   `share/smallwebwaf.run` as runit's `run` script for `smallwebwaf` and