Alerts to a JSON webhook, with a cooldown and an hourly summary #93

Merged
clawbot merged 1 commits from issue-26-alerts into next 2026-10-07 04:21:24 +02:00
Collaborator

Alerts for #26: the queue and the JSON webhook.

  • SWWAF_ALERT_WEBHOOK_URL gets one JSON POST per alert, in SPEC.md's schema, with SWWAF_ALERT_WEBHOOK_HEADERS. No log line shows the headers' values or the URL's path and query.
  • Events: ban and permanent_ban (cause, ban_expires and notes in detail), in observe mode too, for the ban it would have made, worked out only when its alert would be sent, with mode observe in detail; source_failure for GeoJS; file_error, with its file, for a rule or state file with an error. SWWAF_ALERT_EVENTS chooses.
  • SWWAF_ALERT_COOLDOWN holds back the same event on the same netblock, file or source; the next alert carries suppressed_repeats. Past SWWAF_ALERT_MAX_PER_HOUR, the hour's other alerts become one summary; they start no cooldown, so earlier repeats reach the next alert sent.
  • A queue of 1000, oldest dropped first, retried after 1s doubling to a minute; a 4xx other than 408 and 429 gives the alert up, counted as dropped.
  • alerts.json keeps the cooldowns, the hour and the alerts waiting.
  • The ledger now says whether it made a ban, or made one permanent; WouldBanForLimit and WouldBanForAttack give the ban observe mode would have made. Most test edits are that.

Judgement call: the summary's event is summary, which the schema in SPEC.md does not list.
Judgement call: an admin's ban raises no alert.
Judgement call: in observe mode, a would-be ban whose alert would be held back raises nothing and is not counted.
Not in this unit: the anomaly counters of alerts.json, which come with the anomaly thresholds.

Model: opus-5-5

Alerts for https://git.eeqj.de/sneak/smallwebwaf/issues/26: the queue and the JSON webhook. - `SWWAF_ALERT_WEBHOOK_URL` gets one JSON POST per alert, in `SPEC.md`'s schema, with `SWWAF_ALERT_WEBHOOK_HEADERS`. No log line shows the headers' values or the URL's path and query. - Events: `ban` and `permanent_ban` (cause, `ban_expires` and notes in `detail`), in `observe` mode too, for the ban it would have made, worked out only when its alert would be sent, with `mode` `observe` in `detail`; `source_failure` for GeoJS; `file_error`, with its `file`, for a rule or state file with an error. `SWWAF_ALERT_EVENTS` chooses. - `SWWAF_ALERT_COOLDOWN` holds back the same event on the same netblock, file or source; the next alert carries `suppressed_repeats`. Past `SWWAF_ALERT_MAX_PER_HOUR`, the hour's other alerts become one summary; they start no cooldown, so earlier repeats reach the next alert sent. - A queue of 1000, oldest dropped first, retried after 1s doubling to a minute; a 4xx other than `408` and `429` gives the alert up, counted as dropped. - `alerts.json` keeps the cooldowns, the hour and the alerts waiting. - The ledger now says whether it made a ban, or made one permanent; `WouldBanForLimit` and `WouldBanForAttack` give the ban `observe` mode would have made. Most test edits are that. Judgement call: the summary's `event` is `summary`, which the schema in `SPEC.md` does not list. Judgement call: an admin's ban raises no alert. Judgement call: in `observe` mode, a would-be ban whose alert would be held back raises nothing and is not counted. Not in this unit: the anomaly counters of `alerts.json`, which come with the anomaly thresholds. Model: opus-5-5
clawbot added the needs-review label 2026-10-07 02:12:54 +02:00
clawbot self-assigned this 2026-10-07 02:12:54 +02:00
Author
Collaborator

Review failed: needs rework.

  1. Secret in the webhook URL printed. internal/config/config.go (webhookURL, parseWebhookURL) and internal/alerts/alerts.go (send): SWWAF_ALERT_WEBHOOK_URL is logged in full at start, quoted in the start error when it does not parse, and quoted in every "sending an alert to SWWAF_ALERT_WEBHOOK_URL failed" warning, since the HTTP client's error carries the URL. Many webhooks carry their secret in the path or query, and the _FILE form exists for secrets. Acceptable: the start log and the start error show no path or query (or no value, as for the headers), and the failed-send warning carries no URL.

  2. No alert in observe mode. internal/proxy/bans.go, internal/proxy/rules.go, README.md "Alerts": a ban observe mode would have made raises nothing, while SPEC.md "Configuration surface" says observe mode logs and alerts on every decision. Acceptable: in observe mode a request that would have made a ban, or made one permanent, raises that alert, marked as what would have happened, under the same cooldown; TestObserveModeRaisesNoBanAlert turned around.

  3. One cooldown for every file_error. internal/alerts/alerts.go (repeat): a rule file error, or another state file's edit set aside, within the cooldown of any other file_error (a failing write, another file) is held back as a repeat, so the alert naming its file and line is never sent. SPEC.md promises that alert for each, and counts only identical alerts as repeats. Acceptable: a file_error repeats only one for the same file (likewise source_failure, per source, once there is more than one source).

  4. A refused alert holds up every later one. internal/alerts/alerts.go (Run): an alert the webhook answers with a 4xx such as 400 is retried until 1000 newer alerts push it out, and since sending is strictly oldest first, nothing behind it is delivered meanwhile: at the default hourly limit, most of a day. The retry in SPEC.md is for a destination that is down. Acceptable: a 4xx other than 408 and 429 gives that alert up, logged and counted, so the alerts behind it are sent.

  5. The hourly limit loses the cooldown's count. internal/alerts/alerts.go (Raise, repeat): when the first alert after a cooldown is held back by SWWAF_ALERT_MAX_PER_HOUR, its cooldown restarts at 0, so the repeats held back before it reach neither the summary nor the next alert sent. Acceptable: those repeats still reach the webhook, in the next alert sent for that event and netblock or in the summary.

Judgement calls accepted: the summary's event summary; no alert for an admin's ban.

Model: opus-5-5

Review failed: needs rework. 1. Secret in the webhook URL printed. `internal/config/config.go` (`webhookURL`, `parseWebhookURL`) and `internal/alerts/alerts.go` (`send`): `SWWAF_ALERT_WEBHOOK_URL` is logged in full at start, quoted in the start error when it does not parse, and quoted in every "sending an alert to SWWAF_ALERT_WEBHOOK_URL failed" warning, since the HTTP client's error carries the URL. Many webhooks carry their secret in the path or query, and the `_FILE` form exists for secrets. Acceptable: the start log and the start error show no path or query (or no value, as for the headers), and the failed-send warning carries no URL. 2. No alert in `observe` mode. `internal/proxy/bans.go`, `internal/proxy/rules.go`, `README.md` "Alerts": a ban `observe` mode would have made raises nothing, while `SPEC.md` "Configuration surface" says `observe` mode logs and alerts on every decision. Acceptable: in `observe` mode a request that would have made a ban, or made one permanent, raises that alert, marked as what would have happened, under the same cooldown; `TestObserveModeRaisesNoBanAlert` turned around. 3. One cooldown for every `file_error`. `internal/alerts/alerts.go` (`repeat`): a rule file error, or another state file's edit set aside, within the cooldown of any other `file_error` (a failing write, another file) is held back as a repeat, so the alert naming its file and line is never sent. `SPEC.md` promises that alert for each, and counts only identical alerts as repeats. Acceptable: a `file_error` repeats only one for the same file (likewise `source_failure`, per source, once there is more than one source). 4. A refused alert holds up every later one. `internal/alerts/alerts.go` (`Run`): an alert the webhook answers with a 4xx such as `400` is retried until 1000 newer alerts push it out, and since sending is strictly oldest first, nothing behind it is delivered meanwhile: at the default hourly limit, most of a day. The retry in `SPEC.md` is for a destination that is down. Acceptable: a 4xx other than `408` and `429` gives that alert up, logged and counted, so the alerts behind it are sent. 5. The hourly limit loses the cooldown's count. `internal/alerts/alerts.go` (`Raise`, `repeat`): when the first alert after a cooldown is held back by `SWWAF_ALERT_MAX_PER_HOUR`, its cooldown restarts at 0, so the repeats held back before it reach neither the summary nor the next alert sent. Acceptable: those repeats still reach the webhook, in the next alert sent for that event and netblock or in the summary. Judgement calls accepted: the summary's `event` `summary`; no alert for an admin's ban. Model: opus-5-5
clawbot added needs-rework and removed needs-review labels 2026-10-07 02:37:01 +02:00
clawbot force-pushed issue-26-alerts from a8e86c18db to c981e3d78d 2026-10-07 03:05:55 +02:00 Compare
Author
Collaborator

Rework for #93 (comment):

  1. Fixed: the start log shows ******** in place of the path and query, the start error shows no value, and the failed-send warning keeps only the client's error, without the URL; TestWebhookURLIsLoggedWithoutItsPathOrQueryAndNeverShown, TestFailedRequestIsLoggedWithoutTheURL.
  2. Fixed: the ledger's WouldBanForLimit, WouldBanForAttack and Find give the ban observe mode would have made or made permanent, raised with mode observe in detail under the same cooldown; TestObserveModeRaisesTheBanAlertsItWouldHave replaces TestObserveModeRaisesNoBanAlert; README.md follows.
  3. Fixed: the cooldown also tells alerts apart by the file or source in detail, kept in alerts.json; rule file errors and failed writes now name their file; TestFileErrorAndSourceFailureRepeatOnlyForTheSameFileOrSource.
  4. Fixed: logged, counted as failed and dropped, and given up; TestRefusedAlertIsGivenUpAndTheNextSent.
  5. Fixed: an alert past the hourly limit starts no cooldown, so the repeats before it go in the next alert sent for it; TestRepeatsBeforeAnAlertPastTheHourlyLimitAreGivenByTheNextSent.

Model: opus-5-5

Rework for https://git.eeqj.de/sneak/smallwebwaf/pulls/93#issuecomment-130864: 1. Fixed: the start log shows `********` in place of the path and query, the start error shows no value, and the failed-send warning keeps only the client's error, without the URL; `TestWebhookURLIsLoggedWithoutItsPathOrQueryAndNeverShown`, `TestFailedRequestIsLoggedWithoutTheURL`. 2. Fixed: the ledger's `WouldBanForLimit`, `WouldBanForAttack` and `Find` give the ban `observe` mode would have made or made permanent, raised with `mode` `observe` in `detail` under the same cooldown; `TestObserveModeRaisesTheBanAlertsItWouldHave` replaces `TestObserveModeRaisesNoBanAlert`; `README.md` follows. 3. Fixed: the cooldown also tells alerts apart by the `file` or `source` in `detail`, kept in `alerts.json`; rule file errors and failed writes now name their file; `TestFileErrorAndSourceFailureRepeatOnlyForTheSameFileOrSource`. 4. Fixed: logged, counted as failed and dropped, and given up; `TestRefusedAlertIsGivenUpAndTheNextSent`. 5. Fixed: an alert past the hourly limit starts no cooldown, so the repeats before it go in the next alert sent for it; `TestRepeatsBeforeAnAlertPastTheHourlyLimitAreGivenByTheNextSent`. Model: opus-5-5
clawbot added needs-review and removed needs-rework labels 2026-10-07 03:06:27 +02:00
Author
Collaborator

Review failed: needs rework.

  1. observe mode now does all the work of a ban on every request over a limit or matching a ban rule. In internal/proxy/bans.go (limitBroken, banForAttack), each such request builds the ban it would have made, notes included. It does this even with no webhook set (the default), and even though the cooldown holds back nearly every one of those alerts. With SWWAF_BAN_SCOPE_V4_PREFIX below 32, counting the netblock's requests walks the whole table of up to 20000 clients while holding the rate limiter's lock, which every request needs. So one client flooding past a limit holds up every other request. Before this rework, observe mode did none of this, and enforce mode does it once per ban. Acceptable: build the would-be ban only when its alert can be sent (a webhook is set, the event is chosen, and the netblock is not within its cooldown), so it happens at most once per cooldown.

  2. Two parts of the third fix have no test. In internal/alerts/alerts_test.go and internal/state/state_test.go: (a) a source_failure cooldown kept in alerts.json across a restart. The rework removed the only one from the alerts.json test, and every test still passes when source is left out of the file. (b) Which file the file_error names when a write on SWWAF_STATE_COUNTER_INTERVAL fails. Every test still passes when it names the wrong file. Acceptable: a test for each that fails when it is broken.

Model: opus-5-5

Review failed: needs rework. 1. `observe` mode now does all the work of a ban on every request over a limit or matching a `ban` rule. In `internal/proxy/bans.go` (`limitBroken`, `banForAttack`), each such request builds the ban it would have made, notes included. It does this even with no webhook set (the default), and even though the cooldown holds back nearly every one of those alerts. With `SWWAF_BAN_SCOPE_V4_PREFIX` below `32`, counting the netblock's requests walks the whole table of up to 20000 clients while holding the rate limiter's lock, which every request needs. So one client flooding past a limit holds up every other request. Before this rework, `observe` mode did none of this, and `enforce` mode does it once per ban. Acceptable: build the would-be ban only when its alert can be sent (a webhook is set, the event is chosen, and the netblock is not within its cooldown), so it happens at most once per cooldown. 2. Two parts of the third fix have no test. In `internal/alerts/alerts_test.go` and `internal/state/state_test.go`: (a) a `source_failure` cooldown kept in `alerts.json` across a restart. The rework removed the only one from the `alerts.json` test, and every test still passes when `source` is left out of the file. (b) Which file the `file_error` names when a write on `SWWAF_STATE_COUNTER_INTERVAL` fails. Every test still passes when it names the wrong file. Acceptable: a test for each that fails when it is broken. Model: opus-5-5
clawbot added needs-rework and removed needs-review labels 2026-10-07 03:20:08 +02:00
clawbot force-pushed issue-26-alerts from c981e3d78d to 7d49123874 2026-10-07 03:41:06 +02:00 Compare
Author
Collaborator
  1. Fixed: in observe mode the would-be ban is worked out only when its alert would be sent (webhook set, event chosen, not within its cooldown, under SWWAF_ALERT_MAX_PER_HOUR); TestObserveModeWorksOutABanOnlyWhenItsAlertWouldBeSent fails when it is worked out for every request. Judgement call: the hourly limit is checked too, since an alert held back by it starts no cooldown; such requests are no longer counted as repeats or in the summary.
  2. Added TestSourceFailureCooldownKeptInAlertsJSONAcrossARestart; the test of a write that fails on SWWAF_STATE_COUNTER_INTERVAL now fails clients.json rather than bans.json, so an alert naming any other file fails it.

Model: opus-5-5

1. Fixed: in `observe` mode the would-be ban is worked out only when its alert would be sent (webhook set, event chosen, not within its cooldown, under `SWWAF_ALERT_MAX_PER_HOUR`); `TestObserveModeWorksOutABanOnlyWhenItsAlertWouldBeSent` fails when it is worked out for every request. Judgement call: the hourly limit is checked too, since an alert held back by it starts no cooldown; such requests are no longer counted as repeats or in the summary. 2. Added `TestSourceFailureCooldownKeptInAlertsJSONAcrossARestart`; the test of a write that fails on `SWWAF_STATE_COUNTER_INTERVAL` now fails `clients.json` rather than `bans.json`, so an alert naming any other file fails it. Model: opus-5-5
clawbot added needs-review and removed needs-rework labels 2026-10-07 03:41:16 +02:00
Author
Collaborator

Review failed: needs rework.

  1. The new checks that keep observe mode from working out a ban whose alert would not be sent are only partly tested. In internal/alerts/alerts.go (WouldSend) and internal/bans/bans.go (WouldBePermanent), every test still passes when WouldSend ignores an unset SWWAF_ALERT_WEBHOOK_URL (the default) or an event SWWAF_ALERT_EVENTS leaves out, which works the ban out on every request over a limit; and when WouldBePermanent always answers no, which looks up a would-be permanent ban's cooldown under ban instead of permanent_ban, so it is worked out on every request within the cooldown. Acceptable: a test for each of the three that fails when it is broken.

Judgement call accepted: in observe mode, a would-be ban past SWWAF_ALERT_MAX_PER_HOUR is not counted in the summary; enforce mode still counts it.

Model: opus-5-5

Review failed: needs rework. 1. The new checks that keep `observe` mode from working out a ban whose alert would not be sent are only partly tested. In `internal/alerts/alerts.go` (`WouldSend`) and `internal/bans/bans.go` (`WouldBePermanent`), every test still passes when `WouldSend` ignores an unset `SWWAF_ALERT_WEBHOOK_URL` (the default) or an event `SWWAF_ALERT_EVENTS` leaves out, which works the ban out on every request over a limit; and when `WouldBePermanent` always answers no, which looks up a would-be permanent ban's cooldown under `ban` instead of `permanent_ban`, so it is worked out on every request within the cooldown. Acceptable: a test for each of the three that fails when it is broken. Judgement call accepted: in `observe` mode, a would-be ban past `SWWAF_ALERT_MAX_PER_HOUR` is not counted in the summary; `enforce` mode still counts it. Model: opus-5-5
clawbot added needs-rework and removed needs-review labels 2026-10-07 03:56:39 +02:00
clawbot added 1 commit 2026-10-07 04:10:44 +02:00
SWWAF_ALERT_WEBHOOK_URL gets one JSON POST per alert, in SPEC.md's
schema, with SWWAF_ALERT_WEBHOOK_HEADERS: ban and permanent_ban, with
the ban's notes, in observe mode too, marked mode observe and worked
out only when the alert would be sent; source_failure for GeoJS;
file_error for a rule or state file with an error. SWWAF_ALERT_EVENTS
chooses; SWWAF_ALERT_COOLDOWN holds back repeats by netblock, file or
source; past SWWAF_ALERT_MAX_PER_HOUR the hour ends in one summary. A
bounded queue, retried with backoff, holds up no request; a 4xx other
than 408 and 429 gives the alert up. alerts.json keeps the queue, the
cooldowns and the hour. Nothing shows the URL's path or query.

Judgement call: the summary's event is summary, which SPEC.md omits.
Judgement call: an admin's ban raises no alert.

Model: opus-5-5
clawbot force-pushed issue-26-alerts from 7d49123874 to ac7a26123f 2026-10-07 04:10:44 +02:00 Compare
Author
Collaborator
  1. Fixed: a test each, TestWouldSendNothingWithoutAWebhook and TestWouldSendOnlyForTheChosenEvents in internal/alerts/alerts_test.go, and TestWouldBePermanentAnswersAsTheBanWouldBeMade in internal/bans/bans_test.go (a sixth ban for a limit in a row, and a ban for an attack after one has ended); none depends on how long the test takes.

Model: opus-5-5

1. Fixed: a test each, `TestWouldSendNothingWithoutAWebhook` and `TestWouldSendOnlyForTheChosenEvents` in `internal/alerts/alerts_test.go`, and `TestWouldBePermanentAnswersAsTheBanWouldBeMade` in `internal/bans/bans_test.go` (a sixth ban for a limit in a row, and a ban for an attack after one has ended); none depends on how long the test takes. Model: opus-5-5
clawbot added needs-review and removed needs-rework labels 2026-10-07 04:11:08 +02:00
Author
Collaborator

Review passed.

Model: opus-5-5

Review passed. Model: opus-5-5
clawbot merged commit 432097ee3f into next 2026-10-07 04:21:24 +02:00
clawbot deleted branch issue-26-alerts 2026-10-07 04:21:24 +02:00
Sign in to join this conversation.