sneak/smallwebwaf
1
0
Fork 0
Code Issues 44 Pull Requests 1 Actions 45 Packages Projects Releases Wiki Activity

Alerts to Slack and ntfy, each destination with its own queue (closes #90)
check / check (push) Waiting to run
Details

Browse Source 
Each alert is posted as a message to the Slack incoming webhook
SWWAF_ALERT_SLACK_WEBHOOK_URL names, and published to the ntfy topic
SWWAF_ALERT_NTFY_URL names, with SWWAF_ALERT_NTFY_TOKEN as a bearer
token and a priority and tag by event. The cooldown and the hourly
limit stay shared; past them, each destination has its own bounded
queue and backoff, and its own sent, failed and dropped counts.
alerts.json keeps the alerts waiting by destination; one whose waiting
is still a list stops the start, saying what to change. A control
character in the ntfy token, or in the instance name ntfy is sent,
stops the start.

Judgement call: messages also give the detail's file, source, error and mode.
Judgement call: alerts_suppressed_total is the same for every destination.

Model: opus-5-5
This commit is contained in:
clawbot
2026-10-07 03:41:01 +00:00
parent 432097ee3f
commit cd04ec0b35
14 changed files with 1586 additions and 428 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+126 -58
View File
@@ -19,21 +19,21 @@ ledger with the bans you make, keep and lift, the JSON state files with your
edits taken in while it runs and the paths the rate limits do not count, which
come next in the build order, `observe` mode and the rest of the request log's
fields, which come a little later, and the metrics endpoint and the header size
and the idle time as settings, which come last in it. So are four parts of the
stage after it: the rule files, the first part, with the bans for a clear sign
of attack, the other admin endpoints, the second, alerts to a JSON webhook, the
first of the three destinations alerts go to, and remote log sending.
`smallwebwaf` passes each request to the app and the app's answer back,
unchanged, within its timeouts and size limits, works out each client's address,
bans a client that sends too many requests, not counting those for the paths you
choose, refuses a client that comes from a country you refuse or from a network
you refuse, lets the networks you choose through, checks each request against
the rule files and bans a client whose request is a clear sign of attack, keeps
its bans, each client's counters and history, and GeoJS's answers in JSON files
across restarts, takes in your edits of those files, such as a ban you make,
keep or lift, and of the rule files while it runs, writes a JSON log line for
every request, sends its log lines to a syslog server too if you name one, sends
an alert to a webhook you name for each ban it makes or makes permanent, for
and the idle time as settings, which come last in it. So are the four parts of
the stage after it: the rule files, with the bans for a clear sign of attack,
the other admin endpoints, alerts to all three destinations, a JSON webhook,
Slack and ntfy, and remote log sending. `smallwebwaf` passes each request to the
app and the app's answer back, unchanged, within its timeouts and size limits,
works out each client's address, bans a client that sends too many requests, not
counting those for the paths you choose, refuses a client that comes from a
country you refuse or from a network you refuse, lets the networks you choose
through, checks each request against the rule files and bans a client whose
request is a clear sign of attack, keeps its bans, each client's counters and
history, and GeoJS's answers in JSON files across restarts, takes in your edits
of those files, such as a ban you make, keep or lift, and of the rule files
while it runs, writes a JSON log line for every request, sends its log lines to
a syslog server too if you name one, sends an alert to a webhook, to Slack and
to ntfy, each if you name one, for each ban it makes or makes permanent, for
GeoJS failing and for a rule file or state file with an error, serves Prometheus
metrics to a scraper that holds the metrics token, lets an admin who holds the
admin token list, add and lift bans and ask what it knows of a client, and in
@@ -190,10 +190,12 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
- Sends every line it writes on stdout to a syslog server as well, while
`SWWAF_LOG_REMOTE_URL` names one (see "Sending the log to a syslog server"
below).
- Sends an alert, as a JSON object, to the webhook `SWWAF_ALERT_WEBHOOK_URL`
names, while it names one, for each ban it makes or makes permanent, for GeoJS
failing, and for a rule file or state file with an error, holding back repeats
and, past an hourly limit, rolling the rest into one summary (see "Alerts"
- Sends an alert for each ban it makes or makes permanent, for GeoJS failing,
and for a rule file or state file with an error, holding back repeats and,
past an hourly limit, rolling the rest into one summary, to each destination
you name: as a JSON object to the webhook `SWWAF_ALERT_WEBHOOK_URL` names, as
a message to the Slack incoming webhook `SWWAF_ALERT_SLACK_WEBHOOK_URL` names,
and as a message to the ntfy topic `SWWAF_ALERT_NTFY_URL` names (see "Alerts"
below).
## Settings
@@ -340,13 +342,29 @@ effective settings are logged at start.
- `SWWAF_ALERT_WEBHOOK_URL` (default unset): the webhook each alert is posted
to, an `http` or `https` URL without a user or a fragment, such as
`https://alerts.example/smallwebwaf` (see "Alerts" below). Unset or empty, no
alert is sent. Since many webhooks carry their secret in the path or the
query, the settings logged at start show `********` in place of them, and a
value that stops the start is not shown.
alert is sent to a webhook. Since many webhooks carry their secret in the path
or the query, the settings logged at start show `********` in place of them,
and a value that stops the start is not shown.
- `SWWAF_ALERT_WEBHOOK_HEADERS` (default empty): headers sent with each alert,
such as one that authenticates it, as a list of a name, `:` and a value, such
as `Authorization:Bearer 0123456789abcdef`. A value cannot hold a comma. The
settings logged at start show `********` in place of each value.
- `SWWAF_ALERT_SLACK_WEBHOOK_URL` (default unset): the Slack incoming webhook
each alert is posted to as a message, such as
`https://hooks.slack.com/services/T0123/B4567/abcdef`. Unset or empty, no
alert is sent to Slack. It is checked and logged as `SWWAF_ALERT_WEBHOOK_URL`
is.
- `SWWAF_ALERT_NTFY_URL` (default unset): the ntfy topic each alert is published
to, as the topic's full URL, such as `https://ntfy.sh/my-alerts`. Unset or
empty, no alert is sent to ntfy. It is checked and logged as
`SWWAF_ALERT_WEBHOOK_URL` is, since anyone who knows a topic on a server open
to all can read it.
- `SWWAF_ALERT_NTFY_TOKEN` (default unset): an ntfy access token, sent to ntfy
with each alert as `Authorization: Bearer <token>`, for a topic that needs
one. The settings logged at start show `********` in its place. A control
character in it, such as the carriage return of a file saved with Windows line
ends, stops the start, and while `SWWAF_ALERT_NTFY_URL` is set, so does one in
`SWWAF_INSTANCE_NAME`, which ntfy is sent in the title.
- `SWWAF_ALERT_EVENTS` (default
`ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure,file_error`):
the events alerts are sent for. `waf_block`, `anomaly` and `reputation_hit`
@@ -532,11 +550,13 @@ them.
## Alerts
While `SWWAF_ALERT_WEBHOOK_URL` is set, `smallwebwaf` posts each alert to it as
one JSON object, with `Content-Type: application/json` and the headers
`SWWAF_ALERT_WEBHOOK_HEADERS` gives, as "Alert webhook schema" in
[`SPEC.md`](SPEC.md) describes. An alert is for one of these events, and is sent
when `SWWAF_ALERT_EVENTS` names its event:
`smallwebwaf` sends each alert to each destination you name: to
`SWWAF_ALERT_WEBHOOK_URL` it posts the alert as one JSON object, with
`Content-Type: application/json` and the headers `SWWAF_ALERT_WEBHOOK_HEADERS`
gives, as "Alert webhook schema" in [`SPEC.md`](SPEC.md) describes, and to
`SWWAF_ALERT_SLACK_WEBHOOK_URL` and `SWWAF_ALERT_NTFY_URL` a message made from
it, as below. An alert is for one of these events, and is sent when
`SWWAF_ALERT_EVENTS` names its event:
- `ban`: a ban `smallwebwaf` makes, for a broken rate limit or a clear sign of
attack.
@@ -612,6 +632,47 @@ is sent on one line:
- `suppressed_repeats` is how many repeats the cooldown held back before this
alert.
Slack and ntfy are each sent the alert as a message: a title, the instance and
the event, such as `fsn1app1/gitea: ban`, and a text, the `reason`, then a line
for each of the `client`, the `netblock` and the `country`, the `file`, the
`source`, the `error` and the `mode` the `detail` gives, and the
`suppressed_repeats`, leaving out those that are empty or 0. Slack is posted, as
JSON, the title in bold and the text below it, with `&`, `<` and `>` escaped, so
that nothing in them is read as a link or a mention. ntfy is posted the text,
with the title as `Title`, `SWWAF_ALERT_NTFY_TOKEN`, while it is set, as
`Authorization: Bearer <token>`, and the priority and the tag, which ntfy shows
as an emoji, of the alert's event, as `Priority` and `Tags`:
| Event | Priority | Tag |
| ------------------------------ | --------- | -------------------------- |
| `ban` | `default` | `no_entry` |
| `permanent_ban` | `high` | `no_entry` |
| `waf_block` | `default` | `shield` |
| `anomaly` | `high` | `chart_with_upwards_trend` |
| `reputation_hit` | `low` | `label` |
| `source_failure`, `file_error` | `high` | `warning` |
| `summary` | `default` | `bar_chart` |
For the alert above, ntfy is sent these headers and this text:
```text
Title: fsn1app1/gitea: ban
Priority: default
Tags: no_entry
requests per minute over the limit of 1000
client: 203.0.113.9
netblock: 203.0.113.9/32
```
and Slack this JSON object, shown indented; it is sent on one line:
```json
{
"text": "*fsn1app1/gitea: ban*\nrequests per minute over the limit of 1000\nclient: 203.0.113.9\nnetblock: 203.0.113.9/32"
}
```
An alert for the same event as the last one sent, on the same netblock, or for a
`file_error` about the same file, or for a `source_failure` about the same
source, less than `SWWAF_ALERT_COOLDOWN` after it, is a repeat: it is held back
@@ -626,18 +687,19 @@ were held back, and its `detail` gives the `hour` as when it started, the
starts no cooldown, and the repeats held back before it are given by the next
alert sent for the same event and netblock, file or source.
The alerts wait in a queue of at most 1000, from which they are sent one at a
time, the oldest first, so a webhook that is slow or down never holds up a
request. The webhook takes an alert by answering with a 2xx status, and refuses
it with a 4xx status other than `408` and `429`: a refused alert is logged,
counted as dropped, and given up, so that the next is sent. Any other answer, a
redirect included, a connection that fails, or no answer within 10 seconds is a
failure: it is logged, without the webhook's URL, and the alert is sent again a
second later, twice as long after each further failure in a row, up to a minute.
With 1000 alerts waiting, the oldest is dropped to make room for a new one. The
cooldowns, the hour under way and the alerts still waiting are kept in
`alerts.json` (see "State files" below), so that after a restart the alerts
waiting are sent, and the cooldowns go on.
Each destination has a queue of its own, of at most 1000 alerts, from which they
are sent to it one at a time, the oldest first, so a destination that is slow or
down holds up neither the others nor any request. A destination takes an alert
by answering with a 2xx status, and refuses it with a 4xx status other than
`408` and `429`: a refused alert is logged, counted as dropped, and given up, so
that the next is sent. Any other answer, a redirect included, a connection that
fails, or no answer within 10 seconds is a failure: it is logged, naming the
destination's setting and not its URL, and the alert is sent again a second
later, twice as long after each further failure in a row, up to a minute. With
1000 alerts waiting for a destination, the oldest is dropped to make room for a
new one. The cooldowns, the hour under way and the alerts still waiting for each
destination are kept in `alerts.json` (see "State files" below), so that after a
restart the alerts waiting are sent, and the cooldowns go on.
## State files
@@ -668,9 +730,13 @@ entries by client address, but for the alerts waiting, with times in UTC.
`source`, or event alone, when the last alert was sent, `sent`, and the
repeats held back since, `suppressed_repeats`; under `hour`, the hour under
way, from its `start`, the alerts `sent` in it and those `held_back` for its
summary, by event; and under `waiting`, the alerts still waiting to be sent,
the oldest first, each as the webhook is sent it. As an hour ends, the
cooldowns that have run out with no repeat held back are dropped.
summary, by event; and under `waiting`, for each destination you name,
`webhook`, `slack` or `ntfy`, the alerts still waiting to be sent to it, the
oldest first, each as the webhook is sent it. As an hour ends, the cooldowns
that have run out with no repeat held back are dropped. As the file is read,
the alerts waiting for a destination you no longer name are dropped. A file
whose `waiting` is a list, as it was before alerts went to Slack and ntfy too,
stops the start: put the list under `"webhook"`, or remove the file.
`bans.json` is written `SWWAF_STATE_WRITE_DELAY` after a ban is made, lifted
through `DELETE /_smallwebwaf/bans/<client>`, or made permanent, with every such
@@ -695,8 +761,9 @@ without a field it needs, named with the entry's place in the file: a ban's
client's `client`, or the `start` of a window in which it has requests; an
answer's `client`, `country`, which is `""` for a client GeoJS cannot place, or
`answered`; a cooldown's `event` or `sent`; an alert waiting's `event` or
`time`. So does a ban whose `cause` is not `limit`, `attack` or `admin`. The AS
number and AS name come with their lookup.
`time`. So does a ban whose `cause` is not `limit`, `attack` or `admin`, and
alerts waiting for a destination that is not `webhook`, `slack` or `ntfy`. The
AS number and AS name come with their lookup.
While it runs, `smallwebwaf` watches `SWWAF_STATE_DIR` and takes in your edit of
a state file as soon as you save it: what the file then holds replaces what
@@ -705,14 +772,14 @@ yours by comparing the file with what it last read or wrote, and before it
writes a file it takes in any edit made since, so your edit is not overwritten;
a change `smallwebwaf` made after you opened the file, such as a new ban, is
lost when you save over it. An edit that would stop the start, because it does
not parse, has another `version`, leaves out a field an entry needs or gives a
ban another `cause`, does not stop the running `smallwebwaf`: it keeps what it
holds, and at the file's next write renames your file to `<name>.bad`, such as
`bans.json.bad`, writes the file again from memory, logs the file and where the
error is, and raises a `file_error` alert for it. It waits for that write
because an editor's file can be read before the editor has finished writing it.
Mend the `.bad` file and move it back. A file you remove is written again at its
next write.
not parse, has another `version`, leaves out a field an entry needs, gives a ban
another `cause` or names another destination, does not stop the running
`smallwebwaf`: it keeps what it holds, and at the file's next write renames your
file to `<name>.bad`, such as `bans.json.bad`, writes the file again from
memory, logs the file and where the error is, and raises a `file_error` alert
for it. It waits for that write because an editor's file can be read before the
editor has finished writing it. Mend the `.bad` file and move it back. A file
you remove is written again at its next write.
To ban a netblock, add an entry to `bans.json` with its `netblock`, its `start`
and its `expires`, `null` for a ban that never ends; its `reason` and its
@@ -876,12 +943,13 @@ other request. No metric carries a client's address.
`smallwebwaf_remote_log_lines_dropped_total`: those dropped, from a full
buffer or because their sending failed; and
`smallwebwaf_remote_log_buffer_depth`: those waiting in the buffer.
- While `SWWAF_ALERT_WEBHOOK_URL` is set, by `destination`, `webhook`:
`smallwebwaf_alerts_sent_total`: the alerts the webhook took;
- For each destination you name, by `destination`, `webhook`, `slack` or `ntfy`:
`smallwebwaf_alerts_sent_total`: the alerts it took;
`smallwebwaf_alerts_failed_total`: the requests to it that failed;
`smallwebwaf_alerts_suppressed_total`: the alerts held back, as repeats or for
an hour's summary; and `smallwebwaf_alerts_dropped_total`: those dropped from
a full queue, or given up as the webhook refused them.
an hour's summary, which are the same for every destination; and
`smallwebwaf_alerts_dropped_total`: those dropped from its full queue, or
given up as it refused them.
- Go's own `go_` metrics and the process's `process_` metrics.
The requests Go's HTTP server ends before `smallwebwaf` sees them (see "Request
@@ -1259,8 +1327,8 @@ addresses are never sent to GeoJS.
standard library alone, whose `log/syslog` writes only the older syslog
format.
- `internal/alerts`: takes the alerts the other parts raise, holds back repeats
and those past the hourly limit, and sends the others to
`SWWAF_ALERT_WEBHOOK_URL` from a queue of its own.
and those past the hourly limit, and sends the others to the webhook, Slack
and ntfy, each from a queue of its own.
- `Dockerfile`: the lint and test phases, then the image, whose last stage
installs Ubuntu's packages, nixpkgs, `runsvinit` and `smallwebwaf`, with
`share/smallwebwaf.run` as runit's `run` script for `smallwebwaf` and