sneak/smallwebwaf
1
0
Fork 0
Code Issues 44 Pull Requests 2 Actions 47 Packages Projects Releases Wiki Activity

Anomaly thresholds: alerts for unusual traffic, nothing refused (closes #101)
check / check (push) Waiting to run
Details

Browse Source 
SWWAF_ANOMALY_CLIENT_*, _NET_*, _ASN_*, _TOTAL_* and SWWAF_WATCH_* with
SWWAF_WATCH_NETS: requests and bytes per minute and per hour, each off by
default; with all off, nothing is counted. Otherwise every request but the
health check is counted, allow-listed and exempt ones included; a count
over its threshold raises an anomaly alert, with a cooldown per scope. At
most 20,000 counters, kept in alerts.json. A per-AS-number threshold with
lookups off, or a malformed SWWAF_WATCH_NETS, stops the start. A cooldown
that has run out is dropped as the hour ends, whatever it held back; the
hour's summary gives its repeats.

Judgement call: refused requests are counted too.
Judgement call: per-client counters are kept in alerts.json, which SPEC.md does not list.
Judgement call: a request counts for an AS number only if the lookup answered before it ended.

Model: opus-5-5
This commit was merged in pull request #107.
This commit is contained in:
clawbot
2026-10-07 16:12:19 +02:00
parent 2421cdc273
commit 82e20e0cb5
17 changed files with 2132 additions and 297 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+204 -115
View File
@@ -22,32 +22,33 @@ fields, which come a little later, and the metrics endpoint and the header size
and the idle time as settings, which come last in it. So are the four parts of
the stage after it: the rule files, with the bans for a clear sign of attack,
the other admin endpoints, alerts to all three destinations, a JSON webhook,
Slack and ntfy, and remote log sending. So are three parts of the stage after
that: the AS number and country of every client, looked up through GeoJS or in
the IPinfo Lite database file, the byte limits, and the biased thresholds, lower
limits for the AS numbers and countries you list. `smallwebwaf` passes each
request to the app and the app's answer back, unchanged, within its timeouts and
size limits, works out each client's address, looks up its AS number and country
unless you switch that off, bans a client that sends too many requests or too
many bytes, not counting those for the paths you choose, with lower limits for
the clients of the AS numbers and countries you list, refuses a client that
comes from a country you refuse or from a network you refuse, lets the networks
you choose through, checks each request against the rule files and bans a client
whose request is a clear sign of attack, keeps its bans, each client's counters
and history, and GeoJS's answers in JSON files across restarts, takes in your
edits of those files, such as a ban you make, keep or lift, and of the rule
files while it runs, writes a JSON log line for every request, sends its log
lines to a syslog server too if you name one, sends an alert to a webhook, to
Slack and to ntfy, each if you name one, for each ban it makes or makes
permanent, for GeoJS failing, for a rule file or state file with an error and
for a replacement of the lookup database it cannot read, serves Prometheus
metrics to a scraper that holds the metrics token, lets an admin who holds the
admin token list, add and lift bans and ask what it knows of a client, and in
`observe` mode passes on the requests it would refuse, logging what it would
have done with them. It comes as the image the app's own image is built on. The
rest of the design comes after that, in the order of the build order in
[`SPEC.md`](SPEC.md). The survey of existing tools that led to the design is in
[`EVALUATION.md`](EVALUATION.md).
Slack and ntfy, and remote log sending. So is the stage after that: the AS
number and country of every client, looked up through GeoJS or in the IPinfo
Lite database file, the byte limits, the biased thresholds, lower limits for the
AS numbers and countries you list, and the anomaly thresholds, alerts for
unusual traffic that refuse nothing. `smallwebwaf` passes each request to the
app and the app's answer back, unchanged, within its timeouts and size limits,
works out each client's address, looks up its AS number and country unless you
switch that off, bans a client that sends too many requests or too many bytes,
not counting those for the paths you choose, with lower limits for the clients
of the AS numbers and countries you list, refuses a client that comes from a
country you refuse or from a network you refuse, lets the networks you choose
through, checks each request against the rule files and bans a client whose
request is a clear sign of attack, keeps its bans, each client's counters and
history, and GeoJS's answers in JSON files across restarts, takes in your edits
of those files, such as a ban you make, keep or lift, and of the rule files
while it runs, writes a JSON log line for every request, sends its log lines to
a syslog server too if you name one, sends an alert to a webhook, to Slack and
to ntfy, each if you name one, for each ban it makes or makes permanent, for
traffic over an anomaly threshold you set, for GeoJS failing, for a rule file or
state file with an error and for a replacement of the lookup database it cannot
read, serves Prometheus metrics to a scraper that holds the metrics token, lets
an admin who holds the admin token list, add and lift bans and ask what it knows
of a client, and in `observe` mode passes on the requests it would refuse,
logging what it would have done with them. It comes as the image the app's own
image is built on. The rest of the design comes after that, in the order of the
build order in [`SPEC.md`](SPEC.md). The survey of existing tools that led to
the design is in [`EVALUATION.md`](EVALUATION.md).
## Getting started
@@ -177,13 +178,13 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
IPinfo Lite database file while `SWWAF_LOOKUP_SOURCE` is `file`, after the
static lists and bans, unless `SWWAF_LOOKUP_SOURCE` is `off` (see "Country and
AS number lookup" below), for the request log, the client's history, the notes
of its bans, their alerts and the metrics. The file answers at once. With
GeoJS, a request waits for its client's first answer only while a setting acts
on it, a country list, `SWWAF_ADD_LOOKUP_HEADERS` or a biased threshold that
lowers a limit. Otherwise it goes on at once, and the answer reaches the
client's history and the notes of its bans when it comes, but not the log
lines of the requests that went on without it, nor the alerts already raised
for those bans.
of its bans, their alerts, the metrics and the anomaly thresholds per AS
number. The file answers at once. With GeoJS, a request waits for its client's
first answer only while a setting acts on it, a country list,
`SWWAF_ADD_LOOKUP_HEADERS` or a biased threshold that lowers a limit.
Otherwise it goes on at once, and the answer reaches the client's history and
the notes of its bans when it comes, but not the log lines of the requests
that went on without it, nor the alerts already raised for those bans.
- Refuses a request from a country you refuse with `SWWAF_BAN_RESPONSE`, as soon
as the client's country is known and before its body is read; such a request
is not counted for the rate limits. A client on a private, loopback or
@@ -237,13 +238,30 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
- Sends every line it writes on stdout to a syslog server as well, while
`SWWAF_LOG_REMOTE_URL` names one (see "Sending the log to a syslog server"
below).
- Sends an alert for each ban it makes or makes permanent, for GeoJS failing,
for a rule file or state file with an error, and for a replacement of the
lookup database it cannot read, holding back repeats and, past an hourly
limit, rolling the rest into one summary, to each destination you name: as a
JSON object to the webhook `SWWAF_ALERT_WEBHOOK_URL` names, as a message to
the Slack incoming webhook `SWWAF_ALERT_SLACK_WEBHOOK_URL` names, and as a
message to the ntfy topic `SWWAF_ALERT_NTFY_URL` names (see "Alerts" below).
- Sends an alert for each ban it makes or makes permanent, for a count over an
anomaly threshold, for GeoJS failing, for a rule file or state file with an
error, and for a replacement of the lookup database it cannot read, holding
back repeats and, past an hourly limit, rolling the rest into one summary, to
each destination you name: as a JSON object to the webhook
`SWWAF_ALERT_WEBHOOK_URL` names, as a message to the Slack incoming webhook
`SWWAF_ALERT_SLACK_WEBHOOK_URL` names, and as a message to the ntfy topic
`SWWAF_ALERT_NTFY_URL` names (see "Alerts" below).
- Counts requests and their bytes over a minute and an hour, per client, per
netblock around a client, per AS number, for the whole service and per named
netblock, and sends an `anomaly` alert for a count over the anomaly threshold
you set for it (see the anomaly thresholds below). These thresholds only
alert: they refuse and ban nothing. A scope whose four thresholds are all off
is not counted, and within a scope only the counts whose threshold is set are
counted. Every request but the health check is counted, whatever is done with
it: one that is refused, one from a client in `SWWAF_ALLOW_NETS` or
`SWWAF_RATE_LIMIT_EXEMPT_NETS`, and one for a path in
`SWWAF_RATE_LIMIT_EXEMPT_PATHS`, with its body bytes once it has ended, those
of the answer, of the request or both, as `SWWAF_BYTES_COUNT` says. A request
is counted for its client's AS number only if the lookup has given that by the
time the request ends: no request waits for it, and a client in
`SWWAF_ALLOW_NETS`, which is not looked up, counts for no AS number. At most
20,000 counters are kept, the one counted least recently dropped first, and
`alerts.json` keeps them across a restart (see "State files" below).
## Settings
@@ -330,9 +348,10 @@ effective settings are logged at start.
address of every new visitor, `file`, the IPinfo Lite database file
`SWWAF_LOOKUP_DB_PATH` names, or `off`, which looks up no client and sends no
address to GeoJS. With `off`, a country list that is not empty,
`SWWAF_ADD_LOOKUP_HEADERS` set to `true`, or a biased threshold that lowers a
`SWWAF_ADD_LOOKUP_HEADERS` set to `true`, a biased threshold that lowers a
limit, a list of them that is not empty or `SWWAF_UNKNOWN_LIMIT_PERCENT` below
100, stops the start, with a message naming it and `SWWAF_LOOKUP_SOURCE`.
100, or an anomaly threshold per AS number that is not `off`, stops the start,
with a message naming it and `SWWAF_LOOKUP_SOURCE`.
- `SWWAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in its
`.mmdb` form, for `SWWAF_LOOKUP_SOURCE=file`. `file` without it, or it with
any other `SWWAF_LOOKUP_SOURCE`, the default included, stops the start, with a
@@ -467,16 +486,46 @@ effective settings are logged at start.
`SWWAF_INSTANCE_NAME`, which ntfy is sent in the title.
- `SWWAF_ALERT_EVENTS` (default
`ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure,file_error`):
the events alerts are sent for. `waf_block`, `anomaly` and `reputation_hit`
come with the features that raise them; nothing raises them yet.
the events alerts are sent for. `waf_block` and `reputation_hit` come with the
features that raise them; nothing raises them yet.
- `SWWAF_ALERT_COOLDOWN` (default `15m`): how long a repeat of an alert is held
back (see "Alerts" below).
- `SWWAF_ALERT_MAX_PER_HOUR` (default `60`): the most alerts sent in an hour;
the rest of the hour's alerts are rolled into one summary.
- `SWWAF_ANOMALY_CLIENT_REQUESTS_PER_MINUTE`,
`SWWAF_ANOMALY_CLIENT_REQUESTS_PER_HOUR`,
`SWWAF_ANOMALY_CLIENT_BYTES_PER_MINUTE` and
`SWWAF_ANOMALY_CLIENT_BYTES_PER_HOUR` (default `off`): the anomaly thresholds
per client, the most requests and the most bytes a client may have counted in
a minute and in an hour before an `anomaly` alert is sent for it. They refuse
and ban nothing. Each scope below has the same four thresholds, their names
ending in `REQUESTS_PER_MINUTE`, `REQUESTS_PER_HOUR`, `BYTES_PER_MINUTE` and
`BYTES_PER_HOUR`, and each is `off` by default, since what is unusual depends
on each service's normal traffic, which the metrics show.
- `SWWAF_ANOMALY_NET_REQUESTS_PER_MINUTE`, `..._PER_HOUR`,
`SWWAF_ANOMALY_NET_BYTES_PER_MINUTE` and `..._PER_HOUR` (default `off`): the
anomaly thresholds per netblock around a client, which is
`SWWAF_ANOMALY_NET_V4_PREFIX` (default `24`) long, from 0 to 32, for an IPv4
client, and `SWWAF_ANOMALY_NET_V6_PREFIX` (default `48`) long, from 0 to 128,
for an IPv6 one.
- `SWWAF_ANOMALY_ASN_REQUESTS_PER_MINUTE`, `..._PER_HOUR`,
`SWWAF_ANOMALY_ASN_BYTES_PER_MINUTE` and `..._PER_HOUR` (default `off`): the
anomaly thresholds per AS number.
- `SWWAF_ANOMALY_TOTAL_REQUESTS_PER_MINUTE`, `..._PER_HOUR`,
`SWWAF_ANOMALY_TOTAL_BYTES_PER_MINUTE` and `..._PER_HOUR` (default `off`): the
anomaly thresholds for the whole service.
- `SWWAF_WATCH_NETS` (default empty): named netblocks, each a name, `=` and a
netblock, such as `office=203.0.113.0/24,scraper-x=198.51.100.0/22`. An item
without `=`, without a name or without a valid netblock, or a name listed
twice, stops the start. `SWWAF_WATCH_REQUESTS_PER_MINUTE`, `..._PER_HOUR`,
`SWWAF_WATCH_BYTES_PER_MINUTE` and `..._PER_HOUR` (default `off`) are the
anomaly thresholds of each named netblock as a whole, which counts every
client inside it; a client inside several is counted in each.
Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are
bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024
bytes). Rate limits are whole numbers of requests, and byte limits are sizes.
bytes). Rate limits and the anomaly thresholds on requests are whole numbers of
requests, and byte limits and the anomaly thresholds on bytes are sizes.
Netblocks are in CIDR form, and a bare address stands for itself alone.
Countries are the two-letter codes ISO 3166-1 assigns today, and `xk` for
Kosovo, in either case (`de` and `DE` are the same); any other code, such as
@@ -485,15 +534,16 @@ code on both country lists. AS numbers are `AS` and the number, in either case.
Percentages are whole numbers from 0 to 100, and an entry of a list of them is
an AS number or a country, `:` and a percentage; an AS number or a country
listed twice in one of them stops the start. `off` switches a timeout, a size
limit, a rate limit, a byte limit, `SWWAF_ALERT_COOLDOWN` or
`SWWAF_ALERT_MAX_PER_HOUR` off; `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`,
limit, a rate limit, a byte limit, an anomaly threshold, `SWWAF_ALERT_COOLDOWN`
or `SWWAF_ALERT_MAX_PER_HOUR` off; `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`,
`SWWAF_LOOKUP_TIMEOUT`, `SWWAF_UNKNOWN_LIMIT_PERCENT`, the ban settings, the
state settings, `SWWAF_METRICS_TOP_N` and `SWWAF_LOG_REMOTE_BUFFER` cannot be
off.
state settings, `SWWAF_METRICS_TOP_N`, `SWWAF_LOG_REMOTE_BUFFER`,
`SWWAF_ANOMALY_NET_V4_PREFIX` and `SWWAF_ANOMALY_NET_V6_PREFIX` cannot be off.
Several limits are fixed rather than settings. At most 20,000 clients are kept,
with their counters and history, and an IPv6 client is counted by its /64. At
most 100,000 answers from GeoJS are kept, for 7 days each.
most 100,000 answers from GeoJS are kept, for 7 days each, and at most 20,000
anomaly counters.
### Settings given as files
@@ -684,6 +734,9 @@ it, as below. An alert is for one of these events, and is sent when
clear sign of attack.
- `permanent_ban`: a permanent ban it makes, or a ban for a clear sign of attack
that a request made permanent.
- `anomaly`: a count of requests or bytes over an anomaly threshold, raised by
each request that ends with the count over it, in `observe` mode as in
`enforce` mode. It refuses and bans nothing.
- `source_failure`: GeoJS failing or refusing `smallwebwaf`.
- `file_error`: a rule file edited while it runs that has an error, an edit of a
state file set aside as `<name>.bad`, a state file it could not write while
@@ -745,19 +798,29 @@ is sent on one line:
- `instance` is `SWWAF_INSTANCE_NAME`, and `time` when the alert was raised, in
UTC.
- `client` is the address of the client whose request raised the alert, and
`netblock` the netblock of the ban; both are empty for `source_failure` and
`netblock` the netblock of the ban, or for an `anomaly`, the netblock counted:
the client's own, the netblock around it or a named netblock, and none for an
AS number or the whole service; both are empty for `source_failure` and
`file_error`. `asn`, `as_name` and `country` are, for a ban, the client's as
the ban's notes give them when the alert is raised: empty, as in this alert,
when GeoJS had not answered about the client by then.
- `reason` is a short sentence; for a ban, the ban's `reason` in `bans.json`.
when GeoJS had not answered about the client by then; for an `anomaly`, the
client's as the lookup gave them by the time its request ended.
- `reason` is a short sentence; for a ban, the ban's `reason` in `bans.json`;
for an `anomaly`, what was counted over which threshold, such as
`requests per minute of the netblock 203.0.113.0/24 over the threshold of 1000`.
- `detail` is what is particular to the event: for a ban, its `cause`, when it
ends as `ban_expires`, in the form the request log gives it, and its `notes`,
as `bans.json` gives them; for `source_failure`, the `source`, `geojs`, the
`error`, and when GeoJS is asked again, `asking_again_in`; for `file_error`,
the `file`, which for an edit set aside is the file it was renamed to, and the
`error`, which for a file that does not parse names where in it the error is.
as `bans.json` gives them; for an `anomaly`, the `scope`, `client`, `net`,
`asn`, `total` or `watch`, as the settings name them, the `asn` counted for
`asn` and the `name` of the named netblock for `watch`, the `window`, `minute`
or `hour`, the `kind`, `requests` or `bytes`, the `count`, which is weighted
as the rate limits weigh theirs, and the `threshold`; for `source_failure`,
the `source`, `geojs`, the `error`, and when GeoJS is asked again,
`asking_again_in`; for `file_error`, the `file`, which for an edit set aside
is the file it was renamed to, and the `error`, which for a file that does not
parse names where in it the error is.
- `suppressed_repeats` is how many repeats the cooldown held back before this
alert.
alert, and for a `summary`, those no other alert gives (see below).
Slack and ntfy are each sent the alert as a message: a title, the instance and
the event, such as `fsn1app1/gitea: ban`, and a text, the `reason`, then a line
@@ -802,17 +865,21 @@ and Slack this JSON object, shown indented; it is sent on one line:
An alert for the same event as the last one sent, on the same netblock, or for a
`file_error` about the same file, or for a `source_failure` about the same
source, less than `SWWAF_ALERT_COOLDOWN` after it, is a repeat: it is held back
and counted, and the next alert sent for them gives that count as
`suppressed_repeats`.
source, or for an `anomaly` in the same scope, with the same netblock, AS number
or name, whatever its window and kind, less than `SWWAF_ALERT_COOLDOWN` after
it, is a repeat: it is held back and counted, and the next alert sent for them
gives that count as `suppressed_repeats`. As each hour of the clock, in UTC,
ends, the cooldowns that have run out are dropped, and the repeats they held
back, which no alert sent since has given, go in that hour's summary.
Past `SWWAF_ALERT_MAX_PER_HOUR` alerts in an hour of the clock, in UTC, the
hour's other alerts are held back and counted by event. Once the hour has ended,
one alert sums them up: its `event` is `summary`, its `reason` says how many
were held back, and its `detail` gives the `hour` as when it started, the
`count`, and the count for each event, as `events`. An alert held back this way
starts no cooldown, and the repeats held back before it are given by the next
alert sent for the same event and netblock, file or source.
Past `SWWAF_ALERT_MAX_PER_HOUR` alerts in an hour, the hour's other alerts are
held back and counted by event. An alert held back this way starts no cooldown.
Once the hour has ended, one alert sums up the alerts held back and the repeats
of the cooldowns dropped: its `event` is `summary`, its `reason` says how many
of each were held back, its `detail` gives the `hour` as when it started, the
`count` of alerts held back, and the count for each event, as `events`, and its
`suppressed_repeats` gives the repeats. An hour with neither ends without a
summary.
Each destination has a queue of its own, of at most 1000 alerts, from which they
are sent to it one at a time, the oldest first, so a destination that is slow or
@@ -833,7 +900,8 @@ restart the alerts waiting are sent, and the cooldowns go on.
`smallwebwaf` keeps its state in memory and a copy of it in four JSON files in
`SWWAF_STATE_DIR`, `/var/lib/smallwebwaf` by default, as "Persistent state" in
[`SPEC.md`](SPEC.md) describes. Each has a top-level `version`, 1, and lists its
entries by client address, but for the alerts waiting, with times in UTC.
entries by client address, but for the alerts waiting, and the anomaly counters,
which are listed by scope first, with times in UTC.
- `bans.json`: every ban with its notes, indented to be read. A permanent ban's
`expires` is `null`. A ban's `cause` is `limit` for a broken rate limit or
@@ -861,16 +929,23 @@ entries by client address, but for the alerts waiting, with times in UTC.
number, AS name and country, when GeoJS gave it and when it was last used.
- `alerts.json`: the state of the alerts (see "Alerts" above), indented to be
read: under `cooldowns`, for each event and netblock, or event and `file` or
`source`, or event alone, when the last alert was sent, `sent`, and the
repeats held back since, `suppressed_repeats`; under `hour`, the hour under
way, from its `start`, the alerts `sent` in it and those `held_back` for its
summary, by event; and under `waiting`, for each destination you name,
`webhook`, `slack` or `ntfy`, the alerts still waiting to be sent to it, the
oldest first, each as the webhook is sent it. As an hour ends, the cooldowns
that have run out with no repeat held back are dropped. As the file is read,
the alerts waiting for a destination you no longer name are dropped. A file
whose `waiting` is a list, as it was before alerts went to Slack and ntfy too,
stops the start: put the list under `"webhook"`, or remove the file.
`source`, or for an `anomaly`, its `scope` with its `netblock`, `asn` or
`name`, or event alone, when the last alert was sent, `sent`, and the repeats
held back since, `suppressed_repeats`; under `hour`, the hour under way, from
its `start`, the alerts `sent` in it and those `held_back` for its summary, by
event; under `waiting`, for each destination you name, `webhook`, `slack` or
`ntfy`, the alerts still waiting to be sent to it, the oldest first, each as
the webhook is sent it; and under `anomaly_counters`, each anomaly counter:
its `scope`, as an `anomaly` alert names it, with the `netblock` of a client,
of a netblock around a client or of a named netblock, the `asn` of an AS
number and the `name` of a named netblock, and its two buckets of requests in
the minute and the hour, `minute` and `hour`, and of bytes, `minute_bytes` and
`hour_bytes`, each left out while it is empty. As an hour ends, the cooldowns
that have run out are dropped, and the hour's summary gives the repeats they
held back. As the file is read, the alerts waiting for a destination you no
longer name are dropped. A file whose `waiting` is a list, as it was before
alerts went to Slack and ntfy too, stops the start: put the list under
`"webhook"`, or remove the file.
`bans.json` is written `SWWAF_STATE_WRITE_DELAY` after a ban is made, lifted
through `DELETE /_smallwebwaf/bans/<client>`, or made permanent, with every such
@@ -882,22 +957,27 @@ whole. A write that fails is logged, raised as a `file_error` alert while
changed since the last write.
At start the files are read back: each client keeps its counts, so a restart
gives it no fresh allowance, and each ban keeps refusing every client in its
netblock until it ends, even after `SWWAF_BAN_SCOPE_V4_PREFIX` has changed. A
netblock whose address has bits past its length, such as `203.0.113.9/24`, is
read as the netblock it is in, `203.0.113.0/24`. Buckets and answers whose time
has passed are dropped. A missing file is empty state, as on a first start. A
file that does not parse, or has another `version`, stops the start with a
message naming the file, and the line and column where Go's JSON decoder gives
them; so does a state directory `smallwebwaf` cannot write. So does an entry
without a field it needs, named with the entry's place in the file: a ban's
`netblock`, `start` or `expires`, which is `null` for a permanent ban; a
client's `client`, or the `start` of a window in which it has requests or bytes;
an answer's `client`, `country`, which is `""` for a client GeoJS cannot place,
or `answered`; a cooldown's `event` or `sent`; an alert waiting's `event` or
`time`. So does a ban whose `cause` is not `limit`, `attack` or `admin`, and
alerts waiting for a destination that is not `webhook`, `slack` or `ntfy`. An
answer's `asn` or `as_name` left out reads as empty.
gives it no fresh allowance, each anomaly counter keeps its counts, and each ban
keeps refusing every client in its netblock until it ends, even after
`SWWAF_BAN_SCOPE_V4_PREFIX` has changed. A netblock whose address has bits past
its length, such as `203.0.113.9/24`, is read as the netblock it is in,
`203.0.113.0/24`. Buckets and answers whose time has passed are dropped, and so
is an anomaly counter left with no bucket. A missing file is empty state, as on
a first start. A file that does not parse, or has another `version`, stops the
start with a message naming the file, and the line and column where Go's JSON
decoder gives them; so does a state directory `smallwebwaf` cannot write. So
does an entry without a field it needs, named with the entry's place in the
file: a ban's `netblock`, `start` or `expires`, which is `null` for a permanent
ban; a client's `client`, or the `start` of a window in which it has requests or
bytes; an answer's `client`, `country`, which is `""` for a client GeoJS cannot
place, or `answered`; a cooldown's `event` or `sent`; an alert waiting's `event`
or `time`; an anomaly counter's `netblock`, unless it counts an AS number or the
whole service, its `asn`, for an AS number, its `name`, for a named netblock, or
the `start` of a window in which it has requests or bytes. So does a ban whose
`cause` is not `limit`, `attack` or `admin`, alerts waiting for a destination
that is not `webhook`, `slack` or `ntfy`, and an anomaly counter whose `scope`
is not `client`, `net`, `asn`, `total` or `watch`. An answer's `asn` or
`as_name` left out reads as empty.
While it runs, `smallwebwaf` watches `SWWAF_STATE_DIR` and takes in your edit of
a state file as soon as you save it: what the file then holds replaces what
@@ -907,13 +987,14 @@ writes a file it takes in any edit made since, so your edit is not overwritten;
a change `smallwebwaf` made after you opened the file, such as a new ban, is
lost when you save over it. An edit that would stop the start, because it does
not parse, has another `version`, leaves out a field an entry needs, gives a ban
another `cause` or names another destination, does not stop the running
`smallwebwaf`: it keeps what it holds, and at the file's next write renames your
file to `<name>.bad`, such as `bans.json.bad`, writes the file again from
memory, logs the file and where the error is, and raises a `file_error` alert
for it. It waits for that write because an editor's file can be read before the
editor has finished writing it. Mend the `.bad` file and move it back. A file
you remove is written again at its next write.
another `cause`, names another destination or gives an anomaly counter another
`scope`, does not stop the running `smallwebwaf`: it keeps what it holds, and at
the file's next write renames your file to `<name>.bad`, such as
`bans.json.bad`, writes the file again from memory, logs the file and where the
error is, and raises a `file_error` alert for it. It waits for that write
because an editor's file can be read before the editor has finished writing it.
Mend the `.bad` file and move it back. A file you remove is written again at its
next write.
To ban a netblock, add an entry to `bans.json` with its `netblock`, its `start`
and its `expires`, `null` for a ban that never ends; its `reason` and its
@@ -1304,8 +1385,8 @@ For each request `smallwebwaf`:
- forwards it to the app and streams the response back, within the size and time
limits;
- counts the bytes and any refusal by the rule files or the Core Rule Set, bans
the client if it broke a limit, updates its history, sends any alerts that are
due, and writes the log line.
the client if it broke a limit, updates its history and the anomaly counters,
sends any alerts that are due, and writes the log line.
A minimal deployment is the app's own Dockerfile, built on the `smallwebwaf`
image, with no setting. That image is built on Ubuntu 26.04 LTS, the newest
@@ -1394,14 +1475,15 @@ the metrics, failure behaviour and the build order.
`smallwebwaf` looks up the AS number and country of every client through GeoJS,
a free web service that needs no account and no file, for the request log, the
client's history, the notes of its bans, their alerts and the metrics, and for
the country lists when you set them. This means that GeoJS is told the address
of every new visitor, whether or not a setting uses the answer, unless you set
`SWWAF_LOOKUP_SOURCE=off`. The only visitors it is not told about are those in
`SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS`, those whose netblock a ban covers, and
those on a private, loopback or link-local address. An IPv6 visitor is asked
about by the first address of its /64. Each answer is kept for seven days, in
memory and in `lookups.json`, so that it survives a restart, and a visitor whose
answer is kept is not asked about again.
the country lists and the anomaly thresholds per AS number when you set them.
This means that GeoJS is told the address of every new visitor, whether or not a
setting uses the answer, unless you set `SWWAF_LOOKUP_SOURCE=off`. The only
visitors it is not told about are those in `SWWAF_ALLOW_NETS` or
`SWWAF_DENY_NETS`, those whose netblock a ban covers, and those on a private,
loopback or link-local address. An IPv6 visitor is asked about by the first
address of its /64. Each answer is kept for seven days, in memory and in
`lookups.json`, so that it survives a restart, and a visitor whose answer is
kept is not asked about again.
A request waits for its client's first answer only while a setting acts on it
before the request goes on: a country list, `SWWAF_ADD_LOOKUP_HEADERS`, or a
@@ -1470,7 +1552,9 @@ GeoJS.
what it would have refused for noted in the log line. A request under
`/_smallwebwaf/` that `check` lets through is answered by `answerAdmin`
instead of reaching the app. Once the answer to a request passed to the app
has ended, `countBytes` counts its bytes for the byte limits.
has ended, `countBytes` counts its bytes for the byte limits, and once any
request but the health check has ended, `countAnomalies` counts it for the
anomaly thresholds.
- `internal/metrics`: the metrics, counted as the other parts tell it what
happened, and served in the Prometheus text format.
- `internal/bans`: the ban ledger: each netblock's bans with their notes, how
@@ -1486,6 +1570,10 @@ GeoJS.
- `internal/ratelimit`: the table of clients: counts each client's requests and
bytes, tells when they take it over a rate limit or a byte limit, and keeps
each client's history.
- `internal/anomaly`: the anomaly counters: counts each request and its bytes
per client, per netblock around a client, per AS number, for the whole service
and per named netblock, in the buckets `internal/ratelimit` counts in, and
raises an `anomaly` alert for a count over its threshold.
- `internal/state`: reads the state files at start, takes in an admin's edit of
one while running, and writes them when they are due and at the stop.
- `internal/requestlog`: the lines on stdout: the request log line and the
@@ -1506,8 +1594,9 @@ GeoJS.
Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the
table of clients to 20,000 and the GeoJS answers to 100,000, dropping the least
recently seen, and the banned netblocks in the order they were last seen, from
which the ledger picks the ban to drop past `SWWAF_MAX_BANS`, and
recently seen, the anomaly counters to 20,000, dropping the one counted least
recently, and the banned netblocks in the order they were last seen, from which
the ledger picks the ban to drop past `SWWAF_MAX_BANS`, and
`github.com/prometheus/client_golang` keeps the metrics and serves them, and
`github.com/fsnotify/fsnotify` tells `smallwebwaf` when a state file or a rule
file is saved, or the lookup database replaced, and