sneak/smallwebwaf
1
0
Fork 0
Code Issues 47 Pull Requests 1 Actions 40 Packages Projects Releases Wiki Activity

CrowdSec decision list fetched, kept, and its clients banned until the decision ends (closes #106)
check / check (push) Waiting to run
Details

Browse Source 
SWWAF_CROWDSEC_LAPI_URL and SWWAF_CROWDSEC_LAPI_KEY name an engine whose
decision list, <url>/v1/decisions, is fetched every minute with the key in
X-Api-Key and kept as a blocklist is: used while a fetch fails, and across
restarts through reputation.json. Ban decisions on an Ip or a Range end at
the fetch time plus their duration. A listed client's request is refused and
bans its netblock with the cause crowdsec until the decision ends; bans.json,
ban notes and metrics take the cause.

Judgement call: fetched every minute, not a setting.
Judgement call: a crowdsec ban never lengthens a limit ban.
Judgement call: a lifted crowdsec ban is remade while its decision lasts.

Model: opus-5-5
This commit is contained in:
clawbot
2026-10-08 01:47:31 +00:00
parent 04e66d2069
commit 91f69346ea
19 changed files with 1594 additions and 290 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+281 -185
View File
@@ -26,11 +26,12 @@ Slack and ntfy, and remote log sending. So is the stage after that: the AS
number and country of every client, looked up through GeoJS or in the IPinfo
Lite database file, the byte limits, the biased thresholds, lower limits for the
AS numbers and countries you list, and the anomaly thresholds, alerts for
unusual traffic that refuse nothing. So are the first three parts of the stage
after that: the blocklists you name by URL, which it fetches and keeps, with a
file of AS numbers' percentages fetched the same way, the DNS blocklists (DNSBL
zones), which it asks about each client in the background, and AbuseIPDB, which
it asks in the background about each client that has committed an offence.
unusual traffic that refuse nothing. So is the stage after that: the blocklists
you name by URL, which it fetches and keeps, with a file of AS numbers'
percentages fetched the same way, the DNS blocklists (DNSBL zones), which it
asks about each client in the background, AbuseIPDB, which it asks in the
background about each client that has committed an offence, and the decision
list of a CrowdSec engine you run, which it fetches and keeps as a blocklist.
`smallwebwaf` passes each request to the app and the app's answer back,
unchanged, within its timeouts and size limits, works out each client's address,
looks up its AS number and country unless you switch that off, bans a client
@@ -39,26 +40,28 @@ you choose, with lower limits for the clients of the AS numbers and countries
you list, refuses a client that comes from a country you refuse or from a
network you refuse, refuses, limits or only notes a client a blocklist or a
DNSBL zone you name lists, or AbuseIPDB scores at or over the score you set,
lets the networks you choose through, checks each request against the rule files
and bans a client whose request is a clear sign of attack, keeps its bans, each
client's counters and history, GeoJS's answers, the last good copy of each list
it fetches, the DNSBL zones' verdicts, AbuseIPDB's scores and the AbuseIPDB
bans a client your CrowdSec engine's decision list lists until that decision
ends, lets the networks you choose through, checks each request against the rule
files and bans a client whose request is a clear sign of attack, keeps its bans,
each client's counters and history, GeoJS's answers, the last good copy of each
list it fetches, the DNSBL zones' verdicts, AbuseIPDB's scores and the AbuseIPDB
checks spent today in JSON files across restarts, takes in your edits of those
files, such as a ban you make, keep or lift, and of the rule files while it
runs, writes a JSON log line for every request, sends its log lines to a syslog
server too if you name one, sends an alert to a webhook, to Slack and to ntfy,
each if you name one, for each ban it makes or makes permanent, for traffic over
an anomaly threshold you set, for a client a blocklist, a DNSBL zone or
AbuseIPDB lists, for GeoJS failing, a list it cannot fetch, a DNSBL zone or
AbuseIPDB that fails or refuses a query and the day's AbuseIPDB checks used up,
for a rule file or state file with an error and for a replacement of the lookup
database it cannot read, serves Prometheus metrics to a scraper that holds the
metrics token, lets an admin who holds the admin token list, add and lift bans
and ask what it knows of a client, and in `observe` mode passes on the requests
it would refuse, logging what it would have done with them. It comes as the
image the app's own image is built on. The rest of the design comes after that,
in the order of the build order in [`SPEC.md`](SPEC.md). The survey of existing
tools that led to the design is in [`EVALUATION.md`](EVALUATION.md).
an anomaly threshold you set, for a client a blocklist, the CrowdSec decision
list, a DNSBL zone or AbuseIPDB lists, for GeoJS failing, a list it cannot
fetch, a DNSBL zone or AbuseIPDB that fails or refuses a query and the day's
AbuseIPDB checks used up, for a rule file or state file with an error and for a
replacement of the lookup database it cannot read, serves Prometheus metrics to
a scraper that holds the metrics token, lets an admin who holds the admin token
list, add and lift bans and ask what it knows of a client, and in `observe` mode
passes on the requests it would refuse, logging what it would have done with
them. It comes as the image the app's own image is built on. The rest of the
design comes after that, in the order of the build order in
[`SPEC.md`](SPEC.md). The survey of existing tools that led to the design is in
[`EVALUATION.md`](EVALUATION.md).
## Getting started
@@ -163,16 +166,17 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
and the requests or bytes counted in it, the client's percentage of that kind
of limit and the setting that gave it when a biased threshold lowered the
limit, the request that broke it, the client's AS number, AS name and country
once they are looked up, the blocklists, DNSBL zones and AbuseIPDB, with its
score, that listed the client when the ban was made, the netblock's requests
since it was first seen, how many of them the ban has refused, and how many
bans the netblock had before, for a broken limit, for a clear sign of attack
and by an admin. At most `SWWAF_MAX_BANS` bans `smallwebwaf` made are kept,
past, active and permanent; past that, the earliest such ban of the netblock
that has gone longest without a request is dropped first. The bans whose cause
is `admin`, those you make or keep, are kept besides, and never dropped.
`bans.json` shows the bans and their notes, a restart lifts none, and you
make, keep or lift a ban by editing it (see "State files" below).
once they are looked up, the blocklists, the CrowdSec decision list, DNSBL
zones and AbuseIPDB, with its score, that listed the client when the ban was
made, the netblock's requests since it was first seen, how many of them the
ban has refused, and how many bans the netblock had before, for a broken
limit, for a clear sign of attack, by an admin and for CrowdSec's decision. At
most `SWWAF_MAX_BANS` bans `smallwebwaf` made are kept, past, active and
permanent; past that, the earliest such ban of the netblock that has gone
longest without a request is dropped first. The bans whose cause is `admin`,
those you make or keep, are kept besides, and never dropped. `bans.json` shows
the bans and their notes, a restart lifts none, and you make, keep or lift a
ban by editing it (see "State files" below).
- Checks each request against the rules of the rule files (see "Rule files"
below) after the rate limits, and before its body is read. A `log` rule that
matches is noted in the log line; a `block` rule refuses the request with
@@ -214,6 +218,14 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
limit, the lowest of its percentages applying, as for a biased threshold, and
with `log` nothing more is done. Whatever the action, the request's log line
names the lists, and each raises an alert.
- Checks the client's own address against the decision list of the CrowdSec
engine `SWWAF_CROWDSEC_LAPI_URL` names, after the blocklists and before the
DNSBL zones (see "CrowdSec" below). A request from a client a decision in
force bans is refused with `SWWAF_BAN_RESPONSE` before its body is read, and
bans the client's netblock, as any ban covers it, until the decision ends,
with the cause `crowdsec`; the client's next requests are refused under that
ban. The request's log line names the list, and it raises an alert besides the
ban's. A client a blocklist refuses is not checked.
- Checks the client's own address against the DNSBL zones `SWWAF_DNSBL_ZONES`
names, after the blocklists and before the rate limits (see "DNS blocklists"
below), by the verdicts it keeps. A zone is asked about a client in the
@@ -226,7 +238,7 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
before their bodies are read; they are not counted for the rate limits, and
make no ban. With `log`, nothing more is done. Whatever the action, the
request's log line names the zones, and each raises an alert. A client a
blocklist refuses is not checked.
blocklist refuses, or the CrowdSec decision list bans, is not checked.
- Checks the client with AbuseIPDB while `SWWAF_ABUSEIPDB_KEY` is set, after the
DNSBL zones and before the rate limits (see "AbuseIPDB" below), by the scores
it keeps. Only a client whose history counts an offence is checked, so far one
@@ -235,38 +247,39 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
request waits for AbuseIPDB. A score at or over `SWWAF_ABUSEIPDB_MIN_SCORE` is
a hit, and `SWWAF_REPUTATION_ACTION` does with its client what it does with
one a DNSBL zone's verdict lists. The request's log line names AbuseIPDB, and
it raises an alert. A client a blocklist or a DNSBL zone refuses is not
checked.
it raises an alert. A client a blocklist or a DNSBL zone refuses, or the
CrowdSec decision list bans, is not checked.
- Checks the client's own address against the static lists, the three netblock
settings below, before anything else, its lookup included. A client in
`SWWAF_ALLOW_NETS` skips bans, the country lists, the blocklists, the DNSBL
zones, AbuseIPDB, the rate limits, the byte limits and the rule files, and is
not looked up; the timeouts and size limits still apply. A client in
`SWWAF_DENY_NETS` is refused with `SWWAF_BAN_RESPONSE` before its body is
read, and the request is not counted for the rate limits; an address in
`SWWAF_ALLOW_NETS` too is let through. A client in
`SWWAF_ALLOW_NETS` skips bans, the country lists, the blocklists, the CrowdSec
decision list, the DNSBL zones, AbuseIPDB, the rate limits, the byte limits
and the rule files, and is not looked up; the timeouts and size limits still
apply. A client in `SWWAF_DENY_NETS` is refused with `SWWAF_BAN_RESPONSE`
before its body is read, and the request is not counted for the rate limits;
an address in `SWWAF_ALLOW_NETS` too is let through. A client in
`SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither counted nor refused by the rate
limits, and has no bytes counted by the byte limits; the country lists, the
rule files and bans still apply to it.
- In `observe` mode, with `SWWAF_MODE=observe`, refuses none of the requests
that `SWWAF_DENY_NETS`, a ban, the country lists, a blocklist, a DNSBL zone's
verdict, AbuseIPDB's score, a rate limit or a rule would refuse: it passes
them to the app, and their log lines name what `enforce` mode would have done
(see `would_action` in "Request log" below). The checks run, and requests and
bytes are counted, as in `enforce` mode, with three differences: neither a
broken rate limit or byte limit nor a `ban` rule makes a ban; a broken limit
does not set the client's counters back to zero, so each request over a rate
limit is logged as one that would be refused, and each whose bytes keep the
client over a byte limit as breaking it; and a request under a ban does not
make it permanent. As in `enforce` mode, the bytes counted are only those of
the requests `enforce` mode would have passed to the app. A ban it would have
made, or made permanent, raises the alert `enforce` mode would have raised,
marked as what would have happened (see "Alerts" below). The bans in
`bans.json` are kept, and refuse requests again when `smallwebwaf` next runs
in `enforce` mode, as long as they last. The timeouts and size limits still
apply, since they protect `smallwebwaf` and the app themselves, and a request
for one of `smallwebwaf`'s own endpoints without its token is still answered
`401`. It is for trying a configuration before enforcing it.
that `SWWAF_DENY_NETS`, a ban, the country lists, a blocklist, the CrowdSec
decision list, a DNSBL zone's verdict, AbuseIPDB's score, a rate limit or a
rule would refuse: it passes them to the app, and their log lines name what
`enforce` mode would have done (see `would_action` in "Request log" below).
The checks run, and requests and bytes are counted, as in `enforce` mode, with
three differences: neither a broken rate limit or byte limit, a `ban` rule nor
the CrowdSec decision list makes a ban; a broken limit does not set the
client's counters back to zero, so each request over a rate limit is logged as
one that would be refused, and each whose bytes keep the client over a byte
limit as breaking it; and a request under a ban does not make it permanent. As
in `enforce` mode, the bytes counted are only those of the requests `enforce`
mode would have passed to the app. A ban it would have made, or made
permanent, raises the alert `enforce` mode would have raised, marked as what
would have happened (see "Alerts" below). The bans in `bans.json` are kept,
and refuse requests again when `smallwebwaf` next runs in `enforce` mode, as
long as they last. The timeouts and size limits still apply, since they
protect `smallwebwaf` and the app themselves, and a request for one of
`smallwebwaf`'s own endpoints without its token is still answered `401`. It is
for trying a configuration before enforcing it.
- Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any
check and without asking the app, for the image's health check.
- Answers `GET /_smallwebwaf/metrics` with its metrics (see "Metrics" below) for
@@ -286,12 +299,13 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
`SWWAF_LOG_REMOTE_URL` names one (see "Sending the log to a syslog server"
below).
- Sends an alert for each ban it makes or makes permanent, for a count over an
anomaly threshold, for a client a blocklist, a DNSBL zone or AbuseIPDB lists,
for GeoJS failing, a list it cannot fetch, a DNSBL zone or AbuseIPDB that
fails or refuses a query, and the day's AbuseIPDB checks used up, for a rule
file or state file with an error, and for a replacement of the lookup database
it cannot read, holding back repeats and, past an hourly limit, rolling the
rest into one summary, to each destination you name: as a JSON object to the
anomaly threshold, for a client a blocklist, the CrowdSec decision list, a
DNSBL zone or AbuseIPDB lists, for GeoJS failing, a list it cannot fetch, the
CrowdSec decision list among them, a DNSBL zone or AbuseIPDB that fails or
refuses a query, and the day's AbuseIPDB checks used up, for a rule file or
state file with an error, and for a replacement of the lookup database it
cannot read, holding back repeats and, past an hourly limit, rolling the rest
into one summary, to each destination you name: as a JSON object to the
webhook `SWWAF_ALERT_WEBHOOK_URL` names, as a message to the Slack incoming
webhook `SWWAF_ALERT_SLACK_WEBHOOK_URL` names, and as a message to the ntfy
topic `SWWAF_ALERT_NTFY_URL` names (see "Alerts" below).
@@ -370,9 +384,9 @@ effective settings are logged at start, unless `SWWAF_LOG_LEVEL` is `warn` or
- `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body.
- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body.
- `SWWAF_ALLOW_NETS` (default empty): netblocks whose clients skip bans, the
country lists, the blocklists, the DNSBL zones, AbuseIPDB, the rate limits,
the byte limits and the rule files, such as your monitoring or your own
networks.
country lists, the blocklists, the CrowdSec decision list, the DNSBL zones,
AbuseIPDB, the rate limits, the byte limits and the rule files, such as your
monitoring or your own networks.
- `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients the
rate limits and the byte limits do not apply to, such as a machine that talks
to the app all day.
@@ -495,6 +509,17 @@ effective settings are logged at start, unless `SWWAF_LOG_LEVEL` is `warn` or
from 0 to 100, that is a hit.
- `SWWAF_ABUSEIPDB_DAILY_BUDGET` (default `900`): the most checks made in a day,
in UTC, a whole number above zero. AbuseIPDB's free accounts may make 1,000.
- `SWWAF_CROWDSEC_LAPI_URL` (default unset): the local API of a CrowdSec engine
you run, whose decision list is fetched (see "CrowdSec" below), as an `http`
or `https` URL without a user or a fragment, such as `http://172.17.0.1:8080`.
While it is unset, no list is fetched. Set without `SWWAF_CROWDSEC_LAPI_KEY`,
it stops the start, and so does a URL whose decision list, the URL with
`v1/decisions` added to its path, `SWWAF_BLOCKLIST_URLS` or
`SWWAF_ASN_LIMIT_PERCENT_URL` names too.
- `SWWAF_CROWDSEC_LAPI_KEY` (default unset): the key the engine gave for
`smallwebwaf`, as `cscli bouncers add smallwebwaf` prints it. Set without
`SWWAF_CROWDSEC_LAPI_URL`, it stops the start. The settings logged at start
show `********` in its place.
- `SWWAF_REPUTATION_ACTION` (default `limit:25`): what is done with a client a
zone's verdict lists, or whose AbuseIPDB score is a hit, as
`SWWAF_BLOCKLIST_ACTION` is for a blocklist: `deny`, `limit:<percent>` or
@@ -516,7 +541,8 @@ effective settings are logged at start, unless `SWWAF_LOG_LEVEL` is `warn` or
limit or byte limit.
- `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): a rate limit or byte limit
broken again within this time after a ban ended, other than one for a clear
sign of attack, bans for three times as long as that ban.
sign of attack or for CrowdSec's decision, bans for three times as long as
that ban.
- `SWWAF_MAX_BAN_DURATION` (default `7d`): a ban for a broken rate limit or byte
limit that would be longer is permanent instead.
- `SWWAF_ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of
@@ -668,7 +694,8 @@ and `SWWAF_ANOMALY_NET_V6_PREFIX` cannot be off.
Several limits are fixed rather than settings. At most 100,000 answers from
GeoJS are kept, for 7 days each, at most 20,000 anomaly counters, at most
100,000 verdicts of the DNSBL zones, with at most 1,000 queries to them under
way at once, and at most 100,000 scores of AbuseIPDB.
way at once, and at most 100,000 scores of AbuseIPDB, and the CrowdSec decision
list is fetched again a minute after it was last fetched or tried.
### Settings given as files
@@ -754,20 +781,22 @@ which every line has.
refused because its client is in `SWWAF_DENY_NETS`, in a blocklist while
`SWWAF_BLOCKLIST_ACTION` is `deny`, or listed by a DNSBL zone or AbuseIPDB
while `SWWAF_REPUTATION_ACTION` is `deny`, `banned` for one refused because a
ban covers its client or because it matched a `ban` rule, which bans its
client, `country_denied` for one refused for its client's country,
`rate_limited` for one that broke a rate limit and banned its client,
`rule_blocked` for one a `block` rule refused, `too_large` for a request or
response over its size limit, `timed_out` for one that ran out of time,
`upstream_error` when the app could not be reached or its answer broke off,
and `admin` for one `smallwebwaf` answered at its own endpoint.
ban covers its client, or because it matched a `ban` rule or the CrowdSec
decision list lists its client, either of which bans its client,
`country_denied` for one refused for its client's country, `rate_limited` for
one that broke a rate limit and banned its client, `rule_blocked` for one a
`block` rule refused, `too_large` for a request or response over its size
limit, `timed_out` for one that ran out of time, `upstream_error` when the app
could not be reached or its answer broke off, and `admin` for one
`smallwebwaf` answered at its own endpoint.
- `would_action` is there in `observe` mode for a request that
`SWWAF_DENY_NETS`, a ban, the country lists, a blocklist, a DNSBL zone's
verdict, AbuseIPDB's score, a rate limit or a rule would have refused in
`enforce` mode, and names the action that refusal would have had: `denied`,
`banned`, `country_denied`, `rate_limited` or `rule_blocked`. `action` then
names what was done: `forward` for a request passed to the app, and another
action, such as `too_large`, for one a size or time limit refused.
`SWWAF_DENY_NETS`, a ban, the country lists, a blocklist, the CrowdSec
decision list, a DNSBL zone's verdict, AbuseIPDB's score, a rate limit or a
rule would have refused in `enforce` mode, and names the action that refusal
would have had: `denied`, `banned`, `country_denied`, `rate_limited` or
`rule_blocked`. `action` then names what was done: `forward` for a request
passed to the app, and another action, such as `too_large`, for one a size or
time limit refused.
- `limit_percent` is there for a request the rate limits count whose client a
biased threshold, `SWWAF_BLOCKLIST_ACTION` for a blocklist that lists it, or
`SWWAF_REPUTATION_ACTION` for a DNSBL zone whose verdict lists it or an
@@ -785,11 +814,12 @@ which every line has.
health check, one from a client in `SWWAF_ALLOW_NETS` or
`SWWAF_RATE_LIMIT_EXEMPT_NETS`, one for a path that
`SWWAF_RATE_LIMIT_EXEMPT_PATHS` exempts, and one that `SWWAF_DENY_NETS`, a
ban, the country lists, a blocklist, a DNSBL zone's verdict or AbuseIPDB's
score refuse, or would refuse in `observe` mode. Its `minute_bytes`,
`hour_bytes` and `day_bytes` give the client's bytes in each window as the
byte limits count them, in the same way: for a request whose bytes they count,
with its own, once it has ended; for any other, those counted before it.
ban, the country lists, a blocklist, the CrowdSec decision list, a DNSBL
zone's verdict or AbuseIPDB's score refuse, or would refuse in `observe` mode.
Its `minute_bytes`, `hour_bytes` and `day_bytes` give the client's bytes in
each window as the byte limits count them, in the same way: for a request
whose bytes they count, with its own, once it has ended; for any other, those
counted before it.
- `rule_ids` is there for a request that matched rules of the rule files, and
lists their ids in the order they matched, up to the one that refused it.
- `limit_hit` is there for a request that broke a rate limit, or whose bytes
@@ -799,18 +829,22 @@ which every line has.
several. `offence` is then `limit`. A request whose bytes broke a byte limit
is not refused: its `action` is what it would have been otherwise, such as
`forward`.
- `reputation` is there for a request whose client a blocklist or a DNSBL zone's
verdict lists, or whose AbuseIPDB score is a hit, and gives the URLs of the
blocklists that list it, in the order `SWWAF_BLOCKLIST_URLS` names them, then
the zones whose verdict lists it, in the order `SWWAF_DNSBL_ZONES` names them,
then `abuseipdb`, whatever `SWWAF_BLOCKLIST_ACTION` and
`SWWAF_REPUTATION_ACTION` say. A zone whose verdict on the client has not come
yet, or was given `SWWAF_REPUTATION_CACHE_TTL` ago or more, is not named, nor
is AbuseIPDB for such a score. It is left out for a client the blocklists are
not checked for: one in `SWWAF_ALLOW_NETS`, and one `SWWAF_DENY_NETS`, a ban
or the country lists refuse first, or would in `observe` mode. The zones are
not checked either for a client a blocklist refuses, nor AbuseIPDB's score for
one a blocklist or a zone refuses, or would in `observe` mode.
- `reputation` is there for a request whose client a blocklist, the CrowdSec
decision list or a DNSBL zone's verdict lists, or whose AbuseIPDB score is a
hit, and gives the URLs of the blocklists that list it, in the order
`SWWAF_BLOCKLIST_URLS` names them, then the URL of the CrowdSec decision list
when a decision in force bans the client, then the zones whose verdict lists
it, in the order `SWWAF_DNSBL_ZONES` names them, then `abuseipdb`, whatever
`SWWAF_BLOCKLIST_ACTION` and `SWWAF_REPUTATION_ACTION` say. A zone whose
verdict on the client has not come yet, or was given
`SWWAF_REPUTATION_CACHE_TTL` ago or more, is not named, nor is AbuseIPDB for
such a score. It is left out for a client the blocklists are not checked for:
one in `SWWAF_ALLOW_NETS`, and one `SWWAF_DENY_NETS`, a ban or the country
lists refuse first, or would in `observe` mode. The CrowdSec decision list is
not checked either for a client a blocklist refuses, nor are the zones for one
a blocklist refuses or the decision list bans, nor AbuseIPDB's score for one a
blocklist or a zone refuses or the decision list bans, or would in `observe`
mode.
- `ban_expires` is there for a request that made a ban or was refused under one,
or in `observe` mode would have been refused under one, and gives when the ban
ends, in the same form as `time`, or `permanent`.
@@ -874,22 +908,23 @@ gives, as "Alert webhook schema" in [`SPEC.md`](SPEC.md) describes, and to
it, as below. An alert is for one of these events, and is sent when
`SWWAF_ALERT_EVENTS` names its event:
- `ban`: a ban `smallwebwaf` makes, for a broken rate limit or byte limit or a
clear sign of attack.
- `ban`: a ban `smallwebwaf` makes, for a broken rate limit or byte limit, a
clear sign of attack, or a client the CrowdSec decision list lists.
- `permanent_ban`: a permanent ban it makes, or a ban for a clear sign of attack
that a request made permanent.
- `anomaly`: a count of requests or bytes over an anomaly threshold, raised by
each request that ends with the count over it, in `observe` mode as in
`enforce` mode. It refuses and bans nothing.
- `reputation_hit`: a request whose client a blocklist or a DNSBL zone's verdict
lists, or whose AbuseIPDB score is a hit, one alert for each blocklist and
each zone that lists it, and one for AbuseIPDB, whatever
`SWWAF_BLOCKLIST_ACTION` and `SWWAF_REPUTATION_ACTION` say, in `observe` mode
as in `enforce` mode.
- `reputation_hit`: a request whose client a blocklist, the CrowdSec decision
list or a DNSBL zone's verdict lists, or whose AbuseIPDB score is a hit, one
alert for each blocklist and each zone that lists it, one for the decision
list, and one for AbuseIPDB, whatever `SWWAF_BLOCKLIST_ACTION` and
`SWWAF_REPUTATION_ACTION` say, in `observe` mode as in `enforce` mode.
- `source_failure`: GeoJS failing or refusing `smallwebwaf`, a fetch of a list
failing (see "Blocklists" below), a query to a DNSBL zone failing or refused
(see "DNS blocklists" below), or a check with AbuseIPDB failing or refused, or
the check that uses up the day's AbuseIPDB checks (see "AbuseIPDB" below).
failing (see "Blocklists" below), the CrowdSec decision list's among them (see
"CrowdSec" below), a query to a DNSBL zone failing or refused (see "DNS
blocklists" below), or a check with AbuseIPDB failing or refused, or the check
that uses up the day's AbuseIPDB checks (see "AbuseIPDB" below).
- `file_error`: a rule file edited while it runs that has an error, an edit of a
state file set aside as `<name>.bad`, a state file it could not write while
running, or a replacement of the lookup database it could not read, which it
@@ -939,7 +974,8 @@ is sent on one line:
"earlier_bans": {
"limit": 0,
"attack": 0,
"admin": 0
"admin": 0,
"crowdsec": 0
}
}
},
@@ -962,7 +998,8 @@ is sent on one line:
- `reason` is a short sentence; for a ban, the ban's `reason` in `bans.json`;
for an `anomaly`, what was counted over which threshold, such as
`requests per minute of the netblock 203.0.113.0/24 over the threshold of 1000`;
for a `reputation_hit`, `listed by a blocklist`, `listed by a DNSBL zone` or
for a `reputation_hit`, `listed by a blocklist`,
`listed by the CrowdSec decision list`, `listed by a DNSBL zone` or
`scored by AbuseIPDB at or over SWWAF_ABUSEIPDB_MIN_SCORE`.
- `detail` is what is particular to the event: for a ban, its `cause`, when it
ends as `ban_expires`, in the form the request log gives it, and its `notes`,
@@ -971,12 +1008,12 @@ is sent on one line:
`asn` and the `name` of the named netblock for `watch`, the `window`, `minute`
or `hour`, the `kind`, `requests` or `bytes`, the `count`, which is weighted
as the rate limits weigh theirs, and the `threshold`; for a `reputation_hit`,
the `source`, the URL of the blocklist, the zone or `abuseipdb`, and for
AbuseIPDB the `score`; for `source_failure`, the `source`, `geojs`, the URL of
the list, the zone or `abuseipdb`, the `error`, and for GeoJS, when it is
asked again, `asking_again_in`; for `file_error`, the `file`, which for an
edit set aside is the file it was renamed to, and the `error`, which for a
file that does not parse names where in it the error is.
the `source`, the URL of the blocklist or of the CrowdSec decision list, the
zone or `abuseipdb`, and for AbuseIPDB the `score`; for `source_failure`, the
`source`, `geojs`, the URL of the list, the zone or `abuseipdb`, the `error`,
and for GeoJS, when it is asked again, `asking_again_in`; for `file_error`,
the `file`, which for an edit set aside is the file it was renamed to, and the
`error`, which for a file that does not parse names where in it the error is.
- `suppressed_repeats` is how many repeats the cooldown held back before this
alert, and for a `summary`, those no other alert gives (see below).
@@ -1022,14 +1059,14 @@ and Slack this JSON object, shown indented; it is sent on one line:
```
An alert for the same event as the last one sent, on the same netblock, and for
a `reputation_hit` about the same blocklist, zone or AbuseIPDB, or for a
`file_error` about the same file, or for a `source_failure` about the same
source, or for an `anomaly` in the same scope, with the same netblock, AS number
or name, whatever its window and kind, less than `SWWAF_ALERT_COOLDOWN` after
it, is a repeat: it is held back and counted, and the next alert sent for them
gives that count as `suppressed_repeats`. As each hour of the clock, in UTC,
ends, the cooldowns that have run out are dropped, and the repeats they held
back, which no alert sent since has given, go in that hour's summary.
a `reputation_hit` about the same blocklist, decision list, zone or AbuseIPDB,
or for a `file_error` about the same file, or for a `source_failure` about the
same source, or for an `anomaly` in the same scope, with the same netblock, AS
number or name, whatever its window and kind, less than `SWWAF_ALERT_COOLDOWN`
after it, is a repeat: it is held back and counted, and the next alert sent for
them gives that count as `suppressed_repeats`. As each hour of the clock, in
UTC, ends, the cooldowns that have run out are dropped, and the repeats they
held back, which no alert sent since has given, go in that hour's summary.
Past `SWWAF_ALERT_MAX_PER_HOUR` alerts in an hour, the hour's other alerts are
held back and counted by event. An alert held back this way starts no cooldown.
@@ -1065,21 +1102,23 @@ with times in UTC.
- `bans.json`: every ban with its notes, indented to be read. A permanent ban's
`expires` is `null`. A ban's `cause` is `limit` for a broken rate limit or
byte limit or `attack` for a clear sign of attack, for a ban `smallwebwaf`
made, and `admin` for one you made or keep. Its `reason` is a short text: for
a ban `smallwebwaf` made, the limit broken, such as
`requests per minute over the limit of 1000` or
`bytes per hour over the limit of 21474836480`, or the rule that matched, such
as `matched the rule env-file`; for yours, what you wrote. Its `lifted` is
when you lifted it, and is left out until you do. The `kind` in the notes of a
ban for a broken limit is `requests` or `bytes`, what the limit is on. For a
limit a biased threshold lowered, the reason and the notes' `limit` give the
lowered limit, and the notes' `limit_percent` and `limit_percent_setting` the
client's percentage of that kind of limit and the setting that gave it. The
notes' `reputation` gives each blocklist, DNSBL zone or AbuseIPDB that listed
the client when the ban was made, as its `source`, named and ordered as in the
request log's `reputation`, with AbuseIPDB's `score` of the client. It is left
out when none did, and the example below shows it.
byte limit, `attack` for a clear sign of attack or `crowdsec` for a client the
CrowdSec decision list lists, for a ban `smallwebwaf` made, and `admin` for
one you made or keep. Its `reason` is a short text: for a ban `smallwebwaf`
made, the limit broken, such as `requests per minute over the limit of 1000`
or `bytes per hour over the limit of 21474836480`, the rule that matched, such
as `matched the rule env-file`, or the scenario that made CrowdSec's decision,
such as `CrowdSec's decision for crowdsecurity/ssh-bf`; for yours, what you
wrote. Its `lifted` is when you lifted it, and is left out until you do. The
`kind` in the notes of a ban for a broken limit is `requests` or `bytes`, what
the limit is on. For a limit a biased threshold lowered, the reason and the
notes' `limit` give the lowered limit, and the notes' `limit_percent` and
`limit_percent_setting` the client's percentage of that kind of limit and the
setting that gave it. The notes' `reputation` gives each blocklist, the
CrowdSec decision list, each DNSBL zone or AbuseIPDB that listed the client
when the ban was made, as its `source`, named and ordered as in the request
log's `reputation`, with AbuseIPDB's `score` of the client. It is left out
when none did, and the example below shows it.
- `clients.json`: each client's two buckets of requests in the minute, the hour
and the day, its two buckets of bytes in each, `minute_bytes`, `hour_bytes`
and `day_bytes`, and its history: when it was first and last seen, its AS
@@ -1091,20 +1130,20 @@ with times in UTC.
`grep` shows everything about one.
- `lookups.json`: GeoJS's answers, one to a line, each with the client's AS
number, AS name and country, when GeoJS gave it and when it was last used.
- `reputation.json`: each list fetched from a URL (see "Blocklists" below),
indented to be read, under `lists`: its `url`, when it was last `tried`, the
fetch failed or not, and its last good copy: when that was `fetched`, and its
`lines`, as fetched, comment lines included, each on a line of its own, both
left out while no fetch of it has succeeded; and under `verdicts`, each
verdict of a DNSBL zone still in use (see "DNS blocklists" below): its `zone`,
the `client`'s address, whether the zone `listed` the client, and when the
zone gave it, `fetched`; and under `abuseipdb` (see "AbuseIPDB" below), the
`day`, in UTC, whose checks it counts, left out before the first, the checks
`spent` that day, and under `scores`, each score of AbuseIPDB still in use:
the `client`, its IPv4 address as a /32 or its IPv6 group, its `score`, and
when AbuseIPDB gave it, `fetched`. As the file is read, the lists the settings
no longer name, and the verdicts of the zones they no longer name, are
dropped.
- `reputation.json`: each list fetched from a URL (see "Blocklists" and
"CrowdSec" below), indented to be read, under `lists`: its `url`, when it was
last `tried`, the fetch failed or not, and its last good copy: when that was
`fetched`, and its `lines`, as fetched, comment lines included, each on a line
of its own, both left out while no fetch of it has succeeded; and under
`verdicts`, each verdict of a DNSBL zone still in use (see "DNS blocklists"
below): its `zone`, the `client`'s address, whether the zone `listed` the
client, and when the zone gave it, `fetched`; and under `abuseipdb` (see
"AbuseIPDB" below), the `day`, in UTC, whose checks it counts, left out before
the first, the checks `spent` that day, and under `scores`, each score of
AbuseIPDB still in use: the `client`, its IPv4 address as a /32 or its IPv6
group, its `score`, and when AbuseIPDB gave it, `fetched`. As the file is
read, the lists the settings no longer name, and the verdicts of the zones
they no longer name, are dropped.
- `alerts.json`: the state of the alerts (see "Alerts" above), indented to be
read: under `cooldowns`, for each event and netblock, with the `source` too
for a `reputation_hit`, or event and `file` or `source`, or for an `anomaly`,
@@ -1172,7 +1211,8 @@ at its default of `limit:25`, lowered to a quarter:
"earlier_bans": {
"limit": 0,
"attack": 0,
"admin": 0
"admin": 0,
"crowdsec": 0
}
}
}
@@ -1211,11 +1251,11 @@ counter's `netblock`, unless it counts an AS number or the whole service, its
window in which it has requests or bytes; a list's `url`, `fetched` or `lines`,
which is `[]` for an empty list; a verdict's `zone`, `client`, `listed`, which
is `false` for a client the zone does not list, or `fetched`. So does a ban
whose `cause` is not `limit`, `attack` or `admin`, alerts waiting for a
destination that is not `webhook`, `slack` or `ntfy`, an anomaly counter whose
`scope` is not `client`, `net`, `asn`, `total` or `watch`, and a copy of a list
with a line that would make its fetch fail. An answer's `asn` or `as_name` left
out reads as empty.
whose `cause` is not `limit`, `attack`, `admin` or `crowdsec`, alerts waiting
for a destination that is not `webhook`, `slack` or `ntfy`, an anomaly counter
whose `scope` is not `client`, `net`, `asn`, `total` or `watch`, and a copy of a
list with a line that would make its fetch fail. An answer's `asn` or `as_name`
left out reads as empty.
While it runs, `smallwebwaf` watches `SWWAF_STATE_DIR` and takes in your edit of
a state file as soon as you save it: what the file then holds replaces what
@@ -1240,7 +1280,7 @@ and its `expires`, `null` for a ban that never ends; its `reason` and its
written so at the file's next write. A ban whose `cause` is `admin` is never
dropped and does not count toward `SWWAF_MAX_BANS`. A ban whose `cause` is
`attack` becomes permanent at the first request it refuses; one whose `cause` is
`admin` does not. This `bans.json` bans `203.0.113.0/24` for good:
`admin` or `crowdsec` does not. This `bans.json` bans `203.0.113.0/24` for good:
```json
{
@@ -1369,11 +1409,11 @@ scraped, and keeps this one as `exported_instance` unless the scrape sets
and `kind`, `requests` for a rate limit or `bytes` for a byte limit,
`smallwebwaf_size_and_time_limit_hits_total` by `limit`, the setting whose
limit was passed, `smallwebwaf_offences_total` by `kind`, and
`smallwebwaf_bans_made_total` by `cause`, `limit`, `attack` or `admin`, the
last for the bans you add through `POST /_smallwebwaf/bans`, and those whose
`cause` is `admin` that you add to `bans.json` while `smallwebwaf` runs;
`smallwebwaf_active_bans` and `smallwebwaf_permanent_bans`, neither of which
counts a lifted ban.
`smallwebwaf_bans_made_total` by `cause`, `limit`, `attack`, `admin` or
`crowdsec`, `admin` for the bans you add through `POST /_smallwebwaf/bans`,
and those whose `cause` is `admin` that you add to `bans.json` while
`smallwebwaf` runs; `smallwebwaf_active_bans` and
`smallwebwaf_permanent_bans`, neither of which counts a lifted ban.
- `smallwebwaf_rule_matches_total`: the requests that matched each rule, by
`rule_id` and `action`, the rule's own; and `smallwebwaf_rules_loaded`: the
rules read from the rule files.
@@ -1402,8 +1442,9 @@ scraped, and keeps this one as `exported_instance` unless the scrape sets
`smallwebwaf_lookup_database_read_failures_total`: the replacements of it that
could not be read.
- By `source`, the URL of each list `SWWAF_BLOCKLIST_URLS` or
`SWWAF_ASN_LIMIT_PERCENT_URL` names: `smallwebwaf_reputation_hits_total`: the
requests whose client the blocklist lists, a series that comes with the first;
`SWWAF_ASN_LIMIT_PERCENT_URL` names, and of the CrowdSec decision list:
`smallwebwaf_reputation_hits_total`: the requests whose client the blocklist
or the decision list lists, a series that comes with the first;
`smallwebwaf_reputation_failures_total`: the fetches of the list that failed;
and `smallwebwaf_reputation_last_fetch_timestamp_seconds`: when the copy of it
in use was fetched, `0` while there is none.
@@ -1633,7 +1674,8 @@ For each request `smallwebwaf`:
the deny list or currently banned;
- looks up its AS number and country, and refuses it if that country is denied,
or is not among the only ones allowed;
- checks it against the blocklists, and for a cached reputation verdict;
- checks it against the blocklists, bans it if the CrowdSec decision list lists
it, and checks it for a cached reputation verdict;
- picks the client's limit percentage from those;
- checks the minute, hour and day request counters against the limits, and bans
the client if it breaks one;
@@ -1944,6 +1986,58 @@ the metrics nor `reputation.json` hold it. Given as a file, with
`SWWAF_ABUSEIPDB_KEY_FILE`, it can be kept out of the app's reach (see "Settings
given as files" above).
## CrowdSec
While `SWWAF_CROWDSEC_LAPI_URL` names the local API of a CrowdSec engine you
run, `smallwebwaf` fetches the engine's decision list, at that URL with
`v1/decisions` added to its path, with `SWWAF_CROWDSEC_LAPI_KEY`, a key the
engine gave, such as `cscli bouncers add smallwebwaf` makes. This is how a list
kept for a whole fleet of hosts can reach `smallwebwaf` without it depending on
CrowdSec. It is unset by default, for the reason no blocklist is named, and
since it needs an engine of your own.
The list is fetched again a minute after it was last fetched or tried, the fetch
failed or not, and is kept as a blocklist is (see "Blocklists" above): its last
good copy, the engine's answer as it came, stays in use while a fetch fails, and
`reputation.json` keeps it with the time it was fetched, so that a restart keeps
it in use too. A fetch fails when the engine answers other than `200`, such as
`403` for a key it does not know, when it does not finish within a minute, when
the answer is longer than 16 MiB or is not a JSON list of decisions, or when a
decision to ban gives a value that is not an address or a netblock, or a
`duration` that does not read. A failure is counted, logged and raised as a
`source_failure` alert, held back as a repeat within `SWWAF_ALERT_COOLDOWN`.
The decisions of the type `ban` on an address or a netblock, the scopes `Ip` and
`Range`, are used; any other, such as one to show a captcha or one on a country,
is left out. A decision bans until its `duration`, the time it had left as the
engine answered, has passed since the fetch, so that one that has ended no
longer bans, even while the copy in use still holds it because the engine cannot
be reached.
A client in `SWWAF_ALLOW_NETS` is not checked, nor is one a blocklist refuses.
Any other is checked by its own address after the blocklists. A request from a
client a decision in force bans is refused with `SWWAF_BAN_RESPONSE`, and bans
the client's netblock, as any ban covers it, until that decision ends, the one
that ends last if several ban the client. The ban's cause is `crowdsec`, and its
reason names the decision's scenario. Only a client that sends a request is
banned, so that a long list does not fill `bans.json`. Such a ban counts toward
`SWWAF_MAX_BANS`, is never made permanent, and does not make the netblock's next
ban for a broken limit longer. The request's log line and the ban's notes name
the list by its URL, as they name a blocklist, and the request raises a
`reputation_hit` alert besides the ban's.
A ban you lift while its decision is in force is followed by another at the
client's next request. To let a client in, delete its decision in CrowdSec, such
as with `cscli decisions delete --ip 203.0.113.9`, and lift the ban once the
list has been fetched again, a minute or so later. A ban made for a decision
deleted in CrowdSec otherwise lasts until that decision would have ended.
The key is sent to the engine in the `X-Api-Key` header, and nowhere else. The
settings logged at start show `********` in its place, and neither the log, the
alerts, the metrics nor `reputation.json` hold it. Given as a file, with
`SWWAF_CROWDSEC_LAPI_KEY_FILE`, it can be kept out of the app's reach (see
"Settings given as files" above).
## How the code is laid out
- `cmd/smallwebwaf`: the binary, which only calls `internal/smallwebwaf`.
@@ -1957,15 +2051,16 @@ given as files" above).
standard library's `httputil.ReverseProxy` within the timeouts and size
limits, and writes the request's log line. Its `check` method is where a
request is refused before anything reaches the app: for `SWWAF_DENY_NETS`, for
a ban, for the country lists, for a blocklist, for a DNSBL zone's verdict, for
AbuseIPDB's score, for a rate limit, which bans the client, for a `block` or
`ban` rule, the latter banning the client, and for an announced body over the
size limit; in `observe` mode, only for the size limit, with what it would
have refused for noted in the log line. A request under `/_smallwebwaf/` that
`check` lets through is answered by `answerAdmin` instead of reaching the app.
Once the answer to a request passed to the app has ended, `countBytes` counts
its bytes for the byte limits, and once any request but the health check has
ended, `countAnomalies` counts it for the anomaly thresholds.
a ban, for the country lists, for a blocklist, for the CrowdSec decision list,
which bans the client, for a DNSBL zone's verdict, for AbuseIPDB's score, for
a rate limit, which bans the client, for a `block` or `ban` rule, the latter
banning the client, and for an announced body over the size limit; in
`observe` mode, only for the size limit, with what it would have refused for
noted in the log line. A request under `/_smallwebwaf/` that `check` lets
through is answered by `answerAdmin` instead of reaching the app. Once the
answer to a request passed to the app has ended, `countBytes` counts its bytes
for the byte limits, and once any request but the health check has ended,
`countAnomalies` counts it for the anomaly thresholds.
- `internal/metrics`: the metrics, counted as the other parts tell it what
happened, and served in the Prometheus text format.
- `internal/bans`: the ban ledger: each netblock's bans with their notes, how
@@ -1978,14 +2073,15 @@ given as files" above).
client's history and to the notes of its bans; or in the lookup database,
which it reads again when the file is replaced. `internal/lookup/lookuptest`
writes lookup databases for the tests.
- `internal/reputation`: fetches the blocklists and the file
`SWWAF_ASN_LIMIT_PERCENT_URL` names as they are due, keeps the last good copy
of each, and tells which blocklists list an address and what percentage the
file gives an AS number; asks the DNSBL zones about clients in the background,
through the standard library's resolver, keeps their verdicts, and tells which
zones' verdicts list an address; and checks clients with AbuseIPDB in the
background, keeps their scores and the checks spent today, and tells whether a
client's score is a hit.
- `internal/reputation`: fetches the blocklists, the file
`SWWAF_ASN_LIMIT_PERCENT_URL` names and the CrowdSec decision list as they are
due, keeps the last good copy of each, and tells which blocklists list an
address, what percentage the file gives an AS number, and which decision of
the decision list bans an address; asks the DNSBL zones about clients in the
background, through the standard library's resolver, keeps their verdicts, and
tells which zones' verdicts list an address; and checks clients with AbuseIPDB
in the background, keeps their scores and the checks spent today, and tells
whether a client's score is a hit.
- `internal/ratelimit`: the table of clients: counts each client's requests and
bytes, tells when they take it over a rate limit or a byte limit, and keeps
each client's history.