sneak/smallwebwaf
1
0
Fork 0
Code Issues 5 Pull Requests 1 Actions 14 Packages Projects Releases Wiki Activity

SPEC: requirements and reasons only, reference material in docs/
check / check (push) Waiting to run
Details

Browse Source 
docs/settings.md is now the one settings reference; the spec keeps the
conventions, the off list and each group's requirements and reasons, and
links to it for the per-setting entries. The endpoint list, the ban
notes, the state file contents, the rule file format and the alert
object likewise live in docs/ and are linked from the spec. The package
list from the README's Design section joins "Architecture". The nine
docs files are byte-identical to those on the README branch. Package
comments that named "Alert webhook schema", "Request log" and "Metrics
endpoint" for their field lists now name the docs files.

Model: fable-5-1
This commit is contained in:
sneak
2026-10-22 21:08:26 +00:00
parent eff99f0b13
commit d712bb1496
15 changed files with 1515 additions and 561 deletions
Download Patch File Download Diff File Expand all files Collapse all files
SPEC.md
+306 -542
View File
File diff suppressed because it is too large Load Diff
docs/admin-endpoints.md
+35
View File
@@ -0,0 +1,35 @@
# Admin endpoints
`smallwebwaf` answers these requests itself, on the app's own address and
through traefik like any other request. Apart from the health check, each needs
a token, sent as `Authorization: Bearer <token>`, and goes through every check
any other request goes through. It is answered where another request would be
passed to the app: a banned client stays refused, and each request counts toward
rate limiting. A missing or wrong token is answered `401`, in `observe` mode
too. Such a refusal counts toward the error burst, so a client guessing tokens
is soon banned. While a token is unset, its endpoints answer `404`.
- `GET /_smallwebwaf/healthz`: `200` and `ok`, to anyone, before any check and
without asking the app.
- `GET /_smallwebwaf/metrics`: the metrics, for `SWWAF_METRICS_TOKEN` (see
[`metrics.md`](metrics.md)).
- `GET /_smallwebwaf/bans`: every ban held, under `bans`, each as `bans.json`
lists it. Needs `SWWAF_ADMIN_TOKEN`, as do the three below.
- `POST /_smallwebwaf/bans`: bans a netblock. The body is a JSON object of
`netblock`, `duration` and, if you like, `reason`. `netblock` is a netblock
such as `203.0.113.0/24`, or a client's address. A client's address bans its
IPv4 netblock (`SWWAF_BAN_SCOPE_V4_PREFIX`) or its IPv6 group. `duration` is a
duration such as `1h` or `7d`, or `permanent`. The ban starts at once, with
the cause `admin`. A bad body is answered `400`, saying what is wrong:
- one that is not such an object, or has another field;
- one longer than 4 KiB;
- one that names an IPv4-mapped netblock, or an address with a zone.
- `DELETE /_smallwebwaf/bans/<client>`: lifts every active ban covering the
address `<client>`. It answers `404` when none is active.
- `GET /_smallwebwaf/clients/<ip>`: what `smallwebwaf` knows of the client.
Under `client`, its counters and history as `clients.json` holds them, or
`null` when it has never been seen. Under `bans`, every ban covering the
address, with its notes.
An admin whose own address is banned lifts that ban by editing `bans.json` (see
[`bans.md`](bans.md)).
docs/alerts.md
+149
View File
@@ -0,0 +1,149 @@
# Alerts
Each alert goes to each destination you name. `SWWAF_ALERT_WEBHOOK_URL` gets it
as one JSON object. `SWWAF_ALERT_SLACK_WEBHOOK_URL` and `SWWAF_ALERT_NTFY_URL`
get a message made from it. The settings are under "Alert destinations" in
[`settings.md`](settings.md). An alert is one of these events, and is sent when
its event is in `SWWAF_ALERT_EVENTS`:
- `ban`: a ban `smallwebwaf` made.
- `permanent_ban`: a permanent ban it made, or an existing ban made permanent by
a request.
- `waf_block`: the Core Rule Set scored a request at or over the threshold. In
`block` mode the request was refused. In `detect` mode it was not, and the
alert's `detail` gives `mode`.
- `anomaly`: a count over an anomaly threshold.
- `reputation_hit`: a request from a client on a blocklist, the CrowdSec
decision list or a DNS blocklist, or with an AbuseIPDB score that is a hit.
One alert per source.
- `source_failure`: one of:
- GeoJS failing;
- a list that cannot be fetched;
- a DNS blocklist or AbuseIPDB failing or refusing a query;
- the day's AbuseIPDB checks used up.
- `file_error`: one of:
- a rule file or state file edit with an error;
- a state file that cannot be written;
- a replacement lookup database that cannot be read.
- `summary`: the hourly summary described under "Repeats and the hourly cap"
below.
The bans you make raise no alert. In `observe` mode, a request that would have
made a ban still raises the ban's alert, with `mode` set to `observe` in its
`detail`.
## The JSON object
This is the alert for a ban for a broken rate limit, shown indented; it is sent
on one line:
```json
{
"instance": "fsn1app1/gitea",
"time": "2026-10-06T12:00:00.123461Z",
"event": "ban",
"client": "203.0.113.9",
"netblock": "203.0.113.9/32",
"asn": "",
"as_name": "",
"country": "",
"reason": "requests per minute over the limit of 1000",
"detail": {
"ban_expires": "2026-10-06T13:00:00.123Z",
"cause": "limit",
"notes": {
"asn": "",
"as_name": "",
"country": "",
"kind": "requests",
"limit": 1000,
"window": "minute",
"count": 1001,
"request": {
"time": "2026-10-06T12:00:00.123456789Z",
"method": "GET",
"host": "app.example",
"path": "/owner/repo/commits/branch/main?page=812",
"status": 403,
"user_agent": "scraper/1.0"
},
"requests": 5210,
"refused": 0,
"earlier_bans": {
"limit": 0,
"attack": 0,
"admin": 0,
"crowdsec": 0
}
}
},
"suppressed_repeats": 0
}
```
- `instance` is `SWWAF_INSTANCE_NAME`. `time` is when the alert was raised.
`event` is one of the events above.
- `client` is the address whose request raised the alert. `netblock` is the
ban's netblock, the netblock an anomaly counted, or the client's group. Both
are empty for `source_failure` and `file_error`. `asn`, `as_name` and
`country` are the client's as known when the alert was raised. They are empty
when GeoJS had not answered yet.
- `reason` is a short sentence, such as
`requests per minute over the limit of 1000` or `listed by a blocklist`.
- `detail` is what is particular to the event:
- for a ban, its `cause`, `ban_expires` and `notes` (see
[`bans.md`](bans.md) for the notes);
- for an anomaly, the `scope`, `window`, `kind`, `count` and `threshold`;
- for a reputation hit, the `source` and, for AbuseIPDB, the `score`;
- for a Core Rule Set match, the `rule_ids`, `score`, `method` and `path`;
- for a source failure, the `source` and `error`;
- for a file error, the `file` and `error`.
- `suppressed_repeats` is how many repeats the cooldown held back before this
alert.
## Slack and ntfy
Slack and ntfy get a message with a title and a text. The title is the instance
and the event, such as `fsn1app1/gitea: ban`. The text is the `reason`, then one
line for each of these that the alert has:
- the client, the netblock and the country;
- the file, the source, the error and the mode from `detail`;
- the suppressed repeats, if any.
Slack gets the title in bold and the text below it, with `&`, `<` and `>`
escaped. ntfy gets the text as the message, with these headers:
- `Title`: the title;
- `Authorization`: the token, as `Bearer <token>`, while
`SWWAF_ALERT_NTFY_TOKEN` is set;
- `Priority` and `Tags`: by event, as listed below.
The priority and tag by event:
- `ban`: priority `default`, tag `no_entry`.
- `permanent_ban`: priority `high`, tag `no_entry`.
- `waf_block`: priority `default`, tag `shield`.
- `anomaly`: priority `high`, tag `chart_with_upwards_trend`.
- `reputation_hit`: priority `low`, tag `label`.
- `source_failure` and `file_error`: priority `high`, tag `warning`.
- `summary`: priority `default`, tag `bar_chart`.
## Repeats and the hourly cap
A repeat is an alert for the same event on the same netblock, source, file or
thing counted, less than `SWWAF_ALERT_COOLDOWN` after the last. It is held back
and counted, and the next alert sent gives the count. Past
`SWWAF_ALERT_MAX_PER_HOUR` alerts in an hour, the rest are held back and counted
by event. Once the hour ends, one alert with the event `summary` gives those
counts. It also counts the repeats that a cooldown held back and that no later
alert reported.
## Sending
Each destination has a queue of at most 1000 alerts, sent one at a time, so a
slow destination holds up nothing. A `2xx` answer takes an alert. A `4xx` other
than `408` and `429` refuses it, and it is dropped. Anything else is a failure:
the alert is sent again a second later, and twice as long after each further
failure, up to a minute. `alerts.json` keeps the queues and cooldowns across a
restart (see [`state-files.md`](state-files.md)).
docs/bans.md
+84
View File
@@ -0,0 +1,84 @@
# Bans
How bans are made, what they cover and how long they last is under "Bans" in
[`SPEC.md`](../SPEC.md); the settings are under "How long bans last and what
they cover" in [`settings.md`](settings.md). This file says what a ban's notes
hold, and how to make, keep or lift a ban.
## The notes
Each ban carries notes for deciding whether to lift it, drawn from what
`smallwebwaf` already knows. Nothing is looked up to fill them:
- the AS number, AS name and country, added when the lookup answers (empty when
lookups are off);
- what was broken: the rule id and target that matched, or the trap path, or the
limit, its window and the count reached;
- the client's percentage of the limit, with the setting that gave it;
- the reputation sources the client is on;
- the request that caused the ban: time, method, host, path with its query
string, status and user agent, each text cut to 256 bytes;
- the netblock's requests since it was first seen, and how many this ban has
refused so far, kept up to date while the ban lasts;
- how many earlier bans the netblock had, by cause.
## Managing bans by editing `bans.json`
`bans.json` in the state directory holds every ban, indented to be read. A
permanent ban's `expires` is `null`. `smallwebwaf` watches the file and takes in
an edit as soon as it is saved (see [`state-files.md`](state-files.md)).
To ban a netblock, add an entry with its `netblock`, `start` and `expires`.
`expires` is `null` for a ban that never ends. `reason`, `notes` and `cause` may
be left out; the cause is then `admin`:
```json
{
"version": 1,
"bans": [
{
"netblock": "203.0.113.0/24",
"start": "2026-10-06T12:00:00Z",
"expires": null,
"reason": "probes for logins"
}
]
}
```
To keep a ban `smallwebwaf` made, so that it is never dropped, set its `cause`
to `admin`. To lift a ban, add `"lifted"` with the time you lift it, such as
`"lifted": "2026-10-06T13:00:00Z"`. The ban then refuses nothing and does not
make the netblock's next ban longer. It is kept with its notes. To forget a ban,
delete its entry. A ban whose cause is `attack` becomes permanent at the first
request it refuses. One whose cause is `admin` or `crowdsec` does not.
An edit that cannot be read does not stop the running `smallwebwaf`. It keeps
what it holds. At the file's next write it renames your file to `bans.json.bad`,
writes the file again from memory, and logs and alerts the error. Mend the
`.bad` file and move it back.
## Managing bans through the admin endpoints
With `SWWAF_ADMIN_TOKEN` set, and the token in `$TOKEN`:
```sh
# Every ban.
curl -H "Authorization: Bearer $TOKEN" https://app.example/_smallwebwaf/bans
# Ban 203.0.113.0/24 for seven days.
curl -H "Authorization: Bearer $TOKEN" \
--json '{"netblock": "203.0.113.0/24", "duration": "7d", "reason": "probes for logins"}' \
https://app.example/_smallwebwaf/bans
# Lift the bans on 203.0.113.9.
curl -H "Authorization: Bearer $TOKEN" -X DELETE \
https://app.example/_smallwebwaf/bans/203.0.113.9
# What smallwebwaf knows of 203.0.113.9.
curl -H "Authorization: Bearer $TOKEN" \
https://app.example/_smallwebwaf/clients/203.0.113.9
```
See [`admin-endpoints.md`](admin-endpoints.md) for what each answers. An admin
whose own address is banned lifts that ban by editing `bans.json`.
docs/core-rule-set.md
+31
View File
@@ -0,0 +1,31 @@
# The Core Rule Set
The OWASP Core Rule Set 4.25.0 runs on each request after the rule files, unless
`SWWAF_WAF_MODE` is `off`. It reads the method, the URL with its query, and the
headers. Once `SWWAF_WAF_BODY_LIMIT` is set, it also reads a body that is form
data, multipart, JSON or XML. No answer is inspected. The settings are under
"Attack detection" in [`settings.md`](settings.md). The same section of
[`SPEC.md`](../SPEC.md) says why bodies are not read by default, and gives the
six changes `smallwebwaf` makes to the Core Rule Set, with the reason for each.
Each rule that matches adds to the request's score. A score at or over
`SWWAF_WAF_ANOMALY_THRESHOLD` is a match. A match is refused with `403` while
`SWWAF_WAF_MODE` is `block`, and let through while it is `detect`. Either way
the log line gives the rule ids and the score, and a `waf_block` alert is sent.
A request with more than 1000 query parameters or body fields adds 5 to its
score, since Coraza reads only the first 1000. The Core Rule Set inspects only a
request that every earlier check let through. It skips a path under
`SWWAF_WAF_EXEMPT_PATHS` and a client in `SWWAF_ALLOW_NETS`. An engine error on
a request forwards the request, and is logged and counted.
Body inspection suits apps whose forms carry no code. In front of a code forge
it refuses issue and comment text, wiki pages and files saved in the web editor
that hold shell commands or code. The log line names the rule ids, which
`SWWAF_WAF_DISABLED_RULES` can switch off. A body the Core Rule Set cannot parse
adds 5 to the score. So does a multipart body cut by the limit in the middle of
a part's header. The client has until `SWWAF_CLIENT_REQUEST_TIMEOUT` runs out to
send the part that is read. Any other body, such as a git push, reaches the app
uninspected.
[`gitea.md`](gitea.md) records what the Core Rule Set still refuses in front of
gitea, with and without body inspection.
docs/lookup.md
+79
View File
@@ -0,0 +1,79 @@
# GeoJS and the lookup database
`smallwebwaf` looks up each client's AS number and country, through GeoJS by
default or in a database file on your own host. The settings are under "Looking
up the AS number and country" in [`settings.md`](settings.md); the reasons for
the defaults are under "Lookup of AS number and country" in
[`SPEC.md`](../SPEC.md).
## GeoJS
By default `smallwebwaf` asks GeoJS, a free web service that needs no account,
about every new visitor's address. It asks whether or not a setting uses the
answer, since the log line, the history, the ban notes and the metrics carry the
AS number and country. These addresses are never sent:
- one in `SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS`;
- one under a ban;
- a private, loopback or link-local address, which no source can place.
For an IPv6 visitor, GeoJS is asked about the first address of its group.
Addresses are asked about together, up to 200 in one request
(`https://get.geojs.io/v1/ip/geo.json?ip=a,b,c`), one request at a time. Each
answer's `country_code`, `asn` and `organization_name` are read. An answer
without a `country_code`, or with the `asn` `64512`, counts as unknown. Each
answer is kept for seven days, in memory and in `lookups.json`, separately from
the table of clients. Up to 100,000 answers are kept; past that, the answer used
longest ago goes first.
A request waits for its client's first answer only when a setting needs the
answer before the request goes on. It waits up to `SWWAF_LOOKUP_TIMEOUT`. The
settings that make it wait:
- a country list;
- `SWWAF_ADD_LOOKUP_HEADERS`;
- any setting under "Lower limits by AS number and country" in
[`settings.md`](settings.md).
Otherwise the request goes on at once, logged without the answer. The answer
reaches the client's history and ban notes when it comes. A client whose answer
has not come in time counts as from an unknown country until it comes, and its
later requests do not wait. While GeoJS fails, new visitors count as from an
unknown country, which `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses. GeoJS is
then left alone for a second, and twice as long after each further failure, up
to five minutes, and one `source_failure` alert is sent per cooldown.
## The database file
To keep your visitors' addresses on your own host, set
`SWWAF_LOOKUP_SOURCE=off`, or set it to `file`. `file` looks every client up in
the free IPinfo Lite database (`ipinfo_lite.mmdb`) at `SWWAF_LOOKUP_DB_PATH`,
reading its `asn`, `as_name` and `country_code` fields. You download the
database with an IPinfo account and mount it into the container; `smallwebwaf`
never fetches it and holds no account or token for it. It answers at once, with
nothing sent off the host. A client whose address it does not hold counts as
from an unknown country.
`smallwebwaf` reads the whole file at start. It reads the file again once it has
gone 2 seconds without a change after you replace it. A replacement it cannot
read is logged and alerted, and the file read before stays in use. Mount the
directory that holds the file, not the file itself. Many editors and download
tools save a file by replacing it, and a container keeps seeing the old file
when only the file is mounted.
IPinfo updates the database daily, and releases it under the Creative Commons
Attribution-ShareAlike 4.0 licence. It asks for a link to IPinfo on the site
that uses it, such as "IP address data is powered by IPinfo".
## A client without a country
A client the lookup cannot place has an unknown country and AS number. That is
one of:
- a client on a private address;
- one GeoJS or the database does not know;
- one whose GeoJS answer has not come in time.
`SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses such a client, and the lower
limits give it `SWWAF_UNKNOWN_LIMIT_PERCENT`. Nothing else treats it
differently.
docs/reputation.md
+113
View File
@@ -0,0 +1,113 @@
# Reputation sources
A reputation source judges a client by what others saw it do. None is on by
default. The settings are under "Reputation sources" in
[`settings.md`](settings.md); the reasons for the defaults are under
"Reputation" in [`SPEC.md`](../SPEC.md). The sources are checked in this order,
and a source the client is on refuses it, lowers its limits, or only notes it,
as `SWWAF_BLOCKLIST_ACTION` and `SWWAF_REPUTATION_ACTION` say.
## Blocklists
`SWWAF_BLOCKLIST_URLS` lists text files of addresses and netblocks, one to a
line, as the Spamhaus DROP list is written. Anything after a `;` or `#` is left
out. An IPv4-mapped address such as `::ffff:192.0.2.0/120` is read as the IPv4
netblock it stands for.
Each list is fetched again `SWWAF_BLOCKLIST_REFRESH` after its last try, one
list at a time. The file of `SWWAF_ASN_LIMIT_PERCENT_URL` is too.
`reputation.json` keeps when each was last tried, so a restart does not fetch a
list again before its time. A fetch fails when:
- the server answers other than `200`;
- the fetch does not finish within a minute;
- the list is longer than 16 MiB;
- a line is not an address or netblock, or, for the AS number file, not
`AS<number>:<percent>`.
The copy fetched before stays in use, comment lines included. The failure is
logged and raised as a `source_failure` alert. The log, the alerts, the ban
notes and the metrics name each list by its URL, so keep a secret out of it.
The Spamhaus DROP list is The Spamhaus Project's. Its terms, and what they mean
for `SWWAF_BLOCKLIST_REFRESH` and for several instances on one host, are under
"Reputation" in [`SPEC.md`](../SPEC.md). The copy in `reputation.json` keeps the
list's date and copyright lines.
## CrowdSec
While `SWWAF_CROWDSEC_LAPI_URL` is set to the local API of a CrowdSec engine you
run, `smallwebwaf` fetches its decision list every minute. It sends
`SWWAF_CROWDSEC_LAPI_KEY` in the `X-Api-Key` header. The list is kept as a
blocklist is kept. The decisions of the type `ban` on an address or a netblock
are used. Any other, such as a captcha or a country, is left out. A decision
lasts for the `duration` it had left when the engine answered.
A request is refused when a decision still in force names its client. The
client's netblock is banned until the decision ends, with the cause `crowdsec`.
Only a client that sends a request is banned, so a long list does not fill
`bans.json`. Such a ban is never made permanent, and does not lengthen the
netblock's next ban for a broken limit. If you lift a ban while the decision is
in force, the client's next request makes another. Delete the decision in
CrowdSec first, such as with `cscli decisions delete --ip 203.0.113.9`, and lift
the ban once the list has been fetched again. A fetch fails when:
- the engine answers other than `200`, a redirect included;
- the fetch does not finish within a minute;
- the answer is longer than 16 MiB;
- the answer is not a list of decisions.
## DNS blocklists
Each zone in `SWWAF_DNSBL_ZONES` is asked about a client's own address in the
background, the first time the client is seen. The request goes on at once, and
so does every request from the client until the zone has answered. The name
asked about is the one RFC 5782 gives: the numbers of an IPv4 address reversed,
or the 32 hex digits of an IPv6 address reversed. `192.0.2.99` is asked about as
`99.2.0.192.dnsbl.dronebl.org`. An answer in `127.0.0.0/8` lists the client, but
one in `127.255.255.0/24` is Spamhaus refusing the query. A name that does not
exist does not list the client. These are failures:
- an answer in `127.255.255.0/24`;
- an answer outside `127.0.0.0/8`;
- a resolver error;
- no answer within `SWWAF_REPUTATION_TIMEOUT`.
A failure is logged and alerted, and the zone is not asked again for a minute.
Each verdict is kept for `SWWAF_REPUTATION_CACHE_TTL`, in memory and in
`reputation.json`. Up to 100,000 verdicts are kept; past that, the one fetched
longest ago goes first. At most 1,000 queries are under way at once.
The key of a Spamhaus zone under `dq.spamhaus.net` is its first label.
`smallwebwaf` shows `********` in its place wherever it names the zone. Only
`reputation.json` keeps the zone with its key.
## AbuseIPDB
While `SWWAF_ABUSEIPDB_KEY` is set, `smallwebwaf` asks AbuseIPDB's check
endpoint for the abuse confidence score of a client, from 0 to 100. Only a
client that has committed an offence is checked. An offence is one of:
- a broken limit;
- a `ban` rule match or a trap path;
- a refusal by a `block` rule, the Core Rule Set or a token check.
The check runs in the background, at the client's first request after the
offence that no ban refuses, so no request waits. The score applies to the
client's later requests. For an IPv6 client, AbuseIPDB is asked about the
address of the request that set off the check, and the score is used for the
whole group. A score at or over `SWWAF_ABUSEIPDB_MIN_SCORE` is a hit, and
`SWWAF_REPUTATION_ACTION` says what is done. Each score is kept for
`SWWAF_REPUTATION_CACHE_TTL`, up to 100,000 scores.
At most `SWWAF_ABUSEIPDB_DAILY_BUDGET` checks are made in a day, counted in UTC
and kept in `reputation.json`. A restart does not make the budget whole again.
The check that spends the last of the budget is logged and alerted. No client is
checked until the day ends. A check fails when:
- AbuseIPDB answers other than `200`;
- the answer gives no score;
- no answer comes within `SWWAF_REPUTATION_TIMEOUT`.
A failure is logged and alerted, and no client is checked for a minute. The key
is sent in the `Key` header and nowhere else, and is never logged.
docs/rules.md
+114
View File
@@ -0,0 +1,114 @@
# Rule files
`smallwebwaf` reads every `*.rules` file in `SWWAF_RULES_DIR`, in the order of
their names, and checks each request against their rules in that order, unless
the client is in `SWWAF_ALLOW_NETS`. The requirements the files meet are under
"Rule files" in [`SPEC.md`](../SPEC.md). A file whose name starts with `.`, such
as an editor's lock file, is not a rule file.
## The format
A rule is a line of four fields separated by spaces or tabs: an id, a target, an
action and a regex. The regex runs to the end of the line. Blank lines and lines
starting with `#` are ignored.
```
# id target action regex
env-file path ban (?i)^/\.env(\.[a-z]+)?$
scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|wpscan)\b
```
- The id is letters, digits, `-` and `_`. No two rules share one. The log line,
the metrics, the alerts and a ban's notes name the rule by it.
- The target is what the regex is matched against:
- `path`: the path without the query;
- `uri`: the path and the query together;
- `query`: the query as sent;
- `method`, `host`, `user_agent` and `referer`;
- `header:<Name>`: any one request header but `Host` and
`Transfer-Encoding`, which Go's HTTP server takes out of every request.
`path` and `uri` are matched both as sent and once percent-decoded, so that
an encoded probe does not slip past. A header sent more than once is matched
with its values joined by `, `. A header not sent is empty text. No body is
read; reading bodies is the Core Rule Set's job, once `SWWAF_WAF_BODY_LIMIT`
switches it on.
- The action is `log`, `block` or `ban`. `log` only notes the match in the log
line. `block` refuses the request with `403` and bans no one; the refusal
counts toward the error burst. `ban` refuses it and bans the client, as a
clear sign of attack. Matching stops at the first rule that refuses. Keep
`ban` for requests no real visitor sends. Anchor a path at the site root with
`^/`: a file of the same name deeper in a site can be ordinary content.
- The regex is in Go's syntax, RE2. It has no backreferences or lookaround, and
takes time linear in the text it reads, so no rule can be made to stall the
proxy. It matches anywhere in the target unless anchored with `^` and `$`.
`(?i)` at its front makes it ignore case.
## Errors
These stop the start, with a message naming the file and line:
- a line that is not a rule;
- a bad `header:` name, or a rule for `Host` or `Transfer-Encoding`;
- a regex that does not compile;
- an id used twice.
So does a `SWWAF_RULES_DIR` that does not exist. An empty directory is not an
error: the log says that no rules were loaded.
While it runs, `smallwebwaf` watches the directory. It reads the files again
once the directory has had no change for 2 seconds, so that a file still being
written is read whole. If the files then hold such an error, the rules stay as
they were. The log and a `file_error` alert name the file and line. Once the
file is fixed, it is read again.
## The default file
The image ships one file, `share/rules.d/00-default.rules`, installed as
`/etc/smallwebwaf/rules.d/00-default.rules`:
```
# 00-default.rules: probes no real visitor sends, anchored at the site root
# id target action regex
env-file path ban (?i)^/\.env(\.[a-z]+)?$
vcs-dir path ban (?i)^/\.(git|svn|hg|bzr)(/|$)
secrets-dir path ban (?i)^/\.(aws|ssh|docker|kube)/
secret-file path ban (?i)^/\.(htpasswd|htaccess|npmrc|netrc|pgpass|git-credentials|bash_history|DS_Store)$
editor-dir path ban (?i)^/\.(vscode|idea)/
backup-file path ban (?i)^/[^/]+\.(php(\.[a-z0-9]+|~)|sql(\.[a-z0-9]+)?)$
log-file path ban (?i)^/(debug|error|access)\.log$
compose-file path ban (?i)^/(docker-)?compose\.ya?ml$
php-shell path ban (?i)^/(shell|c99|r57|wso|alfa)\.php$
scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|masscan|zgrab|wpscan)\b
path-traversal uri block (\.\./){2,}
empty-agent user_agent log ^$
```
It is kept short and limited to patterns that are wrong for every app, with its
path rules anchored at the site root. In front of gitea, `/.env` is a probe,
while `/<owner>/<repo>/src/branch/main/.env.example` is a file in a repository
that any visitor may open. Anything app-specific belongs in a file the app's
`Dockerfile` adds or the deployer mounts. A request for `/wp-login.php`, for
example, is a clear sign of attack in front of gitea and an ordinary login in
front of WordPress. Such a rule goes in a file named to sort after the default:
```
# 50-gitea.rules: WordPress probes, which gitea never serves
wp-probe path ban (?i)^/(wp-login\.php|xmlrpc\.php|wp-admin/)
```
```dockerfile
COPY 50-gitea.rules /etc/smallwebwaf/rules.d/50-gitea.rules
```
## Mounting rule files
A directory mounted over `/etc/smallwebwaf/rules.d` replaces the default file;
single files mounted into it add to it. Many editors save a file by replacing
it, and a container keeps seeing the old one when the file alone is mounted. So
rules to be edited while `smallwebwaf` runs belong in a mounted directory, with
a copy of the default file if the defaults are to stay. The lookup database is
mounted the same way, for the same reason. To run without rules, mount an empty
directory or set `SWWAF_RULES_ENABLED=false`.
docs/settings.md
+483
View File
@@ -0,0 +1,483 @@
# Settings
Every setting is an environment variable whose name starts with `SWWAF_`. Every
setting has a default, so none has to be set. Settings are read once, at start.
To change one, restart the container.
A setting that is set to a value it cannot take stops the start. The process
writes one line, `"type":"process"` with `"msg":"invalid setting"`, naming the
variable and what is wrong with the value. It exits with status 1, and runit
starts it again every second until the value is fixed. The entries below say
which values each setting takes. Any other value stops the start. At start, the
settings in effect are logged on one line. Tokens, keys and the paths of
webhooks are shown as `********`.
The forms of values:
- A duration is Go's syntax, with `d` for days: `90s`, `15m`, `24h`, `7d`. Zero
and negative durations are refused.
- A size is a number of bytes with an optional `K`, `M` or `G`, which are powers
of 1024: `1K` is 1024 bytes, `512K`, `100M`, `5G`.
- A count is a whole number of requests, such as `1000`.
- `off` switches a setting off, where its entry below says it takes `off`.
- A list is comma separated, with spaces around items ignored. An empty item is
refused. A list set but empty is an empty list, and replaces the default.
- A netblock is an address and a prefix length, such as `10.0.0.0/8`. A bare
address stands for itself alone.
- A country is a two-letter code of ISO 3166-1, in either case: `de` and `DE`
are the same. `xk` stands for Kosovo. A code that is not a country is refused,
such as `nk` (North Korea is `kp`) or the withdrawn `su`.
- An AS number is `AS` and the number, in either case, such as `AS64496`.
- A percentage is a whole number from 0 to 100.
- A path prefix starts with `/`, such as `/assets/`.
## Where it listens and what it protects
- `SWWAF_LISTEN_ADDR` (default `:8080`): where `smallwebwaf` listens, an
optional host and a port. Keep the host empty: `:9000`, never
`127.0.0.1:9000`. The app must not listen on that port. The health check
follows the setting, and traefik's label must name the port.
- `SWWAF_UPSTREAM_URL` (default `http://127.0.0.1:8081`): the app. An `http` or
`https` URL with a host and an optional port, and nothing more: no path, user
or query.
- `SWWAF_INSTANCE_NAME` (default: the container's host name, the first 12
characters of the container's id): the `instance` name in every log line,
alert and metric. Set it to a name that survives deploys and tells instances
apart, such as `fsn1app1/gitea`. It must be valid UTF-8. While
`SWWAF_ALERT_NTFY_URL` is set, it may hold no control character. While
`SWWAF_LOG_REMOTE_URL` is set and `SWWAF_LOG_REMOTE_APP_NAME` is not, it must
be 1 to 48 printable ASCII characters without a space.
- `SWWAF_MODE` (default `enforce`): `enforce`, or `observe` to ban no one and
log what `enforce` would have done. Observe mode still refuses a request over
a size or time limit, and one without the right token.
- `SWWAF_TRUSTED_PROXIES` (default `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`,
the private address ranges): the netblocks whose `X-Forwarded-For` is
believed, normally where traefik comes from. Set but empty, it trusts nothing:
every TCP peer is then the client.
- `SWWAF_IPV6_GROUP_PREFIX` (default `64`): the length of the netblock that is
one IPv6 client, from 32 to 128. After a change, each IPv6 client starts
afresh. Existing bans keep refusing their netblocks until they end.
- `SWWAF_MAX_TRACKED_CLIENTS` (default `20000`): the most clients kept in memory
and in `clients.json`, a whole number above zero. Past it, the least recently
seen client is dropped first.
## Time and size limits
Each request has limits on both sides: between the client and `smallwebwaf`, and
between `smallwebwaf` and the app. Bodies stream through in both directions,
except the part the Core Rule Set reads while `SWWAF_WAF_BODY_LIMIT` is set.
- `SWWAF_CLIENT_REQUEST_TIMEOUT` (default `60s`): how long a client may take to
send its request line and headers, and then, from the end of the headers, its
body. A duration or `off`.
- `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES` (default `32K`): the largest request
line and headers. A size over `4K`. It cannot be `off`: Go's HTTP server
always has such a limit.
- `SWWAF_CLIENT_IDLE_TIMEOUT` (default `120s`): how long a kept-open connection
may wait for its next request before it is closed. A duration or `off`. The
default is longer than the 90 seconds after which traefik closes an unused
connection, so traefik closes first.
- `SWWAF_CLIENT_RESPONSE_TIMEOUT` (default `30m`): how long the answer may take
to reach the client, from the end of the request to the last byte. A duration
or `off`.
- `SWWAF_UPSTREAM_REQUEST_TIMEOUT` (default `60s`): how long connecting to the
app and sending it the whole request may take. A duration or `off`.
- `SWWAF_UPSTREAM_RESPONSE_TIMEOUT` (default `30m`): how long the app may take
to send its whole answer. A duration or `off`.
- `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body. A size
or `off`. A request that announces a larger body is refused before anything
reaches the app.
- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body. A size
or `off`.
In front of a code forge, a clone is a response and a push is a request. At the
defaults, an upload over 100 MiB or slower than 60 seconds is cut off. A forge
that takes large uploads needs `SWWAF_REQUEST_MAX_BYTES`,
`SWWAF_CLIENT_REQUEST_TIMEOUT` and `SWWAF_UPSTREAM_REQUEST_TIMEOUT` raised to
fit.
What the client gets when a limit is passed is under "Size and time limits" in
[`SPEC.md`](../SPEC.md).
## Netblock lists
- `SWWAF_ALLOW_NETS` (default empty): netblocks whose clients skip every check,
such as your monitoring or your own network. The size and time limits still
apply to them. They are not looked up. They still need a token for the admin
endpoints.
- `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients are
exempt from rate limiting and the byte limits, such as a machine that talks to
the app all day. Everything else still applies to them.
- `SWWAF_DENY_NETS` (default empty): netblocks whose clients are always refused.
A client in both lists is let through.
## Rate limiting and byte limits
- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `1000`), `SWWAF_RATE_LIMIT_PER_HOUR`
(default `10000`) and `SWWAF_RATE_LIMIT_PER_DAY` (default `50000`): the most
requests a client may make in each window. A count or `off`. The defaults are
several times what one busy person produces. A browser loading a heavy page
makes a few hundred requests, and several people often share one address.
- `SWWAF_RATE_LIMIT_EXEMPT_PATHS` (default empty): path prefixes exempt from
rate limiting, such as `/assets/`. Their bytes are not counted either. The
path is matched exactly as sent, without the query, and is not
percent-decoded. `/assets/` matches `/assets/app.js`. It does not match
`/Assets/app.js`, `/%61ssets/app.js` or `/static/assets/app.js`. There are no
wildcards. A path holding `..`, a backslash or an encoded slash (`%2F`) is
never exempt, since the app may read it as a path outside the prefix.
- `SWWAF_BYTES_LIMIT_PER_MINUTE` (default `10G`), `SWWAF_BYTES_LIMIT_PER_HOUR`
(default `20G`) and `SWWAF_BYTES_LIMIT_PER_DAY` (default `50G`): the most
bytes a client may have counted in each window. A size or `off`. Bytes are
counted once the answer has ended, so a byte limit never cuts an answer short.
The client's next request is refused instead. Each default is above the
largest request and answer together, so at the defaults no single download
breaks a limit.
- `SWWAF_BYTES_COUNT` (default `both`): which body bytes the byte limits count:
`response`, `request` or `both`. For a WebSocket, what it carried each way
counts once it closes.
## Looking up the AS number and country
- `SWWAF_LOOKUP_SOURCE` (default `geojs`): `geojs`, `file` or `off`. `geojs`
asks the GeoJS web service, which is then told the address of every new
visitor. `file` looks clients up in the IPinfo Lite database file at
`SWWAF_LOOKUP_DB_PATH`, and sends nothing off the host. `off` looks up no
client. With `off`, a setting that needs lookups stops the start:
- a country list;
- `SWWAF_ADD_LOOKUP_HEADERS`;
- an entry in `SWWAF_ASN_LIMIT_PERCENT`, `SWWAF_COUNTRY_LIMIT_PERCENT`,
`SWWAF_ASN_BYTES_PERCENT` or `SWWAF_COUNTRY_BYTES_PERCENT`;
- `SWWAF_UNKNOWN_LIMIT_PERCENT` below 100;
- `SWWAF_ASN_LIMIT_PERCENT_URL`;
- an anomaly threshold per AS number.
- `SWWAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in its
`.mmdb` form, for `SWWAF_LOOKUP_SOURCE=file`. `file` without it stops the
start, and so does setting it with any other source. A file that is missing or
cannot be read stops the start too.
- `SWWAF_LOOKUP_TIMEOUT` (default `1s`): how long a request waits for its
client's first answer from GeoJS when a setting needs it, and how long a
request to GeoJS may take. A duration; it cannot be `off`.
- `SWWAF_ADD_LOOKUP_HEADERS` (default `false`): `true` passes the app the
client's AS number in `X-Client-ASN` and its country in `X-Client-Country`. A
header whose value is unknown is left out. Whatever this setting says, any
such header the client sent is removed.
See [`lookup.md`](lookup.md) for what each source sends where, and how long
answers are kept.
## Country lists
- `SWWAF_DENIED_COUNTRIES` (default empty): countries whose clients are refused,
such as `cn,ru,kp`.
- `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` (default empty): when set, the only
countries whose clients get through, such as `us,de`. A client whose country
cannot be found is refused too. Otherwise every new client would be let in
whenever GeoJS stops answering. A client on a private address has no country,
so it is refused unless it is in `SWWAF_ALLOW_NETS`.
A country on both lists stops the start. What a refused request gets, and why
the lists are empty by default, is under "Country lists" in
[`SPEC.md`](../SPEC.md).
## Lower limits by AS number and country
Clients of the AS numbers and countries listed here get a percentage of every
rate limit and byte limit. A lowered limit is rounded down. "Biased thresholds"
in [`SPEC.md`](../SPEC.md) says why the lists are empty by default and why no
budget is shared by a whole AS number or country.
- `SWWAF_ASN_LIMIT_PERCENT` (default empty): a list of AS numbers, each with `:`
and a percentage, such as `AS14061:50,AS16276:50,AS45102:25`. An AS number
listed twice is refused.
- `SWWAF_COUNTRY_LIMIT_PERCENT` (default empty): the same by country, such as
`cn:25,ru:50`.
- `SWWAF_ASN_BYTES_PERCENT` and `SWWAF_COUNTRY_BYTES_PERCENT` (default empty):
the same for the byte limits alone. For the AS numbers and countries they
list, they take the place of the two settings above.
- `SWWAF_UNKNOWN_LIMIT_PERCENT` (default `100`): the percentage of every limit
for a client without a country:
- one the lookup cannot place;
- one on a private address;
- one whose answer from GeoJS has not come in time.
- `SWWAF_ASN_LIMIT_PERCENT_URL` (default unset): the `http` or `https` URL of a
text file of such entries, one to a line, such as `AS14061:50`. One list can
then serve several instances. It is fetched and kept as a blocklist is. Where
it and `SWWAF_ASN_LIMIT_PERCENT` both give a percentage, the lower applies. A
URL that is also in `SWWAF_BLOCKLIST_URLS` stops the start.
A client to which several percentages apply gets the lowest. The log line of
each counted request, and the notes of each ban, say which setting lowered the
limit.
## Reputation sources
None is on by default. A reputation source judges a client by what others saw it
do. The defaults judge it by what it does to your service.
- `SWWAF_BLOCKLIST_URLS` (default empty): blocklists to fetch, such as
`https://www.spamhaus.org/drop/drop.txt`. Each is an `http` or `https` URL
without a user or a fragment. A URL listed twice is refused.
- `SWWAF_BLOCKLIST_REFRESH` (default `24h`): how long a list waits after its
last fetch, or failed try, before it is fetched again. A duration of `1h` or
more; it cannot be `off`. The Spamhaus lists may be fetched no more than once
an hour.
- `SWWAF_BLOCKLIST_ACTION` (default `deny`): what is done with a client on a
blocklist. `deny` refuses its requests. `limit:<percent>`, such as `limit:25`,
gives it that percentage of every limit. `log` only notes the list in the log
line and raises an alert.
- `SWWAF_DNSBL_ZONES` (default empty): the DNS blocklists to ask about each
client, such as `dnsbl.dronebl.org`. A DNS blocklist answers a DNS query for a
name made from an address, and so says whether the address is on it. A zone is
a DNS name of at most 189 characters. A Spamhaus zone carries its key, as in
`<key>.xbl.dq.spamhaus.net`, and is shown with the key masked. A zone listed
twice is refused, whatever its case or key. Do not name a zone meant for mail:
such a zone holds the residential addresses ordinary visitors come from.
- `SWWAF_DNSBL_RESOLVER` (default unset): the resolver to ask the zones through,
such as `192.0.2.53` or `[2001:db8::53]:5353`. An IP address with an optional
port, 53 when none is given. Unset, the host's resolver is used. Several zones
refuse queries that come through a public resolver.
- `SWWAF_ABUSEIPDB_KEY` (default unset): the key of your AbuseIPDB account.
While it is set, a client that has committed an offence is checked. A control
character in it is refused.
- `SWWAF_ABUSEIPDB_MIN_SCORE` (default `75`): the least abuse confidence score,
from 0 to 100, that is a hit.
- `SWWAF_ABUSEIPDB_DAILY_BUDGET` (default `900`): the most checks made in a day,
in UTC, a whole number above zero. AbuseIPDB's free accounts may make 1,000.
- `SWWAF_CROWDSEC_LAPI_URL` (default unset): the local API of a CrowdSec engine
you run, such as `http://172.17.0.1:8080`. An `http` or `https` URL without a
user or a fragment. Its decision list is at `v1/decisions` under that URL, and
is fetched every minute. Set without the key, it stops the start. So does a
decision list URL that is also in `SWWAF_BLOCKLIST_URLS`, or that is
`SWWAF_ASN_LIMIT_PERCENT_URL`.
- `SWWAF_CROWDSEC_LAPI_KEY` (default unset): the key the engine gave for
`smallwebwaf`, as `cscli bouncers add smallwebwaf` prints it. Set without the
URL, it stops the start.
- `SWWAF_REPUTATION_ACTION` (default `limit:25`): what is done with a client on
a DNS blocklist, or one whose AbuseIPDB score is a hit. `deny`,
`limit:<percent>` or `log`, as for `SWWAF_BLOCKLIST_ACTION`. Such a verdict is
less certain than the Spamhaus DROP list, so by default the client gets a
quarter of every limit.
- `SWWAF_REPUTATION_CACHE_TTL` (default `24h`): how long a zone's verdict or
AbuseIPDB's score is used. A duration; it cannot be `off`.
- `SWWAF_REPUTATION_TIMEOUT` (default `2s`): how long a query to a zone or a
check with AbuseIPDB may take. A duration; it cannot be `off`.
See [`reputation.md`](reputation.md) for how each source is asked and what it
does with the answer.
## Attack detection
- `SWWAF_RULES_DIR` (default `/etc/smallwebwaf/rules.d`): the directory of the
rule files (see [`rules.md`](rules.md)). A directory that cannot be read stops
the start. An empty one is no error.
- `SWWAF_RULES_ENABLED` (default `true`): `false` reads no rule file.
- `SWWAF_TRAP_PATHS` (default empty): paths the app never serves and only
scanners ask for, such as `/wp-login.php,/xmlrpc.php` in front of a code
forge. A request for one bans the client, as a `ban` rule does. Each starts
with `/` and holds no `?`. A trap path is matched whole, without the query,
against the path as sent and once percent-decoded. `/wp-login.php` matches
`/wp-login.php?x=1` and `/%77p-login.php`. It does not match `/wp-login.php/`
or `/blog/wp-login.php`.
- `SWWAF_WAF_MODE` (default `block`): what the Core Rule Set does with a request
it takes for an attack. `block` refuses it with `403`. `detect` lets it
through, logged and alerted. `off` inspects no request.
- `SWWAF_WAF_PARANOIA_LEVEL` (default `1`): the Core Rule Set's paranoia level,
from 1 to 4. Each level up runs more rules. Those find more attacks, and take
more ordinary requests for attacks.
- `SWWAF_WAF_ANOMALY_THRESHOLD` (default `5`): the score at which a request is a
match. Each rule that matches adds to the request's score, and a rule rated
critical adds 5. So by default one such rule is enough. A whole number above
zero, or `off`. With `off` no request is a match, and scores are still logged.
- `SWWAF_WAF_DISABLED_RULES` (default
`920340,920420,920440,920640,930130,930140`): the ids of Core Rule Set rules
to switch off. The log line names the rules a request matched, so a rule that
refuses your app's ordinary requests can be found there. "Attack detection" in
[`SPEC.md`](../SPEC.md) says what the six default ids refuse and why they are
off. A list you give replaces the default, so name these six ids in it to keep
them off. An id must be a whole number above zero. An id from 900000 to 900999
is refused: those are the rules that set the Core Rule Set up, and
`smallwebwaf`'s own.
- `SWWAF_WAF_EXEMPT_PATHS` (default empty): path prefixes the Core Rule Set does
not inspect, matched as `SWWAF_RATE_LIMIT_EXEMPT_PATHS` is.
- `SWWAF_WAF_BODY_LIMIT` (default `off`): `off` reads no request body. A size,
at most `1G`, switches body inspection on. The Core Rule Set then reads a form
data or multipart body up to that size, and the rest streams on to the app. It
reads a JSON or XML body no larger than that size, since those cannot be read
in part. Any other body, such as a git push, reaches the app uninspected. See
[`core-rule-set.md`](core-rule-set.md).
- `SWWAF_ERROR_BURST_THRESHOLD` (default `30`): the most refusals of one client
in a minute for a rule match, a trap path, a Core Rule Set match or a failed
token check. One more bans the client, as a broken rate limit does. A count or
`off`. The app's own `401` and `404` answers are not counted.
## How long bans last and what they cover
- `SWWAF_BAN_RESPONSE` (default `403`): how a refused client is answered: `403`,
`429`, or `close` to close the connection without an answer. A `block` rule
and the Core Rule Set always answer `403`.
- `SWWAF_LIMIT_BAN_DURATION` (default `1h`): the ban for a first broken rate
limit, byte limit or error burst. A duration; it cannot be `off`.
- `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): a client that breaks a limit
again within this time after its ban ended is banned three times as long. A
duration; it cannot be `off`.
- `SWWAF_MAX_BAN_DURATION` (default `7d`): a ban for a broken limit that would
be longer is permanent instead. A duration; it cannot be `off`.
- `SWWAF_ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of
attack. A duration; it cannot be `off`.
- `SWWAF_MAX_BANS` (default `5000`): the most bans `smallwebwaf` made that are
kept, whether past, active or permanent. A whole number above zero. Bans whose
cause is `admin` are kept besides and never dropped (see
[`bans.md`](bans.md)).
- `SWWAF_BAN_SCOPE_V4_PREFIX` (default `32`): the length of the netblock around
an IPv4 client that a ban covers, from 0 to 32. `24` bans the surrounding /24.
An IPv6 ban covers the client's group.
## Where and how often the state files are written
- `SWWAF_STATE_DIR` (default `/var/lib/smallwebwaf`): the directory of the state
files, an absolute path. A directory that does not exist or cannot be written
stops the start.
- `SWWAF_STATE_WRITE_DELAY` (default `10s`): how long `bans.json` waits to be
written after a ban is made, lifted or made permanent. Every change in between
goes into the same write. A duration; it cannot be `off`.
- `SWWAF_STATE_COUNTER_INTERVAL` (default `15m`): how often every state file is
written. A duration; it cannot be `off`.
## Logging
- `SWWAF_LOG_REQUEST_HEADERS` (default
`accept,accept-language,accept-encoding,content-type,origin,range`): the
request headers to record in the log line, in either case. `Authorization`,
`Cookie` and `Set-Cookie` are never logged, even if listed. `Host` and
`Transfer-Encoding` are refused: Go's HTTP server takes them out of the
request. The host is the field `host`.
- `SWWAF_LOG_LEVEL` (default `info`): the least severe of `smallwebwaf`'s own
messages that are written: `debug`, `info`, `warn` or `error`. No message is
at `debug`. It holds back no request log line. `warn` and `error` hold back
the line of settings at start.
- `SWWAF_LOG_REMOTE_URL` (default unset): a syslog server every line on stdout
is also sent to, such as `syslog+tls://logs.example:6514`. The forms are
`syslog+udp://`, `syslog+tcp://` and `syslog+tls://`, each with a host and a
port and nothing more.
- `SWWAF_LOG_REMOTE_TLS_CA_FILE` (default unset): a file of PEM certificates
that a `syslog+tls` server's certificate must chain to, instead of the host's
own. A file that cannot be read, or holds no certificate, stops the start.
- `SWWAF_LOG_REMOTE_BUFFER` (default `10000`): the most lines held while they
wait to be sent, a whole number above zero.
- `SWWAF_LOG_REMOTE_FACILITY` (default `local0`): the syslog facility: `kern`,
`user`, `mail`, `daemon`, `auth`, `syslog`, `lpr`, `news`, `uucp`, `cron`,
`authpriv`, `ftp`, or `local0` to `local7`.
- `SWWAF_LOG_REMOTE_APP_NAME` (default: `SWWAF_INSTANCE_NAME`): the app name in
each syslog record, 1 to 48 printable ASCII characters without a space.
## Tokens and metrics
- `SWWAF_ADMIN_TOKEN` (default unset): the token for the ban endpoints and
`/_smallwebwaf/clients/<ip>`. A long random value of at least 32 characters.
Unset, those endpoints are off. Give it as a file to keep it from the app (see
"Settings given as files" below).
- `SWWAF_METRICS_TOKEN` (default unset): the token a scraper sends for
`/_smallwebwaf/metrics`, at least 32 characters. Unset, the metrics are off.
It is separate from the admin token, so a scraper cannot manage bans.
- `SWWAF_METRICS_TOP_N` (default `50`): how many AS numbers and how many
countries get series of their own in the metrics. The rest count as `other`. A
whole number above zero.
## Alert destinations
- `SWWAF_ALERT_WEBHOOK_URL` (default unset): the webhook that receives each
alert as a JSON object. An `http` or `https` URL without a user or a fragment.
The settings logged at start show its path and query as `********`, since many
webhooks carry their secret there.
- `SWWAF_ALERT_WEBHOOK_HEADERS` (default empty): headers sent with each post,
such as `Authorization:Bearer 0123456789abcdef`. A list of name, `:` and
value. A value cannot hold a comma.
- `SWWAF_ALERT_SLACK_WEBHOOK_URL` (default unset): the Slack incoming webhook
that receives each alert as a message. It is checked and masked as the webhook
URL is.
- `SWWAF_ALERT_NTFY_URL` (default unset): the ntfy topic that receives each
alert, as the topic's full URL, such as `https://ntfy.sh/my-alerts`. It is
checked and masked as the webhook URL is.
- `SWWAF_ALERT_NTFY_TOKEN` (default unset): an ntfy access token, sent as
`Authorization: Bearer <token>`. A control character in it is refused.
- `SWWAF_ALERT_EVENTS` (default
`ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure,file_error`):
the events alerts are sent for. Any other name is refused.
- `SWWAF_ALERT_COOLDOWN` (default `15m`): how long a repeat of an alert is held
back. A duration or `off`.
- `SWWAF_ALERT_MAX_PER_HOUR` (default `60`): the most alerts sent in an hour.
The rest are rolled into one summary. A whole number above zero, or `off`.
See [`alerts.md`](alerts.md) for the events and what each destination gets.
## Anomaly thresholds
These only alert. They refuse and ban nothing. Each scope has four thresholds:
the most requests and the most bytes counted in a minute and in an hour.
Requests are counts, bytes are sizes. Each is `off` by default. What is unusual
depends on each service's normal traffic, which the metrics show. The four
thresholds per client are:
- `SWWAF_ANOMALY_CLIENT_REQUESTS_PER_MINUTE`;
- `SWWAF_ANOMALY_CLIENT_REQUESTS_PER_HOUR`;
- `SWWAF_ANOMALY_CLIENT_BYTES_PER_MINUTE`;
- `SWWAF_ANOMALY_CLIENT_BYTES_PER_HOUR`.
The other scopes' names have another prefix before the same four endings, such
as `REQUESTS_PER_MINUTE`:
- `SWWAF_ANOMALY_NET_...`: per netblock around a client. Two settings give the
netblock's length:
- `SWWAF_ANOMALY_NET_V4_PREFIX` (default `24`, from 0 to 32) for an IPv4
client;
- `SWWAF_ANOMALY_NET_V6_PREFIX` (default `48`, from 0 to 128) for an IPv6
client.
- `SWWAF_ANOMALY_ASN_...`: per AS number. Needs lookups on.
- `SWWAF_ANOMALY_TOTAL_...`: the whole service.
- `SWWAF_WATCH_...`: each netblock in `SWWAF_WATCH_NETS`, as a whole.
`SWWAF_WATCH_NETS` (default empty) is a list of name, `=` and netblock, such
as `office=203.0.113.0/24,scraper-x=198.51.100.0/22`. An item without `=` or a
name, or with a bad netblock, is refused. So is a name used twice.
Every request but the health check is counted, whatever was done with it,
clients in `SWWAF_ALLOW_NETS` included. At most 20,000 counters are kept.
`alerts.json` keeps them across a restart.
## Settings given as files
Any setting may be given as a file holding its value. Add `_FILE` to the
setting's name: `SWWAF_METRICS_TOKEN_FILE` names the file that holds
`SWWAF_METRICS_TOKEN`. The file's contents are the value, less one newline at
the end. They are checked as the value itself would be. Setting both forms stops
the start, and so does naming a file that cannot be read.
`SWWAF_LOG_REMOTE_TLS_CA_FILE` already holds a file name, so it has no `_FILE`
form.
A token given as a file is out of the app's reach only while the `smallwebwaf`
user alone can read the file. Make it on the host, owned by uid 65532 with mode
`0400`:
```sh
mkdir -p /srv/app/tokens
openssl rand -hex 32 > /srv/app/tokens/metrics
chown 65532:65532 /srv/app/tokens/metrics
chmod 0400 /srv/app/tokens/metrics
```
Mount its directory into the container read-only, and name the file:
```sh
docker run \
-v /srv/app/tokens:/etc/smallwebwaf/tokens:ro \
-e SWWAF_METRICS_TOKEN_FILE=/etc/smallwebwaf/tokens/metrics \
...
```
## Fixed limits
A few limits are fixed rather than settings:
- at most 100,000 GeoJS answers are kept, for 7 days each;
- at most 100,000 DNS blocklist verdicts and 100,000 AbuseIPDB scores are kept;
- at most 1,000 DNS blocklist queries are under way at once;
- the CrowdSec decision list is fetched again a minute after the last fetch.
docs/state-files.md
+102
View File
@@ -0,0 +1,102 @@
# State files
`smallwebwaf` keeps its state in memory, and a copy of it in five JSON files in
`SWWAF_STATE_DIR`. The files are meant for people as well: you can read or edit
them at any time, and the running `smallwebwaf` takes the edit in. The
requirements they meet are under "Persistent state" in [`SPEC.md`](../SPEC.md).
## The files
Each has a top-level `version` of 1.
- `bans.json`: active and past bans, up to `SWWAF_MAX_BANS` that `smallwebwaf`
made and every ban you made, indented to be read. Per entry:
- the netblock, start and expiry (`null` for permanent);
- the cause: `attack`, `limit`, `admin` or `crowdsec`;
- a short reason;
- when it was lifted, if you lifted it;
- the notes (see [`bans.md`](bans.md)).
- `clients.json`: per client, its minute, hour and day counters and its history
since it was first seen, one client per line, so `grep` shows everything about
one. The history holds:
- the first and last time seen;
- the AS number, AS name and country last looked up, and when;
- total requests, and how many were forwarded and how many refused;
- bytes in each direction;
- responses by status class;
- offences by kind: `limit`, `attack`, `rule_blocked`, `waf_blocked` and
`token_refused`.
Clients that were never banned are kept too, so every client's history
survives a restart.
- `lookups.json`: GeoJS answers, one per line: the AS number, AS name and
country, when GeoJS was asked and when the answer was last used.
- `reputation.json`:
- the last good copy of each list fetched from a URL, with when it was last
tried;
- each DNS blocklist verdict still in use;
- AbuseIPDB's scores, and the checks spent today.
- `alerts.json`:
- the anomaly counters;
- the alert cooldowns: for each event and netblock, source, file or thing
counted, when it was last sent and the count of repeats held back since;
- the alerts sent and held back in the current hour;
- the alerts waiting for each destination.
Times are RFC 3339 UTC. Durations and sizes are plain numbers, with the unit in
the field name. `bans.json`, `reputation.json` and `alerts.json` are indented;
`clients.json` and `lookups.json` put each entry on one line.
## Sizes
`clients.json` takes about 1 KiB per client, about 20 MiB at the default
`SWWAF_MAX_TRACKED_CLIENTS`. Written every 15 minutes, that is under 2 GiB of
disk writes a day. `bans.json` takes about 1.2 KiB per ban, at most about 2.5
KiB, so about 6 MiB at the default `SWWAF_MAX_BANS`. With a full file, a hundred
new bans a day come to about 600 MiB of writes. `lookups.json` takes about 150
bytes per answer, under 1.5 GiB of writes a day when full. The other two files
are usually a few MiB or less.
## Writing
`bans.json` is written `SWWAF_STATE_WRITE_DELAY` after a ban is made, lifted or
made permanent. Every file is written every `SWWAF_STATE_COUNTER_INTERVAL`, and
when `smallwebwaf` stops. Each write goes to a temporary file that then replaces
the file, so a crash leaves the old file or the new one, whole. Before writing a
file, `smallwebwaf` takes in any edit you made to it, so your edit is not
overwritten. A write that fails is logged, raised as a `file_error` alert, and
tried again at the next write. A hard kill loses what changed since the last
write:
- ban changes from the last `SWWAF_STATE_WRITE_DELAY` (10 seconds);
- everything else from the last `SWWAF_STATE_COUNTER_INTERVAL` (15 minutes):
- counts and history, and the counts in ban notes;
- GeoJS answers;
- reputation verdicts and the AbuseIPDB count;
- anomaly counters and alert cooldowns.
## Reading at start
The files are read back. Each client keeps its counts, so a restart gives no one
a fresh allowance. Each ban keeps refusing until it ends. Counter buckets and
cache entries whose time has passed are discarded. A missing file is empty
state. These stop the start, with a message naming the file and the place:
- a file that does not parse, or has another `version`;
- an entry without a field it needs;
- a ban whose `cause` is not `limit`, `attack`, `admin` or `crowdsec`;
- a ban whose `expires` is not after its `start`.
## Editing while it runs
`smallwebwaf` watches the directory and takes in your edit of a file as soon as
you save it. What the file then holds replaces what `smallwebwaf` held for it.
It tells its own writes from yours by comparing the file with what it last read
or wrote. A change it made after you opened the file, such as a new ban, is lost
when you save over it; most editors warn when a file changed on disk while it
was open. An edit with an error that would stop the start does not stop the
running `smallwebwaf`. It keeps what it holds. At the file's next write it
renames your file to `<name>.bad`, writes the file again from memory, and logs
and alerts the file and the place of the error. Mend the `.bad` file and move it
back. A file you remove is written again at its next write.
internal/alerts/alerts.go
+5 -6
View File
@@ -1,8 +1,8 @@
// Package alerts sends alerts on bans, on traffic over an anomaly
// threshold, on a source that fails and on a file with an error to each
// destination set: to the webhook
// SWWAF_ALERT_WEBHOOK_URL names, each as one JSON object, as the "Alert
// webhook schema" section of SPEC.md describes, to the Slack incoming
// SWWAF_ALERT_WEBHOOK_URL names, each as one JSON object, as "The JSON
// object" in docs/alerts.md describes, to the Slack incoming
// webhook SWWAF_ALERT_SLACK_WEBHOOK_URL names, as a message, and to the
// ntfy topic SWWAF_ALERT_NTFY_URL names. A repeat within
// SWWAF_ALERT_COOLDOWN is held back, and so is an alert past
@@ -149,11 +149,10 @@ type Params struct {
}
// Alert is one alert, as the webhook is sent it and alerts.json holds it,
// with the fields of the "Alert webhook schema" section of SPEC.md. ASN,
// ASName and Country are, for a ban, the client's as the ban's notes give
// them.
// with the fields of "The JSON object" in docs/alerts.md. ASN, ASName and
// Country are, for a ban, the client's as the ban's notes give them.
//
//nolint:tagliatelle // SPEC.md's alert webhook schema names its fields in snake_case
//nolint:tagliatelle // docs/alerts.md names the alert's fields in snake_case
type Alert struct {
Instance string `json:"instance"`
Time time.Time `json:"time"`
internal/metrics/metrics.go
+4 -3
View File
@@ -1,6 +1,7 @@
// Package metrics keeps smallwebwaf's Prometheus metrics, as the "Metrics
// endpoint" section of SPEC.md lists them, and serves them in the
// Prometheus text format. No metric carries a client's address.
// Package metrics keeps smallwebwaf's Prometheus metrics, as
// docs/metrics.md lists them, and serves them in the Prometheus text
// format, as the "Metrics endpoint" section of SPEC.md requires. No
// metric carries a client's address.
package metrics
import (
internal/ratelimit/ratelimit.go
+1 -1
View File
@@ -206,7 +206,7 @@ type Hit struct {
// Counts are a client's requests and bytes in the minute, the hour and
// the day that end at a request, that request's included.
//
//nolint:tagliatelle // SPEC.md's request log names its fields in snake_case
//nolint:tagliatelle // docs/request-log.md names the fields in snake_case
type Counts struct {
Minute float64 `json:"minute"`
Hour float64 `json:"hour"`
internal/remotelog/remotelog.go
+4 -3
View File
@@ -1,8 +1,9 @@
// Package remotelog sends the lines smallwebwaf writes on stdout to the
// remote log endpoint, SWWAF_LOG_REMOTE_URL, as the "Request log" section
// of SPEC.md describes: each line as the message of an RFC 5424 syslog
// record, over UDP, TCP or TLS. Lines wait in a bounded buffer, so a slow
// or unreachable endpoint never holds up a request or stdout.
// of SPEC.md requires and "Sending the log to a syslog server" in
// docs/request-log.md describes: each line as the message of an RFC 5424
// syslog record, over UDP, TCP or TLS. Lines wait in a bounded buffer, so
// a slow or unreachable endpoint never holds up a request or stdout.
package remotelog
import (
internal/requestlog/requestlog.go
+3 -4
View File
@@ -64,11 +64,10 @@ const LimitHitErrorBurst = "error_burst"
const timeLayout = "2006-01-02T15:04:05.000Z07:00"
// Line is one request's line in the request log. The field names, and
// their order, are those of the "Request log" section of SPEC.md. A field
// that may not apply to a request is left out of its line when it does
// not.
// their order, are those of docs/request-log.md. A field that may not
// apply to a request is left out of its line when it does not.
//
//nolint:tagliatelle // SPEC.md's request log names its fields in snake_case
//nolint:tagliatelle // docs/request-log.md names the fields in snake_case
type Line struct {
Type string `json:"type"`