sneak/smallwebwaf
1
0
Fork 0
Code Issues 5 Pull Requests 1 Actions 14 Packages Projects Releases Wiki Activity
Files
e44ba7ee5e197e8dbc8f15d521974dc99ee77304
smallwebwaf/SPEC.md
T
sneak e44ba7ee5e
check / check (push) Waiting to run
Details
SPEC rework after review: each requirement once, gitea notes to docs/ 
Every requirement is stated once, where it belongs, and linked from
elsewhere. The notes on gitea move to docs/gitea.md, which
script/gitea-requests points at. The list of settings that take off
is exact, the non-goal that contradicted the planned item is gone,
and the sentences the review named are split or reordered. The
package comments in internal/waf and internal/anomaly point at the
renamed sections.

Model: fable-5-1
2026-10-22 19:43:03 +00:00

1395 lines
74 KiB
Markdown
Raw Blame History

# smallwebwaf specification
`smallwebwaf` is a web application firewall and reverse proxy for one app. It
runs inside the app's container, between traefik and the app. This document says
what it must do and why, each requirement once. Everything in it is built,
except the last section, "Planned, not built". `README.md` says how to run it.
`EVALUATION.md` says why no existing tool was adopted. `docs/gitea.md` records
how it behaves in front of gitea.
## Purpose
- It protects one self-hosted app on the open internet from scrapers, scanners
and other abusers, from the app's first request, with no setting at all. Every
setting has a default chosen for a service facing the internet in 2026. A
setting that needs an alert destination, an account key or a token stays off
until the operator supplies it.
- It limits each client's requests and bytes per minute, hour and day. It bounds
the size and duration of every request and answer. It detects attacks.
- It bans abusers: briefly at first, for seven days on a clear sign of attack,
and permanently when they keep at it.
- It looks up each client's AS number and country. It can refuse whole countries
and netblocks, and lower its limits for listed AS numbers and countries. It
consults reputation sources. It sends alerts.
- Everything is set by environment variables whose names start with `SWWAF_`,
since `smallwebwaf` shares its container, and so its environment, with the
app. The only other inputs are files: the rule files, the state files, a
lookup database, and a setting given as a file.
## Non-goals
- TLS, certificates and routing by host name. Those are traefik's job.
- More than one app per `smallwebwaf`. Each app's image carries its own.
- Browser challenges such as captchas or proof of work. If wanted, chain Anubis
between `smallwebwaf` and the app.
- Defence against floods that saturate the host's network link. That needs help
upstream of the host.
- A web console or a configuration file.
## Architecture
- One statically linked Go binary. It ships in an image built on Ubuntu 26.04
LTS with nixpkgs installed. That image is the base of the app's own image, so
`smallwebwaf` and the app run in one container (see "Deployment"). runit
starts `smallwebwaf` as a non-root user of its own. `smallwebwaf` writes only
to its state directory.
- One listener, on port 8080, which traefik routes to. It forwards requests to
the app on `127.0.0.1:8081`. It answers its own endpoints under
`/_smallwebwaf/` (see "Admin endpoints"). Those never reach the app.
- No database of any kind. Nothing is read from disk while serving a request.
- The proxy is the standard library's `httputil.ReverseProxy`. The other parts
of the process are each specified below.
- Libraries, checked against the owner's Go dependency defaults:
- the standard library for the proxy, HTTP clients, DNS, logging
(`log/slog`) and the state files (`encoding/json`, `os.Rename`);
- `github.com/corazawaf/coraza/v3` and
`github.com/corazawaf/coraza-coreruleset/v4` at `v4.25.0`, which carries
the Core Rule Set 4.25.0. The changes `smallwebwaf` makes to the Core Rule
Set, and the default `SWWAF_WAF_DISABLED_RULES`, are built for that
version. They are checked again before the version changes;
- `github.com/oschwald/maxminddb-golang/v2` to read the lookup database;
- `github.com/fsnotify/fsnotify` to notice edited files;
- `github.com/prometheus/client_golang` for the metrics;
- remote log sending written in this project, since the standard library's
`log/syslog` is frozen and writes only the older syslog format.
## What happens to a request
The steps run in this order. The first step that produces a final answer ends
the handling. The size and time limits apply to the whole exchange. Each step
names the section that holds its rules.
1. `GET /_smallwebwaf/healthz` is answered at once, before any check.
2. The client is identified.
- If the TCP peer is inside `SWWAF_TRUSTED_PROXIES`, `X-Forwarded-For` is
walked from the right. The first address outside those netblocks is the
client. If every address is inside them, the leftmost is the client, so
that a visitor on a private network is known by its own address. With no
header, the peer is the client.
- If the peer is outside `SWWAF_TRUSTED_PROXIES`, it is the client, and the
header is ignored.
- An IPv6 client is its group: the netblock of `SWWAF_IPV6_GROUP_PREFIX`
around its address, a /64 by default, since one abuser usually controls a
whole /64. A client is therefore one IPv4 address or one IPv6 group.
3. The static lists. A client in `SWWAF_ALLOW_NETS` skips every check below. It
is still counted for the anomaly thresholds. A client in `SWWAF_DENY_NETS` is
refused.
4. The ban ledger. A client under a ban is refused (see "Bans").
5. The lookup, unless `SWWAF_LOOKUP_SOURCE` is `off` (see "Lookup of AS number
and country"). The request goes on at once. It waits for the first GeoJS
answer, up to `SWWAF_LOOKUP_TIMEOUT`, only when one of these needs it:
- a country list;
- a biased threshold;
- `SWWAF_ADD_LOOKUP_HEADERS`.
6. The country lists (see "Country lists"). A refused request is not an offence.
7. The reputation sources (see "Reputation"). A client on a fetched blocklist
gets `SWWAF_BLOCKLIST_ACTION`. A client on the CrowdSec decision list is
banned. A client listed by a DNS blocklist, or scored at least
`SWWAF_ABUSEIPDB_MIN_SCORE` by AbuseIPDB, gets `SWWAF_REPUTATION_ACTION`.
With no cached verdict, a query is queued in the background and the request
goes on. A first request is never delayed by a reputation query.
8. The client's limit percentage (see "Biased thresholds").
9. Rate limiting (see "Request rate limits, per client"). A request over a limit
times the percentage is refused and bans the client.
10. The trap paths (see `SWWAF_TRAP_PATHS`). A request for one is refused and
bans the client.
11. The rule files (see "Rule files"). Matching stops at the first rule that
refuses.
12. The Core Rule Set (see "Attack detection"). In `block` mode a match is
refused with `403`. In `detect` mode it is only logged and alerted.
13. A request under `/_smallwebwaf/` is answered here and goes no further.
14. The request is forwarded to `SWWAF_UPSTREAM_URL`, streaming, with
`X-Forwarded-For` added, and `X-Client-ASN` and `X-Client-Country` when
enabled.
15. After the response:
- The response bytes, the request bytes or both are added to the byte
counters, as `SWWAF_BYTES_COUNT` says. A client whose total passes a byte
limit times the percentage has broken that limit and is banned. The
response in progress is not cut off.
- The client's refusals are counted for the error burst (see
`SWWAF_ERROR_BURST_THRESHOLD`).
- The client's history and the anomaly counters are updated, and the alert
thresholds are evaluated.
In `SWWAF_MODE=observe`, every refusal above is logged as what would have
happened, nothing is refused, and no ban is made. The size and time limits and
the token check still apply.
## Counting method
- Each window (minute, hour, day) uses two adjacent fixed buckets per client.
The earlier bucket is weighted by how much of it still overlaps the sliding
window. This costs a few integers per client per window. It avoids the burst
at bucket boundaries that a single fixed bucket allows.
- Counters are read and updated in memory. They are written to `clients.json`
and loaded again at start (see "Persistent state"), so a restart does not hand
every client a fresh allowance. Buckets whose time has passed are discarded on
load.
- Memory is bounded. When the table of clients holds
`SWWAF_MAX_TRACKED_CLIENTS`, the least recently seen client is dropped first,
with its history. The bans are bounded by `SWWAF_MAX_BANS` (see "Bans"). The
GeoJS answers, the reputation verdicts and the anomaly counters have fixed
bounds, given where each is described.
## Bans
A ban refuses every request from a netblock with `SWWAF_BAN_RESPONSE` until it
ends. The netblock is the client: its IPv4 address, widened to
`SWWAF_BAN_SCOPE_V4_PREFIX`, or its IPv6 group. At the defaults a ban covers one
address or one /64, so a permanent ban does not reach neighbours who did
nothing. Requests refused under a ban are not counted against limits. They are
counted in the client's history and in the ban's notes.
An offence is a request `smallwebwaf` holds against the client. It is one of:
- a clear sign of attack;
- a broken limit;
- a refusal by a rule file, by the Core Rule Set or for a missing or wrong
token.
Offences are counted by kind in the client's history. The first two kinds lead
to a ban, each by its own rule:
- A clear sign of attack: a request that matches a rule whose action is `ban`,
or asks for a path in `SWWAF_TRAP_PATHS`. Examples are a probe for a `.env`
file or a `.git` directory, and a known scanner's user agent.
- The first one bans the netblock for `SWWAF_ATTACK_BAN_DURATION` (default
`7d`).
- Any further request from the netblock while that ban lasts shows it is
malicious: the ban becomes permanent.
- Once that ban has run out, the netblock is served like any other. Its next
clear sign of attack bans it permanently at once.
- A broken limit: a request over a rate limit, a response that takes the
client's byte total past a byte limit, or more than
`SWWAF_ERROR_BURST_THRESHOLD` refusals within a minute.
- The first such ban lasts `SWWAF_LIMIT_BAN_DURATION` (default `1h`). So
does one that comes more than `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default
`24h`) after the last such ban ended.
- Breaking a limit again within that window makes the new ban three times as
long as the last one: one hour, then 3, 9, 27 and 81 hours.
- A ban that would be longer than `SWWAF_MAX_BAN_DURATION` (default `7d`) is
permanent instead. At the defaults, that is the sixth ban in a row.
- Each such ban sets the client's counters for every limit back to zero.
Once it ends, only new requests can break a limit again. The client's
history keeps its totals.
A Core Rule Set match refuses only the request it matched, without a ban,
because the Core Rule Set has false positives. A `block` rule does the same.
Those refusals count toward the error burst, so a client that keeps setting them
off is banned under the second rule.
A client on the CrowdSec decision list is banned until the decision ends, with
the cause `crowdsec`. Only a client that sends a request is banned, so a long
list does not fill `bans.json`. Such a ban is never made permanent. It does not
make the netblock's next ban for a broken limit longer.
Some refusals make no ban at all. `SWWAF_DENY_NETS`, a blocklist or reputation
hit whose action is `deny`, and the country lists refuse each request they cover
and record nothing in `bans.json`.
Every ban carries notes with what an admin needs to decide whether to lift it
(see "Persistent state"). An admin can add or lift a ban at any time by editing
`bans.json`, or through the ban endpoints when `SWWAF_ADMIN_TOKEN` is set. A ban
lifted before it ends is kept, marked as lifted, and does not count toward a
longer ban. Deleting its entry from `bans.json` forgets it entirely.
A permanent ban does not run out. It ends when an admin lifts it, or when the
table of bans is full. When `smallwebwaf` holds `SWWAF_MAX_BANS` bans of its
own, it drops the one that has gone longest without a request, whether past,
active or permanent. A scanner whose ban was dropped that way is refused and
banned again by its next probe. A ban an admin made, whose cause is `admin`, is
never dropped and does not count toward `SWWAF_MAX_BANS`, so such bans can never
fill the table. An admin who wants to keep a ban `smallwebwaf` made sets its
cause to `admin`.
## Settings
Conventions:
- Lists are comma separated. Netblocks are CIDR; a bare address means /32 or
/128. Durations use Go's syntax plus `d` for days (`90s`, `15m`, `24h`, `7d`).
Sizes take `K`, `M` and `G`, powers of 1024. Countries are two-letter ISO
codes in either case. A code that is not a country, such as `nk`, stops the
start with a message naming it.
- These settings take `off`, which switches them off:
- the five timeouts under "Size and time limits";
- `SWWAF_REQUEST_MAX_BYTES` and `SWWAF_RESPONSE_MAX_BYTES`;
- the three rate limits and the three byte limits;
- `SWWAF_WAF_ANOMALY_THRESHOLD`, `SWWAF_WAF_BODY_LIMIT` and
`SWWAF_ERROR_BURST_THRESHOLD`;
- `SWWAF_ALERT_COOLDOWN` and `SWWAF_ALERT_MAX_PER_HOUR`;
- every anomaly threshold.
Every other setting refuses `off`.
- A list set to an empty value is an empty list, and replaces the default.
- Any setting can be read from a file instead, for secrets and long lists:
`SWWAF_ADMIN_TOKEN_FILE` names the file that holds `SWWAF_ADMIN_TOKEN`.
`SWWAF_LOG_REMOTE_TLS_CA_FILE` already holds a file name, so it has no `_FILE`
form.
- Settings, files included, are read once at start. Changing one means
restarting the container. The files `smallwebwaf` watches while it runs are
its state files, its rule files and the lookup database.
- An invalid setting stops the process at start with a message naming the
variable. At start the settings in effect are logged, with secrets masked.
The settings, by group:
### Core
- `SWWAF_UPSTREAM_URL` (default `http://127.0.0.1:8081`): the app.
- `SWWAF_LISTEN_ADDR` (default `:8080`): the one listener, for the requests
`smallwebwaf` forwards and for its own endpoints. The address part stays empty
(see "Ports").
- `SWWAF_INSTANCE_NAME` (default: the container's host name, which docker sets
to the first 12 characters of the container's id): included in every log line,
metric and alert. Set it to a name that survives a redeploy and tells
instances apart, such as `fsn1app1/gitea`.
- `SWWAF_MODE` (default `enforce`): `enforce`, or `observe` to log and alert on
every decision while refusing nothing.
- `SWWAF_TRUSTED_PROXIES` (default `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`,
the private address ranges): the netblocks whose `X-Forwarded-For` is
believed, normally where traefik reaches `smallwebwaf` from. A client that
connects directly cannot claim another address. A list given replaces the
default; set but empty, it trusts nothing.
- `SWWAF_IPV6_GROUP_PREFIX` (default `64`), from 32 to 128.
- `SWWAF_MAX_TRACKED_CLIENTS` (default `20000`): the clients held in memory and
in `clients.json`.
- `SWWAF_MAX_BANS` (default `5000`): the bans `smallwebwaf` made that are held,
past, active and permanent (see "Bans").
### Tokens
- `SWWAF_ADMIN_TOKEN`: the bearer token for the ban endpoints and
`/_smallwebwaf/clients/<ip>`. Unset by default, which switches them off; bans
are then managed by editing `bans.json`.
- `SWWAF_METRICS_TOKEN`: the bearer token a scraper sends for
`/_smallwebwaf/metrics`. Unset by default, which switches the metrics off,
since they would otherwise be open to anyone on the internet. It is separate
from `SWWAF_ADMIN_TOKEN`, so a scraper that holds it cannot manage bans.
- A token is a long random value, since the endpoints can be reached from the
internet. One shorter than 32 characters stops the start. The app starts with
the same environment variables as `smallwebwaf`, so it can read a token given
as one. The file form keeps a token out of the app's reach (see "Tokens" under
"Deployment").
### State files
- `SWWAF_STATE_DIR` (default `/var/lib/smallwebwaf`): the JSON state files, in a
directory of their own beside the app's data, belonging to the `smallwebwaf`
user.
- `SWWAF_STATE_WRITE_DELAY` (default `10s`): `bans.json` is written this long
after a ban is made, lifted or made permanent, so a burst of changes becomes
one write.
- `SWWAF_STATE_COUNTER_INTERVAL` (default `15m`): how often every state file is
written.
### Logging
- The request log on stdout is always on and has no switch.
- `SWWAF_LOG_LEVEL` (default `info`): for the process's own messages, which are
JSON lines on stdout too, marked `"type":"process"`. It does not filter the
request log.
- `SWWAF_LOG_REQUEST_HEADERS` (default
`accept,accept-language,accept-encoding,content-type,origin,range`): the
request headers to record. `Authorization`, `Cookie` and `Set-Cookie` values
are never logged, only whether they were present.
- `SWWAF_LOG_REMOTE_URL`: when set, every log line is also sent there, as
`syslog+udp://host:514`, `syslog+tcp://host:514` or `syslog+tls://host:6514`.
- `SWWAF_LOG_REMOTE_TLS_CA_FILE`: an optional CA certificate file for the `+tls`
form.
- `SWWAF_LOG_REMOTE_BUFFER` (default `10000`): the lines held in memory while
the server is unreachable. When full, the oldest are dropped and counted.
- `SWWAF_LOG_REMOTE_FACILITY` (default `local0`) and `SWWAF_LOG_REMOTE_APP_NAME`
(default `SWWAF_INSTANCE_NAME`): the syslog header fields.
### Metrics
- `SWWAF_METRICS_TOP_N` (default `50`): how many AS numbers and countries get
series of their own. The rest are summed as `other`.
### Static lists
- `SWWAF_ALLOW_NETS`: bypass everything (monitoring, the owner's own networks).
- `SWWAF_RATE_LIMIT_EXEMPT_NETS`: bypass the request and byte limits only. The
error burst, attack detection and bans still apply.
- `SWWAF_DENY_NETS`: always refused.
### Request rate limits, per client
Breaking one bans the client, so the defaults sit several times above what one
busy person produces. A browser loading a heavy page makes a few hundred
requests, a git clone a handful, and several people often share one address. A
machine that talks to the app all day, such as a gitea Actions runner, can still
pass the day limit. Its address belongs in `SWWAF_RATE_LIMIT_EXEMPT_NETS`.
- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `1000`), `SWWAF_RATE_LIMIT_PER_HOUR`
(default `10000`) and `SWWAF_RATE_LIMIT_PER_DAY` (default `50000`).
- `SWWAF_RATE_LIMIT_EXEMPT_PATHS`: path prefixes not counted, such as static
assets or health checks.
### Byte limits, per client
A response's bytes are counted when it ends, so every default sits above the
largest response allowed, and no single download breaks one.
- `SWWAF_BYTES_LIMIT_PER_MINUTE` (default `10G`), `SWWAF_BYTES_LIMIT_PER_HOUR`
(default `20G`) and `SWWAF_BYTES_LIMIT_PER_DAY` (default `50G`).
- `SWWAF_BYTES_COUNT` (default `both`): `response`, `request` or `both`.
### Size and time limits
These apply per request, in both directions. The client-facing timeouts apply
between the client and `smallwebwaf`, the app-facing ones between `smallwebwaf`
and the app, since a slow client and a slow app are separate problems. There is
one size limit for request bodies and one for response bodies, since
`smallwebwaf` passes bodies through unchanged. Bodies stream straight through,
so a request body reaches the app while the client is still sending it.
- `SWWAF_CLIENT_REQUEST_TIMEOUT` (default `60s`): how long a client may take to
send its request line and headers, and then, from the end of the headers, its
body.
- `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES` (default `32K`): the largest request
line and headers. It must be more than `4K`. Go's HTTP server always has such
a limit, and reads 4 KiB past the one it is given before it refuses.
- `SWWAF_CLIENT_IDLE_TIMEOUT` (default `120s`): how long a kept-open connection
may wait for its next request before `smallwebwaf` closes it. It is longer
than the 90 seconds after which traefik, by default, closes an unused
connection. So traefik closes first, and never sends a request on a connection
`smallwebwaf` is closing.
- `SWWAF_CLIENT_RESPONSE_TIMEOUT` (default `30m`): how long one response may
take to reach the client, from the end of the request to the last byte.
- `SWWAF_UPSTREAM_REQUEST_TIMEOUT` (default `60s`): how long connecting to the
app and sending it the whole request may take.
- `SWWAF_UPSTREAM_RESPONSE_TIMEOUT` (default `30m`): how long the app may take
to send one whole response, from the end of the request to the last byte.
- `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body.
- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body.
What the client gets when a limit is passed:
- Go's HTTP server reads a request's line and headers before `smallwebwaf` sees
the request. It answers `431` to headers over
`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`. A client too slow to send them gets
either no answer or an error answer, mostly `400`. Either way the server
closes the connection, nothing reaches the app, and the request log gets no
line.
- A request that announces a body larger than its limit is refused before
anything reaches the app.
- When a limit is passed after the headers have been read, but before the
response has started, `smallwebwaf` answers itself:
- `413` for a request body that is too large;
- `408` for a client too slow to send its body;
- `502` for a response that is too large;
- `504` for an app that is too slow.
- A request body streams through, so each side can hold up the other. A request
timeout that runs out while the body is still on its way answers `408` if
`smallwebwaf` was waiting for the client at that moment. It answers `504` if
it was waiting for the app to take what it had.
- Once the response has started it can only be cut off, and the connection is
closed.
- A WebSocket connection leaves these limits behind once it is upgraded. It
stays open until either side closes it.
### Lookup of AS number and country
On by default through GeoJS, which needs no account or file, so that country
lookup works with no setup. The file is the alternative for a service that keeps
its visitors' addresses on its own host.
- `SWWAF_LOOKUP_SOURCE` (default `geojs`): `geojs`, `file` or `off`. One source
at a time: `file` without `SWWAF_LOOKUP_DB_PATH`, or `SWWAF_LOOKUP_DB_PATH`
with any other source, stops the start with a message naming both. So does a
setting that needs lookups while the source is `off`: the country lists, the
biased thresholds, `SWWAF_ADD_LOOKUP_HEADERS`, and the anomaly thresholds per
AS number.
- `SWWAF_LOOKUP_DB_PATH`: the file for `file`: the IPinfo Lite database in its
`.mmdb` form (`ipinfo_lite.mmdb`), whose `asn`, `as_name` and `country_code`
fields are read. The operator downloads it with a free IPinfo account, mounts
it read-only, and refreshes it; IPinfo updates it daily. `smallwebwaf` never
fetches it and holds no account or token for it. A file that is missing or
unreadable at start stops the start. `smallwebwaf` notices when the file is
replaced and reads it again. A replacement it cannot read is ignored, logged
and sent as one `file_error` alert, and the file it has stays in use. The
directory that holds the file is mounted, not the file itself (see "Rule
files" for why).
- `geojs` asks the free GeoJS web service about up to 200 addresses in one
request (`https://get.geojs.io/v1/ip/geo.json?ip=a,b,c`), one request at a
time, and reads each answer's `country_code`, `asn` and `organization_name`.
An answer without a `country_code`, or with the `asn` `64512`, counts as
unknown. GeoJS is told the address of every new visitor, whether or not a
setting uses the answer, since the request log, the metrics and the ban notes
carry it. Private, loopback and link-local addresses, which no source can
place, are never sent.
- A client the lookup cannot place has an unknown country and AS number. The
exclusive country list refuses it, and the biased thresholds give it
`SWWAF_UNKNOWN_LIMIT_PERCENT`. Nothing else treats it differently.
- Each GeoJS answer is kept for 7 days, in memory and in `lookups.json`, apart
from the table of clients. Up to 100,000 answers are kept, about 15 MiB; past
that, the answer used longest ago goes first.
- `SWWAF_LOOKUP_TIMEOUT` (default `1s`): how long a request waits for its
client's first answer when a setting needs it, and how long a request to GeoJS
may take. A client whose answer has not come in time counts as unknown until
it comes, and its later requests do not wait.
- GeoJS publishes no rate limit, but its terms forbid "an excessive amount of
API requests", judged by GeoJS alone, and let it block a caller. While GeoJS
is slow, down or refusing `smallwebwaf`, new clients count as unknown,
`smallwebwaf` keeps asking with backoff, and one `source_failure` alert is
sent per cooldown. A service that cannot accept this uses the file.
- `SWWAF_ADD_LOOKUP_HEADERS` (default `false`): pass `X-Client-ASN` and
`X-Client-Country` to the app.
### Country lists
Both are empty by default: the defaults judge a client by what it does, not by
where it comes from. Clients in `SWWAF_ALLOW_NETS` are not checked.
- `SWWAF_DENIED_COUNTRIES`: for example `cn,ru,kp,ir,ua,by`. Every request from
a listed country is refused.
- `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES`: for example `us,de`. Only clients
placed in a listed country get through. Every request from any other country
is refused. So is every request from a client the lookup cannot place, such as
a visitor on the local network, another container or internal monitoring.
Those that should reach the app go in `SWWAF_ALLOW_NETS`. A list that let
unplaced clients through would let every new client in whenever GeoJS stops
answering. That is the nearest a failing helper comes to taking the service
down: GeoJS failing while this list is set refuses every new client.
- Both may be set. `SWWAF_DENIED_COUNTRIES` then adds nothing, since the
exclusive list already refuses every other country. A code on both lists stops
the start with a message naming it.
- A refused request is answered with `SWWAF_BAN_RESPONSE`, as soon as the
client's address has been looked up, before its body is read. It skips the
reputation checks, the limits, the rule files and the Core Rule Set. It is
logged with the action `country_denied`, and counted in the metrics. It is a
refusal, not a ban.
### Biased thresholds
The lists are empty by default, for the same reason as the country lists. The
request log and the metrics show which AS numbers and countries a service's
abuse comes from.
- `SWWAF_ASN_LIMIT_PERCENT`: for example `AS14061:50,AS16276:50,AS45102:25`.
Clients in a listed AS number get that percentage of every request and byte
limit, so the rules in "Bans" ban them after fewer requests than others. `0`
is a zero allowance: the client's first request breaks a limit and bans it.
- `SWWAF_COUNTRY_LIMIT_PERCENT`: the same form with country codes, for example
`CN:25,RU:50`.
- `SWWAF_ASN_BYTES_PERCENT` and `SWWAF_COUNTRY_BYTES_PERCENT`: optional
overrides applied to the byte limits only, when the byte percentage should
differ from the request percentage.
- `SWWAF_UNKNOWN_LIMIT_PERCENT` (default `100`): for clients the lookup cannot
place.
- `SWWAF_ASN_LIMIT_PERCENT_URL`: optional URL of a text file of `AS:percent`
lines, so one abuse-source list can be shared by every `smallwebwaf` in the
fleet. It is fetched and refreshed like the blocklists.
- A client's limit percentage is the lowest of those that apply to it: its AS
number's, its country's, or a reputation hit's `limit:<percent>`. It is 100 if
none applies. Clients given a lower percentage are banned after fewer
requests, first for a while and then, for repeated abuse, permanently.
- No budget is shared by a whole country or AS number. One abuser could use it
up and lock out everyone else there: a denial of service nobody chose, and the
one this tool exists to prevent. Refusing a whole country is left to the
operator's choice, through the country lists.
### Attack detection
By default the Core Rule Set reads the method, the URL with its query string,
and the headers of every request. That is where automated attacks show:
injection in query strings, path traversal, attacks carried in headers,
scanners' user agents. It reads no request body. On a code forge, the example
deployment, bodies carry what people write and publish: issue and comment text,
wiki pages, files saved in the web editor, package descriptions. When these hold
shell commands or code, the Core Rule Set takes them for attacks. The default
gives up refusing an attack carried in a body. The rule files, the limits and
the bans still apply to the client that sends it, and `SWWAF_WAF_BODY_LIMIT`
switches body inspection on.
- `SWWAF_RULES_DIR` (default `/etc/smallwebwaf/rules.d`): the directory of rule
files (see "Rule files").
- `SWWAF_RULES_ENABLED` (default `true`): `false` skips rule files entirely.
- `SWWAF_WAF_MODE` (default `block`): `off`, `detect` (log and alert only), or
`block`.
- `SWWAF_WAF_PARANOIA_LEVEL` (default `1`) and `SWWAF_WAF_ANOMALY_THRESHOLD`
(default `5`): the Core Rule Set's own two tuning values, at its own defaults.
- `SWWAF_WAF_DISABLED_RULES` (default
`920340,920420,920440,920640,930130,930140`): rule ids to switch off when an
app trips a false positive. The default switches off six rules:
- 920340, 920420 and 920640 refuse a request for the type of its body, when
the type is missing or not on the Core Rule Set's short list;
- 920440 refuses a request for its file extension, such as `.sh` or `.sql`;
- 930130 and 930140 refuse a file or directory name in the path, such as
`.git/`, `.gitignore`, `Dockerfile`, `package.json` or an editor's
settings directory.
In front of a code forge these refuse git over HTTP, container image and
package uploads, and views of ordinary files in a repository. At the site
root, where no app serves such files, the default rule file bans the common
probes these rules caught (see "Rule files"). A list given replaces the
default, so include them in it.
- `SWWAF_WAF_EXEMPT_PATHS`: path prefixes not inspected.
- `SWWAF_WAF_BODY_LIMIT` (default `off`): `off` reads no request body. A size,
such as `128K`, switches body inspection on. The Core Rule Set then reads form
data and multipart bodies up to that size, streaming the rest of a longer one
on without holding it in memory. It reads JSON and XML bodies no larger than
the size, since those cannot be read in part. Any other body is not read: the
Core Rule Set would read it as form data, where binary content such as a git
push trips rules written for text. Responses are never inspected. Body
inspection suits apps whose forms carry no code. In front of gitea it refuses
the requests below, until their rule ids, named in the request log, are added
to `SWWAF_WAF_DISABLED_RULES`:
- issue and comment text, wiki pages and files saved in the web editor that
hold shell commands or code (932125, 932235, 932250 and others);
- package descriptions that show code;
- PyPI uploads (922130);
- attachments named like `debug.log` or `config.yml` (932180).
- `SWWAF_TRAP_PATHS`: paths the app never serves and only scanners ask for, for
example `/wp-login.php,/xmlrpc.php` in front of gitea. A request for one is a
clear sign of attack. This is the short form of a `path` rule with the `ban`
action, for deployments that mount no rule files.
- `SWWAF_ERROR_BURST_THRESHOLD` (default `30`): the refusals per client per
minute after a rule file match, a trap path, a Core Rule Set match, or a
missing or wrong token. More than this breaks a limit (see "Bans"). A client
trying one attack or token after another is refused many times a minute; a
person rarely more than a few times. Answers the app gives are not counted,
since they do not tell a scanner from an ordinary client. In front of gitea,
registry and git clients are answered `404` and `401` by design, hundreds of
times a minute.
`smallwebwaf` changes the Core Rule Set 4.25.0 in six ways, since in front of
gitea it would otherwise refuse ordinary requests. The changes hold in front of
every app, and no setting undoes them:
- `PUT`, `PATCH` and `DELETE` are allowed methods besides `GET`, `HEAD`, `POST`
and `OPTIONS`. APIs, container image pushes and package uploads use them.
Other methods stay refused.
- The request header `Expect` is allowed, since curl and git send it before some
large uploads. `Content-Encoding` is allowed on a request whose body the Core
Rule Set does not read, which by default is every request. git compresses a
fetch request over 1 KiB and says so in that header. On a body the Core Rule
Set does read, the header stays refused, since a compressed body cannot be
inspected. The other headers the Core Rule Set refuses, such as `Proxy`, stay
refused.
- The `redirect_uri` parameter is not checked for a URL naming an IP address or
`localhost` (931100, 934110). Git Credential Manager, git-credential-oauth and
tea, which gitea registers for OAuth sign-in, ask to be sent back to
`http://127.0.0.1` on the user's own machine. The server never fetches that
address.
- Ten query parameters are not checked against the Core Rule Set's lists of
system files (930120), shell paths (932160) and command names (932260):
- `path`, `files`, `skip-to` and `sub_path`;
- `ref`, `sha`, `branch` and `workflow`;
- `artifactName` and `redirect_to`.
gitea puts file paths, branch and workflow names, and the page to return to
after signing in into them. Any name on those lists can be an ordinary file
or branch, such as `.gitignore` or `docker-build`. In these parameters,
anything only these three rules catch reaches the app, commands such as
`|cat /etc/passwd` included. Path traversal, SQL and script injection, and
PHP, Java and Node.js code are still refused there. Every other parameter
keeps all three rules. Parameter names are matched without regard to case,
as Coraza matches them.
- The Core Rule Set reads the request without the `gitea_flash` and
`redirect_to` cookies. It does not check `Referer` for a Unix command given
without arguments (932340) or for Java starting a process (944110). gitea puts
a message naming what was just done into `gitea_flash`. It puts the page to
return to after signing in into `redirect_to`. The Core Rule Set takes file
names in them, such as `package.json`, for attacks. A browser sends such a
cookie with every request until gitea replaces it. gitea cannot replace it
while those requests are refused, so one refusal would keep the browser out of
the whole site. `Referer` names the page a request comes from, on everything
that page loads. The Core Rule Set takes some page addresses for attacks, such
as a search for `env`, or OpenJDK's `Runtime.java`, whose path holds both
`runtime` and `java.`. gitea shows the message with any script removed and
returns only to its own pages. Every other cookie is read in full, and
`Referer` keeps the other rules.
- Responses are not inspected. A raw file from a repository, such as a shell
script, looks to the response rules like source code leaking from the server.
The fourth and fifth changes leave an app on its own in those places (see "Risks
the design has to handle").
### Ban lengths and response
- `SWWAF_ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of
attack.
- `SWWAF_LIMIT_BAN_DURATION` (default `1h`): the ban for a first broken limit.
- `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): breaking a limit again within
this time after a ban for a broken limit ended makes the next ban three times
as long.
- `SWWAF_MAX_BAN_DURATION` (default `7d`): a ban that would be longer is
permanent instead.
- `SWWAF_BAN_RESPONSE` (default `403`): `403`, `429`, or `close` to drop the
connection without an answer. `403` makes a ban easy to recognise when
debugging. Behind traefik, `close` does not leave the client unanswered:
traefik answers `502`, as it does whenever its backend drops a connection.
- `SWWAF_BAN_SCOPE_V4_PREFIX` (default `32`): widen to for example `24` to ban
the surrounding netblock.
### Reputation
No source is on by default, for the reason the country lists and the biased
thresholds are empty. A source acts on a list kept elsewhere; the defaults judge
a client only by what it does here. AbuseIPDB and CrowdSec also need an account
or an engine of the operator's own. Besides, the list best suited to be on by
default, Spamhaus DROP, must not be downloaded automatically more than once an
hour. Spamhaus may block an address that downloads it too often. Each
`smallwebwaf` fetches its own copy. On a host that runs several apps with it,
which share one address, downloads would often come less than an hour apart.
- `SWWAF_BLOCKLIST_URLS`: text files of addresses and netblocks, one per line;
anything after a `;` or `#` on a line is ignored. For example the Spamhaus
DROP list, `https://www.spamhaus.org/drop/drop.txt`. Spamhaus's terms for it,
on its DROP page:
- automated downloads must be at least one hour apart, and once a day is
more than enough in most cases;
- a product that uses it must credit The Spamhaus Project and keep the
list's date and copyright lines with the data.
- `SWWAF_BLOCKLIST_REFRESH` (default `24h`, never less than `1h`): how often
each list is fetched again. The last good copy is kept whole, comment lines
included, on failure and across restarts.
- `SWWAF_BLOCKLIST_ACTION` (default `deny`): `deny`, `limit:<percent>` or `log`.
`deny` refuses every request from a listed address, which suits lists of
networks that send nothing legitimate, such as DROP. `limit:<percent>` gives
listed clients that percentage of every limit, so they are banned after fewer
requests. It suits a list of addresses shared with ordinary visitors, such as
Tor exits.
- `SWWAF_DNSBL_ZONES`: for example `dnsbl.dronebl.org`. Zones meant for mail
(lists of residential ranges) will block ordinary visitors and should not be
used. Spamhaus zones need their keyed query service, given as a zone name
containing the key.
- `SWWAF_DNSBL_RESOLVER`: an optional resolver address, since public resolvers
are refused by several list operators.
- `SWWAF_ABUSEIPDB_KEY`, `SWWAF_ABUSEIPDB_MIN_SCORE` (default `75`) and
`SWWAF_ABUSEIPDB_DAILY_BUDGET` (default `900`; the free tier allows 1000
checks a day). Only clients that have already committed one offence are
queried, so the budget is spent on suspects.
- `SWWAF_CROWDSEC_LAPI_URL` and `SWWAF_CROWDSEC_LAPI_KEY`: optional. If a
CrowdSec engine exists on the host, its decision list is pulled on a schedule
and treated as a blocklist (see "Bans" for the ban it makes). This is how a
fleet-wide blocklist can arrive without `smallwebwaf` depending on CrowdSec.
- `SWWAF_REPUTATION_ACTION` (default `limit:25`): `deny`, `limit:<percent>` or
`log`, for DNS blocklist and AbuseIPDB hits. Such verdicts are less certain
than a blocklist, so by default a listed client gets a quarter of every limit
and is banned after a quarter of the requests.
- `SWWAF_REPUTATION_CACHE_TTL` (default `24h`) and `SWWAF_REPUTATION_TIMEOUT`
(default `2s`).
- A source that is down or over quota gives no verdict. The service continues,
with one `source_failure` alert per cooldown.
### Alerting
- `SWWAF_ALERT_WEBHOOK_URL`: a JSON POST; schema under "Alert webhook schema".
`SWWAF_ALERT_WEBHOOK_HEADERS`: optional `Name:value` pairs for authentication.
- `SWWAF_ALERT_SLACK_WEBHOOK_URL`: a Slack incoming webhook, sent a formatted
message.
- `SWWAF_ALERT_NTFY_URL` (the full topic URL) and `SWWAF_ALERT_NTFY_TOKEN`:
title, priority and tags set from the event type.
- `SWWAF_ALERT_EVENTS` (default
`ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure,file_error`):
which event types are sent. `source_failure` is a reputation source or GeoJS
failing or refusing `smallwebwaf`. `file_error` is one of:
- a rule file or state file edited while running that does not parse;
- a replacement lookup database that cannot be read;
- a state file that cannot be written.
- `SWWAF_ALERT_COOLDOWN` (default `15m`): the same event for the same netblock,
source, file or anomaly scope is not repeated within this time. A count of
suppressed repeats is included in the next one.
- `SWWAF_ALERT_MAX_PER_HOUR` (default `60`): beyond this, alerts are rolled into
one summary per hour, so a wide attack cannot flood the channel.
- A destination that is down is retried with backoff from a bounded queue, the
oldest alert dropped first, and the drops counted in the metrics.
### Anomaly thresholds
These only alert; nothing is blocked. None is set by default, and a threshold
left unset sends no alert. An alert needs a destination only the operator can
supply, and what counts as unusual depends on each service's normal traffic,
which the metrics show.
- Per client: `SWWAF_ANOMALY_CLIENT_REQUESTS_PER_MINUTE`,
`SWWAF_ANOMALY_CLIENT_REQUESTS_PER_HOUR`,
`SWWAF_ANOMALY_CLIENT_BYTES_PER_MINUTE` and
`SWWAF_ANOMALY_CLIENT_BYTES_PER_HOUR`.
- Per surrounding netblock (`SWWAF_ANOMALY_NET_V4_PREFIX` default `24`,
`SWWAF_ANOMALY_NET_V6_PREFIX` default `48`):
`SWWAF_ANOMALY_NET_REQUESTS_PER_MINUTE`, `..._PER_HOUR`,
`SWWAF_ANOMALY_NET_BYTES_PER_MINUTE` and `..._PER_HOUR`.
- Per AS number, which needs a lookup source:
`SWWAF_ANOMALY_ASN_REQUESTS_PER_MINUTE`, `..._PER_HOUR`,
`SWWAF_ANOMALY_ASN_BYTES_PER_MINUTE` and `..._PER_HOUR`.
- The whole service: `SWWAF_ANOMALY_TOTAL_REQUESTS_PER_MINUTE`, `..._PER_HOUR`,
`SWWAF_ANOMALY_TOTAL_BYTES_PER_MINUTE` and `..._PER_HOUR`.
- Named netblocks: `SWWAF_WATCH_NETS`, for example
`office=203.0.113.0/24,scraper-x=198.51.100.0/22`, with
`SWWAF_WATCH_REQUESTS_PER_MINUTE`, `..._PER_HOUR`,
`SWWAF_WATCH_BYTES_PER_MINUTE` and `..._PER_HOUR` applied to each named
netblock as a whole.
- Anomaly counting includes clients in `SWWAF_ALLOW_NETS` and
`SWWAF_RATE_LIMIT_EXEMPT_NETS`, since an exempt client misbehaving is worth
knowing about. At most 20,000 counters are kept, in memory and in
`alerts.json`.
## Rule files
`smallwebwaf` reads every `*.rules` file in `SWWAF_RULES_DIR` and checks each
request against them. The files are plain text meant to be edited by hand. A new
scanning pattern seen in the request log can be turned into a rule in one line.
`smallwebwaf` watches the directory: a file edited, added or removed there takes
effect while it runs.
- One rule per line, four fields separated by spaces or tabs. The fourth field
runs to the end of the line:
```
<id> <target> <action> <regex>
```
- Blank lines and lines starting with `#` are ignored.
- `id`: a short name of letters, digits, `-` and `_`, unique across all files.
It appears in the request log, the metrics, the alerts and the ban notes.
- `target`: what the regex is matched against.
- `path`: the URL path without the query, both as received and once
percent-decoded, so an encoded probe cannot slip past.
- `query`: the raw query string.
- `uri`: the path and query together, both as received and once
percent-decoded.
- `method`, `host`, `user_agent`, `referer`.
- `header:<Name>`: any one request header but `Host` and
`Transfer-Encoding`, which Go's HTTP server takes out of every request.
- Request bodies are not available to rule files. Reading them is the Core
Rule Set's job, once `SWWAF_WAF_BODY_LIMIT` switches it on.
- `action`:
- `log`: note the match in the request log and do nothing else.
- `block`: refuse the request with `403`. The client is not banned for it,
but the refusal counts toward the error burst.
- `ban`: the request is a clear sign of attack (see "Bans"). Use it only for
requests no real visitor sends. Anchor a path at the site root (`^/`): a
file of the same name deeper in a site can be ordinary content, such as a
file in a gitea repository.
- `regex`: Go regular expression syntax (RE2). It has no backreferences or
lookaround. In exchange matching time is linear in the input, so no rule can
be made to stall the proxy. `(?i)` at the front makes a rule ignore case. A
rule matches if the regex matches anywhere in the target; anchor with `^` and
`$` when that is not wanted.
- Files are read in name order (`00-default.rules` before `50-gitea.rules`),
rules in line order.
- Rules are compiled when their file is read and held in memory. At start, a
line that does not parse, a regex that does not compile, or a duplicate id
stops the process with a message naming the file and line. So does a
`SWWAF_RULES_DIR` that does not exist. While running, the same faults leave
the rules as they were, and the log and one `file_error` alert name the file
and line. Once the file is fixed, it is read again. An empty directory is not
an error. To run without rule files, mount an empty directory or set
`SWWAF_RULES_ENABLED=false`.
- The image ships a default file in `SWWAF_RULES_DIR`. The app's Dockerfile can
copy files of its own into it, beside the default. Mounting a directory over
it replaces the defaults; mounting single files into it adds to them. Many
editors save a file by replacing it, and a container keeps seeing the old one
when the file alone is mounted. So rules meant to be edited while
`smallwebwaf` runs belong in a mounted directory, with a copy of the default
file if the defaults are to stay. The lookup database is mounted the same way,
for the same reason.
- Clients in `SWWAF_ALLOW_NETS` are not checked.
The default file the image ships:
```
# 00-default.rules: probes no real visitor sends, anchored at the site root
# id target action regex
env-file path ban (?i)^/\.env(\.[a-z]+)?$
vcs-dir path ban (?i)^/\.(git|svn|hg|bzr)(/|$)
secrets-dir path ban (?i)^/\.(aws|ssh|docker|kube)/
secret-file path ban (?i)^/\.(htpasswd|htaccess|npmrc|netrc|pgpass|git-credentials|bash_history|DS_Store)$
editor-dir path ban (?i)^/\.(vscode|idea)/
backup-file path ban (?i)^/[^/]+\.(php(\.[a-z0-9]+|~)|sql(\.[a-z0-9]+)?)$
log-file path ban (?i)^/(debug|error|access)\.log$
compose-file path ban (?i)^/(docker-)?compose\.ya?ml$
php-shell path ban (?i)^/(shell|c99|r57|wso|alfa)\.php$
scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|masscan|zgrab|wpscan)\b
path-traversal uri block (\.\./){2,}
empty-agent user_agent log ^$
```
It is kept short and limited to patterns that are wrong for every app, with its
path rules anchored at the site root. In front of gitea, `/.env` is a probe,
while `/<owner>/<repo>/src/branch/main/.env.example` is a file in a repository
that any visitor may open. Anything app-specific belongs in a file the app's
Dockerfile adds or the deployer mounts. A request for `/wp-login.php`, for
example, is a clear sign of attack in front of gitea and an ordinary login in
front of WordPress:
```
# 50-gitea.rules: WordPress probes, which gitea never serves
wp-probe path ban (?i)^/(wp-login\.php|xmlrpc\.php|wp-admin/)
```
## Persistent state
All state lives in memory, and the files in `SWWAF_STATE_DIR` hold a copy of all
of it, so an orderly stop loses nothing. The one exception is log lines still
waiting to be sent to `SWWAF_LOG_REMOTE_URL`, which stdout has already carried.
The files are meant for people as well: an admin can read or edit them at any
time, and the running `smallwebwaf` takes the edit in.
The files, each holding one kind of state:
- `bans.json`: active and past bans, up to `SWWAF_MAX_BANS` that `smallwebwaf`
made and every ban an admin made. Per entry:
- the netblock, start and expiry (`null` for permanent);
- the cause: `attack`, `limit`, `admin` or `crowdsec`;
- a short reason;
- when it was lifted, if an admin lifted it;
- the notes below.
- `clients.json`: per client, its minute, hour and day counters and its history
since it was first seen. The history holds:
- the first and last time seen;
- the AS number, AS name and country last looked up, and when;
- total requests, and how many were forwarded and how many refused;
- bytes in each direction;
- responses by status class;
- offences by kind.
Clients that were never banned are kept too, so every client's history
survives a restart.
- `lookups.json`: GeoJS answers, one per client: the AS number, AS name and
country, when GeoJS was asked and when the answer was last used.
- `reputation.json`: the last good copy of each list fetched from a URL; the
cached DNS blocklist verdicts and AbuseIPDB scores, each with the time it was
fetched; and the AbuseIPDB checks spent today, so that a restart does not
reset the daily budget.
- `alerts.json`:
- the anomaly counters;
- for each event and netblock, source, file or anomaly scope, when it was
last sent and how many repeats the cooldown has held back since;
- the alerts sent this hour;
- the alerts still waiting to be sent.
Ban notes hold what an admin needs to decide whether to lift a ban, drawn from
what `smallwebwaf` already knows. Nothing is looked up to fill them:
- the AS number, AS name and country, added when the lookup answers (empty when
lookups are off);
- what was broken: the rule id and target that matched, or the trap path, or the
limit, its window and the count reached;
- the client's limit percentage, with the setting that gave it;
- the reputation sources that listed the client;
- the request that caused the ban: time, method, host, path with its query
string, status and user agent, each text cut to 256 bytes;
- the netblock's total requests since it was first seen, and the requests
refused under this ban so far, kept up to date while the ban lasts;
- how many earlier bans of each kind the netblock has had.
Each file is rewritten whole, so each has a bound. `clients.json` takes about 1
KiB per client, about 20 MiB at the default `SWWAF_MAX_TRACKED_CLIENTS`, which
written every 15 minutes is under 2 GiB of disk writes a day. `bans.json` takes
about 1.2 KiB per ban, at most about 2.5 KiB, so about 6 MiB at the default
`SWWAF_MAX_BANS`. With a full file, a hundred new bans a day come to about 600
MiB of writes. `lookups.json` takes about 150 bytes per answer, about 15 MiB
when full, under 1.5 GiB of writes a day. The other two files are usually a few
MiB or less. Writing a file of these sizes takes well under a second on an
ordinary disk, in the background, so rewriting whole files needs nothing
cleverer.
Format:
- indented JSON with a top-level `version` number;
- entries sorted by client address;
- times in RFC 3339 UTC;
- durations and sizes as plain numbers, with the unit in the field name;
- `clients.json` and `lookups.json` put each entry on one line instead. That
halves their size and lets `grep` show everything about one client.
The aim is that a person can open `bans.json` in an editor, find an address, and
remove or add an entry.
Writing:
- serialise from a snapshot taken under the lock, so requests are not held up
while the file is written;
- write to a temporary file in the same directory, sync it, rename it over the
real name, sync the directory. A crash at any point leaves either the old
complete file or the new complete file, never a partial one;
- `bans.json` is written `SWWAF_STATE_WRITE_DELAY` after a ban changes. Changes
that only update the counts in its notes wait for the write every
`SWWAF_STATE_COUNTER_INTERVAL`, as the other four files do. All five are
written on orderly shutdown (`SIGTERM`);
- before writing a file, `smallwebwaf` checks whether it changed on disk since
it last read or wrote it. If it did, `smallwebwaf` takes that edit in first,
so an admin's edit is never overwritten.
Reading at start:
- Active bans go into an in-memory prefix lookup; everything else into maps.
Counter buckets and cache entries whose time has passed are discarded.
- A missing file means empty state, and is normal on first run.
- A file that does not parse, or has an unknown `version`, stops the process
with a message naming the file and position. Starting with empty state would
silently forgive every repeat offender. Since writes are atomic, a broken file
can only come from a hand edit, which the editor should hear about.
- `SWWAF_STATE_DIR` not writable at start: the process exits.
Edits while running:
- `smallwebwaf` watches `SWWAF_STATE_DIR` and notices a file that is edited,
replaced or added. It tells its own writes from an admin's by comparing the
file with what it last wrote.
- An edit that parses is taken in at once: what the file says replaces what
`smallwebwaf` held for that file. Removing a ban's entry from `bans.json`
lifts the ban; adding an entry bans. Changes `smallwebwaf` made after the
admin opened the file, such as a new ban, are lost when the admin saves over
them. Most editors warn when a file changed on disk while it was open.
- An edit that does not parse does not stop the running `smallwebwaf`. It keeps
the state it has and renames the edited file to `<name>.bad` (for example
`bans.json.bad`), so the edit is kept. It writes the file again from memory,
logs the file and the position of the error, and sends them in one
`file_error` alert. The admin fixes the `.bad` file and moves it back.
A write that fails while running:
- state stays correct in memory;
- the failure is logged and counted in the metrics;
- one `file_error` alert is sent per cooldown;
- the write is retried at the next write.
What a hard kill can lose:
- ban changes from the last `SWWAF_STATE_WRITE_DELAY` (10 seconds);
- everything else from the last `SWWAF_STATE_COUNTER_INTERVAL` (15 minutes):
counts and history, the counts in ban notes, GeoJS answers, reputation
verdicts and the AbuseIPDB count, anomaly counters and alert cooldowns.
Losing the volume loses ban history, not service.
## Request log
One JSON object per line on stdout for every request, including refused ones,
apart from those Go's HTTP server ends before `smallwebwaf` sees them (see "Size
and time limits"). stdout is always on. When `SWWAF_LOG_REMOTE_URL` is set the
same lines are also sent there, so a deployment can stop depending on docker's
log handling while `docker logs` keeps working.
- Standard web log fields: `time` (RFC 3339 with milliseconds), `instance`,
`client_ip`, `method`, `scheme`, `host`, `path`, `query`, `protocol`,
`status`, `request_bytes`, `response_bytes`, `referer`, `user_agent`.
- Request detail:
- `request_id`, generated if traefik did not supply one, and passed to the
app;
- `peer_ip`, the TCP peer, normally traefik;
- `forwarded_for`, the header as received;
- `client_group`, the /64 or configured prefix used for counting;
- `asn`, `as_name`, `country`;
- `content_type`, `content_length`, and the headers named in
`SWWAF_LOG_REQUEST_HEADERS`;
- `has_authorization` and `has_cookie`, as booleans;
- `websocket`, when the connection was upgraded.
- Response detail: `response_content_type`, `upstream_status` (differs from
`status` when `smallwebwaf` answered itself), `cache_control`, `location` on
redirects, `aborted` when the client went away early.
- Decision:
- `action`: `forward`, `banned`, `denied`, `country_denied`, `rate_limited`,
`rule_blocked`, `waf_blocked`, `too_large`, `timed_out`, `upstream_error`,
or `admin` for a request `smallwebwaf` answered at one of its own
endpoints;
- `would_action`, in `observe` mode;
- `limit_percent`, and which setting gave it;
- `counts`, the client's minute, hour and day request and byte totals after
this request;
- `limit_hit`, which window;
- `rule_ids`, the rule file rules that matched;
- `waf_rule_ids` and `waf_score`;
- `reputation`, the sources that listed the client;
- `offence`, when one was recorded;
- `ban_expires`.
- Timings in milliseconds: `duration_total`, `duration_checks` (everything
`smallwebwaf` did before forwarding), `duration_waf`,
`duration_upstream_connect`, `duration_upstream_first_byte` and
`duration_upstream_total`.
- Bodies are never logged. Query strings are logged as received; an app that
carries secrets in query strings needs that fixed in the app.
- The process's own messages share the stream as JSON lines with
`"type":"process"`; request lines carry `"type":"request"`.
- Remote sending:
- each line is the message of an RFC 5424 syslog record, with octet-counted
framing on TCP and TLS;
- sending happens on its own goroutine from a bounded buffer
(`SWWAF_LOG_REMOTE_BUFFER`). An unreachable or slow server never delays a
request and never stops stdout. The sender reconnects with backoff, drops
the oldest lines when the buffer is full, and counts the drops in the
metrics. UDP gives no delivery signal at all and is offered only for
compatibility.
## Metrics endpoint
Prometheus text format at `/_smallwebwaf/metrics`, for a request carrying
`SWWAF_METRICS_TOKEN`.
- Traffic: requests and bytes in and out, by status class and `action`; request
duration and upstream duration histograms; requests in flight.
- Limits and bans:
- limit hits by window and kind (requests, bytes or refusals);
- size and time limit hits by limit;
- offences by kind;
- bans made by cause, permanent bans, active bans;
- requests refused by the country lists, by country.
- Attack detection:
- rule file matches by rule id and action, and the number of rules loaded;
- Core Rule Set matches by mode and rule id, limited to the rules that
fired.
- Lookup and reputation:
- requests and bytes by AS number and by country, limited to the
`SWWAF_METRICS_TOP_N` busiest of each with the rest summed as `other`, so
the label set stays bounded;
- reputation queries, hits, failures and remaining daily budget by source;
- GeoJS requests and failures, and clients that counted as unknown because
it did not answer in time;
- when each list was last fetched, and the lookup database last read.
- Housekeeping:
- tracked clients;
- state file writes, write failures, last successful write time and size per
file; edits taken in, and edits set aside because they did not parse;
- alerts sent, failed, suppressed and dropped by destination;
- remote log lines sent, dropped and buffer depth;
- the standard Go runtime and process metrics.
- No metric carries a client IP address as a label. Per-address questions are
answered by the request log and `GET /_smallwebwaf/clients/<ip>`.
## Admin endpoints
A request whose path starts with `/_smallwebwaf/` is for `smallwebwaf` itself:
it answers it and never passes it to the app. The prefix carries the tool's
name, so it takes no path an app uses. Admins and scrapers reach these endpoints
through traefik, like any other request.
- `GET /_smallwebwaf/healthz`: answers `200` with `ok` to anyone, without a
token, while `smallwebwaf` is running. It does not ask the app; the
container's health check does. It is answered before any check, so a health
checker never needs an exemption and is never refused, for example by
`SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES`.
- `GET /_smallwebwaf/metrics`: the metrics; needs `SWWAF_METRICS_TOKEN`.
- `GET /_smallwebwaf/bans`, `POST /_smallwebwaf/bans` (a client or netblock, a
duration, a reason) and `DELETE /_smallwebwaf/bans/<client>`: need
`SWWAF_ADMIN_TOKEN`. Editing `bans.json` does the same without a token.
- `GET /_smallwebwaf/clients/<ip>`: needs `SWWAF_ADMIN_TOKEN`. The client's
current counters and history, which holds its lookup result and offences, and
its bans with their notes. It answers "why was this address refused" and "what
has this netblock been doing".
- A token is sent as `Authorization: Bearer <token>` (see "Tokens" under
"Settings"). While a token is unset, the endpoints that need it answer `404`,
as does any other path under the prefix.
- Apart from the health check, these requests go through every check that any
other request goes through. They are answered at the point where another
request would be forwarded to the app: a banned or refused client stays
refused, and each request counts toward the client's limits. A missing or
wrong token is answered `401`, in `SWWAF_MODE=observe` too, and counts toward
the error burst. A client guessing tokens is therefore banned once it passes
`SWWAF_ERROR_BURST_THRESHOLD` guesses in a minute. A client in
`SWWAF_ALLOW_NETS` skips the checks but still needs the token.
- An admin whose own address is banned lifts the ban by editing `bans.json`.
## Alert webhook schema
One JSON object per alert:
- `instance`, `time`, `event` (one of the `SWWAF_ALERT_EVENTS` values, or
`summary` for the hourly summary);
- `client`, `netblock`, `asn`, `as_name`, `country`;
- `reason`: a short human-readable sentence;
- `detail`: event-specific fields, for example `window`, `count`, `limit`,
`limit_percent`, `rule_ids`, `path`, `ban_expires`, for a ban its notes, and
for `file_error` the file and the position of the error;
- `suppressed_repeats`: the number of identical alerts held back by the
cooldown.
## Deployment
`smallwebwaf` runs inside the app's own container: the app's Dockerfile builds
`FROM` the `smallwebwaf` image. Each app remains one container. An app is
hardened by changing its base image. It can add packages from nixpkgs.
The image holds:
- Ubuntu 26.04 LTS, the newest long-term support release of Ubuntu, pinned by
digest. The image moves to the next LTS release when that ships.
- `ca-certificates`, `nix-bin` and `runit` from Ubuntu, installed from a dated
snapshot of its archive and checked by hash (see "Pinning"). The Ubuntu image
has no CA certificates; without them Nix cannot download packages and
`smallwebwaf` cannot reach GeoJS.
- Nix and nixpkgs, fixed at one release (see "Pinning"). Root uses Nix directly;
no Nix daemon runs, and `build-users-group =` in `/etc/nix/nix.conf` lets root
build without the build users `nix-bin` does not create.
- runit, and `runsvinit` as the entrypoint, as the owner's code style guide asks
for service containers. No package carries `runsvinit`, so the image builds it
from its archived source (`github.com/peterbourgon/runsvinit`, unchanged
since 2015) at a fixed commit hash, with a `go.mod` the build writes.
`runsvinit` starts runit's `runsvdir`, which runs the `run` script of each
directory under `/etc/service` and runs it again whenever it exits. Ubuntu's
`sv` looks for services there too.
- The `smallwebwaf` binary, and a user of its own, `smallwebwaf` (uid and gid
65532).
- The service directory `/etc/service/smallwebwaf`. Its `run` script waits one
second (`sleep 1`), makes `SWWAF_STATE_DIR` and every file in it belong to the
`smallwebwaf` user, and starts `smallwebwaf` as that user with runit's
`chpst`.
- The directory `/var/lib/smallwebwaf` for the state files, and
`/etc/smallwebwaf/rules.d` with the default rule file.
- Port 8080 declared (`EXPOSE 8080`), and the health check described below
(`HEALTHCHECK`).
Every `run` script, the app's included, is a bash script. It starts with
`#!/usr/bin/env bash` and `set -euo pipefail`, and puts its code in a `main`
function, called on its last line, as the owner's code style guide asks. Ubuntu
ships bash.
Beyond its `FROM` line, the app's Dockerfile adds:
- its binary;
- any packages it needs, from nixpkgs, with `nix-env -iA nixpkgs.<name>`;
- its runit service: a directory under `/etc/service` named after the app, never
`smallwebwaf`. Its `run` script waits one second (`sleep 1`) and then starts
the app with `chpst`, listening on `127.0.0.1:8081`. The app runs as a user of
its own, which the Dockerfile creates with `useradd`. That user is neither
root nor `smallwebwaf`, so that the app cannot touch the state files or the
process of `smallwebwaf`.
It sets no `ENTRYPOINT` or `USER` of its own. The container must start
`runsvinit`, as root, so that it can start each service as its own user.
An example, for an app whose binary is `app` and which takes the address it
listens on and the proxies it trusts as options of its own:
```dockerfile
# The smallwebwaf image, pinned by digest.
FROM <registry>/smallwebwaf@sha256:<digest>
# Packages the app needs, if any, from the nixpkgs in the image.
RUN nix-env -iA nixpkgs.git
# The app's binary, and a user of its own to run it.
COPY app /usr/local/bin/app
RUN useradd --system --no-create-home --shell /usr/sbin/nologin app
# The app's runit service.
COPY --chmod=755 app.run /etc/service/app/run
```
with `app.run` beside the Dockerfile:
```bash
#!/usr/bin/env bash
set -euo pipefail
main() {
sleep 1
exec chpst -u app:app /usr/local/bin/app \
--listen 127.0.0.1:8081 \
--trusted-proxies 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.1/32,::1/128
}
main "$@"
```
### Pinning
- Ubuntu's packages come from Ubuntu's snapshot service, the archive as it was
at the moment the Dockerfile names (`--snapshot 20261001T000000Z`), never
earlier than the pinned Ubuntu image's date. The Dockerfile names the SHA-256
hash of each of the snapshot's `InRelease` files, which cover every package
list and so every package, and the build checks them before installing. The
hashes are the amd64 archive's, so the image is built for amd64 only. The
snapshot is fetched over HTTPS with the CA certificates of the Go image that
builds `smallwebwaf`, itself pinned by digest, since the Ubuntu image has
none.
- nixpkgs comes from a release file on `releases.nixos.org` (`nixexprs.tar.xz`
of `nixos-26.05.11045.774debe7a0d1`), named with its SHA-256 hash, not from a
GitHub archive, whose bytes can change. It is set up for root as `nixpkgs`.
Root's Nix profile, `/nix/var/nix/profiles/default/bin`, is last on the
`PATH`, so that no package hides the image's own commands; busybox's `sv`, for
one, looks for services elsewhere. Unpacked, nixpkgs takes about 500 MiB.
- Security fixes from Ubuntu and nixpkgs come with a newer `smallwebwaf` image,
which an app takes by changing the digest in its `FROM` line. The image moves
to each new nixpkgs release, every six months.
### The two processes
- `runsvinit`, `runsvdir` and each service's `runsv` run as root. Each `run`
script starts as root and hands over to its program's user with `chpst`. Both
services start with the container's environment variables and write to the
container's output, which `docker logs` shows.
- When `smallwebwaf` or the app exits, runit starts it again a second later. The
other process keeps running, and the container does not stop. While
`smallwebwaf` is down, traefik answers `502`. While the app is down,
`smallwebwaf` answers `502`, which is not a client offence.
- Only four things stop `smallwebwaf`, all at start:
- invalid configuration;
- a lookup database that is missing or unreadable;
- a rule file or state file that does not parse;
- an unwritable `SWWAF_STATE_DIR`.
- A `smallwebwaf` that stops at start is started again every second, logging its
message each time, until the cause is fixed. The health check fails meanwhile.
Once running, a broken helper or a broken edit never takes the protected
service down.
- `docker stop` sends `SIGTERM` to `runsvinit`, which has runit stop both
services at once, each with `SIGTERM`, and exits once they have stopped.
`smallwebwaf` stops taking requests, gives those in progress five seconds to
finish, and writes its state files. Those requests can finish only while the
app still answers. An app that exits at once on `SIGTERM` ends them: the
client gets `502`, or a response cut short. To let them finish, the app must
keep taking and answering requests for five seconds after `SIGTERM`.
- The container's root filesystem stays writable: runit writes each service's
status into its directory under `/etc/service`.
### The health check
The image's `HEALTHCHECK` runs `smallwebwaf healthcheck`. It passes while
`smallwebwaf` answers `GET /_smallwebwaf/healthz` on `127.0.0.1`, at the port in
`SWWAF_LISTEN_ADDR`, and the app accepts connections at the address in
`SWWAF_UPSTREAM_URL`. It fails when either does not. The container therefore
shows as healthy only while both processes are up. traefik sends a container no
requests until it shows as healthy. So the check runs every second from the
container's start until it first passes, for up to a minute, and every 30
seconds after that. An app with a health check of its own can replace the
image's `HEALTHCHECK` with one that checks both.
### Ports
`smallwebwaf` listens on no port but the one in `SWWAF_LISTEN_ADDR`, and its own
endpoints are on that listener. Another port may be set: the health check
follows the setting, traefik's port label
(`traefik.http.services.<name>.loadbalancer.server.port`) must name it, and the
app must leave it free. The address part of `SWWAF_LISTEN_ADDR` stays empty
(`:9000`, never `127.0.0.1:9000`), so that traefik reaches `smallwebwaf` on the
container's address and the health check on `127.0.0.1`. The app listens on
`127.0.0.1:8081` only. An app that listens on every address can be reached
around `smallwebwaf` by anything that reaches the container.
### State
The app keeps its data in directories of its own, such as `/var/lib/app`, and
`smallwebwaf` keeps its state files beside them in `SWWAF_STATE_DIR`. Without a
volume there, the files survive a restart of the container and are lost when a
deploy replaces it. A volume mounted at `/var/lib/smallwebwaf`, named or a host
directory, keeps them across deploys. The `run` script makes it and every file
in it belong to the `smallwebwaf` user, so a host directory mounted there needs
no change of owner. The volume is separate from the app's, holds a few tens of
MiB at most with the defaults, and needs no backup beyond the host's own. The
image declares no volume, since every app image built on it would inherit it.
### Tokens
A token given as a file (`SWWAF_ADMIN_TOKEN_FILE`, `SWWAF_METRICS_TOKEN_FILE`)
is out of the app's reach only while the `smallwebwaf` user alone can read the
file. The operator makes the file on the host, owned by uid 65532, the
`smallwebwaf` user, with mode `0400`, and mounts the directory that holds it
into the container read-only. The container sees the same owner and mode.
### Forwarded headers
The app's TCP peer is `smallwebwaf` on `127.0.0.1`, and the `X-Forwarded-For`
the app receives ends with traefik's address, which `smallwebwaf` adds after the
visitor's. The app must therefore trust `127.0.0.1` and `::1` for forwarded
headers, besides the private address ranges traefik is on. The usual default for
a trusted-proxy setting, the private ranges alone, does not include loopback: an
app left at it ignores the header and sees every visitor as `127.0.0.1`. A list
given replaces that default, so the app's own trusted-proxy setting names them
all, `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.1/32,::1/128`, as in the
example.
### upaas
An app deployed by upaas takes this shape with no change to upaas. upaas builds
the app's Dockerfile and runs the one container as it runs any app. The traefik
labels route to port 8080, and any `SWWAF_` settings go with the app's
environment variables. The volume for `/var/lib/smallwebwaf` and the directory
of any token file go beside the app's own volumes.
### Rollout per service
Build the app's image on the `smallwebwaf` image, with no setting, and deploy
it. Nothing needs tuning first. Afterwards the request log and the ban notes say
why each client was refused. A real visitor refused by mistake is let through
with an exclusion (`SWWAF_WAF_DISABLED_RULES`, `SWWAF_WAF_EXEMPT_PATHS`,
`SWWAF_RATE_LIMIT_EXEMPT_PATHS`) or an exemption
(`SWWAF_RATE_LIMIT_EXEMPT_NETS`, `SWWAF_ALLOW_NETS`), and its ban is lifted in
`bans.json`. Additions to consider at any time:
- an alert destination;
- country lists or biased thresholds;
- reputation sources;
- the metrics and admin tokens;
- a remote log server;
- an app-specific rule file.
gitea is built on the image like any other app. `docs/gitea.md` records what the
defaults do in front of it, and `script/gitea-requests` checks those notes.
## Risks the design has to handle
- Forged `X-Forwarded-For`: handled by step 2 of "What happens to a request".
The default trusts every private address, which in the intended deployment
means traefik. Where other containers reach `smallwebwaf` directly, or
visitors on a private network reach traefik, they can name another address in
the header. `SWWAF_TRUSTED_PROXIES` set to traefik's own network closes that
gap.
- Many clients behind one address (mobile carriers, offices, Tor) share its
limits and its bans. The defaults sit well above one person's use, the static
lists take known shared addresses out, and the ban notes show an admin what
happened before a ban is lifted.
- A `ban` rule that matches a real visitor bans it for a long time (see "Bans").
So the default rule file keeps to requests no real visitor sends, anchored at
the site root, and a ban is lifted in `bans.json`.
- IPv6 address rotation inside a /64: handled by grouping.
- Widely distributed scrapers, thousands of addresses at low rates each:
per-client limits do not see them. The AS number and netblock anomaly alerts
reveal them, and a low `SWWAF_ASN_LIMIT_PERCENT` for that AS number is the
response.
- Core Rule Set false positives: a match refuses only that request and bans no
one by itself. The request log names the rule, and exclusions by rule id and
path fix it. An engine error on a request forwards the request, and is logged
and counted. Attacks in request bodies are not refused by default, for the
reason under "Attack detection".
- The fourth and fifth changes to the Core Rule Set hold in front of every app,
and no setting restores those rules. An app that uses one of those query
parameters as a file on the server, or passes it to a shell, must check the
value itself. A rule file can refuse it instead. The same holds for an app
that reads a cookie named `gitea_flash` or `redirect_to`, or passes `Referer`
to a shell.
- The admin endpoints can be reached from the internet. All but the health check
need a token and are off while it is unset, and token guessing is banned
through the error burst.
- GeoJS, the default lookup source: every new visitor's address goes to a third
party, and a swarm of fresh addresses is when GeoJS may slow down or block
`smallwebwaf`. Kept answers and batched requests keep the number of requests
low. The file source has neither risk.
- Slow-request attacks: bounded by `SWWAF_CLIENT_REQUEST_TIMEOUT`,
`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES` and `SWWAF_CLIENT_IDLE_TIMEOUT`.
- Alert floods: the cooldown and the hourly cap.
## Planned, not built
Everything above is built. These two points are specified as later work, and
nothing in the code does them yet:
- Planned: sharing bans between the instances of several apps, by running one
CrowdSec engine per host that every instance reports its bans to. It is
reconsidered once real ban volumes are known. Until then each instance keeps
its own bans, and can read a CrowdSec engine's decisions as described under
"Reputation".
- Planned: sending the log over RELP, the reliable syslog transport. It is
revisited against the available Go packages. Until then the remote log goes
over UDP, TCP or TLS as described under "Request log".
Reference in New Issue View Git Blame Copy Permalink