diff --git a/SPEC.md b/SPEC.md index b1b7daa..f5c683e 100644 --- a/SPEC.md +++ b/SPEC.md @@ -4,8 +4,9 @@ runs inside the app's container, between traefik and the app. This document says what it must do and why, each requirement once. Everything in it is built, except the last section, "Planned, not built". `README.md` says how to run it. -`EVALUATION.md` says why no existing tool was adopted. `docs/gitea.md` records -how it behaves in front of gitea. +`EVALUATION.md` says why no existing tool was adopted. `docs/` holds the lists +this document links to, and `docs/gitea.md` records how `smallwebwaf` behaves in +front of gitea. ## Purpose @@ -19,8 +20,8 @@ how it behaves in front of gitea. - It bans abusers: briefly at first, for seven days on a clear sign of attack, and permanently when they keep at it. - It looks up each client's AS number and country. It can refuse whole countries - and netblocks, and lower its limits for listed AS numbers and countries. It - consults reputation sources. It sends alerts. + and netblocks, lower its limits for listed AS numbers and countries, consult + reputation sources, and send alerts. - Everything is set by environment variables whose names start with `SWWAF_`, since `smallwebwaf` shares its container, and so its environment, with the app. The only other inputs are files: the rule files, the state files, a @@ -38,25 +39,20 @@ how it behaves in front of gitea. ## Architecture -- One statically linked Go binary. It ships in an image built on Ubuntu 26.04 - LTS with nixpkgs installed. That image is the base of the app's own image, so - `smallwebwaf` and the app run in one container (see "Deployment"). runit - starts `smallwebwaf` as a non-root user of its own. `smallwebwaf` writes only - to its state directory. -- One listener, on port 8080, which traefik routes to. It forwards requests to - the app on `127.0.0.1:8081`. It answers its own endpoints under - `/_smallwebwaf/` (see "Admin endpoints"). Those never reach the app. +- One statically linked Go binary. It writes only to its state directory. It + ships in the image that is the base of the app's own image (see "Deployment"). - No database of any kind. Nothing is read from disk while serving a request. -- The proxy is the standard library's `httputil.ReverseProxy`. The other parts - of the process are each specified below. - Libraries, checked against the owner's Go dependency defaults: - - the standard library for the proxy, HTTP clients, DNS, logging - (`log/slog`) and the state files (`encoding/json`, `os.Rename`); + - the standard library for the proxy (`httputil.ReverseProxy`), HTTP + clients, DNS, logging (`log/slog`) and the state files (`encoding/json`, + `os.Rename`); - `github.com/corazawaf/coraza/v3` and `github.com/corazawaf/coraza-coreruleset/v4` at `v4.25.0`, which carries - the Core Rule Set 4.25.0. The changes `smallwebwaf` makes to the Core Rule - Set, and the default `SWWAF_WAF_DISABLED_RULES`, are built for that - version. They are checked again before the version changes; + the Core Rule Set 4.25.0. The changes `smallwebwaf` makes to it, and the + default `SWWAF_WAF_DISABLED_RULES`, are built for that version. They are + checked again before the version changes, since rule ids, and what each + rule matches, change between versions; + - `github.com/hashicorp/golang-lru/v2` for the bounded tables; - `github.com/oschwald/maxminddb-golang/v2` to read the lookup database; - `github.com/fsnotify/fsnotify` to notice edited files; - `github.com/prometheus/client_golang` for the metrics; @@ -72,32 +68,36 @@ names the section that holds its rules. 1. `GET /_smallwebwaf/healthz` is answered at once, before any check. 2. The client is identified. - If the TCP peer is inside `SWWAF_TRUSTED_PROXIES`, `X-Forwarded-For` is - walked from the right. The first address outside those netblocks is the - client. If every address is inside them, the leftmost is the client, so - that a visitor on a private network is known by its own address. With no - header, the peer is the client. + walked from the right, and the first address outside those netblocks is + the client. If every address is inside them, the leftmost is, so that a + visitor on a private network is known by its own address. With no header, + the peer is the client. - If the peer is outside `SWWAF_TRUSTED_PROXIES`, it is the client, and the header is ignored. - An IPv6 client is its group: the netblock of `SWWAF_IPV6_GROUP_PREFIX` around its address, a /64 by default, since one abuser usually controls a whole /64. A client is therefore one IPv4 address or one IPv6 group. -3. The static lists. A client in `SWWAF_ALLOW_NETS` skips every check below. It - is still counted for the anomaly thresholds. A client in `SWWAF_DENY_NETS` is - refused. -4. The ban ledger. A client under a ban is refused (see "Bans"). +3. The static lists. A client in `SWWAF_ALLOW_NETS` skips steps 4 to 12 and the + byte limits. It is still counted for the anomaly thresholds, and still needs + a token at step 13. A client in `SWWAF_DENY_NETS` is refused; a client in + both lists is let through. +4. Bans. A client under a ban is refused (see "Bans"). 5. The lookup, unless `SWWAF_LOOKUP_SOURCE` is `off` (see "Lookup of AS number and country"). The request goes on at once. It waits for the first GeoJS answer, up to `SWWAF_LOOKUP_TIMEOUT`, only when one of these needs it: - a country list; - a biased threshold; - `SWWAF_ADD_LOOKUP_HEADERS`. -6. The country lists (see "Country lists"). A refused request is not an offence. + + A request that ends before the answer comes is logged without it. + +6. The country lists (see "Country lists"). 7. The reputation sources (see "Reputation"). A client on a fetched blocklist gets `SWWAF_BLOCKLIST_ACTION`. A client on the CrowdSec decision list is banned. A client listed by a DNS blocklist, or scored at least `SWWAF_ABUSEIPDB_MIN_SCORE` by AbuseIPDB, gets `SWWAF_REPUTATION_ACTION`. With no cached verdict, a query is queued in the background and the request - goes on. A first request is never delayed by a reputation query. + goes on, so a first request is never delayed by a reputation query. 8. The client's limit percentage (see "Biased thresholds"). 9. Rate limiting (see "Request rate limits, per client"). A request over a limit times the percentage is refused and bans the client. @@ -105,9 +105,10 @@ names the section that holds its rules. bans the client. 11. The rule files (see "Rule files"). Matching stops at the first rule that refuses. -12. The Core Rule Set (see "Attack detection"). In `block` mode a match is - refused with `403`. In `detect` mode it is only logged and alerted. -13. A request under `/_smallwebwaf/` is answered here and goes no further. +12. The Core Rule Set (see "Attack detection"). A match is refused with `403` + while `SWWAF_WAF_MODE` is `block`, and only logged and alerted while it is + `detect`. +13. A request under `/_smallwebwaf/` is answered here (see "Admin endpoints"). 14. The request is forwarded to `SWWAF_UPSTREAM_URL`, streaming, with `X-Forwarded-For` added, and `X-Client-ASN` and `X-Client-Country` when enabled. @@ -129,7 +130,7 @@ the token check still apply. - Each window (minute, hour, day) uses two adjacent fixed buckets per client. The earlier bucket is weighted by how much of it still overlaps the sliding - window. This costs a few integers per client per window. It avoids the burst + window. This costs a few integers per client per window and avoids the burst at bucket boundaries that a single fixed bucket allows. - Counters are read and updated in memory. They are written to `clients.json` and loaded again at start (see "Persistent state"), so a restart does not hand @@ -137,14 +138,13 @@ the token check still apply. load. - Memory is bounded. When the table of clients holds `SWWAF_MAX_TRACKED_CLIENTS`, the least recently seen client is dropped first, - with its history. The bans are bounded by `SWWAF_MAX_BANS` (see "Bans"). The - GeoJS answers, the reputation verdicts and the anomaly counters have fixed - bounds, given where each is described. + with its history. The bans, the GeoJS answers, the reputation verdicts and the + anomaly counters have bounds of their own, given where each is described. ## Bans -A ban refuses every request from a netblock with `SWWAF_BAN_RESPONSE` until it -ends. The netblock is the client: its IPv4 address, widened to +Until a ban ends, every request from its netblock is refused with +`SWWAF_BAN_RESPONSE`. The netblock is the client: its IPv4 address, widened to `SWWAF_BAN_SCOPE_V4_PREFIX`, or its IPv6 group. At the defaults a ban covers one address or one /64, so a permanent ban does not reach neighbours who did nothing. Requests refused under a ban are not counted against limits. They are @@ -193,20 +193,24 @@ the cause `crowdsec`. Only a client that sends a request is banned, so a long list does not fill `bans.json`. Such a ban is never made permanent. It does not make the netblock's next ban for a broken limit longer. -Some refusals make no ban at all. `SWWAF_DENY_NETS`, a blocklist or reputation -hit whose action is `deny`, and the country lists refuse each request they cover -and record nothing in `bans.json`. +These refuse each request they cover, record nothing in `bans.json`, and count +no offence: -Every ban carries notes with what an admin needs to decide whether to lift it -(see "Persistent state"). An admin can add or lift a ban at any time by editing -`bans.json`, or through the ban endpoints when `SWWAF_ADMIN_TOKEN` is set. A ban -lifted before it ends is kept, marked as lifted, and does not count toward a -longer ban. Deleting its entry from `bans.json` forgets it entirely. +- `SWWAF_DENY_NETS`; +- a blocklist or reputation hit whose action is `deny`; +- the country lists. -A permanent ban does not run out. It ends when an admin lifts it, or when the -table of bans is full. When `smallwebwaf` holds `SWWAF_MAX_BANS` bans of its -own, it drops the one that has gone longest without a request, whether past, -active or permanent. A scanner whose ban was dropped that way is refused and +Every ban carries notes with what an admin needs to decide whether to lift it. +An admin adds or lifts a ban by editing `bans.json` (see "Persistent state"), or +through the ban endpoints when `SWWAF_ADMIN_TOKEN` is set. A ban lifted before +it ends is kept, marked as lifted, and does not count toward a longer ban. +Deleting its entry from `bans.json` forgets it entirely. + +A permanent ban does not run out. It ends when an admin lifts it, or when +`smallwebwaf` drops it from a full table, as below. When `smallwebwaf` holds +`SWWAF_MAX_BANS` bans of its own, it drops the earliest ban of one netblock: the +netblock that has gone longest without a request. A past, active or permanent +ban is dropped alike. A scanner whose ban was dropped that way is refused and banned again by its next probe. A ban an admin made, whose cause is `admin`, is never dropped and does not count toward `SWWAF_MAX_BANS`, so such bans can never fill the table. An admin who wants to keep a ban `smallwebwaf` made sets its @@ -222,6 +226,7 @@ Conventions: codes in either case. A code that is not a country, such as `nk`, stops the start with a message naming it. - These settings take `off`, which switches them off: + - `SWWAF_LOOKUP_SOURCE` and `SWWAF_WAF_MODE`; - the five timeouts under "Size and time limits"; - `SWWAF_REQUEST_MAX_BYTES` and `SWWAF_RESPONSE_MAX_BYTES`; - the three rate limits and the three byte limits; @@ -261,7 +266,10 @@ The settings, by group: the private address ranges): the netblocks whose `X-Forwarded-For` is believed, normally where traefik reaches `smallwebwaf` from. A client that connects directly cannot claim another address. A list given replaces the - default; set but empty, it trusts nothing. + default; set but empty, it trusts nothing. Where other containers reach + `smallwebwaf` directly, or visitors on a private network reach traefik, they + can name another address in the header; this setting, narrowed to traefik's + own network, closes that gap. - `SWWAF_IPV6_GROUP_PREFIX` (default `64`), from 32 to 128. - `SWWAF_MAX_TRACKED_CLIENTS` (default `20000`): the clients held in memory and in `clients.json`. @@ -271,26 +279,25 @@ The settings, by group: ### Tokens - `SWWAF_ADMIN_TOKEN`: the bearer token for the ban endpoints and - `/_smallwebwaf/clients/`. Unset by default, which switches them off; bans - are then managed by editing `bans.json`. + `/_smallwebwaf/clients/`. Unset by default, which switches them off. - `SWWAF_METRICS_TOKEN`: the bearer token a scraper sends for `/_smallwebwaf/metrics`. Unset by default, which switches the metrics off, since they would otherwise be open to anyone on the internet. It is separate from `SWWAF_ADMIN_TOKEN`, so a scraper that holds it cannot manage bans. - A token is a long random value, since the endpoints can be reached from the internet. One shorter than 32 characters stops the start. The app starts with - the same environment variables as `smallwebwaf`, so it can read a token given - as one. The file form keeps a token out of the app's reach (see "Tokens" under - "Deployment"). + the same environment variables as `smallwebwaf`, so it can read a token set as + an environment variable. The file form keeps a token out of the app's reach + (see "Tokens" under "Deployment"). ### State files - `SWWAF_STATE_DIR` (default `/var/lib/smallwebwaf`): the JSON state files, in a directory of their own beside the app's data, belonging to the `smallwebwaf` user. -- `SWWAF_STATE_WRITE_DELAY` (default `10s`): `bans.json` is written this long - after a ban is made, lifted or made permanent, so a burst of changes becomes - one write. +- `SWWAF_STATE_WRITE_DELAY` (default `10s`): how long `bans.json` waits to be + written after a ban is made, lifted or made permanent, so a burst of changes + becomes one write. - `SWWAF_STATE_COUNTER_INTERVAL` (default `15m`): how often every state file is written. @@ -316,7 +323,8 @@ The settings, by group: ### Metrics - `SWWAF_METRICS_TOP_N` (default `50`): how many AS numbers and countries get - series of their own. The rest are summed as `other`. + series of their own. The rest are summed as `other`, so the label set stays + bounded. ### Static lists @@ -325,6 +333,10 @@ The settings, by group: error burst, attack detection and bans still apply. - `SWWAF_DENY_NETS`: always refused. +Many clients share one address (mobile carriers, offices, Tor), and so share its +limits and its bans. The defaults sit well above one person's use, and the first +two lists take known shared addresses out. + ### Request rate limits, per client Breaking one bans the client, so the defaults sit several times above what one @@ -352,9 +364,10 @@ largest response allowed, and no single download breaks one. These apply per request, in both directions. The client-facing timeouts apply between the client and `smallwebwaf`, the app-facing ones between `smallwebwaf` and the app, since a slow client and a slow app are separate problems. There is -one size limit for request bodies and one for response bodies, since -`smallwebwaf` passes bodies through unchanged. Bodies stream straight through, -so a request body reaches the app while the client is still sending it. +one size limit for request bodies and one for response bodies. Bodies pass +through unchanged, so a limit on each side would bound the same bytes, and the +lower one would always decide. Bodies stream straight through, so a request body +reaches the app while the client is still sending it. - `SWWAF_CLIENT_REQUEST_TIMEOUT` (default `60s`): how long a client may take to send its request line and headers, and then, from the end of the headers, its @@ -380,22 +393,21 @@ What the client gets when a limit is passed: - Go's HTTP server reads a request's line and headers before `smallwebwaf` sees the request. It answers `431` to headers over - `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`. A client too slow to send them gets - either no answer or an error answer, mostly `400`. Either way the server - closes the connection, nothing reaches the app, and the request log gets no - line. + `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, and a client too slow to send them + gets an error answer, mostly `400`, or none. Either way the server closes the + connection, nothing reaches the app, and the request log gets no line. - A request that announces a body larger than its limit is refused before anything reaches the app. -- When a limit is passed after the headers have been read, but before the - response has started, `smallwebwaf` answers itself: +- A limit passed after the headers have been read, but before the response has + started, is answered by `smallwebwaf`: - `413` for a request body that is too large; - `408` for a client too slow to send its body; - `502` for a response that is too large; - `504` for an app that is too slow. - A request body streams through, so each side can hold up the other. A request timeout that runs out while the body is still on its way answers `408` if - `smallwebwaf` was waiting for the client at that moment. It answers `504` if - it was waiting for the app to take what it had. + `smallwebwaf` was waiting for the client at that moment, and `504` if it was + waiting for the app. - Once the response has started it can only be cut off, and the connection is closed. - A WebSocket connection leaves these limits behind once it is upgraded. It @@ -404,25 +416,26 @@ What the client gets when a limit is passed: ### Lookup of AS number and country On by default through GeoJS, which needs no account or file, so that country -lookup works with no setup. The file is the alternative for a service that keeps -its visitors' addresses on its own host. +lookup works with no setup. A local database file is the alternative for a +service that keeps its visitors' addresses on its own host. -- `SWWAF_LOOKUP_SOURCE` (default `geojs`): `geojs`, `file` or `off`. One source - at a time: `file` without `SWWAF_LOOKUP_DB_PATH`, or `SWWAF_LOOKUP_DB_PATH` - with any other source, stops the start with a message naming both. So does a - setting that needs lookups while the source is `off`: the country lists, the - biased thresholds, `SWWAF_ADD_LOOKUP_HEADERS`, and the anomaly thresholds per - AS number. -- `SWWAF_LOOKUP_DB_PATH`: the file for `file`: the IPinfo Lite database in its - `.mmdb` form (`ipinfo_lite.mmdb`), whose `asn`, `as_name` and `country_code` +- `SWWAF_LOOKUP_SOURCE` (default `geojs`): `geojs`, `file` or `off`. Only one + source is used. The start stops, naming both settings, when `file` is set + without `SWWAF_LOOKUP_DB_PATH`, or `SWWAF_LOOKUP_DB_PATH` with any other + source. The start also stops when the source is `off` and one of these needs + lookups: + - a country list; + - a biased threshold; + - `SWWAF_ADD_LOOKUP_HEADERS`; + - an anomaly threshold per AS number. +- `SWWAF_LOOKUP_DB_PATH`: the database the `file` source reads: the IPinfo Lite + database (`ipinfo_lite.mmdb`), whose `asn`, `as_name` and `country_code` fields are read. The operator downloads it with a free IPinfo account, mounts - it read-only, and refreshes it; IPinfo updates it daily. `smallwebwaf` never - fetches it and holds no account or token for it. A file that is missing or - unreadable at start stops the start. `smallwebwaf` notices when the file is - replaced and reads it again. A replacement it cannot read is ignored, logged - and sent as one `file_error` alert, and the file it has stays in use. The - directory that holds the file is mounted, not the file itself (see "Rule - files" for why). + the directory that holds it read-only (see "Rule files" for why), and + refreshes it. `smallwebwaf` never fetches it. A file that is missing or + unreadable at start stops the start. A replaced file is read again; a + replacement that cannot be read is logged, sent as one `file_error` alert, and + ignored, the file in use staying in use. - `geojs` asks the free GeoJS web service about up to 200 addresses in one request (`https://get.geojs.io/v1/ip/geo.json?ip=a,b,c`), one request at a time, and reads each answer's `country_code`, `asn` and `organization_name`. @@ -434,18 +447,21 @@ its visitors' addresses on its own host. - A client the lookup cannot place has an unknown country and AS number. The exclusive country list refuses it, and the biased thresholds give it `SWWAF_UNKNOWN_LIMIT_PERCENT`. Nothing else treats it differently. -- Each GeoJS answer is kept for 7 days, in memory and in `lookups.json`, apart - from the table of clients. Up to 100,000 answers are kept, about 15 MiB; past - that, the answer used longest ago goes first. -- `SWWAF_LOOKUP_TIMEOUT` (default `1s`): how long a request waits for its - client's first answer when a setting needs it, and how long a request to GeoJS - may take. A client whose answer has not come in time counts as unknown until - it comes, and its later requests do not wait. +- Each GeoJS answer is kept for 7 days, in memory and in `lookups.json`, + separately from the table of clients, so that it outlasts a client dropped + from that table. Up to 100,000 answers are kept, about 15 MiB; past that, the + answer used longest ago goes first. +- `SWWAF_LOOKUP_TIMEOUT` (default `1s`): how long a visitor's request waits for + its client's first answer when a setting needs it. It is also how long a + request to GeoJS may take. A client whose answer has not come in time counts + as unknown until it comes, and its later requests do not wait. - GeoJS publishes no rate limit, but its terms forbid "an excessive amount of - API requests", judged by GeoJS alone, and let it block a caller. While GeoJS - is slow, down or refusing `smallwebwaf`, new clients count as unknown, - `smallwebwaf` keeps asking with backoff, and one `source_failure` alert is - sent per cooldown. A service that cannot accept this uses the file. + API requests", judged by GeoJS alone, and let it block a caller. Kept answers + and batched requests keep the number of requests low. While GeoJS is slow, + down or refusing `smallwebwaf`, new clients count as unknown, `smallwebwaf` + keeps asking with backoff, and one `source_failure` alert is sent per + cooldown. A service that cannot accept this, or that every new visitor's + address goes to a third party, uses the file. - `SWWAF_ADD_LOOKUP_HEADERS` (default `false`): pass `X-Client-ASN` and `X-Client-Country` to the app. @@ -458,20 +474,20 @@ where it comes from. Clients in `SWWAF_ALLOW_NETS` are not checked. a listed country is refused. - `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES`: for example `us,de`. Only clients placed in a listed country get through. Every request from any other country - is refused. So is every request from a client the lookup cannot place, such as - a visitor on the local network, another container or internal monitoring. - Those that should reach the app go in `SWWAF_ALLOW_NETS`. A list that let - unplaced clients through would let every new client in whenever GeoJS stops - answering. That is the nearest a failing helper comes to taking the service - down: GeoJS failing while this list is set refuses every new client. + is refused. So is every request from a client the lookup cannot place (see + "Lookup of AS number and country"), such as internal monitoring; those that + should reach the app go in `SWWAF_ALLOW_NETS`. A list that let unplaced + clients through would let every new client in whenever GeoJS stops answering. + The cost: while GeoJS fails, this list refuses every new client. No other + failure of GeoJS, a reputation source or an alert destination comes this close + to taking the service down. - Both may be set. `SWWAF_DENIED_COUNTRIES` then adds nothing, since the exclusive list already refuses every other country. A code on both lists stops the start with a message naming it. - A refused request is answered with `SWWAF_BAN_RESPONSE`, as soon as the client's address has been looked up, before its body is read. It skips the reputation checks, the limits, the rule files and the Core Rule Set. It is - logged with the action `country_denied`, and counted in the metrics. It is a - refusal, not a ban. + logged with the action `country_denied`, and counted in the metrics. ### Biased thresholds @@ -481,14 +497,14 @@ abuse comes from. - `SWWAF_ASN_LIMIT_PERCENT`: for example `AS14061:50,AS16276:50,AS45102:25`. Clients in a listed AS number get that percentage of every request and byte - limit, so the rules in "Bans" ban them after fewer requests than others. `0` - is a zero allowance: the client's first request breaks a limit and bans it. + limit. `0` is a zero allowance: the client's first request breaks a limit and + bans it. - `SWWAF_COUNTRY_LIMIT_PERCENT`: the same form with country codes, for example `CN:25,RU:50`. - `SWWAF_ASN_BYTES_PERCENT` and `SWWAF_COUNTRY_BYTES_PERCENT`: optional overrides applied to the byte limits only, when the byte percentage should differ from the request percentage. -- `SWWAF_UNKNOWN_LIMIT_PERCENT` (default `100`): for clients the lookup cannot +- `SWWAF_UNKNOWN_LIMIT_PERCENT` (default `100`): for a client the lookup cannot place. - `SWWAF_ASN_LIMIT_PERCENT_URL`: optional URL of a text file of `AS:percent` lines, so one abuse-source list can be shared by every `smallwebwaf` in the @@ -508,12 +524,11 @@ By default the Core Rule Set reads the method, the URL with its query string, and the headers of every request. That is where automated attacks show: injection in query strings, path traversal, attacks carried in headers, scanners' user agents. It reads no request body. On a code forge, the example -deployment, bodies carry what people write and publish: issue and comment text, -wiki pages, files saved in the web editor, package descriptions. When these hold -shell commands or code, the Core Rule Set takes them for attacks. The default -gives up refusing an attack carried in a body. The rule files, the limits and -the bans still apply to the client that sends it, and `SWWAF_WAF_BODY_LIMIT` -switches body inspection on. +deployment, bodies carry what people write and publish, such as issue text and +files saved in the web editor. The Core Rule Set takes the shell commands and +code in them for attacks. The default gives up refusing an attack carried in a +body; the rule files, the limits and the bans still apply to the client that +sends it, and `SWWAF_WAF_BODY_LIMIT` switches body inspection on. - `SWWAF_RULES_DIR` (default `/etc/smallwebwaf/rules.d`): the directory of rule files (see "Rule files"). @@ -522,6 +537,9 @@ switches body inspection on. `block`. - `SWWAF_WAF_PARANOIA_LEVEL` (default `1`) and `SWWAF_WAF_ANOMALY_THRESHOLD` (default `5`): the Core Rule Set's own two tuning values, at its own defaults. + Each rule that matches adds its score to the request's. A request whose score + reaches the threshold is a match. With the threshold `off`, nothing is a + match, and scores are still logged. - `SWWAF_WAF_DISABLED_RULES` (default `920340,920420,920440,920640,930130,930140`): rule ids to switch off when an app trips a false positive. The default switches off six rules: @@ -536,7 +554,7 @@ switches body inspection on. package uploads, and views of ordinary files in a repository. At the site root, where no app serves such files, the default rule file bans the common probes these rules caught (see "Rule files"). A list given replaces the - default, so include them in it. + default, so name these six ids in it to keep them off. - `SWWAF_WAF_EXEMPT_PATHS`: path prefixes not inspected. - `SWWAF_WAF_BODY_LIMIT` (default `off`): `off` reads no request body. A size, @@ -545,15 +563,8 @@ switches body inspection on. on without holding it in memory. It reads JSON and XML bodies no larger than the size, since those cannot be read in part. Any other body is not read: the Core Rule Set would read it as form data, where binary content such as a git - push trips rules written for text. Responses are never inspected. Body - inspection suits apps whose forms carry no code. In front of gitea it refuses - the requests below, until their rule ids, named in the request log, are added - to `SWWAF_WAF_DISABLED_RULES`: - - issue and comment text, wiki pages and files saved in the web editor that - hold shell commands or code (932125, 932235, 932250 and others); - - package descriptions that show code; - - PyPI uploads (922130); - - attachments named like `debug.log` or `config.yml` (932180). + push trips rules written for text. Body inspection suits apps whose forms + carry no code. `docs/gitea.md` records what it refuses in front of gitea. - `SWWAF_TRAP_PATHS`: paths the app never serves and only scanners ask for, for example `/wp-login.php,/xmlrpc.php` in front of gitea. A request for one is a clear sign of attack. This is the short form of a `path` rule with the `ban` @@ -564,30 +575,32 @@ switches body inspection on. trying one attack or token after another is refused many times a minute; a person rarely more than a few times. Answers the app gives are not counted, since they do not tell a scanner from an ordinary client. In front of gitea, - registry and git clients are answered `404` and `401` by design, hundreds of - times a minute. + registry and git clients get `404` and `401` by design, hundreds of times a + minute. +- A Core Rule Set engine error on a request forwards the request, and is logged + and counted. `smallwebwaf` changes the Core Rule Set 4.25.0 in six ways, since in front of gitea it would otherwise refuse ordinary requests. The changes hold in front of every app, and no setting undoes them: -- `PUT`, `PATCH` and `DELETE` are allowed methods besides `GET`, `HEAD`, `POST` - and `OPTIONS`. APIs, container image pushes and package uploads use them. - Other methods stay refused. -- The request header `Expect` is allowed, since curl and git send it before some - large uploads. `Content-Encoding` is allowed on a request whose body the Core - Rule Set does not read, which by default is every request. git compresses a - fetch request over 1 KiB and says so in that header. On a body the Core Rule - Set does read, the header stays refused, since a compressed body cannot be - inspected. The other headers the Core Rule Set refuses, such as `Proxy`, stay - refused. -- The `redirect_uri` parameter is not checked for a URL naming an IP address or - `localhost` (931100, 934110). Git Credential Manager, git-credential-oauth and - tea, which gitea registers for OAuth sign-in, ask to be sent back to - `http://127.0.0.1` on the user's own machine. The server never fetches that - address. -- Ten query parameters are not checked against the Core Rule Set's lists of - system files (930120), shell paths (932160) and command names (932260): +1. `PUT`, `PATCH` and `DELETE` are allowed methods besides `GET`, `HEAD`, `POST` + and `OPTIONS`. APIs, container image pushes and package uploads use them. + Other methods stay refused. +2. The request header `Expect` is allowed, since curl and git send it before + some large uploads. `Content-Encoding` is allowed on a request whose body the + Core Rule Set does not read, which by default is every request. git + compresses a fetch request over 1 KiB and says so in that header. On a body + the Core Rule Set does read, the header stays refused, since a compressed + body cannot be inspected. The other headers the Core Rule Set refuses, such + as `Proxy`, stay refused. +3. The `redirect_uri` parameter is not checked for a URL naming an IP address or + `localhost` (931100, 934110). Git Credential Manager, git-credential-oauth + and tea, which gitea registers for OAuth sign-in, ask to be sent back to + `http://127.0.0.1` on the user's own machine. The server never fetches that + address. +4. Ten query parameters are not checked against the Core Rule Set's lists of + system files (930120), shell paths (932160) and command names (932260): - `path`, `files`, `skip-to` and `sub_path`; - `ref`, `sha`, `branch` and `workflow`; - `artifactName` and `redirect_to`. @@ -599,55 +612,53 @@ every app, and no setting undoes them: `|cat /etc/passwd` included. Path traversal, SQL and script injection, and PHP, Java and Node.js code are still refused there. Every other parameter keeps all three rules. Parameter names are matched without regard to case, - as Coraza matches them. + as Coraza matches them. An app that uses one of these parameters as a file + on the server, or passes it to a shell, must check the value itself. A rule + file can refuse it instead. -- The Core Rule Set reads the request without the `gitea_flash` and - `redirect_to` cookies. It does not check `Referer` for a Unix command given - without arguments (932340) or for Java starting a process (944110). gitea puts - a message naming what was just done into `gitea_flash`. It puts the page to - return to after signing in into `redirect_to`. The Core Rule Set takes file - names in them, such as `package.json`, for attacks. A browser sends such a - cookie with every request until gitea replaces it. gitea cannot replace it - while those requests are refused, so one refusal would keep the browser out of - the whole site. `Referer` names the page a request comes from, on everything - that page loads. The Core Rule Set takes some page addresses for attacks, such - as a search for `env`, or OpenJDK's `Runtime.java`, whose path holds both - `runtime` and `java.`. gitea shows the message with any script removed and - returns only to its own pages. Every other cookie is read in full, and - `Referer` keeps the other rules. -- Responses are not inspected. A raw file from a repository, such as a shell - script, looks to the response rules like source code leaking from the server. +5. Two cookies are left out, and `Referer` keeps fewer rules: + - The Core Rule Set reads the request without the `gitea_flash` and + `redirect_to` cookies. gitea puts a message naming what was just done into + `gitea_flash`. It puts the page to return to after signing in into + `redirect_to`. The Core Rule Set takes file names in them, such as + `package.json`, for attacks. A browser sends such a cookie with every + request until gitea replaces it. gitea cannot replace it while those + requests are refused, so one refusal would keep the browser out of the + whole site. gitea shows the message with any script removed and returns + only to its own pages. Every other cookie is read in full. An app that + reads a cookie of either name must check it itself. + - `Referer` is not checked for a Unix command given without arguments + (932340) or for Java starting a process (944110). `Referer` names the page + a request comes from, on everything that page loads and on every link + followed from it. The Core Rule Set takes some page addresses for attacks, + such as a search for `env`, or OpenJDK's `Runtime.java`, whose path holds + both `runtime` and `java.`. All of those requests would be refused. + `Referer` keeps the other rules. An app that passes `Referer` to a shell + must check it itself. +6. Responses are not inspected. A raw file from a repository, such as a shell + script, looks to the response rules like source code leaking from the server. -The fourth and fifth changes leave an app on its own in those places (see "Risks -the design has to handle"). +### Ban response and scope -### Ban lengths and response - -- `SWWAF_ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of - attack. -- `SWWAF_LIMIT_BAN_DURATION` (default `1h`): the ban for a first broken limit. -- `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): breaking a limit again within - this time after a ban for a broken limit ended makes the next ban three times - as long. -- `SWWAF_MAX_BAN_DURATION` (default `7d`): a ban that would be longer is - permanent instead. - `SWWAF_BAN_RESPONSE` (default `403`): `403`, `429`, or `close` to drop the connection without an answer. `403` makes a ban easy to recognise when debugging. Behind traefik, `close` does not leave the client unanswered: traefik answers `502`, as it does whenever its backend drops a connection. - `SWWAF_BAN_SCOPE_V4_PREFIX` (default `32`): widen to for example `24` to ban the surrounding netblock. +- `SWWAF_ATTACK_BAN_DURATION`, `SWWAF_LIMIT_BAN_DURATION`, + `SWWAF_LIMIT_BAN_REPEAT_WINDOW` and `SWWAF_MAX_BAN_DURATION`: the ban lengths, + with their defaults, under "Bans". ### Reputation -No source is on by default, for the reason the country lists and the biased -thresholds are empty. A source acts on a list kept elsewhere; the defaults judge -a client only by what it does here. AbuseIPDB and CrowdSec also need an account -or an engine of the operator's own. Besides, the list best suited to be on by -default, Spamhaus DROP, must not be downloaded automatically more than once an -hour. Spamhaus may block an address that downloads it too often. Each -`smallwebwaf` fetches its own copy. On a host that runs several apps with it, -which share one address, downloads would often come less than an hour apart. +No source is on by default. A source acts on a list kept elsewhere; the defaults +judge a client only by what it does here. AbuseIPDB and CrowdSec also need an +account or an engine of the operator's own. The list best suited to be on by +default, Spamhaus DROP, must not be fetched more than once an hour, and Spamhaus +may block an address that does. Each `smallwebwaf` fetches its own copy, so +several apps on one host, sharing one address, would often fetch less than an +hour apart. - `SWWAF_BLOCKLIST_URLS`: text files of addresses and netblocks, one per line; anything after a `;` or `#` on a line is ignored. For example the Spamhaus @@ -663,9 +674,8 @@ which share one address, downloads would often come less than an hour apart. - `SWWAF_BLOCKLIST_ACTION` (default `deny`): `deny`, `limit:` or `log`. `deny` refuses every request from a listed address, which suits lists of networks that send nothing legitimate, such as DROP. `limit:` gives - listed clients that percentage of every limit, so they are banned after fewer - requests. It suits a list of addresses shared with ordinary visitors, such as - Tor exits. + listed clients that percentage of every limit. It suits a list of addresses + shared with ordinary visitors, such as Tor exits. - `SWWAF_DNSBL_ZONES`: for example `dnsbl.dronebl.org`. Zones meant for mail (lists of residential ranges) will block ordinary visitors and should not be used. Spamhaus zones need their keyed query service, given as a zone name @@ -677,15 +687,17 @@ which share one address, downloads would often come less than an hour apart. checks a day). Only clients that have already committed one offence are queried, so the budget is spent on suspects. - `SWWAF_CROWDSEC_LAPI_URL` and `SWWAF_CROWDSEC_LAPI_KEY`: optional. If a - CrowdSec engine exists on the host, its decision list is pulled on a schedule - and treated as a blocklist (see "Bans" for the ban it makes). This is how a - fleet-wide blocklist can arrive without `smallwebwaf` depending on CrowdSec. + CrowdSec engine exists on the host, its decision list is pulled on a schedule. + It is kept like a fetched blocklist, and a client on it is banned (see + "Bans"). This is how a fleet-wide blocklist can arrive without `smallwebwaf` + depending on CrowdSec. - `SWWAF_REPUTATION_ACTION` (default `limit:25`): `deny`, `limit:` or `log`, for DNS blocklist and AbuseIPDB hits. Such verdicts are less certain - than a blocklist, so by default a listed client gets a quarter of every limit - and is banned after a quarter of the requests. + than a blocklist, so by default a listed client gets a quarter of every limit. - `SWWAF_REPUTATION_CACHE_TTL` (default `24h`) and `SWWAF_REPUTATION_TIMEOUT` (default `2s`). +- Up to 100,000 DNS blocklist verdicts and 100,000 AbuseIPDB scores are kept. + Past that, the one fetched longest ago goes first. - A source that is down or over quota gives no verdict. The service continues, with one `source_failure` alert per cooldown. @@ -704,9 +716,10 @@ which share one address, downloads would often come less than an hour apart. - a rule file or state file edited while running that does not parse; - a replacement lookup database that cannot be read; - a state file that cannot be written. -- `SWWAF_ALERT_COOLDOWN` (default `15m`): the same event for the same netblock, - source, file or anomaly scope is not repeated within this time. A count of - suppressed repeats is included in the next one. +- `SWWAF_ALERT_COOLDOWN` (default `15m`): the same event is not repeated within + this time for the same netblock, source or file, or, for an anomaly, for the + same thing counted: one client, netblock, AS number or named netblock, or the + whole service. A count of suppressed repeats is included in the next one. - `SWWAF_ALERT_MAX_PER_HOUR` (default `60`): beyond this, alerts are rolled into one summary per hour, so a wide attack cannot flood the channel. - A destination that is down is retried with backoff from a bounded queue, the @@ -717,7 +730,9 @@ which share one address, downloads would often come less than an hour apart. These only alert; nothing is blocked. None is set by default, and a threshold left unset sends no alert. An alert needs a destination only the operator can supply, and what counts as unusual depends on each service's normal traffic, -which the metrics show. +which the metrics show. They are what reveals a widely distributed scraper, +thousands of addresses at low rates each, which the per-client limits do not +see; a low `SWWAF_ASN_LIMIT_PERCENT` for its AS number is the response. - Per client: `SWWAF_ANOMALY_CLIENT_REQUESTS_PER_MINUTE`, `SWWAF_ANOMALY_CLIENT_REQUESTS_PER_HOUR`, @@ -745,10 +760,10 @@ which the metrics show. ## Rule files `smallwebwaf` reads every `*.rules` file in `SWWAF_RULES_DIR` and checks each -request against them. The files are plain text meant to be edited by hand. A new -scanning pattern seen in the request log can be turned into a rule in one line. -`smallwebwaf` watches the directory: a file edited, added or removed there takes -effect while it runs. +request against them, unless the client is in `SWWAF_ALLOW_NETS`. The files are +plain text meant to be edited by hand, so that a new scanning pattern seen in +the request log becomes a rule in one line. `smallwebwaf` watches the directory: +a file edited, added or removed there takes effect while it runs. - One rule per line, four fields separated by spaces or tabs. The fourth field runs to the end of the line: @@ -773,12 +788,12 @@ effect while it runs. Rule Set's job, once `SWWAF_WAF_BODY_LIMIT` switches it on. - `action`: - `log`: note the match in the request log and do nothing else. - - `block`: refuse the request with `403`. The client is not banned for it, - but the refusal counts toward the error burst. + - `block`: refuse the request with `403`, without a ban (see "Bans"). - `ban`: the request is a clear sign of attack (see "Bans"). Use it only for requests no real visitor sends. Anchor a path at the site root (`^/`): a file of the same name deeper in a site can be ordinary content, such as a - file in a gitea repository. + file in a gitea repository. A visitor banned by mistake is let back in by + lifting the ban. - `regex`: Go regular expression syntax (RE2). It has no backreferences or lookaround. In exchange matching time is linear in the input, so no rule can be made to stall the proxy. `(?i)` at the front makes a rule ignore case. A @@ -792,17 +807,15 @@ effect while it runs. `SWWAF_RULES_DIR` that does not exist. While running, the same faults leave the rules as they were, and the log and one `file_error` alert name the file and line. Once the file is fixed, it is read again. An empty directory is not - an error. To run without rule files, mount an empty directory or set - `SWWAF_RULES_ENABLED=false`. + an error: the log says that no rules were loaded. To run without rule files, + mount an empty directory or set `SWWAF_RULES_ENABLED=false`. - The image ships a default file in `SWWAF_RULES_DIR`. The app's Dockerfile can copy files of its own into it, beside the default. Mounting a directory over it replaces the defaults; mounting single files into it adds to them. Many editors save a file by replacing it, and a container keeps seeing the old one when the file alone is mounted. So rules meant to be edited while - `smallwebwaf` runs belong in a mounted directory, with a copy of the default - file if the defaults are to stay. The lookup database is mounted the same way, - for the same reason. -- Clients in `SWWAF_ALLOW_NETS` are not checked. + `smallwebwaf` runs belong in a mounted directory. The lookup database is + mounted the same way, for the same reason. The default file the image ships: @@ -824,18 +837,12 @@ path-traversal uri block (\.\./){2,} empty-agent user_agent log ^$ ``` -It is kept short and limited to patterns that are wrong for every app, with its -path rules anchored at the site root. In front of gitea, `/.env` is a probe, -while `///src/branch/main/.env.example` is a file in a repository -that any visitor may open. Anything app-specific belongs in a file the app's -Dockerfile adds or the deployer mounts. A request for `/wp-login.php`, for -example, is a clear sign of attack in front of gitea and an ordinary login in -front of WordPress: - -``` -# 50-gitea.rules: WordPress probes, which gitea never serves -wp-probe path ban (?i)^/(wp-login\.php|xmlrpc\.php|wp-admin/) -``` +It is kept short and limited to patterns that are wrong for every app. In front +of gitea, `/.env` is a probe, while +`///src/branch/main/.env.example` is a file in a repository that +any visitor may open. Anything app-specific, such as a rule banning WordPress +probes in front of gitea, belongs in a file the app's Dockerfile adds or the +deployer mounts. ## Persistent state @@ -868,14 +875,16 @@ The files, each holding one kind of state: - `lookups.json`: GeoJS answers, one per client: the AS number, AS name and country, when GeoJS was asked and when the answer was last used. -- `reputation.json`: the last good copy of each list fetched from a URL; the - cached DNS blocklist verdicts and AbuseIPDB scores, each with the time it was - fetched; and the AbuseIPDB checks spent today, so that a restart does not - reset the daily budget. +- `reputation.json`: + - the last good copy of each list fetched from a URL; + - the cached DNS blocklist verdicts and AbuseIPDB scores, each with the time + it was fetched; + - the AbuseIPDB checks spent today, so that a restart does not reset the + daily budget. - `alerts.json`: - the anomaly counters; - - for each event and netblock, source, file or anomaly scope, when it was - last sent and how many repeats the cooldown has held back since; + - for each event and netblock, source, file or thing counted, when it was + last sent, and the count of repeats held back since; - the alerts sent this hour; - the alerts still waiting to be sent. @@ -895,17 +904,17 @@ what `smallwebwaf` already knows. Nothing is looked up to fill them: - how many earlier bans of each kind the netblock has had. Each file is rewritten whole, so each has a bound. `clients.json` takes about 1 -KiB per client, about 20 MiB at the default `SWWAF_MAX_TRACKED_CLIENTS`, which -written every 15 minutes is under 2 GiB of disk writes a day. `bans.json` takes +KiB per client, about 20 MiB at the default `SWWAF_MAX_TRACKED_CLIENTS`. Written +every 15 minutes, that is under 2 GiB of disk writes a day. `bans.json` takes about 1.2 KiB per ban, at most about 2.5 KiB, so about 6 MiB at the default `SWWAF_MAX_BANS`. With a full file, a hundred new bans a day come to about 600 -MiB of writes. `lookups.json` takes about 150 bytes per answer, about 15 MiB -when full, under 1.5 GiB of writes a day. The other two files are usually a few -MiB or less. Writing a file of these sizes takes well under a second on an -ordinary disk, in the background, so rewriting whole files needs nothing -cleverer. +MiB of writes. `lookups.json` takes about 150 bytes per answer, under 1.5 GiB of +writes a day when full. The other two files are usually a few MiB or less. +Writing a file of these sizes takes well under a second, in the background, so +rewriting whole files needs nothing cleverer. -Format: +Format, so that a person can open `bans.json` in an editor, find an address, and +remove or add an entry: - indented JSON with a top-level `version` number; - entries sorted by client address; @@ -914,9 +923,6 @@ Format: - `clients.json` and `lookups.json` put each entry on one line instead. That halves their size and lets `grep` show everything about one client. -The aim is that a person can open `bans.json` in an editor, find an address, and -remove or add an entry. - Writing: - serialise from a snapshot taken under the lock, so requests are not held up @@ -924,8 +930,7 @@ Writing: - write to a temporary file in the same directory, sync it, rename it over the real name, sync the directory. A crash at any point leaves either the old complete file or the new complete file, never a partial one; -- `bans.json` is written `SWWAF_STATE_WRITE_DELAY` after a ban changes. Changes - that only update the counts in its notes wait for the write every +- changes that only update the counts in a ban's notes wait for the write every `SWWAF_STATE_COUNTER_INTERVAL`, as the other four files do. All five are written on orderly shutdown (`SIGTERM`); - before writing a file, `smallwebwaf` checks whether it changed on disk since @@ -950,9 +955,10 @@ Edits while running: file with what it last wrote. - An edit that parses is taken in at once: what the file says replaces what `smallwebwaf` held for that file. Removing a ban's entry from `bans.json` - lifts the ban; adding an entry bans. Changes `smallwebwaf` made after the - admin opened the file, such as a new ban, are lost when the admin saves over - them. Most editors warn when a file changed on disk while it was open. + lifts the ban, an admin's own included; adding an entry bans. Changes + `smallwebwaf` made after the admin opened the file, such as a new ban, are + lost when the admin saves over them. Most editors warn when a file changed on + disk while it was open. - An edit that does not parse does not stop the running `smallwebwaf`. It keeps the state it has and renames the edited file to `.bad` (for example `bans.json.bad`), so the edit is kept. It writes the file again from memory, @@ -970,103 +976,41 @@ What a hard kill can lose: - ban changes from the last `SWWAF_STATE_WRITE_DELAY` (10 seconds); - everything else from the last `SWWAF_STATE_COUNTER_INTERVAL` (15 minutes): - counts and history, the counts in ban notes, GeoJS answers, reputation - verdicts and the AbuseIPDB count, anomaly counters and alert cooldowns. + - counts and history, and the counts in ban notes; + - GeoJS answers; + - reputation verdicts and the AbuseIPDB count; + - anomaly counters and alert cooldowns. Losing the volume loses ban history, not service. ## Request log -One JSON object per line on stdout for every request, including refused ones, +Every request gets one JSON object on one line on stdout, refused ones included, apart from those Go's HTTP server ends before `smallwebwaf` sees them (see "Size -and time limits"). stdout is always on. When `SWWAF_LOG_REMOTE_URL` is set the -same lines are also sent there, so a deployment can stop depending on docker's -log handling while `docker logs` keeps working. +and time limits"). `docs/request-log.md` lists the fields. Bodies are never +logged. Query strings are logged as received; an app that carries secrets in +query strings needs that fixed in the app. The process's own messages share the +stream as JSON lines with `"type":"process"`; request lines carry +`"type":"request"`. -- Standard web log fields: `time` (RFC 3339 with milliseconds), `instance`, - `client_ip`, `method`, `scheme`, `host`, `path`, `query`, `protocol`, - `status`, `request_bytes`, `response_bytes`, `referer`, `user_agent`. -- Request detail: - - `request_id`, generated if traefik did not supply one, and passed to the - app; - - `peer_ip`, the TCP peer, normally traefik; - - `forwarded_for`, the header as received; - - `client_group`, the /64 or configured prefix used for counting; - - `asn`, `as_name`, `country`; - - `content_type`, `content_length`, and the headers named in - `SWWAF_LOG_REQUEST_HEADERS`; - - `has_authorization` and `has_cookie`, as booleans; - - `websocket`, when the connection was upgraded. -- Response detail: `response_content_type`, `upstream_status` (differs from - `status` when `smallwebwaf` answered itself), `cache_control`, `location` on - redirects, `aborted` when the client went away early. -- Decision: - - `action`: `forward`, `banned`, `denied`, `country_denied`, `rate_limited`, - `rule_blocked`, `waf_blocked`, `too_large`, `timed_out`, `upstream_error`, - or `admin` for a request `smallwebwaf` answered at one of its own - endpoints; - - `would_action`, in `observe` mode; - - `limit_percent`, and which setting gave it; - - `counts`, the client's minute, hour and day request and byte totals after - this request; - - `limit_hit`, which window; - - `rule_ids`, the rule file rules that matched; - - `waf_rule_ids` and `waf_score`; - - `reputation`, the sources that listed the client; - - `offence`, when one was recorded; - - `ban_expires`. -- Timings in milliseconds: `duration_total`, `duration_checks` (everything - `smallwebwaf` did before forwarding), `duration_waf`, - `duration_upstream_connect`, `duration_upstream_first_byte` and - `duration_upstream_total`. -- Bodies are never logged. Query strings are logged as received; an app that - carries secrets in query strings needs that fixed in the app. -- The process's own messages share the stream as JSON lines with - `"type":"process"`; request lines carry `"type":"request"`. -- Remote sending: - - each line is the message of an RFC 5424 syslog record, with octet-counted - framing on TCP and TLS; - - sending happens on its own goroutine from a bounded buffer - (`SWWAF_LOG_REMOTE_BUFFER`). An unreachable or slow server never delays a - request and never stops stdout. The sender reconnects with backoff, drops - the oldest lines when the buffer is full, and counts the drops in the - metrics. UDP gives no delivery signal at all and is offered only for - compatibility. +When `SWWAF_LOG_REMOTE_URL` is set, the same lines are also sent there, so a +deployment can stop depending on docker's log handling while `docker logs` keeps +working: + +- each line is the message of an RFC 5424 syslog record, with octet-counted + framing on TCP and TLS; +- sending happens on its own goroutine from a bounded buffer + (`SWWAF_LOG_REMOTE_BUFFER`). An unreachable or slow server never delays a + request and never stops stdout. The sender reconnects with backoff, drops the + oldest lines when the buffer is full, and counts the drops in the metrics. UDP + gives no delivery signal at all and is offered only for compatibility. ## Metrics endpoint -Prometheus text format at `/_smallwebwaf/metrics`, for a request carrying -`SWWAF_METRICS_TOKEN`. - -- Traffic: requests and bytes in and out, by status class and `action`; request - duration and upstream duration histograms; requests in flight. -- Limits and bans: - - limit hits by window and kind (requests, bytes or refusals); - - size and time limit hits by limit; - - offences by kind; - - bans made by cause, permanent bans, active bans; - - requests refused by the country lists, by country. -- Attack detection: - - rule file matches by rule id and action, and the number of rules loaded; - - Core Rule Set matches by mode and rule id, limited to the rules that - fired. -- Lookup and reputation: - - requests and bytes by AS number and by country, limited to the - `SWWAF_METRICS_TOP_N` busiest of each with the rest summed as `other`, so - the label set stays bounded; - - reputation queries, hits, failures and remaining daily budget by source; - - GeoJS requests and failures, and clients that counted as unknown because - it did not answer in time; - - when each list was last fetched, and the lookup database last read. -- Housekeeping: - - tracked clients; - - state file writes, write failures, last successful write time and size per - file; edits taken in, and edits set aside because they did not parse; - - alerts sent, failed, suppressed and dropped by destination; - - remote log lines sent, dropped and buffer depth; - - the standard Go runtime and process metrics. -- No metric carries a client IP address as a label. Per-address questions are - answered by the request log and `GET /_smallwebwaf/clients/`. +`GET /_smallwebwaf/metrics` answers in the Prometheus text format, for a request +carrying `SWWAF_METRICS_TOKEN`. `docs/metrics.md` lists the metrics. No metric +carries a client IP address as a label. Per-address questions are answered by +the request log and `GET /_smallwebwaf/clients/`. ## Admin endpoints @@ -1083,11 +1027,11 @@ through traefik, like any other request. - `GET /_smallwebwaf/metrics`: the metrics; needs `SWWAF_METRICS_TOKEN`. - `GET /_smallwebwaf/bans`, `POST /_smallwebwaf/bans` (a client or netblock, a duration, a reason) and `DELETE /_smallwebwaf/bans/`: need - `SWWAF_ADMIN_TOKEN`. Editing `bans.json` does the same without a token. -- `GET /_smallwebwaf/clients/`: needs `SWWAF_ADMIN_TOKEN`. The client's - current counters and history, which holds its lookup result and offences, and - its bans with their notes. It answers "why was this address refused" and "what - has this netblock been doing". + `SWWAF_ADMIN_TOKEN`. +- `GET /_smallwebwaf/clients/`: needs `SWWAF_ADMIN_TOKEN`. It answers with + the client's current counters, its history (with its lookup result and + offences), and its bans with their notes. It answers "why was this address + refused" and "what has this netblock been doing". - A token is sent as `Authorization: Bearer ` (see "Tokens" under "Settings"). While a token is unset, the endpoints that need it answer `404`, as does any other path under the prefix. @@ -1099,7 +1043,6 @@ through traefik, like any other request. the error burst. A client guessing tokens is therefore banned once it passes `SWWAF_ERROR_BURST_THRESHOLD` guesses in a minute. A client in `SWWAF_ALLOW_NETS` skips the checks but still needs the token. -- An admin whose own address is banned lifts the ban by editing `bans.json`. ## Alert webhook schema @@ -1123,32 +1066,17 @@ hardened by changing its base image. It can add packages from nixpkgs. The image holds: -- Ubuntu 26.04 LTS, the newest long-term support release of Ubuntu, pinned by - digest. The image moves to the next LTS release when that ships. -- `ca-certificates`, `nix-bin` and `runit` from Ubuntu, installed from a dated - snapshot of its archive and checked by hash (see "Pinning"). The Ubuntu image - has no CA certificates; without them Nix cannot download packages and - `smallwebwaf` cannot reach GeoJS. -- Nix and nixpkgs, fixed at one release (see "Pinning"). Root uses Nix directly; - no Nix daemon runs, and `build-users-group =` in `/etc/nix/nix.conf` lets root - build without the build users `nix-bin` does not create. -- runit, and `runsvinit` as the entrypoint, as the owner's code style guide asks - for service containers. No package carries `runsvinit`, so the image builds it - from its archived source (`github.com/peterbourgon/runsvinit`, unchanged - since 2015) at a fixed commit hash, with a `go.mod` the build writes. - `runsvinit` starts runit's `runsvdir`, which runs the `run` script of each - directory under `/etc/service` and runs it again whenever it exits. Ubuntu's - `sv` looks for services there too. -- The `smallwebwaf` binary, and a user of its own, `smallwebwaf` (uid and gid - 65532). -- The service directory `/etc/service/smallwebwaf`. Its `run` script waits one - second (`sleep 1`), makes `SWWAF_STATE_DIR` and every file in it belong to the - `smallwebwaf` user, and starts `smallwebwaf` as that user with runit's - `chpst`. -- The directory `/var/lib/smallwebwaf` for the state files, and - `/etc/smallwebwaf/rules.d` with the default rule file. -- Port 8080 declared (`EXPOSE 8080`), and the health check described below - (`HEALTHCHECK`). +- Ubuntu 26.04 LTS, with `ca-certificates`, `nix-bin` and `runit` from Ubuntu; +- nixpkgs, fixed at one release; +- `runsvinit` as the entrypoint; +- the `smallwebwaf` binary, a user of its own (uid and gid 65532), and its runit + service; +- its state and rules directories; +- `EXPOSE 8080` and the health check below. + +Every input to the image is pinned by digest or hash, so that an app built on it +gets the same packages each time. `docs/image.md` says how each part is pinned, +and why. Every `run` script, the app's included, is a bash script. It starts with `#!/usr/bin/env bash` and `set -euo pipefail`, and puts its code in a `main` @@ -1161,68 +1089,14 @@ Beyond its `FROM` line, the app's Dockerfile adds: - any packages it needs, from nixpkgs, with `nix-env -iA nixpkgs.`; - its runit service: a directory under `/etc/service` named after the app, never `smallwebwaf`. Its `run` script waits one second (`sleep 1`) and then starts - the app with `chpst`, listening on `127.0.0.1:8081`. The app runs as a user of - its own, which the Dockerfile creates with `useradd`. That user is neither - root nor `smallwebwaf`, so that the app cannot touch the state files or the - process of `smallwebwaf`. + the app with `chpst`, listening where `SWWAF_UPSTREAM_URL` points (see + "Ports"). The app runs as a user of its own, which the Dockerfile creates with + `useradd`. That user is neither root nor `smallwebwaf`, so that the app cannot + touch the state files or the process of `smallwebwaf`. It sets no `ENTRYPOINT` or `USER` of its own. The container must start `runsvinit`, as root, so that it can start each service as its own user. - -An example, for an app whose binary is `app` and which takes the address it -listens on and the proxies it trusts as options of its own: - -```dockerfile -# The smallwebwaf image, pinned by digest. -FROM /smallwebwaf@sha256: - -# Packages the app needs, if any, from the nixpkgs in the image. -RUN nix-env -iA nixpkgs.git - -# The app's binary, and a user of its own to run it. -COPY app /usr/local/bin/app -RUN useradd --system --no-create-home --shell /usr/sbin/nologin app - -# The app's runit service. -COPY --chmod=755 app.run /etc/service/app/run -``` - -with `app.run` beside the Dockerfile: - -```bash -#!/usr/bin/env bash -set -euo pipefail - -main() { - sleep 1 - exec chpst -u app:app /usr/local/bin/app \ - --listen 127.0.0.1:8081 \ - --trusted-proxies 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.1/32,::1/128 -} - -main "$@" -``` - -### Pinning - -- Ubuntu's packages come from Ubuntu's snapshot service, the archive as it was - at the moment the Dockerfile names (`--snapshot 20261001T000000Z`), never - earlier than the pinned Ubuntu image's date. The Dockerfile names the SHA-256 - hash of each of the snapshot's `InRelease` files, which cover every package - list and so every package, and the build checks them before installing. The - hashes are the amd64 archive's, so the image is built for amd64 only. The - snapshot is fetched over HTTPS with the CA certificates of the Go image that - builds `smallwebwaf`, itself pinned by digest, since the Ubuntu image has - none. -- nixpkgs comes from a release file on `releases.nixos.org` (`nixexprs.tar.xz` - of `nixos-26.05.11045.774debe7a0d1`), named with its SHA-256 hash, not from a - GitHub archive, whose bytes can change. It is set up for root as `nixpkgs`. - Root's Nix profile, `/nix/var/nix/profiles/default/bin`, is last on the - `PATH`, so that no package hides the image's own commands; busybox's `sv`, for - one, looks for services elsewhere. Unpacked, nixpkgs takes about 500 MiB. -- Security fixes from Ubuntu and nixpkgs come with a newer `smallwebwaf` image, - which an app takes by changing the digest in its `FROM` line. The image moves - to each new nixpkgs release, every six months. +"Getting started" in `README.md` shows such a Dockerfile and `run` script. ### The two processes @@ -1240,16 +1114,15 @@ main "$@" - a rule file or state file that does not parse; - an unwritable `SWWAF_STATE_DIR`. - A `smallwebwaf` that stops at start is started again every second, logging its - message each time, until the cause is fixed. The health check fails meanwhile. - Once running, a broken helper or a broken edit never takes the protected - service down. + message each time, until the cause is fixed; the health check fails meanwhile. + Once running, a failing GeoJS, reputation source or alert destination, or a + broken edit, never takes the protected service down. - `docker stop` sends `SIGTERM` to `runsvinit`, which has runit stop both services at once, each with `SIGTERM`, and exits once they have stopped. `smallwebwaf` stops taking requests, gives those in progress five seconds to - finish, and writes its state files. Those requests can finish only while the - app still answers. An app that exits at once on `SIGTERM` ends them: the - client gets `502`, or a response cut short. To let them finish, the app must - keep taking and answering requests for five seconds after `SIGTERM`. + finish, and writes its state files. An app that exits at once on `SIGTERM` + ends those requests: the client gets `502`, or a response cut short. To let + them finish, the app must keep answering for five seconds after `SIGTERM`. - The container's root filesystem stays writable: runit writes each service's status into its directory under `/etc/service`. @@ -1257,37 +1130,38 @@ main "$@" The image's `HEALTHCHECK` runs `smallwebwaf healthcheck`. It passes while `smallwebwaf` answers `GET /_smallwebwaf/healthz` on `127.0.0.1`, at the port in -`SWWAF_LISTEN_ADDR`, and the app accepts connections at the address in -`SWWAF_UPSTREAM_URL`. It fails when either does not. The container therefore -shows as healthy only while both processes are up. traefik sends a container no -requests until it shows as healthy. So the check runs every second from the -container's start until it first passes, for up to a minute, and every 30 -seconds after that. An app with a health check of its own can replace the +`SWWAF_LISTEN_ADDR`, and the app accepts connections at `SWWAF_UPSTREAM_URL`. So +the container shows as healthy only while both processes are up. traefik sends a +container no requests until it shows as healthy. So the check runs every second +from the container's start until it first passes, for up to a minute, and every +30 seconds after that. An app with a health check of its own can replace the image's `HEALTHCHECK` with one that checks both. ### Ports -`smallwebwaf` listens on no port but the one in `SWWAF_LISTEN_ADDR`, and its own -endpoints are on that listener. Another port may be set: the health check -follows the setting, traefik's port label +`smallwebwaf` listens on no port but the one in `SWWAF_LISTEN_ADDR`, 8080 by +default, which traefik routes to. Its own endpoints are on that listener, and it +forwards every other request to the app at `SWWAF_UPSTREAM_URL`, +`127.0.0.1:8081` by default. Another port may be set: the health check follows +the setting, traefik's port label (`traefik.http.services..loadbalancer.server.port`) must name it, and the app must leave it free. The address part of `SWWAF_LISTEN_ADDR` stays empty (`:9000`, never `127.0.0.1:9000`), so that traefik reaches `smallwebwaf` on the -container's address and the health check on `127.0.0.1`. The app listens on -`127.0.0.1:8081` only. An app that listens on every address can be reached +container's address and the health check on `127.0.0.1`. The app listens on that +loopback address only. An app that listens on every address can be reached around `smallwebwaf` by anything that reaches the container. ### State -The app keeps its data in directories of its own, such as `/var/lib/app`, and -`smallwebwaf` keeps its state files beside them in `SWWAF_STATE_DIR`. Without a -volume there, the files survive a restart of the container and are lost when a -deploy replaces it. A volume mounted at `/var/lib/smallwebwaf`, named or a host -directory, keeps them across deploys. The `run` script makes it and every file -in it belong to the `smallwebwaf` user, so a host directory mounted there needs -no change of owner. The volume is separate from the app's, holds a few tens of -MiB at most with the defaults, and needs no backup beyond the host's own. The -image declares no volume, since every app image built on it would inherit it. +`smallwebwaf` keeps its state files in `SWWAF_STATE_DIR`, beside the app's own +data directories. Without a volume there, the files survive a restart of the +container and are lost when a deploy replaces it. A volume mounted at +`/var/lib/smallwebwaf`, named or a host directory, keeps them across deploys. +The `run` script makes it and every file in it belong to the `smallwebwaf` user, +so a host directory mounted there needs no change of owner. The volume is +separate from the app's, holds a few tens of MiB at most with the defaults, and +needs no backup beyond the host's own. The image declares no volume, since every +app image built on it would inherit it. ### Tokens @@ -1306,8 +1180,7 @@ headers, besides the private address ranges traefik is on. The usual default for a trusted-proxy setting, the private ranges alone, does not include loopback: an app left at it ignores the header and sees every visitor as `127.0.0.1`. A list given replaces that default, so the app's own trusted-proxy setting names them -all, `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.1/32,::1/128`, as in the -example. +all: `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.1/32,::1/128`. ### upaas @@ -1315,69 +1188,9 @@ An app deployed by upaas takes this shape with no change to upaas. upaas builds the app's Dockerfile and runs the one container as it runs any app. The traefik labels route to port 8080, and any `SWWAF_` settings go with the app's environment variables. The volume for `/var/lib/smallwebwaf` and the directory -of any token file go beside the app's own volumes. - -### Rollout per service - -Build the app's image on the `smallwebwaf` image, with no setting, and deploy -it. Nothing needs tuning first. Afterwards the request log and the ban notes say -why each client was refused. A real visitor refused by mistake is let through -with an exclusion (`SWWAF_WAF_DISABLED_RULES`, `SWWAF_WAF_EXEMPT_PATHS`, -`SWWAF_RATE_LIMIT_EXEMPT_PATHS`) or an exemption -(`SWWAF_RATE_LIMIT_EXEMPT_NETS`, `SWWAF_ALLOW_NETS`), and its ban is lifted in -`bans.json`. Additions to consider at any time: - -- an alert destination; -- country lists or biased thresholds; -- reputation sources; -- the metrics and admin tokens; -- a remote log server; -- an app-specific rule file. - -gitea is built on the image like any other app. `docs/gitea.md` records what the -defaults do in front of it, and `script/gitea-requests` checks those notes. - -## Risks the design has to handle - -- Forged `X-Forwarded-For`: handled by step 2 of "What happens to a request". - The default trusts every private address, which in the intended deployment - means traefik. Where other containers reach `smallwebwaf` directly, or - visitors on a private network reach traefik, they can name another address in - the header. `SWWAF_TRUSTED_PROXIES` set to traefik's own network closes that - gap. -- Many clients behind one address (mobile carriers, offices, Tor) share its - limits and its bans. The defaults sit well above one person's use, the static - lists take known shared addresses out, and the ban notes show an admin what - happened before a ban is lifted. -- A `ban` rule that matches a real visitor bans it for a long time (see "Bans"). - So the default rule file keeps to requests no real visitor sends, anchored at - the site root, and a ban is lifted in `bans.json`. -- IPv6 address rotation inside a /64: handled by grouping. -- Widely distributed scrapers, thousands of addresses at low rates each: - per-client limits do not see them. The AS number and netblock anomaly alerts - reveal them, and a low `SWWAF_ASN_LIMIT_PERCENT` for that AS number is the - response. -- Core Rule Set false positives: a match refuses only that request and bans no - one by itself. The request log names the rule, and exclusions by rule id and - path fix it. An engine error on a request forwards the request, and is logged - and counted. Attacks in request bodies are not refused by default, for the - reason under "Attack detection". -- The fourth and fifth changes to the Core Rule Set hold in front of every app, - and no setting restores those rules. An app that uses one of those query - parameters as a file on the server, or passes it to a shell, must check the - value itself. A rule file can refuse it instead. The same holds for an app - that reads a cookie named `gitea_flash` or `redirect_to`, or passes `Referer` - to a shell. -- The admin endpoints can be reached from the internet. All but the health check - need a token and are off while it is unset, and token guessing is banned - through the error burst. -- GeoJS, the default lookup source: every new visitor's address goes to a third - party, and a swarm of fresh addresses is when GeoJS may slow down or block - `smallwebwaf`. Kept answers and batched requests keep the number of requests - low. The file source has neither risk. -- Slow-request attacks: bounded by `SWWAF_CLIENT_REQUEST_TIMEOUT`, - `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES` and `SWWAF_CLIENT_IDLE_TIMEOUT`. -- Alert floods: the cooldown and the hourly cap. +of any token file go beside the app's own volumes. gitea is built on the image +like any other app; `docs/gitea.md` records what the defaults do in front of it, +and `script/gitea-requests` checks those notes. ## Planned, not built diff --git a/docs/gitea.md b/docs/gitea.md index 85d7a40..d703385 100644 --- a/docs/gitea.md +++ b/docs/gitea.md @@ -172,6 +172,18 @@ the runner. More than 30 such refusals from its address within a minute, as when many jobs upload at once, ban it through the error burst. `actions/upload-artifact@v4` does not send the header. +## What body inspection refuses + +With `SWWAF_WAF_BODY_LIMIT` set, the Core Rule Set reads request bodies too. In +front of gitea it then refuses these, until their rule ids, named in the request +log, are added to `SWWAF_WAF_DISABLED_RULES`: + +- issue and comment text, wiki pages and files saved in the web editor that hold + shell commands or code (932125, 932235, 932250 and others); +- package descriptions that show code; +- PyPI uploads (922130); +- attachments named like `debug.log` or `config.yml` (932180). + ## What `script/gitea-requests` sends Of the requests above, the script sends: diff --git a/docs/image.md b/docs/image.md new file mode 100644 index 0000000..c208356 --- /dev/null +++ b/docs/image.md @@ -0,0 +1,56 @@ +# What the image holds + +The `smallwebwaf` image is the base of each app's own image. The requirements +are under "Deployment" in [`SPEC.md`](../SPEC.md); this file lists what the +image holds, how each part is pinned, and why. + +## Contents + +- Ubuntu 26.04 LTS, the newest long-term support release of Ubuntu, pinned by + digest. The image moves to the next LTS release when that ships. +- `ca-certificates`, `nix-bin` and `runit` from Ubuntu, installed from a dated + snapshot of its archive and checked by hash. The Ubuntu image has no CA + certificates; without them Nix cannot download packages and `smallwebwaf` + cannot reach GeoJS. +- Nix and nixpkgs, fixed at one release. Root uses Nix directly; no Nix daemon + runs. `nix-bin` creates no build users, so `build-users-group =` in + `/etc/nix/nix.conf` lets root build without them. +- runit, and `runsvinit` as the entrypoint, as the owner's code style guide asks + for service containers. No package carries `runsvinit`, so the image builds it + from its archived source (`github.com/peterbourgon/runsvinit`, unchanged + since 2015) at a fixed commit hash, with a `go.mod` the build writes. + `runsvinit` starts runit's `runsvdir`, which runs the `run` script of each + directory under `/etc/service` and runs it again whenever it exits. Ubuntu's + `sv` looks for services there too. +- The `smallwebwaf` binary, and a user of its own, `smallwebwaf` (uid and gid + 65532). +- The service directory `/etc/service/smallwebwaf`. Its `run` script waits one + second (`sleep 1`), makes `SWWAF_STATE_DIR` and every file in it belong to the + `smallwebwaf` user, and starts `smallwebwaf` as that user with runit's + `chpst`. +- The directory `/var/lib/smallwebwaf` for the state files, and + `/etc/smallwebwaf/rules.d` with the default rule file. +- Port 8080 declared (`EXPOSE 8080`), and the health check (`HEALTHCHECK`) that + "The health check" in `SPEC.md` describes. + +## Pinning + +- Ubuntu's packages come from Ubuntu's snapshot service, the archive as it was + at the moment the Dockerfile names (`--snapshot 20261001T000000Z`), never + earlier than the pinned Ubuntu image's date. The Dockerfile names the SHA-256 + hash of each of the snapshot's `InRelease` files, which cover every package + list and so every package, and the build checks them before installing. The + hashes are the amd64 archive's, so the image is built for amd64 only. The + snapshot is fetched over HTTPS with the CA certificates of the Go image that + builds `smallwebwaf`, itself pinned by digest, since the Ubuntu image has + none. +- nixpkgs comes from a release file on `releases.nixos.org` (`nixexprs.tar.xz` + of `nixos-26.05.11045.774debe7a0d1`), named with its SHA-256 hash, not from a + GitHub archive, whose bytes can change. It is set up for root as `nixpkgs`. + Root's Nix profile, `/nix/var/nix/profiles/default/bin`, is last on the + `PATH`, so that no package hides the image's own commands; busybox's `sv`, for + one, looks for services elsewhere. Unpacked, nixpkgs takes about 500 MiB. +- An app built on the image gets the same packages each time it is built. + Security fixes from Ubuntu and nixpkgs come with a newer `smallwebwaf` image, + which an app takes by changing the digest in its `FROM` line. The image moves + to each new nixpkgs release, every six months. diff --git a/docs/metrics.md b/docs/metrics.md new file mode 100644 index 0000000..10e9f58 --- /dev/null +++ b/docs/metrics.md @@ -0,0 +1,83 @@ +# The metrics + +`GET /_smallwebwaf/metrics` answers in the Prometheus text format. The +requirements are under "Metrics endpoint" in [`SPEC.md`](../SPEC.md); this file +lists the metrics. Every metric carries the label `instance`, set to +`SWWAF_INSTANCE_NAME`; Prometheus keeps it as `exported_instance` unless the +scrape sets `honor_labels: true`. No metric carries a client's address. + +## Traffic + +- `smallwebwaf_requests_total`, `smallwebwaf_request_bytes_total` and + `smallwebwaf_response_bytes_total`, by `status_class` such as `2xx` and by + `action`; +- `smallwebwaf_request_duration_seconds` and + `smallwebwaf_upstream_duration_seconds`, as histograms; +- `smallwebwaf_requests_in_flight`. + +## Limits and bans + +- `smallwebwaf_rate_limit_hits_total`, by `window` and by `kind` (`requests`, + `bytes` or `refusals`); +- `smallwebwaf_size_and_time_limit_hits_total`, by `limit`, the setting passed; +- `smallwebwaf_offences_total`, by `kind`; +- `smallwebwaf_bans_made_total`, by `cause`; +- `smallwebwaf_active_bans` and `smallwebwaf_permanent_bans`. + +## Attack detection + +- `smallwebwaf_rule_matches_total`, by `rule_id` and `action`, and + `smallwebwaf_rules_loaded`; +- `smallwebwaf_waf_matches_total`, by `mode` and `rule_id`, for the rules that + fired. + +## Countries and AS numbers + +For the `SWWAF_METRICS_TOP_N` busiest of each, the rest as `other`: + +- `smallwebwaf_country_requests_total`, + `smallwebwaf_country_request_bytes_total`, + `smallwebwaf_country_response_bytes_total` and + `smallwebwaf_country_list_refusals_total`, by `country`; +- `smallwebwaf_asn_requests_total`, `smallwebwaf_asn_request_bytes_total` and + `smallwebwaf_asn_response_bytes_total`, by `asn`. + +## Lookups + +- `smallwebwaf_geojs_requests_total` and `smallwebwaf_geojs_failures_total`; +- `smallwebwaf_geojs_unanswered_total`: the requests that needed an answer and + went on without it; +- with the file source, + `smallwebwaf_lookup_database_last_read_timestamp_seconds` and + `smallwebwaf_lookup_database_read_failures_total`. + +## Reputation sources + +By `source`: each list's URL, each zone, and `abuseipdb`: + +- `smallwebwaf_reputation_hits_total`, `smallwebwaf_reputation_queries_total` + and `smallwebwaf_reputation_failures_total`; +- `smallwebwaf_reputation_last_fetch_timestamp_seconds`, for a list; +- `smallwebwaf_reputation_daily_budget_remaining`, for AbuseIPDB. + +## State files + +By `file`: + +- `smallwebwaf_state_file_writes_total`, + `smallwebwaf_state_file_write_failures_total`, + `smallwebwaf_state_file_last_write_timestamp_seconds` and + `smallwebwaf_state_file_size_bytes`; +- `smallwebwaf_state_file_edits_taken_in_total` and + `smallwebwaf_state_file_edits_set_aside_total`; +- `smallwebwaf_tracked_clients`: the clients in the table. + +## Remote log, alerts and the process + +- while a syslog server is set, `smallwebwaf_remote_log_lines_sent_total`, + `smallwebwaf_remote_log_lines_dropped_total` and + `smallwebwaf_remote_log_buffer_depth`; +- by `destination`, `smallwebwaf_alerts_sent_total`, + `smallwebwaf_alerts_failed_total`, `smallwebwaf_alerts_suppressed_total` and + `smallwebwaf_alerts_dropped_total`; +- Go's own `go_` metrics and the process's `process_` metrics. diff --git a/docs/request-log.md b/docs/request-log.md new file mode 100644 index 0000000..e0c1922 --- /dev/null +++ b/docs/request-log.md @@ -0,0 +1,128 @@ +# The request log's fields + +`smallwebwaf` writes one JSON object per line on stdout for every request. The +requirements are under "Request log" in [`SPEC.md`](../SPEC.md); this file lists +the fields. A field that does not apply to a request is left out. + +## Example + +``` +{"type":"request","time":"2026-10-03T12:00:00.123Z","instance":"fsn1app1/gitea","client_ip":"203.0.113.9","method":"GET","scheme":"https","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","peer_ip":"172.18.0.2","forwarded_for":"203.0.113.9","client_group":"203.0.113.9/32","asn":"AS64496","as_name":"Example Net","country":"DE","request_headers":{"accept":"*/*"},"response_content_type":"text/html; charset=utf-8","upstream_status":200,"action":"forward","counts":{"minute":1,"hour":12,"day":40,"minute_bytes":5120,"hour_bytes":61440,"day_bytes":204800},"duration_total":3.217,"duration_checks":0.041,"duration_upstream_connect":0.052,"duration_upstream_first_byte":2.874,"duration_upstream_total":3.104} +``` + +## Standard web log fields + +- `time`: when the request arrived, RFC 3339 with milliseconds, in UTC; +- `instance`: `SWWAF_INSTANCE_NAME`; +- `client_ip`, `method`, `scheme`, `host`, `path`, `query`, `protocol`; +- `status`: what the client was sent, `0` if nothing; +- `request_bytes` and `response_bytes`: body bytes each way; +- `referer` and `user_agent`. + +## Request detail + +- `request_id`: the `X-Request-ID` a trusted proxy sent, or a new random one of + 26 letters and digits. It goes to the app in `X-Request-ID`. +- `peer_ip`: the TCP peer, normally traefik. +- `forwarded_for`: the `X-Forwarded-For` header as received. +- `client_group`: the client as rate limiting counts it, an IPv4 address as a + /32 or an IPv6 group. +- `asn`, `as_name` and `country`: the client's, as looked up. Each is empty + when: + - lookups are off; + - the client is not looked up; + - the source cannot place the client; + - GeoJS had not answered by the time the request went on. +- `content_type` and `content_length`: the request's own. +- `request_headers`: the headers listed in `SWWAF_LOG_REQUEST_HEADERS`, by name + in lower case. +- `has_authorization` and `has_cookie`: true when the request had those headers. + Their values are never logged. +- `websocket`: true when the app switched the connection to another protocol. + +## Response detail + +- `upstream_status`: what the app answered. It differs from `status` when + `smallwebwaf` answered itself. +- `response_content_type`, `cache_control` and `location`: the answer's headers. +- `aborted`: true when the client went away early. + +## Decision + +- `action`: what was done: + - `forward`: passed to the app; + - `denied`: refused for `SWWAF_DENY_NETS`, or for a reputation source with + the action `deny`; + - `banned`: refused under a ban, or for a `ban` rule, a trap path or a + CrowdSec decision, each of which bans the client; + - `country_denied`, `rate_limited`, `rule_blocked` and `waf_blocked`: + refused by that check; + - `too_large` and `timed_out`: a size or time limit was passed; + - `upstream_error`: the app could not be reached, or its answer broke off; + - `admin`: answered at one of `smallwebwaf`'s own endpoints. +- `would_action`: in `observe` mode, what `enforce` mode would have done with a + request it would have refused. `action` then names what was done, normally + `forward`. +- `limit_percent` and `limit_percent_setting`: the client's percentage of the + rate limits and the setting that lowered it, when one did. `bytes_percent` and + `bytes_percent_setting` do the same for the byte limits. +- `counts`: the client's requests in the minute, hour and day as rate limiting + counts them, this request included, as `minute`, `hour` and `day`; its bytes + as `minute_bytes`, `hour_bytes` and `day_bytes`. A count can have a fraction, + since each window is counted in two buckets with the earlier one weighted. + `counts` is left out when rate limiting did not count the request. +- `limit_hit`: the limit broken: `minute`, `hour` or `day` for a rate limit; + `minute_bytes`, `hour_bytes` or `day_bytes` for a byte limit; or + `error_burst`. A request whose bytes broke a limit was passed on whole, so its + `action` stays `forward`. +- `rule_ids`: the rule file rules that matched, in order, up to the one that + refused. +- `waf_score` and `waf_rule_ids`: the Core Rule Set's score and the rules that + matched, for a request it inspected. +- `reputation`: the sources the client is on, in this order: + - blocklist URLs; + - the CrowdSec decision list's URL; + - DNS blocklist zones; + - `abuseipdb`. +- `offence`: the kind of offence recorded, when one was. +- `ban_expires`: when the client's ban ends, for a request that made one or was + refused under one: a time, or `permanent`. + +## Timings + +All in milliseconds: + +- `duration_total`: from the request's headers being read to its log line; +- `duration_checks`: to the end of the checks, before forwarding; +- `duration_waf`: the part of that the Core Rule Set took; +- `duration_upstream_connect`: from the hand-over to the app until there was a + connection; +- `duration_upstream_first_byte`: until the first byte of the answer; +- `duration_upstream_total`: until the end of the answer. + +## Every line + +Every line has `type`, the standard web log fields, `request_id`, `peer_ip`, +`client_group`, `asn`, `as_name`, `country`, `action` and `duration_total`. The +process's own messages share the stream with `"type":"process"`; request lines +carry `"type":"request"`. + +## Sending the log to a syslog server + +While `SWWAF_LOG_REMOTE_URL` is set, every line on stdout is also sent to that +server, as the message of an RFC 5424 syslog record. Over UDP each record is one +datagram. Over TCP and TLS each record follows its length in bytes and a space. +stdout is unchanged. The record carries: + +- the facility from `SWWAF_LOG_REMOTE_FACILITY`; +- the severity informational; +- the time; +- the host's name; +- the app name from `SWWAF_LOG_REMOTE_APP_NAME`. + +Lines wait in a buffer of `SWWAF_LOG_REMOTE_BUFFER` lines, so a server that is +slow or down never holds up a request. When the buffer is full, the oldest line +is dropped. A failed connection is tried again a second later, and twice as long +after each further failure, up to a minute. A line too long for one UDP datagram +is dropped alone. As `smallwebwaf` stops, it spends at most two seconds sending +the lines still waiting, and drops the rest. stdout has carried them.