diff --git a/SPEC.md b/SPEC.md index e7c5586..c0f2dbd 100644 --- a/SPEC.md +++ b/SPEC.md @@ -5,8 +5,8 @@ runs inside the app's container, between traefik and the app. This document says what it must do and why, each requirement once. Everything in it is built, except the last section, "Planned, not built". `README.md` says how to run it. `EVALUATION.md` says why no existing tool was adopted. The reference documents -under `docs/`, which this document links to, give the settings, the formats and -the lists; every fact lives in one of these files and is linked from the others. +under `docs/` give the settings, the formats and the lists. This document links +to them. ## Purpose @@ -58,12 +58,12 @@ the lists; every fact lives in one of these files and is linked from the others. - `internal/ratelimit`: the table of clients. It counts requests, bytes and refusals, and keeps each client's history. - `internal/bans`: the bans. It holds each netblock's bans with their notes, - decides how long a new ban lasts, and picks the ban to drop past - `SWWAF_MAX_BANS`. + decides how long a new ban lasts, and picks the ban to drop when the table + of bans is full. - `internal/rules`: reads the rule files, and again as they change, and tells which rules a request matches. - - `internal/waf`: the Core Rule Set with the six changes, as Coraza - directives. + - `internal/waf`: the Core Rule Set, with the six changes listed under + "Attack detection", as Coraza directives. - `internal/lookup`: looks clients up through GeoJS or in the lookup database. - `internal/reputation`: fetches the lists, asks the DNS blocklists and @@ -116,9 +116,9 @@ names the section that holds its rules. around its address, a /64 by default, since one abuser usually controls a whole /64. A client is therefore one IPv4 address or one IPv6 group. 3. The static lists. A client in `SWWAF_ALLOW_NETS` skips steps 4 to 12 and the - byte limits. It is still counted for the anomaly thresholds, and still needs - a token at step 13. A client in `SWWAF_DENY_NETS` is refused; a client in - both lists is let through. + byte limits and the error burst at step 15. It is still counted for the + anomaly thresholds, and still needs a token at step 13. A client in + `SWWAF_DENY_NETS` is refused; a client in both lists is let through. 4. Bans. A client under a ban is refused (see "Bans"). 5. The lookup, unless `SWWAF_LOOKUP_SOURCE` is `off` (see "Lookup of AS number and country"). The request goes on at once. It waits for the first GeoJS @@ -148,15 +148,19 @@ names the section that holds its rules. `detect`. 13. A request under `/_smallwebwaf/` is answered here (see "Admin endpoints"). 14. The request is forwarded to `SWWAF_UPSTREAM_URL`, streaming, with - `X-Forwarded-For` added, and `X-Client-ASN` and `X-Client-Country` when - enabled. + `X-Forwarded-For` added. With `SWWAF_ADD_LOOKUP_HEADERS`, `X-Client-ASN` and + `X-Client-Country` are added for the app's own logs; a header whose value is + unknown is left out. Whatever that setting says, any such header the client + sent is removed. 15. After the response: - The response bytes, the request bytes or both are added to the byte - counters, as `SWWAF_BYTES_COUNT` says. A client whose total passes a byte - limit times the percentage has broken that limit and is banned. The - response in progress is not cut off. - - The client's refusals are counted for the error burst (see "Attack - detection"). + counters, as `SWWAF_BYTES_COUNT` says. For a WebSocket, what it carried + each way counts once it closes. A client whose total passes a byte limit + times the percentage has broken that limit and is banned. The response in + progress is not cut off. + - A refusal by a rule file, a trap path, the Core Rule Set or a token check + is counted for the error burst (see "Attack detection"). No other refusal + is. - The client's history and the anomaly counters are updated, and the alert thresholds are evaluated. @@ -176,9 +180,10 @@ the token check still apply. load. - Memory is bounded. When the table of clients holds `SWWAF_MAX_TRACKED_CLIENTS`, the least recently seen client is dropped first, - with its history. The bans, the GeoJS answers, the reputation verdicts and the - anomaly counters have bounds of their own; `docs/settings.md` gives the fixed - ones under "Fixed limits". + with its history. The bans are bounded by `SWWAF_MAX_BANS` (see "Bans"). The + GeoJS answers, the reputation verdicts and scores, the DNS queries under way + and the anomaly counters have fixed bounds, listed under "Fixed limits" in + `docs/settings.md`; past a bound, the entry used longest ago goes first. ## Bans @@ -187,7 +192,8 @@ Until a ban ends, every request from its netblock is refused with `SWWAF_BAN_SCOPE_V4_PREFIX`, or its IPv6 group. At the defaults a ban covers one address or one /64, so a permanent ban does not reach neighbours who did nothing. Requests refused under a ban are not counted against limits. They are -counted in the client's history and in the ban's notes. +counted in the client's history and in the ban's notes. A banned client is not +looked up. An offence is a request `smallwebwaf` holds against the client. It is one of: @@ -203,7 +209,7 @@ to a ban, each by its own rule: or asks for a path in `SWWAF_TRAP_PATHS`. Examples are a probe for a `.env` file or a `.git` directory, and a known scanner's user agent. - The first one bans the netblock for `SWWAF_ATTACK_BAN_DURATION` (default - `7d`). + `7d`), with the cause `attack`. - Any further request from the netblock while that ban lasts shows it is malicious: the ban becomes permanent. - Once that ban has run out, the netblock is served like any other. Its next @@ -211,9 +217,10 @@ to a ban, each by its own rule: - A broken limit: a request over a rate limit, a response that takes the client's byte total past a byte limit, or more than `SWWAF_ERROR_BURST_THRESHOLD` refusals within a minute. - - The first such ban lasts `SWWAF_LIMIT_BAN_DURATION` (default `1h`). So - does one that comes more than `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default - `24h`) after the last such ban ended. + - The first such ban lasts `SWWAF_LIMIT_BAN_DURATION` (default `1h`), with + the cause `limit`. So does one that comes more than + `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`) after the last such ban + ended. - Breaking a limit again within that window makes the new ban three times as long as the last one: one hour, then 3, 9, 27 and 81 hours. - A ban that would be longer than `SWWAF_MAX_BAN_DURATION` (default `7d`) is @@ -222,15 +229,18 @@ to a ban, each by its own rule: Once it ends, only new requests can break a limit again. The client's history keeps its totals. -A Core Rule Set match refuses only the request it matched, without a ban, -because the Core Rule Set has false positives. A `block` rule does the same. -Those refusals count toward the error burst, so a client that keeps setting them -off is banned under the second rule. +A Core Rule Set match refuses only the request it matched, with `403` and +without a ban, because the Core Rule Set has false positives. A `block` rule +does the same. Those refusals count toward the error burst, so a client that +keeps setting them off is banned under the second rule. A client on the CrowdSec decision list is banned until the decision ends, with the cause `crowdsec`. Only a client that sends a request is banned, so a long list does not fill `bans.json`. Such a ban is never made permanent. It does not -make the netblock's next ban for a broken limit longer. +make the netblock's next ban for a broken limit longer. A ban lifted while the +decision is in force is followed by another at the client's next request; the +decision is deleted in CrowdSec first, and the ban lifted once the list has been +fetched again. These refuse each request they cover, record nothing in `bans.json`, and count no offence: @@ -240,31 +250,34 @@ no offence: - the country lists. Every ban carries notes with what an admin needs to decide whether to lift it, -drawn from what `smallwebwaf` already knows; nothing is looked up to fill them. -`docs/bans.md` lists the notes and says how a ban is made, kept or lifted, by -editing `bans.json` or through the ban endpoints. A ban lifted before it ends is -kept, marked as lifted, and does not count toward a longer ban. Deleting its -entry from `bans.json` forgets it entirely. +drawn from what `smallwebwaf` already knows. No lookup is made for the notes +alone. `docs/state-files.md` lists the notes with the rest of a ban's entry. + +An admin adds, keeps or lifts a ban by editing `bans.json` (see "Persistent +state") or through the ban endpoints (see "Admin endpoints"). A ban an admin +makes has the cause `admin`; an admin who wants to keep a ban `smallwebwaf` made +sets its cause to `admin`. A ban lifted before it ends is kept, marked as +lifted, and does not count toward a longer ban; it refuses nothing. Deleting its +entry from `bans.json` forgets it entirely. Only a ban whose cause is `attack` +becomes permanent at the first request it refuses. An admin whose own address is +banned lifts the ban by editing `bans.json`. A permanent ban does not run out. It ends when an admin lifts it, or when `smallwebwaf` drops it from a full table, as below. When `smallwebwaf` holds `SWWAF_MAX_BANS` bans of its own, it drops the earliest ban of one netblock: the netblock that has gone longest without a request. A past, active or permanent ban is dropped alike. A scanner whose ban was dropped that way is refused and -banned again by its next probe. A ban an admin made, whose cause is `admin`, is -never dropped and does not count toward `SWWAF_MAX_BANS`, so such bans can never -fill the table. An admin who wants to keep a ban `smallwebwaf` made sets its -cause to `admin`. +banned again by its next probe. A ban whose cause is `admin` is never dropped +and does not count toward `SWWAF_MAX_BANS`, so such bans can never fill the +table. ## Settings +`docs/settings.md` lists every setting with its default and the values it takes, +in the groups below. It also gives the forms of values: lists, netblocks, +durations, sizes, countries, AS numbers, percentages and path prefixes. Conventions: -- Lists are comma separated. Netblocks are CIDR; a bare address means /32 or - /128. Durations use Go's syntax plus `d` for days (`90s`, `15m`, `24h`, `7d`). - Sizes take `K`, `M` and `G`, powers of 1024. Countries are two-letter ISO - codes in either case. A code that is not a country, such as `nk`, stops the - start with a message naming it. - These settings take `off`, which switches them off: - `SWWAF_LOOKUP_SOURCE` and `SWWAF_WAF_MODE`; - the five timeouts under "Size and time limits"; @@ -275,21 +288,24 @@ Conventions: - `SWWAF_ALERT_COOLDOWN` and `SWWAF_ALERT_MAX_PER_HOUR`; - every anomaly threshold. - Every other setting refuses `off`. + No other setting can be switched off with `off`. - A list set to an empty value is an empty list, and replaces the default. - Any setting can be read from a file instead, for secrets and long lists: - `SWWAF_ADMIN_TOKEN_FILE` names the file that holds `SWWAF_ADMIN_TOKEN`. - `SWWAF_LOG_REMOTE_TLS_CA_FILE` already holds a file name, so it has no `_FILE` - form. + `SWWAF_ADMIN_TOKEN_FILE` names the file that holds `SWWAF_ADMIN_TOKEN`. The + file's contents are the value, less one newline at the end, and are checked as + the value itself would be. Setting both forms stops the start, and so does + naming a file that cannot be read. `SWWAF_LOG_REMOTE_TLS_CA_FILE` already + holds a file name, so it has no `_FILE` form. - Settings, files included, are read once at start. Changing one means restarting the container. The files `smallwebwaf` watches while it runs are its state files, its rule files and the lookup database. -- An invalid setting stops the process at start with a message naming the - variable. At start the settings in effect are logged, with secrets masked. - -`docs/settings.md` lists every setting with its default and the values it takes, -in the groups below. What each group must do, and why: +- An invalid setting stops the process at start with one log line, marked + `"type":"process"` with `"msg":"invalid setting"`, naming the variable and + what is wrong with the value. runit then starts the process again every second + until the value is fixed (see "The two processes"). At start the settings in + effect are logged on one line; tokens, keys and the paths of webhooks, which + often carry a secret, are shown as `********`. ### Core @@ -304,16 +320,19 @@ in the groups below. What each group must do, and why: the private address ranges, so a client that connects directly cannot claim another address. Where other containers reach `smallwebwaf` directly, or visitors on a private network reach traefik, they can name another address in - the header; the setting, narrowed to traefik's own network, closes that gap. -- `SWWAF_IPV6_GROUP_PREFIX`, `SWWAF_MAX_TRACKED_CLIENTS` and `SWWAF_MAX_BANS` - set the group size and the bounds under "Counting method" and "Bans". + the header. Narrowing the setting to traefik's own network closes that gap. +- `SWWAF_IPV6_GROUP_PREFIX` sets the group size. After a change, each IPv6 + client starts afresh; existing bans keep refusing their netblocks until they + end. `SWWAF_MAX_TRACKED_CLIENTS` and `SWWAF_MAX_BANS` set the bounds under + "Counting method" and "Bans". ### Tokens - `SWWAF_ADMIN_TOKEN` switches on the ban endpoints and - `/_smallwebwaf/clients/`; `SWWAF_METRICS_TOKEN` switches on the metrics, - which would otherwise be open to anyone on the internet. The two are separate, - so a scraper that holds the metrics token cannot manage bans. + `/_smallwebwaf/clients/`. The metrics are off until `SWWAF_METRICS_TOKEN` + is set, since without a token they would be open to anyone on the internet. + The two are separate, so a scraper that holds the metrics token cannot manage + bans. - A token is a long random value, since the endpoints can be reached from the internet. One shorter than 32 characters stops the start. The app starts with the same environment variables as `smallwebwaf`, so it can read a token set as @@ -322,36 +341,36 @@ in the groups below. What each group must do, and why: ### State files -- The state files live in `SWWAF_STATE_DIR`, a directory of their own beside the - app's data, belonging to the `smallwebwaf` user. +- `SWWAF_STATE_DIR` names the directory of the state files (see "State" under + "Deployment"). - `bans.json` is written `SWWAF_STATE_WRITE_DELAY` after a ban is made, lifted or made permanent, so a burst of changes becomes one write. Every file is written every `SWWAF_STATE_COUNTER_INTERVAL`, and at an orderly stop. ### Logging -- The request log on stdout is always on and has no switch. The process's own - messages are JSON lines on stdout too, marked `"type":"process"`; - `SWWAF_LOG_LEVEL` filters only those. +- The request log on stdout is always on and has no switch (see "Request log"). + The process's own messages share the stream; `SWWAF_LOG_LEVEL` filters only + those, and no message is at `debug`. `warn` and `error` hold back the line of + settings at start. - `SWWAF_LOG_REQUEST_HEADERS` names the request headers to record. - `Authorization`, `Cookie` and `Set-Cookie` values are never logged, only - whether they were present. + `Authorization`, `Cookie` and `Set-Cookie` values are never logged, even if + listed, only whether they were present. - `SWWAF_LOG_REMOTE_URL` sends every log line to a syslog server as well, over - UDP, TCP or TLS. Lines wait in a buffer of `SWWAF_LOG_REMOTE_BUFFER` lines; - when it is full, the oldest are dropped and counted. + UDP, TCP or TLS (see "Request log"). ### Metrics -- `SWWAF_METRICS_TOP_N` bounds the series by AS number and by country: that many - of the busiest each, the rest summed as `other`, so the label set stays +- `SWWAF_METRICS_TOP_N` is how many of the busiest AS numbers and countries get + a series of their own. The rest are summed as `other`, so the label set stays bounded. ### Static lists -- `SWWAF_ALLOW_NETS` bypasses everything (monitoring, the owner's own networks). - `SWWAF_RATE_LIMIT_EXEMPT_NETS` bypasses the request and byte limits only; the - error burst, attack detection and bans still apply. `SWWAF_DENY_NETS` is - always refused. +- `SWWAF_ALLOW_NETS` skips the checks step 3 names, for monitoring or the + owner's own networks. `SWWAF_RATE_LIMIT_EXEMPT_NETS` bypasses the request and + byte limits only; the error burst, attack detection and bans still apply. + `SWWAF_DENY_NETS` is always refused. - Many clients share one address (mobile carriers, offices, Tor), and so share its limits and its bans. The defaults sit well above one person's use, and the first two lists take known shared addresses out. @@ -361,16 +380,24 @@ in the groups below. What each group must do, and why: Breaking one bans the client, so the defaults sit several times above what one busy person produces. A browser loading a heavy page makes a few hundred requests, a git clone a handful, and several people often share one address. A -machine that talks to the app all day, such as a gitea Actions runner, can still -pass the day limit. Its address belongs in `SWWAF_RATE_LIMIT_EXEMPT_NETS`. +machine that talks to the app all day can still pass the day limit. Its address +belongs in `SWWAF_RATE_LIMIT_EXEMPT_NETS`. + `SWWAF_RATE_LIMIT_EXEMPT_PATHS` takes path prefixes out of the count, such as -static assets or health checks. +static assets or health checks; their bytes are not counted either. A prefix is +matched exactly as sent, without the query, and is not percent-decoded: +`/assets/` matches `/assets/app.js`, and does not match `/Assets/app.js`, +`/%61ssets/app.js` or `/static/assets/app.js`. There are no wildcards. A path +holding `..`, a backslash or an encoded slash (`%2F`) is never exempt, since the +app may read it as a path outside the prefix. `SWWAF_WAF_EXEMPT_PATHS` is +matched the same way. ### Byte limits, per client -A response's bytes are counted when it ends, so every default sits above the -largest response allowed, and no single download breaks one. `SWWAF_BYTES_COUNT` -says whether request bytes, response bytes or both are counted. +A response's bytes are counted when it ends, so a byte limit never cuts an +answer short; the client's next request is refused instead. Every default sits +above the largest request and answer together, so at the defaults no single +download breaks a limit. ### Size and time limits @@ -380,7 +407,8 @@ and the app, since a slow client and a slow app are separate problems. There is one size limit for request bodies and one for response bodies. Bodies pass through unchanged, so a limit on each side would bound the same bytes, and the lower one would always decide. Bodies stream straight through, so a request body -reaches the app while the client is still sending it. +reaches the app while the client is still sending it, except the part the Core +Rule Set reads while `SWWAF_WAF_BODY_LIMIT` is set. - `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES` must be more than `4K` and cannot be `off`. Go's HTTP server always has such a limit, and reads 4 KiB past the one @@ -402,12 +430,12 @@ What the client gets when a limit is passed: started, is answered by `smallwebwaf`: - `413` for a request body that is too large; - `408` for a client too slow to send its body; - - `502` for a response that is too large; + - `502` for a response that is too large, or an app that cannot be reached; - `504` for an app that is too slow. - A request body streams through, so each side can hold up the other. A request - timeout that runs out while the body is still on its way answers `408` if - `smallwebwaf` was waiting for the client at that moment, and `504` if it was - waiting for the app. + timeout can run out while the body is on its way. The answer is then `408` if + `smallwebwaf` was waiting for the client, and `504` if it was waiting for the + app. - Once the response has started it can only be cut off, and the connection is closed. - A WebSocket connection leaves these limits behind once it is upgraded. It @@ -416,41 +444,61 @@ What the client gets when a limit is passed: ### Lookup of AS number and country On by default through GeoJS, which needs no account or file, so that country -lookup works with no setup. A local database file, the free IPinfo Lite -database, is the alternative for a service that keeps its visitors' addresses on -its own host; the operator downloads and mounts it, and `smallwebwaf` never -fetches it and holds no account or token for it. `docs/lookup.md` says what each -source sends where, how long answers are kept, and when a request waits. +lookup works with no setup. A local database file is the alternative for a +service that keeps its visitors' addresses on its own host. - Only one source is used. The start stops, naming both settings, when `file` is set without `SWWAF_LOOKUP_DB_PATH`, or `SWWAF_LOOKUP_DB_PATH` with any other - source. The start also stops when the source is `off` and a setting needs - lookups: a country list, a biased threshold, `SWWAF_ADD_LOOKUP_HEADERS` or an - anomaly threshold per AS number. + source. The start also stops when the source is `off` and one of these needs + lookups: + - a country list; + - a biased threshold; + - `SWWAF_ADD_LOOKUP_HEADERS`; + - an anomaly threshold per AS number. - GeoJS is told the address of every new visitor, whether or not a setting uses the answer, since the request log, the metrics and the ban notes carry it. - Private, loopback and link-local addresses, which no source can place, are - never sent. Addresses are asked about in batches, one request at a time, and - answers are kept for days, which keeps the number of requests to GeoJS low. - Answers are kept apart from the table of clients, so that an answer outlasts a - client dropped from that table. + These addresses are never sent: + - one in `SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS`; + - one under a ban; + - a private, loopback or link-local address, which no source can place. +- For an IPv6 visitor, GeoJS is asked about the first address of its group. + Addresses are asked about together, up to 200 in one request + (`https://get.geojs.io/v1/ip/geo.json?ip=a,b,c`), one request at a time, which + keeps the number of requests to GeoJS low. Each answer's `country_code`, `asn` + and `organization_name` are read. An answer without a `country_code`, or with + the `asn` `64512`, counts as unknown. +- Each answer is kept for seven days, in memory and in `lookups.json`. Answers + are kept apart from the table of clients, so that an answer outlasts a client + dropped from that table. The answer reaches the client's history and ban notes + when it comes. +- `SWWAF_LOOKUP_TIMEOUT` is how long a visitor's request waits for its client's + first answer, when a setting needs it (see step 5). It also bounds each + request to GeoJS. A client whose answer has not come in time counts as unknown + until it comes, and its later requests do not wait. Its address is asked about + again in the background. - A client the lookup cannot place has an unknown country and AS number. The - exclusive country list refuses it, and the biased thresholds give it - `SWWAF_UNKNOWN_LIMIT_PERCENT`. Nothing else treats it differently. -- `SWWAF_LOOKUP_TIMEOUT` bounds both how long a visitor's request waits for its - client's first answer, when a setting needs it, and how long a request to - GeoJS may take. A client whose answer has not come in time counts as unknown - until it comes, and its later requests do not wait. + biased thresholds give it `SWWAF_UNKNOWN_LIMIT_PERCENT`, and the exclusive + country list refuses it (see "Country lists"). Nothing else treats it + differently. - GeoJS publishes no rate limit, but its terms forbid "an excessive amount of API requests", judged by GeoJS alone, and let it block a caller. While GeoJS - is slow, down or refusing `smallwebwaf`, new clients count as unknown, - `smallwebwaf` keeps asking with backoff, and one `source_failure` alert is - sent per cooldown. A service that cannot accept this, or that every new - visitor's address goes to a third party, uses the file. -- A replaced database file is read again; one that cannot be read is logged, - sent as one `file_error` alert, and ignored, the file in use staying in use. - The directory that holds the file is mounted, not the file itself (see "Rule - files" for why). + is slow, down or refusing `smallwebwaf`, new clients count as unknown. GeoJS + is then left alone for a second, and twice as long after each further failure, + up to five minutes, and one `source_failure` alert is sent per cooldown. A + service that cannot accept this uses the file. +- The file is the free IPinfo Lite database (`ipinfo_lite.mmdb`), whose `asn`, + `as_name` and `country_code` fields are read. The operator downloads it with a + free IPinfo account and mounts it read-only; `smallwebwaf` never fetches it + and holds no account or token for it. IPinfo updates it daily, and releases it + under the Creative Commons Attribution-ShareAlike 4.0 licence, asking for a + link to IPinfo on the site that uses it, such as "IP address data is powered + by IPinfo". +- The file is read whole at start; one that is missing or unreadable stops the + start. It is read again once it has gone 2 seconds without a change after + being replaced. One that cannot be read is logged, sent as one `file_error` + alert and ignored. The database already loaded stays in use. The directory + that holds the file is mounted, not the file itself (see "Rule files" for + why). A client whose address the file does not hold counts as unknown. ### Country lists @@ -460,12 +508,12 @@ where it comes from. Clients in `SWWAF_ALLOW_NETS` are not checked. - `SWWAF_DENIED_COUNTRIES` refuses every request from a listed country. - `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` lets through only clients placed in a listed country. Every request from any other country is refused. So is every - request from a client the lookup cannot place, such as internal monitoring; - those that should reach the app go in `SWWAF_ALLOW_NETS`. A list that let - unplaced clients through would let every new client in whenever GeoJS stops - answering. The cost: while GeoJS fails, this list refuses every new client. No - other failure of GeoJS, a reputation source or an alert destination comes this - close to taking the service down. + request from a client the lookup cannot place, such as internal monitoring or + a visitor on a private address; those that should reach the app go in + `SWWAF_ALLOW_NETS`. A list that let unplaced clients through would let every + new client in whenever GeoJS stops answering. The cost: while GeoJS fails, + this list refuses every new client. No other failure of GeoJS, a reputation + source or an alert destination comes this close to taking the service down. - Both may be set. `SWWAF_DENIED_COUNTRIES` then adds nothing, since the exclusive list already refuses every other country. A code on both lists stops the start with a message naming it. @@ -481,17 +529,25 @@ request log and the metrics show which AS numbers and countries a service's abuse comes from. - Clients in a listed AS number or country get that percentage of every request - and byte limit (`SWWAF_ASN_LIMIT_PERCENT`, `SWWAF_COUNTRY_LIMIT_PERCENT`), or - of the byte limits alone (`SWWAF_ASN_BYTES_PERCENT`, - `SWWAF_COUNTRY_BYTES_PERCENT`). `0` is a zero allowance: the client's first + and byte limit (`SWWAF_ASN_LIMIT_PERCENT`, `SWWAF_COUNTRY_LIMIT_PERCENT`). A + lowered limit is rounded down. `0` is a zero allowance: the client's first request breaks a limit and bans it. +- `SWWAF_ASN_BYTES_PERCENT` and `SWWAF_COUNTRY_BYTES_PERCENT` set the byte + limits' percentage alone, when it should differ from the request limits'. For + an AS number they list, they replace its percentage from + `SWWAF_ASN_LIMIT_PERCENT` and `SWWAF_ASN_LIMIT_PERCENT_URL`; for a country, + the one from `SWWAF_COUNTRY_LIMIT_PERCENT`. - `SWWAF_ASN_LIMIT_PERCENT_URL` names a text file of `AS:percent` lines, so one abuse-source list can be shared by every `smallwebwaf` in the fleet. It is - fetched and refreshed like the blocklists. + fetched and kept like a blocklist (see "Reputation"). Where it and + `SWWAF_ASN_LIMIT_PERCENT` both give a percentage, the lower applies. - A client's limit percentage is the lowest of those that apply to it: its AS - number's, its country's, or a reputation hit's `limit:`. It is 100 if - none applies. Clients given a lower percentage are banned after fewer - requests, first for a while and then, for repeated abuse, permanently. + number's, its country's, or a reputation hit's `limit:`. A client the + lookup cannot place gets `SWWAF_UNKNOWN_LIMIT_PERCENT` in place of an AS + number's or a country's. The percentage is 100 if none applies. Clients given + a lower percentage are banned after fewer requests, first for a while and + then, for repeated abuse, permanently. The log line of each counted request, + and the notes of each ban, say which setting lowered the limit. - No budget is shared by a whole country or AS number. One abuser could use it up and lock out everyone else there: a denial of service nobody chose, and the one this tool exists to prevent. Refusing a whole country is left to the @@ -506,15 +562,23 @@ scanners' user agents. It reads no request body. On a code forge, the example deployment, bodies carry what people write and publish, such as issue text and files saved in the web editor. The Core Rule Set takes the shell commands and code in them for attacks. The default gives up refusing an attack carried in a -body; the rule files, the limits and the bans still apply to the client that -sends it, and `SWWAF_WAF_BODY_LIMIT` switches body inspection on. -`docs/core-rule-set.md` says what the Core Rule Set reads and how it scores a -request. +body. The rule files, the limits and the bans still apply to the client that +sends it. +- The Core Rule Set inspects only a request that every earlier check let + through. It skips a path under `SWWAF_WAF_EXEMPT_PATHS` and a client in + `SWWAF_ALLOW_NETS`. An engine error on a request forwards the request, and is + logged and counted. - `SWWAF_WAF_PARANOIA_LEVEL` and `SWWAF_WAF_ANOMALY_THRESHOLD` are the Core Rule - Set's own two tuning values, at its own defaults. A request whose score - reaches the threshold is a match. With the threshold `off`, nothing is a - match, and scores are still logged. + Set's own two tuning values, at its own defaults. Each level up runs more + rules, which find more attacks and take more ordinary requests for attacks. + Each rule that matches adds its score to the request's, and a rule rated + critical adds 5. A request whose score reaches the threshold is a match, so by + default one critical rule is enough. With the threshold `off`, nothing is a + match, and scores are still logged. A request with more than 1000 query + parameters or body fields adds 5 to its score, since Coraza reads only the + first 1000. Either way the log line gives the rule ids and the score, and a + `waf_block` alert is sent. - `SWWAF_WAF_DISABLED_RULES` switches off rule ids when an app trips a false positive. The default switches off six rules: - 920340, 920420 and 920640 refuse a request for the type of its body, when @@ -531,25 +595,29 @@ request. default, so name these six ids in it to keep them off. - `SWWAF_WAF_BODY_LIMIT` switches body inspection on, up to that size. Form data - and multipart bodies are read up to the size, the rest streaming on without - being held in memory; JSON and XML bodies are read only when no larger than + and multipart bodies are read up to the size, and the rest streams on without + being held in memory. JSON and XML bodies are read only when no larger than the size, since those cannot be read in part. Any other body is not read: the Core Rule Set would read it as form data, where binary content such as a git - push trips rules written for text. Body inspection suits apps whose forms - carry no code. `docs/gitea.md` records what it refuses in front of gitea. + push trips rules written for text. A body the Core Rule Set cannot parse adds + 5 to the score, and so does a multipart body cut by the limit in the middle of + a part's header. The client has until `SWWAF_CLIENT_REQUEST_TIMEOUT` runs out + to send the part that is read. Body inspection suits apps whose forms carry no + code; `docs/gitea.md` records what it refuses in front of gitea. - `SWWAF_TRAP_PATHS` names paths the app never serves and only scanners ask for, such as `/wp-login.php` in front of gitea. A request for one is a clear sign of attack. This is the short form of a `path` rule with the `ban` action, for - deployments that mount no rule files. -- `SWWAF_ERROR_BURST_THRESHOLD` bounds the refusals per client per minute after - a rule file match, a trap path, a Core Rule Set match, or a missing or wrong - token. More than this breaks a limit (see "Bans"). A client trying one attack - or token after another is refused many times a minute; a person rarely more - than a few times. Answers the app gives are not counted, since they do not - tell a scanner from an ordinary client. In front of gitea, registry and git - clients get `404` and `401` by design, hundreds of times a minute. -- A Core Rule Set engine error on a request forwards the request, and is logged - and counted. + deployments that mount no rule files. A trap path is matched whole, without + the query, against the path as sent and once percent-decoded: `/wp-login.php` + matches `/wp-login.php?x=1` and `/%77p-login.php`, and does not match + `/wp-login.php/` or `/blog/wp-login.php`. +- `SWWAF_ERROR_BURST_THRESHOLD` counts a client's refusals per minute for a rule + file match, a trap path, a Core Rule Set match, or a missing or wrong token. + More than this breaks a limit (see "Bans"). A client trying one attack or + token after another is refused many times a minute; a person rarely more than + a few times. Answers the app gives are not counted, since they do not tell a + scanner from an ordinary client. In front of gitea, registry and git clients + get `404` and `401` by design, hundreds of times a minute. `smallwebwaf` changes the Core Rule Set 4.25.0 in six ways, since in front of gitea it would otherwise refuse ordinary requests. The changes hold in front of @@ -609,7 +677,7 @@ every app, and no setting undoes them: 6. Responses are not inspected. A raw file from a repository, such as a shell script, looks to the response rules like source code leaking from the server. -### Ban response and scope +### Ban lengths, response and scope `SWWAF_BAN_RESPONSE` is `403` by default, which makes a ban easy to recognise when debugging; `429` and `close` are the alternatives. Behind traefik, `close` @@ -625,11 +693,13 @@ account or an engine of the operator's own. The list best suited to be on by default, Spamhaus DROP, must not be fetched more than once an hour, and Spamhaus may block an address that does. Each `smallwebwaf` fetches its own copy, so several apps on one host, sharing one address, would often fetch less than an -hour apart. `docs/reputation.md` says how each source is asked and what is done -with the answer. +hour apart. - Blocklists (`SWWAF_BLOCKLIST_URLS`) are text files of addresses and netblocks, - such as the Spamhaus DROP list. Spamhaus's terms for it, on its DROP page: + one to a line, as the Spamhaus DROP list is written. Anything after a `;` or + `#` on a line is ignored. An IPv4-mapped address such as + `::ffff:192.0.2.0/120` is read as the IPv4 netblock it stands for. Spamhaus's + terms for DROP, on its DROP page: - automated downloads must be at least one hour apart, and once a day is more than enough in most cases; - a product that uses it must credit The Spamhaus Project and keep the @@ -638,95 +708,174 @@ with the answer. So `SWWAF_BLOCKLIST_REFRESH` is never less than `1h`, and the last good copy is kept whole, comment lines included, on failure and across restarts. +- Each list is fetched again `SWWAF_BLOCKLIST_REFRESH` after its last try, one + list at a time. `reputation.json` keeps when each was last tried, so a restart + does not fetch a list again before its time. A fetch fails when: + - the server answers other than `200`; + - the fetch does not finish within a minute; + - the list is longer than 16 MiB; + - a line is not an address or netblock, or, for the AS number file, not + `AS:`. + + The log, the alerts, the ban notes and the metrics name each list by its + URL, so a secret stays out of it. + - `SWWAF_BLOCKLIST_ACTION` is `deny` by default, which refuses every request from a listed address and suits lists of networks that send nothing legitimate, such as DROP. `limit:` gives listed clients that percentage of every limit, and suits a list of addresses shared with ordinary - visitors, such as Tor exits. `log` only notes the hit. -- DNS blocklists (`SWWAF_DNSBL_ZONES`) meant for mail list residential ranges - and would block ordinary visitors, so they should not be used. Spamhaus zones - need their keyed query service, given as a zone name containing the key. - `SWWAF_DNSBL_RESOLVER` names a resolver, since public resolvers are refused by - several list operators. -- AbuseIPDB is asked only about clients that have already committed one offence, - so the daily budget (`SWWAF_ABUSEIPDB_DAILY_BUDGET`, under the free tier's - 1000 checks a day) is spent on suspects. + visitors, such as Tor exits. `log` only notes the hit in the log line and + raises an alert. - A CrowdSec engine on the host (`SWWAF_CROWDSEC_LAPI_URL`) has its decision - list pulled on a schedule. It is kept like a fetched blocklist, and a client - on it is banned (see "Bans"). This is how a fleet-wide blocklist can arrive - without `smallwebwaf` depending on CrowdSec. + list fetched every minute, with `SWWAF_CROWDSEC_LAPI_KEY` in the `X-Api-Key` + header. The list is kept like a fetched blocklist, and a client on it is + banned (see "Bans"). Only decisions of the type `ban` on an address or a + netblock are used; any other, such as a captcha or a country, is left out. A + decision lasts for the `duration` it had left when the engine answered. A + fetch fails on an answer other than `200`, a redirect included, on one not + finished within a minute, on one longer than 16 MiB, and on one that is not a + list of decisions. This is how a fleet-wide blocklist can arrive without + `smallwebwaf` depending on CrowdSec. +- Each DNS blocklist (`SWWAF_DNSBL_ZONES`) is asked about a client's own address + in the background, the first time the client is seen; the request goes on at + once, and so does every request from the client until the zone has answered. + The name asked about is the one RFC 5782 gives: the numbers of an IPv4 address + reversed, or the 32 hex digits of an IPv6 address reversed, so `192.0.2.99` is + asked about as `99.2.0.192.dnsbl.dronebl.org`. An answer in `127.0.0.0/8` + lists the client, but one in `127.255.255.0/24` is Spamhaus refusing the + query. A name that does not exist does not list the client. These are + failures, after which the zone is not asked again for a minute: + - an answer in `127.255.255.0/24`; + - an answer outside `127.0.0.0/8`; + - a resolver error; + - no answer within `SWWAF_REPUTATION_TIMEOUT`. + + DNS blocklists made for mail servers list residential ranges, and would + block ordinary visitors, so they should not be used. Spamhaus zones need + their keyed query service, given as a zone name containing the key, which + `smallwebwaf` shows as `********` wherever it names the zone; only + `reputation.json` keeps the zone with its key. `SWWAF_DNSBL_RESOLVER` names + a resolver, since public resolvers are refused by several list operators. + +- AbuseIPDB (`SWWAF_ABUSEIPDB_KEY`) is asked, at its check endpoint, for the + abuse confidence score of a client, from 0 to 100. Only a client that has + already committed one offence is checked, so the daily budget + (`SWWAF_ABUSEIPDB_DAILY_BUDGET`, under the free tier's 1000 checks a day) is + spent on suspects. The check runs in the background at the client's next + request that no ban refuses, so no request waits; the score applies to the + client's later requests. For an IPv6 client, AbuseIPDB is asked about the + address of the request that set off the check, and the score is used for the + whole group. A score at or over `SWWAF_ABUSEIPDB_MIN_SCORE` is a hit. The + budget is counted in UTC days and kept in `reputation.json`, so a restart does + not make it whole again; the check that spends the last of it is logged and + alerted, and no client is checked until the day ends. A check fails on an + answer other than `200`, on one without a score, and on no answer within + `SWWAF_REPUTATION_TIMEOUT`; after a failure no client is checked for a minute. + The key is sent in the `Key` header and nowhere else, and is never logged. - `SWWAF_REPUTATION_ACTION`, for DNS blocklist and AbuseIPDB hits, is `limit:25` by default. Such verdicts are less certain than a blocklist, so a listed client gets a quarter of every limit rather than a refusal. -- Verdicts and scores are kept for `SWWAF_REPUTATION_CACHE_TTL`, with a fixed - bound on their number. A source that is down or over quota gives no verdict. - The service continues, with one `source_failure` alert per cooldown. +- Verdicts and scores are kept for `SWWAF_REPUTATION_CACHE_TTL`, in memory and + in `reputation.json`, with the fixed bounds under "Fixed limits" in + `docs/settings.md`. A source that is down or over quota gives no verdict. The + service continues, with one `source_failure` alert per cooldown. ### Alerting -- Alerts go to a webhook as JSON (`docs/alerts.md` gives the object), and to - Slack and ntfy as formatted messages. `SWWAF_ALERT_EVENTS` chooses the event - types sent. -- `SWWAF_ALERT_COOLDOWN`: the same event is not repeated within this time for - the same netblock, source or file. For an anomaly, it is the same thing - counted that must match: one client, netblock, AS number or named netblock, or - the whole service. A count of suppressed repeats is included in the next one. -- `SWWAF_ALERT_MAX_PER_HOUR`: beyond this, alerts are rolled into one summary - per hour, so a wide attack cannot flood the channel. -- A destination that is down is retried with backoff from a bounded queue, the - oldest alert dropped first, and the drops counted in the metrics. +- Alerts go to a webhook as JSON, and to Slack and ntfy as formatted messages. + `docs/alerts.md` lists the events, the JSON object and the messages. + `SWWAF_ALERT_EVENTS` chooses which of the seven event types are sent; the + hourly `summary` is sent whatever it says. The bans an admin makes raise no + alert. In `observe` mode, a request that would have made a ban still raises + the ban's alert, with `mode` set to `observe` in its `detail`. +- A repeat is an alert with the same event, and the same netblock, source, file + or anomaly scope, as one sent less than `SWWAF_ALERT_COOLDOWN` before. An + anomaly scope is the thing counted: one client, netblock, AS number or named + netblock, or the whole service. A repeat is held back and counted, and the + next such alert sent gives the count. +- Past `SWWAF_ALERT_MAX_PER_HOUR` alerts in an hour, the rest are held back and + counted by event, so a wide attack cannot flood the channel. Once the hour + ends, one alert with the event `summary` gives those counts. It also counts + the repeats that a cooldown held back and that no later alert reported. +- Each destination has a queue of at most 1000 alerts, sent one at a time, so a + slow destination holds up nothing; when it is full, the oldest alert is + dropped, and the drops are counted in the metrics. A `2xx` answer takes an + alert. A `4xx` other than `408` and `429` refuses it, and it is dropped. + Anything else is a failure: the alert is sent again a second later, and twice + as long after each further failure, up to a minute. `alerts.json` keeps the + queues and cooldowns across a restart. ### Anomaly thresholds These only alert; nothing is blocked. None is set by default, and a threshold left unset sends no alert. An alert needs a destination only the operator can supply, and what counts as unusual depends on each service's normal traffic, -which the metrics show. They are what reveals a widely distributed scraper, -thousands of addresses at low rates each, which the per-client limits do not -see; a low `SWWAF_ASN_LIMIT_PERCENT` for its AS number is the response. +which the metrics show. They reveal a widely distributed scraper, thousands of +addresses at low rates each, which the per-client limits do not see. A low +`SWWAF_ASN_LIMIT_PERCENT` for its AS number is the response. - Requests and bytes are counted per minute and per hour, for each client, each surrounding netblock, each AS number, the whole service, and each named netblock in `SWWAF_WATCH_NETS`. -- Anomaly counting includes clients in `SWWAF_ALLOW_NETS` and - `SWWAF_RATE_LIMIT_EXEMPT_NETS`, since an exempt client misbehaving is worth - knowing about. At most 20,000 counters are kept, in memory and in - `alerts.json`. +- Every request but the health check is counted, whatever was done with it, + clients in `SWWAF_ALLOW_NETS` and `SWWAF_RATE_LIMIT_EXEMPT_NETS` included, + since an exempt client misbehaving is worth knowing about. The counters are + kept in memory and in `alerts.json`, with a fixed bound. ## Rule files `smallwebwaf` reads every `*.rules` file in `SWWAF_RULES_DIR` and checks each request against them, unless the client is in `SWWAF_ALLOW_NETS`. -`docs/rules.md` gives the format, the targets and actions, the errors, and the -default file. +`docs/rules.md` gives the format, the targets and actions, and the default file. - The files are plain text meant to be edited by hand, so that a new scanning pattern seen in the request log becomes a rule in one line. `smallwebwaf` watches the directory: a file edited, added or removed there takes effect - while it runs. + while it runs. It reads the files again once the directory has had no change + for 2 seconds, so that a file still being written is read whole. A file whose + name starts with `.`, such as an editor's lock file, is not a rule file. +- Files are read in name order (`00-default.rules` before `50-gitea.rules`), + rules in line order. Matching stops at the first rule that refuses. - A rule's regex is Go's RE2 syntax. It has no backreferences or lookaround; in exchange matching time is linear in the input, so no rule can be made to stall the proxy. - `path` and `uri` are matched both as received and once percent-decoded, so an - encoded probe cannot slip past. Request bodies are not available to rule - files; reading them is the Core Rule Set's job. -- `block` refuses the request without a ban; `ban` makes the request a clear - sign of attack (see "Bans"). `ban` is for requests no real visitor sends, - anchored at the site root. A file of the same name deeper in a site can be - ordinary content, such as a file in a gitea repository. A visitor banned by - mistake is let back in by lifting the ban. -- At start, a file that does not parse, a regex that does not compile, or a - duplicate id stops the process with a message naming the file and line. So - does a `SWWAF_RULES_DIR` that does not exist. While running, the same faults - leave the rules as they were, and the log and one `file_error` alert name the - file and line. An empty directory is not an error. + encoded probe cannot slip past. A header sent more than once is matched with + its values joined by `, `; a header not sent is empty text. `Host` and + `Transfer-Encoding` cannot be targets, since Go's HTTP server takes them out + of every request. Request bodies are not available to rule files; reading them + is the Core Rule Set's job. +- `log` only notes the match in the request log. `block` refuses the request + without a ban; `ban` makes the request a clear sign of attack (see "Bans"). + `ban` is for requests no real visitor sends, anchored at the site root. A file + of the same name deeper in a site can be ordinary content, such as a file in a + gitea repository. A visitor banned by mistake is let back in by lifting the + ban. +- At start, these stop the process with a message naming the file and line: + - a line that does not parse; + - a regex that does not compile; + - a rule for `Host` or `Transfer-Encoding`; + - a duplicate id. + + So does a `SWWAF_RULES_DIR` that does not exist or cannot be read. While + running, the same faults leave the rules as they were, and the log and one + `file_error` alert name the file and line. Once the file is fixed, it is + read again. An empty directory is not an error: the log says that no rules + were loaded. + - The image ships a default file, kept short and limited to patterns that are - wrong for every app, with its path rules anchored at the site root. Anything - app-specific belongs in a file the app's Dockerfile adds or the deployer - mounts. Rules meant to be edited while `smallwebwaf` runs belong in a mounted + wrong for every app, with its path rules anchored at the site root. In front + of gitea, `/.env` is a probe, while + `///src/branch/main/.env.example` is a file in a repository that + any visitor may open. Anything app-specific, such as a rule banning WordPress + probes in front of gitea, belongs in a file the app's Dockerfile adds or the + deployer mounts. A directory mounted over `SWWAF_RULES_DIR` replaces the + default file; single files mounted into it add to it. +- Rules meant to be edited while `smallwebwaf` runs belong in a mounted directory, not a mounted file. Many editors save a file by replacing it, and a container keeps seeing the old one when the file alone is mounted. The lookup - database is mounted the same way, for the same reason. + database is mounted the same way, for the same reason. To run without rule + files, mount an empty directory or set `SWWAF_RULES_ENABLED=false`. ## Persistent state @@ -735,65 +884,94 @@ of it, so an orderly stop loses nothing. The one exception is log lines still waiting to be sent to `SWWAF_LOG_REMOTE_URL`, which stdout has already carried. The files are meant for people as well: an admin can read or edit them at any time, and the running `smallwebwaf` takes the edit in. `docs/state-files.md` -lists the files, what each holds, their sizes, and how an edit is taken in. +lists the files, what each holds, and their sizes. - Each file holds one kind of state, as JSON with: - a top-level `version` number; - entries sorted by client address; - times in RFC 3339 UTC; - durations and sizes as plain numbers, with the unit in the field name. -- `bans.json` is indented, so that a person can open it in an editor, find an - address, and remove or add an entry. `clients.json` and `lookups.json` put - each entry on one line, which halves their size and lets `grep` show - everything about one client. +- `bans.json`, `reputation.json` and `alerts.json` are indented, so that a + person can open `bans.json` in an editor, find an address, and remove or add + an entry. `clients.json` and `lookups.json` put each entry on one line, which + halves their size and lets `grep` show everything about one client. - Clients that were never banned are kept too, so every client's history survives a restart. -- Each file is rewritten whole, so each has a bound given by its setting. - Writing a file of these sizes takes well under a second, in the background, so - rewriting whole files needs nothing cleverer. +- Each file has a bound, so it can be rewritten whole. Writing a file of these + sizes takes well under a second, in the background, so rewriting whole files + needs nothing cleverer. - A write serialises from a snapshot taken under the lock, so requests are not held up. It goes to a temporary file in the same directory, which is synced - and renamed over the real name. A crash at any point leaves either the old - complete file or the new complete file, never a partial one. Before writing a - file, `smallwebwaf` takes in any edit made to it on disk, so an admin's edit - is never overwritten. -- At start, a file that does not parse, or has an unknown `version`, stops the - process with a message naming the file and position. Starting with empty state - would silently forgive every repeat offender. Since writes are atomic, a - broken file can only come from a hand edit, which the editor should hear - about. A missing file means empty state, and is normal on first run. -- While running, an edit that parses replaces what `smallwebwaf` held for that - file. An edit that does not parse never stops the running `smallwebwaf`. It - keeps the state it has and sets the edited file aside as `.bad`. It - writes the file again from memory, and names the file and the position of the - error in the log and in one `file_error` alert. + and renamed over the real name, and the directory is synced. A crash at any + point leaves either the old complete file or the new complete file, never a + partial one. Before writing a file, `smallwebwaf` takes in any edit made to it + on disk, so an admin's edit is never overwritten. +- At start, the files are read back. Active bans go into an in-memory lookup by + netblock, everything else into maps. Counter buckets and cache entries whose + time has passed are discarded. A missing file means empty state, and is normal + on first run. A file that does not parse, or has an unknown `version`, stops + the process with a message naming the file and position; so does an entry + without a field it needs, a ban whose cause is not one of the four, or a ban + whose `expires` is not after its `start`. Starting with empty state would + silently forgive every repeat offender. Since writes are atomic, a broken file + can only come from a hand edit, which the editor should hear about. +- While running, `smallwebwaf` notices a file that is edited, replaced or added. + It tells its own writes from an admin's by comparing the file with what it + last read or wrote. An edit that parses is taken in at once: what the file + says replaces what `smallwebwaf` held for that file. Removing a ban's entry + from `bans.json` lifts the ban; adding an entry bans; setting `lifted` lifts + it and keeps it. A change `smallwebwaf` made after the admin opened the file, + such as a new ban, is lost when the admin saves over it. Most editors warn + when a file changed on disk while it was open. A file that is removed is + written again at its next write. +- An edit that does not parse never stops the running `smallwebwaf`. It keeps + the state it has and sets the edited file aside as `.bad`, so the edit + is kept. It writes the file again from memory, and names the file and the + position of the error in the log and in one `file_error` alert. The admin + mends the `.bad` file and moves it back. - A write that fails leaves the state correct in memory. It is logged, counted in the metrics and sent as one `file_error` alert per cooldown, and retried at the next write. -- A hard kill loses the changes since the last write. Losing the volume loses - ban history, not service. +- A hard kill loses the changes since the last write: ban changes from the last + `SWWAF_STATE_WRITE_DELAY`, everything else from the last + `SWWAF_STATE_COUNTER_INTERVAL`. Losing the volume loses ban history, not + service. ## Request log Every request gets one JSON object on one line on stdout, refused ones included, apart from those Go's HTTP server ends before `smallwebwaf` sees them (see "Size -and time limits"). `docs/request-log.md` lists the fields and describes the -syslog sending. Bodies are never logged. Query strings are logged as received; -an app that carries secrets in query strings needs that fixed in the app. The -process's own messages share the stream as JSON lines with `"type":"process"`; -request lines carry `"type":"request"`. +and time limits"). `docs/request-log.md` lists the fields. Bodies are never +logged, and no header but those listed there. Query strings are logged as +received; an app that carries secrets in query strings needs that fixed in the +app. The process's own messages (start, the settings, stop, errors) share the +stream as JSON lines with `"type":"process"`; request lines carry +`"type":"request"`. When `SWWAF_LOG_REMOTE_URL` is set, the same lines are also sent there as RFC 5424 syslog records. So a deployment can stop depending on docker's log handling -while `docker logs` keeps working. Sending happens on its own goroutine from a -bounded buffer: an unreachable or slow server never delays a request and never -stops stdout. UDP gives no delivery signal at all and is offered only for -compatibility. +while `docker logs` keeps working. + +- Over UDP each record is one datagram; a line too long for one datagram is + dropped alone. Over TCP and TLS each record follows its length in bytes and a + space. UDP gives no delivery signal at all and is offered only for + compatibility. +- The record carries the facility from `SWWAF_LOG_REMOTE_FACILITY`, the severity + informational, the time, the host's name and the app name from + `SWWAF_LOG_REMOTE_APP_NAME`. +- Sending happens on its own goroutine from a buffer of + `SWWAF_LOG_REMOTE_BUFFER` lines, so a server that is slow or down never holds + up a request and never stops stdout. When the buffer is full, the oldest line + is dropped and counted in the metrics. A failed connection is tried again a + second later, and twice as long after each further failure, up to a minute. +- As `smallwebwaf` stops, it spends at most two seconds sending the lines still + waiting, and drops the rest; stdout has carried them. ## Metrics endpoint `GET /_smallwebwaf/metrics` answers in the Prometheus text format, for a request -carrying `SWWAF_METRICS_TOKEN`. `docs/metrics.md` lists the metrics. No metric +carrying `SWWAF_METRICS_TOKEN`. `docs/metrics.md` lists the metrics. Every +metric carries the label `instance`, set to `SWWAF_INSTANCE_NAME`. No metric carries a client IP address as a label. Per-address questions are answered by the request log and `GET /_smallwebwaf/clients/`. @@ -803,7 +981,7 @@ A request whose path starts with `/_smallwebwaf/` is for `smallwebwaf` itself: it answers it and never passes it to the app. The prefix carries the tool's name, so it takes no path an app uses. Admins and scrapers reach these endpoints through traefik, like any other request. `docs/admin-endpoints.md` lists the -endpoints and what each answers. +endpoints, what each takes and what each answers. - `GET /_smallwebwaf/healthz` answers `200` with `ok` to anyone, without a token, while `smallwebwaf` is running. It does not ask the app; the @@ -819,28 +997,54 @@ endpoints and what each answers. refused, and each request counts toward the client's limits. A missing or wrong token is answered `401`, in `SWWAF_MODE=observe` too, and counts toward the error burst. A client guessing tokens is therefore banned once it passes - `SWWAF_ERROR_BURST_THRESHOLD` guesses in a minute. A client in - `SWWAF_ALLOW_NETS` skips the checks but still needs the token. + `SWWAF_ERROR_BURST_THRESHOLD` guesses in a minute. +- A ban made through the endpoints starts at once, with the cause `admin`. + Lifting through the endpoints lifts every active ban covering the address + given. ## Deployment `smallwebwaf` runs inside the app's own container: the app's Dockerfile builds -`FROM` the `smallwebwaf` image. Each app remains one container. An app is -hardened by changing its base image. It can add packages from nixpkgs. +`FROM` the `smallwebwaf` image. Each app is one container. An app is hardened by +changing its base image. It can add packages from nixpkgs. `docs/image.md` lists +what the image holds and how each part is pinned. -The image holds: - -- Ubuntu 26.04 LTS, with `ca-certificates`, `nix-bin` and `runit` from Ubuntu; -- nixpkgs, fixed at one release; -- `runsvinit` as the entrypoint; -- the `smallwebwaf` binary, a user of its own (uid and gid 65532), and its runit - service; -- its state and rules directories; -- `EXPOSE 8080` and the health check below. - -Every input to the image is pinned by digest or hash, so that an app built on it -gets the same packages each time. `docs/image.md` says how each part is pinned, -and why. +- The image is Ubuntu's newest long-term support release, pinned by digest, and + moves to the next LTS release when that ships. Its packages come from Ubuntu's + snapshot service, the archive as it was at the moment the Dockerfile names, + rather than the live archive, whose packages change with every update. The + snapshot is never older than the pinned Ubuntu image, since older packages can + need older versions of packages the image holds, and it moves forward with the + image's digest. The Dockerfile names the hash of each of the snapshot's + `InRelease` files, which cover every package list and so every package, and + the build checks them before installing. The hashes are the amd64 archive's, + so the image is built for amd64 only. +- The Ubuntu image has no CA certificates. Without them Nix cannot download + packages and `smallwebwaf` cannot reach GeoJS, so `ca-certificates` is + installed by name. The snapshot itself is fetched over HTTPS with the CA + certificates of the Go image that builds `smallwebwaf`, itself pinned by + digest. +- nixpkgs comes from a release file on `releases.nixos.org`, named with its + hash, not from a GitHub archive, whose bytes can change. Root uses Nix + directly; no Nix daemon runs, and since `nix-bin` creates no build users, + `build-users-group =` in `/etc/nix/nix.conf` lets root build without them. + Root's Nix profile is last on the `PATH`, so that no package hides the image's + own commands; busybox's `sv`, for one, looks for services elsewhere. An app + built on the image gets the same packages each time. Security fixes from + Ubuntu and nixpkgs come with a newer `smallwebwaf` image, which an app takes + by changing the digest in its `FROM` line; the image moves to each new nixpkgs + release, every six months. +- runit supervises the two processes, with `runsvinit` as the entrypoint, as the + owner's code style guide asks for service containers. No package carries + `runsvinit`, so the image builds it from its archived source, unchanged since + 2015, at a fixed commit hash, with a `go.mod` the build writes. `runsvinit` + starts runit's `runsvdir`, which runs the `run` script of each directory under + `/etc/service` and runs it again whenever it exits. Ubuntu's `sv` looks for + services there too. +- The image's `run` script for `smallwebwaf` waits one second and makes + `SWWAF_STATE_DIR` and every file in it belong to the `smallwebwaf` user (uid + and gid 65532). Then it starts `smallwebwaf` as that user with runit's + `chpst`. Every `run` script, the app's included, is a bash script. It starts with `#!/usr/bin/env bash` and `set -euo pipefail`, and puts its code in a `main` @@ -917,15 +1121,16 @@ around `smallwebwaf` by anything that reaches the container. ### State -`smallwebwaf` keeps its state files in `SWWAF_STATE_DIR`, beside the app's own -data directories. Without a volume there, the files survive a restart of the -container and are lost when a deploy replaces it. A volume mounted at -`/var/lib/smallwebwaf`, named or a host directory, keeps them across deploys. -The `run` script makes it and every file in it belong to the `smallwebwaf` user, -so a host directory mounted there needs no change of owner. The volume is -separate from the app's, holds a few tens of MiB at most with the defaults, and -needs no backup beyond the host's own. The image declares no volume, since every -app image built on it would inherit it. +`smallwebwaf` keeps its state files in `SWWAF_STATE_DIR`, a directory of its own +beside the app's data directories, belonging to the `smallwebwaf` user. Without +a volume there, the files survive a restart of the container and are lost when a +deploy replaces it. A volume mounted at `/var/lib/smallwebwaf`, named or a host +directory, keeps them across deploys. A host directory mounted there needs no +change of owner, since the `run` script gives it to the `smallwebwaf` user at +each start; files an earlier owner left in it can be read and replaced. The +volume is separate from the app's, holds a few tens of MiB at most with the +defaults, and needs no backup beyond the host's own. The image declares no +volume, since every app image built on it would inherit it. ### Tokens @@ -934,6 +1139,7 @@ is out of the app's reach only while the `smallwebwaf` user alone can read the file. The operator makes the file on the host, owned by uid 65532, the `smallwebwaf` user, with mode `0400`, and mounts the directory that holds it into the container read-only. The container sees the same owner and mode. +"Settings given as files" in `docs/settings.md` shows the commands. ### Forwarded headers diff --git a/docs/admin-endpoints.md b/docs/admin-endpoints.md index 85563f0..dde1d67 100644 --- a/docs/admin-endpoints.md +++ b/docs/admin-endpoints.md @@ -1,26 +1,20 @@ # Admin endpoints -`smallwebwaf` answers these requests itself, on the app's own address and -through traefik like any other request. Apart from the health check, each needs -a token, sent as `Authorization: Bearer `, and goes through every check -any other request goes through. It is answered where another request would be -passed to the app: a banned client stays refused, and each request counts toward -rate limiting. A missing or wrong token is answered `401`, in `observe` mode -too. Such a refusal counts toward the error burst, so a client guessing tokens -is soon banned. While a token is unset, its endpoints answer `404`. +The endpoints under `/_smallwebwaf/`, what each takes and what each answers. +"Admin endpoints" in [`SPEC.md`](../SPEC.md) says which checks they go through +and how their tokens work. -- `GET /_smallwebwaf/healthz`: `200` and `ok`, to anyone, before any check and - without asking the app. +- `GET /_smallwebwaf/healthz`: `200` and `ok`, to anyone. - `GET /_smallwebwaf/metrics`: the metrics, for `SWWAF_METRICS_TOKEN` (see [`metrics.md`](metrics.md)). - `GET /_smallwebwaf/bans`: every ban held, under `bans`, each as `bans.json` lists it. Needs `SWWAF_ADMIN_TOKEN`, as do the three below. - `POST /_smallwebwaf/bans`: bans a netblock. The body is a JSON object of `netblock`, `duration` and, if you like, `reason`. `netblock` is a netblock - such as `203.0.113.0/24`, or a client's address. A client's address bans its - IPv4 netblock (`SWWAF_BAN_SCOPE_V4_PREFIX`) or its IPv6 group. `duration` is a - duration such as `1h` or `7d`, or `permanent`. The ban starts at once, with - the cause `admin`. A bad body is answered `400`, saying what is wrong: + such as `203.0.113.0/24`, or a client's address, which bans its IPv4 netblock + (`SWWAF_BAN_SCOPE_V4_PREFIX`) or its IPv6 group. `duration` is a duration such + as `1h` or `7d`, or `permanent`. A bad body is answered `400`, saying what is + wrong: - one that is not such an object, or has another field; - one longer than 4 KiB; - one that names an IPv4-mapped netblock, or an address with a zone. @@ -28,8 +22,27 @@ is soon banned. While a token is unset, its endpoints answer `404`. address ``. It answers `404` when none is active. - `GET /_smallwebwaf/clients/`: what `smallwebwaf` knows of the client. Under `client`, its counters and history as `clients.json` holds them, or - `null` when it has never been seen. Under `bans`, every ban covering the - address, with its notes. + `null` when `clients.json` does not hold it. Under `bans`, every ban covering + the address, with its notes. -An admin whose own address is banned lifts that ban by editing `bans.json` (see -[`bans.md`](bans.md)). +## Examples + +With `SWWAF_ADMIN_TOKEN` set, and the token in `$TOKEN`: + +```sh +# Every ban. +curl -H "Authorization: Bearer $TOKEN" https://app.example/_smallwebwaf/bans + +# Ban 203.0.113.0/24 for seven days. +curl -H "Authorization: Bearer $TOKEN" \ + --json '{"netblock": "203.0.113.0/24", "duration": "7d", "reason": "probes for logins"}' \ + https://app.example/_smallwebwaf/bans + +# Lift the bans on 203.0.113.9. +curl -H "Authorization: Bearer $TOKEN" -X DELETE \ + https://app.example/_smallwebwaf/bans/203.0.113.9 + +# What smallwebwaf knows of 203.0.113.9. +curl -H "Authorization: Bearer $TOKEN" \ + https://app.example/_smallwebwaf/clients/203.0.113.9 +``` diff --git a/docs/alerts.md b/docs/alerts.md index 6577753..1da7174 100644 --- a/docs/alerts.md +++ b/docs/alerts.md @@ -1,17 +1,17 @@ # Alerts -Each alert goes to each destination you name. `SWWAF_ALERT_WEBHOOK_URL` gets it -as one JSON object. `SWWAF_ALERT_SLACK_WEBHOOK_URL` and `SWWAF_ALERT_NTFY_URL` -get a message made from it. The settings are under "Alert destinations" in -[`settings.md`](settings.md). An alert is one of these events, and is sent when -its event is in `SWWAF_ALERT_EVENTS`: +The alert events, the JSON object the webhook gets, and the messages Slack and +ntfy get. "Alerting" in [`SPEC.md`](../SPEC.md) says when an alert is sent, held +back or dropped. + +## Events - `ban`: a ban `smallwebwaf` made. - `permanent_ban`: a permanent ban it made, or an existing ban made permanent by a request. -- `waf_block`: the Core Rule Set scored a request at or over the threshold. In - `block` mode the request was refused. In `detect` mode it was not, and the - alert's `detail` gives `mode`. +- `waf_block`: the Core Rule Set scored a request at or over the threshold. The + request was refused unless the alert's `detail` gives `mode`: `detect` or + `observe`. - `anomaly`: a count over an anomaly threshold. - `reputation_hit`: a request from a client on a blocklist, the CrowdSec decision list or a DNS blocklist, or with an AbuseIPDB score that is a hit. @@ -25,12 +25,7 @@ its event is in `SWWAF_ALERT_EVENTS`: - a rule file or state file edit with an error; - a state file that cannot be written; - a replacement lookup database that cannot be read. -- `summary`: the hourly summary described under "Repeats and the hourly cap" - below. - -The bans you make raise no alert. In `observe` mode, a request that would have -made a ban still raises the ban's alert, with `mode` set to `observe` in its -`detail`. +- `summary`: the hourly summary of alerts held back. ## The JSON object @@ -81,24 +76,26 @@ on one line: } ``` -- `instance` is `SWWAF_INSTANCE_NAME`. `time` is when the alert was raised. - `event` is one of the events above. -- `client` is the address whose request raised the alert. `netblock` is the - ban's netblock, the netblock an anomaly counted, or the client's group. Both - are empty for `source_failure` and `file_error`. `asn`, `as_name` and - `country` are the client's as known when the alert was raised. They are empty - when GeoJS had not answered yet. -- `reason` is a short sentence, such as +- `instance`: `SWWAF_INSTANCE_NAME`. `time`: when the alert was raised. `event`: + one of the events above. +- `client`: the address whose request raised the alert. `netblock`: the ban's + netblock, the netblock an anomaly counted, or, for the other events, the + client as rate limiting counts it: an IPv4 address as a /32, or an IPv6 group. + Both are empty for `source_failure` and `file_error`. +- `asn`, `as_name` and `country`: the client's as known when the alert was + raised. They are empty when GeoJS had not answered yet. +- `reason`: a short sentence, such as `requests per minute over the limit of 1000` or `listed by a blocklist`. -- `detail` is what is particular to the event: - - for a ban, its `cause`, `ban_expires` and `notes` (see - [`bans.md`](bans.md) for the notes); +- `detail`: what is particular to the event: + - for a ban, its `cause`, `ban_expires` and `notes` (the notes are listed + under `bans.json` in [`state-files.md`](state-files.md)); - for an anomaly, the `scope`, `window`, `kind`, `count` and `threshold`; - for a reputation hit, the `source` and, for AbuseIPDB, the `score`; - for a Core Rule Set match, the `rule_ids`, `score`, `method` and `path`; - for a source failure, the `source` and `error`; - - for a file error, the `file` and `error`. -- `suppressed_repeats` is how many repeats the cooldown held back before this + - for a file error, the `file` and `error`; + - `mode`, when the request was not refused: `detect` or `observe`. +- `suppressed_repeats`: how many repeats the cooldown held back before this alert. ## Slack and ntfy @@ -128,22 +125,3 @@ The priority and tag by event: - `reputation_hit`: priority `low`, tag `label`. - `source_failure` and `file_error`: priority `high`, tag `warning`. - `summary`: priority `default`, tag `bar_chart`. - -## Repeats and the hourly cap - -A repeat is an alert for the same event on the same netblock, source, file or -thing counted, less than `SWWAF_ALERT_COOLDOWN` after the last. It is held back -and counted, and the next alert sent gives the count. Past -`SWWAF_ALERT_MAX_PER_HOUR` alerts in an hour, the rest are held back and counted -by event. Once the hour ends, one alert with the event `summary` gives those -counts. It also counts the repeats that a cooldown held back and that no later -alert reported. - -## Sending - -Each destination has a queue of at most 1000 alerts, sent one at a time, so a -slow destination holds up nothing. A `2xx` answer takes an alert. A `4xx` other -than `408` and `429` refuses it, and it is dropped. Anything else is a failure: -the alert is sent again a second later, and twice as long after each further -failure, up to a minute. `alerts.json` keeps the queues and cooldowns across a -restart (see [`state-files.md`](state-files.md)). diff --git a/docs/bans.md b/docs/bans.md deleted file mode 100644 index bd0f4e6..0000000 --- a/docs/bans.md +++ /dev/null @@ -1,84 +0,0 @@ -# Bans - -How bans are made, what they cover and how long they last is under "Bans" in -[`SPEC.md`](../SPEC.md); the settings are under "How long bans last and what -they cover" in [`settings.md`](settings.md). This file says what a ban's notes -hold, and how to make, keep or lift a ban. - -## The notes - -Each ban carries notes for deciding whether to lift it, drawn from what -`smallwebwaf` already knows. Nothing is looked up to fill them: - -- the AS number, AS name and country, added when the lookup answers (empty when - lookups are off); -- what was broken: the rule id and target that matched, or the trap path, or the - limit, its window and the count reached; -- the client's percentage of the limit, with the setting that gave it; -- the reputation sources the client is on; -- the request that caused the ban: time, method, host, path with its query - string, status and user agent, each text cut to 256 bytes; -- the netblock's requests since it was first seen, and how many this ban has - refused so far, kept up to date while the ban lasts; -- how many earlier bans the netblock had, by cause. - -## Managing bans by editing `bans.json` - -`bans.json` in the state directory holds every ban, indented to be read. A -permanent ban's `expires` is `null`. `smallwebwaf` watches the file and takes in -an edit as soon as it is saved (see [`state-files.md`](state-files.md)). - -To ban a netblock, add an entry with its `netblock`, `start` and `expires`. -`expires` is `null` for a ban that never ends. `reason`, `notes` and `cause` may -be left out; the cause is then `admin`: - -```json -{ - "version": 1, - "bans": [ - { - "netblock": "203.0.113.0/24", - "start": "2026-10-06T12:00:00Z", - "expires": null, - "reason": "probes for logins" - } - ] -} -``` - -To keep a ban `smallwebwaf` made, so that it is never dropped, set its `cause` -to `admin`. To lift a ban, add `"lifted"` with the time you lift it, such as -`"lifted": "2026-10-06T13:00:00Z"`. The ban then refuses nothing and does not -make the netblock's next ban longer. It is kept with its notes. To forget a ban, -delete its entry. A ban whose cause is `attack` becomes permanent at the first -request it refuses. One whose cause is `admin` or `crowdsec` does not. - -An edit that cannot be read does not stop the running `smallwebwaf`. It keeps -what it holds. At the file's next write it renames your file to `bans.json.bad`, -writes the file again from memory, and logs and alerts the error. Mend the -`.bad` file and move it back. - -## Managing bans through the admin endpoints - -With `SWWAF_ADMIN_TOKEN` set, and the token in `$TOKEN`: - -```sh -# Every ban. -curl -H "Authorization: Bearer $TOKEN" https://app.example/_smallwebwaf/bans - -# Ban 203.0.113.0/24 for seven days. -curl -H "Authorization: Bearer $TOKEN" \ - --json '{"netblock": "203.0.113.0/24", "duration": "7d", "reason": "probes for logins"}' \ - https://app.example/_smallwebwaf/bans - -# Lift the bans on 203.0.113.9. -curl -H "Authorization: Bearer $TOKEN" -X DELETE \ - https://app.example/_smallwebwaf/bans/203.0.113.9 - -# What smallwebwaf knows of 203.0.113.9. -curl -H "Authorization: Bearer $TOKEN" \ - https://app.example/_smallwebwaf/clients/203.0.113.9 -``` - -See [`admin-endpoints.md`](admin-endpoints.md) for what each answers. An admin -whose own address is banned lifts that ban by editing `bans.json`. diff --git a/docs/core-rule-set.md b/docs/core-rule-set.md deleted file mode 100644 index 066e749..0000000 --- a/docs/core-rule-set.md +++ /dev/null @@ -1,31 +0,0 @@ -# The Core Rule Set - -The OWASP Core Rule Set 4.25.0 runs on each request after the rule files, unless -`SWWAF_WAF_MODE` is `off`. It reads the method, the URL with its query, and the -headers. Once `SWWAF_WAF_BODY_LIMIT` is set, it also reads a body that is form -data, multipart, JSON or XML. No answer is inspected. The settings are under -"Attack detection" in [`settings.md`](settings.md). The same section of -[`SPEC.md`](../SPEC.md) says why bodies are not read by default, and gives the -six changes `smallwebwaf` makes to the Core Rule Set, with the reason for each. - -Each rule that matches adds to the request's score. A score at or over -`SWWAF_WAF_ANOMALY_THRESHOLD` is a match. A match is refused with `403` while -`SWWAF_WAF_MODE` is `block`, and let through while it is `detect`. Either way -the log line gives the rule ids and the score, and a `waf_block` alert is sent. -A request with more than 1000 query parameters or body fields adds 5 to its -score, since Coraza reads only the first 1000. The Core Rule Set inspects only a -request that every earlier check let through. It skips a path under -`SWWAF_WAF_EXEMPT_PATHS` and a client in `SWWAF_ALLOW_NETS`. An engine error on -a request forwards the request, and is logged and counted. - -Body inspection suits apps whose forms carry no code. In front of a code forge -it refuses issue and comment text, wiki pages and files saved in the web editor -that hold shell commands or code. The log line names the rule ids, which -`SWWAF_WAF_DISABLED_RULES` can switch off. A body the Core Rule Set cannot parse -adds 5 to the score. So does a multipart body cut by the limit in the middle of -a part's header. The client has until `SWWAF_CLIENT_REQUEST_TIMEOUT` runs out to -send the part that is read. Any other body, such as a git push, reaches the app -uninspected. - -[`gitea.md`](gitea.md) records what the Core Rule Set still refuses in front of -gitea, with and without body inspection. diff --git a/docs/gitea.md b/docs/gitea.md index d703385..46b8143 100644 --- a/docs/gitea.md +++ b/docs/gitea.md @@ -13,16 +13,19 @@ checked again when gitea or the Core Rule Set changes version. by it. - A clone is a response. It fits the defaults of 30 minutes and 5 GiB for all but the largest repositories and slowest links. -- A push is a request, and so is every other upload: a git push, an LFS object, - a container image layer, a package file or a release attachment. At the - defaults, an upload larger than 100 MiB or taking more than 60 seconds is cut - off. The client is answered `413` for a body that is too large, before - anything reaches gitea when the request announces its size. It is answered - `408` for a body it sends too slowly, or `504` if gitea is too slow to take - it. The upload fails, and no one is banned for it. A gitea that takes large - uploads needs `SWWAF_REQUEST_MAX_BYTES`, `SWWAF_CLIENT_REQUEST_TIMEOUT` and - `SWWAF_UPSTREAM_REQUEST_TIMEOUT` raised to fit. The Core Rule Set does not - read an upload's body, which streams through without being held in memory. +- A push is a request, and so is every other upload: + - a git push; + - an LFS object; + - a container image layer; + - a package file; + - a release attachment. + + At the defaults, an upload larger than 100 MiB or taking more than 60 + seconds is cut off, and no one is banned for it. "Size and time limits" in + `SPEC.md` gives the answers the client gets. A gitea that takes large + uploads needs `SWWAF_REQUEST_MAX_BYTES`, `SWWAF_CLIENT_REQUEST_TIMEOUT` and + `SWWAF_UPSTREAM_REQUEST_TIMEOUT` raised to fit. + - Archive download and blame or history pages are what scrapers hammer. The request limits do most of the work there. - An Actions runner that reaches gitea through `smallwebwaf` sends requests all @@ -32,8 +35,7 @@ checked again when gitea or the Core Rule Set changes version. jobs. It is then banned, the job it is running fails, and each repeat within a day makes the ban longer. Newer runners ask less often, but a busy one, or several on one address, can still pass it. Put the runners' addresses in - `SWWAF_RATE_LIMIT_EXEMPT_NETS`. That takes them out of the request and byte - limits, while the error burst, attack detection and bans still apply. + `SWWAF_RATE_LIMIT_EXEMPT_NETS`. ## What the Core Rule Set lets through @@ -55,10 +57,9 @@ the refusals in the next section: ## What the Core Rule Set still refuses -Each refusal answers only that request, with `403`, and bans no one by itself. -It counts toward the error burst, which a person does not reach this way. The -request log names the rule in `waf_rule_ids`, which `SWWAF_WAF_DISABLED_RULES` -can switch off. +Each refusal answers only that request ("Bans" in `SPEC.md` says what it counts +toward). The request log names the rule in `waf_rule_ids`, which +`SWWAF_WAF_DISABLED_RULES` can switch off. ### A query string that reads as an attack @@ -151,9 +152,9 @@ carries it. A push can reuse a layer of an image docker has pushed to that gitea before. docker then asks gitea to take the layer from that image, naming it in the query parameter `from`, which 932260 checks. The request is refused when the image's -owner, or its name after the owner, starts with an entry on the list of -commands. Examples are `owner/python3-app`, `owner/docker-base`, and any image -of an owner named `ansible` or `docker-team`. +owner starts with an entry on the list of commands. So is one whose name after +the owner does. Examples are `owner/python3-app`, `owner/docker-base`, and any +image of an owner named `ansible` or `docker-team`. What happens next depends on docker's image store. A docker that keeps its images in containerd's image store fails the push, and fails each new try the diff --git a/docs/image.md b/docs/image.md index c208356..d2955c0 100644 --- a/docs/image.md +++ b/docs/image.md @@ -1,56 +1,36 @@ # What the image holds -The `smallwebwaf` image is the base of each app's own image. The requirements -are under "Deployment" in [`SPEC.md`](../SPEC.md); this file lists what the -image holds, how each part is pinned, and why. +The contents of the `smallwebwaf` image, and how each part is pinned. +"Deployment" in [`SPEC.md`](../SPEC.md) says why each part is there and why it +is pinned that way. ## Contents -- Ubuntu 26.04 LTS, the newest long-term support release of Ubuntu, pinned by - digest. The image moves to the next LTS release when that ships. -- `ca-certificates`, `nix-bin` and `runit` from Ubuntu, installed from a dated - snapshot of its archive and checked by hash. The Ubuntu image has no CA - certificates; without them Nix cannot download packages and `smallwebwaf` - cannot reach GeoJS. -- Nix and nixpkgs, fixed at one release. Root uses Nix directly; no Nix daemon - runs. `nix-bin` creates no build users, so `build-users-group =` in - `/etc/nix/nix.conf` lets root build without them. -- runit, and `runsvinit` as the entrypoint, as the owner's code style guide asks - for service containers. No package carries `runsvinit`, so the image builds it - from its archived source (`github.com/peterbourgon/runsvinit`, unchanged - since 2015) at a fixed commit hash, with a `go.mod` the build writes. - `runsvinit` starts runit's `runsvdir`, which runs the `run` script of each - directory under `/etc/service` and runs it again whenever it exits. Ubuntu's - `sv` looks for services there too. -- The `smallwebwaf` binary, and a user of its own, `smallwebwaf` (uid and gid - 65532). -- The service directory `/etc/service/smallwebwaf`. Its `run` script waits one - second (`sleep 1`), makes `SWWAF_STATE_DIR` and every file in it belong to the - `smallwebwaf` user, and starts `smallwebwaf` as that user with runit's - `chpst`. -- The directory `/var/lib/smallwebwaf` for the state files, and - `/etc/smallwebwaf/rules.d` with the default rule file. -- Port 8080 declared (`EXPOSE 8080`), and the health check (`HEALTHCHECK`) that - "The health check" in `SPEC.md` describes. +- Ubuntu 26.04 LTS, pinned by digest. +- `ca-certificates`, `nix-bin` and `runit` from Ubuntu, from the snapshot of its + archive at `20261001T000000Z`. +- Nix, with nixpkgs `nixos-26.05.11045.774debe7a0d1` set up for root as + `nixpkgs`. No Nix daemon; `build-users-group =` in `/etc/nix/nix.conf`. Root's + Nix profile, `/nix/var/nix/profiles/default/bin`, last on the `PATH`. + Unpacked, nixpkgs takes about 500 MiB. +- `runsvinit` as the entrypoint, built from `github.com/peterbourgon/runsvinit` + at a fixed commit hash. +- The `smallwebwaf` binary, and the user `smallwebwaf` (uid and gid 65532). +- The service directory `/etc/service/smallwebwaf`, with the `run` script + `share/smallwebwaf.run`. +- `/var/lib/smallwebwaf` for the state files, and `/etc/smallwebwaf/rules.d` + with the default rule file `share/rules.d/00-default.rules`. +- `EXPOSE 8080`, and the `HEALTHCHECK` that runs `smallwebwaf healthcheck`. -## Pinning +## Pins -- Ubuntu's packages come from Ubuntu's snapshot service, the archive as it was - at the moment the Dockerfile names (`--snapshot 20261001T000000Z`), never - earlier than the pinned Ubuntu image's date. The Dockerfile names the SHA-256 - hash of each of the snapshot's `InRelease` files, which cover every package - list and so every package, and the build checks them before installing. The - hashes are the amd64 archive's, so the image is built for amd64 only. The - snapshot is fetched over HTTPS with the CA certificates of the Go image that - builds `smallwebwaf`, itself pinned by digest, since the Ubuntu image has - none. -- nixpkgs comes from a release file on `releases.nixos.org` (`nixexprs.tar.xz` - of `nixos-26.05.11045.774debe7a0d1`), named with its SHA-256 hash, not from a - GitHub archive, whose bytes can change. It is set up for root as `nixpkgs`. - Root's Nix profile, `/nix/var/nix/profiles/default/bin`, is last on the - `PATH`, so that no package hides the image's own commands; busybox's `sv`, for - one, looks for services elsewhere. Unpacked, nixpkgs takes about 500 MiB. -- An app built on the image gets the same packages each time it is built. - Security fixes from Ubuntu and nixpkgs come with a newer `smallwebwaf` image, - which an app takes by changing the digest in its `FROM` line. The image moves - to each new nixpkgs release, every six months. +- Ubuntu's packages: the snapshot service, `--snapshot 20261001T000000Z` on both + `apt-get update` and `apt-get install`. The SHA-256 hash of each of the + snapshot's `InRelease` files is named in the `Dockerfile` and checked after + `apt-get update`. The hashes are the amd64 archive's. +- nixpkgs: the release file `nixexprs.tar.xz` of + `nixos-26.05.11045.774debe7a0d1` on `releases.nixos.org`, named with its + SHA-256 hash. +- `runsvinit`: the archived repository, unchanged since 2015, at tag `v2.0.0`, + by commit hash, with a `go.mod` the build writes. +- The Go image that builds `smallwebwaf`: by digest. diff --git a/docs/lookup.md b/docs/lookup.md deleted file mode 100644 index 45909d0..0000000 --- a/docs/lookup.md +++ /dev/null @@ -1,79 +0,0 @@ -# GeoJS and the lookup database - -`smallwebwaf` looks up each client's AS number and country, through GeoJS by -default or in a database file on your own host. The settings are under "Looking -up the AS number and country" in [`settings.md`](settings.md); the reasons for -the defaults are under "Lookup of AS number and country" in -[`SPEC.md`](../SPEC.md). - -## GeoJS - -By default `smallwebwaf` asks GeoJS, a free web service that needs no account, -about every new visitor's address. It asks whether or not a setting uses the -answer, since the log line, the history, the ban notes and the metrics carry the -AS number and country. These addresses are never sent: - -- one in `SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS`; -- one under a ban; -- a private, loopback or link-local address, which no source can place. - -For an IPv6 visitor, GeoJS is asked about the first address of its group. -Addresses are asked about together, up to 200 in one request -(`https://get.geojs.io/v1/ip/geo.json?ip=a,b,c`), one request at a time. Each -answer's `country_code`, `asn` and `organization_name` are read. An answer -without a `country_code`, or with the `asn` `64512`, counts as unknown. Each -answer is kept for seven days, in memory and in `lookups.json`, separately from -the table of clients. Up to 100,000 answers are kept; past that, the answer used -longest ago goes first. - -A request waits for its client's first answer only when a setting needs the -answer before the request goes on. It waits up to `SWWAF_LOOKUP_TIMEOUT`. The -settings that make it wait: - -- a country list; -- `SWWAF_ADD_LOOKUP_HEADERS`; -- any setting under "Lower limits by AS number and country" in - [`settings.md`](settings.md). - -Otherwise the request goes on at once, logged without the answer. The answer -reaches the client's history and ban notes when it comes. A client whose answer -has not come in time counts as from an unknown country until it comes, and its -later requests do not wait. While GeoJS fails, new visitors count as from an -unknown country, which `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses. GeoJS is -then left alone for a second, and twice as long after each further failure, up -to five minutes, and one `source_failure` alert is sent per cooldown. - -## The database file - -To keep your visitors' addresses on your own host, set -`SWWAF_LOOKUP_SOURCE=off`, or set it to `file`. `file` looks every client up in -the free IPinfo Lite database (`ipinfo_lite.mmdb`) at `SWWAF_LOOKUP_DB_PATH`, -reading its `asn`, `as_name` and `country_code` fields. You download the -database with an IPinfo account and mount it into the container; `smallwebwaf` -never fetches it and holds no account or token for it. It answers at once, with -nothing sent off the host. A client whose address it does not hold counts as -from an unknown country. - -`smallwebwaf` reads the whole file at start. It reads the file again once it has -gone 2 seconds without a change after you replace it. A replacement it cannot -read is logged and alerted, and the file read before stays in use. Mount the -directory that holds the file, not the file itself. Many editors and download -tools save a file by replacing it, and a container keeps seeing the old file -when only the file is mounted. - -IPinfo updates the database daily, and releases it under the Creative Commons -Attribution-ShareAlike 4.0 licence. It asks for a link to IPinfo on the site -that uses it, such as "IP address data is powered by IPinfo". - -## A client without a country - -A client the lookup cannot place has an unknown country and AS number. That is -one of: - -- a client on a private address; -- one GeoJS or the database does not know; -- one whose GeoJS answer has not come in time. - -`SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses such a client, and the lower -limits give it `SWWAF_UNKNOWN_LIMIT_PERCENT`. Nothing else treats it -differently. diff --git a/docs/reputation.md b/docs/reputation.md deleted file mode 100644 index 80c6318..0000000 --- a/docs/reputation.md +++ /dev/null @@ -1,113 +0,0 @@ -# Reputation sources - -A reputation source judges a client by what others saw it do. None is on by -default. The settings are under "Reputation sources" in -[`settings.md`](settings.md); the reasons for the defaults are under -"Reputation" in [`SPEC.md`](../SPEC.md). The sources are checked in this order, -and a source the client is on refuses it, lowers its limits, or only notes it, -as `SWWAF_BLOCKLIST_ACTION` and `SWWAF_REPUTATION_ACTION` say. - -## Blocklists - -`SWWAF_BLOCKLIST_URLS` lists text files of addresses and netblocks, one to a -line, as the Spamhaus DROP list is written. Anything after a `;` or `#` is left -out. An IPv4-mapped address such as `::ffff:192.0.2.0/120` is read as the IPv4 -netblock it stands for. - -Each list is fetched again `SWWAF_BLOCKLIST_REFRESH` after its last try, one -list at a time. The file of `SWWAF_ASN_LIMIT_PERCENT_URL` is too. -`reputation.json` keeps when each was last tried, so a restart does not fetch a -list again before its time. A fetch fails when: - -- the server answers other than `200`; -- the fetch does not finish within a minute; -- the list is longer than 16 MiB; -- a line is not an address or netblock, or, for the AS number file, not - `AS:`. - -The copy fetched before stays in use, comment lines included. The failure is -logged and raised as a `source_failure` alert. The log, the alerts, the ban -notes and the metrics name each list by its URL, so keep a secret out of it. - -The Spamhaus DROP list is The Spamhaus Project's. Its terms, and what they mean -for `SWWAF_BLOCKLIST_REFRESH` and for several instances on one host, are under -"Reputation" in [`SPEC.md`](../SPEC.md). The copy in `reputation.json` keeps the -list's date and copyright lines. - -## CrowdSec - -While `SWWAF_CROWDSEC_LAPI_URL` is set to the local API of a CrowdSec engine you -run, `smallwebwaf` fetches its decision list every minute. It sends -`SWWAF_CROWDSEC_LAPI_KEY` in the `X-Api-Key` header. The list is kept as a -blocklist is kept. The decisions of the type `ban` on an address or a netblock -are used. Any other, such as a captcha or a country, is left out. A decision -lasts for the `duration` it had left when the engine answered. - -A request is refused when a decision still in force names its client. The -client's netblock is banned until the decision ends, with the cause `crowdsec`. -Only a client that sends a request is banned, so a long list does not fill -`bans.json`. Such a ban is never made permanent, and does not lengthen the -netblock's next ban for a broken limit. If you lift a ban while the decision is -in force, the client's next request makes another. Delete the decision in -CrowdSec first, such as with `cscli decisions delete --ip 203.0.113.9`, and lift -the ban once the list has been fetched again. A fetch fails when: - -- the engine answers other than `200`, a redirect included; -- the fetch does not finish within a minute; -- the answer is longer than 16 MiB; -- the answer is not a list of decisions. - -## DNS blocklists - -Each zone in `SWWAF_DNSBL_ZONES` is asked about a client's own address in the -background, the first time the client is seen. The request goes on at once, and -so does every request from the client until the zone has answered. The name -asked about is the one RFC 5782 gives: the numbers of an IPv4 address reversed, -or the 32 hex digits of an IPv6 address reversed. `192.0.2.99` is asked about as -`99.2.0.192.dnsbl.dronebl.org`. An answer in `127.0.0.0/8` lists the client, but -one in `127.255.255.0/24` is Spamhaus refusing the query. A name that does not -exist does not list the client. These are failures: - -- an answer in `127.255.255.0/24`; -- an answer outside `127.0.0.0/8`; -- a resolver error; -- no answer within `SWWAF_REPUTATION_TIMEOUT`. - -A failure is logged and alerted, and the zone is not asked again for a minute. -Each verdict is kept for `SWWAF_REPUTATION_CACHE_TTL`, in memory and in -`reputation.json`. Up to 100,000 verdicts are kept; past that, the one fetched -longest ago goes first. At most 1,000 queries are under way at once. - -The key of a Spamhaus zone under `dq.spamhaus.net` is its first label. -`smallwebwaf` shows `********` in its place wherever it names the zone. Only -`reputation.json` keeps the zone with its key. - -## AbuseIPDB - -While `SWWAF_ABUSEIPDB_KEY` is set, `smallwebwaf` asks AbuseIPDB's check -endpoint for the abuse confidence score of a client, from 0 to 100. Only a -client that has committed an offence is checked. An offence is one of: - -- a broken limit; -- a `ban` rule match or a trap path; -- a refusal by a `block` rule, the Core Rule Set or a token check. - -The check runs in the background, at the client's first request after the -offence that no ban refuses, so no request waits. The score applies to the -client's later requests. For an IPv6 client, AbuseIPDB is asked about the -address of the request that set off the check, and the score is used for the -whole group. A score at or over `SWWAF_ABUSEIPDB_MIN_SCORE` is a hit, and -`SWWAF_REPUTATION_ACTION` says what is done. Each score is kept for -`SWWAF_REPUTATION_CACHE_TTL`, up to 100,000 scores. - -At most `SWWAF_ABUSEIPDB_DAILY_BUDGET` checks are made in a day, counted in UTC -and kept in `reputation.json`. A restart does not make the budget whole again. -The check that spends the last of the budget is logged and alerted. No client is -checked until the day ends. A check fails when: - -- AbuseIPDB answers other than `200`; -- the answer gives no score; -- no answer comes within `SWWAF_REPUTATION_TIMEOUT`. - -A failure is logged and alerted, and no client is checked for a minute. The key -is sent in the `Key` header and nowhere else, and is never logged. diff --git a/docs/request-log.md b/docs/request-log.md index e0c1922..1891448 100644 --- a/docs/request-log.md +++ b/docs/request-log.md @@ -1,8 +1,8 @@ # The request log's fields -`smallwebwaf` writes one JSON object per line on stdout for every request. The -requirements are under "Request log" in [`SPEC.md`](../SPEC.md); this file lists -the fields. A field that does not apply to a request is left out. +The fields of the JSON object `smallwebwaf` writes for each request. "Request +log" in [`SPEC.md`](../SPEC.md) says when a line is written and how the log is +sent to a syslog server. A field that does not apply to a request is left out. ## Example @@ -10,6 +10,12 @@ the fields. A field that does not apply to a request is left out. {"type":"request","time":"2026-10-03T12:00:00.123Z","instance":"fsn1app1/gitea","client_ip":"203.0.113.9","method":"GET","scheme":"https","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","peer_ip":"172.18.0.2","forwarded_for":"203.0.113.9","client_group":"203.0.113.9/32","asn":"AS64496","as_name":"Example Net","country":"DE","request_headers":{"accept":"*/*"},"response_content_type":"text/html; charset=utf-8","upstream_status":200,"action":"forward","counts":{"minute":1,"hour":12,"day":40,"minute_bytes":5120,"hour_bytes":61440,"day_bytes":204800},"duration_total":3.217,"duration_checks":0.041,"duration_upstream_connect":0.052,"duration_upstream_first_byte":2.874,"duration_upstream_total":3.104} ``` +## Every line + +Every line has `type` (`request`; the process's own messages have `process`), +the standard web log fields, `request_id`, `peer_ip`, `client_group`, `asn`, +`as_name`, `country`, `action` and `duration_total`. + ## Standard web log fields - `time`: when the request arrived, RFC 3339 with milliseconds, in UTC; @@ -66,11 +72,12 @@ the fields. A field that does not apply to a request is left out. - `limit_percent` and `limit_percent_setting`: the client's percentage of the rate limits and the setting that lowered it, when one did. `bytes_percent` and `bytes_percent_setting` do the same for the byte limits. -- `counts`: the client's requests in the minute, hour and day as rate limiting - counts them, this request included, as `minute`, `hour` and `day`; its bytes - as `minute_bytes`, `hour_bytes` and `day_bytes`. A count can have a fraction, - since each window is counted in two buckets with the earlier one weighted. - `counts` is left out when rate limiting did not count the request. +- `counts`: the client's totals as rate limiting counts them, this request + included. It is left out when rate limiting did not count the request. A count + can have a fraction, since each window is counted in two buckets with the + earlier one weighted: + - `minute`, `hour` and `day`: requests; + - `minute_bytes`, `hour_bytes` and `day_bytes`: bytes. - `limit_hit`: the limit broken: `minute`, `hour` or `day` for a rate limit; `minute_bytes`, `hour_bytes` or `day_bytes` for a byte limit; or `error_burst`. A request whose bytes broke a limit was passed on whole, so its @@ -99,30 +106,3 @@ All in milliseconds: connection; - `duration_upstream_first_byte`: until the first byte of the answer; - `duration_upstream_total`: until the end of the answer. - -## Every line - -Every line has `type`, the standard web log fields, `request_id`, `peer_ip`, -`client_group`, `asn`, `as_name`, `country`, `action` and `duration_total`. The -process's own messages share the stream with `"type":"process"`; request lines -carry `"type":"request"`. - -## Sending the log to a syslog server - -While `SWWAF_LOG_REMOTE_URL` is set, every line on stdout is also sent to that -server, as the message of an RFC 5424 syslog record. Over UDP each record is one -datagram. Over TCP and TLS each record follows its length in bytes and a space. -stdout is unchanged. The record carries: - -- the facility from `SWWAF_LOG_REMOTE_FACILITY`; -- the severity informational; -- the time; -- the host's name; -- the app name from `SWWAF_LOG_REMOTE_APP_NAME`. - -Lines wait in a buffer of `SWWAF_LOG_REMOTE_BUFFER` lines, so a server that is -slow or down never holds up a request. When the buffer is full, the oldest line -is dropped. A failed connection is tried again a second later, and twice as long -after each further failure, up to a minute. A line too long for one UDP datagram -is dropped alone. As `smallwebwaf` stops, it spends at most two seconds sending -the lines still waiting, and drops the rest. stdout has carried them. diff --git a/docs/rules.md b/docs/rules.md index 8af131b..1bd3fd3 100644 --- a/docs/rules.md +++ b/docs/rules.md @@ -1,10 +1,8 @@ # Rule files -`smallwebwaf` reads every `*.rules` file in `SWWAF_RULES_DIR`, in the order of -their names, and checks each request against their rules in that order, unless -the client is in `SWWAF_ALLOW_NETS`. The requirements the files meet are under -"Rule files" in [`SPEC.md`](../SPEC.md). A file whose name starts with `.`, such -as an editor's lock file, is not a rule file. +The format of a rule file, and the default file the image ships. "Rule files" in +[`SPEC.md`](../SPEC.md) says how the files are read, watched and mounted, and +what each action leads to. ## The format @@ -18,7 +16,7 @@ env-file path ban (?i)^/\.env(\.[a-z]+)?$ scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|wpscan)\b ``` -- The id is letters, digits, `-` and `_`. No two rules share one. The log line, +- The id is letters, digits, `-` and `_`, unique across all files. The log line, the metrics, the alerts and a ban's notes name the rule by it. - The target is what the regex is matched against: - `path`: the path without the query; @@ -26,42 +24,10 @@ scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|wpscan)\b - `query`: the query as sent; - `method`, `host`, `user_agent` and `referer`; - `header:`: any one request header but `Host` and - `Transfer-Encoding`, which Go's HTTP server takes out of every request. - - `path` and `uri` are matched both as sent and once percent-decoded, so that - an encoded probe does not slip past. A header sent more than once is matched - with its values joined by `, `. A header not sent is empty text. No body is - read; reading bodies is the Core Rule Set's job, once `SWWAF_WAF_BODY_LIMIT` - switches it on. - -- The action is `log`, `block` or `ban`. `log` only notes the match in the log - line. `block` refuses the request with `403` and bans no one; the refusal - counts toward the error burst. `ban` refuses it and bans the client, as a - clear sign of attack. Matching stops at the first rule that refuses. Keep - `ban` for requests no real visitor sends. Anchor a path at the site root with - `^/`: a file of the same name deeper in a site can be ordinary content. -- The regex is in Go's syntax, RE2. It has no backreferences or lookaround, and - takes time linear in the text it reads, so no rule can be made to stall the - proxy. It matches anywhere in the target unless anchored with `^` and `$`. - `(?i)` at its front makes it ignore case. - -## Errors - -These stop the start, with a message naming the file and line: - -- a line that is not a rule; -- a bad `header:` name, or a rule for `Host` or `Transfer-Encoding`; -- a regex that does not compile; -- an id used twice. - -So does a `SWWAF_RULES_DIR` that does not exist. An empty directory is not an -error: the log says that no rules were loaded. - -While it runs, `smallwebwaf` watches the directory. It reads the files again -once the directory has had no change for 2 seconds, so that a file still being -written is read whole. If the files then hold such an error, the rules stay as -they were. The log and a `file_error` alert name the file and line. Once the -file is fixed, it is read again. + `Transfer-Encoding`. +- The action is `log`, `block` or `ban`. +- The regex is in Go's syntax, RE2. It matches anywhere in the target unless + anchored with `^` and `$`. `(?i)` at its front makes it ignore case. ## The default file @@ -86,13 +52,10 @@ path-traversal uri block (\.\./){2,} empty-agent user_agent log ^$ ``` -It is kept short and limited to patterns that are wrong for every app, with its -path rules anchored at the site root. In front of gitea, `/.env` is a probe, -while `///src/branch/main/.env.example` is a file in a repository -that any visitor may open. Anything app-specific belongs in a file the app's -`Dockerfile` adds or the deployer mounts. A request for `/wp-login.php`, for -example, is a clear sign of attack in front of gitea and an ordinary login in -front of WordPress. Such a rule goes in a file named to sort after the default: +## A file of your own + +An app's `Dockerfile` adds rules of its own in a file named to sort after the +default, such as `50-gitea.rules`: ``` # 50-gitea.rules: WordPress probes, which gitea never serves @@ -102,13 +65,3 @@ wp-probe path ban (?i)^/(wp-login\.php|xmlrpc\.php|wp-admin/) ```dockerfile COPY 50-gitea.rules /etc/smallwebwaf/rules.d/50-gitea.rules ``` - -## Mounting rule files - -A directory mounted over `/etc/smallwebwaf/rules.d` replaces the default file; -single files mounted into it add to it. Many editors save a file by replacing -it, and a container keeps seeing the old one when the file alone is mounted. So -rules to be edited while `smallwebwaf` runs belong in a mounted directory, with -a copy of the default file if the defaults are to stay. The lookup database is -mounted the same way, for the same reason. To run without rules, mount an empty -directory or set `SWWAF_RULES_ENABLED=false`. diff --git a/docs/settings.md b/docs/settings.md index d9088ba..55990fa 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -1,27 +1,19 @@ # Settings -Every setting is an environment variable whose name starts with `SWWAF_`. Every -setting has a default, so none has to be set. Settings are read once, at start. -To change one, restart the container. +Every setting, its default and the values it takes, in the groups of "Settings" +in [`SPEC.md`](../SPEC.md). That section gives the conventions, the settings +that take `off`, the `_FILE` form, and what each group does and why. A setting +set to a value its entry does not allow stops the start. -A setting that is set to a value it cannot take stops the start. The process -writes one line, `"type":"process"` with `"msg":"invalid setting"`, naming the -variable and what is wrong with the value. It exits with status 1, and runit -starts it again every second until the value is fixed. The entries below say -which values each setting takes. Any other value stops the start. At start, the -settings in effect are logged on one line. Tokens, keys and the paths of -webhooks are shown as `********`. - -The forms of values: +## Forms of values - A duration is Go's syntax, with `d` for days: `90s`, `15m`, `24h`, `7d`. Zero and negative durations are refused. - A size is a number of bytes with an optional `K`, `M` or `G`, which are powers - of 1024: `1K` is 1024 bytes, `512K`, `100M`, `5G`. + of 1024, so `1K` is 1024 bytes. Examples: `512K`, `100M`, `5G`. - A count is a whole number of requests, such as `1000`. -- `off` switches a setting off, where its entry below says it takes `off`. - A list is comma separated, with spaces around items ignored. An empty item is - refused. A list set but empty is an empty list, and replaces the default. + refused. - A netblock is an address and a prefix length, such as `10.0.0.0/8`. A bare address stands for itself alone. - A country is a two-letter code of ISO 3166-1, in either case: `de` and `DE` @@ -31,337 +23,58 @@ The forms of values: - A percentage is a whole number from 0 to 100. - A path prefix starts with `/`, such as `/assets/`. -## Where it listens and what it protects +## Core -- `SWWAF_LISTEN_ADDR` (default `:8080`): where `smallwebwaf` listens, an - optional host and a port. Keep the host empty: `:9000`, never - `127.0.0.1:9000`. The app must not listen on that port. The health check - follows the setting, and traefik's label must name the port. - `SWWAF_UPSTREAM_URL` (default `http://127.0.0.1:8081`): the app. An `http` or - `https` URL with a host and an optional port, and nothing more: no path, user - or query. -- `SWWAF_INSTANCE_NAME` (default: the container's host name, the first 12 - characters of the container's id): the `instance` name in every log line, - alert and metric. Set it to a name that survives deploys and tells instances - apart, such as `fsn1app1/gitea`. It must be valid UTF-8. While - `SWWAF_ALERT_NTFY_URL` is set, it may hold no control character. While - `SWWAF_LOG_REMOTE_URL` is set and `SWWAF_LOG_REMOTE_APP_NAME` is not, it must - be 1 to 48 printable ASCII characters without a space. -- `SWWAF_MODE` (default `enforce`): `enforce`, or `observe` to ban no one and - log what `enforce` would have done. Observe mode still refuses a request over - a size or time limit, and one without the right token. -- `SWWAF_TRUSTED_PROXIES` (default `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`, - the private address ranges): the netblocks whose `X-Forwarded-For` is - believed, normally where traefik comes from. Set but empty, it trusts nothing: - every TCP peer is then the client. -- `SWWAF_IPV6_GROUP_PREFIX` (default `64`): the length of the netblock that is - one IPv6 client, from 32 to 128. After a change, each IPv6 client starts - afresh. Existing bans keep refusing their netblocks until they end. -- `SWWAF_MAX_TRACKED_CLIENTS` (default `20000`): the most clients kept in memory - and in `clients.json`, a whole number above zero. Past it, the least recently - seen client is dropped first. - -## Time and size limits - -Each request has limits on both sides: between the client and `smallwebwaf`, and -between `smallwebwaf` and the app. Bodies stream through in both directions, -except the part the Core Rule Set reads while `SWWAF_WAF_BODY_LIMIT` is set. - -- `SWWAF_CLIENT_REQUEST_TIMEOUT` (default `60s`): how long a client may take to - send its request line and headers, and then, from the end of the headers, its - body. A duration or `off`. -- `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES` (default `32K`): the largest request - line and headers. A size over `4K`. It cannot be `off`: Go's HTTP server - always has such a limit. -- `SWWAF_CLIENT_IDLE_TIMEOUT` (default `120s`): how long a kept-open connection - may wait for its next request before it is closed. A duration or `off`. The - default is longer than the 90 seconds after which traefik closes an unused - connection, so traefik closes first. -- `SWWAF_CLIENT_RESPONSE_TIMEOUT` (default `30m`): how long the answer may take - to reach the client, from the end of the request to the last byte. A duration - or `off`. -- `SWWAF_UPSTREAM_REQUEST_TIMEOUT` (default `60s`): how long connecting to the - app and sending it the whole request may take. A duration or `off`. -- `SWWAF_UPSTREAM_RESPONSE_TIMEOUT` (default `30m`): how long the app may take - to send its whole answer. A duration or `off`. -- `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body. A size - or `off`. A request that announces a larger body is refused before anything - reaches the app. -- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body. A size - or `off`. - -In front of a code forge, a clone is a response and a push is a request. At the -defaults, an upload over 100 MiB or slower than 60 seconds is cut off. A forge -that takes large uploads needs `SWWAF_REQUEST_MAX_BYTES`, -`SWWAF_CLIENT_REQUEST_TIMEOUT` and `SWWAF_UPSTREAM_REQUEST_TIMEOUT` raised to -fit. - -What the client gets when a limit is passed is under "Size and time limits" in -[`SPEC.md`](../SPEC.md). - -## Netblock lists - -- `SWWAF_ALLOW_NETS` (default empty): netblocks whose clients skip every check, - such as your monitoring or your own network. The size and time limits still - apply to them. They are not looked up. They still need a token for the admin - endpoints. -- `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients are - exempt from rate limiting and the byte limits, such as a machine that talks to - the app all day. Everything else still applies to them. -- `SWWAF_DENY_NETS` (default empty): netblocks whose clients are always refused. - A client in both lists is let through. - -## Rate limiting and byte limits - -- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `1000`), `SWWAF_RATE_LIMIT_PER_HOUR` - (default `10000`) and `SWWAF_RATE_LIMIT_PER_DAY` (default `50000`): the most - requests a client may make in each window. A count or `off`. The defaults are - several times what one busy person produces. A browser loading a heavy page - makes a few hundred requests, and several people often share one address. -- `SWWAF_RATE_LIMIT_EXEMPT_PATHS` (default empty): path prefixes exempt from - rate limiting, such as `/assets/`. Their bytes are not counted either. The - path is matched exactly as sent, without the query, and is not - percent-decoded. `/assets/` matches `/assets/app.js`. It does not match - `/Assets/app.js`, `/%61ssets/app.js` or `/static/assets/app.js`. There are no - wildcards. A path holding `..`, a backslash or an encoded slash (`%2F`) is - never exempt, since the app may read it as a path outside the prefix. -- `SWWAF_BYTES_LIMIT_PER_MINUTE` (default `10G`), `SWWAF_BYTES_LIMIT_PER_HOUR` - (default `20G`) and `SWWAF_BYTES_LIMIT_PER_DAY` (default `50G`): the most - bytes a client may have counted in each window. A size or `off`. Bytes are - counted once the answer has ended, so a byte limit never cuts an answer short. - The client's next request is refused instead. Each default is above the - largest request and answer together, so at the defaults no single download - breaks a limit. -- `SWWAF_BYTES_COUNT` (default `both`): which body bytes the byte limits count: - `response`, `request` or `both`. For a WebSocket, what it carried each way - counts once it closes. - -## Looking up the AS number and country - -- `SWWAF_LOOKUP_SOURCE` (default `geojs`): `geojs`, `file` or `off`. `geojs` - asks the GeoJS web service, which is then told the address of every new - visitor. `file` looks clients up in the IPinfo Lite database file at - `SWWAF_LOOKUP_DB_PATH`, and sends nothing off the host. `off` looks up no - client. With `off`, a setting that needs lookups stops the start: - - a country list; - - `SWWAF_ADD_LOOKUP_HEADERS`; - - an entry in `SWWAF_ASN_LIMIT_PERCENT`, `SWWAF_COUNTRY_LIMIT_PERCENT`, - `SWWAF_ASN_BYTES_PERCENT` or `SWWAF_COUNTRY_BYTES_PERCENT`; - - `SWWAF_UNKNOWN_LIMIT_PERCENT` below 100; - - `SWWAF_ASN_LIMIT_PERCENT_URL`; - - an anomaly threshold per AS number. -- `SWWAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in its - `.mmdb` form, for `SWWAF_LOOKUP_SOURCE=file`. `file` without it stops the - start, and so does setting it with any other source. A file that is missing or - cannot be read stops the start too. -- `SWWAF_LOOKUP_TIMEOUT` (default `1s`): how long a request waits for its - client's first answer from GeoJS when a setting needs it, and how long a - request to GeoJS may take. A duration; it cannot be `off`. -- `SWWAF_ADD_LOOKUP_HEADERS` (default `false`): `true` passes the app the - client's AS number in `X-Client-ASN` and its country in `X-Client-Country`. A - header whose value is unknown is left out. Whatever this setting says, any - such header the client sent is removed. - -See [`lookup.md`](lookup.md) for what each source sends where, and how long -answers are kept. - -## Country lists - -- `SWWAF_DENIED_COUNTRIES` (default empty): countries whose clients are refused, - such as `cn,ru,kp`. -- `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` (default empty): when set, the only - countries whose clients get through, such as `us,de`. A client whose country - cannot be found is refused too. Otherwise every new client would be let in - whenever GeoJS stops answering. A client on a private address has no country, - so it is refused unless it is in `SWWAF_ALLOW_NETS`. - -A country on both lists stops the start. What a refused request gets, and why -the lists are empty by default, is under "Country lists" in -[`SPEC.md`](../SPEC.md). - -## Lower limits by AS number and country - -Clients of the AS numbers and countries listed here get a percentage of every -rate limit and byte limit. A lowered limit is rounded down. "Biased thresholds" -in [`SPEC.md`](../SPEC.md) says why the lists are empty by default and why no -budget is shared by a whole AS number or country. - -- `SWWAF_ASN_LIMIT_PERCENT` (default empty): a list of AS numbers, each with `:` - and a percentage, such as `AS14061:50,AS16276:50,AS45102:25`. An AS number - listed twice is refused. -- `SWWAF_COUNTRY_LIMIT_PERCENT` (default empty): the same by country, such as - `cn:25,ru:50`. -- `SWWAF_ASN_BYTES_PERCENT` and `SWWAF_COUNTRY_BYTES_PERCENT` (default empty): - the same for the byte limits alone. For the AS numbers and countries they - list, they take the place of the two settings above. -- `SWWAF_UNKNOWN_LIMIT_PERCENT` (default `100`): the percentage of every limit - for a client without a country: - - one the lookup cannot place; - - one on a private address; - - one whose answer from GeoJS has not come in time. -- `SWWAF_ASN_LIMIT_PERCENT_URL` (default unset): the `http` or `https` URL of a - text file of such entries, one to a line, such as `AS14061:50`. One list can - then serve several instances. It is fetched and kept as a blocklist is. Where - it and `SWWAF_ASN_LIMIT_PERCENT` both give a percentage, the lower applies. A - URL that is also in `SWWAF_BLOCKLIST_URLS` stops the start. - -A client to which several percentages apply gets the lowest. The log line of -each counted request, and the notes of each ban, say which setting lowered the -limit. - -## Reputation sources - -None is on by default. A reputation source judges a client by what others saw it -do. The defaults judge it by what it does to your service. - -- `SWWAF_BLOCKLIST_URLS` (default empty): blocklists to fetch, such as - `https://www.spamhaus.org/drop/drop.txt`. Each is an `http` or `https` URL - without a user or a fragment. A URL listed twice is refused. -- `SWWAF_BLOCKLIST_REFRESH` (default `24h`): how long a list waits after its - last fetch, or failed try, before it is fetched again. A duration of `1h` or - more; it cannot be `off`. The Spamhaus lists may be fetched no more than once - an hour. -- `SWWAF_BLOCKLIST_ACTION` (default `deny`): what is done with a client on a - blocklist. `deny` refuses its requests. `limit:`, such as `limit:25`, - gives it that percentage of every limit. `log` only notes the list in the log - line and raises an alert. -- `SWWAF_DNSBL_ZONES` (default empty): the DNS blocklists to ask about each - client, such as `dnsbl.dronebl.org`. A DNS blocklist answers a DNS query for a - name made from an address, and so says whether the address is on it. A zone is - a DNS name of at most 189 characters. A Spamhaus zone carries its key, as in - `.xbl.dq.spamhaus.net`, and is shown with the key masked. A zone listed - twice is refused, whatever its case or key. Do not name a zone meant for mail: - such a zone holds the residential addresses ordinary visitors come from. -- `SWWAF_DNSBL_RESOLVER` (default unset): the resolver to ask the zones through, - such as `192.0.2.53` or `[2001:db8::53]:5353`. An IP address with an optional - port, 53 when none is given. Unset, the host's resolver is used. Several zones - refuse queries that come through a public resolver. -- `SWWAF_ABUSEIPDB_KEY` (default unset): the key of your AbuseIPDB account. - While it is set, a client that has committed an offence is checked. A control - character in it is refused. -- `SWWAF_ABUSEIPDB_MIN_SCORE` (default `75`): the least abuse confidence score, - from 0 to 100, that is a hit. -- `SWWAF_ABUSEIPDB_DAILY_BUDGET` (default `900`): the most checks made in a day, - in UTC, a whole number above zero. AbuseIPDB's free accounts may make 1,000. -- `SWWAF_CROWDSEC_LAPI_URL` (default unset): the local API of a CrowdSec engine - you run, such as `http://172.17.0.1:8080`. An `http` or `https` URL without a - user or a fragment. Its decision list is at `v1/decisions` under that URL, and - is fetched every minute. Set without the key, it stops the start. So does a - decision list URL that is also in `SWWAF_BLOCKLIST_URLS`, or that is - `SWWAF_ASN_LIMIT_PERCENT_URL`. -- `SWWAF_CROWDSEC_LAPI_KEY` (default unset): the key the engine gave for - `smallwebwaf`, as `cscli bouncers add smallwebwaf` prints it. Set without the - URL, it stops the start. -- `SWWAF_REPUTATION_ACTION` (default `limit:25`): what is done with a client on - a DNS blocklist, or one whose AbuseIPDB score is a hit. `deny`, - `limit:` or `log`, as for `SWWAF_BLOCKLIST_ACTION`. Such a verdict is - less certain than the Spamhaus DROP list, so by default the client gets a - quarter of every limit. -- `SWWAF_REPUTATION_CACHE_TTL` (default `24h`): how long a zone's verdict or - AbuseIPDB's score is used. A duration; it cannot be `off`. -- `SWWAF_REPUTATION_TIMEOUT` (default `2s`): how long a query to a zone or a - check with AbuseIPDB may take. A duration; it cannot be `off`. - -See [`reputation.md`](reputation.md) for how each source is asked and what it -does with the answer. - -## Attack detection - -- `SWWAF_RULES_DIR` (default `/etc/smallwebwaf/rules.d`): the directory of the - rule files (see [`rules.md`](rules.md)). A directory that cannot be read stops - the start. An empty one is no error. -- `SWWAF_RULES_ENABLED` (default `true`): `false` reads no rule file. -- `SWWAF_TRAP_PATHS` (default empty): paths the app never serves and only - scanners ask for, such as `/wp-login.php,/xmlrpc.php` in front of a code - forge. A request for one bans the client, as a `ban` rule does. Each starts - with `/` and holds no `?`. A trap path is matched whole, without the query, - against the path as sent and once percent-decoded. `/wp-login.php` matches - `/wp-login.php?x=1` and `/%77p-login.php`. It does not match `/wp-login.php/` - or `/blog/wp-login.php`. -- `SWWAF_WAF_MODE` (default `block`): what the Core Rule Set does with a request - it takes for an attack. `block` refuses it with `403`. `detect` lets it - through, logged and alerted. `off` inspects no request. -- `SWWAF_WAF_PARANOIA_LEVEL` (default `1`): the Core Rule Set's paranoia level, - from 1 to 4. Each level up runs more rules. Those find more attacks, and take - more ordinary requests for attacks. -- `SWWAF_WAF_ANOMALY_THRESHOLD` (default `5`): the score at which a request is a - match. Each rule that matches adds to the request's score, and a rule rated - critical adds 5. So by default one such rule is enough. A whole number above - zero, or `off`. With `off` no request is a match, and scores are still logged. -- `SWWAF_WAF_DISABLED_RULES` (default - `920340,920420,920440,920640,930130,930140`): the ids of Core Rule Set rules - to switch off. The log line names the rules a request matched, so a rule that - refuses your app's ordinary requests can be found there. "Attack detection" in - [`SPEC.md`](../SPEC.md) says what the six default ids refuse and why they are - off. A list you give replaces the default, so name these six ids in it to keep - them off. An id must be a whole number above zero. An id from 900000 to 900999 - is refused: those are the rules that set the Core Rule Set up, and - `smallwebwaf`'s own. -- `SWWAF_WAF_EXEMPT_PATHS` (default empty): path prefixes the Core Rule Set does - not inspect, matched as `SWWAF_RATE_LIMIT_EXEMPT_PATHS` is. -- `SWWAF_WAF_BODY_LIMIT` (default `off`): `off` reads no request body. A size, - at most `1G`, switches body inspection on. The Core Rule Set then reads a form - data or multipart body up to that size, and the rest streams on to the app. It - reads a JSON or XML body no larger than that size, since those cannot be read - in part. Any other body, such as a git push, reaches the app uninspected. See - [`core-rule-set.md`](core-rule-set.md). -- `SWWAF_ERROR_BURST_THRESHOLD` (default `30`): the most refusals of one client - in a minute for a rule match, a trap path, a Core Rule Set match or a failed - token check. One more bans the client, as a broken rate limit does. A count or - `off`. The app's own `401` and `404` answers are not counted. - -## How long bans last and what they cover - -- `SWWAF_BAN_RESPONSE` (default `403`): how a refused client is answered: `403`, - `429`, or `close` to close the connection without an answer. A `block` rule - and the Core Rule Set always answer `403`. -- `SWWAF_LIMIT_BAN_DURATION` (default `1h`): the ban for a first broken rate - limit, byte limit or error burst. A duration; it cannot be `off`. -- `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): a client that breaks a limit - again within this time after its ban ended is banned three times as long. A - duration; it cannot be `off`. -- `SWWAF_MAX_BAN_DURATION` (default `7d`): a ban for a broken limit that would - be longer is permanent instead. A duration; it cannot be `off`. -- `SWWAF_ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of - attack. A duration; it cannot be `off`. + `https` URL with a host and an optional port, and nothing more. +- `SWWAF_LISTEN_ADDR` (default `:8080`): where `smallwebwaf` listens, an + optional host and a port. The host stays empty. +- `SWWAF_INSTANCE_NAME` (default: the container's host name): the `instance` + name in every log line, alert and metric, such as `fsn1app1/gitea`. Valid + UTF-8. No control character while `SWWAF_ALERT_NTFY_URL` is set. 1 to 48 + printable ASCII characters without a space while `SWWAF_LOG_REMOTE_URL` is set + and `SWWAF_LOG_REMOTE_APP_NAME` is not. +- `SWWAF_MODE` (default `enforce`): `enforce` or `observe`. +- `SWWAF_TRUSTED_PROXIES` (default `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`): + the netblocks whose `X-Forwarded-For` is believed. Set but empty, it trusts + nothing. +- `SWWAF_IPV6_GROUP_PREFIX` (default `64`): the prefix length of an IPv6 + client's group, from 32 to 128. +- `SWWAF_MAX_TRACKED_CLIENTS` (default `20000`): the most clients kept, a whole + number above zero. - `SWWAF_MAX_BANS` (default `5000`): the most bans `smallwebwaf` made that are - kept, whether past, active or permanent. A whole number above zero. Bans whose - cause is `admin` are kept besides and never dropped (see - [`bans.md`](bans.md)). -- `SWWAF_BAN_SCOPE_V4_PREFIX` (default `32`): the length of the netblock around - an IPv4 client that a ban covers, from 0 to 32. `24` bans the surrounding /24. - An IPv6 ban covers the client's group. + kept, a whole number above zero. -## Where and how often the state files are written +## Tokens + +- `SWWAF_ADMIN_TOKEN` (default unset): the token for the ban endpoints and + `/_smallwebwaf/clients/`. At least 32 characters. +- `SWWAF_METRICS_TOKEN` (default unset): the token for `/_smallwebwaf/metrics`. + At least 32 characters. + +## State files - `SWWAF_STATE_DIR` (default `/var/lib/smallwebwaf`): the directory of the state - files, an absolute path. A directory that does not exist or cannot be written - stops the start. + files, an absolute path that exists and can be written. - `SWWAF_STATE_WRITE_DELAY` (default `10s`): how long `bans.json` waits to be - written after a ban is made, lifted or made permanent. Every change in between - goes into the same write. A duration; it cannot be `off`. + written after a ban is made, lifted or made permanent. A duration. - `SWWAF_STATE_COUNTER_INTERVAL` (default `15m`): how often every state file is - written. A duration; it cannot be `off`. + written. A duration. ## Logging +- `SWWAF_LOG_LEVEL` (default `info`): the least severe of `smallwebwaf`'s own + messages that are written: `debug`, `info`, `warn` or `error`. - `SWWAF_LOG_REQUEST_HEADERS` (default `accept,accept-language,accept-encoding,content-type,origin,range`): the - request headers to record in the log line, in either case. `Authorization`, - `Cookie` and `Set-Cookie` are never logged, even if listed. `Host` and - `Transfer-Encoding` are refused: Go's HTTP server takes them out of the - request. The host is the field `host`. -- `SWWAF_LOG_LEVEL` (default `info`): the least severe of `smallwebwaf`'s own - messages that are written: `debug`, `info`, `warn` or `error`. No message is - at `debug`. It holds back no request log line. `warn` and `error` hold back - the line of settings at start. -- `SWWAF_LOG_REMOTE_URL` (default unset): a syslog server every line on stdout - is also sent to, such as `syslog+tls://logs.example:6514`. The forms are - `syslog+udp://`, `syslog+tcp://` and `syslog+tls://`, each with a host and a - port and nothing more. + request headers recorded in the log line, in either case. `Host` and + `Transfer-Encoding` are refused; the host is the field `host`. +- `SWWAF_LOG_REMOTE_URL` (default unset): a syslog server that also gets every + line written to stdout: `syslog+udp://`, `syslog+tcp://` or `syslog+tls://`, + each with a host and a port and nothing more. - `SWWAF_LOG_REMOTE_TLS_CA_FILE` (default unset): a file of PEM certificates that a `syslog+tls` server's certificate must chain to, instead of the host's - own. A file that cannot be read, or holds no certificate, stops the start. + own. A file that cannot be read, or holds no certificate, is refused. - `SWWAF_LOG_REMOTE_BUFFER` (default `10000`): the most lines held while they wait to be sent, a whole number above zero. - `SWWAF_LOG_REMOTE_FACILITY` (default `local0`): the syslog facility: `kern`, @@ -370,53 +83,193 @@ does with the answer. - `SWWAF_LOG_REMOTE_APP_NAME` (default: `SWWAF_INSTANCE_NAME`): the app name in each syslog record, 1 to 48 printable ASCII characters without a space. -## Tokens and metrics +## Metrics -- `SWWAF_ADMIN_TOKEN` (default unset): the token for the ban endpoints and - `/_smallwebwaf/clients/`. A long random value of at least 32 characters. - Unset, those endpoints are off. Give it as a file to keep it from the app (see - "Settings given as files" below). -- `SWWAF_METRICS_TOKEN` (default unset): the token a scraper sends for - `/_smallwebwaf/metrics`, at least 32 characters. Unset, the metrics are off. - It is separate from the admin token, so a scraper cannot manage bans. - `SWWAF_METRICS_TOP_N` (default `50`): how many AS numbers and how many - countries get series of their own in the metrics. The rest count as `other`. A - whole number above zero. + countries get series of their own. A whole number above zero. -## Alert destinations +## Static lists + +- `SWWAF_ALLOW_NETS` (default empty): netblocks whose clients are refused only + for a size or time limit or a missing or wrong token, such as your monitoring. +- `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients are + exempt from rate limiting and the byte limits. +- `SWWAF_DENY_NETS` (default empty): netblocks whose clients are always refused. + +## Request rate limits, per client + +- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `1000`), `SWWAF_RATE_LIMIT_PER_HOUR` + (default `10000`) and `SWWAF_RATE_LIMIT_PER_DAY` (default `50000`): the most + requests a client may make in each window. A count or `off`. +- `SWWAF_RATE_LIMIT_EXEMPT_PATHS` (default empty): path prefixes exempt from + rate limiting and the byte limits, such as `/assets/`. + +## Byte limits, per client + +- `SWWAF_BYTES_LIMIT_PER_MINUTE` (default `10G`), `SWWAF_BYTES_LIMIT_PER_HOUR` + (default `20G`) and `SWWAF_BYTES_LIMIT_PER_DAY` (default `50G`): the most + bytes counted for a client in each window. A size or `off`. +- `SWWAF_BYTES_COUNT` (default `both`): which body bytes the byte limits count: + `response`, `request` or `both`. + +## Size and time limits + +- `SWWAF_CLIENT_REQUEST_TIMEOUT` (default `60s`): how long a client may take to + send its request line and headers, and then, from the end of the headers, its + body. A duration or `off`. +- `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES` (default `32K`): the largest request + line and headers. A size over `4K`; never `off`. +- `SWWAF_CLIENT_IDLE_TIMEOUT` (default `120s`): how long a kept-open connection + may wait for its next request before it is closed. A duration or `off`. +- `SWWAF_CLIENT_RESPONSE_TIMEOUT` (default `30m`): how long the answer may take + to reach the client, from the end of the request to the last byte. A duration + or `off`. +- `SWWAF_UPSTREAM_REQUEST_TIMEOUT` (default `60s`): how long connecting to the + app and sending it the whole request may take. A duration or `off`. +- `SWWAF_UPSTREAM_RESPONSE_TIMEOUT` (default `30m`): how long the app may take + to send its whole answer. A duration or `off`. +- `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body. A size + or `off`. +- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body. A size + or `off`. + +## Lookup of AS number and country + +- `SWWAF_LOOKUP_SOURCE` (default `geojs`): `geojs`, `file` or `off`. +- `SWWAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in its + `.mmdb` form, for `SWWAF_LOOKUP_SOURCE=file`. +- `SWWAF_LOOKUP_TIMEOUT` (default `1s`): how long a request waits for its + client's first answer from GeoJS, and how long a request to GeoJS may take. A + duration; never `off`. +- `SWWAF_ADD_LOOKUP_HEADERS` (default `false`): `true` passes the app the + client's AS number in `X-Client-ASN` and its country in `X-Client-Country`. + +## Country lists + +- `SWWAF_DENIED_COUNTRIES` (default empty): countries whose clients are refused, + such as `cn,ru,kp`. +- `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` (default empty): when set, the only + countries whose clients get through, such as `us,de`. A country on both lists + is refused. + +## Biased thresholds + +- `SWWAF_ASN_LIMIT_PERCENT` (default empty): a list of AS numbers, each with `:` + and a percentage, such as `AS14061:50,AS16276:50,AS45102:25`. An AS number + listed twice is refused. +- `SWWAF_COUNTRY_LIMIT_PERCENT` (default empty): the same by country, such as + `cn:25,ru:50`. +- `SWWAF_ASN_BYTES_PERCENT` and `SWWAF_COUNTRY_BYTES_PERCENT` (default empty): + the same, for the byte limits alone. +- `SWWAF_UNKNOWN_LIMIT_PERCENT` (default `100`): the percentage for a client the + lookup cannot place. +- `SWWAF_ASN_LIMIT_PERCENT_URL` (default unset): the `http` or `https` URL of a + text file of `AS:` lines, one to a line. A URL that is also + in `SWWAF_BLOCKLIST_URLS` is refused. + +## Attack detection + +- `SWWAF_RULES_DIR` (default `/etc/smallwebwaf/rules.d`): the directory of the + rule files. +- `SWWAF_RULES_ENABLED` (default `true`): `false` reads no rule file. +- `SWWAF_TRAP_PATHS` (default empty): paths only scanners ask for, such as + `/wp-login.php,/xmlrpc.php`. Each starts with `/` and holds no `?`. +- `SWWAF_WAF_MODE` (default `block`): `off`, `detect` or `block`. +- `SWWAF_WAF_PARANOIA_LEVEL` (default `1`): the Core Rule Set's paranoia level, + from 1 to 4. +- `SWWAF_WAF_ANOMALY_THRESHOLD` (default `5`): the score at which a request is a + match. A whole number above zero, or `off`. +- `SWWAF_WAF_DISABLED_RULES` (default + `920340,920420,920440,920640,930130,930140`): the ids of Core Rule Set rules + to switch off. Each a whole number above zero. An id from 900000 to 900999 is + refused: those are the rules that set the Core Rule Set up, and + `smallwebwaf`'s own. +- `SWWAF_WAF_EXEMPT_PATHS` (default empty): path prefixes the Core Rule Set does + not inspect. +- `SWWAF_WAF_BODY_LIMIT` (default `off`): `off`, or how much of a request body + the Core Rule Set reads, a size of at most `1G`. +- `SWWAF_ERROR_BURST_THRESHOLD` (default `30`): the most refusals of one client + in a minute for a rule match, a trap path, a Core Rule Set match or a failed + token check. A count or `off`. + +## Ban lengths, response and scope + +- `SWWAF_BAN_RESPONSE` (default `403`): `403`, `429` or `close`. +- `SWWAF_LIMIT_BAN_DURATION` (default `1h`): the ban for a first broken limit. A + duration. +- `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): the time after a ban ends + within which breaking a limit again lengthens the next ban. A duration. +- `SWWAF_MAX_BAN_DURATION` (default `7d`): the longest ban for a broken limit; + one that would be longer is permanent. A duration. +- `SWWAF_ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of + attack. A duration. +- `SWWAF_BAN_SCOPE_V4_PREFIX` (default `32`): how much a ban on an IPv4 client + covers: the prefix length of its netblock, from 0 to 32. + +## Reputation + +- `SWWAF_BLOCKLIST_URLS` (default empty): blocklists to fetch, such as + `https://www.spamhaus.org/drop/drop.txt`. Each an `http` or `https` URL + without a user or a fragment. A URL listed twice is refused. +- `SWWAF_BLOCKLIST_REFRESH` (default `24h`): how long a list waits after its + last try before it is fetched again. A duration of `1h` or more. +- `SWWAF_BLOCKLIST_ACTION` (default `deny`): `deny`, `limit:` or `log`. +- `SWWAF_DNSBL_ZONES` (default empty): the DNS blocklists to ask, such as + `dnsbl.dronebl.org`. Each a DNS name of at most 189 characters. A Spamhaus + zone carries its key, as in `.xbl.dq.spamhaus.net`. A zone listed twice + is refused, whatever its case or key. +- `SWWAF_DNSBL_RESOLVER` (default unset): the resolver to ask the zones through, + such as `192.0.2.53` or `[2001:db8::53]:5353`. An IP address with an optional + port, 53 when none is given. Unset, the host's resolver is used. +- `SWWAF_ABUSEIPDB_KEY` (default unset): the key of your AbuseIPDB account. A + control character in it is refused. +- `SWWAF_ABUSEIPDB_MIN_SCORE` (default `75`): the least abuse confidence score, + from 0 to 100, that is a hit. +- `SWWAF_ABUSEIPDB_DAILY_BUDGET` (default `900`): the most checks made in a day, + a whole number above zero. AbuseIPDB's free accounts may make 1,000. +- `SWWAF_CROWDSEC_LAPI_URL` (default unset): the local API of a CrowdSec engine + you run, such as `http://172.17.0.1:8080`. An `http` or `https` URL without a + user or a fragment; its decision list is at `v1/decisions` under it. Set + without the key, it is refused. So is a decision list URL that is also in + `SWWAF_BLOCKLIST_URLS`, or that is `SWWAF_ASN_LIMIT_PERCENT_URL`. +- `SWWAF_CROWDSEC_LAPI_KEY` (default unset): the key the engine gave for + `smallwebwaf`, as `cscli bouncers add smallwebwaf` prints it. Set without the + URL, it is refused. +- `SWWAF_REPUTATION_ACTION` (default `limit:25`): `deny`, `limit:` or + `log`, for DNS blocklist and AbuseIPDB hits. +- `SWWAF_REPUTATION_CACHE_TTL` (default `24h`): how long a zone's verdict or + AbuseIPDB's score is used. A duration. +- `SWWAF_REPUTATION_TIMEOUT` (default `2s`): how long a query to a zone or a + check with AbuseIPDB may take. A duration. + +## Alerting - `SWWAF_ALERT_WEBHOOK_URL` (default unset): the webhook that receives each alert as a JSON object. An `http` or `https` URL without a user or a fragment. - The settings logged at start show its path and query as `********`, since many - webhooks carry their secret there. - `SWWAF_ALERT_WEBHOOK_HEADERS` (default empty): headers sent with each post, such as `Authorization:Bearer 0123456789abcdef`. A list of name, `:` and value. A value cannot hold a comma. - `SWWAF_ALERT_SLACK_WEBHOOK_URL` (default unset): the Slack incoming webhook - that receives each alert as a message. It is checked and masked as the webhook - URL is. + that receives each alert as a message, checked as the webhook URL is. - `SWWAF_ALERT_NTFY_URL` (default unset): the ntfy topic that receives each - alert, as the topic's full URL, such as `https://ntfy.sh/my-alerts`. It is - checked and masked as the webhook URL is. -- `SWWAF_ALERT_NTFY_TOKEN` (default unset): an ntfy access token, sent as - `Authorization: Bearer `. A control character in it is refused. + alert, as the topic's full URL, such as `https://ntfy.sh/my-alerts`, checked + as the webhook URL is. +- `SWWAF_ALERT_NTFY_TOKEN` (default unset): an ntfy access token. A control + character in it is refused. - `SWWAF_ALERT_EVENTS` (default `ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure,file_error`): - the events alerts are sent for. Any other name is refused. + the event types alerts are sent for, from those in [`alerts.md`](alerts.md) + but `summary`. Any other name is refused. - `SWWAF_ALERT_COOLDOWN` (default `15m`): how long a repeat of an alert is held back. A duration or `off`. -- `SWWAF_ALERT_MAX_PER_HOUR` (default `60`): the most alerts sent in an hour. - The rest are rolled into one summary. A whole number above zero, or `off`. - -See [`alerts.md`](alerts.md) for the events and what each destination gets. +- `SWWAF_ALERT_MAX_PER_HOUR` (default `60`): the most alerts sent in an hour. A + whole number above zero, or `off`. ## Anomaly thresholds -These only alert. They refuse and ban nothing. Each scope has four thresholds: -the most requests and the most bytes counted in a minute and in an hour. -Requests are counts, bytes are sizes. Each is `off` by default. What is unusual -depends on each service's normal traffic, which the metrics show. The four -thresholds per client are: +Each scope has four thresholds: the most requests and the most bytes counted in +a minute and in an hour. Requests are counts, bytes are sizes. Each is `off` by +default. The four thresholds per client are: - `SWWAF_ANOMALY_CLIENT_REQUESTS_PER_MINUTE`; - `SWWAF_ANOMALY_CLIENT_REQUESTS_PER_HOUR`; @@ -432,30 +285,19 @@ as `REQUESTS_PER_MINUTE`: client; - `SWWAF_ANOMALY_NET_V6_PREFIX` (default `48`, from 0 to 128) for an IPv6 client. -- `SWWAF_ANOMALY_ASN_...`: per AS number. Needs lookups on. +- `SWWAF_ANOMALY_ASN_...`: per AS number. Needs a lookup source. - `SWWAF_ANOMALY_TOTAL_...`: the whole service. - `SWWAF_WATCH_...`: each netblock in `SWWAF_WATCH_NETS`, as a whole. `SWWAF_WATCH_NETS` (default empty) is a list of name, `=` and netblock, such as `office=203.0.113.0/24,scraper-x=198.51.100.0/22`. An item without `=` or a name, or with a bad netblock, is refused. So is a name used twice. -Every request but the health check is counted, whatever was done with it, -clients in `SWWAF_ALLOW_NETS` included. At most 20,000 counters are kept. -`alerts.json` keeps them across a restart. - ## Settings given as files -Any setting may be given as a file holding its value. Add `_FILE` to the -setting's name: `SWWAF_METRICS_TOKEN_FILE` names the file that holds -`SWWAF_METRICS_TOKEN`. The file's contents are the value, less one newline at -the end. They are checked as the value itself would be. Setting both forms stops -the start, and so does naming a file that cannot be read. -`SWWAF_LOG_REMOTE_TLS_CA_FILE` already holds a file name, so it has no `_FILE` -form. - -A token given as a file is out of the app's reach only while the `smallwebwaf` -user alone can read the file. Make it on the host, owned by uid 65532 with mode -`0400`: +Add `_FILE` to a setting's name to read its value from a file: +`SWWAF_METRICS_TOKEN_FILE` names the file that holds `SWWAF_METRICS_TOKEN`. To +keep a token from the app, make the file on the host, owned by uid 65532 with +mode `0400`: ```sh mkdir -p /srv/app/tokens @@ -477,7 +319,8 @@ docker run \ A few limits are fixed rather than settings: -- at most 100,000 GeoJS answers are kept, for 7 days each; +- at most 100,000 GeoJS answers are kept; - at most 100,000 DNS blocklist verdicts and 100,000 AbuseIPDB scores are kept; - at most 1,000 DNS blocklist queries are under way at once; -- the CrowdSec decision list is fetched again a minute after the last fetch. +- at most 20,000 anomaly counters are kept; +- each alert destination's queue holds at most 1,000 alerts. diff --git a/docs/state-files.md b/docs/state-files.md index dd185cd..0c00910 100644 --- a/docs/state-files.md +++ b/docs/state-files.md @@ -1,102 +1,100 @@ # State files -`smallwebwaf` keeps its state in memory, and a copy of it in five JSON files in -`SWWAF_STATE_DIR`. The files are meant for people as well: you can read or edit -them at any time, and the running `smallwebwaf` takes the edit in. The -requirements they meet are under "Persistent state" in [`SPEC.md`](../SPEC.md). - -## The files - +The five JSON files in `SWWAF_STATE_DIR`, and what each holds. "Persistent +state" in [`SPEC.md`](../SPEC.md) says how they are written, read and edited. Each has a top-level `version` of 1. -- `bans.json`: active and past bans, up to `SWWAF_MAX_BANS` that `smallwebwaf` - made and every ban you made, indented to be read. Per entry: - - the netblock, start and expiry (`null` for permanent); - - the cause: `attack`, `limit`, `admin` or `crowdsec`; - - a short reason; - - when it was lifted, if you lifted it; - - the notes (see [`bans.md`](bans.md)). -- `clients.json`: per client, its minute, hour and day counters and its history - since it was first seen, one client per line, so `grep` shows everything about - one. The history holds: - - the first and last time seen; - - the AS number, AS name and country last looked up, and when; - - total requests, and how many were forwarded and how many refused; - - bytes in each direction; - - responses by status class; - - offences by kind: `limit`, `attack`, `rule_blocked`, `waf_blocked` and - `token_refused`. +## `bans.json` - Clients that were never banned are kept too, so every client's history - survives a restart. +Active and past bans: every ban an admin made, and up to `SWWAF_MAX_BANS` of +those `smallwebwaf` made. Per entry: -- `lookups.json`: GeoJS answers, one per line: the AS number, AS name and - country, when GeoJS was asked and when the answer was last used. -- `reputation.json`: - - the last good copy of each list fetched from a URL, with when it was last - tried; - - each DNS blocklist verdict still in use; - - AbuseIPDB's scores, and the checks spent today. -- `alerts.json`: - - the anomaly counters; - - the alert cooldowns: for each event and netblock, source, file or thing - counted, when it was last sent and the count of repeats held back since; - - the alerts sent and held back in the current hour; - - the alerts waiting for each destination. +- `netblock`, `start` and `expires` (`null` for a permanent ban); +- `cause`: `attack`, `limit`, `admin` or `crowdsec`; +- `reason`: a short reason; +- `lifted`: when an admin lifted it, if one did; +- `notes`: the notes below. -Times are RFC 3339 UTC. Durations and sizes are plain numbers, with the unit in -the field name. `bans.json`, `reputation.json` and `alerts.json` are indented; -`clients.json` and `lookups.json` put each entry on one line. +To ban a netblock, add an entry with its `netblock`, `start` and `expires`. +`reason`, `notes` and `cause` may be left out; the cause is then `admin`: + +```json +{ + "version": 1, + "bans": [ + { + "netblock": "203.0.113.0/24", + "start": "2026-10-06T12:00:00Z", + "expires": null, + "reason": "probes for logins" + } + ] +} +``` + +To lift a ban, add `"lifted"` with the time you lift it, such as +`"lifted": "2026-10-06T13:00:00Z"`. To keep a ban `smallwebwaf` made, set its +`cause` to `admin`. To forget a ban, delete its entry. + +The notes: + +- `asn`, `as_name` and `country`: the client's, added when the lookup answers + (empty when lookups are off); +- what was broken: the rule id and target that matched, or the trap path, or the + limit, its window and the count reached; +- the client's percentage of the limit, with the setting that gave it; +- the reputation sources the client is on; +- `request`: the request that caused the ban: time, method, host, path with its + query string, status and user agent, each text cut to 256 bytes; +- `requests` and `refused`: the netblock's requests since it was first seen, and + how many this ban has refused so far, kept up to date while the ban lasts; +- `earlier_bans`: how many earlier bans the netblock had, by cause. + +## `clients.json` + +Per client, one per line: its minute, hour and day counters, and its history +since it was first seen: + +- the first and last time seen; +- the AS number, AS name and country last looked up, and when; +- total requests, and how many were forwarded and how many refused; +- bytes in each direction; +- responses by status class; +- offences by kind: `limit`, `attack`, `rule_blocked`, `waf_blocked` and + `token_refused`. + +## `lookups.json` + +GeoJS answers, one per line: + +- `client`; +- `asn`, `as_name` and `country`; +- `answered`: when GeoJS answered; +- `used`: when the answer was last used. + +## `reputation.json` + +- the last good copy of each list fetched from a URL, with when it was last + tried; +- each DNS blocklist verdict still in use; +- AbuseIPDB's scores, and the checks spent today. + +## `alerts.json` + +- the anomaly counters; +- the alert cooldowns: for each event and netblock, source, file or anomaly + scope, when it was last sent and the count of repeats held back since; +- the alerts sent and held back in the current hour; +- the alerts waiting for each destination. ## Sizes -`clients.json` takes about 1 KiB per client, about 20 MiB at the default -`SWWAF_MAX_TRACKED_CLIENTS`. Written every 15 minutes, that is under 2 GiB of -disk writes a day. `bans.json` takes about 1.2 KiB per ban, at most about 2.5 -KiB, so about 6 MiB at the default `SWWAF_MAX_BANS`. With a full file, a hundred -new bans a day come to about 600 MiB of writes. `lookups.json` takes about 150 -bytes per answer, under 1.5 GiB of writes a day when full. The other two files -are usually a few MiB or less. - -## Writing - -`bans.json` is written `SWWAF_STATE_WRITE_DELAY` after a ban is made, lifted or -made permanent. Every file is written every `SWWAF_STATE_COUNTER_INTERVAL`, and -when `smallwebwaf` stops. Each write goes to a temporary file that then replaces -the file, so a crash leaves the old file or the new one, whole. Before writing a -file, `smallwebwaf` takes in any edit you made to it, so your edit is not -overwritten. A write that fails is logged, raised as a `file_error` alert, and -tried again at the next write. A hard kill loses what changed since the last -write: - -- ban changes from the last `SWWAF_STATE_WRITE_DELAY` (10 seconds); -- everything else from the last `SWWAF_STATE_COUNTER_INTERVAL` (15 minutes): - - counts and history, and the counts in ban notes; - - GeoJS answers; - - reputation verdicts and the AbuseIPDB count; - - anomaly counters and alert cooldowns. - -## Reading at start - -The files are read back. Each client keeps its counts, so a restart gives no one -a fresh allowance. Each ban keeps refusing until it ends. Counter buckets and -cache entries whose time has passed are discarded. A missing file is empty -state. These stop the start, with a message naming the file and the place: - -- a file that does not parse, or has another `version`; -- an entry without a field it needs; -- a ban whose `cause` is not `limit`, `attack`, `admin` or `crowdsec`; -- a ban whose `expires` is not after its `start`. - -## Editing while it runs - -`smallwebwaf` watches the directory and takes in your edit of a file as soon as -you save it. What the file then holds replaces what `smallwebwaf` held for it. -It tells its own writes from yours by comparing the file with what it last read -or wrote. A change it made after you opened the file, such as a new ban, is lost -when you save over it; most editors warn when a file changed on disk while it -was open. An edit with an error that would stop the start does not stop the -running `smallwebwaf`. It keeps what it holds. At the file's next write it -renames your file to `.bad`, writes the file again from memory, and logs -and alerts the file and the place of the error. Mend the `.bad` file and move it -back. A file you remove is written again at its next write. +- `clients.json`: about 1 KiB per client, about 20 MiB at the default + `SWWAF_MAX_TRACKED_CLIENTS`. Written every 15 minutes, that is under 2 GiB of + disk writes a day. +- `bans.json`: about 1.2 KiB per ban, at most about 2.5 KiB, so about 6 MiB at + the default `SWWAF_MAX_BANS`. With a full file, a hundred new bans a day come + to about 600 MiB of writes. +- `lookups.json`: about 150 bytes per answer, under 1.5 GiB of writes a day when + full. +- `reputation.json` and `alerts.json`: usually a few MiB or less. diff --git a/internal/remotelog/remotelog.go b/internal/remotelog/remotelog.go index 9f5f2ee..d40239b 100644 --- a/internal/remotelog/remotelog.go +++ b/internal/remotelog/remotelog.go @@ -1,9 +1,8 @@ // Package remotelog sends the lines smallwebwaf writes on stdout to the // remote log endpoint, SWWAF_LOG_REMOTE_URL, as the "Request log" section -// of SPEC.md requires and "Sending the log to a syslog server" in -// docs/request-log.md describes: each line as the message of an RFC 5424 -// syslog record, over UDP, TCP or TLS. Lines wait in a bounded buffer, so -// a slow or unreachable endpoint never holds up a request or stdout. +// of SPEC.md describes: each line as the message of an RFC 5424 syslog +// record, over UDP, TCP or TLS. Lines wait in a bounded buffer, so a slow +// or unreachable endpoint never holds up a request or stdout. package remotelog import ( diff --git a/internal/state/state.go b/internal/state/state.go index a6321c4..296fde1 100644 --- a/internal/state/state.go +++ b/internal/state/state.go @@ -1,5 +1,6 @@ // Package state keeps smallwebwaf's state in JSON files in -// SWWAF_STATE_DIR, as the "Persistent state" section of SPEC.md describes: +// SWWAF_STATE_DIR, as the "Persistent state" section of SPEC.md requires +// and docs/state-files.md lists them: // bans.json holds the bans, clients.json each client's counters and // history, lookups.json GeoJS's answers, reputation.json the last try and // last good copy of each list fetched from a URL, the DNSBL zones' diff --git a/internal/state/state_test.go b/internal/state/state_test.go index 0b98d94..97bfccf 100644 --- a/internal/state/state_test.go +++ b/internal/state/state_test.go @@ -1019,8 +1019,8 @@ func TestEditJustBeforeAScheduledWriteSurvivesIt(t *testing.T) { edit(t, dir, bansJSON, permanentBansJSON) // The write takes the edit in first, and writes it back. The second - // ban, made after the admin opened the file, is lost, as "Edits - // while running" in SPEC.md says. + // ban, made after the admin opened the file, is lost, as the + // "Persistent state" section of SPEC.md says. time.Sleep(time.Nanosecond) synctest.Wait() diff --git a/script/example-app b/script/example-app index 18cea8e..c668d2a 100755 --- a/script/example-app +++ b/script/example-app @@ -87,7 +87,7 @@ start_container() { # start_moved_container: run the app's container with the state files on # a new volume at /srv/state, which SWWAF_STATE_DIR_FILE names in a file # only the smallwebwaf user can read, as under "Settings given as files" -# in README.md, and wait until it is healthy. smallwebwaf can write the +# in docs/settings.md, and wait until it is healthy. smallwebwaf can write the # new volume only once the run script has given it to that user. start_moved_container() { docker volume create "$MOVED_VOLUME" >/dev/null