From d712bb1496dcf21c1d8a3c06ee94e6d54d17f13b Mon Sep 17 00:00:00 2001 From: sneak Date: Thu, 22 Oct 2026 21:08:26 +0000 Subject: [PATCH] SPEC: requirements and reasons only, reference material in docs/ docs/settings.md is now the one settings reference; the spec keeps the conventions, the off list and each group's requirements and reasons, and links to it for the per-setting entries. The endpoint list, the ban notes, the state file contents, the rule file format and the alert object likewise live in docs/ and are linked from the spec. The package list from the README's Design section joins "Architecture". The nine docs files are byte-identical to those on the README branch. Package comments that named "Alert webhook schema", "Request log" and "Metrics endpoint" for their field lists now name the docs files. Model: fable-5-1 --- SPEC.md | 852 +++++++++++------------------- docs/admin-endpoints.md | 35 ++ docs/alerts.md | 149 ++++++ docs/bans.md | 84 +++ docs/core-rule-set.md | 31 ++ docs/lookup.md | 79 +++ docs/reputation.md | 113 ++++ docs/rules.md | 114 ++++ docs/settings.md | 483 +++++++++++++++++ docs/state-files.md | 102 ++++ internal/alerts/alerts.go | 11 +- internal/metrics/metrics.go | 7 +- internal/ratelimit/ratelimit.go | 2 +- internal/remotelog/remotelog.go | 7 +- internal/requestlog/requestlog.go | 7 +- 15 files changed, 1515 insertions(+), 561 deletions(-) create mode 100644 docs/admin-endpoints.md create mode 100644 docs/alerts.md create mode 100644 docs/bans.md create mode 100644 docs/core-rule-set.md create mode 100644 docs/lookup.md create mode 100644 docs/reputation.md create mode 100644 docs/rules.md create mode 100644 docs/settings.md create mode 100644 docs/state-files.md diff --git a/SPEC.md b/SPEC.md index f5c683e..e7c5586 100644 --- a/SPEC.md +++ b/SPEC.md @@ -4,9 +4,9 @@ runs inside the app's container, between traefik and the app. This document says what it must do and why, each requirement once. Everything in it is built, except the last section, "Planned, not built". `README.md` says how to run it. -`EVALUATION.md` says why no existing tool was adopted. `docs/` holds the lists -this document links to, and `docs/gitea.md` records how `smallwebwaf` behaves in -front of gitea. +`EVALUATION.md` says why no existing tool was adopted. The reference documents +under `docs/`, which this document links to, give the settings, the formats and +the lists; every fact lives in one of these files and is linked from the others. ## Purpose @@ -42,10 +42,47 @@ front of gitea. - One statically linked Go binary. It writes only to its state directory. It ships in the image that is the base of the app's own image (see "Deployment"). - No database of any kind. Nothing is read from disk while serving a request. +- Packages: + - `cmd/smallwebwaf`: the binary, which only calls `internal/smallwebwaf`. + - `internal/smallwebwaf`: the process. It reads the settings, the rule + files, the lookup database and the state files, listens, and serves until + `SIGTERM` or `SIGINT`. Then it stops, writing the state files. Run as + `smallwebwaf healthcheck`, it is the image's health check instead. + - `internal/config`: reads the settings, the one place they are read. + - `internal/proxy`: what happens to each request. It works out the client + and runs the checks in `check`. It passes the request to the app with the + standard library's `httputil.ReverseProxy`, within the timeouts and size + limits, and answers its own endpoints in `answerAdmin`. Once the request + has ended, it counts bytes, refusals and anomalies, and writes the log + line. + - `internal/ratelimit`: the table of clients. It counts requests, bytes and + refusals, and keeps each client's history. + - `internal/bans`: the bans. It holds each netblock's bans with their notes, + decides how long a new ban lasts, and picks the ban to drop past + `SWWAF_MAX_BANS`. + - `internal/rules`: reads the rule files, and again as they change, and + tells which rules a request matches. + - `internal/waf`: the Core Rule Set with the six changes, as Coraza + directives. + - `internal/lookup`: looks clients up through GeoJS or in the lookup + database. + - `internal/reputation`: fetches the lists, asks the DNS blocklists and + AbuseIPDB in the background, and keeps their answers. + - `internal/anomaly`: the anomaly counters and their alerts. + - `internal/alerts`: holds back repeats and alerts past the hourly limit, + and sends the rest to the webhook, Slack and ntfy, each from a queue of + its own. + - `internal/state`: reads the state files at start, takes in edits while + running, and writes the files when due and at the stop. + - `internal/requestlog` and `internal/remotelog`: the lines on stdout, and + sending them to the syslog server. + - `internal/metrics`: the metrics, in the Prometheus text format. + - `Dockerfile`: the lint and test phases, then the image. + `deploy/example-app` is an app built on the image, which + `script/example-app` checks. - Libraries, checked against the owner's Go dependency defaults: - - the standard library for the proxy (`httputil.ReverseProxy`), HTTP - clients, DNS, logging (`log/slog`) and the state files (`encoding/json`, - `os.Rename`); + - the standard library for the proxy, HTTP clients, DNS, logging + (`log/slog`) and the state files (`encoding/json`, `os.Rename`); - `github.com/corazawaf/coraza/v3` and `github.com/corazawaf/coraza-coreruleset/v4` at `v4.25.0`, which carries the Core Rule Set 4.25.0. The changes `smallwebwaf` makes to it, and the @@ -56,6 +93,7 @@ front of gitea. - `github.com/oschwald/maxminddb-golang/v2` to read the lookup database; - `github.com/fsnotify/fsnotify` to notice edited files; - `github.com/prometheus/client_golang` for the metrics; + - `github.com/maxmind/mmdbwriter`, in the tests, to write lookup databases; - remote log sending written in this project, since the standard library's `log/syslog` is frozen and writes only the older syslog format. @@ -101,7 +139,7 @@ names the section that holds its rules. 8. The client's limit percentage (see "Biased thresholds"). 9. Rate limiting (see "Request rate limits, per client"). A request over a limit times the percentage is refused and bans the client. -10. The trap paths (see `SWWAF_TRAP_PATHS`). A request for one is refused and +10. The trap paths (see "Attack detection"). A request for one is refused and bans the client. 11. The rule files (see "Rule files"). Matching stops at the first rule that refuses. @@ -117,8 +155,8 @@ names the section that holds its rules. counters, as `SWWAF_BYTES_COUNT` says. A client whose total passes a byte limit times the percentage has broken that limit and is banned. The response in progress is not cut off. - - The client's refusals are counted for the error burst (see - `SWWAF_ERROR_BURST_THRESHOLD`). + - The client's refusals are counted for the error burst (see "Attack + detection"). - The client's history and the anomaly counters are updated, and the alert thresholds are evaluated. @@ -139,7 +177,8 @@ the token check still apply. - Memory is bounded. When the table of clients holds `SWWAF_MAX_TRACKED_CLIENTS`, the least recently seen client is dropped first, with its history. The bans, the GeoJS answers, the reputation verdicts and the - anomaly counters have bounds of their own, given where each is described. + anomaly counters have bounds of their own; `docs/settings.md` gives the fixed + ones under "Fixed limits". ## Bans @@ -200,11 +239,12 @@ no offence: - a blocklist or reputation hit whose action is `deny`; - the country lists. -Every ban carries notes with what an admin needs to decide whether to lift it. -An admin adds or lifts a ban by editing `bans.json` (see "Persistent state"), or -through the ban endpoints when `SWWAF_ADMIN_TOKEN` is set. A ban lifted before -it ends is kept, marked as lifted, and does not count toward a longer ban. -Deleting its entry from `bans.json` forgets it entirely. +Every ban carries notes with what an admin needs to decide whether to lift it, +drawn from what `smallwebwaf` already knows; nothing is looked up to fill them. +`docs/bans.md` lists the notes and says how a ban is made, kept or lifted, by +editing `bans.json` or through the ban endpoints. A ban lifted before it ends is +kept, marked as lifted, and does not count toward a longer ban. Deleting its +entry from `bans.json` forgets it entirely. A permanent ban does not run out. It ends when an admin lifts it, or when `smallwebwaf` drops it from a full table, as below. When `smallwebwaf` holds @@ -248,42 +288,32 @@ Conventions: - An invalid setting stops the process at start with a message naming the variable. At start the settings in effect are logged, with secrets masked. -The settings, by group: +`docs/settings.md` lists every setting with its default and the values it takes, +in the groups below. What each group must do, and why: ### Core -- `SWWAF_UPSTREAM_URL` (default `http://127.0.0.1:8081`): the app. -- `SWWAF_LISTEN_ADDR` (default `:8080`): the one listener, for the requests - `smallwebwaf` forwards and for its own endpoints. The address part stays empty - (see "Ports"). -- `SWWAF_INSTANCE_NAME` (default: the container's host name, which docker sets - to the first 12 characters of the container's id): included in every log line, - metric and alert. Set it to a name that survives a redeploy and tells - instances apart, such as `fsn1app1/gitea`. -- `SWWAF_MODE` (default `enforce`): `enforce`, or `observe` to log and alert on - every decision while refusing nothing. -- `SWWAF_TRUSTED_PROXIES` (default `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`, - the private address ranges): the netblocks whose `X-Forwarded-For` is - believed, normally where traefik reaches `smallwebwaf` from. A client that - connects directly cannot claim another address. A list given replaces the - default; set but empty, it trusts nothing. Where other containers reach - `smallwebwaf` directly, or visitors on a private network reach traefik, they - can name another address in the header; this setting, narrowed to traefik's - own network, closes that gap. -- `SWWAF_IPV6_GROUP_PREFIX` (default `64`), from 32 to 128. -- `SWWAF_MAX_TRACKED_CLIENTS` (default `20000`): the clients held in memory and - in `clients.json`. -- `SWWAF_MAX_BANS` (default `5000`): the bans `smallwebwaf` made that are held, - past, active and permanent (see "Bans"). +- One listener, `SWWAF_LISTEN_ADDR`, for the requests `smallwebwaf` forwards to + `SWWAF_UPSTREAM_URL` and for its own endpoints (see "Ports"). +- `SWWAF_INSTANCE_NAME` is in every log line, metric and alert, so that + instances can be told apart. The default, the container's host name, changes + at every redeploy; a name set by the operator survives it. +- `SWWAF_MODE=observe` logs and alerts on every decision while refusing nothing, + so that a configuration can be tried before it is enforced. +- `X-Forwarded-For` is believed only from `SWWAF_TRUSTED_PROXIES`, by default + the private address ranges, so a client that connects directly cannot claim + another address. Where other containers reach `smallwebwaf` directly, or + visitors on a private network reach traefik, they can name another address in + the header; the setting, narrowed to traefik's own network, closes that gap. +- `SWWAF_IPV6_GROUP_PREFIX`, `SWWAF_MAX_TRACKED_CLIENTS` and `SWWAF_MAX_BANS` + set the group size and the bounds under "Counting method" and "Bans". ### Tokens -- `SWWAF_ADMIN_TOKEN`: the bearer token for the ban endpoints and - `/_smallwebwaf/clients/`. Unset by default, which switches them off. -- `SWWAF_METRICS_TOKEN`: the bearer token a scraper sends for - `/_smallwebwaf/metrics`. Unset by default, which switches the metrics off, - since they would otherwise be open to anyone on the internet. It is separate - from `SWWAF_ADMIN_TOKEN`, so a scraper that holds it cannot manage bans. +- `SWWAF_ADMIN_TOKEN` switches on the ban endpoints and + `/_smallwebwaf/clients/`; `SWWAF_METRICS_TOKEN` switches on the metrics, + which would otherwise be open to anyone on the internet. The two are separate, + so a scraper that holds the metrics token cannot manage bans. - A token is a long random value, since the endpoints can be reached from the internet. One shorter than 32 characters stops the start. The app starts with the same environment variables as `smallwebwaf`, so it can read a token set as @@ -292,50 +322,39 @@ The settings, by group: ### State files -- `SWWAF_STATE_DIR` (default `/var/lib/smallwebwaf`): the JSON state files, in a - directory of their own beside the app's data, belonging to the `smallwebwaf` - user. -- `SWWAF_STATE_WRITE_DELAY` (default `10s`): how long `bans.json` waits to be - written after a ban is made, lifted or made permanent, so a burst of changes - becomes one write. -- `SWWAF_STATE_COUNTER_INTERVAL` (default `15m`): how often every state file is - written. +- The state files live in `SWWAF_STATE_DIR`, a directory of their own beside the + app's data, belonging to the `smallwebwaf` user. +- `bans.json` is written `SWWAF_STATE_WRITE_DELAY` after a ban is made, lifted + or made permanent, so a burst of changes becomes one write. Every file is + written every `SWWAF_STATE_COUNTER_INTERVAL`, and at an orderly stop. ### Logging -- The request log on stdout is always on and has no switch. -- `SWWAF_LOG_LEVEL` (default `info`): for the process's own messages, which are - JSON lines on stdout too, marked `"type":"process"`. It does not filter the - request log. -- `SWWAF_LOG_REQUEST_HEADERS` (default - `accept,accept-language,accept-encoding,content-type,origin,range`): the - request headers to record. `Authorization`, `Cookie` and `Set-Cookie` values - are never logged, only whether they were present. -- `SWWAF_LOG_REMOTE_URL`: when set, every log line is also sent there, as - `syslog+udp://host:514`, `syslog+tcp://host:514` or `syslog+tls://host:6514`. -- `SWWAF_LOG_REMOTE_TLS_CA_FILE`: an optional CA certificate file for the `+tls` - form. -- `SWWAF_LOG_REMOTE_BUFFER` (default `10000`): the lines held in memory while - the server is unreachable. When full, the oldest are dropped and counted. -- `SWWAF_LOG_REMOTE_FACILITY` (default `local0`) and `SWWAF_LOG_REMOTE_APP_NAME` - (default `SWWAF_INSTANCE_NAME`): the syslog header fields. +- The request log on stdout is always on and has no switch. The process's own + messages are JSON lines on stdout too, marked `"type":"process"`; + `SWWAF_LOG_LEVEL` filters only those. +- `SWWAF_LOG_REQUEST_HEADERS` names the request headers to record. + `Authorization`, `Cookie` and `Set-Cookie` values are never logged, only + whether they were present. +- `SWWAF_LOG_REMOTE_URL` sends every log line to a syslog server as well, over + UDP, TCP or TLS. Lines wait in a buffer of `SWWAF_LOG_REMOTE_BUFFER` lines; + when it is full, the oldest are dropped and counted. ### Metrics -- `SWWAF_METRICS_TOP_N` (default `50`): how many AS numbers and countries get - series of their own. The rest are summed as `other`, so the label set stays +- `SWWAF_METRICS_TOP_N` bounds the series by AS number and by country: that many + of the busiest each, the rest summed as `other`, so the label set stays bounded. ### Static lists -- `SWWAF_ALLOW_NETS`: bypass everything (monitoring, the owner's own networks). -- `SWWAF_RATE_LIMIT_EXEMPT_NETS`: bypass the request and byte limits only. The - error burst, attack detection and bans still apply. -- `SWWAF_DENY_NETS`: always refused. - -Many clients share one address (mobile carriers, offices, Tor), and so share its -limits and its bans. The defaults sit well above one person's use, and the first -two lists take known shared addresses out. +- `SWWAF_ALLOW_NETS` bypasses everything (monitoring, the owner's own networks). + `SWWAF_RATE_LIMIT_EXEMPT_NETS` bypasses the request and byte limits only; the + error burst, attack detection and bans still apply. `SWWAF_DENY_NETS` is + always refused. +- Many clients share one address (mobile carriers, offices, Tor), and so share + its limits and its bans. The defaults sit well above one person's use, and the + first two lists take known shared addresses out. ### Request rate limits, per client @@ -344,20 +363,14 @@ busy person produces. A browser loading a heavy page makes a few hundred requests, a git clone a handful, and several people often share one address. A machine that talks to the app all day, such as a gitea Actions runner, can still pass the day limit. Its address belongs in `SWWAF_RATE_LIMIT_EXEMPT_NETS`. - -- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `1000`), `SWWAF_RATE_LIMIT_PER_HOUR` - (default `10000`) and `SWWAF_RATE_LIMIT_PER_DAY` (default `50000`). -- `SWWAF_RATE_LIMIT_EXEMPT_PATHS`: path prefixes not counted, such as static - assets or health checks. +`SWWAF_RATE_LIMIT_EXEMPT_PATHS` takes path prefixes out of the count, such as +static assets or health checks. ### Byte limits, per client A response's bytes are counted when it ends, so every default sits above the -largest response allowed, and no single download breaks one. - -- `SWWAF_BYTES_LIMIT_PER_MINUTE` (default `10G`), `SWWAF_BYTES_LIMIT_PER_HOUR` - (default `20G`) and `SWWAF_BYTES_LIMIT_PER_DAY` (default `50G`). -- `SWWAF_BYTES_COUNT` (default `both`): `response`, `request` or `both`. +largest response allowed, and no single download breaks one. `SWWAF_BYTES_COUNT` +says whether request bytes, response bytes or both are counted. ### Size and time limits @@ -369,25 +382,12 @@ through unchanged, so a limit on each side would bound the same bytes, and the lower one would always decide. Bodies stream straight through, so a request body reaches the app while the client is still sending it. -- `SWWAF_CLIENT_REQUEST_TIMEOUT` (default `60s`): how long a client may take to - send its request line and headers, and then, from the end of the headers, its - body. -- `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES` (default `32K`): the largest request - line and headers. It must be more than `4K`. Go's HTTP server always has such - a limit, and reads 4 KiB past the one it is given before it refuses. -- `SWWAF_CLIENT_IDLE_TIMEOUT` (default `120s`): how long a kept-open connection - may wait for its next request before `smallwebwaf` closes it. It is longer - than the 90 seconds after which traefik, by default, closes an unused - connection. So traefik closes first, and never sends a request on a connection - `smallwebwaf` is closing. -- `SWWAF_CLIENT_RESPONSE_TIMEOUT` (default `30m`): how long one response may - take to reach the client, from the end of the request to the last byte. -- `SWWAF_UPSTREAM_REQUEST_TIMEOUT` (default `60s`): how long connecting to the - app and sending it the whole request may take. -- `SWWAF_UPSTREAM_RESPONSE_TIMEOUT` (default `30m`): how long the app may take - to send one whole response, from the end of the request to the last byte. -- `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body. -- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body. +- `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES` must be more than `4K` and cannot be + `off`. Go's HTTP server always has such a limit, and reads 4 KiB past the one + it is given before it refuses. +- `SWWAF_CLIENT_IDLE_TIMEOUT` is longer by default than the 90 seconds after + which traefik, by default, closes an unused connection. So traefik closes + first, and never sends a request on a connection `smallwebwaf` is closing. What the client gets when a limit is passed: @@ -416,71 +416,56 @@ What the client gets when a limit is passed: ### Lookup of AS number and country On by default through GeoJS, which needs no account or file, so that country -lookup works with no setup. A local database file is the alternative for a -service that keeps its visitors' addresses on its own host. +lookup works with no setup. A local database file, the free IPinfo Lite +database, is the alternative for a service that keeps its visitors' addresses on +its own host; the operator downloads and mounts it, and `smallwebwaf` never +fetches it and holds no account or token for it. `docs/lookup.md` says what each +source sends where, how long answers are kept, and when a request waits. -- `SWWAF_LOOKUP_SOURCE` (default `geojs`): `geojs`, `file` or `off`. Only one - source is used. The start stops, naming both settings, when `file` is set - without `SWWAF_LOOKUP_DB_PATH`, or `SWWAF_LOOKUP_DB_PATH` with any other - source. The start also stops when the source is `off` and one of these needs - lookups: - - a country list; - - a biased threshold; - - `SWWAF_ADD_LOOKUP_HEADERS`; - - an anomaly threshold per AS number. -- `SWWAF_LOOKUP_DB_PATH`: the database the `file` source reads: the IPinfo Lite - database (`ipinfo_lite.mmdb`), whose `asn`, `as_name` and `country_code` - fields are read. The operator downloads it with a free IPinfo account, mounts - the directory that holds it read-only (see "Rule files" for why), and - refreshes it. `smallwebwaf` never fetches it. A file that is missing or - unreadable at start stops the start. A replaced file is read again; a - replacement that cannot be read is logged, sent as one `file_error` alert, and - ignored, the file in use staying in use. -- `geojs` asks the free GeoJS web service about up to 200 addresses in one - request (`https://get.geojs.io/v1/ip/geo.json?ip=a,b,c`), one request at a - time, and reads each answer's `country_code`, `asn` and `organization_name`. - An answer without a `country_code`, or with the `asn` `64512`, counts as - unknown. GeoJS is told the address of every new visitor, whether or not a - setting uses the answer, since the request log, the metrics and the ban notes - carry it. Private, loopback and link-local addresses, which no source can - place, are never sent. +- Only one source is used. The start stops, naming both settings, when `file` is + set without `SWWAF_LOOKUP_DB_PATH`, or `SWWAF_LOOKUP_DB_PATH` with any other + source. The start also stops when the source is `off` and a setting needs + lookups: a country list, a biased threshold, `SWWAF_ADD_LOOKUP_HEADERS` or an + anomaly threshold per AS number. +- GeoJS is told the address of every new visitor, whether or not a setting uses + the answer, since the request log, the metrics and the ban notes carry it. + Private, loopback and link-local addresses, which no source can place, are + never sent. Addresses are asked about in batches, one request at a time, and + answers are kept for days, which keeps the number of requests to GeoJS low. + Answers are kept apart from the table of clients, so that an answer outlasts a + client dropped from that table. - A client the lookup cannot place has an unknown country and AS number. The exclusive country list refuses it, and the biased thresholds give it `SWWAF_UNKNOWN_LIMIT_PERCENT`. Nothing else treats it differently. -- Each GeoJS answer is kept for 7 days, in memory and in `lookups.json`, - separately from the table of clients, so that it outlasts a client dropped - from that table. Up to 100,000 answers are kept, about 15 MiB; past that, the - answer used longest ago goes first. -- `SWWAF_LOOKUP_TIMEOUT` (default `1s`): how long a visitor's request waits for - its client's first answer when a setting needs it. It is also how long a - request to GeoJS may take. A client whose answer has not come in time counts - as unknown until it comes, and its later requests do not wait. +- `SWWAF_LOOKUP_TIMEOUT` bounds both how long a visitor's request waits for its + client's first answer, when a setting needs it, and how long a request to + GeoJS may take. A client whose answer has not come in time counts as unknown + until it comes, and its later requests do not wait. - GeoJS publishes no rate limit, but its terms forbid "an excessive amount of - API requests", judged by GeoJS alone, and let it block a caller. Kept answers - and batched requests keep the number of requests low. While GeoJS is slow, - down or refusing `smallwebwaf`, new clients count as unknown, `smallwebwaf` - keeps asking with backoff, and one `source_failure` alert is sent per - cooldown. A service that cannot accept this, or that every new visitor's - address goes to a third party, uses the file. -- `SWWAF_ADD_LOOKUP_HEADERS` (default `false`): pass `X-Client-ASN` and - `X-Client-Country` to the app. + API requests", judged by GeoJS alone, and let it block a caller. While GeoJS + is slow, down or refusing `smallwebwaf`, new clients count as unknown, + `smallwebwaf` keeps asking with backoff, and one `source_failure` alert is + sent per cooldown. A service that cannot accept this, or that every new + visitor's address goes to a third party, uses the file. +- A replaced database file is read again; one that cannot be read is logged, + sent as one `file_error` alert, and ignored, the file in use staying in use. + The directory that holds the file is mounted, not the file itself (see "Rule + files" for why). ### Country lists Both are empty by default: the defaults judge a client by what it does, not by where it comes from. Clients in `SWWAF_ALLOW_NETS` are not checked. -- `SWWAF_DENIED_COUNTRIES`: for example `cn,ru,kp,ir,ua,by`. Every request from - a listed country is refused. -- `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES`: for example `us,de`. Only clients - placed in a listed country get through. Every request from any other country - is refused. So is every request from a client the lookup cannot place (see - "Lookup of AS number and country"), such as internal monitoring; those that - should reach the app go in `SWWAF_ALLOW_NETS`. A list that let unplaced - clients through would let every new client in whenever GeoJS stops answering. - The cost: while GeoJS fails, this list refuses every new client. No other - failure of GeoJS, a reputation source or an alert destination comes this close - to taking the service down. +- `SWWAF_DENIED_COUNTRIES` refuses every request from a listed country. +- `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` lets through only clients placed in a + listed country. Every request from any other country is refused. So is every + request from a client the lookup cannot place, such as internal monitoring; + those that should reach the app go in `SWWAF_ALLOW_NETS`. A list that let + unplaced clients through would let every new client in whenever GeoJS stops + answering. The cost: while GeoJS fails, this list refuses every new client. No + other failure of GeoJS, a reputation source or an alert destination comes this + close to taking the service down. - Both may be set. `SWWAF_DENIED_COUNTRIES` then adds nothing, since the exclusive list already refuses every other country. A code on both lists stops the start with a message naming it. @@ -495,20 +480,14 @@ The lists are empty by default, for the same reason as the country lists. The request log and the metrics show which AS numbers and countries a service's abuse comes from. -- `SWWAF_ASN_LIMIT_PERCENT`: for example `AS14061:50,AS16276:50,AS45102:25`. - Clients in a listed AS number get that percentage of every request and byte - limit. `0` is a zero allowance: the client's first request breaks a limit and - bans it. -- `SWWAF_COUNTRY_LIMIT_PERCENT`: the same form with country codes, for example - `CN:25,RU:50`. -- `SWWAF_ASN_BYTES_PERCENT` and `SWWAF_COUNTRY_BYTES_PERCENT`: optional - overrides applied to the byte limits only, when the byte percentage should - differ from the request percentage. -- `SWWAF_UNKNOWN_LIMIT_PERCENT` (default `100`): for a client the lookup cannot - place. -- `SWWAF_ASN_LIMIT_PERCENT_URL`: optional URL of a text file of `AS:percent` - lines, so one abuse-source list can be shared by every `smallwebwaf` in the - fleet. It is fetched and refreshed like the blocklists. +- Clients in a listed AS number or country get that percentage of every request + and byte limit (`SWWAF_ASN_LIMIT_PERCENT`, `SWWAF_COUNTRY_LIMIT_PERCENT`), or + of the byte limits alone (`SWWAF_ASN_BYTES_PERCENT`, + `SWWAF_COUNTRY_BYTES_PERCENT`). `0` is a zero allowance: the client's first + request breaks a limit and bans it. +- `SWWAF_ASN_LIMIT_PERCENT_URL` names a text file of `AS:percent` lines, so one + abuse-source list can be shared by every `smallwebwaf` in the fleet. It is + fetched and refreshed like the blocklists. - A client's limit percentage is the lowest of those that apply to it: its AS number's, its country's, or a reputation hit's `limit:`. It is 100 if none applies. Clients given a lower percentage are banned after fewer @@ -529,20 +508,15 @@ files saved in the web editor. The Core Rule Set takes the shell commands and code in them for attacks. The default gives up refusing an attack carried in a body; the rule files, the limits and the bans still apply to the client that sends it, and `SWWAF_WAF_BODY_LIMIT` switches body inspection on. +`docs/core-rule-set.md` says what the Core Rule Set reads and how it scores a +request. -- `SWWAF_RULES_DIR` (default `/etc/smallwebwaf/rules.d`): the directory of rule - files (see "Rule files"). -- `SWWAF_RULES_ENABLED` (default `true`): `false` skips rule files entirely. -- `SWWAF_WAF_MODE` (default `block`): `off`, `detect` (log and alert only), or - `block`. -- `SWWAF_WAF_PARANOIA_LEVEL` (default `1`) and `SWWAF_WAF_ANOMALY_THRESHOLD` - (default `5`): the Core Rule Set's own two tuning values, at its own defaults. - Each rule that matches adds its score to the request's. A request whose score +- `SWWAF_WAF_PARANOIA_LEVEL` and `SWWAF_WAF_ANOMALY_THRESHOLD` are the Core Rule + Set's own two tuning values, at its own defaults. A request whose score reaches the threshold is a match. With the threshold `off`, nothing is a match, and scores are still logged. -- `SWWAF_WAF_DISABLED_RULES` (default - `920340,920420,920440,920640,930130,930140`): rule ids to switch off when an - app trips a false positive. The default switches off six rules: +- `SWWAF_WAF_DISABLED_RULES` switches off rule ids when an app trips a false + positive. The default switches off six rules: - 920340, 920420 and 920640 refuse a request for the type of its body, when the type is missing or not on the Core Rule Set's short list; - 920440 refuses a request for its file extension, such as `.sh` or `.sql`; @@ -553,30 +527,27 @@ sends it, and `SWWAF_WAF_BODY_LIMIT` switches body inspection on. In front of a code forge these refuse git over HTTP, container image and package uploads, and views of ordinary files in a repository. At the site root, where no app serves such files, the default rule file bans the common - probes these rules caught (see "Rule files"). A list given replaces the + probes these rules caught (see "Rule files"). A list you give replaces the default, so name these six ids in it to keep them off. -- `SWWAF_WAF_EXEMPT_PATHS`: path prefixes not inspected. -- `SWWAF_WAF_BODY_LIMIT` (default `off`): `off` reads no request body. A size, - such as `128K`, switches body inspection on. The Core Rule Set then reads form - data and multipart bodies up to that size, streaming the rest of a longer one - on without holding it in memory. It reads JSON and XML bodies no larger than +- `SWWAF_WAF_BODY_LIMIT` switches body inspection on, up to that size. Form data + and multipart bodies are read up to the size, the rest streaming on without + being held in memory; JSON and XML bodies are read only when no larger than the size, since those cannot be read in part. Any other body is not read: the Core Rule Set would read it as form data, where binary content such as a git push trips rules written for text. Body inspection suits apps whose forms carry no code. `docs/gitea.md` records what it refuses in front of gitea. -- `SWWAF_TRAP_PATHS`: paths the app never serves and only scanners ask for, for - example `/wp-login.php,/xmlrpc.php` in front of gitea. A request for one is a - clear sign of attack. This is the short form of a `path` rule with the `ban` - action, for deployments that mount no rule files. -- `SWWAF_ERROR_BURST_THRESHOLD` (default `30`): the refusals per client per - minute after a rule file match, a trap path, a Core Rule Set match, or a - missing or wrong token. More than this breaks a limit (see "Bans"). A client - trying one attack or token after another is refused many times a minute; a - person rarely more than a few times. Answers the app gives are not counted, - since they do not tell a scanner from an ordinary client. In front of gitea, - registry and git clients get `404` and `401` by design, hundreds of times a - minute. +- `SWWAF_TRAP_PATHS` names paths the app never serves and only scanners ask for, + such as `/wp-login.php` in front of gitea. A request for one is a clear sign + of attack. This is the short form of a `path` rule with the `ban` action, for + deployments that mount no rule files. +- `SWWAF_ERROR_BURST_THRESHOLD` bounds the refusals per client per minute after + a rule file match, a trap path, a Core Rule Set match, or a missing or wrong + token. More than this breaks a limit (see "Bans"). A client trying one attack + or token after another is refused many times a minute; a person rarely more + than a few times. Answers the app gives are not counted, since they do not + tell a scanner from an ordinary client. In front of gitea, registry and git + clients get `404` and `401` by design, hundreds of times a minute. - A Core Rule Set engine error on a request forwards the request, and is logged and counted. @@ -640,15 +611,11 @@ every app, and no setting undoes them: ### Ban response and scope -- `SWWAF_BAN_RESPONSE` (default `403`): `403`, `429`, or `close` to drop the - connection without an answer. `403` makes a ban easy to recognise when - debugging. Behind traefik, `close` does not leave the client unanswered: - traefik answers `502`, as it does whenever its backend drops a connection. -- `SWWAF_BAN_SCOPE_V4_PREFIX` (default `32`): widen to for example `24` to ban - the surrounding netblock. -- `SWWAF_ATTACK_BAN_DURATION`, `SWWAF_LIMIT_BAN_DURATION`, - `SWWAF_LIMIT_BAN_REPEAT_WINDOW` and `SWWAF_MAX_BAN_DURATION`: the ban lengths, - with their defaults, under "Bans". +`SWWAF_BAN_RESPONSE` is `403` by default, which makes a ban easy to recognise +when debugging; `429` and `close` are the alternatives. Behind traefik, `close` +does not leave the client unanswered: traefik answers `502`, as it does whenever +its backend drops a connection. `SWWAF_BAN_SCOPE_V4_PREFIX` widens an IPv4 ban +to the surrounding netblock. The four ban lengths are under "Bans". ### Reputation @@ -658,70 +625,54 @@ account or an engine of the operator's own. The list best suited to be on by default, Spamhaus DROP, must not be fetched more than once an hour, and Spamhaus may block an address that does. Each `smallwebwaf` fetches its own copy, so several apps on one host, sharing one address, would often fetch less than an -hour apart. +hour apart. `docs/reputation.md` says how each source is asked and what is done +with the answer. -- `SWWAF_BLOCKLIST_URLS`: text files of addresses and netblocks, one per line; - anything after a `;` or `#` on a line is ignored. For example the Spamhaus - DROP list, `https://www.spamhaus.org/drop/drop.txt`. Spamhaus's terms for it, - on its DROP page: +- Blocklists (`SWWAF_BLOCKLIST_URLS`) are text files of addresses and netblocks, + such as the Spamhaus DROP list. Spamhaus's terms for it, on its DROP page: - automated downloads must be at least one hour apart, and once a day is more than enough in most cases; - a product that uses it must credit The Spamhaus Project and keep the list's date and copyright lines with the data. -- `SWWAF_BLOCKLIST_REFRESH` (default `24h`, never less than `1h`): how often - each list is fetched again. The last good copy is kept whole, comment lines - included, on failure and across restarts. -- `SWWAF_BLOCKLIST_ACTION` (default `deny`): `deny`, `limit:` or `log`. - `deny` refuses every request from a listed address, which suits lists of - networks that send nothing legitimate, such as DROP. `limit:` gives - listed clients that percentage of every limit. It suits a list of addresses - shared with ordinary visitors, such as Tor exits. -- `SWWAF_DNSBL_ZONES`: for example `dnsbl.dronebl.org`. Zones meant for mail - (lists of residential ranges) will block ordinary visitors and should not be - used. Spamhaus zones need their keyed query service, given as a zone name - containing the key. -- `SWWAF_DNSBL_RESOLVER`: an optional resolver address, since public resolvers - are refused by several list operators. -- `SWWAF_ABUSEIPDB_KEY`, `SWWAF_ABUSEIPDB_MIN_SCORE` (default `75`) and - `SWWAF_ABUSEIPDB_DAILY_BUDGET` (default `900`; the free tier allows 1000 - checks a day). Only clients that have already committed one offence are - queried, so the budget is spent on suspects. -- `SWWAF_CROWDSEC_LAPI_URL` and `SWWAF_CROWDSEC_LAPI_KEY`: optional. If a - CrowdSec engine exists on the host, its decision list is pulled on a schedule. - It is kept like a fetched blocklist, and a client on it is banned (see - "Bans"). This is how a fleet-wide blocklist can arrive without `smallwebwaf` - depending on CrowdSec. -- `SWWAF_REPUTATION_ACTION` (default `limit:25`): `deny`, `limit:` or - `log`, for DNS blocklist and AbuseIPDB hits. Such verdicts are less certain - than a blocklist, so by default a listed client gets a quarter of every limit. -- `SWWAF_REPUTATION_CACHE_TTL` (default `24h`) and `SWWAF_REPUTATION_TIMEOUT` - (default `2s`). -- Up to 100,000 DNS blocklist verdicts and 100,000 AbuseIPDB scores are kept. - Past that, the one fetched longest ago goes first. -- A source that is down or over quota gives no verdict. The service continues, - with one `source_failure` alert per cooldown. + + So `SWWAF_BLOCKLIST_REFRESH` is never less than `1h`, and the last good copy + is kept whole, comment lines included, on failure and across restarts. + +- `SWWAF_BLOCKLIST_ACTION` is `deny` by default, which refuses every request + from a listed address and suits lists of networks that send nothing + legitimate, such as DROP. `limit:` gives listed clients that + percentage of every limit, and suits a list of addresses shared with ordinary + visitors, such as Tor exits. `log` only notes the hit. +- DNS blocklists (`SWWAF_DNSBL_ZONES`) meant for mail list residential ranges + and would block ordinary visitors, so they should not be used. Spamhaus zones + need their keyed query service, given as a zone name containing the key. + `SWWAF_DNSBL_RESOLVER` names a resolver, since public resolvers are refused by + several list operators. +- AbuseIPDB is asked only about clients that have already committed one offence, + so the daily budget (`SWWAF_ABUSEIPDB_DAILY_BUDGET`, under the free tier's + 1000 checks a day) is spent on suspects. +- A CrowdSec engine on the host (`SWWAF_CROWDSEC_LAPI_URL`) has its decision + list pulled on a schedule. It is kept like a fetched blocklist, and a client + on it is banned (see "Bans"). This is how a fleet-wide blocklist can arrive + without `smallwebwaf` depending on CrowdSec. +- `SWWAF_REPUTATION_ACTION`, for DNS blocklist and AbuseIPDB hits, is `limit:25` + by default. Such verdicts are less certain than a blocklist, so a listed + client gets a quarter of every limit rather than a refusal. +- Verdicts and scores are kept for `SWWAF_REPUTATION_CACHE_TTL`, with a fixed + bound on their number. A source that is down or over quota gives no verdict. + The service continues, with one `source_failure` alert per cooldown. ### Alerting -- `SWWAF_ALERT_WEBHOOK_URL`: a JSON POST; schema under "Alert webhook schema". - `SWWAF_ALERT_WEBHOOK_HEADERS`: optional `Name:value` pairs for authentication. -- `SWWAF_ALERT_SLACK_WEBHOOK_URL`: a Slack incoming webhook, sent a formatted - message. -- `SWWAF_ALERT_NTFY_URL` (the full topic URL) and `SWWAF_ALERT_NTFY_TOKEN`: - title, priority and tags set from the event type. -- `SWWAF_ALERT_EVENTS` (default - `ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure,file_error`): - which event types are sent. `source_failure` is a reputation source or GeoJS - failing or refusing `smallwebwaf`. `file_error` is one of: - - a rule file or state file edited while running that does not parse; - - a replacement lookup database that cannot be read; - - a state file that cannot be written. -- `SWWAF_ALERT_COOLDOWN` (default `15m`): the same event is not repeated within - this time for the same netblock, source or file, or, for an anomaly, for the - same thing counted: one client, netblock, AS number or named netblock, or the - whole service. A count of suppressed repeats is included in the next one. -- `SWWAF_ALERT_MAX_PER_HOUR` (default `60`): beyond this, alerts are rolled into - one summary per hour, so a wide attack cannot flood the channel. +- Alerts go to a webhook as JSON (`docs/alerts.md` gives the object), and to + Slack and ntfy as formatted messages. `SWWAF_ALERT_EVENTS` chooses the event + types sent. +- `SWWAF_ALERT_COOLDOWN`: the same event is not repeated within this time for + the same netblock, source or file. For an anomaly, it is the same thing + counted that must match: one client, netblock, AS number or named netblock, or + the whole service. A count of suppressed repeats is included in the next one. +- `SWWAF_ALERT_MAX_PER_HOUR`: beyond this, alerts are rolled into one summary + per hour, so a wide attack cannot flood the channel. - A destination that is down is retried with backoff from a bounded queue, the oldest alert dropped first, and the drops counted in the metrics. @@ -734,24 +685,9 @@ which the metrics show. They are what reveals a widely distributed scraper, thousands of addresses at low rates each, which the per-client limits do not see; a low `SWWAF_ASN_LIMIT_PERCENT` for its AS number is the response. -- Per client: `SWWAF_ANOMALY_CLIENT_REQUESTS_PER_MINUTE`, - `SWWAF_ANOMALY_CLIENT_REQUESTS_PER_HOUR`, - `SWWAF_ANOMALY_CLIENT_BYTES_PER_MINUTE` and - `SWWAF_ANOMALY_CLIENT_BYTES_PER_HOUR`. -- Per surrounding netblock (`SWWAF_ANOMALY_NET_V4_PREFIX` default `24`, - `SWWAF_ANOMALY_NET_V6_PREFIX` default `48`): - `SWWAF_ANOMALY_NET_REQUESTS_PER_MINUTE`, `..._PER_HOUR`, - `SWWAF_ANOMALY_NET_BYTES_PER_MINUTE` and `..._PER_HOUR`. -- Per AS number, which needs a lookup source: - `SWWAF_ANOMALY_ASN_REQUESTS_PER_MINUTE`, `..._PER_HOUR`, - `SWWAF_ANOMALY_ASN_BYTES_PER_MINUTE` and `..._PER_HOUR`. -- The whole service: `SWWAF_ANOMALY_TOTAL_REQUESTS_PER_MINUTE`, `..._PER_HOUR`, - `SWWAF_ANOMALY_TOTAL_BYTES_PER_MINUTE` and `..._PER_HOUR`. -- Named netblocks: `SWWAF_WATCH_NETS`, for example - `office=203.0.113.0/24,scraper-x=198.51.100.0/22`, with - `SWWAF_WATCH_REQUESTS_PER_MINUTE`, `..._PER_HOUR`, - `SWWAF_WATCH_BYTES_PER_MINUTE` and `..._PER_HOUR` applied to each named - netblock as a whole. +- Requests and bytes are counted per minute and per hour, for each client, each + surrounding netblock, each AS number, the whole service, and each named + netblock in `SWWAF_WATCH_NETS`. - Anomaly counting includes clients in `SWWAF_ALLOW_NETS` and `SWWAF_RATE_LIMIT_EXEMPT_NETS`, since an exempt client misbehaving is worth knowing about. At most 20,000 counters are kept, in memory and in @@ -760,89 +696,37 @@ see; a low `SWWAF_ASN_LIMIT_PERCENT` for its AS number is the response. ## Rule files `smallwebwaf` reads every `*.rules` file in `SWWAF_RULES_DIR` and checks each -request against them, unless the client is in `SWWAF_ALLOW_NETS`. The files are -plain text meant to be edited by hand, so that a new scanning pattern seen in -the request log becomes a rule in one line. `smallwebwaf` watches the directory: -a file edited, added or removed there takes effect while it runs. +request against them, unless the client is in `SWWAF_ALLOW_NETS`. +`docs/rules.md` gives the format, the targets and actions, the errors, and the +default file. -- One rule per line, four fields separated by spaces or tabs. The fourth field - runs to the end of the line: - - ``` - - ``` - -- Blank lines and lines starting with `#` are ignored. -- `id`: a short name of letters, digits, `-` and `_`, unique across all files. - It appears in the request log, the metrics, the alerts and the ban notes. -- `target`: what the regex is matched against. - - `path`: the URL path without the query, both as received and once - percent-decoded, so an encoded probe cannot slip past. - - `query`: the raw query string. - - `uri`: the path and query together, both as received and once - percent-decoded. - - `method`, `host`, `user_agent`, `referer`. - - `header:`: any one request header but `Host` and - `Transfer-Encoding`, which Go's HTTP server takes out of every request. - - Request bodies are not available to rule files. Reading them is the Core - Rule Set's job, once `SWWAF_WAF_BODY_LIMIT` switches it on. -- `action`: - - `log`: note the match in the request log and do nothing else. - - `block`: refuse the request with `403`, without a ban (see "Bans"). - - `ban`: the request is a clear sign of attack (see "Bans"). Use it only for - requests no real visitor sends. Anchor a path at the site root (`^/`): a - file of the same name deeper in a site can be ordinary content, such as a - file in a gitea repository. A visitor banned by mistake is let back in by - lifting the ban. -- `regex`: Go regular expression syntax (RE2). It has no backreferences or - lookaround. In exchange matching time is linear in the input, so no rule can - be made to stall the proxy. `(?i)` at the front makes a rule ignore case. A - rule matches if the regex matches anywhere in the target; anchor with `^` and - `$` when that is not wanted. -- Files are read in name order (`00-default.rules` before `50-gitea.rules`), - rules in line order. -- Rules are compiled when their file is read and held in memory. At start, a - line that does not parse, a regex that does not compile, or a duplicate id - stops the process with a message naming the file and line. So does a - `SWWAF_RULES_DIR` that does not exist. While running, the same faults leave - the rules as they were, and the log and one `file_error` alert name the file - and line. Once the file is fixed, it is read again. An empty directory is not - an error: the log says that no rules were loaded. To run without rule files, - mount an empty directory or set `SWWAF_RULES_ENABLED=false`. -- The image ships a default file in `SWWAF_RULES_DIR`. The app's Dockerfile can - copy files of its own into it, beside the default. Mounting a directory over - it replaces the defaults; mounting single files into it adds to them. Many - editors save a file by replacing it, and a container keeps seeing the old one - when the file alone is mounted. So rules meant to be edited while - `smallwebwaf` runs belong in a mounted directory. The lookup database is - mounted the same way, for the same reason. - -The default file the image ships: - -``` -# 00-default.rules: probes no real visitor sends, anchored at the site root - -# id target action regex -env-file path ban (?i)^/\.env(\.[a-z]+)?$ -vcs-dir path ban (?i)^/\.(git|svn|hg|bzr)(/|$) -secrets-dir path ban (?i)^/\.(aws|ssh|docker|kube)/ -secret-file path ban (?i)^/\.(htpasswd|htaccess|npmrc|netrc|pgpass|git-credentials|bash_history|DS_Store)$ -editor-dir path ban (?i)^/\.(vscode|idea)/ -backup-file path ban (?i)^/[^/]+\.(php(\.[a-z0-9]+|~)|sql(\.[a-z0-9]+)?)$ -log-file path ban (?i)^/(debug|error|access)\.log$ -compose-file path ban (?i)^/(docker-)?compose\.ya?ml$ -php-shell path ban (?i)^/(shell|c99|r57|wso|alfa)\.php$ -scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|masscan|zgrab|wpscan)\b -path-traversal uri block (\.\./){2,} -empty-agent user_agent log ^$ -``` - -It is kept short and limited to patterns that are wrong for every app. In front -of gitea, `/.env` is a probe, while -`///src/branch/main/.env.example` is a file in a repository that -any visitor may open. Anything app-specific, such as a rule banning WordPress -probes in front of gitea, belongs in a file the app's Dockerfile adds or the -deployer mounts. +- The files are plain text meant to be edited by hand, so that a new scanning + pattern seen in the request log becomes a rule in one line. `smallwebwaf` + watches the directory: a file edited, added or removed there takes effect + while it runs. +- A rule's regex is Go's RE2 syntax. It has no backreferences or lookaround; in + exchange matching time is linear in the input, so no rule can be made to stall + the proxy. +- `path` and `uri` are matched both as received and once percent-decoded, so an + encoded probe cannot slip past. Request bodies are not available to rule + files; reading them is the Core Rule Set's job. +- `block` refuses the request without a ban; `ban` makes the request a clear + sign of attack (see "Bans"). `ban` is for requests no real visitor sends, + anchored at the site root. A file of the same name deeper in a site can be + ordinary content, such as a file in a gitea repository. A visitor banned by + mistake is let back in by lifting the ban. +- At start, a file that does not parse, a regex that does not compile, or a + duplicate id stops the process with a message naming the file and line. So + does a `SWWAF_RULES_DIR` that does not exist. While running, the same faults + leave the rules as they were, and the log and one `file_error` alert name the + file and line. An empty directory is not an error. +- The image ships a default file, kept short and limited to patterns that are + wrong for every app, with its path rules anchored at the site root. Anything + app-specific belongs in a file the app's Dockerfile adds or the deployer + mounts. Rules meant to be edited while `smallwebwaf` runs belong in a mounted + directory, not a mounted file. Many editors save a file by replacing it, and a + container keeps seeing the old one when the file alone is mounted. The lookup + database is mounted the same way, for the same reason. ## Persistent state @@ -850,160 +734,61 @@ All state lives in memory, and the files in `SWWAF_STATE_DIR` hold a copy of all of it, so an orderly stop loses nothing. The one exception is log lines still waiting to be sent to `SWWAF_LOG_REMOTE_URL`, which stdout has already carried. The files are meant for people as well: an admin can read or edit them at any -time, and the running `smallwebwaf` takes the edit in. +time, and the running `smallwebwaf` takes the edit in. `docs/state-files.md` +lists the files, what each holds, their sizes, and how an edit is taken in. -The files, each holding one kind of state: - -- `bans.json`: active and past bans, up to `SWWAF_MAX_BANS` that `smallwebwaf` - made and every ban an admin made. Per entry: - - the netblock, start and expiry (`null` for permanent); - - the cause: `attack`, `limit`, `admin` or `crowdsec`; - - a short reason; - - when it was lifted, if an admin lifted it; - - the notes below. -- `clients.json`: per client, its minute, hour and day counters and its history - since it was first seen. The history holds: - - the first and last time seen; - - the AS number, AS name and country last looked up, and when; - - total requests, and how many were forwarded and how many refused; - - bytes in each direction; - - responses by status class; - - offences by kind. - - Clients that were never banned are kept too, so every client's history - survives a restart. - -- `lookups.json`: GeoJS answers, one per client: the AS number, AS name and - country, when GeoJS was asked and when the answer was last used. -- `reputation.json`: - - the last good copy of each list fetched from a URL; - - the cached DNS blocklist verdicts and AbuseIPDB scores, each with the time - it was fetched; - - the AbuseIPDB checks spent today, so that a restart does not reset the - daily budget. -- `alerts.json`: - - the anomaly counters; - - for each event and netblock, source, file or thing counted, when it was - last sent, and the count of repeats held back since; - - the alerts sent this hour; - - the alerts still waiting to be sent. - -Ban notes hold what an admin needs to decide whether to lift a ban, drawn from -what `smallwebwaf` already knows. Nothing is looked up to fill them: - -- the AS number, AS name and country, added when the lookup answers (empty when - lookups are off); -- what was broken: the rule id and target that matched, or the trap path, or the - limit, its window and the count reached; -- the client's limit percentage, with the setting that gave it; -- the reputation sources that listed the client; -- the request that caused the ban: time, method, host, path with its query - string, status and user agent, each text cut to 256 bytes; -- the netblock's total requests since it was first seen, and the requests - refused under this ban so far, kept up to date while the ban lasts; -- how many earlier bans of each kind the netblock has had. - -Each file is rewritten whole, so each has a bound. `clients.json` takes about 1 -KiB per client, about 20 MiB at the default `SWWAF_MAX_TRACKED_CLIENTS`. Written -every 15 minutes, that is under 2 GiB of disk writes a day. `bans.json` takes -about 1.2 KiB per ban, at most about 2.5 KiB, so about 6 MiB at the default -`SWWAF_MAX_BANS`. With a full file, a hundred new bans a day come to about 600 -MiB of writes. `lookups.json` takes about 150 bytes per answer, under 1.5 GiB of -writes a day when full. The other two files are usually a few MiB or less. -Writing a file of these sizes takes well under a second, in the background, so -rewriting whole files needs nothing cleverer. - -Format, so that a person can open `bans.json` in an editor, find an address, and -remove or add an entry: - -- indented JSON with a top-level `version` number; -- entries sorted by client address; -- times in RFC 3339 UTC; -- durations and sizes as plain numbers, with the unit in the field name; -- `clients.json` and `lookups.json` put each entry on one line instead. That - halves their size and lets `grep` show everything about one client. - -Writing: - -- serialise from a snapshot taken under the lock, so requests are not held up - while the file is written; -- write to a temporary file in the same directory, sync it, rename it over the - real name, sync the directory. A crash at any point leaves either the old - complete file or the new complete file, never a partial one; -- changes that only update the counts in a ban's notes wait for the write every - `SWWAF_STATE_COUNTER_INTERVAL`, as the other four files do. All five are - written on orderly shutdown (`SIGTERM`); -- before writing a file, `smallwebwaf` checks whether it changed on disk since - it last read or wrote it. If it did, `smallwebwaf` takes that edit in first, - so an admin's edit is never overwritten. - -Reading at start: - -- Active bans go into an in-memory prefix lookup; everything else into maps. - Counter buckets and cache entries whose time has passed are discarded. -- A missing file means empty state, and is normal on first run. -- A file that does not parse, or has an unknown `version`, stops the process - with a message naming the file and position. Starting with empty state would - silently forgive every repeat offender. Since writes are atomic, a broken file - can only come from a hand edit, which the editor should hear about. -- `SWWAF_STATE_DIR` not writable at start: the process exits. - -Edits while running: - -- `smallwebwaf` watches `SWWAF_STATE_DIR` and notices a file that is edited, - replaced or added. It tells its own writes from an admin's by comparing the - file with what it last wrote. -- An edit that parses is taken in at once: what the file says replaces what - `smallwebwaf` held for that file. Removing a ban's entry from `bans.json` - lifts the ban, an admin's own included; adding an entry bans. Changes - `smallwebwaf` made after the admin opened the file, such as a new ban, are - lost when the admin saves over them. Most editors warn when a file changed on - disk while it was open. -- An edit that does not parse does not stop the running `smallwebwaf`. It keeps - the state it has and renames the edited file to `.bad` (for example - `bans.json.bad`), so the edit is kept. It writes the file again from memory, - logs the file and the position of the error, and sends them in one - `file_error` alert. The admin fixes the `.bad` file and moves it back. - -A write that fails while running: - -- state stays correct in memory; -- the failure is logged and counted in the metrics; -- one `file_error` alert is sent per cooldown; -- the write is retried at the next write. - -What a hard kill can lose: - -- ban changes from the last `SWWAF_STATE_WRITE_DELAY` (10 seconds); -- everything else from the last `SWWAF_STATE_COUNTER_INTERVAL` (15 minutes): - - counts and history, and the counts in ban notes; - - GeoJS answers; - - reputation verdicts and the AbuseIPDB count; - - anomaly counters and alert cooldowns. - -Losing the volume loses ban history, not service. +- Each file holds one kind of state, as JSON with: + - a top-level `version` number; + - entries sorted by client address; + - times in RFC 3339 UTC; + - durations and sizes as plain numbers, with the unit in the field name. +- `bans.json` is indented, so that a person can open it in an editor, find an + address, and remove or add an entry. `clients.json` and `lookups.json` put + each entry on one line, which halves their size and lets `grep` show + everything about one client. +- Clients that were never banned are kept too, so every client's history + survives a restart. +- Each file is rewritten whole, so each has a bound given by its setting. + Writing a file of these sizes takes well under a second, in the background, so + rewriting whole files needs nothing cleverer. +- A write serialises from a snapshot taken under the lock, so requests are not + held up. It goes to a temporary file in the same directory, which is synced + and renamed over the real name. A crash at any point leaves either the old + complete file or the new complete file, never a partial one. Before writing a + file, `smallwebwaf` takes in any edit made to it on disk, so an admin's edit + is never overwritten. +- At start, a file that does not parse, or has an unknown `version`, stops the + process with a message naming the file and position. Starting with empty state + would silently forgive every repeat offender. Since writes are atomic, a + broken file can only come from a hand edit, which the editor should hear + about. A missing file means empty state, and is normal on first run. +- While running, an edit that parses replaces what `smallwebwaf` held for that + file. An edit that does not parse never stops the running `smallwebwaf`. It + keeps the state it has and sets the edited file aside as `.bad`. It + writes the file again from memory, and names the file and the position of the + error in the log and in one `file_error` alert. +- A write that fails leaves the state correct in memory. It is logged, counted + in the metrics and sent as one `file_error` alert per cooldown, and retried at + the next write. +- A hard kill loses the changes since the last write. Losing the volume loses + ban history, not service. ## Request log Every request gets one JSON object on one line on stdout, refused ones included, apart from those Go's HTTP server ends before `smallwebwaf` sees them (see "Size -and time limits"). `docs/request-log.md` lists the fields. Bodies are never -logged. Query strings are logged as received; an app that carries secrets in -query strings needs that fixed in the app. The process's own messages share the -stream as JSON lines with `"type":"process"`; request lines carry -`"type":"request"`. +and time limits"). `docs/request-log.md` lists the fields and describes the +syslog sending. Bodies are never logged. Query strings are logged as received; +an app that carries secrets in query strings needs that fixed in the app. The +process's own messages share the stream as JSON lines with `"type":"process"`; +request lines carry `"type":"request"`. -When `SWWAF_LOG_REMOTE_URL` is set, the same lines are also sent there, so a -deployment can stop depending on docker's log handling while `docker logs` keeps -working: - -- each line is the message of an RFC 5424 syslog record, with octet-counted - framing on TCP and TLS; -- sending happens on its own goroutine from a bounded buffer - (`SWWAF_LOG_REMOTE_BUFFER`). An unreachable or slow server never delays a - request and never stops stdout. The sender reconnects with backoff, drops the - oldest lines when the buffer is full, and counts the drops in the metrics. UDP - gives no delivery signal at all and is offered only for compatibility. +When `SWWAF_LOG_REMOTE_URL` is set, the same lines are also sent there as RFC +5424 syslog records. So a deployment can stop depending on docker's log handling +while `docker logs` keeps working. Sending happens on its own goroutine from a +bounded buffer: an unreachable or slow server never delays a request and never +stops stdout. UDP gives no delivery signal at all and is offered only for +compatibility. ## Metrics endpoint @@ -1017,24 +802,17 @@ the request log and `GET /_smallwebwaf/clients/`. A request whose path starts with `/_smallwebwaf/` is for `smallwebwaf` itself: it answers it and never passes it to the app. The prefix carries the tool's name, so it takes no path an app uses. Admins and scrapers reach these endpoints -through traefik, like any other request. +through traefik, like any other request. `docs/admin-endpoints.md` lists the +endpoints and what each answers. -- `GET /_smallwebwaf/healthz`: answers `200` with `ok` to anyone, without a +- `GET /_smallwebwaf/healthz` answers `200` with `ok` to anyone, without a token, while `smallwebwaf` is running. It does not ask the app; the container's health check does. It is answered before any check, so a health checker never needs an exemption and is never refused, for example by `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES`. -- `GET /_smallwebwaf/metrics`: the metrics; needs `SWWAF_METRICS_TOKEN`. -- `GET /_smallwebwaf/bans`, `POST /_smallwebwaf/bans` (a client or netblock, a - duration, a reason) and `DELETE /_smallwebwaf/bans/`: need - `SWWAF_ADMIN_TOKEN`. -- `GET /_smallwebwaf/clients/`: needs `SWWAF_ADMIN_TOKEN`. It answers with - the client's current counters, its history (with its lookup result and - offences), and its bans with their notes. It answers "why was this address - refused" and "what has this netblock been doing". -- A token is sent as `Authorization: Bearer ` (see "Tokens" under - "Settings"). While a token is unset, the endpoints that need it answer `404`, - as does any other path under the prefix. +- Every other endpoint needs its token, sent as `Authorization: Bearer ` + (see "Tokens" under "Settings"). While a token is unset, the endpoints that + need it answer `404`, as does any other path under the prefix. - Apart from the health check, these requests go through every check that any other request goes through. They are answered at the point where another request would be forwarded to the app: a banned or refused client stays @@ -1044,20 +822,6 @@ through traefik, like any other request. `SWWAF_ERROR_BURST_THRESHOLD` guesses in a minute. A client in `SWWAF_ALLOW_NETS` skips the checks but still needs the token. -## Alert webhook schema - -One JSON object per alert: - -- `instance`, `time`, `event` (one of the `SWWAF_ALERT_EVENTS` values, or - `summary` for the hourly summary); -- `client`, `netblock`, `asn`, `as_name`, `country`; -- `reason`: a short human-readable sentence; -- `detail`: event-specific fields, for example `window`, `count`, `limit`, - `limit_percent`, `rule_ids`, `path`, `ban_expires`, for a ban its notes, and - for `file_error` the file and the position of the error; -- `suppressed_repeats`: the number of identical alerts held back by the - cooldown. - ## Deployment `smallwebwaf` runs inside the app's own container: the app's Dockerfile builds diff --git a/docs/admin-endpoints.md b/docs/admin-endpoints.md new file mode 100644 index 0000000..85563f0 --- /dev/null +++ b/docs/admin-endpoints.md @@ -0,0 +1,35 @@ +# Admin endpoints + +`smallwebwaf` answers these requests itself, on the app's own address and +through traefik like any other request. Apart from the health check, each needs +a token, sent as `Authorization: Bearer `, and goes through every check +any other request goes through. It is answered where another request would be +passed to the app: a banned client stays refused, and each request counts toward +rate limiting. A missing or wrong token is answered `401`, in `observe` mode +too. Such a refusal counts toward the error burst, so a client guessing tokens +is soon banned. While a token is unset, its endpoints answer `404`. + +- `GET /_smallwebwaf/healthz`: `200` and `ok`, to anyone, before any check and + without asking the app. +- `GET /_smallwebwaf/metrics`: the metrics, for `SWWAF_METRICS_TOKEN` (see + [`metrics.md`](metrics.md)). +- `GET /_smallwebwaf/bans`: every ban held, under `bans`, each as `bans.json` + lists it. Needs `SWWAF_ADMIN_TOKEN`, as do the three below. +- `POST /_smallwebwaf/bans`: bans a netblock. The body is a JSON object of + `netblock`, `duration` and, if you like, `reason`. `netblock` is a netblock + such as `203.0.113.0/24`, or a client's address. A client's address bans its + IPv4 netblock (`SWWAF_BAN_SCOPE_V4_PREFIX`) or its IPv6 group. `duration` is a + duration such as `1h` or `7d`, or `permanent`. The ban starts at once, with + the cause `admin`. A bad body is answered `400`, saying what is wrong: + - one that is not such an object, or has another field; + - one longer than 4 KiB; + - one that names an IPv4-mapped netblock, or an address with a zone. +- `DELETE /_smallwebwaf/bans/`: lifts every active ban covering the + address ``. It answers `404` when none is active. +- `GET /_smallwebwaf/clients/`: what `smallwebwaf` knows of the client. + Under `client`, its counters and history as `clients.json` holds them, or + `null` when it has never been seen. Under `bans`, every ban covering the + address, with its notes. + +An admin whose own address is banned lifts that ban by editing `bans.json` (see +[`bans.md`](bans.md)). diff --git a/docs/alerts.md b/docs/alerts.md new file mode 100644 index 0000000..6577753 --- /dev/null +++ b/docs/alerts.md @@ -0,0 +1,149 @@ +# Alerts + +Each alert goes to each destination you name. `SWWAF_ALERT_WEBHOOK_URL` gets it +as one JSON object. `SWWAF_ALERT_SLACK_WEBHOOK_URL` and `SWWAF_ALERT_NTFY_URL` +get a message made from it. The settings are under "Alert destinations" in +[`settings.md`](settings.md). An alert is one of these events, and is sent when +its event is in `SWWAF_ALERT_EVENTS`: + +- `ban`: a ban `smallwebwaf` made. +- `permanent_ban`: a permanent ban it made, or an existing ban made permanent by + a request. +- `waf_block`: the Core Rule Set scored a request at or over the threshold. In + `block` mode the request was refused. In `detect` mode it was not, and the + alert's `detail` gives `mode`. +- `anomaly`: a count over an anomaly threshold. +- `reputation_hit`: a request from a client on a blocklist, the CrowdSec + decision list or a DNS blocklist, or with an AbuseIPDB score that is a hit. + One alert per source. +- `source_failure`: one of: + - GeoJS failing; + - a list that cannot be fetched; + - a DNS blocklist or AbuseIPDB failing or refusing a query; + - the day's AbuseIPDB checks used up. +- `file_error`: one of: + - a rule file or state file edit with an error; + - a state file that cannot be written; + - a replacement lookup database that cannot be read. +- `summary`: the hourly summary described under "Repeats and the hourly cap" + below. + +The bans you make raise no alert. In `observe` mode, a request that would have +made a ban still raises the ban's alert, with `mode` set to `observe` in its +`detail`. + +## The JSON object + +This is the alert for a ban for a broken rate limit, shown indented; it is sent +on one line: + +```json +{ + "instance": "fsn1app1/gitea", + "time": "2026-10-06T12:00:00.123461Z", + "event": "ban", + "client": "203.0.113.9", + "netblock": "203.0.113.9/32", + "asn": "", + "as_name": "", + "country": "", + "reason": "requests per minute over the limit of 1000", + "detail": { + "ban_expires": "2026-10-06T13:00:00.123Z", + "cause": "limit", + "notes": { + "asn": "", + "as_name": "", + "country": "", + "kind": "requests", + "limit": 1000, + "window": "minute", + "count": 1001, + "request": { + "time": "2026-10-06T12:00:00.123456789Z", + "method": "GET", + "host": "app.example", + "path": "/owner/repo/commits/branch/main?page=812", + "status": 403, + "user_agent": "scraper/1.0" + }, + "requests": 5210, + "refused": 0, + "earlier_bans": { + "limit": 0, + "attack": 0, + "admin": 0, + "crowdsec": 0 + } + } + }, + "suppressed_repeats": 0 +} +``` + +- `instance` is `SWWAF_INSTANCE_NAME`. `time` is when the alert was raised. + `event` is one of the events above. +- `client` is the address whose request raised the alert. `netblock` is the + ban's netblock, the netblock an anomaly counted, or the client's group. Both + are empty for `source_failure` and `file_error`. `asn`, `as_name` and + `country` are the client's as known when the alert was raised. They are empty + when GeoJS had not answered yet. +- `reason` is a short sentence, such as + `requests per minute over the limit of 1000` or `listed by a blocklist`. +- `detail` is what is particular to the event: + - for a ban, its `cause`, `ban_expires` and `notes` (see + [`bans.md`](bans.md) for the notes); + - for an anomaly, the `scope`, `window`, `kind`, `count` and `threshold`; + - for a reputation hit, the `source` and, for AbuseIPDB, the `score`; + - for a Core Rule Set match, the `rule_ids`, `score`, `method` and `path`; + - for a source failure, the `source` and `error`; + - for a file error, the `file` and `error`. +- `suppressed_repeats` is how many repeats the cooldown held back before this + alert. + +## Slack and ntfy + +Slack and ntfy get a message with a title and a text. The title is the instance +and the event, such as `fsn1app1/gitea: ban`. The text is the `reason`, then one +line for each of these that the alert has: + +- the client, the netblock and the country; +- the file, the source, the error and the mode from `detail`; +- the suppressed repeats, if any. + +Slack gets the title in bold and the text below it, with `&`, `<` and `>` +escaped. ntfy gets the text as the message, with these headers: + +- `Title`: the title; +- `Authorization`: the token, as `Bearer `, while + `SWWAF_ALERT_NTFY_TOKEN` is set; +- `Priority` and `Tags`: by event, as listed below. + +The priority and tag by event: + +- `ban`: priority `default`, tag `no_entry`. +- `permanent_ban`: priority `high`, tag `no_entry`. +- `waf_block`: priority `default`, tag `shield`. +- `anomaly`: priority `high`, tag `chart_with_upwards_trend`. +- `reputation_hit`: priority `low`, tag `label`. +- `source_failure` and `file_error`: priority `high`, tag `warning`. +- `summary`: priority `default`, tag `bar_chart`. + +## Repeats and the hourly cap + +A repeat is an alert for the same event on the same netblock, source, file or +thing counted, less than `SWWAF_ALERT_COOLDOWN` after the last. It is held back +and counted, and the next alert sent gives the count. Past +`SWWAF_ALERT_MAX_PER_HOUR` alerts in an hour, the rest are held back and counted +by event. Once the hour ends, one alert with the event `summary` gives those +counts. It also counts the repeats that a cooldown held back and that no later +alert reported. + +## Sending + +Each destination has a queue of at most 1000 alerts, sent one at a time, so a +slow destination holds up nothing. A `2xx` answer takes an alert. A `4xx` other +than `408` and `429` refuses it, and it is dropped. Anything else is a failure: +the alert is sent again a second later, and twice as long after each further +failure, up to a minute. `alerts.json` keeps the queues and cooldowns across a +restart (see [`state-files.md`](state-files.md)). diff --git a/docs/bans.md b/docs/bans.md new file mode 100644 index 0000000..bd0f4e6 --- /dev/null +++ b/docs/bans.md @@ -0,0 +1,84 @@ +# Bans + +How bans are made, what they cover and how long they last is under "Bans" in +[`SPEC.md`](../SPEC.md); the settings are under "How long bans last and what +they cover" in [`settings.md`](settings.md). This file says what a ban's notes +hold, and how to make, keep or lift a ban. + +## The notes + +Each ban carries notes for deciding whether to lift it, drawn from what +`smallwebwaf` already knows. Nothing is looked up to fill them: + +- the AS number, AS name and country, added when the lookup answers (empty when + lookups are off); +- what was broken: the rule id and target that matched, or the trap path, or the + limit, its window and the count reached; +- the client's percentage of the limit, with the setting that gave it; +- the reputation sources the client is on; +- the request that caused the ban: time, method, host, path with its query + string, status and user agent, each text cut to 256 bytes; +- the netblock's requests since it was first seen, and how many this ban has + refused so far, kept up to date while the ban lasts; +- how many earlier bans the netblock had, by cause. + +## Managing bans by editing `bans.json` + +`bans.json` in the state directory holds every ban, indented to be read. A +permanent ban's `expires` is `null`. `smallwebwaf` watches the file and takes in +an edit as soon as it is saved (see [`state-files.md`](state-files.md)). + +To ban a netblock, add an entry with its `netblock`, `start` and `expires`. +`expires` is `null` for a ban that never ends. `reason`, `notes` and `cause` may +be left out; the cause is then `admin`: + +```json +{ + "version": 1, + "bans": [ + { + "netblock": "203.0.113.0/24", + "start": "2026-10-06T12:00:00Z", + "expires": null, + "reason": "probes for logins" + } + ] +} +``` + +To keep a ban `smallwebwaf` made, so that it is never dropped, set its `cause` +to `admin`. To lift a ban, add `"lifted"` with the time you lift it, such as +`"lifted": "2026-10-06T13:00:00Z"`. The ban then refuses nothing and does not +make the netblock's next ban longer. It is kept with its notes. To forget a ban, +delete its entry. A ban whose cause is `attack` becomes permanent at the first +request it refuses. One whose cause is `admin` or `crowdsec` does not. + +An edit that cannot be read does not stop the running `smallwebwaf`. It keeps +what it holds. At the file's next write it renames your file to `bans.json.bad`, +writes the file again from memory, and logs and alerts the error. Mend the +`.bad` file and move it back. + +## Managing bans through the admin endpoints + +With `SWWAF_ADMIN_TOKEN` set, and the token in `$TOKEN`: + +```sh +# Every ban. +curl -H "Authorization: Bearer $TOKEN" https://app.example/_smallwebwaf/bans + +# Ban 203.0.113.0/24 for seven days. +curl -H "Authorization: Bearer $TOKEN" \ + --json '{"netblock": "203.0.113.0/24", "duration": "7d", "reason": "probes for logins"}' \ + https://app.example/_smallwebwaf/bans + +# Lift the bans on 203.0.113.9. +curl -H "Authorization: Bearer $TOKEN" -X DELETE \ + https://app.example/_smallwebwaf/bans/203.0.113.9 + +# What smallwebwaf knows of 203.0.113.9. +curl -H "Authorization: Bearer $TOKEN" \ + https://app.example/_smallwebwaf/clients/203.0.113.9 +``` + +See [`admin-endpoints.md`](admin-endpoints.md) for what each answers. An admin +whose own address is banned lifts that ban by editing `bans.json`. diff --git a/docs/core-rule-set.md b/docs/core-rule-set.md new file mode 100644 index 0000000..066e749 --- /dev/null +++ b/docs/core-rule-set.md @@ -0,0 +1,31 @@ +# The Core Rule Set + +The OWASP Core Rule Set 4.25.0 runs on each request after the rule files, unless +`SWWAF_WAF_MODE` is `off`. It reads the method, the URL with its query, and the +headers. Once `SWWAF_WAF_BODY_LIMIT` is set, it also reads a body that is form +data, multipart, JSON or XML. No answer is inspected. The settings are under +"Attack detection" in [`settings.md`](settings.md). The same section of +[`SPEC.md`](../SPEC.md) says why bodies are not read by default, and gives the +six changes `smallwebwaf` makes to the Core Rule Set, with the reason for each. + +Each rule that matches adds to the request's score. A score at or over +`SWWAF_WAF_ANOMALY_THRESHOLD` is a match. A match is refused with `403` while +`SWWAF_WAF_MODE` is `block`, and let through while it is `detect`. Either way +the log line gives the rule ids and the score, and a `waf_block` alert is sent. +A request with more than 1000 query parameters or body fields adds 5 to its +score, since Coraza reads only the first 1000. The Core Rule Set inspects only a +request that every earlier check let through. It skips a path under +`SWWAF_WAF_EXEMPT_PATHS` and a client in `SWWAF_ALLOW_NETS`. An engine error on +a request forwards the request, and is logged and counted. + +Body inspection suits apps whose forms carry no code. In front of a code forge +it refuses issue and comment text, wiki pages and files saved in the web editor +that hold shell commands or code. The log line names the rule ids, which +`SWWAF_WAF_DISABLED_RULES` can switch off. A body the Core Rule Set cannot parse +adds 5 to the score. So does a multipart body cut by the limit in the middle of +a part's header. The client has until `SWWAF_CLIENT_REQUEST_TIMEOUT` runs out to +send the part that is read. Any other body, such as a git push, reaches the app +uninspected. + +[`gitea.md`](gitea.md) records what the Core Rule Set still refuses in front of +gitea, with and without body inspection. diff --git a/docs/lookup.md b/docs/lookup.md new file mode 100644 index 0000000..45909d0 --- /dev/null +++ b/docs/lookup.md @@ -0,0 +1,79 @@ +# GeoJS and the lookup database + +`smallwebwaf` looks up each client's AS number and country, through GeoJS by +default or in a database file on your own host. The settings are under "Looking +up the AS number and country" in [`settings.md`](settings.md); the reasons for +the defaults are under "Lookup of AS number and country" in +[`SPEC.md`](../SPEC.md). + +## GeoJS + +By default `smallwebwaf` asks GeoJS, a free web service that needs no account, +about every new visitor's address. It asks whether or not a setting uses the +answer, since the log line, the history, the ban notes and the metrics carry the +AS number and country. These addresses are never sent: + +- one in `SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS`; +- one under a ban; +- a private, loopback or link-local address, which no source can place. + +For an IPv6 visitor, GeoJS is asked about the first address of its group. +Addresses are asked about together, up to 200 in one request +(`https://get.geojs.io/v1/ip/geo.json?ip=a,b,c`), one request at a time. Each +answer's `country_code`, `asn` and `organization_name` are read. An answer +without a `country_code`, or with the `asn` `64512`, counts as unknown. Each +answer is kept for seven days, in memory and in `lookups.json`, separately from +the table of clients. Up to 100,000 answers are kept; past that, the answer used +longest ago goes first. + +A request waits for its client's first answer only when a setting needs the +answer before the request goes on. It waits up to `SWWAF_LOOKUP_TIMEOUT`. The +settings that make it wait: + +- a country list; +- `SWWAF_ADD_LOOKUP_HEADERS`; +- any setting under "Lower limits by AS number and country" in + [`settings.md`](settings.md). + +Otherwise the request goes on at once, logged without the answer. The answer +reaches the client's history and ban notes when it comes. A client whose answer +has not come in time counts as from an unknown country until it comes, and its +later requests do not wait. While GeoJS fails, new visitors count as from an +unknown country, which `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses. GeoJS is +then left alone for a second, and twice as long after each further failure, up +to five minutes, and one `source_failure` alert is sent per cooldown. + +## The database file + +To keep your visitors' addresses on your own host, set +`SWWAF_LOOKUP_SOURCE=off`, or set it to `file`. `file` looks every client up in +the free IPinfo Lite database (`ipinfo_lite.mmdb`) at `SWWAF_LOOKUP_DB_PATH`, +reading its `asn`, `as_name` and `country_code` fields. You download the +database with an IPinfo account and mount it into the container; `smallwebwaf` +never fetches it and holds no account or token for it. It answers at once, with +nothing sent off the host. A client whose address it does not hold counts as +from an unknown country. + +`smallwebwaf` reads the whole file at start. It reads the file again once it has +gone 2 seconds without a change after you replace it. A replacement it cannot +read is logged and alerted, and the file read before stays in use. Mount the +directory that holds the file, not the file itself. Many editors and download +tools save a file by replacing it, and a container keeps seeing the old file +when only the file is mounted. + +IPinfo updates the database daily, and releases it under the Creative Commons +Attribution-ShareAlike 4.0 licence. It asks for a link to IPinfo on the site +that uses it, such as "IP address data is powered by IPinfo". + +## A client without a country + +A client the lookup cannot place has an unknown country and AS number. That is +one of: + +- a client on a private address; +- one GeoJS or the database does not know; +- one whose GeoJS answer has not come in time. + +`SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses such a client, and the lower +limits give it `SWWAF_UNKNOWN_LIMIT_PERCENT`. Nothing else treats it +differently. diff --git a/docs/reputation.md b/docs/reputation.md new file mode 100644 index 0000000..80c6318 --- /dev/null +++ b/docs/reputation.md @@ -0,0 +1,113 @@ +# Reputation sources + +A reputation source judges a client by what others saw it do. None is on by +default. The settings are under "Reputation sources" in +[`settings.md`](settings.md); the reasons for the defaults are under +"Reputation" in [`SPEC.md`](../SPEC.md). The sources are checked in this order, +and a source the client is on refuses it, lowers its limits, or only notes it, +as `SWWAF_BLOCKLIST_ACTION` and `SWWAF_REPUTATION_ACTION` say. + +## Blocklists + +`SWWAF_BLOCKLIST_URLS` lists text files of addresses and netblocks, one to a +line, as the Spamhaus DROP list is written. Anything after a `;` or `#` is left +out. An IPv4-mapped address such as `::ffff:192.0.2.0/120` is read as the IPv4 +netblock it stands for. + +Each list is fetched again `SWWAF_BLOCKLIST_REFRESH` after its last try, one +list at a time. The file of `SWWAF_ASN_LIMIT_PERCENT_URL` is too. +`reputation.json` keeps when each was last tried, so a restart does not fetch a +list again before its time. A fetch fails when: + +- the server answers other than `200`; +- the fetch does not finish within a minute; +- the list is longer than 16 MiB; +- a line is not an address or netblock, or, for the AS number file, not + `AS:`. + +The copy fetched before stays in use, comment lines included. The failure is +logged and raised as a `source_failure` alert. The log, the alerts, the ban +notes and the metrics name each list by its URL, so keep a secret out of it. + +The Spamhaus DROP list is The Spamhaus Project's. Its terms, and what they mean +for `SWWAF_BLOCKLIST_REFRESH` and for several instances on one host, are under +"Reputation" in [`SPEC.md`](../SPEC.md). The copy in `reputation.json` keeps the +list's date and copyright lines. + +## CrowdSec + +While `SWWAF_CROWDSEC_LAPI_URL` is set to the local API of a CrowdSec engine you +run, `smallwebwaf` fetches its decision list every minute. It sends +`SWWAF_CROWDSEC_LAPI_KEY` in the `X-Api-Key` header. The list is kept as a +blocklist is kept. The decisions of the type `ban` on an address or a netblock +are used. Any other, such as a captcha or a country, is left out. A decision +lasts for the `duration` it had left when the engine answered. + +A request is refused when a decision still in force names its client. The +client's netblock is banned until the decision ends, with the cause `crowdsec`. +Only a client that sends a request is banned, so a long list does not fill +`bans.json`. Such a ban is never made permanent, and does not lengthen the +netblock's next ban for a broken limit. If you lift a ban while the decision is +in force, the client's next request makes another. Delete the decision in +CrowdSec first, such as with `cscli decisions delete --ip 203.0.113.9`, and lift +the ban once the list has been fetched again. A fetch fails when: + +- the engine answers other than `200`, a redirect included; +- the fetch does not finish within a minute; +- the answer is longer than 16 MiB; +- the answer is not a list of decisions. + +## DNS blocklists + +Each zone in `SWWAF_DNSBL_ZONES` is asked about a client's own address in the +background, the first time the client is seen. The request goes on at once, and +so does every request from the client until the zone has answered. The name +asked about is the one RFC 5782 gives: the numbers of an IPv4 address reversed, +or the 32 hex digits of an IPv6 address reversed. `192.0.2.99` is asked about as +`99.2.0.192.dnsbl.dronebl.org`. An answer in `127.0.0.0/8` lists the client, but +one in `127.255.255.0/24` is Spamhaus refusing the query. A name that does not +exist does not list the client. These are failures: + +- an answer in `127.255.255.0/24`; +- an answer outside `127.0.0.0/8`; +- a resolver error; +- no answer within `SWWAF_REPUTATION_TIMEOUT`. + +A failure is logged and alerted, and the zone is not asked again for a minute. +Each verdict is kept for `SWWAF_REPUTATION_CACHE_TTL`, in memory and in +`reputation.json`. Up to 100,000 verdicts are kept; past that, the one fetched +longest ago goes first. At most 1,000 queries are under way at once. + +The key of a Spamhaus zone under `dq.spamhaus.net` is its first label. +`smallwebwaf` shows `********` in its place wherever it names the zone. Only +`reputation.json` keeps the zone with its key. + +## AbuseIPDB + +While `SWWAF_ABUSEIPDB_KEY` is set, `smallwebwaf` asks AbuseIPDB's check +endpoint for the abuse confidence score of a client, from 0 to 100. Only a +client that has committed an offence is checked. An offence is one of: + +- a broken limit; +- a `ban` rule match or a trap path; +- a refusal by a `block` rule, the Core Rule Set or a token check. + +The check runs in the background, at the client's first request after the +offence that no ban refuses, so no request waits. The score applies to the +client's later requests. For an IPv6 client, AbuseIPDB is asked about the +address of the request that set off the check, and the score is used for the +whole group. A score at or over `SWWAF_ABUSEIPDB_MIN_SCORE` is a hit, and +`SWWAF_REPUTATION_ACTION` says what is done. Each score is kept for +`SWWAF_REPUTATION_CACHE_TTL`, up to 100,000 scores. + +At most `SWWAF_ABUSEIPDB_DAILY_BUDGET` checks are made in a day, counted in UTC +and kept in `reputation.json`. A restart does not make the budget whole again. +The check that spends the last of the budget is logged and alerted. No client is +checked until the day ends. A check fails when: + +- AbuseIPDB answers other than `200`; +- the answer gives no score; +- no answer comes within `SWWAF_REPUTATION_TIMEOUT`. + +A failure is logged and alerted, and no client is checked for a minute. The key +is sent in the `Key` header and nowhere else, and is never logged. diff --git a/docs/rules.md b/docs/rules.md new file mode 100644 index 0000000..8af131b --- /dev/null +++ b/docs/rules.md @@ -0,0 +1,114 @@ +# Rule files + +`smallwebwaf` reads every `*.rules` file in `SWWAF_RULES_DIR`, in the order of +their names, and checks each request against their rules in that order, unless +the client is in `SWWAF_ALLOW_NETS`. The requirements the files meet are under +"Rule files" in [`SPEC.md`](../SPEC.md). A file whose name starts with `.`, such +as an editor's lock file, is not a rule file. + +## The format + +A rule is a line of four fields separated by spaces or tabs: an id, a target, an +action and a regex. The regex runs to the end of the line. Blank lines and lines +starting with `#` are ignored. + +``` +# id target action regex +env-file path ban (?i)^/\.env(\.[a-z]+)?$ +scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|wpscan)\b +``` + +- The id is letters, digits, `-` and `_`. No two rules share one. The log line, + the metrics, the alerts and a ban's notes name the rule by it. +- The target is what the regex is matched against: + - `path`: the path without the query; + - `uri`: the path and the query together; + - `query`: the query as sent; + - `method`, `host`, `user_agent` and `referer`; + - `header:`: any one request header but `Host` and + `Transfer-Encoding`, which Go's HTTP server takes out of every request. + + `path` and `uri` are matched both as sent and once percent-decoded, so that + an encoded probe does not slip past. A header sent more than once is matched + with its values joined by `, `. A header not sent is empty text. No body is + read; reading bodies is the Core Rule Set's job, once `SWWAF_WAF_BODY_LIMIT` + switches it on. + +- The action is `log`, `block` or `ban`. `log` only notes the match in the log + line. `block` refuses the request with `403` and bans no one; the refusal + counts toward the error burst. `ban` refuses it and bans the client, as a + clear sign of attack. Matching stops at the first rule that refuses. Keep + `ban` for requests no real visitor sends. Anchor a path at the site root with + `^/`: a file of the same name deeper in a site can be ordinary content. +- The regex is in Go's syntax, RE2. It has no backreferences or lookaround, and + takes time linear in the text it reads, so no rule can be made to stall the + proxy. It matches anywhere in the target unless anchored with `^` and `$`. + `(?i)` at its front makes it ignore case. + +## Errors + +These stop the start, with a message naming the file and line: + +- a line that is not a rule; +- a bad `header:` name, or a rule for `Host` or `Transfer-Encoding`; +- a regex that does not compile; +- an id used twice. + +So does a `SWWAF_RULES_DIR` that does not exist. An empty directory is not an +error: the log says that no rules were loaded. + +While it runs, `smallwebwaf` watches the directory. It reads the files again +once the directory has had no change for 2 seconds, so that a file still being +written is read whole. If the files then hold such an error, the rules stay as +they were. The log and a `file_error` alert name the file and line. Once the +file is fixed, it is read again. + +## The default file + +The image ships one file, `share/rules.d/00-default.rules`, installed as +`/etc/smallwebwaf/rules.d/00-default.rules`: + +``` +# 00-default.rules: probes no real visitor sends, anchored at the site root + +# id target action regex +env-file path ban (?i)^/\.env(\.[a-z]+)?$ +vcs-dir path ban (?i)^/\.(git|svn|hg|bzr)(/|$) +secrets-dir path ban (?i)^/\.(aws|ssh|docker|kube)/ +secret-file path ban (?i)^/\.(htpasswd|htaccess|npmrc|netrc|pgpass|git-credentials|bash_history|DS_Store)$ +editor-dir path ban (?i)^/\.(vscode|idea)/ +backup-file path ban (?i)^/[^/]+\.(php(\.[a-z0-9]+|~)|sql(\.[a-z0-9]+)?)$ +log-file path ban (?i)^/(debug|error|access)\.log$ +compose-file path ban (?i)^/(docker-)?compose\.ya?ml$ +php-shell path ban (?i)^/(shell|c99|r57|wso|alfa)\.php$ +scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|masscan|zgrab|wpscan)\b +path-traversal uri block (\.\./){2,} +empty-agent user_agent log ^$ +``` + +It is kept short and limited to patterns that are wrong for every app, with its +path rules anchored at the site root. In front of gitea, `/.env` is a probe, +while `///src/branch/main/.env.example` is a file in a repository +that any visitor may open. Anything app-specific belongs in a file the app's +`Dockerfile` adds or the deployer mounts. A request for `/wp-login.php`, for +example, is a clear sign of attack in front of gitea and an ordinary login in +front of WordPress. Such a rule goes in a file named to sort after the default: + +``` +# 50-gitea.rules: WordPress probes, which gitea never serves +wp-probe path ban (?i)^/(wp-login\.php|xmlrpc\.php|wp-admin/) +``` + +```dockerfile +COPY 50-gitea.rules /etc/smallwebwaf/rules.d/50-gitea.rules +``` + +## Mounting rule files + +A directory mounted over `/etc/smallwebwaf/rules.d` replaces the default file; +single files mounted into it add to it. Many editors save a file by replacing +it, and a container keeps seeing the old one when the file alone is mounted. So +rules to be edited while `smallwebwaf` runs belong in a mounted directory, with +a copy of the default file if the defaults are to stay. The lookup database is +mounted the same way, for the same reason. To run without rules, mount an empty +directory or set `SWWAF_RULES_ENABLED=false`. diff --git a/docs/settings.md b/docs/settings.md new file mode 100644 index 0000000..d9088ba --- /dev/null +++ b/docs/settings.md @@ -0,0 +1,483 @@ +# Settings + +Every setting is an environment variable whose name starts with `SWWAF_`. Every +setting has a default, so none has to be set. Settings are read once, at start. +To change one, restart the container. + +A setting that is set to a value it cannot take stops the start. The process +writes one line, `"type":"process"` with `"msg":"invalid setting"`, naming the +variable and what is wrong with the value. It exits with status 1, and runit +starts it again every second until the value is fixed. The entries below say +which values each setting takes. Any other value stops the start. At start, the +settings in effect are logged on one line. Tokens, keys and the paths of +webhooks are shown as `********`. + +The forms of values: + +- A duration is Go's syntax, with `d` for days: `90s`, `15m`, `24h`, `7d`. Zero + and negative durations are refused. +- A size is a number of bytes with an optional `K`, `M` or `G`, which are powers + of 1024: `1K` is 1024 bytes, `512K`, `100M`, `5G`. +- A count is a whole number of requests, such as `1000`. +- `off` switches a setting off, where its entry below says it takes `off`. +- A list is comma separated, with spaces around items ignored. An empty item is + refused. A list set but empty is an empty list, and replaces the default. +- A netblock is an address and a prefix length, such as `10.0.0.0/8`. A bare + address stands for itself alone. +- A country is a two-letter code of ISO 3166-1, in either case: `de` and `DE` + are the same. `xk` stands for Kosovo. A code that is not a country is refused, + such as `nk` (North Korea is `kp`) or the withdrawn `su`. +- An AS number is `AS` and the number, in either case, such as `AS64496`. +- A percentage is a whole number from 0 to 100. +- A path prefix starts with `/`, such as `/assets/`. + +## Where it listens and what it protects + +- `SWWAF_LISTEN_ADDR` (default `:8080`): where `smallwebwaf` listens, an + optional host and a port. Keep the host empty: `:9000`, never + `127.0.0.1:9000`. The app must not listen on that port. The health check + follows the setting, and traefik's label must name the port. +- `SWWAF_UPSTREAM_URL` (default `http://127.0.0.1:8081`): the app. An `http` or + `https` URL with a host and an optional port, and nothing more: no path, user + or query. +- `SWWAF_INSTANCE_NAME` (default: the container's host name, the first 12 + characters of the container's id): the `instance` name in every log line, + alert and metric. Set it to a name that survives deploys and tells instances + apart, such as `fsn1app1/gitea`. It must be valid UTF-8. While + `SWWAF_ALERT_NTFY_URL` is set, it may hold no control character. While + `SWWAF_LOG_REMOTE_URL` is set and `SWWAF_LOG_REMOTE_APP_NAME` is not, it must + be 1 to 48 printable ASCII characters without a space. +- `SWWAF_MODE` (default `enforce`): `enforce`, or `observe` to ban no one and + log what `enforce` would have done. Observe mode still refuses a request over + a size or time limit, and one without the right token. +- `SWWAF_TRUSTED_PROXIES` (default `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`, + the private address ranges): the netblocks whose `X-Forwarded-For` is + believed, normally where traefik comes from. Set but empty, it trusts nothing: + every TCP peer is then the client. +- `SWWAF_IPV6_GROUP_PREFIX` (default `64`): the length of the netblock that is + one IPv6 client, from 32 to 128. After a change, each IPv6 client starts + afresh. Existing bans keep refusing their netblocks until they end. +- `SWWAF_MAX_TRACKED_CLIENTS` (default `20000`): the most clients kept in memory + and in `clients.json`, a whole number above zero. Past it, the least recently + seen client is dropped first. + +## Time and size limits + +Each request has limits on both sides: between the client and `smallwebwaf`, and +between `smallwebwaf` and the app. Bodies stream through in both directions, +except the part the Core Rule Set reads while `SWWAF_WAF_BODY_LIMIT` is set. + +- `SWWAF_CLIENT_REQUEST_TIMEOUT` (default `60s`): how long a client may take to + send its request line and headers, and then, from the end of the headers, its + body. A duration or `off`. +- `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES` (default `32K`): the largest request + line and headers. A size over `4K`. It cannot be `off`: Go's HTTP server + always has such a limit. +- `SWWAF_CLIENT_IDLE_TIMEOUT` (default `120s`): how long a kept-open connection + may wait for its next request before it is closed. A duration or `off`. The + default is longer than the 90 seconds after which traefik closes an unused + connection, so traefik closes first. +- `SWWAF_CLIENT_RESPONSE_TIMEOUT` (default `30m`): how long the answer may take + to reach the client, from the end of the request to the last byte. A duration + or `off`. +- `SWWAF_UPSTREAM_REQUEST_TIMEOUT` (default `60s`): how long connecting to the + app and sending it the whole request may take. A duration or `off`. +- `SWWAF_UPSTREAM_RESPONSE_TIMEOUT` (default `30m`): how long the app may take + to send its whole answer. A duration or `off`. +- `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body. A size + or `off`. A request that announces a larger body is refused before anything + reaches the app. +- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body. A size + or `off`. + +In front of a code forge, a clone is a response and a push is a request. At the +defaults, an upload over 100 MiB or slower than 60 seconds is cut off. A forge +that takes large uploads needs `SWWAF_REQUEST_MAX_BYTES`, +`SWWAF_CLIENT_REQUEST_TIMEOUT` and `SWWAF_UPSTREAM_REQUEST_TIMEOUT` raised to +fit. + +What the client gets when a limit is passed is under "Size and time limits" in +[`SPEC.md`](../SPEC.md). + +## Netblock lists + +- `SWWAF_ALLOW_NETS` (default empty): netblocks whose clients skip every check, + such as your monitoring or your own network. The size and time limits still + apply to them. They are not looked up. They still need a token for the admin + endpoints. +- `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients are + exempt from rate limiting and the byte limits, such as a machine that talks to + the app all day. Everything else still applies to them. +- `SWWAF_DENY_NETS` (default empty): netblocks whose clients are always refused. + A client in both lists is let through. + +## Rate limiting and byte limits + +- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `1000`), `SWWAF_RATE_LIMIT_PER_HOUR` + (default `10000`) and `SWWAF_RATE_LIMIT_PER_DAY` (default `50000`): the most + requests a client may make in each window. A count or `off`. The defaults are + several times what one busy person produces. A browser loading a heavy page + makes a few hundred requests, and several people often share one address. +- `SWWAF_RATE_LIMIT_EXEMPT_PATHS` (default empty): path prefixes exempt from + rate limiting, such as `/assets/`. Their bytes are not counted either. The + path is matched exactly as sent, without the query, and is not + percent-decoded. `/assets/` matches `/assets/app.js`. It does not match + `/Assets/app.js`, `/%61ssets/app.js` or `/static/assets/app.js`. There are no + wildcards. A path holding `..`, a backslash or an encoded slash (`%2F`) is + never exempt, since the app may read it as a path outside the prefix. +- `SWWAF_BYTES_LIMIT_PER_MINUTE` (default `10G`), `SWWAF_BYTES_LIMIT_PER_HOUR` + (default `20G`) and `SWWAF_BYTES_LIMIT_PER_DAY` (default `50G`): the most + bytes a client may have counted in each window. A size or `off`. Bytes are + counted once the answer has ended, so a byte limit never cuts an answer short. + The client's next request is refused instead. Each default is above the + largest request and answer together, so at the defaults no single download + breaks a limit. +- `SWWAF_BYTES_COUNT` (default `both`): which body bytes the byte limits count: + `response`, `request` or `both`. For a WebSocket, what it carried each way + counts once it closes. + +## Looking up the AS number and country + +- `SWWAF_LOOKUP_SOURCE` (default `geojs`): `geojs`, `file` or `off`. `geojs` + asks the GeoJS web service, which is then told the address of every new + visitor. `file` looks clients up in the IPinfo Lite database file at + `SWWAF_LOOKUP_DB_PATH`, and sends nothing off the host. `off` looks up no + client. With `off`, a setting that needs lookups stops the start: + - a country list; + - `SWWAF_ADD_LOOKUP_HEADERS`; + - an entry in `SWWAF_ASN_LIMIT_PERCENT`, `SWWAF_COUNTRY_LIMIT_PERCENT`, + `SWWAF_ASN_BYTES_PERCENT` or `SWWAF_COUNTRY_BYTES_PERCENT`; + - `SWWAF_UNKNOWN_LIMIT_PERCENT` below 100; + - `SWWAF_ASN_LIMIT_PERCENT_URL`; + - an anomaly threshold per AS number. +- `SWWAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in its + `.mmdb` form, for `SWWAF_LOOKUP_SOURCE=file`. `file` without it stops the + start, and so does setting it with any other source. A file that is missing or + cannot be read stops the start too. +- `SWWAF_LOOKUP_TIMEOUT` (default `1s`): how long a request waits for its + client's first answer from GeoJS when a setting needs it, and how long a + request to GeoJS may take. A duration; it cannot be `off`. +- `SWWAF_ADD_LOOKUP_HEADERS` (default `false`): `true` passes the app the + client's AS number in `X-Client-ASN` and its country in `X-Client-Country`. A + header whose value is unknown is left out. Whatever this setting says, any + such header the client sent is removed. + +See [`lookup.md`](lookup.md) for what each source sends where, and how long +answers are kept. + +## Country lists + +- `SWWAF_DENIED_COUNTRIES` (default empty): countries whose clients are refused, + such as `cn,ru,kp`. +- `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` (default empty): when set, the only + countries whose clients get through, such as `us,de`. A client whose country + cannot be found is refused too. Otherwise every new client would be let in + whenever GeoJS stops answering. A client on a private address has no country, + so it is refused unless it is in `SWWAF_ALLOW_NETS`. + +A country on both lists stops the start. What a refused request gets, and why +the lists are empty by default, is under "Country lists" in +[`SPEC.md`](../SPEC.md). + +## Lower limits by AS number and country + +Clients of the AS numbers and countries listed here get a percentage of every +rate limit and byte limit. A lowered limit is rounded down. "Biased thresholds" +in [`SPEC.md`](../SPEC.md) says why the lists are empty by default and why no +budget is shared by a whole AS number or country. + +- `SWWAF_ASN_LIMIT_PERCENT` (default empty): a list of AS numbers, each with `:` + and a percentage, such as `AS14061:50,AS16276:50,AS45102:25`. An AS number + listed twice is refused. +- `SWWAF_COUNTRY_LIMIT_PERCENT` (default empty): the same by country, such as + `cn:25,ru:50`. +- `SWWAF_ASN_BYTES_PERCENT` and `SWWAF_COUNTRY_BYTES_PERCENT` (default empty): + the same for the byte limits alone. For the AS numbers and countries they + list, they take the place of the two settings above. +- `SWWAF_UNKNOWN_LIMIT_PERCENT` (default `100`): the percentage of every limit + for a client without a country: + - one the lookup cannot place; + - one on a private address; + - one whose answer from GeoJS has not come in time. +- `SWWAF_ASN_LIMIT_PERCENT_URL` (default unset): the `http` or `https` URL of a + text file of such entries, one to a line, such as `AS14061:50`. One list can + then serve several instances. It is fetched and kept as a blocklist is. Where + it and `SWWAF_ASN_LIMIT_PERCENT` both give a percentage, the lower applies. A + URL that is also in `SWWAF_BLOCKLIST_URLS` stops the start. + +A client to which several percentages apply gets the lowest. The log line of +each counted request, and the notes of each ban, say which setting lowered the +limit. + +## Reputation sources + +None is on by default. A reputation source judges a client by what others saw it +do. The defaults judge it by what it does to your service. + +- `SWWAF_BLOCKLIST_URLS` (default empty): blocklists to fetch, such as + `https://www.spamhaus.org/drop/drop.txt`. Each is an `http` or `https` URL + without a user or a fragment. A URL listed twice is refused. +- `SWWAF_BLOCKLIST_REFRESH` (default `24h`): how long a list waits after its + last fetch, or failed try, before it is fetched again. A duration of `1h` or + more; it cannot be `off`. The Spamhaus lists may be fetched no more than once + an hour. +- `SWWAF_BLOCKLIST_ACTION` (default `deny`): what is done with a client on a + blocklist. `deny` refuses its requests. `limit:`, such as `limit:25`, + gives it that percentage of every limit. `log` only notes the list in the log + line and raises an alert. +- `SWWAF_DNSBL_ZONES` (default empty): the DNS blocklists to ask about each + client, such as `dnsbl.dronebl.org`. A DNS blocklist answers a DNS query for a + name made from an address, and so says whether the address is on it. A zone is + a DNS name of at most 189 characters. A Spamhaus zone carries its key, as in + `.xbl.dq.spamhaus.net`, and is shown with the key masked. A zone listed + twice is refused, whatever its case or key. Do not name a zone meant for mail: + such a zone holds the residential addresses ordinary visitors come from. +- `SWWAF_DNSBL_RESOLVER` (default unset): the resolver to ask the zones through, + such as `192.0.2.53` or `[2001:db8::53]:5353`. An IP address with an optional + port, 53 when none is given. Unset, the host's resolver is used. Several zones + refuse queries that come through a public resolver. +- `SWWAF_ABUSEIPDB_KEY` (default unset): the key of your AbuseIPDB account. + While it is set, a client that has committed an offence is checked. A control + character in it is refused. +- `SWWAF_ABUSEIPDB_MIN_SCORE` (default `75`): the least abuse confidence score, + from 0 to 100, that is a hit. +- `SWWAF_ABUSEIPDB_DAILY_BUDGET` (default `900`): the most checks made in a day, + in UTC, a whole number above zero. AbuseIPDB's free accounts may make 1,000. +- `SWWAF_CROWDSEC_LAPI_URL` (default unset): the local API of a CrowdSec engine + you run, such as `http://172.17.0.1:8080`. An `http` or `https` URL without a + user or a fragment. Its decision list is at `v1/decisions` under that URL, and + is fetched every minute. Set without the key, it stops the start. So does a + decision list URL that is also in `SWWAF_BLOCKLIST_URLS`, or that is + `SWWAF_ASN_LIMIT_PERCENT_URL`. +- `SWWAF_CROWDSEC_LAPI_KEY` (default unset): the key the engine gave for + `smallwebwaf`, as `cscli bouncers add smallwebwaf` prints it. Set without the + URL, it stops the start. +- `SWWAF_REPUTATION_ACTION` (default `limit:25`): what is done with a client on + a DNS blocklist, or one whose AbuseIPDB score is a hit. `deny`, + `limit:` or `log`, as for `SWWAF_BLOCKLIST_ACTION`. Such a verdict is + less certain than the Spamhaus DROP list, so by default the client gets a + quarter of every limit. +- `SWWAF_REPUTATION_CACHE_TTL` (default `24h`): how long a zone's verdict or + AbuseIPDB's score is used. A duration; it cannot be `off`. +- `SWWAF_REPUTATION_TIMEOUT` (default `2s`): how long a query to a zone or a + check with AbuseIPDB may take. A duration; it cannot be `off`. + +See [`reputation.md`](reputation.md) for how each source is asked and what it +does with the answer. + +## Attack detection + +- `SWWAF_RULES_DIR` (default `/etc/smallwebwaf/rules.d`): the directory of the + rule files (see [`rules.md`](rules.md)). A directory that cannot be read stops + the start. An empty one is no error. +- `SWWAF_RULES_ENABLED` (default `true`): `false` reads no rule file. +- `SWWAF_TRAP_PATHS` (default empty): paths the app never serves and only + scanners ask for, such as `/wp-login.php,/xmlrpc.php` in front of a code + forge. A request for one bans the client, as a `ban` rule does. Each starts + with `/` and holds no `?`. A trap path is matched whole, without the query, + against the path as sent and once percent-decoded. `/wp-login.php` matches + `/wp-login.php?x=1` and `/%77p-login.php`. It does not match `/wp-login.php/` + or `/blog/wp-login.php`. +- `SWWAF_WAF_MODE` (default `block`): what the Core Rule Set does with a request + it takes for an attack. `block` refuses it with `403`. `detect` lets it + through, logged and alerted. `off` inspects no request. +- `SWWAF_WAF_PARANOIA_LEVEL` (default `1`): the Core Rule Set's paranoia level, + from 1 to 4. Each level up runs more rules. Those find more attacks, and take + more ordinary requests for attacks. +- `SWWAF_WAF_ANOMALY_THRESHOLD` (default `5`): the score at which a request is a + match. Each rule that matches adds to the request's score, and a rule rated + critical adds 5. So by default one such rule is enough. A whole number above + zero, or `off`. With `off` no request is a match, and scores are still logged. +- `SWWAF_WAF_DISABLED_RULES` (default + `920340,920420,920440,920640,930130,930140`): the ids of Core Rule Set rules + to switch off. The log line names the rules a request matched, so a rule that + refuses your app's ordinary requests can be found there. "Attack detection" in + [`SPEC.md`](../SPEC.md) says what the six default ids refuse and why they are + off. A list you give replaces the default, so name these six ids in it to keep + them off. An id must be a whole number above zero. An id from 900000 to 900999 + is refused: those are the rules that set the Core Rule Set up, and + `smallwebwaf`'s own. +- `SWWAF_WAF_EXEMPT_PATHS` (default empty): path prefixes the Core Rule Set does + not inspect, matched as `SWWAF_RATE_LIMIT_EXEMPT_PATHS` is. +- `SWWAF_WAF_BODY_LIMIT` (default `off`): `off` reads no request body. A size, + at most `1G`, switches body inspection on. The Core Rule Set then reads a form + data or multipart body up to that size, and the rest streams on to the app. It + reads a JSON or XML body no larger than that size, since those cannot be read + in part. Any other body, such as a git push, reaches the app uninspected. See + [`core-rule-set.md`](core-rule-set.md). +- `SWWAF_ERROR_BURST_THRESHOLD` (default `30`): the most refusals of one client + in a minute for a rule match, a trap path, a Core Rule Set match or a failed + token check. One more bans the client, as a broken rate limit does. A count or + `off`. The app's own `401` and `404` answers are not counted. + +## How long bans last and what they cover + +- `SWWAF_BAN_RESPONSE` (default `403`): how a refused client is answered: `403`, + `429`, or `close` to close the connection without an answer. A `block` rule + and the Core Rule Set always answer `403`. +- `SWWAF_LIMIT_BAN_DURATION` (default `1h`): the ban for a first broken rate + limit, byte limit or error burst. A duration; it cannot be `off`. +- `SWWAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): a client that breaks a limit + again within this time after its ban ended is banned three times as long. A + duration; it cannot be `off`. +- `SWWAF_MAX_BAN_DURATION` (default `7d`): a ban for a broken limit that would + be longer is permanent instead. A duration; it cannot be `off`. +- `SWWAF_ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of + attack. A duration; it cannot be `off`. +- `SWWAF_MAX_BANS` (default `5000`): the most bans `smallwebwaf` made that are + kept, whether past, active or permanent. A whole number above zero. Bans whose + cause is `admin` are kept besides and never dropped (see + [`bans.md`](bans.md)). +- `SWWAF_BAN_SCOPE_V4_PREFIX` (default `32`): the length of the netblock around + an IPv4 client that a ban covers, from 0 to 32. `24` bans the surrounding /24. + An IPv6 ban covers the client's group. + +## Where and how often the state files are written + +- `SWWAF_STATE_DIR` (default `/var/lib/smallwebwaf`): the directory of the state + files, an absolute path. A directory that does not exist or cannot be written + stops the start. +- `SWWAF_STATE_WRITE_DELAY` (default `10s`): how long `bans.json` waits to be + written after a ban is made, lifted or made permanent. Every change in between + goes into the same write. A duration; it cannot be `off`. +- `SWWAF_STATE_COUNTER_INTERVAL` (default `15m`): how often every state file is + written. A duration; it cannot be `off`. + +## Logging + +- `SWWAF_LOG_REQUEST_HEADERS` (default + `accept,accept-language,accept-encoding,content-type,origin,range`): the + request headers to record in the log line, in either case. `Authorization`, + `Cookie` and `Set-Cookie` are never logged, even if listed. `Host` and + `Transfer-Encoding` are refused: Go's HTTP server takes them out of the + request. The host is the field `host`. +- `SWWAF_LOG_LEVEL` (default `info`): the least severe of `smallwebwaf`'s own + messages that are written: `debug`, `info`, `warn` or `error`. No message is + at `debug`. It holds back no request log line. `warn` and `error` hold back + the line of settings at start. +- `SWWAF_LOG_REMOTE_URL` (default unset): a syslog server every line on stdout + is also sent to, such as `syslog+tls://logs.example:6514`. The forms are + `syslog+udp://`, `syslog+tcp://` and `syslog+tls://`, each with a host and a + port and nothing more. +- `SWWAF_LOG_REMOTE_TLS_CA_FILE` (default unset): a file of PEM certificates + that a `syslog+tls` server's certificate must chain to, instead of the host's + own. A file that cannot be read, or holds no certificate, stops the start. +- `SWWAF_LOG_REMOTE_BUFFER` (default `10000`): the most lines held while they + wait to be sent, a whole number above zero. +- `SWWAF_LOG_REMOTE_FACILITY` (default `local0`): the syslog facility: `kern`, + `user`, `mail`, `daemon`, `auth`, `syslog`, `lpr`, `news`, `uucp`, `cron`, + `authpriv`, `ftp`, or `local0` to `local7`. +- `SWWAF_LOG_REMOTE_APP_NAME` (default: `SWWAF_INSTANCE_NAME`): the app name in + each syslog record, 1 to 48 printable ASCII characters without a space. + +## Tokens and metrics + +- `SWWAF_ADMIN_TOKEN` (default unset): the token for the ban endpoints and + `/_smallwebwaf/clients/`. A long random value of at least 32 characters. + Unset, those endpoints are off. Give it as a file to keep it from the app (see + "Settings given as files" below). +- `SWWAF_METRICS_TOKEN` (default unset): the token a scraper sends for + `/_smallwebwaf/metrics`, at least 32 characters. Unset, the metrics are off. + It is separate from the admin token, so a scraper cannot manage bans. +- `SWWAF_METRICS_TOP_N` (default `50`): how many AS numbers and how many + countries get series of their own in the metrics. The rest count as `other`. A + whole number above zero. + +## Alert destinations + +- `SWWAF_ALERT_WEBHOOK_URL` (default unset): the webhook that receives each + alert as a JSON object. An `http` or `https` URL without a user or a fragment. + The settings logged at start show its path and query as `********`, since many + webhooks carry their secret there. +- `SWWAF_ALERT_WEBHOOK_HEADERS` (default empty): headers sent with each post, + such as `Authorization:Bearer 0123456789abcdef`. A list of name, `:` and + value. A value cannot hold a comma. +- `SWWAF_ALERT_SLACK_WEBHOOK_URL` (default unset): the Slack incoming webhook + that receives each alert as a message. It is checked and masked as the webhook + URL is. +- `SWWAF_ALERT_NTFY_URL` (default unset): the ntfy topic that receives each + alert, as the topic's full URL, such as `https://ntfy.sh/my-alerts`. It is + checked and masked as the webhook URL is. +- `SWWAF_ALERT_NTFY_TOKEN` (default unset): an ntfy access token, sent as + `Authorization: Bearer `. A control character in it is refused. +- `SWWAF_ALERT_EVENTS` (default + `ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure,file_error`): + the events alerts are sent for. Any other name is refused. +- `SWWAF_ALERT_COOLDOWN` (default `15m`): how long a repeat of an alert is held + back. A duration or `off`. +- `SWWAF_ALERT_MAX_PER_HOUR` (default `60`): the most alerts sent in an hour. + The rest are rolled into one summary. A whole number above zero, or `off`. + +See [`alerts.md`](alerts.md) for the events and what each destination gets. + +## Anomaly thresholds + +These only alert. They refuse and ban nothing. Each scope has four thresholds: +the most requests and the most bytes counted in a minute and in an hour. +Requests are counts, bytes are sizes. Each is `off` by default. What is unusual +depends on each service's normal traffic, which the metrics show. The four +thresholds per client are: + +- `SWWAF_ANOMALY_CLIENT_REQUESTS_PER_MINUTE`; +- `SWWAF_ANOMALY_CLIENT_REQUESTS_PER_HOUR`; +- `SWWAF_ANOMALY_CLIENT_BYTES_PER_MINUTE`; +- `SWWAF_ANOMALY_CLIENT_BYTES_PER_HOUR`. + +The other scopes' names have another prefix before the same four endings, such +as `REQUESTS_PER_MINUTE`: + +- `SWWAF_ANOMALY_NET_...`: per netblock around a client. Two settings give the + netblock's length: + - `SWWAF_ANOMALY_NET_V4_PREFIX` (default `24`, from 0 to 32) for an IPv4 + client; + - `SWWAF_ANOMALY_NET_V6_PREFIX` (default `48`, from 0 to 128) for an IPv6 + client. +- `SWWAF_ANOMALY_ASN_...`: per AS number. Needs lookups on. +- `SWWAF_ANOMALY_TOTAL_...`: the whole service. +- `SWWAF_WATCH_...`: each netblock in `SWWAF_WATCH_NETS`, as a whole. + `SWWAF_WATCH_NETS` (default empty) is a list of name, `=` and netblock, such + as `office=203.0.113.0/24,scraper-x=198.51.100.0/22`. An item without `=` or a + name, or with a bad netblock, is refused. So is a name used twice. + +Every request but the health check is counted, whatever was done with it, +clients in `SWWAF_ALLOW_NETS` included. At most 20,000 counters are kept. +`alerts.json` keeps them across a restart. + +## Settings given as files + +Any setting may be given as a file holding its value. Add `_FILE` to the +setting's name: `SWWAF_METRICS_TOKEN_FILE` names the file that holds +`SWWAF_METRICS_TOKEN`. The file's contents are the value, less one newline at +the end. They are checked as the value itself would be. Setting both forms stops +the start, and so does naming a file that cannot be read. +`SWWAF_LOG_REMOTE_TLS_CA_FILE` already holds a file name, so it has no `_FILE` +form. + +A token given as a file is out of the app's reach only while the `smallwebwaf` +user alone can read the file. Make it on the host, owned by uid 65532 with mode +`0400`: + +```sh +mkdir -p /srv/app/tokens +openssl rand -hex 32 > /srv/app/tokens/metrics +chown 65532:65532 /srv/app/tokens/metrics +chmod 0400 /srv/app/tokens/metrics +``` + +Mount its directory into the container read-only, and name the file: + +```sh +docker run \ + -v /srv/app/tokens:/etc/smallwebwaf/tokens:ro \ + -e SWWAF_METRICS_TOKEN_FILE=/etc/smallwebwaf/tokens/metrics \ + ... +``` + +## Fixed limits + +A few limits are fixed rather than settings: + +- at most 100,000 GeoJS answers are kept, for 7 days each; +- at most 100,000 DNS blocklist verdicts and 100,000 AbuseIPDB scores are kept; +- at most 1,000 DNS blocklist queries are under way at once; +- the CrowdSec decision list is fetched again a minute after the last fetch. diff --git a/docs/state-files.md b/docs/state-files.md new file mode 100644 index 0000000..dd185cd --- /dev/null +++ b/docs/state-files.md @@ -0,0 +1,102 @@ +# State files + +`smallwebwaf` keeps its state in memory, and a copy of it in five JSON files in +`SWWAF_STATE_DIR`. The files are meant for people as well: you can read or edit +them at any time, and the running `smallwebwaf` takes the edit in. The +requirements they meet are under "Persistent state" in [`SPEC.md`](../SPEC.md). + +## The files + +Each has a top-level `version` of 1. + +- `bans.json`: active and past bans, up to `SWWAF_MAX_BANS` that `smallwebwaf` + made and every ban you made, indented to be read. Per entry: + - the netblock, start and expiry (`null` for permanent); + - the cause: `attack`, `limit`, `admin` or `crowdsec`; + - a short reason; + - when it was lifted, if you lifted it; + - the notes (see [`bans.md`](bans.md)). +- `clients.json`: per client, its minute, hour and day counters and its history + since it was first seen, one client per line, so `grep` shows everything about + one. The history holds: + - the first and last time seen; + - the AS number, AS name and country last looked up, and when; + - total requests, and how many were forwarded and how many refused; + - bytes in each direction; + - responses by status class; + - offences by kind: `limit`, `attack`, `rule_blocked`, `waf_blocked` and + `token_refused`. + + Clients that were never banned are kept too, so every client's history + survives a restart. + +- `lookups.json`: GeoJS answers, one per line: the AS number, AS name and + country, when GeoJS was asked and when the answer was last used. +- `reputation.json`: + - the last good copy of each list fetched from a URL, with when it was last + tried; + - each DNS blocklist verdict still in use; + - AbuseIPDB's scores, and the checks spent today. +- `alerts.json`: + - the anomaly counters; + - the alert cooldowns: for each event and netblock, source, file or thing + counted, when it was last sent and the count of repeats held back since; + - the alerts sent and held back in the current hour; + - the alerts waiting for each destination. + +Times are RFC 3339 UTC. Durations and sizes are plain numbers, with the unit in +the field name. `bans.json`, `reputation.json` and `alerts.json` are indented; +`clients.json` and `lookups.json` put each entry on one line. + +## Sizes + +`clients.json` takes about 1 KiB per client, about 20 MiB at the default +`SWWAF_MAX_TRACKED_CLIENTS`. Written every 15 minutes, that is under 2 GiB of +disk writes a day. `bans.json` takes about 1.2 KiB per ban, at most about 2.5 +KiB, so about 6 MiB at the default `SWWAF_MAX_BANS`. With a full file, a hundred +new bans a day come to about 600 MiB of writes. `lookups.json` takes about 150 +bytes per answer, under 1.5 GiB of writes a day when full. The other two files +are usually a few MiB or less. + +## Writing + +`bans.json` is written `SWWAF_STATE_WRITE_DELAY` after a ban is made, lifted or +made permanent. Every file is written every `SWWAF_STATE_COUNTER_INTERVAL`, and +when `smallwebwaf` stops. Each write goes to a temporary file that then replaces +the file, so a crash leaves the old file or the new one, whole. Before writing a +file, `smallwebwaf` takes in any edit you made to it, so your edit is not +overwritten. A write that fails is logged, raised as a `file_error` alert, and +tried again at the next write. A hard kill loses what changed since the last +write: + +- ban changes from the last `SWWAF_STATE_WRITE_DELAY` (10 seconds); +- everything else from the last `SWWAF_STATE_COUNTER_INTERVAL` (15 minutes): + - counts and history, and the counts in ban notes; + - GeoJS answers; + - reputation verdicts and the AbuseIPDB count; + - anomaly counters and alert cooldowns. + +## Reading at start + +The files are read back. Each client keeps its counts, so a restart gives no one +a fresh allowance. Each ban keeps refusing until it ends. Counter buckets and +cache entries whose time has passed are discarded. A missing file is empty +state. These stop the start, with a message naming the file and the place: + +- a file that does not parse, or has another `version`; +- an entry without a field it needs; +- a ban whose `cause` is not `limit`, `attack`, `admin` or `crowdsec`; +- a ban whose `expires` is not after its `start`. + +## Editing while it runs + +`smallwebwaf` watches the directory and takes in your edit of a file as soon as +you save it. What the file then holds replaces what `smallwebwaf` held for it. +It tells its own writes from yours by comparing the file with what it last read +or wrote. A change it made after you opened the file, such as a new ban, is lost +when you save over it; most editors warn when a file changed on disk while it +was open. An edit with an error that would stop the start does not stop the +running `smallwebwaf`. It keeps what it holds. At the file's next write it +renames your file to `.bad`, writes the file again from memory, and logs +and alerts the file and the place of the error. Mend the `.bad` file and move it +back. A file you remove is written again at its next write. diff --git a/internal/alerts/alerts.go b/internal/alerts/alerts.go index 89ea0e9..7864637 100644 --- a/internal/alerts/alerts.go +++ b/internal/alerts/alerts.go @@ -1,8 +1,8 @@ // Package alerts sends alerts on bans, on traffic over an anomaly // threshold, on a source that fails and on a file with an error to each // destination set: to the webhook -// SWWAF_ALERT_WEBHOOK_URL names, each as one JSON object, as the "Alert -// webhook schema" section of SPEC.md describes, to the Slack incoming +// SWWAF_ALERT_WEBHOOK_URL names, each as one JSON object, as "The JSON +// object" in docs/alerts.md describes, to the Slack incoming // webhook SWWAF_ALERT_SLACK_WEBHOOK_URL names, as a message, and to the // ntfy topic SWWAF_ALERT_NTFY_URL names. A repeat within // SWWAF_ALERT_COOLDOWN is held back, and so is an alert past @@ -149,11 +149,10 @@ type Params struct { } // Alert is one alert, as the webhook is sent it and alerts.json holds it, -// with the fields of the "Alert webhook schema" section of SPEC.md. ASN, -// ASName and Country are, for a ban, the client's as the ban's notes give -// them. +// with the fields of "The JSON object" in docs/alerts.md. ASN, ASName and +// Country are, for a ban, the client's as the ban's notes give them. // -//nolint:tagliatelle // SPEC.md's alert webhook schema names its fields in snake_case +//nolint:tagliatelle // docs/alerts.md names the alert's fields in snake_case type Alert struct { Instance string `json:"instance"` Time time.Time `json:"time"` diff --git a/internal/metrics/metrics.go b/internal/metrics/metrics.go index f1f4fc4..c7f7d96 100644 --- a/internal/metrics/metrics.go +++ b/internal/metrics/metrics.go @@ -1,6 +1,7 @@ -// Package metrics keeps smallwebwaf's Prometheus metrics, as the "Metrics -// endpoint" section of SPEC.md lists them, and serves them in the -// Prometheus text format. No metric carries a client's address. +// Package metrics keeps smallwebwaf's Prometheus metrics, as +// docs/metrics.md lists them, and serves them in the Prometheus text +// format, as the "Metrics endpoint" section of SPEC.md requires. No +// metric carries a client's address. package metrics import ( diff --git a/internal/ratelimit/ratelimit.go b/internal/ratelimit/ratelimit.go index 2413ac6..2318102 100644 --- a/internal/ratelimit/ratelimit.go +++ b/internal/ratelimit/ratelimit.go @@ -206,7 +206,7 @@ type Hit struct { // Counts are a client's requests and bytes in the minute, the hour and // the day that end at a request, that request's included. // -//nolint:tagliatelle // SPEC.md's request log names its fields in snake_case +//nolint:tagliatelle // docs/request-log.md names the fields in snake_case type Counts struct { Minute float64 `json:"minute"` Hour float64 `json:"hour"` diff --git a/internal/remotelog/remotelog.go b/internal/remotelog/remotelog.go index d40239b..9f5f2ee 100644 --- a/internal/remotelog/remotelog.go +++ b/internal/remotelog/remotelog.go @@ -1,8 +1,9 @@ // Package remotelog sends the lines smallwebwaf writes on stdout to the // remote log endpoint, SWWAF_LOG_REMOTE_URL, as the "Request log" section -// of SPEC.md describes: each line as the message of an RFC 5424 syslog -// record, over UDP, TCP or TLS. Lines wait in a bounded buffer, so a slow -// or unreachable endpoint never holds up a request or stdout. +// of SPEC.md requires and "Sending the log to a syslog server" in +// docs/request-log.md describes: each line as the message of an RFC 5424 +// syslog record, over UDP, TCP or TLS. Lines wait in a bounded buffer, so +// a slow or unreachable endpoint never holds up a request or stdout. package remotelog import ( diff --git a/internal/requestlog/requestlog.go b/internal/requestlog/requestlog.go index d9b5f66..829f8a0 100644 --- a/internal/requestlog/requestlog.go +++ b/internal/requestlog/requestlog.go @@ -64,11 +64,10 @@ const LimitHitErrorBurst = "error_burst" const timeLayout = "2006-01-02T15:04:05.000Z07:00" // Line is one request's line in the request log. The field names, and -// their order, are those of the "Request log" section of SPEC.md. A field -// that may not apply to a request is left out of its line when it does -// not. +// their order, are those of docs/request-log.md. A field that may not +// apply to a request is left out of its line when it does not. // -//nolint:tagliatelle // SPEC.md's request log names its fields in snake_case +//nolint:tagliatelle // docs/request-log.md names the fields in snake_case type Line struct { Type string `json:"type"`