diff --git a/README.md b/README.md index 6292ef5..83adf48 100644 --- a/README.md +++ b/README.md @@ -150,8 +150,8 @@ request that counts against its client in its history. It is one of: - a broken limit; - a refusal by a rule, by the Core Rule Set or for a missing or wrong token. -`smallwebwaf` handles each request in this order. A request one step refuses -skips the steps after it, apart from step 13. Nothing of a refused request +`smallwebwaf` handles each request in this order. Once a step refuses a request, +the steps after it are skipped, apart from step 13. Nothing of a refused request reaches the app. "What happens to a request" in [`SPEC.md`](SPEC.md) gives each step in full. diff --git a/SPEC.md b/SPEC.md index 13fa229..df217d5 100644 --- a/SPEC.md +++ b/SPEC.md @@ -100,10 +100,12 @@ to them. ## What happens to a request The steps run in this order. The first step that produces a final answer ends -the checks; step 15 runs for every request. The size and time limits apply to -the whole exchange. Each step names the section that holds its rules. +the checks; step 15 runs for every request but the health check. The size and +time limits apply to the whole exchange. Each step names the section that holds +its rules. -1. `GET /_smallwebwaf/healthz` is answered at once, before any check. +1. `GET /_smallwebwaf/healthz` is answered at once, before any check, so a + health checker never needs an exemption and is never refused. 2. The client is identified. - If the TCP peer is inside `SWWAF_TRUSTED_PROXIES`, `X-Forwarded-For` is walked from the right, and the first address outside those netblocks is @@ -116,9 +118,9 @@ the whole exchange. Each step names the section that holds its rules. around its address, a /64 by default, since one abuser usually controls a whole /64. A client is therefore one IPv4 address or one IPv6 group. 3. The static lists. A client in `SWWAF_ALLOW_NETS` skips steps 4 to 12 and the - byte limits and the error burst at step 15. It is still counted for the - anomaly thresholds, and still needs a token at step 13. A client in - `SWWAF_DENY_NETS` is refused; a client in both lists is let through. + byte limits and the error burst at step 15. It still needs a token at + step 13. A client in `SWWAF_DENY_NETS` is refused; a client in both lists is + let through. 4. Bans. A client under a ban is refused (see "Bans"). 5. The lookup, unless `SWWAF_LOOKUP_SOURCE` is `off` (see "Lookup of AS number and country"). The request goes on at once. It waits for the first GeoJS @@ -134,8 +136,6 @@ the whole exchange. Each step names the section that holds its rules. gets `SWWAF_BLOCKLIST_ACTION`. A client on the CrowdSec decision list is banned. A client listed by a DNS blocklist, or scored at least `SWWAF_ABUSEIPDB_MIN_SCORE` by AbuseIPDB, gets `SWWAF_REPUTATION_ACTION`. - With no cached verdict, a query is queued in the background and the request - goes on, so a first request is never delayed by a reputation query. 8. The client's limit percentage (see "Biased thresholds"). 9. Rate limiting (see "Request rate limits, per client"). A request over a limit times the percentage is refused and bans the client. @@ -184,7 +184,7 @@ the token check still apply. and the anomaly counters have fixed bounds, listed under "Fixed limits" in `docs/settings.md`. Past its bound, the GeoJS answer or anomaly counter used longest ago goes first, and the verdict or score fetched longest ago. Past - 1,000 DNS queries under way, a zone is asked at the client's next request. + that bound, a zone is asked at the client's next request. ## Bans @@ -193,8 +193,7 @@ Until a ban ends, every request from its netblock is refused with `SWWAF_BAN_SCOPE_V4_PREFIX`, or its IPv6 group. At the defaults a ban covers one address or one /64, so a permanent ban does not reach neighbours who did nothing. Requests refused under a ban are not counted against limits. They are -counted in the client's history and in the ban's notes. A banned client is not -looked up. +counted in the client's history and in the ban's notes. An offence is a request `smallwebwaf` holds against the client. It is one of: @@ -367,8 +366,8 @@ Conventions: owner's own networks. `SWWAF_RATE_LIMIT_EXEMPT_NETS` bypasses the request and byte limits only; the error burst, attack detection and bans still apply. - Many clients share one address (mobile carriers, offices, Tor), and so share - its limits and its bans. The defaults sit well above one person's use, and the - first two lists take known shared addresses out. + its limits and its bans. `SWWAF_ALLOW_NETS` and `SWWAF_RATE_LIMIT_EXEMPT_NETS` + take known shared addresses out. ### Request rate limits, per client @@ -442,12 +441,8 @@ service that keeps its visitors' addresses on its own host. - Only one source is used. The start stops, naming both settings, when `file` is set without `SWWAF_LOOKUP_DB_PATH`, or `SWWAF_LOOKUP_DB_PATH` with any other - source. The start also stops when the source is `off` and one of these needs - lookups: - - a country list; - - a biased threshold; - - `SWWAF_ADD_LOOKUP_HEADERS`; - - an anomaly threshold per AS number. + source. The start also stops when the source is `off` and a setting that needs + a lookup is set (`docs/settings.md` lists them). - GeoJS is told the address of every new visitor, whether or not a setting uses the answer, since the request log, the metrics and the ban notes carry it. These addresses are never sent: @@ -476,8 +471,8 @@ service that keeps its visitors' addresses on its own host. API requests", judged by GeoJS alone, and let it block a caller. While GeoJS is slow, down or refusing `smallwebwaf`, new clients count as unknown. GeoJS is then left alone for a second, and twice as long after each further failure, - up to five minutes, and one `source_failure` alert is sent per cooldown. A - service that cannot accept this uses the file. + up to five minutes. Each failure raises a `source_failure` alert (see + "Alerting"). A service that cannot accept this uses the file. - The file is the free IPinfo Lite database (`ipinfo_lite.mmdb`), whose `asn`, `as_name` and `country_code` fields are read. The operator downloads it with a free IPinfo account and mounts it read-only; `smallwebwaf` never fetches it @@ -486,9 +481,9 @@ service that keeps its visitors' addresses on its own host. on the site that uses it, such as "IP address data is powered by IPinfo". - The file is read whole at start; one that is missing or unreadable stops the start. It is read again once it has gone 2 seconds without a change after - being replaced. One that cannot be read is logged, sent as one `file_error` - alert and ignored. The database already loaded stays in use. A client whose - address the file does not hold counts as unknown. + being replaced. A replacement that cannot be read is logged, sent as one + `file_error` alert and ignored. The database already loaded stays in use. A + client whose address the file does not hold counts as unknown. ### Country lists @@ -508,9 +503,8 @@ where it comes from. exclusive list already refuses every other country. A code on both lists stops the start with a message naming it. - A refused request is answered with `SWWAF_BAN_RESPONSE`, as soon as the - client's address has been looked up, before its body is read. It skips the - reputation checks, the limits, the rule files and the Core Rule Set. It is - logged with the action `country_denied`, and counted in the metrics. + client's address has been looked up, before its body is read. It is logged + with the action `country_denied`, and counted in the metrics. ### Biased thresholds @@ -519,10 +513,11 @@ countries. The lists are empty by default, for the same reason as the country lists. The request log and the metrics show which AS numbers and countries a service's abuse comes from. -- Clients in a listed AS number or country get that percentage of every request - and byte limit (`SWWAF_ASN_LIMIT_PERCENT`, `SWWAF_COUNTRY_LIMIT_PERCENT`). A - lowered limit is rounded down. `0` is a zero allowance: the client's first - request breaks a limit and bans it. +- Each list gives AS numbers or countries a percentage, such as `AS14061:50`. + Their clients get that percentage of every request and byte limit + (`SWWAF_ASN_LIMIT_PERCENT`, `SWWAF_COUNTRY_LIMIT_PERCENT`). A lowered limit is + rounded down. `0` is a zero allowance: the client's first request breaks a + limit and bans it. - `SWWAF_ASN_BYTES_PERCENT` and `SWWAF_COUNTRY_BYTES_PERCENT` set the byte limits' percentage alone, when it should differ from the request limits'. For an AS number they list, they replace its percentage from @@ -556,9 +551,8 @@ code in them for attacks. The default gives up refusing an attack carried in a body. The rule files, the limits and the bans still apply to the client that sends it. -- The Core Rule Set inspects only a request that every earlier check let - through, and skips a path under `SWWAF_WAF_EXEMPT_PATHS`. An engine error on a - request forwards the request, and is logged and counted. +- The Core Rule Set skips a path under `SWWAF_WAF_EXEMPT_PATHS`. An engine error + on a request forwards the request, and is logged and counted. - `SWWAF_WAF_PARANOIA_LEVEL` and `SWWAF_WAF_ANOMALY_THRESHOLD` are the Core Rule Set's own two tuning values, at its own defaults. Each level up runs more rules, which find more attacks and take more ordinary requests for attacks. @@ -601,14 +595,18 @@ sends it. percent-decoded: `/wp-login.php` matches `/wp-login.php?x=1` and `/%77p-login.php`, and does not match `/wp-login.php/` or `/blog/wp-login.php`. -- `SWWAF_ERROR_BURST_THRESHOLD` counts a client's refusals per minute for a rule - file match, a trap path, a Core Rule Set match, or a missing or wrong token; - no other refusal counts. More than this breaks a limit (see "Bans"). A client - trying one attack or token after another is refused many times a minute; a - person rarely more than a few times. Answers the app gives are not counted, - since they do not tell a scanner from an ordinary client. In front of gitea, - registry and git clients get `404` and `401` by design, hundreds of times a - minute. +- `SWWAF_ERROR_BURST_THRESHOLD` counts a client's refusals per minute of these + kinds, and of no other: + - a rule file match; + - a trap path; + - a Core Rule Set match; + - a missing or wrong token. + + More than this breaks a limit (see "Bans"). A client trying one attack or + token after another is refused many times a minute; a person rarely more + than a few times. Answers the app gives are not counted, since they do not + tell a scanner from an ordinary client. In front of gitea, registry and git + clients get `404` and `401` by design, hundreds of times a minute. `smallwebwaf` changes the Core Rule Set 4.25.0 in six ways, since in front of gitea it would otherwise refuse ordinary requests. The changes hold in front of @@ -773,9 +771,8 @@ too close together. by default. Such verdicts are less certain than a blocklist, so a listed client gets a quarter of every request and byte limit rather than a refusal. - Verdicts and scores are kept for `SWWAF_REPUTATION_CACHE_TTL`, in memory and - in `reputation.json`, with the fixed bounds under "Fixed limits" in - `docs/settings.md`. A source that is down or over quota gives no verdict. The - service continues, with one `source_failure` alert per cooldown. + in `reputation.json`. A source that is down or over quota gives no verdict. + The service continues, with one `source_failure` alert per cooldown. ### Alerting @@ -787,9 +784,8 @@ too close together. the ban's alert, with `mode` set to `observe` in its `detail`. - A repeat is an alert with the same event, and the same netblock, source, file or anomaly scope, as one sent less than `SWWAF_ALERT_COOLDOWN` before. An - anomaly scope is the thing counted: one client, netblock, AS number or named - netblock, or the whole service. A repeat is held back and counted, and the - next such alert sent gives the count. + anomaly scope is one of the things counted under "Anomaly thresholds". A + repeat is held back and counted, and the next such alert sent gives the count. - Past `SWWAF_ALERT_MAX_PER_HOUR` alerts in an hour, the rest are held back and counted by event, so a wide attack cannot flood the channel. Once the hour ends, one alert with the event `summary` gives those counts. It also counts @@ -820,7 +816,7 @@ addresses at low rates each, which the per-client limits do not see. A low - Every request but the health check is counted, whatever was done with it, clients in `SWWAF_ALLOW_NETS` and `SWWAF_RATE_LIMIT_EXEMPT_NETS` included, since an exempt client misbehaving is worth knowing about. The counters are - kept in memory and in `alerts.json`, with a fixed bound. + kept in memory and in `alerts.json`. ## Rule files @@ -846,10 +842,10 @@ and the default file. of every request; a rule naming one stops the start like any bad line. Request bodies are not available to rule files; reading them is the Core Rule Set's job. -- `log` only notes the match in the request log. `block` refuses the request - without a ban; `ban` makes the request a clear sign of attack (see "Bans"), - and is for requests no real visitor sends. A visitor banned by mistake is let - back in by lifting the ban. +- `log` only notes the match in the request log. `block` refuses the request; + `ban` makes the request a clear sign of attack (see "Bans"), and is for + requests no real visitor sends. A visitor banned by mistake is let back in by + lifting the ban. - At start, these stop the process with a message naming the file and line: - a line that does not parse; - a regex that does not compile; @@ -940,9 +936,8 @@ edit them at any time, and the running `smallwebwaf` takes the edit in. ## Request log -Every request gets one JSON object on one line on stdout, refused ones included, -apart from those Go's HTTP server ends before `smallwebwaf` sees them (see "Size -and time limits"). `docs/request-log.md` lists the fields. Bodies are never +Every request `smallwebwaf` sees gets one JSON object on one line on stdout, +refused ones included. `docs/request-log.md` lists the fields. Bodies are never logged, and no header but those listed there. Query strings are logged as received; an app that carries secrets in query strings needs that fixed in the app. The process's own messages (start, the settings, stop, errors) share the @@ -971,8 +966,7 @@ while `docker logs` keeps working. ## Metrics endpoint `GET /_smallwebwaf/metrics` answers in the Prometheus text format, for a request -carrying `SWWAF_METRICS_TOKEN`. `docs/metrics.md` lists the metrics. Every -metric carries the label `instance`, set to `SWWAF_INSTANCE_NAME`. No metric +carrying `SWWAF_METRICS_TOKEN`. `docs/metrics.md` lists the metrics. No metric carries a client IP address as a label. Per-address questions are answered by the request log and `GET /_smallwebwaf/clients/`. @@ -980,15 +974,14 @@ the request log and `GET /_smallwebwaf/clients/`. A request whose path starts with `/_smallwebwaf/` is for `smallwebwaf` itself: it answers it and never passes it to the app. The prefix carries the tool's -name, so it takes no path an app uses. Admins and scrapers reach these endpoints -through traefik, like any other request. `docs/admin-endpoints.md` lists the -endpoints, what each takes and what each answers. +name, so it takes no path an app uses. Admins and the metrics collector reach +these endpoints through traefik, like any other request. +`docs/admin-endpoints.md` lists the endpoints, what each takes and what each +answers. - `GET /_smallwebwaf/healthz` answers `200` with `ok` to anyone, without a token, while `smallwebwaf` is running. It does not ask the app; the - container's health check does. It is answered before any check, so a health - checker never needs an exemption and is never refused, for example by - `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES`. + container's health check does. - Every other endpoint needs its token, sent as `Authorization: Bearer ` (see "Tokens" under "Settings"). While a token is unset, the endpoints that need it answer `404`, as does any other path under the prefix. @@ -996,10 +989,9 @@ endpoints, what each takes and what each answers. other request goes through. They are answered at the point where another request would be forwarded to the app: a banned or refused client stays refused, and each request counts toward the client's limits. A missing or - wrong token is answered `401`, in `SWWAF_MODE=observe` too, and counts toward - the error burst. A client guessing tokens is therefore banned once it passes - `SWWAF_ERROR_BURST_THRESHOLD` guesses in a minute. -- A ban made through the endpoints starts at once, with the cause `admin`. + wrong token is answered `401` (see "Attack detection" for what guessing + costs). +- A ban made through the endpoints starts at once. ## Deployment @@ -1041,26 +1033,26 @@ what the image holds and how each part is pinned. 2015, at a fixed commit hash, with a `go.mod` the build writes. `runsvinit` starts runit's `runsvdir`, which runs the `run` script of each directory under `/etc/service`. Ubuntu's `sv` looks for services there too. -- The image's `run` script for `smallwebwaf` waits one second and makes - `SWWAF_STATE_DIR` and every file in it belong to the `smallwebwaf` user (uid - and gid 65532). Then it starts `smallwebwaf` as that user with runit's - `chpst`. +- The image's `run` script for `smallwebwaf` makes `SWWAF_STATE_DIR` and every + file in it belong to the `smallwebwaf` user (uid and gid 65532). Then it + starts `smallwebwaf` as that user with runit's `chpst`. Every `run` script, the app's included, is a bash script. It starts with `#!/usr/bin/env bash` and `set -euo pipefail`, and puts its code in a `main` function, called on its last line, as the owner's code style guide asks. Ubuntu -ships bash. +ships bash. Each waits one second (`sleep 1`) before starting its program, so a +process that stops at start is started again every second, not in a tight loop. Beyond its `FROM` line, the app's Dockerfile adds: - its binary; - any packages it needs, from nixpkgs, with `nix-env -iA nixpkgs.`; - its runit service: a directory under `/etc/service` named after the app, never - `smallwebwaf`. Its `run` script waits one second (`sleep 1`) and then starts - the app with `chpst`, listening where `SWWAF_UPSTREAM_URL` points (see - "Ports"). The app runs as a user of its own, which the Dockerfile creates with - `useradd`. That user is neither root nor `smallwebwaf`, so that the app cannot - touch the state files or the process of `smallwebwaf`. + `smallwebwaf`. Its `run` script starts the app with `chpst`, listening where + `SWWAF_UPSTREAM_URL` points (see "Ports"). The app runs as a user of its own, + which the Dockerfile creates with `useradd`. That user is neither root nor + `smallwebwaf`, so that the app cannot touch the state files or the process of + `smallwebwaf`. It sets no `ENTRYPOINT` or `USER` of its own. The container must start `runsvinit`, as root, so that it can start each service as its own user. @@ -1072,16 +1064,17 @@ It sets no `ENTRYPOINT` or `USER` of its own. The container must start script starts as root and hands over to its program's user with `chpst`. Both services start with the container's environment variables and write to the container's output, which `docker logs` shows. -- When `smallwebwaf` or the app exits, runit runs its `run` script again, which - waits a second before starting it. The other process keeps running, and the - container does not stop. While `smallwebwaf` is down, traefik answers `502`. - While the app is down, `smallwebwaf` answers `502`, which is not a client - offence. -- Only four things stop `smallwebwaf`, all at start: +- When `smallwebwaf` or the app exits, runit runs its `run` script again. The + other process keeps running, and the container does not stop. While + `smallwebwaf` is down, traefik answers `502`. While the app is down, + `smallwebwaf` answers `502`, which is not a client offence. +- Only faults found at start stop `smallwebwaf`: - invalid configuration; - a lookup database that is missing or unreadable; - a rule file or state file that does not parse; - - an unwritable `SWWAF_STATE_DIR`. + - an unwritable `SWWAF_STATE_DIR`; + - a `SWWAF_RULES_DIR` that does not exist or cannot be read; + - a `SWWAF_LISTEN_ADDR` it cannot listen on. - A `smallwebwaf` that stops at start is therefore started again every second, logging its message each time, until the cause is fixed; the health check fails meanwhile. Once running, a failing GeoJS, reputation source or alert diff --git a/docs/alerts.md b/docs/alerts.md index b35e1a5..cbc4fe2 100644 --- a/docs/alerts.md +++ b/docs/alerts.md @@ -79,10 +79,11 @@ on one line: - `instance`: `SWWAF_INSTANCE_NAME`. `time`: when the alert was raised. `event`: one of the events above. - `client`: the address whose request raised the alert. Empty for - `source_failure` and `file_error`. -- `netblock`: empty for `source_failure` and `file_error`; otherwise one of: + `source_failure`, `file_error` and `summary`. +- `netblock`: empty for `source_failure`, `file_error` and `summary`; otherwise + one of: - the ban's netblock; - - the netblock an anomaly counted; + - the netblock an anomaly counted, empty in the `asn` and `total` scopes; - for the other events, the client as rate limiting counts it: an IPv4 address as a /32, or an IPv6 group. - `asn`, `as_name` and `country`: the client's, as known when the alert was @@ -97,6 +98,8 @@ on one line: - for a Core Rule Set match, the `rule_ids`, `score`, `method` and `path`; - for a source failure, the `source` and `error`; - for a file error, the `file` and `error`; + - for a summary, the `hour` it covers, the `count` of alerts held back, and + that count by event in `events`; - `mode`, when the request was not refused: `detect` or `observe`. - `suppressed_repeats`: how many repeats the cooldown held back before this alert. diff --git a/docs/metrics.md b/docs/metrics.md index b2a0e91..87877e7 100644 --- a/docs/metrics.md +++ b/docs/metrics.md @@ -1,8 +1,8 @@ # The metrics -The metrics `GET /_smallwebwaf/metrics` answers with, as "Metrics endpoint" in -[`SPEC.md`](../SPEC.md) describes it. Prometheus keeps the `instance` label as -`exported_instance` unless the scrape sets `honor_labels: true`. +What `GET /_smallwebwaf/metrics` answers with. "Metrics endpoint" in +[`SPEC.md`](../SPEC.md) describes the endpoint. Prometheus keeps the `instance` +label as `exported_instance` unless the scrape sets `honor_labels: true`. ## Traffic diff --git a/docs/settings.md b/docs/settings.md index 840f7c5..2e32250 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -91,11 +91,11 @@ that take `off`, the `_FILE` form, and what each group does and why. ## Static lists - `SWWAF_ALLOW_NETS` (default empty): netblocks you trust, such as your - monitoring. Their clients are refused only for a size or time limit or a - missing or wrong token. + monitoring (see step 3 in [`SPEC.md`](../SPEC.md)). - `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients are exempt from rate limiting and the byte limits. -- `SWWAF_DENY_NETS` (default empty): netblocks whose clients are always refused. +- `SWWAF_DENY_NETS` (default empty): netblocks whose clients are refused (see + step 3 in [`SPEC.md`](../SPEC.md)). ## Request rate limits, per client @@ -136,7 +136,13 @@ that take `off`, the `_FILE` form, and what each group does and why. ## Lookup of AS number and country -- `SWWAF_LOOKUP_SOURCE` (default `geojs`): `geojs`, `file` or `off`. +- `SWWAF_LOOKUP_SOURCE` (default `geojs`): `geojs`, `file` or `off`. `off` stops + the start while a setting that needs a lookup is set: + - the two country lists; + - `SWWAF_ADD_LOOKUP_HEADERS`; + - the four biased threshold lists and `SWWAF_ASN_LIMIT_PERCENT_URL`; + - `SWWAF_UNKNOWN_LIMIT_PERCENT` below `100`; + - an anomaly threshold per AS number. - `SWWAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in its `.mmdb` form, for `SWWAF_LOOKUP_SOURCE=file`. - `SWWAF_LOOKUP_TIMEOUT` (default `1s`): how long a request waits for its @@ -286,7 +292,7 @@ as `REQUESTS_PER_MINUTE`: client; - `SWWAF_ANOMALY_NET_V6_PREFIX` (default `48`, from 0 to 128) for an IPv6 client. -- `SWWAF_ANOMALY_ASN_...`: per AS number. Needs a lookup source. +- `SWWAF_ANOMALY_ASN_...`: per AS number. - `SWWAF_ANOMALY_TOTAL_...`: the whole service. - `SWWAF_WATCH_...`: each netblock in `SWWAF_WATCH_NETS`, as a whole. `SWWAF_WATCH_NETS` (default empty) is a list of name, `=` and netblock, such