From 9e6abf84969933c325aa1dd4b8eb0df03c2d5ee3 Mon Sep 17 00:00:00 2001 From: sneak Date: Mon, 9 Nov 2026 06:09:38 +0000 Subject: [PATCH] Name log fields, metrics and the Core Rule Set package for what they are (closes #192) Log field, metric, action and event names now say what they count and, for timings, the unit. Package internal/waf is internal/coreruleset. Model: opus-5-5 --- README.md | 243 +++++++++--------- SPEC.md | 18 +- docs/alerts.md | 8 +- docs/metrics.md | 11 +- docs/request-log.md | 45 ++-- docs/settings.md | 2 +- docs/state-files.md | 4 +- internal/alerts/alerts.go | 22 +- internal/alerts/alerts_test.go | 12 +- internal/anomaly/anomaly.go | 10 +- internal/bans/admin_test.go | 32 +-- internal/bans/bans.go | 24 +- internal/bans/bans_test.go | 54 ++-- internal/bans/crowdsec_test.go | 6 +- internal/bans/snapshot_test.go | 28 +- internal/config/config.go | 4 +- internal/config/config_test.go | 2 +- .../waf.go => coreruleset/coreruleset.go} | 12 +- .../coreruleset_test.go} | 88 ++++--- internal/{waf => coreruleset}/gitea.conf | 0 .../testdata/gitea-requests.txt | 2 +- internal/discriminator/admin.go | 4 +- internal/discriminator/admin_test.go | 18 +- internal/discriminator/anomaly_test.go | 2 +- internal/discriminator/answertime_test.go | 22 +- internal/discriminator/bans.go | 6 +- internal/discriminator/bans_test.go | 2 +- internal/discriminator/biased_test.go | 48 ++-- internal/discriminator/bytelimits.go | 2 +- internal/discriminator/coreruleset.go | 26 +- internal/discriminator/coreruleset_test.go | 106 ++++---- internal/discriminator/discriminator.go | 36 +-- internal/discriminator/discriminator_test.go | 46 ++-- internal/discriminator/health_test.go | 3 +- internal/discriminator/history_test.go | 4 +- internal/discriminator/lookup_test.go | 14 +- internal/discriminator/metrics_test.go | 12 +- internal/discriminator/proxy.go | 20 +- internal/discriminator/ratelimits.go | 10 +- internal/discriminator/refusallimit.go | 4 +- internal/discriminator/refusallimit_test.go | 15 +- internal/discriminator/reputation_test.go | 60 +++-- internal/discriminator/request.go | 38 +-- internal/discriminator/requestlog_test.go | 24 +- internal/discriminator/tagdeny_test.go | 9 +- internal/lookup/lookup.go | 6 +- internal/lookup/lookup_test.go | 16 +- internal/metrics/metrics.go | 60 ++--- internal/ratelimit/ratelimit.go | 25 +- internal/requestlog/requestlog.go | 75 +++--- internal/requestlog/requestlog_test.go | 18 +- internal/rproxy/request.go | 16 +- internal/rproxy/rproxy.go | 10 +- internal/rproxy/rproxy_test.go | 12 +- internal/state/state.go | 14 +- internal/state/state_test.go | 10 +- internal/sww/sww.go | 4 +- internal/sww/sww_test.go | 10 +- 58 files changed, 743 insertions(+), 691 deletions(-) rename internal/{waf/waf.go => coreruleset/coreruleset.go} (98%) rename internal/{waf/waf_test.go => coreruleset/coreruleset_test.go} (90%) rename internal/{waf => coreruleset}/gitea.conf (100%) rename internal/{waf => coreruleset}/testdata/gitea-requests.txt (99%) diff --git a/README.md b/README.md index 3d9ebdf..8799b16 100644 --- a/README.md +++ b/README.md @@ -220,123 +220,123 @@ account, a key, a token or an alert destination stays off until you give one. Every setting the code reads, with its default: -| Variable | Default | What it does | -| ---------------------------------------- | ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `SWW_COMMAND` | `rproxy` | Which command the image runs, and its health check checks: `rproxy` or `discriminator`. | -| `SWRP_WAF_BACKEND` | required | `discriminator`'s URL, such as `http://discriminator:8080`. | -| `SWRP_UPSTREAM_URL` | `http://127.0.0.1:8081` | The app's URL: a scheme, a host and an optional port, nothing more. | -| `SWRP_LISTEN_ADDR` | `:8080` | Where `rproxy` listens; leave the host part empty. | -| `SWRP_TRUSTED_PROXIES` | `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16` | The netblocks whose `X-Forwarded-For` `rproxy` believes; set but empty, it believes none. | -| `SWRP_TIMEOUT_MS` | `2000` | How long `rproxy` waits for `discriminator`'s answer to a question and to a report, in milliseconds. | -| `SWRP_DEFAULT_DISPOSITION` | `allow` | What `rproxy` does with a request when `discriminator` cannot be reached or does not answer in time: `allow` passes it to the app, `deny` answers `503`. | -| `SWRP_MAX_REQUEST_SIZE` | `100M` | The largest request body, a size or `off`; a larger one is answered `413`. | -| `SWRP_MAX_RESPONSE_SIZE` | `1G` | The largest response body, a size or `off`. The visitor gets `502` in place of a larger one, or it is cut off if it has already started. | -| `SWRP_MAX_REQUEST_TIME_MS` | `60000` | Two time limits, each this long, in milliseconds or `off`: one for the visitor to send its request line and headers, then one for the whole request to reach the app. | -| `SWRP_MAX_RESPONSE_TIME_MS` | `1800000` | How long the answer may take, from the end of the request to its last byte reaching the visitor, in milliseconds or `off`. | -| `SWW_LISTEN_ADDR` | `:8080` | Where `discriminator` listens; leave the host part empty. | -| `SWW_INSTANCE_NAME` | the container's host name | The name in every line `discriminator` logs, and in every alert and metric, such as `fsn1/discriminator`. | -| `SWW_MODE` | `enforce` | `observe` refuses nothing and makes no ban; each log line says what `enforce` would have done. | -| `SWW_IPV6_GROUP_PREFIX` | `64` | The prefix length of the IPv6 netblock that counts as one client, from 32 to 128. | -| `SWW_MAX_TRACKED_CLIENTS` | `20000` | The most clients kept, in memory and in `clients.json`; the least recently seen is dropped first. | -| `SWW_MAX_BANS` | `5000` | The most bans `discriminator` keeps of its own; bans an admin made do not count. | -| `SWW_ALLOW_NETS` | empty | Netblocks whose clients skip every check, such as your monitoring. | -| `SWW_RATE_LIMIT_EXEMPT_NETS` | empty | Netblocks whose clients are exempt from rate limiting and the byte limits. | -| `SWW_DENY_NETS` | empty | Netblocks whose clients are refused. | -| `SWW_RATE_LIMIT_PER_MINUTE` | `120` | The most requests a client may make in a minute, a count or `off`; the next request is refused and bans the client. | -| `SWW_RATE_LIMIT_PER_HOUR` | `1500` | The most requests a client may make in an hour, a count or `off`; the next request is refused and bans the client. | -| `SWW_RATE_LIMIT_PER_DAY` | `6000` | The most requests a client may make in a day, a count or `off`; the next request is refused and bans the client. | -| `SWW_RATE_LIMIT_EXEMPT_PATHS` | empty | Path prefixes exempt from rate limiting and the byte limits, such as `/assets/`. | -| `SWW_BYTES_LIMIT_PER_MINUTE` | `off` | The most bytes counted for a client in a minute, a size or `off`. `SWW_BYTES_COUNT` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. | -| `SWW_BYTES_LIMIT_PER_HOUR` | `50G` | The most bytes counted for a client in an hour, a size or `off`. `SWW_BYTES_COUNT` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. | -| `SWW_BYTES_LIMIT_PER_DAY` | `100G` | The most bytes counted for a client in a day, a size or `off`. `SWW_BYTES_COUNT` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. | -| `SWW_BYTES_COUNT` | `both` | What the byte limits and the byte anomaly thresholds count: `response`, the app's response bodies; `request`, the visitors' request bodies; or `both`. What the visitor sent over a WebSocket counts as request bytes, and what the app sent as response bytes. | -| `SWW_ERROR_BURST_THRESHOLD` | `30` | The most refusals a client may get in a minute from a rule file, a trap path or the Core Rule Set, a count or `off`. The next one bans the client. | -| `SWW_BAN_RESPONSE` | `403` | How a refused client is answered: `403`, `429` or `close`, which drops the connection. | -| `SWW_LIMIT_BAN_DURATION` | `1h` | The ban for a first broken limit. | -| `SWW_LIMIT_BAN_REPEAT_WINDOW` | `24h` | A client that breaks a limit again within this time after its last such ban ended gets a ban three times as long. | -| `SWW_MAX_BAN_DURATION` | `7d` | A ban for a broken limit that would be longer than this is permanent instead. | -| `SWW_ATTACK_BAN_DURATION` | `7d` | The ban for a first clear sign of attack: a `ban` rule or a trap path. | -| `SWW_BAN_SCOPE_V4_PREFIX` | `32` | The prefix length of the netblock a ban on an IPv4 client covers, from 0 to 32. | -| `SWW_LOOKUP_SOURCE` | `geojs` | Where each client's AS number and country come from: `geojs` (a web service), `file` (a local database) or `off`. | -| `SWW_LOOKUP_DB_PATH` | empty | The IPinfo Lite database file, in its `.mmdb` form, read while the source is `file`. | -| `SWW_LOOKUP_TIMEOUT` | `800ms` | How long a request waits for its client's first GeoJS answer when a setting needs it, and how long a request to GeoJS may take. | -| `SWW_ADD_LOOKUP_HEADERS` | `false` | `true` passes the app the client's AS number and country in `X-Client-ASN` and `X-Client-Country`. | -| `SWW_EXCLUSIVELY_ALLOWED_COUNTRIES` | empty | When set, the only countries whose clients get through; a client the lookup cannot place is refused too. | -| `SWW_TAG_LISTS` | empty | Address lists that each add a tag to their clients' requests, as `name=URL` pairs, such as `torexit=https://example.com/tor.txt`. | -| `SWW_TAG_DENY` | empty | Tags whose requests are refused, such as `torexit,country=CN`. | -| `SWW_TAG_LIMIT_PERCENT` | empty | Tags whose clients get a percentage, from 1 to 99, of every request and byte limit, such as `vpn:50,asn=14061:25`; the lowest applies. | -| `SWW_ROUTEWATCH_URL` | unset | The URL of a routewatch instance, which gives each client's `asn=` and `prefix=` tags. Unset, nothing asks routewatch. | -| `SWW_ASN_BYTES_PERCENT` | empty | A percentage of the byte limits alone for the AS numbers listed, in place of their `asn=` tag's percentage. | -| `SWW_COUNTRY_BYTES_PERCENT` | empty | A percentage of the byte limits alone for the countries listed, in place of their `country=` tag's percentage. | -| `SWW_UNKNOWN_LIMIT_PERCENT` | `100` | The percentage of every limit a client without a country gets. | -| `SWW_ASN_LIMIT_PERCENT_URL` | unset | The URL of a text file of `AS:` lines, fetched like a blocklist; each line counts as an `asn=` tag in `SWW_TAG_LIMIT_PERCENT` (the lowest wins). | -| `SWW_RULES_DIR` | `/etc/sww/rules.d` | The directory of the rule files, read at start and again when a file there changes. | -| `SWW_RULES_ENABLED` | `true` | `false` reads no rule file. | -| `SWW_TRAP_PATHS` | empty | Paths only scanners ask for, such as `/wp-login.php,/xmlrpc.php`; a request for one bans the client. | -| `SWW_WAF_MODE` | `block` | What the Core Rule Set does with a match: `block` refuses it with `403`, `detect` only logs and alerts it, `off` runs no rule. | -| `SWW_WAF_PARANOIA_LEVEL` | `1` | The Core Rule Set's paranoia level, from 1 to 4; each level up finds more attacks and takes more ordinary requests for attacks. | -| `SWW_WAF_ANOMALY_THRESHOLD` | `5` | The Core Rule Set score at which a request is a match, a whole number or `off`; one critical rule scores 5. | -| `SWW_WAF_DISABLED_RULES` | `920340,920420,920440,920640,930130,930140` | Core Rule Set rule ids switched off; a list you give replaces the default, so repeat these six to keep them off. | -| `SWW_WAF_EXEMPT_PATHS` | empty | Path prefixes the Core Rule Set does not inspect. | -| `SWW_WAF_BODY_LIMIT` | `off` | How much of a request body the Core Rule Set reads, a size up to `64K`; `off` reads none. | -| `SWW_CORE_RULE_SET_EXCLUSIONS_FILE` | unset | A file of Core Rule Set exclusions, as Coraza directives, that switch rules off for some fields or paths only. | -| `SWW_BLOCKLIST_URLS` | empty | Blocklists to fetch, text files of addresses and netblocks one to a line, such as `https://www.spamhaus.org/drop/drop.txt`. | -| `SWW_BLOCKLIST_REFRESH` | `24h` | How long after a list was last tried it is fetched again, `1h` or more. | -| `SWW_BLOCKLIST_ACTION` | `deny` | What a client on a blocklist gets: `deny`, `limit:` of every limit, or `log`. | -| `SWW_DNSBL_ZONES` | empty | DNS blocklists asked about each new client in the background, such as `dnsbl.dronebl.org`. | -| `SWW_DNSBL_RESOLVER` | unset | The resolver the zones are asked through, an IP address with an optional port; unset, the host's resolver is used. | -| `SWW_ABUSEIPDB_KEY` | unset | The key of your AbuseIPDB account; set, a client that has committed an offence is checked in the background. | -| `SWW_ABUSEIPDB_MIN_SCORE` | `75` | The least AbuseIPDB abuse confidence score, from 0 to 100, that is a hit. | -| `SWW_ABUSEIPDB_DAILY_BUDGET` | `900` | The most AbuseIPDB checks made in a day. | -| `SWW_CROWDSEC_LAPI_URL` | unset | The local API of a CrowdSec engine you run, such as `http://172.17.0.1:8080`; its decision list is fetched every minute, and a listed client is banned. | -| `SWW_CROWDSEC_LAPI_KEY` | unset | The key the engine gave for `smallwebwaf`, as `cscli bouncers add smallwebwaf` prints it; set with the URL, never alone. | -| `SWW_REPUTATION_ACTION` | `limit:25` | What a client listed by a DNS blocklist, or scored a hit by AbuseIPDB, gets: `deny`, `limit:` or `log`. | -| `SWW_REPUTATION_CACHE_TTL` | `24h` | How long a DNS blocklist verdict or an AbuseIPDB score is used. | -| `SWW_REPUTATION_TIMEOUT` | `2s` | How long a DNS blocklist query or an AbuseIPDB check may take. | -| `SWW_STATE_DIR` | `/var/lib/sww` | The directory of the state files, an absolute path. | -| `SWW_STATE_WRITE_DELAY` | `10s` | How long `bans.json` waits to be written after a ban is made, lifted or made permanent. | -| `SWW_STATE_COUNTER_INTERVAL` | `15m` | How often every state file is written. | -| `SWW_LOG_LEVEL` | `info` | The least severe of `discriminator`'s own messages that are written: `debug`, `info`, `warn` or `error`; request lines are always written. | -| `SWW_LOG_REQUEST_HEADERS` | `accept,accept-language,accept-encoding,content-type,origin,range` | The request headers whose values the request log records; `Authorization` and `Cookie` are never recorded. | -| `SWW_LOG_REMOTE_URL` | unset | A syslog server that also gets every line `discriminator` writes: `syslog+udp://`, `syslog+tcp://` or `syslog+tls://` with a host and a port. | -| `SWW_LOG_REMOTE_TLS_CA_FILE` | unset | A file of PEM certificates a `syslog+tls` server's certificate must chain to, instead of the host's own. | -| `SWW_LOG_REMOTE_BUFFER` | `10000` | The most lines held while they wait to be sent; past it, the oldest is dropped. | -| `SWW_LOG_REMOTE_FACILITY` | `local0` | The syslog facility of each record sent. | -| `SWW_LOG_REMOTE_APP_NAME` | `SWW_INSTANCE_NAME` | The app name in each syslog record, 1 to 48 printable ASCII characters without a space. | -| `SWW_ADMIN_TOKEN` | unset | The bearer token for the ban and client endpoints, at least 32 characters; unset, they answer `404`. | -| `SWW_METRICS_TOKEN` | unset | The bearer token for `/_sww/metrics`, at least 32 characters; unset, the metrics are off. | -| `SWW_METRICS_TOP_N` | `50` | How many AS numbers and how many countries get metric series of their own; the rest are summed as `other`. | -| `SWW_ALERT_WEBHOOK_URL` | unset | A webhook each alert is posted to as a JSON object. | -| `SWW_ALERT_WEBHOOK_HEADERS` | empty | Headers sent with each post, a list of `Name:value`, such as `Authorization:Bearer `. | -| `SWW_ALERT_SLACK_WEBHOOK_URL` | unset | A Slack incoming webhook each alert is posted to as a message. | -| `SWW_ALERT_NTFY_URL` | unset | An ntfy topic each alert is published to, as its full URL, such as `https://ntfy.sh/my-alerts`. | -| `SWW_ALERT_NTFY_TOKEN` | unset | An ntfy access token sent with each alert. | -| `SWW_ALERT_EVENTS` | `ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure,file_error` | The events alerts are sent for; the hourly `summary` is sent whatever it says. | -| `SWW_ALERT_COOLDOWN` | `15m` | How long after an alert its repeats are held back and counted, a duration or `off`. | -| `SWW_ALERT_MAX_PER_HOUR` | `60` | The most alerts sent in an hour, a number or `off`; the rest are counted in the hour's `summary`. | -| `SWW_ANOMALY_CLIENT_REQUESTS_PER_MINUTE` | `off` | Alert when one client makes more requests than this in a minute, a count. | -| `SWW_ANOMALY_CLIENT_REQUESTS_PER_HOUR` | `off` | Alert when one client makes more requests than this in an hour, a count. | -| `SWW_ANOMALY_CLIENT_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for one client in a minute, a size. | -| `SWW_ANOMALY_CLIENT_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for one client in an hour, a size. | -| `SWW_ANOMALY_NET_REQUESTS_PER_MINUTE` | `off` | Alert when the netblock around a client makes more requests than this in a minute, a count. | -| `SWW_ANOMALY_NET_REQUESTS_PER_HOUR` | `off` | Alert when the netblock around a client makes more requests than this in an hour, a count. | -| `SWW_ANOMALY_NET_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for the netblock around a client in a minute, a size. | -| `SWW_ANOMALY_NET_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for the netblock around a client in an hour, a size. | -| `SWW_ANOMALY_NET_V4_PREFIX` | `24` | The prefix length of the netblock around an IPv4 client that the `SWW_ANOMALY_NET_` thresholds count, from 0 to 32. | -| `SWW_ANOMALY_NET_V6_PREFIX` | `48` | The prefix length of the netblock around an IPv6 client that the `SWW_ANOMALY_NET_` thresholds count, from 0 to 128. | -| `SWW_ANOMALY_ASN_REQUESTS_PER_MINUTE` | `off` | Alert when one AS number makes more requests than this in a minute, a count. | -| `SWW_ANOMALY_ASN_REQUESTS_PER_HOUR` | `off` | Alert when one AS number makes more requests than this in an hour, a count. | -| `SWW_ANOMALY_ASN_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for one AS number in a minute, a size. | -| `SWW_ANOMALY_ASN_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for one AS number in an hour, a size. | -| `SWW_ANOMALY_TOTAL_REQUESTS_PER_MINUTE` | `off` | Alert when the whole service gets more requests than this in a minute, a count. | -| `SWW_ANOMALY_TOTAL_REQUESTS_PER_HOUR` | `off` | Alert when the whole service gets more requests than this in an hour, a count. | -| `SWW_ANOMALY_TOTAL_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for the whole service in a minute, a size. | -| `SWW_ANOMALY_TOTAL_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for the whole service in an hour, a size. | -| `SWW_WATCH_NETS` | empty | Named netblocks the `SWW_WATCH_` thresholds count each as a whole, such as `office=203.0.113.0/24,scraper-x=198.51.100.0/22`. | -| `SWW_WATCH_REQUESTS_PER_MINUTE` | `off` | Alert when a watched netblock makes more requests than this in a minute, a count. | -| `SWW_WATCH_REQUESTS_PER_HOUR` | `off` | Alert when a watched netblock makes more requests than this in an hour, a count. | -| `SWW_WATCH_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for a watched netblock in a minute, a size. | -| `SWW_WATCH_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for a watched netblock in an hour, a size. | +| Variable | Default | What it does | +| ---------------------------------------- | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `SWW_COMMAND` | `rproxy` | Which command the image runs, and its health check checks: `rproxy` or `discriminator`. | +| `SWRP_WAF_BACKEND` | required | `discriminator`'s URL, such as `http://discriminator:8080`. | +| `SWRP_UPSTREAM_URL` | `http://127.0.0.1:8081` | The app's URL: a scheme, a host and an optional port, nothing more. | +| `SWRP_LISTEN_ADDR` | `:8080` | Where `rproxy` listens; leave the host part empty. | +| `SWRP_TRUSTED_PROXIES` | `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16` | The netblocks whose `X-Forwarded-For` `rproxy` believes; set but empty, it believes none. | +| `SWRP_TIMEOUT_MS` | `2000` | How long `rproxy` waits for `discriminator`'s answer to a question and to a report, in milliseconds. | +| `SWRP_DEFAULT_DISPOSITION` | `allow` | What `rproxy` does with a request when `discriminator` cannot be reached or does not answer in time: `allow` passes it to the app, `deny` answers `503`. | +| `SWRP_MAX_REQUEST_SIZE` | `100M` | The largest request body, a size or `off`; a larger one is answered `413`. | +| `SWRP_MAX_RESPONSE_SIZE` | `1G` | The largest response body, a size or `off`. The visitor gets `502` in place of a larger one, or it is cut off if it has already started. | +| `SWRP_MAX_REQUEST_TIME_MS` | `60000` | Two time limits, each this long, in milliseconds or `off`: one for the visitor to send its request line and headers, then one for the whole request to reach the app. | +| `SWRP_MAX_RESPONSE_TIME_MS` | `1800000` | How long the answer may take, from the end of the request to its last byte reaching the visitor, in milliseconds or `off`. | +| `SWW_LISTEN_ADDR` | `:8080` | Where `discriminator` listens; leave the host part empty. | +| `SWW_INSTANCE_NAME` | the container's host name | The name in every line `discriminator` logs, and in every alert and metric, such as `fsn1/discriminator`. | +| `SWW_MODE` | `enforce` | `observe` refuses nothing and makes no ban; each log line says what `enforce` would have done. | +| `SWW_IPV6_GROUP_PREFIX` | `64` | The prefix length of the IPv6 netblock that counts as one client, from 32 to 128. | +| `SWW_MAX_TRACKED_CLIENTS` | `20000` | The most clients kept, in memory and in `clients.json`; the least recently seen is dropped first. | +| `SWW_MAX_BANS` | `5000` | The most bans `discriminator` keeps of its own; bans an admin made do not count. | +| `SWW_ALLOW_NETS` | empty | Netblocks whose clients skip every check, such as your monitoring. | +| `SWW_RATE_LIMIT_EXEMPT_NETS` | empty | Netblocks whose clients are exempt from rate limiting and the byte limits. | +| `SWW_DENY_NETS` | empty | Netblocks whose clients are refused. | +| `SWW_RATE_LIMIT_PER_MINUTE` | `120` | The most requests a client may make in a minute, a count or `off`; the next request is refused and bans the client. | +| `SWW_RATE_LIMIT_PER_HOUR` | `1500` | The most requests a client may make in an hour, a count or `off`; the next request is refused and bans the client. | +| `SWW_RATE_LIMIT_PER_DAY` | `6000` | The most requests a client may make in a day, a count or `off`; the next request is refused and bans the client. | +| `SWW_RATE_LIMIT_EXEMPT_PATHS` | empty | Path prefixes exempt from rate limiting and the byte limits, such as `/assets/`. | +| `SWW_BYTES_LIMIT_PER_MINUTE` | `off` | The most bytes counted for a client in a minute, a size or `off`. `SWW_BYTES_COUNT` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. | +| `SWW_BYTES_LIMIT_PER_HOUR` | `50G` | The most bytes counted for a client in an hour, a size or `off`. `SWW_BYTES_COUNT` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. | +| `SWW_BYTES_LIMIT_PER_DAY` | `100G` | The most bytes counted for a client in a day, a size or `off`. `SWW_BYTES_COUNT` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. | +| `SWW_BYTES_COUNT` | `both` | What the byte limits and the byte anomaly thresholds count: `response`, the app's response bodies; `request`, the visitors' request bodies; or `both`. What the visitor sent over a WebSocket counts as request bytes, and what the app sent as response bytes. | +| `SWW_ERROR_BURST_THRESHOLD` | `30` | The most refusals a client may get in a minute from a rule file, a trap path or the Core Rule Set, a count or `off`. The next one bans the client. | +| `SWW_BAN_RESPONSE` | `403` | How a refused client is answered: `403`, `429` or `close`, which drops the connection. | +| `SWW_LIMIT_BAN_DURATION` | `1h` | The ban for a first broken limit. | +| `SWW_LIMIT_BAN_REPEAT_WINDOW` | `24h` | A client that breaks a limit again within this time after its last such ban ended gets a ban three times as long. | +| `SWW_MAX_BAN_DURATION` | `7d` | A ban for a broken limit that would be longer than this is permanent instead. | +| `SWW_ATTACK_BAN_DURATION` | `7d` | The ban for a first clear sign of attack: a `ban` rule or a trap path. | +| `SWW_BAN_SCOPE_V4_PREFIX` | `32` | The prefix length of the netblock a ban on an IPv4 client covers, from 0 to 32. | +| `SWW_LOOKUP_SOURCE` | `geojs` | Where each client's AS number and country come from: `geojs` (a web service), `file` (a local database) or `off`. | +| `SWW_LOOKUP_DB_PATH` | empty | The IPinfo Lite database file, in its `.mmdb` form, read while the source is `file`. | +| `SWW_LOOKUP_TIMEOUT` | `800ms` | How long a request waits for its client's first GeoJS answer when a setting needs it, and how long a request to GeoJS may take. | +| `SWW_ADD_LOOKUP_HEADERS` | `false` | `true` passes the app the client's AS number and country in `X-Client-ASN` and `X-Client-Country`. | +| `SWW_EXCLUSIVELY_ALLOWED_COUNTRIES` | empty | When set, the only countries whose clients get through; a client the lookup cannot place is refused too. | +| `SWW_TAG_LISTS` | empty | Address lists that each add a tag to their clients' requests, as `name=URL` pairs, such as `torexit=https://example.com/tor.txt`. | +| `SWW_TAG_DENY` | empty | Tags whose requests are refused, such as `torexit,country=CN`. | +| `SWW_TAG_LIMIT_PERCENT` | empty | Tags whose clients get a percentage, from 1 to 99, of every request and byte limit, such as `vpn:50,asn=14061:25`; the lowest applies. | +| `SWW_ROUTEWATCH_URL` | unset | The URL of a routewatch instance, which gives each client's `asn=` and `prefix=` tags. Unset, nothing asks routewatch. | +| `SWW_ASN_BYTES_PERCENT` | empty | A percentage of the byte limits alone for the AS numbers listed, in place of their `asn=` tag's percentage. | +| `SWW_COUNTRY_BYTES_PERCENT` | empty | A percentage of the byte limits alone for the countries listed, in place of their `country=` tag's percentage. | +| `SWW_UNKNOWN_LIMIT_PERCENT` | `100` | The percentage of every limit a client without a country gets. | +| `SWW_ASN_LIMIT_PERCENT_URL` | unset | The URL of a text file of `AS:` lines, fetched like a blocklist; each line counts as an `asn=` tag in `SWW_TAG_LIMIT_PERCENT` (the lowest wins). | +| `SWW_RULES_DIR` | `/etc/sww/rules.d` | The directory of the rule files, read at start and again when a file there changes. | +| `SWW_RULES_ENABLED` | `true` | `false` reads no rule file. | +| `SWW_TRAP_PATHS` | empty | Paths only scanners ask for, such as `/wp-login.php,/xmlrpc.php`; a request for one bans the client. | +| `SWW_WAF_MODE` | `block` | What the Core Rule Set does with a match: `block` refuses it with `403`, `detect` only logs and alerts it, `off` runs no rule. | +| `SWW_WAF_PARANOIA_LEVEL` | `1` | The Core Rule Set's paranoia level, from 1 to 4; each level up finds more attacks and takes more ordinary requests for attacks. | +| `SWW_WAF_ANOMALY_THRESHOLD` | `5` | The Core Rule Set score at which a request is a match, a whole number or `off`; one critical rule scores 5. | +| `SWW_WAF_DISABLED_RULES` | `920340,920420,920440,920640,930130,930140` | Core Rule Set rule ids switched off; a list you give replaces the default, so repeat these six to keep them off. | +| `SWW_WAF_EXEMPT_PATHS` | empty | Path prefixes the Core Rule Set does not inspect. | +| `SWW_WAF_BODY_LIMIT` | `off` | How much of a request body the Core Rule Set reads, a size up to `64K`; `off` reads none. | +| `SWW_CORE_RULE_SET_EXCLUSIONS_FILE` | unset | A file of Core Rule Set exclusions, as Coraza directives, that switch rules off for some fields or paths only. | +| `SWW_BLOCKLIST_URLS` | empty | Blocklists to fetch, text files of addresses and netblocks one to a line, such as `https://www.spamhaus.org/drop/drop.txt`. | +| `SWW_BLOCKLIST_REFRESH` | `24h` | How long after a list was last tried it is fetched again, `1h` or more. | +| `SWW_BLOCKLIST_ACTION` | `deny` | What a client on a blocklist gets: `deny`, `limit:` of every limit, or `log`. | +| `SWW_DNSBL_ZONES` | empty | DNS blocklists asked about each new client in the background, such as `dnsbl.dronebl.org`. | +| `SWW_DNSBL_RESOLVER` | unset | The resolver the zones are asked through, an IP address with an optional port; unset, the host's resolver is used. | +| `SWW_ABUSEIPDB_KEY` | unset | The key of your AbuseIPDB account; set, a client that has committed an offence is checked in the background. | +| `SWW_ABUSEIPDB_MIN_SCORE` | `75` | The least AbuseIPDB abuse confidence score, from 0 to 100, that is a hit. | +| `SWW_ABUSEIPDB_DAILY_BUDGET` | `900` | The most AbuseIPDB checks made in a day. | +| `SWW_CROWDSEC_LAPI_URL` | unset | The local API of a CrowdSec engine you run, such as `http://172.17.0.1:8080`; its decision list is fetched every minute, and a listed client is banned. | +| `SWW_CROWDSEC_LAPI_KEY` | unset | The key the engine gave for `smallwebwaf`, as `cscli bouncers add smallwebwaf` prints it; set with the URL, never alone. | +| `SWW_REPUTATION_ACTION` | `limit:25` | What a client listed by a DNS blocklist, or scored a hit by AbuseIPDB, gets: `deny`, `limit:` or `log`. | +| `SWW_REPUTATION_CACHE_TTL` | `24h` | How long a DNS blocklist verdict or an AbuseIPDB score is used. | +| `SWW_REPUTATION_TIMEOUT` | `2s` | How long a DNS blocklist query or an AbuseIPDB check may take. | +| `SWW_STATE_DIR` | `/var/lib/sww` | The directory of the state files, an absolute path. | +| `SWW_STATE_WRITE_DELAY` | `10s` | How long `bans.json` waits to be written after a ban is made, lifted or made permanent. | +| `SWW_STATE_COUNTER_INTERVAL` | `15m` | How often every state file is written. | +| `SWW_LOG_LEVEL` | `info` | The least severe of `discriminator`'s own messages that are written: `debug`, `info`, `warn` or `error`; request lines are always written. | +| `SWW_LOG_REQUEST_HEADERS` | `accept,accept-language,accept-encoding,content-type,origin,range` | The request headers whose values the request log records; `Authorization` and `Cookie` are never recorded. | +| `SWW_LOG_REMOTE_URL` | unset | A syslog server that also gets every line `discriminator` writes: `syslog+udp://`, `syslog+tcp://` or `syslog+tls://` with a host and a port. | +| `SWW_LOG_REMOTE_TLS_CA_FILE` | unset | A file of PEM certificates a `syslog+tls` server's certificate must chain to, instead of the host's own. | +| `SWW_LOG_REMOTE_BUFFER` | `10000` | The most lines held while they wait to be sent; past it, the oldest is dropped. | +| `SWW_LOG_REMOTE_FACILITY` | `local0` | The syslog facility of each record sent. | +| `SWW_LOG_REMOTE_APP_NAME` | `SWW_INSTANCE_NAME` | The app name in each syslog record, 1 to 48 printable ASCII characters without a space. | +| `SWW_ADMIN_TOKEN` | unset | The bearer token for the ban and client endpoints, at least 32 characters; unset, they answer `404`. | +| `SWW_METRICS_TOKEN` | unset | The bearer token for `/_sww/metrics`, at least 32 characters; unset, the metrics are off. | +| `SWW_METRICS_TOP_N` | `50` | How many AS numbers and how many countries get metric series of their own; the rest are summed as `other`. | +| `SWW_ALERT_WEBHOOK_URL` | unset | A webhook each alert is posted to as a JSON object. | +| `SWW_ALERT_WEBHOOK_HEADERS` | empty | Headers sent with each post, a list of `Name:value`, such as `Authorization:Bearer `. | +| `SWW_ALERT_SLACK_WEBHOOK_URL` | unset | A Slack incoming webhook each alert is posted to as a message. | +| `SWW_ALERT_NTFY_URL` | unset | An ntfy topic each alert is published to, as its full URL, such as `https://ntfy.sh/my-alerts`. | +| `SWW_ALERT_NTFY_TOKEN` | unset | An ntfy access token sent with each alert. | +| `SWW_ALERT_EVENTS` | `ban,permanent_ban,core_rule_set_match,anomaly,reputation_hit,source_failure,file_error` | The events alerts are sent for; the hourly `summary` is sent whatever it says. | +| `SWW_ALERT_COOLDOWN` | `15m` | How long after an alert its repeats are held back and counted, a duration or `off`. | +| `SWW_ALERT_MAX_PER_HOUR` | `60` | The most alerts sent in an hour, a number or `off`; the rest are counted in the hour's `summary`. | +| `SWW_ANOMALY_CLIENT_REQUESTS_PER_MINUTE` | `off` | Alert when one client makes more requests than this in a minute, a count. | +| `SWW_ANOMALY_CLIENT_REQUESTS_PER_HOUR` | `off` | Alert when one client makes more requests than this in an hour, a count. | +| `SWW_ANOMALY_CLIENT_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for one client in a minute, a size. | +| `SWW_ANOMALY_CLIENT_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for one client in an hour, a size. | +| `SWW_ANOMALY_NET_REQUESTS_PER_MINUTE` | `off` | Alert when the netblock around a client makes more requests than this in a minute, a count. | +| `SWW_ANOMALY_NET_REQUESTS_PER_HOUR` | `off` | Alert when the netblock around a client makes more requests than this in an hour, a count. | +| `SWW_ANOMALY_NET_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for the netblock around a client in a minute, a size. | +| `SWW_ANOMALY_NET_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for the netblock around a client in an hour, a size. | +| `SWW_ANOMALY_NET_V4_PREFIX` | `24` | The prefix length of the netblock around an IPv4 client that the `SWW_ANOMALY_NET_` thresholds count, from 0 to 32. | +| `SWW_ANOMALY_NET_V6_PREFIX` | `48` | The prefix length of the netblock around an IPv6 client that the `SWW_ANOMALY_NET_` thresholds count, from 0 to 128. | +| `SWW_ANOMALY_ASN_REQUESTS_PER_MINUTE` | `off` | Alert when one AS number makes more requests than this in a minute, a count. | +| `SWW_ANOMALY_ASN_REQUESTS_PER_HOUR` | `off` | Alert when one AS number makes more requests than this in an hour, a count. | +| `SWW_ANOMALY_ASN_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for one AS number in a minute, a size. | +| `SWW_ANOMALY_ASN_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for one AS number in an hour, a size. | +| `SWW_ANOMALY_TOTAL_REQUESTS_PER_MINUTE` | `off` | Alert when the whole service gets more requests than this in a minute, a count. | +| `SWW_ANOMALY_TOTAL_REQUESTS_PER_HOUR` | `off` | Alert when the whole service gets more requests than this in an hour, a count. | +| `SWW_ANOMALY_TOTAL_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for the whole service in a minute, a size. | +| `SWW_ANOMALY_TOTAL_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for the whole service in an hour, a size. | +| `SWW_WATCH_NETS` | empty | Named netblocks the `SWW_WATCH_` thresholds count each as a whole, such as `office=203.0.113.0/24,scraper-x=198.51.100.0/22`. | +| `SWW_WATCH_REQUESTS_PER_MINUTE` | `off` | Alert when a watched netblock makes more requests than this in a minute, a count. | +| `SWW_WATCH_REQUESTS_PER_HOUR` | `off` | Alert when a watched netblock makes more requests than this in an hour, a count. | +| `SWW_WATCH_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for a watched netblock in a minute, a size. | +| `SWW_WATCH_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_BYTES_COUNT` counts them, are counted for a watched netblock in an hour, a size. | [`docs/settings.md`](docs/settings.md) gives the values each setting takes in full, and the limits that are fixed rather than settings. @@ -469,7 +469,7 @@ services: # Alerts to an ntfy topic, with its token read from a file. SWW_ALERT_NTFY_URL: https://ntfy.example/smallwebwaf SWW_ALERT_NTFY_TOKEN_FILE: /etc/sww/tokens/ntfy - SWW_ALERT_EVENTS: ban,permanent_ban,waf_block,source_failure,file_error + SWW_ALERT_EVENTS: ban,permanent_ban,core_rule_set_match,source_failure,file_error volumes: - discriminator-state:/var/lib/sww - /srv/ipinfo:/var/lib/ipinfo:ro @@ -632,9 +632,10 @@ is set aside as `.bad`, and never stops the running process (see - `internal/rproxy`: the reverse proxy in front of one app. - `internal/discriminator`: `discriminator`'s server, which runs the checks and answers its own endpoints. -- `internal/ratelimit`, `internal/bans`, `internal/rules`, `internal/waf`, - `internal/lookup`, `internal/reputation`, `internal/routewatch`, - `internal/anomaly` and `internal/alerts`: one check or source each. +- `internal/ratelimit`, `internal/bans`, `internal/rules`, + `internal/coreruleset`, `internal/lookup`, `internal/reputation`, + `internal/routewatch`, `internal/anomaly` and `internal/alerts`: one check or + source each. - `internal/state`, `internal/requestlog`, `internal/remotelog` and `internal/metrics`: the state files, the log lines, the syslog sender and the metrics. diff --git a/SPEC.md b/SPEC.md index 1371cc8..e3b5123 100644 --- a/SPEC.md +++ b/SPEC.md @@ -81,8 +81,8 @@ to them. of bans is full. - `internal/rules`: reads the rule files at start and again when they change, and tells which rules a request matches. - - `internal/waf`: the Core Rule Set, with the eight changes listed under - "Attack detection", as Coraza directives. + - `internal/coreruleset`: the Core Rule Set, with the eight changes listed + under "Attack detection", as Coraza directives. - `internal/lookup`: looks clients up through GeoJS or in the lookup database. - `internal/reputation`: fetches the lists, asks the DNS blocklists and @@ -655,7 +655,7 @@ files, the limits and the bans still apply to the client that sends it. match, and scores are still logged. A request with more than 1000 query parameters or body fields adds 5 to its score, since Coraza reads only the first 1000. For a match, in `block` and `detect` mode alike, the log line - gives the rule ids and the score, and a `waf_block` alert is sent. + gives the rule ids and the score, and a `core_rule_set_match` alert is sent. - `SWW_WAF_DISABLED_RULES` switches off rule ids when an app trips a false positive. The default switches off six rules: - 920340, 920420 and 920640 refuse a request for the type of its body, when @@ -789,8 +789,8 @@ every app, and no setting undoes them: such as `owner/python3-app` would have that push refused. 8. On gitea's paths, some fields skip the rules that refused ordinary gitea requests there. They are the fields where gitea sends a search, a file name - or a branch name. `internal/waf/gitea.conf` lists each rule and the request - it refused. The fields are: + or a branch name. `internal/coreruleset/gitea.conf` lists each rule and the + request it refused. The fields are: - `q` on gitea's searches of code, of issues and pull requests, and of commits. A code search holds code, such as `