Compare commits
22
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
fde205af50 | ||
|
|
98d7432077 | ||
|
|
11ac8aee50 | ||
|
|
e7cc0adbdb | ||
|
|
805c64a256 | ||
|
|
967ccdf75e | ||
|
|
809d2664d7 | ||
|
|
01e6b5d47e | ||
|
|
5099448f79 | ||
|
|
eb48796f27 | ||
|
|
9f42460f70 | ||
|
|
9cb8f362bc | ||
|
|
48edab5dad | ||
|
|
dadd5768d5 | ||
|
|
e6a081134a | ||
|
|
4cb91a4870 | ||
|
|
b9972af6a0 | ||
|
|
c0d9d5787e | ||
|
|
8c9f20d346 | ||
|
|
caa3079847 | ||
|
|
a93243a294 | ||
|
|
f52b1f9ab5 |
No files matched your search
+14
@@ -161,6 +161,20 @@ RUN git checkout --quiet --detach b4b2c785308b1ce785b6155c7fe5f16879080193 \
|
||||
# ubuntu 26.04, 2026-09-27
|
||||
FROM ubuntu@sha256:f144425ff09be612d6d9ad965196e9cdc23dae1f42110a8a11a3e9a8198759f7
|
||||
|
||||
# The image's own labels, in place of the ones it would inherit from
|
||||
# Ubuntu's. script/docker and script/cibuild pass the build arguments;
|
||||
# a build that passes none labels the image unknown rather than leave
|
||||
# Ubuntu's values.
|
||||
ARG VERSION=unknown
|
||||
ARG CREATED=unknown
|
||||
ARG REVISION=unknown
|
||||
LABEL org.opencontainers.image.title="smallwebwaf" \
|
||||
org.opencontainers.image.description="A web application firewall for self-hosted web apps: sww rproxy in front of an app, and sww waf, which decides." \
|
||||
org.opencontainers.image.version="${VERSION}" \
|
||||
org.opencontainers.image.created="${CREATED}" \
|
||||
org.opencontainers.image.revision="${REVISION}" \
|
||||
org.opencontainers.image.source="https://git.eeqj.de/sneak/smallwebwaf"
|
||||
|
||||
# runit's install creates its _runit-log user with minsysusers, which
|
||||
# reads this file in place of runit's /usr/lib/sysusers.d/runit.conf.
|
||||
# runit's line leaves out the shell, and minsysusers prints a Perl
|
||||
|
||||
@@ -75,7 +75,8 @@ Together, `rproxy` and `waf` protect against:
|
||||
- give the clients of chosen networks and countries a fraction of every
|
||||
limit, so that they are banned sooner;
|
||||
- refuse or limit clients on a blocklist, a DNS blocklist, or with a high
|
||||
AbuseIPDB score.
|
||||
AbuseIPDB score;
|
||||
- ban the clients on AbuseIPDB's blacklist before they send a request.
|
||||
|
||||
For the operator, it:
|
||||
|
||||
@@ -284,129 +285,136 @@ account, a key, a token or an alert destination stays off until you give one.
|
||||
`SWW_WAF_BYTES_COUNTED` says. Headers are never counted. What the visitor sent
|
||||
over a WebSocket counts as request bytes, and what the app sent as response
|
||||
bytes.
|
||||
- An anomaly is more requests or bytes than an `SWW_WAF_ANOMALY_` threshold
|
||||
allows. Crossing a threshold sends one `anomaly` alert to the alert
|
||||
destinations, and does nothing else: no client is banned, rate limited or
|
||||
blocked.
|
||||
- An invalid setting stops the process at start with one log line naming the
|
||||
variable and what is wrong with the value. The settings in effect are logged
|
||||
on one line at start, with tokens, keys and webhook paths shown as `********`.
|
||||
|
||||
Every setting the code reads, with its default:
|
||||
|
||||
| Variable | Default | What it does |
|
||||
| ------------------------------------------------- | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `SWW_COMMAND` | `rproxy` | Which command the image runs, and its health check checks: `rproxy` or `waf`. |
|
||||
| `SWW_RPROXY_WAF_BACKEND` | required | `waf`'s URL, such as `http://waf:8080`. |
|
||||
| `SWW_RPROXY_UPSTREAM_URL` | `http://127.0.0.1:8081` | The app's URL: a scheme, a host and an optional port, nothing more. |
|
||||
| `SWW_RPROXY_LISTEN_ADDR` | `:8080` | Where `rproxy` listens; leave the host part empty. |
|
||||
| `SWW_RPROXY_TRUSTED_PROXIES` | `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16` | The netblocks whose `X-Forwarded-For` `rproxy` believes; set but empty, it believes none. |
|
||||
| `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS` | `2000` | How long `rproxy` waits for `waf`'s answer to a question and to a report, in milliseconds. |
|
||||
| `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` | `allow` | What `rproxy` does with a request when `waf` gives no answer in time, and while the circuit breaker for asks to `waf` is open: `allow` passes it to the app, `deny` answers `503`. |
|
||||
| `SWW_RPROXY_MAX_REQUEST_SIZE` | `100M` | The largest request body, a size or `off`; a larger one is answered `413`. |
|
||||
| `SWW_RPROXY_MAX_RESPONSE_SIZE` | `1G` | The largest response body, a size or `off`. The visitor gets `502` in place of a larger one, or it is cut off if it has already started. |
|
||||
| `SWW_RPROXY_MAX_REQUEST_TIME_MS` | `60000` | Two time limits, each this long, in milliseconds or `off`: one for the visitor to send its request line and headers, then one for the whole request to reach the app. |
|
||||
| `SWW_RPROXY_MAX_RESPONSE_TIME_MS` | `1800000` | How long the answer may take, from the end of the request to its last byte reaching the visitor, in milliseconds or `off`. |
|
||||
| `SWW_WAF_LISTEN_ADDR` | `:8080` | Where `waf` listens; leave the host part empty. |
|
||||
| `SWW_WAF_INSTANCE_NAME` | the container's host name | The name in every line `waf` logs, and in every alert and metric, such as `host1/waf`. |
|
||||
| `SWW_WAF_MODE` | `enforce` | `observe` refuses nothing and makes no ban; each log line says what `enforce` would have done. |
|
||||
| `SWW_WAF_IPV6_GROUP_PREFIX` | `64` | The prefix length of the IPv6 netblock that counts as one client, from 32 to 128. |
|
||||
| `SWW_WAF_MAX_TRACKED_CLIENTS` | `20000` | The most clients kept, in memory and in `clients.json`; the least recently seen is dropped first. |
|
||||
| `SWW_WAF_MAX_BANS` | `5000` | The most bans `waf` keeps of its own; bans an admin made do not count. |
|
||||
| `SWW_WAF_ALLOW_NETS` | empty | Netblocks whose clients skip every check, such as your monitoring. |
|
||||
| `SWW_WAF_RATE_LIMIT_EXEMPT_NETS` | empty | Netblocks whose clients are exempt from rate limiting and the byte limits. |
|
||||
| `SWW_WAF_DENY_NETS` | empty | Netblocks whose clients are refused. |
|
||||
| `SWW_WAF_RATE_LIMIT_PER_MINUTE` | `120` | The most requests a client may make in a minute, a count or `off`; the next request is refused and bans the client. |
|
||||
| `SWW_WAF_RATE_LIMIT_PER_HOUR` | `1500` | The most requests a client may make in an hour, a count or `off`; the next request is refused and bans the client. |
|
||||
| `SWW_WAF_RATE_LIMIT_PER_DAY` | `6000` | The most requests a client may make in a day, a count or `off`; the next request is refused and bans the client. |
|
||||
| `SWW_WAF_RATE_LIMIT_EXEMPT_PATHS` | empty | Path prefixes exempt from rate limiting and the byte limits, such as `/assets/`. |
|
||||
| `SWW_WAF_BYTES_LIMIT_PER_MINUTE` | `off` | The most bytes counted for a client in a minute, a size or `off`. `SWW_WAF_BYTES_COUNTED` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. |
|
||||
| `SWW_WAF_BYTES_LIMIT_PER_HOUR` | `50G` | The most bytes counted for a client in an hour, a size or `off`. `SWW_WAF_BYTES_COUNTED` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. |
|
||||
| `SWW_WAF_BYTES_LIMIT_PER_DAY` | `100G` | The most bytes counted for a client in a day, a size or `off`. `SWW_WAF_BYTES_COUNTED` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. |
|
||||
| `SWW_WAF_BYTES_COUNTED` | `both` | What the byte limits and the byte anomaly thresholds count: `response`, the app's response bodies; `request`, the visitors' request bodies; or `both`. What the visitor sent over a WebSocket counts as request bytes, and what the app sent as response bytes. |
|
||||
| `SWW_WAF_REFUSALS_LIMIT_PER_MINUTE` | `30` | The most refusals a client may get in a minute from a rule file, a trap path or the Core Rule Set, a count or `off`. The next one bans the client. |
|
||||
| `SWW_WAF_BAN_RESPONSE` | `403` | How a refused client is answered: `403`, `429` or `close`, which drops the connection. |
|
||||
| `SWW_WAF_LIMIT_BAN_DURATION` | `1h` | The ban for a first broken limit. |
|
||||
| `SWW_WAF_LIMIT_BAN_REPEAT_WINDOW` | `24h` | A client that breaks a limit again within this time after its last such ban ended gets a ban three times as long. |
|
||||
| `SWW_WAF_LIMIT_BAN_PERMANENT_ABOVE` | `7d` | A ban for a broken limit that would be longer than this is permanent instead. |
|
||||
| `SWW_WAF_ATTACK_BAN_DURATION` | `7d` | The ban for a first clear sign of attack: a `ban` rule or a trap path. |
|
||||
| `SWW_WAF_BAN_SCOPE_V4_PREFIX` | `32` | The prefix length of the netblock a ban on an IPv4 client covers, from 0 to 32. |
|
||||
| `SWW_WAF_LOOKUP_SOURCE` | `geojs` | Where each client's AS number and country come from: `geojs` (a web service), `file` (a local database) or `off`. |
|
||||
| `SWW_WAF_LOOKUP_DB_PATH` | empty | The IPinfo Lite database file, in its `.mmdb` form, read while the source is `file`. |
|
||||
| `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` | `500` | How long a request waits for its client's first GeoJS answer when a setting needs it, and how long a request to GeoJS may take, in milliseconds. |
|
||||
| `SWW_WAF_ADD_LOOKUP_HEADERS` | `true`, or `false` while the lookup source is `off` | `true` passes the app the client's AS number and country in `X-Client-ASN` and `X-Client-Country`. |
|
||||
| `SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES` | empty | When set, the only countries whose clients get through; a client the lookup cannot place is refused too. |
|
||||
| `SWW_WAF_TAG_LISTS` | empty | Address lists that each add a tag to their clients' requests, as `name=URL` pairs, such as `torexit=https://example.com/tor.txt`. |
|
||||
| `SWW_WAF_TAG_DENY` | empty | Tags whose requests are refused, such as `torexit,country=CN`. |
|
||||
| `SWW_WAF_TAG_LIMIT_PERCENT` | empty | Tags whose clients get a percentage, from 1 to 99, of every request and byte limit, such as `vpn:50,asn=14061:25`; the lowest applies. |
|
||||
| `SWW_WAF_ROUTEWATCH_URL` | unset | The URL of a routewatch instance, which gives each client's `asn=` and `prefix=` tags. Unset, nothing asks routewatch. |
|
||||
| `SWW_WAF_ASN_BYTES_LIMIT_PERCENT` | empty | A percentage of the byte limits alone for the AS numbers listed, in place of their `asn=` tag's percentage. |
|
||||
| `SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT` | empty | A percentage of the byte limits alone for the countries listed, in place of their `country=` tag's percentage. |
|
||||
| `SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT` | `100` | The percentage of every limit a client without a country gets. |
|
||||
| `SWW_WAF_ASN_LIMIT_PERCENT_URL` | unset | The URL of a text file of `AS<number>:<percent>` lines, fetched like a blocklist; each line counts as an `asn=` tag in `SWW_WAF_TAG_LIMIT_PERCENT` (the lowest wins). |
|
||||
| `SWW_WAF_RULE_FILES_DIR` | `/etc/sww/rules.d` | The directory of the rule files, read at start and again when a file there changes. |
|
||||
| `SWW_WAF_RULE_FILES_ENABLED` | `true` | `false` reads no rule file. |
|
||||
| `SWW_WAF_TRAP_PATHS` | empty | Paths only scanners ask for, such as `/wp-login.php,/xmlrpc.php`; a request for one bans the client. |
|
||||
| `SWW_WAF_CORE_RULE_SET_MODE` | `block` | What the Core Rule Set does with a match: `block` refuses it with `403`, `detect` only logs and alerts it, `off` runs no rule. |
|
||||
| `SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL` | `1` | The Core Rule Set's paranoia level, from 1 to 4; each level up finds more attacks and takes more ordinary requests for attacks. |
|
||||
| `SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD` | `5` | The Core Rule Set score at which a request is a match, a whole number or `off`; one critical rule scores 5. |
|
||||
| `SWW_WAF_CORE_RULE_SET_DISABLED_RULES` | `920340,920420,920440,920640,930130,930140` | Core Rule Set rule ids switched off; a list you give replaces the default, so repeat these six to keep them off. |
|
||||
| `SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS` | empty | Path prefixes the Core Rule Set does not inspect. |
|
||||
| `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` | `off` | How much of a request body the Core Rule Set reads, a size up to `8K`; `off` reads none. |
|
||||
| `SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE` | unset | A file of Core Rule Set exclusions, as Coraza directives, that switch rules off for some fields or paths only. |
|
||||
| `SWW_WAF_APPS` | unset | The apps each hostname serves, such as `git.example.com:gitea;blog.example.com:wordpress`; only those apps' fixes apply there. "Attack detection" lists the apps. Listing `forgejo` brings gitea's fixes too. Unset, every app's fixes apply to every hostname. |
|
||||
| `SWW_WAF_BLOCKLIST_URLS` | empty | Blocklists to fetch, text files of addresses and netblocks one to a line, such as `https://www.spamhaus.org/drop/drop.txt`. |
|
||||
| `SWW_WAF_BLOCKLIST_REFRESH` | `24h` | How long after a list was last tried it is fetched again, `1h` or more. |
|
||||
| `SWW_WAF_BLOCKLIST_ACTION` | `deny` | What a client on a blocklist gets: `deny`, `limit:<percent>` of every limit, or `log`. |
|
||||
| `SWW_WAF_DNSBL_ZONES` | empty | DNS blocklists asked about each new client in the background, such as `dnsbl.dronebl.org`. |
|
||||
| `SWW_WAF_DNSBL_RESOLVER` | unset | The resolver the zones are asked through, an IP address with an optional port; unset, the host's resolver is used. |
|
||||
| `SWW_WAF_ABUSEIPDB_KEY` | unset | The key of your AbuseIPDB account; set, a client that has committed an offence is checked in the background. |
|
||||
| `SWW_WAF_ABUSEIPDB_MIN_SCORE` | `75` | The least AbuseIPDB abuse confidence score, from 0 to 100, that is a hit. |
|
||||
| `SWW_WAF_ABUSEIPDB_DAILY_BUDGET` | `900` | The most AbuseIPDB checks made in a day. |
|
||||
| `SWW_WAF_DNSBL_ABUSEIPDB_ACTION` | `limit:25` | What a client listed by a DNS blocklist, or scored a hit by AbuseIPDB, gets: `deny`, `limit:<percent>` or `log`. |
|
||||
| `SWW_WAF_IP_REPUTATION_CACHE_TTL` | `24h` | How long a DNS blocklist verdict or an AbuseIPDB score is used. |
|
||||
| `SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS` | `2000` | How long a DNS blocklist query or an AbuseIPDB check may take, in milliseconds. |
|
||||
| `SWW_WAF_STATE_DIR` | `/var/lib/sww` | The directory of the state files, an absolute path. |
|
||||
| `SWW_WAF_BANS_WRITE_DELAY` | `10s` | How long `bans.json` waits to be written after a ban is made, lifted or made permanent. |
|
||||
| `SWW_WAF_STATE_WRITE_INTERVAL` | `15m` | How often every state and cache file is written. |
|
||||
| `SWW_WAF_CACHE_DIR` | `/var/cache/sww` | The directory of the cache files, an absolute path. |
|
||||
| `SWW_WAF_LOG_LEVEL` | `info` | The least severe of `waf`'s own messages that are written: `debug`, `info`, `warn` or `error`; request lines are always written. |
|
||||
| `SWW_WAF_LOG_REQUEST_HEADERS` | `accept,accept-language,accept-encoding,content-type,origin,range` | The request headers whose values the request log records; `Authorization` and `Cookie` are never recorded. |
|
||||
| `SWW_WAF_LOG_REMOTE_URL` | unset | A syslog server that also gets every line `waf` writes: `syslog+udp://`, `syslog+tcp://` or `syslog+tls://` with a host and a port. |
|
||||
| `SWW_WAF_LOG_REMOTE_TLS_CA_FILE` | unset | A file of PEM certificates a `syslog+tls` server's certificate must chain to, instead of the host's own. |
|
||||
| `SWW_WAF_LOG_REMOTE_BUFFER_LINES` | `10000` | The most lines held while they wait to be sent; past it, the oldest is dropped. |
|
||||
| `SWW_WAF_LOG_REMOTE_FACILITY` | `local0` | The syslog facility of each record sent. |
|
||||
| `SWW_WAF_LOG_REMOTE_APP_NAME` | `SWW_WAF_INSTANCE_NAME` | The app name in each syslog record, 1 to 48 printable ASCII characters without a space. |
|
||||
| `SWW_WAF_ADMIN_TOKEN` | unset | The bearer token for the ban and client endpoints, at least 32 characters; unset, they answer `404`. |
|
||||
| `SWW_WAF_METRICS_TOKEN` | unset | The bearer token for `/_sww/metrics`, at least 32 characters; unset, the metrics are off. |
|
||||
| `SWW_WAF_METRICS_BUSIEST_COUNTRIES_AND_ASNS` | `50` | How many AS numbers and how many countries get metric series of their own; the rest are summed as `other`. |
|
||||
| `SWW_WAF_ALERT_WEBHOOK_URL` | unset | A webhook each alert is posted to as a JSON object. |
|
||||
| `SWW_WAF_ALERT_WEBHOOK_HEADERS` | empty | Headers sent with each post, a list of `Name:value`, such as `Authorization:Bearer <token>`. |
|
||||
| `SWW_WAF_ALERT_SLACK_WEBHOOK_URL` | unset | A Slack incoming webhook each alert is posted to as a message. |
|
||||
| `SWW_WAF_ALERT_NTFY_URL` | unset | An ntfy topic each alert is published to, as its full URL, such as `https://ntfy.sh/my-alerts`. |
|
||||
| `SWW_WAF_ALERT_NTFY_TOKEN` | unset | An ntfy access token sent with each alert. |
|
||||
| `SWW_WAF_ALERT_EVENTS` | `ban,permanent_ban,core_rule_set_match,anomaly,reputation_hit,source_failure,file_error` | The events alerts are sent for; the hourly `summary` is sent whatever it says. |
|
||||
| `SWW_WAF_ALERT_COOLDOWN` | `15m` | How long after an alert its repeats are held back and counted, a duration or `off`. |
|
||||
| `SWW_WAF_ALERT_MAX_PER_HOUR` | `60` | The most alerts sent in an hour, a number or `off`; the rest are counted in the hour's `summary`. |
|
||||
| `SWW_WAF_ANOMALY_CLIENT_REQUESTS_PER_MINUTE` | `off` | Alert when one client makes more requests than this in a minute, a count. |
|
||||
| `SWW_WAF_ANOMALY_CLIENT_REQUESTS_PER_HOUR` | `off` | Alert when one client makes more requests than this in an hour, a count. |
|
||||
| `SWW_WAF_ANOMALY_CLIENT_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for one client in a minute, a size. |
|
||||
| `SWW_WAF_ANOMALY_CLIENT_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for one client in an hour, a size. |
|
||||
| `SWW_WAF_ANOMALY_NET_REQUESTS_PER_MINUTE` | `off` | Alert when the netblock around a client makes more requests than this in a minute, a count. |
|
||||
| `SWW_WAF_ANOMALY_NET_REQUESTS_PER_HOUR` | `off` | Alert when the netblock around a client makes more requests than this in an hour, a count. |
|
||||
| `SWW_WAF_ANOMALY_NET_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for the netblock around a client in a minute, a size. |
|
||||
| `SWW_WAF_ANOMALY_NET_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for the netblock around a client in an hour, a size. |
|
||||
| `SWW_WAF_ANOMALY_NET_V4_PREFIX` | `24` | The prefix length of the netblock around an IPv4 client that the `SWW_WAF_ANOMALY_NET_` thresholds count, from 0 to 32. |
|
||||
| `SWW_WAF_ANOMALY_NET_V6_PREFIX` | `48` | The prefix length of the netblock around an IPv6 client that the `SWW_WAF_ANOMALY_NET_` thresholds count, from 0 to 128. |
|
||||
| `SWW_WAF_ANOMALY_ASN_REQUESTS_PER_MINUTE` | `off` | Alert when one AS number makes more requests than this in a minute, a count. |
|
||||
| `SWW_WAF_ANOMALY_ASN_REQUESTS_PER_HOUR` | `off` | Alert when one AS number makes more requests than this in an hour, a count. |
|
||||
| `SWW_WAF_ANOMALY_ASN_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for one AS number in a minute, a size. |
|
||||
| `SWW_WAF_ANOMALY_ASN_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for one AS number in an hour, a size. |
|
||||
| `SWW_WAF_ANOMALY_TOTAL_REQUESTS_PER_MINUTE` | `off` | Alert when the whole service gets more requests than this in a minute, a count. |
|
||||
| `SWW_WAF_ANOMALY_TOTAL_REQUESTS_PER_HOUR` | `off` | Alert when the whole service gets more requests than this in an hour, a count. |
|
||||
| `SWW_WAF_ANOMALY_TOTAL_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for the whole service in a minute, a size. |
|
||||
| `SWW_WAF_ANOMALY_TOTAL_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for the whole service in an hour, a size. |
|
||||
| `SWW_WAF_ANOMALY_WATCHED_NETS` | empty | Named netblocks the `SWW_WAF_ANOMALY_WATCHED_NET_` thresholds count each as a whole, such as `office=203.0.113.0/24,scraper-x=198.51.100.0/22`. |
|
||||
| `SWW_WAF_ANOMALY_WATCHED_NET_REQUESTS_PER_MINUTE` | `off` | Alert when a watched netblock makes more requests than this in a minute, a count. |
|
||||
| `SWW_WAF_ANOMALY_WATCHED_NET_REQUESTS_PER_HOUR` | `off` | Alert when a watched netblock makes more requests than this in an hour, a count. |
|
||||
| `SWW_WAF_ANOMALY_WATCHED_NET_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for a watched netblock in a minute, a size. |
|
||||
| `SWW_WAF_ANOMALY_WATCHED_NET_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for a watched netblock in an hour, a size. |
|
||||
| Variable | Default | What it does |
|
||||
| --------------------------------------------------- | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `SWW_COMMAND` | `rproxy` | Which command the image runs, and its health check checks: `rproxy` or `waf`. |
|
||||
| `SWW_RPROXY_WAF_BACKEND` | required | `waf`'s URL, such as `http://waf:8080`. |
|
||||
| `SWW_RPROXY_UPSTREAM_URL` | `http://127.0.0.1:8081` | The app's URL: a scheme, a host and an optional port, nothing more. |
|
||||
| `SWW_RPROXY_LISTEN_ADDR` | `:8080` | Where `rproxy` listens; leave the host part empty. |
|
||||
| `SWW_RPROXY_TRUSTED_PROXIES` | `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16` | The netblocks whose `X-Forwarded-For` `rproxy` believes; set but empty, it believes none. |
|
||||
| `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS` | `2000` | How long `rproxy` waits for `waf`'s answer to a question and to a report, in milliseconds. |
|
||||
| `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` | `allow` | What `rproxy` does with a request when `waf` gives no answer in time, and while the circuit breaker for asks to `waf` is open: `allow` passes it to the app, `deny` answers `503`. |
|
||||
| `SWW_RPROXY_MAX_REQUEST_SIZE` | `100M` | The largest request body, a size or `off`; a larger one is answered `413`. |
|
||||
| `SWW_RPROXY_MAX_RESPONSE_SIZE` | `1G` | The largest response body, a size or `off`. The visitor gets `502` in place of a larger one, or it is cut off if it has already started. |
|
||||
| `SWW_RPROXY_MAX_REQUEST_TIME_MS` | `60000` | Two time limits, each this long, in milliseconds or `off`: one for the visitor to send its request line and headers, then one for the whole request to reach the app. |
|
||||
| `SWW_RPROXY_MAX_RESPONSE_TIME_MS` | `1800000` | How long the answer may take, from the end of the request to its last byte reaching the visitor, in milliseconds or `off`. |
|
||||
| `SWW_WAF_LISTEN_ADDR` | `:8080` | Where `waf` listens; leave the host part empty. |
|
||||
| `SWW_WAF_INSTANCE_NAME` | the container's host name | The name in every line `waf` logs, and in every alert and metric, such as `host1/waf`. |
|
||||
| `SWW_WAF_MODE` | `enforce` | `observe` refuses nothing and makes no ban; each log line says what `enforce` would have done. |
|
||||
| `SWW_WAF_IPV6_GROUP_PREFIX` | `64` | The prefix length of the IPv6 netblock that counts as one client, from 32 to 128. |
|
||||
| `SWW_WAF_MAX_TRACKED_CLIENTS` | `20000` | The most clients kept, in memory and in `clients.json`; the least recently seen is dropped first. |
|
||||
| `SWW_WAF_MAX_BANS` | `5000` | The most bans `waf` keeps for broken limits and attacks; bans an admin made and bans for AbuseIPDB's blacklist do not count. |
|
||||
| `SWW_WAF_ALLOW_NETS` | empty | Netblocks whose clients skip every check, such as your monitoring. |
|
||||
| `SWW_WAF_RATE_LIMIT_EXEMPT_NETS` | empty | Netblocks whose clients are exempt from rate limiting and the byte limits. |
|
||||
| `SWW_WAF_DENY_NETS` | empty | Netblocks whose clients are refused. |
|
||||
| `SWW_WAF_RATE_LIMIT_PER_MINUTE` | `120` | The most requests a client may make in a minute, a count or `off`; the next request is refused and bans the client. |
|
||||
| `SWW_WAF_RATE_LIMIT_PER_HOUR` | `1500` | The most requests a client may make in an hour, a count or `off`; the next request is refused and bans the client. |
|
||||
| `SWW_WAF_RATE_LIMIT_PER_DAY` | `6000` | The most requests a client may make in a day, a count or `off`; the next request is refused and bans the client. |
|
||||
| `SWW_WAF_RATE_LIMIT_EXEMPT_PATHS` | empty | Path prefixes exempt from rate limiting and the byte limits, such as `/assets/`. |
|
||||
| `SWW_WAF_BYTES_LIMIT_PER_MINUTE` | `off` | The most bytes counted for a client in a minute, a size or `off`. `SWW_WAF_BYTES_COUNTED` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. |
|
||||
| `SWW_WAF_BYTES_LIMIT_PER_HOUR` | `50G` | The most bytes counted for a client in an hour, a size or `off`. `SWW_WAF_BYTES_COUNTED` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. |
|
||||
| `SWW_WAF_BYTES_LIMIT_PER_DAY` | `100G` | The most bytes counted for a client in a day, a size or `off`. `SWW_WAF_BYTES_COUNTED` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. |
|
||||
| `SWW_WAF_BYTES_COUNTED` | `both` | What the byte limits and the byte anomaly thresholds count: `response`, the app's response bodies; `request`, the visitors' request bodies; or `both`. What the visitor sent over a WebSocket counts as request bytes, and what the app sent as response bytes. |
|
||||
| `SWW_WAF_REFUSALS_LIMIT_PER_MINUTE` | `30` | The most refusals a client may get in a minute from a rule file, a trap path or the Core Rule Set, a count or `off`. The next one bans the client. |
|
||||
| `SWW_WAF_BAN_RESPONSE` | `403` | How a refused client is answered: `403`, `429` or `close`, which drops the connection. |
|
||||
| `SWW_WAF_LIMIT_BAN_DURATION` | `1h` | The ban for a first broken limit. |
|
||||
| `SWW_WAF_LIMIT_BAN_REPEAT_WINDOW` | `24h` | A client that breaks a limit again within this time after its last such ban ended gets a ban three times as long. |
|
||||
| `SWW_WAF_LIMIT_BAN_PERMANENT_ABOVE` | `7d` | A ban for a broken limit that would be longer than this is permanent instead. |
|
||||
| `SWW_WAF_ATTACK_BAN_DURATION` | `7d` | The ban for a first clear sign of attack: a `ban` rule or a trap path. |
|
||||
| `SWW_WAF_BAN_SCOPE_V4_PREFIX` | `32` | The prefix length of the netblock a ban on an IPv4 client covers, from 0 to 32. |
|
||||
| `SWW_WAF_LOOKUP_SOURCE` | `geojs` | Where each client's AS number and country come from: `geojs` (a web service), `file` (a local database) or `off`. |
|
||||
| `SWW_WAF_LOOKUP_DB_PATH` | empty | The IPinfo Lite database file, in its `.mmdb` form, read while the source is `file`. |
|
||||
| `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` | `800` | How long a request waits for its client's first GeoJS answer when a setting needs it, in milliseconds. The request to GeoJS goes on, and a later answer is kept. |
|
||||
| `SWW_WAF_ADD_LOOKUP_HEADERS` | `true`, or `false` while the lookup source is `off` | `true` passes the app the client's AS number and country in `X-Client-ASN` and `X-Client-Country`. |
|
||||
| `SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES` | empty | When set, the only countries whose clients get through; a client the lookup cannot place is refused too. |
|
||||
| `SWW_WAF_TAG_LISTS` | empty | Address lists that each add a tag to their clients' requests, as `name=URL` pairs, such as `torexit=https://example.com/tor.txt`. |
|
||||
| `SWW_WAF_TAG_DENY` | empty | Tags whose requests are refused, such as `torexit,country=CN`. |
|
||||
| `SWW_WAF_TAG_LIMIT_PERCENT` | empty | Tags whose clients get a percentage, from 1 to 99, of every request and byte limit, such as `vpn:50,asn=14061:25`; the lowest applies. |
|
||||
| `SWW_WAF_ROUTEWATCH_URL` | unset | The URL of a routewatch instance, which gives each client's `asn=` and `prefix=` tags. Unset, nothing asks routewatch. |
|
||||
| `SWW_WAF_ASN_BYTES_LIMIT_PERCENT` | empty | A percentage of the byte limits alone for the AS numbers listed, in place of their `asn=` tag's percentage. |
|
||||
| `SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT` | empty | A percentage of the byte limits alone for the countries listed, in place of their `country=` tag's percentage. |
|
||||
| `SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT` | `100` | The percentage of every limit a client without a country gets. |
|
||||
| `SWW_WAF_ASN_LIMIT_PERCENT_URL` | unset | The URL of a text file of `AS<number>:<percent>` lines, fetched like a blocklist; each line counts as an `asn=` tag in `SWW_WAF_TAG_LIMIT_PERCENT` (the lowest wins). |
|
||||
| `SWW_WAF_RULE_FILES_DIR` | `/etc/sww/rules.d` | The directory of the rule files, read at start and again when a file there changes. |
|
||||
| `SWW_WAF_RULE_FILES_ENABLED` | `true` | `false` reads no rule file. |
|
||||
| `SWW_WAF_TRAP_PATHS` | empty | Paths only scanners ask for, such as `/wp-login.php,/xmlrpc.php`; a request for one bans the client. |
|
||||
| `SWW_WAF_CORE_RULE_SET_MODE` | `block` | What the Core Rule Set does with a match: `block` refuses it with `403`, `detect` only logs and alerts it, `off` runs no rule. |
|
||||
| `SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL` | `1` | The Core Rule Set's paranoia level, from 1 to 4; each level up finds more attacks and takes more ordinary requests for attacks. |
|
||||
| `SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD` | `5` | The Core Rule Set score at which a request is a match, a whole number or `off`; one critical rule scores 5. |
|
||||
| `SWW_WAF_CORE_RULE_SET_DISABLED_RULES` | `920340,920420,920440,920640,930130,930140` | Core Rule Set rule ids switched off; a list you give replaces the default, so repeat these six to keep them off. |
|
||||
| `SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS` | empty | Path prefixes the Core Rule Set does not inspect. |
|
||||
| `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` | `off` | How much of a request body the Core Rule Set reads, a size up to `8K`; `off` reads none. |
|
||||
| `SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE` | unset | A file of Core Rule Set exclusions, as Coraza directives, that switch rules off for some fields or paths only. |
|
||||
| `SWW_WAF_APPS` | unset | The apps each hostname serves, such as `git.example.com:gitea;blog.example.com:wordpress`; only those apps' fixes apply there. "Attack detection" lists the apps. Listing `forgejo` brings gitea's fixes too. Unset, every app's fixes apply to every hostname. |
|
||||
| `SWW_WAF_BLOCKLIST_URLS` | empty | Blocklists to fetch, text files of addresses and netblocks one to a line, such as `https://www.spamhaus.org/drop/drop.txt`. |
|
||||
| `SWW_WAF_BLOCKLIST_REFRESH` | `24h` | How long after a list was last tried it is fetched again, `1h` or more. |
|
||||
| `SWW_WAF_BLOCKLIST_ACTION` | `deny` | What a client on a blocklist gets: `deny`, `limit:<percent>` of every limit, or `log`. |
|
||||
| `SWW_WAF_DNSBL_ZONES` | empty | DNS blocklists asked about each new client in the background, such as `dnsbl.dronebl.org`. |
|
||||
| `SWW_WAF_DNSBL_RESOLVER` | unset | The resolver the zones are asked through, an IP address with an optional port; unset, the host's resolver is used. |
|
||||
| `SWW_WAF_ABUSEIPDB_KEY` | unset | The key of your AbuseIPDB account. Set, a client that has committed an offence is checked in the background, and the clients on AbuseIPDB's blacklist are banned. |
|
||||
| `SWW_WAF_ABUSEIPDB_MIN_SCORE` | `75` | The least AbuseIPDB abuse confidence score, from 0 to 100, that is a hit. |
|
||||
| `SWW_WAF_ABUSEIPDB_DAILY_BUDGET` | `900` | The most AbuseIPDB checks made in a day. |
|
||||
| `SWW_WAF_DNSBL_ABUSEIPDB_ACTION` | `limit:25` | What a client listed by a DNS blocklist, or scored a hit by AbuseIPDB, gets: `deny`, `limit:<percent>` or `log`. |
|
||||
| `SWW_WAF_IP_REPUTATION_CACHE_TTL` | `24h` | How long a DNS blocklist verdict or an AbuseIPDB score is used. |
|
||||
| `SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS` | `2000` | How long a DNS blocklist query or an AbuseIPDB check may take, in milliseconds. |
|
||||
| `SWW_WAF_ABUSEIPDB_THRESHOLD` | `90` | The least AbuseIPDB score, from 25 to 100, at which a client on its blacklist is banned. Do not set it below 50 to 75: lower scores list ordinary visitors. |
|
||||
| `SWW_WAF_ABUSEIPDB_BLACKLIST_REFRESH_INTERVAL_MINS` | `60` | How often AbuseIPDB's blacklist is fetched again, in minutes. |
|
||||
| `SWW_WAF_ABUSEIPDB_BAN_DAYS` | `30` | How long a client on AbuseIPDB's blacklist is banned, in days. |
|
||||
| `SWW_WAF_STATE_DIR` | `/var/lib/sww` | The directory of the state files, an absolute path. |
|
||||
| `SWW_WAF_BANS_WRITE_DELAY` | `10s` | How long `bans.json` waits to be written after a ban is made, lifted or made permanent. |
|
||||
| `SWW_WAF_STATE_WRITE_INTERVAL` | `15m` | How often every state and cache file is written. |
|
||||
| `SWW_WAF_CACHE_DIR` | `/var/cache/sww` | The directory of the cache files, an absolute path. |
|
||||
| `SWW_WAF_LOG_LEVEL` | `info` | The least severe of `waf`'s own messages that are written: `debug`, `info`, `warn` or `error`; request lines are always written. |
|
||||
| `SWW_WAF_LOG_REQUEST_HEADERS` | `accept,accept-language,accept-encoding,content-type,origin,range` | The request headers whose values the request log records; `Authorization` and `Cookie` are never recorded. |
|
||||
| `SWW_WAF_LOG_REMOTE_URL` | unset | A syslog server that also gets every line `waf` writes: `syslog+udp://`, `syslog+tcp://` or `syslog+tls://` with a host and a port. |
|
||||
| `SWW_WAF_LOG_REMOTE_TLS_CA_FILE` | unset | A file of PEM certificates a `syslog+tls` server's certificate must chain to, instead of the host's own. |
|
||||
| `SWW_WAF_LOG_REMOTE_BUFFER_LINES` | `10000` | The most lines held while they wait to be sent; past it, the oldest is dropped. |
|
||||
| `SWW_WAF_LOG_REMOTE_FACILITY` | `local0` | The syslog facility of each record sent. |
|
||||
| `SWW_WAF_LOG_REMOTE_APP_NAME` | `SWW_WAF_INSTANCE_NAME` | The app name in each syslog record, 1 to 48 printable ASCII characters without a space. |
|
||||
| `SWW_WAF_ADMIN_TOKEN` | unset | The bearer token for the ban and client endpoints, at least 32 characters; unset, they answer `404`. |
|
||||
| `SWW_WAF_METRICS_TOKEN` | unset | The bearer token for `/_sww/metrics`, at least 32 characters; unset, the metrics are off. |
|
||||
| `SWW_WAF_METRICS_BUSIEST_COUNTRIES_AND_ASNS` | `50` | How many AS numbers and how many countries get metric series of their own; the rest are summed as `other`. |
|
||||
| `SWW_WAF_ALERT_WEBHOOK_URL` | unset | A webhook each alert is posted to as a JSON object. |
|
||||
| `SWW_WAF_ALERT_WEBHOOK_HEADERS` | empty | Headers sent with each post, a list of `Name:value`, such as `Authorization:Bearer <token>`. |
|
||||
| `SWW_WAF_ALERT_SLACK_WEBHOOK_URL` | unset | A Slack incoming webhook each alert is posted to as a message. |
|
||||
| `SWW_WAF_ALERT_NTFY_URL` | unset | An ntfy topic each alert is published to, as its full URL, such as `https://ntfy.sh/my-alerts`. |
|
||||
| `SWW_WAF_ALERT_NTFY_TOKEN` | unset | An ntfy access token sent with each alert. |
|
||||
| `SWW_WAF_ALERT_EVENTS` | `ban,permanent_ban,core_rule_set_match,anomaly,reputation_hit,source_failure,file_error` | The events alerts are sent for; the hourly `summary` is sent whatever it says. |
|
||||
| `SWW_WAF_ALERT_COOLDOWN` | `15m` | How long after an alert its repeats are held back and counted, a duration or `off`. |
|
||||
| `SWW_WAF_ALERT_MAX_PER_HOUR` | `60` | The most alerts sent in an hour, a number or `off`; the rest are counted in the hour's `summary`. |
|
||||
| `SWW_WAF_ANOMALY_CLIENT_REQUESTS_PER_MINUTE` | `off` | Sends one alert to the alert destinations when one client makes more requests than this in a minute, a count. No client is banned, rate limited or blocked. |
|
||||
| `SWW_WAF_ANOMALY_CLIENT_REQUESTS_PER_HOUR` | `off` | Sends one alert to the alert destinations when one client makes more requests than this in an hour, a count. No client is banned, rate limited or blocked. |
|
||||
| `SWW_WAF_ANOMALY_CLIENT_BYTES_PER_MINUTE` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for one client in a minute, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
|
||||
| `SWW_WAF_ANOMALY_CLIENT_BYTES_PER_HOUR` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for one client in an hour, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
|
||||
| `SWW_WAF_ANOMALY_NET_REQUESTS_PER_MINUTE` | `off` | Sends one alert to the alert destinations when the netblock around a client makes more requests than this in a minute, a count. No client is banned, rate limited or blocked. |
|
||||
| `SWW_WAF_ANOMALY_NET_REQUESTS_PER_HOUR` | `off` | Sends one alert to the alert destinations when the netblock around a client makes more requests than this in an hour, a count. No client is banned, rate limited or blocked. |
|
||||
| `SWW_WAF_ANOMALY_NET_BYTES_PER_MINUTE` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for the netblock around a client in a minute, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
|
||||
| `SWW_WAF_ANOMALY_NET_BYTES_PER_HOUR` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for the netblock around a client in an hour, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
|
||||
| `SWW_WAF_ANOMALY_NET_V4_PREFIX` | `24` | The prefix length of the netblock around an IPv4 client that the `SWW_WAF_ANOMALY_NET_` thresholds count, from 0 to 32. Those thresholds only send alerts. |
|
||||
| `SWW_WAF_ANOMALY_NET_V6_PREFIX` | `48` | The prefix length of the netblock around an IPv6 client that the `SWW_WAF_ANOMALY_NET_` thresholds count, from 0 to 128. Those thresholds only send alerts. |
|
||||
| `SWW_WAF_ANOMALY_ASN_REQUESTS_PER_MINUTE` | `off` | Sends one alert to the alert destinations when one AS number makes more requests than this in a minute, a count. No client is banned, rate limited or blocked. |
|
||||
| `SWW_WAF_ANOMALY_ASN_REQUESTS_PER_HOUR` | `off` | Sends one alert to the alert destinations when one AS number makes more requests than this in an hour, a count. No client is banned, rate limited or blocked. |
|
||||
| `SWW_WAF_ANOMALY_ASN_BYTES_PER_MINUTE` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for one AS number in a minute, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
|
||||
| `SWW_WAF_ANOMALY_ASN_BYTES_PER_HOUR` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for one AS number in an hour, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
|
||||
| `SWW_WAF_ANOMALY_TOTAL_REQUESTS_PER_MINUTE` | `off` | Sends one alert to the alert destinations when the whole service gets more requests than this in a minute, a count. No client is banned, rate limited or blocked. |
|
||||
| `SWW_WAF_ANOMALY_TOTAL_REQUESTS_PER_HOUR` | `off` | Sends one alert to the alert destinations when the whole service gets more requests than this in an hour, a count. No client is banned, rate limited or blocked. |
|
||||
| `SWW_WAF_ANOMALY_TOTAL_BYTES_PER_MINUTE` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for the whole service in a minute, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
|
||||
| `SWW_WAF_ANOMALY_TOTAL_BYTES_PER_HOUR` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for the whole service in an hour, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
|
||||
| `SWW_WAF_ANOMALY_WATCHED_NETS` | empty | Named netblocks the `SWW_WAF_ANOMALY_WATCHED_NET_` thresholds count each as a whole, such as `office=203.0.113.0/24,scraper-x=198.51.100.0/22`. Those thresholds only send alerts. |
|
||||
| `SWW_WAF_ANOMALY_WATCHED_NET_REQUESTS_PER_MINUTE` | `off` | Sends one alert to the alert destinations when a watched netblock makes more requests than this in a minute, a count. No client is banned, rate limited or blocked. |
|
||||
| `SWW_WAF_ANOMALY_WATCHED_NET_REQUESTS_PER_HOUR` | `off` | Sends one alert to the alert destinations when a watched netblock makes more requests than this in an hour, a count. No client is banned, rate limited or blocked. |
|
||||
| `SWW_WAF_ANOMALY_WATCHED_NET_BYTES_PER_MINUTE` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for a watched netblock in a minute, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
|
||||
| `SWW_WAF_ANOMALY_WATCHED_NET_BYTES_PER_HOUR` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for a watched netblock in an hour, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
|
||||
|
||||
[`docs/settings.md`](docs/settings.md) gives the values each setting takes in
|
||||
full, and the limits that are fixed rather than settings.
|
||||
@@ -416,8 +424,8 @@ full, and the limits that are fixed rather than settings.
|
||||
Each example is a complete `compose.yaml`. `smallwebwaf` is the tag
|
||||
`make docker` gives the image; on another host, use the name you pushed it
|
||||
under, by digest. Docker Compose puts the services on a private network of its
|
||||
own, where each reaches the others by service name. Each example publishes
|
||||
`rproxy`'s port on the host, for trying it out.
|
||||
own, where each reaches the others by service name. Each example but "On the
|
||||
open internet" publishes `rproxy`'s port on the host, for trying it out.
|
||||
|
||||
### Smallest setup
|
||||
|
||||
@@ -449,6 +457,142 @@ volumes:
|
||||
waf-state:
|
||||
```
|
||||
|
||||
### On the open internet
|
||||
|
||||
One gitea at `git.example.com`, with traefik in front of `rproxy`. traefik gets
|
||||
the certificate from Let's Encrypt and is the only service with published ports.
|
||||
`waf` sets what most sites set. More `waf` settings follow, commented out, each
|
||||
with what it does and when to set it.
|
||||
|
||||
```yaml
|
||||
services:
|
||||
traefik:
|
||||
# traefik v3.7.19, 2026-11-13
|
||||
image: traefik@sha256:a6c0194dda77bcb804824580f36fa7edc56e315b1acfe933366c8b52844a78a4
|
||||
restart: unless-stopped
|
||||
command:
|
||||
- --providers.docker=true
|
||||
- --providers.docker.exposedByDefault=false
|
||||
- --entryPoints.web.address=:80
|
||||
- --entryPoints.websecure.address=:443
|
||||
# Redirect plain HTTP to HTTPS.
|
||||
- --entryPoints.web.http.redirections.entryPoint.to=websecure
|
||||
- --entryPoints.web.http.redirections.entryPoint.scheme=https
|
||||
# A Let's Encrypt certificate for each site, by the TLS-ALPN
|
||||
# challenge on port 443.
|
||||
- --entryPoints.websecure.http.tls.certResolver=le
|
||||
- --certificatesResolvers.le.acme.email=hostmaster@example.com
|
||||
- --certificatesResolvers.le.acme.storage=/letsencrypt/acme.json
|
||||
- --certificatesResolvers.le.acme.tlsChallenge=true
|
||||
# Give a request 10 minutes to arrive, rather than 1, for the
|
||||
# large git pushes the rproxy lines below accept.
|
||||
# - --entryPoints.websecure.transport.respondingTimeouts.readTimeout=600s
|
||||
ports:
|
||||
- "80:80"
|
||||
- "443:443"
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
- acme:/letsencrypt
|
||||
|
||||
app:
|
||||
# Your app: here a gitea, which listens on port 3000.
|
||||
image: registry.example/app@sha256:0000000000000000000000000000000000000000000000000000000000000000
|
||||
restart: unless-stopped
|
||||
|
||||
rproxy:
|
||||
image: smallwebwaf
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
SWW_RPROXY_WAF_BACKEND: http://waf:8080
|
||||
SWW_RPROXY_UPSTREAM_URL: http://app:3000
|
||||
# Accept git pushes up to 1 GiB, rather than 100 MiB, sent in
|
||||
# up to 10 minutes, rather than 1, with traefik's readTimeout
|
||||
# line above.
|
||||
# SWW_RPROXY_MAX_REQUEST_SIZE: 1G
|
||||
# SWW_RPROXY_MAX_REQUEST_TIME_MS: "600000"
|
||||
labels:
|
||||
- traefik.enable=true
|
||||
- traefik.http.routers.git.rule=Host(`git.example.com`)
|
||||
- traefik.http.routers.git.entryPoints=websecure
|
||||
- traefik.http.services.git.loadBalancer.server.port=8080
|
||||
|
||||
waf:
|
||||
image: smallwebwaf
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
SWW_COMMAND: waf
|
||||
# The name in every log line, alert and metric. The default,
|
||||
# the container's host name, is a random id under Compose.
|
||||
SWW_WAF_INSTANCE_NAME: host1/waf
|
||||
# The app each hostname serves. Only that app's fixes to the
|
||||
# Core Rule Set apply there.
|
||||
SWW_WAF_APPS: git.example.com:gitea
|
||||
# Monitoring skips every check.
|
||||
SWW_WAF_ALLOW_NETS: 192.0.2.10
|
||||
# The office and the CI runner, which would exceed the rate
|
||||
# limits, are never rate limited.
|
||||
SWW_WAF_RATE_LIMIT_EXEMPT_NETS: 203.0.113.0/24,198.51.100.7
|
||||
# WordPress paths, which only scanners ask a gitea for. One
|
||||
# request for one bans the client.
|
||||
SWW_WAF_TRAP_PATHS: /wp-login.php,/xmlrpc.php
|
||||
# Spamhaus's list of netblocks hijacked or leased by
|
||||
# criminals. Their clients are refused.
|
||||
SWW_WAF_BLOCKLIST_URLS: https://www.spamhaus.org/drop/drop.txt
|
||||
# The token for the ban endpoints, to lift a ban without
|
||||
# editing bans.json.
|
||||
SWW_WAF_ADMIN_TOKEN_FILE: /etc/sww/tokens/admin
|
||||
# Alerts, such as one for each ban, to an ntfy topic.
|
||||
SWW_WAF_ALERT_NTFY_URL: https://ntfy.example.com/smallwebwaf
|
||||
|
||||
# Refuse nothing and ban no one, for the first days. Each log
|
||||
# line says what enforce mode would have done.
|
||||
# SWW_WAF_MODE: observe
|
||||
# The access token for an ntfy topic that is not public.
|
||||
# SWW_WAF_ALERT_NTFY_TOKEN_FILE: /etc/sww/tokens/ntfy
|
||||
# The token for /_sww/metrics, for Prometheus. Unset, the
|
||||
# metrics are off.
|
||||
# SWW_WAF_METRICS_TOKEN_FILE: /etc/sww/tokens/metrics
|
||||
# 600 requests an hour rather than 1500, to ban slower
|
||||
# scrapers. A fast human reader can reach it too.
|
||||
# SWW_WAF_RATE_LIMIT_PER_HOUR: "600"
|
||||
# A quarter of every limit for clients in two hosting
|
||||
# networks, where scrapers run rather than people.
|
||||
# SWW_WAF_TAG_LIMIT_PERCENT: asn=14061:25,asn=16276:25
|
||||
# For a site only for people in a few countries. Clients
|
||||
# elsewhere are refused, and so are any the lookup cannot place.
|
||||
# SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES: de,at,ch
|
||||
# Close a refused request's connection rather than answer
|
||||
# 403. traefik then answers 502, so a scraper cannot tell a
|
||||
# ban from a broken site.
|
||||
# SWW_WAF_BAN_RESPONSE: close
|
||||
# A copy of every log line to a syslog server.
|
||||
# SWW_WAF_LOG_REMOTE_URL: syslog+tls://logs.example.com:6514
|
||||
volumes:
|
||||
- waf-state:/var/lib/sww
|
||||
- /srv/sww/tokens:/etc/sww/tokens:ro
|
||||
|
||||
volumes:
|
||||
acme:
|
||||
waf-state:
|
||||
```
|
||||
|
||||
Change these before the first start:
|
||||
|
||||
- `git.example.com`, in the router rule and in `SWW_WAF_APPS`;
|
||||
- the email address for Let's Encrypt;
|
||||
- the ntfy topic's URL;
|
||||
- the app's image and port;
|
||||
- the netblocks, which are documentation addresses.
|
||||
|
||||
Make the admin token file first, as for any token file:
|
||||
|
||||
```sh
|
||||
mkdir -p /srv/sww/tokens
|
||||
openssl rand -hex 32 > /srv/sww/tokens/admin
|
||||
chown 65532:65532 /srv/sww/tokens/admin
|
||||
chmod 0400 /srv/sww/tokens/admin
|
||||
```
|
||||
|
||||
### Custom per-address limits
|
||||
|
||||
For a site with no API:
|
||||
@@ -599,6 +743,11 @@ request reaches the app.
|
||||
16. A path under `/_sww/` is answered `404`, so that the app never sees that
|
||||
prefix.
|
||||
|
||||
Steps 12 to 14 only read the request. While the lookup waits for GeoJS, they run
|
||||
at the same time as steps 6 to 11. The first of the two sides to refuse the
|
||||
request ends the checks, and `waf` answers at once. Steps 12 to 14 have 500 ms
|
||||
together.
|
||||
|
||||
`rproxy` takes the rest, with `waf` counting at the end:
|
||||
|
||||
17. The request is passed to the app, streaming.
|
||||
@@ -611,17 +760,20 @@ A refused visitor is answered with `SWW_WAF_BAN_RESPONSE`. A `block` rule or a
|
||||
Core Rule Set match answers `403` and bans nobody, since the Core Rule Set has
|
||||
false positives; such refusals count toward `SWW_WAF_REFUSALS_LIMIT_PER_MINUTE`
|
||||
instead. A request over the header limits of step 3 is answered `414` for a
|
||||
request line over 8 KiB and `431` otherwise. It bans nobody either. `rproxy`
|
||||
itself answers `405` to a method it does not pass on, and the status for a size
|
||||
or time limit, such as `413` for a body too large.
|
||||
request line over 8 KiB and `431` otherwise. It bans nobody either. Nor does a
|
||||
request that steps 12 to 14 have not finished within 500 ms, which is answered
|
||||
`503`. `rproxy` itself answers `405` to a method it does not pass on, and the
|
||||
status for a size or time limit, such as `413` for a body too large.
|
||||
|
||||
### Clients and bans
|
||||
|
||||
- A client is one IPv4 address or one IPv6 netblock of
|
||||
`SWW_WAF_IPV6_GROUP_PREFIX`, since one abuser usually controls a whole /64.
|
||||
- Each limit is counted over a sliding window with a few integers per client.
|
||||
The table of clients and the table of bans each have a bound
|
||||
(`SWW_WAF_MAX_TRACKED_CLIENTS`, `SWW_WAF_MAX_BANS`), so memory is bounded too.
|
||||
The table of clients and the bans for limits and attacks each have a bound
|
||||
(`SWW_WAF_MAX_TRACKED_CLIENTS`, `SWW_WAF_MAX_BANS`). The bans for AbuseIPDB's
|
||||
blacklist are bounded by the clients it listed over the last
|
||||
`SWW_WAF_ABUSEIPDB_BAN_DAYS` days. So memory is bounded too.
|
||||
- A ban refuses every request from its netblock until it ends. Requests refused
|
||||
under a ban count toward no limit. A permanent ban ends only when an admin
|
||||
lifts it.
|
||||
@@ -744,14 +896,16 @@ stops the running process (see [`docs/state-files.md`](docs/state-files.md)).
|
||||
|
||||
- `cmd/sww`: the binary, which only calls `internal/sww`.
|
||||
- `internal/sww`: the process, as the command it is given, and the health check.
|
||||
It is the only package that imports `go.uber.org/fx`, which builds the parts
|
||||
of `waf` and `rproxy`.
|
||||
- `internal/config`: reads the settings, the one place they are read.
|
||||
- `internal/rproxy`: the reverse proxy in front of one app.
|
||||
- `internal/waf`: `waf`'s server, which runs the checks and answers its own
|
||||
endpoints.
|
||||
- `internal/ratelimit`, `internal/bans`, `internal/rules`,
|
||||
`internal/coreruleset`, `internal/lookup`, `internal/reputation`,
|
||||
`internal/routewatch`, `internal/anomaly` and `internal/alerts`: one check or
|
||||
source each.
|
||||
`internal/coreruleset`, `internal/geojs`, `internal/lookup`,
|
||||
`internal/reputation`, `internal/routewatch`, `internal/anomaly` and
|
||||
`internal/alerts`: one check or source each.
|
||||
- `internal/state`, `internal/requestlog`, `internal/remotelog` and
|
||||
`internal/metrics`: the state and cache files, the log lines, the syslog
|
||||
sender and the metrics.
|
||||
|
||||
@@ -69,7 +69,9 @@ never as a time.
|
||||
that command's settings, and runs `internal/rproxy` until `SIGTERM` or
|
||||
`SIGINT` (see "`rproxy`"). `sww healthcheck` is the image's health check.
|
||||
With no command, or one it does not know, the binary prints the commands
|
||||
and exits with status 2.
|
||||
and exits with status 2. `internal/sww` is the only package that imports
|
||||
`go.uber.org/fx`. fx builds the parts of `waf` and `rproxy`, and starts
|
||||
and stops their background work.
|
||||
- `internal/config`: reads the settings, the one place they are read.
|
||||
- `internal/waf`: `waf`'s server. It runs the checks in `check`, and holds
|
||||
`waf`'s question, answer and report. It answers `waf`'s own endpoints in
|
||||
@@ -87,10 +89,13 @@ never as a time.
|
||||
change, and tells which rules a request matches.
|
||||
- `internal/coreruleset`: the Core Rule Set, with the five changes and the
|
||||
apps' fixes listed under "Attack detection", as Coraza directives.
|
||||
- `internal/lookup`: looks clients up through GeoJS or in the lookup
|
||||
database.
|
||||
- `internal/geojs`: looks clients up through GeoJS and keeps the answers. It
|
||||
imports no other package of this repository: `waf` gives it what it needs
|
||||
to count and report failures.
|
||||
- `internal/lookup`: looks clients up in the lookup database.
|
||||
- `internal/reputation`: fetches the lists, asks the DNS blocklists and
|
||||
AbuseIPDB in the background, and keeps their answers.
|
||||
AbuseIPDB in the background, and keeps their answers. It also fetches
|
||||
AbuseIPDB's blacklist, which `internal/waf` bans.
|
||||
- `internal/routewatch`: asks a routewatch instance about clients in the
|
||||
background, and keeps its answers.
|
||||
- `internal/anomaly`: the anomaly counters and their alerts.
|
||||
@@ -171,7 +176,8 @@ field gives that name. `refusal_limit` and `byte_limits` give theirs in
|
||||
`country=` tag.
|
||||
|
||||
Otherwise the request goes on at once. A request that ends before the answer
|
||||
comes is logged without it.
|
||||
comes is logged without it. While the request waits, steps 12 to 14 run at
|
||||
the same time as steps 6 to 11 (see "Answering in time").
|
||||
|
||||
7. The country list (`countries`; see "Country list").
|
||||
8. The reputation sources (see "Reputation"), in this order:
|
||||
@@ -235,13 +241,15 @@ apply.
|
||||
load.
|
||||
- Memory is bounded. When the table of clients holds
|
||||
`SWW_WAF_MAX_TRACKED_CLIENTS`, the least recently seen client is dropped
|
||||
first, with its history. The bans are bounded by `SWW_WAF_MAX_BANS` (see
|
||||
"Bans"). The GeoJS answers, the reputation verdicts and scores, the DNS
|
||||
queries under way and the anomaly counters have fixed bounds, listed under
|
||||
"Fixed limits" in `docs/settings.md`. Past its bound, the GeoJS answer or
|
||||
anomaly counter used longest ago goes first, and the verdict or score fetched
|
||||
longest ago. Past the bound on DNS queries under way, a zone is asked at the
|
||||
client's next request.
|
||||
first, with its history. The bans for broken limits and clear signs of attack
|
||||
are bounded by `SWW_WAF_MAX_BANS` (see "Bans"). Those for AbuseIPDB's
|
||||
blacklist are bounded by the clients it listed over the last
|
||||
`SWW_WAF_ABUSEIPDB_BAN_DAYS` days (see "Reputation"). The GeoJS answers, the
|
||||
reputation verdicts and scores, the DNS queries under way and the anomaly
|
||||
counters have fixed bounds, listed under "Fixed limits" in `docs/settings.md`.
|
||||
Past its bound, the GeoJS answer or anomaly counter used longest ago goes
|
||||
first, and the verdict or score fetched longest ago. Past the bound on DNS
|
||||
queries under way, a zone is asked at the client's next request.
|
||||
|
||||
## Bans
|
||||
|
||||
@@ -296,6 +304,9 @@ to a ban, each by its own rule:
|
||||
Once it ends, only new requests can break a limit again. The client's
|
||||
history keeps its totals.
|
||||
|
||||
A client on AbuseIPDB's blacklist is banned before it sends a request, with the
|
||||
cause `abuseipdb` (see "Reputation").
|
||||
|
||||
A Core Rule Set match refuses only the request it matched, with `403` and
|
||||
without a ban, because the Core Rule Set has false positives. A `block` rule
|
||||
does the same. Those refusals count toward the refusal limit, so a client that
|
||||
@@ -322,11 +333,12 @@ does not count toward a longer ban; it refuses nothing. Deleting its entry from
|
||||
|
||||
A permanent ban does not run out. It ends when an admin lifts it, or when `waf`
|
||||
drops it from a full table, as below. When `waf` holds `SWW_WAF_MAX_BANS` bans
|
||||
of its own, it drops the earliest ban of one netblock: the netblock that has
|
||||
gone longest without a request. A past, active or permanent ban is dropped
|
||||
alike. A scanner whose ban was dropped that way is refused and banned again by
|
||||
its next probe. A ban whose cause is `admin` is never dropped and does not count
|
||||
toward `SWW_WAF_MAX_BANS`, so such bans can never fill the table.
|
||||
for broken limits and clear signs of attack, it drops the earliest such ban of
|
||||
one netblock: the netblock that has gone longest without a request. A past,
|
||||
active or permanent ban is dropped alike. A scanner whose ban was dropped that
|
||||
way is refused and banned again by its next probe. Bans whose cause is `admin`
|
||||
or `abuseipdb` do not count toward `SWW_WAF_MAX_BANS` and are never dropped to
|
||||
make room, so they can never push out the others.
|
||||
|
||||
A request refused under a ban that runs out is logged with the action
|
||||
`temporarily_banned`. Under a permanent ban the action is `permanently_banned`.
|
||||
@@ -507,19 +519,25 @@ service that keeps its visitors' addresses on its own host.
|
||||
dropped from that table. The answer reaches the client's history and ban notes
|
||||
when it comes.
|
||||
- `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` is how long a visitor's request
|
||||
waits for its client's first answer, when a setting needs it (see step 6). It
|
||||
also bounds each request to GeoJS. A client whose answer has not come in time
|
||||
counts as unknown until it comes, and its later requests do not wait. Its
|
||||
address is asked about again in the background.
|
||||
waits for its client's first answer, when a setting needs it (see step 6). A
|
||||
client whose answer has not come in time counts as unknown until it comes, and
|
||||
its later requests do not wait.
|
||||
- A request to GeoJS may take up to 10 seconds, however long visitors wait. It
|
||||
goes on after the visitor's request ends, and its answer is kept. Say GeoJS
|
||||
answers after 5 seconds. The visitor's request is decided after 800
|
||||
milliseconds, without a country. A request from the same client 6 seconds
|
||||
later gets the answer.
|
||||
- Requests from one client that come together share one request to GeoJS.
|
||||
- A client the lookup cannot place has an unknown country and AS number, which
|
||||
the country list and the biased thresholds treat as described there. Nothing
|
||||
else treats it differently.
|
||||
- GeoJS publishes no rate limit, but its terms forbid "an excessive amount of
|
||||
API requests", judged by GeoJS alone, and let it block a caller. While GeoJS
|
||||
is slow, down or refusing `waf`, new clients count as unknown. GeoJS is then
|
||||
left alone for a second, and twice as long after each further failure, up to
|
||||
five minutes. Each failure raises a `source_failure` alert (see "Alerting"). A
|
||||
service that cannot accept this uses the file.
|
||||
API requests", judged by GeoJS alone, and let it block a caller. A request to
|
||||
GeoJS fails when it takes over 10 seconds or GeoJS answers with an error.
|
||||
While GeoJS is slow, down or refusing `waf`, new clients count as unknown.
|
||||
After a failure GeoJS is left alone for a second, and twice as long after each
|
||||
further failure, up to five minutes. Each failure raises a `source_failure`
|
||||
alert (see "Alerting"). A service that cannot accept this uses the file.
|
||||
- The file is the free IPinfo Lite database (`ipinfo_lite.mmdb`), whose `asn`,
|
||||
`as_name` and `country_code` fields are read. The operator downloads it with a
|
||||
free IPinfo account and mounts it read-only; `waf` never fetches it and holds
|
||||
@@ -2027,6 +2045,26 @@ host, sharing one address, would often fetch too close together.
|
||||
- AbuseIPDB answers other than `200`;
|
||||
- the answer gives no score;
|
||||
- no answer comes within `SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS`.
|
||||
- While `SWW_WAF_ABUSEIPDB_KEY` is set, AbuseIPDB's blacklist is fetched at the
|
||||
start and every `SWW_WAF_ABUSEIPDB_BLACKLIST_REFRESH_INTERVAL_MINS` minutes
|
||||
(default `60`). It lists the clients AbuseIPDB scores at
|
||||
`SWW_WAF_ABUSEIPDB_THRESHOLD` (default `90`) or more. The blacklist itself is
|
||||
not kept.
|
||||
- Each client on it is banned for `SWW_WAF_ABUSEIPDB_BAN_DAYS` days (default
|
||||
`30`), with the cause `abuseipdb`. The ban covers the client's netblock,
|
||||
as any ban does.
|
||||
- A netblock that already has an active ban is left as it is. So is one
|
||||
whose `abuseipdb` ban an admin lifted, until that ban would have ended.
|
||||
- An `abuseipdb` ban that has ended is forgotten at the next of these:
|
||||
- a fetch of the blacklist in `enforce` mode;
|
||||
- a read of `bans.json`, at the start or after an admin's edit.
|
||||
- In `observe` mode the blacklist makes no ban.
|
||||
- The threshold takes 25 to 100. It should not be set below 50 to 75, since
|
||||
lower scores list ordinary visitors. AbuseIPDB serves free accounts the
|
||||
blacklist at the threshold 100, whatever is asked for.
|
||||
- AbuseIPDB limits the fetches of the blacklist a day: 5 for a free account.
|
||||
A fetch over the limit fails, and is logged and alerted. The default
|
||||
refresh fetches it 24 times a day.
|
||||
- `SWW_WAF_DNSBL_ABUSEIPDB_ACTION`, for DNS blocklist and AbuseIPDB hits, is
|
||||
`limit:25` by default. Such verdicts are less certain than a blocklist, so a
|
||||
listed client gets a quarter of every request and byte limit rather than a
|
||||
@@ -2041,9 +2079,10 @@ host, sharing one address, would often fetch too close together.
|
||||
- Alerts go to a webhook as JSON, and to Slack and ntfy as formatted messages.
|
||||
`docs/alerts.md` lists the events, the JSON object and the messages.
|
||||
`SWW_WAF_ALERT_EVENTS` chooses which of the seven event types are sent; the
|
||||
hourly `summary` is sent whatever it says. The bans an admin makes raise no
|
||||
alert. In `observe` mode, a request that would have made a ban still raises
|
||||
the ban's alert, with `mode` set to `observe` in its `detail`.
|
||||
hourly `summary` is sent whatever it says. The bans an admin makes, and the
|
||||
bans for AbuseIPDB's blacklist, raise no alert. In `observe` mode, a request
|
||||
that would have made a ban still raises the ban's alert, with `mode` set to
|
||||
`observe` in its `detail`.
|
||||
- A repeat is an alert with the same event, and the same netblock, source, file
|
||||
or anomaly scope, as one sent less than `SWW_WAF_ALERT_COOLDOWN` before. An
|
||||
anomaly scope is one of the things counted under "Anomaly thresholds". A
|
||||
@@ -2191,7 +2230,7 @@ lists the files, what each holds, and their sizes.
|
||||
these stop the process with a message naming the file and position:
|
||||
- a file that does not parse, or has an unknown `version`;
|
||||
- an entry without a field it needs;
|
||||
- a ban whose cause is not one of the four;
|
||||
- a ban whose cause is not `limit`, `attack`, `admin` or `abuseipdb`;
|
||||
- a ban whose `expires` is not after its `start`.
|
||||
|
||||
Starting with empty state would silently forgive every repeat offender.
|
||||
@@ -2371,8 +2410,9 @@ runs no check, writes no request log line and counts nothing in the metrics.
|
||||
|
||||
### The answer
|
||||
|
||||
`waf` runs steps 3 to 16 of "What happens to a request", in that order. Every
|
||||
answer but the last below carries `X-Sww-Disposition`.
|
||||
`waf` runs steps 3 to 16 of "What happens to a request", in that order, apart
|
||||
from the steps that run at the same time (see "Answering in time"). Every answer
|
||||
but the last below carries `X-Sww-Disposition`.
|
||||
|
||||
- `X-Sww-Disposition: allow`, status `200`, no body. `rproxy` passes the request
|
||||
to the app. While `SWW_WAF_ADD_LOOKUP_HEADERS` is `true`, the answer also
|
||||
@@ -2384,6 +2424,8 @@ answer but the last below carries `X-Sww-Disposition`.
|
||||
- `SWW_WAF_BAN_RESPONSE`, for a refusal;
|
||||
- `403`, for a `block` rule or the Core Rule Set;
|
||||
- `414` or `431`, for a request over the header limits;
|
||||
- `503`, for a request that steps 12 to 14 have not finished within 500
|
||||
milliseconds;
|
||||
- `404`, for a path under `/_sww/` that passes the checks, so that the app
|
||||
never sees that prefix.
|
||||
- `X-Sww-Disposition: close`, status `200`, no body. This takes the place of a
|
||||
@@ -2404,15 +2446,22 @@ The Core Rule Set inspects the body a question carries only while
|
||||
|
||||
`rproxy` stops waiting for `waf` after `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, by
|
||||
default 2 seconds, and `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` then
|
||||
decides. The longest a question waits on an outside source is
|
||||
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, by default 500 milliseconds, well
|
||||
within that time.
|
||||
decides. Two things in a decision take time:
|
||||
|
||||
- the wait for GeoJS, at most `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, by
|
||||
default 800 milliseconds;
|
||||
- the trap paths, the rule files and the Core Rule Set, which have 500
|
||||
milliseconds.
|
||||
|
||||
They run at the same time, so `waf` answers within the longer of the two, well
|
||||
within `rproxy`'s wait.
|
||||
|
||||
GeoJS is the only outside source a question can wait on. A question waits only
|
||||
when step 6 says so, and for at most
|
||||
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, by default 500 milliseconds. A
|
||||
client without an answer by then counts as unknown. No question waits on these,
|
||||
which run in the background:
|
||||
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, by default 800 milliseconds. A
|
||||
client without an answer by then counts as unknown. The setting bounds only the
|
||||
question's wait: the request to GeoJS goes on, and its answer is kept for later
|
||||
questions. No question waits on these, which run in the background:
|
||||
|
||||
- the DNS blocklists and AbuseIPDB;
|
||||
- fetching the blocklists, the tag lists and the file
|
||||
@@ -2437,10 +2486,38 @@ never wait. These can still wait for GeoJS:
|
||||
- a country list, which needs the answer;
|
||||
- a rate limit a percentage has lowered, which needs the answer;
|
||||
- a DNS blocklist or AbuseIPDB, while `SWW_WAF_DNSBL_ABUSEIPDB_ACTION` is
|
||||
`deny`;
|
||||
- a trap path;
|
||||
- a rule file;
|
||||
- the Core Rule Set.
|
||||
`deny`.
|
||||
|
||||
The trap paths, the rule files and the Core Rule Set, steps 12 to 14, run during
|
||||
that wait. Once a request has passed the bans, the lookup starts. While it waits
|
||||
for GeoJS, steps 12 to 14 run at the same time as steps 6 to 11. A lookup that
|
||||
needs no wait has its answer at once, and steps 7 to 11 then run before steps 12
|
||||
to 14 start.
|
||||
|
||||
- The first of the two sides to refuse the request ends the checks. `waf`
|
||||
answers at once, and does not wait for the other side. The lookup goes on, and
|
||||
its answer is kept.
|
||||
- What the other side finds leaves no trace. It counts nothing, bans nothing,
|
||||
raises no alert, counts no metric and adds nothing to the log line.
|
||||
- When both sides would refuse, the one that finishes first is answered and
|
||||
logged.
|
||||
- When neither refuses, step 15 runs.
|
||||
|
||||
Steps 12 to 14 only read the request. They have 500 milliseconds together, from
|
||||
their start. The limit is fixed, with no setting. A request they have not
|
||||
finished by then is refused:
|
||||
|
||||
- the answer is `deny`, with `503`;
|
||||
- the log line's action is `checks_timed_out`, and its `heuristic` names the
|
||||
step that had not finished;
|
||||
- the refusal makes no ban, is no offence and does not count toward the refusal
|
||||
limit;
|
||||
- it holds in every `SWW_WAF_CORE_RULE_SET_MODE`;
|
||||
- in `observe` mode the request is answered `allow` with `would_action`, like
|
||||
every refusal.
|
||||
|
||||
The limit is measured on the clock. While `waf`'s host is so busy that an
|
||||
ordinary request takes 500 milliseconds, that request is refused too.
|
||||
|
||||
The header limits are fixed, with no setting. A line's length is in bytes and
|
||||
does not count its line ending. `waf` refuses a request whose request line is
|
||||
@@ -2451,11 +2528,8 @@ It refuses a request with `431` for any of:
|
||||
|
||||
- a header line longer than 8 KiB. Each value of a header sent twice is a line
|
||||
of its own. `Host` counts as a line.
|
||||
- more than one `Cookie` line. A browser sends all its cookies on one line, as
|
||||
RFC 6265 requires. Go's server joins the `Cookie` lines of an HTTP/2 request
|
||||
into one before `rproxy` sees them.
|
||||
- more than 50 cookies. Each part of its `Cookie` line between semicolons counts
|
||||
as one, unless it is empty or only spaces and tabs.
|
||||
- more than 50 cookies. Each part of its `Cookie` headers between semicolons
|
||||
counts as one, unless it is empty or only spaces and tabs.
|
||||
- more than 100 header fields. A header sent twice counts twice. `Host` and
|
||||
`Transfer-Encoding` do not count. `X-Request-ID` and `X-Forwarded-Proto` do.
|
||||
|
||||
@@ -2493,11 +2567,11 @@ body can add up to more than the body. From 1 KiB of JSON, Coraza makes at most
|
||||
|
||||
A request can reach all of these at once. At paranoia level 4, with no rule
|
||||
switched off, the Core Rule Set takes under a second on one core of a current
|
||||
server to inspect it, even with every core busy. Its share of the default 2
|
||||
seconds is 1.3 seconds: the rest is the default 500 milliseconds of waiting for
|
||||
GeoJS and 200 milliseconds for the other checks. So `waf` answers it in time.
|
||||
Enough such requests at once can still overload `waf` past
|
||||
`SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`.
|
||||
server to inspect it. `waf` refuses it if the inspection takes over 500
|
||||
milliseconds. Coraza cannot stop an inspection part-way, so the inspection runs
|
||||
to its end with nobody waiting for it. These limits keep one request from
|
||||
holding a processor for more than about a second. Enough such requests at once
|
||||
can still overload `waf` past `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`.
|
||||
|
||||
### The report
|
||||
|
||||
|
||||
@@ -9,6 +9,10 @@
|
||||
ARG SWW_IMAGE=smallwebwaf
|
||||
FROM ${SWW_IMAGE}
|
||||
|
||||
# Its own title and description, in place of the smallwebwaf image's.
|
||||
LABEL org.opencontainers.image.title="smallwebwaf-example-app" \
|
||||
org.opencontainers.image.description="An example app built on the smallwebwaf image."
|
||||
|
||||
# Packages the app needs, from the nixpkgs in the image.
|
||||
RUN nix-env -iA nixpkgs.busybox
|
||||
|
||||
|
||||
+3
-2
@@ -6,7 +6,7 @@ back or dropped.
|
||||
|
||||
## Events
|
||||
|
||||
- `ban`: a ban `waf` made.
|
||||
- `ban`: a ban `waf` made for a broken limit or a clear sign of attack.
|
||||
- `permanent_ban`: a permanent ban it made, or an existing ban made permanent by
|
||||
a request.
|
||||
- `core_rule_set_match`: the Core Rule Set scored a request at or over the
|
||||
@@ -68,7 +68,8 @@ on one line:
|
||||
"earlier_bans": {
|
||||
"limit": 0,
|
||||
"attack": 0,
|
||||
"admin": 0
|
||||
"admin": 0,
|
||||
"abuseipdb": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
+9
-2
@@ -105,6 +105,8 @@ Every one of these lines has:
|
||||
`blocked_by_core_rule_set`: refused by that check;
|
||||
- `request_line_too_long`: a request line over the header limits;
|
||||
- `request_header_too_large`: headers over the header limits;
|
||||
- `checks_timed_out`: refused with `503` because the trap paths, the rule
|
||||
files and the Core Rule Set had not finished within 500 ms;
|
||||
- `own_endpoint`: answered at one of `waf`'s own endpoints, or `404` for a
|
||||
path under `/_sww/`.
|
||||
- `would_action`: in `observe` mode, what `enforce` mode would have done with a
|
||||
@@ -128,12 +130,17 @@ Every one of these lines has:
|
||||
- `rule_files`: `rule_id` and `rule_action`, of the rule that refused it;
|
||||
- `header_limits`: for a request line, `request_line_bytes` and
|
||||
`request_line_bytes_limit`; for headers, `longest_header_line_bytes`,
|
||||
`header_line_bytes_limit`, `fields`, `fields_limit`, `cookie_lines`,
|
||||
`cookie_lines_limit`, `cookies` and `cookies_limit`;
|
||||
`header_line_bytes_limit`, `fields`, `fields_limit`, `cookies` and
|
||||
`cookies_limit`;
|
||||
- `core_rule_set`: `score`, and `threshold`,
|
||||
`SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD`;
|
||||
- `refusal_limit`: `window`, `count` and `limit`, of the refusal limit;
|
||||
- `byte_limits`: `window`, `count` and `limit`, the count in bytes.
|
||||
|
||||
For a request refused as `checks_timed_out`, `heuristic` is the one of
|
||||
`trap_paths`, `rule_files` and `core_rule_set` that had not finished, and
|
||||
its figure is `time_limit_ms`, 500.
|
||||
|
||||
- `after_answer_heuristic`: the heuristic that banned the client after `waf`
|
||||
answered the request, in `observe` mode too: `refusal_limit` or `byte_limits`.
|
||||
- `after_answer_heuristic_figures`: an object of the figures that heuristic
|
||||
|
||||
+24
-12
@@ -42,8 +42,8 @@ chooses between the two (see "The image's command" below).
|
||||
client's group, from 32 to 128.
|
||||
- `SWW_WAF_MAX_TRACKED_CLIENTS` (default `20000`): the most clients kept, a
|
||||
whole number above zero.
|
||||
- `SWW_WAF_MAX_BANS` (default `5000`): how many of its own bans `waf` keeps, a
|
||||
whole number above zero.
|
||||
- `SWW_WAF_MAX_BANS` (default `5000`): the most bans `waf` keeps for broken
|
||||
limits and attacks, a whole number above zero.
|
||||
|
||||
## Tokens
|
||||
|
||||
@@ -158,9 +158,10 @@ cut off; the client is banned.
|
||||
- an anomaly threshold per AS number.
|
||||
- `SWW_WAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in
|
||||
its `.mmdb` form, for `SWW_WAF_LOOKUP_SOURCE=file`.
|
||||
- `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` (default `500`): how long a
|
||||
request waits for its client's first answer from GeoJS, and how long a request
|
||||
to GeoJS may take. A time in milliseconds; never `off`.
|
||||
- `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` (default `800`): how long a
|
||||
request waits for its client's first answer from GeoJS. A time in
|
||||
milliseconds; never `off`. It does not cut off the request to GeoJS, whose
|
||||
answer is kept when it comes later.
|
||||
- `SWW_WAF_ADD_LOOKUP_HEADERS` (default `true`): `true` has `rproxy` pass the
|
||||
app the client's AS number in `X-Client-ASN` and its country in
|
||||
`X-Client-Country`. The default is `false` while `SWW_WAF_LOOKUP_SOURCE` is
|
||||
@@ -337,6 +338,13 @@ tag that no request can get stops the start. A request can get only these:
|
||||
or AbuseIPDB's score is used. A duration.
|
||||
- `SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS` (default `2000`): how long a query to a
|
||||
zone or a check with AbuseIPDB may take. A time in milliseconds.
|
||||
- `SWW_WAF_ABUSEIPDB_THRESHOLD` (default `90`): the least abuse confidence score
|
||||
of the clients on AbuseIPDB's blacklist, a whole number from 25 to 100. It
|
||||
should not be set below 50 to 75, since lower scores list ordinary visitors.
|
||||
- `SWW_WAF_ABUSEIPDB_BLACKLIST_REFRESH_INTERVAL_MINS` (default `60`): how often
|
||||
the blacklist is fetched again, in minutes. A whole number above zero.
|
||||
- `SWW_WAF_ABUSEIPDB_BAN_DAYS` (default `30`): how long the ban of a client on
|
||||
the blacklist lasts, in days. A whole number above zero.
|
||||
|
||||
## Alerting
|
||||
|
||||
@@ -363,6 +371,11 @@ tag that no request can get stops the start. A request can get only these:
|
||||
|
||||
## Anomaly thresholds
|
||||
|
||||
An anomaly is more requests or bytes than one of these thresholds allows.
|
||||
Crossing a threshold sends one `anomaly` alert to the destinations under
|
||||
"Alerting", and does nothing else: no client is banned, rate limited or blocked.
|
||||
`SWW_WAF_ALERT_COOLDOWN` holds back repeats of that alert.
|
||||
|
||||
Each scope has four thresholds: the most requests and the most body bytes
|
||||
counted in a minute and in an hour. The bytes are those `SWW_WAF_BYTES_COUNTED`
|
||||
names. Requests are counts, bytes are sizes. Each is `off` by default. The four
|
||||
@@ -422,12 +435,9 @@ A few limits are fixed rather than settings:
|
||||
- at most 20,000 anomaly counters are kept;
|
||||
- each alert destination's queue holds at most 1,000 alerts;
|
||||
- `waf` holds at most 10,000 answered requests awaiting their reports;
|
||||
- `waf` refuses a request with any of these (see "Answering in time" under
|
||||
"`waf`" in [`SPEC.md`](../SPEC.md)):
|
||||
- a request line or header line over 8 KiB;
|
||||
- more than one `Cookie` line;
|
||||
- more than 50 cookies;
|
||||
- more than 100 header fields;
|
||||
- `waf` refuses a request with a request line or header line over 8 KiB, more
|
||||
than 50 cookies or more than 100 header fields (see "Answering in time" under
|
||||
"`waf`" in [`SPEC.md`](../SPEC.md));
|
||||
- `rproxy` takes a request line and headers of at most 32 KiB, and closes a
|
||||
kept-open connection idle for 120 seconds;
|
||||
- `rproxy` reads at most 1 MiB of the body of `waf`'s answer, and answers `502`
|
||||
@@ -437,7 +447,9 @@ A few limits are fixed rather than settings:
|
||||
|
||||
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` is the longest a question waits on
|
||||
an outside source. Keep it well under `rproxy`'s
|
||||
`SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, as the default is. No question waits for
|
||||
`SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, as the default is. The trap paths, the rule
|
||||
files and the Core Rule Set run during that wait and have 500 ms, so `waf`
|
||||
answers within the longer of the two times. No question waits for
|
||||
`SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS`, since the DNS blocklists and AbuseIPDB are
|
||||
asked in the background (see "Answering in time" under "`waf`" in
|
||||
[`SPEC.md`](../SPEC.md)).
|
||||
|
||||
+19
-5
@@ -21,11 +21,22 @@ top-level `version` of 1.
|
||||
|
||||
## `bans.json`
|
||||
|
||||
Active and past bans: every ban an admin made, and up to `SWW_WAF_MAX_BANS` of
|
||||
those `waf` made. Per entry:
|
||||
Active and past bans:
|
||||
|
||||
- every ban an admin made;
|
||||
- up to `SWW_WAF_MAX_BANS` of those `waf` made for a broken limit or an attack;
|
||||
- the bans for AbuseIPDB's blacklist.
|
||||
|
||||
A ban for AbuseIPDB's blacklist that has ended is forgotten at the next of
|
||||
these:
|
||||
|
||||
- a fetch of the blacklist in `enforce` mode;
|
||||
- a read of `bans.json`, at the start or after an admin's edit.
|
||||
|
||||
Per entry:
|
||||
|
||||
- `netblock`, `start` and `expires` (`null` for a permanent ban);
|
||||
- `cause`: `attack`, `limit` or `admin`;
|
||||
- `cause`: `attack`, `limit`, `admin` or `abuseipdb`;
|
||||
- `reason`: a short reason;
|
||||
- `lifted`: when an admin lifted it, if one did;
|
||||
- `notes`: the notes below.
|
||||
@@ -60,7 +71,8 @@ ban. The notes:
|
||||
- for a clear sign of attack, `rule_id` and `target`, or `trap_path`;
|
||||
- `reputation`: the sources the client was on, each with its `source` and, for
|
||||
AbuseIPDB, its `score`;
|
||||
- `request`: the request that caused the ban, each text cut to 256 bytes:
|
||||
- `request`: the request that caused the ban, each text cut to 256 bytes, and
|
||||
empty for a ban for AbuseIPDB's blacklist:
|
||||
- `time`, `method` and `host`;
|
||||
- `path`, with its query string;
|
||||
- `status` and `user_agent`;
|
||||
@@ -151,7 +163,9 @@ A cache file. routewatch's answers from the last hour, one per line:
|
||||
- `bans.json`: about 1.2 KiB per ban, at most about 2.5 KiB, so about 6 MiB at
|
||||
the default `SWW_WAF_MAX_BANS`. Written every 15 minutes, a full file comes to
|
||||
about 600 MiB of writes a day, and a hundred new bans a day add about 600 MiB
|
||||
more.
|
||||
more. A ban for AbuseIPDB's blacklist takes about 1 KiB. A free account's
|
||||
blacklist lists 10,000 clients, so it adds at least 10 MiB, and 1 GiB of
|
||||
writes a day.
|
||||
- `lookups.json`: about 150 bytes per answer, under 1.5 GiB of writes a day when
|
||||
full.
|
||||
- `routewatch.json`: about 120 bytes per answer, at most about 12 MiB.
|
||||
|
||||
@@ -12,6 +12,7 @@ require (
|
||||
github.com/oschwald/maxminddb-golang/v2 v2.7.0
|
||||
github.com/prometheus/client_golang v1.24.1
|
||||
github.com/spf13/viper v1.21.0
|
||||
go.uber.org/fx v1.24.0
|
||||
golang.org/x/sys v0.48.0
|
||||
)
|
||||
|
||||
@@ -44,6 +45,9 @@ require (
|
||||
github.com/tidwall/match v1.1.1 // indirect
|
||||
github.com/tidwall/pretty v1.2.1 // indirect
|
||||
github.com/valllabh/ocsf-schema-golang v1.0.3 // indirect
|
||||
go.uber.org/dig v1.19.0 // indirect
|
||||
go.uber.org/multierr v1.10.0 // indirect
|
||||
go.uber.org/zap v1.26.0 // indirect
|
||||
go.yaml.in/yaml/v3 v3.0.5 // indirect
|
||||
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba // indirect
|
||||
golang.org/x/net v0.60.0 // indirect
|
||||
|
||||
@@ -97,8 +97,16 @@ github.com/tidwall/pretty v1.2.1 h1:qjsOFOWWQl+N3RsoF5/ssm1pHmJJwhjlSbZ51I6wMl4=
|
||||
github.com/tidwall/pretty v1.2.1/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
|
||||
github.com/valllabh/ocsf-schema-golang v1.0.3 h1:eR8k/3jP/OOqB8LRCtdJ4U+vlgd/gk5y3KMXoodrsrw=
|
||||
github.com/valllabh/ocsf-schema-golang v1.0.3/go.mod h1:sZ3as9xqm1SSK5feFWIR2CuGeGRhsM7TR1MbpBctzPk=
|
||||
go.uber.org/dig v1.19.0 h1:BACLhebsYdpQ7IROQ1AGPjrXcP5dF80U3gKoFzbaq/4=
|
||||
go.uber.org/dig v1.19.0/go.mod h1:Us0rSJiThwCv2GteUN0Q7OKvU7n5J4dxZ9JKUXozFdE=
|
||||
go.uber.org/fx v1.24.0 h1:wE8mruvpg2kiiL1Vqd0CC+tr0/24XIB10Iwp2lLWzkg=
|
||||
go.uber.org/fx v1.24.0/go.mod h1:AmDeGyS+ZARGKM4tlH4FY2Jr63VjbEDJHtqXTGP5hbo=
|
||||
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
|
||||
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
|
||||
go.uber.org/multierr v1.10.0 h1:S0h4aNzvfcFsC3dRF1jLoaov7oRaKqRGC/pUEJ2yvPQ=
|
||||
go.uber.org/multierr v1.10.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
|
||||
go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
|
||||
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
|
||||
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
|
||||
go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
|
||||
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
|
||||
|
||||
+55
-42
@@ -34,65 +34,76 @@ import (
|
||||
"time"
|
||||
)
|
||||
|
||||
// Event is what an alert is for, such as EventBan.
|
||||
type Event string
|
||||
|
||||
// The events an alert is for, as SWW_WAF_ALERT_EVENTS names them.
|
||||
const (
|
||||
// EventBan is a ban sww made.
|
||||
EventBan = "ban"
|
||||
EventBan Event = "ban"
|
||||
// EventPermanentBan is a permanent ban sww made, or a ban it
|
||||
// made permanent.
|
||||
EventPermanentBan = "permanent_ban"
|
||||
EventPermanentBan Event = "permanent_ban"
|
||||
// EventAnomaly is a count of requests or bytes over an anomaly
|
||||
// threshold.
|
||||
EventAnomaly = "anomaly"
|
||||
EventAnomaly Event = "anomaly"
|
||||
// EventCoreRuleSetMatch is a request the Core Rule Set scored at or over
|
||||
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD, refused in block mode, let through in
|
||||
// detect mode.
|
||||
EventCoreRuleSetMatch = "core_rule_set_match"
|
||||
EventCoreRuleSetMatch Event = "core_rule_set_match"
|
||||
// EventReputationHit is a request whose client a blocklist or a DNSBL
|
||||
// zone lists, or whose AbuseIPDB score is a hit.
|
||||
EventReputationHit = "reputation_hit"
|
||||
EventReputationHit Event = "reputation_hit"
|
||||
// EventSourceFailure is GeoJS failing or refusing sww, a fetch
|
||||
// of a list failing, a query to a DNSBL zone or a check with AbuseIPDB
|
||||
// failing or refused, or the day's AbuseIPDB checks used up.
|
||||
EventSourceFailure = "source_failure"
|
||||
EventSourceFailure Event = "source_failure"
|
||||
// EventFileError is one of:
|
||||
// - a rule file or state file edited while sww runs that does not parse;
|
||||
// - a cache file that does not parse, at start or after an edit;
|
||||
// - a state file or cache file that cannot be written;
|
||||
// - a replacement of the lookup database that cannot be read.
|
||||
EventFileError = "file_error"
|
||||
EventFileError Event = "file_error"
|
||||
// EventSummary is the summary sent as an hour ends: of the alerts held
|
||||
// back in it past SWW_WAF_ALERT_MAX_PER_HOUR, and of the repeats held
|
||||
// back by the cooldowns dropped as it ends, which no alert let through
|
||||
// has given. It is sent with SWW_WAF_ALERT_MAX_PER_HOUR off too, for
|
||||
// those repeats. SWW_WAF_ALERT_EVENTS does not name it.
|
||||
EventSummary = "summary"
|
||||
EventSummary Event = "summary"
|
||||
)
|
||||
|
||||
// Events returns every event SWW_WAF_ALERT_EVENTS can name, which is its
|
||||
// default.
|
||||
func Events() []string {
|
||||
return []string{
|
||||
func Events() []Event {
|
||||
return []Event{
|
||||
EventBan, EventPermanentBan, EventCoreRuleSetMatch, EventAnomaly,
|
||||
EventReputationHit, EventSourceFailure, EventFileError,
|
||||
}
|
||||
}
|
||||
|
||||
// Destination is where alerts are sent: DestinationWebhook,
|
||||
// DestinationSlack or DestinationNtfy.
|
||||
type Destination string
|
||||
|
||||
// The destinations alerts are sent to, as the metrics and alerts.json
|
||||
// name them.
|
||||
const (
|
||||
// DestinationWebhook is the webhook SWW_WAF_ALERT_WEBHOOK_URL names.
|
||||
DestinationWebhook = "webhook"
|
||||
DestinationWebhook Destination = "webhook"
|
||||
// DestinationSlack is the Slack incoming webhook
|
||||
// SWW_WAF_ALERT_SLACK_WEBHOOK_URL names.
|
||||
DestinationSlack = "slack"
|
||||
DestinationSlack Destination = "slack"
|
||||
// DestinationNtfy is the ntfy topic SWW_WAF_ALERT_NTFY_URL names.
|
||||
DestinationNtfy = "ntfy"
|
||||
DestinationNtfy Destination = "ntfy"
|
||||
)
|
||||
|
||||
// Destinations returns every destination alerts can be sent to.
|
||||
func Destinations() []string {
|
||||
return []string{DestinationWebhook, DestinationSlack, DestinationNtfy}
|
||||
func Destinations() []Destination {
|
||||
return []Destination{
|
||||
DestinationWebhook,
|
||||
DestinationSlack,
|
||||
DestinationNtfy,
|
||||
}
|
||||
}
|
||||
|
||||
const (
|
||||
@@ -135,7 +146,7 @@ type Params struct {
|
||||
NtfyURL *url.URL
|
||||
NtfyToken string
|
||||
// Events are the events alerts are sent for (SWW_WAF_ALERT_EVENTS).
|
||||
Events []string
|
||||
Events []Event
|
||||
// Cooldown is how long a repeat of an alert is held back
|
||||
// (SWW_WAF_ALERT_COOLDOWN), 0 for no time. MaxPerHour is the most alerts
|
||||
// sent in an hour (SWW_WAF_ALERT_MAX_PER_HOUR), 0 for no limit.
|
||||
@@ -159,7 +170,7 @@ type Params struct {
|
||||
type Alert struct {
|
||||
Instance string `json:"instance"`
|
||||
Time time.Time `json:"time"`
|
||||
Event string `json:"event"`
|
||||
Event Event `json:"event"`
|
||||
Host string `json:"host,omitempty"`
|
||||
Client netip.Addr `json:"client"`
|
||||
Netblock netip.Prefix `json:"netblock"`
|
||||
@@ -186,7 +197,7 @@ type Alert struct {
|
||||
//
|
||||
//nolint:tagliatelle // the state files use snake_case, as the request log does
|
||||
type Cooldown struct {
|
||||
Event string `json:"event"`
|
||||
Event Event `json:"event"`
|
||||
Netblock netip.Prefix `json:"netblock"`
|
||||
File string `json:"file,omitempty"`
|
||||
Source string `json:"source,omitempty"`
|
||||
@@ -203,18 +214,18 @@ type Cooldown struct {
|
||||
//
|
||||
//nolint:tagliatelle // the state files use snake_case, as the request log does
|
||||
type Hour struct {
|
||||
Start time.Time `json:"start"`
|
||||
Sent int `json:"sent"`
|
||||
HeldBack map[string]int `json:"held_back"`
|
||||
Start time.Time `json:"start"`
|
||||
Sent int `json:"sent"`
|
||||
HeldBack map[Event]int `json:"held_back"`
|
||||
}
|
||||
|
||||
// State is what alerts.json holds: the cooldowns, the hour under way, and
|
||||
// for each destination set, the alerts waiting to be sent to it, oldest
|
||||
// first.
|
||||
type State struct {
|
||||
Cooldowns []Cooldown `json:"cooldowns"`
|
||||
Hour Hour `json:"hour"`
|
||||
Waiting map[string][]Alert `json:"waiting"`
|
||||
Cooldowns []Cooldown `json:"cooldowns"`
|
||||
Hour Hour `json:"hour"`
|
||||
Waiting map[Destination][]Alert `json:"waiting"`
|
||||
}
|
||||
|
||||
// Counts are, for a destination, how many alerts it took, how many
|
||||
@@ -248,7 +259,7 @@ type destination struct {
|
||||
// name is how the metrics and alerts.json name the destination, and
|
||||
// setting the setting that is its URL, which the log names in place
|
||||
// of the URL.
|
||||
name string
|
||||
name Destination
|
||||
setting string
|
||||
url *url.URL
|
||||
// message returns the body an alert is posted with, and the headers
|
||||
@@ -273,7 +284,7 @@ type destination struct {
|
||||
// scope with the same AS number or name, as its detail names them. Each
|
||||
// is empty for an alert without one.
|
||||
type cooldownKey struct {
|
||||
event string
|
||||
event Event
|
||||
netblock netip.Prefix
|
||||
file string
|
||||
source string
|
||||
@@ -306,7 +317,7 @@ func New(params Params) *Queue {
|
||||
q := &Queue{
|
||||
params: params,
|
||||
cooldowns: map[cooldownKey]*Cooldown{},
|
||||
hour: Hour{HeldBack: map[string]int{}},
|
||||
hour: Hour{HeldBack: map[Event]int{}},
|
||||
}
|
||||
|
||||
if params.WebhookURL != nil {
|
||||
@@ -377,7 +388,7 @@ func (q *Queue) Raise(alert Alert) {
|
||||
// event, no alert for event on netblock was let through less than
|
||||
// Cooldown before, and fewer than MaxPerHour alerts have been let through
|
||||
// in the hour under way. Unlike Raise, it counts nothing.
|
||||
func (q *Queue) WouldSend(event string, netblock netip.Prefix) bool {
|
||||
func (q *Queue) WouldSend(event Event, netblock netip.Prefix) bool {
|
||||
if len(q.destinations) == 0 || !slices.Contains(q.params.Events, event) {
|
||||
return false
|
||||
}
|
||||
@@ -433,7 +444,7 @@ func (q *Queue) Run(ctx context.Context) {
|
||||
|
||||
// Counts returns the counts of the destination name, all 0 for one not
|
||||
// set.
|
||||
func (q *Queue) Counts(name string) Counts {
|
||||
func (q *Queue) Counts(name Destination) Counts {
|
||||
for _, d := range q.destinations {
|
||||
if d.name == name {
|
||||
return Counts{
|
||||
@@ -449,8 +460,8 @@ func (q *Queue) Counts(name string) Counts {
|
||||
|
||||
// DestinationsSet returns the destinations set, in the order of
|
||||
// Destinations.
|
||||
func (q *Queue) DestinationsSet() []string {
|
||||
names := make([]string, 0, len(q.destinations))
|
||||
func (q *Queue) DestinationsSet() []Destination {
|
||||
names := make([]Destination, 0, len(q.destinations))
|
||||
for _, d := range q.destinations {
|
||||
names = append(names, d.name)
|
||||
}
|
||||
@@ -474,7 +485,7 @@ func (q *Queue) Snapshot() State {
|
||||
state := State{
|
||||
Cooldowns: make([]Cooldown, 0, len(q.cooldowns)),
|
||||
Hour: q.hour,
|
||||
Waiting: map[string][]Alert{},
|
||||
Waiting: map[Destination][]Alert{},
|
||||
}
|
||||
state.Hour.HeldBack = maps.Clone(q.hour.HeldBack)
|
||||
|
||||
@@ -525,7 +536,7 @@ func (q *Queue) Load(state State) {
|
||||
q.hour.HeldBack = maps.Clone(state.Hour.HeldBack)
|
||||
|
||||
if q.hour.HeldBack == nil {
|
||||
q.hour.HeldBack = map[string]int{}
|
||||
q.hour.HeldBack = map[Event]int{}
|
||||
}
|
||||
|
||||
for _, d := range q.destinations {
|
||||
@@ -537,7 +548,7 @@ func (q *Queue) Load(state State) {
|
||||
// its URL, that URL, target, and message, which makes the messages sent
|
||||
// to it.
|
||||
func (q *Queue) addDestination(
|
||||
name, setting string, target *url.URL,
|
||||
name Destination, setting string, target *url.URL,
|
||||
message func(alert *Alert) ([]byte, http.Header, error),
|
||||
) {
|
||||
q.destinations = append(q.destinations, &destination{
|
||||
@@ -649,7 +660,7 @@ func (q *Queue) endHour(now time.Time) {
|
||||
})
|
||||
}
|
||||
|
||||
q.hour = Hour{Start: start, HeldBack: map[string]int{}}
|
||||
q.hour = Hour{Start: start, HeldBack: map[Event]int{}}
|
||||
}
|
||||
|
||||
// queue adds alert to the alerts waiting for each destination.
|
||||
@@ -904,20 +915,21 @@ func (q *Queue) ntfyMessage(alert *Alert) ([]byte, http.Header, error) {
|
||||
|
||||
// ntfyPriority returns the priority an alert for event is published to
|
||||
// ntfy with: high for an event the admin needs to look at.
|
||||
func ntfyPriority(event string) string {
|
||||
func ntfyPriority(event Event) string {
|
||||
switch event {
|
||||
case EventPermanentBan, EventAnomaly, EventSourceFailure, EventFileError:
|
||||
return "high"
|
||||
case EventReputationHit:
|
||||
return "low"
|
||||
default: // ban, core_rule_set_match and summary
|
||||
return "default"
|
||||
case EventBan, EventCoreRuleSetMatch, EventSummary:
|
||||
}
|
||||
|
||||
return "default"
|
||||
}
|
||||
|
||||
// ntfyTag returns the tag an alert for event is published to ntfy with,
|
||||
// which ntfy shows as an emoji.
|
||||
func ntfyTag(event string) string {
|
||||
func ntfyTag(event Event) string {
|
||||
switch event {
|
||||
case EventBan, EventPermanentBan:
|
||||
return "no_entry"
|
||||
@@ -929,15 +941,16 @@ func ntfyTag(event string) string {
|
||||
return "label"
|
||||
case EventSourceFailure, EventFileError:
|
||||
return "warning"
|
||||
default: // summary
|
||||
return "bar_chart"
|
||||
case EventSummary:
|
||||
}
|
||||
|
||||
return "bar_chart"
|
||||
}
|
||||
|
||||
// title returns the title of alert in Slack and ntfy: the instance and
|
||||
// the event.
|
||||
func title(alert *Alert) string {
|
||||
return alert.Instance + ": " + alert.Event
|
||||
return alert.Instance + ": " + string(alert.Event)
|
||||
}
|
||||
|
||||
// text returns the text of alert in Slack and ntfy: its reason, then a
|
||||
|
||||
@@ -126,7 +126,7 @@ func TestOnlyTheChosenEventsAreSent(t *testing.T) {
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
params := newParams()
|
||||
params.Events = []string{
|
||||
params.Events = []alerts.Event{
|
||||
alerts.EventSourceFailure,
|
||||
alerts.EventFileError,
|
||||
}
|
||||
@@ -156,7 +156,7 @@ func TestWouldSendOnlyForTheChosenEvents(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
params := newParams()
|
||||
params.Events = []string{
|
||||
params.Events = []alerts.Event{
|
||||
alerts.EventSourceFailure,
|
||||
alerts.EventFileError,
|
||||
}
|
||||
@@ -186,8 +186,8 @@ func TestRaiseDoesNothingWithoutADestination(t *testing.T) {
|
||||
// No alert waits, no cooldown has started, and the hour counts none.
|
||||
want := alerts.State{
|
||||
Cooldowns: []alerts.Cooldown{},
|
||||
Hour: alerts.Hour{HeldBack: map[string]int{}},
|
||||
Waiting: map[string][]alerts.Alert{},
|
||||
Hour: alerts.Hour{HeldBack: map[alerts.Event]int{}},
|
||||
Waiting: map[alerts.Destination][]alerts.Alert{},
|
||||
}
|
||||
if got := q.Snapshot(); !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("state %+v, want %+v", got, want)
|
||||
@@ -235,7 +235,7 @@ func TestRepeatWithinTheCooldownIsHeldBackAndCountedInTheNext(t *testing.T) {
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
webhook, q := start(t, newParams())
|
||||
raise := func(event string, n int) {
|
||||
raise := func(event alerts.Event, n int) {
|
||||
q.Raise(alerts.Alert{Event: event, Netblock: netblock(n)})
|
||||
}
|
||||
|
||||
@@ -1014,7 +1014,7 @@ func TestNtfyTokenIsSentToNtfyAlone(t *testing.T) {
|
||||
)
|
||||
synctest.Wait()
|
||||
|
||||
for destination, want := range map[string][]string{
|
||||
for destination, want := range map[alerts.Destination][]string{
|
||||
alerts.DestinationWebhook: nil,
|
||||
alerts.DestinationSlack: nil,
|
||||
alerts.DestinationNtfy: tc.want,
|
||||
@@ -1063,7 +1063,7 @@ func TestDestinationThatDoesNotAnswerHoldsUpNeitherOther(t *testing.T) {
|
||||
|
||||
// With no time passed, the webhook and ntfy have taken every alert,
|
||||
// while Slack has not answered the first.
|
||||
for destination, want := range map[string]alerts.Counts{
|
||||
for destination, want := range map[alerts.Destination]alerts.Counts{
|
||||
alerts.DestinationWebhook: {Sent: 3},
|
||||
alerts.DestinationSlack: {},
|
||||
alerts.DestinationNtfy: {Sent: 3},
|
||||
@@ -1126,7 +1126,7 @@ func TestDropsAreCountedForEachDestination(t *testing.T) {
|
||||
// the webhook takes every one.
|
||||
const all = alerts.QueueSize + 1
|
||||
|
||||
for destination, want := range map[string]alerts.Counts{
|
||||
for destination, want := range map[alerts.Destination]alerts.Counts{
|
||||
alerts.DestinationWebhook: {Sent: all},
|
||||
alerts.DestinationSlack: {Dropped: 1},
|
||||
alerts.DestinationNtfy: {SendFailures: all, Dropped: all},
|
||||
@@ -1152,11 +1152,19 @@ func TestEachDestinationIsSentOnlyTheAlertsWaitingForIt(t *testing.T) {
|
||||
|
||||
// As read from alerts.json. The alert waiting for the webhook, which
|
||||
// is not set, is dropped.
|
||||
q.Load(roundTrip(t, alerts.State{Waiting: map[string][]alerts.Alert{
|
||||
alerts.DestinationWebhook: {waiting("first")},
|
||||
alerts.DestinationSlack: {waiting("second"), waiting("third")},
|
||||
alerts.DestinationNtfy: {waiting("fourth")},
|
||||
}}))
|
||||
q.Load(
|
||||
roundTrip(
|
||||
t,
|
||||
alerts.State{Waiting: map[alerts.Destination][]alerts.Alert{
|
||||
alerts.DestinationWebhook: {waiting("first")},
|
||||
alerts.DestinationSlack: {
|
||||
waiting("second"),
|
||||
waiting("third"),
|
||||
},
|
||||
alerts.DestinationNtfy: {waiting("fourth")},
|
||||
}},
|
||||
),
|
||||
)
|
||||
synctest.Wait()
|
||||
|
||||
wantBodies(t, standIns[alerts.DestinationSlack],
|
||||
@@ -1165,7 +1173,7 @@ func TestEachDestinationIsSentOnlyTheAlertsWaitingForIt(t *testing.T) {
|
||||
wantBodies(t, standIns[alerts.DestinationNtfy], "fourth")
|
||||
|
||||
// alerts.json then lists Slack and ntfy alone, with no alert waiting.
|
||||
want := map[string][]alerts.Alert{
|
||||
want := map[alerts.Destination][]alerts.Alert{
|
||||
alerts.DestinationSlack: {}, alerts.DestinationNtfy: {},
|
||||
}
|
||||
if got := q.Snapshot().Waiting; !reflect.DeepEqual(got, want) {
|
||||
@@ -1334,11 +1342,11 @@ func start(t *testing.T, params alerts.Params) (*standIn, *alerts.Queue) {
|
||||
func startAll(
|
||||
t *testing.T,
|
||||
params alerts.Params,
|
||||
) (map[string]*standIn, *alerts.Queue) {
|
||||
) (map[alerts.Destination]*standIn, *alerts.Queue) {
|
||||
t.Helper()
|
||||
|
||||
q := alerts.New(params)
|
||||
standIns := map[string]*standIn{}
|
||||
standIns := map[alerts.Destination]*standIn{}
|
||||
|
||||
for _, destination := range q.DestinationsSet() {
|
||||
standIns[destination] = &standIn{answers: answering}
|
||||
@@ -1404,14 +1412,14 @@ func wantAlert(t *testing.T, got, want map[string]any) {
|
||||
|
||||
// wantEvents checks the events of the alerts the webhook was sent, in
|
||||
// order.
|
||||
func wantEvents(t *testing.T, webhook *standIn, want ...string) {
|
||||
func wantEvents(t *testing.T, webhook *standIn, want ...alerts.Event) {
|
||||
t.Helper()
|
||||
|
||||
got := make([]string, 0, len(webhook.received()))
|
||||
got := make([]alerts.Event, 0, len(webhook.received()))
|
||||
|
||||
for _, request := range webhook.received() {
|
||||
event, _ := request.alert["event"].(string)
|
||||
got = append(got, event)
|
||||
got = append(got, alerts.Event(event))
|
||||
}
|
||||
|
||||
if !slices.Equal(got, want) {
|
||||
@@ -1447,7 +1455,8 @@ func wantCounts(
|
||||
|
||||
// wantDestinationCounts checks q's counts for destination.
|
||||
func wantDestinationCounts(
|
||||
t *testing.T, q *alerts.Queue, destination string, want alerts.Counts,
|
||||
t *testing.T, q *alerts.Queue, destination alerts.Destination,
|
||||
want alerts.Counts,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@ const QueueSize = queueSize
|
||||
|
||||
// SetTransport has q's requests to the destination name go through
|
||||
// transport instead of the network.
|
||||
func (q *Queue) SetTransport(name string, transport http.RoundTripper) {
|
||||
func (q *Queue) SetTransport(name Destination, transport http.RoundTripper) {
|
||||
for _, d := range q.destinations {
|
||||
if d.name == name {
|
||||
d.httpClient.Transport = transport
|
||||
|
||||
@@ -295,9 +295,10 @@ func (c *Counters) netAround(client netip.Addr) netip.Prefix {
|
||||
// overThreshold is a count over its threshold: what it counts, requests or
|
||||
// bytes, its window, the count and the threshold.
|
||||
type overThreshold struct {
|
||||
kind, window string
|
||||
count float64
|
||||
threshold int64
|
||||
kind ratelimit.LimitKind
|
||||
window string
|
||||
count float64
|
||||
threshold int64
|
||||
}
|
||||
|
||||
// add counts a request of bytes at now in each of c's counts whose
|
||||
@@ -346,9 +347,10 @@ func (c *Counter) add(
|
||||
// bucketCount is one of a counter's four counts: requests or bytes, in a
|
||||
// window of length, and the buckets they are counted in.
|
||||
type bucketCount struct {
|
||||
kind, window string
|
||||
length time.Duration
|
||||
buckets *ratelimit.Buckets
|
||||
kind ratelimit.LimitKind
|
||||
window string
|
||||
length time.Duration
|
||||
buckets *ratelimit.Buckets
|
||||
}
|
||||
|
||||
// counts returns c's counts: requests and bytes in the minute, then in
|
||||
@@ -404,8 +406,8 @@ func (k key) counter() *Counter {
|
||||
// counted, the count and the threshold.
|
||||
func alertFor(r Request, k key, o overThreshold) alerts.Alert {
|
||||
detail := map[string]any{
|
||||
"scope": k.scope, "window": o.window, "kind": o.kind, "count": o.count,
|
||||
"threshold": o.threshold,
|
||||
"scope": k.scope, "window": o.window, "kind": string(o.kind),
|
||||
"count": o.count, "threshold": o.threshold,
|
||||
}
|
||||
|
||||
var counted string
|
||||
|
||||
@@ -0,0 +1,165 @@
|
||||
package bans_test
|
||||
|
||||
import (
|
||||
"net/netip"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
)
|
||||
|
||||
func TestBlacklistedNetblockIsBannedUntilTheGivenEnd(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
now := midnight()
|
||||
expires := now.Add(30 * day)
|
||||
|
||||
made := ledger.BanForAbuseIPDB(
|
||||
now, expires, map[netip.Prefix]int64{netblock: 95},
|
||||
)
|
||||
if made != 1 || ledger.Made(bans.CauseAbuseIPDB) != 1 {
|
||||
t.Fatalf("made %d bans, %d counted, want 1 and 1",
|
||||
made, ledger.Made(bans.CauseAbuseIPDB))
|
||||
}
|
||||
|
||||
ban, banned, madePermanent := ledger.Check(netblock.Addr(), now)
|
||||
if !banned || madePermanent || !ban.Expires.Equal(expires) ||
|
||||
ban.Cause != bans.CauseAbuseIPDB ||
|
||||
ban.Reason != "on AbuseIPDB's blacklist with a score of 95" ||
|
||||
len(ban.Notes.Reputation) != 1 ||
|
||||
ban.Notes.Reputation[0].Source != "abuseipdb" ||
|
||||
*ban.Notes.Reputation[0].Score != 95 {
|
||||
t.Errorf(
|
||||
"ban %+v, banned %t, made permanent %t, want the blacklist's "+
|
||||
"ban until %s",
|
||||
ban,
|
||||
banned,
|
||||
madePermanent,
|
||||
expires,
|
||||
)
|
||||
}
|
||||
|
||||
if _, banned, _ := ledger.Check(netblock.Addr(), expires); banned {
|
||||
t.Error("the ban refuses a request at its end")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBlacklistLeavesANetblockWithAnActiveBanAsItIs(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
now := midnight()
|
||||
scores := map[netip.Prefix]int64{netblock: 100}
|
||||
|
||||
limitBan, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
|
||||
|
||||
if made := ledger.BanForAbuseIPDB(now, now.Add(30*day), scores); made != 0 {
|
||||
t.Errorf("made %d bans over a ban for a limit, want none", made)
|
||||
}
|
||||
|
||||
// Once that ban has ended the netblock is banned, and the blacklist
|
||||
// fetched again makes no second ban while the first lasts.
|
||||
later := limitBan.Expires
|
||||
if made := ledger.BanForAbuseIPDB(later, later.Add(30*day), scores); made != 1 {
|
||||
t.Errorf("made %d bans after the ban for a limit, want 1", made)
|
||||
}
|
||||
|
||||
later = later.Add(time.Hour)
|
||||
if made := ledger.BanForAbuseIPDB(later, later.Add(30*day), scores); made != 0 {
|
||||
t.Errorf("made %d bans while the blacklist's lasts, want none", made)
|
||||
}
|
||||
|
||||
wantBans(t, ledger, map[netip.Prefix]int{netblock: 2})
|
||||
|
||||
if got := ledger.Bans(netblock)[1].Notes.EarlierBans; got != (bans.EarlierBans{
|
||||
Limit: 1,
|
||||
}) {
|
||||
t.Errorf("earlier bans %+v, want 1 for a limit", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLiftedBlacklistBanHoldsUntilItsEnd(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultParams())
|
||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||
now := midnight()
|
||||
expires := now.Add(30 * day)
|
||||
scores := map[netip.Prefix]int64{netblock: 100}
|
||||
|
||||
ledger.BanForAbuseIPDB(now, expires, scores)
|
||||
ledger.Lift(netblock.Addr(), now)
|
||||
|
||||
// The admin lifted it: the blacklist fetched again until its end bans
|
||||
// nothing.
|
||||
later := expires.Add(-time.Hour)
|
||||
if made := ledger.BanForAbuseIPDB(later, later.Add(30*day), scores); made != 0 {
|
||||
t.Errorf("made %d bans before the lifted ban's end, want none", made)
|
||||
}
|
||||
|
||||
// From its end on, the blacklist bans the netblock again, and the
|
||||
// lifted ban, ended, is forgotten.
|
||||
if made := ledger.BanForAbuseIPDB(expires, expires.Add(30*day), scores); made != 1 {
|
||||
t.Errorf("made %d bans at the lifted ban's end, want 1", made)
|
||||
}
|
||||
|
||||
wantBans(t, ledger, map[netip.Prefix]int{netblock: 1})
|
||||
}
|
||||
|
||||
func TestEndedBlacklistBansAreForgottenAtTheNextFetch(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ledger := bans.New(defaultParams())
|
||||
listed := netip.MustParsePrefix("203.0.113.9/32")
|
||||
limited := netip.MustParsePrefix("203.0.113.10/32")
|
||||
now := midnight()
|
||||
expires := now.Add(30 * day)
|
||||
|
||||
ledger.BanForAbuseIPDB(now, expires, map[netip.Prefix]int64{listed: 100})
|
||||
ledger.BanForLimit(limited, now, bans.Notes{})
|
||||
|
||||
// listed is off the blacklist when it is fetched after the ban's end.
|
||||
if made := ledger.BanForAbuseIPDB(expires, expires.Add(30*day), nil); made != 0 {
|
||||
t.Errorf("made %d bans for an empty blacklist, want none", made)
|
||||
}
|
||||
|
||||
wantBans(t, ledger, map[netip.Prefix]int{listed: 0, limited: 1})
|
||||
}
|
||||
|
||||
func TestBlacklistBansDoNotCountTowardMaxBans(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
params := defaultParams()
|
||||
params.MaxBans = 1
|
||||
ledger := bans.New(params)
|
||||
limited := netip.MustParsePrefix("203.0.113.1/32")
|
||||
attacker := netip.MustParsePrefix("203.0.113.2/32")
|
||||
now := midnight()
|
||||
|
||||
scores := map[netip.Prefix]int64{}
|
||||
for i := range 3 {
|
||||
scores[netip.PrefixFrom(netip.AddrFrom4([4]byte{198, 51, 100, byte(i)}), 32)] = 100
|
||||
}
|
||||
|
||||
ledger.BanForLimit(limited, now, bans.Notes{})
|
||||
|
||||
if made := ledger.BanForAbuseIPDB(now, now.Add(30*day), scores); made != 3 {
|
||||
t.Errorf("made %d bans for 3 netblocks, want 3", made)
|
||||
}
|
||||
|
||||
wantBans(t, ledger, map[netip.Prefix]int{limited: 1})
|
||||
|
||||
// The ban for an attack takes the place of the one for a limit, and no
|
||||
// blacklist ban goes.
|
||||
ledger.BanForAttack(attacker, now, bans.Notes{})
|
||||
|
||||
want := map[netip.Prefix]int{limited: 0, attacker: 1}
|
||||
for netblock := range scores {
|
||||
want[netblock] = 1
|
||||
}
|
||||
|
||||
wantBans(t, ledger, want)
|
||||
}
|
||||
+180
-47
@@ -1,8 +1,9 @@
|
||||
// Package bans is the ban ledger: the bans sww makes on the
|
||||
// netblocks of clients that break a rate limit, a byte limit or the refusal
|
||||
// limit or show a clear sign of attack, and those an admin makes, with their
|
||||
// notes, as the "Bans" section of SPEC.md describes. The bans are kept in
|
||||
// memory, and written to bans.json and read from it by the state package.
|
||||
// limit or show a clear sign of attack, or are on AbuseIPDB's blacklist, and
|
||||
// those an admin makes, with their notes, as the "Bans" section of SPEC.md
|
||||
// describes. The bans are kept in memory, and written to bans.json and read
|
||||
// from it by the state package.
|
||||
package bans
|
||||
|
||||
import (
|
||||
@@ -15,17 +16,26 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/hashicorp/golang-lru/v2/simplelru"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||
)
|
||||
|
||||
// Cause is why a ban was made: CauseLimit, CauseAttack, CauseAdmin or
|
||||
// CauseAbuseIPDB.
|
||||
type Cause string
|
||||
|
||||
// The causes of bans.
|
||||
const (
|
||||
// CauseLimit is a ban sww made for a broken limit.
|
||||
CauseLimit = "limit"
|
||||
CauseLimit Cause = "limit"
|
||||
// CauseAttack is a ban sww made for a clear sign of attack.
|
||||
CauseAttack = "attack"
|
||||
CauseAttack Cause = "attack"
|
||||
// CauseAdmin is a ban an admin made, or one sww made that an
|
||||
// admin keeps. It is never dropped.
|
||||
CauseAdmin = "admin"
|
||||
CauseAdmin Cause = "admin"
|
||||
// CauseAbuseIPDB is a ban sww made for a client on AbuseIPDB's
|
||||
// blacklist. It is forgotten once it has ended.
|
||||
CauseAbuseIPDB Cause = "abuseipdb"
|
||||
)
|
||||
|
||||
// repeatFactor is how many times as long as the netblock's last ban a ban
|
||||
@@ -50,10 +60,11 @@ type Params struct {
|
||||
// AttackBanDuration is how long a first ban for a clear sign of attack
|
||||
// lasts.
|
||||
AttackBanDuration time.Duration
|
||||
// MaxBans is the most bans held whose cause is not CauseAdmin, at
|
||||
// least one. Past it, the earliest such ban of the netblock that has
|
||||
// gone longest without a request is dropped. Bans whose cause is
|
||||
// CauseAdmin are held besides, and never dropped.
|
||||
// MaxBans is the most bans held whose cause is CauseLimit or
|
||||
// CauseAttack, at least one. Past it, the earliest such ban of the
|
||||
// netblock that has gone longest without a request is dropped. Bans
|
||||
// whose cause is CauseAdmin or CauseAbuseIPDB are held besides, and
|
||||
// never dropped this way.
|
||||
MaxBans int
|
||||
}
|
||||
|
||||
@@ -63,10 +74,10 @@ type Ban struct {
|
||||
Start time.Time
|
||||
// Expires is when the ban ends, zero for a permanent ban.
|
||||
Expires time.Time
|
||||
// Cause is CauseLimit, CauseAttack or CauseAdmin.
|
||||
Cause string
|
||||
// Reason is a short text: for a ban sww made, the limit broken or
|
||||
// the rule that matched; for an admin's, what the admin wrote.
|
||||
Cause Cause
|
||||
// Reason is a short text: for a ban sww made, the limit broken, the
|
||||
// rule that matched or AbuseIPDB's score; for an admin's, what the
|
||||
// admin wrote.
|
||||
Reason string
|
||||
// Lifted is when an admin lifted the ban, zero while no admin has. A
|
||||
// lifted ban refuses nothing, and does not make the netblock's next
|
||||
@@ -105,10 +116,10 @@ type Notes struct {
|
||||
// request that broke the limit included.
|
||||
// These are what counted toward the ban, and the window is the time
|
||||
// over which they came.
|
||||
Kind string `json:"kind,omitempty"`
|
||||
Limit int64 `json:"limit,omitempty"`
|
||||
Window string `json:"window,omitempty"`
|
||||
Count float64 `json:"count,omitempty"`
|
||||
Kind ratelimit.LimitKind `json:"kind,omitempty"`
|
||||
Limit int64 `json:"limit,omitempty"`
|
||||
Window string `json:"window,omitempty"`
|
||||
Count float64 `json:"count,omitempty"`
|
||||
// LimitPercent and LimitPercentSetting are, for a ban for a limit a
|
||||
// biased threshold lowered, the client's percentage of that kind of
|
||||
// limit, of which Limit is the result, and the setting that gave it.
|
||||
@@ -127,7 +138,8 @@ type Notes struct {
|
||||
// log's reputation names them. It is left out when none did.
|
||||
Reputation []ReputationHit `json:"reputation,omitempty"`
|
||||
// Request is the request that broke the limit, or whose bytes broke
|
||||
// it, or that was the clear sign of attack.
|
||||
// it, or that was the clear sign of attack. A ban for AbuseIPDB's
|
||||
// blacklist has none, and leaves it zero.
|
||||
Request Request `json:"request"`
|
||||
// Requests is how many requests the netblock has sent since it was
|
||||
// first seen, and Refused how many of them the ban has refused so
|
||||
@@ -150,9 +162,10 @@ type ReputationHit struct {
|
||||
|
||||
// EarlierBans counts a netblock's bans before a ban, by cause.
|
||||
type EarlierBans struct {
|
||||
Limit int `json:"limit"`
|
||||
Attack int `json:"attack"`
|
||||
Admin int `json:"admin"`
|
||||
Limit int `json:"limit"`
|
||||
Attack int `json:"attack"`
|
||||
Admin int `json:"admin"`
|
||||
AbuseIPDB int `json:"abuseipdb"`
|
||||
}
|
||||
|
||||
// Request is a request in a ban's notes. Each text is cut to 256 bytes.
|
||||
@@ -180,13 +193,13 @@ type Ledger struct {
|
||||
// netblocks holds each banned netblock's bans, oldest first. Check and
|
||||
// Find make each netblock they find the most recently seen.
|
||||
netblocks *simplelru.LRU[netip.Prefix, *[]Ban]
|
||||
// held is how many bans netblocks holds whose cause is not CauseAdmin,
|
||||
// at most params.MaxBans.
|
||||
// held is how many bans netblocks holds whose cause is CauseLimit or
|
||||
// CauseAttack, at most params.MaxBans.
|
||||
held int
|
||||
// made is how many bans have been made since the start, by cause: by
|
||||
// the ledger, and by an admin, through BanForAdmin or in an edit of
|
||||
// bans.json.
|
||||
made map[string]int
|
||||
made map[Cause]int
|
||||
// v4Lengths and v6Lengths are the lengths of the IPv4 and IPv6
|
||||
// netblocks that have been banned. Check looks for a ban at each of
|
||||
// them, so that a ban read from bans.json refuses every client in its
|
||||
@@ -198,8 +211,9 @@ type Ledger struct {
|
||||
// New returns a Ledger with no ban yet.
|
||||
func New(params Params) *Ledger {
|
||||
// The ledger drops bans itself, and never those whose cause is
|
||||
// CauseAdmin, however many there are, so the LRU has no limit of its
|
||||
// own: it keeps the netblocks in the order they were last seen.
|
||||
// CauseAdmin or CauseAbuseIPDB to make room, however many there are, so
|
||||
// the LRU has no limit of its own: it keeps the netblocks in the order
|
||||
// they were last seen.
|
||||
netblocks, err := simplelru.NewLRU[netip.Prefix, *[]Ban](
|
||||
math.MaxInt,
|
||||
nil,
|
||||
@@ -212,7 +226,7 @@ func New(params Params) *Ledger {
|
||||
params: params,
|
||||
changed: make(chan struct{}, 1),
|
||||
netblocks: netblocks,
|
||||
made: map[string]int{},
|
||||
made: map[Cause]int{},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -367,7 +381,7 @@ func (l *Ledger) WouldBanForAttack(
|
||||
// or CauseAttack, made at now would be permanent, as BanForLimit or
|
||||
// BanForAttack would make it. It works out nothing else of the ban.
|
||||
func (l *Ledger) WouldBePermanent(
|
||||
netblock netip.Prefix, now time.Time, cause string,
|
||||
netblock netip.Prefix, now time.Time, cause Cause,
|
||||
) bool {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
@@ -429,6 +443,88 @@ func (l *Ledger) BanForAdmin(
|
||||
return ban
|
||||
}
|
||||
|
||||
// BanForAbuseIPDB bans each netblock in scores at now until expires, and
|
||||
// returns how many bans it made. scores are the netblocks of the clients
|
||||
// on AbuseIPDB's blacklist, each with the highest score of its clients.
|
||||
// A netblock is left as it is while a ban on it is active, or while a ban
|
||||
// for the blacklist that an admin lifted has not reached its end: the
|
||||
// admin's lift holds that long. Each ban whose cause is CauseAbuseIPDB
|
||||
// and that has ended is forgotten first, so that the bans held for the
|
||||
// blacklist are those of the clients listed within one ban's length.
|
||||
func (l *Ledger) BanForAbuseIPDB(
|
||||
now, expires time.Time, scores map[netip.Prefix]int64,
|
||||
) int {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
|
||||
changed := l.forgetEndedAbuseIPDBBans(now)
|
||||
made := 0
|
||||
|
||||
for netblock, score := range scores {
|
||||
netblock = netblock.Masked()
|
||||
|
||||
var notes Notes
|
||||
|
||||
if held, found := l.netblocks.Peek(netblock); found {
|
||||
if activeBan(*held, now) != nil ||
|
||||
liftedAbuseIPDBBan(*held, now) {
|
||||
continue
|
||||
}
|
||||
|
||||
notes.EarlierBans = earlierBans(*held)
|
||||
}
|
||||
|
||||
// The source is named as the request log names AbuseIPDB.
|
||||
notes.Reputation = []ReputationHit{
|
||||
{Source: string(CauseAbuseIPDB), Score: &score},
|
||||
}
|
||||
|
||||
l.add(Ban{
|
||||
Netblock: netblock,
|
||||
Start: now,
|
||||
Expires: expires,
|
||||
Cause: CauseAbuseIPDB,
|
||||
Reason: fmt.Sprintf(
|
||||
"on AbuseIPDB's blacklist with a score of %d", score,
|
||||
),
|
||||
Notes: notes,
|
||||
})
|
||||
|
||||
made++
|
||||
}
|
||||
|
||||
l.made[CauseAbuseIPDB] += made
|
||||
|
||||
if changed || made > 0 {
|
||||
l.markChanged()
|
||||
}
|
||||
|
||||
return made
|
||||
}
|
||||
|
||||
// ForgetEndedAbuseIPDBBans forgets the bans whose cause is CauseAbuseIPDB
|
||||
// and that have ended at now, as BanForAbuseIPDB does first. The state
|
||||
// package calls it each time it reads bans.json, so that these bans are
|
||||
// also forgotten while no fetch of the blacklist reaches the ledger.
|
||||
func (l *Ledger) ForgetEndedAbuseIPDBBans(now time.Time) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
|
||||
if l.forgetEndedAbuseIPDBBans(now) {
|
||||
l.markChanged()
|
||||
}
|
||||
}
|
||||
|
||||
// liftedAbuseIPDBBan reports whether bans, a netblock's, hold a ban whose
|
||||
// cause is CauseAbuseIPDB that an admin lifted, and whose end is not
|
||||
// reached at now.
|
||||
func liftedAbuseIPDBBan(bans []Ban, now time.Time) bool {
|
||||
return slices.ContainsFunc(bans, func(ban Ban) bool {
|
||||
return ban.Cause == CauseAbuseIPDB && !ban.Lifted.IsZero() &&
|
||||
(ban.Permanent() || now.Before(ban.Expires))
|
||||
})
|
||||
}
|
||||
|
||||
// Lift lifts, at now, every ban active then on a netblock client is in,
|
||||
// as an admin does, and returns those bans. A lifted ban is kept, refuses
|
||||
// nothing, and does not make the netblock's next ban longer.
|
||||
@@ -515,10 +611,10 @@ func (l *Ledger) AddLookup(
|
||||
}
|
||||
|
||||
// Made returns how many bans for cause have been made since the start:
|
||||
// for CauseLimit and CauseAttack, by the ledger; for CauseAdmin, by an
|
||||
// admin, with BanForAdmin or in an edit of bans.json, as LoadEdit counts
|
||||
// them. The bans read from bans.json at the start are not among them.
|
||||
func (l *Ledger) Made(cause string) int {
|
||||
// for CauseLimit, CauseAttack and CauseAbuseIPDB, by the ledger; for
|
||||
// CauseAdmin, by an admin, with BanForAdmin or in an edit of bans.json, as
|
||||
// LoadEdit counts them. The bans read from bans.json at the start are not among them.
|
||||
func (l *Ledger) Made(cause Cause) int {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
|
||||
@@ -574,8 +670,8 @@ func (l *Ledger) Snapshot() []Ban {
|
||||
// seen. A ban without a cause is an admin's, and gets CauseAdmin. Each
|
||||
// netblock is masked to its length, so that 203.0.113.9/24 is
|
||||
// 203.0.113.0/24, and each text in the notes is cut to 256 bytes. Past
|
||||
// MaxBans the earliest bans whose cause is not CauseAdmin are dropped, as
|
||||
// when they are made.
|
||||
// MaxBans the earliest bans that count toward it are dropped, as when they
|
||||
// are made.
|
||||
func (l *Ledger) Load(bans []Ban) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
@@ -639,6 +735,35 @@ func (l *Ledger) holds(netblock netip.Prefix, start time.Time) bool {
|
||||
})
|
||||
}
|
||||
|
||||
// forgetEndedAbuseIPDBBans forgets each ban whose cause is CauseAbuseIPDB
|
||||
// and whose end is reached at now, lifted or not, and each netblock left
|
||||
// without a ban. It reports whether it forgot one.
|
||||
func (l *Ledger) forgetEndedAbuseIPDBBans(now time.Time) bool {
|
||||
forgot := false
|
||||
|
||||
for _, netblock := range l.netblocks.Keys() {
|
||||
bans, _ := l.netblocks.Peek(netblock)
|
||||
|
||||
kept := slices.DeleteFunc(*bans, func(ban Ban) bool {
|
||||
return ban.Cause == CauseAbuseIPDB && !ban.Permanent() &&
|
||||
!now.Before(ban.Expires)
|
||||
})
|
||||
if len(kept) == len(*bans) {
|
||||
continue
|
||||
}
|
||||
|
||||
forgot = true
|
||||
|
||||
if len(kept) == 0 {
|
||||
l.netblocks.Remove(netblock)
|
||||
} else {
|
||||
*bans = kept
|
||||
}
|
||||
}
|
||||
|
||||
return forgot
|
||||
}
|
||||
|
||||
// ban bans netblock at now for cause, CauseLimit or CauseAttack, with
|
||||
// reason and notes, as BanForLimit and BanForAttack describe, and returns
|
||||
// the ban, and whether it made it. Unless keep is true, the ban is not
|
||||
@@ -646,7 +771,8 @@ func (l *Ledger) holds(netblock netip.Prefix, start time.Time) bool {
|
||||
func (l *Ledger) ban(
|
||||
netblock netip.Prefix,
|
||||
now time.Time,
|
||||
cause, reason string,
|
||||
cause Cause,
|
||||
reason string,
|
||||
notes Notes,
|
||||
keep bool,
|
||||
) (Ban, bool) {
|
||||
@@ -714,13 +840,16 @@ func earlierBans(held []Ban) EarlierBans {
|
||||
fromHere.Attack++
|
||||
case CauseAdmin:
|
||||
fromHere.Admin++
|
||||
case CauseAbuseIPDB:
|
||||
fromHere.AbuseIPDB++
|
||||
}
|
||||
|
||||
notes := ban.Notes.EarlierBans
|
||||
count := EarlierBans{
|
||||
Limit: notes.Limit + fromHere.Limit,
|
||||
Attack: notes.Attack + fromHere.Attack,
|
||||
Admin: notes.Admin + fromHere.Admin,
|
||||
Limit: notes.Limit + fromHere.Limit,
|
||||
Attack: notes.Attack + fromHere.Attack,
|
||||
Admin: notes.Admin + fromHere.Admin,
|
||||
AbuseIPDB: notes.AbuseIPDB + fromHere.AbuseIPDB,
|
||||
}
|
||||
|
||||
// Of counts that name as many bans, the earlier ban's is taken: it
|
||||
@@ -735,7 +864,7 @@ func earlierBans(held []Ban) EarlierBans {
|
||||
|
||||
// total returns how many bans e counts, of all causes.
|
||||
func (e EarlierBans) total() int {
|
||||
return e.Limit + e.Attack + e.Admin
|
||||
return e.Limit + e.Attack + e.Admin + e.AbuseIPDB
|
||||
}
|
||||
|
||||
// markChanged has Changed receive a value, unless one is waiting already.
|
||||
@@ -806,9 +935,9 @@ func (l *Ledger) covering(client netip.Addr) []*[]Ban {
|
||||
|
||||
// add adds ban to its netblock's bans, after the last, and makes its
|
||||
// netblock the most recently seen. With MaxBans held, it drops one first,
|
||||
// unless ban's cause is CauseAdmin, which does not count toward MaxBans.
|
||||
// unless ban does not count toward MaxBans.
|
||||
func (l *Ledger) add(ban Ban) {
|
||||
counted := ban.Cause != CauseAdmin
|
||||
counted := countsTowardMaxBans(ban)
|
||||
if counted && l.held == l.params.MaxBans {
|
||||
l.dropOne()
|
||||
}
|
||||
@@ -889,7 +1018,13 @@ func (l *Ledger) attackExpiry(held []Ban, now time.Time) time.Time {
|
||||
return now.Add(l.params.AttackBanDuration)
|
||||
}
|
||||
|
||||
// dropOne drops the earliest ban whose cause is not CauseAdmin of the
|
||||
// countsTowardMaxBans reports whether ban counts toward MaxBans: its cause
|
||||
// is CauseLimit or CauseAttack.
|
||||
func countsTowardMaxBans(ban Ban) bool {
|
||||
return ban.Cause == CauseLimit || ban.Cause == CauseAttack
|
||||
}
|
||||
|
||||
// dropOne drops the earliest ban that counts toward MaxBans of the
|
||||
// netblock that has gone longest without a request, of those that hold
|
||||
// such a ban, and the netblock with it if that was its only ban. It is
|
||||
// called with at least one such ban held. It looks at each netblock once
|
||||
@@ -898,11 +1033,9 @@ func (l *Ledger) dropOne() {
|
||||
for range l.netblocks.Len() {
|
||||
netblock, bans, _ := l.netblocks.GetOldest()
|
||||
|
||||
i := slices.IndexFunc(*bans, func(ban Ban) bool {
|
||||
return ban.Cause != CauseAdmin
|
||||
})
|
||||
i := slices.IndexFunc(*bans, countsTowardMaxBans)
|
||||
if i < 0 {
|
||||
// Its bans are all an admin's, and never dropped. Get makes
|
||||
// None of its bans is dropped to make room. Get makes
|
||||
// it the most recently seen, so that the next netblock is
|
||||
// looked at; when it was seen matters only for dropping a
|
||||
// ban, and a ban added to it makes it the most recently seen
|
||||
|
||||
+204
-60
@@ -27,6 +27,19 @@ import (
|
||||
"sneak.berlin/go/smallwebwaf/internal/remotelog"
|
||||
)
|
||||
|
||||
// The names of the settings that waf writes into log lines and ban notes,
|
||||
// as the setting that refused a request or lowered a client's limits.
|
||||
const (
|
||||
ExclusivelyAllowedCountriesSetting = "SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES"
|
||||
TagLimitPercentSetting = "SWW_WAF_TAG_LIMIT_PERCENT"
|
||||
ASNBytesLimitPercentSetting = "SWW_WAF_ASN_BYTES_LIMIT_PERCENT"
|
||||
CountryBytesLimitPercentSetting = "SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT"
|
||||
UnknownCountryLimitPercentSetting = "SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT"
|
||||
ASNLimitPercentURLSetting = "SWW_WAF_ASN_LIMIT_PERCENT_URL"
|
||||
BlocklistActionSetting = "SWW_WAF_BLOCKLIST_ACTION"
|
||||
DNSBLAbuseIPDBActionSetting = "SWW_WAF_DNSBL_ABUSEIPDB_ACTION"
|
||||
)
|
||||
|
||||
// WAF is the settings of `sww waf`. A time or size limit of zero is off.
|
||||
type WAF struct {
|
||||
// ListenAddr is where waf listens (SWW_WAF_LISTEN_ADDR).
|
||||
@@ -75,7 +88,7 @@ type WAF struct {
|
||||
BytesLimitPerMinute int64
|
||||
BytesLimitPerHour int64
|
||||
BytesLimitPerDay int64
|
||||
BytesCounted string
|
||||
BytesCounted BytesCounted
|
||||
// LookupSource is where each client's AS number and country are looked
|
||||
// up (SWW_WAF_LOOKUP_SOURCE): geojs, file, or off for nowhere.
|
||||
// LookupDBPath is the lookup database, the IPinfo Lite file looked up in
|
||||
@@ -86,7 +99,7 @@ type WAF struct {
|
||||
// AddLookupHeaders is true when the app is passed the client's AS number
|
||||
// and country in headers (SWW_WAF_ADD_LOOKUP_HEADERS), by default unless
|
||||
// LookupSource is off.
|
||||
LookupSource string
|
||||
LookupSource LookupSource
|
||||
LookupDBPath string
|
||||
ASNAndCountryLookupTimeout time.Duration
|
||||
AddLookupHeaders bool
|
||||
@@ -131,7 +144,7 @@ type WAF struct {
|
||||
// BlocklistLimitPercent is the percentage of every limit it gets.
|
||||
BlocklistURLs []string
|
||||
BlocklistRefresh time.Duration
|
||||
BlocklistAction string
|
||||
BlocklistAction ReputationAction
|
||||
BlocklistLimitPercent int64
|
||||
// DNSBLZones are the DNSBL zones clients are asked about
|
||||
// (SWW_WAF_DNSBL_ZONES), through DNSBLResolver (SWW_WAF_DNSBL_RESOLVER), or
|
||||
@@ -154,10 +167,18 @@ type WAF struct {
|
||||
AbuseIPDBKey string
|
||||
AbuseIPDBMinScore int64
|
||||
AbuseIPDBDailyBudget int
|
||||
DNSBLAbuseIPDBAction string
|
||||
DNSBLAbuseIPDBAction ReputationAction
|
||||
DNSBLAbuseIPDBLimitPercent int64
|
||||
IPReputationCacheTTL time.Duration
|
||||
DNSBLAbuseIPDBTimeout time.Duration
|
||||
// While AbuseIPDBKey is set, AbuseIPDB's blacklist of the clients it
|
||||
// scores AbuseIPDBThreshold or more (SWW_WAF_ABUSEIPDB_THRESHOLD), from
|
||||
// 25 to 100, is fetched at the start and every AbuseIPDBBlacklistRefresh
|
||||
// (SWW_WAF_ABUSEIPDB_BLACKLIST_REFRESH_INTERVAL_MINS). Each client on it
|
||||
// is banned for AbuseIPDBBanDuration (SWW_WAF_ABUSEIPDB_BAN_DAYS).
|
||||
AbuseIPDBThreshold int64
|
||||
AbuseIPDBBlacklistRefresh time.Duration
|
||||
AbuseIPDBBanDuration time.Duration
|
||||
// BanResponse is the status a refused client is answered with, 403
|
||||
// or 429, or 0 to close the connection without an answer
|
||||
// (SWW_WAF_BAN_RESPONSE). It answers a banned client, a request that
|
||||
@@ -229,7 +250,7 @@ type WAF struct {
|
||||
// reads none.
|
||||
// CoreRuleSetExclusionsFile is the path of the file of its exclusions
|
||||
// (SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE), "" while there is none.
|
||||
CoreRuleSetMode string
|
||||
CoreRuleSetMode CoreRuleSetMode
|
||||
CoreRuleSetParanoiaLevel int
|
||||
CoreRuleSetScoreThreshold int
|
||||
CoreRuleSetDisabledRules []int
|
||||
@@ -282,7 +303,7 @@ type WAF struct {
|
||||
AlertSlackWebhookURL *url.URL
|
||||
AlertNtfyURL *url.URL
|
||||
AlertNtfyToken string
|
||||
AlertEvents []string
|
||||
AlertEvents []alerts.Event
|
||||
AlertCooldown time.Duration
|
||||
AlertMaxPerHour int
|
||||
// The anomaly thresholds, which only raise alerts: the most requests
|
||||
@@ -308,20 +329,66 @@ type WAF struct {
|
||||
settings []slog.Attr
|
||||
}
|
||||
|
||||
// BytesCounted is which body bytes count toward the byte limits and the
|
||||
// byte anomaly thresholds, a value of SWW_WAF_BYTES_COUNTED.
|
||||
type BytesCounted string
|
||||
|
||||
// The values of SWW_WAF_BYTES_COUNTED.
|
||||
const (
|
||||
// BytesCountedResponse counts the response's body bytes.
|
||||
BytesCountedResponse BytesCounted = "response"
|
||||
// BytesCountedRequest counts the request's body bytes.
|
||||
BytesCountedRequest BytesCounted = "request"
|
||||
// BytesCountedBoth counts the body bytes of both.
|
||||
BytesCountedBoth BytesCounted = "both"
|
||||
)
|
||||
|
||||
// LookupSource is where each client's AS number and country are looked up,
|
||||
// a value of SWW_WAF_LOOKUP_SOURCE.
|
||||
type LookupSource string
|
||||
|
||||
// The values of SWW_WAF_LOOKUP_SOURCE.
|
||||
const (
|
||||
// LookupSourceGeoJS asks GeoJS.
|
||||
LookupSourceGeoJS LookupSource = "geojs"
|
||||
// LookupSourceFile looks clients up in the lookup database, the file
|
||||
// SWW_WAF_LOOKUP_DB_PATH names.
|
||||
LookupSourceFile LookupSource = "file"
|
||||
// LookupSourceOff looks no client up.
|
||||
LookupSourceOff LookupSource = "off"
|
||||
)
|
||||
|
||||
// ReputationAction is what is done with a client a reputation source
|
||||
// lists, a value of SWW_WAF_BLOCKLIST_ACTION and of
|
||||
// SWW_WAF_DNSBL_ABUSEIPDB_ACTION.
|
||||
type ReputationAction string
|
||||
|
||||
// The values of SWW_WAF_BLOCKLIST_ACTION and SWW_WAF_DNSBL_ABUSEIPDB_ACTION.
|
||||
const (
|
||||
// ReputationActionDeny refuses the client's requests.
|
||||
ReputationActionDeny ReputationAction = "deny"
|
||||
// ReputationActionLimit gives the client a percentage of every limit,
|
||||
// written limit:<percent> in the setting.
|
||||
ReputationActionLimit ReputationAction = "limit"
|
||||
// ReputationActionLog only notes the hit in the log line and raises an
|
||||
// alert.
|
||||
ReputationActionLog ReputationAction = "log"
|
||||
)
|
||||
|
||||
// CoreRuleSetMode is what the Core Rule Set does, a value of
|
||||
// SWW_WAF_CORE_RULE_SET_MODE.
|
||||
type CoreRuleSetMode string
|
||||
|
||||
// The values of SWW_WAF_CORE_RULE_SET_MODE.
|
||||
const (
|
||||
// CoreRuleSetModeOff runs no request through the Core Rule Set.
|
||||
CoreRuleSetModeOff = off
|
||||
CoreRuleSetModeOff CoreRuleSetMode = "off"
|
||||
// CoreRuleSetModeDetect logs and alerts a match, and refuses nothing.
|
||||
CoreRuleSetModeDetect = "detect"
|
||||
CoreRuleSetModeDetect CoreRuleSetMode = "detect"
|
||||
// CoreRuleSetModeBlock refuses a match with 403.
|
||||
CoreRuleSetModeBlock = "block"
|
||||
CoreRuleSetModeBlock CoreRuleSetMode = "block"
|
||||
)
|
||||
|
||||
// fileSource is the SWW_WAF_LOOKUP_SOURCE that looks clients up in the
|
||||
// lookup database, the file SWW_WAF_LOOKUP_DB_PATH names.
|
||||
const fileSource = "file"
|
||||
|
||||
const (
|
||||
day = 24 * time.Hour
|
||||
ipv4Bits = 32
|
||||
@@ -330,6 +397,9 @@ const (
|
||||
// netblock a provider is usually given: a shorter one would make one
|
||||
// client of the customers of several providers.
|
||||
minIPv6GroupPrefix = 32
|
||||
// minAbuseIPDBThreshold is the lowest score AbuseIPDB's blacklist can
|
||||
// be cut at.
|
||||
minAbuseIPDBThreshold = 25
|
||||
// minTokenLength is the fewest characters a token may have.
|
||||
minTokenLength = 32
|
||||
// maxParanoiaLevel is the Core Rule Set's highest paranoia level.
|
||||
@@ -446,6 +516,13 @@ var (
|
||||
errNotPercent = errors.New(
|
||||
"is not a percentage, a whole number from 0 to 100",
|
||||
)
|
||||
errNotAbuseIPDBThreshold = errors.New(
|
||||
"is not a whole number from 25 to 100, the scores AbuseIPDB's " +
|
||||
"blacklist can be cut at",
|
||||
)
|
||||
errNotWholeUnits = errors.New(
|
||||
"is not a whole number above zero, such as 30, that fits a duration",
|
||||
)
|
||||
errListedTwice = errors.New("is listed twice")
|
||||
errNotListURL = errors.New(
|
||||
"is not an http or https URL without a user or a fragment, " +
|
||||
@@ -540,35 +617,38 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
|
||||
),
|
||||
BytesLimitPerHour: env.size("SWW_WAF_BYTES_LIMIT_PER_HOUR", "50G"),
|
||||
BytesLimitPerDay: env.size("SWW_WAF_BYTES_LIMIT_PER_DAY", "100G"),
|
||||
BytesCounted: env.bytesCounted("SWW_WAF_BYTES_COUNTED", "both"),
|
||||
BytesCounted: env.bytesCounted(
|
||||
"SWW_WAF_BYTES_COUNTED",
|
||||
BytesCountedBoth,
|
||||
),
|
||||
LookupSource: env.lookupSource(
|
||||
"SWW_WAF_LOOKUP_SOURCE",
|
||||
"geojs",
|
||||
LookupSourceGeoJS,
|
||||
),
|
||||
LookupDBPath: env.value("SWW_WAF_LOOKUP_DB_PATH", ""),
|
||||
ASNAndCountryLookupTimeout: env.milliseconds(
|
||||
"SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS",
|
||||
"500",
|
||||
"800",
|
||||
),
|
||||
ExclusivelyAllowedCountries: env.countries(
|
||||
"SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES", ""),
|
||||
ExclusivelyAllowedCountriesSetting, ""),
|
||||
TagLists: env.tagLists("SWW_WAF_TAG_LISTS"),
|
||||
TagDeny: env.tags("SWW_WAF_TAG_DENY"),
|
||||
TagLimitPercent: env.tagPercents("SWW_WAF_TAG_LIMIT_PERCENT"),
|
||||
TagLimitPercent: env.tagPercents(TagLimitPercentSetting),
|
||||
RouteWatchURL: env.routeWatchURL("SWW_WAF_ROUTEWATCH_URL"),
|
||||
ASNBytesLimitPercent: env.percents(
|
||||
"SWW_WAF_ASN_BYTES_LIMIT_PERCENT",
|
||||
ASNBytesLimitPercentSetting,
|
||||
ParseASN,
|
||||
),
|
||||
CountryBytesLimitPercent: env.percents(
|
||||
"SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT",
|
||||
CountryBytesLimitPercentSetting,
|
||||
parseCountry,
|
||||
),
|
||||
UnknownCountryLimitPercent: env.percent(
|
||||
"SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT",
|
||||
UnknownCountryLimitPercentSetting,
|
||||
"100",
|
||||
),
|
||||
ASNLimitPercentURL: env.listURL("SWW_WAF_ASN_LIMIT_PERCENT_URL"),
|
||||
ASNLimitPercentURL: env.listURL(ASNLimitPercentURLSetting),
|
||||
BlocklistURLs: env.listURLs("SWW_WAF_BLOCKLIST_URLS"),
|
||||
BlocklistRefresh: env.refresh("SWW_WAF_BLOCKLIST_REFRESH", "24h"),
|
||||
DNSBLZones: env.zones("SWW_WAF_DNSBL_ZONES"),
|
||||
@@ -587,6 +667,20 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
|
||||
"SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS",
|
||||
"2000",
|
||||
),
|
||||
AbuseIPDBThreshold: env.abuseIPDBThreshold(
|
||||
"SWW_WAF_ABUSEIPDB_THRESHOLD",
|
||||
"90",
|
||||
),
|
||||
AbuseIPDBBlacklistRefresh: env.wholeUnits(
|
||||
"SWW_WAF_ABUSEIPDB_BLACKLIST_REFRESH_INTERVAL_MINS",
|
||||
"60",
|
||||
time.Minute,
|
||||
),
|
||||
AbuseIPDBBanDuration: env.wholeUnits(
|
||||
"SWW_WAF_ABUSEIPDB_BAN_DAYS",
|
||||
"30",
|
||||
day,
|
||||
),
|
||||
BanResponse: env.banResponse("SWW_WAF_BAN_RESPONSE", "403"),
|
||||
LimitBanDuration: env.durationNotOff(
|
||||
"SWW_WAF_LIMIT_BAN_DURATION",
|
||||
@@ -697,9 +791,8 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
|
||||
),
|
||||
AlertNtfyURL: env.webhookURL("SWW_WAF_ALERT_NTFY_URL"),
|
||||
AlertNtfyToken: env.secret("SWW_WAF_ALERT_NTFY_TOKEN"),
|
||||
AlertEvents: env.alertEvents("SWW_WAF_ALERT_EVENTS",
|
||||
strings.Join(alerts.Events(), ",")),
|
||||
AlertCooldown: env.duration("SWW_WAF_ALERT_COOLDOWN", "15m"),
|
||||
AlertEvents: env.alertEvents("SWW_WAF_ALERT_EVENTS"),
|
||||
AlertCooldown: env.duration("SWW_WAF_ALERT_COOLDOWN", "15m"),
|
||||
AlertMaxPerHour: env.numberOrOff(
|
||||
"SWW_WAF_ALERT_MAX_PER_HOUR",
|
||||
"60",
|
||||
@@ -726,11 +819,11 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
|
||||
cfg.InstanceName, cfg.LogRemoteURL != nil)
|
||||
// The app gets the headers by default, unless no client is looked up.
|
||||
cfg.AddLookupHeaders = env.boolean("SWW_WAF_ADD_LOOKUP_HEADERS",
|
||||
strconv.FormatBool(cfg.LookupSource != off))
|
||||
strconv.FormatBool(cfg.LookupSource != LookupSourceOff))
|
||||
cfg.BlocklistAction, cfg.BlocklistLimitPercent = env.action(
|
||||
"SWW_WAF_BLOCKLIST_ACTION", "deny")
|
||||
BlocklistActionSetting, string(ReputationActionDeny))
|
||||
cfg.DNSBLAbuseIPDBAction, cfg.DNSBLAbuseIPDBLimitPercent = env.action(
|
||||
"SWW_WAF_DNSBL_ABUSEIPDB_ACTION", "limit:25")
|
||||
DNSBLAbuseIPDBActionSetting, "limit:25")
|
||||
|
||||
env.checkInstanceNameForNtfy(cfg.InstanceName, cfg.AlertNtfyURL != nil)
|
||||
env.checkLookupDBPath(cfg)
|
||||
@@ -826,9 +919,12 @@ func (e *environment) count(name, defaultValue string) int64 {
|
||||
|
||||
// bytesCounted reads the setting that is which body bytes count toward the
|
||||
// byte limits and the byte anomaly thresholds: response, request or both.
|
||||
func (e *environment) bytesCounted(name, defaultValue string) string {
|
||||
value := e.value(name, defaultValue)
|
||||
if value != "response" && value != "request" && value != "both" {
|
||||
func (e *environment) bytesCounted(
|
||||
name string, defaultValue BytesCounted,
|
||||
) BytesCounted {
|
||||
value := BytesCounted(e.value(name, string(defaultValue)))
|
||||
if value != BytesCountedResponse && value != BytesCountedRequest &&
|
||||
value != BytesCountedBoth {
|
||||
e.check(name, fmt.Errorf("%q %w", value, errNotBytesCounted))
|
||||
}
|
||||
|
||||
@@ -873,8 +969,10 @@ func (e *environment) trapPaths(name string) []string {
|
||||
|
||||
// coreRuleSetMode reads the setting that is what the Core Rule Set does: off,
|
||||
// detect or block.
|
||||
func (e *environment) coreRuleSetMode(name, defaultValue string) string {
|
||||
mode := e.value(name, defaultValue)
|
||||
func (e *environment) coreRuleSetMode(
|
||||
name string, defaultValue CoreRuleSetMode,
|
||||
) CoreRuleSetMode {
|
||||
mode := CoreRuleSetMode(e.value(name, string(defaultValue)))
|
||||
if mode != CoreRuleSetModeOff && mode != CoreRuleSetModeDetect &&
|
||||
mode != CoreRuleSetModeBlock {
|
||||
e.check(name, fmt.Errorf("%q %w", mode, errNotCoreRuleSetMode))
|
||||
@@ -957,6 +1055,35 @@ func (e *environment) percent(name, defaultValue string) int64 {
|
||||
return percent
|
||||
}
|
||||
|
||||
// abuseIPDBThreshold reads the setting that is the least score of the
|
||||
// clients on AbuseIPDB's blacklist: a whole number from 25 to 100, the
|
||||
// range AbuseIPDB takes.
|
||||
func (e *environment) abuseIPDBThreshold(name, defaultValue string) int64 {
|
||||
value := e.value(name, defaultValue)
|
||||
|
||||
threshold, err := strconv.ParseInt(value, 10, 64)
|
||||
if err != nil || threshold < minAbuseIPDBThreshold || threshold > 100 {
|
||||
e.check(name, fmt.Errorf("%q %w", value, errNotAbuseIPDBThreshold))
|
||||
}
|
||||
|
||||
return threshold
|
||||
}
|
||||
|
||||
// wholeUnits reads a setting that is a whole number above zero of unit,
|
||||
// such as minutes or days, and returns it as a duration.
|
||||
func (e *environment) wholeUnits(
|
||||
name, defaultValue string, unit time.Duration,
|
||||
) time.Duration {
|
||||
value := e.value(name, defaultValue)
|
||||
|
||||
n, err := strconv.ParseInt(value, 10, 64)
|
||||
if err != nil || n <= 0 || n > math.MaxInt64/int64(unit) {
|
||||
e.check(name, fmt.Errorf("%q %w", value, errNotWholeUnits))
|
||||
}
|
||||
|
||||
return time.Duration(n) * unit
|
||||
}
|
||||
|
||||
// listURL reads a setting that is the URL a list is fetched from, "" while
|
||||
// it is unset or empty.
|
||||
func (e *environment) listURL(name string) string {
|
||||
@@ -995,20 +1122,24 @@ func (e *environment) refresh(name, defaultValue string) time.Duration {
|
||||
// action reads a setting that is what is done with a client a blocklist
|
||||
// or a DNSBL zone lists: deny, log, or limit:<percent>, which it returns
|
||||
// as limit and the percentage.
|
||||
func (e *environment) action(name, defaultValue string) (string, int64) {
|
||||
value := e.value(name, defaultValue)
|
||||
if value == "deny" || value == "log" {
|
||||
func (e *environment) action(
|
||||
name, defaultValue string,
|
||||
) (ReputationAction, int64) {
|
||||
value := ReputationAction(e.value(name, defaultValue))
|
||||
if value == ReputationActionDeny || value == ReputationActionLog {
|
||||
return value, 0
|
||||
}
|
||||
|
||||
percentText, isLimit := strings.CutPrefix(value, "limit:")
|
||||
percentText, isLimit := strings.CutPrefix(
|
||||
string(value), string(ReputationActionLimit)+":",
|
||||
)
|
||||
|
||||
percent, err := ParsePercent(percentText)
|
||||
if !isLimit || err != nil {
|
||||
e.check(name, fmt.Errorf("%q %w", value, errNotAction))
|
||||
}
|
||||
|
||||
return "limit", percent
|
||||
return ReputationActionLimit, percent
|
||||
}
|
||||
|
||||
// zones reads the setting that is the list of DNSBL zones. It is empty by
|
||||
@@ -1059,9 +1190,12 @@ func (e *environment) routeWatchURL(name string) string {
|
||||
|
||||
// lookupSource reads the setting that is where clients are looked up:
|
||||
// geojs, file, or off.
|
||||
func (e *environment) lookupSource(name, defaultValue string) string {
|
||||
source := e.value(name, defaultValue)
|
||||
if source != "geojs" && source != fileSource && source != off {
|
||||
func (e *environment) lookupSource(
|
||||
name string, defaultValue LookupSource,
|
||||
) LookupSource {
|
||||
source := LookupSource(e.value(name, string(defaultValue)))
|
||||
if source != LookupSourceGeoJS && source != LookupSourceFile &&
|
||||
source != LookupSourceOff {
|
||||
e.check(name, fmt.Errorf("%q %w", source, errNotLookupSource))
|
||||
}
|
||||
|
||||
@@ -1073,12 +1207,12 @@ func (e *environment) lookupSource(name, defaultValue string) string {
|
||||
// one source at a time.
|
||||
func (e *environment) checkLookupDBPath(cfg *WAF) {
|
||||
switch {
|
||||
case cfg.LookupSource == fileSource && cfg.LookupDBPath == "":
|
||||
case cfg.LookupSource == LookupSourceFile && cfg.LookupDBPath == "":
|
||||
e.check("SWW_WAF_LOOKUP_SOURCE", fmt.Errorf(
|
||||
"is file while SWW_WAF_LOOKUP_DB_PATH is unset; %w",
|
||||
errNeedsDBPath,
|
||||
))
|
||||
case cfg.LookupSource != fileSource && cfg.LookupDBPath != "":
|
||||
case cfg.LookupSource != LookupSourceFile && cfg.LookupDBPath != "":
|
||||
e.check("SWW_WAF_LOOKUP_DB_PATH", fmt.Errorf(
|
||||
"is set while SWW_WAF_LOOKUP_SOURCE is %s; %w",
|
||||
cfg.LookupSource,
|
||||
@@ -1098,12 +1232,12 @@ func (e *environment) checkLookupDBPath(cfg *WAF) {
|
||||
func (e *environment) checkCountriesAndLookups(cfg *WAF) {
|
||||
for _, country := range cfg.ExclusivelyAllowedCountries {
|
||||
if slices.Contains(cfg.TagDeny, CountryTagPrefix+country) {
|
||||
e.check("SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES",
|
||||
e.check(ExclusivelyAllowedCountriesSetting,
|
||||
fmt.Errorf("%q %w", country, errOnBothLists))
|
||||
}
|
||||
}
|
||||
|
||||
if cfg.LookupSource != off {
|
||||
if cfg.LookupSource != LookupSourceOff {
|
||||
return
|
||||
}
|
||||
|
||||
@@ -1117,15 +1251,15 @@ func (e *environment) checkCountriesAndLookups(cfg *WAF) {
|
||||
name string
|
||||
set bool
|
||||
}{
|
||||
{"SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES", len(cfg.ExclusivelyAllowedCountries) > 0},
|
||||
{ExclusivelyAllowedCountriesSetting, len(cfg.ExclusivelyAllowedCountries) > 0},
|
||||
{"SWW_WAF_ADD_LOOKUP_HEADERS", cfg.AddLookupHeaders},
|
||||
{"SWW_WAF_TAG_DENY", slices.ContainsFunc(cfg.TagDeny, onlyLookupGives)},
|
||||
{"SWW_WAF_TAG_LIMIT_PERCENT", slices.ContainsFunc(
|
||||
{TagLimitPercentSetting, slices.ContainsFunc(
|
||||
slices.Collect(maps.Keys(cfg.TagLimitPercent)), onlyLookupGives)},
|
||||
{"SWW_WAF_ASN_BYTES_LIMIT_PERCENT", len(cfg.ASNBytesLimitPercent) > 0},
|
||||
{"SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT", len(cfg.CountryBytesLimitPercent) > 0},
|
||||
{"SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT", cfg.UnknownCountryLimitPercent < 100},
|
||||
{"SWW_WAF_ASN_LIMIT_PERCENT_URL", cfg.ASNLimitPercentURL != ""},
|
||||
{ASNBytesLimitPercentSetting, len(cfg.ASNBytesLimitPercent) > 0},
|
||||
{CountryBytesLimitPercentSetting, len(cfg.CountryBytesLimitPercent) > 0},
|
||||
{UnknownCountryLimitPercentSetting, cfg.UnknownCountryLimitPercent < 100},
|
||||
{ASNLimitPercentURLSetting, cfg.ASNLimitPercentURL != ""},
|
||||
{"SWW_WAF_ANOMALY_ASN_REQUESTS_PER_MINUTE", cfg.AnomalyASN.RequestsPerMinute > 0},
|
||||
{"SWW_WAF_ANOMALY_ASN_REQUESTS_PER_HOUR", cfg.AnomalyASN.RequestsPerHour > 0},
|
||||
{"SWW_WAF_ANOMALY_ASN_BYTES_PER_MINUTE", cfg.AnomalyASN.BytesPerMinute > 0},
|
||||
@@ -1145,7 +1279,7 @@ func (e *environment) checkCountriesAndLookups(cfg *WAF) {
|
||||
// blocklist too: the file at a URL is fetched as one list or the other.
|
||||
func (e *environment) checkASNLimitPercentURL(cfg *WAF) {
|
||||
if slices.Contains(cfg.BlocklistURLs, cfg.ASNLimitPercentURL) {
|
||||
e.check("SWW_WAF_ASN_LIMIT_PERCENT_URL",
|
||||
e.check(ASNLimitPercentURLSetting,
|
||||
fmt.Errorf("%q %w", cfg.ASNLimitPercentURL, errInBlocklistURLs))
|
||||
}
|
||||
}
|
||||
@@ -1172,7 +1306,7 @@ func (e *environment) checkTags(cfg *WAF) {
|
||||
}
|
||||
|
||||
for tag := range cfg.TagLimitPercent {
|
||||
e.check("SWW_WAF_TAG_LIMIT_PERCENT", checkTag(cfg, tag))
|
||||
e.check(TagLimitPercentSetting, checkTag(cfg, tag))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1549,9 +1683,15 @@ func (e *environment) secret(name string) string {
|
||||
return value
|
||||
}
|
||||
|
||||
// alertEvents reads the setting that is the events alerts are sent for.
|
||||
func (e *environment) alertEvents(name, defaultValue string) []string {
|
||||
events, err := parseAlertEvents(e.value(name, defaultValue))
|
||||
// alertEvents reads the setting that is the events alerts are sent for,
|
||||
// every event by default.
|
||||
func (e *environment) alertEvents(name string) []alerts.Event {
|
||||
every := make([]string, 0, len(alerts.Events()))
|
||||
for _, event := range alerts.Events() {
|
||||
every = append(every, string(event))
|
||||
}
|
||||
|
||||
events, err := parseAlertEvents(e.value(name, strings.Join(every, ",")))
|
||||
e.check(name, err)
|
||||
|
||||
return events
|
||||
@@ -2470,16 +2610,20 @@ func parseWebhookHeaders(value string) (http.Header, string, error) {
|
||||
|
||||
// parseAlertEvents reads a comma-separated list of the events alerts can
|
||||
// be sent for.
|
||||
func parseAlertEvents(value string) ([]string, error) {
|
||||
events, err := parseList(value)
|
||||
func parseAlertEvents(value string) ([]alerts.Event, error) {
|
||||
names, err := parseList(value)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
for _, event := range events {
|
||||
events := make([]alerts.Event, 0, len(names))
|
||||
for _, name := range names {
|
||||
event := alerts.Event(name)
|
||||
if !slices.Contains(alerts.Events(), event) {
|
||||
return nil, fmt.Errorf("%q %w", event, errNotAlertEvent)
|
||||
return nil, fmt.Errorf("%q %w", name, errNotAlertEvent)
|
||||
}
|
||||
|
||||
events = append(events, event)
|
||||
}
|
||||
|
||||
return events, nil
|
||||
|
||||
+100
-10
@@ -19,6 +19,7 @@ import (
|
||||
|
||||
"github.com/spf13/viper"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
|
||||
@@ -61,6 +62,9 @@ const (
|
||||
abuseIPDBKey = "SWW_WAF_ABUSEIPDB_KEY"
|
||||
abuseIPDBMinScore = "SWW_WAF_ABUSEIPDB_MIN_SCORE"
|
||||
abuseIPDBDailyBudget = "SWW_WAF_ABUSEIPDB_DAILY_BUDGET"
|
||||
abuseIPDBThreshold = "SWW_WAF_ABUSEIPDB_THRESHOLD"
|
||||
abuseIPDBBlacklistRefresh = "SWW_WAF_ABUSEIPDB_BLACKLIST_REFRESH_INTERVAL_MINS"
|
||||
abuseIPDBBanDays = "SWW_WAF_ABUSEIPDB_BAN_DAYS"
|
||||
dnsblAbuseIPDBAction = "SWW_WAF_DNSBL_ABUSEIPDB_ACTION"
|
||||
ipReputationCacheTTL = "SWW_WAF_IP_REPUTATION_CACHE_TTL"
|
||||
dnsblAbuseIPDBTimeout = "SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS"
|
||||
@@ -183,6 +187,16 @@ const (
|
||||
defaultAlertCooldown = "15m"
|
||||
)
|
||||
|
||||
// joinEvents returns events as SWW_WAF_ALERT_EVENTS lists them.
|
||||
func joinEvents(events []alerts.Event) string {
|
||||
names := make([]string, 0, len(events))
|
||||
for _, event := range events {
|
||||
names = append(names, string(event))
|
||||
}
|
||||
|
||||
return strings.Join(names, ",")
|
||||
}
|
||||
|
||||
// defaultIPReputationCacheTTL is the default of SWW_WAF_IP_REPUTATION_CACHE_TTL.
|
||||
const defaultIPReputationCacheTTL = "24h"
|
||||
|
||||
@@ -286,7 +300,7 @@ func TestDefaults(t *testing.T) {
|
||||
RuleFilesEnabled: true,
|
||||
})
|
||||
wantLookupSettings(t, cfg, config.WAF{
|
||||
LookupSource: defaultLookupSource, ASNAndCountryLookupTimeout: 500 * time.Millisecond,
|
||||
LookupSource: defaultLookupSource, ASNAndCountryLookupTimeout: 800 * time.Millisecond,
|
||||
AddLookupHeaders: true,
|
||||
})
|
||||
wantByteLimitSettings(t, cfg, config.WAF{
|
||||
@@ -611,7 +625,7 @@ func TestCoreRuleSetSettings(t *testing.T) {
|
||||
},
|
||||
{
|
||||
environment{
|
||||
coreRuleSetMode: config.CoreRuleSetModeDetect,
|
||||
coreRuleSetMode: string(config.CoreRuleSetModeDetect),
|
||||
coreRuleSetParanoiaLevel: "4",
|
||||
coreRuleSetScoreThreshold: "10",
|
||||
coreRuleSetDisabledRules: "942100, 920350",
|
||||
@@ -1079,7 +1093,7 @@ func TestAlertSettingsDefaults(t *testing.T) {
|
||||
if cfg.AlertWebhookURL != nil || len(cfg.AlertWebhookHeaders) != 0 ||
|
||||
cfg.AlertSlackWebhookURL != nil || cfg.AlertNtfyURL != nil ||
|
||||
cfg.AlertNtfyToken != "" ||
|
||||
strings.Join(cfg.AlertEvents, ",") != defaultAlertEvents ||
|
||||
joinEvents(cfg.AlertEvents) != defaultAlertEvents ||
|
||||
cfg.AlertCooldown != 15*time.Minute || cfg.AlertMaxPerHour != 60 {
|
||||
t.Errorf(
|
||||
"alert settings %v, %v, %v, %v, %q, %v, %s and %d, want no URLs, "+
|
||||
@@ -1126,7 +1140,10 @@ func TestAlertSettingsAsSet(t *testing.T) {
|
||||
!reflect.DeepEqual(cfg.AlertWebhookHeaders, headers) ||
|
||||
cfg.AlertSlackWebhookURL.String() != slack || cfg.AlertNtfyURL.String() != ntfy ||
|
||||
cfg.AlertNtfyToken != token ||
|
||||
!slices.Equal(cfg.AlertEvents, []string{"ban", "file_error"}) ||
|
||||
!slices.Equal(
|
||||
cfg.AlertEvents,
|
||||
[]alerts.Event{"ban", "file_error"},
|
||||
) ||
|
||||
cfg.AlertCooldown != time.Hour || cfg.AlertMaxPerHour != 10 {
|
||||
t.Errorf(
|
||||
"alert settings %v, %v, %v, %v, %q, %v, %s and %d",
|
||||
@@ -1407,7 +1424,7 @@ func TestLookupSettingsAsSet(t *testing.T) {
|
||||
// With no client looked up, the headers are off unless set.
|
||||
cfg = wafFromEnvironment(t, environment{lookupSource: off})
|
||||
wantLookupSettings(t, cfg, config.WAF{
|
||||
LookupSource: off, ASNAndCountryLookupTimeout: 500 * time.Millisecond,
|
||||
LookupSource: off, ASNAndCountryLookupTimeout: 800 * time.Millisecond,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -2030,8 +2047,9 @@ func TestReputationSettingsAsSet(t *testing.T) {
|
||||
}
|
||||
|
||||
for _, tc := range []struct {
|
||||
value, action string
|
||||
percent int64
|
||||
value string
|
||||
action config.ReputationAction
|
||||
percent int64
|
||||
}{
|
||||
{actionDeny, actionDeny, 0},
|
||||
{actionLog, actionLog, 0},
|
||||
@@ -2164,7 +2182,7 @@ func TestDNSBLSettingsAsSet(t *testing.T) {
|
||||
// want are the zones, resolver, action and percent config.WAF gives.
|
||||
want []string
|
||||
wantResolver string
|
||||
wantAction string
|
||||
wantAction config.ReputationAction
|
||||
wantPercent int64
|
||||
}{
|
||||
{
|
||||
@@ -2399,6 +2417,75 @@ func TestAbuseIPDBSettingsAsSet(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAbuseIPDBBlacklistSettingsAsSet(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg := wafFromEnvironment(t, environment{})
|
||||
if cfg.AbuseIPDBThreshold != 90 ||
|
||||
cfg.AbuseIPDBBlacklistRefresh != time.Hour ||
|
||||
cfg.AbuseIPDBBanDuration != 30*24*time.Hour {
|
||||
t.Errorf(
|
||||
"by default, the threshold %d, the refresh %s and the ban %s, "+
|
||||
"want 90, 1h and 720h",
|
||||
cfg.AbuseIPDBThreshold,
|
||||
cfg.AbuseIPDBBlacklistRefresh,
|
||||
cfg.AbuseIPDBBanDuration,
|
||||
)
|
||||
}
|
||||
|
||||
cfg = wafFromEnvironment(t, environment{
|
||||
abuseIPDBThreshold: "25", abuseIPDBBlacklistRefresh: "1440",
|
||||
abuseIPDBBanDays: "1",
|
||||
})
|
||||
if cfg.AbuseIPDBThreshold != 25 ||
|
||||
cfg.AbuseIPDBBlacklistRefresh != 24*time.Hour ||
|
||||
cfg.AbuseIPDBBanDuration != 24*time.Hour {
|
||||
t.Errorf(
|
||||
"set, the threshold %d, the refresh %s and the ban %s, "+
|
||||
"want 25, 24h and 24h",
|
||||
cfg.AbuseIPDBThreshold,
|
||||
cfg.AbuseIPDBBlacklistRefresh,
|
||||
cfg.AbuseIPDBBanDuration,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInvalidAbuseIPDBBlacklistSettingStopsTheStartSayingWhatIsWrong(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
notThreshold = " is not a whole number from 25 to 100, the scores " +
|
||||
"AbuseIPDB's blacklist can be cut at"
|
||||
notWhole = " is not a whole number above zero, such as 30, that fits " +
|
||||
"a duration"
|
||||
)
|
||||
|
||||
for _, tc := range []struct{ name, value, want string }{
|
||||
{abuseIPDBThreshold, "24", `"24"` + notThreshold},
|
||||
{abuseIPDBThreshold, "101", `"101"` + notThreshold},
|
||||
{abuseIPDBThreshold, off, `"off"` + notThreshold},
|
||||
{abuseIPDBBlacklistRefresh, "0", `"0"` + notWhole},
|
||||
{abuseIPDBBlacklistRefresh, "1h", `"1h"` + notWhole},
|
||||
{abuseIPDBBanDays, "-1", `"-1"` + notWhole},
|
||||
{abuseIPDBBanDays, "200000", `"200000"` + notWhole},
|
||||
} {
|
||||
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := config.WAFFromEnvironment(
|
||||
environment{tc.name: tc.value}.viper(),
|
||||
)
|
||||
|
||||
want := tc.name + ": " + tc.want
|
||||
if err == nil || err.Error() != want {
|
||||
t.Errorf("error %v, want %s", err, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestInvalidAbuseIPDBSettingStopsTheStartSayingWhatIsWrong(
|
||||
t *testing.T,
|
||||
) {
|
||||
@@ -2933,7 +3020,7 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
||||
bytesCounted: "both",
|
||||
lookupSource: defaultLookupSource,
|
||||
lookupDBPath: "",
|
||||
asnAndCountryLookupTimeout: "500",
|
||||
asnAndCountryLookupTimeout: "800",
|
||||
addLookupHeaders: enabled,
|
||||
allowedCountries: "",
|
||||
tagLists: "",
|
||||
@@ -2951,6 +3038,9 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
||||
abuseIPDBKey: "",
|
||||
abuseIPDBMinScore: "75",
|
||||
abuseIPDBDailyBudget: "900",
|
||||
abuseIPDBThreshold: "90",
|
||||
abuseIPDBBlacklistRefresh: "60",
|
||||
abuseIPDBBanDays: "30",
|
||||
dnsblAbuseIPDBAction: "limit:25",
|
||||
ipReputationCacheTTL: defaultIPReputationCacheTTL,
|
||||
dnsblAbuseIPDBTimeout: "2000",
|
||||
@@ -2974,7 +3064,7 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
||||
logLevel: "info",
|
||||
ruleFilesDir: "/etc/sww/rules.d",
|
||||
ruleFilesEnabled: "true",
|
||||
coreRuleSetMode: config.CoreRuleSetModeBlock,
|
||||
coreRuleSetMode: string(config.CoreRuleSetModeBlock),
|
||||
coreRuleSetParanoiaLevel: "1",
|
||||
coreRuleSetScoreThreshold: "5",
|
||||
coreRuleSetDisabledRules: defaultCoreRuleSetDisabledRules,
|
||||
|
||||
@@ -35,20 +35,21 @@ const (
|
||||
mostJSONRead = 1 << 10
|
||||
)
|
||||
|
||||
// The Core Rule Set has 1.3 seconds for one request. That is the default of
|
||||
// rproxy's SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS, 2 seconds, less the default of
|
||||
// SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS, 500 milliseconds, and less 200
|
||||
// milliseconds for waf's other checks.
|
||||
// waf refuses a request the Core Rule Set has not inspected within 500
|
||||
// milliseconds, but Coraza cannot stop an inspection part-way, so it runs to
|
||||
// its end with nobody waiting for it. The test checks that an inspection ends
|
||||
// within about 1 second, so that a visitor cannot keep a processor busy for
|
||||
// longer with one request.
|
||||
//
|
||||
// A busy host slows all work, so the test does not compare an inspection with
|
||||
// 1.3 seconds. It compares it with referenceTime, timed on the same thread
|
||||
// just before and after. inspectionTimeLimitMultiple is how many times
|
||||
// 1 second. It compares it with referenceTime, timed on the same thread just
|
||||
// before and after. inspectionTimeLimitMultiple is how many times
|
||||
// referenceTime an inspection may take.
|
||||
//
|
||||
// On the host that runs our checks, referenceTime took at least 78
|
||||
// milliseconds, so 16 times it is about 1.25 seconds. With that host fully
|
||||
// busy, no try of the slowest request took more than 6.7 times referenceTime.
|
||||
const inspectionTimeLimitMultiple = 16
|
||||
// milliseconds, so 13 times it is about 1 second. With that host fully busy,
|
||||
// no try of the slowest request took more than 9.5 times referenceTime.
|
||||
const inspectionTimeLimitMultiple = 13
|
||||
|
||||
// referencePattern is what referenceTime matches. Like the Core Rule Set's
|
||||
// rules, it is a regular expression with alternatives and repeats.
|
||||
@@ -221,22 +222,20 @@ const fullLine = lineLimit - 64
|
||||
|
||||
// worstCaseRequests are, for each kind of thing a visitor sends, the requests
|
||||
// that keep the Core Rule Set busiest, within the bounds above. The last
|
||||
// holds the most of everything at once. waf refuses a second Cookie line, so
|
||||
// it has one, and User-Agent and Referer lines, which keep the Core Rule Set
|
||||
// busier than other header lines. The JSON body of the most elements under a
|
||||
// key keeps it busier than the other bodies.
|
||||
// holds the most of everything at once: Cookie lines keep it busier than
|
||||
// other header lines, and the JSON body of the most elements under a key
|
||||
// keeps it busier than the other bodies.
|
||||
func worstCaseRequests() []visitorRequest {
|
||||
fullCookieLine := cookieLine(1)[:fullLine-len("Cookie: ")]
|
||||
|
||||
return append(
|
||||
append(worstCaseHeaders(), worstCaseBodies()...),
|
||||
visitorRequest{
|
||||
name: "the longest request line, Cookie, User-Agent and " +
|
||||
"Referer lines and body",
|
||||
name: "the longest request line, three Cookie lines and body",
|
||||
target: longestTarget()[:fullLine-len("POST HTTP/1.1")],
|
||||
header: http.Header{
|
||||
cookieHeader: {cookieLine(1)[:fullLine-len("Cookie: ")]},
|
||||
"User-Agent": {headerValue("User-Agent", fullLine)},
|
||||
"Referer": {headerValue("Referer", fullLine)},
|
||||
},
|
||||
header: http.Header{cookieHeader: {
|
||||
fullCookieLine, fullCookieLine, fullCookieLine,
|
||||
}},
|
||||
contentType: jsonType, body: jsonOfMostElementsUnderAKey(),
|
||||
},
|
||||
)
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
package lookup
|
||||
package geojs
|
||||
|
||||
import "net/http"
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
// Package lookup looks up each client's AS number and country, through
|
||||
// the GeoJS web service or in the lookup database, the IPinfo Lite file
|
||||
// SWW_WAF_LOOKUP_DB_PATH names. GeoJS's answers are kept in memory, for at
|
||||
// most 100,000 clients and for 7 days each, and are written to
|
||||
// lookups.json and read from it by the state package.
|
||||
package lookup
|
||||
// Package geojs looks up clients' AS numbers and countries through the GeoJS
|
||||
// web service. Its answers are kept in memory, for at most 100,000 clients and
|
||||
// for 7 days each. A caller waits for an answer as long as it says. A request to
|
||||
// GeoJS goes on after its callers stop waiting, and its answer is kept. The
|
||||
// package imports no other package of this repository.
|
||||
package geojs
|
||||
|
||||
import (
|
||||
"context"
|
||||
@@ -11,7 +11,6 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"slices"
|
||||
@@ -21,8 +20,6 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/hashicorp/golang-lru/v2/simplelru"
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||
)
|
||||
|
||||
// URL is GeoJS's endpoint for an address's place and network. Asked about
|
||||
@@ -43,6 +40,9 @@ const (
|
||||
maxWaiting = 10000
|
||||
// maxPerRequest is how many addresses one request to GeoJS asks about.
|
||||
maxPerRequest = 200
|
||||
// requestTimeout is how long a request to GeoJS may take before it is
|
||||
// abandoned, which is a failure. It is apart from how long callers wait.
|
||||
requestTimeout = 10 * time.Second
|
||||
// unknownASN is the AS number GeoJS gives when it knows none.
|
||||
unknownASN = 64512
|
||||
// After a failure GeoJS is not asked again for a second, and for
|
||||
@@ -64,39 +64,32 @@ var (
|
||||
type Params struct {
|
||||
// URL is where GeoJS is asked, normally URL.
|
||||
URL string
|
||||
// Timeout is how long a request waits for its client's first answer,
|
||||
// and how long a request to GeoJS may take before it is abandoned
|
||||
// (SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS).
|
||||
Timeout time.Duration
|
||||
// Wait is true when a setting needs each request's answer before the
|
||||
// request goes on. Otherwise no request waits for one.
|
||||
Wait bool
|
||||
// Answered, unless nil, is given each answer GeoJS gives, once it is
|
||||
// kept.
|
||||
Answered func(Answer)
|
||||
// Now tells the time, normally time.Now in UTC.
|
||||
Now func() time.Time
|
||||
// ProcessLog receives GeoJS's failures.
|
||||
ProcessLog *slog.Logger
|
||||
// Metrics count the requests to GeoJS, those that failed, and the
|
||||
// clients that go without an answer.
|
||||
Metrics *metrics.Metrics
|
||||
// Alerts receive a source_failure alert each time GeoJS fails.
|
||||
Alerts *alerts.Queue
|
||||
// Asked is called for each request to GeoJS.
|
||||
Asked func()
|
||||
// Failed is called for each failure of GeoJS, with what went wrong and
|
||||
// how long GeoJS is left alone after it. The error never names the
|
||||
// addresses asked about.
|
||||
Failed func(err error, askingAgainIn time.Duration)
|
||||
// WentWithoutAnswer is called each time a caller that waits for an
|
||||
// answer goes on without one.
|
||||
WentWithoutAnswer func()
|
||||
}
|
||||
|
||||
// GeoJS looks up clients' AS numbers and countries through GeoJS. At most
|
||||
// one request to GeoJS is under way at a time, and it asks about every
|
||||
// client waiting, up to maxPerRequest. It is safe for concurrent use.
|
||||
type GeoJS struct {
|
||||
url string
|
||||
timeout time.Duration
|
||||
wait bool
|
||||
answered func(Answer)
|
||||
now func() time.Time
|
||||
processLog *slog.Logger
|
||||
metrics *metrics.Metrics
|
||||
alerts *alerts.Queue
|
||||
url string
|
||||
answered func(Answer)
|
||||
now func() time.Time
|
||||
asked func()
|
||||
failed func(err error, askingAgainIn time.Duration)
|
||||
wentWithoutAnswer func()
|
||||
// httpClient follows no redirect, so that visitors' addresses go to
|
||||
// GeoJS alone: a redirect is a failure.
|
||||
httpClient *http.Client
|
||||
@@ -106,20 +99,23 @@ type GeoJS struct {
|
||||
// waiting are the clients without an answer: those to ask GeoJS about,
|
||||
// and those it is being asked about.
|
||||
waiting map[netip.Prefix]*wait
|
||||
// asking is true while a request to GeoJS is under way.
|
||||
asking bool
|
||||
// asking is true while a request to GeoJS is under way, and endAsking
|
||||
// ends it. stopped is true once StopAsking has been called, and
|
||||
// askGoroutines waits for the goroutine that makes the requests.
|
||||
asking bool
|
||||
endAsking context.CancelFunc
|
||||
stopped bool
|
||||
askGoroutines sync.WaitGroup
|
||||
// retryDelay is how long GeoJS is left alone after its last failure,
|
||||
// zero after an answer; retryAt is when it may be asked again.
|
||||
retryDelay time.Duration
|
||||
retryAt time.Time
|
||||
}
|
||||
|
||||
// Answer is what GeoJS or the lookup database said about a client: its AS
|
||||
// number, such as AS64496, and the AS's name, both "" when the source knows
|
||||
// no AS number for it; its country, "" when the source cannot place it;
|
||||
// when the source said so; and, for GeoJS's answers, which lookups.json
|
||||
// holds, when the answer was last used. The zero Answer is that of a
|
||||
// client with no answer.
|
||||
// Answer is what GeoJS said about a client: its AS number, such as AS64496,
|
||||
// and the AS's name, both "" when GeoJS knows no AS number for it; its
|
||||
// country, "" when GeoJS cannot place it; when GeoJS said so; and when the
|
||||
// answer was last used. The zero Answer is that of a client with no answer.
|
||||
//
|
||||
//nolint:tagliatelle // the state files use snake_case, as the request log does
|
||||
type Answer struct {
|
||||
@@ -137,7 +133,7 @@ type wait struct {
|
||||
// replaced each time GeoJS fails before then.
|
||||
asked chan struct{}
|
||||
// late is true once the client has gone without an answer, for a
|
||||
// whole timeout or because GeoJS failed: its requests no longer wait.
|
||||
// whole wait or because GeoJS failed: its callers no longer wait.
|
||||
late bool
|
||||
}
|
||||
|
||||
@@ -149,14 +145,12 @@ func New(params Params) *GeoJS {
|
||||
}
|
||||
|
||||
return &GeoJS{
|
||||
url: params.URL,
|
||||
timeout: params.Timeout,
|
||||
wait: params.Wait,
|
||||
answered: params.Answered,
|
||||
now: params.Now,
|
||||
processLog: params.ProcessLog,
|
||||
metrics: params.Metrics,
|
||||
alerts: params.Alerts,
|
||||
url: params.URL,
|
||||
answered: params.Answered,
|
||||
now: params.Now,
|
||||
asked: params.Asked,
|
||||
failed: params.Failed,
|
||||
wentWithoutAnswer: params.WentWithoutAnswer,
|
||||
httpClient: &http.Client{
|
||||
CheckRedirect: func(*http.Request, []*http.Request) error {
|
||||
return http.ErrUseLastResponse
|
||||
@@ -170,21 +164,22 @@ func New(params Params) *GeoJS {
|
||||
// LookUp returns the answer GeoJS gave about client, with its country as
|
||||
// a two-letter code in capitals, or the zero Answer when there is none
|
||||
// yet. An answer is kept for 7 days. Without one, the client is asked
|
||||
// about in the background, and, while Wait is set, the request waits up
|
||||
// to Timeout for the answer, unless the client has gone without one
|
||||
// before. ctx is the context of the client's request, and ends the wait
|
||||
// when it ends.
|
||||
// about in the background, and LookUp waits up to wait for the answer,
|
||||
// unless the client has gone without one before. The wait ends early when
|
||||
// ctx ends. Neither ends the request to GeoJS.
|
||||
//
|
||||
// GeoJS is asked about the client's first address, which is the client's
|
||||
// own address for IPv4, and an address in the same place for an IPv6
|
||||
// netblock.
|
||||
func (g *GeoJS) LookUp(ctx context.Context, client netip.Prefix) Answer {
|
||||
answer, asked := g.answerOrWait(ctx, client, g.wait)
|
||||
func (g *GeoJS) LookUp(
|
||||
ctx context.Context, client netip.Prefix, wait time.Duration,
|
||||
) Answer {
|
||||
answer, asked := g.answerOrWait(ctx, client, true)
|
||||
if asked == nil {
|
||||
return answer
|
||||
}
|
||||
|
||||
timer := time.NewTimer(g.timeout)
|
||||
timer := time.NewTimer(wait)
|
||||
defer timer.Stop()
|
||||
|
||||
select {
|
||||
@@ -198,7 +193,7 @@ func (g *GeoJS) LookUp(ctx context.Context, client netip.Prefix) Answer {
|
||||
|
||||
answer, found := g.kept(client)
|
||||
if !found {
|
||||
g.metrics.RequestsWithoutGeoJSAnswer.Inc()
|
||||
g.wentWithoutAnswer()
|
||||
}
|
||||
|
||||
w, waiting := g.waiting[client]
|
||||
@@ -209,8 +204,8 @@ func (g *GeoJS) LookUp(ctx context.Context, client netip.Prefix) Answer {
|
||||
return answer
|
||||
}
|
||||
|
||||
// LookUpWithoutWaiting is LookUp for a request that does not need the answer,
|
||||
// whatever Wait says: it returns the zero Answer at once when none is kept.
|
||||
// LookUpWithoutWaiting is LookUp for a caller that does not need the answer:
|
||||
// it returns the zero Answer at once when none is kept.
|
||||
func (g *GeoJS) LookUpWithoutWaiting(
|
||||
ctx context.Context,
|
||||
client netip.Prefix,
|
||||
@@ -220,6 +215,23 @@ func (g *GeoJS) LookUpWithoutWaiting(
|
||||
return answer
|
||||
}
|
||||
|
||||
// StopAsking ends the request to GeoJS under way, and returns once the
|
||||
// goroutine that made it has. A request it ends is no failure: Failed is not
|
||||
// called, and GeoJS is not left alone after it. After StopAsking, GeoJS is
|
||||
// asked nothing.
|
||||
func (g *GeoJS) StopAsking() {
|
||||
g.mu.Lock()
|
||||
|
||||
g.stopped = true
|
||||
if g.asking {
|
||||
g.endAsking()
|
||||
}
|
||||
|
||||
g.mu.Unlock()
|
||||
|
||||
g.askGoroutines.Wait()
|
||||
}
|
||||
|
||||
// Kept returns client's answer, if one is kept, without asking GeoJS.
|
||||
func (g *GeoJS) Kept(client netip.Prefix) (Answer, bool) {
|
||||
g.mu.Lock()
|
||||
@@ -228,8 +240,7 @@ func (g *GeoJS) Kept(client netip.Prefix) (Answer, bool) {
|
||||
return g.kept(client)
|
||||
}
|
||||
|
||||
// Snapshot returns every answer kept, sorted by client, as lookups.json
|
||||
// lists them.
|
||||
// Snapshot returns every answer kept, sorted by client.
|
||||
func (g *GeoJS) Snapshot() []Answer {
|
||||
g.mu.Lock()
|
||||
|
||||
@@ -247,9 +258,9 @@ func (g *GeoJS) Snapshot() []Answer {
|
||||
return answers
|
||||
}
|
||||
|
||||
// Load keeps answers read from lookups.json, in place of the answers it
|
||||
// keeps, in the order they were last used, so that the one used longest
|
||||
// ago is dropped first. Answers GeoJS gave keepFor ago or more are
|
||||
// Load keeps answers, such as those of an earlier Snapshot, in place of the
|
||||
// answers it keeps, in the order they were last used, so that the one used
|
||||
// longest ago is dropped first. Answers GeoJS gave keepFor ago or more are
|
||||
// dropped.
|
||||
func (g *GeoJS) Load(answers []Answer) {
|
||||
answers = slices.Clone(answers)
|
||||
@@ -295,22 +306,23 @@ func (g *GeoJS) answerOrWait(
|
||||
g.ask(ctx)
|
||||
|
||||
if !mustWait {
|
||||
return Answer{}, nil // the answer is not needed before the request goes on
|
||||
return Answer{}, nil // the answer is not needed before the caller goes on
|
||||
}
|
||||
|
||||
if w == nil {
|
||||
g.metrics.RequestsWithoutGeoJSAnswer.Inc()
|
||||
g.wentWithoutAnswer()
|
||||
|
||||
return Answer{}, nil // too many clients wait already
|
||||
}
|
||||
|
||||
if !g.asking {
|
||||
// GeoJS is left alone after a failure, so no answer can come.
|
||||
// GeoJS is left alone after a failure, or asked nothing after
|
||||
// StopAsking, so no answer can come.
|
||||
w.late = true
|
||||
}
|
||||
|
||||
if w.late {
|
||||
g.metrics.RequestsWithoutGeoJSAnswer.Inc()
|
||||
g.wentWithoutAnswer()
|
||||
|
||||
return Answer{}, nil
|
||||
}
|
||||
@@ -334,22 +346,28 @@ func (g *GeoJS) kept(client netip.Prefix) (Answer, bool) {
|
||||
}
|
||||
|
||||
// ask starts asking GeoJS about the waiting clients, unless a request to
|
||||
// it is under way or it is left alone after a failure. The requests to
|
||||
// GeoJS are for every client waiting, so they go on when the client's
|
||||
// request whose ctx is given ends.
|
||||
// it is under way, it is left alone after a failure, or StopAsking has
|
||||
// been called. The requests to GeoJS are for every client waiting, so they
|
||||
// go on when the caller whose ctx is given stops waiting, until StopAsking.
|
||||
func (g *GeoJS) ask(ctx context.Context) {
|
||||
if g.asking || g.now().Before(g.retryAt) {
|
||||
if g.asking || g.stopped || g.now().Before(g.retryAt) {
|
||||
return
|
||||
}
|
||||
|
||||
g.asking = true
|
||||
askCtx, end := context.WithCancel(context.WithoutCancel(ctx))
|
||||
g.asking, g.endAsking = true, end
|
||||
|
||||
go g.askAboutWaiting(context.WithoutCancel(ctx))
|
||||
g.askGoroutines.Go(func() {
|
||||
defer end()
|
||||
|
||||
g.askAboutWaiting(askCtx)
|
||||
})
|
||||
}
|
||||
|
||||
// askAboutWaiting asks GeoJS about the waiting clients, one request at a
|
||||
// time, until none is left or GeoJS fails. Each answer kept is given to
|
||||
// Answered, outside the lock, since Answered takes locks of its own.
|
||||
// time, until none is left, GeoJS fails, or StopAsking ends ctx. Each
|
||||
// answer kept is given to Answered, outside the lock, since Answered takes
|
||||
// locks of its own.
|
||||
func (g *GeoJS) askAboutWaiting(ctx context.Context) {
|
||||
for {
|
||||
clients := g.nextClients()
|
||||
@@ -358,6 +376,10 @@ func (g *GeoJS) askAboutWaiting(ctx context.Context) {
|
||||
}
|
||||
|
||||
given, err := g.request(ctx, clients)
|
||||
if err != nil && ctx.Err() != nil {
|
||||
continue // ended by StopAsking, which is no failure
|
||||
}
|
||||
|
||||
kept, answered := g.keep(clients, given, err)
|
||||
|
||||
if g.answered != nil {
|
||||
@@ -373,13 +395,13 @@ func (g *GeoJS) askAboutWaiting(ctx context.Context) {
|
||||
}
|
||||
|
||||
// nextClients returns up to maxPerRequest of the waiting clients. When
|
||||
// none is waiting, it returns none and notes that no request to GeoJS is
|
||||
// under way.
|
||||
// none is waiting, or StopAsking has been called, it returns none and
|
||||
// notes that no request to GeoJS is under way.
|
||||
func (g *GeoJS) nextClients() []netip.Prefix {
|
||||
g.mu.Lock()
|
||||
defer g.mu.Unlock()
|
||||
|
||||
if len(g.waiting) == 0 {
|
||||
if len(g.waiting) == 0 || g.stopped {
|
||||
g.asking = false
|
||||
|
||||
return nil
|
||||
@@ -441,8 +463,6 @@ func (g *GeoJS) keep(
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
g.metrics.GeoJSFailures.Inc()
|
||||
|
||||
g.retryDelay = min(
|
||||
max(retryDelayFactor*g.retryDelay, firstRetryDelay),
|
||||
maxRetryDelay,
|
||||
@@ -457,16 +477,7 @@ func (g *GeoJS) keep(
|
||||
w.late = true
|
||||
}
|
||||
|
||||
g.processLog.Warn("asking GeoJS failed",
|
||||
"error", err.Error(), "asking_again_in", g.retryDelay.String())
|
||||
g.alerts.Raise(alerts.Alert{
|
||||
Event: alerts.EventSourceFailure,
|
||||
Reason: "asking GeoJS failed",
|
||||
Detail: map[string]any{
|
||||
"source": "geojs", "error": err.Error(),
|
||||
"asking_again_in": g.retryDelay.String(),
|
||||
},
|
||||
})
|
||||
g.failed(err, g.retryDelay)
|
||||
|
||||
return kept, false
|
||||
}
|
||||
@@ -479,7 +490,8 @@ func (g *GeoJS) keep(
|
||||
// request asks GeoJS about clients in one request, and returns the answer
|
||||
// for each address GeoJS's answer names: its AS number and the AS's name,
|
||||
// both "" for the AS number 64512, which GeoJS gives when it knows none,
|
||||
// and its country, in capitals.
|
||||
// and its country, in capitals. The request is abandoned after
|
||||
// requestTimeout.
|
||||
func (g *GeoJS) request(
|
||||
ctx context.Context, clients []netip.Prefix,
|
||||
) (map[netip.Addr]Answer, error) {
|
||||
@@ -489,7 +501,7 @@ func (g *GeoJS) request(
|
||||
addrs = append(addrs, client.Addr().String())
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(ctx, g.timeout)
|
||||
ctx, cancel := context.WithTimeout(ctx, requestTimeout)
|
||||
defer cancel()
|
||||
|
||||
req, err := http.NewRequestWithContext(
|
||||
@@ -504,7 +516,7 @@ func (g *GeoJS) request(
|
||||
|
||||
req.URL.RawQuery = "ip=" + strings.Join(addrs, ",")
|
||||
|
||||
g.metrics.GeoJSRequests.Inc()
|
||||
g.asked()
|
||||
|
||||
res, err := g.httpClient.Do(req)
|
||||
if err != nil {
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,4 +1,4 @@
|
||||
package lookup_test
|
||||
package geojs_test
|
||||
|
||||
import (
|
||||
"net/netip"
|
||||
@@ -7,7 +7,7 @@ import (
|
||||
"testing/synctest"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||
)
|
||||
|
||||
func TestSnapshotHoldsEachAnswerAndWhenItWasLastUsed(t *testing.T) {
|
||||
@@ -25,7 +25,7 @@ func TestSnapshotHoldsEachAnswerAndWhenItWasLastUsed(t *testing.T) {
|
||||
clock.advance(time.Hour)
|
||||
wantCountry(t, g, placed, germany)
|
||||
|
||||
want := []lookup.Answer{
|
||||
want := []geojs.Answer{
|
||||
{Client: notPlaced, Answered: asked, Used: asked},
|
||||
{
|
||||
Client: placed, ASN: asn, ASName: asName, Country: germany,
|
||||
@@ -41,24 +41,24 @@ func TestSnapshotHoldsEachAnswerAndWhenItWasLastUsed(t *testing.T) {
|
||||
func TestLoadedAnswersAreKeptFor7DaysFromWhenGeoJSGaveThem(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
geojs, clock, g := start()
|
||||
service, clock, g := start()
|
||||
now := clock.Now()
|
||||
kept := lookup.Answer{
|
||||
kept := geojs.Answer{
|
||||
Client: netip.MustParsePrefix("203.0.113.9/32"),
|
||||
Country: "FR",
|
||||
Answered: now.Add(-week + time.Second),
|
||||
Used: now.Add(-time.Hour),
|
||||
}
|
||||
stale := lookup.Answer{
|
||||
stale := geojs.Answer{
|
||||
Client: netip.MustParsePrefix("203.0.113.10/32"),
|
||||
Country: "FR",
|
||||
Answered: now.Add(-week),
|
||||
Used: now.Add(-time.Hour),
|
||||
}
|
||||
|
||||
g.Load([]lookup.Answer{kept, stale})
|
||||
g.Load([]geojs.Answer{kept, stale})
|
||||
|
||||
if got := g.Snapshot(); !slices.Equal(got, []lookup.Answer{kept}) {
|
||||
if got := g.Snapshot(); !slices.Equal(got, []geojs.Answer{kept}) {
|
||||
t.Errorf(
|
||||
"kept %+v, want only the answer GeoJS gave less than 7 days ago",
|
||||
got,
|
||||
@@ -66,7 +66,7 @@ func TestLoadedAnswersAreKeptFor7DaysFromWhenGeoJSGaveThem(t *testing.T) {
|
||||
}
|
||||
|
||||
wantCountry(t, g, kept.Client, "FR")
|
||||
wantRequests(t, geojs, 0)
|
||||
wantRequests(t, service, 0)
|
||||
}
|
||||
|
||||
func TestLoadDropsTheAnswerUsedLongestAgoFirst(t *testing.T) {
|
||||
@@ -80,11 +80,11 @@ func TestLoadDropsTheAnswerUsedLongestAgoFirst(t *testing.T) {
|
||||
// lookups.json lists the answers by client. Here each was last used a
|
||||
// second before the one listed before it, so the last listed is the
|
||||
// one used longest ago, and the one dropped.
|
||||
answers := make([]lookup.Answer, maxAnswers+1)
|
||||
answers := make([]geojs.Answer, maxAnswers+1)
|
||||
addr := netip.MustParseAddr("10.0.0.0")
|
||||
|
||||
for i := range answers {
|
||||
answers[i] = lookup.Answer{
|
||||
answers[i] = geojs.Answer{
|
||||
Client: netip.PrefixFrom(addr, addr.BitLen()),
|
||||
Country: germany,
|
||||
Answered: now,
|
||||
@@ -1,3 +1,6 @@
|
||||
// Package lookup looks up each client's AS number and country in the lookup
|
||||
// database, the IPinfo Lite file SWW_WAF_LOOKUP_DB_PATH names. Its answers
|
||||
// take the form of GeoJS's, so that waf handles both alike.
|
||||
package lookup
|
||||
|
||||
import (
|
||||
@@ -14,6 +17,7 @@ import (
|
||||
"github.com/oschwald/maxminddb-golang/v2"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||
)
|
||||
|
||||
// quietTime is how long the lookup database must go without a change
|
||||
@@ -77,7 +81,7 @@ func OpenFile(params FileParams) (*File, error) {
|
||||
// each "" when the database does not give it, as for an address missing
|
||||
// from it. The database is asked about the client's first address, as
|
||||
// GeoJS is.
|
||||
func (f *File) LookUp(client netip.Prefix) Answer {
|
||||
func (f *File) LookUp(client netip.Prefix) geojs.Answer {
|
||||
f.mu.Lock()
|
||||
reader := f.reader
|
||||
f.mu.Unlock()
|
||||
@@ -91,7 +95,7 @@ func (f *File) LookUp(client netip.Prefix) Answer {
|
||||
found = record{}
|
||||
}
|
||||
|
||||
return Answer{
|
||||
return geojs.Answer{
|
||||
Client: client,
|
||||
ASN: found.ASN,
|
||||
ASName: found.ASName,
|
||||
|
||||
@@ -16,6 +16,7 @@ import (
|
||||
"github.com/maxmind/mmdbwriter/mmdbtype"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
|
||||
)
|
||||
|
||||
@@ -68,7 +69,7 @@ func TestFilePlacesClientsAndCountsAnAddressMissingFromItAsUnknown(
|
||||
prefix := netip.MustParsePrefix(client)
|
||||
|
||||
got := f.LookUp(prefix)
|
||||
if got != (Answer{
|
||||
if got != (geojs.Answer{
|
||||
Client: prefix, ASN: want.ASN, ASName: want.ASName, Country: want.Country,
|
||||
Answered: now,
|
||||
}) {
|
||||
|
||||
+12
-12
@@ -184,14 +184,14 @@ func New(busiestCount int, instanceName string) *Metrics {
|
||||
func (m *Metrics) AddBansAndClients(
|
||||
ledger *bans.Ledger, limiter *ratelimit.Limiter, now func() time.Time,
|
||||
) {
|
||||
for _, cause := range []string{
|
||||
bans.CauseLimit, bans.CauseAttack, bans.CauseAdmin,
|
||||
for _, cause := range []bans.Cause{
|
||||
bans.CauseLimit, bans.CauseAttack, bans.CauseAdmin, bans.CauseAbuseIPDB,
|
||||
} {
|
||||
m.registry.MustRegister(
|
||||
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
||||
Name: "sww_bans_made_total",
|
||||
Help: "Bans made, by cause.",
|
||||
ConstLabels: prometheus.Labels{"cause": cause},
|
||||
ConstLabels: prometheus.Labels{"cause": string(cause)},
|
||||
}, func() float64 {
|
||||
return float64(ledger.Made(cause))
|
||||
}),
|
||||
@@ -274,8 +274,7 @@ func (m *Metrics) AddRemoteLog(remote *remotelog.Sender) {
|
||||
// AddLookupFile adds the metrics of the lookup database, read as the
|
||||
// metrics are asked for: when the file in use was read, which lastRead
|
||||
// returns, and the replacements of it that could not be read, which
|
||||
// readFailures returns. The lookup package's File, which has both, cannot
|
||||
// be named here: that package counts GeoJS's requests in these metrics.
|
||||
// readFailures returns.
|
||||
func (m *Metrics) AddLookupFile(
|
||||
lastRead func() time.Time,
|
||||
readFailures func() int,
|
||||
@@ -391,7 +390,7 @@ func (m *Metrics) ReputationHit(source string) {
|
||||
// dropped. With no destination set, it adds none.
|
||||
func (m *Metrics) AddAlerts(queue *alerts.Queue) {
|
||||
for _, name := range queue.DestinationsSet() {
|
||||
destination := prometheus.Labels{"destination": name}
|
||||
destination := prometheus.Labels{"destination": string(name)}
|
||||
|
||||
m.registry.MustRegister(
|
||||
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
||||
@@ -466,7 +465,7 @@ func (m *Metrics) RequestReported(
|
||||
// LimitHit counts a request that broke a rate limit, a byte limit or the
|
||||
// refusal limit, by the window and the kind of hit.
|
||||
func (m *Metrics) LimitHit(hit ratelimit.Hit) {
|
||||
m.rateLimitHits.WithLabelValues(hit.Window, hit.Kind).Inc()
|
||||
m.rateLimitHits.WithLabelValues(hit.Window, string(hit.Kind)).Inc()
|
||||
}
|
||||
|
||||
// Offences counts the offences of r, a request that has ended, as its
|
||||
@@ -487,8 +486,8 @@ func (m *Metrics) Offences(r ratelimit.Request) {
|
||||
|
||||
// RuleMatched counts a request that matched the rule id, whose action is
|
||||
// action.
|
||||
func (m *Metrics) RuleMatched(id, action string) {
|
||||
m.ruleMatches.WithLabelValues(id, action).Inc()
|
||||
func (m *Metrics) RuleMatched(id string, action rules.Action) {
|
||||
m.ruleMatches.WithLabelValues(id, string(action)).Inc()
|
||||
}
|
||||
|
||||
// CoreRuleSetMatched counts a request that matched the Core Rule Set's rule id,
|
||||
@@ -534,10 +533,11 @@ func (m *Metrics) count(
|
||||
line *requestlog.Line, duration, upstreamDuration time.Duration,
|
||||
) {
|
||||
class := statusClass(line.Status)
|
||||
m.requests.WithLabelValues(class, line.Action).Inc()
|
||||
m.requestBytes.WithLabelValues(class, line.Action).
|
||||
action := string(line.Action)
|
||||
m.requests.WithLabelValues(class, action).Inc()
|
||||
m.requestBytes.WithLabelValues(class, action).
|
||||
Add(float64(line.RequestBytes))
|
||||
m.responseBytes.WithLabelValues(class, line.Action).
|
||||
m.responseBytes.WithLabelValues(class, action).
|
||||
Add(float64(line.ResponseBytes))
|
||||
m.requestDuration.Observe(duration.Seconds())
|
||||
|
||||
|
||||
@@ -19,15 +19,19 @@ import (
|
||||
|
||||
const day = 24 * time.Hour
|
||||
|
||||
// LimitKind is what a limit counts: KindRequests, KindBytes or
|
||||
// KindRefusals.
|
||||
type LimitKind string
|
||||
|
||||
// The kinds of limits, as the metrics name them.
|
||||
const (
|
||||
// KindRequests is a rate limit, on a client's requests.
|
||||
KindRequests = "requests"
|
||||
KindRequests LimitKind = "requests"
|
||||
// KindBytes is a byte limit, on a client's bytes.
|
||||
KindBytes = "bytes"
|
||||
KindBytes LimitKind = "bytes"
|
||||
// KindRefusals is the refusal limit, on a client's requests sww
|
||||
// refused after a rule file match or for a missing or wrong token.
|
||||
KindRefusals = "refusals"
|
||||
KindRefusals LimitKind = "refusals"
|
||||
)
|
||||
|
||||
// Limits are the most requests a client may make in a minute, an hour and
|
||||
@@ -188,9 +192,7 @@ func New(limits Limits, maxClients int) *Limiter {
|
||||
// Hit is a request that takes a client over a rate limit or the refusal
|
||||
// limit, or whose bytes take it over a byte limit.
|
||||
type Hit struct {
|
||||
// Kind is KindRequests for a rate limit, KindBytes for a byte limit,
|
||||
// KindRefusals for the refusal limit.
|
||||
Kind string
|
||||
Kind LimitKind
|
||||
// Window is "minute", "hour" or "day".
|
||||
Window string
|
||||
// Limit is the window's limit, as the client's percentage of it.
|
||||
|
||||
@@ -628,7 +628,7 @@ func wantBytesCount(
|
||||
client netip.Prefix,
|
||||
now time.Time,
|
||||
bytes int64,
|
||||
want string,
|
||||
want ratelimit.LimitKind,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
|
||||
@@ -11,6 +11,7 @@ import (
|
||||
"net/netip"
|
||||
"net/url"
|
||||
"slices"
|
||||
"strconv"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
@@ -23,6 +24,9 @@ const (
|
||||
// AbuseIPDBURL is where clients are checked: the check endpoint of
|
||||
// AbuseIPDB's API.
|
||||
AbuseIPDBURL = "https://api.abuseipdb.com/api/v2/check"
|
||||
// AbuseIPDBBlacklistURL is where AbuseIPDB's blacklist is fetched from:
|
||||
// the blacklist endpoint of AbuseIPDB's API.
|
||||
AbuseIPDBBlacklistURL = "https://api.abuseipdb.com/api/v2/blacklist"
|
||||
// AbuseIPDBSource is how the request log, the alerts and the metrics
|
||||
// name AbuseIPDB.
|
||||
AbuseIPDBSource = "abuseipdb"
|
||||
@@ -32,6 +36,8 @@ const (
|
||||
|
||||
var (
|
||||
errNoScore = errors.New("the answer gives no abuseConfidenceScore")
|
||||
errNoBlacklist = errors.New("the answer gives no data")
|
||||
errNotAddress = errors.New("is not an address")
|
||||
errBudgetUsedUp = errors.New(
|
||||
"checks spent; none is made until the day ends at 00:00 UTC")
|
||||
)
|
||||
@@ -53,6 +59,13 @@ type ChecksSpent struct {
|
||||
Spent int
|
||||
}
|
||||
|
||||
// BlacklistEntry is a client on AbuseIPDB's blacklist: its address and its
|
||||
// abuse confidence score.
|
||||
type BlacklistEntry struct {
|
||||
Addr netip.Addr
|
||||
Score int64
|
||||
}
|
||||
|
||||
// AbuseIPDBParams are what NewAbuseIPDB needs.
|
||||
type AbuseIPDBParams struct {
|
||||
// URL is where clients are checked, normally AbuseIPDBURL, with Key,
|
||||
@@ -69,10 +82,19 @@ type AbuseIPDBParams struct {
|
||||
// take (SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS).
|
||||
CacheTTL time.Duration
|
||||
Timeout time.Duration
|
||||
// BlacklistURL is where the blacklist is fetched from, normally
|
||||
// AbuseIPDBBlacklistURL, with the clients scored Threshold or more
|
||||
// (SWW_WAF_ABUSEIPDB_THRESHOLD), every BlacklistRefresh
|
||||
// (SWW_WAF_ABUSEIPDB_BLACKLIST_REFRESH_INTERVAL_MINS). Blacklisted
|
||||
// receives the clients on each blacklist fetched.
|
||||
BlacklistURL string
|
||||
Threshold int64
|
||||
BlacklistRefresh time.Duration
|
||||
Blacklisted func([]BlacklistEntry)
|
||||
// Now tells the time, normally time.Now in UTC.
|
||||
Now func() time.Time
|
||||
// ProcessLog receives each check that fails, and why, and the day's
|
||||
// budget used up.
|
||||
// ProcessLog receives each check or fetch of the blacklist that fails,
|
||||
// and why, and the day's budget used up.
|
||||
ProcessLog *slog.Logger
|
||||
// Alerts receive a source_failure alert for each.
|
||||
Alerts *alerts.Queue
|
||||
@@ -88,8 +110,12 @@ type AbuseIPDB struct {
|
||||
// scores are by client. Each is added as it is fetched and never moved
|
||||
// up, so that the one fetched longest ago is the first dropped.
|
||||
scores *simplelru.LRU[netip.Prefix, Score]
|
||||
// checking are the clients whose check is under way.
|
||||
checking map[netip.Prefix]bool
|
||||
// checking are the clients whose check is under way, each with the
|
||||
// function that ends it. stopped is true once StopChecking has been
|
||||
// called, and checkGoroutines waits for the goroutine of each check.
|
||||
checking map[netip.Prefix]context.CancelFunc
|
||||
stopped bool
|
||||
checkGoroutines sync.WaitGroup
|
||||
// day is the day, in UTC, of the checks spent counts.
|
||||
day civil.Date
|
||||
spent int
|
||||
@@ -112,7 +138,7 @@ func NewAbuseIPDB(params AbuseIPDBParams) *AbuseIPDB {
|
||||
params: params,
|
||||
httpClient: &http.Client{},
|
||||
scores: scores,
|
||||
checking: map[netip.Prefix]bool{},
|
||||
checking: map[netip.Prefix]context.CancelFunc{},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -122,11 +148,11 @@ func NewAbuseIPDB(params AbuseIPDBParams) *AbuseIPDB {
|
||||
// addresses its request comes from. A client without one is checked in
|
||||
// the background, by addr, the address its request came from, if
|
||||
// offender, if it has committed an offence, unless its check is under
|
||||
// way, a check failed less than failureDelay ago, or the day's checks
|
||||
// have used up DailyBudget; Hit never waits for a check. The check that
|
||||
// uses the budget up is logged and raised as a source_failure alert. ctx
|
||||
// is the context of the client's request, and a check goes on after the
|
||||
// request ends.
|
||||
// way, a check failed less than failureDelay ago, the day's checks have
|
||||
// used up DailyBudget, or StopChecking has been called; Hit never waits
|
||||
// for a check. The check that uses the budget up is logged and raised as a
|
||||
// source_failure alert. ctx is the context of the client's request, and a
|
||||
// check goes on after the request ends, until StopChecking.
|
||||
func (a *AbuseIPDB) Hit(
|
||||
ctx context.Context, client netip.Prefix, addr netip.Addr, offender bool,
|
||||
) (int64, bool) {
|
||||
@@ -145,14 +171,21 @@ func (a *AbuseIPDB) Hit(
|
||||
a.day, a.spent = today, 0
|
||||
}
|
||||
|
||||
check := offender && !a.checking[client] && !now.Before(a.retryAt) &&
|
||||
a.spent < a.params.DailyBudget
|
||||
_, checking := a.checking[client]
|
||||
|
||||
check := offender && !checking && !a.stopped &&
|
||||
!now.Before(a.retryAt) && a.spent < a.params.DailyBudget
|
||||
if check {
|
||||
a.checking[client] = true
|
||||
checkCtx, end := context.WithCancel(context.WithoutCancel(ctx))
|
||||
a.checking[client] = end
|
||||
a.checks++
|
||||
a.spent++
|
||||
|
||||
go a.check(context.WithoutCancel(ctx), client, addr)
|
||||
a.checkGoroutines.Go(func() {
|
||||
defer end()
|
||||
|
||||
a.check(checkCtx, client, addr)
|
||||
})
|
||||
}
|
||||
|
||||
usedUp := check && a.spent == a.params.DailyBudget
|
||||
@@ -167,6 +200,24 @@ func (a *AbuseIPDB) Hit(
|
||||
return 0, false
|
||||
}
|
||||
|
||||
// StopChecking ends the checks under way, and returns once their goroutines
|
||||
// have. A check it ends gives no score and is no failure: it is neither
|
||||
// counted, logged nor raised as an alert, and checks are not left alone
|
||||
// after it. The ended check stays spent from the day's budget. After
|
||||
// StopChecking, Hit checks no client.
|
||||
func (a *AbuseIPDB) StopChecking() {
|
||||
a.mu.Lock()
|
||||
|
||||
a.stopped = true
|
||||
for _, end := range a.checking {
|
||||
end()
|
||||
}
|
||||
|
||||
a.mu.Unlock()
|
||||
|
||||
a.checkGoroutines.Wait()
|
||||
}
|
||||
|
||||
// Checked returns how many checks were made.
|
||||
func (a *AbuseIPDB) Checked() int {
|
||||
a.mu.Lock()
|
||||
@@ -256,10 +307,45 @@ func (a *AbuseIPDB) LoadChecksSpent(checks ChecksSpent) {
|
||||
a.day, a.spent = checks.Day, checks.Spent
|
||||
}
|
||||
|
||||
// RunBlacklist fetches AbuseIPDB's blacklist at once, and again every
|
||||
// BlacklistRefresh, until ctx is done, and gives the clients on each copy
|
||||
// fetched to Blacklisted. While Key is unset it does nothing. A fetch that
|
||||
// fails is logged and raised as a source_failure alert, unless it was cut
|
||||
// off as ctx ended.
|
||||
func (a *AbuseIPDB) RunBlacklist(ctx context.Context) {
|
||||
if a.params.Key == "" {
|
||||
return
|
||||
}
|
||||
|
||||
for {
|
||||
listed, err := a.fetchBlacklist(ctx)
|
||||
|
||||
switch {
|
||||
case ctx.Err() != nil:
|
||||
return
|
||||
case err != nil:
|
||||
a.alert("fetching AbuseIPDB's blacklist failed", err)
|
||||
default:
|
||||
a.params.Blacklisted(listed)
|
||||
}
|
||||
|
||||
timer := time.NewTimer(a.params.BlacklistRefresh)
|
||||
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
timer.Stop()
|
||||
|
||||
return
|
||||
case <-timer.C:
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// check checks client with AbuseIPDB by addr, one of its addresses, keeps
|
||||
// the score as client's, and notes the check as no longer under way. A
|
||||
// check that fails gives no score: it is counted, logged and raised as a
|
||||
// source_failure alert, and no client is checked for failureDelay.
|
||||
// source_failure alert, and no client is checked for failureDelay. A check
|
||||
// ended as ctx ends, by StopChecking, is no failure.
|
||||
func (a *AbuseIPDB) check(
|
||||
ctx context.Context,
|
||||
client netip.Prefix,
|
||||
@@ -267,24 +353,26 @@ func (a *AbuseIPDB) check(
|
||||
) {
|
||||
score, err := a.ask(ctx, addr)
|
||||
now := a.params.Now()
|
||||
isFailure := err != nil && ctx.Err() == nil
|
||||
|
||||
a.mu.Lock()
|
||||
|
||||
delete(a.checking, client)
|
||||
|
||||
if err == nil {
|
||||
switch {
|
||||
case err == nil:
|
||||
a.scores.Add(
|
||||
client,
|
||||
Score{Client: client, Score: score, Fetched: now},
|
||||
)
|
||||
} else {
|
||||
case isFailure:
|
||||
a.failures++
|
||||
a.retryAt = now.Add(failureDelay)
|
||||
}
|
||||
|
||||
a.mu.Unlock()
|
||||
|
||||
if err != nil {
|
||||
if isFailure {
|
||||
a.alert("checking a client with AbuseIPDB failed", err)
|
||||
}
|
||||
}
|
||||
@@ -344,6 +432,94 @@ func (a *AbuseIPDB) ask(
|
||||
return *answer.Data.AbuseConfidenceScore, nil
|
||||
}
|
||||
|
||||
// fetchBlacklist fetches the clients on AbuseIPDB's blacklist scored
|
||||
// Threshold or more, sending the key in the header Key. An answer other
|
||||
// than 200, one longer than maxListBytes or without the list, an entry
|
||||
// whose address does not read, and none within fetchTimeout, fail.
|
||||
func (a *AbuseIPDB) fetchBlacklist(
|
||||
ctx context.Context,
|
||||
) ([]BlacklistEntry, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, fetchTimeout)
|
||||
defer cancel()
|
||||
|
||||
query := url.Values{
|
||||
"confidenceMinimum": {strconv.FormatInt(a.params.Threshold, 10)},
|
||||
}
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet,
|
||||
a.params.BlacklistURL+"?"+query.Encode(), http.NoBody)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("make the request: %w", err)
|
||||
}
|
||||
|
||||
req.Header.Set("Key", a.params.Key)
|
||||
req.Header.Set("Accept", "application/json")
|
||||
|
||||
res, err := a.httpClient.Do(req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("fetch the blacklist: %w", errors.Unwrap(err))
|
||||
}
|
||||
|
||||
defer func() {
|
||||
_ = res.Body.Close()
|
||||
}()
|
||||
|
||||
if res.StatusCode != http.StatusOK {
|
||||
return nil, fmt.Errorf("%w %s", errStatus, res.Status)
|
||||
}
|
||||
|
||||
body, err := io.ReadAll(io.LimitReader(res.Body, maxListBytes+1))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read the blacklist: %w", err)
|
||||
}
|
||||
|
||||
if len(body) > maxListBytes {
|
||||
return nil, errTooLong
|
||||
}
|
||||
|
||||
return parseBlacklist(body)
|
||||
}
|
||||
|
||||
// parseBlacklist reads body, AbuseIPDB's answer with its blacklist.
|
||||
func parseBlacklist(body []byte) ([]BlacklistEntry, error) {
|
||||
var answer struct {
|
||||
Data *[]struct {
|
||||
IPAddress string `json:"ipAddress"`
|
||||
AbuseConfidenceScore int64 `json:"abuseConfidenceScore"`
|
||||
} `json:"data"`
|
||||
}
|
||||
|
||||
err := json.Unmarshal(body, &answer)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read the blacklist: %w", err)
|
||||
}
|
||||
|
||||
if answer.Data == nil {
|
||||
return nil, errNoBlacklist
|
||||
}
|
||||
|
||||
listed := make([]BlacklistEntry, 0, len(*answer.Data))
|
||||
|
||||
for i, entry := range *answer.Data {
|
||||
addr, err := netip.ParseAddr(entry.IPAddress)
|
||||
if err != nil || addr.Zone() != "" {
|
||||
// The error leaves out what the entry holds, as it does a
|
||||
// client's address.
|
||||
return nil, fmt.Errorf(
|
||||
"entry %d's ipAddress %w",
|
||||
i+1,
|
||||
errNotAddress,
|
||||
)
|
||||
}
|
||||
|
||||
listed = append(listed, BlacklistEntry{
|
||||
Addr: addr.Unmap(), Score: entry.AbuseConfidenceScore,
|
||||
})
|
||||
}
|
||||
|
||||
return listed, nil
|
||||
}
|
||||
|
||||
// alert raises a source_failure alert from AbuseIPDB with reason and err,
|
||||
// and logs them.
|
||||
func (a *AbuseIPDB) alert(reason string, err error) {
|
||||
|
||||
@@ -394,6 +394,43 @@ func TestCheckNotAnsweredWithinTheTimeoutFailsAndHitNeverWaits(
|
||||
})
|
||||
}
|
||||
|
||||
func TestStopCheckingEndsTheCheckUnderWayAsNoFailureAndChecksNoMore(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
queue := newQueue()
|
||||
p := abuseIPDBParams()
|
||||
p.Alerts = queue
|
||||
abuseIPDB := &abuseIPDBStandIn{hanging: true}
|
||||
checker := newAbuseIPDB(abuseIPDB, p)
|
||||
|
||||
wantScore(t, checker, suspect, true, 0, false)
|
||||
synctest.Wait()
|
||||
|
||||
// The check is ended, not waited out, and is no failure.
|
||||
began := time.Now()
|
||||
|
||||
checker.StopChecking()
|
||||
|
||||
if waited := time.Since(began); waited != 0 {
|
||||
t.Errorf("StopChecking returned after %s, want at once", waited)
|
||||
}
|
||||
|
||||
wantFailures(t, checker, 0)
|
||||
|
||||
if got := waiting(queue); len(got) != 0 {
|
||||
t.Errorf("alerts waiting %+v, want none", got)
|
||||
}
|
||||
|
||||
// An offender without a score is then not checked.
|
||||
wantScore(t, checker, other, true, 0, false)
|
||||
synctest.Wait()
|
||||
wantChecked(t, abuseIPDB, suspect)
|
||||
})
|
||||
}
|
||||
|
||||
func TestKeyIsSentInTheKeyHeaderAndNeverShown(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -0,0 +1,252 @@
|
||||
package reputation_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"testing/synctest"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||
)
|
||||
|
||||
// The tests of the blacklist run in synctest bubbles, as those of the
|
||||
// checks do, with AbuseIPDB a stand-in reached without the network.
|
||||
|
||||
// blacklistURL is where the tests fetch the blacklist from.
|
||||
const blacklistURL = "https://abuseipdb.example/api/v2/blacklist"
|
||||
|
||||
func TestBlacklistIsFetchedAtOnceAndEveryRefresh(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
abuseIPDB := &blacklistStandIn{body: `{"meta":{},"data":[` +
|
||||
`{"ipAddress":"203.0.113.9","abuseConfidenceScore":100},` +
|
||||
`{"ipAddress":"::ffff:198.51.100.7","abuseConfidenceScore":95},` +
|
||||
`{"ipAddress":"2001:db8::9","abuseConfidenceScore":90}]}`}
|
||||
got := runBlacklist(t, abuseIPDB, key, newQueue())
|
||||
|
||||
want := []reputation.BlacklistEntry{
|
||||
{Addr: netip.MustParseAddr("203.0.113.9"), Score: 100},
|
||||
{Addr: netip.MustParseAddr("198.51.100.7"), Score: 95},
|
||||
{Addr: netip.MustParseAddr("2001:db8::9"), Score: 90},
|
||||
}
|
||||
|
||||
synctest.Wait()
|
||||
got.want(t, want)
|
||||
|
||||
time.Sleep(time.Hour - time.Nanosecond)
|
||||
synctest.Wait()
|
||||
got.want(t, want)
|
||||
|
||||
time.Sleep(time.Nanosecond)
|
||||
synctest.Wait()
|
||||
got.want(t, want, want)
|
||||
|
||||
// Each fetch asks for the clients scored 90 or more, with the key
|
||||
// in the header Key alone.
|
||||
abuseIPDB.mu.Lock()
|
||||
defer abuseIPDB.mu.Unlock()
|
||||
|
||||
for _, req := range abuseIPDB.asked {
|
||||
if req.URL.String() != blacklistURL+"?confidenceMinimum=90" ||
|
||||
req.Header.Get("Key") != key {
|
||||
t.Errorf(
|
||||
"asked %s with the key %q, want %s?confidenceMinimum=90 "+
|
||||
"with %q",
|
||||
req.URL,
|
||||
req.Header.Get("Key"),
|
||||
blacklistURL,
|
||||
key,
|
||||
)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestFailedBlacklistFetchIsAlertedAndTriedAgainAtTheNextRefresh(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
for _, tc := range []struct{ name, body, error string }{
|
||||
{
|
||||
"a refusal", "",
|
||||
"the server answered 429 Too Many Requests",
|
||||
},
|
||||
{
|
||||
"an answer without the list", `{"errors":[]}`,
|
||||
"the answer gives no data",
|
||||
},
|
||||
{
|
||||
"an entry that is not an address",
|
||||
`{"data":[{"ipAddress":"203.0.113.9"},{"ipAddress":"example"}]}`,
|
||||
"entry 2's ipAddress is not an address",
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
abuseIPDB := &blacklistStandIn{body: tc.body}
|
||||
if tc.body == "" {
|
||||
abuseIPDB.status = http.StatusTooManyRequests
|
||||
}
|
||||
|
||||
queue := newQueue()
|
||||
got := runBlacklist(t, abuseIPDB, key, queue)
|
||||
|
||||
synctest.Wait()
|
||||
|
||||
got.want(t)
|
||||
|
||||
alerted := waiting(queue)
|
||||
if len(alerted) != 1 ||
|
||||
alerted[0].Event != alerts.EventSourceFailure ||
|
||||
alerted[0].Reason != "fetching AbuseIPDB's blacklist failed" ||
|
||||
alerted[0].Detail["source"] != reputation.AbuseIPDBSource ||
|
||||
alerted[0].Detail["error"] != tc.error {
|
||||
t.Errorf(
|
||||
"alerts %+v, want the failure %q",
|
||||
alerted,
|
||||
tc.error,
|
||||
)
|
||||
}
|
||||
|
||||
abuseIPDB.answer(`{"data":[]}`)
|
||||
time.Sleep(time.Hour)
|
||||
synctest.Wait()
|
||||
got.want(t, []reputation.BlacklistEntry{})
|
||||
})
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestBlacklistIsNotFetchedWithoutAKey(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
abuseIPDB := &blacklistStandIn{body: `{"data":[]}`}
|
||||
got := runBlacklist(t, abuseIPDB, "", newQueue())
|
||||
|
||||
time.Sleep(time.Hour)
|
||||
synctest.Wait()
|
||||
got.want(t)
|
||||
|
||||
if len(abuseIPDB.asked) != 0 {
|
||||
t.Errorf("fetched the blacklist %d times, want none",
|
||||
len(abuseIPDB.asked))
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// blacklistStandIn is a stand-in for AbuseIPDB that answers each fetch of
|
||||
// the blacklist with status, 200 when it is zero, and body, and keeps the
|
||||
// requests.
|
||||
type blacklistStandIn struct {
|
||||
mu sync.Mutex
|
||||
status int
|
||||
body string
|
||||
asked []*http.Request
|
||||
}
|
||||
|
||||
// RoundTrip has the stand-in answer req, in place of the network.
|
||||
func (s *blacklistStandIn) RoundTrip(
|
||||
req *http.Request,
|
||||
) (*http.Response, error) {
|
||||
s.mu.Lock()
|
||||
s.asked = append(s.asked, req)
|
||||
status, body := s.status, s.body
|
||||
s.mu.Unlock()
|
||||
|
||||
if status == 0 {
|
||||
status = http.StatusOK
|
||||
}
|
||||
|
||||
return &http.Response{
|
||||
StatusCode: status,
|
||||
Status: fmt.Sprintf("%d %s", status, http.StatusText(status)),
|
||||
Header: http.Header{},
|
||||
Body: io.NopCloser(strings.NewReader(body)),
|
||||
Request: req,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// answer has the stand-in answer each fetch with 200 and body.
|
||||
func (s *blacklistStandIn) answer(body string) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
|
||||
s.status, s.body = 0, body
|
||||
}
|
||||
|
||||
// blacklisted holds the clients on each blacklist given to Blacklisted.
|
||||
type blacklisted struct {
|
||||
mu sync.Mutex
|
||||
fetched [][]reputation.BlacklistEntry
|
||||
}
|
||||
|
||||
// want checks the clients on each blacklist given so far.
|
||||
func (b *blacklisted) want(
|
||||
t *testing.T,
|
||||
want ...[]reputation.BlacklistEntry,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
|
||||
if !slices.EqualFunc(b.fetched, want, slices.Equal) {
|
||||
t.Errorf("blacklists %v, want %v", b.fetched, want)
|
||||
}
|
||||
}
|
||||
|
||||
// runBlacklist runs RunBlacklist, until the test ends, with the key
|
||||
// accountKey, the threshold 90 and a refresh of an hour, fetching from
|
||||
// abuseIPDB and raising alerts into queue, and returns what it gives
|
||||
// Blacklisted.
|
||||
func runBlacklist(
|
||||
t *testing.T, abuseIPDB *blacklistStandIn, accountKey string,
|
||||
queue *alerts.Queue,
|
||||
) *blacklisted {
|
||||
t.Helper()
|
||||
|
||||
got := &blacklisted{}
|
||||
p := abuseIPDBParams()
|
||||
p.Key = accountKey
|
||||
p.BlacklistURL = blacklistURL
|
||||
p.Threshold = 90
|
||||
p.BlacklistRefresh = time.Hour
|
||||
p.Alerts = queue
|
||||
p.Blacklisted = func(listed []reputation.BlacklistEntry) {
|
||||
got.mu.Lock()
|
||||
defer got.mu.Unlock()
|
||||
|
||||
got.fetched = append(got.fetched, listed)
|
||||
}
|
||||
|
||||
checker := reputation.NewAbuseIPDB(p)
|
||||
checker.SetTransport(abuseIPDB)
|
||||
|
||||
ctx, cancel := context.WithCancel(t.Context())
|
||||
done := make(chan struct{})
|
||||
|
||||
go func() {
|
||||
checker.RunBlacklist(ctx)
|
||||
close(done)
|
||||
}()
|
||||
|
||||
t.Cleanup(func() {
|
||||
cancel()
|
||||
<-done
|
||||
})
|
||||
|
||||
return got
|
||||
}
|
||||
@@ -82,8 +82,12 @@ type DNSBL struct {
|
||||
// verdicts are by query. Each is added as it is fetched and never moved
|
||||
// up, so that the one fetched longest ago is the first dropped.
|
||||
verdicts *simplelru.LRU[query, Verdict]
|
||||
// asking are the queries under way.
|
||||
asking map[query]bool
|
||||
// asking are the queries under way, each with the function that ends
|
||||
// it. stopped is true once StopAsking has been called, and
|
||||
// queryGoroutines waits for the goroutine of each query.
|
||||
asking map[query]context.CancelFunc
|
||||
stopped bool
|
||||
queryGoroutines sync.WaitGroup
|
||||
// queries and failures count, by zone, the queries made and those that
|
||||
// failed, and retryAt is when a zone whose last query failed may be
|
||||
// asked again.
|
||||
@@ -120,7 +124,7 @@ func NewDNSBL(params DNSBLParams) *DNSBL {
|
||||
params: params,
|
||||
resolver: resolver,
|
||||
verdicts: verdicts,
|
||||
asking: map[query]bool{},
|
||||
asking: map[query]context.CancelFunc{},
|
||||
queries: map[string]int{},
|
||||
failures: map[string]int{},
|
||||
retryAt: map[string]time.Time{},
|
||||
@@ -138,9 +142,10 @@ func (d *DNSBL) Zones() []string {
|
||||
// log, the alerts and the metrics. A verdict is used until CacheTTL has
|
||||
// passed since it was fetched. Each zone without one is asked about addr
|
||||
// in the background, unless a query about addr to it is under way, the
|
||||
// zone is left alone after a failure, or maxQueries are under way;
|
||||
// ListedBy never waits for a query. ctx is the context of the client's
|
||||
// request, and a query goes on after the request ends.
|
||||
// zone is left alone after a failure, maxQueries are under way, or
|
||||
// StopAsking has been called; ListedBy never waits for a query. ctx is the
|
||||
// context of the client's request, and a query goes on after the request
|
||||
// ends, until StopAsking.
|
||||
func (d *DNSBL) ListedBy(ctx context.Context, addr netip.Addr) []string {
|
||||
d.mu.Lock()
|
||||
defer d.mu.Unlock()
|
||||
@@ -153,23 +158,47 @@ func (d *DNSBL) ListedBy(ctx context.Context, addr netip.Addr) []string {
|
||||
q := query{zone: zone, client: addr}
|
||||
|
||||
kept, found := d.verdicts.Peek(q)
|
||||
_, asking := d.asking[q]
|
||||
|
||||
switch {
|
||||
case found && now.Sub(kept.Fetched) < d.params.CacheTTL:
|
||||
if kept.Listed {
|
||||
listedBy = append(listedBy, config.MaskZoneKey(zone))
|
||||
}
|
||||
case !d.asking[q] && !now.Before(d.retryAt[zone]) && len(d.asking) < maxQueries:
|
||||
d.asking[q] = true
|
||||
case !asking && !d.stopped && !now.Before(d.retryAt[zone]) &&
|
||||
len(d.asking) < maxQueries:
|
||||
queryCtx, end := context.WithCancel(context.WithoutCancel(ctx))
|
||||
d.asking[q] = end
|
||||
d.queries[zone]++
|
||||
|
||||
go d.ask(context.WithoutCancel(ctx), q)
|
||||
d.queryGoroutines.Go(func() {
|
||||
defer end()
|
||||
|
||||
d.ask(queryCtx, q)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
return listedBy
|
||||
}
|
||||
|
||||
// StopAsking ends the queries under way, and returns once their
|
||||
// goroutines have. A query it ends gives no verdict and is no failure: it
|
||||
// is neither counted, logged nor raised as an alert, and the zone is not
|
||||
// left alone after it. After StopAsking, ListedBy asks no zone.
|
||||
func (d *DNSBL) StopAsking() {
|
||||
d.mu.Lock()
|
||||
|
||||
d.stopped = true
|
||||
for _, end := range d.asking {
|
||||
end()
|
||||
}
|
||||
|
||||
d.mu.Unlock()
|
||||
|
||||
d.queryGoroutines.Wait()
|
||||
}
|
||||
|
||||
// Queries returns how many queries were made to zone.
|
||||
func (d *DNSBL) Queries(zone string) int {
|
||||
d.mu.Lock()
|
||||
@@ -238,27 +267,29 @@ func (d *DNSBL) Load(verdicts []Verdict) {
|
||||
// query as no longer under way. A query that fails gives no verdict: it
|
||||
// is counted, logged and raised as a source_failure alert, which show the
|
||||
// zone with its key masked, and the zone is not asked again for
|
||||
// failureDelay.
|
||||
// failureDelay. A query ended as ctx ends, by StopAsking, is no failure.
|
||||
func (d *DNSBL) ask(ctx context.Context, q query) {
|
||||
listed, err := d.lookUp(ctx, q)
|
||||
now := d.params.Now()
|
||||
isFailure := err != nil && ctx.Err() == nil
|
||||
|
||||
d.mu.Lock()
|
||||
|
||||
delete(d.asking, q)
|
||||
|
||||
if err == nil {
|
||||
switch {
|
||||
case err == nil:
|
||||
d.verdicts.Add(q, Verdict{
|
||||
Zone: q.zone, Client: q.client, Listed: listed, Fetched: now,
|
||||
})
|
||||
} else {
|
||||
case isFailure:
|
||||
d.failures[q.zone]++
|
||||
d.retryAt[q.zone] = now.Add(failureDelay)
|
||||
}
|
||||
|
||||
d.mu.Unlock()
|
||||
|
||||
if err != nil {
|
||||
if isFailure {
|
||||
const failed = "asking a DNSBL zone failed"
|
||||
|
||||
shown := config.MaskZoneKey(q.zone)
|
||||
|
||||
@@ -31,10 +31,12 @@ import (
|
||||
// this package: Go's resolver counts the queries under way in one
|
||||
// sync.WaitGroup for the whole process, and the process fails when
|
||||
// queries from two bubbles, or from a bubble and from outside one, are
|
||||
// under way at once. TestMain has the resolver make its configuration,
|
||||
// which it makes on its first query, outside every bubble, since the
|
||||
// configuration holds a channel, which the bubble it was made in would
|
||||
// keep to itself.
|
||||
// under way at once. The one test outside a bubble is parallel, so it
|
||||
// starts only once every test that is not parallel has ended; no other
|
||||
// parallel test of this package queries a resolver. TestMain has the
|
||||
// resolver make its configuration, which it makes on its first query,
|
||||
// outside every bubble, since the configuration holds a channel, which
|
||||
// the bubble it was made in would keep to itself.
|
||||
|
||||
const (
|
||||
// zone and otherZone are the DNSBL zones the tests name.
|
||||
@@ -208,6 +210,45 @@ func TestQueryNotAnsweredWithinTheTimeoutFails(t *testing.T) {
|
||||
})
|
||||
}
|
||||
|
||||
//
|
||||
//nolint:paralleltest // one at a time, as the comment at the top of this file says
|
||||
func TestStopAskingEndsTheQueryUnderWayAsNoFailureAndAsksNoMore(
|
||||
t *testing.T,
|
||||
) {
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
queue := newQueue()
|
||||
p := dnsblParams(zone)
|
||||
p.Alerts = queue
|
||||
resolver := &resolverStandIn{hanging: true}
|
||||
dnsbl := newDNSBL(resolver, p)
|
||||
|
||||
wantZones(t, dnsbl, listed)
|
||||
synctest.Wait()
|
||||
|
||||
// The query is ended, not waited out, and is no failure.
|
||||
began := time.Now()
|
||||
|
||||
dnsbl.StopAsking()
|
||||
|
||||
if waited := time.Since(began); waited != 0 {
|
||||
t.Errorf("StopAsking returned after %s, want at once", waited)
|
||||
}
|
||||
|
||||
wantQueries(t, dnsbl, 1, 0)
|
||||
|
||||
if got := waiting(queue); len(got) != 0 {
|
||||
t.Errorf("alerts waiting %+v, want none", got)
|
||||
}
|
||||
|
||||
// A client without a verdict is then asked about in no zone.
|
||||
wantZones(t, dnsbl, unlisted)
|
||||
synctest.Wait()
|
||||
wantQueries(t, dnsbl, 1, 0)
|
||||
wantAsked(t, resolver, listedName)
|
||||
waitForTheResolver()
|
||||
})
|
||||
}
|
||||
|
||||
//
|
||||
//nolint:paralleltest // one at a time, as the comment at the top of this file says
|
||||
func TestZoneThatFailsOrRefusesGivesNoVerdictAndIsLeftAloneForAMinute(
|
||||
@@ -525,8 +566,9 @@ func TestAtMost100000VerdictsKeptTheOneFetchedLongestAgoDroppedFirst(
|
||||
}
|
||||
}
|
||||
|
||||
//nolint:paralleltest // one at a time, as the comment at the top of this file says
|
||||
func TestQueriesGoToTheResolverSWWDNSBLResolverNames(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
resolver := &resolverStandIn{answers: map[string]answer{
|
||||
listedName: {addrs: []string{listing}},
|
||||
}}
|
||||
|
||||
@@ -7,7 +7,8 @@
|
||||
// SWW_WAF_DNSBL_ZONES about clients, and keeps their verdicts, and checks
|
||||
// clients with AbuseIPDB, and keeps their scores and the checks spent
|
||||
// today. The state package writes all of these to files and reads them
|
||||
// from there, so that a restart keeps them too.
|
||||
// from there, so that a restart keeps them too. Last, it fetches
|
||||
// AbuseIPDB's blacklist, which it does not keep.
|
||||
package reputation
|
||||
|
||||
import (
|
||||
|
||||
@@ -15,74 +15,91 @@ import (
|
||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||
)
|
||||
|
||||
// Action is what waf did with a request, as its request line names it,
|
||||
// such as ActionForward.
|
||||
type Action string
|
||||
|
||||
// The action a request line names: what waf did with the request.
|
||||
const (
|
||||
// ActionForward is a request waf allowed, which rproxy passes to
|
||||
// the app.
|
||||
ActionForward = "forward"
|
||||
ActionForward Action = "forward"
|
||||
|
||||
// ActionRequestLineTooLong is a request whose request line is too long for
|
||||
// the header limits.
|
||||
ActionRequestLineTooLong = "request_line_too_long"
|
||||
ActionRequestLineTooLong Action = "request_line_too_long"
|
||||
|
||||
// ActionRequestHeaderTooLarge is a request whose headers are over the
|
||||
// header limits.
|
||||
ActionRequestHeaderTooLarge = "request_header_too_large"
|
||||
ActionRequestHeaderTooLarge Action = "request_header_too_large"
|
||||
|
||||
// ActionClientIsRateLimited is a request refused because it took its
|
||||
// client over a rate limit, which bans the client.
|
||||
ActionClientIsRateLimited = "client_is_rate_limited"
|
||||
ActionClientIsRateLimited Action = "client_is_rate_limited"
|
||||
|
||||
// ActionTemporarilyBanned is a request refused because a ban that runs
|
||||
// out covers its client, or because it matched a ban rule or asked for a
|
||||
// trap path, and so made such a ban.
|
||||
ActionTemporarilyBanned = "temporarily_banned"
|
||||
ActionTemporarilyBanned Action = "temporarily_banned"
|
||||
|
||||
// ActionPermanentlyBanned is a request refused because a ban that never
|
||||
// runs out covers its client, or because the request made such a ban or
|
||||
// made its client's ban permanent.
|
||||
ActionPermanentlyBanned = "permanently_banned"
|
||||
ActionPermanentlyBanned Action = "permanently_banned"
|
||||
|
||||
// ActionBlockedByBlockRule is a request refused because it matched a
|
||||
// block rule.
|
||||
ActionBlockedByBlockRule = "blocked_by_block_rule"
|
||||
ActionBlockedByBlockRule Action = "blocked_by_block_rule"
|
||||
|
||||
// ActionBlockedByCoreRuleSet is a request refused because the Core Rule
|
||||
// Set scored it at or over SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD.
|
||||
ActionBlockedByCoreRuleSet = "blocked_by_core_rule_set"
|
||||
ActionBlockedByCoreRuleSet Action = "blocked_by_core_rule_set"
|
||||
|
||||
// ActionClientDenied is a request refused because its client is in
|
||||
// SWW_WAF_DENY_NETS, in a blocklist while SWW_WAF_BLOCKLIST_ACTION is deny,
|
||||
// or listed by a DNSBL zone, or scored a hit by AbuseIPDB, while
|
||||
// SWW_WAF_DNSBL_ABUSEIPDB_ACTION is deny, or that carries a tag in
|
||||
// SWW_WAF_TAG_DENY.
|
||||
ActionClientDenied = "client_denied"
|
||||
ActionClientDenied Action = "client_denied"
|
||||
|
||||
// ActionCountryDenied is a request refused because its client's country
|
||||
// is not in SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES.
|
||||
ActionCountryDenied = "country_denied"
|
||||
ActionCountryDenied Action = "country_denied"
|
||||
|
||||
// ActionChecksTimedOut is a request refused because the trap paths, the
|
||||
// rule files and the Core Rule Set had not finished with it within 500
|
||||
// milliseconds.
|
||||
ActionChecksTimedOut Action = "checks_timed_out"
|
||||
|
||||
// ActionOwnEndpoint is a request waf answered at one of its own
|
||||
// endpoints, or a question about a path under /_sww/.
|
||||
ActionOwnEndpoint = "own_endpoint"
|
||||
ActionOwnEndpoint Action = "own_endpoint"
|
||||
)
|
||||
|
||||
// Offence is what a request was held as, as its request line names it.
|
||||
type Offence string
|
||||
|
||||
// OffenceLimit is the offence a request line names for a request that
|
||||
// broke a rate limit or the refusal limit, or whose bytes broke a byte
|
||||
// limit.
|
||||
const OffenceLimit = "limit"
|
||||
const OffenceLimit Offence = "limit"
|
||||
|
||||
// LimitHitMinuteRefusals is the limit_hit a request line names for a request
|
||||
// that broke the refusal limit.
|
||||
const LimitHitMinuteRefusals = "minute_refusals"
|
||||
|
||||
// WAFLine is which of waf's two lines about a question a request line is:
|
||||
// WAFLineDecision or WAFLineReport.
|
||||
type WAFLine string
|
||||
|
||||
// The waf_line a request line names, which says which of
|
||||
// waf's two lines about a question it is.
|
||||
const (
|
||||
// WAFLineDecision is waf's line at its decision.
|
||||
WAFLineDecision = "decision"
|
||||
WAFLineDecision WAFLine = "decision"
|
||||
// WAFLineReport is waf's line when rproxy's report
|
||||
// comes, with the decision's fields and the report's together.
|
||||
WAFLineReport = "report"
|
||||
WAFLineReport WAFLine = "report"
|
||||
)
|
||||
|
||||
// timeLayout is RFC 3339 with milliseconds.
|
||||
@@ -95,9 +112,8 @@ const timeLayout = "2006-01-02T15:04:05.000Z07:00"
|
||||
//nolint:tagliatelle // docs/request-log.md names the fields in snake_case
|
||||
type Line struct {
|
||||
Type string `json:"type"`
|
||||
// WAFLine is WAFLineDecision or WAFLineReport
|
||||
// on waf's two lines about a question.
|
||||
WAFLine string `json:"waf_line,omitempty"`
|
||||
// WAFLine is set on waf's two lines about a question.
|
||||
WAFLine WAFLine `json:"waf_line,omitempty"`
|
||||
|
||||
// The standard web log fields. Scheme is how the client reached rproxy, or
|
||||
// the trusted proxy in front of it.
|
||||
@@ -143,14 +159,15 @@ type Line struct {
|
||||
Aborted bool `json:"aborted,omitempty"`
|
||||
|
||||
// The decision.
|
||||
Action string `json:"action"`
|
||||
Action Action `json:"action"`
|
||||
// WouldAction is, in observe mode, the action enforce mode would have
|
||||
// taken with a request it would have refused: ActionClientDenied,
|
||||
// ActionTemporarilyBanned, ActionPermanentlyBanned, ActionCountryDenied,
|
||||
// ActionClientIsRateLimited, ActionBlockedByBlockRule,
|
||||
// ActionBlockedByCoreRuleSet, or, in waf, ActionRequestLineTooLong or
|
||||
// ActionRequestHeaderTooLarge for the header limits.
|
||||
WouldAction string `json:"would_action,omitempty"`
|
||||
// ActionRequestHeaderTooLarge for the header limits, or
|
||||
// ActionChecksTimedOut.
|
||||
WouldAction Action `json:"would_action,omitempty"`
|
||||
// Heuristic is the name of the heuristic that refused the request, in
|
||||
// observe mode too, and HeuristicFigures are the figures it acted on, by
|
||||
// name.
|
||||
@@ -196,9 +213,8 @@ type Line struct {
|
||||
// RouteWatchTagsMissing, while SWW_WAF_ROUTEWATCH_URL is set, says why the
|
||||
// request got no tags from routewatch: no_answer_yet, request_failed or
|
||||
// no_route.
|
||||
RouteWatchTagsMissing string `json:"routewatch_tags_missing,omitempty"`
|
||||
// Offence is the offence the request was held as, OffenceLimit.
|
||||
Offence string `json:"offence,omitempty"`
|
||||
RouteWatchTagsMissing string `json:"routewatch_tags_missing,omitempty"`
|
||||
Offence Offence `json:"offence,omitempty"`
|
||||
// BanExpires is when the ban the request made, or was refused under,
|
||||
// ends: a time, or "permanent".
|
||||
BanExpires string `json:"ban_expires,omitempty"`
|
||||
|
||||
@@ -99,8 +99,12 @@ type Client struct {
|
||||
// answers are by client. Each is added as it comes and never moved up,
|
||||
// so that the one that came longest ago is the first dropped.
|
||||
answers *simplelru.LRU[netip.Prefix, Answer]
|
||||
// asking are the clients whose query is under way.
|
||||
asking map[netip.Prefix]bool
|
||||
// asking are the clients whose query is under way, each with the
|
||||
// function that ends it. stopped is true once StopAsking has been
|
||||
// called, and queryGoroutines waits for the goroutine of each query.
|
||||
asking map[netip.Prefix]context.CancelFunc
|
||||
stopped bool
|
||||
queryGoroutines sync.WaitGroup
|
||||
// retryAt is when routewatch may be asked again after a query failed.
|
||||
retryAt time.Time
|
||||
}
|
||||
@@ -116,7 +120,7 @@ func New(params Params) *Client {
|
||||
params: params,
|
||||
httpClient: &http.Client{},
|
||||
answers: answers,
|
||||
asking: map[netip.Prefix]bool{},
|
||||
asking: map[netip.Prefix]context.CancelFunc{},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -124,8 +128,9 @@ func New(params Params) *Client {
|
||||
// IPv6 group, or why it has no route. It never waits for a query. A client
|
||||
// without an answer is asked about in the background, by addr, the address its
|
||||
// request came from, unless its query is under way, a query failed less than
|
||||
// failureKept ago, or maxQueries are under way. ctx is the context of the
|
||||
// client's request, and a query goes on after the request ends.
|
||||
// failureKept ago, maxQueries are under way, or StopAsking has been called. ctx
|
||||
// is the context of the client's request, and a query goes on after the request
|
||||
// ends, until StopAsking.
|
||||
func (c *Client) Route(
|
||||
ctx context.Context, client netip.Prefix, addr netip.Addr,
|
||||
) (Route, Missing) {
|
||||
@@ -147,15 +152,38 @@ func (c *Client) Route(
|
||||
return Route{}, RequestFailed
|
||||
}
|
||||
|
||||
if !c.asking[client] && len(c.asking) < maxQueries {
|
||||
c.asking[client] = true
|
||||
_, asking := c.asking[client]
|
||||
if !asking && !c.stopped && len(c.asking) < maxQueries {
|
||||
queryCtx, end := context.WithCancel(context.WithoutCancel(ctx))
|
||||
c.asking[client] = end
|
||||
|
||||
go c.ask(context.WithoutCancel(ctx), client, addr)
|
||||
c.queryGoroutines.Go(func() {
|
||||
defer end()
|
||||
|
||||
c.ask(queryCtx, client, addr)
|
||||
})
|
||||
}
|
||||
|
||||
return Route{}, NoAnswerYet
|
||||
}
|
||||
|
||||
// StopAsking ends the queries under way, and returns once their goroutines
|
||||
// have. A query it ends gives no answer and is no failure: it is not logged,
|
||||
// and routewatch is not left alone after it. After StopAsking, Route asks
|
||||
// routewatch nothing.
|
||||
func (c *Client) StopAsking() {
|
||||
c.mu.Lock()
|
||||
|
||||
c.stopped = true
|
||||
for _, end := range c.asking {
|
||||
end()
|
||||
}
|
||||
|
||||
c.mu.Unlock()
|
||||
|
||||
c.queryGoroutines.Wait()
|
||||
}
|
||||
|
||||
// Snapshot returns every answer still in use, sorted by client, as
|
||||
// routewatch.json lists them.
|
||||
func (c *Client) Snapshot() []Answer {
|
||||
@@ -205,7 +233,8 @@ func (c *Client) Load(answers []Answer) {
|
||||
// ask asks routewatch about addr, keeps the answer as client's, and notes the
|
||||
// query as no longer under way. A query that fails gives no answer, and
|
||||
// routewatch is not asked again for failureKept. Only the first failure of
|
||||
// those under way is logged.
|
||||
// those under way is logged. A query ended as ctx ends, by StopAsking, is no
|
||||
// failure.
|
||||
func (c *Client) ask(
|
||||
ctx context.Context,
|
||||
client netip.Prefix,
|
||||
@@ -213,18 +242,20 @@ func (c *Client) ask(
|
||||
) {
|
||||
route, err := c.query(ctx, addr)
|
||||
now := c.params.Now()
|
||||
isFailure := err != nil && ctx.Err() == nil
|
||||
|
||||
c.mu.Lock()
|
||||
|
||||
delete(c.asking, client)
|
||||
|
||||
firstFailure := err != nil && !now.Before(c.retryAt)
|
||||
firstFailure := isFailure && !now.Before(c.retryAt)
|
||||
|
||||
if err == nil {
|
||||
switch {
|
||||
case err == nil:
|
||||
c.answers.Add(client, Answer{
|
||||
Client: client, Netblock: route.Netblock, ASN: route.ASN, Answered: now,
|
||||
})
|
||||
} else {
|
||||
case isFailure:
|
||||
c.retryAt = now.Add(failureKept)
|
||||
}
|
||||
|
||||
|
||||
@@ -361,6 +361,54 @@ func TestHangingRoutewatchNeverDelaysAndFailsTheQuery(t *testing.T) {
|
||||
})
|
||||
}
|
||||
|
||||
func TestStopAskingEndsTheQueryUnderWayAsNoFailureAndAsksNoMore(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
// In a synctest bubble, as TestHangingRoutewatchNeverDelaysAndFailsTheQuery
|
||||
// says.
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
f := &fixture{t: t, log: &syncBuffer{}}
|
||||
f.client = routewatch.New(routewatch.Params{
|
||||
URL: silentURL, Now: time.Now,
|
||||
ProcessLog: slog.New(slog.NewTextHandler(f.log, nil)),
|
||||
})
|
||||
|
||||
f.route()
|
||||
synctest.Wait()
|
||||
|
||||
// The query is ended, not waited out.
|
||||
began := time.Now()
|
||||
|
||||
f.client.StopAsking()
|
||||
|
||||
if waited := time.Since(began); waited != 0 {
|
||||
t.Errorf("StopAsking returned after %s, want at once", waited)
|
||||
}
|
||||
|
||||
// A query started now would fail within waitLimit. None is, and the
|
||||
// one ended is no failure: nothing is logged, and routewatch is not
|
||||
// left alone.
|
||||
for range 2 {
|
||||
if _, missing := f.route(); missing != routewatch.NoAnswerYet {
|
||||
t.Errorf(
|
||||
"Route gave %q, want %q",
|
||||
missing,
|
||||
routewatch.NoAnswerYet,
|
||||
)
|
||||
}
|
||||
|
||||
time.Sleep(waitLimit)
|
||||
synctest.Wait()
|
||||
}
|
||||
|
||||
if f.log.String() != "" {
|
||||
t.Errorf("process log %q, want nothing", f.log.String())
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestRepeatedRequestsMakeOneQueryAndTheAnswerExpires(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -71,6 +71,59 @@ func TestServerHasTheFixedLimits(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// getOfSize is a GET whose request line and headers, the blank line that ends
|
||||
// them included, are size bytes long.
|
||||
func getOfSize(size int) string {
|
||||
const start, end = "GET / HTTP/1.1\r\nHost: site.example\r\nX-Pad: ", "\r\n\r\n"
|
||||
|
||||
return start + strings.Repeat("x", size-len(start)-len(end)) + end
|
||||
}
|
||||
|
||||
func TestARequestLineAndHeadersOf32KiBReachTheApp(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
standIn := startWAF(t, allow)
|
||||
app := startApp(t, greet)
|
||||
addr, _ := startRProxy(t, standIn.URL, app.URL, nil)
|
||||
|
||||
conn := dial(t, addr)
|
||||
send(t, conn, getOfSize(32<<10))
|
||||
|
||||
if status := readStatus(t, conn); status != http.StatusOK {
|
||||
t.Errorf("status %d, want the app's 200", status)
|
||||
}
|
||||
}
|
||||
|
||||
func TestARequestLineAndHeadersOver32KiBAreRefusedAndTheConnectionClosed(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
standIn := startWAF(t, allow)
|
||||
app := startApp(t, func(http.ResponseWriter, *http.Request) {
|
||||
t.Error("the app was asked")
|
||||
})
|
||||
addr, _ := startRProxy(t, standIn.URL, app.URL, nil)
|
||||
|
||||
conn := dial(t, addr)
|
||||
send(t, conn, getOfSize(32<<10+1))
|
||||
|
||||
// io.ReadAll returns without an error only once rproxy has closed the
|
||||
// connection.
|
||||
text, err := io.ReadAll(conn)
|
||||
if err != nil || !strings.HasPrefix(string(text), "HTTP/1.1 431 ") {
|
||||
t.Errorf(
|
||||
"read %q (%v), want 431 and the connection closed",
|
||||
text,
|
||||
err,
|
||||
)
|
||||
}
|
||||
|
||||
if len(standIn.questions) != 0 {
|
||||
t.Error("waf was asked")
|
||||
}
|
||||
}
|
||||
|
||||
func TestABodyAnnouncedOverItsLimitIsRefusedOnceWAFAllowsIt(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -67,7 +67,7 @@ type request struct {
|
||||
// disposition is what is done with the request: waf's valid disposition,
|
||||
// or SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION. It is "" when neither
|
||||
// gave one.
|
||||
disposition string
|
||||
disposition waf.Disposition
|
||||
// answered is true when waf gave disposition, which makes a report due,
|
||||
// and wafNoAnswer when SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION did.
|
||||
answered bool
|
||||
@@ -454,7 +454,7 @@ func (rq *request) finish(ctx context.Context) {
|
||||
requestEnd.Sub(rq.start),
|
||||
),
|
||||
DurationResponseMS: requestlog.Milliseconds(end.Sub(requestEnd)),
|
||||
Disposition: rq.disposition,
|
||||
Disposition: string(rq.disposition),
|
||||
WAFBackendNoAnswer: rq.wafNoAnswer,
|
||||
},
|
||||
)
|
||||
|
||||
@@ -215,7 +215,7 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
// This panic has Go's server close the connection without an answer,
|
||||
// and log nothing; the deferred finish still runs.
|
||||
panic(http.ErrAbortHandler)
|
||||
default: // a deny, an error, or an answer without a valid disposition
|
||||
case waf.DispositionDeny, "": // "" is an error, or no valid disposition
|
||||
rq.answerWith(answer)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -72,9 +72,9 @@ const (
|
||||
|
||||
// What the stand-in for waf answers.
|
||||
const (
|
||||
dispositionAllow = waf.DispositionAllow
|
||||
dispositionDeny = waf.DispositionDeny
|
||||
dispositionClose = waf.DispositionClose
|
||||
dispositionAllow = string(waf.DispositionAllow)
|
||||
dispositionDeny = string(waf.DispositionDeny)
|
||||
dispositionClose = string(waf.DispositionClose)
|
||||
// deniedBody is the body of a deny.
|
||||
deniedBody = "refused by waf"
|
||||
)
|
||||
@@ -324,7 +324,8 @@ func greet(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
// startRProxy starts rproxy with waf at backend and the app at appURL, and
|
||||
// the settings in env on top of the defaults. It returns where rproxy listens
|
||||
// and what it writes.
|
||||
// and what it writes. SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS is a minute unless env
|
||||
// sets it, so that a busy host cannot make rproxy drop a question or a report.
|
||||
func startRProxy(
|
||||
t *testing.T, backend, appURL string, env map[string]string,
|
||||
) (string, *output) {
|
||||
@@ -357,6 +358,7 @@ func startServerLogging(
|
||||
|
||||
settings := map[string]string{
|
||||
"SWW_RPROXY_WAF_BACKEND": backend, "SWW_RPROXY_UPSTREAM_URL": appURL,
|
||||
wafBackendTimeout: "60000",
|
||||
}
|
||||
maps.Copy(settings, env)
|
||||
|
||||
|
||||
@@ -41,7 +41,7 @@ type wafAnswer struct {
|
||||
// notes nothing, and no report is due for it: waf holds no decision to report
|
||||
// on.
|
||||
func (rq *request) noteAnswer(answer *wafAnswer) {
|
||||
disposition := answer.header.Get(waf.DispositionHeader)
|
||||
disposition := waf.Disposition(answer.header.Get(waf.DispositionHeader))
|
||||
|
||||
switch disposition {
|
||||
case waf.DispositionAllow, waf.DispositionDeny,
|
||||
@@ -77,7 +77,9 @@ func (rq *request) handleNoUsableAnswer(ctx context.Context, err error) {
|
||||
// https://git.eeqj.de/sneak/smallwebwaf/issues/182.
|
||||
rq.h.processLog.Warn("waf gave no answer", "error", err.Error())
|
||||
|
||||
rq.disposition = rq.h.config.WAFBackendNoAnswerDisposition
|
||||
rq.disposition = waf.Disposition(
|
||||
rq.h.config.WAFBackendNoAnswerDisposition,
|
||||
)
|
||||
rq.wafNoAnswer = true
|
||||
|
||||
if rq.disposition == waf.DispositionAllow {
|
||||
|
||||
+10
-7
@@ -26,15 +26,19 @@ import (
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
)
|
||||
|
||||
// Action is what a rule does when it matches: ActionLog, ActionBlock or
|
||||
// ActionBan.
|
||||
type Action string
|
||||
|
||||
// The actions a rule takes when it matches.
|
||||
const (
|
||||
// ActionLog notes the match in the request log, and does nothing else.
|
||||
ActionLog = "log"
|
||||
ActionLog Action = "log"
|
||||
// ActionBlock refuses the request with 403.
|
||||
ActionBlock = "block"
|
||||
ActionBlock Action = "block"
|
||||
// ActionBan refuses the request and bans the client's netblock: the
|
||||
// request is a clear sign of attack.
|
||||
ActionBan = "ban"
|
||||
ActionBan Action = "ban"
|
||||
)
|
||||
|
||||
// extension ends the name of every rule file.
|
||||
@@ -95,8 +99,7 @@ type Rule struct {
|
||||
// Target is what the regex is matched against, such as path or
|
||||
// header:Accept.
|
||||
Target string
|
||||
// Action is ActionLog, ActionBlock or ActionBan.
|
||||
Action string
|
||||
Action Action
|
||||
|
||||
regex *regexp.Regexp
|
||||
}
|
||||
@@ -366,7 +369,7 @@ func parse(line string) (Rule, bool, error) {
|
||||
return Rule{}, false, errNotRule
|
||||
}
|
||||
|
||||
rule := Rule{ID: fields[1], Target: fields[2], Action: fields[3]}
|
||||
rule := Rule{ID: fields[1], Target: fields[2], Action: Action(fields[3])}
|
||||
headerName, isHeader := strings.CutPrefix(rule.Target, headerTarget)
|
||||
|
||||
switch {
|
||||
@@ -394,7 +397,7 @@ func parse(line string) (Rule, bool, error) {
|
||||
rule.Target,
|
||||
errHeaderTakenOut,
|
||||
)
|
||||
case !slices.Contains([]string{ActionLog, ActionBlock, ActionBan}, rule.Action):
|
||||
case !slices.Contains([]Action{ActionLog, ActionBlock, ActionBan}, rule.Action):
|
||||
return Rule{}, false, fmt.Errorf(
|
||||
"the action %q %w",
|
||||
rule.Action,
|
||||
|
||||
+16
-13
@@ -46,7 +46,7 @@ import (
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||
@@ -104,7 +104,7 @@ var (
|
||||
errVersion = errors.New("unknown version")
|
||||
// errMissing is for an entry without a field it needs.
|
||||
errMissing = errors.New("has no")
|
||||
errCause = errors.New("is not limit, attack or admin")
|
||||
errCause = errors.New("is not limit, attack, admin or abuseipdb")
|
||||
errExpires = errors.New("is not after its start")
|
||||
errDestination = errors.New("is not webhook, slack or ntfy")
|
||||
errScope = errors.New(
|
||||
@@ -135,7 +135,7 @@ type Params struct {
|
||||
// sww runs.
|
||||
Ledger *bans.Ledger
|
||||
Limiter *ratelimit.Limiter
|
||||
GeoJS *lookup.GeoJS
|
||||
GeoJS *geojs.GeoJS
|
||||
Lists *reputation.Lists
|
||||
DNSBL *reputation.DNSBL
|
||||
AbuseIPDB *reputation.AbuseIPDB
|
||||
@@ -181,7 +181,7 @@ type BanEntry struct {
|
||||
Netblock netip.Prefix `json:"netblock"`
|
||||
Start time.Time `json:"start"`
|
||||
Expires *time.Time `json:"expires"`
|
||||
Cause string `json:"cause"`
|
||||
Cause bans.Cause `json:"cause"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
Lifted *time.Time `json:"lifted,omitempty"`
|
||||
Notes bans.Notes `json:"notes"`
|
||||
@@ -195,8 +195,8 @@ type clientsFile struct {
|
||||
|
||||
// lookupsFile is lookups.json, with each answer on a line of its own.
|
||||
type lookupsFile struct {
|
||||
Version int `json:"version"`
|
||||
Lookups []lookup.Answer `json:"lookups"`
|
||||
Version int `json:"version"`
|
||||
Lookups []geojs.Answer `json:"lookups"`
|
||||
}
|
||||
|
||||
// abuseIPDBChecksSpentFile is abuseipdb-checks-spent.json, indented for an
|
||||
@@ -237,11 +237,11 @@ type routeWatchFile struct {
|
||||
//
|
||||
//nolint:tagliatelle // the state files use snake_case, as the request log does
|
||||
type alertsFile struct {
|
||||
Version int `json:"version"`
|
||||
Cooldowns []alerts.Cooldown `json:"cooldowns"`
|
||||
Hour alerts.Hour `json:"hour"`
|
||||
Waiting map[string][]alerts.Alert `json:"waiting"`
|
||||
AnomalyCounters []anomaly.Counter `json:"anomaly_counters"`
|
||||
Version int `json:"version"`
|
||||
Cooldowns []alerts.Cooldown `json:"cooldowns"`
|
||||
Hour alerts.Hour `json:"hour"`
|
||||
Waiting map[alerts.Destination][]alerts.Alert `json:"waiting"`
|
||||
AnomalyCounters []anomaly.Counter `json:"anomaly_counters"`
|
||||
}
|
||||
|
||||
// stateFile is the struct of a state file. Once the file is decoded, its
|
||||
@@ -638,6 +638,8 @@ func (f *Files) takeInState(
|
||||
f.params.Ledger.Load(held)
|
||||
}
|
||||
|
||||
f.params.Ledger.ForgetEndedAbuseIPDBBans(f.params.Now())
|
||||
|
||||
entries = len(held)
|
||||
case clientsJSON:
|
||||
var file clientsFile
|
||||
@@ -967,7 +969,7 @@ func (e BanEntry) ban() bans.Ban {
|
||||
// each expires is read again as written. An expires not after the start
|
||||
// is refused too, since the netblock's next ban for a broken limit, three
|
||||
// times as long as this one, would then refuse nothing; and so is a cause
|
||||
// other than limit, attack or admin, most likely misspelt.
|
||||
// other than limit, attack, admin or abuseipdb, most likely misspelt.
|
||||
func (f *bansFile) check(data []byte) error {
|
||||
var written struct {
|
||||
Bans []struct {
|
||||
@@ -991,7 +993,8 @@ func (f *bansFile) check(data []byte) error {
|
||||
case entry.Expires != nil && !entry.Expires.After(entry.Start):
|
||||
return fmt.Errorf("entry %d's expires %w", i+1, errExpires)
|
||||
case entry.Cause != "" && entry.Cause != bans.CauseLimit &&
|
||||
entry.Cause != bans.CauseAttack && entry.Cause != bans.CauseAdmin:
|
||||
entry.Cause != bans.CauseAttack && entry.Cause != bans.CauseAdmin &&
|
||||
entry.Cause != bans.CauseAbuseIPDB:
|
||||
return fmt.Errorf(
|
||||
"entry %d's cause %q %w",
|
||||
i+1,
|
||||
|
||||
@@ -26,7 +26,7 @@ import (
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||
@@ -114,7 +114,8 @@ const permanentBansJSON = `{
|
||||
"earlier_bans": {
|
||||
"limit": 3,
|
||||
"attack": 1,
|
||||
"admin": 1
|
||||
"admin": 1,
|
||||
"abuseipdb": 0
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1190,7 +1191,7 @@ func TestBanWithAnotherCauseStopsTheStart(t *testing.T) {
|
||||
`"expires": "2026-10-06T04:00:00Z", "cause": "limit"}, `+
|
||||
`{"netblock": "203.0.113.12/32", "start": "2026-10-06T00:00:00Z", `+
|
||||
`"expires": null, "cause": "atack"}]}`,
|
||||
`: entry 4's cause "atack" is not limit, attack or admin`)
|
||||
`: entry 4's cause "atack" is not limit, attack, admin or abuseipdb`)
|
||||
}
|
||||
|
||||
func TestBanThatDoesNotEndAfterItsStartStopsTheStart(t *testing.T) {
|
||||
@@ -1700,7 +1701,7 @@ func TestEditOfEachFileTakenIn(t *testing.T) {
|
||||
t,
|
||||
lookupsJSON,
|
||||
params.GeoJS.Snapshot(),
|
||||
[]lookup.Answer{
|
||||
[]geojs.Answer{
|
||||
{Client: client, Country: "FR", Answered: midnight()},
|
||||
},
|
||||
)
|
||||
@@ -1757,8 +1758,8 @@ func TestEditOfRouteWatchAndAlertsFilesTakenIn(t *testing.T) {
|
||||
Event: alerts.EventBan, Netblock: netip.MustParsePrefix("198.51.100.0/24"),
|
||||
Sent: midnight(),
|
||||
}},
|
||||
Hour: alerts.Hour{HeldBack: map[string]int{}},
|
||||
Waiting: map[string][]alerts.Alert{
|
||||
Hour: alerts.Hour{HeldBack: map[alerts.Event]int{}},
|
||||
Waiting: map[alerts.Destination][]alerts.Alert{
|
||||
alerts.DestinationWebhook: {
|
||||
{Event: alerts.EventFileError, Time: midnight()},
|
||||
},
|
||||
@@ -2045,8 +2046,8 @@ func TestBanWithoutACauseTakenInAsAnAdmins(t *testing.T) {
|
||||
|
||||
var written struct {
|
||||
Bans []struct {
|
||||
Cause string `json:"cause"`
|
||||
Reason string `json:"reason"`
|
||||
Cause bans.Cause `json:"cause"`
|
||||
Reason string `json:"reason"`
|
||||
} `json:"bans"`
|
||||
}
|
||||
|
||||
@@ -2090,6 +2091,39 @@ func TestLiftedBanReadAtTheStart(t *testing.T) {
|
||||
wantLiftedBanKept(t, load(t, params), dir, params.Ledger)
|
||||
}
|
||||
|
||||
func TestEndedAbuseIPDBBanForgottenWhenBansJSONIsReadAtTheStart(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
ended := netip.MustParsePrefix("203.0.113.9/32")
|
||||
lasting := netip.MustParsePrefix("203.0.113.10/32")
|
||||
limited := netip.MustParsePrefix("203.0.113.11/32")
|
||||
|
||||
// At midnight, the start's time, the first ban has ended and the second
|
||||
// lasts. The third has ended too, but is for a limit.
|
||||
dir := t.TempDir()
|
||||
edit(t, dir, bansJSON, `{"version": 1, "bans": [`+
|
||||
`{"netblock": "203.0.113.9/32", "start": "2026-09-06T00:00:00Z", `+
|
||||
`"expires": "2026-10-06T00:00:00Z", "cause": "abuseipdb"}, `+
|
||||
`{"netblock": "203.0.113.10/32", "start": "2026-09-07T00:00:00Z", `+
|
||||
`"expires": "2026-10-07T00:00:00Z", "cause": "abuseipdb"}, `+
|
||||
`{"netblock": "203.0.113.11/32", "start": "2026-10-05T00:00:00Z", `+
|
||||
`"expires": "2026-10-05T01:00:00Z", "cause": "limit"}]}`)
|
||||
|
||||
params := newParams(dir)
|
||||
load(t, params)
|
||||
|
||||
for netblock, want := range map[netip.Prefix]int{
|
||||
ended: 0, lasting: 1, limited: 1,
|
||||
} {
|
||||
if got := len(params.Ledger.Bans(netblock)); got != want {
|
||||
t.Errorf("the ledger holds %d bans on %s, want %d",
|
||||
got, netblock, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestBanLiftedByAnEditWhileRunning(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -2415,9 +2449,7 @@ func newParams(dir string) state.Params {
|
||||
MaxBans: 5000,
|
||||
}),
|
||||
Limiter: ratelimit.New(ratelimit.Limits{}, 20000),
|
||||
GeoJS: lookup.New(lookup.Params{
|
||||
Now: midnight, ProcessLog: discard, Metrics: m,
|
||||
}),
|
||||
GeoJS: geojs.New(geojs.Params{Now: midnight}),
|
||||
Lists: reputation.New(reputation.Params{
|
||||
BlocklistURLs: []string{
|
||||
blocklistURL,
|
||||
@@ -2494,7 +2526,7 @@ func fill(params state.Params) {
|
||||
})
|
||||
params.Limiter.AddLookup(client, now.Add(-time.Hour), asn, asName, "DE")
|
||||
|
||||
params.GeoJS.Load([]lookup.Answer{
|
||||
params.GeoJS.Load([]geojs.Answer{
|
||||
{
|
||||
Client: client, ASN: asn, ASName: asName, Country: "DE",
|
||||
Answered: now.Add(-time.Hour), Used: now,
|
||||
|
||||
@@ -0,0 +1,198 @@
|
||||
package sww
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/http"
|
||||
"time"
|
||||
|
||||
"go.uber.org/fx"
|
||||
"go.uber.org/fx/fxevent"
|
||||
)
|
||||
|
||||
// shutdownTimeout is how long requests in progress may take to finish
|
||||
// once sww is told to stop, before their connections are closed.
|
||||
// runit and docker wait a little longer before they kill the process.
|
||||
const shutdownTimeout = 5 * time.Second
|
||||
|
||||
// requestLog is where a daemon's server writes the request log: stdout, and
|
||||
// the remote log sender while there is one. fx tells parts apart by type, so
|
||||
// it has a type of its own.
|
||||
type requestLog struct {
|
||||
io.Writer
|
||||
}
|
||||
|
||||
// servingFailed gets why serving failed, for run, which then stops the
|
||||
// daemon. fx tells parts apart by type, so it has a type of its own.
|
||||
type servingFailed chan error
|
||||
|
||||
// utcNow is the time the state files, the alerts and the rate limits go by.
|
||||
func utcNow() time.Time {
|
||||
return time.Now().UTC()
|
||||
}
|
||||
|
||||
// startError is why a daemon cannot start: the message its process log gives,
|
||||
// and the error it logs with it.
|
||||
type startError struct {
|
||||
message string
|
||||
err error
|
||||
}
|
||||
|
||||
func (e startError) Error() string {
|
||||
return e.message + ": " + e.err.Error()
|
||||
}
|
||||
|
||||
func (e startError) Unwrap() error {
|
||||
return e.err
|
||||
}
|
||||
|
||||
// cannotStart logs why a daemon cannot start, with the message of the
|
||||
// startError in err, and returns its exit status.
|
||||
func cannotStart(processLog *slog.Logger, err error) int {
|
||||
message, cause := "cannot start", err
|
||||
|
||||
failed, ok := errors.AsType[startError](err)
|
||||
if ok {
|
||||
message, cause = failed.message, failed.err
|
||||
}
|
||||
|
||||
processLog.Error(message, "error", cause.Error())
|
||||
|
||||
return 1
|
||||
}
|
||||
|
||||
// listen listens on address, which the setting names.
|
||||
func listen(address, setting string) (net.Listener, error) {
|
||||
listener, err := (&net.ListenConfig{}).Listen(
|
||||
context.Background(), "tcp", address)
|
||||
if err != nil {
|
||||
return nil, startError{"cannot listen on " + setting, err}
|
||||
}
|
||||
|
||||
return listener, nil
|
||||
}
|
||||
|
||||
// background returns the hook that runs loop on a goroutine of its own from
|
||||
// the start, and at the stop ends it and waits for it to return.
|
||||
func background(loop func(context.Context)) fx.Hook {
|
||||
running, stop := context.WithCancel(context.Background())
|
||||
ended := make(chan struct{})
|
||||
|
||||
return fx.StartStopHook(
|
||||
func() {
|
||||
go func() {
|
||||
loop(running)
|
||||
close(ended)
|
||||
}()
|
||||
},
|
||||
func() {
|
||||
stop()
|
||||
<-ended
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
// serve serves requests on listener from the start. At the stop it gives the
|
||||
// requests in progress shutdownTimeout to finish, then closes their
|
||||
// connections. When serving fails, it logs why and sends the error to
|
||||
// failed.
|
||||
func serve(
|
||||
lc fx.Lifecycle, server *http.Server, listener net.Listener,
|
||||
processLog *slog.Logger, failed servingFailed,
|
||||
) {
|
||||
served := make(chan error, 1)
|
||||
|
||||
lc.Append(fx.Hook{
|
||||
OnStart: func(context.Context) error {
|
||||
go func() {
|
||||
err := server.Serve(listener)
|
||||
if !errors.Is(err, http.ErrServerClosed) {
|
||||
processLog.Error("serving failed", "error", err.Error())
|
||||
|
||||
failed <- err
|
||||
}
|
||||
|
||||
served <- err
|
||||
}()
|
||||
|
||||
return nil
|
||||
},
|
||||
OnStop: func(ctx context.Context) error {
|
||||
processLog.Info("stopping")
|
||||
|
||||
shutdownCtx, cancel := context.WithTimeout(ctx, shutdownTimeout)
|
||||
defer cancel()
|
||||
|
||||
err := server.Shutdown(shutdownCtx)
|
||||
if err != nil {
|
||||
processLog.Warn("requests still in progress were cut off",
|
||||
"error", err.Error())
|
||||
|
||||
_ = server.Close()
|
||||
}
|
||||
|
||||
err = <-served
|
||||
if !errors.Is(err, http.ErrServerClosed) {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
// run starts app, which serves until ctx is done or serving fails, as
|
||||
// failed tells. Then it stops app and takes the daemon's last step. It
|
||||
// returns the process's exit status, 1 when serving failed, a part could
|
||||
// not stop in order or the last step failed; each logs why.
|
||||
func run(
|
||||
ctx context.Context, app *fx.App, failed servingFailed,
|
||||
processLog *slog.Logger, lastStep func() error,
|
||||
) int {
|
||||
// The hooks only start goroutines, and end them at the stop, so neither
|
||||
// is cut short by a deadline or by ctx.
|
||||
err := app.Start(context.WithoutCancel(ctx))
|
||||
if err != nil {
|
||||
return cannotStart(processLog, err)
|
||||
}
|
||||
|
||||
status := 0
|
||||
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
case <-failed:
|
||||
status = 1
|
||||
}
|
||||
|
||||
err = app.Stop(context.WithoutCancel(ctx))
|
||||
if err != nil {
|
||||
status = 1
|
||||
}
|
||||
|
||||
err = lastStep()
|
||||
if err != nil {
|
||||
status = 1
|
||||
}
|
||||
|
||||
if status == 0 {
|
||||
processLog.Info("stopped")
|
||||
}
|
||||
|
||||
return status
|
||||
}
|
||||
|
||||
// fxEventLogger returns what fx.WithLogger takes to write fx's own events to
|
||||
// processLog at debug, which the daemons' log levels normally hold back. Its
|
||||
// errors are at debug too: the daemon logs each one itself.
|
||||
func fxEventLogger(processLog *slog.Logger) func() fxevent.Logger {
|
||||
return func() fxevent.Logger {
|
||||
logger := &fxevent.SlogLogger{Logger: processLog}
|
||||
logger.UseLogLevel(slog.LevelDebug)
|
||||
logger.UseErrorLevel(slog.LevelDebug)
|
||||
|
||||
return logger
|
||||
}
|
||||
}
|
||||
+42
-63
@@ -2,12 +2,13 @@ package sww
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
|
||||
"go.uber.org/fx"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
"sneak.berlin/go/smallwebwaf/internal/rproxy"
|
||||
@@ -30,26 +31,31 @@ func RunRProxy(ctx context.Context, params Params) int {
|
||||
|
||||
cfg, err := config.RProxyFromEnvironment(params.Viper)
|
||||
if err != nil {
|
||||
processLog.Error("invalid setting", "error", err.Error())
|
||||
|
||||
return 1
|
||||
return cannotStart(processLog, startError{"invalid setting", err})
|
||||
}
|
||||
|
||||
server := rproxy.New(rproxy.Params{
|
||||
Config: cfg,
|
||||
RequestLog: params.Stdout,
|
||||
ProcessLog: processLog,
|
||||
})
|
||||
var (
|
||||
server *rproxy.Server
|
||||
listener net.Listener
|
||||
)
|
||||
|
||||
listener, err := (&net.ListenConfig{}).Listen(ctx, "tcp", cfg.ListenAddr)
|
||||
failed := make(servingFailed, 1)
|
||||
|
||||
app := fx.New(
|
||||
fx.Supply(cfg, processLog, requestLog{params.Stdout}, failed),
|
||||
fx.Provide(
|
||||
newRProxyServer,
|
||||
func(server *rproxy.Server) *http.Server { return server.Server },
|
||||
newRProxyListener,
|
||||
),
|
||||
fx.Invoke(serve),
|
||||
fx.Populate(&server, &listener),
|
||||
fx.WithLogger(fxEventLogger(processLog)),
|
||||
)
|
||||
|
||||
err = app.Err()
|
||||
if err != nil {
|
||||
processLog.Error(
|
||||
"cannot listen on SWW_RPROXY_LISTEN_ADDR",
|
||||
"error",
|
||||
err.Error(),
|
||||
)
|
||||
|
||||
return 1
|
||||
return cannotStart(processLog, err)
|
||||
}
|
||||
|
||||
processLog.Info("starting",
|
||||
@@ -57,52 +63,25 @@ func RunRProxy(ctx context.Context, params Params) int {
|
||||
"address", listener.Addr().String(),
|
||||
"settings", cfg)
|
||||
|
||||
return serveRProxy(ctx, server, listener, processLog)
|
||||
return run(ctx, app, failed, processLog, func() error {
|
||||
server.WaitForReports()
|
||||
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// serveRProxy serves requests on listener until ctx is done, and then stops as
|
||||
// RunRProxy says.
|
||||
func serveRProxy(
|
||||
ctx context.Context, server *rproxy.Server, listener net.Listener,
|
||||
processLog *slog.Logger,
|
||||
) int {
|
||||
served := make(chan error, 1)
|
||||
|
||||
go func() {
|
||||
served <- server.Serve(listener)
|
||||
}()
|
||||
|
||||
select {
|
||||
case err := <-served:
|
||||
processLog.Error("serving failed", "error", err.Error())
|
||||
|
||||
return 1
|
||||
case <-ctx.Done():
|
||||
}
|
||||
|
||||
processLog.Info("stopping")
|
||||
|
||||
shutdownCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx),
|
||||
shutdownTimeout)
|
||||
defer cancel()
|
||||
|
||||
err := server.Shutdown(shutdownCtx)
|
||||
if err != nil {
|
||||
processLog.Warn("requests still in progress were cut off",
|
||||
"error", err.Error())
|
||||
|
||||
_ = server.Close()
|
||||
}
|
||||
|
||||
err = <-served
|
||||
if !errors.Is(err, http.ErrServerClosed) {
|
||||
processLog.Error("serving failed", "error", err.Error())
|
||||
|
||||
return 1
|
||||
}
|
||||
|
||||
server.WaitForReports()
|
||||
processLog.Info("stopped")
|
||||
|
||||
return 0
|
||||
// newRProxyServer returns the server rproxy runs.
|
||||
func newRProxyServer(
|
||||
cfg *config.RProxy, out requestLog, processLog *slog.Logger,
|
||||
) *rproxy.Server {
|
||||
return rproxy.New(rproxy.Params{
|
||||
Config: cfg,
|
||||
RequestLog: out,
|
||||
ProcessLog: processLog,
|
||||
})
|
||||
}
|
||||
|
||||
// newRProxyListener listens on SWW_RPROXY_LISTEN_ADDR.
|
||||
func newRProxyListener(cfg *config.RProxy) (net.Listener, error) {
|
||||
return listen(cfg.ListenAddr, "SWW_RPROXY_LISTEN_ADDR")
|
||||
}
|
||||
@@ -51,7 +51,9 @@ func runRProxy(ctx context.Context, env map[string]string, out *output) int {
|
||||
|
||||
// startRProxy runs rproxy with waf at backend and the app at appURL, and
|
||||
// returns its URL and what it writes. It believes X-Forwarded-For from the
|
||||
// tests. It stops as the test ends, and that it stopped in order is checked.
|
||||
// tests. It waits a minute for waf's answer, so that a slow answer on a busy
|
||||
// host is not taken for no answer. It stops as the test ends, and that it
|
||||
// stopped in order is checked.
|
||||
func startRProxy(t *testing.T, backend, appURL string) (string, *output) {
|
||||
t.Helper()
|
||||
|
||||
@@ -65,6 +67,7 @@ func startRProxy(t *testing.T, backend, appURL string) (string, *output) {
|
||||
"SWW_RPROXY_UPSTREAM_URL": appURL,
|
||||
"SWW_RPROXY_TRUSTED_PROXIES": localhost + "/32",
|
||||
wafBackend: backend,
|
||||
wafBackendTimeout: "60000",
|
||||
}, out)
|
||||
}()
|
||||
|
||||
@@ -161,7 +164,7 @@ func TestABanThroughOneSiteRefusesTheClientOnTheOther(t *testing.T) {
|
||||
|
||||
// One request's three lines carry one id, rproxy's with waf's answer.
|
||||
line := firstOut.line(t, "path", bannedProbe)
|
||||
if line["disposition"] != waf.DispositionDeny ||
|
||||
if line["disposition"] != string(waf.DispositionDeny) ||
|
||||
line["client_ip"] != placed {
|
||||
t.Errorf("rproxy's line is %v", line)
|
||||
}
|
||||
@@ -246,7 +249,7 @@ func TestTheSiteAnswersWhileWAFIsStoppedAndStartedAgain(t *testing.T) {
|
||||
wantStatus(t, site+"/after-stop", placed, http.StatusOK)
|
||||
|
||||
line := out.line(t, "path", "/after-stop")
|
||||
if line["disposition"] != waf.DispositionAllow ||
|
||||
if line["disposition"] != string(waf.DispositionAllow) ||
|
||||
line["waf_backend_no_answer"] != true {
|
||||
t.Errorf("rproxy's line is %v", line)
|
||||
}
|
||||
@@ -549,7 +552,9 @@ func TestAStatusWAFDoesNotTakeIsReportedAs599(t *testing.T) {
|
||||
// startRProxyProcess runs `sww rproxy` in a process of its own, with the
|
||||
// settings in env, each as NAME=value. It returns where rproxy listens and a
|
||||
// function that sends rproxy SIGTERM and fails the test unless it exits with
|
||||
// status 0 within waitLimit.
|
||||
// status 0 within waitLimit. It waits a minute for waf's answers unless env
|
||||
// sets SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS, so that a busy host cannot make it
|
||||
// drop a question.
|
||||
func startRProxyProcess(t *testing.T, env ...string) (string, func()) {
|
||||
t.Helper()
|
||||
|
||||
@@ -565,7 +570,8 @@ func startRProxyProcess(t *testing.T, env ...string) (string, func()) {
|
||||
|
||||
cmd.Env = append(os.Environ(),
|
||||
commandForMain+"=rproxy",
|
||||
"SWW_RPROXY_LISTEN_ADDR="+localhost+":0")
|
||||
"SWW_RPROXY_LISTEN_ADDR="+localhost+":0",
|
||||
wafBackendTimeout+"=60000")
|
||||
cmd.Env = append(cmd.Env, env...)
|
||||
|
||||
exited := startProcess(t, cmd)
|
||||
@@ -602,7 +608,8 @@ func TestRProxyAsksAWAFThatSpeaksHTTP2OverHTTP1(t *testing.T) {
|
||||
if r.Header.Get(waf.ClientHeader) != "" {
|
||||
protocols <- r.Proto
|
||||
|
||||
w.Header().Set(waf.DispositionHeader, waf.DispositionDeny)
|
||||
w.Header().
|
||||
Set(waf.DispositionHeader, string(waf.DispositionDeny))
|
||||
w.WriteHeader(http.StatusForbidden)
|
||||
}
|
||||
}))
|
||||
@@ -685,7 +692,8 @@ func TestRProxyStoppingWaitsForAReportBeingSent(t *testing.T) {
|
||||
backend := httptest.NewServer(http.HandlerFunc(
|
||||
func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Header.Get(waf.ClientHeader) != "" {
|
||||
w.Header().Set(waf.DispositionHeader, waf.DispositionAllow)
|
||||
w.Header().
|
||||
Set(waf.DispositionHeader, string(waf.DispositionAllow))
|
||||
|
||||
return
|
||||
}
|
||||
@@ -796,7 +804,8 @@ func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
|
||||
|
||||
backend := httptest.NewServer(http.HandlerFunc(
|
||||
func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set(waf.DispositionHeader, waf.DispositionAllow)
|
||||
w.Header().
|
||||
Set(waf.DispositionHeader, string(waf.DispositionAllow))
|
||||
}))
|
||||
t.Cleanup(backend.Close)
|
||||
|
||||
|
||||
+5
-393
@@ -4,17 +4,14 @@
|
||||
// stop, and then stops in an orderly way, writing the state and cache files.
|
||||
// `sww rproxy` reads rproxy's settings alone, and runs the reverse proxy in
|
||||
// front of one app that asks waf about each request. `sww healthcheck` is the
|
||||
// container's health check.
|
||||
// container's health check. fx builds the parts of each daemon, starts their
|
||||
// background work and stops it.
|
||||
package sww
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
@@ -22,28 +19,9 @@ import (
|
||||
|
||||
"github.com/spf13/viper"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||
"sneak.berlin/go/smallwebwaf/internal/remotelog"
|
||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||
"sneak.berlin/go/smallwebwaf/internal/state"
|
||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||
)
|
||||
|
||||
// shutdownTimeout is how long requests in progress may take to finish
|
||||
// once sww is told to stop, before their connections are closed.
|
||||
// runit and docker wait a little longer before they kill the process.
|
||||
const shutdownTimeout = 5 * time.Second
|
||||
|
||||
// remoteLogStopTimeout is how long, as waf stops, the log lines still
|
||||
// waiting are sent to SWW_WAF_LOG_REMOTE_URL before they are given up. stdout has
|
||||
// carried them.
|
||||
const remoteLogStopTimeout = 2 * time.Second
|
||||
|
||||
// usageStatus is the exit status with no command, or one sww does not
|
||||
// know.
|
||||
const usageStatus = 2
|
||||
@@ -66,6 +44,9 @@ type Params struct {
|
||||
Viper *viper.Viper
|
||||
// Stdout receives the request log and the process's own messages.
|
||||
Stdout io.Writer
|
||||
// BesideLookupTimeLimit is the waf.Params.BesideLookupTimeLimit RunWAF
|
||||
// gives waf. Main leaves it zero, waf's own limit; tests lengthen it.
|
||||
BesideLookupTimeLimit time.Duration
|
||||
}
|
||||
|
||||
// Main runs the command os.Args names, and returns the process's exit status.
|
||||
@@ -106,372 +87,3 @@ func printUsage(w io.Writer) int {
|
||||
|
||||
return usageStatus
|
||||
}
|
||||
|
||||
// RunWAF is `sww waf`: it reads the settings, the rule files,
|
||||
// the lookup database and the state and cache files, then serves requests until
|
||||
// ctx is done. It returns the process's exit status, 1 when waf cannot start.
|
||||
func RunWAF(ctx context.Context, params Params) int {
|
||||
// Until the settings are read, the one message is an invalid setting's
|
||||
// error, which every SWW_WAF_LOG_LEVEL lets through.
|
||||
processLog := requestlog.NewProcessLogger(params.Stdout,
|
||||
config.InstanceName(params.Viper), slog.LevelError)
|
||||
|
||||
cfg, err := config.WAFFromEnvironment(params.Viper)
|
||||
if err != nil {
|
||||
return cannotStart(processLog, "invalid setting", err)
|
||||
}
|
||||
|
||||
// While SWW_WAF_LOG_REMOTE_URL is set, every line on stdout from here on
|
||||
// is sent there too.
|
||||
stdout := params.Stdout
|
||||
|
||||
var remote *remotelog.Sender
|
||||
|
||||
if cfg.LogRemoteURL != nil {
|
||||
remote = newRemoteLogSender(cfg)
|
||||
stdout = io.MultiWriter(params.Stdout, remote)
|
||||
}
|
||||
|
||||
processLog = requestlog.NewProcessLogger(
|
||||
stdout,
|
||||
cfg.InstanceName,
|
||||
cfg.LogLevel,
|
||||
)
|
||||
|
||||
if remote != nil {
|
||||
stopSending := startSending(ctx, remote, processLog)
|
||||
defer stopSending()
|
||||
}
|
||||
|
||||
return runWAFWithSettings(
|
||||
ctx,
|
||||
params.Version,
|
||||
cfg,
|
||||
stdout,
|
||||
remote,
|
||||
processLog,
|
||||
)
|
||||
}
|
||||
|
||||
// runWAFWithSettings is the rest of RunWAF once the settings are read and
|
||||
// the process log goes where they say. remote is nil unless
|
||||
// SWW_WAF_LOG_REMOTE_URL is set.
|
||||
func runWAFWithSettings(
|
||||
ctx context.Context,
|
||||
version string,
|
||||
cfg *config.WAF,
|
||||
stdout io.Writer,
|
||||
remote *remotelog.Sender,
|
||||
processLog *slog.Logger,
|
||||
) int {
|
||||
// The state files and the alerts give times in UTC.
|
||||
now := func() time.Time { return time.Now().UTC() }
|
||||
|
||||
alertQueue := newAlertQueue(cfg, now, processLog)
|
||||
|
||||
ruleFiles, err := rules.Load(rules.Params{
|
||||
Dir: cfg.RuleFilesDir,
|
||||
Enabled: cfg.RuleFilesEnabled,
|
||||
ProcessLog: processLog,
|
||||
Alerts: alertQueue,
|
||||
})
|
||||
if err != nil {
|
||||
return cannotStart(processLog, "cannot use the rule files", err)
|
||||
}
|
||||
|
||||
crs, err := waf.NewCoreRuleSet(cfg)
|
||||
if err != nil {
|
||||
return cannotStart(processLog, "cannot use the Core Rule Set", err)
|
||||
}
|
||||
|
||||
server, err := newServer(
|
||||
cfg,
|
||||
stdout,
|
||||
processLog,
|
||||
now,
|
||||
ruleFiles,
|
||||
alertQueue,
|
||||
crs,
|
||||
)
|
||||
if err != nil {
|
||||
return cannotStart(processLog, "cannot use the lookup database", err)
|
||||
}
|
||||
|
||||
if remote != nil {
|
||||
server.Metrics.AddRemoteLog(remote)
|
||||
}
|
||||
|
||||
files, err := loadStateFiles(cfg, server, alertQueue, now, processLog)
|
||||
if err != nil {
|
||||
return cannotStart(
|
||||
processLog,
|
||||
"cannot use the state and cache files",
|
||||
err,
|
||||
)
|
||||
}
|
||||
|
||||
listener, err := (&net.ListenConfig{}).Listen(ctx, "tcp", cfg.ListenAddr)
|
||||
if err != nil {
|
||||
return cannotStart(
|
||||
processLog,
|
||||
"cannot listen on SWW_WAF_LISTEN_ADDR",
|
||||
err,
|
||||
)
|
||||
}
|
||||
|
||||
processLog.Info("starting",
|
||||
"version", version,
|
||||
"address", listener.Addr().String(),
|
||||
"settings", cfg)
|
||||
|
||||
return serve(
|
||||
ctx,
|
||||
server,
|
||||
listener,
|
||||
files,
|
||||
ruleFiles,
|
||||
alertQueue,
|
||||
processLog,
|
||||
)
|
||||
}
|
||||
|
||||
// cannotStart logs why waf cannot start, and returns its exit status.
|
||||
func cannotStart(processLog *slog.Logger, message string, err error) int {
|
||||
processLog.Error(message, "error", err.Error())
|
||||
|
||||
return 1
|
||||
}
|
||||
|
||||
// newServer returns the server waf runs, with the metrics of the alerts,
|
||||
// after reading the lookup database while SWW_WAF_LOOKUP_SOURCE is file. A lookup
|
||||
// database that cannot be read is an error.
|
||||
func newServer(
|
||||
cfg *config.WAF, stdout io.Writer, processLog *slog.Logger,
|
||||
now func() time.Time, ruleFiles *rules.Files, alertQueue *alerts.Queue,
|
||||
crs *coreruleset.CoreRuleSet,
|
||||
) (*waf.Server, error) {
|
||||
var lookupFile *lookup.File
|
||||
|
||||
if cfg.LookupSource == "file" {
|
||||
var err error
|
||||
|
||||
lookupFile, err = lookup.OpenFile(lookup.FileParams{
|
||||
Path: cfg.LookupDBPath,
|
||||
Now: now,
|
||||
ProcessLog: processLog,
|
||||
Alerts: alertQueue,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
server := waf.New(waf.Params{
|
||||
Config: cfg,
|
||||
RequestLog: stdout,
|
||||
ProcessLog: processLog,
|
||||
GeoJSURL: lookup.URL,
|
||||
AbuseIPDBURL: reputation.AbuseIPDBURL,
|
||||
LookupFile: lookupFile,
|
||||
CoreRuleSet: crs,
|
||||
Now: now,
|
||||
Rules: ruleFiles,
|
||||
Alerts: alertQueue,
|
||||
})
|
||||
server.Metrics.AddAlerts(alertQueue)
|
||||
|
||||
if lookupFile != nil {
|
||||
server.Metrics.AddLookupFile(
|
||||
lookupFile.LastRead,
|
||||
lookupFile.ReadFailures,
|
||||
)
|
||||
}
|
||||
|
||||
return server, nil
|
||||
}
|
||||
|
||||
// loadStateFiles reads the state and cache files into the parts of server and
|
||||
// into alertQueue, as state.Load does.
|
||||
func loadStateFiles(
|
||||
cfg *config.WAF, server *waf.Server, alertQueue *alerts.Queue,
|
||||
now func() time.Time, processLog *slog.Logger,
|
||||
) (*state.Files, error) {
|
||||
return state.Load(state.Params{
|
||||
Dir: cfg.StateDir,
|
||||
CacheDir: cfg.CacheDir,
|
||||
BansWriteDelay: cfg.BansWriteDelay,
|
||||
WriteInterval: cfg.StateWriteInterval,
|
||||
Ledger: server.Ledger,
|
||||
Limiter: server.Limiter,
|
||||
GeoJS: server.GeoJS,
|
||||
Lists: server.Lists,
|
||||
DNSBL: server.DNSBL,
|
||||
AbuseIPDB: server.AbuseIPDB,
|
||||
RouteWatch: server.RouteWatch,
|
||||
Alerts: alertQueue,
|
||||
Anomalies: server.Anomalies,
|
||||
Now: now,
|
||||
ProcessLog: processLog,
|
||||
Metrics: server.Metrics,
|
||||
})
|
||||
}
|
||||
|
||||
// newAlertQueue returns the queue of the alerts to the webhook, Slack and
|
||||
// ntfy, with the settings for them.
|
||||
func newAlertQueue(
|
||||
cfg *config.WAF, now func() time.Time, processLog *slog.Logger,
|
||||
) *alerts.Queue {
|
||||
return alerts.New(alerts.Params{
|
||||
WebhookURL: cfg.AlertWebhookURL,
|
||||
WebhookHeaders: cfg.AlertWebhookHeaders,
|
||||
SlackURL: cfg.AlertSlackWebhookURL,
|
||||
NtfyURL: cfg.AlertNtfyURL,
|
||||
NtfyToken: cfg.AlertNtfyToken,
|
||||
Events: cfg.AlertEvents,
|
||||
Cooldown: cfg.AlertCooldown,
|
||||
MaxPerHour: cfg.AlertMaxPerHour,
|
||||
Instance: cfg.InstanceName,
|
||||
Now: now,
|
||||
ProcessLog: processLog,
|
||||
})
|
||||
}
|
||||
|
||||
// newRemoteLogSender returns a sender of the log lines to
|
||||
// SWW_WAF_LOG_REMOTE_URL, with the settings for it.
|
||||
func newRemoteLogSender(cfg *config.WAF) *remotelog.Sender {
|
||||
return remotelog.New(remotelog.Params{
|
||||
URL: cfg.LogRemoteURL,
|
||||
RootCAs: cfg.LogRemoteTLSCAs,
|
||||
BufferLines: cfg.LogRemoteBufferLines,
|
||||
Facility: cfg.LogRemoteFacility,
|
||||
AppName: cfg.LogRemoteAppName,
|
||||
})
|
||||
}
|
||||
|
||||
// startSending runs remote until the function it returns is called, which
|
||||
// then waits at most remoteLogStopTimeout for the lines still waiting to
|
||||
// be sent. Sending goes on after ctx is done, so that the lines written
|
||||
// while sww stops are sent too.
|
||||
func startSending(
|
||||
ctx context.Context, remote *remotelog.Sender, processLog *slog.Logger,
|
||||
) func() {
|
||||
sending, stop := context.WithCancel(context.WithoutCancel(ctx))
|
||||
sent := make(chan struct{})
|
||||
|
||||
go func() {
|
||||
remote.Run(sending, processLog)
|
||||
close(sent)
|
||||
}()
|
||||
|
||||
return func() {
|
||||
stop()
|
||||
|
||||
select {
|
||||
case <-sent:
|
||||
case <-time.After(remoteLogStopTimeout):
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// serve serves requests on listener, writes the state and cache files as they
|
||||
// are due, takes in an admin's edits of them, reads the rule files again as
|
||||
// they change, and the lookup database when it is replaced, fetches the
|
||||
// lists the settings name by URL as they are due, and sends the alerts,
|
||||
// until ctx is done. Then it gives the requests in progress
|
||||
// shutdownTimeout to finish, and writes every state and cache file,
|
||||
// alerts.json with the alerts still waiting.
|
||||
func serve(
|
||||
ctx context.Context, server *waf.Server, listener net.Listener,
|
||||
files *state.Files, ruleFiles *rules.Files, alertQueue *alerts.Queue,
|
||||
processLog *slog.Logger,
|
||||
) int {
|
||||
served := make(chan error, 1)
|
||||
|
||||
go func() {
|
||||
served <- server.Serve(listener)
|
||||
}()
|
||||
|
||||
writing, stopWriting := context.WithCancel(ctx)
|
||||
defer stopWriting()
|
||||
|
||||
written := inBackground(func() { files.Run(writing) })
|
||||
watched := inBackground(func() { files.Watch(writing) })
|
||||
rulesWatched := inBackground(func() { ruleFiles.Watch(writing) })
|
||||
lookupFileWatched := inBackground(func() {
|
||||
if server.LookupFile != nil {
|
||||
server.LookupFile.Watch(writing)
|
||||
}
|
||||
})
|
||||
listsFetched := inBackground(func() { server.Lists.Run(writing) })
|
||||
alertsSent := inBackground(func() { alertQueue.Run(writing) })
|
||||
|
||||
select {
|
||||
case err := <-served:
|
||||
processLog.Error("serving failed", "error", err.Error())
|
||||
|
||||
return 1
|
||||
case <-ctx.Done():
|
||||
}
|
||||
|
||||
processLog.Info("stopping")
|
||||
|
||||
shutdownCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx),
|
||||
shutdownTimeout)
|
||||
defer cancel()
|
||||
|
||||
err := server.Shutdown(shutdownCtx)
|
||||
if err != nil {
|
||||
processLog.Warn("requests still in progress were cut off",
|
||||
"error", err.Error())
|
||||
|
||||
_ = server.Close()
|
||||
}
|
||||
|
||||
err = <-served
|
||||
if !errors.Is(err, http.ErrServerClosed) {
|
||||
processLog.Error("serving failed", "error", err.Error())
|
||||
|
||||
return 1
|
||||
}
|
||||
|
||||
// Run and Watch have ended, so nothing else reads or writes the
|
||||
// files, and no alert is being sent, so that alerts.json keeps every
|
||||
// alert not yet sent. Every request has ended too, but for one cut off
|
||||
// because Shutdown timed out, which adds to its client's history only as
|
||||
// it ends. That can be after this write, and then that request is missing
|
||||
// from clients.json.
|
||||
<-written
|
||||
<-watched
|
||||
<-rulesWatched
|
||||
<-lookupFileWatched
|
||||
<-listsFetched
|
||||
<-alertsSent
|
||||
|
||||
err = files.WriteAll()
|
||||
if err != nil {
|
||||
processLog.Error(
|
||||
"writing the state and cache files failed",
|
||||
"error",
|
||||
err.Error(),
|
||||
)
|
||||
|
||||
return 1
|
||||
}
|
||||
|
||||
processLog.Info("stopped")
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
// inBackground runs task on a goroutine of its own, and returns a channel
|
||||
// that is closed once task has returned.
|
||||
func inBackground(task func()) <-chan struct{} {
|
||||
done := make(chan struct{})
|
||||
|
||||
go func() {
|
||||
task()
|
||||
close(done)
|
||||
}()
|
||||
|
||||
return done
|
||||
}
|
||||
@@ -173,6 +173,9 @@ func run(ctx context.Context, env map[string]string, out *output) int {
|
||||
Version: testVersion,
|
||||
Viper: v,
|
||||
Stdout: out,
|
||||
// The race detector and a busy host can slow the Core Rule Set's
|
||||
// inspections past waf's own limit.
|
||||
BesideLookupTimeLimit: time.Hour,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -657,7 +660,9 @@ func TestCoreRuleSetExclusionsFileThatCannotBeUsedStopsTheStartWhileTheCoreRuleS
|
||||
|
||||
status := run(t.Context(), map[string]string{
|
||||
listenAddr: localhost + ":0", stateDir: t.TempDir(), ruleFilesDir: t.TempDir(),
|
||||
"SWW_WAF_CORE_RULE_SET_MODE": config.CoreRuleSetModeOff,
|
||||
"SWW_WAF_CORE_RULE_SET_MODE": string(
|
||||
config.CoreRuleSetModeOff,
|
||||
),
|
||||
"SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE": path,
|
||||
}, out)
|
||||
if status != 1 {
|
||||
|
||||
@@ -0,0 +1,554 @@
|
||||
package sww
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/http"
|
||||
"time"
|
||||
|
||||
"go.uber.org/fx"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||
"sneak.berlin/go/smallwebwaf/internal/remotelog"
|
||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
"sneak.berlin/go/smallwebwaf/internal/routewatch"
|
||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||
"sneak.berlin/go/smallwebwaf/internal/state"
|
||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||
)
|
||||
|
||||
// remoteLogStopTimeout is how long, as waf stops, the log lines still
|
||||
// waiting are sent to SWW_WAF_LOG_REMOTE_URL before they are given up. stdout has
|
||||
// carried them.
|
||||
const remoteLogStopTimeout = 2 * time.Second
|
||||
|
||||
// RunWAF is `sww waf`: it reads the settings, the rule files,
|
||||
// the lookup database and the state and cache files, then serves requests until
|
||||
// ctx is done. Then it gives the requests in progress shutdownTimeout to
|
||||
// finish, ends its background work and writes every state and cache file,
|
||||
// alerts.json with the alerts still waiting. It returns the process's exit
|
||||
// status, 1 when waf cannot start or stop in order.
|
||||
func RunWAF(ctx context.Context, params Params) int {
|
||||
// Until the settings are read, the one message is an invalid setting's
|
||||
// error, which every SWW_WAF_LOG_LEVEL lets through.
|
||||
processLog := requestlog.NewProcessLogger(params.Stdout,
|
||||
config.InstanceName(params.Viper), slog.LevelError)
|
||||
|
||||
cfg, err := config.WAFFromEnvironment(params.Viper)
|
||||
if err != nil {
|
||||
return cannotStart(processLog, startError{"invalid setting", err})
|
||||
}
|
||||
|
||||
// While SWW_WAF_LOG_REMOTE_URL is set, every line on stdout from here on
|
||||
// is sent there too. The sender runs before any part is built, so that it
|
||||
// sends why waf cannot start.
|
||||
stdout := params.Stdout
|
||||
|
||||
var remote *remotelog.Sender
|
||||
|
||||
if cfg.LogRemoteURL != nil {
|
||||
remote = newRemoteLogSender(cfg)
|
||||
stdout = io.MultiWriter(params.Stdout, remote)
|
||||
}
|
||||
|
||||
processLog = requestlog.NewProcessLogger(
|
||||
stdout,
|
||||
cfg.InstanceName,
|
||||
cfg.LogLevel,
|
||||
)
|
||||
|
||||
if remote != nil {
|
||||
stopSending := startSending(ctx, remote, processLog)
|
||||
defer stopSending()
|
||||
}
|
||||
|
||||
var (
|
||||
files *state.Files
|
||||
listener net.Listener
|
||||
)
|
||||
|
||||
failed := make(servingFailed, 1)
|
||||
|
||||
app := fx.New(
|
||||
fx.Supply(
|
||||
params,
|
||||
cfg,
|
||||
processLog,
|
||||
requestLog{stdout},
|
||||
remote,
|
||||
failed,
|
||||
),
|
||||
fx.Provide(
|
||||
newAlertQueue, newRuleFiles, newCoreRuleSet, newLookupFile,
|
||||
newLimiter, newLedger, newAnomalyCounters,
|
||||
newLists, newDNSBL, newAbuseIPDB, newRouteWatch,
|
||||
newMetrics, newGeoJS, newWAFServer,
|
||||
func(server *waf.Server) *http.Server { return server.Server },
|
||||
newWAFListener, newStateFiles,
|
||||
),
|
||||
// fx stops hooks in the reverse of the order they were added in. The
|
||||
// state files are read before serving is set up, so that serving
|
||||
// stops before their loops do.
|
||||
fx.Populate(&files),
|
||||
fx.Invoke(serve),
|
||||
fx.Populate(&listener),
|
||||
fx.WithLogger(fxEventLogger(processLog)),
|
||||
)
|
||||
|
||||
err = app.Err()
|
||||
if err != nil {
|
||||
return cannotStart(processLog, err)
|
||||
}
|
||||
|
||||
processLog.Info("starting",
|
||||
"version", params.Version,
|
||||
"address", listener.Addr().String(),
|
||||
"settings", cfg)
|
||||
|
||||
// The last write follows the stop of every hook, and so the end of every
|
||||
// loop that changes the state or raises an alert, and of every ask of
|
||||
// GeoJS, the DNSBL zones, AbuseIPDB and routewatch. A hook of the state
|
||||
// files would stop before the loops of the parts built ahead of them.
|
||||
return run(ctx, app, failed, processLog, func() error {
|
||||
err := files.WriteAll()
|
||||
if err != nil {
|
||||
processLog.Error("writing the state and cache files failed",
|
||||
"error", err.Error())
|
||||
}
|
||||
|
||||
return err
|
||||
})
|
||||
}
|
||||
|
||||
// newAlertQueue returns the queue of the alerts to the webhook, Slack and
|
||||
// ntfy, with the settings for them. Until waf stops, it sends them.
|
||||
func newAlertQueue(
|
||||
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
|
||||
) *alerts.Queue {
|
||||
alertQueue := alerts.New(alerts.Params{
|
||||
WebhookURL: cfg.AlertWebhookURL,
|
||||
WebhookHeaders: cfg.AlertWebhookHeaders,
|
||||
SlackURL: cfg.AlertSlackWebhookURL,
|
||||
NtfyURL: cfg.AlertNtfyURL,
|
||||
NtfyToken: cfg.AlertNtfyToken,
|
||||
Events: cfg.AlertEvents,
|
||||
Cooldown: cfg.AlertCooldown,
|
||||
MaxPerHour: cfg.AlertMaxPerHour,
|
||||
Instance: cfg.InstanceName,
|
||||
Now: utcNow,
|
||||
ProcessLog: processLog,
|
||||
})
|
||||
|
||||
lc.Append(background(alertQueue.Run))
|
||||
|
||||
return alertQueue
|
||||
}
|
||||
|
||||
// newRuleFiles reads the rule files, and reads them again as they change
|
||||
// until waf stops.
|
||||
func newRuleFiles(
|
||||
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
|
||||
alertQueue *alerts.Queue,
|
||||
) (*rules.Files, error) {
|
||||
ruleFiles, err := rules.Load(rules.Params{
|
||||
Dir: cfg.RuleFilesDir,
|
||||
Enabled: cfg.RuleFilesEnabled,
|
||||
ProcessLog: processLog,
|
||||
Alerts: alertQueue,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, startError{"cannot use the rule files", err}
|
||||
}
|
||||
|
||||
lc.Append(background(ruleFiles.Watch))
|
||||
|
||||
return ruleFiles, nil
|
||||
}
|
||||
|
||||
// newCoreRuleSet returns the Core Rule Set at SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL,
|
||||
// without the rules SWW_WAF_CORE_RULE_SET_DISABLED_RULES switches off, reading bodies
|
||||
// up to SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE, with the exclusions in
|
||||
// SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE and the fixes of the apps in SWW_WAF_APPS,
|
||||
// or nil while SWW_WAF_CORE_RULE_SET_MODE is off. An exclusions file that
|
||||
// coreruleset.New refuses is an error in every mode. While the mode is off, a
|
||||
// set file is loaded with the Core Rule Set only to check it.
|
||||
func newCoreRuleSet(cfg *config.WAF) (*coreruleset.CoreRuleSet, error) {
|
||||
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff &&
|
||||
cfg.CoreRuleSetExclusionsFile == "" {
|
||||
return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error
|
||||
}
|
||||
|
||||
crs, err := coreruleset.New(coreruleset.Params{
|
||||
ParanoiaLevel: cfg.CoreRuleSetParanoiaLevel,
|
||||
DisabledRules: cfg.CoreRuleSetDisabledRules,
|
||||
BodyInspectionSize: cfg.CoreRuleSetBodyInspectionSize,
|
||||
ExclusionsFile: cfg.CoreRuleSetExclusionsFile,
|
||||
Apps: cfg.Apps,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, startError{
|
||||
"cannot use the Core Rule Set",
|
||||
fmt.Errorf("SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE: %w", err),
|
||||
}
|
||||
}
|
||||
|
||||
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff {
|
||||
return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error
|
||||
}
|
||||
|
||||
return crs, nil
|
||||
}
|
||||
|
||||
// newLookupFile reads the lookup database while SWW_WAF_LOOKUP_SOURCE is
|
||||
// file, and reads it again when it is replaced until waf stops. It returns
|
||||
// nil while the source is anything else.
|
||||
func newLookupFile(
|
||||
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
|
||||
alertQueue *alerts.Queue,
|
||||
) (*lookup.File, error) {
|
||||
if cfg.LookupSource != config.LookupSourceFile {
|
||||
return nil, nil //nolint:nilnil // no lookup database, and no error
|
||||
}
|
||||
|
||||
lookupFile, err := lookup.OpenFile(lookup.FileParams{
|
||||
Path: cfg.LookupDBPath,
|
||||
Now: utcNow,
|
||||
ProcessLog: processLog,
|
||||
Alerts: alertQueue,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, startError{"cannot use the lookup database", err}
|
||||
}
|
||||
|
||||
lc.Append(background(lookupFile.Watch))
|
||||
|
||||
return lookupFile, nil
|
||||
}
|
||||
|
||||
// newLimiter returns the table of clients, with the rate and byte limits.
|
||||
func newLimiter(cfg *config.WAF) *ratelimit.Limiter {
|
||||
return ratelimit.New(ratelimit.Limits{
|
||||
PerMinute: cfg.RateLimitPerMinute,
|
||||
PerHour: cfg.RateLimitPerHour,
|
||||
PerDay: cfg.RateLimitPerDay,
|
||||
BytesPerMinute: cfg.BytesLimitPerMinute,
|
||||
BytesPerHour: cfg.BytesLimitPerHour,
|
||||
BytesPerDay: cfg.BytesLimitPerDay,
|
||||
}, cfg.MaxTrackedClients)
|
||||
}
|
||||
|
||||
// newLedger returns the ledger of the bans, with the settings for them.
|
||||
func newLedger(cfg *config.WAF) *bans.Ledger {
|
||||
return bans.New(bans.Params{
|
||||
LimitBanDuration: cfg.LimitBanDuration,
|
||||
LimitBanRepeatWindow: cfg.LimitBanRepeatWindow,
|
||||
LimitBanPermanentAbove: cfg.LimitBanPermanentAbove,
|
||||
AttackBanDuration: cfg.AttackBanDuration,
|
||||
MaxBans: cfg.MaxBans,
|
||||
})
|
||||
}
|
||||
|
||||
// newAnomalyCounters returns the counters of the anomaly thresholds.
|
||||
func newAnomalyCounters(
|
||||
cfg *config.WAF, alertQueue *alerts.Queue,
|
||||
) *anomaly.Counters {
|
||||
return anomaly.New(anomaly.Params{
|
||||
Client: cfg.AnomalyClient,
|
||||
Net: cfg.AnomalyNet,
|
||||
ASN: cfg.AnomalyASN,
|
||||
Total: cfg.AnomalyTotal,
|
||||
WatchedNet: cfg.AnomalyWatchedNet,
|
||||
NetV4Prefix: cfg.AnomalyNetV4Prefix,
|
||||
NetV6Prefix: cfg.AnomalyNetV6Prefix,
|
||||
NamedNetblocks: cfg.AnomalyWatchedNets,
|
||||
Alerts: alertQueue,
|
||||
})
|
||||
}
|
||||
|
||||
// newLists returns the lists the settings name by URL, none fetched yet.
|
||||
// Until waf stops, it fetches them as they are due.
|
||||
func newLists(
|
||||
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
|
||||
alertQueue *alerts.Queue,
|
||||
) *reputation.Lists {
|
||||
lists := reputation.New(reputation.Params{
|
||||
BlocklistURLs: cfg.BlocklistURLs,
|
||||
Refresh: cfg.BlocklistRefresh,
|
||||
ASNLimitPercentURL: cfg.ASNLimitPercentURL,
|
||||
TagLists: cfg.TagLists,
|
||||
Now: utcNow,
|
||||
ProcessLog: processLog,
|
||||
Alerts: alertQueue,
|
||||
})
|
||||
|
||||
lc.Append(background(lists.Run))
|
||||
|
||||
return lists
|
||||
}
|
||||
|
||||
// newDNSBL returns what asks the DNSBL zones, none asked yet. At the stop it
|
||||
// ends the queries under way, as reputation.DNSBL.StopAsking says.
|
||||
func newDNSBL(
|
||||
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
|
||||
alertQueue *alerts.Queue,
|
||||
) *reputation.DNSBL {
|
||||
dnsbl := reputation.NewDNSBL(reputation.DNSBLParams{
|
||||
Zones: cfg.DNSBLZones,
|
||||
Resolver: cfg.DNSBLResolver,
|
||||
CacheTTL: cfg.IPReputationCacheTTL,
|
||||
Timeout: cfg.DNSBLAbuseIPDBTimeout,
|
||||
Now: utcNow,
|
||||
ProcessLog: processLog,
|
||||
Alerts: alertQueue,
|
||||
})
|
||||
|
||||
lc.Append(fx.StopHook(dnsbl.StopAsking))
|
||||
|
||||
return dnsbl
|
||||
}
|
||||
|
||||
// newAbuseIPDB returns what checks clients with AbuseIPDB while
|
||||
// SWW_WAF_ABUSEIPDB_KEY is set, none checked yet. Until waf stops, it
|
||||
// fetches AbuseIPDB's blacklist as it is due, and bans the clients on it in
|
||||
// ledger, as waf.BanAbuseIPDBBlacklist says. At the stop it ends the checks
|
||||
// under way, as reputation.AbuseIPDB.StopChecking says.
|
||||
func newAbuseIPDB(
|
||||
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
|
||||
alertQueue *alerts.Queue, ledger *bans.Ledger,
|
||||
) *reputation.AbuseIPDB {
|
||||
abuseIPDB := reputation.NewAbuseIPDB(reputation.AbuseIPDBParams{
|
||||
URL: reputation.AbuseIPDBURL,
|
||||
Key: cfg.AbuseIPDBKey,
|
||||
MinScore: cfg.AbuseIPDBMinScore,
|
||||
DailyBudget: cfg.AbuseIPDBDailyBudget,
|
||||
CacheTTL: cfg.IPReputationCacheTTL,
|
||||
Timeout: cfg.DNSBLAbuseIPDBTimeout,
|
||||
BlacklistURL: reputation.AbuseIPDBBlacklistURL,
|
||||
Threshold: cfg.AbuseIPDBThreshold,
|
||||
BlacklistRefresh: cfg.AbuseIPDBBlacklistRefresh,
|
||||
Blacklisted: func(listed []reputation.BlacklistEntry) {
|
||||
waf.BanAbuseIPDBBlacklist(
|
||||
cfg, ledger, processLog, utcNow(), listed,
|
||||
)
|
||||
},
|
||||
Now: utcNow,
|
||||
ProcessLog: processLog,
|
||||
Alerts: alertQueue,
|
||||
})
|
||||
|
||||
lc.Append(background(abuseIPDB.RunBlacklist))
|
||||
lc.Append(fx.StopHook(abuseIPDB.StopChecking))
|
||||
|
||||
return abuseIPDB
|
||||
}
|
||||
|
||||
// newRouteWatch returns what asks routewatch about clients, or nil while
|
||||
// SWW_WAF_ROUTEWATCH_URL is unset. At the stop it ends the queries under
|
||||
// way, as routewatch.Client.StopAsking says.
|
||||
func newRouteWatch(
|
||||
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
|
||||
) *routewatch.Client {
|
||||
if cfg.RouteWatchURL == "" {
|
||||
return nil
|
||||
}
|
||||
|
||||
routeWatch := routewatch.New(routewatch.Params{
|
||||
URL: cfg.RouteWatchURL,
|
||||
Now: utcNow,
|
||||
ProcessLog: processLog,
|
||||
})
|
||||
|
||||
lc.Append(fx.StopHook(routeWatch.StopAsking))
|
||||
|
||||
return routeWatch
|
||||
}
|
||||
|
||||
// newMetrics returns the metrics, with those read from the other parts as
|
||||
// the metrics are asked for: AbuseIPDB's while SWW_WAF_ABUSEIPDB_KEY is set,
|
||||
// the lookup database's while there is one, and the remote log sender's
|
||||
// while there is one.
|
||||
func newMetrics(
|
||||
cfg *config.WAF, ledger *bans.Ledger, limiter *ratelimit.Limiter,
|
||||
ruleFiles *rules.Files, lists *reputation.Lists,
|
||||
dnsbl *reputation.DNSBL, abuseIPDB *reputation.AbuseIPDB,
|
||||
alertQueue *alerts.Queue, lookupFile *lookup.File,
|
||||
remote *remotelog.Sender,
|
||||
) *metrics.Metrics {
|
||||
m := metrics.New(cfg.MetricsBusiestCountriesAndASNs, cfg.InstanceName)
|
||||
m.AddBansAndClients(ledger, limiter, utcNow)
|
||||
m.AddRules(ruleFiles)
|
||||
m.AddReputation(lists, dnsbl)
|
||||
|
||||
if cfg.AbuseIPDBKey != "" {
|
||||
m.AddAbuseIPDB(abuseIPDB)
|
||||
}
|
||||
|
||||
m.AddAlerts(alertQueue)
|
||||
|
||||
if lookupFile != nil {
|
||||
m.AddLookupFile(lookupFile.LastRead, lookupFile.ReadFailures)
|
||||
}
|
||||
|
||||
if remote != nil {
|
||||
m.AddRemoteLog(remote)
|
||||
}
|
||||
|
||||
return m
|
||||
}
|
||||
|
||||
// newGeoJS returns what looks clients up at GeoJS, none looked up yet. Each
|
||||
// answer goes to the limiter and the ledger, as waf.AddLookup says, and each
|
||||
// failure is handled as waf.GeoJSFailed says. At the stop it ends the request
|
||||
// under way, as geojs.GeoJS.StopAsking says.
|
||||
func newGeoJS(
|
||||
lc fx.Lifecycle,
|
||||
cfg *config.WAF,
|
||||
m *metrics.Metrics,
|
||||
limiter *ratelimit.Limiter,
|
||||
ledger *bans.Ledger,
|
||||
alertQueue *alerts.Queue,
|
||||
processLog *slog.Logger,
|
||||
) *geojs.GeoJS {
|
||||
geoJS := geojs.New(geojs.Params{
|
||||
URL: geojs.URL,
|
||||
Answered: func(answer geojs.Answer) {
|
||||
waf.AddLookup(cfg, limiter, ledger, answer)
|
||||
},
|
||||
Now: utcNow,
|
||||
Asked: m.GeoJSRequests.Inc,
|
||||
Failed: func(err error, askingAgainIn time.Duration) {
|
||||
waf.GeoJSFailed(processLog, alertQueue, m, err, askingAgainIn)
|
||||
},
|
||||
WentWithoutAnswer: m.RequestsWithoutGeoJSAnswer.Inc,
|
||||
})
|
||||
|
||||
lc.Append(fx.StopHook(geoJS.StopAsking))
|
||||
|
||||
return geoJS
|
||||
}
|
||||
|
||||
// newWAFServer returns the server waf runs, with its parts and the time
|
||||
// limit params gives.
|
||||
func newWAFServer(
|
||||
params Params, cfg *config.WAF, out requestLog, processLog *slog.Logger,
|
||||
m *metrics.Metrics, limiter *ratelimit.Limiter, ledger *bans.Ledger,
|
||||
anomalies *anomaly.Counters, lists *reputation.Lists,
|
||||
dnsbl *reputation.DNSBL, abuseIPDB *reputation.AbuseIPDB,
|
||||
routeWatch *routewatch.Client, geoJS *geojs.GeoJS,
|
||||
lookupFile *lookup.File, crs *coreruleset.CoreRuleSet,
|
||||
ruleFiles *rules.Files, alertQueue *alerts.Queue,
|
||||
) *waf.Server {
|
||||
return waf.New(waf.Params{
|
||||
Config: cfg,
|
||||
RequestLog: out,
|
||||
ProcessLog: processLog,
|
||||
Metrics: m,
|
||||
Limiter: limiter,
|
||||
Ledger: ledger,
|
||||
Anomalies: anomalies,
|
||||
Lists: lists,
|
||||
DNSBL: dnsbl,
|
||||
AbuseIPDB: abuseIPDB,
|
||||
RouteWatch: routeWatch,
|
||||
GeoJS: geoJS,
|
||||
LookupFile: lookupFile,
|
||||
CoreRuleSet: crs,
|
||||
Now: utcNow,
|
||||
Rules: ruleFiles,
|
||||
Alerts: alertQueue,
|
||||
|
||||
BesideLookupTimeLimit: params.BesideLookupTimeLimit,
|
||||
})
|
||||
}
|
||||
|
||||
// newWAFListener listens on SWW_WAF_LISTEN_ADDR.
|
||||
func newWAFListener(cfg *config.WAF) (net.Listener, error) {
|
||||
return listen(cfg.ListenAddr, "SWW_WAF_LISTEN_ADDR")
|
||||
}
|
||||
|
||||
// newStateFiles reads the state and cache files into the parts and the
|
||||
// alert queue, as state.Load does. Until waf stops, it writes them as they
|
||||
// are due and takes in an admin's edits of them.
|
||||
func newStateFiles(
|
||||
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
|
||||
m *metrics.Metrics, ledger *bans.Ledger, limiter *ratelimit.Limiter,
|
||||
geoJS *geojs.GeoJS, lists *reputation.Lists, dnsbl *reputation.DNSBL,
|
||||
abuseIPDB *reputation.AbuseIPDB, routeWatch *routewatch.Client,
|
||||
alertQueue *alerts.Queue, anomalies *anomaly.Counters,
|
||||
) (*state.Files, error) {
|
||||
files, err := state.Load(state.Params{
|
||||
Dir: cfg.StateDir,
|
||||
CacheDir: cfg.CacheDir,
|
||||
BansWriteDelay: cfg.BansWriteDelay,
|
||||
WriteInterval: cfg.StateWriteInterval,
|
||||
Ledger: ledger,
|
||||
Limiter: limiter,
|
||||
GeoJS: geoJS,
|
||||
Lists: lists,
|
||||
DNSBL: dnsbl,
|
||||
AbuseIPDB: abuseIPDB,
|
||||
RouteWatch: routeWatch,
|
||||
Alerts: alertQueue,
|
||||
Anomalies: anomalies,
|
||||
Now: utcNow,
|
||||
ProcessLog: processLog,
|
||||
Metrics: m,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, startError{"cannot use the state and cache files", err}
|
||||
}
|
||||
|
||||
lc.Append(background(files.Run))
|
||||
lc.Append(background(files.Watch))
|
||||
|
||||
return files, nil
|
||||
}
|
||||
|
||||
// newRemoteLogSender returns a sender of the log lines to
|
||||
// SWW_WAF_LOG_REMOTE_URL, with the settings for it.
|
||||
func newRemoteLogSender(cfg *config.WAF) *remotelog.Sender {
|
||||
return remotelog.New(remotelog.Params{
|
||||
URL: cfg.LogRemoteURL,
|
||||
RootCAs: cfg.LogRemoteTLSCAs,
|
||||
BufferLines: cfg.LogRemoteBufferLines,
|
||||
Facility: cfg.LogRemoteFacility,
|
||||
AppName: cfg.LogRemoteAppName,
|
||||
})
|
||||
}
|
||||
|
||||
// startSending runs remote until the function it returns is called, which
|
||||
// then waits at most remoteLogStopTimeout for the lines still waiting to
|
||||
// be sent. Sending goes on after ctx is done, so that the lines written
|
||||
// while sww stops are sent too.
|
||||
func startSending(
|
||||
ctx context.Context, remote *remotelog.Sender, processLog *slog.Logger,
|
||||
) func() {
|
||||
sending, stop := context.WithCancel(context.WithoutCancel(ctx))
|
||||
sent := make(chan struct{})
|
||||
|
||||
go func() {
|
||||
remote.Run(sending, processLog)
|
||||
close(sent)
|
||||
}()
|
||||
|
||||
return func() {
|
||||
stop()
|
||||
|
||||
select {
|
||||
case <-sent:
|
||||
case <-time.After(remoteLogStopTimeout):
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -90,6 +90,8 @@ func TestMainRunsWAF(t *testing.T) {
|
||||
cmd := exec.CommandContext(t.Context(), binary)
|
||||
cmd.Stdout = out
|
||||
|
||||
// The Core Rule Set is off. Main keeps waf's own limit for its checks,
|
||||
// and on a busy host the Core Rule Set's inspection can pass it.
|
||||
cmd.Env = append(os.Environ(),
|
||||
"TZ="+zone,
|
||||
commandForMain+"="+wafCommand,
|
||||
@@ -97,7 +99,8 @@ func TestMainRunsWAF(t *testing.T) {
|
||||
stateDir+"="+dir,
|
||||
cacheDir+"="+dir,
|
||||
ruleFilesDir+"="+t.TempDir(),
|
||||
lookupSource+"=off")
|
||||
lookupSource+"=off",
|
||||
"SWW_WAF_CORE_RULE_SET_MODE=off")
|
||||
|
||||
exited := startProcess(t, cmd)
|
||||
|
||||
@@ -205,7 +208,9 @@ func TestMainWithoutACommandItKnowsPrintsTheCommands(t *testing.T) {
|
||||
|
||||
// ask sends waf at addr rproxy's question about a request for path from
|
||||
// placed, checks the answer's status and disposition, and closes it.
|
||||
func ask(t *testing.T, addr, path string, status int, disposition string) {
|
||||
func ask(
|
||||
t *testing.T, addr, path string, status int, disposition waf.Disposition,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
req, err := http.NewRequestWithContext(t.Context(), http.MethodGet,
|
||||
@@ -229,7 +234,7 @@ func ask(t *testing.T, addr, path string, status int, disposition string) {
|
||||
_ = res.Body.Close()
|
||||
|
||||
if res.StatusCode != status ||
|
||||
res.Header.Get(waf.DispositionHeader) != disposition {
|
||||
res.Header.Get(waf.DispositionHeader) != string(disposition) {
|
||||
t.Errorf(
|
||||
"answered %d with disposition %q, want %d with %q",
|
||||
res.StatusCode,
|
||||
|
||||
@@ -35,7 +35,7 @@ func (abuseIPDB) check(
|
||||
return nil
|
||||
}
|
||||
|
||||
client := rq.h.clientGroup(rq.client)
|
||||
client := clientGroup(rq.h.config, rq.client)
|
||||
held, _ := rq.h.limiter.Client(client)
|
||||
offender := held.History.Offences != ratelimit.Offences{}
|
||||
|
||||
@@ -54,7 +54,7 @@ func (abuseIPDB) check(
|
||||
"scored by AbuseIPDB at or over SWW_WAF_ABUSEIPDB_MIN_SCORE",
|
||||
)
|
||||
|
||||
if rq.h.config.DNSBLAbuseIPDBAction != deny {
|
||||
if rq.h.config.DNSBLAbuseIPDBAction != config.ReputationActionDeny {
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
package waf
|
||||
|
||||
// The bans for AbuseIPDB's blacklist. They read SWW_WAF_ABUSEIPDB_BAN_DAYS and
|
||||
// SWW_WAF_MODE; the blacklist is fetched by the reputation package.
|
||||
|
||||
import (
|
||||
"log/slog"
|
||||
"net/netip"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||
)
|
||||
|
||||
// BanAbuseIPDBBlacklist bans in ledger the netblock of each client on
|
||||
// AbuseIPDB's blacklist, listed, for SWW_WAF_ABUSEIPDB_BAN_DAYS from now, as
|
||||
// the ledger's BanForAbuseIPDB describes. It logs to processLog how many
|
||||
// clients were listed and how many bans were made. Clients in one netblock
|
||||
// make one ban, with the highest of their scores. In observe mode it makes
|
||||
// no ban.
|
||||
func BanAbuseIPDBBlacklist(
|
||||
cfg *config.WAF, ledger *bans.Ledger, processLog *slog.Logger,
|
||||
now time.Time, listed []reputation.BlacklistEntry,
|
||||
) {
|
||||
scores := make(map[netip.Prefix]int64, len(listed))
|
||||
|
||||
for _, client := range listed {
|
||||
netblock := banNetblock(cfg, client.Addr)
|
||||
scores[netblock] = max(scores[netblock], client.Score)
|
||||
}
|
||||
|
||||
made := 0
|
||||
|
||||
if !cfg.Observe {
|
||||
made = ledger.BanForAbuseIPDB(
|
||||
now, now.Add(cfg.AbuseIPDBBanDuration), scores,
|
||||
)
|
||||
}
|
||||
|
||||
processLog.Info(
|
||||
"fetched AbuseIPDB's blacklist",
|
||||
"clients", len(listed),
|
||||
"bans_made", made,
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,100 @@
|
||||
package waf_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// abuseIPDBBanDays is the setting of how long a ban for AbuseIPDB's
|
||||
// blacklist lasts, in days.
|
||||
const abuseIPDBBanDays = "SWW_WAF_ABUSEIPDB_BAN_DAYS"
|
||||
|
||||
func TestClientOnTheFetchedAbuseIPDBBlacklistIsRefused(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, clk, server := startWithClock(t, "", map[string]string{
|
||||
abuseIPDBKey: accountKey,
|
||||
})
|
||||
|
||||
ctx, cancel := context.WithCancel(t.Context())
|
||||
stopped := make(chan struct{})
|
||||
|
||||
go func() {
|
||||
server.AbuseIPDB.RunBlacklist(ctx)
|
||||
close(stopped)
|
||||
}()
|
||||
|
||||
t.Cleanup(func() {
|
||||
cancel()
|
||||
<-stopped
|
||||
})
|
||||
|
||||
waitUntil(func() bool {
|
||||
active, _ := server.Ledger.Count(clk.Now())
|
||||
|
||||
return active == 1
|
||||
})
|
||||
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
s.get(otherClient, http.StatusOK, requestlog.ActionForward)
|
||||
}
|
||||
|
||||
func TestClientsOnTheAbuseIPDBBlacklistAreBannedForTheBanDays(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, clk, server := startWithClock(t, "", map[string]string{
|
||||
abuseIPDBBanDays: "2", banScopeV4Prefix: "24",
|
||||
})
|
||||
|
||||
// Two clients in one netblock make one ban, with the higher score.
|
||||
server.BanAbuseIPDBBlacklist([]reputation.BlacklistEntry{
|
||||
{Addr: netip.MustParseAddr(client), Score: 95},
|
||||
{Addr: netip.MustParseAddr(otherClient), Score: 100},
|
||||
{Addr: netip.MustParseAddr("2001:db8:5::1"), Score: 90},
|
||||
})
|
||||
|
||||
held := server.Ledger.Bans(netip.MustParsePrefix(clientsNet))
|
||||
if len(held) != 1 || held[0].Cause != bans.CauseAbuseIPDB ||
|
||||
*held[0].Notes.Reputation[0].Score != 100 {
|
||||
t.Errorf("bans on %s %+v, want one for the blacklist, with 100",
|
||||
clientsNet, held)
|
||||
}
|
||||
|
||||
// Every client of a banned netblock is refused, the IPv6 address's /64
|
||||
// included.
|
||||
for _, from := range []string{"203.0.113.200", "2001:db8:5::ffff:1"} {
|
||||
s.get(from, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||
}
|
||||
|
||||
s.get("2001:db8:5:1::1", http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
clk.advance(2 * 24 * time.Hour)
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
}
|
||||
|
||||
func TestAbuseIPDBBlacklistMakesNoBanInObserveMode(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, clk, server := startWithClock(
|
||||
t,
|
||||
"",
|
||||
map[string]string{mode: "observe"},
|
||||
)
|
||||
|
||||
server.BanAbuseIPDBBlacklist([]reputation.BlacklistEntry{
|
||||
{Addr: netip.MustParseAddr(client), Score: 100},
|
||||
})
|
||||
|
||||
if active, _ := server.Ledger.Count(clk.Now()); active != 0 {
|
||||
t.Errorf("%d bans active, want none", active)
|
||||
}
|
||||
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
}
|
||||
@@ -139,7 +139,7 @@ func (rq *request) addBan() {
|
||||
)
|
||||
|
||||
if err == nil {
|
||||
netblock, err = rq.h.banNetblock(toAdd.Netblock)
|
||||
netblock, err = rq.h.parseBanNetblock(toAdd.Netblock)
|
||||
}
|
||||
|
||||
if err == nil {
|
||||
@@ -190,12 +190,12 @@ func (rq *request) readBanToAdd() (banToAdd, error) {
|
||||
return toAdd, nil
|
||||
}
|
||||
|
||||
// banNetblock reads value, a netblock such as 203.0.113.0/24, or a
|
||||
// parseBanNetblock reads value, a netblock such as 203.0.113.0/24, or a
|
||||
// client's address, which stands for the netblock a ban on that client
|
||||
// covers. An IPv4-mapped netblock, such as ::ffff:203.0.113.0/120, is
|
||||
// refused, since a client's address is looked up as IPv4 and a ban on it
|
||||
// would refuse nothing, and so is a value with a zone.
|
||||
func (h *handler) banNetblock(value string) (netip.Prefix, error) {
|
||||
func (h *handler) parseBanNetblock(value string) (netip.Prefix, error) {
|
||||
netblock, err := netip.ParsePrefix(value)
|
||||
if err == nil {
|
||||
if netblock.Addr().Is4In6() {
|
||||
@@ -224,7 +224,7 @@ func (h *handler) banNetblock(value string) (netip.Prefix, error) {
|
||||
return netip.Prefix{}, fmt.Errorf("netblock %q %w", value, errZone)
|
||||
}
|
||||
|
||||
return h.netblock(addr), nil
|
||||
return banNetblock(h.config, addr), nil
|
||||
}
|
||||
|
||||
// expiry returns when a ban made at now for duration ends: duration
|
||||
@@ -295,7 +295,7 @@ func (rq *request) showClient() {
|
||||
Bans: state.BanEntries(rq.h.ledger.Covering(addr)),
|
||||
}
|
||||
|
||||
client, seen := rq.h.limiter.Client(rq.h.clientGroup(addr))
|
||||
client, seen := rq.h.limiter.Client(clientGroup(rq.h.config, addr))
|
||||
if seen {
|
||||
answer.Client = &client
|
||||
}
|
||||
|
||||
@@ -19,6 +19,8 @@ const (
|
||||
alertMaxPerHour = "SWW_WAF_ALERT_MAX_PER_HOUR"
|
||||
// alertInstance is the instance every alert of these tests gives.
|
||||
alertInstance = "host1/blog"
|
||||
// alertWebhook is where the tests send alerts.
|
||||
alertWebhook = "https://alerts.example/sww"
|
||||
)
|
||||
|
||||
func TestBanForABrokenLimitRaisesABanAlertWithItsNotes(t *testing.T) {
|
||||
@@ -270,7 +272,7 @@ func startAppWithAlerts(
|
||||
clk := &clock{now: time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)}
|
||||
settings := map[string]string{
|
||||
trustedProxies: trustLocalhost,
|
||||
alertWebhookURL: "https://alerts.example/sww",
|
||||
alertWebhookURL: alertWebhook,
|
||||
instanceName: alertInstance,
|
||||
}
|
||||
maps.Copy(settings, env)
|
||||
@@ -292,7 +294,7 @@ func startAppWithAlerts(
|
||||
// the time raised, for ban, with its netblock, cause, reason and notes,
|
||||
// which ends at expires, as the log line gives it.
|
||||
func banAlert(
|
||||
event string,
|
||||
event alerts.Event,
|
||||
raised time.Time,
|
||||
client string,
|
||||
ban bans.Ban,
|
||||
@@ -307,7 +309,9 @@ func banAlert(
|
||||
Netblock: ban.Netblock,
|
||||
Reason: ban.Reason,
|
||||
Detail: map[string]any{
|
||||
"cause": ban.Cause, "ban_expires": expires, "notes": ban.Notes,
|
||||
"cause": string(
|
||||
ban.Cause,
|
||||
), "ban_expires": expires, "notes": ban.Notes,
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -315,7 +319,7 @@ func banAlert(
|
||||
// attackAlert is banAlert for a ban for the probe rule of testRules, with
|
||||
// the netblock and the notes of ban.
|
||||
func attackAlert(
|
||||
event string,
|
||||
event alerts.Event,
|
||||
raised time.Time,
|
||||
client string,
|
||||
ban bans.Ban,
|
||||
|
||||
@@ -86,10 +86,10 @@ func TestEachScopeAndWindowOverItsThresholdAlertsOncePerCooldown(
|
||||
value int64
|
||||
count float64
|
||||
}{
|
||||
{requestsPerMinute, ratelimit.KindRequests, minute, 2, 3},
|
||||
{requestsPerHour, ratelimit.KindRequests, hour, 2, 3},
|
||||
{bytesPerMinute, ratelimit.KindBytes, minute, 250, 300},
|
||||
{bytesPerHour, ratelimit.KindBytes, hour, 250, 300},
|
||||
{requestsPerMinute, string(ratelimit.KindRequests), minute, 2, 3},
|
||||
{requestsPerHour, string(ratelimit.KindRequests), hour, 2, 3},
|
||||
{bytesPerMinute, string(ratelimit.KindBytes), minute, 250, 300},
|
||||
{bytesPerHour, string(ratelimit.KindBytes), hour, 250, 300},
|
||||
} {
|
||||
setting := scope.prefix + threshold.end
|
||||
value := strconv.FormatInt(threshold.value, 10)
|
||||
@@ -186,7 +186,7 @@ func TestEveryRequestIsCountedWhateverIsDoneWithIt(t *testing.T) {
|
||||
for _, sent := range []struct {
|
||||
from, path string
|
||||
status int
|
||||
action string
|
||||
action requestlog.Action
|
||||
}{
|
||||
{allowed, "/", http.StatusOK, requestlog.ActionForward},
|
||||
{exempt, "/", http.StatusOK, requestlog.ActionForward},
|
||||
|
||||
@@ -20,7 +20,7 @@ import (
|
||||
"github.com/prometheus/client_golang/prometheus/testutil"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
|
||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||
@@ -33,7 +33,7 @@ const (
|
||||
// SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS, the time waf has to answer.
|
||||
answerTimeLimit = 2 * time.Second
|
||||
// defaultLookupWait is the default of SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS.
|
||||
defaultLookupWait = 500 * time.Millisecond
|
||||
defaultLookupWait = 800 * time.Millisecond
|
||||
// defaultDNSBLAbuseIPDBWait is the default of SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS.
|
||||
defaultDNSBLAbuseIPDBWait = 2 * time.Second
|
||||
// The header limits SPEC.md gives.
|
||||
@@ -111,8 +111,7 @@ func TestHeaderLimitRefusalNamesItsHeuristicCountsAndLimits(t *testing.T) {
|
||||
s.out.requestLine(t),
|
||||
"header_limits",
|
||||
`{"longest_header_line_bytes": 40, "header_line_bytes_limit": 8192,
|
||||
"fields": 101, "fields_limit": 100, "cookie_lines": 0,
|
||||
"cookie_lines_limit": 1, "cookies": 0, "cookies_limit": 50}`,
|
||||
"fields": 101, "fields_limit": 100, "cookies": 0, "cookies_limit": 50}`,
|
||||
)
|
||||
}
|
||||
|
||||
@@ -135,7 +134,7 @@ func TestARequestLineOverTheLimitIsRefusedWith414BeforeAnyOtherCheck(
|
||||
// cookies is how many cookies the request carries.
|
||||
cookies int
|
||||
status int
|
||||
action string
|
||||
action requestlog.Action
|
||||
}{
|
||||
{
|
||||
"at the limit", lineLimit, 0, http.StatusOK,
|
||||
@@ -199,9 +198,6 @@ func TestAHeaderLineOverTheLimitIsRefusedWith431(t *testing.T) {
|
||||
{"a header line over the limit", func(req *http.Request) {
|
||||
req.Header.Set("X-Long", strings.Repeat("a", lineLimit+1-len("X-Long: ")))
|
||||
}, true},
|
||||
{"a Cookie line at the limit", func(req *http.Request) {
|
||||
req.Header.Set(cookieHeader, "a="+strings.Repeat("a", lineLimit-len("Cookie: a=")))
|
||||
}, false},
|
||||
{"one cookie over the limit", func(req *http.Request) {
|
||||
req.Header.Set(cookieHeader, "a="+strings.Repeat("a", lineLimit))
|
||||
}, true},
|
||||
@@ -271,8 +267,8 @@ func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
|
||||
{"parts of no-break spaces count", func(req *http.Request) {
|
||||
req.Header.Set(cookieHeader, strings.Repeat(" ;", cookieLimit+1))
|
||||
}, true},
|
||||
{"two Cookie lines", func(req *http.Request) {
|
||||
req.Header[cookieHeader] = []string{"a=1", "b=1"}
|
||||
{"cookies over the limit across two headers", func(req *http.Request) {
|
||||
req.Header[cookieHeader] = []string{cookies(cookieLimit), "last=1"}
|
||||
}, true},
|
||||
{"cookies without a value count", func(req *http.Request) {
|
||||
req.Header.Set(cookieHeader, strings.Repeat("a;", cookieLimit+1))
|
||||
@@ -595,7 +591,7 @@ func TestAClientGeoJSPlacedInNoCountryCountsAsUnknownWithoutWaiting(
|
||||
s := startWAF(t, geojsURL, map[string]string{
|
||||
unknownCountryLimitPercent: "0", rateLimitPerMinute: "1",
|
||||
})
|
||||
s.server.GeoJS.Load([]lookup.Answer{{
|
||||
s.server.GeoJS.Load([]geojs.Answer{{
|
||||
Client: netip.MustParsePrefix(client + "/32"), ASN: "AS64512",
|
||||
Answered: s.clock.Now(), Used: s.clock.Now(),
|
||||
}})
|
||||
@@ -734,7 +730,7 @@ func TestARefusedRequestDoesNotWaitForASlowGeoJS(t *testing.T) {
|
||||
// cookie is the request's Cookie header.
|
||||
cookie string
|
||||
status int
|
||||
action string
|
||||
action requestlog.Action
|
||||
}{
|
||||
{
|
||||
name: "SWW_WAF_DENY_NETS", env: map[string]string{denyNets: client},
|
||||
@@ -774,6 +770,8 @@ func TestARefusedRequestDoesNotWaitForASlowGeoJS(t *testing.T) {
|
||||
// TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays says,
|
||||
// so that a request that does not wait takes no time at all.
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
defer waitForGeoJSRequestsToBeAbandoned()
|
||||
|
||||
server, out := newWAFInBubble(t, tc.env)
|
||||
tc.prepare(t, server, time.Now())
|
||||
|
||||
@@ -815,7 +813,7 @@ func TestAQuestionWaitsForASlowGeoJSOnlyAsLongAsTheDefaultSays(
|
||||
// prepare gets the server ready, at now, before the question.
|
||||
prepare func(t *testing.T, server *waf.Server, now time.Time)
|
||||
status int
|
||||
action string
|
||||
action requestlog.Action
|
||||
}{
|
||||
{"nothing refuses it", nil, leaveAsItIs, http.StatusOK, requestlog.ActionForward},
|
||||
{
|
||||
@@ -842,6 +840,8 @@ func TestAQuestionWaitsForASlowGeoJSOnlyAsLongAsTheDefaultSays(
|
||||
t.Parallel()
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
defer waitForGeoJSRequestsToBeAbandoned()
|
||||
|
||||
server, out := newWAFInBubble(t, tc.env)
|
||||
tc.prepare(t, server, time.Now())
|
||||
|
||||
@@ -877,18 +877,13 @@ func TestWAFAnswersWithinTheTimeLimitWhileEveryOutsideSourceHangs(
|
||||
held, _, _ := startHeldGeoJS(t)
|
||||
resolver, releaseResolver := startHeldResolver(t)
|
||||
|
||||
server, out, _ := newWAFWith(t, func(
|
||||
params waf.Params,
|
||||
) *waf.Server {
|
||||
params.AbuseIPDBURL = silentAbuseIPDBURL
|
||||
|
||||
return waf.New(params)
|
||||
}, silentGeoJSURL, time.Now, map[string]string{
|
||||
tagDeny: "country=KP", coreRuleSetMode: block,
|
||||
dnsblZones: dnsblZone, dnsblResolver: resolver,
|
||||
abuseIPDBKey: accountKey,
|
||||
blocklistURLs: held + "/drop.txt", asnLimitPercentURL: held + "/asn.txt",
|
||||
})
|
||||
server, out, _ := newWAFWith(t, waf.New,
|
||||
silentGeoJSURL, silentAbuseIPDBURL, time.Now, map[string]string{
|
||||
tagDeny: "country=KP", coreRuleSetMode: block,
|
||||
dnsblZones: dnsblZone, dnsblResolver: resolver,
|
||||
abuseIPDBKey: accountKey,
|
||||
blocklistURLs: held + "/drop.txt", asnLimitPercentURL: held + "/asn.txt",
|
||||
})
|
||||
|
||||
// The lists are fetched outside the synctest bubble below: a wait on the
|
||||
// network inside it would keep its clock from moving on.
|
||||
@@ -954,10 +949,11 @@ func TestWAFAnswersWithinTheTimeLimitWhileEveryOutsideSourceHangs(
|
||||
t.Errorf("%d checks with AbuseIPDB, want 1", checked)
|
||||
}
|
||||
|
||||
// The query is answered, and the check with AbuseIPDB abandoned,
|
||||
// before the bubble ends.
|
||||
// The query is answered, and the check with AbuseIPDB and the request
|
||||
// to GeoJS abandoned, before the bubble ends.
|
||||
releaseResolver()
|
||||
time.Sleep(defaultDNSBLAbuseIPDBWait)
|
||||
waitForGeoJSRequestsToBeAbandoned()
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
+24
-17
@@ -10,13 +10,14 @@ import (
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// banResponse is a refusal answered with SWW_WAF_BAN_RESPONSE, and logged
|
||||
// with action.
|
||||
func (rq *request) banResponse(action string) *refusal {
|
||||
func (rq *request) banResponse(action requestlog.Action) *refusal {
|
||||
return &refusal{status: rq.h.config.BanResponse, action: action}
|
||||
}
|
||||
|
||||
@@ -62,7 +63,7 @@ func (activeBan) check(
|
||||
|
||||
// banAction is the action of a request refused under a ban or for making one:
|
||||
// permanent says the ban never runs out.
|
||||
func banAction(permanent bool) string {
|
||||
func banAction(permanent bool) requestlog.Action {
|
||||
if permanent {
|
||||
return requestlog.ActionPermanentlyBanned
|
||||
}
|
||||
@@ -85,13 +86,14 @@ func (rq *request) countedBytes() int64 {
|
||||
response, request := rq.out.bytes, rq.requestBytes
|
||||
|
||||
switch rq.h.config.BytesCounted {
|
||||
case "response":
|
||||
case config.BytesCountedResponse:
|
||||
return response
|
||||
case "request":
|
||||
case config.BytesCountedRequest:
|
||||
return request
|
||||
default: // both
|
||||
return response + request
|
||||
case config.BytesCountedBoth:
|
||||
}
|
||||
|
||||
return response + request
|
||||
}
|
||||
|
||||
// banForLimit bans the client's netblock at now for a broken limit, the
|
||||
@@ -114,14 +116,14 @@ func (rq *request) banForLimit(
|
||||
rq.line.LimitHit = hit.Window + "_bytes" // as counts names the byte totals
|
||||
case ratelimit.KindRefusals:
|
||||
rq.line.LimitHit = requestlog.LimitHitMinuteRefusals
|
||||
default:
|
||||
case ratelimit.KindRequests:
|
||||
rq.line.LimitHit = hit.Window
|
||||
}
|
||||
|
||||
rq.line.Offence = requestlog.OffenceLimit
|
||||
rq.h.metrics.LimitHit(hit)
|
||||
|
||||
netblock := rq.h.netblock(rq.client)
|
||||
netblock := banNetblock(rq.h.config, rq.client)
|
||||
if rq.h.config.Observe &&
|
||||
!rq.wouldAlertBan(netblock, now, bans.CauseLimit) {
|
||||
return
|
||||
@@ -145,6 +147,7 @@ func (rq *request) banForLimit(
|
||||
notes.LimitPercent, notes.LimitPercentSetting = rq.rateLimitPercent.logged()
|
||||
case ratelimit.KindBytes:
|
||||
notes.LimitPercent, notes.LimitPercentSetting = rq.bytesLimitPercent.logged()
|
||||
case ratelimit.KindRefusals: // the refusal limit is not lowered
|
||||
}
|
||||
|
||||
if rq.h.config.Observe {
|
||||
@@ -157,7 +160,7 @@ func (rq *request) banForLimit(
|
||||
}
|
||||
|
||||
ban, made := rq.h.ledger.BanForLimit(netblock, now, notes)
|
||||
rq.h.limiter.Reset(rq.h.clientGroup(rq.client))
|
||||
rq.h.limiter.Reset(clientGroup(rq.h.config, rq.client))
|
||||
rq.line.BanExpires = banExpires(ban)
|
||||
|
||||
if made {
|
||||
@@ -170,8 +173,10 @@ func (rq *request) banForLimit(
|
||||
// asked for. It fills in the rest of the notes. In observe mode it makes
|
||||
// no ban, and raises the alert for the ban it would have made, if that
|
||||
// alert would be sent. It returns the action for that ban.
|
||||
func (rq *request) banForAttack(now time.Time, notes bans.Notes) string {
|
||||
netblock := rq.h.netblock(rq.client)
|
||||
func (rq *request) banForAttack(
|
||||
now time.Time, notes bans.Notes,
|
||||
) requestlog.Action {
|
||||
netblock := banNetblock(rq.h.config, rq.client)
|
||||
if rq.h.config.Observe &&
|
||||
!rq.wouldAlertBan(netblock, now, bans.CauseAttack) {
|
||||
return banAction(
|
||||
@@ -211,7 +216,7 @@ func (rq *request) banForAttack(now time.Time, notes bans.Notes) string {
|
||||
// SWW_WAF_ALERT_COOLDOWN and never with no webhook set: its notes count the
|
||||
// netblock's requests, which can mean going through every client.
|
||||
func (rq *request) wouldAlertBan(
|
||||
netblock netip.Prefix, now time.Time, cause string,
|
||||
netblock netip.Prefix, now time.Time, cause bans.Cause,
|
||||
) bool {
|
||||
event := alerts.EventBan
|
||||
if rq.h.ledger.WouldBePermanent(netblock, now, cause) {
|
||||
@@ -233,7 +238,9 @@ func (rq *request) alertBan(ban bans.Ban) {
|
||||
}
|
||||
|
||||
detail := map[string]any{
|
||||
"cause": ban.Cause, "ban_expires": banExpires(ban), "notes": ban.Notes,
|
||||
"cause": string(ban.Cause),
|
||||
"ban_expires": banExpires(ban),
|
||||
"notes": ban.Notes,
|
||||
}
|
||||
if rq.h.config.Observe {
|
||||
detail["mode"] = "observe"
|
||||
@@ -273,16 +280,16 @@ func (rq *request) netblockRequests(netblock netip.Prefix) int64 {
|
||||
return rq.h.limiter.Requests(netblock) + 1
|
||||
}
|
||||
|
||||
// netblock is the netblock a ban on client covers: its IPv4 address,
|
||||
// banNetblock is the netblock a ban on client covers: its IPv4 address,
|
||||
// widened to SWW_WAF_BAN_SCOPE_V4_PREFIX, or the IPv6 group clientGroup
|
||||
// counts it in.
|
||||
func (h *handler) netblock(client netip.Addr) netip.Prefix {
|
||||
func banNetblock(cfg *config.WAF, client netip.Addr) netip.Prefix {
|
||||
addr := client.Unmap()
|
||||
if addr.Is4() {
|
||||
return netip.PrefixFrom(addr, h.config.BanScopeV4Prefix).Masked()
|
||||
return netip.PrefixFrom(addr, cfg.BanScopeV4Prefix).Masked()
|
||||
}
|
||||
|
||||
return h.clientGroup(addr)
|
||||
return clientGroup(cfg, addr)
|
||||
}
|
||||
|
||||
// banExpires is when ban ends, as the log line gives it: a time, or
|
||||
|
||||
@@ -504,7 +504,9 @@ type sender struct {
|
||||
}
|
||||
|
||||
// get sends a GET request for / from the client at from.
|
||||
func (s *sender) get(from string, status int, action string) logLine {
|
||||
func (s *sender) get(
|
||||
from string, status int, action requestlog.Action,
|
||||
) logLine {
|
||||
s.t.Helper()
|
||||
|
||||
return s.request(from, "/", status, action)
|
||||
@@ -517,7 +519,7 @@ func (s *sender) get(from string, status int, action string) logLine {
|
||||
func (s *sender) request(
|
||||
from, path string,
|
||||
status int,
|
||||
action string,
|
||||
action requestlog.Action,
|
||||
) logLine {
|
||||
s.t.Helper()
|
||||
|
||||
@@ -527,7 +529,7 @@ func (s *sender) request(
|
||||
// requestWithHeader is request with header, such as "Authorization:
|
||||
// Bearer x", added to the request unless it is "".
|
||||
func (s *sender) requestWithHeader(
|
||||
from, path, header string, status int, action string,
|
||||
from, path, header string, status int, action requestlog.Action,
|
||||
) logLine {
|
||||
s.t.Helper()
|
||||
|
||||
@@ -549,7 +551,9 @@ func (s *sender) requestWithHeader(
|
||||
// Content-Length or Transfer-Encoding. header may hold several lines,
|
||||
// separated by "\r\n". It returns the whole answer.
|
||||
func (s *sender) requestWithBody(
|
||||
method, from, path, header, body string, status int, action string,
|
||||
method, from, path, header, body string,
|
||||
status int,
|
||||
action requestlog.Action,
|
||||
) (logLine, answer) {
|
||||
s.t.Helper()
|
||||
|
||||
|
||||
@@ -0,0 +1,172 @@
|
||||
package waf
|
||||
|
||||
// The heuristics of besideLookup, and how they run beside the lookup.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||
)
|
||||
|
||||
// besideLookupTimeLimit is how long the heuristics of besideLookup have for a
|
||||
// request, from when they start. They only read the request, and no request
|
||||
// should take them that long. A request they have not finished by then is
|
||||
// refused with 503, as ActionChecksTimedOut.
|
||||
const besideLookupTimeLimit = 500 * time.Millisecond
|
||||
|
||||
// besideLookupFindings is what the heuristics of besideLookup found of a
|
||||
// request, before any of it is applied to the request.
|
||||
type besideLookupFindings struct {
|
||||
// trapPath is the request's path when it is in SWW_WAF_TRAP_PATHS, and ""
|
||||
// otherwise.
|
||||
trapPath string
|
||||
// rules are the rules of the rule files the request matched.
|
||||
rules []rules.Rule
|
||||
// inspected is true when the Core Rule Set inspected the request.
|
||||
// inspection, read and inspectionTime are then what it found, what it read
|
||||
// of the body, and how long it took.
|
||||
inspected bool
|
||||
inspection coreruleset.Result
|
||||
read []byte
|
||||
inspectionTime time.Duration
|
||||
// refuses is true when one of the heuristics refuses the request, or would
|
||||
// in observe mode. Those after it did no work.
|
||||
refuses bool
|
||||
}
|
||||
|
||||
// runBesideLookup runs the heuristics of besideLookup while the lookup waits
|
||||
// for GeoJS, and returns the first verdict, or nil when none gives one.
|
||||
// lookedUp is the channel the lookup's result comes on, or nil once afterLookup
|
||||
// has let the request through.
|
||||
//
|
||||
// The first of the two to refuse the request ends the checks: waf answers at
|
||||
// once, and what the other found is dropped. besideLookup has
|
||||
// besideLookupTimeLimit; past it, the request is refused as
|
||||
// ActionChecksTimedOut. Once both have let the request through, the heuristics
|
||||
// of afterBesideLookup run. ctx is the request's own context.
|
||||
func (rq *request) runBesideLookup(
|
||||
ctx context.Context, lookedUp <-chan lookupResult,
|
||||
) *verdict {
|
||||
var running atomic.Pointer[string]
|
||||
|
||||
running.Store(new(trapPaths{}.name()))
|
||||
|
||||
found := make(chan *besideLookupFindings, 1)
|
||||
|
||||
go func() { found <- rq.findBesideLookup(&running) }()
|
||||
|
||||
timeUp := time.NewTimer(rq.h.besideLookupLimit)
|
||||
defer timeUp.Stop()
|
||||
|
||||
var findings *besideLookupFindings
|
||||
|
||||
for findings == nil || lookedUp != nil {
|
||||
select {
|
||||
case result := <-lookedUp:
|
||||
lookedUp = nil
|
||||
|
||||
v := rq.runAfterLookup(ctx, result)
|
||||
if v != nil {
|
||||
return v
|
||||
}
|
||||
case findings = <-found:
|
||||
found = nil
|
||||
|
||||
timeUp.Stop()
|
||||
|
||||
if findings.refuses {
|
||||
return rq.applyFindings(ctx, findings)
|
||||
}
|
||||
case <-timeUp.C:
|
||||
return rq.timedOut(*running.Load())
|
||||
}
|
||||
}
|
||||
|
||||
v := rq.applyFindings(ctx, findings)
|
||||
if v != nil {
|
||||
return v
|
||||
}
|
||||
|
||||
return rq.runHeuristics(ctx, afterBesideLookup(), rq.h.now())
|
||||
}
|
||||
|
||||
// findBesideLookup does the work of the heuristics of besideLookup on the
|
||||
// request, in their order, until one refuses it, and returns what they found.
|
||||
// It changes nothing of the request or of what waf keeps, so that it can run
|
||||
// while the lookup and afterLookup do. running holds the name of the heuristic
|
||||
// at work, the first one's when findBesideLookup is called.
|
||||
func (rq *request) findBesideLookup(
|
||||
running *atomic.Pointer[string],
|
||||
) *besideLookupFindings {
|
||||
findings := &besideLookupFindings{}
|
||||
|
||||
findings.trapPath = rq.trapPath()
|
||||
if findings.trapPath != "" {
|
||||
findings.refuses = true
|
||||
|
||||
return findings
|
||||
}
|
||||
|
||||
running.Store(new(ruleFiles{}.name()))
|
||||
|
||||
findings.rules = rq.h.rules.Match(rq.in)
|
||||
if refusedByRule(findings.rules) {
|
||||
findings.refuses = true
|
||||
|
||||
return findings
|
||||
}
|
||||
|
||||
running.Store(new(coreRuleSet{}.name()))
|
||||
|
||||
cfg := rq.h.config
|
||||
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff ||
|
||||
pathExempt(rq.in, cfg.CoreRuleSetExemptPaths) {
|
||||
return findings
|
||||
}
|
||||
|
||||
start := time.Now()
|
||||
// A question's body has been read into memory, so that reading it here
|
||||
// cannot fail.
|
||||
findings.inspection, findings.read, _ = rq.h.coreRuleSet.Inspect(
|
||||
rq.in, rq.client, rq.in.Body,
|
||||
)
|
||||
findings.inspectionTime = time.Since(start)
|
||||
findings.inspected = true
|
||||
findings.refuses = rq.refusedByCoreRuleSet(findings.inspection)
|
||||
|
||||
return findings
|
||||
}
|
||||
|
||||
// applyFindings applies findings, what findBesideLookup found, to the
|
||||
// request: it runs the heuristics of besideLookup, which read them, at the
|
||||
// time it is called, and returns the first verdict, or nil when none gives
|
||||
// one.
|
||||
func (rq *request) applyFindings(
|
||||
ctx context.Context, findings *besideLookupFindings,
|
||||
) *verdict {
|
||||
rq.findings = findings
|
||||
|
||||
return rq.runHeuristics(ctx, besideLookup(), rq.h.now())
|
||||
}
|
||||
|
||||
// timedOut refuses the request for the heuristics of besideLookup not having
|
||||
// finished with it within besideLookupTimeLimit. running is the name of the
|
||||
// one at work, which the log line's heuristic gives. The refusal makes no ban
|
||||
// and is no offence.
|
||||
func (rq *request) timedOut(running string) *verdict {
|
||||
v := &verdict{
|
||||
action: requestlog.ActionChecksTimedOut,
|
||||
figures: map[string]any{
|
||||
"time_limit_ms": rq.h.besideLookupLimit.Milliseconds(),
|
||||
},
|
||||
}
|
||||
rq.line.Heuristic = running
|
||||
rq.line.HeuristicFigures = v.figures
|
||||
|
||||
return v
|
||||
}
|
||||
@@ -0,0 +1,426 @@
|
||||
package waf_test
|
||||
|
||||
import (
|
||||
"io"
|
||||
"maps"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/netip"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"testing/synctest"
|
||||
"time"
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus/testutil"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||
)
|
||||
|
||||
// checksTimeLimit is how long the trap paths, the rule files and the Core Rule
|
||||
// Set have for a request, as SPEC.md gives it, and checksTimedOut the status of
|
||||
// a request they have not finished by then.
|
||||
const (
|
||||
checksTimeLimit = 500 * time.Millisecond
|
||||
checksTimedOut = http.StatusServiceUnavailable
|
||||
)
|
||||
|
||||
// heldCoreRuleSet is a stand-in for the Core Rule Set: each inspection takes
|
||||
// took, then finds result. asked counts the inspections.
|
||||
type heldCoreRuleSet struct {
|
||||
took time.Duration
|
||||
result coreruleset.Result
|
||||
asked atomic.Int64
|
||||
}
|
||||
|
||||
// Inspect inspects a request as the stand-in does, and reads all of body.
|
||||
func (c *heldCoreRuleSet) Inspect(
|
||||
_ *http.Request, _ netip.Addr, body io.Reader,
|
||||
) (coreruleset.Result, []byte, error) {
|
||||
c.asked.Add(1)
|
||||
time.Sleep(c.took)
|
||||
|
||||
read, err := io.ReadAll(body)
|
||||
|
||||
return c.result, read, err
|
||||
}
|
||||
|
||||
// coreRuleSetMatch returns what the stand-in finds of an attack: rule 942100,
|
||||
// with a score of 5, the default SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD.
|
||||
func coreRuleSetMatch() coreruleset.Result {
|
||||
return coreruleset.Result{RuleIDs: []int{942100}, Score: 5}
|
||||
}
|
||||
|
||||
// newWAFHoldingTheCoreRuleSet makes a waf, to be asked in a synctest bubble,
|
||||
// with crs in place of the Core Rule Set, in block mode. GeoJS never answers,
|
||||
// a setting needs its answer, alerts go to a webhook, and everything else is
|
||||
// at its default but for env. No request to it may use the network.
|
||||
func newWAFHoldingTheCoreRuleSet(
|
||||
t *testing.T, crs *heldCoreRuleSet, env map[string]string,
|
||||
) (*waf.Server, *output, *alerts.Queue) {
|
||||
t.Helper()
|
||||
|
||||
settings := map[string]string{
|
||||
tagDeny: denyKP, coreRuleSetMode: block,
|
||||
alertWebhookURL: alertWebhook,
|
||||
}
|
||||
maps.Copy(settings, env)
|
||||
|
||||
server, out, queue := newWAFWith(t, func(params waf.Params) *waf.Server {
|
||||
params.CoreRuleSet = crs
|
||||
params.BesideLookupTimeLimit = waf.BesideLookupTimeLimit
|
||||
|
||||
return waf.New(params)
|
||||
}, silentGeoJSURL, abuseIPDBURL, time.Now, settings)
|
||||
|
||||
return server, out, queue
|
||||
}
|
||||
|
||||
// lookupWait is SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS for wait.
|
||||
func lookupWait(wait time.Duration) string {
|
||||
return strconv.FormatInt(wait.Milliseconds(), 10)
|
||||
}
|
||||
|
||||
// wantAnswer checks the status of an answer and how long it took.
|
||||
func wantAnswer(
|
||||
t *testing.T, status int, took time.Duration,
|
||||
wantStatus int, wantTook time.Duration, out *output,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
if status != wantStatus || took != wantTook {
|
||||
t.Errorf(
|
||||
"answered %d after %s, want %d after %s; output:\n%s",
|
||||
status, took, wantStatus, wantTook, out.text(),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// wantNoCoreRuleSetTrace checks that what the Core Rule Set found of a request
|
||||
// left no trace: line has none of its fields, and server has counted no match
|
||||
// of it and raised no alert for one.
|
||||
func wantNoCoreRuleSetTrace(
|
||||
t *testing.T, server *waf.Server, queue *alerts.Queue, line logLine,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
if line.DurationCoreRuleSetMS != nil || line.CoreRuleSetScore != nil ||
|
||||
line.CoreRuleSetRuleIDs != nil {
|
||||
t.Errorf(
|
||||
"log line has Core Rule Set time %v, score %v and rule ids %v, want none",
|
||||
line.DurationCoreRuleSetMS,
|
||||
line.CoreRuleSetScore,
|
||||
line.CoreRuleSetRuleIDs,
|
||||
)
|
||||
}
|
||||
|
||||
recorder := httptest.NewRecorder()
|
||||
server.Metrics.ServeHTTP(
|
||||
recorder,
|
||||
httptest.NewRequestWithContext(
|
||||
t.Context(),
|
||||
http.MethodGet,
|
||||
waf.MetricsPath,
|
||||
http.NoBody,
|
||||
),
|
||||
)
|
||||
|
||||
if strings.Contains(
|
||||
recorder.Body.String(),
|
||||
"sww_core_rule_set_matches_total{",
|
||||
) {
|
||||
t.Errorf("a Core Rule Set match is counted")
|
||||
}
|
||||
|
||||
for _, alert := range queue.Snapshot().Waiting[alerts.DestinationWebhook] {
|
||||
if alert.Event == alerts.EventCoreRuleSetMatch {
|
||||
t.Errorf("alert raised: %+v", alert)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheLookupAndTheCoreRuleSetTakeTheLongerOfTheirTimesNotTheirSum(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
inspection := checksTimeLimit - time.Millisecond
|
||||
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
// wait is how long the request waits for GeoJS.
|
||||
wait time.Duration
|
||||
}{
|
||||
{"the lookup waits longer", defaultLookupWait},
|
||||
{"the inspection takes longer", inspection / 2},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
crs := &heldCoreRuleSet{took: inspection}
|
||||
server, out, _ := newWAFHoldingTheCoreRuleSet(
|
||||
t,
|
||||
crs,
|
||||
map[string]string{
|
||||
asnAndCountryLookupTimeout: lookupWait(tc.wait),
|
||||
},
|
||||
)
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
defer waitForGeoJSRequestsToBeAbandoned()
|
||||
|
||||
status, took := askDirectly(
|
||||
t,
|
||||
server,
|
||||
func(*http.Request) {},
|
||||
)
|
||||
|
||||
wantAnswer(t, status, took, http.StatusOK,
|
||||
max(tc.wait, inspection), out)
|
||||
wantLine(t, out.requestLine(t), http.StatusOK,
|
||||
requestlog.ActionForward)
|
||||
})
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// firstToRefuse is a case of
|
||||
// TestTheFirstOfTheLookupAndTheRequestChecksToRefuseIsAnswered: a request
|
||||
// under env, whose inspection by the Core Rule Set takes inspection, refused as
|
||||
// action after took.
|
||||
type firstToRefuse struct {
|
||||
name string
|
||||
env map[string]string
|
||||
inspection time.Duration
|
||||
took time.Duration
|
||||
action requestlog.Action
|
||||
}
|
||||
|
||||
// firstToRefuseCases are the cases of
|
||||
// TestTheFirstOfTheLookupAndTheRequestChecksToRefuseIsAnswered.
|
||||
func firstToRefuseCases() []firstToRefuse {
|
||||
return []firstToRefuse{
|
||||
{
|
||||
name: "a trap path, before GeoJS answers",
|
||||
env: map[string]string{trapPaths: "/", allowedCountries: "DE"},
|
||||
took: 0, action: requestlog.ActionTemporarilyBanned,
|
||||
},
|
||||
{
|
||||
name: "the Core Rule Set, before GeoJS answers",
|
||||
env: map[string]string{allowedCountries: "DE"},
|
||||
inspection: 100 * time.Millisecond,
|
||||
took: 100 * time.Millisecond, action: requestlog.ActionBlockedByCoreRuleSet,
|
||||
},
|
||||
{
|
||||
name: "the country list, while the Core Rule Set inspects",
|
||||
env: map[string]string{
|
||||
allowedCountries: "DE", asnAndCountryLookupTimeout: "100",
|
||||
},
|
||||
inspection: 400 * time.Millisecond,
|
||||
took: 100 * time.Millisecond, action: requestlog.ActionCountryDenied,
|
||||
},
|
||||
{
|
||||
name: "the rate limits, while the Core Rule Set inspects",
|
||||
env: map[string]string{
|
||||
rateLimitPerMinute: "2", unknownCountryLimitPercent: "50",
|
||||
asnAndCountryLookupTimeout: "100",
|
||||
},
|
||||
inspection: 400 * time.Millisecond,
|
||||
took: 100 * time.Millisecond, action: requestlog.ActionClientIsRateLimited,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheFirstOfTheLookupAndTheRequestChecksToRefuseIsAnswered(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
const forbidden = http.StatusForbidden
|
||||
|
||||
for _, tc := range firstToRefuseCases() {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
crs := &heldCoreRuleSet{
|
||||
took: tc.inspection,
|
||||
result: coreRuleSetMatch(),
|
||||
}
|
||||
server, out, queue := newWAFHoldingTheCoreRuleSet(t, crs, tc.env)
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
defer waitForGeoJSRequestsToBeAbandoned()
|
||||
|
||||
// Under the rate limits of the last case, the client's next
|
||||
// request takes it over a limit only at the percentage of an
|
||||
// unknown country, so it waits for GeoJS.
|
||||
countClient(t, server, time.Now())
|
||||
|
||||
status, took := askDirectly(
|
||||
t,
|
||||
server,
|
||||
func(*http.Request) {},
|
||||
)
|
||||
|
||||
wantAnswer(t, status, took, forbidden, tc.took, out)
|
||||
|
||||
line := out.requestLine(t)
|
||||
wantLine(t, line, forbidden, tc.action)
|
||||
|
||||
if tc.action == requestlog.ActionBlockedByCoreRuleSet {
|
||||
return
|
||||
}
|
||||
|
||||
if tc.action != requestlog.ActionClientIsRateLimited &&
|
||||
line.Counts != (ratelimit.Counts{}) {
|
||||
t.Errorf(
|
||||
"the rate limits counted the request: %+v",
|
||||
line.Counts,
|
||||
)
|
||||
}
|
||||
|
||||
// What the Core Rule Set finds once the request is answered is
|
||||
// dropped.
|
||||
time.Sleep(tc.inspection)
|
||||
wantNoCoreRuleSetTrace(t, server, queue, line)
|
||||
})
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestARequestRefusedBeforeTheLookupStartsNoInspection(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
crs := &heldCoreRuleSet{took: time.Second}
|
||||
server, out, _ := newWAFHoldingTheCoreRuleSet(t, crs, nil)
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
banClient(t, server, time.Now())
|
||||
|
||||
status, took := askDirectly(t, server, func(*http.Request) {})
|
||||
|
||||
wantAnswer(t, status, took, http.StatusForbidden, 0, out)
|
||||
|
||||
requests := testutil.ToFloat64(server.Metrics.GeoJSRequests)
|
||||
if asked := crs.asked.Load(); asked != 0 || requests != 0 {
|
||||
t.Errorf(
|
||||
"%d inspections and %v requests to GeoJS, want none",
|
||||
asked,
|
||||
requests,
|
||||
)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestAClientInTheAllowNetsIsAllowedAtOnceAndNothingElseStarts(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
crs := &heldCoreRuleSet{took: time.Second, result: coreRuleSetMatch()}
|
||||
server, out, _ := newWAFHoldingTheCoreRuleSet(t, crs,
|
||||
map[string]string{allowNets: client, allowedCountries: "DE"})
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
status, took := askDirectly(t, server, func(*http.Request) {})
|
||||
|
||||
wantAnswer(t, status, took, http.StatusOK, 0, out)
|
||||
|
||||
requests := testutil.ToFloat64(server.Metrics.GeoJSRequests)
|
||||
if asked := crs.asked.Load(); asked != 0 || requests != 0 {
|
||||
t.Errorf(
|
||||
"%d inspections and %v requests to GeoJS, want none",
|
||||
asked,
|
||||
requests,
|
||||
)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestARequestTheCoreRuleSetHasNotFinishedWithIn500MillisecondsIsRefused(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
crs := &heldCoreRuleSet{took: checksTimeLimit + time.Second}
|
||||
server, out, _ := newWAFHoldingTheCoreRuleSet(t, crs,
|
||||
map[string]string{refusalsLimitPerMinute: "1"})
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
defer waitForGeoJSRequestsToBeAbandoned()
|
||||
|
||||
// Were it counted toward the refusal limit, the second refusal would
|
||||
// ban the client, and the third request would be refused for the ban.
|
||||
for i := range 3 {
|
||||
status, took := askDirectly(t, server, func(req *http.Request) {
|
||||
req.Header.Set(requestIDHeader, "id-"+strconv.Itoa(i))
|
||||
})
|
||||
|
||||
wantAnswer(t, status, took, checksTimedOut, checksTimeLimit, out)
|
||||
|
||||
line := out.requestLines(t, i+1)[i]
|
||||
wantLine(
|
||||
t,
|
||||
line,
|
||||
checksTimedOut,
|
||||
requestlog.ActionChecksTimedOut,
|
||||
)
|
||||
wantHeuristic(t, line, "core_rule_set", `{"time_limit_ms": 500}`)
|
||||
|
||||
if line.Offence != "" || line.BanExpires != "" {
|
||||
t.Errorf(
|
||||
"log line has offence %q and ban_expires %q, want neither",
|
||||
line.Offence,
|
||||
line.BanExpires,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
if bans := server.Ledger.Snapshot(); len(bans) != 0 {
|
||||
t.Errorf("bans %+v, want none", bans)
|
||||
}
|
||||
|
||||
offences := historyOf(t, server, client).Offences
|
||||
if offences != (ratelimit.Offences{}) {
|
||||
t.Errorf("history counts the offences %+v, want none", offences)
|
||||
}
|
||||
|
||||
time.Sleep(crs.took)
|
||||
})
|
||||
}
|
||||
|
||||
func TestObserveModeAllowsARequestTheCoreRuleSetHasNotFinishedWith(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
crs := &heldCoreRuleSet{took: checksTimeLimit + time.Second}
|
||||
server, out, _ := newWAFHoldingTheCoreRuleSet(t, crs,
|
||||
map[string]string{mode: observe})
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
defer waitForGeoJSRequestsToBeAbandoned()
|
||||
|
||||
status, took := askDirectly(t, server, func(*http.Request) {})
|
||||
|
||||
wantAnswer(t, status, took, http.StatusOK, checksTimeLimit, out)
|
||||
|
||||
line := out.requestLine(t)
|
||||
wantLine(t, line, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
if line.WouldAction != requestlog.ActionChecksTimedOut {
|
||||
t.Errorf(
|
||||
"log line has would_action %q, want %q",
|
||||
line.WouldAction,
|
||||
requestlog.ActionChecksTimedOut,
|
||||
)
|
||||
}
|
||||
|
||||
time.Sleep(crs.took)
|
||||
})
|
||||
}
|
||||
+10
-9
@@ -53,33 +53,34 @@ func (rq *request) limitPercentages() (percentage, percentage) {
|
||||
if country == "" && !rq.skippedLookupWait {
|
||||
unknown = percentage{
|
||||
cfg.UnknownCountryLimitPercent,
|
||||
"SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT",
|
||||
config.UnknownCountryLimitPercentSetting,
|
||||
}
|
||||
}
|
||||
|
||||
fetched := percentage{percent: whole}
|
||||
if percent, listed := rq.h.lists.ASNLimitPercent(asn); listed {
|
||||
fetched = percentage{percent, "SWW_WAF_ASN_LIMIT_PERCENT_URL"}
|
||||
fetched = percentage{percent, config.ASNLimitPercentURLSetting}
|
||||
}
|
||||
|
||||
blocklisted := percentage{percent: whole}
|
||||
if rq.blocklisted && cfg.BlocklistAction == "limit" {
|
||||
if rq.blocklisted &&
|
||||
cfg.BlocklistAction == config.ReputationActionLimit {
|
||||
blocklisted = percentage{
|
||||
cfg.BlocklistLimitPercent,
|
||||
"SWW_WAF_BLOCKLIST_ACTION",
|
||||
config.BlocklistActionSetting,
|
||||
}
|
||||
}
|
||||
|
||||
dnsblAbuseIPDBListed := percentage{percent: whole}
|
||||
if (rq.dnsblListed || rq.abuseIPDBHit) &&
|
||||
cfg.DNSBLAbuseIPDBAction == "limit" {
|
||||
cfg.DNSBLAbuseIPDBAction == config.ReputationActionLimit {
|
||||
dnsblAbuseIPDBListed = percentage{
|
||||
cfg.DNSBLAbuseIPDBLimitPercent,
|
||||
"SWW_WAF_DNSBL_ABUSEIPDB_ACTION",
|
||||
config.DNSBLAbuseIPDBActionSetting,
|
||||
}
|
||||
}
|
||||
|
||||
const tagSetting = "SWW_WAF_TAG_LIMIT_PERCENT"
|
||||
const tagSetting = config.TagLimitPercentSetting
|
||||
|
||||
asnRequests := lowest(
|
||||
given(cfg.TagLimitPercent, asnTag(asn), tagSetting),
|
||||
@@ -107,7 +108,7 @@ func (rq *request) limitPercentages() (percentage, percentage) {
|
||||
asnBytes = given(
|
||||
cfg.ASNBytesLimitPercent,
|
||||
asn,
|
||||
"SWW_WAF_ASN_BYTES_LIMIT_PERCENT",
|
||||
config.ASNBytesLimitPercentSetting,
|
||||
)
|
||||
}
|
||||
|
||||
@@ -115,7 +116,7 @@ func (rq *request) limitPercentages() (percentage, percentage) {
|
||||
countryBytes = given(
|
||||
cfg.CountryBytesLimitPercent,
|
||||
country,
|
||||
"SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT",
|
||||
config.CountryBytesLimitPercentSetting,
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
@@ -73,7 +73,7 @@ func TestEachBiasedThresholdLowersTheRateLimits(t *testing.T) {
|
||||
// Half of 4 requests a minute: the third breaks the limit.
|
||||
for _, sent := range []struct {
|
||||
status int
|
||||
action string
|
||||
action requestlog.Action
|
||||
}{
|
||||
{http.StatusOK, requestlog.ActionForward},
|
||||
{http.StatusOK, requestlog.ActionForward},
|
||||
@@ -339,6 +339,8 @@ func TestClientWithoutAnAnswerInTimeHasTheUnknownCountryLimitPercent(
|
||||
// TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays says, with
|
||||
// a GeoJS that never answers.
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
defer waitForGeoJSRequestsToBeAbandoned()
|
||||
|
||||
server, out, _ := newWAF(t, silentGeoJSURL, time.Now,
|
||||
map[string]string{unknownCountryLimitPercent: "0"})
|
||||
|
||||
@@ -390,6 +392,8 @@ func TestRequestWaitsForItsLookupWhileABiasedThresholdIsSet(t *testing.T) {
|
||||
// with a GeoJS that never answers. The headers, which need the
|
||||
// answer too, are off.
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
defer waitForGeoJSRequestsToBeAbandoned()
|
||||
|
||||
server, out, _ := newWAF(t, silentGeoJSURL, time.Now,
|
||||
map[string]string{
|
||||
asnAndCountryLookupTimeout: strconv.FormatInt(
|
||||
@@ -448,7 +452,7 @@ func TestBanForALoweredLimitGivesThePercentageInItsNotesAndItsAlert(
|
||||
// limit, which is answered with status and logged with action.
|
||||
before int
|
||||
status int
|
||||
action string
|
||||
action requestlog.Action
|
||||
// reason and want are the ban's reason, and its notes' limit
|
||||
// percentage, as percentText gives it.
|
||||
reason, want string
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
@@ -41,5 +42,6 @@ func (blocklists) check(
|
||||
// client, given listedBy, the blocklists that list it: it does when one lists
|
||||
// it and SWW_WAF_BLOCKLIST_ACTION is deny.
|
||||
func (rq *request) refusedByBlocklists(listedBy []string) bool {
|
||||
return len(listedBy) > 0 && rq.h.config.BlocklistAction == deny
|
||||
return len(listedBy) > 0 && rq.h.config.BlocklistAction ==
|
||||
config.ReputationActionDeny
|
||||
}
|
||||
@@ -30,7 +30,7 @@ func (byteLimits) check(
|
||||
}
|
||||
|
||||
counts, hit, over := rq.h.limiter.CountBytes(
|
||||
rq.h.clientGroup(rq.client),
|
||||
clientGroup(rq.h.config, rq.client),
|
||||
now,
|
||||
rq.countedBytes(),
|
||||
rq.bytesLimitPercent.percent,
|
||||
|
||||
@@ -5,6 +5,8 @@ import (
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"slices"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
)
|
||||
|
||||
// peerAddress is the address of the request's TCP peer.
|
||||
@@ -45,10 +47,10 @@ func scheme(r *http.Request, isQuestion bool) string {
|
||||
// address, or its IPv6 group, the netblock its IPv6 address is in of the
|
||||
// length SWW_WAF_IPV6_GROUP_PREFIX sets, a /64 by default, since one abuser
|
||||
// usually holds a whole /64. An IPv4 address in IPv6 form counts as IPv4.
|
||||
func (h *handler) clientGroup(addr netip.Addr) netip.Prefix {
|
||||
func clientGroup(cfg *config.WAF, addr netip.Addr) netip.Prefix {
|
||||
addr = addr.Unmap()
|
||||
if addr.Is6() {
|
||||
return netip.PrefixFrom(addr, h.config.IPv6GroupPrefix).Masked()
|
||||
return netip.PrefixFrom(addr, cfg.IPv6GroupPrefix).Masked()
|
||||
}
|
||||
|
||||
return netip.PrefixFrom(addr, addr.BitLen())
|
||||
|
||||
+26
-18
@@ -13,10 +13,11 @@ import (
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// coreRuleSet inspects the request with the Core Rule Set, unless
|
||||
// SWW_WAF_CORE_RULE_SET_MODE is off or SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS
|
||||
// exempts its path, as pathExempt decides, and notes the rules it matched and
|
||||
// its score in the log line, and the rules in the metrics. A score at or over
|
||||
// coreRuleSet takes what the Core Rule Set found of the request, which
|
||||
// findBesideLookup inspected with it unless SWW_WAF_CORE_RULE_SET_MODE is off
|
||||
// or SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS exempts its path, as pathExempt
|
||||
// decides. It notes the rules the request matched and its score in the log
|
||||
// line, and the rules in the metrics. A score at or over
|
||||
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD is a match: it adds the tag
|
||||
// exploitsignature, raises the core_rule_set_match alert, and in block mode
|
||||
// refuses the request with 403, which is an offence its client's history
|
||||
@@ -30,27 +31,23 @@ func (coreRuleSet) check(
|
||||
rq *request,
|
||||
_ time.Time,
|
||||
) *verdict {
|
||||
cfg := rq.h.config
|
||||
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff ||
|
||||
pathExempt(rq.in, cfg.CoreRuleSetExemptPaths) {
|
||||
findings := rq.findings
|
||||
if !findings.inspected {
|
||||
return nil
|
||||
}
|
||||
|
||||
start := time.Now()
|
||||
|
||||
// A question's body has been read into memory, so that reading it here
|
||||
// cannot fail.
|
||||
result, read, _ := rq.h.coreRuleSet.Inspect(rq.in, rq.client, rq.in.Body)
|
||||
rq.requestBytes = int64(len(read))
|
||||
cfg := rq.h.config
|
||||
result := findings.inspection
|
||||
rq.requestBytes = int64(len(findings.read))
|
||||
|
||||
rq.line.DurationCoreRuleSetMS = new(
|
||||
requestlog.Milliseconds(time.Since(start)),
|
||||
requestlog.Milliseconds(findings.inspectionTime),
|
||||
)
|
||||
rq.line.CoreRuleSetRuleIDs = result.RuleIDs
|
||||
rq.line.CoreRuleSetScore = &result.Score
|
||||
|
||||
for _, id := range result.RuleIDs {
|
||||
rq.h.metrics.CoreRuleSetMatched(cfg.CoreRuleSetMode, id)
|
||||
rq.h.metrics.CoreRuleSetMatched(string(cfg.CoreRuleSetMode), id)
|
||||
}
|
||||
|
||||
threshold := cfg.CoreRuleSetScoreThreshold
|
||||
@@ -61,7 +58,7 @@ func (coreRuleSet) check(
|
||||
rq.addTag(config.ExploitSignatureTag)
|
||||
rq.alertCoreRuleSetMatch(result)
|
||||
|
||||
if cfg.CoreRuleSetMode == config.CoreRuleSetModeDetect {
|
||||
if !rq.refusedByCoreRuleSet(result) {
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -76,6 +73,17 @@ func (coreRuleSet) check(
|
||||
}
|
||||
}
|
||||
|
||||
// refusedByCoreRuleSet reports whether coreRuleSet refuses a request the Core
|
||||
// Rule Set found result of: whether the score is at or over
|
||||
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD in block mode.
|
||||
func (rq *request) refusedByCoreRuleSet(result coreruleset.Result) bool {
|
||||
cfg := rq.h.config
|
||||
threshold := cfg.CoreRuleSetScoreThreshold
|
||||
|
||||
return cfg.CoreRuleSetMode == config.CoreRuleSetModeBlock &&
|
||||
threshold != 0 && result.Score >= threshold
|
||||
}
|
||||
|
||||
// alertCoreRuleSetMatch raises the core_rule_set_match alert for the request,
|
||||
// which the Core Rule Set scored at result, at or over
|
||||
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD. Its detail gives the rule ids, the
|
||||
@@ -91,7 +99,7 @@ func (rq *request) alertCoreRuleSetMatch(result coreruleset.Result) {
|
||||
|
||||
switch {
|
||||
case rq.h.config.CoreRuleSetMode == config.CoreRuleSetModeDetect:
|
||||
detail["mode"] = config.CoreRuleSetModeDetect
|
||||
detail["mode"] = string(config.CoreRuleSetModeDetect)
|
||||
case rq.h.config.Observe:
|
||||
detail["mode"] = "observe"
|
||||
}
|
||||
@@ -100,7 +108,7 @@ func (rq *request) alertCoreRuleSetMatch(result coreruleset.Result) {
|
||||
Event: alerts.EventCoreRuleSetMatch,
|
||||
Host: rq.site,
|
||||
Client: rq.client,
|
||||
Netblock: rq.h.clientGroup(rq.client),
|
||||
Netblock: clientGroup(rq.h.config, rq.client),
|
||||
ASN: rq.line.ASN,
|
||||
ASName: rq.line.ASName,
|
||||
Country: rq.line.Country,
|
||||
|
||||
@@ -84,7 +84,7 @@ func TestCoreRuleSetMatchAddsTheExploitSignatureTag(t *testing.T) {
|
||||
name string
|
||||
mode string
|
||||
status int
|
||||
action string
|
||||
action requestlog.Action
|
||||
}{
|
||||
{block, block, http.StatusForbidden, requestlog.ActionBlockedByCoreRuleSet},
|
||||
{detect, detect, http.StatusOK, requestlog.ActionForward},
|
||||
@@ -158,8 +158,9 @@ func TestCoreRuleSetRefusesAttacksInBlockModeAndOnlyLogsThemInDetectMode(
|
||||
t.Parallel()
|
||||
|
||||
for _, tc := range []struct {
|
||||
mode, action string
|
||||
status int
|
||||
mode string
|
||||
action requestlog.Action
|
||||
status int
|
||||
}{
|
||||
{block, requestlog.ActionBlockedByCoreRuleSet, http.StatusForbidden},
|
||||
{detect, requestlog.ActionForward, http.StatusOK},
|
||||
@@ -652,8 +653,9 @@ func TestCoreRuleSetMatchRaisesTheCoreRuleSetMatchAlert(t *testing.T) {
|
||||
// status and action are what the request is answered and logged
|
||||
// with, and alertMode what the alert's detail gives as mode, if
|
||||
// anything.
|
||||
status int
|
||||
action, alertMode string
|
||||
status int
|
||||
action requestlog.Action
|
||||
alertMode string
|
||||
}{
|
||||
{
|
||||
"block", map[string]string{coreRuleSetMode: block},
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"slices"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
@@ -33,7 +34,8 @@ func (countries) check(
|
||||
return &verdict{
|
||||
action: requestlog.ActionCountryDenied,
|
||||
figures: map[string]any{
|
||||
"country": country, "setting": "SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES",
|
||||
"country": country,
|
||||
"setting": config.ExclusivelyAllowedCountriesSetting,
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -316,7 +316,7 @@ func TestExclusiveListRefusesAPrivateAddressUnlessAllowed(t *testing.T) {
|
||||
name string
|
||||
allowNets string
|
||||
status int
|
||||
action string
|
||||
action requestlog.Action
|
||||
}{
|
||||
{
|
||||
"not in SWW_WAF_ALLOW_NETS", "",
|
||||
|
||||
@@ -32,7 +32,8 @@ func (dnsbl) check(ctx context.Context, rq *request, _ time.Time) *verdict {
|
||||
rq.addTag(config.DNSBLTagPrefix + strings.ToLower(zone))
|
||||
}
|
||||
|
||||
if !rq.dnsblListed || rq.h.config.DNSBLAbuseIPDBAction != deny {
|
||||
if !rq.dnsblListed ||
|
||||
rq.h.config.DNSBLAbuseIPDBAction != config.ReputationActionDeny {
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
package waf
|
||||
|
||||
import "sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||
|
||||
// BesideLookupTimeLimit is besideLookupTimeLimit.
|
||||
const BesideLookupTimeLimit = besideLookupTimeLimit
|
||||
|
||||
// BanAbuseIPDBBlacklist does on s what a fetch of AbuseIPDB's blacklist with
|
||||
// the clients on listed does.
|
||||
func (s *Server) BanAbuseIPDBBlacklist(listed []reputation.BlacklistEntry) {
|
||||
h := s.Handler.(*handler) //nolint:forcetypeassert // New makes it one
|
||||
BanAbuseIPDBBlacklist(h.config, h.ledger, h.processLog, h.now(), listed)
|
||||
}
|
||||
@@ -10,15 +10,11 @@ import (
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// The header limits: the most Cookie lines, cookies and header fields a request
|
||||
// may have, and the longest its request line and each header line may be, in
|
||||
// bytes, not counting the line ending. Real clients rarely reach them.
|
||||
// maxLineBytes is what nginx allows by default. A browser sends all its cookies
|
||||
// on one Cookie line, as RFC 6265 requires, and Go's server joins the Cookie
|
||||
// lines of an HTTP/2 request into one. A second line would only give a visitor
|
||||
// another 8 KiB for the Core Rule Set to read.
|
||||
// The header limits: the most cookies and header fields a request may have,
|
||||
// and the longest its request line and each header line may be, in bytes, not
|
||||
// counting the line ending. Real clients rarely reach them. maxLineBytes is
|
||||
// what nginx allows by default.
|
||||
const (
|
||||
maxCookieLines = 1
|
||||
maxCookies = 50
|
||||
maxHeaderFields = 100
|
||||
maxLineBytes = 8 << 10
|
||||
@@ -26,11 +22,10 @@ const (
|
||||
|
||||
// headerLimits refuses a request with a request line longer than maxLineBytes,
|
||||
// with 414. It refuses a request with a header line longer than maxLineBytes,
|
||||
// more than maxHeaderFields header fields, more than maxCookieLines Cookie
|
||||
// lines or more than maxCookies cookies, with 431. Its figures are the request
|
||||
// line's length and its limit for 414. For 431 they are the longest header
|
||||
// line, the header fields, the Cookie lines and the cookies, each with its
|
||||
// limit.
|
||||
// more than maxHeaderFields header fields or more than maxCookies cookies, with
|
||||
// 431. Its figures are the request line's length and its limit for 414. For
|
||||
// 431 they are the longest header line, the header fields and the cookies,
|
||||
// each with its limit.
|
||||
type headerLimits struct{}
|
||||
|
||||
func (headerLimits) name() string { return "header_limits" }
|
||||
@@ -53,10 +48,9 @@ func (headerLimits) check(
|
||||
|
||||
longest := rq.longestHeaderLineBytes()
|
||||
fields, cookies := rq.headerCounts()
|
||||
cookieLines := len(rq.in.Header.Values("Cookie"))
|
||||
|
||||
if longest <= maxLineBytes && fields <= maxHeaderFields &&
|
||||
cookieLines <= maxCookieLines && cookies <= maxCookies {
|
||||
cookies <= maxCookies {
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -66,7 +60,6 @@ func (headerLimits) check(
|
||||
"longest_header_line_bytes": longest,
|
||||
"header_line_bytes_limit": maxLineBytes,
|
||||
"fields": fields, "fields_limit": maxHeaderFields,
|
||||
"cookie_lines": cookieLines, "cookie_lines_limit": maxCookieLines,
|
||||
"cookies": cookies, "cookies_limit": maxCookies,
|
||||
},
|
||||
}
|
||||
|
||||
+23
-12
@@ -3,6 +3,8 @@ package waf
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// To add a heuristic, write one file with a type that implements heuristic,
|
||||
@@ -20,10 +22,10 @@ type heuristic interface {
|
||||
}
|
||||
|
||||
// verdict is a heuristic refusing a request, or banning its client: the action
|
||||
// the log line names, one of requestlog's Action constants, and the figures the
|
||||
// heuristic acted on, by name, such as the count and the limit.
|
||||
// the log line names, and the figures the heuristic acted on, by name, such as
|
||||
// the count and the limit.
|
||||
type verdict struct {
|
||||
action string
|
||||
action requestlog.Action
|
||||
figures map[string]any
|
||||
}
|
||||
|
||||
@@ -54,8 +56,6 @@ func beforeLookup() []heuristic {
|
||||
}
|
||||
|
||||
// afterLookup are the heuristics that run once the lookup has been made.
|
||||
// tag_deny runs twice: before the rate limits, on the tags known by then, and
|
||||
// last, on the tag rule_files and core_rule_set can add.
|
||||
func afterLookup() []heuristic {
|
||||
return []heuristic{
|
||||
countries{},
|
||||
@@ -64,9 +64,26 @@ func afterLookup() []heuristic {
|
||||
abuseIPDB{},
|
||||
tagDeny{},
|
||||
rateLimits{},
|
||||
}
|
||||
}
|
||||
|
||||
// besideLookup are the heuristics that only read the request. Their work runs
|
||||
// beside the lookup and afterLookup, in findBesideLookup, which changes nothing
|
||||
// of the request. Their check then applies what that work found, and runs only
|
||||
// once waf takes it.
|
||||
func besideLookup() []heuristic {
|
||||
return []heuristic{
|
||||
trapPaths{},
|
||||
ruleFiles{},
|
||||
coreRuleSet{},
|
||||
}
|
||||
}
|
||||
|
||||
// afterBesideLookup are the heuristics that run once afterLookup and
|
||||
// besideLookup have both let the request through. tag_deny runs a second time,
|
||||
// on the tag rule_files and core_rule_set can add.
|
||||
func afterBesideLookup() []heuristic {
|
||||
return []heuristic{
|
||||
tagDeny{},
|
||||
}
|
||||
}
|
||||
@@ -89,13 +106,7 @@ func onReport() []heuristic {
|
||||
}
|
||||
}
|
||||
|
||||
// runAfterAnswer runs the heuristics of afterAnswer on the request, at the time
|
||||
// it is called.
|
||||
func (rq *request) runAfterAnswer(ctx context.Context) {
|
||||
rq.runBanningHeuristics(ctx, afterAnswer())
|
||||
}
|
||||
|
||||
// runHeuristics runs list, beforeLookup or afterLookup, on the request at now,
|
||||
// runHeuristics runs list, one of those checkClient runs, on the request at now,
|
||||
// and returns the first verdict, or nil when none gives one. The log line's
|
||||
// heuristic and heuristic_figures name the heuristic that gave it, and give its
|
||||
// figures. ctx is the request's own context.
|
||||
|
||||
+107
-30
@@ -2,13 +2,18 @@ package waf
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||
)
|
||||
|
||||
// The headers in which an answer of allow gives rproxy the client's AS number
|
||||
@@ -20,33 +25,81 @@ const (
|
||||
countryHeader = "X-Client-Country"
|
||||
)
|
||||
|
||||
// lookUp looks up the client's AS number and country, in the lookup
|
||||
// database or through GeoJS, and notes them for the log line and as the
|
||||
// request's asn= and country= tags, unless
|
||||
// SWW_WAF_LOOKUP_SOURCE is off or the client is on a private, loopback or
|
||||
// link-local address, which no lookup can place. The lookup database
|
||||
// answers at once. With GeoJS, while a setting needs the answer, such as a
|
||||
// country list or a biased threshold, a new client's request waits for it,
|
||||
// unless refusedWhateverTheAnswer reports true. ctx is the request's own
|
||||
// context.
|
||||
func (rq *request) lookUp(ctx context.Context, now time.Time) {
|
||||
if rq.h.config.LookupSource == "off" || !canBePlaced(rq.client) {
|
||||
// lookupResult is what the lookup gave for a request: whether its client was
|
||||
// looked up, the answer, and whether the request skipped the wait for GeoJS.
|
||||
type lookupResult struct {
|
||||
lookedUp, skippedWait bool
|
||||
answer geojs.Answer
|
||||
}
|
||||
|
||||
// startLookUp starts looking up the client's AS number and country, in the
|
||||
// lookup database or through GeoJS, unless SWW_WAF_LOOKUP_SOURCE is off or the
|
||||
// client is on a private, loopback or link-local address, which no lookup can
|
||||
// place. It returns the channel the result comes on. A result that needs no
|
||||
// wait is on it when startLookUp returns: the lookup database's, an answer
|
||||
// GeoJS gave earlier, or none. With GeoJS, while a setting needs the answer,
|
||||
// such as a country list or a biased threshold, a new client's request waits
|
||||
// for it up to SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS, unless
|
||||
// refusedWhateverTheAnswer reports true. The wait goes on to its end even once
|
||||
// the request has been answered, so that it ends as it would have for a
|
||||
// request that waited. startLookUp changes nothing of the request. ctx is the
|
||||
// request's own context.
|
||||
func (rq *request) startLookUp(
|
||||
ctx context.Context, now time.Time,
|
||||
) <-chan lookupResult {
|
||||
result := make(chan lookupResult, 1)
|
||||
|
||||
if rq.h.config.LookupSource == config.LookupSourceOff ||
|
||||
!canBePlaced(rq.client) {
|
||||
result <- lookupResult{}
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
client := clientGroup(rq.h.config, rq.client)
|
||||
|
||||
if rq.h.config.LookupSource == config.LookupSourceFile {
|
||||
result <- lookupResult{
|
||||
lookedUp: true, answer: rq.h.lookupFile.LookUp(client),
|
||||
}
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
answer := rq.h.geojs.LookUpWithoutWaiting(ctx, client)
|
||||
refused := rq.refusedWhateverTheAnswer(now)
|
||||
|
||||
if !answer.Answered.IsZero() || refused || !rq.h.waitForGeoJS {
|
||||
result <- lookupResult{
|
||||
lookedUp: true,
|
||||
skippedWait: refused && answer.Answered.IsZero(),
|
||||
answer: answer,
|
||||
}
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
go func() {
|
||||
result <- lookupResult{
|
||||
lookedUp: true,
|
||||
answer: rq.h.geojs.LookUp(context.WithoutCancel(ctx), client,
|
||||
rq.h.config.ASNAndCountryLookupTimeout),
|
||||
}
|
||||
}()
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
// noteLookup notes result for the log line and as the request's asn= and
|
||||
// country= tags.
|
||||
func (rq *request) noteLookup(result lookupResult) {
|
||||
if !result.lookedUp {
|
||||
return
|
||||
}
|
||||
|
||||
client := rq.h.clientGroup(rq.client)
|
||||
|
||||
switch {
|
||||
case rq.h.config.LookupSource == "file":
|
||||
rq.lookupAnswer = rq.h.lookupFile.LookUp(client)
|
||||
case rq.refusedWhateverTheAnswer(now):
|
||||
rq.lookupAnswer = rq.h.geojs.LookUpWithoutWaiting(ctx, client)
|
||||
rq.skippedLookupWait = rq.lookupAnswer.Answered.IsZero()
|
||||
default:
|
||||
rq.lookupAnswer = rq.h.geojs.LookUp(ctx, client)
|
||||
}
|
||||
|
||||
rq.lookedUp = true
|
||||
rq.lookupAnswer = result.answer
|
||||
rq.skippedLookupWait = result.skippedWait
|
||||
rq.line.ASN = rq.lookupAnswer.ASN
|
||||
rq.line.ASName = rq.lookupAnswer.ASName
|
||||
rq.line.Country = rq.lookupAnswer.Country
|
||||
@@ -80,16 +133,40 @@ func countryTag(country string) string {
|
||||
return config.CountryTagPrefix + country
|
||||
}
|
||||
|
||||
// addLookup adds answer, an answer about a client from the lookup
|
||||
// database or GeoJS, to the client's history, and to the notes of the bans
|
||||
// on its netblock that have no AS number, AS name or country yet.
|
||||
func (h *handler) addLookup(answer lookup.Answer) {
|
||||
h.limiter.AddLookup(answer.Client, answer.Answered,
|
||||
// AddLookup adds answer, an answer about a client from the lookup database
|
||||
// or GeoJS, to the client's history in limiter, and to the notes of the bans
|
||||
// in ledger on its netblock that have no AS number, AS name or country yet.
|
||||
// cfg sets the netblock's size.
|
||||
func AddLookup(
|
||||
cfg *config.WAF, limiter *ratelimit.Limiter, ledger *bans.Ledger,
|
||||
answer geojs.Answer,
|
||||
) {
|
||||
limiter.AddLookup(answer.Client, answer.Answered,
|
||||
answer.ASN, answer.ASName, answer.Country)
|
||||
h.ledger.AddLookup(h.netblock(answer.Client.Addr()),
|
||||
ledger.AddLookup(banNetblock(cfg, answer.Client.Addr()),
|
||||
answer.ASN, answer.ASName, answer.Country)
|
||||
}
|
||||
|
||||
// GeoJSFailed logs a failure of GeoJS to processLog, raises a
|
||||
// source_failure alert for it on alertQueue and counts it in m.
|
||||
// askingAgainIn is how long GeoJS is left alone after it.
|
||||
func GeoJSFailed(
|
||||
processLog *slog.Logger, alertQueue *alerts.Queue, m *metrics.Metrics,
|
||||
err error, askingAgainIn time.Duration,
|
||||
) {
|
||||
processLog.Warn("asking GeoJS failed",
|
||||
"error", err.Error(), "asking_again_in", askingAgainIn.String())
|
||||
alertQueue.Raise(alerts.Alert{
|
||||
Event: alerts.EventSourceFailure,
|
||||
Reason: "asking GeoJS failed",
|
||||
Detail: map[string]any{
|
||||
"source": "geojs", "error": err.Error(),
|
||||
"asking_again_in": askingAgainIn.String(),
|
||||
},
|
||||
})
|
||||
m.GeoJSFailures.Inc()
|
||||
}
|
||||
|
||||
// setLookupHeaders sets the headers that give the client's AS number and
|
||||
// country, leaving out one that is unknown.
|
||||
func setLookupHeaders(header http.Header, asn, country string) {
|
||||
|
||||
@@ -12,7 +12,7 @@ import (
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
@@ -138,7 +138,7 @@ func TestRequestThatWaitedForItsLookupIsCountedWhenItGoesOn(t *testing.T) {
|
||||
// of lookups.json would keep it, while the first request still waits
|
||||
// for GeoJS. The second request goes on at once, and is counted.
|
||||
clk.advance(2500 * time.Millisecond)
|
||||
server.GeoJS.Load([]lookup.Answer{{
|
||||
server.GeoJS.Load([]geojs.Answer{{
|
||||
Client: netip.MustParsePrefix(
|
||||
fromDE + "/32",
|
||||
), ASN: asnDE, ASName: asNameDE,
|
||||
@@ -190,7 +190,7 @@ func TestASNumberAndNameInTheLogLineTheHistoryTheBanNotesAndTheAlert(
|
||||
clk.Now,
|
||||
map[string]string{
|
||||
trustedProxies: trustLocalhost,
|
||||
alertWebhookURL: "https://alerts.example/sww",
|
||||
alertWebhookURL: alertWebhook,
|
||||
rateLimitPerMinute: "1",
|
||||
},
|
||||
)
|
||||
@@ -199,7 +199,7 @@ func TestASNumberAndNameInTheLogLineTheHistoryTheBanNotesAndTheAlert(
|
||||
// The answer is kept before the requests, so GeoJS is not asked, and
|
||||
// gives no answer of its own.
|
||||
netblock := netip.MustParsePrefix(fromDE + "/32")
|
||||
server.GeoJS.Load([]lookup.Answer{{
|
||||
server.GeoJS.Load([]geojs.Answer{{
|
||||
Client: netblock, ASN: asnDE, ASName: asNameDE, Country: "DE",
|
||||
Answered: clk.Now(), Used: clk.Now(),
|
||||
}})
|
||||
@@ -249,7 +249,7 @@ func TestLookupSourceOffLooksNoClientUp(t *testing.T) {
|
||||
)
|
||||
|
||||
// Even an answer kept from before is not used.
|
||||
server.GeoJS.Load([]lookup.Answer{{
|
||||
server.GeoJS.Load([]geojs.Answer{{
|
||||
Client: netip.MustParsePrefix(
|
||||
fromDE + "/32",
|
||||
), ASN: asnDE, ASName: asNameDE,
|
||||
@@ -450,6 +450,8 @@ func TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays(
|
||||
// network, which would keep that clock from moving on: the request is
|
||||
// handed to waf's handler, and GeoJS is one that never answers.
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
defer waitForGeoJSRequestsToBeAbandoned()
|
||||
|
||||
// Not the default. The exclusive list needs the answer, and the app
|
||||
// is never reached.
|
||||
const timeout = 3 * time.Second
|
||||
@@ -495,6 +497,17 @@ func TestMain(m *testing.M) {
|
||||
m.Run()
|
||||
}
|
||||
|
||||
// geojsRequestTimeout is how long a request to GeoJS may take before it is
|
||||
// abandoned.
|
||||
const geojsRequestTimeout = 10 * time.Second
|
||||
|
||||
// waitForGeoJSRequestsToBeAbandoned sleeps until every request to the GeoJS at
|
||||
// silentGeoJSURL made so far has been abandoned. A synctest bubble that asks
|
||||
// it defers this: the bubble cannot end while a request waits on it.
|
||||
func waitForGeoJSRequestsToBeAbandoned() {
|
||||
time.Sleep(geojsRequestTimeout)
|
||||
}
|
||||
|
||||
// silentGeoJS is the GeoJS at silentGeoJSURL.
|
||||
type silentGeoJS struct{}
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package waf_test
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
@@ -10,8 +11,9 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||
)
|
||||
@@ -248,7 +250,7 @@ func TestMetricsByCountryKeepTheBusiestAndCountTheRestAsOther(t *testing.T) {
|
||||
|
||||
// The answers are kept before the requests, so that none waits for
|
||||
// GeoJS.
|
||||
server.GeoJS.Load([]lookup.Answer{
|
||||
server.GeoJS.Load([]geojs.Answer{
|
||||
keptAnswer(fromKP, "KP"), keptAnswer(fromDE, "DE"),
|
||||
keptAnswer(fromFR, "FR"),
|
||||
})
|
||||
@@ -329,13 +331,13 @@ func TestMetricsByASNumberKeepTheBusiestAndCountTheRestAsOther(
|
||||
|
||||
// The answers are kept before the requests, so that GeoJS gives none
|
||||
// of its own. Each client is in an AS of its own.
|
||||
answer := func(addr, asn string) lookup.Answer {
|
||||
return lookup.Answer{
|
||||
answer := func(addr, asn string) geojs.Answer {
|
||||
return geojs.Answer{
|
||||
Client: netip.MustParsePrefix(addr + "/32"), ASN: asn,
|
||||
Answered: clk.Now(), Used: clk.Now(),
|
||||
}
|
||||
}
|
||||
server.GeoJS.Load([]lookup.Answer{
|
||||
server.GeoJS.Load([]geojs.Answer{
|
||||
answer(fromDE, "AS64501"), answer(fromKP, "AS64502"),
|
||||
})
|
||||
|
||||
@@ -357,21 +359,23 @@ func TestMetricsByASNumberKeepTheBusiestAndCountTheRestAsOther(
|
||||
`sww_asn_requests_total{asn="AS64502",instance="app"}`)
|
||||
}
|
||||
|
||||
func TestMetricsCountGeoJSRequestsAndFailures(t *testing.T) {
|
||||
func TestGeoJSFailureIsCountedLoggedAndAlerted(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
geojs := httptest.NewServer(http.HandlerFunc(
|
||||
failingGeoJS := httptest.NewServer(http.HandlerFunc(
|
||||
func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(http.StatusServiceUnavailable)
|
||||
}))
|
||||
t.Cleanup(geojs.Close)
|
||||
t.Cleanup(failingGeoJS.Close)
|
||||
|
||||
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
|
||||
addr, _, server := startProxyWithClock(t, app.URL, geojs.URL, time.Now,
|
||||
addr, out, server, queue := startProxyWithAlerts(t, app.URL,
|
||||
failingGeoJS.URL, time.Now,
|
||||
map[string]string{
|
||||
trustedProxies: trustLocalhost,
|
||||
metricsToken: token,
|
||||
tagDeny: denyKP,
|
||||
trustedProxies: trustLocalhost,
|
||||
metricsToken: token,
|
||||
tagDeny: denyKP,
|
||||
alertWebhookURL: "https://alerts.example/waf",
|
||||
})
|
||||
|
||||
// GeoJS fails, so the client has no country= tag, and SWW_WAF_TAG_DENY does
|
||||
@@ -401,14 +405,46 @@ func TestMetricsCountGeoJSRequestsAndFailures(t *testing.T) {
|
||||
`sww_requests_without_geojs_answer_total{instance="app"}`,
|
||||
1,
|
||||
)
|
||||
|
||||
// The process log's line names the failure but not the address asked
|
||||
// about.
|
||||
logged := ""
|
||||
|
||||
for line := range strings.Lines(out.text()) {
|
||||
if strings.Contains(line, "asking GeoJS failed") {
|
||||
logged = line
|
||||
}
|
||||
}
|
||||
|
||||
if logged == "" || strings.Contains(logged, fromDE) {
|
||||
t.Errorf(
|
||||
"logged %q, want the failure without the address asked about",
|
||||
logged,
|
||||
)
|
||||
}
|
||||
|
||||
// fmt prints a map sorted by its keys.
|
||||
const want = "map[asking_again_in:1s " +
|
||||
"error:GeoJS answered 503 Service Unavailable source:geojs]"
|
||||
|
||||
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
|
||||
if len(waiting) != 1 || waiting[0].Event != alerts.EventSourceFailure ||
|
||||
waiting[0].Reason != "asking GeoJS failed" ||
|
||||
fmt.Sprint(waiting[0].Detail) != want {
|
||||
t.Errorf(
|
||||
"alerts waiting %+v, want one source_failure alert with %v",
|
||||
waiting,
|
||||
want,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// keptAnswer returns GeoJS's answer that the client at addr is in
|
||||
// country, given now.
|
||||
func keptAnswer(addr, country string) lookup.Answer {
|
||||
func keptAnswer(addr, country string) geojs.Answer {
|
||||
now := time.Now()
|
||||
|
||||
return lookup.Answer{
|
||||
return geojs.Answer{
|
||||
Client: netip.MustParsePrefix(addr + "/32"), Country: country,
|
||||
Answered: now, Used: now,
|
||||
}
|
||||
|
||||
@@ -22,12 +22,12 @@ func TestBanActionSaysWhetherTheBanRunsOut(t *testing.T) {
|
||||
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
cause string
|
||||
cause bans.Cause
|
||||
// endsIn is how long after the request the ban ends: negative for a
|
||||
// ban that has ended, and zero for a permanent ban.
|
||||
endsIn time.Duration
|
||||
path string
|
||||
want string
|
||||
want requestlog.Action
|
||||
}{
|
||||
{
|
||||
"temporary ban", bans.CauseAdmin, time.Hour, "/",
|
||||
@@ -170,7 +170,10 @@ func TestObserveModeForwardsWhatEnforceModeRefuses(t *testing.T) {
|
||||
// and its second breaks it.
|
||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
for _, sent := range []struct{ from, refusal string }{
|
||||
for _, sent := range []struct {
|
||||
from string
|
||||
refusal requestlog.Action
|
||||
}{
|
||||
{denied, requestlog.ActionClientDenied},
|
||||
{banned, requestlog.ActionTemporarilyBanned},
|
||||
{fromKP, requestlog.ActionClientDenied},
|
||||
@@ -253,7 +256,7 @@ func TestObserveModeMakesNoBanAndKeepsTheBansItHas(t *testing.T) {
|
||||
|
||||
// wantWouldAction checks the request log line's would_action, and that a
|
||||
// line that should have none has no such field.
|
||||
func wantWouldAction(t *testing.T, line logLine, want string) {
|
||||
func wantWouldAction(t *testing.T, line logLine, want requestlog.Action) {
|
||||
t.Helper()
|
||||
|
||||
got, present := line.fields["would_action"]
|
||||
@@ -261,7 +264,7 @@ func wantWouldAction(t *testing.T, line logLine, want string) {
|
||||
switch {
|
||||
case want == "" && present:
|
||||
t.Errorf("log line has would_action %v, want none", got)
|
||||
case want != "" && got != want:
|
||||
case want != "" && got != string(want):
|
||||
t.Errorf("log line has would_action %v, want %s", got, want)
|
||||
}
|
||||
}
|
||||
+77
-156
@@ -1,13 +1,15 @@
|
||||
// Package waf is the server the waf command runs. It answers
|
||||
// rproxy's questions, takes rproxy's reports and answers its own endpoints, and
|
||||
// writes a log line for each.
|
||||
// writes a log line for each. It also bans the clients on AbuseIPDB's
|
||||
// blacklist each time the reputation package fetches it.
|
||||
package waf
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"cmp"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"time"
|
||||
|
||||
lru "github.com/hashicorp/golang-lru/v2"
|
||||
@@ -17,6 +19,7 @@ import (
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||
@@ -54,18 +57,34 @@ type Params struct {
|
||||
RequestLog io.Writer
|
||||
// ProcessLog receives the process's own messages.
|
||||
ProcessLog *slog.Logger
|
||||
// GeoJSURL is where clients' AS numbers and countries are looked up
|
||||
// while SWW_WAF_LOOKUP_SOURCE is geojs, normally lookup.URL.
|
||||
GeoJSURL string
|
||||
// AbuseIPDBURL is where clients are checked with AbuseIPDB while
|
||||
// SWW_WAF_ABUSEIPDB_KEY is set, normally reputation.AbuseIPDBURL.
|
||||
AbuseIPDBURL string
|
||||
// Metrics count what waf does. Whoever builds them adds the metrics
|
||||
// read from the other parts.
|
||||
Metrics *metrics.Metrics
|
||||
// Limiter is the table of clients the rate limits count, with each
|
||||
// client's history.
|
||||
Limiter *ratelimit.Limiter
|
||||
// Ledger holds the bans.
|
||||
Ledger *bans.Ledger
|
||||
// Anomalies are the counters of the anomaly thresholds.
|
||||
Anomalies *anomaly.Counters
|
||||
// Lists are the lists fetched from URLs, DNSBL the DNSBL zones'
|
||||
// verdicts, and AbuseIPDB AbuseIPDB's scores and the checks spent.
|
||||
Lists *reputation.Lists
|
||||
DNSBL *reputation.DNSBL
|
||||
AbuseIPDB *reputation.AbuseIPDB
|
||||
// RouteWatch asks routewatch about clients, and is nil while
|
||||
// SWW_WAF_ROUTEWATCH_URL is unset.
|
||||
RouteWatch *routewatch.Client
|
||||
// GeoJS looks clients' AS numbers and countries up while
|
||||
// SWW_WAF_LOOKUP_SOURCE is geojs. It is given AddLookup as its Answered
|
||||
// and GeoJSFailed as its Failed.
|
||||
GeoJS *geojs.GeoJS
|
||||
// LookupFile is the lookup database they are looked up in while
|
||||
// SWW_WAF_LOOKUP_SOURCE is file, and nil otherwise.
|
||||
LookupFile *lookup.File
|
||||
// CoreRuleSet is what NewCoreRuleSet returns: what inspects requests
|
||||
// while SWW_WAF_CORE_RULE_SET_MODE is not off, and nil while it is.
|
||||
CoreRuleSet *coreruleset.CoreRuleSet
|
||||
// CoreRuleSet is what inspects requests while
|
||||
// SWW_WAF_CORE_RULE_SET_MODE is not off, and nil while it is.
|
||||
CoreRuleSet CoreRuleSetInspector
|
||||
// Now tells the time by which requests are counted for the rate
|
||||
// limits, bans are made and run out, and GeoJS's answers are kept,
|
||||
// normally time.Now in UTC, the time the state files give.
|
||||
@@ -81,19 +100,29 @@ type Params struct {
|
||||
// failing, a query to a DNSBL zone or a check with AbuseIPDB failing,
|
||||
// or the day's AbuseIPDB checks used up.
|
||||
Alerts *alerts.Queue
|
||||
// BesideLookupTimeLimit is how long the heuristics of besideLookup have
|
||||
// for a request. Zero means besideLookupTimeLimit. Tests that run the
|
||||
// Core Rule Set lengthen it, since the race detector and a busy host can
|
||||
// slow an inspection past besideLookupTimeLimit.
|
||||
BesideLookupTimeLimit time.Duration
|
||||
}
|
||||
|
||||
// Server is the server waf runs, with the parts whose state the state
|
||||
// files keep, the lookup database, nil unless SWW_WAF_LOOKUP_SOURCE is file, the
|
||||
// lists fetched from URLs, which its Run fetches, the DNSBL zones' verdicts,
|
||||
// AbuseIPDB's scores and checks spent, routewatch's answers, and the metrics.
|
||||
// RouteWatch is nil while SWW_WAF_ROUTEWATCH_URL is unset.
|
||||
// CoreRuleSetInspector inspects a request with the Core Rule Set, as
|
||||
// coreruleset.CoreRuleSet's Inspect says. Tests give waf a stand-in for it.
|
||||
type CoreRuleSetInspector interface {
|
||||
Inspect(
|
||||
r *http.Request, client netip.Addr, body io.Reader,
|
||||
) (coreruleset.Result, []byte, error)
|
||||
}
|
||||
|
||||
// Server is the server waf runs, with the parts of its Params. The tests
|
||||
// read them.
|
||||
type Server struct {
|
||||
*http.Server
|
||||
|
||||
Ledger *bans.Ledger
|
||||
Limiter *ratelimit.Limiter
|
||||
GeoJS *lookup.GeoJS
|
||||
GeoJS *geojs.GeoJS
|
||||
Anomalies *anomaly.Counters
|
||||
LookupFile *lookup.File
|
||||
Lists *reputation.Lists
|
||||
@@ -103,159 +132,45 @@ type Server struct {
|
||||
Metrics *metrics.Metrics
|
||||
}
|
||||
|
||||
// newHandler returns the handler with what every request shares, as the
|
||||
// settings in params make it.
|
||||
// newHandler returns the handler with what every request shares, the parts
|
||||
// in params among it.
|
||||
func newHandler(params Params) *handler {
|
||||
m := metrics.New(
|
||||
params.Config.MetricsBusiestCountriesAndASNs,
|
||||
params.Config.InstanceName,
|
||||
)
|
||||
lists, dnsbl, abuseIPDB := newReputation(params, m)
|
||||
|
||||
awaitingReport, err := lru.New[string, *decision](maxAwaitingReport)
|
||||
if err != nil {
|
||||
panic(err) // New fails only for a size below one
|
||||
}
|
||||
|
||||
h := &handler{
|
||||
return &handler{
|
||||
config: params.Config,
|
||||
requestLog: params.RequestLog,
|
||||
processLog: params.ProcessLog,
|
||||
now: params.Now,
|
||||
metrics: m,
|
||||
metrics: params.Metrics,
|
||||
awaitingReport: awaitingReport,
|
||||
limiter: ratelimit.New(ratelimit.Limits{
|
||||
PerMinute: params.Config.RateLimitPerMinute,
|
||||
PerHour: params.Config.RateLimitPerHour,
|
||||
PerDay: params.Config.RateLimitPerDay,
|
||||
BytesPerMinute: params.Config.BytesLimitPerMinute,
|
||||
BytesPerHour: params.Config.BytesLimitPerHour,
|
||||
BytesPerDay: params.Config.BytesLimitPerDay,
|
||||
}, params.Config.MaxTrackedClients),
|
||||
ledger: bans.New(bans.Params{
|
||||
LimitBanDuration: params.Config.LimitBanDuration,
|
||||
LimitBanRepeatWindow: params.Config.LimitBanRepeatWindow,
|
||||
LimitBanPermanentAbove: params.Config.LimitBanPermanentAbove,
|
||||
AttackBanDuration: params.Config.AttackBanDuration,
|
||||
MaxBans: params.Config.MaxBans,
|
||||
}),
|
||||
anomalies: anomaly.New(anomaly.Params{
|
||||
Client: params.Config.AnomalyClient,
|
||||
Net: params.Config.AnomalyNet,
|
||||
ASN: params.Config.AnomalyASN,
|
||||
Total: params.Config.AnomalyTotal,
|
||||
WatchedNet: params.Config.AnomalyWatchedNet,
|
||||
NetV4Prefix: params.Config.AnomalyNetV4Prefix,
|
||||
NetV6Prefix: params.Config.AnomalyNetV6Prefix,
|
||||
NamedNetblocks: params.Config.AnomalyWatchedNets,
|
||||
Alerts: params.Alerts,
|
||||
}),
|
||||
lookupFile: params.LookupFile,
|
||||
lists: lists,
|
||||
dnsbl: dnsbl,
|
||||
abuseIPDB: abuseIPDB,
|
||||
rules: params.Rules,
|
||||
coreRuleSet: params.CoreRuleSet,
|
||||
alerts: params.Alerts,
|
||||
}
|
||||
if params.Config.RouteWatchURL != "" {
|
||||
h.routeWatch = routewatch.New(routewatch.Params{
|
||||
URL: params.Config.RouteWatchURL, Now: params.Now, ProcessLog: params.ProcessLog,
|
||||
})
|
||||
}
|
||||
|
||||
h.geojs = lookup.New(lookup.Params{
|
||||
URL: params.GeoJSURL,
|
||||
Timeout: params.Config.ASNAndCountryLookupTimeout,
|
||||
limiter: params.Limiter,
|
||||
ledger: params.Ledger,
|
||||
geojs: params.GeoJS,
|
||||
anomalies: params.Anomalies,
|
||||
lookupFile: params.LookupFile,
|
||||
lists: params.Lists,
|
||||
dnsbl: params.DNSBL,
|
||||
abuseIPDB: params.AbuseIPDB,
|
||||
routeWatch: params.RouteWatch,
|
||||
rules: params.Rules,
|
||||
coreRuleSet: params.CoreRuleSet,
|
||||
alerts: params.Alerts,
|
||||
// The country list, the headers, the tag settings that name an asn=
|
||||
// or country= tag and the biased thresholds act on the answer before
|
||||
// the request goes on.
|
||||
Wait: len(params.Config.ExclusivelyAllowedCountries) > 0 ||
|
||||
params.Config.AddLookupHeaders || params.Config.TagsNeedLookup() ||
|
||||
waitForGeoJS: len(params.Config.ExclusivelyAllowedCountries) > 0 ||
|
||||
params.Config.AddLookupHeaders ||
|
||||
params.Config.TagsNeedLookup() ||
|
||||
biasedThresholdsSet(params.Config),
|
||||
Answered: h.addLookup,
|
||||
Now: params.Now,
|
||||
ProcessLog: params.ProcessLog,
|
||||
Metrics: m,
|
||||
Alerts: params.Alerts,
|
||||
})
|
||||
m.AddBansAndClients(h.ledger, h.limiter, params.Now)
|
||||
m.AddRules(params.Rules)
|
||||
|
||||
return h
|
||||
}
|
||||
|
||||
// newReputation returns the lists fetched from URLs, the DNSBL zones'
|
||||
// verdicts and AbuseIPDB's scores, as the settings in params name them,
|
||||
// with none fetched, asked for or checked yet, and adds their metrics to
|
||||
// m, AbuseIPDB's while SWW_WAF_ABUSEIPDB_KEY is set.
|
||||
func newReputation(
|
||||
params Params, m *metrics.Metrics,
|
||||
) (*reputation.Lists, *reputation.DNSBL, *reputation.AbuseIPDB) {
|
||||
cfg := params.Config
|
||||
lists := reputation.New(reputation.Params{
|
||||
BlocklistURLs: cfg.BlocklistURLs,
|
||||
Refresh: cfg.BlocklistRefresh,
|
||||
ASNLimitPercentURL: cfg.ASNLimitPercentURL,
|
||||
TagLists: cfg.TagLists,
|
||||
Now: params.Now,
|
||||
ProcessLog: params.ProcessLog,
|
||||
Alerts: params.Alerts,
|
||||
})
|
||||
dnsbl := reputation.NewDNSBL(reputation.DNSBLParams{
|
||||
Zones: cfg.DNSBLZones, Resolver: cfg.DNSBLResolver,
|
||||
CacheTTL: cfg.IPReputationCacheTTL,
|
||||
Timeout: cfg.DNSBLAbuseIPDBTimeout, Now: params.Now, ProcessLog: params.ProcessLog,
|
||||
Alerts: params.Alerts,
|
||||
})
|
||||
abuseIPDB := reputation.NewAbuseIPDB(reputation.AbuseIPDBParams{
|
||||
URL: params.AbuseIPDBURL, Key: cfg.AbuseIPDBKey, MinScore: cfg.AbuseIPDBMinScore,
|
||||
DailyBudget: cfg.AbuseIPDBDailyBudget, CacheTTL: cfg.IPReputationCacheTTL,
|
||||
Timeout: cfg.DNSBLAbuseIPDBTimeout, Now: params.Now, ProcessLog: params.ProcessLog,
|
||||
Alerts: params.Alerts,
|
||||
})
|
||||
|
||||
m.AddReputation(lists, dnsbl)
|
||||
|
||||
if cfg.AbuseIPDBKey != "" {
|
||||
m.AddAbuseIPDB(abuseIPDB)
|
||||
besideLookupLimit: cmp.Or(
|
||||
params.BesideLookupTimeLimit,
|
||||
besideLookupTimeLimit,
|
||||
),
|
||||
}
|
||||
|
||||
return lists, dnsbl, abuseIPDB
|
||||
}
|
||||
|
||||
// NewCoreRuleSet returns the Core Rule Set at SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL,
|
||||
// without the rules SWW_WAF_CORE_RULE_SET_DISABLED_RULES switches off, reading bodies
|
||||
// up to SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE, with the exclusions in
|
||||
// SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE and the fixes of the apps in SWW_WAF_APPS,
|
||||
// or nil while SWW_WAF_CORE_RULE_SET_MODE is off. An exclusions file that
|
||||
// coreruleset.New refuses is an error in every mode. While the mode is off, a
|
||||
// set file is loaded with the Core Rule Set only to check it.
|
||||
func NewCoreRuleSet(cfg *config.WAF) (*coreruleset.CoreRuleSet, error) {
|
||||
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff &&
|
||||
cfg.CoreRuleSetExclusionsFile == "" {
|
||||
return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error
|
||||
}
|
||||
|
||||
coreRuleSet, err := coreruleset.New(coreruleset.Params{
|
||||
ParanoiaLevel: cfg.CoreRuleSetParanoiaLevel,
|
||||
DisabledRules: cfg.CoreRuleSetDisabledRules,
|
||||
BodyInspectionSize: cfg.CoreRuleSetBodyInspectionSize,
|
||||
ExclusionsFile: cfg.CoreRuleSetExclusionsFile,
|
||||
Apps: cfg.Apps,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf(
|
||||
"SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE: %w",
|
||||
err,
|
||||
)
|
||||
}
|
||||
|
||||
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff {
|
||||
return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error
|
||||
}
|
||||
|
||||
return coreRuleSet, nil
|
||||
}
|
||||
|
||||
// handler is waf's handler. It holds what every request shares; what
|
||||
@@ -268,7 +183,7 @@ type handler struct {
|
||||
metrics *metrics.Metrics
|
||||
limiter *ratelimit.Limiter
|
||||
ledger *bans.Ledger
|
||||
geojs *lookup.GeoJS
|
||||
geojs *geojs.GeoJS
|
||||
anomalies *anomaly.Counters
|
||||
lookupFile *lookup.File
|
||||
lists *reputation.Lists
|
||||
@@ -276,8 +191,14 @@ type handler struct {
|
||||
abuseIPDB *reputation.AbuseIPDB
|
||||
routeWatch *routewatch.Client
|
||||
rules *rules.Files
|
||||
coreRuleSet *coreruleset.CoreRuleSet
|
||||
coreRuleSet CoreRuleSetInspector
|
||||
alerts *alerts.Queue
|
||||
// waitForGeoJS is true when a setting needs each request's answer from
|
||||
// GeoJS before the request goes on.
|
||||
waitForGeoJS bool
|
||||
// besideLookupLimit is how long the heuristics of besideLookup have for a
|
||||
// request, as Params.BesideLookupTimeLimit says.
|
||||
besideLookupLimit time.Duration
|
||||
// awaitingReport holds the decision on each answered question by its id
|
||||
// until its report comes, and nil for a question still being handled.
|
||||
awaitingReport *lru.Cache[string, *decision]
|
||||
|
||||
+180
-18
@@ -16,10 +16,18 @@ import (
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
|
||||
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
"sneak.berlin/go/smallwebwaf/internal/routewatch"
|
||||
"sneak.berlin/go/smallwebwaf/internal/rproxy"
|
||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||
@@ -424,7 +432,8 @@ func startRProxy(
|
||||
// With no geojsURL, there is no stand-in for GeoJS to look clients up at, and
|
||||
// SWW_WAF_LOOKUP_SOURCE is off unless env sets it. While it is file, the lookup
|
||||
// database SWW_WAF_LOOKUP_DB_PATH names is read. Clients are checked with AbuseIPDB
|
||||
// at abuseIPDBURL while env sets SWW_WAF_ABUSEIPDB_KEY.
|
||||
// at abuseIPDBURL while env sets SWW_WAF_ABUSEIPDB_KEY, and its blacklist is
|
||||
// fetched from abuseIPDBBlacklistURL.
|
||||
// SWW_WAF_CORE_RULE_SET_MODE is off unless env sets it, so that only the tests of the
|
||||
// Core Rule Set have their requests inspected by it.
|
||||
func newWAF(
|
||||
@@ -435,13 +444,15 @@ func newWAF(
|
||||
) (*waf.Server, *output, *alerts.Queue) {
|
||||
t.Helper()
|
||||
|
||||
return newWAFWith(t, waf.New, geojsURL, now, env)
|
||||
return newWAFWith(t, waf.New, geojsURL, abuseIPDBURL, now, env)
|
||||
}
|
||||
|
||||
// newWAFWith is newWAF for the server build makes.
|
||||
// newWAFWith is newWAF for the server build makes, with clients checked with
|
||||
// AbuseIPDB at abuseIPDBCheckURL.
|
||||
func newWAFWith(
|
||||
t *testing.T, build func(waf.Params) *waf.Server,
|
||||
geojsURL string, now func() time.Time, env map[string]string,
|
||||
geojsURL, abuseIPDBCheckURL string, now func() time.Time,
|
||||
env map[string]string,
|
||||
) (*waf.Server, *output, *alerts.Queue) {
|
||||
t.Helper()
|
||||
|
||||
@@ -494,25 +505,174 @@ func newWAFWith(
|
||||
}
|
||||
}
|
||||
|
||||
coreRuleSet, err := waf.NewCoreRuleSet(cfg)
|
||||
params := waf.Params{
|
||||
Config: cfg,
|
||||
RequestLog: out,
|
||||
ProcessLog: processLog,
|
||||
LookupFile: lookupFile,
|
||||
Now: now,
|
||||
Rules: ruleFiles,
|
||||
Alerts: alertQueue,
|
||||
// The tests of the limit set it back, and hold the Core Rule Set
|
||||
// with a stand-in.
|
||||
BesideLookupTimeLimit: time.Hour,
|
||||
}
|
||||
|
||||
addCoreRuleSet(t, ¶ms)
|
||||
addCounters(¶ms)
|
||||
addOutsideSources(¶ms, geojsURL, abuseIPDBCheckURL)
|
||||
|
||||
server := build(params)
|
||||
|
||||
return server, out, alertQueue
|
||||
}
|
||||
|
||||
// addCoreRuleSet adds to params the Core Rule Set, with the settings for it,
|
||||
// while SWW_WAF_CORE_RULE_SET_MODE is not off.
|
||||
func addCoreRuleSet(t *testing.T, params *waf.Params) {
|
||||
t.Helper()
|
||||
|
||||
cfg := params.Config
|
||||
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff {
|
||||
return
|
||||
}
|
||||
|
||||
coreRuleSet, err := coreruleset.New(coreruleset.Params{
|
||||
ParanoiaLevel: cfg.CoreRuleSetParanoiaLevel,
|
||||
DisabledRules: cfg.CoreRuleSetDisabledRules,
|
||||
BodyInspectionSize: cfg.CoreRuleSetBodyInspectionSize,
|
||||
ExclusionsFile: cfg.CoreRuleSetExclusionsFile,
|
||||
Apps: cfg.Apps,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Core Rule Set: %v", err)
|
||||
}
|
||||
|
||||
server := build(waf.Params{
|
||||
Config: cfg,
|
||||
RequestLog: out,
|
||||
ProcessLog: processLog,
|
||||
GeoJSURL: geojsURL,
|
||||
AbuseIPDBURL: abuseIPDBURL,
|
||||
LookupFile: lookupFile,
|
||||
CoreRuleSet: coreRuleSet,
|
||||
Now: now,
|
||||
Rules: ruleFiles,
|
||||
Alerts: alertQueue,
|
||||
params.CoreRuleSet = coreRuleSet
|
||||
}
|
||||
|
||||
// addCounters adds to params the metrics, the table of clients, the ledger
|
||||
// and the anomaly counters, as sww builds them from params' settings. The
|
||||
// metrics read the ledger, the table of clients and the rules.
|
||||
func addCounters(params *waf.Params) {
|
||||
cfg := params.Config
|
||||
|
||||
params.Metrics = metrics.New(
|
||||
cfg.MetricsBusiestCountriesAndASNs,
|
||||
cfg.InstanceName,
|
||||
)
|
||||
params.Limiter = ratelimit.New(ratelimit.Limits{
|
||||
PerMinute: cfg.RateLimitPerMinute,
|
||||
PerHour: cfg.RateLimitPerHour,
|
||||
PerDay: cfg.RateLimitPerDay,
|
||||
BytesPerMinute: cfg.BytesLimitPerMinute,
|
||||
BytesPerHour: cfg.BytesLimitPerHour,
|
||||
BytesPerDay: cfg.BytesLimitPerDay,
|
||||
}, cfg.MaxTrackedClients)
|
||||
params.Ledger = bans.New(bans.Params{
|
||||
LimitBanDuration: cfg.LimitBanDuration,
|
||||
LimitBanRepeatWindow: cfg.LimitBanRepeatWindow,
|
||||
LimitBanPermanentAbove: cfg.LimitBanPermanentAbove,
|
||||
AttackBanDuration: cfg.AttackBanDuration,
|
||||
MaxBans: cfg.MaxBans,
|
||||
})
|
||||
params.Anomalies = anomaly.New(anomaly.Params{
|
||||
Client: cfg.AnomalyClient,
|
||||
Net: cfg.AnomalyNet,
|
||||
ASN: cfg.AnomalyASN,
|
||||
Total: cfg.AnomalyTotal,
|
||||
WatchedNet: cfg.AnomalyWatchedNet,
|
||||
NetV4Prefix: cfg.AnomalyNetV4Prefix,
|
||||
NetV6Prefix: cfg.AnomalyNetV6Prefix,
|
||||
NamedNetblocks: cfg.AnomalyWatchedNets,
|
||||
Alerts: params.Alerts,
|
||||
})
|
||||
|
||||
return server, out, alertQueue
|
||||
params.Metrics.AddBansAndClients(
|
||||
params.Ledger,
|
||||
params.Limiter,
|
||||
params.Now,
|
||||
)
|
||||
params.Metrics.AddRules(params.Rules)
|
||||
}
|
||||
|
||||
// addOutsideSources adds to params the lists fetched from URLs, the DNSBL
|
||||
// zones' verdicts, AbuseIPDB's scores from abuseIPDBCheckURL and its
|
||||
// blacklist from abuseIPDBBlacklistURL, routewatch's answers while its
|
||||
// setting is set, and GeoJS at geojsURL, as sww builds
|
||||
// them from params' settings and the parts addCounters added. The metrics
|
||||
// read the lists, the zones' verdicts, and AbuseIPDB's while its key is set.
|
||||
func addOutsideSources(
|
||||
params *waf.Params, geojsURL, abuseIPDBCheckURL string,
|
||||
) {
|
||||
cfg := params.Config
|
||||
m, limiter, ledger := params.Metrics, params.Limiter, params.Ledger
|
||||
processLog, alertQueue := params.ProcessLog, params.Alerts
|
||||
|
||||
params.Lists = reputation.New(reputation.Params{
|
||||
BlocklistURLs: cfg.BlocklistURLs,
|
||||
Refresh: cfg.BlocklistRefresh,
|
||||
ASNLimitPercentURL: cfg.ASNLimitPercentURL,
|
||||
TagLists: cfg.TagLists,
|
||||
Now: params.Now,
|
||||
ProcessLog: processLog,
|
||||
Alerts: alertQueue,
|
||||
})
|
||||
params.DNSBL = reputation.NewDNSBL(reputation.DNSBLParams{
|
||||
Zones: cfg.DNSBLZones,
|
||||
Resolver: cfg.DNSBLResolver,
|
||||
CacheTTL: cfg.IPReputationCacheTTL,
|
||||
Timeout: cfg.DNSBLAbuseIPDBTimeout,
|
||||
Now: params.Now,
|
||||
ProcessLog: processLog,
|
||||
Alerts: alertQueue,
|
||||
})
|
||||
params.AbuseIPDB = reputation.NewAbuseIPDB(reputation.AbuseIPDBParams{
|
||||
URL: abuseIPDBCheckURL,
|
||||
Key: cfg.AbuseIPDBKey,
|
||||
MinScore: cfg.AbuseIPDBMinScore,
|
||||
DailyBudget: cfg.AbuseIPDBDailyBudget,
|
||||
CacheTTL: cfg.IPReputationCacheTTL,
|
||||
Timeout: cfg.DNSBLAbuseIPDBTimeout,
|
||||
BlacklistURL: abuseIPDBBlacklistURL,
|
||||
Threshold: cfg.AbuseIPDBThreshold,
|
||||
BlacklistRefresh: cfg.AbuseIPDBBlacklistRefresh,
|
||||
Blacklisted: func(listed []reputation.BlacklistEntry) {
|
||||
waf.BanAbuseIPDBBlacklist(
|
||||
cfg, ledger, processLog, params.Now(), listed,
|
||||
)
|
||||
},
|
||||
Now: params.Now,
|
||||
ProcessLog: processLog,
|
||||
Alerts: alertQueue,
|
||||
})
|
||||
|
||||
if cfg.RouteWatchURL != "" {
|
||||
params.RouteWatch = routewatch.New(routewatch.Params{
|
||||
URL: cfg.RouteWatchURL,
|
||||
Now: params.Now,
|
||||
ProcessLog: processLog,
|
||||
})
|
||||
}
|
||||
|
||||
params.GeoJS = geojs.New(geojs.Params{
|
||||
URL: geojsURL,
|
||||
Answered: func(answer geojs.Answer) {
|
||||
waf.AddLookup(cfg, limiter, ledger, answer)
|
||||
},
|
||||
Now: params.Now,
|
||||
Asked: m.GeoJSRequests.Inc,
|
||||
Failed: func(err error, askingAgainIn time.Duration) {
|
||||
waf.GeoJSFailed(processLog, alertQueue, m, err, askingAgainIn)
|
||||
},
|
||||
WentWithoutAnswer: m.RequestsWithoutGeoJSAnswer.Inc,
|
||||
})
|
||||
|
||||
m.AddReputation(params.Lists, params.DNSBL)
|
||||
|
||||
if cfg.AbuseIPDBKey != "" {
|
||||
m.AddAbuseIPDB(params.AbuseIPDB)
|
||||
}
|
||||
}
|
||||
|
||||
// newClient returns an HTTP client that sends requests as they are made,
|
||||
@@ -634,7 +794,9 @@ func readResponse(t *testing.T, conn net.Conn) answer {
|
||||
}
|
||||
|
||||
// wantLine checks the request log line's status and action.
|
||||
func wantLine(t *testing.T, line logLine, status int, action string) {
|
||||
func wantLine(
|
||||
t *testing.T, line logLine, status int, action requestlog.Action,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
if line.Status != status || line.Action != action {
|
||||
|
||||
@@ -38,8 +38,11 @@ func (rateLimits) check(
|
||||
rq.line.BytesLimitPercent, rq.line.BytesLimitPercentSetting =
|
||||
rq.bytesLimitPercent.logged()
|
||||
|
||||
counts, hit, over := rq.h.limiter.Count(rq.h.clientGroup(rq.client), now,
|
||||
rq.rateLimitPercent.percent)
|
||||
counts, hit, over := rq.h.limiter.Count(
|
||||
clientGroup(rq.h.config, rq.client),
|
||||
now,
|
||||
rq.rateLimitPercent.percent,
|
||||
)
|
||||
rq.line.Counts = counts
|
||||
|
||||
if !over {
|
||||
@@ -59,7 +62,7 @@ func (rateLimits) check(
|
||||
// limit. It counts nothing.
|
||||
func (rq *request) overAWholeRateLimit(now time.Time) bool {
|
||||
return rq.countedByRateLimits() &&
|
||||
rq.h.limiter.OverARateLimit(rq.h.clientGroup(rq.client), now)
|
||||
rq.h.limiter.OverARateLimit(clientGroup(rq.h.config, rq.client), now)
|
||||
}
|
||||
|
||||
// countedByRateLimits reports whether the rate limits count the request: its
|
||||
|
||||
@@ -5,7 +5,7 @@ import (
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
@@ -121,7 +121,7 @@ func TestIPv6GroupPrefixSetsTheClientTheLimitsCount(t *testing.T) {
|
||||
// status and action are those of the request that breaks the
|
||||
// limit: a rate limit refuses it, a byte limit passes it on.
|
||||
status int
|
||||
action string
|
||||
action requestlog.Action
|
||||
}{
|
||||
{rateLimitPerMinute, "1", http.StatusForbidden, requestlog.ActionClientIsRateLimited},
|
||||
{bytesLimitPerMinute, byteLimit, http.StatusOK, requestlog.ActionForward},
|
||||
@@ -162,7 +162,7 @@ func TestRateLimitExemptPathsAreNeitherCountedNorRefused(t *testing.T) {
|
||||
|
||||
// The answers are kept before the requests, so that none waits for
|
||||
// GeoJS.
|
||||
server.GeoJS.Load([]lookup.Answer{
|
||||
server.GeoJS.Load([]geojs.Answer{
|
||||
keptAnswer(client, "DE"), keptAnswer(fromKP, "KP"),
|
||||
})
|
||||
|
||||
|
||||
@@ -36,8 +36,11 @@ func (refusalLimit) check(
|
||||
return nil
|
||||
}
|
||||
|
||||
hit, over := rq.h.limiter.CountRefusal(rq.h.clientGroup(rq.client), now,
|
||||
cfg.RefusalsLimitPerMinute)
|
||||
hit, over := rq.h.limiter.CountRefusal(
|
||||
clientGroup(rq.h.config, rq.client),
|
||||
now,
|
||||
cfg.RefusalsLimitPerMinute,
|
||||
)
|
||||
if !over {
|
||||
return nil
|
||||
}
|
||||
@@ -52,6 +55,14 @@ func (refusalLimit) check(
|
||||
case requestlog.ActionTemporarilyBanned,
|
||||
requestlog.ActionPermanentlyBanned:
|
||||
status = cfg.BanResponse
|
||||
case requestlog.ActionForward,
|
||||
requestlog.ActionRequestLineTooLong,
|
||||
requestlog.ActionRequestHeaderTooLarge,
|
||||
requestlog.ActionClientIsRateLimited,
|
||||
requestlog.ActionClientDenied,
|
||||
requestlog.ActionCountryDenied,
|
||||
requestlog.ActionChecksTimedOut,
|
||||
requestlog.ActionOwnEndpoint:
|
||||
}
|
||||
|
||||
rq.banForLimit(now, hit, status)
|
||||
|
||||
@@ -74,7 +74,7 @@ func TestRefusalLimitBreaksAtOneOverTheLimit(t *testing.T) {
|
||||
refusals []refused
|
||||
// nextAction is the action of the client's next request: the next
|
||||
// request after a ban for a ban rule makes it permanent.
|
||||
nextAction string
|
||||
nextAction requestlog.Action
|
||||
}{
|
||||
{
|
||||
"block rule",
|
||||
|
||||
@@ -7,10 +7,6 @@ import (
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
)
|
||||
|
||||
// deny is the SWW_WAF_BLOCKLIST_ACTION and the SWW_WAF_DNSBL_ABUSEIPDB_ACTION that
|
||||
// refuses the requests of a client a source lists.
|
||||
const deny = "deny"
|
||||
|
||||
// noteListed notes each of sources, the URLs of the blocklists or the
|
||||
// DNSBL zones, their keys masked, that list the client, as noteHit does,
|
||||
// with reason.
|
||||
@@ -37,7 +33,7 @@ func (rq *request) noteHit(hit bans.ReputationHit, reason string) {
|
||||
Event: alerts.EventReputationHit,
|
||||
Host: rq.site,
|
||||
Client: rq.client,
|
||||
Netblock: rq.h.clientGroup(rq.client),
|
||||
Netblock: clientGroup(rq.h.config, rq.client),
|
||||
ASN: rq.line.ASN,
|
||||
ASName: rq.line.ASName,
|
||||
Country: rq.line.Country,
|
||||
|
||||
@@ -58,22 +58,22 @@ func TestEachBlocklistActionForAListedAddressAndAListedNetblock(
|
||||
// statuses and actions are those of a listed client's three
|
||||
// requests, and percent their rate_limit_percent, as percentText gives it.
|
||||
statuses []int
|
||||
actions []string
|
||||
actions []requestlog.Action
|
||||
percent string
|
||||
}{
|
||||
{
|
||||
actionDeny, []int{http.StatusForbidden, http.StatusForbidden, http.StatusForbidden},
|
||||
[]string{denied, denied, denied}, none,
|
||||
[]requestlog.Action{denied, denied, denied}, none,
|
||||
},
|
||||
{
|
||||
// Half of 4 requests a minute: the third breaks the limit.
|
||||
limitHalf, []int{http.StatusOK, http.StatusOK, http.StatusForbidden},
|
||||
[]string{forward, forward, requestlog.ActionClientIsRateLimited},
|
||||
[]requestlog.Action{forward, forward, requestlog.ActionClientIsRateLimited},
|
||||
"50 from " + blocklistAction,
|
||||
},
|
||||
{
|
||||
actionLog, []int{http.StatusOK, http.StatusOK, http.StatusOK},
|
||||
[]string{forward, forward, forward}, none,
|
||||
[]requestlog.Action{forward, forward, forward}, none,
|
||||
},
|
||||
} {
|
||||
t.Run(tc.action, func(t *testing.T) {
|
||||
@@ -442,22 +442,22 @@ func TestEachDNSBLAbuseIPDBActionForAClientADNSBLZoneLists(t *testing.T) {
|
||||
// statuses and actions are those of a listed client's three
|
||||
// requests, and percent their rate_limit_percent, as percentText gives it.
|
||||
statuses []int
|
||||
actions []string
|
||||
actions []requestlog.Action
|
||||
percent string
|
||||
}{
|
||||
{
|
||||
actionDeny, []int{http.StatusForbidden, http.StatusForbidden, http.StatusForbidden},
|
||||
[]string{denied, denied, denied}, none,
|
||||
[]requestlog.Action{denied, denied, denied}, none,
|
||||
},
|
||||
{
|
||||
// Half of 4 requests a minute: the third breaks the limit.
|
||||
limitHalf, []int{http.StatusOK, http.StatusOK, http.StatusForbidden},
|
||||
[]string{forward, forward, requestlog.ActionClientIsRateLimited},
|
||||
[]requestlog.Action{forward, forward, requestlog.ActionClientIsRateLimited},
|
||||
"50 from " + dnsblAbuseIPDBAction,
|
||||
},
|
||||
{
|
||||
actionLog, []int{http.StatusOK, http.StatusOK, http.StatusOK},
|
||||
[]string{forward, forward, forward}, none,
|
||||
[]requestlog.Action{forward, forward, forward}, none,
|
||||
},
|
||||
} {
|
||||
t.Run(tc.action, func(t *testing.T) {
|
||||
@@ -822,22 +822,39 @@ const abuseipdb = reputation.AbuseIPDBSource
|
||||
// through which AbuseIPDB is asked.
|
||||
const abuseIPDBURL = "abuseipdb://stand-in/api/v2/check"
|
||||
|
||||
// abuseIPDBBlacklistURL is where newWAF has AbuseIPDB's blacklist fetched
|
||||
// from: at abuseIPDBStandIn too.
|
||||
const abuseIPDBBlacklistURL = "abuseipdb://stand-in/api/v2/blacklist"
|
||||
|
||||
// abuseIPDBStandIn is a stand-in for AbuseIPDB that gives every client the
|
||||
// score 100, at once and without the network.
|
||||
// score 100, at once and without the network. Its blacklist lists client
|
||||
// alone, with the score 100, to a fetch with accountKey that asks for the
|
||||
// clients scored 90 or more, the default threshold; it refuses any other
|
||||
// fetch of it with 400.
|
||||
type abuseIPDBStandIn struct{}
|
||||
|
||||
// RoundTrip answers req with the score 100.
|
||||
// RoundTrip answers req with the score 100, or with the blacklist.
|
||||
func (abuseIPDBStandIn) RoundTrip(
|
||||
req *http.Request,
|
||||
) (*http.Response, error) {
|
||||
status, body := http.StatusOK, `{"data":{"abuseConfidenceScore":100}}`
|
||||
|
||||
if req.URL.Path == "/api/v2/blacklist" {
|
||||
body = `{"data":[{"ipAddress":"` + client +
|
||||
`","abuseConfidenceScore":100}]}`
|
||||
|
||||
if req.Header.Get("Key") != accountKey ||
|
||||
req.URL.RawQuery != "confidenceMinimum=90" {
|
||||
status, body = http.StatusBadRequest, `{"errors":[]}`
|
||||
}
|
||||
}
|
||||
|
||||
return &http.Response{
|
||||
StatusCode: http.StatusOK,
|
||||
Status: "200 OK",
|
||||
StatusCode: status,
|
||||
Status: fmt.Sprintf("%d %s", status, http.StatusText(status)),
|
||||
Header: http.Header{},
|
||||
Body: io.NopCloser(
|
||||
strings.NewReader(`{"data":{"abuseConfidenceScore":100}}`),
|
||||
),
|
||||
Request: req,
|
||||
Body: io.NopCloser(strings.NewReader(body)),
|
||||
Request: req,
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -937,7 +954,7 @@ func TestClientRefusedForAnOffenceIsCheckedWithAbuseIPDBAtItsNextRequest(
|
||||
// history then counts.
|
||||
path string
|
||||
status int
|
||||
action string
|
||||
action requestlog.Action
|
||||
want ratelimit.Offences
|
||||
}{
|
||||
{
|
||||
@@ -1001,22 +1018,22 @@ func TestEachDNSBLAbuseIPDBActionForAClientAbuseIPDBScoresAtOrOverTheMinimum(
|
||||
// statuses and actions are those of fromDE's three requests, and
|
||||
// percent their rate_limit_percent, as percentText gives it.
|
||||
statuses []int
|
||||
actions []string
|
||||
actions []requestlog.Action
|
||||
percent string
|
||||
}{
|
||||
{
|
||||
actionDeny, []int{http.StatusForbidden, http.StatusForbidden, http.StatusForbidden},
|
||||
[]string{denied, denied, denied}, none,
|
||||
[]requestlog.Action{denied, denied, denied}, none,
|
||||
},
|
||||
{
|
||||
// Half of 4 requests a minute: the third breaks the limit.
|
||||
limitHalf, []int{http.StatusOK, http.StatusOK, http.StatusForbidden},
|
||||
[]string{forward, forward, requestlog.ActionClientIsRateLimited},
|
||||
[]requestlog.Action{forward, forward, requestlog.ActionClientIsRateLimited},
|
||||
"50 from " + dnsblAbuseIPDBAction,
|
||||
},
|
||||
{
|
||||
actionLog, []int{http.StatusOK, http.StatusOK, http.StatusOK},
|
||||
[]string{forward, forward, forward}, none,
|
||||
[]requestlog.Action{forward, forward, forward}, none,
|
||||
},
|
||||
} {
|
||||
t.Run(tc.action, func(t *testing.T) {
|
||||
|
||||
+64
-20
@@ -11,7 +11,7 @@ import (
|
||||
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||
@@ -21,7 +21,7 @@ import (
|
||||
// to close its connection without an answer, and the action the log line names.
|
||||
type refusal struct {
|
||||
status int
|
||||
action string
|
||||
action requestlog.Action
|
||||
}
|
||||
|
||||
// decision is what the checks note of a request: its log line, and what
|
||||
@@ -42,7 +42,7 @@ type decision struct {
|
||||
// looked up, whether or not an answer was there, and lookupAnswer is
|
||||
// what the lookup gave then, the zero Answer while GeoJS had given none.
|
||||
lookedUp bool
|
||||
lookupAnswer lookup.Answer
|
||||
lookupAnswer geojs.Answer
|
||||
// skippedLookupWait is true for a request that did not wait for GeoJS and
|
||||
// found no answer kept. SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT does not apply to it,
|
||||
// so that not waiting never lowers its client's limits.
|
||||
@@ -81,6 +81,9 @@ type request struct {
|
||||
checked time.Time
|
||||
// refused is what the request was refused with, nil if it was not.
|
||||
refused *refusal
|
||||
// findings is what the heuristics of besideLookup found of the request,
|
||||
// once waf has taken it, and nil before.
|
||||
findings *besideLookupFindings
|
||||
// requestBytes is how many bytes of the request's body have been read.
|
||||
requestBytes int64
|
||||
}
|
||||
@@ -119,7 +122,7 @@ func (h *handler) startRequest(
|
||||
RequestID: requestID(r, isQuestion),
|
||||
PeerIP: peer.String(),
|
||||
ForwardedFor: strings.Join(forwardedFor, ", "),
|
||||
ClientGroup: h.clientGroup(client).String(),
|
||||
ClientGroup: clientGroup(h.config, client).String(),
|
||||
ContentType: r.Header.Get("Content-Type"),
|
||||
RequestHeaders: requestHeaders(
|
||||
r,
|
||||
@@ -177,9 +180,9 @@ func requestHeaders(r *http.Request, names []string) map[string]string {
|
||||
// check is the one place where a request can be refused. It returns nil to let
|
||||
// the request through. The heuristics of checkClient decide: a request they
|
||||
// refuse is answered with SWW_WAF_BAN_RESPONSE, 403 for a block rule or the Core
|
||||
// Rule Set, or 414 or 431 for the header limits. In observe mode nothing is refused,
|
||||
// and the log line names what enforce mode would have done. ctx is the
|
||||
// request's own context.
|
||||
// Rule Set, 503 for heuristics that ran out of time, or 414 or 431 for the
|
||||
// header limits. In observe mode nothing is refused, and the log line names
|
||||
// what enforce mode would have done. ctx is the request's own context.
|
||||
func (rq *request) check(ctx context.Context) *refusal {
|
||||
v := rq.checkClient(ctx)
|
||||
if v == nil {
|
||||
@@ -196,6 +199,11 @@ func (rq *request) check(ctx context.Context) *refusal {
|
||||
case action == requestlog.ActionBlockedByBlockRule ||
|
||||
action == requestlog.ActionBlockedByCoreRuleSet:
|
||||
return &refusal{status: http.StatusForbidden, action: action}
|
||||
case action == requestlog.ActionChecksTimedOut:
|
||||
return &refusal{
|
||||
status: http.StatusServiceUnavailable,
|
||||
action: action,
|
||||
}
|
||||
case action == requestlog.ActionRequestLineTooLong:
|
||||
return &refusal{status: http.StatusRequestURITooLong, action: action}
|
||||
case action == requestlog.ActionRequestHeaderTooLarge:
|
||||
@@ -208,11 +216,16 @@ func (rq *request) check(ctx context.Context) *refusal {
|
||||
}
|
||||
}
|
||||
|
||||
// checkClient runs the heuristics of beforeLookup on the request, then looks
|
||||
// up the client's AS number and country, then runs those of afterLookup. It
|
||||
// returns the first verdict, or nil when none gives one. A client in
|
||||
// SWW_WAF_ALLOW_NETS skips them all, and is not looked up. ctx is the request's
|
||||
// own context.
|
||||
// checkClient runs the heuristics on the request, and returns the first
|
||||
// verdict, or nil when none gives one. A client in SWW_WAF_ALLOW_NETS skips them
|
||||
// all, and is not looked up. ctx is the request's own context.
|
||||
//
|
||||
// The heuristics of beforeLookup run first. Then the client's AS number and
|
||||
// country are looked up, and afterLookup runs once the lookup has its result.
|
||||
// While the lookup waits for GeoJS, besideLookup runs too, as runBesideLookup
|
||||
// says. A lookup that needs no wait is done, and afterLookup with it, before
|
||||
// besideLookup starts, so that besideLookup does no work for a request
|
||||
// afterLookup refuses.
|
||||
func (rq *request) checkClient(ctx context.Context) *verdict {
|
||||
if isInside(rq.client, rq.h.config.AllowNets) {
|
||||
return nil
|
||||
@@ -225,7 +238,28 @@ func (rq *request) checkClient(ctx context.Context) *verdict {
|
||||
return v
|
||||
}
|
||||
|
||||
rq.lookUp(ctx, now)
|
||||
lookedUp := rq.startLookUp(ctx, now)
|
||||
|
||||
select {
|
||||
case result := <-lookedUp:
|
||||
v := rq.runAfterLookup(ctx, result)
|
||||
if v != nil {
|
||||
return v
|
||||
}
|
||||
|
||||
return rq.runBesideLookup(ctx, nil)
|
||||
default:
|
||||
return rq.runBesideLookup(ctx, lookedUp)
|
||||
}
|
||||
}
|
||||
|
||||
// runAfterLookup notes result, the lookup's, and runs the heuristics of
|
||||
// afterLookup. It returns the first verdict, or nil when none gives one. ctx is
|
||||
// the request's own context.
|
||||
func (rq *request) runAfterLookup(
|
||||
ctx context.Context, result lookupResult,
|
||||
) *verdict {
|
||||
rq.noteLookup(result)
|
||||
|
||||
// The lookup can wait for GeoJS, so the checks after it read the time
|
||||
// again: a request counted at the time it arrived could be counted
|
||||
@@ -311,6 +345,16 @@ func timing(start, end time.Time) *float64 {
|
||||
return new(requestlog.Milliseconds(end.Sub(start)))
|
||||
}
|
||||
|
||||
// banAndCount runs list, afterAnswer or onReport, whose heuristics only ban.
|
||||
// It then counts the request for the anomaly thresholds and adds it to its
|
||||
// client's history. list runs first, so that a refusal limit it finds broken
|
||||
// is in that history. ctx is the request's own context.
|
||||
func (rq *request) banAndCount(ctx context.Context, list []heuristic) {
|
||||
rq.runBanningHeuristics(ctx, list)
|
||||
rq.countAnomalies()
|
||||
rq.addToHistory()
|
||||
}
|
||||
|
||||
// addToHistory adds the request, which has ended, to its client's
|
||||
// history, and counts its offences in the metrics, and then the lookup's
|
||||
// answer about the client, as answerAtTheEnd gives it, to that history and
|
||||
@@ -331,7 +375,7 @@ func (rq *request) addToHistory() {
|
||||
}
|
||||
|
||||
rq.h.limiter.AddToHistory(
|
||||
rq.h.clientGroup(rq.client),
|
||||
clientGroup(rq.h.config, rq.client),
|
||||
rq.h.now(),
|
||||
request,
|
||||
)
|
||||
@@ -339,7 +383,7 @@ func (rq *request) addToHistory() {
|
||||
|
||||
answer, found := rq.answerAtTheEnd()
|
||||
if found {
|
||||
rq.h.addLookup(answer)
|
||||
AddLookup(rq.h.config, rq.h.limiter, rq.h.ledger, answer)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -360,7 +404,7 @@ func (rq *request) countAnomalies() {
|
||||
rq.h.anomalies.Count(rq.h.now(), anomaly.Request{
|
||||
Host: rq.site,
|
||||
Client: rq.client,
|
||||
ClientGroup: rq.h.clientGroup(rq.client),
|
||||
ClientGroup: clientGroup(rq.h.config, rq.client),
|
||||
ASN: answer.ASN,
|
||||
ASName: answer.ASName,
|
||||
Country: answer.Country,
|
||||
@@ -382,16 +426,16 @@ func anomalyThresholdsSet(cfg *config.WAF) bool {
|
||||
// answer about it as the request ends, and whether there is one: the
|
||||
// lookup database's, which was there at once, or the one GeoJS has given
|
||||
// by then, which a request does not wait for unless a setting needs it.
|
||||
func (rq *request) answerAtTheEnd() (lookup.Answer, bool) {
|
||||
func (rq *request) answerAtTheEnd() (geojs.Answer, bool) {
|
||||
if !rq.lookedUp {
|
||||
return lookup.Answer{}, false
|
||||
return geojs.Answer{}, false
|
||||
}
|
||||
|
||||
if rq.h.config.LookupSource == "file" {
|
||||
if rq.h.config.LookupSource == config.LookupSourceFile {
|
||||
return rq.lookupAnswer, true
|
||||
}
|
||||
|
||||
return rq.h.geojs.Kept(rq.h.clientGroup(rq.client))
|
||||
return rq.h.geojs.Kept(clientGroup(rq.h.config, rq.client))
|
||||
}
|
||||
|
||||
// responseWriter is waf's answer to a request. It notes the status and
|
||||
|
||||
@@ -32,7 +32,7 @@ func (routeWatchTags) check(
|
||||
|
||||
route, missing := rq.h.routeWatch.Route(
|
||||
ctx,
|
||||
rq.h.clientGroup(rq.client),
|
||||
clientGroup(rq.h.config, rq.client),
|
||||
rq.client,
|
||||
)
|
||||
if missing != "" {
|
||||
|
||||
@@ -13,8 +13,8 @@ import (
|
||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||
)
|
||||
|
||||
// ruleFiles checks the request against the rules of the rule files at now,
|
||||
// notes the ids of those it matches in the log line, adds the tag
|
||||
// ruleFiles takes the rules of the rule files the request matched, which
|
||||
// findBesideLookup found, notes their ids in the log line, adds the tag
|
||||
// exploitsignature when it matches any, and refuses the request
|
||||
// when the last rule it matches refuses it: with ActionBlockedByBlockRule for a
|
||||
// block rule, and for a ban rule with the action for the ban it makes. A ban
|
||||
@@ -31,7 +31,7 @@ func (ruleFiles) check(
|
||||
rq *request,
|
||||
now time.Time,
|
||||
) *verdict {
|
||||
matched := rq.h.rules.Match(rq.in)
|
||||
matched := rq.findings.rules
|
||||
|
||||
for _, rule := range matched {
|
||||
rq.line.RuleIDs = append(rq.line.RuleIDs, rule.ID)
|
||||
@@ -66,7 +66,20 @@ func (ruleFiles) check(
|
||||
),
|
||||
figures: figures,
|
||||
}
|
||||
default:
|
||||
return nil
|
||||
case rules.ActionLog:
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// refusedByRule reports whether ruleFiles refuses a request that matched
|
||||
// matched: whether the last of them is a block rule or a ban rule.
|
||||
func refusedByRule(matched []rules.Rule) bool {
|
||||
if len(matched) == 0 {
|
||||
return false
|
||||
}
|
||||
|
||||
action := matched[len(matched)-1].Action
|
||||
|
||||
return action == rules.ActionBlock || action == rules.ActionBan
|
||||
}
|
||||
@@ -184,7 +184,7 @@ func TestRateLimitExemptNetsAreNeitherCountedNorRefused(t *testing.T) {
|
||||
type sentRequest struct {
|
||||
client string
|
||||
status int
|
||||
action string
|
||||
action requestlog.Action
|
||||
}
|
||||
|
||||
// wantAnswers sends requests to sww at addr one after another and
|
||||
|
||||
@@ -11,9 +11,8 @@ import (
|
||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||
)
|
||||
|
||||
// trapPaths refuses a request for a path in SWW_WAF_TRAP_PATHS: its path as a
|
||||
// path rule sees it, without the query, as sent or once percent-decoded, is one
|
||||
// of them. Such a request is a clear sign of attack, as a ban rule's match is:
|
||||
// trapPaths refuses a request for a path in SWW_WAF_TRAP_PATHS, as trapPath
|
||||
// finds it. Such a request is a clear sign of attack, as a ban rule's match is:
|
||||
// it bans the client's netblock, or in observe mode raises the alert for the
|
||||
// ban it would have made. The ban's notes give the path as sent, and so does
|
||||
// its figure.
|
||||
@@ -26,9 +25,8 @@ func (trapPaths) check(
|
||||
rq *request,
|
||||
now time.Time,
|
||||
) *verdict {
|
||||
path := rules.Path(rq.in)
|
||||
if !slices.Contains(rq.h.config.TrapPaths, path) &&
|
||||
!slices.Contains(rq.h.config.TrapPaths, rules.DecodeOnce(path)) {
|
||||
path := rq.findings.trapPath
|
||||
if path == "" {
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -39,3 +37,16 @@ func (trapPaths) check(
|
||||
figures: map[string]any{"path": path},
|
||||
}
|
||||
}
|
||||
|
||||
// trapPath returns the request's path as sent, when its path as a path rule
|
||||
// sees it, without the query, as sent or once percent-decoded, is in
|
||||
// SWW_WAF_TRAP_PATHS, and "" otherwise.
|
||||
func (rq *request) trapPath() string {
|
||||
path := rules.Path(rq.in)
|
||||
if !slices.Contains(rq.h.config.TrapPaths, path) &&
|
||||
!slices.Contains(rq.h.config.TrapPaths, rules.DecodeOnce(path)) {
|
||||
return ""
|
||||
}
|
||||
|
||||
return path
|
||||
}
|
||||
+31
-34
@@ -32,18 +32,22 @@ const (
|
||||
DispositionHeader = "X-Sww-Disposition"
|
||||
)
|
||||
|
||||
// Disposition is what waf tells rproxy to do with a request, in
|
||||
// DispositionHeader.
|
||||
type Disposition string
|
||||
|
||||
// The values of DispositionHeader.
|
||||
const (
|
||||
// DispositionAllow tells rproxy to pass the request to the app.
|
||||
DispositionAllow = "allow"
|
||||
DispositionAllow Disposition = "allow"
|
||||
// DispositionDeny tells rproxy to give the visitor the answer's status,
|
||||
// headers and body.
|
||||
DispositionDeny = "deny"
|
||||
DispositionDeny Disposition = "deny"
|
||||
// DispositionClose tells rproxy to close the visitor's connection
|
||||
// without an answer, which SWW_WAF_BAN_RESPONSE asks for when it is close.
|
||||
// waf closing the question's connection instead would read as
|
||||
// waf being down, and rproxy would then let the request through.
|
||||
DispositionClose = "close"
|
||||
DispositionClose Disposition = "close"
|
||||
)
|
||||
|
||||
// ReportPath is where rproxy sends the report of each request waf
|
||||
@@ -177,11 +181,13 @@ func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
|
||||
peer.Unmap(),
|
||||
true,
|
||||
)
|
||||
// The last deferred runs first, so that what is held has the finished log
|
||||
// line. Go's server sends the answer only once this function has
|
||||
// returned, so the report cannot come before it is held.
|
||||
defer rq.holdForReport()
|
||||
defer rq.finish()
|
||||
// These run even when a check panics. finish runs first, so that what is
|
||||
// held has the finished log line. Go's server sends the answer only once
|
||||
// this function has returned, so the report cannot come before it is held.
|
||||
defer func() {
|
||||
rq.finish()
|
||||
rq.holdForReport()
|
||||
}()
|
||||
|
||||
rq.site = r.Host
|
||||
rq.line.WAFLine = requestlog.WAFLineDecision
|
||||
@@ -190,35 +196,28 @@ func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
|
||||
underOwnPrefix := strings.HasPrefix(r.URL.Path, ownEndpointPrefix)
|
||||
|
||||
if refused == nil && !underOwnPrefix {
|
||||
// In observe mode, what enforce mode would have refused counts here.
|
||||
defer rq.runAfterAnswer(r.Context())
|
||||
|
||||
rq.allow()
|
||||
// In observe mode, what enforce mode would have refused counts here.
|
||||
rq.runBanningHeuristics(r.Context(), afterAnswer())
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
// The last deferred runs first: runAfterAnswer before addToHistory, so that
|
||||
// a broken refusal limit is in the client's history.
|
||||
defer rq.addToHistory()
|
||||
defer rq.countAnomalies()
|
||||
defer rq.runAfterAnswer(r.Context())
|
||||
|
||||
if refused != nil {
|
||||
rq.deny(*refused)
|
||||
|
||||
return
|
||||
} else {
|
||||
// The app never sees this prefix. waf's own endpoints are at its
|
||||
// own address, so this answers as a path with no endpoint does.
|
||||
rq.line.Action = requestlog.ActionOwnEndpoint
|
||||
rq.out.Header().Set(DispositionHeader, string(DispositionDeny))
|
||||
http.Error(
|
||||
rq.out,
|
||||
http.StatusText(http.StatusNotFound),
|
||||
http.StatusNotFound,
|
||||
)
|
||||
}
|
||||
|
||||
// The app never sees this prefix. waf's own endpoints are at its
|
||||
// own address, so this answers as a path with no endpoint does.
|
||||
rq.line.Action = requestlog.ActionOwnEndpoint
|
||||
rq.out.Header().Set(DispositionHeader, DispositionDeny)
|
||||
http.Error(
|
||||
rq.out,
|
||||
http.StatusText(http.StatusNotFound),
|
||||
http.StatusNotFound,
|
||||
)
|
||||
rq.banAndCount(r.Context(), afterAnswer())
|
||||
}
|
||||
|
||||
// readQuestion returns the visitor's address and the body a question carries,
|
||||
@@ -325,7 +324,7 @@ func (rq *request) allow() {
|
||||
rq.allowed = true
|
||||
|
||||
header := rq.out.Header()
|
||||
header.Set(DispositionHeader, DispositionAllow)
|
||||
header.Set(DispositionHeader, string(DispositionAllow))
|
||||
|
||||
if rq.h.config.AddLookupHeaders {
|
||||
setLookupHeaders(header, rq.line.ASN, rq.line.Country)
|
||||
@@ -348,13 +347,13 @@ func (rq *request) deny(r refusal) {
|
||||
if r.status == 0 {
|
||||
// SWW_WAF_BAN_RESPONSE is close. Written past rq.out, since the log line's
|
||||
// status stays 0: the visitor is sent nothing.
|
||||
rq.out.Header().Set(DispositionHeader, DispositionClose)
|
||||
rq.out.Header().Set(DispositionHeader, string(DispositionClose))
|
||||
rq.out.ResponseWriter.WriteHeader(http.StatusOK)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
rq.out.Header().Set(DispositionHeader, DispositionDeny)
|
||||
rq.out.Header().Set(DispositionHeader, string(DispositionDeny))
|
||||
http.Error(rq.out, http.StatusText(r.status), r.status)
|
||||
}
|
||||
|
||||
@@ -491,9 +490,7 @@ func (h *handler) finishWith(
|
||||
if rq.allowed {
|
||||
rq.line.DurationUpstreamTotalMS = &report.DurationUpstreamTotalMS
|
||||
|
||||
rq.runBanningHeuristics(ctx, onReport())
|
||||
rq.countAnomalies()
|
||||
rq.addToHistory()
|
||||
rq.banAndCount(ctx, onReport())
|
||||
|
||||
rq.h.metrics.RequestReported(
|
||||
&rq.line,
|
||||
|
||||
@@ -205,13 +205,13 @@ func wantDisposition(
|
||||
t *testing.T,
|
||||
got answer,
|
||||
status int,
|
||||
disposition string,
|
||||
disposition waf.Disposition,
|
||||
) {
|
||||
t.Helper()
|
||||
wantStatus(t, got, status)
|
||||
|
||||
if answered := got.header.Get(waf.DispositionHeader); answered !=
|
||||
disposition {
|
||||
string(disposition) {
|
||||
t.Errorf(
|
||||
"answered with disposition %q, want %q",
|
||||
answered,
|
||||
@@ -1480,7 +1480,7 @@ func TestEveryDeniedRequestsReportGetsItsLastLine(t *testing.T) {
|
||||
env map[string]string
|
||||
path string
|
||||
status int
|
||||
action string
|
||||
action requestlog.Action
|
||||
}{
|
||||
{
|
||||
"deny", map[string]string{denyNets: client}, "/", http.StatusForbidden,
|
||||
|
||||
+9
-1
@@ -2,6 +2,9 @@
|
||||
# script/cibuild: run the CI build. It bootstraps first: a CI runner
|
||||
# checks out and runs this and nothing else, and script/fmt-check runs
|
||||
# the formatter on the host, which a pristine checkout cannot do.
|
||||
# --no-cache for the same reason as script/docker: the gate phases the
|
||||
# final stage depends on are RUN steps, and a cached one is a check that
|
||||
# did not run.
|
||||
set -eu
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||
@@ -17,8 +20,13 @@ main() {
|
||||
# the version a build stage derives from the .git in the context.
|
||||
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
||||
[ -n "$version" ] || version="unknown"
|
||||
docker build \
|
||||
revision="$(git rev-parse HEAD 2>/dev/null || true)"
|
||||
[ -n "$revision" ] || revision="unknown"
|
||||
created="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
docker build --no-cache \
|
||||
--build-arg VERSION="$version" \
|
||||
--build-arg REVISION="$revision" \
|
||||
--build-arg CREATED="$created" \
|
||||
-t "$("$SCRIPT_DIR/projectname")" .
|
||||
}
|
||||
|
||||
|
||||
+8
-1
@@ -1,6 +1,8 @@
|
||||
#!/bin/sh
|
||||
# script/docker: build the Docker image tagged with the project name.
|
||||
# Identical in all repos; the tag comes from script/projectname.
|
||||
# --no-cache because the gate phases the final stage depends on are RUN
|
||||
# steps, and a cached one is a check that did not run.
|
||||
set -eu
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||
@@ -14,8 +16,13 @@ main() {
|
||||
# the version a build stage derives from the .git in the context.
|
||||
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
||||
[ -n "$version" ] || version="unknown"
|
||||
docker build \
|
||||
revision="$(git rev-parse HEAD 2>/dev/null || true)"
|
||||
[ -n "$revision" ] || revision="unknown"
|
||||
created="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
docker build --no-cache \
|
||||
--build-arg VERSION="$version" \
|
||||
--build-arg REVISION="$revision" \
|
||||
--build-arg CREATED="$created" \
|
||||
-t "$("$SCRIPT_DIR/projectname")" .
|
||||
}
|
||||
|
||||
|
||||
+2
-2
@@ -170,8 +170,8 @@ main() {
|
||||
trap cleanup EXIT
|
||||
trap 'exit 1' HUP INT TERM
|
||||
|
||||
docker build -t "$IMAGE" .
|
||||
docker build --build-arg SWW_IMAGE="$IMAGE" \
|
||||
docker build --no-cache -t "$IMAGE" .
|
||||
docker build --no-cache --build-arg SWW_IMAGE="$IMAGE" \
|
||||
-t "$APP_IMAGE" deploy/example-app
|
||||
|
||||
docker network create "$NETWORK" >/dev/null
|
||||
|
||||
+3
-2
@@ -5,7 +5,8 @@
|
||||
# make its answer untrustworthy.
|
||||
#
|
||||
# The phase is not the last stage in the file, so it is built only when
|
||||
# --target names it. The tag makes each build replace the previous image
|
||||
# --target names it. --no-cache because a cached lint layer is a lint
|
||||
# that did not run. The tag makes each build replace the previous image
|
||||
# instead of leaving a dangling one behind.
|
||||
set -eu
|
||||
|
||||
@@ -14,7 +15,7 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
||||
|
||||
main() {
|
||||
cd "$ROOT"
|
||||
docker build \
|
||||
docker build --no-cache \
|
||||
--target lint \
|
||||
-t "$("$SCRIPT_DIR/projectname")-lint" .
|
||||
}
|
||||
|
||||
+3
-2
@@ -2,7 +2,8 @@
|
||||
# script/test: run the test suite. Testing is a phase of the Dockerfile
|
||||
# and this builds that phase alone, on the same terms as script/lint:
|
||||
# --target because a phase that is not the last stage is built only when
|
||||
# named, and a tag so each build replaces the previous image.
|
||||
# named, --no-cache because a cached test layer is a test that did not
|
||||
# run, and a tag so each build replaces the previous image.
|
||||
set -eu
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||
@@ -10,7 +11,7 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
||||
|
||||
main() {
|
||||
cd "$ROOT"
|
||||
docker build \
|
||||
docker build --no-cache \
|
||||
--target test \
|
||||
-t "$("$SCRIPT_DIR/projectname")-test" .
|
||||
}
|
||||
|
||||
Loaded 100 of 101 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user