diff --git a/README.md b/README.md index 1ed7527..c947ac5 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ shipped in one image, with two commands that do this together: - `sww rproxy` is a small reverse proxy in front of one app. It sits behind a TLS-terminating proxy, such as traefik, nginx or Caddy. It asks `waf` about each request, passes the allowed ones to the app, and passes the app's answers - back. It keeps nothing between requests and writes nothing to disk. + back. It writes nothing to disk. - `sww waf` decides. It runs the checks and keeps the bans, the counters and each client's history. One `waf` serves many sites: a client banned through one site is refused on all of them, and its requests on all of them count @@ -35,7 +35,26 @@ shipped in one image, with two commands that do this together: The site keeps answering while `waf` is down, slow or restarting: without an answer in time, `rproxy` passes the request to the app. -It protects against: +`rproxy` also protects the app on its own, whether or not `waf` answers. With +the default settings, `rproxy`: + +- answers `400` to a malformed request, such as a broken header line or a + missing `Host`; +- answers `405` to a method it does not pass on, such as `TRACE`; +- answers `431` to a request line and headers over 32 KiB together; +- gives the visitor 60 seconds to send the request line and headers, and then 60 + seconds more for the whole request to reach the app; +- answers `413` to a request body over 100 MiB; +- gives the app's answer 30 minutes to reach the visitor, counted from the end + of the request; +- answers `502` in place of an app's answer whose body is over 1 GiB, or cuts it + off if it has started; +- closes a kept-open connection after 120 seconds idle. + +`rproxy` does not limit the number of headers. `waf` limits the number of +headers and cookies, and the length of the request line and of each header line. + +Together, `rproxy` and `waf` protect against: - Scrapers that fetch every page of a site, all day. Each client gets a limit on requests and on body bytes per minute, per hour and per day. The defaults @@ -57,8 +76,6 @@ It protects against: limit, so that they are banned sooner; - refuse or limit clients on a blocklist, a DNS blocklist, or with a high AbuseIPDB score. -- Slow and oversized requests. Every request and every answer has a time limit - and a size limit. For the operator, it: @@ -281,13 +298,13 @@ Every setting the code reads, with its default: | `SWW_RPROXY_LISTEN_ADDR` | `:8080` | Where `rproxy` listens; leave the host part empty. | | `SWW_RPROXY_TRUSTED_PROXIES` | `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16` | The netblocks whose `X-Forwarded-For` `rproxy` believes; set but empty, it believes none. | | `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS` | `2000` | How long `rproxy` waits for `waf`'s answer to a question and to a report, in milliseconds. | -| `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` | `allow` | What `rproxy` does with a request when `waf` cannot be reached or does not answer in time: `allow` passes it to the app, `deny` answers `503`. | +| `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` | `allow` | What `rproxy` does with a request when `waf` gives no answer in time, and while the circuit breaker for asks to `waf` is open: `allow` passes it to the app, `deny` answers `503`. | | `SWW_RPROXY_MAX_REQUEST_SIZE` | `100M` | The largest request body, a size or `off`; a larger one is answered `413`. | | `SWW_RPROXY_MAX_RESPONSE_SIZE` | `1G` | The largest response body, a size or `off`. The visitor gets `502` in place of a larger one, or it is cut off if it has already started. | | `SWW_RPROXY_MAX_REQUEST_TIME_MS` | `60000` | Two time limits, each this long, in milliseconds or `off`: one for the visitor to send its request line and headers, then one for the whole request to reach the app. | | `SWW_RPROXY_MAX_RESPONSE_TIME_MS` | `1800000` | How long the answer may take, from the end of the request to its last byte reaching the visitor, in milliseconds or `off`. | | `SWW_WAF_LISTEN_ADDR` | `:8080` | Where `waf` listens; leave the host part empty. | -| `SWW_WAF_INSTANCE_NAME` | the container's host name | The name in every line `waf` logs, and in every alert and metric, such as `fsn1/waf`. | +| `SWW_WAF_INSTANCE_NAME` | the container's host name | The name in every line `waf` logs, and in every alert and metric, such as `host1/waf`. | | `SWW_WAF_MODE` | `enforce` | `observe` refuses nothing and makes no ban; each log line says what `enforce` would have done. | | `SWW_WAF_IPV6_GROUP_PREFIX` | `64` | The prefix length of the IPv6 netblock that counts as one client, from 32 to 128. | | `SWW_WAF_MAX_TRACKED_CLIENTS` | `20000` | The most clients kept, in memory and in `clients.json`; the least recently seen is dropped first. | @@ -312,7 +329,7 @@ Every setting the code reads, with its default: | `SWW_WAF_BAN_SCOPE_V4_PREFIX` | `32` | The prefix length of the netblock a ban on an IPv4 client covers, from 0 to 32. | | `SWW_WAF_LOOKUP_SOURCE` | `geojs` | Where each client's AS number and country come from: `geojs` (a web service), `file` (a local database) or `off`. | | `SWW_WAF_LOOKUP_DB_PATH` | empty | The IPinfo Lite database file, in its `.mmdb` form, read while the source is `file`. | -| `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` | `800` | How long a request waits for its client's first GeoJS answer when a setting needs it, and how long a request to GeoJS may take, in milliseconds. | +| `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` | `800` | How long a request waits for its client's first GeoJS answer when a setting needs it, in milliseconds. The request to GeoJS goes on, and a later answer is kept. | | `SWW_WAF_ADD_LOOKUP_HEADERS` | `true`, or `false` while the lookup source is `off` | `true` passes the app the client's AS number and country in `X-Client-ASN` and `X-Client-Country`. | | `SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES` | empty | When set, the only countries whose clients get through; a client the lookup cannot place is refused too. | | `SWW_WAF_TAG_LISTS` | empty | Address lists that each add a tag to their clients' requests, as `name=URL` pairs, such as `torexit=https://example.com/tor.txt`. | @@ -331,7 +348,7 @@ Every setting the code reads, with its default: | `SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD` | `5` | The Core Rule Set score at which a request is a match, a whole number or `off`; one critical rule scores 5. | | `SWW_WAF_CORE_RULE_SET_DISABLED_RULES` | `920340,920420,920440,920640,930130,930140` | Core Rule Set rule ids switched off; a list you give replaces the default, so repeat these six to keep them off. | | `SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS` | empty | Path prefixes the Core Rule Set does not inspect. | -| `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` | `off` | How much of a request body the Core Rule Set reads, a size up to `64K`; `off` reads none. | +| `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` | `off` | How much of a request body the Core Rule Set reads, a size up to `8K`; `off` reads none. | | `SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE` | unset | A file of Core Rule Set exclusions, as Coraza directives, that switch rules off for some fields or paths only. | | `SWW_WAF_APPS` | unset | The apps each hostname serves, such as `git.example.com:gitea;blog.example.com:wordpress`; only those apps' fixes apply there. "Attack detection" lists the apps. Listing `forgejo` brings gitea's fixes too. Unset, every app's fixes apply to every hostname. | | `SWW_WAF_BLOCKLIST_URLS` | empty | Blocklists to fetch, text files of addresses and netblocks one to a line, such as `https://www.spamhaus.org/drop/drop.txt`. | @@ -513,7 +530,7 @@ services: image: smallwebwaf environment: SWW_COMMAND: waf - SWW_WAF_INSTANCE_NAME: fsn1/waf + SWW_WAF_INSTANCE_NAME: host1/waf # Refuse nothing yet; log what enforce mode would have done. SWW_WAF_MODE: observe # Look addresses up in a local IPinfo Lite database, never GeoJS. @@ -582,6 +599,11 @@ request reaches the app. 16. A path under `/_sww/` is answered `404`, so that the app never sees that prefix. +Steps 12 to 14 only read the request. While the lookup waits for GeoJS, they run +at the same time as steps 6 to 11. The first of the two sides to refuse the +request ends the checks, and `waf` answers at once. Steps 12 to 14 have 500 ms +together. + `rproxy` takes the rest, with `waf` counting at the end: 17. The request is passed to the app, streaming. @@ -594,9 +616,10 @@ A refused visitor is answered with `SWW_WAF_BAN_RESPONSE`. A `block` rule or a Core Rule Set match answers `403` and bans nobody, since the Core Rule Set has false positives; such refusals count toward `SWW_WAF_REFUSALS_LIMIT_PER_MINUTE` instead. A request over the header limits of step 3 is answered `414` for a -request line over 8 KiB and `431` otherwise. It bans nobody either. `rproxy` -itself answers `405` to a method it does not pass on, and the status for a size -or time limit, such as `413` for a body too large. +request line over 8 KiB and `431` otherwise. It bans nobody either. Nor does a +request that steps 12 to 14 have not finished within 500 ms, which is answered +`503`. `rproxy` itself answers `405` to a method it does not pass on, and the +status for a size or time limit, such as `413` for a body too large. ### Clients and bans @@ -732,9 +755,9 @@ stops the running process (see [`docs/state-files.md`](docs/state-files.md)). - `internal/waf`: `waf`'s server, which runs the checks and answers its own endpoints. - `internal/ratelimit`, `internal/bans`, `internal/rules`, - `internal/coreruleset`, `internal/lookup`, `internal/reputation`, - `internal/routewatch`, `internal/anomaly` and `internal/alerts`: one check or - source each. + `internal/coreruleset`, `internal/geojs`, `internal/lookup`, + `internal/reputation`, `internal/routewatch`, `internal/anomaly` and + `internal/alerts`: one check or source each. - `internal/state`, `internal/requestlog`, `internal/remotelog` and `internal/metrics`: the state and cache files, the log lines, the syslog sender and the metrics. @@ -801,10 +824,9 @@ on the host. - `script/docker`: builds the image, whose build runs the tests and the linter first. `make docker`. - `script/publish`: runs `script/docker`, whose build happens on the Docker host - named by `DOCKER_HOST` (`ssh://root@fsn1app1` unless set). It then pushes the - image to `PUBLISH_IMAGE` (`git.eeqj.de/sneak/smallwebwaf` unless set). It - refuses a working tree with uncommitted changes. `make publish`. It pushes - four tags: + named by `DOCKER_HOST`, then pushes the image to `PUBLISH_IMAGE`, such as + `registry.example.com/smallwebwaf`. Both variables are required. It refuses a + working tree with uncommitted changes. `make publish`. It pushes four tags: - the UTC date as `YYYY-MM-DD`; - the short commit hash; - the branch name; diff --git a/SPEC.md b/SPEC.md index b062148..ca67a0a 100644 --- a/SPEC.md +++ b/SPEC.md @@ -87,8 +87,10 @@ never as a time. change, and tells which rules a request matches. - `internal/coreruleset`: the Core Rule Set, with the five changes and the apps' fixes listed under "Attack detection", as Coraza directives. - - `internal/lookup`: looks clients up through GeoJS or in the lookup - database. + - `internal/geojs`: looks clients up through GeoJS and keeps the answers. It + imports no other package of this repository: `waf` gives it what it needs + to count and report failures. + - `internal/lookup`: looks clients up in the lookup database. - `internal/reputation`: fetches the lists, asks the DNS blocklists and AbuseIPDB in the background, and keeps their answers. - `internal/routewatch`: asks a routewatch instance about clients in the @@ -171,7 +173,8 @@ field gives that name. `refusal_limit` and `byte_limits` give theirs in `country=` tag. Otherwise the request goes on at once. A request that ends before the answer - comes is logged without it. + comes is logged without it. While the request waits, steps 12 to 14 run at + the same time as steps 6 to 11 (see "Answering in time"). 7. The country list (`countries`; see "Country list"). 8. The reputation sources (see "Reputation"), in this order: @@ -507,19 +510,25 @@ service that keeps its visitors' addresses on its own host. dropped from that table. The answer reaches the client's history and ban notes when it comes. - `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` is how long a visitor's request - waits for its client's first answer, when a setting needs it (see step 6). It - also bounds each request to GeoJS. A client whose answer has not come in time - counts as unknown until it comes, and its later requests do not wait. Its - address is asked about again in the background. + waits for its client's first answer, when a setting needs it (see step 6). A + client whose answer has not come in time counts as unknown until it comes, and + its later requests do not wait. +- A request to GeoJS may take up to 10 seconds, however long visitors wait. It + goes on after the visitor's request ends, and its answer is kept. Say GeoJS + answers after 5 seconds. The visitor's request is decided after 800 + milliseconds, without a country. A request from the same client 6 seconds + later gets the answer. +- Requests from one client that come together share one request to GeoJS. - A client the lookup cannot place has an unknown country and AS number, which the country list and the biased thresholds treat as described there. Nothing else treats it differently. - GeoJS publishes no rate limit, but its terms forbid "an excessive amount of - API requests", judged by GeoJS alone, and let it block a caller. While GeoJS - is slow, down or refusing `waf`, new clients count as unknown. GeoJS is then - left alone for a second, and twice as long after each further failure, up to - five minutes. Each failure raises a `source_failure` alert (see "Alerting"). A - service that cannot accept this uses the file. + API requests", judged by GeoJS alone, and let it block a caller. A request to + GeoJS fails when it takes over 10 seconds or GeoJS answers with an error. + While GeoJS is slow, down or refusing `waf`, new clients count as unknown. + After a failure GeoJS is left alone for a second, and twice as long after each + further failure, up to five minutes. Each failure raises a `source_failure` + alert (see "Alerting"). A service that cannot accept this uses the file. - The file is the free IPinfo Lite database (`ipinfo_lite.mmdb`), whose `asn`, `as_name` and `country_code` fields are read. The operator downloads it with a free IPinfo account and mounts it read-only; `waf` never fetches it and holds @@ -704,11 +713,12 @@ files, the limits and the bans still apply to the client that sends it. rules is loaded yet. - `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` switches body inspection on, up - to that size. It takes at most `64K`, since `waf` gets no more of a body than - that (see "The answer"). Form data and multipart bodies are read up to the - size. JSON and XML bodies are read only when no larger than the size, since - those cannot be read in part. Any other body is not read: the Core Rule Set - would read it as form data, where binary content such as a git push trips + to that size. It takes at most `8K`, so that `waf` answers in time (see + "Answering in time"). Form data and multipart bodies are read up to the size. + JSON and XML bodies are read only when no larger than the size, since those + cannot be read in part. A JSON body over 1 KiB is not read either, whatever + the size (see "Answering in time"). Any other body is not read: the Core Rule + Set would read it as form data, where binary content such as a git push trips rules written for text. A body the Core Rule Set cannot parse adds 5 to the score, and so does a multipart body cut by the limit in the middle of a part's header. Body inspection suits apps whose forms carry no code. @@ -2370,8 +2380,9 @@ runs no check, writes no request log line and counts nothing in the metrics. ### The answer -`waf` runs steps 3 to 16 of "What happens to a request", in that order. Every -answer but the last below carries `X-Sww-Disposition`. +`waf` runs steps 3 to 16 of "What happens to a request", in that order, apart +from the steps that run at the same time (see "Answering in time"). Every answer +but the last below carries `X-Sww-Disposition`. - `X-Sww-Disposition: allow`, status `200`, no body. `rproxy` passes the request to the app. While `SWW_WAF_ADD_LOOKUP_HEADERS` is `true`, the answer also @@ -2383,6 +2394,8 @@ answer but the last below carries `X-Sww-Disposition`. - `SWW_WAF_BAN_RESPONSE`, for a refusal; - `403`, for a `block` rule or the Core Rule Set; - `414` or `431`, for a request over the header limits; + - `503`, for a request that steps 12 to 14 have not finished within 500 + milliseconds; - `404`, for a path under `/_sww/` that passes the checks, so that the app never sees that prefix. - `X-Sww-Disposition: close`, status `200`, no body. This takes the place of a @@ -2397,23 +2410,28 @@ In `observe` mode a request that `enforce` mode would refuse is answered answered `404`. The Core Rule Set inspects the body a question carries only while -`SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` is set. It inspects at most 64 KiB -less one byte. So it does not inspect a JSON or XML body of 64 KiB or more, -which may have been cut short. +`SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` is set, and at most 8 KiB of it. ### Answering in time `rproxy` stops waiting for `waf` after `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, by default 2 seconds, and `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` then -decides. The longest a question waits on an outside source is -`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, by default 800 milliseconds, well -within that time. +decides. Two things in a decision take time: + +- the wait for GeoJS, at most `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, by + default 800 milliseconds; +- the trap paths, the rule files and the Core Rule Set, which have 500 + milliseconds. + +They run at the same time, so `waf` answers within the longer of the two, well +within `rproxy`'s wait. GeoJS is the only outside source a question can wait on. A question waits only when step 6 says so, and for at most `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, by default 800 milliseconds. A -client without an answer by then counts as unknown. No question waits on these, -which run in the background: +client without an answer by then counts as unknown. The setting bounds only the +question's wait: the request to GeoJS goes on, and its answer is kept for later +questions. No question waits on these, which run in the background: - the DNS blocklists and AbuseIPDB; - fetching the blocklists, the tag lists and the file @@ -2438,10 +2456,38 @@ never wait. These can still wait for GeoJS: - a country list, which needs the answer; - a rate limit a percentage has lowered, which needs the answer; - a DNS blocklist or AbuseIPDB, while `SWW_WAF_DNSBL_ABUSEIPDB_ACTION` is - `deny`; -- a trap path; -- a rule file; -- the Core Rule Set. + `deny`. + +The trap paths, the rule files and the Core Rule Set, steps 12 to 14, run during +that wait. Once a request has passed the bans, the lookup starts. While it waits +for GeoJS, steps 12 to 14 run at the same time as steps 6 to 11. A lookup that +needs no wait has its answer at once, and steps 7 to 11 then run before steps 12 +to 14 start. + +- The first of the two sides to refuse the request ends the checks. `waf` + answers at once, and does not wait for the other side. The lookup goes on, and + its answer is kept. +- What the other side finds leaves no trace. It counts nothing, bans nothing, + raises no alert, counts no metric and adds nothing to the log line. +- When both sides would refuse, the one that finishes first is answered and + logged. +- When neither refuses, step 15 runs. + +Steps 12 to 14 only read the request. They have 500 milliseconds together, from +their start. The limit is fixed, with no setting. A request they have not +finished by then is refused: + +- the answer is `deny`, with `503`; +- the log line's action is `checks_timed_out`, and its `heuristic` names the + step that had not finished; +- the refusal makes no ban, is no offence and does not count toward the refusal + limit; +- it holds in every `SWW_WAF_CORE_RULE_SET_MODE`; +- in `observe` mode the request is answered `allow` with `would_action`, like + every refusal. + +The limit is measured on the clock. While `waf`'s host is so busy that an +ordinary request takes 500 milliseconds, that request is refused too. The header limits are fixed, with no setting. A line's length is in bytes and does not count its line ending. `waf` refuses a request whose request line is @@ -2474,12 +2520,28 @@ client in `SWW_WAF_ALLOW_NETS` skips them, as it skips the Core Rule Set. In `observe` mode such a request is answered `allow` with `would_action`, and no other check runs on it. -The Core Rule Set runs each of its rules on each cookie, so thousands of small -cookies would keep it busy for over a second. One 30 KB cookie or query string -would keep it busy for a third of a second or more. A request with several long -`Cookie` lines, each within the limits, can keep it as busy. Enough such -requests at once can still overload `waf` past -`SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`. +The Core Rule Set is the one check a visitor can keep busy for long. It runs +each of its rules on each part of a request, so thousands of small cookies would +keep it busy for over a second. These fixed limits bound what it reads: + +- the header limits above; +- `rproxy`'s limit of 32 KiB on the request line and headers together (see + "`rproxy`'s limits"); +- 8 KiB of a body, the most `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` takes, + or 1 KiB of a JSON body. + +Coraza names each value in a JSON body by the whole path to it, such as +`json.users.0.name`. So the names and values the Core Rule Set reads in a JSON +body can add up to more than the body. From 1 KiB of JSON, Coraza makes at most +8 KiB of them, no more than the largest body of another kind. + +A request can reach all of these at once. At paranoia level 4, with no rule +switched off, the Core Rule Set takes under a second on one core of a current +server to inspect it. `waf` refuses it if the inspection takes over 500 +milliseconds. Coraza cannot stop an inspection part-way, so the inspection runs +to its end with nobody waiting for it. These limits keep one request from +holding a processor for more than about a second. Enough such requests at once +can still overload `waf` past `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`. ### The report @@ -2583,11 +2645,11 @@ request and ban notes name the site by its host. ## `rproxy` `sww rproxy` is the reverse proxy in front of one app. It asks `waf` about each -request, and passes on what `waf` allows. It keeps nothing between requests and -writes nothing to disk. When `waf` cannot be connected to, does not answer at -all, or does not answer within `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, -`SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` decides. Its default, `allow`, -keeps the site answering. +request, and passes on what `waf` allows. It writes nothing to disk. The only +thing it keeps between requests is its circuit breaker for asks to `waf`. +`SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` decides what is done with a +request when `waf` gives no answer in time, and while the circuit breaker is +open. Its default, `allow`, keeps the site answering. `rproxy` reads only its `SWW_RPROXY_` settings, which `docs/settings.md` lists. One of them is required: `SWW_RPROXY_WAF_BACKEND`, `waf`'s URL. It has no @@ -2605,7 +2667,8 @@ only. The image runs it by default (see "Deployment"). 4. It refuses a request with a method it does not pass on, or one it cannot put to `waf` as the visitor sent it (see "Requests `rproxy` refuses itself"). 5. It reads up to the first 64 KiB of the body into memory. -6. It sends `waf` the question (see "The question"). It speaks HTTP/1.1 to +6. It sends `waf` the question (see "The question"), unless the circuit breaker + keeps it from asking (see "The circuit breaker"). It speaks HTTP/1.1 to `waf`, whatever `waf` offers. Go's HTTP/2 client would refuse to send a visitor's `Upgrade` header, and most `Connection` headers. The question never carries the visitor's `Trailer`. @@ -2679,23 +2742,24 @@ log line has status `0`. ### Without an answer from `waf` `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` decides what is done with the -request only when `waf`: +request when `waf`: - cannot be connected to; - does not answer at all; - does not answer within `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, which covers connecting, sending the question and reading the whole answer. -An answer whose body breaks off counts as no answer at all. +An answer whose body breaks off counts as no answer at all. The setting also +decides, without asking `waf`, while the circuit breaker is open. `allow`, the default, passes the request to the app. `deny` answers `503`. Either way `rproxy` writes a process warning, its log line has `waf_backend_no_answer`, and it sends no report. -A visitor can make `waf` slow. A request that keeps `waf` busy past -`SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, like any request while `waf` is overloaded, -gets `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION`. With `deny`, such a request -gets `503` instead of reaching the app. +`waf` taking longer than `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS` to decide a request +is a bug in `waf`. Such a request gets +`SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION`, like any request while `waf` is +overloaded. With `deny`, it gets `503` instead of reaching the app. A path under `/_sww/` that `allow` would pass on is answered `404` instead, so that the app never sees that prefix. @@ -2710,6 +2774,28 @@ A visitor that goes away while `waf` is being asked is no failure of `waf`. `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` does not apply. Its log line has status `0` and no `disposition`. +### The circuit breaker + +`rproxy` stops asking `waf` for a while when `waf` keeps failing to answer. An +ask fails when `waf` gives no answer in time (see "Without an answer from +`waf`"), or an answer `rproxy` cannot pass on. Any other answer in time +succeeds, whatever it decides. A visitor that goes away while `waf` is being +asked counts as neither. + +The breaker works like this: + +- It opens when 5 asks fail within 5 seconds. Fewer failures never open it. +- While it is open, a request gets + `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` at once, without asking `waf`. + It gets the same process warning and `waf_backend_no_answer` as a request + whose ask failed. +- While it is open, one request a second still asks `waf`. +- Each ask that succeeds clears the failures counted so far, and closes the + breaker if it is open. + +The breaker lives in memory, and a restart of `rproxy` closes it. It has no +settings. + ### Passing a request to the app `rproxy` sends the app at `SWW_RPROXY_UPSTREAM_URL` the visitor's request: the @@ -2983,8 +3069,8 @@ host directory, keeps them across deploys. A host directory mounted there needs no change of owner, and files an earlier owner left in it can be read and replaced. The volume holds a few tens of MiB at most with the defaults, and needs no backup beyond the host's own. The image declares no volume, since every -app image built on it would inherit it. `rproxy` keeps nothing, and needs no -volume. +app image built on it would inherit it. `rproxy` writes nothing to disk, and +needs no volume. `waf` keeps its cache files in `SWW_WAF_CACHE_DIR`, `/var/cache/sww` by default, which the `run` script makes. The cache directory needs no volume and can be diff --git a/docs/alerts.md b/docs/alerts.md index 11918c5..6f14a37 100644 --- a/docs/alerts.md +++ b/docs/alerts.md @@ -34,7 +34,7 @@ on one line: ```json { - "instance": "fsn1/waf", + "instance": "host1/waf", "time": "2026-10-06T12:00:50.123461Z", "event": "ban", "host": "app.example", @@ -109,7 +109,7 @@ on one line: ## Slack and ntfy Slack and ntfy get a message with a title and a text. The title is the instance -and the event, such as `fsn1/waf: ban`. The text is the `reason`, then one line +and the event, such as `host1/waf: ban`. The text is the `reason`, then one line for each of these that the alert has: - the host, the client, the netblock and the country; diff --git a/docs/request-log.md b/docs/request-log.md index 26f76ef..77bf096 100644 --- a/docs/request-log.md +++ b/docs/request-log.md @@ -26,7 +26,7 @@ These questions write no line: A `report` line: ``` -{"type":"request","waf_line":"report","time":"2026-10-03T12:00:00.123Z","instance":"fsn1/waf","client_ip":"203.0.113.9","method":"GET","scheme":"https","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","peer_ip":"172.18.0.2","forwarded_for":"203.0.113.9","client_group":"203.0.113.9/32","asn":"AS64496","as_name":"Example Net","country":"DE","request_headers":{"accept":"*/*"},"action":"forward","counts":{"minute":1,"hour":12,"day":40,"minute_bytes":5120,"hour_bytes":61440,"day_bytes":204800},"core_rule_set_score":0,"duration_total_ms":3.217,"duration_checks_ms":0.041,"duration_core_rule_set_ms":0.029,"duration_upstream_total_ms":3.104} +{"type":"request","waf_line":"report","time":"2026-10-03T12:00:00.123Z","instance":"host1/waf","client_ip":"203.0.113.9","method":"GET","scheme":"https","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","peer_ip":"172.18.0.2","forwarded_for":"203.0.113.9","client_group":"203.0.113.9/32","asn":"AS64496","as_name":"Example Net","country":"DE","request_headers":{"accept":"*/*"},"action":"forward","counts":{"minute":1,"hour":12,"day":40,"minute_bytes":5120,"hour_bytes":61440,"day_bytes":204800},"core_rule_set_score":0,"duration_total_ms":3.217,"duration_checks_ms":0.041,"duration_core_rule_set_ms":0.029,"duration_upstream_total_ms":3.104} ``` ### Every line @@ -105,6 +105,8 @@ Every one of these lines has: `blocked_by_core_rule_set`: refused by that check; - `request_line_too_long`: a request line over the header limits; - `request_header_too_large`: headers over the header limits; + - `checks_timed_out`: refused with `503` because the trap paths, the rule + files and the Core Rule Set had not finished within 500 ms; - `own_endpoint`: answered at one of `waf`'s own endpoints, or `404` for a path under `/_sww/`. - `would_action`: in `observe` mode, what `enforce` mode would have done with a @@ -134,6 +136,11 @@ Every one of these lines has: `SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD`; - `refusal_limit`: `window`, `count` and `limit`, of the refusal limit; - `byte_limits`: `window`, `count` and `limit`, the count in bytes. + + For a request refused as `checks_timed_out`, `heuristic` is the one of + `trap_paths`, `rule_files` and `core_rule_set` that had not finished, and + its figure is `time_limit_ms`, 500. + - `after_answer_heuristic`: the heuristic that banned the client after `waf` answered the request, in `observe` mode too: `refusal_limit` or `byte_limits`. - `after_answer_heuristic_figures`: an object of the figures that heuristic @@ -238,7 +245,8 @@ meanings where they differ: - `disposition`: what was done with the request: `allow`, `deny` or `close`. It is `waf`'s answer, or the value of `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION`. It is left out when: - - `waf` was not asked, as for the health check; + - `waf` was not asked and `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` did + not apply, as for the health check; - the visitor went away while `waf` was being asked; - `waf`'s answer had no valid `X-Sww-Disposition`; - `rproxy` could not pass on `waf`'s answer, and answered `502`. diff --git a/docs/settings.md b/docs/settings.md index 46b30fb..0f07055 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -34,7 +34,7 @@ chooses between the two (see "The image's command" below). such as `:9000`. Leave the host out (see "Ports" in [`SPEC.md`](../SPEC.md)). - `SWW_WAF_INSTANCE_NAME` (default: the container's host name): the `instance` name in each line `waf` logs, and in each alert and metric, such as - `fsn1/waf`. Valid UTF-8. No control character while `SWW_WAF_ALERT_NTFY_URL` + `host1/waf`. Valid UTF-8. No control character while `SWW_WAF_ALERT_NTFY_URL` is set. 1 to 48 printable ASCII characters without a space while `SWW_WAF_LOG_REMOTE_URL` is set and `SWW_WAF_LOG_REMOTE_APP_NAME` is not. - `SWW_WAF_MODE` (default `enforce`): `enforce` or `observe`. @@ -159,8 +159,9 @@ cut off; the client is banned. - `SWW_WAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in its `.mmdb` form, for `SWW_WAF_LOOKUP_SOURCE=file`. - `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` (default `800`): how long a - request waits for its client's first answer from GeoJS, and how long a request - to GeoJS may take. A time in milliseconds; never `off`. + request waits for its client's first answer from GeoJS. A time in + milliseconds; never `off`. It does not cut off the request to GeoJS, whose + answer is kept when it comes later. - `SWW_WAF_ADD_LOOKUP_HEADERS` (default `true`): `true` has `rproxy` pass the app the client's AS number in `X-Client-ASN` and its country in `X-Client-Country`. The default is `false` while `SWW_WAF_LOOKUP_SOURCE` is @@ -251,10 +252,9 @@ tag that no request can get stops the start. A request can get only these: - `SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS` (default empty): path prefixes the Core Rule Set does not inspect. - `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` (default `off`): `off`, or how - much of a request body the Core Rule Set reads, a size of at most `64K`. A - question carries no more of a body than 64 KiB, and the Core Rule Set reads at - most 64 KiB less one byte of it, whatever the value (see "The answer" in - [`SPEC.md`](../SPEC.md)). + much of a request body the Core Rule Set reads, a size of at most `8K`. A + larger size would let one request keep it busy past `rproxy`'s timeout (see + "Answering in time" in [`SPEC.md`](../SPEC.md)). - `SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE` (default unset): a file of Core Rule Set exclusions, written as Coraza directives. It is loaded before the Core Rule Set's rules, so it holds only `SecAction` and `SecRule` lines that @@ -435,7 +435,9 @@ A few limits are fixed rather than settings: `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` is the longest a question waits on an outside source. Keep it well under `rproxy`'s -`SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, as the default is. No question waits for +`SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, as the default is. The trap paths, the rule +files and the Core Rule Set run during that wait and have 500 ms, so `waf` +answers within the longer of the two times. No question waits for `SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS`, since the DNS blocklists and AbuseIPDB are asked in the background (see "Answering in time" under "`waf`" in [`SPEC.md`](../SPEC.md)). @@ -453,10 +455,9 @@ start, and an invalid value stops the start. Any can be given as a file, as a for `waf`'s answer, to a question and to a report. A time in milliseconds; never `off`. - `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` (default `allow`): what is done - with a request when `waf` cannot be connected to, does not answer at all, or - does not answer within `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`. `allow` passes it - to the app, and `deny` answers `503`. It does not apply to an answer `waf` - gives, such as an error. + with a request when `waf` gives no answer in time, and while the circuit + breaker for asks to `waf` is open. `allow` passes it to the app, and `deny` + answers `503`. It does not apply to an answer `waf` gives, such as an error. - `SWW_RPROXY_MAX_REQUEST_SIZE` (default `100M`): the largest request body. A size or `off`. - `SWW_RPROXY_MAX_RESPONSE_SIZE` (default `1G`): the largest response body. A diff --git a/go.mod b/go.mod index 346ad93..5f24224 100644 --- a/go.mod +++ b/go.mod @@ -12,6 +12,7 @@ require ( github.com/oschwald/maxminddb-golang/v2 v2.7.0 github.com/prometheus/client_golang v1.24.1 github.com/spf13/viper v1.21.0 + golang.org/x/sys v0.48.0 ) require ( @@ -47,7 +48,6 @@ require ( go4.org/netipx v0.0.0-20231129151722-fdeea329fbba // indirect golang.org/x/net v0.60.0 // indirect golang.org/x/sync v0.23.0 // indirect - golang.org/x/sys v0.48.0 // indirect golang.org/x/text v0.42.0 // indirect google.golang.org/protobuf v1.36.11 // indirect rsc.io/binaryregexp v0.2.0 // indirect diff --git a/internal/alerts/alerts_test.go b/internal/alerts/alerts_test.go index ac09131..190eea9 100644 --- a/internal/alerts/alerts_test.go +++ b/internal/alerts/alerts_test.go @@ -39,7 +39,7 @@ const ( ntfyURL = "https://ntfy.example/sww-alerts" ntfyToken = "tk_0123456789abcdefghijklmnopq" // instance is the instance name every alert gives. - instance = "fsn1app1/blog" + instance = "host1/blog" // started is when each test starts, as an alert gives it, and // anHourOn an hour later. started = "2000-01-01T00:00:00Z" @@ -1160,8 +1160,8 @@ func TestEachDestinationIsSentOnlyTheAlertsWaitingForIt(t *testing.T) { synctest.Wait() wantBodies(t, standIns[alerts.DestinationSlack], - `{"text":"*fsn1app1/blog: file_error*\nsecond"}`, - `{"text":"*fsn1app1/blog: file_error*\nthird"}`) + `{"text":"*host1/blog: file_error*\nsecond"}`, + `{"text":"*host1/blog: file_error*\nthird"}`) wantBodies(t, standIns[alerts.DestinationNtfy], "fourth") // alerts.json then lists Slack and ntfy alone, with no alert waiting. @@ -1477,7 +1477,7 @@ func anAlertForEachEvent() []eventMessage { Reason: "requests per hour over the limit of 10000", Detail: map[string]any{"mode": "observe"}, }, - "fsn1app1/blog: ban", "default no_entry", + "host1/blog: ban", "default no_entry", "requests per hour over the limit of 10000\nclient: 203.0.113.9\n" + "netblock: 203.0.113.0/24\ncountry: DE\nmode: observe", }, @@ -1489,7 +1489,7 @@ func anAlertForEachEvent() []eventMessage { ), Country: "FR", Reason: "matched the rule env-file", }, - "fsn1app1/blog: permanent_ban", "high no_entry", + "host1/blog: permanent_ban", "high no_entry", "matched the rule env-file\nclient: 198.51.100.7\n" + "netblock: 198.51.100.7/32\ncountry: FR", }, @@ -1500,7 +1500,7 @@ func anAlertForEachEvent() []eventMessage { Reason: "refused by the Core Rule Set", Host: "app.example", }, - "fsn1app1/blog: core_rule_set_match", "default shield", + "host1/blog: core_rule_set_match", "default shield", "refused by the Core Rule Set\nhost: app.example\nclient: 192.0.2.1\n" + "netblock: 192.0.2.1/32", }, @@ -1509,7 +1509,7 @@ func anAlertForEachEvent() []eventMessage { Event: alerts.EventAnomaly, Netblock: netip.MustParsePrefix("192.0.2.0/24"), Reason: "requests per minute over the threshold of 5000", }, - "fsn1app1/blog: anomaly", "high chart_with_upwards_trend", + "host1/blog: anomaly", "high chart_with_upwards_trend", "requests per minute over the threshold of 5000\nnetblock: 192.0.2.0/24", }, { @@ -1518,7 +1518,7 @@ func anAlertForEachEvent() []eventMessage { Netblock: netip.MustParsePrefix("192.0.2.2/32"), Reason: "listed by a DNS blocklist", }, - "fsn1app1/blog: reputation_hit", "low label", + "host1/blog: reputation_hit", "low label", "listed by a DNS blocklist\nclient: 192.0.2.2\nnetblock: 192.0.2.2/32", }, { @@ -1526,7 +1526,7 @@ func anAlertForEachEvent() []eventMessage { Event: alerts.EventSourceFailure, Reason: "asking GeoJS failed", Detail: map[string]any{"source": "geojs"}, }, - "fsn1app1/blog: source_failure", "high warning", + "host1/blog: source_failure", "high warning", "asking GeoJS failed\nsource: geojs", }, { @@ -1537,7 +1537,7 @@ func anAlertForEachEvent() []eventMessage { "file": "/etc/sww/rules.d/50-app.rules", "error": "line 2: no action", }, }, - "fsn1app1/blog: file_error", "high warning", + "host1/blog: file_error", "high warning", "a rule file has an error, and the rules stay as they were\n" + "file: /etc/sww/rules.d/50-app.rules\nerror: line 2: no action", }, diff --git a/internal/config/waf.go b/internal/config/waf.go index cc166e3..513d253 100644 --- a/internal/config/waf.go +++ b/internal/config/waf.go @@ -365,8 +365,8 @@ var ( "it names the file to look clients up in", ) errDBPathUnused = errors.New("only file reads it") - errOver64K = errors.New( - "is more than 64K, the most waf gets of a body") + errOver8K = errors.New( + "is more than 8K, the most the Core Rule Set can read in time") errNotDurationAboveZero = errors.New( "is not a duration above zero, such as 1h or 7d") errNotNumberAboveZero = errors.New( @@ -1601,16 +1601,16 @@ func durationOrDays(value string) (time.Duration, error) { } // parseCoreRuleSetBodyInspectionSize reads the most of a request body the Core Rule Set -// reads: a size as parseSize reads it, or off, but at most 64K, since waf -// gets no more of a body than that. +// reads: a size as parseSize reads it, or off, but at most 8K, so that one +// request cannot keep the Core Rule Set busy past rproxy's timeout. func parseCoreRuleSetBodyInspectionSize(value string) (int64, error) { limit, err := parseSize(value) if err != nil { return 0, err } - if limit > 64*kibibyte { - return 0, fmt.Errorf("%q %w", value, errOver64K) + if limit > 8*kibibyte { + return 0, fmt.Errorf("%q %w", value, errOver8K) } return limit, nil diff --git a/internal/config/waf_test.go b/internal/config/waf_test.go index c90c0f2..f3ea741 100644 --- a/internal/config/waf_test.go +++ b/internal/config/waf_test.go @@ -218,7 +218,7 @@ const ( // instance is an SWW_WAF_INSTANCE_NAME that is a valid app name too, and // remoteURL an SWW_WAF_LOG_REMOTE_URL, for the tests that send the lines. const ( - instance = "fsn1app1/blog" + instance = "host1/blog" remoteURL = "syslog+udp://192.0.2.1:514" ) @@ -616,7 +616,7 @@ func TestCoreRuleSetSettings(t *testing.T) { coreRuleSetScoreThreshold: "10", coreRuleSetDisabledRules: "942100, 920350", coreRuleSetExemptPaths: "/api/, /static/", - coreRuleSetBodyInspectionSize: "32K", + coreRuleSetBodyInspectionSize: "4K", exclusionsFile: "/etc/sww/crs-exclusions.conf", }, config.WAF{ @@ -625,7 +625,7 @@ func TestCoreRuleSetSettings(t *testing.T) { CoreRuleSetScoreThreshold: 10, CoreRuleSetDisabledRules: []int{942100, 920350}, CoreRuleSetExemptPaths: []string{"/api/", "/static/"}, - CoreRuleSetBodyInspectionSize: 32 << 10, + CoreRuleSetBodyInspectionSize: 4 << 10, CoreRuleSetExclusionsFile: "/etc/sww/crs-exclusions.conf", }, }, @@ -762,15 +762,15 @@ func TestInvalidAppsStopsTheStart(t *testing.T) { } } -func TestCoreRuleSetBodyInspectionSizeOf64K(t *testing.T) { +func TestCoreRuleSetBodyInspectionSizeOf8K(t *testing.T) { t.Parallel() cfg := wafFromEnvironment( t, - environment{coreRuleSetBodyInspectionSize: "64K"}, + environment{coreRuleSetBodyInspectionSize: "8K"}, ) - if cfg.CoreRuleSetBodyInspectionSize != 64<<10 { - t.Errorf("64K read as %d", cfg.CoreRuleSetBodyInspectionSize) + if cfg.CoreRuleSetBodyInspectionSize != 8<<10 { + t.Errorf("8K read as %d", cfg.CoreRuleSetBodyInspectionSize) } } @@ -819,11 +819,13 @@ func TestInvalidCoreRuleSetSettingStopsTheStart(t *testing.T) { coreRuleSetBodyInspectionSize, "128KB", `"128KB" is not a size such as 512K, 100M or 5G, or off`, }, - {coreRuleSetBodyInspectionSize, "65K", `"65K" is more than 64K, the most waf ` + - `gets of a body`}, { - coreRuleSetBodyInspectionSize, "65537", - `"65537" is more than 64K, the most waf gets of a body`, + coreRuleSetBodyInspectionSize, "9K", + `"9K" is more than 8K, the most the Core Rule Set can read in time`, + }, + { + coreRuleSetBodyInspectionSize, "8193", + `"8193" is more than 8K, the most the Core Rule Set can read in time`, }, } { t.Run(tc.name+"="+tc.value, func(t *testing.T) { @@ -845,11 +847,11 @@ func TestInstanceNameAndLoggedHeadersAsSet(t *testing.T) { t.Parallel() cfg := wafFromEnvironment(t, environment{ - instanceName: "fsn1app1/blog", + instanceName: "host1/blog", logRequestHeaders: " Accept , X-Custom", }) - if cfg.InstanceName != "fsn1app1/blog" || + if cfg.InstanceName != "host1/blog" || !slices.Equal( cfg.LogRequestHeaders, []string{"accept", "x-custom"}, @@ -1022,14 +1024,14 @@ func TestInstanceNameNotAnAppNameStopsTheStartOnlyWhileSending( ) { t.Parallel() - const spaced = "fsn1 app1" + const spaced = "host1 blog" sending := environment{logRemoteURL: remoteURL, instanceName: spaced} _, err := config.WAFFromEnvironment(sending.viper()) want := logRemoteAppName + `: is unset, and ` + instanceName + - ` "fsn1 app1", its default, is not 1 to 48 printable ASCII characters ` + + ` "host1 blog", its default, is not 1 to 48 printable ASCII characters ` + `without a space, such as blog` if err == nil || err.Error() != want { t.Errorf("error %v, want %s", err, want) @@ -1331,13 +1333,13 @@ func TestInstanceNameWithAControlCharacterStopsTheStartOnlyWithNtfySet( ) { t.Parallel() - const name = "fsn1app1\r" + const name = "host1\r" _, err := config.WAFFromEnvironment(environment{ instanceName: name, alertNtfyURL: "https://ntfy.example/sww-alerts", }.viper()) - want := instanceName + `: "fsn1app1\r" holds a control character, such as the ` + + want := instanceName + `: "host1\r" holds a control character, such as the ` + `carriage return of a Windows line end, and is sent to ntfy in a header ` + `while ` + alertNtfyURL + ` is set` if err == nil || err.Error() != want { diff --git a/internal/coreruleset/coreruleset.go b/internal/coreruleset/coreruleset.go index 4a3b9ed..83ae46a 100644 --- a/internal/coreruleset/coreruleset.go +++ b/internal/coreruleset/coreruleset.go @@ -481,12 +481,21 @@ func (c *CoreRuleSet) Inspect( return Result{RuleIDs: ids, Score: score(tx)}, read, err } +// mostJSONRead is the largest JSON body the Core Rule Set reads, even while +// SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is larger. Coraza names each +// value in a JSON body by the whole path to it, so what it gives the Core +// Rule Set to read can outgrow the body: up to 8 times its size, and up to +// 4 KiB for a small body. 1 KiB of JSON thus gives at most 8 KiB, no more +// than the largest body of another kind. +const mostJSONRead = 1 << 10 + // readBody reads body, the body of the request in tx, which has run on // the request's headers, while SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is // set and the body is of a kind the Core Rule Set reads: form data and // multipart, of which it reads the first c.bodyInspectionSize bytes, and // JSON and XML, which it reads only when they are no longer than that, -// since they cannot be read in part. +// since they cannot be read in part. The Core Rule Set reads no JSON body +// longer than mostJSONRead either. // readBody reads one byte past c.bodyInspectionSize, to tell which they // are, gives the Core Rule Set what it reads, and returns what it read and // the error that ended the reading early, if one did. @@ -499,6 +508,8 @@ func (c *CoreRuleSet) readBody( } inPart := false + // largest is the largest JSON or XML body the Core Rule Set reads. + largest := c.bodyInspectionSize // How the body is read is a variable of the transaction, which only // Coraza's interface for plugins reads. state := tx.(plugintypes.TransactionState) //nolint:forcetypeassert // every one is @@ -506,14 +517,16 @@ func (c *CoreRuleSet) readBody( switch state.Variables().RequestBodyProcessor().Get() { case "URLENCODED", "MULTIPART": inPart = true - case "JSON", "XML": + case "JSON": + largest = min(largest, mostJSONRead) + case "XML": default: return nil, nil } read, err := io.ReadAll(io.LimitReader(body, c.bodyInspectionSize+1)) - if inPart || (err == nil && int64(len(read)) <= c.bodyInspectionSize) { + if inPart || (err == nil && int64(len(read)) <= largest) { // Coraza holds what it reads of the body in memory, up to the // limit, so this cannot fail. _, _, _ = tx.WriteRequestBody(read) diff --git a/internal/coreruleset/coreruleset_test.go b/internal/coreruleset/coreruleset_test.go index 7db965d..52a3146 100644 --- a/internal/coreruleset/coreruleset_test.go +++ b/internal/coreruleset/coreruleset_test.go @@ -390,22 +390,15 @@ func TestMoreParametersThanCorazaKeepsIsAMatch(t *testing.T) { matched(900300), ) - // So it is with the fields of a form data or JSON body. + // So it is with the fields of a form data body. A JSON body the Core + // Rule Set reads, of at most mostJSONRead bytes, cannot hold that many. crs = readingBodies(t) - for _, tc := range []struct{ header, body string }{ - {formData, strings.Repeat("a=1&", 999) + attack}, - {jsonBody, `{"a":[` + strings.Repeat("1,", 998) + `1],"id":"` + injection + `"}`}, - } { - wantBody(t, crs, post(tc.header), tc.body, matched(942100), tc.body) - } + body := strings.Repeat("a=1&", 999) + attack + wantBody(t, crs, post(formData), body, matched(942100), body) - for _, tc := range []struct{ header, body string }{ - {formData, strings.Repeat("a=1&", 1000) + attack}, - {jsonBody, `{"a":[` + strings.Repeat("1,", 999) + `1],"id":"` + injection + `"}`}, - } { - wantBody(t, crs, post(tc.header), tc.body, matched(900300), tc.body) - } + body = strings.Repeat("a=1&", 1000) + attack + wantBody(t, crs, post(formData), body, matched(900300), body) } func TestRedirectURIMayNameALocalAddress(t *testing.T) { @@ -1512,13 +1505,15 @@ const ( wpTags = "/wp-json/wp/v2/tags" ) -// readingBodiesUpTo64K returns the Core Rule Set as sww runs it by default, +// readingBodiesUpTo8K returns the Core Rule Set as sww runs it by default, // but reading bodies up to the largest SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE. -func readingBodiesUpTo64K(t *testing.T) *coreruleset.CoreRuleSet { +func readingBodiesUpTo8K(t *testing.T) *coreruleset.CoreRuleSet { t.Helper() return shared(t, coreruleset.Params{ - ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, BodyInspectionSize: 64<<10 - 1, + ParanoiaLevel: 1, + DisabledRules: defaultDisabledRules, + BodyInspectionSize: mostBodyRead, }) } @@ -1582,7 +1577,7 @@ func TestWordPressAttacksAreRefused(t *testing.T) { const threshold = 5 // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default - defaults, bodies := atDefaults(t), readingBodiesUpTo64K(t) + defaults, bodies := atDefaults(t), readingBodiesUpTo8K(t) for _, attack := range []string{ sqlInjection, @@ -1693,7 +1688,7 @@ func wordpressRESTAddresses(route string) []string { func TestWordPressExclusionsFollowEveryRESTAddress(t *testing.T) { t.Parallel() - crs := readingBodiesUpTo64K(t) + crs := readingBodiesUpTo8K(t) content := `{"content":"` + wordpressBlock + `"}` for _, route := range []string{ @@ -1919,7 +1914,7 @@ func TestWordPressCommandAfterASeparatorIsRefusedOnSearchesAndNames( wantResult(t, crs, get(target+command), matched(932235)) } - bodies := readingBodiesUpTo64K(t) + bodies := readingBodiesUpTo8K(t) for _, p := range wordpressBodyFields(command) { r := request{http.MethodPost, p.target, []string{p.contentType}} @@ -2143,7 +2138,7 @@ func TestWordPressBodyFieldsAreCheckedInTheQuery(t *testing.T) { func TestWordPressBodyNamesAndAddressesAreExcludedOnlyThere(t *testing.T) { t.Parallel() - crs := readingBodiesUpTo64K(t) + crs := readingBodiesUpTo8K(t) for _, v := range wordpressWords() { for _, field := range []string{"title", "excerpt", "name"} { @@ -2262,7 +2257,7 @@ func TestWordPressExclusionsAreNarrow(t *testing.T) { length = "Content-Length: 2" ) - crs := readingBodiesUpTo64K(t) + crs := readingBodiesUpTo8K(t) saving := request{ http.MethodPost, wpPost, @@ -2473,19 +2468,35 @@ func TestFormDataAndMultipartAreReadUpToTheLimit(t *testing.T) { func TestJSONAndXMLAreReadOnlyWhenNoLargerThanTheLimit(t *testing.T) { t.Parallel() - crs := readingBodies(t) + // The limit is the body inspection size, but at most mostJSONRead for + // JSON. smallSize is a body inspection size under mostJSONRead. + const smallSize = 512 - for _, tc := range []struct{ header, head, tail string }{ - {jsonBody, `{"q":"` + injection + `","pad":"`, `"}`}, - {xmlBody, "" + injection + "", ""}, + bodies, smallBodies := readingBodies(t), shared(t, coreruleset.Params{ + ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, + BodyInspectionSize: smallSize, + }) + jsonHead, jsonTail := `{"q":"`+injection+`","pad":"`, `"}` + + for _, tc := range []struct { + crs *coreruleset.CoreRuleSet + header, head, tail string + limit int + }{ + {bodies, jsonBody, jsonHead, jsonTail, mostJSONRead}, + {smallBodies, jsonBody, jsonHead, jsonTail, smallSize}, + { + bodies, xmlBody, "" + injection + "", "", + bodyInspectionSize, + }, } { - fits := padded(tc.head, tc.tail, bodyInspectionSize) - wantBody(t, crs, post(tc.header), fits, matched(942100), fits) + fits := padded(tc.head, tc.tail, tc.limit) + wantBody(t, tc.crs, post(tc.header), fits, matched(942100), fits) - larger := padded(tc.head, tc.tail, bodyInspectionSize+1) + larger := padded(tc.head, tc.tail, tc.limit+1) wantBody( t, - crs, + tc.crs, post(tc.header), larger, coreruleset.Result{}, diff --git a/internal/coreruleset/inspectiontime_test.go b/internal/coreruleset/inspectiontime_test.go new file mode 100644 index 0000000..c6a688a --- /dev/null +++ b/internal/coreruleset/inspectiontime_test.go @@ -0,0 +1,437 @@ +package coreruleset_test + +import ( + "maps" + "net/http" + "net/http/httptest" + "net/netip" + "regexp" + "runtime" + "slices" + "strconv" + "strings" + "testing" + "time" + + "golang.org/x/sys/unix" + + "sneak.berlin/go/smallwebwaf/internal/coreruleset" +) + +// The bounds on what a visitor can send, as SPEC.md gives them. +const ( + // lineLimit is the longest request line or header line. + lineLimit = 8 << 10 + // headerLimit is the most bytes rproxy reads of a request line and + // headers together. + headerLimit = 32 << 10 + // mostCookies and mostHeaderFields are waf's limits on cookies and header + // fields. + mostCookies = 50 + mostHeaderFields = 100 + // mostBodyRead is the largest SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE. + mostBodyRead = 8 << 10 + // mostJSONRead is the largest JSON body the Core Rule Set reads. + mostJSONRead = 1 << 10 +) + +// waf refuses a request the Core Rule Set has not inspected within 500 +// milliseconds, but Coraza cannot stop an inspection part-way, so it runs to +// its end with nobody waiting for it. The test checks that an inspection ends +// within about 1 second, so that a visitor cannot keep a processor busy for +// longer with one request. +// +// A busy host slows all work, so the test does not compare an inspection with +// 1 second. It compares it with referenceTime, timed on the same thread just +// before and after. inspectionTimeLimitMultiple is how many times +// referenceTime an inspection may take. +// +// On the host that runs our checks, referenceTime took at least 78 +// milliseconds, so 13 times it is about 1 second. With that host fully busy, +// no try of the slowest request took more than 9.5 times referenceTime. +const inspectionTimeLimitMultiple = 13 + +// referencePattern is what referenceTime matches. Like the Core Rule Set's +// rules, it is a regular expression with alternatives and repeats. +var referencePattern = regexp.MustCompile( + `(?i)(union|select|insert|delete)\s+\w+\s+(from|into)\s+\w+[0-9]+;`, +) + +// referenceTime returns the processor time the calling thread takes to do a +// fixed amount of work unrelated to the Core Rule Set, so that a change to the +// rules does not change it. referencePattern matches nowhere in the text, so +// each match reads all of it. +func referenceTime(t *testing.T) time.Duration { + t.Helper() + + referenceText := text(4 << 10) + began := threadCPUTime(t) + + for range 200 { + if referencePattern.MatchString(referenceText) { + t.Fatal("the reference pattern matched") + } + } + + return threadCPUTime(t) - began +} + +// text returns n bytes of letters, digits and spaces, which the Core Rule Set +// takes longer over than one letter repeated. +func text(n int) string { + const words = "select 0x2f from tbl_1 where name like abc_9 " + + return strings.Repeat(words, n/len(words)+1)[:n] +} + +// visitorRequest is a request a visitor can send, at the most of one or more +// of the bounds above. +type visitorRequest struct { + name string + target string + // header holds the headers besides Host. + header http.Header + // contentType is the body's type, "" for no body. + contentType string + body string +} + +// longestTarget returns a target whose request line, "POST +// HTTP/1.1", is lineLimit bytes long. +func longestTarget() string { + const around = len("POST /?q= HTTP/1.1") + + return "/?q=" + strings.ReplaceAll(text(lineLimit-around), " ", "+") +} + +// cookieLine returns a Cookie value whose line, "Cookie: ", is +// lineLimit bytes long, made of count cookies. +func cookieLine(count int) string { + const linePrefix = len("Cookie: ") + + cookies := make([]string, count) + for i := range cookies { + cookies[i] = "c" + strconv.Itoa(i) + "=" + } + + // Each cookie is followed by "; ", but for the last. + room := lineLimit - linePrefix - len(strings.Join(cookies, "; ")) + for i := range cookies { + share := room / count + if i == 0 { + share += room % count + } + + cookies[i] += strings.ReplaceAll(text(share), " ", "_") + } + + return strings.Join(cookies, "; ") +} + +// headerValue returns a value whose line, ": ", is n bytes long. +func headerValue(name string, n int) string { + return text(n - len(name) - len(": ")) +} + +// formFields returns a form body of mostBodyRead bytes holding count fields. +func formFields(count int) string { + fields := make([]string, count) + for i := range fields { + fields[i] = "f" + strconv.Itoa(i) + "=" + } + + room := mostBodyRead - len(strings.Join(fields, "&")) + for i := range fields { + share := room / count + if i == 0 { + share += room % count + } + + fields[i] += strings.ReplaceAll(text(share), " ", "+") + } + + return strings.Join(fields, "&") +} + +// multipartBody returns a multipart body of mostBodyRead bytes, with the +// boundary "b", holding fields of about size bytes each. +func multipartBody(size int) string { + var body strings.Builder + + const end = "--b--\r\n" + + for i := 0; ; i++ { + head := "--b\r\nContent-Disposition: form-data; name=\"f" + + strconv.Itoa(i) + "\"\r\n\r\n" + + room := mostBodyRead - body.Len() - len( + end, + ) - len( + head, + ) - len( + "\r\n", + ) + if room <= 0 { + break + } + + body.WriteString(head + text(min(size, room)) + "\r\n") + } + + body.WriteString(end) + + return body.String() +} + +// jsonArrayOfMostElements returns a JSON array of at most mostJSONRead bytes +// holding as many elements as fit, each the number 1. +func jsonArrayOfMostElements() string { + return "[1" + strings.Repeat( + ",1", + (mostJSONRead-len("[1]"))/len(",1"), + ) + "]" +} + +// jsonOfMostElementsUnderAKey returns a JSON body of mostJSONRead bytes +// holding as many elements as fit, each the number 1, in an array under a +// key of 6 bytes. Coraza names each element "json..", so a +// longer key makes longer names. 6 bytes is the longest key that keeps the +// names and values of this body within the 8 KiB Coraza allows. +func jsonOfMostElementsUnderAKey() string { + head := `{"` + text(6) + `":[1` + + return head + strings.Repeat( + ",1", + (mostJSONRead-len(head+"]}"))/len(",1"), + ) + "]}" +} + +// The kinds of body the Core Rule Set reads, and the header that names a +// request's cookies. +const ( + formType = "application/x-www-form-urlencoded" + jsonType = "application/json" + multipartType = "multipart/form-data; boundary=b" + cookieHeader = "Cookie" +) + +// fullLine is the length of each of the request line and the three or four +// header lines that together fill rproxy's limit, with room for the line +// endings and the other headers. +const fullLine = lineLimit - 64 + +// worstCaseRequests are, for each kind of thing a visitor sends, the requests +// that keep the Core Rule Set busiest, within the bounds above. The last +// holds the most of everything at once: Cookie lines keep it busier than +// other header lines, and the JSON body of the most elements under a key +// keeps it busier than the other bodies. +func worstCaseRequests() []visitorRequest { + fullCookieLine := cookieLine(1)[:fullLine-len("Cookie: ")] + + return append( + append(worstCaseHeaders(), worstCaseBodies()...), + visitorRequest{ + name: "the longest request line, three Cookie lines and body", + target: longestTarget()[:fullLine-len("POST HTTP/1.1")], + header: http.Header{cookieHeader: { + fullCookieLine, fullCookieLine, fullCookieLine, + }}, + contentType: jsonType, body: jsonOfMostElementsUnderAKey(), + }, + ) +} + +// worstCaseHeaders are the requests of worstCaseRequests with no body. +func worstCaseHeaders() []visitorRequest { + headerFields := http.Header{} + + for i := range mostHeaderFields - 1 { + name := "X-Field-" + strconv.Itoa(i) + headerFields.Set( + name, + headerValue(name, headerLimit/mostHeaderFields), + ) + } + + return []visitorRequest{ + {name: "the longest request line", target: longestTarget()}, + { + name: "the longest request line, as query parameters", + target: "/?" + strings.Repeat( + "a=1&", + (lineLimit-len("POST /? HTTP/1.1"))/len("a=1&"), + ), + }, + { + name: "the longest Cookie line, as one cookie", + target: "/", + header: http.Header{cookieHeader: {cookieLine(1)}}, + }, + { + name: "the longest Cookie line, as the most cookies", + target: "/", + header: http.Header{cookieHeader: {cookieLine(mostCookies)}}, + }, + {name: "the most header fields", target: "/", header: headerFields}, + { + name: "the longest header lines", + target: "/", + header: http.Header{ + "User-Agent": {headerValue("User-Agent", fullLine)}, + "Referer": {headerValue("Referer", fullLine)}, + "Accept": {headerValue("Accept", fullLine)}, + "X-Long": {headerValue("X-Long", fullLine)}, + }, + }, + } +} + +// worstCaseBodies are the requests of worstCaseRequests whose body holds +// the most the Core Rule Set reads. +func worstCaseBodies() []visitorRequest { + const nested = (mostJSONRead - len(`{"a":1}`)) / len(`{"a":}`) + + return []visitorRequest{ + { + name: "a form body of one field", target: "/", + contentType: formType, body: formFields(1), + }, + { + name: "a form body of the most fields Coraza keeps", target: "/", + contentType: formType, body: formFields(1000), + }, + { + name: "a JSON body of one string", target: "/", + contentType: jsonType, + body: `{"a":"` + text( + mostJSONRead-len(`{"a":""}`), + ) + `"}`, + }, + { + name: "a JSON body of the most elements", target: "/", + contentType: jsonType, body: jsonArrayOfMostElements(), + }, + { + name: "a JSON body of the most elements under a key", target: "/", + contentType: jsonType, body: jsonOfMostElementsUnderAKey(), + }, + { + name: "a JSON body nested deep", target: "/", + contentType: jsonType, + body: strings.Repeat(`{"a":`, nested) + "1" + + strings.Repeat("}", nested), + }, + { + name: "an XML body of many elements", target: "/", + contentType: "application/xml", + body: "" + strings.Repeat( + "x", + (mostBodyRead-len(""))/len("x"), + ) + "", + }, + { + name: "a multipart body of one field", target: "/", + contentType: multipartType, body: multipartBody(mostBodyRead), + }, + { + name: "a multipart body of many fields", target: "/", + contentType: multipartType, body: multipartBody(1), + }, + } +} + +// threadCPUTime returns the processor time the calling thread has used. +func threadCPUTime(t *testing.T) time.Duration { + t.Helper() + + var now unix.Timespec + + err := unix.ClockGettime(unix.CLOCK_THREAD_CPUTIME_ID, &now) + if err != nil { + t.Fatalf("read the thread's processor time: %v", err) + } + + return time.Duration(now.Nano()) +} + +// inspectionTime returns the processor time crs takes to inspect tc, and the +// mean of referenceTime just before and after. It fails t unless crs reads all +// of tc's body and Coraza parses it. +// +// All of it runs on this goroutine, kept on one thread, so the thread's +// processor time measures it. Time other work holds the processor does not +// count. +func inspectionTime( + t *testing.T, crs *coreruleset.CoreRuleSet, tc visitorRequest, +) (time.Duration, time.Duration) { + t.Helper() + + req := httptest.NewRequestWithContext( + t.Context(), http.MethodPost, tc.target, nil, + ) + maps.Copy(req.Header, tc.header) + + if tc.contentType != "" { + req.Header.Set("Content-Type", tc.contentType) + } + + runtime.LockOSThread() + + defer runtime.UnlockOSThread() + + before := referenceTime(t) + + began := threadCPUTime(t) + result, read, _ := crs.Inspect( + req, netip.MustParseAddr("192.0.2.1"), strings.NewReader(tc.body), + ) + took := threadCPUTime(t) - began + + reference := (before + referenceTime(t)) / 2 + + if string(read) != tc.body { + t.Errorf("read %d bytes of %d", len(read), len(tc.body)) + } + + // Rule 900440 matches a body Coraza could not parse. + if slices.Contains(result.RuleIDs, 900440) { + t.Errorf("Coraza could not parse the body") + } + + return took, reference +} + +// TestTheCoreRuleSetInspectsEachWorstCaseRequestInTime runs one at a time, so +// that the tests running in parallel do not slow it. Other work on the host +// can still slow one try more than the reference work around it, so a request +// fails only when three tries in a row take too long. +// +//nolint:paralleltest // one at a time, as above +func TestTheCoreRuleSetInspectsEachWorstCaseRequestInTime(t *testing.T) { + const tries = 3 + + // The settings that make the Core Rule Set slowest: the highest paranoia + // level, no rule switched off, and the most of a body read. + crs := shared(t, coreruleset.Params{ + ParanoiaLevel: 4, BodyInspectionSize: mostBodyRead, + }) + + for _, tc := range worstCaseRequests() { + t.Run(tc.name, func(t *testing.T) { + var took, reference time.Duration + + for range tries { + took, reference = inspectionTime(t, crs, tc) + if took <= inspectionTimeLimitMultiple*reference { + return + } + } + + t.Errorf( + "took %s on the last of %d tries, over %d times the %s "+ + "of the reference work", + took, tries, inspectionTimeLimitMultiple, reference, + ) + }) + } +} diff --git a/internal/lookup/export_test.go b/internal/geojs/export_test.go similarity index 93% rename from internal/lookup/export_test.go rename to internal/geojs/export_test.go index 3276a6c..360ff2d 100644 --- a/internal/lookup/export_test.go +++ b/internal/geojs/export_test.go @@ -1,4 +1,4 @@ -package lookup +package geojs import "net/http" diff --git a/internal/lookup/lookup.go b/internal/geojs/geojs.go similarity index 75% rename from internal/lookup/lookup.go rename to internal/geojs/geojs.go index d5d5db0..962bc51 100644 --- a/internal/lookup/lookup.go +++ b/internal/geojs/geojs.go @@ -1,9 +1,9 @@ -// Package lookup looks up each client's AS number and country, through -// the GeoJS web service or in the lookup database, the IPinfo Lite file -// SWW_WAF_LOOKUP_DB_PATH names. GeoJS's answers are kept in memory, for at -// most 100,000 clients and for 7 days each, and are written to -// lookups.json and read from it by the state package. -package lookup +// Package geojs looks up clients' AS numbers and countries through the GeoJS +// web service. Its answers are kept in memory, for at most 100,000 clients and +// for 7 days each. A caller waits for an answer as long as it says. A request to +// GeoJS goes on after its callers stop waiting, and its answer is kept. The +// package imports no other package of this repository. +package geojs import ( "context" @@ -11,7 +11,6 @@ import ( "errors" "fmt" "io" - "log/slog" "net/http" "net/netip" "slices" @@ -21,8 +20,6 @@ import ( "time" "github.com/hashicorp/golang-lru/v2/simplelru" - "sneak.berlin/go/smallwebwaf/internal/alerts" - "sneak.berlin/go/smallwebwaf/internal/metrics" ) // URL is GeoJS's endpoint for an address's place and network. Asked about @@ -43,6 +40,9 @@ const ( maxWaiting = 10000 // maxPerRequest is how many addresses one request to GeoJS asks about. maxPerRequest = 200 + // requestTimeout is how long a request to GeoJS may take before it is + // abandoned, which is a failure. It is apart from how long callers wait. + requestTimeout = 10 * time.Second // unknownASN is the AS number GeoJS gives when it knows none. unknownASN = 64512 // After a failure GeoJS is not asked again for a second, and for @@ -64,39 +64,32 @@ var ( type Params struct { // URL is where GeoJS is asked, normally URL. URL string - // Timeout is how long a request waits for its client's first answer, - // and how long a request to GeoJS may take before it is abandoned - // (SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS). - Timeout time.Duration - // Wait is true when a setting needs each request's answer before the - // request goes on. Otherwise no request waits for one. - Wait bool // Answered, unless nil, is given each answer GeoJS gives, once it is // kept. Answered func(Answer) // Now tells the time, normally time.Now in UTC. Now func() time.Time - // ProcessLog receives GeoJS's failures. - ProcessLog *slog.Logger - // Metrics count the requests to GeoJS, those that failed, and the - // clients that go without an answer. - Metrics *metrics.Metrics - // Alerts receive a source_failure alert each time GeoJS fails. - Alerts *alerts.Queue + // Asked is called for each request to GeoJS. + Asked func() + // Failed is called for each failure of GeoJS, with what went wrong and + // how long GeoJS is left alone after it. The error never names the + // addresses asked about. + Failed func(err error, askingAgainIn time.Duration) + // WentWithoutAnswer is called each time a caller that waits for an + // answer goes on without one. + WentWithoutAnswer func() } // GeoJS looks up clients' AS numbers and countries through GeoJS. At most // one request to GeoJS is under way at a time, and it asks about every // client waiting, up to maxPerRequest. It is safe for concurrent use. type GeoJS struct { - url string - timeout time.Duration - wait bool - answered func(Answer) - now func() time.Time - processLog *slog.Logger - metrics *metrics.Metrics - alerts *alerts.Queue + url string + answered func(Answer) + now func() time.Time + asked func() + failed func(err error, askingAgainIn time.Duration) + wentWithoutAnswer func() // httpClient follows no redirect, so that visitors' addresses go to // GeoJS alone: a redirect is a failure. httpClient *http.Client @@ -114,12 +107,10 @@ type GeoJS struct { retryAt time.Time } -// Answer is what GeoJS or the lookup database said about a client: its AS -// number, such as AS64496, and the AS's name, both "" when the source knows -// no AS number for it; its country, "" when the source cannot place it; -// when the source said so; and, for GeoJS's answers, which lookups.json -// holds, when the answer was last used. The zero Answer is that of a -// client with no answer. +// Answer is what GeoJS said about a client: its AS number, such as AS64496, +// and the AS's name, both "" when GeoJS knows no AS number for it; its +// country, "" when GeoJS cannot place it; when GeoJS said so; and when the +// answer was last used. The zero Answer is that of a client with no answer. // //nolint:tagliatelle // the state files use snake_case, as the request log does type Answer struct { @@ -137,7 +128,7 @@ type wait struct { // replaced each time GeoJS fails before then. asked chan struct{} // late is true once the client has gone without an answer, for a - // whole timeout or because GeoJS failed: its requests no longer wait. + // whole wait or because GeoJS failed: its callers no longer wait. late bool } @@ -149,14 +140,12 @@ func New(params Params) *GeoJS { } return &GeoJS{ - url: params.URL, - timeout: params.Timeout, - wait: params.Wait, - answered: params.Answered, - now: params.Now, - processLog: params.ProcessLog, - metrics: params.Metrics, - alerts: params.Alerts, + url: params.URL, + answered: params.Answered, + now: params.Now, + asked: params.Asked, + failed: params.Failed, + wentWithoutAnswer: params.WentWithoutAnswer, httpClient: &http.Client{ CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse @@ -170,21 +159,22 @@ func New(params Params) *GeoJS { // LookUp returns the answer GeoJS gave about client, with its country as // a two-letter code in capitals, or the zero Answer when there is none // yet. An answer is kept for 7 days. Without one, the client is asked -// about in the background, and, while Wait is set, the request waits up -// to Timeout for the answer, unless the client has gone without one -// before. ctx is the context of the client's request, and ends the wait -// when it ends. +// about in the background, and LookUp waits up to wait for the answer, +// unless the client has gone without one before. The wait ends early when +// ctx ends. Neither ends the request to GeoJS. // // GeoJS is asked about the client's first address, which is the client's // own address for IPv4, and an address in the same place for an IPv6 // netblock. -func (g *GeoJS) LookUp(ctx context.Context, client netip.Prefix) Answer { - answer, asked := g.answerOrWait(ctx, client, g.wait) +func (g *GeoJS) LookUp( + ctx context.Context, client netip.Prefix, wait time.Duration, +) Answer { + answer, asked := g.answerOrWait(ctx, client, true) if asked == nil { return answer } - timer := time.NewTimer(g.timeout) + timer := time.NewTimer(wait) defer timer.Stop() select { @@ -198,7 +188,7 @@ func (g *GeoJS) LookUp(ctx context.Context, client netip.Prefix) Answer { answer, found := g.kept(client) if !found { - g.metrics.RequestsWithoutGeoJSAnswer.Inc() + g.wentWithoutAnswer() } w, waiting := g.waiting[client] @@ -209,8 +199,8 @@ func (g *GeoJS) LookUp(ctx context.Context, client netip.Prefix) Answer { return answer } -// LookUpWithoutWaiting is LookUp for a request that does not need the answer, -// whatever Wait says: it returns the zero Answer at once when none is kept. +// LookUpWithoutWaiting is LookUp for a caller that does not need the answer: +// it returns the zero Answer at once when none is kept. func (g *GeoJS) LookUpWithoutWaiting( ctx context.Context, client netip.Prefix, @@ -228,8 +218,7 @@ func (g *GeoJS) Kept(client netip.Prefix) (Answer, bool) { return g.kept(client) } -// Snapshot returns every answer kept, sorted by client, as lookups.json -// lists them. +// Snapshot returns every answer kept, sorted by client. func (g *GeoJS) Snapshot() []Answer { g.mu.Lock() @@ -247,9 +236,9 @@ func (g *GeoJS) Snapshot() []Answer { return answers } -// Load keeps answers read from lookups.json, in place of the answers it -// keeps, in the order they were last used, so that the one used longest -// ago is dropped first. Answers GeoJS gave keepFor ago or more are +// Load keeps answers, such as those of an earlier Snapshot, in place of the +// answers it keeps, in the order they were last used, so that the one used +// longest ago is dropped first. Answers GeoJS gave keepFor ago or more are // dropped. func (g *GeoJS) Load(answers []Answer) { answers = slices.Clone(answers) @@ -295,11 +284,11 @@ func (g *GeoJS) answerOrWait( g.ask(ctx) if !mustWait { - return Answer{}, nil // the answer is not needed before the request goes on + return Answer{}, nil // the answer is not needed before the caller goes on } if w == nil { - g.metrics.RequestsWithoutGeoJSAnswer.Inc() + g.wentWithoutAnswer() return Answer{}, nil // too many clients wait already } @@ -310,7 +299,7 @@ func (g *GeoJS) answerOrWait( } if w.late { - g.metrics.RequestsWithoutGeoJSAnswer.Inc() + g.wentWithoutAnswer() return Answer{}, nil } @@ -335,8 +324,8 @@ func (g *GeoJS) kept(client netip.Prefix) (Answer, bool) { // ask starts asking GeoJS about the waiting clients, unless a request to // it is under way or it is left alone after a failure. The requests to -// GeoJS are for every client waiting, so they go on when the client's -// request whose ctx is given ends. +// GeoJS are for every client waiting, so they go on when the caller whose +// ctx is given stops waiting. func (g *GeoJS) ask(ctx context.Context) { if g.asking || g.now().Before(g.retryAt) { return @@ -441,8 +430,6 @@ func (g *GeoJS) keep( } if err != nil { - g.metrics.GeoJSFailures.Inc() - g.retryDelay = min( max(retryDelayFactor*g.retryDelay, firstRetryDelay), maxRetryDelay, @@ -457,16 +444,7 @@ func (g *GeoJS) keep( w.late = true } - g.processLog.Warn("asking GeoJS failed", - "error", err.Error(), "asking_again_in", g.retryDelay.String()) - g.alerts.Raise(alerts.Alert{ - Event: alerts.EventSourceFailure, - Reason: "asking GeoJS failed", - Detail: map[string]any{ - "source": "geojs", "error": err.Error(), - "asking_again_in": g.retryDelay.String(), - }, - }) + g.failed(err, g.retryDelay) return kept, false } @@ -479,7 +457,8 @@ func (g *GeoJS) keep( // request asks GeoJS about clients in one request, and returns the answer // for each address GeoJS's answer names: its AS number and the AS's name, // both "" for the AS number 64512, which GeoJS gives when it knows none, -// and its country, in capitals. +// and its country, in capitals. The request is abandoned after +// requestTimeout. func (g *GeoJS) request( ctx context.Context, clients []netip.Prefix, ) (map[netip.Addr]Answer, error) { @@ -489,7 +468,7 @@ func (g *GeoJS) request( addrs = append(addrs, client.Addr().String()) } - ctx, cancel := context.WithTimeout(ctx, g.timeout) + ctx, cancel := context.WithTimeout(ctx, requestTimeout) defer cancel() req, err := http.NewRequestWithContext( @@ -504,7 +483,7 @@ func (g *GeoJS) request( req.URL.RawQuery = "ip=" + strings.Join(addrs, ",") - g.metrics.GeoJSRequests.Inc() + g.asked() res, err := g.httpClient.Do(req) if err != nil { diff --git a/internal/lookup/lookup_test.go b/internal/geojs/geojs_test.go similarity index 60% rename from internal/lookup/lookup_test.go rename to internal/geojs/geojs_test.go index 4f67d02..a793f6b 100644 --- a/internal/lookup/lookup_test.go +++ b/internal/geojs/geojs_test.go @@ -1,13 +1,11 @@ -package lookup_test +package geojs_test import ( + "context" "encoding/json" - "log/slog" "net/http" "net/http/httptest" "net/netip" - "net/url" - "reflect" "slices" "strings" "sync" @@ -15,10 +13,7 @@ import ( "testing/synctest" "time" - "github.com/prometheus/client_golang/prometheus/testutil" - "sneak.berlin/go/smallwebwaf/internal/alerts" - "sneak.berlin/go/smallwebwaf/internal/lookup" - "sneak.berlin/go/smallwebwaf/internal/metrics" + "sneak.berlin/go/smallwebwaf/internal/geojs" ) const ( @@ -34,8 +29,14 @@ const ( // leftOut is the address it leaves out of its answer when // answeringWithoutLeftOut. leftOut = "203.0.113.7" - // timeout is how long a new client waits for its answer. - timeout = time.Second + // wait is how long a lookup waits for a new client's answer. + wait = time.Second + // requestTimeout is how long a request to GeoJS may take before it is + // abandoned. + requestTimeout = 10 * time.Second + // lateAnswer is how long the stand-in takes to answer when + // answeringLate. + lateAnswer = 5 * time.Second // week is how long an answer is kept. week = 7 * 24 * time.Hour ) @@ -51,24 +52,24 @@ func TestKeptAnswerIsUsedFor7DaysThenAskedAgain(t *testing.T) { t.Parallel() synctest.Test(t, func(t *testing.T) { - geojs, clock, g := start() + service, clock, g := start() placed := netip.MustParsePrefix("203.0.113.9/32") notPlaced := netip.MustParsePrefix(unplaced + "/32") wantCountry(t, g, placed, germany) wantCountry(t, g, notPlaced, "") - wantRequests(t, geojs, 2) + wantRequests(t, service, 2) // An answer without a country is kept too. clock.advance(week - time.Second) wantCountry(t, g, placed, germany) wantCountry(t, g, notPlaced, "") - wantRequests(t, geojs, 2) + wantRequests(t, service, 2) clock.advance(time.Second) wantCountry(t, g, placed, germany) - wantRequests(t, geojs, 3) - wantAsked(t, geojs, 2, "203.0.113.9") + wantRequests(t, service, 3) + wantAsked(t, service, 2, "203.0.113.9") }) } @@ -76,17 +77,17 @@ func TestLookUpWithoutWaitingAsksGeoJSAndGoesOnAtOnce(t *testing.T) { t.Parallel() synctest.Test(t, func(t *testing.T) { - geojs, clock, g := start() + service, clock, g := start() client := netip.MustParsePrefix("203.0.113.9/32") - // Wait is set, and GeoJS does not answer. - geojs.set(hanging) + // GeoJS does not answer. + service.set(hanging) began := time.Now() answer := g.LookUpWithoutWaiting(t.Context(), client) if waited := time.Since(began); waited != 0 || - answer != (lookup.Answer{}) { + answer != (geojs.Answer{}) { t.Errorf( "got %+v after %s, want no answer at once", answer, @@ -94,14 +95,14 @@ func TestLookUpWithoutWaitingAsksGeoJSAndGoesOnAtOnce(t *testing.T) { ) } - waitForRequests(t, geojs, 1) - wantAsked(t, geojs, 0, "203.0.113.9") + waitForRequests(t, service, 1) + wantAsked(t, service, 0, "203.0.113.9") // That request is abandoned, and has failed once this returns. GeoJS // answers when it is asked again, and the answer is kept. - time.Sleep(timeout) + time.Sleep(requestTimeout) synctest.Wait() - geojs.set(answering) + service.set(answering) waitForCountry(t, g, clock, client, germany) if got := g.LookUpWithoutWaiting(t.Context(), client).Country; got != germany { @@ -114,39 +115,45 @@ func TestLookUpWithoutWaitingAsksGeoJSAndGoesOnAtOnce(t *testing.T) { }) } -func TestNewClientWaitsAtMostOneSecondThenCountsAsNotFound(t *testing.T) { +func TestNewClientWaitsAsLongAsItsCallerSaysThenCountsAsNotFound( + t *testing.T, +) { t.Parallel() synctest.Test(t, func(t *testing.T) { - geojs, clock, g := start() + service, clock, g := start() client := netip.MustParsePrefix("203.0.113.9/32") // The client comes while GeoJS is asked about an earlier client, which // it answers most of a second later. It is then asked about the client - // and does not answer: that request is abandoned a second after it + // and does not answer: that request is abandoned ten seconds after it // began, well after the client's wait is over. - geojs.set(answeringSlowly) + service.set(answeringSlowly) var earlier sync.WaitGroup - earlier.Go( - func() { g.LookUp(t.Context(), netip.MustParsePrefix("203.0.113.1/32")) }, - ) + earlier.Go(func() { + g.LookUp( + t.Context(), + netip.MustParsePrefix("203.0.113.1/32"), + wait, + ) + }) defer earlier.Wait() - waitForRequests(t, geojs, 1) - geojs.set(hanging) + waitForRequests(t, service, 1) + service.set(hanging) began := time.Now() wantCountry(t, g, client, "") took := time.Since(began) - if took != timeout { - t.Errorf("waited %s for the answer, want %s", took, timeout) + if took != wait { + t.Errorf("waited %s for the answer, want %s", took, wait) } - // Its next request does not wait. + // Its next lookup does not wait. began = time.Now() wantCountry(t, g, client, "") @@ -158,66 +165,212 @@ func TestNewClientWaitsAtMostOneSecondThenCountsAsNotFound(t *testing.T) { // Once GeoJS answers, the client is asked about again in the // background, and has its country. - geojs.set(answering) + service.set(answering) waitForCountry(t, g, clock, client, germany) }) } -func TestRequestWaitsAsLongAsTheTimeoutSaysAndGeoJSIsAbandonedAfterIt( +func TestAnswerAfterTheWaitIsKeptAndIsNoFailure(t *testing.T) { + t.Parallel() + + synctest.Test(t, func(t *testing.T) { + service, _, g, reports := startWithReports() + client := netip.MustParsePrefix("203.0.113.9/32") + + // GeoJS answers after the lookup's wait is over. + service.set(answeringLate) + wantCountry(t, g, client, "") + + time.Sleep(lateAnswer - wait) + synctest.Wait() + + // The answer is kept, and the next lookup returns it without asking + // GeoJS again. + answer, found := g.Kept(client) + if !found || answer.Country != germany { + t.Errorf( + "kept %+v, %t; want the answer in %s", + answer, + found, + germany, + ) + } + + wantCountry(t, g, client, germany) + wantRequests(t, service, 1) + reports.wantFailures(t) + + // GeoJS is not left alone: a new client is asked about at once. + service.set(answering) + wantCountry(t, g, netip.MustParsePrefix("203.0.113.10/32"), germany) + wantRequests(t, service, 2) + }) +} + +func TestAnswerAfter5SecondsIsReadFromTheCache6SecondsAfterTheFirstLookUp( t *testing.T, ) { t.Parallel() synctest.Test(t, func(t *testing.T) { - // A timeout longer than the default, and a GeoJS that does not - // answer. - const longerTimeout = 3 * time.Second + const shortWait = 800 * time.Millisecond - m := metrics.New(1, "app") - g := lookup.New(lookup.Params{ - URL: lookup.URL, - Timeout: longerTimeout, - Wait: true, - Now: time.Now, - ProcessLog: slog.New(slog.DiscardHandler), - Metrics: m, - Alerts: alerts.New(alerts.Params{}), - }) - g.SetTransport(&standIn{answers: hanging}) + service, _, g := start() + client := netip.MustParsePrefix("203.0.113.9/32") - var ( - request sync.WaitGroup - waited time.Duration - ) + service.set(answeringLate) - request.Go(func() { - began := time.Now() + began := time.Now() - wantCountry(t, g, netip.MustParsePrefix("203.0.113.9/32"), "") - - waited = time.Since(began) - }) - - // A moment before the timeout runs out, GeoJS is still being asked: - // the request to it has not failed. - time.Sleep(longerTimeout - time.Millisecond) - synctest.Wait() - wantFailures(t, m, 0) - - // As it runs out, the client's request goes on, and the request to - // GeoJS is abandoned, which counts as a failure. - request.Wait() - synctest.Wait() - - if waited != longerTimeout { + answer := g.LookUp(t.Context(), client, shortWait) + if took := time.Since(began); took != shortWait || + answer != (geojs.Answer{}) { t.Errorf( - "waited %s for the answer, want %s", - waited, - longerTimeout, + "got %+v after %s, want no answer after %s", + answer, + took, + shortWait, ) } - wantFailures(t, m, 1) + time.Sleep(6*time.Second - shortWait) + + began = time.Now() + + answer = g.LookUp(t.Context(), client, shortWait) + if took := time.Since(began); took != 0 || + answer.Country != germany { + t.Errorf( + "got %+v after %s, want the answer in %s at once", + answer, + took, + germany, + ) + } + + wantRequests(t, service, 1) + }) +} + +func TestTenLookUpsAtOnceOfOneClientCauseOneRequest(t *testing.T) { + t.Parallel() + + synctest.Test(t, func(t *testing.T) { + service, _, g := start() + client := netip.MustParsePrefix("203.0.113.9/32") + + service.set(answeringSlowly) + + var lookUps sync.WaitGroup + + for range 10 { + lookUps.Go(func() { wantCountry(t, g, client, germany) }) + } + + lookUps.Wait() + wantRequests(t, service, 1) + wantAsked(t, service, 0, "203.0.113.9") + }) +} + +func TestCallerWhoseContextEndsEarlyDoesNotStopTheRequest(t *testing.T) { + t.Parallel() + + synctest.Test(t, func(t *testing.T) { + const early = 100 * time.Millisecond + + service, _, g, reports := startWithReports() + client := netip.MustParsePrefix("203.0.113.9/32") + + service.set(answeringLate) + + ctx, cancel := context.WithTimeout(t.Context(), early) + defer cancel() + + began := time.Now() + + answer := g.LookUp(ctx, client, wait) + if took := time.Since(began); took != early || + answer != (geojs.Answer{}) { + t.Errorf( + "got %+v after %s, want no answer after %s", + answer, + took, + early, + ) + } + + time.Sleep(lateAnswer) + synctest.Wait() + + answer, found := g.Kept(client) + if !found || answer.Country != germany { + t.Errorf( + "kept %+v, %t; want the answer in %s", + answer, + found, + germany, + ) + } + + wantRequests(t, service, 1) + reports.wantFailures(t) + }) +} + +func TestRequestOver10SecondsIsAFailure(t *testing.T) { + t.Parallel() + + synctest.Test(t, func(t *testing.T) { + service, _, g, reports := startWithReports() + + // GeoJS does not answer. + service.set(hanging) + wantCountry(t, g, netip.MustParsePrefix("203.0.113.9/32"), "") + + // A moment before 10 seconds are up, the request has not failed. + time.Sleep(requestTimeout - wait - time.Millisecond) + synctest.Wait() + reports.wantFailures(t) + + // As they run out, it is abandoned, which is a failure. The error + // does not name the address asked about. + time.Sleep(time.Millisecond) + synctest.Wait() + reports.wantFailures(t, failure{ + err: "ask GeoJS: context deadline exceeded", + askingAgainIn: time.Second, + }) + + // GeoJS is then left alone, so a new client is not asked about. + wantCountry(t, g, netip.MustParsePrefix("203.0.113.10/32"), "") + wantRequests(t, service, 1) + }) +} + +func TestFailureIsReportedWithHowLongGeoJSIsLeftAlone(t *testing.T) { + t.Parallel() + + synctest.Test(t, func(t *testing.T) { + service, clock, g, reports := startWithReports() + clients := newClients() + + service.set(failing) + wantCountry(t, g, clients(), "") + + clock.advance(time.Second) + wantCountry(t, g, clients(), "") + + reports.wantFailures(t, + failure{ + err: "GeoJS answered 503 Service Unavailable", + askingAgainIn: time.Second, + }, + failure{ + err: "GeoJS answered 503 Service Unavailable", + askingAgainIn: 2 * time.Second, + }, + ) }) } @@ -238,32 +391,32 @@ func TestAddressLeftOutOfAnAnswerIsAskedAboutAgain(t *testing.T) { t.Parallel() synctest.Test(t, func(t *testing.T) { - geojs, clock, g := start() + service, clock, g := start() other := netip.MustParsePrefix("203.0.113.1/32") client := netip.MustParsePrefix(leftOut + "/32") // GeoJS fails, and is left alone for a second while the client // comes too, so that the next request asks about both. - geojs.set(failing) + service.set(failing) wantCountry(t, g, other, "") wantCountry(t, g, client, "") - geojs.set(tc.answers) + service.set(tc.answers) clock.advance(time.Second) wantCountry(t, g, other, "") - waitForRequests(t, geojs, 2) + waitForRequests(t, service, 2) // The answer counts as a failure, and the client is asked about // again, with the other client only if the answer left it out too. - geojs.set(answering) + service.set(answering) waitForCountry(t, g, clock, client, germany) wantCountry(t, g, other, germany) - wantRequests(t, geojs, 3) + wantRequests(t, service, 3) if tc.named { - wantAsked(t, geojs, 2, leftOut) + wantAsked(t, service, 2, leftOut) } else { - wantAsked(t, geojs, 2, leftOut, "203.0.113.1") + wantAsked(t, service, 2, leftOut, "203.0.113.1") } }) }) @@ -274,11 +427,11 @@ func TestRedirectCountsAsFailure(t *testing.T) { t.Parallel() synctest.Test(t, func(t *testing.T) { - geojs, _, g := start() - geojs.set(redirecting) + service, _, g := start() + service.set(redirecting) wantCountry(t, g, netip.MustParsePrefix("203.0.113.9/32"), "") - wantRequests(t, geojs, 1) + wantRequests(t, service, 1) }) } @@ -286,8 +439,8 @@ func TestCountryIsKeptInCapitals(t *testing.T) { t.Parallel() synctest.Test(t, func(t *testing.T) { - geojs, _, g := start() - geojs.set(answeringInLowerCase) + service, _, g := start() + service.set(answeringInLowerCase) wantCountry(t, g, netip.MustParsePrefix("203.0.113.9/32"), germany) }) @@ -304,14 +457,14 @@ func TestAnswerHoldsTheASNumberTheASNameAndTheCountry(t *testing.T) { // For the client it cannot place, GeoJS gives the AS number 64512 // and the AS name Unknown, which count as unknown. - for client, want := range map[netip.Prefix]lookup.Answer{ + for client, want := range map[netip.Prefix]geojs.Answer{ placed: { Client: placed, ASN: asn, ASName: asName, Country: germany, Answered: now, Used: now, }, notPlaced: {Client: notPlaced, Answered: now, Used: now}, } { - got := g.LookUp(t.Context(), client) + got := g.LookUp(t.Context(), client, wait) if got != want { t.Errorf("answer for %s\n%+v\nwant\n%+v", client, got, want) } @@ -319,146 +472,54 @@ func TestAnswerHoldsTheASNumberTheASNameAndTheCountry(t *testing.T) { }) } -func TestWithoutWaitTheRequestGoesOnAtOnceAndTheAnswerIsGivenWhenItComes( - t *testing.T, -) { +func TestEachAnswerIsGivenToAnsweredOnceKept(t *testing.T) { t.Parallel() synctest.Test(t, func(t *testing.T) { var ( mu sync.Mutex - given []lookup.Answer + given []geojs.Answer ) - geojs := &standIn{answers: answeringSlowly} + service := &standIn{answers: answeringSlowly} clock := newClock() - m := metrics.New(1, "app") - g := lookup.New(lookup.Params{ - URL: lookup.URL, - Timeout: timeout, - Answered: func(answer lookup.Answer) { + g, reports := newGeoJS(service, geojs.Params{ + Answered: func(answer geojs.Answer) { mu.Lock() defer mu.Unlock() given = append(given, answer) }, - Now: clock.Now, - ProcessLog: slog.New(slog.DiscardHandler), - Metrics: m, - Alerts: alerts.New(alerts.Params{}), + Now: clock.Now, }) - g.SetTransport(geojs) client := netip.MustParsePrefix("203.0.113.9/32") - // The request goes on at once, without an answer, and GeoJS is asked - // about the client, which it answers most of a second later. - began := time.Now() - - got := g.LookUp(t.Context(), client) - if took := time.Since(began); took != 0 || got != (lookup.Answer{}) { - t.Errorf( - "waited %s for %+v, want no wait and no answer", - took, - got, - ) + // GeoJS answers about the client most of a second later. + if got := g.LookUpWithoutWaiting(t.Context(), client); got != (geojs.Answer{}) { + t.Errorf("got %+v, want no answer", got) } - waitForRequests(t, geojs, 1) - wantAsked(t, geojs, 0, "203.0.113.9") + waitForRequests(t, service, 1) + wantAsked(t, service, 0, "203.0.113.9") - time.Sleep(timeout) + time.Sleep(wait) synctest.Wait() now := clock.Now() - want := lookup.Answer{ + want := geojs.Answer{ Client: client, ASN: asn, ASName: asName, Country: germany, Answered: now, Used: now, } mu.Lock() - if !slices.Equal(given, []lookup.Answer{want}) { + if !slices.Equal(given, []geojs.Answer{want}) { t.Errorf("answers given %+v, want only %+v", given, want) } mu.Unlock() wantCountry(t, g, client, germany) - wantRequestsWithoutGeoJSAnswer(t, m, 0) - }) -} - -func TestFailureIsLoggedWithoutTheAddressesAskedAbout(t *testing.T) { - t.Parallel() - - synctest.Test(t, func(t *testing.T) { - var log strings.Builder - - // GeoJS does not answer, so the request to it is abandoned, and fails. - geojs := &standIn{answers: hanging} - g := lookup.New(lookup.Params{ - URL: lookup.URL, - Timeout: timeout, - Wait: true, - Now: time.Now, - ProcessLog: slog.New(slog.NewTextHandler(&log, nil)), - Metrics: metrics.New(1, "app"), - Alerts: alerts.New(alerts.Params{}), - }) - g.SetTransport(geojs) - - wantCountry(t, g, netip.MustParsePrefix("203.0.113.9/32"), "") - synctest.Wait() - - logged := log.String() - if !strings.Contains(logged, "asking GeoJS failed") || - strings.Contains(logged, "203.0.113.9") { - t.Errorf( - "logged %q, want the failure without the address asked about", - logged, - ) - } - }) -} - -func TestFailureRaisesASourceFailureAlertOncePerCooldown(t *testing.T) { - t.Parallel() - - synctest.Test(t, func(t *testing.T) { - geojs, clock, g, queue := startWithAlerts() - clients := newClients() - - geojs.set(failing) - - wantCountry(t, g, clients(), "") - - want := alerts.Alert{ - Time: clock.Now(), - Event: alerts.EventSourceFailure, - Reason: "asking GeoJS failed", - Detail: map[string]any{ - "source": "geojs", - "error": "GeoJS answered 503 Service Unavailable", - "asking_again_in": "1s", - }, - } - - // The next failure, a second later, is a repeat within the - // cooldown. - clock.advance(time.Second) - wantCountry(t, g, clients(), "") - wantRequests(t, geojs, 2) - - waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook] - if len(waiting) != 1 || !reflect.DeepEqual(waiting[0], want) { - t.Errorf("alerts waiting %+v, want only %+v", waiting, want) - } - - if queue.Suppressed() != 1 { - t.Errorf( - "%d alerts held back, want the repeat", - queue.Suppressed(), - ) - } + reports.wantWithoutAnswer(t, 0) }) } @@ -466,24 +527,24 @@ func TestWaitingClientsAreAskedAboutInOneRequest(t *testing.T) { t.Parallel() synctest.Test(t, func(t *testing.T) { - geojs, clock, g := start() + service, clock, g := start() // GeoJS fails, and is then left alone for a second, while three more // clients come. An IPv6 client is a /64, and GeoJS is asked about its // first address. - geojs.set(failing) + service.set(failing) wantCountry(t, g, netip.MustParsePrefix("203.0.113.1/32"), "") wantCountry(t, g, netip.MustParsePrefix("203.0.113.2/32"), "") wantCountry(t, g, netip.MustParsePrefix("2001:db8:1:2::/64"), "") - wantRequests(t, geojs, 1) + wantRequests(t, service, 1) - geojs.set(answering) + service.set(answering) clock.advance(time.Second) wantCountry(t, g, netip.MustParsePrefix("203.0.113.3/32"), germany) - wantRequests(t, geojs, 2) + wantRequests(t, service, 2) wantAsked( t, - geojs, + service, 1, "203.0.113.1", "203.0.113.2", @@ -499,15 +560,15 @@ func TestKeptAnswersUnaffectedWhileGeoJSFailsAndAskedAgainWithBackoff( t.Parallel() synctest.Test(t, func(t *testing.T) { - geojs, clock, g := start() + service, clock, g := start() clients := newClients() kept := clients() wantCountry(t, g, kept, germany) - geojs.set(failing) + service.set(failing) wantCountry(t, g, kept, germany) - wantRequests(t, geojs, 1) + wantRequests(t, service, 1) // Each failure leaves GeoJS alone twice as long as the one before, up // to five minutes. New clients meanwhile count as not found, and the @@ -523,22 +584,22 @@ func TestKeptAnswersUnaffectedWhileGeoJSFailsAndAskedAgainWithBackoff( wantCountry(t, g, clients(), "") requests++ - wantRequests(t, geojs, requests) + wantRequests(t, service, requests) clock.advance(delay - time.Millisecond) wantCountry(t, g, clients(), "") wantCountry(t, g, kept, germany) - wantRequests(t, geojs, requests) + wantRequests(t, service, requests) clock.advance(time.Millisecond) } // Once GeoJS answers again, it is asked about every client waiting. - geojs.set(answering) + service.set(answering) wantCountry(t, g, clients(), germany) - wantRequests(t, geojs, requests+1) + wantRequests(t, service, requests+1) - asked := waitForRequests(t, geojs, requests+1) + asked := waitForRequests(t, service, requests+1) if len(asked[requests]) != 23 { t.Errorf( "GeoJS was asked about %d clients, want 23", @@ -552,24 +613,24 @@ func TestAtMost200AddressesInOneRequest(t *testing.T) { t.Parallel() synctest.Test(t, func(t *testing.T) { - geojs, clock, g := start() + service, clock, g := start() clients := newClients() first := clients() // 201 clients wait while GeoJS is left alone after a failure. - geojs.set(failing) + service.set(failing) wantCountry(t, g, first, "") for range 200 { wantCountry(t, g, clients(), "") } - // The first one's next request has GeoJS asked again. - geojs.set(answering) + // The first one's next lookup has GeoJS asked again. + service.set(answering) clock.advance(time.Second) wantCountry(t, g, first, "") - asked := waitForRequests(t, geojs, 3) + asked := waitForRequests(t, service, 3) if len(asked[1]) != 200 || len(asked[2]) != 1 { t.Errorf( "GeoJS was asked about %d and then %d clients, want 200 and 1", @@ -584,13 +645,13 @@ func TestAtMost10000ClientsWait(t *testing.T) { t.Parallel() synctest.Test(t, func(t *testing.T) { - geojs, clock, g := start() + service, clock, g := start() clients := newClients() first := clients() // 10,000 clients wait while GeoJS is left alone after a failure, and // one more cannot join them. - geojs.set(failing) + service.set(failing) wantCountry(t, g, first, "") for range 9999 { @@ -600,13 +661,13 @@ func TestAtMost10000ClientsWait(t *testing.T) { extra := clients() wantCountry(t, g, extra, "") - // The first one's next request has GeoJS asked about the 10,000, 200 + // The first one's next lookup has GeoJS asked about the 10,000, 200 // at a time, and not about the one more. - geojs.set(answering) + service.set(answering) clock.advance(time.Second) wantCountry(t, g, first, "") - asked := waitForRequests(t, geojs, 51) + asked := waitForRequests(t, service, 51) for i, request := range asked { if slices.Contains(request, extra.Addr().String()) { t.Errorf("request %d asked about %s", i, extra.Addr()) @@ -622,24 +683,15 @@ func TestClientsWithoutAnAnswerAreCounted(t *testing.T) { t.Parallel() synctest.Test(t, func(t *testing.T) { - m := metrics.New(1, "app") - g := lookup.New(lookup.Params{ - URL: lookup.URL, - Timeout: timeout, - Wait: true, - Now: time.Now, - ProcessLog: slog.New(slog.DiscardHandler), - Metrics: m, - Alerts: alerts.New(alerts.Params{}), - }) - g.SetTransport(&standIn{answers: failing}) - + service, _, g, reports := startWithReports() clients := newClients() + service.set(failing) + // GeoJS fails, so the first client goes without an answer, and GeoJS // is left alone for a second, which does not pass in this test. wantCountry(t, g, clients(), "") - wantRequestsWithoutGeoJSAnswer(t, m, 1) + reports.wantWithoutAnswer(t, 1) // Meanwhile each new client goes without one at once, while there is // room for it among the 10,000 that may wait. @@ -647,11 +699,11 @@ func TestClientsWithoutAnAnswerAreCounted(t *testing.T) { wantCountry(t, g, clients(), "") } - wantRequestsWithoutGeoJSAnswer(t, m, 10000) + reports.wantWithoutAnswer(t, 10000) // One more, for which there is no room, goes without one too. wantCountry(t, g, clients(), "") - wantRequestsWithoutGeoJSAnswer(t, m, 10001) + reports.wantWithoutAnswer(t, 10001) }) } @@ -659,6 +711,7 @@ func TestClientsWithoutAnAnswerAreCounted(t *testing.T) { const ( answering = iota answeringSlowly // most of a second later + answeringLate // lateAnswer later answeringInLowerCase // with each country in lower case answeringWithoutLeftOut // with a list that leaves leftOut out answeringEmptyList // with [] @@ -712,9 +765,9 @@ func (s *standIn) ServeHTTP(w http.ResponseWriter, r *http.Request) { http.Redirect(w, r, "/", http.StatusFound) return - case answeringSlowly: + case answeringSlowly, answeringLate: select { - case <-time.After(timeout * 4 / 5): + case <-time.After(delay(answers)): case <-r.Context().Done(): return } @@ -759,6 +812,16 @@ func (s *standIn) ServeHTTP(w http.ResponseWriter, r *http.Request) { } } +// delay returns how long the stand-in takes to answer when answering +// slowly or late. +func delay(answers int) time.Duration { + if answers == answeringLate { + return lateAnswer + } + + return wait * 4 / 5 +} + // set sets how the stand-in answers. func (s *standIn) set(answers int) { s.mu.Lock() @@ -775,6 +838,46 @@ func (s *standIn) asked() [][]string { return slices.Clone(s.requests) } +// reports are what a GeoJS reported through Failed and WentWithoutAnswer. +type reports struct { + mu sync.Mutex + failures []failure + withoutAnswer int +} + +// failure is one report of Failed: the error, and how long GeoJS is left +// alone. +type failure struct { + err string + askingAgainIn time.Duration +} + +// wantFailures checks the failures reported so far. +func (r *reports) wantFailures(t *testing.T, want ...failure) { + t.Helper() + + r.mu.Lock() + defer r.mu.Unlock() + + if !slices.Equal(r.failures, want) { + t.Errorf("failures %+v, want %+v", r.failures, want) + } +} + +// wantWithoutAnswer checks how many times a caller has gone without an +// answer. +func (r *reports) wantWithoutAnswer(t *testing.T, want int) { + t.Helper() + + r.mu.Lock() + defer r.mu.Unlock() + + if r.withoutAnswer != want { + t.Errorf("%d lookups went without an answer, want %d", + r.withoutAnswer, want) + } +} + // testClock is a clock the test sets. GeoJS tells the time by it, while // waits run on the bubble's clock. type testClock struct { @@ -799,38 +902,47 @@ func (c *testClock) advance(d time.Duration) { } // start returns a stand-in for GeoJS that answers, a clock, and a GeoJS -// asking the stand-in by that clock, for which a request waits for its -// client's first answer. -func start() (*standIn, *testClock, *lookup.GeoJS) { - geojs, clock, g, _ := startWithAlerts() +// asking the stand-in by that clock. +func start() (*standIn, *testClock, *geojs.GeoJS) { + service, clock, g, _ := startWithReports() - return geojs, clock, g + return service, clock, g } -// startWithAlerts is start, and returns the queue of the alerts GeoJS -// raises as well, for a webhook that is never sent them, with the default -// cooldown, by the same clock. -func startWithAlerts() (*standIn, *testClock, *lookup.GeoJS, *alerts.Queue) { - geojs := &standIn{} +// startWithReports is start, and returns what the GeoJS reports as well. +func startWithReports() (*standIn, *testClock, *geojs.GeoJS, *reports) { + service := &standIn{} clock := newClock() - queue := alerts.New(alerts.Params{ - WebhookURL: &url.URL{Scheme: "https", Host: "alerts.example"}, - Events: alerts.Events(), - Cooldown: 15 * time.Minute, - Now: clock.Now, - }) - g := lookup.New(lookup.Params{ - URL: lookup.URL, - Timeout: timeout, - Wait: true, - Now: clock.Now, - ProcessLog: slog.New(slog.DiscardHandler), - Metrics: metrics.New(1, "app"), - Alerts: queue, - }) - g.SetTransport(geojs) + g, reports := newGeoJS(service, geojs.Params{Now: clock.Now}) - return geojs, clock, g, queue + return service, clock, g, reports +} + +// newGeoJS returns a GeoJS that asks service, made with params and with +// what it reports noted in the reports returned. +func newGeoJS( + service *standIn, params geojs.Params, +) (*geojs.GeoJS, *reports) { + r := &reports{} + params.URL = geojs.URL + params.Asked = func() {} + params.Failed = func(err error, askingAgainIn time.Duration) { + r.mu.Lock() + defer r.mu.Unlock() + + r.failures = append(r.failures, failure{err.Error(), askingAgainIn}) + } + params.WentWithoutAnswer = func() { + r.mu.Lock() + defer r.mu.Unlock() + + r.withoutAnswer++ + } + + g := geojs.New(params) + g.SetTransport(service) + + return g, r } // newClock returns a clock set to the start of a day. @@ -850,36 +962,37 @@ func newClients() func() netip.Prefix { } } -// wantCountry checks the country g gives client. +// wantCountry checks the country g gives client, waiting up to wait for +// it. func wantCountry( t *testing.T, - g *lookup.GeoJS, + g *geojs.GeoJS, client netip.Prefix, want string, ) { t.Helper() - got := g.LookUp(t.Context(), client).Country + got := g.LookUp(t.Context(), client, wait).Country if got != want { t.Errorf("%s is in %q, want %q", client, got, want) } } // wantRequests checks how many requests GeoJS has had. -func wantRequests(t *testing.T, geojs *standIn, want int) { +func wantRequests(t *testing.T, service *standIn, want int) { t.Helper() - got := len(geojs.asked()) + got := len(service.asked()) if got != want { t.Errorf("GeoJS had %d requests, want %d", got, want) } } // wantAsked checks the addresses request i asked about, in any order. -func wantAsked(t *testing.T, geojs *standIn, i int, want ...string) { +func wantAsked(t *testing.T, service *standIn, i int, want ...string) { t.Helper() - asked := geojs.asked() + asked := service.asked() if len(asked) <= i { t.Fatalf("GeoJS had %d requests, want more than %d", len(asked), i) } @@ -893,40 +1006,15 @@ func wantAsked(t *testing.T, geojs *standIn, i int, want ...string) { } } -// wantRequestsWithoutGeoJSAnswer checks how many requests m counts as having -// gone without an answer from GeoJS. -func wantRequestsWithoutGeoJSAnswer( - t *testing.T, - m *metrics.Metrics, - want float64, -) { - t.Helper() - - got := testutil.ToFloat64(m.RequestsWithoutGeoJSAnswer) - if got != want { - t.Errorf("%v requests went without an answer, want %v", got, want) - } -} - -// wantFailures checks how many requests to GeoJS m counts as failed. -func wantFailures(t *testing.T, m *metrics.Metrics, want float64) { - t.Helper() - - got := testutil.ToFloat64(m.GeoJSFailures) - if got != want { - t.Errorf("%v requests to GeoJS failed, want %v", got, want) - } -} - // waitForRequests waits until g has done all it can before time passes, // checks that GeoJS has had count requests, and returns the addresses each // asked about. -func waitForRequests(t *testing.T, geojs *standIn, count int) [][]string { +func waitForRequests(t *testing.T, service *standIn, count int) [][]string { t.Helper() synctest.Wait() - asked := geojs.asked() + asked := service.asked() if len(asked) != count { t.Fatalf("GeoJS had %d requests, want %d", len(asked), count) } @@ -936,18 +1024,18 @@ func waitForRequests(t *testing.T, geojs *standIn, count int) [][]string { // waitForCountry lets a request to GeoJS under way be abandoned, and moves // the clock on a minute, so that GeoJS may be asked again after a failure. -// It then checks that client's next request does not wait but has it asked +// It then checks that client's next lookup does not wait but has it asked // about again in the background, after which g gives it the country want. func waitForCountry( t *testing.T, - g *lookup.GeoJS, + g *geojs.GeoJS, clock *testClock, client netip.Prefix, want string, ) { t.Helper() - time.Sleep(timeout) + time.Sleep(requestTimeout) clock.advance(time.Minute) wantCountry(t, g, client, "") synctest.Wait() diff --git a/internal/lookup/snapshot_test.go b/internal/geojs/snapshot_test.go similarity index 85% rename from internal/lookup/snapshot_test.go rename to internal/geojs/snapshot_test.go index af854a3..27c71bc 100644 --- a/internal/lookup/snapshot_test.go +++ b/internal/geojs/snapshot_test.go @@ -1,4 +1,4 @@ -package lookup_test +package geojs_test import ( "net/netip" @@ -7,7 +7,7 @@ import ( "testing/synctest" "time" - "sneak.berlin/go/smallwebwaf/internal/lookup" + "sneak.berlin/go/smallwebwaf/internal/geojs" ) func TestSnapshotHoldsEachAnswerAndWhenItWasLastUsed(t *testing.T) { @@ -25,7 +25,7 @@ func TestSnapshotHoldsEachAnswerAndWhenItWasLastUsed(t *testing.T) { clock.advance(time.Hour) wantCountry(t, g, placed, germany) - want := []lookup.Answer{ + want := []geojs.Answer{ {Client: notPlaced, Answered: asked, Used: asked}, { Client: placed, ASN: asn, ASName: asName, Country: germany, @@ -41,24 +41,24 @@ func TestSnapshotHoldsEachAnswerAndWhenItWasLastUsed(t *testing.T) { func TestLoadedAnswersAreKeptFor7DaysFromWhenGeoJSGaveThem(t *testing.T) { t.Parallel() - geojs, clock, g := start() + service, clock, g := start() now := clock.Now() - kept := lookup.Answer{ + kept := geojs.Answer{ Client: netip.MustParsePrefix("203.0.113.9/32"), Country: "FR", Answered: now.Add(-week + time.Second), Used: now.Add(-time.Hour), } - stale := lookup.Answer{ + stale := geojs.Answer{ Client: netip.MustParsePrefix("203.0.113.10/32"), Country: "FR", Answered: now.Add(-week), Used: now.Add(-time.Hour), } - g.Load([]lookup.Answer{kept, stale}) + g.Load([]geojs.Answer{kept, stale}) - if got := g.Snapshot(); !slices.Equal(got, []lookup.Answer{kept}) { + if got := g.Snapshot(); !slices.Equal(got, []geojs.Answer{kept}) { t.Errorf( "kept %+v, want only the answer GeoJS gave less than 7 days ago", got, @@ -66,7 +66,7 @@ func TestLoadedAnswersAreKeptFor7DaysFromWhenGeoJSGaveThem(t *testing.T) { } wantCountry(t, g, kept.Client, "FR") - wantRequests(t, geojs, 0) + wantRequests(t, service, 0) } func TestLoadDropsTheAnswerUsedLongestAgoFirst(t *testing.T) { @@ -80,11 +80,11 @@ func TestLoadDropsTheAnswerUsedLongestAgoFirst(t *testing.T) { // lookups.json lists the answers by client. Here each was last used a // second before the one listed before it, so the last listed is the // one used longest ago, and the one dropped. - answers := make([]lookup.Answer, maxAnswers+1) + answers := make([]geojs.Answer, maxAnswers+1) addr := netip.MustParseAddr("10.0.0.0") for i := range answers { - answers[i] = lookup.Answer{ + answers[i] = geojs.Answer{ Client: netip.PrefixFrom(addr, addr.BitLen()), Country: germany, Answered: now, diff --git a/internal/lookup/file.go b/internal/lookup/file.go index 74292d1..1c028f5 100644 --- a/internal/lookup/file.go +++ b/internal/lookup/file.go @@ -1,3 +1,6 @@ +// Package lookup looks up each client's AS number and country in the lookup +// database, the IPinfo Lite file SWW_WAF_LOOKUP_DB_PATH names. Its answers +// take the form of GeoJS's, so that waf handles both alike. package lookup import ( @@ -14,6 +17,7 @@ import ( "github.com/oschwald/maxminddb-golang/v2" "sneak.berlin/go/smallwebwaf/internal/alerts" + "sneak.berlin/go/smallwebwaf/internal/geojs" ) // quietTime is how long the lookup database must go without a change @@ -77,7 +81,7 @@ func OpenFile(params FileParams) (*File, error) { // each "" when the database does not give it, as for an address missing // from it. The database is asked about the client's first address, as // GeoJS is. -func (f *File) LookUp(client netip.Prefix) Answer { +func (f *File) LookUp(client netip.Prefix) geojs.Answer { f.mu.Lock() reader := f.reader f.mu.Unlock() @@ -91,7 +95,7 @@ func (f *File) LookUp(client netip.Prefix) Answer { found = record{} } - return Answer{ + return geojs.Answer{ Client: client, ASN: found.ASN, ASName: found.ASName, diff --git a/internal/lookup/file_internal_test.go b/internal/lookup/file_internal_test.go index c51c3bd..a49c68b 100644 --- a/internal/lookup/file_internal_test.go +++ b/internal/lookup/file_internal_test.go @@ -16,6 +16,7 @@ import ( "github.com/maxmind/mmdbwriter/mmdbtype" "sneak.berlin/go/smallwebwaf/internal/alerts" + "sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest" ) @@ -68,7 +69,7 @@ func TestFilePlacesClientsAndCountsAnAddressMissingFromItAsUnknown( prefix := netip.MustParsePrefix(client) got := f.LookUp(prefix) - if got != (Answer{ + if got != (geojs.Answer{ Client: prefix, ASN: want.ASN, ASName: want.ASName, Country: want.Country, Answered: now, }) { diff --git a/internal/metrics/metrics.go b/internal/metrics/metrics.go index 24e92f3..2f787e6 100644 --- a/internal/metrics/metrics.go +++ b/internal/metrics/metrics.go @@ -274,8 +274,7 @@ func (m *Metrics) AddRemoteLog(remote *remotelog.Sender) { // AddLookupFile adds the metrics of the lookup database, read as the // metrics are asked for: when the file in use was read, which lastRead // returns, and the replacements of it that could not be read, which -// readFailures returns. The lookup package's File, which has both, cannot -// be named here: that package counts GeoJS's requests in these metrics. +// readFailures returns. func (m *Metrics) AddLookupFile( lastRead func() time.Time, readFailures func() int, diff --git a/internal/remotelog/remotelog_test.go b/internal/remotelog/remotelog_test.go index ffa8862..91ebacc 100644 --- a/internal/remotelog/remotelog_test.go +++ b/internal/remotelog/remotelog_test.go @@ -44,7 +44,7 @@ import ( const ( // started is the time a record made as a test starts gives. started = "2000-01-01T00:00:00.000Z" - appName = "fsn1app1/blog" + appName = "host1/blog" // local0 is the number of the default facility, and local0Info the // priority of its records. local0 = 16 diff --git a/internal/reputation/dnsbl_test.go b/internal/reputation/dnsbl_test.go index 0be1a3d..dfe1e74 100644 --- a/internal/reputation/dnsbl_test.go +++ b/internal/reputation/dnsbl_test.go @@ -31,10 +31,12 @@ import ( // this package: Go's resolver counts the queries under way in one // sync.WaitGroup for the whole process, and the process fails when // queries from two bubbles, or from a bubble and from outside one, are -// under way at once. TestMain has the resolver make its configuration, -// which it makes on its first query, outside every bubble, since the -// configuration holds a channel, which the bubble it was made in would -// keep to itself. +// under way at once. The one test outside a bubble is parallel, so it +// starts only once every test that is not parallel has ended; no other +// parallel test of this package queries a resolver. TestMain has the +// resolver make its configuration, which it makes on its first query, +// outside every bubble, since the configuration holds a channel, which +// the bubble it was made in would keep to itself. const ( // zone and otherZone are the DNSBL zones the tests name. @@ -525,8 +527,9 @@ func TestAtMost100000VerdictsKeptTheOneFetchedLongestAgoDroppedFirst( } } -//nolint:paralleltest // one at a time, as the comment at the top of this file says func TestQueriesGoToTheResolverSWWDNSBLResolverNames(t *testing.T) { + t.Parallel() + resolver := &resolverStandIn{answers: map[string]answer{ listedName: {addrs: []string{listing}}, }} diff --git a/internal/requestlog/requestlog.go b/internal/requestlog/requestlog.go index 7ebb8b7..c432c17 100644 --- a/internal/requestlog/requestlog.go +++ b/internal/requestlog/requestlog.go @@ -61,6 +61,12 @@ const ( // ActionCountryDenied is a request refused because its client's country // is not in SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES. ActionCountryDenied = "country_denied" + + // ActionChecksTimedOut is a request refused because the trap paths, the + // rule files and the Core Rule Set had not finished with it within 500 + // milliseconds. + ActionChecksTimedOut = "checks_timed_out" + // ActionOwnEndpoint is a request waf answered at one of its own // endpoints, or a question about a path under /_sww/. ActionOwnEndpoint = "own_endpoint" @@ -149,7 +155,8 @@ type Line struct { // ActionTemporarilyBanned, ActionPermanentlyBanned, ActionCountryDenied, // ActionClientIsRateLimited, ActionBlockedByBlockRule, // ActionBlockedByCoreRuleSet, or, in waf, ActionRequestLineTooLong or - // ActionRequestHeaderTooLarge for the header limits. + // ActionRequestHeaderTooLarge for the header limits, or + // ActionChecksTimedOut. WouldAction string `json:"would_action,omitempty"` // Heuristic is the name of the heuristic that refused the request, in // observe mode too, and HeuristicFigures are the figures it acted on, by diff --git a/internal/requestlog/requestlog_test.go b/internal/requestlog/requestlog_test.go index 11194d7..b14faac 100644 --- a/internal/requestlog/requestlog_test.go +++ b/internal/requestlog/requestlog_test.go @@ -140,7 +140,7 @@ func TestProcessLinesAreMarkedProcessAndGiveTheInstance(t *testing.T) { var out bytes.Buffer - requestlog.NewProcessLogger(&out, "fsn1app1/blog", slog.LevelInfo). + requestlog.NewProcessLogger(&out, "host1/blog", slog.LevelInfo). Info("starting", "version", "v1") @@ -152,7 +152,7 @@ func TestProcessLinesAreMarkedProcessAndGiveTheInstance(t *testing.T) { } if fields["type"] != "process" || - fields["instance"] != "fsn1app1/blog" || + fields["instance"] != "host1/blog" || fields["msg"] != "starting" || fields["level"] != "INFO" || fields["version"] != "v1" { @@ -187,7 +187,7 @@ func TestProcessLoggerWritesTheMessagesAtItsLevelOrMoreSevere(t *testing.T) { processLog := requestlog.NewProcessLogger( &out, - "fsn1app1/blog", + "host1/blog", level, ) for _, at := range levels { diff --git a/internal/rproxy/breaker.go b/internal/rproxy/breaker.go new file mode 100644 index 0000000..21e6cdb --- /dev/null +++ b/internal/rproxy/breaker.go @@ -0,0 +1,82 @@ +package rproxy + +import ( + "errors" + "sync" + "time" +) + +// What opens the circuit breaker, and how often a request asks waf while it is +// open. +const ( + breakerFailures = 5 + breakerWindow = 5 * time.Second + breakerAskInterval = time.Second +) + +// errBreakerOpen is why a request did not ask waf. +var errBreakerOpen = errors.New( + "waf is not asked while the circuit breaker is open", +) + +// breaker is the circuit breaker for asks to waf. It opens when +// breakerFailures asks fail within breakerWindow. An ask fails when it gets no +// answer in time, or an answer rproxy cannot pass on. While it is open, at +// most one request each breakerAskInterval asks waf. Any usable answer closes +// it, whatever it decides, and forgets the failures before it. +type breaker struct { + mu sync.Mutex + // failures holds the times of the latest failed asks, oldest first, at most + // breakerFailures of them. + failures []time.Time + open bool + // lastAsk is when a request last asked waf while the breaker was open. + lastAsk time.Time +} + +// mayAsk reports whether a request may ask waf at now. While the breaker is +// open, it says yes once each breakerAskInterval. +func (b *breaker) mayAsk(now time.Time) bool { + b.mu.Lock() + defer b.mu.Unlock() + + if !b.open { + return true + } + + if now.Sub(b.lastAsk) < breakerAskInterval { + return false + } + + b.lastAsk = now + + return true +} + +// noteAnswer notes that an ask got an answer rproxy can use. It closes the +// breaker and forgets the failures before it. +func (b *breaker) noteAnswer() { + b.mu.Lock() + defer b.mu.Unlock() + + b.open = false + b.failures = nil +} + +// noteFailure notes that an ask that ended at now got no answer in time, and +// opens the breaker if breakerFailures asks have failed within breakerWindow. +func (b *breaker) noteFailure(now time.Time) { + b.mu.Lock() + defer b.mu.Unlock() + + b.failures = append(b.failures, now) + if len(b.failures) > breakerFailures { + b.failures = b.failures[1:] + } + + if !b.open && len(b.failures) == breakerFailures && + now.Sub(b.failures[0]) <= breakerWindow { + b.open = true + b.lastAsk = now + } +} diff --git a/internal/rproxy/breaker_test.go b/internal/rproxy/breaker_test.go new file mode 100644 index 0000000..ef0ead3 --- /dev/null +++ b/internal/rproxy/breaker_test.go @@ -0,0 +1,142 @@ +package rproxy_test + +import ( + "testing" + "time" + + "sneak.berlin/go/smallwebwaf/internal/rproxy" +) + +// The breaker tests give the breaker each time themselves, counted from +// time.Now(), and never wait. + +// fail notes count failed asks on b, one each step from at. +func fail(b *rproxy.Breaker, at time.Time, count int, step time.Duration) { + for i := range count { + b.NoteFailure(at.Add(time.Duration(i) * step)) + } +} + +// openBreaker returns a breaker opened at now by five failed asks, one a +// second. +func openBreaker(now time.Time) *rproxy.Breaker { + b := &rproxy.Breaker{} + fail(b, now.Add(-4*time.Second), 5, time.Second) + + return b +} + +func TestFourFailedAsksWithinFiveSecondsLeaveTheBreakerClosed(t *testing.T) { + t.Parallel() + + now := time.Now() + b := &rproxy.Breaker{} + fail(b, now, 4, time.Second) + + if !b.MayAsk(now.Add(3 * time.Second)) { + t.Error("a request may not ask, want it to") + } +} + +func TestFiveFailedAsksWithinFiveSecondsOpenTheBreaker(t *testing.T) { + t.Parallel() + + now := time.Now() + b := openBreaker(now) + + if b.MayAsk(now) { + t.Error("a request may ask, want it not to") + } +} + +func TestFiveFailedAsksOverMoreThanFiveSecondsLeaveTheBreakerClosed( + t *testing.T, +) { + t.Parallel() + + now := time.Now() + b := &rproxy.Breaker{} + fail(b, now, 5, 1300*time.Millisecond) + + if !b.MayAsk(now.Add(6 * time.Second)) { + t.Error("a request may not ask, want it to") + } +} + +func TestAnAnswerBetweenFailedAsksKeepsTheBreakerClosed(t *testing.T) { + t.Parallel() + + now := time.Now() + b := &rproxy.Breaker{} + fail(b, now, 4, 100*time.Millisecond) + b.NoteAnswer() + fail(b, now.Add(500*time.Millisecond), 4, 100*time.Millisecond) + + if !b.MayAsk(now.Add(900 * time.Millisecond)) { + t.Error("a request may not ask, want it to") + } +} + +func TestAnOpenBreakerLetsOneRequestAskEachSecond(t *testing.T) { + t.Parallel() + + now := time.Now() + b := openBreaker(now) + + for _, tc := range []struct { + after time.Duration + want bool + }{ + {999 * time.Millisecond, false}, + {time.Second, true}, + {time.Second, false}, + {1500 * time.Millisecond, false}, + {2 * time.Second, true}, + } { + got := b.MayAsk(now.Add(tc.after)) + if got != tc.want { + t.Errorf("may ask %v after it opened: %v, want %v", + tc.after, got, tc.want) + } + } +} + +func TestAFailedAskKeepsTheBreakerOpen(t *testing.T) { + t.Parallel() + + opened := time.Now() + b := openBreaker(opened) + now := opened.Add(time.Second) + b.MayAsk(now) + b.NoteFailure(now) + + if b.MayAsk(now.Add(999 * time.Millisecond)) { + t.Error("a request may ask, want it not to") + } + + if !b.MayAsk(now.Add(time.Second)) { + t.Error("a request may not ask a second later, want it to") + } +} + +func TestOneAnswerClosesTheBreaker(t *testing.T) { + t.Parallel() + + opened := time.Now() + b := openBreaker(opened) + now := opened.Add(time.Second) + b.MayAsk(now) + b.NoteAnswer() + + if !b.MayAsk(now) { + t.Error("a request may not ask, want it to") + } + + // The failures before the answer no longer count, so one more does not open + // the breaker again. + b.NoteFailure(now) + + if !b.MayAsk(now) { + t.Error("a request may not ask after one failure, want it to") + } +} diff --git a/internal/rproxy/export_test.go b/internal/rproxy/export_test.go new file mode 100644 index 0000000..975298b --- /dev/null +++ b/internal/rproxy/export_test.go @@ -0,0 +1,23 @@ +package rproxy + +import "time" + +// Breaker is the circuit breaker, for tests that give it the time. +type Breaker struct { + b breaker +} + +// MayAsk reports whether a request may ask waf at now. +func (b *Breaker) MayAsk(now time.Time) bool { + return b.b.mayAsk(now) +} + +// NoteAnswer notes that an ask got an answer rproxy can use. +func (b *Breaker) NoteAnswer() { + b.b.noteAnswer() +} + +// NoteFailure notes that an ask that ended at now got no answer in time. +func (b *Breaker) NoteFailure(now time.Time) { + b.b.noteFailure(now) +} diff --git a/internal/rproxy/rproxy.go b/internal/rproxy/rproxy.go index de545b6..489c4c6 100644 --- a/internal/rproxy/rproxy.go +++ b/internal/rproxy/rproxy.go @@ -4,10 +4,11 @@ // waf goes to the visitor as waf gave it. // SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION decides only when waf cannot // be connected to, does not answer at all, or does not answer within -// SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS. rproxy itself refuses a request with a -// method it does not pass on, or one it cannot put to waf as the visitor -// sent it. It keeps nothing between requests and writes nothing to disk. "rproxy" in -// SPEC.md says what it does. +// SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS, or while the circuit breaker for asks to +// waf is open. rproxy itself refuses a request with a method it does not pass +// on, or one it cannot put to waf as the visitor sent it. It keeps nothing +// between requests but the circuit breaker, and writes nothing to disk. +// "rproxy" in SPEC.md says what it does. package rproxy import ( @@ -132,6 +133,8 @@ type handler struct { wafTransport http.RoundTripper // reportURL is where the reports go. reportURL string + // breaker is the circuit breaker for asks to waf. + breaker breaker // reports waits for the reports being sent. reportsMu guards starting one // against reportsStopped, which is set once nothing is to wait for another. reports sync.WaitGroup @@ -196,7 +199,7 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) { return } - answer, err := rq.ask(r.Context()) + answer, err := rq.askUnlessBreakerOpen(r.Context()) if err != nil { rq.handleNoUsableAnswer(r.Context(), err) diff --git a/internal/rproxy/rproxy_test.go b/internal/rproxy/rproxy_test.go index b68f733..d45e5f6 100644 --- a/internal/rproxy/rproxy_test.go +++ b/internal/rproxy/rproxy_test.go @@ -15,6 +15,7 @@ import ( "strconv" "strings" "sync" + "sync/atomic" "testing" "time" @@ -1395,6 +1396,163 @@ func TestSendsNoReportWithoutAnAnswer(t *testing.T) { standIn.wantNoReport(t) } +func TestFiveFailedAsksKeepTheNextRequestFromAskingWAF(t *testing.T) { + t.Parallel() + + for name, failure := range map[string]http.HandlerFunc{ + "no answer at all": func(http.ResponseWriter, *http.Request) { + panic(http.ErrAbortHandler) + }, + "a slow answer": func(_ http.ResponseWriter, r *http.Request) { + <-r.Context().Done() + }, + } { + t.Run(name, func(t *testing.T) { + t.Parallel() + + var answering atomic.Bool + + standIn := startWAF( + t, + func(w http.ResponseWriter, r *http.Request) { + if answering.Load() { + allow(w, r) + + return + } + + failure(w, r) + }, + ) + nextID := make(chan string, 1) + app := startApp(t, func(w http.ResponseWriter, r *http.Request) { + if answering.Load() { + nextID <- r.Header.Get(requestIDHeader) + } + + greet(w, r) + }) + addr, out := startRProxy( + t, + standIn.URL, + app.URL, + map[string]string{ + wafBackendTimeout: "200", + }, + ) + + // An ask can time out before it reaches the stand-in, so these + // may leave fewer than five questions, or leave some late. + for range 5 { + wantGreeting(t, get(t, addr, "/")) + } + + // rproxy writes this line before it answers a request it kept + // from asking waf, so no wait is needed. + if out.has("the circuit breaker is open") { + t.Error("the circuit breaker opened before five asks failed") + } + + answering.Store(true) + wantGreeting(t, get(t, addr, "/")) + out.waitFor(t, "the circuit breaker is open") + + // Had the next request asked waf, its question would have reached + // the stand-in before the request reached the app. + var id string + select { + case id = <-nextID: + default: + t.Fatal("the app did not get the next request") + } + + for len(standIn.questions) > 0 { + asked := <-standIn.questions + if asked.header.Get(requestIDHeader) == id { + t.Error("waf was asked, want it not to be") + } + } + }) + } +} + +func TestVisitorsThatGoAwayWhileWAFIsAskedLeaveTheBreakerClosed( + t *testing.T, +) { + t.Parallel() + + var answering atomic.Bool + + standIn := startWAF(t, func(w http.ResponseWriter, r *http.Request) { + if answering.Load() { + allow(w, r) + + return + } + + <-r.Context().Done() + }) + app := startApp(t, greet) + addr, out := startRProxy(t, standIn.URL, app.URL, map[string]string{ + wafBackendTimeout: "60000", + }) + + // Five of these would open the breaker if they counted as failures. + for range 5 { + conn := dial(t, addr) + send(t, conn, "GET / HTTP/1.1\r\nHost: site.example\r\n\r\n") + id := standIn.question(t).header.Get(requestIDHeader) + + _ = conn.Close() + + // rproxy writes the request's log line once it is done with the ask. + out.waitFor(t, id) + } + + answering.Store(true) + wantGreeting(t, get(t, addr, "/")) + standIn.question(t) +} + +func TestAnswersThatCannotBePassedOnOpenTheBreaker(t *testing.T) { + t.Parallel() + + var answering atomic.Bool + + tooLarge := overOneMiB(dispositionDeny) + standIn := startWAF(t, func(w http.ResponseWriter, r *http.Request) { + if answering.Load() { + allow(w, r) + + return + } + + tooLarge(w, r) + }) + app := startApp(t, greet) + addr, out := startRProxy(t, standIn.URL, app.URL, map[string]string{ + wafBackendTimeout: "60000", + }) + + for range 5 { + if got := get(t, addr, "/"); got.status != http.StatusBadGateway { + t.Errorf("answered %d, want 502", got.status) + } + + standIn.question(t) + } + + answering.Store(true) + wantGreeting(t, get(t, addr, "/")) + out.waitFor(t, "the circuit breaker is open") + + select { + case <-standIn.questions: + t.Error("waf was asked, want it not to be") + case <-time.After(10 * pollInterval): + } +} + func TestAPathUnderOwnPrefixNeverReachesTheApp(t *testing.T) { t.Parallel() diff --git a/internal/rproxy/waf.go b/internal/rproxy/waf.go index e5f298a..630329b 100644 --- a/internal/rproxy/waf.go +++ b/internal/rproxy/waf.go @@ -10,6 +10,7 @@ import ( "maps" "net/http" "strings" + "time" "sneak.berlin/go/smallwebwaf/internal/waf" ) @@ -67,7 +68,8 @@ func (rq *request) handleNoUsableAnswer(ctx context.Context, err error) { rq.answer(http.StatusBadGateway) default: // waf cannot be connected to, does not answer at all, or does not answer - // within SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS. A request that keeps waf + // within SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS; or the circuit breaker + // kept the request from asking. A request that keeps waf // busy past SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS, like any request while // waf is overloaded, gets SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION // here. deny closes this, at the cost of answering 503 whenever waf is @@ -86,6 +88,31 @@ func (rq *request) handleNoUsableAnswer(ctx context.Context, err error) { } } +// askUnlessBreakerOpen asks waf, unless the circuit breaker is open and it is +// not yet time for a request to ask. It then tells the breaker whether waf +// answered in time. An answer rproxy cannot pass on counts as a failure. A +// visitor that goes away tells the breaker nothing. ctx is the request's own +// context. +func (rq *request) askUnlessBreakerOpen( + ctx context.Context, +) (*wafAnswer, error) { + b := &rq.h.breaker + if !b.mayAsk(time.Now()) { + return nil, errBreakerOpen + } + + answer, err := rq.ask(ctx) + + switch { + case err == nil: + b.noteAnswer() + case ctx.Err() == nil: + b.noteFailure(time.Now()) + } + + return answer, err +} + // ask sends waf the question, and returns its answer, which must have come // whole within SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS. It ends when ctx does, as // when the visitor goes away. diff --git a/internal/state/state.go b/internal/state/state.go index 9e9d4f4..9a04cc9 100644 --- a/internal/state/state.go +++ b/internal/state/state.go @@ -46,7 +46,7 @@ import ( "sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/anomaly" "sneak.berlin/go/smallwebwaf/internal/bans" - "sneak.berlin/go/smallwebwaf/internal/lookup" + "sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/metrics" "sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/reputation" @@ -135,7 +135,7 @@ type Params struct { // sww runs. Ledger *bans.Ledger Limiter *ratelimit.Limiter - GeoJS *lookup.GeoJS + GeoJS *geojs.GeoJS Lists *reputation.Lists DNSBL *reputation.DNSBL AbuseIPDB *reputation.AbuseIPDB @@ -195,8 +195,8 @@ type clientsFile struct { // lookupsFile is lookups.json, with each answer on a line of its own. type lookupsFile struct { - Version int `json:"version"` - Lookups []lookup.Answer `json:"lookups"` + Version int `json:"version"` + Lookups []geojs.Answer `json:"lookups"` } // abuseIPDBChecksSpentFile is abuseipdb-checks-spent.json, indented for an diff --git a/internal/state/state_test.go b/internal/state/state_test.go index af3c840..2ee13d9 100644 --- a/internal/state/state_test.go +++ b/internal/state/state_test.go @@ -26,7 +26,7 @@ import ( "sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/anomaly" "sneak.berlin/go/smallwebwaf/internal/bans" - "sneak.berlin/go/smallwebwaf/internal/lookup" + "sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/metrics" "sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/reputation" @@ -159,7 +159,7 @@ const filledAlertsJSON = `{ "waiting": { "webhook": [ { - "instance": "fsn1app1/blog", + "instance": "host1/blog", "time": "2026-10-06T00:00:00Z", "event": "ban", "client": "203.0.113.9", @@ -174,7 +174,7 @@ const filledAlertsJSON = `{ "suppressed_repeats": 0 }, { - "instance": "fsn1app1/blog", + "instance": "host1/blog", "time": "2026-10-06T00:00:00Z", "event": "file_error", "client": "", @@ -1700,7 +1700,7 @@ func TestEditOfEachFileTakenIn(t *testing.T) { t, lookupsJSON, params.GeoJS.Snapshot(), - []lookup.Answer{ + []geojs.Answer{ {Client: client, Country: "FR", Answered: midnight()}, }, ) @@ -2398,7 +2398,7 @@ func newParams(dir string) state.Params { Events: alerts.Events(), Cooldown: 15 * time.Minute, MaxPerHour: 2, - Instance: "fsn1app1/blog", + Instance: "host1/blog", Now: midnight, }) @@ -2415,9 +2415,7 @@ func newParams(dir string) state.Params { MaxBans: 5000, }), Limiter: ratelimit.New(ratelimit.Limits{}, 20000), - GeoJS: lookup.New(lookup.Params{ - Now: midnight, ProcessLog: discard, Metrics: m, - }), + GeoJS: geojs.New(geojs.Params{Now: midnight}), Lists: reputation.New(reputation.Params{ BlocklistURLs: []string{ blocklistURL, @@ -2494,7 +2492,7 @@ func fill(params state.Params) { }) params.Limiter.AddLookup(client, now.Add(-time.Hour), asn, asName, "DE") - params.GeoJS.Load([]lookup.Answer{ + params.GeoJS.Load([]geojs.Answer{ { Client: client, ASN: asn, ASName: asName, Country: "DE", Answered: now.Add(-time.Hour), Used: now, diff --git a/internal/sww/rproxy_test.go b/internal/sww/rproxy_test.go index 934d92f..267704e 100644 --- a/internal/sww/rproxy_test.go +++ b/internal/sww/rproxy_test.go @@ -260,7 +260,17 @@ func TestTheSiteAnswersWhileWAFIsStoppedAndStartedAgain(t *testing.T) { stop() }() - wantStatus(t, site+"/", placed, http.StatusForbidden) + // The visitor's asks have opened rproxy's circuit breaker, which lets one + // request a second ask until one is answered. + deadline := time.Now().Add(waitLimit) + for statusFrom(t, site+"/", placed) != http.StatusForbidden { + if time.Now().After(deadline) { + t.Fatalf("request from %s: never answered %d", placed, + http.StatusForbidden) + } + + time.Sleep(pollInterval) + } close(visitorStop) <-visitorDone diff --git a/internal/sww/sww.go b/internal/sww/sww.go index 3da2b60..7befcde 100644 --- a/internal/sww/sww.go +++ b/internal/sww/sww.go @@ -25,6 +25,7 @@ import ( "sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/coreruleset" + "sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/lookup" "sneak.berlin/go/smallwebwaf/internal/remotelog" "sneak.berlin/go/smallwebwaf/internal/reputation" @@ -270,7 +271,7 @@ func newServer( Config: cfg, RequestLog: stdout, ProcessLog: processLog, - GeoJSURL: lookup.URL, + GeoJSURL: geojs.URL, AbuseIPDBURL: reputation.AbuseIPDBURL, LookupFile: lookupFile, CoreRuleSet: crs, diff --git a/internal/sww/sww_test.go b/internal/sww/sww_test.go index 42c4a52..4cfa9a7 100644 --- a/internal/sww/sww_test.go +++ b/internal/sww/sww_test.go @@ -52,7 +52,7 @@ const ( adminSecret = "fedcba9876543210fedcba9876543210" // instance is the SWW_WAF_INSTANCE_NAME the tests set where they look at // it. - instance = "fsn1app1/blog" + instance = "host1/blog" // greeting is what the tests' app answers. greeting = "hello from the app" // placed is the client the tests' lookup databases place. @@ -386,8 +386,8 @@ func TestEveryLogLineAndMetricCarriesTheInstanceName(t *testing.T) { // the comments. for line := range strings.Lines(metrics) { if !strings.HasPrefix(line, "#") && - !strings.Contains(line, `instance="fsn1app1/blog"`) { - t.Errorf("series %q, without instance=\"fsn1app1/blog\"", line) + !strings.Contains(line, `instance="host1/blog"`) { + t.Errorf("series %q, without instance=\"host1/blog\"", line) } } } @@ -734,12 +734,12 @@ func TestLookupDatabaseReplacedWhileRunningTakesEffect(t *testing.T) { t, url+"_sww/metrics", token, - `sww_lookup_database_read_failures_total{instance="fsn1app1/blog"} 1`, + `sww_lookup_database_read_failures_total{instance="host1/blog"} 1`, ) lastRead = seriesValue( t, metrics, - `sww_lookup_database_last_read_timestamp_seconds{instance="fsn1app1/blog"}`, + `sww_lookup_database_last_read_timestamp_seconds{instance="host1/blog"}`, ) wantStatus(t, url, placed, http.StatusOK) @@ -801,7 +801,7 @@ func TestBlocklistTriesAndCopiesKeptInListsJSONAcrossRestarts(t *testing.T) { // too. rateLimitExemptNets: placed + "," + localhost, } - failures := `sww_reputation_failures_total{instance="fsn1app1/blog",` + + failures := `sww_reputation_failures_total{instance="host1/blog",` + `source="` + torURL + `"} ` // tor.txt cannot be fetched at first, which is counted. @@ -984,8 +984,8 @@ func TestStalledRemoteLogEndpointHoldsUpNoRequest(t *testing.T) { // last. metrics := metricsText(t, url+"_sww/metrics", token) for _, series := range []string{ - `sww_remote_log_lines_sent_total{instance="fsn1app1/blog"} 0`, - `sww_remote_log_lines_waiting{instance="fsn1app1/blog"} 1`, + `sww_remote_log_lines_sent_total{instance="host1/blog"} 0`, + `sww_remote_log_lines_waiting{instance="host1/blog"} 1`, } { if !strings.Contains(metrics, "\n"+series+"\n") { t.Errorf("no %q in the metrics:\n%s", series, metrics) @@ -993,7 +993,7 @@ func TestStalledRemoteLogEndpointHoldsUpNoRequest(t *testing.T) { } const dropped = "\nsww_remote_log_lines_dropped_total" + - `{instance="fsn1app1/blog"} ` + `{instance="host1/blog"} ` if strings.Contains(metrics, dropped+"0\n") || !strings.Contains(metrics, dropped) { t.Errorf("no line dropped in the metrics:\n%s", metrics) @@ -1089,7 +1089,7 @@ func TestBanIsAlertedAndAnAlertNotSentIsKeptAcrossARestart(t *testing.T) { runUntilStopped(t, env, func(url string) { webhook.waitFor(t, "permanent_ban", true) - const ofWebhook = `{destination="webhook",instance="fsn1app1/blog"}` + const ofWebhook = `{destination="webhook",instance="host1/blog"}` metrics := metricsWith(t, url+"_sww/metrics", token, "\nsww_alerts_sent_total"+ofWebhook+" 1\n") @@ -1176,13 +1176,13 @@ func TestBanIsAlertedToSlackAndNtfy(t *testing.T) { err := json.Unmarshal([]byte(slackPost.body), &message) if err != nil || - !strings.HasPrefix(message.Text, "*fsn1app1/blog: ban*\n") || + !strings.HasPrefix(message.Text, "*host1/blog: ban*\n") || !strings.Contains(message.Text, "\nclient: "+client+"\n") { t.Errorf("Slack was sent %s", slackPost.body) } ntfyPost := ntfy.firstPost(t) - if ntfyPost.header.Get("Title") != "fsn1app1/blog: ban" || + if ntfyPost.header.Get("Title") != "host1/blog: ban" || ntfyPost.header.Get("Authorization") != "Bearer "+ntfyToken || !strings.Contains(ntfyPost.body, "\nclient: "+client+"\n") { t.Errorf("ntfy was sent %s, with the headers %v", ntfyPost.body, @@ -1194,9 +1194,9 @@ func TestBanIsAlertedToSlackAndNtfy(t *testing.T) { // test process cannot fail the test. sent := []string{ "\nsww_alerts_sent_total{destination=\"slack\"," + - "instance=\"fsn1app1/blog\"} 1\n", + "instance=\"host1/blog\"} 1\n", "\nsww_alerts_sent_total{destination=\"ntfy\"," + - "instance=\"fsn1app1/blog\"} 1\n", + "instance=\"host1/blog\"} 1\n", } metrics := metricsWith(t, url+"_sww/metrics", token, sent...) diff --git a/internal/waf/alerts_test.go b/internal/waf/alerts_test.go index c0c7616..42f1487 100644 --- a/internal/waf/alerts_test.go +++ b/internal/waf/alerts_test.go @@ -18,7 +18,9 @@ const ( alertWebhookURL = "SWW_WAF_ALERT_WEBHOOK_URL" alertMaxPerHour = "SWW_WAF_ALERT_MAX_PER_HOUR" // alertInstance is the instance every alert of these tests gives. - alertInstance = "fsn1app1/blog" + alertInstance = "host1/blog" + // alertWebhook is where the tests send alerts. + alertWebhook = "https://alerts.example/sww" ) func TestBanForABrokenLimitRaisesABanAlertWithItsNotes(t *testing.T) { @@ -270,7 +272,7 @@ func startAppWithAlerts( clk := &clock{now: time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)} settings := map[string]string{ trustedProxies: trustLocalhost, - alertWebhookURL: "https://alerts.example/sww", + alertWebhookURL: alertWebhook, instanceName: alertInstance, } maps.Copy(settings, env) diff --git a/internal/waf/answertime_test.go b/internal/waf/answertime_test.go index 408e9e6..3e184a0 100644 --- a/internal/waf/answertime_test.go +++ b/internal/waf/answertime_test.go @@ -20,7 +20,7 @@ import ( "github.com/prometheus/client_golang/prometheus/testutil" "sneak.berlin/go/smallwebwaf/internal/bans" - "sneak.berlin/go/smallwebwaf/internal/lookup" + "sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest" "sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/reputation" @@ -591,7 +591,7 @@ func TestAClientGeoJSPlacedInNoCountryCountsAsUnknownWithoutWaiting( s := startWAF(t, geojsURL, map[string]string{ unknownCountryLimitPercent: "0", rateLimitPerMinute: "1", }) - s.server.GeoJS.Load([]lookup.Answer{{ + s.server.GeoJS.Load([]geojs.Answer{{ Client: netip.MustParsePrefix(client + "/32"), ASN: "AS64512", Answered: s.clock.Now(), Used: s.clock.Now(), }}) @@ -770,6 +770,8 @@ func TestARefusedRequestDoesNotWaitForASlowGeoJS(t *testing.T) { // TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays says, // so that a request that does not wait takes no time at all. synctest.Test(t, func(t *testing.T) { + defer waitForGeoJSRequestsToBeAbandoned() + server, out := newWAFInBubble(t, tc.env) tc.prepare(t, server, time.Now()) @@ -838,6 +840,8 @@ func TestAQuestionWaitsForASlowGeoJSOnlyAsLongAsTheDefaultSays( t.Parallel() synctest.Test(t, func(t *testing.T) { + defer waitForGeoJSRequestsToBeAbandoned() + server, out := newWAFInBubble(t, tc.env) tc.prepare(t, server, time.Now()) @@ -950,10 +954,11 @@ func TestWAFAnswersWithinTheTimeLimitWhileEveryOutsideSourceHangs( t.Errorf("%d checks with AbuseIPDB, want 1", checked) } - // The query is answered, and the check with AbuseIPDB abandoned, - // before the bubble ends. + // The query is answered, and the check with AbuseIPDB and the request + // to GeoJS abandoned, before the bubble ends. releaseResolver() time.Sleep(defaultDNSBLAbuseIPDBWait) + waitForGeoJSRequestsToBeAbandoned() }) } diff --git a/internal/waf/besidelookup.go b/internal/waf/besidelookup.go new file mode 100644 index 0000000..d1edf44 --- /dev/null +++ b/internal/waf/besidelookup.go @@ -0,0 +1,172 @@ +package waf + +// The heuristics of besideLookup, and how they run beside the lookup. + +import ( + "context" + "sync/atomic" + "time" + + "sneak.berlin/go/smallwebwaf/internal/config" + "sneak.berlin/go/smallwebwaf/internal/coreruleset" + "sneak.berlin/go/smallwebwaf/internal/requestlog" + "sneak.berlin/go/smallwebwaf/internal/rules" +) + +// besideLookupTimeLimit is how long the heuristics of besideLookup have for a +// request, from when they start. They only read the request, and no request +// should take them that long. A request they have not finished by then is +// refused with 503, as ActionChecksTimedOut. +const besideLookupTimeLimit = 500 * time.Millisecond + +// besideLookupFindings is what the heuristics of besideLookup found of a +// request, before any of it is applied to the request. +type besideLookupFindings struct { + // trapPath is the request's path when it is in SWW_WAF_TRAP_PATHS, and "" + // otherwise. + trapPath string + // rules are the rules of the rule files the request matched. + rules []rules.Rule + // inspected is true when the Core Rule Set inspected the request. + // inspection, read and inspectionTime are then what it found, what it read + // of the body, and how long it took. + inspected bool + inspection coreruleset.Result + read []byte + inspectionTime time.Duration + // refuses is true when one of the heuristics refuses the request, or would + // in observe mode. Those after it did no work. + refuses bool +} + +// runBesideLookup runs the heuristics of besideLookup while the lookup waits +// for GeoJS, and returns the first verdict, or nil when none gives one. +// lookedUp is the channel the lookup's result comes on, or nil once afterLookup +// has let the request through. +// +// The first of the two to refuse the request ends the checks: waf answers at +// once, and what the other found is dropped. besideLookup has +// besideLookupTimeLimit; past it, the request is refused as +// ActionChecksTimedOut. Once both have let the request through, the heuristics +// of afterBesideLookup run. ctx is the request's own context. +func (rq *request) runBesideLookup( + ctx context.Context, lookedUp <-chan lookupResult, +) *verdict { + var running atomic.Pointer[string] + + running.Store(new(trapPaths{}.name())) + + found := make(chan *besideLookupFindings, 1) + + go func() { found <- rq.findBesideLookup(&running) }() + + timeUp := time.NewTimer(rq.h.besideLookupLimit) + defer timeUp.Stop() + + var findings *besideLookupFindings + + for findings == nil || lookedUp != nil { + select { + case result := <-lookedUp: + lookedUp = nil + + v := rq.runAfterLookup(ctx, result) + if v != nil { + return v + } + case findings = <-found: + found = nil + + timeUp.Stop() + + if findings.refuses { + return rq.applyFindings(ctx, findings) + } + case <-timeUp.C: + return rq.timedOut(*running.Load()) + } + } + + v := rq.applyFindings(ctx, findings) + if v != nil { + return v + } + + return rq.runHeuristics(ctx, afterBesideLookup(), rq.h.now()) +} + +// findBesideLookup does the work of the heuristics of besideLookup on the +// request, in their order, until one refuses it, and returns what they found. +// It changes nothing of the request or of what waf keeps, so that it can run +// while the lookup and afterLookup do. running holds the name of the heuristic +// at work, the first one's when findBesideLookup is called. +func (rq *request) findBesideLookup( + running *atomic.Pointer[string], +) *besideLookupFindings { + findings := &besideLookupFindings{} + + findings.trapPath = rq.trapPath() + if findings.trapPath != "" { + findings.refuses = true + + return findings + } + + running.Store(new(ruleFiles{}.name())) + + findings.rules = rq.h.rules.Match(rq.in) + if refusedByRule(findings.rules) { + findings.refuses = true + + return findings + } + + running.Store(new(coreRuleSet{}.name())) + + cfg := rq.h.config + if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff || + pathExempt(rq.in, cfg.CoreRuleSetExemptPaths) { + return findings + } + + start := time.Now() + // A question's body has been read into memory, so that reading it here + // cannot fail. + findings.inspection, findings.read, _ = rq.h.coreRuleSet.Inspect( + rq.in, rq.client, rq.in.Body, + ) + findings.inspectionTime = time.Since(start) + findings.inspected = true + findings.refuses = rq.refusedByCoreRuleSet(findings.inspection) + + return findings +} + +// applyFindings applies findings, what findBesideLookup found, to the +// request: it runs the heuristics of besideLookup, which read them, at the +// time it is called, and returns the first verdict, or nil when none gives +// one. +func (rq *request) applyFindings( + ctx context.Context, findings *besideLookupFindings, +) *verdict { + rq.findings = findings + + return rq.runHeuristics(ctx, besideLookup(), rq.h.now()) +} + +// timedOut refuses the request for the heuristics of besideLookup not having +// finished with it within besideLookupTimeLimit. running is the name of the +// one at work, which the log line's heuristic gives. The refusal makes no ban +// and is no offence. +func (rq *request) timedOut(running string) *verdict { + v := &verdict{ + action: requestlog.ActionChecksTimedOut, + figures: map[string]any{ + "time_limit_ms": rq.h.besideLookupLimit.Milliseconds(), + }, + } + rq.line.Heuristic = running + rq.line.HeuristicFigures = v.figures + + return v +} diff --git a/internal/waf/besidelookup_test.go b/internal/waf/besidelookup_test.go new file mode 100644 index 0000000..7ecb49e --- /dev/null +++ b/internal/waf/besidelookup_test.go @@ -0,0 +1,426 @@ +package waf_test + +import ( + "io" + "maps" + "net/http" + "net/http/httptest" + "net/netip" + "strconv" + "strings" + "sync/atomic" + "testing" + "testing/synctest" + "time" + + "github.com/prometheus/client_golang/prometheus/testutil" + + "sneak.berlin/go/smallwebwaf/internal/alerts" + "sneak.berlin/go/smallwebwaf/internal/coreruleset" + "sneak.berlin/go/smallwebwaf/internal/ratelimit" + "sneak.berlin/go/smallwebwaf/internal/requestlog" + "sneak.berlin/go/smallwebwaf/internal/waf" +) + +// checksTimeLimit is how long the trap paths, the rule files and the Core Rule +// Set have for a request, as SPEC.md gives it, and checksTimedOut the status of +// a request they have not finished by then. +const ( + checksTimeLimit = 500 * time.Millisecond + checksTimedOut = http.StatusServiceUnavailable +) + +// heldCoreRuleSet is a stand-in for the Core Rule Set: each inspection takes +// took, then finds result. asked counts the inspections. +type heldCoreRuleSet struct { + took time.Duration + result coreruleset.Result + asked atomic.Int64 +} + +// Inspect inspects a request as the stand-in does, and reads all of body. +func (c *heldCoreRuleSet) Inspect( + _ *http.Request, _ netip.Addr, body io.Reader, +) (coreruleset.Result, []byte, error) { + c.asked.Add(1) + time.Sleep(c.took) + + read, err := io.ReadAll(body) + + return c.result, read, err +} + +// coreRuleSetMatch returns what the stand-in finds of an attack: rule 942100, +// with a score of 5, the default SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD. +func coreRuleSetMatch() coreruleset.Result { + return coreruleset.Result{RuleIDs: []int{942100}, Score: 5} +} + +// newWAFHoldingTheCoreRuleSet makes a waf, to be asked in a synctest bubble, +// with crs in place of the Core Rule Set, in block mode. GeoJS never answers, +// a setting needs its answer, alerts go to a webhook, and everything else is +// at its default but for env. No request to it may use the network. +func newWAFHoldingTheCoreRuleSet( + t *testing.T, crs *heldCoreRuleSet, env map[string]string, +) (*waf.Server, *output, *alerts.Queue) { + t.Helper() + + settings := map[string]string{ + tagDeny: denyKP, coreRuleSetMode: block, + alertWebhookURL: alertWebhook, + } + maps.Copy(settings, env) + + server, out, queue := newWAFWith(t, func(params waf.Params) *waf.Server { + params.CoreRuleSet = crs + + return waf.New(params) + }, silentGeoJSURL, time.Now, settings) + server.SetBesideLookupTimeLimit(waf.BesideLookupTimeLimit) + + return server, out, queue +} + +// lookupWait is SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS for wait. +func lookupWait(wait time.Duration) string { + return strconv.FormatInt(wait.Milliseconds(), 10) +} + +// wantAnswer checks the status of an answer and how long it took. +func wantAnswer( + t *testing.T, status int, took time.Duration, + wantStatus int, wantTook time.Duration, out *output, +) { + t.Helper() + + if status != wantStatus || took != wantTook { + t.Errorf( + "answered %d after %s, want %d after %s; output:\n%s", + status, took, wantStatus, wantTook, out.text(), + ) + } +} + +// wantNoCoreRuleSetTrace checks that what the Core Rule Set found of a request +// left no trace: line has none of its fields, and server has counted no match +// of it and raised no alert for one. +func wantNoCoreRuleSetTrace( + t *testing.T, server *waf.Server, queue *alerts.Queue, line logLine, +) { + t.Helper() + + if line.DurationCoreRuleSetMS != nil || line.CoreRuleSetScore != nil || + line.CoreRuleSetRuleIDs != nil { + t.Errorf( + "log line has Core Rule Set time %v, score %v and rule ids %v, want none", + line.DurationCoreRuleSetMS, + line.CoreRuleSetScore, + line.CoreRuleSetRuleIDs, + ) + } + + recorder := httptest.NewRecorder() + server.Metrics.ServeHTTP( + recorder, + httptest.NewRequestWithContext( + t.Context(), + http.MethodGet, + waf.MetricsPath, + http.NoBody, + ), + ) + + if strings.Contains( + recorder.Body.String(), + "sww_core_rule_set_matches_total{", + ) { + t.Errorf("a Core Rule Set match is counted") + } + + for _, alert := range queue.Snapshot().Waiting[alerts.DestinationWebhook] { + if alert.Event == alerts.EventCoreRuleSetMatch { + t.Errorf("alert raised: %+v", alert) + } + } +} + +func TestTheLookupAndTheCoreRuleSetTakeTheLongerOfTheirTimesNotTheirSum( + t *testing.T, +) { + t.Parallel() + + inspection := checksTimeLimit - time.Millisecond + + for _, tc := range []struct { + name string + // wait is how long the request waits for GeoJS. + wait time.Duration + }{ + {"the lookup waits longer", defaultLookupWait}, + {"the inspection takes longer", inspection / 2}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + crs := &heldCoreRuleSet{took: inspection} + server, out, _ := newWAFHoldingTheCoreRuleSet( + t, + crs, + map[string]string{ + asnAndCountryLookupTimeout: lookupWait(tc.wait), + }, + ) + + synctest.Test(t, func(t *testing.T) { + defer waitForGeoJSRequestsToBeAbandoned() + + status, took := askDirectly( + t, + server, + func(*http.Request) {}, + ) + + wantAnswer(t, status, took, http.StatusOK, + max(tc.wait, inspection), out) + wantLine(t, out.requestLine(t), http.StatusOK, + requestlog.ActionForward) + }) + }) + } +} + +// firstToRefuse is a case of +// TestTheFirstOfTheLookupAndTheRequestChecksToRefuseIsAnswered: a request +// under env, whose inspection by the Core Rule Set takes inspection, refused as +// action after took. +type firstToRefuse struct { + name string + env map[string]string + inspection time.Duration + took time.Duration + action string +} + +// firstToRefuseCases are the cases of +// TestTheFirstOfTheLookupAndTheRequestChecksToRefuseIsAnswered. +func firstToRefuseCases() []firstToRefuse { + return []firstToRefuse{ + { + name: "a trap path, before GeoJS answers", + env: map[string]string{trapPaths: "/", allowedCountries: "DE"}, + took: 0, action: requestlog.ActionTemporarilyBanned, + }, + { + name: "the Core Rule Set, before GeoJS answers", + env: map[string]string{allowedCountries: "DE"}, + inspection: 100 * time.Millisecond, + took: 100 * time.Millisecond, action: requestlog.ActionBlockedByCoreRuleSet, + }, + { + name: "the country list, while the Core Rule Set inspects", + env: map[string]string{ + allowedCountries: "DE", asnAndCountryLookupTimeout: "100", + }, + inspection: 400 * time.Millisecond, + took: 100 * time.Millisecond, action: requestlog.ActionCountryDenied, + }, + { + name: "the rate limits, while the Core Rule Set inspects", + env: map[string]string{ + rateLimitPerMinute: "2", unknownCountryLimitPercent: "50", + asnAndCountryLookupTimeout: "100", + }, + inspection: 400 * time.Millisecond, + took: 100 * time.Millisecond, action: requestlog.ActionClientIsRateLimited, + }, + } +} + +func TestTheFirstOfTheLookupAndTheRequestChecksToRefuseIsAnswered( + t *testing.T, +) { + t.Parallel() + + const forbidden = http.StatusForbidden + + for _, tc := range firstToRefuseCases() { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + crs := &heldCoreRuleSet{ + took: tc.inspection, + result: coreRuleSetMatch(), + } + server, out, queue := newWAFHoldingTheCoreRuleSet(t, crs, tc.env) + + synctest.Test(t, func(t *testing.T) { + defer waitForGeoJSRequestsToBeAbandoned() + + // Under the rate limits of the last case, the client's next + // request takes it over a limit only at the percentage of an + // unknown country, so it waits for GeoJS. + countClient(t, server, time.Now()) + + status, took := askDirectly( + t, + server, + func(*http.Request) {}, + ) + + wantAnswer(t, status, took, forbidden, tc.took, out) + + line := out.requestLine(t) + wantLine(t, line, forbidden, tc.action) + + if tc.action == requestlog.ActionBlockedByCoreRuleSet { + return + } + + if tc.action != requestlog.ActionClientIsRateLimited && + line.Counts != (ratelimit.Counts{}) { + t.Errorf( + "the rate limits counted the request: %+v", + line.Counts, + ) + } + + // What the Core Rule Set finds once the request is answered is + // dropped. + time.Sleep(tc.inspection) + wantNoCoreRuleSetTrace(t, server, queue, line) + }) + }) + } +} + +func TestARequestRefusedBeforeTheLookupStartsNoInspection(t *testing.T) { + t.Parallel() + + crs := &heldCoreRuleSet{took: time.Second} + server, out, _ := newWAFHoldingTheCoreRuleSet(t, crs, nil) + + synctest.Test(t, func(t *testing.T) { + banClient(t, server, time.Now()) + + status, took := askDirectly(t, server, func(*http.Request) {}) + + wantAnswer(t, status, took, http.StatusForbidden, 0, out) + + requests := testutil.ToFloat64(server.Metrics.GeoJSRequests) + if asked := crs.asked.Load(); asked != 0 || requests != 0 { + t.Errorf( + "%d inspections and %v requests to GeoJS, want none", + asked, + requests, + ) + } + }) +} + +func TestAClientInTheAllowNetsIsAllowedAtOnceAndNothingElseStarts( + t *testing.T, +) { + t.Parallel() + + crs := &heldCoreRuleSet{took: time.Second, result: coreRuleSetMatch()} + server, out, _ := newWAFHoldingTheCoreRuleSet(t, crs, + map[string]string{allowNets: client, allowedCountries: "DE"}) + + synctest.Test(t, func(t *testing.T) { + status, took := askDirectly(t, server, func(*http.Request) {}) + + wantAnswer(t, status, took, http.StatusOK, 0, out) + + requests := testutil.ToFloat64(server.Metrics.GeoJSRequests) + if asked := crs.asked.Load(); asked != 0 || requests != 0 { + t.Errorf( + "%d inspections and %v requests to GeoJS, want none", + asked, + requests, + ) + } + }) +} + +func TestARequestTheCoreRuleSetHasNotFinishedWithIn500MillisecondsIsRefused( + t *testing.T, +) { + t.Parallel() + + crs := &heldCoreRuleSet{took: checksTimeLimit + time.Second} + server, out, _ := newWAFHoldingTheCoreRuleSet(t, crs, + map[string]string{refusalsLimitPerMinute: "1"}) + + synctest.Test(t, func(t *testing.T) { + defer waitForGeoJSRequestsToBeAbandoned() + + // Were it counted toward the refusal limit, the second refusal would + // ban the client, and the third request would be refused for the ban. + for i := range 3 { + status, took := askDirectly(t, server, func(req *http.Request) { + req.Header.Set(requestIDHeader, "id-"+strconv.Itoa(i)) + }) + + wantAnswer(t, status, took, checksTimedOut, checksTimeLimit, out) + + line := out.requestLines(t, i+1)[i] + wantLine( + t, + line, + checksTimedOut, + requestlog.ActionChecksTimedOut, + ) + wantHeuristic(t, line, "core_rule_set", `{"time_limit_ms": 500}`) + + if line.Offence != "" || line.BanExpires != "" { + t.Errorf( + "log line has offence %q and ban_expires %q, want neither", + line.Offence, + line.BanExpires, + ) + } + } + + if bans := server.Ledger.Snapshot(); len(bans) != 0 { + t.Errorf("bans %+v, want none", bans) + } + + offences := historyOf(t, server, client).Offences + if offences != (ratelimit.Offences{}) { + t.Errorf("history counts the offences %+v, want none", offences) + } + + time.Sleep(crs.took) + }) +} + +func TestObserveModeAllowsARequestTheCoreRuleSetHasNotFinishedWith( + t *testing.T, +) { + t.Parallel() + + crs := &heldCoreRuleSet{took: checksTimeLimit + time.Second} + server, out, _ := newWAFHoldingTheCoreRuleSet(t, crs, + map[string]string{mode: observe}) + + synctest.Test(t, func(t *testing.T) { + defer waitForGeoJSRequestsToBeAbandoned() + + status, took := askDirectly(t, server, func(*http.Request) {}) + + wantAnswer(t, status, took, http.StatusOK, checksTimeLimit, out) + + line := out.requestLine(t) + wantLine(t, line, http.StatusOK, requestlog.ActionForward) + + if line.WouldAction != requestlog.ActionChecksTimedOut { + t.Errorf( + "log line has would_action %q, want %q", + line.WouldAction, + requestlog.ActionChecksTimedOut, + ) + } + + time.Sleep(crs.took) + }) +} diff --git a/internal/waf/biased_test.go b/internal/waf/biased_test.go index 17dadf7..0835a4f 100644 --- a/internal/waf/biased_test.go +++ b/internal/waf/biased_test.go @@ -339,6 +339,8 @@ func TestClientWithoutAnAnswerInTimeHasTheUnknownCountryLimitPercent( // TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays says, with // a GeoJS that never answers. synctest.Test(t, func(t *testing.T) { + defer waitForGeoJSRequestsToBeAbandoned() + server, out, _ := newWAF(t, silentGeoJSURL, time.Now, map[string]string{unknownCountryLimitPercent: "0"}) @@ -390,6 +392,8 @@ func TestRequestWaitsForItsLookupWhileABiasedThresholdIsSet(t *testing.T) { // with a GeoJS that never answers. The headers, which need the // answer too, are off. synctest.Test(t, func(t *testing.T) { + defer waitForGeoJSRequestsToBeAbandoned() + server, out, _ := newWAF(t, silentGeoJSURL, time.Now, map[string]string{ asnAndCountryLookupTimeout: strconv.FormatInt( diff --git a/internal/waf/coreruleset.go b/internal/waf/coreruleset.go index e05ca11..4acc2cc 100644 --- a/internal/waf/coreruleset.go +++ b/internal/waf/coreruleset.go @@ -13,10 +13,11 @@ import ( "sneak.berlin/go/smallwebwaf/internal/requestlog" ) -// coreRuleSet inspects the request with the Core Rule Set, unless -// SWW_WAF_CORE_RULE_SET_MODE is off or SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS -// exempts its path, as pathExempt decides, and notes the rules it matched and -// its score in the log line, and the rules in the metrics. A score at or over +// coreRuleSet takes what the Core Rule Set found of the request, which +// findBesideLookup inspected with it unless SWW_WAF_CORE_RULE_SET_MODE is off +// or SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS exempts its path, as pathExempt +// decides. It notes the rules the request matched and its score in the log +// line, and the rules in the metrics. A score at or over // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD is a match: it adds the tag // exploitsignature, raises the core_rule_set_match alert, and in block mode // refuses the request with 403, which is an offence its client's history @@ -30,21 +31,17 @@ func (coreRuleSet) check( rq *request, _ time.Time, ) *verdict { - cfg := rq.h.config - if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff || - pathExempt(rq.in, cfg.CoreRuleSetExemptPaths) { + findings := rq.findings + if !findings.inspected { return nil } - start := time.Now() - - // A question's body has been read into memory, so that reading it here - // cannot fail. - result, read, _ := rq.h.coreRuleSet.Inspect(rq.in, rq.client, rq.in.Body) - rq.requestBytes = int64(len(read)) + cfg := rq.h.config + result := findings.inspection + rq.requestBytes = int64(len(findings.read)) rq.line.DurationCoreRuleSetMS = new( - requestlog.Milliseconds(time.Since(start)), + requestlog.Milliseconds(findings.inspectionTime), ) rq.line.CoreRuleSetRuleIDs = result.RuleIDs rq.line.CoreRuleSetScore = &result.Score @@ -61,7 +58,7 @@ func (coreRuleSet) check( rq.addTag(config.ExploitSignatureTag) rq.alertCoreRuleSetMatch(result) - if cfg.CoreRuleSetMode == config.CoreRuleSetModeDetect { + if !rq.refusedByCoreRuleSet(result) { return nil } @@ -76,6 +73,17 @@ func (coreRuleSet) check( } } +// refusedByCoreRuleSet reports whether coreRuleSet refuses a request the Core +// Rule Set found result of: whether the score is at or over +// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD in block mode. +func (rq *request) refusedByCoreRuleSet(result coreruleset.Result) bool { + cfg := rq.h.config + threshold := cfg.CoreRuleSetScoreThreshold + + return cfg.CoreRuleSetMode == config.CoreRuleSetModeBlock && + threshold != 0 && result.Score >= threshold +} + // alertCoreRuleSetMatch raises the core_rule_set_match alert for the request, // which the Core Rule Set scored at result, at or over // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD. Its detail gives the rule ids, the diff --git a/internal/waf/export_test.go b/internal/waf/export_test.go new file mode 100644 index 0000000..60b6e9b --- /dev/null +++ b/internal/waf/export_test.go @@ -0,0 +1,13 @@ +package waf + +import "time" + +// BesideLookupTimeLimit is besideLookupTimeLimit. +const BesideLookupTimeLimit = besideLookupTimeLimit + +// SetBesideLookupTimeLimit gives the heuristics of besideLookup limit on s in +// place of besideLookupTimeLimit. +func (s *Server) SetBesideLookupTimeLimit(limit time.Duration) { + h := s.Handler.(*handler) //nolint:forcetypeassert // New makes it one + h.besideLookupLimit = limit +} diff --git a/internal/waf/heuristics.go b/internal/waf/heuristics.go index 76153e2..9fb7e59 100644 --- a/internal/waf/heuristics.go +++ b/internal/waf/heuristics.go @@ -54,8 +54,6 @@ func beforeLookup() []heuristic { } // afterLookup are the heuristics that run once the lookup has been made. -// tag_deny runs twice: before the rate limits, on the tags known by then, and -// last, on the tag rule_files and core_rule_set can add. func afterLookup() []heuristic { return []heuristic{ countries{}, @@ -64,9 +62,26 @@ func afterLookup() []heuristic { abuseIPDB{}, tagDeny{}, rateLimits{}, + } +} + +// besideLookup are the heuristics that only read the request. Their work runs +// beside the lookup and afterLookup, in findBesideLookup, which changes nothing +// of the request. Their check then applies what that work found, and runs only +// once waf takes it. +func besideLookup() []heuristic { + return []heuristic{ trapPaths{}, ruleFiles{}, coreRuleSet{}, + } +} + +// afterBesideLookup are the heuristics that run once afterLookup and +// besideLookup have both let the request through. tag_deny runs a second time, +// on the tag rule_files and core_rule_set can add. +func afterBesideLookup() []heuristic { + return []heuristic{ tagDeny{}, } } @@ -95,7 +110,7 @@ func (rq *request) runAfterAnswer(ctx context.Context) { rq.runBanningHeuristics(ctx, afterAnswer()) } -// runHeuristics runs list, beforeLookup or afterLookup, on the request at now, +// runHeuristics runs list, one of those checkClient runs, on the request at now, // and returns the first verdict, or nil when none gives one. The log line's // heuristic and heuristic_figures name the heuristic that gave it, and give its // figures. ctx is the request's own context. diff --git a/internal/waf/lookup.go b/internal/waf/lookup.go index 8627cd2..8250667 100644 --- a/internal/waf/lookup.go +++ b/internal/waf/lookup.go @@ -7,8 +7,9 @@ import ( "strings" "time" + "sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/config" - "sneak.berlin/go/smallwebwaf/internal/lookup" + "sneak.berlin/go/smallwebwaf/internal/geojs" ) // The headers in which an answer of allow gives rproxy the client's AS number @@ -20,33 +21,80 @@ const ( countryHeader = "X-Client-Country" ) -// lookUp looks up the client's AS number and country, in the lookup -// database or through GeoJS, and notes them for the log line and as the -// request's asn= and country= tags, unless -// SWW_WAF_LOOKUP_SOURCE is off or the client is on a private, loopback or -// link-local address, which no lookup can place. The lookup database -// answers at once. With GeoJS, while a setting needs the answer, such as a -// country list or a biased threshold, a new client's request waits for it, -// unless refusedWhateverTheAnswer reports true. ctx is the request's own -// context. -func (rq *request) lookUp(ctx context.Context, now time.Time) { +// lookupResult is what the lookup gave for a request: whether its client was +// looked up, the answer, and whether the request skipped the wait for GeoJS. +type lookupResult struct { + lookedUp, skippedWait bool + answer geojs.Answer +} + +// startLookUp starts looking up the client's AS number and country, in the +// lookup database or through GeoJS, unless SWW_WAF_LOOKUP_SOURCE is off or the +// client is on a private, loopback or link-local address, which no lookup can +// place. It returns the channel the result comes on. A result that needs no +// wait is on it when startLookUp returns: the lookup database's, an answer +// GeoJS gave earlier, or none. With GeoJS, while a setting needs the answer, +// such as a country list or a biased threshold, a new client's request waits +// for it up to SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS, unless +// refusedWhateverTheAnswer reports true. The wait goes on to its end even once +// the request has been answered, so that it ends as it would have for a +// request that waited. startLookUp changes nothing of the request. ctx is the +// request's own context. +func (rq *request) startLookUp( + ctx context.Context, now time.Time, +) <-chan lookupResult { + result := make(chan lookupResult, 1) + if rq.h.config.LookupSource == "off" || !canBePlaced(rq.client) { - return + result <- lookupResult{} + + return result } client := rq.h.clientGroup(rq.client) - switch { - case rq.h.config.LookupSource == "file": - rq.lookupAnswer = rq.h.lookupFile.LookUp(client) - case rq.refusedWhateverTheAnswer(now): - rq.lookupAnswer = rq.h.geojs.LookUpWithoutWaiting(ctx, client) - rq.skippedLookupWait = rq.lookupAnswer.Answered.IsZero() - default: - rq.lookupAnswer = rq.h.geojs.LookUp(ctx, client) + if rq.h.config.LookupSource == "file" { + result <- lookupResult{ + lookedUp: true, answer: rq.h.lookupFile.LookUp(client), + } + + return result + } + + answer := rq.h.geojs.LookUpWithoutWaiting(ctx, client) + refused := rq.refusedWhateverTheAnswer(now) + + if !answer.Answered.IsZero() || refused || !rq.h.waitForGeoJS { + result <- lookupResult{ + lookedUp: true, + skippedWait: refused && answer.Answered.IsZero(), + answer: answer, + } + + return result + } + + go func() { + result <- lookupResult{ + lookedUp: true, + answer: rq.h.geojs.LookUp(context.WithoutCancel(ctx), client, + rq.h.config.ASNAndCountryLookupTimeout), + } + }() + + return result +} + +// noteLookup notes result for the log line and as the request's asn= and +// country= tags. +func (rq *request) noteLookup(result lookupResult) { + if !result.lookedUp { + return } rq.lookedUp = true + rq.lookupAnswer = result.answer + rq.skippedLookupWait = result.skippedWait rq.line.ASN = rq.lookupAnswer.ASN rq.line.ASName = rq.lookupAnswer.ASName rq.line.Country = rq.lookupAnswer.Country @@ -83,13 +131,29 @@ func countryTag(country string) string { // addLookup adds answer, an answer about a client from the lookup // database or GeoJS, to the client's history, and to the notes of the bans // on its netblock that have no AS number, AS name or country yet. -func (h *handler) addLookup(answer lookup.Answer) { +func (h *handler) addLookup(answer geojs.Answer) { h.limiter.AddLookup(answer.Client, answer.Answered, answer.ASN, answer.ASName, answer.Country) h.ledger.AddLookup(h.netblock(answer.Client.Addr()), answer.ASN, answer.ASName, answer.Country) } +// geojsFailed logs a failure of GeoJS, raises a source_failure alert for it +// and counts it. askingAgainIn is how long GeoJS is left alone after it. +func (h *handler) geojsFailed(err error, askingAgainIn time.Duration) { + h.processLog.Warn("asking GeoJS failed", + "error", err.Error(), "asking_again_in", askingAgainIn.String()) + h.alerts.Raise(alerts.Alert{ + Event: alerts.EventSourceFailure, + Reason: "asking GeoJS failed", + Detail: map[string]any{ + "source": "geojs", "error": err.Error(), + "asking_again_in": askingAgainIn.String(), + }, + }) + h.metrics.GeoJSFailures.Inc() +} + // setLookupHeaders sets the headers that give the client's AS number and // country, leaving out one that is unknown. func setLookupHeaders(header http.Header, asn, country string) { diff --git a/internal/waf/lookup_test.go b/internal/waf/lookup_test.go index 407c87c..3c2d413 100644 --- a/internal/waf/lookup_test.go +++ b/internal/waf/lookup_test.go @@ -12,7 +12,7 @@ import ( "time" "sneak.berlin/go/smallwebwaf/internal/alerts" - "sneak.berlin/go/smallwebwaf/internal/lookup" + "sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest" "sneak.berlin/go/smallwebwaf/internal/requestlog" ) @@ -138,7 +138,7 @@ func TestRequestThatWaitedForItsLookupIsCountedWhenItGoesOn(t *testing.T) { // of lookups.json would keep it, while the first request still waits // for GeoJS. The second request goes on at once, and is counted. clk.advance(2500 * time.Millisecond) - server.GeoJS.Load([]lookup.Answer{{ + server.GeoJS.Load([]geojs.Answer{{ Client: netip.MustParsePrefix( fromDE + "/32", ), ASN: asnDE, ASName: asNameDE, @@ -190,7 +190,7 @@ func TestASNumberAndNameInTheLogLineTheHistoryTheBanNotesAndTheAlert( clk.Now, map[string]string{ trustedProxies: trustLocalhost, - alertWebhookURL: "https://alerts.example/sww", + alertWebhookURL: alertWebhook, rateLimitPerMinute: "1", }, ) @@ -199,7 +199,7 @@ func TestASNumberAndNameInTheLogLineTheHistoryTheBanNotesAndTheAlert( // The answer is kept before the requests, so GeoJS is not asked, and // gives no answer of its own. netblock := netip.MustParsePrefix(fromDE + "/32") - server.GeoJS.Load([]lookup.Answer{{ + server.GeoJS.Load([]geojs.Answer{{ Client: netblock, ASN: asnDE, ASName: asNameDE, Country: "DE", Answered: clk.Now(), Used: clk.Now(), }}) @@ -249,7 +249,7 @@ func TestLookupSourceOffLooksNoClientUp(t *testing.T) { ) // Even an answer kept from before is not used. - server.GeoJS.Load([]lookup.Answer{{ + server.GeoJS.Load([]geojs.Answer{{ Client: netip.MustParsePrefix( fromDE + "/32", ), ASN: asnDE, ASName: asNameDE, @@ -450,6 +450,8 @@ func TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays( // network, which would keep that clock from moving on: the request is // handed to waf's handler, and GeoJS is one that never answers. synctest.Test(t, func(t *testing.T) { + defer waitForGeoJSRequestsToBeAbandoned() + // Not the default. The exclusive list needs the answer, and the app // is never reached. const timeout = 3 * time.Second @@ -495,6 +497,17 @@ func TestMain(m *testing.M) { m.Run() } +// geojsRequestTimeout is how long a request to GeoJS may take before it is +// abandoned. +const geojsRequestTimeout = 10 * time.Second + +// waitForGeoJSRequestsToBeAbandoned sleeps until every request to the GeoJS at +// silentGeoJSURL made so far has been abandoned. A synctest bubble that asks +// it defers this: the bubble cannot end while a request waits on it. +func waitForGeoJSRequestsToBeAbandoned() { + time.Sleep(geojsRequestTimeout) +} + // silentGeoJS is the GeoJS at silentGeoJSURL. type silentGeoJS struct{} diff --git a/internal/waf/metrics_test.go b/internal/waf/metrics_test.go index bf96455..dac7175 100644 --- a/internal/waf/metrics_test.go +++ b/internal/waf/metrics_test.go @@ -1,6 +1,7 @@ package waf_test import ( + "fmt" "io" "net/http" "net/http/httptest" @@ -10,8 +11,9 @@ import ( "testing" "time" + "sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/bans" - "sneak.berlin/go/smallwebwaf/internal/lookup" + "sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/waf" ) @@ -248,7 +250,7 @@ func TestMetricsByCountryKeepTheBusiestAndCountTheRestAsOther(t *testing.T) { // The answers are kept before the requests, so that none waits for // GeoJS. - server.GeoJS.Load([]lookup.Answer{ + server.GeoJS.Load([]geojs.Answer{ keptAnswer(fromKP, "KP"), keptAnswer(fromDE, "DE"), keptAnswer(fromFR, "FR"), }) @@ -329,13 +331,13 @@ func TestMetricsByASNumberKeepTheBusiestAndCountTheRestAsOther( // The answers are kept before the requests, so that GeoJS gives none // of its own. Each client is in an AS of its own. - answer := func(addr, asn string) lookup.Answer { - return lookup.Answer{ + answer := func(addr, asn string) geojs.Answer { + return geojs.Answer{ Client: netip.MustParsePrefix(addr + "/32"), ASN: asn, Answered: clk.Now(), Used: clk.Now(), } } - server.GeoJS.Load([]lookup.Answer{ + server.GeoJS.Load([]geojs.Answer{ answer(fromDE, "AS64501"), answer(fromKP, "AS64502"), }) @@ -357,21 +359,23 @@ func TestMetricsByASNumberKeepTheBusiestAndCountTheRestAsOther( `sww_asn_requests_total{asn="AS64502",instance="app"}`) } -func TestMetricsCountGeoJSRequestsAndFailures(t *testing.T) { +func TestGeoJSFailureIsCountedLoggedAndAlerted(t *testing.T) { t.Parallel() - geojs := httptest.NewServer(http.HandlerFunc( + failingGeoJS := httptest.NewServer(http.HandlerFunc( func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusServiceUnavailable) })) - t.Cleanup(geojs.Close) + t.Cleanup(failingGeoJS.Close) app := startApp(t, func(http.ResponseWriter, *http.Request) {}) - addr, _, server := startProxyWithClock(t, app.URL, geojs.URL, time.Now, + addr, out, server, queue := startProxyWithAlerts(t, app.URL, + failingGeoJS.URL, time.Now, map[string]string{ - trustedProxies: trustLocalhost, - metricsToken: token, - tagDeny: denyKP, + trustedProxies: trustLocalhost, + metricsToken: token, + tagDeny: denyKP, + alertWebhookURL: "https://alerts.example/waf", }) // GeoJS fails, so the client has no country= tag, and SWW_WAF_TAG_DENY does @@ -401,14 +405,46 @@ func TestMetricsCountGeoJSRequestsAndFailures(t *testing.T) { `sww_requests_without_geojs_answer_total{instance="app"}`, 1, ) + + // The process log's line names the failure but not the address asked + // about. + logged := "" + + for line := range strings.Lines(out.text()) { + if strings.Contains(line, "asking GeoJS failed") { + logged = line + } + } + + if logged == "" || strings.Contains(logged, fromDE) { + t.Errorf( + "logged %q, want the failure without the address asked about", + logged, + ) + } + + // fmt prints a map sorted by its keys. + const want = "map[asking_again_in:1s " + + "error:GeoJS answered 503 Service Unavailable source:geojs]" + + waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook] + if len(waiting) != 1 || waiting[0].Event != alerts.EventSourceFailure || + waiting[0].Reason != "asking GeoJS failed" || + fmt.Sprint(waiting[0].Detail) != want { + t.Errorf( + "alerts waiting %+v, want one source_failure alert with %v", + waiting, + want, + ) + } } // keptAnswer returns GeoJS's answer that the client at addr is in // country, given now. -func keptAnswer(addr, country string) lookup.Answer { +func keptAnswer(addr, country string) geojs.Answer { now := time.Now() - return lookup.Answer{ + return geojs.Answer{ Client: netip.MustParsePrefix(addr + "/32"), Country: country, Answered: now, Used: now, } diff --git a/internal/waf/proxy.go b/internal/waf/proxy.go index 59500fe..9179ae4 100644 --- a/internal/waf/proxy.go +++ b/internal/waf/proxy.go @@ -8,6 +8,7 @@ import ( "io" "log/slog" "net/http" + "net/netip" "time" lru "github.com/hashicorp/golang-lru/v2" @@ -17,6 +18,7 @@ import ( "sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/coreruleset" + "sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/lookup" "sneak.berlin/go/smallwebwaf/internal/metrics" "sneak.berlin/go/smallwebwaf/internal/ratelimit" @@ -55,7 +57,7 @@ type Params struct { // ProcessLog receives the process's own messages. ProcessLog *slog.Logger // GeoJSURL is where clients' AS numbers and countries are looked up - // while SWW_WAF_LOOKUP_SOURCE is geojs, normally lookup.URL. + // while SWW_WAF_LOOKUP_SOURCE is geojs, normally geojs.URL. GeoJSURL string // AbuseIPDBURL is where clients are checked with AbuseIPDB while // SWW_WAF_ABUSEIPDB_KEY is set, normally reputation.AbuseIPDBURL. @@ -65,7 +67,7 @@ type Params struct { LookupFile *lookup.File // CoreRuleSet is what NewCoreRuleSet returns: what inspects requests // while SWW_WAF_CORE_RULE_SET_MODE is not off, and nil while it is. - CoreRuleSet *coreruleset.CoreRuleSet + CoreRuleSet CoreRuleSetInspector // Now tells the time by which requests are counted for the rate // limits, bans are made and run out, and GeoJS's answers are kept, // normally time.Now in UTC, the time the state files give. @@ -83,6 +85,14 @@ type Params struct { Alerts *alerts.Queue } +// CoreRuleSetInspector inspects a request with the Core Rule Set, as +// coreruleset.CoreRuleSet's Inspect says. Tests give waf a stand-in for it. +type CoreRuleSetInspector interface { + Inspect( + r *http.Request, client netip.Addr, body io.Reader, + ) (coreruleset.Result, []byte, error) +} + // Server is the server waf runs, with the parts whose state the state // files keep, the lookup database, nil unless SWW_WAF_LOOKUP_SOURCE is file, the // lists fetched from URLs, which its Run fetches, the DNSBL zones' verdicts, @@ -93,7 +103,7 @@ type Server struct { Ledger *bans.Ledger Limiter *ratelimit.Limiter - GeoJS *lookup.GeoJS + GeoJS *geojs.GeoJS Anomalies *anomaly.Counters LookupFile *lookup.File Lists *reputation.Lists @@ -157,6 +167,8 @@ func newHandler(params Params) *handler { rules: params.Rules, coreRuleSet: params.CoreRuleSet, alerts: params.Alerts, + + besideLookupLimit: besideLookupTimeLimit, } if params.Config.RouteWatchURL != "" { h.routeWatch = routewatch.New(routewatch.Params{ @@ -164,20 +176,19 @@ func newHandler(params Params) *handler { }) } - h.geojs = lookup.New(lookup.Params{ - URL: params.GeoJSURL, - Timeout: params.Config.ASNAndCountryLookupTimeout, - // The country list, the headers, the tag settings that name an asn= - // or country= tag and the biased thresholds act on the answer before - // the request goes on. - Wait: len(params.Config.ExclusivelyAllowedCountries) > 0 || - params.Config.AddLookupHeaders || params.Config.TagsNeedLookup() || - biasedThresholdsSet(params.Config), - Answered: h.addLookup, - Now: params.Now, - ProcessLog: params.ProcessLog, - Metrics: m, - Alerts: params.Alerts, + // The country list, the headers, the tag settings that name an asn= or + // country= tag and the biased thresholds act on the answer before the + // request goes on. + h.waitForGeoJS = len(params.Config.ExclusivelyAllowedCountries) > 0 || + params.Config.AddLookupHeaders || params.Config.TagsNeedLookup() || + biasedThresholdsSet(params.Config) + h.geojs = geojs.New(geojs.Params{ + URL: params.GeoJSURL, + Answered: h.addLookup, + Now: params.Now, + Asked: m.GeoJSRequests.Inc, + Failed: h.geojsFailed, + WentWithoutAnswer: m.RequestsWithoutGeoJSAnswer.Inc, }) m.AddBansAndClients(h.ledger, h.limiter, params.Now) m.AddRules(params.Rules) @@ -228,12 +239,9 @@ func newReputation( // without the rules SWW_WAF_CORE_RULE_SET_DISABLED_RULES switches off, reading bodies // up to SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE, with the exclusions in // SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE and the fixes of the apps in SWW_WAF_APPS, -// or nil while SWW_WAF_CORE_RULE_SET_MODE is off. It -// reads at most QuestionBodyMaxBytes less one byte: a question's body of -// QuestionBodyMaxBytes may be cut short, and the Core Rule Set does not read a -// JSON or XML body over its limit. An exclusions file that coreruleset.New refuses is -// an error in every mode. While the mode is off, a set file is loaded with the -// Core Rule Set only to check it. +// or nil while SWW_WAF_CORE_RULE_SET_MODE is off. An exclusions file that +// coreruleset.New refuses is an error in every mode. While the mode is off, a +// set file is loaded with the Core Rule Set only to check it. func NewCoreRuleSet(cfg *config.WAF) (*coreruleset.CoreRuleSet, error) { if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff && cfg.CoreRuleSetExclusionsFile == "" { @@ -241,14 +249,11 @@ func NewCoreRuleSet(cfg *config.WAF) (*coreruleset.CoreRuleSet, error) { } coreRuleSet, err := coreruleset.New(coreruleset.Params{ - ParanoiaLevel: cfg.CoreRuleSetParanoiaLevel, - DisabledRules: cfg.CoreRuleSetDisabledRules, - BodyInspectionSize: min( - cfg.CoreRuleSetBodyInspectionSize, - QuestionBodyMaxBytes-1, - ), - ExclusionsFile: cfg.CoreRuleSetExclusionsFile, - Apps: cfg.Apps, + ParanoiaLevel: cfg.CoreRuleSetParanoiaLevel, + DisabledRules: cfg.CoreRuleSetDisabledRules, + BodyInspectionSize: cfg.CoreRuleSetBodyInspectionSize, + ExclusionsFile: cfg.CoreRuleSetExclusionsFile, + Apps: cfg.Apps, }) if err != nil { return nil, fmt.Errorf( @@ -274,7 +279,7 @@ type handler struct { metrics *metrics.Metrics limiter *ratelimit.Limiter ledger *bans.Ledger - geojs *lookup.GeoJS + geojs *geojs.GeoJS anomalies *anomaly.Counters lookupFile *lookup.File lists *reputation.Lists @@ -282,8 +287,13 @@ type handler struct { abuseIPDB *reputation.AbuseIPDB routeWatch *routewatch.Client rules *rules.Files - coreRuleSet *coreruleset.CoreRuleSet + coreRuleSet CoreRuleSetInspector alerts *alerts.Queue + // waitForGeoJS is true when a setting needs each request's answer from + // GeoJS before the request goes on. + waitForGeoJS bool + // besideLookupLimit is besideLookupTimeLimit. Only tests change it. + besideLookupLimit time.Duration // awaitingReport holds the decision on each answered question by its id // until its report comes, and nil for a question still being handled. awaitingReport *lru.Cache[string, *decision] diff --git a/internal/waf/proxy_test.go b/internal/waf/proxy_test.go index e32b864..054b46b 100644 --- a/internal/waf/proxy_test.go +++ b/internal/waf/proxy_test.go @@ -512,6 +512,11 @@ func newWAFWith( Alerts: alertQueue, }) + // The race detector and a busy host can slow the Core Rule Set's + // inspections past the time limit. The tests of the limit set it back, and + // hold the Core Rule Set with a stand-in. + server.SetBesideLookupTimeLimit(time.Hour) + return server, out, alertQueue } diff --git a/internal/waf/ratelimits_test.go b/internal/waf/ratelimits_test.go index 47aca6f..962668a 100644 --- a/internal/waf/ratelimits_test.go +++ b/internal/waf/ratelimits_test.go @@ -5,7 +5,7 @@ import ( "sync/atomic" "testing" - "sneak.berlin/go/smallwebwaf/internal/lookup" + "sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/requestlog" ) @@ -162,7 +162,7 @@ func TestRateLimitExemptPathsAreNeitherCountedNorRefused(t *testing.T) { // The answers are kept before the requests, so that none waits for // GeoJS. - server.GeoJS.Load([]lookup.Answer{ + server.GeoJS.Load([]geojs.Answer{ keptAnswer(client, "DE"), keptAnswer(fromKP, "KP"), }) diff --git a/internal/waf/request.go b/internal/waf/request.go index c945d32..06bec96 100644 --- a/internal/waf/request.go +++ b/internal/waf/request.go @@ -11,7 +11,7 @@ import ( "sneak.berlin/go/smallwebwaf/internal/anomaly" "sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/config" - "sneak.berlin/go/smallwebwaf/internal/lookup" + "sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/rules" @@ -42,7 +42,7 @@ type decision struct { // looked up, whether or not an answer was there, and lookupAnswer is // what the lookup gave then, the zero Answer while GeoJS had given none. lookedUp bool - lookupAnswer lookup.Answer + lookupAnswer geojs.Answer // skippedLookupWait is true for a request that did not wait for GeoJS and // found no answer kept. SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT does not apply to it, // so that not waiting never lowers its client's limits. @@ -81,6 +81,9 @@ type request struct { checked time.Time // refused is what the request was refused with, nil if it was not. refused *refusal + // findings is what the heuristics of besideLookup found of the request, + // once waf has taken it, and nil before. + findings *besideLookupFindings // requestBytes is how many bytes of the request's body have been read. requestBytes int64 } @@ -177,9 +180,9 @@ func requestHeaders(r *http.Request, names []string) map[string]string { // check is the one place where a request can be refused. It returns nil to let // the request through. The heuristics of checkClient decide: a request they // refuse is answered with SWW_WAF_BAN_RESPONSE, 403 for a block rule or the Core -// Rule Set, or 414 or 431 for the header limits. In observe mode nothing is refused, -// and the log line names what enforce mode would have done. ctx is the -// request's own context. +// Rule Set, 503 for heuristics that ran out of time, or 414 or 431 for the +// header limits. In observe mode nothing is refused, and the log line names +// what enforce mode would have done. ctx is the request's own context. func (rq *request) check(ctx context.Context) *refusal { v := rq.checkClient(ctx) if v == nil { @@ -196,6 +199,11 @@ func (rq *request) check(ctx context.Context) *refusal { case action == requestlog.ActionBlockedByBlockRule || action == requestlog.ActionBlockedByCoreRuleSet: return &refusal{status: http.StatusForbidden, action: action} + case action == requestlog.ActionChecksTimedOut: + return &refusal{ + status: http.StatusServiceUnavailable, + action: action, + } case action == requestlog.ActionRequestLineTooLong: return &refusal{status: http.StatusRequestURITooLong, action: action} case action == requestlog.ActionRequestHeaderTooLarge: @@ -208,11 +216,16 @@ func (rq *request) check(ctx context.Context) *refusal { } } -// checkClient runs the heuristics of beforeLookup on the request, then looks -// up the client's AS number and country, then runs those of afterLookup. It -// returns the first verdict, or nil when none gives one. A client in -// SWW_WAF_ALLOW_NETS skips them all, and is not looked up. ctx is the request's -// own context. +// checkClient runs the heuristics on the request, and returns the first +// verdict, or nil when none gives one. A client in SWW_WAF_ALLOW_NETS skips them +// all, and is not looked up. ctx is the request's own context. +// +// The heuristics of beforeLookup run first. Then the client's AS number and +// country are looked up, and afterLookup runs once the lookup has its result. +// While the lookup waits for GeoJS, besideLookup runs too, as runBesideLookup +// says. A lookup that needs no wait is done, and afterLookup with it, before +// besideLookup starts, so that besideLookup does no work for a request +// afterLookup refuses. func (rq *request) checkClient(ctx context.Context) *verdict { if isInside(rq.client, rq.h.config.AllowNets) { return nil @@ -225,7 +238,28 @@ func (rq *request) checkClient(ctx context.Context) *verdict { return v } - rq.lookUp(ctx, now) + lookedUp := rq.startLookUp(ctx, now) + + select { + case result := <-lookedUp: + v := rq.runAfterLookup(ctx, result) + if v != nil { + return v + } + + return rq.runBesideLookup(ctx, nil) + default: + return rq.runBesideLookup(ctx, lookedUp) + } +} + +// runAfterLookup notes result, the lookup's, and runs the heuristics of +// afterLookup. It returns the first verdict, or nil when none gives one. ctx is +// the request's own context. +func (rq *request) runAfterLookup( + ctx context.Context, result lookupResult, +) *verdict { + rq.noteLookup(result) // The lookup can wait for GeoJS, so the checks after it read the time // again: a request counted at the time it arrived could be counted @@ -382,9 +416,9 @@ func anomalyThresholdsSet(cfg *config.WAF) bool { // answer about it as the request ends, and whether there is one: the // lookup database's, which was there at once, or the one GeoJS has given // by then, which a request does not wait for unless a setting needs it. -func (rq *request) answerAtTheEnd() (lookup.Answer, bool) { +func (rq *request) answerAtTheEnd() (geojs.Answer, bool) { if !rq.lookedUp { - return lookup.Answer{}, false + return geojs.Answer{}, false } if rq.h.config.LookupSource == "file" { diff --git a/internal/waf/requestlog_test.go b/internal/waf/requestlog_test.go index 241f505..871e7e3 100644 --- a/internal/waf/requestlog_test.go +++ b/internal/waf/requestlog_test.go @@ -20,7 +20,7 @@ const ( // requestIDHeader carries the request's id. requestIDHeader = "X-Request-ID" // instance is the SWW_WAF_INSTANCE_NAME a test sets. - instance = "fsn1app1/blog" + instance = "host1/blog" // ipv6Client is a client on IPv6, and ipv6Group the netblock the rate // limits count it as. ipv6Client = "2001:db8::7" diff --git a/internal/waf/rulefiles.go b/internal/waf/rulefiles.go index b3c55d2..22b211f 100644 --- a/internal/waf/rulefiles.go +++ b/internal/waf/rulefiles.go @@ -13,8 +13,8 @@ import ( "sneak.berlin/go/smallwebwaf/internal/rules" ) -// ruleFiles checks the request against the rules of the rule files at now, -// notes the ids of those it matches in the log line, adds the tag +// ruleFiles takes the rules of the rule files the request matched, which +// findBesideLookup found, notes their ids in the log line, adds the tag // exploitsignature when it matches any, and refuses the request // when the last rule it matches refuses it: with ActionBlockedByBlockRule for a // block rule, and for a ban rule with the action for the ban it makes. A ban @@ -31,7 +31,7 @@ func (ruleFiles) check( rq *request, now time.Time, ) *verdict { - matched := rq.h.rules.Match(rq.in) + matched := rq.findings.rules for _, rule := range matched { rq.line.RuleIDs = append(rq.line.RuleIDs, rule.ID) @@ -70,3 +70,15 @@ func (ruleFiles) check( return nil } } + +// refusedByRule reports whether ruleFiles refuses a request that matched +// matched: whether the last of them is a block rule or a ban rule. +func refusedByRule(matched []rules.Rule) bool { + if len(matched) == 0 { + return false + } + + action := matched[len(matched)-1].Action + + return action == rules.ActionBlock || action == rules.ActionBan +} diff --git a/internal/waf/trappaths.go b/internal/waf/trappaths.go index 5c554cc..0941500 100644 --- a/internal/waf/trappaths.go +++ b/internal/waf/trappaths.go @@ -11,9 +11,8 @@ import ( "sneak.berlin/go/smallwebwaf/internal/rules" ) -// trapPaths refuses a request for a path in SWW_WAF_TRAP_PATHS: its path as a -// path rule sees it, without the query, as sent or once percent-decoded, is one -// of them. Such a request is a clear sign of attack, as a ban rule's match is: +// trapPaths refuses a request for a path in SWW_WAF_TRAP_PATHS, as trapPath +// finds it. Such a request is a clear sign of attack, as a ban rule's match is: // it bans the client's netblock, or in observe mode raises the alert for the // ban it would have made. The ban's notes give the path as sent, and so does // its figure. @@ -26,9 +25,8 @@ func (trapPaths) check( rq *request, now time.Time, ) *verdict { - path := rules.Path(rq.in) - if !slices.Contains(rq.h.config.TrapPaths, path) && - !slices.Contains(rq.h.config.TrapPaths, rules.DecodeOnce(path)) { + path := rq.findings.trapPath + if path == "" { return nil } @@ -39,3 +37,16 @@ func (trapPaths) check( figures: map[string]any{"path": path}, } } + +// trapPath returns the request's path as sent, when its path as a path rule +// sees it, without the query, as sent or once percent-decoded, is in +// SWW_WAF_TRAP_PATHS, and "" otherwise. +func (rq *request) trapPath() string { + path := rules.Path(rq.in) + if !slices.Contains(rq.h.config.TrapPaths, path) && + !slices.Contains(rq.h.config.TrapPaths, rules.DecodeOnce(path)) { + return "" + } + + return path +} diff --git a/internal/waf/waf_test.go b/internal/waf/waf_test.go index f93f88f..93a0e15 100644 --- a/internal/waf/waf_test.go +++ b/internal/waf/waf_test.go @@ -1787,7 +1787,7 @@ func TestCoreRuleSetReadsTheBodyAQuestionCarries(t *testing.T) { t.Parallel() s := startWAF(t, "", map[string]string{ - coreRuleSetMode: block, coreRuleSetBodyInspectionSize: "64K", + coreRuleSetMode: block, coreRuleSetBodyInspectionSize: "8K", }) ask := func(contentType, body string) answer { @@ -1802,10 +1802,10 @@ func TestCoreRuleSetReadsTheBodyAQuestionCarries(t *testing.T) { got := ask(formData, "id=1'%20OR%20'1'='1") wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny) - // The first 64 KiB of a longer JSON body cannot be parsed, which is no + // The first 8 KiB of a longer JSON body cannot be parsed, which is no // attack, so the Core Rule Set does not read it. - cut := `{"text":"` + strings.Repeat("x", waf.QuestionBodyMaxBytes) - got = ask("application/json", cut[:waf.QuestionBodyMaxBytes]) + cut := `{"text":"` + strings.Repeat("x", 8<<10) + got = ask("application/json", cut) wantDisposition(t, got, http.StatusOK, waf.DispositionAllow) } diff --git a/script/publish b/script/publish index 614ff19..32db2c8 100755 --- a/script/publish +++ b/script/publish @@ -2,8 +2,9 @@ # script/publish: build the image on the Docker host and push it to the # registry under four tags: the UTC date (YYYY-MM-DD), the short commit # hash, the branch name and latest. The build is script/docker's, so the -# tests and the linter run first. The host and the registry name are -# environment variables with defaults, for running it against another. +# tests and the linter run first. DOCKER_HOST names the Docker host and +# PUBLISH_IMAGE the image in the registry. Both must be set; neither has +# a default. # # The working tree must be clean: the hash tag names a commit, and an # image built from uncommitted changes would carry a hash that does not @@ -13,11 +14,16 @@ set -eu SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)" -: "${DOCKER_HOST:=ssh://root@fsn1app1}" -: "${PUBLISH_IMAGE:=git.eeqj.de/sneak/smallwebwaf}" -export DOCKER_HOST - main() { + if [ -z "${DOCKER_HOST:-}" ]; then + echo "script/publish: DOCKER_HOST is not set; set it to the Docker host that builds the image, such as ssh://user@build.example.com" >&2 + exit 1 + fi + if [ -z "${PUBLISH_IMAGE:-}" ]; then + echo "script/publish: PUBLISH_IMAGE is not set; set it to the image name in the registry, such as registry.example.com/smallwebwaf" >&2 + exit 1 + fi + cd "$ROOT" # Own line: a failing command substitution inside an argument does