diff --git a/README.md b/README.md
index 1ed7527..c947ac5 100644
--- a/README.md
+++ b/README.md
@@ -26,7 +26,7 @@ shipped in one image, with two commands that do this together:
- `sww rproxy` is a small reverse proxy in front of one app. It sits behind a
TLS-terminating proxy, such as traefik, nginx or Caddy. It asks `waf` about
each request, passes the allowed ones to the app, and passes the app's answers
- back. It keeps nothing between requests and writes nothing to disk.
+ back. It writes nothing to disk.
- `sww waf` decides. It runs the checks and keeps the bans, the counters and
each client's history. One `waf` serves many sites: a client banned through
one site is refused on all of them, and its requests on all of them count
@@ -35,7 +35,26 @@ shipped in one image, with two commands that do this together:
The site keeps answering while `waf` is down, slow or restarting: without an
answer in time, `rproxy` passes the request to the app.
-It protects against:
+`rproxy` also protects the app on its own, whether or not `waf` answers. With
+the default settings, `rproxy`:
+
+- answers `400` to a malformed request, such as a broken header line or a
+ missing `Host`;
+- answers `405` to a method it does not pass on, such as `TRACE`;
+- answers `431` to a request line and headers over 32 KiB together;
+- gives the visitor 60 seconds to send the request line and headers, and then 60
+ seconds more for the whole request to reach the app;
+- answers `413` to a request body over 100 MiB;
+- gives the app's answer 30 minutes to reach the visitor, counted from the end
+ of the request;
+- answers `502` in place of an app's answer whose body is over 1 GiB, or cuts it
+ off if it has started;
+- closes a kept-open connection after 120 seconds idle.
+
+`rproxy` does not limit the number of headers. `waf` limits the number of
+headers and cookies, and the length of the request line and of each header line.
+
+Together, `rproxy` and `waf` protect against:
- Scrapers that fetch every page of a site, all day. Each client gets a limit on
requests and on body bytes per minute, per hour and per day. The defaults
@@ -57,8 +76,6 @@ It protects against:
limit, so that they are banned sooner;
- refuse or limit clients on a blocklist, a DNS blocklist, or with a high
AbuseIPDB score.
-- Slow and oversized requests. Every request and every answer has a time limit
- and a size limit.
For the operator, it:
@@ -281,13 +298,13 @@ Every setting the code reads, with its default:
| `SWW_RPROXY_LISTEN_ADDR` | `:8080` | Where `rproxy` listens; leave the host part empty. |
| `SWW_RPROXY_TRUSTED_PROXIES` | `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16` | The netblocks whose `X-Forwarded-For` `rproxy` believes; set but empty, it believes none. |
| `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS` | `2000` | How long `rproxy` waits for `waf`'s answer to a question and to a report, in milliseconds. |
-| `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` | `allow` | What `rproxy` does with a request when `waf` cannot be reached or does not answer in time: `allow` passes it to the app, `deny` answers `503`. |
+| `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` | `allow` | What `rproxy` does with a request when `waf` gives no answer in time, and while the circuit breaker for asks to `waf` is open: `allow` passes it to the app, `deny` answers `503`. |
| `SWW_RPROXY_MAX_REQUEST_SIZE` | `100M` | The largest request body, a size or `off`; a larger one is answered `413`. |
| `SWW_RPROXY_MAX_RESPONSE_SIZE` | `1G` | The largest response body, a size or `off`. The visitor gets `502` in place of a larger one, or it is cut off if it has already started. |
| `SWW_RPROXY_MAX_REQUEST_TIME_MS` | `60000` | Two time limits, each this long, in milliseconds or `off`: one for the visitor to send its request line and headers, then one for the whole request to reach the app. |
| `SWW_RPROXY_MAX_RESPONSE_TIME_MS` | `1800000` | How long the answer may take, from the end of the request to its last byte reaching the visitor, in milliseconds or `off`. |
| `SWW_WAF_LISTEN_ADDR` | `:8080` | Where `waf` listens; leave the host part empty. |
-| `SWW_WAF_INSTANCE_NAME` | the container's host name | The name in every line `waf` logs, and in every alert and metric, such as `fsn1/waf`. |
+| `SWW_WAF_INSTANCE_NAME` | the container's host name | The name in every line `waf` logs, and in every alert and metric, such as `host1/waf`. |
| `SWW_WAF_MODE` | `enforce` | `observe` refuses nothing and makes no ban; each log line says what `enforce` would have done. |
| `SWW_WAF_IPV6_GROUP_PREFIX` | `64` | The prefix length of the IPv6 netblock that counts as one client, from 32 to 128. |
| `SWW_WAF_MAX_TRACKED_CLIENTS` | `20000` | The most clients kept, in memory and in `clients.json`; the least recently seen is dropped first. |
@@ -312,7 +329,7 @@ Every setting the code reads, with its default:
| `SWW_WAF_BAN_SCOPE_V4_PREFIX` | `32` | The prefix length of the netblock a ban on an IPv4 client covers, from 0 to 32. |
| `SWW_WAF_LOOKUP_SOURCE` | `geojs` | Where each client's AS number and country come from: `geojs` (a web service), `file` (a local database) or `off`. |
| `SWW_WAF_LOOKUP_DB_PATH` | empty | The IPinfo Lite database file, in its `.mmdb` form, read while the source is `file`. |
-| `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` | `800` | How long a request waits for its client's first GeoJS answer when a setting needs it, and how long a request to GeoJS may take, in milliseconds. |
+| `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` | `800` | How long a request waits for its client's first GeoJS answer when a setting needs it, in milliseconds. The request to GeoJS goes on, and a later answer is kept. |
| `SWW_WAF_ADD_LOOKUP_HEADERS` | `true`, or `false` while the lookup source is `off` | `true` passes the app the client's AS number and country in `X-Client-ASN` and `X-Client-Country`. |
| `SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES` | empty | When set, the only countries whose clients get through; a client the lookup cannot place is refused too. |
| `SWW_WAF_TAG_LISTS` | empty | Address lists that each add a tag to their clients' requests, as `name=URL` pairs, such as `torexit=https://example.com/tor.txt`. |
@@ -331,7 +348,7 @@ Every setting the code reads, with its default:
| `SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD` | `5` | The Core Rule Set score at which a request is a match, a whole number or `off`; one critical rule scores 5. |
| `SWW_WAF_CORE_RULE_SET_DISABLED_RULES` | `920340,920420,920440,920640,930130,930140` | Core Rule Set rule ids switched off; a list you give replaces the default, so repeat these six to keep them off. |
| `SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS` | empty | Path prefixes the Core Rule Set does not inspect. |
-| `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` | `off` | How much of a request body the Core Rule Set reads, a size up to `64K`; `off` reads none. |
+| `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` | `off` | How much of a request body the Core Rule Set reads, a size up to `8K`; `off` reads none. |
| `SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE` | unset | A file of Core Rule Set exclusions, as Coraza directives, that switch rules off for some fields or paths only. |
| `SWW_WAF_APPS` | unset | The apps each hostname serves, such as `git.example.com:gitea;blog.example.com:wordpress`; only those apps' fixes apply there. "Attack detection" lists the apps. Listing `forgejo` brings gitea's fixes too. Unset, every app's fixes apply to every hostname. |
| `SWW_WAF_BLOCKLIST_URLS` | empty | Blocklists to fetch, text files of addresses and netblocks one to a line, such as `https://www.spamhaus.org/drop/drop.txt`. |
@@ -513,7 +530,7 @@ services:
image: smallwebwaf
environment:
SWW_COMMAND: waf
- SWW_WAF_INSTANCE_NAME: fsn1/waf
+ SWW_WAF_INSTANCE_NAME: host1/waf
# Refuse nothing yet; log what enforce mode would have done.
SWW_WAF_MODE: observe
# Look addresses up in a local IPinfo Lite database, never GeoJS.
@@ -582,6 +599,11 @@ request reaches the app.
16. A path under `/_sww/` is answered `404`, so that the app never sees that
prefix.
+Steps 12 to 14 only read the request. While the lookup waits for GeoJS, they run
+at the same time as steps 6 to 11. The first of the two sides to refuse the
+request ends the checks, and `waf` answers at once. Steps 12 to 14 have 500 ms
+together.
+
`rproxy` takes the rest, with `waf` counting at the end:
17. The request is passed to the app, streaming.
@@ -594,9 +616,10 @@ A refused visitor is answered with `SWW_WAF_BAN_RESPONSE`. A `block` rule or a
Core Rule Set match answers `403` and bans nobody, since the Core Rule Set has
false positives; such refusals count toward `SWW_WAF_REFUSALS_LIMIT_PER_MINUTE`
instead. A request over the header limits of step 3 is answered `414` for a
-request line over 8 KiB and `431` otherwise. It bans nobody either. `rproxy`
-itself answers `405` to a method it does not pass on, and the status for a size
-or time limit, such as `413` for a body too large.
+request line over 8 KiB and `431` otherwise. It bans nobody either. Nor does a
+request that steps 12 to 14 have not finished within 500 ms, which is answered
+`503`. `rproxy` itself answers `405` to a method it does not pass on, and the
+status for a size or time limit, such as `413` for a body too large.
### Clients and bans
@@ -732,9 +755,9 @@ stops the running process (see [`docs/state-files.md`](docs/state-files.md)).
- `internal/waf`: `waf`'s server, which runs the checks and answers its own
endpoints.
- `internal/ratelimit`, `internal/bans`, `internal/rules`,
- `internal/coreruleset`, `internal/lookup`, `internal/reputation`,
- `internal/routewatch`, `internal/anomaly` and `internal/alerts`: one check or
- source each.
+ `internal/coreruleset`, `internal/geojs`, `internal/lookup`,
+ `internal/reputation`, `internal/routewatch`, `internal/anomaly` and
+ `internal/alerts`: one check or source each.
- `internal/state`, `internal/requestlog`, `internal/remotelog` and
`internal/metrics`: the state and cache files, the log lines, the syslog
sender and the metrics.
@@ -801,10 +824,9 @@ on the host.
- `script/docker`: builds the image, whose build runs the tests and the linter
first. `make docker`.
- `script/publish`: runs `script/docker`, whose build happens on the Docker host
- named by `DOCKER_HOST` (`ssh://root@fsn1app1` unless set). It then pushes the
- image to `PUBLISH_IMAGE` (`git.eeqj.de/sneak/smallwebwaf` unless set). It
- refuses a working tree with uncommitted changes. `make publish`. It pushes
- four tags:
+ named by `DOCKER_HOST`, then pushes the image to `PUBLISH_IMAGE`, such as
+ `registry.example.com/smallwebwaf`. Both variables are required. It refuses a
+ working tree with uncommitted changes. `make publish`. It pushes four tags:
- the UTC date as `YYYY-MM-DD`;
- the short commit hash;
- the branch name;
diff --git a/SPEC.md b/SPEC.md
index b062148..ca67a0a 100644
--- a/SPEC.md
+++ b/SPEC.md
@@ -87,8 +87,10 @@ never as a time.
change, and tells which rules a request matches.
- `internal/coreruleset`: the Core Rule Set, with the five changes and the
apps' fixes listed under "Attack detection", as Coraza directives.
- - `internal/lookup`: looks clients up through GeoJS or in the lookup
- database.
+ - `internal/geojs`: looks clients up through GeoJS and keeps the answers. It
+ imports no other package of this repository: `waf` gives it what it needs
+ to count and report failures.
+ - `internal/lookup`: looks clients up in the lookup database.
- `internal/reputation`: fetches the lists, asks the DNS blocklists and
AbuseIPDB in the background, and keeps their answers.
- `internal/routewatch`: asks a routewatch instance about clients in the
@@ -171,7 +173,8 @@ field gives that name. `refusal_limit` and `byte_limits` give theirs in
`country=` tag.
Otherwise the request goes on at once. A request that ends before the answer
- comes is logged without it.
+ comes is logged without it. While the request waits, steps 12 to 14 run at
+ the same time as steps 6 to 11 (see "Answering in time").
7. The country list (`countries`; see "Country list").
8. The reputation sources (see "Reputation"), in this order:
@@ -507,19 +510,25 @@ service that keeps its visitors' addresses on its own host.
dropped from that table. The answer reaches the client's history and ban notes
when it comes.
- `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` is how long a visitor's request
- waits for its client's first answer, when a setting needs it (see step 6). It
- also bounds each request to GeoJS. A client whose answer has not come in time
- counts as unknown until it comes, and its later requests do not wait. Its
- address is asked about again in the background.
+ waits for its client's first answer, when a setting needs it (see step 6). A
+ client whose answer has not come in time counts as unknown until it comes, and
+ its later requests do not wait.
+- A request to GeoJS may take up to 10 seconds, however long visitors wait. It
+ goes on after the visitor's request ends, and its answer is kept. Say GeoJS
+ answers after 5 seconds. The visitor's request is decided after 800
+ milliseconds, without a country. A request from the same client 6 seconds
+ later gets the answer.
+- Requests from one client that come together share one request to GeoJS.
- A client the lookup cannot place has an unknown country and AS number, which
the country list and the biased thresholds treat as described there. Nothing
else treats it differently.
- GeoJS publishes no rate limit, but its terms forbid "an excessive amount of
- API requests", judged by GeoJS alone, and let it block a caller. While GeoJS
- is slow, down or refusing `waf`, new clients count as unknown. GeoJS is then
- left alone for a second, and twice as long after each further failure, up to
- five minutes. Each failure raises a `source_failure` alert (see "Alerting"). A
- service that cannot accept this uses the file.
+ API requests", judged by GeoJS alone, and let it block a caller. A request to
+ GeoJS fails when it takes over 10 seconds or GeoJS answers with an error.
+ While GeoJS is slow, down or refusing `waf`, new clients count as unknown.
+ After a failure GeoJS is left alone for a second, and twice as long after each
+ further failure, up to five minutes. Each failure raises a `source_failure`
+ alert (see "Alerting"). A service that cannot accept this uses the file.
- The file is the free IPinfo Lite database (`ipinfo_lite.mmdb`), whose `asn`,
`as_name` and `country_code` fields are read. The operator downloads it with a
free IPinfo account and mounts it read-only; `waf` never fetches it and holds
@@ -704,11 +713,12 @@ files, the limits and the bans still apply to the client that sends it.
rules is loaded yet.
- `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` switches body inspection on, up
- to that size. It takes at most `64K`, since `waf` gets no more of a body than
- that (see "The answer"). Form data and multipart bodies are read up to the
- size. JSON and XML bodies are read only when no larger than the size, since
- those cannot be read in part. Any other body is not read: the Core Rule Set
- would read it as form data, where binary content such as a git push trips
+ to that size. It takes at most `8K`, so that `waf` answers in time (see
+ "Answering in time"). Form data and multipart bodies are read up to the size.
+ JSON and XML bodies are read only when no larger than the size, since those
+ cannot be read in part. A JSON body over 1 KiB is not read either, whatever
+ the size (see "Answering in time"). Any other body is not read: the Core Rule
+ Set would read it as form data, where binary content such as a git push trips
rules written for text. A body the Core Rule Set cannot parse adds 5 to the
score, and so does a multipart body cut by the limit in the middle of a part's
header. Body inspection suits apps whose forms carry no code.
@@ -2370,8 +2380,9 @@ runs no check, writes no request log line and counts nothing in the metrics.
### The answer
-`waf` runs steps 3 to 16 of "What happens to a request", in that order. Every
-answer but the last below carries `X-Sww-Disposition`.
+`waf` runs steps 3 to 16 of "What happens to a request", in that order, apart
+from the steps that run at the same time (see "Answering in time"). Every answer
+but the last below carries `X-Sww-Disposition`.
- `X-Sww-Disposition: allow`, status `200`, no body. `rproxy` passes the request
to the app. While `SWW_WAF_ADD_LOOKUP_HEADERS` is `true`, the answer also
@@ -2383,6 +2394,8 @@ answer but the last below carries `X-Sww-Disposition`.
- `SWW_WAF_BAN_RESPONSE`, for a refusal;
- `403`, for a `block` rule or the Core Rule Set;
- `414` or `431`, for a request over the header limits;
+ - `503`, for a request that steps 12 to 14 have not finished within 500
+ milliseconds;
- `404`, for a path under `/_sww/` that passes the checks, so that the app
never sees that prefix.
- `X-Sww-Disposition: close`, status `200`, no body. This takes the place of a
@@ -2397,23 +2410,28 @@ In `observe` mode a request that `enforce` mode would refuse is answered
answered `404`.
The Core Rule Set inspects the body a question carries only while
-`SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` is set. It inspects at most 64 KiB
-less one byte. So it does not inspect a JSON or XML body of 64 KiB or more,
-which may have been cut short.
+`SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` is set, and at most 8 KiB of it.
### Answering in time
`rproxy` stops waiting for `waf` after `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, by
default 2 seconds, and `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` then
-decides. The longest a question waits on an outside source is
-`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, by default 800 milliseconds, well
-within that time.
+decides. Two things in a decision take time:
+
+- the wait for GeoJS, at most `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, by
+ default 800 milliseconds;
+- the trap paths, the rule files and the Core Rule Set, which have 500
+ milliseconds.
+
+They run at the same time, so `waf` answers within the longer of the two, well
+within `rproxy`'s wait.
GeoJS is the only outside source a question can wait on. A question waits only
when step 6 says so, and for at most
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, by default 800 milliseconds. A
-client without an answer by then counts as unknown. No question waits on these,
-which run in the background:
+client without an answer by then counts as unknown. The setting bounds only the
+question's wait: the request to GeoJS goes on, and its answer is kept for later
+questions. No question waits on these, which run in the background:
- the DNS blocklists and AbuseIPDB;
- fetching the blocklists, the tag lists and the file
@@ -2438,10 +2456,38 @@ never wait. These can still wait for GeoJS:
- a country list, which needs the answer;
- a rate limit a percentage has lowered, which needs the answer;
- a DNS blocklist or AbuseIPDB, while `SWW_WAF_DNSBL_ABUSEIPDB_ACTION` is
- `deny`;
-- a trap path;
-- a rule file;
-- the Core Rule Set.
+ `deny`.
+
+The trap paths, the rule files and the Core Rule Set, steps 12 to 14, run during
+that wait. Once a request has passed the bans, the lookup starts. While it waits
+for GeoJS, steps 12 to 14 run at the same time as steps 6 to 11. A lookup that
+needs no wait has its answer at once, and steps 7 to 11 then run before steps 12
+to 14 start.
+
+- The first of the two sides to refuse the request ends the checks. `waf`
+ answers at once, and does not wait for the other side. The lookup goes on, and
+ its answer is kept.
+- What the other side finds leaves no trace. It counts nothing, bans nothing,
+ raises no alert, counts no metric and adds nothing to the log line.
+- When both sides would refuse, the one that finishes first is answered and
+ logged.
+- When neither refuses, step 15 runs.
+
+Steps 12 to 14 only read the request. They have 500 milliseconds together, from
+their start. The limit is fixed, with no setting. A request they have not
+finished by then is refused:
+
+- the answer is `deny`, with `503`;
+- the log line's action is `checks_timed_out`, and its `heuristic` names the
+ step that had not finished;
+- the refusal makes no ban, is no offence and does not count toward the refusal
+ limit;
+- it holds in every `SWW_WAF_CORE_RULE_SET_MODE`;
+- in `observe` mode the request is answered `allow` with `would_action`, like
+ every refusal.
+
+The limit is measured on the clock. While `waf`'s host is so busy that an
+ordinary request takes 500 milliseconds, that request is refused too.
The header limits are fixed, with no setting. A line's length is in bytes and
does not count its line ending. `waf` refuses a request whose request line is
@@ -2474,12 +2520,28 @@ client in `SWW_WAF_ALLOW_NETS` skips them, as it skips the Core Rule Set. In
`observe` mode such a request is answered `allow` with `would_action`, and no
other check runs on it.
-The Core Rule Set runs each of its rules on each cookie, so thousands of small
-cookies would keep it busy for over a second. One 30 KB cookie or query string
-would keep it busy for a third of a second or more. A request with several long
-`Cookie` lines, each within the limits, can keep it as busy. Enough such
-requests at once can still overload `waf` past
-`SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`.
+The Core Rule Set is the one check a visitor can keep busy for long. It runs
+each of its rules on each part of a request, so thousands of small cookies would
+keep it busy for over a second. These fixed limits bound what it reads:
+
+- the header limits above;
+- `rproxy`'s limit of 32 KiB on the request line and headers together (see
+ "`rproxy`'s limits");
+- 8 KiB of a body, the most `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` takes,
+ or 1 KiB of a JSON body.
+
+Coraza names each value in a JSON body by the whole path to it, such as
+`json.users.0.name`. So the names and values the Core Rule Set reads in a JSON
+body can add up to more than the body. From 1 KiB of JSON, Coraza makes at most
+8 KiB of them, no more than the largest body of another kind.
+
+A request can reach all of these at once. At paranoia level 4, with no rule
+switched off, the Core Rule Set takes under a second on one core of a current
+server to inspect it. `waf` refuses it if the inspection takes over 500
+milliseconds. Coraza cannot stop an inspection part-way, so the inspection runs
+to its end with nobody waiting for it. These limits keep one request from
+holding a processor for more than about a second. Enough such requests at once
+can still overload `waf` past `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`.
### The report
@@ -2583,11 +2645,11 @@ request and ban notes name the site by its host.
## `rproxy`
`sww rproxy` is the reverse proxy in front of one app. It asks `waf` about each
-request, and passes on what `waf` allows. It keeps nothing between requests and
-writes nothing to disk. When `waf` cannot be connected to, does not answer at
-all, or does not answer within `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`,
-`SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` decides. Its default, `allow`,
-keeps the site answering.
+request, and passes on what `waf` allows. It writes nothing to disk. The only
+thing it keeps between requests is its circuit breaker for asks to `waf`.
+`SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` decides what is done with a
+request when `waf` gives no answer in time, and while the circuit breaker is
+open. Its default, `allow`, keeps the site answering.
`rproxy` reads only its `SWW_RPROXY_` settings, which `docs/settings.md` lists.
One of them is required: `SWW_RPROXY_WAF_BACKEND`, `waf`'s URL. It has no
@@ -2605,7 +2667,8 @@ only. The image runs it by default (see "Deployment").
4. It refuses a request with a method it does not pass on, or one it cannot put
to `waf` as the visitor sent it (see "Requests `rproxy` refuses itself").
5. It reads up to the first 64 KiB of the body into memory.
-6. It sends `waf` the question (see "The question"). It speaks HTTP/1.1 to
+6. It sends `waf` the question (see "The question"), unless the circuit breaker
+ keeps it from asking (see "The circuit breaker"). It speaks HTTP/1.1 to
`waf`, whatever `waf` offers. Go's HTTP/2 client would refuse to send a
visitor's `Upgrade` header, and most `Connection` headers. The question never
carries the visitor's `Trailer`.
@@ -2679,23 +2742,24 @@ log line has status `0`.
### Without an answer from `waf`
`SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` decides what is done with the
-request only when `waf`:
+request when `waf`:
- cannot be connected to;
- does not answer at all;
- does not answer within `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, which covers
connecting, sending the question and reading the whole answer.
-An answer whose body breaks off counts as no answer at all.
+An answer whose body breaks off counts as no answer at all. The setting also
+decides, without asking `waf`, while the circuit breaker is open.
`allow`, the default, passes the request to the app. `deny` answers `503`.
Either way `rproxy` writes a process warning, its log line has
`waf_backend_no_answer`, and it sends no report.
-A visitor can make `waf` slow. A request that keeps `waf` busy past
-`SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, like any request while `waf` is overloaded,
-gets `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION`. With `deny`, such a request
-gets `503` instead of reaching the app.
+`waf` taking longer than `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS` to decide a request
+is a bug in `waf`. Such a request gets
+`SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION`, like any request while `waf` is
+overloaded. With `deny`, it gets `503` instead of reaching the app.
A path under `/_sww/` that `allow` would pass on is answered `404` instead, so
that the app never sees that prefix.
@@ -2710,6 +2774,28 @@ A visitor that goes away while `waf` is being asked is no failure of `waf`.
`SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` does not apply. Its log line has
status `0` and no `disposition`.
+### The circuit breaker
+
+`rproxy` stops asking `waf` for a while when `waf` keeps failing to answer. An
+ask fails when `waf` gives no answer in time (see "Without an answer from
+`waf`"), or an answer `rproxy` cannot pass on. Any other answer in time
+succeeds, whatever it decides. A visitor that goes away while `waf` is being
+asked counts as neither.
+
+The breaker works like this:
+
+- It opens when 5 asks fail within 5 seconds. Fewer failures never open it.
+- While it is open, a request gets
+ `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` at once, without asking `waf`.
+ It gets the same process warning and `waf_backend_no_answer` as a request
+ whose ask failed.
+- While it is open, one request a second still asks `waf`.
+- Each ask that succeeds clears the failures counted so far, and closes the
+ breaker if it is open.
+
+The breaker lives in memory, and a restart of `rproxy` closes it. It has no
+settings.
+
### Passing a request to the app
`rproxy` sends the app at `SWW_RPROXY_UPSTREAM_URL` the visitor's request: the
@@ -2983,8 +3069,8 @@ host directory, keeps them across deploys. A host directory mounted there needs
no change of owner, and files an earlier owner left in it can be read and
replaced. The volume holds a few tens of MiB at most with the defaults, and
needs no backup beyond the host's own. The image declares no volume, since every
-app image built on it would inherit it. `rproxy` keeps nothing, and needs no
-volume.
+app image built on it would inherit it. `rproxy` writes nothing to disk, and
+needs no volume.
`waf` keeps its cache files in `SWW_WAF_CACHE_DIR`, `/var/cache/sww` by default,
which the `run` script makes. The cache directory needs no volume and can be
diff --git a/docs/alerts.md b/docs/alerts.md
index 11918c5..6f14a37 100644
--- a/docs/alerts.md
+++ b/docs/alerts.md
@@ -34,7 +34,7 @@ on one line:
```json
{
- "instance": "fsn1/waf",
+ "instance": "host1/waf",
"time": "2026-10-06T12:00:50.123461Z",
"event": "ban",
"host": "app.example",
@@ -109,7 +109,7 @@ on one line:
## Slack and ntfy
Slack and ntfy get a message with a title and a text. The title is the instance
-and the event, such as `fsn1/waf: ban`. The text is the `reason`, then one line
+and the event, such as `host1/waf: ban`. The text is the `reason`, then one line
for each of these that the alert has:
- the host, the client, the netblock and the country;
diff --git a/docs/request-log.md b/docs/request-log.md
index 26f76ef..77bf096 100644
--- a/docs/request-log.md
+++ b/docs/request-log.md
@@ -26,7 +26,7 @@ These questions write no line:
A `report` line:
```
-{"type":"request","waf_line":"report","time":"2026-10-03T12:00:00.123Z","instance":"fsn1/waf","client_ip":"203.0.113.9","method":"GET","scheme":"https","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","peer_ip":"172.18.0.2","forwarded_for":"203.0.113.9","client_group":"203.0.113.9/32","asn":"AS64496","as_name":"Example Net","country":"DE","request_headers":{"accept":"*/*"},"action":"forward","counts":{"minute":1,"hour":12,"day":40,"minute_bytes":5120,"hour_bytes":61440,"day_bytes":204800},"core_rule_set_score":0,"duration_total_ms":3.217,"duration_checks_ms":0.041,"duration_core_rule_set_ms":0.029,"duration_upstream_total_ms":3.104}
+{"type":"request","waf_line":"report","time":"2026-10-03T12:00:00.123Z","instance":"host1/waf","client_ip":"203.0.113.9","method":"GET","scheme":"https","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","peer_ip":"172.18.0.2","forwarded_for":"203.0.113.9","client_group":"203.0.113.9/32","asn":"AS64496","as_name":"Example Net","country":"DE","request_headers":{"accept":"*/*"},"action":"forward","counts":{"minute":1,"hour":12,"day":40,"minute_bytes":5120,"hour_bytes":61440,"day_bytes":204800},"core_rule_set_score":0,"duration_total_ms":3.217,"duration_checks_ms":0.041,"duration_core_rule_set_ms":0.029,"duration_upstream_total_ms":3.104}
```
### Every line
@@ -105,6 +105,8 @@ Every one of these lines has:
`blocked_by_core_rule_set`: refused by that check;
- `request_line_too_long`: a request line over the header limits;
- `request_header_too_large`: headers over the header limits;
+ - `checks_timed_out`: refused with `503` because the trap paths, the rule
+ files and the Core Rule Set had not finished within 500 ms;
- `own_endpoint`: answered at one of `waf`'s own endpoints, or `404` for a
path under `/_sww/`.
- `would_action`: in `observe` mode, what `enforce` mode would have done with a
@@ -134,6 +136,11 @@ Every one of these lines has:
`SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD`;
- `refusal_limit`: `window`, `count` and `limit`, of the refusal limit;
- `byte_limits`: `window`, `count` and `limit`, the count in bytes.
+
+ For a request refused as `checks_timed_out`, `heuristic` is the one of
+ `trap_paths`, `rule_files` and `core_rule_set` that had not finished, and
+ its figure is `time_limit_ms`, 500.
+
- `after_answer_heuristic`: the heuristic that banned the client after `waf`
answered the request, in `observe` mode too: `refusal_limit` or `byte_limits`.
- `after_answer_heuristic_figures`: an object of the figures that heuristic
@@ -238,7 +245,8 @@ meanings where they differ:
- `disposition`: what was done with the request: `allow`, `deny` or `close`. It
is `waf`'s answer, or the value of
`SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION`. It is left out when:
- - `waf` was not asked, as for the health check;
+ - `waf` was not asked and `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` did
+ not apply, as for the health check;
- the visitor went away while `waf` was being asked;
- `waf`'s answer had no valid `X-Sww-Disposition`;
- `rproxy` could not pass on `waf`'s answer, and answered `502`.
diff --git a/docs/settings.md b/docs/settings.md
index 46b30fb..0f07055 100644
--- a/docs/settings.md
+++ b/docs/settings.md
@@ -34,7 +34,7 @@ chooses between the two (see "The image's command" below).
such as `:9000`. Leave the host out (see "Ports" in [`SPEC.md`](../SPEC.md)).
- `SWW_WAF_INSTANCE_NAME` (default: the container's host name): the `instance`
name in each line `waf` logs, and in each alert and metric, such as
- `fsn1/waf`. Valid UTF-8. No control character while `SWW_WAF_ALERT_NTFY_URL`
+ `host1/waf`. Valid UTF-8. No control character while `SWW_WAF_ALERT_NTFY_URL`
is set. 1 to 48 printable ASCII characters without a space while
`SWW_WAF_LOG_REMOTE_URL` is set and `SWW_WAF_LOG_REMOTE_APP_NAME` is not.
- `SWW_WAF_MODE` (default `enforce`): `enforce` or `observe`.
@@ -159,8 +159,9 @@ cut off; the client is banned.
- `SWW_WAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in
its `.mmdb` form, for `SWW_WAF_LOOKUP_SOURCE=file`.
- `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` (default `800`): how long a
- request waits for its client's first answer from GeoJS, and how long a request
- to GeoJS may take. A time in milliseconds; never `off`.
+ request waits for its client's first answer from GeoJS. A time in
+ milliseconds; never `off`. It does not cut off the request to GeoJS, whose
+ answer is kept when it comes later.
- `SWW_WAF_ADD_LOOKUP_HEADERS` (default `true`): `true` has `rproxy` pass the
app the client's AS number in `X-Client-ASN` and its country in
`X-Client-Country`. The default is `false` while `SWW_WAF_LOOKUP_SOURCE` is
@@ -251,10 +252,9 @@ tag that no request can get stops the start. A request can get only these:
- `SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS` (default empty): path prefixes the Core
Rule Set does not inspect.
- `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` (default `off`): `off`, or how
- much of a request body the Core Rule Set reads, a size of at most `64K`. A
- question carries no more of a body than 64 KiB, and the Core Rule Set reads at
- most 64 KiB less one byte of it, whatever the value (see "The answer" in
- [`SPEC.md`](../SPEC.md)).
+ much of a request body the Core Rule Set reads, a size of at most `8K`. A
+ larger size would let one request keep it busy past `rproxy`'s timeout (see
+ "Answering in time" in [`SPEC.md`](../SPEC.md)).
- `SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE` (default unset): a file of Core Rule
Set exclusions, written as Coraza directives. It is loaded before the Core
Rule Set's rules, so it holds only `SecAction` and `SecRule` lines that
@@ -435,7 +435,9 @@ A few limits are fixed rather than settings:
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` is the longest a question waits on
an outside source. Keep it well under `rproxy`'s
-`SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, as the default is. No question waits for
+`SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, as the default is. The trap paths, the rule
+files and the Core Rule Set run during that wait and have 500 ms, so `waf`
+answers within the longer of the two times. No question waits for
`SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS`, since the DNS blocklists and AbuseIPDB are
asked in the background (see "Answering in time" under "`waf`" in
[`SPEC.md`](../SPEC.md)).
@@ -453,10 +455,9 @@ start, and an invalid value stops the start. Any can be given as a file, as a
for `waf`'s answer, to a question and to a report. A time in milliseconds;
never `off`.
- `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` (default `allow`): what is done
- with a request when `waf` cannot be connected to, does not answer at all, or
- does not answer within `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`. `allow` passes it
- to the app, and `deny` answers `503`. It does not apply to an answer `waf`
- gives, such as an error.
+ with a request when `waf` gives no answer in time, and while the circuit
+ breaker for asks to `waf` is open. `allow` passes it to the app, and `deny`
+ answers `503`. It does not apply to an answer `waf` gives, such as an error.
- `SWW_RPROXY_MAX_REQUEST_SIZE` (default `100M`): the largest request body. A
size or `off`.
- `SWW_RPROXY_MAX_RESPONSE_SIZE` (default `1G`): the largest response body. A
diff --git a/go.mod b/go.mod
index 346ad93..5f24224 100644
--- a/go.mod
+++ b/go.mod
@@ -12,6 +12,7 @@ require (
github.com/oschwald/maxminddb-golang/v2 v2.7.0
github.com/prometheus/client_golang v1.24.1
github.com/spf13/viper v1.21.0
+ golang.org/x/sys v0.48.0
)
require (
@@ -47,7 +48,6 @@ require (
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba // indirect
golang.org/x/net v0.60.0 // indirect
golang.org/x/sync v0.23.0 // indirect
- golang.org/x/sys v0.48.0 // indirect
golang.org/x/text v0.42.0 // indirect
google.golang.org/protobuf v1.36.11 // indirect
rsc.io/binaryregexp v0.2.0 // indirect
diff --git a/internal/alerts/alerts_test.go b/internal/alerts/alerts_test.go
index ac09131..190eea9 100644
--- a/internal/alerts/alerts_test.go
+++ b/internal/alerts/alerts_test.go
@@ -39,7 +39,7 @@ const (
ntfyURL = "https://ntfy.example/sww-alerts"
ntfyToken = "tk_0123456789abcdefghijklmnopq"
// instance is the instance name every alert gives.
- instance = "fsn1app1/blog"
+ instance = "host1/blog"
// started is when each test starts, as an alert gives it, and
// anHourOn an hour later.
started = "2000-01-01T00:00:00Z"
@@ -1160,8 +1160,8 @@ func TestEachDestinationIsSentOnlyTheAlertsWaitingForIt(t *testing.T) {
synctest.Wait()
wantBodies(t, standIns[alerts.DestinationSlack],
- `{"text":"*fsn1app1/blog: file_error*\nsecond"}`,
- `{"text":"*fsn1app1/blog: file_error*\nthird"}`)
+ `{"text":"*host1/blog: file_error*\nsecond"}`,
+ `{"text":"*host1/blog: file_error*\nthird"}`)
wantBodies(t, standIns[alerts.DestinationNtfy], "fourth")
// alerts.json then lists Slack and ntfy alone, with no alert waiting.
@@ -1477,7 +1477,7 @@ func anAlertForEachEvent() []eventMessage {
Reason: "requests per hour over the limit of 10000",
Detail: map[string]any{"mode": "observe"},
},
- "fsn1app1/blog: ban", "default no_entry",
+ "host1/blog: ban", "default no_entry",
"requests per hour over the limit of 10000\nclient: 203.0.113.9\n" +
"netblock: 203.0.113.0/24\ncountry: DE\nmode: observe",
},
@@ -1489,7 +1489,7 @@ func anAlertForEachEvent() []eventMessage {
), Country: "FR",
Reason: "matched the rule env-file",
},
- "fsn1app1/blog: permanent_ban", "high no_entry",
+ "host1/blog: permanent_ban", "high no_entry",
"matched the rule env-file\nclient: 198.51.100.7\n" +
"netblock: 198.51.100.7/32\ncountry: FR",
},
@@ -1500,7 +1500,7 @@ func anAlertForEachEvent() []eventMessage {
Reason: "refused by the Core Rule Set",
Host: "app.example",
},
- "fsn1app1/blog: core_rule_set_match", "default shield",
+ "host1/blog: core_rule_set_match", "default shield",
"refused by the Core Rule Set\nhost: app.example\nclient: 192.0.2.1\n" +
"netblock: 192.0.2.1/32",
},
@@ -1509,7 +1509,7 @@ func anAlertForEachEvent() []eventMessage {
Event: alerts.EventAnomaly, Netblock: netip.MustParsePrefix("192.0.2.0/24"),
Reason: "requests per minute over the threshold of 5000",
},
- "fsn1app1/blog: anomaly", "high chart_with_upwards_trend",
+ "host1/blog: anomaly", "high chart_with_upwards_trend",
"requests per minute over the threshold of 5000\nnetblock: 192.0.2.0/24",
},
{
@@ -1518,7 +1518,7 @@ func anAlertForEachEvent() []eventMessage {
Netblock: netip.MustParsePrefix("192.0.2.2/32"),
Reason: "listed by a DNS blocklist",
},
- "fsn1app1/blog: reputation_hit", "low label",
+ "host1/blog: reputation_hit", "low label",
"listed by a DNS blocklist\nclient: 192.0.2.2\nnetblock: 192.0.2.2/32",
},
{
@@ -1526,7 +1526,7 @@ func anAlertForEachEvent() []eventMessage {
Event: alerts.EventSourceFailure, Reason: "asking GeoJS failed",
Detail: map[string]any{"source": "geojs"},
},
- "fsn1app1/blog: source_failure", "high warning",
+ "host1/blog: source_failure", "high warning",
"asking GeoJS failed\nsource: geojs",
},
{
@@ -1537,7 +1537,7 @@ func anAlertForEachEvent() []eventMessage {
"file": "/etc/sww/rules.d/50-app.rules", "error": "line 2: no action",
},
},
- "fsn1app1/blog: file_error", "high warning",
+ "host1/blog: file_error", "high warning",
"a rule file has an error, and the rules stay as they were\n" +
"file: /etc/sww/rules.d/50-app.rules\nerror: line 2: no action",
},
diff --git a/internal/config/waf.go b/internal/config/waf.go
index cc166e3..513d253 100644
--- a/internal/config/waf.go
+++ b/internal/config/waf.go
@@ -365,8 +365,8 @@ var (
"it names the file to look clients up in",
)
errDBPathUnused = errors.New("only file reads it")
- errOver64K = errors.New(
- "is more than 64K, the most waf gets of a body")
+ errOver8K = errors.New(
+ "is more than 8K, the most the Core Rule Set can read in time")
errNotDurationAboveZero = errors.New(
"is not a duration above zero, such as 1h or 7d")
errNotNumberAboveZero = errors.New(
@@ -1601,16 +1601,16 @@ func durationOrDays(value string) (time.Duration, error) {
}
// parseCoreRuleSetBodyInspectionSize reads the most of a request body the Core Rule Set
-// reads: a size as parseSize reads it, or off, but at most 64K, since waf
-// gets no more of a body than that.
+// reads: a size as parseSize reads it, or off, but at most 8K, so that one
+// request cannot keep the Core Rule Set busy past rproxy's timeout.
func parseCoreRuleSetBodyInspectionSize(value string) (int64, error) {
limit, err := parseSize(value)
if err != nil {
return 0, err
}
- if limit > 64*kibibyte {
- return 0, fmt.Errorf("%q %w", value, errOver64K)
+ if limit > 8*kibibyte {
+ return 0, fmt.Errorf("%q %w", value, errOver8K)
}
return limit, nil
diff --git a/internal/config/waf_test.go b/internal/config/waf_test.go
index c90c0f2..f3ea741 100644
--- a/internal/config/waf_test.go
+++ b/internal/config/waf_test.go
@@ -218,7 +218,7 @@ const (
// instance is an SWW_WAF_INSTANCE_NAME that is a valid app name too, and
// remoteURL an SWW_WAF_LOG_REMOTE_URL, for the tests that send the lines.
const (
- instance = "fsn1app1/blog"
+ instance = "host1/blog"
remoteURL = "syslog+udp://192.0.2.1:514"
)
@@ -616,7 +616,7 @@ func TestCoreRuleSetSettings(t *testing.T) {
coreRuleSetScoreThreshold: "10",
coreRuleSetDisabledRules: "942100, 920350",
coreRuleSetExemptPaths: "/api/, /static/",
- coreRuleSetBodyInspectionSize: "32K",
+ coreRuleSetBodyInspectionSize: "4K",
exclusionsFile: "/etc/sww/crs-exclusions.conf",
},
config.WAF{
@@ -625,7 +625,7 @@ func TestCoreRuleSetSettings(t *testing.T) {
CoreRuleSetScoreThreshold: 10,
CoreRuleSetDisabledRules: []int{942100, 920350},
CoreRuleSetExemptPaths: []string{"/api/", "/static/"},
- CoreRuleSetBodyInspectionSize: 32 << 10,
+ CoreRuleSetBodyInspectionSize: 4 << 10,
CoreRuleSetExclusionsFile: "/etc/sww/crs-exclusions.conf",
},
},
@@ -762,15 +762,15 @@ func TestInvalidAppsStopsTheStart(t *testing.T) {
}
}
-func TestCoreRuleSetBodyInspectionSizeOf64K(t *testing.T) {
+func TestCoreRuleSetBodyInspectionSizeOf8K(t *testing.T) {
t.Parallel()
cfg := wafFromEnvironment(
t,
- environment{coreRuleSetBodyInspectionSize: "64K"},
+ environment{coreRuleSetBodyInspectionSize: "8K"},
)
- if cfg.CoreRuleSetBodyInspectionSize != 64<<10 {
- t.Errorf("64K read as %d", cfg.CoreRuleSetBodyInspectionSize)
+ if cfg.CoreRuleSetBodyInspectionSize != 8<<10 {
+ t.Errorf("8K read as %d", cfg.CoreRuleSetBodyInspectionSize)
}
}
@@ -819,11 +819,13 @@ func TestInvalidCoreRuleSetSettingStopsTheStart(t *testing.T) {
coreRuleSetBodyInspectionSize, "128KB",
`"128KB" is not a size such as 512K, 100M or 5G, or off`,
},
- {coreRuleSetBodyInspectionSize, "65K", `"65K" is more than 64K, the most waf ` +
- `gets of a body`},
{
- coreRuleSetBodyInspectionSize, "65537",
- `"65537" is more than 64K, the most waf gets of a body`,
+ coreRuleSetBodyInspectionSize, "9K",
+ `"9K" is more than 8K, the most the Core Rule Set can read in time`,
+ },
+ {
+ coreRuleSetBodyInspectionSize, "8193",
+ `"8193" is more than 8K, the most the Core Rule Set can read in time`,
},
} {
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
@@ -845,11 +847,11 @@ func TestInstanceNameAndLoggedHeadersAsSet(t *testing.T) {
t.Parallel()
cfg := wafFromEnvironment(t, environment{
- instanceName: "fsn1app1/blog",
+ instanceName: "host1/blog",
logRequestHeaders: " Accept , X-Custom",
})
- if cfg.InstanceName != "fsn1app1/blog" ||
+ if cfg.InstanceName != "host1/blog" ||
!slices.Equal(
cfg.LogRequestHeaders,
[]string{"accept", "x-custom"},
@@ -1022,14 +1024,14 @@ func TestInstanceNameNotAnAppNameStopsTheStartOnlyWhileSending(
) {
t.Parallel()
- const spaced = "fsn1 app1"
+ const spaced = "host1 blog"
sending := environment{logRemoteURL: remoteURL, instanceName: spaced}
_, err := config.WAFFromEnvironment(sending.viper())
want := logRemoteAppName + `: is unset, and ` + instanceName +
- ` "fsn1 app1", its default, is not 1 to 48 printable ASCII characters ` +
+ ` "host1 blog", its default, is not 1 to 48 printable ASCII characters ` +
`without a space, such as blog`
if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want)
@@ -1331,13 +1333,13 @@ func TestInstanceNameWithAControlCharacterStopsTheStartOnlyWithNtfySet(
) {
t.Parallel()
- const name = "fsn1app1\r"
+ const name = "host1\r"
_, err := config.WAFFromEnvironment(environment{
instanceName: name, alertNtfyURL: "https://ntfy.example/sww-alerts",
}.viper())
- want := instanceName + `: "fsn1app1\r" holds a control character, such as the ` +
+ want := instanceName + `: "host1\r" holds a control character, such as the ` +
`carriage return of a Windows line end, and is sent to ntfy in a header ` +
`while ` + alertNtfyURL + ` is set`
if err == nil || err.Error() != want {
diff --git a/internal/coreruleset/coreruleset.go b/internal/coreruleset/coreruleset.go
index 4a3b9ed..83ae46a 100644
--- a/internal/coreruleset/coreruleset.go
+++ b/internal/coreruleset/coreruleset.go
@@ -481,12 +481,21 @@ func (c *CoreRuleSet) Inspect(
return Result{RuleIDs: ids, Score: score(tx)}, read, err
}
+// mostJSONRead is the largest JSON body the Core Rule Set reads, even while
+// SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is larger. Coraza names each
+// value in a JSON body by the whole path to it, so what it gives the Core
+// Rule Set to read can outgrow the body: up to 8 times its size, and up to
+// 4 KiB for a small body. 1 KiB of JSON thus gives at most 8 KiB, no more
+// than the largest body of another kind.
+const mostJSONRead = 1 << 10
+
// readBody reads body, the body of the request in tx, which has run on
// the request's headers, while SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is
// set and the body is of a kind the Core Rule Set reads: form data and
// multipart, of which it reads the first c.bodyInspectionSize bytes, and
// JSON and XML, which it reads only when they are no longer than that,
-// since they cannot be read in part.
+// since they cannot be read in part. The Core Rule Set reads no JSON body
+// longer than mostJSONRead either.
// readBody reads one byte past c.bodyInspectionSize, to tell which they
// are, gives the Core Rule Set what it reads, and returns what it read and
// the error that ended the reading early, if one did.
@@ -499,6 +508,8 @@ func (c *CoreRuleSet) readBody(
}
inPart := false
+ // largest is the largest JSON or XML body the Core Rule Set reads.
+ largest := c.bodyInspectionSize
// How the body is read is a variable of the transaction, which only
// Coraza's interface for plugins reads.
state := tx.(plugintypes.TransactionState) //nolint:forcetypeassert // every one is
@@ -506,14 +517,16 @@ func (c *CoreRuleSet) readBody(
switch state.Variables().RequestBodyProcessor().Get() {
case "URLENCODED", "MULTIPART":
inPart = true
- case "JSON", "XML":
+ case "JSON":
+ largest = min(largest, mostJSONRead)
+ case "XML":
default:
return nil, nil
}
read, err := io.ReadAll(io.LimitReader(body, c.bodyInspectionSize+1))
- if inPart || (err == nil && int64(len(read)) <= c.bodyInspectionSize) {
+ if inPart || (err == nil && int64(len(read)) <= largest) {
// Coraza holds what it reads of the body in memory, up to the
// limit, so this cannot fail.
_, _, _ = tx.WriteRequestBody(read)
diff --git a/internal/coreruleset/coreruleset_test.go b/internal/coreruleset/coreruleset_test.go
index 7db965d..52a3146 100644
--- a/internal/coreruleset/coreruleset_test.go
+++ b/internal/coreruleset/coreruleset_test.go
@@ -390,22 +390,15 @@ func TestMoreParametersThanCorazaKeepsIsAMatch(t *testing.T) {
matched(900300),
)
- // So it is with the fields of a form data or JSON body.
+ // So it is with the fields of a form data body. A JSON body the Core
+ // Rule Set reads, of at most mostJSONRead bytes, cannot hold that many.
crs = readingBodies(t)
- for _, tc := range []struct{ header, body string }{
- {formData, strings.Repeat("a=1&", 999) + attack},
- {jsonBody, `{"a":[` + strings.Repeat("1,", 998) + `1],"id":"` + injection + `"}`},
- } {
- wantBody(t, crs, post(tc.header), tc.body, matched(942100), tc.body)
- }
+ body := strings.Repeat("a=1&", 999) + attack
+ wantBody(t, crs, post(formData), body, matched(942100), body)
- for _, tc := range []struct{ header, body string }{
- {formData, strings.Repeat("a=1&", 1000) + attack},
- {jsonBody, `{"a":[` + strings.Repeat("1,", 999) + `1],"id":"` + injection + `"}`},
- } {
- wantBody(t, crs, post(tc.header), tc.body, matched(900300), tc.body)
- }
+ body = strings.Repeat("a=1&", 1000) + attack
+ wantBody(t, crs, post(formData), body, matched(900300), body)
}
func TestRedirectURIMayNameALocalAddress(t *testing.T) {
@@ -1512,13 +1505,15 @@ const (
wpTags = "/wp-json/wp/v2/tags"
)
-// readingBodiesUpTo64K returns the Core Rule Set as sww runs it by default,
+// readingBodiesUpTo8K returns the Core Rule Set as sww runs it by default,
// but reading bodies up to the largest SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE.
-func readingBodiesUpTo64K(t *testing.T) *coreruleset.CoreRuleSet {
+func readingBodiesUpTo8K(t *testing.T) *coreruleset.CoreRuleSet {
t.Helper()
return shared(t, coreruleset.Params{
- ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, BodyInspectionSize: 64<<10 - 1,
+ ParanoiaLevel: 1,
+ DisabledRules: defaultDisabledRules,
+ BodyInspectionSize: mostBodyRead,
})
}
@@ -1582,7 +1577,7 @@ func TestWordPressAttacksAreRefused(t *testing.T) {
const threshold = 5 // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default
- defaults, bodies := atDefaults(t), readingBodiesUpTo64K(t)
+ defaults, bodies := atDefaults(t), readingBodiesUpTo8K(t)
for _, attack := range []string{
sqlInjection,
@@ -1693,7 +1688,7 @@ func wordpressRESTAddresses(route string) []string {
func TestWordPressExclusionsFollowEveryRESTAddress(t *testing.T) {
t.Parallel()
- crs := readingBodiesUpTo64K(t)
+ crs := readingBodiesUpTo8K(t)
content := `{"content":"` + wordpressBlock + `"}`
for _, route := range []string{
@@ -1919,7 +1914,7 @@ func TestWordPressCommandAfterASeparatorIsRefusedOnSearchesAndNames(
wantResult(t, crs, get(target+command), matched(932235))
}
- bodies := readingBodiesUpTo64K(t)
+ bodies := readingBodiesUpTo8K(t)
for _, p := range wordpressBodyFields(command) {
r := request{http.MethodPost, p.target, []string{p.contentType}}
@@ -2143,7 +2138,7 @@ func TestWordPressBodyFieldsAreCheckedInTheQuery(t *testing.T) {
func TestWordPressBodyNamesAndAddressesAreExcludedOnlyThere(t *testing.T) {
t.Parallel()
- crs := readingBodiesUpTo64K(t)
+ crs := readingBodiesUpTo8K(t)
for _, v := range wordpressWords() {
for _, field := range []string{"title", "excerpt", "name"} {
@@ -2262,7 +2257,7 @@ func TestWordPressExclusionsAreNarrow(t *testing.T) {
length = "Content-Length: 2"
)
- crs := readingBodiesUpTo64K(t)
+ crs := readingBodiesUpTo8K(t)
saving := request{
http.MethodPost,
wpPost,
@@ -2473,19 +2468,35 @@ func TestFormDataAndMultipartAreReadUpToTheLimit(t *testing.T) {
func TestJSONAndXMLAreReadOnlyWhenNoLargerThanTheLimit(t *testing.T) {
t.Parallel()
- crs := readingBodies(t)
+ // The limit is the body inspection size, but at most mostJSONRead for
+ // JSON. smallSize is a body inspection size under mostJSONRead.
+ const smallSize = 512
- for _, tc := range []struct{ header, head, tail string }{
- {jsonBody, `{"q":"` + injection + `","pad":"`, `"}`},
- {xmlBody, "" + injection + "
", ""},
+ bodies, smallBodies := readingBodies(t), shared(t, coreruleset.Params{
+ ParanoiaLevel: 1, DisabledRules: defaultDisabledRules,
+ BodyInspectionSize: smallSize,
+ })
+ jsonHead, jsonTail := `{"q":"`+injection+`","pad":"`, `"}`
+
+ for _, tc := range []struct {
+ crs *coreruleset.CoreRuleSet
+ header, head, tail string
+ limit int
+ }{
+ {bodies, jsonBody, jsonHead, jsonTail, mostJSONRead},
+ {smallBodies, jsonBody, jsonHead, jsonTail, smallSize},
+ {
+ bodies, xmlBody, "" + injection + "
", "",
+ bodyInspectionSize,
+ },
} {
- fits := padded(tc.head, tc.tail, bodyInspectionSize)
- wantBody(t, crs, post(tc.header), fits, matched(942100), fits)
+ fits := padded(tc.head, tc.tail, tc.limit)
+ wantBody(t, tc.crs, post(tc.header), fits, matched(942100), fits)
- larger := padded(tc.head, tc.tail, bodyInspectionSize+1)
+ larger := padded(tc.head, tc.tail, tc.limit+1)
wantBody(
t,
- crs,
+ tc.crs,
post(tc.header),
larger,
coreruleset.Result{},
diff --git a/internal/coreruleset/inspectiontime_test.go b/internal/coreruleset/inspectiontime_test.go
new file mode 100644
index 0000000..c6a688a
--- /dev/null
+++ b/internal/coreruleset/inspectiontime_test.go
@@ -0,0 +1,437 @@
+package coreruleset_test
+
+import (
+ "maps"
+ "net/http"
+ "net/http/httptest"
+ "net/netip"
+ "regexp"
+ "runtime"
+ "slices"
+ "strconv"
+ "strings"
+ "testing"
+ "time"
+
+ "golang.org/x/sys/unix"
+
+ "sneak.berlin/go/smallwebwaf/internal/coreruleset"
+)
+
+// The bounds on what a visitor can send, as SPEC.md gives them.
+const (
+ // lineLimit is the longest request line or header line.
+ lineLimit = 8 << 10
+ // headerLimit is the most bytes rproxy reads of a request line and
+ // headers together.
+ headerLimit = 32 << 10
+ // mostCookies and mostHeaderFields are waf's limits on cookies and header
+ // fields.
+ mostCookies = 50
+ mostHeaderFields = 100
+ // mostBodyRead is the largest SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE.
+ mostBodyRead = 8 << 10
+ // mostJSONRead is the largest JSON body the Core Rule Set reads.
+ mostJSONRead = 1 << 10
+)
+
+// waf refuses a request the Core Rule Set has not inspected within 500
+// milliseconds, but Coraza cannot stop an inspection part-way, so it runs to
+// its end with nobody waiting for it. The test checks that an inspection ends
+// within about 1 second, so that a visitor cannot keep a processor busy for
+// longer with one request.
+//
+// A busy host slows all work, so the test does not compare an inspection with
+// 1 second. It compares it with referenceTime, timed on the same thread just
+// before and after. inspectionTimeLimitMultiple is how many times
+// referenceTime an inspection may take.
+//
+// On the host that runs our checks, referenceTime took at least 78
+// milliseconds, so 13 times it is about 1 second. With that host fully busy,
+// no try of the slowest request took more than 9.5 times referenceTime.
+const inspectionTimeLimitMultiple = 13
+
+// referencePattern is what referenceTime matches. Like the Core Rule Set's
+// rules, it is a regular expression with alternatives and repeats.
+var referencePattern = regexp.MustCompile(
+ `(?i)(union|select|insert|delete)\s+\w+\s+(from|into)\s+\w+[0-9]+;`,
+)
+
+// referenceTime returns the processor time the calling thread takes to do a
+// fixed amount of work unrelated to the Core Rule Set, so that a change to the
+// rules does not change it. referencePattern matches nowhere in the text, so
+// each match reads all of it.
+func referenceTime(t *testing.T) time.Duration {
+ t.Helper()
+
+ referenceText := text(4 << 10)
+ began := threadCPUTime(t)
+
+ for range 200 {
+ if referencePattern.MatchString(referenceText) {
+ t.Fatal("the reference pattern matched")
+ }
+ }
+
+ return threadCPUTime(t) - began
+}
+
+// text returns n bytes of letters, digits and spaces, which the Core Rule Set
+// takes longer over than one letter repeated.
+func text(n int) string {
+ const words = "select 0x2f from tbl_1 where name like abc_9 "
+
+ return strings.Repeat(words, n/len(words)+1)[:n]
+}
+
+// visitorRequest is a request a visitor can send, at the most of one or more
+// of the bounds above.
+type visitorRequest struct {
+ name string
+ target string
+ // header holds the headers besides Host.
+ header http.Header
+ // contentType is the body's type, "" for no body.
+ contentType string
+ body string
+}
+
+// longestTarget returns a target whose request line, "POST
+// HTTP/1.1", is lineLimit bytes long.
+func longestTarget() string {
+ const around = len("POST /?q= HTTP/1.1")
+
+ return "/?q=" + strings.ReplaceAll(text(lineLimit-around), " ", "+")
+}
+
+// cookieLine returns a Cookie value whose line, "Cookie: ", is
+// lineLimit bytes long, made of count cookies.
+func cookieLine(count int) string {
+ const linePrefix = len("Cookie: ")
+
+ cookies := make([]string, count)
+ for i := range cookies {
+ cookies[i] = "c" + strconv.Itoa(i) + "="
+ }
+
+ // Each cookie is followed by "; ", but for the last.
+ room := lineLimit - linePrefix - len(strings.Join(cookies, "; "))
+ for i := range cookies {
+ share := room / count
+ if i == 0 {
+ share += room % count
+ }
+
+ cookies[i] += strings.ReplaceAll(text(share), " ", "_")
+ }
+
+ return strings.Join(cookies, "; ")
+}
+
+// headerValue returns a value whose line, ": ", is n bytes long.
+func headerValue(name string, n int) string {
+ return text(n - len(name) - len(": "))
+}
+
+// formFields returns a form body of mostBodyRead bytes holding count fields.
+func formFields(count int) string {
+ fields := make([]string, count)
+ for i := range fields {
+ fields[i] = "f" + strconv.Itoa(i) + "="
+ }
+
+ room := mostBodyRead - len(strings.Join(fields, "&"))
+ for i := range fields {
+ share := room / count
+ if i == 0 {
+ share += room % count
+ }
+
+ fields[i] += strings.ReplaceAll(text(share), " ", "+")
+ }
+
+ return strings.Join(fields, "&")
+}
+
+// multipartBody returns a multipart body of mostBodyRead bytes, with the
+// boundary "b", holding fields of about size bytes each.
+func multipartBody(size int) string {
+ var body strings.Builder
+
+ const end = "--b--\r\n"
+
+ for i := 0; ; i++ {
+ head := "--b\r\nContent-Disposition: form-data; name=\"f" +
+ strconv.Itoa(i) + "\"\r\n\r\n"
+
+ room := mostBodyRead - body.Len() - len(
+ end,
+ ) - len(
+ head,
+ ) - len(
+ "\r\n",
+ )
+ if room <= 0 {
+ break
+ }
+
+ body.WriteString(head + text(min(size, room)) + "\r\n")
+ }
+
+ body.WriteString(end)
+
+ return body.String()
+}
+
+// jsonArrayOfMostElements returns a JSON array of at most mostJSONRead bytes
+// holding as many elements as fit, each the number 1.
+func jsonArrayOfMostElements() string {
+ return "[1" + strings.Repeat(
+ ",1",
+ (mostJSONRead-len("[1]"))/len(",1"),
+ ) + "]"
+}
+
+// jsonOfMostElementsUnderAKey returns a JSON body of mostJSONRead bytes
+// holding as many elements as fit, each the number 1, in an array under a
+// key of 6 bytes. Coraza names each element "json..", so a
+// longer key makes longer names. 6 bytes is the longest key that keeps the
+// names and values of this body within the 8 KiB Coraza allows.
+func jsonOfMostElementsUnderAKey() string {
+ head := `{"` + text(6) + `":[1`
+
+ return head + strings.Repeat(
+ ",1",
+ (mostJSONRead-len(head+"]}"))/len(",1"),
+ ) + "]}"
+}
+
+// The kinds of body the Core Rule Set reads, and the header that names a
+// request's cookies.
+const (
+ formType = "application/x-www-form-urlencoded"
+ jsonType = "application/json"
+ multipartType = "multipart/form-data; boundary=b"
+ cookieHeader = "Cookie"
+)
+
+// fullLine is the length of each of the request line and the three or four
+// header lines that together fill rproxy's limit, with room for the line
+// endings and the other headers.
+const fullLine = lineLimit - 64
+
+// worstCaseRequests are, for each kind of thing a visitor sends, the requests
+// that keep the Core Rule Set busiest, within the bounds above. The last
+// holds the most of everything at once: Cookie lines keep it busier than
+// other header lines, and the JSON body of the most elements under a key
+// keeps it busier than the other bodies.
+func worstCaseRequests() []visitorRequest {
+ fullCookieLine := cookieLine(1)[:fullLine-len("Cookie: ")]
+
+ return append(
+ append(worstCaseHeaders(), worstCaseBodies()...),
+ visitorRequest{
+ name: "the longest request line, three Cookie lines and body",
+ target: longestTarget()[:fullLine-len("POST HTTP/1.1")],
+ header: http.Header{cookieHeader: {
+ fullCookieLine, fullCookieLine, fullCookieLine,
+ }},
+ contentType: jsonType, body: jsonOfMostElementsUnderAKey(),
+ },
+ )
+}
+
+// worstCaseHeaders are the requests of worstCaseRequests with no body.
+func worstCaseHeaders() []visitorRequest {
+ headerFields := http.Header{}
+
+ for i := range mostHeaderFields - 1 {
+ name := "X-Field-" + strconv.Itoa(i)
+ headerFields.Set(
+ name,
+ headerValue(name, headerLimit/mostHeaderFields),
+ )
+ }
+
+ return []visitorRequest{
+ {name: "the longest request line", target: longestTarget()},
+ {
+ name: "the longest request line, as query parameters",
+ target: "/?" + strings.Repeat(
+ "a=1&",
+ (lineLimit-len("POST /? HTTP/1.1"))/len("a=1&"),
+ ),
+ },
+ {
+ name: "the longest Cookie line, as one cookie",
+ target: "/",
+ header: http.Header{cookieHeader: {cookieLine(1)}},
+ },
+ {
+ name: "the longest Cookie line, as the most cookies",
+ target: "/",
+ header: http.Header{cookieHeader: {cookieLine(mostCookies)}},
+ },
+ {name: "the most header fields", target: "/", header: headerFields},
+ {
+ name: "the longest header lines",
+ target: "/",
+ header: http.Header{
+ "User-Agent": {headerValue("User-Agent", fullLine)},
+ "Referer": {headerValue("Referer", fullLine)},
+ "Accept": {headerValue("Accept", fullLine)},
+ "X-Long": {headerValue("X-Long", fullLine)},
+ },
+ },
+ }
+}
+
+// worstCaseBodies are the requests of worstCaseRequests whose body holds
+// the most the Core Rule Set reads.
+func worstCaseBodies() []visitorRequest {
+ const nested = (mostJSONRead - len(`{"a":1}`)) / len(`{"a":}`)
+
+ return []visitorRequest{
+ {
+ name: "a form body of one field", target: "/",
+ contentType: formType, body: formFields(1),
+ },
+ {
+ name: "a form body of the most fields Coraza keeps", target: "/",
+ contentType: formType, body: formFields(1000),
+ },
+ {
+ name: "a JSON body of one string", target: "/",
+ contentType: jsonType,
+ body: `{"a":"` + text(
+ mostJSONRead-len(`{"a":""}`),
+ ) + `"}`,
+ },
+ {
+ name: "a JSON body of the most elements", target: "/",
+ contentType: jsonType, body: jsonArrayOfMostElements(),
+ },
+ {
+ name: "a JSON body of the most elements under a key", target: "/",
+ contentType: jsonType, body: jsonOfMostElementsUnderAKey(),
+ },
+ {
+ name: "a JSON body nested deep", target: "/",
+ contentType: jsonType,
+ body: strings.Repeat(`{"a":`, nested) + "1" +
+ strings.Repeat("}", nested),
+ },
+ {
+ name: "an XML body of many elements", target: "/",
+ contentType: "application/xml",
+ body: "" + strings.Repeat(
+ "x",
+ (mostBodyRead-len(""))/len("x"),
+ ) + "",
+ },
+ {
+ name: "a multipart body of one field", target: "/",
+ contentType: multipartType, body: multipartBody(mostBodyRead),
+ },
+ {
+ name: "a multipart body of many fields", target: "/",
+ contentType: multipartType, body: multipartBody(1),
+ },
+ }
+}
+
+// threadCPUTime returns the processor time the calling thread has used.
+func threadCPUTime(t *testing.T) time.Duration {
+ t.Helper()
+
+ var now unix.Timespec
+
+ err := unix.ClockGettime(unix.CLOCK_THREAD_CPUTIME_ID, &now)
+ if err != nil {
+ t.Fatalf("read the thread's processor time: %v", err)
+ }
+
+ return time.Duration(now.Nano())
+}
+
+// inspectionTime returns the processor time crs takes to inspect tc, and the
+// mean of referenceTime just before and after. It fails t unless crs reads all
+// of tc's body and Coraza parses it.
+//
+// All of it runs on this goroutine, kept on one thread, so the thread's
+// processor time measures it. Time other work holds the processor does not
+// count.
+func inspectionTime(
+ t *testing.T, crs *coreruleset.CoreRuleSet, tc visitorRequest,
+) (time.Duration, time.Duration) {
+ t.Helper()
+
+ req := httptest.NewRequestWithContext(
+ t.Context(), http.MethodPost, tc.target, nil,
+ )
+ maps.Copy(req.Header, tc.header)
+
+ if tc.contentType != "" {
+ req.Header.Set("Content-Type", tc.contentType)
+ }
+
+ runtime.LockOSThread()
+
+ defer runtime.UnlockOSThread()
+
+ before := referenceTime(t)
+
+ began := threadCPUTime(t)
+ result, read, _ := crs.Inspect(
+ req, netip.MustParseAddr("192.0.2.1"), strings.NewReader(tc.body),
+ )
+ took := threadCPUTime(t) - began
+
+ reference := (before + referenceTime(t)) / 2
+
+ if string(read) != tc.body {
+ t.Errorf("read %d bytes of %d", len(read), len(tc.body))
+ }
+
+ // Rule 900440 matches a body Coraza could not parse.
+ if slices.Contains(result.RuleIDs, 900440) {
+ t.Errorf("Coraza could not parse the body")
+ }
+
+ return took, reference
+}
+
+// TestTheCoreRuleSetInspectsEachWorstCaseRequestInTime runs one at a time, so
+// that the tests running in parallel do not slow it. Other work on the host
+// can still slow one try more than the reference work around it, so a request
+// fails only when three tries in a row take too long.
+//
+//nolint:paralleltest // one at a time, as above
+func TestTheCoreRuleSetInspectsEachWorstCaseRequestInTime(t *testing.T) {
+ const tries = 3
+
+ // The settings that make the Core Rule Set slowest: the highest paranoia
+ // level, no rule switched off, and the most of a body read.
+ crs := shared(t, coreruleset.Params{
+ ParanoiaLevel: 4, BodyInspectionSize: mostBodyRead,
+ })
+
+ for _, tc := range worstCaseRequests() {
+ t.Run(tc.name, func(t *testing.T) {
+ var took, reference time.Duration
+
+ for range tries {
+ took, reference = inspectionTime(t, crs, tc)
+ if took <= inspectionTimeLimitMultiple*reference {
+ return
+ }
+ }
+
+ t.Errorf(
+ "took %s on the last of %d tries, over %d times the %s "+
+ "of the reference work",
+ took, tries, inspectionTimeLimitMultiple, reference,
+ )
+ })
+ }
+}
diff --git a/internal/lookup/export_test.go b/internal/geojs/export_test.go
similarity index 93%
rename from internal/lookup/export_test.go
rename to internal/geojs/export_test.go
index 3276a6c..360ff2d 100644
--- a/internal/lookup/export_test.go
+++ b/internal/geojs/export_test.go
@@ -1,4 +1,4 @@
-package lookup
+package geojs
import "net/http"
diff --git a/internal/lookup/lookup.go b/internal/geojs/geojs.go
similarity index 75%
rename from internal/lookup/lookup.go
rename to internal/geojs/geojs.go
index d5d5db0..962bc51 100644
--- a/internal/lookup/lookup.go
+++ b/internal/geojs/geojs.go
@@ -1,9 +1,9 @@
-// Package lookup looks up each client's AS number and country, through
-// the GeoJS web service or in the lookup database, the IPinfo Lite file
-// SWW_WAF_LOOKUP_DB_PATH names. GeoJS's answers are kept in memory, for at
-// most 100,000 clients and for 7 days each, and are written to
-// lookups.json and read from it by the state package.
-package lookup
+// Package geojs looks up clients' AS numbers and countries through the GeoJS
+// web service. Its answers are kept in memory, for at most 100,000 clients and
+// for 7 days each. A caller waits for an answer as long as it says. A request to
+// GeoJS goes on after its callers stop waiting, and its answer is kept. The
+// package imports no other package of this repository.
+package geojs
import (
"context"
@@ -11,7 +11,6 @@ import (
"errors"
"fmt"
"io"
- "log/slog"
"net/http"
"net/netip"
"slices"
@@ -21,8 +20,6 @@ import (
"time"
"github.com/hashicorp/golang-lru/v2/simplelru"
- "sneak.berlin/go/smallwebwaf/internal/alerts"
- "sneak.berlin/go/smallwebwaf/internal/metrics"
)
// URL is GeoJS's endpoint for an address's place and network. Asked about
@@ -43,6 +40,9 @@ const (
maxWaiting = 10000
// maxPerRequest is how many addresses one request to GeoJS asks about.
maxPerRequest = 200
+ // requestTimeout is how long a request to GeoJS may take before it is
+ // abandoned, which is a failure. It is apart from how long callers wait.
+ requestTimeout = 10 * time.Second
// unknownASN is the AS number GeoJS gives when it knows none.
unknownASN = 64512
// After a failure GeoJS is not asked again for a second, and for
@@ -64,39 +64,32 @@ var (
type Params struct {
// URL is where GeoJS is asked, normally URL.
URL string
- // Timeout is how long a request waits for its client's first answer,
- // and how long a request to GeoJS may take before it is abandoned
- // (SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS).
- Timeout time.Duration
- // Wait is true when a setting needs each request's answer before the
- // request goes on. Otherwise no request waits for one.
- Wait bool
// Answered, unless nil, is given each answer GeoJS gives, once it is
// kept.
Answered func(Answer)
// Now tells the time, normally time.Now in UTC.
Now func() time.Time
- // ProcessLog receives GeoJS's failures.
- ProcessLog *slog.Logger
- // Metrics count the requests to GeoJS, those that failed, and the
- // clients that go without an answer.
- Metrics *metrics.Metrics
- // Alerts receive a source_failure alert each time GeoJS fails.
- Alerts *alerts.Queue
+ // Asked is called for each request to GeoJS.
+ Asked func()
+ // Failed is called for each failure of GeoJS, with what went wrong and
+ // how long GeoJS is left alone after it. The error never names the
+ // addresses asked about.
+ Failed func(err error, askingAgainIn time.Duration)
+ // WentWithoutAnswer is called each time a caller that waits for an
+ // answer goes on without one.
+ WentWithoutAnswer func()
}
// GeoJS looks up clients' AS numbers and countries through GeoJS. At most
// one request to GeoJS is under way at a time, and it asks about every
// client waiting, up to maxPerRequest. It is safe for concurrent use.
type GeoJS struct {
- url string
- timeout time.Duration
- wait bool
- answered func(Answer)
- now func() time.Time
- processLog *slog.Logger
- metrics *metrics.Metrics
- alerts *alerts.Queue
+ url string
+ answered func(Answer)
+ now func() time.Time
+ asked func()
+ failed func(err error, askingAgainIn time.Duration)
+ wentWithoutAnswer func()
// httpClient follows no redirect, so that visitors' addresses go to
// GeoJS alone: a redirect is a failure.
httpClient *http.Client
@@ -114,12 +107,10 @@ type GeoJS struct {
retryAt time.Time
}
-// Answer is what GeoJS or the lookup database said about a client: its AS
-// number, such as AS64496, and the AS's name, both "" when the source knows
-// no AS number for it; its country, "" when the source cannot place it;
-// when the source said so; and, for GeoJS's answers, which lookups.json
-// holds, when the answer was last used. The zero Answer is that of a
-// client with no answer.
+// Answer is what GeoJS said about a client: its AS number, such as AS64496,
+// and the AS's name, both "" when GeoJS knows no AS number for it; its
+// country, "" when GeoJS cannot place it; when GeoJS said so; and when the
+// answer was last used. The zero Answer is that of a client with no answer.
//
//nolint:tagliatelle // the state files use snake_case, as the request log does
type Answer struct {
@@ -137,7 +128,7 @@ type wait struct {
// replaced each time GeoJS fails before then.
asked chan struct{}
// late is true once the client has gone without an answer, for a
- // whole timeout or because GeoJS failed: its requests no longer wait.
+ // whole wait or because GeoJS failed: its callers no longer wait.
late bool
}
@@ -149,14 +140,12 @@ func New(params Params) *GeoJS {
}
return &GeoJS{
- url: params.URL,
- timeout: params.Timeout,
- wait: params.Wait,
- answered: params.Answered,
- now: params.Now,
- processLog: params.ProcessLog,
- metrics: params.Metrics,
- alerts: params.Alerts,
+ url: params.URL,
+ answered: params.Answered,
+ now: params.Now,
+ asked: params.Asked,
+ failed: params.Failed,
+ wentWithoutAnswer: params.WentWithoutAnswer,
httpClient: &http.Client{
CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse
@@ -170,21 +159,22 @@ func New(params Params) *GeoJS {
// LookUp returns the answer GeoJS gave about client, with its country as
// a two-letter code in capitals, or the zero Answer when there is none
// yet. An answer is kept for 7 days. Without one, the client is asked
-// about in the background, and, while Wait is set, the request waits up
-// to Timeout for the answer, unless the client has gone without one
-// before. ctx is the context of the client's request, and ends the wait
-// when it ends.
+// about in the background, and LookUp waits up to wait for the answer,
+// unless the client has gone without one before. The wait ends early when
+// ctx ends. Neither ends the request to GeoJS.
//
// GeoJS is asked about the client's first address, which is the client's
// own address for IPv4, and an address in the same place for an IPv6
// netblock.
-func (g *GeoJS) LookUp(ctx context.Context, client netip.Prefix) Answer {
- answer, asked := g.answerOrWait(ctx, client, g.wait)
+func (g *GeoJS) LookUp(
+ ctx context.Context, client netip.Prefix, wait time.Duration,
+) Answer {
+ answer, asked := g.answerOrWait(ctx, client, true)
if asked == nil {
return answer
}
- timer := time.NewTimer(g.timeout)
+ timer := time.NewTimer(wait)
defer timer.Stop()
select {
@@ -198,7 +188,7 @@ func (g *GeoJS) LookUp(ctx context.Context, client netip.Prefix) Answer {
answer, found := g.kept(client)
if !found {
- g.metrics.RequestsWithoutGeoJSAnswer.Inc()
+ g.wentWithoutAnswer()
}
w, waiting := g.waiting[client]
@@ -209,8 +199,8 @@ func (g *GeoJS) LookUp(ctx context.Context, client netip.Prefix) Answer {
return answer
}
-// LookUpWithoutWaiting is LookUp for a request that does not need the answer,
-// whatever Wait says: it returns the zero Answer at once when none is kept.
+// LookUpWithoutWaiting is LookUp for a caller that does not need the answer:
+// it returns the zero Answer at once when none is kept.
func (g *GeoJS) LookUpWithoutWaiting(
ctx context.Context,
client netip.Prefix,
@@ -228,8 +218,7 @@ func (g *GeoJS) Kept(client netip.Prefix) (Answer, bool) {
return g.kept(client)
}
-// Snapshot returns every answer kept, sorted by client, as lookups.json
-// lists them.
+// Snapshot returns every answer kept, sorted by client.
func (g *GeoJS) Snapshot() []Answer {
g.mu.Lock()
@@ -247,9 +236,9 @@ func (g *GeoJS) Snapshot() []Answer {
return answers
}
-// Load keeps answers read from lookups.json, in place of the answers it
-// keeps, in the order they were last used, so that the one used longest
-// ago is dropped first. Answers GeoJS gave keepFor ago or more are
+// Load keeps answers, such as those of an earlier Snapshot, in place of the
+// answers it keeps, in the order they were last used, so that the one used
+// longest ago is dropped first. Answers GeoJS gave keepFor ago or more are
// dropped.
func (g *GeoJS) Load(answers []Answer) {
answers = slices.Clone(answers)
@@ -295,11 +284,11 @@ func (g *GeoJS) answerOrWait(
g.ask(ctx)
if !mustWait {
- return Answer{}, nil // the answer is not needed before the request goes on
+ return Answer{}, nil // the answer is not needed before the caller goes on
}
if w == nil {
- g.metrics.RequestsWithoutGeoJSAnswer.Inc()
+ g.wentWithoutAnswer()
return Answer{}, nil // too many clients wait already
}
@@ -310,7 +299,7 @@ func (g *GeoJS) answerOrWait(
}
if w.late {
- g.metrics.RequestsWithoutGeoJSAnswer.Inc()
+ g.wentWithoutAnswer()
return Answer{}, nil
}
@@ -335,8 +324,8 @@ func (g *GeoJS) kept(client netip.Prefix) (Answer, bool) {
// ask starts asking GeoJS about the waiting clients, unless a request to
// it is under way or it is left alone after a failure. The requests to
-// GeoJS are for every client waiting, so they go on when the client's
-// request whose ctx is given ends.
+// GeoJS are for every client waiting, so they go on when the caller whose
+// ctx is given stops waiting.
func (g *GeoJS) ask(ctx context.Context) {
if g.asking || g.now().Before(g.retryAt) {
return
@@ -441,8 +430,6 @@ func (g *GeoJS) keep(
}
if err != nil {
- g.metrics.GeoJSFailures.Inc()
-
g.retryDelay = min(
max(retryDelayFactor*g.retryDelay, firstRetryDelay),
maxRetryDelay,
@@ -457,16 +444,7 @@ func (g *GeoJS) keep(
w.late = true
}
- g.processLog.Warn("asking GeoJS failed",
- "error", err.Error(), "asking_again_in", g.retryDelay.String())
- g.alerts.Raise(alerts.Alert{
- Event: alerts.EventSourceFailure,
- Reason: "asking GeoJS failed",
- Detail: map[string]any{
- "source": "geojs", "error": err.Error(),
- "asking_again_in": g.retryDelay.String(),
- },
- })
+ g.failed(err, g.retryDelay)
return kept, false
}
@@ -479,7 +457,8 @@ func (g *GeoJS) keep(
// request asks GeoJS about clients in one request, and returns the answer
// for each address GeoJS's answer names: its AS number and the AS's name,
// both "" for the AS number 64512, which GeoJS gives when it knows none,
-// and its country, in capitals.
+// and its country, in capitals. The request is abandoned after
+// requestTimeout.
func (g *GeoJS) request(
ctx context.Context, clients []netip.Prefix,
) (map[netip.Addr]Answer, error) {
@@ -489,7 +468,7 @@ func (g *GeoJS) request(
addrs = append(addrs, client.Addr().String())
}
- ctx, cancel := context.WithTimeout(ctx, g.timeout)
+ ctx, cancel := context.WithTimeout(ctx, requestTimeout)
defer cancel()
req, err := http.NewRequestWithContext(
@@ -504,7 +483,7 @@ func (g *GeoJS) request(
req.URL.RawQuery = "ip=" + strings.Join(addrs, ",")
- g.metrics.GeoJSRequests.Inc()
+ g.asked()
res, err := g.httpClient.Do(req)
if err != nil {
diff --git a/internal/lookup/lookup_test.go b/internal/geojs/geojs_test.go
similarity index 60%
rename from internal/lookup/lookup_test.go
rename to internal/geojs/geojs_test.go
index 4f67d02..a793f6b 100644
--- a/internal/lookup/lookup_test.go
+++ b/internal/geojs/geojs_test.go
@@ -1,13 +1,11 @@
-package lookup_test
+package geojs_test
import (
+ "context"
"encoding/json"
- "log/slog"
"net/http"
"net/http/httptest"
"net/netip"
- "net/url"
- "reflect"
"slices"
"strings"
"sync"
@@ -15,10 +13,7 @@ import (
"testing/synctest"
"time"
- "github.com/prometheus/client_golang/prometheus/testutil"
- "sneak.berlin/go/smallwebwaf/internal/alerts"
- "sneak.berlin/go/smallwebwaf/internal/lookup"
- "sneak.berlin/go/smallwebwaf/internal/metrics"
+ "sneak.berlin/go/smallwebwaf/internal/geojs"
)
const (
@@ -34,8 +29,14 @@ const (
// leftOut is the address it leaves out of its answer when
// answeringWithoutLeftOut.
leftOut = "203.0.113.7"
- // timeout is how long a new client waits for its answer.
- timeout = time.Second
+ // wait is how long a lookup waits for a new client's answer.
+ wait = time.Second
+ // requestTimeout is how long a request to GeoJS may take before it is
+ // abandoned.
+ requestTimeout = 10 * time.Second
+ // lateAnswer is how long the stand-in takes to answer when
+ // answeringLate.
+ lateAnswer = 5 * time.Second
// week is how long an answer is kept.
week = 7 * 24 * time.Hour
)
@@ -51,24 +52,24 @@ func TestKeptAnswerIsUsedFor7DaysThenAskedAgain(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
- geojs, clock, g := start()
+ service, clock, g := start()
placed := netip.MustParsePrefix("203.0.113.9/32")
notPlaced := netip.MustParsePrefix(unplaced + "/32")
wantCountry(t, g, placed, germany)
wantCountry(t, g, notPlaced, "")
- wantRequests(t, geojs, 2)
+ wantRequests(t, service, 2)
// An answer without a country is kept too.
clock.advance(week - time.Second)
wantCountry(t, g, placed, germany)
wantCountry(t, g, notPlaced, "")
- wantRequests(t, geojs, 2)
+ wantRequests(t, service, 2)
clock.advance(time.Second)
wantCountry(t, g, placed, germany)
- wantRequests(t, geojs, 3)
- wantAsked(t, geojs, 2, "203.0.113.9")
+ wantRequests(t, service, 3)
+ wantAsked(t, service, 2, "203.0.113.9")
})
}
@@ -76,17 +77,17 @@ func TestLookUpWithoutWaitingAsksGeoJSAndGoesOnAtOnce(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
- geojs, clock, g := start()
+ service, clock, g := start()
client := netip.MustParsePrefix("203.0.113.9/32")
- // Wait is set, and GeoJS does not answer.
- geojs.set(hanging)
+ // GeoJS does not answer.
+ service.set(hanging)
began := time.Now()
answer := g.LookUpWithoutWaiting(t.Context(), client)
if waited := time.Since(began); waited != 0 ||
- answer != (lookup.Answer{}) {
+ answer != (geojs.Answer{}) {
t.Errorf(
"got %+v after %s, want no answer at once",
answer,
@@ -94,14 +95,14 @@ func TestLookUpWithoutWaitingAsksGeoJSAndGoesOnAtOnce(t *testing.T) {
)
}
- waitForRequests(t, geojs, 1)
- wantAsked(t, geojs, 0, "203.0.113.9")
+ waitForRequests(t, service, 1)
+ wantAsked(t, service, 0, "203.0.113.9")
// That request is abandoned, and has failed once this returns. GeoJS
// answers when it is asked again, and the answer is kept.
- time.Sleep(timeout)
+ time.Sleep(requestTimeout)
synctest.Wait()
- geojs.set(answering)
+ service.set(answering)
waitForCountry(t, g, clock, client, germany)
if got := g.LookUpWithoutWaiting(t.Context(), client).Country; got != germany {
@@ -114,39 +115,45 @@ func TestLookUpWithoutWaitingAsksGeoJSAndGoesOnAtOnce(t *testing.T) {
})
}
-func TestNewClientWaitsAtMostOneSecondThenCountsAsNotFound(t *testing.T) {
+func TestNewClientWaitsAsLongAsItsCallerSaysThenCountsAsNotFound(
+ t *testing.T,
+) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
- geojs, clock, g := start()
+ service, clock, g := start()
client := netip.MustParsePrefix("203.0.113.9/32")
// The client comes while GeoJS is asked about an earlier client, which
// it answers most of a second later. It is then asked about the client
- // and does not answer: that request is abandoned a second after it
+ // and does not answer: that request is abandoned ten seconds after it
// began, well after the client's wait is over.
- geojs.set(answeringSlowly)
+ service.set(answeringSlowly)
var earlier sync.WaitGroup
- earlier.Go(
- func() { g.LookUp(t.Context(), netip.MustParsePrefix("203.0.113.1/32")) },
- )
+ earlier.Go(func() {
+ g.LookUp(
+ t.Context(),
+ netip.MustParsePrefix("203.0.113.1/32"),
+ wait,
+ )
+ })
defer earlier.Wait()
- waitForRequests(t, geojs, 1)
- geojs.set(hanging)
+ waitForRequests(t, service, 1)
+ service.set(hanging)
began := time.Now()
wantCountry(t, g, client, "")
took := time.Since(began)
- if took != timeout {
- t.Errorf("waited %s for the answer, want %s", took, timeout)
+ if took != wait {
+ t.Errorf("waited %s for the answer, want %s", took, wait)
}
- // Its next request does not wait.
+ // Its next lookup does not wait.
began = time.Now()
wantCountry(t, g, client, "")
@@ -158,66 +165,212 @@ func TestNewClientWaitsAtMostOneSecondThenCountsAsNotFound(t *testing.T) {
// Once GeoJS answers, the client is asked about again in the
// background, and has its country.
- geojs.set(answering)
+ service.set(answering)
waitForCountry(t, g, clock, client, germany)
})
}
-func TestRequestWaitsAsLongAsTheTimeoutSaysAndGeoJSIsAbandonedAfterIt(
+func TestAnswerAfterTheWaitIsKeptAndIsNoFailure(t *testing.T) {
+ t.Parallel()
+
+ synctest.Test(t, func(t *testing.T) {
+ service, _, g, reports := startWithReports()
+ client := netip.MustParsePrefix("203.0.113.9/32")
+
+ // GeoJS answers after the lookup's wait is over.
+ service.set(answeringLate)
+ wantCountry(t, g, client, "")
+
+ time.Sleep(lateAnswer - wait)
+ synctest.Wait()
+
+ // The answer is kept, and the next lookup returns it without asking
+ // GeoJS again.
+ answer, found := g.Kept(client)
+ if !found || answer.Country != germany {
+ t.Errorf(
+ "kept %+v, %t; want the answer in %s",
+ answer,
+ found,
+ germany,
+ )
+ }
+
+ wantCountry(t, g, client, germany)
+ wantRequests(t, service, 1)
+ reports.wantFailures(t)
+
+ // GeoJS is not left alone: a new client is asked about at once.
+ service.set(answering)
+ wantCountry(t, g, netip.MustParsePrefix("203.0.113.10/32"), germany)
+ wantRequests(t, service, 2)
+ })
+}
+
+func TestAnswerAfter5SecondsIsReadFromTheCache6SecondsAfterTheFirstLookUp(
t *testing.T,
) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
- // A timeout longer than the default, and a GeoJS that does not
- // answer.
- const longerTimeout = 3 * time.Second
+ const shortWait = 800 * time.Millisecond
- m := metrics.New(1, "app")
- g := lookup.New(lookup.Params{
- URL: lookup.URL,
- Timeout: longerTimeout,
- Wait: true,
- Now: time.Now,
- ProcessLog: slog.New(slog.DiscardHandler),
- Metrics: m,
- Alerts: alerts.New(alerts.Params{}),
- })
- g.SetTransport(&standIn{answers: hanging})
+ service, _, g := start()
+ client := netip.MustParsePrefix("203.0.113.9/32")
- var (
- request sync.WaitGroup
- waited time.Duration
- )
+ service.set(answeringLate)
- request.Go(func() {
- began := time.Now()
+ began := time.Now()
- wantCountry(t, g, netip.MustParsePrefix("203.0.113.9/32"), "")
-
- waited = time.Since(began)
- })
-
- // A moment before the timeout runs out, GeoJS is still being asked:
- // the request to it has not failed.
- time.Sleep(longerTimeout - time.Millisecond)
- synctest.Wait()
- wantFailures(t, m, 0)
-
- // As it runs out, the client's request goes on, and the request to
- // GeoJS is abandoned, which counts as a failure.
- request.Wait()
- synctest.Wait()
-
- if waited != longerTimeout {
+ answer := g.LookUp(t.Context(), client, shortWait)
+ if took := time.Since(began); took != shortWait ||
+ answer != (geojs.Answer{}) {
t.Errorf(
- "waited %s for the answer, want %s",
- waited,
- longerTimeout,
+ "got %+v after %s, want no answer after %s",
+ answer,
+ took,
+ shortWait,
)
}
- wantFailures(t, m, 1)
+ time.Sleep(6*time.Second - shortWait)
+
+ began = time.Now()
+
+ answer = g.LookUp(t.Context(), client, shortWait)
+ if took := time.Since(began); took != 0 ||
+ answer.Country != germany {
+ t.Errorf(
+ "got %+v after %s, want the answer in %s at once",
+ answer,
+ took,
+ germany,
+ )
+ }
+
+ wantRequests(t, service, 1)
+ })
+}
+
+func TestTenLookUpsAtOnceOfOneClientCauseOneRequest(t *testing.T) {
+ t.Parallel()
+
+ synctest.Test(t, func(t *testing.T) {
+ service, _, g := start()
+ client := netip.MustParsePrefix("203.0.113.9/32")
+
+ service.set(answeringSlowly)
+
+ var lookUps sync.WaitGroup
+
+ for range 10 {
+ lookUps.Go(func() { wantCountry(t, g, client, germany) })
+ }
+
+ lookUps.Wait()
+ wantRequests(t, service, 1)
+ wantAsked(t, service, 0, "203.0.113.9")
+ })
+}
+
+func TestCallerWhoseContextEndsEarlyDoesNotStopTheRequest(t *testing.T) {
+ t.Parallel()
+
+ synctest.Test(t, func(t *testing.T) {
+ const early = 100 * time.Millisecond
+
+ service, _, g, reports := startWithReports()
+ client := netip.MustParsePrefix("203.0.113.9/32")
+
+ service.set(answeringLate)
+
+ ctx, cancel := context.WithTimeout(t.Context(), early)
+ defer cancel()
+
+ began := time.Now()
+
+ answer := g.LookUp(ctx, client, wait)
+ if took := time.Since(began); took != early ||
+ answer != (geojs.Answer{}) {
+ t.Errorf(
+ "got %+v after %s, want no answer after %s",
+ answer,
+ took,
+ early,
+ )
+ }
+
+ time.Sleep(lateAnswer)
+ synctest.Wait()
+
+ answer, found := g.Kept(client)
+ if !found || answer.Country != germany {
+ t.Errorf(
+ "kept %+v, %t; want the answer in %s",
+ answer,
+ found,
+ germany,
+ )
+ }
+
+ wantRequests(t, service, 1)
+ reports.wantFailures(t)
+ })
+}
+
+func TestRequestOver10SecondsIsAFailure(t *testing.T) {
+ t.Parallel()
+
+ synctest.Test(t, func(t *testing.T) {
+ service, _, g, reports := startWithReports()
+
+ // GeoJS does not answer.
+ service.set(hanging)
+ wantCountry(t, g, netip.MustParsePrefix("203.0.113.9/32"), "")
+
+ // A moment before 10 seconds are up, the request has not failed.
+ time.Sleep(requestTimeout - wait - time.Millisecond)
+ synctest.Wait()
+ reports.wantFailures(t)
+
+ // As they run out, it is abandoned, which is a failure. The error
+ // does not name the address asked about.
+ time.Sleep(time.Millisecond)
+ synctest.Wait()
+ reports.wantFailures(t, failure{
+ err: "ask GeoJS: context deadline exceeded",
+ askingAgainIn: time.Second,
+ })
+
+ // GeoJS is then left alone, so a new client is not asked about.
+ wantCountry(t, g, netip.MustParsePrefix("203.0.113.10/32"), "")
+ wantRequests(t, service, 1)
+ })
+}
+
+func TestFailureIsReportedWithHowLongGeoJSIsLeftAlone(t *testing.T) {
+ t.Parallel()
+
+ synctest.Test(t, func(t *testing.T) {
+ service, clock, g, reports := startWithReports()
+ clients := newClients()
+
+ service.set(failing)
+ wantCountry(t, g, clients(), "")
+
+ clock.advance(time.Second)
+ wantCountry(t, g, clients(), "")
+
+ reports.wantFailures(t,
+ failure{
+ err: "GeoJS answered 503 Service Unavailable",
+ askingAgainIn: time.Second,
+ },
+ failure{
+ err: "GeoJS answered 503 Service Unavailable",
+ askingAgainIn: 2 * time.Second,
+ },
+ )
})
}
@@ -238,32 +391,32 @@ func TestAddressLeftOutOfAnAnswerIsAskedAboutAgain(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
- geojs, clock, g := start()
+ service, clock, g := start()
other := netip.MustParsePrefix("203.0.113.1/32")
client := netip.MustParsePrefix(leftOut + "/32")
// GeoJS fails, and is left alone for a second while the client
// comes too, so that the next request asks about both.
- geojs.set(failing)
+ service.set(failing)
wantCountry(t, g, other, "")
wantCountry(t, g, client, "")
- geojs.set(tc.answers)
+ service.set(tc.answers)
clock.advance(time.Second)
wantCountry(t, g, other, "")
- waitForRequests(t, geojs, 2)
+ waitForRequests(t, service, 2)
// The answer counts as a failure, and the client is asked about
// again, with the other client only if the answer left it out too.
- geojs.set(answering)
+ service.set(answering)
waitForCountry(t, g, clock, client, germany)
wantCountry(t, g, other, germany)
- wantRequests(t, geojs, 3)
+ wantRequests(t, service, 3)
if tc.named {
- wantAsked(t, geojs, 2, leftOut)
+ wantAsked(t, service, 2, leftOut)
} else {
- wantAsked(t, geojs, 2, leftOut, "203.0.113.1")
+ wantAsked(t, service, 2, leftOut, "203.0.113.1")
}
})
})
@@ -274,11 +427,11 @@ func TestRedirectCountsAsFailure(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
- geojs, _, g := start()
- geojs.set(redirecting)
+ service, _, g := start()
+ service.set(redirecting)
wantCountry(t, g, netip.MustParsePrefix("203.0.113.9/32"), "")
- wantRequests(t, geojs, 1)
+ wantRequests(t, service, 1)
})
}
@@ -286,8 +439,8 @@ func TestCountryIsKeptInCapitals(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
- geojs, _, g := start()
- geojs.set(answeringInLowerCase)
+ service, _, g := start()
+ service.set(answeringInLowerCase)
wantCountry(t, g, netip.MustParsePrefix("203.0.113.9/32"), germany)
})
@@ -304,14 +457,14 @@ func TestAnswerHoldsTheASNumberTheASNameAndTheCountry(t *testing.T) {
// For the client it cannot place, GeoJS gives the AS number 64512
// and the AS name Unknown, which count as unknown.
- for client, want := range map[netip.Prefix]lookup.Answer{
+ for client, want := range map[netip.Prefix]geojs.Answer{
placed: {
Client: placed, ASN: asn, ASName: asName, Country: germany,
Answered: now, Used: now,
},
notPlaced: {Client: notPlaced, Answered: now, Used: now},
} {
- got := g.LookUp(t.Context(), client)
+ got := g.LookUp(t.Context(), client, wait)
if got != want {
t.Errorf("answer for %s\n%+v\nwant\n%+v", client, got, want)
}
@@ -319,146 +472,54 @@ func TestAnswerHoldsTheASNumberTheASNameAndTheCountry(t *testing.T) {
})
}
-func TestWithoutWaitTheRequestGoesOnAtOnceAndTheAnswerIsGivenWhenItComes(
- t *testing.T,
-) {
+func TestEachAnswerIsGivenToAnsweredOnceKept(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
var (
mu sync.Mutex
- given []lookup.Answer
+ given []geojs.Answer
)
- geojs := &standIn{answers: answeringSlowly}
+ service := &standIn{answers: answeringSlowly}
clock := newClock()
- m := metrics.New(1, "app")
- g := lookup.New(lookup.Params{
- URL: lookup.URL,
- Timeout: timeout,
- Answered: func(answer lookup.Answer) {
+ g, reports := newGeoJS(service, geojs.Params{
+ Answered: func(answer geojs.Answer) {
mu.Lock()
defer mu.Unlock()
given = append(given, answer)
},
- Now: clock.Now,
- ProcessLog: slog.New(slog.DiscardHandler),
- Metrics: m,
- Alerts: alerts.New(alerts.Params{}),
+ Now: clock.Now,
})
- g.SetTransport(geojs)
client := netip.MustParsePrefix("203.0.113.9/32")
- // The request goes on at once, without an answer, and GeoJS is asked
- // about the client, which it answers most of a second later.
- began := time.Now()
-
- got := g.LookUp(t.Context(), client)
- if took := time.Since(began); took != 0 || got != (lookup.Answer{}) {
- t.Errorf(
- "waited %s for %+v, want no wait and no answer",
- took,
- got,
- )
+ // GeoJS answers about the client most of a second later.
+ if got := g.LookUpWithoutWaiting(t.Context(), client); got != (geojs.Answer{}) {
+ t.Errorf("got %+v, want no answer", got)
}
- waitForRequests(t, geojs, 1)
- wantAsked(t, geojs, 0, "203.0.113.9")
+ waitForRequests(t, service, 1)
+ wantAsked(t, service, 0, "203.0.113.9")
- time.Sleep(timeout)
+ time.Sleep(wait)
synctest.Wait()
now := clock.Now()
- want := lookup.Answer{
+ want := geojs.Answer{
Client: client, ASN: asn, ASName: asName, Country: germany,
Answered: now, Used: now,
}
mu.Lock()
- if !slices.Equal(given, []lookup.Answer{want}) {
+ if !slices.Equal(given, []geojs.Answer{want}) {
t.Errorf("answers given %+v, want only %+v", given, want)
}
mu.Unlock()
wantCountry(t, g, client, germany)
- wantRequestsWithoutGeoJSAnswer(t, m, 0)
- })
-}
-
-func TestFailureIsLoggedWithoutTheAddressesAskedAbout(t *testing.T) {
- t.Parallel()
-
- synctest.Test(t, func(t *testing.T) {
- var log strings.Builder
-
- // GeoJS does not answer, so the request to it is abandoned, and fails.
- geojs := &standIn{answers: hanging}
- g := lookup.New(lookup.Params{
- URL: lookup.URL,
- Timeout: timeout,
- Wait: true,
- Now: time.Now,
- ProcessLog: slog.New(slog.NewTextHandler(&log, nil)),
- Metrics: metrics.New(1, "app"),
- Alerts: alerts.New(alerts.Params{}),
- })
- g.SetTransport(geojs)
-
- wantCountry(t, g, netip.MustParsePrefix("203.0.113.9/32"), "")
- synctest.Wait()
-
- logged := log.String()
- if !strings.Contains(logged, "asking GeoJS failed") ||
- strings.Contains(logged, "203.0.113.9") {
- t.Errorf(
- "logged %q, want the failure without the address asked about",
- logged,
- )
- }
- })
-}
-
-func TestFailureRaisesASourceFailureAlertOncePerCooldown(t *testing.T) {
- t.Parallel()
-
- synctest.Test(t, func(t *testing.T) {
- geojs, clock, g, queue := startWithAlerts()
- clients := newClients()
-
- geojs.set(failing)
-
- wantCountry(t, g, clients(), "")
-
- want := alerts.Alert{
- Time: clock.Now(),
- Event: alerts.EventSourceFailure,
- Reason: "asking GeoJS failed",
- Detail: map[string]any{
- "source": "geojs",
- "error": "GeoJS answered 503 Service Unavailable",
- "asking_again_in": "1s",
- },
- }
-
- // The next failure, a second later, is a repeat within the
- // cooldown.
- clock.advance(time.Second)
- wantCountry(t, g, clients(), "")
- wantRequests(t, geojs, 2)
-
- waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
- if len(waiting) != 1 || !reflect.DeepEqual(waiting[0], want) {
- t.Errorf("alerts waiting %+v, want only %+v", waiting, want)
- }
-
- if queue.Suppressed() != 1 {
- t.Errorf(
- "%d alerts held back, want the repeat",
- queue.Suppressed(),
- )
- }
+ reports.wantWithoutAnswer(t, 0)
})
}
@@ -466,24 +527,24 @@ func TestWaitingClientsAreAskedAboutInOneRequest(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
- geojs, clock, g := start()
+ service, clock, g := start()
// GeoJS fails, and is then left alone for a second, while three more
// clients come. An IPv6 client is a /64, and GeoJS is asked about its
// first address.
- geojs.set(failing)
+ service.set(failing)
wantCountry(t, g, netip.MustParsePrefix("203.0.113.1/32"), "")
wantCountry(t, g, netip.MustParsePrefix("203.0.113.2/32"), "")
wantCountry(t, g, netip.MustParsePrefix("2001:db8:1:2::/64"), "")
- wantRequests(t, geojs, 1)
+ wantRequests(t, service, 1)
- geojs.set(answering)
+ service.set(answering)
clock.advance(time.Second)
wantCountry(t, g, netip.MustParsePrefix("203.0.113.3/32"), germany)
- wantRequests(t, geojs, 2)
+ wantRequests(t, service, 2)
wantAsked(
t,
- geojs,
+ service,
1,
"203.0.113.1",
"203.0.113.2",
@@ -499,15 +560,15 @@ func TestKeptAnswersUnaffectedWhileGeoJSFailsAndAskedAgainWithBackoff(
t.Parallel()
synctest.Test(t, func(t *testing.T) {
- geojs, clock, g := start()
+ service, clock, g := start()
clients := newClients()
kept := clients()
wantCountry(t, g, kept, germany)
- geojs.set(failing)
+ service.set(failing)
wantCountry(t, g, kept, germany)
- wantRequests(t, geojs, 1)
+ wantRequests(t, service, 1)
// Each failure leaves GeoJS alone twice as long as the one before, up
// to five minutes. New clients meanwhile count as not found, and the
@@ -523,22 +584,22 @@ func TestKeptAnswersUnaffectedWhileGeoJSFailsAndAskedAgainWithBackoff(
wantCountry(t, g, clients(), "")
requests++
- wantRequests(t, geojs, requests)
+ wantRequests(t, service, requests)
clock.advance(delay - time.Millisecond)
wantCountry(t, g, clients(), "")
wantCountry(t, g, kept, germany)
- wantRequests(t, geojs, requests)
+ wantRequests(t, service, requests)
clock.advance(time.Millisecond)
}
// Once GeoJS answers again, it is asked about every client waiting.
- geojs.set(answering)
+ service.set(answering)
wantCountry(t, g, clients(), germany)
- wantRequests(t, geojs, requests+1)
+ wantRequests(t, service, requests+1)
- asked := waitForRequests(t, geojs, requests+1)
+ asked := waitForRequests(t, service, requests+1)
if len(asked[requests]) != 23 {
t.Errorf(
"GeoJS was asked about %d clients, want 23",
@@ -552,24 +613,24 @@ func TestAtMost200AddressesInOneRequest(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
- geojs, clock, g := start()
+ service, clock, g := start()
clients := newClients()
first := clients()
// 201 clients wait while GeoJS is left alone after a failure.
- geojs.set(failing)
+ service.set(failing)
wantCountry(t, g, first, "")
for range 200 {
wantCountry(t, g, clients(), "")
}
- // The first one's next request has GeoJS asked again.
- geojs.set(answering)
+ // The first one's next lookup has GeoJS asked again.
+ service.set(answering)
clock.advance(time.Second)
wantCountry(t, g, first, "")
- asked := waitForRequests(t, geojs, 3)
+ asked := waitForRequests(t, service, 3)
if len(asked[1]) != 200 || len(asked[2]) != 1 {
t.Errorf(
"GeoJS was asked about %d and then %d clients, want 200 and 1",
@@ -584,13 +645,13 @@ func TestAtMost10000ClientsWait(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
- geojs, clock, g := start()
+ service, clock, g := start()
clients := newClients()
first := clients()
// 10,000 clients wait while GeoJS is left alone after a failure, and
// one more cannot join them.
- geojs.set(failing)
+ service.set(failing)
wantCountry(t, g, first, "")
for range 9999 {
@@ -600,13 +661,13 @@ func TestAtMost10000ClientsWait(t *testing.T) {
extra := clients()
wantCountry(t, g, extra, "")
- // The first one's next request has GeoJS asked about the 10,000, 200
+ // The first one's next lookup has GeoJS asked about the 10,000, 200
// at a time, and not about the one more.
- geojs.set(answering)
+ service.set(answering)
clock.advance(time.Second)
wantCountry(t, g, first, "")
- asked := waitForRequests(t, geojs, 51)
+ asked := waitForRequests(t, service, 51)
for i, request := range asked {
if slices.Contains(request, extra.Addr().String()) {
t.Errorf("request %d asked about %s", i, extra.Addr())
@@ -622,24 +683,15 @@ func TestClientsWithoutAnAnswerAreCounted(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
- m := metrics.New(1, "app")
- g := lookup.New(lookup.Params{
- URL: lookup.URL,
- Timeout: timeout,
- Wait: true,
- Now: time.Now,
- ProcessLog: slog.New(slog.DiscardHandler),
- Metrics: m,
- Alerts: alerts.New(alerts.Params{}),
- })
- g.SetTransport(&standIn{answers: failing})
-
+ service, _, g, reports := startWithReports()
clients := newClients()
+ service.set(failing)
+
// GeoJS fails, so the first client goes without an answer, and GeoJS
// is left alone for a second, which does not pass in this test.
wantCountry(t, g, clients(), "")
- wantRequestsWithoutGeoJSAnswer(t, m, 1)
+ reports.wantWithoutAnswer(t, 1)
// Meanwhile each new client goes without one at once, while there is
// room for it among the 10,000 that may wait.
@@ -647,11 +699,11 @@ func TestClientsWithoutAnAnswerAreCounted(t *testing.T) {
wantCountry(t, g, clients(), "")
}
- wantRequestsWithoutGeoJSAnswer(t, m, 10000)
+ reports.wantWithoutAnswer(t, 10000)
// One more, for which there is no room, goes without one too.
wantCountry(t, g, clients(), "")
- wantRequestsWithoutGeoJSAnswer(t, m, 10001)
+ reports.wantWithoutAnswer(t, 10001)
})
}
@@ -659,6 +711,7 @@ func TestClientsWithoutAnAnswerAreCounted(t *testing.T) {
const (
answering = iota
answeringSlowly // most of a second later
+ answeringLate // lateAnswer later
answeringInLowerCase // with each country in lower case
answeringWithoutLeftOut // with a list that leaves leftOut out
answeringEmptyList // with []
@@ -712,9 +765,9 @@ func (s *standIn) ServeHTTP(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "/", http.StatusFound)
return
- case answeringSlowly:
+ case answeringSlowly, answeringLate:
select {
- case <-time.After(timeout * 4 / 5):
+ case <-time.After(delay(answers)):
case <-r.Context().Done():
return
}
@@ -759,6 +812,16 @@ func (s *standIn) ServeHTTP(w http.ResponseWriter, r *http.Request) {
}
}
+// delay returns how long the stand-in takes to answer when answering
+// slowly or late.
+func delay(answers int) time.Duration {
+ if answers == answeringLate {
+ return lateAnswer
+ }
+
+ return wait * 4 / 5
+}
+
// set sets how the stand-in answers.
func (s *standIn) set(answers int) {
s.mu.Lock()
@@ -775,6 +838,46 @@ func (s *standIn) asked() [][]string {
return slices.Clone(s.requests)
}
+// reports are what a GeoJS reported through Failed and WentWithoutAnswer.
+type reports struct {
+ mu sync.Mutex
+ failures []failure
+ withoutAnswer int
+}
+
+// failure is one report of Failed: the error, and how long GeoJS is left
+// alone.
+type failure struct {
+ err string
+ askingAgainIn time.Duration
+}
+
+// wantFailures checks the failures reported so far.
+func (r *reports) wantFailures(t *testing.T, want ...failure) {
+ t.Helper()
+
+ r.mu.Lock()
+ defer r.mu.Unlock()
+
+ if !slices.Equal(r.failures, want) {
+ t.Errorf("failures %+v, want %+v", r.failures, want)
+ }
+}
+
+// wantWithoutAnswer checks how many times a caller has gone without an
+// answer.
+func (r *reports) wantWithoutAnswer(t *testing.T, want int) {
+ t.Helper()
+
+ r.mu.Lock()
+ defer r.mu.Unlock()
+
+ if r.withoutAnswer != want {
+ t.Errorf("%d lookups went without an answer, want %d",
+ r.withoutAnswer, want)
+ }
+}
+
// testClock is a clock the test sets. GeoJS tells the time by it, while
// waits run on the bubble's clock.
type testClock struct {
@@ -799,38 +902,47 @@ func (c *testClock) advance(d time.Duration) {
}
// start returns a stand-in for GeoJS that answers, a clock, and a GeoJS
-// asking the stand-in by that clock, for which a request waits for its
-// client's first answer.
-func start() (*standIn, *testClock, *lookup.GeoJS) {
- geojs, clock, g, _ := startWithAlerts()
+// asking the stand-in by that clock.
+func start() (*standIn, *testClock, *geojs.GeoJS) {
+ service, clock, g, _ := startWithReports()
- return geojs, clock, g
+ return service, clock, g
}
-// startWithAlerts is start, and returns the queue of the alerts GeoJS
-// raises as well, for a webhook that is never sent them, with the default
-// cooldown, by the same clock.
-func startWithAlerts() (*standIn, *testClock, *lookup.GeoJS, *alerts.Queue) {
- geojs := &standIn{}
+// startWithReports is start, and returns what the GeoJS reports as well.
+func startWithReports() (*standIn, *testClock, *geojs.GeoJS, *reports) {
+ service := &standIn{}
clock := newClock()
- queue := alerts.New(alerts.Params{
- WebhookURL: &url.URL{Scheme: "https", Host: "alerts.example"},
- Events: alerts.Events(),
- Cooldown: 15 * time.Minute,
- Now: clock.Now,
- })
- g := lookup.New(lookup.Params{
- URL: lookup.URL,
- Timeout: timeout,
- Wait: true,
- Now: clock.Now,
- ProcessLog: slog.New(slog.DiscardHandler),
- Metrics: metrics.New(1, "app"),
- Alerts: queue,
- })
- g.SetTransport(geojs)
+ g, reports := newGeoJS(service, geojs.Params{Now: clock.Now})
- return geojs, clock, g, queue
+ return service, clock, g, reports
+}
+
+// newGeoJS returns a GeoJS that asks service, made with params and with
+// what it reports noted in the reports returned.
+func newGeoJS(
+ service *standIn, params geojs.Params,
+) (*geojs.GeoJS, *reports) {
+ r := &reports{}
+ params.URL = geojs.URL
+ params.Asked = func() {}
+ params.Failed = func(err error, askingAgainIn time.Duration) {
+ r.mu.Lock()
+ defer r.mu.Unlock()
+
+ r.failures = append(r.failures, failure{err.Error(), askingAgainIn})
+ }
+ params.WentWithoutAnswer = func() {
+ r.mu.Lock()
+ defer r.mu.Unlock()
+
+ r.withoutAnswer++
+ }
+
+ g := geojs.New(params)
+ g.SetTransport(service)
+
+ return g, r
}
// newClock returns a clock set to the start of a day.
@@ -850,36 +962,37 @@ func newClients() func() netip.Prefix {
}
}
-// wantCountry checks the country g gives client.
+// wantCountry checks the country g gives client, waiting up to wait for
+// it.
func wantCountry(
t *testing.T,
- g *lookup.GeoJS,
+ g *geojs.GeoJS,
client netip.Prefix,
want string,
) {
t.Helper()
- got := g.LookUp(t.Context(), client).Country
+ got := g.LookUp(t.Context(), client, wait).Country
if got != want {
t.Errorf("%s is in %q, want %q", client, got, want)
}
}
// wantRequests checks how many requests GeoJS has had.
-func wantRequests(t *testing.T, geojs *standIn, want int) {
+func wantRequests(t *testing.T, service *standIn, want int) {
t.Helper()
- got := len(geojs.asked())
+ got := len(service.asked())
if got != want {
t.Errorf("GeoJS had %d requests, want %d", got, want)
}
}
// wantAsked checks the addresses request i asked about, in any order.
-func wantAsked(t *testing.T, geojs *standIn, i int, want ...string) {
+func wantAsked(t *testing.T, service *standIn, i int, want ...string) {
t.Helper()
- asked := geojs.asked()
+ asked := service.asked()
if len(asked) <= i {
t.Fatalf("GeoJS had %d requests, want more than %d", len(asked), i)
}
@@ -893,40 +1006,15 @@ func wantAsked(t *testing.T, geojs *standIn, i int, want ...string) {
}
}
-// wantRequestsWithoutGeoJSAnswer checks how many requests m counts as having
-// gone without an answer from GeoJS.
-func wantRequestsWithoutGeoJSAnswer(
- t *testing.T,
- m *metrics.Metrics,
- want float64,
-) {
- t.Helper()
-
- got := testutil.ToFloat64(m.RequestsWithoutGeoJSAnswer)
- if got != want {
- t.Errorf("%v requests went without an answer, want %v", got, want)
- }
-}
-
-// wantFailures checks how many requests to GeoJS m counts as failed.
-func wantFailures(t *testing.T, m *metrics.Metrics, want float64) {
- t.Helper()
-
- got := testutil.ToFloat64(m.GeoJSFailures)
- if got != want {
- t.Errorf("%v requests to GeoJS failed, want %v", got, want)
- }
-}
-
// waitForRequests waits until g has done all it can before time passes,
// checks that GeoJS has had count requests, and returns the addresses each
// asked about.
-func waitForRequests(t *testing.T, geojs *standIn, count int) [][]string {
+func waitForRequests(t *testing.T, service *standIn, count int) [][]string {
t.Helper()
synctest.Wait()
- asked := geojs.asked()
+ asked := service.asked()
if len(asked) != count {
t.Fatalf("GeoJS had %d requests, want %d", len(asked), count)
}
@@ -936,18 +1024,18 @@ func waitForRequests(t *testing.T, geojs *standIn, count int) [][]string {
// waitForCountry lets a request to GeoJS under way be abandoned, and moves
// the clock on a minute, so that GeoJS may be asked again after a failure.
-// It then checks that client's next request does not wait but has it asked
+// It then checks that client's next lookup does not wait but has it asked
// about again in the background, after which g gives it the country want.
func waitForCountry(
t *testing.T,
- g *lookup.GeoJS,
+ g *geojs.GeoJS,
clock *testClock,
client netip.Prefix,
want string,
) {
t.Helper()
- time.Sleep(timeout)
+ time.Sleep(requestTimeout)
clock.advance(time.Minute)
wantCountry(t, g, client, "")
synctest.Wait()
diff --git a/internal/lookup/snapshot_test.go b/internal/geojs/snapshot_test.go
similarity index 85%
rename from internal/lookup/snapshot_test.go
rename to internal/geojs/snapshot_test.go
index af854a3..27c71bc 100644
--- a/internal/lookup/snapshot_test.go
+++ b/internal/geojs/snapshot_test.go
@@ -1,4 +1,4 @@
-package lookup_test
+package geojs_test
import (
"net/netip"
@@ -7,7 +7,7 @@ import (
"testing/synctest"
"time"
- "sneak.berlin/go/smallwebwaf/internal/lookup"
+ "sneak.berlin/go/smallwebwaf/internal/geojs"
)
func TestSnapshotHoldsEachAnswerAndWhenItWasLastUsed(t *testing.T) {
@@ -25,7 +25,7 @@ func TestSnapshotHoldsEachAnswerAndWhenItWasLastUsed(t *testing.T) {
clock.advance(time.Hour)
wantCountry(t, g, placed, germany)
- want := []lookup.Answer{
+ want := []geojs.Answer{
{Client: notPlaced, Answered: asked, Used: asked},
{
Client: placed, ASN: asn, ASName: asName, Country: germany,
@@ -41,24 +41,24 @@ func TestSnapshotHoldsEachAnswerAndWhenItWasLastUsed(t *testing.T) {
func TestLoadedAnswersAreKeptFor7DaysFromWhenGeoJSGaveThem(t *testing.T) {
t.Parallel()
- geojs, clock, g := start()
+ service, clock, g := start()
now := clock.Now()
- kept := lookup.Answer{
+ kept := geojs.Answer{
Client: netip.MustParsePrefix("203.0.113.9/32"),
Country: "FR",
Answered: now.Add(-week + time.Second),
Used: now.Add(-time.Hour),
}
- stale := lookup.Answer{
+ stale := geojs.Answer{
Client: netip.MustParsePrefix("203.0.113.10/32"),
Country: "FR",
Answered: now.Add(-week),
Used: now.Add(-time.Hour),
}
- g.Load([]lookup.Answer{kept, stale})
+ g.Load([]geojs.Answer{kept, stale})
- if got := g.Snapshot(); !slices.Equal(got, []lookup.Answer{kept}) {
+ if got := g.Snapshot(); !slices.Equal(got, []geojs.Answer{kept}) {
t.Errorf(
"kept %+v, want only the answer GeoJS gave less than 7 days ago",
got,
@@ -66,7 +66,7 @@ func TestLoadedAnswersAreKeptFor7DaysFromWhenGeoJSGaveThem(t *testing.T) {
}
wantCountry(t, g, kept.Client, "FR")
- wantRequests(t, geojs, 0)
+ wantRequests(t, service, 0)
}
func TestLoadDropsTheAnswerUsedLongestAgoFirst(t *testing.T) {
@@ -80,11 +80,11 @@ func TestLoadDropsTheAnswerUsedLongestAgoFirst(t *testing.T) {
// lookups.json lists the answers by client. Here each was last used a
// second before the one listed before it, so the last listed is the
// one used longest ago, and the one dropped.
- answers := make([]lookup.Answer, maxAnswers+1)
+ answers := make([]geojs.Answer, maxAnswers+1)
addr := netip.MustParseAddr("10.0.0.0")
for i := range answers {
- answers[i] = lookup.Answer{
+ answers[i] = geojs.Answer{
Client: netip.PrefixFrom(addr, addr.BitLen()),
Country: germany,
Answered: now,
diff --git a/internal/lookup/file.go b/internal/lookup/file.go
index 74292d1..1c028f5 100644
--- a/internal/lookup/file.go
+++ b/internal/lookup/file.go
@@ -1,3 +1,6 @@
+// Package lookup looks up each client's AS number and country in the lookup
+// database, the IPinfo Lite file SWW_WAF_LOOKUP_DB_PATH names. Its answers
+// take the form of GeoJS's, so that waf handles both alike.
package lookup
import (
@@ -14,6 +17,7 @@ import (
"github.com/oschwald/maxminddb-golang/v2"
"sneak.berlin/go/smallwebwaf/internal/alerts"
+ "sneak.berlin/go/smallwebwaf/internal/geojs"
)
// quietTime is how long the lookup database must go without a change
@@ -77,7 +81,7 @@ func OpenFile(params FileParams) (*File, error) {
// each "" when the database does not give it, as for an address missing
// from it. The database is asked about the client's first address, as
// GeoJS is.
-func (f *File) LookUp(client netip.Prefix) Answer {
+func (f *File) LookUp(client netip.Prefix) geojs.Answer {
f.mu.Lock()
reader := f.reader
f.mu.Unlock()
@@ -91,7 +95,7 @@ func (f *File) LookUp(client netip.Prefix) Answer {
found = record{}
}
- return Answer{
+ return geojs.Answer{
Client: client,
ASN: found.ASN,
ASName: found.ASName,
diff --git a/internal/lookup/file_internal_test.go b/internal/lookup/file_internal_test.go
index c51c3bd..a49c68b 100644
--- a/internal/lookup/file_internal_test.go
+++ b/internal/lookup/file_internal_test.go
@@ -16,6 +16,7 @@ import (
"github.com/maxmind/mmdbwriter/mmdbtype"
"sneak.berlin/go/smallwebwaf/internal/alerts"
+ "sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
)
@@ -68,7 +69,7 @@ func TestFilePlacesClientsAndCountsAnAddressMissingFromItAsUnknown(
prefix := netip.MustParsePrefix(client)
got := f.LookUp(prefix)
- if got != (Answer{
+ if got != (geojs.Answer{
Client: prefix, ASN: want.ASN, ASName: want.ASName, Country: want.Country,
Answered: now,
}) {
diff --git a/internal/metrics/metrics.go b/internal/metrics/metrics.go
index 24e92f3..2f787e6 100644
--- a/internal/metrics/metrics.go
+++ b/internal/metrics/metrics.go
@@ -274,8 +274,7 @@ func (m *Metrics) AddRemoteLog(remote *remotelog.Sender) {
// AddLookupFile adds the metrics of the lookup database, read as the
// metrics are asked for: when the file in use was read, which lastRead
// returns, and the replacements of it that could not be read, which
-// readFailures returns. The lookup package's File, which has both, cannot
-// be named here: that package counts GeoJS's requests in these metrics.
+// readFailures returns.
func (m *Metrics) AddLookupFile(
lastRead func() time.Time,
readFailures func() int,
diff --git a/internal/remotelog/remotelog_test.go b/internal/remotelog/remotelog_test.go
index ffa8862..91ebacc 100644
--- a/internal/remotelog/remotelog_test.go
+++ b/internal/remotelog/remotelog_test.go
@@ -44,7 +44,7 @@ import (
const (
// started is the time a record made as a test starts gives.
started = "2000-01-01T00:00:00.000Z"
- appName = "fsn1app1/blog"
+ appName = "host1/blog"
// local0 is the number of the default facility, and local0Info the
// priority of its records.
local0 = 16
diff --git a/internal/reputation/dnsbl_test.go b/internal/reputation/dnsbl_test.go
index 0be1a3d..dfe1e74 100644
--- a/internal/reputation/dnsbl_test.go
+++ b/internal/reputation/dnsbl_test.go
@@ -31,10 +31,12 @@ import (
// this package: Go's resolver counts the queries under way in one
// sync.WaitGroup for the whole process, and the process fails when
// queries from two bubbles, or from a bubble and from outside one, are
-// under way at once. TestMain has the resolver make its configuration,
-// which it makes on its first query, outside every bubble, since the
-// configuration holds a channel, which the bubble it was made in would
-// keep to itself.
+// under way at once. The one test outside a bubble is parallel, so it
+// starts only once every test that is not parallel has ended; no other
+// parallel test of this package queries a resolver. TestMain has the
+// resolver make its configuration, which it makes on its first query,
+// outside every bubble, since the configuration holds a channel, which
+// the bubble it was made in would keep to itself.
const (
// zone and otherZone are the DNSBL zones the tests name.
@@ -525,8 +527,9 @@ func TestAtMost100000VerdictsKeptTheOneFetchedLongestAgoDroppedFirst(
}
}
-//nolint:paralleltest // one at a time, as the comment at the top of this file says
func TestQueriesGoToTheResolverSWWDNSBLResolverNames(t *testing.T) {
+ t.Parallel()
+
resolver := &resolverStandIn{answers: map[string]answer{
listedName: {addrs: []string{listing}},
}}
diff --git a/internal/requestlog/requestlog.go b/internal/requestlog/requestlog.go
index 7ebb8b7..c432c17 100644
--- a/internal/requestlog/requestlog.go
+++ b/internal/requestlog/requestlog.go
@@ -61,6 +61,12 @@ const (
// ActionCountryDenied is a request refused because its client's country
// is not in SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES.
ActionCountryDenied = "country_denied"
+
+ // ActionChecksTimedOut is a request refused because the trap paths, the
+ // rule files and the Core Rule Set had not finished with it within 500
+ // milliseconds.
+ ActionChecksTimedOut = "checks_timed_out"
+
// ActionOwnEndpoint is a request waf answered at one of its own
// endpoints, or a question about a path under /_sww/.
ActionOwnEndpoint = "own_endpoint"
@@ -149,7 +155,8 @@ type Line struct {
// ActionTemporarilyBanned, ActionPermanentlyBanned, ActionCountryDenied,
// ActionClientIsRateLimited, ActionBlockedByBlockRule,
// ActionBlockedByCoreRuleSet, or, in waf, ActionRequestLineTooLong or
- // ActionRequestHeaderTooLarge for the header limits.
+ // ActionRequestHeaderTooLarge for the header limits, or
+ // ActionChecksTimedOut.
WouldAction string `json:"would_action,omitempty"`
// Heuristic is the name of the heuristic that refused the request, in
// observe mode too, and HeuristicFigures are the figures it acted on, by
diff --git a/internal/requestlog/requestlog_test.go b/internal/requestlog/requestlog_test.go
index 11194d7..b14faac 100644
--- a/internal/requestlog/requestlog_test.go
+++ b/internal/requestlog/requestlog_test.go
@@ -140,7 +140,7 @@ func TestProcessLinesAreMarkedProcessAndGiveTheInstance(t *testing.T) {
var out bytes.Buffer
- requestlog.NewProcessLogger(&out, "fsn1app1/blog", slog.LevelInfo).
+ requestlog.NewProcessLogger(&out, "host1/blog", slog.LevelInfo).
Info("starting",
"version", "v1")
@@ -152,7 +152,7 @@ func TestProcessLinesAreMarkedProcessAndGiveTheInstance(t *testing.T) {
}
if fields["type"] != "process" ||
- fields["instance"] != "fsn1app1/blog" ||
+ fields["instance"] != "host1/blog" ||
fields["msg"] != "starting" ||
fields["level"] != "INFO" ||
fields["version"] != "v1" {
@@ -187,7 +187,7 @@ func TestProcessLoggerWritesTheMessagesAtItsLevelOrMoreSevere(t *testing.T) {
processLog := requestlog.NewProcessLogger(
&out,
- "fsn1app1/blog",
+ "host1/blog",
level,
)
for _, at := range levels {
diff --git a/internal/rproxy/breaker.go b/internal/rproxy/breaker.go
new file mode 100644
index 0000000..21e6cdb
--- /dev/null
+++ b/internal/rproxy/breaker.go
@@ -0,0 +1,82 @@
+package rproxy
+
+import (
+ "errors"
+ "sync"
+ "time"
+)
+
+// What opens the circuit breaker, and how often a request asks waf while it is
+// open.
+const (
+ breakerFailures = 5
+ breakerWindow = 5 * time.Second
+ breakerAskInterval = time.Second
+)
+
+// errBreakerOpen is why a request did not ask waf.
+var errBreakerOpen = errors.New(
+ "waf is not asked while the circuit breaker is open",
+)
+
+// breaker is the circuit breaker for asks to waf. It opens when
+// breakerFailures asks fail within breakerWindow. An ask fails when it gets no
+// answer in time, or an answer rproxy cannot pass on. While it is open, at
+// most one request each breakerAskInterval asks waf. Any usable answer closes
+// it, whatever it decides, and forgets the failures before it.
+type breaker struct {
+ mu sync.Mutex
+ // failures holds the times of the latest failed asks, oldest first, at most
+ // breakerFailures of them.
+ failures []time.Time
+ open bool
+ // lastAsk is when a request last asked waf while the breaker was open.
+ lastAsk time.Time
+}
+
+// mayAsk reports whether a request may ask waf at now. While the breaker is
+// open, it says yes once each breakerAskInterval.
+func (b *breaker) mayAsk(now time.Time) bool {
+ b.mu.Lock()
+ defer b.mu.Unlock()
+
+ if !b.open {
+ return true
+ }
+
+ if now.Sub(b.lastAsk) < breakerAskInterval {
+ return false
+ }
+
+ b.lastAsk = now
+
+ return true
+}
+
+// noteAnswer notes that an ask got an answer rproxy can use. It closes the
+// breaker and forgets the failures before it.
+func (b *breaker) noteAnswer() {
+ b.mu.Lock()
+ defer b.mu.Unlock()
+
+ b.open = false
+ b.failures = nil
+}
+
+// noteFailure notes that an ask that ended at now got no answer in time, and
+// opens the breaker if breakerFailures asks have failed within breakerWindow.
+func (b *breaker) noteFailure(now time.Time) {
+ b.mu.Lock()
+ defer b.mu.Unlock()
+
+ b.failures = append(b.failures, now)
+ if len(b.failures) > breakerFailures {
+ b.failures = b.failures[1:]
+ }
+
+ if !b.open && len(b.failures) == breakerFailures &&
+ now.Sub(b.failures[0]) <= breakerWindow {
+ b.open = true
+ b.lastAsk = now
+ }
+}
diff --git a/internal/rproxy/breaker_test.go b/internal/rproxy/breaker_test.go
new file mode 100644
index 0000000..ef0ead3
--- /dev/null
+++ b/internal/rproxy/breaker_test.go
@@ -0,0 +1,142 @@
+package rproxy_test
+
+import (
+ "testing"
+ "time"
+
+ "sneak.berlin/go/smallwebwaf/internal/rproxy"
+)
+
+// The breaker tests give the breaker each time themselves, counted from
+// time.Now(), and never wait.
+
+// fail notes count failed asks on b, one each step from at.
+func fail(b *rproxy.Breaker, at time.Time, count int, step time.Duration) {
+ for i := range count {
+ b.NoteFailure(at.Add(time.Duration(i) * step))
+ }
+}
+
+// openBreaker returns a breaker opened at now by five failed asks, one a
+// second.
+func openBreaker(now time.Time) *rproxy.Breaker {
+ b := &rproxy.Breaker{}
+ fail(b, now.Add(-4*time.Second), 5, time.Second)
+
+ return b
+}
+
+func TestFourFailedAsksWithinFiveSecondsLeaveTheBreakerClosed(t *testing.T) {
+ t.Parallel()
+
+ now := time.Now()
+ b := &rproxy.Breaker{}
+ fail(b, now, 4, time.Second)
+
+ if !b.MayAsk(now.Add(3 * time.Second)) {
+ t.Error("a request may not ask, want it to")
+ }
+}
+
+func TestFiveFailedAsksWithinFiveSecondsOpenTheBreaker(t *testing.T) {
+ t.Parallel()
+
+ now := time.Now()
+ b := openBreaker(now)
+
+ if b.MayAsk(now) {
+ t.Error("a request may ask, want it not to")
+ }
+}
+
+func TestFiveFailedAsksOverMoreThanFiveSecondsLeaveTheBreakerClosed(
+ t *testing.T,
+) {
+ t.Parallel()
+
+ now := time.Now()
+ b := &rproxy.Breaker{}
+ fail(b, now, 5, 1300*time.Millisecond)
+
+ if !b.MayAsk(now.Add(6 * time.Second)) {
+ t.Error("a request may not ask, want it to")
+ }
+}
+
+func TestAnAnswerBetweenFailedAsksKeepsTheBreakerClosed(t *testing.T) {
+ t.Parallel()
+
+ now := time.Now()
+ b := &rproxy.Breaker{}
+ fail(b, now, 4, 100*time.Millisecond)
+ b.NoteAnswer()
+ fail(b, now.Add(500*time.Millisecond), 4, 100*time.Millisecond)
+
+ if !b.MayAsk(now.Add(900 * time.Millisecond)) {
+ t.Error("a request may not ask, want it to")
+ }
+}
+
+func TestAnOpenBreakerLetsOneRequestAskEachSecond(t *testing.T) {
+ t.Parallel()
+
+ now := time.Now()
+ b := openBreaker(now)
+
+ for _, tc := range []struct {
+ after time.Duration
+ want bool
+ }{
+ {999 * time.Millisecond, false},
+ {time.Second, true},
+ {time.Second, false},
+ {1500 * time.Millisecond, false},
+ {2 * time.Second, true},
+ } {
+ got := b.MayAsk(now.Add(tc.after))
+ if got != tc.want {
+ t.Errorf("may ask %v after it opened: %v, want %v",
+ tc.after, got, tc.want)
+ }
+ }
+}
+
+func TestAFailedAskKeepsTheBreakerOpen(t *testing.T) {
+ t.Parallel()
+
+ opened := time.Now()
+ b := openBreaker(opened)
+ now := opened.Add(time.Second)
+ b.MayAsk(now)
+ b.NoteFailure(now)
+
+ if b.MayAsk(now.Add(999 * time.Millisecond)) {
+ t.Error("a request may ask, want it not to")
+ }
+
+ if !b.MayAsk(now.Add(time.Second)) {
+ t.Error("a request may not ask a second later, want it to")
+ }
+}
+
+func TestOneAnswerClosesTheBreaker(t *testing.T) {
+ t.Parallel()
+
+ opened := time.Now()
+ b := openBreaker(opened)
+ now := opened.Add(time.Second)
+ b.MayAsk(now)
+ b.NoteAnswer()
+
+ if !b.MayAsk(now) {
+ t.Error("a request may not ask, want it to")
+ }
+
+ // The failures before the answer no longer count, so one more does not open
+ // the breaker again.
+ b.NoteFailure(now)
+
+ if !b.MayAsk(now) {
+ t.Error("a request may not ask after one failure, want it to")
+ }
+}
diff --git a/internal/rproxy/export_test.go b/internal/rproxy/export_test.go
new file mode 100644
index 0000000..975298b
--- /dev/null
+++ b/internal/rproxy/export_test.go
@@ -0,0 +1,23 @@
+package rproxy
+
+import "time"
+
+// Breaker is the circuit breaker, for tests that give it the time.
+type Breaker struct {
+ b breaker
+}
+
+// MayAsk reports whether a request may ask waf at now.
+func (b *Breaker) MayAsk(now time.Time) bool {
+ return b.b.mayAsk(now)
+}
+
+// NoteAnswer notes that an ask got an answer rproxy can use.
+func (b *Breaker) NoteAnswer() {
+ b.b.noteAnswer()
+}
+
+// NoteFailure notes that an ask that ended at now got no answer in time.
+func (b *Breaker) NoteFailure(now time.Time) {
+ b.b.noteFailure(now)
+}
diff --git a/internal/rproxy/rproxy.go b/internal/rproxy/rproxy.go
index de545b6..489c4c6 100644
--- a/internal/rproxy/rproxy.go
+++ b/internal/rproxy/rproxy.go
@@ -4,10 +4,11 @@
// waf goes to the visitor as waf gave it.
// SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION decides only when waf cannot
// be connected to, does not answer at all, or does not answer within
-// SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS. rproxy itself refuses a request with a
-// method it does not pass on, or one it cannot put to waf as the visitor
-// sent it. It keeps nothing between requests and writes nothing to disk. "rproxy" in
-// SPEC.md says what it does.
+// SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS, or while the circuit breaker for asks to
+// waf is open. rproxy itself refuses a request with a method it does not pass
+// on, or one it cannot put to waf as the visitor sent it. It keeps nothing
+// between requests but the circuit breaker, and writes nothing to disk.
+// "rproxy" in SPEC.md says what it does.
package rproxy
import (
@@ -132,6 +133,8 @@ type handler struct {
wafTransport http.RoundTripper
// reportURL is where the reports go.
reportURL string
+ // breaker is the circuit breaker for asks to waf.
+ breaker breaker
// reports waits for the reports being sent. reportsMu guards starting one
// against reportsStopped, which is set once nothing is to wait for another.
reports sync.WaitGroup
@@ -196,7 +199,7 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
return
}
- answer, err := rq.ask(r.Context())
+ answer, err := rq.askUnlessBreakerOpen(r.Context())
if err != nil {
rq.handleNoUsableAnswer(r.Context(), err)
diff --git a/internal/rproxy/rproxy_test.go b/internal/rproxy/rproxy_test.go
index b68f733..d45e5f6 100644
--- a/internal/rproxy/rproxy_test.go
+++ b/internal/rproxy/rproxy_test.go
@@ -15,6 +15,7 @@ import (
"strconv"
"strings"
"sync"
+ "sync/atomic"
"testing"
"time"
@@ -1395,6 +1396,163 @@ func TestSendsNoReportWithoutAnAnswer(t *testing.T) {
standIn.wantNoReport(t)
}
+func TestFiveFailedAsksKeepTheNextRequestFromAskingWAF(t *testing.T) {
+ t.Parallel()
+
+ for name, failure := range map[string]http.HandlerFunc{
+ "no answer at all": func(http.ResponseWriter, *http.Request) {
+ panic(http.ErrAbortHandler)
+ },
+ "a slow answer": func(_ http.ResponseWriter, r *http.Request) {
+ <-r.Context().Done()
+ },
+ } {
+ t.Run(name, func(t *testing.T) {
+ t.Parallel()
+
+ var answering atomic.Bool
+
+ standIn := startWAF(
+ t,
+ func(w http.ResponseWriter, r *http.Request) {
+ if answering.Load() {
+ allow(w, r)
+
+ return
+ }
+
+ failure(w, r)
+ },
+ )
+ nextID := make(chan string, 1)
+ app := startApp(t, func(w http.ResponseWriter, r *http.Request) {
+ if answering.Load() {
+ nextID <- r.Header.Get(requestIDHeader)
+ }
+
+ greet(w, r)
+ })
+ addr, out := startRProxy(
+ t,
+ standIn.URL,
+ app.URL,
+ map[string]string{
+ wafBackendTimeout: "200",
+ },
+ )
+
+ // An ask can time out before it reaches the stand-in, so these
+ // may leave fewer than five questions, or leave some late.
+ for range 5 {
+ wantGreeting(t, get(t, addr, "/"))
+ }
+
+ // rproxy writes this line before it answers a request it kept
+ // from asking waf, so no wait is needed.
+ if out.has("the circuit breaker is open") {
+ t.Error("the circuit breaker opened before five asks failed")
+ }
+
+ answering.Store(true)
+ wantGreeting(t, get(t, addr, "/"))
+ out.waitFor(t, "the circuit breaker is open")
+
+ // Had the next request asked waf, its question would have reached
+ // the stand-in before the request reached the app.
+ var id string
+ select {
+ case id = <-nextID:
+ default:
+ t.Fatal("the app did not get the next request")
+ }
+
+ for len(standIn.questions) > 0 {
+ asked := <-standIn.questions
+ if asked.header.Get(requestIDHeader) == id {
+ t.Error("waf was asked, want it not to be")
+ }
+ }
+ })
+ }
+}
+
+func TestVisitorsThatGoAwayWhileWAFIsAskedLeaveTheBreakerClosed(
+ t *testing.T,
+) {
+ t.Parallel()
+
+ var answering atomic.Bool
+
+ standIn := startWAF(t, func(w http.ResponseWriter, r *http.Request) {
+ if answering.Load() {
+ allow(w, r)
+
+ return
+ }
+
+ <-r.Context().Done()
+ })
+ app := startApp(t, greet)
+ addr, out := startRProxy(t, standIn.URL, app.URL, map[string]string{
+ wafBackendTimeout: "60000",
+ })
+
+ // Five of these would open the breaker if they counted as failures.
+ for range 5 {
+ conn := dial(t, addr)
+ send(t, conn, "GET / HTTP/1.1\r\nHost: site.example\r\n\r\n")
+ id := standIn.question(t).header.Get(requestIDHeader)
+
+ _ = conn.Close()
+
+ // rproxy writes the request's log line once it is done with the ask.
+ out.waitFor(t, id)
+ }
+
+ answering.Store(true)
+ wantGreeting(t, get(t, addr, "/"))
+ standIn.question(t)
+}
+
+func TestAnswersThatCannotBePassedOnOpenTheBreaker(t *testing.T) {
+ t.Parallel()
+
+ var answering atomic.Bool
+
+ tooLarge := overOneMiB(dispositionDeny)
+ standIn := startWAF(t, func(w http.ResponseWriter, r *http.Request) {
+ if answering.Load() {
+ allow(w, r)
+
+ return
+ }
+
+ tooLarge(w, r)
+ })
+ app := startApp(t, greet)
+ addr, out := startRProxy(t, standIn.URL, app.URL, map[string]string{
+ wafBackendTimeout: "60000",
+ })
+
+ for range 5 {
+ if got := get(t, addr, "/"); got.status != http.StatusBadGateway {
+ t.Errorf("answered %d, want 502", got.status)
+ }
+
+ standIn.question(t)
+ }
+
+ answering.Store(true)
+ wantGreeting(t, get(t, addr, "/"))
+ out.waitFor(t, "the circuit breaker is open")
+
+ select {
+ case <-standIn.questions:
+ t.Error("waf was asked, want it not to be")
+ case <-time.After(10 * pollInterval):
+ }
+}
+
func TestAPathUnderOwnPrefixNeverReachesTheApp(t *testing.T) {
t.Parallel()
diff --git a/internal/rproxy/waf.go b/internal/rproxy/waf.go
index e5f298a..630329b 100644
--- a/internal/rproxy/waf.go
+++ b/internal/rproxy/waf.go
@@ -10,6 +10,7 @@ import (
"maps"
"net/http"
"strings"
+ "time"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
@@ -67,7 +68,8 @@ func (rq *request) handleNoUsableAnswer(ctx context.Context, err error) {
rq.answer(http.StatusBadGateway)
default:
// waf cannot be connected to, does not answer at all, or does not answer
- // within SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS. A request that keeps waf
+ // within SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS; or the circuit breaker
+ // kept the request from asking. A request that keeps waf
// busy past SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS, like any request while
// waf is overloaded, gets SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION
// here. deny closes this, at the cost of answering 503 whenever waf is
@@ -86,6 +88,31 @@ func (rq *request) handleNoUsableAnswer(ctx context.Context, err error) {
}
}
+// askUnlessBreakerOpen asks waf, unless the circuit breaker is open and it is
+// not yet time for a request to ask. It then tells the breaker whether waf
+// answered in time. An answer rproxy cannot pass on counts as a failure. A
+// visitor that goes away tells the breaker nothing. ctx is the request's own
+// context.
+func (rq *request) askUnlessBreakerOpen(
+ ctx context.Context,
+) (*wafAnswer, error) {
+ b := &rq.h.breaker
+ if !b.mayAsk(time.Now()) {
+ return nil, errBreakerOpen
+ }
+
+ answer, err := rq.ask(ctx)
+
+ switch {
+ case err == nil:
+ b.noteAnswer()
+ case ctx.Err() == nil:
+ b.noteFailure(time.Now())
+ }
+
+ return answer, err
+}
+
// ask sends waf the question, and returns its answer, which must have come
// whole within SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS. It ends when ctx does, as
// when the visitor goes away.
diff --git a/internal/state/state.go b/internal/state/state.go
index 9e9d4f4..9a04cc9 100644
--- a/internal/state/state.go
+++ b/internal/state/state.go
@@ -46,7 +46,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/anomaly"
"sneak.berlin/go/smallwebwaf/internal/bans"
- "sneak.berlin/go/smallwebwaf/internal/lookup"
+ "sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation"
@@ -135,7 +135,7 @@ type Params struct {
// sww runs.
Ledger *bans.Ledger
Limiter *ratelimit.Limiter
- GeoJS *lookup.GeoJS
+ GeoJS *geojs.GeoJS
Lists *reputation.Lists
DNSBL *reputation.DNSBL
AbuseIPDB *reputation.AbuseIPDB
@@ -195,8 +195,8 @@ type clientsFile struct {
// lookupsFile is lookups.json, with each answer on a line of its own.
type lookupsFile struct {
- Version int `json:"version"`
- Lookups []lookup.Answer `json:"lookups"`
+ Version int `json:"version"`
+ Lookups []geojs.Answer `json:"lookups"`
}
// abuseIPDBChecksSpentFile is abuseipdb-checks-spent.json, indented for an
diff --git a/internal/state/state_test.go b/internal/state/state_test.go
index af3c840..2ee13d9 100644
--- a/internal/state/state_test.go
+++ b/internal/state/state_test.go
@@ -26,7 +26,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/anomaly"
"sneak.berlin/go/smallwebwaf/internal/bans"
- "sneak.berlin/go/smallwebwaf/internal/lookup"
+ "sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation"
@@ -159,7 +159,7 @@ const filledAlertsJSON = `{
"waiting": {
"webhook": [
{
- "instance": "fsn1app1/blog",
+ "instance": "host1/blog",
"time": "2026-10-06T00:00:00Z",
"event": "ban",
"client": "203.0.113.9",
@@ -174,7 +174,7 @@ const filledAlertsJSON = `{
"suppressed_repeats": 0
},
{
- "instance": "fsn1app1/blog",
+ "instance": "host1/blog",
"time": "2026-10-06T00:00:00Z",
"event": "file_error",
"client": "",
@@ -1700,7 +1700,7 @@ func TestEditOfEachFileTakenIn(t *testing.T) {
t,
lookupsJSON,
params.GeoJS.Snapshot(),
- []lookup.Answer{
+ []geojs.Answer{
{Client: client, Country: "FR", Answered: midnight()},
},
)
@@ -2398,7 +2398,7 @@ func newParams(dir string) state.Params {
Events: alerts.Events(),
Cooldown: 15 * time.Minute,
MaxPerHour: 2,
- Instance: "fsn1app1/blog",
+ Instance: "host1/blog",
Now: midnight,
})
@@ -2415,9 +2415,7 @@ func newParams(dir string) state.Params {
MaxBans: 5000,
}),
Limiter: ratelimit.New(ratelimit.Limits{}, 20000),
- GeoJS: lookup.New(lookup.Params{
- Now: midnight, ProcessLog: discard, Metrics: m,
- }),
+ GeoJS: geojs.New(geojs.Params{Now: midnight}),
Lists: reputation.New(reputation.Params{
BlocklistURLs: []string{
blocklistURL,
@@ -2494,7 +2492,7 @@ func fill(params state.Params) {
})
params.Limiter.AddLookup(client, now.Add(-time.Hour), asn, asName, "DE")
- params.GeoJS.Load([]lookup.Answer{
+ params.GeoJS.Load([]geojs.Answer{
{
Client: client, ASN: asn, ASName: asName, Country: "DE",
Answered: now.Add(-time.Hour), Used: now,
diff --git a/internal/sww/rproxy_test.go b/internal/sww/rproxy_test.go
index 934d92f..267704e 100644
--- a/internal/sww/rproxy_test.go
+++ b/internal/sww/rproxy_test.go
@@ -260,7 +260,17 @@ func TestTheSiteAnswersWhileWAFIsStoppedAndStartedAgain(t *testing.T) {
stop()
}()
- wantStatus(t, site+"/", placed, http.StatusForbidden)
+ // The visitor's asks have opened rproxy's circuit breaker, which lets one
+ // request a second ask until one is answered.
+ deadline := time.Now().Add(waitLimit)
+ for statusFrom(t, site+"/", placed) != http.StatusForbidden {
+ if time.Now().After(deadline) {
+ t.Fatalf("request from %s: never answered %d", placed,
+ http.StatusForbidden)
+ }
+
+ time.Sleep(pollInterval)
+ }
close(visitorStop)
<-visitorDone
diff --git a/internal/sww/sww.go b/internal/sww/sww.go
index 3da2b60..7befcde 100644
--- a/internal/sww/sww.go
+++ b/internal/sww/sww.go
@@ -25,6 +25,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
+ "sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/remotelog"
"sneak.berlin/go/smallwebwaf/internal/reputation"
@@ -270,7 +271,7 @@ func newServer(
Config: cfg,
RequestLog: stdout,
ProcessLog: processLog,
- GeoJSURL: lookup.URL,
+ GeoJSURL: geojs.URL,
AbuseIPDBURL: reputation.AbuseIPDBURL,
LookupFile: lookupFile,
CoreRuleSet: crs,
diff --git a/internal/sww/sww_test.go b/internal/sww/sww_test.go
index 42c4a52..4cfa9a7 100644
--- a/internal/sww/sww_test.go
+++ b/internal/sww/sww_test.go
@@ -52,7 +52,7 @@ const (
adminSecret = "fedcba9876543210fedcba9876543210"
// instance is the SWW_WAF_INSTANCE_NAME the tests set where they look at
// it.
- instance = "fsn1app1/blog"
+ instance = "host1/blog"
// greeting is what the tests' app answers.
greeting = "hello from the app"
// placed is the client the tests' lookup databases place.
@@ -386,8 +386,8 @@ func TestEveryLogLineAndMetricCarriesTheInstanceName(t *testing.T) {
// the comments.
for line := range strings.Lines(metrics) {
if !strings.HasPrefix(line, "#") &&
- !strings.Contains(line, `instance="fsn1app1/blog"`) {
- t.Errorf("series %q, without instance=\"fsn1app1/blog\"", line)
+ !strings.Contains(line, `instance="host1/blog"`) {
+ t.Errorf("series %q, without instance=\"host1/blog\"", line)
}
}
}
@@ -734,12 +734,12 @@ func TestLookupDatabaseReplacedWhileRunningTakesEffect(t *testing.T) {
t,
url+"_sww/metrics",
token,
- `sww_lookup_database_read_failures_total{instance="fsn1app1/blog"} 1`,
+ `sww_lookup_database_read_failures_total{instance="host1/blog"} 1`,
)
lastRead = seriesValue(
t,
metrics,
- `sww_lookup_database_last_read_timestamp_seconds{instance="fsn1app1/blog"}`,
+ `sww_lookup_database_last_read_timestamp_seconds{instance="host1/blog"}`,
)
wantStatus(t, url, placed, http.StatusOK)
@@ -801,7 +801,7 @@ func TestBlocklistTriesAndCopiesKeptInListsJSONAcrossRestarts(t *testing.T) {
// too.
rateLimitExemptNets: placed + "," + localhost,
}
- failures := `sww_reputation_failures_total{instance="fsn1app1/blog",` +
+ failures := `sww_reputation_failures_total{instance="host1/blog",` +
`source="` + torURL + `"} `
// tor.txt cannot be fetched at first, which is counted.
@@ -984,8 +984,8 @@ func TestStalledRemoteLogEndpointHoldsUpNoRequest(t *testing.T) {
// last.
metrics := metricsText(t, url+"_sww/metrics", token)
for _, series := range []string{
- `sww_remote_log_lines_sent_total{instance="fsn1app1/blog"} 0`,
- `sww_remote_log_lines_waiting{instance="fsn1app1/blog"} 1`,
+ `sww_remote_log_lines_sent_total{instance="host1/blog"} 0`,
+ `sww_remote_log_lines_waiting{instance="host1/blog"} 1`,
} {
if !strings.Contains(metrics, "\n"+series+"\n") {
t.Errorf("no %q in the metrics:\n%s", series, metrics)
@@ -993,7 +993,7 @@ func TestStalledRemoteLogEndpointHoldsUpNoRequest(t *testing.T) {
}
const dropped = "\nsww_remote_log_lines_dropped_total" +
- `{instance="fsn1app1/blog"} `
+ `{instance="host1/blog"} `
if strings.Contains(metrics, dropped+"0\n") ||
!strings.Contains(metrics, dropped) {
t.Errorf("no line dropped in the metrics:\n%s", metrics)
@@ -1089,7 +1089,7 @@ func TestBanIsAlertedAndAnAlertNotSentIsKeptAcrossARestart(t *testing.T) {
runUntilStopped(t, env, func(url string) {
webhook.waitFor(t, "permanent_ban", true)
- const ofWebhook = `{destination="webhook",instance="fsn1app1/blog"}`
+ const ofWebhook = `{destination="webhook",instance="host1/blog"}`
metrics := metricsWith(t, url+"_sww/metrics", token,
"\nsww_alerts_sent_total"+ofWebhook+" 1\n")
@@ -1176,13 +1176,13 @@ func TestBanIsAlertedToSlackAndNtfy(t *testing.T) {
err := json.Unmarshal([]byte(slackPost.body), &message)
if err != nil ||
- !strings.HasPrefix(message.Text, "*fsn1app1/blog: ban*\n") ||
+ !strings.HasPrefix(message.Text, "*host1/blog: ban*\n") ||
!strings.Contains(message.Text, "\nclient: "+client+"\n") {
t.Errorf("Slack was sent %s", slackPost.body)
}
ntfyPost := ntfy.firstPost(t)
- if ntfyPost.header.Get("Title") != "fsn1app1/blog: ban" ||
+ if ntfyPost.header.Get("Title") != "host1/blog: ban" ||
ntfyPost.header.Get("Authorization") != "Bearer "+ntfyToken ||
!strings.Contains(ntfyPost.body, "\nclient: "+client+"\n") {
t.Errorf("ntfy was sent %s, with the headers %v", ntfyPost.body,
@@ -1194,9 +1194,9 @@ func TestBanIsAlertedToSlackAndNtfy(t *testing.T) {
// test process cannot fail the test.
sent := []string{
"\nsww_alerts_sent_total{destination=\"slack\"," +
- "instance=\"fsn1app1/blog\"} 1\n",
+ "instance=\"host1/blog\"} 1\n",
"\nsww_alerts_sent_total{destination=\"ntfy\"," +
- "instance=\"fsn1app1/blog\"} 1\n",
+ "instance=\"host1/blog\"} 1\n",
}
metrics := metricsWith(t, url+"_sww/metrics", token, sent...)
diff --git a/internal/waf/alerts_test.go b/internal/waf/alerts_test.go
index c0c7616..42f1487 100644
--- a/internal/waf/alerts_test.go
+++ b/internal/waf/alerts_test.go
@@ -18,7 +18,9 @@ const (
alertWebhookURL = "SWW_WAF_ALERT_WEBHOOK_URL"
alertMaxPerHour = "SWW_WAF_ALERT_MAX_PER_HOUR"
// alertInstance is the instance every alert of these tests gives.
- alertInstance = "fsn1app1/blog"
+ alertInstance = "host1/blog"
+ // alertWebhook is where the tests send alerts.
+ alertWebhook = "https://alerts.example/sww"
)
func TestBanForABrokenLimitRaisesABanAlertWithItsNotes(t *testing.T) {
@@ -270,7 +272,7 @@ func startAppWithAlerts(
clk := &clock{now: time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)}
settings := map[string]string{
trustedProxies: trustLocalhost,
- alertWebhookURL: "https://alerts.example/sww",
+ alertWebhookURL: alertWebhook,
instanceName: alertInstance,
}
maps.Copy(settings, env)
diff --git a/internal/waf/answertime_test.go b/internal/waf/answertime_test.go
index 408e9e6..3e184a0 100644
--- a/internal/waf/answertime_test.go
+++ b/internal/waf/answertime_test.go
@@ -20,7 +20,7 @@ import (
"github.com/prometheus/client_golang/prometheus/testutil"
"sneak.berlin/go/smallwebwaf/internal/bans"
- "sneak.berlin/go/smallwebwaf/internal/lookup"
+ "sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation"
@@ -591,7 +591,7 @@ func TestAClientGeoJSPlacedInNoCountryCountsAsUnknownWithoutWaiting(
s := startWAF(t, geojsURL, map[string]string{
unknownCountryLimitPercent: "0", rateLimitPerMinute: "1",
})
- s.server.GeoJS.Load([]lookup.Answer{{
+ s.server.GeoJS.Load([]geojs.Answer{{
Client: netip.MustParsePrefix(client + "/32"), ASN: "AS64512",
Answered: s.clock.Now(), Used: s.clock.Now(),
}})
@@ -770,6 +770,8 @@ func TestARefusedRequestDoesNotWaitForASlowGeoJS(t *testing.T) {
// TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays says,
// so that a request that does not wait takes no time at all.
synctest.Test(t, func(t *testing.T) {
+ defer waitForGeoJSRequestsToBeAbandoned()
+
server, out := newWAFInBubble(t, tc.env)
tc.prepare(t, server, time.Now())
@@ -838,6 +840,8 @@ func TestAQuestionWaitsForASlowGeoJSOnlyAsLongAsTheDefaultSays(
t.Parallel()
synctest.Test(t, func(t *testing.T) {
+ defer waitForGeoJSRequestsToBeAbandoned()
+
server, out := newWAFInBubble(t, tc.env)
tc.prepare(t, server, time.Now())
@@ -950,10 +954,11 @@ func TestWAFAnswersWithinTheTimeLimitWhileEveryOutsideSourceHangs(
t.Errorf("%d checks with AbuseIPDB, want 1", checked)
}
- // The query is answered, and the check with AbuseIPDB abandoned,
- // before the bubble ends.
+ // The query is answered, and the check with AbuseIPDB and the request
+ // to GeoJS abandoned, before the bubble ends.
releaseResolver()
time.Sleep(defaultDNSBLAbuseIPDBWait)
+ waitForGeoJSRequestsToBeAbandoned()
})
}
diff --git a/internal/waf/besidelookup.go b/internal/waf/besidelookup.go
new file mode 100644
index 0000000..d1edf44
--- /dev/null
+++ b/internal/waf/besidelookup.go
@@ -0,0 +1,172 @@
+package waf
+
+// The heuristics of besideLookup, and how they run beside the lookup.
+
+import (
+ "context"
+ "sync/atomic"
+ "time"
+
+ "sneak.berlin/go/smallwebwaf/internal/config"
+ "sneak.berlin/go/smallwebwaf/internal/coreruleset"
+ "sneak.berlin/go/smallwebwaf/internal/requestlog"
+ "sneak.berlin/go/smallwebwaf/internal/rules"
+)
+
+// besideLookupTimeLimit is how long the heuristics of besideLookup have for a
+// request, from when they start. They only read the request, and no request
+// should take them that long. A request they have not finished by then is
+// refused with 503, as ActionChecksTimedOut.
+const besideLookupTimeLimit = 500 * time.Millisecond
+
+// besideLookupFindings is what the heuristics of besideLookup found of a
+// request, before any of it is applied to the request.
+type besideLookupFindings struct {
+ // trapPath is the request's path when it is in SWW_WAF_TRAP_PATHS, and ""
+ // otherwise.
+ trapPath string
+ // rules are the rules of the rule files the request matched.
+ rules []rules.Rule
+ // inspected is true when the Core Rule Set inspected the request.
+ // inspection, read and inspectionTime are then what it found, what it read
+ // of the body, and how long it took.
+ inspected bool
+ inspection coreruleset.Result
+ read []byte
+ inspectionTime time.Duration
+ // refuses is true when one of the heuristics refuses the request, or would
+ // in observe mode. Those after it did no work.
+ refuses bool
+}
+
+// runBesideLookup runs the heuristics of besideLookup while the lookup waits
+// for GeoJS, and returns the first verdict, or nil when none gives one.
+// lookedUp is the channel the lookup's result comes on, or nil once afterLookup
+// has let the request through.
+//
+// The first of the two to refuse the request ends the checks: waf answers at
+// once, and what the other found is dropped. besideLookup has
+// besideLookupTimeLimit; past it, the request is refused as
+// ActionChecksTimedOut. Once both have let the request through, the heuristics
+// of afterBesideLookup run. ctx is the request's own context.
+func (rq *request) runBesideLookup(
+ ctx context.Context, lookedUp <-chan lookupResult,
+) *verdict {
+ var running atomic.Pointer[string]
+
+ running.Store(new(trapPaths{}.name()))
+
+ found := make(chan *besideLookupFindings, 1)
+
+ go func() { found <- rq.findBesideLookup(&running) }()
+
+ timeUp := time.NewTimer(rq.h.besideLookupLimit)
+ defer timeUp.Stop()
+
+ var findings *besideLookupFindings
+
+ for findings == nil || lookedUp != nil {
+ select {
+ case result := <-lookedUp:
+ lookedUp = nil
+
+ v := rq.runAfterLookup(ctx, result)
+ if v != nil {
+ return v
+ }
+ case findings = <-found:
+ found = nil
+
+ timeUp.Stop()
+
+ if findings.refuses {
+ return rq.applyFindings(ctx, findings)
+ }
+ case <-timeUp.C:
+ return rq.timedOut(*running.Load())
+ }
+ }
+
+ v := rq.applyFindings(ctx, findings)
+ if v != nil {
+ return v
+ }
+
+ return rq.runHeuristics(ctx, afterBesideLookup(), rq.h.now())
+}
+
+// findBesideLookup does the work of the heuristics of besideLookup on the
+// request, in their order, until one refuses it, and returns what they found.
+// It changes nothing of the request or of what waf keeps, so that it can run
+// while the lookup and afterLookup do. running holds the name of the heuristic
+// at work, the first one's when findBesideLookup is called.
+func (rq *request) findBesideLookup(
+ running *atomic.Pointer[string],
+) *besideLookupFindings {
+ findings := &besideLookupFindings{}
+
+ findings.trapPath = rq.trapPath()
+ if findings.trapPath != "" {
+ findings.refuses = true
+
+ return findings
+ }
+
+ running.Store(new(ruleFiles{}.name()))
+
+ findings.rules = rq.h.rules.Match(rq.in)
+ if refusedByRule(findings.rules) {
+ findings.refuses = true
+
+ return findings
+ }
+
+ running.Store(new(coreRuleSet{}.name()))
+
+ cfg := rq.h.config
+ if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff ||
+ pathExempt(rq.in, cfg.CoreRuleSetExemptPaths) {
+ return findings
+ }
+
+ start := time.Now()
+ // A question's body has been read into memory, so that reading it here
+ // cannot fail.
+ findings.inspection, findings.read, _ = rq.h.coreRuleSet.Inspect(
+ rq.in, rq.client, rq.in.Body,
+ )
+ findings.inspectionTime = time.Since(start)
+ findings.inspected = true
+ findings.refuses = rq.refusedByCoreRuleSet(findings.inspection)
+
+ return findings
+}
+
+// applyFindings applies findings, what findBesideLookup found, to the
+// request: it runs the heuristics of besideLookup, which read them, at the
+// time it is called, and returns the first verdict, or nil when none gives
+// one.
+func (rq *request) applyFindings(
+ ctx context.Context, findings *besideLookupFindings,
+) *verdict {
+ rq.findings = findings
+
+ return rq.runHeuristics(ctx, besideLookup(), rq.h.now())
+}
+
+// timedOut refuses the request for the heuristics of besideLookup not having
+// finished with it within besideLookupTimeLimit. running is the name of the
+// one at work, which the log line's heuristic gives. The refusal makes no ban
+// and is no offence.
+func (rq *request) timedOut(running string) *verdict {
+ v := &verdict{
+ action: requestlog.ActionChecksTimedOut,
+ figures: map[string]any{
+ "time_limit_ms": rq.h.besideLookupLimit.Milliseconds(),
+ },
+ }
+ rq.line.Heuristic = running
+ rq.line.HeuristicFigures = v.figures
+
+ return v
+}
diff --git a/internal/waf/besidelookup_test.go b/internal/waf/besidelookup_test.go
new file mode 100644
index 0000000..7ecb49e
--- /dev/null
+++ b/internal/waf/besidelookup_test.go
@@ -0,0 +1,426 @@
+package waf_test
+
+import (
+ "io"
+ "maps"
+ "net/http"
+ "net/http/httptest"
+ "net/netip"
+ "strconv"
+ "strings"
+ "sync/atomic"
+ "testing"
+ "testing/synctest"
+ "time"
+
+ "github.com/prometheus/client_golang/prometheus/testutil"
+
+ "sneak.berlin/go/smallwebwaf/internal/alerts"
+ "sneak.berlin/go/smallwebwaf/internal/coreruleset"
+ "sneak.berlin/go/smallwebwaf/internal/ratelimit"
+ "sneak.berlin/go/smallwebwaf/internal/requestlog"
+ "sneak.berlin/go/smallwebwaf/internal/waf"
+)
+
+// checksTimeLimit is how long the trap paths, the rule files and the Core Rule
+// Set have for a request, as SPEC.md gives it, and checksTimedOut the status of
+// a request they have not finished by then.
+const (
+ checksTimeLimit = 500 * time.Millisecond
+ checksTimedOut = http.StatusServiceUnavailable
+)
+
+// heldCoreRuleSet is a stand-in for the Core Rule Set: each inspection takes
+// took, then finds result. asked counts the inspections.
+type heldCoreRuleSet struct {
+ took time.Duration
+ result coreruleset.Result
+ asked atomic.Int64
+}
+
+// Inspect inspects a request as the stand-in does, and reads all of body.
+func (c *heldCoreRuleSet) Inspect(
+ _ *http.Request, _ netip.Addr, body io.Reader,
+) (coreruleset.Result, []byte, error) {
+ c.asked.Add(1)
+ time.Sleep(c.took)
+
+ read, err := io.ReadAll(body)
+
+ return c.result, read, err
+}
+
+// coreRuleSetMatch returns what the stand-in finds of an attack: rule 942100,
+// with a score of 5, the default SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD.
+func coreRuleSetMatch() coreruleset.Result {
+ return coreruleset.Result{RuleIDs: []int{942100}, Score: 5}
+}
+
+// newWAFHoldingTheCoreRuleSet makes a waf, to be asked in a synctest bubble,
+// with crs in place of the Core Rule Set, in block mode. GeoJS never answers,
+// a setting needs its answer, alerts go to a webhook, and everything else is
+// at its default but for env. No request to it may use the network.
+func newWAFHoldingTheCoreRuleSet(
+ t *testing.T, crs *heldCoreRuleSet, env map[string]string,
+) (*waf.Server, *output, *alerts.Queue) {
+ t.Helper()
+
+ settings := map[string]string{
+ tagDeny: denyKP, coreRuleSetMode: block,
+ alertWebhookURL: alertWebhook,
+ }
+ maps.Copy(settings, env)
+
+ server, out, queue := newWAFWith(t, func(params waf.Params) *waf.Server {
+ params.CoreRuleSet = crs
+
+ return waf.New(params)
+ }, silentGeoJSURL, time.Now, settings)
+ server.SetBesideLookupTimeLimit(waf.BesideLookupTimeLimit)
+
+ return server, out, queue
+}
+
+// lookupWait is SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS for wait.
+func lookupWait(wait time.Duration) string {
+ return strconv.FormatInt(wait.Milliseconds(), 10)
+}
+
+// wantAnswer checks the status of an answer and how long it took.
+func wantAnswer(
+ t *testing.T, status int, took time.Duration,
+ wantStatus int, wantTook time.Duration, out *output,
+) {
+ t.Helper()
+
+ if status != wantStatus || took != wantTook {
+ t.Errorf(
+ "answered %d after %s, want %d after %s; output:\n%s",
+ status, took, wantStatus, wantTook, out.text(),
+ )
+ }
+}
+
+// wantNoCoreRuleSetTrace checks that what the Core Rule Set found of a request
+// left no trace: line has none of its fields, and server has counted no match
+// of it and raised no alert for one.
+func wantNoCoreRuleSetTrace(
+ t *testing.T, server *waf.Server, queue *alerts.Queue, line logLine,
+) {
+ t.Helper()
+
+ if line.DurationCoreRuleSetMS != nil || line.CoreRuleSetScore != nil ||
+ line.CoreRuleSetRuleIDs != nil {
+ t.Errorf(
+ "log line has Core Rule Set time %v, score %v and rule ids %v, want none",
+ line.DurationCoreRuleSetMS,
+ line.CoreRuleSetScore,
+ line.CoreRuleSetRuleIDs,
+ )
+ }
+
+ recorder := httptest.NewRecorder()
+ server.Metrics.ServeHTTP(
+ recorder,
+ httptest.NewRequestWithContext(
+ t.Context(),
+ http.MethodGet,
+ waf.MetricsPath,
+ http.NoBody,
+ ),
+ )
+
+ if strings.Contains(
+ recorder.Body.String(),
+ "sww_core_rule_set_matches_total{",
+ ) {
+ t.Errorf("a Core Rule Set match is counted")
+ }
+
+ for _, alert := range queue.Snapshot().Waiting[alerts.DestinationWebhook] {
+ if alert.Event == alerts.EventCoreRuleSetMatch {
+ t.Errorf("alert raised: %+v", alert)
+ }
+ }
+}
+
+func TestTheLookupAndTheCoreRuleSetTakeTheLongerOfTheirTimesNotTheirSum(
+ t *testing.T,
+) {
+ t.Parallel()
+
+ inspection := checksTimeLimit - time.Millisecond
+
+ for _, tc := range []struct {
+ name string
+ // wait is how long the request waits for GeoJS.
+ wait time.Duration
+ }{
+ {"the lookup waits longer", defaultLookupWait},
+ {"the inspection takes longer", inspection / 2},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ t.Parallel()
+
+ crs := &heldCoreRuleSet{took: inspection}
+ server, out, _ := newWAFHoldingTheCoreRuleSet(
+ t,
+ crs,
+ map[string]string{
+ asnAndCountryLookupTimeout: lookupWait(tc.wait),
+ },
+ )
+
+ synctest.Test(t, func(t *testing.T) {
+ defer waitForGeoJSRequestsToBeAbandoned()
+
+ status, took := askDirectly(
+ t,
+ server,
+ func(*http.Request) {},
+ )
+
+ wantAnswer(t, status, took, http.StatusOK,
+ max(tc.wait, inspection), out)
+ wantLine(t, out.requestLine(t), http.StatusOK,
+ requestlog.ActionForward)
+ })
+ })
+ }
+}
+
+// firstToRefuse is a case of
+// TestTheFirstOfTheLookupAndTheRequestChecksToRefuseIsAnswered: a request
+// under env, whose inspection by the Core Rule Set takes inspection, refused as
+// action after took.
+type firstToRefuse struct {
+ name string
+ env map[string]string
+ inspection time.Duration
+ took time.Duration
+ action string
+}
+
+// firstToRefuseCases are the cases of
+// TestTheFirstOfTheLookupAndTheRequestChecksToRefuseIsAnswered.
+func firstToRefuseCases() []firstToRefuse {
+ return []firstToRefuse{
+ {
+ name: "a trap path, before GeoJS answers",
+ env: map[string]string{trapPaths: "/", allowedCountries: "DE"},
+ took: 0, action: requestlog.ActionTemporarilyBanned,
+ },
+ {
+ name: "the Core Rule Set, before GeoJS answers",
+ env: map[string]string{allowedCountries: "DE"},
+ inspection: 100 * time.Millisecond,
+ took: 100 * time.Millisecond, action: requestlog.ActionBlockedByCoreRuleSet,
+ },
+ {
+ name: "the country list, while the Core Rule Set inspects",
+ env: map[string]string{
+ allowedCountries: "DE", asnAndCountryLookupTimeout: "100",
+ },
+ inspection: 400 * time.Millisecond,
+ took: 100 * time.Millisecond, action: requestlog.ActionCountryDenied,
+ },
+ {
+ name: "the rate limits, while the Core Rule Set inspects",
+ env: map[string]string{
+ rateLimitPerMinute: "2", unknownCountryLimitPercent: "50",
+ asnAndCountryLookupTimeout: "100",
+ },
+ inspection: 400 * time.Millisecond,
+ took: 100 * time.Millisecond, action: requestlog.ActionClientIsRateLimited,
+ },
+ }
+}
+
+func TestTheFirstOfTheLookupAndTheRequestChecksToRefuseIsAnswered(
+ t *testing.T,
+) {
+ t.Parallel()
+
+ const forbidden = http.StatusForbidden
+
+ for _, tc := range firstToRefuseCases() {
+ t.Run(tc.name, func(t *testing.T) {
+ t.Parallel()
+
+ crs := &heldCoreRuleSet{
+ took: tc.inspection,
+ result: coreRuleSetMatch(),
+ }
+ server, out, queue := newWAFHoldingTheCoreRuleSet(t, crs, tc.env)
+
+ synctest.Test(t, func(t *testing.T) {
+ defer waitForGeoJSRequestsToBeAbandoned()
+
+ // Under the rate limits of the last case, the client's next
+ // request takes it over a limit only at the percentage of an
+ // unknown country, so it waits for GeoJS.
+ countClient(t, server, time.Now())
+
+ status, took := askDirectly(
+ t,
+ server,
+ func(*http.Request) {},
+ )
+
+ wantAnswer(t, status, took, forbidden, tc.took, out)
+
+ line := out.requestLine(t)
+ wantLine(t, line, forbidden, tc.action)
+
+ if tc.action == requestlog.ActionBlockedByCoreRuleSet {
+ return
+ }
+
+ if tc.action != requestlog.ActionClientIsRateLimited &&
+ line.Counts != (ratelimit.Counts{}) {
+ t.Errorf(
+ "the rate limits counted the request: %+v",
+ line.Counts,
+ )
+ }
+
+ // What the Core Rule Set finds once the request is answered is
+ // dropped.
+ time.Sleep(tc.inspection)
+ wantNoCoreRuleSetTrace(t, server, queue, line)
+ })
+ })
+ }
+}
+
+func TestARequestRefusedBeforeTheLookupStartsNoInspection(t *testing.T) {
+ t.Parallel()
+
+ crs := &heldCoreRuleSet{took: time.Second}
+ server, out, _ := newWAFHoldingTheCoreRuleSet(t, crs, nil)
+
+ synctest.Test(t, func(t *testing.T) {
+ banClient(t, server, time.Now())
+
+ status, took := askDirectly(t, server, func(*http.Request) {})
+
+ wantAnswer(t, status, took, http.StatusForbidden, 0, out)
+
+ requests := testutil.ToFloat64(server.Metrics.GeoJSRequests)
+ if asked := crs.asked.Load(); asked != 0 || requests != 0 {
+ t.Errorf(
+ "%d inspections and %v requests to GeoJS, want none",
+ asked,
+ requests,
+ )
+ }
+ })
+}
+
+func TestAClientInTheAllowNetsIsAllowedAtOnceAndNothingElseStarts(
+ t *testing.T,
+) {
+ t.Parallel()
+
+ crs := &heldCoreRuleSet{took: time.Second, result: coreRuleSetMatch()}
+ server, out, _ := newWAFHoldingTheCoreRuleSet(t, crs,
+ map[string]string{allowNets: client, allowedCountries: "DE"})
+
+ synctest.Test(t, func(t *testing.T) {
+ status, took := askDirectly(t, server, func(*http.Request) {})
+
+ wantAnswer(t, status, took, http.StatusOK, 0, out)
+
+ requests := testutil.ToFloat64(server.Metrics.GeoJSRequests)
+ if asked := crs.asked.Load(); asked != 0 || requests != 0 {
+ t.Errorf(
+ "%d inspections and %v requests to GeoJS, want none",
+ asked,
+ requests,
+ )
+ }
+ })
+}
+
+func TestARequestTheCoreRuleSetHasNotFinishedWithIn500MillisecondsIsRefused(
+ t *testing.T,
+) {
+ t.Parallel()
+
+ crs := &heldCoreRuleSet{took: checksTimeLimit + time.Second}
+ server, out, _ := newWAFHoldingTheCoreRuleSet(t, crs,
+ map[string]string{refusalsLimitPerMinute: "1"})
+
+ synctest.Test(t, func(t *testing.T) {
+ defer waitForGeoJSRequestsToBeAbandoned()
+
+ // Were it counted toward the refusal limit, the second refusal would
+ // ban the client, and the third request would be refused for the ban.
+ for i := range 3 {
+ status, took := askDirectly(t, server, func(req *http.Request) {
+ req.Header.Set(requestIDHeader, "id-"+strconv.Itoa(i))
+ })
+
+ wantAnswer(t, status, took, checksTimedOut, checksTimeLimit, out)
+
+ line := out.requestLines(t, i+1)[i]
+ wantLine(
+ t,
+ line,
+ checksTimedOut,
+ requestlog.ActionChecksTimedOut,
+ )
+ wantHeuristic(t, line, "core_rule_set", `{"time_limit_ms": 500}`)
+
+ if line.Offence != "" || line.BanExpires != "" {
+ t.Errorf(
+ "log line has offence %q and ban_expires %q, want neither",
+ line.Offence,
+ line.BanExpires,
+ )
+ }
+ }
+
+ if bans := server.Ledger.Snapshot(); len(bans) != 0 {
+ t.Errorf("bans %+v, want none", bans)
+ }
+
+ offences := historyOf(t, server, client).Offences
+ if offences != (ratelimit.Offences{}) {
+ t.Errorf("history counts the offences %+v, want none", offences)
+ }
+
+ time.Sleep(crs.took)
+ })
+}
+
+func TestObserveModeAllowsARequestTheCoreRuleSetHasNotFinishedWith(
+ t *testing.T,
+) {
+ t.Parallel()
+
+ crs := &heldCoreRuleSet{took: checksTimeLimit + time.Second}
+ server, out, _ := newWAFHoldingTheCoreRuleSet(t, crs,
+ map[string]string{mode: observe})
+
+ synctest.Test(t, func(t *testing.T) {
+ defer waitForGeoJSRequestsToBeAbandoned()
+
+ status, took := askDirectly(t, server, func(*http.Request) {})
+
+ wantAnswer(t, status, took, http.StatusOK, checksTimeLimit, out)
+
+ line := out.requestLine(t)
+ wantLine(t, line, http.StatusOK, requestlog.ActionForward)
+
+ if line.WouldAction != requestlog.ActionChecksTimedOut {
+ t.Errorf(
+ "log line has would_action %q, want %q",
+ line.WouldAction,
+ requestlog.ActionChecksTimedOut,
+ )
+ }
+
+ time.Sleep(crs.took)
+ })
+}
diff --git a/internal/waf/biased_test.go b/internal/waf/biased_test.go
index 17dadf7..0835a4f 100644
--- a/internal/waf/biased_test.go
+++ b/internal/waf/biased_test.go
@@ -339,6 +339,8 @@ func TestClientWithoutAnAnswerInTimeHasTheUnknownCountryLimitPercent(
// TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays says, with
// a GeoJS that never answers.
synctest.Test(t, func(t *testing.T) {
+ defer waitForGeoJSRequestsToBeAbandoned()
+
server, out, _ := newWAF(t, silentGeoJSURL, time.Now,
map[string]string{unknownCountryLimitPercent: "0"})
@@ -390,6 +392,8 @@ func TestRequestWaitsForItsLookupWhileABiasedThresholdIsSet(t *testing.T) {
// with a GeoJS that never answers. The headers, which need the
// answer too, are off.
synctest.Test(t, func(t *testing.T) {
+ defer waitForGeoJSRequestsToBeAbandoned()
+
server, out, _ := newWAF(t, silentGeoJSURL, time.Now,
map[string]string{
asnAndCountryLookupTimeout: strconv.FormatInt(
diff --git a/internal/waf/coreruleset.go b/internal/waf/coreruleset.go
index e05ca11..4acc2cc 100644
--- a/internal/waf/coreruleset.go
+++ b/internal/waf/coreruleset.go
@@ -13,10 +13,11 @@ import (
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
-// coreRuleSet inspects the request with the Core Rule Set, unless
-// SWW_WAF_CORE_RULE_SET_MODE is off or SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS
-// exempts its path, as pathExempt decides, and notes the rules it matched and
-// its score in the log line, and the rules in the metrics. A score at or over
+// coreRuleSet takes what the Core Rule Set found of the request, which
+// findBesideLookup inspected with it unless SWW_WAF_CORE_RULE_SET_MODE is off
+// or SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS exempts its path, as pathExempt
+// decides. It notes the rules the request matched and its score in the log
+// line, and the rules in the metrics. A score at or over
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD is a match: it adds the tag
// exploitsignature, raises the core_rule_set_match alert, and in block mode
// refuses the request with 403, which is an offence its client's history
@@ -30,21 +31,17 @@ func (coreRuleSet) check(
rq *request,
_ time.Time,
) *verdict {
- cfg := rq.h.config
- if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff ||
- pathExempt(rq.in, cfg.CoreRuleSetExemptPaths) {
+ findings := rq.findings
+ if !findings.inspected {
return nil
}
- start := time.Now()
-
- // A question's body has been read into memory, so that reading it here
- // cannot fail.
- result, read, _ := rq.h.coreRuleSet.Inspect(rq.in, rq.client, rq.in.Body)
- rq.requestBytes = int64(len(read))
+ cfg := rq.h.config
+ result := findings.inspection
+ rq.requestBytes = int64(len(findings.read))
rq.line.DurationCoreRuleSetMS = new(
- requestlog.Milliseconds(time.Since(start)),
+ requestlog.Milliseconds(findings.inspectionTime),
)
rq.line.CoreRuleSetRuleIDs = result.RuleIDs
rq.line.CoreRuleSetScore = &result.Score
@@ -61,7 +58,7 @@ func (coreRuleSet) check(
rq.addTag(config.ExploitSignatureTag)
rq.alertCoreRuleSetMatch(result)
- if cfg.CoreRuleSetMode == config.CoreRuleSetModeDetect {
+ if !rq.refusedByCoreRuleSet(result) {
return nil
}
@@ -76,6 +73,17 @@ func (coreRuleSet) check(
}
}
+// refusedByCoreRuleSet reports whether coreRuleSet refuses a request the Core
+// Rule Set found result of: whether the score is at or over
+// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD in block mode.
+func (rq *request) refusedByCoreRuleSet(result coreruleset.Result) bool {
+ cfg := rq.h.config
+ threshold := cfg.CoreRuleSetScoreThreshold
+
+ return cfg.CoreRuleSetMode == config.CoreRuleSetModeBlock &&
+ threshold != 0 && result.Score >= threshold
+}
+
// alertCoreRuleSetMatch raises the core_rule_set_match alert for the request,
// which the Core Rule Set scored at result, at or over
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD. Its detail gives the rule ids, the
diff --git a/internal/waf/export_test.go b/internal/waf/export_test.go
new file mode 100644
index 0000000..60b6e9b
--- /dev/null
+++ b/internal/waf/export_test.go
@@ -0,0 +1,13 @@
+package waf
+
+import "time"
+
+// BesideLookupTimeLimit is besideLookupTimeLimit.
+const BesideLookupTimeLimit = besideLookupTimeLimit
+
+// SetBesideLookupTimeLimit gives the heuristics of besideLookup limit on s in
+// place of besideLookupTimeLimit.
+func (s *Server) SetBesideLookupTimeLimit(limit time.Duration) {
+ h := s.Handler.(*handler) //nolint:forcetypeassert // New makes it one
+ h.besideLookupLimit = limit
+}
diff --git a/internal/waf/heuristics.go b/internal/waf/heuristics.go
index 76153e2..9fb7e59 100644
--- a/internal/waf/heuristics.go
+++ b/internal/waf/heuristics.go
@@ -54,8 +54,6 @@ func beforeLookup() []heuristic {
}
// afterLookup are the heuristics that run once the lookup has been made.
-// tag_deny runs twice: before the rate limits, on the tags known by then, and
-// last, on the tag rule_files and core_rule_set can add.
func afterLookup() []heuristic {
return []heuristic{
countries{},
@@ -64,9 +62,26 @@ func afterLookup() []heuristic {
abuseIPDB{},
tagDeny{},
rateLimits{},
+ }
+}
+
+// besideLookup are the heuristics that only read the request. Their work runs
+// beside the lookup and afterLookup, in findBesideLookup, which changes nothing
+// of the request. Their check then applies what that work found, and runs only
+// once waf takes it.
+func besideLookup() []heuristic {
+ return []heuristic{
trapPaths{},
ruleFiles{},
coreRuleSet{},
+ }
+}
+
+// afterBesideLookup are the heuristics that run once afterLookup and
+// besideLookup have both let the request through. tag_deny runs a second time,
+// on the tag rule_files and core_rule_set can add.
+func afterBesideLookup() []heuristic {
+ return []heuristic{
tagDeny{},
}
}
@@ -95,7 +110,7 @@ func (rq *request) runAfterAnswer(ctx context.Context) {
rq.runBanningHeuristics(ctx, afterAnswer())
}
-// runHeuristics runs list, beforeLookup or afterLookup, on the request at now,
+// runHeuristics runs list, one of those checkClient runs, on the request at now,
// and returns the first verdict, or nil when none gives one. The log line's
// heuristic and heuristic_figures name the heuristic that gave it, and give its
// figures. ctx is the request's own context.
diff --git a/internal/waf/lookup.go b/internal/waf/lookup.go
index 8627cd2..8250667 100644
--- a/internal/waf/lookup.go
+++ b/internal/waf/lookup.go
@@ -7,8 +7,9 @@ import (
"strings"
"time"
+ "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/config"
- "sneak.berlin/go/smallwebwaf/internal/lookup"
+ "sneak.berlin/go/smallwebwaf/internal/geojs"
)
// The headers in which an answer of allow gives rproxy the client's AS number
@@ -20,33 +21,80 @@ const (
countryHeader = "X-Client-Country"
)
-// lookUp looks up the client's AS number and country, in the lookup
-// database or through GeoJS, and notes them for the log line and as the
-// request's asn= and country= tags, unless
-// SWW_WAF_LOOKUP_SOURCE is off or the client is on a private, loopback or
-// link-local address, which no lookup can place. The lookup database
-// answers at once. With GeoJS, while a setting needs the answer, such as a
-// country list or a biased threshold, a new client's request waits for it,
-// unless refusedWhateverTheAnswer reports true. ctx is the request's own
-// context.
-func (rq *request) lookUp(ctx context.Context, now time.Time) {
+// lookupResult is what the lookup gave for a request: whether its client was
+// looked up, the answer, and whether the request skipped the wait for GeoJS.
+type lookupResult struct {
+ lookedUp, skippedWait bool
+ answer geojs.Answer
+}
+
+// startLookUp starts looking up the client's AS number and country, in the
+// lookup database or through GeoJS, unless SWW_WAF_LOOKUP_SOURCE is off or the
+// client is on a private, loopback or link-local address, which no lookup can
+// place. It returns the channel the result comes on. A result that needs no
+// wait is on it when startLookUp returns: the lookup database's, an answer
+// GeoJS gave earlier, or none. With GeoJS, while a setting needs the answer,
+// such as a country list or a biased threshold, a new client's request waits
+// for it up to SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS, unless
+// refusedWhateverTheAnswer reports true. The wait goes on to its end even once
+// the request has been answered, so that it ends as it would have for a
+// request that waited. startLookUp changes nothing of the request. ctx is the
+// request's own context.
+func (rq *request) startLookUp(
+ ctx context.Context, now time.Time,
+) <-chan lookupResult {
+ result := make(chan lookupResult, 1)
+
if rq.h.config.LookupSource == "off" || !canBePlaced(rq.client) {
- return
+ result <- lookupResult{}
+
+ return result
}
client := rq.h.clientGroup(rq.client)
- switch {
- case rq.h.config.LookupSource == "file":
- rq.lookupAnswer = rq.h.lookupFile.LookUp(client)
- case rq.refusedWhateverTheAnswer(now):
- rq.lookupAnswer = rq.h.geojs.LookUpWithoutWaiting(ctx, client)
- rq.skippedLookupWait = rq.lookupAnswer.Answered.IsZero()
- default:
- rq.lookupAnswer = rq.h.geojs.LookUp(ctx, client)
+ if rq.h.config.LookupSource == "file" {
+ result <- lookupResult{
+ lookedUp: true, answer: rq.h.lookupFile.LookUp(client),
+ }
+
+ return result
+ }
+
+ answer := rq.h.geojs.LookUpWithoutWaiting(ctx, client)
+ refused := rq.refusedWhateverTheAnswer(now)
+
+ if !answer.Answered.IsZero() || refused || !rq.h.waitForGeoJS {
+ result <- lookupResult{
+ lookedUp: true,
+ skippedWait: refused && answer.Answered.IsZero(),
+ answer: answer,
+ }
+
+ return result
+ }
+
+ go func() {
+ result <- lookupResult{
+ lookedUp: true,
+ answer: rq.h.geojs.LookUp(context.WithoutCancel(ctx), client,
+ rq.h.config.ASNAndCountryLookupTimeout),
+ }
+ }()
+
+ return result
+}
+
+// noteLookup notes result for the log line and as the request's asn= and
+// country= tags.
+func (rq *request) noteLookup(result lookupResult) {
+ if !result.lookedUp {
+ return
}
rq.lookedUp = true
+ rq.lookupAnswer = result.answer
+ rq.skippedLookupWait = result.skippedWait
rq.line.ASN = rq.lookupAnswer.ASN
rq.line.ASName = rq.lookupAnswer.ASName
rq.line.Country = rq.lookupAnswer.Country
@@ -83,13 +131,29 @@ func countryTag(country string) string {
// addLookup adds answer, an answer about a client from the lookup
// database or GeoJS, to the client's history, and to the notes of the bans
// on its netblock that have no AS number, AS name or country yet.
-func (h *handler) addLookup(answer lookup.Answer) {
+func (h *handler) addLookup(answer geojs.Answer) {
h.limiter.AddLookup(answer.Client, answer.Answered,
answer.ASN, answer.ASName, answer.Country)
h.ledger.AddLookup(h.netblock(answer.Client.Addr()),
answer.ASN, answer.ASName, answer.Country)
}
+// geojsFailed logs a failure of GeoJS, raises a source_failure alert for it
+// and counts it. askingAgainIn is how long GeoJS is left alone after it.
+func (h *handler) geojsFailed(err error, askingAgainIn time.Duration) {
+ h.processLog.Warn("asking GeoJS failed",
+ "error", err.Error(), "asking_again_in", askingAgainIn.String())
+ h.alerts.Raise(alerts.Alert{
+ Event: alerts.EventSourceFailure,
+ Reason: "asking GeoJS failed",
+ Detail: map[string]any{
+ "source": "geojs", "error": err.Error(),
+ "asking_again_in": askingAgainIn.String(),
+ },
+ })
+ h.metrics.GeoJSFailures.Inc()
+}
+
// setLookupHeaders sets the headers that give the client's AS number and
// country, leaving out one that is unknown.
func setLookupHeaders(header http.Header, asn, country string) {
diff --git a/internal/waf/lookup_test.go b/internal/waf/lookup_test.go
index 407c87c..3c2d413 100644
--- a/internal/waf/lookup_test.go
+++ b/internal/waf/lookup_test.go
@@ -12,7 +12,7 @@ import (
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
- "sneak.berlin/go/smallwebwaf/internal/lookup"
+ "sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
@@ -138,7 +138,7 @@ func TestRequestThatWaitedForItsLookupIsCountedWhenItGoesOn(t *testing.T) {
// of lookups.json would keep it, while the first request still waits
// for GeoJS. The second request goes on at once, and is counted.
clk.advance(2500 * time.Millisecond)
- server.GeoJS.Load([]lookup.Answer{{
+ server.GeoJS.Load([]geojs.Answer{{
Client: netip.MustParsePrefix(
fromDE + "/32",
), ASN: asnDE, ASName: asNameDE,
@@ -190,7 +190,7 @@ func TestASNumberAndNameInTheLogLineTheHistoryTheBanNotesAndTheAlert(
clk.Now,
map[string]string{
trustedProxies: trustLocalhost,
- alertWebhookURL: "https://alerts.example/sww",
+ alertWebhookURL: alertWebhook,
rateLimitPerMinute: "1",
},
)
@@ -199,7 +199,7 @@ func TestASNumberAndNameInTheLogLineTheHistoryTheBanNotesAndTheAlert(
// The answer is kept before the requests, so GeoJS is not asked, and
// gives no answer of its own.
netblock := netip.MustParsePrefix(fromDE + "/32")
- server.GeoJS.Load([]lookup.Answer{{
+ server.GeoJS.Load([]geojs.Answer{{
Client: netblock, ASN: asnDE, ASName: asNameDE, Country: "DE",
Answered: clk.Now(), Used: clk.Now(),
}})
@@ -249,7 +249,7 @@ func TestLookupSourceOffLooksNoClientUp(t *testing.T) {
)
// Even an answer kept from before is not used.
- server.GeoJS.Load([]lookup.Answer{{
+ server.GeoJS.Load([]geojs.Answer{{
Client: netip.MustParsePrefix(
fromDE + "/32",
), ASN: asnDE, ASName: asNameDE,
@@ -450,6 +450,8 @@ func TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays(
// network, which would keep that clock from moving on: the request is
// handed to waf's handler, and GeoJS is one that never answers.
synctest.Test(t, func(t *testing.T) {
+ defer waitForGeoJSRequestsToBeAbandoned()
+
// Not the default. The exclusive list needs the answer, and the app
// is never reached.
const timeout = 3 * time.Second
@@ -495,6 +497,17 @@ func TestMain(m *testing.M) {
m.Run()
}
+// geojsRequestTimeout is how long a request to GeoJS may take before it is
+// abandoned.
+const geojsRequestTimeout = 10 * time.Second
+
+// waitForGeoJSRequestsToBeAbandoned sleeps until every request to the GeoJS at
+// silentGeoJSURL made so far has been abandoned. A synctest bubble that asks
+// it defers this: the bubble cannot end while a request waits on it.
+func waitForGeoJSRequestsToBeAbandoned() {
+ time.Sleep(geojsRequestTimeout)
+}
+
// silentGeoJS is the GeoJS at silentGeoJSURL.
type silentGeoJS struct{}
diff --git a/internal/waf/metrics_test.go b/internal/waf/metrics_test.go
index bf96455..dac7175 100644
--- a/internal/waf/metrics_test.go
+++ b/internal/waf/metrics_test.go
@@ -1,6 +1,7 @@
package waf_test
import (
+ "fmt"
"io"
"net/http"
"net/http/httptest"
@@ -10,8 +11,9 @@ import (
"testing"
"time"
+ "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans"
- "sneak.berlin/go/smallwebwaf/internal/lookup"
+ "sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
@@ -248,7 +250,7 @@ func TestMetricsByCountryKeepTheBusiestAndCountTheRestAsOther(t *testing.T) {
// The answers are kept before the requests, so that none waits for
// GeoJS.
- server.GeoJS.Load([]lookup.Answer{
+ server.GeoJS.Load([]geojs.Answer{
keptAnswer(fromKP, "KP"), keptAnswer(fromDE, "DE"),
keptAnswer(fromFR, "FR"),
})
@@ -329,13 +331,13 @@ func TestMetricsByASNumberKeepTheBusiestAndCountTheRestAsOther(
// The answers are kept before the requests, so that GeoJS gives none
// of its own. Each client is in an AS of its own.
- answer := func(addr, asn string) lookup.Answer {
- return lookup.Answer{
+ answer := func(addr, asn string) geojs.Answer {
+ return geojs.Answer{
Client: netip.MustParsePrefix(addr + "/32"), ASN: asn,
Answered: clk.Now(), Used: clk.Now(),
}
}
- server.GeoJS.Load([]lookup.Answer{
+ server.GeoJS.Load([]geojs.Answer{
answer(fromDE, "AS64501"), answer(fromKP, "AS64502"),
})
@@ -357,21 +359,23 @@ func TestMetricsByASNumberKeepTheBusiestAndCountTheRestAsOther(
`sww_asn_requests_total{asn="AS64502",instance="app"}`)
}
-func TestMetricsCountGeoJSRequestsAndFailures(t *testing.T) {
+func TestGeoJSFailureIsCountedLoggedAndAlerted(t *testing.T) {
t.Parallel()
- geojs := httptest.NewServer(http.HandlerFunc(
+ failingGeoJS := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusServiceUnavailable)
}))
- t.Cleanup(geojs.Close)
+ t.Cleanup(failingGeoJS.Close)
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
- addr, _, server := startProxyWithClock(t, app.URL, geojs.URL, time.Now,
+ addr, out, server, queue := startProxyWithAlerts(t, app.URL,
+ failingGeoJS.URL, time.Now,
map[string]string{
- trustedProxies: trustLocalhost,
- metricsToken: token,
- tagDeny: denyKP,
+ trustedProxies: trustLocalhost,
+ metricsToken: token,
+ tagDeny: denyKP,
+ alertWebhookURL: "https://alerts.example/waf",
})
// GeoJS fails, so the client has no country= tag, and SWW_WAF_TAG_DENY does
@@ -401,14 +405,46 @@ func TestMetricsCountGeoJSRequestsAndFailures(t *testing.T) {
`sww_requests_without_geojs_answer_total{instance="app"}`,
1,
)
+
+ // The process log's line names the failure but not the address asked
+ // about.
+ logged := ""
+
+ for line := range strings.Lines(out.text()) {
+ if strings.Contains(line, "asking GeoJS failed") {
+ logged = line
+ }
+ }
+
+ if logged == "" || strings.Contains(logged, fromDE) {
+ t.Errorf(
+ "logged %q, want the failure without the address asked about",
+ logged,
+ )
+ }
+
+ // fmt prints a map sorted by its keys.
+ const want = "map[asking_again_in:1s " +
+ "error:GeoJS answered 503 Service Unavailable source:geojs]"
+
+ waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
+ if len(waiting) != 1 || waiting[0].Event != alerts.EventSourceFailure ||
+ waiting[0].Reason != "asking GeoJS failed" ||
+ fmt.Sprint(waiting[0].Detail) != want {
+ t.Errorf(
+ "alerts waiting %+v, want one source_failure alert with %v",
+ waiting,
+ want,
+ )
+ }
}
// keptAnswer returns GeoJS's answer that the client at addr is in
// country, given now.
-func keptAnswer(addr, country string) lookup.Answer {
+func keptAnswer(addr, country string) geojs.Answer {
now := time.Now()
- return lookup.Answer{
+ return geojs.Answer{
Client: netip.MustParsePrefix(addr + "/32"), Country: country,
Answered: now, Used: now,
}
diff --git a/internal/waf/proxy.go b/internal/waf/proxy.go
index 59500fe..9179ae4 100644
--- a/internal/waf/proxy.go
+++ b/internal/waf/proxy.go
@@ -8,6 +8,7 @@ import (
"io"
"log/slog"
"net/http"
+ "net/netip"
"time"
lru "github.com/hashicorp/golang-lru/v2"
@@ -17,6 +18,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
+ "sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
@@ -55,7 +57,7 @@ type Params struct {
// ProcessLog receives the process's own messages.
ProcessLog *slog.Logger
// GeoJSURL is where clients' AS numbers and countries are looked up
- // while SWW_WAF_LOOKUP_SOURCE is geojs, normally lookup.URL.
+ // while SWW_WAF_LOOKUP_SOURCE is geojs, normally geojs.URL.
GeoJSURL string
// AbuseIPDBURL is where clients are checked with AbuseIPDB while
// SWW_WAF_ABUSEIPDB_KEY is set, normally reputation.AbuseIPDBURL.
@@ -65,7 +67,7 @@ type Params struct {
LookupFile *lookup.File
// CoreRuleSet is what NewCoreRuleSet returns: what inspects requests
// while SWW_WAF_CORE_RULE_SET_MODE is not off, and nil while it is.
- CoreRuleSet *coreruleset.CoreRuleSet
+ CoreRuleSet CoreRuleSetInspector
// Now tells the time by which requests are counted for the rate
// limits, bans are made and run out, and GeoJS's answers are kept,
// normally time.Now in UTC, the time the state files give.
@@ -83,6 +85,14 @@ type Params struct {
Alerts *alerts.Queue
}
+// CoreRuleSetInspector inspects a request with the Core Rule Set, as
+// coreruleset.CoreRuleSet's Inspect says. Tests give waf a stand-in for it.
+type CoreRuleSetInspector interface {
+ Inspect(
+ r *http.Request, client netip.Addr, body io.Reader,
+ ) (coreruleset.Result, []byte, error)
+}
+
// Server is the server waf runs, with the parts whose state the state
// files keep, the lookup database, nil unless SWW_WAF_LOOKUP_SOURCE is file, the
// lists fetched from URLs, which its Run fetches, the DNSBL zones' verdicts,
@@ -93,7 +103,7 @@ type Server struct {
Ledger *bans.Ledger
Limiter *ratelimit.Limiter
- GeoJS *lookup.GeoJS
+ GeoJS *geojs.GeoJS
Anomalies *anomaly.Counters
LookupFile *lookup.File
Lists *reputation.Lists
@@ -157,6 +167,8 @@ func newHandler(params Params) *handler {
rules: params.Rules,
coreRuleSet: params.CoreRuleSet,
alerts: params.Alerts,
+
+ besideLookupLimit: besideLookupTimeLimit,
}
if params.Config.RouteWatchURL != "" {
h.routeWatch = routewatch.New(routewatch.Params{
@@ -164,20 +176,19 @@ func newHandler(params Params) *handler {
})
}
- h.geojs = lookup.New(lookup.Params{
- URL: params.GeoJSURL,
- Timeout: params.Config.ASNAndCountryLookupTimeout,
- // The country list, the headers, the tag settings that name an asn=
- // or country= tag and the biased thresholds act on the answer before
- // the request goes on.
- Wait: len(params.Config.ExclusivelyAllowedCountries) > 0 ||
- params.Config.AddLookupHeaders || params.Config.TagsNeedLookup() ||
- biasedThresholdsSet(params.Config),
- Answered: h.addLookup,
- Now: params.Now,
- ProcessLog: params.ProcessLog,
- Metrics: m,
- Alerts: params.Alerts,
+ // The country list, the headers, the tag settings that name an asn= or
+ // country= tag and the biased thresholds act on the answer before the
+ // request goes on.
+ h.waitForGeoJS = len(params.Config.ExclusivelyAllowedCountries) > 0 ||
+ params.Config.AddLookupHeaders || params.Config.TagsNeedLookup() ||
+ biasedThresholdsSet(params.Config)
+ h.geojs = geojs.New(geojs.Params{
+ URL: params.GeoJSURL,
+ Answered: h.addLookup,
+ Now: params.Now,
+ Asked: m.GeoJSRequests.Inc,
+ Failed: h.geojsFailed,
+ WentWithoutAnswer: m.RequestsWithoutGeoJSAnswer.Inc,
})
m.AddBansAndClients(h.ledger, h.limiter, params.Now)
m.AddRules(params.Rules)
@@ -228,12 +239,9 @@ func newReputation(
// without the rules SWW_WAF_CORE_RULE_SET_DISABLED_RULES switches off, reading bodies
// up to SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE, with the exclusions in
// SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE and the fixes of the apps in SWW_WAF_APPS,
-// or nil while SWW_WAF_CORE_RULE_SET_MODE is off. It
-// reads at most QuestionBodyMaxBytes less one byte: a question's body of
-// QuestionBodyMaxBytes may be cut short, and the Core Rule Set does not read a
-// JSON or XML body over its limit. An exclusions file that coreruleset.New refuses is
-// an error in every mode. While the mode is off, a set file is loaded with the
-// Core Rule Set only to check it.
+// or nil while SWW_WAF_CORE_RULE_SET_MODE is off. An exclusions file that
+// coreruleset.New refuses is an error in every mode. While the mode is off, a
+// set file is loaded with the Core Rule Set only to check it.
func NewCoreRuleSet(cfg *config.WAF) (*coreruleset.CoreRuleSet, error) {
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff &&
cfg.CoreRuleSetExclusionsFile == "" {
@@ -241,14 +249,11 @@ func NewCoreRuleSet(cfg *config.WAF) (*coreruleset.CoreRuleSet, error) {
}
coreRuleSet, err := coreruleset.New(coreruleset.Params{
- ParanoiaLevel: cfg.CoreRuleSetParanoiaLevel,
- DisabledRules: cfg.CoreRuleSetDisabledRules,
- BodyInspectionSize: min(
- cfg.CoreRuleSetBodyInspectionSize,
- QuestionBodyMaxBytes-1,
- ),
- ExclusionsFile: cfg.CoreRuleSetExclusionsFile,
- Apps: cfg.Apps,
+ ParanoiaLevel: cfg.CoreRuleSetParanoiaLevel,
+ DisabledRules: cfg.CoreRuleSetDisabledRules,
+ BodyInspectionSize: cfg.CoreRuleSetBodyInspectionSize,
+ ExclusionsFile: cfg.CoreRuleSetExclusionsFile,
+ Apps: cfg.Apps,
})
if err != nil {
return nil, fmt.Errorf(
@@ -274,7 +279,7 @@ type handler struct {
metrics *metrics.Metrics
limiter *ratelimit.Limiter
ledger *bans.Ledger
- geojs *lookup.GeoJS
+ geojs *geojs.GeoJS
anomalies *anomaly.Counters
lookupFile *lookup.File
lists *reputation.Lists
@@ -282,8 +287,13 @@ type handler struct {
abuseIPDB *reputation.AbuseIPDB
routeWatch *routewatch.Client
rules *rules.Files
- coreRuleSet *coreruleset.CoreRuleSet
+ coreRuleSet CoreRuleSetInspector
alerts *alerts.Queue
+ // waitForGeoJS is true when a setting needs each request's answer from
+ // GeoJS before the request goes on.
+ waitForGeoJS bool
+ // besideLookupLimit is besideLookupTimeLimit. Only tests change it.
+ besideLookupLimit time.Duration
// awaitingReport holds the decision on each answered question by its id
// until its report comes, and nil for a question still being handled.
awaitingReport *lru.Cache[string, *decision]
diff --git a/internal/waf/proxy_test.go b/internal/waf/proxy_test.go
index e32b864..054b46b 100644
--- a/internal/waf/proxy_test.go
+++ b/internal/waf/proxy_test.go
@@ -512,6 +512,11 @@ func newWAFWith(
Alerts: alertQueue,
})
+ // The race detector and a busy host can slow the Core Rule Set's
+ // inspections past the time limit. The tests of the limit set it back, and
+ // hold the Core Rule Set with a stand-in.
+ server.SetBesideLookupTimeLimit(time.Hour)
+
return server, out, alertQueue
}
diff --git a/internal/waf/ratelimits_test.go b/internal/waf/ratelimits_test.go
index 47aca6f..962668a 100644
--- a/internal/waf/ratelimits_test.go
+++ b/internal/waf/ratelimits_test.go
@@ -5,7 +5,7 @@ import (
"sync/atomic"
"testing"
- "sneak.berlin/go/smallwebwaf/internal/lookup"
+ "sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
@@ -162,7 +162,7 @@ func TestRateLimitExemptPathsAreNeitherCountedNorRefused(t *testing.T) {
// The answers are kept before the requests, so that none waits for
// GeoJS.
- server.GeoJS.Load([]lookup.Answer{
+ server.GeoJS.Load([]geojs.Answer{
keptAnswer(client, "DE"), keptAnswer(fromKP, "KP"),
})
diff --git a/internal/waf/request.go b/internal/waf/request.go
index c945d32..06bec96 100644
--- a/internal/waf/request.go
+++ b/internal/waf/request.go
@@ -11,7 +11,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/anomaly"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config"
- "sneak.berlin/go/smallwebwaf/internal/lookup"
+ "sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
@@ -42,7 +42,7 @@ type decision struct {
// looked up, whether or not an answer was there, and lookupAnswer is
// what the lookup gave then, the zero Answer while GeoJS had given none.
lookedUp bool
- lookupAnswer lookup.Answer
+ lookupAnswer geojs.Answer
// skippedLookupWait is true for a request that did not wait for GeoJS and
// found no answer kept. SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT does not apply to it,
// so that not waiting never lowers its client's limits.
@@ -81,6 +81,9 @@ type request struct {
checked time.Time
// refused is what the request was refused with, nil if it was not.
refused *refusal
+ // findings is what the heuristics of besideLookup found of the request,
+ // once waf has taken it, and nil before.
+ findings *besideLookupFindings
// requestBytes is how many bytes of the request's body have been read.
requestBytes int64
}
@@ -177,9 +180,9 @@ func requestHeaders(r *http.Request, names []string) map[string]string {
// check is the one place where a request can be refused. It returns nil to let
// the request through. The heuristics of checkClient decide: a request they
// refuse is answered with SWW_WAF_BAN_RESPONSE, 403 for a block rule or the Core
-// Rule Set, or 414 or 431 for the header limits. In observe mode nothing is refused,
-// and the log line names what enforce mode would have done. ctx is the
-// request's own context.
+// Rule Set, 503 for heuristics that ran out of time, or 414 or 431 for the
+// header limits. In observe mode nothing is refused, and the log line names
+// what enforce mode would have done. ctx is the request's own context.
func (rq *request) check(ctx context.Context) *refusal {
v := rq.checkClient(ctx)
if v == nil {
@@ -196,6 +199,11 @@ func (rq *request) check(ctx context.Context) *refusal {
case action == requestlog.ActionBlockedByBlockRule ||
action == requestlog.ActionBlockedByCoreRuleSet:
return &refusal{status: http.StatusForbidden, action: action}
+ case action == requestlog.ActionChecksTimedOut:
+ return &refusal{
+ status: http.StatusServiceUnavailable,
+ action: action,
+ }
case action == requestlog.ActionRequestLineTooLong:
return &refusal{status: http.StatusRequestURITooLong, action: action}
case action == requestlog.ActionRequestHeaderTooLarge:
@@ -208,11 +216,16 @@ func (rq *request) check(ctx context.Context) *refusal {
}
}
-// checkClient runs the heuristics of beforeLookup on the request, then looks
-// up the client's AS number and country, then runs those of afterLookup. It
-// returns the first verdict, or nil when none gives one. A client in
-// SWW_WAF_ALLOW_NETS skips them all, and is not looked up. ctx is the request's
-// own context.
+// checkClient runs the heuristics on the request, and returns the first
+// verdict, or nil when none gives one. A client in SWW_WAF_ALLOW_NETS skips them
+// all, and is not looked up. ctx is the request's own context.
+//
+// The heuristics of beforeLookup run first. Then the client's AS number and
+// country are looked up, and afterLookup runs once the lookup has its result.
+// While the lookup waits for GeoJS, besideLookup runs too, as runBesideLookup
+// says. A lookup that needs no wait is done, and afterLookup with it, before
+// besideLookup starts, so that besideLookup does no work for a request
+// afterLookup refuses.
func (rq *request) checkClient(ctx context.Context) *verdict {
if isInside(rq.client, rq.h.config.AllowNets) {
return nil
@@ -225,7 +238,28 @@ func (rq *request) checkClient(ctx context.Context) *verdict {
return v
}
- rq.lookUp(ctx, now)
+ lookedUp := rq.startLookUp(ctx, now)
+
+ select {
+ case result := <-lookedUp:
+ v := rq.runAfterLookup(ctx, result)
+ if v != nil {
+ return v
+ }
+
+ return rq.runBesideLookup(ctx, nil)
+ default:
+ return rq.runBesideLookup(ctx, lookedUp)
+ }
+}
+
+// runAfterLookup notes result, the lookup's, and runs the heuristics of
+// afterLookup. It returns the first verdict, or nil when none gives one. ctx is
+// the request's own context.
+func (rq *request) runAfterLookup(
+ ctx context.Context, result lookupResult,
+) *verdict {
+ rq.noteLookup(result)
// The lookup can wait for GeoJS, so the checks after it read the time
// again: a request counted at the time it arrived could be counted
@@ -382,9 +416,9 @@ func anomalyThresholdsSet(cfg *config.WAF) bool {
// answer about it as the request ends, and whether there is one: the
// lookup database's, which was there at once, or the one GeoJS has given
// by then, which a request does not wait for unless a setting needs it.
-func (rq *request) answerAtTheEnd() (lookup.Answer, bool) {
+func (rq *request) answerAtTheEnd() (geojs.Answer, bool) {
if !rq.lookedUp {
- return lookup.Answer{}, false
+ return geojs.Answer{}, false
}
if rq.h.config.LookupSource == "file" {
diff --git a/internal/waf/requestlog_test.go b/internal/waf/requestlog_test.go
index 241f505..871e7e3 100644
--- a/internal/waf/requestlog_test.go
+++ b/internal/waf/requestlog_test.go
@@ -20,7 +20,7 @@ const (
// requestIDHeader carries the request's id.
requestIDHeader = "X-Request-ID"
// instance is the SWW_WAF_INSTANCE_NAME a test sets.
- instance = "fsn1app1/blog"
+ instance = "host1/blog"
// ipv6Client is a client on IPv6, and ipv6Group the netblock the rate
// limits count it as.
ipv6Client = "2001:db8::7"
diff --git a/internal/waf/rulefiles.go b/internal/waf/rulefiles.go
index b3c55d2..22b211f 100644
--- a/internal/waf/rulefiles.go
+++ b/internal/waf/rulefiles.go
@@ -13,8 +13,8 @@ import (
"sneak.berlin/go/smallwebwaf/internal/rules"
)
-// ruleFiles checks the request against the rules of the rule files at now,
-// notes the ids of those it matches in the log line, adds the tag
+// ruleFiles takes the rules of the rule files the request matched, which
+// findBesideLookup found, notes their ids in the log line, adds the tag
// exploitsignature when it matches any, and refuses the request
// when the last rule it matches refuses it: with ActionBlockedByBlockRule for a
// block rule, and for a ban rule with the action for the ban it makes. A ban
@@ -31,7 +31,7 @@ func (ruleFiles) check(
rq *request,
now time.Time,
) *verdict {
- matched := rq.h.rules.Match(rq.in)
+ matched := rq.findings.rules
for _, rule := range matched {
rq.line.RuleIDs = append(rq.line.RuleIDs, rule.ID)
@@ -70,3 +70,15 @@ func (ruleFiles) check(
return nil
}
}
+
+// refusedByRule reports whether ruleFiles refuses a request that matched
+// matched: whether the last of them is a block rule or a ban rule.
+func refusedByRule(matched []rules.Rule) bool {
+ if len(matched) == 0 {
+ return false
+ }
+
+ action := matched[len(matched)-1].Action
+
+ return action == rules.ActionBlock || action == rules.ActionBan
+}
diff --git a/internal/waf/trappaths.go b/internal/waf/trappaths.go
index 5c554cc..0941500 100644
--- a/internal/waf/trappaths.go
+++ b/internal/waf/trappaths.go
@@ -11,9 +11,8 @@ import (
"sneak.berlin/go/smallwebwaf/internal/rules"
)
-// trapPaths refuses a request for a path in SWW_WAF_TRAP_PATHS: its path as a
-// path rule sees it, without the query, as sent or once percent-decoded, is one
-// of them. Such a request is a clear sign of attack, as a ban rule's match is:
+// trapPaths refuses a request for a path in SWW_WAF_TRAP_PATHS, as trapPath
+// finds it. Such a request is a clear sign of attack, as a ban rule's match is:
// it bans the client's netblock, or in observe mode raises the alert for the
// ban it would have made. The ban's notes give the path as sent, and so does
// its figure.
@@ -26,9 +25,8 @@ func (trapPaths) check(
rq *request,
now time.Time,
) *verdict {
- path := rules.Path(rq.in)
- if !slices.Contains(rq.h.config.TrapPaths, path) &&
- !slices.Contains(rq.h.config.TrapPaths, rules.DecodeOnce(path)) {
+ path := rq.findings.trapPath
+ if path == "" {
return nil
}
@@ -39,3 +37,16 @@ func (trapPaths) check(
figures: map[string]any{"path": path},
}
}
+
+// trapPath returns the request's path as sent, when its path as a path rule
+// sees it, without the query, as sent or once percent-decoded, is in
+// SWW_WAF_TRAP_PATHS, and "" otherwise.
+func (rq *request) trapPath() string {
+ path := rules.Path(rq.in)
+ if !slices.Contains(rq.h.config.TrapPaths, path) &&
+ !slices.Contains(rq.h.config.TrapPaths, rules.DecodeOnce(path)) {
+ return ""
+ }
+
+ return path
+}
diff --git a/internal/waf/waf_test.go b/internal/waf/waf_test.go
index f93f88f..93a0e15 100644
--- a/internal/waf/waf_test.go
+++ b/internal/waf/waf_test.go
@@ -1787,7 +1787,7 @@ func TestCoreRuleSetReadsTheBodyAQuestionCarries(t *testing.T) {
t.Parallel()
s := startWAF(t, "", map[string]string{
- coreRuleSetMode: block, coreRuleSetBodyInspectionSize: "64K",
+ coreRuleSetMode: block, coreRuleSetBodyInspectionSize: "8K",
})
ask := func(contentType, body string) answer {
@@ -1802,10 +1802,10 @@ func TestCoreRuleSetReadsTheBodyAQuestionCarries(t *testing.T) {
got := ask(formData, "id=1'%20OR%20'1'='1")
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
- // The first 64 KiB of a longer JSON body cannot be parsed, which is no
+ // The first 8 KiB of a longer JSON body cannot be parsed, which is no
// attack, so the Core Rule Set does not read it.
- cut := `{"text":"` + strings.Repeat("x", waf.QuestionBodyMaxBytes)
- got = ask("application/json", cut[:waf.QuestionBodyMaxBytes])
+ cut := `{"text":"` + strings.Repeat("x", 8<<10)
+ got = ask("application/json", cut)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
}
diff --git a/script/publish b/script/publish
index 614ff19..32db2c8 100755
--- a/script/publish
+++ b/script/publish
@@ -2,8 +2,9 @@
# script/publish: build the image on the Docker host and push it to the
# registry under four tags: the UTC date (YYYY-MM-DD), the short commit
# hash, the branch name and latest. The build is script/docker's, so the
-# tests and the linter run first. The host and the registry name are
-# environment variables with defaults, for running it against another.
+# tests and the linter run first. DOCKER_HOST names the Docker host and
+# PUBLISH_IMAGE the image in the registry. Both must be set; neither has
+# a default.
#
# The working tree must be clean: the hash tag names a commit, and an
# image built from uncommitted changes would carry a hash that does not
@@ -13,11 +14,16 @@ set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
-: "${DOCKER_HOST:=ssh://root@fsn1app1}"
-: "${PUBLISH_IMAGE:=git.eeqj.de/sneak/smallwebwaf}"
-export DOCKER_HOST
-
main() {
+ if [ -z "${DOCKER_HOST:-}" ]; then
+ echo "script/publish: DOCKER_HOST is not set; set it to the Docker host that builds the image, such as ssh://user@build.example.com" >&2
+ exit 1
+ fi
+ if [ -z "${PUBLISH_IMAGE:-}" ]; then
+ echo "script/publish: PUBLISH_IMAGE is not set; set it to the image name in the registry, such as registry.example.com/smallwebwaf" >&2
+ exit 1
+ fi
+
cd "$ROOT"
# Own line: a failing command substitution inside an argument does