waf decides in parallel: the GeoJS wait and the local checks overlap, the first answer wins, and the local checks get 500 ms #405

Open
opened 2026-11-16 23:54:45 +01:00 by clawbot · 1 comment
Collaborator

Split from #401. Second of three.

Start only once #404 (the GeoJS lookup as a package of its own) is merged into next. This issue uses that package's separate caller wait.

What sneak asked for

Reworded from his comments:

  • Time limits run in parallel and never add up. As much of the deciding as possible happens at the same time (#401 (comment)).
  • The first check to refuse stops all other processing (#403 (comment)).
  • The first check to approve outright also stops all other processing, and its approval is returned. His examples are an allow-listed address, and a country or AS number that is exempt from waf (#403 (comment)).
  • The checks that only compute on the request get 500 ms. Taking longer is a failure, since no request should be that hard to interpret (#401 (comment)).
  • Earlier: every request a visitor can send is decided within the time rproxy waits. A visitor able to deny service is a bug in waf (#385 (comment)).

Where the code is today

checkClient in internal/waf/request.go does everything one after another:

  1. the checks of beforeLookup;
  2. the wait for GeoJS, up to 800 ms;
  3. the checks of afterLookup, which end with the Core Rule Set. It takes up to about a second on the slowest request a visitor can send.

So the two waits add up. They are the only two things in a decision that take time. Every other check reads the request or waf's memory and takes microseconds.

One fact shapes the design. Coraza cannot stop an inspection part-way. The pinned version carries a context on each transaction, but its rule evaluation never looks at it. So "stopping" the Core Rule Set can only mean that waf stops waiting and drops the result. The inspection ends by itself, bounded by the fixed size limits in "Answering in time" of SPEC.md.

To build

  1. The checks of beforeLookup stay first, one after another, in today's order: header limits, tag lists, routewatch tags, SWW_WAF_DENY_NETS, bans. A request they refuse starts no lookup wait and no inspection, as now. Otherwise a banned client could keep a processor busy with inspections nobody waits for.
  2. Then two things start at the same time:
    • the lookup, and after it the checks that follow it today up to the rate limits, in today's order: country list, blocklists, DNS blocklists, AbuseIPDB, tag refusals, rate limits;
    • the local checks, in today's order: trap paths, rule files, the Core Rule Set.
  3. The first of the two to refuse ends the request. waf answers at once and does not wait for the other. The lookup itself goes on and its answer is kept.
  4. If neither refuses, the tag refusals are checked the second time, as now, and the request is allowed.
  5. The local checks have 500 ms together, from their start. The limit is fixed and has no setting. If they are not done by then, waf refuses the request:
    • the answer is deny with status 503;
    • the log line has an action of its own, and its heuristic names the check that had not finished;
    • it makes no ban, is no offence and does not count toward the refusal limit;
    • it holds whatever SWW_WAF_CORE_RULE_SET_MODE says;
    • in observe mode it is logged as would_action and the request is allowed, like every refusal.
  6. A check that was stopped, or whose result came after the answer, leaves no trace. It counts nothing, bans nothing, raises no alert, counts no metric and adds nothing to the log line.
  7. When both would refuse, the one that finishes first is the one answered and logged. Nothing else about a decision may depend on which finished first.

Keep as it is:

  • A client in SWW_WAF_ALLOW_NETS is approved before every check, and nothing else starts. That is the only outright approval waf has. No setting exempts a country or an AS number from waf, and this issue adds none.
  • The requests that do not wait for GeoJS today (see "Answering in time") still do not.
  • In observe mode the first refusal still ends the checks and is logged as what would have happened.

Do not patch Coraza, and build nothing to stop an inspection.

The timing test

TestTheCoreRuleSetInspectsEachWorstCaseRequestInTime in internal/coreruleset/inspectiontime_test.go checks that the slowest request a visitor can send is inspected within about 1 second of processor time.

  • Leave the test as it is. Change only what its comment and "Answering in time" say it is for. It no longer shows that the slowest request is decided in time: the 500 ms limit does that. It shows that an inspection nobody waits for ends within about a second, so a visitor cannot keep a processor busy for longer with one request.
  • Measure once, for the PR body: how long the slowest of the apps' recorded requests in internal/coreruleset takes to inspect, at paranoia level 4 with the most of a body read. Give it with no other load and with every core busy. Commit no new timing test for it.
  • If a recorded request needs over 500 ms, do not raise the limit and do not drop the request. Say so on this issue and stop. A real request that slow is sneak's to weigh.

Done when

Each point has a test. Hold the lookup and the local checks with stand-ins, under testing/synctest where time matters, so no test compares real work with the clock.

  • GeoJS never answers and a local check takes its whole 500 ms. waf answers after the longer of the two limits, not after their sum.
  • A request the local checks refuse is answered at once while GeoJS has not answered. The rate limits have not counted it.
  • A request the country list or the rate limits refuse is answered as soon as the lookup answers, while the Core Rule Set is still inspecting. Its log line has no Core Rule Set fields. No Core Rule Set metric is counted and no alert is raised, also after the inspection ends.
  • A request refused by a check of beforeLookup starts no lookup wait and no inspection. The tests of this in internal/waf/answertime_test.go still hold.
  • Local checks held past 500 ms: the request gets 503, the log line has the new action, and there is no ban and no offence. In observe mode the request is allowed with would_action.
  • A client in SWW_WAF_ALLOW_NETS is allowed at once and nothing else starts.
  • The tests cover both orders of finishing. They are in internal/waf, which the race detector checks.
  • Docs match:
    • SPEC.md: "What happens to a request", "The answer" and "Answering in time";
    • docs/request-log.md: the new action;
    • README.md and docs/settings.md, wherever they give the order of the checks or these times.
  • make check passes.

Readings taken

Each is mine, not sneak's ruling, and each is cheap to reverse.

  • A request whose local checks pass 500 ms is refused. It is not left without an answer, and not let through.
    • Why: his earlier ruling requires an answer in time. Letting it through would let a visitor skip the attack filter by sending a request that is slow to inspect.
    • The catch: time on the clock also depends on load. While waf's host is so busy that an ordinary request takes over 500 ms, that request is refused too. Today it would get SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION after 2 seconds.
    • Reversing it is one branch.
  • Status 503, no ban, no offence. rproxy already answers 503 when waf gives no answer in time and the setting is deny. A ban would turn an overloaded host into bans of ordinary visitors. Reversing it is one constant and two lines.
  • "As much as possible at the same time" is read as the two things that take time. The instant checks stay first, for the reason in step 1.
  • The 500 ms also hold while the Core Rule Set is in detect mode, where a match refuses nothing. The limit is on the local checks as a whole, not a finding of the Core Rule Set.

Not checked

  • That Coraza cannot stop an inspection part-way is read from the source of the pinned version, not tested.
  • How long real requests take against 500 ms. No test was started while writing this issue. The measurement above settles it.

Model: fable-5-2

Split from https://git.eeqj.de/sneak/smallwebwaf/issues/401. Second of three. Start only once https://git.eeqj.de/sneak/smallwebwaf/issues/404 (the GeoJS lookup as a package of its own) is merged into `next`. This issue uses that package's separate caller wait. ## What sneak asked for Reworded from his comments: - Time limits run in parallel and never add up. As much of the deciding as possible happens at the same time (https://git.eeqj.de/sneak/smallwebwaf/issues/401#issuecomment-182196). - The first check to refuse stops all other processing (https://git.eeqj.de/sneak/smallwebwaf/pulls/403#issuecomment-182174). - The first check to approve outright also stops all other processing, and its approval is returned. His examples are an allow-listed address, and a country or AS number that is exempt from `waf` (https://git.eeqj.de/sneak/smallwebwaf/pulls/403#issuecomment-182175). - The checks that only compute on the request get 500 ms. Taking longer is a failure, since no request should be that hard to interpret (https://git.eeqj.de/sneak/smallwebwaf/issues/401#issuecomment-182195). - Earlier: every request a visitor can send is decided within the time `rproxy` waits. A visitor able to deny service is a bug in `waf` (https://git.eeqj.de/sneak/smallwebwaf/issues/385#issuecomment-181334). ## Where the code is today `checkClient` in `internal/waf/request.go` does everything one after another: 1. the checks of `beforeLookup`; 2. the wait for GeoJS, up to 800 ms; 3. the checks of `afterLookup`, which end with the Core Rule Set. It takes up to about a second on the slowest request a visitor can send. So the two waits add up. They are the only two things in a decision that take time. Every other check reads the request or `waf`'s memory and takes microseconds. One fact shapes the design. Coraza cannot stop an inspection part-way. The pinned version carries a context on each transaction, but its rule evaluation never looks at it. So "stopping" the Core Rule Set can only mean that `waf` stops waiting and drops the result. The inspection ends by itself, bounded by the fixed size limits in "Answering in time" of `SPEC.md`. ## To build 1. The checks of `beforeLookup` stay first, one after another, in today's order: header limits, tag lists, routewatch tags, `SWW_WAF_DENY_NETS`, bans. A request they refuse starts no lookup wait and no inspection, as now. Otherwise a banned client could keep a processor busy with inspections nobody waits for. 2. Then two things start at the same time: - the lookup, and after it the checks that follow it today up to the rate limits, in today's order: country list, blocklists, DNS blocklists, AbuseIPDB, tag refusals, rate limits; - the local checks, in today's order: trap paths, rule files, the Core Rule Set. 3. The first of the two to refuse ends the request. `waf` answers at once and does not wait for the other. The lookup itself goes on and its answer is kept. 4. If neither refuses, the tag refusals are checked the second time, as now, and the request is allowed. 5. The local checks have 500 ms together, from their start. The limit is fixed and has no setting. If they are not done by then, `waf` refuses the request: - the answer is `deny` with status `503`; - the log line has an action of its own, and its `heuristic` names the check that had not finished; - it makes no ban, is no offence and does not count toward the refusal limit; - it holds whatever `SWW_WAF_CORE_RULE_SET_MODE` says; - in `observe` mode it is logged as `would_action` and the request is allowed, like every refusal. 6. A check that was stopped, or whose result came after the answer, leaves no trace. It counts nothing, bans nothing, raises no alert, counts no metric and adds nothing to the log line. 7. When both would refuse, the one that finishes first is the one answered and logged. Nothing else about a decision may depend on which finished first. Keep as it is: - A client in `SWW_WAF_ALLOW_NETS` is approved before every check, and nothing else starts. That is the only outright approval `waf` has. No setting exempts a country or an AS number from `waf`, and this issue adds none. - The requests that do not wait for GeoJS today (see "Answering in time") still do not. - In `observe` mode the first refusal still ends the checks and is logged as what would have happened. Do not patch Coraza, and build nothing to stop an inspection. ## The timing test `TestTheCoreRuleSetInspectsEachWorstCaseRequestInTime` in `internal/coreruleset/inspectiontime_test.go` checks that the slowest request a visitor can send is inspected within about 1 second of processor time. - Leave the test as it is. Change only what its comment and "Answering in time" say it is for. It no longer shows that the slowest request is decided in time: the 500 ms limit does that. It shows that an inspection nobody waits for ends within about a second, so a visitor cannot keep a processor busy for longer with one request. - Measure once, for the PR body: how long the slowest of the apps' recorded requests in `internal/coreruleset` takes to inspect, at paranoia level 4 with the most of a body read. Give it with no other load and with every core busy. Commit no new timing test for it. - If a recorded request needs over 500 ms, do not raise the limit and do not drop the request. Say so on this issue and stop. A real request that slow is sneak's to weigh. ## Done when Each point has a test. Hold the lookup and the local checks with stand-ins, under `testing/synctest` where time matters, so no test compares real work with the clock. - GeoJS never answers and a local check takes its whole 500 ms. `waf` answers after the longer of the two limits, not after their sum. - A request the local checks refuse is answered at once while GeoJS has not answered. The rate limits have not counted it. - A request the country list or the rate limits refuse is answered as soon as the lookup answers, while the Core Rule Set is still inspecting. Its log line has no Core Rule Set fields. No Core Rule Set metric is counted and no alert is raised, also after the inspection ends. - A request refused by a check of `beforeLookup` starts no lookup wait and no inspection. The tests of this in `internal/waf/answertime_test.go` still hold. - Local checks held past 500 ms: the request gets `503`, the log line has the new action, and there is no ban and no offence. In `observe` mode the request is allowed with `would_action`. - A client in `SWW_WAF_ALLOW_NETS` is allowed at once and nothing else starts. - The tests cover both orders of finishing. They are in `internal/waf`, which the race detector checks. - Docs match: - `SPEC.md`: "What happens to a request", "The answer" and "Answering in time"; - `docs/request-log.md`: the new action; - `README.md` and `docs/settings.md`, wherever they give the order of the checks or these times. - `make check` passes. ## Readings taken Each is mine, not sneak's ruling, and each is cheap to reverse. - A request whose local checks pass 500 ms is refused. It is not left without an answer, and not let through. - Why: his earlier ruling requires an answer in time. Letting it through would let a visitor skip the attack filter by sending a request that is slow to inspect. - The catch: time on the clock also depends on load. While `waf`'s host is so busy that an ordinary request takes over 500 ms, that request is refused too. Today it would get `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` after 2 seconds. - Reversing it is one branch. - Status `503`, no ban, no offence. `rproxy` already answers `503` when `waf` gives no answer in time and the setting is `deny`. A ban would turn an overloaded host into bans of ordinary visitors. Reversing it is one constant and two lines. - "As much as possible at the same time" is read as the two things that take time. The instant checks stay first, for the reason in step 1. - The 500 ms also hold while the Core Rule Set is in `detect` mode, where a match refuses nothing. The limit is on the local checks as a whole, not a finding of the Core Rule Set. ## Not checked - That Coraza cannot stop an inspection part-way is read from the source of the pinned version, not tested. - How long real requests take against 500 ms. No test was started while writing this issue. The measurement above settles it. Model: fable-5-2
Author
Collaborator

Not started: this issue waits on the GeoJS lookup package (#404), which is still open with no PR merged into next.

Model: opus-5-5

Not started: this issue waits on the GeoJS lookup package (https://git.eeqj.de/sneak/smallwebwaf/issues/404), which is still open with no PR merged into `next`. Model: opus-5-5
clawbot added the needs-decision label 2026-11-17 00:02:11 +01:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Reference: sneak/smallwebwaf#405