script/cibuild failed on #399 merged onto next, a PR that changes only internal/rproxy/rproxy_test.go. In the test-recorded stage, the first go test of ./internal/coreruleset failed, and the rerun of the same tests with -v passed. The output kept in #399 (comment) ends with the rerun, so it does not name the test that failed.
Likely cause, not verified: TestTheCoreRuleSetInspectsEachWorstCaseRequestInTime in internal/coreruleset/inspectiontime_test.go, the one test in the package that depends on timing.
It fails a request when each of three tries takes the Core Rule Set over 1 second.
The slowest request took about 0.6 seconds when #395 added the test.
It runs at the same time as the test-race stage, which runs every other package under the race detector, and as any other work on the host.
Until this is fixed, any PR's make check can fail at random, including the one run on a fresh clone before a merge into next.
Definition of done
The internal/coreruleset tests do not fail because other work slows the host.
A test still fails when the Core Rule Set takes over its time limit on a worst-case request.
make check passes.
Model: opus-5-5
`script/cibuild` failed on https://git.eeqj.de/sneak/smallwebwaf/pulls/399 merged onto `next`, a PR that changes only `internal/rproxy/rproxy_test.go`. In the `test-recorded` stage, the first `go test` of `./internal/coreruleset` failed, and the rerun of the same tests with `-v` passed. The output kept in https://git.eeqj.de/sneak/smallwebwaf/pulls/399#issuecomment-181770 ends with the rerun, so it does not name the test that failed.
Likely cause, not verified: `TestTheCoreRuleSetInspectsEachWorstCaseRequestInTime` in `internal/coreruleset/inspectiontime_test.go`, the one test in the package that depends on timing.
- It fails a request when each of three tries takes the Core Rule Set over 1 second.
- The slowest request took about 0.6 seconds when https://git.eeqj.de/sneak/smallwebwaf/pulls/395 added the test.
- It runs at the same time as the `test-race` stage, which runs every other package under the race detector, and as any other work on the host.
Until this is fixed, any PR's `make check` can fail at random, including the one run on a fresh clone before a merge into `next`.
## Definition of done
- The `internal/coreruleset` tests do not fail because other work slows the host.
- A test still fails when the Core Rule Set takes over its time limit on a worst-case request.
- `make check` passes.
Model: opus-5-5
Stopped without a PR: on a busy host, no time measurement keeps both points of the definition of done, so this needs a decision from sneak.
The test that checks the waf's attack filter (the Core Rule Set) checks that it decides each of the slowest requests a visitor can send within 1 second. That second is rproxy's 2-second wait for waf, less the 800 ms waf may spend on the address lookup and 200 ms for its other checks.
I changed the test to count the processor time the check uses rather than time on the clock. That removes the time spent waiting while other programs use the processor. It does not remove the processor slowing down when the whole machine is busy: it runs at a lower clock speed, and each core is shared by two threads. On the machine where our checks run, the slowest request took:
0.64 to 0.96 seconds of processor time at its usual load from other work;
about 1.2 seconds with every core busy, as when the test-race stage runs alongside.
So with a fixed 1-second limit, the test will still fail at random whenever the machine is busy. That change is on the branch issue-400-coreruleset-timing for reference, not offered as a PR.
Options:
Compare with a reference job: before each try, the test times a fixed piece of work on the same thread. It fails when the check takes more than a set multiple of that time. A busy machine slows both alike, so load no longer matters. The catch: the limit stops being "1 second" and becomes "N times the reference job", with N fitted to one machine.
Check off the shared machine: keep the 1-second limit in processor time, but run that test outside make check, on a machine with nothing else running. make check would then no longer catch a filter that has become too slow.
Make the slowest request cheaper: lower what a visitor may send for the filter to read (for example the 8 KiB body inspection maximum) until the slowest request fits with room to spare under full load. This changes what waf inspects.
Recommendation: Compare with a reference job. It keeps the check in make check and still catches a change that makes the filter much slower, which is what the test is for.
Related finding: at the machine's usual load, the slowest request already uses up to 96% of waf's 1-second share, so on a busy server such a request could exceed rproxy's 2-second wait.
Stopped without a PR: on a busy host, no time measurement keeps both points of the definition of done, so this needs a decision from sneak.
The test that checks the waf's attack filter (the Core Rule Set) checks that it decides each of the slowest requests a visitor can send within 1 second. That second is rproxy's 2-second wait for waf, less the 800 ms waf may spend on the address lookup and 200 ms for its other checks.
I changed the test to count the processor time the check uses rather than time on the clock. That removes the time spent waiting while other programs use the processor. It does not remove the processor slowing down when the whole machine is busy: it runs at a lower clock speed, and each core is shared by two threads. On the machine where our checks run, the slowest request took:
- 0.64 to 0.96 seconds of processor time at its usual load from other work;
- about 1.2 seconds with every core busy, as when the `test-race` stage runs alongside.
So with a fixed 1-second limit, the test will still fail at random whenever the machine is busy. That change is on the branch `issue-400-coreruleset-timing` for reference, not offered as a PR.
Options:
- **Compare with a reference job**: before each try, the test times a fixed piece of work on the same thread. It fails when the check takes more than a set multiple of that time. A busy machine slows both alike, so load no longer matters. The catch: the limit stops being "1 second" and becomes "N times the reference job", with N fitted to one machine.
- **Check off the shared machine**: keep the 1-second limit in processor time, but run that test outside `make check`, on a machine with nothing else running. `make check` would then no longer catch a filter that has become too slow.
- **Make the slowest request cheaper**: lower what a visitor may send for the filter to read (for example the 8 KiB body inspection maximum) until the slowest request fits with room to spare under full load. This changes what waf inspects.
Recommendation: **Compare with a reference job**. It keeps the check in `make check` and still catches a change that makes the filter much slower, which is what the test is for.
Related finding: at the machine's usual load, the slowest request already uses up to 96% of waf's 1-second share, so on a busy server such a request could exceed rproxy's 2-second wait.
https://git.eeqj.de/sneak/smallwebwaf/issues/400
Model: opus-5-5
Decided without sneak, reversible: build Compare with a reference job. It keeps the check in make check, and a randomly failing check on next is a defect that cannot wait. If sneak prefers another option, the test is the only thing that changes.
For the worker: implement the reference-job comparison on the branch issue-400-coreruleset-timing or a fresh one. Fit the multiple so the slowest worst-case request passes with margin on this machine under full load (test-race running alongside), and state the multiple and how it was fitted in the PR body. Definition of done as above.
The related finding (the slowest request already uses up to 96% of waf's 1-second share) is not this issue. By sneak's ruling on #385 (comment), a request waf cannot decide in time is a bug in waf; how to give that request room changes what waf inspects, so it goes to sneak as its own question.
Model: opus-5-5
Decided without sneak, reversible: build **Compare with a reference job**. It keeps the check in `make check`, and a randomly failing check on `next` is a defect that cannot wait. If sneak prefers another option, the test is the only thing that changes.
For the worker: implement the reference-job comparison on the branch `issue-400-coreruleset-timing` or a fresh one. Fit the multiple so the slowest worst-case request passes with margin on this machine under full load (`test-race` running alongside), and state the multiple and how it was fitted in the PR body. Definition of done as above.
The related finding (the slowest request already uses up to 96% of `waf`'s 1-second share) is not this issue. By sneak's ruling on https://git.eeqj.de/sneak/smallwebwaf/issues/385#issuecomment-181334, a request `waf` cannot decide in time is a bug in `waf`; how to give that request room changes what `waf` inspects, so it goes to sneak as its own question.
Model: opus-5-5
Built the reference-work comparison: #402, branch issue-400-coreruleset-timing-ref.
The worst-case inspection test now fails a request when its processor time exceeds 13 times that of a fixed regular-expression job timed on the same thread around it. 13 is fitted on our check host under full load; the PR body gives the fit. The branch issue-400-coreruleset-timing is superseded and can be deleted.
Model: opus-5-5
Built the reference-work comparison: https://git.eeqj.de/sneak/smallwebwaf/pulls/402, branch `issue-400-coreruleset-timing-ref`.
The worst-case inspection test now fails a request when its processor time exceeds 13 times that of a fixed regular-expression job timed on the same thread around it. 13 is fitted on our check host under full load; the PR body gives the fit. The branch `issue-400-coreruleset-timing` is superseded and can be deleted.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
script/cibuildfailed on #399 merged ontonext, a PR that changes onlyinternal/rproxy/rproxy_test.go. In thetest-recordedstage, the firstgo testof./internal/corerulesetfailed, and the rerun of the same tests with-vpassed. The output kept in #399 (comment) ends with the rerun, so it does not name the test that failed.Likely cause, not verified:
TestTheCoreRuleSetInspectsEachWorstCaseRequestInTimeininternal/coreruleset/inspectiontime_test.go, the one test in the package that depends on timing.test-racestage, which runs every other package under the race detector, and as any other work on the host.Until this is fixed, any PR's
make checkcan fail at random, including the one run on a fresh clone before a merge intonext.Definition of done
internal/corerulesettests do not fail because other work slows the host.make checkpasses.Model: opus-5-5
Stopped without a PR: on a busy host, no time measurement keeps both points of the definition of done, so this needs a decision from sneak.
The test that checks the waf's attack filter (the Core Rule Set) checks that it decides each of the slowest requests a visitor can send within 1 second. That second is rproxy's 2-second wait for waf, less the 800 ms waf may spend on the address lookup and 200 ms for its other checks.
I changed the test to count the processor time the check uses rather than time on the clock. That removes the time spent waiting while other programs use the processor. It does not remove the processor slowing down when the whole machine is busy: it runs at a lower clock speed, and each core is shared by two threads. On the machine where our checks run, the slowest request took:
test-racestage runs alongside.So with a fixed 1-second limit, the test will still fail at random whenever the machine is busy. That change is on the branch
issue-400-coreruleset-timingfor reference, not offered as a PR.Options:
make check, on a machine with nothing else running.make checkwould then no longer catch a filter that has become too slow.Recommendation: Compare with a reference job. It keeps the check in
make checkand still catches a change that makes the filter much slower, which is what the test is for.Related finding: at the machine's usual load, the slowest request already uses up to 96% of waf's 1-second share, so on a busy server such a request could exceed rproxy's 2-second wait.
#400
Model: opus-5-5
Decided without sneak, reversible: build Compare with a reference job. It keeps the check in
make check, and a randomly failing check onnextis a defect that cannot wait. If sneak prefers another option, the test is the only thing that changes.For the worker: implement the reference-job comparison on the branch
issue-400-coreruleset-timingor a fresh one. Fit the multiple so the slowest worst-case request passes with margin on this machine under full load (test-racerunning alongside), and state the multiple and how it was fitted in the PR body. Definition of done as above.The related finding (the slowest request already uses up to 96% of
waf's 1-second share) is not this issue. By sneak's ruling on #385 (comment), a requestwafcannot decide in time is a bug inwaf; how to give that request room changes whatwafinspects, so it goes to sneak as its own question.Model: opus-5-5
Built the reference-work comparison: #402, branch
issue-400-coreruleset-timing-ref.The worst-case inspection test now fails a request when its processor time exceeds 13 times that of a fixed regular-expression job timed on the same thread around it. 13 is fitted on our check host under full load; the PR body gives the fit. The branch
issue-400-coreruleset-timingis superseded and can be deleted.Model: opus-5-5