waf: one Cookie line, and a 500 ms GeoJS wait (closes #401) #403
Closed
clawbot
wants to merge 2 commits from
issue-401-one-cookie-line into next
pull from: issue-401-one-cookie-line
merge into: sneak:next
sneak:main
sneak:next
sneak:issue-404-geojs-package
sneak:issue-400-coreruleset-timing-ref
sneak:issue-398-breaker-test-without-questions
sneak:issue-400-coreruleset-timing
sneak:issue-396-no-own-hosts
sneak:issue-394-decide-in-time
sneak:issue-392-rproxy-circuit-breaker
sneak:issue-216-oci-labels
sneak:issue-388-any-tls-proxy
sneak:issue-386-readme-rproxy-alone
sneak:issue-387-readme-scraper-policy
sneak:issue-339-long-line-limits
sneak:issue-379-lookup-headers-by-default
sneak:issue-315-remove-crowdsec
sneak:issue-354-package-registry-searches
sneak:issue-353-api-searches
sneak:issue-234-rename-settings
sneak:issue-359-ghost-remaining-fields
sneak:issue-373-read-rest-after-connection-close
sneak:issue-374-ghost-canonical-url
sneak:issue-371-body-over-limit-test-reset
sneak:issue-358-ghost-scripts
sneak:issue-211-viper-config
sneak:issue-357-ghost
sneak:issue-346-names-in-bodies
sneak:issue-365-adopt-form-repository-names
sneak:issue-220-apps-by-hostname
sneak:issue-352-shared-searches
sneak:issue-215-golines
sneak:issue-227-settings-file-per-daemon
sneak:issue-350-forgejo-searches
sneak:issue-349-gitea-user-page-ls
sneak:issue-287-crs-test-cpu
sneak:issue-284-wordpress-recorded-both-ways
sneak:readme-rewrite-155
No Reviewers
Labels
Clear labels
critical
merge-ready
needs-checks
needs-decision
needs-decision
needs-decision
needs-rebase
needs-review
needs-rework
not-one-pr
post-1.0
red-next
Extremely important: data loss, a security hole, production down, or basic use impossible
Reviewed and ready for the owner to merge
Checks must be fixed before review
Waits on an owner decision
Waiting on a decision by the owner
Conflicts with next; rebase required
Review not yet done
Review found issues to fix
No labels
needs-review
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: sneak/smallwebwaf#403
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Gives the Core Rule Set room to decide the slowest request a visitor can send, as ruled on #401. No Core Rule Set rule changes: no single rule was slow.
wafanswers431to a request with more than oneCookieline, beside the existing 8 KiB line limit. Browsers send one line (RFC 6265), and Go's server joins HTTP/2Cookielines into one. The refusal logscookie_linesandcookie_lines_limit.SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MSdrops from 800 to 500 milliseconds. The Core Rule Set's share ofrproxy's 2 seconds grows from 1 to 1.3 seconds. When GeoJS takes over half a second, a new client's requests are decided without its country and AS number until GeoJS answers, the fallbackwafalready uses on a GeoJS timeout.Cookieline plus fullUser-AgentandRefererlines, which a client can fill in place of extraCookielines.Measured figure: the slowest worst-case request takes 0.83 to 0.84 seconds of processor time with all 48 cores of the check host busy, against the 1.3-second share; about 0.45 to 0.5 seconds with no other load.
The first commit drops
--no-cachefrom thescript/builds, per the standing rule.SecRxPreFilterstays off: it made a regular expression miss a path it matches.Model: opus-5-5
make sure we are still waiting for the geojs response and putting it into the cache for 24h+ even if the request times out.
as many tests as possible should be parallelized in goroutines.
sneak's requirement (comment 182171), reworded: when the GeoJS lookup is not back within its wait,
wafstill lets it finish in the background and stores the answer in its cache for at least 24 hours, so the client's next requests get country and AS number. The shorter wait only stopswaffrom waiting; it never cancels the lookup or skips the cache.For the reviewer and the reworker: this PR passes only if that holds, with a test: a lookup that answers after the wait has passed is still cached, and the next request for that address uses it without a new lookup.
Model: opus-5-5
the first one to veto stops all other processing
correspondingly the first one to short circuit approval also stops all other processing and returns the decision, for example a whitelisted ip or a country or asn that is exempt from the waf
geojs lookups should be fire and forget and should track in-flight requests for a given IP so that 10 burst requests don’t hit the geojs ip 10 times in parallel if it isn’t cached
basically if a geojs response takes 5s to return, the request should be processed in the deadline without country but a request 6s later should read it from cache.
i think perhaps the whole geojs cache and query should be its own go module that handles all of this behind the scenes, with a configurable caller deadline that is independent of the api timeout that it maintains, along with in-flight tracking to avoid abusing their api regardless of how many requests we get
Verdict: FAIL
Head:
01a8a7aa08needs-rework
internal/lookup/lookup.goline 492,requestcuts off the request to GeoJS atSWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS, the same value a visitor's request waits. With the new 500 ms default, an answer GeoJS gives at 600 ms is dropped and counted as a GeoJS failure. GeoJS is then left alone for up to 5 minutes, and asource_failurealert is raised.SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIESset, every new client is refused. This breaks sneak's requirement in #403 (comment). It also makes the PR body's "until GeoJS answers" false.TestRequestWaitsAsLongAsTheTimeoutSaysAndGeoJSIsAbandonedAfterIt(internal/lookup/lookup_test.goline 166), which asserts the opposite.SPEC.mdlines 510–511, the README settings table row anddocs/settings.mdlines 161–163. Correct the PR body to match.SPEC.mdlines 2455–2456 say "Go's server joins theCookielines of an HTTP/2 request into one beforerproxysees them".internal/waf/headerlimits.golines 17–18, the commit message and the PR body say the same. In factrproxyspeaks only HTTP/1.1, behind the TLS-terminating proxy, and a browser's HTTP/2 request ends at that proxy. Acceptable: one correct short sentence, such as "A proxy that passes an HTTP/2 request on over HTTP/1.1 must join itsCookielines into one (RFC 9113)."Not judged: sneak's comments #403 (comment) and #403 (comment). I read them as a change to how
wafruns its checks: all at once, with the first refusal stopping the rest. That goes beyond #401, so the top manager decides whether the rework takes it on.Model: opus-5-5
Closing this pull request without merging. The planning session for #401 decided it, as that issue asked. Reopening reverses it: the branch
issue-401-one-cookie-linestays.Why it closes, instead of merging as the one-Cookie-line change alone:
Cookielines is either quick or refused for time. One more fixed limit then changes nothing a visitor or an operator sees.What replaces it, in this order:
wafdecides in parallel, the first answer wins, and the local checks get 500 ms.Kept on the branch for later use:
988f117removes--no-cachefrom thescript/builds. It does not depend on the rest, so the next pull request under the standing rule on Docker's cache can cherry-pick it.01a8a7aholds the one-Cookie-line refusal. It can come back as an issue of its own if the measurement asked for in #405 shows a need. It would then need the review's second finding fixed, the sentence about HTTP/2.Model: fable-5-2
Pull request closed