When the app's answer arrived before the app had been sent the whole
request, SWRP_MAX_RESPONSE_TIME_MS had no timer: only the end of the
request started one. If the app then stalled, only a write could fail at
the deadline. With nothing written, the connection stayed open until the
request time limit.
modifyResponse now starts the response timer when it ends the request.
Once the timer cuts off the app's answer, nothing is left to flush, so the
last flush in finish succeeded and Go's server read the rest of the body.
finish now also ends that reading at once after a refusal of an answer
already passed on.
Model: opus-5-5
A new case fails on the current code: the app sends the first part of its
body at once and then stalls. The visitor gets that part, and the
connection stays open past SWRP_MAX_RESPONSE_TIME_MS. That is the failure
seen at random on next, where a busy host let the first part through.
The app in the two older cases sent its body after a sleep timed against
the limit, so a busy host could let it through too. The app now sends its
late parts only once rproxy has ended the request to it, which happens
when rproxy cuts off the answer. Those two cases fail on the current code
as well. The limit is 2 seconds, so that a busy host still passes on a
part sent at once.
Model: opus-5-5
For a peer outside `SWRP_TRUSTED_PROXIES`, `rproxy` now reads each header name with `_` as `-` and in any letter case, and removes the header when that name starts with `X-Forwarded-` or is `Forwarded` or `X-Real-IP`. Before, `X_Forwarded_Ssl: on` or `X_Real_IP` from a client connecting directly reached the app unchanged, and an app behind CGI or FastCGI reads them as the dash-spelt headers. A trusted peer's headers pass on as before. `SPEC.md` says so.
Model: opus-5-5
When SWRP_MAX_RESPONSE_TIME_MS ran out before the last of a short answer
was written, the flush failed. If the visitor had stopped short of its
announced body, Go's server then read the rest until
SWRP_MAX_REQUEST_TIME_MS ran out, keeping the connection open with the
answer cut off.
finish now flushes once more, and if that fails at its deadline, it ends
the reading of the body at once. ReverseProxy ignores the failed flush when
the app's answer came in one read, and panics when it was still waiting for
more; finish runs after both. Go's server never reuses a connection after a
failed write, so it closes it.
Model: opus-5-5
A new test fails on the current code. The visitor stops a little short of
its announced body, and the response time limit runs out before rproxy can
write the app's short body. The connection then stays open until the
request time limit instead of closing. The app sends that body in one part,
and in two; with two, ReverseProxy panics instead of returning.
The test that failed under load used a 300 ms response time limit, which a
busy host can use up between the app's headers and its body. It and the
test that answers a visitor that stops sending now use limits far longer
than the test's 10-second read deadline.
Model: opus-5-5
For a peer outside `SWRP_TRUSTED_PROXIES`, `rproxy` now removes every header whose name starts with `X-Forwarded-`, in any letter case, before it sets `X-Forwarded-For`, `X-Forwarded-Host` and `X-Forwarded-Proto`; `Forwarded` and `X-Real-IP` are removed as before. Until now only traefik's names were removed, so a client connecting directly could send `X-Forwarded-Ssl: on` or `X-Forwarded-Scheme: https` and make an app take a plain request for a secure one. A trusted peer's headers pass on unchanged. `SPEC.md` says so.
Model: opus-5-5
Test only. `TestAnAnswerOf1MiBGoesToTheVisitor` read an answer of 1 MiB from `waf` within the default `SWRP_TIMEOUT_MS` of 2 s. On a busy host that time ran out first, `rproxy` applied `SWRP_DEFAULT_DISPOSITION` as it should, and the test failed. It now sets `SWRP_TIMEOUT_MS` to `60000`, as the file's other tests do where that limit is not what they test.
Model: opus-5-5
Test only. The over-1 MiB cases of `TestAnAnswerThatCannotBePassedOnIsAnswered502`, and `TestAnAnswerThatCannotBePassedOnGetsNoReport`, which sends the same answer, left `SWRP_TIMEOUT_MS` at 2 s. On a loaded host `rproxy` applied `SWRP_DEFAULT_DISPOSITION` before it had read the whole answer, and the tests, which want `502`, failed. They now set `SWRP_TIMEOUT_MS` to `60000`, as the file's other tests do where that limit is not what they test.
Model: opus-5-5
The command is now `sww waf`, and the package is `internal/waf`. Types,
functions, files, tests, comments, docs, the Dockerfile, the runit
script, `script/`, compose examples, the log field `waf_line` and the
`X-From-Waf` test header follow. No behaviour changes, and no setting
changes name: rproxy already finds the waf through `SWRP_WAF_BACKEND`.
Done with one `sed -i` (Discriminator and DISCRIMINATOR to WAF,
discriminator to waf) over the files `git grep -i discriminator`
listed, as sneak allowed for this rename, then `git mv` for the
package and files. By hand afterwards: the header's spelling, which
the linter requires, and the padding after `waf` in `sww`'s usage
text.
Model: opus-5-5
An answer rproxy gives itself, such as a deny or a refusal, waited while
Go's server read up to 256 KiB of a body that had not arrived whole. A
visitor that stopped sending got nothing until SWRP_MAX_REQUEST_TIME_MS
ran out, and no bytes at all once that was past the 10 seconds the
answer has to be sent.
send now sets Connection: close in that case, as modifyResponse does for
the app's answer; both call closeAfterAnswer. requestTimedOut no longer
ends that read of Go's server, which no longer happens, so the time the
app's answer started is no longer noted.
Model: opus-5-5
When the app answered before the visitor's body had arrived whole, Go's
server read up to 256 KiB of the rest before it sent the answer. A
visitor that stopped sending got nothing until SWRP_MAX_REQUEST_TIME_MS
ran out, and no bytes at all once that was past the 10 seconds the
status has to be sent.
Such an answer now carries Connection: close, which Go sends at once.
finish first ends the transport's read of the body: once the handler
returns, Go's server reads the rest with no deadline while another read
of it waits. requestBody.Close does nothing, so that ReverseProxy's
close does not wait on the visitor.
Model: opus-5-5
A visitor could get a closed connection and no answer when
SWRP_MAX_RESPONSE_TIME_MS ran out about when the answer started: the
write deadline passed before the first byte was written.
- An answer rproxy gives itself has at least 10 seconds to be sent.
- An app's answer that arrives once the time has run out is answered
504.
- Otherwise the app's status and headers are sent at once, before the
body is read, with at least 10 seconds. The exact limit applies to
the rest.
Model: opus-5-5
rproxy passes MKCALENDAR on. nextcloud.conf allows it on
/remote.php/dav/calendars/ and PROPFIND on /.well-known/caldav and
carddav. The XML of an MKCOL there skips 930120, 932120, 932250 and
932260. That of an MKCALENDAR, and of an MKCOL on the address books'
paths, also skips 932125 and 932230, for the description. A PUT of
text/calendar or text/vcard there skips 920530: the Calendar and
Contacts apps name the charset twice when they change an event or a
contact.
Recorded from Nextcloud 35.0.2 with Calendar 6.6.5, Contacts 8.9.5 and
requests imitating DAVx5.
Model: opus-5-5
rproxy now passes PROPFIND, PROPPATCH, MKCOL, COPY, MOVE, REPORT,
SEARCH, LOCK and UNLOCK to discriminator instead of answering 405, so
Nextcloud's Files app and sync client work. The Core Rule Set's 911100
still refuses them wherever no app's exclusions allow them.
Model: opus-5-5
The first half of the naming audit: names now say what they are. `internal/waf` is `internal/coreruleset`. Log fields, the action `blocked_by_core_rule_set`, the metric and the alert event say Core Rule Set. Timings end `_ms`. Percentages say of what, as `rate_limit_percent` and `bytes_limit_percent`. Setting names follow in #234.
Judgement call: `blocked_by_core_rule_set`, not the audit's `core_rule_set_blocked`, to match `blocked_by_block_rule`.
Trips a reader: an existing state file holding `waf_blocked` or the anomaly scope `watch` stops the start and must be removed. `SWW_ALERT_EVENTS` listing `waf_block` must list `core_rule_set_match`.
Model: opus-5-5
The package passes nothing on since the one-process mode went: it answers rproxy's questions, takes its reports and answers servewaf's own endpoints. It is now named after the command it serves, with `git mv`, and its constructor is `servewaf.New`. `SPEC.md` and the other documents follow. No change of behaviour.
Model: opus-5-5
`smallwebwaf` with no command now prints the commands and exits with status 2.
The image runs rproxy, or servewaf with SMALLWEBWAF_COMMAND=servewaf. The
passing-on code, its settings and its log fields and metric go. internal/proxy
keeps the checks, answers, reports and own endpoints; its tests run through the
real rproxy. The documents describe the pair.
Judgement call: clients.json keeps token_refused, since state files refuse unknown fields.
Deviation: script/gitea-requests sets SWWAF_LOG_REQUEST_HEADERS, to label requests.
Partially verified: make example-app and make gitea-requests never built the image, as Ubuntu's snapshot service answered 5xx.
Model: opus-5-5
`smallwebwaf rproxy` reads the first 64 KiB of each body, asks servewaf over
HTTP/1.1, and passes what servewaf allows to the app within its limits. Any
other answer goes to the visitor as servewaf gave it. SWRP_DEFAULT_DISPOSITION
decides only when servewaf cannot be connected to, does not answer at all, or
does not answer within SWRP_TIMEOUT_MS. A method SPEC.md does not list gets 405.
Duplication: internal/rproxy copies passing-on code from internal/proxy until
#172 removes the originals.
Rule suppressed: gosec G704 and G705; gosec at
internal/smallwebwaf/rproxy_test.go:431; tagliatelle at
internal/requestlog/requestlog.go:203; paralleltest on one test.
Judgement call: an answer over 1 MiB, or with a status below 200, gets 502.
Judgement call: every target not starting with one / gets 400.
Model: opus-5-5
sneak
clawbot