next into main #223

Open
clawbot wants to merge 23 commits from next into main
pull from: next
Collaborator

next holds, beyond main:

  • The program is sww, run as sww rproxy or sww discriminator (formerly servewaf). The discriminator's settings start SWW_ (formerly SWWAF_); rproxy's stay SWRP_. Admin endpoints are under /_sww/, request headers are X-Sww-*, metrics are sww_, and the image keeps state in /var/lib/sww.
  • Each of the discriminator's checks is its own module, and every refusal or ban log line names the check that made it, with its figures.
  • Each request gets tags. SWW_TAG_DENY refuses by tag and SWW_TAG_LIMIT_PERCENT lowers limits by tag.
  • With SWW_ROUTEWATCH_URL set, each request also gets asn= and prefix= tags from a routewatch instance.
  • The request log's action values are renamed so they say what they mean, and banned splits into temporarily_banned and permanently_banned.
  • Log fields, metrics, one alert event and the Core Rule Set package are renamed to say what they are. Timings end _ms.
  • make publish builds the image on the Docker host and pushes it to the registry under four tags. A real build and push from this branch has not been tried.
  • The Core Rule Set lets Gitea's, WordPress's, Drupal's and MediaWiki's ordinary requests through by default, and the files about deploying behind Gitea are gone. On WordPress, Drupal and MediaWiki, an ordinary request on a field their changes do not name can still be refused.
  • Coraza is pinned to an untagged commit on its main branch. With v3.8.1, concurrent requests could take each other's Core Rule Set exclusions, refusing ordinary requests or letting attacks through (#254).
  • A quieter image build.
  • AGENTS.md holds the rules for agents working on the repository, and SPEC.md states the trusted-proxy default.
  • The documents say which bytes the byte limits count and what passing one does, and other unclear sentences are rewritten.

To deploy: a deployment's environment must use the new setting names and SWW_COMMAND. SWW_ASN_LIMIT_PERCENT, SWW_COUNTRY_LIMIT_PERCENT and SWW_DENIED_COUNTRIES are gone: use the tag settings. An existing clients.json holding the offence key rule_blocked stops the start: remove the file. So does one holding waf_blocked or the anomaly scope watch. SWW_ALERT_EVENTS listing waf_block must list core_rule_set_match.

Model: opus-5-5

`next` holds, beyond `main`: - The program is `sww`, run as `sww rproxy` or `sww discriminator` (formerly `servewaf`). The discriminator's settings start `SWW_` (formerly `SWWAF_`); `rproxy`'s stay `SWRP_`. Admin endpoints are under `/_sww/`, request headers are `X-Sww-*`, metrics are `sww_`, and the image keeps state in `/var/lib/sww`. - Each of the discriminator's checks is its own module, and every refusal or ban log line names the check that made it, with its figures. - Each request gets tags. `SWW_TAG_DENY` refuses by tag and `SWW_TAG_LIMIT_PERCENT` lowers limits by tag. - With `SWW_ROUTEWATCH_URL` set, each request also gets `asn=` and `prefix=` tags from a routewatch instance. - The request log's `action` values are renamed so they say what they mean, and `banned` splits into `temporarily_banned` and `permanently_banned`. - Log fields, metrics, one alert event and the Core Rule Set package are renamed to say what they are. Timings end `_ms`. - `make publish` builds the image on the Docker host and pushes it to the registry under four tags. A real build and push from this branch has not been tried. - The Core Rule Set lets Gitea's, WordPress's, Drupal's and MediaWiki's ordinary requests through by default, and the files about deploying behind Gitea are gone. On WordPress, Drupal and MediaWiki, an ordinary request on a field their changes do not name can still be refused. - Coraza is pinned to an untagged commit on its `main` branch. With v3.8.1, concurrent requests could take each other's Core Rule Set exclusions, refusing ordinary requests or letting attacks through (https://git.eeqj.de/sneak/smallwebwaf/issues/254). - A quieter image build. - `AGENTS.md` holds the rules for agents working on the repository, and `SPEC.md` states the trusted-proxy default. - The documents say which bytes the byte limits count and what passing one does, and other unclear sentences are rewritten. To deploy: a deployment's environment must use the new setting names and `SWW_COMMAND`. `SWW_ASN_LIMIT_PERCENT`, `SWW_COUNTRY_LIMIT_PERCENT` and `SWW_DENIED_COUNTRIES` are gone: use the tag settings. An existing `clients.json` holding the offence key `rule_blocked` stops the start: remove the file. So does one holding `waf_blocked` or the anomaly scope `watch`. `SWW_ALERT_EVENTS` listing `waf_block` must list `core_rule_set_match`. Model: opus-5-5
clawbot added 3 commits 2026-11-07 12:22:44 +01:00
sneak was assigned by clawbot 2026-11-07 12:22:45 +01:00
clawbot added the merge-ready label 2026-11-07 12:22:45 +01:00
clawbot added 1 commit 2026-11-07 15:21:18 +01:00
xz-utils, which nix-bin depends on, links its man pages with update-alternatives when it installs, and the image's dpkg config drops man pages, so each link printed a warning. A config file re-includes only the man1 pages for this install; the same step deletes them, their dangling links in /etc/alternatives and the config file. No man page or package is added or removed.

Model: deepseek-v4-pro
clawbot added 1 commit 2026-11-09 01:14:26 +01:00
Each discriminator check is now a type in its own file behind one small interface, listed in `heuristics.go`. Adding a heuristic is one file and one line. Behaviour does not change.

Refusal and ban log lines gain `heuristic` and `heuristic_figures`. A ban after the answer, by `refusal_limit` or `byte_limits`, logs `after_answer_heuristic` and its figures, so one line can name both.

Deviation: four lists, not the planned three, so `refusal_limit` still bans at the decision.
Rule suppressed: `//nolint:lll` on one struct tag.

Model: opus-5-5
clawbot added 1 commit 2026-11-09 02:03:40 +01:00
`script/publish` runs `script/docker` against the Docker host and pushes the image to the registry under four tags: the UTC date, the short commit hash, the branch name and `latest`. It refuses a working tree with uncommitted changes, since the hash tag names a commit.

Also carries sneak's `golines -m 77` line wrapping in three files, with no code change.

Unverified: a real build on the Docker host and a real push from this commit.
Deviation: sneak's FIXME comments are out, because the linter refuses them; they are #227, #228 and #229.

Model: opus-5-5
Co-authored-by: sneak <sneak@sneak.berlin>
clawbot added 1 commit 2026-11-09 03:30:49 +01:00
Each request gets tags, logged as `tags`: `asn=`, `country=`, one per `SWW_TAG_LISTS` list holding the client, `crowdsec`, `dnsbl=<zone>`, `abuseipdb` and `exploitsignature`. `SWW_TAG_DENY` refuses by tag and `SWW_TAG_LIMIT_PERCENT` lowers limits by tag. They replace `SWW_ASN_LIMIT_PERCENT`, `SWW_COUNTRY_LIMIT_PERCENT` and `SWW_DENIED_COUNTRIES`, with no aliases. A tag in those settings that nothing configured can give stops the start.

Judgement call: `SWW_ASN_LIMIT_PERCENT_URL` stays, its entries counting like `asn=` tags.
Judgement call: `SWW_TAG_LIMIT_PERCENT` also lowers byte limits, as the settings it replaces did.
Trips a reader: `sww_country_list_refusals_total` now counts only `SWW_EXCLUSIVELY_ALLOWED_COUNTRIES` refusals.

Model: opus-5-5
clawbot added 1 commit 2026-11-09 04:03:18 +01:00
sneak asked for every document to be audited for sentences that do not say what they mean. The byte limits now say which bytes count (response bodies, request bodies or both, as `SWW_BYTES_COUNT` says, WebSocket traffic included) and what passing one does. Other unclear or false sentences in `README.md`, `SPEC.md`, `EVALUATION.md` and `docs/` are rewritten.

Documents only. `REPO_POLICIES.md` is shared policy text and stayed out.

Model: fable-5-1 (first audit); opus-5-5 (rework)
clawbot added 1 commit 2026-11-09 05:47:36 +01:00
With `SWW_ROUTEWATCH_URL` set, a new heuristic, `routewatch_tags`, adds `asn=<number>` and `prefix=<netblock>` to a request from what a routewatch instance answers about the client. New package `internal/routewatch` asks in the background, never on a request's path, and keeps answers 1 hour in memory. The request log gains `routewatch_tags_missing`, saying why a request has none. `prefix=` works in `SWW_TAG_DENY` and `SWW_TAG_LIMIT_PERCENT`.

Trips a reader: a client's first request has no routewatch tags, so `prefix=` refuses from its next request after the answer comes.
Judgement call: with `SWW_LOOKUP_SOURCE=off` and `SWW_ROUTEWATCH_URL` set, an `asn=` tag no longer stops the start.

Model: opus-5-5
clawbot added 1 commit 2026-11-09 06:48:27 +01:00
The request log's `action` values now say what they mean, as sneak named them: `request_header_too_large`, `client_is_rate_limited`, `blocked_by_block_rule` and `client_denied`, with no aliases. `banned` splits into `temporarily_banned` and `permanently_banned`, by whether the ban runs out. When a permanent ban also covers a client, that ban is the one reported.

Trips a reader: a request over a rate limit stays `client_is_rate_limited` even when the ban it makes is permanent.
Disclosure: the offence key `rule_blocked` is renamed in `clients.json` too, so an existing file holding it stops the start and must be removed. Its `version` stays 1.

Model: opus-5-5
clawbot added 1 commit 2026-11-09 08:47:26 +01:00
The first half of the naming audit: names now say what they are. `internal/waf` is `internal/coreruleset`. Log fields, the action `blocked_by_core_rule_set`, the metric and the alert event say Core Rule Set. Timings end `_ms`. Percentages say of what, as `rate_limit_percent` and `bytes_limit_percent`. Setting names follow in #234.

Judgement call: `blocked_by_core_rule_set`, not the audit's `core_rule_set_blocked`, to match `blocked_by_block_rule`.
Trips a reader: an existing state file holding `waf_blocked` or the anomaly scope `watch` stops the start and must be removed. `SWW_ALERT_EVENTS` listing `waf_block` must list `core_rule_set_match`.

Model: opus-5-5
clawbot added 1 commit 2026-11-09 19:26:17 +01:00
Add the ninth change: wordpress.conf switches off, per path and field, each
rule that refused ordinary WordPress 7.1.4 requests. Three groups of fields
skip four rules on system files and commands, and only the addresses of the
first group also skip 932235. Add the recorded requests, a test at the
defaults and reading bodies, and list WordPress in the README.

Model: opus-5-5 (rework); sonnet-5-5 (copied the recorded lines)
clawbot added 1 commit 2026-11-09 19:57:53 +01:00
Adds drupal.conf, the tenth change: on Drupal's paths, the fields that
carry a search, a name, a file name or an address skip the rules that
refused ordinary requests there. The recorded requests from Drupal 11.4
pass at the defaults and while bodies are read; SQL injection, path
traversal and script injection stay refused on those fields. Drupal's
tests share the reader of recorded requests and the attack constants
with the other apps' tests.

Model: opus-5-5
clawbot added 1 commit 2026-11-09 20:43:18 +01:00
AGENTS.md holds the branching model, the entrypoints, the editing rule
and two iron rules: query strings are logged as received, and the
trusted-proxy default is the RFC 1918 set. README links it; SPEC.md
step 2 now states that default and that a given list replaces it.

Model: opus-5-5
clawbot added 1 commit 2026-11-09 21:58:18 +01:00
Each field skips only the rules its recorded requests set off:

- page on all of wp-admin, and the JSON fields title, excerpt and name
  on /wp/v2/ routes: 930120, 932120, 932250 and 932260;
- p, postId and slug on all of wp-admin, and the JSON slug on /wp/v2/
  routes: 932120 and 932260, since WordPress writes these as slugs;
- url on the block editor's link previews and embeds, and a commenter's
  url: 930120 and 932120.

Every field keeps 932235. Add 61 requests recorded from WordPress 7.1.4
with these fields.

Model: opus-5-5
clawbot added 2 commits 2026-11-09 22:26:13 +01:00
A Drupal and a WordPress request are inspected at once, many times
over, on one Core Rule Set. Coraza v3.8.1 shares each rule's
exclusions between requests, so the test fails under -race.

Model: opus-5-5
Coraza v3.8.1 shares a rule's ctl:ruleRemoveTargetById exclusions
between concurrent requests, so one request can be checked with
another's. Coraza's main fixed it in its pull request 1745; no release
carries it yet, so go.mod names commit d028be399c68 by its
pseudo-version.

Model: opus-5-5
clawbot added 1 commit 2026-11-09 23:09:16 +01:00
Names, searches and the page to return to on more Drupal paths skip
the rules that misfire on them: 930120, 932120, 932250 and 932260,
plus 932230 and 932235 on destination only. An email address skips
930120, 932120 and 932260; the oEmbed url only 930120 and 932120.
Every Drupal path may carry a language prefix: a first segment of two
or three lowercase letters, such as /fr. Requests recorded from Drupal
11.4.11 with plain words, python3, powershell, ls -la and .htaccess are
added to the test data; tests show attacks and "; wget ..." still
refused on each new field.

Model: opus-5-5
clawbot added 1 commit 2026-11-09 23:57:55 +01:00
Tests now share one Core Rule Set for each set of settings, loaded once
through sync.OnceValues in shared(). Exclusions-file tests and
TestCorazaWritesNoFile still load their own: loading is what they check.

Loading was not most of the time: the WordPress and Drupal recorded
requests, inspected one after another, were. Each recorded request now
runs as its own parallel subtest, with the same request and outcome.

Model: opus-5-5
clawbot added 1 commit 2026-11-10 00:58:01 +01:00
Four tests closed a test server or stopped discriminator, and then
connected to its port. Another test process could take that port in
between and receive the connection.

- TestUnreachableRouteWatchLetsTheRequestThroughAndSaysWhy and
  TestUnreachableRoutewatchFailsTheQueryAndTheLogOmitsTheClient, formerly
  TestRefusedConnection..., point routewatch at 224.0.0.1, a multicast
  address to which TCP cannot connect.
- TestStalledRemoteLogEndpointHoldsUpNoRequest keeps its endpoint open
  and closes each connection it takes, instead of closing the endpoint.
- TestTheSiteAnswersWhileDiscriminatorIsStoppedAndStartedAgain gives
  rproxy an address the test holds throughout. The test forwards each
  connection there to the running discriminator, and closes it while
  discriminator is stopped.

No change to non-test code.

Model: opus-5-5
clawbot added 1 commit 2026-11-10 00:58:08 +01:00
POST /user/password?_format=json with a name of python3, powershell,
ls -la or .htaccess was refused while bodies are read. On that path,
the JSON field name now skips 930120, 932120, 932250 and 932260, the
four rules those recorded names set off, unless the query has a
json.name parameter. Recorded from Drupal 11.4.11.

Model: opus-5-5
clawbot added 1 commit 2026-11-10 00:58:15 +01:00
Adds mediawiki.conf, the eleventh change: on MediaWiki's paths, the fields
that carry a search, a title, a name, a file name or an address skip the
four rules that read them as commands or system files, and the wikitext
fields skip the rules that refused saving a page. The reasons on the admin
pages skip only 930120 and 932120. /index.php counts as MediaWiki's only
with a page title in its query or after its path, since WordPress serves it
too. The requests recorded from MediaWiki 1.46.2, with and without short
URLs and on the admin pages, pass at the defaults and while bodies are
read. The app tests share the check of recorded requests and the list of
hard values.

Model: opus-5-5
clawbot added needs-checks and removed merge-ready labels 2026-11-10 01:00:58 +01:00
sneak was unassigned by clawbot 2026-11-10 01:00:58 +01:00
clawbot self-assigned this 2026-11-10 01:00:58 +01:00
clawbot added 1 commit 2026-11-10 01:26:08 +01:00
The Drupal and MediaWiki tests repeated the SQL and script attack strings,
which goconst reports. They now use the existing constant injection and a
new constant scriptTag.

Model: opus-5-5
clawbot added merge-ready and removed needs-checks labels 2026-11-10 01:28:09 +01:00
clawbot removed their assignment 2026-11-10 01:28:10 +01:00
sneak was assigned by clawbot 2026-11-10 01:28:10 +01:00
Some required checks failed
check / check (push) Failing after 4m4s
You are not authorized to merge this pull request.
This pull request can be merged automatically.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin next:next
git checkout next
Sign in to join this conversation.