Compare commits

...
Author SHA1 Message Date
clawbot eca8f4d8cc waf: the server's types and constructor in waf.go, active_ban in its own file (closes #435)
check / check (push) Waiting to run
Moves only. waf.go now holds the package comment, Params, Server, New,
newHandler, handler and ServeHTTP. The code that answers questions and
takes reports moves to questionsandreports.go, and activeBan to
activeban.go. refusedWhateverTheAnswer moves to lookup.go, beside its
only caller. proxy.go is gone. The only changed lines are imports and
the file comment of bans.go, split between it and activeban.go.

Model: opus-5-5
2026-11-20 01:54:31 +00:00
clawbot 4774518cb3 waf: a verdict carries the status the visitor gets (closes #443)
check / check (push) Waiting to run
Each heuristic that refuses a request sets the status on its verdict.
check copies it into the refusal, or in observe mode notes it, and no
longer works the status out from the action. refusalLimit takes the
status from that verdict instead of from a switch over the action.

Model: opus-5-5
2026-11-20 02:42:08 +01:00
clawbot 5d5e9542f8 coreruleset: embed the apps' fixes as one directory (closes #430)
check / check (push) Waiting to run
The nine app files move to internal/coreruleset/apps/ and are embedded
once as appFixes. An app's name is its file's name; the per-app
variables go, and apps keeps only gitea's and Forgejo's cookies and
forkOf. Comments and SPEC name each of sww's changes to the Core Rule
Set by what it does instead of by number. The directives Coraza gets
are unchanged.

Model: opus-5-5
2026-11-20 02:13:14 +01:00
clawbot 1b9ca94774 Hold Go's server privately in waf.Server and rproxy.Server (closes #437)
check / check (push) Failing after 1m46s
Neither Server embeds *http.Server any more. Each holds it as a private
field, is its handler, and has Serve, Shutdown, Close and ServeHTTP.
sww's serve hook takes the new daemonServer interface of those first
three, and fx provides each Server as it.

Model: opus-5-5
2026-11-20 01:58:25 +01:00
clawbot 9afbd1b683 config: each waf setting's code in one place (closes #449)
check / check (push) Failing after 3m59s
internal/config/waf.go is reordered by setting, in the order of the
settings sections of SPEC and docs/settings.md. The struct WAF and
WAFFromEnvironment stay first. Each setting's reader, parser, errors
and check sit together. A helper several settings share sits before
the first of them.

Moves only. The one error block and the one block of constants are
split into a block beside each user, so gofmt re-aligns the = of some
of those lines.

Model: opus-5-5
2026-11-20 01:44:08 +01:00
clawbot cc77435f2a rproxy: one type for the goroutines sending reports (closes #433)
check / check (push) Successful in 4m31s
handler held a WaitGroup, a mutex and a stopped flag for the reports,
and the rule that no report starts once waiting has begun was split
between sendReport and Server.WaitForReports. reportGoroutines now
holds all three, and its two methods are the only code that touches
the lock and the flag.

Model: opus-5-5
2026-11-20 01:21:06 +01:00
clawbot dcfdd12c51 waf: Server exports no parts (closes #436)
check / check (push) Successful in 4m9s
waf.Server held its parts as public fields only for the tests to read.
The tests now keep the Params the server was built from in a test-only
wafAndParts, which embeds both, so every assertion reads as before.
newWAFWith takes a change to the Params instead of a build function, so
the Params kept are the ones the server was built from.

Model: opus-5-5
2026-11-20 00:57:04 +01:00
clawbot 2301a76029 Give ban causes, limit kinds, alert events, actions and dispositions named types (closes #428)
check / check (push) Failing after 1m41s
Each fixed set of string constants becomes a named string type in the
package that owns it, and the fields and parameters that carry it use
the type: bans.Cause, ratelimit.LimitKind, alerts.Event,
alerts.DestinationName, rules.Action, requestlog.Action,
requestlog.Offence, requestlog.WAFLineKind and wafprotocol.Disposition.
Every value keeps its string, so the logs, state files, alerts and
metrics do not change.

The switches exhaustive now checks list their missing cases.

Model: opus-5-5
2026-11-20 00:43:20 +01:00
clawbot abf6c8d517 rproxy: the two time limits get a type and a file of their own (closes #432)
check / check (push) Failing after 3m36s
timeLimits in internal/rproxy/timelimits.go owns the mutex, the two
timers and the end of the request; request holds one field of it. It
is given refuse, whether the visitor's body is being waited on, and a
function that ends that read. Tests of the type alone run in a
synctest bubble.

Model: opus-5-5
2026-11-20 00:28:55 +01:00
clawbot 17400f68b8 waf: heuristics pass facts through typed fields of decision (closes #442)
check / check (push) Failing after 1m35s
The tags, what observe mode would have done, the broken limit and the
three reputation flags are fields of decision, each naming what sets it.
The log line is written from them and from the lookup answer, and no
heuristic, ban note or alert reads it back. A ban's notes take the
request's method, Host and User-Agent from decision, since the report's
steps have no request to read them from.

Model: opus-5-5
2026-11-19 23:43:00 +01:00
clawbot 9d008ed31a waf: a heuristic says itself that it refuses whatever the lookup answers (closes #446)
check / check (push) Failing after 1m37s
refusedWhateverTheAnswer asks each heuristic of afterLookup that
implements the new refusesWithoutLookup interface, instead of naming
blocklists and rate_limits. The comment on adding a heuristic names the
places outside its file that may change with it.

Model: opus-5-5
2026-11-19 23:28:52 +01:00
clawbot 86bb33bf07 waf: list its own endpoints on a chi router in routes.go (closes #426)
check / check (push) Successful in 7m7s
routes.go lists every own endpoint. One middleware checks a token and
another writes the log line. answerOwnEndpoint, answerAdmin and
endpoint() are gone; the handlers return http.HandlerFunc. The router
matches the decoded path, as the old code did. A test checks the
request_bytes of an own endpoint's log line.

Model: opus-5-5
2026-11-19 23:13:24 +01:00
clawbot 91444a37b3 metrics: build from one Params in place of seven Add calls (closes #438)
check / check (push) Failing after 1m48s
metrics.New takes a Params holding every part at once, each behind a
small interface declared in metrics beside the metrics it reads. A part
that is off is left nil and adds no metric. The rule-match and
reputation-hit counters are made in New, so no metric field is nil and
no call order is needed. metrics no longer imports remotelog or rules.

Model: opus-5-5
2026-11-19 22:59:44 +01:00
clawbot 51989235d6 rproxy: the body wrappers hold only what they call (closes #431)
check / check (push) Failing after 4m1s
requestBody and responseBody are read on goroutines other than the
handler's. They held the whole request; now they hold only the
functions they call and, for responseBody, the request's context.

Model: opus-5-5
2026-11-19 22:43:57 +01:00
clawbot e15edef583 Move what rproxy and waf send each other into wafprotocol (closes #425)
check / check (push) Failing after 3m42s
internal/wafprotocol writes and reads the question, the answer and the
report, and declares once each header, path and limit both daemons use.
rproxy no longer imports internal/waf. Nothing on the wire changes.

Model: opus-5-5
2026-11-19 22:27:16 +01:00
clawbot 06faf0877c dirwatch: one loop that watches a directory, for rules and lookup (closes #429)
check / check (push) Successful in 4m35s
rules and lookup each had the same loop: watch the directory, start a
timer at once, restart it at each change that counts, read again when it
fires. It now lives once in internal/dirwatch, with its quiet time and
its tests. Each package keeps its own log messages, its readAgain, and
its test for which changes count. internal/state keeps its own copy.

Model: opus-5-5
2026-11-19 22:13:55 +01:00
clawbot b2b1716907 sww: the three commands as cobra commands (closes #440)
check / check (push) Failing after 1m56s
Main builds a cobra root command with rproxy, waf and healthcheck under
it, in place of the switch on os.Args and the hand-kept usage text.
Exit statuses stay: 2 with no command or an unknown one, otherwise what
the command returns. cobra's completion command is off. `sww help` and
`sww --help` now print the commands and exit 0.

Model: opus-5-5
2026-11-19 21:59:25 +01:00
clawbot 56a57125f1 waf: each heuristic holds its own settings and parts (closes #441)
check / check (push) Failing after 1m42s
Every heuristic is a struct built once in newHandler with the settings
and collaborators it reads; no check reads the handler. The six lists
are handler fields built there. Code several heuristics share takes what
it needs as arguments or lives on a struct they hold: banMaker for bans
and their alerts, reputationHitNoter for reputation hits,
biasedThresholds for limit percentages. The handler keeps blocklists,
rateLimits, trapPaths, ruleFiles and coreRuleSet as fields too, since
refusedWhateverTheAnswer and findBesideLookup use them.

Model: opus-5-5
2026-11-19 21:43:55 +01:00
clawbot 9a8ed745bd Constructors take Params: ratelimit.New and processlog.New (closes #450)
check / check (push) Failing after 6m12s
ratelimit.New takes ratelimit.Params (Limits, MaxClients) in place of
two arguments by position. The process logger moves out of requestlog
into internal/processlog, whose New takes Params (Output, InstanceName,
Level); its two tests move with it unchanged. geojs.GeoJS keeps its
params whole, as DNSBL, AbuseIPDB and routewatch.Client do. SPEC and
README list the new package.

Model: opus-5-5
2026-11-19 21:28:01 +01:00
clawbot 0a230c86ce rproxy tests: leave room for a busy host in five timing tests (closes #451)
check / check (push) Failing after 15m10s
Three tests that wait out a 2- or 3-second request time limit now use 5
seconds, so the request reaches the app before it runs out under load.
The WebSocket test also uses 5-second limits, and renews its
connection's deadline after the wait.

TestFiveFailedAsksKeepTheNextRequestFromAskingWAF stops the breaker's
clock like TestAnswersThatCannotBePassedOnOpenTheBreaker.

Model: opus-5-5
2026-11-19 21:14:01 +01:00
clawbot 40386dce6e waf: wait as long as it takes for a test request's answer (closes #454)
check / check (push) Failing after 2m29s
requestWithBody gave the answer 10 seconds, and endedLines gave the log
line 10 seconds. On a busy host the Core Rule Set's inspection of a
request with 1000 query parameters, under the race detector, took
longer. Both now wait until the answer or line comes; go test's
-timeout ends a test where it never does.

Model: opus-5-5
2026-11-19 20:46:01 +01:00
clawbot 433d5cc0a9 bans: one ledger method makes a request's ban, one place decides observe mode (closes #444)
check / check (push) Successful in 5m40s
The ledger's BanForLimit, WouldBanForLimit, BanForAttack and
WouldBanForAttack become Ban and WouldBan, which take the cause and the
notes in one bans.Grounds. The waf's banForLimit and banForAttack fill in
what differs and call banClient, the one function that tests observe
mode on the way to a ban. Behaviour is unchanged; tests change only
where they called the four methods.

Model: opus-5-5
2026-11-19 20:29:53 +01:00
clawbot 419ee1c441 No part imports config (closes #447)
check / check (push) Failing after 2m13s
TagList, MaskZoneKey, ParseNetblock, ParseASN and ParsePercent move
from config to reputation; config imports them from there. rules and
config check header names with httpguts.ValidHeaderFieldName, and
IsHeaderName goes. A new test pins that check to the characters
IsHeaderName took, for the empty name and every single-byte name.

Model: opus-5-5
2026-11-19 20:14:27 +01:00
clawbot aaa28f541b rproxy, sww: keep the breaker and stop tests from failing under load (closes #474)
check / check (push) Failing after 2m10s
The circuit breaker reads the time through a clock the handler holds,
which is time.Now outside tests. The test that answers waf's
questions with answers rproxy cannot pass on stops that clock, so how
long its requests take on a slow host no longer decides whether the
breaker opens or turns the next request away.

The test that stops rproxy mid-request sets its read deadline from the
stop, not from the send, so a slow wait for the app no longer shortens
it.

Model: opus-5-5
2026-11-19 19:59:34 +01:00
clawbot 98d7432077 waf: the steps after the answer as plain statements (closes #445)
check / check (push) Failing after 2m21s
banAndCount runs the heuristics that only ban, then countAnomalies,
then addToHistory. A refused request's decision and an allowed
request's report both call it, each with its own list. answerQuestion
keeps one defer, which writes the log line and holds the decision for
the report.

A check that panics now leaves the request out of its client's history
and the anomaly counts, as the issue states.

Model: opus-5-5
2026-11-19 19:45:13 +01:00
clawbot 11ac8aee50 config: export the setting names waf logs (closes #448)
check / check (push) Failing after 2m25s
waf wrote eight setting names as string literals into log lines and
ban notes, and config wrote each again where it reads the setting. A
rename that missed one would log a setting that does not exist. config
now exports a constant for each name, used by its reader and by waf.

Model: opus-5-5
2026-11-19 19:29:49 +01:00
clawbot e7cc0adbdb waf: end the asks to GeoJS, the DNSBL zones, AbuseIPDB and routewatch at the stop (closes #439)
check / check (push) Failing after 1m58s
Each of the four keeps the cancel function of every ask it started, and a
wait group of their goroutines. Its new method (GeoJS.StopAsking,
DNSBL.StopAsking, AbuseIPDB.StopChecking, routewatch Client.StopAsking)
refuses further asks, ends those under way, and returns once their
goroutines have. An ask ended that way is no failure: no alert, no failure
count, no wait before the next. Each provider in internal/sww adds the
method as a stop hook, so the last write of the state files follows it.

Model: opus-5-5
2026-11-19 19:02:12 +01:00
133 changed files with 7700 additions and 5156 deletions

No files matched your search

+9 -3
View File
@@ -902,13 +902,19 @@ stops the running process (see [`docs/state-files.md`](docs/state-files.md)).
- `internal/rproxy`: the reverse proxy in front of one app. - `internal/rproxy`: the reverse proxy in front of one app.
- `internal/waf`: `waf`'s server, which runs the checks and answers its own - `internal/waf`: `waf`'s server, which runs the checks and answers its own
endpoints. endpoints.
- `internal/wafprotocol`: the question, the answer and the report that `rproxy`
and `waf` send each other.
- `internal/ratelimit`, `internal/bans`, `internal/rules`, - `internal/ratelimit`, `internal/bans`, `internal/rules`,
`internal/coreruleset`, `internal/geojs`, `internal/lookup`, `internal/coreruleset`, `internal/geojs`, `internal/lookup`,
`internal/reputation`, `internal/routewatch`, `internal/anomaly` and `internal/reputation`, `internal/routewatch`, `internal/anomaly` and
`internal/alerts`: one check or source each. `internal/alerts`: one check or source each.
- `internal/state`, `internal/requestlog`, `internal/remotelog` and - `internal/dirwatch`: watches a directory, so that `internal/rules` and
`internal/metrics`: the state and cache files, the log lines, the syslog `internal/lookup` read their files again when they change.
sender and the metrics. - `internal/state`: the state and cache files.
- `internal/requestlog`: the request log.
- `internal/processlog`: the process's own messages.
- `internal/remotelog`: the syslog sender.
- `internal/metrics`: the metrics.
## Rationale ## Rationale
+49 -37
View File
@@ -69,17 +69,21 @@ never as a time.
that command's settings, and runs `internal/rproxy` until `SIGTERM` or that command's settings, and runs `internal/rproxy` until `SIGTERM` or
`SIGINT` (see "`rproxy`"). `sww healthcheck` is the image's health check. `SIGINT` (see "`rproxy`"). `sww healthcheck` is the image's health check.
With no command, or one it does not know, the binary prints the commands With no command, or one it does not know, the binary prints the commands
and exits with status 2. `internal/sww` is the only package that imports and exits with status 2. `sww help` and `sww --help` print them and exit
with status 0. `internal/sww` is the only package that imports
`go.uber.org/fx`. fx builds the parts of `waf` and `rproxy`, and starts `go.uber.org/fx`. fx builds the parts of `waf` and `rproxy`, and starts
and stops their background work. and stops their background work.
- `internal/config`: reads the settings, the one place they are read. - `internal/config`: reads the settings, the one place they are read.
- `internal/waf`: `waf`'s server. It runs the checks in `check`, and holds - `internal/waf`: `waf`'s server. It runs the checks in `check`. Its
`waf`'s question, answer and report. It answers `waf`'s own endpoints in `routes.go` lists `waf`'s own endpoints. It counts bytes, refusals and
`answerOwnEndpoint`. It counts bytes, refusals and anomalies, and writes anomalies, and writes the log lines.
the log lines.
- `internal/rproxy`: the reverse proxy in front of one app that asks `waf` - `internal/rproxy`: the reverse proxy in front of one app that asks `waf`
about each request. It passes a request to the app with the standard about each request. It passes a request to the app with the standard
library's `httputil.ReverseProxy`, within the time and size limits. library's `httputil.ReverseProxy`, within the time and size limits.
- `internal/wafprotocol`: the question, the answer and the report that
`rproxy` and `waf` send each other (see "`waf`"). It writes and reads each
one, and declares every header, path and limit both sides use.
`internal/rproxy` imports it, not `internal/waf`.
- `internal/ratelimit`: the table of clients. It counts requests, bytes and - `internal/ratelimit`: the table of clients. It counts requests, bytes and
refusals, and keeps each client's history. refusals, and keeps each client's history.
- `internal/bans`: the bans. It holds each netblock's bans with their notes, - `internal/bans`: the bans. It holds each netblock's bans with their notes,
@@ -93,6 +97,9 @@ never as a time.
imports no other package of this repository: `waf` gives it what it needs imports no other package of this repository: `waf` gives it what it needs
to count and report failures. to count and report failures.
- `internal/lookup`: looks clients up in the lookup database. - `internal/lookup`: looks clients up in the lookup database.
- `internal/dirwatch`: watches a directory, and calls a function once it has
gone a short time without a change. `internal/rules` and `internal/lookup`
read their files again this way.
- `internal/reputation`: fetches the lists, asks the DNS blocklists and - `internal/reputation`: fetches the lists, asks the DNS blocklists and
AbuseIPDB in the background, and keeps their answers. It also fetches AbuseIPDB in the background, and keeps their answers. It also fetches
AbuseIPDB's blacklist, which `internal/waf` bans. AbuseIPDB's blacklist, which `internal/waf` bans.
@@ -104,8 +111,10 @@ never as a time.
its own. its own.
- `internal/state`: reads the state and cache files at start, takes in edits - `internal/state`: reads the state and cache files at start, takes in edits
while running, and writes the files when due and at the stop. while running, and writes the files when due and at the stop.
- `internal/requestlog` and `internal/remotelog`: the lines on stdout, and - `internal/requestlog`: the request log, as JSON lines on stdout.
sending them to the syslog server. - `internal/processlog`: the process lines on stdout, which hold the
process's own messages.
- `internal/remotelog`: sends the lines on stdout to the syslog server.
- `internal/metrics`: the metrics, in the Prometheus text format. - `internal/metrics`: the metrics, in the Prometheus text format.
- `Dockerfile`: the lint and test phases, then the image. - `Dockerfile`: the lint and test phases, then the image.
`deploy/example-app` is an app built on the image, with `rproxy` in its `deploy/example-app` is an app built on the image, with `rproxy` in its
@@ -119,11 +128,13 @@ never as a time.
default `SWW_WAF_CORE_RULE_SET_DISABLED_RULES`, are built for that default `SWW_WAF_CORE_RULE_SET_DISABLED_RULES`, are built for that
version. They are checked again before the version changes, since rule version. They are checked again before the version changes, since rule
ids, and what each rule matches, change between versions; ids, and what each rule matches, change between versions;
- `github.com/go-chi/chi/v5` to route the requests to `waf`'s own endpoints;
- `github.com/hashicorp/golang-lru/v2` for the bounded tables; - `github.com/hashicorp/golang-lru/v2` for the bounded tables;
- `github.com/oschwald/maxminddb-golang/v2` to read the lookup database; - `github.com/oschwald/maxminddb-golang/v2` to read the lookup database;
- `github.com/fsnotify/fsnotify` to notice edited files; - `github.com/fsnotify/fsnotify` to notice edited files;
- `github.com/prometheus/client_golang` for the metrics; - `github.com/prometheus/client_golang` for the metrics;
- `github.com/spf13/viper` to read the settings from the environment; - `github.com/spf13/viper` to read the settings from the environment;
- `github.com/spf13/cobra` for the commands and their help text;
- `github.com/maxmind/mmdbwriter`, in the tests, to write lookup databases; - `github.com/maxmind/mmdbwriter`, in the tests, to write lookup databases;
- remote log sending written in this project, since the standard library's - remote log sending written in this project, since the standard library's
`log/syslog` is frozen and writes only the older syslog format. `log/syslog` is frozen and writes only the older syslog format.
@@ -817,8 +828,8 @@ The supported apps are:
- `vaultwarden`; - `vaultwarden`;
- `wordpress`. - `wordpress`.
Each app's fixes are in one file, such as `internal/coreruleset/gitea.conf`, and Each app's fixes are in one file named for the app, such as
one entry of `apps` in `internal/coreruleset/coreruleset.go`. gitea's fixes are: `internal/coreruleset/apps/gitea.conf`. gitea's fixes are:
1. Ten parameters are not checked against the Core Rule Set's lists of system 1. Ten parameters are not checked against the Core Rule Set's lists of system
files (930120), shell paths (932160) and command names (932260), on any path: files (930120), shell paths (932160) and command names (932260), on any path:
@@ -851,8 +862,8 @@ one entry of `apps` in `internal/coreruleset/coreruleset.go`. gitea's fixes are:
3. On gitea's paths, some fields skip the rules that refused ordinary gitea 3. On gitea's paths, some fields skip the rules that refused ordinary gitea
requests there. They are the fields where gitea sends a search, a file's name requests there. They are the fields where gitea sends a search, a file's name
or text, a commit message, a branch name or a repository's name. or text, a commit message, a branch name or a repository's name.
`internal/coreruleset/gitea.conf` lists each rule and the request it refused. `internal/coreruleset/apps/gitea.conf` lists each rule and the request it
The fields are: refused. The fields are:
- `q` on gitea's searches of code, of issues and pull requests, and of - `q` on gitea's searches of code, of issues and pull requests, and of
commits. A code search holds code, such as `<script>`, commits. A code search holds code, such as `<script>`,
`SELECT * FROM users WHERE` or `../`. The searches of issues and pull `SELECT * FROM users WHERE` or `../`. The searches of issues and pull
@@ -1037,7 +1048,7 @@ one entry of `apps` in `internal/coreruleset/coreruleset.go`. gitea's fixes are:
- The `Content-Range` header of an artifact upload from - The `Content-Range` header of an artifact upload from
`actions/upload-artifact@v3` (920450). `actions/upload-artifact@v3` (920450).
- The passwords in a body, for each rule that random passwords set off. The - The passwords in a body, for each rule that random passwords set off. The
numbers are in `internal/coreruleset/gitea.conf`. numbers are in `internal/coreruleset/apps/gitea.conf`.
- `password` on these paths: - `password` on these paths:
- `/user/login`, `/user/sign_up`, `/user/activate`; - `/user/login`, `/user/sign_up`, `/user/activate`;
- `/user/link_account_signin` and `/user/link_account_signup`; - `/user/link_account_signin` and `/user/link_account_signup`;
@@ -1069,8 +1080,8 @@ one entry of `apps` in `internal/coreruleset/coreruleset.go`. gitea's fixes are:
such as `owner/python3-app` would have that push refused. such as `owner/python3-app` would have that push refused.
WordPress's fixes: on WordPress's paths, some fields skip the rules that refused WordPress's fixes: on WordPress's paths, some fields skip the rules that refused
ordinary WordPress requests there. `internal/coreruleset/wordpress.conf` lists ordinary WordPress requests there. `internal/coreruleset/apps/wordpress.conf`
each rule and the request it refused. The REST API has four addresses: lists each rule and the request it refused. The REST API has four addresses:
- `/wp-json/...` - `/wp-json/...`
- `/index.php/wp-json/...` - `/index.php/wp-json/...`
@@ -1162,9 +1173,9 @@ Elsewhere WordPress's fixes do not apply. The paths are those of WordPress at
the site root. the site root.
Drupal's fixes: on Drupal's paths, some fields skip the rules that refused Drupal's fixes: on Drupal's paths, some fields skip the rules that refused
ordinary Drupal requests there. `internal/coreruleset/drupal.conf` lists each ordinary Drupal requests there. `internal/coreruleset/apps/drupal.conf` lists
rule and the request it refused. These query fields skip the lists of system each rule and the request it refused. These query fields skip the lists of
files and command names (930120, 932120, 932250, 932260): system files and command names (930120, 932120, 932250, 932260):
- On every page under `/admin`: `title`, `name`, `user`, `filename`, `filemime`, - On every page under `/admin`: `title`, `name`, `user`, `filename`, `filemime`,
`info`, `subject`, `author_name`, `search`, `q` and `destination`. `info`, `subject`, `author_name`, `search`, `q` and `destination`.
@@ -1254,10 +1265,10 @@ first path segment of two or three lowercase letters, optionally followed by a
hyphen and more letters, so `/api` counts too. hyphen and more letters, so `/api` counts too.
MediaWiki's fixes: on MediaWiki's paths, some fields skip the rules that refused MediaWiki's fixes: on MediaWiki's paths, some fields skip the rules that refused
ordinary MediaWiki requests there. `internal/coreruleset/mediawiki.conf` lists ordinary MediaWiki requests there. `internal/coreruleset/apps/mediawiki.conf`
each rule and the request it refused. These fields skip the lists of system lists each rule and the request it refused. These fields skip the lists of
files and command names (930120, 932120, 932250, 932260), in the query or the system files and command names (930120, 932120, 932250, 932260), in the query or
body: the body:
- On MediaWiki's pages, which are `/index.php` and the paths under `/index.php/` - On MediaWiki's pages, which are `/index.php` and the paths under `/index.php/`
and `/wiki/`: and `/wiki/`:
@@ -1363,9 +1374,9 @@ Elsewhere these fields keep every rule. The paths are those of MediaWiki at the
site root or under `/w/`. site root or under `/w/`.
Mastodon's fixes: on Mastodon's paths, some fields skip the rules that refused Mastodon's fixes: on Mastodon's paths, some fields skip the rules that refused
ordinary Mastodon requests there. `internal/coreruleset/mastodon.conf` lists ordinary Mastodon requests there. `internal/coreruleset/apps/mastodon.conf`
each rule and the request it refused. These fields skip the lists of system lists each rule and the request it refused. These fields skip the lists of
files and command names (930120, 932120, 932250, 932260), in the query: system files and command names (930120, 932120, 932250, 932260), in the query:
- On the API's search, accounts and admin accounts, under `/api/v1/` and - On the API's search, accounts and admin accounts, under `/api/v1/` and
`/api/v2/`, the searches and names `q`, `acct`, `username`, `display_name`, `/api/v2/`, the searches and names `q`, `acct`, `username`, `display_name`,
@@ -1526,7 +1537,7 @@ every rule.
Vaultwarden's fixes: on Vaultwarden's paths, some fields skip the rules that Vaultwarden's fixes: on Vaultwarden's paths, some fields skip the rules that
refused ordinary Vaultwarden requests there. refused ordinary Vaultwarden requests there.
`internal/coreruleset/vaultwarden.conf` lists each rule and the request it `internal/coreruleset/apps/vaultwarden.conf` lists each rule and the request it
refused. The apps encrypt vault items, folders, collections and sends before refused. The apps encrypt vault items, folders, collections and sends before
sending them, so those fields skip no rules. The fields below hold plain text. sending them, so those fields skip no rules. The fields below hold plain text.
@@ -1577,8 +1588,8 @@ has a parameter of the same name.
`token` in the body of `/admin`, the admin token typed on the admin page, skips `token` in the body of `/admin`, the admin token typed on the admin page, skips
each rule that random passwords set off. The numbers are in each rule that random passwords set off. The numbers are in
`internal/coreruleset/vaultwarden.conf`. It applies only when the query has no `internal/coreruleset/apps/vaultwarden.conf`. It applies only when the query has
`token` parameter. `' OR '1'='1`, `../../etc/passwd`, `; cat /etc/passwd` and no `token` parameter. `' OR '1'='1`, `../../etc/passwd`, `; cat /etc/passwd` and
`<img src=x onerror=x()>` sent as the token reach Vaultwarden; `<img src=x onerror=x()>` sent as the token reach Vaultwarden;
`<script>alert(1)</script>` and `javascript:alert(1)` are still refused. `<script>alert(1)</script>` and `javascript:alert(1)` are still refused.
@@ -1600,7 +1611,7 @@ Elsewhere these fields keep every rule.
Nextcloud's fixes: on Nextcloud's paths, some fields skip the rules that refused Nextcloud's fixes: on Nextcloud's paths, some fields skip the rules that refused
ordinary Nextcloud requests there, and its WebDAV methods are allowed. ordinary Nextcloud requests there, and its WebDAV methods are allowed.
`internal/coreruleset/nextcloud.conf` lists each rule and the request it `internal/coreruleset/apps/nextcloud.conf` lists each rule and the request it
refused. refused.
The sync client and the Files app use WebDAV on these paths: The sync client and the Files app use WebDAV on these paths:
@@ -1610,7 +1621,8 @@ The sync client and the Files app use WebDAV on these paths:
- `/remote.php/webdav/`; - `/remote.php/webdav/`;
- `/public.php/webdav/`. - `/public.php/webdav/`.
These methods are allowed there, besides those of the first change: These methods are allowed there, besides `PUT`, `PATCH` and `DELETE`, which
every app gets:
- `PROPFIND` and `PROPPATCH`; - `PROPFIND` and `PROPPATCH`;
- `MKCOL`; - `MKCOL`;
@@ -1754,7 +1766,7 @@ gets gitea's fixes too. Forgejo's own fixes hold only what Forgejo sends and
gitea does not. gitea does not.
On Forgejo's paths, some fields skip the rules that refused ordinary Forgejo On Forgejo's paths, some fields skip the rules that refused ordinary Forgejo
requests there. `internal/coreruleset/forgejo.conf` lists each rule and the requests there. `internal/coreruleset/apps/forgejo.conf` lists each rule and the
request it refused. request it refused.
`q` skips rules on some of Forgejo's searches. The searches are: `q` skips rules on some of Forgejo's searches. The searches are:
@@ -1800,10 +1812,10 @@ The Core Rule Set reads the request without the `flash` cookie, in which Forgejo
puts what gitea puts in `gitea_flash` (gitea's second fix). puts what gitea puts in `gitea_flash` (gitea's second fix).
Ghost's fixes: on Ghost's paths, some fields skip the rules that refused Ghost's fixes: on Ghost's paths, some fields skip the rules that refused
ordinary Ghost requests there. `internal/coreruleset/ghost.conf` lists each rule ordinary Ghost requests there. `internal/coreruleset/apps/ghost.conf` lists each
and the request it refused. Ghost's admin and integrations call the Admin API rule and the request it refused. Ghost's admin and integrations call the Admin
under `/ghost/api/admin/`. Themes and Ghost's scripts call the Content API under API under `/ghost/api/admin/`. Themes and Ghost's scripts call the Content API
`/ghost/api/content/`, and members call `/members/api/`. The admin calls under `/ghost/api/content/`, and members call `/members/api/`. The admin calls
ActivityPub, a service of its own, under `/.ghost/activitypub/`. ActivityPub, a service of its own, under `/.ghost/activitypub/`.
Eight rules refused words that name commands and files, such as `python3`, Eight rules refused words that name commands and files, such as `python3`,
@@ -1941,9 +1953,9 @@ That name holds `.profile`, a file on the list of system files, so the name
skips 930120 when a profile is saved. skips 930120 when a profile is saved.
The passwords skip each rule that random passwords set off. The numbers are in The passwords skip each rule that random passwords set off. The numbers are in
`internal/coreruleset/ghost.conf`. They are `password` when signing in, and the `internal/coreruleset/apps/ghost.conf`. They are `password` when signing in, and
passwords sent when creating the first staff user, accepting an invitation, and the passwords sent when creating the first staff user, accepting an invitation,
resetting or changing a password. `' OR '1'='1`, `../../etc/passwd`, and resetting or changing a password. `' OR '1'='1`, `../../etc/passwd`,
`; cat /etc/passwd` and `<img src=x onerror=x()>` sent as a password reach `; cat /etc/passwd` and `<img src=x onerror=x()>` sent as a password reach
Ghost; `<script>alert(1)</script>` and `javascript:alert(1)` are still refused. Ghost; `<script>alert(1)</script>` and `javascript:alert(1)` are still refused.
+4 -1
View File
@@ -7,12 +7,15 @@ require (
github.com/corazawaf/coraza-coreruleset/v4 v4.25.0 github.com/corazawaf/coraza-coreruleset/v4 v4.25.0
github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68 github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68
github.com/fsnotify/fsnotify v1.10.1 github.com/fsnotify/fsnotify v1.10.1
github.com/go-chi/chi/v5 v5.5.0
github.com/hashicorp/golang-lru/v2 v2.0.7 github.com/hashicorp/golang-lru/v2 v2.0.7
github.com/maxmind/mmdbwriter v1.2.0 github.com/maxmind/mmdbwriter v1.2.0
github.com/oschwald/maxminddb-golang/v2 v2.7.0 github.com/oschwald/maxminddb-golang/v2 v2.7.0
github.com/prometheus/client_golang v1.24.1 github.com/prometheus/client_golang v1.24.1
github.com/spf13/cobra v1.10.2
github.com/spf13/viper v1.21.0 github.com/spf13/viper v1.21.0
go.uber.org/fx v1.24.0 go.uber.org/fx v1.24.0
golang.org/x/net v0.60.0
golang.org/x/sys v0.48.0 golang.org/x/sys v0.48.0
) )
@@ -25,6 +28,7 @@ require (
github.com/goccy/go-yaml v1.19.2 // indirect github.com/goccy/go-yaml v1.19.2 // indirect
github.com/gotnospirit/makeplural v0.0.0-20180622080156-a5f48d94d976 // indirect github.com/gotnospirit/makeplural v0.0.0-20180622080156-a5f48d94d976 // indirect
github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 // indirect github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/kaptinlin/go-i18n v0.1.4 // indirect github.com/kaptinlin/go-i18n v0.1.4 // indirect
github.com/kaptinlin/jsonschema v0.4.6 // indirect github.com/kaptinlin/jsonschema v0.4.6 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect github.com/kylelemons/godebug v1.1.0 // indirect
@@ -50,7 +54,6 @@ require (
go.uber.org/zap v1.26.0 // indirect go.uber.org/zap v1.26.0 // indirect
go.yaml.in/yaml/v3 v3.0.5 // indirect go.yaml.in/yaml/v3 v3.0.5 // indirect
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba // indirect go4.org/netipx v0.0.0-20231129151722-fdeea329fbba // indirect
golang.org/x/net v0.60.0 // indirect
golang.org/x/sync v0.23.0 // indirect golang.org/x/sync v0.23.0 // indirect
golang.org/x/text v0.42.0 // indirect golang.org/x/text v0.42.0 // indirect
google.golang.org/protobuf v1.36.11 // indirect google.golang.org/protobuf v1.36.11 // indirect
+11
View File
@@ -14,12 +14,15 @@ github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68 h1:KLEjvBBSw
github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68/go.mod h1:9XocblQEwK6EYylRth7PrPIifF2l0iSoYo4FAXhXU/8= github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68/go.mod h1:9XocblQEwK6EYylRth7PrPIifF2l0iSoYo4FAXhXU/8=
github.com/corazawaf/libinjection-go v0.3.4 h1:jX2Tit2xZxYJJyVryNXucYBZtQqS7473x9Nc6lgRRbk= github.com/corazawaf/libinjection-go v0.3.4 h1:jX2Tit2xZxYJJyVryNXucYBZtQqS7473x9Nc6lgRRbk=
github.com/corazawaf/libinjection-go v0.3.4/go.mod h1:Ik/+w3UmTWH9yn366RgS9D95K3y7Atb5m/H/gXzzPCk= github.com/corazawaf/libinjection-go v0.3.4/go.mod h1:Ik/+w3UmTWH9yn366RgS9D95K3y7Atb5m/H/gXzzPCk=
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/foxcpp/go-mockdns v1.2.0 h1:omK3OrHRD1IWJz1FuFBCFquhXslXoF17OvBS6JPzZF0= github.com/foxcpp/go-mockdns v1.2.0 h1:omK3OrHRD1IWJz1FuFBCFquhXslXoF17OvBS6JPzZF0=
github.com/foxcpp/go-mockdns v1.2.0/go.mod h1:IhLeSFGed3mJIAXPH2aiRQB+kqz7oqu8ld2qVbOu7Wk= github.com/foxcpp/go-mockdns v1.2.0/go.mod h1:IhLeSFGed3mJIAXPH2aiRQB+kqz7oqu8ld2qVbOu7Wk=
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8= github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0= github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho= github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho=
github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo= github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo=
github.com/go-chi/chi/v5 v5.5.0 h1:xdKQ1dJmVJsmgahrXSKhUx8NyVPzpwdlyrrQFpMPJgg=
github.com/go-chi/chi/v5 v5.5.0/go.mod h1:d1IG7GtKPSjPyBP365PH8sAGVHk3qfB/IhjFNRH1wWM=
github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs= github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM= github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4= github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
@@ -34,6 +37,8 @@ github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 h1:c7gcN
github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092/go.mod h1:ZZAN4fkkful3l1lpJwF8JbW41ZiG9TwJ2ZlqzQovBNU= github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092/go.mod h1:ZZAN4fkkful3l1lpJwF8JbW41ZiG9TwJ2ZlqzQovBNU=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k= github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM= github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/jcchavezs/mergefs v0.1.1 h1:D45R17m6dHnSVZefnhynoeZvcK2Uw0oTrRfoUOQ0S5Y= github.com/jcchavezs/mergefs v0.1.1 h1:D45R17m6dHnSVZefnhynoeZvcK2Uw0oTrRfoUOQ0S5Y=
github.com/jcchavezs/mergefs v0.1.1/go.mod h1:eRLTrsA+vFwQZ48hj8p8gki/5v9C2bFtHH5Mnn4bcGk= github.com/jcchavezs/mergefs v0.1.1/go.mod h1:eRLTrsA+vFwQZ48hj8p8gki/5v9C2bFtHH5Mnn4bcGk=
github.com/kaptinlin/go-i18n v0.1.4 h1:wCiwAn1LOcvymvWIVAM4m5dUAMiHunTdEubLDk4hTGs= github.com/kaptinlin/go-i18n v0.1.4 h1:wCiwAn1LOcvymvWIVAM4m5dUAMiHunTdEubLDk4hTGs=
@@ -72,6 +77,7 @@ github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+
github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY= github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY=
github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8= github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs= github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/sagikazarmark/locafero v0.11.0 h1:1iurJgmM9G3PA/I+wWYIOw/5SyBtxapeHDcg+AAIFXc= github.com/sagikazarmark/locafero v0.11.0 h1:1iurJgmM9G3PA/I+wWYIOw/5SyBtxapeHDcg+AAIFXc=
github.com/sagikazarmark/locafero v0.11.0/go.mod h1:nVIGvgyzw595SUSUE6tvCp3YYTeHs15MvlmU87WwIik= github.com/sagikazarmark/locafero v0.11.0/go.mod h1:nVIGvgyzw595SUSUE6tvCp3YYTeHs15MvlmU87WwIik=
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 h1:+jumHNA0Wrelhe64i8F6HNlS8pkoyMv5sreGx2Ry5Rw= github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 h1:+jumHNA0Wrelhe64i8F6HNlS8pkoyMv5sreGx2Ry5Rw=
@@ -80,6 +86,9 @@ github.com/spf13/afero v1.15.0 h1:b/YBCLWAJdFWJTN9cLhiXXcD7mzKn9Dm86dNnfyQw1I=
github.com/spf13/afero v1.15.0/go.mod h1:NC2ByUVxtQs4b3sIUphxK0NioZnmxgyCrfzeuq8lxMg= github.com/spf13/afero v1.15.0/go.mod h1:NC2ByUVxtQs4b3sIUphxK0NioZnmxgyCrfzeuq8lxMg=
github.com/spf13/cast v1.10.0 h1:h2x0u2shc1QuLHfxi+cTJvs30+ZAHOGRic8uyGTDWxY= github.com/spf13/cast v1.10.0 h1:h2x0u2shc1QuLHfxi+cTJvs30+ZAHOGRic8uyGTDWxY=
github.com/spf13/cast v1.10.0/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqeHo= github.com/spf13/cast v1.10.0/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqeHo=
github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk= github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.21.0 h1:x5S+0EU27Lbphp4UKm1C+1oQO+rKx36vfCoaVebLFSU= github.com/spf13/viper v1.21.0 h1:x5S+0EU27Lbphp4UKm1C+1oQO+rKx36vfCoaVebLFSU=
@@ -109,6 +118,7 @@ go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so= go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ= go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ= go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw= go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg= go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba h1:0b9z3AuHCjxk0x/opv64kcgZLBseWJUpBw5I82+2U4M= go4.org/netipx v0.0.0-20231129151722-fdeea329fbba h1:0b9z3AuHCjxk0x/opv64kcgZLBseWJUpBw5I82+2U4M=
@@ -127,6 +137,7 @@ golang.org/x/tools v0.50.0 h1:c2ifzfcuY7L90lZ2aKd8S4K2NpASF08SZx9ZuJkHmSU=
golang.org/x/tools v0.50.0/go.mod h1:7ulVMw3831Mwi5EZD6RomGyffr4VFjuNYXf2BbCEAV0= golang.org/x/tools v0.50.0/go.mod h1:7ulVMw3831Mwi5EZD6RomGyffr4VFjuNYXf2BbCEAV0=
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE= google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
rsc.io/binaryregexp v0.2.0 h1:HfqmD5MEmC0zvwBuF187nq9mdnXjXsSivRiXN7SmRkE= rsc.io/binaryregexp v0.2.0 h1:HfqmD5MEmC0zvwBuF187nq9mdnXjXsSivRiXN7SmRkE=
+55 -42
View File
@@ -34,65 +34,76 @@ import (
"time" "time"
) )
// Event is what an alert is for, such as EventBan.
type Event string
// The events an alert is for, as SWW_WAF_ALERT_EVENTS names them. // The events an alert is for, as SWW_WAF_ALERT_EVENTS names them.
const ( const (
// EventBan is a ban sww made. // EventBan is a ban sww made.
EventBan = "ban" EventBan Event = "ban"
// EventPermanentBan is a permanent ban sww made, or a ban it // EventPermanentBan is a permanent ban sww made, or a ban it
// made permanent. // made permanent.
EventPermanentBan = "permanent_ban" EventPermanentBan Event = "permanent_ban"
// EventAnomaly is a count of requests or bytes over an anomaly // EventAnomaly is a count of requests or bytes over an anomaly
// threshold. // threshold.
EventAnomaly = "anomaly" EventAnomaly Event = "anomaly"
// EventCoreRuleSetMatch is a request the Core Rule Set scored at or over // EventCoreRuleSetMatch is a request the Core Rule Set scored at or over
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD, refused in block mode, let through in // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD, refused in block mode, let through in
// detect mode. // detect mode.
EventCoreRuleSetMatch = "core_rule_set_match" EventCoreRuleSetMatch Event = "core_rule_set_match"
// EventReputationHit is a request whose client a blocklist or a DNSBL // EventReputationHit is a request whose client a blocklist or a DNSBL
// zone lists, or whose AbuseIPDB score is a hit. // zone lists, or whose AbuseIPDB score is a hit.
EventReputationHit = "reputation_hit" EventReputationHit Event = "reputation_hit"
// EventSourceFailure is GeoJS failing or refusing sww, a fetch // EventSourceFailure is GeoJS failing or refusing sww, a fetch
// of a list failing, a query to a DNSBL zone or a check with AbuseIPDB // of a list failing, a query to a DNSBL zone or a check with AbuseIPDB
// failing or refused, or the day's AbuseIPDB checks used up. // failing or refused, or the day's AbuseIPDB checks used up.
EventSourceFailure = "source_failure" EventSourceFailure Event = "source_failure"
// EventFileError is one of: // EventFileError is one of:
// - a rule file or state file edited while sww runs that does not parse; // - a rule file or state file edited while sww runs that does not parse;
// - a cache file that does not parse, at start or after an edit; // - a cache file that does not parse, at start or after an edit;
// - a state file or cache file that cannot be written; // - a state file or cache file that cannot be written;
// - a replacement of the lookup database that cannot be read. // - a replacement of the lookup database that cannot be read.
EventFileError = "file_error" EventFileError Event = "file_error"
// EventSummary is the summary sent as an hour ends: of the alerts held // EventSummary is the summary sent as an hour ends: of the alerts held
// back in it past SWW_WAF_ALERT_MAX_PER_HOUR, and of the repeats held // back in it past SWW_WAF_ALERT_MAX_PER_HOUR, and of the repeats held
// back by the cooldowns dropped as it ends, which no alert let through // back by the cooldowns dropped as it ends, which no alert let through
// has given. It is sent with SWW_WAF_ALERT_MAX_PER_HOUR off too, for // has given. It is sent with SWW_WAF_ALERT_MAX_PER_HOUR off too, for
// those repeats. SWW_WAF_ALERT_EVENTS does not name it. // those repeats. SWW_WAF_ALERT_EVENTS does not name it.
EventSummary = "summary" EventSummary Event = "summary"
) )
// Events returns every event SWW_WAF_ALERT_EVENTS can name, which is its // Events returns every event SWW_WAF_ALERT_EVENTS can name, which is its
// default. // default.
func Events() []string { func Events() []Event {
return []string{ return []Event{
EventBan, EventPermanentBan, EventCoreRuleSetMatch, EventAnomaly, EventBan, EventPermanentBan, EventCoreRuleSetMatch, EventAnomaly,
EventReputationHit, EventSourceFailure, EventFileError, EventReputationHit, EventSourceFailure, EventFileError,
} }
} }
// DestinationName is the name of a destination alerts are sent to:
// DestinationWebhook, DestinationSlack or DestinationNtfy.
type DestinationName string
// The destinations alerts are sent to, as the metrics and alerts.json // The destinations alerts are sent to, as the metrics and alerts.json
// name them. // name them.
const ( const (
// DestinationWebhook is the webhook SWW_WAF_ALERT_WEBHOOK_URL names. // DestinationWebhook is the webhook SWW_WAF_ALERT_WEBHOOK_URL names.
DestinationWebhook = "webhook" DestinationWebhook DestinationName = "webhook"
// DestinationSlack is the Slack incoming webhook // DestinationSlack is the Slack incoming webhook
// SWW_WAF_ALERT_SLACK_WEBHOOK_URL names. // SWW_WAF_ALERT_SLACK_WEBHOOK_URL names.
DestinationSlack = "slack" DestinationSlack DestinationName = "slack"
// DestinationNtfy is the ntfy topic SWW_WAF_ALERT_NTFY_URL names. // DestinationNtfy is the ntfy topic SWW_WAF_ALERT_NTFY_URL names.
DestinationNtfy = "ntfy" DestinationNtfy DestinationName = "ntfy"
) )
// Destinations returns every destination alerts can be sent to. // Destinations returns every destination alerts can be sent to.
func Destinations() []string { func Destinations() []DestinationName {
return []string{DestinationWebhook, DestinationSlack, DestinationNtfy} return []DestinationName{
DestinationWebhook,
DestinationSlack,
DestinationNtfy,
}
} }
const ( const (
@@ -135,7 +146,7 @@ type Params struct {
NtfyURL *url.URL NtfyURL *url.URL
NtfyToken string NtfyToken string
// Events are the events alerts are sent for (SWW_WAF_ALERT_EVENTS). // Events are the events alerts are sent for (SWW_WAF_ALERT_EVENTS).
Events []string Events []Event
// Cooldown is how long a repeat of an alert is held back // Cooldown is how long a repeat of an alert is held back
// (SWW_WAF_ALERT_COOLDOWN), 0 for no time. MaxPerHour is the most alerts // (SWW_WAF_ALERT_COOLDOWN), 0 for no time. MaxPerHour is the most alerts
// sent in an hour (SWW_WAF_ALERT_MAX_PER_HOUR), 0 for no limit. // sent in an hour (SWW_WAF_ALERT_MAX_PER_HOUR), 0 for no limit.
@@ -159,7 +170,7 @@ type Params struct {
type Alert struct { type Alert struct {
Instance string `json:"instance"` Instance string `json:"instance"`
Time time.Time `json:"time"` Time time.Time `json:"time"`
Event string `json:"event"` Event Event `json:"event"`
Host string `json:"host,omitempty"` Host string `json:"host,omitempty"`
Client netip.Addr `json:"client"` Client netip.Addr `json:"client"`
Netblock netip.Prefix `json:"netblock"` Netblock netip.Prefix `json:"netblock"`
@@ -186,7 +197,7 @@ type Alert struct {
// //
//nolint:tagliatelle // the state files use snake_case, as the request log does //nolint:tagliatelle // the state files use snake_case, as the request log does
type Cooldown struct { type Cooldown struct {
Event string `json:"event"` Event Event `json:"event"`
Netblock netip.Prefix `json:"netblock"` Netblock netip.Prefix `json:"netblock"`
File string `json:"file,omitempty"` File string `json:"file,omitempty"`
Source string `json:"source,omitempty"` Source string `json:"source,omitempty"`
@@ -203,18 +214,18 @@ type Cooldown struct {
// //
//nolint:tagliatelle // the state files use snake_case, as the request log does //nolint:tagliatelle // the state files use snake_case, as the request log does
type Hour struct { type Hour struct {
Start time.Time `json:"start"` Start time.Time `json:"start"`
Sent int `json:"sent"` Sent int `json:"sent"`
HeldBack map[string]int `json:"held_back"` HeldBack map[Event]int `json:"held_back"`
} }
// State is what alerts.json holds: the cooldowns, the hour under way, and // State is what alerts.json holds: the cooldowns, the hour under way, and
// for each destination set, the alerts waiting to be sent to it, oldest // for each destination set, the alerts waiting to be sent to it, oldest
// first. // first.
type State struct { type State struct {
Cooldowns []Cooldown `json:"cooldowns"` Cooldowns []Cooldown `json:"cooldowns"`
Hour Hour `json:"hour"` Hour Hour `json:"hour"`
Waiting map[string][]Alert `json:"waiting"` Waiting map[DestinationName][]Alert `json:"waiting"`
} }
// Counts are, for a destination, how many alerts it took, how many // Counts are, for a destination, how many alerts it took, how many
@@ -248,7 +259,7 @@ type destination struct {
// name is how the metrics and alerts.json name the destination, and // name is how the metrics and alerts.json name the destination, and
// setting the setting that is its URL, which the log names in place // setting the setting that is its URL, which the log names in place
// of the URL. // of the URL.
name string name DestinationName
setting string setting string
url *url.URL url *url.URL
// message returns the body an alert is posted with, and the headers // message returns the body an alert is posted with, and the headers
@@ -273,7 +284,7 @@ type destination struct {
// scope with the same AS number or name, as its detail names them. Each // scope with the same AS number or name, as its detail names them. Each
// is empty for an alert without one. // is empty for an alert without one.
type cooldownKey struct { type cooldownKey struct {
event string event Event
netblock netip.Prefix netblock netip.Prefix
file string file string
source string source string
@@ -306,7 +317,7 @@ func New(params Params) *Queue {
q := &Queue{ q := &Queue{
params: params, params: params,
cooldowns: map[cooldownKey]*Cooldown{}, cooldowns: map[cooldownKey]*Cooldown{},
hour: Hour{HeldBack: map[string]int{}}, hour: Hour{HeldBack: map[Event]int{}},
} }
if params.WebhookURL != nil { if params.WebhookURL != nil {
@@ -377,7 +388,7 @@ func (q *Queue) Raise(alert Alert) {
// event, no alert for event on netblock was let through less than // event, no alert for event on netblock was let through less than
// Cooldown before, and fewer than MaxPerHour alerts have been let through // Cooldown before, and fewer than MaxPerHour alerts have been let through
// in the hour under way. Unlike Raise, it counts nothing. // in the hour under way. Unlike Raise, it counts nothing.
func (q *Queue) WouldSend(event string, netblock netip.Prefix) bool { func (q *Queue) WouldSend(event Event, netblock netip.Prefix) bool {
if len(q.destinations) == 0 || !slices.Contains(q.params.Events, event) { if len(q.destinations) == 0 || !slices.Contains(q.params.Events, event) {
return false return false
} }
@@ -433,7 +444,7 @@ func (q *Queue) Run(ctx context.Context) {
// Counts returns the counts of the destination name, all 0 for one not // Counts returns the counts of the destination name, all 0 for one not
// set. // set.
func (q *Queue) Counts(name string) Counts { func (q *Queue) Counts(name DestinationName) Counts {
for _, d := range q.destinations { for _, d := range q.destinations {
if d.name == name { if d.name == name {
return Counts{ return Counts{
@@ -449,8 +460,8 @@ func (q *Queue) Counts(name string) Counts {
// DestinationsSet returns the destinations set, in the order of // DestinationsSet returns the destinations set, in the order of
// Destinations. // Destinations.
func (q *Queue) DestinationsSet() []string { func (q *Queue) DestinationsSet() []DestinationName {
names := make([]string, 0, len(q.destinations)) names := make([]DestinationName, 0, len(q.destinations))
for _, d := range q.destinations { for _, d := range q.destinations {
names = append(names, d.name) names = append(names, d.name)
} }
@@ -474,7 +485,7 @@ func (q *Queue) Snapshot() State {
state := State{ state := State{
Cooldowns: make([]Cooldown, 0, len(q.cooldowns)), Cooldowns: make([]Cooldown, 0, len(q.cooldowns)),
Hour: q.hour, Hour: q.hour,
Waiting: map[string][]Alert{}, Waiting: map[DestinationName][]Alert{},
} }
state.Hour.HeldBack = maps.Clone(q.hour.HeldBack) state.Hour.HeldBack = maps.Clone(q.hour.HeldBack)
@@ -525,7 +536,7 @@ func (q *Queue) Load(state State) {
q.hour.HeldBack = maps.Clone(state.Hour.HeldBack) q.hour.HeldBack = maps.Clone(state.Hour.HeldBack)
if q.hour.HeldBack == nil { if q.hour.HeldBack == nil {
q.hour.HeldBack = map[string]int{} q.hour.HeldBack = map[Event]int{}
} }
for _, d := range q.destinations { for _, d := range q.destinations {
@@ -537,7 +548,7 @@ func (q *Queue) Load(state State) {
// its URL, that URL, target, and message, which makes the messages sent // its URL, that URL, target, and message, which makes the messages sent
// to it. // to it.
func (q *Queue) addDestination( func (q *Queue) addDestination(
name, setting string, target *url.URL, name DestinationName, setting string, target *url.URL,
message func(alert *Alert) ([]byte, http.Header, error), message func(alert *Alert) ([]byte, http.Header, error),
) { ) {
q.destinations = append(q.destinations, &destination{ q.destinations = append(q.destinations, &destination{
@@ -649,7 +660,7 @@ func (q *Queue) endHour(now time.Time) {
}) })
} }
q.hour = Hour{Start: start, HeldBack: map[string]int{}} q.hour = Hour{Start: start, HeldBack: map[Event]int{}}
} }
// queue adds alert to the alerts waiting for each destination. // queue adds alert to the alerts waiting for each destination.
@@ -904,20 +915,21 @@ func (q *Queue) ntfyMessage(alert *Alert) ([]byte, http.Header, error) {
// ntfyPriority returns the priority an alert for event is published to // ntfyPriority returns the priority an alert for event is published to
// ntfy with: high for an event the admin needs to look at. // ntfy with: high for an event the admin needs to look at.
func ntfyPriority(event string) string { func ntfyPriority(event Event) string {
switch event { switch event {
case EventPermanentBan, EventAnomaly, EventSourceFailure, EventFileError: case EventPermanentBan, EventAnomaly, EventSourceFailure, EventFileError:
return "high" return "high"
case EventReputationHit: case EventReputationHit:
return "low" return "low"
default: // ban, core_rule_set_match and summary case EventBan, EventCoreRuleSetMatch, EventSummary:
return "default"
} }
return "default"
} }
// ntfyTag returns the tag an alert for event is published to ntfy with, // ntfyTag returns the tag an alert for event is published to ntfy with,
// which ntfy shows as an emoji. // which ntfy shows as an emoji.
func ntfyTag(event string) string { func ntfyTag(event Event) string {
switch event { switch event {
case EventBan, EventPermanentBan: case EventBan, EventPermanentBan:
return "no_entry" return "no_entry"
@@ -929,15 +941,16 @@ func ntfyTag(event string) string {
return "label" return "label"
case EventSourceFailure, EventFileError: case EventSourceFailure, EventFileError:
return "warning" return "warning"
default: // summary case EventSummary:
return "bar_chart"
} }
return "bar_chart"
} }
// title returns the title of alert in Slack and ntfy: the instance and // title returns the title of alert in Slack and ntfy: the instance and
// the event. // the event.
func title(alert *Alert) string { func title(alert *Alert) string {
return alert.Instance + ": " + alert.Event return alert.Instance + ": " + string(alert.Event)
} }
// text returns the text of alert in Slack and ntfy: its reason, then a // text returns the text of alert in Slack and ntfy: its reason, then a
+31 -20
View File
@@ -126,7 +126,7 @@ func TestOnlyTheChosenEventsAreSent(t *testing.T) {
synctest.Test(t, func(t *testing.T) { synctest.Test(t, func(t *testing.T) {
params := newParams() params := newParams()
params.Events = []string{ params.Events = []alerts.Event{
alerts.EventSourceFailure, alerts.EventSourceFailure,
alerts.EventFileError, alerts.EventFileError,
} }
@@ -156,7 +156,7 @@ func TestWouldSendOnlyForTheChosenEvents(t *testing.T) {
t.Parallel() t.Parallel()
params := newParams() params := newParams()
params.Events = []string{ params.Events = []alerts.Event{
alerts.EventSourceFailure, alerts.EventSourceFailure,
alerts.EventFileError, alerts.EventFileError,
} }
@@ -186,8 +186,8 @@ func TestRaiseDoesNothingWithoutADestination(t *testing.T) {
// No alert waits, no cooldown has started, and the hour counts none. // No alert waits, no cooldown has started, and the hour counts none.
want := alerts.State{ want := alerts.State{
Cooldowns: []alerts.Cooldown{}, Cooldowns: []alerts.Cooldown{},
Hour: alerts.Hour{HeldBack: map[string]int{}}, Hour: alerts.Hour{HeldBack: map[alerts.Event]int{}},
Waiting: map[string][]alerts.Alert{}, Waiting: map[alerts.DestinationName][]alerts.Alert{},
} }
if got := q.Snapshot(); !reflect.DeepEqual(got, want) { if got := q.Snapshot(); !reflect.DeepEqual(got, want) {
t.Errorf("state %+v, want %+v", got, want) t.Errorf("state %+v, want %+v", got, want)
@@ -235,7 +235,7 @@ func TestRepeatWithinTheCooldownIsHeldBackAndCountedInTheNext(t *testing.T) {
synctest.Test(t, func(t *testing.T) { synctest.Test(t, func(t *testing.T) {
webhook, q := start(t, newParams()) webhook, q := start(t, newParams())
raise := func(event string, n int) { raise := func(event alerts.Event, n int) {
q.Raise(alerts.Alert{Event: event, Netblock: netblock(n)}) q.Raise(alerts.Alert{Event: event, Netblock: netblock(n)})
} }
@@ -1014,7 +1014,7 @@ func TestNtfyTokenIsSentToNtfyAlone(t *testing.T) {
) )
synctest.Wait() synctest.Wait()
for destination, want := range map[string][]string{ for destination, want := range map[alerts.DestinationName][]string{
alerts.DestinationWebhook: nil, alerts.DestinationWebhook: nil,
alerts.DestinationSlack: nil, alerts.DestinationSlack: nil,
alerts.DestinationNtfy: tc.want, alerts.DestinationNtfy: tc.want,
@@ -1063,7 +1063,7 @@ func TestDestinationThatDoesNotAnswerHoldsUpNeitherOther(t *testing.T) {
// With no time passed, the webhook and ntfy have taken every alert, // With no time passed, the webhook and ntfy have taken every alert,
// while Slack has not answered the first. // while Slack has not answered the first.
for destination, want := range map[string]alerts.Counts{ for destination, want := range map[alerts.DestinationName]alerts.Counts{
alerts.DestinationWebhook: {Sent: 3}, alerts.DestinationWebhook: {Sent: 3},
alerts.DestinationSlack: {}, alerts.DestinationSlack: {},
alerts.DestinationNtfy: {Sent: 3}, alerts.DestinationNtfy: {Sent: 3},
@@ -1126,7 +1126,7 @@ func TestDropsAreCountedForEachDestination(t *testing.T) {
// the webhook takes every one. // the webhook takes every one.
const all = alerts.QueueSize + 1 const all = alerts.QueueSize + 1
for destination, want := range map[string]alerts.Counts{ for destination, want := range map[alerts.DestinationName]alerts.Counts{
alerts.DestinationWebhook: {Sent: all}, alerts.DestinationWebhook: {Sent: all},
alerts.DestinationSlack: {Dropped: 1}, alerts.DestinationSlack: {Dropped: 1},
alerts.DestinationNtfy: {SendFailures: all, Dropped: all}, alerts.DestinationNtfy: {SendFailures: all, Dropped: all},
@@ -1152,11 +1152,21 @@ func TestEachDestinationIsSentOnlyTheAlertsWaitingForIt(t *testing.T) {
// As read from alerts.json. The alert waiting for the webhook, which // As read from alerts.json. The alert waiting for the webhook, which
// is not set, is dropped. // is not set, is dropped.
q.Load(roundTrip(t, alerts.State{Waiting: map[string][]alerts.Alert{ q.Load(
alerts.DestinationWebhook: {waiting("first")}, roundTrip(
alerts.DestinationSlack: {waiting("second"), waiting("third")}, t,
alerts.DestinationNtfy: {waiting("fourth")}, alerts.State{
}})) Waiting: map[alerts.DestinationName][]alerts.Alert{
alerts.DestinationWebhook: {waiting("first")},
alerts.DestinationSlack: {
waiting("second"),
waiting("third"),
},
alerts.DestinationNtfy: {waiting("fourth")},
},
},
),
)
synctest.Wait() synctest.Wait()
wantBodies(t, standIns[alerts.DestinationSlack], wantBodies(t, standIns[alerts.DestinationSlack],
@@ -1165,7 +1175,7 @@ func TestEachDestinationIsSentOnlyTheAlertsWaitingForIt(t *testing.T) {
wantBodies(t, standIns[alerts.DestinationNtfy], "fourth") wantBodies(t, standIns[alerts.DestinationNtfy], "fourth")
// alerts.json then lists Slack and ntfy alone, with no alert waiting. // alerts.json then lists Slack and ntfy alone, with no alert waiting.
want := map[string][]alerts.Alert{ want := map[alerts.DestinationName][]alerts.Alert{
alerts.DestinationSlack: {}, alerts.DestinationNtfy: {}, alerts.DestinationSlack: {}, alerts.DestinationNtfy: {},
} }
if got := q.Snapshot().Waiting; !reflect.DeepEqual(got, want) { if got := q.Snapshot().Waiting; !reflect.DeepEqual(got, want) {
@@ -1334,11 +1344,11 @@ func start(t *testing.T, params alerts.Params) (*standIn, *alerts.Queue) {
func startAll( func startAll(
t *testing.T, t *testing.T,
params alerts.Params, params alerts.Params,
) (map[string]*standIn, *alerts.Queue) { ) (map[alerts.DestinationName]*standIn, *alerts.Queue) {
t.Helper() t.Helper()
q := alerts.New(params) q := alerts.New(params)
standIns := map[string]*standIn{} standIns := map[alerts.DestinationName]*standIn{}
for _, destination := range q.DestinationsSet() { for _, destination := range q.DestinationsSet() {
standIns[destination] = &standIn{answers: answering} standIns[destination] = &standIn{answers: answering}
@@ -1404,14 +1414,14 @@ func wantAlert(t *testing.T, got, want map[string]any) {
// wantEvents checks the events of the alerts the webhook was sent, in // wantEvents checks the events of the alerts the webhook was sent, in
// order. // order.
func wantEvents(t *testing.T, webhook *standIn, want ...string) { func wantEvents(t *testing.T, webhook *standIn, want ...alerts.Event) {
t.Helper() t.Helper()
got := make([]string, 0, len(webhook.received())) got := make([]alerts.Event, 0, len(webhook.received()))
for _, request := range webhook.received() { for _, request := range webhook.received() {
event, _ := request.alert["event"].(string) event, _ := request.alert["event"].(string)
got = append(got, event) got = append(got, alerts.Event(event))
} }
if !slices.Equal(got, want) { if !slices.Equal(got, want) {
@@ -1447,7 +1457,8 @@ func wantCounts(
// wantDestinationCounts checks q's counts for destination. // wantDestinationCounts checks q's counts for destination.
func wantDestinationCounts( func wantDestinationCounts(
t *testing.T, q *alerts.Queue, destination string, want alerts.Counts, t *testing.T, q *alerts.Queue, destination alerts.DestinationName,
want alerts.Counts,
) { ) {
t.Helper() t.Helper()
+3 -1
View File
@@ -7,7 +7,9 @@ const QueueSize = queueSize
// SetTransport has q's requests to the destination name go through // SetTransport has q's requests to the destination name go through
// transport instead of the network. // transport instead of the network.
func (q *Queue) SetTransport(name string, transport http.RoundTripper) { func (q *Queue) SetTransport(
name DestinationName, transport http.RoundTripper,
) {
for _, d := range q.destinations { for _, d := range q.destinations {
if d.name == name { if d.name == name {
d.httpClient.Transport = transport d.httpClient.Transport = transport
+10 -8
View File
@@ -295,9 +295,10 @@ func (c *Counters) netAround(client netip.Addr) netip.Prefix {
// overThreshold is a count over its threshold: what it counts, requests or // overThreshold is a count over its threshold: what it counts, requests or
// bytes, its window, the count and the threshold. // bytes, its window, the count and the threshold.
type overThreshold struct { type overThreshold struct {
kind, window string kind ratelimit.LimitKind
count float64 window string
threshold int64 count float64
threshold int64
} }
// add counts a request of bytes at now in each of c's counts whose // add counts a request of bytes at now in each of c's counts whose
@@ -346,9 +347,10 @@ func (c *Counter) add(
// bucketCount is one of a counter's four counts: requests or bytes, in a // bucketCount is one of a counter's four counts: requests or bytes, in a
// window of length, and the buckets they are counted in. // window of length, and the buckets they are counted in.
type bucketCount struct { type bucketCount struct {
kind, window string kind ratelimit.LimitKind
length time.Duration window string
buckets *ratelimit.Buckets length time.Duration
buckets *ratelimit.Buckets
} }
// counts returns c's counts: requests and bytes in the minute, then in // counts returns c's counts: requests and bytes in the minute, then in
@@ -404,8 +406,8 @@ func (k key) counter() *Counter {
// counted, the count and the threshold. // counted, the count and the threshold.
func alertFor(r Request, k key, o overThreshold) alerts.Alert { func alertFor(r Request, k key, o overThreshold) alerts.Alert {
detail := map[string]any{ detail := map[string]any{
"scope": k.scope, "window": o.window, "kind": o.kind, "count": o.count, "scope": k.scope, "window": o.window, "kind": string(o.kind),
"threshold": o.threshold, "count": o.count, "threshold": o.threshold,
} }
var counted string var counted string
+8 -4
View File
@@ -54,7 +54,11 @@ func TestBlacklistLeavesANetblockWithAnActiveBanAsItIs(t *testing.T) {
now := midnight() now := midnight()
scores := map[netip.Prefix]int64{netblock: 100} scores := map[netip.Prefix]int64{netblock: 100}
limitBan, _ := ledger.BanForLimit(netblock, now, bans.Notes{}) limitBan, _ := ledger.Ban(
netblock,
now,
bans.Grounds{Cause: bans.CauseLimit},
)
if made := ledger.BanForAbuseIPDB(now, now.Add(30*day), scores); made != 0 { if made := ledger.BanForAbuseIPDB(now, now.Add(30*day), scores); made != 0 {
t.Errorf("made %d bans over a ban for a limit, want none", made) t.Errorf("made %d bans over a ban for a limit, want none", made)
@@ -119,7 +123,7 @@ func TestEndedBlacklistBansAreForgottenAtTheNextFetch(t *testing.T) {
expires := now.Add(30 * day) expires := now.Add(30 * day)
ledger.BanForAbuseIPDB(now, expires, map[netip.Prefix]int64{listed: 100}) ledger.BanForAbuseIPDB(now, expires, map[netip.Prefix]int64{listed: 100})
ledger.BanForLimit(limited, now, bans.Notes{}) ledger.Ban(limited, now, bans.Grounds{Cause: bans.CauseLimit})
// listed is off the blacklist when it is fetched after the ban's end. // listed is off the blacklist when it is fetched after the ban's end.
if made := ledger.BanForAbuseIPDB(expires, expires.Add(30*day), nil); made != 0 { if made := ledger.BanForAbuseIPDB(expires, expires.Add(30*day), nil); made != 0 {
@@ -144,7 +148,7 @@ func TestBlacklistBansDoNotCountTowardMaxBans(t *testing.T) {
scores[netip.PrefixFrom(netip.AddrFrom4([4]byte{198, 51, 100, byte(i)}), 32)] = 100 scores[netip.PrefixFrom(netip.AddrFrom4([4]byte{198, 51, 100, byte(i)}), 32)] = 100
} }
ledger.BanForLimit(limited, now, bans.Notes{}) ledger.Ban(limited, now, bans.Grounds{Cause: bans.CauseLimit})
if made := ledger.BanForAbuseIPDB(now, now.Add(30*day), scores); made != 3 { if made := ledger.BanForAbuseIPDB(now, now.Add(30*day), scores); made != 3 {
t.Errorf("made %d bans for 3 netblocks, want 3", made) t.Errorf("made %d bans for 3 netblocks, want 3", made)
@@ -154,7 +158,7 @@ func TestBlacklistBansDoNotCountTowardMaxBans(t *testing.T) {
// The ban for an attack takes the place of the one for a limit, and no // The ban for an attack takes the place of the one for a limit, and no
// blacklist ban goes. // blacklist ban goes.
ledger.BanForAttack(attacker, now, bans.Notes{}) ledger.Ban(attacker, now, bans.Grounds{Cause: bans.CauseAttack})
want := map[netip.Prefix]int{limited: 0, attacker: 1} want := map[netip.Prefix]int{limited: 0, attacker: 1}
for netblock := range scores { for netblock := range scores {
+64 -23
View File
@@ -61,7 +61,7 @@ func TestAdminsBansAreNeverDroppedAndDoNotCountTowardMaxBans(t *testing.T) {
wantBans(t, ledger, map[netip.Prefix]int{adminsOnly: 2, both: 2}) wantBans(t, ledger, map[netip.Prefix]int{adminsOnly: 2, both: 2})
// A new ban drops the ban sww made, and only that one. // A new ban drops the ban sww made, and only that one.
ledger.BanForLimit(second, midnight(), bans.Notes{}) ledger.Ban(second, midnight(), bans.Grounds{Cause: bans.CauseLimit})
wantBans( wantBans(
t, t,
ledger, ledger,
@@ -77,7 +77,7 @@ func TestAdminsBansAreNeverDroppedAndDoNotCountTowardMaxBans(t *testing.T) {
} }
// And the next drops that one. // And the next drops that one.
ledger.BanForLimit(third, midnight(), bans.Notes{}) ledger.Ban(third, midnight(), bans.Grounds{Cause: bans.CauseLimit})
wantBans( wantBans(
t, t,
ledger, ledger,
@@ -98,20 +98,28 @@ func TestEarlierBansCountABanDroppedAfterAnAdminsBan(t *testing.T) {
// twice, each time after its last ban has ended. // twice, each time after its last ban has ended.
ledger.BanForAdmin(netblock, midnight(), midnight().Add(time.Minute), ledger.BanForAdmin(netblock, midnight(), midnight().Add(time.Minute),
"probes for logins") "probes for logins")
first, _ := ledger.BanForLimit( first, _ := ledger.Ban(
netblock, netblock,
midnight().Add(time.Minute), midnight().Add(time.Minute),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
)
second, _ := ledger.Ban(
netblock,
first.Expires,
bans.Grounds{Cause: bans.CauseLimit},
) )
second, _ := ledger.BanForLimit(netblock, first.Expires, bans.Notes{})
// A ban on another netblock fills the ledger, and the netblock's first // A ban on another netblock fills the ledger, and the netblock's first
// ban for a limit, its earliest that is not an admin's, goes. // ban for a limit, its earliest that is not an admin's, goes.
ledger.BanForLimit(other, second.Start, bans.Notes{}) ledger.Ban(other, second.Start, bans.Grounds{Cause: bans.CauseLimit})
wantBans(t, ledger, map[netip.Prefix]int{netblock: 2, other: 1}) wantBans(t, ledger, map[netip.Prefix]int{netblock: 2, other: 1})
// The netblock's next ban still counts it. // The netblock's next ban still counts it.
third, _ := ledger.BanForLimit(netblock, second.Expires, bans.Notes{}) third, _ := ledger.Ban(
netblock,
second.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
if third.Notes.EarlierBans != (bans.EarlierBans{Limit: 2, Admin: 1}) { if third.Notes.EarlierBans != (bans.EarlierBans{Limit: 2, Admin: 1}) {
t.Errorf("the third ban for a limit has earlier bans %+v, "+ t.Errorf("the third ban for a limit has earlier bans %+v, "+
"want 2 for a limit and 1 an admin's", third.Notes.EarlierBans) "want 2 for a limit and 1 an admin's", third.Notes.EarlierBans)
@@ -127,15 +135,27 @@ func TestEarlierBansCountABanAnAdminKeepsOnce(t *testing.T) {
// The netblock breaks a limit twice, the second time after its first // The netblock breaks a limit twice, the second time after its first
// ban has ended, and the second ban's notes count the first. An admin // ban has ended, and the second ban's notes count the first. An admin
// then keeps the first ban. // then keeps the first ban.
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{}) first, _ := ledger.Ban(
second, _ := ledger.BanForLimit(netblock, first.Expires, bans.Notes{}) netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
second, _ := ledger.Ban(
netblock,
first.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
kept := first kept := first
kept.Cause = bans.CauseAdmin kept.Cause = bans.CauseAdmin
ledger.LoadEdit([]bans.Ban{kept, second}) ledger.LoadEdit([]bans.Ban{kept, second})
// The netblock's next ban counts the kept ban once, as an admin's. // The netblock's next ban counts the kept ban once, as an admin's.
third, _ := ledger.BanForLimit(netblock, second.Expires, bans.Notes{}) third, _ := ledger.Ban(
netblock,
second.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
if third.Notes.EarlierBans != (bans.EarlierBans{Limit: 1, Admin: 1}) { if third.Notes.EarlierBans != (bans.EarlierBans{Limit: 1, Admin: 1}) {
t.Errorf("the third ban for a limit has earlier bans %+v, "+ t.Errorf("the third ban for a limit has earlier bans %+v, "+
"want 1 for a limit and 1 an admin's", third.Notes.EarlierBans) "want 1 for a limit and 1 an admin's", third.Notes.EarlierBans)
@@ -147,20 +167,37 @@ func TestReasonOfTheBansSwwMakes(t *testing.T) {
ledger := bans.New(defaultParams()) ledger := bans.New(defaultParams())
limit, _ := ledger.BanForLimit( limit, _ := ledger.Ban(
netip.MustParsePrefix("203.0.113.1/32"), netip.MustParsePrefix("203.0.113.1/32"),
midnight(), midnight(),
bans.Notes{Kind: "requests", Limit: 1000, Window: "minute"}, bans.Grounds{
Cause: bans.CauseLimit,
Notes: bans.Notes{
Kind: "requests",
Limit: 1000,
Window: "minute",
},
},
) )
byteLimit, _ := ledger.BanForLimit( byteLimit, _ := ledger.Ban(
netip.MustParsePrefix("203.0.113.3/32"), netip.MustParsePrefix("203.0.113.3/32"),
midnight(), midnight(),
bans.Notes{Kind: "bytes", Limit: 10 << 30, Window: "hour"}, bans.Grounds{
Cause: bans.CauseLimit,
Notes: bans.Notes{
Kind: "bytes",
Limit: 10 << 30,
Window: "hour",
},
},
) )
attack, _ := ledger.BanForAttack( attack, _ := ledger.Ban(
netip.MustParsePrefix("203.0.113.2/32"), netip.MustParsePrefix("203.0.113.2/32"),
midnight(), midnight(),
bans.Notes{RuleID: "git-dir", Target: "path"}, bans.Grounds{
Cause: bans.CauseAttack,
Notes: bans.Notes{RuleID: "git-dir", Target: "path"},
},
) )
for _, tc := range []struct{ got, want string }{ for _, tc := range []struct{ got, want string }{
@@ -200,7 +237,7 @@ func TestLiftedBanForALimitRefusesNothingAndMakesNoBanLonger(t *testing.T) {
t.Error("the lifted ban refuses") t.Error("the lifted ban refuses")
} }
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{}) ban, _ := ledger.Ban(netblock, now, bans.Grounds{Cause: bans.CauseLimit})
if ban.Expires.Sub(ban.Start) != time.Hour || if ban.Expires.Sub(ban.Start) != time.Hour ||
ban.Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) { ban.Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) {
t.Errorf( t.Errorf(
@@ -251,7 +288,11 @@ func TestLiftedBanForAnAttackRefusesNothingAndMakesNoBanLonger(
} }
// The next clear sign of attack bans for seven days, as a first does. // The next clear sign of attack bans for seven days, as a first does.
ban, _ := ledger.BanForAttack(netblock, now, bans.Notes{}) ban, _ := ledger.Ban(
netblock,
now,
bans.Grounds{Cause: bans.CauseAttack},
)
if ban.Expires.Sub(ban.Start) != 7*day { if ban.Expires.Sub(ban.Start) != 7*day {
t.Errorf( t.Errorf(
"the next ban for an attack ends at %s, want seven days on", "the next ban for an attack ends at %s, want seven days on",
@@ -264,10 +305,10 @@ func TestLoadEditCountsTheBansAnAdminMade(t *testing.T) {
t.Parallel() t.Parallel()
ledger := bans.New(defaultParams()) ledger := bans.New(defaultParams())
made, _ := ledger.BanForLimit( made, _ := ledger.Ban(
netip.MustParsePrefix("203.0.113.1/32"), netip.MustParsePrefix("203.0.113.1/32"),
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
atStart := bans.Ban{ atStart := bans.Ban{
Netblock: netip.MustParsePrefix("203.0.113.2/32"), Netblock: netip.MustParsePrefix("203.0.113.2/32"),
@@ -311,7 +352,7 @@ func TestAdminsBanIsMadeWhileAnotherLasts(t *testing.T) {
ledger := bans.New(defaultParams()) ledger := bans.New(defaultParams())
// An hour's ban for a broken limit. // An hour's ban for a broken limit.
ledger.BanForLimit(netblock, midnight(), bans.Notes{}) ledger.Ban(netblock, midnight(), bans.Grounds{Cause: bans.CauseLimit})
wantChanged(t, ledger, true) wantChanged(t, ledger, true)
// A minute later an admin bans the netblock for good, named by an // A minute later an admin bans the netblock for good, named by an
@@ -431,10 +472,10 @@ func TestPermanentBanOnAWiderNetblockIsReturnedOverATemporaryBan(
ledger := bans.New(defaultParams()) ledger := bans.New(defaultParams())
// The temporary ban is made first, so that its netblock is looked at first. // The temporary ban is made first, so that its netblock is looked at first.
ledger.BanForLimit( ledger.Ban(
netip.MustParsePrefix("203.0.113.9/32"), netip.MustParsePrefix("203.0.113.9/32"),
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
ledger.BanForAdmin( ledger.BanForAdmin(
netip.MustParsePrefix("203.0.113.0/24"), netip.MustParsePrefix("203.0.113.0/24"),
+76 -97
View File
@@ -16,20 +16,26 @@ import (
"time" "time"
"github.com/hashicorp/golang-lru/v2/simplelru" "github.com/hashicorp/golang-lru/v2/simplelru"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
) )
// Cause is why a ban was made: CauseLimit, CauseAttack, CauseAdmin or
// CauseAbuseIPDB.
type Cause string
// The causes of bans. // The causes of bans.
const ( const (
// CauseLimit is a ban sww made for a broken limit. // CauseLimit is a ban sww made for a broken limit.
CauseLimit = "limit" CauseLimit Cause = "limit"
// CauseAttack is a ban sww made for a clear sign of attack. // CauseAttack is a ban sww made for a clear sign of attack.
CauseAttack = "attack" CauseAttack Cause = "attack"
// CauseAdmin is a ban an admin made, or one sww made that an // CauseAdmin is a ban an admin made, or one sww made that an
// admin keeps. It is never dropped. // admin keeps. It is never dropped.
CauseAdmin = "admin" CauseAdmin Cause = "admin"
// CauseAbuseIPDB is a ban sww made for a client on AbuseIPDB's // CauseAbuseIPDB is a ban sww made for a client on AbuseIPDB's
// blacklist. It is forgotten once it has ended. // blacklist. It is forgotten once it has ended.
CauseAbuseIPDB = "abuseipdb" CauseAbuseIPDB Cause = "abuseipdb"
) )
// repeatFactor is how many times as long as the netblock's last ban a ban // repeatFactor is how many times as long as the netblock's last ban a ban
@@ -68,8 +74,7 @@ type Ban struct {
Start time.Time Start time.Time
// Expires is when the ban ends, zero for a permanent ban. // Expires is when the ban ends, zero for a permanent ban.
Expires time.Time Expires time.Time
// Cause is CauseLimit, CauseAttack, CauseAdmin or CauseAbuseIPDB. Cause Cause
Cause string
// Reason is a short text: for a ban sww made, the limit broken, the // Reason is a short text: for a ban sww made, the limit broken, the
// rule that matched or AbuseIPDB's score; for an admin's, what the // rule that matched or AbuseIPDB's score; for an admin's, what the
// admin wrote. // admin wrote.
@@ -111,10 +116,10 @@ type Notes struct {
// request that broke the limit included. // request that broke the limit included.
// These are what counted toward the ban, and the window is the time // These are what counted toward the ban, and the window is the time
// over which they came. // over which they came.
Kind string `json:"kind,omitempty"` Kind ratelimit.LimitKind `json:"kind,omitempty"`
Limit int64 `json:"limit,omitempty"` Limit int64 `json:"limit,omitempty"`
Window string `json:"window,omitempty"` Window string `json:"window,omitempty"`
Count float64 `json:"count,omitempty"` Count float64 `json:"count,omitempty"`
// LimitPercent and LimitPercentSetting are, for a ban for a limit a // LimitPercent and LimitPercentSetting are, for a ban for a limit a
// biased threshold lowered, the client's percentage of that kind of // biased threshold lowered, the client's percentage of that kind of
// limit, of which Limit is the result, and the setting that gave it. // limit, of which Limit is the result, and the setting that gave it.
@@ -194,7 +199,7 @@ type Ledger struct {
// made is how many bans have been made since the start, by cause: by // made is how many bans have been made since the start, by cause: by
// the ledger, and by an admin, through BanForAdmin or in an edit of // the ledger, and by an admin, through BanForAdmin or in an edit of
// bans.json. // bans.json.
made map[string]int made map[Cause]int
// v4Lengths and v6Lengths are the lengths of the IPv4 and IPv6 // v4Lengths and v6Lengths are the lengths of the IPv4 and IPv6
// netblocks that have been banned. Check looks for a ban at each of // netblocks that have been banned. Check looks for a ban at each of
// them, so that a ban read from bans.json refuses every client in its // them, so that a ban read from bans.json refuses every client in its
@@ -221,7 +226,7 @@ func New(params Params) *Ledger {
params: params, params: params,
changed: make(chan struct{}, 1), changed: make(chan struct{}, 1),
netblocks: netblocks, netblocks: netblocks,
made: map[string]int{}, made: map[Cause]int{},
} }
} }
@@ -300,83 +305,52 @@ func activeBan(bans []Ban, now time.Time) *Ban {
return newest return newest
} }
// BanForLimit bans netblock at now for a broken limit, with notes, and // Grounds are why a request makes a ban: its cause, CauseLimit or
// returns the ban, and true. A first ban lasts LimitBanDuration. A ban // CauseAttack, and its notes.
// made within LimitBanRepeatWindow after the netblock's ban that ended type Grounds struct {
// last, other than one for a clear sign of attack or a lifted one, lasts Cause Cause
// repeatFactor times as long as that one. A ban that would be longer Notes Notes
// than LimitBanPermanentAbove is permanent instead. If a ban on netblock is still
// active, as when two of its requests break a limit at once, that ban is
// returned with false, and no other is made. The ledger fills in the
// notes' Refused and EarlierBans itself, and gives the ban the reason
// "<Kind> per <Window> over the limit of <Limit>", from the notes, such
// as "requests per minute over the limit of 1000".
func (l *Ledger) BanForLimit(
netblock netip.Prefix, now time.Time, notes Notes,
) (Ban, bool) {
return l.ban(
netblock,
now,
CauseLimit,
limitReason(notes),
notes,
true,
)
} }
// WouldBanForLimit returns what BanForLimit would, without making the ban: // Ban bans netblock at now on grounds, and returns the ban, and true. If
// what observe mode would have done. // a ban on netblock is still active, as when two of its requests break a
func (l *Ledger) WouldBanForLimit( // limit at once, that ban is returned with false, and no other is made.
netblock netip.Prefix, now time.Time, notes Notes, // The ledger fills in the notes' Refused and EarlierBans itself.
//
// A ban for a broken limit:
// - lasts LimitBanDuration the first time;
// - lasts repeatFactor times as long as the netblock's ban that ended
// last, when made within LimitBanRepeatWindow after it, unless that
// ban was for a clear sign of attack or was lifted;
// - is permanent instead when it would be longer than
// LimitBanPermanentAbove;
// - has the reason "<Kind> per <Window> over the limit of <Limit>", from
// the notes, such as "requests per minute over the limit of 1000".
//
// A ban for a clear sign of attack:
// - lasts AttackBanDuration the first time;
// - is permanent once the netblock has had one that was not lifted;
// - has the reason "matched the rule <RuleID>", or "asked for the trap
// path <TrapPath>".
func (l *Ledger) Ban(
netblock netip.Prefix, now time.Time, grounds Grounds,
) (Ban, bool) { ) (Ban, bool) {
return l.ban( return l.ban(netblock, now, grounds, true)
netblock,
now,
CauseLimit,
limitReason(notes),
notes,
false,
)
} }
// BanForAttack bans netblock at now for a clear sign of attack, with // WouldBan returns what Ban would, without making the ban: what observe
// notes, and returns the ban, and whether it made it, as BanForLimit // mode would have done.
// does. A first ban lasts AttackBanDuration; once the netblock has had func (l *Ledger) WouldBan(
// one that was not lifted, the next is permanent. Its reason is "matched netblock netip.Prefix, now time.Time, grounds Grounds,
// the rule <RuleID>", or "asked for the trap path <TrapPath>".
func (l *Ledger) BanForAttack(
netblock netip.Prefix, now time.Time, notes Notes,
) (Ban, bool) { ) (Ban, bool) {
return l.ban( return l.ban(netblock, now, grounds, false)
netblock,
now,
CauseAttack,
attackReason(notes),
notes,
true,
)
}
// WouldBanForAttack returns what BanForAttack would, without making the
// ban: what observe mode would have done.
func (l *Ledger) WouldBanForAttack(
netblock netip.Prefix, now time.Time, notes Notes,
) (Ban, bool) {
return l.ban(
netblock,
now,
CauseAttack,
attackReason(notes),
notes,
false,
)
} }
// WouldBePermanent reports whether a ban on netblock for cause, CauseLimit // WouldBePermanent reports whether a ban on netblock for cause, CauseLimit
// or CauseAttack, made at now would be permanent, as BanForLimit or // or CauseAttack, made at now would be permanent, as Ban would make it. It
// BanForAttack would make it. It works out nothing else of the ban. // works out nothing else of the ban.
func (l *Ledger) WouldBePermanent( func (l *Ledger) WouldBePermanent(
netblock netip.Prefix, now time.Time, cause string, netblock netip.Prefix, now time.Time, cause Cause,
) bool { ) bool {
l.mu.Lock() l.mu.Lock()
defer l.mu.Unlock() defer l.mu.Unlock()
@@ -411,10 +385,9 @@ func attackReason(notes Notes) string {
// BanForAdmin bans netblock at now for an admin, with reason, until // BanForAdmin bans netblock at now for an admin, with reason, until
// expires, or for good when expires is zero, and returns the ban, whose // expires, or for good when expires is zero, and returns the ban, whose
// cause is CauseAdmin. Unlike BanForLimit and BanForAttack, it makes the // cause is CauseAdmin. Unlike Ban, it makes the ban even while another
// ban even while another on netblock is active, since the admin asked // on netblock is active, since the admin asked for this one. The ledger
// for this one. The ledger fills in the notes' EarlierBans, and counts // fills in the notes' EarlierBans, and counts the ban among those made.
// the ban among those made.
func (l *Ledger) BanForAdmin( func (l *Ledger) BanForAdmin(
netblock netip.Prefix, now, expires time.Time, reason string, netblock netip.Prefix, now, expires time.Time, reason string,
) Ban { ) Ban {
@@ -471,7 +444,7 @@ func (l *Ledger) BanForAbuseIPDB(
// The source is named as the request log names AbuseIPDB. // The source is named as the request log names AbuseIPDB.
notes.Reputation = []ReputationHit{ notes.Reputation = []ReputationHit{
{Source: CauseAbuseIPDB, Score: &score}, {Source: string(CauseAbuseIPDB), Score: &score},
} }
l.add(Ban{ l.add(Ban{
@@ -609,7 +582,7 @@ func (l *Ledger) AddLookup(
// for CauseLimit, CauseAttack and CauseAbuseIPDB, by the ledger; for // for CauseLimit, CauseAttack and CauseAbuseIPDB, by the ledger; for
// CauseAdmin, by an admin, with BanForAdmin or in an edit of bans.json, as // CauseAdmin, by an admin, with BanForAdmin or in an edit of bans.json, as
// LoadEdit counts them. The bans read from bans.json at the start are not among them. // LoadEdit counts them. The bans read from bans.json at the start are not among them.
func (l *Ledger) Made(cause string) int { func (l *Ledger) Made(cause Cause) int {
l.mu.Lock() l.mu.Lock()
defer l.mu.Unlock() defer l.mu.Unlock()
@@ -759,15 +732,13 @@ func (l *Ledger) forgetEndedAbuseIPDBBans(now time.Time) bool {
return forgot return forgot
} }
// ban bans netblock at now for cause, CauseLimit or CauseAttack, with // ban bans netblock at now on grounds, as Ban describes, and returns the
// reason and notes, as BanForLimit and BanForAttack describe, and returns // ban, and whether it made it. Unless keep is true, the ban is not made,
// the ban, and whether it made it. Unless keep is true, the ban is not // only returned: it is the ban that would have been made.
// made, only returned: it is the ban that would have been made.
func (l *Ledger) ban( func (l *Ledger) ban(
netblock netip.Prefix, netblock netip.Prefix,
now time.Time, now time.Time,
cause, reason string, grounds Grounds,
notes Notes,
keep bool, keep bool,
) (Ban, bool) { ) (Ban, bool) {
l.mu.Lock() l.mu.Lock()
@@ -776,6 +747,8 @@ func (l *Ledger) ban(
// held are the netblock's bans, none of them active. // held are the netblock's bans, none of them active.
var held []Ban var held []Ban
notes := grounds.Notes
bans, found := l.netblocks.Get(netblock) bans, found := l.netblocks.Get(netblock)
if found { if found {
active := activeBan(*bans, now) active := activeBan(*bans, now)
@@ -791,15 +764,21 @@ func (l *Ledger) ban(
ban := Ban{ ban := Ban{
Netblock: netblock, Netblock: netblock,
Start: now, Start: now,
Cause: cause, Cause: grounds.Cause,
Reason: reason,
Notes: notes, Notes: notes,
} }
if cause == CauseAttack { switch grounds.Cause {
ban.Expires = l.attackExpiry(held, now) case CauseLimit:
} else { ban.Reason = limitReason(notes)
ban.Expires = l.limitExpiry(held, now) ban.Expires = l.limitExpiry(held, now)
case CauseAttack:
ban.Reason = attackReason(notes)
ban.Expires = l.attackExpiry(held, now)
case CauseAdmin, CauseAbuseIPDB:
fallthrough
default:
panic("bans: a request's ban has the cause " + string(grounds.Cause))
} }
if !keep { if !keep {
@@ -807,7 +786,7 @@ func (l *Ledger) ban(
} }
l.add(ban) l.add(ban)
l.made[cause]++ l.made[grounds.Cause]++
l.markChanged() l.markChanged()
return ban, true return ban, true
+121 -51
View File
@@ -22,7 +22,11 @@ func TestRepeatsTripleUntilPermanent(t *testing.T) {
// Each ban is followed by another as soon as it ends: 1, 3, 9, 27 and // Each ban is followed by another as soon as it ends: 1, 3, 9, 27 and
// 81 hours. // 81 hours.
for i, hours := range []int{1, 3, 9, 27, 81} { for i, hours := range []int{1, 3, 9, 27, 81} {
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{}) ban, _ := ledger.Ban(
netblock,
now,
bans.Grounds{Cause: bans.CauseLimit},
)
length := time.Duration(hours) * time.Hour length := time.Duration(hours) * time.Hour
if !ban.Expires.Equal(now.Add(length)) || if !ban.Expires.Equal(now.Add(length)) ||
@@ -42,7 +46,7 @@ func TestRepeatsTripleUntilPermanent(t *testing.T) {
// The sixth would last 243 hours, more than seven days: it is // The sixth would last 243 hours, more than seven days: it is
// permanent, and never ends. // permanent, and never ends.
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{}) ban, _ := ledger.Ban(netblock, now, bans.Grounds{Cause: bans.CauseLimit})
if !ban.Permanent() { if !ban.Permanent() {
t.Fatalf("sixth ban ends at %s, want a permanent one", ban.Expires) t.Fatalf("sixth ban ends at %s, want a permanent one", ban.Expires)
} }
@@ -71,15 +75,15 @@ func TestRepeatWindowRunsOut(t *testing.T) {
ledger := bans.New(defaultParams()) ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
first, _ := ledger.BanForLimit( first, _ := ledger.Ban(
netblock, netblock,
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
second, _ := ledger.BanForLimit( second, _ := ledger.Ban(
netblock, netblock,
first.Expires.Add(tc.gap), first.Expires.Add(tc.gap),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
if second.Expires.Sub(second.Start) != tc.want || if second.Expires.Sub(second.Start) != tc.want ||
@@ -104,10 +108,10 @@ func TestFirstBanLongerThanLimitBanPermanentAboveIsPermanent(t *testing.T) {
params.LimitBanDuration = params.LimitBanPermanentAbove + time.Hour params.LimitBanDuration = params.LimitBanPermanentAbove + time.Hour
ledger := bans.New(params) ledger := bans.New(params)
ban, _ := ledger.BanForLimit( ban, _ := ledger.Ban(
netip.MustParsePrefix("203.0.113.9/32"), netip.MustParsePrefix("203.0.113.9/32"),
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
if !ban.Permanent() { if !ban.Permanent() {
t.Errorf("first ban ends at %s, want a permanent one", ban.Expires) t.Errorf("first ban ends at %s, want a permanent one", ban.Expires)
@@ -127,7 +131,11 @@ func TestLimitBanPermanentAboveSetFarOffDoesNotOverflow(t *testing.T) {
now := midnight() now := midnight()
for i := range 14 { for i := range 14 {
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{}) ban, _ := ledger.Ban(
netblock,
now,
bans.Grounds{Cause: bans.CauseLimit},
)
if !ban.Expires.After(ban.Start) { if !ban.Expires.After(ban.Start) {
t.Fatalf( t.Fatalf(
"ban %d starts at %s and ends at %s", "ban %d starts at %s and ends at %s",
@@ -140,7 +148,7 @@ func TestLimitBanPermanentAboveSetFarOffDoesNotOverflow(t *testing.T) {
now = ban.Expires now = ban.Expires
} }
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{}) ban, _ := ledger.Ban(netblock, now, bans.Grounds{Cause: bans.CauseLimit})
if !ban.Permanent() { if !ban.Permanent() {
t.Errorf("15th ban ends at %s, want a permanent one", ban.Expires) t.Errorf("15th ban ends at %s, want a permanent one", ban.Expires)
} }
@@ -152,15 +160,19 @@ func TestBrokenLimitDuringABanMakesNoOther(t *testing.T) {
ledger := bans.New(defaultParams()) ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
first, made := ledger.BanForLimit(netblock, midnight(), bans.Notes{}) first, made := ledger.Ban(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
if !made { if !made {
t.Error("the first ban was not made") t.Error("the first ban was not made")
} }
again, made := ledger.BanForLimit( again, made := ledger.Ban(
netblock, netblock,
midnight().Add(time.Minute), midnight().Add(time.Minute),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
if made || !reflect.DeepEqual(again, first) || if made || !reflect.DeepEqual(again, first) ||
@@ -175,10 +187,10 @@ func TestBrokenLimitDuringABanMakesNoOther(t *testing.T) {
) )
} }
again, made = ledger.BanForAttack( again, made = ledger.Ban(
netblock, netblock,
midnight().Add(time.Minute), midnight().Add(time.Minute),
bans.Notes{}, bans.Grounds{Cause: bans.CauseAttack},
) )
if made || !reflect.DeepEqual(again, first) { if made || !reflect.DeepEqual(again, first) {
t.Errorf( t.Errorf(
@@ -195,10 +207,10 @@ func TestCheckRefusesWhileTheBanLastsAndCountsTheRefusals(t *testing.T) {
ledger := bans.New(defaultParams()) ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
ban, _ := ledger.BanForLimit( ban, _ := ledger.Ban(
netblock, netblock,
midnight(), midnight(),
bans.Notes{Requests: 5}, bans.Grounds{Cause: bans.CauseLimit, Notes: bans.Notes{Requests: 5}},
) )
for range 3 { for range 3 {
@@ -237,10 +249,10 @@ func TestFindCountsNothing(t *testing.T) {
ledger := bans.New(defaultParams()) ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
ban, _ := ledger.BanForLimit( ban, _ := ledger.Ban(
netblock, netblock,
midnight(), midnight(),
bans.Notes{Requests: 5}, bans.Grounds{Cause: bans.CauseLimit, Notes: bans.Notes{Requests: 5}},
) )
got, banned, _ := ledger.Find( got, banned, _ := ledger.Find(
@@ -287,24 +299,24 @@ func TestMaxBansDropsTheEarliestBanOfTheNetblockSeenLongestAgo(
d := netip.MustParsePrefix("2001:db8::/64") d := netip.MustParsePrefix("2001:db8::/64")
now := midnight() now := midnight()
first, _ := ledger.BanForLimit(a, now, bans.Notes{}) first, _ := ledger.Ban(a, now, bans.Grounds{Cause: bans.CauseLimit})
ledger.BanForLimit(b, now, bans.Notes{}) ledger.Ban(b, now, bans.Grounds{Cause: bans.CauseLimit})
ledger.BanForLimit(c, now, bans.Notes{}) ledger.Ban(c, now, bans.Grounds{Cause: bans.CauseLimit})
// A request from a makes b the netblock seen longest ago, and its ban // A request from a makes b the netblock seen longest ago, and its ban
// goes to make room for d's. // goes to make room for d's.
ledger.Check(a.Addr(), now) ledger.Check(a.Addr(), now)
ledger.BanForLimit(d, now, bans.Notes{}) ledger.Ban(d, now, bans.Grounds{Cause: bans.CauseLimit})
wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 0, c: 1, d: 1}) wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 0, c: 1, d: 1})
// a is banned again once its ban has ended; c, seen longest ago, goes. // a is banned again once its ban has ended; c, seen longest ago, goes.
ledger.BanForLimit(a, first.Expires, bans.Notes{}) ledger.Ban(a, first.Expires, bans.Grounds{Cause: bans.CauseLimit})
wantBans(t, ledger, map[netip.Prefix]int{a: 2, c: 0, d: 1}) wantBans(t, ledger, map[netip.Prefix]int{a: 2, c: 0, d: 1})
// With d seen since, a is seen longest ago, and its earlier ban goes // With d seen since, a is seen longest ago, and its earlier ban goes
// first. // first.
ledger.Check(d.Addr(), first.Expires) ledger.Check(d.Addr(), first.Expires)
ledger.BanForLimit(b, first.Expires, bans.Notes{}) ledger.Ban(b, first.Expires, bans.Grounds{Cause: bans.CauseLimit})
wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 1, d: 1}) wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 1, d: 1})
if !ledger.Bans(a)[0].Start.Equal(first.Expires) { if !ledger.Bans(a)[0].Start.Equal(first.Expires) {
@@ -325,8 +337,16 @@ func TestFullLedgerDropsTheEarlierBanOfTheNetblockBannedAgain(t *testing.T) {
ledger := bans.New(params) ledger := bans.New(params)
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{}) first, _ := ledger.Ban(
second, _ := ledger.BanForLimit(netblock, first.Expires, bans.Notes{}) netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
second, _ := ledger.Ban(
netblock,
first.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
held := ledger.Bans(netblock) held := ledger.Bans(netblock)
if len(held) != 1 || !reflect.DeepEqual(held[0], second) || if len(held) != 1 || !reflect.DeepEqual(held[0], second) ||
@@ -343,7 +363,11 @@ func TestRequestDuringAnAttackBanMakesItPermanent(t *testing.T) {
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
notes := bans.Notes{RuleID: "env-file", Target: "path"} notes := bans.Notes{RuleID: "env-file", Target: "path"}
ban, _ := ledger.BanForAttack(netblock, midnight(), notes) ban, _ := ledger.Ban(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseAttack, Notes: notes},
)
if !ban.Expires.Equal(midnight().Add(7*day)) || if !ban.Expires.Equal(midnight().Add(7*day)) ||
ban.Cause != bans.CauseAttack || ban.Cause != bans.CauseAttack ||
ban.Notes.RuleID != "env-file" || ban.Notes.RuleID != "env-file" ||
@@ -418,8 +442,16 @@ func TestAttackAfterAnAttackBanHasEndedBansPermanently(t *testing.T) {
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
// A ban for a broken limit before does not count. // A ban for a broken limit before does not count.
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{}) first, _ := ledger.Ban(
second, _ := ledger.BanForAttack(netblock, first.Expires, bans.Notes{}) netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
second, _ := ledger.Ban(
netblock,
first.Expires,
bans.Grounds{Cause: bans.CauseAttack},
)
if second.Expires.Sub(second.Start) != 7*day { if second.Expires.Sub(second.Start) != 7*day {
t.Fatalf("the first ban for an attack lasts %s, want 7 days", t.Fatalf("the first ban for an attack lasts %s, want 7 days",
@@ -435,10 +467,10 @@ func TestAttackAfterAnAttackBanHasEndedBansPermanently(t *testing.T) {
// Its notes show the earlier ban for an attack that makes it permanent, // Its notes show the earlier ban for an attack that makes it permanent,
// beside the one for a limit. // beside the one for a limit.
third, _ := ledger.BanForAttack( third, _ := ledger.Ban(
netblock, netblock,
second.Expires.Add(30*day), second.Expires.Add(30*day),
bans.Notes{}, bans.Grounds{Cause: bans.CauseAttack},
) )
if !third.Permanent() || if !third.Permanent() ||
third.Notes.EarlierBans != (bans.EarlierBans{Limit: 1, Attack: 1}) { third.Notes.EarlierBans != (bans.EarlierBans{Limit: 1, Attack: 1}) {
@@ -452,14 +484,18 @@ func TestWouldBanGivesTheBanWithoutMakingIt(t *testing.T) {
ledger := bans.New(defaultParams()) ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{}) first, _ := ledger.Ban(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
wantChanged(t, ledger, true) wantChanged(t, ledger, true)
// While the first ban lasts, none would be made. // While the first ban lasts, none would be made.
during, would := ledger.WouldBanForAttack( during, would := ledger.WouldBan(
netblock, netblock,
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseAttack},
) )
if would || !reflect.DeepEqual(during, first) { if would || !reflect.DeepEqual(during, first) {
t.Errorf( t.Errorf(
@@ -472,13 +508,19 @@ func TestWouldBanGivesTheBanWithoutMakingIt(t *testing.T) {
// As it ends, a clear sign of attack would ban for seven days, and a // As it ends, a clear sign of attack would ban for seven days, and a
// limit broken again for three hours, but neither is made. // limit broken again for three hours, but neither is made.
limitNotes := bans.Notes{Kind: "requests", Limit: 1, Window: "minute"} limitGrounds := bans.Grounds{
attack, wouldAttack := ledger.WouldBanForAttack(netblock, first.Expires, Cause: bans.CauseLimit,
bans.Notes{RuleID: "git-dir"}) Notes: bans.Notes{Kind: "requests", Limit: 1, Window: "minute"},
limit, wouldLimit := ledger.WouldBanForLimit( }
attack, wouldAttack := ledger.WouldBan(netblock, first.Expires,
bans.Grounds{
Cause: bans.CauseAttack,
Notes: bans.Notes{RuleID: "git-dir"},
})
limit, wouldLimit := ledger.WouldBan(
netblock, netblock,
first.Expires, first.Expires,
limitNotes, limitGrounds,
) )
if !wouldAttack || !attack.Expires.Equal(first.Expires.Add(7*day)) || if !wouldAttack || !attack.Expires.Equal(first.Expires.Add(7*day)) ||
@@ -505,7 +547,7 @@ func TestWouldBanGivesTheBanWithoutMakingIt(t *testing.T) {
wantChanged(t, ledger, false) wantChanged(t, ledger, false)
// The ban made is the one that would have been. // The ban made is the one that would have been.
made, _ := ledger.BanForLimit(netblock, first.Expires, limitNotes) made, _ := ledger.Ban(netblock, first.Expires, limitGrounds)
if !reflect.DeepEqual(made, limit) { if !reflect.DeepEqual(made, limit) {
t.Errorf("the ban made is %+v, want %+v", made, limit) t.Errorf("the ban made is %+v, want %+v", made, limit)
} }
@@ -526,7 +568,11 @@ func TestWouldBePermanentAnswersAsTheBanWouldBeMade(t *testing.T) {
t.Fatalf("ban %d for a limit would be permanent", i+1) t.Fatalf("ban %d for a limit would be permanent", i+1)
} }
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{}) ban, _ := ledger.Ban(
netblock,
now,
bans.Grounds{Cause: bans.CauseLimit},
)
now = ban.Expires now = ban.Expires
} }
@@ -539,7 +585,11 @@ func TestWouldBePermanentAnswersAsTheBanWouldBeMade(t *testing.T) {
} }
// Once a first ban for an attack has ended, the next would be permanent. // Once a first ban for an attack has ended, the next would be permanent.
attack, _ := ledger.BanForAttack(netblock, now, bans.Notes{}) attack, _ := ledger.Ban(
netblock,
now,
bans.Grounds{Cause: bans.CauseAttack},
)
if !ledger.WouldBePermanent(netblock, attack.Expires, bans.CauseAttack) { if !ledger.WouldBePermanent(netblock, attack.Expires, bans.CauseAttack) {
t.Error("a second ban for an attack would not be permanent") t.Error("a second ban for an attack would not be permanent")
} }
@@ -553,8 +603,16 @@ func TestAttackBanDoesNotLengthenTheNextBanForALimit(t *testing.T) {
// Three times the seven days would be permanent; a limit broken as the // Three times the seven days would be permanent; a limit broken as the
// ban for an attack ends bans for an hour, as a first broken limit does. // ban for an attack ends bans for an hour, as a first broken limit does.
attack, _ := ledger.BanForAttack(netblock, midnight(), bans.Notes{}) attack, _ := ledger.Ban(
limit, _ := ledger.BanForLimit(netblock, attack.Expires, bans.Notes{}) netblock,
midnight(),
bans.Grounds{Cause: bans.CauseAttack},
)
limit, _ := ledger.Ban(
netblock,
attack.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
if limit.Expires.Sub(limit.Start) != time.Hour || if limit.Expires.Sub(limit.Start) != time.Hour ||
limit.Cause != bans.CauseLimit { limit.Cause != bans.CauseLimit {
@@ -578,10 +636,13 @@ func TestRequestTextsAreCutTo256Bytes(t *testing.T) {
Time: midnight(), Method: long, Host: long, Path: long, Status: 403, UserAgent: long, Time: midnight(), Method: long, Host: long, Path: long, Status: 403, UserAgent: long,
} }
ban, _ := ledger.BanForLimit( ban, _ := ledger.Ban(
netblock, netblock,
midnight(), midnight(),
bans.Notes{Request: request}, bans.Grounds{
Cause: bans.CauseLimit,
Notes: bans.Notes{Request: request},
},
) )
cut := long[:256] cut := long[:256]
@@ -607,11 +668,20 @@ func TestLookupFillsTheNotesOfTheNetblocksBansWithoutOne(t *testing.T) {
// A ban made with the client's lookup, one made before it came, after // A ban made with the client's lookup, one made before it came, after
// the first ended, and one on another netblock. // the first ended, and one on another netblock.
ledger.BanForLimit(netblock, midnight(), bans.Notes{ ledger.Ban(netblock, midnight(), bans.Grounds{
ASN: "AS64497", ASName: "Other Net", Country: "FR", Cause: bans.CauseLimit,
Notes: bans.Notes{
ASN: "AS64497",
ASName: "Other Net",
Country: "FR",
},
}) })
ledger.BanForLimit(netblock, midnight().Add(time.Hour), bans.Notes{}) ledger.Ban(
ledger.BanForLimit(other, midnight(), bans.Notes{}) netblock,
midnight().Add(time.Hour),
bans.Grounds{Cause: bans.CauseLimit},
)
ledger.Ban(other, midnight(), bans.Grounds{Cause: bans.CauseLimit})
ledger.AddLookup(netblock, "AS64496", "Example Net", "DE") ledger.AddLookup(netblock, "AS64496", "Example Net", "DE")
+36 -21
View File
@@ -19,25 +19,29 @@ func TestChangedAfterABanIsMade(t *testing.T) {
wantChanged(t, ledger, false) wantChanged(t, ledger, false)
ledger.BanForLimit(netblock, midnight(), bans.Notes{}) ledger.Ban(netblock, midnight(), bans.Grounds{Cause: bans.CauseLimit})
wantChanged(t, ledger, true) wantChanged(t, ledger, true)
// A limit broken during the ban makes no other, and a refusal changes // A limit broken during the ban makes no other, and a refusal changes
// only the counts in the notes, which wait for the interval's write. // only the counts in the notes, which wait for the interval's write.
ledger.BanForLimit(netblock, midnight().Add(time.Minute), bans.Notes{}) ledger.Ban(
netblock,
midnight().Add(time.Minute),
bans.Grounds{Cause: bans.CauseLimit},
)
ledger.Check(netblock.Addr(), midnight().Add(time.Minute)) ledger.Check(netblock.Addr(), midnight().Add(time.Minute))
wantChanged(t, ledger, false) wantChanged(t, ledger, false)
// Two bans before the value is read leave one. // Two bans before the value is read leave one.
ledger.BanForLimit( ledger.Ban(
netip.MustParsePrefix("203.0.113.10/32"), netip.MustParsePrefix("203.0.113.10/32"),
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
ledger.BanForLimit( ledger.Ban(
netip.MustParsePrefix("203.0.113.11/32"), netip.MustParsePrefix("203.0.113.11/32"),
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
wantChanged(t, ledger, true) wantChanged(t, ledger, true)
wantChanged(t, ledger, false) wantChanged(t, ledger, false)
@@ -51,10 +55,14 @@ func TestSnapshotListsEveryBanByNetblock(t *testing.T) {
high := netip.MustParsePrefix("203.0.113.10/32") high := netip.MustParsePrefix("203.0.113.10/32")
low := netip.MustParsePrefix("203.0.113.9/32") low := netip.MustParsePrefix("203.0.113.9/32")
first, _ := ledger.BanForLimit(v6, midnight(), bans.Notes{}) first, _ := ledger.Ban(
ledger.BanForLimit(high, midnight(), bans.Notes{}) v6,
ledger.BanForLimit(low, midnight(), bans.Notes{}) midnight(),
ledger.BanForLimit(v6, first.Expires, bans.Notes{}) bans.Grounds{Cause: bans.CauseLimit},
)
ledger.Ban(high, midnight(), bans.Grounds{Cause: bans.CauseLimit})
ledger.Ban(low, midnight(), bans.Grounds{Cause: bans.CauseLimit})
ledger.Ban(v6, first.Expires, bans.Grounds{Cause: bans.CauseLimit})
snapshot := ledger.Snapshot() snapshot := ledger.Snapshot()
@@ -80,7 +88,10 @@ func TestLoadedBansCarryOn(t *testing.T) {
before := bans.New(defaultParams()) before := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
ban, _ := before.BanForLimit(netblock, midnight(), bans.Notes{Limit: 1}) ban, _ := before.Ban(netblock, midnight(), bans.Grounds{
Cause: bans.CauseLimit,
Notes: bans.Notes{Limit: 1},
})
// Loaded into a new ledger, as across a restart, the ban still refuses // Loaded into a new ledger, as across a restart, the ban still refuses
// while it lasts, and once it has ended a broken limit bans for three // while it lasts, and once it has ended a broken limit bans for three
@@ -96,7 +107,11 @@ func TestLoadedBansCarryOn(t *testing.T) {
t.Error("the loaded ban does not refuse") t.Error("the loaded ban does not refuse")
} }
again, _ := after.BanForLimit(netblock, ban.Expires, bans.Notes{}) again, _ := after.Ban(
netblock,
ban.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
if again.Expires.Sub(again.Start) != 3*time.Hour || if again.Expires.Sub(again.Start) != 3*time.Hour ||
again.Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) { again.Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) {
t.Errorf( t.Errorf(
@@ -124,10 +139,10 @@ func TestLoadedBanRefusesEveryClientInItsNetblock(t *testing.T) {
Start: midnight(), Start: midnight(),
}, },
}) })
ledger.BanForLimit( ledger.Ban(
netip.MustParsePrefix("198.51.100.7/32"), netip.MustParsePrefix("198.51.100.7/32"),
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
for client, want := range map[string]bool{ for client, want := range map[string]bool{
@@ -200,7 +215,7 @@ func TestPermanentBanStartedBeforeAnEndedOneRefuses(t *testing.T) {
} }
// A limit broken now makes no shorter ban over the permanent one. // A limit broken now makes no shorter ban over the permanent one.
ban, _ = ledger.BanForLimit(netblock, now, bans.Notes{}) ban, _ = ledger.Ban(netblock, now, bans.Grounds{Cause: bans.CauseLimit})
if !ban.Permanent() || len(ledger.Bans(netblock)) != 2 { if !ban.Permanent() || len(ledger.Bans(netblock)) != 2 {
t.Errorf( t.Errorf(
"a broken limit returned %+v and left the netblock %d bans, "+ "a broken limit returned %+v and left the netblock %d bans, "+
@@ -236,10 +251,10 @@ func TestNextBanWorkedOutFromTheBanThatEndedLast(t *testing.T) {
// Once both have ended, a limit broken within the repeat window bans // Once both have ended, a limit broken within the repeat window bans
// for three times the 9 hours, and the notes count the two bans // for three times the 9 hours, and the notes count the two bans
// before the 9-hour one and it, for a limit, and the admin's. // before the 9-hour one and it, for a limit, and the admin's.
ban, _ := ledger.BanForLimit( ban, _ := ledger.Ban(
netblock, netblock,
nineHours.Expires.Add(time.Hour), nineHours.Expires.Add(time.Hour),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
if ban.Expires.Sub(ban.Start) != 27*time.Hour || if ban.Expires.Sub(ban.Start) != 27*time.Hour ||
ban.Notes.EarlierBans != (bans.EarlierBans{Limit: 3, Admin: 1}) { ban.Notes.EarlierBans != (bans.EarlierBans{Limit: 3, Admin: 1}) {
@@ -313,15 +328,15 @@ func TestLoadReplacesTheBansHeld(t *testing.T) {
} }
// The ledger holds one ban, so it makes two more without dropping any. // The ledger holds one ban, so it makes two more without dropping any.
first, _ := ledger.BanForLimit( first, _ := ledger.Ban(
netip.MustParsePrefix("198.51.100.7/32"), netip.MustParsePrefix("198.51.100.7/32"),
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
second, _ := ledger.BanForLimit( second, _ := ledger.Ban(
netip.MustParsePrefix("198.51.100.8/32"), netip.MustParsePrefix("198.51.100.8/32"),
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
want := []bans.Ban{first, second, kept} want := []bans.Ban{first, second, kept}
+4 -25
View File
@@ -19,6 +19,8 @@ import (
"time" "time"
"github.com/spf13/viper" "github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/reputation"
) )
// off is the value that switches a timeout, a size limit or a rate limit // off is the value that switches a timeout, a size limit or a rate limit
@@ -40,9 +42,7 @@ var (
errNotPositive = errors.New("must be more than zero, or off") errNotPositive = errors.New("must be more than zero, or off")
errNotMilliseconds = errors.New( errNotMilliseconds = errors.New(
"is not a whole number of milliseconds above zero, such as 2000") "is not a whole number of milliseconds above zero, such as 2000")
errEmptyItem = errors.New("has an empty item in its list") errEmptyItem = errors.New("has an empty item in its list")
errNotNetblock = errors.New(
"is not a netblock such as 10.0.0.0/8, or an address")
errNotListenAddr = errors.New( errNotListenAddr = errors.New(
"is not an address to listen on, such as :8080") "is not an address to listen on, such as :8080")
errNotCommand = errors.New("is not rproxy or waf") errNotCommand = errors.New("is not rproxy or waf")
@@ -253,7 +253,7 @@ func parseNetblocks(value string) ([]netip.Prefix, error) {
netblocks := make([]netip.Prefix, 0, len(items)) netblocks := make([]netip.Prefix, 0, len(items))
for _, item := range items { for _, item := range items {
netblock, err := ParseNetblock(item) netblock, err := reputation.ParseNetblock(item)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -264,27 +264,6 @@ func parseNetblocks(value string) ([]netip.Prefix, error) {
return netblocks, nil return netblocks, nil
} }
// ParseNetblock reads a netblock in CIDR form, such as 10.0.0.0/8. A bare
// address is a netblock of that address alone, a /32 or a /128. A
// blocklist's lines are read with it too.
func ParseNetblock(value string) (netip.Prefix, error) {
if strings.Contains(value, "/") {
netblock, err := netip.ParsePrefix(value)
if err != nil {
return netip.Prefix{}, fmt.Errorf("%q %w", value, errNotNetblock)
}
return netblock.Masked(), nil
}
addr, err := netip.ParseAddr(value)
if err != nil || addr.Zone() != "" {
return netip.Prefix{}, fmt.Errorf("%q %w", value, errNotNetblock)
}
return netip.PrefixFrom(addr, addr.BitLen()), nil
}
// parseListenAddr checks an address to listen on: an optional host and a // parseListenAddr checks an address to listen on: an optional host and a
// port number. // port number.
func parseListenAddr(value string) (string, error) { func parseListenAddr(value string) (string, error) {
+10 -5
View File
@@ -10,6 +10,8 @@ import (
"time" "time"
"github.com/spf13/viper" "github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// RProxy is the settings of `sww rproxy`. A size or time limit of zero // RProxy is the settings of `sww rproxy`. A size or time limit of zero
@@ -32,7 +34,7 @@ type RProxy struct {
// cannot be connected to, does not answer at all, or does not answer // cannot be connected to, does not answer at all, or does not answer
// within WAFBackendTimeout: allow or deny // within WAFBackendTimeout: allow or deny
// (SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION). // (SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION).
WAFBackendNoAnswerDisposition string WAFBackendNoAnswerDisposition wafprotocol.Disposition
// MaxRequestSize is the largest request body (SWW_RPROXY_MAX_REQUEST_SIZE), and // MaxRequestSize is the largest request body (SWW_RPROXY_MAX_REQUEST_SIZE), and
// MaxResponseSize the largest response body (SWW_RPROXY_MAX_RESPONSE_SIZE). // MaxResponseSize the largest response body (SWW_RPROXY_MAX_RESPONSE_SIZE).
MaxRequestSize int64 MaxRequestSize int64
@@ -94,7 +96,7 @@ func RProxyFromEnvironment(v *viper.Viper) (*RProxy, error) {
), ),
WAFBackendNoAnswerDisposition: env.disposition( WAFBackendNoAnswerDisposition: env.disposition(
"SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION", "SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION",
"allow", wafprotocol.DispositionAllow,
), ),
MaxRequestSize: env.size("SWW_RPROXY_MAX_REQUEST_SIZE", "100M"), MaxRequestSize: env.size("SWW_RPROXY_MAX_REQUEST_SIZE", "100M"),
MaxResponseSize: env.size("SWW_RPROXY_MAX_RESPONSE_SIZE", "1G"), MaxResponseSize: env.size("SWW_RPROXY_MAX_RESPONSE_SIZE", "1G"),
@@ -166,9 +168,12 @@ func (e *environment) wafBackend(name string) *url.URL {
// disposition reads the setting that is what is done with a request when waf // disposition reads the setting that is what is done with a request when waf
// cannot be connected to, does not answer at all, or does not answer in time: // cannot be connected to, does not answer at all, or does not answer in time:
// allow or deny. // allow or deny.
func (e *environment) disposition(name, defaultValue string) string { func (e *environment) disposition(
value := e.value(name, defaultValue) name string, defaultValue wafprotocol.Disposition,
if value != "allow" && value != "deny" { ) wafprotocol.Disposition {
value := wafprotocol.Disposition(e.value(name, string(defaultValue)))
if value != wafprotocol.DispositionAllow &&
value != wafprotocol.DispositionDeny {
e.check(name, fmt.Errorf("%q %w", value, errNotDisposition)) e.check(name, fmt.Errorf("%q %w", value, errNotDisposition))
} }
+1637 -1495
View File
File diff suppressed because it is too large. Load diff
+45 -22
View File
@@ -18,10 +18,13 @@ import (
"time" "time"
"github.com/spf13/viper" "github.com/spf13/viper"
"golang.org/x/net/http/httpguts"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/anomaly" "sneak.berlin/go/smallwebwaf/internal/anomaly"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/config/configtest" "sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/reputation"
) )
// The settings, by name. // The settings, by name.
@@ -186,6 +189,16 @@ const (
defaultAlertCooldown = "15m" defaultAlertCooldown = "15m"
) )
// joinEvents returns events as SWW_WAF_ALERT_EVENTS lists them.
func joinEvents(events []alerts.Event) string {
names := make([]string, 0, len(events))
for _, event := range events {
names = append(names, string(event))
}
return strings.Join(names, ",")
}
// defaultIPReputationCacheTTL is the default of SWW_WAF_IP_REPUTATION_CACHE_TTL. // defaultIPReputationCacheTTL is the default of SWW_WAF_IP_REPUTATION_CACHE_TTL.
const defaultIPReputationCacheTTL = "24h" const defaultIPReputationCacheTTL = "24h"
@@ -864,6 +877,32 @@ func TestInstanceNameAndLoggedHeadersAsSet(t *testing.T) {
} }
} }
// config and rules check header names with httpguts.ValidHeaderFieldName.
// It takes a name of one or more of the characters RFC 9110 allows in one,
// as the check it replaced did: letters, digits and these marks.
func TestHeaderNameCheckTakesTheRFC9110Characters(t *testing.T) {
t.Parallel()
const marks = "!#$%&'*+-.^_`|~"
if httpguts.ValidHeaderFieldName("") {
t.Error("the empty name is taken")
}
for b := range 256 {
char := byte(b)
want := 'a' <= char && char <= 'z' || 'A' <= char && char <= 'Z' ||
'0' <= char && char <= '9' || strings.IndexByte(marks, char) >= 0
// string(char) would be the UTF-8 of U+0080 to U+00FF, two bytes,
// for a byte above 0x7f.
name := string([]byte{char})
if got := httpguts.ValidHeaderFieldName(name); got != want {
t.Errorf("the name %q is taken: %t, want %t", name, got, want)
}
}
}
func TestRemoteLogSettingsDefaults(t *testing.T) { func TestRemoteLogSettingsDefaults(t *testing.T) {
t.Parallel() t.Parallel()
@@ -1082,7 +1121,7 @@ func TestAlertSettingsDefaults(t *testing.T) {
if cfg.AlertWebhookURL != nil || len(cfg.AlertWebhookHeaders) != 0 || if cfg.AlertWebhookURL != nil || len(cfg.AlertWebhookHeaders) != 0 ||
cfg.AlertSlackWebhookURL != nil || cfg.AlertNtfyURL != nil || cfg.AlertSlackWebhookURL != nil || cfg.AlertNtfyURL != nil ||
cfg.AlertNtfyToken != "" || cfg.AlertNtfyToken != "" ||
strings.Join(cfg.AlertEvents, ",") != defaultAlertEvents || joinEvents(cfg.AlertEvents) != defaultAlertEvents ||
cfg.AlertCooldown != 15*time.Minute || cfg.AlertMaxPerHour != 60 { cfg.AlertCooldown != 15*time.Minute || cfg.AlertMaxPerHour != 60 {
t.Errorf( t.Errorf(
"alert settings %v, %v, %v, %v, %q, %v, %s and %d, want no URLs, "+ "alert settings %v, %v, %v, %v, %q, %v, %s and %d, want no URLs, "+
@@ -1129,7 +1168,10 @@ func TestAlertSettingsAsSet(t *testing.T) {
!reflect.DeepEqual(cfg.AlertWebhookHeaders, headers) || !reflect.DeepEqual(cfg.AlertWebhookHeaders, headers) ||
cfg.AlertSlackWebhookURL.String() != slack || cfg.AlertNtfyURL.String() != ntfy || cfg.AlertSlackWebhookURL.String() != slack || cfg.AlertNtfyURL.String() != ntfy ||
cfg.AlertNtfyToken != token || cfg.AlertNtfyToken != token ||
!slices.Equal(cfg.AlertEvents, []string{"ban", "file_error"}) || !slices.Equal(
cfg.AlertEvents,
[]alerts.Event{"ban", "file_error"},
) ||
cfg.AlertCooldown != time.Hour || cfg.AlertMaxPerHour != 10 { cfg.AlertCooldown != time.Hour || cfg.AlertMaxPerHour != 10 {
t.Errorf( t.Errorf(
"alert settings %v, %v, %v, %v, %q, %v, %s and %d", "alert settings %v, %v, %v, %v, %q, %v, %s and %d",
@@ -1694,7 +1736,7 @@ func TestTagSettingsAsSet(t *testing.T) {
tagLimitPercent: "vpn=mullvad :50, asn=014061:25, COUNTRY=cn:1, dnsbl=zen.example:99", tagLimitPercent: "vpn=mullvad :50, asn=014061:25, COUNTRY=cn:1, dnsbl=zen.example:99",
}) })
wantLists := []config.TagList{ wantLists := []reputation.TagList{
{Name: torExitTag, URL: torURL}, {Name: torExitTag, URL: torURL},
{Name: "vpn=mullvad", URL: "https://example.com/mullvad.txt"}, {Name: "vpn=mullvad", URL: "https://example.com/mullvad.txt"},
} }
@@ -2288,25 +2330,6 @@ func TestInvalidDNSBLSettingStopsTheStartSayingWhatIsWrong(t *testing.T) {
} }
} }
func TestMaskZoneKeyMasksTheFirstLabelOfAZoneUnderDqSpamhausNet(
t *testing.T,
) {
t.Parallel()
for zone, want := range map[string]string{
spamhaus: spamhausMasked,
spamhaus + ".": spamhausMasked + ".",
"KEY.ZEN.DQ.SPAMHAUS.NET": "********.ZEN.DQ.SPAMHAUS.NET",
dronebl: dronebl,
"dq.spamhaus.net": "dq.spamhaus.net",
spamhaus + ".example": spamhaus + ".example",
} {
if got := config.MaskZoneKey(zone); got != want {
t.Errorf("MaskZoneKey(%q) is %q, want %q", zone, got, want)
}
}
}
func TestDNSBLZoneKeyIsLoggedMaskedAndNeverShown(t *testing.T) { func TestDNSBLZoneKeyIsLoggedMaskedAndNeverShown(t *testing.T) {
t.Parallel() t.Parallel()
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
@@ -37,9 +37,10 @@
SecRule &TX:app_nextcloud "@eq 0" \ SecRule &TX:app_nextcloud "@eq 0" \
"id:900090,phase:1,pass,nolog,skipAfter:END_NEXTCLOUD" "id:900090,phase:1,pass,nolog,skipAfter:END_NEXTCLOUD"
# The sync client and the Files app use these WebDAV methods besides those # The sync client and the Files app use these WebDAV methods besides PUT,
# the first change allows, on /remote.php/dav/ and /public.php/dav/ and on the # PATCH and DELETE, which sww allows for every app, on /remote.php/dav/ and
# older /remote.php/webdav/ and /public.php/webdav/. 911100 refused each: # /public.php/dav/ and on the older /remote.php/webdav/ and /public.php/webdav/.
# 911100 refused each:
# PROPFIND, PROPPATCH: listing a folder, marking a file a favourite # PROPFIND, PROPPATCH: listing a folder, marking a file a favourite
# MKCOL: making a folder, and starting a chunked upload # MKCOL: making a folder, and starting a chunked upload
# COPY, MOVE: copying, moving and renaming, and ending a chunked upload # COPY, MOVE: copying, moving and renaming, and ending a chunked upload
File renamed without changes.
+43 -65
View File
@@ -12,11 +12,10 @@
package coreruleset package coreruleset
import ( import (
_ "embed" "embed"
"errors" "errors"
"fmt" "fmt"
"io" "io"
"maps"
"net" "net"
"net/http" "net/http"
"net/netip" "net/netip"
@@ -35,7 +34,7 @@ import (
// directives are the Core Rule Set as sww runs it, with the // directives are the Core Rule Set as sww runs it, with the
// paranoia level for %d, bodyDirectives for the first %s while // paranoia level for %d, bodyDirectives for the first %s while
// SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is set, the exclusions file for // SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is set, the exclusions file for
// the second, and the directives of every app in apps for the third. // the second, and the fixes of every app in appFixes for the third.
// Each rule sww adds has an id from 900000 to 900999, the ids the // Each rule sww adds has an id from 900000 to 900999, the ids the
// Core Rule Set keeps for the rules that set it up, which // Core Rule Set keeps for the rules that set it up, which
// SWW_WAF_CORE_RULE_SET_DISABLED_RULES and the exclusions file refuse, so that no // SWW_WAF_CORE_RULE_SET_DISABLED_RULES and the exclusions file refuse, so that no
@@ -54,13 +53,12 @@ Include @crs-setup.conf.example
SecAction "id:900000,phase:1,pass,nolog,\ SecAction "id:900000,phase:1,pass,nolog,\
setvar:tx.blocking_paranoia_level=%d" setvar:tx.blocking_paranoia_level=%d"
# The first change: PUT, PATCH and DELETE are allowed besides GET, HEAD, # PUT, PATCH and DELETE are allowed besides GET, HEAD, POST and OPTIONS.
# POST and OPTIONS.
SecAction "id:900200,phase:1,pass,nolog,\ SecAction "id:900200,phase:1,pass,nolog,\
setvar:'tx.allowed_methods=GET HEAD POST OPTIONS PUT PATCH DELETE'" setvar:'tx.allowed_methods=GET HEAD POST OPTIONS PUT PATCH DELETE'"
# The second: Expect and Content-Encoding are taken off the Core Rule Set's # Expect and Content-Encoding are taken off the Core Rule Set's list of the
# list of the headers it refuses. Content-Encoding goes back on it for a # headers it refuses. Content-Encoding goes back on it for a
# body the Core Rule Set reads (900260 in bodyDirectives). # body the Core Rule Set reads (900260 in bodyDirectives).
SecAction "id:900250,phase:1,pass,nolog,\ SecAction "id:900250,phase:1,pass,nolog,\
setvar:'tx.restricted_headers_basic=/proxy/ /lock-token/ /content-range/ \ setvar:'tx.restricted_headers_basic=/proxy/ /lock-token/ /content-range/ \
@@ -86,18 +84,17 @@ SecRule ARGUMENTS_LIMIT_REACHED "@eq 1" "id:900300,phase:2,pass,\
# The apps' fixes come before the rules for the same reason. # The apps' fixes come before the rules for the same reason.
%s %s
# The fifth: only the rules for requests are loaded, and no response is # Only the rules for requests are loaded, and no response is inspected.
# inspected.
Include @owasp_crs/REQUEST-*.conf Include @owasp_crs/REQUEST-*.conf
# The third: redirect_uri is not checked for a URL naming an IP address or # redirect_uri is not checked for a URL naming an IP address or
# localhost. Coraza matches a parameter name here without regard to case. # localhost. Coraza matches a parameter name here without regard to case.
# ARGS holds the fields of a form data or multipart body Coraza reads as # ARGS holds the fields of a form data or multipart body Coraza reads as
# well as the query parameters, so a field of this name is left out too. # well as the query parameters, so a field of this name is left out too.
SecRuleUpdateTargetById 931100 "!ARGS:redirect_uri" SecRuleUpdateTargetById 931100 "!ARGS:redirect_uri"
SecRuleUpdateTargetById 934110 "!ARGS:redirect_uri" SecRuleUpdateTargetById 934110 "!ARGS:redirect_uri"
# The fourth: Referer is not checked for a Unix command without arguments # Referer is not checked for a Unix command without arguments
# or for Java starting a process. # or for Java starting a process.
SecRuleUpdateTargetById 932340 "!REQUEST_HEADERS:Referer" SecRuleUpdateTargetById 932340 "!REQUEST_HEADERS:Referer"
SecRuleUpdateTargetById 944110 "!REQUEST_HEADERS:Referer" SecRuleUpdateTargetById 944110 "!REQUEST_HEADERS:Referer"
@@ -122,9 +119,9 @@ SecRule REQUEST_HEADERS:Content-Type \
"@rx ^(?:application|text)/(?:[a-z0-9.-]+[+])?json" \ "@rx ^(?:application|text)/(?:[a-z0-9.-]+[+])?json" \
"id:900420,phase:1,pass,nolog,t:none,t:lowercase,ctl:requestBodyProcessor=JSON" "id:900420,phase:1,pass,nolog,t:none,t:lowercase,ctl:requestBodyProcessor=JSON"
# The rest of the second change: Content-Encoding is refused again on a # Content-Encoding goes back on the list of the headers the Core Rule Set
# body of a kind the Core Rule Set reads, since a compressed body cannot be # refuses (900250 in directives) for a body of a kind it reads, since a
# inspected. # compressed body cannot be inspected.
SecRule REQBODY_PROCESSOR "@rx ^(?:URLENCODED|MULTIPART|JSON|XML)$" \ SecRule REQBODY_PROCESSOR "@rx ^(?:URLENCODED|MULTIPART|JSON|XML)$" \
"id:900260,phase:1,pass,nolog,\ "id:900260,phase:1,pass,nolog,\
setvar:'tx.restricted_headers_basic=%%{tx.restricted_headers_basic} \ setvar:'tx.restricted_headers_basic=%%{tx.restricted_headers_basic} \
@@ -145,15 +142,19 @@ SecRule MULTIPART_STRICT_ERROR "!@eq 0" "id:900450,phase:2,pass,\
severity:'CRITICAL',setvar:'tx.inbound_anomaly_score_pl1=+5'" severity:'CRITICAL',setvar:'tx.inbound_anomaly_score_pl1=+5'"
` `
// app is an app whose ordinary requests the Core Rule Set would refuse, // appFixes holds the fixes of each app whose ordinary requests the Core
// with the fixes that let them pass. // Rule Set would refuse, in a file named for the app by the name
// SWW_WAF_APPS gives it, such as apps/gitea.conf. The fixes are Coraza
// exclusions, loaded before the Core Rule Set's rules. Their first rule
// skips to the marker they end with, unless Inspect sets the transaction
// variable named for the app, such as app_gitea. The skip reaches only
// phase 1, so a rule of a later phase must check that variable itself.
//
//go:embed apps/*.conf
var appFixes embed.FS
// app is what an app's file in appFixes cannot say about it.
type app struct { type app struct {
// directives are Coraza exclusions, loaded before the Core Rule Set's
// rules. Their first rule skips to the marker they end with, unless
// Inspect sets the transaction variable named for the app, such as
// app_gitea. The skip reaches only phase 1, so a rule of a later phase
// must check that variable itself.
directives string
// cookies are the cookies the Core Rule Set reads a request without. // cookies are the cookies the Core Rule Set reads a request without.
cookies []string cookies []string
// forkOf names the app this one is a fork of, "" for none. A hostname // forkOf names the app this one is a fork of, "" for none. A hostname
@@ -161,60 +162,34 @@ type app struct {
forkOf string forkOf string
} }
//go:embed gitea.conf // apps are the apps that need more than their file in appFixes, by the
var giteaDirectives string // name SWW_WAF_APPS gives each. flash is Forgejo's name for gitea_flash.
//go:embed wordpress.conf
var wordpressDirectives string
//go:embed drupal.conf
var drupalDirectives string
//go:embed mediawiki.conf
var mediawikiDirectives string
//go:embed mastodon.conf
var mastodonDirectives string
//go:embed vaultwarden.conf
var vaultwardenDirectives string
//go:embed nextcloud.conf
var nextcloudDirectives string
//go:embed forgejo.conf
var forgejoDirectives string
//go:embed ghost.conf
var ghostDirectives string
// apps are the apps whose fixes sww has, by the name SWW_WAF_APPS gives each.
// flash is Forgejo's name for gitea_flash.
// //
//nolint:gochecknoglobals // a constant cannot be a map //nolint:gochecknoglobals // a constant cannot be a map
var apps = map[string]app{ var apps = map[string]app{
"drupal": {directives: drupalDirectives},
"forgejo": { "forgejo": {
directives: forgejoDirectives, cookies: []string{"flash"},
cookies: []string{"flash"}, forkOf: "gitea",
forkOf: "gitea",
}, },
"ghost": {directives: ghostDirectives},
"gitea": { "gitea": {
directives: giteaDirectives, cookies: []string{"gitea_flash", "redirect_to"},
cookies: []string{"gitea_flash", "redirect_to"},
}, },
"mastodon": {directives: mastodonDirectives},
"mediawiki": {directives: mediawikiDirectives},
"nextcloud": {directives: nextcloudDirectives},
"vaultwarden": {directives: vaultwardenDirectives},
"wordpress": {directives: wordpressDirectives},
} }
// AppNames returns the names of the apps whose fixes sww has, in // AppNames returns the names of the apps whose fixes sww has, in
// alphabetical order. // alphabetical order.
func AppNames() []string { func AppNames() []string {
return slices.Sorted(maps.Keys(apps)) // An embedded directory always reads.
files, _ := appFixes.ReadDir("apps")
names := make([]string, len(files))
for i, file := range files {
names[i] = strings.TrimSuffix(file.Name(), ".conf")
}
slices.Sort(names)
return names
} }
// Params are what New needs. // Params are what New needs.
@@ -373,8 +348,11 @@ func New(params Params) (*CoreRuleSet, error) {
} }
var fixes strings.Builder var fixes strings.Builder
for _, name := range AppNames() { for _, name := range AppNames() {
fixes.WriteString(apps[name].directives + "\n") // An embedded file always reads.
file, _ := appFixes.ReadFile("apps/" + name + ".conf")
fixes.WriteString(string(file) + "\n")
} }
text := fmt.Sprintf(directives, params.ParanoiaLevel, body, exclusions, text := fmt.Sprintf(directives, params.ParanoiaLevel, body, exclusions,
+3 -3
View File
@@ -17,7 +17,7 @@ const (
) )
// giteaSearches are searches of the pages gitea and Forgejo share whose q // giteaSearches are searches of the pages gitea and Forgejo share whose q
// skips the lists of system files and command names, in the eighth change. // skips the lists of system files and command names, in gitea's fixes.
// gitea serves its admin pages under /-/admin/ and Forgejo under /admin/. // gitea serves its admin pages under /-/admin/ and Forgejo under /admin/.
// //
//nolint:gochecknoglobals // a constant cannot be a list //nolint:gochecknoglobals // a constant cannot be a list
@@ -222,8 +222,8 @@ func TestGiteaSearchExclusionsAreNarrow(t *testing.T) {
} }
// Under /admin/, where Forgejo serves its admin pages, only the hard // Under /admin/, where Forgejo serves its admin pages, only the hard
// values are left out: the tenth change lets them through in q on every // values are left out: Drupal's fixes let them through in q on every path
// path there, for Drupal. // there.
for _, target := range []string{ for _, target := range []string{
"/admin/packages/delete?q=", "/admin/packages/delete?q=",
"/admin/packages?other=", "/admin/packages?other=",
+90
View File
@@ -0,0 +1,90 @@
// Package dirwatch watches a directory, and calls a function once the
// directory has gone a short time without a change after one. The rule
// files and the lookup database are read again this way.
package dirwatch
import (
"context"
"time"
"github.com/fsnotify/fsnotify"
)
// quietTime is how long the directory must go without a change before the
// function is called, so that a file still being written, such as one
// saved in place, appended to or copied in with scp, is read only once
// whole.
const quietTime = 2 * time.Second
// Watcher watches one directory. The directory is watched, rather than a
// file in it, so that a file replaced by a rename is seen.
type Watcher struct {
watcher *fsnotify.Watcher
}
// New starts watching dir, or returns why it cannot.
func New(dir string) (*Watcher, error) {
watcher, err := fsnotify.NewWatcher()
if err != nil {
return nil, err
}
err = watcher.Add(dir)
if err != nil {
_ = watcher.Close()
return nil, err
}
return &Watcher{watcher: watcher}, nil
}
// Close stops watching the directory.
func (w *Watcher) Close() error {
return w.watcher.Close()
}
// CallWhenQuiet calls changed each time the directory has gone quietTime
// without a change, after a change to a path for which counts is true, until
// ctx is done. It also calls failed with each error the watch reports. The
// wait starts at once, as if for a change, so that a change made before the
// watch began is taken in too.
func (w *Watcher) CallWhenQuiet(
ctx context.Context,
counts func(path string) bool,
changed func(),
failed func(error),
) {
callWhenQuiet(
ctx, w.watcher.Events, w.watcher.Errors, counts, changed, failed,
)
}
// callWhenQuiet is CallWhenQuiet, for the changes from events and the
// errors from errs.
func callWhenQuiet(
ctx context.Context,
events <-chan fsnotify.Event,
errs <-chan error,
counts func(path string) bool,
changed func(),
failed func(error),
) {
quiet := time.NewTimer(quietTime)
defer quiet.Stop()
for {
select {
case <-ctx.Done():
return
case event := <-events:
if counts(event.Name) {
quiet.Reset(quietTime)
}
case <-quiet.C:
changed()
case err := <-errs:
failed(err)
}
}
}
+167
View File
@@ -0,0 +1,167 @@
package dirwatch
import (
"context"
"errors"
"path/filepath"
"sync/atomic"
"testing"
"testing/synctest"
"time"
"github.com/fsnotify/fsnotify"
)
// errWatch is an error the watch reports.
var errWatch = errors.New("too many changes")
func TestDirectoryThatCannotBeWatchedIsAnError(t *testing.T) {
t.Parallel()
_, err := New(filepath.Join(t.TempDir(), "missing"))
if err == nil {
t.Error("a missing directory is watched, want an error")
}
}
// The tests below run callWhenQuiet in a synctest bubble, where time is a
// clock of the test's own: time.Sleep moves it on at once, and
// synctest.Wait returns once callWhenQuiet waits again, so that every call
// due by then is made. The test sends the changes itself, as the watch of
// a directory cannot run in a bubble.
func TestSecondChangeStartsTheWaitAgain(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
loop := start(t, everyPath)
loop.changes <- fsnotify.Event{Name: "50-app.rules", Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
loop.wantCalls(t, 0)
// The second part starts the wait again.
loop.changes <- fsnotify.Event{Name: "50-app.rules", Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
loop.wantCalls(t, 0)
time.Sleep(time.Nanosecond)
synctest.Wait()
loop.wantCalls(t, 1)
})
}
func TestCalledOnceQuietAfterTheStart(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
// An edit saved before the watch began sends no change, and is
// taken in by this call.
loop := start(t, everyPath)
time.Sleep(quietTime)
synctest.Wait()
loop.wantCalls(t, 1)
// Long after, it has been called only once.
time.Sleep(time.Hour)
synctest.Wait()
loop.wantCalls(t, 1)
})
}
func TestChangeThatDoesNotCountIsNoChange(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
loop := start(t, func(path string) bool {
return path == "ipinfo_lite.mmdb"
})
// The wait that starts with the watch ends with a call.
time.Sleep(quietTime)
synctest.Wait()
loop.changes <- fsnotify.Event{Name: "other.mmdb", Op: fsnotify.Create}
time.Sleep(quietTime)
synctest.Wait()
loop.wantCalls(t, 1)
loop.changes <- fsnotify.Event{Name: "ipinfo_lite.mmdb", Op: fsnotify.Write}
time.Sleep(quietTime)
synctest.Wait()
loop.wantCalls(t, 2)
})
}
func TestErrorsOfTheWatchHandedOn(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
loop := start(t, everyPath)
loop.errs <- errWatch
synctest.Wait()
if !errors.Is(loop.failure, errWatch) {
t.Errorf("failed with %v, want %v", loop.failure, errWatch)
}
})
}
// loop is a callWhenQuiet that a test runs.
type loop struct {
// changes and errs send it changes and errors.
changes chan fsnotify.Event
errs chan error
// calls counts its calls of changed, and failure is the last error it
// handed to failed.
calls atomic.Int32
failure error
}
// start runs callWhenQuiet, with counts, until the test ends.
func start(t *testing.T, counts func(path string) bool) *loop {
t.Helper()
l := &loop{changes: make(chan fsnotify.Event), errs: make(chan error)}
ctx, stop := context.WithCancel(t.Context())
stopped := make(chan struct{})
go func() {
callWhenQuiet(ctx, l.changes, l.errs, counts,
func() { l.calls.Add(1) },
func(err error) { l.failure = err },
)
close(stopped)
}()
t.Cleanup(func() {
stop()
<-stopped
})
return l
}
// everyPath counts a change to any path.
func everyPath(string) bool {
return true
}
// wantCalls checks how many times changed has been called.
func (l *loop) wantCalls(t *testing.T, want int32) {
t.Helper()
got := l.calls.Load()
if got != want {
t.Errorf("changed called %d times, want %d", got, want)
}
}
+60 -37
View File
@@ -84,12 +84,7 @@ type Params struct {
// one request to GeoJS is under way at a time, and it asks about every // one request to GeoJS is under way at a time, and it asks about every
// client waiting, up to maxPerRequest. It is safe for concurrent use. // client waiting, up to maxPerRequest. It is safe for concurrent use.
type GeoJS struct { type GeoJS struct {
url string params Params
answered func(Answer)
now func() time.Time
asked func()
failed func(err error, askingAgainIn time.Duration)
wentWithoutAnswer func()
// httpClient follows no redirect, so that visitors' addresses go to // httpClient follows no redirect, so that visitors' addresses go to
// GeoJS alone: a redirect is a failure. // GeoJS alone: a redirect is a failure.
httpClient *http.Client httpClient *http.Client
@@ -99,8 +94,13 @@ type GeoJS struct {
// waiting are the clients without an answer: those to ask GeoJS about, // waiting are the clients without an answer: those to ask GeoJS about,
// and those it is being asked about. // and those it is being asked about.
waiting map[netip.Prefix]*wait waiting map[netip.Prefix]*wait
// asking is true while a request to GeoJS is under way. // asking is true while a request to GeoJS is under way, and endAsking
asking bool // ends it. stopped is true once StopAsking has been called, and
// askGoroutines waits for the goroutine that makes the requests.
asking bool
endAsking context.CancelFunc
stopped bool
askGoroutines sync.WaitGroup
// retryDelay is how long GeoJS is left alone after its last failure, // retryDelay is how long GeoJS is left alone after its last failure,
// zero after an answer; retryAt is when it may be asked again. // zero after an answer; retryAt is when it may be asked again.
retryDelay time.Duration retryDelay time.Duration
@@ -140,12 +140,7 @@ func New(params Params) *GeoJS {
} }
return &GeoJS{ return &GeoJS{
url: params.URL, params: params,
answered: params.Answered,
now: params.Now,
asked: params.Asked,
failed: params.Failed,
wentWithoutAnswer: params.WentWithoutAnswer,
httpClient: &http.Client{ httpClient: &http.Client{
CheckRedirect: func(*http.Request, []*http.Request) error { CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse return http.ErrUseLastResponse
@@ -188,7 +183,7 @@ func (g *GeoJS) LookUp(
answer, found := g.kept(client) answer, found := g.kept(client)
if !found { if !found {
g.wentWithoutAnswer() g.params.WentWithoutAnswer()
} }
w, waiting := g.waiting[client] w, waiting := g.waiting[client]
@@ -210,6 +205,23 @@ func (g *GeoJS) LookUpWithoutWaiting(
return answer return answer
} }
// StopAsking ends the request to GeoJS under way, and returns once the
// goroutine that made it has. A request it ends is no failure: Failed is not
// called, and GeoJS is not left alone after it. After StopAsking, GeoJS is
// asked nothing.
func (g *GeoJS) StopAsking() {
g.mu.Lock()
g.stopped = true
if g.asking {
g.endAsking()
}
g.mu.Unlock()
g.askGoroutines.Wait()
}
// Kept returns client's answer, if one is kept, without asking GeoJS. // Kept returns client's answer, if one is kept, without asking GeoJS.
func (g *GeoJS) Kept(client netip.Prefix) (Answer, bool) { func (g *GeoJS) Kept(client netip.Prefix) (Answer, bool) {
g.mu.Lock() g.mu.Lock()
@@ -251,7 +263,7 @@ func (g *GeoJS) Load(answers []Answer) {
g.answers.Purge() g.answers.Purge()
now := g.now() now := g.params.Now()
for _, answer := range answers { for _, answer := range answers {
if now.Sub(answer.Answered) < keepFor { if now.Sub(answer.Answered) < keepFor {
@@ -288,18 +300,19 @@ func (g *GeoJS) answerOrWait(
} }
if w == nil { if w == nil {
g.wentWithoutAnswer() g.params.WentWithoutAnswer()
return Answer{}, nil // too many clients wait already return Answer{}, nil // too many clients wait already
} }
if !g.asking { if !g.asking {
// GeoJS is left alone after a failure, so no answer can come. // GeoJS is left alone after a failure, or asked nothing after
// StopAsking, so no answer can come.
w.late = true w.late = true
} }
if w.late { if w.late {
g.wentWithoutAnswer() g.params.WentWithoutAnswer()
return Answer{}, nil return Answer{}, nil
} }
@@ -310,7 +323,7 @@ func (g *GeoJS) answerOrWait(
// kept returns client's answer, if GeoJS gave it less than keepFor ago, // kept returns client's answer, if GeoJS gave it less than keepFor ago,
// and notes that it was used. // and notes that it was used.
func (g *GeoJS) kept(client netip.Prefix) (Answer, bool) { func (g *GeoJS) kept(client netip.Prefix) (Answer, bool) {
now := g.now() now := g.params.Now()
kept, found := g.answers.Get(client) kept, found := g.answers.Get(client)
if !found || now.Sub(kept.Answered) >= keepFor { if !found || now.Sub(kept.Answered) >= keepFor {
@@ -323,22 +336,28 @@ func (g *GeoJS) kept(client netip.Prefix) (Answer, bool) {
} }
// ask starts asking GeoJS about the waiting clients, unless a request to // ask starts asking GeoJS about the waiting clients, unless a request to
// it is under way or it is left alone after a failure. The requests to // it is under way, it is left alone after a failure, or StopAsking has
// GeoJS are for every client waiting, so they go on when the caller whose // been called. The requests to GeoJS are for every client waiting, so they
// ctx is given stops waiting. // go on when the caller whose ctx is given stops waiting, until StopAsking.
func (g *GeoJS) ask(ctx context.Context) { func (g *GeoJS) ask(ctx context.Context) {
if g.asking || g.now().Before(g.retryAt) { if g.asking || g.stopped || g.params.Now().Before(g.retryAt) {
return return
} }
g.asking = true askCtx, end := context.WithCancel(context.WithoutCancel(ctx))
g.asking, g.endAsking = true, end
go g.askAboutWaiting(context.WithoutCancel(ctx)) g.askGoroutines.Go(func() {
defer end()
g.askAboutWaiting(askCtx)
})
} }
// askAboutWaiting asks GeoJS about the waiting clients, one request at a // askAboutWaiting asks GeoJS about the waiting clients, one request at a
// time, until none is left or GeoJS fails. Each answer kept is given to // time, until none is left, GeoJS fails, or StopAsking ends ctx. Each
// Answered, outside the lock, since Answered takes locks of its own. // answer kept is given to Answered, outside the lock, since Answered takes
// locks of its own.
func (g *GeoJS) askAboutWaiting(ctx context.Context) { func (g *GeoJS) askAboutWaiting(ctx context.Context) {
for { for {
clients := g.nextClients() clients := g.nextClients()
@@ -347,11 +366,15 @@ func (g *GeoJS) askAboutWaiting(ctx context.Context) {
} }
given, err := g.request(ctx, clients) given, err := g.request(ctx, clients)
if err != nil && ctx.Err() != nil {
continue // ended by StopAsking, which is no failure
}
kept, answered := g.keep(clients, given, err) kept, answered := g.keep(clients, given, err)
if g.answered != nil { if g.params.Answered != nil {
for _, answer := range kept { for _, answer := range kept {
g.answered(answer) g.params.Answered(answer)
} }
} }
@@ -362,13 +385,13 @@ func (g *GeoJS) askAboutWaiting(ctx context.Context) {
} }
// nextClients returns up to maxPerRequest of the waiting clients. When // nextClients returns up to maxPerRequest of the waiting clients. When
// none is waiting, it returns none and notes that no request to GeoJS is // none is waiting, or StopAsking has been called, it returns none and
// under way. // notes that no request to GeoJS is under way.
func (g *GeoJS) nextClients() []netip.Prefix { func (g *GeoJS) nextClients() []netip.Prefix {
g.mu.Lock() g.mu.Lock()
defer g.mu.Unlock() defer g.mu.Unlock()
if len(g.waiting) == 0 { if len(g.waiting) == 0 || g.stopped {
g.asking = false g.asking = false
return nil return nil
@@ -400,7 +423,7 @@ func (g *GeoJS) keep(
g.mu.Lock() g.mu.Lock()
defer g.mu.Unlock() defer g.mu.Unlock()
now := g.now() now := g.params.Now()
kept := make([]Answer, 0, len(clients)) kept := make([]Answer, 0, len(clients))
leftOut := 0 leftOut := 0
@@ -444,7 +467,7 @@ func (g *GeoJS) keep(
w.late = true w.late = true
} }
g.failed(err, g.retryDelay) g.params.Failed(err, g.retryDelay)
return kept, false return kept, false
} }
@@ -474,7 +497,7 @@ func (g *GeoJS) request(
req, err := http.NewRequestWithContext( req, err := http.NewRequestWithContext(
ctx, ctx,
http.MethodGet, http.MethodGet,
g.url, g.params.URL,
http.NoBody, http.NoBody,
) )
if err != nil { if err != nil {
@@ -483,7 +506,7 @@ func (g *GeoJS) request(
req.URL.RawQuery = "ip=" + strings.Join(addrs, ",") req.URL.RawQuery = "ip=" + strings.Join(addrs, ",")
g.asked() g.params.Asked()
res, err := g.httpClient.Do(req) res, err := g.httpClient.Do(req)
if err != nil { if err != nil {
+38
View File
@@ -318,6 +318,44 @@ func TestCallerWhoseContextEndsEarlyDoesNotStopTheRequest(t *testing.T) {
}) })
} }
func TestStopAskingEndsTheRequestUnderWayAsNoFailureAndAsksNoMore(
t *testing.T,
) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
service, _, g, reports := startWithReports()
clients := newClients()
service.set(hanging)
g.LookUpWithoutWaiting(t.Context(), clients())
waitForRequests(t, service, 1)
// The request is ended, not waited out, and is no failure.
began := time.Now()
g.StopAsking()
if waited := time.Since(began); waited != 0 {
t.Errorf("StopAsking returned after %s, want at once", waited)
}
reports.wantFailures(t)
// A new client is then not asked about, and its lookup does not
// wait.
service.set(answering)
wantCountry(t, g, clients(), "")
if waited := time.Since(began); waited != 0 {
t.Errorf("the lookup returned after %s, want at once", waited)
}
waitForRequests(t, service, 1)
reports.wantFailures(t)
})
}
func TestRequestOver10SecondsIsAFailure(t *testing.T) { func TestRequestOver10SecondsIsAFailure(t *testing.T) {
t.Parallel() t.Parallel()
+18 -49
View File
@@ -13,18 +13,13 @@ import (
"sync" "sync"
"time" "time"
"github.com/fsnotify/fsnotify"
"github.com/oschwald/maxminddb-golang/v2" "github.com/oschwald/maxminddb-golang/v2"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/dirwatch"
"sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/geojs"
) )
// quietTime is how long the lookup database must go without a change
// before it is read again, so that a file still being copied in is read
// only once whole.
const quietTime = 2 * time.Second
// FileParams are what OpenFile needs. // FileParams are what OpenFile needs.
type FileParams struct { type FileParams struct {
// Path is the lookup database, the IPinfo Lite file in its .mmdb form // Path is the lookup database, the IPinfo Lite file in its .mmdb form
@@ -122,20 +117,13 @@ func (f *File) ReadFailures() int {
} }
// Watch watches the directory of the lookup database until ctx is done, // Watch watches the directory of the lookup database until ctx is done,
// and reads the file again once it has gone without a change for // and reads the file again once it has gone without a change for a short
// quietTime, after it is replaced, written or removed, and after Watch // time, after it is replaced, written or removed, and after Watch starts
// starts watching. If the directory cannot be watched, that is logged, and // watching, as dirwatch describes. A change to another file in its
// the database read at start stays in use. // directory does not count. If the directory cannot be watched, that is
// logged, and the database read at start stays in use.
func (f *File) Watch(ctx context.Context) { func (f *File) Watch(ctx context.Context) {
watcher, err := fsnotify.NewWatcher() watcher, err := dirwatch.New(filepath.Dir(f.params.Path))
if err == nil {
defer func() {
_ = watcher.Close()
}()
err = watcher.Add(filepath.Dir(f.params.Path))
}
if err != nil { if err != nil {
f.params.ProcessLog.Error( f.params.ProcessLog.Error(
"cannot watch the lookup database for replacements", "cannot watch the lookup database for replacements",
@@ -146,41 +134,22 @@ func (f *File) Watch(ctx context.Context) {
return return
} }
defer func() {
_ = watcher.Close()
}()
f.params.ProcessLog.Info("watching the lookup database for replacements", f.params.ProcessLog.Info("watching the lookup database for replacements",
"file", f.params.Path) "file", f.params.Path)
f.readAfterChanges(ctx, watcher.Events, watcher.Errors) watcher.CallWhenQuiet(ctx, f.isDatabase, f.readAgain, func(err error) {
f.params.ProcessLog.Warn("watching the lookup database failed",
"error", err.Error())
})
} }
// readAfterChanges reads the lookup database again once quietTime has // isDatabase reports whether path names the lookup database.
// passed without a change to it from events, until ctx is done, and logs func (f *File) isDatabase(path string) bool {
// the errors from errs. A change to another file in its directory does not return filepath.Clean(path) == filepath.Clean(f.params.Path)
// count. The wait starts at once, as if for a change, so that a file
// replaced after OpenFile read it, and before its directory was watched,
// is read too.
func (f *File) readAfterChanges(
ctx context.Context, events <-chan fsnotify.Event, errs <-chan error,
) {
path := filepath.Clean(f.params.Path)
quiet := time.NewTimer(quietTime)
defer quiet.Stop()
for {
select {
case <-ctx.Done():
return
case event := <-events:
if filepath.Clean(event.Name) == path {
quiet.Reset(quietTime)
}
case <-quiet.C:
f.readAgain()
case err := <-errs:
f.params.ProcessLog.Warn("watching the lookup database failed",
"error", err.Error())
}
}
} }
// readAgain reads the lookup database again, in place of the one in use, // readAgain reads the lookup database again, in place of the one in use,
+35 -141
View File
@@ -1,7 +1,6 @@
package lookup package lookup
import ( import (
"context"
"log/slog" "log/slog"
"net/netip" "net/netip"
"net/url" "net/url"
@@ -12,7 +11,6 @@ import (
"testing/synctest" "testing/synctest"
"time" "time"
"github.com/fsnotify/fsnotify"
"github.com/maxmind/mmdbwriter/mmdbtype" "github.com/maxmind/mmdbwriter/mmdbtype"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
@@ -133,66 +131,48 @@ func TestFileThatCannotBeReadIsAnError(t *testing.T) {
} }
} }
// The tests below run readAfterChanges in a synctest bubble, where time is func TestChangeOfAnotherFileInTheDirectoryIsNoReplacement(t *testing.T) {
// a clock of the test's own: time.Sleep moves it on at once, and t.Parallel()
// synctest.Wait returns once readAfterChanges waits again, so that every
// reading due by then is done. The test sends the changes itself, as the
// watch of a directory cannot run in a bubble.
func TestReplacementCopiedOverTheFileInTwoPartsIsReadOnlyWhole( dir := t.TempDir()
t *testing.T, path := filepath.Join(dir, "ipinfo_lite.mmdb")
) { writeDatabase(t, path, "DE")
f := openFile(t, path, newQueue())
for changed, want := range map[string]bool{
path: true,
dir + "/./ipinfo_lite.mmdb": true,
filepath.Join(dir, "other.mmdb"): false,
} {
if f.isDatabase(changed) != want {
t.Errorf(
"a change to %s counts: %t, want %t",
changed,
!want,
want,
)
}
}
}
// The tests below run in a synctest bubble, where time is a clock of the
// test's own, so that the time a file is read can be checked. The wait for
// the directory to go quiet before a reading is tested in dirwatch.
func TestReplacementIsReadAgain(t *testing.T) {
t.Parallel() t.Parallel()
synctest.Test(t, func(t *testing.T) { synctest.Test(t, func(t *testing.T) {
dir := t.TempDir() path := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb")
path := filepath.Join(dir, "ipinfo_lite.mmdb")
writeDatabase(t, path, "DE") writeDatabase(t, path, "DE")
queue := newQueue() queue := newQueue()
f := openFile(t, path, queue) f := openFile(t, path, queue)
changes := watch(t, f)
other := filepath.Join(dir, "replacement.mmdb") writeDatabase(t, path, "KP")
writeDatabase(t, other, "KP") time.Sleep(time.Hour)
f.readAgain()
//nolint:gosec // a file the test wrote
replacement, err := os.ReadFile(other)
if err != nil {
t.Fatalf("read %s: %v", other, err)
}
// The file in use is overwritten in place, and keeps giving what
// it gave. Its first part alone is not a .mmdb file.
file, err := os.Create(path) //nolint:gosec // a file the test wrote
if err != nil {
t.Fatalf("create %s: %v", path, err)
}
defer func() {
_ = file.Close()
}()
half := len(replacement) / 2
write(t, file, replacement[:half])
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantCountry(t, f, "DE")
// The second part starts the wait again.
write(t, file, replacement[half:])
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantCountry(t, f, "DE")
time.Sleep(time.Nanosecond)
synctest.Wait()
wantCountry(t, f, "KP") wantCountry(t, f, "KP")
if !f.LastRead().Equal(time.Now()) || f.ReadFailures() != 0 { if !f.LastRead().Equal(time.Now()) || f.ReadFailures() != 0 {
@@ -219,15 +199,10 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(
queue := newQueue() queue := newQueue()
f := openFile(t, path, queue) f := openFile(t, path, queue)
read := f.LastRead() read := f.LastRead()
changes := watch(t, f)
writeFile(t, path, "not a lookup database\n") writeFile(t, path, "not a lookup database\n")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
// Long after, the replacement has been read once.
time.Sleep(time.Hour) time.Sleep(time.Hour)
synctest.Wait() f.readAgain()
wantCountry(t, f, "DE") wantCountry(t, f, "DE")
if !f.LastRead().Equal(read) || f.ReadFailures() != 1 { if !f.LastRead().Equal(read) || f.ReadFailures() != 1 {
@@ -240,7 +215,7 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(
} }
wantAlerts(t, queue, alerts.Alert{ wantAlerts(t, queue, alerts.Alert{
Time: read.Add(quietTime), Time: read.Add(time.Hour),
Event: alerts.EventFileError, Event: alerts.EventFileError,
Reason: "the lookup database cannot be read, and the one read before stays in use", Reason: "the lookup database cannot be read, and the one read before stays in use",
Detail: map[string]any{ Detail: map[string]any{
@@ -252,55 +227,6 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(
}) })
} }
func TestChangeOfAnotherFileInTheDirectoryIsNoReplacement(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "ipinfo_lite.mmdb")
writeDatabase(t, path, "DE")
f := openFile(t, path, newQueue())
changes := watch(t, f)
// The wait that starts with the watch ends with a reading.
time.Sleep(quietTime)
synctest.Wait()
writeDatabase(t, path, "KP")
changes <- fsnotify.Event{Name: filepath.Join(dir, "other.mmdb"), Op: fsnotify.Create}
time.Sleep(quietTime)
synctest.Wait()
wantCountry(t, f, "DE")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime)
synctest.Wait()
wantCountry(t, f, "KP")
})
}
func TestReplacementSavedBeforeTheWatchStartsIsRead(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb")
writeDatabase(t, path, "DE")
f := openFile(t, path, newQueue())
// Saved after OpenFile read the file, and before its directory was
// watched, so that no change is seen for it.
writeDatabase(t, path, "KP")
watch(t, f)
time.Sleep(quietTime)
synctest.Wait()
wantCountry(t, f, "KP")
})
}
// newQueue returns a queue of alerts for a webhook that is never sent // newQueue returns a queue of alerts for a webhook that is never sent
// them, so that they wait in it for the test to look at. // them, so that they wait in it for the test to look at.
func newQueue() *alerts.Queue { func newQueue() *alerts.Queue {
@@ -344,28 +270,6 @@ func openFile(t *testing.T, path string, queue *alerts.Queue) *File {
return f return f
} }
// watch runs f's readAfterChanges until the test ends, and returns the
// channel that sends it changes.
func watch(t *testing.T, f *File) chan<- fsnotify.Event {
t.Helper()
changes := make(chan fsnotify.Event)
ctx, stop := context.WithCancel(t.Context())
stopped := make(chan struct{})
go func() {
f.readAfterChanges(ctx, changes, nil)
close(stopped)
}()
t.Cleanup(func() {
stop()
<-stopped
})
return changes
}
// wantCountry checks the country f gives testClient. // wantCountry checks the country f gives testClient.
func wantCountry(t *testing.T, f *File, want string) { func wantCountry(t *testing.T, f *File, want string) {
t.Helper() t.Helper()
@@ -401,13 +305,3 @@ func writeFile(t *testing.T, path, content string) {
t.Fatalf("write %s: %v", path, err) t.Fatalf("write %s: %v", path, err)
} }
} }
// write writes data to the end of file.
func write(t *testing.T, file *os.File, data []byte) {
t.Helper()
_, err := file.Write(data)
if err != nil {
t.Fatalf("write: %v", err)
}
}
+400 -272
View File
@@ -14,14 +14,35 @@ import (
"github.com/prometheus/client_golang/prometheus/promhttp" "github.com/prometheus/client_golang/prometheus/promhttp"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/remotelog"
"sneak.berlin/go/smallwebwaf/internal/reputation" "sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules" "sneak.berlin/go/smallwebwaf/internal/rules"
) )
// Params are what New builds the metrics from. Each part is nil while it is
// off, and a part that is nil adds no metric of its own.
type Params struct {
// BusiestCount is how many countries and how many AS numbers get series
// of their own (SWW_WAF_METRICS_BUSIEST_COUNTRIES_AND_ASNS).
BusiestCount int
// InstanceName is the label instance every metric carries
// (SWW_WAF_INSTANCE_NAME).
InstanceName string
// Now returns the time at which Bans counts the bans active.
Now func() time.Time
Bans banLedger
Clients sized
Rules sized
Lists fetchedLists
DNSBL dnsblZones
AbuseIPDB abuseIPDBChecker
Alerts alertQueue
LookupFile lookupDatabase
RemoteLog remoteLogSender
}
// Metrics are sww's metrics. They are safe for concurrent use. // Metrics are sww's metrics. They are safe for concurrent use.
type Metrics struct { type Metrics struct {
// registry gives every metric registered with it the label instance. // registry gives every metric registered with it the label instance.
@@ -37,12 +58,10 @@ type Metrics struct {
rateLimitHits *prometheus.CounterVec rateLimitHits *prometheus.CounterVec
offences *prometheus.CounterVec offences *prometheus.CounterVec
coreRuleSetMatches *prometheus.CounterVec coreRuleSetMatches *prometheus.CounterVec
// ruleMatches are made by AddRules, and reputationHits by ruleMatches *prometheus.CounterVec
// AddReputation. reputationHits *prometheus.CounterVec
ruleMatches *prometheus.CounterVec countries *busiest
reputationHits *prometheus.CounterVec asns *busiest
countries *busiest
asns *busiest
// GeoJSRequests are the requests to GeoJS, and GeoJSFailures those // GeoJSRequests are the requests to GeoJS, and GeoJSFailures those
// that failed. RequestsWithoutGeoJSAnswer are the requests that needed // that failed. RequestsWithoutGeoJSAnswer are the requests that needed
@@ -60,20 +79,18 @@ type Metrics struct {
stateFileEditsSetAside *prometheus.CounterVec stateFileEditsSetAside *prometheus.CounterVec
} }
// New returns the metrics, with the Go runtime's and the process's own. // New returns the metrics, with the Go runtime's and the process's own, and
// busiestCount is how many countries and how many AS numbers get series of // those read from each part in params as the metrics are asked for.
// their own (SWW_WAF_METRICS_BUSIEST_COUNTRIES_AND_ASNS). Every metric
// carries instanceName (SWW_WAF_INSTANCE_NAME) as its label instance.
// //
//nolint:funlen // a few lines for each metric, a list that grows with them //nolint:funlen // a few lines for each metric, a list that grows with them
func New(busiestCount int, instanceName string) *Metrics { func New(params Params) *Metrics {
byStatus := []string{"status_class", "action"} byStatus := []string{"status_class", "action"}
byFile := []string{"file"} byFile := []string{"file"}
registry := prometheus.NewRegistry() registry := prometheus.NewRegistry()
m := &Metrics{ m := &Metrics{
registry: prometheus.WrapRegistererWith( registry: prometheus.WrapRegistererWith(
prometheus.Labels{"instance": instanceName}, registry), prometheus.Labels{"instance": params.InstanceName}, registry),
handler: promhttp.HandlerFor(registry, promhttp.HandlerOpts{}), handler: promhttp.HandlerFor(registry, promhttp.HandlerOpts{}),
inFlight: prometheus.NewGauge(prometheus.GaugeOpts{ inFlight: prometheus.NewGauge(prometheus.GaugeOpts{
Name: "sww_requests_in_flight", Name: "sww_requests_in_flight",
@@ -114,8 +131,19 @@ func New(busiestCount int, instanceName string) *Metrics {
"and the rule's id.", "and the rule's id.",
[]string{"mode", "rule_id"}, []string{"mode", "rule_id"},
), ),
countries: newCountries(busiestCount), ruleMatches: counterVec(
asns: newASNs(busiestCount), "sww_rule_matches_total",
"Requests that matched a rule of the rule files, by its id and action.",
[]string{"rule_id", "action"},
),
reputationHits: counterVec(
"sww_reputation_hits_total",
"Requests whose client a blocklist, a DNSBL zone or AbuseIPDB lists, "+
"by the list's URL, the zone, or abuseipdb.",
[]string{sourceLabel},
),
countries: newCountries(params.BusiestCount),
asns: newASNs(params.BusiestCount),
GeoJSRequests: prometheus.NewCounter(prometheus.CounterOpts{ GeoJSRequests: prometheus.NewCounter(prometheus.CounterOpts{
Name: "sww_geojs_requests_total", Name: "sww_geojs_requests_total",
Help: "Requests to GeoJS.", Help: "Requests to GeoJS.",
@@ -167,6 +195,7 @@ func New(busiestCount int, instanceName string) *Metrics {
m.inFlight, m.requests, m.requestBytes, m.responseBytes, m.inFlight, m.requests, m.requestBytes, m.responseBytes,
m.requestDuration, m.upstreamDuration, m.requestDuration, m.upstreamDuration,
m.rateLimitHits, m.offences, m.coreRuleSetMatches, m.rateLimitHits, m.offences, m.coreRuleSetMatches,
m.ruleMatches, m.reputationHits,
m.countries, m.asns, m.countries, m.asns,
m.GeoJSRequests, m.GeoJSFailures, m.RequestsWithoutGeoJSAnswer, m.GeoJSRequests, m.GeoJSFailures, m.RequestsWithoutGeoJSAnswer,
m.stateFileWrites, m.stateFileWriteFailures, m.stateFileWrites, m.stateFileWriteFailures,
@@ -174,259 +203,45 @@ func New(busiestCount int, instanceName string) *Metrics {
m.stateFileEditsTakenIn, m.stateFileEditsSetAside, m.stateFileEditsTakenIn, m.stateFileEditsSetAside,
) )
if params.Bans != nil {
m.addBans(params.Bans, params.Now)
}
if params.Clients != nil {
m.addClients(params.Clients)
}
if params.Rules != nil {
m.addRules(params.Rules)
}
if params.Lists != nil {
m.addLists(params.Lists)
}
if params.DNSBL != nil {
m.addDNSBL(params.DNSBL)
}
if params.AbuseIPDB != nil {
m.addAbuseIPDB(params.AbuseIPDB)
}
if params.Alerts != nil {
m.addAlerts(params.Alerts)
}
if params.LookupFile != nil {
m.addLookupFile(params.LookupFile)
}
if params.RemoteLog != nil {
m.addRemoteLog(params.RemoteLog)
}
return m return m
} }
// AddBansAndClients adds the metrics read from the ledger and the table
// of clients as the metrics are asked for: the bans made since the start,
// by cause, the bans active and permanent at now, and the clients in the
// table.
func (m *Metrics) AddBansAndClients(
ledger *bans.Ledger, limiter *ratelimit.Limiter, now func() time.Time,
) {
for _, cause := range []string{
bans.CauseLimit, bans.CauseAttack, bans.CauseAdmin, bans.CauseAbuseIPDB,
} {
m.registry.MustRegister(
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_bans_made_total",
Help: "Bans made, by cause.",
ConstLabels: prometheus.Labels{"cause": cause},
}, func() float64 {
return float64(ledger.Made(cause))
}),
)
}
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_active_bans",
Help: "Bans active now, the permanent ones included.",
}, func() float64 {
active, _ := ledger.Count(now())
return float64(active)
}),
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_permanent_bans",
Help: "Permanent bans not lifted.",
}, func() float64 {
_, permanent := ledger.Count(now())
return float64(permanent)
}),
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_tracked_clients",
Help: "Clients in the table of clients.",
}, func() float64 {
return float64(limiter.Len())
}),
)
}
// AddRules adds the metrics of the rule files: the requests that matched
// each rule, which RuleMatched counts, and the rules loaded from
// ruleFiles, read as the metrics are asked for. It is called once, before
// RuleMatched.
func (m *Metrics) AddRules(ruleFiles *rules.Files) {
m.ruleMatches = counterVec(
"sww_rule_matches_total",
"Requests that matched a rule of the rule files, by its id and action.",
[]string{"rule_id", "action"},
)
m.registry.MustRegister(m.ruleMatches,
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_rules_loaded",
Help: "Rules loaded from the rule files.",
}, func() float64 {
return float64(ruleFiles.Len())
}))
}
// AddRemoteLog adds the metrics of sending the log lines to
// SWW_WAF_LOG_REMOTE_URL, read from remote as the metrics are asked for: the
// lines sent, those dropped, and those waiting in the buffer.
func (m *Metrics) AddRemoteLog(remote *remotelog.Sender) {
m.registry.MustRegister(
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_remote_log_lines_sent_total",
Help: "Log lines sent to SWW_WAF_LOG_REMOTE_URL.",
}, func() float64 {
return float64(remote.Sent())
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_remote_log_lines_dropped_total",
Help: "Log lines dropped: the oldest in a full buffer, and those " +
"whose sending failed.",
}, func() float64 {
return float64(remote.Dropped())
}),
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_remote_log_lines_waiting",
Help: "Log lines in the buffer, waiting to be sent.",
}, func() float64 {
return float64(remote.Depth())
}),
)
}
// AddLookupFile adds the metrics of the lookup database, read as the
// metrics are asked for: when the file in use was read, which lastRead
// returns, and the replacements of it that could not be read, which
// readFailures returns.
func (m *Metrics) AddLookupFile(
lastRead func() time.Time,
readFailures func() int,
) {
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_lookup_database_last_read_timestamp_seconds",
Help: "When the lookup database in use was read, in seconds since 1970.",
}, func() float64 {
return float64(lastRead().Unix())
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_lookup_database_read_failures_total",
Help: "Replacements of the lookup database that could not be read.",
}, func() float64 {
return float64(readFailures())
}),
)
}
// sourceLabel is the label of the reputation metrics: a list's URL, a
// DNSBL zone, its key masked, or abuseipdb.
const sourceLabel = "source"
// AddReputation adds the metrics of the lists fetched from URLs and of the
// DNSBL zones, by source, each list's URL or each zone, its key masked as
// config.MaskZoneKey masks it: the requests whose client a blocklist, a
// zone's verdict or AbuseIPDB's score lists, which ReputationHit counts,
// and, read from lists and dnsbl as the metrics are asked for, for a list,
// the fetches that failed and when the copy in use was fetched, and for a
// zone, the queries made and those that failed. It is called once, before
// ReputationHit.
func (m *Metrics) AddReputation(
lists *reputation.Lists,
dnsbl *reputation.DNSBL,
) {
m.reputationHits = counterVec(
"sww_reputation_hits_total",
"Requests whose client a blocklist, a DNSBL zone or AbuseIPDB lists, "+
"by the list's URL, the zone, or abuseipdb.",
[]string{sourceLabel},
)
m.registry.MustRegister(m.reputationHits)
for _, zone := range dnsbl.Zones() {
source := prometheus.Labels{sourceLabel: config.MaskZoneKey(zone)}
m.addReputationQueries(
source,
func() int { return dnsbl.Queries(zone) },
)
m.addReputationFailures(
source,
func() int { return dnsbl.Failures(zone) },
)
}
for _, listURL := range lists.URLs() {
source := prometheus.Labels{sourceLabel: listURL}
m.addReputationFailures(
source,
func() int { return lists.Failures(listURL) },
)
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_reputation_last_fetch_timestamp_seconds",
Help: "When the copy of the list in use was fetched, in seconds since " +
"1970, or 0 while there is none.",
ConstLabels: source,
}, func() float64 {
fetched := lists.Fetched(listURL)
if fetched.IsZero() {
return 0
}
return float64(fetched.Unix())
}),
)
}
}
// AddAbuseIPDB adds the metrics of AbuseIPDB, with the source abuseipdb,
// read from abuseIPDB as the metrics are asked for: the checks made, those
// that failed, and how many checks the day's budget has left. It is
// called once, after AddReputation, while SWW_WAF_ABUSEIPDB_KEY is set.
func (m *Metrics) AddAbuseIPDB(abuseIPDB *reputation.AbuseIPDB) {
source := prometheus.Labels{sourceLabel: reputation.AbuseIPDBSource}
m.addReputationQueries(source, abuseIPDB.Checked)
m.addReputationFailures(source, abuseIPDB.Failures)
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_reputation_daily_budget_remaining",
Help: "Checks of the day's SWW_WAF_ABUSEIPDB_DAILY_BUDGET not yet spent.",
ConstLabels: source,
}, func() float64 {
return float64(abuseIPDB.BudgetLeft())
}),
)
}
// ReputationHit counts a request whose client source lists: a blocklist,
// by its URL, a DNSBL zone, its key masked, or AbuseIPDB, abuseipdb.
func (m *Metrics) ReputationHit(source string) {
m.reputationHits.WithLabelValues(source).Inc()
}
// AddAlerts adds the metrics of the alerts sent to each destination set,
// read from queue as the metrics are asked for, by destination: the
// alerts sent, the requests to the destination that failed, the alerts
// held back, which are the same for every destination, and those
// dropped. With no destination set, it adds none.
func (m *Metrics) AddAlerts(queue *alerts.Queue) {
for _, name := range queue.DestinationsSet() {
destination := prometheus.Labels{"destination": name}
m.registry.MustRegister(
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alerts_sent_total",
Help: "Alerts the destination took.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Counts(name).Sent)
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alert_send_failures_total",
Help: "Requests to the destination that failed.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Counts(name).SendFailures)
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alerts_suppressed_total",
Help: "Alerts held back: repeats within SWW_WAF_ALERT_COOLDOWN, and " +
"alerts past SWW_WAF_ALERT_MAX_PER_HOUR, for the hour's summary.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Suppressed())
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alerts_dropped_total",
Help: "Alerts dropped, the oldest first, from a full queue, and " +
"alerts given up as the destination refused them.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Counts(name).Dropped)
}),
)
}
}
// ServeHTTP answers with the metrics in the Prometheus text format. // ServeHTTP answers with the metrics in the Prometheus text format.
func (m *Metrics) ServeHTTP(w http.ResponseWriter, r *http.Request) { func (m *Metrics) ServeHTTP(w http.ResponseWriter, r *http.Request) {
m.handler.ServeHTTP(w, r) m.handler.ServeHTTP(w, r)
@@ -465,7 +280,7 @@ func (m *Metrics) RequestReported(
// LimitHit counts a request that broke a rate limit, a byte limit or the // LimitHit counts a request that broke a rate limit, a byte limit or the
// refusal limit, by the window and the kind of hit. // refusal limit, by the window and the kind of hit.
func (m *Metrics) LimitHit(hit ratelimit.Hit) { func (m *Metrics) LimitHit(hit ratelimit.Hit) {
m.rateLimitHits.WithLabelValues(hit.Window, hit.Kind).Inc() m.rateLimitHits.WithLabelValues(hit.Window, string(hit.Kind)).Inc()
} }
// Offences counts the offences of r, a request that has ended, as its // Offences counts the offences of r, a request that has ended, as its
@@ -486,8 +301,8 @@ func (m *Metrics) Offences(r ratelimit.Request) {
// RuleMatched counts a request that matched the rule id, whose action is // RuleMatched counts a request that matched the rule id, whose action is
// action. // action.
func (m *Metrics) RuleMatched(id, action string) { func (m *Metrics) RuleMatched(id string, action rules.Action) {
m.ruleMatches.WithLabelValues(id, action).Inc() m.ruleMatches.WithLabelValues(id, string(action)).Inc()
} }
// CoreRuleSetMatched counts a request that matched the Core Rule Set's rule id, // CoreRuleSetMatched counts a request that matched the Core Rule Set's rule id,
@@ -496,6 +311,12 @@ func (m *Metrics) CoreRuleSetMatched(mode string, id int) {
m.coreRuleSetMatches.WithLabelValues(mode, strconv.Itoa(id)).Inc() m.coreRuleSetMatches.WithLabelValues(mode, strconv.Itoa(id)).Inc()
} }
// ReputationHit counts a request whose client source lists: a blocklist,
// by its URL, a DNSBL zone, its key masked, or AbuseIPDB, abuseipdb.
func (m *Metrics) ReputationHit(source string) {
m.reputationHits.WithLabelValues(source).Inc()
}
// StateFileWritten counts a write of the state or cache file name, of size // StateFileWritten counts a write of the state or cache file name, of size
// bytes, that ended with err. // bytes, that ended with err.
func (m *Metrics) StateFileWritten(name string, size int, err error) { func (m *Metrics) StateFileWritten(name string, size int, err error) {
@@ -527,16 +348,323 @@ func (m *Metrics) StateFileEditSetAside(name string) {
m.stateFileEditsSetAside.WithLabelValues(name).Inc() m.stateFileEditsSetAside.WithLabelValues(name).Inc()
} }
// banLedger is what the metrics read from the ledger of bans.
type banLedger interface {
// Made returns the bans made since the start with cause.
Made(cause bans.Cause) int
// Count returns the bans active at now, the permanent ones included,
// and the permanent ones.
Count(now time.Time) (active, permanent int)
}
// addBans adds the metrics read from ledger as the metrics are asked for:
// the bans made since the start, by cause, and the bans active and
// permanent at now.
func (m *Metrics) addBans(ledger banLedger, now func() time.Time) {
for _, cause := range []bans.Cause{
bans.CauseLimit, bans.CauseAttack, bans.CauseAdmin, bans.CauseAbuseIPDB,
} {
m.registry.MustRegister(
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_bans_made_total",
Help: "Bans made, by cause.",
ConstLabels: prometheus.Labels{"cause": string(cause)},
}, func() float64 {
return float64(ledger.Made(cause))
}),
)
}
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_active_bans",
Help: "Bans active now, the permanent ones included.",
}, func() float64 {
active, _ := ledger.Count(now())
return float64(active)
}),
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_permanent_bans",
Help: "Permanent bans not lifted.",
}, func() float64 {
_, permanent := ledger.Count(now())
return float64(permanent)
}),
)
}
// sized is what the metrics read from the table of clients and from the
// rule files.
type sized interface {
// Len returns how many clients the table holds, or how many rules were
// loaded from the rule files.
Len() int
}
// addClients adds the metric of the clients in table, read as the metrics
// are asked for.
func (m *Metrics) addClients(table sized) {
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_tracked_clients",
Help: "Clients in the table of clients.",
}, func() float64 {
return float64(table.Len())
}),
)
}
// addRules adds the metric of the rules loaded from files, read as the
// metrics are asked for.
func (m *Metrics) addRules(files sized) {
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_rules_loaded",
Help: "Rules loaded from the rule files.",
}, func() float64 {
return float64(files.Len())
}))
}
// remoteLogSender is what the metrics read from what sends the log lines to
// SWW_WAF_LOG_REMOTE_URL.
type remoteLogSender interface {
// Sent returns the lines sent.
Sent() int64
// Dropped returns the lines dropped.
Dropped() int64
// Depth returns the lines waiting in the buffer.
Depth() int
}
// addRemoteLog adds the metrics of sending the log lines to
// SWW_WAF_LOG_REMOTE_URL, read from remote as the metrics are asked for: the
// lines sent, those dropped, and those waiting in the buffer.
func (m *Metrics) addRemoteLog(remote remoteLogSender) {
m.registry.MustRegister(
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_remote_log_lines_sent_total",
Help: "Log lines sent to SWW_WAF_LOG_REMOTE_URL.",
}, func() float64 {
return float64(remote.Sent())
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_remote_log_lines_dropped_total",
Help: "Log lines dropped: the oldest in a full buffer, and those " +
"whose sending failed.",
}, func() float64 {
return float64(remote.Dropped())
}),
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_remote_log_lines_waiting",
Help: "Log lines in the buffer, waiting to be sent.",
}, func() float64 {
return float64(remote.Depth())
}),
)
}
// lookupDatabase is what the metrics read from the lookup database.
type lookupDatabase interface {
// LastRead returns when the file in use was read.
LastRead() time.Time
// ReadFailures returns the replacements of the file that could not be
// read.
ReadFailures() int
}
// addLookupFile adds the metrics of the lookup database, read from file as
// the metrics are asked for: when the file in use was read, and the
// replacements of it that could not be read.
func (m *Metrics) addLookupFile(file lookupDatabase) {
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_lookup_database_last_read_timestamp_seconds",
Help: "When the lookup database in use was read, in seconds since 1970.",
}, func() float64 {
return float64(file.LastRead().Unix())
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_lookup_database_read_failures_total",
Help: "Replacements of the lookup database that could not be read.",
}, func() float64 {
return float64(file.ReadFailures())
}),
)
}
// sourceLabel is the label of the reputation metrics: a list's URL, a
// DNSBL zone, its key masked, or abuseipdb.
const sourceLabel = "source"
// dnsblZones is what the metrics read from what asks the DNSBL zones.
type dnsblZones interface {
// Zones returns the zones asked, their keys unmasked.
Zones() []string
// Queries returns the queries made to zone.
Queries(zone string) int
// Failures returns the queries to zone that failed.
Failures(zone string) int
}
// addDNSBL adds the metrics of each zone dnsbl asks, by source, the zone,
// its key masked as reputation.MaskZoneKey masks it, read as the metrics are
// asked for: the queries made and those that failed.
func (m *Metrics) addDNSBL(dnsbl dnsblZones) {
for _, zone := range dnsbl.Zones() {
source := prometheus.Labels{
sourceLabel: reputation.MaskZoneKey(zone),
}
m.addReputationQueries(
source,
func() int { return dnsbl.Queries(zone) },
)
m.addReputationFailures(
source,
func() int { return dnsbl.Failures(zone) },
)
}
}
// fetchedLists is what the metrics read from the lists fetched from URLs.
type fetchedLists interface {
// URLs returns the lists' URLs.
URLs() []string
// Failures returns the fetches of the list at listURL that failed.
Failures(listURL string) int
// Fetched returns when the copy in use of the list at listURL was
// fetched, or the zero time while there is none.
Fetched(listURL string) time.Time
}
// addLists adds the metrics of each list fetched from a URL, by source, the
// list's URL, read from lists as the metrics are asked for: the fetches
// that failed and when the copy in use was fetched.
func (m *Metrics) addLists(lists fetchedLists) {
for _, listURL := range lists.URLs() {
source := prometheus.Labels{sourceLabel: listURL}
m.addReputationFailures(
source,
func() int { return lists.Failures(listURL) },
)
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_reputation_last_fetch_timestamp_seconds",
Help: "When the copy of the list in use was fetched, in seconds since " +
"1970, or 0 while there is none.",
ConstLabels: source,
}, func() float64 {
fetched := lists.Fetched(listURL)
if fetched.IsZero() {
return 0
}
return float64(fetched.Unix())
}),
)
}
}
// abuseIPDBChecker is what the metrics read from what checks clients with
// AbuseIPDB.
type abuseIPDBChecker interface {
// Checked returns the checks made.
Checked() int
// Failures returns the checks that failed.
Failures() int
// BudgetLeft returns how many checks the day's budget has left.
BudgetLeft() int
}
// addAbuseIPDB adds the metrics of AbuseIPDB, with the source abuseipdb,
// read from abuseIPDB as the metrics are asked for: the checks made, those
// that failed, and how many checks the day's budget has left.
func (m *Metrics) addAbuseIPDB(abuseIPDB abuseIPDBChecker) {
source := prometheus.Labels{sourceLabel: reputation.AbuseIPDBSource}
m.addReputationQueries(source, abuseIPDB.Checked)
m.addReputationFailures(source, abuseIPDB.Failures)
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_reputation_daily_budget_remaining",
Help: "Checks of the day's SWW_WAF_ABUSEIPDB_DAILY_BUDGET not yet spent.",
ConstLabels: source,
}, func() float64 {
return float64(abuseIPDB.BudgetLeft())
}),
)
}
// alertQueue is what the metrics read from the queue of alerts.
type alertQueue interface {
// DestinationsSet returns the names of the destinations set.
DestinationsSet() []alerts.DestinationName
// Counts returns the alerts sent to the destination name, the requests
// to it that failed, and the alerts dropped.
Counts(name alerts.DestinationName) alerts.Counts
// Suppressed returns the alerts held back.
Suppressed() int64
}
// addAlerts adds the metrics of the alerts sent to each destination set,
// read from queue as the metrics are asked for, by destination: the
// alerts sent, the requests to the destination that failed, the alerts
// held back, which are the same for every destination, and those
// dropped. With no destination set, it adds none.
func (m *Metrics) addAlerts(queue alertQueue) {
for _, name := range queue.DestinationsSet() {
destination := prometheus.Labels{"destination": string(name)}
m.registry.MustRegister(
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alerts_sent_total",
Help: "Alerts the destination took.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Counts(name).Sent)
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alert_send_failures_total",
Help: "Requests to the destination that failed.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Counts(name).SendFailures)
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alerts_suppressed_total",
Help: "Alerts held back: repeats within SWW_WAF_ALERT_COOLDOWN, and " +
"alerts past SWW_WAF_ALERT_MAX_PER_HOUR, for the hour's summary.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Suppressed())
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alerts_dropped_total",
Help: "Alerts dropped, the oldest first, from a full queue, and " +
"alerts given up as the destination refused them.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Counts(name).Dropped)
}),
)
}
}
// count counts a request that has ended, as RequestEnded and RequestReported // count counts a request that has ended, as RequestEnded and RequestReported
// describe. // describe.
func (m *Metrics) count( func (m *Metrics) count(
line *requestlog.Line, duration, upstreamDuration time.Duration, line *requestlog.Line, duration, upstreamDuration time.Duration,
) { ) {
class := statusClass(line.Status) class := statusClass(line.Status)
m.requests.WithLabelValues(class, line.Action).Inc() action := string(line.Action)
m.requestBytes.WithLabelValues(class, line.Action). m.requests.WithLabelValues(class, action).Inc()
m.requestBytes.WithLabelValues(class, action).
Add(float64(line.RequestBytes)) Add(float64(line.RequestBytes))
m.responseBytes.WithLabelValues(class, line.Action). m.responseBytes.WithLabelValues(class, action).
Add(float64(line.ResponseBytes)) Add(float64(line.ResponseBytes))
m.requestDuration.Observe(duration.Seconds()) m.requestDuration.Observe(duration.Seconds())
+161
View File
@@ -0,0 +1,161 @@
package metrics_test
import (
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/metrics"
)
const (
listURL = "https://lists.example/block.txt"
zone = "dnsbl.example"
destination = "webhook"
)
// Each stand-in answers with a number of its own, so that each metric
// shows which part it was read from.
type ledgerStandIn struct{}
func (ledgerStandIn) Made(bans.Cause) int { return 1 }
func (ledgerStandIn) Count(time.Time) (int, int) { return 2, 1 }
type clientTableStandIn struct{}
func (clientTableStandIn) Len() int { return 3 }
type ruleFilesStandIn struct{}
func (ruleFilesStandIn) Len() int { return 4 }
type listsStandIn struct{}
func (listsStandIn) URLs() []string { return []string{listURL} }
func (listsStandIn) Failures(string) int { return 5 }
func (listsStandIn) Fetched(string) time.Time { return time.Time{} }
type dnsblStandIn struct{}
func (dnsblStandIn) Zones() []string { return []string{zone} }
func (dnsblStandIn) Queries(string) int { return 6 }
func (dnsblStandIn) Failures(string) int { return 0 }
type abuseIPDBStandIn struct{}
func (abuseIPDBStandIn) Checked() int { return 0 }
func (abuseIPDBStandIn) Failures() int { return 0 }
func (abuseIPDBStandIn) BudgetLeft() int { return 7 }
type alertQueueStandIn struct{}
func (alertQueueStandIn) DestinationsSet() []alerts.DestinationName {
return []alerts.DestinationName{destination}
}
func (alertQueueStandIn) Counts(alerts.DestinationName) alerts.Counts {
return alerts.Counts{Sent: 8}
}
func (alertQueueStandIn) Suppressed() int64 { return 0 }
type lookupDatabaseStandIn struct{}
func (lookupDatabaseStandIn) LastRead() time.Time { return time.Time{} }
func (lookupDatabaseStandIn) ReadFailures() int { return 9 }
type remoteLogStandIn struct{}
func (remoteLogStandIn) Sent() int64 { return 10 }
func (remoteLogStandIn) Dropped() int64 { return 0 }
func (remoteLogStandIn) Depth() int { return 0 }
func TestMetricsAreReadFromEachPart(t *testing.T) {
t.Parallel()
m := metrics.New(metrics.Params{
BusiestCount: 1,
InstanceName: "app",
Now: func() time.Time { return time.Time{} },
Bans: ledgerStandIn{},
Clients: clientTableStandIn{},
Rules: ruleFilesStandIn{},
Lists: listsStandIn{},
DNSBL: dnsblStandIn{},
AbuseIPDB: abuseIPDBStandIn{},
Alerts: alertQueueStandIn{},
LookupFile: lookupDatabaseStandIn{},
RemoteLog: remoteLogStandIn{},
})
scraped := scrape(t, m)
for _, line := range []string{
`sww_bans_made_total{cause="limit",instance="app"} 1`,
`sww_active_bans{instance="app"} 2`,
`sww_tracked_clients{instance="app"} 3`,
`sww_rules_loaded{instance="app"} 4`,
`sww_reputation_failures_total{instance="app",source="` + listURL + `"} 5`,
`sww_reputation_queries_total{instance="app",source="` + zone + `"} 6`,
`sww_reputation_daily_budget_remaining{instance="app",source="abuseipdb"} 7`,
`sww_alerts_sent_total{destination="` + destination + `",instance="app"} 8`,
`sww_lookup_database_read_failures_total{instance="app"} 9`,
`sww_remote_log_lines_sent_total{instance="app"} 10`,
} {
if !strings.Contains(scraped, "\n"+line+"\n") {
t.Errorf("metrics\n%s\nwant\n%s", scraped, line)
}
}
}
func TestPartThatIsOffAddsNoMetric(t *testing.T) {
t.Parallel()
scraped := scrape(t, metrics.New(metrics.Params{
BusiestCount: 1,
InstanceName: "app",
}))
for _, name := range []string{
"sww_bans_made_total",
"sww_tracked_clients",
"sww_rules_loaded",
"sww_reputation_",
"sww_alert",
"sww_lookup_database_",
"sww_remote_log_",
} {
if strings.Contains(scraped, name) {
t.Errorf("metrics\n%s\nshow %s", scraped, name)
}
}
}
// scrape returns the metrics m serves.
func scrape(t *testing.T, m *metrics.Metrics) string {
t.Helper()
scraped := httptest.NewRecorder()
m.ServeHTTP(
scraped,
httptest.NewRequestWithContext(
t.Context(), http.MethodGet, "/", http.NoBody,
),
)
return scraped.Body.String()
}
+44
View File
@@ -0,0 +1,44 @@
// Package processlog builds the logger for the process's own messages,
// which it writes as JSON lines marked "type":"process". The request log
// is package requestlog's.
package processlog
import (
"io"
"log/slog"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// Params are what New needs.
type Params struct {
// Output is where the lines are written.
Output io.Writer
// InstanceName, SWW_WAF_INSTANCE_NAME, is each line's instance.
InstanceName string
// Level, SWW_WAF_LOG_LEVEL, is the least severe level written.
Level slog.Level
}
// New returns the logger for the process's own messages: JSON lines on
// params.Output, marked "type":"process", with the time in the same form
// as a request line's. It writes only the messages at params.Level or more
// severe; the request lines requestlog.Write writes are never held back.
func New(params Params) *slog.Logger {
handler := slog.NewJSONHandler(params.Output, &slog.HandlerOptions{
Level: params.Level,
ReplaceAttr: func(groups []string, attr slog.Attr) slog.Attr {
if attr.Key == slog.TimeKey && len(groups) == 0 {
return slog.String(
slog.TimeKey,
requestlog.FormatTime(attr.Value.Time()),
)
}
return attr
},
})
return slog.New(handler).
With("type", "process", "instance", params.InstanceName)
}
+105
View File
@@ -0,0 +1,105 @@
package processlog_test
import (
"bytes"
"encoding/json"
"log/slog"
"slices"
"strings"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/processlog"
)
// instanceName is the instance the tests' loggers are built with.
const instanceName = "host1/blog"
func TestProcessLinesAreMarkedProcessAndGiveTheInstance(t *testing.T) {
t.Parallel()
var out bytes.Buffer
processlog.New(processlog.Params{
Output: &out,
InstanceName: instanceName,
Level: slog.LevelInfo,
}).
Info("starting",
"version", "v1")
var fields map[string]any
err := json.Unmarshal(out.Bytes(), &fields)
if err != nil {
t.Fatalf("decode %q: %v", out.String(), err)
}
if fields["type"] != "process" ||
fields["instance"] != "host1/blog" ||
fields["msg"] != "starting" ||
fields["level"] != "INFO" ||
fields["version"] != "v1" {
t.Errorf("process line %v", fields)
}
timeText, _ := fields["time"].(string)
logged, err := time.Parse(time.RFC3339, timeText)
if err != nil || !strings.HasSuffix(timeText, "Z") ||
len(timeText) != len("2006-01-02T15:04:05.000Z") ||
time.Since(logged) > time.Minute {
t.Errorf(
"process line time %q, want now in UTC with milliseconds",
timeText,
)
}
}
func TestProcessLoggerWritesTheMessagesAtItsLevelOrMoreSevere(t *testing.T) {
t.Parallel()
levels := []slog.Level{
slog.LevelDebug, slog.LevelInfo, slog.LevelWarn, slog.LevelError,
}
for i, level := range levels {
t.Run(level.String(), func(t *testing.T) {
t.Parallel()
var out bytes.Buffer
processLog := processlog.New(processlog.Params{
Output: &out,
InstanceName: instanceName,
Level: level,
})
for _, at := range levels {
processLog.Log(t.Context(), at, "message")
}
var got, want []string
for line := range strings.Lines(out.String()) {
var fields struct {
Level string `json:"level"`
}
err := json.Unmarshal([]byte(line), &fields)
if err != nil {
t.Fatalf("decode %q: %v", line, err)
}
got = append(got, fields.Level)
}
for _, written := range levels[i:] {
want = append(want, written.String())
}
if !slices.Equal(got, want) {
t.Errorf("lines at %v, want %v", got, want)
}
})
}
}
+7 -4
View File
@@ -11,7 +11,7 @@ import (
func TestHistoryKeepsEveryRequest(t *testing.T) { func TestHistoryKeepsEveryRequest(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize) limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -57,7 +57,7 @@ func TestHistoryKeepsEveryRequest(t *testing.T) {
func TestLookupReachesTheHistoryOfAClientInTheTable(t *testing.T) { func TestLookupReachesTheHistoryOfAClientInTheTable(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize) limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
other := netip.MustParsePrefix("198.51.100.7/32") other := netip.MustParsePrefix("198.51.100.7/32")
start := midnight() start := midnight()
@@ -100,7 +100,10 @@ func TestLookupReachesTheHistoryOfAClientInTheTable(t *testing.T) {
func TestResetKeepsTheHistory(t *testing.T) { func TestResetKeepsTheHistory(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit}, tableSize) limiter := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{PerMinute: limit},
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -123,7 +126,7 @@ func TestResetKeepsTheHistory(t *testing.T) {
func TestRequestsAddsUpTheClientsInsideTheNetblock(t *testing.T) { func TestRequestsAddsUpTheClientsInsideTheNetblock(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize) limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
for client, requests := range map[string]int{ for client, requests := range map[string]int{
"198.51.100.9/32": 2, "198.51.100.9/32": 2,
+26 -12
View File
@@ -19,15 +19,19 @@ import (
const day = 24 * time.Hour const day = 24 * time.Hour
// LimitKind is what a limit counts: KindRequests, KindBytes or
// KindRefusals.
type LimitKind string
// The kinds of limits, as the metrics name them. // The kinds of limits, as the metrics name them.
const ( const (
// KindRequests is a rate limit, on a client's requests. // KindRequests is a rate limit, on a client's requests.
KindRequests = "requests" KindRequests LimitKind = "requests"
// KindBytes is a byte limit, on a client's bytes. // KindBytes is a byte limit, on a client's bytes.
KindBytes = "bytes" KindBytes LimitKind = "bytes"
// KindRefusals is the refusal limit, on a client's requests sww // KindRefusals is the refusal limit, on a client's requests sww
// refused after a rule file match or for a missing or wrong token. // refused after a rule file match or for a missing or wrong token.
KindRefusals = "refusals" KindRefusals LimitKind = "refusals"
) )
// Limits are the most requests a client may make in a minute, an hour and // Limits are the most requests a client may make in a minute, an hour and
@@ -156,16 +160,28 @@ type Request struct {
BlockedByCoreRuleSet bool BlockedByCoreRuleSet bool
} }
// New returns a Limiter for limits, with no client counted yet, whose // Params are what New needs.
// table holds at most maxClients clients (SWW_WAF_MAX_TRACKED_CLIENTS). Past type Params struct {
// it, the least recently seen client is dropped, with its history, and // Limits are what each client's requests and bytes are counted against.
// starts afresh if it comes back. Limits Limits
func New(limits Limits, maxClients int) *Limiter { // MaxClients is the most clients the table holds,
clients, err := simplelru.NewLRU[netip.Prefix, *Client](maxClients, nil) // SWW_WAF_MAX_TRACKED_CLIENTS. Past it, the least recently seen client
// is dropped, with its history, and starts afresh if it comes back.
MaxClients int
}
// New returns a Limiter with no client counted yet.
func New(params Params) *Limiter {
clients, err := simplelru.NewLRU[netip.Prefix, *Client](
params.MaxClients,
nil,
)
if err != nil { if err != nil {
panic(err) // NewLRU fails only for a size below one panic(err) // NewLRU fails only for a size below one
} }
limits := params.Limits
return &Limiter{ return &Limiter{
windows: [3]window{ windows: [3]window{
{ {
@@ -188,9 +204,7 @@ func New(limits Limits, maxClients int) *Limiter {
// Hit is a request that takes a client over a rate limit or the refusal // Hit is a request that takes a client over a rate limit or the refusal
// limit, or whose bytes take it over a byte limit. // limit, or whose bytes take it over a byte limit.
type Hit struct { type Hit struct {
// Kind is KindRequests for a rate limit, KindBytes for a byte limit, Kind LimitKind
// KindRefusals for the refusal limit.
Kind string
// Window is "minute", "hour" or "day". // Window is "minute", "hour" or "day".
Window string Window string
// Limit is the window's limit, as the client's percentage of it. // Limit is the window's limit, as the client's percentage of it.
+64 -40
View File
@@ -33,10 +33,10 @@ func TestOverARateLimitSaysWhatTheNextRequestWouldDoAndCountsNothing(
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
now := midnight() now := midnight()
limiter := ratelimit.New( limiter := ratelimit.New(ratelimit.Params{
ratelimit.Limits{PerMinute: limit, PerDay: limit + 1}, Limits: ratelimit.Limits{PerMinute: limit, PerDay: limit + 1},
tableSize, MaxClients: tableSize,
) })
if limiter.OverARateLimit(client, now) { if limiter.OverARateLimit(client, now) {
t.Error("a client never seen is over a rate limit") t.Error("a client never seen is over a rate limit")
@@ -72,7 +72,7 @@ func TestOverARateLimitSaysWhatTheNextRequestWouldDoAndCountsNothing(
wantCount(t, limiter, client, now, minute) wantCount(t, limiter, client, now, minute)
// A limit that is off is never gone over. // A limit that is off is never gone over.
off := ratelimit.New(ratelimit.Limits{}, tableSize) off := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
off.Count(client, now, whole) off.Count(client, now, whole)
if off.OverARateLimit(client, now) { if off.OverARateLimit(client, now) {
@@ -95,7 +95,10 @@ func TestEachWindowRefusesAtItsLimitAndLetsTheClientBack(t *testing.T) {
t.Run(tc.window, func(t *testing.T) { t.Run(tc.window, func(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(tc.limits, tableSize) limiter := ratelimit.New(ratelimit.Params{
Limits: tc.limits,
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
quarter := tc.length / 4 quarter := tc.length / 4
@@ -126,10 +129,10 @@ func TestEachWindowRefusesAtItsLimitAndLetsTheClientBack(t *testing.T) {
func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) { func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New( limiter := ratelimit.New(ratelimit.Params{
ratelimit.Limits{PerMinute: limit, PerHour: limit}, Limits: ratelimit.Limits{PerMinute: limit, PerHour: limit},
tableSize, MaxClients: tableSize,
) })
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -159,10 +162,13 @@ func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
func TestClientGetsItsPercentageOfEachLimitRoundedDown(t *testing.T) { func TestClientGetsItsPercentageOfEachLimitRoundedDown(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New( limiter := ratelimit.New(ratelimit.Params{
ratelimit.Limits{PerMinute: 5, BytesPerDay: math.MaxInt64}, Limits: ratelimit.Limits{
tableSize, PerMinute: 5,
) BytesPerDay: math.MaxInt64,
},
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -203,10 +209,10 @@ func TestZeroPercentIsAZeroAllowanceAndALimitOffStaysOff(t *testing.T) {
t.Parallel() t.Parallel()
// Only the hour has limits: the minute's and the day's are off. // Only the hour has limits: the minute's and the day's are off.
limiter := ratelimit.New( limiter := ratelimit.New(ratelimit.Params{
ratelimit.Limits{PerHour: limit, BytesPerHour: 1000}, Limits: ratelimit.Limits{PerHour: limit, BytesPerHour: 1000},
tableSize, MaxClients: tableSize,
) })
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -253,7 +259,10 @@ func TestEachByteLimitIsBrokenByTheBytesCounted(t *testing.T) {
t.Run(tc.window, func(t *testing.T) { t.Run(tc.window, func(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(tc.limits, tableSize) limiter := ratelimit.New(ratelimit.Params{
Limits: tc.limits,
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
// 600 bytes are within the limit, 600 more over it. // 600 bytes are within the limit, 600 more over it.
@@ -287,10 +296,10 @@ func TestEachByteLimitIsBrokenByTheBytesCounted(t *testing.T) {
func TestALimitIsBrokenOnlyByWhatIsAddedToIt(t *testing.T) { func TestALimitIsBrokenOnlyByWhatIsAddedToIt(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New( limiter := ratelimit.New(ratelimit.Params{
ratelimit.Limits{PerMinute: 2, BytesPerMinute: 1000}, Limits: ratelimit.Limits{PerMinute: 2, BytesPerMinute: 1000},
tableSize, MaxClients: tableSize,
) })
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
other := netip.MustParsePrefix("203.0.113.10/32") other := netip.MustParsePrefix("203.0.113.10/32")
start := midnight() start := midnight()
@@ -314,7 +323,7 @@ func TestALimitIsBrokenOnlyByWhatIsAddedToIt(t *testing.T) {
func TestCountGivesTheBytesInEachWindow(t *testing.T) { func TestCountGivesTheBytesInEachWindow(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize) limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -340,7 +349,10 @@ func TestCountGivesTheBytesInEachWindow(t *testing.T) {
func TestResetSetsTheBytesBackToZero(t *testing.T) { func TestResetSetsTheBytesBackToZero(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{BytesPerDay: 1000}, tableSize) limiter := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{BytesPerDay: 1000},
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -356,7 +368,7 @@ func TestRefusalsOverTheLimitInAMinuteBreakTheRefusalLimit(
) { ) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize) limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -411,7 +423,7 @@ func TestRefusalsOverTheLimitInAMinuteBreakTheRefusalLimit(
func TestCountGivesTheRequestsInEachWindow(t *testing.T) { func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize) limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -438,10 +450,10 @@ func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
func TestResetSetsTheCountsBackToZero(t *testing.T) { func TestResetSetsTheCountsBackToZero(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New( limiter := ratelimit.New(ratelimit.Params{
ratelimit.Limits{PerMinute: limit, PerDay: limit}, Limits: ratelimit.Limits{PerMinute: limit, PerDay: limit},
tableSize, MaxClients: tableSize,
) })
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -463,7 +475,10 @@ func TestResetSetsTheCountsBackToZero(t *testing.T) {
func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) { func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize) limiter := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{PerHour: limit},
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -482,10 +497,10 @@ func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) {
func TestRefusedRequestsCount(t *testing.T) { func TestRefusedRequestsCount(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New( limiter := ratelimit.New(ratelimit.Params{
ratelimit.Limits{PerMinute: limit, PerHour: 2 * limit}, Limits: ratelimit.Limits{PerMinute: limit, PerHour: 2 * limit},
tableSize, MaxClients: tableSize,
) })
refused := netip.MustParsePrefix("203.0.113.9/32") refused := netip.MustParsePrefix("203.0.113.9/32")
within := netip.MustParsePrefix("203.0.113.10/32") within := netip.MustParsePrefix("203.0.113.10/32")
start := midnight() start := midnight()
@@ -518,7 +533,10 @@ func TestRefusedRequestsCount(t *testing.T) {
func TestRequestCountedLateGoesInTheBucketUnderWay(t *testing.T) { func TestRequestCountedLateGoesInTheBucketUnderWay(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit}, tableSize) limiter := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{PerMinute: limit},
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -534,7 +552,10 @@ func TestRequestCountedLateGoesInTheBucketUnderWay(t *testing.T) {
func TestClockSetBackStartsTheBucketsAfresh(t *testing.T) { func TestClockSetBackStartsTheBucketsAfresh(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize) limiter := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{PerHour: limit},
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -562,7 +583,10 @@ func TestKeepsAtMostMaxClientsDroppingTheLeastRecentlySeen(t *testing.T) {
const maxClients = 3 const maxClients = 3
limiter := ratelimit.New(ratelimit.Limits{PerMinute: 1}, maxClients) limiter := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{PerMinute: 1},
MaxClients: maxClients,
})
now := midnight() now := midnight()
clients := make([]netip.Prefix, maxClients+1) clients := make([]netip.Prefix, maxClients+1)
@@ -628,7 +652,7 @@ func wantBytesCount(
client netip.Prefix, client netip.Prefix,
now time.Time, now time.Time,
bytes int64, bytes int64,
want string, want ratelimit.LimitKind,
) { ) {
t.Helper() t.Helper()
+12 -6
View File
@@ -19,7 +19,7 @@ func TestSnapshotListsTheClientsByAddress(t *testing.T) {
"2001:db8::/64", "2001:db8::/64",
} }
limiter := ratelimit.New(ratelimit.Limits{}, tableSize) limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
for _, i := range []int{2, 3, 0, 1} { for _, i := range []int{2, 3, 0, 1} {
limiter.Count(netip.MustParsePrefix(want[i]), midnight(), whole) limiter.Count(netip.MustParsePrefix(want[i]), midnight(), whole)
} }
@@ -52,7 +52,10 @@ func TestLoadedCountsCarryOn(t *testing.T) {
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
before := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize) before := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{PerHour: limit},
MaxClients: tableSize,
})
for range limit { for range limit {
wantCount(t, before, client, start, "") wantCount(t, before, client, start, "")
} }
@@ -60,7 +63,10 @@ func TestLoadedCountsCarryOn(t *testing.T) {
// Loaded into a new limiter, as across a restart, the client has no // Loaded into a new limiter, as across a restart, the client has no
// fresh allowance. // fresh allowance.
later := start.Add(time.Minute) later := start.Add(time.Minute)
after := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize) after := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{PerHour: limit},
MaxClients: tableSize,
})
after.Load(before.Snapshot(), later) after.Load(before.Snapshot(), later)
wantCount(t, after, client, later, hour) wantCount(t, after, client, later, hour)
} }
@@ -71,7 +77,7 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize) limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
limiter.Count(client, start, whole) limiter.Count(client, start, whole)
limiter.CountBytes(client, start, 5, whole) limiter.CountBytes(client, start, 5, whole)
limiter.CountRefusal(client, start, limit) limiter.CountRefusal(client, start, limit)
@@ -80,7 +86,7 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
loaded := func(now time.Time) ratelimit.Client { loaded := func(now time.Time) ratelimit.Client {
t.Helper() t.Helper()
after := ratelimit.New(ratelimit.Limits{}, tableSize) after := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
after.Load(limiter.Snapshot(), now) after.Load(limiter.Snapshot(), now)
return after.Snapshot()[0] return after.Snapshot()[0]
@@ -142,7 +148,7 @@ func TestLoadDropsTheLeastRecentlySeenFirst(t *testing.T) {
addr = addr.Next() addr = addr.Next()
} }
limiter := ratelimit.New(ratelimit.Limits{}, maxClients) limiter := ratelimit.New(ratelimit.Params{MaxClients: maxClients})
limiter.Load(clients, midnight()) limiter.Load(clients, midnight())
got := limiter.Snapshot() got := limiter.Snapshot()
+48 -16
View File
@@ -110,8 +110,12 @@ type AbuseIPDB struct {
// scores are by client. Each is added as it is fetched and never moved // scores are by client. Each is added as it is fetched and never moved
// up, so that the one fetched longest ago is the first dropped. // up, so that the one fetched longest ago is the first dropped.
scores *simplelru.LRU[netip.Prefix, Score] scores *simplelru.LRU[netip.Prefix, Score]
// checking are the clients whose check is under way. // checking are the clients whose check is under way, each with the
checking map[netip.Prefix]bool // function that ends it. stopped is true once StopChecking has been
// called, and checkGoroutines waits for the goroutine of each check.
checking map[netip.Prefix]context.CancelFunc
stopped bool
checkGoroutines sync.WaitGroup
// day is the day, in UTC, of the checks spent counts. // day is the day, in UTC, of the checks spent counts.
day civil.Date day civil.Date
spent int spent int
@@ -134,7 +138,7 @@ func NewAbuseIPDB(params AbuseIPDBParams) *AbuseIPDB {
params: params, params: params,
httpClient: &http.Client{}, httpClient: &http.Client{},
scores: scores, scores: scores,
checking: map[netip.Prefix]bool{}, checking: map[netip.Prefix]context.CancelFunc{},
} }
} }
@@ -144,11 +148,11 @@ func NewAbuseIPDB(params AbuseIPDBParams) *AbuseIPDB {
// addresses its request comes from. A client without one is checked in // addresses its request comes from. A client without one is checked in
// the background, by addr, the address its request came from, if // the background, by addr, the address its request came from, if
// offender, if it has committed an offence, unless its check is under // offender, if it has committed an offence, unless its check is under
// way, a check failed less than failureDelay ago, or the day's checks // way, a check failed less than failureDelay ago, the day's checks have
// have used up DailyBudget; Hit never waits for a check. The check that // used up DailyBudget, or StopChecking has been called; Hit never waits
// uses the budget up is logged and raised as a source_failure alert. ctx // for a check. The check that uses the budget up is logged and raised as a
// is the context of the client's request, and a check goes on after the // source_failure alert. ctx is the context of the client's request, and a
// request ends. // check goes on after the request ends, until StopChecking.
func (a *AbuseIPDB) Hit( func (a *AbuseIPDB) Hit(
ctx context.Context, client netip.Prefix, addr netip.Addr, offender bool, ctx context.Context, client netip.Prefix, addr netip.Addr, offender bool,
) (int64, bool) { ) (int64, bool) {
@@ -167,14 +171,21 @@ func (a *AbuseIPDB) Hit(
a.day, a.spent = today, 0 a.day, a.spent = today, 0
} }
check := offender && !a.checking[client] && !now.Before(a.retryAt) && _, checking := a.checking[client]
a.spent < a.params.DailyBudget
check := offender && !checking && !a.stopped &&
!now.Before(a.retryAt) && a.spent < a.params.DailyBudget
if check { if check {
a.checking[client] = true checkCtx, end := context.WithCancel(context.WithoutCancel(ctx))
a.checking[client] = end
a.checks++ a.checks++
a.spent++ a.spent++
go a.check(context.WithoutCancel(ctx), client, addr) a.checkGoroutines.Go(func() {
defer end()
a.check(checkCtx, client, addr)
})
} }
usedUp := check && a.spent == a.params.DailyBudget usedUp := check && a.spent == a.params.DailyBudget
@@ -189,6 +200,24 @@ func (a *AbuseIPDB) Hit(
return 0, false return 0, false
} }
// StopChecking ends the checks under way, and returns once their goroutines
// have. A check it ends gives no score and is no failure: it is neither
// counted, logged nor raised as an alert, and checks are not left alone
// after it. The ended check stays spent from the day's budget. After
// StopChecking, Hit checks no client.
func (a *AbuseIPDB) StopChecking() {
a.mu.Lock()
a.stopped = true
for _, end := range a.checking {
end()
}
a.mu.Unlock()
a.checkGoroutines.Wait()
}
// Checked returns how many checks were made. // Checked returns how many checks were made.
func (a *AbuseIPDB) Checked() int { func (a *AbuseIPDB) Checked() int {
a.mu.Lock() a.mu.Lock()
@@ -315,7 +344,8 @@ func (a *AbuseIPDB) RunBlacklist(ctx context.Context) {
// check checks client with AbuseIPDB by addr, one of its addresses, keeps // check checks client with AbuseIPDB by addr, one of its addresses, keeps
// the score as client's, and notes the check as no longer under way. A // the score as client's, and notes the check as no longer under way. A
// check that fails gives no score: it is counted, logged and raised as a // check that fails gives no score: it is counted, logged and raised as a
// source_failure alert, and no client is checked for failureDelay. // source_failure alert, and no client is checked for failureDelay. A check
// ended as ctx ends, by StopChecking, is no failure.
func (a *AbuseIPDB) check( func (a *AbuseIPDB) check(
ctx context.Context, ctx context.Context,
client netip.Prefix, client netip.Prefix,
@@ -323,24 +353,26 @@ func (a *AbuseIPDB) check(
) { ) {
score, err := a.ask(ctx, addr) score, err := a.ask(ctx, addr)
now := a.params.Now() now := a.params.Now()
isFailure := err != nil && ctx.Err() == nil
a.mu.Lock() a.mu.Lock()
delete(a.checking, client) delete(a.checking, client)
if err == nil { switch {
case err == nil:
a.scores.Add( a.scores.Add(
client, client,
Score{Client: client, Score: score, Fetched: now}, Score{Client: client, Score: score, Fetched: now},
) )
} else { case isFailure:
a.failures++ a.failures++
a.retryAt = now.Add(failureDelay) a.retryAt = now.Add(failureDelay)
} }
a.mu.Unlock() a.mu.Unlock()
if err != nil { if isFailure {
a.alert("checking a client with AbuseIPDB failed", err) a.alert("checking a client with AbuseIPDB failed", err)
} }
} }
+46 -12
View File
@@ -394,6 +394,43 @@ func TestCheckNotAnsweredWithinTheTimeoutFailsAndHitNeverWaits(
}) })
} }
func TestStopCheckingEndsTheCheckUnderWayAsNoFailureAndChecksNoMore(
t *testing.T,
) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
queue := newQueue()
p := abuseIPDBParams()
p.Alerts = queue
abuseIPDB := &abuseIPDBStandIn{hanging: true}
checker := newAbuseIPDB(abuseIPDB, p)
wantScore(t, checker, suspect, true, 0, false)
synctest.Wait()
// The check is ended, not waited out, and is no failure.
began := time.Now()
checker.StopChecking()
if waited := time.Since(began); waited != 0 {
t.Errorf("StopChecking returned after %s, want at once", waited)
}
wantFailures(t, checker, 0)
if got := waiting(queue); len(got) != 0 {
t.Errorf("alerts waiting %+v, want none", got)
}
// An offender without a score is then not checked.
wantScore(t, checker, other, true, 0, false)
synctest.Wait()
wantChecked(t, abuseIPDB, suspect)
})
}
func TestKeyIsSentInTheKeyHeaderAndNeverShown(t *testing.T) { func TestKeyIsSentInTheKeyHeaderAndNeverShown(t *testing.T) {
t.Parallel() t.Parallel()
@@ -408,12 +445,9 @@ func TestKeyIsSentInTheKeyHeaderAndNeverShown(t *testing.T) {
scores: map[string]int64{suspect: 100}, scores: map[string]int64{suspect: 100},
} }
checker := newAbuseIPDB(abuseIPDB, p) checker := newAbuseIPDB(abuseIPDB, p)
m := metrics.New(1, "app") m := metrics.New(metrics.Params{
m.AddReputation( BusiestCount: 1, InstanceName: instanceName, AbuseIPDB: checker,
reputation.New(params()), })
reputation.NewDNSBL(dnsblParams()),
)
m.AddAbuseIPDB(checker)
// One check that AbuseIPDB answers, and one it refuses with an answer // One check that AbuseIPDB answers, and one it refuses with an answer
// that names the key. // that names the key.
@@ -466,12 +500,9 @@ func TestMetricsCountTheChecksTheFailuresAndTheBudgetLeft(t *testing.T) {
p := abuseIPDBParams() p := abuseIPDBParams()
p.DailyBudget = 5 p.DailyBudget = 5
checker := newAbuseIPDB(abuseIPDB, p) checker := newAbuseIPDB(abuseIPDB, p)
m := metrics.New(1, "app") m := metrics.New(metrics.Params{
m.AddReputation( BusiestCount: 1, InstanceName: instanceName, AbuseIPDB: checker,
reputation.New(params()), })
reputation.NewDNSBL(dnsblParams()),
)
m.AddAbuseIPDB(checker)
// One check that AbuseIPDB answers, and one that fails. // One check that AbuseIPDB answers, and one that fails.
hitFrom(t, checker, suspect, true) hitFrom(t, checker, suspect, true)
@@ -774,6 +805,9 @@ func wantBudgetLeft(t *testing.T, checker *reputation.AbuseIPDB, want int) {
} }
} }
// instanceName is the label instance of the metrics the tests build.
const instanceName = "app"
// scrapeMetrics returns the metrics m serves. // scrapeMetrics returns the metrics m serves.
func scrapeMetrics(t *testing.T, m *metrics.Metrics) string { func scrapeMetrics(t *testing.T, m *metrics.Metrics) string {
t.Helper() t.Helper()
+67 -17
View File
@@ -16,7 +16,6 @@ import (
"github.com/hashicorp/golang-lru/v2/simplelru" "github.com/hashicorp/golang-lru/v2/simplelru"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/config"
) )
const ( const (
@@ -32,6 +31,8 @@ const (
// it fails, and no client is checked with AbuseIPDB after a check // it fails, and no client is checked with AbuseIPDB after a check
// fails, so that a source refusing them is not asked on every request. // fails, so that a source refusing them is not asked on every request.
failureDelay = time.Minute failureDelay = time.Minute
// maskedKey is what MaskZoneKey shows in place of a zone's key.
maskedKey = "********"
) )
var ( var (
@@ -82,8 +83,12 @@ type DNSBL struct {
// verdicts are by query. Each is added as it is fetched and never moved // verdicts are by query. Each is added as it is fetched and never moved
// up, so that the one fetched longest ago is the first dropped. // up, so that the one fetched longest ago is the first dropped.
verdicts *simplelru.LRU[query, Verdict] verdicts *simplelru.LRU[query, Verdict]
// asking are the queries under way. // asking are the queries under way, each with the function that ends
asking map[query]bool // it. stopped is true once StopAsking has been called, and
// queryGoroutines waits for the goroutine of each query.
asking map[query]context.CancelFunc
stopped bool
queryGoroutines sync.WaitGroup
// queries and failures count, by zone, the queries made and those that // queries and failures count, by zone, the queries made and those that
// failed, and retryAt is when a zone whose last query failed may be // failed, and retryAt is when a zone whose last query failed may be
// asked again. // asked again.
@@ -120,13 +125,31 @@ func NewDNSBL(params DNSBLParams) *DNSBL {
params: params, params: params,
resolver: resolver, resolver: resolver,
verdicts: verdicts, verdicts: verdicts,
asking: map[query]bool{}, asking: map[query]context.CancelFunc{},
queries: map[string]int{}, queries: map[string]int{},
failures: map[string]int{}, failures: map[string]int{},
retryAt: map[string]time.Time{}, retryAt: map[string]time.Time{},
} }
} }
// MaskZoneKey returns zone with ******** in place of its key, if it is a
// zone of Spamhaus's keyed query service, a name under dq.spamhaus.net,
// such as <key>.xbl.dq.spamhaus.net, whose first label is the key. Any
// other zone it returns as it is. A zone is shown so wherever it leaves
// the process: in the log, the alerts and the metrics.
func MaskZoneKey(zone string) string {
// DNS names ignore case, and a name may be written with a dot at its
// end.
name := strings.TrimSuffix(strings.ToLower(zone), ".")
if !strings.HasSuffix(name, ".dq.spamhaus.net") {
return zone
}
_, rest, _ := strings.Cut(zone, ".")
return maskedKey + "." + rest
}
// Zones returns the zones, in the order SWW_WAF_DNSBL_ZONES names them. // Zones returns the zones, in the order SWW_WAF_DNSBL_ZONES names them.
func (d *DNSBL) Zones() []string { func (d *DNSBL) Zones() []string {
return slices.Clone(d.params.Zones) return slices.Clone(d.params.Zones)
@@ -134,13 +157,14 @@ func (d *DNSBL) Zones() []string {
// ListedBy returns the zones whose verdict on addr, a client's address, // ListedBy returns the zones whose verdict on addr, a client's address,
// lists it, in the order SWW_WAF_DNSBL_ZONES names them, each with its key // lists it, in the order SWW_WAF_DNSBL_ZONES names them, each with its key
// masked, as config.MaskZoneKey masks it, since they go to the request // masked, as MaskZoneKey masks it, since they go to the request
// log, the alerts and the metrics. A verdict is used until CacheTTL has // log, the alerts and the metrics. A verdict is used until CacheTTL has
// passed since it was fetched. Each zone without one is asked about addr // passed since it was fetched. Each zone without one is asked about addr
// in the background, unless a query about addr to it is under way, the // in the background, unless a query about addr to it is under way, the
// zone is left alone after a failure, or maxQueries are under way; // zone is left alone after a failure, maxQueries are under way, or
// ListedBy never waits for a query. ctx is the context of the client's // StopAsking has been called; ListedBy never waits for a query. ctx is the
// request, and a query goes on after the request ends. // context of the client's request, and a query goes on after the request
// ends, until StopAsking.
func (d *DNSBL) ListedBy(ctx context.Context, addr netip.Addr) []string { func (d *DNSBL) ListedBy(ctx context.Context, addr netip.Addr) []string {
d.mu.Lock() d.mu.Lock()
defer d.mu.Unlock() defer d.mu.Unlock()
@@ -153,23 +177,47 @@ func (d *DNSBL) ListedBy(ctx context.Context, addr netip.Addr) []string {
q := query{zone: zone, client: addr} q := query{zone: zone, client: addr}
kept, found := d.verdicts.Peek(q) kept, found := d.verdicts.Peek(q)
_, asking := d.asking[q]
switch { switch {
case found && now.Sub(kept.Fetched) < d.params.CacheTTL: case found && now.Sub(kept.Fetched) < d.params.CacheTTL:
if kept.Listed { if kept.Listed {
listedBy = append(listedBy, config.MaskZoneKey(zone)) listedBy = append(listedBy, MaskZoneKey(zone))
} }
case !d.asking[q] && !now.Before(d.retryAt[zone]) && len(d.asking) < maxQueries: case !asking && !d.stopped && !now.Before(d.retryAt[zone]) &&
d.asking[q] = true len(d.asking) < maxQueries:
queryCtx, end := context.WithCancel(context.WithoutCancel(ctx))
d.asking[q] = end
d.queries[zone]++ d.queries[zone]++
go d.ask(context.WithoutCancel(ctx), q) d.queryGoroutines.Go(func() {
defer end()
d.ask(queryCtx, q)
})
} }
} }
return listedBy return listedBy
} }
// StopAsking ends the queries under way, and returns once their
// goroutines have. A query it ends gives no verdict and is no failure: it
// is neither counted, logged nor raised as an alert, and the zone is not
// left alone after it. After StopAsking, ListedBy asks no zone.
func (d *DNSBL) StopAsking() {
d.mu.Lock()
d.stopped = true
for _, end := range d.asking {
end()
}
d.mu.Unlock()
d.queryGoroutines.Wait()
}
// Queries returns how many queries were made to zone. // Queries returns how many queries were made to zone.
func (d *DNSBL) Queries(zone string) int { func (d *DNSBL) Queries(zone string) int {
d.mu.Lock() d.mu.Lock()
@@ -238,30 +286,32 @@ func (d *DNSBL) Load(verdicts []Verdict) {
// query as no longer under way. A query that fails gives no verdict: it // query as no longer under way. A query that fails gives no verdict: it
// is counted, logged and raised as a source_failure alert, which show the // is counted, logged and raised as a source_failure alert, which show the
// zone with its key masked, and the zone is not asked again for // zone with its key masked, and the zone is not asked again for
// failureDelay. // failureDelay. A query ended as ctx ends, by StopAsking, is no failure.
func (d *DNSBL) ask(ctx context.Context, q query) { func (d *DNSBL) ask(ctx context.Context, q query) {
listed, err := d.lookUp(ctx, q) listed, err := d.lookUp(ctx, q)
now := d.params.Now() now := d.params.Now()
isFailure := err != nil && ctx.Err() == nil
d.mu.Lock() d.mu.Lock()
delete(d.asking, q) delete(d.asking, q)
if err == nil { switch {
case err == nil:
d.verdicts.Add(q, Verdict{ d.verdicts.Add(q, Verdict{
Zone: q.zone, Client: q.client, Listed: listed, Fetched: now, Zone: q.zone, Client: q.client, Listed: listed, Fetched: now,
}) })
} else { case isFailure:
d.failures[q.zone]++ d.failures[q.zone]++
d.retryAt[q.zone] = now.Add(failureDelay) d.retryAt[q.zone] = now.Add(failureDelay)
} }
d.mu.Unlock() d.mu.Unlock()
if err != nil { if isFailure {
const failed = "asking a DNSBL zone failed" const failed = "asking a DNSBL zone failed"
shown := config.MaskZoneKey(q.zone) shown := MaskZoneKey(q.zone)
// Raised before it is logged, so that the alert is there once the // Raised before it is logged, so that the alert is there once the
// log line is. // log line is.
+70 -4
View File
@@ -210,6 +210,45 @@ func TestQueryNotAnsweredWithinTheTimeoutFails(t *testing.T) {
}) })
} }
//
//nolint:paralleltest // one at a time, as the comment at the top of this file says
func TestStopAskingEndsTheQueryUnderWayAsNoFailureAndAsksNoMore(
t *testing.T,
) {
synctest.Test(t, func(t *testing.T) {
queue := newQueue()
p := dnsblParams(zone)
p.Alerts = queue
resolver := &resolverStandIn{hanging: true}
dnsbl := newDNSBL(resolver, p)
wantZones(t, dnsbl, listed)
synctest.Wait()
// The query is ended, not waited out, and is no failure.
began := time.Now()
dnsbl.StopAsking()
if waited := time.Since(began); waited != 0 {
t.Errorf("StopAsking returned after %s, want at once", waited)
}
wantQueries(t, dnsbl, 1, 0)
if got := waiting(queue); len(got) != 0 {
t.Errorf("alerts waiting %+v, want none", got)
}
// A client without a verdict is then asked about in no zone.
wantZones(t, dnsbl, unlisted)
synctest.Wait()
wantQueries(t, dnsbl, 1, 0)
wantAsked(t, resolver, listedName)
waitForTheResolver()
})
}
// //
//nolint:paralleltest // one at a time, as the comment at the top of this file says //nolint:paralleltest // one at a time, as the comment at the top of this file says
func TestZoneThatFailsOrRefusesGivesNoVerdictAndIsLeftAloneForAMinute( func TestZoneThatFailsOrRefusesGivesNoVerdictAndIsLeftAloneForAMinute(
@@ -319,8 +358,9 @@ func TestMetricsCountEachZonesQueriesAndThoseThatFailed(t *testing.T) {
"99.2.0.192." + otherZone + ".": {rcode: serverFailure}, "99.2.0.192." + otherZone + ".": {rcode: serverFailure},
}} }}
dnsbl := newDNSBL(resolver, dnsblParams(zone, otherZone)) dnsbl := newDNSBL(resolver, dnsblParams(zone, otherZone))
m := metrics.New(1, "app") m := metrics.New(metrics.Params{
m.AddReputation(reputation.New(params()), dnsbl) BusiestCount: 1, InstanceName: instanceName, DNSBL: dnsbl,
})
wantZones(t, dnsbl, listed) wantZones(t, dnsbl, listed)
synctest.Wait() synctest.Wait()
@@ -365,8 +405,9 @@ func TestZoneKeyIsMaskedInTheVerdictsTheFailuresAndTheMetrics(t *testing.T) {
"99.2.0.192." + keyed + ".": {addrs: []string{listing}}, "99.2.0.192." + keyed + ".": {addrs: []string{listing}},
"100.2.0.192." + keyed + ".": {rcode: serverFailure}, "100.2.0.192." + keyed + ".": {rcode: serverFailure},
}}, p) }}, p)
m := metrics.New(1, "app") m := metrics.New(metrics.Params{
m.AddReputation(reputation.New(params()), dnsbl) BusiestCount: 1, InstanceName: instanceName, DNSBL: dnsbl,
})
// Both clients are asked about before either answer comes, so that // Both clients are asked about before either answer comes, so that
// the failure does not keep the zone from the other query. // the failure does not keep the zone from the other query.
@@ -821,3 +862,28 @@ func wantAsked(t *testing.T, resolver *resolverStandIn, want ...string) {
t.Errorf("asked about %v, want %v", got, want) t.Errorf("asked about %v, want %v", got, want)
} }
} }
func TestMaskZoneKeyMasksTheFirstLabelOfAZoneUnderDqSpamhausNet(
t *testing.T,
) {
t.Parallel()
const (
dronebl = "dnsbl.dronebl.org"
spamhaus = "abcdefghijklmnopqrstuvwxyz.xbl.dq.spamhaus.net"
spamhausMasked = "********.xbl.dq.spamhaus.net"
)
for zone, want := range map[string]string{
spamhaus: spamhausMasked,
spamhaus + ".": spamhausMasked + ".",
"KEY.ZEN.DQ.SPAMHAUS.NET": "********.ZEN.DQ.SPAMHAUS.NET",
dronebl: dronebl,
"dq.spamhaus.net": "dq.spamhaus.net",
spamhaus + ".example": spamhaus + ".example",
} {
if got := reputation.MaskZoneKey(zone); got != want {
t.Errorf("MaskZoneKey(%q) is %q, want %q", zone, got, want)
}
}
}
+77 -11
View File
@@ -7,8 +7,10 @@
// SWW_WAF_DNSBL_ZONES about clients, and keeps their verdicts, and checks // SWW_WAF_DNSBL_ZONES about clients, and keeps their verdicts, and checks
// clients with AbuseIPDB, and keeps their scores and the checks spent // clients with AbuseIPDB, and keeps their scores and the checks spent
// today. The state package writes all of these to files and reads them // today. The state package writes all of these to files and reads them
// from there, so that a restart keeps them too. Last, it fetches // from there, so that a restart keeps them too. It fetches AbuseIPDB's
// AbuseIPDB's blacklist, which it does not keep. // blacklist, which it does not keep. Last, it reads the netblocks, AS
// numbers and percentages that both the settings and the lists hold, and
// masks the keys of DNSBL zones.
package reputation package reputation
import ( import (
@@ -20,12 +22,12 @@ import (
"net/http" "net/http"
"net/netip" "net/netip"
"slices" "slices"
"strconv"
"strings" "strings"
"sync" "sync"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/config"
) )
const ( const (
@@ -40,15 +42,28 @@ const (
) )
var ( var (
errStatus = errors.New("the server answered") errStatus = errors.New("the server answered")
errTooLong = errors.New("the list is longer than 16 MiB") errTooLong = errors.New("the list is longer than 16 MiB")
errNotNetblock = errors.New( errNotNetblockLine = errors.New(
"is not an address or a netblock, such as 192.0.2.0/24", "is not an address or a netblock, such as 192.0.2.0/24",
) )
errNotASNPercent = errors.New( errNotASNPercent = errors.New(
"is not an AS number, : and a percentage, such as AS64496:50") "is not an AS number, : and a percentage, such as AS64496:50")
errNotNetblock = errors.New(
"is not a netblock such as 10.0.0.0/8, or an address")
errNotASN = errors.New("is not an AS number such as AS64496")
errNotPercent = errors.New(
"is not a percentage, a whole number from 0 to 100",
)
) )
// TagList is an address list of SWW_WAF_TAG_LISTS: the tag it adds to the
// requests of the clients it holds, and the URL it is fetched from.
type TagList struct {
Name string
URL string
}
// List is a list as lists.json holds it: the URL it is fetched from, // List is a list as lists.json holds it: the URL it is fetched from,
// when it was last tried, the fetch failed or not, and its last good copy: // when it was last tried, the fetch failed or not, and its last good copy:
// when that was fetched, and its lines, as fetched, comment lines // when that was fetched, and its lines, as fetched, comment lines
@@ -68,7 +83,7 @@ type Params struct {
BlocklistURLs []string BlocklistURLs []string
ASNLimitPercentURL string ASNLimitPercentURL string
// TagLists are the address lists that add a tag (SWW_WAF_TAG_LISTS). // TagLists are the address lists that add a tag (SWW_WAF_TAG_LISTS).
TagLists []config.TagList TagLists []TagList
// Refresh is how long after a list was last fetched or tried it is // Refresh is how long after a list was last fetched or tried it is
// fetched again (SWW_WAF_BLOCKLIST_REFRESH). // fetched again (SWW_WAF_BLOCKLIST_REFRESH).
Refresh time.Duration Refresh time.Duration
@@ -465,7 +480,11 @@ func parseNetblocks(lines []string) (entries, error) {
netblock, ok := parseNetblock(text) netblock, ok := parseNetblock(text)
if !ok { if !ok {
return entries{}, fmt.Errorf("line %d %w", i+1, errNotNetblock) return entries{}, fmt.Errorf(
"line %d %w",
i+1,
errNotNetblockLine,
)
} }
found.netblocks[netblock] = true found.netblocks[netblock] = true
@@ -485,7 +504,7 @@ func parseNetblocks(lines []string) (entries, error) {
// stands for, 192.0.2.0/24, and a mapped netblock shorter than /96, which // stands for, 192.0.2.0/24, and a mapped netblock shorter than /96, which
// stands for none, is refused. // stands for none, is refused.
func parseNetblock(text string) (netip.Prefix, bool) { func parseNetblock(text string) (netip.Prefix, bool) {
netblock, err := config.ParseNetblock(text) netblock, err := ParseNetblock(text)
if err != nil { if err != nil {
return netip.Prefix{}, false return netip.Prefix{}, false
} }
@@ -520,9 +539,9 @@ func parsePercents(lines []string) (entries, error) {
} }
asnText, percentText, _ := strings.Cut(text, ":") asnText, percentText, _ := strings.Cut(text, ":")
asn, asnErr := config.ParseASN(asnText) asn, asnErr := ParseASN(asnText)
percent, percentErr := config.ParsePercent(percentText) percent, percentErr := ParsePercent(percentText)
if asnErr != nil || percentErr != nil { if asnErr != nil || percentErr != nil {
return entries{}, fmt.Errorf("line %d %w", i+1, errNotASNPercent) return entries{}, fmt.Errorf("line %d %w", i+1, errNotASNPercent)
} }
@@ -536,6 +555,53 @@ func parsePercents(lines []string) (entries, error) {
return found, nil return found, nil
} }
// ParseNetblock reads a netblock in CIDR form, such as 10.0.0.0/8. A bare
// address is a netblock of that address alone, a /32 or a /128. A
// blocklist's lines are read with it too.
func ParseNetblock(value string) (netip.Prefix, error) {
if strings.Contains(value, "/") {
netblock, err := netip.ParsePrefix(value)
if err != nil {
return netip.Prefix{}, fmt.Errorf("%q %w", value, errNotNetblock)
}
return netblock.Masked(), nil
}
addr, err := netip.ParseAddr(value)
if err != nil || addr.Zone() != "" {
return netip.Prefix{}, fmt.Errorf("%q %w", value, errNotNetblock)
}
return netip.PrefixFrom(addr, addr.BitLen()), nil
}
// ParseASN reads an AS number such as AS64496, in either case, and
// returns it as the lookup gives it: AS and the number, in capitals and
// without leading zeros. The file SWW_WAF_ASN_LIMIT_PERCENT_URL names is
// read with it too.
func ParseASN(value string) (string, error) {
digits, hasAS := strings.CutPrefix(strings.ToUpper(value), "AS")
number, err := strconv.ParseUint(digits, 10, 32)
if !hasAS || err != nil {
return "", fmt.Errorf("%q %w", value, errNotASN)
}
return "AS" + strconv.FormatUint(number, 10), nil
}
// ParsePercent reads a percentage, a whole number from 0 to 100. The file
// SWW_WAF_ASN_LIMIT_PERCENT_URL names is read with it too.
func ParsePercent(value string) (int64, error) {
percent, err := strconv.ParseInt(value, 10, 64)
if err != nil || percent < 0 || percent > 100 {
return 0, fmt.Errorf("%q %w", value, errNotPercent)
}
return percent, nil
}
// withoutComment returns line without anything after a ; or a #, and // withoutComment returns line without anything after a ; or a #, and
// without the spaces around what is left. // without the spaces around what is left.
func withoutComment(line string) string { func withoutComment(line string) string {
+39 -58
View File
@@ -1,7 +1,7 @@
// Package requestlog writes the lines sww prints on stdout: one // Package requestlog writes the request log sww prints on stdout: JSON
// JSON object per request, two from waf and a short one from rproxy, // lines marked "type":"request", at most two from waf and a short one
// marked "type":"request", and the process's own messages as JSON lines marked // from rproxy for each request. Package processlog writes the process's
// "type":"process". // own messages.
package requestlog package requestlog
import ( import (
@@ -9,86 +9,97 @@ import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"io" "io"
"log/slog"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// Action is what waf did with a request, as its request line names it,
// such as ActionForward.
type Action string
// The action a request line names: what waf did with the request. // The action a request line names: what waf did with the request.
const ( const (
// ActionForward is a request waf allowed, which rproxy passes to // ActionForward is a request waf allowed, which rproxy passes to
// the app. // the app.
ActionForward = "forward" ActionForward Action = "forward"
// ActionRequestLineTooLong is a request whose request line is too long for // ActionRequestLineTooLong is a request whose request line is too long for
// the header limits. // the header limits.
ActionRequestLineTooLong = "request_line_too_long" ActionRequestLineTooLong Action = "request_line_too_long"
// ActionRequestHeaderTooLarge is a request whose headers are over the // ActionRequestHeaderTooLarge is a request whose headers are over the
// header limits. // header limits.
ActionRequestHeaderTooLarge = "request_header_too_large" ActionRequestHeaderTooLarge Action = "request_header_too_large"
// ActionClientIsRateLimited is a request refused because it took its // ActionClientIsRateLimited is a request refused because it took its
// client over a rate limit, which bans the client. // client over a rate limit, which bans the client.
ActionClientIsRateLimited = "client_is_rate_limited" ActionClientIsRateLimited Action = "client_is_rate_limited"
// ActionTemporarilyBanned is a request refused because a ban that runs // ActionTemporarilyBanned is a request refused because a ban that runs
// out covers its client, or because it matched a ban rule or asked for a // out covers its client, or because it matched a ban rule or asked for a
// trap path, and so made such a ban. // trap path, and so made such a ban.
ActionTemporarilyBanned = "temporarily_banned" ActionTemporarilyBanned Action = "temporarily_banned"
// ActionPermanentlyBanned is a request refused because a ban that never // ActionPermanentlyBanned is a request refused because a ban that never
// runs out covers its client, or because the request made such a ban or // runs out covers its client, or because the request made such a ban or
// made its client's ban permanent. // made its client's ban permanent.
ActionPermanentlyBanned = "permanently_banned" ActionPermanentlyBanned Action = "permanently_banned"
// ActionBlockedByBlockRule is a request refused because it matched a // ActionBlockedByBlockRule is a request refused because it matched a
// block rule. // block rule.
ActionBlockedByBlockRule = "blocked_by_block_rule" ActionBlockedByBlockRule Action = "blocked_by_block_rule"
// ActionBlockedByCoreRuleSet is a request refused because the Core Rule // ActionBlockedByCoreRuleSet is a request refused because the Core Rule
// Set scored it at or over SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD. // Set scored it at or over SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD.
ActionBlockedByCoreRuleSet = "blocked_by_core_rule_set" ActionBlockedByCoreRuleSet Action = "blocked_by_core_rule_set"
// ActionClientDenied is a request refused because its client is in // ActionClientDenied is a request refused because its client is in
// SWW_WAF_DENY_NETS, in a blocklist while SWW_WAF_BLOCKLIST_ACTION is deny, // SWW_WAF_DENY_NETS, in a blocklist while SWW_WAF_BLOCKLIST_ACTION is deny,
// or listed by a DNSBL zone, or scored a hit by AbuseIPDB, while // or listed by a DNSBL zone, or scored a hit by AbuseIPDB, while
// SWW_WAF_DNSBL_ABUSEIPDB_ACTION is deny, or that carries a tag in // SWW_WAF_DNSBL_ABUSEIPDB_ACTION is deny, or that carries a tag in
// SWW_WAF_TAG_DENY. // SWW_WAF_TAG_DENY.
ActionClientDenied = "client_denied" ActionClientDenied Action = "client_denied"
// ActionCountryDenied is a request refused because its client's country // ActionCountryDenied is a request refused because its client's country
// is not in SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES. // is not in SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES.
ActionCountryDenied = "country_denied" ActionCountryDenied Action = "country_denied"
// ActionChecksTimedOut is a request refused because the trap paths, the // ActionChecksTimedOut is a request refused because the trap paths, the
// rule files and the Core Rule Set had not finished with it within 500 // rule files and the Core Rule Set had not finished with it within 500
// milliseconds. // milliseconds.
ActionChecksTimedOut = "checks_timed_out" ActionChecksTimedOut Action = "checks_timed_out"
// ActionOwnEndpoint is a request waf answered at one of its own // ActionOwnEndpoint is a request waf answered at one of its own
// endpoints, or a question about a path under /_sww/. // endpoints, or a question about a path under /_sww/.
ActionOwnEndpoint = "own_endpoint" ActionOwnEndpoint Action = "own_endpoint"
) )
// Offence is what a request was held as, as its request line names it.
type Offence string
// OffenceLimit is the offence a request line names for a request that // OffenceLimit is the offence a request line names for a request that
// broke a rate limit or the refusal limit, or whose bytes broke a byte // broke a rate limit or the refusal limit, or whose bytes broke a byte
// limit. // limit.
const OffenceLimit = "limit" const OffenceLimit Offence = "limit"
// LimitHitMinuteRefusals is the limit_hit a request line names for a request // LimitHitMinuteRefusals is the limit_hit a request line names for a request
// that broke the refusal limit. // that broke the refusal limit.
const LimitHitMinuteRefusals = "minute_refusals" const LimitHitMinuteRefusals = "minute_refusals"
// WAFLineKind is which of waf's two lines about a question a request line
// is: WAFLineDecision or WAFLineReport.
type WAFLineKind string
// The waf_line a request line names, which says which of // The waf_line a request line names, which says which of
// waf's two lines about a question it is. // waf's two lines about a question it is.
const ( const (
// WAFLineDecision is waf's line at its decision. // WAFLineDecision is waf's line at its decision.
WAFLineDecision = "decision" WAFLineDecision WAFLineKind = "decision"
// WAFLineReport is waf's line when rproxy's report // WAFLineReport is waf's line when rproxy's report
// comes, with the decision's fields and the report's together. // comes, with the decision's fields and the report's together.
WAFLineReport = "report" WAFLineReport WAFLineKind = "report"
) )
// timeLayout is RFC 3339 with milliseconds. // timeLayout is RFC 3339 with milliseconds.
@@ -101,9 +112,8 @@ const timeLayout = "2006-01-02T15:04:05.000Z07:00"
//nolint:tagliatelle // docs/request-log.md names the fields in snake_case //nolint:tagliatelle // docs/request-log.md names the fields in snake_case
type Line struct { type Line struct {
Type string `json:"type"` Type string `json:"type"`
// WAFLine is WAFLineDecision or WAFLineReport // WAFLine is set on waf's two lines about a question.
// on waf's two lines about a question. WAFLine WAFLineKind `json:"waf_line,omitempty"`
WAFLine string `json:"waf_line,omitempty"`
// The standard web log fields. Scheme is how the client reached rproxy, or // The standard web log fields. Scheme is how the client reached rproxy, or
// the trusted proxy in front of it. // the trusted proxy in front of it.
@@ -149,7 +159,7 @@ type Line struct {
Aborted bool `json:"aborted,omitempty"` Aborted bool `json:"aborted,omitempty"`
// The decision. // The decision.
Action string `json:"action"` Action Action `json:"action"`
// WouldAction is, in observe mode, the action enforce mode would have // WouldAction is, in observe mode, the action enforce mode would have
// taken with a request it would have refused: ActionClientDenied, // taken with a request it would have refused: ActionClientDenied,
// ActionTemporarilyBanned, ActionPermanentlyBanned, ActionCountryDenied, // ActionTemporarilyBanned, ActionPermanentlyBanned, ActionCountryDenied,
@@ -157,7 +167,7 @@ type Line struct {
// ActionBlockedByCoreRuleSet, or, in waf, ActionRequestLineTooLong or // ActionBlockedByCoreRuleSet, or, in waf, ActionRequestLineTooLong or
// ActionRequestHeaderTooLarge for the header limits, or // ActionRequestHeaderTooLarge for the header limits, or
// ActionChecksTimedOut. // ActionChecksTimedOut.
WouldAction string `json:"would_action,omitempty"` WouldAction Action `json:"would_action,omitempty"`
// Heuristic is the name of the heuristic that refused the request, in // Heuristic is the name of the heuristic that refused the request, in
// observe mode too, and HeuristicFigures are the figures it acted on, by // observe mode too, and HeuristicFigures are the figures it acted on, by
// name. // name.
@@ -203,9 +213,8 @@ type Line struct {
// RouteWatchTagsMissing, while SWW_WAF_ROUTEWATCH_URL is set, says why the // RouteWatchTagsMissing, while SWW_WAF_ROUTEWATCH_URL is set, says why the
// request got no tags from routewatch: no_answer_yet, request_failed or // request got no tags from routewatch: no_answer_yet, request_failed or
// no_route. // no_route.
RouteWatchTagsMissing string `json:"routewatch_tags_missing,omitempty"` RouteWatchTagsMissing string `json:"routewatch_tags_missing,omitempty"`
// Offence is the offence the request was held as, OffenceLimit. Offence Offence `json:"offence,omitempty"`
Offence string `json:"offence,omitempty"`
// BanExpires is when the ban the request made, or was refused under, // BanExpires is when the ban the request made, or was refused under,
// ends: a time, or "permanent". // ends: a time, or "permanent".
BanExpires string `json:"ban_expires,omitempty"` BanExpires string `json:"ban_expires,omitempty"`
@@ -247,8 +256,8 @@ type RProxyLine struct {
// WAFBackendNoAnswer is true, the value of // WAFBackendNoAnswer is true, the value of
// SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION. It is "" when neither // SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION. It is "" when neither
// gave one. // gave one.
Disposition string `json:"disposition,omitempty"` Disposition wafprotocol.Disposition `json:"disposition,omitempty"`
WAFBackendNoAnswer bool `json:"waf_backend_no_answer,omitempty"` WAFBackendNoAnswer bool `json:"waf_backend_no_answer,omitempty"`
} }
// Write writes line to w as one JSON line marked "type":"request". // Write writes line to w as one JSON line marked "type":"request".
@@ -300,31 +309,3 @@ func Milliseconds(d time.Duration) float64 {
time.Millisecond/time.Microsecond, time.Millisecond/time.Microsecond,
) )
} }
// NewProcessLogger returns the logger for the process's own messages:
// JSON lines on w, marked "type":"process", with the time in the same form
// as a request line's, and instanceName, SWW_WAF_INSTANCE_NAME, as instance.
// It writes only the messages at level, SWW_WAF_LOG_LEVEL, or more severe;
// the request lines Write writes are never held back.
func NewProcessLogger(
w io.Writer,
instanceName string,
level slog.Level,
) *slog.Logger {
handler := slog.NewJSONHandler(w, &slog.HandlerOptions{
Level: level,
ReplaceAttr: func(groups []string, attr slog.Attr) slog.Attr {
if attr.Key == slog.TimeKey && len(groups) == 0 {
return slog.String(
slog.TimeKey,
FormatTime(attr.Value.Time()),
)
}
return attr
},
})
return slog.New(handler).
With("type", "process", "instance", instanceName)
}
-87
View File
@@ -3,8 +3,6 @@ package requestlog_test
import ( import (
"bytes" "bytes"
"encoding/json" "encoding/json"
"log/slog"
"slices"
"strings" "strings"
"testing" "testing"
"time" "time"
@@ -134,88 +132,3 @@ func TestWriteRProxyLineWritesOneJSONLineWithOnlyItsFields(t *testing.T) {
t.Errorf("wrote %s, want %s", out.String(), want) t.Errorf("wrote %s, want %s", out.String(), want)
} }
} }
func TestProcessLinesAreMarkedProcessAndGiveTheInstance(t *testing.T) {
t.Parallel()
var out bytes.Buffer
requestlog.NewProcessLogger(&out, "host1/blog", slog.LevelInfo).
Info("starting",
"version", "v1")
var fields map[string]any
err := json.Unmarshal(out.Bytes(), &fields)
if err != nil {
t.Fatalf("decode %q: %v", out.String(), err)
}
if fields["type"] != "process" ||
fields["instance"] != "host1/blog" ||
fields["msg"] != "starting" ||
fields["level"] != "INFO" ||
fields["version"] != "v1" {
t.Errorf("process line %v", fields)
}
timeText, _ := fields["time"].(string)
logged, err := time.Parse(time.RFC3339, timeText)
if err != nil || !strings.HasSuffix(timeText, "Z") ||
len(timeText) != len("2006-01-02T15:04:05.000Z") ||
time.Since(logged) > time.Minute {
t.Errorf(
"process line time %q, want now in UTC with milliseconds",
timeText,
)
}
}
func TestProcessLoggerWritesTheMessagesAtItsLevelOrMoreSevere(t *testing.T) {
t.Parallel()
levels := []slog.Level{
slog.LevelDebug, slog.LevelInfo, slog.LevelWarn, slog.LevelError,
}
for i, level := range levels {
t.Run(level.String(), func(t *testing.T) {
t.Parallel()
var out bytes.Buffer
processLog := requestlog.NewProcessLogger(
&out,
"host1/blog",
level,
)
for _, at := range levels {
processLog.Log(t.Context(), at, "message")
}
var got, want []string
for line := range strings.Lines(out.String()) {
var fields struct {
Level string `json:"level"`
}
err := json.Unmarshal([]byte(line), &fields)
if err != nil {
t.Fatalf("decode %q: %v", line, err)
}
got = append(got, fields.Level)
}
for _, written := range levels[i:] {
want = append(want, written.String())
}
if !slices.Equal(got, want) {
t.Errorf("lines at %v, want %v", got, want)
}
})
}
}
+43 -12
View File
@@ -99,8 +99,12 @@ type Client struct {
// answers are by client. Each is added as it comes and never moved up, // answers are by client. Each is added as it comes and never moved up,
// so that the one that came longest ago is the first dropped. // so that the one that came longest ago is the first dropped.
answers *simplelru.LRU[netip.Prefix, Answer] answers *simplelru.LRU[netip.Prefix, Answer]
// asking are the clients whose query is under way. // asking are the clients whose query is under way, each with the
asking map[netip.Prefix]bool // function that ends it. stopped is true once StopAsking has been
// called, and queryGoroutines waits for the goroutine of each query.
asking map[netip.Prefix]context.CancelFunc
stopped bool
queryGoroutines sync.WaitGroup
// retryAt is when routewatch may be asked again after a query failed. // retryAt is when routewatch may be asked again after a query failed.
retryAt time.Time retryAt time.Time
} }
@@ -116,7 +120,7 @@ func New(params Params) *Client {
params: params, params: params,
httpClient: &http.Client{}, httpClient: &http.Client{},
answers: answers, answers: answers,
asking: map[netip.Prefix]bool{}, asking: map[netip.Prefix]context.CancelFunc{},
} }
} }
@@ -124,8 +128,9 @@ func New(params Params) *Client {
// IPv6 group, or why it has no route. It never waits for a query. A client // IPv6 group, or why it has no route. It never waits for a query. A client
// without an answer is asked about in the background, by addr, the address its // without an answer is asked about in the background, by addr, the address its
// request came from, unless its query is under way, a query failed less than // request came from, unless its query is under way, a query failed less than
// failureKept ago, or maxQueries are under way. ctx is the context of the // failureKept ago, maxQueries are under way, or StopAsking has been called. ctx
// client's request, and a query goes on after the request ends. // is the context of the client's request, and a query goes on after the request
// ends, until StopAsking.
func (c *Client) Route( func (c *Client) Route(
ctx context.Context, client netip.Prefix, addr netip.Addr, ctx context.Context, client netip.Prefix, addr netip.Addr,
) (Route, Missing) { ) (Route, Missing) {
@@ -147,15 +152,38 @@ func (c *Client) Route(
return Route{}, RequestFailed return Route{}, RequestFailed
} }
if !c.asking[client] && len(c.asking) < maxQueries { _, asking := c.asking[client]
c.asking[client] = true if !asking && !c.stopped && len(c.asking) < maxQueries {
queryCtx, end := context.WithCancel(context.WithoutCancel(ctx))
c.asking[client] = end
go c.ask(context.WithoutCancel(ctx), client, addr) c.queryGoroutines.Go(func() {
defer end()
c.ask(queryCtx, client, addr)
})
} }
return Route{}, NoAnswerYet return Route{}, NoAnswerYet
} }
// StopAsking ends the queries under way, and returns once their goroutines
// have. A query it ends gives no answer and is no failure: it is not logged,
// and routewatch is not left alone after it. After StopAsking, Route asks
// routewatch nothing.
func (c *Client) StopAsking() {
c.mu.Lock()
c.stopped = true
for _, end := range c.asking {
end()
}
c.mu.Unlock()
c.queryGoroutines.Wait()
}
// Snapshot returns every answer still in use, sorted by client, as // Snapshot returns every answer still in use, sorted by client, as
// routewatch.json lists them. // routewatch.json lists them.
func (c *Client) Snapshot() []Answer { func (c *Client) Snapshot() []Answer {
@@ -205,7 +233,8 @@ func (c *Client) Load(answers []Answer) {
// ask asks routewatch about addr, keeps the answer as client's, and notes the // ask asks routewatch about addr, keeps the answer as client's, and notes the
// query as no longer under way. A query that fails gives no answer, and // query as no longer under way. A query that fails gives no answer, and
// routewatch is not asked again for failureKept. Only the first failure of // routewatch is not asked again for failureKept. Only the first failure of
// those under way is logged. // those under way is logged. A query ended as ctx ends, by StopAsking, is no
// failure.
func (c *Client) ask( func (c *Client) ask(
ctx context.Context, ctx context.Context,
client netip.Prefix, client netip.Prefix,
@@ -213,18 +242,20 @@ func (c *Client) ask(
) { ) {
route, err := c.query(ctx, addr) route, err := c.query(ctx, addr)
now := c.params.Now() now := c.params.Now()
isFailure := err != nil && ctx.Err() == nil
c.mu.Lock() c.mu.Lock()
delete(c.asking, client) delete(c.asking, client)
firstFailure := err != nil && !now.Before(c.retryAt) firstFailure := isFailure && !now.Before(c.retryAt)
if err == nil { switch {
case err == nil:
c.answers.Add(client, Answer{ c.answers.Add(client, Answer{
Client: client, Netblock: route.Netblock, ASN: route.ASN, Answered: now, Client: client, Netblock: route.Netblock, ASN: route.ASN, Answered: now,
}) })
} else { case isFailure:
c.retryAt = now.Add(failureKept) c.retryAt = now.Add(failureKept)
} }
+48
View File
@@ -361,6 +361,54 @@ func TestHangingRoutewatchNeverDelaysAndFailsTheQuery(t *testing.T) {
}) })
} }
func TestStopAskingEndsTheQueryUnderWayAsNoFailureAndAsksNoMore(
t *testing.T,
) {
t.Parallel()
// In a synctest bubble, as TestHangingRoutewatchNeverDelaysAndFailsTheQuery
// says.
synctest.Test(t, func(t *testing.T) {
f := &fixture{t: t, log: &syncBuffer{}}
f.client = routewatch.New(routewatch.Params{
URL: silentURL, Now: time.Now,
ProcessLog: slog.New(slog.NewTextHandler(f.log, nil)),
})
f.route()
synctest.Wait()
// The query is ended, not waited out.
began := time.Now()
f.client.StopAsking()
if waited := time.Since(began); waited != 0 {
t.Errorf("StopAsking returned after %s, want at once", waited)
}
// A query started now would fail within waitLimit. None is, and the
// one ended is no failure: nothing is logged, and routewatch is not
// left alone.
for range 2 {
if _, missing := f.route(); missing != routewatch.NoAnswerYet {
t.Errorf(
"Route gave %q, want %q",
missing,
routewatch.NoAnswerYet,
)
}
time.Sleep(waitLimit)
synctest.Wait()
}
if f.log.String() != "" {
t.Errorf("process log %q, want nothing", f.log.String())
}
})
}
func TestRepeatedRequestsMakeOneQueryAndTheAnswerExpires(t *testing.T) { func TestRepeatedRequestsMakeOneQueryAndTheAnswerExpires(t *testing.T) {
t.Parallel() t.Parallel()
+16 -7
View File
@@ -2,6 +2,7 @@ package rproxy
import ( import (
"bufio" "bufio"
"context"
"errors" "errors"
"io" "io"
"net" "net"
@@ -22,7 +23,9 @@ type requestBody struct {
// body is the visitor's body, ending in an *http.MaxBytesError past // body is the visitor's body, ending in an *http.MaxBytesError past
// SWW_RPROXY_MAX_REQUEST_SIZE. // SWW_RPROXY_MAX_REQUEST_SIZE.
body io.ReadCloser body io.ReadCloser
rq *request // refuse refuses the request with a status, as the request's own refuse
// does.
refuse func(status int)
// held is the part read before the request goes to the app, which Read gives // held is the part read before the request goes to the app, which Read gives
// first. // first.
held []byte held []byte
@@ -68,7 +71,7 @@ func (b *requestBody) Read(p []byte) (int, error) {
case errors.Is(err, io.EOF): case errors.Is(err, io.EOF):
b.received.Store(true) b.received.Store(true)
case errors.As(err, &tooLarge): case errors.As(err, &tooLarge):
b.rq.refuse(http.StatusRequestEntityTooLarge) b.refuse(http.StatusRequestEntityTooLarge)
} }
return n, err return n, err
@@ -86,7 +89,13 @@ type responseBody struct {
// body is the app's body, ending in an *http.MaxBytesError past // body is the app's body, ending in an *http.MaxBytesError past
// SWW_RPROXY_MAX_RESPONSE_SIZE. // SWW_RPROXY_MAX_RESPONSE_SIZE.
body io.ReadCloser body io.ReadCloser
rq *request // refuse refuses the request with a status, as the request's own refuse
// does.
refuse func(status int)
// responseReceived is called once the app's whole answer has arrived.
responseReceived func()
// ctx is the visitor's request's context.
ctx context.Context //nolint:containedctx // the body ends with the request
} }
// Read reads from the app's body. // Read reads from the app's body.
@@ -100,15 +109,15 @@ func (b *responseBody) Read(p []byte) (int, error) {
switch { switch {
case errors.Is(err, io.EOF): case errors.Is(err, io.EOF):
b.rq.responseReceived() b.responseReceived()
case errors.As(err, &tooLarge): case errors.As(err, &tooLarge):
b.rq.refuse(http.StatusBadGateway) b.refuse(http.StatusBadGateway)
return n, errResponseTooLarge return n, errResponseTooLarge
case b.rq.in.Context().Err() == nil: case b.ctx.Err() == nil:
// The answer broke off, not because the visitor went away. If a time // The answer broke off, not because the visitor went away. If a time
// limit cut it, that refusal came first and is the one kept. // limit cut it, that refusal came first and is the one kept.
b.rq.refuse(http.StatusBadGateway) b.refuse(http.StatusBadGateway)
} }
return n, err return n, err
-10
View File
@@ -7,16 +7,6 @@ import (
"strings" "strings"
) )
// The headers rproxy sets on the requests it passes to the app. Go writes every
// header name in this form, as it sends it and as it receives it, so
// X-Client-ASN arrives as X-Client-Asn, and Del removes a visitor's own whatever
// their case; header names are not case-sensitive.
const (
requestIDHeader = "X-Request-ID"
asnHeader = "X-Client-Asn"
countryHeader = "X-Client-Country"
)
// peerAddress is the address of the request's TCP peer, normally traefik, without // peerAddress is the address of the request's TCP peer, normally traefik, without
// any IPv6 zone, which waf refuses. // any IPv6 zone, which waf refuses.
func peerAddress(r *http.Request) netip.Addr { func peerAddress(r *http.Request) netip.Addr {
+15 -1
View File
@@ -1,6 +1,20 @@
package rproxy package rproxy
import "time" import (
"net/http"
"time"
)
// HTTPServer is Go's server s holds, for tests that read its settings.
func (s *Server) HTTPServer() *http.Server {
return s.httpServer
}
// StopBreakerClock has the circuit breaker of s read at as the time, so that how
// long requests take cannot matter to it. It is called before s serves.
func (s *Server) StopBreakerClock(at time.Time) {
s.h.breakerClock = func() time.Time { return at }
}
// Breaker is the circuit breaker, for tests that give it the time. // Breaker is the circuit breaker, for tests that give it the time.
type Breaker struct { type Breaker struct {
+27 -18
View File
@@ -18,9 +18,9 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/config/configtest" "sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/processlog"
"sneak.berlin/go/smallwebwaf/internal/rproxy" "sneak.berlin/go/smallwebwaf/internal/rproxy"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// post is the start of a POST whose headers end with what follows it. // post is the start of a POST whose headers end with what follows it.
@@ -51,12 +51,12 @@ func TestServerHasTheFixedLimits(t *testing.T) {
server := rproxy.New(rproxy.Params{ server := rproxy.New(rproxy.Params{
Config: cfg, Config: cfg,
RequestLog: io.Discard, RequestLog: io.Discard,
ProcessLog: requestlog.NewProcessLogger( ProcessLog: processlog.New(processlog.Params{
io.Discard, Output: io.Discard,
"test", InstanceName: "test",
slog.LevelInfo, Level: slog.LevelInfo,
), }),
}) }).HTTPServer()
if server.Addr != ":8080" || server.MaxHeaderBytes != 28<<10 || if server.Addr != ":8080" || server.MaxHeaderBytes != 28<<10 ||
server.IdleTimeout != 2*time.Minute || server.ReadHeaderTimeout != time.Minute { server.IdleTimeout != 2*time.Minute || server.ReadHeaderTimeout != time.Minute {
@@ -492,7 +492,7 @@ func TestASettingOfOffSetsNoLimit(t *testing.T) {
strings.NewReader("a body"))), strings.NewReader("a body"))),
) )
if got := get(t, addr, waf.HealthPath); got.status != http.StatusOK { if got := get(t, addr, wafprotocol.HealthPath); got.status != http.StatusOK {
t.Errorf("answered %+v, want rproxy's own 200", got) t.Errorf("answered %+v, want rproxy's own 200", got)
} }
}) })
@@ -709,7 +709,7 @@ func TestABodyLeftUnreadIsReadOnlyUntilTheRequestTimeLimit(t *testing.T) {
before := time.Now() before := time.Now()
server.Handler.ServeHTTP(w, req) server.ServeHTTP(w, req)
if w.Code != http.StatusNoContent { if w.Code != http.StatusNoContent {
t.Errorf("status %d, want the app's 204", w.Code) t.Errorf("status %d, want the app's 204", w.Code)
@@ -762,7 +762,7 @@ func TestABodyLeftUnreadIsReadNoMoreOnceTheTimeToSendTheAnswerRunsOut(
before := time.Now() before := time.Now()
server.Handler.ServeHTTP(w, req) server.ServeHTTP(w, req)
if w.Code != http.StatusOK { if w.Code != http.StatusOK {
t.Errorf("status %d, want the app's 200", w.Code) t.Errorf("status %d, want the app's 200", w.Code)
@@ -815,7 +815,7 @@ func TestAnAnswerRProxyGivesItselfHasTheResponseTimeToBeSent(t *testing.T) {
before := time.Now() before := time.Now()
server.Handler.ServeHTTP(w, req) server.ServeHTTP(w, req)
if w.Code != tc.status { if w.Code != tc.status {
t.Errorf("status %d, want %d", w.Code, tc.status) t.Errorf("status %d, want %d", w.Code, tc.status)
@@ -856,7 +856,7 @@ func TestAnAnswerRProxyGivesItselfHasTenSecondsToBeSentAtLeast(
before := time.Now() before := time.Now()
server.Handler.ServeHTTP(w, req) server.ServeHTTP(w, req)
if w.writeDeadline.Before(before.Add(least)) { if w.writeDeadline.Before(before.Add(least)) {
t.Errorf( t.Errorf(
@@ -870,10 +870,14 @@ func TestAnAnswerRProxyGivesItselfHasTenSecondsToBeSentAtLeast(
func TestAVisitorTooSlowToSendTheRestIsAnswered408(t *testing.T) { func TestAVisitorTooSlowToSendTheRestIsAnswered408(t *testing.T) {
t.Parallel() t.Parallel()
// rproxy is waiting for the rest of the body well within this limit, even on
// a busy host, and answers when it runs out, well before dial's deadline.
const limit = 5 * time.Second
standIn := startWAF(t, allow) standIn := startWAF(t, allow)
app := startApp(t, greet) app := startApp(t, greet)
addr, _ := startRProxy(t, standIn.URL, app.URL, map[string]string{ addr, _ := startRProxy(t, standIn.URL, app.URL, map[string]string{
maxRequestTime: "3000", maxRequestTime: strconv.FormatInt(limit.Milliseconds(), 10),
}) })
conn := dial(t, addr) conn := dial(t, addr)
@@ -1262,7 +1266,9 @@ func TestAVisitorThatStopsALittleShortIsReadUntilTheRequestTimeLimit(
) { ) {
t.Parallel() t.Parallel()
const limit = 2 * time.Second // The app's answer arrives well within this limit, even on a busy host, and
// the connection closes when it runs out, well before dial's deadline.
const limit = 5 * time.Second
standIn := startWAF(t, allow) standIn := startWAF(t, allow)
app := startApp(t, answerBeforeTheBody) app := startApp(t, answerBeforeTheBody)
@@ -1427,17 +1433,20 @@ func TestTheRequestTimeLimitDoesNotCutAnAnswerOnceTheAppHasTheRequest(
const answer = "the end of a long answer" const answer = "the end of a long answer"
// The request reaches the app well within this limit, even on a busy host.
const limit = 5 * time.Second
standIn := startWAF(t, allow) standIn := startWAF(t, allow)
app := startApp(t, func(w http.ResponseWriter, r *http.Request) { app := startApp(t, func(w http.ResponseWriter, r *http.Request) {
_, _ = io.Copy(io.Discard, r.Body) _, _ = io.Copy(io.Discard, r.Body)
_ = http.NewResponseController(w).Flush() _ = http.NewResponseController(w).Flush()
time.Sleep(2100 * time.Millisecond) time.Sleep(limit + 100*time.Millisecond)
_, _ = io.WriteString(w, answer) _, _ = io.WriteString(w, answer)
}) })
addr, _ := startRProxy(t, standIn.URL, app.URL, map[string]string{ addr, _ := startRProxy(t, standIn.URL, app.URL, map[string]string{
maxRequestTime: "2000", maxRequestTime: strconv.FormatInt(limit.Milliseconds(), 10),
}) })
got := visit( got := visit(
@@ -1471,7 +1480,7 @@ func TestAVisitorThatStopsALittleShortGetsAnAnswerRProxyGivesAtOnce(
{"a deny", post, http.StatusForbidden}, {"a deny", post, http.StatusForbidden},
{ {
"the health check", "the health check",
"GET " + waf.HealthPath + " HTTP/1.1\r\nHost: site.example\r\n", "GET " + wafprotocol.HealthPath + " HTTP/1.1\r\nHost: site.example\r\n",
http.StatusOK, http.StatusOK,
}, },
} { } {
+14 -14
View File
@@ -12,11 +12,11 @@ import (
"strings" "strings"
"testing" "testing"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// questionBodyMaxBytes is the most of a body a question carries. // questionBodyMaxBytes is the most of a body a question carries.
const questionBodyMaxBytes = waf.QuestionBodyMaxBytes const questionBodyMaxBytes = wafprotocol.QuestionBodyMaxBytes
// chunked is the encoding of a body whose length the visitor did not announce. // chunked is the encoding of a body whose length the visitor did not announce.
const chunked = "chunked" const chunked = "chunked"
@@ -42,9 +42,9 @@ func TestQuestionIsTheVisitorsRequestWithRProxysHeaders(t *testing.T) {
req.Header.Set(forwardedForHeader, "198.51.100.1") req.Header.Set(forwardedForHeader, "198.51.100.1")
req.Header.Set(forwardedProtoHeader, "gopher") req.Header.Set(forwardedProtoHeader, "gopher")
req.Header.Set(requestIDHeader, "the-visitor's") req.Header.Set(requestIDHeader, "the-visitor's")
req.Header.Set(waf.ClientHeader, "198.51.100.2") req.Header.Set(wafprotocol.ClientHeader, "198.51.100.2")
req.Header.Set(waf.PeerHeader, "198.51.100.3") req.Header.Set(wafprotocol.PeerHeader, "198.51.100.3")
req.Header.Set(waf.ContentLengthHeader, "12345") req.Header.Set(wafprotocol.ContentLengthHeader, "12345")
// Go's client then sends no User-Agent, so that one in the question would be // Go's client then sends no User-Agent, so that one in the question would be
// Go's own. // Go's own.
req.Header["User-Agent"] = nil req.Header["User-Agent"] = nil
@@ -73,8 +73,8 @@ func TestQuestionIsTheVisitorsRequestWithRProxysHeaders(t *testing.T) {
want.Set(forwardedForHeader, "198.51.100.1") want.Set(forwardedForHeader, "198.51.100.1")
want.Set(forwardedProtoHeader, "http") want.Set(forwardedProtoHeader, "http")
want.Set(requestIDHeader, asked.header.Get(requestIDHeader)) want.Set(requestIDHeader, asked.header.Get(requestIDHeader))
want.Set(waf.ClientHeader, localhost) want.Set(wafprotocol.ClientHeader, localhost)
want.Set(waf.PeerHeader, localhost) want.Set(wafprotocol.PeerHeader, localhost)
if !maps.EqualFunc(asked.header, want, slices.Equal) { if !maps.EqualFunc(asked.header, want, slices.Equal) {
t.Errorf( t.Errorf(
@@ -124,8 +124,8 @@ func TestQuestionNamesTheVisitorATrustedProxySpeaksFor(t *testing.T) {
wantGreeting(t, visit(t, req)) wantGreeting(t, visit(t, req))
asked := standIn.question(t) asked := standIn.question(t)
if asked.header.Get(waf.ClientHeader) != tc.client || if asked.header.Get(wafprotocol.ClientHeader) != tc.client ||
asked.header.Get(waf.PeerHeader) != localhost || asked.header.Get(wafprotocol.PeerHeader) != localhost ||
asked.header.Get(forwardedForHeader) != visitor || asked.header.Get(forwardedForHeader) != visitor ||
asked.header.Get(forwardedProtoHeader) != tc.proto { asked.header.Get(forwardedProtoHeader) != tc.proto {
t.Errorf("the question's headers are %v", asked.header) t.Errorf("the question's headers are %v", asked.header)
@@ -191,7 +191,7 @@ func TestQuestionNamesTheClientAsTheForwardedAddressesSay(t *testing.T) {
req.Header[forwardedForHeader] = tc.header req.Header[forwardedForHeader] = tc.header
wantGreeting(t, visit(t, req)) wantGreeting(t, visit(t, req))
if got := standIn.question(t).header.Get(waf.ClientHeader); got != if got := standIn.question(t).header.Get(wafprotocol.ClientHeader); got !=
tc.client { tc.client {
t.Errorf("the question names %s, want %s", got, tc.client) t.Errorf("the question names %s, want %s", got, tc.client)
} }
@@ -282,7 +282,7 @@ func wantQuestionBody(
) )
} }
values := asked.header.Values(waf.ContentLengthHeader) values := asked.header.Values(wafprotocol.ContentLengthHeader)
if announced { if announced {
if !slices.Equal(values, []string{strconv.Itoa(size)}) || if !slices.Equal(values, []string{strconv.Itoa(size)}) ||
@@ -583,11 +583,11 @@ func TestAQuestionNamesNoZone(t *testing.T) {
req.RemoteAddr = tc.remoteAddr req.RemoteAddr = tc.remoteAddr
req.Header.Set(forwardedForHeader, tc.forwardedFor) req.Header.Set(forwardedForHeader, tc.forwardedFor)
server.Handler.ServeHTTP(httptest.NewRecorder(), req) server.ServeHTTP(httptest.NewRecorder(), req)
asked := standIn.question(t) asked := standIn.question(t)
if asked.header.Get(waf.ClientHeader) != tc.client || if asked.header.Get(wafprotocol.ClientHeader) != tc.client ||
asked.header.Get(waf.PeerHeader) != tc.peer { asked.header.Get(wafprotocol.PeerHeader) != tc.peer {
t.Errorf("the question's headers are %v", asked.header) t.Errorf("the question's headers are %v", asked.header)
} }
+49 -178
View File
@@ -12,23 +12,18 @@ import (
"net/url" "net/url"
"os" "os"
"strconv" "strconv"
"sync"
"sync/atomic" "sync/atomic"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules" "sneak.berlin/go/smallwebwaf/internal/rules"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// flushAfterEachWrite has ReverseProxy pass on each part of the app's answer as // flushAfterEachWrite has ReverseProxy pass on each part of the app's answer as
// soon as it arrives. // soon as it arrives.
const flushAfterEachWrite time.Duration = -1 const flushAfterEachWrite time.Duration = -1
// reportStatusMax is the highest status waf takes in a report. An app can
// send a higher one, up to 999, and the visitor gets that as it is.
const reportStatusMax = 599
// plainText is the Content-Type of the answers rproxy gives itself. // plainText is the Content-Type of the answers rproxy gives itself.
const plainText = "text/plain; charset=utf-8" const plainText = "text/plain; charset=utf-8"
@@ -67,7 +62,7 @@ type request struct {
// disposition is what is done with the request: waf's valid disposition, // disposition is what is done with the request: waf's valid disposition,
// or SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION. It is "" when neither // or SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION. It is "" when neither
// gave one. // gave one.
disposition string disposition wafprotocol.Disposition
// answered is true when waf gave disposition, which makes a report due, // answered is true when waf gave disposition, which makes a report due,
// and wafNoAnswer when SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION did. // and wafNoAnswer when SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION did.
answered bool answered bool
@@ -86,17 +81,9 @@ type request struct {
// upgraded is the connection to the app once the app has switched protocols, // upgraded is the connection to the app once the app has switched protocols,
// as for a WebSocket, and nil otherwise. // as for a WebSocket, and nil otherwise.
upgraded *upgradedConn upgraded *upgradedConn
// limits are the request's SWW_RPROXY_MAX_REQUEST_TIME_MS and
// mu guards what follows. The time limits run on goroutines of their own, // SWW_RPROXY_MAX_RESPONSE_TIME_MS.
// and the transport starts and stops them, and notes requestEnd, from its limits *timeLimits
// own; once timersStopped is set, neither time limit acts any more.
mu sync.Mutex
timersStopped bool
requestTimer *time.Timer
responseTimer *time.Timer
// requestEnd is when the app had been sent the whole request, or the answer
// started, if that was sooner.
requestEnd time.Time
} }
// newRequest starts handling r: it notes the time, works out the visitor from // newRequest starts handling r: it notes the time, works out the visitor from
@@ -126,17 +113,30 @@ func (h *handler) newRequest(
if r.Body != http.NoBody { if r.Body != http.NoBody {
rq.body = &requestBody{ rq.body = &requestBody{
body: limitBody(r.Body, h.config.MaxRequestSize), body: limitBody(r.Body, h.config.MaxRequestSize),
rq: rq, refuse: rq.refuse,
} }
} }
rq.limits = &timeLimits{
maxRequestTime: h.config.MaxRequestTime,
maxResponseTime: h.config.MaxResponseTime,
start: rq.start,
refuse: rq.refuse,
waitingForVisitor: func() bool {
return rq.body != nil && rq.body.waiting.Load()
},
endVisitorRead: func() {
_ = rq.rc.SetReadDeadline(time.Now())
},
}
return rq return rq
} }
// answerHealth answers the health check. // answerHealth answers the health check.
func (rq *request) answerHealth() { func (rq *request) answerHealth() {
rq.prepareOwnAnswer(rq.requestDeadline()) rq.prepareOwnAnswer(rq.limits.requestDeadline())
rq.out.Header().Set("Content-Type", plainText) rq.out.Header().Set("Content-Type", plainText)
rq.send(http.StatusOK, []byte("ok\n")) rq.send(http.StatusOK, []byte("ok\n"))
} }
@@ -152,9 +152,9 @@ func (rq *request) readFirstPartOfBody() bool {
return true return true
} }
_ = rq.rc.SetReadDeadline(rq.requestDeadline()) _ = rq.rc.SetReadDeadline(rq.limits.requestDeadline())
held, err := io.ReadAll( held, err := io.ReadAll(
io.LimitReader(rq.body, waf.QuestionBodyMaxBytes), io.LimitReader(rq.body, wafprotocol.QuestionBodyMaxBytes),
) )
// From here on, the request timer ends a read of the rest of the body, so that // From here on, the request timer ends a read of the rest of the body, so that
// the answer can name the side rproxy was waiting on. // the answer can name the side rproxy was waiting on.
@@ -192,7 +192,7 @@ func (rq *request) forward(ctx context.Context) {
rq.cancel = cancel rq.cancel = cancel
ctx = httptrace.WithClientTrace(ctx, &httptrace.ClientTrace{ ctx = httptrace.WithClientTrace(ctx, &httptrace.ClientTrace{
WroteRequest: rq.wroteRequest, WroteRequest: rq.limits.wroteRequest,
}) })
out := rq.in.WithContext(ctx) out := rq.in.WithContext(ctx)
@@ -209,7 +209,7 @@ func (rq *request) forward(ctx context.Context) {
ErrorHandler: rq.answerError, ErrorHandler: rq.answerError,
} }
rq.startRequestTimer() rq.limits.startRequestTimer()
rq.upstreamStart = time.Now() rq.upstreamStart = time.Now()
reverseProxy.ServeHTTP(rq.out, out) reverseProxy.ServeHTTP(rq.out, out)
} }
@@ -228,16 +228,17 @@ func (rq *request) rewrite(pr *httputil.ProxyRequest) {
setPathAsSent(pr.Out.URL, pr.In) setPathAsSent(pr.Out.URL, pr.In)
setForwardedHeaders(pr.In, pr.Out, rq.peer, rq.peerTrusted) setForwardedHeaders(pr.In, pr.Out, rq.peer, rq.peerTrusted)
pr.Out.Header.Set(requestIDHeader, rq.id) pr.Out.Header.Set(wafprotocol.RequestIDHeader, rq.id)
pr.Out.Header.Del(asnHeader) // Del removes a visitor's own, whatever their case.
pr.Out.Header.Del(countryHeader) pr.Out.Header.Del(wafprotocol.ASNHeader)
pr.Out.Header.Del(wafprotocol.CountryHeader)
if rq.asn != "" { if rq.asn != "" {
pr.Out.Header.Set(asnHeader, rq.asn) pr.Out.Header.Set(wafprotocol.ASNHeader, rq.asn)
} }
if rq.country != "" { if rq.country != "" {
pr.Out.Header.Set(countryHeader, rq.country) pr.Out.Header.Set(wafprotocol.CountryHeader, rq.country)
} }
} }
@@ -251,7 +252,7 @@ func (rq *request) modifyResponse(res *http.Response) error {
// switch, as when the app's Upgrade is not the visitor's, so the status // switch, as when the app's Upgrade is not the visitor's, so the status
// is noted only once it has taken over the connection, by rq.out's // is noted only once it has taken over the connection, by rq.out's
// Hijack. // Hijack.
rq.stopTimers() rq.limits.stop()
conn, ok := res.Body.(io.ReadWriteCloser) conn, ok := res.Body.(io.ReadWriteCloser)
if ok { if ok {
@@ -262,7 +263,7 @@ func (rq *request) modifyResponse(res *http.Response) error {
return nil return nil
} }
requestEnd := rq.endRequest() requestEnd := rq.limits.endRequest()
// An answer that arrives once SWW_RPROXY_MAX_RESPONSE_TIME_MS has run out is not // An answer that arrives once SWW_RPROXY_MAX_RESPONSE_TIME_MS has run out is not
// passed on: the visitor is answered 504. // passed on: the visitor is answered 504.
@@ -292,9 +293,14 @@ func (rq *request) modifyResponse(res *http.Response) error {
// An answer that arrives before the app has been sent the whole request ends // An answer that arrives before the app has been sent the whole request ends
// the request, and SWW_RPROXY_MAX_RESPONSE_TIME_MS starts. // the request, and SWW_RPROXY_MAX_RESPONSE_TIME_MS starts.
rq.startResponseTimer(requestEnd) rq.limits.startResponseTimer(requestEnd)
res.Body = &responseBody{body: limitBody(res.Body, maxBytes), rq: rq} res.Body = &responseBody{
body: limitBody(res.Body, maxBytes),
refuse: rq.refuse,
responseReceived: rq.limits.stop,
ctx: rq.in.Context(),
}
rq.out.afterStatus = rq.setResponseDeadline rq.out.afterStatus = rq.setResponseDeadline
return nil return nil
@@ -345,7 +351,7 @@ func (rq *request) answer(status int) {
// A visitor found too slow is read no more; any other may go on sending until // A visitor found too slow is read no more; any other may go on sending until
// its time is up, so that the rest of the body is read and the request ends // its time is up, so that the rest of the body is read and the request ends
// cleanly: by Go's server, or by readRestOfBody. // cleanly: by Go's server, or by readRestOfBody.
deadline := rq.requestDeadline() deadline := rq.limits.requestDeadline()
if status == http.StatusRequestTimeout { if status == http.StatusRequestTimeout {
deadline = time.Now() deadline = time.Now()
} }
@@ -391,7 +397,7 @@ func (rq *request) closeAfterAnswer() {
// used that time up first, and at least minSendTime. // used that time up first, and at least minSendTime.
func (rq *request) prepareOwnAnswer(deadline time.Time) { func (rq *request) prepareOwnAnswer(deadline time.Time) {
rq.stopReadingBody(deadline) rq.stopReadingBody(deadline)
rq.endRequest() rq.limits.endRequest()
limit := rq.h.config.MaxResponseTime limit := rq.h.config.MaxResponseTime
if limit > 0 { if limit > 0 {
@@ -412,7 +418,7 @@ func (rq *request) refuse(status int) {
// finish ends the request's time limits, writes its log line and, if waf // finish ends the request's time limits, writes its log line and, if waf
// answered the question, sends the report. ctx is the request's own context. // answered the question, sends the report. ctx is the request's own context.
func (rq *request) finish(ctx context.Context) { func (rq *request) finish(ctx context.Context) {
rq.stopTimers() rq.limits.stop()
// After a cut-off answer, the reading of a body that has not arrived whole // After a cut-off answer, the reading of a body that has not arrived whole
// ends now. Go's server would otherwise keep the connection open while it // ends now. Go's server would otherwise keep the connection open while it
@@ -430,10 +436,10 @@ func (rq *request) finish(ctx context.Context) {
errors.Is(rq.rc.Flush(), os.ErrDeadlineExceeded): errors.Is(rq.rc.Flush(), os.ErrDeadlineExceeded):
rq.stopPassingOnBody(time.Now()) rq.stopPassingOnBody(time.Now())
case rq.refused.Load() == 0: case rq.refused.Load() == 0:
rq.stopPassingOnBody(rq.requestDeadline()) rq.stopPassingOnBody(rq.limits.requestDeadline())
} }
requestEnd := rq.endRequest() requestEnd := rq.limits.endRequest()
end := time.Now() end := time.Now()
requestBytes, responseBytes := rq.bytesEachWay() requestBytes, responseBytes := rq.bytesEachWay()
@@ -470,9 +476,10 @@ func (rq *request) finish(ctx context.Context) {
return return
} }
report := &waf.Report{ // An app can send a status up to 999, and the visitor gets it as it is.
report := &wafprotocol.Report{
RequestID: rq.id, RequestID: rq.id,
Status: min(rq.out.status, reportStatusMax), Status: min(rq.out.status, wafprotocol.ReportStatusMax),
RequestBytes: requestBytes, RequestBytes: requestBytes,
ResponseBytes: responseBytes, ResponseBytes: responseBytes,
DurationTotalMS: requestlog.Milliseconds(end.Sub(rq.start)), DurationTotalMS: requestlog.Milliseconds(end.Sub(rq.start)),
@@ -506,17 +513,6 @@ func (rq *request) bytesEachWay() (int64, int64) {
return requestBytes, responseBytes return requestBytes, responseBytes
} }
// requestDeadline is when the visitor must have sent its whole request, or zero
// when SWW_RPROXY_MAX_REQUEST_TIME_MS is off.
func (rq *request) requestDeadline() time.Time {
limit := rq.h.config.MaxRequestTime
if limit == 0 {
return time.Time{}
}
return rq.start.Add(limit)
}
// stopReadingBody ends, at deadline, the reading of a visitor's body that has not // stopReadingBody ends, at deadline, the reading of a visitor's body that has not
// arrived whole: Go's server then reads no more of it, and closes the connection // arrived whole: Go's server then reads no more of it, and closes the connection
// after the answer. // after the answer.
@@ -573,100 +569,6 @@ func (rq *request) stopPassingOnBody(deadline time.Time) {
rq.stopReadingBody(deadline) rq.stopReadingBody(deadline)
} }
// startRequestTimer starts SWW_RPROXY_MAX_REQUEST_TIME_MS, which runs from the
// request's headers being read until the app has been sent the whole request.
func (rq *request) startRequestTimer() {
deadline := rq.requestDeadline()
if deadline.IsZero() {
return
}
rq.mu.Lock()
defer rq.mu.Unlock()
rq.requestTimer = time.AfterFunc(
time.Until(deadline),
rq.requestTimedOut,
)
}
// requestTimedOut is called when SWW_RPROXY_MAX_REQUEST_TIME_MS runs out while the
// request is still on its way to the app. The answer names the side rproxy was
// waiting on at that moment: 408 when it was waiting for the visitor to send more
// of its body, 504 when it was waiting for the app to be reached or to take what
// it had.
func (rq *request) requestTimedOut() {
rq.mu.Lock()
defer rq.mu.Unlock()
if rq.timersStopped {
return
}
if rq.body == nil || !rq.body.waiting.Load() {
rq.refuse(http.StatusGatewayTimeout)
return
}
rq.refuse(http.StatusRequestTimeout)
// The transport gives up on the app only once its Read of the visitor's body
// returns, so that Read is ended now. The lock keeps this from reaching the
// connection after the request is handled.
_ = rq.rc.SetReadDeadline(time.Now())
}
// wroteRequest is called once the app has been sent the whole request:
// SWW_RPROXY_MAX_REQUEST_TIME_MS ends and SWW_RPROXY_MAX_RESPONSE_TIME_MS starts.
func (rq *request) wroteRequest(info httptrace.WroteRequestInfo) {
if info.Err != nil {
return // the transport gives up, or tries again
}
requestEnd := rq.endRequest()
rq.mu.Lock()
stopTimer(rq.requestTimer)
rq.mu.Unlock()
rq.startResponseTimer(requestEnd)
}
// startResponseTimer starts SWW_RPROXY_MAX_RESPONSE_TIME_MS, which runs from
// requestEnd, unless it has started already.
func (rq *request) startResponseTimer(requestEnd time.Time) {
limit := rq.h.config.MaxResponseTime
if limit == 0 {
return
}
rq.mu.Lock()
defer rq.mu.Unlock()
if rq.timersStopped || rq.responseTimer != nil {
return
}
rq.responseTimer = time.AfterFunc(time.Until(requestEnd.Add(limit)),
rq.responseTimedOut)
}
// responseTimedOut is called when SWW_RPROXY_MAX_RESPONSE_TIME_MS runs out before the
// app has sent its whole answer.
func (rq *request) responseTimedOut() {
rq.mu.Lock()
defer rq.mu.Unlock()
if !rq.timersStopped {
rq.refuse(http.StatusGatewayTimeout)
}
}
// responseReceived is called once the app has sent its whole answer.
func (rq *request) responseReceived() {
rq.stopTimers()
}
// setResponseDeadline sends the status and headers of the app's answer, with // setResponseDeadline sends the status and headers of the app's answer, with
// at least minSendTime, before ReverseProxy reads the answer's body, and then // at least minSendTime, before ReverseProxy reads the answer's body, and then
// the rest has until SWW_RPROXY_MAX_RESPONSE_TIME_MS after the end of the request. // the rest has until SWW_RPROXY_MAX_RESPONSE_TIME_MS after the end of the request.
@@ -676,7 +578,7 @@ func (rq *request) setResponseDeadline() {
return return
} }
deadline := rq.endRequest().Add(limit) deadline := rq.limits.endRequest().Add(limit)
_ = rq.rc.SetWriteDeadline( _ = rq.rc.SetWriteDeadline(
time.Now().Add(max(time.Until(deadline), minSendTime)), time.Now().Add(max(time.Until(deadline), minSendTime)),
@@ -684,34 +586,3 @@ func (rq *request) setResponseDeadline() {
_ = rq.rc.Flush() _ = rq.rc.Flush()
_ = rq.rc.SetWriteDeadline(deadline) _ = rq.rc.SetWriteDeadline(deadline)
} }
// endRequest notes now as the end of the request, unless an end is noted, and
// returns the end noted.
func (rq *request) endRequest() time.Time {
rq.mu.Lock()
defer rq.mu.Unlock()
if rq.requestEnd.IsZero() {
rq.requestEnd = time.Now()
}
return rq.requestEnd
}
// stopTimers stops the request's time limits and keeps any from starting later:
// the app's answer is complete, the connection upgraded, or the request handled.
func (rq *request) stopTimers() {
rq.mu.Lock()
defer rq.mu.Unlock()
rq.timersStopped = true
stopTimer(rq.requestTimer)
stopTimer(rq.responseTimer)
}
// stopTimer stops t, which is nil when its time limit is off.
func stopTimer(t *time.Timer) {
if t != nil {
t.Stop()
}
}
+56 -39
View File
@@ -16,15 +16,15 @@ import (
"io" "io"
"log" "log"
"log/slog" "log/slog"
"net"
"net/http" "net/http"
"net/url" "net/url"
"slices" "slices"
"strings" "strings"
"sync"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// How rproxy keeps connections to the app and to waf open between // How rproxy keeps connections to the app and to waf open between
@@ -47,10 +47,6 @@ const (
idleTimeout = 120 * time.Second idleTimeout = 120 * time.Second
) )
// ownEndpointPrefix starts the path of every request for sww itself, which
// never reaches the app.
const ownEndpointPrefix = "/_sww/"
// allowedMethods lists the methods rproxy passes on, as an Allow header gives // allowedMethods lists the methods rproxy passes on, as an Allow header gives
// them. It answers any other method 405 itself. // them. It answers any other method 405 itself.
// The WebDAV ones are for Nextcloud. // The WebDAV ones are for Nextcloud.
@@ -68,9 +64,8 @@ type Params struct {
// Server is the server `sww rproxy` runs. // Server is the server `sww rproxy` runs.
type Server struct { type Server struct {
*http.Server httpServer *http.Server
h *handler
h *handler
} }
// New returns the server `sww rproxy` runs. Go's server itself refuses // New returns the server `sww rproxy` runs. Go's server itself refuses
@@ -92,32 +87,54 @@ func New(params Params) *Server {
transport: newTransport(), transport: newTransport(),
wafTransport: newWAFTransport(), wafTransport: newWAFTransport(),
reportURL: (&url.URL{ reportURL: (&url.URL{
Scheme: backend.Scheme, Host: backend.Host, Path: waf.ReportPath, Scheme: backend.Scheme,
Host: backend.Host,
Path: wafprotocol.ReportPath,
}).String(), }).String(),
breakerClock: time.Now,
} }
return &Server{ s := &Server{h: h}
Server: &http.Server{ s.httpServer = &http.Server{
Addr: params.Config.ListenAddr, Addr: params.Config.ListenAddr,
Handler: h, Handler: s,
ErrorLog: errorLog, ErrorLog: errorLog,
ReadHeaderTimeout: params.Config.MaxRequestTime, ReadHeaderTimeout: params.Config.MaxRequestTime,
IdleTimeout: idleTimeout, IdleTimeout: idleTimeout,
MaxHeaderBytes: requestHeaderMaxBytes - goServerExtraHeaderBytes, MaxHeaderBytes: requestHeaderMaxBytes - goServerExtraHeaderBytes,
},
h: h,
} }
return s
}
// Serve serves requests on listener until Shutdown or Close, as Go's
// http.Server.Serve does.
func (s *Server) Serve(listener net.Listener) error {
return s.httpServer.Serve(listener)
}
// Shutdown stops serving once the requests in progress have finished, or
// when ctx is done, as Go's http.Server.Shutdown does.
func (s *Server) Shutdown(ctx context.Context) error {
return s.httpServer.Shutdown(ctx)
}
// Close stops serving at once and closes every connection, as Go's
// http.Server.Close does.
func (s *Server) Close() error {
return s.httpServer.Close()
}
// ServeHTTP handles one request from a visitor, as handler.ServeHTTP says.
func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
s.h.ServeHTTP(w, r)
} }
// WaitForReports returns once the reports being sent have gone or failed, // WaitForReports returns once the reports being sent have gone or failed,
// which SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS bounds. Any report after this call // which SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS bounds. Any report after this call
// is dropped: that of a WebSocket that closes while rproxy stops, for one. // is dropped: that of a WebSocket that closes while rproxy stops, for one.
func (s *Server) WaitForReports() { func (s *Server) WaitForReports() {
s.h.reportsMu.Lock() s.h.reports.stopAndWait()
s.h.reportsStopped = true
s.h.reportsMu.Unlock()
s.h.reports.Wait()
} }
// handler is the proxy. It holds what every request shares; what belongs to // handler is the proxy. It holds what every request shares; what belongs to
@@ -133,13 +150,12 @@ type handler struct {
wafTransport http.RoundTripper wafTransport http.RoundTripper
// reportURL is where the reports go. // reportURL is where the reports go.
reportURL string reportURL string
// breaker is the circuit breaker for asks to waf. // breaker is the circuit breaker for asks to waf, and breakerClock gives it
breaker breaker // the time.
// reports waits for the reports being sent. reportsMu guards starting one breaker breaker
// against reportsStopped, which is set once nothing is to wait for another. breakerClock func() time.Time
reports sync.WaitGroup // reports are the goroutines sending reports to waf.
reportsMu sync.Mutex reports reportGoroutines
reportsStopped bool
} }
// newTransport returns what carries requests to the app. It never goes through // newTransport returns what carries requests to the app. It never goes through
@@ -176,7 +192,7 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
defer rq.readRestOfBody() defer rq.readRestOfBody()
defer rq.finish(r.Context()) defer rq.finish(r.Context())
if r.Method == http.MethodGet && r.URL.Path == waf.HealthPath { if r.Method == http.MethodGet && r.URL.Path == wafprotocol.HealthPath {
rq.answerHealth() rq.answerHealth()
return return
@@ -209,20 +225,21 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
rq.noteAnswer(answer) rq.noteAnswer(answer)
switch rq.disposition { switch rq.disposition {
case waf.DispositionAllow: case wafprotocol.DispositionAllow:
rq.passToApp(r.Context()) rq.passToApp(r.Context())
case waf.DispositionClose: case wafprotocol.DispositionClose:
// This panic has Go's server close the connection without an answer, // This panic has Go's server close the connection without an answer,
// and log nothing; the deferred finish still runs. // and log nothing; the deferred finish still runs.
panic(http.ErrAbortHandler) panic(http.ErrAbortHandler)
default: // a deny, an error, or an answer without a valid disposition case wafprotocol.DispositionDeny, "":
// "" is an error, or no valid disposition
rq.answerWith(answer) rq.answerWith(answer)
} }
} }
// passToApp passes the request on to the app, unless a limit or its path keeps // passToApp passes the request on to the app, unless a limit or its path keeps
// it from the app. waf answers a path under ownEndpointPrefix itself, so // it from the app. waf answers a path under wafprotocol.OwnEndpointPrefix
// such a request gets here only when // itself, so such a request gets here only when
// SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION allowed it. // SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION allowed it.
// ctx is the request's own context. // ctx is the request's own context.
func (rq *request) passToApp(ctx context.Context) { func (rq *request) passToApp(ctx context.Context) {
@@ -231,7 +248,7 @@ func (rq *request) passToApp(ctx context.Context) {
switch { switch {
case rq.refused.Load() != 0 || (maxBytes > 0 && rq.in.ContentLength > maxBytes): case rq.refused.Load() != 0 || (maxBytes > 0 && rq.in.ContentLength > maxBytes):
rq.answer(http.StatusRequestEntityTooLarge) rq.answer(http.StatusRequestEntityTooLarge)
case strings.HasPrefix(rq.in.URL.Path, ownEndpointPrefix): case strings.HasPrefix(rq.in.URL.Path, wafprotocol.OwnEndpointPrefix):
rq.answer(http.StatusNotFound) rq.answer(http.StatusNotFound)
default: default:
rq.forward(ctx) rq.forward(ctx)
+73 -40
View File
@@ -21,9 +21,9 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/config/configtest" "sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/processlog"
"sneak.berlin/go/smallwebwaf/internal/rproxy" "sneak.berlin/go/smallwebwaf/internal/rproxy"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
const ( const (
@@ -56,7 +56,7 @@ const (
// The headers the tests set and look at. // The headers the tests set and look at.
const ( const (
dispositionHeader = waf.DispositionHeader dispositionHeader = wafprotocol.DispositionHeader
forwardedForHeader = "X-Forwarded-For" forwardedForHeader = "X-Forwarded-For"
forwardedProtoHeader = "X-Forwarded-Proto" forwardedProtoHeader = "X-Forwarded-Proto"
requestIDHeader = "X-Request-ID" requestIDHeader = "X-Request-ID"
@@ -72,9 +72,9 @@ const (
// What the stand-in for waf answers. // What the stand-in for waf answers.
const ( const (
dispositionAllow = waf.DispositionAllow dispositionAllow = string(wafprotocol.DispositionAllow)
dispositionDeny = waf.DispositionDeny dispositionDeny = string(wafprotocol.DispositionDeny)
dispositionClose = waf.DispositionClose dispositionClose = string(wafprotocol.DispositionClose)
// deniedBody is the body of a deny. // deniedBody is the body of a deny.
deniedBody = "refused by waf" deniedBody = "refused by waf"
) )
@@ -175,7 +175,7 @@ type WAF struct {
*httptest.Server *httptest.Server
questions chan question questions chan question
reports chan waf.Report reports chan wafprotocol.Report
} }
// startWAF starts a stand-in for waf. // startWAF starts a stand-in for waf.
@@ -184,11 +184,11 @@ func startWAF(t *testing.T, answer http.HandlerFunc) *WAF {
s := &WAF{ s := &WAF{
questions: make(chan question, 10), questions: make(chan question, 10),
reports: make(chan waf.Report, 10), reports: make(chan wafprotocol.Report, 10),
} }
s.Server = httptest.NewServer(http.HandlerFunc( s.Server = httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) { func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(waf.ClientHeader) == "" { if r.Header.Get(wafprotocol.ClientHeader) == "" {
s.takeReport(t, w, r) s.takeReport(t, w, r)
return return
@@ -217,11 +217,11 @@ func (s *WAF) takeReport(
) { ) {
t.Helper() t.Helper()
var report waf.Report var report wafprotocol.Report
err := json.NewDecoder(r.Body).Decode(&report) err := json.NewDecoder(r.Body).Decode(&report)
if err != nil || r.Method != http.MethodPost || if err != nil || r.Method != http.MethodPost ||
r.URL.Path != waf.ReportPath { r.URL.Path != wafprotocol.ReportPath {
t.Errorf("%s %s is no report: %v", r.Method, r.URL.Path, err) t.Errorf("%s %s is no report: %v", r.Method, r.URL.Path, err)
} }
@@ -242,7 +242,7 @@ func (s *WAF) wantNoReport(t *testing.T) {
} }
// report waits for a report. // report waits for a report.
func (s *WAF) report(t *testing.T) waf.Report { func (s *WAF) report(t *testing.T) wafprotocol.Report {
t.Helper() t.Helper()
select { select {
@@ -251,7 +251,7 @@ func (s *WAF) report(t *testing.T) waf.Report {
case <-time.After(waitLimit): case <-time.After(waitLimit):
t.Fatal("no report") t.Fatal("no report")
return waf.Report{} return wafprotocol.Report{}
} }
} }
@@ -279,7 +279,7 @@ func startWAFAnsweringReports(
backend := httptest.NewServer(http.HandlerFunc( backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) { func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(waf.ClientHeader) != "" { if r.Header.Get(wafprotocol.ClientHeader) != "" {
answer(w, r) answer(w, r)
return return
@@ -356,6 +356,19 @@ func startServerLogging(
) (string, *rproxy.Server) { ) (string, *rproxy.Server) {
t.Helper() t.Helper()
server := newServer(t, backend, appURL, env, requestLog, processLog)
return serve(t, server), server
}
// newServer returns rproxy, not yet serving, with its request lines going to
// requestLog and its process lines to processLog.
func newServer(
t *testing.T, backend, appURL string, env map[string]string,
requestLog, processLog io.Writer,
) *rproxy.Server {
t.Helper()
settings := map[string]string{ settings := map[string]string{
"SWW_RPROXY_WAF_BACKEND": backend, "SWW_RPROXY_UPSTREAM_URL": appURL, "SWW_RPROXY_WAF_BACKEND": backend, "SWW_RPROXY_UPSTREAM_URL": appURL,
wafBackendTimeout: "60000", wafBackendTimeout: "60000",
@@ -367,15 +380,20 @@ func startServerLogging(
t.Fatalf("settings %v: %v", settings, err) t.Fatalf("settings %v: %v", settings, err)
} }
server := rproxy.New(rproxy.Params{ return rproxy.New(rproxy.Params{
Config: cfg, Config: cfg,
RequestLog: requestLog, RequestLog: requestLog,
ProcessLog: requestlog.NewProcessLogger( ProcessLog: processlog.New(processlog.Params{
processLog, Output: processLog,
"test", InstanceName: "test",
slog.LevelInfo, Level: slog.LevelInfo,
), }),
}) })
}
// serve has server serve until the test ends, and returns its address.
func serve(t *testing.T, server *rproxy.Server) string {
t.Helper()
listener, err := (&net.ListenConfig{}).Listen( listener, err := (&net.ListenConfig{}).Listen(
t.Context(), t.Context(),
@@ -396,7 +414,7 @@ func startServerLogging(
server.WaitForReports() server.WaitForReports()
}) })
return listener.Addr().String(), server return listener.Addr().String()
} }
// answered is the visitor's answer. // answered is the visitor's answer.
@@ -692,7 +710,7 @@ func TestAnAnswerWithoutAContentTypeGetsNone(t *testing.T) {
) )
w := httptest.NewRecorder() w := httptest.NewRecorder()
server.Handler.ServeHTTP(w, req) server.ServeHTTP(w, req)
// Go's server guesses a type when the first part of the body goes before the // Go's server guesses a type when the first part of the body goes before the
// headers are flushed, which is a matter of timing here. So this looks for what // headers are flushed, which is a matter of timing here. So this looks for what
@@ -811,7 +829,7 @@ func TestAnAnswerRProxyHoldsWholeIsSentBeforeItsLogLine(t *testing.T) {
}{ }{
{"a deny", "GET /" + host, deniedBody, http.StatusForbidden}, {"a deny", "GET /" + host, deniedBody, http.StatusForbidden},
{"rproxy's own answer", "GET x:admin" + host, "Bad Request\n", http.StatusBadRequest}, {"rproxy's own answer", "GET x:admin" + host, "Bad Request\n", http.StatusBadRequest},
{"the health check", "GET " + waf.HealthPath + host, "ok\n", http.StatusOK}, {"the health check", "GET " + wafprotocol.HealthPath + host, "ok\n", http.StatusOK},
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
t.Parallel() t.Parallel()
@@ -1434,14 +1452,18 @@ func TestFiveFailedAsksKeepTheNextRequestFromAskingWAF(t *testing.T) {
greet(w, r) greet(w, r)
}) })
addr, out := startRProxy( out := &output{}
t, server := newServer(
standIn.URL, t, standIn.URL, app.URL,
app.URL, map[string]string{wafBackendTimeout: "200"},
map[string]string{ out, out,
wafBackendTimeout: "200",
},
) )
// The breaker's clock stands still, as in
// TestAnswersThatCannotBePassedOnOpenTheBreaker: on a slow host, the five
// failures could take more than five seconds, and the next request could
// come more than a second later.
server.StopBreakerClock(time.Now())
addr := serve(t, server)
// An ask can time out before it reaches the stand-in, so these // An ask can time out before it reaches the stand-in, so these
// may leave fewer than five questions, or leave some late. // may leave fewer than five questions, or leave some late.
@@ -1532,9 +1554,13 @@ func TestAnswersThatCannotBePassedOnOpenTheBreaker(t *testing.T) {
tooLarge(w, r) tooLarge(w, r)
}) })
app := startApp(t, greet) app := startApp(t, greet)
addr, out := startRProxy(t, standIn.URL, app.URL, map[string]string{ out := &output{}
wafBackendTimeout: "60000", server := newServer(t, standIn.URL, app.URL, nil, out, out)
}) // The breaker's clock stands still. On a slow host, the five failures could
// otherwise take more than the five seconds that open the breaker. The next
// request could come more than a second later, when it may ask waf again.
server.StopBreakerClock(time.Now())
addr := serve(t, server)
for range 5 { for range 5 {
if got := get(t, addr, "/"); got.status != http.StatusBadGateway { if got := get(t, addr, "/"); got.status != http.StatusBadGateway {
@@ -1576,19 +1602,19 @@ func TestHealthCheckIsAnsweredWhileWAFIsDown(t *testing.T) {
app := startApp(t, greet) app := startApp(t, greet)
addr, out := startRProxy(t, nowhere, app.URL, nil) addr, out := startRProxy(t, nowhere, app.URL, nil)
got := get(t, addr, waf.HealthPath) got := get(t, addr, wafprotocol.HealthPath)
if got.status != http.StatusOK || got.body != "ok\n" { if got.status != http.StatusOK || got.body != "ok\n" {
t.Errorf("answered %+v, want ok", got) t.Errorf("answered %+v, want ok", got)
} }
line := out.requestLine(t) line := out.requestLine(t)
if line["path"] != waf.HealthPath || line["disposition"] != nil { if line["path"] != wafprotocol.HealthPath || line["disposition"] != nil {
t.Errorf("line %v, want one with no disposition", line) t.Errorf("line %v, want one with no disposition", line)
} }
// With waf up too, it is not asked. // With waf up too, it is not asked.
addr, _ = startRProxy(t, standIn.URL, app.URL, nil) addr, _ = startRProxy(t, standIn.URL, app.URL, nil)
get(t, addr, waf.HealthPath) get(t, addr, wafprotocol.HealthPath)
if len(standIn.questions) != 0 { if len(standIn.questions) != 0 {
t.Error("waf was asked about the health check") t.Error("waf was asked about the health check")
@@ -1749,7 +1775,7 @@ func TestWaitForReportsWaitsForAReportBeingSent(t *testing.T) {
release := make(chan struct{}) release := make(chan struct{})
backend := httptest.NewServer(http.HandlerFunc( backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) { func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(waf.ClientHeader) != "" { if r.Header.Get(wafprotocol.ClientHeader) != "" {
allow(w, r) allow(w, r)
return return
@@ -1809,11 +1835,15 @@ func TestWaitForReportsWaitsForAReportBeingSent(t *testing.T) {
func TestReportOfAWebSocketCountsWhatItCarried(t *testing.T) { func TestReportOfAWebSocketCountsWhatItCarried(t *testing.T) {
t.Parallel() t.Parallel()
// The app's answer to the upgrade arrives well within these limits, even on a
// busy host.
const limit = 5 * time.Second
standIn := startWAF(t, allow) standIn := startWAF(t, allow)
app := startApp(t, echoAfterUpgrade) app := startApp(t, echoAfterUpgrade)
addr, out := startRProxy(t, standIn.URL, app.URL, map[string]string{ addr, out := startRProxy(t, standIn.URL, app.URL, map[string]string{
maxRequestTime: "2000", maxRequestTime: strconv.FormatInt(limit.Milliseconds(), 10),
maxResponseTime: "2000", maxResponseTime: strconv.FormatInt(limit.Milliseconds(), 10),
}) })
conn := dial(t, addr) conn := dial(t, addr)
@@ -1838,7 +1868,10 @@ func TestReportOfAWebSocketCountsWhatItCarried(t *testing.T) {
} }
// Past both time limits, the connection still carries what is sent. // Past both time limits, the connection still carries what is sent.
time.Sleep(2100 * time.Millisecond) time.Sleep(limit + 100*time.Millisecond)
// dial's deadline ran from before the upgrade and this wait.
_ = conn.SetDeadline(time.Now().Add(waitLimit))
const message = "still here\n" const message = "still here\n"
+165
View File
@@ -0,0 +1,165 @@
package rproxy
import (
"net/http"
"net/http/httptrace"
"sync"
"time"
)
// timeLimits are one request's two time limits: SWW_RPROXY_MAX_REQUEST_TIME_MS,
// which runs from the request's headers being read until the app has been sent
// the whole request, and SWW_RPROXY_MAX_RESPONSE_TIME_MS, which runs from then
// until the app has sent its whole answer. A limit of zero is off. Each limit
// runs on a goroutine of its own, and the transport starts and stops them from
// its own.
type timeLimits struct {
maxRequestTime time.Duration
maxResponseTime time.Duration
// start is when the request's headers had been read.
start time.Time
// refuse refuses the request with 408 or 504 when a limit runs out.
refuse func(status int)
// waitingForVisitor reports whether a read of the visitor's body is waiting
// for the visitor to send more.
waitingForVisitor func() bool
// endVisitorRead ends at once a read of the visitor's body that is waiting.
endVisitorRead func()
// mu guards what follows. Once stopped is set, neither limit acts any more.
mu sync.Mutex
stopped bool
requestTimer *time.Timer
responseTimer *time.Timer
// requestEnd is when the app had been sent the whole request, or the answer
// started, if that was sooner.
requestEnd time.Time
}
// requestDeadline is when the visitor must have sent its whole request, or zero
// when SWW_RPROXY_MAX_REQUEST_TIME_MS is off.
func (l *timeLimits) requestDeadline() time.Time {
if l.maxRequestTime == 0 {
return time.Time{}
}
return l.start.Add(l.maxRequestTime)
}
// startRequestTimer starts SWW_RPROXY_MAX_REQUEST_TIME_MS.
func (l *timeLimits) startRequestTimer() {
deadline := l.requestDeadline()
if deadline.IsZero() {
return
}
l.mu.Lock()
defer l.mu.Unlock()
l.requestTimer = time.AfterFunc(time.Until(deadline), l.requestTimedOut)
}
// requestTimedOut is called when SWW_RPROXY_MAX_REQUEST_TIME_MS runs out while
// the request is still on its way to the app. The answer names the side rproxy
// was waiting on at that moment: 408 when it was waiting for the visitor to send
// more of its body, 504 when it was waiting for the app to be reached or to take
// what it had.
func (l *timeLimits) requestTimedOut() {
l.mu.Lock()
defer l.mu.Unlock()
if l.stopped {
return
}
if !l.waitingForVisitor() {
l.refuse(http.StatusGatewayTimeout)
return
}
l.refuse(http.StatusRequestTimeout)
// The transport gives up on the app only once its Read of the visitor's body
// returns, so that Read is ended now. The lock keeps this from reaching the
// connection after the request is handled.
l.endVisitorRead()
}
// wroteRequest is called once the app has been sent the whole request:
// SWW_RPROXY_MAX_REQUEST_TIME_MS ends and SWW_RPROXY_MAX_RESPONSE_TIME_MS
// starts.
func (l *timeLimits) wroteRequest(info httptrace.WroteRequestInfo) {
if info.Err != nil {
return // the transport gives up, or tries again
}
requestEnd := l.endRequest()
l.mu.Lock()
stopTimer(l.requestTimer)
l.mu.Unlock()
l.startResponseTimer(requestEnd)
}
// startResponseTimer starts SWW_RPROXY_MAX_RESPONSE_TIME_MS, which runs from
// requestEnd, unless it has started already.
func (l *timeLimits) startResponseTimer(requestEnd time.Time) {
if l.maxResponseTime == 0 {
return
}
l.mu.Lock()
defer l.mu.Unlock()
if l.stopped || l.responseTimer != nil {
return
}
l.responseTimer = time.AfterFunc(
time.Until(requestEnd.Add(l.maxResponseTime)),
l.responseTimedOut,
)
}
// responseTimedOut is called when SWW_RPROXY_MAX_RESPONSE_TIME_MS runs out
// before the app has sent its whole answer.
func (l *timeLimits) responseTimedOut() {
l.mu.Lock()
defer l.mu.Unlock()
if !l.stopped {
l.refuse(http.StatusGatewayTimeout)
}
}
// endRequest notes now as the end of the request, unless an end is noted, and
// returns the end noted.
func (l *timeLimits) endRequest() time.Time {
l.mu.Lock()
defer l.mu.Unlock()
if l.requestEnd.IsZero() {
l.requestEnd = time.Now()
}
return l.requestEnd
}
// stop stops both limits and keeps either from starting later: the app's
// answer is complete, the connection upgraded, or the request handled.
func (l *timeLimits) stop() {
l.mu.Lock()
defer l.mu.Unlock()
l.stopped = true
stopTimer(l.requestTimer)
stopTimer(l.responseTimer)
}
// stopTimer stops t, which is nil when its limit is off.
func stopTimer(t *time.Timer) {
if t != nil {
t.Stop()
}
}
+171
View File
@@ -0,0 +1,171 @@
package rproxy
import (
"net/http"
"net/http/httptrace"
"sync/atomic"
"testing"
"testing/synctest"
"time"
)
// The tests below run timeLimits in a synctest bubble, where time is a clock of
// the test's own: time.Sleep moves it on at once, and synctest.Wait returns once
// a limit that ran out has refused the request.
const (
testRequestTime = 3 * time.Second
testResponseTime = 5 * time.Second
)
// limitsSeen is what a timeLimits under test did to its request.
type limitsSeen struct {
// refused is the status of the last refusal, or zero.
refused atomic.Int64
// readEnded is true once the read of the visitor's body was ended.
readEnded atomic.Bool
// waiting is what waitingForVisitor reports.
waiting atomic.Bool
}
// newTestLimits returns time limits that started now, with what they do noted
// in seen.
func newTestLimits(seen *limitsSeen) *timeLimits {
return &timeLimits{
maxRequestTime: testRequestTime,
maxResponseTime: testResponseTime,
start: time.Now(),
refuse: func(status int) {
seen.refused.Store(int64(status))
},
waitingForVisitor: seen.waiting.Load,
endVisitorRead: func() {
seen.readEnded.Store(true)
},
}
}
// sleepThenWait moves the bubble's clock on by d and waits for what is then
// due.
func sleepThenWait(d time.Duration) {
time.Sleep(d)
synctest.Wait()
}
func TestRequestTimeRunningOutWhileTheVisitorIsWaitedOnRefuses408(
t *testing.T,
) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
seen := &limitsSeen{}
seen.waiting.Store(true)
limits := newTestLimits(seen)
limits.startRequestTimer()
sleepThenWait(testRequestTime - time.Millisecond)
if got := seen.refused.Load(); got != 0 {
t.Fatalf(
"refused %d before the limit ran out, want nothing",
got,
)
}
sleepThenWait(time.Millisecond)
if got := seen.refused.Load(); got != http.StatusRequestTimeout {
t.Errorf("refused %d, want 408", got)
}
if !seen.readEnded.Load() {
t.Error("the read of the visitor's body goes on, want it ended")
}
})
}
func TestRequestTimeRunningOutWhileTheAppIsWaitedOnRefuses504(
t *testing.T,
) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
seen := &limitsSeen{}
limits := newTestLimits(seen)
limits.startRequestTimer()
sleepThenWait(testRequestTime)
if got := seen.refused.Load(); got != http.StatusGatewayTimeout {
t.Errorf("refused %d, want 504", got)
}
if seen.readEnded.Load() {
t.Error("the read of the visitor's body was ended, want it not")
}
})
}
func TestResponseTimeRunsFromTheEndOfTheRequest(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
seen := &limitsSeen{}
limits := newTestLimits(seen)
limits.startRequestTimer()
sleepThenWait(time.Second)
end := time.Now()
limits.wroteRequest(httptrace.WroteRequestInfo{})
if got := limits.endRequest(); !got.Equal(end) {
t.Errorf("the request ended at %v, want %v", got, end)
}
// Past SWW_RPROXY_MAX_REQUEST_TIME_MS, which the end of the request
// stopped, and just short of SWW_RPROXY_MAX_RESPONSE_TIME_MS.
sleepThenWait(testResponseTime - time.Millisecond)
if got := seen.refused.Load(); got != 0 {
t.Fatalf(
"refused %d before the limit ran out, want nothing",
got,
)
}
sleepThenWait(time.Millisecond)
if got := seen.refused.Load(); got != http.StatusGatewayTimeout {
t.Errorf("refused %d, want 504", got)
}
})
}
func TestStoppedLimitsRefuseNothing(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
seen := &limitsSeen{}
seen.waiting.Store(true)
limits := newTestLimits(seen)
limits.startRequestTimer()
limits.stop()
limits.startResponseTimer(limits.endRequest())
sleepThenWait(testRequestTime + testResponseTime)
if got := seen.refused.Load(); got != 0 {
t.Errorf("refused %d, want nothing", got)
}
if seen.readEnded.Load() {
t.Error("the read of the visitor's body was ended, want it not")
}
})
}
+70 -94
View File
@@ -1,18 +1,17 @@
package rproxy package rproxy
import ( import (
"bytes"
"context" "context"
"encoding/json"
"errors" "errors"
"fmt" "fmt"
"io" "io"
"maps" "maps"
"net/http" "net/http"
"strings" "strings"
"time" "sync"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/rules"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// answerBodyMaxBytes is the most of the body of waf's answer that is read. // answerBodyMaxBytes is the most of the body of waf's answer that is read.
@@ -41,15 +40,12 @@ type wafAnswer struct {
// notes nothing, and no report is due for it: waf holds no decision to report // notes nothing, and no report is due for it: waf holds no decision to report
// on. // on.
func (rq *request) noteAnswer(answer *wafAnswer) { func (rq *request) noteAnswer(answer *wafAnswer) {
disposition := answer.header.Get(waf.DispositionHeader) read, valid := wafprotocol.ReadAnswer(answer.header)
if valid {
switch disposition { rq.disposition = read.Disposition
case waf.DispositionAllow, waf.DispositionDeny,
waf.DispositionClose:
rq.disposition = disposition
rq.answered = true rq.answered = true
rq.asn = answer.header.Get(asnHeader) rq.asn = read.ASN
rq.country = answer.header.Get(countryHeader) rq.country = read.Country
} }
} }
@@ -80,7 +76,7 @@ func (rq *request) handleNoUsableAnswer(ctx context.Context, err error) {
rq.disposition = rq.h.config.WAFBackendNoAnswerDisposition rq.disposition = rq.h.config.WAFBackendNoAnswerDisposition
rq.wafNoAnswer = true rq.wafNoAnswer = true
if rq.disposition == waf.DispositionAllow { if rq.disposition == wafprotocol.DispositionAllow {
rq.passToApp(ctx) rq.passToApp(ctx)
} else { } else {
rq.answer(http.StatusServiceUnavailable) rq.answer(http.StatusServiceUnavailable)
@@ -97,7 +93,7 @@ func (rq *request) askUnlessBreakerOpen(
ctx context.Context, ctx context.Context,
) (*wafAnswer, error) { ) (*wafAnswer, error) {
b := &rq.h.breaker b := &rq.h.breaker
if !b.mayAsk(time.Now()) { if !b.mayAsk(rq.h.breakerClock()) {
return nil, errBreakerOpen return nil, errBreakerOpen
} }
@@ -107,7 +103,7 @@ func (rq *request) askUnlessBreakerOpen(
case err == nil: case err == nil:
b.noteAnswer() b.noteAnswer()
case ctx.Err() == nil: case ctx.Err() == nil:
b.noteFailure(time.Now()) b.noteFailure(rq.h.breakerClock())
} }
return answer, err return answer, err
@@ -120,7 +116,21 @@ func (rq *request) ask(ctx context.Context) (*wafAnswer, error) {
ctx, cancel := context.WithTimeout(ctx, rq.h.config.WAFBackendTimeout) ctx, cancel := context.WithTimeout(ctx, rq.h.config.WAFBackendTimeout)
defer cancel() defer cancel()
question, err := rq.newQuestion(ctx) var held []byte
if rq.body != nil {
held = rq.body.held
}
question, err := wafprotocol.NewQuestion(ctx, wafprotocol.QuestionParams{
Visitor: rq.in,
PathAsSent: rules.Path(rq.in),
Body: held,
WAFBackend: rq.h.config.WAFBackend,
Client: rq.client,
Peer: rq.peer,
RequestID: rq.id,
Scheme: scheme(rq.in, rq.peerTrusted),
})
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -177,69 +187,13 @@ func canBeAsked(r *http.Request) bool {
len(r.Header.Values("User-Agent")) <= 1 len(r.Header.Values("User-Agent")) <= 1
} }
// newQuestion returns the question about the request, as "The question" in
// SPEC.md gives it: the visitor's request as the visitor sent it, with the part
// of the body it holds and rproxy's headers.
func (rq *request) newQuestion(ctx context.Context) (*http.Request, error) {
in := rq.in
var held []byte
if rq.body != nil {
held = rq.body.held
}
question, err := http.NewRequestWithContext( //nolint:gosec // a setting's URL
ctx,
in.Method,
rq.h.config.WAFBackend.String(),
bytes.NewReader(held),
)
if err != nil {
return nil, fmt.Errorf("make the question: %w", err)
}
backend := question.URL
target := *in.URL
target.Scheme = backend.Scheme
target.Host = backend.Host
setPathAsSent(&target, in)
question.URL = &target
question.Host = in.Host
// The length NewRequestWithContext set is that of what is held. This line makes
// a body whose length the visitor did not announce go chunked, as "The question"
// in SPEC.md requires.
question.TransferEncoding = in.TransferEncoding
question.Header = in.Header.Clone()
question.Header.Set(waf.ClientHeader, rq.client.String())
question.Header.Set(waf.PeerHeader, rq.peer.String())
question.Header.Del(waf.ContentLengthHeader)
announced := in.Header.Get("Content-Length")
if announced != "" {
question.Header.Set(waf.ContentLengthHeader, announced)
}
question.Header.Set(requestIDHeader, rq.id)
question.Header.Set("X-Forwarded-Proto", scheme(in, rq.peerTrusted))
// Go's client adds a User-Agent of its own to a request without the key.
_, sent := question.Header["User-Agent"]
if !sent {
question.Header["User-Agent"] = nil
}
return question, nil
}
// answerWith gives the visitor waf's answer, less its disposition. // answerWith gives the visitor waf's answer, less its disposition.
func (rq *request) answerWith(answer *wafAnswer) { func (rq *request) answerWith(answer *wafAnswer) {
rq.prepareOwnAnswer(rq.requestDeadline()) rq.prepareOwnAnswer(rq.limits.requestDeadline())
header := rq.out.Header() header := rq.out.Header()
maps.Copy(header, answer.header) maps.Copy(header, answer.header)
header.Del(waf.DispositionHeader) header.Del(wafprotocol.DispositionHeader)
// Go's server adds a Content-Type it guesses from the body to a response whose // Go's server adds a Content-Type it guesses from the body to a response whose
// headers have no Content-Type key. The key with no value sends none. // headers have no Content-Type key. The key with no value sends none.
@@ -253,15 +207,11 @@ func (rq *request) answerWith(answer *wafAnswer) {
// sendReport sends waf report on a goroutine of its own, so that the // sendReport sends waf report on a goroutine of its own, so that the
// visitor does not wait for it. The report goes on after ctx, the request's own // visitor does not wait for it. The report goes on after ctx, the request's own
// context, has ended. // context, has ended.
func (h *handler) sendReport(ctx context.Context, report *waf.Report) { func (h *handler) sendReport(
h.reportsMu.Lock() ctx context.Context,
defer h.reportsMu.Unlock() report *wafprotocol.Report,
) {
if h.reportsStopped { h.reports.startUnlessStopped(func() {
return
}
h.reports.Go(func() {
err := h.postReport(context.WithoutCancel(ctx), report) err := h.postReport(context.WithoutCancel(ctx), report)
if err != nil { if err != nil {
h.processLog.Warn("a report is dropped", "error", err.Error()) h.processLog.Warn("a report is dropped", "error", err.Error())
@@ -269,25 +219,51 @@ func (h *handler) sendReport(ctx context.Context, report *waf.Report) {
}) })
} }
// postReport posts report to waf, and waits for its answer for up to // reportGoroutines are the goroutines sending reports to waf. None starts once
// SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS. // stopAndWait has been called.
func (h *handler) postReport(ctx context.Context, report *waf.Report) error { type reportGoroutines struct {
body, err := json.Marshal(report) mu sync.Mutex
if err != nil { stopped bool
return fmt.Errorf("encode the report: %w", err) running sync.WaitGroup
}
// startUnlessStopped runs send on a goroutine of its own, or drops it once
// stopAndWait has been called.
func (g *reportGoroutines) startUnlessStopped(send func()) {
g.mu.Lock()
defer g.mu.Unlock()
if g.stopped {
return
} }
g.running.Go(send)
}
// stopAndWait stops any more goroutines from starting, and returns once those
// running have ended.
func (g *reportGoroutines) stopAndWait() {
g.mu.Lock()
g.stopped = true
g.mu.Unlock()
g.running.Wait()
}
// postReport posts report to waf, and waits for its answer for up to
// SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS.
func (h *handler) postReport(
ctx context.Context,
report *wafprotocol.Report,
) error {
ctx, cancel := context.WithTimeout(ctx, h.config.WAFBackendTimeout) ctx, cancel := context.WithTimeout(ctx, h.config.WAFBackendTimeout)
defer cancel() defer cancel()
req, err := http.NewRequestWithContext( //nolint:gosec // a setting's URL req, err := wafprotocol.NewReportRequest(ctx, h.reportURL, report)
ctx, http.MethodPost, h.reportURL, bytes.NewReader(body))
if err != nil { if err != nil {
return fmt.Errorf("make the request: %w", err) return err
} }
req.Header.Set("Content-Type", "application/json")
res, err := h.wafTransport.RoundTrip(req) res, err := h.wafTransport.RoundTrip(req)
if err != nil { if err != nil {
return fmt.Errorf("send the report: %w", err) return fmt.Errorf("send the report: %w", err)
+31 -57
View File
@@ -18,34 +18,31 @@ import (
"slices" "slices"
"strings" "strings"
"sync/atomic" "sync/atomic"
"time"
"github.com/fsnotify/fsnotify" "golang.org/x/net/http/httpguts"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/dirwatch"
) )
// Action is what a rule does when it matches: ActionLog, ActionBlock or
// ActionBan.
type Action string
// The actions a rule takes when it matches. // The actions a rule takes when it matches.
const ( const (
// ActionLog notes the match in the request log, and does nothing else. // ActionLog notes the match in the request log, and does nothing else.
ActionLog = "log" ActionLog Action = "log"
// ActionBlock refuses the request with 403. // ActionBlock refuses the request with 403.
ActionBlock = "block" ActionBlock Action = "block"
// ActionBan refuses the request and bans the client's netblock: the // ActionBan refuses the request and bans the client's netblock: the
// request is a clear sign of attack. // request is a clear sign of attack.
ActionBan = "ban" ActionBan Action = "ban"
) )
// extension ends the name of every rule file. // extension ends the name of every rule file.
const extension = ".rules" const extension = ".rules"
// quietTime is how long SWW_WAF_RULE_FILES_DIR must go without a change before
// the rule files are read again, so that a file still being written, such
// as one saved in place, appended to or copied in with scp, is read only
// once whole.
const quietTime = 2 * time.Second
// headerTarget starts the target that is one request header, // headerTarget starts the target that is one request header,
// header:<Name>. // header:<Name>.
const headerTarget = "header:" const headerTarget = "header:"
@@ -95,8 +92,7 @@ type Rule struct {
// Target is what the regex is matched against, such as path or // Target is what the regex is matched against, such as path or
// header:Accept. // header:Accept.
Target string Target string
// Action is ActionLog, ActionBlock or ActionBan. Action Action
Action string
regex *regexp.Regexp regex *regexp.Regexp
} }
@@ -176,26 +172,18 @@ func (f *Files) Len() int {
} }
// Watch watches Dir until ctx is done, and reads the rule files again // Watch watches Dir until ctx is done, and reads the rule files again
// once Dir has had no change for quietTime, after one is edited, added or // once Dir has had no change for a short time, after one is edited, added
// removed, and after Watch starts watching. If they then hold an error, // or removed, and after Watch starts watching, as dirwatch describes. If
// the rules stay as they were, the error is logged with its file and // they then hold an error, the rules stay as they were, the error is
// line, and the files are read again after the next change. If Dir cannot // logged with its file and line, and the files are read again after the
// be watched, that is logged, and the rules stay as they were loaded. // next change. If Dir cannot be watched, that is logged, and the rules
// While Enabled is false, Watch returns at once. // stay as they were loaded. While Enabled is false, Watch returns at once.
func (f *Files) Watch(ctx context.Context) { func (f *Files) Watch(ctx context.Context) {
if !f.params.Enabled { if !f.params.Enabled {
return return
} }
watcher, err := fsnotify.NewWatcher() watcher, err := dirwatch.New(f.params.Dir)
if err == nil {
defer func() {
_ = watcher.Close()
}()
err = watcher.Add(f.params.Dir)
}
if err != nil { if err != nil {
f.params.ProcessLog.Error("cannot watch the rule files for edits", f.params.ProcessLog.Error("cannot watch the rule files for edits",
"error", err.Error()) "error", err.Error())
@@ -203,36 +191,22 @@ func (f *Files) Watch(ctx context.Context) {
return return
} }
defer func() {
_ = watcher.Close()
}()
f.params.ProcessLog.Info("watching the rule files for edits", f.params.ProcessLog.Info("watching the rule files for edits",
"directory", f.params.Dir) "directory", f.params.Dir)
f.readAfterChanges(ctx, watcher.Events, watcher.Errors) // A change to any file in Dir counts.
} watcher.CallWhenQuiet(ctx,
func(string) bool { return true },
// readAfterChanges reads the rule files again once quietTime has passed f.readAgain,
// without a change from events, until ctx is done, and logs the errors func(err error) {
// from errs. The wait starts at once, as if for a change, so that an edit
// saved after Load read the files, and before Dir was watched, is taken
// in too.
func (f *Files) readAfterChanges(
ctx context.Context, events <-chan fsnotify.Event, errs <-chan error,
) {
quiet := time.NewTimer(quietTime)
defer quiet.Stop()
for {
select {
case <-ctx.Done():
return
case <-events:
quiet.Reset(quietTime)
case <-quiet.C:
f.readAgain()
case err := <-errs:
f.params.ProcessLog.Warn("watching the rule files failed", f.params.ProcessLog.Warn("watching the rule files failed",
"error", err.Error()) "error", err.Error())
} },
} )
} }
// readAgain reads the rule files again, in place of the rules loaded, or // readAgain reads the rule files again, in place of the rules loaded, or
@@ -366,7 +340,7 @@ func parse(line string) (Rule, bool, error) {
return Rule{}, false, errNotRule return Rule{}, false, errNotRule
} }
rule := Rule{ID: fields[1], Target: fields[2], Action: fields[3]} rule := Rule{ID: fields[1], Target: fields[2], Action: Action(fields[3])}
headerName, isHeader := strings.CutPrefix(rule.Target, headerTarget) headerName, isHeader := strings.CutPrefix(rule.Target, headerTarget)
switch { switch {
@@ -378,7 +352,7 @@ func parse(line string) (Rule, bool, error) {
rule.Target, rule.Target,
errNotTarget, errNotTarget,
) )
case isHeader && !config.IsHeaderName(headerName): case isHeader && !httpguts.ValidHeaderFieldName(headerName):
return Rule{}, false, fmt.Errorf( return Rule{}, false, fmt.Errorf(
"the target %q %w", "the target %q %w",
rule.Target, rule.Target,
@@ -394,7 +368,7 @@ func parse(line string) (Rule, bool, error) {
rule.Target, rule.Target,
errHeaderTakenOut, errHeaderTakenOut,
) )
case !slices.Contains([]string{ActionLog, ActionBlock, ActionBan}, rule.Action): case !slices.Contains([]Action{ActionLog, ActionBlock, ActionBan}, rule.Action):
return Rule{}, false, fmt.Errorf( return Rule{}, false, fmt.Errorf(
"the action %q %w", "the action %q %w",
rule.Action, rule.Action,
-165
View File
@@ -1,165 +0,0 @@
package rules
import (
"context"
"log/slog"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"slices"
"testing"
"testing/synctest"
"time"
"github.com/fsnotify/fsnotify"
"sneak.berlin/go/smallwebwaf/internal/alerts"
)
// The tests below run readAfterChanges in a synctest bubble, where time is
// a clock of the test's own: time.Sleep moves it on at once, and
// synctest.Wait returns once readAfterChanges waits again, so that every
// reading due by then is done. The test sends the changes itself, as the
// watch of a directory cannot run in a bubble.
func TestFileWrittenInTwoPartsTakenInOnlyWhole(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "50-app.rules")
writeFile(t, path, "first path block ^/first\n")
files := load(t, dir)
changes := run(t, files)
file, err := os.Create(path) //nolint:gosec // a file the test wrote
if err != nil {
t.Fatalf("create: %v", err)
}
defer func() {
_ = file.Close()
}()
// The first part ends in the middle of a ban rule's regex, which,
// read then, would ban every request.
write(t, file, "first path block ^/first\nprobe path ban ^/")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantMatched(t, files, "/anything")
// The second part starts the wait again.
write(t, file, `\.env$`+"\n")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantMatched(t, files, "/.env")
time.Sleep(time.Nanosecond)
synctest.Wait()
wantMatched(t, files, "/.env", "probe")
wantMatched(t, files, "/anything")
})
}
func TestEditSavedBeforeTheWatchStartsTakenIn(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "50-app.rules")
writeFile(t, path, "first path block ^/first\n")
files := load(t, dir)
// Saved after Load read the files, and before the directory was
// watched, so that no change is seen for it.
writeFile(t, path, "first path block ^/edited\n")
run(t, files)
time.Sleep(quietTime)
synctest.Wait()
wantMatched(t, files, "/edited", "first")
})
}
// load loads the rules in dir.
func load(t *testing.T, dir string) *Files {
t.Helper()
files, err := Load(Params{
Dir: dir, Enabled: true, ProcessLog: slog.New(slog.DiscardHandler),
Alerts: alerts.New(alerts.Params{}),
})
if err != nil {
t.Fatalf("load: %v", err)
}
return files
}
// run runs files' readAfterChanges until the test ends, and returns the
// channel that sends it changes.
func run(t *testing.T, files *Files) chan<- fsnotify.Event {
t.Helper()
changes := make(chan fsnotify.Event)
ctx, stop := context.WithCancel(t.Context())
stopped := make(chan struct{})
go func() {
files.readAfterChanges(ctx, changes, nil)
close(stopped)
}()
t.Cleanup(func() {
stop()
<-stopped
})
return changes
}
// writeFile writes content to the file at path.
func writeFile(t *testing.T, path, content string) {
t.Helper()
err := os.WriteFile(path, []byte(content), 0o600)
if err != nil {
t.Fatalf("write %s: %v", path, err)
}
}
// write writes text to the end of file.
func write(t *testing.T, file *os.File, text string) {
t.Helper()
_, err := file.WriteString(text)
if err != nil {
t.Fatalf("write: %v", err)
}
}
// wantMatched checks the ids of the rules that a GET request for path
// matches, in order.
func wantMatched(t *testing.T, files *Files, path string, want ...string) {
t.Helper()
r := httptest.NewRequestWithContext(t.Context(), http.MethodGet,
"http://app.example"+path, nil)
matched := files.Match(r)
got := make([]string, 0, len(matched))
for _, rule := range matched {
got = append(got, rule.ID)
}
if !slices.Equal(got, want) {
t.Errorf("%s matched %v, want %v", path, got, want)
}
}
+6 -6
View File
@@ -181,7 +181,7 @@ type BanEntry struct {
Netblock netip.Prefix `json:"netblock"` Netblock netip.Prefix `json:"netblock"`
Start time.Time `json:"start"` Start time.Time `json:"start"`
Expires *time.Time `json:"expires"` Expires *time.Time `json:"expires"`
Cause string `json:"cause"` Cause bans.Cause `json:"cause"`
Reason string `json:"reason,omitempty"` Reason string `json:"reason,omitempty"`
Lifted *time.Time `json:"lifted,omitempty"` Lifted *time.Time `json:"lifted,omitempty"`
Notes bans.Notes `json:"notes"` Notes bans.Notes `json:"notes"`
@@ -237,11 +237,11 @@ type routeWatchFile struct {
// //
//nolint:tagliatelle // the state files use snake_case, as the request log does //nolint:tagliatelle // the state files use snake_case, as the request log does
type alertsFile struct { type alertsFile struct {
Version int `json:"version"` Version int `json:"version"`
Cooldowns []alerts.Cooldown `json:"cooldowns"` Cooldowns []alerts.Cooldown `json:"cooldowns"`
Hour alerts.Hour `json:"hour"` Hour alerts.Hour `json:"hour"`
Waiting map[string][]alerts.Alert `json:"waiting"` Waiting map[alerts.DestinationName][]alerts.Alert `json:"waiting"`
AnomalyCounters []anomaly.Counter `json:"anomaly_counters"` AnomalyCounters []anomaly.Counter `json:"anomaly_counters"`
} }
// stateFile is the struct of a state file. Once the file is decoded, its // stateFile is the struct of a state file. Once the file is decoded, its
+36 -24
View File
@@ -1284,18 +1284,18 @@ func TestBansWrittenOnceBansWriteDelayAfterABan(t *testing.T) {
// A second ban, made while the first waits to be written, puts the // A second ban, made while the first waits to be written, puts the
// write off no further, and is written with it. // write off no further, and is written with it.
first, _ := params.Ledger.BanForLimit( first, _ := params.Ledger.Ban(
netip.MustParsePrefix("203.0.113.9/32"), netip.MustParsePrefix("203.0.113.9/32"),
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
time.Sleep(5 * time.Second) time.Sleep(5 * time.Second)
second, _ := params.Ledger.BanForLimit( second, _ := params.Ledger.Ban(
netip.MustParsePrefix("203.0.113.10/32"), netip.MustParsePrefix("203.0.113.10/32"),
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
time.Sleep(5*time.Second - time.Nanosecond) time.Sleep(5*time.Second - time.Nanosecond)
@@ -1358,16 +1358,16 @@ func TestEditJustBeforeAScheduledWriteSurvivesIt(t *testing.T) {
run(t, load(t, params).Run) run(t, load(t, params).Run)
// A ban, and bans.json written with it. // A ban, and bans.json written with it.
params.Ledger.BanForLimit(netip.MustParsePrefix("203.0.113.9/32"), params.Ledger.Ban(netip.MustParsePrefix("203.0.113.9/32"),
midnight(), bans.Notes{}) midnight(), bans.Grounds{Cause: bans.CauseLimit})
time.Sleep(params.BansWriteDelay) time.Sleep(params.BansWriteDelay)
synctest.Wait() synctest.Wait()
// A second ban is to be written BansWriteDelay later. Just before // A second ban is to be written BansWriteDelay later. Just before
// then, an admin saves bans.json with the first ban lifted and // then, an admin saves bans.json with the first ban lifted and
// another added. // another added.
params.Ledger.BanForLimit(netip.MustParsePrefix("203.0.113.10/32"), params.Ledger.Ban(netip.MustParsePrefix("203.0.113.10/32"),
midnight(), bans.Notes{}) midnight(), bans.Grounds{Cause: bans.CauseLimit})
time.Sleep(params.BansWriteDelay - time.Nanosecond) time.Sleep(params.BansWriteDelay - time.Nanosecond)
synctest.Wait() synctest.Wait()
edit(t, dir, bansJSON, permanentBansJSON) edit(t, dir, bansJSON, permanentBansJSON)
@@ -1483,10 +1483,10 @@ func TestFailedWriteLeavesTheFileAsItWas(t *testing.T) {
t.Fatalf("mkdir: %v", err) t.Fatalf("mkdir: %v", err)
} }
params.Ledger.BanForLimit( params.Ledger.Ban(
netip.MustParsePrefix("203.0.113.9/32"), netip.MustParsePrefix("203.0.113.9/32"),
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
params.Limiter.Count( params.Limiter.Count(
netip.MustParsePrefix("203.0.113.9/32"), netip.MustParsePrefix("203.0.113.9/32"),
@@ -1556,10 +1556,10 @@ func TestWritesAreCountedInTheMetrics(t *testing.T) {
t.Fatalf("mkdir: %v", err) t.Fatalf("mkdir: %v", err)
} }
params.Ledger.BanForLimit( params.Ledger.Ban(
netip.MustParsePrefix("203.0.113.9/32"), netip.MustParsePrefix("203.0.113.9/32"),
midnight(), midnight(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
err = files.WriteAll() err = files.WriteAll()
@@ -1758,8 +1758,8 @@ func TestEditOfRouteWatchAndAlertsFilesTakenIn(t *testing.T) {
Event: alerts.EventBan, Netblock: netip.MustParsePrefix("198.51.100.0/24"), Event: alerts.EventBan, Netblock: netip.MustParsePrefix("198.51.100.0/24"),
Sent: midnight(), Sent: midnight(),
}}, }},
Hour: alerts.Hour{HeldBack: map[string]int{}}, Hour: alerts.Hour{HeldBack: map[alerts.Event]int{}},
Waiting: map[string][]alerts.Alert{ Waiting: map[alerts.DestinationName][]alerts.Alert{
alerts.DestinationWebhook: { alerts.DestinationWebhook: {
{Event: alerts.EventFileError, Time: midnight()}, {Event: alerts.EventFileError, Time: midnight()},
}, },
@@ -2046,8 +2046,8 @@ func TestBanWithoutACauseTakenInAsAnAdmins(t *testing.T) {
var written struct { var written struct {
Bans []struct { Bans []struct {
Cause string `json:"cause"` Cause bans.Cause `json:"cause"`
Reason string `json:"reason"` Reason string `json:"reason"`
} `json:"bans"` } `json:"bans"`
} }
@@ -2135,7 +2135,11 @@ func TestBanLiftedByAnEditWhileRunning(t *testing.T) {
// The ban that liftedBansJSON lifts, before it is lifted. // The ban that liftedBansJSON lifts, before it is lifted.
netblock := netip.MustParsePrefix(liftedClient + "/32") netblock := netip.MustParsePrefix(liftedClient + "/32")
params.Ledger.BanForLimit(netblock, midnight(), bans.Notes{}) params.Ledger.Ban(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
_, banned, _ := params.Ledger.Find(netblock.Addr(), afterLifting()) _, banned, _ := params.Ledger.Find(netblock.Addr(), afterLifting())
if !banned { if !banned {
@@ -2426,7 +2430,7 @@ func midnight() time.Time {
// counts, with thresholds fill does not reach. // counts, with thresholds fill does not reach.
func newParams(dir string) state.Params { func newParams(dir string) state.Params {
discard := slog.New(slog.DiscardHandler) discard := slog.New(slog.DiscardHandler)
m := metrics.New(1, "app") m := metrics.New(metrics.Params{BusiestCount: 1, InstanceName: "app"})
queue := alerts.New(alerts.Params{ queue := alerts.New(alerts.Params{
WebhookURL: &url.URL{Scheme: "https", Host: "alerts.example"}, WebhookURL: &url.URL{Scheme: "https", Host: "alerts.example"},
Events: alerts.Events(), Events: alerts.Events(),
@@ -2448,7 +2452,7 @@ func newParams(dir string) state.Params {
AttackBanDuration: 7 * 24 * time.Hour, AttackBanDuration: 7 * 24 * time.Hour,
MaxBans: 5000, MaxBans: 5000,
}), }),
Limiter: ratelimit.New(ratelimit.Limits{}, 20000), Limiter: ratelimit.New(ratelimit.Params{MaxClients: 20000}),
GeoJS: geojs.New(geojs.Params{Now: midnight}), GeoJS: geojs.New(geojs.Params{Now: midnight}),
Lists: reputation.New(reputation.Params{ Lists: reputation.New(reputation.Params{
BlocklistURLs: []string{ BlocklistURLs: []string{
@@ -2510,11 +2514,15 @@ func fill(params state.Params) {
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
params.Ledger.Load([]bans.Ban{permanentBan()}) params.Ledger.Load([]bans.Ban{permanentBan()})
params.Ledger.BanForLimit(client, now, bans.Notes{ params.Ledger.Ban(client, now, bans.Grounds{
ASN: asn, ASName: asName, Country: "DE", Limit: 1, Cause: bans.CauseLimit,
Notes: bans.Notes{ASN: asn, ASName: asName, Country: "DE", Limit: 1},
}) })
params.Ledger.BanForAttack(netip.MustParsePrefix("192.0.2.1/32"), now, params.Ledger.Ban(netip.MustParsePrefix("192.0.2.1/32"), now,
bans.Notes{RuleID: "env-file", Target: "path"}) bans.Grounds{
Cause: bans.CauseAttack,
Notes: bans.Notes{RuleID: "env-file", Target: "path"},
})
for _, c := range []string{"2001:db8::/64", "203.0.113.9/32", "192.0.2.1/32"} { for _, c := range []string{"2001:db8::/64", "203.0.113.9/32", "192.0.2.1/32"} {
params.Limiter.Count(netip.MustParsePrefix(c), now, whole) params.Limiter.Count(netip.MustParsePrefix(c), now, whole)
@@ -2700,7 +2708,11 @@ func wantLiftedBanKept(
} }
// Were the lifted ban counted, the next would last three hours. // Were the lifted ban counted, the next would last three hours.
ban, _ := ledger.BanForLimit(netblock, afterLifting(), bans.Notes{}) ban, _ := ledger.Ban(
netblock,
afterLifting(),
bans.Grounds{Cause: bans.CauseLimit},
)
if ban.Expires.Sub(ban.Start) != time.Hour { if ban.Expires.Sub(ban.Start) != time.Hour {
t.Errorf( t.Errorf(
"the next ban lasts %s, want 1h", "the next ban lasts %s, want 1h",
+2 -2
View File
@@ -13,7 +13,7 @@ import (
"github.com/spf13/viper" "github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// healthCheckTimeout bounds the whole health check. // healthCheckTimeout bounds the whole health check.
@@ -116,7 +116,7 @@ func askHealthEndpoint(ctx context.Context, listenAddr string) error {
health := "http://" + net.JoinHostPort( health := "http://" + net.JoinHostPort(
"127.0.0.1", "127.0.0.1",
port, port,
) + waf.HealthPath ) + wafprotocol.HealthPath
req, err := http.NewRequestWithContext( req, err := http.NewRequestWithContext(
ctx, ctx,
+9 -1
View File
@@ -95,12 +95,20 @@ func background(loop func(context.Context)) fx.Hook {
) )
} }
// daemonServer is the server a daemon runs, waf.Server or rproxy.Server: the
// methods of Go's http.Server that serve starts and stops it with.
type daemonServer interface {
Serve(listener net.Listener) error
Shutdown(ctx context.Context) error
Close() error
}
// serve serves requests on listener from the start. At the stop it gives the // serve serves requests on listener from the start. At the stop it gives the
// requests in progress shutdownTimeout to finish, then closes their // requests in progress shutdownTimeout to finish, then closes their
// connections. When serving fails, it logs why and sends the error to // connections. When serving fails, it logs why and sends the error to
// failed. // failed.
func serve( func serve(
lc fx.Lifecycle, server *http.Server, listener net.Listener, lc fx.Lifecycle, server daemonServer, listener net.Listener,
processLog *slog.Logger, failed servingFailed, processLog *slog.Logger, failed servingFailed,
) { ) {
served := make(chan error, 1) served := make(chan error, 1)
+13 -9
View File
@@ -4,13 +4,12 @@ import (
"context" "context"
"log/slog" "log/slog"
"net" "net"
"net/http"
"os" "os"
"go.uber.org/fx" "go.uber.org/fx"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/processlog"
"sneak.berlin/go/smallwebwaf/internal/rproxy" "sneak.berlin/go/smallwebwaf/internal/rproxy"
) )
@@ -23,11 +22,11 @@ import (
// setting for either. // setting for either.
func RunRProxy(ctx context.Context, params Params) int { func RunRProxy(ctx context.Context, params Params) int {
hostname, _ := os.Hostname() // "" when the host has no name to give hostname, _ := os.Hostname() // "" when the host has no name to give
processLog := requestlog.NewProcessLogger( processLog := processlog.New(processlog.Params{
params.Stdout, Output: params.Stdout,
hostname, InstanceName: hostname,
slog.LevelInfo, Level: slog.LevelInfo,
) })
cfg, err := config.RProxyFromEnvironment(params.Viper) cfg, err := config.RProxyFromEnvironment(params.Viper)
if err != nil { if err != nil {
@@ -44,8 +43,13 @@ func RunRProxy(ctx context.Context, params Params) int {
app := fx.New( app := fx.New(
fx.Supply(cfg, processLog, requestLog{params.Stdout}, failed), fx.Supply(cfg, processLog, requestLog{params.Stdout}, failed),
fx.Provide( fx.Provide(
newRProxyServer, // run waits for rproxy.Server's reports, and serve takes it as a
func(server *rproxy.Server) *http.Server { return server.Server }, // daemonServer.
fx.Annotate(
newRProxyServer,
fx.As(fx.Self()),
fx.As(new(daemonServer)),
),
newRProxyListener, newRProxyListener,
), ),
fx.Invoke(serve), fx.Invoke(serve),
+21 -24
View File
@@ -21,7 +21,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config/configtest" "sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/sww" "sneak.berlin/go/smallwebwaf/internal/sww"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
const ( const (
@@ -164,7 +164,7 @@ func TestABanThroughOneSiteRefusesTheClientOnTheOther(t *testing.T) {
// One request's three lines carry one id, rproxy's with waf's answer. // One request's three lines carry one id, rproxy's with waf's answer.
line := firstOut.line(t, "path", bannedProbe) line := firstOut.line(t, "path", bannedProbe)
if line["disposition"] != waf.DispositionDeny || if line["disposition"] != string(wafprotocol.DispositionDeny) ||
line["client_ip"] != placed { line["client_ip"] != placed {
t.Errorf("rproxy's line is %v", line) t.Errorf("rproxy's line is %v", line)
} }
@@ -249,7 +249,7 @@ func TestTheSiteAnswersWhileWAFIsStoppedAndStartedAgain(t *testing.T) {
wantStatus(t, site+"/after-stop", placed, http.StatusOK) wantStatus(t, site+"/after-stop", placed, http.StatusOK)
line := out.line(t, "path", "/after-stop") line := out.line(t, "path", "/after-stop")
if line["disposition"] != waf.DispositionAllow || if line["disposition"] != string(wafprotocol.DispositionAllow) ||
line["waf_backend_no_answer"] != true { line["waf_backend_no_answer"] != true {
t.Errorf("rproxy's line is %v", line) t.Errorf("rproxy's line is %v", line)
} }
@@ -437,7 +437,7 @@ func TestAChunkedBodyStaysABodyAtWAF(t *testing.T) {
const smuggledID = "smuggled-request" const smuggledID = "smuggled-request"
smuggled := "GET " + bannedProbe + " HTTP/1.1\r\nHost: site.example\r\n" + smuggled := "GET " + bannedProbe + " HTTP/1.1\r\nHost: site.example\r\n" +
waf.ClientHeader + ": " + otherClient + "\r\n" + wafprotocol.ClientHeader + ": " + otherClient + "\r\n" +
"X-Request-ID: " + smuggledID + "\r\n\r\n" "X-Request-ID: " + smuggledID + "\r\n\r\n"
status := sendRaw( status := sendRaw(
@@ -605,10 +605,11 @@ func TestRProxyAsksAWAFThatSpeaksHTTP2OverHTTP1(t *testing.T) {
protocols := make(chan string, 10) protocols := make(chan string, 10)
backend := httptest.NewUnstartedServer(http.HandlerFunc( backend := httptest.NewUnstartedServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) { func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(waf.ClientHeader) != "" { if r.Header.Get(wafprotocol.ClientHeader) != "" {
protocols <- r.Proto protocols <- r.Proto
w.Header().Set(waf.DispositionHeader, waf.DispositionDeny) w.Header().Set(wafprotocol.DispositionHeader,
string(wafprotocol.DispositionDeny))
w.WriteHeader(http.StatusForbidden) w.WriteHeader(http.StatusForbidden)
} }
})) }))
@@ -690,8 +691,9 @@ func TestRProxyStoppingWaitsForAReportBeingSent(t *testing.T) {
release := make(chan struct{}) release := make(chan struct{})
backend := httptest.NewServer(http.HandlerFunc( backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) { func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(waf.ClientHeader) != "" { if r.Header.Get(wafprotocol.ClientHeader) != "" {
w.Header().Set(waf.DispositionHeader, waf.DispositionAllow) w.Header().Set(wafprotocol.DispositionHeader,
string(wafprotocol.DispositionAllow))
return return
} }
@@ -755,8 +757,7 @@ func TestRProxyStoppingWaitsForAReportBeingSent(t *testing.T) {
} }
} }
// sendAndKeepOpen sends request to rproxy at addr, and returns the connection. A // sendAndKeepOpen sends request to rproxy at addr, and returns the connection.
// read from it ends with an error once waitLimit is up.
func sendAndKeepOpen(t *testing.T, addr, request string) net.Conn { func sendAndKeepOpen(t *testing.T, addr, request string) net.Conn {
t.Helper() t.Helper()
@@ -769,11 +770,6 @@ func sendAndKeepOpen(t *testing.T, addr, request string) net.Conn {
_ = conn.Close() _ = conn.Close()
}) })
err = conn.SetDeadline(time.Now().Add(waitLimit))
if err != nil {
t.Fatalf("set the deadline: %v", err)
}
_, err = io.WriteString(conn, request) _, err = io.WriteString(conn, request)
if err != nil { if err != nil {
t.Fatalf("send: %v", err) t.Fatalf("send: %v", err)
@@ -785,10 +781,7 @@ func sendAndKeepOpen(t *testing.T, addr, request string) net.Conn {
func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) { func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
t.Parallel() t.Parallel()
const ( const cutOffAfter = 5 * time.Second
cutOffAfter = 5 * time.Second
upstreamSetting = "SWW_RPROXY_UPSTREAM_URL"
)
asked := make(chan struct{}) asked := make(chan struct{})
release := make(chan struct{}) release := make(chan struct{})
@@ -802,7 +795,8 @@ func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
backend := httptest.NewServer(http.HandlerFunc( backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) { func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set(waf.DispositionHeader, waf.DispositionAllow) w.Header().Set(wafprotocol.DispositionHeader,
string(wafprotocol.DispositionAllow))
})) }))
t.Cleanup(backend.Close) t.Cleanup(backend.Close)
@@ -810,9 +804,9 @@ func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
defer stop() defer stop()
env := map[string]string{ env := map[string]string{
rproxyListenAddr: localhost + ":0", rproxyListenAddr: localhost + ":0",
upstreamSetting: app.URL, rproxyUpstreamURL: app.URL,
wafBackend: backend.URL, wafBackend: backend.URL,
} }
out := &output{} out := &output{}
exited := make(chan int, 1) exited := make(chan int, 1)
@@ -838,6 +832,9 @@ func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
stop() stop()
// Counted from the stop, so a slow wait for the app cannot shorten it.
_ = conn.SetReadDeadline(stopped.Add(cutOffAfter + waitLimit))
text, err := io.ReadAll(conn) text, err := io.ReadAll(conn)
if err != nil || len(text) != 0 { if err != nil || len(text) != 0 {
t.Errorf("read %q (%v), want the connection cut off", text, err) t.Errorf("read %q (%v), want the connection cut off", text, err)
@@ -935,7 +932,7 @@ func TestMainRunsRProxy(t *testing.T) {
addr, stop := startRProxyProcess(t, wafBackend+"=http://127.0.0.1:9") addr, stop := startRProxyProcess(t, wafBackend+"=http://127.0.0.1:9")
req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, req, err := http.NewRequestWithContext(t.Context(), http.MethodGet,
"http://"+addr+waf.HealthPath, http.NoBody) "http://"+addr+wafprotocol.HealthPath, http.NoBody)
if err != nil { if err != nil {
t.Fatalf("new request: %v", err) t.Fatalf("new request: %v", err)
} }
+52 -30
View File
@@ -10,13 +10,14 @@ package sww
import ( import (
"context" "context"
"fmt" "errors"
"io" "io"
"os" "os"
"os/signal" "os/signal"
"syscall" "syscall"
"time" "time"
"github.com/spf13/cobra"
"github.com/spf13/viper" "github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
@@ -26,14 +27,7 @@ import (
// know. // know.
const usageStatus = 2 const usageStatus = 2
// usage is what sww prints then. var errNoCommand = errors.New("no command given")
const usage = `usage: sww <command>
commands:
rproxy the reverse proxy in front of one app
waf decides for rproxy, and keeps bans, counters and history
healthcheck the container's health check
`
// Params are what RunWAF and RunRProxy need from the process. // Params are what RunWAF and RunRProxy need from the process.
type Params struct { type Params struct {
@@ -51,14 +45,9 @@ type Params struct {
// Main runs the command os.Args names, and returns the process's exit status. // Main runs the command os.Args names, and returns the process's exit status.
// `sww rproxy` and `sww waf` run until SIGTERM or SIGINT. // `sww rproxy` and `sww waf` run until SIGTERM or SIGINT.
// With no command, or one it does not know, it prints the commands and returns // With no command, or one it does not know, cobra prints the error and the
// usageStatus. // commands to stderr, and Main returns usageStatus.
func Main(version string) int { func Main(version string) int {
args := os.Args[1:]
if len(args) == 0 {
return printUsage(os.Stderr)
}
ctx, stop := signal.NotifyContext(context.Background(), ctx, stop := signal.NotifyContext(context.Background(),
syscall.SIGTERM, os.Interrupt) syscall.SIGTERM, os.Interrupt)
defer stop() defer stop()
@@ -69,21 +58,54 @@ func Main(version string) int {
Stdout: os.Stdout, Stdout: os.Stdout,
} }
switch args[0] { // status is what the command that ran returns.
case "rproxy": status := 0
return RunRProxy(ctx, params)
case "waf": root := &cobra.Command{
return RunWAF(ctx, params) Use: "sww",
case "healthcheck": Short: "a web application firewall",
return HealthCheck(ctx, args[1:], params.Viper, os.Stderr) Args: cobra.NoArgs,
default: RunE: func(*cobra.Command, []string) error {
return printUsage(os.Stderr) return errNoCommand
},
} }
} root.CompletionOptions.DisableDefaultCmd = true
// printUsage writes the commands to w, and returns usageStatus. root.AddCommand(
func printUsage(w io.Writer) int { &cobra.Command{
_, _ = fmt.Fprint(w, usage) Use: "rproxy",
Short: "the reverse proxy in front of one app",
Run: func(cmd *cobra.Command, _ []string) {
status = RunRProxy(cmd.Context(), params)
},
},
&cobra.Command{
Use: "waf",
Short: "decides for rproxy, and keeps bans, counters and history",
Run: func(cmd *cobra.Command, _ []string) {
status = RunWAF(cmd.Context(), params)
},
},
&cobra.Command{
Use: "healthcheck",
Short: "the container's health check",
// HealthCheck refuses every argument itself, flags included.
DisableFlagParsing: true,
Run: func(cmd *cobra.Command, args []string) {
status = HealthCheck(
cmd.Context(),
args,
params.Viper,
cmd.ErrOrStderr(),
)
},
},
)
return usageStatus err := root.ExecuteContext(ctx)
if err != nil {
return usageStatus
}
return status
} }
+2 -2
View File
@@ -23,7 +23,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config/configtest" "sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest" "sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
"sneak.berlin/go/smallwebwaf/internal/sww" "sneak.berlin/go/smallwebwaf/internal/sww"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
const ( const (
@@ -1813,7 +1813,7 @@ func statusFrom(t *testing.T, url, from string) int {
} }
req.Header.Set("X-Forwarded-For", from) req.Header.Set("X-Forwarded-For", from)
req.Header.Set(waf.ClientHeader, from) req.Header.Set(wafprotocol.ClientHeader, from)
req.Header.Set("X-Request-ID", rand.Text()) req.Header.Set("X-Request-ID", rand.Text())
transport := &http.Transport{} transport := &http.Transport{}
+78 -42
View File
@@ -6,7 +6,6 @@ import (
"io" "io"
"log/slog" "log/slog"
"net" "net"
"net/http"
"time" "time"
"go.uber.org/fx" "go.uber.org/fx"
@@ -19,10 +18,10 @@ import (
"sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/lookup" "sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/metrics" "sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/processlog"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/remotelog" "sneak.berlin/go/smallwebwaf/internal/remotelog"
"sneak.berlin/go/smallwebwaf/internal/reputation" "sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/routewatch" "sneak.berlin/go/smallwebwaf/internal/routewatch"
"sneak.berlin/go/smallwebwaf/internal/rules" "sneak.berlin/go/smallwebwaf/internal/rules"
"sneak.berlin/go/smallwebwaf/internal/state" "sneak.berlin/go/smallwebwaf/internal/state"
@@ -43,8 +42,11 @@ const remoteLogStopTimeout = 2 * time.Second
func RunWAF(ctx context.Context, params Params) int { func RunWAF(ctx context.Context, params Params) int {
// Until the settings are read, the one message is an invalid setting's // Until the settings are read, the one message is an invalid setting's
// error, which every SWW_WAF_LOG_LEVEL lets through. // error, which every SWW_WAF_LOG_LEVEL lets through.
processLog := requestlog.NewProcessLogger(params.Stdout, processLog := processlog.New(processlog.Params{
config.InstanceName(params.Viper), slog.LevelError) Output: params.Stdout,
InstanceName: config.InstanceName(params.Viper),
Level: slog.LevelError,
})
cfg, err := config.WAFFromEnvironment(params.Viper) cfg, err := config.WAFFromEnvironment(params.Viper)
if err != nil { if err != nil {
@@ -63,11 +65,11 @@ func RunWAF(ctx context.Context, params Params) int {
stdout = io.MultiWriter(params.Stdout, remote) stdout = io.MultiWriter(params.Stdout, remote)
} }
processLog = requestlog.NewProcessLogger( processLog = processlog.New(processlog.Params{
stdout, Output: stdout,
cfg.InstanceName, InstanceName: cfg.InstanceName,
cfg.LogLevel, Level: cfg.LogLevel,
) })
if remote != nil { if remote != nil {
stopSending := startSending(ctx, remote, processLog) stopSending := startSending(ctx, remote, processLog)
@@ -94,8 +96,8 @@ func RunWAF(ctx context.Context, params Params) int {
newAlertQueue, newRuleFiles, newCoreRuleSet, newLookupFile, newAlertQueue, newRuleFiles, newCoreRuleSet, newLookupFile,
newLimiter, newLedger, newAnomalyCounters, newLimiter, newLedger, newAnomalyCounters,
newLists, newDNSBL, newAbuseIPDB, newRouteWatch, newLists, newDNSBL, newAbuseIPDB, newRouteWatch,
newMetrics, newGeoJS, newWAFServer, newMetrics, newGeoJS,
func(server *waf.Server) *http.Server { return server.Server }, fx.Annotate(newWAFServer, fx.As(new(daemonServer))),
newWAFListener, newStateFiles, newWAFListener, newStateFiles,
), ),
// fx stops hooks in the reverse of the order they were added in. The // fx stops hooks in the reverse of the order they were added in. The
@@ -118,7 +120,8 @@ func RunWAF(ctx context.Context, params Params) int {
"settings", cfg) "settings", cfg)
// The last write follows the stop of every hook, and so the end of every // The last write follows the stop of every hook, and so the end of every
// loop that changes the state or raises an alert. A hook of the state // loop that changes the state or raises an alert, and of every ask of
// GeoJS, the DNSBL zones, AbuseIPDB and routewatch. A hook of the state
// files would stop before the loops of the parts built ahead of them. // files would stop before the loops of the parts built ahead of them.
return run(ctx, app, failed, processLog, func() error { return run(ctx, app, failed, processLog, func() error {
err := files.WriteAll() err := files.WriteAll()
@@ -238,14 +241,17 @@ func newLookupFile(
// newLimiter returns the table of clients, with the rate and byte limits. // newLimiter returns the table of clients, with the rate and byte limits.
func newLimiter(cfg *config.WAF) *ratelimit.Limiter { func newLimiter(cfg *config.WAF) *ratelimit.Limiter {
return ratelimit.New(ratelimit.Limits{ return ratelimit.New(ratelimit.Params{
PerMinute: cfg.RateLimitPerMinute, Limits: ratelimit.Limits{
PerHour: cfg.RateLimitPerHour, PerMinute: cfg.RateLimitPerMinute,
PerDay: cfg.RateLimitPerDay, PerHour: cfg.RateLimitPerHour,
BytesPerMinute: cfg.BytesLimitPerMinute, PerDay: cfg.RateLimitPerDay,
BytesPerHour: cfg.BytesLimitPerHour, BytesPerMinute: cfg.BytesLimitPerMinute,
BytesPerDay: cfg.BytesLimitPerDay, BytesPerHour: cfg.BytesLimitPerHour,
}, cfg.MaxTrackedClients) BytesPerDay: cfg.BytesLimitPerDay,
},
MaxClients: cfg.MaxTrackedClients,
})
} }
// newLedger returns the ledger of the bans, with the settings for them. // newLedger returns the ledger of the bans, with the settings for them.
@@ -297,11 +303,13 @@ func newLists(
return lists return lists
} }
// newDNSBL returns what asks the DNSBL zones, none asked yet. // newDNSBL returns what asks the DNSBL zones, none asked yet. At the stop it
// ends the queries under way, as reputation.DNSBL.StopAsking says.
func newDNSBL( func newDNSBL(
cfg *config.WAF, processLog *slog.Logger, alertQueue *alerts.Queue, lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
alertQueue *alerts.Queue,
) *reputation.DNSBL { ) *reputation.DNSBL {
return reputation.NewDNSBL(reputation.DNSBLParams{ dnsbl := reputation.NewDNSBL(reputation.DNSBLParams{
Zones: cfg.DNSBLZones, Zones: cfg.DNSBLZones,
Resolver: cfg.DNSBLResolver, Resolver: cfg.DNSBLResolver,
CacheTTL: cfg.IPReputationCacheTTL, CacheTTL: cfg.IPReputationCacheTTL,
@@ -310,12 +318,17 @@ func newDNSBL(
ProcessLog: processLog, ProcessLog: processLog,
Alerts: alertQueue, Alerts: alertQueue,
}) })
lc.Append(fx.StopHook(dnsbl.StopAsking))
return dnsbl
} }
// newAbuseIPDB returns what checks clients with AbuseIPDB while // newAbuseIPDB returns what checks clients with AbuseIPDB while
// SWW_WAF_ABUSEIPDB_KEY is set, none checked yet. Until waf stops, it // SWW_WAF_ABUSEIPDB_KEY is set, none checked yet. Until waf stops, it
// fetches AbuseIPDB's blacklist as it is due, and bans the clients on it in // fetches AbuseIPDB's blacklist as it is due, and bans the clients on it in
// ledger, as waf.BanAbuseIPDBBlacklist says. // ledger, as waf.BanAbuseIPDBBlacklist says. At the stop it ends the checks
// under way, as reputation.AbuseIPDB.StopChecking says.
func newAbuseIPDB( func newAbuseIPDB(
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger, lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
alertQueue *alerts.Queue, ledger *bans.Ledger, alertQueue *alerts.Queue, ledger *bans.Ledger,
@@ -341,24 +354,30 @@ func newAbuseIPDB(
}) })
lc.Append(background(abuseIPDB.RunBlacklist)) lc.Append(background(abuseIPDB.RunBlacklist))
lc.Append(fx.StopHook(abuseIPDB.StopChecking))
return abuseIPDB return abuseIPDB
} }
// newRouteWatch returns what asks routewatch about clients, or nil while // newRouteWatch returns what asks routewatch about clients, or nil while
// SWW_WAF_ROUTEWATCH_URL is unset. // SWW_WAF_ROUTEWATCH_URL is unset. At the stop it ends the queries under
// way, as routewatch.Client.StopAsking says.
func newRouteWatch( func newRouteWatch(
cfg *config.WAF, processLog *slog.Logger, lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
) *routewatch.Client { ) *routewatch.Client {
if cfg.RouteWatchURL == "" { if cfg.RouteWatchURL == "" {
return nil return nil
} }
return routewatch.New(routewatch.Params{ routeWatch := routewatch.New(routewatch.Params{
URL: cfg.RouteWatchURL, URL: cfg.RouteWatchURL,
Now: utcNow, Now: utcNow,
ProcessLog: processLog, ProcessLog: processLog,
}) })
lc.Append(fx.StopHook(routeWatch.StopAsking))
return routeWatch
} }
// newMetrics returns the metrics, with those read from the other parts as // newMetrics returns the metrics, with those read from the other parts as
@@ -372,36 +391,49 @@ func newMetrics(
alertQueue *alerts.Queue, lookupFile *lookup.File, alertQueue *alerts.Queue, lookupFile *lookup.File,
remote *remotelog.Sender, remote *remotelog.Sender,
) *metrics.Metrics { ) *metrics.Metrics {
m := metrics.New(cfg.MetricsBusiestCountriesAndASNs, cfg.InstanceName) params := metrics.Params{
m.AddBansAndClients(ledger, limiter, utcNow) BusiestCount: cfg.MetricsBusiestCountriesAndASNs,
m.AddRules(ruleFiles) InstanceName: cfg.InstanceName,
m.AddReputation(lists, dnsbl) Now: utcNow,
Bans: ledger,
if cfg.AbuseIPDBKey != "" { Clients: limiter,
m.AddAbuseIPDB(abuseIPDB) Rules: ruleFiles,
Lists: lists,
DNSBL: dnsbl,
Alerts: alertQueue,
} }
m.AddAlerts(alertQueue) // A nil pointer in an interface field is not a nil interface, so a part
// that is off is left out rather than set to nil.
if cfg.AbuseIPDBKey != "" {
params.AbuseIPDB = abuseIPDB
}
if lookupFile != nil { if lookupFile != nil {
m.AddLookupFile(lookupFile.LastRead, lookupFile.ReadFailures) params.LookupFile = lookupFile
} }
if remote != nil { if remote != nil {
m.AddRemoteLog(remote) params.RemoteLog = remote
} }
return m return metrics.New(params)
} }
// newGeoJS returns what looks clients up at GeoJS, none looked up yet. Each // newGeoJS returns what looks clients up at GeoJS, none looked up yet. Each
// answer goes to the limiter and the ledger, as waf.AddLookup says, and each // answer goes to the limiter and the ledger, as waf.AddLookup says, and each
// failure is handled as waf.GeoJSFailed says. // failure is handled as waf.GeoJSFailed says. At the stop it ends the request
// under way, as geojs.GeoJS.StopAsking says.
func newGeoJS( func newGeoJS(
cfg *config.WAF, m *metrics.Metrics, limiter *ratelimit.Limiter, lc fx.Lifecycle,
ledger *bans.Ledger, alertQueue *alerts.Queue, processLog *slog.Logger, cfg *config.WAF,
m *metrics.Metrics,
limiter *ratelimit.Limiter,
ledger *bans.Ledger,
alertQueue *alerts.Queue,
processLog *slog.Logger,
) *geojs.GeoJS { ) *geojs.GeoJS {
return geojs.New(geojs.Params{ geoJS := geojs.New(geojs.Params{
URL: geojs.URL, URL: geojs.URL,
Answered: func(answer geojs.Answer) { Answered: func(answer geojs.Answer) {
waf.AddLookup(cfg, limiter, ledger, answer) waf.AddLookup(cfg, limiter, ledger, answer)
@@ -413,6 +445,10 @@ func newGeoJS(
}, },
WentWithoutAnswer: m.RequestsWithoutGeoJSAnswer.Inc, WentWithoutAnswer: m.RequestsWithoutGeoJSAnswer.Inc,
}) })
lc.Append(fx.StopHook(geoJS.StopAsking))
return geoJS
} }
// newWAFServer returns the server waf runs, with its parts and the time // newWAFServer returns the server waf runs, with its parts and the time
+110 -40
View File
@@ -4,6 +4,7 @@ import (
"bytes" "bytes"
"context" "context"
"crypto/rand" "crypto/rand"
"errors"
"io" "io"
"net/http" "net/http"
"os" "os"
@@ -20,7 +21,7 @@ import (
_ "time/tzdata" _ "time/tzdata"
"sneak.berlin/go/smallwebwaf/internal/sww" "sneak.berlin/go/smallwebwaf/internal/sww"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// commandForMain, while set in the environment, has this test binary run Main // commandForMain, while set in the environment, has this test binary run Main
@@ -106,7 +107,7 @@ func TestMainRunsWAF(t *testing.T) {
starting := out.lineBeforeExit(t, "starting", exited) starting := out.lineBeforeExit(t, "starting", exited)
addr, _ := starting["address"].(string) addr, _ := starting["address"].(string)
ask(t, addr, "/", http.StatusOK, waf.DispositionAllow) ask(t, addr, "/", http.StatusOK, wafprotocol.DispositionAllow)
// The files are written as sww stops. // The files are written as sww stops.
err = cmd.Process.Signal(syscall.SIGTERM) err = cmd.Process.Signal(syscall.SIGTERM)
@@ -164,51 +165,114 @@ func wantUTC(t *testing.T, where string, value any) {
func TestMainWithoutACommandItKnowsPrintsTheCommands(t *testing.T) { func TestMainWithoutACommandItKnowsPrintsTheCommands(t *testing.T) {
t.Parallel() t.Parallel()
binary, err := os.Executable() // completion is cobra's own command, which sww turns off.
if err != nil { for _, command := range []string{"", "serve", "completion"} {
t.Fatalf("find the test binary: %v", err)
}
for _, command := range []string{"", "serve"} {
t.Run("command "+command, func(t *testing.T) { t.Run("command "+command, func(t *testing.T) {
t.Parallel() t.Parallel()
ctx, cancel := context.WithTimeout(t.Context(), waitLimit) code, stdout, stderr := runMain(t, command)
defer cancel()
var stdout, stderr bytes.Buffer if code != 2 {
t.Errorf("exit status %d, want 2", code)
cmd := exec.CommandContext(ctx, binary)
cmd.Stdout, cmd.Stderr = &stdout, &stderr
cmd.Env = append(os.Environ(), commandForMain+"="+command)
err := cmd.Run()
if code := cmd.ProcessState.ExitCode(); code != 2 {
t.Errorf("exit status %d (%v), want 2", code, err)
} }
for _, name := range []string{"rproxy", wafCommand, "healthcheck"} { wantTheCommands(t, "stderr", stderr)
if !strings.Contains(stderr.String(), name) {
t.Errorf(
"stderr %q does not name %s",
stderr.String(),
name,
)
}
}
if stdout.Len() != 0 { if stdout != "" {
t.Errorf("stdout %q, want nothing", stdout.String()) t.Errorf("stdout %q, want nothing", stdout)
} }
}) })
} }
} }
func TestMainHelpPrintsTheCommands(t *testing.T) {
t.Parallel()
for _, command := range []string{"help", "--help"} {
t.Run("command "+command, func(t *testing.T) {
t.Parallel()
code, stdout, _ := runMain(t, command)
if code != 0 {
t.Errorf("exit status %d, want 0", code)
}
wantTheCommands(t, "stdout", stdout)
})
}
}
// wantTheCommands checks that text, what Main wrote to where, names each
// command.
func wantTheCommands(t *testing.T, where, text string) {
t.Helper()
for _, name := range []string{"rproxy", wafCommand, "healthcheck"} {
if !strings.Contains(text, name) {
t.Errorf("%s %q does not name %s", where, text, name)
}
}
}
func TestMainHealthCheckRefusesAnArgument(t *testing.T) {
t.Parallel()
for _, argument := range []string{"now", "--x", "-h"} {
t.Run("argument "+argument, func(t *testing.T) {
t.Parallel()
code, stdout, stderr := runMain(t, "healthcheck "+argument)
want := "sww healthcheck: unexpected argument \"" +
argument + "\"\n"
if code != 1 || stderr != want || stdout != "" {
t.Errorf("exit status %d, stdout %q and stderr %q, "+
"want 1, nothing and %q", code, stdout, stderr, want)
}
})
}
}
// runMain runs Main with command in a process of its own, and returns its
// exit status and what it wrote to stdout and stderr.
func runMain(t *testing.T, command string) (int, string, string) {
t.Helper()
binary, err := os.Executable()
if err != nil {
t.Fatalf("find the test binary: %v", err)
}
ctx, cancel := context.WithTimeout(t.Context(), waitLimit)
defer cancel()
var stdout, stderr bytes.Buffer
//nolint:gosec // this test binary
cmd := exec.CommandContext(ctx, binary)
cmd.Stdout, cmd.Stderr = &stdout, &stderr
cmd.Env = append(os.Environ(), commandForMain+"="+command)
err = cmd.Run()
var exitErr *exec.ExitError
if err != nil && !errors.As(err, &exitErr) {
t.Fatalf("run: %v", err)
}
return cmd.ProcessState.ExitCode(), stdout.String(), stderr.String()
}
// ask sends waf at addr rproxy's question about a request for path from // ask sends waf at addr rproxy's question about a request for path from
// placed, checks the answer's status and disposition, and closes it. // placed, checks the answer's status and disposition, and closes it.
func ask(t *testing.T, addr, path string, status int, disposition string) { func ask(
t *testing.T,
addr, path string,
status int,
disposition wafprotocol.Disposition,
) {
t.Helper() t.Helper()
req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, req, err := http.NewRequestWithContext(t.Context(), http.MethodGet,
@@ -217,7 +281,7 @@ func ask(t *testing.T, addr, path string, status int, disposition string) {
t.Fatalf("new request: %v", err) t.Fatalf("new request: %v", err)
} }
req.Header.Set(waf.ClientHeader, placed) req.Header.Set(wafprotocol.ClientHeader, placed)
req.Header.Set("X-Request-ID", rand.Text()) req.Header.Set("X-Request-ID", rand.Text())
transport := &http.Transport{} transport := &http.Transport{}
@@ -231,12 +295,12 @@ func ask(t *testing.T, addr, path string, status int, disposition string) {
_, _ = io.Copy(io.Discard, res.Body) _, _ = io.Copy(io.Discard, res.Body)
_ = res.Body.Close() _ = res.Body.Close()
if res.StatusCode != status || answered := res.Header.Get(wafprotocol.DispositionHeader)
res.Header.Get(waf.DispositionHeader) != disposition { if res.StatusCode != status || answered != string(disposition) {
t.Errorf( t.Errorf(
"answered %d with disposition %q, want %d with %q", "answered %d with disposition %q, want %d with %q",
res.StatusCode, res.StatusCode,
res.Header.Get(waf.DispositionHeader), answered,
status, status,
disposition, disposition,
) )
@@ -287,12 +351,18 @@ func TestWAFKeepsItsBansAcrossARestartAndNeedsNoApp(t *testing.T) {
} }
runWAF(t, env, func(addr string) { runWAF(t, env, func(addr string) {
ask(t, addr, "/", http.StatusOK, waf.DispositionAllow) ask(t, addr, "/", http.StatusOK, wafprotocol.DispositionAllow)
// The default rule file bans a probe for this. // The default rule file bans a probe for this.
ask(t, addr, "/.env", http.StatusForbidden, waf.DispositionDeny) ask(
t,
addr,
"/.env",
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
}) })
runWAF(t, env, func(addr string) { runWAF(t, env, func(addr string) {
ask(t, addr, "/", http.StatusForbidden, waf.DispositionDeny) ask(t, addr, "/", http.StatusForbidden, wafprotocol.DispositionDeny)
}) })
} }
+29 -14
View File
@@ -1,7 +1,8 @@
package waf package waf
// The heuristic abuseipdb. It reads SWW_WAF_ABUSEIPDB_KEY, // The heuristic abuseipdb. It reads SWW_WAF_ABUSEIPDB_KEY,
// SWW_WAF_ABUSEIPDB_MIN_SCORE and SWW_WAF_DNSBL_ABUSEIPDB_ACTION. // SWW_WAF_ABUSEIPDB_MIN_SCORE, SWW_WAF_DNSBL_ABUSEIPDB_ACTION and
// SWW_WAF_BAN_RESPONSE.
import ( import (
"context" "context"
@@ -16,37 +17,50 @@ import (
// abuseIPDB adds the tag abuseipdb and notes AbuseIPDB, as noteHit does, with // abuseIPDB adds the tag abuseipdb and notes AbuseIPDB, as noteHit does, with
// the score, when its score of the client is a hit, and refuses the request // the score, when its score of the client is a hit, and refuses the request
// when SWW_WAF_DNSBL_ABUSEIPDB_ACTION is deny, as dnsbl does for a zone. While // with SWW_WAF_BAN_RESPONSE when SWW_WAF_DNSBL_ABUSEIPDB_ACTION is deny, as
// SWW_WAF_ABUSEIPDB_KEY is unset it does nothing. A client without a score is // dnsbl does for a zone. While SWW_WAF_ABUSEIPDB_KEY is unset it does nothing.
// checked in the background, by the request's address, if its history counts // A client without a score is checked in the background, by the request's
// an offence, and the request does not wait for the answer. The score is then // address, if its history counts an offence, and the request does not wait for
// the answer. The score is then
// used for each address of the client. Its figures are the score and // used for each address of the client. Its figures are the score and
// SWW_WAF_ABUSEIPDB_MIN_SCORE. // SWW_WAF_ABUSEIPDB_MIN_SCORE.
type abuseIPDB struct{} type abuseIPDB struct {
abuseIPDB *reputation.AbuseIPDB
// limiter is the table of clients the rate limits count, whose histories
// say which clients are offenders.
limiter *ratelimit.Limiter
keySet bool // SWW_WAF_ABUSEIPDB_KEY is set
minScore int64 // SWW_WAF_ABUSEIPDB_MIN_SCORE
action config.ReputationAction // SWW_WAF_DNSBL_ABUSEIPDB_ACTION
banResponse int // SWW_WAF_BAN_RESPONSE
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
hitNoter reputationHitNoter
}
func (abuseIPDB) name() string { return "abuseipdb" } func (abuseIPDB) name() string { return "abuseipdb" }
func (abuseIPDB) check( func (a abuseIPDB) check(
ctx context.Context, ctx context.Context,
rq *request, rq *request,
_ time.Time, _ time.Time,
) *verdict { ) *verdict {
if rq.h.config.AbuseIPDBKey == "" { if !a.keySet {
return nil return nil
} }
client := clientGroup(rq.h.config, rq.client) client := clientGroup(a.ipv6GroupPrefix, rq.client)
held, _ := rq.h.limiter.Client(client) held, _ := a.limiter.Client(client)
offender := held.History.Offences != ratelimit.Offences{} offender := held.History.Offences != ratelimit.Offences{}
score, hit := rq.h.abuseIPDB.Hit(ctx, client, rq.client, offender) score, hit := a.abuseIPDB.Hit(ctx, client, rq.client, offender)
if !hit { if !hit {
return nil return nil
} }
rq.abuseIPDBHit = true rq.abuseIPDBHit = true
rq.addTag(config.AbuseIPDBTag) rq.addTag(config.AbuseIPDBTag)
rq.noteHit( a.hitNoter.noteHit(
rq,
bans.ReputationHit{ bans.ReputationHit{
Source: reputation.AbuseIPDBSource, Source: reputation.AbuseIPDBSource,
Score: &score, Score: &score,
@@ -54,14 +68,15 @@ func (abuseIPDB) check(
"scored by AbuseIPDB at or over SWW_WAF_ABUSEIPDB_MIN_SCORE", "scored by AbuseIPDB at or over SWW_WAF_ABUSEIPDB_MIN_SCORE",
) )
if rq.h.config.DNSBLAbuseIPDBAction != config.ReputationActionDeny { if a.action != config.ReputationActionDeny {
return nil return nil
} }
return &verdict{ return &verdict{
action: requestlog.ActionClientDenied, action: requestlog.ActionClientDenied,
status: a.banResponse,
figures: map[string]any{ figures: map[string]any{
figureScore: score, "min_score": rq.h.config.AbuseIPDBMinScore, figureScore: score, "min_score": a.minScore,
}, },
} }
} }
+3 -1
View File
@@ -26,7 +26,9 @@ func BanAbuseIPDBBlacklist(
scores := make(map[netip.Prefix]int64, len(listed)) scores := make(map[netip.Prefix]int64, len(listed))
for _, client := range listed { for _, client := range listed {
netblock := banNetblock(cfg, client.Addr) netblock := banNetblock(
cfg.BanScopeV4Prefix, cfg.IPv6GroupPrefix, client.Addr,
)
scores[netblock] = max(scores[netblock], client.Score) scores[netblock] = max(scores[netblock], client.Score)
} }
+2 -2
View File
@@ -54,7 +54,7 @@ func TestClientsOnTheAbuseIPDBBlacklistAreBannedForTheBanDays(t *testing.T) {
}) })
// Two clients in one netblock make one ban, with the higher score. // Two clients in one netblock make one ban, with the higher score.
server.BanAbuseIPDBBlacklist([]reputation.BlacklistEntry{ server.banAbuseIPDBBlacklist([]reputation.BlacklistEntry{
{Addr: netip.MustParseAddr(client), Score: 95}, {Addr: netip.MustParseAddr(client), Score: 95},
{Addr: netip.MustParseAddr(otherClient), Score: 100}, {Addr: netip.MustParseAddr(otherClient), Score: 100},
{Addr: netip.MustParseAddr("2001:db8:5::1"), Score: 90}, {Addr: netip.MustParseAddr("2001:db8:5::1"), Score: 90},
@@ -88,7 +88,7 @@ func TestAbuseIPDBBlacklistMakesNoBanInObserveMode(t *testing.T) {
map[string]string{mode: "observe"}, map[string]string{mode: "observe"},
) )
server.BanAbuseIPDBBlacklist([]reputation.BlacklistEntry{ server.banAbuseIPDBBlacklist([]reputation.BlacklistEntry{
{Addr: netip.MustParseAddr(client), Score: 100}, {Addr: netip.MustParseAddr(client), Score: 100},
}) })
+58
View File
@@ -0,0 +1,58 @@
package waf
// The heuristic active_ban. It reads SWW_WAF_MODE and SWW_WAF_BAN_RESPONSE; the
// bans are the ban ledger's.
import (
"context"
"time"
)
// activeBan refuses a client a ban covers, with SWW_WAF_BAN_RESPONSE: a ban on
// a netblock the client is in, in force at now. It notes for the log line when
// that ban ends. A request that makes the ban permanent, or in observe mode
// would have, raises the alert for it. Its figures are the ban's netblock and
// when it ends.
type activeBan struct {
banMaker banMaker
}
func (activeBan) name() string { return "active_ban" }
func (a activeBan) check(
_ context.Context,
rq *request,
now time.Time,
) *verdict {
ledger := a.banMaker.ledger
check := ledger.Check
if a.banMaker.observe {
check = ledger.Find // the ban refuses nothing, and stays as it is
}
ban, banned, madePermanent := check(rq.client, now)
var expires string
if banned {
expires = banExpires(ban)
rq.line.BanExpires = expires
}
if madePermanent {
ban.Expires = time.Time{} // the ban made permanent, which Find leaves as it is
a.banMaker.alertBan(rq, ban)
}
if !banned {
return nil
}
return &verdict{
action: banAction(ban.Permanent()),
status: a.banMaker.banResponse,
figures: map[string]any{
figureNetblock: ban.Netblock.String(), "ban_expires": expires,
},
}
}
+97 -146
View File
@@ -2,7 +2,6 @@ package waf
import ( import (
"bytes" "bytes"
"crypto/subtle"
"encoding/json" "encoding/json"
"errors" "errors"
"fmt" "fmt"
@@ -15,7 +14,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/state" "sneak.berlin/go/smallwebwaf/internal/state"
) )
@@ -41,80 +39,23 @@ var (
errNotAddress = errors.New("is not an address, such as 203.0.113.9") errNotAddress = errors.New("is not an address, such as 203.0.113.9")
) )
// answerHealth answers a request for the health endpoint. // handleHealth answers GET wafprotocol.HealthPath.
func (rq *request) answerHealth() { func handleHealth() http.HandlerFunc {
rq.line.Action = requestlog.ActionOwnEndpoint return func(w http.ResponseWriter, _ *http.Request) {
// Set here rather than left to Go's server, which would set it only // Set here rather than left to Go's server, which would set it only
// after the log line has taken the response's headers. // after the log line has taken the response's headers.
rq.out.Header().Set("Content-Type", "text/plain; charset=utf-8") w.Header().Set("Content-Type", "text/plain; charset=utf-8")
_, _ = io.WriteString(rq.out, "ok\n") _, _ = io.WriteString(w, "ok\n")
}
// answerAdmin answers a request for one of waf's own endpoints. Each
// needs a token, sent as Authorization: Bearer <token>: the metrics
// SWW_WAF_METRICS_TOKEN, the others SWW_WAF_ADMIN_TOKEN. A request without it is
// refused with 401. An endpoint whose token is unset answers 404, as any other
// path does.
func (rq *request) answerAdmin() {
rq.line.Action = requestlog.ActionOwnEndpoint
token, answer := rq.endpoint()
switch {
case token == "":
http.Error(
rq.out,
http.StatusText(http.StatusNotFound),
http.StatusNotFound,
)
case !hasToken(rq.in, token):
rq.out.Header().Set("WWW-Authenticate", "Bearer")
http.Error(
rq.out,
http.StatusText(http.StatusUnauthorized),
http.StatusUnauthorized,
)
default:
answer()
} }
} }
// endpoint returns the token the request's endpoint needs, and what // handleListBans answers GET BansPath with every ban held.
// answers the request there; "" when there is no such endpoint. func (h *handler) handleListBans() http.HandlerFunc {
func (rq *request) endpoint() (string, func()) { return func(w http.ResponseWriter, _ *http.Request) {
cfg := rq.h.config h.answerBans(w, h.ledger.Snapshot())
method, path := rq.in.Method, rq.in.URL.Path
switch {
case method == http.MethodGet && path == MetricsPath:
return cfg.MetricsToken, func() { rq.h.metrics.ServeHTTP(rq.out, rq.in) }
case method == http.MethodGet && path == BansPath:
return cfg.AdminToken, rq.listBans
case method == http.MethodPost && path == BansPath:
return cfg.AdminToken, rq.addBan
case method == http.MethodDelete && strings.HasPrefix(path, BansPath+"/"):
return cfg.AdminToken, rq.liftBans
case method == http.MethodGet && strings.HasPrefix(path, ClientsPath):
return cfg.AdminToken, rq.showClient
default:
return "", nil
} }
} }
// hasToken reports whether r carries token, as Authorization: Bearer
// <token>.
func hasToken(r *http.Request, token string) bool {
scheme, sent, _ := strings.Cut(r.Header.Get("Authorization"), " ")
return strings.EqualFold(scheme, "Bearer") &&
subtle.ConstantTimeCompare([]byte(sent), []byte(token)) == 1
}
// listBans answers GET BansPath with every ban held.
func (rq *request) listBans() {
rq.answerBans(rq.h.ledger.Snapshot())
}
// banToAdd is the body of POST BansPath. // banToAdd is the body of POST BansPath.
type banToAdd struct { type banToAdd struct {
// Netblock is a netblock, or a client's address, which stands for the // Netblock is a netblock, or a client's address, which stands for the
@@ -126,44 +67,44 @@ type banToAdd struct {
Reason string `json:"reason"` Reason string `json:"reason"`
} }
// addBan answers POST BansPath: it bans the netblock the body names, as // handleAddBan answers POST BansPath: it bans the netblock the body names,
// an admin, from now for the duration the body gives, with its reason, // as an admin, from now for the duration the body gives, with its reason,
// and answers with that ban. // and answers with that ban.
func (rq *request) addBan() { func (h *handler) handleAddBan() http.HandlerFunc {
toAdd, err := rq.readBanToAdd() return func(w http.ResponseWriter, r *http.Request) {
toAdd, err := readBanToAdd(r)
var ( var (
netblock netip.Prefix netblock netip.Prefix
expires time.Time expires time.Time
now = rq.h.now() now = h.now()
) )
if err == nil { if err == nil {
netblock, err = rq.h.parseBanNetblock(toAdd.Netblock) netblock, err = h.parseBanNetblock(toAdd.Netblock)
}
if err == nil {
expires, err = expiry(toAdd.Duration, now)
}
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
ban := h.ledger.BanForAdmin(netblock, now, expires, toAdd.Reason)
h.answerBans(w, []bans.Ban{ban})
} }
if err == nil {
expires, err = expiry(toAdd.Duration, now)
}
if err != nil {
http.Error(rq.out, err.Error(), http.StatusBadRequest)
return
}
ban := rq.h.ledger.BanForAdmin(netblock, now, expires, toAdd.Reason)
rq.answerBans([]bans.Ban{ban})
} }
// readBanToAdd reads the body of POST BansPath: a JSON object with // readBanToAdd reads the body of POST BansPath: a JSON object with
// nothing but whitespace after it, in at most banBodyMaxBytes. // nothing but whitespace after it, in at most banBodyMaxBytes.
func (rq *request) readBanToAdd() (banToAdd, error) { func readBanToAdd(r *http.Request) (banToAdd, error) {
data, err := io.ReadAll( data, err := io.ReadAll(
http.MaxBytesReader(nil, rq.in.Body, banBodyMaxBytes), http.MaxBytesReader(nil, r.Body, banBodyMaxBytes),
) )
rq.requestBytes = int64(len(data))
if err != nil { if err != nil {
return banToAdd{}, fmt.Errorf("%w: %w", errNotBanToAdd, err) return banToAdd{}, fmt.Errorf("%w: %w", errNotBanToAdd, err)
} }
@@ -224,7 +165,11 @@ func (h *handler) parseBanNetblock(value string) (netip.Prefix, error) {
return netip.Prefix{}, fmt.Errorf("netblock %q %w", value, errZone) return netip.Prefix{}, fmt.Errorf("netblock %q %w", value, errZone)
} }
return banNetblock(h.config, addr), nil return banNetblock(
h.config.BanScopeV4Prefix,
h.config.IPv6GroupPrefix,
addr,
), nil
} }
// expiry returns when a ban made at now for duration ends: duration // expiry returns when a ban made at now for duration ends: duration
@@ -246,29 +191,31 @@ func expiry(duration string, now time.Time) (time.Time, error) {
return now.Add(length), nil return now.Add(length), nil
} }
// liftBans answers DELETE BansPath/<client>: it lifts every ban active on // handleLiftBans answers DELETE BansPath/<client>: it lifts every ban
// a netblock the client's address is in, and answers with those bans, or // active on a netblock the client's address is in, and answers with those
// with 404 when none is active. // bans, or with 404 when none is active.
func (rq *request) liftBans() { func (h *handler) handleLiftBans() http.HandlerFunc {
client, err := pathAddress(rq.in.URL.Path, BansPath+"/") return func(w http.ResponseWriter, r *http.Request) {
if err != nil { client, err := pathAddress(r.URL.Path, BansPath+"/")
http.Error(rq.out, err.Error(), http.StatusBadRequest) if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return return
}
lifted := h.ledger.Lift(client, h.now())
if len(lifted) == 0 {
http.Error(
w,
"no ban is active on "+client.String(),
http.StatusNotFound,
)
return
}
h.answerBans(w, lifted)
} }
lifted := rq.h.ledger.Lift(client, rq.h.now())
if len(lifted) == 0 {
http.Error(
rq.out,
"no ban is active on "+client.String(),
http.StatusNotFound,
)
return
}
rq.answerBans(lifted)
} }
// clientAnswer is the answer to GET ClientsPath<ip>: the client the // clientAnswer is the answer to GET ClientsPath<ip>: the client the
@@ -280,27 +227,31 @@ type clientAnswer struct {
Bans []state.BanEntry `json:"bans"` Bans []state.BanEntry `json:"bans"`
} }
// showClient answers GET ClientsPath<ip> with what waf knows of the client: // handleShowClient answers GET ClientsPath<ip> with what waf knows of the
// its counters, its history, which holds its country as last looked up and its // client: its counters, its history, which holds its country as last looked
// offences, and its bans with their notes. // up and its offences, and its bans with their notes.
func (rq *request) showClient() { func (h *handler) handleShowClient() http.HandlerFunc {
addr, err := pathAddress(rq.in.URL.Path, ClientsPath) return func(w http.ResponseWriter, r *http.Request) {
if err != nil { addr, err := pathAddress(r.URL.Path, ClientsPath)
http.Error(rq.out, err.Error(), http.StatusBadRequest) if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return return
}
answer := clientAnswer{
Bans: state.BanEntries(h.ledger.Covering(addr)),
}
client, seen := h.limiter.Client(
clientGroup(h.config.IPv6GroupPrefix, addr),
)
if seen {
answer.Client = &client
}
h.answerJSON(w, answer)
} }
answer := clientAnswer{
Bans: state.BanEntries(rq.h.ledger.Covering(addr)),
}
client, seen := rq.h.limiter.Client(clientGroup(rq.h.config, addr))
if seen {
answer.Client = &client
}
rq.answerJSON(answer)
} }
// pathAddress reads the client's address that follows prefix in path. // pathAddress reads the client's address that follows prefix in path.
@@ -316,27 +267,27 @@ func pathAddress(path, prefix string) (netip.Addr, error) {
} }
// answerBans answers with held under bans, as bans.json lists them. // answerBans answers with held under bans, as bans.json lists them.
func (rq *request) answerBans(held []bans.Ban) { func (h *handler) answerBans(w http.ResponseWriter, held []bans.Ban) {
rq.answerJSON(struct { h.answerJSON(w, struct {
Bans []state.BanEntry `json:"bans"` Bans []state.BanEntry `json:"bans"`
}{state.BanEntries(held)}) }{state.BanEntries(held)})
} }
// answerJSON answers with value as indented JSON. // answerJSON answers with value as indented JSON.
func (rq *request) answerJSON(value any) { func (h *handler) answerJSON(w http.ResponseWriter, value any) {
body, err := json.MarshalIndent(value, "", " ") body, err := json.MarshalIndent(value, "", " ")
if err != nil { if err != nil {
rq.h.processLog.Error( h.processLog.Error(
"encoding an answer failed", "encoding an answer failed",
"error", "error",
err.Error(), err.Error(),
) )
http.Error(rq.out, http.StatusText(http.StatusInternalServerError), http.Error(w, http.StatusText(http.StatusInternalServerError),
http.StatusInternalServerError) http.StatusInternalServerError)
return return
} }
rq.out.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
_, _ = rq.out.Write(append(body, '\n')) _, _ = w.Write(append(body, '\n'))
} }
+36 -2
View File
@@ -14,6 +14,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/state" "sneak.berlin/go/smallwebwaf/internal/state"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
const ( const (
@@ -37,10 +38,10 @@ func TestAdminEndpointsAreOffWhileTheTokenIsUnset(t *testing.T) {
"", "",
map[string]string{metricsToken: token}, map[string]string{metricsToken: token},
) )
server.Ledger.BanForLimit( server.Ledger.Ban(
netip.MustParsePrefix(otherClient+"/32"), netip.MustParsePrefix(otherClient+"/32"),
clk.Now(), clk.Now(),
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
before := server.Ledger.Snapshot() before := server.Ledger.Snapshot()
@@ -333,6 +334,39 @@ func TestBanToAddThatCannotBeReadIsRefused(t *testing.T) {
} }
} }
func TestBanToAddIsLoggedWithTheBytesOfItsBody(t *testing.T) {
t.Parallel()
s, _, _ := startWithClock(
t,
"",
map[string]string{adminToken: adminSecret},
)
s.admin(http.MethodPost, waf.BansPath, banOtherClient, http.StatusOK)
line := s.out.endedLines(t, s.sent)[s.sent-1]
if line.RequestBytes != int64(len(banOtherClient)) {
t.Errorf(
"the ban to add was logged with request_bytes %d, want %d",
line.RequestBytes,
len(banOtherClient),
)
}
// The health endpoint reads none of a body sent to it.
s.ownEndpointRequest("", http.MethodGet, wafprotocol.HealthPath, "ok",
http.StatusOK)
line = s.out.endedLines(t, s.sent)[s.sent-1]
if line.RequestBytes != 0 {
t.Errorf(
"the health request was logged with request_bytes %d, want 0",
line.RequestBytes,
)
}
}
func TestClientEndpointShowsTheClientAndItsBans(t *testing.T) { func TestClientEndpointShowsTheClientAndItsBans(t *testing.T) {
t.Parallel() t.Parallel()
+10 -11
View File
@@ -11,7 +11,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
) )
const ( const (
@@ -130,11 +129,9 @@ func TestObserveModeRaisesTheBanAlertsItWouldHave(t *testing.T) {
// A ban for a clear sign of attack, which a request under it would make // A ban for a clear sign of attack, which a request under it would make
// permanent. // permanent.
group := netip.MustParsePrefix(ipv6Group) group := netip.MustParsePrefix(ipv6Group)
attackBan, _ := server.Ledger.BanForAttack( attackBan, _ := server.Ledger.Ban(group, start, bans.Grounds{
group, Cause: bans.CauseAttack, Notes: bans.Notes{RuleID: "probe"},
start, })
bans.Notes{RuleID: "probe"},
)
// The third request breaks the limit, and so does the fourth, within the // The third request breaks the limit, and so does the fourth, within the
// cooldown, which raises nothing. The probe is a clear sign of attack. // cooldown, which raises nothing. The probe is a clear sign of attack.
@@ -252,7 +249,7 @@ func TestObserveModeWorksOutABanOnlyWhenItsAlertWouldBeSent(t *testing.T) {
// never sent them, and returns the queue they wait in as well. // never sent them, and returns the queue they wait in as well.
func startWithAlerts( func startWithAlerts(
t *testing.T, env map[string]string, t *testing.T, env map[string]string,
) (*sender, *clock, *waf.Server, *alerts.Queue) { ) (*sender, *clock, *wafAndParts, *alerts.Queue) {
t.Helper() t.Helper()
return startAppWithAlerts( return startAppWithAlerts(
@@ -265,7 +262,7 @@ func startWithAlerts(
// startAppWithAlerts is startWithAlerts in front of the app handler. // startAppWithAlerts is startWithAlerts in front of the app handler.
func startAppWithAlerts( func startAppWithAlerts(
t *testing.T, handler http.HandlerFunc, env map[string]string, t *testing.T, handler http.HandlerFunc, env map[string]string,
) (*sender, *clock, *waf.Server, *alerts.Queue) { ) (*sender, *clock, *wafAndParts, *alerts.Queue) {
t.Helper() t.Helper()
app := startApp(t, handler) app := startApp(t, handler)
@@ -294,7 +291,7 @@ func startAppWithAlerts(
// the time raised, for ban, with its netblock, cause, reason and notes, // the time raised, for ban, with its netblock, cause, reason and notes,
// which ends at expires, as the log line gives it. // which ends at expires, as the log line gives it.
func banAlert( func banAlert(
event string, event alerts.Event,
raised time.Time, raised time.Time,
client string, client string,
ban bans.Ban, ban bans.Ban,
@@ -309,7 +306,9 @@ func banAlert(
Netblock: ban.Netblock, Netblock: ban.Netblock,
Reason: ban.Reason, Reason: ban.Reason,
Detail: map[string]any{ Detail: map[string]any{
"cause": ban.Cause, "ban_expires": expires, "notes": ban.Notes, "cause": string(
ban.Cause,
), "ban_expires": expires, "notes": ban.Notes,
}, },
} }
} }
@@ -317,7 +316,7 @@ func banAlert(
// attackAlert is banAlert for a ban for the probe rule of testRules, with // attackAlert is banAlert for a ban for the probe rule of testRules, with
// the netblock and the notes of ban. // the netblock and the notes of ban.
func attackAlert( func attackAlert(
event string, event alerts.Event,
raised time.Time, raised time.Time,
client string, client string,
ban bans.Ban, ban bans.Ban,
+6 -7
View File
@@ -15,7 +15,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/anomaly" "sneak.berlin/go/smallwebwaf/internal/anomaly"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
) )
// The anomaly thresholds: the prefix of a scope followed by the end of a // The anomaly thresholds: the prefix of a scope followed by the end of a
@@ -86,10 +85,10 @@ func TestEachScopeAndWindowOverItsThresholdAlertsOncePerCooldown(
value int64 value int64
count float64 count float64
}{ }{
{requestsPerMinute, ratelimit.KindRequests, minute, 2, 3}, {requestsPerMinute, string(ratelimit.KindRequests), minute, 2, 3},
{requestsPerHour, ratelimit.KindRequests, hour, 2, 3}, {requestsPerHour, string(ratelimit.KindRequests), hour, 2, 3},
{bytesPerMinute, ratelimit.KindBytes, minute, 250, 300}, {bytesPerMinute, string(ratelimit.KindBytes), minute, 250, 300},
{bytesPerHour, ratelimit.KindBytes, hour, 250, 300}, {bytesPerHour, string(ratelimit.KindBytes), hour, 250, 300},
} { } {
setting := scope.prefix + threshold.end setting := scope.prefix + threshold.end
value := strconv.FormatInt(threshold.value, 10) value := strconv.FormatInt(threshold.value, 10)
@@ -186,7 +185,7 @@ func TestEveryRequestIsCountedWhateverIsDoneWithIt(t *testing.T) {
for _, sent := range []struct { for _, sent := range []struct {
from, path string from, path string
status int status int
action string action requestlog.Action
}{ }{
{allowed, "/", http.StatusOK, requestlog.ActionForward}, {allowed, "/", http.StatusOK, requestlog.ActionForward},
{exempt, "/", http.StatusOK, requestlog.ActionForward}, {exempt, "/", http.StatusOK, requestlog.ActionForward},
@@ -332,7 +331,7 @@ func TestRequestCountsForTheASNumberGeoJSGivesBeforeItEnds(t *testing.T) {
// kept. // kept.
geojsURL, _, release := startHeldGeoJS(t) geojsURL, _, release := startHeldGeoJS(t)
var server atomic.Pointer[waf.Server] var server atomic.Pointer[wafAndParts]
app := startApp(t, func(http.ResponseWriter, *http.Request) { app := startApp(t, func(http.ResponseWriter, *http.Request) {
release() release()
+80 -42
View File
@@ -26,6 +26,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/reputation" "sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
const ( const (
@@ -84,7 +85,7 @@ func TestSixteenThousandCookiesAreRefusedBeforeTheCoreRuleSetReadsThem(
got := do(t, req) got := do(t, req)
wantDisposition(t, got, tooLarge, waf.DispositionDeny) wantDisposition(t, got, tooLarge, wafprotocol.DispositionDeny)
line := s.out.requestLine(t) line := s.out.requestLine(t)
wantLine(t, line, tooLarge, requestlog.ActionRequestHeaderTooLarge) wantLine(t, line, tooLarge, requestlog.ActionRequestHeaderTooLarge)
@@ -134,7 +135,7 @@ func TestARequestLineOverTheLimitIsRefusedWith414BeforeAnyOtherCheck(
// cookies is how many cookies the request carries. // cookies is how many cookies the request carries.
cookies int cookies int
status int status int
action string action requestlog.Action
}{ }{
{ {
"at the limit", lineLimit, 0, http.StatusOK, "at the limit", lineLimit, 0, http.StatusOK,
@@ -164,9 +165,14 @@ func TestARequestLineOverTheLimitIsRefusedWith414BeforeAnyOtherCheck(
line := s.out.requestLine(t) line := s.out.requestLine(t)
if tc.status == http.StatusOK { if tc.status == http.StatusOK {
wantDisposition(t, got, tc.status, waf.DispositionAllow) wantDisposition(
t,
got,
tc.status,
wafprotocol.DispositionAllow,
)
} else { } else {
wantDisposition(t, got, tc.status, waf.DispositionDeny) wantDisposition(t, got, tc.status, wafprotocol.DispositionDeny)
} }
wantLine(t, line, tc.status, tc.action) wantLine(t, line, tc.status, tc.action)
@@ -219,9 +225,14 @@ func TestAHeaderLineOverTheLimitIsRefusedWith431(t *testing.T) {
got := do(t, req) got := do(t, req)
if tc.refused { if tc.refused {
wantDisposition(t, got, tooLarge, waf.DispositionDeny) wantDisposition(
t,
got,
tooLarge,
wafprotocol.DispositionDeny,
)
} else { } else {
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow) wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
} }
}) })
} }
@@ -244,10 +255,7 @@ func TestRequestLineRefusalNamesItsHeuristicLengthAndLimit(t *testing.T) {
func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) { func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
t.Parallel() t.Parallel()
repeated := make([]string, headerFieldLimit) repeated := slices.Repeat([]string{"1"}, headerFieldLimit)
for i := range repeated {
repeated[i] = "1"
}
for _, tc := range []struct { for _, tc := range []struct {
name string name string
@@ -294,7 +302,12 @@ func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
line := s.out.requestLine(t) line := s.out.requestLine(t)
if tc.refused { if tc.refused {
wantDisposition(t, got, tooLarge, waf.DispositionDeny) wantDisposition(
t,
got,
tooLarge,
wafprotocol.DispositionDeny,
)
wantLine( wantLine(
t, t,
line, line,
@@ -305,7 +318,12 @@ func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
return return
} }
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow) wantDisposition(
t,
got,
http.StatusOK,
wafprotocol.DispositionAllow,
)
wantLine(t, line, http.StatusOK, requestlog.ActionForward) wantLine(t, line, http.StatusOK, requestlog.ActionForward)
if line.DurationCoreRuleSetMS == nil { if line.DurationCoreRuleSetMS == nil {
@@ -332,7 +350,12 @@ func TestHeaderLimitsHoldWhateverTheCoreRuleSetSettingsSay(t *testing.T) {
req := s.question(siteA, client, "/") req := s.question(siteA, client, "/")
req.Header.Set(cookieHeader, cookies(cookieLimit+1)) req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(t, do(t, req), tooLarge, waf.DispositionDeny) wantDisposition(
t,
do(t, req),
tooLarge,
wafprotocol.DispositionDeny,
)
}) })
} }
} }
@@ -348,7 +371,12 @@ func TestClientInTheAllowNetsSkipsTheHeaderLimits(t *testing.T) {
req := s.question(siteA, client, "/") req := s.question(siteA, client, "/")
req.Header.Set(cookieHeader, cookies(cookieLimit+1)) req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow) wantDisposition(
t,
do(t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
)
} }
func TestObserveModeAllowsARequestOverTheHeaderLimitsUnread(t *testing.T) { func TestObserveModeAllowsARequestOverTheHeaderLimitsUnread(t *testing.T) {
@@ -364,7 +392,7 @@ func TestObserveModeAllowsARequestOverTheHeaderLimitsUnread(t *testing.T) {
got := do(t, req) got := do(t, req)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow) wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
line := s.out.requestLine(t) line := s.out.requestLine(t)
if line.WouldAction != requestlog.ActionRequestHeaderTooLarge || if line.WouldAction != requestlog.ActionRequestHeaderTooLarge ||
@@ -386,14 +414,14 @@ func TestTheRateLimitsDoNotCountRequestsOverTheHeaderLimits(t *testing.T) {
for range 2 { for range 2 {
req := s.question(siteA, client, "/") req := s.question(siteA, client, "/")
req.Header.Set(cookieHeader, cookies(cookieLimit+1)) req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(t, do(t, req), tooLarge, waf.DispositionDeny) wantDisposition(t, do(t, req), tooLarge, wafprotocol.DispositionDeny)
} }
wantDisposition( wantDisposition(
t, t,
s.ask(client, "/"), s.ask(client, "/"),
http.StatusOK, http.StatusOK,
waf.DispositionAllow, wafprotocol.DispositionAllow,
) )
} }
@@ -411,7 +439,7 @@ func TestTheHeaderLimitsComeBeforeEveryCheckThatRefuses(t *testing.T) {
env map[string]string env map[string]string
path string path string
// prepare gets the server ready, at now, before the question. // prepare gets the server ready, at now, before the question.
prepare func(t *testing.T, server *waf.Server, now time.Time) prepare func(t *testing.T, server *wafAndParts, now time.Time)
}{ }{
{"SWW_WAF_DENY_NETS", map[string]string{denyNets: client}, "/", leaveAsItIs}, {"SWW_WAF_DENY_NETS", map[string]string{denyNets: client}, "/", leaveAsItIs},
{"a ban", nil, "/", banClient}, {"a ban", nil, "/", banClient},
@@ -426,21 +454,21 @@ func TestTheHeaderLimitsComeBeforeEveryCheckThatRefuses(t *testing.T) {
"a DNS blocklist", map[string]string{ "a DNS blocklist", map[string]string{
dnsblZones: dnsblZone, dnsblResolver: noResolver, dnsblAbuseIPDBAction: actionDeny, dnsblZones: dnsblZone, dnsblResolver: noResolver, dnsblAbuseIPDBAction: actionDeny,
}, },
"/", func(_ *testing.T, server *waf.Server, _ time.Time) { "/", func(_ *testing.T, server *wafAndParts, _ time.Time) {
loadVerdicts(server, map[string][]string{client: {dnsblZone}}) loadVerdicts(server, map[string][]string{client: {dnsblZone}})
}, },
}, },
{ {
"AbuseIPDB", "AbuseIPDB",
map[string]string{abuseIPDBKey: accountKey, dnsblAbuseIPDBAction: actionDeny}, map[string]string{abuseIPDBKey: accountKey, dnsblAbuseIPDBAction: actionDeny},
"/", func(_ *testing.T, server *waf.Server, _ time.Time) { "/", func(_ *testing.T, server *wafAndParts, _ time.Time) {
// Over SWW_WAF_ABUSEIPDB_MIN_SCORE, 75 by default. // Over SWW_WAF_ABUSEIPDB_MIN_SCORE, 75 by default.
loadScores(server, map[string]int64{client: 80}) loadScores(server, map[string]int64{client: 80})
}, },
}, },
{ {
"SWW_WAF_TAG_DENY", map[string]string{tagLists: "vpn=" + vpnList, tagDeny: "vpn"}, "SWW_WAF_TAG_DENY", map[string]string{tagLists: "vpn=" + vpnList, tagDeny: "vpn"},
"/", func(t *testing.T, server *waf.Server, _ time.Time) { "/", func(t *testing.T, server *wafAndParts, _ time.Time) {
t.Helper() t.Helper()
loadLists(t, server, map[string][]string{vpnList: {client}}) loadLists(t, server, map[string][]string{vpnList: {client}})
}, },
@@ -466,7 +494,12 @@ func TestTheHeaderLimitsComeBeforeEveryCheckThatRefuses(t *testing.T) {
req := s.question(siteA, client, tc.path) req := s.question(siteA, client, tc.path)
req.Header.Set(cookieHeader, cookies(cookieLimit+1)) req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(t, do(t, req), tooLarge, waf.DispositionDeny) wantDisposition(
t,
do(t, req),
tooLarge,
wafprotocol.DispositionDeny,
)
wantLine( wantLine(
t, t,
s.out.requestLine(t), s.out.requestLine(t),
@@ -503,11 +536,16 @@ func TestARequestOverTheHeaderLimitsRunsNoOtherCheck(t *testing.T) {
over := s.question(siteA, client, "/") over := s.question(siteA, client, "/")
over.Header.Set(cookieHeader, cookies(cookieLimit+1)) over.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(t, do(t, over), tooLarge, waf.DispositionDeny) wantDisposition(t, do(t, over), tooLarge, wafprotocol.DispositionDeny)
within := s.question(siteA, client, "/") within := s.question(siteA, client, "/")
within.Header.Set(cookieHeader, cookies(cookieLimit)) within.Header.Set(cookieHeader, cookies(cookieLimit))
wantDisposition(t, do(t, within), http.StatusOK, waf.DispositionAllow) wantDisposition(
t,
do(t, within),
http.StatusOK,
wafprotocol.DispositionAllow,
)
lines := s.out.requestLines(t, 2) lines := s.out.requestLines(t, 2)
@@ -564,7 +602,7 @@ func TestNotWaitingForGeoJSNeverLowersAClientsLimit(t *testing.T) {
t, t,
s.ask(fromDE, "/"), s.ask(fromDE, "/"),
http.StatusForbidden, http.StatusForbidden,
waf.DispositionDeny, wafprotocol.DispositionDeny,
) )
line := s.out.requestLine(t) line := s.out.requestLine(t)
@@ -601,7 +639,7 @@ func TestAClientGeoJSPlacedInNoCountryCountsAsUnknownWithoutWaiting(
t, t,
s.ask(client, "/"), s.ask(client, "/"),
http.StatusForbidden, http.StatusForbidden,
waf.DispositionDeny, wafprotocol.DispositionDeny,
) )
line := s.out.requestLine(t) line := s.out.requestLine(t)
@@ -630,7 +668,7 @@ func TestObserveModeAsksGeoJSNothingForARequestOverTheHeaderLimits(
req.Header.Set(cookieHeader, cookies(cookieLimit+1)) req.Header.Set(cookieHeader, cookies(cookieLimit+1))
got := do(t, req) got := do(t, req)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow) wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
if len(asked()) != 0 || got.header.Get("X-Client-Asn") != "" { if len(asked()) != 0 || got.header.Get("X-Client-Asn") != "" {
t.Errorf( t.Errorf(
@@ -647,7 +685,7 @@ func TestObserveModeAsksGeoJSNothingForARequestOverTheHeaderLimits(
// for env. No request to it may use the network. // for env. No request to it may use the network.
func newWAFInBubble( func newWAFInBubble(
t *testing.T, env map[string]string, t *testing.T, env map[string]string,
) (*waf.Server, *output) { ) (*wafAndParts, *output) {
t.Helper() t.Helper()
settings := map[string]string{tagDeny: "country=KP"} settings := map[string]string{tagDeny: "country=KP"}
@@ -661,7 +699,7 @@ func newWAFInBubble(
// askDirectly hands a question about a request from client straight to server, // askDirectly hands a question about a request from client straight to server,
// as change leaves it, and returns the answer's status and how long it took. // as change leaves it, and returns the answer's status and how long it took.
func askDirectly( func askDirectly(
t *testing.T, server *waf.Server, change func(*http.Request), t *testing.T, server *wafAndParts, change func(*http.Request),
) (int, time.Duration) { ) (int, time.Duration) {
t.Helper() t.Helper()
@@ -672,29 +710,29 @@ func askDirectly(
http.NoBody, http.NoBody,
) )
req.RemoteAddr = net.JoinHostPort(rproxyPeer, "1234") req.RemoteAddr = net.JoinHostPort(rproxyPeer, "1234")
req.Header.Set(waf.ClientHeader, client) req.Header.Set(wafprotocol.ClientHeader, client)
req.Header.Set(requestIDHeader, "the-id") req.Header.Set(requestIDHeader, "the-id")
change(req) change(req)
recorder := httptest.NewRecorder() recorder := httptest.NewRecorder()
began := time.Now() began := time.Now()
server.Handler.ServeHTTP(recorder, req) server.ServeHTTP(recorder, req)
return recorder.Code, time.Since(began) return recorder.Code, time.Since(began)
} }
// banClient bans client at now. // banClient bans client at now.
func banClient(_ *testing.T, server *waf.Server, now time.Time) { func banClient(_ *testing.T, server *wafAndParts, now time.Time) {
server.Ledger.BanForLimit( server.Ledger.Ban(
netip.MustParsePrefix(client+"/32"), netip.MustParsePrefix(client+"/32"),
now, now,
bans.Notes{}, bans.Grounds{Cause: bans.CauseLimit},
) )
} }
// countClient counts one request from client at now for the rate limits. // countClient counts one request from client at now for the rate limits.
func countClient(_ *testing.T, server *waf.Server, now time.Time) { func countClient(_ *testing.T, server *wafAndParts, now time.Time) {
server.Limiter.Count( server.Limiter.Count(
netip.MustParsePrefix(client+"/32"), netip.MustParsePrefix(client+"/32"),
now, now,
@@ -703,7 +741,7 @@ func countClient(_ *testing.T, server *waf.Server, now time.Time) {
} }
// listClient puts client on the blocklist at dropURL. // listClient puts client on the blocklist at dropURL.
func listClient(t *testing.T, server *waf.Server, now time.Time) { func listClient(t *testing.T, server *wafAndParts, now time.Time) {
t.Helper() t.Helper()
err := server.Lists.Load([]reputation.List{{ err := server.Lists.Load([]reputation.List{{
@@ -715,7 +753,7 @@ func listClient(t *testing.T, server *waf.Server, now time.Time) {
} }
// leaveAsItIs prepares nothing. // leaveAsItIs prepares nothing.
func leaveAsItIs(*testing.T, *waf.Server, time.Time) {} func leaveAsItIs(*testing.T, *wafAndParts, time.Time) {}
func TestARefusedRequestDoesNotWaitForASlowGeoJS(t *testing.T) { func TestARefusedRequestDoesNotWaitForASlowGeoJS(t *testing.T) {
t.Parallel() t.Parallel()
@@ -726,11 +764,11 @@ func TestARefusedRequestDoesNotWaitForASlowGeoJS(t *testing.T) {
name string name string
env map[string]string env map[string]string
// prepare gets the server ready, at now, before the question. // prepare gets the server ready, at now, before the question.
prepare func(t *testing.T, server *waf.Server, now time.Time) prepare func(t *testing.T, server *wafAndParts, now time.Time)
// cookie is the request's Cookie header. // cookie is the request's Cookie header.
cookie string cookie string
status int status int
action string action requestlog.Action
}{ }{
{ {
name: "SWW_WAF_DENY_NETS", env: map[string]string{denyNets: client}, name: "SWW_WAF_DENY_NETS", env: map[string]string{denyNets: client},
@@ -811,9 +849,9 @@ func TestAQuestionWaitsForASlowGeoJSOnlyAsLongAsTheDefaultSays(
name string name string
env map[string]string env map[string]string
// prepare gets the server ready, at now, before the question. // prepare gets the server ready, at now, before the question.
prepare func(t *testing.T, server *waf.Server, now time.Time) prepare func(t *testing.T, server *wafAndParts, now time.Time)
status int status int
action string action requestlog.Action
}{ }{
{"nothing refuses it", nil, leaveAsItIs, http.StatusOK, requestlog.ActionForward}, {"nothing refuses it", nil, leaveAsItIs, http.StatusOK, requestlog.ActionForward},
{ {
@@ -877,7 +915,7 @@ func TestWAFAnswersWithinTheTimeLimitWhileEveryOutsideSourceHangs(
held, _, _ := startHeldGeoJS(t) held, _, _ := startHeldGeoJS(t)
resolver, releaseResolver := startHeldResolver(t) resolver, releaseResolver := startHeldResolver(t)
server, out, _ := newWAFWith(t, waf.New, server, out, _ := newWAFWith(t, func(*waf.Params) {},
silentGeoJSURL, silentAbuseIPDBURL, time.Now, map[string]string{ silentGeoJSURL, silentAbuseIPDBURL, time.Now, map[string]string{
tagDeny: "country=KP", coreRuleSetMode: block, tagDeny: "country=KP", coreRuleSetMode: block,
dnsblZones: dnsblZone, dnsblResolver: resolver, dnsblZones: dnsblZone, dnsblResolver: resolver,
+159 -142
View File
@@ -1,69 +1,23 @@
package waf package waf
// The heuristic active_ban, and the code that makes bans, which several // banMaker, which makes the bans of several heuristics. The bans are the ban
// heuristics share. active_ban reads SWW_WAF_MODE; the bans are the ban ledger's. // ledger's.
import ( import (
"context"
"net/netip" "net/netip"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
// banResponse is a refusal answered with SWW_WAF_BAN_RESPONSE, and logged
// with action.
func (rq *request) banResponse(action string) *refusal {
return &refusal{status: rq.h.config.BanResponse, action: action}
}
// activeBan refuses a client a ban covers: a ban on a netblock the client is
// in, in force at now. It notes for the log line when that ban ends. A request
// that makes the ban permanent, or in observe mode would have, raises the alert
// for it. Its figures are the ban's netblock and when it ends.
type activeBan struct{}
func (activeBan) name() string { return "active_ban" }
func (activeBan) check(
_ context.Context,
rq *request,
now time.Time,
) *verdict {
check := rq.h.ledger.Check
if rq.h.config.Observe {
check = rq.h.ledger.Find // the ban refuses nothing, and stays as it is
}
ban, banned, madePermanent := check(rq.client, now)
if banned {
rq.line.BanExpires = banExpires(ban)
}
if madePermanent {
ban.Expires = time.Time{} // the ban made permanent, which Find leaves as it is
rq.alertBan(ban)
}
if !banned {
return nil
}
return &verdict{
action: banAction(ban.Permanent()),
figures: map[string]any{
figureNetblock: ban.Netblock.String(), "ban_expires": rq.line.BanExpires,
},
}
}
// banAction is the action of a request refused under a ban or for making one: // banAction is the action of a request refused under a ban or for making one:
// permanent says the ban never runs out. // permanent says the ban never runs out.
func banAction(permanent bool) string { func banAction(permanent bool) requestlog.Action {
if permanent { if permanent {
return requestlog.ActionPermanentlyBanned return requestlog.ActionPermanentlyBanned
} }
@@ -81,11 +35,12 @@ func limitFigures(hit ratelimit.Hit) map[string]any {
// countedBytes returns the request's bytes, once it has ended, as the // countedBytes returns the request's bytes, once it has ended, as the
// byte limits and the anomaly thresholds count them: the response's body // byte limits and the anomaly thresholds count them: the response's body
// bytes, the request's, or both, as SWW_WAF_BYTES_COUNTED says. // bytes, the request's, or both, as bytesCounted, SWW_WAF_BYTES_COUNTED,
func (rq *request) countedBytes() int64 { // says.
func (rq *request) countedBytes(bytesCounted config.BytesCounted) int64 {
response, request := rq.out.bytes, rq.requestBytes response, request := rq.out.bytes, rq.requestBytes
switch rq.h.config.BytesCounted { switch bytesCounted {
case config.BytesCountedResponse: case config.BytesCountedResponse:
return response return response
case config.BytesCountedRequest: case config.BytesCountedRequest:
@@ -96,17 +51,40 @@ func (rq *request) countedBytes() int64 {
return response + request return response + request
} }
// banForLimit bans the client's netblock at now for a broken limit, the // banMaker makes the bans of the heuristics that ban a client, in the ban
// ledger, and raises the alert for each. In observe mode it makes no ban, and
// raises the alert for the ban it would have made.
type banMaker struct {
ledger *bans.Ledger
// limiter is the table of clients the rate limits count. A ban for a limit
// sets the client's counters back to zero, and a ban's notes give how many
// requests its netblock has sent.
limiter *ratelimit.Limiter
metrics *metrics.Metrics
alerts *alerts.Queue
// observe is true in observe mode (SWW_WAF_MODE).
observe bool
// banResponse is SWW_WAF_BAN_RESPONSE: the status of a request refused
// under a ban or for a clear sign of attack, which the notes of a ban for
// a clear sign of attack give.
banResponse int
// banScopeV4Prefix is SWW_WAF_BAN_SCOPE_V4_PREFIX and ipv6GroupPrefix
// SWW_WAF_IPV6_GROUP_PREFIX, which set the netblock a ban covers.
banScopeV4Prefix int
ipv6GroupPrefix int
}
// banForLimit bans the netblock of rq's client at now for a broken limit, the
// one hit names, notes the offence for the log line and counts the hit in // one hit names, notes the offence for the log line and counts the hit in
// the metrics. status is what the client was sent, or is sent: // the metrics. status is what the client was sent, or is sent:
// SWW_WAF_BAN_RESPONSE for a request over a rate limit, what the report says for // SWW_WAF_BAN_RESPONSE for a request over a rate limit, what the report says for
// one whose bytes broke a byte limit, the refusal for one that broke the // one whose bytes broke a byte limit, the refusal for one that broke the
// refusal limit. The ban's notes give the client's limit percentage for a // refusal limit. The ban's notes give the client's limit percentage for a
// rate limit or a byte limit; the refusal limit is not lowered. The ban sets // rate limit or a byte limit; the refusal limit is not lowered. The ban sets
// the client's counters back to zero. In observe mode it makes no ban and // the client's counters back to zero. In observe mode, banClient says what
// sets nothing back, and raises the alert for the ban it would have made, // happens instead.
// if that alert would be sent. func (b banMaker) banForLimit(
func (rq *request) banForLimit( rq *request,
now time.Time, now time.Time,
hit ratelimit.Hit, hit ratelimit.Hit,
status int, status int,
@@ -116,30 +94,20 @@ func (rq *request) banForLimit(
rq.line.LimitHit = hit.Window + "_bytes" // as counts names the byte totals rq.line.LimitHit = hit.Window + "_bytes" // as counts names the byte totals
case ratelimit.KindRefusals: case ratelimit.KindRefusals:
rq.line.LimitHit = requestlog.LimitHitMinuteRefusals rq.line.LimitHit = requestlog.LimitHitMinuteRefusals
default: case ratelimit.KindRequests:
rq.line.LimitHit = hit.Window rq.line.LimitHit = hit.Window
} }
rq.brokeLimit = true
rq.line.Offence = requestlog.OffenceLimit rq.line.Offence = requestlog.OffenceLimit
rq.h.metrics.LimitHit(hit)
netblock := banNetblock(rq.h.config, rq.client) b.metrics.LimitHit(hit)
if rq.h.config.Observe &&
!rq.wouldAlertBan(netblock, now, bans.CauseLimit) {
return
}
notes := bans.Notes{ notes := bans.Notes{
ASN: rq.line.ASN, Kind: hit.Kind,
ASName: rq.line.ASName, Limit: hit.Limit,
Country: rq.line.Country, Window: hit.Window,
Kind: hit.Kind, Count: hit.Count,
Limit: hit.Limit,
Window: hit.Window,
Count: hit.Count,
Reputation: rq.reputation,
Request: rq.noted(now, status),
Requests: rq.netblockRequests(netblock),
} }
switch hit.Kind { switch hit.Kind {
@@ -147,101 +115,148 @@ func (rq *request) banForLimit(
notes.LimitPercent, notes.LimitPercentSetting = rq.rateLimitPercent.logged() notes.LimitPercent, notes.LimitPercentSetting = rq.rateLimitPercent.logged()
case ratelimit.KindBytes: case ratelimit.KindBytes:
notes.LimitPercent, notes.LimitPercentSetting = rq.bytesLimitPercent.logged() notes.LimitPercent, notes.LimitPercentSetting = rq.bytesLimitPercent.logged()
case ratelimit.KindRefusals: // the refusal limit is not lowered
} }
if rq.h.config.Observe { b.banClient(
ban, wouldBan := rq.h.ledger.WouldBanForLimit(netblock, now, notes) rq,
if wouldBan { now,
rq.alertBan(ban) status,
} bans.Grounds{Cause: bans.CauseLimit, Notes: notes},
)
return
}
ban, made := rq.h.ledger.BanForLimit(netblock, now, notes)
rq.h.limiter.Reset(clientGroup(rq.h.config, rq.client))
rq.line.BanExpires = banExpires(ban)
if made {
rq.alertBan(ban)
}
} }
// banForAttack bans the client's netblock at now for a clear sign of // banForAttack bans the netblock of rq's client at now for a clear sign of
// attack, which notes name: the ban rule that matched, or the trap path // attack, which notes name: the ban rule that matched, or the trap path
// asked for. It fills in the rest of the notes. In observe mode it makes // asked for. It returns the action for that ban. In observe mode, banClient
// no ban, and raises the alert for the ban it would have made, if that // says what happens instead.
// alert would be sent. It returns the action for that ban. func (b banMaker) banForAttack(
func (rq *request) banForAttack(now time.Time, notes bans.Notes) string { rq *request, now time.Time, notes bans.Notes,
netblock := banNetblock(rq.h.config, rq.client) ) requestlog.Action {
if rq.h.config.Observe && return b.banClient(
!rq.wouldAlertBan(netblock, now, bans.CauseAttack) { rq,
return banAction( now,
rq.h.ledger.WouldBePermanent(netblock, now, bans.CauseAttack), b.banResponse,
bans.Grounds{Cause: bans.CauseAttack, Notes: notes},
)
}
// banClient bans the netblock of rq's client at now on grounds, and returns
// the action for that ban. It adds to the notes what the request tells of the
// client, with status, what the client was sent. A ban for a broken limit
// sets the client's counters back to zero.
//
// In observe mode it makes no ban and sets nothing back. It raises the
// alert for the ban it would have made, if that alert would be sent.
func (b banMaker) banClient(
rq *request,
now time.Time,
status int,
grounds bans.Grounds,
) requestlog.Action {
netblock := banNetblock(b.banScopeV4Prefix, b.ipv6GroupPrefix, rq.client)
if b.observe {
if !b.wouldAlertBan(netblock, now, grounds.Cause) {
return banAction(
b.ledger.WouldBePermanent(netblock, now, grounds.Cause),
)
}
grounds.Notes = b.withClientNotes(
rq,
grounds.Notes,
now,
status,
netblock,
) )
}
notes.ASN = rq.line.ASN ban, wouldBan := b.ledger.WouldBan(netblock, now, grounds)
notes.ASName = rq.line.ASName
notes.Country = rq.line.Country
notes.Reputation = rq.reputation
notes.Request = rq.noted(now, rq.h.config.BanResponse)
notes.Requests = rq.netblockRequests(netblock)
if rq.h.config.Observe {
ban, wouldBan := rq.h.ledger.WouldBanForAttack(netblock, now, notes)
if wouldBan { if wouldBan {
rq.alertBan(ban) b.alertBan(rq, ban)
} }
return banAction(ban.Permanent()) return banAction(ban.Permanent())
} }
ban, made := rq.h.ledger.BanForAttack(netblock, now, notes) grounds.Notes = b.withClientNotes(
rq,
grounds.Notes,
now,
status,
netblock,
)
ban, made := b.ledger.Ban(netblock, now, grounds)
if grounds.Cause == bans.CauseLimit {
b.limiter.Reset(clientGroup(b.ipv6GroupPrefix, rq.client))
}
rq.line.BanExpires = banExpires(ban) rq.line.BanExpires = banExpires(ban)
if made { if made {
rq.alertBan(ban) b.alertBan(rq, ban)
} }
return banAction(ban.Permanent()) return banAction(ban.Permanent())
} }
// withClientNotes is notes with what rq tells of its client added: its AS,
// country and reputation, the request at now with status, and how many
// requests netblock, the client's, has sent.
func (b banMaker) withClientNotes(
rq *request,
notes bans.Notes,
now time.Time,
status int,
netblock netip.Prefix,
) bans.Notes {
notes.ASN = rq.lookupAnswer.ASN
notes.ASName = rq.lookupAnswer.ASName
notes.Country = rq.lookupAnswer.Country
notes.Reputation = rq.reputation
notes.Request = rq.noted(now, status)
notes.Requests = b.netblockRequests(netblock)
return notes
}
// wouldAlertBan reports whether the alert for a ban on netblock for cause // wouldAlertBan reports whether the alert for a ban on netblock for cause
// made at now would be sent. In observe mode the ban the request would // made at now would be sent. In observe mode the ban the request would
// have made is worked out only then, at most once per // have made is worked out only then, at most once per
// SWW_WAF_ALERT_COOLDOWN and never with no webhook set: its notes count the // SWW_WAF_ALERT_COOLDOWN and never with no webhook set: its notes count the
// netblock's requests, which can mean going through every client. // netblock's requests, which can mean going through every client.
func (rq *request) wouldAlertBan( func (b banMaker) wouldAlertBan(
netblock netip.Prefix, now time.Time, cause string, netblock netip.Prefix, now time.Time, cause bans.Cause,
) bool { ) bool {
event := alerts.EventBan event := alerts.EventBan
if rq.h.ledger.WouldBePermanent(netblock, now, cause) { if b.ledger.WouldBePermanent(netblock, now, cause) {
event = alerts.EventPermanentBan event = alerts.EventPermanentBan
} }
return rq.h.alerts.WouldSend(event, netblock) return b.alerts.WouldSend(event, netblock)
} }
// alertBan raises the alert for ban, which the request made, or made // alertBan raises the alert for ban, which rq made, or made permanent:
// permanent: permanent_ban for a permanent ban, ban for another. Its // permanent_ban for a permanent ban, ban for another. Its detail gives the
// detail gives the ban's cause, when it ends, and its notes, and in // ban's cause, when it ends, and its notes, and in observe mode, where ban is
// observe mode, where ban is the ban that would have been made, or made // the ban that would have been made, or made permanent, mode, observe.
// permanent, mode, observe. func (b banMaker) alertBan(rq *request, ban bans.Ban) {
func (rq *request) alertBan(ban bans.Ban) {
event := alerts.EventBan event := alerts.EventBan
if ban.Permanent() { if ban.Permanent() {
event = alerts.EventPermanentBan event = alerts.EventPermanentBan
} }
detail := map[string]any{ detail := map[string]any{
"cause": ban.Cause, "ban_expires": banExpires(ban), "notes": ban.Notes, "cause": string(ban.Cause),
"ban_expires": banExpires(ban),
"notes": ban.Notes,
} }
if rq.h.config.Observe { if b.observe {
detail["mode"] = "observe" detail["mode"] = "observe"
} }
rq.h.alerts.Raise(alerts.Alert{ b.alerts.Raise(alerts.Alert{
Event: event, Event: event,
Host: rq.site, Host: rq.site,
Client: rq.client, Client: rq.client,
@@ -254,37 +269,39 @@ func (rq *request) alertBan(ban bans.Ban) {
}) })
} }
// netblockRequests is how many requests netblock has sent since it was
// first seen, the request being banned for included: the histories count it
// only once it has ended.
func (b banMaker) netblockRequests(netblock netip.Prefix) int64 {
return b.limiter.Requests(netblock) + 1
}
// noted is the request, at now, with status, what the client was sent, or // noted is the request, at now, with status, what the client was sent, or
// in observe mode would have been, as the notes of the ban it makes keep // in observe mode would have been, as the notes of the ban it makes keep
// it. // it.
func (rq *request) noted(now time.Time, status int) bans.Request { func (rq *request) noted(now time.Time, status int) bans.Request {
return bans.Request{ return bans.Request{
Time: now, Time: now,
Method: rq.line.Method, Method: rq.method,
Host: rq.line.Host, Host: rq.site,
Path: rq.requestURI, Path: rq.requestURI,
Status: status, Status: status,
UserAgent: rq.line.UserAgent, UserAgent: rq.userAgent,
} }
} }
// netblockRequests is how many requests netblock has sent since it was
// first seen, this one included: the histories count it only once it has
// ended.
func (rq *request) netblockRequests(netblock netip.Prefix) int64 {
return rq.h.limiter.Requests(netblock) + 1
}
// banNetblock is the netblock a ban on client covers: its IPv4 address, // banNetblock is the netblock a ban on client covers: its IPv4 address,
// widened to SWW_WAF_BAN_SCOPE_V4_PREFIX, or the IPv6 group clientGroup // widened to banScopeV4Prefix, SWW_WAF_BAN_SCOPE_V4_PREFIX, or the IPv6 group
// counts it in. // clientGroup counts it in, of length ipv6GroupPrefix.
func banNetblock(cfg *config.WAF, client netip.Addr) netip.Prefix { func banNetblock(
banScopeV4Prefix, ipv6GroupPrefix int, client netip.Addr,
) netip.Prefix {
addr := client.Unmap() addr := client.Unmap()
if addr.Is4() { if addr.Is4() {
return netip.PrefixFrom(addr, cfg.BanScopeV4Prefix).Masked() return netip.PrefixFrom(addr, banScopeV4Prefix).Masked()
} }
return clientGroup(cfg, addr) return clientGroup(ipv6GroupPrefix, addr)
} }
// banExpires is when ban ends, as the log line gives it: a time, or // banExpires is when ban ends, as the log line gives it: a time, or
+13 -11
View File
@@ -16,7 +16,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
) )
const ( const (
@@ -466,7 +465,7 @@ func (c *clock) advance(d time.Duration) {
// window. // window.
func startWithClock( func startWithClock(
t *testing.T, geojsURL string, env map[string]string, t *testing.T, geojsURL string, env map[string]string,
) (*sender, *clock, *waf.Server) { ) (*sender, *clock, *wafAndParts) {
t.Helper() t.Helper()
app := startApp(t, func(http.ResponseWriter, *http.Request) {}) app := startApp(t, func(http.ResponseWriter, *http.Request) {})
@@ -504,7 +503,9 @@ type sender struct {
} }
// get sends a GET request for / from the client at from. // get sends a GET request for / from the client at from.
func (s *sender) get(from string, status int, action string) logLine { func (s *sender) get(
from string, status int, action requestlog.Action,
) logLine {
s.t.Helper() s.t.Helper()
return s.request(from, "/", status, action) return s.request(from, "/", status, action)
@@ -517,7 +518,7 @@ func (s *sender) get(from string, status int, action string) logLine {
func (s *sender) request( func (s *sender) request(
from, path string, from, path string,
status int, status int,
action string, action requestlog.Action,
) logLine { ) logLine {
s.t.Helper() s.t.Helper()
@@ -527,7 +528,7 @@ func (s *sender) request(
// requestWithHeader is request with header, such as "Authorization: // requestWithHeader is request with header, such as "Authorization:
// Bearer x", added to the request unless it is "". // Bearer x", added to the request unless it is "".
func (s *sender) requestWithHeader( func (s *sender) requestWithHeader(
from, path, header string, status int, action string, from, path, header string, status int, action requestlog.Action,
) logLine { ) logLine {
s.t.Helper() s.t.Helper()
@@ -548,8 +549,14 @@ func (s *sender) requestWithHeader(
// body is sent as it is after the headers, header holding its // body is sent as it is after the headers, header holding its
// Content-Length or Transfer-Encoding. header may hold several lines, // Content-Length or Transfer-Encoding. header may hold several lines,
// separated by "\r\n". It returns the whole answer. // separated by "\r\n". It returns the whole answer.
//
// It waits for the answer as long as it takes, since a busy host can slow
// the Core Rule Set past any shorter wait. go test's -timeout ends a test
// whose answer never comes.
func (s *sender) requestWithBody( func (s *sender) requestWithBody(
method, from, path, header, body string, status int, action string, method, from, path, header, body string,
status int,
action requestlog.Action,
) (logLine, answer) { ) (logLine, answer) {
s.t.Helper() s.t.Helper()
@@ -566,11 +573,6 @@ func (s *sender) requestWithBody(
send(s.t, conn, method+" "+path+" HTTP/1.1\r\nHost: "+appHost+"\r\n"+ send(s.t, conn, method+" "+path+" HTTP/1.1\r\nHost: "+appHost+"\r\n"+
forwardedFor+": "+from+"\r\n"+header+"\r\n"+body) forwardedFor+": "+from+"\r\n"+header+"\r\n"+body)
err := conn.SetReadDeadline(time.Now().Add(waitLimit))
if err != nil {
s.t.Fatalf("set read deadline: %v", err)
}
var got answer var got answer
res, err := http.ReadResponse(bufio.NewReader(conn), nil) res, err := http.ReadResponse(bufio.NewReader(conn), nil)
+14 -12
View File
@@ -4,6 +4,7 @@ package waf
import ( import (
"context" "context"
"net/http"
"sync/atomic" "sync/atomic"
"time" "time"
@@ -54,7 +55,7 @@ func (rq *request) runBesideLookup(
) *verdict { ) *verdict {
var running atomic.Pointer[string] var running atomic.Pointer[string]
running.Store(new(trapPaths{}.name())) running.Store(new(rq.h.trapPaths.name()))
found := make(chan *besideLookupFindings, 1) found := make(chan *besideLookupFindings, 1)
@@ -92,7 +93,7 @@ func (rq *request) runBesideLookup(
return v return v
} }
return rq.runHeuristics(ctx, afterBesideLookup(), rq.h.now()) return rq.runHeuristics(ctx, rq.h.afterBesideLookup, rq.h.now())
} }
// findBesideLookup does the work of the heuristics of besideLookup on the // findBesideLookup does the work of the heuristics of besideLookup on the
@@ -105,39 +106,39 @@ func (rq *request) findBesideLookup(
) *besideLookupFindings { ) *besideLookupFindings {
findings := &besideLookupFindings{} findings := &besideLookupFindings{}
findings.trapPath = rq.trapPath() findings.trapPath = rq.h.trapPaths.trapPath(rq)
if findings.trapPath != "" { if findings.trapPath != "" {
findings.refuses = true findings.refuses = true
return findings return findings
} }
running.Store(new(ruleFiles{}.name())) running.Store(new(rq.h.ruleFiles.name()))
findings.rules = rq.h.rules.Match(rq.in) findings.rules = rq.h.ruleFiles.rules.Match(rq.in)
if refusedByRule(findings.rules) { if refusedByRule(findings.rules) {
findings.refuses = true findings.refuses = true
return findings return findings
} }
running.Store(new(coreRuleSet{}.name())) coreRuleSet := rq.h.coreRuleSet
running.Store(new(coreRuleSet.name()))
cfg := rq.h.config if coreRuleSet.mode == config.CoreRuleSetModeOff ||
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff || pathExempt(rq.in, coreRuleSet.exemptPaths) {
pathExempt(rq.in, cfg.CoreRuleSetExemptPaths) {
return findings return findings
} }
start := time.Now() start := time.Now()
// A question's body has been read into memory, so that reading it here // A question's body has been read into memory, so that reading it here
// cannot fail. // cannot fail.
findings.inspection, findings.read, _ = rq.h.coreRuleSet.Inspect( findings.inspection, findings.read, _ = coreRuleSet.inspector.Inspect(
rq.in, rq.client, rq.in.Body, rq.in, rq.client, rq.in.Body,
) )
findings.inspectionTime = time.Since(start) findings.inspectionTime = time.Since(start)
findings.inspected = true findings.inspected = true
findings.refuses = rq.refusedByCoreRuleSet(findings.inspection) findings.refuses = coreRuleSet.refusedByCoreRuleSet(findings.inspection)
return findings return findings
} }
@@ -151,7 +152,7 @@ func (rq *request) applyFindings(
) *verdict { ) *verdict {
rq.findings = findings rq.findings = findings
return rq.runHeuristics(ctx, besideLookup(), rq.h.now()) return rq.runHeuristics(ctx, rq.h.besideLookup, rq.h.now())
} }
// timedOut refuses the request for the heuristics of besideLookup not having // timedOut refuses the request for the heuristics of besideLookup not having
@@ -161,6 +162,7 @@ func (rq *request) applyFindings(
func (rq *request) timedOut(running string) *verdict { func (rq *request) timedOut(running string) *verdict {
v := &verdict{ v := &verdict{
action: requestlog.ActionChecksTimedOut, action: requestlog.ActionChecksTimedOut,
status: http.StatusServiceUnavailable,
figures: map[string]any{ figures: map[string]any{
"time_limit_ms": rq.h.besideLookupLimit.Milliseconds(), "time_limit_ms": rq.h.besideLookupLimit.Milliseconds(),
}, },
+4 -6
View File
@@ -62,7 +62,7 @@ func coreRuleSetMatch() coreruleset.Result {
// at its default but for env. No request to it may use the network. // at its default but for env. No request to it may use the network.
func newWAFHoldingTheCoreRuleSet( func newWAFHoldingTheCoreRuleSet(
t *testing.T, crs *heldCoreRuleSet, env map[string]string, t *testing.T, crs *heldCoreRuleSet, env map[string]string,
) (*waf.Server, *output, *alerts.Queue) { ) (*wafAndParts, *output, *alerts.Queue) {
t.Helper() t.Helper()
settings := map[string]string{ settings := map[string]string{
@@ -71,11 +71,9 @@ func newWAFHoldingTheCoreRuleSet(
} }
maps.Copy(settings, env) maps.Copy(settings, env)
server, out, queue := newWAFWith(t, func(params waf.Params) *waf.Server { server, out, queue := newWAFWith(t, func(params *waf.Params) {
params.CoreRuleSet = crs params.CoreRuleSet = crs
params.BesideLookupTimeLimit = waf.BesideLookupTimeLimit params.BesideLookupTimeLimit = waf.BesideLookupTimeLimit
return waf.New(params)
}, silentGeoJSURL, abuseIPDBURL, time.Now, settings) }, silentGeoJSURL, abuseIPDBURL, time.Now, settings)
return server, out, queue return server, out, queue
@@ -105,7 +103,7 @@ func wantAnswer(
// left no trace: line has none of its fields, and server has counted no match // left no trace: line has none of its fields, and server has counted no match
// of it and raised no alert for one. // of it and raised no alert for one.
func wantNoCoreRuleSetTrace( func wantNoCoreRuleSetTrace(
t *testing.T, server *waf.Server, queue *alerts.Queue, line logLine, t *testing.T, server *wafAndParts, queue *alerts.Queue, line logLine,
) { ) {
t.Helper() t.Helper()
@@ -198,7 +196,7 @@ type firstToRefuse struct {
env map[string]string env map[string]string
inspection time.Duration inspection time.Duration
took time.Duration took time.Duration
action string action requestlog.Action
} }
// firstToRefuseCases are the cases of // firstToRefuseCases are the cases of
+45 -24
View File
@@ -2,6 +2,7 @@ package waf
import ( import (
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/reputation"
) )
// whole is the percentage of each limit a client gets when no biased // whole is the percentage of each limit a client gets when no biased
@@ -29,6 +30,25 @@ func biasedThresholdsSet(cfg *config.WAF) bool {
cfg.ASNLimitPercentURL != "" cfg.ASNLimitPercentURL != ""
} }
// biasedThresholds are the settings and the lists by which limitPercentages
// lowers a client's limits.
type biasedThresholds struct {
// lists hold the file SWW_WAF_ASN_LIMIT_PERCENT_URL names.
lists *reputation.Lists
tagLimitPercent map[string]int64 // SWW_WAF_TAG_LIMIT_PERCENT
asnBytesLimitPercent map[string]int64 // SWW_WAF_ASN_BYTES_LIMIT_PERCENT
countryBytesLimitPercent map[string]int64 // SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT
unknownCountryLimitPercent int64 // SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT
// blocklistAction and blocklistLimitPercent are SWW_WAF_BLOCKLIST_ACTION
// and its percentage.
blocklistAction config.ReputationAction
blocklistLimitPercent int64
// dnsblAbuseIPDBAction and dnsblAbuseIPDBLimitPercent are
// SWW_WAF_DNSBL_ABUSEIPDB_ACTION and its percentage.
dnsblAbuseIPDBAction config.ReputationAction
dnsblAbuseIPDBLimitPercent int64
}
// limitPercentages returns the client's limit percentages, for the rate // limitPercentages returns the client's limit percentages, for the rate
// limits and for the byte limits, by its tags, its AS number and country // limits and for the byte limits, by its tags, its AS number and country
// as looked up, each "" when unknown, and the blocklists, DNSBL zones and // as looked up, each "" when unknown, and the blocklists, DNSBL zones and
@@ -45,78 +65,79 @@ func biasedThresholdsSet(cfg *config.WAF) bool {
// the byte limits, SWW_WAF_ASN_BYTES_LIMIT_PERCENT and // the byte limits, SWW_WAF_ASN_BYTES_LIMIT_PERCENT and
// SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT take the place of the first three for // SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT take the place of the first three for
// an AS number or a country they list. // an AS number or a country they list.
func (rq *request) limitPercentages() (percentage, percentage) { func (b biasedThresholds) limitPercentages(
cfg := rq.h.config rq *request,
asn, country := rq.line.ASN, rq.line.Country ) (percentage, percentage) {
asn, country := rq.lookupAnswer.ASN, rq.lookupAnswer.Country
unknown := percentage{percent: whole} unknown := percentage{percent: whole}
if country == "" && !rq.skippedLookupWait { if country == "" && !rq.skippedLookupWait {
unknown = percentage{ unknown = percentage{
cfg.UnknownCountryLimitPercent, b.unknownCountryLimitPercent,
"SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT", config.UnknownCountryLimitPercentSetting,
} }
} }
fetched := percentage{percent: whole} fetched := percentage{percent: whole}
if percent, listed := rq.h.lists.ASNLimitPercent(asn); listed { if percent, listed := b.lists.ASNLimitPercent(asn); listed {
fetched = percentage{percent, "SWW_WAF_ASN_LIMIT_PERCENT_URL"} fetched = percentage{percent, config.ASNLimitPercentURLSetting}
} }
blocklisted := percentage{percent: whole} blocklisted := percentage{percent: whole}
if rq.blocklisted && if rq.blocklisted &&
cfg.BlocklistAction == config.ReputationActionLimit { b.blocklistAction == config.ReputationActionLimit {
blocklisted = percentage{ blocklisted = percentage{
cfg.BlocklistLimitPercent, b.blocklistLimitPercent,
"SWW_WAF_BLOCKLIST_ACTION", config.BlocklistActionSetting,
} }
} }
dnsblAbuseIPDBListed := percentage{percent: whole} dnsblAbuseIPDBListed := percentage{percent: whole}
if (rq.dnsblListed || rq.abuseIPDBHit) && if (rq.dnsblListed || rq.abuseIPDBHit) &&
cfg.DNSBLAbuseIPDBAction == config.ReputationActionLimit { b.dnsblAbuseIPDBAction == config.ReputationActionLimit {
dnsblAbuseIPDBListed = percentage{ dnsblAbuseIPDBListed = percentage{
cfg.DNSBLAbuseIPDBLimitPercent, b.dnsblAbuseIPDBLimitPercent,
"SWW_WAF_DNSBL_ABUSEIPDB_ACTION", config.DNSBLAbuseIPDBActionSetting,
} }
} }
const tagSetting = "SWW_WAF_TAG_LIMIT_PERCENT" const tagSetting = config.TagLimitPercentSetting
asnRequests := lowest( asnRequests := lowest(
given(cfg.TagLimitPercent, asnTag(asn), tagSetting), given(b.tagLimitPercent, asnTag(asn), tagSetting),
fetched, fetched,
) )
countryRequests := given( countryRequests := given(
cfg.TagLimitPercent, b.tagLimitPercent,
countryTag(country), countryTag(country),
tagSetting, tagSetting,
) )
otherTags := percentage{percent: whole} otherTags := percentage{percent: whole}
for _, tag := range rq.line.Tags { for _, tag := range rq.tags {
if tag != asnTag(asn) && tag != countryTag(country) { if tag != asnTag(asn) && tag != countryTag(country) {
otherTags = lowest( otherTags = lowest(
otherTags, otherTags,
given(cfg.TagLimitPercent, tag, tagSetting), given(b.tagLimitPercent, tag, tagSetting),
) )
} }
} }
asnBytes, countryBytes := asnRequests, countryRequests asnBytes, countryBytes := asnRequests, countryRequests
if _, listed := cfg.ASNBytesLimitPercent[asn]; listed { if _, listed := b.asnBytesLimitPercent[asn]; listed {
asnBytes = given( asnBytes = given(
cfg.ASNBytesLimitPercent, b.asnBytesLimitPercent,
asn, asn,
"SWW_WAF_ASN_BYTES_LIMIT_PERCENT", config.ASNBytesLimitPercentSetting,
) )
} }
if _, listed := cfg.CountryBytesLimitPercent[country]; listed { if _, listed := b.countryBytesLimitPercent[country]; listed {
countryBytes = given( countryBytes = given(
cfg.CountryBytesLimitPercent, b.countryBytesLimitPercent,
country, country,
"SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT", config.CountryBytesLimitPercentSetting,
) )
} }
+8 -8
View File
@@ -18,7 +18,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest" "sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// The biased thresholds. // The biased thresholds.
@@ -73,7 +73,7 @@ func TestEachBiasedThresholdLowersTheRateLimits(t *testing.T) {
// Half of 4 requests a minute: the third breaks the limit. // Half of 4 requests a minute: the third breaks the limit.
for _, sent := range []struct { for _, sent := range []struct {
status int status int
action string action requestlog.Action
}{ }{
{http.StatusOK, requestlog.ActionForward}, {http.StatusOK, requestlog.ActionForward},
{http.StatusOK, requestlog.ActionForward}, {http.StatusOK, requestlog.ActionForward},
@@ -452,7 +452,7 @@ func TestBanForALoweredLimitGivesThePercentageInItsNotesAndItsAlert(
// limit, which is answered with status and logged with action. // limit, which is answered with status and logged with action.
before int before int
status int status int
action string action requestlog.Action
// reason and want are the ban's reason, and its notes' limit // reason and want are the ban's reason, and its notes' limit
// percentage, as percentText gives it. // percentage, as percentText gives it.
reason, want string reason, want string
@@ -529,7 +529,7 @@ func TestBanForALoweredLimitGivesThePercentageInItsNotesAndItsAlert(
// clock. // clock.
func startWithLookups( func startWithLookups(
t *testing.T, env map[string]string, t *testing.T, env map[string]string,
) (*sender, *waf.Server, *alerts.Queue) { ) (*sender, *wafAndParts, *alerts.Queue) {
t.Helper() t.Helper()
s, _, server, queue := startWithLookupsAndClock(t, env) s, _, server, queue := startWithLookupsAndClock(t, env)
@@ -544,7 +544,7 @@ func startWithLookups(
// country, and no other address. // country, and no other address.
func startWithLookupsAndClock( func startWithLookupsAndClock(
t *testing.T, env map[string]string, t *testing.T, env map[string]string,
) (*sender, *clock, *waf.Server, *alerts.Queue) { ) (*sender, *clock, *wafAndParts, *alerts.Queue) {
t.Helper() t.Helper()
path := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb") path := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb")
@@ -584,15 +584,15 @@ func (s *sender) uploadFrom(from string) logLine {
// straight to server's handler, without the network, and returns once it is // straight to server's handler, without the network, and returns once it is
// answered. // answered.
func serveFromDE( func serveFromDE(
t *testing.T, server *waf.Server, method string, body io.Reader, t *testing.T, server *wafAndParts, method string, body io.Reader,
) { ) {
t.Helper() t.Helper()
req := httptest.NewRequestWithContext(t.Context(), method, "/", body) req := httptest.NewRequestWithContext(t.Context(), method, "/", body)
req.Header.Set(waf.ClientHeader, fromDE) req.Header.Set(wafprotocol.ClientHeader, fromDE)
req.Header.Set(requestIDHeader, "the-id") req.Header.Set(requestIDHeader, "the-id")
server.Handler.ServeHTTP(httptest.NewRecorder(), req) server.ServeHTTP(httptest.NewRecorder(), req)
} }
// wantPercent checks a limit percentage that a log line or a ban's notes // wantPercent checks a limit percentage that a log line or a ban's notes
+29 -13
View File
@@ -1,47 +1,63 @@
package waf package waf
// The heuristic blocklists. It reads SWW_WAF_BLOCKLIST_ACTION, and the blocklists // The heuristic blocklists. It reads SWW_WAF_BLOCKLIST_ACTION and
// through the reputation lists. // SWW_WAF_BAN_RESPONSE, and the blocklists through the reputation lists.
import ( import (
"context" "context"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
// blocklists notes the blocklists that list the client, as noteListed does, // blocklists notes the blocklists that list the client, as noteListed does,
// and refuses the request when SWW_WAF_BLOCKLIST_ACTION is deny. Being limit, it // and refuses the request with SWW_WAF_BAN_RESPONSE when
// lowers the client's limits instead (see limitPercentages), and being log, it // SWW_WAF_BLOCKLIST_ACTION is deny. Being limit, it lowers the client's limits
// does nothing more. Its figure is the blocklists that list the client. // instead (see limitPercentages), and being log, it does nothing more. Its
type blocklists struct{} // figure is the blocklists that list the client.
type blocklists struct {
lists *reputation.Lists
action config.ReputationAction // SWW_WAF_BLOCKLIST_ACTION
banResponse int // SWW_WAF_BAN_RESPONSE
hitNoter reputationHitNoter
}
func (blocklists) name() string { return "blocklists" } func (blocklists) name() string { return "blocklists" }
func (blocklists) check( func (b blocklists) check(
_ context.Context, _ context.Context,
rq *request, rq *request,
_ time.Time, _ time.Time,
) *verdict { ) *verdict {
listedBy := rq.h.lists.ListedBy(rq.client) listedBy := b.lists.ListedBy(rq.client)
rq.blocklisted = len(listedBy) > 0 rq.blocklisted = len(listedBy) > 0
rq.noteListed(listedBy, "listed by a blocklist") b.hitNoter.noteListed(rq, listedBy, "listed by a blocklist")
if !rq.refusedByBlocklists(listedBy) { if !b.refusedByBlocklists(listedBy) {
return nil return nil
} }
return &verdict{ return &verdict{
action: requestlog.ActionClientDenied, action: requestlog.ActionClientDenied,
status: b.banResponse,
figures: map[string]any{"lists": listedBy}, figures: map[string]any{"lists": listedBy},
} }
} }
// refusesWhateverTheLookupAnswers reports whether blocklists refuses rq, as
// refusedByBlocklists says. The lookup's answer plays no part in it.
func (b blocklists) refusesWhateverTheLookupAnswers(
rq *request,
_ time.Time,
) bool {
return b.refusedByBlocklists(b.lists.ListedBy(rq.client))
}
// refusedByBlocklists reports whether the heuristic blocklists refuses a // refusedByBlocklists reports whether the heuristic blocklists refuses a
// client, given listedBy, the blocklists that list it: it does when one lists // client, given listedBy, the blocklists that list it: it does when one lists
// it and SWW_WAF_BLOCKLIST_ACTION is deny. // it and SWW_WAF_BLOCKLIST_ACTION is deny.
func (rq *request) refusedByBlocklists(listedBy []string) bool { func (b blocklists) refusedByBlocklists(listedBy []string) bool {
return len(listedBy) > 0 && rq.h.config.BlocklistAction == return len(listedBy) > 0 && b.action == config.ReputationActionDeny
config.ReputationActionDeny
} }
+15 -7
View File
@@ -6,6 +6,9 @@ package waf
import ( import (
"context" "context"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
) )
// byteLimits counts the request's bytes, as countedBytes gives them, for the // byteLimits counts the request's bytes, as countedBytes gives them, for the
@@ -16,23 +19,28 @@ import (
// the client over a byte limit, as its limit percentage for the byte limits // the client over a byte limit, as its limit percentage for the byte limits
// lowers it, break it, which bans the client; the response was passed on whole. // lowers it, break it, which bans the client; the response was passed on whole.
// Its figures are the window, the count and the limit. // Its figures are the window, the count and the limit.
type byteLimits struct{} type byteLimits struct {
limiter *ratelimit.Limiter
bytesCounted config.BytesCounted // SWW_WAF_BYTES_COUNTED
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
banMaker banMaker
}
func (byteLimits) name() string { return "byte_limits" } func (byteLimits) name() string { return "byte_limits" }
func (byteLimits) check( func (b byteLimits) check(
_ context.Context, _ context.Context,
rq *request, rq *request,
now time.Time, now time.Time,
) *verdict { ) *verdict {
if !rq.counted || rq.line.WouldAction != "" { if !rq.counted || rq.wouldAction != "" {
return nil return nil
} }
counts, hit, over := rq.h.limiter.CountBytes( counts, hit, over := b.limiter.CountBytes(
clientGroup(rq.h.config, rq.client), clientGroup(b.ipv6GroupPrefix, rq.client),
now, now,
rq.countedBytes(), rq.countedBytes(b.bytesCounted),
rq.bytesLimitPercent.percent, rq.bytesLimitPercent.percent,
) )
rq.line.Counts.MinuteBytes = counts.MinuteBytes rq.line.Counts.MinuteBytes = counts.MinuteBytes
@@ -43,7 +51,7 @@ func (byteLimits) check(
return nil return nil
} }
rq.banForLimit(now, hit, rq.out.status) b.banMaker.banForLimit(rq, now, hit, rq.out.status)
return &verdict{figures: limitFigures(hit)} return &verdict{figures: limitFigures(hit)}
} }
+7 -9
View File
@@ -6,7 +6,7 @@ import (
"net/netip" "net/netip"
"slices" "slices"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// peerAddress is the address of the request's TCP peer. // peerAddress is the address of the request's TCP peer.
@@ -19,14 +19,11 @@ func peerAddress(r *http.Request) netip.Addr {
return addrPort.Addr().Unmap() return addrPort.Addr().Unmap()
} }
// requestIDHeader carries the request's id, from rproxy.
const requestIDHeader = "X-Request-ID"
// requestID is the request's id: the one a question gives, or for any other // requestID is the request's id: the one a question gives, or for any other
// request a new random one. // request a new random one.
func requestID(r *http.Request, isQuestion bool) string { func requestID(r *http.Request, isQuestion bool) string {
if isQuestion { if isQuestion {
return r.Header.Get(requestIDHeader) return r.Header.Get(wafprotocol.RequestIDHeader)
} }
return rand.Text() return rand.Text()
@@ -45,12 +42,13 @@ func scheme(r *http.Request, isQuestion bool) string {
// clientGroup is the client a request is counted toward: its IPv4 // clientGroup is the client a request is counted toward: its IPv4
// address, or its IPv6 group, the netblock its IPv6 address is in of the // address, or its IPv6 group, the netblock its IPv6 address is in of the
// length SWW_WAF_IPV6_GROUP_PREFIX sets, a /64 by default, since one abuser // length ipv6GroupPrefix, SWW_WAF_IPV6_GROUP_PREFIX, a /64 by default, since
// usually holds a whole /64. An IPv4 address in IPv6 form counts as IPv4. // one abuser usually holds a whole /64. An IPv4 address in IPv6 form counts as
func clientGroup(cfg *config.WAF, addr netip.Addr) netip.Prefix { // IPv4.
func clientGroup(ipv6GroupPrefix int, addr netip.Addr) netip.Prefix {
addr = addr.Unmap() addr = addr.Unmap()
if addr.Is6() { if addr.Is6() {
return netip.PrefixFrom(addr, cfg.IPv6GroupPrefix).Masked() return netip.PrefixFrom(addr, ipv6GroupPrefix).Masked()
} }
return netip.PrefixFrom(addr, addr.BitLen()) return netip.PrefixFrom(addr, addr.BitLen())
+35 -21
View File
@@ -5,11 +5,13 @@ package waf
import ( import (
"context" "context"
"net/http"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/coreruleset" "sneak.berlin/go/smallwebwaf/internal/coreruleset"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
@@ -22,11 +24,22 @@ import (
// exploitsignature, raises the core_rule_set_match alert, and in block mode // exploitsignature, raises the core_rule_set_match alert, and in block mode
// refuses the request with 403, which is an offence its client's history // refuses the request with 403, which is an offence its client's history
// counts. Its figures are the score and the threshold. // counts. Its figures are the score and the threshold.
type coreRuleSet struct{} type coreRuleSet struct {
// inspector is the Core Rule Set, nil while SWW_WAF_CORE_RULE_SET_MODE is
// off.
inspector CoreRuleSetInspector
metrics *metrics.Metrics
alerts *alerts.Queue
mode config.CoreRuleSetMode // SWW_WAF_CORE_RULE_SET_MODE
exemptPaths []string // SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS
scoreThreshold int // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD
observe bool // SWW_WAF_MODE is observe
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
}
func (coreRuleSet) name() string { return "core_rule_set" } func (coreRuleSet) name() string { return "core_rule_set" }
func (coreRuleSet) check( func (c coreRuleSet) check(
_ context.Context, _ context.Context,
rq *request, rq *request,
_ time.Time, _ time.Time,
@@ -36,7 +49,6 @@ func (coreRuleSet) check(
return nil return nil
} }
cfg := rq.h.config
result := findings.inspection result := findings.inspection
rq.requestBytes = int64(len(findings.read)) rq.requestBytes = int64(len(findings.read))
@@ -47,18 +59,18 @@ func (coreRuleSet) check(
rq.line.CoreRuleSetScore = &result.Score rq.line.CoreRuleSetScore = &result.Score
for _, id := range result.RuleIDs { for _, id := range result.RuleIDs {
rq.h.metrics.CoreRuleSetMatched(string(cfg.CoreRuleSetMode), id) c.metrics.CoreRuleSetMatched(string(c.mode), id)
} }
threshold := cfg.CoreRuleSetScoreThreshold threshold := c.scoreThreshold
if threshold == 0 || result.Score < threshold { if threshold == 0 || result.Score < threshold {
return nil return nil
} }
rq.addTag(config.ExploitSignatureTag) rq.addTag(config.ExploitSignatureTag)
rq.alertCoreRuleSetMatch(result) c.alertCoreRuleSetMatch(rq, result)
if !rq.refusedByCoreRuleSet(result) { if !c.refusedByCoreRuleSet(result) {
return nil return nil
} }
@@ -66,6 +78,7 @@ func (coreRuleSet) check(
return &verdict{ return &verdict{
action: requestlog.ActionBlockedByCoreRuleSet, action: requestlog.ActionBlockedByCoreRuleSet,
status: http.StatusForbidden,
figures: map[string]any{ figures: map[string]any{
figureScore: result.Score, figureScore: result.Score,
"threshold": threshold, "threshold": threshold,
@@ -76,20 +89,21 @@ func (coreRuleSet) check(
// refusedByCoreRuleSet reports whether coreRuleSet refuses a request the Core // refusedByCoreRuleSet reports whether coreRuleSet refuses a request the Core
// Rule Set found result of: whether the score is at or over // Rule Set found result of: whether the score is at or over
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD in block mode. // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD in block mode.
func (rq *request) refusedByCoreRuleSet(result coreruleset.Result) bool { func (c coreRuleSet) refusedByCoreRuleSet(result coreruleset.Result) bool {
cfg := rq.h.config threshold := c.scoreThreshold
threshold := cfg.CoreRuleSetScoreThreshold
return cfg.CoreRuleSetMode == config.CoreRuleSetModeBlock && return c.mode == config.CoreRuleSetModeBlock &&
threshold != 0 && result.Score >= threshold threshold != 0 && result.Score >= threshold
} }
// alertCoreRuleSetMatch raises the core_rule_set_match alert for the request, // alertCoreRuleSetMatch raises the core_rule_set_match alert for rq, which the
// which the Core Rule Set scored at result, at or over // Core Rule Set scored at result, at or over
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD. Its detail gives the rule ids, the // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD. Its detail gives the rule ids, the
// score, the method and the path with the query, and, for a request that is // score, the method and the path with the query, and, for a request that is
// not refused for it, the mode: detect, or observe in observe mode. // not refused for it, the mode: detect, or observe in observe mode.
func (rq *request) alertCoreRuleSetMatch(result coreruleset.Result) { func (c coreRuleSet) alertCoreRuleSetMatch(
rq *request, result coreruleset.Result,
) {
detail := map[string]any{ detail := map[string]any{
"rule_ids": result.RuleIDs, "rule_ids": result.RuleIDs,
"score": result.Score, "score": result.Score,
@@ -98,20 +112,20 @@ func (rq *request) alertCoreRuleSetMatch(result coreruleset.Result) {
} }
switch { switch {
case rq.h.config.CoreRuleSetMode == config.CoreRuleSetModeDetect: case c.mode == config.CoreRuleSetModeDetect:
detail["mode"] = string(config.CoreRuleSetModeDetect) detail["mode"] = string(config.CoreRuleSetModeDetect)
case rq.h.config.Observe: case c.observe:
detail["mode"] = "observe" detail["mode"] = "observe"
} }
rq.h.alerts.Raise(alerts.Alert{ c.alerts.Raise(alerts.Alert{
Event: alerts.EventCoreRuleSetMatch, Event: alerts.EventCoreRuleSetMatch,
Host: rq.site, Host: rq.site,
Client: rq.client, Client: rq.client,
Netblock: clientGroup(rq.h.config, rq.client), Netblock: clientGroup(c.ipv6GroupPrefix, rq.client),
ASN: rq.line.ASN, ASN: rq.lookupAnswer.ASN,
ASName: rq.line.ASName, ASName: rq.lookupAnswer.ASName,
Country: rq.line.Country, Country: rq.lookupAnswer.Country,
Reason: "scored by the Core Rule Set at or over " + Reason: "scored by the Core Rule Set at or over " +
"SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD", "SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD",
Detail: detail, Detail: detail,
+10 -8
View File
@@ -12,7 +12,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// The Core Rule Set's settings the tests set, besides SWW_WAF_CORE_RULE_SET_MODE, and // The Core Rule Set's settings the tests set, besides SWW_WAF_CORE_RULE_SET_MODE, and
@@ -84,7 +84,7 @@ func TestCoreRuleSetMatchAddsTheExploitSignatureTag(t *testing.T) {
name string name string
mode string mode string
status int status int
action string action requestlog.Action
}{ }{
{block, block, http.StatusForbidden, requestlog.ActionBlockedByCoreRuleSet}, {block, block, http.StatusForbidden, requestlog.ActionBlockedByCoreRuleSet},
{detect, detect, http.StatusOK, requestlog.ActionForward}, {detect, detect, http.StatusOK, requestlog.ActionForward},
@@ -158,8 +158,9 @@ func TestCoreRuleSetRefusesAttacksInBlockModeAndOnlyLogsThemInDetectMode(
t.Parallel() t.Parallel()
for _, tc := range []struct { for _, tc := range []struct {
mode, action string mode string
status int action requestlog.Action
status int
}{ }{
{block, requestlog.ActionBlockedByCoreRuleSet, http.StatusForbidden}, {block, requestlog.ActionBlockedByCoreRuleSet, http.StatusForbidden},
{detect, requestlog.ActionForward, http.StatusOK}, {detect, requestlog.ActionForward, http.StatusOK},
@@ -395,9 +396,9 @@ func TestGiteasFixesApplyOnlyToAHostnameThatServesGitea(t *testing.T) {
}) })
wantDisposition(t, do(t, s.question(siteA, client, fileName)), wantDisposition(t, do(t, s.question(siteA, client, fileName)),
http.StatusOK, waf.DispositionAllow) http.StatusOK, wafprotocol.DispositionAllow)
wantDisposition(t, do(t, s.question(siteB, client, fileName)), wantDisposition(t, do(t, s.question(siteB, client, fileName)),
http.StatusForbidden, waf.DispositionDeny) http.StatusForbidden, wafprotocol.DispositionDeny)
lines := s.out.requestLines(t, 2) lines := s.out.requestLines(t, 2)
wantLine(t, lines[0], http.StatusOK, requestlog.ActionForward) wantLine(t, lines[0], http.StatusOK, requestlog.ActionForward)
@@ -652,8 +653,9 @@ func TestCoreRuleSetMatchRaisesTheCoreRuleSetMatchAlert(t *testing.T) {
// status and action are what the request is answered and logged // status and action are what the request is answered and logged
// with, and alertMode what the alert's detail gives as mode, if // with, and alertMode what the alert's detail gives as mode, if
// anything. // anything.
status int status int
action, alertMode string action requestlog.Action
alertMode string
}{ }{
{ {
"block", map[string]string{coreRuleSetMode: block}, "block", map[string]string{coreRuleSetMode: block},
+16 -8
View File
@@ -1,30 +1,36 @@
package waf package waf
// The heuristic countries. It reads SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES. // The heuristic countries. It reads SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES and
// SWW_WAF_BAN_RESPONSE.
import ( import (
"context" "context"
"slices" "slices"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
// countries refuses a request when SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES is set and // countries refuses a request when SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES is set and
// does not hold the client's country as it was looked up. A client without a // does not hold the client's country as it was looked up. A client without a
// country, or whose country cannot be found, is refused. Its figures are the // country, or whose country cannot be found, is refused. It refuses with
// country and the setting that refused it. // SWW_WAF_BAN_RESPONSE. Its figures are the country and the setting that
type countries struct{} // refused it.
type countries struct {
allowed []string // SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES
banResponse int // SWW_WAF_BAN_RESPONSE
}
func (countries) name() string { return "countries" } func (countries) name() string { return "countries" }
func (countries) check( func (c countries) check(
_ context.Context, _ context.Context,
rq *request, rq *request,
_ time.Time, _ time.Time,
) *verdict { ) *verdict {
allowed := rq.h.config.ExclusivelyAllowedCountries allowed := c.allowed
country := rq.line.Country country := rq.lookupAnswer.Country
if len(allowed) == 0 || slices.Contains(allowed, country) { if len(allowed) == 0 || slices.Contains(allowed, country) {
return nil return nil
@@ -32,8 +38,10 @@ func (countries) check(
return &verdict{ return &verdict{
action: requestlog.ActionCountryDenied, action: requestlog.ActionCountryDenied,
status: c.banResponse,
figures: map[string]any{ figures: map[string]any{
"country": country, "setting": "SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES", "country": country,
"setting": config.ExclusivelyAllowedCountriesSetting,
}, },
} }
} }
+1 -1
View File
@@ -316,7 +316,7 @@ func TestExclusiveListRefusesAPrivateAddressUnlessAllowed(t *testing.T) {
name string name string
allowNets string allowNets string
status int status int
action string action requestlog.Action
}{ }{
{ {
"not in SWW_WAF_ALLOW_NETS", "", "not in SWW_WAF_ALLOW_NETS", "",
+15 -6
View File
@@ -1,25 +1,34 @@
package waf package waf
// The heuristic deny_nets. It reads SWW_WAF_DENY_NETS. // The heuristic deny_nets. It reads SWW_WAF_DENY_NETS and SWW_WAF_BAN_RESPONSE.
import ( import (
"context" "context"
"net/netip"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
// denyNets refuses a client in SWW_WAF_DENY_NETS. Its figure is the netblock of // denyNets refuses a client in SWW_WAF_DENY_NETS, with SWW_WAF_BAN_RESPONSE.
// the setting the client is in. // Its figure is the netblock of the setting the client is in.
type denyNets struct{} type denyNets struct {
netblocks []netip.Prefix // SWW_WAF_DENY_NETS
banResponse int // SWW_WAF_BAN_RESPONSE
}
func (denyNets) name() string { return "deny_nets" } func (denyNets) name() string { return "deny_nets" }
func (denyNets) check(_ context.Context, rq *request, _ time.Time) *verdict { func (d denyNets) check(
for _, netblock := range rq.h.config.DenyNets { _ context.Context,
rq *request,
_ time.Time,
) *verdict {
for _, netblock := range d.netblocks {
if netblock.Contains(rq.client) { if netblock.Contains(rq.client) {
return &verdict{ return &verdict{
action: requestlog.ActionClientDenied, action: requestlog.ActionClientDenied,
status: d.banResponse,
figures: map[string]any{figureNetblock: netblock.String()}, figures: map[string]any{figureNetblock: netblock.String()},
} }
} }
+47
View File
@@ -0,0 +1,47 @@
package waf
import (
"net/netip"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
func TestDenyNetsRefusesOnlyAClientInsideItsNetblocks(t *testing.T) {
t.Parallel()
heuristic := denyNets{
netblocks: []netip.Prefix{netip.MustParsePrefix("198.51.100.0/24")},
}
// The request has no handler: the heuristic reads only what it holds.
inside := &request{
decision: decision{client: netip.MustParseAddr("198.51.100.7")},
}
v := heuristic.check(t.Context(), inside, time.Time{})
if v == nil {
t.Fatal("a client inside the netblock was let through")
}
if v.action != requestlog.ActionClientDenied {
t.Errorf(
"action = %q, want %q",
v.action,
requestlog.ActionClientDenied,
)
}
if got := v.figures[figureNetblock]; got != "198.51.100.0/24" {
t.Errorf("netblock figure = %v, want 198.51.100.0/24", got)
}
outside := &request{
decision: decision{client: netip.MustParseAddr("203.0.113.7")},
}
if v := heuristic.check(t.Context(), outside, time.Time{}); v != nil {
t.Errorf("a client outside the netblock was refused: %+v", v)
}
}
+25 -13
View File
@@ -1,7 +1,8 @@
package waf package waf
// The heuristic dnsbl. It reads SWW_WAF_DNSBL_ABUSEIPDB_ACTION, and the DNSBL zones' // The heuristic dnsbl. It reads SWW_WAF_DNSBL_ABUSEIPDB_ACTION and
// verdicts through the DNSBL checker. // SWW_WAF_BAN_RESPONSE, and the DNSBL zones' verdicts through the DNSBL
// checker.
import ( import (
"context" "context"
@@ -9,36 +10,47 @@ import (
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
// dnsbl notes the DNSBL zones whose verdict lists the client, as noteListed // dnsbl notes the DNSBL zones whose verdict lists the client, as noteListed
// does, adds the tag dnsbl=<zone> for each, the zone's key masked, and refuses // does, adds the tag dnsbl=<zone> for each, the zone's key masked, and refuses
// the request when SWW_WAF_DNSBL_ABUSEIPDB_ACTION is deny. Being // the request with SWW_WAF_BAN_RESPONSE when SWW_WAF_DNSBL_ABUSEIPDB_ACTION is
// limit, it lowers the client's limits instead (see limitPercentages), and // deny. Being limit, it lowers the client's limits instead (see
// being log, it does nothing more. A zone without a verdict on the client is // limitPercentages), and being log, it does nothing more. A zone without a
// asked about it in the background, and the request does not wait for the // verdict on the client is asked about it in the background, and the request
// answer. Its figure is the zones that list the client, their keys masked. // does not wait for the answer. Its figure is the zones that list the client,
type dnsbl struct{} // their keys masked.
type dnsbl struct {
dnsbl *reputation.DNSBL
action config.ReputationAction // SWW_WAF_DNSBL_ABUSEIPDB_ACTION
banResponse int // SWW_WAF_BAN_RESPONSE
hitNoter reputationHitNoter
}
func (dnsbl) name() string { return "dnsbl" } func (dnsbl) name() string { return "dnsbl" }
func (dnsbl) check(ctx context.Context, rq *request, _ time.Time) *verdict { func (d dnsbl) check(
listedBy := rq.h.dnsbl.ListedBy(ctx, rq.client) ctx context.Context,
rq *request,
_ time.Time,
) *verdict {
listedBy := d.dnsbl.ListedBy(ctx, rq.client)
rq.dnsblListed = len(listedBy) > 0 rq.dnsblListed = len(listedBy) > 0
rq.noteListed(listedBy, "listed by a DNSBL zone") d.hitNoter.noteListed(rq, listedBy, "listed by a DNSBL zone")
for _, zone := range listedBy { for _, zone := range listedBy {
rq.addTag(config.DNSBLTagPrefix + strings.ToLower(zone)) rq.addTag(config.DNSBLTagPrefix + strings.ToLower(zone))
} }
if !rq.dnsblListed || if !rq.dnsblListed || d.action != config.ReputationActionDeny {
rq.h.config.DNSBLAbuseIPDBAction != config.ReputationActionDeny {
return nil return nil
} }
return &verdict{ return &verdict{
action: requestlog.ActionClientDenied, action: requestlog.ActionClientDenied,
status: d.banResponse,
figures: map[string]any{"zones": listedBy}, figures: map[string]any{"zones": listedBy},
} }
} }
+4 -6
View File
@@ -1,13 +1,11 @@
package waf package waf
import "sneak.berlin/go/smallwebwaf/internal/reputation" import "net/http"
// BesideLookupTimeLimit is besideLookupTimeLimit. // BesideLookupTimeLimit is besideLookupTimeLimit.
const BesideLookupTimeLimit = besideLookupTimeLimit const BesideLookupTimeLimit = besideLookupTimeLimit
// BanAbuseIPDBBlacklist does on s what a fetch of AbuseIPDB's blacklist with // HTTPServer is Go's server s holds, for tests that read its settings.
// the clients on listed does. func (s *Server) HTTPServer() *http.Server {
func (s *Server) BanAbuseIPDBBlacklist(listed []reputation.BlacklistEntry) { return s.httpServer
h := s.Handler.(*handler) //nolint:forcetypeassert // New makes it one
BanAbuseIPDBBlacklist(h.config, h.ledger, h.processLog, h.now(), listed)
} }
Loaded 100 of 133 files, more files were not shown because too many files have changed in this diff. Show more