Compare commits

..
Author SHA1 Message Date
sneak bfdf17126c rproxy tests: leave room for a busy host in five timing tests (closes #451)
check / check (push) Waiting to run
Three tests that wait out a 2- or 3-second request time limit now use 5
seconds, so the request reaches the app before it runs out under load.
The WebSocket test also uses 5-second limits, and renews its
connection's deadline after the wait.

TestFiveFailedAsksKeepTheNextRequestFromAskingWAF stops the breaker's
clock like TestAnswersThatCannotBePassedOnOpenTheBreaker.

Model: opus-5-5
2026-11-19 19:24:37 +00:00
89 changed files with 2330 additions and 3887 deletions

No files matched your search

+3 -9
View File
@@ -902,19 +902,13 @@ stops the running process (see [`docs/state-files.md`](docs/state-files.md)).
- `internal/rproxy`: the reverse proxy in front of one app. - `internal/rproxy`: the reverse proxy in front of one app.
- `internal/waf`: `waf`'s server, which runs the checks and answers its own - `internal/waf`: `waf`'s server, which runs the checks and answers its own
endpoints. endpoints.
- `internal/wafprotocol`: the question, the answer and the report that `rproxy`
and `waf` send each other.
- `internal/ratelimit`, `internal/bans`, `internal/rules`, - `internal/ratelimit`, `internal/bans`, `internal/rules`,
`internal/coreruleset`, `internal/geojs`, `internal/lookup`, `internal/coreruleset`, `internal/geojs`, `internal/lookup`,
`internal/reputation`, `internal/routewatch`, `internal/anomaly` and `internal/reputation`, `internal/routewatch`, `internal/anomaly` and
`internal/alerts`: one check or source each. `internal/alerts`: one check or source each.
- `internal/dirwatch`: watches a directory, so that `internal/rules` and - `internal/state`, `internal/requestlog`, `internal/remotelog` and
`internal/lookup` read their files again when they change. `internal/metrics`: the state and cache files, the log lines, the syslog
- `internal/state`: the state and cache files. sender and the metrics.
- `internal/requestlog`: the request log.
- `internal/processlog`: the process's own messages.
- `internal/remotelog`: the syslog sender.
- `internal/metrics`: the metrics.
## Rationale ## Rationale
+7 -18
View File
@@ -69,21 +69,17 @@ never as a time.
that command's settings, and runs `internal/rproxy` until `SIGTERM` or that command's settings, and runs `internal/rproxy` until `SIGTERM` or
`SIGINT` (see "`rproxy`"). `sww healthcheck` is the image's health check. `SIGINT` (see "`rproxy`"). `sww healthcheck` is the image's health check.
With no command, or one it does not know, the binary prints the commands With no command, or one it does not know, the binary prints the commands
and exits with status 2. `sww help` and `sww --help` print them and exit and exits with status 2. `internal/sww` is the only package that imports
with status 0. `internal/sww` is the only package that imports
`go.uber.org/fx`. fx builds the parts of `waf` and `rproxy`, and starts `go.uber.org/fx`. fx builds the parts of `waf` and `rproxy`, and starts
and stops their background work. and stops their background work.
- `internal/config`: reads the settings, the one place they are read. - `internal/config`: reads the settings, the one place they are read.
- `internal/waf`: `waf`'s server. It runs the checks in `check`. Its - `internal/waf`: `waf`'s server. It runs the checks in `check`, and holds
`routes.go` lists `waf`'s own endpoints. It counts bytes, refusals and `waf`'s question, answer and report. It answers `waf`'s own endpoints in
anomalies, and writes the log lines. `answerOwnEndpoint`. It counts bytes, refusals and anomalies, and writes
the log lines.
- `internal/rproxy`: the reverse proxy in front of one app that asks `waf` - `internal/rproxy`: the reverse proxy in front of one app that asks `waf`
about each request. It passes a request to the app with the standard about each request. It passes a request to the app with the standard
library's `httputil.ReverseProxy`, within the time and size limits. library's `httputil.ReverseProxy`, within the time and size limits.
- `internal/wafprotocol`: the question, the answer and the report that
`rproxy` and `waf` send each other (see "`waf`"). It writes and reads each
one, and declares every header, path and limit both sides use.
`internal/rproxy` imports it, not `internal/waf`.
- `internal/ratelimit`: the table of clients. It counts requests, bytes and - `internal/ratelimit`: the table of clients. It counts requests, bytes and
refusals, and keeps each client's history. refusals, and keeps each client's history.
- `internal/bans`: the bans. It holds each netblock's bans with their notes, - `internal/bans`: the bans. It holds each netblock's bans with their notes,
@@ -97,9 +93,6 @@ never as a time.
imports no other package of this repository: `waf` gives it what it needs imports no other package of this repository: `waf` gives it what it needs
to count and report failures. to count and report failures.
- `internal/lookup`: looks clients up in the lookup database. - `internal/lookup`: looks clients up in the lookup database.
- `internal/dirwatch`: watches a directory, and calls a function once it has
gone a short time without a change. `internal/rules` and `internal/lookup`
read their files again this way.
- `internal/reputation`: fetches the lists, asks the DNS blocklists and - `internal/reputation`: fetches the lists, asks the DNS blocklists and
AbuseIPDB in the background, and keeps their answers. It also fetches AbuseIPDB in the background, and keeps their answers. It also fetches
AbuseIPDB's blacklist, which `internal/waf` bans. AbuseIPDB's blacklist, which `internal/waf` bans.
@@ -111,10 +104,8 @@ never as a time.
its own. its own.
- `internal/state`: reads the state and cache files at start, takes in edits - `internal/state`: reads the state and cache files at start, takes in edits
while running, and writes the files when due and at the stop. while running, and writes the files when due and at the stop.
- `internal/requestlog`: the request log, as JSON lines on stdout. - `internal/requestlog` and `internal/remotelog`: the lines on stdout, and
- `internal/processlog`: the process lines on stdout, which hold the sending them to the syslog server.
process's own messages.
- `internal/remotelog`: sends the lines on stdout to the syslog server.
- `internal/metrics`: the metrics, in the Prometheus text format. - `internal/metrics`: the metrics, in the Prometheus text format.
- `Dockerfile`: the lint and test phases, then the image. - `Dockerfile`: the lint and test phases, then the image.
`deploy/example-app` is an app built on the image, with `rproxy` in its `deploy/example-app` is an app built on the image, with `rproxy` in its
@@ -128,13 +119,11 @@ never as a time.
default `SWW_WAF_CORE_RULE_SET_DISABLED_RULES`, are built for that default `SWW_WAF_CORE_RULE_SET_DISABLED_RULES`, are built for that
version. They are checked again before the version changes, since rule version. They are checked again before the version changes, since rule
ids, and what each rule matches, change between versions; ids, and what each rule matches, change between versions;
- `github.com/go-chi/chi/v5` to route the requests to `waf`'s own endpoints;
- `github.com/hashicorp/golang-lru/v2` for the bounded tables; - `github.com/hashicorp/golang-lru/v2` for the bounded tables;
- `github.com/oschwald/maxminddb-golang/v2` to read the lookup database; - `github.com/oschwald/maxminddb-golang/v2` to read the lookup database;
- `github.com/fsnotify/fsnotify` to notice edited files; - `github.com/fsnotify/fsnotify` to notice edited files;
- `github.com/prometheus/client_golang` for the metrics; - `github.com/prometheus/client_golang` for the metrics;
- `github.com/spf13/viper` to read the settings from the environment; - `github.com/spf13/viper` to read the settings from the environment;
- `github.com/spf13/cobra` for the commands and their help text;
- `github.com/maxmind/mmdbwriter`, in the tests, to write lookup databases; - `github.com/maxmind/mmdbwriter`, in the tests, to write lookup databases;
- remote log sending written in this project, since the standard library's - remote log sending written in this project, since the standard library's
`log/syslog` is frozen and writes only the older syslog format. `log/syslog` is frozen and writes only the older syslog format.
-3
View File
@@ -7,12 +7,10 @@ require (
github.com/corazawaf/coraza-coreruleset/v4 v4.25.0 github.com/corazawaf/coraza-coreruleset/v4 v4.25.0
github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68 github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68
github.com/fsnotify/fsnotify v1.10.1 github.com/fsnotify/fsnotify v1.10.1
github.com/go-chi/chi/v5 v5.5.0
github.com/hashicorp/golang-lru/v2 v2.0.7 github.com/hashicorp/golang-lru/v2 v2.0.7
github.com/maxmind/mmdbwriter v1.2.0 github.com/maxmind/mmdbwriter v1.2.0
github.com/oschwald/maxminddb-golang/v2 v2.7.0 github.com/oschwald/maxminddb-golang/v2 v2.7.0
github.com/prometheus/client_golang v1.24.1 github.com/prometheus/client_golang v1.24.1
github.com/spf13/cobra v1.10.2
github.com/spf13/viper v1.21.0 github.com/spf13/viper v1.21.0
go.uber.org/fx v1.24.0 go.uber.org/fx v1.24.0
golang.org/x/net v0.60.0 golang.org/x/net v0.60.0
@@ -28,7 +26,6 @@ require (
github.com/goccy/go-yaml v1.19.2 // indirect github.com/goccy/go-yaml v1.19.2 // indirect
github.com/gotnospirit/makeplural v0.0.0-20180622080156-a5f48d94d976 // indirect github.com/gotnospirit/makeplural v0.0.0-20180622080156-a5f48d94d976 // indirect
github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 // indirect github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/kaptinlin/go-i18n v0.1.4 // indirect github.com/kaptinlin/go-i18n v0.1.4 // indirect
github.com/kaptinlin/jsonschema v0.4.6 // indirect github.com/kaptinlin/jsonschema v0.4.6 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect github.com/kylelemons/godebug v1.1.0 // indirect
-11
View File
@@ -14,15 +14,12 @@ github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68 h1:KLEjvBBSw
github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68/go.mod h1:9XocblQEwK6EYylRth7PrPIifF2l0iSoYo4FAXhXU/8= github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68/go.mod h1:9XocblQEwK6EYylRth7PrPIifF2l0iSoYo4FAXhXU/8=
github.com/corazawaf/libinjection-go v0.3.4 h1:jX2Tit2xZxYJJyVryNXucYBZtQqS7473x9Nc6lgRRbk= github.com/corazawaf/libinjection-go v0.3.4 h1:jX2Tit2xZxYJJyVryNXucYBZtQqS7473x9Nc6lgRRbk=
github.com/corazawaf/libinjection-go v0.3.4/go.mod h1:Ik/+w3UmTWH9yn366RgS9D95K3y7Atb5m/H/gXzzPCk= github.com/corazawaf/libinjection-go v0.3.4/go.mod h1:Ik/+w3UmTWH9yn366RgS9D95K3y7Atb5m/H/gXzzPCk=
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/foxcpp/go-mockdns v1.2.0 h1:omK3OrHRD1IWJz1FuFBCFquhXslXoF17OvBS6JPzZF0= github.com/foxcpp/go-mockdns v1.2.0 h1:omK3OrHRD1IWJz1FuFBCFquhXslXoF17OvBS6JPzZF0=
github.com/foxcpp/go-mockdns v1.2.0/go.mod h1:IhLeSFGed3mJIAXPH2aiRQB+kqz7oqu8ld2qVbOu7Wk= github.com/foxcpp/go-mockdns v1.2.0/go.mod h1:IhLeSFGed3mJIAXPH2aiRQB+kqz7oqu8ld2qVbOu7Wk=
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8= github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0= github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho= github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho=
github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo= github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo=
github.com/go-chi/chi/v5 v5.5.0 h1:xdKQ1dJmVJsmgahrXSKhUx8NyVPzpwdlyrrQFpMPJgg=
github.com/go-chi/chi/v5 v5.5.0/go.mod h1:d1IG7GtKPSjPyBP365PH8sAGVHk3qfB/IhjFNRH1wWM=
github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs= github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM= github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4= github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
@@ -37,8 +34,6 @@ github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 h1:c7gcN
github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092/go.mod h1:ZZAN4fkkful3l1lpJwF8JbW41ZiG9TwJ2ZlqzQovBNU= github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092/go.mod h1:ZZAN4fkkful3l1lpJwF8JbW41ZiG9TwJ2ZlqzQovBNU=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k= github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM= github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/jcchavezs/mergefs v0.1.1 h1:D45R17m6dHnSVZefnhynoeZvcK2Uw0oTrRfoUOQ0S5Y= github.com/jcchavezs/mergefs v0.1.1 h1:D45R17m6dHnSVZefnhynoeZvcK2Uw0oTrRfoUOQ0S5Y=
github.com/jcchavezs/mergefs v0.1.1/go.mod h1:eRLTrsA+vFwQZ48hj8p8gki/5v9C2bFtHH5Mnn4bcGk= github.com/jcchavezs/mergefs v0.1.1/go.mod h1:eRLTrsA+vFwQZ48hj8p8gki/5v9C2bFtHH5Mnn4bcGk=
github.com/kaptinlin/go-i18n v0.1.4 h1:wCiwAn1LOcvymvWIVAM4m5dUAMiHunTdEubLDk4hTGs= github.com/kaptinlin/go-i18n v0.1.4 h1:wCiwAn1LOcvymvWIVAM4m5dUAMiHunTdEubLDk4hTGs=
@@ -77,7 +72,6 @@ github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+
github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY= github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY=
github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8= github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs= github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/sagikazarmark/locafero v0.11.0 h1:1iurJgmM9G3PA/I+wWYIOw/5SyBtxapeHDcg+AAIFXc= github.com/sagikazarmark/locafero v0.11.0 h1:1iurJgmM9G3PA/I+wWYIOw/5SyBtxapeHDcg+AAIFXc=
github.com/sagikazarmark/locafero v0.11.0/go.mod h1:nVIGvgyzw595SUSUE6tvCp3YYTeHs15MvlmU87WwIik= github.com/sagikazarmark/locafero v0.11.0/go.mod h1:nVIGvgyzw595SUSUE6tvCp3YYTeHs15MvlmU87WwIik=
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 h1:+jumHNA0Wrelhe64i8F6HNlS8pkoyMv5sreGx2Ry5Rw= github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 h1:+jumHNA0Wrelhe64i8F6HNlS8pkoyMv5sreGx2Ry5Rw=
@@ -86,9 +80,6 @@ github.com/spf13/afero v1.15.0 h1:b/YBCLWAJdFWJTN9cLhiXXcD7mzKn9Dm86dNnfyQw1I=
github.com/spf13/afero v1.15.0/go.mod h1:NC2ByUVxtQs4b3sIUphxK0NioZnmxgyCrfzeuq8lxMg= github.com/spf13/afero v1.15.0/go.mod h1:NC2ByUVxtQs4b3sIUphxK0NioZnmxgyCrfzeuq8lxMg=
github.com/spf13/cast v1.10.0 h1:h2x0u2shc1QuLHfxi+cTJvs30+ZAHOGRic8uyGTDWxY= github.com/spf13/cast v1.10.0 h1:h2x0u2shc1QuLHfxi+cTJvs30+ZAHOGRic8uyGTDWxY=
github.com/spf13/cast v1.10.0/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqeHo= github.com/spf13/cast v1.10.0/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqeHo=
github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk= github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.21.0 h1:x5S+0EU27Lbphp4UKm1C+1oQO+rKx36vfCoaVebLFSU= github.com/spf13/viper v1.21.0 h1:x5S+0EU27Lbphp4UKm1C+1oQO+rKx36vfCoaVebLFSU=
@@ -118,7 +109,6 @@ go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so= go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ= go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ= go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw= go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg= go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba h1:0b9z3AuHCjxk0x/opv64kcgZLBseWJUpBw5I82+2U4M= go4.org/netipx v0.0.0-20231129151722-fdeea329fbba h1:0b9z3AuHCjxk0x/opv64kcgZLBseWJUpBw5I82+2U4M=
@@ -137,7 +127,6 @@ golang.org/x/tools v0.50.0 h1:c2ifzfcuY7L90lZ2aKd8S4K2NpASF08SZx9ZuJkHmSU=
golang.org/x/tools v0.50.0/go.mod h1:7ulVMw3831Mwi5EZD6RomGyffr4VFjuNYXf2BbCEAV0= golang.org/x/tools v0.50.0/go.mod h1:7ulVMw3831Mwi5EZD6RomGyffr4VFjuNYXf2BbCEAV0=
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE= google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
rsc.io/binaryregexp v0.2.0 h1:HfqmD5MEmC0zvwBuF187nq9mdnXjXsSivRiXN7SmRkE= rsc.io/binaryregexp v0.2.0 h1:HfqmD5MEmC0zvwBuF187nq9mdnXjXsSivRiXN7SmRkE=
+4 -8
View File
@@ -54,11 +54,7 @@ func TestBlacklistLeavesANetblockWithAnActiveBanAsItIs(t *testing.T) {
now := midnight() now := midnight()
scores := map[netip.Prefix]int64{netblock: 100} scores := map[netip.Prefix]int64{netblock: 100}
limitBan, _ := ledger.Ban( limitBan, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
netblock,
now,
bans.Grounds{Cause: bans.CauseLimit},
)
if made := ledger.BanForAbuseIPDB(now, now.Add(30*day), scores); made != 0 { if made := ledger.BanForAbuseIPDB(now, now.Add(30*day), scores); made != 0 {
t.Errorf("made %d bans over a ban for a limit, want none", made) t.Errorf("made %d bans over a ban for a limit, want none", made)
@@ -123,7 +119,7 @@ func TestEndedBlacklistBansAreForgottenAtTheNextFetch(t *testing.T) {
expires := now.Add(30 * day) expires := now.Add(30 * day)
ledger.BanForAbuseIPDB(now, expires, map[netip.Prefix]int64{listed: 100}) ledger.BanForAbuseIPDB(now, expires, map[netip.Prefix]int64{listed: 100})
ledger.Ban(limited, now, bans.Grounds{Cause: bans.CauseLimit}) ledger.BanForLimit(limited, now, bans.Notes{})
// listed is off the blacklist when it is fetched after the ban's end. // listed is off the blacklist when it is fetched after the ban's end.
if made := ledger.BanForAbuseIPDB(expires, expires.Add(30*day), nil); made != 0 { if made := ledger.BanForAbuseIPDB(expires, expires.Add(30*day), nil); made != 0 {
@@ -148,7 +144,7 @@ func TestBlacklistBansDoNotCountTowardMaxBans(t *testing.T) {
scores[netip.PrefixFrom(netip.AddrFrom4([4]byte{198, 51, 100, byte(i)}), 32)] = 100 scores[netip.PrefixFrom(netip.AddrFrom4([4]byte{198, 51, 100, byte(i)}), 32)] = 100
} }
ledger.Ban(limited, now, bans.Grounds{Cause: bans.CauseLimit}) ledger.BanForLimit(limited, now, bans.Notes{})
if made := ledger.BanForAbuseIPDB(now, now.Add(30*day), scores); made != 3 { if made := ledger.BanForAbuseIPDB(now, now.Add(30*day), scores); made != 3 {
t.Errorf("made %d bans for 3 netblocks, want 3", made) t.Errorf("made %d bans for 3 netblocks, want 3", made)
@@ -158,7 +154,7 @@ func TestBlacklistBansDoNotCountTowardMaxBans(t *testing.T) {
// The ban for an attack takes the place of the one for a limit, and no // The ban for an attack takes the place of the one for a limit, and no
// blacklist ban goes. // blacklist ban goes.
ledger.Ban(attacker, now, bans.Grounds{Cause: bans.CauseAttack}) ledger.BanForAttack(attacker, now, bans.Notes{})
want := map[netip.Prefix]int{limited: 0, attacker: 1} want := map[netip.Prefix]int{limited: 0, attacker: 1}
for netblock := range scores { for netblock := range scores {
+23 -64
View File
@@ -61,7 +61,7 @@ func TestAdminsBansAreNeverDroppedAndDoNotCountTowardMaxBans(t *testing.T) {
wantBans(t, ledger, map[netip.Prefix]int{adminsOnly: 2, both: 2}) wantBans(t, ledger, map[netip.Prefix]int{adminsOnly: 2, both: 2})
// A new ban drops the ban sww made, and only that one. // A new ban drops the ban sww made, and only that one.
ledger.Ban(second, midnight(), bans.Grounds{Cause: bans.CauseLimit}) ledger.BanForLimit(second, midnight(), bans.Notes{})
wantBans( wantBans(
t, t,
ledger, ledger,
@@ -77,7 +77,7 @@ func TestAdminsBansAreNeverDroppedAndDoNotCountTowardMaxBans(t *testing.T) {
} }
// And the next drops that one. // And the next drops that one.
ledger.Ban(third, midnight(), bans.Grounds{Cause: bans.CauseLimit}) ledger.BanForLimit(third, midnight(), bans.Notes{})
wantBans( wantBans(
t, t,
ledger, ledger,
@@ -98,28 +98,20 @@ func TestEarlierBansCountABanDroppedAfterAnAdminsBan(t *testing.T) {
// twice, each time after its last ban has ended. // twice, each time after its last ban has ended.
ledger.BanForAdmin(netblock, midnight(), midnight().Add(time.Minute), ledger.BanForAdmin(netblock, midnight(), midnight().Add(time.Minute),
"probes for logins") "probes for logins")
first, _ := ledger.Ban( first, _ := ledger.BanForLimit(
netblock, netblock,
midnight().Add(time.Minute), midnight().Add(time.Minute),
bans.Grounds{Cause: bans.CauseLimit}, bans.Notes{},
)
second, _ := ledger.Ban(
netblock,
first.Expires,
bans.Grounds{Cause: bans.CauseLimit},
) )
second, _ := ledger.BanForLimit(netblock, first.Expires, bans.Notes{})
// A ban on another netblock fills the ledger, and the netblock's first // A ban on another netblock fills the ledger, and the netblock's first
// ban for a limit, its earliest that is not an admin's, goes. // ban for a limit, its earliest that is not an admin's, goes.
ledger.Ban(other, second.Start, bans.Grounds{Cause: bans.CauseLimit}) ledger.BanForLimit(other, second.Start, bans.Notes{})
wantBans(t, ledger, map[netip.Prefix]int{netblock: 2, other: 1}) wantBans(t, ledger, map[netip.Prefix]int{netblock: 2, other: 1})
// The netblock's next ban still counts it. // The netblock's next ban still counts it.
third, _ := ledger.Ban( third, _ := ledger.BanForLimit(netblock, second.Expires, bans.Notes{})
netblock,
second.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
if third.Notes.EarlierBans != (bans.EarlierBans{Limit: 2, Admin: 1}) { if third.Notes.EarlierBans != (bans.EarlierBans{Limit: 2, Admin: 1}) {
t.Errorf("the third ban for a limit has earlier bans %+v, "+ t.Errorf("the third ban for a limit has earlier bans %+v, "+
"want 2 for a limit and 1 an admin's", third.Notes.EarlierBans) "want 2 for a limit and 1 an admin's", third.Notes.EarlierBans)
@@ -135,27 +127,15 @@ func TestEarlierBansCountABanAnAdminKeepsOnce(t *testing.T) {
// The netblock breaks a limit twice, the second time after its first // The netblock breaks a limit twice, the second time after its first
// ban has ended, and the second ban's notes count the first. An admin // ban has ended, and the second ban's notes count the first. An admin
// then keeps the first ban. // then keeps the first ban.
first, _ := ledger.Ban( first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
netblock, second, _ := ledger.BanForLimit(netblock, first.Expires, bans.Notes{})
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
second, _ := ledger.Ban(
netblock,
first.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
kept := first kept := first
kept.Cause = bans.CauseAdmin kept.Cause = bans.CauseAdmin
ledger.LoadEdit([]bans.Ban{kept, second}) ledger.LoadEdit([]bans.Ban{kept, second})
// The netblock's next ban counts the kept ban once, as an admin's. // The netblock's next ban counts the kept ban once, as an admin's.
third, _ := ledger.Ban( third, _ := ledger.BanForLimit(netblock, second.Expires, bans.Notes{})
netblock,
second.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
if third.Notes.EarlierBans != (bans.EarlierBans{Limit: 1, Admin: 1}) { if third.Notes.EarlierBans != (bans.EarlierBans{Limit: 1, Admin: 1}) {
t.Errorf("the third ban for a limit has earlier bans %+v, "+ t.Errorf("the third ban for a limit has earlier bans %+v, "+
"want 1 for a limit and 1 an admin's", third.Notes.EarlierBans) "want 1 for a limit and 1 an admin's", third.Notes.EarlierBans)
@@ -167,37 +147,20 @@ func TestReasonOfTheBansSwwMakes(t *testing.T) {
ledger := bans.New(defaultParams()) ledger := bans.New(defaultParams())
limit, _ := ledger.Ban( limit, _ := ledger.BanForLimit(
netip.MustParsePrefix("203.0.113.1/32"), netip.MustParsePrefix("203.0.113.1/32"),
midnight(), midnight(),
bans.Grounds{ bans.Notes{Kind: "requests", Limit: 1000, Window: "minute"},
Cause: bans.CauseLimit,
Notes: bans.Notes{
Kind: "requests",
Limit: 1000,
Window: "minute",
},
},
) )
byteLimit, _ := ledger.Ban( byteLimit, _ := ledger.BanForLimit(
netip.MustParsePrefix("203.0.113.3/32"), netip.MustParsePrefix("203.0.113.3/32"),
midnight(), midnight(),
bans.Grounds{ bans.Notes{Kind: "bytes", Limit: 10 << 30, Window: "hour"},
Cause: bans.CauseLimit,
Notes: bans.Notes{
Kind: "bytes",
Limit: 10 << 30,
Window: "hour",
},
},
) )
attack, _ := ledger.Ban( attack, _ := ledger.BanForAttack(
netip.MustParsePrefix("203.0.113.2/32"), netip.MustParsePrefix("203.0.113.2/32"),
midnight(), midnight(),
bans.Grounds{ bans.Notes{RuleID: "git-dir", Target: "path"},
Cause: bans.CauseAttack,
Notes: bans.Notes{RuleID: "git-dir", Target: "path"},
},
) )
for _, tc := range []struct{ got, want string }{ for _, tc := range []struct{ got, want string }{
@@ -237,7 +200,7 @@ func TestLiftedBanForALimitRefusesNothingAndMakesNoBanLonger(t *testing.T) {
t.Error("the lifted ban refuses") t.Error("the lifted ban refuses")
} }
ban, _ := ledger.Ban(netblock, now, bans.Grounds{Cause: bans.CauseLimit}) ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
if ban.Expires.Sub(ban.Start) != time.Hour || if ban.Expires.Sub(ban.Start) != time.Hour ||
ban.Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) { ban.Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) {
t.Errorf( t.Errorf(
@@ -288,11 +251,7 @@ func TestLiftedBanForAnAttackRefusesNothingAndMakesNoBanLonger(
} }
// The next clear sign of attack bans for seven days, as a first does. // The next clear sign of attack bans for seven days, as a first does.
ban, _ := ledger.Ban( ban, _ := ledger.BanForAttack(netblock, now, bans.Notes{})
netblock,
now,
bans.Grounds{Cause: bans.CauseAttack},
)
if ban.Expires.Sub(ban.Start) != 7*day { if ban.Expires.Sub(ban.Start) != 7*day {
t.Errorf( t.Errorf(
"the next ban for an attack ends at %s, want seven days on", "the next ban for an attack ends at %s, want seven days on",
@@ -305,10 +264,10 @@ func TestLoadEditCountsTheBansAnAdminMade(t *testing.T) {
t.Parallel() t.Parallel()
ledger := bans.New(defaultParams()) ledger := bans.New(defaultParams())
made, _ := ledger.Ban( made, _ := ledger.BanForLimit(
netip.MustParsePrefix("203.0.113.1/32"), netip.MustParsePrefix("203.0.113.1/32"),
midnight(), midnight(),
bans.Grounds{Cause: bans.CauseLimit}, bans.Notes{},
) )
atStart := bans.Ban{ atStart := bans.Ban{
Netblock: netip.MustParsePrefix("203.0.113.2/32"), Netblock: netip.MustParsePrefix("203.0.113.2/32"),
@@ -352,7 +311,7 @@ func TestAdminsBanIsMadeWhileAnotherLasts(t *testing.T) {
ledger := bans.New(defaultParams()) ledger := bans.New(defaultParams())
// An hour's ban for a broken limit. // An hour's ban for a broken limit.
ledger.Ban(netblock, midnight(), bans.Grounds{Cause: bans.CauseLimit}) ledger.BanForLimit(netblock, midnight(), bans.Notes{})
wantChanged(t, ledger, true) wantChanged(t, ledger, true)
// A minute later an admin bans the netblock for good, named by an // A minute later an admin bans the netblock for good, named by an
@@ -472,10 +431,10 @@ func TestPermanentBanOnAWiderNetblockIsReturnedOverATemporaryBan(
ledger := bans.New(defaultParams()) ledger := bans.New(defaultParams())
// The temporary ban is made first, so that its netblock is looked at first. // The temporary ban is made first, so that its netblock is looked at first.
ledger.Ban( ledger.BanForLimit(
netip.MustParsePrefix("203.0.113.9/32"), netip.MustParsePrefix("203.0.113.9/32"),
midnight(), midnight(),
bans.Grounds{Cause: bans.CauseLimit}, bans.Notes{},
) )
ledger.BanForAdmin( ledger.BanForAdmin(
netip.MustParsePrefix("203.0.113.0/24"), netip.MustParsePrefix("203.0.113.0/24"),
+82 -54
View File
@@ -300,50 +300,81 @@ func activeBan(bans []Ban, now time.Time) *Ban {
return newest return newest
} }
// Grounds are why a request makes a ban: its cause, CauseLimit or // BanForLimit bans netblock at now for a broken limit, with notes, and
// CauseAttack, and its notes. // returns the ban, and true. A first ban lasts LimitBanDuration. A ban
type Grounds struct { // made within LimitBanRepeatWindow after the netblock's ban that ended
Cause string // last, other than one for a clear sign of attack or a lifted one, lasts
Notes Notes // repeatFactor times as long as that one. A ban that would be longer
// than LimitBanPermanentAbove is permanent instead. If a ban on netblock is still
// active, as when two of its requests break a limit at once, that ban is
// returned with false, and no other is made. The ledger fills in the
// notes' Refused and EarlierBans itself, and gives the ban the reason
// "<Kind> per <Window> over the limit of <Limit>", from the notes, such
// as "requests per minute over the limit of 1000".
func (l *Ledger) BanForLimit(
netblock netip.Prefix, now time.Time, notes Notes,
) (Ban, bool) {
return l.ban(
netblock,
now,
CauseLimit,
limitReason(notes),
notes,
true,
)
} }
// Ban bans netblock at now on grounds, and returns the ban, and true. If // WouldBanForLimit returns what BanForLimit would, without making the ban:
// a ban on netblock is still active, as when two of its requests break a // what observe mode would have done.
// limit at once, that ban is returned with false, and no other is made. func (l *Ledger) WouldBanForLimit(
// The ledger fills in the notes' Refused and EarlierBans itself. netblock netip.Prefix, now time.Time, notes Notes,
//
// A ban for a broken limit:
// - lasts LimitBanDuration the first time;
// - lasts repeatFactor times as long as the netblock's ban that ended
// last, when made within LimitBanRepeatWindow after it, unless that
// ban was for a clear sign of attack or was lifted;
// - is permanent instead when it would be longer than
// LimitBanPermanentAbove;
// - has the reason "<Kind> per <Window> over the limit of <Limit>", from
// the notes, such as "requests per minute over the limit of 1000".
//
// A ban for a clear sign of attack:
// - lasts AttackBanDuration the first time;
// - is permanent once the netblock has had one that was not lifted;
// - has the reason "matched the rule <RuleID>", or "asked for the trap
// path <TrapPath>".
func (l *Ledger) Ban(
netblock netip.Prefix, now time.Time, grounds Grounds,
) (Ban, bool) { ) (Ban, bool) {
return l.ban(netblock, now, grounds, true) return l.ban(
netblock,
now,
CauseLimit,
limitReason(notes),
notes,
false,
)
} }
// WouldBan returns what Ban would, without making the ban: what observe // BanForAttack bans netblock at now for a clear sign of attack, with
// mode would have done. // notes, and returns the ban, and whether it made it, as BanForLimit
func (l *Ledger) WouldBan( // does. A first ban lasts AttackBanDuration; once the netblock has had
netblock netip.Prefix, now time.Time, grounds Grounds, // one that was not lifted, the next is permanent. Its reason is "matched
// the rule <RuleID>", or "asked for the trap path <TrapPath>".
func (l *Ledger) BanForAttack(
netblock netip.Prefix, now time.Time, notes Notes,
) (Ban, bool) { ) (Ban, bool) {
return l.ban(netblock, now, grounds, false) return l.ban(
netblock,
now,
CauseAttack,
attackReason(notes),
notes,
true,
)
}
// WouldBanForAttack returns what BanForAttack would, without making the
// ban: what observe mode would have done.
func (l *Ledger) WouldBanForAttack(
netblock netip.Prefix, now time.Time, notes Notes,
) (Ban, bool) {
return l.ban(
netblock,
now,
CauseAttack,
attackReason(notes),
notes,
false,
)
} }
// WouldBePermanent reports whether a ban on netblock for cause, CauseLimit // WouldBePermanent reports whether a ban on netblock for cause, CauseLimit
// or CauseAttack, made at now would be permanent, as Ban would make it. It // or CauseAttack, made at now would be permanent, as BanForLimit or
// works out nothing else of the ban. // BanForAttack would make it. It works out nothing else of the ban.
func (l *Ledger) WouldBePermanent( func (l *Ledger) WouldBePermanent(
netblock netip.Prefix, now time.Time, cause string, netblock netip.Prefix, now time.Time, cause string,
) bool { ) bool {
@@ -380,9 +411,10 @@ func attackReason(notes Notes) string {
// BanForAdmin bans netblock at now for an admin, with reason, until // BanForAdmin bans netblock at now for an admin, with reason, until
// expires, or for good when expires is zero, and returns the ban, whose // expires, or for good when expires is zero, and returns the ban, whose
// cause is CauseAdmin. Unlike Ban, it makes the ban even while another // cause is CauseAdmin. Unlike BanForLimit and BanForAttack, it makes the
// on netblock is active, since the admin asked for this one. The ledger // ban even while another on netblock is active, since the admin asked
// fills in the notes' EarlierBans, and counts the ban among those made. // for this one. The ledger fills in the notes' EarlierBans, and counts
// the ban among those made.
func (l *Ledger) BanForAdmin( func (l *Ledger) BanForAdmin(
netblock netip.Prefix, now, expires time.Time, reason string, netblock netip.Prefix, now, expires time.Time, reason string,
) Ban { ) Ban {
@@ -727,13 +759,15 @@ func (l *Ledger) forgetEndedAbuseIPDBBans(now time.Time) bool {
return forgot return forgot
} }
// ban bans netblock at now on grounds, as Ban describes, and returns the // ban bans netblock at now for cause, CauseLimit or CauseAttack, with
// ban, and whether it made it. Unless keep is true, the ban is not made, // reason and notes, as BanForLimit and BanForAttack describe, and returns
// only returned: it is the ban that would have been made. // the ban, and whether it made it. Unless keep is true, the ban is not
// made, only returned: it is the ban that would have been made.
func (l *Ledger) ban( func (l *Ledger) ban(
netblock netip.Prefix, netblock netip.Prefix,
now time.Time, now time.Time,
grounds Grounds, cause, reason string,
notes Notes,
keep bool, keep bool,
) (Ban, bool) { ) (Ban, bool) {
l.mu.Lock() l.mu.Lock()
@@ -742,8 +776,6 @@ func (l *Ledger) ban(
// held are the netblock's bans, none of them active. // held are the netblock's bans, none of them active.
var held []Ban var held []Ban
notes := grounds.Notes
bans, found := l.netblocks.Get(netblock) bans, found := l.netblocks.Get(netblock)
if found { if found {
active := activeBan(*bans, now) active := activeBan(*bans, now)
@@ -759,19 +791,15 @@ func (l *Ledger) ban(
ban := Ban{ ban := Ban{
Netblock: netblock, Netblock: netblock,
Start: now, Start: now,
Cause: grounds.Cause, Cause: cause,
Reason: reason,
Notes: notes, Notes: notes,
} }
switch grounds.Cause { if cause == CauseAttack {
case CauseLimit:
ban.Reason = limitReason(notes)
ban.Expires = l.limitExpiry(held, now)
case CauseAttack:
ban.Reason = attackReason(notes)
ban.Expires = l.attackExpiry(held, now) ban.Expires = l.attackExpiry(held, now)
default: } else {
panic("bans: a request's ban has the cause " + grounds.Cause) ban.Expires = l.limitExpiry(held, now)
} }
if !keep { if !keep {
@@ -779,7 +807,7 @@ func (l *Ledger) ban(
} }
l.add(ban) l.add(ban)
l.made[grounds.Cause]++ l.made[cause]++
l.markChanged() l.markChanged()
return ban, true return ban, true
+51 -121
View File
@@ -22,11 +22,7 @@ func TestRepeatsTripleUntilPermanent(t *testing.T) {
// Each ban is followed by another as soon as it ends: 1, 3, 9, 27 and // Each ban is followed by another as soon as it ends: 1, 3, 9, 27 and
// 81 hours. // 81 hours.
for i, hours := range []int{1, 3, 9, 27, 81} { for i, hours := range []int{1, 3, 9, 27, 81} {
ban, _ := ledger.Ban( ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
netblock,
now,
bans.Grounds{Cause: bans.CauseLimit},
)
length := time.Duration(hours) * time.Hour length := time.Duration(hours) * time.Hour
if !ban.Expires.Equal(now.Add(length)) || if !ban.Expires.Equal(now.Add(length)) ||
@@ -46,7 +42,7 @@ func TestRepeatsTripleUntilPermanent(t *testing.T) {
// The sixth would last 243 hours, more than seven days: it is // The sixth would last 243 hours, more than seven days: it is
// permanent, and never ends. // permanent, and never ends.
ban, _ := ledger.Ban(netblock, now, bans.Grounds{Cause: bans.CauseLimit}) ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
if !ban.Permanent() { if !ban.Permanent() {
t.Fatalf("sixth ban ends at %s, want a permanent one", ban.Expires) t.Fatalf("sixth ban ends at %s, want a permanent one", ban.Expires)
} }
@@ -75,15 +71,15 @@ func TestRepeatWindowRunsOut(t *testing.T) {
ledger := bans.New(defaultParams()) ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
first, _ := ledger.Ban( first, _ := ledger.BanForLimit(
netblock, netblock,
midnight(), midnight(),
bans.Grounds{Cause: bans.CauseLimit}, bans.Notes{},
) )
second, _ := ledger.Ban( second, _ := ledger.BanForLimit(
netblock, netblock,
first.Expires.Add(tc.gap), first.Expires.Add(tc.gap),
bans.Grounds{Cause: bans.CauseLimit}, bans.Notes{},
) )
if second.Expires.Sub(second.Start) != tc.want || if second.Expires.Sub(second.Start) != tc.want ||
@@ -108,10 +104,10 @@ func TestFirstBanLongerThanLimitBanPermanentAboveIsPermanent(t *testing.T) {
params.LimitBanDuration = params.LimitBanPermanentAbove + time.Hour params.LimitBanDuration = params.LimitBanPermanentAbove + time.Hour
ledger := bans.New(params) ledger := bans.New(params)
ban, _ := ledger.Ban( ban, _ := ledger.BanForLimit(
netip.MustParsePrefix("203.0.113.9/32"), netip.MustParsePrefix("203.0.113.9/32"),
midnight(), midnight(),
bans.Grounds{Cause: bans.CauseLimit}, bans.Notes{},
) )
if !ban.Permanent() { if !ban.Permanent() {
t.Errorf("first ban ends at %s, want a permanent one", ban.Expires) t.Errorf("first ban ends at %s, want a permanent one", ban.Expires)
@@ -131,11 +127,7 @@ func TestLimitBanPermanentAboveSetFarOffDoesNotOverflow(t *testing.T) {
now := midnight() now := midnight()
for i := range 14 { for i := range 14 {
ban, _ := ledger.Ban( ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
netblock,
now,
bans.Grounds{Cause: bans.CauseLimit},
)
if !ban.Expires.After(ban.Start) { if !ban.Expires.After(ban.Start) {
t.Fatalf( t.Fatalf(
"ban %d starts at %s and ends at %s", "ban %d starts at %s and ends at %s",
@@ -148,7 +140,7 @@ func TestLimitBanPermanentAboveSetFarOffDoesNotOverflow(t *testing.T) {
now = ban.Expires now = ban.Expires
} }
ban, _ := ledger.Ban(netblock, now, bans.Grounds{Cause: bans.CauseLimit}) ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
if !ban.Permanent() { if !ban.Permanent() {
t.Errorf("15th ban ends at %s, want a permanent one", ban.Expires) t.Errorf("15th ban ends at %s, want a permanent one", ban.Expires)
} }
@@ -160,19 +152,15 @@ func TestBrokenLimitDuringABanMakesNoOther(t *testing.T) {
ledger := bans.New(defaultParams()) ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
first, made := ledger.Ban( first, made := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
if !made { if !made {
t.Error("the first ban was not made") t.Error("the first ban was not made")
} }
again, made := ledger.Ban( again, made := ledger.BanForLimit(
netblock, netblock,
midnight().Add(time.Minute), midnight().Add(time.Minute),
bans.Grounds{Cause: bans.CauseLimit}, bans.Notes{},
) )
if made || !reflect.DeepEqual(again, first) || if made || !reflect.DeepEqual(again, first) ||
@@ -187,10 +175,10 @@ func TestBrokenLimitDuringABanMakesNoOther(t *testing.T) {
) )
} }
again, made = ledger.Ban( again, made = ledger.BanForAttack(
netblock, netblock,
midnight().Add(time.Minute), midnight().Add(time.Minute),
bans.Grounds{Cause: bans.CauseAttack}, bans.Notes{},
) )
if made || !reflect.DeepEqual(again, first) { if made || !reflect.DeepEqual(again, first) {
t.Errorf( t.Errorf(
@@ -207,10 +195,10 @@ func TestCheckRefusesWhileTheBanLastsAndCountsTheRefusals(t *testing.T) {
ledger := bans.New(defaultParams()) ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
ban, _ := ledger.Ban( ban, _ := ledger.BanForLimit(
netblock, netblock,
midnight(), midnight(),
bans.Grounds{Cause: bans.CauseLimit, Notes: bans.Notes{Requests: 5}}, bans.Notes{Requests: 5},
) )
for range 3 { for range 3 {
@@ -249,10 +237,10 @@ func TestFindCountsNothing(t *testing.T) {
ledger := bans.New(defaultParams()) ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
ban, _ := ledger.Ban( ban, _ := ledger.BanForLimit(
netblock, netblock,
midnight(), midnight(),
bans.Grounds{Cause: bans.CauseLimit, Notes: bans.Notes{Requests: 5}}, bans.Notes{Requests: 5},
) )
got, banned, _ := ledger.Find( got, banned, _ := ledger.Find(
@@ -299,24 +287,24 @@ func TestMaxBansDropsTheEarliestBanOfTheNetblockSeenLongestAgo(
d := netip.MustParsePrefix("2001:db8::/64") d := netip.MustParsePrefix("2001:db8::/64")
now := midnight() now := midnight()
first, _ := ledger.Ban(a, now, bans.Grounds{Cause: bans.CauseLimit}) first, _ := ledger.BanForLimit(a, now, bans.Notes{})
ledger.Ban(b, now, bans.Grounds{Cause: bans.CauseLimit}) ledger.BanForLimit(b, now, bans.Notes{})
ledger.Ban(c, now, bans.Grounds{Cause: bans.CauseLimit}) ledger.BanForLimit(c, now, bans.Notes{})
// A request from a makes b the netblock seen longest ago, and its ban // A request from a makes b the netblock seen longest ago, and its ban
// goes to make room for d's. // goes to make room for d's.
ledger.Check(a.Addr(), now) ledger.Check(a.Addr(), now)
ledger.Ban(d, now, bans.Grounds{Cause: bans.CauseLimit}) ledger.BanForLimit(d, now, bans.Notes{})
wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 0, c: 1, d: 1}) wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 0, c: 1, d: 1})
// a is banned again once its ban has ended; c, seen longest ago, goes. // a is banned again once its ban has ended; c, seen longest ago, goes.
ledger.Ban(a, first.Expires, bans.Grounds{Cause: bans.CauseLimit}) ledger.BanForLimit(a, first.Expires, bans.Notes{})
wantBans(t, ledger, map[netip.Prefix]int{a: 2, c: 0, d: 1}) wantBans(t, ledger, map[netip.Prefix]int{a: 2, c: 0, d: 1})
// With d seen since, a is seen longest ago, and its earlier ban goes // With d seen since, a is seen longest ago, and its earlier ban goes
// first. // first.
ledger.Check(d.Addr(), first.Expires) ledger.Check(d.Addr(), first.Expires)
ledger.Ban(b, first.Expires, bans.Grounds{Cause: bans.CauseLimit}) ledger.BanForLimit(b, first.Expires, bans.Notes{})
wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 1, d: 1}) wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 1, d: 1})
if !ledger.Bans(a)[0].Start.Equal(first.Expires) { if !ledger.Bans(a)[0].Start.Equal(first.Expires) {
@@ -337,16 +325,8 @@ func TestFullLedgerDropsTheEarlierBanOfTheNetblockBannedAgain(t *testing.T) {
ledger := bans.New(params) ledger := bans.New(params)
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
first, _ := ledger.Ban( first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
netblock, second, _ := ledger.BanForLimit(netblock, first.Expires, bans.Notes{})
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
second, _ := ledger.Ban(
netblock,
first.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
held := ledger.Bans(netblock) held := ledger.Bans(netblock)
if len(held) != 1 || !reflect.DeepEqual(held[0], second) || if len(held) != 1 || !reflect.DeepEqual(held[0], second) ||
@@ -363,11 +343,7 @@ func TestRequestDuringAnAttackBanMakesItPermanent(t *testing.T) {
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
notes := bans.Notes{RuleID: "env-file", Target: "path"} notes := bans.Notes{RuleID: "env-file", Target: "path"}
ban, _ := ledger.Ban( ban, _ := ledger.BanForAttack(netblock, midnight(), notes)
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseAttack, Notes: notes},
)
if !ban.Expires.Equal(midnight().Add(7*day)) || if !ban.Expires.Equal(midnight().Add(7*day)) ||
ban.Cause != bans.CauseAttack || ban.Cause != bans.CauseAttack ||
ban.Notes.RuleID != "env-file" || ban.Notes.RuleID != "env-file" ||
@@ -442,16 +418,8 @@ func TestAttackAfterAnAttackBanHasEndedBansPermanently(t *testing.T) {
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
// A ban for a broken limit before does not count. // A ban for a broken limit before does not count.
first, _ := ledger.Ban( first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
netblock, second, _ := ledger.BanForAttack(netblock, first.Expires, bans.Notes{})
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
second, _ := ledger.Ban(
netblock,
first.Expires,
bans.Grounds{Cause: bans.CauseAttack},
)
if second.Expires.Sub(second.Start) != 7*day { if second.Expires.Sub(second.Start) != 7*day {
t.Fatalf("the first ban for an attack lasts %s, want 7 days", t.Fatalf("the first ban for an attack lasts %s, want 7 days",
@@ -467,10 +435,10 @@ func TestAttackAfterAnAttackBanHasEndedBansPermanently(t *testing.T) {
// Its notes show the earlier ban for an attack that makes it permanent, // Its notes show the earlier ban for an attack that makes it permanent,
// beside the one for a limit. // beside the one for a limit.
third, _ := ledger.Ban( third, _ := ledger.BanForAttack(
netblock, netblock,
second.Expires.Add(30*day), second.Expires.Add(30*day),
bans.Grounds{Cause: bans.CauseAttack}, bans.Notes{},
) )
if !third.Permanent() || if !third.Permanent() ||
third.Notes.EarlierBans != (bans.EarlierBans{Limit: 1, Attack: 1}) { third.Notes.EarlierBans != (bans.EarlierBans{Limit: 1, Attack: 1}) {
@@ -484,18 +452,14 @@ func TestWouldBanGivesTheBanWithoutMakingIt(t *testing.T) {
ledger := bans.New(defaultParams()) ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
first, _ := ledger.Ban( first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
wantChanged(t, ledger, true) wantChanged(t, ledger, true)
// While the first ban lasts, none would be made. // While the first ban lasts, none would be made.
during, would := ledger.WouldBan( during, would := ledger.WouldBanForAttack(
netblock, netblock,
midnight(), midnight(),
bans.Grounds{Cause: bans.CauseAttack}, bans.Notes{},
) )
if would || !reflect.DeepEqual(during, first) { if would || !reflect.DeepEqual(during, first) {
t.Errorf( t.Errorf(
@@ -508,19 +472,13 @@ func TestWouldBanGivesTheBanWithoutMakingIt(t *testing.T) {
// As it ends, a clear sign of attack would ban for seven days, and a // As it ends, a clear sign of attack would ban for seven days, and a
// limit broken again for three hours, but neither is made. // limit broken again for three hours, but neither is made.
limitGrounds := bans.Grounds{ limitNotes := bans.Notes{Kind: "requests", Limit: 1, Window: "minute"}
Cause: bans.CauseLimit, attack, wouldAttack := ledger.WouldBanForAttack(netblock, first.Expires,
Notes: bans.Notes{Kind: "requests", Limit: 1, Window: "minute"}, bans.Notes{RuleID: "git-dir"})
} limit, wouldLimit := ledger.WouldBanForLimit(
attack, wouldAttack := ledger.WouldBan(netblock, first.Expires,
bans.Grounds{
Cause: bans.CauseAttack,
Notes: bans.Notes{RuleID: "git-dir"},
})
limit, wouldLimit := ledger.WouldBan(
netblock, netblock,
first.Expires, first.Expires,
limitGrounds, limitNotes,
) )
if !wouldAttack || !attack.Expires.Equal(first.Expires.Add(7*day)) || if !wouldAttack || !attack.Expires.Equal(first.Expires.Add(7*day)) ||
@@ -547,7 +505,7 @@ func TestWouldBanGivesTheBanWithoutMakingIt(t *testing.T) {
wantChanged(t, ledger, false) wantChanged(t, ledger, false)
// The ban made is the one that would have been. // The ban made is the one that would have been.
made, _ := ledger.Ban(netblock, first.Expires, limitGrounds) made, _ := ledger.BanForLimit(netblock, first.Expires, limitNotes)
if !reflect.DeepEqual(made, limit) { if !reflect.DeepEqual(made, limit) {
t.Errorf("the ban made is %+v, want %+v", made, limit) t.Errorf("the ban made is %+v, want %+v", made, limit)
} }
@@ -568,11 +526,7 @@ func TestWouldBePermanentAnswersAsTheBanWouldBeMade(t *testing.T) {
t.Fatalf("ban %d for a limit would be permanent", i+1) t.Fatalf("ban %d for a limit would be permanent", i+1)
} }
ban, _ := ledger.Ban( ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
netblock,
now,
bans.Grounds{Cause: bans.CauseLimit},
)
now = ban.Expires now = ban.Expires
} }
@@ -585,11 +539,7 @@ func TestWouldBePermanentAnswersAsTheBanWouldBeMade(t *testing.T) {
} }
// Once a first ban for an attack has ended, the next would be permanent. // Once a first ban for an attack has ended, the next would be permanent.
attack, _ := ledger.Ban( attack, _ := ledger.BanForAttack(netblock, now, bans.Notes{})
netblock,
now,
bans.Grounds{Cause: bans.CauseAttack},
)
if !ledger.WouldBePermanent(netblock, attack.Expires, bans.CauseAttack) { if !ledger.WouldBePermanent(netblock, attack.Expires, bans.CauseAttack) {
t.Error("a second ban for an attack would not be permanent") t.Error("a second ban for an attack would not be permanent")
} }
@@ -603,16 +553,8 @@ func TestAttackBanDoesNotLengthenTheNextBanForALimit(t *testing.T) {
// Three times the seven days would be permanent; a limit broken as the // Three times the seven days would be permanent; a limit broken as the
// ban for an attack ends bans for an hour, as a first broken limit does. // ban for an attack ends bans for an hour, as a first broken limit does.
attack, _ := ledger.Ban( attack, _ := ledger.BanForAttack(netblock, midnight(), bans.Notes{})
netblock, limit, _ := ledger.BanForLimit(netblock, attack.Expires, bans.Notes{})
midnight(),
bans.Grounds{Cause: bans.CauseAttack},
)
limit, _ := ledger.Ban(
netblock,
attack.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
if limit.Expires.Sub(limit.Start) != time.Hour || if limit.Expires.Sub(limit.Start) != time.Hour ||
limit.Cause != bans.CauseLimit { limit.Cause != bans.CauseLimit {
@@ -636,13 +578,10 @@ func TestRequestTextsAreCutTo256Bytes(t *testing.T) {
Time: midnight(), Method: long, Host: long, Path: long, Status: 403, UserAgent: long, Time: midnight(), Method: long, Host: long, Path: long, Status: 403, UserAgent: long,
} }
ban, _ := ledger.Ban( ban, _ := ledger.BanForLimit(
netblock, netblock,
midnight(), midnight(),
bans.Grounds{ bans.Notes{Request: request},
Cause: bans.CauseLimit,
Notes: bans.Notes{Request: request},
},
) )
cut := long[:256] cut := long[:256]
@@ -668,20 +607,11 @@ func TestLookupFillsTheNotesOfTheNetblocksBansWithoutOne(t *testing.T) {
// A ban made with the client's lookup, one made before it came, after // A ban made with the client's lookup, one made before it came, after
// the first ended, and one on another netblock. // the first ended, and one on another netblock.
ledger.Ban(netblock, midnight(), bans.Grounds{ ledger.BanForLimit(netblock, midnight(), bans.Notes{
Cause: bans.CauseLimit, ASN: "AS64497", ASName: "Other Net", Country: "FR",
Notes: bans.Notes{
ASN: "AS64497",
ASName: "Other Net",
Country: "FR",
},
}) })
ledger.Ban( ledger.BanForLimit(netblock, midnight().Add(time.Hour), bans.Notes{})
netblock, ledger.BanForLimit(other, midnight(), bans.Notes{})
midnight().Add(time.Hour),
bans.Grounds{Cause: bans.CauseLimit},
)
ledger.Ban(other, midnight(), bans.Grounds{Cause: bans.CauseLimit})
ledger.AddLookup(netblock, "AS64496", "Example Net", "DE") ledger.AddLookup(netblock, "AS64496", "Example Net", "DE")
+21 -36
View File
@@ -19,29 +19,25 @@ func TestChangedAfterABanIsMade(t *testing.T) {
wantChanged(t, ledger, false) wantChanged(t, ledger, false)
ledger.Ban(netblock, midnight(), bans.Grounds{Cause: bans.CauseLimit}) ledger.BanForLimit(netblock, midnight(), bans.Notes{})
wantChanged(t, ledger, true) wantChanged(t, ledger, true)
// A limit broken during the ban makes no other, and a refusal changes // A limit broken during the ban makes no other, and a refusal changes
// only the counts in the notes, which wait for the interval's write. // only the counts in the notes, which wait for the interval's write.
ledger.Ban( ledger.BanForLimit(netblock, midnight().Add(time.Minute), bans.Notes{})
netblock,
midnight().Add(time.Minute),
bans.Grounds{Cause: bans.CauseLimit},
)
ledger.Check(netblock.Addr(), midnight().Add(time.Minute)) ledger.Check(netblock.Addr(), midnight().Add(time.Minute))
wantChanged(t, ledger, false) wantChanged(t, ledger, false)
// Two bans before the value is read leave one. // Two bans before the value is read leave one.
ledger.Ban( ledger.BanForLimit(
netip.MustParsePrefix("203.0.113.10/32"), netip.MustParsePrefix("203.0.113.10/32"),
midnight(), midnight(),
bans.Grounds{Cause: bans.CauseLimit}, bans.Notes{},
) )
ledger.Ban( ledger.BanForLimit(
netip.MustParsePrefix("203.0.113.11/32"), netip.MustParsePrefix("203.0.113.11/32"),
midnight(), midnight(),
bans.Grounds{Cause: bans.CauseLimit}, bans.Notes{},
) )
wantChanged(t, ledger, true) wantChanged(t, ledger, true)
wantChanged(t, ledger, false) wantChanged(t, ledger, false)
@@ -55,14 +51,10 @@ func TestSnapshotListsEveryBanByNetblock(t *testing.T) {
high := netip.MustParsePrefix("203.0.113.10/32") high := netip.MustParsePrefix("203.0.113.10/32")
low := netip.MustParsePrefix("203.0.113.9/32") low := netip.MustParsePrefix("203.0.113.9/32")
first, _ := ledger.Ban( first, _ := ledger.BanForLimit(v6, midnight(), bans.Notes{})
v6, ledger.BanForLimit(high, midnight(), bans.Notes{})
midnight(), ledger.BanForLimit(low, midnight(), bans.Notes{})
bans.Grounds{Cause: bans.CauseLimit}, ledger.BanForLimit(v6, first.Expires, bans.Notes{})
)
ledger.Ban(high, midnight(), bans.Grounds{Cause: bans.CauseLimit})
ledger.Ban(low, midnight(), bans.Grounds{Cause: bans.CauseLimit})
ledger.Ban(v6, first.Expires, bans.Grounds{Cause: bans.CauseLimit})
snapshot := ledger.Snapshot() snapshot := ledger.Snapshot()
@@ -88,10 +80,7 @@ func TestLoadedBansCarryOn(t *testing.T) {
before := bans.New(defaultParams()) before := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32") netblock := netip.MustParsePrefix("203.0.113.9/32")
ban, _ := before.Ban(netblock, midnight(), bans.Grounds{ ban, _ := before.BanForLimit(netblock, midnight(), bans.Notes{Limit: 1})
Cause: bans.CauseLimit,
Notes: bans.Notes{Limit: 1},
})
// Loaded into a new ledger, as across a restart, the ban still refuses // Loaded into a new ledger, as across a restart, the ban still refuses
// while it lasts, and once it has ended a broken limit bans for three // while it lasts, and once it has ended a broken limit bans for three
@@ -107,11 +96,7 @@ func TestLoadedBansCarryOn(t *testing.T) {
t.Error("the loaded ban does not refuse") t.Error("the loaded ban does not refuse")
} }
again, _ := after.Ban( again, _ := after.BanForLimit(netblock, ban.Expires, bans.Notes{})
netblock,
ban.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
if again.Expires.Sub(again.Start) != 3*time.Hour || if again.Expires.Sub(again.Start) != 3*time.Hour ||
again.Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) { again.Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) {
t.Errorf( t.Errorf(
@@ -139,10 +124,10 @@ func TestLoadedBanRefusesEveryClientInItsNetblock(t *testing.T) {
Start: midnight(), Start: midnight(),
}, },
}) })
ledger.Ban( ledger.BanForLimit(
netip.MustParsePrefix("198.51.100.7/32"), netip.MustParsePrefix("198.51.100.7/32"),
midnight(), midnight(),
bans.Grounds{Cause: bans.CauseLimit}, bans.Notes{},
) )
for client, want := range map[string]bool{ for client, want := range map[string]bool{
@@ -215,7 +200,7 @@ func TestPermanentBanStartedBeforeAnEndedOneRefuses(t *testing.T) {
} }
// A limit broken now makes no shorter ban over the permanent one. // A limit broken now makes no shorter ban over the permanent one.
ban, _ = ledger.Ban(netblock, now, bans.Grounds{Cause: bans.CauseLimit}) ban, _ = ledger.BanForLimit(netblock, now, bans.Notes{})
if !ban.Permanent() || len(ledger.Bans(netblock)) != 2 { if !ban.Permanent() || len(ledger.Bans(netblock)) != 2 {
t.Errorf( t.Errorf(
"a broken limit returned %+v and left the netblock %d bans, "+ "a broken limit returned %+v and left the netblock %d bans, "+
@@ -251,10 +236,10 @@ func TestNextBanWorkedOutFromTheBanThatEndedLast(t *testing.T) {
// Once both have ended, a limit broken within the repeat window bans // Once both have ended, a limit broken within the repeat window bans
// for three times the 9 hours, and the notes count the two bans // for three times the 9 hours, and the notes count the two bans
// before the 9-hour one and it, for a limit, and the admin's. // before the 9-hour one and it, for a limit, and the admin's.
ban, _ := ledger.Ban( ban, _ := ledger.BanForLimit(
netblock, netblock,
nineHours.Expires.Add(time.Hour), nineHours.Expires.Add(time.Hour),
bans.Grounds{Cause: bans.CauseLimit}, bans.Notes{},
) )
if ban.Expires.Sub(ban.Start) != 27*time.Hour || if ban.Expires.Sub(ban.Start) != 27*time.Hour ||
ban.Notes.EarlierBans != (bans.EarlierBans{Limit: 3, Admin: 1}) { ban.Notes.EarlierBans != (bans.EarlierBans{Limit: 3, Admin: 1}) {
@@ -328,15 +313,15 @@ func TestLoadReplacesTheBansHeld(t *testing.T) {
} }
// The ledger holds one ban, so it makes two more without dropping any. // The ledger holds one ban, so it makes two more without dropping any.
first, _ := ledger.Ban( first, _ := ledger.BanForLimit(
netip.MustParsePrefix("198.51.100.7/32"), netip.MustParsePrefix("198.51.100.7/32"),
midnight(), midnight(),
bans.Grounds{Cause: bans.CauseLimit}, bans.Notes{},
) )
second, _ := ledger.Ban( second, _ := ledger.BanForLimit(
netip.MustParsePrefix("198.51.100.8/32"), netip.MustParsePrefix("198.51.100.8/32"),
midnight(), midnight(),
bans.Grounds{Cause: bans.CauseLimit}, bans.Notes{},
) )
want := []bans.Ban{first, second, kept} want := []bans.Ban{first, second, kept}
-90
View File
@@ -1,90 +0,0 @@
// Package dirwatch watches a directory, and calls a function once the
// directory has gone a short time without a change after one. The rule
// files and the lookup database are read again this way.
package dirwatch
import (
"context"
"time"
"github.com/fsnotify/fsnotify"
)
// quietTime is how long the directory must go without a change before the
// function is called, so that a file still being written, such as one
// saved in place, appended to or copied in with scp, is read only once
// whole.
const quietTime = 2 * time.Second
// Watcher watches one directory. The directory is watched, rather than a
// file in it, so that a file replaced by a rename is seen.
type Watcher struct {
watcher *fsnotify.Watcher
}
// New starts watching dir, or returns why it cannot.
func New(dir string) (*Watcher, error) {
watcher, err := fsnotify.NewWatcher()
if err != nil {
return nil, err
}
err = watcher.Add(dir)
if err != nil {
_ = watcher.Close()
return nil, err
}
return &Watcher{watcher: watcher}, nil
}
// Close stops watching the directory.
func (w *Watcher) Close() error {
return w.watcher.Close()
}
// CallWhenQuiet calls changed each time the directory has gone quietTime
// without a change, after a change to a path for which counts is true, until
// ctx is done. It also calls failed with each error the watch reports. The
// wait starts at once, as if for a change, so that a change made before the
// watch began is taken in too.
func (w *Watcher) CallWhenQuiet(
ctx context.Context,
counts func(path string) bool,
changed func(),
failed func(error),
) {
callWhenQuiet(
ctx, w.watcher.Events, w.watcher.Errors, counts, changed, failed,
)
}
// callWhenQuiet is CallWhenQuiet, for the changes from events and the
// errors from errs.
func callWhenQuiet(
ctx context.Context,
events <-chan fsnotify.Event,
errs <-chan error,
counts func(path string) bool,
changed func(),
failed func(error),
) {
quiet := time.NewTimer(quietTime)
defer quiet.Stop()
for {
select {
case <-ctx.Done():
return
case event := <-events:
if counts(event.Name) {
quiet.Reset(quietTime)
}
case <-quiet.C:
changed()
case err := <-errs:
failed(err)
}
}
}
-167
View File
@@ -1,167 +0,0 @@
package dirwatch
import (
"context"
"errors"
"path/filepath"
"sync/atomic"
"testing"
"testing/synctest"
"time"
"github.com/fsnotify/fsnotify"
)
// errWatch is an error the watch reports.
var errWatch = errors.New("too many changes")
func TestDirectoryThatCannotBeWatchedIsAnError(t *testing.T) {
t.Parallel()
_, err := New(filepath.Join(t.TempDir(), "missing"))
if err == nil {
t.Error("a missing directory is watched, want an error")
}
}
// The tests below run callWhenQuiet in a synctest bubble, where time is a
// clock of the test's own: time.Sleep moves it on at once, and
// synctest.Wait returns once callWhenQuiet waits again, so that every call
// due by then is made. The test sends the changes itself, as the watch of
// a directory cannot run in a bubble.
func TestSecondChangeStartsTheWaitAgain(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
loop := start(t, everyPath)
loop.changes <- fsnotify.Event{Name: "50-app.rules", Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
loop.wantCalls(t, 0)
// The second part starts the wait again.
loop.changes <- fsnotify.Event{Name: "50-app.rules", Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
loop.wantCalls(t, 0)
time.Sleep(time.Nanosecond)
synctest.Wait()
loop.wantCalls(t, 1)
})
}
func TestCalledOnceQuietAfterTheStart(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
// An edit saved before the watch began sends no change, and is
// taken in by this call.
loop := start(t, everyPath)
time.Sleep(quietTime)
synctest.Wait()
loop.wantCalls(t, 1)
// Long after, it has been called only once.
time.Sleep(time.Hour)
synctest.Wait()
loop.wantCalls(t, 1)
})
}
func TestChangeThatDoesNotCountIsNoChange(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
loop := start(t, func(path string) bool {
return path == "ipinfo_lite.mmdb"
})
// The wait that starts with the watch ends with a call.
time.Sleep(quietTime)
synctest.Wait()
loop.changes <- fsnotify.Event{Name: "other.mmdb", Op: fsnotify.Create}
time.Sleep(quietTime)
synctest.Wait()
loop.wantCalls(t, 1)
loop.changes <- fsnotify.Event{Name: "ipinfo_lite.mmdb", Op: fsnotify.Write}
time.Sleep(quietTime)
synctest.Wait()
loop.wantCalls(t, 2)
})
}
func TestErrorsOfTheWatchHandedOn(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
loop := start(t, everyPath)
loop.errs <- errWatch
synctest.Wait()
if !errors.Is(loop.failure, errWatch) {
t.Errorf("failed with %v, want %v", loop.failure, errWatch)
}
})
}
// loop is a callWhenQuiet that a test runs.
type loop struct {
// changes and errs send it changes and errors.
changes chan fsnotify.Event
errs chan error
// calls counts its calls of changed, and failure is the last error it
// handed to failed.
calls atomic.Int32
failure error
}
// start runs callWhenQuiet, with counts, until the test ends.
func start(t *testing.T, counts func(path string) bool) *loop {
t.Helper()
l := &loop{changes: make(chan fsnotify.Event), errs: make(chan error)}
ctx, stop := context.WithCancel(t.Context())
stopped := make(chan struct{})
go func() {
callWhenQuiet(ctx, l.changes, l.errs, counts,
func() { l.calls.Add(1) },
func(err error) { l.failure = err },
)
close(stopped)
}()
t.Cleanup(func() {
stop()
<-stopped
})
return l
}
// everyPath counts a change to any path.
func everyPath(string) bool {
return true
}
// wantCalls checks how many times changed has been called.
func (l *loop) wantCalls(t *testing.T, want int32) {
t.Helper()
got := l.calls.Load()
if got != want {
t.Errorf("changed called %d times, want %d", got, want)
}
}
+24 -14
View File
@@ -84,7 +84,12 @@ type Params struct {
// one request to GeoJS is under way at a time, and it asks about every // one request to GeoJS is under way at a time, and it asks about every
// client waiting, up to maxPerRequest. It is safe for concurrent use. // client waiting, up to maxPerRequest. It is safe for concurrent use.
type GeoJS struct { type GeoJS struct {
params Params url string
answered func(Answer)
now func() time.Time
asked func()
failed func(err error, askingAgainIn time.Duration)
wentWithoutAnswer func()
// httpClient follows no redirect, so that visitors' addresses go to // httpClient follows no redirect, so that visitors' addresses go to
// GeoJS alone: a redirect is a failure. // GeoJS alone: a redirect is a failure.
httpClient *http.Client httpClient *http.Client
@@ -140,7 +145,12 @@ func New(params Params) *GeoJS {
} }
return &GeoJS{ return &GeoJS{
params: params, url: params.URL,
answered: params.Answered,
now: params.Now,
asked: params.Asked,
failed: params.Failed,
wentWithoutAnswer: params.WentWithoutAnswer,
httpClient: &http.Client{ httpClient: &http.Client{
CheckRedirect: func(*http.Request, []*http.Request) error { CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse return http.ErrUseLastResponse
@@ -183,7 +193,7 @@ func (g *GeoJS) LookUp(
answer, found := g.kept(client) answer, found := g.kept(client)
if !found { if !found {
g.params.WentWithoutAnswer() g.wentWithoutAnswer()
} }
w, waiting := g.waiting[client] w, waiting := g.waiting[client]
@@ -263,7 +273,7 @@ func (g *GeoJS) Load(answers []Answer) {
g.answers.Purge() g.answers.Purge()
now := g.params.Now() now := g.now()
for _, answer := range answers { for _, answer := range answers {
if now.Sub(answer.Answered) < keepFor { if now.Sub(answer.Answered) < keepFor {
@@ -300,7 +310,7 @@ func (g *GeoJS) answerOrWait(
} }
if w == nil { if w == nil {
g.params.WentWithoutAnswer() g.wentWithoutAnswer()
return Answer{}, nil // too many clients wait already return Answer{}, nil // too many clients wait already
} }
@@ -312,7 +322,7 @@ func (g *GeoJS) answerOrWait(
} }
if w.late { if w.late {
g.params.WentWithoutAnswer() g.wentWithoutAnswer()
return Answer{}, nil return Answer{}, nil
} }
@@ -323,7 +333,7 @@ func (g *GeoJS) answerOrWait(
// kept returns client's answer, if GeoJS gave it less than keepFor ago, // kept returns client's answer, if GeoJS gave it less than keepFor ago,
// and notes that it was used. // and notes that it was used.
func (g *GeoJS) kept(client netip.Prefix) (Answer, bool) { func (g *GeoJS) kept(client netip.Prefix) (Answer, bool) {
now := g.params.Now() now := g.now()
kept, found := g.answers.Get(client) kept, found := g.answers.Get(client)
if !found || now.Sub(kept.Answered) >= keepFor { if !found || now.Sub(kept.Answered) >= keepFor {
@@ -340,7 +350,7 @@ func (g *GeoJS) kept(client netip.Prefix) (Answer, bool) {
// been called. The requests to GeoJS are for every client waiting, so they // been called. The requests to GeoJS are for every client waiting, so they
// go on when the caller whose ctx is given stops waiting, until StopAsking. // go on when the caller whose ctx is given stops waiting, until StopAsking.
func (g *GeoJS) ask(ctx context.Context) { func (g *GeoJS) ask(ctx context.Context) {
if g.asking || g.stopped || g.params.Now().Before(g.retryAt) { if g.asking || g.stopped || g.now().Before(g.retryAt) {
return return
} }
@@ -372,9 +382,9 @@ func (g *GeoJS) askAboutWaiting(ctx context.Context) {
kept, answered := g.keep(clients, given, err) kept, answered := g.keep(clients, given, err)
if g.params.Answered != nil { if g.answered != nil {
for _, answer := range kept { for _, answer := range kept {
g.params.Answered(answer) g.answered(answer)
} }
} }
@@ -423,7 +433,7 @@ func (g *GeoJS) keep(
g.mu.Lock() g.mu.Lock()
defer g.mu.Unlock() defer g.mu.Unlock()
now := g.params.Now() now := g.now()
kept := make([]Answer, 0, len(clients)) kept := make([]Answer, 0, len(clients))
leftOut := 0 leftOut := 0
@@ -467,7 +477,7 @@ func (g *GeoJS) keep(
w.late = true w.late = true
} }
g.params.Failed(err, g.retryDelay) g.failed(err, g.retryDelay)
return kept, false return kept, false
} }
@@ -497,7 +507,7 @@ func (g *GeoJS) request(
req, err := http.NewRequestWithContext( req, err := http.NewRequestWithContext(
ctx, ctx,
http.MethodGet, http.MethodGet,
g.params.URL, g.url,
http.NoBody, http.NoBody,
) )
if err != nil { if err != nil {
@@ -506,7 +516,7 @@ func (g *GeoJS) request(
req.URL.RawQuery = "ip=" + strings.Join(addrs, ",") req.URL.RawQuery = "ip=" + strings.Join(addrs, ",")
g.params.Asked() g.asked()
res, err := g.httpClient.Do(req) res, err := g.httpClient.Do(req)
if err != nil { if err != nil {
+49 -18
View File
@@ -13,13 +13,18 @@ import (
"sync" "sync"
"time" "time"
"github.com/fsnotify/fsnotify"
"github.com/oschwald/maxminddb-golang/v2" "github.com/oschwald/maxminddb-golang/v2"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/dirwatch"
"sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/geojs"
) )
// quietTime is how long the lookup database must go without a change
// before it is read again, so that a file still being copied in is read
// only once whole.
const quietTime = 2 * time.Second
// FileParams are what OpenFile needs. // FileParams are what OpenFile needs.
type FileParams struct { type FileParams struct {
// Path is the lookup database, the IPinfo Lite file in its .mmdb form // Path is the lookup database, the IPinfo Lite file in its .mmdb form
@@ -117,13 +122,20 @@ func (f *File) ReadFailures() int {
} }
// Watch watches the directory of the lookup database until ctx is done, // Watch watches the directory of the lookup database until ctx is done,
// and reads the file again once it has gone without a change for a short // and reads the file again once it has gone without a change for
// time, after it is replaced, written or removed, and after Watch starts // quietTime, after it is replaced, written or removed, and after Watch
// watching, as dirwatch describes. A change to another file in its // starts watching. If the directory cannot be watched, that is logged, and
// directory does not count. If the directory cannot be watched, that is // the database read at start stays in use.
// logged, and the database read at start stays in use.
func (f *File) Watch(ctx context.Context) { func (f *File) Watch(ctx context.Context) {
watcher, err := dirwatch.New(filepath.Dir(f.params.Path)) watcher, err := fsnotify.NewWatcher()
if err == nil {
defer func() {
_ = watcher.Close()
}()
err = watcher.Add(filepath.Dir(f.params.Path))
}
if err != nil { if err != nil {
f.params.ProcessLog.Error( f.params.ProcessLog.Error(
"cannot watch the lookup database for replacements", "cannot watch the lookup database for replacements",
@@ -134,22 +146,41 @@ func (f *File) Watch(ctx context.Context) {
return return
} }
defer func() {
_ = watcher.Close()
}()
f.params.ProcessLog.Info("watching the lookup database for replacements", f.params.ProcessLog.Info("watching the lookup database for replacements",
"file", f.params.Path) "file", f.params.Path)
watcher.CallWhenQuiet(ctx, f.isDatabase, f.readAgain, func(err error) { f.readAfterChanges(ctx, watcher.Events, watcher.Errors)
f.params.ProcessLog.Warn("watching the lookup database failed",
"error", err.Error())
})
} }
// isDatabase reports whether path names the lookup database. // readAfterChanges reads the lookup database again once quietTime has
func (f *File) isDatabase(path string) bool { // passed without a change to it from events, until ctx is done, and logs
return filepath.Clean(path) == filepath.Clean(f.params.Path) // the errors from errs. A change to another file in its directory does not
// count. The wait starts at once, as if for a change, so that a file
// replaced after OpenFile read it, and before its directory was watched,
// is read too.
func (f *File) readAfterChanges(
ctx context.Context, events <-chan fsnotify.Event, errs <-chan error,
) {
path := filepath.Clean(f.params.Path)
quiet := time.NewTimer(quietTime)
defer quiet.Stop()
for {
select {
case <-ctx.Done():
return
case event := <-events:
if filepath.Clean(event.Name) == path {
quiet.Reset(quietTime)
}
case <-quiet.C:
f.readAgain()
case err := <-errs:
f.params.ProcessLog.Warn("watching the lookup database failed",
"error", err.Error())
}
}
} }
// readAgain reads the lookup database again, in place of the one in use, // readAgain reads the lookup database again, in place of the one in use,
+141 -35
View File
@@ -1,6 +1,7 @@
package lookup package lookup
import ( import (
"context"
"log/slog" "log/slog"
"net/netip" "net/netip"
"net/url" "net/url"
@@ -11,6 +12,7 @@ import (
"testing/synctest" "testing/synctest"
"time" "time"
"github.com/fsnotify/fsnotify"
"github.com/maxmind/mmdbwriter/mmdbtype" "github.com/maxmind/mmdbwriter/mmdbtype"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
@@ -131,48 +133,66 @@ func TestFileThatCannotBeReadIsAnError(t *testing.T) {
} }
} }
func TestChangeOfAnotherFileInTheDirectoryIsNoReplacement(t *testing.T) { // The tests below run readAfterChanges in a synctest bubble, where time is
// a clock of the test's own: time.Sleep moves it on at once, and
// synctest.Wait returns once readAfterChanges waits again, so that every
// reading due by then is done. The test sends the changes itself, as the
// watch of a directory cannot run in a bubble.
func TestReplacementCopiedOverTheFileInTwoPartsIsReadOnlyWhole(
t *testing.T,
) {
t.Parallel() t.Parallel()
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir() dir := t.TempDir()
path := filepath.Join(dir, "ipinfo_lite.mmdb") path := filepath.Join(dir, "ipinfo_lite.mmdb")
writeDatabase(t, path, "DE") writeDatabase(t, path, "DE")
f := openFile(t, path, newQueue())
for changed, want := range map[string]bool{
path: true,
dir + "/./ipinfo_lite.mmdb": true,
filepath.Join(dir, "other.mmdb"): false,
} {
if f.isDatabase(changed) != want {
t.Errorf(
"a change to %s counts: %t, want %t",
changed,
!want,
want,
)
}
}
}
// The tests below run in a synctest bubble, where time is a clock of the
// test's own, so that the time a file is read can be checked. The wait for
// the directory to go quiet before a reading is tested in dirwatch.
func TestReplacementIsReadAgain(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb")
writeDatabase(t, path, "DE")
queue := newQueue() queue := newQueue()
f := openFile(t, path, queue) f := openFile(t, path, queue)
changes := watch(t, f)
writeDatabase(t, path, "KP") other := filepath.Join(dir, "replacement.mmdb")
time.Sleep(time.Hour) writeDatabase(t, other, "KP")
f.readAgain()
//nolint:gosec // a file the test wrote
replacement, err := os.ReadFile(other)
if err != nil {
t.Fatalf("read %s: %v", other, err)
}
// The file in use is overwritten in place, and keeps giving what
// it gave. Its first part alone is not a .mmdb file.
file, err := os.Create(path) //nolint:gosec // a file the test wrote
if err != nil {
t.Fatalf("create %s: %v", path, err)
}
defer func() {
_ = file.Close()
}()
half := len(replacement) / 2
write(t, file, replacement[:half])
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantCountry(t, f, "DE")
// The second part starts the wait again.
write(t, file, replacement[half:])
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantCountry(t, f, "DE")
time.Sleep(time.Nanosecond)
synctest.Wait()
wantCountry(t, f, "KP") wantCountry(t, f, "KP")
if !f.LastRead().Equal(time.Now()) || f.ReadFailures() != 0 { if !f.LastRead().Equal(time.Now()) || f.ReadFailures() != 0 {
@@ -199,10 +219,15 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(
queue := newQueue() queue := newQueue()
f := openFile(t, path, queue) f := openFile(t, path, queue)
read := f.LastRead() read := f.LastRead()
changes := watch(t, f)
writeFile(t, path, "not a lookup database\n") writeFile(t, path, "not a lookup database\n")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
// Long after, the replacement has been read once.
time.Sleep(time.Hour) time.Sleep(time.Hour)
f.readAgain() synctest.Wait()
wantCountry(t, f, "DE") wantCountry(t, f, "DE")
if !f.LastRead().Equal(read) || f.ReadFailures() != 1 { if !f.LastRead().Equal(read) || f.ReadFailures() != 1 {
@@ -215,7 +240,7 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(
} }
wantAlerts(t, queue, alerts.Alert{ wantAlerts(t, queue, alerts.Alert{
Time: read.Add(time.Hour), Time: read.Add(quietTime),
Event: alerts.EventFileError, Event: alerts.EventFileError,
Reason: "the lookup database cannot be read, and the one read before stays in use", Reason: "the lookup database cannot be read, and the one read before stays in use",
Detail: map[string]any{ Detail: map[string]any{
@@ -227,6 +252,55 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(
}) })
} }
func TestChangeOfAnotherFileInTheDirectoryIsNoReplacement(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "ipinfo_lite.mmdb")
writeDatabase(t, path, "DE")
f := openFile(t, path, newQueue())
changes := watch(t, f)
// The wait that starts with the watch ends with a reading.
time.Sleep(quietTime)
synctest.Wait()
writeDatabase(t, path, "KP")
changes <- fsnotify.Event{Name: filepath.Join(dir, "other.mmdb"), Op: fsnotify.Create}
time.Sleep(quietTime)
synctest.Wait()
wantCountry(t, f, "DE")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime)
synctest.Wait()
wantCountry(t, f, "KP")
})
}
func TestReplacementSavedBeforeTheWatchStartsIsRead(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb")
writeDatabase(t, path, "DE")
f := openFile(t, path, newQueue())
// Saved after OpenFile read the file, and before its directory was
// watched, so that no change is seen for it.
writeDatabase(t, path, "KP")
watch(t, f)
time.Sleep(quietTime)
synctest.Wait()
wantCountry(t, f, "KP")
})
}
// newQueue returns a queue of alerts for a webhook that is never sent // newQueue returns a queue of alerts for a webhook that is never sent
// them, so that they wait in it for the test to look at. // them, so that they wait in it for the test to look at.
func newQueue() *alerts.Queue { func newQueue() *alerts.Queue {
@@ -270,6 +344,28 @@ func openFile(t *testing.T, path string, queue *alerts.Queue) *File {
return f return f
} }
// watch runs f's readAfterChanges until the test ends, and returns the
// channel that sends it changes.
func watch(t *testing.T, f *File) chan<- fsnotify.Event {
t.Helper()
changes := make(chan fsnotify.Event)
ctx, stop := context.WithCancel(t.Context())
stopped := make(chan struct{})
go func() {
f.readAfterChanges(ctx, changes, nil)
close(stopped)
}()
t.Cleanup(func() {
stop()
<-stopped
})
return changes
}
// wantCountry checks the country f gives testClient. // wantCountry checks the country f gives testClient.
func wantCountry(t *testing.T, f *File, want string) { func wantCountry(t *testing.T, f *File, want string) {
t.Helper() t.Helper()
@@ -305,3 +401,13 @@ func writeFile(t *testing.T, path, content string) {
t.Fatalf("write %s: %v", path, err) t.Fatalf("write %s: %v", path, err)
} }
} }
// write writes data to the end of file.
func write(t *testing.T, file *os.File, data []byte) {
t.Helper()
_, err := file.Write(data)
if err != nil {
t.Fatalf("write: %v", err)
}
}
+264 -389
View File
@@ -15,33 +15,12 @@ import (
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/remotelog"
"sneak.berlin/go/smallwebwaf/internal/reputation" "sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
) )
// Params are what New builds the metrics from. Each part is nil while it is
// off, and a part that is nil adds no metric of its own.
type Params struct {
// BusiestCount is how many countries and how many AS numbers get series
// of their own (SWW_WAF_METRICS_BUSIEST_COUNTRIES_AND_ASNS).
BusiestCount int
// InstanceName is the label instance every metric carries
// (SWW_WAF_INSTANCE_NAME).
InstanceName string
// Now returns the time at which Bans counts the bans active.
Now func() time.Time
Bans banLedger
Clients sized
Rules sized
Lists fetchedLists
DNSBL dnsblZones
AbuseIPDB abuseIPDBChecker
Alerts alertQueue
LookupFile lookupDatabase
RemoteLog remoteLogSender
}
// Metrics are sww's metrics. They are safe for concurrent use. // Metrics are sww's metrics. They are safe for concurrent use.
type Metrics struct { type Metrics struct {
// registry gives every metric registered with it the label instance. // registry gives every metric registered with it the label instance.
@@ -57,6 +36,8 @@ type Metrics struct {
rateLimitHits *prometheus.CounterVec rateLimitHits *prometheus.CounterVec
offences *prometheus.CounterVec offences *prometheus.CounterVec
coreRuleSetMatches *prometheus.CounterVec coreRuleSetMatches *prometheus.CounterVec
// ruleMatches are made by AddRules, and reputationHits by
// AddReputation.
ruleMatches *prometheus.CounterVec ruleMatches *prometheus.CounterVec
reputationHits *prometheus.CounterVec reputationHits *prometheus.CounterVec
countries *busiest countries *busiest
@@ -78,18 +59,20 @@ type Metrics struct {
stateFileEditsSetAside *prometheus.CounterVec stateFileEditsSetAside *prometheus.CounterVec
} }
// New returns the metrics, with the Go runtime's and the process's own, and // New returns the metrics, with the Go runtime's and the process's own.
// those read from each part in params as the metrics are asked for. // busiestCount is how many countries and how many AS numbers get series of
// their own (SWW_WAF_METRICS_BUSIEST_COUNTRIES_AND_ASNS). Every metric
// carries instanceName (SWW_WAF_INSTANCE_NAME) as its label instance.
// //
//nolint:funlen // a few lines for each metric, a list that grows with them //nolint:funlen // a few lines for each metric, a list that grows with them
func New(params Params) *Metrics { func New(busiestCount int, instanceName string) *Metrics {
byStatus := []string{"status_class", "action"} byStatus := []string{"status_class", "action"}
byFile := []string{"file"} byFile := []string{"file"}
registry := prometheus.NewRegistry() registry := prometheus.NewRegistry()
m := &Metrics{ m := &Metrics{
registry: prometheus.WrapRegistererWith( registry: prometheus.WrapRegistererWith(
prometheus.Labels{"instance": params.InstanceName}, registry), prometheus.Labels{"instance": instanceName}, registry),
handler: promhttp.HandlerFor(registry, promhttp.HandlerOpts{}), handler: promhttp.HandlerFor(registry, promhttp.HandlerOpts{}),
inFlight: prometheus.NewGauge(prometheus.GaugeOpts{ inFlight: prometheus.NewGauge(prometheus.GaugeOpts{
Name: "sww_requests_in_flight", Name: "sww_requests_in_flight",
@@ -130,19 +113,8 @@ func New(params Params) *Metrics {
"and the rule's id.", "and the rule's id.",
[]string{"mode", "rule_id"}, []string{"mode", "rule_id"},
), ),
ruleMatches: counterVec( countries: newCountries(busiestCount),
"sww_rule_matches_total", asns: newASNs(busiestCount),
"Requests that matched a rule of the rule files, by its id and action.",
[]string{"rule_id", "action"},
),
reputationHits: counterVec(
"sww_reputation_hits_total",
"Requests whose client a blocklist, a DNSBL zone or AbuseIPDB lists, "+
"by the list's URL, the zone, or abuseipdb.",
[]string{sourceLabel},
),
countries: newCountries(params.BusiestCount),
asns: newASNs(params.BusiestCount),
GeoJSRequests: prometheus.NewCounter(prometheus.CounterOpts{ GeoJSRequests: prometheus.NewCounter(prometheus.CounterOpts{
Name: "sww_geojs_requests_total", Name: "sww_geojs_requests_total",
Help: "Requests to GeoJS.", Help: "Requests to GeoJS.",
@@ -194,7 +166,6 @@ func New(params Params) *Metrics {
m.inFlight, m.requests, m.requestBytes, m.responseBytes, m.inFlight, m.requests, m.requestBytes, m.responseBytes,
m.requestDuration, m.upstreamDuration, m.requestDuration, m.upstreamDuration,
m.rateLimitHits, m.offences, m.coreRuleSetMatches, m.rateLimitHits, m.offences, m.coreRuleSetMatches,
m.ruleMatches, m.reputationHits,
m.countries, m.asns, m.countries, m.asns,
m.GeoJSRequests, m.GeoJSFailures, m.RequestsWithoutGeoJSAnswer, m.GeoJSRequests, m.GeoJSFailures, m.RequestsWithoutGeoJSAnswer,
m.stateFileWrites, m.stateFileWriteFailures, m.stateFileWrites, m.stateFileWriteFailures,
@@ -202,45 +173,261 @@ func New(params Params) *Metrics {
m.stateFileEditsTakenIn, m.stateFileEditsSetAside, m.stateFileEditsTakenIn, m.stateFileEditsSetAside,
) )
if params.Bans != nil {
m.addBans(params.Bans, params.Now)
}
if params.Clients != nil {
m.addClients(params.Clients)
}
if params.Rules != nil {
m.addRules(params.Rules)
}
if params.Lists != nil {
m.addLists(params.Lists)
}
if params.DNSBL != nil {
m.addDNSBL(params.DNSBL)
}
if params.AbuseIPDB != nil {
m.addAbuseIPDB(params.AbuseIPDB)
}
if params.Alerts != nil {
m.addAlerts(params.Alerts)
}
if params.LookupFile != nil {
m.addLookupFile(params.LookupFile)
}
if params.RemoteLog != nil {
m.addRemoteLog(params.RemoteLog)
}
return m return m
} }
// AddBansAndClients adds the metrics read from the ledger and the table
// of clients as the metrics are asked for: the bans made since the start,
// by cause, the bans active and permanent at now, and the clients in the
// table.
func (m *Metrics) AddBansAndClients(
ledger *bans.Ledger, limiter *ratelimit.Limiter, now func() time.Time,
) {
for _, cause := range []string{
bans.CauseLimit, bans.CauseAttack, bans.CauseAdmin, bans.CauseAbuseIPDB,
} {
m.registry.MustRegister(
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_bans_made_total",
Help: "Bans made, by cause.",
ConstLabels: prometheus.Labels{"cause": cause},
}, func() float64 {
return float64(ledger.Made(cause))
}),
)
}
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_active_bans",
Help: "Bans active now, the permanent ones included.",
}, func() float64 {
active, _ := ledger.Count(now())
return float64(active)
}),
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_permanent_bans",
Help: "Permanent bans not lifted.",
}, func() float64 {
_, permanent := ledger.Count(now())
return float64(permanent)
}),
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_tracked_clients",
Help: "Clients in the table of clients.",
}, func() float64 {
return float64(limiter.Len())
}),
)
}
// AddRules adds the metrics of the rule files: the requests that matched
// each rule, which RuleMatched counts, and the rules loaded from
// ruleFiles, read as the metrics are asked for. It is called once, before
// RuleMatched.
func (m *Metrics) AddRules(ruleFiles *rules.Files) {
m.ruleMatches = counterVec(
"sww_rule_matches_total",
"Requests that matched a rule of the rule files, by its id and action.",
[]string{"rule_id", "action"},
)
m.registry.MustRegister(m.ruleMatches,
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_rules_loaded",
Help: "Rules loaded from the rule files.",
}, func() float64 {
return float64(ruleFiles.Len())
}))
}
// AddRemoteLog adds the metrics of sending the log lines to
// SWW_WAF_LOG_REMOTE_URL, read from remote as the metrics are asked for: the
// lines sent, those dropped, and those waiting in the buffer.
func (m *Metrics) AddRemoteLog(remote *remotelog.Sender) {
m.registry.MustRegister(
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_remote_log_lines_sent_total",
Help: "Log lines sent to SWW_WAF_LOG_REMOTE_URL.",
}, func() float64 {
return float64(remote.Sent())
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_remote_log_lines_dropped_total",
Help: "Log lines dropped: the oldest in a full buffer, and those " +
"whose sending failed.",
}, func() float64 {
return float64(remote.Dropped())
}),
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_remote_log_lines_waiting",
Help: "Log lines in the buffer, waiting to be sent.",
}, func() float64 {
return float64(remote.Depth())
}),
)
}
// AddLookupFile adds the metrics of the lookup database, read as the
// metrics are asked for: when the file in use was read, which lastRead
// returns, and the replacements of it that could not be read, which
// readFailures returns.
func (m *Metrics) AddLookupFile(
lastRead func() time.Time,
readFailures func() int,
) {
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_lookup_database_last_read_timestamp_seconds",
Help: "When the lookup database in use was read, in seconds since 1970.",
}, func() float64 {
return float64(lastRead().Unix())
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_lookup_database_read_failures_total",
Help: "Replacements of the lookup database that could not be read.",
}, func() float64 {
return float64(readFailures())
}),
)
}
// sourceLabel is the label of the reputation metrics: a list's URL, a
// DNSBL zone, its key masked, or abuseipdb.
const sourceLabel = "source"
// AddReputation adds the metrics of the lists fetched from URLs and of the
// DNSBL zones, by source, each list's URL or each zone, its key masked as
// reputation.MaskZoneKey masks it: the requests whose client a blocklist, a
// zone's verdict or AbuseIPDB's score lists, which ReputationHit counts,
// and, read from lists and dnsbl as the metrics are asked for, for a list,
// the fetches that failed and when the copy in use was fetched, and for a
// zone, the queries made and those that failed. It is called once, before
// ReputationHit.
func (m *Metrics) AddReputation(
lists *reputation.Lists,
dnsbl *reputation.DNSBL,
) {
m.reputationHits = counterVec(
"sww_reputation_hits_total",
"Requests whose client a blocklist, a DNSBL zone or AbuseIPDB lists, "+
"by the list's URL, the zone, or abuseipdb.",
[]string{sourceLabel},
)
m.registry.MustRegister(m.reputationHits)
for _, zone := range dnsbl.Zones() {
source := prometheus.Labels{
sourceLabel: reputation.MaskZoneKey(zone),
}
m.addReputationQueries(
source,
func() int { return dnsbl.Queries(zone) },
)
m.addReputationFailures(
source,
func() int { return dnsbl.Failures(zone) },
)
}
for _, listURL := range lists.URLs() {
source := prometheus.Labels{sourceLabel: listURL}
m.addReputationFailures(
source,
func() int { return lists.Failures(listURL) },
)
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_reputation_last_fetch_timestamp_seconds",
Help: "When the copy of the list in use was fetched, in seconds since " +
"1970, or 0 while there is none.",
ConstLabels: source,
}, func() float64 {
fetched := lists.Fetched(listURL)
if fetched.IsZero() {
return 0
}
return float64(fetched.Unix())
}),
)
}
}
// AddAbuseIPDB adds the metrics of AbuseIPDB, with the source abuseipdb,
// read from abuseIPDB as the metrics are asked for: the checks made, those
// that failed, and how many checks the day's budget has left. It is
// called once, after AddReputation, while SWW_WAF_ABUSEIPDB_KEY is set.
func (m *Metrics) AddAbuseIPDB(abuseIPDB *reputation.AbuseIPDB) {
source := prometheus.Labels{sourceLabel: reputation.AbuseIPDBSource}
m.addReputationQueries(source, abuseIPDB.Checked)
m.addReputationFailures(source, abuseIPDB.Failures)
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_reputation_daily_budget_remaining",
Help: "Checks of the day's SWW_WAF_ABUSEIPDB_DAILY_BUDGET not yet spent.",
ConstLabels: source,
}, func() float64 {
return float64(abuseIPDB.BudgetLeft())
}),
)
}
// ReputationHit counts a request whose client source lists: a blocklist,
// by its URL, a DNSBL zone, its key masked, or AbuseIPDB, abuseipdb.
func (m *Metrics) ReputationHit(source string) {
m.reputationHits.WithLabelValues(source).Inc()
}
// AddAlerts adds the metrics of the alerts sent to each destination set,
// read from queue as the metrics are asked for, by destination: the
// alerts sent, the requests to the destination that failed, the alerts
// held back, which are the same for every destination, and those
// dropped. With no destination set, it adds none.
func (m *Metrics) AddAlerts(queue *alerts.Queue) {
for _, name := range queue.DestinationsSet() {
destination := prometheus.Labels{"destination": name}
m.registry.MustRegister(
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alerts_sent_total",
Help: "Alerts the destination took.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Counts(name).Sent)
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alert_send_failures_total",
Help: "Requests to the destination that failed.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Counts(name).SendFailures)
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alerts_suppressed_total",
Help: "Alerts held back: repeats within SWW_WAF_ALERT_COOLDOWN, and " +
"alerts past SWW_WAF_ALERT_MAX_PER_HOUR, for the hour's summary.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Suppressed())
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alerts_dropped_total",
Help: "Alerts dropped, the oldest first, from a full queue, and " +
"alerts given up as the destination refused them.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Counts(name).Dropped)
}),
)
}
}
// ServeHTTP answers with the metrics in the Prometheus text format. // ServeHTTP answers with the metrics in the Prometheus text format.
func (m *Metrics) ServeHTTP(w http.ResponseWriter, r *http.Request) { func (m *Metrics) ServeHTTP(w http.ResponseWriter, r *http.Request) {
m.handler.ServeHTTP(w, r) m.handler.ServeHTTP(w, r)
@@ -310,12 +497,6 @@ func (m *Metrics) CoreRuleSetMatched(mode string, id int) {
m.coreRuleSetMatches.WithLabelValues(mode, strconv.Itoa(id)).Inc() m.coreRuleSetMatches.WithLabelValues(mode, strconv.Itoa(id)).Inc()
} }
// ReputationHit counts a request whose client source lists: a blocklist,
// by its URL, a DNSBL zone, its key masked, or AbuseIPDB, abuseipdb.
func (m *Metrics) ReputationHit(source string) {
m.reputationHits.WithLabelValues(source).Inc()
}
// StateFileWritten counts a write of the state or cache file name, of size // StateFileWritten counts a write of the state or cache file name, of size
// bytes, that ended with err. // bytes, that ended with err.
func (m *Metrics) StateFileWritten(name string, size int, err error) { func (m *Metrics) StateFileWritten(name string, size int, err error) {
@@ -347,312 +528,6 @@ func (m *Metrics) StateFileEditSetAside(name string) {
m.stateFileEditsSetAside.WithLabelValues(name).Inc() m.stateFileEditsSetAside.WithLabelValues(name).Inc()
} }
// banLedger is what the metrics read from the ledger of bans.
type banLedger interface {
// Made returns the bans made since the start with cause.
Made(cause string) int
// Count returns the bans active at now, the permanent ones included,
// and the permanent ones.
Count(now time.Time) (active, permanent int)
}
// addBans adds the metrics read from ledger as the metrics are asked for:
// the bans made since the start, by cause, and the bans active and
// permanent at now.
func (m *Metrics) addBans(ledger banLedger, now func() time.Time) {
for _, cause := range []string{
bans.CauseLimit, bans.CauseAttack, bans.CauseAdmin, bans.CauseAbuseIPDB,
} {
m.registry.MustRegister(
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_bans_made_total",
Help: "Bans made, by cause.",
ConstLabels: prometheus.Labels{"cause": cause},
}, func() float64 {
return float64(ledger.Made(cause))
}),
)
}
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_active_bans",
Help: "Bans active now, the permanent ones included.",
}, func() float64 {
active, _ := ledger.Count(now())
return float64(active)
}),
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_permanent_bans",
Help: "Permanent bans not lifted.",
}, func() float64 {
_, permanent := ledger.Count(now())
return float64(permanent)
}),
)
}
// sized is what the metrics read from the table of clients and from the
// rule files.
type sized interface {
// Len returns how many clients the table holds, or how many rules were
// loaded from the rule files.
Len() int
}
// addClients adds the metric of the clients in table, read as the metrics
// are asked for.
func (m *Metrics) addClients(table sized) {
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_tracked_clients",
Help: "Clients in the table of clients.",
}, func() float64 {
return float64(table.Len())
}),
)
}
// addRules adds the metric of the rules loaded from files, read as the
// metrics are asked for.
func (m *Metrics) addRules(files sized) {
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_rules_loaded",
Help: "Rules loaded from the rule files.",
}, func() float64 {
return float64(files.Len())
}))
}
// remoteLogSender is what the metrics read from what sends the log lines to
// SWW_WAF_LOG_REMOTE_URL.
type remoteLogSender interface {
// Sent returns the lines sent.
Sent() int64
// Dropped returns the lines dropped.
Dropped() int64
// Depth returns the lines waiting in the buffer.
Depth() int
}
// addRemoteLog adds the metrics of sending the log lines to
// SWW_WAF_LOG_REMOTE_URL, read from remote as the metrics are asked for: the
// lines sent, those dropped, and those waiting in the buffer.
func (m *Metrics) addRemoteLog(remote remoteLogSender) {
m.registry.MustRegister(
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_remote_log_lines_sent_total",
Help: "Log lines sent to SWW_WAF_LOG_REMOTE_URL.",
}, func() float64 {
return float64(remote.Sent())
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_remote_log_lines_dropped_total",
Help: "Log lines dropped: the oldest in a full buffer, and those " +
"whose sending failed.",
}, func() float64 {
return float64(remote.Dropped())
}),
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_remote_log_lines_waiting",
Help: "Log lines in the buffer, waiting to be sent.",
}, func() float64 {
return float64(remote.Depth())
}),
)
}
// lookupDatabase is what the metrics read from the lookup database.
type lookupDatabase interface {
// LastRead returns when the file in use was read.
LastRead() time.Time
// ReadFailures returns the replacements of the file that could not be
// read.
ReadFailures() int
}
// addLookupFile adds the metrics of the lookup database, read from file as
// the metrics are asked for: when the file in use was read, and the
// replacements of it that could not be read.
func (m *Metrics) addLookupFile(file lookupDatabase) {
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_lookup_database_last_read_timestamp_seconds",
Help: "When the lookup database in use was read, in seconds since 1970.",
}, func() float64 {
return float64(file.LastRead().Unix())
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_lookup_database_read_failures_total",
Help: "Replacements of the lookup database that could not be read.",
}, func() float64 {
return float64(file.ReadFailures())
}),
)
}
// sourceLabel is the label of the reputation metrics: a list's URL, a
// DNSBL zone, its key masked, or abuseipdb.
const sourceLabel = "source"
// dnsblZones is what the metrics read from what asks the DNSBL zones.
type dnsblZones interface {
// Zones returns the zones asked, their keys unmasked.
Zones() []string
// Queries returns the queries made to zone.
Queries(zone string) int
// Failures returns the queries to zone that failed.
Failures(zone string) int
}
// addDNSBL adds the metrics of each zone dnsbl asks, by source, the zone,
// its key masked as reputation.MaskZoneKey masks it, read as the metrics are
// asked for: the queries made and those that failed.
func (m *Metrics) addDNSBL(dnsbl dnsblZones) {
for _, zone := range dnsbl.Zones() {
source := prometheus.Labels{
sourceLabel: reputation.MaskZoneKey(zone),
}
m.addReputationQueries(
source,
func() int { return dnsbl.Queries(zone) },
)
m.addReputationFailures(
source,
func() int { return dnsbl.Failures(zone) },
)
}
}
// fetchedLists is what the metrics read from the lists fetched from URLs.
type fetchedLists interface {
// URLs returns the lists' URLs.
URLs() []string
// Failures returns the fetches of the list at listURL that failed.
Failures(listURL string) int
// Fetched returns when the copy in use of the list at listURL was
// fetched, or the zero time while there is none.
Fetched(listURL string) time.Time
}
// addLists adds the metrics of each list fetched from a URL, by source, the
// list's URL, read from lists as the metrics are asked for: the fetches
// that failed and when the copy in use was fetched.
func (m *Metrics) addLists(lists fetchedLists) {
for _, listURL := range lists.URLs() {
source := prometheus.Labels{sourceLabel: listURL}
m.addReputationFailures(
source,
func() int { return lists.Failures(listURL) },
)
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_reputation_last_fetch_timestamp_seconds",
Help: "When the copy of the list in use was fetched, in seconds since " +
"1970, or 0 while there is none.",
ConstLabels: source,
}, func() float64 {
fetched := lists.Fetched(listURL)
if fetched.IsZero() {
return 0
}
return float64(fetched.Unix())
}),
)
}
}
// abuseIPDBChecker is what the metrics read from what checks clients with
// AbuseIPDB.
type abuseIPDBChecker interface {
// Checked returns the checks made.
Checked() int
// Failures returns the checks that failed.
Failures() int
// BudgetLeft returns how many checks the day's budget has left.
BudgetLeft() int
}
// addAbuseIPDB adds the metrics of AbuseIPDB, with the source abuseipdb,
// read from abuseIPDB as the metrics are asked for: the checks made, those
// that failed, and how many checks the day's budget has left.
func (m *Metrics) addAbuseIPDB(abuseIPDB abuseIPDBChecker) {
source := prometheus.Labels{sourceLabel: reputation.AbuseIPDBSource}
m.addReputationQueries(source, abuseIPDB.Checked)
m.addReputationFailures(source, abuseIPDB.Failures)
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_reputation_daily_budget_remaining",
Help: "Checks of the day's SWW_WAF_ABUSEIPDB_DAILY_BUDGET not yet spent.",
ConstLabels: source,
}, func() float64 {
return float64(abuseIPDB.BudgetLeft())
}),
)
}
// alertQueue is what the metrics read from the queue of alerts.
type alertQueue interface {
// DestinationsSet returns the names of the destinations set.
DestinationsSet() []string
// Counts returns the alerts sent to the destination name, the requests
// to it that failed, and the alerts dropped.
Counts(name string) alerts.Counts
// Suppressed returns the alerts held back.
Suppressed() int64
}
// addAlerts adds the metrics of the alerts sent to each destination set,
// read from queue as the metrics are asked for, by destination: the
// alerts sent, the requests to the destination that failed, the alerts
// held back, which are the same for every destination, and those
// dropped. With no destination set, it adds none.
func (m *Metrics) addAlerts(queue alertQueue) {
for _, name := range queue.DestinationsSet() {
destination := prometheus.Labels{"destination": name}
m.registry.MustRegister(
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alerts_sent_total",
Help: "Alerts the destination took.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Counts(name).Sent)
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alert_send_failures_total",
Help: "Requests to the destination that failed.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Counts(name).SendFailures)
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alerts_suppressed_total",
Help: "Alerts held back: repeats within SWW_WAF_ALERT_COOLDOWN, and " +
"alerts past SWW_WAF_ALERT_MAX_PER_HOUR, for the hour's summary.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Suppressed())
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alerts_dropped_total",
Help: "Alerts dropped, the oldest first, from a full queue, and " +
"alerts given up as the destination refused them.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Counts(name).Dropped)
}),
)
}
}
// count counts a request that has ended, as RequestEnded and RequestReported // count counts a request that has ended, as RequestEnded and RequestReported
// describe. // describe.
func (m *Metrics) count( func (m *Metrics) count(
-160
View File
@@ -1,160 +0,0 @@
package metrics_test
import (
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/metrics"
)
const (
listURL = "https://lists.example/block.txt"
zone = "dnsbl.example"
destination = "webhook"
)
// Each stand-in answers with a number of its own, so that each metric
// shows which part it was read from.
type ledgerStandIn struct{}
func (ledgerStandIn) Made(string) int { return 1 }
func (ledgerStandIn) Count(time.Time) (int, int) { return 2, 1 }
type clientTableStandIn struct{}
func (clientTableStandIn) Len() int { return 3 }
type ruleFilesStandIn struct{}
func (ruleFilesStandIn) Len() int { return 4 }
type listsStandIn struct{}
func (listsStandIn) URLs() []string { return []string{listURL} }
func (listsStandIn) Failures(string) int { return 5 }
func (listsStandIn) Fetched(string) time.Time { return time.Time{} }
type dnsblStandIn struct{}
func (dnsblStandIn) Zones() []string { return []string{zone} }
func (dnsblStandIn) Queries(string) int { return 6 }
func (dnsblStandIn) Failures(string) int { return 0 }
type abuseIPDBStandIn struct{}
func (abuseIPDBStandIn) Checked() int { return 0 }
func (abuseIPDBStandIn) Failures() int { return 0 }
func (abuseIPDBStandIn) BudgetLeft() int { return 7 }
type alertQueueStandIn struct{}
func (alertQueueStandIn) DestinationsSet() []string {
return []string{destination}
}
func (alertQueueStandIn) Counts(string) alerts.Counts {
return alerts.Counts{Sent: 8}
}
func (alertQueueStandIn) Suppressed() int64 { return 0 }
type lookupDatabaseStandIn struct{}
func (lookupDatabaseStandIn) LastRead() time.Time { return time.Time{} }
func (lookupDatabaseStandIn) ReadFailures() int { return 9 }
type remoteLogStandIn struct{}
func (remoteLogStandIn) Sent() int64 { return 10 }
func (remoteLogStandIn) Dropped() int64 { return 0 }
func (remoteLogStandIn) Depth() int { return 0 }
func TestMetricsAreReadFromEachPart(t *testing.T) {
t.Parallel()
m := metrics.New(metrics.Params{
BusiestCount: 1,
InstanceName: "app",
Now: func() time.Time { return time.Time{} },
Bans: ledgerStandIn{},
Clients: clientTableStandIn{},
Rules: ruleFilesStandIn{},
Lists: listsStandIn{},
DNSBL: dnsblStandIn{},
AbuseIPDB: abuseIPDBStandIn{},
Alerts: alertQueueStandIn{},
LookupFile: lookupDatabaseStandIn{},
RemoteLog: remoteLogStandIn{},
})
scraped := scrape(t, m)
for _, line := range []string{
`sww_bans_made_total{cause="limit",instance="app"} 1`,
`sww_active_bans{instance="app"} 2`,
`sww_tracked_clients{instance="app"} 3`,
`sww_rules_loaded{instance="app"} 4`,
`sww_reputation_failures_total{instance="app",source="` + listURL + `"} 5`,
`sww_reputation_queries_total{instance="app",source="` + zone + `"} 6`,
`sww_reputation_daily_budget_remaining{instance="app",source="abuseipdb"} 7`,
`sww_alerts_sent_total{destination="` + destination + `",instance="app"} 8`,
`sww_lookup_database_read_failures_total{instance="app"} 9`,
`sww_remote_log_lines_sent_total{instance="app"} 10`,
} {
if !strings.Contains(scraped, "\n"+line+"\n") {
t.Errorf("metrics\n%s\nwant\n%s", scraped, line)
}
}
}
func TestPartThatIsOffAddsNoMetric(t *testing.T) {
t.Parallel()
scraped := scrape(t, metrics.New(metrics.Params{
BusiestCount: 1,
InstanceName: "app",
}))
for _, name := range []string{
"sww_bans_made_total",
"sww_tracked_clients",
"sww_rules_loaded",
"sww_reputation_",
"sww_alert",
"sww_lookup_database_",
"sww_remote_log_",
} {
if strings.Contains(scraped, name) {
t.Errorf("metrics\n%s\nshow %s", scraped, name)
}
}
}
// scrape returns the metrics m serves.
func scrape(t *testing.T, m *metrics.Metrics) string {
t.Helper()
scraped := httptest.NewRecorder()
m.ServeHTTP(
scraped,
httptest.NewRequestWithContext(
t.Context(), http.MethodGet, "/", http.NoBody,
),
)
return scraped.Body.String()
}
-44
View File
@@ -1,44 +0,0 @@
// Package processlog builds the logger for the process's own messages,
// which it writes as JSON lines marked "type":"process". The request log
// is package requestlog's.
package processlog
import (
"io"
"log/slog"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// Params are what New needs.
type Params struct {
// Output is where the lines are written.
Output io.Writer
// InstanceName, SWW_WAF_INSTANCE_NAME, is each line's instance.
InstanceName string
// Level, SWW_WAF_LOG_LEVEL, is the least severe level written.
Level slog.Level
}
// New returns the logger for the process's own messages: JSON lines on
// params.Output, marked "type":"process", with the time in the same form
// as a request line's. It writes only the messages at params.Level or more
// severe; the request lines requestlog.Write writes are never held back.
func New(params Params) *slog.Logger {
handler := slog.NewJSONHandler(params.Output, &slog.HandlerOptions{
Level: params.Level,
ReplaceAttr: func(groups []string, attr slog.Attr) slog.Attr {
if attr.Key == slog.TimeKey && len(groups) == 0 {
return slog.String(
slog.TimeKey,
requestlog.FormatTime(attr.Value.Time()),
)
}
return attr
},
})
return slog.New(handler).
With("type", "process", "instance", params.InstanceName)
}
-105
View File
@@ -1,105 +0,0 @@
package processlog_test
import (
"bytes"
"encoding/json"
"log/slog"
"slices"
"strings"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/processlog"
)
// instanceName is the instance the tests' loggers are built with.
const instanceName = "host1/blog"
func TestProcessLinesAreMarkedProcessAndGiveTheInstance(t *testing.T) {
t.Parallel()
var out bytes.Buffer
processlog.New(processlog.Params{
Output: &out,
InstanceName: instanceName,
Level: slog.LevelInfo,
}).
Info("starting",
"version", "v1")
var fields map[string]any
err := json.Unmarshal(out.Bytes(), &fields)
if err != nil {
t.Fatalf("decode %q: %v", out.String(), err)
}
if fields["type"] != "process" ||
fields["instance"] != "host1/blog" ||
fields["msg"] != "starting" ||
fields["level"] != "INFO" ||
fields["version"] != "v1" {
t.Errorf("process line %v", fields)
}
timeText, _ := fields["time"].(string)
logged, err := time.Parse(time.RFC3339, timeText)
if err != nil || !strings.HasSuffix(timeText, "Z") ||
len(timeText) != len("2006-01-02T15:04:05.000Z") ||
time.Since(logged) > time.Minute {
t.Errorf(
"process line time %q, want now in UTC with milliseconds",
timeText,
)
}
}
func TestProcessLoggerWritesTheMessagesAtItsLevelOrMoreSevere(t *testing.T) {
t.Parallel()
levels := []slog.Level{
slog.LevelDebug, slog.LevelInfo, slog.LevelWarn, slog.LevelError,
}
for i, level := range levels {
t.Run(level.String(), func(t *testing.T) {
t.Parallel()
var out bytes.Buffer
processLog := processlog.New(processlog.Params{
Output: &out,
InstanceName: instanceName,
Level: level,
})
for _, at := range levels {
processLog.Log(t.Context(), at, "message")
}
var got, want []string
for line := range strings.Lines(out.String()) {
var fields struct {
Level string `json:"level"`
}
err := json.Unmarshal([]byte(line), &fields)
if err != nil {
t.Fatalf("decode %q: %v", line, err)
}
got = append(got, fields.Level)
}
for _, written := range levels[i:] {
want = append(want, written.String())
}
if !slices.Equal(got, want) {
t.Errorf("lines at %v, want %v", got, want)
}
})
}
}
+4 -7
View File
@@ -11,7 +11,7 @@ import (
func TestHistoryKeepsEveryRequest(t *testing.T) { func TestHistoryKeepsEveryRequest(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize}) limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -57,7 +57,7 @@ func TestHistoryKeepsEveryRequest(t *testing.T) {
func TestLookupReachesTheHistoryOfAClientInTheTable(t *testing.T) { func TestLookupReachesTheHistoryOfAClientInTheTable(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize}) limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
other := netip.MustParsePrefix("198.51.100.7/32") other := netip.MustParsePrefix("198.51.100.7/32")
start := midnight() start := midnight()
@@ -100,10 +100,7 @@ func TestLookupReachesTheHistoryOfAClientInTheTable(t *testing.T) {
func TestResetKeepsTheHistory(t *testing.T) { func TestResetKeepsTheHistory(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit}, tableSize)
Limits: ratelimit.Limits{PerMinute: limit},
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -126,7 +123,7 @@ func TestResetKeepsTheHistory(t *testing.T) {
func TestRequestsAddsUpTheClientsInsideTheNetblock(t *testing.T) { func TestRequestsAddsUpTheClientsInsideTheNetblock(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize}) limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
for client, requests := range map[string]int{ for client, requests := range map[string]int{
"198.51.100.9/32": 2, "198.51.100.9/32": 2,
+6 -18
View File
@@ -156,28 +156,16 @@ type Request struct {
BlockedByCoreRuleSet bool BlockedByCoreRuleSet bool
} }
// Params are what New needs. // New returns a Limiter for limits, with no client counted yet, whose
type Params struct { // table holds at most maxClients clients (SWW_WAF_MAX_TRACKED_CLIENTS). Past
// Limits are what each client's requests and bytes are counted against. // it, the least recently seen client is dropped, with its history, and
Limits Limits // starts afresh if it comes back.
// MaxClients is the most clients the table holds, func New(limits Limits, maxClients int) *Limiter {
// SWW_WAF_MAX_TRACKED_CLIENTS. Past it, the least recently seen client clients, err := simplelru.NewLRU[netip.Prefix, *Client](maxClients, nil)
// is dropped, with its history, and starts afresh if it comes back.
MaxClients int
}
// New returns a Limiter with no client counted yet.
func New(params Params) *Limiter {
clients, err := simplelru.NewLRU[netip.Prefix, *Client](
params.MaxClients,
nil,
)
if err != nil { if err != nil {
panic(err) // NewLRU fails only for a size below one panic(err) // NewLRU fails only for a size below one
} }
limits := params.Limits
return &Limiter{ return &Limiter{
windows: [3]window{ windows: [3]window{
{ {
+39 -63
View File
@@ -33,10 +33,10 @@ func TestOverARateLimitSaysWhatTheNextRequestWouldDoAndCountsNothing(
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
now := midnight() now := midnight()
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(
Limits: ratelimit.Limits{PerMinute: limit, PerDay: limit + 1}, ratelimit.Limits{PerMinute: limit, PerDay: limit + 1},
MaxClients: tableSize, tableSize,
}) )
if limiter.OverARateLimit(client, now) { if limiter.OverARateLimit(client, now) {
t.Error("a client never seen is over a rate limit") t.Error("a client never seen is over a rate limit")
@@ -72,7 +72,7 @@ func TestOverARateLimitSaysWhatTheNextRequestWouldDoAndCountsNothing(
wantCount(t, limiter, client, now, minute) wantCount(t, limiter, client, now, minute)
// A limit that is off is never gone over. // A limit that is off is never gone over.
off := ratelimit.New(ratelimit.Params{MaxClients: tableSize}) off := ratelimit.New(ratelimit.Limits{}, tableSize)
off.Count(client, now, whole) off.Count(client, now, whole)
if off.OverARateLimit(client, now) { if off.OverARateLimit(client, now) {
@@ -95,10 +95,7 @@ func TestEachWindowRefusesAtItsLimitAndLetsTheClientBack(t *testing.T) {
t.Run(tc.window, func(t *testing.T) { t.Run(tc.window, func(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(tc.limits, tableSize)
Limits: tc.limits,
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
quarter := tc.length / 4 quarter := tc.length / 4
@@ -129,10 +126,10 @@ func TestEachWindowRefusesAtItsLimitAndLetsTheClientBack(t *testing.T) {
func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) { func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(
Limits: ratelimit.Limits{PerMinute: limit, PerHour: limit}, ratelimit.Limits{PerMinute: limit, PerHour: limit},
MaxClients: tableSize, tableSize,
}) )
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -162,13 +159,10 @@ func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
func TestClientGetsItsPercentageOfEachLimitRoundedDown(t *testing.T) { func TestClientGetsItsPercentageOfEachLimitRoundedDown(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(
Limits: ratelimit.Limits{ ratelimit.Limits{PerMinute: 5, BytesPerDay: math.MaxInt64},
PerMinute: 5, tableSize,
BytesPerDay: math.MaxInt64, )
},
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -209,10 +203,10 @@ func TestZeroPercentIsAZeroAllowanceAndALimitOffStaysOff(t *testing.T) {
t.Parallel() t.Parallel()
// Only the hour has limits: the minute's and the day's are off. // Only the hour has limits: the minute's and the day's are off.
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(
Limits: ratelimit.Limits{PerHour: limit, BytesPerHour: 1000}, ratelimit.Limits{PerHour: limit, BytesPerHour: 1000},
MaxClients: tableSize, tableSize,
}) )
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -259,10 +253,7 @@ func TestEachByteLimitIsBrokenByTheBytesCounted(t *testing.T) {
t.Run(tc.window, func(t *testing.T) { t.Run(tc.window, func(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(tc.limits, tableSize)
Limits: tc.limits,
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
// 600 bytes are within the limit, 600 more over it. // 600 bytes are within the limit, 600 more over it.
@@ -296,10 +287,10 @@ func TestEachByteLimitIsBrokenByTheBytesCounted(t *testing.T) {
func TestALimitIsBrokenOnlyByWhatIsAddedToIt(t *testing.T) { func TestALimitIsBrokenOnlyByWhatIsAddedToIt(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(
Limits: ratelimit.Limits{PerMinute: 2, BytesPerMinute: 1000}, ratelimit.Limits{PerMinute: 2, BytesPerMinute: 1000},
MaxClients: tableSize, tableSize,
}) )
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
other := netip.MustParsePrefix("203.0.113.10/32") other := netip.MustParsePrefix("203.0.113.10/32")
start := midnight() start := midnight()
@@ -323,7 +314,7 @@ func TestALimitIsBrokenOnlyByWhatIsAddedToIt(t *testing.T) {
func TestCountGivesTheBytesInEachWindow(t *testing.T) { func TestCountGivesTheBytesInEachWindow(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize}) limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -349,10 +340,7 @@ func TestCountGivesTheBytesInEachWindow(t *testing.T) {
func TestResetSetsTheBytesBackToZero(t *testing.T) { func TestResetSetsTheBytesBackToZero(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(ratelimit.Limits{BytesPerDay: 1000}, tableSize)
Limits: ratelimit.Limits{BytesPerDay: 1000},
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -368,7 +356,7 @@ func TestRefusalsOverTheLimitInAMinuteBreakTheRefusalLimit(
) { ) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize}) limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -423,7 +411,7 @@ func TestRefusalsOverTheLimitInAMinuteBreakTheRefusalLimit(
func TestCountGivesTheRequestsInEachWindow(t *testing.T) { func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize}) limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -450,10 +438,10 @@ func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
func TestResetSetsTheCountsBackToZero(t *testing.T) { func TestResetSetsTheCountsBackToZero(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(
Limits: ratelimit.Limits{PerMinute: limit, PerDay: limit}, ratelimit.Limits{PerMinute: limit, PerDay: limit},
MaxClients: tableSize, tableSize,
}) )
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -475,10 +463,7 @@ func TestResetSetsTheCountsBackToZero(t *testing.T) {
func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) { func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize)
Limits: ratelimit.Limits{PerHour: limit},
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -497,10 +482,10 @@ func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) {
func TestRefusedRequestsCount(t *testing.T) { func TestRefusedRequestsCount(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(
Limits: ratelimit.Limits{PerMinute: limit, PerHour: 2 * limit}, ratelimit.Limits{PerMinute: limit, PerHour: 2 * limit},
MaxClients: tableSize, tableSize,
}) )
refused := netip.MustParsePrefix("203.0.113.9/32") refused := netip.MustParsePrefix("203.0.113.9/32")
within := netip.MustParsePrefix("203.0.113.10/32") within := netip.MustParsePrefix("203.0.113.10/32")
start := midnight() start := midnight()
@@ -533,10 +518,7 @@ func TestRefusedRequestsCount(t *testing.T) {
func TestRequestCountedLateGoesInTheBucketUnderWay(t *testing.T) { func TestRequestCountedLateGoesInTheBucketUnderWay(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit}, tableSize)
Limits: ratelimit.Limits{PerMinute: limit},
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -552,10 +534,7 @@ func TestRequestCountedLateGoesInTheBucketUnderWay(t *testing.T) {
func TestClockSetBackStartsTheBucketsAfresh(t *testing.T) { func TestClockSetBackStartsTheBucketsAfresh(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize)
Limits: ratelimit.Limits{PerHour: limit},
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -583,10 +562,7 @@ func TestKeepsAtMostMaxClientsDroppingTheLeastRecentlySeen(t *testing.T) {
const maxClients = 3 const maxClients = 3
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(ratelimit.Limits{PerMinute: 1}, maxClients)
Limits: ratelimit.Limits{PerMinute: 1},
MaxClients: maxClients,
})
now := midnight() now := midnight()
clients := make([]netip.Prefix, maxClients+1) clients := make([]netip.Prefix, maxClients+1)
+6 -12
View File
@@ -19,7 +19,7 @@ func TestSnapshotListsTheClientsByAddress(t *testing.T) {
"2001:db8::/64", "2001:db8::/64",
} }
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize}) limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
for _, i := range []int{2, 3, 0, 1} { for _, i := range []int{2, 3, 0, 1} {
limiter.Count(netip.MustParsePrefix(want[i]), midnight(), whole) limiter.Count(netip.MustParsePrefix(want[i]), midnight(), whole)
} }
@@ -52,10 +52,7 @@ func TestLoadedCountsCarryOn(t *testing.T) {
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
before := ratelimit.New(ratelimit.Params{ before := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize)
Limits: ratelimit.Limits{PerHour: limit},
MaxClients: tableSize,
})
for range limit { for range limit {
wantCount(t, before, client, start, "") wantCount(t, before, client, start, "")
} }
@@ -63,10 +60,7 @@ func TestLoadedCountsCarryOn(t *testing.T) {
// Loaded into a new limiter, as across a restart, the client has no // Loaded into a new limiter, as across a restart, the client has no
// fresh allowance. // fresh allowance.
later := start.Add(time.Minute) later := start.Add(time.Minute)
after := ratelimit.New(ratelimit.Params{ after := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize)
Limits: ratelimit.Limits{PerHour: limit},
MaxClients: tableSize,
})
after.Load(before.Snapshot(), later) after.Load(before.Snapshot(), later)
wantCount(t, after, client, later, hour) wantCount(t, after, client, later, hour)
} }
@@ -77,7 +71,7 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize}) limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
limiter.Count(client, start, whole) limiter.Count(client, start, whole)
limiter.CountBytes(client, start, 5, whole) limiter.CountBytes(client, start, 5, whole)
limiter.CountRefusal(client, start, limit) limiter.CountRefusal(client, start, limit)
@@ -86,7 +80,7 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
loaded := func(now time.Time) ratelimit.Client { loaded := func(now time.Time) ratelimit.Client {
t.Helper() t.Helper()
after := ratelimit.New(ratelimit.Params{MaxClients: tableSize}) after := ratelimit.New(ratelimit.Limits{}, tableSize)
after.Load(limiter.Snapshot(), now) after.Load(limiter.Snapshot(), now)
return after.Snapshot()[0] return after.Snapshot()[0]
@@ -148,7 +142,7 @@ func TestLoadDropsTheLeastRecentlySeenFirst(t *testing.T) {
addr = addr.Next() addr = addr.Next()
} }
limiter := ratelimit.New(ratelimit.Params{MaxClients: maxClients}) limiter := ratelimit.New(ratelimit.Limits{}, maxClients)
limiter.Load(clients, midnight()) limiter.Load(clients, midnight())
got := limiter.Snapshot() got := limiter.Snapshot()
+12 -9
View File
@@ -445,9 +445,12 @@ func TestKeyIsSentInTheKeyHeaderAndNeverShown(t *testing.T) {
scores: map[string]int64{suspect: 100}, scores: map[string]int64{suspect: 100},
} }
checker := newAbuseIPDB(abuseIPDB, p) checker := newAbuseIPDB(abuseIPDB, p)
m := metrics.New(metrics.Params{ m := metrics.New(1, "app")
BusiestCount: 1, InstanceName: instanceName, AbuseIPDB: checker, m.AddReputation(
}) reputation.New(params()),
reputation.NewDNSBL(dnsblParams()),
)
m.AddAbuseIPDB(checker)
// One check that AbuseIPDB answers, and one it refuses with an answer // One check that AbuseIPDB answers, and one it refuses with an answer
// that names the key. // that names the key.
@@ -500,9 +503,12 @@ func TestMetricsCountTheChecksTheFailuresAndTheBudgetLeft(t *testing.T) {
p := abuseIPDBParams() p := abuseIPDBParams()
p.DailyBudget = 5 p.DailyBudget = 5
checker := newAbuseIPDB(abuseIPDB, p) checker := newAbuseIPDB(abuseIPDB, p)
m := metrics.New(metrics.Params{ m := metrics.New(1, "app")
BusiestCount: 1, InstanceName: instanceName, AbuseIPDB: checker, m.AddReputation(
}) reputation.New(params()),
reputation.NewDNSBL(dnsblParams()),
)
m.AddAbuseIPDB(checker)
// One check that AbuseIPDB answers, and one that fails. // One check that AbuseIPDB answers, and one that fails.
hitFrom(t, checker, suspect, true) hitFrom(t, checker, suspect, true)
@@ -805,9 +811,6 @@ func wantBudgetLeft(t *testing.T, checker *reputation.AbuseIPDB, want int) {
} }
} }
// instanceName is the label instance of the metrics the tests build.
const instanceName = "app"
// scrapeMetrics returns the metrics m serves. // scrapeMetrics returns the metrics m serves.
func scrapeMetrics(t *testing.T, m *metrics.Metrics) string { func scrapeMetrics(t *testing.T, m *metrics.Metrics) string {
t.Helper() t.Helper()
+4 -6
View File
@@ -358,9 +358,8 @@ func TestMetricsCountEachZonesQueriesAndThoseThatFailed(t *testing.T) {
"99.2.0.192." + otherZone + ".": {rcode: serverFailure}, "99.2.0.192." + otherZone + ".": {rcode: serverFailure},
}} }}
dnsbl := newDNSBL(resolver, dnsblParams(zone, otherZone)) dnsbl := newDNSBL(resolver, dnsblParams(zone, otherZone))
m := metrics.New(metrics.Params{ m := metrics.New(1, "app")
BusiestCount: 1, InstanceName: instanceName, DNSBL: dnsbl, m.AddReputation(reputation.New(params()), dnsbl)
})
wantZones(t, dnsbl, listed) wantZones(t, dnsbl, listed)
synctest.Wait() synctest.Wait()
@@ -405,9 +404,8 @@ func TestZoneKeyIsMaskedInTheVerdictsTheFailuresAndTheMetrics(t *testing.T) {
"99.2.0.192." + keyed + ".": {addrs: []string{listing}}, "99.2.0.192." + keyed + ".": {addrs: []string{listing}},
"100.2.0.192." + keyed + ".": {rcode: serverFailure}, "100.2.0.192." + keyed + ".": {rcode: serverFailure},
}}, p) }}, p)
m := metrics.New(metrics.Params{ m := metrics.New(1, "app")
BusiestCount: 1, InstanceName: instanceName, DNSBL: dnsbl, m.AddReputation(reputation.New(params()), dnsbl)
})
// Both clients are asked about before either answer comes, so that // Both clients are asked about before either answer comes, so that
// the failure does not keep the zone from the other query. // the failure does not keep the zone from the other query.
+33 -4
View File
@@ -1,7 +1,7 @@
// Package requestlog writes the request log sww prints on stdout: JSON // Package requestlog writes the lines sww prints on stdout: one
// lines marked "type":"request", at most two from waf and a short one // JSON object per request, two from waf and a short one from rproxy,
// from rproxy for each request. Package processlog writes the process's // marked "type":"request", and the process's own messages as JSON lines marked
// own messages. // "type":"process".
package requestlog package requestlog
import ( import (
@@ -9,6 +9,7 @@ import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"io" "io"
"log/slog"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
@@ -299,3 +300,31 @@ func Milliseconds(d time.Duration) float64 {
time.Millisecond/time.Microsecond, time.Millisecond/time.Microsecond,
) )
} }
// NewProcessLogger returns the logger for the process's own messages:
// JSON lines on w, marked "type":"process", with the time in the same form
// as a request line's, and instanceName, SWW_WAF_INSTANCE_NAME, as instance.
// It writes only the messages at level, SWW_WAF_LOG_LEVEL, or more severe;
// the request lines Write writes are never held back.
func NewProcessLogger(
w io.Writer,
instanceName string,
level slog.Level,
) *slog.Logger {
handler := slog.NewJSONHandler(w, &slog.HandlerOptions{
Level: level,
ReplaceAttr: func(groups []string, attr slog.Attr) slog.Attr {
if attr.Key == slog.TimeKey && len(groups) == 0 {
return slog.String(
slog.TimeKey,
FormatTime(attr.Value.Time()),
)
}
return attr
},
})
return slog.New(handler).
With("type", "process", "instance", instanceName)
}
+87
View File
@@ -3,6 +3,8 @@ package requestlog_test
import ( import (
"bytes" "bytes"
"encoding/json" "encoding/json"
"log/slog"
"slices"
"strings" "strings"
"testing" "testing"
"time" "time"
@@ -132,3 +134,88 @@ func TestWriteRProxyLineWritesOneJSONLineWithOnlyItsFields(t *testing.T) {
t.Errorf("wrote %s, want %s", out.String(), want) t.Errorf("wrote %s, want %s", out.String(), want)
} }
} }
func TestProcessLinesAreMarkedProcessAndGiveTheInstance(t *testing.T) {
t.Parallel()
var out bytes.Buffer
requestlog.NewProcessLogger(&out, "host1/blog", slog.LevelInfo).
Info("starting",
"version", "v1")
var fields map[string]any
err := json.Unmarshal(out.Bytes(), &fields)
if err != nil {
t.Fatalf("decode %q: %v", out.String(), err)
}
if fields["type"] != "process" ||
fields["instance"] != "host1/blog" ||
fields["msg"] != "starting" ||
fields["level"] != "INFO" ||
fields["version"] != "v1" {
t.Errorf("process line %v", fields)
}
timeText, _ := fields["time"].(string)
logged, err := time.Parse(time.RFC3339, timeText)
if err != nil || !strings.HasSuffix(timeText, "Z") ||
len(timeText) != len("2006-01-02T15:04:05.000Z") ||
time.Since(logged) > time.Minute {
t.Errorf(
"process line time %q, want now in UTC with milliseconds",
timeText,
)
}
}
func TestProcessLoggerWritesTheMessagesAtItsLevelOrMoreSevere(t *testing.T) {
t.Parallel()
levels := []slog.Level{
slog.LevelDebug, slog.LevelInfo, slog.LevelWarn, slog.LevelError,
}
for i, level := range levels {
t.Run(level.String(), func(t *testing.T) {
t.Parallel()
var out bytes.Buffer
processLog := requestlog.NewProcessLogger(
&out,
"host1/blog",
level,
)
for _, at := range levels {
processLog.Log(t.Context(), at, "message")
}
var got, want []string
for line := range strings.Lines(out.String()) {
var fields struct {
Level string `json:"level"`
}
err := json.Unmarshal([]byte(line), &fields)
if err != nil {
t.Fatalf("decode %q: %v", line, err)
}
got = append(got, fields.Level)
}
for _, written := range levels[i:] {
want = append(want, written.String())
}
if !slices.Equal(got, want) {
t.Errorf("lines at %v, want %v", got, want)
}
})
}
}
+7 -16
View File
@@ -2,7 +2,6 @@ package rproxy
import ( import (
"bufio" "bufio"
"context"
"errors" "errors"
"io" "io"
"net" "net"
@@ -23,9 +22,7 @@ type requestBody struct {
// body is the visitor's body, ending in an *http.MaxBytesError past // body is the visitor's body, ending in an *http.MaxBytesError past
// SWW_RPROXY_MAX_REQUEST_SIZE. // SWW_RPROXY_MAX_REQUEST_SIZE.
body io.ReadCloser body io.ReadCloser
// refuse refuses the request with a status, as the request's own refuse rq *request
// does.
refuse func(status int)
// held is the part read before the request goes to the app, which Read gives // held is the part read before the request goes to the app, which Read gives
// first. // first.
held []byte held []byte
@@ -71,7 +68,7 @@ func (b *requestBody) Read(p []byte) (int, error) {
case errors.Is(err, io.EOF): case errors.Is(err, io.EOF):
b.received.Store(true) b.received.Store(true)
case errors.As(err, &tooLarge): case errors.As(err, &tooLarge):
b.refuse(http.StatusRequestEntityTooLarge) b.rq.refuse(http.StatusRequestEntityTooLarge)
} }
return n, err return n, err
@@ -89,13 +86,7 @@ type responseBody struct {
// body is the app's body, ending in an *http.MaxBytesError past // body is the app's body, ending in an *http.MaxBytesError past
// SWW_RPROXY_MAX_RESPONSE_SIZE. // SWW_RPROXY_MAX_RESPONSE_SIZE.
body io.ReadCloser body io.ReadCloser
// refuse refuses the request with a status, as the request's own refuse rq *request
// does.
refuse func(status int)
// responseReceived is called once the app's whole answer has arrived.
responseReceived func()
// ctx is the visitor's request's context.
ctx context.Context //nolint:containedctx // the body ends with the request
} }
// Read reads from the app's body. // Read reads from the app's body.
@@ -109,15 +100,15 @@ func (b *responseBody) Read(p []byte) (int, error) {
switch { switch {
case errors.Is(err, io.EOF): case errors.Is(err, io.EOF):
b.responseReceived() b.rq.responseReceived()
case errors.As(err, &tooLarge): case errors.As(err, &tooLarge):
b.refuse(http.StatusBadGateway) b.rq.refuse(http.StatusBadGateway)
return n, errResponseTooLarge return n, errResponseTooLarge
case b.ctx.Err() == nil: case b.rq.in.Context().Err() == nil:
// The answer broke off, not because the visitor went away. If a time // The answer broke off, not because the visitor went away. If a time
// limit cut it, that refusal came first and is the one kept. // limit cut it, that refusal came first and is the one kept.
b.refuse(http.StatusBadGateway) b.rq.refuse(http.StatusBadGateway)
} }
return n, err return n, err
+10
View File
@@ -7,6 +7,16 @@ import (
"strings" "strings"
) )
// The headers rproxy sets on the requests it passes to the app. Go writes every
// header name in this form, as it sends it and as it receives it, so
// X-Client-ASN arrives as X-Client-Asn, and Del removes a visitor's own whatever
// their case; header names are not case-sensitive.
const (
requestIDHeader = "X-Request-ID"
asnHeader = "X-Client-Asn"
countryHeader = "X-Client-Country"
)
// peerAddress is the address of the request's TCP peer, normally traefik, without // peerAddress is the address of the request's TCP peer, normally traefik, without
// any IPv6 zone, which waf refuses. // any IPv6 zone, which waf refuses.
func peerAddress(r *http.Request) netip.Addr { func peerAddress(r *http.Request) netip.Addr {
+9 -9
View File
@@ -18,9 +18,9 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/config/configtest" "sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/processlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rproxy" "sneak.berlin/go/smallwebwaf/internal/rproxy"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/waf"
) )
// post is the start of a POST whose headers end with what follows it. // post is the start of a POST whose headers end with what follows it.
@@ -51,11 +51,11 @@ func TestServerHasTheFixedLimits(t *testing.T) {
server := rproxy.New(rproxy.Params{ server := rproxy.New(rproxy.Params{
Config: cfg, Config: cfg,
RequestLog: io.Discard, RequestLog: io.Discard,
ProcessLog: processlog.New(processlog.Params{ ProcessLog: requestlog.NewProcessLogger(
Output: io.Discard, io.Discard,
InstanceName: "test", "test",
Level: slog.LevelInfo, slog.LevelInfo,
}), ),
}) })
if server.Addr != ":8080" || server.MaxHeaderBytes != 28<<10 || if server.Addr != ":8080" || server.MaxHeaderBytes != 28<<10 ||
@@ -492,7 +492,7 @@ func TestASettingOfOffSetsNoLimit(t *testing.T) {
strings.NewReader("a body"))), strings.NewReader("a body"))),
) )
if got := get(t, addr, wafprotocol.HealthPath); got.status != http.StatusOK { if got := get(t, addr, waf.HealthPath); got.status != http.StatusOK {
t.Errorf("answered %+v, want rproxy's own 200", got) t.Errorf("answered %+v, want rproxy's own 200", got)
} }
}) })
@@ -1480,7 +1480,7 @@ func TestAVisitorThatStopsALittleShortGetsAnAnswerRProxyGivesAtOnce(
{"a deny", post, http.StatusForbidden}, {"a deny", post, http.StatusForbidden},
{ {
"the health check", "the health check",
"GET " + wafprotocol.HealthPath + " HTTP/1.1\r\nHost: site.example\r\n", "GET " + waf.HealthPath + " HTTP/1.1\r\nHost: site.example\r\n",
http.StatusOK, http.StatusOK,
}, },
} { } {
+13 -13
View File
@@ -12,11 +12,11 @@ import (
"strings" "strings"
"testing" "testing"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/waf"
) )
// questionBodyMaxBytes is the most of a body a question carries. // questionBodyMaxBytes is the most of a body a question carries.
const questionBodyMaxBytes = wafprotocol.QuestionBodyMaxBytes const questionBodyMaxBytes = waf.QuestionBodyMaxBytes
// chunked is the encoding of a body whose length the visitor did not announce. // chunked is the encoding of a body whose length the visitor did not announce.
const chunked = "chunked" const chunked = "chunked"
@@ -42,9 +42,9 @@ func TestQuestionIsTheVisitorsRequestWithRProxysHeaders(t *testing.T) {
req.Header.Set(forwardedForHeader, "198.51.100.1") req.Header.Set(forwardedForHeader, "198.51.100.1")
req.Header.Set(forwardedProtoHeader, "gopher") req.Header.Set(forwardedProtoHeader, "gopher")
req.Header.Set(requestIDHeader, "the-visitor's") req.Header.Set(requestIDHeader, "the-visitor's")
req.Header.Set(wafprotocol.ClientHeader, "198.51.100.2") req.Header.Set(waf.ClientHeader, "198.51.100.2")
req.Header.Set(wafprotocol.PeerHeader, "198.51.100.3") req.Header.Set(waf.PeerHeader, "198.51.100.3")
req.Header.Set(wafprotocol.ContentLengthHeader, "12345") req.Header.Set(waf.ContentLengthHeader, "12345")
// Go's client then sends no User-Agent, so that one in the question would be // Go's client then sends no User-Agent, so that one in the question would be
// Go's own. // Go's own.
req.Header["User-Agent"] = nil req.Header["User-Agent"] = nil
@@ -73,8 +73,8 @@ func TestQuestionIsTheVisitorsRequestWithRProxysHeaders(t *testing.T) {
want.Set(forwardedForHeader, "198.51.100.1") want.Set(forwardedForHeader, "198.51.100.1")
want.Set(forwardedProtoHeader, "http") want.Set(forwardedProtoHeader, "http")
want.Set(requestIDHeader, asked.header.Get(requestIDHeader)) want.Set(requestIDHeader, asked.header.Get(requestIDHeader))
want.Set(wafprotocol.ClientHeader, localhost) want.Set(waf.ClientHeader, localhost)
want.Set(wafprotocol.PeerHeader, localhost) want.Set(waf.PeerHeader, localhost)
if !maps.EqualFunc(asked.header, want, slices.Equal) { if !maps.EqualFunc(asked.header, want, slices.Equal) {
t.Errorf( t.Errorf(
@@ -124,8 +124,8 @@ func TestQuestionNamesTheVisitorATrustedProxySpeaksFor(t *testing.T) {
wantGreeting(t, visit(t, req)) wantGreeting(t, visit(t, req))
asked := standIn.question(t) asked := standIn.question(t)
if asked.header.Get(wafprotocol.ClientHeader) != tc.client || if asked.header.Get(waf.ClientHeader) != tc.client ||
asked.header.Get(wafprotocol.PeerHeader) != localhost || asked.header.Get(waf.PeerHeader) != localhost ||
asked.header.Get(forwardedForHeader) != visitor || asked.header.Get(forwardedForHeader) != visitor ||
asked.header.Get(forwardedProtoHeader) != tc.proto { asked.header.Get(forwardedProtoHeader) != tc.proto {
t.Errorf("the question's headers are %v", asked.header) t.Errorf("the question's headers are %v", asked.header)
@@ -191,7 +191,7 @@ func TestQuestionNamesTheClientAsTheForwardedAddressesSay(t *testing.T) {
req.Header[forwardedForHeader] = tc.header req.Header[forwardedForHeader] = tc.header
wantGreeting(t, visit(t, req)) wantGreeting(t, visit(t, req))
if got := standIn.question(t).header.Get(wafprotocol.ClientHeader); got != if got := standIn.question(t).header.Get(waf.ClientHeader); got !=
tc.client { tc.client {
t.Errorf("the question names %s, want %s", got, tc.client) t.Errorf("the question names %s, want %s", got, tc.client)
} }
@@ -282,7 +282,7 @@ func wantQuestionBody(
) )
} }
values := asked.header.Values(wafprotocol.ContentLengthHeader) values := asked.header.Values(waf.ContentLengthHeader)
if announced { if announced {
if !slices.Equal(values, []string{strconv.Itoa(size)}) || if !slices.Equal(values, []string{strconv.Itoa(size)}) ||
@@ -586,8 +586,8 @@ func TestAQuestionNamesNoZone(t *testing.T) {
server.Handler.ServeHTTP(httptest.NewRecorder(), req) server.Handler.ServeHTTP(httptest.NewRecorder(), req)
asked := standIn.question(t) asked := standIn.question(t)
if asked.header.Get(wafprotocol.ClientHeader) != tc.client || if asked.header.Get(waf.ClientHeader) != tc.client ||
asked.header.Get(wafprotocol.PeerHeader) != tc.peer { asked.header.Get(waf.PeerHeader) != tc.peer {
t.Errorf("the question's headers are %v", asked.header) t.Errorf("the question's headers are %v", asked.header)
} }
+15 -18
View File
@@ -18,13 +18,17 @@ import (
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules" "sneak.berlin/go/smallwebwaf/internal/rules"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/waf"
) )
// flushAfterEachWrite has ReverseProxy pass on each part of the app's answer as // flushAfterEachWrite has ReverseProxy pass on each part of the app's answer as
// soon as it arrives. // soon as it arrives.
const flushAfterEachWrite time.Duration = -1 const flushAfterEachWrite time.Duration = -1
// reportStatusMax is the highest status waf takes in a report. An app can
// send a higher one, up to 999, and the visitor gets that as it is.
const reportStatusMax = 599
// plainText is the Content-Type of the answers rproxy gives itself. // plainText is the Content-Type of the answers rproxy gives itself.
const plainText = "text/plain; charset=utf-8" const plainText = "text/plain; charset=utf-8"
@@ -123,7 +127,7 @@ func (h *handler) newRequest(
if r.Body != http.NoBody { if r.Body != http.NoBody {
rq.body = &requestBody{ rq.body = &requestBody{
body: limitBody(r.Body, h.config.MaxRequestSize), body: limitBody(r.Body, h.config.MaxRequestSize),
refuse: rq.refuse, rq: rq,
} }
} }
@@ -150,7 +154,7 @@ func (rq *request) readFirstPartOfBody() bool {
_ = rq.rc.SetReadDeadline(rq.requestDeadline()) _ = rq.rc.SetReadDeadline(rq.requestDeadline())
held, err := io.ReadAll( held, err := io.ReadAll(
io.LimitReader(rq.body, wafprotocol.QuestionBodyMaxBytes), io.LimitReader(rq.body, waf.QuestionBodyMaxBytes),
) )
// From here on, the request timer ends a read of the rest of the body, so that // From here on, the request timer ends a read of the rest of the body, so that
// the answer can name the side rproxy was waiting on. // the answer can name the side rproxy was waiting on.
@@ -224,17 +228,16 @@ func (rq *request) rewrite(pr *httputil.ProxyRequest) {
setPathAsSent(pr.Out.URL, pr.In) setPathAsSent(pr.Out.URL, pr.In)
setForwardedHeaders(pr.In, pr.Out, rq.peer, rq.peerTrusted) setForwardedHeaders(pr.In, pr.Out, rq.peer, rq.peerTrusted)
pr.Out.Header.Set(wafprotocol.RequestIDHeader, rq.id) pr.Out.Header.Set(requestIDHeader, rq.id)
// Del removes a visitor's own, whatever their case. pr.Out.Header.Del(asnHeader)
pr.Out.Header.Del(wafprotocol.ASNHeader) pr.Out.Header.Del(countryHeader)
pr.Out.Header.Del(wafprotocol.CountryHeader)
if rq.asn != "" { if rq.asn != "" {
pr.Out.Header.Set(wafprotocol.ASNHeader, rq.asn) pr.Out.Header.Set(asnHeader, rq.asn)
} }
if rq.country != "" { if rq.country != "" {
pr.Out.Header.Set(wafprotocol.CountryHeader, rq.country) pr.Out.Header.Set(countryHeader, rq.country)
} }
} }
@@ -291,12 +294,7 @@ func (rq *request) modifyResponse(res *http.Response) error {
// the request, and SWW_RPROXY_MAX_RESPONSE_TIME_MS starts. // the request, and SWW_RPROXY_MAX_RESPONSE_TIME_MS starts.
rq.startResponseTimer(requestEnd) rq.startResponseTimer(requestEnd)
res.Body = &responseBody{ res.Body = &responseBody{body: limitBody(res.Body, maxBytes), rq: rq}
body: limitBody(res.Body, maxBytes),
refuse: rq.refuse,
responseReceived: rq.responseReceived,
ctx: rq.in.Context(),
}
rq.out.afterStatus = rq.setResponseDeadline rq.out.afterStatus = rq.setResponseDeadline
return nil return nil
@@ -472,10 +470,9 @@ func (rq *request) finish(ctx context.Context) {
return return
} }
// An app can send a status up to 999, and the visitor gets it as it is. report := &waf.Report{
report := &wafprotocol.Report{
RequestID: rq.id, RequestID: rq.id,
Status: min(rq.out.status, wafprotocol.ReportStatusMax), Status: min(rq.out.status, reportStatusMax),
RequestBytes: requestBytes, RequestBytes: requestBytes,
ResponseBytes: responseBytes, ResponseBytes: responseBytes,
DurationTotalMS: requestlog.Milliseconds(end.Sub(rq.start)), DurationTotalMS: requestlog.Milliseconds(end.Sub(rq.start)),
+12 -10
View File
@@ -24,7 +24,7 @@ import (
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/waf"
) )
// How rproxy keeps connections to the app and to waf open between // How rproxy keeps connections to the app and to waf open between
@@ -47,6 +47,10 @@ const (
idleTimeout = 120 * time.Second idleTimeout = 120 * time.Second
) )
// ownEndpointPrefix starts the path of every request for sww itself, which
// never reaches the app.
const ownEndpointPrefix = "/_sww/"
// allowedMethods lists the methods rproxy passes on, as an Allow header gives // allowedMethods lists the methods rproxy passes on, as an Allow header gives
// them. It answers any other method 405 itself. // them. It answers any other method 405 itself.
// The WebDAV ones are for Nextcloud. // The WebDAV ones are for Nextcloud.
@@ -88,9 +92,7 @@ func New(params Params) *Server {
transport: newTransport(), transport: newTransport(),
wafTransport: newWAFTransport(), wafTransport: newWAFTransport(),
reportURL: (&url.URL{ reportURL: (&url.URL{
Scheme: backend.Scheme, Scheme: backend.Scheme, Host: backend.Host, Path: waf.ReportPath,
Host: backend.Host,
Path: wafprotocol.ReportPath,
}).String(), }).String(),
breakerClock: time.Now, breakerClock: time.Now,
} }
@@ -177,7 +179,7 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
defer rq.readRestOfBody() defer rq.readRestOfBody()
defer rq.finish(r.Context()) defer rq.finish(r.Context())
if r.Method == http.MethodGet && r.URL.Path == wafprotocol.HealthPath { if r.Method == http.MethodGet && r.URL.Path == waf.HealthPath {
rq.answerHealth() rq.answerHealth()
return return
@@ -210,9 +212,9 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
rq.noteAnswer(answer) rq.noteAnswer(answer)
switch rq.disposition { switch rq.disposition {
case wafprotocol.DispositionAllow: case waf.DispositionAllow:
rq.passToApp(r.Context()) rq.passToApp(r.Context())
case wafprotocol.DispositionClose: case waf.DispositionClose:
// This panic has Go's server close the connection without an answer, // This panic has Go's server close the connection without an answer,
// and log nothing; the deferred finish still runs. // and log nothing; the deferred finish still runs.
panic(http.ErrAbortHandler) panic(http.ErrAbortHandler)
@@ -222,8 +224,8 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
} }
// passToApp passes the request on to the app, unless a limit or its path keeps // passToApp passes the request on to the app, unless a limit or its path keeps
// it from the app. waf answers a path under wafprotocol.OwnEndpointPrefix // it from the app. waf answers a path under ownEndpointPrefix itself, so
// itself, so such a request gets here only when // such a request gets here only when
// SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION allowed it. // SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION allowed it.
// ctx is the request's own context. // ctx is the request's own context.
func (rq *request) passToApp(ctx context.Context) { func (rq *request) passToApp(ctx context.Context) {
@@ -232,7 +234,7 @@ func (rq *request) passToApp(ctx context.Context) {
switch { switch {
case rq.refused.Load() != 0 || (maxBytes > 0 && rq.in.ContentLength > maxBytes): case rq.refused.Load() != 0 || (maxBytes > 0 && rq.in.ContentLength > maxBytes):
rq.answer(http.StatusRequestEntityTooLarge) rq.answer(http.StatusRequestEntityTooLarge)
case strings.HasPrefix(rq.in.URL.Path, wafprotocol.OwnEndpointPrefix): case strings.HasPrefix(rq.in.URL.Path, ownEndpointPrefix):
rq.answer(http.StatusNotFound) rq.answer(http.StatusNotFound)
default: default:
rq.forward(ctx) rq.forward(ctx)
+24 -24
View File
@@ -21,9 +21,9 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/config/configtest" "sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/processlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rproxy" "sneak.berlin/go/smallwebwaf/internal/rproxy"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/waf"
) )
const ( const (
@@ -56,7 +56,7 @@ const (
// The headers the tests set and look at. // The headers the tests set and look at.
const ( const (
dispositionHeader = wafprotocol.DispositionHeader dispositionHeader = waf.DispositionHeader
forwardedForHeader = "X-Forwarded-For" forwardedForHeader = "X-Forwarded-For"
forwardedProtoHeader = "X-Forwarded-Proto" forwardedProtoHeader = "X-Forwarded-Proto"
requestIDHeader = "X-Request-ID" requestIDHeader = "X-Request-ID"
@@ -72,9 +72,9 @@ const (
// What the stand-in for waf answers. // What the stand-in for waf answers.
const ( const (
dispositionAllow = wafprotocol.DispositionAllow dispositionAllow = waf.DispositionAllow
dispositionDeny = wafprotocol.DispositionDeny dispositionDeny = waf.DispositionDeny
dispositionClose = wafprotocol.DispositionClose dispositionClose = waf.DispositionClose
// deniedBody is the body of a deny. // deniedBody is the body of a deny.
deniedBody = "refused by waf" deniedBody = "refused by waf"
) )
@@ -175,7 +175,7 @@ type WAF struct {
*httptest.Server *httptest.Server
questions chan question questions chan question
reports chan wafprotocol.Report reports chan waf.Report
} }
// startWAF starts a stand-in for waf. // startWAF starts a stand-in for waf.
@@ -184,11 +184,11 @@ func startWAF(t *testing.T, answer http.HandlerFunc) *WAF {
s := &WAF{ s := &WAF{
questions: make(chan question, 10), questions: make(chan question, 10),
reports: make(chan wafprotocol.Report, 10), reports: make(chan waf.Report, 10),
} }
s.Server = httptest.NewServer(http.HandlerFunc( s.Server = httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) { func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(wafprotocol.ClientHeader) == "" { if r.Header.Get(waf.ClientHeader) == "" {
s.takeReport(t, w, r) s.takeReport(t, w, r)
return return
@@ -217,11 +217,11 @@ func (s *WAF) takeReport(
) { ) {
t.Helper() t.Helper()
var report wafprotocol.Report var report waf.Report
err := json.NewDecoder(r.Body).Decode(&report) err := json.NewDecoder(r.Body).Decode(&report)
if err != nil || r.Method != http.MethodPost || if err != nil || r.Method != http.MethodPost ||
r.URL.Path != wafprotocol.ReportPath { r.URL.Path != waf.ReportPath {
t.Errorf("%s %s is no report: %v", r.Method, r.URL.Path, err) t.Errorf("%s %s is no report: %v", r.Method, r.URL.Path, err)
} }
@@ -242,7 +242,7 @@ func (s *WAF) wantNoReport(t *testing.T) {
} }
// report waits for a report. // report waits for a report.
func (s *WAF) report(t *testing.T) wafprotocol.Report { func (s *WAF) report(t *testing.T) waf.Report {
t.Helper() t.Helper()
select { select {
@@ -251,7 +251,7 @@ func (s *WAF) report(t *testing.T) wafprotocol.Report {
case <-time.After(waitLimit): case <-time.After(waitLimit):
t.Fatal("no report") t.Fatal("no report")
return wafprotocol.Report{} return waf.Report{}
} }
} }
@@ -279,7 +279,7 @@ func startWAFAnsweringReports(
backend := httptest.NewServer(http.HandlerFunc( backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) { func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(wafprotocol.ClientHeader) != "" { if r.Header.Get(waf.ClientHeader) != "" {
answer(w, r) answer(w, r)
return return
@@ -383,11 +383,11 @@ func newServer(
return rproxy.New(rproxy.Params{ return rproxy.New(rproxy.Params{
Config: cfg, Config: cfg,
RequestLog: requestLog, RequestLog: requestLog,
ProcessLog: processlog.New(processlog.Params{ ProcessLog: requestlog.NewProcessLogger(
Output: processLog, processLog,
InstanceName: "test", "test",
Level: slog.LevelInfo, slog.LevelInfo,
}), ),
}) })
} }
@@ -829,7 +829,7 @@ func TestAnAnswerRProxyHoldsWholeIsSentBeforeItsLogLine(t *testing.T) {
}{ }{
{"a deny", "GET /" + host, deniedBody, http.StatusForbidden}, {"a deny", "GET /" + host, deniedBody, http.StatusForbidden},
{"rproxy's own answer", "GET x:admin" + host, "Bad Request\n", http.StatusBadRequest}, {"rproxy's own answer", "GET x:admin" + host, "Bad Request\n", http.StatusBadRequest},
{"the health check", "GET " + wafprotocol.HealthPath + host, "ok\n", http.StatusOK}, {"the health check", "GET " + waf.HealthPath + host, "ok\n", http.StatusOK},
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
t.Parallel() t.Parallel()
@@ -1602,19 +1602,19 @@ func TestHealthCheckIsAnsweredWhileWAFIsDown(t *testing.T) {
app := startApp(t, greet) app := startApp(t, greet)
addr, out := startRProxy(t, nowhere, app.URL, nil) addr, out := startRProxy(t, nowhere, app.URL, nil)
got := get(t, addr, wafprotocol.HealthPath) got := get(t, addr, waf.HealthPath)
if got.status != http.StatusOK || got.body != "ok\n" { if got.status != http.StatusOK || got.body != "ok\n" {
t.Errorf("answered %+v, want ok", got) t.Errorf("answered %+v, want ok", got)
} }
line := out.requestLine(t) line := out.requestLine(t)
if line["path"] != wafprotocol.HealthPath || line["disposition"] != nil { if line["path"] != waf.HealthPath || line["disposition"] != nil {
t.Errorf("line %v, want one with no disposition", line) t.Errorf("line %v, want one with no disposition", line)
} }
// With waf up too, it is not asked. // With waf up too, it is not asked.
addr, _ = startRProxy(t, standIn.URL, app.URL, nil) addr, _ = startRProxy(t, standIn.URL, app.URL, nil)
get(t, addr, wafprotocol.HealthPath) get(t, addr, waf.HealthPath)
if len(standIn.questions) != 0 { if len(standIn.questions) != 0 {
t.Error("waf was asked about the health check") t.Error("waf was asked about the health check")
@@ -1775,7 +1775,7 @@ func TestWaitForReportsWaitsForAReportBeingSent(t *testing.T) {
release := make(chan struct{}) release := make(chan struct{})
backend := httptest.NewServer(http.HandlerFunc( backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) { func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(wafprotocol.ClientHeader) != "" { if r.Header.Get(waf.ClientHeader) != "" {
allow(w, r) allow(w, r)
return return
+82 -34
View File
@@ -1,7 +1,9 @@
package rproxy package rproxy
import ( import (
"bytes"
"context" "context"
"encoding/json"
"errors" "errors"
"fmt" "fmt"
"io" "io"
@@ -9,8 +11,7 @@ import (
"net/http" "net/http"
"strings" "strings"
"sneak.berlin/go/smallwebwaf/internal/rules" "sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// answerBodyMaxBytes is the most of the body of waf's answer that is read. // answerBodyMaxBytes is the most of the body of waf's answer that is read.
@@ -39,12 +40,15 @@ type wafAnswer struct {
// notes nothing, and no report is due for it: waf holds no decision to report // notes nothing, and no report is due for it: waf holds no decision to report
// on. // on.
func (rq *request) noteAnswer(answer *wafAnswer) { func (rq *request) noteAnswer(answer *wafAnswer) {
read, valid := wafprotocol.ReadAnswer(answer.header) disposition := answer.header.Get(waf.DispositionHeader)
if valid {
rq.disposition = read.Disposition switch disposition {
case waf.DispositionAllow, waf.DispositionDeny,
waf.DispositionClose:
rq.disposition = disposition
rq.answered = true rq.answered = true
rq.asn = read.ASN rq.asn = answer.header.Get(asnHeader)
rq.country = read.Country rq.country = answer.header.Get(countryHeader)
} }
} }
@@ -75,7 +79,7 @@ func (rq *request) handleNoUsableAnswer(ctx context.Context, err error) {
rq.disposition = rq.h.config.WAFBackendNoAnswerDisposition rq.disposition = rq.h.config.WAFBackendNoAnswerDisposition
rq.wafNoAnswer = true rq.wafNoAnswer = true
if rq.disposition == wafprotocol.DispositionAllow { if rq.disposition == waf.DispositionAllow {
rq.passToApp(ctx) rq.passToApp(ctx)
} else { } else {
rq.answer(http.StatusServiceUnavailable) rq.answer(http.StatusServiceUnavailable)
@@ -115,21 +119,7 @@ func (rq *request) ask(ctx context.Context) (*wafAnswer, error) {
ctx, cancel := context.WithTimeout(ctx, rq.h.config.WAFBackendTimeout) ctx, cancel := context.WithTimeout(ctx, rq.h.config.WAFBackendTimeout)
defer cancel() defer cancel()
var held []byte question, err := rq.newQuestion(ctx)
if rq.body != nil {
held = rq.body.held
}
question, err := wafprotocol.NewQuestion(ctx, wafprotocol.QuestionParams{
Visitor: rq.in,
PathAsSent: rules.Path(rq.in),
Body: held,
WAFBackend: rq.h.config.WAFBackend,
Client: rq.client,
Peer: rq.peer,
RequestID: rq.id,
Scheme: scheme(rq.in, rq.peerTrusted),
})
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -186,13 +176,69 @@ func canBeAsked(r *http.Request) bool {
len(r.Header.Values("User-Agent")) <= 1 len(r.Header.Values("User-Agent")) <= 1
} }
// newQuestion returns the question about the request, as "The question" in
// SPEC.md gives it: the visitor's request as the visitor sent it, with the part
// of the body it holds and rproxy's headers.
func (rq *request) newQuestion(ctx context.Context) (*http.Request, error) {
in := rq.in
var held []byte
if rq.body != nil {
held = rq.body.held
}
question, err := http.NewRequestWithContext( //nolint:gosec // a setting's URL
ctx,
in.Method,
rq.h.config.WAFBackend.String(),
bytes.NewReader(held),
)
if err != nil {
return nil, fmt.Errorf("make the question: %w", err)
}
backend := question.URL
target := *in.URL
target.Scheme = backend.Scheme
target.Host = backend.Host
setPathAsSent(&target, in)
question.URL = &target
question.Host = in.Host
// The length NewRequestWithContext set is that of what is held. This line makes
// a body whose length the visitor did not announce go chunked, as "The question"
// in SPEC.md requires.
question.TransferEncoding = in.TransferEncoding
question.Header = in.Header.Clone()
question.Header.Set(waf.ClientHeader, rq.client.String())
question.Header.Set(waf.PeerHeader, rq.peer.String())
question.Header.Del(waf.ContentLengthHeader)
announced := in.Header.Get("Content-Length")
if announced != "" {
question.Header.Set(waf.ContentLengthHeader, announced)
}
question.Header.Set(requestIDHeader, rq.id)
question.Header.Set("X-Forwarded-Proto", scheme(in, rq.peerTrusted))
// Go's client adds a User-Agent of its own to a request without the key.
_, sent := question.Header["User-Agent"]
if !sent {
question.Header["User-Agent"] = nil
}
return question, nil
}
// answerWith gives the visitor waf's answer, less its disposition. // answerWith gives the visitor waf's answer, less its disposition.
func (rq *request) answerWith(answer *wafAnswer) { func (rq *request) answerWith(answer *wafAnswer) {
rq.prepareOwnAnswer(rq.requestDeadline()) rq.prepareOwnAnswer(rq.requestDeadline())
header := rq.out.Header() header := rq.out.Header()
maps.Copy(header, answer.header) maps.Copy(header, answer.header)
header.Del(wafprotocol.DispositionHeader) header.Del(waf.DispositionHeader)
// Go's server adds a Content-Type it guesses from the body to a response whose // Go's server adds a Content-Type it guesses from the body to a response whose
// headers have no Content-Type key. The key with no value sends none. // headers have no Content-Type key. The key with no value sends none.
@@ -206,10 +252,7 @@ func (rq *request) answerWith(answer *wafAnswer) {
// sendReport sends waf report on a goroutine of its own, so that the // sendReport sends waf report on a goroutine of its own, so that the
// visitor does not wait for it. The report goes on after ctx, the request's own // visitor does not wait for it. The report goes on after ctx, the request's own
// context, has ended. // context, has ended.
func (h *handler) sendReport( func (h *handler) sendReport(ctx context.Context, report *waf.Report) {
ctx context.Context,
report *wafprotocol.Report,
) {
h.reportsMu.Lock() h.reportsMu.Lock()
defer h.reportsMu.Unlock() defer h.reportsMu.Unlock()
@@ -227,18 +270,23 @@ func (h *handler) sendReport(
// postReport posts report to waf, and waits for its answer for up to // postReport posts report to waf, and waits for its answer for up to
// SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS. // SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS.
func (h *handler) postReport( func (h *handler) postReport(ctx context.Context, report *waf.Report) error {
ctx context.Context, body, err := json.Marshal(report)
report *wafprotocol.Report, if err != nil {
) error { return fmt.Errorf("encode the report: %w", err)
}
ctx, cancel := context.WithTimeout(ctx, h.config.WAFBackendTimeout) ctx, cancel := context.WithTimeout(ctx, h.config.WAFBackendTimeout)
defer cancel() defer cancel()
req, err := wafprotocol.NewReportRequest(ctx, h.reportURL, report) req, err := http.NewRequestWithContext( //nolint:gosec // a setting's URL
ctx, http.MethodPost, h.reportURL, bytes.NewReader(body))
if err != nil { if err != nil {
return err return fmt.Errorf("make the request: %w", err)
} }
req.Header.Set("Content-Type", "application/json")
res, err := h.wafTransport.RoundTrip(req) res, err := h.wafTransport.RoundTrip(req)
if err != nil { if err != nil {
return fmt.Errorf("send the report: %w", err) return fmt.Errorf("send the report: %w", err)
+48 -19
View File
@@ -18,11 +18,12 @@ import (
"slices" "slices"
"strings" "strings"
"sync/atomic" "sync/atomic"
"time"
"github.com/fsnotify/fsnotify"
"golang.org/x/net/http/httpguts" "golang.org/x/net/http/httpguts"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/dirwatch"
) )
// The actions a rule takes when it matches. // The actions a rule takes when it matches.
@@ -39,6 +40,12 @@ const (
// extension ends the name of every rule file. // extension ends the name of every rule file.
const extension = ".rules" const extension = ".rules"
// quietTime is how long SWW_WAF_RULE_FILES_DIR must go without a change before
// the rule files are read again, so that a file still being written, such
// as one saved in place, appended to or copied in with scp, is read only
// once whole.
const quietTime = 2 * time.Second
// headerTarget starts the target that is one request header, // headerTarget starts the target that is one request header,
// header:<Name>. // header:<Name>.
const headerTarget = "header:" const headerTarget = "header:"
@@ -169,18 +176,26 @@ func (f *Files) Len() int {
} }
// Watch watches Dir until ctx is done, and reads the rule files again // Watch watches Dir until ctx is done, and reads the rule files again
// once Dir has had no change for a short time, after one is edited, added // once Dir has had no change for quietTime, after one is edited, added or
// or removed, and after Watch starts watching, as dirwatch describes. If // removed, and after Watch starts watching. If they then hold an error,
// they then hold an error, the rules stay as they were, the error is // the rules stay as they were, the error is logged with its file and
// logged with its file and line, and the files are read again after the // line, and the files are read again after the next change. If Dir cannot
// next change. If Dir cannot be watched, that is logged, and the rules // be watched, that is logged, and the rules stay as they were loaded.
// stay as they were loaded. While Enabled is false, Watch returns at once. // While Enabled is false, Watch returns at once.
func (f *Files) Watch(ctx context.Context) { func (f *Files) Watch(ctx context.Context) {
if !f.params.Enabled { if !f.params.Enabled {
return return
} }
watcher, err := dirwatch.New(f.params.Dir) watcher, err := fsnotify.NewWatcher()
if err == nil {
defer func() {
_ = watcher.Close()
}()
err = watcher.Add(f.params.Dir)
}
if err != nil { if err != nil {
f.params.ProcessLog.Error("cannot watch the rule files for edits", f.params.ProcessLog.Error("cannot watch the rule files for edits",
"error", err.Error()) "error", err.Error())
@@ -188,22 +203,36 @@ func (f *Files) Watch(ctx context.Context) {
return return
} }
defer func() {
_ = watcher.Close()
}()
f.params.ProcessLog.Info("watching the rule files for edits", f.params.ProcessLog.Info("watching the rule files for edits",
"directory", f.params.Dir) "directory", f.params.Dir)
// A change to any file in Dir counts. f.readAfterChanges(ctx, watcher.Events, watcher.Errors)
watcher.CallWhenQuiet(ctx, }
func(string) bool { return true },
f.readAgain, // readAfterChanges reads the rule files again once quietTime has passed
func(err error) { // without a change from events, until ctx is done, and logs the errors
// from errs. The wait starts at once, as if for a change, so that an edit
// saved after Load read the files, and before Dir was watched, is taken
// in too.
func (f *Files) readAfterChanges(
ctx context.Context, events <-chan fsnotify.Event, errs <-chan error,
) {
quiet := time.NewTimer(quietTime)
defer quiet.Stop()
for {
select {
case <-ctx.Done():
return
case <-events:
quiet.Reset(quietTime)
case <-quiet.C:
f.readAgain()
case err := <-errs:
f.params.ProcessLog.Warn("watching the rule files failed", f.params.ProcessLog.Warn("watching the rule files failed",
"error", err.Error()) "error", err.Error())
}, }
) }
} }
// readAgain reads the rule files again, in place of the rules loaded, or // readAgain reads the rule files again, in place of the rules loaded, or
+165
View File
@@ -0,0 +1,165 @@
package rules
import (
"context"
"log/slog"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"slices"
"testing"
"testing/synctest"
"time"
"github.com/fsnotify/fsnotify"
"sneak.berlin/go/smallwebwaf/internal/alerts"
)
// The tests below run readAfterChanges in a synctest bubble, where time is
// a clock of the test's own: time.Sleep moves it on at once, and
// synctest.Wait returns once readAfterChanges waits again, so that every
// reading due by then is done. The test sends the changes itself, as the
// watch of a directory cannot run in a bubble.
func TestFileWrittenInTwoPartsTakenInOnlyWhole(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "50-app.rules")
writeFile(t, path, "first path block ^/first\n")
files := load(t, dir)
changes := run(t, files)
file, err := os.Create(path) //nolint:gosec // a file the test wrote
if err != nil {
t.Fatalf("create: %v", err)
}
defer func() {
_ = file.Close()
}()
// The first part ends in the middle of a ban rule's regex, which,
// read then, would ban every request.
write(t, file, "first path block ^/first\nprobe path ban ^/")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantMatched(t, files, "/anything")
// The second part starts the wait again.
write(t, file, `\.env$`+"\n")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantMatched(t, files, "/.env")
time.Sleep(time.Nanosecond)
synctest.Wait()
wantMatched(t, files, "/.env", "probe")
wantMatched(t, files, "/anything")
})
}
func TestEditSavedBeforeTheWatchStartsTakenIn(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "50-app.rules")
writeFile(t, path, "first path block ^/first\n")
files := load(t, dir)
// Saved after Load read the files, and before the directory was
// watched, so that no change is seen for it.
writeFile(t, path, "first path block ^/edited\n")
run(t, files)
time.Sleep(quietTime)
synctest.Wait()
wantMatched(t, files, "/edited", "first")
})
}
// load loads the rules in dir.
func load(t *testing.T, dir string) *Files {
t.Helper()
files, err := Load(Params{
Dir: dir, Enabled: true, ProcessLog: slog.New(slog.DiscardHandler),
Alerts: alerts.New(alerts.Params{}),
})
if err != nil {
t.Fatalf("load: %v", err)
}
return files
}
// run runs files' readAfterChanges until the test ends, and returns the
// channel that sends it changes.
func run(t *testing.T, files *Files) chan<- fsnotify.Event {
t.Helper()
changes := make(chan fsnotify.Event)
ctx, stop := context.WithCancel(t.Context())
stopped := make(chan struct{})
go func() {
files.readAfterChanges(ctx, changes, nil)
close(stopped)
}()
t.Cleanup(func() {
stop()
<-stopped
})
return changes
}
// writeFile writes content to the file at path.
func writeFile(t *testing.T, path, content string) {
t.Helper()
err := os.WriteFile(path, []byte(content), 0o600)
if err != nil {
t.Fatalf("write %s: %v", path, err)
}
}
// write writes text to the end of file.
func write(t *testing.T, file *os.File, text string) {
t.Helper()
_, err := file.WriteString(text)
if err != nil {
t.Fatalf("write: %v", err)
}
}
// wantMatched checks the ids of the rules that a GET request for path
// matches, in order.
func wantMatched(t *testing.T, files *Files, path string, want ...string) {
t.Helper()
r := httptest.NewRequestWithContext(t.Context(), http.MethodGet,
"http://app.example"+path, nil)
matched := files.Match(r)
got := make([]string, 0, len(matched))
for _, rule := range matched {
got = append(got, rule.ID)
}
if !slices.Equal(got, want) {
t.Errorf("%s matched %v, want %v", path, got, want)
}
}
+20 -32
View File
@@ -1284,18 +1284,18 @@ func TestBansWrittenOnceBansWriteDelayAfterABan(t *testing.T) {
// A second ban, made while the first waits to be written, puts the // A second ban, made while the first waits to be written, puts the
// write off no further, and is written with it. // write off no further, and is written with it.
first, _ := params.Ledger.Ban( first, _ := params.Ledger.BanForLimit(
netip.MustParsePrefix("203.0.113.9/32"), netip.MustParsePrefix("203.0.113.9/32"),
midnight(), midnight(),
bans.Grounds{Cause: bans.CauseLimit}, bans.Notes{},
) )
time.Sleep(5 * time.Second) time.Sleep(5 * time.Second)
second, _ := params.Ledger.Ban( second, _ := params.Ledger.BanForLimit(
netip.MustParsePrefix("203.0.113.10/32"), netip.MustParsePrefix("203.0.113.10/32"),
midnight(), midnight(),
bans.Grounds{Cause: bans.CauseLimit}, bans.Notes{},
) )
time.Sleep(5*time.Second - time.Nanosecond) time.Sleep(5*time.Second - time.Nanosecond)
@@ -1358,16 +1358,16 @@ func TestEditJustBeforeAScheduledWriteSurvivesIt(t *testing.T) {
run(t, load(t, params).Run) run(t, load(t, params).Run)
// A ban, and bans.json written with it. // A ban, and bans.json written with it.
params.Ledger.Ban(netip.MustParsePrefix("203.0.113.9/32"), params.Ledger.BanForLimit(netip.MustParsePrefix("203.0.113.9/32"),
midnight(), bans.Grounds{Cause: bans.CauseLimit}) midnight(), bans.Notes{})
time.Sleep(params.BansWriteDelay) time.Sleep(params.BansWriteDelay)
synctest.Wait() synctest.Wait()
// A second ban is to be written BansWriteDelay later. Just before // A second ban is to be written BansWriteDelay later. Just before
// then, an admin saves bans.json with the first ban lifted and // then, an admin saves bans.json with the first ban lifted and
// another added. // another added.
params.Ledger.Ban(netip.MustParsePrefix("203.0.113.10/32"), params.Ledger.BanForLimit(netip.MustParsePrefix("203.0.113.10/32"),
midnight(), bans.Grounds{Cause: bans.CauseLimit}) midnight(), bans.Notes{})
time.Sleep(params.BansWriteDelay - time.Nanosecond) time.Sleep(params.BansWriteDelay - time.Nanosecond)
synctest.Wait() synctest.Wait()
edit(t, dir, bansJSON, permanentBansJSON) edit(t, dir, bansJSON, permanentBansJSON)
@@ -1483,10 +1483,10 @@ func TestFailedWriteLeavesTheFileAsItWas(t *testing.T) {
t.Fatalf("mkdir: %v", err) t.Fatalf("mkdir: %v", err)
} }
params.Ledger.Ban( params.Ledger.BanForLimit(
netip.MustParsePrefix("203.0.113.9/32"), netip.MustParsePrefix("203.0.113.9/32"),
midnight(), midnight(),
bans.Grounds{Cause: bans.CauseLimit}, bans.Notes{},
) )
params.Limiter.Count( params.Limiter.Count(
netip.MustParsePrefix("203.0.113.9/32"), netip.MustParsePrefix("203.0.113.9/32"),
@@ -1556,10 +1556,10 @@ func TestWritesAreCountedInTheMetrics(t *testing.T) {
t.Fatalf("mkdir: %v", err) t.Fatalf("mkdir: %v", err)
} }
params.Ledger.Ban( params.Ledger.BanForLimit(
netip.MustParsePrefix("203.0.113.9/32"), netip.MustParsePrefix("203.0.113.9/32"),
midnight(), midnight(),
bans.Grounds{Cause: bans.CauseLimit}, bans.Notes{},
) )
err = files.WriteAll() err = files.WriteAll()
@@ -2135,11 +2135,7 @@ func TestBanLiftedByAnEditWhileRunning(t *testing.T) {
// The ban that liftedBansJSON lifts, before it is lifted. // The ban that liftedBansJSON lifts, before it is lifted.
netblock := netip.MustParsePrefix(liftedClient + "/32") netblock := netip.MustParsePrefix(liftedClient + "/32")
params.Ledger.Ban( params.Ledger.BanForLimit(netblock, midnight(), bans.Notes{})
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
_, banned, _ := params.Ledger.Find(netblock.Addr(), afterLifting()) _, banned, _ := params.Ledger.Find(netblock.Addr(), afterLifting())
if !banned { if !banned {
@@ -2430,7 +2426,7 @@ func midnight() time.Time {
// counts, with thresholds fill does not reach. // counts, with thresholds fill does not reach.
func newParams(dir string) state.Params { func newParams(dir string) state.Params {
discard := slog.New(slog.DiscardHandler) discard := slog.New(slog.DiscardHandler)
m := metrics.New(metrics.Params{BusiestCount: 1, InstanceName: "app"}) m := metrics.New(1, "app")
queue := alerts.New(alerts.Params{ queue := alerts.New(alerts.Params{
WebhookURL: &url.URL{Scheme: "https", Host: "alerts.example"}, WebhookURL: &url.URL{Scheme: "https", Host: "alerts.example"},
Events: alerts.Events(), Events: alerts.Events(),
@@ -2452,7 +2448,7 @@ func newParams(dir string) state.Params {
AttackBanDuration: 7 * 24 * time.Hour, AttackBanDuration: 7 * 24 * time.Hour,
MaxBans: 5000, MaxBans: 5000,
}), }),
Limiter: ratelimit.New(ratelimit.Params{MaxClients: 20000}), Limiter: ratelimit.New(ratelimit.Limits{}, 20000),
GeoJS: geojs.New(geojs.Params{Now: midnight}), GeoJS: geojs.New(geojs.Params{Now: midnight}),
Lists: reputation.New(reputation.Params{ Lists: reputation.New(reputation.Params{
BlocklistURLs: []string{ BlocklistURLs: []string{
@@ -2514,15 +2510,11 @@ func fill(params state.Params) {
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
params.Ledger.Load([]bans.Ban{permanentBan()}) params.Ledger.Load([]bans.Ban{permanentBan()})
params.Ledger.Ban(client, now, bans.Grounds{ params.Ledger.BanForLimit(client, now, bans.Notes{
Cause: bans.CauseLimit, ASN: asn, ASName: asName, Country: "DE", Limit: 1,
Notes: bans.Notes{ASN: asn, ASName: asName, Country: "DE", Limit: 1},
})
params.Ledger.Ban(netip.MustParsePrefix("192.0.2.1/32"), now,
bans.Grounds{
Cause: bans.CauseAttack,
Notes: bans.Notes{RuleID: "env-file", Target: "path"},
}) })
params.Ledger.BanForAttack(netip.MustParsePrefix("192.0.2.1/32"), now,
bans.Notes{RuleID: "env-file", Target: "path"})
for _, c := range []string{"2001:db8::/64", "203.0.113.9/32", "192.0.2.1/32"} { for _, c := range []string{"2001:db8::/64", "203.0.113.9/32", "192.0.2.1/32"} {
params.Limiter.Count(netip.MustParsePrefix(c), now, whole) params.Limiter.Count(netip.MustParsePrefix(c), now, whole)
@@ -2708,11 +2700,7 @@ func wantLiftedBanKept(
} }
// Were the lifted ban counted, the next would last three hours. // Were the lifted ban counted, the next would last three hours.
ban, _ := ledger.Ban( ban, _ := ledger.BanForLimit(netblock, afterLifting(), bans.Notes{})
netblock,
afterLifting(),
bans.Grounds{Cause: bans.CauseLimit},
)
if ban.Expires.Sub(ban.Start) != time.Hour { if ban.Expires.Sub(ban.Start) != time.Hour {
t.Errorf( t.Errorf(
"the next ban lasts %s, want 1h", "the next ban lasts %s, want 1h",
+2 -2
View File
@@ -13,7 +13,7 @@ import (
"github.com/spf13/viper" "github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/waf"
) )
// healthCheckTimeout bounds the whole health check. // healthCheckTimeout bounds the whole health check.
@@ -116,7 +116,7 @@ func askHealthEndpoint(ctx context.Context, listenAddr string) error {
health := "http://" + net.JoinHostPort( health := "http://" + net.JoinHostPort(
"127.0.0.1", "127.0.0.1",
port, port,
) + wafprotocol.HealthPath ) + waf.HealthPath
req, err := http.NewRequestWithContext( req, err := http.NewRequestWithContext(
ctx, ctx,
+6 -6
View File
@@ -10,7 +10,7 @@ import (
"go.uber.org/fx" "go.uber.org/fx"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/processlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rproxy" "sneak.berlin/go/smallwebwaf/internal/rproxy"
) )
@@ -23,11 +23,11 @@ import (
// setting for either. // setting for either.
func RunRProxy(ctx context.Context, params Params) int { func RunRProxy(ctx context.Context, params Params) int {
hostname, _ := os.Hostname() // "" when the host has no name to give hostname, _ := os.Hostname() // "" when the host has no name to give
processLog := processlog.New(processlog.Params{ processLog := requestlog.NewProcessLogger(
Output: params.Stdout, params.Stdout,
InstanceName: hostname, hostname,
Level: slog.LevelInfo, slog.LevelInfo,
}) )
cfg, err := config.RProxyFromEnvironment(params.Viper) cfg, err := config.RProxyFromEnvironment(params.Viper)
if err != nil { if err != nil {
+15 -15
View File
@@ -21,7 +21,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config/configtest" "sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/sww" "sneak.berlin/go/smallwebwaf/internal/sww"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/waf"
) )
const ( const (
@@ -164,7 +164,7 @@ func TestABanThroughOneSiteRefusesTheClientOnTheOther(t *testing.T) {
// One request's three lines carry one id, rproxy's with waf's answer. // One request's three lines carry one id, rproxy's with waf's answer.
line := firstOut.line(t, "path", bannedProbe) line := firstOut.line(t, "path", bannedProbe)
if line["disposition"] != wafprotocol.DispositionDeny || if line["disposition"] != waf.DispositionDeny ||
line["client_ip"] != placed { line["client_ip"] != placed {
t.Errorf("rproxy's line is %v", line) t.Errorf("rproxy's line is %v", line)
} }
@@ -249,7 +249,7 @@ func TestTheSiteAnswersWhileWAFIsStoppedAndStartedAgain(t *testing.T) {
wantStatus(t, site+"/after-stop", placed, http.StatusOK) wantStatus(t, site+"/after-stop", placed, http.StatusOK)
line := out.line(t, "path", "/after-stop") line := out.line(t, "path", "/after-stop")
if line["disposition"] != wafprotocol.DispositionAllow || if line["disposition"] != waf.DispositionAllow ||
line["waf_backend_no_answer"] != true { line["waf_backend_no_answer"] != true {
t.Errorf("rproxy's line is %v", line) t.Errorf("rproxy's line is %v", line)
} }
@@ -437,7 +437,7 @@ func TestAChunkedBodyStaysABodyAtWAF(t *testing.T) {
const smuggledID = "smuggled-request" const smuggledID = "smuggled-request"
smuggled := "GET " + bannedProbe + " HTTP/1.1\r\nHost: site.example\r\n" + smuggled := "GET " + bannedProbe + " HTTP/1.1\r\nHost: site.example\r\n" +
wafprotocol.ClientHeader + ": " + otherClient + "\r\n" + waf.ClientHeader + ": " + otherClient + "\r\n" +
"X-Request-ID: " + smuggledID + "\r\n\r\n" "X-Request-ID: " + smuggledID + "\r\n\r\n"
status := sendRaw( status := sendRaw(
@@ -605,11 +605,10 @@ func TestRProxyAsksAWAFThatSpeaksHTTP2OverHTTP1(t *testing.T) {
protocols := make(chan string, 10) protocols := make(chan string, 10)
backend := httptest.NewUnstartedServer(http.HandlerFunc( backend := httptest.NewUnstartedServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) { func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(wafprotocol.ClientHeader) != "" { if r.Header.Get(waf.ClientHeader) != "" {
protocols <- r.Proto protocols <- r.Proto
w.Header(). w.Header().Set(waf.DispositionHeader, waf.DispositionDeny)
Set(wafprotocol.DispositionHeader, wafprotocol.DispositionDeny)
w.WriteHeader(http.StatusForbidden) w.WriteHeader(http.StatusForbidden)
} }
})) }))
@@ -691,9 +690,8 @@ func TestRProxyStoppingWaitsForAReportBeingSent(t *testing.T) {
release := make(chan struct{}) release := make(chan struct{})
backend := httptest.NewServer(http.HandlerFunc( backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) { func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(wafprotocol.ClientHeader) != "" { if r.Header.Get(waf.ClientHeader) != "" {
w.Header(). w.Header().Set(waf.DispositionHeader, waf.DispositionAllow)
Set(wafprotocol.DispositionHeader, wafprotocol.DispositionAllow)
return return
} }
@@ -781,7 +779,10 @@ func sendAndKeepOpen(t *testing.T, addr, request string) net.Conn {
func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) { func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
t.Parallel() t.Parallel()
const cutOffAfter = 5 * time.Second const (
cutOffAfter = 5 * time.Second
upstreamSetting = "SWW_RPROXY_UPSTREAM_URL"
)
asked := make(chan struct{}) asked := make(chan struct{})
release := make(chan struct{}) release := make(chan struct{})
@@ -795,8 +796,7 @@ func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
backend := httptest.NewServer(http.HandlerFunc( backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) { func(w http.ResponseWriter, _ *http.Request) {
w.Header(). w.Header().Set(waf.DispositionHeader, waf.DispositionAllow)
Set(wafprotocol.DispositionHeader, wafprotocol.DispositionAllow)
})) }))
t.Cleanup(backend.Close) t.Cleanup(backend.Close)
@@ -805,7 +805,7 @@ func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
env := map[string]string{ env := map[string]string{
rproxyListenAddr: localhost + ":0", rproxyListenAddr: localhost + ":0",
rproxyUpstreamURL: app.URL, upstreamSetting: app.URL,
wafBackend: backend.URL, wafBackend: backend.URL,
} }
out := &output{} out := &output{}
@@ -932,7 +932,7 @@ func TestMainRunsRProxy(t *testing.T) {
addr, stop := startRProxyProcess(t, wafBackend+"=http://127.0.0.1:9") addr, stop := startRProxyProcess(t, wafBackend+"=http://127.0.0.1:9")
req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, req, err := http.NewRequestWithContext(t.Context(), http.MethodGet,
"http://"+addr+wafprotocol.HealthPath, http.NoBody) "http://"+addr+waf.HealthPath, http.NoBody)
if err != nil { if err != nil {
t.Fatalf("new request: %v", err) t.Fatalf("new request: %v", err)
} }
+29 -51
View File
@@ -10,14 +10,13 @@ package sww
import ( import (
"context" "context"
"errors" "fmt"
"io" "io"
"os" "os"
"os/signal" "os/signal"
"syscall" "syscall"
"time" "time"
"github.com/spf13/cobra"
"github.com/spf13/viper" "github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
@@ -27,7 +26,14 @@ import (
// know. // know.
const usageStatus = 2 const usageStatus = 2
var errNoCommand = errors.New("no command given") // usage is what sww prints then.
const usage = `usage: sww <command>
commands:
rproxy the reverse proxy in front of one app
waf decides for rproxy, and keeps bans, counters and history
healthcheck the container's health check
`
// Params are what RunWAF and RunRProxy need from the process. // Params are what RunWAF and RunRProxy need from the process.
type Params struct { type Params struct {
@@ -45,9 +51,14 @@ type Params struct {
// Main runs the command os.Args names, and returns the process's exit status. // Main runs the command os.Args names, and returns the process's exit status.
// `sww rproxy` and `sww waf` run until SIGTERM or SIGINT. // `sww rproxy` and `sww waf` run until SIGTERM or SIGINT.
// With no command, or one it does not know, cobra prints the error and the // With no command, or one it does not know, it prints the commands and returns
// commands to stderr, and Main returns usageStatus. // usageStatus.
func Main(version string) int { func Main(version string) int {
args := os.Args[1:]
if len(args) == 0 {
return printUsage(os.Stderr)
}
ctx, stop := signal.NotifyContext(context.Background(), ctx, stop := signal.NotifyContext(context.Background(),
syscall.SIGTERM, os.Interrupt) syscall.SIGTERM, os.Interrupt)
defer stop() defer stop()
@@ -58,54 +69,21 @@ func Main(version string) int {
Stdout: os.Stdout, Stdout: os.Stdout,
} }
// status is what the command that ran returns. switch args[0] {
status := 0 case "rproxy":
return RunRProxy(ctx, params)
root := &cobra.Command{ case "waf":
Use: "sww", return RunWAF(ctx, params)
Short: "a web application firewall", case "healthcheck":
Args: cobra.NoArgs, return HealthCheck(ctx, args[1:], params.Viper, os.Stderr)
RunE: func(*cobra.Command, []string) error { default:
return errNoCommand return printUsage(os.Stderr)
}, }
} }
root.CompletionOptions.DisableDefaultCmd = true
root.AddCommand( // printUsage writes the commands to w, and returns usageStatus.
&cobra.Command{ func printUsage(w io.Writer) int {
Use: "rproxy", _, _ = fmt.Fprint(w, usage)
Short: "the reverse proxy in front of one app",
Run: func(cmd *cobra.Command, _ []string) {
status = RunRProxy(cmd.Context(), params)
},
},
&cobra.Command{
Use: "waf",
Short: "decides for rproxy, and keeps bans, counters and history",
Run: func(cmd *cobra.Command, _ []string) {
status = RunWAF(cmd.Context(), params)
},
},
&cobra.Command{
Use: "healthcheck",
Short: "the container's health check",
// HealthCheck refuses every argument itself, flags included.
DisableFlagParsing: true,
Run: func(cmd *cobra.Command, args []string) {
status = HealthCheck(
cmd.Context(),
args,
params.Viper,
cmd.ErrOrStderr(),
)
},
},
)
err := root.ExecuteContext(ctx)
if err != nil {
return usageStatus return usageStatus
} }
return status
}
+2 -2
View File
@@ -23,7 +23,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config/configtest" "sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest" "sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
"sneak.berlin/go/smallwebwaf/internal/sww" "sneak.berlin/go/smallwebwaf/internal/sww"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/waf"
) )
const ( const (
@@ -1813,7 +1813,7 @@ func statusFrom(t *testing.T, url, from string) int {
} }
req.Header.Set("X-Forwarded-For", from) req.Header.Set("X-Forwarded-For", from)
req.Header.Set(wafprotocol.ClientHeader, from) req.Header.Set(waf.ClientHeader, from)
req.Header.Set("X-Request-ID", rand.Text()) req.Header.Set("X-Request-ID", rand.Text())
transport := &http.Transport{} transport := &http.Transport{}
+21 -34
View File
@@ -19,10 +19,10 @@ import (
"sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/lookup" "sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/metrics" "sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/processlog"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/remotelog" "sneak.berlin/go/smallwebwaf/internal/remotelog"
"sneak.berlin/go/smallwebwaf/internal/reputation" "sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/routewatch" "sneak.berlin/go/smallwebwaf/internal/routewatch"
"sneak.berlin/go/smallwebwaf/internal/rules" "sneak.berlin/go/smallwebwaf/internal/rules"
"sneak.berlin/go/smallwebwaf/internal/state" "sneak.berlin/go/smallwebwaf/internal/state"
@@ -43,11 +43,8 @@ const remoteLogStopTimeout = 2 * time.Second
func RunWAF(ctx context.Context, params Params) int { func RunWAF(ctx context.Context, params Params) int {
// Until the settings are read, the one message is an invalid setting's // Until the settings are read, the one message is an invalid setting's
// error, which every SWW_WAF_LOG_LEVEL lets through. // error, which every SWW_WAF_LOG_LEVEL lets through.
processLog := processlog.New(processlog.Params{ processLog := requestlog.NewProcessLogger(params.Stdout,
Output: params.Stdout, config.InstanceName(params.Viper), slog.LevelError)
InstanceName: config.InstanceName(params.Viper),
Level: slog.LevelError,
})
cfg, err := config.WAFFromEnvironment(params.Viper) cfg, err := config.WAFFromEnvironment(params.Viper)
if err != nil { if err != nil {
@@ -66,11 +63,11 @@ func RunWAF(ctx context.Context, params Params) int {
stdout = io.MultiWriter(params.Stdout, remote) stdout = io.MultiWriter(params.Stdout, remote)
} }
processLog = processlog.New(processlog.Params{ processLog = requestlog.NewProcessLogger(
Output: stdout, stdout,
InstanceName: cfg.InstanceName, cfg.InstanceName,
Level: cfg.LogLevel, cfg.LogLevel,
}) )
if remote != nil { if remote != nil {
stopSending := startSending(ctx, remote, processLog) stopSending := startSending(ctx, remote, processLog)
@@ -242,17 +239,14 @@ func newLookupFile(
// newLimiter returns the table of clients, with the rate and byte limits. // newLimiter returns the table of clients, with the rate and byte limits.
func newLimiter(cfg *config.WAF) *ratelimit.Limiter { func newLimiter(cfg *config.WAF) *ratelimit.Limiter {
return ratelimit.New(ratelimit.Params{ return ratelimit.New(ratelimit.Limits{
Limits: ratelimit.Limits{
PerMinute: cfg.RateLimitPerMinute, PerMinute: cfg.RateLimitPerMinute,
PerHour: cfg.RateLimitPerHour, PerHour: cfg.RateLimitPerHour,
PerDay: cfg.RateLimitPerDay, PerDay: cfg.RateLimitPerDay,
BytesPerMinute: cfg.BytesLimitPerMinute, BytesPerMinute: cfg.BytesLimitPerMinute,
BytesPerHour: cfg.BytesLimitPerHour, BytesPerHour: cfg.BytesLimitPerHour,
BytesPerDay: cfg.BytesLimitPerDay, BytesPerDay: cfg.BytesLimitPerDay,
}, }, cfg.MaxTrackedClients)
MaxClients: cfg.MaxTrackedClients,
})
} }
// newLedger returns the ledger of the bans, with the settings for them. // newLedger returns the ledger of the bans, with the settings for them.
@@ -392,33 +386,26 @@ func newMetrics(
alertQueue *alerts.Queue, lookupFile *lookup.File, alertQueue *alerts.Queue, lookupFile *lookup.File,
remote *remotelog.Sender, remote *remotelog.Sender,
) *metrics.Metrics { ) *metrics.Metrics {
params := metrics.Params{ m := metrics.New(cfg.MetricsBusiestCountriesAndASNs, cfg.InstanceName)
BusiestCount: cfg.MetricsBusiestCountriesAndASNs, m.AddBansAndClients(ledger, limiter, utcNow)
InstanceName: cfg.InstanceName, m.AddRules(ruleFiles)
Now: utcNow, m.AddReputation(lists, dnsbl)
Bans: ledger,
Clients: limiter, if cfg.AbuseIPDBKey != "" {
Rules: ruleFiles, m.AddAbuseIPDB(abuseIPDB)
Lists: lists,
DNSBL: dnsbl,
Alerts: alertQueue,
} }
// A nil pointer in an interface field is not a nil interface, so a part m.AddAlerts(alertQueue)
// that is off is left out rather than set to nil.
if cfg.AbuseIPDBKey != "" {
params.AbuseIPDB = abuseIPDB
}
if lookupFile != nil { if lookupFile != nil {
params.LookupFile = lookupFile m.AddLookupFile(lookupFile.LastRead, lookupFile.ReadFailures)
} }
if remote != nil { if remote != nil {
params.RemoteLog = remote m.AddRemoteLog(remote)
} }
return metrics.New(params) return m
} }
// newGeoJS returns what looks clients up at GeoJS, none looked up yet. Each // newGeoJS returns what looks clients up at GeoJS, none looked up yet. Each
+30 -95
View File
@@ -4,7 +4,6 @@ import (
"bytes" "bytes"
"context" "context"
"crypto/rand" "crypto/rand"
"errors"
"io" "io"
"net/http" "net/http"
"os" "os"
@@ -21,7 +20,7 @@ import (
_ "time/tzdata" _ "time/tzdata"
"sneak.berlin/go/smallwebwaf/internal/sww" "sneak.berlin/go/smallwebwaf/internal/sww"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/waf"
) )
// commandForMain, while set in the environment, has this test binary run Main // commandForMain, while set in the environment, has this test binary run Main
@@ -107,7 +106,7 @@ func TestMainRunsWAF(t *testing.T) {
starting := out.lineBeforeExit(t, "starting", exited) starting := out.lineBeforeExit(t, "starting", exited)
addr, _ := starting["address"].(string) addr, _ := starting["address"].(string)
ask(t, addr, "/", http.StatusOK, wafprotocol.DispositionAllow) ask(t, addr, "/", http.StatusOK, waf.DispositionAllow)
// The files are written as sww stops. // The files are written as sww stops.
err = cmd.Process.Signal(syscall.SIGTERM) err = cmd.Process.Signal(syscall.SIGTERM)
@@ -165,104 +164,46 @@ func wantUTC(t *testing.T, where string, value any) {
func TestMainWithoutACommandItKnowsPrintsTheCommands(t *testing.T) { func TestMainWithoutACommandItKnowsPrintsTheCommands(t *testing.T) {
t.Parallel() t.Parallel()
// completion is cobra's own command, which sww turns off.
for _, command := range []string{"", "serve", "completion"} {
t.Run("command "+command, func(t *testing.T) {
t.Parallel()
code, stdout, stderr := runMain(t, command)
if code != 2 {
t.Errorf("exit status %d, want 2", code)
}
wantTheCommands(t, "stderr", stderr)
if stdout != "" {
t.Errorf("stdout %q, want nothing", stdout)
}
})
}
}
func TestMainHelpPrintsTheCommands(t *testing.T) {
t.Parallel()
for _, command := range []string{"help", "--help"} {
t.Run("command "+command, func(t *testing.T) {
t.Parallel()
code, stdout, _ := runMain(t, command)
if code != 0 {
t.Errorf("exit status %d, want 0", code)
}
wantTheCommands(t, "stdout", stdout)
})
}
}
// wantTheCommands checks that text, what Main wrote to where, names each
// command.
func wantTheCommands(t *testing.T, where, text string) {
t.Helper()
for _, name := range []string{"rproxy", wafCommand, "healthcheck"} {
if !strings.Contains(text, name) {
t.Errorf("%s %q does not name %s", where, text, name)
}
}
}
func TestMainHealthCheckRefusesAnArgument(t *testing.T) {
t.Parallel()
for _, argument := range []string{"now", "--x", "-h"} {
t.Run("argument "+argument, func(t *testing.T) {
t.Parallel()
code, stdout, stderr := runMain(t, "healthcheck "+argument)
want := "sww healthcheck: unexpected argument \"" +
argument + "\"\n"
if code != 1 || stderr != want || stdout != "" {
t.Errorf("exit status %d, stdout %q and stderr %q, "+
"want 1, nothing and %q", code, stdout, stderr, want)
}
})
}
}
// runMain runs Main with command in a process of its own, and returns its
// exit status and what it wrote to stdout and stderr.
func runMain(t *testing.T, command string) (int, string, string) {
t.Helper()
binary, err := os.Executable() binary, err := os.Executable()
if err != nil { if err != nil {
t.Fatalf("find the test binary: %v", err) t.Fatalf("find the test binary: %v", err)
} }
for _, command := range []string{"", "serve"} {
t.Run("command "+command, func(t *testing.T) {
t.Parallel()
ctx, cancel := context.WithTimeout(t.Context(), waitLimit) ctx, cancel := context.WithTimeout(t.Context(), waitLimit)
defer cancel() defer cancel()
var stdout, stderr bytes.Buffer var stdout, stderr bytes.Buffer
//nolint:gosec // this test binary
cmd := exec.CommandContext(ctx, binary) cmd := exec.CommandContext(ctx, binary)
cmd.Stdout, cmd.Stderr = &stdout, &stderr cmd.Stdout, cmd.Stderr = &stdout, &stderr
cmd.Env = append(os.Environ(), commandForMain+"="+command) cmd.Env = append(os.Environ(), commandForMain+"="+command)
err = cmd.Run() err := cmd.Run()
var exitErr *exec.ExitError if code := cmd.ProcessState.ExitCode(); code != 2 {
if err != nil && !errors.As(err, &exitErr) { t.Errorf("exit status %d (%v), want 2", code, err)
t.Fatalf("run: %v", err)
} }
return cmd.ProcessState.ExitCode(), stdout.String(), stderr.String() for _, name := range []string{"rproxy", wafCommand, "healthcheck"} {
if !strings.Contains(stderr.String(), name) {
t.Errorf(
"stderr %q does not name %s",
stderr.String(),
name,
)
}
}
if stdout.Len() != 0 {
t.Errorf("stdout %q, want nothing", stdout.String())
}
})
}
} }
// ask sends waf at addr rproxy's question about a request for path from // ask sends waf at addr rproxy's question about a request for path from
@@ -276,7 +217,7 @@ func ask(t *testing.T, addr, path string, status int, disposition string) {
t.Fatalf("new request: %v", err) t.Fatalf("new request: %v", err)
} }
req.Header.Set(wafprotocol.ClientHeader, placed) req.Header.Set(waf.ClientHeader, placed)
req.Header.Set("X-Request-ID", rand.Text()) req.Header.Set("X-Request-ID", rand.Text())
transport := &http.Transport{} transport := &http.Transport{}
@@ -291,11 +232,11 @@ func ask(t *testing.T, addr, path string, status int, disposition string) {
_ = res.Body.Close() _ = res.Body.Close()
if res.StatusCode != status || if res.StatusCode != status ||
res.Header.Get(wafprotocol.DispositionHeader) != disposition { res.Header.Get(waf.DispositionHeader) != disposition {
t.Errorf( t.Errorf(
"answered %d with disposition %q, want %d with %q", "answered %d with disposition %q, want %d with %q",
res.StatusCode, res.StatusCode,
res.Header.Get(wafprotocol.DispositionHeader), res.Header.Get(waf.DispositionHeader),
status, status,
disposition, disposition,
) )
@@ -346,18 +287,12 @@ func TestWAFKeepsItsBansAcrossARestartAndNeedsNoApp(t *testing.T) {
} }
runWAF(t, env, func(addr string) { runWAF(t, env, func(addr string) {
ask(t, addr, "/", http.StatusOK, wafprotocol.DispositionAllow) ask(t, addr, "/", http.StatusOK, waf.DispositionAllow)
// The default rule file bans a probe for this. // The default rule file bans a probe for this.
ask( ask(t, addr, "/.env", http.StatusForbidden, waf.DispositionDeny)
t,
addr,
"/.env",
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
}) })
runWAF(t, env, func(addr string) { runWAF(t, env, func(addr string) {
ask(t, addr, "/", http.StatusForbidden, wafprotocol.DispositionDeny) ask(t, addr, "/", http.StatusForbidden, waf.DispositionDeny)
}) })
} }
+9 -20
View File
@@ -22,42 +22,31 @@ import (
// an offence, and the request does not wait for the answer. The score is then // an offence, and the request does not wait for the answer. The score is then
// used for each address of the client. Its figures are the score and // used for each address of the client. Its figures are the score and
// SWW_WAF_ABUSEIPDB_MIN_SCORE. // SWW_WAF_ABUSEIPDB_MIN_SCORE.
type abuseIPDB struct { type abuseIPDB struct{}
abuseIPDB *reputation.AbuseIPDB
// limiter is the table of clients the rate limits count, whose histories
// say which clients are offenders.
limiter *ratelimit.Limiter
keySet bool // SWW_WAF_ABUSEIPDB_KEY is set
minScore int64 // SWW_WAF_ABUSEIPDB_MIN_SCORE
action config.ReputationAction // SWW_WAF_DNSBL_ABUSEIPDB_ACTION
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
hitNoter reputationHitNoter
}
func (abuseIPDB) name() string { return "abuseipdb" } func (abuseIPDB) name() string { return "abuseipdb" }
func (a abuseIPDB) check( func (abuseIPDB) check(
ctx context.Context, ctx context.Context,
rq *request, rq *request,
_ time.Time, _ time.Time,
) *verdict { ) *verdict {
if !a.keySet { if rq.h.config.AbuseIPDBKey == "" {
return nil return nil
} }
client := clientGroup(a.ipv6GroupPrefix, rq.client) client := clientGroup(rq.h.config, rq.client)
held, _ := a.limiter.Client(client) held, _ := rq.h.limiter.Client(client)
offender := held.History.Offences != ratelimit.Offences{} offender := held.History.Offences != ratelimit.Offences{}
score, hit := a.abuseIPDB.Hit(ctx, client, rq.client, offender) score, hit := rq.h.abuseIPDB.Hit(ctx, client, rq.client, offender)
if !hit { if !hit {
return nil return nil
} }
rq.abuseIPDBHit = true rq.abuseIPDBHit = true
rq.addTag(config.AbuseIPDBTag) rq.addTag(config.AbuseIPDBTag)
a.hitNoter.noteHit( rq.noteHit(
rq,
bans.ReputationHit{ bans.ReputationHit{
Source: reputation.AbuseIPDBSource, Source: reputation.AbuseIPDBSource,
Score: &score, Score: &score,
@@ -65,14 +54,14 @@ func (a abuseIPDB) check(
"scored by AbuseIPDB at or over SWW_WAF_ABUSEIPDB_MIN_SCORE", "scored by AbuseIPDB at or over SWW_WAF_ABUSEIPDB_MIN_SCORE",
) )
if a.action != config.ReputationActionDeny { if rq.h.config.DNSBLAbuseIPDBAction != config.ReputationActionDeny {
return nil return nil
} }
return &verdict{ return &verdict{
action: requestlog.ActionClientDenied, action: requestlog.ActionClientDenied,
figures: map[string]any{ figures: map[string]any{
figureScore: score, "min_score": a.minScore, figureScore: score, "min_score": rq.h.config.AbuseIPDBMinScore,
}, },
} }
} }
+1 -3
View File
@@ -26,9 +26,7 @@ func BanAbuseIPDBBlacklist(
scores := make(map[netip.Prefix]int64, len(listed)) scores := make(map[netip.Prefix]int64, len(listed))
for _, client := range listed { for _, client := range listed {
netblock := banNetblock( netblock := banNetblock(cfg, client.Addr)
cfg.BanScopeV4Prefix, cfg.IPv6GroupPrefix, client.Addr,
)
scores[netblock] = max(scores[netblock], client.Score) scores[netblock] = max(scores[netblock], client.Score)
} }
+107 -58
View File
@@ -2,6 +2,7 @@ package waf
import ( import (
"bytes" "bytes"
"crypto/subtle"
"encoding/json" "encoding/json"
"errors" "errors"
"fmt" "fmt"
@@ -14,6 +15,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/state" "sneak.berlin/go/smallwebwaf/internal/state"
) )
@@ -39,23 +41,80 @@ var (
errNotAddress = errors.New("is not an address, such as 203.0.113.9") errNotAddress = errors.New("is not an address, such as 203.0.113.9")
) )
// handleHealth answers GET wafprotocol.HealthPath. // answerHealth answers a request for the health endpoint.
func handleHealth() http.HandlerFunc { func (rq *request) answerHealth() {
return func(w http.ResponseWriter, _ *http.Request) { rq.line.Action = requestlog.ActionOwnEndpoint
// Set here rather than left to Go's server, which would set it only // Set here rather than left to Go's server, which would set it only
// after the log line has taken the response's headers. // after the log line has taken the response's headers.
w.Header().Set("Content-Type", "text/plain; charset=utf-8") rq.out.Header().Set("Content-Type", "text/plain; charset=utf-8")
_, _ = io.WriteString(w, "ok\n") _, _ = io.WriteString(rq.out, "ok\n")
}
// answerAdmin answers a request for one of waf's own endpoints. Each
// needs a token, sent as Authorization: Bearer <token>: the metrics
// SWW_WAF_METRICS_TOKEN, the others SWW_WAF_ADMIN_TOKEN. A request without it is
// refused with 401. An endpoint whose token is unset answers 404, as any other
// path does.
func (rq *request) answerAdmin() {
rq.line.Action = requestlog.ActionOwnEndpoint
token, answer := rq.endpoint()
switch {
case token == "":
http.Error(
rq.out,
http.StatusText(http.StatusNotFound),
http.StatusNotFound,
)
case !hasToken(rq.in, token):
rq.out.Header().Set("WWW-Authenticate", "Bearer")
http.Error(
rq.out,
http.StatusText(http.StatusUnauthorized),
http.StatusUnauthorized,
)
default:
answer()
} }
} }
// handleListBans answers GET BansPath with every ban held. // endpoint returns the token the request's endpoint needs, and what
func (h *handler) handleListBans() http.HandlerFunc { // answers the request there; "" when there is no such endpoint.
return func(w http.ResponseWriter, _ *http.Request) { func (rq *request) endpoint() (string, func()) {
h.answerBans(w, h.ledger.Snapshot()) cfg := rq.h.config
method, path := rq.in.Method, rq.in.URL.Path
switch {
case method == http.MethodGet && path == MetricsPath:
return cfg.MetricsToken, func() { rq.h.metrics.ServeHTTP(rq.out, rq.in) }
case method == http.MethodGet && path == BansPath:
return cfg.AdminToken, rq.listBans
case method == http.MethodPost && path == BansPath:
return cfg.AdminToken, rq.addBan
case method == http.MethodDelete && strings.HasPrefix(path, BansPath+"/"):
return cfg.AdminToken, rq.liftBans
case method == http.MethodGet && strings.HasPrefix(path, ClientsPath):
return cfg.AdminToken, rq.showClient
default:
return "", nil
} }
} }
// hasToken reports whether r carries token, as Authorization: Bearer
// <token>.
func hasToken(r *http.Request, token string) bool {
scheme, sent, _ := strings.Cut(r.Header.Get("Authorization"), " ")
return strings.EqualFold(scheme, "Bearer") &&
subtle.ConstantTimeCompare([]byte(sent), []byte(token)) == 1
}
// listBans answers GET BansPath with every ban held.
func (rq *request) listBans() {
rq.answerBans(rq.h.ledger.Snapshot())
}
// banToAdd is the body of POST BansPath. // banToAdd is the body of POST BansPath.
type banToAdd struct { type banToAdd struct {
// Netblock is a netblock, or a client's address, which stands for the // Netblock is a netblock, or a client's address, which stands for the
@@ -67,21 +126,20 @@ type banToAdd struct {
Reason string `json:"reason"` Reason string `json:"reason"`
} }
// handleAddBan answers POST BansPath: it bans the netblock the body names, // addBan answers POST BansPath: it bans the netblock the body names, as
// as an admin, from now for the duration the body gives, with its reason, // an admin, from now for the duration the body gives, with its reason,
// and answers with that ban. // and answers with that ban.
func (h *handler) handleAddBan() http.HandlerFunc { func (rq *request) addBan() {
return func(w http.ResponseWriter, r *http.Request) { toAdd, err := rq.readBanToAdd()
toAdd, err := readBanToAdd(r)
var ( var (
netblock netip.Prefix netblock netip.Prefix
expires time.Time expires time.Time
now = h.now() now = rq.h.now()
) )
if err == nil { if err == nil {
netblock, err = h.parseBanNetblock(toAdd.Netblock) netblock, err = rq.h.parseBanNetblock(toAdd.Netblock)
} }
if err == nil { if err == nil {
@@ -89,22 +147,23 @@ func (h *handler) handleAddBan() http.HandlerFunc {
} }
if err != nil { if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest) http.Error(rq.out, err.Error(), http.StatusBadRequest)
return return
} }
ban := h.ledger.BanForAdmin(netblock, now, expires, toAdd.Reason) ban := rq.h.ledger.BanForAdmin(netblock, now, expires, toAdd.Reason)
h.answerBans(w, []bans.Ban{ban}) rq.answerBans([]bans.Ban{ban})
}
} }
// readBanToAdd reads the body of POST BansPath: a JSON object with // readBanToAdd reads the body of POST BansPath: a JSON object with
// nothing but whitespace after it, in at most banBodyMaxBytes. // nothing but whitespace after it, in at most banBodyMaxBytes.
func readBanToAdd(r *http.Request) (banToAdd, error) { func (rq *request) readBanToAdd() (banToAdd, error) {
data, err := io.ReadAll( data, err := io.ReadAll(
http.MaxBytesReader(nil, r.Body, banBodyMaxBytes), http.MaxBytesReader(nil, rq.in.Body, banBodyMaxBytes),
) )
rq.requestBytes = int64(len(data))
if err != nil { if err != nil {
return banToAdd{}, fmt.Errorf("%w: %w", errNotBanToAdd, err) return banToAdd{}, fmt.Errorf("%w: %w", errNotBanToAdd, err)
} }
@@ -165,11 +224,7 @@ func (h *handler) parseBanNetblock(value string) (netip.Prefix, error) {
return netip.Prefix{}, fmt.Errorf("netblock %q %w", value, errZone) return netip.Prefix{}, fmt.Errorf("netblock %q %w", value, errZone)
} }
return banNetblock( return banNetblock(h.config, addr), nil
h.config.BanScopeV4Prefix,
h.config.IPv6GroupPrefix,
addr,
), nil
} }
// expiry returns when a ban made at now for duration ends: duration // expiry returns when a ban made at now for duration ends: duration
@@ -191,22 +246,21 @@ func expiry(duration string, now time.Time) (time.Time, error) {
return now.Add(length), nil return now.Add(length), nil
} }
// handleLiftBans answers DELETE BansPath/<client>: it lifts every ban // liftBans answers DELETE BansPath/<client>: it lifts every ban active on
// active on a netblock the client's address is in, and answers with those // a netblock the client's address is in, and answers with those bans, or
// bans, or with 404 when none is active. // with 404 when none is active.
func (h *handler) handleLiftBans() http.HandlerFunc { func (rq *request) liftBans() {
return func(w http.ResponseWriter, r *http.Request) { client, err := pathAddress(rq.in.URL.Path, BansPath+"/")
client, err := pathAddress(r.URL.Path, BansPath+"/")
if err != nil { if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest) http.Error(rq.out, err.Error(), http.StatusBadRequest)
return return
} }
lifted := h.ledger.Lift(client, h.now()) lifted := rq.h.ledger.Lift(client, rq.h.now())
if len(lifted) == 0 { if len(lifted) == 0 {
http.Error( http.Error(
w, rq.out,
"no ban is active on "+client.String(), "no ban is active on "+client.String(),
http.StatusNotFound, http.StatusNotFound,
) )
@@ -214,8 +268,7 @@ func (h *handler) handleLiftBans() http.HandlerFunc {
return return
} }
h.answerBans(w, lifted) rq.answerBans(lifted)
}
} }
// clientAnswer is the answer to GET ClientsPath<ip>: the client the // clientAnswer is the answer to GET ClientsPath<ip>: the client the
@@ -227,31 +280,27 @@ type clientAnswer struct {
Bans []state.BanEntry `json:"bans"` Bans []state.BanEntry `json:"bans"`
} }
// handleShowClient answers GET ClientsPath<ip> with what waf knows of the // showClient answers GET ClientsPath<ip> with what waf knows of the client:
// client: its counters, its history, which holds its country as last looked // its counters, its history, which holds its country as last looked up and its
// up and its offences, and its bans with their notes. // offences, and its bans with their notes.
func (h *handler) handleShowClient() http.HandlerFunc { func (rq *request) showClient() {
return func(w http.ResponseWriter, r *http.Request) { addr, err := pathAddress(rq.in.URL.Path, ClientsPath)
addr, err := pathAddress(r.URL.Path, ClientsPath)
if err != nil { if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest) http.Error(rq.out, err.Error(), http.StatusBadRequest)
return return
} }
answer := clientAnswer{ answer := clientAnswer{
Bans: state.BanEntries(h.ledger.Covering(addr)), Bans: state.BanEntries(rq.h.ledger.Covering(addr)),
} }
client, seen := h.limiter.Client( client, seen := rq.h.limiter.Client(clientGroup(rq.h.config, addr))
clientGroup(h.config.IPv6GroupPrefix, addr),
)
if seen { if seen {
answer.Client = &client answer.Client = &client
} }
h.answerJSON(w, answer) rq.answerJSON(answer)
}
} }
// pathAddress reads the client's address that follows prefix in path. // pathAddress reads the client's address that follows prefix in path.
@@ -267,27 +316,27 @@ func pathAddress(path, prefix string) (netip.Addr, error) {
} }
// answerBans answers with held under bans, as bans.json lists them. // answerBans answers with held under bans, as bans.json lists them.
func (h *handler) answerBans(w http.ResponseWriter, held []bans.Ban) { func (rq *request) answerBans(held []bans.Ban) {
h.answerJSON(w, struct { rq.answerJSON(struct {
Bans []state.BanEntry `json:"bans"` Bans []state.BanEntry `json:"bans"`
}{state.BanEntries(held)}) }{state.BanEntries(held)})
} }
// answerJSON answers with value as indented JSON. // answerJSON answers with value as indented JSON.
func (h *handler) answerJSON(w http.ResponseWriter, value any) { func (rq *request) answerJSON(value any) {
body, err := json.MarshalIndent(value, "", " ") body, err := json.MarshalIndent(value, "", " ")
if err != nil { if err != nil {
h.processLog.Error( rq.h.processLog.Error(
"encoding an answer failed", "encoding an answer failed",
"error", "error",
err.Error(), err.Error(),
) )
http.Error(w, http.StatusText(http.StatusInternalServerError), http.Error(rq.out, http.StatusText(http.StatusInternalServerError),
http.StatusInternalServerError) http.StatusInternalServerError)
return return
} }
w.Header().Set("Content-Type", "application/json") rq.out.Header().Set("Content-Type", "application/json")
_, _ = w.Write(append(body, '\n')) _, _ = rq.out.Write(append(body, '\n'))
} }
+2 -36
View File
@@ -14,7 +14,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/state" "sneak.berlin/go/smallwebwaf/internal/state"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
const ( const (
@@ -38,10 +37,10 @@ func TestAdminEndpointsAreOffWhileTheTokenIsUnset(t *testing.T) {
"", "",
map[string]string{metricsToken: token}, map[string]string{metricsToken: token},
) )
server.Ledger.Ban( server.Ledger.BanForLimit(
netip.MustParsePrefix(otherClient+"/32"), netip.MustParsePrefix(otherClient+"/32"),
clk.Now(), clk.Now(),
bans.Grounds{Cause: bans.CauseLimit}, bans.Notes{},
) )
before := server.Ledger.Snapshot() before := server.Ledger.Snapshot()
@@ -334,39 +333,6 @@ func TestBanToAddThatCannotBeReadIsRefused(t *testing.T) {
} }
} }
func TestBanToAddIsLoggedWithTheBytesOfItsBody(t *testing.T) {
t.Parallel()
s, _, _ := startWithClock(
t,
"",
map[string]string{adminToken: adminSecret},
)
s.admin(http.MethodPost, waf.BansPath, banOtherClient, http.StatusOK)
line := s.out.endedLines(t, s.sent)[s.sent-1]
if line.RequestBytes != int64(len(banOtherClient)) {
t.Errorf(
"the ban to add was logged with request_bytes %d, want %d",
line.RequestBytes,
len(banOtherClient),
)
}
// The health endpoint reads none of a body sent to it.
s.ownEndpointRequest("", http.MethodGet, wafprotocol.HealthPath, "ok",
http.StatusOK)
line = s.out.endedLines(t, s.sent)[s.sent-1]
if line.RequestBytes != 0 {
t.Errorf(
"the health request was logged with request_bytes %d, want 0",
line.RequestBytes,
)
}
}
func TestClientEndpointShowsTheClientAndItsBans(t *testing.T) { func TestClientEndpointShowsTheClientAndItsBans(t *testing.T) {
t.Parallel() t.Parallel()
+5 -3
View File
@@ -130,9 +130,11 @@ func TestObserveModeRaisesTheBanAlertsItWouldHave(t *testing.T) {
// A ban for a clear sign of attack, which a request under it would make // A ban for a clear sign of attack, which a request under it would make
// permanent. // permanent.
group := netip.MustParsePrefix(ipv6Group) group := netip.MustParsePrefix(ipv6Group)
attackBan, _ := server.Ledger.Ban(group, start, bans.Grounds{ attackBan, _ := server.Ledger.BanForAttack(
Cause: bans.CauseAttack, Notes: bans.Notes{RuleID: "probe"}, group,
}) start,
bans.Notes{RuleID: "probe"},
)
// The third request breaks the limit, and so does the fourth, within the // The third request breaks the limit, and so does the fourth, within the
// cooldown, which raises nothing. The probe is a clear sign of attack. // cooldown, which raises nothing. The probe is a clear sign of attack.
+25 -63
View File
@@ -26,7 +26,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/reputation" "sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
const ( const (
@@ -85,7 +84,7 @@ func TestSixteenThousandCookiesAreRefusedBeforeTheCoreRuleSetReadsThem(
got := do(t, req) got := do(t, req)
wantDisposition(t, got, tooLarge, wafprotocol.DispositionDeny) wantDisposition(t, got, tooLarge, waf.DispositionDeny)
line := s.out.requestLine(t) line := s.out.requestLine(t)
wantLine(t, line, tooLarge, requestlog.ActionRequestHeaderTooLarge) wantLine(t, line, tooLarge, requestlog.ActionRequestHeaderTooLarge)
@@ -165,14 +164,9 @@ func TestARequestLineOverTheLimitIsRefusedWith414BeforeAnyOtherCheck(
line := s.out.requestLine(t) line := s.out.requestLine(t)
if tc.status == http.StatusOK { if tc.status == http.StatusOK {
wantDisposition( wantDisposition(t, got, tc.status, waf.DispositionAllow)
t,
got,
tc.status,
wafprotocol.DispositionAllow,
)
} else { } else {
wantDisposition(t, got, tc.status, wafprotocol.DispositionDeny) wantDisposition(t, got, tc.status, waf.DispositionDeny)
} }
wantLine(t, line, tc.status, tc.action) wantLine(t, line, tc.status, tc.action)
@@ -225,14 +219,9 @@ func TestAHeaderLineOverTheLimitIsRefusedWith431(t *testing.T) {
got := do(t, req) got := do(t, req)
if tc.refused { if tc.refused {
wantDisposition( wantDisposition(t, got, tooLarge, waf.DispositionDeny)
t,
got,
tooLarge,
wafprotocol.DispositionDeny,
)
} else { } else {
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow) wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
} }
}) })
} }
@@ -255,7 +244,10 @@ func TestRequestLineRefusalNamesItsHeuristicLengthAndLimit(t *testing.T) {
func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) { func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
t.Parallel() t.Parallel()
repeated := slices.Repeat([]string{"1"}, headerFieldLimit) repeated := make([]string, headerFieldLimit)
for i := range repeated {
repeated[i] = "1"
}
for _, tc := range []struct { for _, tc := range []struct {
name string name string
@@ -302,12 +294,7 @@ func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
line := s.out.requestLine(t) line := s.out.requestLine(t)
if tc.refused { if tc.refused {
wantDisposition( wantDisposition(t, got, tooLarge, waf.DispositionDeny)
t,
got,
tooLarge,
wafprotocol.DispositionDeny,
)
wantLine( wantLine(
t, t,
line, line,
@@ -318,12 +305,7 @@ func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
return return
} }
wantDisposition( wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
t,
got,
http.StatusOK,
wafprotocol.DispositionAllow,
)
wantLine(t, line, http.StatusOK, requestlog.ActionForward) wantLine(t, line, http.StatusOK, requestlog.ActionForward)
if line.DurationCoreRuleSetMS == nil { if line.DurationCoreRuleSetMS == nil {
@@ -350,12 +332,7 @@ func TestHeaderLimitsHoldWhateverTheCoreRuleSetSettingsSay(t *testing.T) {
req := s.question(siteA, client, "/") req := s.question(siteA, client, "/")
req.Header.Set(cookieHeader, cookies(cookieLimit+1)) req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition( wantDisposition(t, do(t, req), tooLarge, waf.DispositionDeny)
t,
do(t, req),
tooLarge,
wafprotocol.DispositionDeny,
)
}) })
} }
} }
@@ -371,12 +348,7 @@ func TestClientInTheAllowNetsSkipsTheHeaderLimits(t *testing.T) {
req := s.question(siteA, client, "/") req := s.question(siteA, client, "/")
req.Header.Set(cookieHeader, cookies(cookieLimit+1)) req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition( wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow)
t,
do(t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
)
} }
func TestObserveModeAllowsARequestOverTheHeaderLimitsUnread(t *testing.T) { func TestObserveModeAllowsARequestOverTheHeaderLimitsUnread(t *testing.T) {
@@ -392,7 +364,7 @@ func TestObserveModeAllowsARequestOverTheHeaderLimitsUnread(t *testing.T) {
got := do(t, req) got := do(t, req)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow) wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
line := s.out.requestLine(t) line := s.out.requestLine(t)
if line.WouldAction != requestlog.ActionRequestHeaderTooLarge || if line.WouldAction != requestlog.ActionRequestHeaderTooLarge ||
@@ -414,14 +386,14 @@ func TestTheRateLimitsDoNotCountRequestsOverTheHeaderLimits(t *testing.T) {
for range 2 { for range 2 {
req := s.question(siteA, client, "/") req := s.question(siteA, client, "/")
req.Header.Set(cookieHeader, cookies(cookieLimit+1)) req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(t, do(t, req), tooLarge, wafprotocol.DispositionDeny) wantDisposition(t, do(t, req), tooLarge, waf.DispositionDeny)
} }
wantDisposition( wantDisposition(
t, t,
s.ask(client, "/"), s.ask(client, "/"),
http.StatusOK, http.StatusOK,
wafprotocol.DispositionAllow, waf.DispositionAllow,
) )
} }
@@ -494,12 +466,7 @@ func TestTheHeaderLimitsComeBeforeEveryCheckThatRefuses(t *testing.T) {
req := s.question(siteA, client, tc.path) req := s.question(siteA, client, tc.path)
req.Header.Set(cookieHeader, cookies(cookieLimit+1)) req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition( wantDisposition(t, do(t, req), tooLarge, waf.DispositionDeny)
t,
do(t, req),
tooLarge,
wafprotocol.DispositionDeny,
)
wantLine( wantLine(
t, t,
s.out.requestLine(t), s.out.requestLine(t),
@@ -536,16 +503,11 @@ func TestARequestOverTheHeaderLimitsRunsNoOtherCheck(t *testing.T) {
over := s.question(siteA, client, "/") over := s.question(siteA, client, "/")
over.Header.Set(cookieHeader, cookies(cookieLimit+1)) over.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(t, do(t, over), tooLarge, wafprotocol.DispositionDeny) wantDisposition(t, do(t, over), tooLarge, waf.DispositionDeny)
within := s.question(siteA, client, "/") within := s.question(siteA, client, "/")
within.Header.Set(cookieHeader, cookies(cookieLimit)) within.Header.Set(cookieHeader, cookies(cookieLimit))
wantDisposition( wantDisposition(t, do(t, within), http.StatusOK, waf.DispositionAllow)
t,
do(t, within),
http.StatusOK,
wafprotocol.DispositionAllow,
)
lines := s.out.requestLines(t, 2) lines := s.out.requestLines(t, 2)
@@ -602,7 +564,7 @@ func TestNotWaitingForGeoJSNeverLowersAClientsLimit(t *testing.T) {
t, t,
s.ask(fromDE, "/"), s.ask(fromDE, "/"),
http.StatusForbidden, http.StatusForbidden,
wafprotocol.DispositionDeny, waf.DispositionDeny,
) )
line := s.out.requestLine(t) line := s.out.requestLine(t)
@@ -639,7 +601,7 @@ func TestAClientGeoJSPlacedInNoCountryCountsAsUnknownWithoutWaiting(
t, t,
s.ask(client, "/"), s.ask(client, "/"),
http.StatusForbidden, http.StatusForbidden,
wafprotocol.DispositionDeny, waf.DispositionDeny,
) )
line := s.out.requestLine(t) line := s.out.requestLine(t)
@@ -668,7 +630,7 @@ func TestObserveModeAsksGeoJSNothingForARequestOverTheHeaderLimits(
req.Header.Set(cookieHeader, cookies(cookieLimit+1)) req.Header.Set(cookieHeader, cookies(cookieLimit+1))
got := do(t, req) got := do(t, req)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow) wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
if len(asked()) != 0 || got.header.Get("X-Client-Asn") != "" { if len(asked()) != 0 || got.header.Get("X-Client-Asn") != "" {
t.Errorf( t.Errorf(
@@ -710,7 +672,7 @@ func askDirectly(
http.NoBody, http.NoBody,
) )
req.RemoteAddr = net.JoinHostPort(rproxyPeer, "1234") req.RemoteAddr = net.JoinHostPort(rproxyPeer, "1234")
req.Header.Set(wafprotocol.ClientHeader, client) req.Header.Set(waf.ClientHeader, client)
req.Header.Set(requestIDHeader, "the-id") req.Header.Set(requestIDHeader, "the-id")
change(req) change(req)
@@ -724,10 +686,10 @@ func askDirectly(
// banClient bans client at now. // banClient bans client at now.
func banClient(_ *testing.T, server *waf.Server, now time.Time) { func banClient(_ *testing.T, server *waf.Server, now time.Time) {
server.Ledger.Ban( server.Ledger.BanForLimit(
netip.MustParsePrefix(client+"/32"), netip.MustParsePrefix(client+"/32"),
now, now,
bans.Grounds{Cause: bans.CauseLimit}, bans.Notes{},
) )
} }
+98 -165
View File
@@ -1,7 +1,7 @@
package waf package waf
// The heuristic active_ban, and banMaker, which makes the bans of several // The heuristic active_ban, and the code that makes bans, which several
// heuristics. active_ban reads SWW_WAF_MODE; the bans are the ban ledger's. // heuristics share. active_ban reads SWW_WAF_MODE; the bans are the ban ledger's.
import ( import (
"context" "context"
@@ -11,7 +11,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
@@ -26,35 +25,28 @@ func (rq *request) banResponse(action string) *refusal {
// in, in force at now. It notes for the log line when that ban ends. A request // in, in force at now. It notes for the log line when that ban ends. A request
// that makes the ban permanent, or in observe mode would have, raises the alert // that makes the ban permanent, or in observe mode would have, raises the alert
// for it. Its figures are the ban's netblock and when it ends. // for it. Its figures are the ban's netblock and when it ends.
type activeBan struct { type activeBan struct{}
banMaker banMaker
}
func (activeBan) name() string { return "active_ban" } func (activeBan) name() string { return "active_ban" }
func (a activeBan) check( func (activeBan) check(
_ context.Context, _ context.Context,
rq *request, rq *request,
now time.Time, now time.Time,
) *verdict { ) *verdict {
ledger := a.banMaker.ledger check := rq.h.ledger.Check
if rq.h.config.Observe {
check := ledger.Check check = rq.h.ledger.Find // the ban refuses nothing, and stays as it is
if a.banMaker.observe {
check = ledger.Find // the ban refuses nothing, and stays as it is
} }
ban, banned, madePermanent := check(rq.client, now) ban, banned, madePermanent := check(rq.client, now)
var expires string
if banned { if banned {
expires = banExpires(ban) rq.line.BanExpires = banExpires(ban)
rq.line.BanExpires = expires
} }
if madePermanent { if madePermanent {
ban.Expires = time.Time{} // the ban made permanent, which Find leaves as it is ban.Expires = time.Time{} // the ban made permanent, which Find leaves as it is
a.banMaker.alertBan(rq, ban) rq.alertBan(ban)
} }
if !banned { if !banned {
@@ -64,7 +56,7 @@ func (a activeBan) check(
return &verdict{ return &verdict{
action: banAction(ban.Permanent()), action: banAction(ban.Permanent()),
figures: map[string]any{ figures: map[string]any{
figureNetblock: ban.Netblock.String(), "ban_expires": expires, figureNetblock: ban.Netblock.String(), "ban_expires": rq.line.BanExpires,
}, },
} }
} }
@@ -89,12 +81,11 @@ func limitFigures(hit ratelimit.Hit) map[string]any {
// countedBytes returns the request's bytes, once it has ended, as the // countedBytes returns the request's bytes, once it has ended, as the
// byte limits and the anomaly thresholds count them: the response's body // byte limits and the anomaly thresholds count them: the response's body
// bytes, the request's, or both, as bytesCounted, SWW_WAF_BYTES_COUNTED, // bytes, the request's, or both, as SWW_WAF_BYTES_COUNTED says.
// says. func (rq *request) countedBytes() int64 {
func (rq *request) countedBytes(bytesCounted config.BytesCounted) int64 {
response, request := rq.out.bytes, rq.requestBytes response, request := rq.out.bytes, rq.requestBytes
switch bytesCounted { switch rq.h.config.BytesCounted {
case config.BytesCountedResponse: case config.BytesCountedResponse:
return response return response
case config.BytesCountedRequest: case config.BytesCountedRequest:
@@ -105,39 +96,17 @@ func (rq *request) countedBytes(bytesCounted config.BytesCounted) int64 {
return response + request return response + request
} }
// banMaker makes the bans of the heuristics that ban a client, in the ban // banForLimit bans the client's netblock at now for a broken limit, the
// ledger, and raises the alert for each. In observe mode it makes no ban, and
// raises the alert for the ban it would have made.
type banMaker struct {
ledger *bans.Ledger
// limiter is the table of clients the rate limits count. A ban for a limit
// sets the client's counters back to zero, and a ban's notes give how many
// requests its netblock has sent.
limiter *ratelimit.Limiter
metrics *metrics.Metrics
alerts *alerts.Queue
// observe is true in observe mode (SWW_WAF_MODE).
observe bool
// banResponse is SWW_WAF_BAN_RESPONSE, the status the notes of a ban for a
// clear sign of attack give.
banResponse int
// banScopeV4Prefix is SWW_WAF_BAN_SCOPE_V4_PREFIX and ipv6GroupPrefix
// SWW_WAF_IPV6_GROUP_PREFIX, which set the netblock a ban covers.
banScopeV4Prefix int
ipv6GroupPrefix int
}
// banForLimit bans the netblock of rq's client at now for a broken limit, the
// one hit names, notes the offence for the log line and counts the hit in // one hit names, notes the offence for the log line and counts the hit in
// the metrics. status is what the client was sent, or is sent: // the metrics. status is what the client was sent, or is sent:
// SWW_WAF_BAN_RESPONSE for a request over a rate limit, what the report says for // SWW_WAF_BAN_RESPONSE for a request over a rate limit, what the report says for
// one whose bytes broke a byte limit, the refusal for one that broke the // one whose bytes broke a byte limit, the refusal for one that broke the
// refusal limit. The ban's notes give the client's limit percentage for a // refusal limit. The ban's notes give the client's limit percentage for a
// rate limit or a byte limit; the refusal limit is not lowered. The ban sets // rate limit or a byte limit; the refusal limit is not lowered. The ban sets
// the client's counters back to zero. In observe mode, banClient says what // the client's counters back to zero. In observe mode it makes no ban and
// happens instead. // sets nothing back, and raises the alert for the ban it would have made,
func (b banMaker) banForLimit( // if that alert would be sent.
rq *request, func (rq *request) banForLimit(
now time.Time, now time.Time,
hit ratelimit.Hit, hit ratelimit.Hit,
status int, status int,
@@ -151,16 +120,26 @@ func (b banMaker) banForLimit(
rq.line.LimitHit = hit.Window rq.line.LimitHit = hit.Window
} }
rq.brokeLimit = true
rq.line.Offence = requestlog.OffenceLimit rq.line.Offence = requestlog.OffenceLimit
rq.h.metrics.LimitHit(hit)
b.metrics.LimitHit(hit) netblock := banNetblock(rq.h.config, rq.client)
if rq.h.config.Observe &&
!rq.wouldAlertBan(netblock, now, bans.CauseLimit) {
return
}
notes := bans.Notes{ notes := bans.Notes{
ASN: rq.line.ASN,
ASName: rq.line.ASName,
Country: rq.line.Country,
Kind: hit.Kind, Kind: hit.Kind,
Limit: hit.Limit, Limit: hit.Limit,
Window: hit.Window, Window: hit.Window,
Count: hit.Count, Count: hit.Count,
Reputation: rq.reputation,
Request: rq.noted(now, status),
Requests: rq.netblockRequests(netblock),
} }
switch hit.Kind { switch hit.Kind {
@@ -170,107 +149,62 @@ func (b banMaker) banForLimit(
notes.LimitPercent, notes.LimitPercentSetting = rq.bytesLimitPercent.logged() notes.LimitPercent, notes.LimitPercentSetting = rq.bytesLimitPercent.logged()
} }
b.banClient( if rq.h.config.Observe {
rq, ban, wouldBan := rq.h.ledger.WouldBanForLimit(netblock, now, notes)
now,
status,
bans.Grounds{Cause: bans.CauseLimit, Notes: notes},
)
}
// banForAttack bans the netblock of rq's client at now for a clear sign of
// attack, which notes name: the ban rule that matched, or the trap path
// asked for. It returns the action for that ban. In observe mode, banClient
// says what happens instead.
func (b banMaker) banForAttack(
rq *request, now time.Time, notes bans.Notes,
) string {
return b.banClient(
rq,
now,
b.banResponse,
bans.Grounds{Cause: bans.CauseAttack, Notes: notes},
)
}
// banClient bans the netblock of rq's client at now on grounds, and returns
// the action for that ban. It adds to the notes what the request tells of the
// client, with status, what the client was sent. A ban for a broken limit
// sets the client's counters back to zero.
//
// In observe mode it makes no ban and sets nothing back. It raises the
// alert for the ban it would have made, if that alert would be sent.
func (b banMaker) banClient(
rq *request,
now time.Time,
status int,
grounds bans.Grounds,
) string {
netblock := banNetblock(b.banScopeV4Prefix, b.ipv6GroupPrefix, rq.client)
if b.observe {
if !b.wouldAlertBan(netblock, now, grounds.Cause) {
return banAction(
b.ledger.WouldBePermanent(netblock, now, grounds.Cause),
)
}
grounds.Notes = b.withClientNotes(
rq,
grounds.Notes,
now,
status,
netblock,
)
ban, wouldBan := b.ledger.WouldBan(netblock, now, grounds)
if wouldBan { if wouldBan {
b.alertBan(rq, ban) rq.alertBan(ban)
} }
return banAction(ban.Permanent()) return
}
grounds.Notes = b.withClientNotes(
rq,
grounds.Notes,
now,
status,
netblock,
)
ban, made := b.ledger.Ban(netblock, now, grounds)
if grounds.Cause == bans.CauseLimit {
b.limiter.Reset(clientGroup(b.ipv6GroupPrefix, rq.client))
} }
ban, made := rq.h.ledger.BanForLimit(netblock, now, notes)
rq.h.limiter.Reset(clientGroup(rq.h.config, rq.client))
rq.line.BanExpires = banExpires(ban) rq.line.BanExpires = banExpires(ban)
if made { if made {
b.alertBan(rq, ban) rq.alertBan(ban)
}
}
// banForAttack bans the client's netblock at now for a clear sign of
// attack, which notes name: the ban rule that matched, or the trap path
// asked for. It fills in the rest of the notes. In observe mode it makes
// no ban, and raises the alert for the ban it would have made, if that
// alert would be sent. It returns the action for that ban.
func (rq *request) banForAttack(now time.Time, notes bans.Notes) string {
netblock := banNetblock(rq.h.config, rq.client)
if rq.h.config.Observe &&
!rq.wouldAlertBan(netblock, now, bans.CauseAttack) {
return banAction(
rq.h.ledger.WouldBePermanent(netblock, now, bans.CauseAttack),
)
}
notes.ASN = rq.line.ASN
notes.ASName = rq.line.ASName
notes.Country = rq.line.Country
notes.Reputation = rq.reputation
notes.Request = rq.noted(now, rq.h.config.BanResponse)
notes.Requests = rq.netblockRequests(netblock)
if rq.h.config.Observe {
ban, wouldBan := rq.h.ledger.WouldBanForAttack(netblock, now, notes)
if wouldBan {
rq.alertBan(ban)
} }
return banAction(ban.Permanent()) return banAction(ban.Permanent())
} }
// withClientNotes is notes with what rq tells of its client added: its AS, ban, made := rq.h.ledger.BanForAttack(netblock, now, notes)
// country and reputation, the request at now with status, and how many rq.line.BanExpires = banExpires(ban)
// requests netblock, the client's, has sent.
func (b banMaker) withClientNotes(
rq *request,
notes bans.Notes,
now time.Time,
status int,
netblock netip.Prefix,
) bans.Notes {
notes.ASN = rq.lookupAnswer.ASN
notes.ASName = rq.lookupAnswer.ASName
notes.Country = rq.lookupAnswer.Country
notes.Reputation = rq.reputation
notes.Request = rq.noted(now, status)
notes.Requests = b.netblockRequests(netblock)
return notes if made {
rq.alertBan(ban)
}
return banAction(ban.Permanent())
} }
// wouldAlertBan reports whether the alert for a ban on netblock for cause // wouldAlertBan reports whether the alert for a ban on netblock for cause
@@ -278,22 +212,23 @@ func (b banMaker) withClientNotes(
// have made is worked out only then, at most once per // have made is worked out only then, at most once per
// SWW_WAF_ALERT_COOLDOWN and never with no webhook set: its notes count the // SWW_WAF_ALERT_COOLDOWN and never with no webhook set: its notes count the
// netblock's requests, which can mean going through every client. // netblock's requests, which can mean going through every client.
func (b banMaker) wouldAlertBan( func (rq *request) wouldAlertBan(
netblock netip.Prefix, now time.Time, cause string, netblock netip.Prefix, now time.Time, cause string,
) bool { ) bool {
event := alerts.EventBan event := alerts.EventBan
if b.ledger.WouldBePermanent(netblock, now, cause) { if rq.h.ledger.WouldBePermanent(netblock, now, cause) {
event = alerts.EventPermanentBan event = alerts.EventPermanentBan
} }
return b.alerts.WouldSend(event, netblock) return rq.h.alerts.WouldSend(event, netblock)
} }
// alertBan raises the alert for ban, which rq made, or made permanent: // alertBan raises the alert for ban, which the request made, or made
// permanent_ban for a permanent ban, ban for another. Its detail gives the // permanent: permanent_ban for a permanent ban, ban for another. Its
// ban's cause, when it ends, and its notes, and in observe mode, where ban is // detail gives the ban's cause, when it ends, and its notes, and in
// the ban that would have been made, or made permanent, mode, observe. // observe mode, where ban is the ban that would have been made, or made
func (b banMaker) alertBan(rq *request, ban bans.Ban) { // permanent, mode, observe.
func (rq *request) alertBan(ban bans.Ban) {
event := alerts.EventBan event := alerts.EventBan
if ban.Permanent() { if ban.Permanent() {
event = alerts.EventPermanentBan event = alerts.EventPermanentBan
@@ -302,11 +237,11 @@ func (b banMaker) alertBan(rq *request, ban bans.Ban) {
detail := map[string]any{ detail := map[string]any{
"cause": ban.Cause, "ban_expires": banExpires(ban), "notes": ban.Notes, "cause": ban.Cause, "ban_expires": banExpires(ban), "notes": ban.Notes,
} }
if b.observe { if rq.h.config.Observe {
detail["mode"] = "observe" detail["mode"] = "observe"
} }
b.alerts.Raise(alerts.Alert{ rq.h.alerts.Raise(alerts.Alert{
Event: event, Event: event,
Host: rq.site, Host: rq.site,
Client: rq.client, Client: rq.client,
@@ -319,39 +254,37 @@ func (b banMaker) alertBan(rq *request, ban bans.Ban) {
}) })
} }
// netblockRequests is how many requests netblock has sent since it was
// first seen, the request being banned for included: the histories count it
// only once it has ended.
func (b banMaker) netblockRequests(netblock netip.Prefix) int64 {
return b.limiter.Requests(netblock) + 1
}
// noted is the request, at now, with status, what the client was sent, or // noted is the request, at now, with status, what the client was sent, or
// in observe mode would have been, as the notes of the ban it makes keep // in observe mode would have been, as the notes of the ban it makes keep
// it. // it.
func (rq *request) noted(now time.Time, status int) bans.Request { func (rq *request) noted(now time.Time, status int) bans.Request {
return bans.Request{ return bans.Request{
Time: now, Time: now,
Method: rq.method, Method: rq.line.Method,
Host: rq.site, Host: rq.line.Host,
Path: rq.requestURI, Path: rq.requestURI,
Status: status, Status: status,
UserAgent: rq.userAgent, UserAgent: rq.line.UserAgent,
} }
} }
// netblockRequests is how many requests netblock has sent since it was
// first seen, this one included: the histories count it only once it has
// ended.
func (rq *request) netblockRequests(netblock netip.Prefix) int64 {
return rq.h.limiter.Requests(netblock) + 1
}
// banNetblock is the netblock a ban on client covers: its IPv4 address, // banNetblock is the netblock a ban on client covers: its IPv4 address,
// widened to banScopeV4Prefix, SWW_WAF_BAN_SCOPE_V4_PREFIX, or the IPv6 group // widened to SWW_WAF_BAN_SCOPE_V4_PREFIX, or the IPv6 group clientGroup
// clientGroup counts it in, of length ipv6GroupPrefix. // counts it in.
func banNetblock( func banNetblock(cfg *config.WAF, client netip.Addr) netip.Prefix {
banScopeV4Prefix, ipv6GroupPrefix int, client netip.Addr,
) netip.Prefix {
addr := client.Unmap() addr := client.Unmap()
if addr.Is4() { if addr.Is4() {
return netip.PrefixFrom(addr, banScopeV4Prefix).Masked() return netip.PrefixFrom(addr, cfg.BanScopeV4Prefix).Masked()
} }
return clientGroup(ipv6GroupPrefix, addr) return clientGroup(cfg, addr)
} }
// banExpires is when ban ends, as the log line gives it: a time, or // banExpires is when ban ends, as the log line gives it: a time, or
+5 -4
View File
@@ -548,10 +548,6 @@ func (s *sender) requestWithHeader(
// body is sent as it is after the headers, header holding its // body is sent as it is after the headers, header holding its
// Content-Length or Transfer-Encoding. header may hold several lines, // Content-Length or Transfer-Encoding. header may hold several lines,
// separated by "\r\n". It returns the whole answer. // separated by "\r\n". It returns the whole answer.
//
// It waits for the answer as long as it takes, since a busy host can slow
// the Core Rule Set past any shorter wait. go test's -timeout ends a test
// whose answer never comes.
func (s *sender) requestWithBody( func (s *sender) requestWithBody(
method, from, path, header, body string, status int, action string, method, from, path, header, body string, status int, action string,
) (logLine, answer) { ) (logLine, answer) {
@@ -570,6 +566,11 @@ func (s *sender) requestWithBody(
send(s.t, conn, method+" "+path+" HTTP/1.1\r\nHost: "+appHost+"\r\n"+ send(s.t, conn, method+" "+path+" HTTP/1.1\r\nHost: "+appHost+"\r\n"+
forwardedFor+": "+from+"\r\n"+header+"\r\n"+body) forwardedFor+": "+from+"\r\n"+header+"\r\n"+body)
err := conn.SetReadDeadline(time.Now().Add(waitLimit))
if err != nil {
s.t.Fatalf("set read deadline: %v", err)
}
var got answer var got answer
res, err := http.ReadResponse(bufio.NewReader(conn), nil) res, err := http.ReadResponse(bufio.NewReader(conn), nil)
+12 -12
View File
@@ -54,7 +54,7 @@ func (rq *request) runBesideLookup(
) *verdict { ) *verdict {
var running atomic.Pointer[string] var running atomic.Pointer[string]
running.Store(new(rq.h.trapPaths.name())) running.Store(new(trapPaths{}.name()))
found := make(chan *besideLookupFindings, 1) found := make(chan *besideLookupFindings, 1)
@@ -92,7 +92,7 @@ func (rq *request) runBesideLookup(
return v return v
} }
return rq.runHeuristics(ctx, rq.h.afterBesideLookup, rq.h.now()) return rq.runHeuristics(ctx, afterBesideLookup(), rq.h.now())
} }
// findBesideLookup does the work of the heuristics of besideLookup on the // findBesideLookup does the work of the heuristics of besideLookup on the
@@ -105,39 +105,39 @@ func (rq *request) findBesideLookup(
) *besideLookupFindings { ) *besideLookupFindings {
findings := &besideLookupFindings{} findings := &besideLookupFindings{}
findings.trapPath = rq.h.trapPaths.trapPath(rq) findings.trapPath = rq.trapPath()
if findings.trapPath != "" { if findings.trapPath != "" {
findings.refuses = true findings.refuses = true
return findings return findings
} }
running.Store(new(rq.h.ruleFiles.name())) running.Store(new(ruleFiles{}.name()))
findings.rules = rq.h.ruleFiles.rules.Match(rq.in) findings.rules = rq.h.rules.Match(rq.in)
if refusedByRule(findings.rules) { if refusedByRule(findings.rules) {
findings.refuses = true findings.refuses = true
return findings return findings
} }
coreRuleSet := rq.h.coreRuleSet running.Store(new(coreRuleSet{}.name()))
running.Store(new(coreRuleSet.name()))
if coreRuleSet.mode == config.CoreRuleSetModeOff || cfg := rq.h.config
pathExempt(rq.in, coreRuleSet.exemptPaths) { if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff ||
pathExempt(rq.in, cfg.CoreRuleSetExemptPaths) {
return findings return findings
} }
start := time.Now() start := time.Now()
// A question's body has been read into memory, so that reading it here // A question's body has been read into memory, so that reading it here
// cannot fail. // cannot fail.
findings.inspection, findings.read, _ = coreRuleSet.inspector.Inspect( findings.inspection, findings.read, _ = rq.h.coreRuleSet.Inspect(
rq.in, rq.client, rq.in.Body, rq.in, rq.client, rq.in.Body,
) )
findings.inspectionTime = time.Since(start) findings.inspectionTime = time.Since(start)
findings.inspected = true findings.inspected = true
findings.refuses = coreRuleSet.refusedByCoreRuleSet(findings.inspection) findings.refuses = rq.refusedByCoreRuleSet(findings.inspection)
return findings return findings
} }
@@ -151,7 +151,7 @@ func (rq *request) applyFindings(
) *verdict { ) *verdict {
rq.findings = findings rq.findings = findings
return rq.runHeuristics(ctx, rq.h.besideLookup, rq.h.now()) return rq.runHeuristics(ctx, besideLookup(), rq.h.now())
} }
// timedOut refuses the request for the heuristics of besideLookup not having // timedOut refuses the request for the heuristics of besideLookup not having
+17 -38
View File
@@ -2,7 +2,6 @@ package waf
import ( import (
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/reputation"
) )
// whole is the percentage of each limit a client gets when no biased // whole is the percentage of each limit a client gets when no biased
@@ -30,25 +29,6 @@ func biasedThresholdsSet(cfg *config.WAF) bool {
cfg.ASNLimitPercentURL != "" cfg.ASNLimitPercentURL != ""
} }
// biasedThresholds are the settings and the lists by which limitPercentages
// lowers a client's limits.
type biasedThresholds struct {
// lists hold the file SWW_WAF_ASN_LIMIT_PERCENT_URL names.
lists *reputation.Lists
tagLimitPercent map[string]int64 // SWW_WAF_TAG_LIMIT_PERCENT
asnBytesLimitPercent map[string]int64 // SWW_WAF_ASN_BYTES_LIMIT_PERCENT
countryBytesLimitPercent map[string]int64 // SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT
unknownCountryLimitPercent int64 // SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT
// blocklistAction and blocklistLimitPercent are SWW_WAF_BLOCKLIST_ACTION
// and its percentage.
blocklistAction config.ReputationAction
blocklistLimitPercent int64
// dnsblAbuseIPDBAction and dnsblAbuseIPDBLimitPercent are
// SWW_WAF_DNSBL_ABUSEIPDB_ACTION and its percentage.
dnsblAbuseIPDBAction config.ReputationAction
dnsblAbuseIPDBLimitPercent int64
}
// limitPercentages returns the client's limit percentages, for the rate // limitPercentages returns the client's limit percentages, for the rate
// limits and for the byte limits, by its tags, its AS number and country // limits and for the byte limits, by its tags, its AS number and country
// as looked up, each "" when unknown, and the blocklists, DNSBL zones and // as looked up, each "" when unknown, and the blocklists, DNSBL zones and
@@ -65,38 +45,37 @@ type biasedThresholds struct {
// the byte limits, SWW_WAF_ASN_BYTES_LIMIT_PERCENT and // the byte limits, SWW_WAF_ASN_BYTES_LIMIT_PERCENT and
// SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT take the place of the first three for // SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT take the place of the first three for
// an AS number or a country they list. // an AS number or a country they list.
func (b biasedThresholds) limitPercentages( func (rq *request) limitPercentages() (percentage, percentage) {
rq *request, cfg := rq.h.config
) (percentage, percentage) { asn, country := rq.line.ASN, rq.line.Country
asn, country := rq.lookupAnswer.ASN, rq.lookupAnswer.Country
unknown := percentage{percent: whole} unknown := percentage{percent: whole}
if country == "" && !rq.skippedLookupWait { if country == "" && !rq.skippedLookupWait {
unknown = percentage{ unknown = percentage{
b.unknownCountryLimitPercent, cfg.UnknownCountryLimitPercent,
config.UnknownCountryLimitPercentSetting, config.UnknownCountryLimitPercentSetting,
} }
} }
fetched := percentage{percent: whole} fetched := percentage{percent: whole}
if percent, listed := b.lists.ASNLimitPercent(asn); listed { if percent, listed := rq.h.lists.ASNLimitPercent(asn); listed {
fetched = percentage{percent, config.ASNLimitPercentURLSetting} fetched = percentage{percent, config.ASNLimitPercentURLSetting}
} }
blocklisted := percentage{percent: whole} blocklisted := percentage{percent: whole}
if rq.blocklisted && if rq.blocklisted &&
b.blocklistAction == config.ReputationActionLimit { cfg.BlocklistAction == config.ReputationActionLimit {
blocklisted = percentage{ blocklisted = percentage{
b.blocklistLimitPercent, cfg.BlocklistLimitPercent,
config.BlocklistActionSetting, config.BlocklistActionSetting,
} }
} }
dnsblAbuseIPDBListed := percentage{percent: whole} dnsblAbuseIPDBListed := percentage{percent: whole}
if (rq.dnsblListed || rq.abuseIPDBHit) && if (rq.dnsblListed || rq.abuseIPDBHit) &&
b.dnsblAbuseIPDBAction == config.ReputationActionLimit { cfg.DNSBLAbuseIPDBAction == config.ReputationActionLimit {
dnsblAbuseIPDBListed = percentage{ dnsblAbuseIPDBListed = percentage{
b.dnsblAbuseIPDBLimitPercent, cfg.DNSBLAbuseIPDBLimitPercent,
config.DNSBLAbuseIPDBActionSetting, config.DNSBLAbuseIPDBActionSetting,
} }
} }
@@ -104,38 +83,38 @@ func (b biasedThresholds) limitPercentages(
const tagSetting = config.TagLimitPercentSetting const tagSetting = config.TagLimitPercentSetting
asnRequests := lowest( asnRequests := lowest(
given(b.tagLimitPercent, asnTag(asn), tagSetting), given(cfg.TagLimitPercent, asnTag(asn), tagSetting),
fetched, fetched,
) )
countryRequests := given( countryRequests := given(
b.tagLimitPercent, cfg.TagLimitPercent,
countryTag(country), countryTag(country),
tagSetting, tagSetting,
) )
otherTags := percentage{percent: whole} otherTags := percentage{percent: whole}
for _, tag := range rq.tags { for _, tag := range rq.line.Tags {
if tag != asnTag(asn) && tag != countryTag(country) { if tag != asnTag(asn) && tag != countryTag(country) {
otherTags = lowest( otherTags = lowest(
otherTags, otherTags,
given(b.tagLimitPercent, tag, tagSetting), given(cfg.TagLimitPercent, tag, tagSetting),
) )
} }
} }
asnBytes, countryBytes := asnRequests, countryRequests asnBytes, countryBytes := asnRequests, countryRequests
if _, listed := b.asnBytesLimitPercent[asn]; listed { if _, listed := cfg.ASNBytesLimitPercent[asn]; listed {
asnBytes = given( asnBytes = given(
b.asnBytesLimitPercent, cfg.ASNBytesLimitPercent,
asn, asn,
config.ASNBytesLimitPercentSetting, config.ASNBytesLimitPercentSetting,
) )
} }
if _, listed := b.countryBytesLimitPercent[country]; listed { if _, listed := cfg.CountryBytesLimitPercent[country]; listed {
countryBytes = given( countryBytes = given(
b.countryBytesLimitPercent, cfg.CountryBytesLimitPercent,
country, country,
config.CountryBytesLimitPercentSetting, config.CountryBytesLimitPercentSetting,
) )
+1 -2
View File
@@ -19,7 +19,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest" "sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// The biased thresholds. // The biased thresholds.
@@ -590,7 +589,7 @@ func serveFromDE(
t.Helper() t.Helper()
req := httptest.NewRequestWithContext(t.Context(), method, "/", body) req := httptest.NewRequestWithContext(t.Context(), method, "/", body)
req.Header.Set(wafprotocol.ClientHeader, fromDE) req.Header.Set(waf.ClientHeader, fromDE)
req.Header.Set(requestIDHeader, "the-id") req.Header.Set(requestIDHeader, "the-id")
server.Handler.ServeHTTP(httptest.NewRecorder(), req) server.Handler.ServeHTTP(httptest.NewRecorder(), req)
+8 -21
View File
@@ -8,7 +8,6 @@ import (
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
@@ -16,24 +15,20 @@ import (
// and refuses the request when SWW_WAF_BLOCKLIST_ACTION is deny. Being limit, it // and refuses the request when SWW_WAF_BLOCKLIST_ACTION is deny. Being limit, it
// lowers the client's limits instead (see limitPercentages), and being log, it // lowers the client's limits instead (see limitPercentages), and being log, it
// does nothing more. Its figure is the blocklists that list the client. // does nothing more. Its figure is the blocklists that list the client.
type blocklists struct { type blocklists struct{}
lists *reputation.Lists
action config.ReputationAction // SWW_WAF_BLOCKLIST_ACTION
hitNoter reputationHitNoter
}
func (blocklists) name() string { return "blocklists" } func (blocklists) name() string { return "blocklists" }
func (b blocklists) check( func (blocklists) check(
_ context.Context, _ context.Context,
rq *request, rq *request,
_ time.Time, _ time.Time,
) *verdict { ) *verdict {
listedBy := b.lists.ListedBy(rq.client) listedBy := rq.h.lists.ListedBy(rq.client)
rq.blocklisted = len(listedBy) > 0 rq.blocklisted = len(listedBy) > 0
b.hitNoter.noteListed(rq, listedBy, "listed by a blocklist") rq.noteListed(listedBy, "listed by a blocklist")
if !b.refusedByBlocklists(listedBy) { if !rq.refusedByBlocklists(listedBy) {
return nil return nil
} }
@@ -43,18 +38,10 @@ func (b blocklists) check(
} }
} }
// refusesWhateverTheLookupAnswers reports whether blocklists refuses rq, as
// refusedByBlocklists says. The lookup's answer plays no part in it.
func (b blocklists) refusesWhateverTheLookupAnswers(
rq *request,
_ time.Time,
) bool {
return b.refusedByBlocklists(b.lists.ListedBy(rq.client))
}
// refusedByBlocklists reports whether the heuristic blocklists refuses a // refusedByBlocklists reports whether the heuristic blocklists refuses a
// client, given listedBy, the blocklists that list it: it does when one lists // client, given listedBy, the blocklists that list it: it does when one lists
// it and SWW_WAF_BLOCKLIST_ACTION is deny. // it and SWW_WAF_BLOCKLIST_ACTION is deny.
func (b blocklists) refusedByBlocklists(listedBy []string) bool { func (rq *request) refusedByBlocklists(listedBy []string) bool {
return len(listedBy) > 0 && b.action == config.ReputationActionDeny return len(listedBy) > 0 && rq.h.config.BlocklistAction ==
config.ReputationActionDeny
} }
+7 -15
View File
@@ -6,9 +6,6 @@ package waf
import ( import (
"context" "context"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
) )
// byteLimits counts the request's bytes, as countedBytes gives them, for the // byteLimits counts the request's bytes, as countedBytes gives them, for the
@@ -19,28 +16,23 @@ import (
// the client over a byte limit, as its limit percentage for the byte limits // the client over a byte limit, as its limit percentage for the byte limits
// lowers it, break it, which bans the client; the response was passed on whole. // lowers it, break it, which bans the client; the response was passed on whole.
// Its figures are the window, the count and the limit. // Its figures are the window, the count and the limit.
type byteLimits struct { type byteLimits struct{}
limiter *ratelimit.Limiter
bytesCounted config.BytesCounted // SWW_WAF_BYTES_COUNTED
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
banMaker banMaker
}
func (byteLimits) name() string { return "byte_limits" } func (byteLimits) name() string { return "byte_limits" }
func (b byteLimits) check( func (byteLimits) check(
_ context.Context, _ context.Context,
rq *request, rq *request,
now time.Time, now time.Time,
) *verdict { ) *verdict {
if !rq.counted || rq.wouldAction != "" { if !rq.counted || rq.line.WouldAction != "" {
return nil return nil
} }
counts, hit, over := b.limiter.CountBytes( counts, hit, over := rq.h.limiter.CountBytes(
clientGroup(b.ipv6GroupPrefix, rq.client), clientGroup(rq.h.config, rq.client),
now, now,
rq.countedBytes(b.bytesCounted), rq.countedBytes(),
rq.bytesLimitPercent.percent, rq.bytesLimitPercent.percent,
) )
rq.line.Counts.MinuteBytes = counts.MinuteBytes rq.line.Counts.MinuteBytes = counts.MinuteBytes
@@ -51,7 +43,7 @@ func (b byteLimits) check(
return nil return nil
} }
b.banMaker.banForLimit(rq, now, hit, rq.out.status) rq.banForLimit(now, hit, rq.out.status)
return &verdict{figures: limitFigures(hit)} return &verdict{figures: limitFigures(hit)}
} }
+9 -7
View File
@@ -6,7 +6,7 @@ import (
"net/netip" "net/netip"
"slices" "slices"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/config"
) )
// peerAddress is the address of the request's TCP peer. // peerAddress is the address of the request's TCP peer.
@@ -19,11 +19,14 @@ func peerAddress(r *http.Request) netip.Addr {
return addrPort.Addr().Unmap() return addrPort.Addr().Unmap()
} }
// requestIDHeader carries the request's id, from rproxy.
const requestIDHeader = "X-Request-ID"
// requestID is the request's id: the one a question gives, or for any other // requestID is the request's id: the one a question gives, or for any other
// request a new random one. // request a new random one.
func requestID(r *http.Request, isQuestion bool) string { func requestID(r *http.Request, isQuestion bool) string {
if isQuestion { if isQuestion {
return r.Header.Get(wafprotocol.RequestIDHeader) return r.Header.Get(requestIDHeader)
} }
return rand.Text() return rand.Text()
@@ -42,13 +45,12 @@ func scheme(r *http.Request, isQuestion bool) string {
// clientGroup is the client a request is counted toward: its IPv4 // clientGroup is the client a request is counted toward: its IPv4
// address, or its IPv6 group, the netblock its IPv6 address is in of the // address, or its IPv6 group, the netblock its IPv6 address is in of the
// length ipv6GroupPrefix, SWW_WAF_IPV6_GROUP_PREFIX, a /64 by default, since // length SWW_WAF_IPV6_GROUP_PREFIX sets, a /64 by default, since one abuser
// one abuser usually holds a whole /64. An IPv4 address in IPv6 form counts as // usually holds a whole /64. An IPv4 address in IPv6 form counts as IPv4.
// IPv4. func clientGroup(cfg *config.WAF, addr netip.Addr) netip.Prefix {
func clientGroup(ipv6GroupPrefix int, addr netip.Addr) netip.Prefix {
addr = addr.Unmap() addr = addr.Unmap()
if addr.Is6() { if addr.Is6() {
return netip.PrefixFrom(addr, ipv6GroupPrefix).Masked() return netip.PrefixFrom(addr, cfg.IPv6GroupPrefix).Masked()
} }
return netip.PrefixFrom(addr, addr.BitLen()) return netip.PrefixFrom(addr, addr.BitLen())
+21 -33
View File
@@ -10,7 +10,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/coreruleset" "sneak.berlin/go/smallwebwaf/internal/coreruleset"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
@@ -23,22 +22,11 @@ import (
// exploitsignature, raises the core_rule_set_match alert, and in block mode // exploitsignature, raises the core_rule_set_match alert, and in block mode
// refuses the request with 403, which is an offence its client's history // refuses the request with 403, which is an offence its client's history
// counts. Its figures are the score and the threshold. // counts. Its figures are the score and the threshold.
type coreRuleSet struct { type coreRuleSet struct{}
// inspector is the Core Rule Set, nil while SWW_WAF_CORE_RULE_SET_MODE is
// off.
inspector CoreRuleSetInspector
metrics *metrics.Metrics
alerts *alerts.Queue
mode config.CoreRuleSetMode // SWW_WAF_CORE_RULE_SET_MODE
exemptPaths []string // SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS
scoreThreshold int // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD
observe bool // SWW_WAF_MODE is observe
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
}
func (coreRuleSet) name() string { return "core_rule_set" } func (coreRuleSet) name() string { return "core_rule_set" }
func (c coreRuleSet) check( func (coreRuleSet) check(
_ context.Context, _ context.Context,
rq *request, rq *request,
_ time.Time, _ time.Time,
@@ -48,6 +36,7 @@ func (c coreRuleSet) check(
return nil return nil
} }
cfg := rq.h.config
result := findings.inspection result := findings.inspection
rq.requestBytes = int64(len(findings.read)) rq.requestBytes = int64(len(findings.read))
@@ -58,18 +47,18 @@ func (c coreRuleSet) check(
rq.line.CoreRuleSetScore = &result.Score rq.line.CoreRuleSetScore = &result.Score
for _, id := range result.RuleIDs { for _, id := range result.RuleIDs {
c.metrics.CoreRuleSetMatched(string(c.mode), id) rq.h.metrics.CoreRuleSetMatched(string(cfg.CoreRuleSetMode), id)
} }
threshold := c.scoreThreshold threshold := cfg.CoreRuleSetScoreThreshold
if threshold == 0 || result.Score < threshold { if threshold == 0 || result.Score < threshold {
return nil return nil
} }
rq.addTag(config.ExploitSignatureTag) rq.addTag(config.ExploitSignatureTag)
c.alertCoreRuleSetMatch(rq, result) rq.alertCoreRuleSetMatch(result)
if !c.refusedByCoreRuleSet(result) { if !rq.refusedByCoreRuleSet(result) {
return nil return nil
} }
@@ -87,21 +76,20 @@ func (c coreRuleSet) check(
// refusedByCoreRuleSet reports whether coreRuleSet refuses a request the Core // refusedByCoreRuleSet reports whether coreRuleSet refuses a request the Core
// Rule Set found result of: whether the score is at or over // Rule Set found result of: whether the score is at or over
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD in block mode. // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD in block mode.
func (c coreRuleSet) refusedByCoreRuleSet(result coreruleset.Result) bool { func (rq *request) refusedByCoreRuleSet(result coreruleset.Result) bool {
threshold := c.scoreThreshold cfg := rq.h.config
threshold := cfg.CoreRuleSetScoreThreshold
return c.mode == config.CoreRuleSetModeBlock && return cfg.CoreRuleSetMode == config.CoreRuleSetModeBlock &&
threshold != 0 && result.Score >= threshold threshold != 0 && result.Score >= threshold
} }
// alertCoreRuleSetMatch raises the core_rule_set_match alert for rq, which the // alertCoreRuleSetMatch raises the core_rule_set_match alert for the request,
// Core Rule Set scored at result, at or over // which the Core Rule Set scored at result, at or over
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD. Its detail gives the rule ids, the // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD. Its detail gives the rule ids, the
// score, the method and the path with the query, and, for a request that is // score, the method and the path with the query, and, for a request that is
// not refused for it, the mode: detect, or observe in observe mode. // not refused for it, the mode: detect, or observe in observe mode.
func (c coreRuleSet) alertCoreRuleSetMatch( func (rq *request) alertCoreRuleSetMatch(result coreruleset.Result) {
rq *request, result coreruleset.Result,
) {
detail := map[string]any{ detail := map[string]any{
"rule_ids": result.RuleIDs, "rule_ids": result.RuleIDs,
"score": result.Score, "score": result.Score,
@@ -110,20 +98,20 @@ func (c coreRuleSet) alertCoreRuleSetMatch(
} }
switch { switch {
case c.mode == config.CoreRuleSetModeDetect: case rq.h.config.CoreRuleSetMode == config.CoreRuleSetModeDetect:
detail["mode"] = string(config.CoreRuleSetModeDetect) detail["mode"] = string(config.CoreRuleSetModeDetect)
case c.observe: case rq.h.config.Observe:
detail["mode"] = "observe" detail["mode"] = "observe"
} }
c.alerts.Raise(alerts.Alert{ rq.h.alerts.Raise(alerts.Alert{
Event: alerts.EventCoreRuleSetMatch, Event: alerts.EventCoreRuleSetMatch,
Host: rq.site, Host: rq.site,
Client: rq.client, Client: rq.client,
Netblock: clientGroup(c.ipv6GroupPrefix, rq.client), Netblock: clientGroup(rq.h.config, rq.client),
ASN: rq.lookupAnswer.ASN, ASN: rq.line.ASN,
ASName: rq.lookupAnswer.ASName, ASName: rq.line.ASName,
Country: rq.lookupAnswer.Country, Country: rq.line.Country,
Reason: "scored by the Core Rule Set at or over " + Reason: "scored by the Core Rule Set at or over " +
"SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD", "SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD",
Detail: detail, Detail: detail,
+3 -3
View File
@@ -12,7 +12,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/waf"
) )
// The Core Rule Set's settings the tests set, besides SWW_WAF_CORE_RULE_SET_MODE, and // The Core Rule Set's settings the tests set, besides SWW_WAF_CORE_RULE_SET_MODE, and
@@ -395,9 +395,9 @@ func TestGiteasFixesApplyOnlyToAHostnameThatServesGitea(t *testing.T) {
}) })
wantDisposition(t, do(t, s.question(siteA, client, fileName)), wantDisposition(t, do(t, s.question(siteA, client, fileName)),
http.StatusOK, wafprotocol.DispositionAllow) http.StatusOK, waf.DispositionAllow)
wantDisposition(t, do(t, s.question(siteB, client, fileName)), wantDisposition(t, do(t, s.question(siteB, client, fileName)),
http.StatusForbidden, wafprotocol.DispositionDeny) http.StatusForbidden, waf.DispositionDeny)
lines := s.out.requestLines(t, 2) lines := s.out.requestLines(t, 2)
wantLine(t, lines[0], http.StatusOK, requestlog.ActionForward) wantLine(t, lines[0], http.StatusOK, requestlog.ActionForward)
+4 -6
View File
@@ -15,19 +15,17 @@ import (
// does not hold the client's country as it was looked up. A client without a // does not hold the client's country as it was looked up. A client without a
// country, or whose country cannot be found, is refused. Its figures are the // country, or whose country cannot be found, is refused. Its figures are the
// country and the setting that refused it. // country and the setting that refused it.
type countries struct { type countries struct{}
allowed []string // SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES
}
func (countries) name() string { return "countries" } func (countries) name() string { return "countries" }
func (c countries) check( func (countries) check(
_ context.Context, _ context.Context,
rq *request, rq *request,
_ time.Time, _ time.Time,
) *verdict { ) *verdict {
allowed := c.allowed allowed := rq.h.config.ExclusivelyAllowedCountries
country := rq.lookupAnswer.Country country := rq.line.Country
if len(allowed) == 0 || slices.Contains(allowed, country) { if len(allowed) == 0 || slices.Contains(allowed, country) {
return nil return nil
+3 -10
View File
@@ -4,7 +4,6 @@ package waf
import ( import (
"context" "context"
"net/netip"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
@@ -12,18 +11,12 @@ import (
// denyNets refuses a client in SWW_WAF_DENY_NETS. Its figure is the netblock of // denyNets refuses a client in SWW_WAF_DENY_NETS. Its figure is the netblock of
// the setting the client is in. // the setting the client is in.
type denyNets struct { type denyNets struct{}
netblocks []netip.Prefix // SWW_WAF_DENY_NETS
}
func (denyNets) name() string { return "deny_nets" } func (denyNets) name() string { return "deny_nets" }
func (d denyNets) check( func (denyNets) check(_ context.Context, rq *request, _ time.Time) *verdict {
_ context.Context, for _, netblock := range rq.h.config.DenyNets {
rq *request,
_ time.Time,
) *verdict {
for _, netblock := range d.netblocks {
if netblock.Contains(rq.client) { if netblock.Contains(rq.client) {
return &verdict{ return &verdict{
action: requestlog.ActionClientDenied, action: requestlog.ActionClientDenied,
-47
View File
@@ -1,47 +0,0 @@
package waf
import (
"net/netip"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
func TestDenyNetsRefusesOnlyAClientInsideItsNetblocks(t *testing.T) {
t.Parallel()
heuristic := denyNets{
netblocks: []netip.Prefix{netip.MustParsePrefix("198.51.100.0/24")},
}
// The request has no handler: the heuristic reads only what it holds.
inside := &request{
decision: decision{client: netip.MustParseAddr("198.51.100.7")},
}
v := heuristic.check(t.Context(), inside, time.Time{})
if v == nil {
t.Fatal("a client inside the netblock was let through")
}
if v.action != requestlog.ActionClientDenied {
t.Errorf(
"action = %q, want %q",
v.action,
requestlog.ActionClientDenied,
)
}
if got := v.figures[figureNetblock]; got != "198.51.100.0/24" {
t.Errorf("netblock figure = %v, want 198.51.100.0/24", got)
}
outside := &request{
decision: decision{client: netip.MustParseAddr("203.0.113.7")},
}
if v := heuristic.check(t.Context(), outside, time.Time{}); v != nil {
t.Errorf("a client outside the netblock was refused: %+v", v)
}
}
+6 -14
View File
@@ -9,7 +9,6 @@ import (
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
@@ -20,28 +19,21 @@ import (
// being log, it does nothing more. A zone without a verdict on the client is // being log, it does nothing more. A zone without a verdict on the client is
// asked about it in the background, and the request does not wait for the // asked about it in the background, and the request does not wait for the
// answer. Its figure is the zones that list the client, their keys masked. // answer. Its figure is the zones that list the client, their keys masked.
type dnsbl struct { type dnsbl struct{}
dnsbl *reputation.DNSBL
action config.ReputationAction // SWW_WAF_DNSBL_ABUSEIPDB_ACTION
hitNoter reputationHitNoter
}
func (dnsbl) name() string { return "dnsbl" } func (dnsbl) name() string { return "dnsbl" }
func (d dnsbl) check( func (dnsbl) check(ctx context.Context, rq *request, _ time.Time) *verdict {
ctx context.Context, listedBy := rq.h.dnsbl.ListedBy(ctx, rq.client)
rq *request,
_ time.Time,
) *verdict {
listedBy := d.dnsbl.ListedBy(ctx, rq.client)
rq.dnsblListed = len(listedBy) > 0 rq.dnsblListed = len(listedBy) > 0
d.hitNoter.noteListed(rq, listedBy, "listed by a DNSBL zone") rq.noteListed(listedBy, "listed by a DNSBL zone")
for _, zone := range listedBy { for _, zone := range listedBy {
rq.addTag(config.DNSBLTagPrefix + strings.ToLower(zone)) rq.addTag(config.DNSBLTagPrefix + strings.ToLower(zone))
} }
if !rq.dnsblListed || d.action != config.ReputationActionDeny { if !rq.dnsblListed ||
rq.h.config.DNSBLAbuseIPDBAction != config.ReputationActionDeny {
return nil return nil
} }
+3 -19
View File
@@ -5,7 +5,7 @@ import (
"testing" "testing"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/waf"
) )
func TestHealthEndpointCountsTowardNoLimit(t *testing.T) { func TestHealthEndpointCountsTowardNoLimit(t *testing.T) {
@@ -25,13 +25,8 @@ func TestHealthEndpointCountsTowardNoLimit(t *testing.T) {
) )
for range healthChecks { for range healthChecks {
got := s.ownEndpointRequest( got := s.ownEndpointRequest("", http.MethodGet, waf.HealthPath, "",
"", http.StatusOK)
http.MethodGet,
wafprotocol.HealthPath,
"",
http.StatusOK,
)
if string(got.body) != "ok\n" || if string(got.body) != "ok\n" ||
got.header.Get("Content-Type") != contentType { got.header.Get("Content-Type") != contentType {
@@ -58,14 +53,3 @@ func TestHealthEndpointCountsTowardNoLimit(t *testing.T) {
} }
} }
} }
func TestHealthEndpointIsFoundAtItsPathPercentEncoded(t *testing.T) {
t.Parallel()
// %68 is h. An endpoint is found by the decoded path.
s, _, _ := startWithClock(t, "", nil)
got := s.ownEndpointRequest("", http.MethodGet, "/_sww/%68ealthz", "",
http.StatusOK)
wantBody(t, got, "ok\n")
}
+71 -23
View File
@@ -5,20 +5,9 @@ import (
"time" "time"
) )
// To add a heuristic, write one file with a struct that implements heuristic // To add a heuristic, write one file with a type that implements heuristic,
// and holds the settings and parts it reads. Add one line for it to one of the // and add one line for it to one of the lists below. Its settings go in
// lists that newHandler builds, which hands it those settings and parts. Its // config.WAF, and the comment at the top of its file names them.
// settings go in config.WAF, and the comment at the top of its file names them.
// A heuristic of afterLookup that can refuse a request whatever the lookup
// answers also implements refusesWithoutLookup.
//
// Three places outside its file may change with it:
// - waitForGeoJS in newHandler lists the settings that need the lookup's
// answer;
// - requestCanGetTag in internal/config/waf.go holds which setting can give a
// request which tag;
// - findBesideLookup does the work of each heuristic of besideLookup, by
// name.
// heuristic is one of the checks that decide what waf does with a // heuristic is one of the checks that decide what waf does with a
// request. // request.
@@ -30,15 +19,6 @@ type heuristic interface {
check(ctx context.Context, rq *request, now time.Time) *verdict check(ctx context.Context, rq *request, now time.Time) *verdict
} }
// refusesWithoutLookup is a heuristic of afterLookup that can tell, before the
// lookup answers, that it will refuse a request whatever the answer.
type refusesWithoutLookup interface {
// refusesWhateverTheLookupAnswers reports whether the heuristic will refuse
// rq at now, whatever the lookup answers about its client. It changes
// nothing.
refusesWhateverTheLookupAnswers(rq *request, now time.Time) bool
}
// verdict is a heuristic refusing a request, or banning its client: the action // verdict is a heuristic refusing a request, or banning its client: the action
// the log line names, one of requestlog's Action constants, and the figures the // the log line names, one of requestlog's Action constants, and the figures the
// heuristic acted on, by name, such as the count and the limit. // heuristic acted on, by name, such as the count and the limit.
@@ -56,6 +36,74 @@ const (
figureNetblock = "netblock" figureNetblock = "netblock"
) )
// The lists of heuristics, each in the order its heuristics run. A client in
// SWW_WAF_ALLOW_NETS skips them all.
// beforeLookup are the heuristics that run before the client's AS number and
// country are looked up, so that a client they refuse is not looked up.
// header_limits comes first: it only measures and counts, and a request it
// refuses causes no other work.
func beforeLookup() []heuristic {
return []heuristic{
headerLimits{},
tagLists{},
routeWatchTags{},
denyNets{},
activeBan{},
}
}
// afterLookup are the heuristics that run once the lookup has been made.
func afterLookup() []heuristic {
return []heuristic{
countries{},
blocklists{},
dnsbl{},
abuseIPDB{},
tagDeny{},
rateLimits{},
}
}
// besideLookup are the heuristics that only read the request. Their work runs
// beside the lookup and afterLookup, in findBesideLookup, which changes nothing
// of the request. Their check then applies what that work found, and runs only
// once waf takes it.
func besideLookup() []heuristic {
return []heuristic{
trapPaths{},
ruleFiles{},
coreRuleSet{},
}
}
// afterBesideLookup are the heuristics that run once afterLookup and
// besideLookup have both let the request through. tag_deny runs a second time,
// on the tag rule_files and core_rule_set can add.
func afterBesideLookup() []heuristic {
return []heuristic{
tagDeny{},
}
}
// afterAnswer are the heuristics that run on a request once waf has
// answered it, before its decision line is written. They ban the client, and do
// not change what the request got: their verdict's action is not used.
func afterAnswer() []heuristic {
return []heuristic{
refusalLimit{},
}
}
// onReport are the heuristics that run on an allowed request when rproxy's
// report of it comes, before its report line is written. Like afterAnswer, they
// only ban.
func onReport() []heuristic {
return []heuristic{
byteLimits{},
}
}
// runHeuristics runs list, one of those checkClient runs, on the request at now, // runHeuristics runs list, one of those checkClient runs, on the request at now,
// and returns the first verdict, or nil when none gives one. The log line's // and returns the first verdict, or nil when none gives one. The log line's
// heuristic and heuristic_figures name the heuristic that gave it, and give its // heuristic and heuristic_figures name the heuristic that gave it, and give its
+1 -2
View File
@@ -11,7 +11,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
func TestHistoryKeepsEachRequestOfTheClient(t *testing.T) { func TestHistoryKeepsEachRequestOfTheClient(t *testing.T) {
@@ -119,7 +118,7 @@ func TestOwnEndpointsAreNotInTheHistory(t *testing.T) {
s.ownEndpointRequest( s.ownEndpointRequest(
"", "",
http.MethodGet, http.MethodGet,
wafprotocol.HealthPath, waf.HealthPath,
"", "",
http.StatusOK, http.StatusOK,
) )
+27 -7
View File
@@ -3,6 +3,7 @@ package waf
import ( import (
"context" "context"
"log/slog" "log/slog"
"net/http"
"net/netip" "net/netip"
"strings" "strings"
"time" "time"
@@ -15,6 +16,15 @@ import (
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
) )
// The headers in which an answer of allow gives rproxy the client's AS number
// and country, for the app, while SWW_WAF_ADD_LOOKUP_HEADERS is true. Go writes
// every header name in this form, as it sends it and as it receives it, so
// X-Client-ASN arrives as X-Client-Asn; header names are not case-sensitive.
const (
asnHeader = "X-Client-Asn"
countryHeader = "X-Client-Country"
)
// lookupResult is what the lookup gave for a request: whether its client was // lookupResult is what the lookup gave for a request: whether its client was
// looked up, the answer, and whether the request skipped the wait for GeoJS. // looked up, the answer, and whether the request skipped the wait for GeoJS.
type lookupResult struct { type lookupResult struct {
@@ -46,7 +56,7 @@ func (rq *request) startLookUp(
return result return result
} }
client := clientGroup(rq.h.config.IPv6GroupPrefix, rq.client) client := clientGroup(rq.h.config, rq.client)
if rq.h.config.LookupSource == config.LookupSourceFile { if rq.h.config.LookupSource == config.LookupSourceFile {
result <- lookupResult{ result <- lookupResult{
@@ -94,11 +104,11 @@ func (rq *request) noteLookup(result lookupResult) {
rq.line.ASName = rq.lookupAnswer.ASName rq.line.ASName = rq.lookupAnswer.ASName
rq.line.Country = rq.lookupAnswer.Country rq.line.Country = rq.lookupAnswer.Country
if tag := asnTag(rq.lookupAnswer.ASN); tag != "" { if tag := asnTag(rq.line.ASN); tag != "" {
rq.addTag(tag) rq.addTag(tag)
} }
if tag := countryTag(rq.lookupAnswer.Country); tag != "" { if tag := countryTag(rq.line.Country); tag != "" {
rq.addTag(tag) rq.addTag(tag)
} }
} }
@@ -133,10 +143,8 @@ func AddLookup(
) { ) {
limiter.AddLookup(answer.Client, answer.Answered, limiter.AddLookup(answer.Client, answer.Answered,
answer.ASN, answer.ASName, answer.Country) answer.ASN, answer.ASName, answer.Country)
netblock := banNetblock( ledger.AddLookup(banNetblock(cfg, answer.Client.Addr()),
cfg.BanScopeV4Prefix, cfg.IPv6GroupPrefix, answer.Client.Addr(), answer.ASN, answer.ASName, answer.Country)
)
ledger.AddLookup(netblock, answer.ASN, answer.ASName, answer.Country)
} }
// GeoJSFailed logs a failure of GeoJS to processLog, raises a // GeoJSFailed logs a failure of GeoJS to processLog, raises a
@@ -159,6 +167,18 @@ func GeoJSFailed(
m.GeoJSFailures.Inc() m.GeoJSFailures.Inc()
} }
// setLookupHeaders sets the headers that give the client's AS number and
// country, leaving out one that is unknown.
func setLookupHeaders(header http.Header, asn, country string) {
if asn != "" {
header.Set(asnHeader, asn)
}
if country != "" {
header.Set(countryHeader, country)
}
}
// canBePlaced reports whether a lookup can place addr: private, loopback // canBePlaced reports whether a lookup can place addr: private, loopback
// and link-local addresses have no AS number or country. // and link-local addresses have no AS number or country.
func canBePlaced(addr netip.Addr) bool { func canBePlaced(addr netip.Addr) bool {
+1 -2
View File
@@ -16,7 +16,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
const ( const (
@@ -67,7 +66,7 @@ func TestMetricsNeedTheTokenAndOtherPathsAreNotFound(t *testing.T) {
{http.MethodPost, waf.MetricsPath, bearer, http.StatusNotFound}, {http.MethodPost, waf.MetricsPath, bearer, http.StatusNotFound},
{http.MethodGet, waf.MetricsPath + "/", bearer, http.StatusNotFound}, {http.MethodGet, waf.MetricsPath + "/", bearer, http.StatusNotFound},
{http.MethodGet, "/_sww/bans", bearer, http.StatusNotFound}, {http.MethodGet, "/_sww/bans", bearer, http.StatusNotFound},
{http.MethodPost, wafprotocol.HealthPath, "", http.StatusNotFound}, {http.MethodPost, waf.HealthPath, "", http.StatusNotFound},
} { } {
got := s.ownEndpointRequest( got := s.ownEndpointRequest(
tc.authorization, tc.authorization,
+29 -157
View File
@@ -28,6 +28,14 @@ import (
"sneak.berlin/go/smallwebwaf/internal/rules" "sneak.berlin/go/smallwebwaf/internal/rules"
) )
// ownEndpointPrefix starts the path of each of waf's own endpoints. The app
// never sees a path under it.
const ownEndpointPrefix = "/_sww/"
// HealthPath is the health endpoint of waf and of rproxy, which the
// container's health check asks.
const HealthPath = "/_sww/healthz"
// MetricsPath is where the metrics are, for a request that carries // MetricsPath is where the metrics are, for a request that carries
// SWW_WAF_METRICS_TOKEN. // SWW_WAF_METRICS_TOKEN.
const MetricsPath = "/_sww/metrics" const MetricsPath = "/_sww/metrics"
@@ -125,19 +133,15 @@ type Server struct {
} }
// newHandler returns the handler with what every request shares, the parts // newHandler returns the handler with what every request shares, the parts
// in params among it. It builds the heuristics, each with the settings and // in params among it.
// parts it reads, and the lists they run in.
//
//nolint:funlen // a few lines for each heuristic, a list that grows with them
func newHandler(params Params) *handler { func newHandler(params Params) *handler {
awaitingReport, err := lru.New[string, *decision](maxAwaitingReport) awaitingReport, err := lru.New[string, *decision](maxAwaitingReport)
if err != nil { if err != nil {
panic(err) // New fails only for a size below one panic(err) // New fails only for a size below one
} }
cfg := params.Config return &handler{
h := &handler{ config: params.Config,
config: cfg,
requestLog: params.RequestLog, requestLog: params.RequestLog,
processLog: params.ProcessLog, processLog: params.ProcessLog,
now: params.Now, now: params.Now,
@@ -148,127 +152,25 @@ func newHandler(params Params) *handler {
geojs: params.GeoJS, geojs: params.GeoJS,
anomalies: params.Anomalies, anomalies: params.Anomalies,
lookupFile: params.LookupFile, lookupFile: params.LookupFile,
lists: params.Lists,
dnsbl: params.DNSBL,
abuseIPDB: params.AbuseIPDB,
routeWatch: params.RouteWatch,
rules: params.Rules,
coreRuleSet: params.CoreRuleSet,
alerts: params.Alerts,
// The country list, the headers, the tag settings that name an asn= // The country list, the headers, the tag settings that name an asn=
// or country= tag and the biased thresholds act on the answer before // or country= tag and the biased thresholds act on the answer before
// the request goes on. // the request goes on.
waitForGeoJS: len(cfg.ExclusivelyAllowedCountries) > 0 || waitForGeoJS: len(params.Config.ExclusivelyAllowedCountries) > 0 ||
cfg.AddLookupHeaders || params.Config.AddLookupHeaders ||
cfg.TagsNeedLookup() || params.Config.TagsNeedLookup() ||
biasedThresholdsSet(cfg), biasedThresholdsSet(params.Config),
besideLookupLimit: cmp.Or( besideLookupLimit: cmp.Or(
params.BesideLookupTimeLimit, params.BesideLookupTimeLimit,
besideLookupTimeLimit, besideLookupTimeLimit,
), ),
} }
sharedBanMaker := banMaker{
ledger: params.Ledger,
limiter: params.Limiter,
metrics: params.Metrics,
alerts: params.Alerts,
observe: cfg.Observe,
banResponse: cfg.BanResponse,
banScopeV4Prefix: cfg.BanScopeV4Prefix,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
}
hitNoter := reputationHitNoter{
metrics: params.Metrics,
alerts: params.Alerts,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
}
h.trapPaths = trapPaths{paths: cfg.TrapPaths, banMaker: sharedBanMaker}
h.ruleFiles = ruleFiles{
rules: params.Rules,
metrics: params.Metrics,
banMaker: sharedBanMaker,
}
h.coreRuleSet = coreRuleSet{
inspector: params.CoreRuleSet,
metrics: params.Metrics,
alerts: params.Alerts,
mode: cfg.CoreRuleSetMode,
exemptPaths: cfg.CoreRuleSetExemptPaths,
scoreThreshold: cfg.CoreRuleSetScoreThreshold,
observe: cfg.Observe,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
}
h.beforeLookup = []heuristic{
headerLimits{},
tagLists{lists: params.Lists},
routeWatchTags{
routeWatch: params.RouteWatch,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
},
denyNets{netblocks: cfg.DenyNets},
activeBan{banMaker: sharedBanMaker},
}
h.afterLookup = []heuristic{
countries{allowed: cfg.ExclusivelyAllowedCountries},
blocklists{
lists: params.Lists,
action: cfg.BlocklistAction,
hitNoter: hitNoter,
},
dnsbl{
dnsbl: params.DNSBL,
action: cfg.DNSBLAbuseIPDBAction,
hitNoter: hitNoter,
},
abuseIPDB{
abuseIPDB: params.AbuseIPDB,
limiter: params.Limiter,
keySet: cfg.AbuseIPDBKey != "",
minScore: cfg.AbuseIPDBMinScore,
action: cfg.DNSBLAbuseIPDBAction,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
hitNoter: hitNoter,
},
tagDeny{tags: cfg.TagDeny},
rateLimits{
limiter: params.Limiter,
exemptNets: cfg.RateLimitExemptNets,
exemptPaths: cfg.RateLimitExemptPaths,
banResponse: cfg.BanResponse,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
biasedThresholds: biasedThresholds{
lists: params.Lists,
tagLimitPercent: cfg.TagLimitPercent,
asnBytesLimitPercent: cfg.ASNBytesLimitPercent,
countryBytesLimitPercent: cfg.CountryBytesLimitPercent,
unknownCountryLimitPercent: cfg.UnknownCountryLimitPercent,
blocklistAction: cfg.BlocklistAction,
blocklistLimitPercent: cfg.BlocklistLimitPercent,
dnsblAbuseIPDBAction: cfg.DNSBLAbuseIPDBAction,
dnsblAbuseIPDBLimitPercent: cfg.DNSBLAbuseIPDBLimitPercent,
},
banMaker: sharedBanMaker,
},
}
h.besideLookup = []heuristic{h.trapPaths, h.ruleFiles, h.coreRuleSet}
h.afterBesideLookup = []heuristic{tagDeny{tags: cfg.TagDeny}}
h.afterAnswer = []heuristic{
refusalLimit{
limiter: params.Limiter,
perMinute: cfg.RefusalsLimitPerMinute,
banResponse: cfg.BanResponse,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
banMaker: sharedBanMaker,
},
}
h.onReport = []heuristic{
byteLimits{
limiter: params.Limiter,
bytesCounted: cfg.BytesCounted,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
banMaker: sharedBanMaker,
},
}
h.ownEndpoints = h.routes()
return h
} }
// handler is waf's handler. It holds what every request shares; what // handler is waf's handler. It holds what every request shares; what
@@ -284,6 +186,13 @@ type handler struct {
geojs *geojs.GeoJS geojs *geojs.GeoJS
anomalies *anomaly.Counters anomalies *anomaly.Counters
lookupFile *lookup.File lookupFile *lookup.File
lists *reputation.Lists
dnsbl *reputation.DNSBL
abuseIPDB *reputation.AbuseIPDB
routeWatch *routewatch.Client
rules *rules.Files
coreRuleSet CoreRuleSetInspector
alerts *alerts.Queue
// waitForGeoJS is true when a setting needs each request's answer from // waitForGeoJS is true when a setting needs each request's answer from
// GeoJS before the request goes on. // GeoJS before the request goes on.
waitForGeoJS bool waitForGeoJS bool
@@ -293,41 +202,4 @@ type handler struct {
// awaitingReport holds the decision on each answered question by its id // awaitingReport holds the decision on each answered question by its id
// until its report comes, and nil for a question still being handled. // until its report comes, and nil for a question still being handled.
awaitingReport *lru.Cache[string, *decision] awaitingReport *lru.Cache[string, *decision]
// ownEndpoints answers every request that is not a question, as routes
// lists the endpoints.
ownEndpoints http.Handler
// The lists of heuristics, each in the order its heuristics run. A client
// in SWW_WAF_ALLOW_NETS skips them all.
// beforeLookup are the heuristics that run before the client's AS number
// and country are looked up, so that a client they refuse is not looked
// up. header_limits comes first: it only measures and counts, and a
// request it refuses causes no other work.
beforeLookup []heuristic
// afterLookup are the heuristics that run once the lookup has been made.
afterLookup []heuristic
// besideLookup are the heuristics that only read the request. Their work
// runs beside the lookup and afterLookup, in findBesideLookup, which
// changes nothing of the request. Their check then applies what that work
// found, and runs only once waf takes it.
besideLookup []heuristic
// afterBesideLookup are the heuristics that run once afterLookup and
// besideLookup have both let the request through. tag_deny runs a second
// time, on the tag rule_files and core_rule_set can add.
afterBesideLookup []heuristic
// afterAnswer are the heuristics that run on a request once waf has
// answered it, before its decision line is written. They ban the client,
// and do not change what the request got: their verdict's action is not
// used.
afterAnswer []heuristic
// onReport are the heuristics that run on an allowed request when
// rproxy's report of it comes, before its report line is written. Like
// afterAnswer, they only ban.
onReport []heuristic
// The heuristics of besideLookup, whose work findBesideLookup does.
trapPaths trapPaths
ruleFiles ruleFiles
coreRuleSet coreRuleSet
} }
+38 -45
View File
@@ -24,7 +24,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/lookup" "sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/metrics" "sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/processlog"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation" "sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
@@ -194,11 +193,11 @@ func (o *output) requestLine(t *testing.T) logLine {
// endedLines waits for count requests that went through rproxy, or to waf's // endedLines waits for count requests that went through rproxy, or to waf's
// own endpoints, to have ended, and returns the last log line of each, in the // own endpoints, to have ended, and returns the last log line of each, in the
// order waf took them. A question's last line is the one its report writes. // order waf took them. A question's last line is the one its report writes.
// It waits as long as it takes, as requestWithBody does.
func (o *output) endedLines(t *testing.T, count int) []logLine { func (o *output) endedLines(t *testing.T, count int) []logLine {
t.Helper() t.Helper()
for { deadline := time.Now().Add(waitLimit)
for time.Now().Before(deadline) {
found := o.lastLines(t) found := o.lastLines(t)
ended := len(found) >= count ended := len(found) >= count
@@ -214,6 +213,10 @@ func (o *output) endedLines(t *testing.T, count int) []logLine {
time.Sleep(pollInterval) time.Sleep(pollInterval)
} }
t.Fatalf("fewer than %d requests have ended after %s", count, waitLimit)
return nil
} }
// lastLines returns the last request log line written so far of each request, // lastLines returns the last request log line written so far of each request,
@@ -395,11 +398,11 @@ func startRProxy(
server := rproxy.New(rproxy.Params{ server := rproxy.New(rproxy.Params{
Config: cfg, Config: cfg,
RequestLog: io.Discard, RequestLog: io.Discard,
ProcessLog: processlog.New(processlog.Params{ ProcessLog: requestlog.NewProcessLogger(
Output: io.Discard, io.Discard,
InstanceName: "rproxy", "rproxy",
Level: slog.LevelInfo, slog.LevelInfo,
}), ),
}) })
listener, err := (&net.ListenConfig{}).Listen( listener, err := (&net.ListenConfig{}).Listen(
@@ -468,11 +471,11 @@ func newWAFWith(
} }
out := &output{} out := &output{}
processLog := processlog.New(processlog.Params{ processLog := requestlog.NewProcessLogger(
Output: out, out,
InstanceName: cfg.InstanceName, cfg.InstanceName,
Level: cfg.LogLevel, cfg.LogLevel,
}) )
ruleFiles, err := rules.Load(rules.Params{ ruleFiles, err := rules.Load(rules.Params{
Dir: cfg.RuleFilesDir, Enabled: cfg.RuleFilesEnabled, ProcessLog: processLog, Dir: cfg.RuleFilesDir, Enabled: cfg.RuleFilesEnabled, ProcessLog: processLog,
@@ -548,22 +551,24 @@ func addCoreRuleSet(t *testing.T, params *waf.Params) {
params.CoreRuleSet = coreRuleSet params.CoreRuleSet = coreRuleSet
} }
// addCounters adds to params the table of clients, the ledger and the // addCounters adds to params the metrics, the table of clients, the ledger
// anomaly counters, as sww builds them from params' settings. // and the anomaly counters, as sww builds them from params' settings. The
// metrics read the ledger, the table of clients and the rules.
func addCounters(params *waf.Params) { func addCounters(params *waf.Params) {
cfg := params.Config cfg := params.Config
params.Limiter = ratelimit.New(ratelimit.Params{ params.Metrics = metrics.New(
Limits: ratelimit.Limits{ cfg.MetricsBusiestCountriesAndASNs,
cfg.InstanceName,
)
params.Limiter = ratelimit.New(ratelimit.Limits{
PerMinute: cfg.RateLimitPerMinute, PerMinute: cfg.RateLimitPerMinute,
PerHour: cfg.RateLimitPerHour, PerHour: cfg.RateLimitPerHour,
PerDay: cfg.RateLimitPerDay, PerDay: cfg.RateLimitPerDay,
BytesPerMinute: cfg.BytesLimitPerMinute, BytesPerMinute: cfg.BytesLimitPerMinute,
BytesPerHour: cfg.BytesLimitPerHour, BytesPerHour: cfg.BytesLimitPerHour,
BytesPerDay: cfg.BytesLimitPerDay, BytesPerDay: cfg.BytesLimitPerDay,
}, }, cfg.MaxTrackedClients)
MaxClients: cfg.MaxTrackedClients,
})
params.Ledger = bans.New(bans.Params{ params.Ledger = bans.New(bans.Params{
LimitBanDuration: cfg.LimitBanDuration, LimitBanDuration: cfg.LimitBanDuration,
LimitBanRepeatWindow: cfg.LimitBanRepeatWindow, LimitBanRepeatWindow: cfg.LimitBanRepeatWindow,
@@ -582,18 +587,26 @@ func addCounters(params *waf.Params) {
NamedNetblocks: cfg.AnomalyWatchedNets, NamedNetblocks: cfg.AnomalyWatchedNets,
Alerts: params.Alerts, Alerts: params.Alerts,
}) })
params.Metrics.AddBansAndClients(
params.Ledger,
params.Limiter,
params.Now,
)
params.Metrics.AddRules(params.Rules)
} }
// addOutsideSources adds to params the lists fetched from URLs, the DNSBL // addOutsideSources adds to params the lists fetched from URLs, the DNSBL
// zones' verdicts, AbuseIPDB's scores from abuseIPDBCheckURL and its // zones' verdicts, AbuseIPDB's scores from abuseIPDBCheckURL and its
// blacklist from abuseIPDBBlacklistURL, routewatch's answers while its // blacklist from abuseIPDBBlacklistURL, routewatch's answers while its
// setting is set, the metrics newMetrics returns, and GeoJS at geojsURL, as // setting is set, and GeoJS at geojsURL, as sww builds
// sww builds them from params' settings and the parts addCounters added. // them from params' settings and the parts addCounters added. The metrics
// read the lists, the zones' verdicts, and AbuseIPDB's while its key is set.
func addOutsideSources( func addOutsideSources(
params *waf.Params, geojsURL, abuseIPDBCheckURL string, params *waf.Params, geojsURL, abuseIPDBCheckURL string,
) { ) {
cfg := params.Config cfg := params.Config
limiter, ledger := params.Limiter, params.Ledger m, limiter, ledger := params.Metrics, params.Limiter, params.Ledger
processLog, alertQueue := params.ProcessLog, params.Alerts processLog, alertQueue := params.ProcessLog, params.Alerts
params.Lists = reputation.New(reputation.Params{ params.Lists = reputation.New(reputation.Params{
@@ -642,9 +655,6 @@ func addOutsideSources(
}) })
} }
m := newMetrics(params)
params.Metrics = m
params.GeoJS = geojs.New(geojs.Params{ params.GeoJS = geojs.New(geojs.Params{
URL: geojsURL, URL: geojsURL,
Answered: func(answer geojs.Answer) { Answered: func(answer geojs.Answer) {
@@ -657,29 +667,12 @@ func addOutsideSources(
}, },
WentWithoutAnswer: m.RequestsWithoutGeoJSAnswer.Inc, WentWithoutAnswer: m.RequestsWithoutGeoJSAnswer.Inc,
}) })
}
// newMetrics returns the metrics, as sww builds them from params' settings, m.AddReputation(params.Lists, params.DNSBL)
// read from the ledger, the table of clients, the rules, the lists, the
// zones' verdicts, and AbuseIPDB's while its key is set.
func newMetrics(params *waf.Params) *metrics.Metrics {
cfg := params.Config
metricsParams := metrics.Params{
BusiestCount: cfg.MetricsBusiestCountriesAndASNs,
InstanceName: cfg.InstanceName,
Now: params.Now,
Bans: params.Ledger,
Clients: params.Limiter,
Rules: params.Rules,
Lists: params.Lists,
DNSBL: params.DNSBL,
}
if cfg.AbuseIPDBKey != "" { if cfg.AbuseIPDBKey != "" {
metricsParams.AbuseIPDB = params.AbuseIPDB m.AddAbuseIPDB(params.AbuseIPDB)
} }
return metrics.New(metricsParams)
} }
// newClient returns an HTTP client that sends requests as they are made, // newClient returns an HTTP client that sends requests as they are made,
+22 -37
View File
@@ -6,10 +6,8 @@ package waf
import ( import (
"context" "context"
"net/netip"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
@@ -20,37 +18,28 @@ import (
// refused, and bans the client. A request exempt from the rate limits is exempt // refused, and bans the client. A request exempt from the rate limits is exempt
// from the byte limits too. Its figures are the window, the count and the // from the byte limits too. Its figures are the window, the count and the
// limit. // limit.
type rateLimits struct { type rateLimits struct{}
limiter *ratelimit.Limiter
exemptNets []netip.Prefix // SWW_WAF_RATE_LIMIT_EXEMPT_NETS
exemptPaths []string // SWW_WAF_RATE_LIMIT_EXEMPT_PATHS
banResponse int // SWW_WAF_BAN_RESPONSE
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
biasedThresholds biasedThresholds
banMaker banMaker
}
func (rateLimits) name() string { return "rate_limits" } func (rateLimits) name() string { return "rate_limits" }
func (r rateLimits) check( func (rateLimits) check(
_ context.Context, _ context.Context,
rq *request, rq *request,
now time.Time, now time.Time,
) *verdict { ) *verdict {
rq.counted = r.countedByRateLimits(rq) rq.counted = rq.countedByRateLimits()
if !rq.counted { if !rq.counted {
return nil return nil
} }
rq.rateLimitPercent, rq.bytesLimitPercent = rq.rateLimitPercent, rq.bytesLimitPercent = rq.limitPercentages()
r.biasedThresholds.limitPercentages(rq)
rq.line.RateLimitPercent, rq.line.RateLimitPercentSetting = rq.line.RateLimitPercent, rq.line.RateLimitPercentSetting =
rq.rateLimitPercent.logged() rq.rateLimitPercent.logged()
rq.line.BytesLimitPercent, rq.line.BytesLimitPercentSetting = rq.line.BytesLimitPercent, rq.line.BytesLimitPercentSetting =
rq.bytesLimitPercent.logged() rq.bytesLimitPercent.logged()
counts, hit, over := r.limiter.Count( counts, hit, over := rq.h.limiter.Count(
clientGroup(r.ipv6GroupPrefix, rq.client), clientGroup(rq.h.config, rq.client),
now, now,
rq.rateLimitPercent.percent, rq.rateLimitPercent.percent,
) )
@@ -60,32 +49,28 @@ func (r rateLimits) check(
return nil return nil
} }
r.banMaker.banForLimit(rq, now, hit, r.banResponse) rq.banForLimit(now, hit, rq.h.config.BanResponse)
return &verdict{ return &verdict{
action: requestlog.ActionClientIsRateLimited, figures: limitFigures(hit), action: requestlog.ActionClientIsRateLimited, figures: limitFigures(hit),
} }
} }
// refusesWhateverTheLookupAnswers reports whether the heuristic rate_limits // overAWholeRateLimit reports whether the heuristic rate_limits will refuse the
// will refuse rq at now, whatever its client's limit percentages, which the // request at now, whatever its client's limit percentages: the rate limits
// lookup's answer can lower: the rate limits count it, and one more request // count it, and one more request from its client takes it over a whole rate
// from its client takes it over a whole rate limit. It counts nothing. // limit. It counts nothing.
func (r rateLimits) refusesWhateverTheLookupAnswers( func (rq *request) overAWholeRateLimit(now time.Time) bool {
rq *request, return rq.countedByRateLimits() &&
now time.Time, rq.h.limiter.OverARateLimit(clientGroup(rq.h.config, rq.client), now)
) bool {
return r.countedByRateLimits(rq) &&
r.limiter.OverARateLimit(
clientGroup(r.ipv6GroupPrefix, rq.client),
now,
)
} }
// countedByRateLimits reports whether the rate limits count rq: its client is // countedByRateLimits reports whether the rate limits count the request: its
// not in SWW_WAF_RATE_LIMIT_EXEMPT_NETS, and its path is not exempt under // client is not in SWW_WAF_RATE_LIMIT_EXEMPT_NETS, and its path is not exempt
// SWW_WAF_RATE_LIMIT_EXEMPT_PATHS. // under SWW_WAF_RATE_LIMIT_EXEMPT_PATHS.
func (r rateLimits) countedByRateLimits(rq *request) bool { func (rq *request) countedByRateLimits() bool {
return !isInside(rq.client, r.exemptNets) && cfg := rq.h.config
!pathExempt(rq.in, r.exemptPaths)
return !isInside(rq.client, cfg.RateLimitExemptNets) &&
!pathExempt(rq.in, cfg.RateLimitExemptPaths)
} }
+10 -16
View File
@@ -8,7 +8,6 @@ import (
"net/http" "net/http"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
@@ -19,22 +18,17 @@ import (
// a limit, which bans the client. A client in SWW_WAF_ALLOW_NETS, which the // a limit, which bans the client. A client in SWW_WAF_ALLOW_NETS, which the
// heuristics skip, is not counted, and nothing is while the limit is off. // heuristics skip, is not counted, and nothing is while the limit is off.
// Its figures are the window, the count and the limit. // Its figures are the window, the count and the limit.
type refusalLimit struct { type refusalLimit struct{}
limiter *ratelimit.Limiter
perMinute int64 // SWW_WAF_REFUSALS_LIMIT_PER_MINUTE
banResponse int // SWW_WAF_BAN_RESPONSE
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
banMaker banMaker
}
func (refusalLimit) name() string { return "refusal_limit" } func (refusalLimit) name() string { return "refusal_limit" }
func (r refusalLimit) check( func (refusalLimit) check(
_ context.Context, _ context.Context,
rq *request, rq *request,
now time.Time, now time.Time,
) *verdict { ) *verdict {
if r.perMinute == 0 { cfg := rq.h.config
if cfg.RefusalsLimitPerMinute == 0 {
return nil return nil
} }
@@ -42,10 +36,10 @@ func (r refusalLimit) check(
return nil return nil
} }
hit, over := r.limiter.CountRefusal( hit, over := rq.h.limiter.CountRefusal(
clientGroup(r.ipv6GroupPrefix, rq.client), clientGroup(rq.h.config, rq.client),
now, now,
r.perMinute, cfg.RefusalsLimitPerMinute,
) )
if !over { if !over {
return nil return nil
@@ -54,16 +48,16 @@ func (r refusalLimit) check(
// What the client was sent, or in observe mode would have been. // What the client was sent, or in observe mode would have been.
status := rq.out.status status := rq.out.status
switch rq.wouldAction { switch rq.line.WouldAction {
case requestlog.ActionBlockedByBlockRule, case requestlog.ActionBlockedByBlockRule,
requestlog.ActionBlockedByCoreRuleSet: requestlog.ActionBlockedByCoreRuleSet:
status = http.StatusForbidden status = http.StatusForbidden
case requestlog.ActionTemporarilyBanned, case requestlog.ActionTemporarilyBanned,
requestlog.ActionPermanentlyBanned: requestlog.ActionPermanentlyBanned:
status = r.banResponse status = cfg.BanResponse
} }
r.banMaker.banForLimit(rq, now, hit, status) rq.banForLimit(now, hit, status)
return &verdict{figures: limitFigures(hit)} return &verdict{figures: limitFigures(hit)}
} }
+11 -24
View File
@@ -5,35 +5,22 @@ package waf
import ( import (
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/metrics"
) )
// reputationHitNoter notes the reputation sources that list a request's
// client, for the reputation heuristics.
type reputationHitNoter struct {
metrics *metrics.Metrics
alerts *alerts.Queue
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
}
// noteListed notes each of sources, the URLs of the blocklists or the // noteListed notes each of sources, the URLs of the blocklists or the
// DNSBL zones, their keys masked, that list rq's client, as noteHit does, // DNSBL zones, their keys masked, that list the client, as noteHit does,
// with reason. // with reason.
func (n reputationHitNoter) noteListed( func (rq *request) noteListed(sources []string, reason string) {
rq *request, sources []string, reason string,
) {
for _, source := range sources { for _, source := range sources {
n.noteHit(rq, bans.ReputationHit{Source: source}, reason) rq.noteHit(bans.ReputationHit{Source: source}, reason)
} }
} }
// noteHit adds hit's source, which lists rq's client, to the log line's // noteHit adds hit's source, which lists the client, to the log line's
// reputation, and hit to the notes of a ban the request makes, counts the // reputation, and hit to the notes of a ban the request makes, counts the
// source in the metrics, and raises a reputation_hit alert with reason, // source in the metrics, and raises a reputation_hit alert with reason,
// whose detail gives hit's source and score. // whose detail gives hit's source and score.
func (n reputationHitNoter) noteHit( func (rq *request) noteHit(hit bans.ReputationHit, reason string) {
rq *request, hit bans.ReputationHit, reason string,
) {
detail := map[string]any{"source": hit.Source} detail := map[string]any{"source": hit.Source}
if hit.Score != nil { if hit.Score != nil {
detail["score"] = *hit.Score detail["score"] = *hit.Score
@@ -41,15 +28,15 @@ func (n reputationHitNoter) noteHit(
rq.line.Reputation = append(rq.line.Reputation, hit.Source) rq.line.Reputation = append(rq.line.Reputation, hit.Source)
rq.reputation = append(rq.reputation, hit) rq.reputation = append(rq.reputation, hit)
n.metrics.ReputationHit(hit.Source) rq.h.metrics.ReputationHit(hit.Source)
n.alerts.Raise(alerts.Alert{ rq.h.alerts.Raise(alerts.Alert{
Event: alerts.EventReputationHit, Event: alerts.EventReputationHit,
Host: rq.site, Host: rq.site,
Client: rq.client, Client: rq.client,
Netblock: clientGroup(n.ipv6GroupPrefix, rq.client), Netblock: clientGroup(rq.h.config, rq.client),
ASN: rq.lookupAnswer.ASN, ASN: rq.line.ASN,
ASName: rq.lookupAnswer.ASName, ASName: rq.line.ASName,
Country: rq.lookupAnswer.Country, Country: rq.line.Country,
Reason: reason, Reason: reason,
Detail: detail, Detail: detail,
}) })
+26 -53
View File
@@ -24,64 +24,42 @@ type refusal struct {
action string action string
} }
// decision is what the checks note of a request: its log line, what the // decision is what the checks note of a request: its log line, and what
// heuristics tell each other, and what counting the request needs once it has // counting the request needs once it has ended. waf keeps a copy of it
// ended. Nothing reads the log line back to decide. waf keeps a copy of it // until the request's report comes. It holds no credentials, since the log line
// until the request's report comes. It holds no credentials, since the log // never does, and neither the request's objects nor their bodies.
// line never does, and neither the request's objects nor their bodies.
type decision struct { type decision struct {
line requestlog.Line line requestlog.Line
client netip.Addr client netip.Addr
// method and userAgent are the request's method and User-Agent, and // requestURI is the path with the query, as a ban's notes keep it.
// requestURI its path with the query, as a ban's notes keep them. requestURI string
// startRequest sets them.
method, userAgent, requestURI string
// tags are the request's tags, sorted, each once. addTag adds them, for
// tagLists, routeWatchTags, noteLookup, dnsbl, abuseIPDB, ruleFiles and
// coreRuleSet.
tags []string
// site is, for a question, the host the visitor asked for, which the // site is, for a question, the host the visitor asked for, which the
// alerts and a ban's notes name, since one waf serves many sites. // alerts name, since one waf serves many sites.
// answerQuestion sets it.
site string site string
// allowed is true for a question answered with DispositionAllow. // allowed is true for a question answered with DispositionAllow.
allowed bool allowed bool
// lookedUp is true once the client's AS number and country have been // lookedUp is true once the client's AS number and country have been
// looked up, whether or not an answer was there, and lookupAnswer is // looked up, whether or not an answer was there, and lookupAnswer is
// what the lookup gave then, the zero Answer while GeoJS had given none. // what the lookup gave then, the zero Answer while GeoJS had given none.
// noteLookup sets them.
lookedUp bool lookedUp bool
lookupAnswer geojs.Answer lookupAnswer geojs.Answer
// skippedLookupWait is true for a request that did not wait for GeoJS and // skippedLookupWait is true for a request that did not wait for GeoJS and
// found no answer kept. SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT does not apply to it, // found no answer kept. SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT does not apply to it,
// so that not waiting never lowers its client's limits. noteLookup sets // so that not waiting never lowers its client's limits.
// it.
skippedLookupWait bool skippedLookupWait bool
// counted is true for a request the rate limits counted, whose bytes // counted is true for a request the rate limits counted, whose bytes
// the byte limits count once it has ended. rateLimitPercent and // the byte limits count once it has ended. rateLimitPercent and
// bytesLimitPercent are then its client's limit percentages for the rate // bytesLimitPercent are then its client's limit percentages for the rate
// limits and for the byte limits. rateLimits sets them. // limits and for the byte limits.
counted bool counted bool
rateLimitPercent, bytesLimitPercent percentage rateLimitPercent, bytesLimitPercent percentage
// attack is true for a request that matched a ban rule or asked for a // attack is true for a request that matched a ban rule or asked for a
// trap path, blockedByBlockRule for one a block rule refused, and // trap path, blockedByBlockRule for one a block rule refused, and
// blockedByCoreRuleSet for one the Core Rule Set refused, each an offence // blockedByCoreRuleSet for one the Core Rule Set refused, each an offence
// its client's history counts. trapPaths and ruleFiles set attack, // its client's history counts.
// ruleFiles blockedByBlockRule, and coreRuleSet blockedByCoreRuleSet.
attack, blockedByBlockRule, blockedByCoreRuleSet bool attack, blockedByBlockRule, blockedByCoreRuleSet bool
// wouldAction is, in observe mode, the action enforce mode would have
// refused the request with, one of requestlog's Action constants, and ""
// otherwise. check sets it.
wouldAction string
// brokeLimit is true for a request that broke a limit, an offence its
// client's history counts. banForLimit sets it.
brokeLimit bool
// blocklisted is true once blocklists finds a blocklist that lists the
// client, dnsblListed once dnsbl finds a DNSBL zone's verdict that does,
// and abuseIPDBHit once abuseIPDB finds AbuseIPDB's score of it a hit.
blocklisted, dnsblListed, abuseIPDBHit bool
// reputation is the reputation sources that list the client, for the // reputation is the reputation sources that list the client, for the
// notes of a ban the request makes. noteHit adds them. // notes of a ban the request makes.
reputation []bans.ReputationHit reputation []bans.ReputationHit
} }
@@ -94,6 +72,10 @@ type request struct {
in *http.Request in *http.Request
out *responseWriter out *responseWriter
// blocklisted is true once a blocklist is found to list the client,
// dnsblListed once a DNSBL zone's verdict is, and abuseIPDBHit once
// AbuseIPDB's score of it is a hit.
blocklisted, dnsblListed, abuseIPDBHit bool
start time.Time start time.Time
// checked is when the checks were done. // checked is when the checks were done.
checked time.Time checked time.Time
@@ -124,8 +106,6 @@ func (h *handler) startRequest(
rq := &request{ rq := &request{
decision: decision{ decision: decision{
client: client, client: client,
method: r.Method,
userAgent: r.UserAgent(),
requestURI: r.URL.RequestURI(), requestURI: r.URL.RequestURI(),
line: requestlog.Line{ line: requestlog.Line{
Time: requestlog.FormatTime(start), Time: requestlog.FormatTime(start),
@@ -142,10 +122,7 @@ func (h *handler) startRequest(
RequestID: requestID(r, isQuestion), RequestID: requestID(r, isQuestion),
PeerIP: peer.String(), PeerIP: peer.String(),
ForwardedFor: strings.Join(forwardedFor, ", "), ForwardedFor: strings.Join(forwardedFor, ", "),
ClientGroup: clientGroup( ClientGroup: clientGroup(h.config, client).String(),
h.config.IPv6GroupPrefix,
client,
).String(),
ContentType: r.Header.Get("Content-Type"), ContentType: r.Header.Get("Content-Type"),
RequestHeaders: requestHeaders( RequestHeaders: requestHeaders(
r, r,
@@ -171,12 +148,11 @@ func (h *handler) startRequest(
} }
// addTag adds tag to the request's tags, which stay sorted and hold each tag // addTag adds tag to the request's tags, which stay sorted and hold each tag
// once, and to its log line. // once.
func (rq *request) addTag(tag string) { func (rq *request) addTag(tag string) {
at, found := slices.BinarySearch(rq.tags, tag) at, found := slices.BinarySearch(rq.line.Tags, tag)
if !found { if !found {
rq.tags = slices.Insert(rq.tags, at, tag) rq.line.Tags = slices.Insert(rq.line.Tags, at, tag)
rq.line.Tags = rq.tags
} }
} }
@@ -217,7 +193,6 @@ func (rq *request) check(ctx context.Context) *refusal {
switch { switch {
case rq.h.config.Observe: case rq.h.config.Observe:
rq.wouldAction = action
rq.line.WouldAction = action rq.line.WouldAction = action
return nil return nil
@@ -258,7 +233,7 @@ func (rq *request) checkClient(ctx context.Context) *verdict {
now := rq.h.now() now := rq.h.now()
v := rq.runHeuristics(ctx, rq.h.beforeLookup, now) v := rq.runHeuristics(ctx, beforeLookup(), now)
if v != nil { if v != nil {
return v return v
} }
@@ -290,7 +265,7 @@ func (rq *request) runAfterLookup(
// again: a request counted at the time it arrived could be counted // again: a request counted at the time it arrived could be counted
// after a later one of its client's, which the rate limits would take // after a later one of its client's, which the rate limits would take
// for the clock set back. // for the clock set back.
return rq.runHeuristics(ctx, rq.h.afterLookup, rq.h.now()) return rq.runHeuristics(ctx, afterLookup(), rq.h.now())
} }
// pathExempt reports whether r is exempt under prefixes, // pathExempt reports whether r is exempt under prefixes,
@@ -393,14 +368,14 @@ func (rq *request) addToHistory() {
Status: rq.out.status, Status: rq.out.status,
RequestBytes: rq.requestBytes, RequestBytes: rq.requestBytes,
ResponseBytes: rq.out.bytes, ResponseBytes: rq.out.bytes,
BrokeLimit: rq.brokeLimit, BrokeLimit: rq.line.Offence == requestlog.OffenceLimit,
Attack: rq.attack, Attack: rq.attack,
BlockedByBlockRule: rq.blockedByBlockRule, BlockedByBlockRule: rq.blockedByBlockRule,
BlockedByCoreRuleSet: rq.blockedByCoreRuleSet, BlockedByCoreRuleSet: rq.blockedByCoreRuleSet,
} }
rq.h.limiter.AddToHistory( rq.h.limiter.AddToHistory(
clientGroup(rq.h.config.IPv6GroupPrefix, rq.client), clientGroup(rq.h.config, rq.client),
rq.h.now(), rq.h.now(),
request, request,
) )
@@ -429,11 +404,11 @@ func (rq *request) countAnomalies() {
rq.h.anomalies.Count(rq.h.now(), anomaly.Request{ rq.h.anomalies.Count(rq.h.now(), anomaly.Request{
Host: rq.site, Host: rq.site,
Client: rq.client, Client: rq.client,
ClientGroup: clientGroup(rq.h.config.IPv6GroupPrefix, rq.client), ClientGroup: clientGroup(rq.h.config, rq.client),
ASN: answer.ASN, ASN: answer.ASN,
ASName: answer.ASName, ASName: answer.ASName,
Country: answer.Country, Country: answer.Country,
Bytes: rq.countedBytes(rq.h.config.BytesCounted), Bytes: rq.countedBytes(),
}) })
} }
@@ -460,9 +435,7 @@ func (rq *request) answerAtTheEnd() (geojs.Answer, bool) {
return rq.lookupAnswer, true return rq.lookupAnswer, true
} }
return rq.h.geojs.Kept( return rq.h.geojs.Kept(clientGroup(rq.h.config, rq.client))
clientGroup(rq.h.config.IPv6GroupPrefix, rq.client),
)
} }
// responseWriter is waf's answer to a request. It notes the status and // responseWriter is waf's answer to a request. It notes the status and
+2 -2
View File
@@ -13,7 +13,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/waf"
) )
const ( const (
@@ -248,7 +248,7 @@ func TestTimingsAreInOrder(t *testing.T) {
wantStatus(t, do(t, req), http.StatusForbidden) wantStatus(t, do(t, req), http.StatusForbidden)
refused := out.endedLines(t, 2)[1] refused := out.endedLines(t, 2)[1]
wantStatus(t, get(t, server.Addr, wafprotocol.HealthPath), http.StatusOK) wantStatus(t, get(t, server.Addr, waf.HealthPath), http.StatusOK)
health := out.endedLines(t, 3)[2] health := out.endedLines(t, 3)[2]
// A request passed to the app has every timing; one refused, none of // A request passed to the app has every timing; one refused, none of
-142
View File
@@ -1,142 +0,0 @@
package waf
import (
"crypto/subtle"
"io"
"net/http"
"strings"
"github.com/go-chi/chi/v5"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// routes returns the router of waf's own endpoints. A question never reaches
// it, since ServeHTTP answers a question first. No check runs on a request
// here: waf's address is on the private network, where the peers are rproxy
// and the admin's tools, not visitors.
func (h *handler) routes() http.Handler {
r := chi.NewRouter()
r.Use(matchDecodedPath)
// The report writes no log line of its own: it finishes the line of the
// request it reports.
r.Post(wafprotocol.ReportPath, h.handleReport())
r.Group(func(r chi.Router) {
r.Use(h.logOwnEndpoint)
// The answer to a path with no endpoint, and to a method the path has
// no endpoint for, is set in this group, so that it is logged too.
r.NotFound(answerNotFound)
r.MethodNotAllowed(answerNotFound)
r.Get(wafprotocol.HealthPath, handleHealth())
r.Group(func(r chi.Router) {
r.Use(requireToken(h.config.MetricsToken))
r.Get(MetricsPath, h.metrics.ServeHTTP)
})
// The handlers under BansPath/ and ClientsPath read the address from
// the path themselves, so that one that is not an address gets 400
// after the token check.
r.Group(func(r chi.Router) {
r.Use(requireToken(h.config.AdminToken))
r.Get(BansPath, h.handleListBans())
r.Post(BansPath, h.handleAddBan())
r.Delete(BansPath+"/*", h.handleLiftBans())
r.Get(ClientsPath+"*", h.handleShowClient())
})
})
return r
}
// matchDecodedPath has the router match the decoded path, which the handlers
// read. chi matches the path as sent when it is encoded in an unusual way,
// which would make /_sww/%68ealthz a path with no endpoint.
func matchDecodedPath(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
chi.RouteContext(r.Context()).RoutePath = r.URL.Path
next.ServeHTTP(w, r)
})
}
// logOwnEndpoint writes one log line for each request, with the action
// own_endpoint. The line's request_bytes is how much of the body the handler
// read.
func (h *handler) logOwnEndpoint(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
peer := peerAddress(r)
rq := h.startRequest(w, r, peer, peer, false)
defer rq.finish()
rq.line.Action = requestlog.ActionOwnEndpoint
counted := r.Clone(r.Context())
counted.Body = &countedBody{
ReadCloser: r.Body,
read: &rq.requestBytes,
}
next.ServeHTTP(rq.out, counted)
})
}
// countedBody is a request's body that adds the bytes read of it to read.
type countedBody struct {
io.ReadCloser
read *int64
}
// Read reads part of the body.
func (b *countedBody) Read(p []byte) (int, error) {
n, err := b.ReadCloser.Read(p)
*b.read += int64(n)
return n, err
}
// requireToken returns middleware that lets a request through only when it
// carries token, as Authorization: Bearer <token>. It answers 401 to a request
// without it. While token is unset, it answers 404, as a path with no endpoint
// does.
func requireToken(token string) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
switch {
case token == "":
answerNotFound(w, r)
case !hasToken(r, token):
w.Header().Set("WWW-Authenticate", "Bearer")
http.Error(
w,
http.StatusText(http.StatusUnauthorized),
http.StatusUnauthorized,
)
default:
next.ServeHTTP(w, r)
}
},
)
}
}
// hasToken reports whether r carries token, as Authorization: Bearer
// <token>.
func hasToken(r *http.Request, token string) bool {
scheme, sent, _ := strings.Cut(r.Header.Get("Authorization"), " ")
return strings.EqualFold(scheme, "Bearer") &&
subtle.ConstantTimeCompare([]byte(sent), []byte(token)) == 1
}
// answerNotFound answers 404, for a path with no endpoint, and for a method
// the path has no endpoint for.
func answerNotFound(w http.ResponseWriter, _ *http.Request) {
http.Error(w, http.StatusText(http.StatusNotFound), http.StatusNotFound)
}
+5 -10
View File
@@ -8,7 +8,6 @@ import (
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/routewatch"
) )
// routeWatchTags adds to the request the tags asn=<number> and prefix=<netblock> // routeWatchTags adds to the request the tags asn=<number> and prefix=<netblock>
@@ -18,26 +17,22 @@ import (
// SWW_WAF_ROUTEWATCH_URL is unset it does nothing. Answers are kept by client and // SWW_WAF_ROUTEWATCH_URL is unset it does nothing. Answers are kept by client and
// asked for by the request's address, and a route without an origin AS gives only // asked for by the request's address, and a route without an origin AS gives only
// the prefix= tag. // the prefix= tag.
type routeWatchTags struct { type routeWatchTags struct{}
// routeWatch is nil while SWW_WAF_ROUTEWATCH_URL is unset.
routeWatch *routewatch.Client
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
}
func (routeWatchTags) name() string { return "routewatch_tags" } func (routeWatchTags) name() string { return "routewatch_tags" }
func (r routeWatchTags) check( func (routeWatchTags) check(
ctx context.Context, ctx context.Context,
rq *request, rq *request,
_ time.Time, _ time.Time,
) *verdict { ) *verdict {
if r.routeWatch == nil { if rq.h.routeWatch == nil {
return nil return nil
} }
route, missing := r.routeWatch.Route( route, missing := rq.h.routeWatch.Route(
ctx, ctx,
clientGroup(r.ipv6GroupPrefix, rq.client), clientGroup(rq.h.config, rq.client),
rq.client, rq.client,
) )
if missing != "" { if missing != "" {
+4 -10
View File
@@ -9,7 +9,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules" "sneak.berlin/go/smallwebwaf/internal/rules"
) )
@@ -23,15 +22,11 @@ import (
// mode raises the alert for the ban // mode raises the alert for the ban
// it would have made. Either rule's match is noted as an offence, for the // it would have made. Either rule's match is noted as an offence, for the
// client's history. Its figures are the rule's id and action. // client's history. Its figures are the rule's id and action.
type ruleFiles struct { type ruleFiles struct{}
rules *rules.Files
metrics *metrics.Metrics
banMaker banMaker
}
func (ruleFiles) name() string { return "rule_files" } func (ruleFiles) name() string { return "rule_files" }
func (r ruleFiles) check( func (ruleFiles) check(
_ context.Context, _ context.Context,
rq *request, rq *request,
now time.Time, now time.Time,
@@ -40,7 +35,7 @@ func (r ruleFiles) check(
for _, rule := range matched { for _, rule := range matched {
rq.line.RuleIDs = append(rq.line.RuleIDs, rule.ID) rq.line.RuleIDs = append(rq.line.RuleIDs, rule.ID)
r.metrics.RuleMatched(rule.ID, rule.Action) rq.h.metrics.RuleMatched(rule.ID, rule.Action)
} }
if len(matched) == 0 { if len(matched) == 0 {
@@ -65,8 +60,7 @@ func (r ruleFiles) check(
rq.attack = true rq.attack = true
return &verdict{ return &verdict{
action: r.banMaker.banForAttack( action: rq.banForAttack(
rq,
now, now,
bans.Notes{RuleID: last.ID, Target: last.Target}, bans.Notes{RuleID: last.ID, Target: last.Target},
), ),
+4 -10
View File
@@ -12,19 +12,13 @@ import (
// tagDeny refuses a request that carries a tag in SWW_WAF_TAG_DENY. Its figure is // tagDeny refuses a request that carries a tag in SWW_WAF_TAG_DENY. Its figure is
// the first such tag, in sorted order. // the first such tag, in sorted order.
type tagDeny struct { type tagDeny struct{}
tags []string // SWW_WAF_TAG_DENY
}
func (tagDeny) name() string { return "tag_deny" } func (tagDeny) name() string { return "tag_deny" }
func (t tagDeny) check( func (tagDeny) check(_ context.Context, rq *request, _ time.Time) *verdict {
_ context.Context, for _, tag := range rq.line.Tags {
rq *request, if slices.Contains(rq.h.config.TagDeny, tag) {
_ time.Time,
) *verdict {
for _, tag := range rq.tags {
if slices.Contains(t.tags, tag) {
return &verdict{ return &verdict{
action: requestlog.ActionClientDenied, action: requestlog.ActionClientDenied,
figures: map[string]any{"tag": tag}, figures: map[string]any{"tag": tag},
+3 -11
View File
@@ -6,25 +6,17 @@ package waf
import ( import (
"context" "context"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/reputation"
) )
// tagLists adds to the request one tag for each list of SWW_WAF_TAG_LISTS that // tagLists adds to the request one tag for each list of SWW_WAF_TAG_LISTS that
// holds the client, the tag being the list's name. It refuses nothing, and runs // holds the client, the tag being the list's name. It refuses nothing, and runs
// first so that every refusal's log line carries these tags. // first so that every refusal's log line carries these tags.
type tagLists struct { type tagLists struct{}
lists *reputation.Lists
}
func (tagLists) name() string { return "tag_lists" } func (tagLists) name() string { return "tag_lists" }
func (t tagLists) check( func (tagLists) check(_ context.Context, rq *request, _ time.Time) *verdict {
_ context.Context, for _, name := range rq.h.lists.TagsOf(rq.client) {
rq *request,
_ time.Time,
) *verdict {
for _, name := range t.lists.TagsOf(rq.client) {
rq.addTag(name) rq.addTag(name)
} }
+8 -15
View File
@@ -16,14 +16,11 @@ import (
// it bans the client's netblock, or in observe mode raises the alert for the // it bans the client's netblock, or in observe mode raises the alert for the
// ban it would have made. The ban's notes give the path as sent, and so does // ban it would have made. The ban's notes give the path as sent, and so does
// its figure. // its figure.
type trapPaths struct { type trapPaths struct{}
paths []string // SWW_WAF_TRAP_PATHS
banMaker banMaker
}
func (trapPaths) name() string { return "trap_paths" } func (trapPaths) name() string { return "trap_paths" }
func (t trapPaths) check( func (trapPaths) check(
_ context.Context, _ context.Context,
rq *request, rq *request,
now time.Time, now time.Time,
@@ -36,22 +33,18 @@ func (t trapPaths) check(
rq.attack = true rq.attack = true
return &verdict{ return &verdict{
action: t.banMaker.banForAttack( action: rq.banForAttack(now, bans.Notes{TrapPath: path}),
rq,
now,
bans.Notes{TrapPath: path},
),
figures: map[string]any{"path": path}, figures: map[string]any{"path": path},
} }
} }
// trapPath returns rq's path as sent, when its path as a path rule sees it, // trapPath returns the request's path as sent, when its path as a path rule
// without the query, as sent or once percent-decoded, is in // sees it, without the query, as sent or once percent-decoded, is in
// SWW_WAF_TRAP_PATHS, and "" otherwise. // SWW_WAF_TRAP_PATHS, and "" otherwise.
func (t trapPaths) trapPath(rq *request) string { func (rq *request) trapPath() string {
path := rules.Path(rq.in) path := rules.Path(rq.in)
if !slices.Contains(t.paths, path) && if !slices.Contains(rq.h.config.TrapPaths, path) &&
!slices.Contains(t.paths, rules.DecodeOnce(path)) { !slices.Contains(rq.h.config.TrapPaths, rules.DecodeOnce(path)) {
return "" return ""
} }
+266 -79
View File
@@ -1,21 +1,75 @@
package waf package waf
import ( import (
"bytes"
"context" "context"
"encoding/json"
"errors" "errors"
"fmt"
"io"
"log/slog" "log/slog"
"net/http" "net/http"
"net/netip"
"strconv"
"strings" "strings"
"time" "time"
"unicode/utf8"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// The headers of rproxy's question and of waf's answer, as
// "waf" in SPEC.md gives them.
const (
// ClientHeader marks a request as rproxy's question, and gives the
// visitor's address.
ClientHeader = "X-Sww-Client"
// PeerHeader gives the address of rproxy's own TCP peer.
PeerHeader = "X-Sww-Peer"
// ContentLengthHeader gives the body length the visitor announced.
ContentLengthHeader = "X-Sww-Content-Length"
// DispositionHeader gives waf's answer.
DispositionHeader = "X-Sww-Disposition"
)
// The values of DispositionHeader.
const (
// DispositionAllow tells rproxy to pass the request to the app.
DispositionAllow = "allow"
// DispositionDeny tells rproxy to give the visitor the answer's status,
// headers and body.
DispositionDeny = "deny"
// DispositionClose tells rproxy to close the visitor's connection
// without an answer, which SWW_WAF_BAN_RESPONSE asks for when it is close.
// waf closing the question's connection instead would read as
// waf being down, and rproxy would then let the request through.
DispositionClose = "close"
)
// ReportPath is where rproxy sends the report of each request waf
// answered.
const ReportPath = "/_sww/report"
// QuestionBodyMaxBytes is the most of a visitor's body a question carries.
const QuestionBodyMaxBytes = 64 << 10
const ( const (
// maxAwaitingReport is the most answered requests held until their // maxAwaitingReport is the most answered requests held until their
// reports come. Past it, the request answered longest ago is dropped. // reports come. Past it, the request answered longest ago is dropped.
maxAwaitingReport = 10000 maxAwaitingReport = 10000
// maxRequestIDBytes is the longest id a question may give.
maxRequestIDBytes = 128
// reportBodyMaxBytes is the most of a report's body that is read. It is
// far above any report's.
reportBodyMaxBytes = 1 << 20
// maxHTTPStatus is the highest status a report may give.
maxHTTPStatus = 599
// maxReportedBytes is the most bytes a report may give for each way, 1
// PiB, so that adding them up cannot overflow.
maxReportedBytes = 1 << 50
// maxReportedMilliseconds is the longest duration a report may give,
// about 31 years, so that it fits a time.Duration.
maxReportedMilliseconds = 1e12
// wafTimeout bounds how long a request to waf takes to // wafTimeout bounds how long a request to waf takes to
// arrive, and its answer to leave, and wafIdleTimeout how long a // arrive, and its answer to leave, and wafIdleTimeout how long a
// kept-open connection waits for its next request. // kept-open connection waits for its next request.
@@ -23,6 +77,26 @@ const (
wafIdleTimeout = 120 * time.Second wafIdleTimeout = 120 * time.Second
) )
var (
errReportFieldMissing = errors.New("a field is missing")
errReportValue = errors.New("a value is out of range")
)
// Report is the JSON object rproxy posts to ReportPath once the visitor's
// answer has ended, its fields named as the request log's of the same
// meaning. RequestID is the question's id. Status is what the visitor got, 0
// if nothing. The durations are in milliseconds.
//
//nolint:tagliatelle // docs/request-log.md names the fields in snake_case
type Report struct {
RequestID string `json:"request_id"`
Status int `json:"status"`
RequestBytes int64 `json:"request_bytes"`
ResponseBytes int64 `json:"response_bytes"`
DurationTotalMS float64 `json:"duration_total_ms"`
DurationUpstreamTotalMS float64 `json:"duration_upstream_total_ms"`
}
// New returns the server `sww waf` runs. It answers // New returns the server `sww waf` runs. It answers
// rproxy's questions, takes rproxy's reports and answers its own endpoints. It // rproxy's questions, takes rproxy's reports and answers its own endpoints. It
// passes nothing to an app. rproxy applies the size and time limits to the // passes nothing to an app. rproxy applies the size and time limits to the
@@ -49,61 +123,58 @@ func New(params Params) *Server {
GeoJS: h.geojs, GeoJS: h.geojs,
Anomalies: h.anomalies, Anomalies: h.anomalies,
LookupFile: h.lookupFile, LookupFile: h.lookupFile,
Lists: params.Lists, Lists: h.lists,
DNSBL: params.DNSBL, DNSBL: h.dnsbl,
AbuseIPDB: params.AbuseIPDB, AbuseIPDB: h.abuseIPDB,
RouteWatch: params.RouteWatch, RouteWatch: h.routeWatch,
Metrics: h.metrics, Metrics: h.metrics,
} }
} }
// ServeHTTP handles one request to waf: a request carrying // ServeHTTP handles one request to waf: a request carrying
// wafprotocol.ClientHeader is rproxy's question, and any other is for one of // ClientHeader is rproxy's question, and any other is for waf
// waf's own endpoints. // itself.
func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) { func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
if len(r.Header.Values(wafprotocol.ClientHeader)) > 0 { if len(r.Header.Values(ClientHeader)) > 0 {
h.answerQuestion(w, r) h.answerQuestion(w, r)
return return
} }
h.ownEndpoints.ServeHTTP(w, r) h.answerOwnEndpoint(w, r)
} }
// refusedWhateverTheAnswer reports whether the request is one that does not wait // refusedWhateverTheAnswer reports whether the request is one that does not wait
// for GeoJS, since the checks will refuse it at now whatever GeoJS answers about // for GeoJS, since the checks will refuse it at now whatever GeoJS answers about
// its client: a heuristic of afterLookup that implements refusesWithoutLookup // its client: the heuristic blocklists or rate_limits will refuse it, as a
// says it will refuse it. In observe mode, which refuses nothing, none is. // function in that heuristic's file says. In observe mode, which refuses
// rproxy lets a request through when waf answers too late. // nothing, none is. rproxy lets a request through when waf answers too late.
func (rq *request) refusedWhateverTheAnswer(now time.Time) bool { func (rq *request) refusedWhateverTheAnswer(now time.Time) bool {
if rq.h.config.Observe { if rq.h.config.Observe {
return false return false
} }
for _, each := range rq.h.afterLookup { return rq.refusedByBlocklists(rq.h.lists.ListedBy(rq.client)) ||
knows, ok := each.(refusesWithoutLookup) rq.overAWholeRateLimit(now)
if ok && knows.refusesWhateverTheLookupAnswers(rq, now) {
return true
}
}
return false
} }
// answerQuestion runs the checks on the visitor's request a question // answerQuestion runs the checks on the visitor's request a question
// carries, and answers rproxy what to do with it. A denied request's counting // carries, and answers rproxy what to do with it. A denied request's counting
// is finished here, and an allowed one's by its report. // is finished here, and an allowed one's by its report.
func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) { func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
question, ok := h.readQuestion(w, r) client, body, ok := h.readQuestion(w, r)
if !ok { if !ok {
return return
} }
// One that cannot be read leaves the log line's peer_ip invalid.
peer, _ := netip.ParseAddr(r.Header.Get(PeerHeader))
rq := h.startRequest( rq := h.startRequest(
w, w,
question.Visitor, asTheVisitorSentIt(r, body),
question.Client, client.Unmap(),
question.Peer, peer.Unmap(),
true, true,
) )
// These run even when a check panics. finish runs first, so that what is // These run even when a check panics. finish runs first, so that what is
@@ -118,15 +189,12 @@ func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
rq.line.WAFLine = requestlog.WAFLineDecision rq.line.WAFLine = requestlog.WAFLineDecision
refused := rq.check(r.Context()) refused := rq.check(r.Context())
rq.checked = time.Now() rq.checked = time.Now()
underOwnPrefix := strings.HasPrefix( underOwnPrefix := strings.HasPrefix(r.URL.Path, ownEndpointPrefix)
r.URL.Path,
wafprotocol.OwnEndpointPrefix,
)
if refused == nil && !underOwnPrefix { if refused == nil && !underOwnPrefix {
rq.allow() rq.allow()
// In observe mode, what enforce mode would have refused counts here. // In observe mode, what enforce mode would have refused counts here.
rq.runBanningHeuristics(r.Context(), rq.h.afterAnswer) rq.runBanningHeuristics(r.Context(), afterAnswer())
return return
} }
@@ -137,10 +205,7 @@ func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
// The app never sees this prefix. waf's own endpoints are at its // The app never sees this prefix. waf's own endpoints are at its
// own address, so this answers as a path with no endpoint does. // own address, so this answers as a path with no endpoint does.
rq.line.Action = requestlog.ActionOwnEndpoint rq.line.Action = requestlog.ActionOwnEndpoint
wafprotocol.SetAnswerHeaders( rq.out.Header().Set(DispositionHeader, DispositionDeny)
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionDeny},
)
http.Error( http.Error(
rq.out, rq.out,
http.StatusText(http.StatusNotFound), http.StatusText(http.StatusNotFound),
@@ -148,22 +213,48 @@ func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
) )
} }
rq.banAndCount(r.Context(), rq.h.afterAnswer) rq.banAndCount(r.Context(), afterAnswer())
} }
// readQuestion reads a question, and reports whether waf takes it. It answers // readQuestion returns the visitor's address and the body a question carries,
// a question it does not take with 400 and no disposition. rproxy passes that // and whether waf takes the question. It answers a question it does not
// 400 to the visitor, and SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION does // take with 400 and no disposition. rproxy passes that 400 to the visitor,
// not apply. The id of a question waf takes is held from here on, without a // and SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION does not apply. The id of
// decision yet, so that another question with that id is refused while this // a question waf takes is held from here on, without a decision yet, so
// one is being handled. // that another question with that id is refused while this one is being
// handled.
// //
// A question whose body stops arriving gets no answer, and is not looked at. // The body is read before any check, since Go's server does not send an answer
// until it has read the body. A question whose body stops arriving gets no
// answer, and is not looked at.
func (h *handler) readQuestion( func (h *handler) readQuestion(
w http.ResponseWriter, r *http.Request, w http.ResponseWriter, r *http.Request,
) (wafprotocol.Question, bool) { ) (netip.Addr, []byte, bool) {
question, err := wafprotocol.ReadQuestion(w, r) client, err := netip.ParseAddr(r.Header.Get(ClientHeader))
if errors.Is(err, wafprotocol.ErrBodyStoppedArriving) { if err != nil || client.Zone() != "" {
h.refuseQuestion(w, ClientHeader+" is not an address without a zone")
return netip.Addr{}, nil, false
}
id := r.Header.Get(requestIDHeader)
if id == "" || len(id) > maxRequestIDBytes || !utf8.ValidString(id) {
h.refuseQuestion(w, requestIDHeader+" is not a valid id")
return netip.Addr{}, nil, false
}
body, err := io.ReadAll(
http.MaxBytesReader(w, r.Body, QuestionBodyMaxBytes),
)
if _, tooLarge := errors.AsType[*http.MaxBytesError](err); tooLarge {
h.refuseQuestion(w, "the body is over 64 KiB")
return netip.Addr{}, nil, false
}
if err != nil {
h.processLog.Warn( h.processLog.Warn(
"a question's body stopped arriving", "a question's body stopped arriving",
"error", "error",
@@ -174,23 +265,17 @@ func (h *handler) readQuestion(
panic(http.ErrAbortHandler) panic(http.ErrAbortHandler)
} }
if err != nil { held, _ := h.awaitingReport.ContainsOrAdd(id, nil)
h.refuseQuestion(w, err.Error())
return question, false
}
held, _ := h.awaitingReport.ContainsOrAdd(question.RequestID, nil)
if held { if held {
h.refuseQuestion( h.refuseQuestion(
w, w,
wafprotocol.RequestIDHeader+" is the id of a request waf holds", requestIDHeader+" is the id of a request waf holds",
) )
return question, false return netip.Addr{}, nil, false
} }
return question, true return client, body, true
} }
// refuseQuestion answers a question waf does not take, for reason. // refuseQuestion answers a question waf does not take, for reason.
@@ -199,18 +284,48 @@ func (h *handler) refuseQuestion(w http.ResponseWriter, reason string) {
http.Error(w, reason, http.StatusBadRequest) http.Error(w, reason, http.StatusBadRequest)
} }
// allow answers a question with wafprotocol.DispositionAllow and, while // asTheVisitorSentIt returns a copy of r, a question, as the checks and the
// log line are to see it: without the headers rproxy adds for waf,
// with body, which has been read of r, and with the length the visitor
// announced, as -1 when it announced none. The question's own length covers
// only the part of the body it carries.
func asTheVisitorSentIt(r *http.Request, body []byte) *http.Request {
sent := r.Clone(r.Context())
announced := sent.Header.Get(ContentLengthHeader)
sent.Header.Del(ClientHeader)
sent.Header.Del(PeerHeader)
sent.Header.Del(ContentLengthHeader)
sent.Header.Del("Content-Length")
sent.ContentLength = -1
length, err := strconv.ParseInt(announced, 10, 64)
if err == nil && length >= 0 {
sent.ContentLength = length
sent.Header.Set("Content-Length", announced)
}
sent.Body = http.NoBody
if len(body) > 0 {
sent.Body = io.NopCloser(bytes.NewReader(body))
}
return sent
}
// allow answers a question with DispositionAllow and, while
// SWW_WAF_ADD_LOOKUP_HEADERS is true, the client's AS number and country. // SWW_WAF_ADD_LOOKUP_HEADERS is true, the client's AS number and country.
func (rq *request) allow() { func (rq *request) allow() {
rq.allowed = true rq.allowed = true
answer := wafprotocol.Answer{Disposition: wafprotocol.DispositionAllow} header := rq.out.Header()
header.Set(DispositionHeader, DispositionAllow)
if rq.h.config.AddLookupHeaders { if rq.h.config.AddLookupHeaders {
answer.ASN = rq.lookupAnswer.ASN setLookupHeaders(header, rq.line.ASN, rq.line.Country)
answer.Country = rq.lookupAnswer.Country
} }
wafprotocol.SetAnswerHeaders(rq.out.Header(), answer)
rq.out.WriteHeader(http.StatusOK) rq.out.WriteHeader(http.StatusOK)
} }
@@ -228,28 +343,45 @@ func (rq *request) deny(r refusal) {
if r.status == 0 { if r.status == 0 {
// SWW_WAF_BAN_RESPONSE is close. Written past rq.out, since the log line's // SWW_WAF_BAN_RESPONSE is close. Written past rq.out, since the log line's
// status stays 0: the visitor is sent nothing. // status stays 0: the visitor is sent nothing.
wafprotocol.SetAnswerHeaders( rq.out.Header().Set(DispositionHeader, DispositionClose)
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionClose},
)
rq.out.ResponseWriter.WriteHeader(http.StatusOK) rq.out.ResponseWriter.WriteHeader(http.StatusOK)
return return
} }
wafprotocol.SetAnswerHeaders( rq.out.Header().Set(DispositionHeader, DispositionDeny)
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionDeny},
)
http.Error(rq.out, http.StatusText(r.status), r.status) http.Error(rq.out, http.StatusText(r.status), r.status)
} }
// handleReport answers POST wafprotocol.ReportPath: it finishes the request // answerOwnEndpoint answers a request to waf's own address. No check
// the report names, if it holds that request's decision, and ignores the // runs on it: the address is on the private network, where the peers are
// report otherwise. // rproxy and the admin's tools, not visitors.
func (h *handler) handleReport() http.HandlerFunc { func (h *handler) answerOwnEndpoint(w http.ResponseWriter, r *http.Request) {
return func(w http.ResponseWriter, r *http.Request) { if r.Method == http.MethodPost && r.URL.Path == ReportPath {
report, err := wafprotocol.ReadReport(w, r) h.takeReport(w, r)
return
}
peer := peerAddress(r)
rq := h.startRequest(w, r, peer, peer, false)
defer rq.finish()
if r.Method == http.MethodGet && r.URL.Path == HealthPath {
rq.answerHealth()
return
}
rq.answerAdmin()
}
// takeReport answers POST ReportPath: it finishes the request the report
// names, if it holds that request's decision, and ignores the report
// otherwise.
func (h *handler) takeReport(w http.ResponseWriter, r *http.Request) {
report, err := readReport(w, r)
if err != nil { if err != nil {
h.processLog.Warn("a report is refused", "error", err.Error()) h.processLog.Warn("a report is refused", "error", err.Error())
http.Error(w, "the body is not a report", http.StatusBadRequest) http.Error(w, "the body is not a report", http.StatusBadRequest)
@@ -258,15 +390,70 @@ func (h *handler) handleReport() http.HandlerFunc {
} }
held, found := h.awaitingReport.Peek(report.RequestID) held, found := h.awaitingReport.Peek(report.RequestID)
// Remove reports true to one report only, of several that name the // Remove reports true to one report only, of several that name the id. A
// id. A question still being handled is held as nil. // question still being handled is held as nil.
if found && held != nil && if found && held != nil && h.awaitingReport.Remove(report.RequestID) {
h.awaitingReport.Remove(report.RequestID) {
h.finishWith(r.Context(), held, report) h.finishWith(r.Context(), held, report)
} }
w.WriteHeader(http.StatusNoContent) w.WriteHeader(http.StatusNoContent)
} }
// readReport reads a report from r's body, which must be one JSON object with
// all of Report's fields, in the ranges SPEC.md gives.
func readReport(w http.ResponseWriter, r *http.Request) (Report, error) {
var report Report
body, err := io.ReadAll(
http.MaxBytesReader(w, r.Body, reportBodyMaxBytes),
)
if err != nil {
return report, fmt.Errorf("read the body: %w", err)
}
var fields map[string]json.RawMessage
err = json.Unmarshal(body, &fields)
if err != nil {
return report, fmt.Errorf("decode the body: %w", err)
}
for _, name := range []string{
"request_id", "status", "request_bytes", "response_bytes",
"duration_total_ms", "duration_upstream_total_ms",
} {
value, there := fields[name]
if !there || string(value) == "null" {
return report, fmt.Errorf("%w: %s", errReportFieldMissing, name)
}
}
err = json.Unmarshal(body, &report)
if err != nil {
return report, fmt.Errorf("decode the body: %w", err)
}
return report, report.checkRanges()
}
// checkRanges returns an error for a value of the report that cannot be true.
func (report *Report) checkRanges() error {
switch {
case report.Status != 0 &&
(report.Status < http.StatusContinue || report.Status > maxHTTPStatus):
return fmt.Errorf("%w: status", errReportValue)
case report.RequestBytes < 0 || report.RequestBytes > maxReportedBytes:
return fmt.Errorf("%w: request_bytes", errReportValue)
case report.ResponseBytes < 0 || report.ResponseBytes > maxReportedBytes:
return fmt.Errorf("%w: response_bytes", errReportValue)
case report.DurationTotalMS < 0 || report.DurationTotalMS > maxReportedMilliseconds:
return fmt.Errorf("%w: duration_total_ms", errReportValue)
case report.DurationUpstreamTotalMS < 0 ||
report.DurationUpstreamTotalMS > maxReportedMilliseconds:
return fmt.Errorf("%w: duration_upstream_total_ms", errReportValue)
}
return nil
} }
// finishWith finishes a request with what its report says, and writes the // finishWith finishes a request with what its report says, and writes the
@@ -278,7 +465,7 @@ func (h *handler) handleReport() http.HandlerFunc {
func (h *handler) finishWith( func (h *handler) finishWith(
ctx context.Context, ctx context.Context,
held *decision, held *decision,
report wafprotocol.Report, report Report,
) { ) {
rq := &request{ rq := &request{
decision: *held, decision: *held,
@@ -299,7 +486,7 @@ func (h *handler) finishWith(
if rq.allowed { if rq.allowed {
rq.line.DurationUpstreamTotalMS = &report.DurationUpstreamTotalMS rq.line.DurationUpstreamTotalMS = &report.DurationUpstreamTotalMS
rq.banAndCount(ctx, rq.h.onReport) rq.banAndCount(ctx, onReport())
rq.h.metrics.RequestReported( rq.h.metrics.RequestReported(
&rq.line, &rq.line,
+86 -195
View File
@@ -22,7 +22,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
const ( const (
@@ -107,8 +106,8 @@ func (s *WAF) questionWith(
req := newRequest(s.t, method, s.addr, path, body) req := newRequest(s.t, method, s.addr, path, body)
req.Host = host req.Host = host
req.Header.Set(wafprotocol.ClientHeader, from) req.Header.Set(waf.ClientHeader, from)
req.Header.Set(wafprotocol.PeerHeader, rproxyPeer) req.Header.Set(waf.PeerHeader, rproxyPeer)
req.Header.Set(requestIDHeader, rand.Text()) req.Header.Set(requestIDHeader, rand.Text())
return req return req
@@ -135,18 +134,13 @@ func (s *WAF) allowed(from string) string {
s.t.Helper() s.t.Helper()
req := s.question(siteA, from, "/") req := s.question(siteA, from, "/")
wantDisposition( wantDisposition(s.t, do(s.t, req), http.StatusOK, waf.DispositionAllow)
s.t,
do(s.t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
)
return req.Header.Get(requestIDHeader) return req.Header.Get(requestIDHeader)
} }
// report sends rproxy's report and returns the answer. // report sends rproxy's report and returns the answer.
func (s *WAF) report(report wafprotocol.Report) answer { func (s *WAF) report(report waf.Report) answer {
s.t.Helper() s.t.Helper()
body, err := json.Marshal(report) body, err := json.Marshal(report)
@@ -161,11 +155,8 @@ func (s *WAF) report(report wafprotocol.Report) answer {
func (s *WAF) reportBody(body string) answer { func (s *WAF) reportBody(body string) answer {
s.t.Helper() s.t.Helper()
return do( return do(s.t, newRequest(s.t, http.MethodPost, s.addr, waf.ReportPath,
s.t, strings.NewReader(body)))
newRequest(s.t, http.MethodPost, s.addr, wafprotocol.ReportPath,
strings.NewReader(body)),
)
} }
// passedOn asks about a request from the visitor at from on siteA, and reports // passedOn asks about a request from the visitor at from on siteA, and reports
@@ -173,7 +164,7 @@ func (s *WAF) reportBody(body string) answer {
func (s *WAF) passedOn(from string) { func (s *WAF) passedOn(from string) {
s.t.Helper() s.t.Helper()
got := s.report(wafprotocol.Report{ got := s.report(waf.Report{
RequestID: s.allowed(from), RequestID: s.allowed(from),
Status: http.StatusOK, Status: http.StatusOK,
RequestBytes: bodyBytes, RequestBytes: bodyBytes,
@@ -219,7 +210,7 @@ func wantDisposition(
t.Helper() t.Helper()
wantStatus(t, got, status) wantStatus(t, got, status)
if answered := got.header.Get(wafprotocol.DispositionHeader); answered != if answered := got.header.Get(waf.DispositionHeader); answered !=
disposition { disposition {
t.Errorf( t.Errorf(
"answered with disposition %q, want %q", "answered with disposition %q, want %q",
@@ -281,10 +272,10 @@ func TestQuestionAllowedIsAnsweredAndLogged(t *testing.T) {
req := s.question(siteA, client, "/page?q=1&token=visible") req := s.question(siteA, client, "/page?q=1&token=visible")
req.Header.Set(requestIDHeader, "the-id") req.Header.Set(requestIDHeader, "the-id")
req.Header.Set("X-Forwarded-Proto", "https") req.Header.Set("X-Forwarded-Proto", "https")
req.Header.Set(wafprotocol.ContentLengthHeader, "1234") req.Header.Set(waf.ContentLengthHeader, "1234")
got := do(t, req) got := do(t, req)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow) wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
wantBody(t, got, "") wantBody(t, got, "")
if id := got.header.Get(requestIDHeader); id != "" { if id := got.header.Get(requestIDHeader); id != "" {
@@ -317,12 +308,7 @@ func TestIPv6VisitorIsCountedInItsGroup(t *testing.T) {
s.allowed("2001:db8::1") s.allowed("2001:db8::1")
got := s.ask("2001:db8::2", "/") got := s.ask("2001:db8::2", "/")
wantDisposition( wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
} }
func TestIPv4VisitorInIPv6FormIsTheIPv4Client(t *testing.T) { func TestIPv4VisitorInIPv6FormIsTheIPv4Client(t *testing.T) {
@@ -331,14 +317,9 @@ func TestIPv4VisitorInIPv6FormIsTheIPv4Client(t *testing.T) {
s := startWAF(t, "", map[string]string{denyNets: client}) s := startWAF(t, "", map[string]string{denyNets: client})
req := s.question(siteA, "::ffff:"+client, "/") req := s.question(siteA, "::ffff:"+client, "/")
req.Header.Set(wafprotocol.PeerHeader, "::ffff:"+rproxyPeer) req.Header.Set(waf.PeerHeader, "::ffff:"+rproxyPeer)
wantDisposition( wantDisposition(t, do(t, req), http.StatusForbidden, waf.DispositionDeny)
t,
do(t, req),
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
line := s.out.requestLine(t) line := s.out.requestLine(t)
if line.ClientIP != client || line.PeerIP != rproxyPeer { if line.ClientIP != client || line.PeerIP != rproxyPeer {
@@ -358,13 +339,13 @@ func TestMalformedQuestionGets400AndAWarningAndNoLine(t *testing.T) {
change func(*http.Request) change func(*http.Request)
}{ }{
{"client not an address", func(req *http.Request) { {"client not an address", func(req *http.Request) {
req.Header.Set(wafprotocol.ClientHeader, "not an address") req.Header.Set(waf.ClientHeader, "not an address")
}}, }},
{"client empty", func(req *http.Request) { {"client empty", func(req *http.Request) {
req.Header.Set(wafprotocol.ClientHeader, "") req.Header.Set(waf.ClientHeader, "")
}}, }},
{"client address with a zone", func(req *http.Request) { {"client address with a zone", func(req *http.Request) {
req.Header.Set(wafprotocol.ClientHeader, "2001:db8::1%eth0") req.Header.Set(waf.ClientHeader, "2001:db8::1%eth0")
}}, }},
{"id absent", func(req *http.Request) { {"id absent", func(req *http.Request) {
req.Header.Del(requestIDHeader) req.Header.Del(requestIDHeader)
@@ -409,12 +390,7 @@ func TestLongestValidIDIsTaken(t *testing.T) {
req := s.question(siteA, client, "/") req := s.question(siteA, client, "/")
req.Header.Set(requestIDHeader, strings.Repeat("x", 128)) req.Header.Set(requestIDHeader, strings.Repeat("x", 128))
wantDisposition( wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow)
t,
do(t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
)
} }
func TestQuestionWithABodyOver64KiBGets400(t *testing.T) { func TestQuestionWithABodyOver64KiBGets400(t *testing.T) {
@@ -423,18 +399,16 @@ func TestQuestionWithABodyOver64KiBGets400(t *testing.T) {
s := startWAF(t, "", nil) s := startWAF(t, "", nil)
over := strings.NewReader( over := strings.NewReader(
strings.Repeat("x", wafprotocol.QuestionBodyMaxBytes+1), strings.Repeat("x", waf.QuestionBodyMaxBytes+1),
) )
got := do(t, s.questionWith(http.MethodPost, siteA, client, "/", over)) got := do(t, s.questionWith(http.MethodPost, siteA, client, "/", over))
wantDisposition(t, got, http.StatusBadRequest, "") wantDisposition(t, got, http.StatusBadRequest, "")
wantProcessMessages(t, s, "a question is refused", 1) wantProcessMessages(t, s, "a question is refused", 1)
wantNoRequestLines(t, s) wantNoRequestLines(t, s)
whole := strings.NewReader( whole := strings.NewReader(strings.Repeat("x", waf.QuestionBodyMaxBytes))
strings.Repeat("x", wafprotocol.QuestionBodyMaxBytes),
)
got = do(t, s.questionWith(http.MethodPost, siteA, client, "/", whole)) got = do(t, s.questionWith(http.MethodPost, siteA, client, "/", whole))
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow) wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
} }
func TestQuestionWithAHeldIDIsRefused(t *testing.T) { func TestQuestionWithAHeldIDIsRefused(t *testing.T) {
@@ -450,12 +424,12 @@ func TestQuestionWithAHeldIDIsRefused(t *testing.T) {
wantProcessMessages(t, s, "a question is refused", 1) wantProcessMessages(t, s, "a question is refused", 1)
// A denied request's id is held as well. // A denied request's id is held as well.
denied := s.question(siteA, otherClient, wafprotocol.HealthPath) denied := s.question(siteA, otherClient, waf.HealthPath)
wantDisposition( wantDisposition(
t, t,
do(t, denied), do(t, denied),
http.StatusNotFound, http.StatusNotFound,
wafprotocol.DispositionDeny, waf.DispositionDeny,
) )
again = s.question(siteA, client, "/") again = s.question(siteA, client, "/")
@@ -463,9 +437,7 @@ func TestQuestionWithAHeldIDIsRefused(t *testing.T) {
wantDisposition(t, do(t, again), http.StatusBadRequest, "") wantDisposition(t, do(t, again), http.StatusBadRequest, "")
// The report ends the holding, and the refused questions changed nothing. // The report ends the holding, and the refused questions changed nothing.
got := s.report( got := s.report(waf.Report{RequestID: id, ResponseBytes: answerBytes})
wafprotocol.Report{RequestID: id, ResponseBytes: answerBytes},
)
wantStatus(t, got, http.StatusNoContent) wantStatus(t, got, http.StatusNoContent)
if history := s.history(); history.ResponseBytes != answerBytes { if history := s.history(); history.ResponseBytes != answerBytes {
@@ -474,12 +446,7 @@ func TestQuestionWithAHeldIDIsRefused(t *testing.T) {
again = s.question(siteA, client, "/") again = s.question(siteA, client, "/")
again.Header.Set(requestIDHeader, id) again.Header.Set(requestIDHeader, id)
wantDisposition( wantDisposition(t, do(t, again), http.StatusOK, waf.DispositionAllow)
t,
do(t, again),
http.StatusOK,
wafprotocol.DispositionAllow,
)
} }
func TestReportNeverReachesAQuestionStillBeingHandled(t *testing.T) { func TestReportNeverReachesAQuestionStillBeingHandled(t *testing.T) {
@@ -514,7 +481,7 @@ func TestReportNeverReachesAQuestionStillBeingHandled(t *testing.T) {
// A report for a question not yet answered is ignored, and that question's // A report for a question not yet answered is ignored, and that question's
// id is taken. // id is taken.
got := s.report(wafprotocol.Report{RequestID: id, ResponseBytes: 1000}) got := s.report(waf.Report{RequestID: id, ResponseBytes: 1000})
wantStatus(t, got, http.StatusNoContent) wantStatus(t, got, http.StatusNoContent)
// This would wait on the same lookup were the id not taken. // This would wait on the same lookup were the id not taken.
@@ -529,7 +496,7 @@ func TestReportNeverReachesAQuestionStillBeingHandled(t *testing.T) {
select { select {
case got := <-answered: case got := <-answered:
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow) wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
case <-time.After(waitLimit): case <-time.After(waitLimit):
t.Fatal("the question got no answer") t.Fatal("the question got no answer")
} }
@@ -548,9 +515,7 @@ func TestReportNeverReachesAQuestionStillBeingHandled(t *testing.T) {
} }
// The true report comes after, and is the one that counts. // The true report comes after, and is the one that counts.
got = s.report( got = s.report(waf.Report{RequestID: id, ResponseBytes: answerBytes})
wafprotocol.Report{RequestID: id, ResponseBytes: answerBytes},
)
wantStatus(t, got, http.StatusNoContent) wantStatus(t, got, http.StatusNoContent)
if history := s.history(); history.ResponseBytes != answerBytes { if history := s.history(); history.ResponseBytes != answerBytes {
@@ -568,17 +533,17 @@ func TestPeerAbsentOrUnreadableIsAnsweredAndLoggedInvalid(t *testing.T) {
s := startWAF(t, "", nil) s := startWAF(t, "", nil)
req := s.question(siteA, client, "/") req := s.question(siteA, client, "/")
req.Header.Del(wafprotocol.PeerHeader) req.Header.Del(waf.PeerHeader)
if peer != "" { if peer != "" {
req.Header.Set(wafprotocol.PeerHeader, peer) req.Header.Set(waf.PeerHeader, peer)
} }
wantDisposition( wantDisposition(
t, t,
do(t, req), do(t, req),
http.StatusOK, http.StatusOK,
wafprotocol.DispositionAllow, waf.DispositionAllow,
) )
if line := s.out.requestLine(t); line.PeerIP != "invalid IP" { if line := s.out.requestLine(t); line.PeerIP != "invalid IP" {
@@ -606,12 +571,12 @@ func TestContentLengthUnreadableOrNegativeCountsAsAbsent(t *testing.T) {
}) })
req := s.question(siteA, client, "/") req := s.question(siteA, client, "/")
req.Header.Set(wafprotocol.ContentLengthHeader, announced) req.Header.Set(waf.ContentLengthHeader, announced)
wantDisposition( wantDisposition(
t, t,
do(t, req), do(t, req),
http.StatusOK, http.StatusOK,
wafprotocol.DispositionAllow, waf.DispositionAllow,
) )
if line := s.out.requestLine(t); line.ContentLength != 0 { if line := s.out.requestLine(t); line.ContentLength != 0 {
@@ -630,18 +595,13 @@ func TestHeaderSentTwiceUsesItsFirstValue(t *testing.T) {
s := startWAF(t, "", nil) s := startWAF(t, "", nil)
req := s.question(siteA, client, "/") req := s.question(siteA, client, "/")
req.Header.Add(wafprotocol.ClientHeader, otherClient) req.Header.Add(waf.ClientHeader, otherClient)
req.Header.Add(wafprotocol.PeerHeader, "172.18.0.3") req.Header.Add(waf.PeerHeader, "172.18.0.3")
req.Header.Set(requestIDHeader, "first") req.Header.Set(requestIDHeader, "first")
req.Header.Add(requestIDHeader, "second") req.Header.Add(requestIDHeader, "second")
req.Header.Set(wafprotocol.ContentLengthHeader, "1234") req.Header.Set(waf.ContentLengthHeader, "1234")
req.Header.Add(wafprotocol.ContentLengthHeader, "5678") req.Header.Add(waf.ContentLengthHeader, "5678")
wantDisposition( wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow)
t,
do(t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
)
line := s.out.requestLine(t) line := s.out.requestLine(t)
if line.ClientIP != client || line.PeerIP != rproxyPeer || if line.ClientIP != client || line.PeerIP != rproxyPeer ||
@@ -670,7 +630,7 @@ func sendQuestionWithPartOfItsBody(t *testing.T, conn net.Conn) {
_, err := fmt.Fprintf(conn, _, err := fmt.Fprintf(conn,
"POST / HTTP/1.1\r\nHost: %s\r\n%s: %s\r\n%s: an-id\r\n"+ "POST / HTTP/1.1\r\nHost: %s\r\n%s: %s\r\n%s: an-id\r\n"+
"Content-Length: 100\r\n\r\npart", "Content-Length: 100\r\n\r\npart",
siteA, wafprotocol.ClientHeader, client, requestIDHeader) siteA, waf.ClientHeader, client, requestIDHeader)
if err != nil { if err != nil {
t.Fatalf("send the question: %v", err) t.Fatalf("send the question: %v", err)
} }
@@ -739,7 +699,7 @@ func TestQuestionWhoseBodyStopsArrivingGetsNoAnswer(t *testing.T) {
t, t,
do(t, again), do(t, again),
http.StatusOK, http.StatusOK,
wafprotocol.DispositionAllow, waf.DispositionAllow,
) )
}) })
} }
@@ -757,7 +717,7 @@ func TestQuestionDeniedGetsTheBanResponse(t *testing.T) {
}) })
got := s.ask(client, "/") got := s.ask(client, "/")
wantDisposition(t, got, status, wafprotocol.DispositionDeny) wantDisposition(t, got, status, waf.DispositionDeny)
wantBody(t, got, http.StatusText(status)+"\n") wantBody(t, got, http.StatusText(status)+"\n")
wantLine( wantLine(
t, t,
@@ -777,7 +737,7 @@ func TestBanResponseCloseIsAnAnswerOfItsOwn(t *testing.T) {
}) })
got := s.ask(client, "/") got := s.ask(client, "/")
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionClose) wantDisposition(t, got, http.StatusOK, waf.DispositionClose)
wantBody(t, got, "") wantBody(t, got, "")
// The visitor is sent nothing. // The visitor is sent nothing.
wantLine(t, s.out.requestLine(t), 0, requestlog.ActionClientDenied) wantLine(t, s.out.requestLine(t), 0, requestlog.ActionClientDenied)
@@ -801,22 +761,12 @@ func TestBlockRuleIsDeniedWith403AndABanRuleWithTheBanResponse(
}) })
got := s.ask(client, "/blocked") got := s.ask(client, "/blocked")
wantDisposition( wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
wantLine(t, s.out.requestLine(t), http.StatusForbidden, wantLine(t, s.out.requestLine(t), http.StatusForbidden,
requestlog.ActionBlockedByBlockRule) requestlog.ActionBlockedByBlockRule)
got = s.ask(client, "/.env") got = s.ask(client, "/.env")
wantDisposition( wantDisposition(t, got, http.StatusTooManyRequests, waf.DispositionDeny)
t,
got,
http.StatusTooManyRequests,
wafprotocol.DispositionDeny,
)
wantLine(t, s.out.requestLines(t, 2)[1], http.StatusTooManyRequests, wantLine(t, s.out.requestLines(t, 2)[1], http.StatusTooManyRequests,
requestlog.ActionTemporarilyBanned) requestlog.ActionTemporarilyBanned)
} }
@@ -831,15 +781,10 @@ func TestQuestionUnderTheOwnPrefixIsDeniedWith404AfterTheChecks(
}) })
for _, path := range []string{ for _, path := range []string{
wafprotocol.HealthPath, waf.MetricsPath, wafprotocol.ReportPath, waf.HealthPath, waf.MetricsPath, waf.ReportPath,
} { } {
got := s.ask(client, path) got := s.ask(client, path)
wantDisposition( wantDisposition(t, got, http.StatusNotFound, waf.DispositionDeny)
t,
got,
http.StatusNotFound,
wafprotocol.DispositionDeny,
)
} }
wantLine( wantLine(
@@ -850,12 +795,7 @@ func TestQuestionUnderTheOwnPrefixIsDeniedWith404AfterTheChecks(
) )
got := s.ask(otherClient, waf.MetricsPath) got := s.ask(otherClient, waf.MetricsPath)
wantDisposition( wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
} }
func TestObserveModeAllowsAndLogsWhatEnforceModeWouldHaveDone(t *testing.T) { func TestObserveModeAllowsAndLogsWhatEnforceModeWouldHaveDone(t *testing.T) {
@@ -864,7 +804,7 @@ func TestObserveModeAllowsAndLogsWhatEnforceModeWouldHaveDone(t *testing.T) {
s := startWAF(t, "", map[string]string{mode: observe, denyNets: client}) s := startWAF(t, "", map[string]string{mode: observe, denyNets: client})
got := s.ask(client, "/") got := s.ask(client, "/")
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow) wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
line := s.out.requestLine(t) line := s.out.requestLine(t)
wantLine(t, line, http.StatusOK, requestlog.ActionForward) wantLine(t, line, http.StatusOK, requestlog.ActionForward)
@@ -882,7 +822,7 @@ func TestObserveModeRefusalAnsweredAllowCountsForTheRefusalLimit(
for range 2 { for range 2 {
got := s.ask(client, "/blocked") got := s.ask(client, "/blocked")
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow) wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
} }
lines := s.out.requestLines(t, 2) lines := s.out.requestLines(t, 2)
@@ -906,12 +846,7 @@ func TestDeniedQuestionCountsForTheRefusalLimit(t *testing.T) {
for range 2 { for range 2 {
got := s.ask(client, "/blocked") got := s.ask(client, "/blocked")
wantDisposition( wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
} }
lines := s.out.requestLines(t, 2) lines := s.out.requestLines(t, 2)
@@ -947,12 +882,7 @@ func TestAllowCarriesTheLookupHeadersWhileTheSettingIsSet(t *testing.T) {
}) })
got := s.ask(fromDE, "/") got := s.ask(fromDE, "/")
wantDisposition( wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
t,
got,
http.StatusOK,
wafprotocol.DispositionAllow,
)
asn, country := got.header.Get( asn, country := got.header.Get(
"X-Client-Asn", "X-Client-Asn",
@@ -986,7 +916,7 @@ func TestReportCountsTowardAByteLimitWhichThenBans(t *testing.T) {
} }
// 100 bytes are over the limit of 99. // 100 bytes are over the limit of 99.
got := s.report(wafprotocol.Report{ got := s.report(waf.Report{
RequestID: id, Status: http.StatusBadGateway, RequestID: id, Status: http.StatusBadGateway,
RequestBytes: bodyBytes, ResponseBytes: answerBytes, RequestBytes: bodyBytes, ResponseBytes: answerBytes,
}) })
@@ -1010,12 +940,7 @@ func TestReportCountsTowardAByteLimitWhichThenBans(t *testing.T) {
} }
got = s.ask(client, "/") got = s.ask(client, "/")
wantDisposition( wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
} }
func TestReportCountsAtThePercentageItsQuestionWasGiven(t *testing.T) { func TestReportCountsAtThePercentageItsQuestionWasGiven(t *testing.T) {
@@ -1033,12 +958,12 @@ func TestReportCountsAtThePercentageItsQuestionWasGiven(t *testing.T) {
t, t,
s.ask(fromKP, "/"), s.ask(fromKP, "/"),
http.StatusOK, http.StatusOK,
wafprotocol.DispositionAllow, waf.DispositionAllow,
) )
s.passedOn(fromDE) s.passedOn(fromDE)
wantDisposition(t, s.ask(fromDE, "/"), http.StatusForbidden, wantDisposition(t, s.ask(fromDE, "/"), http.StatusForbidden,
wafprotocol.DispositionDeny) waf.DispositionDeny)
} }
func TestReportAddsTheRequestToItsClientsHistoryAndTheAnomalyCounters( func TestReportAddsTheRequestToItsClientsHistoryAndTheAnomalyCounters(
@@ -1064,7 +989,7 @@ func TestReportAddsTheRequestToItsClientsHistoryAndTheAnomalyCounters(
) )
} }
got := s.report(wafprotocol.Report{ got := s.report(waf.Report{
RequestID: id, Status: http.StatusBadGateway, RequestID: id, Status: http.StatusBadGateway,
RequestBytes: bodyBytes, ResponseBytes: answerBytes, RequestBytes: bodyBytes, ResponseBytes: answerBytes,
}) })
@@ -1096,7 +1021,7 @@ func TestDeniedRequestIsFinishedAtItsDecision(t *testing.T) {
t, t,
do(t, denied), do(t, denied),
http.StatusForbidden, http.StatusForbidden,
wafprotocol.DispositionDeny, waf.DispositionDeny,
) )
counters := s.server.Anomalies.Snapshot() counters := s.server.Anomalies.Snapshot()
@@ -1114,7 +1039,7 @@ func TestDeniedRequestIsFinishedAtItsDecision(t *testing.T) {
wantMetric(t, s.metrics(), series, 1) wantMetric(t, s.metrics(), series, 1)
got := s.report(wafprotocol.Report{ got := s.report(waf.Report{
RequestID: denied.Header.Get( RequestID: denied.Header.Get(
requestIDHeader, requestIDHeader,
), Status: http.StatusForbidden, ), Status: http.StatusForbidden,
@@ -1157,7 +1082,7 @@ func TestMetricsCountAnAllowedRequestFromItsReport(t *testing.T) {
1, 1,
) // the scrape itself ) // the scrape itself
got := s.report(wafprotocol.Report{ got := s.report(waf.Report{
RequestID: id, Status: http.StatusBadGateway, RequestID: id, Status: http.StatusBadGateway,
RequestBytes: bodyBytes, ResponseBytes: answerBytes, RequestBytes: bodyBytes, ResponseBytes: answerBytes,
DurationTotalMS: 5, DurationUpstreamTotalMS: 4, DurationTotalMS: 5, DurationUpstreamTotalMS: 4,
@@ -1175,7 +1100,7 @@ func TestMetricsCountAnAllowedRequestFromItsReport(t *testing.T) {
// check needs a server that has not been scraped yet. // check needs a server that has not been scraped yet.
unscraped := startWAF(t, "", map[string]string{metricsToken: token}) unscraped := startWAF(t, "", map[string]string{metricsToken: token})
got = unscraped.report(wafprotocol.Report{ got = unscraped.report(waf.Report{
RequestID: unscraped.allowed(client), Status: http.StatusBadGateway, RequestID: unscraped.allowed(client), Status: http.StatusBadGateway,
DurationTotalMS: 5, DurationTotalMS: 5,
}) })
@@ -1193,7 +1118,7 @@ func TestReportForARequestNotHeldIsIgnored(t *testing.T) {
s := startWAF(t, "", map[string]string{bytesLimitPerMinute: byteLimit}) s := startWAF(t, "", map[string]string{bytesLimitPerMinute: byteLimit})
got := s.report(wafprotocol.Report{ got := s.report(waf.Report{
RequestID: "no-such-request", ResponseBytes: 1000, RequestID: "no-such-request", ResponseBytes: 1000,
}) })
wantStatus(t, got, http.StatusNoContent) wantStatus(t, got, http.StatusNoContent)
@@ -1212,9 +1137,7 @@ func TestReportForARequestNotHeldIsIgnored(t *testing.T) {
// A second report of the same request does not count it twice. // A second report of the same request does not count it twice.
id := s.allowed(client) id := s.allowed(client)
for range 2 { for range 2 {
got = s.report( got = s.report(waf.Report{RequestID: id, ResponseBytes: answerBytes})
wafprotocol.Report{RequestID: id, ResponseBytes: answerBytes},
)
wantStatus(t, got, http.StatusNoContent) wantStatus(t, got, http.StatusNoContent)
} }
@@ -1424,7 +1347,7 @@ func TestBytesAtTheirMostEachWayBanRatherThanOverflow(t *testing.T) {
s := startWAF(t, "", map[string]string{bytesLimitPerMinute: byteLimit}) s := startWAF(t, "", map[string]string{bytesLimitPerMinute: byteLimit})
got := s.report(wafprotocol.Report{ got := s.report(waf.Report{
RequestID: s.allowed(client), Status: http.StatusOK, RequestID: s.allowed(client), Status: http.StatusOK,
RequestBytes: reportedBytesLimit, ResponseBytes: reportedBytesLimit, RequestBytes: reportedBytesLimit, ResponseBytes: reportedBytesLimit,
}) })
@@ -1518,14 +1441,9 @@ func TestReportWritesTheLastLineWithTheDecisionsAndTheReportsFields(
req.Header.Set("X-Forwarded-Proto", "https") req.Header.Set("X-Forwarded-Proto", "https")
id := req.Header.Get(requestIDHeader) id := req.Header.Get(requestIDHeader)
wantDisposition( wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow)
t,
do(t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
)
got := s.report(wafprotocol.Report{ got := s.report(waf.Report{
RequestID: id, Status: http.StatusCreated, RequestID: id, Status: http.StatusCreated,
RequestBytes: bodyBytes, ResponseBytes: answerBytes, RequestBytes: bodyBytes, ResponseBytes: answerBytes,
DurationTotalMS: 12.5, DurationUpstreamTotalMS: 10.25, DurationTotalMS: 12.5, DurationUpstreamTotalMS: 10.25,
@@ -1572,7 +1490,7 @@ func TestEveryDeniedRequestsReportGetsItsLastLine(t *testing.T) {
"close", map[string]string{denyNets: client, banResponse: banResponseClose}, "/", "close", map[string]string{denyNets: client, banResponse: banResponseClose}, "/",
0, requestlog.ActionClientDenied, 0, requestlog.ActionClientDenied,
}, },
{"own prefix", nil, wafprotocol.HealthPath, http.StatusNotFound, {"own prefix", nil, waf.HealthPath, http.StatusNotFound,
requestlog.ActionOwnEndpoint}, requestlog.ActionOwnEndpoint},
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
@@ -1584,7 +1502,7 @@ func TestEveryDeniedRequestsReportGetsItsLastLine(t *testing.T) {
id := req.Header.Get(requestIDHeader) id := req.Header.Get(requestIDHeader)
do(t, req) do(t, req)
got := s.report(wafprotocol.Report{ got := s.report(waf.Report{
RequestID: id, Status: tc.status, ResponseBytes: 8, DurationTotalMS: 3, RequestID: id, Status: tc.status, ResponseBytes: 8, DurationTotalMS: 3,
}) })
wantStatus(t, got, http.StatusNoContent) wantStatus(t, got, http.StatusNoContent)
@@ -1624,13 +1542,13 @@ func TestOldestAnsweredRequestIsDroppedPastTheBound(t *testing.T) {
t, t,
readAnswer(res), readAnswer(res),
http.StatusOK, http.StatusOK,
wafprotocol.DispositionAllow, waf.DispositionAllow,
) )
} }
// The table is full again with next and those, without oldest. // The table is full again with next and those, without oldest.
got := s.report( got := s.report(
wafprotocol.Report{RequestID: oldest, ResponseBytes: answerBytes}, waf.Report{RequestID: oldest, ResponseBytes: answerBytes},
) )
wantStatus(t, got, http.StatusNoContent) wantStatus(t, got, http.StatusNoContent)
@@ -1641,9 +1559,7 @@ func TestOldestAnsweredRequestIsDroppedPastTheBound(t *testing.T) {
) )
} }
got = s.report( got = s.report(waf.Report{RequestID: next, ResponseBytes: answerBytes})
wafprotocol.Report{RequestID: next, ResponseBytes: answerBytes},
)
wantStatus(t, got, http.StatusNoContent) wantStatus(t, got, http.StatusNoContent)
if counted := s.history().Requests; counted != 1 { if counted := s.history().Requests; counted != 1 {
@@ -1664,20 +1580,10 @@ func TestBanMadeThroughOneHostRefusesTheClientOnAnother(t *testing.T) {
) )
got := do(t, s.question(siteA, client, "/.env")) got := do(t, s.question(siteA, client, "/.env"))
wantDisposition( wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
got = do(t, s.question(siteB, client, "/")) got = do(t, s.question(siteB, client, "/"))
wantDisposition( wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
lines := s.out.requestLines(t, 2) lines := s.out.requestLines(t, 2)
if lines[0].Host != siteA || lines[1].Host != siteB { if lines[0].Host != siteA || lines[1].Host != siteB {
@@ -1715,12 +1621,7 @@ func TestRequestsOnEverySiteCountTowardOneSetOfLimits(t *testing.T) {
s.allowed(client) s.allowed(client)
got := do(t, s.question(siteB, client, "/")) got := do(t, s.question(siteB, client, "/"))
wantDisposition( wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
} }
func TestOwnEndpointsAnswerOnlyWithoutTheClientHeader(t *testing.T) { func TestOwnEndpointsAnswerOnlyWithoutTheClientHeader(t *testing.T) {
@@ -1735,13 +1636,13 @@ func TestOwnEndpointsAnswerOnlyWithoutTheClientHeader(t *testing.T) {
path, authorization string path, authorization string
want int want int
}{ }{
{wafprotocol.HealthPath, "", http.StatusOK}, {waf.HealthPath, "", http.StatusOK},
{waf.MetricsPath, bearer, http.StatusOK}, {waf.MetricsPath, bearer, http.StatusOK},
{waf.MetricsPath, "", http.StatusUnauthorized}, {waf.MetricsPath, "", http.StatusUnauthorized},
{waf.BansPath, adminBearer, http.StatusOK}, {waf.BansPath, adminBearer, http.StatusOK},
{waf.BansPath, "", http.StatusUnauthorized}, {waf.BansPath, "", http.StatusUnauthorized},
{waf.ClientsPath + client, adminBearer, http.StatusOK}, {waf.ClientsPath + client, adminBearer, http.StatusOK},
{wafprotocol.ReportPath, "", http.StatusNotFound}, {waf.ReportPath, "", http.StatusNotFound},
{"/_sww/other", "", http.StatusNotFound}, {"/_sww/other", "", http.StatusNotFound},
{"/", "", http.StatusNotFound}, {"/", "", http.StatusNotFound},
} { } {
@@ -1758,10 +1659,10 @@ func TestOwnEndpointsAnswerOnlyWithoutTheClientHeader(t *testing.T) {
t, t,
do(t, req), do(t, req),
http.StatusNotFound, http.StatusNotFound,
wafprotocol.DispositionDeny, waf.DispositionDeny,
) )
} else { } else {
wantDisposition(t, do(t, req), http.StatusOK, wafprotocol.DispositionAllow) wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow)
} }
} }
} }
@@ -1771,11 +1672,11 @@ func TestOwnEndpointsLinesDoNotSayWhichOfTheTwoTheyAre(t *testing.T) {
s := startWAF(t, "", nil) s := startWAF(t, "", nil)
wantStatus(t, get(t, s.addr, wafprotocol.HealthPath), http.StatusOK) wantStatus(t, get(t, s.addr, waf.HealthPath), http.StatusOK)
line := s.out.requestLine(t) line := s.out.requestLine(t)
if _, there := line.fields["waf_line"]; there || if _, there := line.fields["waf_line"]; there ||
line.Path != wafprotocol.HealthPath { line.Path != waf.HealthPath {
t.Errorf("the health check's log line is %+v", line.fields) t.Errorf("the health check's log line is %+v", line.fields)
} }
} }
@@ -1787,7 +1688,7 @@ func TestALargeBodyAnnouncedAndLongHeadersAreNoReasonToRefuse(t *testing.T) {
req := s.questionWith(http.MethodPost, siteA, client, "/", req := s.questionWith(http.MethodPost, siteA, client, "/",
strings.NewReader("a body")) strings.NewReader("a body"))
req.Header.Set(wafprotocol.ContentLengthHeader, "1073741824") req.Header.Set(waf.ContentLengthHeader, "1073741824")
req.Header.Set(forwardedFor, otherClient) req.Header.Set(forwardedFor, otherClient)
// 40,000 bytes of headers, over rproxy's 32 KiB, in lines within the // 40,000 bytes of headers, over rproxy's 32 KiB, in lines within the
@@ -1797,7 +1698,7 @@ func TestALargeBodyAnnouncedAndLongHeadersAreNoReasonToRefuse(t *testing.T) {
} }
got := do(t, req) got := do(t, req)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow) wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
line := s.out.requestLine(t) line := s.out.requestLine(t)
if line.ClientIP != client || line.ContentLength != 1<<30 { if line.ClientIP != client || line.ContentLength != 1<<30 {
@@ -1875,7 +1776,7 @@ carried header:Content-Length block ^6$
strings.NewReader("a body")) strings.NewReader("a body"))
if tc.announced != "" { if tc.announced != "" {
req.Header.Set(wafprotocol.ContentLengthHeader, tc.announced) req.Header.Set(waf.ContentLengthHeader, tc.announced)
} }
wantStatus(t, do(t, req), tc.want) wantStatus(t, do(t, req), tc.want)
@@ -1893,24 +1794,19 @@ func TestCoreRuleSetReadsTheBodyAQuestionCarries(t *testing.T) {
req := s.questionWith(http.MethodPost, siteA, client, "/", req := s.questionWith(http.MethodPost, siteA, client, "/",
strings.NewReader(body)) strings.NewReader(body))
req.Header.Set("Content-Type", contentType) req.Header.Set("Content-Type", contentType)
req.Header.Set(wafprotocol.ContentLengthHeader, "100000") req.Header.Set(waf.ContentLengthHeader, "100000")
return do(t, req) return do(t, req)
} }
got := ask(formData, "id=1'%20OR%20'1'='1") got := ask(formData, "id=1'%20OR%20'1'='1")
wantDisposition( wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
// The first 8 KiB of a longer JSON body cannot be parsed, which is no // The first 8 KiB of a longer JSON body cannot be parsed, which is no
// attack, so the Core Rule Set does not read it. // attack, so the Core Rule Set does not read it.
cut := `{"text":"` + strings.Repeat("x", 8<<10) cut := `{"text":"` + strings.Repeat("x", 8<<10)
got = ask("application/json", cut) got = ask("application/json", cut)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow) wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
} }
func TestCoreRuleSetAlertNamesTheSite(t *testing.T) { func TestCoreRuleSetAlertNamesTheSite(t *testing.T) {
@@ -1919,12 +1815,7 @@ func TestCoreRuleSetAlertNamesTheSite(t *testing.T) {
s := startWAF(t, "", map[string]string{coreRuleSetMode: block}) s := startWAF(t, "", map[string]string{coreRuleSetMode: block})
got := s.ask(client, sqlInjection) got := s.ask(client, sqlInjection)
wantDisposition( wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
waiting := s.queue.Snapshot().Waiting[alerts.DestinationWebhook] waiting := s.queue.Snapshot().Waiting[alerts.DestinationWebhook]
if len(waiting) != 1 || if len(waiting) != 1 ||
@@ -1945,7 +1836,7 @@ func TestReputationHitAlertNamesTheSite(t *testing.T) {
loadLists(t, s.server, map[string][]string{dropURL: {client}}) loadLists(t, s.server, map[string][]string{dropURL: {client}})
wantDisposition(t, s.ask(client, "/"), http.StatusForbidden, wantDisposition(t, s.ask(client, "/"), http.StatusForbidden,
wafprotocol.DispositionDeny) waf.DispositionDeny)
waiting := s.queue.Snapshot().Waiting[alerts.DestinationWebhook] waiting := s.queue.Snapshot().Waiting[alerts.DestinationWebhook]
if len(waiting) != 1 || waiting[0].Event != alerts.EventReputationHit || if len(waiting) != 1 || waiting[0].Event != alerts.EventReputationHit ||
-46
View File
@@ -1,46 +0,0 @@
package wafprotocol
import "net/http"
// Answer is what waf's answer to a question tells rproxy in its headers.
// The status, the other headers and the body of an answer of deny are what
// the visitor gets.
type Answer struct {
// Disposition is what rproxy does with the request: DispositionAllow,
// DispositionDeny or DispositionClose.
Disposition string
// ASN and Country are the client's AS number and country, each empty
// when the answer does not give it.
ASN, Country string
}
// SetAnswerHeaders sets the headers of waf's answer that tell rproxy answer,
// leaving out an ASN or Country that is empty.
func SetAnswerHeaders(header http.Header, answer Answer) {
header.Set(DispositionHeader, answer.Disposition)
if answer.ASN != "" {
header.Set(ASNHeader, answer.ASN)
}
if answer.Country != "" {
header.Set(CountryHeader, answer.Country)
}
}
// ReadAnswer returns what the headers of waf's answer tell rproxy, and
// whether they give a valid disposition.
func ReadAnswer(header http.Header) (Answer, bool) {
disposition := header.Get(DispositionHeader)
switch disposition {
case DispositionAllow, DispositionDeny, DispositionClose:
return Answer{
Disposition: disposition,
ASN: header.Get(ASNHeader),
Country: header.Get(CountryHeader),
}, true
}
return Answer{}, false
}
-189
View File
@@ -1,189 +0,0 @@
package wafprotocol
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"net/http"
"net/netip"
"net/url"
"strconv"
"unicode/utf8"
)
// maxRequestIDBytes is the longest id a question may give.
const maxRequestIDBytes = 128
// ErrBodyStoppedArriving is the error ReadQuestion wraps when the question's
// body stops arriving. Every other error it returns is why waf refuses the
// question.
var ErrBodyStoppedArriving = errors.New("the body stopped arriving")
// Why waf refuses a question. Each is also the body of the refusal.
var (
errClientNotAnAddress = errors.New(
ClientHeader + " is not an address without a zone",
)
errRequestIDNotValid = errors.New(RequestIDHeader + " is not a valid id")
errBodyTooLarge = errors.New("the body is over 64 KiB")
)
// QuestionParams are what NewQuestion needs.
type QuestionParams struct {
// Visitor is the visitor's request.
Visitor *http.Request
// PathAsSent is Visitor's path as the visitor sent it, which the question
// sends unchanged.
PathAsSent string
// Body is the part of Visitor's body the question carries, at most
// QuestionBodyMaxBytes.
Body []byte
// WAFBackend is waf's address. The question takes its scheme and host.
WAFBackend *url.URL
// Client is the visitor's address, and Peer that of rproxy's own TCP
// peer.
Client, Peer netip.Addr
// RequestID is the request's id.
RequestID string
// Scheme is how the visitor reached rproxy, or the trusted proxy in front
// of it.
Scheme string
}
// Question is a question as waf reads it.
type Question struct {
// Client is the visitor's address.
Client netip.Addr
// Peer is the address of rproxy's own TCP peer. It is not valid when the
// question gives none that can be read.
Peer netip.Addr
// RequestID is the request's id.
RequestID string
// Visitor is the visitor's request, as asTheVisitorSentIt gives it.
Visitor *http.Request
}
// NewQuestion returns the question about a visitor's request, as "The
// question" in SPEC.md gives it: the request as the visitor sent it, with the
// part of the body it carries and rproxy's headers.
func NewQuestion(
ctx context.Context,
params QuestionParams,
) (*http.Request, error) {
in := params.Visitor
question, err := http.NewRequestWithContext(
ctx,
in.Method,
params.WAFBackend.String(),
bytes.NewReader(params.Body),
)
if err != nil {
return nil, fmt.Errorf("make the question: %w", err)
}
backend := question.URL
target := *in.URL
target.Scheme = backend.Scheme
target.Host = backend.Host
target.Opaque = params.PathAsSent
question.URL = &target
question.Host = in.Host
// The length NewRequestWithContext set is that of the body carried. This
// line makes a body whose length the visitor did not announce go chunked,
// as "The question" in SPEC.md requires.
question.TransferEncoding = in.TransferEncoding
question.Header = in.Header.Clone()
question.Header.Set(ClientHeader, params.Client.String())
question.Header.Set(PeerHeader, params.Peer.String())
question.Header.Del(ContentLengthHeader)
announced := in.Header.Get("Content-Length")
if announced != "" {
question.Header.Set(ContentLengthHeader, announced)
}
question.Header.Set(RequestIDHeader, params.RequestID)
question.Header.Set("X-Forwarded-Proto", params.Scheme)
// Go's client adds a User-Agent of its own to a request without the key.
_, sent := question.Header["User-Agent"]
if !sent {
question.Header["User-Agent"] = nil
}
return question, nil
}
// ReadQuestion reads the question r, with the body it carries. An error
// wrapping ErrBodyStoppedArriving means the body stopped arriving. Any other
// error is why waf refuses the question, and the question's body is then not
// read. The body is read whole, since Go's server does not send an answer
// until it has read the body.
func ReadQuestion(w http.ResponseWriter, r *http.Request) (Question, error) {
client, err := netip.ParseAddr(r.Header.Get(ClientHeader))
if err != nil || client.Zone() != "" {
return Question{}, errClientNotAnAddress
}
id := r.Header.Get(RequestIDHeader)
if id == "" || len(id) > maxRequestIDBytes || !utf8.ValidString(id) {
return Question{}, errRequestIDNotValid
}
body, err := io.ReadAll(
http.MaxBytesReader(w, r.Body, QuestionBodyMaxBytes),
)
if _, tooLarge := errors.AsType[*http.MaxBytesError](err); tooLarge {
return Question{}, errBodyTooLarge
}
if err != nil {
return Question{}, fmt.Errorf("%w: %w", ErrBodyStoppedArriving, err)
}
// One that cannot be read leaves the log line's peer_ip invalid.
peer, _ := netip.ParseAddr(r.Header.Get(PeerHeader))
return Question{
Client: client.Unmap(),
Peer: peer.Unmap(),
RequestID: id,
Visitor: asTheVisitorSentIt(r, body),
}, nil
}
// asTheVisitorSentIt returns a copy of r, a question, as the checks and the
// log line are to see it: without the headers rproxy adds for waf,
// with body, which has been read of r, and with the length the visitor
// announced, as -1 when it announced none. The question's own length covers
// only the part of the body it carries.
func asTheVisitorSentIt(r *http.Request, body []byte) *http.Request {
sent := r.Clone(r.Context())
announced := sent.Header.Get(ContentLengthHeader)
sent.Header.Del(ClientHeader)
sent.Header.Del(PeerHeader)
sent.Header.Del(ContentLengthHeader)
sent.Header.Del("Content-Length")
sent.ContentLength = -1
length, err := strconv.ParseInt(announced, 10, 64)
if err == nil && length >= 0 {
sent.ContentLength = length
sent.Header.Set("Content-Length", announced)
}
sent.Body = http.NoBody
if len(body) > 0 {
sent.Body = io.NopCloser(bytes.NewReader(body))
}
return sent
}
-123
View File
@@ -1,123 +0,0 @@
package wafprotocol
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
)
const (
// reportBodyMaxBytes is the most of a report's body that is read. It is
// far above any report's.
reportBodyMaxBytes = 1 << 20
// maxReportedBytes is the most bytes a report may give for each way, 1
// PiB, so that adding them up cannot overflow.
maxReportedBytes = 1 << 50
// maxReportedMilliseconds is the longest duration a report may give,
// about 31 years, so that it fits a time.Duration.
maxReportedMilliseconds = 1e12
)
var (
errReportFieldMissing = errors.New("a field is missing")
errReportValue = errors.New("a value is out of range")
)
// Report is the JSON object rproxy posts to ReportPath once the visitor's
// answer has ended, its fields named as the request log's of the same
// meaning. RequestID is the question's id. Status is what the visitor got, 0
// if nothing. The durations are in milliseconds.
//
//nolint:tagliatelle // docs/request-log.md names the fields in snake_case
type Report struct {
RequestID string `json:"request_id"`
Status int `json:"status"`
RequestBytes int64 `json:"request_bytes"`
ResponseBytes int64 `json:"response_bytes"`
DurationTotalMS float64 `json:"duration_total_ms"`
DurationUpstreamTotalMS float64 `json:"duration_upstream_total_ms"`
}
// NewReportRequest returns the request that posts report to reportURL, which
// is waf's ReportPath.
func NewReportRequest(
ctx context.Context,
reportURL string,
report *Report,
) (*http.Request, error) {
body, err := json.Marshal(report)
if err != nil {
return nil, fmt.Errorf("encode the report: %w", err)
}
req, err := http.NewRequestWithContext(
ctx, http.MethodPost, reportURL, bytes.NewReader(body))
if err != nil {
return nil, fmt.Errorf("make the request: %w", err)
}
req.Header.Set("Content-Type", "application/json")
return req, nil
}
// ReadReport reads a report from r's body, which must be one JSON object with
// all of Report's fields, in the ranges SPEC.md gives.
func ReadReport(w http.ResponseWriter, r *http.Request) (Report, error) {
var report Report
body, err := io.ReadAll(
http.MaxBytesReader(w, r.Body, reportBodyMaxBytes),
)
if err != nil {
return report, fmt.Errorf("read the body: %w", err)
}
var fields map[string]json.RawMessage
err = json.Unmarshal(body, &fields)
if err != nil {
return report, fmt.Errorf("decode the body: %w", err)
}
for _, name := range []string{
"request_id", "status", "request_bytes", "response_bytes",
"duration_total_ms", "duration_upstream_total_ms",
} {
value, there := fields[name]
if !there || string(value) == "null" {
return report, fmt.Errorf("%w: %s", errReportFieldMissing, name)
}
}
err = json.Unmarshal(body, &report)
if err != nil {
return report, fmt.Errorf("decode the body: %w", err)
}
return report, report.checkRanges()
}
// checkRanges returns an error for a value of the report that cannot be true.
func (report *Report) checkRanges() error {
switch {
case report.Status != 0 &&
(report.Status < http.StatusContinue || report.Status > ReportStatusMax):
return fmt.Errorf("%w: status", errReportValue)
case report.RequestBytes < 0 || report.RequestBytes > maxReportedBytes:
return fmt.Errorf("%w: request_bytes", errReportValue)
case report.ResponseBytes < 0 || report.ResponseBytes > maxReportedBytes:
return fmt.Errorf("%w: response_bytes", errReportValue)
case report.DurationTotalMS < 0 || report.DurationTotalMS > maxReportedMilliseconds:
return fmt.Errorf("%w: duration_total_ms", errReportValue)
case report.DurationUpstreamTotalMS < 0 ||
report.DurationUpstreamTotalMS > maxReportedMilliseconds:
return fmt.Errorf("%w: duration_upstream_total_ms", errReportValue)
}
return nil
}
-61
View File
@@ -1,61 +0,0 @@
// Package wafprotocol holds the three messages rproxy and waf send each
// other, as "`waf`" in SPEC.md gives them. rproxy asks waf a question about
// each visitor's request, and waf answers it. Once the visitor's answer has
// ended, rproxy sends waf a report. This package writes and reads each
// message, and declares every name and limit both sides must agree on.
package wafprotocol
// The headers of the question and of the answer. Go writes every header name
// in this form, as it sends it and as it receives it, so X-Client-ASN arrives
// as X-Client-Asn; header names are not case-sensitive.
const (
// ClientHeader marks a request as rproxy's question, and gives the
// visitor's address.
ClientHeader = "X-Sww-Client"
// PeerHeader gives the address of rproxy's own TCP peer.
PeerHeader = "X-Sww-Peer"
// ContentLengthHeader gives the body length the visitor announced.
ContentLengthHeader = "X-Sww-Content-Length"
// RequestIDHeader gives the request's id, which rproxy makes. rproxy
// also sets it on the request it passes to the app.
RequestIDHeader = "X-Request-ID"
// DispositionHeader gives waf's answer.
DispositionHeader = "X-Sww-Disposition"
// ASNHeader and CountryHeader give the client's AS number and country in
// an answer of allow. rproxy passes them to the app.
ASNHeader = "X-Client-Asn"
CountryHeader = "X-Client-Country"
)
// The values of DispositionHeader.
const (
// DispositionAllow tells rproxy to pass the request to the app.
DispositionAllow = "allow"
// DispositionDeny tells rproxy to give the visitor the answer's status,
// headers and body.
DispositionDeny = "deny"
// DispositionClose tells rproxy to close the visitor's connection
// without an answer, which SWW_WAF_BAN_RESPONSE asks for when it is close.
// waf closing the question's connection instead would read as
// waf being down, and rproxy would then let the request through.
DispositionClose = "close"
)
// OwnEndpointPrefix starts the path of every request for sww itself: waf's
// own endpoints and rproxy's health endpoint. The app never sees a path under
// it.
const OwnEndpointPrefix = "/_sww/"
// HealthPath is the health endpoint of waf and of rproxy, which the
// container's health check asks.
const HealthPath = "/_sww/healthz"
// ReportPath is where rproxy sends the report of each request waf
// answered.
const ReportPath = "/_sww/report"
// QuestionBodyMaxBytes is the most of a visitor's body a question carries.
const QuestionBodyMaxBytes = 64 << 10
// ReportStatusMax is the highest status a report may give.
const ReportStatusMax = 599
-204
View File
@@ -1,204 +0,0 @@
package wafprotocol_test
import (
"io"
"net/http"
"net/http/httptest"
"net/netip"
"net/url"
"strings"
"testing"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// requestID is the id the tests' questions and reports give.
const requestID = "an-id"
// questionRead is what the server startQuestionReader starts read of a
// question: the question, and the body it carries.
type questionRead struct {
question wafprotocol.Question
body string
}
// startQuestionReader starts a server that reads each question it gets with
// ReadQuestion. It returns the server's URL and the channel what it reads goes
// to.
func startQuestionReader(t *testing.T) (*url.URL, <-chan questionRead) {
t.Helper()
read := make(chan questionRead, 1)
server := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
question, err := wafprotocol.ReadQuestion(w, r)
if err != nil {
t.Errorf("read the question: %v", err)
}
var body []byte
if question.Visitor != nil {
body, _ = io.ReadAll(question.Visitor.Body)
}
read <- questionRead{question, string(body)}
},
))
t.Cleanup(server.Close)
serverURL, err := url.Parse(server.URL)
if err != nil {
t.Fatalf("parse the server's URL: %v", err)
}
return serverURL, read
}
// TestQuestionReadsAsWritten sends a question NewQuestion writes over HTTP,
// and checks that ReadQuestion gives back the visitor's request and what
// rproxy added.
func TestQuestionReadsAsWritten(t *testing.T) {
t.Parallel()
const (
target = "/a%2Fb?q=1"
body = "name=value"
held = "name"
)
client := netip.MustParseAddr("2001:db8::1")
peer := netip.MustParseAddr("192.0.2.1")
backend, read := startQuestionReader(t)
visitor := httptest.NewRequestWithContext(
t.Context(),
http.MethodPost,
"http://site.example.com"+target,
strings.NewReader(body),
)
visitor.Header.Set("Content-Length", "10")
visitor.Header.Set("User-Agent", "a visitor")
visitor.Header.Set(wafprotocol.PeerHeader, "the visitor's own")
question, err := wafprotocol.NewQuestion(
t.Context(),
wafprotocol.QuestionParams{
Visitor: visitor,
PathAsSent: "/a%2Fb",
Body: []byte(held),
WAFBackend: backend,
Client: client,
Peer: peer,
RequestID: requestID,
Scheme: "https",
},
)
if err != nil {
t.Fatalf("write the question: %v", err)
}
res, err := http.DefaultTransport.RoundTrip(question)
if err != nil {
t.Fatalf("send the question: %v", err)
}
_ = res.Body.Close()
got := <-read
if got.question.Visitor == nil {
t.FailNow()
}
if got.question.Client != client || got.question.Peer != peer ||
got.question.RequestID != requestID {
t.Errorf("read %+v", got.question)
}
sent := got.question.Visitor
if sent.Host != "site.example.com" || sent.RequestURI != target ||
sent.ContentLength != int64(len(body)) || got.body != held {
t.Errorf(
"read %s at %s, of length %d, carrying %q",
sent.RequestURI, sent.Host, sent.ContentLength, got.body,
)
}
for name, want := range map[string]string{
"Content-Length": "10",
"User-Agent": "a visitor",
"X-Forwarded-Proto": "https",
wafprotocol.RequestIDHeader: requestID,
wafprotocol.ClientHeader: "",
wafprotocol.PeerHeader: "",
wafprotocol.ContentLengthHeader: "",
} {
if sent.Header.Get(name) != want {
t.Errorf("%s is %q, want %q", name, sent.Header.Get(name), want)
}
}
}
// TestAnswerReadsAsWritten checks that ReadAnswer gives back what
// SetAnswerHeaders sets, and that headers without a valid disposition give
// nothing.
func TestAnswerReadsAsWritten(t *testing.T) {
t.Parallel()
for _, want := range []wafprotocol.Answer{
{Disposition: wafprotocol.DispositionAllow, ASN: "64496", Country: "DE"},
{Disposition: wafprotocol.DispositionAllow, Country: "DE"},
{Disposition: wafprotocol.DispositionDeny},
{Disposition: wafprotocol.DispositionClose},
} {
header := http.Header{}
wafprotocol.SetAnswerHeaders(header, want)
got, valid := wafprotocol.ReadAnswer(header)
if !valid || got != want {
t.Errorf("read %+v (valid: %t), want %+v", got, valid, want)
}
}
header := http.Header{}
header.Set(wafprotocol.DispositionHeader, "maybe")
header.Set(wafprotocol.ASNHeader, "64496")
got, valid := wafprotocol.ReadAnswer(header)
if valid || got != (wafprotocol.Answer{}) {
t.Errorf("read %+v (valid: %t), want nothing", got, valid)
}
}
// TestReportReadsAsWritten checks that ReadReport gives back the report
// NewReportRequest posts.
func TestReportReadsAsWritten(t *testing.T) {
t.Parallel()
want := wafprotocol.Report{
RequestID: requestID,
Status: http.StatusOK,
RequestBytes: 10,
ResponseBytes: 20,
DurationTotalMS: 1.5,
DurationUpstreamTotalMS: 0.5,
}
req, err := wafprotocol.NewReportRequest(
t.Context(),
"http://waf.example.com"+wafprotocol.ReportPath,
&want,
)
if err != nil {
t.Fatalf("write the report: %v", err)
}
if req.Method != http.MethodPost ||
req.URL.Path != wafprotocol.ReportPath {
t.Errorf("the report is %s %s", req.Method, req.URL.Path)
}
got, err := wafprotocol.ReadReport(httptest.NewRecorder(), req)
if err != nil || got != want {
t.Errorf("read %+v (%v), want %+v", got, err, want)
}
}