Compare commits

..
Author SHA1 Message Date
sneak a46770ed58 waf: each heuristic holds its own settings and parts (closes #441)
check / check (push) Waiting to run
Every heuristic is a struct built once in newHandler with the settings
and collaborators it reads; no check reads the handler. The six lists
are handler fields built there. Code several heuristics share takes what
it needs as arguments or lives on a struct they hold: banMaker for bans
and their alerts, reputationHitNoter for reputation hits,
biasedThresholds for limit percentages. The handler keeps blocklists,
rateLimits, trapPaths, ruleFiles and coreRuleSet as fields too, since
refusedWhateverTheAnswer and findBesideLookup use them.

Model: opus-5-5
2026-11-19 19:41:05 +00:00
54 changed files with 1338 additions and 2015 deletions

No files matched your search

+3 -9
View File
@@ -902,19 +902,13 @@ stops the running process (see [`docs/state-files.md`](docs/state-files.md)).
- `internal/rproxy`: the reverse proxy in front of one app. - `internal/rproxy`: the reverse proxy in front of one app.
- `internal/waf`: `waf`'s server, which runs the checks and answers its own - `internal/waf`: `waf`'s server, which runs the checks and answers its own
endpoints. endpoints.
- `internal/wafprotocol`: the question, the answer and the report that `rproxy`
and `waf` send each other.
- `internal/ratelimit`, `internal/bans`, `internal/rules`, - `internal/ratelimit`, `internal/bans`, `internal/rules`,
`internal/coreruleset`, `internal/geojs`, `internal/lookup`, `internal/coreruleset`, `internal/geojs`, `internal/lookup`,
`internal/reputation`, `internal/routewatch`, `internal/anomaly` and `internal/reputation`, `internal/routewatch`, `internal/anomaly` and
`internal/alerts`: one check or source each. `internal/alerts`: one check or source each.
- `internal/dirwatch`: watches a directory, so that `internal/rules` and - `internal/state`, `internal/requestlog`, `internal/remotelog` and
`internal/lookup` read their files again when they change. `internal/metrics`: the state and cache files, the log lines, the syslog
- `internal/state`: the state and cache files. sender and the metrics.
- `internal/requestlog`: the request log.
- `internal/processlog`: the process's own messages.
- `internal/remotelog`: the syslog sender.
- `internal/metrics`: the metrics.
## Rationale ## Rationale
+7 -17
View File
@@ -69,21 +69,17 @@ never as a time.
that command's settings, and runs `internal/rproxy` until `SIGTERM` or that command's settings, and runs `internal/rproxy` until `SIGTERM` or
`SIGINT` (see "`rproxy`"). `sww healthcheck` is the image's health check. `SIGINT` (see "`rproxy`"). `sww healthcheck` is the image's health check.
With no command, or one it does not know, the binary prints the commands With no command, or one it does not know, the binary prints the commands
and exits with status 2. `sww help` and `sww --help` print them and exit and exits with status 2. `internal/sww` is the only package that imports
with status 0. `internal/sww` is the only package that imports
`go.uber.org/fx`. fx builds the parts of `waf` and `rproxy`, and starts `go.uber.org/fx`. fx builds the parts of `waf` and `rproxy`, and starts
and stops their background work. and stops their background work.
- `internal/config`: reads the settings, the one place they are read. - `internal/config`: reads the settings, the one place they are read.
- `internal/waf`: `waf`'s server. It runs the checks in `check`, and answers - `internal/waf`: `waf`'s server. It runs the checks in `check`, and holds
`waf`'s own endpoints in `answerOwnEndpoint`. It counts bytes, refusals `waf`'s question, answer and report. It answers `waf`'s own endpoints in
and anomalies, and writes the log lines. `answerOwnEndpoint`. It counts bytes, refusals and anomalies, and writes
the log lines.
- `internal/rproxy`: the reverse proxy in front of one app that asks `waf` - `internal/rproxy`: the reverse proxy in front of one app that asks `waf`
about each request. It passes a request to the app with the standard about each request. It passes a request to the app with the standard
library's `httputil.ReverseProxy`, within the time and size limits. library's `httputil.ReverseProxy`, within the time and size limits.
- `internal/wafprotocol`: the question, the answer and the report that
`rproxy` and `waf` send each other (see "`waf`"). It writes and reads each
one, and declares every header, path and limit both sides use.
`internal/rproxy` imports it, not `internal/waf`.
- `internal/ratelimit`: the table of clients. It counts requests, bytes and - `internal/ratelimit`: the table of clients. It counts requests, bytes and
refusals, and keeps each client's history. refusals, and keeps each client's history.
- `internal/bans`: the bans. It holds each netblock's bans with their notes, - `internal/bans`: the bans. It holds each netblock's bans with their notes,
@@ -97,9 +93,6 @@ never as a time.
imports no other package of this repository: `waf` gives it what it needs imports no other package of this repository: `waf` gives it what it needs
to count and report failures. to count and report failures.
- `internal/lookup`: looks clients up in the lookup database. - `internal/lookup`: looks clients up in the lookup database.
- `internal/dirwatch`: watches a directory, and calls a function once it has
gone a short time without a change. `internal/rules` and `internal/lookup`
read their files again this way.
- `internal/reputation`: fetches the lists, asks the DNS blocklists and - `internal/reputation`: fetches the lists, asks the DNS blocklists and
AbuseIPDB in the background, and keeps their answers. It also fetches AbuseIPDB in the background, and keeps their answers. It also fetches
AbuseIPDB's blacklist, which `internal/waf` bans. AbuseIPDB's blacklist, which `internal/waf` bans.
@@ -111,10 +104,8 @@ never as a time.
its own. its own.
- `internal/state`: reads the state and cache files at start, takes in edits - `internal/state`: reads the state and cache files at start, takes in edits
while running, and writes the files when due and at the stop. while running, and writes the files when due and at the stop.
- `internal/requestlog`: the request log, as JSON lines on stdout. - `internal/requestlog` and `internal/remotelog`: the lines on stdout, and
- `internal/processlog`: the process lines on stdout, which hold the sending them to the syslog server.
process's own messages.
- `internal/remotelog`: sends the lines on stdout to the syslog server.
- `internal/metrics`: the metrics, in the Prometheus text format. - `internal/metrics`: the metrics, in the Prometheus text format.
- `Dockerfile`: the lint and test phases, then the image. - `Dockerfile`: the lint and test phases, then the image.
`deploy/example-app` is an app built on the image, with `rproxy` in its `deploy/example-app` is an app built on the image, with `rproxy` in its
@@ -133,7 +124,6 @@ never as a time.
- `github.com/fsnotify/fsnotify` to notice edited files; - `github.com/fsnotify/fsnotify` to notice edited files;
- `github.com/prometheus/client_golang` for the metrics; - `github.com/prometheus/client_golang` for the metrics;
- `github.com/spf13/viper` to read the settings from the environment; - `github.com/spf13/viper` to read the settings from the environment;
- `github.com/spf13/cobra` for the commands and their help text;
- `github.com/maxmind/mmdbwriter`, in the tests, to write lookup databases; - `github.com/maxmind/mmdbwriter`, in the tests, to write lookup databases;
- remote log sending written in this project, since the standard library's - remote log sending written in this project, since the standard library's
`log/syslog` is frozen and writes only the older syslog format. `log/syslog` is frozen and writes only the older syslog format.
-2
View File
@@ -11,7 +11,6 @@ require (
github.com/maxmind/mmdbwriter v1.2.0 github.com/maxmind/mmdbwriter v1.2.0
github.com/oschwald/maxminddb-golang/v2 v2.7.0 github.com/oschwald/maxminddb-golang/v2 v2.7.0
github.com/prometheus/client_golang v1.24.1 github.com/prometheus/client_golang v1.24.1
github.com/spf13/cobra v1.10.2
github.com/spf13/viper v1.21.0 github.com/spf13/viper v1.21.0
go.uber.org/fx v1.24.0 go.uber.org/fx v1.24.0
golang.org/x/net v0.60.0 golang.org/x/net v0.60.0
@@ -27,7 +26,6 @@ require (
github.com/goccy/go-yaml v1.19.2 // indirect github.com/goccy/go-yaml v1.19.2 // indirect
github.com/gotnospirit/makeplural v0.0.0-20180622080156-a5f48d94d976 // indirect github.com/gotnospirit/makeplural v0.0.0-20180622080156-a5f48d94d976 // indirect
github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 // indirect github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/kaptinlin/go-i18n v0.1.4 // indirect github.com/kaptinlin/go-i18n v0.1.4 // indirect
github.com/kaptinlin/jsonschema v0.4.6 // indirect github.com/kaptinlin/jsonschema v0.4.6 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect github.com/kylelemons/godebug v1.1.0 // indirect
-9
View File
@@ -14,7 +14,6 @@ github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68 h1:KLEjvBBSw
github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68/go.mod h1:9XocblQEwK6EYylRth7PrPIifF2l0iSoYo4FAXhXU/8= github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68/go.mod h1:9XocblQEwK6EYylRth7PrPIifF2l0iSoYo4FAXhXU/8=
github.com/corazawaf/libinjection-go v0.3.4 h1:jX2Tit2xZxYJJyVryNXucYBZtQqS7473x9Nc6lgRRbk= github.com/corazawaf/libinjection-go v0.3.4 h1:jX2Tit2xZxYJJyVryNXucYBZtQqS7473x9Nc6lgRRbk=
github.com/corazawaf/libinjection-go v0.3.4/go.mod h1:Ik/+w3UmTWH9yn366RgS9D95K3y7Atb5m/H/gXzzPCk= github.com/corazawaf/libinjection-go v0.3.4/go.mod h1:Ik/+w3UmTWH9yn366RgS9D95K3y7Atb5m/H/gXzzPCk=
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/foxcpp/go-mockdns v1.2.0 h1:omK3OrHRD1IWJz1FuFBCFquhXslXoF17OvBS6JPzZF0= github.com/foxcpp/go-mockdns v1.2.0 h1:omK3OrHRD1IWJz1FuFBCFquhXslXoF17OvBS6JPzZF0=
github.com/foxcpp/go-mockdns v1.2.0/go.mod h1:IhLeSFGed3mJIAXPH2aiRQB+kqz7oqu8ld2qVbOu7Wk= github.com/foxcpp/go-mockdns v1.2.0/go.mod h1:IhLeSFGed3mJIAXPH2aiRQB+kqz7oqu8ld2qVbOu7Wk=
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8= github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
@@ -35,8 +34,6 @@ github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 h1:c7gcN
github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092/go.mod h1:ZZAN4fkkful3l1lpJwF8JbW41ZiG9TwJ2ZlqzQovBNU= github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092/go.mod h1:ZZAN4fkkful3l1lpJwF8JbW41ZiG9TwJ2ZlqzQovBNU=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k= github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM= github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/jcchavezs/mergefs v0.1.1 h1:D45R17m6dHnSVZefnhynoeZvcK2Uw0oTrRfoUOQ0S5Y= github.com/jcchavezs/mergefs v0.1.1 h1:D45R17m6dHnSVZefnhynoeZvcK2Uw0oTrRfoUOQ0S5Y=
github.com/jcchavezs/mergefs v0.1.1/go.mod h1:eRLTrsA+vFwQZ48hj8p8gki/5v9C2bFtHH5Mnn4bcGk= github.com/jcchavezs/mergefs v0.1.1/go.mod h1:eRLTrsA+vFwQZ48hj8p8gki/5v9C2bFtHH5Mnn4bcGk=
github.com/kaptinlin/go-i18n v0.1.4 h1:wCiwAn1LOcvymvWIVAM4m5dUAMiHunTdEubLDk4hTGs= github.com/kaptinlin/go-i18n v0.1.4 h1:wCiwAn1LOcvymvWIVAM4m5dUAMiHunTdEubLDk4hTGs=
@@ -75,7 +72,6 @@ github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+
github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY= github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY=
github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8= github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs= github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/sagikazarmark/locafero v0.11.0 h1:1iurJgmM9G3PA/I+wWYIOw/5SyBtxapeHDcg+AAIFXc= github.com/sagikazarmark/locafero v0.11.0 h1:1iurJgmM9G3PA/I+wWYIOw/5SyBtxapeHDcg+AAIFXc=
github.com/sagikazarmark/locafero v0.11.0/go.mod h1:nVIGvgyzw595SUSUE6tvCp3YYTeHs15MvlmU87WwIik= github.com/sagikazarmark/locafero v0.11.0/go.mod h1:nVIGvgyzw595SUSUE6tvCp3YYTeHs15MvlmU87WwIik=
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 h1:+jumHNA0Wrelhe64i8F6HNlS8pkoyMv5sreGx2Ry5Rw= github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 h1:+jumHNA0Wrelhe64i8F6HNlS8pkoyMv5sreGx2Ry5Rw=
@@ -84,9 +80,6 @@ github.com/spf13/afero v1.15.0 h1:b/YBCLWAJdFWJTN9cLhiXXcD7mzKn9Dm86dNnfyQw1I=
github.com/spf13/afero v1.15.0/go.mod h1:NC2ByUVxtQs4b3sIUphxK0NioZnmxgyCrfzeuq8lxMg= github.com/spf13/afero v1.15.0/go.mod h1:NC2ByUVxtQs4b3sIUphxK0NioZnmxgyCrfzeuq8lxMg=
github.com/spf13/cast v1.10.0 h1:h2x0u2shc1QuLHfxi+cTJvs30+ZAHOGRic8uyGTDWxY= github.com/spf13/cast v1.10.0 h1:h2x0u2shc1QuLHfxi+cTJvs30+ZAHOGRic8uyGTDWxY=
github.com/spf13/cast v1.10.0/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqeHo= github.com/spf13/cast v1.10.0/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqeHo=
github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk= github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.21.0 h1:x5S+0EU27Lbphp4UKm1C+1oQO+rKx36vfCoaVebLFSU= github.com/spf13/viper v1.21.0 h1:x5S+0EU27Lbphp4UKm1C+1oQO+rKx36vfCoaVebLFSU=
@@ -116,7 +109,6 @@ go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so= go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ= go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ= go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw= go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg= go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba h1:0b9z3AuHCjxk0x/opv64kcgZLBseWJUpBw5I82+2U4M= go4.org/netipx v0.0.0-20231129151722-fdeea329fbba h1:0b9z3AuHCjxk0x/opv64kcgZLBseWJUpBw5I82+2U4M=
@@ -135,7 +127,6 @@ golang.org/x/tools v0.50.0 h1:c2ifzfcuY7L90lZ2aKd8S4K2NpASF08SZx9ZuJkHmSU=
golang.org/x/tools v0.50.0/go.mod h1:7ulVMw3831Mwi5EZD6RomGyffr4VFjuNYXf2BbCEAV0= golang.org/x/tools v0.50.0/go.mod h1:7ulVMw3831Mwi5EZD6RomGyffr4VFjuNYXf2BbCEAV0=
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE= google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
rsc.io/binaryregexp v0.2.0 h1:HfqmD5MEmC0zvwBuF187nq9mdnXjXsSivRiXN7SmRkE= rsc.io/binaryregexp v0.2.0 h1:HfqmD5MEmC0zvwBuF187nq9mdnXjXsSivRiXN7SmRkE=
-90
View File
@@ -1,90 +0,0 @@
// Package dirwatch watches a directory, and calls a function once the
// directory has gone a short time without a change after one. The rule
// files and the lookup database are read again this way.
package dirwatch
import (
"context"
"time"
"github.com/fsnotify/fsnotify"
)
// quietTime is how long the directory must go without a change before the
// function is called, so that a file still being written, such as one
// saved in place, appended to or copied in with scp, is read only once
// whole.
const quietTime = 2 * time.Second
// Watcher watches one directory. The directory is watched, rather than a
// file in it, so that a file replaced by a rename is seen.
type Watcher struct {
watcher *fsnotify.Watcher
}
// New starts watching dir, or returns why it cannot.
func New(dir string) (*Watcher, error) {
watcher, err := fsnotify.NewWatcher()
if err != nil {
return nil, err
}
err = watcher.Add(dir)
if err != nil {
_ = watcher.Close()
return nil, err
}
return &Watcher{watcher: watcher}, nil
}
// Close stops watching the directory.
func (w *Watcher) Close() error {
return w.watcher.Close()
}
// CallWhenQuiet calls changed each time the directory has gone quietTime
// without a change, after a change to a path for which counts is true, until
// ctx is done. It also calls failed with each error the watch reports. The
// wait starts at once, as if for a change, so that a change made before the
// watch began is taken in too.
func (w *Watcher) CallWhenQuiet(
ctx context.Context,
counts func(path string) bool,
changed func(),
failed func(error),
) {
callWhenQuiet(
ctx, w.watcher.Events, w.watcher.Errors, counts, changed, failed,
)
}
// callWhenQuiet is CallWhenQuiet, for the changes from events and the
// errors from errs.
func callWhenQuiet(
ctx context.Context,
events <-chan fsnotify.Event,
errs <-chan error,
counts func(path string) bool,
changed func(),
failed func(error),
) {
quiet := time.NewTimer(quietTime)
defer quiet.Stop()
for {
select {
case <-ctx.Done():
return
case event := <-events:
if counts(event.Name) {
quiet.Reset(quietTime)
}
case <-quiet.C:
changed()
case err := <-errs:
failed(err)
}
}
}
-167
View File
@@ -1,167 +0,0 @@
package dirwatch
import (
"context"
"errors"
"path/filepath"
"sync/atomic"
"testing"
"testing/synctest"
"time"
"github.com/fsnotify/fsnotify"
)
// errWatch is an error the watch reports.
var errWatch = errors.New("too many changes")
func TestDirectoryThatCannotBeWatchedIsAnError(t *testing.T) {
t.Parallel()
_, err := New(filepath.Join(t.TempDir(), "missing"))
if err == nil {
t.Error("a missing directory is watched, want an error")
}
}
// The tests below run callWhenQuiet in a synctest bubble, where time is a
// clock of the test's own: time.Sleep moves it on at once, and
// synctest.Wait returns once callWhenQuiet waits again, so that every call
// due by then is made. The test sends the changes itself, as the watch of
// a directory cannot run in a bubble.
func TestSecondChangeStartsTheWaitAgain(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
loop := start(t, everyPath)
loop.changes <- fsnotify.Event{Name: "50-app.rules", Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
loop.wantCalls(t, 0)
// The second part starts the wait again.
loop.changes <- fsnotify.Event{Name: "50-app.rules", Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
loop.wantCalls(t, 0)
time.Sleep(time.Nanosecond)
synctest.Wait()
loop.wantCalls(t, 1)
})
}
func TestCalledOnceQuietAfterTheStart(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
// An edit saved before the watch began sends no change, and is
// taken in by this call.
loop := start(t, everyPath)
time.Sleep(quietTime)
synctest.Wait()
loop.wantCalls(t, 1)
// Long after, it has been called only once.
time.Sleep(time.Hour)
synctest.Wait()
loop.wantCalls(t, 1)
})
}
func TestChangeThatDoesNotCountIsNoChange(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
loop := start(t, func(path string) bool {
return path == "ipinfo_lite.mmdb"
})
// The wait that starts with the watch ends with a call.
time.Sleep(quietTime)
synctest.Wait()
loop.changes <- fsnotify.Event{Name: "other.mmdb", Op: fsnotify.Create}
time.Sleep(quietTime)
synctest.Wait()
loop.wantCalls(t, 1)
loop.changes <- fsnotify.Event{Name: "ipinfo_lite.mmdb", Op: fsnotify.Write}
time.Sleep(quietTime)
synctest.Wait()
loop.wantCalls(t, 2)
})
}
func TestErrorsOfTheWatchHandedOn(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
loop := start(t, everyPath)
loop.errs <- errWatch
synctest.Wait()
if !errors.Is(loop.failure, errWatch) {
t.Errorf("failed with %v, want %v", loop.failure, errWatch)
}
})
}
// loop is a callWhenQuiet that a test runs.
type loop struct {
// changes and errs send it changes and errors.
changes chan fsnotify.Event
errs chan error
// calls counts its calls of changed, and failure is the last error it
// handed to failed.
calls atomic.Int32
failure error
}
// start runs callWhenQuiet, with counts, until the test ends.
func start(t *testing.T, counts func(path string) bool) *loop {
t.Helper()
l := &loop{changes: make(chan fsnotify.Event), errs: make(chan error)}
ctx, stop := context.WithCancel(t.Context())
stopped := make(chan struct{})
go func() {
callWhenQuiet(ctx, l.changes, l.errs, counts,
func() { l.calls.Add(1) },
func(err error) { l.failure = err },
)
close(stopped)
}()
t.Cleanup(func() {
stop()
<-stopped
})
return l
}
// everyPath counts a change to any path.
func everyPath(string) bool {
return true
}
// wantCalls checks how many times changed has been called.
func (l *loop) wantCalls(t *testing.T, want int32) {
t.Helper()
got := l.calls.Load()
if got != want {
t.Errorf("changed called %d times, want %d", got, want)
}
}
+24 -14
View File
@@ -84,7 +84,12 @@ type Params struct {
// one request to GeoJS is under way at a time, and it asks about every // one request to GeoJS is under way at a time, and it asks about every
// client waiting, up to maxPerRequest. It is safe for concurrent use. // client waiting, up to maxPerRequest. It is safe for concurrent use.
type GeoJS struct { type GeoJS struct {
params Params url string
answered func(Answer)
now func() time.Time
asked func()
failed func(err error, askingAgainIn time.Duration)
wentWithoutAnswer func()
// httpClient follows no redirect, so that visitors' addresses go to // httpClient follows no redirect, so that visitors' addresses go to
// GeoJS alone: a redirect is a failure. // GeoJS alone: a redirect is a failure.
httpClient *http.Client httpClient *http.Client
@@ -140,7 +145,12 @@ func New(params Params) *GeoJS {
} }
return &GeoJS{ return &GeoJS{
params: params, url: params.URL,
answered: params.Answered,
now: params.Now,
asked: params.Asked,
failed: params.Failed,
wentWithoutAnswer: params.WentWithoutAnswer,
httpClient: &http.Client{ httpClient: &http.Client{
CheckRedirect: func(*http.Request, []*http.Request) error { CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse return http.ErrUseLastResponse
@@ -183,7 +193,7 @@ func (g *GeoJS) LookUp(
answer, found := g.kept(client) answer, found := g.kept(client)
if !found { if !found {
g.params.WentWithoutAnswer() g.wentWithoutAnswer()
} }
w, waiting := g.waiting[client] w, waiting := g.waiting[client]
@@ -263,7 +273,7 @@ func (g *GeoJS) Load(answers []Answer) {
g.answers.Purge() g.answers.Purge()
now := g.params.Now() now := g.now()
for _, answer := range answers { for _, answer := range answers {
if now.Sub(answer.Answered) < keepFor { if now.Sub(answer.Answered) < keepFor {
@@ -300,7 +310,7 @@ func (g *GeoJS) answerOrWait(
} }
if w == nil { if w == nil {
g.params.WentWithoutAnswer() g.wentWithoutAnswer()
return Answer{}, nil // too many clients wait already return Answer{}, nil // too many clients wait already
} }
@@ -312,7 +322,7 @@ func (g *GeoJS) answerOrWait(
} }
if w.late { if w.late {
g.params.WentWithoutAnswer() g.wentWithoutAnswer()
return Answer{}, nil return Answer{}, nil
} }
@@ -323,7 +333,7 @@ func (g *GeoJS) answerOrWait(
// kept returns client's answer, if GeoJS gave it less than keepFor ago, // kept returns client's answer, if GeoJS gave it less than keepFor ago,
// and notes that it was used. // and notes that it was used.
func (g *GeoJS) kept(client netip.Prefix) (Answer, bool) { func (g *GeoJS) kept(client netip.Prefix) (Answer, bool) {
now := g.params.Now() now := g.now()
kept, found := g.answers.Get(client) kept, found := g.answers.Get(client)
if !found || now.Sub(kept.Answered) >= keepFor { if !found || now.Sub(kept.Answered) >= keepFor {
@@ -340,7 +350,7 @@ func (g *GeoJS) kept(client netip.Prefix) (Answer, bool) {
// been called. The requests to GeoJS are for every client waiting, so they // been called. The requests to GeoJS are for every client waiting, so they
// go on when the caller whose ctx is given stops waiting, until StopAsking. // go on when the caller whose ctx is given stops waiting, until StopAsking.
func (g *GeoJS) ask(ctx context.Context) { func (g *GeoJS) ask(ctx context.Context) {
if g.asking || g.stopped || g.params.Now().Before(g.retryAt) { if g.asking || g.stopped || g.now().Before(g.retryAt) {
return return
} }
@@ -372,9 +382,9 @@ func (g *GeoJS) askAboutWaiting(ctx context.Context) {
kept, answered := g.keep(clients, given, err) kept, answered := g.keep(clients, given, err)
if g.params.Answered != nil { if g.answered != nil {
for _, answer := range kept { for _, answer := range kept {
g.params.Answered(answer) g.answered(answer)
} }
} }
@@ -423,7 +433,7 @@ func (g *GeoJS) keep(
g.mu.Lock() g.mu.Lock()
defer g.mu.Unlock() defer g.mu.Unlock()
now := g.params.Now() now := g.now()
kept := make([]Answer, 0, len(clients)) kept := make([]Answer, 0, len(clients))
leftOut := 0 leftOut := 0
@@ -467,7 +477,7 @@ func (g *GeoJS) keep(
w.late = true w.late = true
} }
g.params.Failed(err, g.retryDelay) g.failed(err, g.retryDelay)
return kept, false return kept, false
} }
@@ -497,7 +507,7 @@ func (g *GeoJS) request(
req, err := http.NewRequestWithContext( req, err := http.NewRequestWithContext(
ctx, ctx,
http.MethodGet, http.MethodGet,
g.params.URL, g.url,
http.NoBody, http.NoBody,
) )
if err != nil { if err != nil {
@@ -506,7 +516,7 @@ func (g *GeoJS) request(
req.URL.RawQuery = "ip=" + strings.Join(addrs, ",") req.URL.RawQuery = "ip=" + strings.Join(addrs, ",")
g.params.Asked() g.asked()
res, err := g.httpClient.Do(req) res, err := g.httpClient.Do(req)
if err != nil { if err != nil {
+49 -18
View File
@@ -13,13 +13,18 @@ import (
"sync" "sync"
"time" "time"
"github.com/fsnotify/fsnotify"
"github.com/oschwald/maxminddb-golang/v2" "github.com/oschwald/maxminddb-golang/v2"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/dirwatch"
"sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/geojs"
) )
// quietTime is how long the lookup database must go without a change
// before it is read again, so that a file still being copied in is read
// only once whole.
const quietTime = 2 * time.Second
// FileParams are what OpenFile needs. // FileParams are what OpenFile needs.
type FileParams struct { type FileParams struct {
// Path is the lookup database, the IPinfo Lite file in its .mmdb form // Path is the lookup database, the IPinfo Lite file in its .mmdb form
@@ -117,13 +122,20 @@ func (f *File) ReadFailures() int {
} }
// Watch watches the directory of the lookup database until ctx is done, // Watch watches the directory of the lookup database until ctx is done,
// and reads the file again once it has gone without a change for a short // and reads the file again once it has gone without a change for
// time, after it is replaced, written or removed, and after Watch starts // quietTime, after it is replaced, written or removed, and after Watch
// watching, as dirwatch describes. A change to another file in its // starts watching. If the directory cannot be watched, that is logged, and
// directory does not count. If the directory cannot be watched, that is // the database read at start stays in use.
// logged, and the database read at start stays in use.
func (f *File) Watch(ctx context.Context) { func (f *File) Watch(ctx context.Context) {
watcher, err := dirwatch.New(filepath.Dir(f.params.Path)) watcher, err := fsnotify.NewWatcher()
if err == nil {
defer func() {
_ = watcher.Close()
}()
err = watcher.Add(filepath.Dir(f.params.Path))
}
if err != nil { if err != nil {
f.params.ProcessLog.Error( f.params.ProcessLog.Error(
"cannot watch the lookup database for replacements", "cannot watch the lookup database for replacements",
@@ -134,22 +146,41 @@ func (f *File) Watch(ctx context.Context) {
return return
} }
defer func() {
_ = watcher.Close()
}()
f.params.ProcessLog.Info("watching the lookup database for replacements", f.params.ProcessLog.Info("watching the lookup database for replacements",
"file", f.params.Path) "file", f.params.Path)
watcher.CallWhenQuiet(ctx, f.isDatabase, f.readAgain, func(err error) { f.readAfterChanges(ctx, watcher.Events, watcher.Errors)
f.params.ProcessLog.Warn("watching the lookup database failed",
"error", err.Error())
})
} }
// isDatabase reports whether path names the lookup database. // readAfterChanges reads the lookup database again once quietTime has
func (f *File) isDatabase(path string) bool { // passed without a change to it from events, until ctx is done, and logs
return filepath.Clean(path) == filepath.Clean(f.params.Path) // the errors from errs. A change to another file in its directory does not
// count. The wait starts at once, as if for a change, so that a file
// replaced after OpenFile read it, and before its directory was watched,
// is read too.
func (f *File) readAfterChanges(
ctx context.Context, events <-chan fsnotify.Event, errs <-chan error,
) {
path := filepath.Clean(f.params.Path)
quiet := time.NewTimer(quietTime)
defer quiet.Stop()
for {
select {
case <-ctx.Done():
return
case event := <-events:
if filepath.Clean(event.Name) == path {
quiet.Reset(quietTime)
}
case <-quiet.C:
f.readAgain()
case err := <-errs:
f.params.ProcessLog.Warn("watching the lookup database failed",
"error", err.Error())
}
}
} }
// readAgain reads the lookup database again, in place of the one in use, // readAgain reads the lookup database again, in place of the one in use,
+141 -35
View File
@@ -1,6 +1,7 @@
package lookup package lookup
import ( import (
"context"
"log/slog" "log/slog"
"net/netip" "net/netip"
"net/url" "net/url"
@@ -11,6 +12,7 @@ import (
"testing/synctest" "testing/synctest"
"time" "time"
"github.com/fsnotify/fsnotify"
"github.com/maxmind/mmdbwriter/mmdbtype" "github.com/maxmind/mmdbwriter/mmdbtype"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
@@ -131,48 +133,66 @@ func TestFileThatCannotBeReadIsAnError(t *testing.T) {
} }
} }
func TestChangeOfAnotherFileInTheDirectoryIsNoReplacement(t *testing.T) { // The tests below run readAfterChanges in a synctest bubble, where time is
t.Parallel() // a clock of the test's own: time.Sleep moves it on at once, and
// synctest.Wait returns once readAfterChanges waits again, so that every
// reading due by then is done. The test sends the changes itself, as the
// watch of a directory cannot run in a bubble.
dir := t.TempDir() func TestReplacementCopiedOverTheFileInTwoPartsIsReadOnlyWhole(
path := filepath.Join(dir, "ipinfo_lite.mmdb") t *testing.T,
writeDatabase(t, path, "DE") ) {
f := openFile(t, path, newQueue())
for changed, want := range map[string]bool{
path: true,
dir + "/./ipinfo_lite.mmdb": true,
filepath.Join(dir, "other.mmdb"): false,
} {
if f.isDatabase(changed) != want {
t.Errorf(
"a change to %s counts: %t, want %t",
changed,
!want,
want,
)
}
}
}
// The tests below run in a synctest bubble, where time is a clock of the
// test's own, so that the time a file is read can be checked. The wait for
// the directory to go quiet before a reading is tested in dirwatch.
func TestReplacementIsReadAgain(t *testing.T) {
t.Parallel() t.Parallel()
synctest.Test(t, func(t *testing.T) { synctest.Test(t, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb") dir := t.TempDir()
path := filepath.Join(dir, "ipinfo_lite.mmdb")
writeDatabase(t, path, "DE") writeDatabase(t, path, "DE")
queue := newQueue() queue := newQueue()
f := openFile(t, path, queue) f := openFile(t, path, queue)
changes := watch(t, f)
writeDatabase(t, path, "KP") other := filepath.Join(dir, "replacement.mmdb")
time.Sleep(time.Hour) writeDatabase(t, other, "KP")
f.readAgain()
//nolint:gosec // a file the test wrote
replacement, err := os.ReadFile(other)
if err != nil {
t.Fatalf("read %s: %v", other, err)
}
// The file in use is overwritten in place, and keeps giving what
// it gave. Its first part alone is not a .mmdb file.
file, err := os.Create(path) //nolint:gosec // a file the test wrote
if err != nil {
t.Fatalf("create %s: %v", path, err)
}
defer func() {
_ = file.Close()
}()
half := len(replacement) / 2
write(t, file, replacement[:half])
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantCountry(t, f, "DE")
// The second part starts the wait again.
write(t, file, replacement[half:])
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantCountry(t, f, "DE")
time.Sleep(time.Nanosecond)
synctest.Wait()
wantCountry(t, f, "KP") wantCountry(t, f, "KP")
if !f.LastRead().Equal(time.Now()) || f.ReadFailures() != 0 { if !f.LastRead().Equal(time.Now()) || f.ReadFailures() != 0 {
@@ -199,10 +219,15 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(
queue := newQueue() queue := newQueue()
f := openFile(t, path, queue) f := openFile(t, path, queue)
read := f.LastRead() read := f.LastRead()
changes := watch(t, f)
writeFile(t, path, "not a lookup database\n") writeFile(t, path, "not a lookup database\n")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
// Long after, the replacement has been read once.
time.Sleep(time.Hour) time.Sleep(time.Hour)
f.readAgain() synctest.Wait()
wantCountry(t, f, "DE") wantCountry(t, f, "DE")
if !f.LastRead().Equal(read) || f.ReadFailures() != 1 { if !f.LastRead().Equal(read) || f.ReadFailures() != 1 {
@@ -215,7 +240,7 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(
} }
wantAlerts(t, queue, alerts.Alert{ wantAlerts(t, queue, alerts.Alert{
Time: read.Add(time.Hour), Time: read.Add(quietTime),
Event: alerts.EventFileError, Event: alerts.EventFileError,
Reason: "the lookup database cannot be read, and the one read before stays in use", Reason: "the lookup database cannot be read, and the one read before stays in use",
Detail: map[string]any{ Detail: map[string]any{
@@ -227,6 +252,55 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(
}) })
} }
func TestChangeOfAnotherFileInTheDirectoryIsNoReplacement(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "ipinfo_lite.mmdb")
writeDatabase(t, path, "DE")
f := openFile(t, path, newQueue())
changes := watch(t, f)
// The wait that starts with the watch ends with a reading.
time.Sleep(quietTime)
synctest.Wait()
writeDatabase(t, path, "KP")
changes <- fsnotify.Event{Name: filepath.Join(dir, "other.mmdb"), Op: fsnotify.Create}
time.Sleep(quietTime)
synctest.Wait()
wantCountry(t, f, "DE")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime)
synctest.Wait()
wantCountry(t, f, "KP")
})
}
func TestReplacementSavedBeforeTheWatchStartsIsRead(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb")
writeDatabase(t, path, "DE")
f := openFile(t, path, newQueue())
// Saved after OpenFile read the file, and before its directory was
// watched, so that no change is seen for it.
writeDatabase(t, path, "KP")
watch(t, f)
time.Sleep(quietTime)
synctest.Wait()
wantCountry(t, f, "KP")
})
}
// newQueue returns a queue of alerts for a webhook that is never sent // newQueue returns a queue of alerts for a webhook that is never sent
// them, so that they wait in it for the test to look at. // them, so that they wait in it for the test to look at.
func newQueue() *alerts.Queue { func newQueue() *alerts.Queue {
@@ -270,6 +344,28 @@ func openFile(t *testing.T, path string, queue *alerts.Queue) *File {
return f return f
} }
// watch runs f's readAfterChanges until the test ends, and returns the
// channel that sends it changes.
func watch(t *testing.T, f *File) chan<- fsnotify.Event {
t.Helper()
changes := make(chan fsnotify.Event)
ctx, stop := context.WithCancel(t.Context())
stopped := make(chan struct{})
go func() {
f.readAfterChanges(ctx, changes, nil)
close(stopped)
}()
t.Cleanup(func() {
stop()
<-stopped
})
return changes
}
// wantCountry checks the country f gives testClient. // wantCountry checks the country f gives testClient.
func wantCountry(t *testing.T, f *File, want string) { func wantCountry(t *testing.T, f *File, want string) {
t.Helper() t.Helper()
@@ -305,3 +401,13 @@ func writeFile(t *testing.T, path, content string) {
t.Fatalf("write %s: %v", path, err) t.Fatalf("write %s: %v", path, err)
} }
} }
// write writes data to the end of file.
func write(t *testing.T, file *os.File, data []byte) {
t.Helper()
_, err := file.Write(data)
if err != nil {
t.Fatalf("write: %v", err)
}
}
-44
View File
@@ -1,44 +0,0 @@
// Package processlog builds the logger for the process's own messages,
// which it writes as JSON lines marked "type":"process". The request log
// is package requestlog's.
package processlog
import (
"io"
"log/slog"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// Params are what New needs.
type Params struct {
// Output is where the lines are written.
Output io.Writer
// InstanceName, SWW_WAF_INSTANCE_NAME, is each line's instance.
InstanceName string
// Level, SWW_WAF_LOG_LEVEL, is the least severe level written.
Level slog.Level
}
// New returns the logger for the process's own messages: JSON lines on
// params.Output, marked "type":"process", with the time in the same form
// as a request line's. It writes only the messages at params.Level or more
// severe; the request lines requestlog.Write writes are never held back.
func New(params Params) *slog.Logger {
handler := slog.NewJSONHandler(params.Output, &slog.HandlerOptions{
Level: params.Level,
ReplaceAttr: func(groups []string, attr slog.Attr) slog.Attr {
if attr.Key == slog.TimeKey && len(groups) == 0 {
return slog.String(
slog.TimeKey,
requestlog.FormatTime(attr.Value.Time()),
)
}
return attr
},
})
return slog.New(handler).
With("type", "process", "instance", params.InstanceName)
}
-105
View File
@@ -1,105 +0,0 @@
package processlog_test
import (
"bytes"
"encoding/json"
"log/slog"
"slices"
"strings"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/processlog"
)
// instanceName is the instance the tests' loggers are built with.
const instanceName = "host1/blog"
func TestProcessLinesAreMarkedProcessAndGiveTheInstance(t *testing.T) {
t.Parallel()
var out bytes.Buffer
processlog.New(processlog.Params{
Output: &out,
InstanceName: instanceName,
Level: slog.LevelInfo,
}).
Info("starting",
"version", "v1")
var fields map[string]any
err := json.Unmarshal(out.Bytes(), &fields)
if err != nil {
t.Fatalf("decode %q: %v", out.String(), err)
}
if fields["type"] != "process" ||
fields["instance"] != "host1/blog" ||
fields["msg"] != "starting" ||
fields["level"] != "INFO" ||
fields["version"] != "v1" {
t.Errorf("process line %v", fields)
}
timeText, _ := fields["time"].(string)
logged, err := time.Parse(time.RFC3339, timeText)
if err != nil || !strings.HasSuffix(timeText, "Z") ||
len(timeText) != len("2006-01-02T15:04:05.000Z") ||
time.Since(logged) > time.Minute {
t.Errorf(
"process line time %q, want now in UTC with milliseconds",
timeText,
)
}
}
func TestProcessLoggerWritesTheMessagesAtItsLevelOrMoreSevere(t *testing.T) {
t.Parallel()
levels := []slog.Level{
slog.LevelDebug, slog.LevelInfo, slog.LevelWarn, slog.LevelError,
}
for i, level := range levels {
t.Run(level.String(), func(t *testing.T) {
t.Parallel()
var out bytes.Buffer
processLog := processlog.New(processlog.Params{
Output: &out,
InstanceName: instanceName,
Level: level,
})
for _, at := range levels {
processLog.Log(t.Context(), at, "message")
}
var got, want []string
for line := range strings.Lines(out.String()) {
var fields struct {
Level string `json:"level"`
}
err := json.Unmarshal([]byte(line), &fields)
if err != nil {
t.Fatalf("decode %q: %v", line, err)
}
got = append(got, fields.Level)
}
for _, written := range levels[i:] {
want = append(want, written.String())
}
if !slices.Equal(got, want) {
t.Errorf("lines at %v, want %v", got, want)
}
})
}
}
+4 -7
View File
@@ -11,7 +11,7 @@ import (
func TestHistoryKeepsEveryRequest(t *testing.T) { func TestHistoryKeepsEveryRequest(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize}) limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -57,7 +57,7 @@ func TestHistoryKeepsEveryRequest(t *testing.T) {
func TestLookupReachesTheHistoryOfAClientInTheTable(t *testing.T) { func TestLookupReachesTheHistoryOfAClientInTheTable(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize}) limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
other := netip.MustParsePrefix("198.51.100.7/32") other := netip.MustParsePrefix("198.51.100.7/32")
start := midnight() start := midnight()
@@ -100,10 +100,7 @@ func TestLookupReachesTheHistoryOfAClientInTheTable(t *testing.T) {
func TestResetKeepsTheHistory(t *testing.T) { func TestResetKeepsTheHistory(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit}, tableSize)
Limits: ratelimit.Limits{PerMinute: limit},
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -126,7 +123,7 @@ func TestResetKeepsTheHistory(t *testing.T) {
func TestRequestsAddsUpTheClientsInsideTheNetblock(t *testing.T) { func TestRequestsAddsUpTheClientsInsideTheNetblock(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize}) limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
for client, requests := range map[string]int{ for client, requests := range map[string]int{
"198.51.100.9/32": 2, "198.51.100.9/32": 2,
+6 -18
View File
@@ -156,28 +156,16 @@ type Request struct {
BlockedByCoreRuleSet bool BlockedByCoreRuleSet bool
} }
// Params are what New needs. // New returns a Limiter for limits, with no client counted yet, whose
type Params struct { // table holds at most maxClients clients (SWW_WAF_MAX_TRACKED_CLIENTS). Past
// Limits are what each client's requests and bytes are counted against. // it, the least recently seen client is dropped, with its history, and
Limits Limits // starts afresh if it comes back.
// MaxClients is the most clients the table holds, func New(limits Limits, maxClients int) *Limiter {
// SWW_WAF_MAX_TRACKED_CLIENTS. Past it, the least recently seen client clients, err := simplelru.NewLRU[netip.Prefix, *Client](maxClients, nil)
// is dropped, with its history, and starts afresh if it comes back.
MaxClients int
}
// New returns a Limiter with no client counted yet.
func New(params Params) *Limiter {
clients, err := simplelru.NewLRU[netip.Prefix, *Client](
params.MaxClients,
nil,
)
if err != nil { if err != nil {
panic(err) // NewLRU fails only for a size below one panic(err) // NewLRU fails only for a size below one
} }
limits := params.Limits
return &Limiter{ return &Limiter{
windows: [3]window{ windows: [3]window{
{ {
+39 -63
View File
@@ -33,10 +33,10 @@ func TestOverARateLimitSaysWhatTheNextRequestWouldDoAndCountsNothing(
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
now := midnight() now := midnight()
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(
Limits: ratelimit.Limits{PerMinute: limit, PerDay: limit + 1}, ratelimit.Limits{PerMinute: limit, PerDay: limit + 1},
MaxClients: tableSize, tableSize,
}) )
if limiter.OverARateLimit(client, now) { if limiter.OverARateLimit(client, now) {
t.Error("a client never seen is over a rate limit") t.Error("a client never seen is over a rate limit")
@@ -72,7 +72,7 @@ func TestOverARateLimitSaysWhatTheNextRequestWouldDoAndCountsNothing(
wantCount(t, limiter, client, now, minute) wantCount(t, limiter, client, now, minute)
// A limit that is off is never gone over. // A limit that is off is never gone over.
off := ratelimit.New(ratelimit.Params{MaxClients: tableSize}) off := ratelimit.New(ratelimit.Limits{}, tableSize)
off.Count(client, now, whole) off.Count(client, now, whole)
if off.OverARateLimit(client, now) { if off.OverARateLimit(client, now) {
@@ -95,10 +95,7 @@ func TestEachWindowRefusesAtItsLimitAndLetsTheClientBack(t *testing.T) {
t.Run(tc.window, func(t *testing.T) { t.Run(tc.window, func(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(tc.limits, tableSize)
Limits: tc.limits,
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
quarter := tc.length / 4 quarter := tc.length / 4
@@ -129,10 +126,10 @@ func TestEachWindowRefusesAtItsLimitAndLetsTheClientBack(t *testing.T) {
func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) { func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(
Limits: ratelimit.Limits{PerMinute: limit, PerHour: limit}, ratelimit.Limits{PerMinute: limit, PerHour: limit},
MaxClients: tableSize, tableSize,
}) )
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -162,13 +159,10 @@ func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
func TestClientGetsItsPercentageOfEachLimitRoundedDown(t *testing.T) { func TestClientGetsItsPercentageOfEachLimitRoundedDown(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(
Limits: ratelimit.Limits{ ratelimit.Limits{PerMinute: 5, BytesPerDay: math.MaxInt64},
PerMinute: 5, tableSize,
BytesPerDay: math.MaxInt64, )
},
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -209,10 +203,10 @@ func TestZeroPercentIsAZeroAllowanceAndALimitOffStaysOff(t *testing.T) {
t.Parallel() t.Parallel()
// Only the hour has limits: the minute's and the day's are off. // Only the hour has limits: the minute's and the day's are off.
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(
Limits: ratelimit.Limits{PerHour: limit, BytesPerHour: 1000}, ratelimit.Limits{PerHour: limit, BytesPerHour: 1000},
MaxClients: tableSize, tableSize,
}) )
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -259,10 +253,7 @@ func TestEachByteLimitIsBrokenByTheBytesCounted(t *testing.T) {
t.Run(tc.window, func(t *testing.T) { t.Run(tc.window, func(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(tc.limits, tableSize)
Limits: tc.limits,
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
// 600 bytes are within the limit, 600 more over it. // 600 bytes are within the limit, 600 more over it.
@@ -296,10 +287,10 @@ func TestEachByteLimitIsBrokenByTheBytesCounted(t *testing.T) {
func TestALimitIsBrokenOnlyByWhatIsAddedToIt(t *testing.T) { func TestALimitIsBrokenOnlyByWhatIsAddedToIt(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(
Limits: ratelimit.Limits{PerMinute: 2, BytesPerMinute: 1000}, ratelimit.Limits{PerMinute: 2, BytesPerMinute: 1000},
MaxClients: tableSize, tableSize,
}) )
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
other := netip.MustParsePrefix("203.0.113.10/32") other := netip.MustParsePrefix("203.0.113.10/32")
start := midnight() start := midnight()
@@ -323,7 +314,7 @@ func TestALimitIsBrokenOnlyByWhatIsAddedToIt(t *testing.T) {
func TestCountGivesTheBytesInEachWindow(t *testing.T) { func TestCountGivesTheBytesInEachWindow(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize}) limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -349,10 +340,7 @@ func TestCountGivesTheBytesInEachWindow(t *testing.T) {
func TestResetSetsTheBytesBackToZero(t *testing.T) { func TestResetSetsTheBytesBackToZero(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(ratelimit.Limits{BytesPerDay: 1000}, tableSize)
Limits: ratelimit.Limits{BytesPerDay: 1000},
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -368,7 +356,7 @@ func TestRefusalsOverTheLimitInAMinuteBreakTheRefusalLimit(
) { ) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize}) limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -423,7 +411,7 @@ func TestRefusalsOverTheLimitInAMinuteBreakTheRefusalLimit(
func TestCountGivesTheRequestsInEachWindow(t *testing.T) { func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize}) limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -450,10 +438,10 @@ func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
func TestResetSetsTheCountsBackToZero(t *testing.T) { func TestResetSetsTheCountsBackToZero(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(
Limits: ratelimit.Limits{PerMinute: limit, PerDay: limit}, ratelimit.Limits{PerMinute: limit, PerDay: limit},
MaxClients: tableSize, tableSize,
}) )
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -475,10 +463,7 @@ func TestResetSetsTheCountsBackToZero(t *testing.T) {
func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) { func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize)
Limits: ratelimit.Limits{PerHour: limit},
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -497,10 +482,10 @@ func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) {
func TestRefusedRequestsCount(t *testing.T) { func TestRefusedRequestsCount(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(
Limits: ratelimit.Limits{PerMinute: limit, PerHour: 2 * limit}, ratelimit.Limits{PerMinute: limit, PerHour: 2 * limit},
MaxClients: tableSize, tableSize,
}) )
refused := netip.MustParsePrefix("203.0.113.9/32") refused := netip.MustParsePrefix("203.0.113.9/32")
within := netip.MustParsePrefix("203.0.113.10/32") within := netip.MustParsePrefix("203.0.113.10/32")
start := midnight() start := midnight()
@@ -533,10 +518,7 @@ func TestRefusedRequestsCount(t *testing.T) {
func TestRequestCountedLateGoesInTheBucketUnderWay(t *testing.T) { func TestRequestCountedLateGoesInTheBucketUnderWay(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit}, tableSize)
Limits: ratelimit.Limits{PerMinute: limit},
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -552,10 +534,7 @@ func TestRequestCountedLateGoesInTheBucketUnderWay(t *testing.T) {
func TestClockSetBackStartsTheBucketsAfresh(t *testing.T) { func TestClockSetBackStartsTheBucketsAfresh(t *testing.T) {
t.Parallel() t.Parallel()
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize)
Limits: ratelimit.Limits{PerHour: limit},
MaxClients: tableSize,
})
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
@@ -583,10 +562,7 @@ func TestKeepsAtMostMaxClientsDroppingTheLeastRecentlySeen(t *testing.T) {
const maxClients = 3 const maxClients = 3
limiter := ratelimit.New(ratelimit.Params{ limiter := ratelimit.New(ratelimit.Limits{PerMinute: 1}, maxClients)
Limits: ratelimit.Limits{PerMinute: 1},
MaxClients: maxClients,
})
now := midnight() now := midnight()
clients := make([]netip.Prefix, maxClients+1) clients := make([]netip.Prefix, maxClients+1)
+6 -12
View File
@@ -19,7 +19,7 @@ func TestSnapshotListsTheClientsByAddress(t *testing.T) {
"2001:db8::/64", "2001:db8::/64",
} }
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize}) limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
for _, i := range []int{2, 3, 0, 1} { for _, i := range []int{2, 3, 0, 1} {
limiter.Count(netip.MustParsePrefix(want[i]), midnight(), whole) limiter.Count(netip.MustParsePrefix(want[i]), midnight(), whole)
} }
@@ -52,10 +52,7 @@ func TestLoadedCountsCarryOn(t *testing.T) {
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
before := ratelimit.New(ratelimit.Params{ before := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize)
Limits: ratelimit.Limits{PerHour: limit},
MaxClients: tableSize,
})
for range limit { for range limit {
wantCount(t, before, client, start, "") wantCount(t, before, client, start, "")
} }
@@ -63,10 +60,7 @@ func TestLoadedCountsCarryOn(t *testing.T) {
// Loaded into a new limiter, as across a restart, the client has no // Loaded into a new limiter, as across a restart, the client has no
// fresh allowance. // fresh allowance.
later := start.Add(time.Minute) later := start.Add(time.Minute)
after := ratelimit.New(ratelimit.Params{ after := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize)
Limits: ratelimit.Limits{PerHour: limit},
MaxClients: tableSize,
})
after.Load(before.Snapshot(), later) after.Load(before.Snapshot(), later)
wantCount(t, after, client, later, hour) wantCount(t, after, client, later, hour)
} }
@@ -77,7 +71,7 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
client := netip.MustParsePrefix("203.0.113.9/32") client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight() start := midnight()
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize}) limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
limiter.Count(client, start, whole) limiter.Count(client, start, whole)
limiter.CountBytes(client, start, 5, whole) limiter.CountBytes(client, start, 5, whole)
limiter.CountRefusal(client, start, limit) limiter.CountRefusal(client, start, limit)
@@ -86,7 +80,7 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
loaded := func(now time.Time) ratelimit.Client { loaded := func(now time.Time) ratelimit.Client {
t.Helper() t.Helper()
after := ratelimit.New(ratelimit.Params{MaxClients: tableSize}) after := ratelimit.New(ratelimit.Limits{}, tableSize)
after.Load(limiter.Snapshot(), now) after.Load(limiter.Snapshot(), now)
return after.Snapshot()[0] return after.Snapshot()[0]
@@ -148,7 +142,7 @@ func TestLoadDropsTheLeastRecentlySeenFirst(t *testing.T) {
addr = addr.Next() addr = addr.Next()
} }
limiter := ratelimit.New(ratelimit.Params{MaxClients: maxClients}) limiter := ratelimit.New(ratelimit.Limits{}, maxClients)
limiter.Load(clients, midnight()) limiter.Load(clients, midnight())
got := limiter.Snapshot() got := limiter.Snapshot()
+33 -4
View File
@@ -1,7 +1,7 @@
// Package requestlog writes the request log sww prints on stdout: JSON // Package requestlog writes the lines sww prints on stdout: one
// lines marked "type":"request", at most two from waf and a short one // JSON object per request, two from waf and a short one from rproxy,
// from rproxy for each request. Package processlog writes the process's // marked "type":"request", and the process's own messages as JSON lines marked
// own messages. // "type":"process".
package requestlog package requestlog
import ( import (
@@ -9,6 +9,7 @@ import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"io" "io"
"log/slog"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
@@ -299,3 +300,31 @@ func Milliseconds(d time.Duration) float64 {
time.Millisecond/time.Microsecond, time.Millisecond/time.Microsecond,
) )
} }
// NewProcessLogger returns the logger for the process's own messages:
// JSON lines on w, marked "type":"process", with the time in the same form
// as a request line's, and instanceName, SWW_WAF_INSTANCE_NAME, as instance.
// It writes only the messages at level, SWW_WAF_LOG_LEVEL, or more severe;
// the request lines Write writes are never held back.
func NewProcessLogger(
w io.Writer,
instanceName string,
level slog.Level,
) *slog.Logger {
handler := slog.NewJSONHandler(w, &slog.HandlerOptions{
Level: level,
ReplaceAttr: func(groups []string, attr slog.Attr) slog.Attr {
if attr.Key == slog.TimeKey && len(groups) == 0 {
return slog.String(
slog.TimeKey,
FormatTime(attr.Value.Time()),
)
}
return attr
},
})
return slog.New(handler).
With("type", "process", "instance", instanceName)
}
+87
View File
@@ -3,6 +3,8 @@ package requestlog_test
import ( import (
"bytes" "bytes"
"encoding/json" "encoding/json"
"log/slog"
"slices"
"strings" "strings"
"testing" "testing"
"time" "time"
@@ -132,3 +134,88 @@ func TestWriteRProxyLineWritesOneJSONLineWithOnlyItsFields(t *testing.T) {
t.Errorf("wrote %s, want %s", out.String(), want) t.Errorf("wrote %s, want %s", out.String(), want)
} }
} }
func TestProcessLinesAreMarkedProcessAndGiveTheInstance(t *testing.T) {
t.Parallel()
var out bytes.Buffer
requestlog.NewProcessLogger(&out, "host1/blog", slog.LevelInfo).
Info("starting",
"version", "v1")
var fields map[string]any
err := json.Unmarshal(out.Bytes(), &fields)
if err != nil {
t.Fatalf("decode %q: %v", out.String(), err)
}
if fields["type"] != "process" ||
fields["instance"] != "host1/blog" ||
fields["msg"] != "starting" ||
fields["level"] != "INFO" ||
fields["version"] != "v1" {
t.Errorf("process line %v", fields)
}
timeText, _ := fields["time"].(string)
logged, err := time.Parse(time.RFC3339, timeText)
if err != nil || !strings.HasSuffix(timeText, "Z") ||
len(timeText) != len("2006-01-02T15:04:05.000Z") ||
time.Since(logged) > time.Minute {
t.Errorf(
"process line time %q, want now in UTC with milliseconds",
timeText,
)
}
}
func TestProcessLoggerWritesTheMessagesAtItsLevelOrMoreSevere(t *testing.T) {
t.Parallel()
levels := []slog.Level{
slog.LevelDebug, slog.LevelInfo, slog.LevelWarn, slog.LevelError,
}
for i, level := range levels {
t.Run(level.String(), func(t *testing.T) {
t.Parallel()
var out bytes.Buffer
processLog := requestlog.NewProcessLogger(
&out,
"host1/blog",
level,
)
for _, at := range levels {
processLog.Log(t.Context(), at, "message")
}
var got, want []string
for line := range strings.Lines(out.String()) {
var fields struct {
Level string `json:"level"`
}
err := json.Unmarshal([]byte(line), &fields)
if err != nil {
t.Fatalf("decode %q: %v", line, err)
}
got = append(got, fields.Level)
}
for _, written := range levels[i:] {
want = append(want, written.String())
}
if !slices.Equal(got, want) {
t.Errorf("lines at %v, want %v", got, want)
}
})
}
}
+7 -16
View File
@@ -2,7 +2,6 @@ package rproxy
import ( import (
"bufio" "bufio"
"context"
"errors" "errors"
"io" "io"
"net" "net"
@@ -23,9 +22,7 @@ type requestBody struct {
// body is the visitor's body, ending in an *http.MaxBytesError past // body is the visitor's body, ending in an *http.MaxBytesError past
// SWW_RPROXY_MAX_REQUEST_SIZE. // SWW_RPROXY_MAX_REQUEST_SIZE.
body io.ReadCloser body io.ReadCloser
// refuse refuses the request with a status, as the request's own refuse rq *request
// does.
refuse func(status int)
// held is the part read before the request goes to the app, which Read gives // held is the part read before the request goes to the app, which Read gives
// first. // first.
held []byte held []byte
@@ -71,7 +68,7 @@ func (b *requestBody) Read(p []byte) (int, error) {
case errors.Is(err, io.EOF): case errors.Is(err, io.EOF):
b.received.Store(true) b.received.Store(true)
case errors.As(err, &tooLarge): case errors.As(err, &tooLarge):
b.refuse(http.StatusRequestEntityTooLarge) b.rq.refuse(http.StatusRequestEntityTooLarge)
} }
return n, err return n, err
@@ -89,13 +86,7 @@ type responseBody struct {
// body is the app's body, ending in an *http.MaxBytesError past // body is the app's body, ending in an *http.MaxBytesError past
// SWW_RPROXY_MAX_RESPONSE_SIZE. // SWW_RPROXY_MAX_RESPONSE_SIZE.
body io.ReadCloser body io.ReadCloser
// refuse refuses the request with a status, as the request's own refuse rq *request
// does.
refuse func(status int)
// responseReceived is called once the app's whole answer has arrived.
responseReceived func()
// ctx is the visitor's request's context.
ctx context.Context //nolint:containedctx // the body ends with the request
} }
// Read reads from the app's body. // Read reads from the app's body.
@@ -109,15 +100,15 @@ func (b *responseBody) Read(p []byte) (int, error) {
switch { switch {
case errors.Is(err, io.EOF): case errors.Is(err, io.EOF):
b.responseReceived() b.rq.responseReceived()
case errors.As(err, &tooLarge): case errors.As(err, &tooLarge):
b.refuse(http.StatusBadGateway) b.rq.refuse(http.StatusBadGateway)
return n, errResponseTooLarge return n, errResponseTooLarge
case b.ctx.Err() == nil: case b.rq.in.Context().Err() == nil:
// The answer broke off, not because the visitor went away. If a time // The answer broke off, not because the visitor went away. If a time
// limit cut it, that refusal came first and is the one kept. // limit cut it, that refusal came first and is the one kept.
b.refuse(http.StatusBadGateway) b.rq.refuse(http.StatusBadGateway)
} }
return n, err return n, err
+10
View File
@@ -7,6 +7,16 @@ import (
"strings" "strings"
) )
// The headers rproxy sets on the requests it passes to the app. Go writes every
// header name in this form, as it sends it and as it receives it, so
// X-Client-ASN arrives as X-Client-Asn, and Del removes a visitor's own whatever
// their case; header names are not case-sensitive.
const (
requestIDHeader = "X-Request-ID"
asnHeader = "X-Client-Asn"
countryHeader = "X-Client-Country"
)
// peerAddress is the address of the request's TCP peer, normally traefik, without // peerAddress is the address of the request's TCP peer, normally traefik, without
// any IPv6 zone, which waf refuses. // any IPv6 zone, which waf refuses.
func peerAddress(r *http.Request) netip.Addr { func peerAddress(r *http.Request) netip.Addr {
+13 -22
View File
@@ -18,9 +18,9 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/config/configtest" "sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/processlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rproxy" "sneak.berlin/go/smallwebwaf/internal/rproxy"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/waf"
) )
// post is the start of a POST whose headers end with what follows it. // post is the start of a POST whose headers end with what follows it.
@@ -51,11 +51,11 @@ func TestServerHasTheFixedLimits(t *testing.T) {
server := rproxy.New(rproxy.Params{ server := rproxy.New(rproxy.Params{
Config: cfg, Config: cfg,
RequestLog: io.Discard, RequestLog: io.Discard,
ProcessLog: processlog.New(processlog.Params{ ProcessLog: requestlog.NewProcessLogger(
Output: io.Discard, io.Discard,
InstanceName: "test", "test",
Level: slog.LevelInfo, slog.LevelInfo,
}), ),
}) })
if server.Addr != ":8080" || server.MaxHeaderBytes != 28<<10 || if server.Addr != ":8080" || server.MaxHeaderBytes != 28<<10 ||
@@ -492,7 +492,7 @@ func TestASettingOfOffSetsNoLimit(t *testing.T) {
strings.NewReader("a body"))), strings.NewReader("a body"))),
) )
if got := get(t, addr, wafprotocol.HealthPath); got.status != http.StatusOK { if got := get(t, addr, waf.HealthPath); got.status != http.StatusOK {
t.Errorf("answered %+v, want rproxy's own 200", got) t.Errorf("answered %+v, want rproxy's own 200", got)
} }
}) })
@@ -870,14 +870,10 @@ func TestAnAnswerRProxyGivesItselfHasTenSecondsToBeSentAtLeast(
func TestAVisitorTooSlowToSendTheRestIsAnswered408(t *testing.T) { func TestAVisitorTooSlowToSendTheRestIsAnswered408(t *testing.T) {
t.Parallel() t.Parallel()
// rproxy is waiting for the rest of the body well within this limit, even on
// a busy host, and answers when it runs out, well before dial's deadline.
const limit = 5 * time.Second
standIn := startWAF(t, allow) standIn := startWAF(t, allow)
app := startApp(t, greet) app := startApp(t, greet)
addr, _ := startRProxy(t, standIn.URL, app.URL, map[string]string{ addr, _ := startRProxy(t, standIn.URL, app.URL, map[string]string{
maxRequestTime: strconv.FormatInt(limit.Milliseconds(), 10), maxRequestTime: "3000",
}) })
conn := dial(t, addr) conn := dial(t, addr)
@@ -1266,9 +1262,7 @@ func TestAVisitorThatStopsALittleShortIsReadUntilTheRequestTimeLimit(
) { ) {
t.Parallel() t.Parallel()
// The app's answer arrives well within this limit, even on a busy host, and const limit = 2 * time.Second
// the connection closes when it runs out, well before dial's deadline.
const limit = 5 * time.Second
standIn := startWAF(t, allow) standIn := startWAF(t, allow)
app := startApp(t, answerBeforeTheBody) app := startApp(t, answerBeforeTheBody)
@@ -1433,20 +1427,17 @@ func TestTheRequestTimeLimitDoesNotCutAnAnswerOnceTheAppHasTheRequest(
const answer = "the end of a long answer" const answer = "the end of a long answer"
// The request reaches the app well within this limit, even on a busy host.
const limit = 5 * time.Second
standIn := startWAF(t, allow) standIn := startWAF(t, allow)
app := startApp(t, func(w http.ResponseWriter, r *http.Request) { app := startApp(t, func(w http.ResponseWriter, r *http.Request) {
_, _ = io.Copy(io.Discard, r.Body) _, _ = io.Copy(io.Discard, r.Body)
_ = http.NewResponseController(w).Flush() _ = http.NewResponseController(w).Flush()
time.Sleep(limit + 100*time.Millisecond) time.Sleep(2100 * time.Millisecond)
_, _ = io.WriteString(w, answer) _, _ = io.WriteString(w, answer)
}) })
addr, _ := startRProxy(t, standIn.URL, app.URL, map[string]string{ addr, _ := startRProxy(t, standIn.URL, app.URL, map[string]string{
maxRequestTime: strconv.FormatInt(limit.Milliseconds(), 10), maxRequestTime: "2000",
}) })
got := visit( got := visit(
@@ -1480,7 +1471,7 @@ func TestAVisitorThatStopsALittleShortGetsAnAnswerRProxyGivesAtOnce(
{"a deny", post, http.StatusForbidden}, {"a deny", post, http.StatusForbidden},
{ {
"the health check", "the health check",
"GET " + wafprotocol.HealthPath + " HTTP/1.1\r\nHost: site.example\r\n", "GET " + waf.HealthPath + " HTTP/1.1\r\nHost: site.example\r\n",
http.StatusOK, http.StatusOK,
}, },
} { } {
+13 -13
View File
@@ -12,11 +12,11 @@ import (
"strings" "strings"
"testing" "testing"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/waf"
) )
// questionBodyMaxBytes is the most of a body a question carries. // questionBodyMaxBytes is the most of a body a question carries.
const questionBodyMaxBytes = wafprotocol.QuestionBodyMaxBytes const questionBodyMaxBytes = waf.QuestionBodyMaxBytes
// chunked is the encoding of a body whose length the visitor did not announce. // chunked is the encoding of a body whose length the visitor did not announce.
const chunked = "chunked" const chunked = "chunked"
@@ -42,9 +42,9 @@ func TestQuestionIsTheVisitorsRequestWithRProxysHeaders(t *testing.T) {
req.Header.Set(forwardedForHeader, "198.51.100.1") req.Header.Set(forwardedForHeader, "198.51.100.1")
req.Header.Set(forwardedProtoHeader, "gopher") req.Header.Set(forwardedProtoHeader, "gopher")
req.Header.Set(requestIDHeader, "the-visitor's") req.Header.Set(requestIDHeader, "the-visitor's")
req.Header.Set(wafprotocol.ClientHeader, "198.51.100.2") req.Header.Set(waf.ClientHeader, "198.51.100.2")
req.Header.Set(wafprotocol.PeerHeader, "198.51.100.3") req.Header.Set(waf.PeerHeader, "198.51.100.3")
req.Header.Set(wafprotocol.ContentLengthHeader, "12345") req.Header.Set(waf.ContentLengthHeader, "12345")
// Go's client then sends no User-Agent, so that one in the question would be // Go's client then sends no User-Agent, so that one in the question would be
// Go's own. // Go's own.
req.Header["User-Agent"] = nil req.Header["User-Agent"] = nil
@@ -73,8 +73,8 @@ func TestQuestionIsTheVisitorsRequestWithRProxysHeaders(t *testing.T) {
want.Set(forwardedForHeader, "198.51.100.1") want.Set(forwardedForHeader, "198.51.100.1")
want.Set(forwardedProtoHeader, "http") want.Set(forwardedProtoHeader, "http")
want.Set(requestIDHeader, asked.header.Get(requestIDHeader)) want.Set(requestIDHeader, asked.header.Get(requestIDHeader))
want.Set(wafprotocol.ClientHeader, localhost) want.Set(waf.ClientHeader, localhost)
want.Set(wafprotocol.PeerHeader, localhost) want.Set(waf.PeerHeader, localhost)
if !maps.EqualFunc(asked.header, want, slices.Equal) { if !maps.EqualFunc(asked.header, want, slices.Equal) {
t.Errorf( t.Errorf(
@@ -124,8 +124,8 @@ func TestQuestionNamesTheVisitorATrustedProxySpeaksFor(t *testing.T) {
wantGreeting(t, visit(t, req)) wantGreeting(t, visit(t, req))
asked := standIn.question(t) asked := standIn.question(t)
if asked.header.Get(wafprotocol.ClientHeader) != tc.client || if asked.header.Get(waf.ClientHeader) != tc.client ||
asked.header.Get(wafprotocol.PeerHeader) != localhost || asked.header.Get(waf.PeerHeader) != localhost ||
asked.header.Get(forwardedForHeader) != visitor || asked.header.Get(forwardedForHeader) != visitor ||
asked.header.Get(forwardedProtoHeader) != tc.proto { asked.header.Get(forwardedProtoHeader) != tc.proto {
t.Errorf("the question's headers are %v", asked.header) t.Errorf("the question's headers are %v", asked.header)
@@ -191,7 +191,7 @@ func TestQuestionNamesTheClientAsTheForwardedAddressesSay(t *testing.T) {
req.Header[forwardedForHeader] = tc.header req.Header[forwardedForHeader] = tc.header
wantGreeting(t, visit(t, req)) wantGreeting(t, visit(t, req))
if got := standIn.question(t).header.Get(wafprotocol.ClientHeader); got != if got := standIn.question(t).header.Get(waf.ClientHeader); got !=
tc.client { tc.client {
t.Errorf("the question names %s, want %s", got, tc.client) t.Errorf("the question names %s, want %s", got, tc.client)
} }
@@ -282,7 +282,7 @@ func wantQuestionBody(
) )
} }
values := asked.header.Values(wafprotocol.ContentLengthHeader) values := asked.header.Values(waf.ContentLengthHeader)
if announced { if announced {
if !slices.Equal(values, []string{strconv.Itoa(size)}) || if !slices.Equal(values, []string{strconv.Itoa(size)}) ||
@@ -586,8 +586,8 @@ func TestAQuestionNamesNoZone(t *testing.T) {
server.Handler.ServeHTTP(httptest.NewRecorder(), req) server.Handler.ServeHTTP(httptest.NewRecorder(), req)
asked := standIn.question(t) asked := standIn.question(t)
if asked.header.Get(wafprotocol.ClientHeader) != tc.client || if asked.header.Get(waf.ClientHeader) != tc.client ||
asked.header.Get(wafprotocol.PeerHeader) != tc.peer { asked.header.Get(waf.PeerHeader) != tc.peer {
t.Errorf("the question's headers are %v", asked.header) t.Errorf("the question's headers are %v", asked.header)
} }
+16 -19
View File
@@ -18,13 +18,17 @@ import (
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules" "sneak.berlin/go/smallwebwaf/internal/rules"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/waf"
) )
// flushAfterEachWrite has ReverseProxy pass on each part of the app's answer as // flushAfterEachWrite has ReverseProxy pass on each part of the app's answer as
// soon as it arrives. // soon as it arrives.
const flushAfterEachWrite time.Duration = -1 const flushAfterEachWrite time.Duration = -1
// reportStatusMax is the highest status waf takes in a report. An app can
// send a higher one, up to 999, and the visitor gets that as it is.
const reportStatusMax = 599
// plainText is the Content-Type of the answers rproxy gives itself. // plainText is the Content-Type of the answers rproxy gives itself.
const plainText = "text/plain; charset=utf-8" const plainText = "text/plain; charset=utf-8"
@@ -122,8 +126,8 @@ func (h *handler) newRequest(
if r.Body != http.NoBody { if r.Body != http.NoBody {
rq.body = &requestBody{ rq.body = &requestBody{
body: limitBody(r.Body, h.config.MaxRequestSize), body: limitBody(r.Body, h.config.MaxRequestSize),
refuse: rq.refuse, rq: rq,
} }
} }
@@ -150,7 +154,7 @@ func (rq *request) readFirstPartOfBody() bool {
_ = rq.rc.SetReadDeadline(rq.requestDeadline()) _ = rq.rc.SetReadDeadline(rq.requestDeadline())
held, err := io.ReadAll( held, err := io.ReadAll(
io.LimitReader(rq.body, wafprotocol.QuestionBodyMaxBytes), io.LimitReader(rq.body, waf.QuestionBodyMaxBytes),
) )
// From here on, the request timer ends a read of the rest of the body, so that // From here on, the request timer ends a read of the rest of the body, so that
// the answer can name the side rproxy was waiting on. // the answer can name the side rproxy was waiting on.
@@ -224,17 +228,16 @@ func (rq *request) rewrite(pr *httputil.ProxyRequest) {
setPathAsSent(pr.Out.URL, pr.In) setPathAsSent(pr.Out.URL, pr.In)
setForwardedHeaders(pr.In, pr.Out, rq.peer, rq.peerTrusted) setForwardedHeaders(pr.In, pr.Out, rq.peer, rq.peerTrusted)
pr.Out.Header.Set(wafprotocol.RequestIDHeader, rq.id) pr.Out.Header.Set(requestIDHeader, rq.id)
// Del removes a visitor's own, whatever their case. pr.Out.Header.Del(asnHeader)
pr.Out.Header.Del(wafprotocol.ASNHeader) pr.Out.Header.Del(countryHeader)
pr.Out.Header.Del(wafprotocol.CountryHeader)
if rq.asn != "" { if rq.asn != "" {
pr.Out.Header.Set(wafprotocol.ASNHeader, rq.asn) pr.Out.Header.Set(asnHeader, rq.asn)
} }
if rq.country != "" { if rq.country != "" {
pr.Out.Header.Set(wafprotocol.CountryHeader, rq.country) pr.Out.Header.Set(countryHeader, rq.country)
} }
} }
@@ -291,12 +294,7 @@ func (rq *request) modifyResponse(res *http.Response) error {
// the request, and SWW_RPROXY_MAX_RESPONSE_TIME_MS starts. // the request, and SWW_RPROXY_MAX_RESPONSE_TIME_MS starts.
rq.startResponseTimer(requestEnd) rq.startResponseTimer(requestEnd)
res.Body = &responseBody{ res.Body = &responseBody{body: limitBody(res.Body, maxBytes), rq: rq}
body: limitBody(res.Body, maxBytes),
refuse: rq.refuse,
responseReceived: rq.responseReceived,
ctx: rq.in.Context(),
}
rq.out.afterStatus = rq.setResponseDeadline rq.out.afterStatus = rq.setResponseDeadline
return nil return nil
@@ -472,10 +470,9 @@ func (rq *request) finish(ctx context.Context) {
return return
} }
// An app can send a status up to 999, and the visitor gets it as it is. report := &waf.Report{
report := &wafprotocol.Report{
RequestID: rq.id, RequestID: rq.id,
Status: min(rq.out.status, wafprotocol.ReportStatusMax), Status: min(rq.out.status, reportStatusMax),
RequestBytes: requestBytes, RequestBytes: requestBytes,
ResponseBytes: responseBytes, ResponseBytes: responseBytes,
DurationTotalMS: requestlog.Milliseconds(end.Sub(rq.start)), DurationTotalMS: requestlog.Milliseconds(end.Sub(rq.start)),
+23 -13
View File
@@ -20,10 +20,11 @@ import (
"net/url" "net/url"
"slices" "slices"
"strings" "strings"
"sync"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/waf"
) )
// How rproxy keeps connections to the app and to waf open between // How rproxy keeps connections to the app and to waf open between
@@ -46,6 +47,10 @@ const (
idleTimeout = 120 * time.Second idleTimeout = 120 * time.Second
) )
// ownEndpointPrefix starts the path of every request for sww itself, which
// never reaches the app.
const ownEndpointPrefix = "/_sww/"
// allowedMethods lists the methods rproxy passes on, as an Allow header gives // allowedMethods lists the methods rproxy passes on, as an Allow header gives
// them. It answers any other method 405 itself. // them. It answers any other method 405 itself.
// The WebDAV ones are for Nextcloud. // The WebDAV ones are for Nextcloud.
@@ -87,9 +92,7 @@ func New(params Params) *Server {
transport: newTransport(), transport: newTransport(),
wafTransport: newWAFTransport(), wafTransport: newWAFTransport(),
reportURL: (&url.URL{ reportURL: (&url.URL{
Scheme: backend.Scheme, Scheme: backend.Scheme, Host: backend.Host, Path: waf.ReportPath,
Host: backend.Host,
Path: wafprotocol.ReportPath,
}).String(), }).String(),
breakerClock: time.Now, breakerClock: time.Now,
} }
@@ -111,7 +114,11 @@ func New(params Params) *Server {
// which SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS bounds. Any report after this call // which SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS bounds. Any report after this call
// is dropped: that of a WebSocket that closes while rproxy stops, for one. // is dropped: that of a WebSocket that closes while rproxy stops, for one.
func (s *Server) WaitForReports() { func (s *Server) WaitForReports() {
s.h.reports.stopAndWait() s.h.reportsMu.Lock()
s.h.reportsStopped = true
s.h.reportsMu.Unlock()
s.h.reports.Wait()
} }
// handler is the proxy. It holds what every request shares; what belongs to // handler is the proxy. It holds what every request shares; what belongs to
@@ -131,8 +138,11 @@ type handler struct {
// the time. // the time.
breaker breaker breaker breaker
breakerClock func() time.Time breakerClock func() time.Time
// reports are the goroutines sending reports to waf. // reports waits for the reports being sent. reportsMu guards starting one
reports reportGoroutines // against reportsStopped, which is set once nothing is to wait for another.
reports sync.WaitGroup
reportsMu sync.Mutex
reportsStopped bool
} }
// newTransport returns what carries requests to the app. It never goes through // newTransport returns what carries requests to the app. It never goes through
@@ -169,7 +179,7 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
defer rq.readRestOfBody() defer rq.readRestOfBody()
defer rq.finish(r.Context()) defer rq.finish(r.Context())
if r.Method == http.MethodGet && r.URL.Path == wafprotocol.HealthPath { if r.Method == http.MethodGet && r.URL.Path == waf.HealthPath {
rq.answerHealth() rq.answerHealth()
return return
@@ -202,9 +212,9 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
rq.noteAnswer(answer) rq.noteAnswer(answer)
switch rq.disposition { switch rq.disposition {
case wafprotocol.DispositionAllow: case waf.DispositionAllow:
rq.passToApp(r.Context()) rq.passToApp(r.Context())
case wafprotocol.DispositionClose: case waf.DispositionClose:
// This panic has Go's server close the connection without an answer, // This panic has Go's server close the connection without an answer,
// and log nothing; the deferred finish still runs. // and log nothing; the deferred finish still runs.
panic(http.ErrAbortHandler) panic(http.ErrAbortHandler)
@@ -214,8 +224,8 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
} }
// passToApp passes the request on to the app, unless a limit or its path keeps // passToApp passes the request on to the app, unless a limit or its path keeps
// it from the app. waf answers a path under wafprotocol.OwnEndpointPrefix // it from the app. waf answers a path under ownEndpointPrefix itself, so
// itself, so such a request gets here only when // such a request gets here only when
// SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION allowed it. // SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION allowed it.
// ctx is the request's own context. // ctx is the request's own context.
func (rq *request) passToApp(ctx context.Context) { func (rq *request) passToApp(ctx context.Context) {
@@ -224,7 +234,7 @@ func (rq *request) passToApp(ctx context.Context) {
switch { switch {
case rq.refused.Load() != 0 || (maxBytes > 0 && rq.in.ContentLength > maxBytes): case rq.refused.Load() != 0 || (maxBytes > 0 && rq.in.ContentLength > maxBytes):
rq.answer(http.StatusRequestEntityTooLarge) rq.answer(http.StatusRequestEntityTooLarge)
case strings.HasPrefix(rq.in.URL.Path, wafprotocol.OwnEndpointPrefix): case strings.HasPrefix(rq.in.URL.Path, ownEndpointPrefix):
rq.answer(http.StatusNotFound) rq.answer(http.StatusNotFound)
default: default:
rq.forward(ctx) rq.forward(ctx)
+34 -45
View File
@@ -21,9 +21,9 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/config/configtest" "sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/processlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rproxy" "sneak.berlin/go/smallwebwaf/internal/rproxy"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/waf"
) )
const ( const (
@@ -56,7 +56,7 @@ const (
// The headers the tests set and look at. // The headers the tests set and look at.
const ( const (
dispositionHeader = wafprotocol.DispositionHeader dispositionHeader = waf.DispositionHeader
forwardedForHeader = "X-Forwarded-For" forwardedForHeader = "X-Forwarded-For"
forwardedProtoHeader = "X-Forwarded-Proto" forwardedProtoHeader = "X-Forwarded-Proto"
requestIDHeader = "X-Request-ID" requestIDHeader = "X-Request-ID"
@@ -72,9 +72,9 @@ const (
// What the stand-in for waf answers. // What the stand-in for waf answers.
const ( const (
dispositionAllow = wafprotocol.DispositionAllow dispositionAllow = waf.DispositionAllow
dispositionDeny = wafprotocol.DispositionDeny dispositionDeny = waf.DispositionDeny
dispositionClose = wafprotocol.DispositionClose dispositionClose = waf.DispositionClose
// deniedBody is the body of a deny. // deniedBody is the body of a deny.
deniedBody = "refused by waf" deniedBody = "refused by waf"
) )
@@ -175,7 +175,7 @@ type WAF struct {
*httptest.Server *httptest.Server
questions chan question questions chan question
reports chan wafprotocol.Report reports chan waf.Report
} }
// startWAF starts a stand-in for waf. // startWAF starts a stand-in for waf.
@@ -184,11 +184,11 @@ func startWAF(t *testing.T, answer http.HandlerFunc) *WAF {
s := &WAF{ s := &WAF{
questions: make(chan question, 10), questions: make(chan question, 10),
reports: make(chan wafprotocol.Report, 10), reports: make(chan waf.Report, 10),
} }
s.Server = httptest.NewServer(http.HandlerFunc( s.Server = httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) { func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(wafprotocol.ClientHeader) == "" { if r.Header.Get(waf.ClientHeader) == "" {
s.takeReport(t, w, r) s.takeReport(t, w, r)
return return
@@ -217,11 +217,11 @@ func (s *WAF) takeReport(
) { ) {
t.Helper() t.Helper()
var report wafprotocol.Report var report waf.Report
err := json.NewDecoder(r.Body).Decode(&report) err := json.NewDecoder(r.Body).Decode(&report)
if err != nil || r.Method != http.MethodPost || if err != nil || r.Method != http.MethodPost ||
r.URL.Path != wafprotocol.ReportPath { r.URL.Path != waf.ReportPath {
t.Errorf("%s %s is no report: %v", r.Method, r.URL.Path, err) t.Errorf("%s %s is no report: %v", r.Method, r.URL.Path, err)
} }
@@ -242,7 +242,7 @@ func (s *WAF) wantNoReport(t *testing.T) {
} }
// report waits for a report. // report waits for a report.
func (s *WAF) report(t *testing.T) wafprotocol.Report { func (s *WAF) report(t *testing.T) waf.Report {
t.Helper() t.Helper()
select { select {
@@ -251,7 +251,7 @@ func (s *WAF) report(t *testing.T) wafprotocol.Report {
case <-time.After(waitLimit): case <-time.After(waitLimit):
t.Fatal("no report") t.Fatal("no report")
return wafprotocol.Report{} return waf.Report{}
} }
} }
@@ -279,7 +279,7 @@ func startWAFAnsweringReports(
backend := httptest.NewServer(http.HandlerFunc( backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) { func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(wafprotocol.ClientHeader) != "" { if r.Header.Get(waf.ClientHeader) != "" {
answer(w, r) answer(w, r)
return return
@@ -383,11 +383,11 @@ func newServer(
return rproxy.New(rproxy.Params{ return rproxy.New(rproxy.Params{
Config: cfg, Config: cfg,
RequestLog: requestLog, RequestLog: requestLog,
ProcessLog: processlog.New(processlog.Params{ ProcessLog: requestlog.NewProcessLogger(
Output: processLog, processLog,
InstanceName: "test", "test",
Level: slog.LevelInfo, slog.LevelInfo,
}), ),
}) })
} }
@@ -829,7 +829,7 @@ func TestAnAnswerRProxyHoldsWholeIsSentBeforeItsLogLine(t *testing.T) {
}{ }{
{"a deny", "GET /" + host, deniedBody, http.StatusForbidden}, {"a deny", "GET /" + host, deniedBody, http.StatusForbidden},
{"rproxy's own answer", "GET x:admin" + host, "Bad Request\n", http.StatusBadRequest}, {"rproxy's own answer", "GET x:admin" + host, "Bad Request\n", http.StatusBadRequest},
{"the health check", "GET " + wafprotocol.HealthPath + host, "ok\n", http.StatusOK}, {"the health check", "GET " + waf.HealthPath + host, "ok\n", http.StatusOK},
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
t.Parallel() t.Parallel()
@@ -1452,18 +1452,14 @@ func TestFiveFailedAsksKeepTheNextRequestFromAskingWAF(t *testing.T) {
greet(w, r) greet(w, r)
}) })
out := &output{} addr, out := startRProxy(
server := newServer( t,
t, standIn.URL, app.URL, standIn.URL,
map[string]string{wafBackendTimeout: "200"}, app.URL,
out, out, map[string]string{
wafBackendTimeout: "200",
},
) )
// The breaker's clock stands still, as in
// TestAnswersThatCannotBePassedOnOpenTheBreaker: on a slow host, the five
// failures could take more than five seconds, and the next request could
// come more than a second later.
server.StopBreakerClock(time.Now())
addr := serve(t, server)
// An ask can time out before it reaches the stand-in, so these // An ask can time out before it reaches the stand-in, so these
// may leave fewer than five questions, or leave some late. // may leave fewer than five questions, or leave some late.
@@ -1602,19 +1598,19 @@ func TestHealthCheckIsAnsweredWhileWAFIsDown(t *testing.T) {
app := startApp(t, greet) app := startApp(t, greet)
addr, out := startRProxy(t, nowhere, app.URL, nil) addr, out := startRProxy(t, nowhere, app.URL, nil)
got := get(t, addr, wafprotocol.HealthPath) got := get(t, addr, waf.HealthPath)
if got.status != http.StatusOK || got.body != "ok\n" { if got.status != http.StatusOK || got.body != "ok\n" {
t.Errorf("answered %+v, want ok", got) t.Errorf("answered %+v, want ok", got)
} }
line := out.requestLine(t) line := out.requestLine(t)
if line["path"] != wafprotocol.HealthPath || line["disposition"] != nil { if line["path"] != waf.HealthPath || line["disposition"] != nil {
t.Errorf("line %v, want one with no disposition", line) t.Errorf("line %v, want one with no disposition", line)
} }
// With waf up too, it is not asked. // With waf up too, it is not asked.
addr, _ = startRProxy(t, standIn.URL, app.URL, nil) addr, _ = startRProxy(t, standIn.URL, app.URL, nil)
get(t, addr, wafprotocol.HealthPath) get(t, addr, waf.HealthPath)
if len(standIn.questions) != 0 { if len(standIn.questions) != 0 {
t.Error("waf was asked about the health check") t.Error("waf was asked about the health check")
@@ -1775,7 +1771,7 @@ func TestWaitForReportsWaitsForAReportBeingSent(t *testing.T) {
release := make(chan struct{}) release := make(chan struct{})
backend := httptest.NewServer(http.HandlerFunc( backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) { func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(wafprotocol.ClientHeader) != "" { if r.Header.Get(waf.ClientHeader) != "" {
allow(w, r) allow(w, r)
return return
@@ -1835,15 +1831,11 @@ func TestWaitForReportsWaitsForAReportBeingSent(t *testing.T) {
func TestReportOfAWebSocketCountsWhatItCarried(t *testing.T) { func TestReportOfAWebSocketCountsWhatItCarried(t *testing.T) {
t.Parallel() t.Parallel()
// The app's answer to the upgrade arrives well within these limits, even on a
// busy host.
const limit = 5 * time.Second
standIn := startWAF(t, allow) standIn := startWAF(t, allow)
app := startApp(t, echoAfterUpgrade) app := startApp(t, echoAfterUpgrade)
addr, out := startRProxy(t, standIn.URL, app.URL, map[string]string{ addr, out := startRProxy(t, standIn.URL, app.URL, map[string]string{
maxRequestTime: strconv.FormatInt(limit.Milliseconds(), 10), maxRequestTime: "2000",
maxResponseTime: strconv.FormatInt(limit.Milliseconds(), 10), maxResponseTime: "2000",
}) })
conn := dial(t, addr) conn := dial(t, addr)
@@ -1868,10 +1860,7 @@ func TestReportOfAWebSocketCountsWhatItCarried(t *testing.T) {
} }
// Past both time limits, the connection still carries what is sent. // Past both time limits, the connection still carries what is sent.
time.Sleep(limit + 100*time.Millisecond) time.Sleep(2100 * time.Millisecond)
// dial's deadline ran from before the upgrade and this wait.
_ = conn.SetDeadline(time.Now().Add(waitLimit))
const message = "still here\n" const message = "still here\n"
+90 -67
View File
@@ -1,17 +1,17 @@
package rproxy package rproxy
import ( import (
"bytes"
"context" "context"
"encoding/json"
"errors" "errors"
"fmt" "fmt"
"io" "io"
"maps" "maps"
"net/http" "net/http"
"strings" "strings"
"sync"
"sneak.berlin/go/smallwebwaf/internal/rules" "sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// answerBodyMaxBytes is the most of the body of waf's answer that is read. // answerBodyMaxBytes is the most of the body of waf's answer that is read.
@@ -40,12 +40,15 @@ type wafAnswer struct {
// notes nothing, and no report is due for it: waf holds no decision to report // notes nothing, and no report is due for it: waf holds no decision to report
// on. // on.
func (rq *request) noteAnswer(answer *wafAnswer) { func (rq *request) noteAnswer(answer *wafAnswer) {
read, valid := wafprotocol.ReadAnswer(answer.header) disposition := answer.header.Get(waf.DispositionHeader)
if valid {
rq.disposition = read.Disposition switch disposition {
case waf.DispositionAllow, waf.DispositionDeny,
waf.DispositionClose:
rq.disposition = disposition
rq.answered = true rq.answered = true
rq.asn = read.ASN rq.asn = answer.header.Get(asnHeader)
rq.country = read.Country rq.country = answer.header.Get(countryHeader)
} }
} }
@@ -76,7 +79,7 @@ func (rq *request) handleNoUsableAnswer(ctx context.Context, err error) {
rq.disposition = rq.h.config.WAFBackendNoAnswerDisposition rq.disposition = rq.h.config.WAFBackendNoAnswerDisposition
rq.wafNoAnswer = true rq.wafNoAnswer = true
if rq.disposition == wafprotocol.DispositionAllow { if rq.disposition == waf.DispositionAllow {
rq.passToApp(ctx) rq.passToApp(ctx)
} else { } else {
rq.answer(http.StatusServiceUnavailable) rq.answer(http.StatusServiceUnavailable)
@@ -116,21 +119,7 @@ func (rq *request) ask(ctx context.Context) (*wafAnswer, error) {
ctx, cancel := context.WithTimeout(ctx, rq.h.config.WAFBackendTimeout) ctx, cancel := context.WithTimeout(ctx, rq.h.config.WAFBackendTimeout)
defer cancel() defer cancel()
var held []byte question, err := rq.newQuestion(ctx)
if rq.body != nil {
held = rq.body.held
}
question, err := wafprotocol.NewQuestion(ctx, wafprotocol.QuestionParams{
Visitor: rq.in,
PathAsSent: rules.Path(rq.in),
Body: held,
WAFBackend: rq.h.config.WAFBackend,
Client: rq.client,
Peer: rq.peer,
RequestID: rq.id,
Scheme: scheme(rq.in, rq.peerTrusted),
})
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -187,13 +176,69 @@ func canBeAsked(r *http.Request) bool {
len(r.Header.Values("User-Agent")) <= 1 len(r.Header.Values("User-Agent")) <= 1
} }
// newQuestion returns the question about the request, as "The question" in
// SPEC.md gives it: the visitor's request as the visitor sent it, with the part
// of the body it holds and rproxy's headers.
func (rq *request) newQuestion(ctx context.Context) (*http.Request, error) {
in := rq.in
var held []byte
if rq.body != nil {
held = rq.body.held
}
question, err := http.NewRequestWithContext( //nolint:gosec // a setting's URL
ctx,
in.Method,
rq.h.config.WAFBackend.String(),
bytes.NewReader(held),
)
if err != nil {
return nil, fmt.Errorf("make the question: %w", err)
}
backend := question.URL
target := *in.URL
target.Scheme = backend.Scheme
target.Host = backend.Host
setPathAsSent(&target, in)
question.URL = &target
question.Host = in.Host
// The length NewRequestWithContext set is that of what is held. This line makes
// a body whose length the visitor did not announce go chunked, as "The question"
// in SPEC.md requires.
question.TransferEncoding = in.TransferEncoding
question.Header = in.Header.Clone()
question.Header.Set(waf.ClientHeader, rq.client.String())
question.Header.Set(waf.PeerHeader, rq.peer.String())
question.Header.Del(waf.ContentLengthHeader)
announced := in.Header.Get("Content-Length")
if announced != "" {
question.Header.Set(waf.ContentLengthHeader, announced)
}
question.Header.Set(requestIDHeader, rq.id)
question.Header.Set("X-Forwarded-Proto", scheme(in, rq.peerTrusted))
// Go's client adds a User-Agent of its own to a request without the key.
_, sent := question.Header["User-Agent"]
if !sent {
question.Header["User-Agent"] = nil
}
return question, nil
}
// answerWith gives the visitor waf's answer, less its disposition. // answerWith gives the visitor waf's answer, less its disposition.
func (rq *request) answerWith(answer *wafAnswer) { func (rq *request) answerWith(answer *wafAnswer) {
rq.prepareOwnAnswer(rq.requestDeadline()) rq.prepareOwnAnswer(rq.requestDeadline())
header := rq.out.Header() header := rq.out.Header()
maps.Copy(header, answer.header) maps.Copy(header, answer.header)
header.Del(wafprotocol.DispositionHeader) header.Del(waf.DispositionHeader)
// Go's server adds a Content-Type it guesses from the body to a response whose // Go's server adds a Content-Type it guesses from the body to a response whose
// headers have no Content-Type key. The key with no value sends none. // headers have no Content-Type key. The key with no value sends none.
@@ -207,11 +252,15 @@ func (rq *request) answerWith(answer *wafAnswer) {
// sendReport sends waf report on a goroutine of its own, so that the // sendReport sends waf report on a goroutine of its own, so that the
// visitor does not wait for it. The report goes on after ctx, the request's own // visitor does not wait for it. The report goes on after ctx, the request's own
// context, has ended. // context, has ended.
func (h *handler) sendReport( func (h *handler) sendReport(ctx context.Context, report *waf.Report) {
ctx context.Context, h.reportsMu.Lock()
report *wafprotocol.Report, defer h.reportsMu.Unlock()
) {
h.reports.startUnlessStopped(func() { if h.reportsStopped {
return
}
h.reports.Go(func() {
err := h.postReport(context.WithoutCancel(ctx), report) err := h.postReport(context.WithoutCancel(ctx), report)
if err != nil { if err != nil {
h.processLog.Warn("a report is dropped", "error", err.Error()) h.processLog.Warn("a report is dropped", "error", err.Error())
@@ -219,51 +268,25 @@ func (h *handler) sendReport(
}) })
} }
// reportGoroutines are the goroutines sending reports to waf. None starts once
// stopAndWait has been called.
type reportGoroutines struct {
mu sync.Mutex
stopped bool
running sync.WaitGroup
}
// startUnlessStopped runs send on a goroutine of its own, or drops it once
// stopAndWait has been called.
func (g *reportGoroutines) startUnlessStopped(send func()) {
g.mu.Lock()
defer g.mu.Unlock()
if g.stopped {
return
}
g.running.Go(send)
}
// stopAndWait stops any more goroutines from starting, and returns once those
// running have ended.
func (g *reportGoroutines) stopAndWait() {
g.mu.Lock()
g.stopped = true
g.mu.Unlock()
g.running.Wait()
}
// postReport posts report to waf, and waits for its answer for up to // postReport posts report to waf, and waits for its answer for up to
// SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS. // SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS.
func (h *handler) postReport( func (h *handler) postReport(ctx context.Context, report *waf.Report) error {
ctx context.Context, body, err := json.Marshal(report)
report *wafprotocol.Report, if err != nil {
) error { return fmt.Errorf("encode the report: %w", err)
}
ctx, cancel := context.WithTimeout(ctx, h.config.WAFBackendTimeout) ctx, cancel := context.WithTimeout(ctx, h.config.WAFBackendTimeout)
defer cancel() defer cancel()
req, err := wafprotocol.NewReportRequest(ctx, h.reportURL, report) req, err := http.NewRequestWithContext( //nolint:gosec // a setting's URL
ctx, http.MethodPost, h.reportURL, bytes.NewReader(body))
if err != nil { if err != nil {
return err return fmt.Errorf("make the request: %w", err)
} }
req.Header.Set("Content-Type", "application/json")
res, err := h.wafTransport.RoundTrip(req) res, err := h.wafTransport.RoundTrip(req)
if err != nil { if err != nil {
return fmt.Errorf("send the report: %w", err) return fmt.Errorf("send the report: %w", err)
+48 -19
View File
@@ -18,11 +18,12 @@ import (
"slices" "slices"
"strings" "strings"
"sync/atomic" "sync/atomic"
"time"
"github.com/fsnotify/fsnotify"
"golang.org/x/net/http/httpguts" "golang.org/x/net/http/httpguts"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/dirwatch"
) )
// The actions a rule takes when it matches. // The actions a rule takes when it matches.
@@ -39,6 +40,12 @@ const (
// extension ends the name of every rule file. // extension ends the name of every rule file.
const extension = ".rules" const extension = ".rules"
// quietTime is how long SWW_WAF_RULE_FILES_DIR must go without a change before
// the rule files are read again, so that a file still being written, such
// as one saved in place, appended to or copied in with scp, is read only
// once whole.
const quietTime = 2 * time.Second
// headerTarget starts the target that is one request header, // headerTarget starts the target that is one request header,
// header:<Name>. // header:<Name>.
const headerTarget = "header:" const headerTarget = "header:"
@@ -169,18 +176,26 @@ func (f *Files) Len() int {
} }
// Watch watches Dir until ctx is done, and reads the rule files again // Watch watches Dir until ctx is done, and reads the rule files again
// once Dir has had no change for a short time, after one is edited, added // once Dir has had no change for quietTime, after one is edited, added or
// or removed, and after Watch starts watching, as dirwatch describes. If // removed, and after Watch starts watching. If they then hold an error,
// they then hold an error, the rules stay as they were, the error is // the rules stay as they were, the error is logged with its file and
// logged with its file and line, and the files are read again after the // line, and the files are read again after the next change. If Dir cannot
// next change. If Dir cannot be watched, that is logged, and the rules // be watched, that is logged, and the rules stay as they were loaded.
// stay as they were loaded. While Enabled is false, Watch returns at once. // While Enabled is false, Watch returns at once.
func (f *Files) Watch(ctx context.Context) { func (f *Files) Watch(ctx context.Context) {
if !f.params.Enabled { if !f.params.Enabled {
return return
} }
watcher, err := dirwatch.New(f.params.Dir) watcher, err := fsnotify.NewWatcher()
if err == nil {
defer func() {
_ = watcher.Close()
}()
err = watcher.Add(f.params.Dir)
}
if err != nil { if err != nil {
f.params.ProcessLog.Error("cannot watch the rule files for edits", f.params.ProcessLog.Error("cannot watch the rule files for edits",
"error", err.Error()) "error", err.Error())
@@ -188,22 +203,36 @@ func (f *Files) Watch(ctx context.Context) {
return return
} }
defer func() {
_ = watcher.Close()
}()
f.params.ProcessLog.Info("watching the rule files for edits", f.params.ProcessLog.Info("watching the rule files for edits",
"directory", f.params.Dir) "directory", f.params.Dir)
// A change to any file in Dir counts. f.readAfterChanges(ctx, watcher.Events, watcher.Errors)
watcher.CallWhenQuiet(ctx, }
func(string) bool { return true },
f.readAgain, // readAfterChanges reads the rule files again once quietTime has passed
func(err error) { // without a change from events, until ctx is done, and logs the errors
// from errs. The wait starts at once, as if for a change, so that an edit
// saved after Load read the files, and before Dir was watched, is taken
// in too.
func (f *Files) readAfterChanges(
ctx context.Context, events <-chan fsnotify.Event, errs <-chan error,
) {
quiet := time.NewTimer(quietTime)
defer quiet.Stop()
for {
select {
case <-ctx.Done():
return
case <-events:
quiet.Reset(quietTime)
case <-quiet.C:
f.readAgain()
case err := <-errs:
f.params.ProcessLog.Warn("watching the rule files failed", f.params.ProcessLog.Warn("watching the rule files failed",
"error", err.Error()) "error", err.Error())
}, }
) }
} }
// readAgain reads the rule files again, in place of the rules loaded, or // readAgain reads the rule files again, in place of the rules loaded, or
+165
View File
@@ -0,0 +1,165 @@
package rules
import (
"context"
"log/slog"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"slices"
"testing"
"testing/synctest"
"time"
"github.com/fsnotify/fsnotify"
"sneak.berlin/go/smallwebwaf/internal/alerts"
)
// The tests below run readAfterChanges in a synctest bubble, where time is
// a clock of the test's own: time.Sleep moves it on at once, and
// synctest.Wait returns once readAfterChanges waits again, so that every
// reading due by then is done. The test sends the changes itself, as the
// watch of a directory cannot run in a bubble.
func TestFileWrittenInTwoPartsTakenInOnlyWhole(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "50-app.rules")
writeFile(t, path, "first path block ^/first\n")
files := load(t, dir)
changes := run(t, files)
file, err := os.Create(path) //nolint:gosec // a file the test wrote
if err != nil {
t.Fatalf("create: %v", err)
}
defer func() {
_ = file.Close()
}()
// The first part ends in the middle of a ban rule's regex, which,
// read then, would ban every request.
write(t, file, "first path block ^/first\nprobe path ban ^/")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantMatched(t, files, "/anything")
// The second part starts the wait again.
write(t, file, `\.env$`+"\n")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantMatched(t, files, "/.env")
time.Sleep(time.Nanosecond)
synctest.Wait()
wantMatched(t, files, "/.env", "probe")
wantMatched(t, files, "/anything")
})
}
func TestEditSavedBeforeTheWatchStartsTakenIn(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "50-app.rules")
writeFile(t, path, "first path block ^/first\n")
files := load(t, dir)
// Saved after Load read the files, and before the directory was
// watched, so that no change is seen for it.
writeFile(t, path, "first path block ^/edited\n")
run(t, files)
time.Sleep(quietTime)
synctest.Wait()
wantMatched(t, files, "/edited", "first")
})
}
// load loads the rules in dir.
func load(t *testing.T, dir string) *Files {
t.Helper()
files, err := Load(Params{
Dir: dir, Enabled: true, ProcessLog: slog.New(slog.DiscardHandler),
Alerts: alerts.New(alerts.Params{}),
})
if err != nil {
t.Fatalf("load: %v", err)
}
return files
}
// run runs files' readAfterChanges until the test ends, and returns the
// channel that sends it changes.
func run(t *testing.T, files *Files) chan<- fsnotify.Event {
t.Helper()
changes := make(chan fsnotify.Event)
ctx, stop := context.WithCancel(t.Context())
stopped := make(chan struct{})
go func() {
files.readAfterChanges(ctx, changes, nil)
close(stopped)
}()
t.Cleanup(func() {
stop()
<-stopped
})
return changes
}
// writeFile writes content to the file at path.
func writeFile(t *testing.T, path, content string) {
t.Helper()
err := os.WriteFile(path, []byte(content), 0o600)
if err != nil {
t.Fatalf("write %s: %v", path, err)
}
}
// write writes text to the end of file.
func write(t *testing.T, file *os.File, text string) {
t.Helper()
_, err := file.WriteString(text)
if err != nil {
t.Fatalf("write: %v", err)
}
}
// wantMatched checks the ids of the rules that a GET request for path
// matches, in order.
func wantMatched(t *testing.T, files *Files, path string, want ...string) {
t.Helper()
r := httptest.NewRequestWithContext(t.Context(), http.MethodGet,
"http://app.example"+path, nil)
matched := files.Match(r)
got := make([]string, 0, len(matched))
for _, rule := range matched {
got = append(got, rule.ID)
}
if !slices.Equal(got, want) {
t.Errorf("%s matched %v, want %v", path, got, want)
}
}
+1 -1
View File
@@ -2452,7 +2452,7 @@ func newParams(dir string) state.Params {
AttackBanDuration: 7 * 24 * time.Hour, AttackBanDuration: 7 * 24 * time.Hour,
MaxBans: 5000, MaxBans: 5000,
}), }),
Limiter: ratelimit.New(ratelimit.Params{MaxClients: 20000}), Limiter: ratelimit.New(ratelimit.Limits{}, 20000),
GeoJS: geojs.New(geojs.Params{Now: midnight}), GeoJS: geojs.New(geojs.Params{Now: midnight}),
Lists: reputation.New(reputation.Params{ Lists: reputation.New(reputation.Params{
BlocklistURLs: []string{ BlocklistURLs: []string{
+2 -2
View File
@@ -13,7 +13,7 @@ import (
"github.com/spf13/viper" "github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/waf"
) )
// healthCheckTimeout bounds the whole health check. // healthCheckTimeout bounds the whole health check.
@@ -116,7 +116,7 @@ func askHealthEndpoint(ctx context.Context, listenAddr string) error {
health := "http://" + net.JoinHostPort( health := "http://" + net.JoinHostPort(
"127.0.0.1", "127.0.0.1",
port, port,
) + wafprotocol.HealthPath ) + waf.HealthPath
req, err := http.NewRequestWithContext( req, err := http.NewRequestWithContext(
ctx, ctx,
+6 -6
View File
@@ -10,7 +10,7 @@ import (
"go.uber.org/fx" "go.uber.org/fx"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/processlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rproxy" "sneak.berlin/go/smallwebwaf/internal/rproxy"
) )
@@ -23,11 +23,11 @@ import (
// setting for either. // setting for either.
func RunRProxy(ctx context.Context, params Params) int { func RunRProxy(ctx context.Context, params Params) int {
hostname, _ := os.Hostname() // "" when the host has no name to give hostname, _ := os.Hostname() // "" when the host has no name to give
processLog := processlog.New(processlog.Params{ processLog := requestlog.NewProcessLogger(
Output: params.Stdout, params.Stdout,
InstanceName: hostname, hostname,
Level: slog.LevelInfo, slog.LevelInfo,
}) )
cfg, err := config.RProxyFromEnvironment(params.Viper) cfg, err := config.RProxyFromEnvironment(params.Viper)
if err != nil { if err != nil {
+17 -17
View File
@@ -21,7 +21,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config/configtest" "sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/sww" "sneak.berlin/go/smallwebwaf/internal/sww"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/waf"
) )
const ( const (
@@ -164,7 +164,7 @@ func TestABanThroughOneSiteRefusesTheClientOnTheOther(t *testing.T) {
// One request's three lines carry one id, rproxy's with waf's answer. // One request's three lines carry one id, rproxy's with waf's answer.
line := firstOut.line(t, "path", bannedProbe) line := firstOut.line(t, "path", bannedProbe)
if line["disposition"] != wafprotocol.DispositionDeny || if line["disposition"] != waf.DispositionDeny ||
line["client_ip"] != placed { line["client_ip"] != placed {
t.Errorf("rproxy's line is %v", line) t.Errorf("rproxy's line is %v", line)
} }
@@ -249,7 +249,7 @@ func TestTheSiteAnswersWhileWAFIsStoppedAndStartedAgain(t *testing.T) {
wantStatus(t, site+"/after-stop", placed, http.StatusOK) wantStatus(t, site+"/after-stop", placed, http.StatusOK)
line := out.line(t, "path", "/after-stop") line := out.line(t, "path", "/after-stop")
if line["disposition"] != wafprotocol.DispositionAllow || if line["disposition"] != waf.DispositionAllow ||
line["waf_backend_no_answer"] != true { line["waf_backend_no_answer"] != true {
t.Errorf("rproxy's line is %v", line) t.Errorf("rproxy's line is %v", line)
} }
@@ -437,7 +437,7 @@ func TestAChunkedBodyStaysABodyAtWAF(t *testing.T) {
const smuggledID = "smuggled-request" const smuggledID = "smuggled-request"
smuggled := "GET " + bannedProbe + " HTTP/1.1\r\nHost: site.example\r\n" + smuggled := "GET " + bannedProbe + " HTTP/1.1\r\nHost: site.example\r\n" +
wafprotocol.ClientHeader + ": " + otherClient + "\r\n" + waf.ClientHeader + ": " + otherClient + "\r\n" +
"X-Request-ID: " + smuggledID + "\r\n\r\n" "X-Request-ID: " + smuggledID + "\r\n\r\n"
status := sendRaw( status := sendRaw(
@@ -605,11 +605,10 @@ func TestRProxyAsksAWAFThatSpeaksHTTP2OverHTTP1(t *testing.T) {
protocols := make(chan string, 10) protocols := make(chan string, 10)
backend := httptest.NewUnstartedServer(http.HandlerFunc( backend := httptest.NewUnstartedServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) { func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(wafprotocol.ClientHeader) != "" { if r.Header.Get(waf.ClientHeader) != "" {
protocols <- r.Proto protocols <- r.Proto
w.Header(). w.Header().Set(waf.DispositionHeader, waf.DispositionDeny)
Set(wafprotocol.DispositionHeader, wafprotocol.DispositionDeny)
w.WriteHeader(http.StatusForbidden) w.WriteHeader(http.StatusForbidden)
} }
})) }))
@@ -691,9 +690,8 @@ func TestRProxyStoppingWaitsForAReportBeingSent(t *testing.T) {
release := make(chan struct{}) release := make(chan struct{})
backend := httptest.NewServer(http.HandlerFunc( backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) { func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(wafprotocol.ClientHeader) != "" { if r.Header.Get(waf.ClientHeader) != "" {
w.Header(). w.Header().Set(waf.DispositionHeader, waf.DispositionAllow)
Set(wafprotocol.DispositionHeader, wafprotocol.DispositionAllow)
return return
} }
@@ -781,7 +779,10 @@ func sendAndKeepOpen(t *testing.T, addr, request string) net.Conn {
func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) { func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
t.Parallel() t.Parallel()
const cutOffAfter = 5 * time.Second const (
cutOffAfter = 5 * time.Second
upstreamSetting = "SWW_RPROXY_UPSTREAM_URL"
)
asked := make(chan struct{}) asked := make(chan struct{})
release := make(chan struct{}) release := make(chan struct{})
@@ -795,8 +796,7 @@ func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
backend := httptest.NewServer(http.HandlerFunc( backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) { func(w http.ResponseWriter, _ *http.Request) {
w.Header(). w.Header().Set(waf.DispositionHeader, waf.DispositionAllow)
Set(wafprotocol.DispositionHeader, wafprotocol.DispositionAllow)
})) }))
t.Cleanup(backend.Close) t.Cleanup(backend.Close)
@@ -804,9 +804,9 @@ func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
defer stop() defer stop()
env := map[string]string{ env := map[string]string{
rproxyListenAddr: localhost + ":0", rproxyListenAddr: localhost + ":0",
rproxyUpstreamURL: app.URL, upstreamSetting: app.URL,
wafBackend: backend.URL, wafBackend: backend.URL,
} }
out := &output{} out := &output{}
exited := make(chan int, 1) exited := make(chan int, 1)
@@ -932,7 +932,7 @@ func TestMainRunsRProxy(t *testing.T) {
addr, stop := startRProxyProcess(t, wafBackend+"=http://127.0.0.1:9") addr, stop := startRProxyProcess(t, wafBackend+"=http://127.0.0.1:9")
req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, req, err := http.NewRequestWithContext(t.Context(), http.MethodGet,
"http://"+addr+wafprotocol.HealthPath, http.NoBody) "http://"+addr+waf.HealthPath, http.NoBody)
if err != nil { if err != nil {
t.Fatalf("new request: %v", err) t.Fatalf("new request: %v", err)
} }
+30 -52
View File
@@ -10,14 +10,13 @@ package sww
import ( import (
"context" "context"
"errors" "fmt"
"io" "io"
"os" "os"
"os/signal" "os/signal"
"syscall" "syscall"
"time" "time"
"github.com/spf13/cobra"
"github.com/spf13/viper" "github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
@@ -27,7 +26,14 @@ import (
// know. // know.
const usageStatus = 2 const usageStatus = 2
var errNoCommand = errors.New("no command given") // usage is what sww prints then.
const usage = `usage: sww <command>
commands:
rproxy the reverse proxy in front of one app
waf decides for rproxy, and keeps bans, counters and history
healthcheck the container's health check
`
// Params are what RunWAF and RunRProxy need from the process. // Params are what RunWAF and RunRProxy need from the process.
type Params struct { type Params struct {
@@ -45,9 +51,14 @@ type Params struct {
// Main runs the command os.Args names, and returns the process's exit status. // Main runs the command os.Args names, and returns the process's exit status.
// `sww rproxy` and `sww waf` run until SIGTERM or SIGINT. // `sww rproxy` and `sww waf` run until SIGTERM or SIGINT.
// With no command, or one it does not know, cobra prints the error and the // With no command, or one it does not know, it prints the commands and returns
// commands to stderr, and Main returns usageStatus. // usageStatus.
func Main(version string) int { func Main(version string) int {
args := os.Args[1:]
if len(args) == 0 {
return printUsage(os.Stderr)
}
ctx, stop := signal.NotifyContext(context.Background(), ctx, stop := signal.NotifyContext(context.Background(),
syscall.SIGTERM, os.Interrupt) syscall.SIGTERM, os.Interrupt)
defer stop() defer stop()
@@ -58,54 +69,21 @@ func Main(version string) int {
Stdout: os.Stdout, Stdout: os.Stdout,
} }
// status is what the command that ran returns. switch args[0] {
status := 0 case "rproxy":
return RunRProxy(ctx, params)
root := &cobra.Command{ case "waf":
Use: "sww", return RunWAF(ctx, params)
Short: "a web application firewall", case "healthcheck":
Args: cobra.NoArgs, return HealthCheck(ctx, args[1:], params.Viper, os.Stderr)
RunE: func(*cobra.Command, []string) error { default:
return errNoCommand return printUsage(os.Stderr)
},
} }
root.CompletionOptions.DisableDefaultCmd = true }
root.AddCommand( // printUsage writes the commands to w, and returns usageStatus.
&cobra.Command{ func printUsage(w io.Writer) int {
Use: "rproxy", _, _ = fmt.Fprint(w, usage)
Short: "the reverse proxy in front of one app",
Run: func(cmd *cobra.Command, _ []string) {
status = RunRProxy(cmd.Context(), params)
},
},
&cobra.Command{
Use: "waf",
Short: "decides for rproxy, and keeps bans, counters and history",
Run: func(cmd *cobra.Command, _ []string) {
status = RunWAF(cmd.Context(), params)
},
},
&cobra.Command{
Use: "healthcheck",
Short: "the container's health check",
// HealthCheck refuses every argument itself, flags included.
DisableFlagParsing: true,
Run: func(cmd *cobra.Command, args []string) {
status = HealthCheck(
cmd.Context(),
args,
params.Viper,
cmd.ErrOrStderr(),
)
},
},
)
err := root.ExecuteContext(ctx) return usageStatus
if err != nil {
return usageStatus
}
return status
} }
+2 -2
View File
@@ -23,7 +23,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config/configtest" "sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest" "sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
"sneak.berlin/go/smallwebwaf/internal/sww" "sneak.berlin/go/smallwebwaf/internal/sww"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/waf"
) )
const ( const (
@@ -1813,7 +1813,7 @@ func statusFrom(t *testing.T, url, from string) int {
} }
req.Header.Set("X-Forwarded-For", from) req.Header.Set("X-Forwarded-For", from)
req.Header.Set(wafprotocol.ClientHeader, from) req.Header.Set(waf.ClientHeader, from)
req.Header.Set("X-Request-ID", rand.Text()) req.Header.Set("X-Request-ID", rand.Text())
transport := &http.Transport{} transport := &http.Transport{}
+16 -22
View File
@@ -19,10 +19,10 @@ import (
"sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/lookup" "sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/metrics" "sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/processlog"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/remotelog" "sneak.berlin/go/smallwebwaf/internal/remotelog"
"sneak.berlin/go/smallwebwaf/internal/reputation" "sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/routewatch" "sneak.berlin/go/smallwebwaf/internal/routewatch"
"sneak.berlin/go/smallwebwaf/internal/rules" "sneak.berlin/go/smallwebwaf/internal/rules"
"sneak.berlin/go/smallwebwaf/internal/state" "sneak.berlin/go/smallwebwaf/internal/state"
@@ -43,11 +43,8 @@ const remoteLogStopTimeout = 2 * time.Second
func RunWAF(ctx context.Context, params Params) int { func RunWAF(ctx context.Context, params Params) int {
// Until the settings are read, the one message is an invalid setting's // Until the settings are read, the one message is an invalid setting's
// error, which every SWW_WAF_LOG_LEVEL lets through. // error, which every SWW_WAF_LOG_LEVEL lets through.
processLog := processlog.New(processlog.Params{ processLog := requestlog.NewProcessLogger(params.Stdout,
Output: params.Stdout, config.InstanceName(params.Viper), slog.LevelError)
InstanceName: config.InstanceName(params.Viper),
Level: slog.LevelError,
})
cfg, err := config.WAFFromEnvironment(params.Viper) cfg, err := config.WAFFromEnvironment(params.Viper)
if err != nil { if err != nil {
@@ -66,11 +63,11 @@ func RunWAF(ctx context.Context, params Params) int {
stdout = io.MultiWriter(params.Stdout, remote) stdout = io.MultiWriter(params.Stdout, remote)
} }
processLog = processlog.New(processlog.Params{ processLog = requestlog.NewProcessLogger(
Output: stdout, stdout,
InstanceName: cfg.InstanceName, cfg.InstanceName,
Level: cfg.LogLevel, cfg.LogLevel,
}) )
if remote != nil { if remote != nil {
stopSending := startSending(ctx, remote, processLog) stopSending := startSending(ctx, remote, processLog)
@@ -242,17 +239,14 @@ func newLookupFile(
// newLimiter returns the table of clients, with the rate and byte limits. // newLimiter returns the table of clients, with the rate and byte limits.
func newLimiter(cfg *config.WAF) *ratelimit.Limiter { func newLimiter(cfg *config.WAF) *ratelimit.Limiter {
return ratelimit.New(ratelimit.Params{ return ratelimit.New(ratelimit.Limits{
Limits: ratelimit.Limits{ PerMinute: cfg.RateLimitPerMinute,
PerMinute: cfg.RateLimitPerMinute, PerHour: cfg.RateLimitPerHour,
PerHour: cfg.RateLimitPerHour, PerDay: cfg.RateLimitPerDay,
PerDay: cfg.RateLimitPerDay, BytesPerMinute: cfg.BytesLimitPerMinute,
BytesPerMinute: cfg.BytesLimitPerMinute, BytesPerHour: cfg.BytesLimitPerHour,
BytesPerHour: cfg.BytesLimitPerHour, BytesPerDay: cfg.BytesLimitPerDay,
BytesPerDay: cfg.BytesLimitPerDay, }, cfg.MaxTrackedClients)
},
MaxClients: cfg.MaxTrackedClients,
})
} }
// newLedger returns the ledger of the bans, with the settings for them. // newLedger returns the ledger of the bans, with the settings for them.
+37 -102
View File
@@ -4,7 +4,6 @@ import (
"bytes" "bytes"
"context" "context"
"crypto/rand" "crypto/rand"
"errors"
"io" "io"
"net/http" "net/http"
"os" "os"
@@ -21,7 +20,7 @@ import (
_ "time/tzdata" _ "time/tzdata"
"sneak.berlin/go/smallwebwaf/internal/sww" "sneak.berlin/go/smallwebwaf/internal/sww"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/waf"
) )
// commandForMain, while set in the environment, has this test binary run Main // commandForMain, while set in the environment, has this test binary run Main
@@ -107,7 +106,7 @@ func TestMainRunsWAF(t *testing.T) {
starting := out.lineBeforeExit(t, "starting", exited) starting := out.lineBeforeExit(t, "starting", exited)
addr, _ := starting["address"].(string) addr, _ := starting["address"].(string)
ask(t, addr, "/", http.StatusOK, wafprotocol.DispositionAllow) ask(t, addr, "/", http.StatusOK, waf.DispositionAllow)
// The files are written as sww stops. // The files are written as sww stops.
err = cmd.Process.Signal(syscall.SIGTERM) err = cmd.Process.Signal(syscall.SIGTERM)
@@ -165,104 +164,46 @@ func wantUTC(t *testing.T, where string, value any) {
func TestMainWithoutACommandItKnowsPrintsTheCommands(t *testing.T) { func TestMainWithoutACommandItKnowsPrintsTheCommands(t *testing.T) {
t.Parallel() t.Parallel()
// completion is cobra's own command, which sww turns off.
for _, command := range []string{"", "serve", "completion"} {
t.Run("command "+command, func(t *testing.T) {
t.Parallel()
code, stdout, stderr := runMain(t, command)
if code != 2 {
t.Errorf("exit status %d, want 2", code)
}
wantTheCommands(t, "stderr", stderr)
if stdout != "" {
t.Errorf("stdout %q, want nothing", stdout)
}
})
}
}
func TestMainHelpPrintsTheCommands(t *testing.T) {
t.Parallel()
for _, command := range []string{"help", "--help"} {
t.Run("command "+command, func(t *testing.T) {
t.Parallel()
code, stdout, _ := runMain(t, command)
if code != 0 {
t.Errorf("exit status %d, want 0", code)
}
wantTheCommands(t, "stdout", stdout)
})
}
}
// wantTheCommands checks that text, what Main wrote to where, names each
// command.
func wantTheCommands(t *testing.T, where, text string) {
t.Helper()
for _, name := range []string{"rproxy", wafCommand, "healthcheck"} {
if !strings.Contains(text, name) {
t.Errorf("%s %q does not name %s", where, text, name)
}
}
}
func TestMainHealthCheckRefusesAnArgument(t *testing.T) {
t.Parallel()
for _, argument := range []string{"now", "--x", "-h"} {
t.Run("argument "+argument, func(t *testing.T) {
t.Parallel()
code, stdout, stderr := runMain(t, "healthcheck "+argument)
want := "sww healthcheck: unexpected argument \"" +
argument + "\"\n"
if code != 1 || stderr != want || stdout != "" {
t.Errorf("exit status %d, stdout %q and stderr %q, "+
"want 1, nothing and %q", code, stdout, stderr, want)
}
})
}
}
// runMain runs Main with command in a process of its own, and returns its
// exit status and what it wrote to stdout and stderr.
func runMain(t *testing.T, command string) (int, string, string) {
t.Helper()
binary, err := os.Executable() binary, err := os.Executable()
if err != nil { if err != nil {
t.Fatalf("find the test binary: %v", err) t.Fatalf("find the test binary: %v", err)
} }
ctx, cancel := context.WithTimeout(t.Context(), waitLimit) for _, command := range []string{"", "serve"} {
defer cancel() t.Run("command "+command, func(t *testing.T) {
t.Parallel()
var stdout, stderr bytes.Buffer ctx, cancel := context.WithTimeout(t.Context(), waitLimit)
defer cancel()
//nolint:gosec // this test binary var stdout, stderr bytes.Buffer
cmd := exec.CommandContext(ctx, binary)
cmd.Stdout, cmd.Stderr = &stdout, &stderr
cmd.Env = append(os.Environ(), commandForMain+"="+command) cmd := exec.CommandContext(ctx, binary)
cmd.Stdout, cmd.Stderr = &stdout, &stderr
err = cmd.Run() cmd.Env = append(os.Environ(), commandForMain+"="+command)
var exitErr *exec.ExitError err := cmd.Run()
if err != nil && !errors.As(err, &exitErr) {
t.Fatalf("run: %v", err) if code := cmd.ProcessState.ExitCode(); code != 2 {
t.Errorf("exit status %d (%v), want 2", code, err)
}
for _, name := range []string{"rproxy", wafCommand, "healthcheck"} {
if !strings.Contains(stderr.String(), name) {
t.Errorf(
"stderr %q does not name %s",
stderr.String(),
name,
)
}
}
if stdout.Len() != 0 {
t.Errorf("stdout %q, want nothing", stdout.String())
}
})
} }
return cmd.ProcessState.ExitCode(), stdout.String(), stderr.String()
} }
// ask sends waf at addr rproxy's question about a request for path from // ask sends waf at addr rproxy's question about a request for path from
@@ -276,7 +217,7 @@ func ask(t *testing.T, addr, path string, status int, disposition string) {
t.Fatalf("new request: %v", err) t.Fatalf("new request: %v", err)
} }
req.Header.Set(wafprotocol.ClientHeader, placed) req.Header.Set(waf.ClientHeader, placed)
req.Header.Set("X-Request-ID", rand.Text()) req.Header.Set("X-Request-ID", rand.Text())
transport := &http.Transport{} transport := &http.Transport{}
@@ -291,11 +232,11 @@ func ask(t *testing.T, addr, path string, status int, disposition string) {
_ = res.Body.Close() _ = res.Body.Close()
if res.StatusCode != status || if res.StatusCode != status ||
res.Header.Get(wafprotocol.DispositionHeader) != disposition { res.Header.Get(waf.DispositionHeader) != disposition {
t.Errorf( t.Errorf(
"answered %d with disposition %q, want %d with %q", "answered %d with disposition %q, want %d with %q",
res.StatusCode, res.StatusCode,
res.Header.Get(wafprotocol.DispositionHeader), res.Header.Get(waf.DispositionHeader),
status, status,
disposition, disposition,
) )
@@ -346,18 +287,12 @@ func TestWAFKeepsItsBansAcrossARestartAndNeedsNoApp(t *testing.T) {
} }
runWAF(t, env, func(addr string) { runWAF(t, env, func(addr string) {
ask(t, addr, "/", http.StatusOK, wafprotocol.DispositionAllow) ask(t, addr, "/", http.StatusOK, waf.DispositionAllow)
// The default rule file bans a probe for this. // The default rule file bans a probe for this.
ask( ask(t, addr, "/.env", http.StatusForbidden, waf.DispositionDeny)
t,
addr,
"/.env",
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
}) })
runWAF(t, env, func(addr string) { runWAF(t, env, func(addr string) {
ask(t, addr, "/", http.StatusForbidden, wafprotocol.DispositionDeny) ask(t, addr, "/", http.StatusForbidden, waf.DispositionDeny)
}) })
} }
+23 -61
View File
@@ -26,7 +26,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/reputation" "sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
const ( const (
@@ -85,7 +84,7 @@ func TestSixteenThousandCookiesAreRefusedBeforeTheCoreRuleSetReadsThem(
got := do(t, req) got := do(t, req)
wantDisposition(t, got, tooLarge, wafprotocol.DispositionDeny) wantDisposition(t, got, tooLarge, waf.DispositionDeny)
line := s.out.requestLine(t) line := s.out.requestLine(t)
wantLine(t, line, tooLarge, requestlog.ActionRequestHeaderTooLarge) wantLine(t, line, tooLarge, requestlog.ActionRequestHeaderTooLarge)
@@ -165,14 +164,9 @@ func TestARequestLineOverTheLimitIsRefusedWith414BeforeAnyOtherCheck(
line := s.out.requestLine(t) line := s.out.requestLine(t)
if tc.status == http.StatusOK { if tc.status == http.StatusOK {
wantDisposition( wantDisposition(t, got, tc.status, waf.DispositionAllow)
t,
got,
tc.status,
wafprotocol.DispositionAllow,
)
} else { } else {
wantDisposition(t, got, tc.status, wafprotocol.DispositionDeny) wantDisposition(t, got, tc.status, waf.DispositionDeny)
} }
wantLine(t, line, tc.status, tc.action) wantLine(t, line, tc.status, tc.action)
@@ -225,14 +219,9 @@ func TestAHeaderLineOverTheLimitIsRefusedWith431(t *testing.T) {
got := do(t, req) got := do(t, req)
if tc.refused { if tc.refused {
wantDisposition( wantDisposition(t, got, tooLarge, waf.DispositionDeny)
t,
got,
tooLarge,
wafprotocol.DispositionDeny,
)
} else { } else {
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow) wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
} }
}) })
} }
@@ -255,7 +244,10 @@ func TestRequestLineRefusalNamesItsHeuristicLengthAndLimit(t *testing.T) {
func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) { func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
t.Parallel() t.Parallel()
repeated := slices.Repeat([]string{"1"}, headerFieldLimit) repeated := make([]string, headerFieldLimit)
for i := range repeated {
repeated[i] = "1"
}
for _, tc := range []struct { for _, tc := range []struct {
name string name string
@@ -302,12 +294,7 @@ func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
line := s.out.requestLine(t) line := s.out.requestLine(t)
if tc.refused { if tc.refused {
wantDisposition( wantDisposition(t, got, tooLarge, waf.DispositionDeny)
t,
got,
tooLarge,
wafprotocol.DispositionDeny,
)
wantLine( wantLine(
t, t,
line, line,
@@ -318,12 +305,7 @@ func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
return return
} }
wantDisposition( wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
t,
got,
http.StatusOK,
wafprotocol.DispositionAllow,
)
wantLine(t, line, http.StatusOK, requestlog.ActionForward) wantLine(t, line, http.StatusOK, requestlog.ActionForward)
if line.DurationCoreRuleSetMS == nil { if line.DurationCoreRuleSetMS == nil {
@@ -350,12 +332,7 @@ func TestHeaderLimitsHoldWhateverTheCoreRuleSetSettingsSay(t *testing.T) {
req := s.question(siteA, client, "/") req := s.question(siteA, client, "/")
req.Header.Set(cookieHeader, cookies(cookieLimit+1)) req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition( wantDisposition(t, do(t, req), tooLarge, waf.DispositionDeny)
t,
do(t, req),
tooLarge,
wafprotocol.DispositionDeny,
)
}) })
} }
} }
@@ -371,12 +348,7 @@ func TestClientInTheAllowNetsSkipsTheHeaderLimits(t *testing.T) {
req := s.question(siteA, client, "/") req := s.question(siteA, client, "/")
req.Header.Set(cookieHeader, cookies(cookieLimit+1)) req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition( wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow)
t,
do(t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
)
} }
func TestObserveModeAllowsARequestOverTheHeaderLimitsUnread(t *testing.T) { func TestObserveModeAllowsARequestOverTheHeaderLimitsUnread(t *testing.T) {
@@ -392,7 +364,7 @@ func TestObserveModeAllowsARequestOverTheHeaderLimitsUnread(t *testing.T) {
got := do(t, req) got := do(t, req)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow) wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
line := s.out.requestLine(t) line := s.out.requestLine(t)
if line.WouldAction != requestlog.ActionRequestHeaderTooLarge || if line.WouldAction != requestlog.ActionRequestHeaderTooLarge ||
@@ -414,14 +386,14 @@ func TestTheRateLimitsDoNotCountRequestsOverTheHeaderLimits(t *testing.T) {
for range 2 { for range 2 {
req := s.question(siteA, client, "/") req := s.question(siteA, client, "/")
req.Header.Set(cookieHeader, cookies(cookieLimit+1)) req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(t, do(t, req), tooLarge, wafprotocol.DispositionDeny) wantDisposition(t, do(t, req), tooLarge, waf.DispositionDeny)
} }
wantDisposition( wantDisposition(
t, t,
s.ask(client, "/"), s.ask(client, "/"),
http.StatusOK, http.StatusOK,
wafprotocol.DispositionAllow, waf.DispositionAllow,
) )
} }
@@ -494,12 +466,7 @@ func TestTheHeaderLimitsComeBeforeEveryCheckThatRefuses(t *testing.T) {
req := s.question(siteA, client, tc.path) req := s.question(siteA, client, tc.path)
req.Header.Set(cookieHeader, cookies(cookieLimit+1)) req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition( wantDisposition(t, do(t, req), tooLarge, waf.DispositionDeny)
t,
do(t, req),
tooLarge,
wafprotocol.DispositionDeny,
)
wantLine( wantLine(
t, t,
s.out.requestLine(t), s.out.requestLine(t),
@@ -536,16 +503,11 @@ func TestARequestOverTheHeaderLimitsRunsNoOtherCheck(t *testing.T) {
over := s.question(siteA, client, "/") over := s.question(siteA, client, "/")
over.Header.Set(cookieHeader, cookies(cookieLimit+1)) over.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(t, do(t, over), tooLarge, wafprotocol.DispositionDeny) wantDisposition(t, do(t, over), tooLarge, waf.DispositionDeny)
within := s.question(siteA, client, "/") within := s.question(siteA, client, "/")
within.Header.Set(cookieHeader, cookies(cookieLimit)) within.Header.Set(cookieHeader, cookies(cookieLimit))
wantDisposition( wantDisposition(t, do(t, within), http.StatusOK, waf.DispositionAllow)
t,
do(t, within),
http.StatusOK,
wafprotocol.DispositionAllow,
)
lines := s.out.requestLines(t, 2) lines := s.out.requestLines(t, 2)
@@ -602,7 +564,7 @@ func TestNotWaitingForGeoJSNeverLowersAClientsLimit(t *testing.T) {
t, t,
s.ask(fromDE, "/"), s.ask(fromDE, "/"),
http.StatusForbidden, http.StatusForbidden,
wafprotocol.DispositionDeny, waf.DispositionDeny,
) )
line := s.out.requestLine(t) line := s.out.requestLine(t)
@@ -639,7 +601,7 @@ func TestAClientGeoJSPlacedInNoCountryCountsAsUnknownWithoutWaiting(
t, t,
s.ask(client, "/"), s.ask(client, "/"),
http.StatusForbidden, http.StatusForbidden,
wafprotocol.DispositionDeny, waf.DispositionDeny,
) )
line := s.out.requestLine(t) line := s.out.requestLine(t)
@@ -668,7 +630,7 @@ func TestObserveModeAsksGeoJSNothingForARequestOverTheHeaderLimits(
req.Header.Set(cookieHeader, cookies(cookieLimit+1)) req.Header.Set(cookieHeader, cookies(cookieLimit+1))
got := do(t, req) got := do(t, req)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow) wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
if len(asked()) != 0 || got.header.Get("X-Client-Asn") != "" { if len(asked()) != 0 || got.header.Get("X-Client-Asn") != "" {
t.Errorf( t.Errorf(
@@ -710,7 +672,7 @@ func askDirectly(
http.NoBody, http.NoBody,
) )
req.RemoteAddr = net.JoinHostPort(rproxyPeer, "1234") req.RemoteAddr = net.JoinHostPort(rproxyPeer, "1234")
req.Header.Set(wafprotocol.ClientHeader, client) req.Header.Set(waf.ClientHeader, client)
req.Header.Set(requestIDHeader, "the-id") req.Header.Set(requestIDHeader, "the-id")
change(req) change(req)
+5 -4
View File
@@ -548,10 +548,6 @@ func (s *sender) requestWithHeader(
// body is sent as it is after the headers, header holding its // body is sent as it is after the headers, header holding its
// Content-Length or Transfer-Encoding. header may hold several lines, // Content-Length or Transfer-Encoding. header may hold several lines,
// separated by "\r\n". It returns the whole answer. // separated by "\r\n". It returns the whole answer.
//
// It waits for the answer as long as it takes, since a busy host can slow
// the Core Rule Set past any shorter wait. go test's -timeout ends a test
// whose answer never comes.
func (s *sender) requestWithBody( func (s *sender) requestWithBody(
method, from, path, header, body string, status int, action string, method, from, path, header, body string, status int, action string,
) (logLine, answer) { ) (logLine, answer) {
@@ -570,6 +566,11 @@ func (s *sender) requestWithBody(
send(s.t, conn, method+" "+path+" HTTP/1.1\r\nHost: "+appHost+"\r\n"+ send(s.t, conn, method+" "+path+" HTTP/1.1\r\nHost: "+appHost+"\r\n"+
forwardedFor+": "+from+"\r\n"+header+"\r\n"+body) forwardedFor+": "+from+"\r\n"+header+"\r\n"+body)
err := conn.SetReadDeadline(time.Now().Add(waitLimit))
if err != nil {
s.t.Fatalf("set read deadline: %v", err)
}
var got answer var got answer
res, err := http.ReadResponse(bufio.NewReader(conn), nil) res, err := http.ReadResponse(bufio.NewReader(conn), nil)
+1 -2
View File
@@ -19,7 +19,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest" "sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// The biased thresholds. // The biased thresholds.
@@ -590,7 +589,7 @@ func serveFromDE(
t.Helper() t.Helper()
req := httptest.NewRequestWithContext(t.Context(), method, "/", body) req := httptest.NewRequestWithContext(t.Context(), method, "/", body)
req.Header.Set(wafprotocol.ClientHeader, fromDE) req.Header.Set(waf.ClientHeader, fromDE)
req.Header.Set(requestIDHeader, "the-id") req.Header.Set(requestIDHeader, "the-id")
server.Handler.ServeHTTP(httptest.NewRecorder(), req) server.Handler.ServeHTTP(httptest.NewRecorder(), req)
+4 -3
View File
@@ -5,8 +5,6 @@ import (
"net/http" "net/http"
"net/netip" "net/netip"
"slices" "slices"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// peerAddress is the address of the request's TCP peer. // peerAddress is the address of the request's TCP peer.
@@ -19,11 +17,14 @@ func peerAddress(r *http.Request) netip.Addr {
return addrPort.Addr().Unmap() return addrPort.Addr().Unmap()
} }
// requestIDHeader carries the request's id, from rproxy.
const requestIDHeader = "X-Request-ID"
// requestID is the request's id: the one a question gives, or for any other // requestID is the request's id: the one a question gives, or for any other
// request a new random one. // request a new random one.
func requestID(r *http.Request, isQuestion bool) string { func requestID(r *http.Request, isQuestion bool) string {
if isQuestion { if isQuestion {
return r.Header.Get(wafprotocol.RequestIDHeader) return r.Header.Get(requestIDHeader)
} }
return rand.Text() return rand.Text()
+3 -3
View File
@@ -12,7 +12,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/waf"
) )
// The Core Rule Set's settings the tests set, besides SWW_WAF_CORE_RULE_SET_MODE, and // The Core Rule Set's settings the tests set, besides SWW_WAF_CORE_RULE_SET_MODE, and
@@ -395,9 +395,9 @@ func TestGiteasFixesApplyOnlyToAHostnameThatServesGitea(t *testing.T) {
}) })
wantDisposition(t, do(t, s.question(siteA, client, fileName)), wantDisposition(t, do(t, s.question(siteA, client, fileName)),
http.StatusOK, wafprotocol.DispositionAllow) http.StatusOK, waf.DispositionAllow)
wantDisposition(t, do(t, s.question(siteB, client, fileName)), wantDisposition(t, do(t, s.question(siteB, client, fileName)),
http.StatusForbidden, wafprotocol.DispositionDeny) http.StatusForbidden, waf.DispositionDeny)
lines := s.out.requestLines(t, 2) lines := s.out.requestLines(t, 2)
wantLine(t, lines[0], http.StatusOK, requestlog.ActionForward) wantLine(t, lines[0], http.StatusOK, requestlog.ActionForward)
+3 -8
View File
@@ -5,7 +5,7 @@ import (
"testing" "testing"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/waf"
) )
func TestHealthEndpointCountsTowardNoLimit(t *testing.T) { func TestHealthEndpointCountsTowardNoLimit(t *testing.T) {
@@ -25,13 +25,8 @@ func TestHealthEndpointCountsTowardNoLimit(t *testing.T) {
) )
for range healthChecks { for range healthChecks {
got := s.ownEndpointRequest( got := s.ownEndpointRequest("", http.MethodGet, waf.HealthPath, "",
"", http.StatusOK)
http.MethodGet,
wafprotocol.HealthPath,
"",
http.StatusOK,
)
if string(got.body) != "ok\n" || if string(got.body) != "ok\n" ||
got.header.Get("Content-Type") != contentType { got.header.Get("Content-Type") != contentType {
+1 -2
View File
@@ -11,7 +11,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
func TestHistoryKeepsEachRequestOfTheClient(t *testing.T) { func TestHistoryKeepsEachRequestOfTheClient(t *testing.T) {
@@ -119,7 +118,7 @@ func TestOwnEndpointsAreNotInTheHistory(t *testing.T) {
s.ownEndpointRequest( s.ownEndpointRequest(
"", "",
http.MethodGet, http.MethodGet,
wafprotocol.HealthPath, waf.HealthPath,
"", "",
http.StatusOK, http.StatusOK,
) )
+22
View File
@@ -3,6 +3,7 @@ package waf
import ( import (
"context" "context"
"log/slog" "log/slog"
"net/http"
"net/netip" "net/netip"
"strings" "strings"
"time" "time"
@@ -15,6 +16,15 @@ import (
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
) )
// The headers in which an answer of allow gives rproxy the client's AS number
// and country, for the app, while SWW_WAF_ADD_LOOKUP_HEADERS is true. Go writes
// every header name in this form, as it sends it and as it receives it, so
// X-Client-ASN arrives as X-Client-Asn; header names are not case-sensitive.
const (
asnHeader = "X-Client-Asn"
countryHeader = "X-Client-Country"
)
// lookupResult is what the lookup gave for a request: whether its client was // lookupResult is what the lookup gave for a request: whether its client was
// looked up, the answer, and whether the request skipped the wait for GeoJS. // looked up, the answer, and whether the request skipped the wait for GeoJS.
type lookupResult struct { type lookupResult struct {
@@ -159,6 +169,18 @@ func GeoJSFailed(
m.GeoJSFailures.Inc() m.GeoJSFailures.Inc()
} }
// setLookupHeaders sets the headers that give the client's AS number and
// country, leaving out one that is unknown.
func setLookupHeaders(header http.Header, asn, country string) {
if asn != "" {
header.Set(asnHeader, asn)
}
if country != "" {
header.Set(countryHeader, country)
}
}
// canBePlaced reports whether a lookup can place addr: private, loopback // canBePlaced reports whether a lookup can place addr: private, loopback
// and link-local addresses have no AS number or country. // and link-local addresses have no AS number or country.
func canBePlaced(addr netip.Addr) bool { func canBePlaced(addr netip.Addr) bool {
+1 -2
View File
@@ -16,7 +16,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
const ( const (
@@ -67,7 +66,7 @@ func TestMetricsNeedTheTokenAndOtherPathsAreNotFound(t *testing.T) {
{http.MethodPost, waf.MetricsPath, bearer, http.StatusNotFound}, {http.MethodPost, waf.MetricsPath, bearer, http.StatusNotFound},
{http.MethodGet, waf.MetricsPath + "/", bearer, http.StatusNotFound}, {http.MethodGet, waf.MetricsPath + "/", bearer, http.StatusNotFound},
{http.MethodGet, "/_sww/bans", bearer, http.StatusNotFound}, {http.MethodGet, "/_sww/bans", bearer, http.StatusNotFound},
{http.MethodPost, wafprotocol.HealthPath, "", http.StatusNotFound}, {http.MethodPost, waf.HealthPath, "", http.StatusNotFound},
} { } {
got := s.ownEndpointRequest( got := s.ownEndpointRequest(
tc.authorization, tc.authorization,
+8
View File
@@ -28,6 +28,14 @@ import (
"sneak.berlin/go/smallwebwaf/internal/rules" "sneak.berlin/go/smallwebwaf/internal/rules"
) )
// ownEndpointPrefix starts the path of each of waf's own endpoints. The app
// never sees a path under it.
const ownEndpointPrefix = "/_sww/"
// HealthPath is the health endpoint of waf and of rproxy, which the
// container's health check asks.
const HealthPath = "/_sww/healthz"
// MetricsPath is where the metrics are, for a request that carries // MetricsPath is where the metrics are, for a request that carries
// SWW_WAF_METRICS_TOKEN. // SWW_WAF_METRICS_TOKEN.
const MetricsPath = "/_sww/metrics" const MetricsPath = "/_sww/metrics"
+24 -24
View File
@@ -24,7 +24,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/lookup" "sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/metrics" "sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/processlog"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation" "sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
@@ -194,11 +193,11 @@ func (o *output) requestLine(t *testing.T) logLine {
// endedLines waits for count requests that went through rproxy, or to waf's // endedLines waits for count requests that went through rproxy, or to waf's
// own endpoints, to have ended, and returns the last log line of each, in the // own endpoints, to have ended, and returns the last log line of each, in the
// order waf took them. A question's last line is the one its report writes. // order waf took them. A question's last line is the one its report writes.
// It waits as long as it takes, as requestWithBody does.
func (o *output) endedLines(t *testing.T, count int) []logLine { func (o *output) endedLines(t *testing.T, count int) []logLine {
t.Helper() t.Helper()
for { deadline := time.Now().Add(waitLimit)
for time.Now().Before(deadline) {
found := o.lastLines(t) found := o.lastLines(t)
ended := len(found) >= count ended := len(found) >= count
@@ -214,6 +213,10 @@ func (o *output) endedLines(t *testing.T, count int) []logLine {
time.Sleep(pollInterval) time.Sleep(pollInterval)
} }
t.Fatalf("fewer than %d requests have ended after %s", count, waitLimit)
return nil
} }
// lastLines returns the last request log line written so far of each request, // lastLines returns the last request log line written so far of each request,
@@ -395,11 +398,11 @@ func startRProxy(
server := rproxy.New(rproxy.Params{ server := rproxy.New(rproxy.Params{
Config: cfg, Config: cfg,
RequestLog: io.Discard, RequestLog: io.Discard,
ProcessLog: processlog.New(processlog.Params{ ProcessLog: requestlog.NewProcessLogger(
Output: io.Discard, io.Discard,
InstanceName: "rproxy", "rproxy",
Level: slog.LevelInfo, slog.LevelInfo,
}), ),
}) })
listener, err := (&net.ListenConfig{}).Listen( listener, err := (&net.ListenConfig{}).Listen(
@@ -468,11 +471,11 @@ func newWAFWith(
} }
out := &output{} out := &output{}
processLog := processlog.New(processlog.Params{ processLog := requestlog.NewProcessLogger(
Output: out, out,
InstanceName: cfg.InstanceName, cfg.InstanceName,
Level: cfg.LogLevel, cfg.LogLevel,
}) )
ruleFiles, err := rules.Load(rules.Params{ ruleFiles, err := rules.Load(rules.Params{
Dir: cfg.RuleFilesDir, Enabled: cfg.RuleFilesEnabled, ProcessLog: processLog, Dir: cfg.RuleFilesDir, Enabled: cfg.RuleFilesEnabled, ProcessLog: processLog,
@@ -558,17 +561,14 @@ func addCounters(params *waf.Params) {
cfg.MetricsBusiestCountriesAndASNs, cfg.MetricsBusiestCountriesAndASNs,
cfg.InstanceName, cfg.InstanceName,
) )
params.Limiter = ratelimit.New(ratelimit.Params{ params.Limiter = ratelimit.New(ratelimit.Limits{
Limits: ratelimit.Limits{ PerMinute: cfg.RateLimitPerMinute,
PerMinute: cfg.RateLimitPerMinute, PerHour: cfg.RateLimitPerHour,
PerHour: cfg.RateLimitPerHour, PerDay: cfg.RateLimitPerDay,
PerDay: cfg.RateLimitPerDay, BytesPerMinute: cfg.BytesLimitPerMinute,
BytesPerMinute: cfg.BytesLimitPerMinute, BytesPerHour: cfg.BytesLimitPerHour,
BytesPerHour: cfg.BytesLimitPerHour, BytesPerDay: cfg.BytesLimitPerDay,
BytesPerDay: cfg.BytesLimitPerDay, }, cfg.MaxTrackedClients)
},
MaxClients: cfg.MaxTrackedClients,
})
params.Ledger = bans.New(bans.Params{ params.Ledger = bans.New(bans.Params{
LimitBanDuration: cfg.LimitBanDuration, LimitBanDuration: cfg.LimitBanDuration,
LimitBanRepeatWindow: cfg.LimitBanRepeatWindow, LimitBanRepeatWindow: cfg.LimitBanRepeatWindow,
+2 -2
View File
@@ -13,7 +13,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/waf"
) )
const ( const (
@@ -248,7 +248,7 @@ func TestTimingsAreInOrder(t *testing.T) {
wantStatus(t, do(t, req), http.StatusForbidden) wantStatus(t, do(t, req), http.StatusForbidden)
refused := out.endedLines(t, 2)[1] refused := out.endedLines(t, 2)[1]
wantStatus(t, get(t, server.Addr, wafprotocol.HealthPath), http.StatusOK) wantStatus(t, get(t, server.Addr, waf.HealthPath), http.StatusOK)
health := out.endedLines(t, 3)[2] health := out.endedLines(t, 3)[2]
// A request passed to the app has every timing; one refused, none of // A request passed to the app has every timing; one refused, none of
+226 -54
View File
@@ -1,21 +1,75 @@
package waf package waf
import ( import (
"bytes"
"context" "context"
"encoding/json"
"errors" "errors"
"fmt"
"io"
"log/slog" "log/slog"
"net/http" "net/http"
"net/netip"
"strconv"
"strings" "strings"
"time" "time"
"unicode/utf8"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// The headers of rproxy's question and of waf's answer, as
// "waf" in SPEC.md gives them.
const (
// ClientHeader marks a request as rproxy's question, and gives the
// visitor's address.
ClientHeader = "X-Sww-Client"
// PeerHeader gives the address of rproxy's own TCP peer.
PeerHeader = "X-Sww-Peer"
// ContentLengthHeader gives the body length the visitor announced.
ContentLengthHeader = "X-Sww-Content-Length"
// DispositionHeader gives waf's answer.
DispositionHeader = "X-Sww-Disposition"
)
// The values of DispositionHeader.
const (
// DispositionAllow tells rproxy to pass the request to the app.
DispositionAllow = "allow"
// DispositionDeny tells rproxy to give the visitor the answer's status,
// headers and body.
DispositionDeny = "deny"
// DispositionClose tells rproxy to close the visitor's connection
// without an answer, which SWW_WAF_BAN_RESPONSE asks for when it is close.
// waf closing the question's connection instead would read as
// waf being down, and rproxy would then let the request through.
DispositionClose = "close"
)
// ReportPath is where rproxy sends the report of each request waf
// answered.
const ReportPath = "/_sww/report"
// QuestionBodyMaxBytes is the most of a visitor's body a question carries.
const QuestionBodyMaxBytes = 64 << 10
const ( const (
// maxAwaitingReport is the most answered requests held until their // maxAwaitingReport is the most answered requests held until their
// reports come. Past it, the request answered longest ago is dropped. // reports come. Past it, the request answered longest ago is dropped.
maxAwaitingReport = 10000 maxAwaitingReport = 10000
// maxRequestIDBytes is the longest id a question may give.
maxRequestIDBytes = 128
// reportBodyMaxBytes is the most of a report's body that is read. It is
// far above any report's.
reportBodyMaxBytes = 1 << 20
// maxHTTPStatus is the highest status a report may give.
maxHTTPStatus = 599
// maxReportedBytes is the most bytes a report may give for each way, 1
// PiB, so that adding them up cannot overflow.
maxReportedBytes = 1 << 50
// maxReportedMilliseconds is the longest duration a report may give,
// about 31 years, so that it fits a time.Duration.
maxReportedMilliseconds = 1e12
// wafTimeout bounds how long a request to waf takes to // wafTimeout bounds how long a request to waf takes to
// arrive, and its answer to leave, and wafIdleTimeout how long a // arrive, and its answer to leave, and wafIdleTimeout how long a
// kept-open connection waits for its next request. // kept-open connection waits for its next request.
@@ -23,6 +77,26 @@ const (
wafIdleTimeout = 120 * time.Second wafIdleTimeout = 120 * time.Second
) )
var (
errReportFieldMissing = errors.New("a field is missing")
errReportValue = errors.New("a value is out of range")
)
// Report is the JSON object rproxy posts to ReportPath once the visitor's
// answer has ended, its fields named as the request log's of the same
// meaning. RequestID is the question's id. Status is what the visitor got, 0
// if nothing. The durations are in milliseconds.
//
//nolint:tagliatelle // docs/request-log.md names the fields in snake_case
type Report struct {
RequestID string `json:"request_id"`
Status int `json:"status"`
RequestBytes int64 `json:"request_bytes"`
ResponseBytes int64 `json:"response_bytes"`
DurationTotalMS float64 `json:"duration_total_ms"`
DurationUpstreamTotalMS float64 `json:"duration_upstream_total_ms"`
}
// New returns the server `sww waf` runs. It answers // New returns the server `sww waf` runs. It answers
// rproxy's questions, takes rproxy's reports and answers its own endpoints. It // rproxy's questions, takes rproxy's reports and answers its own endpoints. It
// passes nothing to an app. rproxy applies the size and time limits to the // passes nothing to an app. rproxy applies the size and time limits to the
@@ -58,10 +132,10 @@ func New(params Params) *Server {
} }
// ServeHTTP handles one request to waf: a request carrying // ServeHTTP handles one request to waf: a request carrying
// wafprotocol.ClientHeader is rproxy's question, and any other is for waf // ClientHeader is rproxy's question, and any other is for waf
// itself. // itself.
func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) { func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
if len(r.Header.Values(wafprotocol.ClientHeader)) > 0 { if len(r.Header.Values(ClientHeader)) > 0 {
h.answerQuestion(w, r) h.answerQuestion(w, r)
return return
@@ -90,16 +164,19 @@ func (rq *request) refusedWhateverTheAnswer(now time.Time) bool {
// carries, and answers rproxy what to do with it. A denied request's counting // carries, and answers rproxy what to do with it. A denied request's counting
// is finished here, and an allowed one's by its report. // is finished here, and an allowed one's by its report.
func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) { func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
question, ok := h.readQuestion(w, r) client, body, ok := h.readQuestion(w, r)
if !ok { if !ok {
return return
} }
// One that cannot be read leaves the log line's peer_ip invalid.
peer, _ := netip.ParseAddr(r.Header.Get(PeerHeader))
rq := h.startRequest( rq := h.startRequest(
w, w,
question.Visitor, asTheVisitorSentIt(r, body),
question.Client, client.Unmap(),
question.Peer, peer.Unmap(),
true, true,
) )
// These run even when a check panics. finish runs first, so that what is // These run even when a check panics. finish runs first, so that what is
@@ -114,10 +191,7 @@ func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
rq.line.WAFLine = requestlog.WAFLineDecision rq.line.WAFLine = requestlog.WAFLineDecision
refused := rq.check(r.Context()) refused := rq.check(r.Context())
rq.checked = time.Now() rq.checked = time.Now()
underOwnPrefix := strings.HasPrefix( underOwnPrefix := strings.HasPrefix(r.URL.Path, ownEndpointPrefix)
r.URL.Path,
wafprotocol.OwnEndpointPrefix,
)
if refused == nil && !underOwnPrefix { if refused == nil && !underOwnPrefix {
rq.allow() rq.allow()
@@ -133,10 +207,7 @@ func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
// The app never sees this prefix. waf's own endpoints are at its // The app never sees this prefix. waf's own endpoints are at its
// own address, so this answers as a path with no endpoint does. // own address, so this answers as a path with no endpoint does.
rq.line.Action = requestlog.ActionOwnEndpoint rq.line.Action = requestlog.ActionOwnEndpoint
wafprotocol.SetAnswerHeaders( rq.out.Header().Set(DispositionHeader, DispositionDeny)
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionDeny},
)
http.Error( http.Error(
rq.out, rq.out,
http.StatusText(http.StatusNotFound), http.StatusText(http.StatusNotFound),
@@ -147,19 +218,45 @@ func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
rq.banAndCount(r.Context(), rq.h.afterAnswer) rq.banAndCount(r.Context(), rq.h.afterAnswer)
} }
// readQuestion reads a question, and reports whether waf takes it. It answers // readQuestion returns the visitor's address and the body a question carries,
// a question it does not take with 400 and no disposition. rproxy passes that // and whether waf takes the question. It answers a question it does not
// 400 to the visitor, and SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION does // take with 400 and no disposition. rproxy passes that 400 to the visitor,
// not apply. The id of a question waf takes is held from here on, without a // and SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION does not apply. The id of
// decision yet, so that another question with that id is refused while this // a question waf takes is held from here on, without a decision yet, so
// one is being handled. // that another question with that id is refused while this one is being
// handled.
// //
// A question whose body stops arriving gets no answer, and is not looked at. // The body is read before any check, since Go's server does not send an answer
// until it has read the body. A question whose body stops arriving gets no
// answer, and is not looked at.
func (h *handler) readQuestion( func (h *handler) readQuestion(
w http.ResponseWriter, r *http.Request, w http.ResponseWriter, r *http.Request,
) (wafprotocol.Question, bool) { ) (netip.Addr, []byte, bool) {
question, err := wafprotocol.ReadQuestion(w, r) client, err := netip.ParseAddr(r.Header.Get(ClientHeader))
if errors.Is(err, wafprotocol.ErrBodyStoppedArriving) { if err != nil || client.Zone() != "" {
h.refuseQuestion(w, ClientHeader+" is not an address without a zone")
return netip.Addr{}, nil, false
}
id := r.Header.Get(requestIDHeader)
if id == "" || len(id) > maxRequestIDBytes || !utf8.ValidString(id) {
h.refuseQuestion(w, requestIDHeader+" is not a valid id")
return netip.Addr{}, nil, false
}
body, err := io.ReadAll(
http.MaxBytesReader(w, r.Body, QuestionBodyMaxBytes),
)
if _, tooLarge := errors.AsType[*http.MaxBytesError](err); tooLarge {
h.refuseQuestion(w, "the body is over 64 KiB")
return netip.Addr{}, nil, false
}
if err != nil {
h.processLog.Warn( h.processLog.Warn(
"a question's body stopped arriving", "a question's body stopped arriving",
"error", "error",
@@ -170,23 +267,17 @@ func (h *handler) readQuestion(
panic(http.ErrAbortHandler) panic(http.ErrAbortHandler)
} }
if err != nil { held, _ := h.awaitingReport.ContainsOrAdd(id, nil)
h.refuseQuestion(w, err.Error())
return question, false
}
held, _ := h.awaitingReport.ContainsOrAdd(question.RequestID, nil)
if held { if held {
h.refuseQuestion( h.refuseQuestion(
w, w,
wafprotocol.RequestIDHeader+" is the id of a request waf holds", requestIDHeader+" is the id of a request waf holds",
) )
return question, false return netip.Addr{}, nil, false
} }
return question, true return client, body, true
} }
// refuseQuestion answers a question waf does not take, for reason. // refuseQuestion answers a question waf does not take, for reason.
@@ -195,18 +286,48 @@ func (h *handler) refuseQuestion(w http.ResponseWriter, reason string) {
http.Error(w, reason, http.StatusBadRequest) http.Error(w, reason, http.StatusBadRequest)
} }
// allow answers a question with wafprotocol.DispositionAllow and, while // asTheVisitorSentIt returns a copy of r, a question, as the checks and the
// log line are to see it: without the headers rproxy adds for waf,
// with body, which has been read of r, and with the length the visitor
// announced, as -1 when it announced none. The question's own length covers
// only the part of the body it carries.
func asTheVisitorSentIt(r *http.Request, body []byte) *http.Request {
sent := r.Clone(r.Context())
announced := sent.Header.Get(ContentLengthHeader)
sent.Header.Del(ClientHeader)
sent.Header.Del(PeerHeader)
sent.Header.Del(ContentLengthHeader)
sent.Header.Del("Content-Length")
sent.ContentLength = -1
length, err := strconv.ParseInt(announced, 10, 64)
if err == nil && length >= 0 {
sent.ContentLength = length
sent.Header.Set("Content-Length", announced)
}
sent.Body = http.NoBody
if len(body) > 0 {
sent.Body = io.NopCloser(bytes.NewReader(body))
}
return sent
}
// allow answers a question with DispositionAllow and, while
// SWW_WAF_ADD_LOOKUP_HEADERS is true, the client's AS number and country. // SWW_WAF_ADD_LOOKUP_HEADERS is true, the client's AS number and country.
func (rq *request) allow() { func (rq *request) allow() {
rq.allowed = true rq.allowed = true
answer := wafprotocol.Answer{Disposition: wafprotocol.DispositionAllow} header := rq.out.Header()
header.Set(DispositionHeader, DispositionAllow)
if rq.h.config.AddLookupHeaders { if rq.h.config.AddLookupHeaders {
answer.ASN = rq.line.ASN setLookupHeaders(header, rq.line.ASN, rq.line.Country)
answer.Country = rq.line.Country
} }
wafprotocol.SetAnswerHeaders(rq.out.Header(), answer)
rq.out.WriteHeader(http.StatusOK) rq.out.WriteHeader(http.StatusOK)
} }
@@ -224,19 +345,13 @@ func (rq *request) deny(r refusal) {
if r.status == 0 { if r.status == 0 {
// SWW_WAF_BAN_RESPONSE is close. Written past rq.out, since the log line's // SWW_WAF_BAN_RESPONSE is close. Written past rq.out, since the log line's
// status stays 0: the visitor is sent nothing. // status stays 0: the visitor is sent nothing.
wafprotocol.SetAnswerHeaders( rq.out.Header().Set(DispositionHeader, DispositionClose)
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionClose},
)
rq.out.ResponseWriter.WriteHeader(http.StatusOK) rq.out.ResponseWriter.WriteHeader(http.StatusOK)
return return
} }
wafprotocol.SetAnswerHeaders( rq.out.Header().Set(DispositionHeader, DispositionDeny)
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionDeny},
)
http.Error(rq.out, http.StatusText(r.status), r.status) http.Error(rq.out, http.StatusText(r.status), r.status)
} }
@@ -244,7 +359,7 @@ func (rq *request) deny(r refusal) {
// runs on it: the address is on the private network, where the peers are // runs on it: the address is on the private network, where the peers are
// rproxy and the admin's tools, not visitors. // rproxy and the admin's tools, not visitors.
func (h *handler) answerOwnEndpoint(w http.ResponseWriter, r *http.Request) { func (h *handler) answerOwnEndpoint(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodPost && r.URL.Path == wafprotocol.ReportPath { if r.Method == http.MethodPost && r.URL.Path == ReportPath {
h.takeReport(w, r) h.takeReport(w, r)
return return
@@ -255,7 +370,7 @@ func (h *handler) answerOwnEndpoint(w http.ResponseWriter, r *http.Request) {
rq := h.startRequest(w, r, peer, peer, false) rq := h.startRequest(w, r, peer, peer, false)
defer rq.finish() defer rq.finish()
if r.Method == http.MethodGet && r.URL.Path == wafprotocol.HealthPath { if r.Method == http.MethodGet && r.URL.Path == HealthPath {
rq.answerHealth() rq.answerHealth()
return return
@@ -264,11 +379,11 @@ func (h *handler) answerOwnEndpoint(w http.ResponseWriter, r *http.Request) {
rq.answerAdmin() rq.answerAdmin()
} }
// takeReport answers POST wafprotocol.ReportPath: it finishes the request the // takeReport answers POST ReportPath: it finishes the request the report
// report names, if it holds that request's decision, and ignores the report // names, if it holds that request's decision, and ignores the report
// otherwise. // otherwise.
func (h *handler) takeReport(w http.ResponseWriter, r *http.Request) { func (h *handler) takeReport(w http.ResponseWriter, r *http.Request) {
report, err := wafprotocol.ReadReport(w, r) report, err := readReport(w, r)
if err != nil { if err != nil {
h.processLog.Warn("a report is refused", "error", err.Error()) h.processLog.Warn("a report is refused", "error", err.Error())
http.Error(w, "the body is not a report", http.StatusBadRequest) http.Error(w, "the body is not a report", http.StatusBadRequest)
@@ -286,6 +401,63 @@ func (h *handler) takeReport(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusNoContent) w.WriteHeader(http.StatusNoContent)
} }
// readReport reads a report from r's body, which must be one JSON object with
// all of Report's fields, in the ranges SPEC.md gives.
func readReport(w http.ResponseWriter, r *http.Request) (Report, error) {
var report Report
body, err := io.ReadAll(
http.MaxBytesReader(w, r.Body, reportBodyMaxBytes),
)
if err != nil {
return report, fmt.Errorf("read the body: %w", err)
}
var fields map[string]json.RawMessage
err = json.Unmarshal(body, &fields)
if err != nil {
return report, fmt.Errorf("decode the body: %w", err)
}
for _, name := range []string{
"request_id", "status", "request_bytes", "response_bytes",
"duration_total_ms", "duration_upstream_total_ms",
} {
value, there := fields[name]
if !there || string(value) == "null" {
return report, fmt.Errorf("%w: %s", errReportFieldMissing, name)
}
}
err = json.Unmarshal(body, &report)
if err != nil {
return report, fmt.Errorf("decode the body: %w", err)
}
return report, report.checkRanges()
}
// checkRanges returns an error for a value of the report that cannot be true.
func (report *Report) checkRanges() error {
switch {
case report.Status != 0 &&
(report.Status < http.StatusContinue || report.Status > maxHTTPStatus):
return fmt.Errorf("%w: status", errReportValue)
case report.RequestBytes < 0 || report.RequestBytes > maxReportedBytes:
return fmt.Errorf("%w: request_bytes", errReportValue)
case report.ResponseBytes < 0 || report.ResponseBytes > maxReportedBytes:
return fmt.Errorf("%w: response_bytes", errReportValue)
case report.DurationTotalMS < 0 || report.DurationTotalMS > maxReportedMilliseconds:
return fmt.Errorf("%w: duration_total_ms", errReportValue)
case report.DurationUpstreamTotalMS < 0 ||
report.DurationUpstreamTotalMS > maxReportedMilliseconds:
return fmt.Errorf("%w: duration_upstream_total_ms", errReportValue)
}
return nil
}
// finishWith finishes a request with what its report says, and writes the // finishWith finishes a request with what its report says, and writes the
// request's last log line, which holds its decision's fields and the report's. // request's last log line, which holds its decision's fields and the report's.
// For an allowed request it first runs the heuristics of onReport, and adds the // For an allowed request it first runs the heuristics of onReport, and adds the
@@ -295,7 +467,7 @@ func (h *handler) takeReport(w http.ResponseWriter, r *http.Request) {
func (h *handler) finishWith( func (h *handler) finishWith(
ctx context.Context, ctx context.Context,
held *decision, held *decision,
report wafprotocol.Report, report Report,
) { ) {
rq := &request{ rq := &request{
decision: *held, decision: *held,
+86 -195
View File
@@ -22,7 +22,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
const ( const (
@@ -107,8 +106,8 @@ func (s *WAF) questionWith(
req := newRequest(s.t, method, s.addr, path, body) req := newRequest(s.t, method, s.addr, path, body)
req.Host = host req.Host = host
req.Header.Set(wafprotocol.ClientHeader, from) req.Header.Set(waf.ClientHeader, from)
req.Header.Set(wafprotocol.PeerHeader, rproxyPeer) req.Header.Set(waf.PeerHeader, rproxyPeer)
req.Header.Set(requestIDHeader, rand.Text()) req.Header.Set(requestIDHeader, rand.Text())
return req return req
@@ -135,18 +134,13 @@ func (s *WAF) allowed(from string) string {
s.t.Helper() s.t.Helper()
req := s.question(siteA, from, "/") req := s.question(siteA, from, "/")
wantDisposition( wantDisposition(s.t, do(s.t, req), http.StatusOK, waf.DispositionAllow)
s.t,
do(s.t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
)
return req.Header.Get(requestIDHeader) return req.Header.Get(requestIDHeader)
} }
// report sends rproxy's report and returns the answer. // report sends rproxy's report and returns the answer.
func (s *WAF) report(report wafprotocol.Report) answer { func (s *WAF) report(report waf.Report) answer {
s.t.Helper() s.t.Helper()
body, err := json.Marshal(report) body, err := json.Marshal(report)
@@ -161,11 +155,8 @@ func (s *WAF) report(report wafprotocol.Report) answer {
func (s *WAF) reportBody(body string) answer { func (s *WAF) reportBody(body string) answer {
s.t.Helper() s.t.Helper()
return do( return do(s.t, newRequest(s.t, http.MethodPost, s.addr, waf.ReportPath,
s.t, strings.NewReader(body)))
newRequest(s.t, http.MethodPost, s.addr, wafprotocol.ReportPath,
strings.NewReader(body)),
)
} }
// passedOn asks about a request from the visitor at from on siteA, and reports // passedOn asks about a request from the visitor at from on siteA, and reports
@@ -173,7 +164,7 @@ func (s *WAF) reportBody(body string) answer {
func (s *WAF) passedOn(from string) { func (s *WAF) passedOn(from string) {
s.t.Helper() s.t.Helper()
got := s.report(wafprotocol.Report{ got := s.report(waf.Report{
RequestID: s.allowed(from), RequestID: s.allowed(from),
Status: http.StatusOK, Status: http.StatusOK,
RequestBytes: bodyBytes, RequestBytes: bodyBytes,
@@ -219,7 +210,7 @@ func wantDisposition(
t.Helper() t.Helper()
wantStatus(t, got, status) wantStatus(t, got, status)
if answered := got.header.Get(wafprotocol.DispositionHeader); answered != if answered := got.header.Get(waf.DispositionHeader); answered !=
disposition { disposition {
t.Errorf( t.Errorf(
"answered with disposition %q, want %q", "answered with disposition %q, want %q",
@@ -281,10 +272,10 @@ func TestQuestionAllowedIsAnsweredAndLogged(t *testing.T) {
req := s.question(siteA, client, "/page?q=1&token=visible") req := s.question(siteA, client, "/page?q=1&token=visible")
req.Header.Set(requestIDHeader, "the-id") req.Header.Set(requestIDHeader, "the-id")
req.Header.Set("X-Forwarded-Proto", "https") req.Header.Set("X-Forwarded-Proto", "https")
req.Header.Set(wafprotocol.ContentLengthHeader, "1234") req.Header.Set(waf.ContentLengthHeader, "1234")
got := do(t, req) got := do(t, req)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow) wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
wantBody(t, got, "") wantBody(t, got, "")
if id := got.header.Get(requestIDHeader); id != "" { if id := got.header.Get(requestIDHeader); id != "" {
@@ -317,12 +308,7 @@ func TestIPv6VisitorIsCountedInItsGroup(t *testing.T) {
s.allowed("2001:db8::1") s.allowed("2001:db8::1")
got := s.ask("2001:db8::2", "/") got := s.ask("2001:db8::2", "/")
wantDisposition( wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
} }
func TestIPv4VisitorInIPv6FormIsTheIPv4Client(t *testing.T) { func TestIPv4VisitorInIPv6FormIsTheIPv4Client(t *testing.T) {
@@ -331,14 +317,9 @@ func TestIPv4VisitorInIPv6FormIsTheIPv4Client(t *testing.T) {
s := startWAF(t, "", map[string]string{denyNets: client}) s := startWAF(t, "", map[string]string{denyNets: client})
req := s.question(siteA, "::ffff:"+client, "/") req := s.question(siteA, "::ffff:"+client, "/")
req.Header.Set(wafprotocol.PeerHeader, "::ffff:"+rproxyPeer) req.Header.Set(waf.PeerHeader, "::ffff:"+rproxyPeer)
wantDisposition( wantDisposition(t, do(t, req), http.StatusForbidden, waf.DispositionDeny)
t,
do(t, req),
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
line := s.out.requestLine(t) line := s.out.requestLine(t)
if line.ClientIP != client || line.PeerIP != rproxyPeer { if line.ClientIP != client || line.PeerIP != rproxyPeer {
@@ -358,13 +339,13 @@ func TestMalformedQuestionGets400AndAWarningAndNoLine(t *testing.T) {
change func(*http.Request) change func(*http.Request)
}{ }{
{"client not an address", func(req *http.Request) { {"client not an address", func(req *http.Request) {
req.Header.Set(wafprotocol.ClientHeader, "not an address") req.Header.Set(waf.ClientHeader, "not an address")
}}, }},
{"client empty", func(req *http.Request) { {"client empty", func(req *http.Request) {
req.Header.Set(wafprotocol.ClientHeader, "") req.Header.Set(waf.ClientHeader, "")
}}, }},
{"client address with a zone", func(req *http.Request) { {"client address with a zone", func(req *http.Request) {
req.Header.Set(wafprotocol.ClientHeader, "2001:db8::1%eth0") req.Header.Set(waf.ClientHeader, "2001:db8::1%eth0")
}}, }},
{"id absent", func(req *http.Request) { {"id absent", func(req *http.Request) {
req.Header.Del(requestIDHeader) req.Header.Del(requestIDHeader)
@@ -409,12 +390,7 @@ func TestLongestValidIDIsTaken(t *testing.T) {
req := s.question(siteA, client, "/") req := s.question(siteA, client, "/")
req.Header.Set(requestIDHeader, strings.Repeat("x", 128)) req.Header.Set(requestIDHeader, strings.Repeat("x", 128))
wantDisposition( wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow)
t,
do(t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
)
} }
func TestQuestionWithABodyOver64KiBGets400(t *testing.T) { func TestQuestionWithABodyOver64KiBGets400(t *testing.T) {
@@ -423,18 +399,16 @@ func TestQuestionWithABodyOver64KiBGets400(t *testing.T) {
s := startWAF(t, "", nil) s := startWAF(t, "", nil)
over := strings.NewReader( over := strings.NewReader(
strings.Repeat("x", wafprotocol.QuestionBodyMaxBytes+1), strings.Repeat("x", waf.QuestionBodyMaxBytes+1),
) )
got := do(t, s.questionWith(http.MethodPost, siteA, client, "/", over)) got := do(t, s.questionWith(http.MethodPost, siteA, client, "/", over))
wantDisposition(t, got, http.StatusBadRequest, "") wantDisposition(t, got, http.StatusBadRequest, "")
wantProcessMessages(t, s, "a question is refused", 1) wantProcessMessages(t, s, "a question is refused", 1)
wantNoRequestLines(t, s) wantNoRequestLines(t, s)
whole := strings.NewReader( whole := strings.NewReader(strings.Repeat("x", waf.QuestionBodyMaxBytes))
strings.Repeat("x", wafprotocol.QuestionBodyMaxBytes),
)
got = do(t, s.questionWith(http.MethodPost, siteA, client, "/", whole)) got = do(t, s.questionWith(http.MethodPost, siteA, client, "/", whole))
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow) wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
} }
func TestQuestionWithAHeldIDIsRefused(t *testing.T) { func TestQuestionWithAHeldIDIsRefused(t *testing.T) {
@@ -450,12 +424,12 @@ func TestQuestionWithAHeldIDIsRefused(t *testing.T) {
wantProcessMessages(t, s, "a question is refused", 1) wantProcessMessages(t, s, "a question is refused", 1)
// A denied request's id is held as well. // A denied request's id is held as well.
denied := s.question(siteA, otherClient, wafprotocol.HealthPath) denied := s.question(siteA, otherClient, waf.HealthPath)
wantDisposition( wantDisposition(
t, t,
do(t, denied), do(t, denied),
http.StatusNotFound, http.StatusNotFound,
wafprotocol.DispositionDeny, waf.DispositionDeny,
) )
again = s.question(siteA, client, "/") again = s.question(siteA, client, "/")
@@ -463,9 +437,7 @@ func TestQuestionWithAHeldIDIsRefused(t *testing.T) {
wantDisposition(t, do(t, again), http.StatusBadRequest, "") wantDisposition(t, do(t, again), http.StatusBadRequest, "")
// The report ends the holding, and the refused questions changed nothing. // The report ends the holding, and the refused questions changed nothing.
got := s.report( got := s.report(waf.Report{RequestID: id, ResponseBytes: answerBytes})
wafprotocol.Report{RequestID: id, ResponseBytes: answerBytes},
)
wantStatus(t, got, http.StatusNoContent) wantStatus(t, got, http.StatusNoContent)
if history := s.history(); history.ResponseBytes != answerBytes { if history := s.history(); history.ResponseBytes != answerBytes {
@@ -474,12 +446,7 @@ func TestQuestionWithAHeldIDIsRefused(t *testing.T) {
again = s.question(siteA, client, "/") again = s.question(siteA, client, "/")
again.Header.Set(requestIDHeader, id) again.Header.Set(requestIDHeader, id)
wantDisposition( wantDisposition(t, do(t, again), http.StatusOK, waf.DispositionAllow)
t,
do(t, again),
http.StatusOK,
wafprotocol.DispositionAllow,
)
} }
func TestReportNeverReachesAQuestionStillBeingHandled(t *testing.T) { func TestReportNeverReachesAQuestionStillBeingHandled(t *testing.T) {
@@ -514,7 +481,7 @@ func TestReportNeverReachesAQuestionStillBeingHandled(t *testing.T) {
// A report for a question not yet answered is ignored, and that question's // A report for a question not yet answered is ignored, and that question's
// id is taken. // id is taken.
got := s.report(wafprotocol.Report{RequestID: id, ResponseBytes: 1000}) got := s.report(waf.Report{RequestID: id, ResponseBytes: 1000})
wantStatus(t, got, http.StatusNoContent) wantStatus(t, got, http.StatusNoContent)
// This would wait on the same lookup were the id not taken. // This would wait on the same lookup were the id not taken.
@@ -529,7 +496,7 @@ func TestReportNeverReachesAQuestionStillBeingHandled(t *testing.T) {
select { select {
case got := <-answered: case got := <-answered:
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow) wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
case <-time.After(waitLimit): case <-time.After(waitLimit):
t.Fatal("the question got no answer") t.Fatal("the question got no answer")
} }
@@ -548,9 +515,7 @@ func TestReportNeverReachesAQuestionStillBeingHandled(t *testing.T) {
} }
// The true report comes after, and is the one that counts. // The true report comes after, and is the one that counts.
got = s.report( got = s.report(waf.Report{RequestID: id, ResponseBytes: answerBytes})
wafprotocol.Report{RequestID: id, ResponseBytes: answerBytes},
)
wantStatus(t, got, http.StatusNoContent) wantStatus(t, got, http.StatusNoContent)
if history := s.history(); history.ResponseBytes != answerBytes { if history := s.history(); history.ResponseBytes != answerBytes {
@@ -568,17 +533,17 @@ func TestPeerAbsentOrUnreadableIsAnsweredAndLoggedInvalid(t *testing.T) {
s := startWAF(t, "", nil) s := startWAF(t, "", nil)
req := s.question(siteA, client, "/") req := s.question(siteA, client, "/")
req.Header.Del(wafprotocol.PeerHeader) req.Header.Del(waf.PeerHeader)
if peer != "" { if peer != "" {
req.Header.Set(wafprotocol.PeerHeader, peer) req.Header.Set(waf.PeerHeader, peer)
} }
wantDisposition( wantDisposition(
t, t,
do(t, req), do(t, req),
http.StatusOK, http.StatusOK,
wafprotocol.DispositionAllow, waf.DispositionAllow,
) )
if line := s.out.requestLine(t); line.PeerIP != "invalid IP" { if line := s.out.requestLine(t); line.PeerIP != "invalid IP" {
@@ -606,12 +571,12 @@ func TestContentLengthUnreadableOrNegativeCountsAsAbsent(t *testing.T) {
}) })
req := s.question(siteA, client, "/") req := s.question(siteA, client, "/")
req.Header.Set(wafprotocol.ContentLengthHeader, announced) req.Header.Set(waf.ContentLengthHeader, announced)
wantDisposition( wantDisposition(
t, t,
do(t, req), do(t, req),
http.StatusOK, http.StatusOK,
wafprotocol.DispositionAllow, waf.DispositionAllow,
) )
if line := s.out.requestLine(t); line.ContentLength != 0 { if line := s.out.requestLine(t); line.ContentLength != 0 {
@@ -630,18 +595,13 @@ func TestHeaderSentTwiceUsesItsFirstValue(t *testing.T) {
s := startWAF(t, "", nil) s := startWAF(t, "", nil)
req := s.question(siteA, client, "/") req := s.question(siteA, client, "/")
req.Header.Add(wafprotocol.ClientHeader, otherClient) req.Header.Add(waf.ClientHeader, otherClient)
req.Header.Add(wafprotocol.PeerHeader, "172.18.0.3") req.Header.Add(waf.PeerHeader, "172.18.0.3")
req.Header.Set(requestIDHeader, "first") req.Header.Set(requestIDHeader, "first")
req.Header.Add(requestIDHeader, "second") req.Header.Add(requestIDHeader, "second")
req.Header.Set(wafprotocol.ContentLengthHeader, "1234") req.Header.Set(waf.ContentLengthHeader, "1234")
req.Header.Add(wafprotocol.ContentLengthHeader, "5678") req.Header.Add(waf.ContentLengthHeader, "5678")
wantDisposition( wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow)
t,
do(t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
)
line := s.out.requestLine(t) line := s.out.requestLine(t)
if line.ClientIP != client || line.PeerIP != rproxyPeer || if line.ClientIP != client || line.PeerIP != rproxyPeer ||
@@ -670,7 +630,7 @@ func sendQuestionWithPartOfItsBody(t *testing.T, conn net.Conn) {
_, err := fmt.Fprintf(conn, _, err := fmt.Fprintf(conn,
"POST / HTTP/1.1\r\nHost: %s\r\n%s: %s\r\n%s: an-id\r\n"+ "POST / HTTP/1.1\r\nHost: %s\r\n%s: %s\r\n%s: an-id\r\n"+
"Content-Length: 100\r\n\r\npart", "Content-Length: 100\r\n\r\npart",
siteA, wafprotocol.ClientHeader, client, requestIDHeader) siteA, waf.ClientHeader, client, requestIDHeader)
if err != nil { if err != nil {
t.Fatalf("send the question: %v", err) t.Fatalf("send the question: %v", err)
} }
@@ -739,7 +699,7 @@ func TestQuestionWhoseBodyStopsArrivingGetsNoAnswer(t *testing.T) {
t, t,
do(t, again), do(t, again),
http.StatusOK, http.StatusOK,
wafprotocol.DispositionAllow, waf.DispositionAllow,
) )
}) })
} }
@@ -757,7 +717,7 @@ func TestQuestionDeniedGetsTheBanResponse(t *testing.T) {
}) })
got := s.ask(client, "/") got := s.ask(client, "/")
wantDisposition(t, got, status, wafprotocol.DispositionDeny) wantDisposition(t, got, status, waf.DispositionDeny)
wantBody(t, got, http.StatusText(status)+"\n") wantBody(t, got, http.StatusText(status)+"\n")
wantLine( wantLine(
t, t,
@@ -777,7 +737,7 @@ func TestBanResponseCloseIsAnAnswerOfItsOwn(t *testing.T) {
}) })
got := s.ask(client, "/") got := s.ask(client, "/")
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionClose) wantDisposition(t, got, http.StatusOK, waf.DispositionClose)
wantBody(t, got, "") wantBody(t, got, "")
// The visitor is sent nothing. // The visitor is sent nothing.
wantLine(t, s.out.requestLine(t), 0, requestlog.ActionClientDenied) wantLine(t, s.out.requestLine(t), 0, requestlog.ActionClientDenied)
@@ -801,22 +761,12 @@ func TestBlockRuleIsDeniedWith403AndABanRuleWithTheBanResponse(
}) })
got := s.ask(client, "/blocked") got := s.ask(client, "/blocked")
wantDisposition( wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
wantLine(t, s.out.requestLine(t), http.StatusForbidden, wantLine(t, s.out.requestLine(t), http.StatusForbidden,
requestlog.ActionBlockedByBlockRule) requestlog.ActionBlockedByBlockRule)
got = s.ask(client, "/.env") got = s.ask(client, "/.env")
wantDisposition( wantDisposition(t, got, http.StatusTooManyRequests, waf.DispositionDeny)
t,
got,
http.StatusTooManyRequests,
wafprotocol.DispositionDeny,
)
wantLine(t, s.out.requestLines(t, 2)[1], http.StatusTooManyRequests, wantLine(t, s.out.requestLines(t, 2)[1], http.StatusTooManyRequests,
requestlog.ActionTemporarilyBanned) requestlog.ActionTemporarilyBanned)
} }
@@ -831,15 +781,10 @@ func TestQuestionUnderTheOwnPrefixIsDeniedWith404AfterTheChecks(
}) })
for _, path := range []string{ for _, path := range []string{
wafprotocol.HealthPath, waf.MetricsPath, wafprotocol.ReportPath, waf.HealthPath, waf.MetricsPath, waf.ReportPath,
} { } {
got := s.ask(client, path) got := s.ask(client, path)
wantDisposition( wantDisposition(t, got, http.StatusNotFound, waf.DispositionDeny)
t,
got,
http.StatusNotFound,
wafprotocol.DispositionDeny,
)
} }
wantLine( wantLine(
@@ -850,12 +795,7 @@ func TestQuestionUnderTheOwnPrefixIsDeniedWith404AfterTheChecks(
) )
got := s.ask(otherClient, waf.MetricsPath) got := s.ask(otherClient, waf.MetricsPath)
wantDisposition( wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
} }
func TestObserveModeAllowsAndLogsWhatEnforceModeWouldHaveDone(t *testing.T) { func TestObserveModeAllowsAndLogsWhatEnforceModeWouldHaveDone(t *testing.T) {
@@ -864,7 +804,7 @@ func TestObserveModeAllowsAndLogsWhatEnforceModeWouldHaveDone(t *testing.T) {
s := startWAF(t, "", map[string]string{mode: observe, denyNets: client}) s := startWAF(t, "", map[string]string{mode: observe, denyNets: client})
got := s.ask(client, "/") got := s.ask(client, "/")
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow) wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
line := s.out.requestLine(t) line := s.out.requestLine(t)
wantLine(t, line, http.StatusOK, requestlog.ActionForward) wantLine(t, line, http.StatusOK, requestlog.ActionForward)
@@ -882,7 +822,7 @@ func TestObserveModeRefusalAnsweredAllowCountsForTheRefusalLimit(
for range 2 { for range 2 {
got := s.ask(client, "/blocked") got := s.ask(client, "/blocked")
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow) wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
} }
lines := s.out.requestLines(t, 2) lines := s.out.requestLines(t, 2)
@@ -906,12 +846,7 @@ func TestDeniedQuestionCountsForTheRefusalLimit(t *testing.T) {
for range 2 { for range 2 {
got := s.ask(client, "/blocked") got := s.ask(client, "/blocked")
wantDisposition( wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
} }
lines := s.out.requestLines(t, 2) lines := s.out.requestLines(t, 2)
@@ -947,12 +882,7 @@ func TestAllowCarriesTheLookupHeadersWhileTheSettingIsSet(t *testing.T) {
}) })
got := s.ask(fromDE, "/") got := s.ask(fromDE, "/")
wantDisposition( wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
t,
got,
http.StatusOK,
wafprotocol.DispositionAllow,
)
asn, country := got.header.Get( asn, country := got.header.Get(
"X-Client-Asn", "X-Client-Asn",
@@ -986,7 +916,7 @@ func TestReportCountsTowardAByteLimitWhichThenBans(t *testing.T) {
} }
// 100 bytes are over the limit of 99. // 100 bytes are over the limit of 99.
got := s.report(wafprotocol.Report{ got := s.report(waf.Report{
RequestID: id, Status: http.StatusBadGateway, RequestID: id, Status: http.StatusBadGateway,
RequestBytes: bodyBytes, ResponseBytes: answerBytes, RequestBytes: bodyBytes, ResponseBytes: answerBytes,
}) })
@@ -1010,12 +940,7 @@ func TestReportCountsTowardAByteLimitWhichThenBans(t *testing.T) {
} }
got = s.ask(client, "/") got = s.ask(client, "/")
wantDisposition( wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
} }
func TestReportCountsAtThePercentageItsQuestionWasGiven(t *testing.T) { func TestReportCountsAtThePercentageItsQuestionWasGiven(t *testing.T) {
@@ -1033,12 +958,12 @@ func TestReportCountsAtThePercentageItsQuestionWasGiven(t *testing.T) {
t, t,
s.ask(fromKP, "/"), s.ask(fromKP, "/"),
http.StatusOK, http.StatusOK,
wafprotocol.DispositionAllow, waf.DispositionAllow,
) )
s.passedOn(fromDE) s.passedOn(fromDE)
wantDisposition(t, s.ask(fromDE, "/"), http.StatusForbidden, wantDisposition(t, s.ask(fromDE, "/"), http.StatusForbidden,
wafprotocol.DispositionDeny) waf.DispositionDeny)
} }
func TestReportAddsTheRequestToItsClientsHistoryAndTheAnomalyCounters( func TestReportAddsTheRequestToItsClientsHistoryAndTheAnomalyCounters(
@@ -1064,7 +989,7 @@ func TestReportAddsTheRequestToItsClientsHistoryAndTheAnomalyCounters(
) )
} }
got := s.report(wafprotocol.Report{ got := s.report(waf.Report{
RequestID: id, Status: http.StatusBadGateway, RequestID: id, Status: http.StatusBadGateway,
RequestBytes: bodyBytes, ResponseBytes: answerBytes, RequestBytes: bodyBytes, ResponseBytes: answerBytes,
}) })
@@ -1096,7 +1021,7 @@ func TestDeniedRequestIsFinishedAtItsDecision(t *testing.T) {
t, t,
do(t, denied), do(t, denied),
http.StatusForbidden, http.StatusForbidden,
wafprotocol.DispositionDeny, waf.DispositionDeny,
) )
counters := s.server.Anomalies.Snapshot() counters := s.server.Anomalies.Snapshot()
@@ -1114,7 +1039,7 @@ func TestDeniedRequestIsFinishedAtItsDecision(t *testing.T) {
wantMetric(t, s.metrics(), series, 1) wantMetric(t, s.metrics(), series, 1)
got := s.report(wafprotocol.Report{ got := s.report(waf.Report{
RequestID: denied.Header.Get( RequestID: denied.Header.Get(
requestIDHeader, requestIDHeader,
), Status: http.StatusForbidden, ), Status: http.StatusForbidden,
@@ -1157,7 +1082,7 @@ func TestMetricsCountAnAllowedRequestFromItsReport(t *testing.T) {
1, 1,
) // the scrape itself ) // the scrape itself
got := s.report(wafprotocol.Report{ got := s.report(waf.Report{
RequestID: id, Status: http.StatusBadGateway, RequestID: id, Status: http.StatusBadGateway,
RequestBytes: bodyBytes, ResponseBytes: answerBytes, RequestBytes: bodyBytes, ResponseBytes: answerBytes,
DurationTotalMS: 5, DurationUpstreamTotalMS: 4, DurationTotalMS: 5, DurationUpstreamTotalMS: 4,
@@ -1175,7 +1100,7 @@ func TestMetricsCountAnAllowedRequestFromItsReport(t *testing.T) {
// check needs a server that has not been scraped yet. // check needs a server that has not been scraped yet.
unscraped := startWAF(t, "", map[string]string{metricsToken: token}) unscraped := startWAF(t, "", map[string]string{metricsToken: token})
got = unscraped.report(wafprotocol.Report{ got = unscraped.report(waf.Report{
RequestID: unscraped.allowed(client), Status: http.StatusBadGateway, RequestID: unscraped.allowed(client), Status: http.StatusBadGateway,
DurationTotalMS: 5, DurationTotalMS: 5,
}) })
@@ -1193,7 +1118,7 @@ func TestReportForARequestNotHeldIsIgnored(t *testing.T) {
s := startWAF(t, "", map[string]string{bytesLimitPerMinute: byteLimit}) s := startWAF(t, "", map[string]string{bytesLimitPerMinute: byteLimit})
got := s.report(wafprotocol.Report{ got := s.report(waf.Report{
RequestID: "no-such-request", ResponseBytes: 1000, RequestID: "no-such-request", ResponseBytes: 1000,
}) })
wantStatus(t, got, http.StatusNoContent) wantStatus(t, got, http.StatusNoContent)
@@ -1212,9 +1137,7 @@ func TestReportForARequestNotHeldIsIgnored(t *testing.T) {
// A second report of the same request does not count it twice. // A second report of the same request does not count it twice.
id := s.allowed(client) id := s.allowed(client)
for range 2 { for range 2 {
got = s.report( got = s.report(waf.Report{RequestID: id, ResponseBytes: answerBytes})
wafprotocol.Report{RequestID: id, ResponseBytes: answerBytes},
)
wantStatus(t, got, http.StatusNoContent) wantStatus(t, got, http.StatusNoContent)
} }
@@ -1424,7 +1347,7 @@ func TestBytesAtTheirMostEachWayBanRatherThanOverflow(t *testing.T) {
s := startWAF(t, "", map[string]string{bytesLimitPerMinute: byteLimit}) s := startWAF(t, "", map[string]string{bytesLimitPerMinute: byteLimit})
got := s.report(wafprotocol.Report{ got := s.report(waf.Report{
RequestID: s.allowed(client), Status: http.StatusOK, RequestID: s.allowed(client), Status: http.StatusOK,
RequestBytes: reportedBytesLimit, ResponseBytes: reportedBytesLimit, RequestBytes: reportedBytesLimit, ResponseBytes: reportedBytesLimit,
}) })
@@ -1518,14 +1441,9 @@ func TestReportWritesTheLastLineWithTheDecisionsAndTheReportsFields(
req.Header.Set("X-Forwarded-Proto", "https") req.Header.Set("X-Forwarded-Proto", "https")
id := req.Header.Get(requestIDHeader) id := req.Header.Get(requestIDHeader)
wantDisposition( wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow)
t,
do(t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
)
got := s.report(wafprotocol.Report{ got := s.report(waf.Report{
RequestID: id, Status: http.StatusCreated, RequestID: id, Status: http.StatusCreated,
RequestBytes: bodyBytes, ResponseBytes: answerBytes, RequestBytes: bodyBytes, ResponseBytes: answerBytes,
DurationTotalMS: 12.5, DurationUpstreamTotalMS: 10.25, DurationTotalMS: 12.5, DurationUpstreamTotalMS: 10.25,
@@ -1572,7 +1490,7 @@ func TestEveryDeniedRequestsReportGetsItsLastLine(t *testing.T) {
"close", map[string]string{denyNets: client, banResponse: banResponseClose}, "/", "close", map[string]string{denyNets: client, banResponse: banResponseClose}, "/",
0, requestlog.ActionClientDenied, 0, requestlog.ActionClientDenied,
}, },
{"own prefix", nil, wafprotocol.HealthPath, http.StatusNotFound, {"own prefix", nil, waf.HealthPath, http.StatusNotFound,
requestlog.ActionOwnEndpoint}, requestlog.ActionOwnEndpoint},
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
@@ -1584,7 +1502,7 @@ func TestEveryDeniedRequestsReportGetsItsLastLine(t *testing.T) {
id := req.Header.Get(requestIDHeader) id := req.Header.Get(requestIDHeader)
do(t, req) do(t, req)
got := s.report(wafprotocol.Report{ got := s.report(waf.Report{
RequestID: id, Status: tc.status, ResponseBytes: 8, DurationTotalMS: 3, RequestID: id, Status: tc.status, ResponseBytes: 8, DurationTotalMS: 3,
}) })
wantStatus(t, got, http.StatusNoContent) wantStatus(t, got, http.StatusNoContent)
@@ -1624,13 +1542,13 @@ func TestOldestAnsweredRequestIsDroppedPastTheBound(t *testing.T) {
t, t,
readAnswer(res), readAnswer(res),
http.StatusOK, http.StatusOK,
wafprotocol.DispositionAllow, waf.DispositionAllow,
) )
} }
// The table is full again with next and those, without oldest. // The table is full again with next and those, without oldest.
got := s.report( got := s.report(
wafprotocol.Report{RequestID: oldest, ResponseBytes: answerBytes}, waf.Report{RequestID: oldest, ResponseBytes: answerBytes},
) )
wantStatus(t, got, http.StatusNoContent) wantStatus(t, got, http.StatusNoContent)
@@ -1641,9 +1559,7 @@ func TestOldestAnsweredRequestIsDroppedPastTheBound(t *testing.T) {
) )
} }
got = s.report( got = s.report(waf.Report{RequestID: next, ResponseBytes: answerBytes})
wafprotocol.Report{RequestID: next, ResponseBytes: answerBytes},
)
wantStatus(t, got, http.StatusNoContent) wantStatus(t, got, http.StatusNoContent)
if counted := s.history().Requests; counted != 1 { if counted := s.history().Requests; counted != 1 {
@@ -1664,20 +1580,10 @@ func TestBanMadeThroughOneHostRefusesTheClientOnAnother(t *testing.T) {
) )
got := do(t, s.question(siteA, client, "/.env")) got := do(t, s.question(siteA, client, "/.env"))
wantDisposition( wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
got = do(t, s.question(siteB, client, "/")) got = do(t, s.question(siteB, client, "/"))
wantDisposition( wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
lines := s.out.requestLines(t, 2) lines := s.out.requestLines(t, 2)
if lines[0].Host != siteA || lines[1].Host != siteB { if lines[0].Host != siteA || lines[1].Host != siteB {
@@ -1715,12 +1621,7 @@ func TestRequestsOnEverySiteCountTowardOneSetOfLimits(t *testing.T) {
s.allowed(client) s.allowed(client)
got := do(t, s.question(siteB, client, "/")) got := do(t, s.question(siteB, client, "/"))
wantDisposition( wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
} }
func TestOwnEndpointsAnswerOnlyWithoutTheClientHeader(t *testing.T) { func TestOwnEndpointsAnswerOnlyWithoutTheClientHeader(t *testing.T) {
@@ -1735,13 +1636,13 @@ func TestOwnEndpointsAnswerOnlyWithoutTheClientHeader(t *testing.T) {
path, authorization string path, authorization string
want int want int
}{ }{
{wafprotocol.HealthPath, "", http.StatusOK}, {waf.HealthPath, "", http.StatusOK},
{waf.MetricsPath, bearer, http.StatusOK}, {waf.MetricsPath, bearer, http.StatusOK},
{waf.MetricsPath, "", http.StatusUnauthorized}, {waf.MetricsPath, "", http.StatusUnauthorized},
{waf.BansPath, adminBearer, http.StatusOK}, {waf.BansPath, adminBearer, http.StatusOK},
{waf.BansPath, "", http.StatusUnauthorized}, {waf.BansPath, "", http.StatusUnauthorized},
{waf.ClientsPath + client, adminBearer, http.StatusOK}, {waf.ClientsPath + client, adminBearer, http.StatusOK},
{wafprotocol.ReportPath, "", http.StatusNotFound}, {waf.ReportPath, "", http.StatusNotFound},
{"/_sww/other", "", http.StatusNotFound}, {"/_sww/other", "", http.StatusNotFound},
{"/", "", http.StatusNotFound}, {"/", "", http.StatusNotFound},
} { } {
@@ -1758,10 +1659,10 @@ func TestOwnEndpointsAnswerOnlyWithoutTheClientHeader(t *testing.T) {
t, t,
do(t, req), do(t, req),
http.StatusNotFound, http.StatusNotFound,
wafprotocol.DispositionDeny, waf.DispositionDeny,
) )
} else { } else {
wantDisposition(t, do(t, req), http.StatusOK, wafprotocol.DispositionAllow) wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow)
} }
} }
} }
@@ -1771,11 +1672,11 @@ func TestOwnEndpointsLinesDoNotSayWhichOfTheTwoTheyAre(t *testing.T) {
s := startWAF(t, "", nil) s := startWAF(t, "", nil)
wantStatus(t, get(t, s.addr, wafprotocol.HealthPath), http.StatusOK) wantStatus(t, get(t, s.addr, waf.HealthPath), http.StatusOK)
line := s.out.requestLine(t) line := s.out.requestLine(t)
if _, there := line.fields["waf_line"]; there || if _, there := line.fields["waf_line"]; there ||
line.Path != wafprotocol.HealthPath { line.Path != waf.HealthPath {
t.Errorf("the health check's log line is %+v", line.fields) t.Errorf("the health check's log line is %+v", line.fields)
} }
} }
@@ -1787,7 +1688,7 @@ func TestALargeBodyAnnouncedAndLongHeadersAreNoReasonToRefuse(t *testing.T) {
req := s.questionWith(http.MethodPost, siteA, client, "/", req := s.questionWith(http.MethodPost, siteA, client, "/",
strings.NewReader("a body")) strings.NewReader("a body"))
req.Header.Set(wafprotocol.ContentLengthHeader, "1073741824") req.Header.Set(waf.ContentLengthHeader, "1073741824")
req.Header.Set(forwardedFor, otherClient) req.Header.Set(forwardedFor, otherClient)
// 40,000 bytes of headers, over rproxy's 32 KiB, in lines within the // 40,000 bytes of headers, over rproxy's 32 KiB, in lines within the
@@ -1797,7 +1698,7 @@ func TestALargeBodyAnnouncedAndLongHeadersAreNoReasonToRefuse(t *testing.T) {
} }
got := do(t, req) got := do(t, req)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow) wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
line := s.out.requestLine(t) line := s.out.requestLine(t)
if line.ClientIP != client || line.ContentLength != 1<<30 { if line.ClientIP != client || line.ContentLength != 1<<30 {
@@ -1875,7 +1776,7 @@ carried header:Content-Length block ^6$
strings.NewReader("a body")) strings.NewReader("a body"))
if tc.announced != "" { if tc.announced != "" {
req.Header.Set(wafprotocol.ContentLengthHeader, tc.announced) req.Header.Set(waf.ContentLengthHeader, tc.announced)
} }
wantStatus(t, do(t, req), tc.want) wantStatus(t, do(t, req), tc.want)
@@ -1893,24 +1794,19 @@ func TestCoreRuleSetReadsTheBodyAQuestionCarries(t *testing.T) {
req := s.questionWith(http.MethodPost, siteA, client, "/", req := s.questionWith(http.MethodPost, siteA, client, "/",
strings.NewReader(body)) strings.NewReader(body))
req.Header.Set("Content-Type", contentType) req.Header.Set("Content-Type", contentType)
req.Header.Set(wafprotocol.ContentLengthHeader, "100000") req.Header.Set(waf.ContentLengthHeader, "100000")
return do(t, req) return do(t, req)
} }
got := ask(formData, "id=1'%20OR%20'1'='1") got := ask(formData, "id=1'%20OR%20'1'='1")
wantDisposition( wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
// The first 8 KiB of a longer JSON body cannot be parsed, which is no // The first 8 KiB of a longer JSON body cannot be parsed, which is no
// attack, so the Core Rule Set does not read it. // attack, so the Core Rule Set does not read it.
cut := `{"text":"` + strings.Repeat("x", 8<<10) cut := `{"text":"` + strings.Repeat("x", 8<<10)
got = ask("application/json", cut) got = ask("application/json", cut)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow) wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
} }
func TestCoreRuleSetAlertNamesTheSite(t *testing.T) { func TestCoreRuleSetAlertNamesTheSite(t *testing.T) {
@@ -1919,12 +1815,7 @@ func TestCoreRuleSetAlertNamesTheSite(t *testing.T) {
s := startWAF(t, "", map[string]string{coreRuleSetMode: block}) s := startWAF(t, "", map[string]string{coreRuleSetMode: block})
got := s.ask(client, sqlInjection) got := s.ask(client, sqlInjection)
wantDisposition( wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
waiting := s.queue.Snapshot().Waiting[alerts.DestinationWebhook] waiting := s.queue.Snapshot().Waiting[alerts.DestinationWebhook]
if len(waiting) != 1 || if len(waiting) != 1 ||
@@ -1945,7 +1836,7 @@ func TestReputationHitAlertNamesTheSite(t *testing.T) {
loadLists(t, s.server, map[string][]string{dropURL: {client}}) loadLists(t, s.server, map[string][]string{dropURL: {client}})
wantDisposition(t, s.ask(client, "/"), http.StatusForbidden, wantDisposition(t, s.ask(client, "/"), http.StatusForbidden,
wafprotocol.DispositionDeny) waf.DispositionDeny)
waiting := s.queue.Snapshot().Waiting[alerts.DestinationWebhook] waiting := s.queue.Snapshot().Waiting[alerts.DestinationWebhook]
if len(waiting) != 1 || waiting[0].Event != alerts.EventReputationHit || if len(waiting) != 1 || waiting[0].Event != alerts.EventReputationHit ||
-46
View File
@@ -1,46 +0,0 @@
package wafprotocol
import "net/http"
// Answer is what waf's answer to a question tells rproxy in its headers.
// The status, the other headers and the body of an answer of deny are what
// the visitor gets.
type Answer struct {
// Disposition is what rproxy does with the request: DispositionAllow,
// DispositionDeny or DispositionClose.
Disposition string
// ASN and Country are the client's AS number and country, each empty
// when the answer does not give it.
ASN, Country string
}
// SetAnswerHeaders sets the headers of waf's answer that tell rproxy answer,
// leaving out an ASN or Country that is empty.
func SetAnswerHeaders(header http.Header, answer Answer) {
header.Set(DispositionHeader, answer.Disposition)
if answer.ASN != "" {
header.Set(ASNHeader, answer.ASN)
}
if answer.Country != "" {
header.Set(CountryHeader, answer.Country)
}
}
// ReadAnswer returns what the headers of waf's answer tell rproxy, and
// whether they give a valid disposition.
func ReadAnswer(header http.Header) (Answer, bool) {
disposition := header.Get(DispositionHeader)
switch disposition {
case DispositionAllow, DispositionDeny, DispositionClose:
return Answer{
Disposition: disposition,
ASN: header.Get(ASNHeader),
Country: header.Get(CountryHeader),
}, true
}
return Answer{}, false
}
-189
View File
@@ -1,189 +0,0 @@
package wafprotocol
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"net/http"
"net/netip"
"net/url"
"strconv"
"unicode/utf8"
)
// maxRequestIDBytes is the longest id a question may give.
const maxRequestIDBytes = 128
// ErrBodyStoppedArriving is the error ReadQuestion wraps when the question's
// body stops arriving. Every other error it returns is why waf refuses the
// question.
var ErrBodyStoppedArriving = errors.New("the body stopped arriving")
// Why waf refuses a question. Each is also the body of the refusal.
var (
errClientNotAnAddress = errors.New(
ClientHeader + " is not an address without a zone",
)
errRequestIDNotValid = errors.New(RequestIDHeader + " is not a valid id")
errBodyTooLarge = errors.New("the body is over 64 KiB")
)
// QuestionParams are what NewQuestion needs.
type QuestionParams struct {
// Visitor is the visitor's request.
Visitor *http.Request
// PathAsSent is Visitor's path as the visitor sent it, which the question
// sends unchanged.
PathAsSent string
// Body is the part of Visitor's body the question carries, at most
// QuestionBodyMaxBytes.
Body []byte
// WAFBackend is waf's address. The question takes its scheme and host.
WAFBackend *url.URL
// Client is the visitor's address, and Peer that of rproxy's own TCP
// peer.
Client, Peer netip.Addr
// RequestID is the request's id.
RequestID string
// Scheme is how the visitor reached rproxy, or the trusted proxy in front
// of it.
Scheme string
}
// Question is a question as waf reads it.
type Question struct {
// Client is the visitor's address.
Client netip.Addr
// Peer is the address of rproxy's own TCP peer. It is not valid when the
// question gives none that can be read.
Peer netip.Addr
// RequestID is the request's id.
RequestID string
// Visitor is the visitor's request, as asTheVisitorSentIt gives it.
Visitor *http.Request
}
// NewQuestion returns the question about a visitor's request, as "The
// question" in SPEC.md gives it: the request as the visitor sent it, with the
// part of the body it carries and rproxy's headers.
func NewQuestion(
ctx context.Context,
params QuestionParams,
) (*http.Request, error) {
in := params.Visitor
question, err := http.NewRequestWithContext(
ctx,
in.Method,
params.WAFBackend.String(),
bytes.NewReader(params.Body),
)
if err != nil {
return nil, fmt.Errorf("make the question: %w", err)
}
backend := question.URL
target := *in.URL
target.Scheme = backend.Scheme
target.Host = backend.Host
target.Opaque = params.PathAsSent
question.URL = &target
question.Host = in.Host
// The length NewRequestWithContext set is that of the body carried. This
// line makes a body whose length the visitor did not announce go chunked,
// as "The question" in SPEC.md requires.
question.TransferEncoding = in.TransferEncoding
question.Header = in.Header.Clone()
question.Header.Set(ClientHeader, params.Client.String())
question.Header.Set(PeerHeader, params.Peer.String())
question.Header.Del(ContentLengthHeader)
announced := in.Header.Get("Content-Length")
if announced != "" {
question.Header.Set(ContentLengthHeader, announced)
}
question.Header.Set(RequestIDHeader, params.RequestID)
question.Header.Set("X-Forwarded-Proto", params.Scheme)
// Go's client adds a User-Agent of its own to a request without the key.
_, sent := question.Header["User-Agent"]
if !sent {
question.Header["User-Agent"] = nil
}
return question, nil
}
// ReadQuestion reads the question r, with the body it carries. An error
// wrapping ErrBodyStoppedArriving means the body stopped arriving. Any other
// error is why waf refuses the question, and the question's body is then not
// read. The body is read whole, since Go's server does not send an answer
// until it has read the body.
func ReadQuestion(w http.ResponseWriter, r *http.Request) (Question, error) {
client, err := netip.ParseAddr(r.Header.Get(ClientHeader))
if err != nil || client.Zone() != "" {
return Question{}, errClientNotAnAddress
}
id := r.Header.Get(RequestIDHeader)
if id == "" || len(id) > maxRequestIDBytes || !utf8.ValidString(id) {
return Question{}, errRequestIDNotValid
}
body, err := io.ReadAll(
http.MaxBytesReader(w, r.Body, QuestionBodyMaxBytes),
)
if _, tooLarge := errors.AsType[*http.MaxBytesError](err); tooLarge {
return Question{}, errBodyTooLarge
}
if err != nil {
return Question{}, fmt.Errorf("%w: %w", ErrBodyStoppedArriving, err)
}
// One that cannot be read leaves the log line's peer_ip invalid.
peer, _ := netip.ParseAddr(r.Header.Get(PeerHeader))
return Question{
Client: client.Unmap(),
Peer: peer.Unmap(),
RequestID: id,
Visitor: asTheVisitorSentIt(r, body),
}, nil
}
// asTheVisitorSentIt returns a copy of r, a question, as the checks and the
// log line are to see it: without the headers rproxy adds for waf,
// with body, which has been read of r, and with the length the visitor
// announced, as -1 when it announced none. The question's own length covers
// only the part of the body it carries.
func asTheVisitorSentIt(r *http.Request, body []byte) *http.Request {
sent := r.Clone(r.Context())
announced := sent.Header.Get(ContentLengthHeader)
sent.Header.Del(ClientHeader)
sent.Header.Del(PeerHeader)
sent.Header.Del(ContentLengthHeader)
sent.Header.Del("Content-Length")
sent.ContentLength = -1
length, err := strconv.ParseInt(announced, 10, 64)
if err == nil && length >= 0 {
sent.ContentLength = length
sent.Header.Set("Content-Length", announced)
}
sent.Body = http.NoBody
if len(body) > 0 {
sent.Body = io.NopCloser(bytes.NewReader(body))
}
return sent
}
-123
View File
@@ -1,123 +0,0 @@
package wafprotocol
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
)
const (
// reportBodyMaxBytes is the most of a report's body that is read. It is
// far above any report's.
reportBodyMaxBytes = 1 << 20
// maxReportedBytes is the most bytes a report may give for each way, 1
// PiB, so that adding them up cannot overflow.
maxReportedBytes = 1 << 50
// maxReportedMilliseconds is the longest duration a report may give,
// about 31 years, so that it fits a time.Duration.
maxReportedMilliseconds = 1e12
)
var (
errReportFieldMissing = errors.New("a field is missing")
errReportValue = errors.New("a value is out of range")
)
// Report is the JSON object rproxy posts to ReportPath once the visitor's
// answer has ended, its fields named as the request log's of the same
// meaning. RequestID is the question's id. Status is what the visitor got, 0
// if nothing. The durations are in milliseconds.
//
//nolint:tagliatelle // docs/request-log.md names the fields in snake_case
type Report struct {
RequestID string `json:"request_id"`
Status int `json:"status"`
RequestBytes int64 `json:"request_bytes"`
ResponseBytes int64 `json:"response_bytes"`
DurationTotalMS float64 `json:"duration_total_ms"`
DurationUpstreamTotalMS float64 `json:"duration_upstream_total_ms"`
}
// NewReportRequest returns the request that posts report to reportURL, which
// is waf's ReportPath.
func NewReportRequest(
ctx context.Context,
reportURL string,
report *Report,
) (*http.Request, error) {
body, err := json.Marshal(report)
if err != nil {
return nil, fmt.Errorf("encode the report: %w", err)
}
req, err := http.NewRequestWithContext(
ctx, http.MethodPost, reportURL, bytes.NewReader(body))
if err != nil {
return nil, fmt.Errorf("make the request: %w", err)
}
req.Header.Set("Content-Type", "application/json")
return req, nil
}
// ReadReport reads a report from r's body, which must be one JSON object with
// all of Report's fields, in the ranges SPEC.md gives.
func ReadReport(w http.ResponseWriter, r *http.Request) (Report, error) {
var report Report
body, err := io.ReadAll(
http.MaxBytesReader(w, r.Body, reportBodyMaxBytes),
)
if err != nil {
return report, fmt.Errorf("read the body: %w", err)
}
var fields map[string]json.RawMessage
err = json.Unmarshal(body, &fields)
if err != nil {
return report, fmt.Errorf("decode the body: %w", err)
}
for _, name := range []string{
"request_id", "status", "request_bytes", "response_bytes",
"duration_total_ms", "duration_upstream_total_ms",
} {
value, there := fields[name]
if !there || string(value) == "null" {
return report, fmt.Errorf("%w: %s", errReportFieldMissing, name)
}
}
err = json.Unmarshal(body, &report)
if err != nil {
return report, fmt.Errorf("decode the body: %w", err)
}
return report, report.checkRanges()
}
// checkRanges returns an error for a value of the report that cannot be true.
func (report *Report) checkRanges() error {
switch {
case report.Status != 0 &&
(report.Status < http.StatusContinue || report.Status > ReportStatusMax):
return fmt.Errorf("%w: status", errReportValue)
case report.RequestBytes < 0 || report.RequestBytes > maxReportedBytes:
return fmt.Errorf("%w: request_bytes", errReportValue)
case report.ResponseBytes < 0 || report.ResponseBytes > maxReportedBytes:
return fmt.Errorf("%w: response_bytes", errReportValue)
case report.DurationTotalMS < 0 || report.DurationTotalMS > maxReportedMilliseconds:
return fmt.Errorf("%w: duration_total_ms", errReportValue)
case report.DurationUpstreamTotalMS < 0 ||
report.DurationUpstreamTotalMS > maxReportedMilliseconds:
return fmt.Errorf("%w: duration_upstream_total_ms", errReportValue)
}
return nil
}
-61
View File
@@ -1,61 +0,0 @@
// Package wafprotocol holds the three messages rproxy and waf send each
// other, as "`waf`" in SPEC.md gives them. rproxy asks waf a question about
// each visitor's request, and waf answers it. Once the visitor's answer has
// ended, rproxy sends waf a report. This package writes and reads each
// message, and declares every name and limit both sides must agree on.
package wafprotocol
// The headers of the question and of the answer. Go writes every header name
// in this form, as it sends it and as it receives it, so X-Client-ASN arrives
// as X-Client-Asn; header names are not case-sensitive.
const (
// ClientHeader marks a request as rproxy's question, and gives the
// visitor's address.
ClientHeader = "X-Sww-Client"
// PeerHeader gives the address of rproxy's own TCP peer.
PeerHeader = "X-Sww-Peer"
// ContentLengthHeader gives the body length the visitor announced.
ContentLengthHeader = "X-Sww-Content-Length"
// RequestIDHeader gives the request's id, which rproxy makes. rproxy
// also sets it on the request it passes to the app.
RequestIDHeader = "X-Request-ID"
// DispositionHeader gives waf's answer.
DispositionHeader = "X-Sww-Disposition"
// ASNHeader and CountryHeader give the client's AS number and country in
// an answer of allow. rproxy passes them to the app.
ASNHeader = "X-Client-Asn"
CountryHeader = "X-Client-Country"
)
// The values of DispositionHeader.
const (
// DispositionAllow tells rproxy to pass the request to the app.
DispositionAllow = "allow"
// DispositionDeny tells rproxy to give the visitor the answer's status,
// headers and body.
DispositionDeny = "deny"
// DispositionClose tells rproxy to close the visitor's connection
// without an answer, which SWW_WAF_BAN_RESPONSE asks for when it is close.
// waf closing the question's connection instead would read as
// waf being down, and rproxy would then let the request through.
DispositionClose = "close"
)
// OwnEndpointPrefix starts the path of every request for sww itself: waf's
// own endpoints and rproxy's health endpoint. The app never sees a path under
// it.
const OwnEndpointPrefix = "/_sww/"
// HealthPath is the health endpoint of waf and of rproxy, which the
// container's health check asks.
const HealthPath = "/_sww/healthz"
// ReportPath is where rproxy sends the report of each request waf
// answered.
const ReportPath = "/_sww/report"
// QuestionBodyMaxBytes is the most of a visitor's body a question carries.
const QuestionBodyMaxBytes = 64 << 10
// ReportStatusMax is the highest status a report may give.
const ReportStatusMax = 599
-204
View File
@@ -1,204 +0,0 @@
package wafprotocol_test
import (
"io"
"net/http"
"net/http/httptest"
"net/netip"
"net/url"
"strings"
"testing"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// requestID is the id the tests' questions and reports give.
const requestID = "an-id"
// questionRead is what the server startQuestionReader starts read of a
// question: the question, and the body it carries.
type questionRead struct {
question wafprotocol.Question
body string
}
// startQuestionReader starts a server that reads each question it gets with
// ReadQuestion. It returns the server's URL and the channel what it reads goes
// to.
func startQuestionReader(t *testing.T) (*url.URL, <-chan questionRead) {
t.Helper()
read := make(chan questionRead, 1)
server := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
question, err := wafprotocol.ReadQuestion(w, r)
if err != nil {
t.Errorf("read the question: %v", err)
}
var body []byte
if question.Visitor != nil {
body, _ = io.ReadAll(question.Visitor.Body)
}
read <- questionRead{question, string(body)}
},
))
t.Cleanup(server.Close)
serverURL, err := url.Parse(server.URL)
if err != nil {
t.Fatalf("parse the server's URL: %v", err)
}
return serverURL, read
}
// TestQuestionReadsAsWritten sends a question NewQuestion writes over HTTP,
// and checks that ReadQuestion gives back the visitor's request and what
// rproxy added.
func TestQuestionReadsAsWritten(t *testing.T) {
t.Parallel()
const (
target = "/a%2Fb?q=1"
body = "name=value"
held = "name"
)
client := netip.MustParseAddr("2001:db8::1")
peer := netip.MustParseAddr("192.0.2.1")
backend, read := startQuestionReader(t)
visitor := httptest.NewRequestWithContext(
t.Context(),
http.MethodPost,
"http://site.example.com"+target,
strings.NewReader(body),
)
visitor.Header.Set("Content-Length", "10")
visitor.Header.Set("User-Agent", "a visitor")
visitor.Header.Set(wafprotocol.PeerHeader, "the visitor's own")
question, err := wafprotocol.NewQuestion(
t.Context(),
wafprotocol.QuestionParams{
Visitor: visitor,
PathAsSent: "/a%2Fb",
Body: []byte(held),
WAFBackend: backend,
Client: client,
Peer: peer,
RequestID: requestID,
Scheme: "https",
},
)
if err != nil {
t.Fatalf("write the question: %v", err)
}
res, err := http.DefaultTransport.RoundTrip(question)
if err != nil {
t.Fatalf("send the question: %v", err)
}
_ = res.Body.Close()
got := <-read
if got.question.Visitor == nil {
t.FailNow()
}
if got.question.Client != client || got.question.Peer != peer ||
got.question.RequestID != requestID {
t.Errorf("read %+v", got.question)
}
sent := got.question.Visitor
if sent.Host != "site.example.com" || sent.RequestURI != target ||
sent.ContentLength != int64(len(body)) || got.body != held {
t.Errorf(
"read %s at %s, of length %d, carrying %q",
sent.RequestURI, sent.Host, sent.ContentLength, got.body,
)
}
for name, want := range map[string]string{
"Content-Length": "10",
"User-Agent": "a visitor",
"X-Forwarded-Proto": "https",
wafprotocol.RequestIDHeader: requestID,
wafprotocol.ClientHeader: "",
wafprotocol.PeerHeader: "",
wafprotocol.ContentLengthHeader: "",
} {
if sent.Header.Get(name) != want {
t.Errorf("%s is %q, want %q", name, sent.Header.Get(name), want)
}
}
}
// TestAnswerReadsAsWritten checks that ReadAnswer gives back what
// SetAnswerHeaders sets, and that headers without a valid disposition give
// nothing.
func TestAnswerReadsAsWritten(t *testing.T) {
t.Parallel()
for _, want := range []wafprotocol.Answer{
{Disposition: wafprotocol.DispositionAllow, ASN: "64496", Country: "DE"},
{Disposition: wafprotocol.DispositionAllow, Country: "DE"},
{Disposition: wafprotocol.DispositionDeny},
{Disposition: wafprotocol.DispositionClose},
} {
header := http.Header{}
wafprotocol.SetAnswerHeaders(header, want)
got, valid := wafprotocol.ReadAnswer(header)
if !valid || got != want {
t.Errorf("read %+v (valid: %t), want %+v", got, valid, want)
}
}
header := http.Header{}
header.Set(wafprotocol.DispositionHeader, "maybe")
header.Set(wafprotocol.ASNHeader, "64496")
got, valid := wafprotocol.ReadAnswer(header)
if valid || got != (wafprotocol.Answer{}) {
t.Errorf("read %+v (valid: %t), want nothing", got, valid)
}
}
// TestReportReadsAsWritten checks that ReadReport gives back the report
// NewReportRequest posts.
func TestReportReadsAsWritten(t *testing.T) {
t.Parallel()
want := wafprotocol.Report{
RequestID: requestID,
Status: http.StatusOK,
RequestBytes: 10,
ResponseBytes: 20,
DurationTotalMS: 1.5,
DurationUpstreamTotalMS: 0.5,
}
req, err := wafprotocol.NewReportRequest(
t.Context(),
"http://waf.example.com"+wafprotocol.ReportPath,
&want,
)
if err != nil {
t.Fatalf("write the report: %v", err)
}
if req.Method != http.MethodPost ||
req.URL.Path != wafprotocol.ReportPath {
t.Errorf("the report is %s %s", req.Method, req.URL.Path)
}
got, err := wafprotocol.ReadReport(httptest.NewRecorder(), req)
if err != nil || got != want {
t.Errorf("read %+v (%v), want %+v", got, err, want)
}
}