Compare commits

..
Author SHA1 Message Date
sneak 9d3f212b0d waf: a heuristic says itself that it refuses whatever the lookup answers (closes #446)
check / check (push) Waiting to run
refusedWhateverTheAnswer asks each heuristic of afterLookup that
implements the new refusesWithoutLookup interface, instead of naming
blocklists and rate_limits. The comment on adding a heuristic names the
places outside its file that may change with it.

Model: opus-5-5
2026-11-19 21:59:05 +00:00
clawbot 51989235d6 rproxy: the body wrappers hold only what they call (closes #431)
check / check (push) Waiting to run
requestBody and responseBody are read on goroutines other than the
handler's. They held the whole request; now they hold only the
functions they call and, for responseBody, the request's context.

Model: opus-5-5
2026-11-19 22:43:57 +01:00
clawbot e15edef583 Move what rproxy and waf send each other into wafprotocol (closes #425)
check / check (push) Waiting to run
internal/wafprotocol writes and reads the question, the answer and the
report, and declares once each header, path and limit both daemons use.
rproxy no longer imports internal/waf. Nothing on the wire changes.

Model: opus-5-5
2026-11-19 22:27:16 +01:00
clawbot 06faf0877c dirwatch: one loop that watches a directory, for rules and lookup (closes #429)
check / check (push) Waiting to run
rules and lookup each had the same loop: watch the directory, start a
timer at once, restart it at each change that counts, read again when it
fires. It now lives once in internal/dirwatch, with its quiet time and
its tests. Each package keeps its own log messages, its readAgain, and
its test for which changes count. internal/state keeps its own copy.

Model: opus-5-5
2026-11-19 22:13:55 +01:00
clawbot b2b1716907 sww: the three commands as cobra commands (closes #440)
check / check (push) Waiting to run
Main builds a cobra root command with rproxy, waf and healthcheck under
it, in place of the switch on os.Args and the hand-kept usage text.
Exit statuses stay: 2 with no command or an unknown one, otherwise what
the command returns. cobra's completion command is off. `sww help` and
`sww --help` now print the commands and exit 0.

Model: opus-5-5
2026-11-19 21:59:25 +01:00
19 changed files with 583 additions and 515 deletions

No files matched your search

+2
View File
@@ -908,6 +908,8 @@ stops the running process (see [`docs/state-files.md`](docs/state-files.md)).
`internal/coreruleset`, `internal/geojs`, `internal/lookup`, `internal/coreruleset`, `internal/geojs`, `internal/lookup`,
`internal/reputation`, `internal/routewatch`, `internal/anomaly` and `internal/reputation`, `internal/routewatch`, `internal/anomaly` and
`internal/alerts`: one check or source each. `internal/alerts`: one check or source each.
- `internal/dirwatch`: watches a directory, so that `internal/rules` and
`internal/lookup` read their files again when they change.
- `internal/state`: the state and cache files. - `internal/state`: the state and cache files.
- `internal/requestlog`: the request log. - `internal/requestlog`: the request log.
- `internal/processlog`: the process's own messages. - `internal/processlog`: the process's own messages.
+6 -1
View File
@@ -69,7 +69,8 @@ never as a time.
that command's settings, and runs `internal/rproxy` until `SIGTERM` or that command's settings, and runs `internal/rproxy` until `SIGTERM` or
`SIGINT` (see "`rproxy`"). `sww healthcheck` is the image's health check. `SIGINT` (see "`rproxy`"). `sww healthcheck` is the image's health check.
With no command, or one it does not know, the binary prints the commands With no command, or one it does not know, the binary prints the commands
and exits with status 2. `internal/sww` is the only package that imports and exits with status 2. `sww help` and `sww --help` print them and exit
with status 0. `internal/sww` is the only package that imports
`go.uber.org/fx`. fx builds the parts of `waf` and `rproxy`, and starts `go.uber.org/fx`. fx builds the parts of `waf` and `rproxy`, and starts
and stops their background work. and stops their background work.
- `internal/config`: reads the settings, the one place they are read. - `internal/config`: reads the settings, the one place they are read.
@@ -96,6 +97,9 @@ never as a time.
imports no other package of this repository: `waf` gives it what it needs imports no other package of this repository: `waf` gives it what it needs
to count and report failures. to count and report failures.
- `internal/lookup`: looks clients up in the lookup database. - `internal/lookup`: looks clients up in the lookup database.
- `internal/dirwatch`: watches a directory, and calls a function once it has
gone a short time without a change. `internal/rules` and `internal/lookup`
read their files again this way.
- `internal/reputation`: fetches the lists, asks the DNS blocklists and - `internal/reputation`: fetches the lists, asks the DNS blocklists and
AbuseIPDB in the background, and keeps their answers. It also fetches AbuseIPDB in the background, and keeps their answers. It also fetches
AbuseIPDB's blacklist, which `internal/waf` bans. AbuseIPDB's blacklist, which `internal/waf` bans.
@@ -129,6 +133,7 @@ never as a time.
- `github.com/fsnotify/fsnotify` to notice edited files; - `github.com/fsnotify/fsnotify` to notice edited files;
- `github.com/prometheus/client_golang` for the metrics; - `github.com/prometheus/client_golang` for the metrics;
- `github.com/spf13/viper` to read the settings from the environment; - `github.com/spf13/viper` to read the settings from the environment;
- `github.com/spf13/cobra` for the commands and their help text;
- `github.com/maxmind/mmdbwriter`, in the tests, to write lookup databases; - `github.com/maxmind/mmdbwriter`, in the tests, to write lookup databases;
- remote log sending written in this project, since the standard library's - remote log sending written in this project, since the standard library's
`log/syslog` is frozen and writes only the older syslog format. `log/syslog` is frozen and writes only the older syslog format.
+2
View File
@@ -11,6 +11,7 @@ require (
github.com/maxmind/mmdbwriter v1.2.0 github.com/maxmind/mmdbwriter v1.2.0
github.com/oschwald/maxminddb-golang/v2 v2.7.0 github.com/oschwald/maxminddb-golang/v2 v2.7.0
github.com/prometheus/client_golang v1.24.1 github.com/prometheus/client_golang v1.24.1
github.com/spf13/cobra v1.10.2
github.com/spf13/viper v1.21.0 github.com/spf13/viper v1.21.0
go.uber.org/fx v1.24.0 go.uber.org/fx v1.24.0
golang.org/x/net v0.60.0 golang.org/x/net v0.60.0
@@ -26,6 +27,7 @@ require (
github.com/goccy/go-yaml v1.19.2 // indirect github.com/goccy/go-yaml v1.19.2 // indirect
github.com/gotnospirit/makeplural v0.0.0-20180622080156-a5f48d94d976 // indirect github.com/gotnospirit/makeplural v0.0.0-20180622080156-a5f48d94d976 // indirect
github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 // indirect github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/kaptinlin/go-i18n v0.1.4 // indirect github.com/kaptinlin/go-i18n v0.1.4 // indirect
github.com/kaptinlin/jsonschema v0.4.6 // indirect github.com/kaptinlin/jsonschema v0.4.6 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect github.com/kylelemons/godebug v1.1.0 // indirect
+9
View File
@@ -14,6 +14,7 @@ github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68 h1:KLEjvBBSw
github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68/go.mod h1:9XocblQEwK6EYylRth7PrPIifF2l0iSoYo4FAXhXU/8= github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68/go.mod h1:9XocblQEwK6EYylRth7PrPIifF2l0iSoYo4FAXhXU/8=
github.com/corazawaf/libinjection-go v0.3.4 h1:jX2Tit2xZxYJJyVryNXucYBZtQqS7473x9Nc6lgRRbk= github.com/corazawaf/libinjection-go v0.3.4 h1:jX2Tit2xZxYJJyVryNXucYBZtQqS7473x9Nc6lgRRbk=
github.com/corazawaf/libinjection-go v0.3.4/go.mod h1:Ik/+w3UmTWH9yn366RgS9D95K3y7Atb5m/H/gXzzPCk= github.com/corazawaf/libinjection-go v0.3.4/go.mod h1:Ik/+w3UmTWH9yn366RgS9D95K3y7Atb5m/H/gXzzPCk=
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/foxcpp/go-mockdns v1.2.0 h1:omK3OrHRD1IWJz1FuFBCFquhXslXoF17OvBS6JPzZF0= github.com/foxcpp/go-mockdns v1.2.0 h1:omK3OrHRD1IWJz1FuFBCFquhXslXoF17OvBS6JPzZF0=
github.com/foxcpp/go-mockdns v1.2.0/go.mod h1:IhLeSFGed3mJIAXPH2aiRQB+kqz7oqu8ld2qVbOu7Wk= github.com/foxcpp/go-mockdns v1.2.0/go.mod h1:IhLeSFGed3mJIAXPH2aiRQB+kqz7oqu8ld2qVbOu7Wk=
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8= github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
@@ -34,6 +35,8 @@ github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 h1:c7gcN
github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092/go.mod h1:ZZAN4fkkful3l1lpJwF8JbW41ZiG9TwJ2ZlqzQovBNU= github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092/go.mod h1:ZZAN4fkkful3l1lpJwF8JbW41ZiG9TwJ2ZlqzQovBNU=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k= github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM= github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/jcchavezs/mergefs v0.1.1 h1:D45R17m6dHnSVZefnhynoeZvcK2Uw0oTrRfoUOQ0S5Y= github.com/jcchavezs/mergefs v0.1.1 h1:D45R17m6dHnSVZefnhynoeZvcK2Uw0oTrRfoUOQ0S5Y=
github.com/jcchavezs/mergefs v0.1.1/go.mod h1:eRLTrsA+vFwQZ48hj8p8gki/5v9C2bFtHH5Mnn4bcGk= github.com/jcchavezs/mergefs v0.1.1/go.mod h1:eRLTrsA+vFwQZ48hj8p8gki/5v9C2bFtHH5Mnn4bcGk=
github.com/kaptinlin/go-i18n v0.1.4 h1:wCiwAn1LOcvymvWIVAM4m5dUAMiHunTdEubLDk4hTGs= github.com/kaptinlin/go-i18n v0.1.4 h1:wCiwAn1LOcvymvWIVAM4m5dUAMiHunTdEubLDk4hTGs=
@@ -72,6 +75,7 @@ github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+
github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY= github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY=
github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8= github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs= github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/sagikazarmark/locafero v0.11.0 h1:1iurJgmM9G3PA/I+wWYIOw/5SyBtxapeHDcg+AAIFXc= github.com/sagikazarmark/locafero v0.11.0 h1:1iurJgmM9G3PA/I+wWYIOw/5SyBtxapeHDcg+AAIFXc=
github.com/sagikazarmark/locafero v0.11.0/go.mod h1:nVIGvgyzw595SUSUE6tvCp3YYTeHs15MvlmU87WwIik= github.com/sagikazarmark/locafero v0.11.0/go.mod h1:nVIGvgyzw595SUSUE6tvCp3YYTeHs15MvlmU87WwIik=
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 h1:+jumHNA0Wrelhe64i8F6HNlS8pkoyMv5sreGx2Ry5Rw= github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 h1:+jumHNA0Wrelhe64i8F6HNlS8pkoyMv5sreGx2Ry5Rw=
@@ -80,6 +84,9 @@ github.com/spf13/afero v1.15.0 h1:b/YBCLWAJdFWJTN9cLhiXXcD7mzKn9Dm86dNnfyQw1I=
github.com/spf13/afero v1.15.0/go.mod h1:NC2ByUVxtQs4b3sIUphxK0NioZnmxgyCrfzeuq8lxMg= github.com/spf13/afero v1.15.0/go.mod h1:NC2ByUVxtQs4b3sIUphxK0NioZnmxgyCrfzeuq8lxMg=
github.com/spf13/cast v1.10.0 h1:h2x0u2shc1QuLHfxi+cTJvs30+ZAHOGRic8uyGTDWxY= github.com/spf13/cast v1.10.0 h1:h2x0u2shc1QuLHfxi+cTJvs30+ZAHOGRic8uyGTDWxY=
github.com/spf13/cast v1.10.0/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqeHo= github.com/spf13/cast v1.10.0/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqeHo=
github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk= github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.21.0 h1:x5S+0EU27Lbphp4UKm1C+1oQO+rKx36vfCoaVebLFSU= github.com/spf13/viper v1.21.0 h1:x5S+0EU27Lbphp4UKm1C+1oQO+rKx36vfCoaVebLFSU=
@@ -109,6 +116,7 @@ go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so= go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ= go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ= go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw= go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg= go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba h1:0b9z3AuHCjxk0x/opv64kcgZLBseWJUpBw5I82+2U4M= go4.org/netipx v0.0.0-20231129151722-fdeea329fbba h1:0b9z3AuHCjxk0x/opv64kcgZLBseWJUpBw5I82+2U4M=
@@ -127,6 +135,7 @@ golang.org/x/tools v0.50.0 h1:c2ifzfcuY7L90lZ2aKd8S4K2NpASF08SZx9ZuJkHmSU=
golang.org/x/tools v0.50.0/go.mod h1:7ulVMw3831Mwi5EZD6RomGyffr4VFjuNYXf2BbCEAV0= golang.org/x/tools v0.50.0/go.mod h1:7ulVMw3831Mwi5EZD6RomGyffr4VFjuNYXf2BbCEAV0=
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE= google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
rsc.io/binaryregexp v0.2.0 h1:HfqmD5MEmC0zvwBuF187nq9mdnXjXsSivRiXN7SmRkE= rsc.io/binaryregexp v0.2.0 h1:HfqmD5MEmC0zvwBuF187nq9mdnXjXsSivRiXN7SmRkE=
+90
View File
@@ -0,0 +1,90 @@
// Package dirwatch watches a directory, and calls a function once the
// directory has gone a short time without a change after one. The rule
// files and the lookup database are read again this way.
package dirwatch
import (
"context"
"time"
"github.com/fsnotify/fsnotify"
)
// quietTime is how long the directory must go without a change before the
// function is called, so that a file still being written, such as one
// saved in place, appended to or copied in with scp, is read only once
// whole.
const quietTime = 2 * time.Second
// Watcher watches one directory. The directory is watched, rather than a
// file in it, so that a file replaced by a rename is seen.
type Watcher struct {
watcher *fsnotify.Watcher
}
// New starts watching dir, or returns why it cannot.
func New(dir string) (*Watcher, error) {
watcher, err := fsnotify.NewWatcher()
if err != nil {
return nil, err
}
err = watcher.Add(dir)
if err != nil {
_ = watcher.Close()
return nil, err
}
return &Watcher{watcher: watcher}, nil
}
// Close stops watching the directory.
func (w *Watcher) Close() error {
return w.watcher.Close()
}
// CallWhenQuiet calls changed each time the directory has gone quietTime
// without a change, after a change to a path for which counts is true, until
// ctx is done. It also calls failed with each error the watch reports. The
// wait starts at once, as if for a change, so that a change made before the
// watch began is taken in too.
func (w *Watcher) CallWhenQuiet(
ctx context.Context,
counts func(path string) bool,
changed func(),
failed func(error),
) {
callWhenQuiet(
ctx, w.watcher.Events, w.watcher.Errors, counts, changed, failed,
)
}
// callWhenQuiet is CallWhenQuiet, for the changes from events and the
// errors from errs.
func callWhenQuiet(
ctx context.Context,
events <-chan fsnotify.Event,
errs <-chan error,
counts func(path string) bool,
changed func(),
failed func(error),
) {
quiet := time.NewTimer(quietTime)
defer quiet.Stop()
for {
select {
case <-ctx.Done():
return
case event := <-events:
if counts(event.Name) {
quiet.Reset(quietTime)
}
case <-quiet.C:
changed()
case err := <-errs:
failed(err)
}
}
}
+167
View File
@@ -0,0 +1,167 @@
package dirwatch
import (
"context"
"errors"
"path/filepath"
"sync/atomic"
"testing"
"testing/synctest"
"time"
"github.com/fsnotify/fsnotify"
)
// errWatch is an error the watch reports.
var errWatch = errors.New("too many changes")
func TestDirectoryThatCannotBeWatchedIsAnError(t *testing.T) {
t.Parallel()
_, err := New(filepath.Join(t.TempDir(), "missing"))
if err == nil {
t.Error("a missing directory is watched, want an error")
}
}
// The tests below run callWhenQuiet in a synctest bubble, where time is a
// clock of the test's own: time.Sleep moves it on at once, and
// synctest.Wait returns once callWhenQuiet waits again, so that every call
// due by then is made. The test sends the changes itself, as the watch of
// a directory cannot run in a bubble.
func TestSecondChangeStartsTheWaitAgain(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
loop := start(t, everyPath)
loop.changes <- fsnotify.Event{Name: "50-app.rules", Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
loop.wantCalls(t, 0)
// The second part starts the wait again.
loop.changes <- fsnotify.Event{Name: "50-app.rules", Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
loop.wantCalls(t, 0)
time.Sleep(time.Nanosecond)
synctest.Wait()
loop.wantCalls(t, 1)
})
}
func TestCalledOnceQuietAfterTheStart(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
// An edit saved before the watch began sends no change, and is
// taken in by this call.
loop := start(t, everyPath)
time.Sleep(quietTime)
synctest.Wait()
loop.wantCalls(t, 1)
// Long after, it has been called only once.
time.Sleep(time.Hour)
synctest.Wait()
loop.wantCalls(t, 1)
})
}
func TestChangeThatDoesNotCountIsNoChange(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
loop := start(t, func(path string) bool {
return path == "ipinfo_lite.mmdb"
})
// The wait that starts with the watch ends with a call.
time.Sleep(quietTime)
synctest.Wait()
loop.changes <- fsnotify.Event{Name: "other.mmdb", Op: fsnotify.Create}
time.Sleep(quietTime)
synctest.Wait()
loop.wantCalls(t, 1)
loop.changes <- fsnotify.Event{Name: "ipinfo_lite.mmdb", Op: fsnotify.Write}
time.Sleep(quietTime)
synctest.Wait()
loop.wantCalls(t, 2)
})
}
func TestErrorsOfTheWatchHandedOn(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
loop := start(t, everyPath)
loop.errs <- errWatch
synctest.Wait()
if !errors.Is(loop.failure, errWatch) {
t.Errorf("failed with %v, want %v", loop.failure, errWatch)
}
})
}
// loop is a callWhenQuiet that a test runs.
type loop struct {
// changes and errs send it changes and errors.
changes chan fsnotify.Event
errs chan error
// calls counts its calls of changed, and failure is the last error it
// handed to failed.
calls atomic.Int32
failure error
}
// start runs callWhenQuiet, with counts, until the test ends.
func start(t *testing.T, counts func(path string) bool) *loop {
t.Helper()
l := &loop{changes: make(chan fsnotify.Event), errs: make(chan error)}
ctx, stop := context.WithCancel(t.Context())
stopped := make(chan struct{})
go func() {
callWhenQuiet(ctx, l.changes, l.errs, counts,
func() { l.calls.Add(1) },
func(err error) { l.failure = err },
)
close(stopped)
}()
t.Cleanup(func() {
stop()
<-stopped
})
return l
}
// everyPath counts a change to any path.
func everyPath(string) bool {
return true
}
// wantCalls checks how many times changed has been called.
func (l *loop) wantCalls(t *testing.T, want int32) {
t.Helper()
got := l.calls.Load()
if got != want {
t.Errorf("changed called %d times, want %d", got, want)
}
}
+18 -49
View File
@@ -13,18 +13,13 @@ import (
"sync" "sync"
"time" "time"
"github.com/fsnotify/fsnotify"
"github.com/oschwald/maxminddb-golang/v2" "github.com/oschwald/maxminddb-golang/v2"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/dirwatch"
"sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/geojs"
) )
// quietTime is how long the lookup database must go without a change
// before it is read again, so that a file still being copied in is read
// only once whole.
const quietTime = 2 * time.Second
// FileParams are what OpenFile needs. // FileParams are what OpenFile needs.
type FileParams struct { type FileParams struct {
// Path is the lookup database, the IPinfo Lite file in its .mmdb form // Path is the lookup database, the IPinfo Lite file in its .mmdb form
@@ -122,20 +117,13 @@ func (f *File) ReadFailures() int {
} }
// Watch watches the directory of the lookup database until ctx is done, // Watch watches the directory of the lookup database until ctx is done,
// and reads the file again once it has gone without a change for // and reads the file again once it has gone without a change for a short
// quietTime, after it is replaced, written or removed, and after Watch // time, after it is replaced, written or removed, and after Watch starts
// starts watching. If the directory cannot be watched, that is logged, and // watching, as dirwatch describes. A change to another file in its
// the database read at start stays in use. // directory does not count. If the directory cannot be watched, that is
// logged, and the database read at start stays in use.
func (f *File) Watch(ctx context.Context) { func (f *File) Watch(ctx context.Context) {
watcher, err := fsnotify.NewWatcher() watcher, err := dirwatch.New(filepath.Dir(f.params.Path))
if err == nil {
defer func() {
_ = watcher.Close()
}()
err = watcher.Add(filepath.Dir(f.params.Path))
}
if err != nil { if err != nil {
f.params.ProcessLog.Error( f.params.ProcessLog.Error(
"cannot watch the lookup database for replacements", "cannot watch the lookup database for replacements",
@@ -146,41 +134,22 @@ func (f *File) Watch(ctx context.Context) {
return return
} }
defer func() {
_ = watcher.Close()
}()
f.params.ProcessLog.Info("watching the lookup database for replacements", f.params.ProcessLog.Info("watching the lookup database for replacements",
"file", f.params.Path) "file", f.params.Path)
f.readAfterChanges(ctx, watcher.Events, watcher.Errors) watcher.CallWhenQuiet(ctx, f.isDatabase, f.readAgain, func(err error) {
f.params.ProcessLog.Warn("watching the lookup database failed",
"error", err.Error())
})
} }
// readAfterChanges reads the lookup database again once quietTime has // isDatabase reports whether path names the lookup database.
// passed without a change to it from events, until ctx is done, and logs func (f *File) isDatabase(path string) bool {
// the errors from errs. A change to another file in its directory does not return filepath.Clean(path) == filepath.Clean(f.params.Path)
// count. The wait starts at once, as if for a change, so that a file
// replaced after OpenFile read it, and before its directory was watched,
// is read too.
func (f *File) readAfterChanges(
ctx context.Context, events <-chan fsnotify.Event, errs <-chan error,
) {
path := filepath.Clean(f.params.Path)
quiet := time.NewTimer(quietTime)
defer quiet.Stop()
for {
select {
case <-ctx.Done():
return
case event := <-events:
if filepath.Clean(event.Name) == path {
quiet.Reset(quietTime)
}
case <-quiet.C:
f.readAgain()
case err := <-errs:
f.params.ProcessLog.Warn("watching the lookup database failed",
"error", err.Error())
}
}
} }
// readAgain reads the lookup database again, in place of the one in use, // readAgain reads the lookup database again, in place of the one in use,
+35 -141
View File
@@ -1,7 +1,6 @@
package lookup package lookup
import ( import (
"context"
"log/slog" "log/slog"
"net/netip" "net/netip"
"net/url" "net/url"
@@ -12,7 +11,6 @@ import (
"testing/synctest" "testing/synctest"
"time" "time"
"github.com/fsnotify/fsnotify"
"github.com/maxmind/mmdbwriter/mmdbtype" "github.com/maxmind/mmdbwriter/mmdbtype"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
@@ -133,66 +131,48 @@ func TestFileThatCannotBeReadIsAnError(t *testing.T) {
} }
} }
// The tests below run readAfterChanges in a synctest bubble, where time is func TestChangeOfAnotherFileInTheDirectoryIsNoReplacement(t *testing.T) {
// a clock of the test's own: time.Sleep moves it on at once, and t.Parallel()
// synctest.Wait returns once readAfterChanges waits again, so that every
// reading due by then is done. The test sends the changes itself, as the
// watch of a directory cannot run in a bubble.
func TestReplacementCopiedOverTheFileInTwoPartsIsReadOnlyWhole( dir := t.TempDir()
t *testing.T, path := filepath.Join(dir, "ipinfo_lite.mmdb")
) { writeDatabase(t, path, "DE")
f := openFile(t, path, newQueue())
for changed, want := range map[string]bool{
path: true,
dir + "/./ipinfo_lite.mmdb": true,
filepath.Join(dir, "other.mmdb"): false,
} {
if f.isDatabase(changed) != want {
t.Errorf(
"a change to %s counts: %t, want %t",
changed,
!want,
want,
)
}
}
}
// The tests below run in a synctest bubble, where time is a clock of the
// test's own, so that the time a file is read can be checked. The wait for
// the directory to go quiet before a reading is tested in dirwatch.
func TestReplacementIsReadAgain(t *testing.T) {
t.Parallel() t.Parallel()
synctest.Test(t, func(t *testing.T) { synctest.Test(t, func(t *testing.T) {
dir := t.TempDir() path := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb")
path := filepath.Join(dir, "ipinfo_lite.mmdb")
writeDatabase(t, path, "DE") writeDatabase(t, path, "DE")
queue := newQueue() queue := newQueue()
f := openFile(t, path, queue) f := openFile(t, path, queue)
changes := watch(t, f)
other := filepath.Join(dir, "replacement.mmdb") writeDatabase(t, path, "KP")
writeDatabase(t, other, "KP") time.Sleep(time.Hour)
f.readAgain()
//nolint:gosec // a file the test wrote
replacement, err := os.ReadFile(other)
if err != nil {
t.Fatalf("read %s: %v", other, err)
}
// The file in use is overwritten in place, and keeps giving what
// it gave. Its first part alone is not a .mmdb file.
file, err := os.Create(path) //nolint:gosec // a file the test wrote
if err != nil {
t.Fatalf("create %s: %v", path, err)
}
defer func() {
_ = file.Close()
}()
half := len(replacement) / 2
write(t, file, replacement[:half])
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantCountry(t, f, "DE")
// The second part starts the wait again.
write(t, file, replacement[half:])
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantCountry(t, f, "DE")
time.Sleep(time.Nanosecond)
synctest.Wait()
wantCountry(t, f, "KP") wantCountry(t, f, "KP")
if !f.LastRead().Equal(time.Now()) || f.ReadFailures() != 0 { if !f.LastRead().Equal(time.Now()) || f.ReadFailures() != 0 {
@@ -219,15 +199,10 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(
queue := newQueue() queue := newQueue()
f := openFile(t, path, queue) f := openFile(t, path, queue)
read := f.LastRead() read := f.LastRead()
changes := watch(t, f)
writeFile(t, path, "not a lookup database\n") writeFile(t, path, "not a lookup database\n")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
// Long after, the replacement has been read once.
time.Sleep(time.Hour) time.Sleep(time.Hour)
synctest.Wait() f.readAgain()
wantCountry(t, f, "DE") wantCountry(t, f, "DE")
if !f.LastRead().Equal(read) || f.ReadFailures() != 1 { if !f.LastRead().Equal(read) || f.ReadFailures() != 1 {
@@ -240,7 +215,7 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(
} }
wantAlerts(t, queue, alerts.Alert{ wantAlerts(t, queue, alerts.Alert{
Time: read.Add(quietTime), Time: read.Add(time.Hour),
Event: alerts.EventFileError, Event: alerts.EventFileError,
Reason: "the lookup database cannot be read, and the one read before stays in use", Reason: "the lookup database cannot be read, and the one read before stays in use",
Detail: map[string]any{ Detail: map[string]any{
@@ -252,55 +227,6 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(
}) })
} }
func TestChangeOfAnotherFileInTheDirectoryIsNoReplacement(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "ipinfo_lite.mmdb")
writeDatabase(t, path, "DE")
f := openFile(t, path, newQueue())
changes := watch(t, f)
// The wait that starts with the watch ends with a reading.
time.Sleep(quietTime)
synctest.Wait()
writeDatabase(t, path, "KP")
changes <- fsnotify.Event{Name: filepath.Join(dir, "other.mmdb"), Op: fsnotify.Create}
time.Sleep(quietTime)
synctest.Wait()
wantCountry(t, f, "DE")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime)
synctest.Wait()
wantCountry(t, f, "KP")
})
}
func TestReplacementSavedBeforeTheWatchStartsIsRead(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb")
writeDatabase(t, path, "DE")
f := openFile(t, path, newQueue())
// Saved after OpenFile read the file, and before its directory was
// watched, so that no change is seen for it.
writeDatabase(t, path, "KP")
watch(t, f)
time.Sleep(quietTime)
synctest.Wait()
wantCountry(t, f, "KP")
})
}
// newQueue returns a queue of alerts for a webhook that is never sent // newQueue returns a queue of alerts for a webhook that is never sent
// them, so that they wait in it for the test to look at. // them, so that they wait in it for the test to look at.
func newQueue() *alerts.Queue { func newQueue() *alerts.Queue {
@@ -344,28 +270,6 @@ func openFile(t *testing.T, path string, queue *alerts.Queue) *File {
return f return f
} }
// watch runs f's readAfterChanges until the test ends, and returns the
// channel that sends it changes.
func watch(t *testing.T, f *File) chan<- fsnotify.Event {
t.Helper()
changes := make(chan fsnotify.Event)
ctx, stop := context.WithCancel(t.Context())
stopped := make(chan struct{})
go func() {
f.readAfterChanges(ctx, changes, nil)
close(stopped)
}()
t.Cleanup(func() {
stop()
<-stopped
})
return changes
}
// wantCountry checks the country f gives testClient. // wantCountry checks the country f gives testClient.
func wantCountry(t *testing.T, f *File, want string) { func wantCountry(t *testing.T, f *File, want string) {
t.Helper() t.Helper()
@@ -401,13 +305,3 @@ func writeFile(t *testing.T, path, content string) {
t.Fatalf("write %s: %v", path, err) t.Fatalf("write %s: %v", path, err)
} }
} }
// write writes data to the end of file.
func write(t *testing.T, file *os.File, data []byte) {
t.Helper()
_, err := file.Write(data)
if err != nil {
t.Fatalf("write: %v", err)
}
}
+16 -7
View File
@@ -2,6 +2,7 @@ package rproxy
import ( import (
"bufio" "bufio"
"context"
"errors" "errors"
"io" "io"
"net" "net"
@@ -22,7 +23,9 @@ type requestBody struct {
// body is the visitor's body, ending in an *http.MaxBytesError past // body is the visitor's body, ending in an *http.MaxBytesError past
// SWW_RPROXY_MAX_REQUEST_SIZE. // SWW_RPROXY_MAX_REQUEST_SIZE.
body io.ReadCloser body io.ReadCloser
rq *request // refuse refuses the request with a status, as the request's own refuse
// does.
refuse func(status int)
// held is the part read before the request goes to the app, which Read gives // held is the part read before the request goes to the app, which Read gives
// first. // first.
held []byte held []byte
@@ -68,7 +71,7 @@ func (b *requestBody) Read(p []byte) (int, error) {
case errors.Is(err, io.EOF): case errors.Is(err, io.EOF):
b.received.Store(true) b.received.Store(true)
case errors.As(err, &tooLarge): case errors.As(err, &tooLarge):
b.rq.refuse(http.StatusRequestEntityTooLarge) b.refuse(http.StatusRequestEntityTooLarge)
} }
return n, err return n, err
@@ -86,7 +89,13 @@ type responseBody struct {
// body is the app's body, ending in an *http.MaxBytesError past // body is the app's body, ending in an *http.MaxBytesError past
// SWW_RPROXY_MAX_RESPONSE_SIZE. // SWW_RPROXY_MAX_RESPONSE_SIZE.
body io.ReadCloser body io.ReadCloser
rq *request // refuse refuses the request with a status, as the request's own refuse
// does.
refuse func(status int)
// responseReceived is called once the app's whole answer has arrived.
responseReceived func()
// ctx is the visitor's request's context.
ctx context.Context //nolint:containedctx // the body ends with the request
} }
// Read reads from the app's body. // Read reads from the app's body.
@@ -100,15 +109,15 @@ func (b *responseBody) Read(p []byte) (int, error) {
switch { switch {
case errors.Is(err, io.EOF): case errors.Is(err, io.EOF):
b.rq.responseReceived() b.responseReceived()
case errors.As(err, &tooLarge): case errors.As(err, &tooLarge):
b.rq.refuse(http.StatusBadGateway) b.refuse(http.StatusBadGateway)
return n, errResponseTooLarge return n, errResponseTooLarge
case b.rq.in.Context().Err() == nil: case b.ctx.Err() == nil:
// The answer broke off, not because the visitor went away. If a time // The answer broke off, not because the visitor went away. If a time
// limit cut it, that refusal came first and is the one kept. // limit cut it, that refusal came first and is the one kept.
b.rq.refuse(http.StatusBadGateway) b.refuse(http.StatusBadGateway)
} }
return n, err return n, err
+8 -3
View File
@@ -122,8 +122,8 @@ func (h *handler) newRequest(
if r.Body != http.NoBody { if r.Body != http.NoBody {
rq.body = &requestBody{ rq.body = &requestBody{
body: limitBody(r.Body, h.config.MaxRequestSize), body: limitBody(r.Body, h.config.MaxRequestSize),
rq: rq, refuse: rq.refuse,
} }
} }
@@ -291,7 +291,12 @@ func (rq *request) modifyResponse(res *http.Response) error {
// the request, and SWW_RPROXY_MAX_RESPONSE_TIME_MS starts. // the request, and SWW_RPROXY_MAX_RESPONSE_TIME_MS starts.
rq.startResponseTimer(requestEnd) rq.startResponseTimer(requestEnd)
res.Body = &responseBody{body: limitBody(res.Body, maxBytes), rq: rq} res.Body = &responseBody{
body: limitBody(res.Body, maxBytes),
refuse: rq.refuse,
responseReceived: rq.responseReceived,
ctx: rq.in.Context(),
}
rq.out.afterStatus = rq.setResponseDeadline rq.out.afterStatus = rq.setResponseDeadline
return nil return nil
+19 -48
View File
@@ -18,12 +18,11 @@ import (
"slices" "slices"
"strings" "strings"
"sync/atomic" "sync/atomic"
"time"
"github.com/fsnotify/fsnotify"
"golang.org/x/net/http/httpguts" "golang.org/x/net/http/httpguts"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/dirwatch"
) )
// The actions a rule takes when it matches. // The actions a rule takes when it matches.
@@ -40,12 +39,6 @@ const (
// extension ends the name of every rule file. // extension ends the name of every rule file.
const extension = ".rules" const extension = ".rules"
// quietTime is how long SWW_WAF_RULE_FILES_DIR must go without a change before
// the rule files are read again, so that a file still being written, such
// as one saved in place, appended to or copied in with scp, is read only
// once whole.
const quietTime = 2 * time.Second
// headerTarget starts the target that is one request header, // headerTarget starts the target that is one request header,
// header:<Name>. // header:<Name>.
const headerTarget = "header:" const headerTarget = "header:"
@@ -176,26 +169,18 @@ func (f *Files) Len() int {
} }
// Watch watches Dir until ctx is done, and reads the rule files again // Watch watches Dir until ctx is done, and reads the rule files again
// once Dir has had no change for quietTime, after one is edited, added or // once Dir has had no change for a short time, after one is edited, added
// removed, and after Watch starts watching. If they then hold an error, // or removed, and after Watch starts watching, as dirwatch describes. If
// the rules stay as they were, the error is logged with its file and // they then hold an error, the rules stay as they were, the error is
// line, and the files are read again after the next change. If Dir cannot // logged with its file and line, and the files are read again after the
// be watched, that is logged, and the rules stay as they were loaded. // next change. If Dir cannot be watched, that is logged, and the rules
// While Enabled is false, Watch returns at once. // stay as they were loaded. While Enabled is false, Watch returns at once.
func (f *Files) Watch(ctx context.Context) { func (f *Files) Watch(ctx context.Context) {
if !f.params.Enabled { if !f.params.Enabled {
return return
} }
watcher, err := fsnotify.NewWatcher() watcher, err := dirwatch.New(f.params.Dir)
if err == nil {
defer func() {
_ = watcher.Close()
}()
err = watcher.Add(f.params.Dir)
}
if err != nil { if err != nil {
f.params.ProcessLog.Error("cannot watch the rule files for edits", f.params.ProcessLog.Error("cannot watch the rule files for edits",
"error", err.Error()) "error", err.Error())
@@ -203,36 +188,22 @@ func (f *Files) Watch(ctx context.Context) {
return return
} }
defer func() {
_ = watcher.Close()
}()
f.params.ProcessLog.Info("watching the rule files for edits", f.params.ProcessLog.Info("watching the rule files for edits",
"directory", f.params.Dir) "directory", f.params.Dir)
f.readAfterChanges(ctx, watcher.Events, watcher.Errors) // A change to any file in Dir counts.
} watcher.CallWhenQuiet(ctx,
func(string) bool { return true },
// readAfterChanges reads the rule files again once quietTime has passed f.readAgain,
// without a change from events, until ctx is done, and logs the errors func(err error) {
// from errs. The wait starts at once, as if for a change, so that an edit
// saved after Load read the files, and before Dir was watched, is taken
// in too.
func (f *Files) readAfterChanges(
ctx context.Context, events <-chan fsnotify.Event, errs <-chan error,
) {
quiet := time.NewTimer(quietTime)
defer quiet.Stop()
for {
select {
case <-ctx.Done():
return
case <-events:
quiet.Reset(quietTime)
case <-quiet.C:
f.readAgain()
case err := <-errs:
f.params.ProcessLog.Warn("watching the rule files failed", f.params.ProcessLog.Warn("watching the rule files failed",
"error", err.Error()) "error", err.Error())
} },
} )
} }
// readAgain reads the rule files again, in place of the rules loaded, or // readAgain reads the rule files again, in place of the rules loaded, or
-165
View File
@@ -1,165 +0,0 @@
package rules
import (
"context"
"log/slog"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"slices"
"testing"
"testing/synctest"
"time"
"github.com/fsnotify/fsnotify"
"sneak.berlin/go/smallwebwaf/internal/alerts"
)
// The tests below run readAfterChanges in a synctest bubble, where time is
// a clock of the test's own: time.Sleep moves it on at once, and
// synctest.Wait returns once readAfterChanges waits again, so that every
// reading due by then is done. The test sends the changes itself, as the
// watch of a directory cannot run in a bubble.
func TestFileWrittenInTwoPartsTakenInOnlyWhole(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "50-app.rules")
writeFile(t, path, "first path block ^/first\n")
files := load(t, dir)
changes := run(t, files)
file, err := os.Create(path) //nolint:gosec // a file the test wrote
if err != nil {
t.Fatalf("create: %v", err)
}
defer func() {
_ = file.Close()
}()
// The first part ends in the middle of a ban rule's regex, which,
// read then, would ban every request.
write(t, file, "first path block ^/first\nprobe path ban ^/")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantMatched(t, files, "/anything")
// The second part starts the wait again.
write(t, file, `\.env$`+"\n")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantMatched(t, files, "/.env")
time.Sleep(time.Nanosecond)
synctest.Wait()
wantMatched(t, files, "/.env", "probe")
wantMatched(t, files, "/anything")
})
}
func TestEditSavedBeforeTheWatchStartsTakenIn(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "50-app.rules")
writeFile(t, path, "first path block ^/first\n")
files := load(t, dir)
// Saved after Load read the files, and before the directory was
// watched, so that no change is seen for it.
writeFile(t, path, "first path block ^/edited\n")
run(t, files)
time.Sleep(quietTime)
synctest.Wait()
wantMatched(t, files, "/edited", "first")
})
}
// load loads the rules in dir.
func load(t *testing.T, dir string) *Files {
t.Helper()
files, err := Load(Params{
Dir: dir, Enabled: true, ProcessLog: slog.New(slog.DiscardHandler),
Alerts: alerts.New(alerts.Params{}),
})
if err != nil {
t.Fatalf("load: %v", err)
}
return files
}
// run runs files' readAfterChanges until the test ends, and returns the
// channel that sends it changes.
func run(t *testing.T, files *Files) chan<- fsnotify.Event {
t.Helper()
changes := make(chan fsnotify.Event)
ctx, stop := context.WithCancel(t.Context())
stopped := make(chan struct{})
go func() {
files.readAfterChanges(ctx, changes, nil)
close(stopped)
}()
t.Cleanup(func() {
stop()
<-stopped
})
return changes
}
// writeFile writes content to the file at path.
func writeFile(t *testing.T, path, content string) {
t.Helper()
err := os.WriteFile(path, []byte(content), 0o600)
if err != nil {
t.Fatalf("write %s: %v", path, err)
}
}
// write writes text to the end of file.
func write(t *testing.T, file *os.File, text string) {
t.Helper()
_, err := file.WriteString(text)
if err != nil {
t.Fatalf("write: %v", err)
}
}
// wantMatched checks the ids of the rules that a GET request for path
// matches, in order.
func wantMatched(t *testing.T, files *Files, path string, want ...string) {
t.Helper()
r := httptest.NewRequestWithContext(t.Context(), http.MethodGet,
"http://app.example"+path, nil)
matched := files.Match(r)
got := make([]string, 0, len(matched))
for _, rule := range matched {
got = append(got, rule.ID)
}
if !slices.Equal(got, want) {
t.Errorf("%s matched %v, want %v", path, got, want)
}
}
+52 -30
View File
@@ -10,13 +10,14 @@ package sww
import ( import (
"context" "context"
"fmt" "errors"
"io" "io"
"os" "os"
"os/signal" "os/signal"
"syscall" "syscall"
"time" "time"
"github.com/spf13/cobra"
"github.com/spf13/viper" "github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
@@ -26,14 +27,7 @@ import (
// know. // know.
const usageStatus = 2 const usageStatus = 2
// usage is what sww prints then. var errNoCommand = errors.New("no command given")
const usage = `usage: sww <command>
commands:
rproxy the reverse proxy in front of one app
waf decides for rproxy, and keeps bans, counters and history
healthcheck the container's health check
`
// Params are what RunWAF and RunRProxy need from the process. // Params are what RunWAF and RunRProxy need from the process.
type Params struct { type Params struct {
@@ -51,14 +45,9 @@ type Params struct {
// Main runs the command os.Args names, and returns the process's exit status. // Main runs the command os.Args names, and returns the process's exit status.
// `sww rproxy` and `sww waf` run until SIGTERM or SIGINT. // `sww rproxy` and `sww waf` run until SIGTERM or SIGINT.
// With no command, or one it does not know, it prints the commands and returns // With no command, or one it does not know, cobra prints the error and the
// usageStatus. // commands to stderr, and Main returns usageStatus.
func Main(version string) int { func Main(version string) int {
args := os.Args[1:]
if len(args) == 0 {
return printUsage(os.Stderr)
}
ctx, stop := signal.NotifyContext(context.Background(), ctx, stop := signal.NotifyContext(context.Background(),
syscall.SIGTERM, os.Interrupt) syscall.SIGTERM, os.Interrupt)
defer stop() defer stop()
@@ -69,21 +58,54 @@ func Main(version string) int {
Stdout: os.Stdout, Stdout: os.Stdout,
} }
switch args[0] { // status is what the command that ran returns.
case "rproxy": status := 0
return RunRProxy(ctx, params)
case "waf": root := &cobra.Command{
return RunWAF(ctx, params) Use: "sww",
case "healthcheck": Short: "a web application firewall",
return HealthCheck(ctx, args[1:], params.Viper, os.Stderr) Args: cobra.NoArgs,
default: RunE: func(*cobra.Command, []string) error {
return printUsage(os.Stderr) return errNoCommand
},
} }
} root.CompletionOptions.DisableDefaultCmd = true
// printUsage writes the commands to w, and returns usageStatus. root.AddCommand(
func printUsage(w io.Writer) int { &cobra.Command{
_, _ = fmt.Fprint(w, usage) Use: "rproxy",
Short: "the reverse proxy in front of one app",
Run: func(cmd *cobra.Command, _ []string) {
status = RunRProxy(cmd.Context(), params)
},
},
&cobra.Command{
Use: "waf",
Short: "decides for rproxy, and keeps bans, counters and history",
Run: func(cmd *cobra.Command, _ []string) {
status = RunWAF(cmd.Context(), params)
},
},
&cobra.Command{
Use: "healthcheck",
Short: "the container's health check",
// HealthCheck refuses every argument itself, flags included.
DisableFlagParsing: true,
Run: func(cmd *cobra.Command, args []string) {
status = HealthCheck(
cmd.Context(),
args,
params.Viper,
cmd.ErrOrStderr(),
)
},
},
)
return usageStatus err := root.ExecuteContext(ctx)
if err != nil {
return usageStatus
}
return status
} }
+88 -29
View File
@@ -4,6 +4,7 @@ import (
"bytes" "bytes"
"context" "context"
"crypto/rand" "crypto/rand"
"errors"
"io" "io"
"net/http" "net/http"
"os" "os"
@@ -164,46 +165,104 @@ func wantUTC(t *testing.T, where string, value any) {
func TestMainWithoutACommandItKnowsPrintsTheCommands(t *testing.T) { func TestMainWithoutACommandItKnowsPrintsTheCommands(t *testing.T) {
t.Parallel() t.Parallel()
// completion is cobra's own command, which sww turns off.
for _, command := range []string{"", "serve", "completion"} {
t.Run("command "+command, func(t *testing.T) {
t.Parallel()
code, stdout, stderr := runMain(t, command)
if code != 2 {
t.Errorf("exit status %d, want 2", code)
}
wantTheCommands(t, "stderr", stderr)
if stdout != "" {
t.Errorf("stdout %q, want nothing", stdout)
}
})
}
}
func TestMainHelpPrintsTheCommands(t *testing.T) {
t.Parallel()
for _, command := range []string{"help", "--help"} {
t.Run("command "+command, func(t *testing.T) {
t.Parallel()
code, stdout, _ := runMain(t, command)
if code != 0 {
t.Errorf("exit status %d, want 0", code)
}
wantTheCommands(t, "stdout", stdout)
})
}
}
// wantTheCommands checks that text, what Main wrote to where, names each
// command.
func wantTheCommands(t *testing.T, where, text string) {
t.Helper()
for _, name := range []string{"rproxy", wafCommand, "healthcheck"} {
if !strings.Contains(text, name) {
t.Errorf("%s %q does not name %s", where, text, name)
}
}
}
func TestMainHealthCheckRefusesAnArgument(t *testing.T) {
t.Parallel()
for _, argument := range []string{"now", "--x", "-h"} {
t.Run("argument "+argument, func(t *testing.T) {
t.Parallel()
code, stdout, stderr := runMain(t, "healthcheck "+argument)
want := "sww healthcheck: unexpected argument \"" +
argument + "\"\n"
if code != 1 || stderr != want || stdout != "" {
t.Errorf("exit status %d, stdout %q and stderr %q, "+
"want 1, nothing and %q", code, stdout, stderr, want)
}
})
}
}
// runMain runs Main with command in a process of its own, and returns its
// exit status and what it wrote to stdout and stderr.
func runMain(t *testing.T, command string) (int, string, string) {
t.Helper()
binary, err := os.Executable() binary, err := os.Executable()
if err != nil { if err != nil {
t.Fatalf("find the test binary: %v", err) t.Fatalf("find the test binary: %v", err)
} }
for _, command := range []string{"", "serve"} { ctx, cancel := context.WithTimeout(t.Context(), waitLimit)
t.Run("command "+command, func(t *testing.T) { defer cancel()
t.Parallel()
ctx, cancel := context.WithTimeout(t.Context(), waitLimit) var stdout, stderr bytes.Buffer
defer cancel()
var stdout, stderr bytes.Buffer //nolint:gosec // this test binary
cmd := exec.CommandContext(ctx, binary)
cmd.Stdout, cmd.Stderr = &stdout, &stderr
cmd := exec.CommandContext(ctx, binary) cmd.Env = append(os.Environ(), commandForMain+"="+command)
cmd.Stdout, cmd.Stderr = &stdout, &stderr
cmd.Env = append(os.Environ(), commandForMain+"="+command) err = cmd.Run()
err := cmd.Run() var exitErr *exec.ExitError
if err != nil && !errors.As(err, &exitErr) {
if code := cmd.ProcessState.ExitCode(); code != 2 { t.Fatalf("run: %v", err)
t.Errorf("exit status %d (%v), want 2", code, err)
}
for _, name := range []string{"rproxy", wafCommand, "healthcheck"} {
if !strings.Contains(stderr.String(), name) {
t.Errorf(
"stderr %q does not name %s",
stderr.String(),
name,
)
}
}
if stdout.Len() != 0 {
t.Errorf("stdout %q, want nothing", stdout.String())
}
})
} }
return cmd.ProcessState.ExitCode(), stdout.String(), stderr.String()
} }
// ask sends waf at addr rproxy's question about a request for path from // ask sends waf at addr rproxy's question about a request for path from
+9
View File
@@ -43,6 +43,15 @@ func (b blocklists) check(
} }
} }
// refusesWhateverTheLookupAnswers reports whether blocklists refuses rq, as
// refusedByBlocklists says. The lookup's answer plays no part in it.
func (b blocklists) refusesWhateverTheLookupAnswers(
rq *request,
_ time.Time,
) bool {
return b.refusedByBlocklists(b.lists.ListedBy(rq.client))
}
// refusedByBlocklists reports whether the heuristic blocklists refuses a // refusedByBlocklists reports whether the heuristic blocklists refuses a
// client, given listedBy, the blocklists that list it: it does when one lists // client, given listedBy, the blocklists that list it: it does when one lists
// it and SWW_WAF_BLOCKLIST_ACTION is deny. // it and SWW_WAF_BLOCKLIST_ACTION is deny.
+19
View File
@@ -9,6 +9,16 @@ import (
// and holds the settings and parts it reads. Add one line for it to one of the // and holds the settings and parts it reads. Add one line for it to one of the
// lists that newHandler builds, which hands it those settings and parts. Its // lists that newHandler builds, which hands it those settings and parts. Its
// settings go in config.WAF, and the comment at the top of its file names them. // settings go in config.WAF, and the comment at the top of its file names them.
// A heuristic of afterLookup that can refuse a request whatever the lookup
// answers also implements refusesWithoutLookup.
//
// Three places outside its file may change with it:
// - waitForGeoJS in newHandler lists the settings that need the lookup's
// answer;
// - requestCanGetTag in internal/config/waf.go holds which setting can give a
// request which tag;
// - findBesideLookup does the work of each heuristic of besideLookup, by
// name.
// heuristic is one of the checks that decide what waf does with a // heuristic is one of the checks that decide what waf does with a
// request. // request.
@@ -20,6 +30,15 @@ type heuristic interface {
check(ctx context.Context, rq *request, now time.Time) *verdict check(ctx context.Context, rq *request, now time.Time) *verdict
} }
// refusesWithoutLookup is a heuristic of afterLookup that can tell, before the
// lookup answers, that it will refuse a request whatever the answer.
type refusesWithoutLookup interface {
// refusesWhateverTheLookupAnswers reports whether the heuristic will refuse
// rq at now, whatever the lookup answers about its client. It changes
// nothing.
refusesWhateverTheLookupAnswers(rq *request, now time.Time) bool
}
// verdict is a heuristic refusing a request, or banning its client: the action // verdict is a heuristic refusing a request, or banning its client: the action
// the log line names, one of requestlog's Action constants, and the figures the // the log line names, one of requestlog's Action constants, and the figures the
// heuristic acted on, by name, such as the count and the limit. // heuristic acted on, by name, such as the count and the limit.
+25 -31
View File
@@ -177,30 +177,6 @@ func newHandler(params Params) *handler {
ipv6GroupPrefix: cfg.IPv6GroupPrefix, ipv6GroupPrefix: cfg.IPv6GroupPrefix,
} }
h.blocklists = blocklists{
lists: params.Lists,
action: cfg.BlocklistAction,
hitNoter: hitNoter,
}
h.rateLimits = rateLimits{
limiter: params.Limiter,
exemptNets: cfg.RateLimitExemptNets,
exemptPaths: cfg.RateLimitExemptPaths,
banResponse: cfg.BanResponse,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
biasedThresholds: biasedThresholds{
lists: params.Lists,
tagLimitPercent: cfg.TagLimitPercent,
asnBytesLimitPercent: cfg.ASNBytesLimitPercent,
countryBytesLimitPercent: cfg.CountryBytesLimitPercent,
unknownCountryLimitPercent: cfg.UnknownCountryLimitPercent,
blocklistAction: cfg.BlocklistAction,
blocklistLimitPercent: cfg.BlocklistLimitPercent,
dnsblAbuseIPDBAction: cfg.DNSBLAbuseIPDBAction,
dnsblAbuseIPDBLimitPercent: cfg.DNSBLAbuseIPDBLimitPercent,
},
banMaker: sharedBanMaker,
}
h.trapPaths = trapPaths{paths: cfg.TrapPaths, banMaker: sharedBanMaker} h.trapPaths = trapPaths{paths: cfg.TrapPaths, banMaker: sharedBanMaker}
h.ruleFiles = ruleFiles{ h.ruleFiles = ruleFiles{
rules: params.Rules, rules: params.Rules,
@@ -230,7 +206,11 @@ func newHandler(params Params) *handler {
} }
h.afterLookup = []heuristic{ h.afterLookup = []heuristic{
countries{allowed: cfg.ExclusivelyAllowedCountries}, countries{allowed: cfg.ExclusivelyAllowedCountries},
h.blocklists, blocklists{
lists: params.Lists,
action: cfg.BlocklistAction,
hitNoter: hitNoter,
},
dnsbl{ dnsbl{
dnsbl: params.DNSBL, dnsbl: params.DNSBL,
action: cfg.DNSBLAbuseIPDBAction, action: cfg.DNSBLAbuseIPDBAction,
@@ -246,7 +226,25 @@ func newHandler(params Params) *handler {
hitNoter: hitNoter, hitNoter: hitNoter,
}, },
tagDeny{tags: cfg.TagDeny}, tagDeny{tags: cfg.TagDeny},
h.rateLimits, rateLimits{
limiter: params.Limiter,
exemptNets: cfg.RateLimitExemptNets,
exemptPaths: cfg.RateLimitExemptPaths,
banResponse: cfg.BanResponse,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
biasedThresholds: biasedThresholds{
lists: params.Lists,
tagLimitPercent: cfg.TagLimitPercent,
asnBytesLimitPercent: cfg.ASNBytesLimitPercent,
countryBytesLimitPercent: cfg.CountryBytesLimitPercent,
unknownCountryLimitPercent: cfg.UnknownCountryLimitPercent,
blocklistAction: cfg.BlocklistAction,
blocklistLimitPercent: cfg.BlocklistLimitPercent,
dnsblAbuseIPDBAction: cfg.DNSBLAbuseIPDBAction,
dnsblAbuseIPDBLimitPercent: cfg.DNSBLAbuseIPDBLimitPercent,
},
banMaker: sharedBanMaker,
},
} }
h.besideLookup = []heuristic{h.trapPaths, h.ruleFiles, h.coreRuleSet} h.besideLookup = []heuristic{h.trapPaths, h.ruleFiles, h.coreRuleSet}
h.afterBesideLookup = []heuristic{tagDeny{tags: cfg.TagDeny}} h.afterBesideLookup = []heuristic{tagDeny{tags: cfg.TagDeny}}
@@ -323,11 +321,7 @@ type handler struct {
// afterAnswer, they only ban. // afterAnswer, they only ban.
onReport []heuristic onReport []heuristic
// The heuristics of the lists that other steps of a request use too: // The heuristics of besideLookup, whose work findBesideLookup does.
// refusedWhateverTheAnswer asks blocklists and rateLimits, and
// findBesideLookup does the work of trapPaths, ruleFiles and coreRuleSet.
blocklists blocklists
rateLimits rateLimits
trapPaths trapPaths trapPaths trapPaths
ruleFiles ruleFiles ruleFiles ruleFiles
coreRuleSet coreRuleSet coreRuleSet coreRuleSet
+8 -5
View File
@@ -67,11 +67,14 @@ func (r rateLimits) check(
} }
} }
// overAWholeRateLimit reports whether the heuristic rate_limits will refuse rq // refusesWhateverTheLookupAnswers reports whether the heuristic rate_limits
// at now, whatever its client's limit percentages: the rate limits count it, // will refuse rq at now, whatever its client's limit percentages, which the
// and one more request from its client takes it over a whole rate limit. It // lookup's answer can lower: the rate limits count it, and one more request
// counts nothing. // from its client takes it over a whole rate limit. It counts nothing.
func (r rateLimits) overAWholeRateLimit(rq *request, now time.Time) bool { func (r rateLimits) refusesWhateverTheLookupAnswers(
rq *request,
now time.Time,
) bool {
return r.countedByRateLimits(rq) && return r.countedByRateLimits(rq) &&
r.limiter.OverARateLimit( r.limiter.OverARateLimit(
clientGroup(r.ipv6GroupPrefix, rq.client), clientGroup(r.ipv6GroupPrefix, rq.client),
+10 -6
View File
@@ -72,18 +72,22 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// refusedWhateverTheAnswer reports whether the request is one that does not wait // refusedWhateverTheAnswer reports whether the request is one that does not wait
// for GeoJS, since the checks will refuse it at now whatever GeoJS answers about // for GeoJS, since the checks will refuse it at now whatever GeoJS answers about
// its client: the heuristic blocklists or rate_limits will refuse it, as a // its client: a heuristic of afterLookup that implements refusesWithoutLookup
// function in that heuristic's file says. In observe mode, which refuses // says it will refuse it. In observe mode, which refuses nothing, none is.
// nothing, none is. rproxy lets a request through when waf answers too late. // rproxy lets a request through when waf answers too late.
func (rq *request) refusedWhateverTheAnswer(now time.Time) bool { func (rq *request) refusedWhateverTheAnswer(now time.Time) bool {
if rq.h.config.Observe { if rq.h.config.Observe {
return false return false
} }
listedBy := rq.h.blocklists.lists.ListedBy(rq.client) for _, each := range rq.h.afterLookup {
knows, ok := each.(refusesWithoutLookup)
if ok && knows.refusesWhateverTheLookupAnswers(rq, now) {
return true
}
}
return rq.h.blocklists.refusedByBlocklists(listedBy) || return false
rq.h.rateLimits.overAWholeRateLimit(rq, now)
} }
// answerQuestion runs the checks on the visitor's request a question // answerQuestion runs the checks on the visitor's request a question