Compare commits

..
Author SHA1 Message Date
clawbot 68bb6c539e waf: name the blacklist bans in the package doc
check / check (push) Failing after 1m38s
The package doc said waf answers rproxy's questions, takes its reports
and answers its own endpoints. The bans for AbuseIPDB's blacklist, made
as the reputation package fetches it, are none of these, so the new
file abuseipdbblacklist.go was outside it.

Model: opus-5-5
2026-11-19 11:13:01 +00:00
clawbot bba3805372 waf: ban the clients on AbuseIPDB's blacklist (closes #411)
While SWW_WAF_ABUSEIPDB_KEY is set, waf fetches AbuseIPDB's blacklist at
the start and every SWW_WAF_ABUSEIPDB_BLACKLIST_REFRESH_INTERVAL_MINS
(60), cut at SWW_WAF_ABUSEIPDB_THRESHOLD (90, from 25 to 100). Each
listed client's netblock gets a ban of SWW_WAF_ABUSEIPDB_BAN_DAYS (30)
with the new cause abuseipdb. The blacklist itself is not kept.

These bans do not count toward SWW_WAF_MAX_BANS, so a 10,000-address
blacklist cannot push out the bans for limits and attacks. An ended
abuseipdb ban is forgotten at each fetch and each read of bans.json. A
netblock with an active ban is left alone, and so is one whose abuseipdb
ban an admin lifted, until that ban would have ended. Observe mode makes
no ban.

Model: opus-5-5
2026-11-19 11:13:01 +00:00
98 changed files with 2479 additions and 4174 deletions

No files matched your search

+3 -9
View File
@@ -902,19 +902,13 @@ stops the running process (see [`docs/state-files.md`](docs/state-files.md)).
- `internal/rproxy`: the reverse proxy in front of one app.
- `internal/waf`: `waf`'s server, which runs the checks and answers its own
endpoints.
- `internal/wafprotocol`: the question, the answer and the report that `rproxy`
and `waf` send each other.
- `internal/ratelimit`, `internal/bans`, `internal/rules`,
`internal/coreruleset`, `internal/geojs`, `internal/lookup`,
`internal/reputation`, `internal/routewatch`, `internal/anomaly` and
`internal/alerts`: one check or source each.
- `internal/dirwatch`: watches a directory, so that `internal/rules` and
`internal/lookup` read their files again when they change.
- `internal/state`: the state and cache files.
- `internal/requestlog`: the request log.
- `internal/processlog`: the process's own messages.
- `internal/remotelog`: the syslog sender.
- `internal/metrics`: the metrics.
- `internal/state`, `internal/requestlog`, `internal/remotelog` and
`internal/metrics`: the state and cache files, the log lines, the syslog
sender and the metrics.
## Rationale
+7 -18
View File
@@ -69,21 +69,17 @@ never as a time.
that command's settings, and runs `internal/rproxy` until `SIGTERM` or
`SIGINT` (see "`rproxy`"). `sww healthcheck` is the image's health check.
With no command, or one it does not know, the binary prints the commands
and exits with status 2. `sww help` and `sww --help` print them and exit
with status 0. `internal/sww` is the only package that imports
and exits with status 2. `internal/sww` is the only package that imports
`go.uber.org/fx`. fx builds the parts of `waf` and `rproxy`, and starts
and stops their background work.
- `internal/config`: reads the settings, the one place they are read.
- `internal/waf`: `waf`'s server. It runs the checks in `check`. Its
`routes.go` lists `waf`'s own endpoints. It counts bytes, refusals and
anomalies, and writes the log lines.
- `internal/waf`: `waf`'s server. It runs the checks in `check`, and holds
`waf`'s question, answer and report. It answers `waf`'s own endpoints in
`answerOwnEndpoint`. It counts bytes, refusals and anomalies, and writes
the log lines.
- `internal/rproxy`: the reverse proxy in front of one app that asks `waf`
about each request. It passes a request to the app with the standard
library's `httputil.ReverseProxy`, within the time and size limits.
- `internal/wafprotocol`: the question, the answer and the report that
`rproxy` and `waf` send each other (see "`waf`"). It writes and reads each
one, and declares every header, path and limit both sides use.
`internal/rproxy` imports it, not `internal/waf`.
- `internal/ratelimit`: the table of clients. It counts requests, bytes and
refusals, and keeps each client's history.
- `internal/bans`: the bans. It holds each netblock's bans with their notes,
@@ -97,9 +93,6 @@ never as a time.
imports no other package of this repository: `waf` gives it what it needs
to count and report failures.
- `internal/lookup`: looks clients up in the lookup database.
- `internal/dirwatch`: watches a directory, and calls a function once it has
gone a short time without a change. `internal/rules` and `internal/lookup`
read their files again this way.
- `internal/reputation`: fetches the lists, asks the DNS blocklists and
AbuseIPDB in the background, and keeps their answers. It also fetches
AbuseIPDB's blacklist, which `internal/waf` bans.
@@ -111,10 +104,8 @@ never as a time.
its own.
- `internal/state`: reads the state and cache files at start, takes in edits
while running, and writes the files when due and at the stop.
- `internal/requestlog`: the request log, as JSON lines on stdout.
- `internal/processlog`: the process lines on stdout, which hold the
process's own messages.
- `internal/remotelog`: sends the lines on stdout to the syslog server.
- `internal/requestlog` and `internal/remotelog`: the lines on stdout, and
sending them to the syslog server.
- `internal/metrics`: the metrics, in the Prometheus text format.
- `Dockerfile`: the lint and test phases, then the image.
`deploy/example-app` is an app built on the image, with `rproxy` in its
@@ -128,13 +119,11 @@ never as a time.
default `SWW_WAF_CORE_RULE_SET_DISABLED_RULES`, are built for that
version. They are checked again before the version changes, since rule
ids, and what each rule matches, change between versions;
- `github.com/go-chi/chi/v5` to route the requests to `waf`'s own endpoints;
- `github.com/hashicorp/golang-lru/v2` for the bounded tables;
- `github.com/oschwald/maxminddb-golang/v2` to read the lookup database;
- `github.com/fsnotify/fsnotify` to notice edited files;
- `github.com/prometheus/client_golang` for the metrics;
- `github.com/spf13/viper` to read the settings from the environment;
- `github.com/spf13/cobra` for the commands and their help text;
- `github.com/maxmind/mmdbwriter`, in the tests, to write lookup databases;
- remote log sending written in this project, since the standard library's
`log/syslog` is frozen and writes only the older syslog format.
+1 -4
View File
@@ -7,15 +7,12 @@ require (
github.com/corazawaf/coraza-coreruleset/v4 v4.25.0
github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68
github.com/fsnotify/fsnotify v1.10.1
github.com/go-chi/chi/v5 v5.5.0
github.com/hashicorp/golang-lru/v2 v2.0.7
github.com/maxmind/mmdbwriter v1.2.0
github.com/oschwald/maxminddb-golang/v2 v2.7.0
github.com/prometheus/client_golang v1.24.1
github.com/spf13/cobra v1.10.2
github.com/spf13/viper v1.21.0
go.uber.org/fx v1.24.0
golang.org/x/net v0.60.0
golang.org/x/sys v0.48.0
)
@@ -28,7 +25,6 @@ require (
github.com/goccy/go-yaml v1.19.2 // indirect
github.com/gotnospirit/makeplural v0.0.0-20180622080156-a5f48d94d976 // indirect
github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/kaptinlin/go-i18n v0.1.4 // indirect
github.com/kaptinlin/jsonschema v0.4.6 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
@@ -54,6 +50,7 @@ require (
go.uber.org/zap v1.26.0 // indirect
go.yaml.in/yaml/v3 v3.0.5 // indirect
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba // indirect
golang.org/x/net v0.60.0 // indirect
golang.org/x/sync v0.23.0 // indirect
golang.org/x/text v0.42.0 // indirect
google.golang.org/protobuf v1.36.11 // indirect
-11
View File
@@ -14,15 +14,12 @@ github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68 h1:KLEjvBBSw
github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68/go.mod h1:9XocblQEwK6EYylRth7PrPIifF2l0iSoYo4FAXhXU/8=
github.com/corazawaf/libinjection-go v0.3.4 h1:jX2Tit2xZxYJJyVryNXucYBZtQqS7473x9Nc6lgRRbk=
github.com/corazawaf/libinjection-go v0.3.4/go.mod h1:Ik/+w3UmTWH9yn366RgS9D95K3y7Atb5m/H/gXzzPCk=
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/foxcpp/go-mockdns v1.2.0 h1:omK3OrHRD1IWJz1FuFBCFquhXslXoF17OvBS6JPzZF0=
github.com/foxcpp/go-mockdns v1.2.0/go.mod h1:IhLeSFGed3mJIAXPH2aiRQB+kqz7oqu8ld2qVbOu7Wk=
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho=
github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo=
github.com/go-chi/chi/v5 v5.5.0 h1:xdKQ1dJmVJsmgahrXSKhUx8NyVPzpwdlyrrQFpMPJgg=
github.com/go-chi/chi/v5 v5.5.0/go.mod h1:d1IG7GtKPSjPyBP365PH8sAGVHk3qfB/IhjFNRH1wWM=
github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
@@ -37,8 +34,6 @@ github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 h1:c7gcN
github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092/go.mod h1:ZZAN4fkkful3l1lpJwF8JbW41ZiG9TwJ2ZlqzQovBNU=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/jcchavezs/mergefs v0.1.1 h1:D45R17m6dHnSVZefnhynoeZvcK2Uw0oTrRfoUOQ0S5Y=
github.com/jcchavezs/mergefs v0.1.1/go.mod h1:eRLTrsA+vFwQZ48hj8p8gki/5v9C2bFtHH5Mnn4bcGk=
github.com/kaptinlin/go-i18n v0.1.4 h1:wCiwAn1LOcvymvWIVAM4m5dUAMiHunTdEubLDk4hTGs=
@@ -77,7 +72,6 @@ github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+
github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY=
github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/sagikazarmark/locafero v0.11.0 h1:1iurJgmM9G3PA/I+wWYIOw/5SyBtxapeHDcg+AAIFXc=
github.com/sagikazarmark/locafero v0.11.0/go.mod h1:nVIGvgyzw595SUSUE6tvCp3YYTeHs15MvlmU87WwIik=
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 h1:+jumHNA0Wrelhe64i8F6HNlS8pkoyMv5sreGx2Ry5Rw=
@@ -86,9 +80,6 @@ github.com/spf13/afero v1.15.0 h1:b/YBCLWAJdFWJTN9cLhiXXcD7mzKn9Dm86dNnfyQw1I=
github.com/spf13/afero v1.15.0/go.mod h1:NC2ByUVxtQs4b3sIUphxK0NioZnmxgyCrfzeuq8lxMg=
github.com/spf13/cast v1.10.0 h1:h2x0u2shc1QuLHfxi+cTJvs30+ZAHOGRic8uyGTDWxY=
github.com/spf13/cast v1.10.0/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqeHo=
github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.21.0 h1:x5S+0EU27Lbphp4UKm1C+1oQO+rKx36vfCoaVebLFSU=
@@ -118,7 +109,6 @@ go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba h1:0b9z3AuHCjxk0x/opv64kcgZLBseWJUpBw5I82+2U4M=
@@ -137,7 +127,6 @@ golang.org/x/tools v0.50.0 h1:c2ifzfcuY7L90lZ2aKd8S4K2NpASF08SZx9ZuJkHmSU=
golang.org/x/tools v0.50.0/go.mod h1:7ulVMw3831Mwi5EZD6RomGyffr4VFjuNYXf2BbCEAV0=
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
rsc.io/binaryregexp v0.2.0 h1:HfqmD5MEmC0zvwBuF187nq9mdnXjXsSivRiXN7SmRkE=
+4 -8
View File
@@ -54,11 +54,7 @@ func TestBlacklistLeavesANetblockWithAnActiveBanAsItIs(t *testing.T) {
now := midnight()
scores := map[netip.Prefix]int64{netblock: 100}
limitBan, _ := ledger.Ban(
netblock,
now,
bans.Grounds{Cause: bans.CauseLimit},
)
limitBan, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
if made := ledger.BanForAbuseIPDB(now, now.Add(30*day), scores); made != 0 {
t.Errorf("made %d bans over a ban for a limit, want none", made)
@@ -123,7 +119,7 @@ func TestEndedBlacklistBansAreForgottenAtTheNextFetch(t *testing.T) {
expires := now.Add(30 * day)
ledger.BanForAbuseIPDB(now, expires, map[netip.Prefix]int64{listed: 100})
ledger.Ban(limited, now, bans.Grounds{Cause: bans.CauseLimit})
ledger.BanForLimit(limited, now, bans.Notes{})
// listed is off the blacklist when it is fetched after the ban's end.
if made := ledger.BanForAbuseIPDB(expires, expires.Add(30*day), nil); made != 0 {
@@ -148,7 +144,7 @@ func TestBlacklistBansDoNotCountTowardMaxBans(t *testing.T) {
scores[netip.PrefixFrom(netip.AddrFrom4([4]byte{198, 51, 100, byte(i)}), 32)] = 100
}
ledger.Ban(limited, now, bans.Grounds{Cause: bans.CauseLimit})
ledger.BanForLimit(limited, now, bans.Notes{})
if made := ledger.BanForAbuseIPDB(now, now.Add(30*day), scores); made != 3 {
t.Errorf("made %d bans for 3 netblocks, want 3", made)
@@ -158,7 +154,7 @@ func TestBlacklistBansDoNotCountTowardMaxBans(t *testing.T) {
// The ban for an attack takes the place of the one for a limit, and no
// blacklist ban goes.
ledger.Ban(attacker, now, bans.Grounds{Cause: bans.CauseAttack})
ledger.BanForAttack(attacker, now, bans.Notes{})
want := map[netip.Prefix]int{limited: 0, attacker: 1}
for netblock := range scores {
+23 -64
View File
@@ -61,7 +61,7 @@ func TestAdminsBansAreNeverDroppedAndDoNotCountTowardMaxBans(t *testing.T) {
wantBans(t, ledger, map[netip.Prefix]int{adminsOnly: 2, both: 2})
// A new ban drops the ban sww made, and only that one.
ledger.Ban(second, midnight(), bans.Grounds{Cause: bans.CauseLimit})
ledger.BanForLimit(second, midnight(), bans.Notes{})
wantBans(
t,
ledger,
@@ -77,7 +77,7 @@ func TestAdminsBansAreNeverDroppedAndDoNotCountTowardMaxBans(t *testing.T) {
}
// And the next drops that one.
ledger.Ban(third, midnight(), bans.Grounds{Cause: bans.CauseLimit})
ledger.BanForLimit(third, midnight(), bans.Notes{})
wantBans(
t,
ledger,
@@ -98,28 +98,20 @@ func TestEarlierBansCountABanDroppedAfterAnAdminsBan(t *testing.T) {
// twice, each time after its last ban has ended.
ledger.BanForAdmin(netblock, midnight(), midnight().Add(time.Minute),
"probes for logins")
first, _ := ledger.Ban(
first, _ := ledger.BanForLimit(
netblock,
midnight().Add(time.Minute),
bans.Grounds{Cause: bans.CauseLimit},
)
second, _ := ledger.Ban(
netblock,
first.Expires,
bans.Grounds{Cause: bans.CauseLimit},
bans.Notes{},
)
second, _ := ledger.BanForLimit(netblock, first.Expires, bans.Notes{})
// A ban on another netblock fills the ledger, and the netblock's first
// ban for a limit, its earliest that is not an admin's, goes.
ledger.Ban(other, second.Start, bans.Grounds{Cause: bans.CauseLimit})
ledger.BanForLimit(other, second.Start, bans.Notes{})
wantBans(t, ledger, map[netip.Prefix]int{netblock: 2, other: 1})
// The netblock's next ban still counts it.
third, _ := ledger.Ban(
netblock,
second.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
third, _ := ledger.BanForLimit(netblock, second.Expires, bans.Notes{})
if third.Notes.EarlierBans != (bans.EarlierBans{Limit: 2, Admin: 1}) {
t.Errorf("the third ban for a limit has earlier bans %+v, "+
"want 2 for a limit and 1 an admin's", third.Notes.EarlierBans)
@@ -135,27 +127,15 @@ func TestEarlierBansCountABanAnAdminKeepsOnce(t *testing.T) {
// The netblock breaks a limit twice, the second time after its first
// ban has ended, and the second ban's notes count the first. An admin
// then keeps the first ban.
first, _ := ledger.Ban(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
second, _ := ledger.Ban(
netblock,
first.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
second, _ := ledger.BanForLimit(netblock, first.Expires, bans.Notes{})
kept := first
kept.Cause = bans.CauseAdmin
ledger.LoadEdit([]bans.Ban{kept, second})
// The netblock's next ban counts the kept ban once, as an admin's.
third, _ := ledger.Ban(
netblock,
second.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
third, _ := ledger.BanForLimit(netblock, second.Expires, bans.Notes{})
if third.Notes.EarlierBans != (bans.EarlierBans{Limit: 1, Admin: 1}) {
t.Errorf("the third ban for a limit has earlier bans %+v, "+
"want 1 for a limit and 1 an admin's", third.Notes.EarlierBans)
@@ -167,37 +147,20 @@ func TestReasonOfTheBansSwwMakes(t *testing.T) {
ledger := bans.New(defaultParams())
limit, _ := ledger.Ban(
limit, _ := ledger.BanForLimit(
netip.MustParsePrefix("203.0.113.1/32"),
midnight(),
bans.Grounds{
Cause: bans.CauseLimit,
Notes: bans.Notes{
Kind: "requests",
Limit: 1000,
Window: "minute",
},
},
bans.Notes{Kind: "requests", Limit: 1000, Window: "minute"},
)
byteLimit, _ := ledger.Ban(
byteLimit, _ := ledger.BanForLimit(
netip.MustParsePrefix("203.0.113.3/32"),
midnight(),
bans.Grounds{
Cause: bans.CauseLimit,
Notes: bans.Notes{
Kind: "bytes",
Limit: 10 << 30,
Window: "hour",
},
},
bans.Notes{Kind: "bytes", Limit: 10 << 30, Window: "hour"},
)
attack, _ := ledger.Ban(
attack, _ := ledger.BanForAttack(
netip.MustParsePrefix("203.0.113.2/32"),
midnight(),
bans.Grounds{
Cause: bans.CauseAttack,
Notes: bans.Notes{RuleID: "git-dir", Target: "path"},
},
bans.Notes{RuleID: "git-dir", Target: "path"},
)
for _, tc := range []struct{ got, want string }{
@@ -237,7 +200,7 @@ func TestLiftedBanForALimitRefusesNothingAndMakesNoBanLonger(t *testing.T) {
t.Error("the lifted ban refuses")
}
ban, _ := ledger.Ban(netblock, now, bans.Grounds{Cause: bans.CauseLimit})
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
if ban.Expires.Sub(ban.Start) != time.Hour ||
ban.Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) {
t.Errorf(
@@ -288,11 +251,7 @@ func TestLiftedBanForAnAttackRefusesNothingAndMakesNoBanLonger(
}
// The next clear sign of attack bans for seven days, as a first does.
ban, _ := ledger.Ban(
netblock,
now,
bans.Grounds{Cause: bans.CauseAttack},
)
ban, _ := ledger.BanForAttack(netblock, now, bans.Notes{})
if ban.Expires.Sub(ban.Start) != 7*day {
t.Errorf(
"the next ban for an attack ends at %s, want seven days on",
@@ -305,10 +264,10 @@ func TestLoadEditCountsTheBansAnAdminMade(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultParams())
made, _ := ledger.Ban(
made, _ := ledger.BanForLimit(
netip.MustParsePrefix("203.0.113.1/32"),
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
bans.Notes{},
)
atStart := bans.Ban{
Netblock: netip.MustParsePrefix("203.0.113.2/32"),
@@ -352,7 +311,7 @@ func TestAdminsBanIsMadeWhileAnotherLasts(t *testing.T) {
ledger := bans.New(defaultParams())
// An hour's ban for a broken limit.
ledger.Ban(netblock, midnight(), bans.Grounds{Cause: bans.CauseLimit})
ledger.BanForLimit(netblock, midnight(), bans.Notes{})
wantChanged(t, ledger, true)
// A minute later an admin bans the netblock for good, named by an
@@ -472,10 +431,10 @@ func TestPermanentBanOnAWiderNetblockIsReturnedOverATemporaryBan(
ledger := bans.New(defaultParams())
// The temporary ban is made first, so that its netblock is looked at first.
ledger.Ban(
ledger.BanForLimit(
netip.MustParsePrefix("203.0.113.9/32"),
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
bans.Notes{},
)
ledger.BanForAdmin(
netip.MustParsePrefix("203.0.113.0/24"),
+82 -54
View File
@@ -300,50 +300,81 @@ func activeBan(bans []Ban, now time.Time) *Ban {
return newest
}
// Grounds are why a request makes a ban: its cause, CauseLimit or
// CauseAttack, and its notes.
type Grounds struct {
Cause string
Notes Notes
// BanForLimit bans netblock at now for a broken limit, with notes, and
// returns the ban, and true. A first ban lasts LimitBanDuration. A ban
// made within LimitBanRepeatWindow after the netblock's ban that ended
// last, other than one for a clear sign of attack or a lifted one, lasts
// repeatFactor times as long as that one. A ban that would be longer
// than LimitBanPermanentAbove is permanent instead. If a ban on netblock is still
// active, as when two of its requests break a limit at once, that ban is
// returned with false, and no other is made. The ledger fills in the
// notes' Refused and EarlierBans itself, and gives the ban the reason
// "<Kind> per <Window> over the limit of <Limit>", from the notes, such
// as "requests per minute over the limit of 1000".
func (l *Ledger) BanForLimit(
netblock netip.Prefix, now time.Time, notes Notes,
) (Ban, bool) {
return l.ban(
netblock,
now,
CauseLimit,
limitReason(notes),
notes,
true,
)
}
// Ban bans netblock at now on grounds, and returns the ban, and true. If
// a ban on netblock is still active, as when two of its requests break a
// limit at once, that ban is returned with false, and no other is made.
// The ledger fills in the notes' Refused and EarlierBans itself.
//
// A ban for a broken limit:
// - lasts LimitBanDuration the first time;
// - lasts repeatFactor times as long as the netblock's ban that ended
// last, when made within LimitBanRepeatWindow after it, unless that
// ban was for a clear sign of attack or was lifted;
// - is permanent instead when it would be longer than
// LimitBanPermanentAbove;
// - has the reason "<Kind> per <Window> over the limit of <Limit>", from
// the notes, such as "requests per minute over the limit of 1000".
//
// A ban for a clear sign of attack:
// - lasts AttackBanDuration the first time;
// - is permanent once the netblock has had one that was not lifted;
// - has the reason "matched the rule <RuleID>", or "asked for the trap
// path <TrapPath>".
func (l *Ledger) Ban(
netblock netip.Prefix, now time.Time, grounds Grounds,
// WouldBanForLimit returns what BanForLimit would, without making the ban:
// what observe mode would have done.
func (l *Ledger) WouldBanForLimit(
netblock netip.Prefix, now time.Time, notes Notes,
) (Ban, bool) {
return l.ban(netblock, now, grounds, true)
return l.ban(
netblock,
now,
CauseLimit,
limitReason(notes),
notes,
false,
)
}
// WouldBan returns what Ban would, without making the ban: what observe
// mode would have done.
func (l *Ledger) WouldBan(
netblock netip.Prefix, now time.Time, grounds Grounds,
// BanForAttack bans netblock at now for a clear sign of attack, with
// notes, and returns the ban, and whether it made it, as BanForLimit
// does. A first ban lasts AttackBanDuration; once the netblock has had
// one that was not lifted, the next is permanent. Its reason is "matched
// the rule <RuleID>", or "asked for the trap path <TrapPath>".
func (l *Ledger) BanForAttack(
netblock netip.Prefix, now time.Time, notes Notes,
) (Ban, bool) {
return l.ban(netblock, now, grounds, false)
return l.ban(
netblock,
now,
CauseAttack,
attackReason(notes),
notes,
true,
)
}
// WouldBanForAttack returns what BanForAttack would, without making the
// ban: what observe mode would have done.
func (l *Ledger) WouldBanForAttack(
netblock netip.Prefix, now time.Time, notes Notes,
) (Ban, bool) {
return l.ban(
netblock,
now,
CauseAttack,
attackReason(notes),
notes,
false,
)
}
// WouldBePermanent reports whether a ban on netblock for cause, CauseLimit
// or CauseAttack, made at now would be permanent, as Ban would make it. It
// works out nothing else of the ban.
// or CauseAttack, made at now would be permanent, as BanForLimit or
// BanForAttack would make it. It works out nothing else of the ban.
func (l *Ledger) WouldBePermanent(
netblock netip.Prefix, now time.Time, cause string,
) bool {
@@ -380,9 +411,10 @@ func attackReason(notes Notes) string {
// BanForAdmin bans netblock at now for an admin, with reason, until
// expires, or for good when expires is zero, and returns the ban, whose
// cause is CauseAdmin. Unlike Ban, it makes the ban even while another
// on netblock is active, since the admin asked for this one. The ledger
// fills in the notes' EarlierBans, and counts the ban among those made.
// cause is CauseAdmin. Unlike BanForLimit and BanForAttack, it makes the
// ban even while another on netblock is active, since the admin asked
// for this one. The ledger fills in the notes' EarlierBans, and counts
// the ban among those made.
func (l *Ledger) BanForAdmin(
netblock netip.Prefix, now, expires time.Time, reason string,
) Ban {
@@ -727,13 +759,15 @@ func (l *Ledger) forgetEndedAbuseIPDBBans(now time.Time) bool {
return forgot
}
// ban bans netblock at now on grounds, as Ban describes, and returns the
// ban, and whether it made it. Unless keep is true, the ban is not made,
// only returned: it is the ban that would have been made.
// ban bans netblock at now for cause, CauseLimit or CauseAttack, with
// reason and notes, as BanForLimit and BanForAttack describe, and returns
// the ban, and whether it made it. Unless keep is true, the ban is not
// made, only returned: it is the ban that would have been made.
func (l *Ledger) ban(
netblock netip.Prefix,
now time.Time,
grounds Grounds,
cause, reason string,
notes Notes,
keep bool,
) (Ban, bool) {
l.mu.Lock()
@@ -742,8 +776,6 @@ func (l *Ledger) ban(
// held are the netblock's bans, none of them active.
var held []Ban
notes := grounds.Notes
bans, found := l.netblocks.Get(netblock)
if found {
active := activeBan(*bans, now)
@@ -759,19 +791,15 @@ func (l *Ledger) ban(
ban := Ban{
Netblock: netblock,
Start: now,
Cause: grounds.Cause,
Cause: cause,
Reason: reason,
Notes: notes,
}
switch grounds.Cause {
case CauseLimit:
ban.Reason = limitReason(notes)
ban.Expires = l.limitExpiry(held, now)
case CauseAttack:
ban.Reason = attackReason(notes)
if cause == CauseAttack {
ban.Expires = l.attackExpiry(held, now)
default:
panic("bans: a request's ban has the cause " + grounds.Cause)
} else {
ban.Expires = l.limitExpiry(held, now)
}
if !keep {
@@ -779,7 +807,7 @@ func (l *Ledger) ban(
}
l.add(ban)
l.made[grounds.Cause]++
l.made[cause]++
l.markChanged()
return ban, true
+51 -121
View File
@@ -22,11 +22,7 @@ func TestRepeatsTripleUntilPermanent(t *testing.T) {
// Each ban is followed by another as soon as it ends: 1, 3, 9, 27 and
// 81 hours.
for i, hours := range []int{1, 3, 9, 27, 81} {
ban, _ := ledger.Ban(
netblock,
now,
bans.Grounds{Cause: bans.CauseLimit},
)
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
length := time.Duration(hours) * time.Hour
if !ban.Expires.Equal(now.Add(length)) ||
@@ -46,7 +42,7 @@ func TestRepeatsTripleUntilPermanent(t *testing.T) {
// The sixth would last 243 hours, more than seven days: it is
// permanent, and never ends.
ban, _ := ledger.Ban(netblock, now, bans.Grounds{Cause: bans.CauseLimit})
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
if !ban.Permanent() {
t.Fatalf("sixth ban ends at %s, want a permanent one", ban.Expires)
}
@@ -75,15 +71,15 @@ func TestRepeatWindowRunsOut(t *testing.T) {
ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32")
first, _ := ledger.Ban(
first, _ := ledger.BanForLimit(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
bans.Notes{},
)
second, _ := ledger.Ban(
second, _ := ledger.BanForLimit(
netblock,
first.Expires.Add(tc.gap),
bans.Grounds{Cause: bans.CauseLimit},
bans.Notes{},
)
if second.Expires.Sub(second.Start) != tc.want ||
@@ -108,10 +104,10 @@ func TestFirstBanLongerThanLimitBanPermanentAboveIsPermanent(t *testing.T) {
params.LimitBanDuration = params.LimitBanPermanentAbove + time.Hour
ledger := bans.New(params)
ban, _ := ledger.Ban(
ban, _ := ledger.BanForLimit(
netip.MustParsePrefix("203.0.113.9/32"),
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
bans.Notes{},
)
if !ban.Permanent() {
t.Errorf("first ban ends at %s, want a permanent one", ban.Expires)
@@ -131,11 +127,7 @@ func TestLimitBanPermanentAboveSetFarOffDoesNotOverflow(t *testing.T) {
now := midnight()
for i := range 14 {
ban, _ := ledger.Ban(
netblock,
now,
bans.Grounds{Cause: bans.CauseLimit},
)
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
if !ban.Expires.After(ban.Start) {
t.Fatalf(
"ban %d starts at %s and ends at %s",
@@ -148,7 +140,7 @@ func TestLimitBanPermanentAboveSetFarOffDoesNotOverflow(t *testing.T) {
now = ban.Expires
}
ban, _ := ledger.Ban(netblock, now, bans.Grounds{Cause: bans.CauseLimit})
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
if !ban.Permanent() {
t.Errorf("15th ban ends at %s, want a permanent one", ban.Expires)
}
@@ -160,19 +152,15 @@ func TestBrokenLimitDuringABanMakesNoOther(t *testing.T) {
ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32")
first, made := ledger.Ban(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
first, made := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
if !made {
t.Error("the first ban was not made")
}
again, made := ledger.Ban(
again, made := ledger.BanForLimit(
netblock,
midnight().Add(time.Minute),
bans.Grounds{Cause: bans.CauseLimit},
bans.Notes{},
)
if made || !reflect.DeepEqual(again, first) ||
@@ -187,10 +175,10 @@ func TestBrokenLimitDuringABanMakesNoOther(t *testing.T) {
)
}
again, made = ledger.Ban(
again, made = ledger.BanForAttack(
netblock,
midnight().Add(time.Minute),
bans.Grounds{Cause: bans.CauseAttack},
bans.Notes{},
)
if made || !reflect.DeepEqual(again, first) {
t.Errorf(
@@ -207,10 +195,10 @@ func TestCheckRefusesWhileTheBanLastsAndCountsTheRefusals(t *testing.T) {
ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32")
ban, _ := ledger.Ban(
ban, _ := ledger.BanForLimit(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit, Notes: bans.Notes{Requests: 5}},
bans.Notes{Requests: 5},
)
for range 3 {
@@ -249,10 +237,10 @@ func TestFindCountsNothing(t *testing.T) {
ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32")
ban, _ := ledger.Ban(
ban, _ := ledger.BanForLimit(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit, Notes: bans.Notes{Requests: 5}},
bans.Notes{Requests: 5},
)
got, banned, _ := ledger.Find(
@@ -299,24 +287,24 @@ func TestMaxBansDropsTheEarliestBanOfTheNetblockSeenLongestAgo(
d := netip.MustParsePrefix("2001:db8::/64")
now := midnight()
first, _ := ledger.Ban(a, now, bans.Grounds{Cause: bans.CauseLimit})
ledger.Ban(b, now, bans.Grounds{Cause: bans.CauseLimit})
ledger.Ban(c, now, bans.Grounds{Cause: bans.CauseLimit})
first, _ := ledger.BanForLimit(a, now, bans.Notes{})
ledger.BanForLimit(b, now, bans.Notes{})
ledger.BanForLimit(c, now, bans.Notes{})
// A request from a makes b the netblock seen longest ago, and its ban
// goes to make room for d's.
ledger.Check(a.Addr(), now)
ledger.Ban(d, now, bans.Grounds{Cause: bans.CauseLimit})
ledger.BanForLimit(d, now, bans.Notes{})
wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 0, c: 1, d: 1})
// a is banned again once its ban has ended; c, seen longest ago, goes.
ledger.Ban(a, first.Expires, bans.Grounds{Cause: bans.CauseLimit})
ledger.BanForLimit(a, first.Expires, bans.Notes{})
wantBans(t, ledger, map[netip.Prefix]int{a: 2, c: 0, d: 1})
// With d seen since, a is seen longest ago, and its earlier ban goes
// first.
ledger.Check(d.Addr(), first.Expires)
ledger.Ban(b, first.Expires, bans.Grounds{Cause: bans.CauseLimit})
ledger.BanForLimit(b, first.Expires, bans.Notes{})
wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 1, d: 1})
if !ledger.Bans(a)[0].Start.Equal(first.Expires) {
@@ -337,16 +325,8 @@ func TestFullLedgerDropsTheEarlierBanOfTheNetblockBannedAgain(t *testing.T) {
ledger := bans.New(params)
netblock := netip.MustParsePrefix("203.0.113.9/32")
first, _ := ledger.Ban(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
second, _ := ledger.Ban(
netblock,
first.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
second, _ := ledger.BanForLimit(netblock, first.Expires, bans.Notes{})
held := ledger.Bans(netblock)
if len(held) != 1 || !reflect.DeepEqual(held[0], second) ||
@@ -363,11 +343,7 @@ func TestRequestDuringAnAttackBanMakesItPermanent(t *testing.T) {
netblock := netip.MustParsePrefix("203.0.113.9/32")
notes := bans.Notes{RuleID: "env-file", Target: "path"}
ban, _ := ledger.Ban(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseAttack, Notes: notes},
)
ban, _ := ledger.BanForAttack(netblock, midnight(), notes)
if !ban.Expires.Equal(midnight().Add(7*day)) ||
ban.Cause != bans.CauseAttack ||
ban.Notes.RuleID != "env-file" ||
@@ -442,16 +418,8 @@ func TestAttackAfterAnAttackBanHasEndedBansPermanently(t *testing.T) {
netblock := netip.MustParsePrefix("203.0.113.9/32")
// A ban for a broken limit before does not count.
first, _ := ledger.Ban(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
second, _ := ledger.Ban(
netblock,
first.Expires,
bans.Grounds{Cause: bans.CauseAttack},
)
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
second, _ := ledger.BanForAttack(netblock, first.Expires, bans.Notes{})
if second.Expires.Sub(second.Start) != 7*day {
t.Fatalf("the first ban for an attack lasts %s, want 7 days",
@@ -467,10 +435,10 @@ func TestAttackAfterAnAttackBanHasEndedBansPermanently(t *testing.T) {
// Its notes show the earlier ban for an attack that makes it permanent,
// beside the one for a limit.
third, _ := ledger.Ban(
third, _ := ledger.BanForAttack(
netblock,
second.Expires.Add(30*day),
bans.Grounds{Cause: bans.CauseAttack},
bans.Notes{},
)
if !third.Permanent() ||
third.Notes.EarlierBans != (bans.EarlierBans{Limit: 1, Attack: 1}) {
@@ -484,18 +452,14 @@ func TestWouldBanGivesTheBanWithoutMakingIt(t *testing.T) {
ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32")
first, _ := ledger.Ban(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
wantChanged(t, ledger, true)
// While the first ban lasts, none would be made.
during, would := ledger.WouldBan(
during, would := ledger.WouldBanForAttack(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseAttack},
bans.Notes{},
)
if would || !reflect.DeepEqual(during, first) {
t.Errorf(
@@ -508,19 +472,13 @@ func TestWouldBanGivesTheBanWithoutMakingIt(t *testing.T) {
// As it ends, a clear sign of attack would ban for seven days, and a
// limit broken again for three hours, but neither is made.
limitGrounds := bans.Grounds{
Cause: bans.CauseLimit,
Notes: bans.Notes{Kind: "requests", Limit: 1, Window: "minute"},
}
attack, wouldAttack := ledger.WouldBan(netblock, first.Expires,
bans.Grounds{
Cause: bans.CauseAttack,
Notes: bans.Notes{RuleID: "git-dir"},
})
limit, wouldLimit := ledger.WouldBan(
limitNotes := bans.Notes{Kind: "requests", Limit: 1, Window: "minute"}
attack, wouldAttack := ledger.WouldBanForAttack(netblock, first.Expires,
bans.Notes{RuleID: "git-dir"})
limit, wouldLimit := ledger.WouldBanForLimit(
netblock,
first.Expires,
limitGrounds,
limitNotes,
)
if !wouldAttack || !attack.Expires.Equal(first.Expires.Add(7*day)) ||
@@ -547,7 +505,7 @@ func TestWouldBanGivesTheBanWithoutMakingIt(t *testing.T) {
wantChanged(t, ledger, false)
// The ban made is the one that would have been.
made, _ := ledger.Ban(netblock, first.Expires, limitGrounds)
made, _ := ledger.BanForLimit(netblock, first.Expires, limitNotes)
if !reflect.DeepEqual(made, limit) {
t.Errorf("the ban made is %+v, want %+v", made, limit)
}
@@ -568,11 +526,7 @@ func TestWouldBePermanentAnswersAsTheBanWouldBeMade(t *testing.T) {
t.Fatalf("ban %d for a limit would be permanent", i+1)
}
ban, _ := ledger.Ban(
netblock,
now,
bans.Grounds{Cause: bans.CauseLimit},
)
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
now = ban.Expires
}
@@ -585,11 +539,7 @@ func TestWouldBePermanentAnswersAsTheBanWouldBeMade(t *testing.T) {
}
// Once a first ban for an attack has ended, the next would be permanent.
attack, _ := ledger.Ban(
netblock,
now,
bans.Grounds{Cause: bans.CauseAttack},
)
attack, _ := ledger.BanForAttack(netblock, now, bans.Notes{})
if !ledger.WouldBePermanent(netblock, attack.Expires, bans.CauseAttack) {
t.Error("a second ban for an attack would not be permanent")
}
@@ -603,16 +553,8 @@ func TestAttackBanDoesNotLengthenTheNextBanForALimit(t *testing.T) {
// Three times the seven days would be permanent; a limit broken as the
// ban for an attack ends bans for an hour, as a first broken limit does.
attack, _ := ledger.Ban(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseAttack},
)
limit, _ := ledger.Ban(
netblock,
attack.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
attack, _ := ledger.BanForAttack(netblock, midnight(), bans.Notes{})
limit, _ := ledger.BanForLimit(netblock, attack.Expires, bans.Notes{})
if limit.Expires.Sub(limit.Start) != time.Hour ||
limit.Cause != bans.CauseLimit {
@@ -636,13 +578,10 @@ func TestRequestTextsAreCutTo256Bytes(t *testing.T) {
Time: midnight(), Method: long, Host: long, Path: long, Status: 403, UserAgent: long,
}
ban, _ := ledger.Ban(
ban, _ := ledger.BanForLimit(
netblock,
midnight(),
bans.Grounds{
Cause: bans.CauseLimit,
Notes: bans.Notes{Request: request},
},
bans.Notes{Request: request},
)
cut := long[:256]
@@ -668,20 +607,11 @@ func TestLookupFillsTheNotesOfTheNetblocksBansWithoutOne(t *testing.T) {
// A ban made with the client's lookup, one made before it came, after
// the first ended, and one on another netblock.
ledger.Ban(netblock, midnight(), bans.Grounds{
Cause: bans.CauseLimit,
Notes: bans.Notes{
ASN: "AS64497",
ASName: "Other Net",
Country: "FR",
},
ledger.BanForLimit(netblock, midnight(), bans.Notes{
ASN: "AS64497", ASName: "Other Net", Country: "FR",
})
ledger.Ban(
netblock,
midnight().Add(time.Hour),
bans.Grounds{Cause: bans.CauseLimit},
)
ledger.Ban(other, midnight(), bans.Grounds{Cause: bans.CauseLimit})
ledger.BanForLimit(netblock, midnight().Add(time.Hour), bans.Notes{})
ledger.BanForLimit(other, midnight(), bans.Notes{})
ledger.AddLookup(netblock, "AS64496", "Example Net", "DE")
+21 -36
View File
@@ -19,29 +19,25 @@ func TestChangedAfterABanIsMade(t *testing.T) {
wantChanged(t, ledger, false)
ledger.Ban(netblock, midnight(), bans.Grounds{Cause: bans.CauseLimit})
ledger.BanForLimit(netblock, midnight(), bans.Notes{})
wantChanged(t, ledger, true)
// A limit broken during the ban makes no other, and a refusal changes
// only the counts in the notes, which wait for the interval's write.
ledger.Ban(
netblock,
midnight().Add(time.Minute),
bans.Grounds{Cause: bans.CauseLimit},
)
ledger.BanForLimit(netblock, midnight().Add(time.Minute), bans.Notes{})
ledger.Check(netblock.Addr(), midnight().Add(time.Minute))
wantChanged(t, ledger, false)
// Two bans before the value is read leave one.
ledger.Ban(
ledger.BanForLimit(
netip.MustParsePrefix("203.0.113.10/32"),
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
bans.Notes{},
)
ledger.Ban(
ledger.BanForLimit(
netip.MustParsePrefix("203.0.113.11/32"),
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
bans.Notes{},
)
wantChanged(t, ledger, true)
wantChanged(t, ledger, false)
@@ -55,14 +51,10 @@ func TestSnapshotListsEveryBanByNetblock(t *testing.T) {
high := netip.MustParsePrefix("203.0.113.10/32")
low := netip.MustParsePrefix("203.0.113.9/32")
first, _ := ledger.Ban(
v6,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
ledger.Ban(high, midnight(), bans.Grounds{Cause: bans.CauseLimit})
ledger.Ban(low, midnight(), bans.Grounds{Cause: bans.CauseLimit})
ledger.Ban(v6, first.Expires, bans.Grounds{Cause: bans.CauseLimit})
first, _ := ledger.BanForLimit(v6, midnight(), bans.Notes{})
ledger.BanForLimit(high, midnight(), bans.Notes{})
ledger.BanForLimit(low, midnight(), bans.Notes{})
ledger.BanForLimit(v6, first.Expires, bans.Notes{})
snapshot := ledger.Snapshot()
@@ -88,10 +80,7 @@ func TestLoadedBansCarryOn(t *testing.T) {
before := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32")
ban, _ := before.Ban(netblock, midnight(), bans.Grounds{
Cause: bans.CauseLimit,
Notes: bans.Notes{Limit: 1},
})
ban, _ := before.BanForLimit(netblock, midnight(), bans.Notes{Limit: 1})
// Loaded into a new ledger, as across a restart, the ban still refuses
// while it lasts, and once it has ended a broken limit bans for three
@@ -107,11 +96,7 @@ func TestLoadedBansCarryOn(t *testing.T) {
t.Error("the loaded ban does not refuse")
}
again, _ := after.Ban(
netblock,
ban.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
again, _ := after.BanForLimit(netblock, ban.Expires, bans.Notes{})
if again.Expires.Sub(again.Start) != 3*time.Hour ||
again.Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) {
t.Errorf(
@@ -139,10 +124,10 @@ func TestLoadedBanRefusesEveryClientInItsNetblock(t *testing.T) {
Start: midnight(),
},
})
ledger.Ban(
ledger.BanForLimit(
netip.MustParsePrefix("198.51.100.7/32"),
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
bans.Notes{},
)
for client, want := range map[string]bool{
@@ -215,7 +200,7 @@ func TestPermanentBanStartedBeforeAnEndedOneRefuses(t *testing.T) {
}
// A limit broken now makes no shorter ban over the permanent one.
ban, _ = ledger.Ban(netblock, now, bans.Grounds{Cause: bans.CauseLimit})
ban, _ = ledger.BanForLimit(netblock, now, bans.Notes{})
if !ban.Permanent() || len(ledger.Bans(netblock)) != 2 {
t.Errorf(
"a broken limit returned %+v and left the netblock %d bans, "+
@@ -251,10 +236,10 @@ func TestNextBanWorkedOutFromTheBanThatEndedLast(t *testing.T) {
// Once both have ended, a limit broken within the repeat window bans
// for three times the 9 hours, and the notes count the two bans
// before the 9-hour one and it, for a limit, and the admin's.
ban, _ := ledger.Ban(
ban, _ := ledger.BanForLimit(
netblock,
nineHours.Expires.Add(time.Hour),
bans.Grounds{Cause: bans.CauseLimit},
bans.Notes{},
)
if ban.Expires.Sub(ban.Start) != 27*time.Hour ||
ban.Notes.EarlierBans != (bans.EarlierBans{Limit: 3, Admin: 1}) {
@@ -328,15 +313,15 @@ func TestLoadReplacesTheBansHeld(t *testing.T) {
}
// The ledger holds one ban, so it makes two more without dropping any.
first, _ := ledger.Ban(
first, _ := ledger.BanForLimit(
netip.MustParsePrefix("198.51.100.7/32"),
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
bans.Notes{},
)
second, _ := ledger.Ban(
second, _ := ledger.BanForLimit(
netip.MustParsePrefix("198.51.100.8/32"),
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
bans.Notes{},
)
want := []bans.Ban{first, second, kept}
+25 -4
View File
@@ -19,8 +19,6 @@ import (
"time"
"github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/reputation"
)
// off is the value that switches a timeout, a size limit or a rate limit
@@ -42,7 +40,9 @@ var (
errNotPositive = errors.New("must be more than zero, or off")
errNotMilliseconds = errors.New(
"is not a whole number of milliseconds above zero, such as 2000")
errEmptyItem = errors.New("has an empty item in its list")
errEmptyItem = errors.New("has an empty item in its list")
errNotNetblock = errors.New(
"is not a netblock such as 10.0.0.0/8, or an address")
errNotListenAddr = errors.New(
"is not an address to listen on, such as :8080")
errNotCommand = errors.New("is not rproxy or waf")
@@ -253,7 +253,7 @@ func parseNetblocks(value string) ([]netip.Prefix, error) {
netblocks := make([]netip.Prefix, 0, len(items))
for _, item := range items {
netblock, err := reputation.ParseNetblock(item)
netblock, err := ParseNetblock(item)
if err != nil {
return nil, err
}
@@ -264,6 +264,27 @@ func parseNetblocks(value string) ([]netip.Prefix, error) {
return netblocks, nil
}
// ParseNetblock reads a netblock in CIDR form, such as 10.0.0.0/8. A bare
// address is a netblock of that address alone, a /32 or a /128. A
// blocklist's lines are read with it too.
func ParseNetblock(value string) (netip.Prefix, error) {
if strings.Contains(value, "/") {
netblock, err := netip.ParsePrefix(value)
if err != nil {
return netip.Prefix{}, fmt.Errorf("%q %w", value, errNotNetblock)
}
return netblock.Masked(), nil
}
addr, err := netip.ParseAddr(value)
if err != nil || addr.Zone() != "" {
return netip.Prefix{}, fmt.Errorf("%q %w", value, errNotNetblock)
}
return netip.PrefixFrom(addr, addr.BitLen()), nil
}
// parseListenAddr checks an address to listen on: an optional host and a
// port number.
func parseListenAddr(value string) (string, error) {
+152 -169
View File
@@ -20,26 +20,11 @@ import (
"unicode/utf8"
"github.com/spf13/viper"
"golang.org/x/net/http/httpguts"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/anomaly"
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
"sneak.berlin/go/smallwebwaf/internal/remotelog"
"sneak.berlin/go/smallwebwaf/internal/reputation"
)
// The names of the settings that waf writes into log lines and ban notes,
// as the setting that refused a request or lowered a client's limits.
const (
ExclusivelyAllowedCountriesSetting = "SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES"
TagLimitPercentSetting = "SWW_WAF_TAG_LIMIT_PERCENT"
ASNBytesLimitPercentSetting = "SWW_WAF_ASN_BYTES_LIMIT_PERCENT"
CountryBytesLimitPercentSetting = "SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT"
UnknownCountryLimitPercentSetting = "SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT"
ASNLimitPercentURLSetting = "SWW_WAF_ASN_LIMIT_PERCENT_URL"
BlocklistActionSetting = "SWW_WAF_BLOCKLIST_ACTION"
DNSBLAbuseIPDBActionSetting = "SWW_WAF_DNSBL_ABUSEIPDB_ACTION"
)
// WAF is the settings of `sww waf`. A time or size limit of zero is off.
@@ -90,7 +75,7 @@ type WAF struct {
BytesLimitPerMinute int64
BytesLimitPerHour int64
BytesLimitPerDay int64
BytesCounted BytesCounted
BytesCounted string
// LookupSource is where each client's AS number and country are looked
// up (SWW_WAF_LOOKUP_SOURCE): geojs, file, or off for nowhere.
// LookupDBPath is the lookup database, the IPinfo Lite file looked up in
@@ -101,7 +86,7 @@ type WAF struct {
// AddLookupHeaders is true when the app is passed the client's AS number
// and country in headers (SWW_WAF_ADD_LOOKUP_HEADERS), by default unless
// LookupSource is off.
LookupSource LookupSource
LookupSource string
LookupDBPath string
ASNAndCountryLookupTimeout time.Duration
AddLookupHeaders bool
@@ -115,7 +100,7 @@ type WAF struct {
// with a tag that percentage of every rate limit and byte limit, from 1
// to 99 (SWW_WAF_TAG_LIMIT_PERCENT). All hold tags as requests carry them: in
// lower case, a country code in capitals.
TagLists []reputation.TagList
TagLists []TagList
TagDeny []string
TagLimitPercent map[string]int64
// RouteWatchURL is where the routewatch instance is that adds the asn=
@@ -146,7 +131,7 @@ type WAF struct {
// BlocklistLimitPercent is the percentage of every limit it gets.
BlocklistURLs []string
BlocklistRefresh time.Duration
BlocklistAction ReputationAction
BlocklistAction string
BlocklistLimitPercent int64
// DNSBLZones are the DNSBL zones clients are asked about
// (SWW_WAF_DNSBL_ZONES), through DNSBLResolver (SWW_WAF_DNSBL_RESOLVER), or
@@ -169,7 +154,7 @@ type WAF struct {
AbuseIPDBKey string
AbuseIPDBMinScore int64
AbuseIPDBDailyBudget int
DNSBLAbuseIPDBAction ReputationAction
DNSBLAbuseIPDBAction string
DNSBLAbuseIPDBLimitPercent int64
IPReputationCacheTTL time.Duration
DNSBLAbuseIPDBTimeout time.Duration
@@ -252,7 +237,7 @@ type WAF struct {
// reads none.
// CoreRuleSetExclusionsFile is the path of the file of its exclusions
// (SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE), "" while there is none.
CoreRuleSetMode CoreRuleSetMode
CoreRuleSetMode string
CoreRuleSetParanoiaLevel int
CoreRuleSetScoreThreshold int
CoreRuleSetDisabledRules []int
@@ -331,66 +316,20 @@ type WAF struct {
settings []slog.Attr
}
// BytesCounted is which body bytes count toward the byte limits and the
// byte anomaly thresholds, a value of SWW_WAF_BYTES_COUNTED.
type BytesCounted string
// The values of SWW_WAF_BYTES_COUNTED.
const (
// BytesCountedResponse counts the response's body bytes.
BytesCountedResponse BytesCounted = "response"
// BytesCountedRequest counts the request's body bytes.
BytesCountedRequest BytesCounted = "request"
// BytesCountedBoth counts the body bytes of both.
BytesCountedBoth BytesCounted = "both"
)
// LookupSource is where each client's AS number and country are looked up,
// a value of SWW_WAF_LOOKUP_SOURCE.
type LookupSource string
// The values of SWW_WAF_LOOKUP_SOURCE.
const (
// LookupSourceGeoJS asks GeoJS.
LookupSourceGeoJS LookupSource = "geojs"
// LookupSourceFile looks clients up in the lookup database, the file
// SWW_WAF_LOOKUP_DB_PATH names.
LookupSourceFile LookupSource = "file"
// LookupSourceOff looks no client up.
LookupSourceOff LookupSource = "off"
)
// ReputationAction is what is done with a client a reputation source
// lists, a value of SWW_WAF_BLOCKLIST_ACTION and of
// SWW_WAF_DNSBL_ABUSEIPDB_ACTION.
type ReputationAction string
// The values of SWW_WAF_BLOCKLIST_ACTION and SWW_WAF_DNSBL_ABUSEIPDB_ACTION.
const (
// ReputationActionDeny refuses the client's requests.
ReputationActionDeny ReputationAction = "deny"
// ReputationActionLimit gives the client a percentage of every limit,
// written limit:<percent> in the setting.
ReputationActionLimit ReputationAction = "limit"
// ReputationActionLog only notes the hit in the log line and raises an
// alert.
ReputationActionLog ReputationAction = "log"
)
// CoreRuleSetMode is what the Core Rule Set does, a value of
// SWW_WAF_CORE_RULE_SET_MODE.
type CoreRuleSetMode string
// The values of SWW_WAF_CORE_RULE_SET_MODE.
const (
// CoreRuleSetModeOff runs no request through the Core Rule Set.
CoreRuleSetModeOff CoreRuleSetMode = "off"
CoreRuleSetModeOff = off
// CoreRuleSetModeDetect logs and alerts a match, and refuses nothing.
CoreRuleSetModeDetect CoreRuleSetMode = "detect"
CoreRuleSetModeDetect = "detect"
// CoreRuleSetModeBlock refuses a match with 403.
CoreRuleSetModeBlock CoreRuleSetMode = "block"
CoreRuleSetModeBlock = "block"
)
// fileSource is the SWW_WAF_LOOKUP_SOURCE that looks clients up in the
// lookup database, the file SWW_WAF_LOOKUP_DB_PATH names.
const fileSource = "file"
const (
day = 24 * time.Hour
ipv4Bits = 32
@@ -512,8 +451,12 @@ var (
"is not a whole number above zero, such as 60, or off",
)
errNotUTF8 = errors.New("is not valid UTF-8")
errNotASN = errors.New("is not an AS number such as AS64496")
errNotPercentItem = errors.New(
"is not a code, : and a percentage, such as AS64496:50 or cn:25")
errNotPercent = errors.New(
"is not a percentage, a whole number from 0 to 100",
)
errNotAbuseIPDBThreshold = errors.New(
"is not a whole number from 25 to 100, the scores AbuseIPDB's " +
"blacklist can be cut at",
@@ -615,13 +558,10 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
),
BytesLimitPerHour: env.size("SWW_WAF_BYTES_LIMIT_PER_HOUR", "50G"),
BytesLimitPerDay: env.size("SWW_WAF_BYTES_LIMIT_PER_DAY", "100G"),
BytesCounted: env.bytesCounted(
"SWW_WAF_BYTES_COUNTED",
BytesCountedBoth,
),
BytesCounted: env.bytesCounted("SWW_WAF_BYTES_COUNTED", "both"),
LookupSource: env.lookupSource(
"SWW_WAF_LOOKUP_SOURCE",
LookupSourceGeoJS,
"geojs",
),
LookupDBPath: env.value("SWW_WAF_LOOKUP_DB_PATH", ""),
ASNAndCountryLookupTimeout: env.milliseconds(
@@ -629,24 +569,24 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
"800",
),
ExclusivelyAllowedCountries: env.countries(
ExclusivelyAllowedCountriesSetting, ""),
"SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES", ""),
TagLists: env.tagLists("SWW_WAF_TAG_LISTS"),
TagDeny: env.tags("SWW_WAF_TAG_DENY"),
TagLimitPercent: env.tagPercents(TagLimitPercentSetting),
TagLimitPercent: env.tagPercents("SWW_WAF_TAG_LIMIT_PERCENT"),
RouteWatchURL: env.routeWatchURL("SWW_WAF_ROUTEWATCH_URL"),
ASNBytesLimitPercent: env.percents(
ASNBytesLimitPercentSetting,
reputation.ParseASN,
"SWW_WAF_ASN_BYTES_LIMIT_PERCENT",
ParseASN,
),
CountryBytesLimitPercent: env.percents(
CountryBytesLimitPercentSetting,
"SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT",
parseCountry,
),
UnknownCountryLimitPercent: env.percent(
UnknownCountryLimitPercentSetting,
"SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT",
"100",
),
ASNLimitPercentURL: env.listURL(ASNLimitPercentURLSetting),
ASNLimitPercentURL: env.listURL("SWW_WAF_ASN_LIMIT_PERCENT_URL"),
BlocklistURLs: env.listURLs("SWW_WAF_BLOCKLIST_URLS"),
BlocklistRefresh: env.refresh("SWW_WAF_BLOCKLIST_REFRESH", "24h"),
DNSBLZones: env.zones("SWW_WAF_DNSBL_ZONES"),
@@ -818,11 +758,11 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
cfg.InstanceName, cfg.LogRemoteURL != nil)
// The app gets the headers by default, unless no client is looked up.
cfg.AddLookupHeaders = env.boolean("SWW_WAF_ADD_LOOKUP_HEADERS",
strconv.FormatBool(cfg.LookupSource != LookupSourceOff))
strconv.FormatBool(cfg.LookupSource != off))
cfg.BlocklistAction, cfg.BlocklistLimitPercent = env.action(
BlocklistActionSetting, string(ReputationActionDeny))
"SWW_WAF_BLOCKLIST_ACTION", "deny")
cfg.DNSBLAbuseIPDBAction, cfg.DNSBLAbuseIPDBLimitPercent = env.action(
DNSBLAbuseIPDBActionSetting, "limit:25")
"SWW_WAF_DNSBL_ABUSEIPDB_ACTION", "limit:25")
env.checkInstanceNameForNtfy(cfg.InstanceName, cfg.AlertNtfyURL != nil)
env.checkLookupDBPath(cfg)
@@ -918,12 +858,9 @@ func (e *environment) count(name, defaultValue string) int64 {
// bytesCounted reads the setting that is which body bytes count toward the
// byte limits and the byte anomaly thresholds: response, request or both.
func (e *environment) bytesCounted(
name string, defaultValue BytesCounted,
) BytesCounted {
value := BytesCounted(e.value(name, string(defaultValue)))
if value != BytesCountedResponse && value != BytesCountedRequest &&
value != BytesCountedBoth {
func (e *environment) bytesCounted(name, defaultValue string) string {
value := e.value(name, defaultValue)
if value != "response" && value != "request" && value != "both" {
e.check(name, fmt.Errorf("%q %w", value, errNotBytesCounted))
}
@@ -968,10 +905,8 @@ func (e *environment) trapPaths(name string) []string {
// coreRuleSetMode reads the setting that is what the Core Rule Set does: off,
// detect or block.
func (e *environment) coreRuleSetMode(
name string, defaultValue CoreRuleSetMode,
) CoreRuleSetMode {
mode := CoreRuleSetMode(e.value(name, string(defaultValue)))
func (e *environment) coreRuleSetMode(name, defaultValue string) string {
mode := e.value(name, defaultValue)
if mode != CoreRuleSetModeOff && mode != CoreRuleSetModeDetect &&
mode != CoreRuleSetModeBlock {
e.check(name, fmt.Errorf("%q %w", mode, errNotCoreRuleSetMode))
@@ -1022,7 +957,7 @@ func (e *environment) percents(
}
// tagLists reads SWW_WAF_TAG_LISTS, which is empty by default.
func (e *environment) tagLists(name string) []reputation.TagList {
func (e *environment) tagLists(name string) []TagList {
lists, err := parseTagLists(e.value(name, ""))
e.check(name, err)
@@ -1048,7 +983,7 @@ func (e *environment) tagPercents(name string) map[string]int64 {
// percent reads a setting that is a percentage, from 0 to 100.
func (e *environment) percent(name, defaultValue string) int64 {
percent, err := reputation.ParsePercent(e.value(name, defaultValue))
percent, err := ParsePercent(e.value(name, defaultValue))
e.check(name, err)
return percent
@@ -1121,29 +1056,25 @@ func (e *environment) refresh(name, defaultValue string) time.Duration {
// action reads a setting that is what is done with a client a blocklist
// or a DNSBL zone lists: deny, log, or limit:<percent>, which it returns
// as limit and the percentage.
func (e *environment) action(
name, defaultValue string,
) (ReputationAction, int64) {
value := ReputationAction(e.value(name, defaultValue))
if value == ReputationActionDeny || value == ReputationActionLog {
func (e *environment) action(name, defaultValue string) (string, int64) {
value := e.value(name, defaultValue)
if value == "deny" || value == "log" {
return value, 0
}
percentText, isLimit := strings.CutPrefix(
string(value), string(ReputationActionLimit)+":",
)
percentText, isLimit := strings.CutPrefix(value, "limit:")
percent, err := reputation.ParsePercent(percentText)
percent, err := ParsePercent(percentText)
if !isLimit || err != nil {
e.check(name, fmt.Errorf("%q %w", value, errNotAction))
}
return ReputationActionLimit, percent
return "limit", percent
}
// zones reads the setting that is the list of DNSBL zones. It is empty by
// default. The log shows each zone with its key masked, as
// reputation.MaskZoneKey masks it.
// default. The log shows each zone with its key masked, as MaskZoneKey
// masks it.
func (e *environment) zones(name string) []string {
value, _ := e.lookup(name)
zones, err := parseZones(value)
@@ -1151,7 +1082,7 @@ func (e *environment) zones(name string) []string {
logged := make([]string, len(zones))
for i, zone := range zones {
logged[i] = reputation.MaskZoneKey(zone)
logged[i] = MaskZoneKey(zone)
}
e.settings = append(
@@ -1189,12 +1120,9 @@ func (e *environment) routeWatchURL(name string) string {
// lookupSource reads the setting that is where clients are looked up:
// geojs, file, or off.
func (e *environment) lookupSource(
name string, defaultValue LookupSource,
) LookupSource {
source := LookupSource(e.value(name, string(defaultValue)))
if source != LookupSourceGeoJS && source != LookupSourceFile &&
source != LookupSourceOff {
func (e *environment) lookupSource(name, defaultValue string) string {
source := e.value(name, defaultValue)
if source != "geojs" && source != fileSource && source != off {
e.check(name, fmt.Errorf("%q %w", source, errNotLookupSource))
}
@@ -1206,12 +1134,12 @@ func (e *environment) lookupSource(
// one source at a time.
func (e *environment) checkLookupDBPath(cfg *WAF) {
switch {
case cfg.LookupSource == LookupSourceFile && cfg.LookupDBPath == "":
case cfg.LookupSource == fileSource && cfg.LookupDBPath == "":
e.check("SWW_WAF_LOOKUP_SOURCE", fmt.Errorf(
"is file while SWW_WAF_LOOKUP_DB_PATH is unset; %w",
errNeedsDBPath,
))
case cfg.LookupSource != LookupSourceFile && cfg.LookupDBPath != "":
case cfg.LookupSource != fileSource && cfg.LookupDBPath != "":
e.check("SWW_WAF_LOOKUP_DB_PATH", fmt.Errorf(
"is set while SWW_WAF_LOOKUP_SOURCE is %s; %w",
cfg.LookupSource,
@@ -1231,12 +1159,12 @@ func (e *environment) checkLookupDBPath(cfg *WAF) {
func (e *environment) checkCountriesAndLookups(cfg *WAF) {
for _, country := range cfg.ExclusivelyAllowedCountries {
if slices.Contains(cfg.TagDeny, CountryTagPrefix+country) {
e.check(ExclusivelyAllowedCountriesSetting,
e.check("SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES",
fmt.Errorf("%q %w", country, errOnBothLists))
}
}
if cfg.LookupSource != LookupSourceOff {
if cfg.LookupSource != off {
return
}
@@ -1250,15 +1178,15 @@ func (e *environment) checkCountriesAndLookups(cfg *WAF) {
name string
set bool
}{
{ExclusivelyAllowedCountriesSetting, len(cfg.ExclusivelyAllowedCountries) > 0},
{"SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES", len(cfg.ExclusivelyAllowedCountries) > 0},
{"SWW_WAF_ADD_LOOKUP_HEADERS", cfg.AddLookupHeaders},
{"SWW_WAF_TAG_DENY", slices.ContainsFunc(cfg.TagDeny, onlyLookupGives)},
{TagLimitPercentSetting, slices.ContainsFunc(
{"SWW_WAF_TAG_LIMIT_PERCENT", slices.ContainsFunc(
slices.Collect(maps.Keys(cfg.TagLimitPercent)), onlyLookupGives)},
{ASNBytesLimitPercentSetting, len(cfg.ASNBytesLimitPercent) > 0},
{CountryBytesLimitPercentSetting, len(cfg.CountryBytesLimitPercent) > 0},
{UnknownCountryLimitPercentSetting, cfg.UnknownCountryLimitPercent < 100},
{ASNLimitPercentURLSetting, cfg.ASNLimitPercentURL != ""},
{"SWW_WAF_ASN_BYTES_LIMIT_PERCENT", len(cfg.ASNBytesLimitPercent) > 0},
{"SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT", len(cfg.CountryBytesLimitPercent) > 0},
{"SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT", cfg.UnknownCountryLimitPercent < 100},
{"SWW_WAF_ASN_LIMIT_PERCENT_URL", cfg.ASNLimitPercentURL != ""},
{"SWW_WAF_ANOMALY_ASN_REQUESTS_PER_MINUTE", cfg.AnomalyASN.RequestsPerMinute > 0},
{"SWW_WAF_ANOMALY_ASN_REQUESTS_PER_HOUR", cfg.AnomalyASN.RequestsPerHour > 0},
{"SWW_WAF_ANOMALY_ASN_BYTES_PER_MINUTE", cfg.AnomalyASN.BytesPerMinute > 0},
@@ -1278,7 +1206,7 @@ func (e *environment) checkCountriesAndLookups(cfg *WAF) {
// blocklist too: the file at a URL is fetched as one list or the other.
func (e *environment) checkASNLimitPercentURL(cfg *WAF) {
if slices.Contains(cfg.BlocklistURLs, cfg.ASNLimitPercentURL) {
e.check(ASNLimitPercentURLSetting,
e.check("SWW_WAF_ASN_LIMIT_PERCENT_URL",
fmt.Errorf("%q %w", cfg.ASNLimitPercentURL, errInBlocklistURLs))
}
}
@@ -1305,18 +1233,14 @@ func (e *environment) checkTags(cfg *WAF) {
}
for tag := range cfg.TagLimitPercent {
e.check(TagLimitPercentSetting, checkTag(cfg, tag))
e.check("SWW_WAF_TAG_LIMIT_PERCENT", checkTag(cfg, tag))
}
}
// checkTag returns an error when tag would never apply, as checkTags says.
func checkTag(cfg *WAF, tag string) error {
if !requestCanGetTag(cfg, tag) {
return fmt.Errorf(
"%q %w",
reputation.MaskZoneKey(tag),
errNoRequestGetsTag,
)
return fmt.Errorf("%q %w", MaskZoneKey(tag), errNoRequestGetsTag)
}
return nil
@@ -1324,7 +1248,7 @@ func checkTag(cfg *WAF, tag string) error {
// requestCanGetTag reports whether anything cfg sets up can give a request tag.
func requestCanGetTag(cfg *WAF, tag string) bool {
isListName := func(list reputation.TagList) bool { return list.Name == tag }
isListName := func(list TagList) bool { return list.Name == tag }
if isLookupTag(tag) || slices.ContainsFunc(cfg.TagLists, isListName) {
return true
}
@@ -1348,7 +1272,7 @@ func requestCanGetTag(cfg *WAF, tag string) bool {
return isZoneTag &&
slices.ContainsFunc(cfg.DNSBLZones, func(each string) bool {
return strings.ToLower(reputation.MaskZoneKey(each)) == zone
return strings.ToLower(MaskZoneKey(each)) == zone
})
}
@@ -1846,9 +1770,7 @@ func parseNamedNetblocks(value string) ([]anomaly.NamedNetblock, error) {
return nil, fmt.Errorf("%q %w", item, errNotNamedNetblock)
}
netblock, err := reputation.ParseNetblock(
strings.TrimSpace(netblockText),
)
netblock, err := ParseNetblock(strings.TrimSpace(netblockText))
if err != nil {
return nil, err
}
@@ -2054,6 +1976,21 @@ func parseCountry(value string) (string, error) {
return country, nil
}
// ParseASN reads an AS number such as AS64496, in either case, and
// returns it as the lookup gives it: AS and the number, in capitals and
// without leading zeros. The file SWW_WAF_ASN_LIMIT_PERCENT_URL names is
// read with it too.
func ParseASN(value string) (string, error) {
digits, hasAS := strings.CutPrefix(strings.ToUpper(value), "AS")
number, err := strconv.ParseUint(digits, 10, 32)
if !hasAS || err != nil {
return "", fmt.Errorf("%q %w", value, errNotASN)
}
return "AS" + strconv.FormatUint(number, 10), nil
}
// The tags a request can get that the settings can name, as waf
// writes them. The prefixes are followed by a value.
const (
@@ -2067,17 +2004,24 @@ const (
ExploitSignatureTag = "exploitsignature"
)
// TagList is an address list of SWW_WAF_TAG_LISTS: the tag it adds to the
// requests of the clients it holds, and the URL it is fetched from.
type TagList struct {
Name string
URL string
}
// parseTagLists reads a comma-separated list of name=URL pairs, such as
// torexit=https://example.com/tor.txt, and returns each list with its name
// in lower case. A name can hold a =, as vpn=mullvad does, and ends at the
// = before http. A name or a URL listed twice is an error.
func parseTagLists(value string) ([]reputation.TagList, error) {
func parseTagLists(value string) ([]TagList, error) {
items, err := parseList(value)
if err != nil {
return nil, err
}
lists := make([]reputation.TagList, 0, len(items))
lists := make([]TagList, 0, len(items))
for _, item := range items {
end := strings.Index(item, "=http")
@@ -2085,12 +2029,9 @@ func parseTagLists(value string) ([]reputation.TagList, error) {
return nil, fmt.Errorf("%q %w", item, errNotTagList)
}
list := reputation.TagList{
Name: strings.ToLower(item[:end]),
URL: item[end+1:],
}
list := TagList{Name: strings.ToLower(item[:end]), URL: item[end+1:]}
if slices.ContainsFunc(lists, func(earlier reputation.TagList) bool {
if slices.ContainsFunc(lists, func(earlier TagList) bool {
return earlier.Name == list.Name || earlier.URL == list.URL
}) {
return nil, fmt.Errorf("%q %w", item, errListedTwice)
@@ -2103,8 +2044,7 @@ func parseTagLists(value string) ([]reputation.TagList, error) {
}
// parseTags reads a comma-separated list of tags, which parseTag reads. A
// tag listed twice is an error, which shows the tag as
// reputation.MaskZoneKey does.
// tag listed twice is an error, which shows the tag as MaskZoneKey does.
func parseTags(value string) ([]string, error) {
items, err := parseList(value)
if err != nil {
@@ -2124,7 +2064,7 @@ func parseTags(value string) ([]string, error) {
if slices.Contains(tags, tag) {
return nil, fmt.Errorf(
"%q %w",
reputation.MaskZoneKey(item),
MaskZoneKey(item),
errListedTwice,
)
}
@@ -2193,7 +2133,7 @@ func parseTagPercents(value string) (map[string]int64, error) {
if colon <= 0 {
return nil, fmt.Errorf(
"%q %w",
reputation.MaskZoneKey(item),
MaskZoneKey(item),
errNotTagPercentItem,
)
}
@@ -2207,7 +2147,7 @@ func parseTagPercents(value string) (map[string]int64, error) {
if err != nil || percent < 1 || percent > 99 {
return nil, fmt.Errorf(
"%q %w",
reputation.MaskZoneKey(item[colon+1:]),
MaskZoneKey(item[colon+1:]),
errNotTagPercent,
)
}
@@ -2219,11 +2159,7 @@ func parseTagPercents(value string) (map[string]int64, error) {
if _, listed := percents[tag]; listed {
// tag, since item[:colon] may end in a space, which MaskZoneKey
// would not mask.
return nil, fmt.Errorf(
"%q %w",
reputation.MaskZoneKey(tag),
errListedTwice,
)
return nil, fmt.Errorf("%q %w", MaskZoneKey(tag), errListedTwice)
}
percents[tag] = percent
@@ -2257,7 +2193,7 @@ func parsePercents(
return nil, err
}
percent, err := reputation.ParsePercent(percentText)
percent, err := ParsePercent(percentText)
if err != nil {
return nil, err
}
@@ -2272,6 +2208,38 @@ func parsePercents(
return percents, nil
}
// ParsePercent reads a percentage, a whole number from 0 to 100. The file
// SWW_WAF_ASN_LIMIT_PERCENT_URL names is read with it too.
func ParsePercent(value string) (int64, error) {
percent, err := strconv.ParseInt(value, 10, 64)
if err != nil || percent < 0 || percent > 100 {
return 0, fmt.Errorf("%q %w", value, errNotPercent)
}
return percent, nil
}
// headerNameChars are the characters RFC 9110 allows in a header name:
// letters, digits and these marks.
const headerNameChars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz" +
"0123456789!#$%&'*+-.^_`|~"
// IsHeaderName reports whether name can be a header name: one or more of
// the characters RFC 9110 allows in one.
func IsHeaderName(name string) bool {
if name == "" {
return false
}
for _, char := range name {
if !strings.ContainsRune(headerNameChars, char) {
return false
}
}
return true
}
// parseHeaderNames reads a comma-separated list of header names in either
// case, and returns them in lower case. Host and Transfer-Encoding are
// refused: Go's HTTP server takes them out of the request's headers.
@@ -2284,7 +2252,7 @@ func parseHeaderNames(value string) ([]string, error) {
headers := make([]string, 0, len(items))
for _, item := range items {
if !httpguts.ValidHeaderFieldName(item) {
if !IsHeaderName(item) {
return nil, fmt.Errorf("%q %w", item, errNotHeaderName)
}
@@ -2440,8 +2408,8 @@ const (
// other name for one that does not exist, so that the zone would list no
// client. A zone listed twice is an error, whatever the case of its
// letters, which DNS names ignore, and whatever its key, since
// reputation.MaskZoneKey shows two keys of one zone alike. An error shows a
// zone as it does.
// MaskZoneKey shows two keys of one zone alike. An error shows a zone as
// MaskZoneKey does.
func parseZones(value string) ([]string, error) {
zones, err := parseList(value)
if err != nil {
@@ -2451,14 +2419,11 @@ func parseZones(value string) ([]string, error) {
}
for i, zone := range zones {
shown := reputation.MaskZoneKey(zone)
shown := MaskZoneKey(zone)
listedBefore := slices.ContainsFunc(
zones[:i],
func(earlier string) bool {
return strings.EqualFold(
reputation.MaskZoneKey(earlier),
shown,
)
return strings.EqualFold(MaskZoneKey(earlier), shown)
},
)
@@ -2475,6 +2440,24 @@ func parseZones(value string) ([]string, error) {
return zones, nil
}
// MaskZoneKey returns zone with ******** in place of its key, if it is a
// zone of Spamhaus's keyed query service, a name under dq.spamhaus.net,
// such as <key>.xbl.dq.spamhaus.net, whose first label is the key. Any
// other zone it returns as it is. A zone is shown so wherever it leaves
// the process: in the log, the alerts and the metrics.
func MaskZoneKey(zone string) string {
// DNS names ignore case, and a name may be written with a dot at its
// end.
name := strings.TrimSuffix(strings.ToLower(zone), ".")
if !strings.HasSuffix(name, ".dq.spamhaus.net") {
return zone
}
_, rest, _ := strings.Cut(zone, ".")
return masked + "." + rest
}
// isZone reports whether each label of zone is as parseZones takes it. It
// also checks the hostnames of SWW_WAF_APPS, which are DNS names too.
func isZone(zone string) bool {
@@ -2530,7 +2513,7 @@ func parseWebhookHeaders(value string) (http.Header, string, error) {
name, headerValue, found := strings.Cut(item, ":")
name = strings.TrimSpace(name)
if !found || !httpguts.ValidHeaderFieldName(name) ||
if !found || !IsHeaderName(name) ||
strings.ContainsAny(headerValue, "\r\n\x00") {
return nil, "", fmt.Errorf(
"item %d %w",
+25 -35
View File
@@ -18,12 +18,10 @@ import (
"time"
"github.com/spf13/viper"
"golang.org/x/net/http/httpguts"
"sneak.berlin/go/smallwebwaf/internal/anomaly"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/reputation"
)
// The settings, by name.
@@ -616,7 +614,7 @@ func TestCoreRuleSetSettings(t *testing.T) {
},
{
environment{
coreRuleSetMode: string(config.CoreRuleSetModeDetect),
coreRuleSetMode: config.CoreRuleSetModeDetect,
coreRuleSetParanoiaLevel: "4",
coreRuleSetScoreThreshold: "10",
coreRuleSetDisabledRules: "942100, 920350",
@@ -866,32 +864,6 @@ func TestInstanceNameAndLoggedHeadersAsSet(t *testing.T) {
}
}
// config and rules check header names with httpguts.ValidHeaderFieldName.
// It takes a name of one or more of the characters RFC 9110 allows in one,
// as the check it replaced did: letters, digits and these marks.
func TestHeaderNameCheckTakesTheRFC9110Characters(t *testing.T) {
t.Parallel()
const marks = "!#$%&'*+-.^_`|~"
if httpguts.ValidHeaderFieldName("") {
t.Error("the empty name is taken")
}
for b := range 256 {
char := byte(b)
want := 'a' <= char && char <= 'z' || 'A' <= char && char <= 'Z' ||
'0' <= char && char <= '9' || strings.IndexByte(marks, char) >= 0
// string(char) would be the UTF-8 of U+0080 to U+00FF, two bytes,
// for a byte above 0x7f.
name := string([]byte{char})
if got := httpguts.ValidHeaderFieldName(name); got != want {
t.Errorf("the name %q is taken: %t, want %t", name, got, want)
}
}
}
func TestRemoteLogSettingsDefaults(t *testing.T) {
t.Parallel()
@@ -1722,7 +1694,7 @@ func TestTagSettingsAsSet(t *testing.T) {
tagLimitPercent: "vpn=mullvad :50, asn=014061:25, COUNTRY=cn:1, dnsbl=zen.example:99",
})
wantLists := []reputation.TagList{
wantLists := []config.TagList{
{Name: torExitTag, URL: torURL},
{Name: "vpn=mullvad", URL: "https://example.com/mullvad.txt"},
}
@@ -2061,9 +2033,8 @@ func TestReputationSettingsAsSet(t *testing.T) {
}
for _, tc := range []struct {
value string
action config.ReputationAction
percent int64
value, action string
percent int64
}{
{actionDeny, actionDeny, 0},
{actionLog, actionLog, 0},
@@ -2196,7 +2167,7 @@ func TestDNSBLSettingsAsSet(t *testing.T) {
// want are the zones, resolver, action and percent config.WAF gives.
want []string
wantResolver string
wantAction config.ReputationAction
wantAction string
wantPercent int64
}{
{
@@ -2316,6 +2287,25 @@ func TestInvalidDNSBLSettingStopsTheStartSayingWhatIsWrong(t *testing.T) {
}
}
func TestMaskZoneKeyMasksTheFirstLabelOfAZoneUnderDqSpamhausNet(
t *testing.T,
) {
t.Parallel()
for zone, want := range map[string]string{
spamhaus: spamhausMasked,
spamhaus + ".": spamhausMasked + ".",
"KEY.ZEN.DQ.SPAMHAUS.NET": "********.ZEN.DQ.SPAMHAUS.NET",
dronebl: dronebl,
"dq.spamhaus.net": "dq.spamhaus.net",
spamhaus + ".example": spamhaus + ".example",
} {
if got := config.MaskZoneKey(zone); got != want {
t.Errorf("MaskZoneKey(%q) is %q, want %q", zone, got, want)
}
}
}
func TestDNSBLZoneKeyIsLoggedMaskedAndNeverShown(t *testing.T) {
t.Parallel()
@@ -3059,7 +3049,7 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
logLevel: "info",
ruleFilesDir: "/etc/sww/rules.d",
ruleFilesEnabled: "true",
coreRuleSetMode: string(config.CoreRuleSetModeBlock),
coreRuleSetMode: config.CoreRuleSetModeBlock,
coreRuleSetParanoiaLevel: "1",
coreRuleSetScoreThreshold: "5",
coreRuleSetDisabledRules: defaultCoreRuleSetDisabledRules,
-90
View File
@@ -1,90 +0,0 @@
// Package dirwatch watches a directory, and calls a function once the
// directory has gone a short time without a change after one. The rule
// files and the lookup database are read again this way.
package dirwatch
import (
"context"
"time"
"github.com/fsnotify/fsnotify"
)
// quietTime is how long the directory must go without a change before the
// function is called, so that a file still being written, such as one
// saved in place, appended to or copied in with scp, is read only once
// whole.
const quietTime = 2 * time.Second
// Watcher watches one directory. The directory is watched, rather than a
// file in it, so that a file replaced by a rename is seen.
type Watcher struct {
watcher *fsnotify.Watcher
}
// New starts watching dir, or returns why it cannot.
func New(dir string) (*Watcher, error) {
watcher, err := fsnotify.NewWatcher()
if err != nil {
return nil, err
}
err = watcher.Add(dir)
if err != nil {
_ = watcher.Close()
return nil, err
}
return &Watcher{watcher: watcher}, nil
}
// Close stops watching the directory.
func (w *Watcher) Close() error {
return w.watcher.Close()
}
// CallWhenQuiet calls changed each time the directory has gone quietTime
// without a change, after a change to a path for which counts is true, until
// ctx is done. It also calls failed with each error the watch reports. The
// wait starts at once, as if for a change, so that a change made before the
// watch began is taken in too.
func (w *Watcher) CallWhenQuiet(
ctx context.Context,
counts func(path string) bool,
changed func(),
failed func(error),
) {
callWhenQuiet(
ctx, w.watcher.Events, w.watcher.Errors, counts, changed, failed,
)
}
// callWhenQuiet is CallWhenQuiet, for the changes from events and the
// errors from errs.
func callWhenQuiet(
ctx context.Context,
events <-chan fsnotify.Event,
errs <-chan error,
counts func(path string) bool,
changed func(),
failed func(error),
) {
quiet := time.NewTimer(quietTime)
defer quiet.Stop()
for {
select {
case <-ctx.Done():
return
case event := <-events:
if counts(event.Name) {
quiet.Reset(quietTime)
}
case <-quiet.C:
changed()
case err := <-errs:
failed(err)
}
}
}
-167
View File
@@ -1,167 +0,0 @@
package dirwatch
import (
"context"
"errors"
"path/filepath"
"sync/atomic"
"testing"
"testing/synctest"
"time"
"github.com/fsnotify/fsnotify"
)
// errWatch is an error the watch reports.
var errWatch = errors.New("too many changes")
func TestDirectoryThatCannotBeWatchedIsAnError(t *testing.T) {
t.Parallel()
_, err := New(filepath.Join(t.TempDir(), "missing"))
if err == nil {
t.Error("a missing directory is watched, want an error")
}
}
// The tests below run callWhenQuiet in a synctest bubble, where time is a
// clock of the test's own: time.Sleep moves it on at once, and
// synctest.Wait returns once callWhenQuiet waits again, so that every call
// due by then is made. The test sends the changes itself, as the watch of
// a directory cannot run in a bubble.
func TestSecondChangeStartsTheWaitAgain(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
loop := start(t, everyPath)
loop.changes <- fsnotify.Event{Name: "50-app.rules", Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
loop.wantCalls(t, 0)
// The second part starts the wait again.
loop.changes <- fsnotify.Event{Name: "50-app.rules", Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
loop.wantCalls(t, 0)
time.Sleep(time.Nanosecond)
synctest.Wait()
loop.wantCalls(t, 1)
})
}
func TestCalledOnceQuietAfterTheStart(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
// An edit saved before the watch began sends no change, and is
// taken in by this call.
loop := start(t, everyPath)
time.Sleep(quietTime)
synctest.Wait()
loop.wantCalls(t, 1)
// Long after, it has been called only once.
time.Sleep(time.Hour)
synctest.Wait()
loop.wantCalls(t, 1)
})
}
func TestChangeThatDoesNotCountIsNoChange(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
loop := start(t, func(path string) bool {
return path == "ipinfo_lite.mmdb"
})
// The wait that starts with the watch ends with a call.
time.Sleep(quietTime)
synctest.Wait()
loop.changes <- fsnotify.Event{Name: "other.mmdb", Op: fsnotify.Create}
time.Sleep(quietTime)
synctest.Wait()
loop.wantCalls(t, 1)
loop.changes <- fsnotify.Event{Name: "ipinfo_lite.mmdb", Op: fsnotify.Write}
time.Sleep(quietTime)
synctest.Wait()
loop.wantCalls(t, 2)
})
}
func TestErrorsOfTheWatchHandedOn(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
loop := start(t, everyPath)
loop.errs <- errWatch
synctest.Wait()
if !errors.Is(loop.failure, errWatch) {
t.Errorf("failed with %v, want %v", loop.failure, errWatch)
}
})
}
// loop is a callWhenQuiet that a test runs.
type loop struct {
// changes and errs send it changes and errors.
changes chan fsnotify.Event
errs chan error
// calls counts its calls of changed, and failure is the last error it
// handed to failed.
calls atomic.Int32
failure error
}
// start runs callWhenQuiet, with counts, until the test ends.
func start(t *testing.T, counts func(path string) bool) *loop {
t.Helper()
l := &loop{changes: make(chan fsnotify.Event), errs: make(chan error)}
ctx, stop := context.WithCancel(t.Context())
stopped := make(chan struct{})
go func() {
callWhenQuiet(ctx, l.changes, l.errs, counts,
func() { l.calls.Add(1) },
func(err error) { l.failure = err },
)
close(stopped)
}()
t.Cleanup(func() {
stop()
<-stopped
})
return l
}
// everyPath counts a change to any path.
func everyPath(string) bool {
return true
}
// wantCalls checks how many times changed has been called.
func (l *loop) wantCalls(t *testing.T, want int32) {
t.Helper()
got := l.calls.Load()
if got != want {
t.Errorf("changed called %d times, want %d", got, want)
}
}
+37 -60
View File
@@ -84,7 +84,12 @@ type Params struct {
// one request to GeoJS is under way at a time, and it asks about every
// client waiting, up to maxPerRequest. It is safe for concurrent use.
type GeoJS struct {
params Params
url string
answered func(Answer)
now func() time.Time
asked func()
failed func(err error, askingAgainIn time.Duration)
wentWithoutAnswer func()
// httpClient follows no redirect, so that visitors' addresses go to
// GeoJS alone: a redirect is a failure.
httpClient *http.Client
@@ -94,13 +99,8 @@ type GeoJS struct {
// waiting are the clients without an answer: those to ask GeoJS about,
// and those it is being asked about.
waiting map[netip.Prefix]*wait
// asking is true while a request to GeoJS is under way, and endAsking
// ends it. stopped is true once StopAsking has been called, and
// askGoroutines waits for the goroutine that makes the requests.
asking bool
endAsking context.CancelFunc
stopped bool
askGoroutines sync.WaitGroup
// asking is true while a request to GeoJS is under way.
asking bool
// retryDelay is how long GeoJS is left alone after its last failure,
// zero after an answer; retryAt is when it may be asked again.
retryDelay time.Duration
@@ -140,7 +140,12 @@ func New(params Params) *GeoJS {
}
return &GeoJS{
params: params,
url: params.URL,
answered: params.Answered,
now: params.Now,
asked: params.Asked,
failed: params.Failed,
wentWithoutAnswer: params.WentWithoutAnswer,
httpClient: &http.Client{
CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse
@@ -183,7 +188,7 @@ func (g *GeoJS) LookUp(
answer, found := g.kept(client)
if !found {
g.params.WentWithoutAnswer()
g.wentWithoutAnswer()
}
w, waiting := g.waiting[client]
@@ -205,23 +210,6 @@ func (g *GeoJS) LookUpWithoutWaiting(
return answer
}
// StopAsking ends the request to GeoJS under way, and returns once the
// goroutine that made it has. A request it ends is no failure: Failed is not
// called, and GeoJS is not left alone after it. After StopAsking, GeoJS is
// asked nothing.
func (g *GeoJS) StopAsking() {
g.mu.Lock()
g.stopped = true
if g.asking {
g.endAsking()
}
g.mu.Unlock()
g.askGoroutines.Wait()
}
// Kept returns client's answer, if one is kept, without asking GeoJS.
func (g *GeoJS) Kept(client netip.Prefix) (Answer, bool) {
g.mu.Lock()
@@ -263,7 +251,7 @@ func (g *GeoJS) Load(answers []Answer) {
g.answers.Purge()
now := g.params.Now()
now := g.now()
for _, answer := range answers {
if now.Sub(answer.Answered) < keepFor {
@@ -300,19 +288,18 @@ func (g *GeoJS) answerOrWait(
}
if w == nil {
g.params.WentWithoutAnswer()
g.wentWithoutAnswer()
return Answer{}, nil // too many clients wait already
}
if !g.asking {
// GeoJS is left alone after a failure, or asked nothing after
// StopAsking, so no answer can come.
// GeoJS is left alone after a failure, so no answer can come.
w.late = true
}
if w.late {
g.params.WentWithoutAnswer()
g.wentWithoutAnswer()
return Answer{}, nil
}
@@ -323,7 +310,7 @@ func (g *GeoJS) answerOrWait(
// kept returns client's answer, if GeoJS gave it less than keepFor ago,
// and notes that it was used.
func (g *GeoJS) kept(client netip.Prefix) (Answer, bool) {
now := g.params.Now()
now := g.now()
kept, found := g.answers.Get(client)
if !found || now.Sub(kept.Answered) >= keepFor {
@@ -336,28 +323,22 @@ func (g *GeoJS) kept(client netip.Prefix) (Answer, bool) {
}
// ask starts asking GeoJS about the waiting clients, unless a request to
// it is under way, it is left alone after a failure, or StopAsking has
// been called. The requests to GeoJS are for every client waiting, so they
// go on when the caller whose ctx is given stops waiting, until StopAsking.
// it is under way or it is left alone after a failure. The requests to
// GeoJS are for every client waiting, so they go on when the caller whose
// ctx is given stops waiting.
func (g *GeoJS) ask(ctx context.Context) {
if g.asking || g.stopped || g.params.Now().Before(g.retryAt) {
if g.asking || g.now().Before(g.retryAt) {
return
}
askCtx, end := context.WithCancel(context.WithoutCancel(ctx))
g.asking, g.endAsking = true, end
g.asking = true
g.askGoroutines.Go(func() {
defer end()
g.askAboutWaiting(askCtx)
})
go g.askAboutWaiting(context.WithoutCancel(ctx))
}
// askAboutWaiting asks GeoJS about the waiting clients, one request at a
// time, until none is left, GeoJS fails, or StopAsking ends ctx. Each
// answer kept is given to Answered, outside the lock, since Answered takes
// locks of its own.
// time, until none is left or GeoJS fails. Each answer kept is given to
// Answered, outside the lock, since Answered takes locks of its own.
func (g *GeoJS) askAboutWaiting(ctx context.Context) {
for {
clients := g.nextClients()
@@ -366,15 +347,11 @@ func (g *GeoJS) askAboutWaiting(ctx context.Context) {
}
given, err := g.request(ctx, clients)
if err != nil && ctx.Err() != nil {
continue // ended by StopAsking, which is no failure
}
kept, answered := g.keep(clients, given, err)
if g.params.Answered != nil {
if g.answered != nil {
for _, answer := range kept {
g.params.Answered(answer)
g.answered(answer)
}
}
@@ -385,13 +362,13 @@ func (g *GeoJS) askAboutWaiting(ctx context.Context) {
}
// nextClients returns up to maxPerRequest of the waiting clients. When
// none is waiting, or StopAsking has been called, it returns none and
// notes that no request to GeoJS is under way.
// none is waiting, it returns none and notes that no request to GeoJS is
// under way.
func (g *GeoJS) nextClients() []netip.Prefix {
g.mu.Lock()
defer g.mu.Unlock()
if len(g.waiting) == 0 || g.stopped {
if len(g.waiting) == 0 {
g.asking = false
return nil
@@ -423,7 +400,7 @@ func (g *GeoJS) keep(
g.mu.Lock()
defer g.mu.Unlock()
now := g.params.Now()
now := g.now()
kept := make([]Answer, 0, len(clients))
leftOut := 0
@@ -467,7 +444,7 @@ func (g *GeoJS) keep(
w.late = true
}
g.params.Failed(err, g.retryDelay)
g.failed(err, g.retryDelay)
return kept, false
}
@@ -497,7 +474,7 @@ func (g *GeoJS) request(
req, err := http.NewRequestWithContext(
ctx,
http.MethodGet,
g.params.URL,
g.url,
http.NoBody,
)
if err != nil {
@@ -506,7 +483,7 @@ func (g *GeoJS) request(
req.URL.RawQuery = "ip=" + strings.Join(addrs, ",")
g.params.Asked()
g.asked()
res, err := g.httpClient.Do(req)
if err != nil {
-38
View File
@@ -318,44 +318,6 @@ func TestCallerWhoseContextEndsEarlyDoesNotStopTheRequest(t *testing.T) {
})
}
func TestStopAskingEndsTheRequestUnderWayAsNoFailureAndAsksNoMore(
t *testing.T,
) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
service, _, g, reports := startWithReports()
clients := newClients()
service.set(hanging)
g.LookUpWithoutWaiting(t.Context(), clients())
waitForRequests(t, service, 1)
// The request is ended, not waited out, and is no failure.
began := time.Now()
g.StopAsking()
if waited := time.Since(began); waited != 0 {
t.Errorf("StopAsking returned after %s, want at once", waited)
}
reports.wantFailures(t)
// A new client is then not asked about, and its lookup does not
// wait.
service.set(answering)
wantCountry(t, g, clients(), "")
if waited := time.Since(began); waited != 0 {
t.Errorf("the lookup returned after %s, want at once", waited)
}
waitForRequests(t, service, 1)
reports.wantFailures(t)
})
}
func TestRequestOver10SecondsIsAFailure(t *testing.T) {
t.Parallel()
+49 -18
View File
@@ -13,13 +13,18 @@ import (
"sync"
"time"
"github.com/fsnotify/fsnotify"
"github.com/oschwald/maxminddb-golang/v2"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/dirwatch"
"sneak.berlin/go/smallwebwaf/internal/geojs"
)
// quietTime is how long the lookup database must go without a change
// before it is read again, so that a file still being copied in is read
// only once whole.
const quietTime = 2 * time.Second
// FileParams are what OpenFile needs.
type FileParams struct {
// Path is the lookup database, the IPinfo Lite file in its .mmdb form
@@ -117,13 +122,20 @@ func (f *File) ReadFailures() int {
}
// Watch watches the directory of the lookup database until ctx is done,
// and reads the file again once it has gone without a change for a short
// time, after it is replaced, written or removed, and after Watch starts
// watching, as dirwatch describes. A change to another file in its
// directory does not count. If the directory cannot be watched, that is
// logged, and the database read at start stays in use.
// and reads the file again once it has gone without a change for
// quietTime, after it is replaced, written or removed, and after Watch
// starts watching. If the directory cannot be watched, that is logged, and
// the database read at start stays in use.
func (f *File) Watch(ctx context.Context) {
watcher, err := dirwatch.New(filepath.Dir(f.params.Path))
watcher, err := fsnotify.NewWatcher()
if err == nil {
defer func() {
_ = watcher.Close()
}()
err = watcher.Add(filepath.Dir(f.params.Path))
}
if err != nil {
f.params.ProcessLog.Error(
"cannot watch the lookup database for replacements",
@@ -134,22 +146,41 @@ func (f *File) Watch(ctx context.Context) {
return
}
defer func() {
_ = watcher.Close()
}()
f.params.ProcessLog.Info("watching the lookup database for replacements",
"file", f.params.Path)
watcher.CallWhenQuiet(ctx, f.isDatabase, f.readAgain, func(err error) {
f.params.ProcessLog.Warn("watching the lookup database failed",
"error", err.Error())
})
f.readAfterChanges(ctx, watcher.Events, watcher.Errors)
}
// isDatabase reports whether path names the lookup database.
func (f *File) isDatabase(path string) bool {
return filepath.Clean(path) == filepath.Clean(f.params.Path)
// readAfterChanges reads the lookup database again once quietTime has
// passed without a change to it from events, until ctx is done, and logs
// the errors from errs. A change to another file in its directory does not
// count. The wait starts at once, as if for a change, so that a file
// replaced after OpenFile read it, and before its directory was watched,
// is read too.
func (f *File) readAfterChanges(
ctx context.Context, events <-chan fsnotify.Event, errs <-chan error,
) {
path := filepath.Clean(f.params.Path)
quiet := time.NewTimer(quietTime)
defer quiet.Stop()
for {
select {
case <-ctx.Done():
return
case event := <-events:
if filepath.Clean(event.Name) == path {
quiet.Reset(quietTime)
}
case <-quiet.C:
f.readAgain()
case err := <-errs:
f.params.ProcessLog.Warn("watching the lookup database failed",
"error", err.Error())
}
}
}
// readAgain reads the lookup database again, in place of the one in use,
+141 -35
View File
@@ -1,6 +1,7 @@
package lookup
import (
"context"
"log/slog"
"net/netip"
"net/url"
@@ -11,6 +12,7 @@ import (
"testing/synctest"
"time"
"github.com/fsnotify/fsnotify"
"github.com/maxmind/mmdbwriter/mmdbtype"
"sneak.berlin/go/smallwebwaf/internal/alerts"
@@ -131,48 +133,66 @@ func TestFileThatCannotBeReadIsAnError(t *testing.T) {
}
}
func TestChangeOfAnotherFileInTheDirectoryIsNoReplacement(t *testing.T) {
t.Parallel()
// The tests below run readAfterChanges in a synctest bubble, where time is
// a clock of the test's own: time.Sleep moves it on at once, and
// synctest.Wait returns once readAfterChanges waits again, so that every
// reading due by then is done. The test sends the changes itself, as the
// watch of a directory cannot run in a bubble.
dir := t.TempDir()
path := filepath.Join(dir, "ipinfo_lite.mmdb")
writeDatabase(t, path, "DE")
f := openFile(t, path, newQueue())
for changed, want := range map[string]bool{
path: true,
dir + "/./ipinfo_lite.mmdb": true,
filepath.Join(dir, "other.mmdb"): false,
} {
if f.isDatabase(changed) != want {
t.Errorf(
"a change to %s counts: %t, want %t",
changed,
!want,
want,
)
}
}
}
// The tests below run in a synctest bubble, where time is a clock of the
// test's own, so that the time a file is read can be checked. The wait for
// the directory to go quiet before a reading is tested in dirwatch.
func TestReplacementIsReadAgain(t *testing.T) {
func TestReplacementCopiedOverTheFileInTwoPartsIsReadOnlyWhole(
t *testing.T,
) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb")
dir := t.TempDir()
path := filepath.Join(dir, "ipinfo_lite.mmdb")
writeDatabase(t, path, "DE")
queue := newQueue()
f := openFile(t, path, queue)
changes := watch(t, f)
writeDatabase(t, path, "KP")
time.Sleep(time.Hour)
f.readAgain()
other := filepath.Join(dir, "replacement.mmdb")
writeDatabase(t, other, "KP")
//nolint:gosec // a file the test wrote
replacement, err := os.ReadFile(other)
if err != nil {
t.Fatalf("read %s: %v", other, err)
}
// The file in use is overwritten in place, and keeps giving what
// it gave. Its first part alone is not a .mmdb file.
file, err := os.Create(path) //nolint:gosec // a file the test wrote
if err != nil {
t.Fatalf("create %s: %v", path, err)
}
defer func() {
_ = file.Close()
}()
half := len(replacement) / 2
write(t, file, replacement[:half])
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantCountry(t, f, "DE")
// The second part starts the wait again.
write(t, file, replacement[half:])
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantCountry(t, f, "DE")
time.Sleep(time.Nanosecond)
synctest.Wait()
wantCountry(t, f, "KP")
if !f.LastRead().Equal(time.Now()) || f.ReadFailures() != 0 {
@@ -199,10 +219,15 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(
queue := newQueue()
f := openFile(t, path, queue)
read := f.LastRead()
changes := watch(t, f)
writeFile(t, path, "not a lookup database\n")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
// Long after, the replacement has been read once.
time.Sleep(time.Hour)
f.readAgain()
synctest.Wait()
wantCountry(t, f, "DE")
if !f.LastRead().Equal(read) || f.ReadFailures() != 1 {
@@ -215,7 +240,7 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(
}
wantAlerts(t, queue, alerts.Alert{
Time: read.Add(time.Hour),
Time: read.Add(quietTime),
Event: alerts.EventFileError,
Reason: "the lookup database cannot be read, and the one read before stays in use",
Detail: map[string]any{
@@ -227,6 +252,55 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(
})
}
func TestChangeOfAnotherFileInTheDirectoryIsNoReplacement(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "ipinfo_lite.mmdb")
writeDatabase(t, path, "DE")
f := openFile(t, path, newQueue())
changes := watch(t, f)
// The wait that starts with the watch ends with a reading.
time.Sleep(quietTime)
synctest.Wait()
writeDatabase(t, path, "KP")
changes <- fsnotify.Event{Name: filepath.Join(dir, "other.mmdb"), Op: fsnotify.Create}
time.Sleep(quietTime)
synctest.Wait()
wantCountry(t, f, "DE")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime)
synctest.Wait()
wantCountry(t, f, "KP")
})
}
func TestReplacementSavedBeforeTheWatchStartsIsRead(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb")
writeDatabase(t, path, "DE")
f := openFile(t, path, newQueue())
// Saved after OpenFile read the file, and before its directory was
// watched, so that no change is seen for it.
writeDatabase(t, path, "KP")
watch(t, f)
time.Sleep(quietTime)
synctest.Wait()
wantCountry(t, f, "KP")
})
}
// newQueue returns a queue of alerts for a webhook that is never sent
// them, so that they wait in it for the test to look at.
func newQueue() *alerts.Queue {
@@ -270,6 +344,28 @@ func openFile(t *testing.T, path string, queue *alerts.Queue) *File {
return f
}
// watch runs f's readAfterChanges until the test ends, and returns the
// channel that sends it changes.
func watch(t *testing.T, f *File) chan<- fsnotify.Event {
t.Helper()
changes := make(chan fsnotify.Event)
ctx, stop := context.WithCancel(t.Context())
stopped := make(chan struct{})
go func() {
f.readAfterChanges(ctx, changes, nil)
close(stopped)
}()
t.Cleanup(func() {
stop()
<-stopped
})
return changes
}
// wantCountry checks the country f gives testClient.
func wantCountry(t *testing.T, f *File, want string) {
t.Helper()
@@ -305,3 +401,13 @@ func writeFile(t *testing.T, path, content string) {
t.Fatalf("write %s: %v", path, err)
}
}
// write writes data to the end of file.
func write(t *testing.T, file *os.File, data []byte) {
t.Helper()
_, err := file.Write(data)
if err != nil {
t.Fatalf("write: %v", err)
}
}
+3 -4
View File
@@ -14,6 +14,7 @@ import (
"github.com/prometheus/client_golang/prometheus/promhttp"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/remotelog"
"sneak.berlin/go/smallwebwaf/internal/reputation"
@@ -300,7 +301,7 @@ const sourceLabel = "source"
// AddReputation adds the metrics of the lists fetched from URLs and of the
// DNSBL zones, by source, each list's URL or each zone, its key masked as
// reputation.MaskZoneKey masks it: the requests whose client a blocklist, a
// config.MaskZoneKey masks it: the requests whose client a blocklist, a
// zone's verdict or AbuseIPDB's score lists, which ReputationHit counts,
// and, read from lists and dnsbl as the metrics are asked for, for a list,
// the fetches that failed and when the copy in use was fetched, and for a
@@ -319,9 +320,7 @@ func (m *Metrics) AddReputation(
m.registry.MustRegister(m.reputationHits)
for _, zone := range dnsbl.Zones() {
source := prometheus.Labels{
sourceLabel: reputation.MaskZoneKey(zone),
}
source := prometheus.Labels{sourceLabel: config.MaskZoneKey(zone)}
m.addReputationQueries(
source,
-44
View File
@@ -1,44 +0,0 @@
// Package processlog builds the logger for the process's own messages,
// which it writes as JSON lines marked "type":"process". The request log
// is package requestlog's.
package processlog
import (
"io"
"log/slog"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// Params are what New needs.
type Params struct {
// Output is where the lines are written.
Output io.Writer
// InstanceName, SWW_WAF_INSTANCE_NAME, is each line's instance.
InstanceName string
// Level, SWW_WAF_LOG_LEVEL, is the least severe level written.
Level slog.Level
}
// New returns the logger for the process's own messages: JSON lines on
// params.Output, marked "type":"process", with the time in the same form
// as a request line's. It writes only the messages at params.Level or more
// severe; the request lines requestlog.Write writes are never held back.
func New(params Params) *slog.Logger {
handler := slog.NewJSONHandler(params.Output, &slog.HandlerOptions{
Level: params.Level,
ReplaceAttr: func(groups []string, attr slog.Attr) slog.Attr {
if attr.Key == slog.TimeKey && len(groups) == 0 {
return slog.String(
slog.TimeKey,
requestlog.FormatTime(attr.Value.Time()),
)
}
return attr
},
})
return slog.New(handler).
With("type", "process", "instance", params.InstanceName)
}
-105
View File
@@ -1,105 +0,0 @@
package processlog_test
import (
"bytes"
"encoding/json"
"log/slog"
"slices"
"strings"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/processlog"
)
// instanceName is the instance the tests' loggers are built with.
const instanceName = "host1/blog"
func TestProcessLinesAreMarkedProcessAndGiveTheInstance(t *testing.T) {
t.Parallel()
var out bytes.Buffer
processlog.New(processlog.Params{
Output: &out,
InstanceName: instanceName,
Level: slog.LevelInfo,
}).
Info("starting",
"version", "v1")
var fields map[string]any
err := json.Unmarshal(out.Bytes(), &fields)
if err != nil {
t.Fatalf("decode %q: %v", out.String(), err)
}
if fields["type"] != "process" ||
fields["instance"] != "host1/blog" ||
fields["msg"] != "starting" ||
fields["level"] != "INFO" ||
fields["version"] != "v1" {
t.Errorf("process line %v", fields)
}
timeText, _ := fields["time"].(string)
logged, err := time.Parse(time.RFC3339, timeText)
if err != nil || !strings.HasSuffix(timeText, "Z") ||
len(timeText) != len("2006-01-02T15:04:05.000Z") ||
time.Since(logged) > time.Minute {
t.Errorf(
"process line time %q, want now in UTC with milliseconds",
timeText,
)
}
}
func TestProcessLoggerWritesTheMessagesAtItsLevelOrMoreSevere(t *testing.T) {
t.Parallel()
levels := []slog.Level{
slog.LevelDebug, slog.LevelInfo, slog.LevelWarn, slog.LevelError,
}
for i, level := range levels {
t.Run(level.String(), func(t *testing.T) {
t.Parallel()
var out bytes.Buffer
processLog := processlog.New(processlog.Params{
Output: &out,
InstanceName: instanceName,
Level: level,
})
for _, at := range levels {
processLog.Log(t.Context(), at, "message")
}
var got, want []string
for line := range strings.Lines(out.String()) {
var fields struct {
Level string `json:"level"`
}
err := json.Unmarshal([]byte(line), &fields)
if err != nil {
t.Fatalf("decode %q: %v", line, err)
}
got = append(got, fields.Level)
}
for _, written := range levels[i:] {
want = append(want, written.String())
}
if !slices.Equal(got, want) {
t.Errorf("lines at %v, want %v", got, want)
}
})
}
}
+4 -7
View File
@@ -11,7 +11,7 @@ import (
func TestHistoryKeepsEveryRequest(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
@@ -57,7 +57,7 @@ func TestHistoryKeepsEveryRequest(t *testing.T) {
func TestLookupReachesTheHistoryOfAClientInTheTable(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
client := netip.MustParsePrefix("203.0.113.9/32")
other := netip.MustParsePrefix("198.51.100.7/32")
start := midnight()
@@ -100,10 +100,7 @@ func TestLookupReachesTheHistoryOfAClientInTheTable(t *testing.T) {
func TestResetKeepsTheHistory(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{PerMinute: limit},
MaxClients: tableSize,
})
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit}, tableSize)
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
@@ -126,7 +123,7 @@ func TestResetKeepsTheHistory(t *testing.T) {
func TestRequestsAddsUpTheClientsInsideTheNetblock(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
for client, requests := range map[string]int{
"198.51.100.9/32": 2,
+6 -18
View File
@@ -156,28 +156,16 @@ type Request struct {
BlockedByCoreRuleSet bool
}
// Params are what New needs.
type Params struct {
// Limits are what each client's requests and bytes are counted against.
Limits Limits
// MaxClients is the most clients the table holds,
// SWW_WAF_MAX_TRACKED_CLIENTS. Past it, the least recently seen client
// is dropped, with its history, and starts afresh if it comes back.
MaxClients int
}
// New returns a Limiter with no client counted yet.
func New(params Params) *Limiter {
clients, err := simplelru.NewLRU[netip.Prefix, *Client](
params.MaxClients,
nil,
)
// New returns a Limiter for limits, with no client counted yet, whose
// table holds at most maxClients clients (SWW_WAF_MAX_TRACKED_CLIENTS). Past
// it, the least recently seen client is dropped, with its history, and
// starts afresh if it comes back.
func New(limits Limits, maxClients int) *Limiter {
clients, err := simplelru.NewLRU[netip.Prefix, *Client](maxClients, nil)
if err != nil {
panic(err) // NewLRU fails only for a size below one
}
limits := params.Limits
return &Limiter{
windows: [3]window{
{
+39 -63
View File
@@ -33,10 +33,10 @@ func TestOverARateLimitSaysWhatTheNextRequestWouldDoAndCountsNothing(
client := netip.MustParsePrefix("203.0.113.9/32")
now := midnight()
limiter := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{PerMinute: limit, PerDay: limit + 1},
MaxClients: tableSize,
})
limiter := ratelimit.New(
ratelimit.Limits{PerMinute: limit, PerDay: limit + 1},
tableSize,
)
if limiter.OverARateLimit(client, now) {
t.Error("a client never seen is over a rate limit")
@@ -72,7 +72,7 @@ func TestOverARateLimitSaysWhatTheNextRequestWouldDoAndCountsNothing(
wantCount(t, limiter, client, now, minute)
// A limit that is off is never gone over.
off := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
off := ratelimit.New(ratelimit.Limits{}, tableSize)
off.Count(client, now, whole)
if off.OverARateLimit(client, now) {
@@ -95,10 +95,7 @@ func TestEachWindowRefusesAtItsLimitAndLetsTheClientBack(t *testing.T) {
t.Run(tc.window, func(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Params{
Limits: tc.limits,
MaxClients: tableSize,
})
limiter := ratelimit.New(tc.limits, tableSize)
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
quarter := tc.length / 4
@@ -129,10 +126,10 @@ func TestEachWindowRefusesAtItsLimitAndLetsTheClientBack(t *testing.T) {
func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{PerMinute: limit, PerHour: limit},
MaxClients: tableSize,
})
limiter := ratelimit.New(
ratelimit.Limits{PerMinute: limit, PerHour: limit},
tableSize,
)
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
@@ -162,13 +159,10 @@ func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
func TestClientGetsItsPercentageOfEachLimitRoundedDown(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{
PerMinute: 5,
BytesPerDay: math.MaxInt64,
},
MaxClients: tableSize,
})
limiter := ratelimit.New(
ratelimit.Limits{PerMinute: 5, BytesPerDay: math.MaxInt64},
tableSize,
)
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
@@ -209,10 +203,10 @@ func TestZeroPercentIsAZeroAllowanceAndALimitOffStaysOff(t *testing.T) {
t.Parallel()
// Only the hour has limits: the minute's and the day's are off.
limiter := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{PerHour: limit, BytesPerHour: 1000},
MaxClients: tableSize,
})
limiter := ratelimit.New(
ratelimit.Limits{PerHour: limit, BytesPerHour: 1000},
tableSize,
)
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
@@ -259,10 +253,7 @@ func TestEachByteLimitIsBrokenByTheBytesCounted(t *testing.T) {
t.Run(tc.window, func(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Params{
Limits: tc.limits,
MaxClients: tableSize,
})
limiter := ratelimit.New(tc.limits, tableSize)
client := netip.MustParsePrefix("203.0.113.9/32")
// 600 bytes are within the limit, 600 more over it.
@@ -296,10 +287,10 @@ func TestEachByteLimitIsBrokenByTheBytesCounted(t *testing.T) {
func TestALimitIsBrokenOnlyByWhatIsAddedToIt(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{PerMinute: 2, BytesPerMinute: 1000},
MaxClients: tableSize,
})
limiter := ratelimit.New(
ratelimit.Limits{PerMinute: 2, BytesPerMinute: 1000},
tableSize,
)
client := netip.MustParsePrefix("203.0.113.9/32")
other := netip.MustParsePrefix("203.0.113.10/32")
start := midnight()
@@ -323,7 +314,7 @@ func TestALimitIsBrokenOnlyByWhatIsAddedToIt(t *testing.T) {
func TestCountGivesTheBytesInEachWindow(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
@@ -349,10 +340,7 @@ func TestCountGivesTheBytesInEachWindow(t *testing.T) {
func TestResetSetsTheBytesBackToZero(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{BytesPerDay: 1000},
MaxClients: tableSize,
})
limiter := ratelimit.New(ratelimit.Limits{BytesPerDay: 1000}, tableSize)
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
@@ -368,7 +356,7 @@ func TestRefusalsOverTheLimitInAMinuteBreakTheRefusalLimit(
) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
@@ -423,7 +411,7 @@ func TestRefusalsOverTheLimitInAMinuteBreakTheRefusalLimit(
func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
@@ -450,10 +438,10 @@ func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
func TestResetSetsTheCountsBackToZero(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{PerMinute: limit, PerDay: limit},
MaxClients: tableSize,
})
limiter := ratelimit.New(
ratelimit.Limits{PerMinute: limit, PerDay: limit},
tableSize,
)
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
@@ -475,10 +463,7 @@ func TestResetSetsTheCountsBackToZero(t *testing.T) {
func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{PerHour: limit},
MaxClients: tableSize,
})
limiter := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize)
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
@@ -497,10 +482,10 @@ func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) {
func TestRefusedRequestsCount(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{PerMinute: limit, PerHour: 2 * limit},
MaxClients: tableSize,
})
limiter := ratelimit.New(
ratelimit.Limits{PerMinute: limit, PerHour: 2 * limit},
tableSize,
)
refused := netip.MustParsePrefix("203.0.113.9/32")
within := netip.MustParsePrefix("203.0.113.10/32")
start := midnight()
@@ -533,10 +518,7 @@ func TestRefusedRequestsCount(t *testing.T) {
func TestRequestCountedLateGoesInTheBucketUnderWay(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{PerMinute: limit},
MaxClients: tableSize,
})
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit}, tableSize)
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
@@ -552,10 +534,7 @@ func TestRequestCountedLateGoesInTheBucketUnderWay(t *testing.T) {
func TestClockSetBackStartsTheBucketsAfresh(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{PerHour: limit},
MaxClients: tableSize,
})
limiter := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize)
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
@@ -583,10 +562,7 @@ func TestKeepsAtMostMaxClientsDroppingTheLeastRecentlySeen(t *testing.T) {
const maxClients = 3
limiter := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{PerMinute: 1},
MaxClients: maxClients,
})
limiter := ratelimit.New(ratelimit.Limits{PerMinute: 1}, maxClients)
now := midnight()
clients := make([]netip.Prefix, maxClients+1)
+6 -12
View File
@@ -19,7 +19,7 @@ func TestSnapshotListsTheClientsByAddress(t *testing.T) {
"2001:db8::/64",
}
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
for _, i := range []int{2, 3, 0, 1} {
limiter.Count(netip.MustParsePrefix(want[i]), midnight(), whole)
}
@@ -52,10 +52,7 @@ func TestLoadedCountsCarryOn(t *testing.T) {
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
before := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{PerHour: limit},
MaxClients: tableSize,
})
before := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize)
for range limit {
wantCount(t, before, client, start, "")
}
@@ -63,10 +60,7 @@ func TestLoadedCountsCarryOn(t *testing.T) {
// Loaded into a new limiter, as across a restart, the client has no
// fresh allowance.
later := start.Add(time.Minute)
after := ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{PerHour: limit},
MaxClients: tableSize,
})
after := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize)
after.Load(before.Snapshot(), later)
wantCount(t, after, client, later, hour)
}
@@ -77,7 +71,7 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
limiter.Count(client, start, whole)
limiter.CountBytes(client, start, 5, whole)
limiter.CountRefusal(client, start, limit)
@@ -86,7 +80,7 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
loaded := func(now time.Time) ratelimit.Client {
t.Helper()
after := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
after := ratelimit.New(ratelimit.Limits{}, tableSize)
after.Load(limiter.Snapshot(), now)
return after.Snapshot()[0]
@@ -148,7 +142,7 @@ func TestLoadDropsTheLeastRecentlySeenFirst(t *testing.T) {
addr = addr.Next()
}
limiter := ratelimit.New(ratelimit.Params{MaxClients: maxClients})
limiter := ratelimit.New(ratelimit.Limits{}, maxClients)
limiter.Load(clients, midnight())
got := limiter.Snapshot()
+16 -48
View File
@@ -110,12 +110,8 @@ type AbuseIPDB struct {
// scores are by client. Each is added as it is fetched and never moved
// up, so that the one fetched longest ago is the first dropped.
scores *simplelru.LRU[netip.Prefix, Score]
// checking are the clients whose check is under way, each with the
// function that ends it. stopped is true once StopChecking has been
// called, and checkGoroutines waits for the goroutine of each check.
checking map[netip.Prefix]context.CancelFunc
stopped bool
checkGoroutines sync.WaitGroup
// checking are the clients whose check is under way.
checking map[netip.Prefix]bool
// day is the day, in UTC, of the checks spent counts.
day civil.Date
spent int
@@ -138,7 +134,7 @@ func NewAbuseIPDB(params AbuseIPDBParams) *AbuseIPDB {
params: params,
httpClient: &http.Client{},
scores: scores,
checking: map[netip.Prefix]context.CancelFunc{},
checking: map[netip.Prefix]bool{},
}
}
@@ -148,11 +144,11 @@ func NewAbuseIPDB(params AbuseIPDBParams) *AbuseIPDB {
// addresses its request comes from. A client without one is checked in
// the background, by addr, the address its request came from, if
// offender, if it has committed an offence, unless its check is under
// way, a check failed less than failureDelay ago, the day's checks have
// used up DailyBudget, or StopChecking has been called; Hit never waits
// for a check. The check that uses the budget up is logged and raised as a
// source_failure alert. ctx is the context of the client's request, and a
// check goes on after the request ends, until StopChecking.
// way, a check failed less than failureDelay ago, or the day's checks
// have used up DailyBudget; Hit never waits for a check. The check that
// uses the budget up is logged and raised as a source_failure alert. ctx
// is the context of the client's request, and a check goes on after the
// request ends.
func (a *AbuseIPDB) Hit(
ctx context.Context, client netip.Prefix, addr netip.Addr, offender bool,
) (int64, bool) {
@@ -171,21 +167,14 @@ func (a *AbuseIPDB) Hit(
a.day, a.spent = today, 0
}
_, checking := a.checking[client]
check := offender && !checking && !a.stopped &&
!now.Before(a.retryAt) && a.spent < a.params.DailyBudget
check := offender && !a.checking[client] && !now.Before(a.retryAt) &&
a.spent < a.params.DailyBudget
if check {
checkCtx, end := context.WithCancel(context.WithoutCancel(ctx))
a.checking[client] = end
a.checking[client] = true
a.checks++
a.spent++
a.checkGoroutines.Go(func() {
defer end()
a.check(checkCtx, client, addr)
})
go a.check(context.WithoutCancel(ctx), client, addr)
}
usedUp := check && a.spent == a.params.DailyBudget
@@ -200,24 +189,6 @@ func (a *AbuseIPDB) Hit(
return 0, false
}
// StopChecking ends the checks under way, and returns once their goroutines
// have. A check it ends gives no score and is no failure: it is neither
// counted, logged nor raised as an alert, and checks are not left alone
// after it. The ended check stays spent from the day's budget. After
// StopChecking, Hit checks no client.
func (a *AbuseIPDB) StopChecking() {
a.mu.Lock()
a.stopped = true
for _, end := range a.checking {
end()
}
a.mu.Unlock()
a.checkGoroutines.Wait()
}
// Checked returns how many checks were made.
func (a *AbuseIPDB) Checked() int {
a.mu.Lock()
@@ -344,8 +315,7 @@ func (a *AbuseIPDB) RunBlacklist(ctx context.Context) {
// check checks client with AbuseIPDB by addr, one of its addresses, keeps
// the score as client's, and notes the check as no longer under way. A
// check that fails gives no score: it is counted, logged and raised as a
// source_failure alert, and no client is checked for failureDelay. A check
// ended as ctx ends, by StopChecking, is no failure.
// source_failure alert, and no client is checked for failureDelay.
func (a *AbuseIPDB) check(
ctx context.Context,
client netip.Prefix,
@@ -353,26 +323,24 @@ func (a *AbuseIPDB) check(
) {
score, err := a.ask(ctx, addr)
now := a.params.Now()
isFailure := err != nil && ctx.Err() == nil
a.mu.Lock()
delete(a.checking, client)
switch {
case err == nil:
if err == nil {
a.scores.Add(
client,
Score{Client: client, Score: score, Fetched: now},
)
case isFailure:
} else {
a.failures++
a.retryAt = now.Add(failureDelay)
}
a.mu.Unlock()
if isFailure {
if err != nil {
a.alert("checking a client with AbuseIPDB failed", err)
}
}
-37
View File
@@ -394,43 +394,6 @@ func TestCheckNotAnsweredWithinTheTimeoutFailsAndHitNeverWaits(
})
}
func TestStopCheckingEndsTheCheckUnderWayAsNoFailureAndChecksNoMore(
t *testing.T,
) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
queue := newQueue()
p := abuseIPDBParams()
p.Alerts = queue
abuseIPDB := &abuseIPDBStandIn{hanging: true}
checker := newAbuseIPDB(abuseIPDB, p)
wantScore(t, checker, suspect, true, 0, false)
synctest.Wait()
// The check is ended, not waited out, and is no failure.
began := time.Now()
checker.StopChecking()
if waited := time.Since(began); waited != 0 {
t.Errorf("StopChecking returned after %s, want at once", waited)
}
wantFailures(t, checker, 0)
if got := waiting(queue); len(got) != 0 {
t.Errorf("alerts waiting %+v, want none", got)
}
// An offender without a score is then not checked.
wantScore(t, checker, other, true, 0, false)
synctest.Wait()
wantChecked(t, abuseIPDB, suspect)
})
}
func TestKeyIsSentInTheKeyHeaderAndNeverShown(t *testing.T) {
t.Parallel()
+17 -67
View File
@@ -16,6 +16,7 @@ import (
"github.com/hashicorp/golang-lru/v2/simplelru"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/config"
)
const (
@@ -31,8 +32,6 @@ const (
// it fails, and no client is checked with AbuseIPDB after a check
// fails, so that a source refusing them is not asked on every request.
failureDelay = time.Minute
// maskedKey is what MaskZoneKey shows in place of a zone's key.
maskedKey = "********"
)
var (
@@ -83,12 +82,8 @@ type DNSBL struct {
// verdicts are by query. Each is added as it is fetched and never moved
// up, so that the one fetched longest ago is the first dropped.
verdicts *simplelru.LRU[query, Verdict]
// asking are the queries under way, each with the function that ends
// it. stopped is true once StopAsking has been called, and
// queryGoroutines waits for the goroutine of each query.
asking map[query]context.CancelFunc
stopped bool
queryGoroutines sync.WaitGroup
// asking are the queries under way.
asking map[query]bool
// queries and failures count, by zone, the queries made and those that
// failed, and retryAt is when a zone whose last query failed may be
// asked again.
@@ -125,31 +120,13 @@ func NewDNSBL(params DNSBLParams) *DNSBL {
params: params,
resolver: resolver,
verdicts: verdicts,
asking: map[query]context.CancelFunc{},
asking: map[query]bool{},
queries: map[string]int{},
failures: map[string]int{},
retryAt: map[string]time.Time{},
}
}
// MaskZoneKey returns zone with ******** in place of its key, if it is a
// zone of Spamhaus's keyed query service, a name under dq.spamhaus.net,
// such as <key>.xbl.dq.spamhaus.net, whose first label is the key. Any
// other zone it returns as it is. A zone is shown so wherever it leaves
// the process: in the log, the alerts and the metrics.
func MaskZoneKey(zone string) string {
// DNS names ignore case, and a name may be written with a dot at its
// end.
name := strings.TrimSuffix(strings.ToLower(zone), ".")
if !strings.HasSuffix(name, ".dq.spamhaus.net") {
return zone
}
_, rest, _ := strings.Cut(zone, ".")
return maskedKey + "." + rest
}
// Zones returns the zones, in the order SWW_WAF_DNSBL_ZONES names them.
func (d *DNSBL) Zones() []string {
return slices.Clone(d.params.Zones)
@@ -157,14 +134,13 @@ func (d *DNSBL) Zones() []string {
// ListedBy returns the zones whose verdict on addr, a client's address,
// lists it, in the order SWW_WAF_DNSBL_ZONES names them, each with its key
// masked, as MaskZoneKey masks it, since they go to the request
// masked, as config.MaskZoneKey masks it, since they go to the request
// log, the alerts and the metrics. A verdict is used until CacheTTL has
// passed since it was fetched. Each zone without one is asked about addr
// in the background, unless a query about addr to it is under way, the
// zone is left alone after a failure, maxQueries are under way, or
// StopAsking has been called; ListedBy never waits for a query. ctx is the
// context of the client's request, and a query goes on after the request
// ends, until StopAsking.
// zone is left alone after a failure, or maxQueries are under way;
// ListedBy never waits for a query. ctx is the context of the client's
// request, and a query goes on after the request ends.
func (d *DNSBL) ListedBy(ctx context.Context, addr netip.Addr) []string {
d.mu.Lock()
defer d.mu.Unlock()
@@ -177,47 +153,23 @@ func (d *DNSBL) ListedBy(ctx context.Context, addr netip.Addr) []string {
q := query{zone: zone, client: addr}
kept, found := d.verdicts.Peek(q)
_, asking := d.asking[q]
switch {
case found && now.Sub(kept.Fetched) < d.params.CacheTTL:
if kept.Listed {
listedBy = append(listedBy, MaskZoneKey(zone))
listedBy = append(listedBy, config.MaskZoneKey(zone))
}
case !asking && !d.stopped && !now.Before(d.retryAt[zone]) &&
len(d.asking) < maxQueries:
queryCtx, end := context.WithCancel(context.WithoutCancel(ctx))
d.asking[q] = end
case !d.asking[q] && !now.Before(d.retryAt[zone]) && len(d.asking) < maxQueries:
d.asking[q] = true
d.queries[zone]++
d.queryGoroutines.Go(func() {
defer end()
d.ask(queryCtx, q)
})
go d.ask(context.WithoutCancel(ctx), q)
}
}
return listedBy
}
// StopAsking ends the queries under way, and returns once their
// goroutines have. A query it ends gives no verdict and is no failure: it
// is neither counted, logged nor raised as an alert, and the zone is not
// left alone after it. After StopAsking, ListedBy asks no zone.
func (d *DNSBL) StopAsking() {
d.mu.Lock()
d.stopped = true
for _, end := range d.asking {
end()
}
d.mu.Unlock()
d.queryGoroutines.Wait()
}
// Queries returns how many queries were made to zone.
func (d *DNSBL) Queries(zone string) int {
d.mu.Lock()
@@ -286,32 +238,30 @@ func (d *DNSBL) Load(verdicts []Verdict) {
// query as no longer under way. A query that fails gives no verdict: it
// is counted, logged and raised as a source_failure alert, which show the
// zone with its key masked, and the zone is not asked again for
// failureDelay. A query ended as ctx ends, by StopAsking, is no failure.
// failureDelay.
func (d *DNSBL) ask(ctx context.Context, q query) {
listed, err := d.lookUp(ctx, q)
now := d.params.Now()
isFailure := err != nil && ctx.Err() == nil
d.mu.Lock()
delete(d.asking, q)
switch {
case err == nil:
if err == nil {
d.verdicts.Add(q, Verdict{
Zone: q.zone, Client: q.client, Listed: listed, Fetched: now,
})
case isFailure:
} else {
d.failures[q.zone]++
d.retryAt[q.zone] = now.Add(failureDelay)
}
d.mu.Unlock()
if isFailure {
if err != nil {
const failed = "asking a DNSBL zone failed"
shown := MaskZoneKey(q.zone)
shown := config.MaskZoneKey(q.zone)
// Raised before it is logged, so that the alert is there once the
// log line is.
-64
View File
@@ -210,45 +210,6 @@ func TestQueryNotAnsweredWithinTheTimeoutFails(t *testing.T) {
})
}
//
//nolint:paralleltest // one at a time, as the comment at the top of this file says
func TestStopAskingEndsTheQueryUnderWayAsNoFailureAndAsksNoMore(
t *testing.T,
) {
synctest.Test(t, func(t *testing.T) {
queue := newQueue()
p := dnsblParams(zone)
p.Alerts = queue
resolver := &resolverStandIn{hanging: true}
dnsbl := newDNSBL(resolver, p)
wantZones(t, dnsbl, listed)
synctest.Wait()
// The query is ended, not waited out, and is no failure.
began := time.Now()
dnsbl.StopAsking()
if waited := time.Since(began); waited != 0 {
t.Errorf("StopAsking returned after %s, want at once", waited)
}
wantQueries(t, dnsbl, 1, 0)
if got := waiting(queue); len(got) != 0 {
t.Errorf("alerts waiting %+v, want none", got)
}
// A client without a verdict is then asked about in no zone.
wantZones(t, dnsbl, unlisted)
synctest.Wait()
wantQueries(t, dnsbl, 1, 0)
wantAsked(t, resolver, listedName)
waitForTheResolver()
})
}
//
//nolint:paralleltest // one at a time, as the comment at the top of this file says
func TestZoneThatFailsOrRefusesGivesNoVerdictAndIsLeftAloneForAMinute(
@@ -860,28 +821,3 @@ func wantAsked(t *testing.T, resolver *resolverStandIn, want ...string) {
t.Errorf("asked about %v, want %v", got, want)
}
}
func TestMaskZoneKeyMasksTheFirstLabelOfAZoneUnderDqSpamhausNet(
t *testing.T,
) {
t.Parallel()
const (
dronebl = "dnsbl.dronebl.org"
spamhaus = "abcdefghijklmnopqrstuvwxyz.xbl.dq.spamhaus.net"
spamhausMasked = "********.xbl.dq.spamhaus.net"
)
for zone, want := range map[string]string{
spamhaus: spamhausMasked,
spamhaus + ".": spamhausMasked + ".",
"KEY.ZEN.DQ.SPAMHAUS.NET": "********.ZEN.DQ.SPAMHAUS.NET",
dronebl: dronebl,
"dq.spamhaus.net": "dq.spamhaus.net",
spamhaus + ".example": spamhaus + ".example",
} {
if got := reputation.MaskZoneKey(zone); got != want {
t.Errorf("MaskZoneKey(%q) is %q, want %q", zone, got, want)
}
}
}
+11 -77
View File
@@ -7,10 +7,8 @@
// SWW_WAF_DNSBL_ZONES about clients, and keeps their verdicts, and checks
// clients with AbuseIPDB, and keeps their scores and the checks spent
// today. The state package writes all of these to files and reads them
// from there, so that a restart keeps them too. It fetches AbuseIPDB's
// blacklist, which it does not keep. Last, it reads the netblocks, AS
// numbers and percentages that both the settings and the lists hold, and
// masks the keys of DNSBL zones.
// from there, so that a restart keeps them too. Last, it fetches
// AbuseIPDB's blacklist, which it does not keep.
package reputation
import (
@@ -22,12 +20,12 @@ import (
"net/http"
"net/netip"
"slices"
"strconv"
"strings"
"sync"
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/config"
)
const (
@@ -42,28 +40,15 @@ const (
)
var (
errStatus = errors.New("the server answered")
errTooLong = errors.New("the list is longer than 16 MiB")
errNotNetblockLine = errors.New(
errStatus = errors.New("the server answered")
errTooLong = errors.New("the list is longer than 16 MiB")
errNotNetblock = errors.New(
"is not an address or a netblock, such as 192.0.2.0/24",
)
errNotASNPercent = errors.New(
"is not an AS number, : and a percentage, such as AS64496:50")
errNotNetblock = errors.New(
"is not a netblock such as 10.0.0.0/8, or an address")
errNotASN = errors.New("is not an AS number such as AS64496")
errNotPercent = errors.New(
"is not a percentage, a whole number from 0 to 100",
)
)
// TagList is an address list of SWW_WAF_TAG_LISTS: the tag it adds to the
// requests of the clients it holds, and the URL it is fetched from.
type TagList struct {
Name string
URL string
}
// List is a list as lists.json holds it: the URL it is fetched from,
// when it was last tried, the fetch failed or not, and its last good copy:
// when that was fetched, and its lines, as fetched, comment lines
@@ -83,7 +68,7 @@ type Params struct {
BlocklistURLs []string
ASNLimitPercentURL string
// TagLists are the address lists that add a tag (SWW_WAF_TAG_LISTS).
TagLists []TagList
TagLists []config.TagList
// Refresh is how long after a list was last fetched or tried it is
// fetched again (SWW_WAF_BLOCKLIST_REFRESH).
Refresh time.Duration
@@ -480,11 +465,7 @@ func parseNetblocks(lines []string) (entries, error) {
netblock, ok := parseNetblock(text)
if !ok {
return entries{}, fmt.Errorf(
"line %d %w",
i+1,
errNotNetblockLine,
)
return entries{}, fmt.Errorf("line %d %w", i+1, errNotNetblock)
}
found.netblocks[netblock] = true
@@ -504,7 +485,7 @@ func parseNetblocks(lines []string) (entries, error) {
// stands for, 192.0.2.0/24, and a mapped netblock shorter than /96, which
// stands for none, is refused.
func parseNetblock(text string) (netip.Prefix, bool) {
netblock, err := ParseNetblock(text)
netblock, err := config.ParseNetblock(text)
if err != nil {
return netip.Prefix{}, false
}
@@ -539,9 +520,9 @@ func parsePercents(lines []string) (entries, error) {
}
asnText, percentText, _ := strings.Cut(text, ":")
asn, asnErr := ParseASN(asnText)
asn, asnErr := config.ParseASN(asnText)
percent, percentErr := ParsePercent(percentText)
percent, percentErr := config.ParsePercent(percentText)
if asnErr != nil || percentErr != nil {
return entries{}, fmt.Errorf("line %d %w", i+1, errNotASNPercent)
}
@@ -555,53 +536,6 @@ func parsePercents(lines []string) (entries, error) {
return found, nil
}
// ParseNetblock reads a netblock in CIDR form, such as 10.0.0.0/8. A bare
// address is a netblock of that address alone, a /32 or a /128. A
// blocklist's lines are read with it too.
func ParseNetblock(value string) (netip.Prefix, error) {
if strings.Contains(value, "/") {
netblock, err := netip.ParsePrefix(value)
if err != nil {
return netip.Prefix{}, fmt.Errorf("%q %w", value, errNotNetblock)
}
return netblock.Masked(), nil
}
addr, err := netip.ParseAddr(value)
if err != nil || addr.Zone() != "" {
return netip.Prefix{}, fmt.Errorf("%q %w", value, errNotNetblock)
}
return netip.PrefixFrom(addr, addr.BitLen()), nil
}
// ParseASN reads an AS number such as AS64496, in either case, and
// returns it as the lookup gives it: AS and the number, in capitals and
// without leading zeros. The file SWW_WAF_ASN_LIMIT_PERCENT_URL names is
// read with it too.
func ParseASN(value string) (string, error) {
digits, hasAS := strings.CutPrefix(strings.ToUpper(value), "AS")
number, err := strconv.ParseUint(digits, 10, 32)
if !hasAS || err != nil {
return "", fmt.Errorf("%q %w", value, errNotASN)
}
return "AS" + strconv.FormatUint(number, 10), nil
}
// ParsePercent reads a percentage, a whole number from 0 to 100. The file
// SWW_WAF_ASN_LIMIT_PERCENT_URL names is read with it too.
func ParsePercent(value string) (int64, error) {
percent, err := strconv.ParseInt(value, 10, 64)
if err != nil || percent < 0 || percent > 100 {
return 0, fmt.Errorf("%q %w", value, errNotPercent)
}
return percent, nil
}
// withoutComment returns line without anything after a ; or a #, and
// without the spaces around what is left.
func withoutComment(line string) string {
+33 -4
View File
@@ -1,7 +1,7 @@
// Package requestlog writes the request log sww prints on stdout: JSON
// lines marked "type":"request", at most two from waf and a short one
// from rproxy for each request. Package processlog writes the process's
// own messages.
// Package requestlog writes the lines sww prints on stdout: one
// JSON object per request, two from waf and a short one from rproxy,
// marked "type":"request", and the process's own messages as JSON lines marked
// "type":"process".
package requestlog
import (
@@ -9,6 +9,7 @@ import (
"encoding/json"
"fmt"
"io"
"log/slog"
"time"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
@@ -299,3 +300,31 @@ func Milliseconds(d time.Duration) float64 {
time.Millisecond/time.Microsecond,
)
}
// NewProcessLogger returns the logger for the process's own messages:
// JSON lines on w, marked "type":"process", with the time in the same form
// as a request line's, and instanceName, SWW_WAF_INSTANCE_NAME, as instance.
// It writes only the messages at level, SWW_WAF_LOG_LEVEL, or more severe;
// the request lines Write writes are never held back.
func NewProcessLogger(
w io.Writer,
instanceName string,
level slog.Level,
) *slog.Logger {
handler := slog.NewJSONHandler(w, &slog.HandlerOptions{
Level: level,
ReplaceAttr: func(groups []string, attr slog.Attr) slog.Attr {
if attr.Key == slog.TimeKey && len(groups) == 0 {
return slog.String(
slog.TimeKey,
FormatTime(attr.Value.Time()),
)
}
return attr
},
})
return slog.New(handler).
With("type", "process", "instance", instanceName)
}
+87
View File
@@ -3,6 +3,8 @@ package requestlog_test
import (
"bytes"
"encoding/json"
"log/slog"
"slices"
"strings"
"testing"
"time"
@@ -132,3 +134,88 @@ func TestWriteRProxyLineWritesOneJSONLineWithOnlyItsFields(t *testing.T) {
t.Errorf("wrote %s, want %s", out.String(), want)
}
}
func TestProcessLinesAreMarkedProcessAndGiveTheInstance(t *testing.T) {
t.Parallel()
var out bytes.Buffer
requestlog.NewProcessLogger(&out, "host1/blog", slog.LevelInfo).
Info("starting",
"version", "v1")
var fields map[string]any
err := json.Unmarshal(out.Bytes(), &fields)
if err != nil {
t.Fatalf("decode %q: %v", out.String(), err)
}
if fields["type"] != "process" ||
fields["instance"] != "host1/blog" ||
fields["msg"] != "starting" ||
fields["level"] != "INFO" ||
fields["version"] != "v1" {
t.Errorf("process line %v", fields)
}
timeText, _ := fields["time"].(string)
logged, err := time.Parse(time.RFC3339, timeText)
if err != nil || !strings.HasSuffix(timeText, "Z") ||
len(timeText) != len("2006-01-02T15:04:05.000Z") ||
time.Since(logged) > time.Minute {
t.Errorf(
"process line time %q, want now in UTC with milliseconds",
timeText,
)
}
}
func TestProcessLoggerWritesTheMessagesAtItsLevelOrMoreSevere(t *testing.T) {
t.Parallel()
levels := []slog.Level{
slog.LevelDebug, slog.LevelInfo, slog.LevelWarn, slog.LevelError,
}
for i, level := range levels {
t.Run(level.String(), func(t *testing.T) {
t.Parallel()
var out bytes.Buffer
processLog := requestlog.NewProcessLogger(
&out,
"host1/blog",
level,
)
for _, at := range levels {
processLog.Log(t.Context(), at, "message")
}
var got, want []string
for line := range strings.Lines(out.String()) {
var fields struct {
Level string `json:"level"`
}
err := json.Unmarshal([]byte(line), &fields)
if err != nil {
t.Fatalf("decode %q: %v", line, err)
}
got = append(got, fields.Level)
}
for _, written := range levels[i:] {
want = append(want, written.String())
}
if !slices.Equal(got, want) {
t.Errorf("lines at %v, want %v", got, want)
}
})
}
}
+12 -43
View File
@@ -99,12 +99,8 @@ type Client struct {
// answers are by client. Each is added as it comes and never moved up,
// so that the one that came longest ago is the first dropped.
answers *simplelru.LRU[netip.Prefix, Answer]
// asking are the clients whose query is under way, each with the
// function that ends it. stopped is true once StopAsking has been
// called, and queryGoroutines waits for the goroutine of each query.
asking map[netip.Prefix]context.CancelFunc
stopped bool
queryGoroutines sync.WaitGroup
// asking are the clients whose query is under way.
asking map[netip.Prefix]bool
// retryAt is when routewatch may be asked again after a query failed.
retryAt time.Time
}
@@ -120,7 +116,7 @@ func New(params Params) *Client {
params: params,
httpClient: &http.Client{},
answers: answers,
asking: map[netip.Prefix]context.CancelFunc{},
asking: map[netip.Prefix]bool{},
}
}
@@ -128,9 +124,8 @@ func New(params Params) *Client {
// IPv6 group, or why it has no route. It never waits for a query. A client
// without an answer is asked about in the background, by addr, the address its
// request came from, unless its query is under way, a query failed less than
// failureKept ago, maxQueries are under way, or StopAsking has been called. ctx
// is the context of the client's request, and a query goes on after the request
// ends, until StopAsking.
// failureKept ago, or maxQueries are under way. ctx is the context of the
// client's request, and a query goes on after the request ends.
func (c *Client) Route(
ctx context.Context, client netip.Prefix, addr netip.Addr,
) (Route, Missing) {
@@ -152,38 +147,15 @@ func (c *Client) Route(
return Route{}, RequestFailed
}
_, asking := c.asking[client]
if !asking && !c.stopped && len(c.asking) < maxQueries {
queryCtx, end := context.WithCancel(context.WithoutCancel(ctx))
c.asking[client] = end
if !c.asking[client] && len(c.asking) < maxQueries {
c.asking[client] = true
c.queryGoroutines.Go(func() {
defer end()
c.ask(queryCtx, client, addr)
})
go c.ask(context.WithoutCancel(ctx), client, addr)
}
return Route{}, NoAnswerYet
}
// StopAsking ends the queries under way, and returns once their goroutines
// have. A query it ends gives no answer and is no failure: it is not logged,
// and routewatch is not left alone after it. After StopAsking, Route asks
// routewatch nothing.
func (c *Client) StopAsking() {
c.mu.Lock()
c.stopped = true
for _, end := range c.asking {
end()
}
c.mu.Unlock()
c.queryGoroutines.Wait()
}
// Snapshot returns every answer still in use, sorted by client, as
// routewatch.json lists them.
func (c *Client) Snapshot() []Answer {
@@ -233,8 +205,7 @@ func (c *Client) Load(answers []Answer) {
// ask asks routewatch about addr, keeps the answer as client's, and notes the
// query as no longer under way. A query that fails gives no answer, and
// routewatch is not asked again for failureKept. Only the first failure of
// those under way is logged. A query ended as ctx ends, by StopAsking, is no
// failure.
// those under way is logged.
func (c *Client) ask(
ctx context.Context,
client netip.Prefix,
@@ -242,20 +213,18 @@ func (c *Client) ask(
) {
route, err := c.query(ctx, addr)
now := c.params.Now()
isFailure := err != nil && ctx.Err() == nil
c.mu.Lock()
delete(c.asking, client)
firstFailure := isFailure && !now.Before(c.retryAt)
firstFailure := err != nil && !now.Before(c.retryAt)
switch {
case err == nil:
if err == nil {
c.answers.Add(client, Answer{
Client: client, Netblock: route.Netblock, ASN: route.ASN, Answered: now,
})
case isFailure:
} else {
c.retryAt = now.Add(failureKept)
}
-48
View File
@@ -361,54 +361,6 @@ func TestHangingRoutewatchNeverDelaysAndFailsTheQuery(t *testing.T) {
})
}
func TestStopAskingEndsTheQueryUnderWayAsNoFailureAndAsksNoMore(
t *testing.T,
) {
t.Parallel()
// In a synctest bubble, as TestHangingRoutewatchNeverDelaysAndFailsTheQuery
// says.
synctest.Test(t, func(t *testing.T) {
f := &fixture{t: t, log: &syncBuffer{}}
f.client = routewatch.New(routewatch.Params{
URL: silentURL, Now: time.Now,
ProcessLog: slog.New(slog.NewTextHandler(f.log, nil)),
})
f.route()
synctest.Wait()
// The query is ended, not waited out.
began := time.Now()
f.client.StopAsking()
if waited := time.Since(began); waited != 0 {
t.Errorf("StopAsking returned after %s, want at once", waited)
}
// A query started now would fail within waitLimit. None is, and the
// one ended is no failure: nothing is logged, and routewatch is not
// left alone.
for range 2 {
if _, missing := f.route(); missing != routewatch.NoAnswerYet {
t.Errorf(
"Route gave %q, want %q",
missing,
routewatch.NoAnswerYet,
)
}
time.Sleep(waitLimit)
synctest.Wait()
}
if f.log.String() != "" {
t.Errorf("process log %q, want nothing", f.log.String())
}
})
}
func TestRepeatedRequestsMakeOneQueryAndTheAnswerExpires(t *testing.T) {
t.Parallel()
+7 -16
View File
@@ -2,7 +2,6 @@ package rproxy
import (
"bufio"
"context"
"errors"
"io"
"net"
@@ -23,9 +22,7 @@ type requestBody struct {
// body is the visitor's body, ending in an *http.MaxBytesError past
// SWW_RPROXY_MAX_REQUEST_SIZE.
body io.ReadCloser
// refuse refuses the request with a status, as the request's own refuse
// does.
refuse func(status int)
rq *request
// held is the part read before the request goes to the app, which Read gives
// first.
held []byte
@@ -71,7 +68,7 @@ func (b *requestBody) Read(p []byte) (int, error) {
case errors.Is(err, io.EOF):
b.received.Store(true)
case errors.As(err, &tooLarge):
b.refuse(http.StatusRequestEntityTooLarge)
b.rq.refuse(http.StatusRequestEntityTooLarge)
}
return n, err
@@ -89,13 +86,7 @@ type responseBody struct {
// body is the app's body, ending in an *http.MaxBytesError past
// SWW_RPROXY_MAX_RESPONSE_SIZE.
body io.ReadCloser
// refuse refuses the request with a status, as the request's own refuse
// does.
refuse func(status int)
// responseReceived is called once the app's whole answer has arrived.
responseReceived func()
// ctx is the visitor's request's context.
ctx context.Context //nolint:containedctx // the body ends with the request
rq *request
}
// Read reads from the app's body.
@@ -109,15 +100,15 @@ func (b *responseBody) Read(p []byte) (int, error) {
switch {
case errors.Is(err, io.EOF):
b.responseReceived()
b.rq.responseReceived()
case errors.As(err, &tooLarge):
b.refuse(http.StatusBadGateway)
b.rq.refuse(http.StatusBadGateway)
return n, errResponseTooLarge
case b.ctx.Err() == nil:
case b.rq.in.Context().Err() == nil:
// The answer broke off, not because the visitor went away. If a time
// limit cut it, that refusal came first and is the one kept.
b.refuse(http.StatusBadGateway)
b.rq.refuse(http.StatusBadGateway)
}
return n, err
+10
View File
@@ -7,6 +7,16 @@ import (
"strings"
)
// The headers rproxy sets on the requests it passes to the app. Go writes every
// header name in this form, as it sends it and as it receives it, so
// X-Client-ASN arrives as X-Client-Asn, and Del removes a visitor's own whatever
// their case; header names are not case-sensitive.
const (
requestIDHeader = "X-Request-ID"
asnHeader = "X-Client-Asn"
countryHeader = "X-Client-Country"
)
// peerAddress is the address of the request's TCP peer, normally traefik, without
// any IPv6 zone, which waf refuses.
func peerAddress(r *http.Request) netip.Addr {
-6
View File
@@ -2,12 +2,6 @@ package rproxy
import "time"
// StopBreakerClock has the circuit breaker of s read at as the time, so that how
// long requests take cannot matter to it. It is called before s serves.
func (s *Server) StopBreakerClock(at time.Time) {
s.h.breakerClock = func() time.Time { return at }
}
// Breaker is the circuit breaker, for tests that give it the time.
type Breaker struct {
b breaker
+13 -22
View File
@@ -18,9 +18,9 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/processlog"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rproxy"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// post is the start of a POST whose headers end with what follows it.
@@ -51,11 +51,11 @@ func TestServerHasTheFixedLimits(t *testing.T) {
server := rproxy.New(rproxy.Params{
Config: cfg,
RequestLog: io.Discard,
ProcessLog: processlog.New(processlog.Params{
Output: io.Discard,
InstanceName: "test",
Level: slog.LevelInfo,
}),
ProcessLog: requestlog.NewProcessLogger(
io.Discard,
"test",
slog.LevelInfo,
),
})
if server.Addr != ":8080" || server.MaxHeaderBytes != 28<<10 ||
@@ -492,7 +492,7 @@ func TestASettingOfOffSetsNoLimit(t *testing.T) {
strings.NewReader("a body"))),
)
if got := get(t, addr, wafprotocol.HealthPath); got.status != http.StatusOK {
if got := get(t, addr, waf.HealthPath); got.status != http.StatusOK {
t.Errorf("answered %+v, want rproxy's own 200", got)
}
})
@@ -870,14 +870,10 @@ func TestAnAnswerRProxyGivesItselfHasTenSecondsToBeSentAtLeast(
func TestAVisitorTooSlowToSendTheRestIsAnswered408(t *testing.T) {
t.Parallel()
// rproxy is waiting for the rest of the body well within this limit, even on
// a busy host, and answers when it runs out, well before dial's deadline.
const limit = 5 * time.Second
standIn := startWAF(t, allow)
app := startApp(t, greet)
addr, _ := startRProxy(t, standIn.URL, app.URL, map[string]string{
maxRequestTime: strconv.FormatInt(limit.Milliseconds(), 10),
maxRequestTime: "3000",
})
conn := dial(t, addr)
@@ -1266,9 +1262,7 @@ func TestAVisitorThatStopsALittleShortIsReadUntilTheRequestTimeLimit(
) {
t.Parallel()
// The app's answer arrives well within this limit, even on a busy host, and
// the connection closes when it runs out, well before dial's deadline.
const limit = 5 * time.Second
const limit = 2 * time.Second
standIn := startWAF(t, allow)
app := startApp(t, answerBeforeTheBody)
@@ -1433,20 +1427,17 @@ func TestTheRequestTimeLimitDoesNotCutAnAnswerOnceTheAppHasTheRequest(
const answer = "the end of a long answer"
// The request reaches the app well within this limit, even on a busy host.
const limit = 5 * time.Second
standIn := startWAF(t, allow)
app := startApp(t, func(w http.ResponseWriter, r *http.Request) {
_, _ = io.Copy(io.Discard, r.Body)
_ = http.NewResponseController(w).Flush()
time.Sleep(limit + 100*time.Millisecond)
time.Sleep(2100 * time.Millisecond)
_, _ = io.WriteString(w, answer)
})
addr, _ := startRProxy(t, standIn.URL, app.URL, map[string]string{
maxRequestTime: strconv.FormatInt(limit.Milliseconds(), 10),
maxRequestTime: "2000",
})
got := visit(
@@ -1480,7 +1471,7 @@ func TestAVisitorThatStopsALittleShortGetsAnAnswerRProxyGivesAtOnce(
{"a deny", post, http.StatusForbidden},
{
"the health check",
"GET " + wafprotocol.HealthPath + " HTTP/1.1\r\nHost: site.example\r\n",
"GET " + waf.HealthPath + " HTTP/1.1\r\nHost: site.example\r\n",
http.StatusOK,
},
} {
+13 -13
View File
@@ -12,11 +12,11 @@ import (
"strings"
"testing"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// questionBodyMaxBytes is the most of a body a question carries.
const questionBodyMaxBytes = wafprotocol.QuestionBodyMaxBytes
const questionBodyMaxBytes = waf.QuestionBodyMaxBytes
// chunked is the encoding of a body whose length the visitor did not announce.
const chunked = "chunked"
@@ -42,9 +42,9 @@ func TestQuestionIsTheVisitorsRequestWithRProxysHeaders(t *testing.T) {
req.Header.Set(forwardedForHeader, "198.51.100.1")
req.Header.Set(forwardedProtoHeader, "gopher")
req.Header.Set(requestIDHeader, "the-visitor's")
req.Header.Set(wafprotocol.ClientHeader, "198.51.100.2")
req.Header.Set(wafprotocol.PeerHeader, "198.51.100.3")
req.Header.Set(wafprotocol.ContentLengthHeader, "12345")
req.Header.Set(waf.ClientHeader, "198.51.100.2")
req.Header.Set(waf.PeerHeader, "198.51.100.3")
req.Header.Set(waf.ContentLengthHeader, "12345")
// Go's client then sends no User-Agent, so that one in the question would be
// Go's own.
req.Header["User-Agent"] = nil
@@ -73,8 +73,8 @@ func TestQuestionIsTheVisitorsRequestWithRProxysHeaders(t *testing.T) {
want.Set(forwardedForHeader, "198.51.100.1")
want.Set(forwardedProtoHeader, "http")
want.Set(requestIDHeader, asked.header.Get(requestIDHeader))
want.Set(wafprotocol.ClientHeader, localhost)
want.Set(wafprotocol.PeerHeader, localhost)
want.Set(waf.ClientHeader, localhost)
want.Set(waf.PeerHeader, localhost)
if !maps.EqualFunc(asked.header, want, slices.Equal) {
t.Errorf(
@@ -124,8 +124,8 @@ func TestQuestionNamesTheVisitorATrustedProxySpeaksFor(t *testing.T) {
wantGreeting(t, visit(t, req))
asked := standIn.question(t)
if asked.header.Get(wafprotocol.ClientHeader) != tc.client ||
asked.header.Get(wafprotocol.PeerHeader) != localhost ||
if asked.header.Get(waf.ClientHeader) != tc.client ||
asked.header.Get(waf.PeerHeader) != localhost ||
asked.header.Get(forwardedForHeader) != visitor ||
asked.header.Get(forwardedProtoHeader) != tc.proto {
t.Errorf("the question's headers are %v", asked.header)
@@ -191,7 +191,7 @@ func TestQuestionNamesTheClientAsTheForwardedAddressesSay(t *testing.T) {
req.Header[forwardedForHeader] = tc.header
wantGreeting(t, visit(t, req))
if got := standIn.question(t).header.Get(wafprotocol.ClientHeader); got !=
if got := standIn.question(t).header.Get(waf.ClientHeader); got !=
tc.client {
t.Errorf("the question names %s, want %s", got, tc.client)
}
@@ -282,7 +282,7 @@ func wantQuestionBody(
)
}
values := asked.header.Values(wafprotocol.ContentLengthHeader)
values := asked.header.Values(waf.ContentLengthHeader)
if announced {
if !slices.Equal(values, []string{strconv.Itoa(size)}) ||
@@ -586,8 +586,8 @@ func TestAQuestionNamesNoZone(t *testing.T) {
server.Handler.ServeHTTP(httptest.NewRecorder(), req)
asked := standIn.question(t)
if asked.header.Get(wafprotocol.ClientHeader) != tc.client ||
asked.header.Get(wafprotocol.PeerHeader) != tc.peer {
if asked.header.Get(waf.ClientHeader) != tc.client ||
asked.header.Get(waf.PeerHeader) != tc.peer {
t.Errorf("the question's headers are %v", asked.header)
}
+16 -19
View File
@@ -18,13 +18,17 @@ import (
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// flushAfterEachWrite has ReverseProxy pass on each part of the app's answer as
// soon as it arrives.
const flushAfterEachWrite time.Duration = -1
// reportStatusMax is the highest status waf takes in a report. An app can
// send a higher one, up to 999, and the visitor gets that as it is.
const reportStatusMax = 599
// plainText is the Content-Type of the answers rproxy gives itself.
const plainText = "text/plain; charset=utf-8"
@@ -122,8 +126,8 @@ func (h *handler) newRequest(
if r.Body != http.NoBody {
rq.body = &requestBody{
body: limitBody(r.Body, h.config.MaxRequestSize),
refuse: rq.refuse,
body: limitBody(r.Body, h.config.MaxRequestSize),
rq: rq,
}
}
@@ -150,7 +154,7 @@ func (rq *request) readFirstPartOfBody() bool {
_ = rq.rc.SetReadDeadline(rq.requestDeadline())
held, err := io.ReadAll(
io.LimitReader(rq.body, wafprotocol.QuestionBodyMaxBytes),
io.LimitReader(rq.body, waf.QuestionBodyMaxBytes),
)
// From here on, the request timer ends a read of the rest of the body, so that
// the answer can name the side rproxy was waiting on.
@@ -224,17 +228,16 @@ func (rq *request) rewrite(pr *httputil.ProxyRequest) {
setPathAsSent(pr.Out.URL, pr.In)
setForwardedHeaders(pr.In, pr.Out, rq.peer, rq.peerTrusted)
pr.Out.Header.Set(wafprotocol.RequestIDHeader, rq.id)
// Del removes a visitor's own, whatever their case.
pr.Out.Header.Del(wafprotocol.ASNHeader)
pr.Out.Header.Del(wafprotocol.CountryHeader)
pr.Out.Header.Set(requestIDHeader, rq.id)
pr.Out.Header.Del(asnHeader)
pr.Out.Header.Del(countryHeader)
if rq.asn != "" {
pr.Out.Header.Set(wafprotocol.ASNHeader, rq.asn)
pr.Out.Header.Set(asnHeader, rq.asn)
}
if rq.country != "" {
pr.Out.Header.Set(wafprotocol.CountryHeader, rq.country)
pr.Out.Header.Set(countryHeader, rq.country)
}
}
@@ -291,12 +294,7 @@ func (rq *request) modifyResponse(res *http.Response) error {
// the request, and SWW_RPROXY_MAX_RESPONSE_TIME_MS starts.
rq.startResponseTimer(requestEnd)
res.Body = &responseBody{
body: limitBody(res.Body, maxBytes),
refuse: rq.refuse,
responseReceived: rq.responseReceived,
ctx: rq.in.Context(),
}
res.Body = &responseBody{body: limitBody(res.Body, maxBytes), rq: rq}
rq.out.afterStatus = rq.setResponseDeadline
return nil
@@ -472,10 +470,9 @@ func (rq *request) finish(ctx context.Context) {
return
}
// An app can send a status up to 999, and the visitor gets it as it is.
report := &wafprotocol.Report{
report := &waf.Report{
RequestID: rq.id,
Status: min(rq.out.status, wafprotocol.ReportStatusMax),
Status: min(rq.out.status, reportStatusMax),
RequestBytes: requestBytes,
ResponseBytes: responseBytes,
DurationTotalMS: requestlog.Milliseconds(end.Sub(rq.start)),
+14 -15
View File
@@ -24,7 +24,7 @@ import (
"time"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// How rproxy keeps connections to the app and to waf open between
@@ -47,6 +47,10 @@ const (
idleTimeout = 120 * time.Second
)
// ownEndpointPrefix starts the path of every request for sww itself, which
// never reaches the app.
const ownEndpointPrefix = "/_sww/"
// allowedMethods lists the methods rproxy passes on, as an Allow header gives
// them. It answers any other method 405 itself.
// The WebDAV ones are for Nextcloud.
@@ -88,11 +92,8 @@ func New(params Params) *Server {
transport: newTransport(),
wafTransport: newWAFTransport(),
reportURL: (&url.URL{
Scheme: backend.Scheme,
Host: backend.Host,
Path: wafprotocol.ReportPath,
Scheme: backend.Scheme, Host: backend.Host, Path: waf.ReportPath,
}).String(),
breakerClock: time.Now,
}
return &Server{
@@ -132,10 +133,8 @@ type handler struct {
wafTransport http.RoundTripper
// reportURL is where the reports go.
reportURL string
// breaker is the circuit breaker for asks to waf, and breakerClock gives it
// the time.
breaker breaker
breakerClock func() time.Time
// breaker is the circuit breaker for asks to waf.
breaker breaker
// reports waits for the reports being sent. reportsMu guards starting one
// against reportsStopped, which is set once nothing is to wait for another.
reports sync.WaitGroup
@@ -177,7 +176,7 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
defer rq.readRestOfBody()
defer rq.finish(r.Context())
if r.Method == http.MethodGet && r.URL.Path == wafprotocol.HealthPath {
if r.Method == http.MethodGet && r.URL.Path == waf.HealthPath {
rq.answerHealth()
return
@@ -210,9 +209,9 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
rq.noteAnswer(answer)
switch rq.disposition {
case wafprotocol.DispositionAllow:
case waf.DispositionAllow:
rq.passToApp(r.Context())
case wafprotocol.DispositionClose:
case waf.DispositionClose:
// This panic has Go's server close the connection without an answer,
// and log nothing; the deferred finish still runs.
panic(http.ErrAbortHandler)
@@ -222,8 +221,8 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
}
// passToApp passes the request on to the app, unless a limit or its path keeps
// it from the app. waf answers a path under wafprotocol.OwnEndpointPrefix
// itself, so such a request gets here only when
// it from the app. waf answers a path under ownEndpointPrefix itself, so
// such a request gets here only when
// SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION allowed it.
// ctx is the request's own context.
func (rq *request) passToApp(ctx context.Context) {
@@ -232,7 +231,7 @@ func (rq *request) passToApp(ctx context.Context) {
switch {
case rq.refused.Load() != 0 || (maxBytes > 0 && rq.in.ContentLength > maxBytes):
rq.answer(http.StatusRequestEntityTooLarge)
case strings.HasPrefix(rq.in.URL.Path, wafprotocol.OwnEndpointPrefix):
case strings.HasPrefix(rq.in.URL.Path, ownEndpointPrefix):
rq.answer(http.StatusNotFound)
default:
rq.forward(ctx)
+39 -72
View File
@@ -21,9 +21,9 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/processlog"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rproxy"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
const (
@@ -56,7 +56,7 @@ const (
// The headers the tests set and look at.
const (
dispositionHeader = wafprotocol.DispositionHeader
dispositionHeader = waf.DispositionHeader
forwardedForHeader = "X-Forwarded-For"
forwardedProtoHeader = "X-Forwarded-Proto"
requestIDHeader = "X-Request-ID"
@@ -72,9 +72,9 @@ const (
// What the stand-in for waf answers.
const (
dispositionAllow = wafprotocol.DispositionAllow
dispositionDeny = wafprotocol.DispositionDeny
dispositionClose = wafprotocol.DispositionClose
dispositionAllow = waf.DispositionAllow
dispositionDeny = waf.DispositionDeny
dispositionClose = waf.DispositionClose
// deniedBody is the body of a deny.
deniedBody = "refused by waf"
)
@@ -175,7 +175,7 @@ type WAF struct {
*httptest.Server
questions chan question
reports chan wafprotocol.Report
reports chan waf.Report
}
// startWAF starts a stand-in for waf.
@@ -184,11 +184,11 @@ func startWAF(t *testing.T, answer http.HandlerFunc) *WAF {
s := &WAF{
questions: make(chan question, 10),
reports: make(chan wafprotocol.Report, 10),
reports: make(chan waf.Report, 10),
}
s.Server = httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(wafprotocol.ClientHeader) == "" {
if r.Header.Get(waf.ClientHeader) == "" {
s.takeReport(t, w, r)
return
@@ -217,11 +217,11 @@ func (s *WAF) takeReport(
) {
t.Helper()
var report wafprotocol.Report
var report waf.Report
err := json.NewDecoder(r.Body).Decode(&report)
if err != nil || r.Method != http.MethodPost ||
r.URL.Path != wafprotocol.ReportPath {
r.URL.Path != waf.ReportPath {
t.Errorf("%s %s is no report: %v", r.Method, r.URL.Path, err)
}
@@ -242,7 +242,7 @@ func (s *WAF) wantNoReport(t *testing.T) {
}
// report waits for a report.
func (s *WAF) report(t *testing.T) wafprotocol.Report {
func (s *WAF) report(t *testing.T) waf.Report {
t.Helper()
select {
@@ -251,7 +251,7 @@ func (s *WAF) report(t *testing.T) wafprotocol.Report {
case <-time.After(waitLimit):
t.Fatal("no report")
return wafprotocol.Report{}
return waf.Report{}
}
}
@@ -279,7 +279,7 @@ func startWAFAnsweringReports(
backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(wafprotocol.ClientHeader) != "" {
if r.Header.Get(waf.ClientHeader) != "" {
answer(w, r)
return
@@ -356,19 +356,6 @@ func startServerLogging(
) (string, *rproxy.Server) {
t.Helper()
server := newServer(t, backend, appURL, env, requestLog, processLog)
return serve(t, server), server
}
// newServer returns rproxy, not yet serving, with its request lines going to
// requestLog and its process lines to processLog.
func newServer(
t *testing.T, backend, appURL string, env map[string]string,
requestLog, processLog io.Writer,
) *rproxy.Server {
t.Helper()
settings := map[string]string{
"SWW_RPROXY_WAF_BACKEND": backend, "SWW_RPROXY_UPSTREAM_URL": appURL,
wafBackendTimeout: "60000",
@@ -380,20 +367,15 @@ func newServer(
t.Fatalf("settings %v: %v", settings, err)
}
return rproxy.New(rproxy.Params{
server := rproxy.New(rproxy.Params{
Config: cfg,
RequestLog: requestLog,
ProcessLog: processlog.New(processlog.Params{
Output: processLog,
InstanceName: "test",
Level: slog.LevelInfo,
}),
ProcessLog: requestlog.NewProcessLogger(
processLog,
"test",
slog.LevelInfo,
),
})
}
// serve has server serve until the test ends, and returns its address.
func serve(t *testing.T, server *rproxy.Server) string {
t.Helper()
listener, err := (&net.ListenConfig{}).Listen(
t.Context(),
@@ -414,7 +396,7 @@ func serve(t *testing.T, server *rproxy.Server) string {
server.WaitForReports()
})
return listener.Addr().String()
return listener.Addr().String(), server
}
// answered is the visitor's answer.
@@ -829,7 +811,7 @@ func TestAnAnswerRProxyHoldsWholeIsSentBeforeItsLogLine(t *testing.T) {
}{
{"a deny", "GET /" + host, deniedBody, http.StatusForbidden},
{"rproxy's own answer", "GET x:admin" + host, "Bad Request\n", http.StatusBadRequest},
{"the health check", "GET " + wafprotocol.HealthPath + host, "ok\n", http.StatusOK},
{"the health check", "GET " + waf.HealthPath + host, "ok\n", http.StatusOK},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
@@ -1452,18 +1434,14 @@ func TestFiveFailedAsksKeepTheNextRequestFromAskingWAF(t *testing.T) {
greet(w, r)
})
out := &output{}
server := newServer(
t, standIn.URL, app.URL,
map[string]string{wafBackendTimeout: "200"},
out, out,
addr, out := startRProxy(
t,
standIn.URL,
app.URL,
map[string]string{
wafBackendTimeout: "200",
},
)
// The breaker's clock stands still, as in
// TestAnswersThatCannotBePassedOnOpenTheBreaker: on a slow host, the five
// failures could take more than five seconds, and the next request could
// come more than a second later.
server.StopBreakerClock(time.Now())
addr := serve(t, server)
// An ask can time out before it reaches the stand-in, so these
// may leave fewer than five questions, or leave some late.
@@ -1554,13 +1532,9 @@ func TestAnswersThatCannotBePassedOnOpenTheBreaker(t *testing.T) {
tooLarge(w, r)
})
app := startApp(t, greet)
out := &output{}
server := newServer(t, standIn.URL, app.URL, nil, out, out)
// The breaker's clock stands still. On a slow host, the five failures could
// otherwise take more than the five seconds that open the breaker. The next
// request could come more than a second later, when it may ask waf again.
server.StopBreakerClock(time.Now())
addr := serve(t, server)
addr, out := startRProxy(t, standIn.URL, app.URL, map[string]string{
wafBackendTimeout: "60000",
})
for range 5 {
if got := get(t, addr, "/"); got.status != http.StatusBadGateway {
@@ -1602,19 +1576,19 @@ func TestHealthCheckIsAnsweredWhileWAFIsDown(t *testing.T) {
app := startApp(t, greet)
addr, out := startRProxy(t, nowhere, app.URL, nil)
got := get(t, addr, wafprotocol.HealthPath)
got := get(t, addr, waf.HealthPath)
if got.status != http.StatusOK || got.body != "ok\n" {
t.Errorf("answered %+v, want ok", got)
}
line := out.requestLine(t)
if line["path"] != wafprotocol.HealthPath || line["disposition"] != nil {
if line["path"] != waf.HealthPath || line["disposition"] != nil {
t.Errorf("line %v, want one with no disposition", line)
}
// With waf up too, it is not asked.
addr, _ = startRProxy(t, standIn.URL, app.URL, nil)
get(t, addr, wafprotocol.HealthPath)
get(t, addr, waf.HealthPath)
if len(standIn.questions) != 0 {
t.Error("waf was asked about the health check")
@@ -1775,7 +1749,7 @@ func TestWaitForReportsWaitsForAReportBeingSent(t *testing.T) {
release := make(chan struct{})
backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(wafprotocol.ClientHeader) != "" {
if r.Header.Get(waf.ClientHeader) != "" {
allow(w, r)
return
@@ -1835,15 +1809,11 @@ func TestWaitForReportsWaitsForAReportBeingSent(t *testing.T) {
func TestReportOfAWebSocketCountsWhatItCarried(t *testing.T) {
t.Parallel()
// The app's answer to the upgrade arrives well within these limits, even on a
// busy host.
const limit = 5 * time.Second
standIn := startWAF(t, allow)
app := startApp(t, echoAfterUpgrade)
addr, out := startRProxy(t, standIn.URL, app.URL, map[string]string{
maxRequestTime: strconv.FormatInt(limit.Milliseconds(), 10),
maxResponseTime: strconv.FormatInt(limit.Milliseconds(), 10),
maxRequestTime: "2000",
maxResponseTime: "2000",
})
conn := dial(t, addr)
@@ -1868,10 +1838,7 @@ func TestReportOfAWebSocketCountsWhatItCarried(t *testing.T) {
}
// Past both time limits, the connection still carries what is sent.
time.Sleep(limit + 100*time.Millisecond)
// dial's deadline ran from before the upgrade and this wait.
_ = conn.SetDeadline(time.Now().Add(waitLimit))
time.Sleep(2100 * time.Millisecond)
const message = "still here\n"
+85 -36
View File
@@ -1,16 +1,18 @@
package rproxy
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"maps"
"net/http"
"strings"
"time"
"sneak.berlin/go/smallwebwaf/internal/rules"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// answerBodyMaxBytes is the most of the body of waf's answer that is read.
@@ -39,12 +41,15 @@ type wafAnswer struct {
// notes nothing, and no report is due for it: waf holds no decision to report
// on.
func (rq *request) noteAnswer(answer *wafAnswer) {
read, valid := wafprotocol.ReadAnswer(answer.header)
if valid {
rq.disposition = read.Disposition
disposition := answer.header.Get(waf.DispositionHeader)
switch disposition {
case waf.DispositionAllow, waf.DispositionDeny,
waf.DispositionClose:
rq.disposition = disposition
rq.answered = true
rq.asn = read.ASN
rq.country = read.Country
rq.asn = answer.header.Get(asnHeader)
rq.country = answer.header.Get(countryHeader)
}
}
@@ -75,7 +80,7 @@ func (rq *request) handleNoUsableAnswer(ctx context.Context, err error) {
rq.disposition = rq.h.config.WAFBackendNoAnswerDisposition
rq.wafNoAnswer = true
if rq.disposition == wafprotocol.DispositionAllow {
if rq.disposition == waf.DispositionAllow {
rq.passToApp(ctx)
} else {
rq.answer(http.StatusServiceUnavailable)
@@ -92,7 +97,7 @@ func (rq *request) askUnlessBreakerOpen(
ctx context.Context,
) (*wafAnswer, error) {
b := &rq.h.breaker
if !b.mayAsk(rq.h.breakerClock()) {
if !b.mayAsk(time.Now()) {
return nil, errBreakerOpen
}
@@ -102,7 +107,7 @@ func (rq *request) askUnlessBreakerOpen(
case err == nil:
b.noteAnswer()
case ctx.Err() == nil:
b.noteFailure(rq.h.breakerClock())
b.noteFailure(time.Now())
}
return answer, err
@@ -115,21 +120,7 @@ func (rq *request) ask(ctx context.Context) (*wafAnswer, error) {
ctx, cancel := context.WithTimeout(ctx, rq.h.config.WAFBackendTimeout)
defer cancel()
var held []byte
if rq.body != nil {
held = rq.body.held
}
question, err := wafprotocol.NewQuestion(ctx, wafprotocol.QuestionParams{
Visitor: rq.in,
PathAsSent: rules.Path(rq.in),
Body: held,
WAFBackend: rq.h.config.WAFBackend,
Client: rq.client,
Peer: rq.peer,
RequestID: rq.id,
Scheme: scheme(rq.in, rq.peerTrusted),
})
question, err := rq.newQuestion(ctx)
if err != nil {
return nil, err
}
@@ -186,13 +177,69 @@ func canBeAsked(r *http.Request) bool {
len(r.Header.Values("User-Agent")) <= 1
}
// newQuestion returns the question about the request, as "The question" in
// SPEC.md gives it: the visitor's request as the visitor sent it, with the part
// of the body it holds and rproxy's headers.
func (rq *request) newQuestion(ctx context.Context) (*http.Request, error) {
in := rq.in
var held []byte
if rq.body != nil {
held = rq.body.held
}
question, err := http.NewRequestWithContext( //nolint:gosec // a setting's URL
ctx,
in.Method,
rq.h.config.WAFBackend.String(),
bytes.NewReader(held),
)
if err != nil {
return nil, fmt.Errorf("make the question: %w", err)
}
backend := question.URL
target := *in.URL
target.Scheme = backend.Scheme
target.Host = backend.Host
setPathAsSent(&target, in)
question.URL = &target
question.Host = in.Host
// The length NewRequestWithContext set is that of what is held. This line makes
// a body whose length the visitor did not announce go chunked, as "The question"
// in SPEC.md requires.
question.TransferEncoding = in.TransferEncoding
question.Header = in.Header.Clone()
question.Header.Set(waf.ClientHeader, rq.client.String())
question.Header.Set(waf.PeerHeader, rq.peer.String())
question.Header.Del(waf.ContentLengthHeader)
announced := in.Header.Get("Content-Length")
if announced != "" {
question.Header.Set(waf.ContentLengthHeader, announced)
}
question.Header.Set(requestIDHeader, rq.id)
question.Header.Set("X-Forwarded-Proto", scheme(in, rq.peerTrusted))
// Go's client adds a User-Agent of its own to a request without the key.
_, sent := question.Header["User-Agent"]
if !sent {
question.Header["User-Agent"] = nil
}
return question, nil
}
// answerWith gives the visitor waf's answer, less its disposition.
func (rq *request) answerWith(answer *wafAnswer) {
rq.prepareOwnAnswer(rq.requestDeadline())
header := rq.out.Header()
maps.Copy(header, answer.header)
header.Del(wafprotocol.DispositionHeader)
header.Del(waf.DispositionHeader)
// Go's server adds a Content-Type it guesses from the body to a response whose
// headers have no Content-Type key. The key with no value sends none.
@@ -206,10 +253,7 @@ func (rq *request) answerWith(answer *wafAnswer) {
// sendReport sends waf report on a goroutine of its own, so that the
// visitor does not wait for it. The report goes on after ctx, the request's own
// context, has ended.
func (h *handler) sendReport(
ctx context.Context,
report *wafprotocol.Report,
) {
func (h *handler) sendReport(ctx context.Context, report *waf.Report) {
h.reportsMu.Lock()
defer h.reportsMu.Unlock()
@@ -227,18 +271,23 @@ func (h *handler) sendReport(
// postReport posts report to waf, and waits for its answer for up to
// SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS.
func (h *handler) postReport(
ctx context.Context,
report *wafprotocol.Report,
) error {
func (h *handler) postReport(ctx context.Context, report *waf.Report) error {
body, err := json.Marshal(report)
if err != nil {
return fmt.Errorf("encode the report: %w", err)
}
ctx, cancel := context.WithTimeout(ctx, h.config.WAFBackendTimeout)
defer cancel()
req, err := wafprotocol.NewReportRequest(ctx, h.reportURL, report)
req, err := http.NewRequestWithContext( //nolint:gosec // a setting's URL
ctx, http.MethodPost, h.reportURL, bytes.NewReader(body))
if err != nil {
return err
return fmt.Errorf("make the request: %w", err)
}
req.Header.Set("Content-Type", "application/json")
res, err := h.wafTransport.RoundTrip(req)
if err != nil {
return fmt.Errorf("send the report: %w", err)
+50 -21
View File
@@ -18,11 +18,12 @@ import (
"slices"
"strings"
"sync/atomic"
"time"
"golang.org/x/net/http/httpguts"
"github.com/fsnotify/fsnotify"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/dirwatch"
"sneak.berlin/go/smallwebwaf/internal/config"
)
// The actions a rule takes when it matches.
@@ -39,6 +40,12 @@ const (
// extension ends the name of every rule file.
const extension = ".rules"
// quietTime is how long SWW_WAF_RULE_FILES_DIR must go without a change before
// the rule files are read again, so that a file still being written, such
// as one saved in place, appended to or copied in with scp, is read only
// once whole.
const quietTime = 2 * time.Second
// headerTarget starts the target that is one request header,
// header:<Name>.
const headerTarget = "header:"
@@ -169,18 +176,26 @@ func (f *Files) Len() int {
}
// Watch watches Dir until ctx is done, and reads the rule files again
// once Dir has had no change for a short time, after one is edited, added
// or removed, and after Watch starts watching, as dirwatch describes. If
// they then hold an error, the rules stay as they were, the error is
// logged with its file and line, and the files are read again after the
// next change. If Dir cannot be watched, that is logged, and the rules
// stay as they were loaded. While Enabled is false, Watch returns at once.
// once Dir has had no change for quietTime, after one is edited, added or
// removed, and after Watch starts watching. If they then hold an error,
// the rules stay as they were, the error is logged with its file and
// line, and the files are read again after the next change. If Dir cannot
// be watched, that is logged, and the rules stay as they were loaded.
// While Enabled is false, Watch returns at once.
func (f *Files) Watch(ctx context.Context) {
if !f.params.Enabled {
return
}
watcher, err := dirwatch.New(f.params.Dir)
watcher, err := fsnotify.NewWatcher()
if err == nil {
defer func() {
_ = watcher.Close()
}()
err = watcher.Add(f.params.Dir)
}
if err != nil {
f.params.ProcessLog.Error("cannot watch the rule files for edits",
"error", err.Error())
@@ -188,22 +203,36 @@ func (f *Files) Watch(ctx context.Context) {
return
}
defer func() {
_ = watcher.Close()
}()
f.params.ProcessLog.Info("watching the rule files for edits",
"directory", f.params.Dir)
// A change to any file in Dir counts.
watcher.CallWhenQuiet(ctx,
func(string) bool { return true },
f.readAgain,
func(err error) {
f.readAfterChanges(ctx, watcher.Events, watcher.Errors)
}
// readAfterChanges reads the rule files again once quietTime has passed
// without a change from events, until ctx is done, and logs the errors
// from errs. The wait starts at once, as if for a change, so that an edit
// saved after Load read the files, and before Dir was watched, is taken
// in too.
func (f *Files) readAfterChanges(
ctx context.Context, events <-chan fsnotify.Event, errs <-chan error,
) {
quiet := time.NewTimer(quietTime)
defer quiet.Stop()
for {
select {
case <-ctx.Done():
return
case <-events:
quiet.Reset(quietTime)
case <-quiet.C:
f.readAgain()
case err := <-errs:
f.params.ProcessLog.Warn("watching the rule files failed",
"error", err.Error())
},
)
}
}
}
// readAgain reads the rule files again, in place of the rules loaded, or
@@ -349,7 +378,7 @@ func parse(line string) (Rule, bool, error) {
rule.Target,
errNotTarget,
)
case isHeader && !httpguts.ValidHeaderFieldName(headerName):
case isHeader && !config.IsHeaderName(headerName):
return Rule{}, false, fmt.Errorf(
"the target %q %w",
rule.Target,
+165
View File
@@ -0,0 +1,165 @@
package rules
import (
"context"
"log/slog"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"slices"
"testing"
"testing/synctest"
"time"
"github.com/fsnotify/fsnotify"
"sneak.berlin/go/smallwebwaf/internal/alerts"
)
// The tests below run readAfterChanges in a synctest bubble, where time is
// a clock of the test's own: time.Sleep moves it on at once, and
// synctest.Wait returns once readAfterChanges waits again, so that every
// reading due by then is done. The test sends the changes itself, as the
// watch of a directory cannot run in a bubble.
func TestFileWrittenInTwoPartsTakenInOnlyWhole(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "50-app.rules")
writeFile(t, path, "first path block ^/first\n")
files := load(t, dir)
changes := run(t, files)
file, err := os.Create(path) //nolint:gosec // a file the test wrote
if err != nil {
t.Fatalf("create: %v", err)
}
defer func() {
_ = file.Close()
}()
// The first part ends in the middle of a ban rule's regex, which,
// read then, would ban every request.
write(t, file, "first path block ^/first\nprobe path ban ^/")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantMatched(t, files, "/anything")
// The second part starts the wait again.
write(t, file, `\.env$`+"\n")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantMatched(t, files, "/.env")
time.Sleep(time.Nanosecond)
synctest.Wait()
wantMatched(t, files, "/.env", "probe")
wantMatched(t, files, "/anything")
})
}
func TestEditSavedBeforeTheWatchStartsTakenIn(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "50-app.rules")
writeFile(t, path, "first path block ^/first\n")
files := load(t, dir)
// Saved after Load read the files, and before the directory was
// watched, so that no change is seen for it.
writeFile(t, path, "first path block ^/edited\n")
run(t, files)
time.Sleep(quietTime)
synctest.Wait()
wantMatched(t, files, "/edited", "first")
})
}
// load loads the rules in dir.
func load(t *testing.T, dir string) *Files {
t.Helper()
files, err := Load(Params{
Dir: dir, Enabled: true, ProcessLog: slog.New(slog.DiscardHandler),
Alerts: alerts.New(alerts.Params{}),
})
if err != nil {
t.Fatalf("load: %v", err)
}
return files
}
// run runs files' readAfterChanges until the test ends, and returns the
// channel that sends it changes.
func run(t *testing.T, files *Files) chan<- fsnotify.Event {
t.Helper()
changes := make(chan fsnotify.Event)
ctx, stop := context.WithCancel(t.Context())
stopped := make(chan struct{})
go func() {
files.readAfterChanges(ctx, changes, nil)
close(stopped)
}()
t.Cleanup(func() {
stop()
<-stopped
})
return changes
}
// writeFile writes content to the file at path.
func writeFile(t *testing.T, path, content string) {
t.Helper()
err := os.WriteFile(path, []byte(content), 0o600)
if err != nil {
t.Fatalf("write %s: %v", path, err)
}
}
// write writes text to the end of file.
func write(t *testing.T, file *os.File, text string) {
t.Helper()
_, err := file.WriteString(text)
if err != nil {
t.Fatalf("write: %v", err)
}
}
// wantMatched checks the ids of the rules that a GET request for path
// matches, in order.
func wantMatched(t *testing.T, files *Files, path string, want ...string) {
t.Helper()
r := httptest.NewRequestWithContext(t.Context(), http.MethodGet,
"http://app.example"+path, nil)
matched := files.Match(r)
got := make([]string, 0, len(matched))
for _, rule := range matched {
got = append(got, rule.ID)
}
if !slices.Equal(got, want) {
t.Errorf("%s matched %v, want %v", path, got, want)
}
}
+19 -31
View File
@@ -1284,18 +1284,18 @@ func TestBansWrittenOnceBansWriteDelayAfterABan(t *testing.T) {
// A second ban, made while the first waits to be written, puts the
// write off no further, and is written with it.
first, _ := params.Ledger.Ban(
first, _ := params.Ledger.BanForLimit(
netip.MustParsePrefix("203.0.113.9/32"),
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
bans.Notes{},
)
time.Sleep(5 * time.Second)
second, _ := params.Ledger.Ban(
second, _ := params.Ledger.BanForLimit(
netip.MustParsePrefix("203.0.113.10/32"),
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
bans.Notes{},
)
time.Sleep(5*time.Second - time.Nanosecond)
@@ -1358,16 +1358,16 @@ func TestEditJustBeforeAScheduledWriteSurvivesIt(t *testing.T) {
run(t, load(t, params).Run)
// A ban, and bans.json written with it.
params.Ledger.Ban(netip.MustParsePrefix("203.0.113.9/32"),
midnight(), bans.Grounds{Cause: bans.CauseLimit})
params.Ledger.BanForLimit(netip.MustParsePrefix("203.0.113.9/32"),
midnight(), bans.Notes{})
time.Sleep(params.BansWriteDelay)
synctest.Wait()
// A second ban is to be written BansWriteDelay later. Just before
// then, an admin saves bans.json with the first ban lifted and
// another added.
params.Ledger.Ban(netip.MustParsePrefix("203.0.113.10/32"),
midnight(), bans.Grounds{Cause: bans.CauseLimit})
params.Ledger.BanForLimit(netip.MustParsePrefix("203.0.113.10/32"),
midnight(), bans.Notes{})
time.Sleep(params.BansWriteDelay - time.Nanosecond)
synctest.Wait()
edit(t, dir, bansJSON, permanentBansJSON)
@@ -1483,10 +1483,10 @@ func TestFailedWriteLeavesTheFileAsItWas(t *testing.T) {
t.Fatalf("mkdir: %v", err)
}
params.Ledger.Ban(
params.Ledger.BanForLimit(
netip.MustParsePrefix("203.0.113.9/32"),
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
bans.Notes{},
)
params.Limiter.Count(
netip.MustParsePrefix("203.0.113.9/32"),
@@ -1556,10 +1556,10 @@ func TestWritesAreCountedInTheMetrics(t *testing.T) {
t.Fatalf("mkdir: %v", err)
}
params.Ledger.Ban(
params.Ledger.BanForLimit(
netip.MustParsePrefix("203.0.113.9/32"),
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
bans.Notes{},
)
err = files.WriteAll()
@@ -2135,11 +2135,7 @@ func TestBanLiftedByAnEditWhileRunning(t *testing.T) {
// The ban that liftedBansJSON lifts, before it is lifted.
netblock := netip.MustParsePrefix(liftedClient + "/32")
params.Ledger.Ban(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
params.Ledger.BanForLimit(netblock, midnight(), bans.Notes{})
_, banned, _ := params.Ledger.Find(netblock.Addr(), afterLifting())
if !banned {
@@ -2452,7 +2448,7 @@ func newParams(dir string) state.Params {
AttackBanDuration: 7 * 24 * time.Hour,
MaxBans: 5000,
}),
Limiter: ratelimit.New(ratelimit.Params{MaxClients: 20000}),
Limiter: ratelimit.New(ratelimit.Limits{}, 20000),
GeoJS: geojs.New(geojs.Params{Now: midnight}),
Lists: reputation.New(reputation.Params{
BlocklistURLs: []string{
@@ -2514,15 +2510,11 @@ func fill(params state.Params) {
client := netip.MustParsePrefix("203.0.113.9/32")
params.Ledger.Load([]bans.Ban{permanentBan()})
params.Ledger.Ban(client, now, bans.Grounds{
Cause: bans.CauseLimit,
Notes: bans.Notes{ASN: asn, ASName: asName, Country: "DE", Limit: 1},
params.Ledger.BanForLimit(client, now, bans.Notes{
ASN: asn, ASName: asName, Country: "DE", Limit: 1,
})
params.Ledger.Ban(netip.MustParsePrefix("192.0.2.1/32"), now,
bans.Grounds{
Cause: bans.CauseAttack,
Notes: bans.Notes{RuleID: "env-file", Target: "path"},
})
params.Ledger.BanForAttack(netip.MustParsePrefix("192.0.2.1/32"), now,
bans.Notes{RuleID: "env-file", Target: "path"})
for _, c := range []string{"2001:db8::/64", "203.0.113.9/32", "192.0.2.1/32"} {
params.Limiter.Count(netip.MustParsePrefix(c), now, whole)
@@ -2708,11 +2700,7 @@ func wantLiftedBanKept(
}
// Were the lifted ban counted, the next would last three hours.
ban, _ := ledger.Ban(
netblock,
afterLifting(),
bans.Grounds{Cause: bans.CauseLimit},
)
ban, _ := ledger.BanForLimit(netblock, afterLifting(), bans.Notes{})
if ban.Expires.Sub(ban.Start) != time.Hour {
t.Errorf(
"the next ban lasts %s, want 1h",
+2 -2
View File
@@ -13,7 +13,7 @@ import (
"github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// healthCheckTimeout bounds the whole health check.
@@ -116,7 +116,7 @@ func askHealthEndpoint(ctx context.Context, listenAddr string) error {
health := "http://" + net.JoinHostPort(
"127.0.0.1",
port,
) + wafprotocol.HealthPath
) + waf.HealthPath
req, err := http.NewRequestWithContext(
ctx,
+6 -6
View File
@@ -10,7 +10,7 @@ import (
"go.uber.org/fx"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/processlog"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rproxy"
)
@@ -23,11 +23,11 @@ import (
// setting for either.
func RunRProxy(ctx context.Context, params Params) int {
hostname, _ := os.Hostname() // "" when the host has no name to give
processLog := processlog.New(processlog.Params{
Output: params.Stdout,
InstanceName: hostname,
Level: slog.LevelInfo,
})
processLog := requestlog.NewProcessLogger(
params.Stdout,
hostname,
slog.LevelInfo,
)
cfg, err := config.RProxyFromEnvironment(params.Viper)
if err != nil {
+24 -21
View File
@@ -21,7 +21,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/sww"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
const (
@@ -164,7 +164,7 @@ func TestABanThroughOneSiteRefusesTheClientOnTheOther(t *testing.T) {
// One request's three lines carry one id, rproxy's with waf's answer.
line := firstOut.line(t, "path", bannedProbe)
if line["disposition"] != wafprotocol.DispositionDeny ||
if line["disposition"] != waf.DispositionDeny ||
line["client_ip"] != placed {
t.Errorf("rproxy's line is %v", line)
}
@@ -249,7 +249,7 @@ func TestTheSiteAnswersWhileWAFIsStoppedAndStartedAgain(t *testing.T) {
wantStatus(t, site+"/after-stop", placed, http.StatusOK)
line := out.line(t, "path", "/after-stop")
if line["disposition"] != wafprotocol.DispositionAllow ||
if line["disposition"] != waf.DispositionAllow ||
line["waf_backend_no_answer"] != true {
t.Errorf("rproxy's line is %v", line)
}
@@ -437,7 +437,7 @@ func TestAChunkedBodyStaysABodyAtWAF(t *testing.T) {
const smuggledID = "smuggled-request"
smuggled := "GET " + bannedProbe + " HTTP/1.1\r\nHost: site.example\r\n" +
wafprotocol.ClientHeader + ": " + otherClient + "\r\n" +
waf.ClientHeader + ": " + otherClient + "\r\n" +
"X-Request-ID: " + smuggledID + "\r\n\r\n"
status := sendRaw(
@@ -605,11 +605,10 @@ func TestRProxyAsksAWAFThatSpeaksHTTP2OverHTTP1(t *testing.T) {
protocols := make(chan string, 10)
backend := httptest.NewUnstartedServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(wafprotocol.ClientHeader) != "" {
if r.Header.Get(waf.ClientHeader) != "" {
protocols <- r.Proto
w.Header().
Set(wafprotocol.DispositionHeader, wafprotocol.DispositionDeny)
w.Header().Set(waf.DispositionHeader, waf.DispositionDeny)
w.WriteHeader(http.StatusForbidden)
}
}))
@@ -691,9 +690,8 @@ func TestRProxyStoppingWaitsForAReportBeingSent(t *testing.T) {
release := make(chan struct{})
backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(wafprotocol.ClientHeader) != "" {
w.Header().
Set(wafprotocol.DispositionHeader, wafprotocol.DispositionAllow)
if r.Header.Get(waf.ClientHeader) != "" {
w.Header().Set(waf.DispositionHeader, waf.DispositionAllow)
return
}
@@ -757,7 +755,8 @@ func TestRProxyStoppingWaitsForAReportBeingSent(t *testing.T) {
}
}
// sendAndKeepOpen sends request to rproxy at addr, and returns the connection.
// sendAndKeepOpen sends request to rproxy at addr, and returns the connection. A
// read from it ends with an error once waitLimit is up.
func sendAndKeepOpen(t *testing.T, addr, request string) net.Conn {
t.Helper()
@@ -770,6 +769,11 @@ func sendAndKeepOpen(t *testing.T, addr, request string) net.Conn {
_ = conn.Close()
})
err = conn.SetDeadline(time.Now().Add(waitLimit))
if err != nil {
t.Fatalf("set the deadline: %v", err)
}
_, err = io.WriteString(conn, request)
if err != nil {
t.Fatalf("send: %v", err)
@@ -781,7 +785,10 @@ func sendAndKeepOpen(t *testing.T, addr, request string) net.Conn {
func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
t.Parallel()
const cutOffAfter = 5 * time.Second
const (
cutOffAfter = 5 * time.Second
upstreamSetting = "SWW_RPROXY_UPSTREAM_URL"
)
asked := make(chan struct{})
release := make(chan struct{})
@@ -795,8 +802,7 @@ func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
w.Header().
Set(wafprotocol.DispositionHeader, wafprotocol.DispositionAllow)
w.Header().Set(waf.DispositionHeader, waf.DispositionAllow)
}))
t.Cleanup(backend.Close)
@@ -804,9 +810,9 @@ func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
defer stop()
env := map[string]string{
rproxyListenAddr: localhost + ":0",
rproxyUpstreamURL: app.URL,
wafBackend: backend.URL,
rproxyListenAddr: localhost + ":0",
upstreamSetting: app.URL,
wafBackend: backend.URL,
}
out := &output{}
exited := make(chan int, 1)
@@ -832,9 +838,6 @@ func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
stop()
// Counted from the stop, so a slow wait for the app cannot shorten it.
_ = conn.SetReadDeadline(stopped.Add(cutOffAfter + waitLimit))
text, err := io.ReadAll(conn)
if err != nil || len(text) != 0 {
t.Errorf("read %q (%v), want the connection cut off", text, err)
@@ -932,7 +935,7 @@ func TestMainRunsRProxy(t *testing.T) {
addr, stop := startRProxyProcess(t, wafBackend+"=http://127.0.0.1:9")
req, err := http.NewRequestWithContext(t.Context(), http.MethodGet,
"http://"+addr+wafprotocol.HealthPath, http.NoBody)
"http://"+addr+waf.HealthPath, http.NoBody)
if err != nil {
t.Fatalf("new request: %v", err)
}
+30 -52
View File
@@ -10,14 +10,13 @@ package sww
import (
"context"
"errors"
"fmt"
"io"
"os"
"os/signal"
"syscall"
"time"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/config"
@@ -27,7 +26,14 @@ import (
// know.
const usageStatus = 2
var errNoCommand = errors.New("no command given")
// usage is what sww prints then.
const usage = `usage: sww <command>
commands:
rproxy the reverse proxy in front of one app
waf decides for rproxy, and keeps bans, counters and history
healthcheck the container's health check
`
// Params are what RunWAF and RunRProxy need from the process.
type Params struct {
@@ -45,9 +51,14 @@ type Params struct {
// Main runs the command os.Args names, and returns the process's exit status.
// `sww rproxy` and `sww waf` run until SIGTERM or SIGINT.
// With no command, or one it does not know, cobra prints the error and the
// commands to stderr, and Main returns usageStatus.
// With no command, or one it does not know, it prints the commands and returns
// usageStatus.
func Main(version string) int {
args := os.Args[1:]
if len(args) == 0 {
return printUsage(os.Stderr)
}
ctx, stop := signal.NotifyContext(context.Background(),
syscall.SIGTERM, os.Interrupt)
defer stop()
@@ -58,54 +69,21 @@ func Main(version string) int {
Stdout: os.Stdout,
}
// status is what the command that ran returns.
status := 0
root := &cobra.Command{
Use: "sww",
Short: "a web application firewall",
Args: cobra.NoArgs,
RunE: func(*cobra.Command, []string) error {
return errNoCommand
},
switch args[0] {
case "rproxy":
return RunRProxy(ctx, params)
case "waf":
return RunWAF(ctx, params)
case "healthcheck":
return HealthCheck(ctx, args[1:], params.Viper, os.Stderr)
default:
return printUsage(os.Stderr)
}
root.CompletionOptions.DisableDefaultCmd = true
}
root.AddCommand(
&cobra.Command{
Use: "rproxy",
Short: "the reverse proxy in front of one app",
Run: func(cmd *cobra.Command, _ []string) {
status = RunRProxy(cmd.Context(), params)
},
},
&cobra.Command{
Use: "waf",
Short: "decides for rproxy, and keeps bans, counters and history",
Run: func(cmd *cobra.Command, _ []string) {
status = RunWAF(cmd.Context(), params)
},
},
&cobra.Command{
Use: "healthcheck",
Short: "the container's health check",
// HealthCheck refuses every argument itself, flags included.
DisableFlagParsing: true,
Run: func(cmd *cobra.Command, args []string) {
status = HealthCheck(
cmd.Context(),
args,
params.Viper,
cmd.ErrOrStderr(),
)
},
},
)
// printUsage writes the commands to w, and returns usageStatus.
func printUsage(w io.Writer) int {
_, _ = fmt.Fprint(w, usage)
err := root.ExecuteContext(ctx)
if err != nil {
return usageStatus
}
return status
return usageStatus
}
+3 -5
View File
@@ -23,7 +23,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
"sneak.berlin/go/smallwebwaf/internal/sww"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
const (
@@ -660,9 +660,7 @@ func TestCoreRuleSetExclusionsFileThatCannotBeUsedStopsTheStartWhileTheCoreRuleS
status := run(t.Context(), map[string]string{
listenAddr: localhost + ":0", stateDir: t.TempDir(), ruleFilesDir: t.TempDir(),
"SWW_WAF_CORE_RULE_SET_MODE": string(
config.CoreRuleSetModeOff,
),
"SWW_WAF_CORE_RULE_SET_MODE": config.CoreRuleSetModeOff,
"SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE": path,
}, out)
if status != 1 {
@@ -1813,7 +1811,7 @@ func statusFrom(t *testing.T, url, from string) int {
}
req.Header.Set("X-Forwarded-For", from)
req.Header.Set(wafprotocol.ClientHeader, from)
req.Header.Set(waf.ClientHeader, from)
req.Header.Set("X-Request-ID", rand.Text())
transport := &http.Transport{}
+32 -87
View File
@@ -2,7 +2,6 @@ package sww
import (
"context"
"fmt"
"io"
"log/slog"
"net"
@@ -19,10 +18,10 @@ import (
"sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/processlog"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/remotelog"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/routewatch"
"sneak.berlin/go/smallwebwaf/internal/rules"
"sneak.berlin/go/smallwebwaf/internal/state"
@@ -43,11 +42,8 @@ const remoteLogStopTimeout = 2 * time.Second
func RunWAF(ctx context.Context, params Params) int {
// Until the settings are read, the one message is an invalid setting's
// error, which every SWW_WAF_LOG_LEVEL lets through.
processLog := processlog.New(processlog.Params{
Output: params.Stdout,
InstanceName: config.InstanceName(params.Viper),
Level: slog.LevelError,
})
processLog := requestlog.NewProcessLogger(params.Stdout,
config.InstanceName(params.Viper), slog.LevelError)
cfg, err := config.WAFFromEnvironment(params.Viper)
if err != nil {
@@ -66,11 +62,11 @@ func RunWAF(ctx context.Context, params Params) int {
stdout = io.MultiWriter(params.Stdout, remote)
}
processLog = processlog.New(processlog.Params{
Output: stdout,
InstanceName: cfg.InstanceName,
Level: cfg.LogLevel,
})
processLog = requestlog.NewProcessLogger(
stdout,
cfg.InstanceName,
cfg.LogLevel,
)
if remote != nil {
stopSending := startSending(ctx, remote, processLog)
@@ -121,8 +117,7 @@ func RunWAF(ctx context.Context, params Params) int {
"settings", cfg)
// The last write follows the stop of every hook, and so the end of every
// loop that changes the state or raises an alert, and of every ask of
// GeoJS, the DNSBL zones, AbuseIPDB and routewatch. A hook of the state
// loop that changes the state or raises an alert. A hook of the state
// files would stop before the loops of the parts built ahead of them.
return run(ctx, app, failed, processLog, func() error {
err := files.WriteAll()
@@ -180,35 +175,11 @@ func newRuleFiles(
return ruleFiles, nil
}
// newCoreRuleSet returns the Core Rule Set at SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL,
// without the rules SWW_WAF_CORE_RULE_SET_DISABLED_RULES switches off, reading bodies
// up to SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE, with the exclusions in
// SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE and the fixes of the apps in SWW_WAF_APPS,
// or nil while SWW_WAF_CORE_RULE_SET_MODE is off. An exclusions file that
// coreruleset.New refuses is an error in every mode. While the mode is off, a
// set file is loaded with the Core Rule Set only to check it.
// newCoreRuleSet returns the Core Rule Set, as waf.NewCoreRuleSet does.
func newCoreRuleSet(cfg *config.WAF) (*coreruleset.CoreRuleSet, error) {
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff &&
cfg.CoreRuleSetExclusionsFile == "" {
return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error
}
crs, err := coreruleset.New(coreruleset.Params{
ParanoiaLevel: cfg.CoreRuleSetParanoiaLevel,
DisabledRules: cfg.CoreRuleSetDisabledRules,
BodyInspectionSize: cfg.CoreRuleSetBodyInspectionSize,
ExclusionsFile: cfg.CoreRuleSetExclusionsFile,
Apps: cfg.Apps,
})
crs, err := waf.NewCoreRuleSet(cfg)
if err != nil {
return nil, startError{
"cannot use the Core Rule Set",
fmt.Errorf("SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE: %w", err),
}
}
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff {
return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error
return nil, startError{"cannot use the Core Rule Set", err}
}
return crs, nil
@@ -221,7 +192,7 @@ func newLookupFile(
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
alertQueue *alerts.Queue,
) (*lookup.File, error) {
if cfg.LookupSource != config.LookupSourceFile {
if cfg.LookupSource != "file" {
return nil, nil //nolint:nilnil // no lookup database, and no error
}
@@ -242,17 +213,14 @@ func newLookupFile(
// newLimiter returns the table of clients, with the rate and byte limits.
func newLimiter(cfg *config.WAF) *ratelimit.Limiter {
return ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{
PerMinute: cfg.RateLimitPerMinute,
PerHour: cfg.RateLimitPerHour,
PerDay: cfg.RateLimitPerDay,
BytesPerMinute: cfg.BytesLimitPerMinute,
BytesPerHour: cfg.BytesLimitPerHour,
BytesPerDay: cfg.BytesLimitPerDay,
},
MaxClients: cfg.MaxTrackedClients,
})
return ratelimit.New(ratelimit.Limits{
PerMinute: cfg.RateLimitPerMinute,
PerHour: cfg.RateLimitPerHour,
PerDay: cfg.RateLimitPerDay,
BytesPerMinute: cfg.BytesLimitPerMinute,
BytesPerHour: cfg.BytesLimitPerHour,
BytesPerDay: cfg.BytesLimitPerDay,
}, cfg.MaxTrackedClients)
}
// newLedger returns the ledger of the bans, with the settings for them.
@@ -304,13 +272,11 @@ func newLists(
return lists
}
// newDNSBL returns what asks the DNSBL zones, none asked yet. At the stop it
// ends the queries under way, as reputation.DNSBL.StopAsking says.
// newDNSBL returns what asks the DNSBL zones, none asked yet.
func newDNSBL(
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
alertQueue *alerts.Queue,
cfg *config.WAF, processLog *slog.Logger, alertQueue *alerts.Queue,
) *reputation.DNSBL {
dnsbl := reputation.NewDNSBL(reputation.DNSBLParams{
return reputation.NewDNSBL(reputation.DNSBLParams{
Zones: cfg.DNSBLZones,
Resolver: cfg.DNSBLResolver,
CacheTTL: cfg.IPReputationCacheTTL,
@@ -319,17 +285,12 @@ func newDNSBL(
ProcessLog: processLog,
Alerts: alertQueue,
})
lc.Append(fx.StopHook(dnsbl.StopAsking))
return dnsbl
}
// newAbuseIPDB returns what checks clients with AbuseIPDB while
// SWW_WAF_ABUSEIPDB_KEY is set, none checked yet. Until waf stops, it
// fetches AbuseIPDB's blacklist as it is due, and bans the clients on it in
// ledger, as waf.BanAbuseIPDBBlacklist says. At the stop it ends the checks
// under way, as reputation.AbuseIPDB.StopChecking says.
// ledger, as waf.BanAbuseIPDBBlacklist says.
func newAbuseIPDB(
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
alertQueue *alerts.Queue, ledger *bans.Ledger,
@@ -355,30 +316,24 @@ func newAbuseIPDB(
})
lc.Append(background(abuseIPDB.RunBlacklist))
lc.Append(fx.StopHook(abuseIPDB.StopChecking))
return abuseIPDB
}
// newRouteWatch returns what asks routewatch about clients, or nil while
// SWW_WAF_ROUTEWATCH_URL is unset. At the stop it ends the queries under
// way, as routewatch.Client.StopAsking says.
// SWW_WAF_ROUTEWATCH_URL is unset.
func newRouteWatch(
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
cfg *config.WAF, processLog *slog.Logger,
) *routewatch.Client {
if cfg.RouteWatchURL == "" {
return nil
}
routeWatch := routewatch.New(routewatch.Params{
return routewatch.New(routewatch.Params{
URL: cfg.RouteWatchURL,
Now: utcNow,
ProcessLog: processLog,
})
lc.Append(fx.StopHook(routeWatch.StopAsking))
return routeWatch
}
// newMetrics returns the metrics, with those read from the other parts as
@@ -416,18 +371,12 @@ func newMetrics(
// newGeoJS returns what looks clients up at GeoJS, none looked up yet. Each
// answer goes to the limiter and the ledger, as waf.AddLookup says, and each
// failure is handled as waf.GeoJSFailed says. At the stop it ends the request
// under way, as geojs.GeoJS.StopAsking says.
// failure is handled as waf.GeoJSFailed says.
func newGeoJS(
lc fx.Lifecycle,
cfg *config.WAF,
m *metrics.Metrics,
limiter *ratelimit.Limiter,
ledger *bans.Ledger,
alertQueue *alerts.Queue,
processLog *slog.Logger,
cfg *config.WAF, m *metrics.Metrics, limiter *ratelimit.Limiter,
ledger *bans.Ledger, alertQueue *alerts.Queue, processLog *slog.Logger,
) *geojs.GeoJS {
geoJS := geojs.New(geojs.Params{
return geojs.New(geojs.Params{
URL: geojs.URL,
Answered: func(answer geojs.Answer) {
waf.AddLookup(cfg, limiter, ledger, answer)
@@ -439,10 +388,6 @@ func newGeoJS(
},
WentWithoutAnswer: m.RequestsWithoutGeoJSAnswer.Inc,
})
lc.Append(fx.StopHook(geoJS.StopAsking))
return geoJS
}
// newWAFServer returns the server waf runs, with its parts and the time
+37 -102
View File
@@ -4,7 +4,6 @@ import (
"bytes"
"context"
"crypto/rand"
"errors"
"io"
"net/http"
"os"
@@ -21,7 +20,7 @@ import (
_ "time/tzdata"
"sneak.berlin/go/smallwebwaf/internal/sww"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// commandForMain, while set in the environment, has this test binary run Main
@@ -107,7 +106,7 @@ func TestMainRunsWAF(t *testing.T) {
starting := out.lineBeforeExit(t, "starting", exited)
addr, _ := starting["address"].(string)
ask(t, addr, "/", http.StatusOK, wafprotocol.DispositionAllow)
ask(t, addr, "/", http.StatusOK, waf.DispositionAllow)
// The files are written as sww stops.
err = cmd.Process.Signal(syscall.SIGTERM)
@@ -165,104 +164,46 @@ func wantUTC(t *testing.T, where string, value any) {
func TestMainWithoutACommandItKnowsPrintsTheCommands(t *testing.T) {
t.Parallel()
// completion is cobra's own command, which sww turns off.
for _, command := range []string{"", "serve", "completion"} {
t.Run("command "+command, func(t *testing.T) {
t.Parallel()
code, stdout, stderr := runMain(t, command)
if code != 2 {
t.Errorf("exit status %d, want 2", code)
}
wantTheCommands(t, "stderr", stderr)
if stdout != "" {
t.Errorf("stdout %q, want nothing", stdout)
}
})
}
}
func TestMainHelpPrintsTheCommands(t *testing.T) {
t.Parallel()
for _, command := range []string{"help", "--help"} {
t.Run("command "+command, func(t *testing.T) {
t.Parallel()
code, stdout, _ := runMain(t, command)
if code != 0 {
t.Errorf("exit status %d, want 0", code)
}
wantTheCommands(t, "stdout", stdout)
})
}
}
// wantTheCommands checks that text, what Main wrote to where, names each
// command.
func wantTheCommands(t *testing.T, where, text string) {
t.Helper()
for _, name := range []string{"rproxy", wafCommand, "healthcheck"} {
if !strings.Contains(text, name) {
t.Errorf("%s %q does not name %s", where, text, name)
}
}
}
func TestMainHealthCheckRefusesAnArgument(t *testing.T) {
t.Parallel()
for _, argument := range []string{"now", "--x", "-h"} {
t.Run("argument "+argument, func(t *testing.T) {
t.Parallel()
code, stdout, stderr := runMain(t, "healthcheck "+argument)
want := "sww healthcheck: unexpected argument \"" +
argument + "\"\n"
if code != 1 || stderr != want || stdout != "" {
t.Errorf("exit status %d, stdout %q and stderr %q, "+
"want 1, nothing and %q", code, stdout, stderr, want)
}
})
}
}
// runMain runs Main with command in a process of its own, and returns its
// exit status and what it wrote to stdout and stderr.
func runMain(t *testing.T, command string) (int, string, string) {
t.Helper()
binary, err := os.Executable()
if err != nil {
t.Fatalf("find the test binary: %v", err)
}
ctx, cancel := context.WithTimeout(t.Context(), waitLimit)
defer cancel()
for _, command := range []string{"", "serve"} {
t.Run("command "+command, func(t *testing.T) {
t.Parallel()
var stdout, stderr bytes.Buffer
ctx, cancel := context.WithTimeout(t.Context(), waitLimit)
defer cancel()
//nolint:gosec // this test binary
cmd := exec.CommandContext(ctx, binary)
cmd.Stdout, cmd.Stderr = &stdout, &stderr
var stdout, stderr bytes.Buffer
cmd.Env = append(os.Environ(), commandForMain+"="+command)
cmd := exec.CommandContext(ctx, binary)
cmd.Stdout, cmd.Stderr = &stdout, &stderr
err = cmd.Run()
cmd.Env = append(os.Environ(), commandForMain+"="+command)
var exitErr *exec.ExitError
if err != nil && !errors.As(err, &exitErr) {
t.Fatalf("run: %v", err)
err := cmd.Run()
if code := cmd.ProcessState.ExitCode(); code != 2 {
t.Errorf("exit status %d (%v), want 2", code, err)
}
for _, name := range []string{"rproxy", wafCommand, "healthcheck"} {
if !strings.Contains(stderr.String(), name) {
t.Errorf(
"stderr %q does not name %s",
stderr.String(),
name,
)
}
}
if stdout.Len() != 0 {
t.Errorf("stdout %q, want nothing", stdout.String())
}
})
}
return cmd.ProcessState.ExitCode(), stdout.String(), stderr.String()
}
// ask sends waf at addr rproxy's question about a request for path from
@@ -276,7 +217,7 @@ func ask(t *testing.T, addr, path string, status int, disposition string) {
t.Fatalf("new request: %v", err)
}
req.Header.Set(wafprotocol.ClientHeader, placed)
req.Header.Set(waf.ClientHeader, placed)
req.Header.Set("X-Request-ID", rand.Text())
transport := &http.Transport{}
@@ -291,11 +232,11 @@ func ask(t *testing.T, addr, path string, status int, disposition string) {
_ = res.Body.Close()
if res.StatusCode != status ||
res.Header.Get(wafprotocol.DispositionHeader) != disposition {
res.Header.Get(waf.DispositionHeader) != disposition {
t.Errorf(
"answered %d with disposition %q, want %d with %q",
res.StatusCode,
res.Header.Get(wafprotocol.DispositionHeader),
res.Header.Get(waf.DispositionHeader),
status,
disposition,
)
@@ -346,18 +287,12 @@ func TestWAFKeepsItsBansAcrossARestartAndNeedsNoApp(t *testing.T) {
}
runWAF(t, env, func(addr string) {
ask(t, addr, "/", http.StatusOK, wafprotocol.DispositionAllow)
ask(t, addr, "/", http.StatusOK, waf.DispositionAllow)
// The default rule file bans a probe for this.
ask(
t,
addr,
"/.env",
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
ask(t, addr, "/.env", http.StatusForbidden, waf.DispositionDeny)
})
runWAF(t, env, func(addr string) {
ask(t, addr, "/", http.StatusForbidden, wafprotocol.DispositionDeny)
ask(t, addr, "/", http.StatusForbidden, waf.DispositionDeny)
})
}
+9 -20
View File
@@ -22,42 +22,31 @@ import (
// an offence, and the request does not wait for the answer. The score is then
// used for each address of the client. Its figures are the score and
// SWW_WAF_ABUSEIPDB_MIN_SCORE.
type abuseIPDB struct {
abuseIPDB *reputation.AbuseIPDB
// limiter is the table of clients the rate limits count, whose histories
// say which clients are offenders.
limiter *ratelimit.Limiter
keySet bool // SWW_WAF_ABUSEIPDB_KEY is set
minScore int64 // SWW_WAF_ABUSEIPDB_MIN_SCORE
action config.ReputationAction // SWW_WAF_DNSBL_ABUSEIPDB_ACTION
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
hitNoter reputationHitNoter
}
type abuseIPDB struct{}
func (abuseIPDB) name() string { return "abuseipdb" }
func (a abuseIPDB) check(
func (abuseIPDB) check(
ctx context.Context,
rq *request,
_ time.Time,
) *verdict {
if !a.keySet {
if rq.h.config.AbuseIPDBKey == "" {
return nil
}
client := clientGroup(a.ipv6GroupPrefix, rq.client)
held, _ := a.limiter.Client(client)
client := clientGroup(rq.h.config, rq.client)
held, _ := rq.h.limiter.Client(client)
offender := held.History.Offences != ratelimit.Offences{}
score, hit := a.abuseIPDB.Hit(ctx, client, rq.client, offender)
score, hit := rq.h.abuseIPDB.Hit(ctx, client, rq.client, offender)
if !hit {
return nil
}
rq.abuseIPDBHit = true
rq.addTag(config.AbuseIPDBTag)
a.hitNoter.noteHit(
rq,
rq.noteHit(
bans.ReputationHit{
Source: reputation.AbuseIPDBSource,
Score: &score,
@@ -65,14 +54,14 @@ func (a abuseIPDB) check(
"scored by AbuseIPDB at or over SWW_WAF_ABUSEIPDB_MIN_SCORE",
)
if a.action != config.ReputationActionDeny {
if rq.h.config.DNSBLAbuseIPDBAction != deny {
return nil
}
return &verdict{
action: requestlog.ActionClientDenied,
figures: map[string]any{
figureScore: score, "min_score": a.minScore,
figureScore: score, "min_score": rq.h.config.AbuseIPDBMinScore,
},
}
}
+1 -3
View File
@@ -26,9 +26,7 @@ func BanAbuseIPDBBlacklist(
scores := make(map[netip.Prefix]int64, len(listed))
for _, client := range listed {
netblock := banNetblock(
cfg.BanScopeV4Prefix, cfg.IPv6GroupPrefix, client.Addr,
)
netblock := banNetblock(cfg, client.Addr)
scores[netblock] = max(scores[netblock], client.Score)
}
+146 -97
View File
@@ -2,6 +2,7 @@ package waf
import (
"bytes"
"crypto/subtle"
"encoding/json"
"errors"
"fmt"
@@ -14,6 +15,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/state"
)
@@ -39,23 +41,80 @@ var (
errNotAddress = errors.New("is not an address, such as 203.0.113.9")
)
// handleHealth answers GET wafprotocol.HealthPath.
func handleHealth() http.HandlerFunc {
return func(w http.ResponseWriter, _ *http.Request) {
// Set here rather than left to Go's server, which would set it only
// after the log line has taken the response's headers.
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
_, _ = io.WriteString(w, "ok\n")
// answerHealth answers a request for the health endpoint.
func (rq *request) answerHealth() {
rq.line.Action = requestlog.ActionOwnEndpoint
// Set here rather than left to Go's server, which would set it only
// after the log line has taken the response's headers.
rq.out.Header().Set("Content-Type", "text/plain; charset=utf-8")
_, _ = io.WriteString(rq.out, "ok\n")
}
// answerAdmin answers a request for one of waf's own endpoints. Each
// needs a token, sent as Authorization: Bearer <token>: the metrics
// SWW_WAF_METRICS_TOKEN, the others SWW_WAF_ADMIN_TOKEN. A request without it is
// refused with 401. An endpoint whose token is unset answers 404, as any other
// path does.
func (rq *request) answerAdmin() {
rq.line.Action = requestlog.ActionOwnEndpoint
token, answer := rq.endpoint()
switch {
case token == "":
http.Error(
rq.out,
http.StatusText(http.StatusNotFound),
http.StatusNotFound,
)
case !hasToken(rq.in, token):
rq.out.Header().Set("WWW-Authenticate", "Bearer")
http.Error(
rq.out,
http.StatusText(http.StatusUnauthorized),
http.StatusUnauthorized,
)
default:
answer()
}
}
// handleListBans answers GET BansPath with every ban held.
func (h *handler) handleListBans() http.HandlerFunc {
return func(w http.ResponseWriter, _ *http.Request) {
h.answerBans(w, h.ledger.Snapshot())
// endpoint returns the token the request's endpoint needs, and what
// answers the request there; "" when there is no such endpoint.
func (rq *request) endpoint() (string, func()) {
cfg := rq.h.config
method, path := rq.in.Method, rq.in.URL.Path
switch {
case method == http.MethodGet && path == MetricsPath:
return cfg.MetricsToken, func() { rq.h.metrics.ServeHTTP(rq.out, rq.in) }
case method == http.MethodGet && path == BansPath:
return cfg.AdminToken, rq.listBans
case method == http.MethodPost && path == BansPath:
return cfg.AdminToken, rq.addBan
case method == http.MethodDelete && strings.HasPrefix(path, BansPath+"/"):
return cfg.AdminToken, rq.liftBans
case method == http.MethodGet && strings.HasPrefix(path, ClientsPath):
return cfg.AdminToken, rq.showClient
default:
return "", nil
}
}
// hasToken reports whether r carries token, as Authorization: Bearer
// <token>.
func hasToken(r *http.Request, token string) bool {
scheme, sent, _ := strings.Cut(r.Header.Get("Authorization"), " ")
return strings.EqualFold(scheme, "Bearer") &&
subtle.ConstantTimeCompare([]byte(sent), []byte(token)) == 1
}
// listBans answers GET BansPath with every ban held.
func (rq *request) listBans() {
rq.answerBans(rq.h.ledger.Snapshot())
}
// banToAdd is the body of POST BansPath.
type banToAdd struct {
// Netblock is a netblock, or a client's address, which stands for the
@@ -67,44 +126,44 @@ type banToAdd struct {
Reason string `json:"reason"`
}
// handleAddBan answers POST BansPath: it bans the netblock the body names,
// as an admin, from now for the duration the body gives, with its reason,
// addBan answers POST BansPath: it bans the netblock the body names, as
// an admin, from now for the duration the body gives, with its reason,
// and answers with that ban.
func (h *handler) handleAddBan() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
toAdd, err := readBanToAdd(r)
func (rq *request) addBan() {
toAdd, err := rq.readBanToAdd()
var (
netblock netip.Prefix
expires time.Time
now = h.now()
)
var (
netblock netip.Prefix
expires time.Time
now = rq.h.now()
)
if err == nil {
netblock, err = h.parseBanNetblock(toAdd.Netblock)
}
if err == nil {
expires, err = expiry(toAdd.Duration, now)
}
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
ban := h.ledger.BanForAdmin(netblock, now, expires, toAdd.Reason)
h.answerBans(w, []bans.Ban{ban})
if err == nil {
netblock, err = rq.h.parseBanNetblock(toAdd.Netblock)
}
if err == nil {
expires, err = expiry(toAdd.Duration, now)
}
if err != nil {
http.Error(rq.out, err.Error(), http.StatusBadRequest)
return
}
ban := rq.h.ledger.BanForAdmin(netblock, now, expires, toAdd.Reason)
rq.answerBans([]bans.Ban{ban})
}
// readBanToAdd reads the body of POST BansPath: a JSON object with
// nothing but whitespace after it, in at most banBodyMaxBytes.
func readBanToAdd(r *http.Request) (banToAdd, error) {
func (rq *request) readBanToAdd() (banToAdd, error) {
data, err := io.ReadAll(
http.MaxBytesReader(nil, r.Body, banBodyMaxBytes),
http.MaxBytesReader(nil, rq.in.Body, banBodyMaxBytes),
)
rq.requestBytes = int64(len(data))
if err != nil {
return banToAdd{}, fmt.Errorf("%w: %w", errNotBanToAdd, err)
}
@@ -165,11 +224,7 @@ func (h *handler) parseBanNetblock(value string) (netip.Prefix, error) {
return netip.Prefix{}, fmt.Errorf("netblock %q %w", value, errZone)
}
return banNetblock(
h.config.BanScopeV4Prefix,
h.config.IPv6GroupPrefix,
addr,
), nil
return banNetblock(h.config, addr), nil
}
// expiry returns when a ban made at now for duration ends: duration
@@ -191,31 +246,29 @@ func expiry(duration string, now time.Time) (time.Time, error) {
return now.Add(length), nil
}
// handleLiftBans answers DELETE BansPath/<client>: it lifts every ban
// active on a netblock the client's address is in, and answers with those
// bans, or with 404 when none is active.
func (h *handler) handleLiftBans() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
client, err := pathAddress(r.URL.Path, BansPath+"/")
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
// liftBans answers DELETE BansPath/<client>: it lifts every ban active on
// a netblock the client's address is in, and answers with those bans, or
// with 404 when none is active.
func (rq *request) liftBans() {
client, err := pathAddress(rq.in.URL.Path, BansPath+"/")
if err != nil {
http.Error(rq.out, err.Error(), http.StatusBadRequest)
return
}
lifted := h.ledger.Lift(client, h.now())
if len(lifted) == 0 {
http.Error(
w,
"no ban is active on "+client.String(),
http.StatusNotFound,
)
return
}
h.answerBans(w, lifted)
return
}
lifted := rq.h.ledger.Lift(client, rq.h.now())
if len(lifted) == 0 {
http.Error(
rq.out,
"no ban is active on "+client.String(),
http.StatusNotFound,
)
return
}
rq.answerBans(lifted)
}
// clientAnswer is the answer to GET ClientsPath<ip>: the client the
@@ -227,31 +280,27 @@ type clientAnswer struct {
Bans []state.BanEntry `json:"bans"`
}
// handleShowClient answers GET ClientsPath<ip> with what waf knows of the
// client: its counters, its history, which holds its country as last looked
// up and its offences, and its bans with their notes.
func (h *handler) handleShowClient() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
addr, err := pathAddress(r.URL.Path, ClientsPath)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
// showClient answers GET ClientsPath<ip> with what waf knows of the client:
// its counters, its history, which holds its country as last looked up and its
// offences, and its bans with their notes.
func (rq *request) showClient() {
addr, err := pathAddress(rq.in.URL.Path, ClientsPath)
if err != nil {
http.Error(rq.out, err.Error(), http.StatusBadRequest)
return
}
answer := clientAnswer{
Bans: state.BanEntries(h.ledger.Covering(addr)),
}
client, seen := h.limiter.Client(
clientGroup(h.config.IPv6GroupPrefix, addr),
)
if seen {
answer.Client = &client
}
h.answerJSON(w, answer)
return
}
answer := clientAnswer{
Bans: state.BanEntries(rq.h.ledger.Covering(addr)),
}
client, seen := rq.h.limiter.Client(clientGroup(rq.h.config, addr))
if seen {
answer.Client = &client
}
rq.answerJSON(answer)
}
// pathAddress reads the client's address that follows prefix in path.
@@ -267,27 +316,27 @@ func pathAddress(path, prefix string) (netip.Addr, error) {
}
// answerBans answers with held under bans, as bans.json lists them.
func (h *handler) answerBans(w http.ResponseWriter, held []bans.Ban) {
h.answerJSON(w, struct {
func (rq *request) answerBans(held []bans.Ban) {
rq.answerJSON(struct {
Bans []state.BanEntry `json:"bans"`
}{state.BanEntries(held)})
}
// answerJSON answers with value as indented JSON.
func (h *handler) answerJSON(w http.ResponseWriter, value any) {
func (rq *request) answerJSON(value any) {
body, err := json.MarshalIndent(value, "", " ")
if err != nil {
h.processLog.Error(
rq.h.processLog.Error(
"encoding an answer failed",
"error",
err.Error(),
)
http.Error(w, http.StatusText(http.StatusInternalServerError),
http.Error(rq.out, http.StatusText(http.StatusInternalServerError),
http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write(append(body, '\n'))
rq.out.Header().Set("Content-Type", "application/json")
_, _ = rq.out.Write(append(body, '\n'))
}
+2 -36
View File
@@ -14,7 +14,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/state"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
const (
@@ -38,10 +37,10 @@ func TestAdminEndpointsAreOffWhileTheTokenIsUnset(t *testing.T) {
"",
map[string]string{metricsToken: token},
)
server.Ledger.Ban(
server.Ledger.BanForLimit(
netip.MustParsePrefix(otherClient+"/32"),
clk.Now(),
bans.Grounds{Cause: bans.CauseLimit},
bans.Notes{},
)
before := server.Ledger.Snapshot()
@@ -334,39 +333,6 @@ func TestBanToAddThatCannotBeReadIsRefused(t *testing.T) {
}
}
func TestBanToAddIsLoggedWithTheBytesOfItsBody(t *testing.T) {
t.Parallel()
s, _, _ := startWithClock(
t,
"",
map[string]string{adminToken: adminSecret},
)
s.admin(http.MethodPost, waf.BansPath, banOtherClient, http.StatusOK)
line := s.out.endedLines(t, s.sent)[s.sent-1]
if line.RequestBytes != int64(len(banOtherClient)) {
t.Errorf(
"the ban to add was logged with request_bytes %d, want %d",
line.RequestBytes,
len(banOtherClient),
)
}
// The health endpoint reads none of a body sent to it.
s.ownEndpointRequest("", http.MethodGet, wafprotocol.HealthPath, "ok",
http.StatusOK)
line = s.out.endedLines(t, s.sent)[s.sent-1]
if line.RequestBytes != 0 {
t.Errorf(
"the health request was logged with request_bytes %d, want 0",
line.RequestBytes,
)
}
}
func TestClientEndpointShowsTheClientAndItsBans(t *testing.T) {
t.Parallel()
+5 -3
View File
@@ -130,9 +130,11 @@ func TestObserveModeRaisesTheBanAlertsItWouldHave(t *testing.T) {
// A ban for a clear sign of attack, which a request under it would make
// permanent.
group := netip.MustParsePrefix(ipv6Group)
attackBan, _ := server.Ledger.Ban(group, start, bans.Grounds{
Cause: bans.CauseAttack, Notes: bans.Notes{RuleID: "probe"},
})
attackBan, _ := server.Ledger.BanForAttack(
group,
start,
bans.Notes{RuleID: "probe"},
)
// The third request breaks the limit, and so does the fourth, within the
// cooldown, which raises nothing. The probe is a clear sign of attack.
+25 -63
View File
@@ -26,7 +26,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
const (
@@ -85,7 +84,7 @@ func TestSixteenThousandCookiesAreRefusedBeforeTheCoreRuleSetReadsThem(
got := do(t, req)
wantDisposition(t, got, tooLarge, wafprotocol.DispositionDeny)
wantDisposition(t, got, tooLarge, waf.DispositionDeny)
line := s.out.requestLine(t)
wantLine(t, line, tooLarge, requestlog.ActionRequestHeaderTooLarge)
@@ -165,14 +164,9 @@ func TestARequestLineOverTheLimitIsRefusedWith414BeforeAnyOtherCheck(
line := s.out.requestLine(t)
if tc.status == http.StatusOK {
wantDisposition(
t,
got,
tc.status,
wafprotocol.DispositionAllow,
)
wantDisposition(t, got, tc.status, waf.DispositionAllow)
} else {
wantDisposition(t, got, tc.status, wafprotocol.DispositionDeny)
wantDisposition(t, got, tc.status, waf.DispositionDeny)
}
wantLine(t, line, tc.status, tc.action)
@@ -225,14 +219,9 @@ func TestAHeaderLineOverTheLimitIsRefusedWith431(t *testing.T) {
got := do(t, req)
if tc.refused {
wantDisposition(
t,
got,
tooLarge,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, tooLarge, waf.DispositionDeny)
} else {
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
}
})
}
@@ -255,7 +244,10 @@ func TestRequestLineRefusalNamesItsHeuristicLengthAndLimit(t *testing.T) {
func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
t.Parallel()
repeated := slices.Repeat([]string{"1"}, headerFieldLimit)
repeated := make([]string, headerFieldLimit)
for i := range repeated {
repeated[i] = "1"
}
for _, tc := range []struct {
name string
@@ -302,12 +294,7 @@ func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
line := s.out.requestLine(t)
if tc.refused {
wantDisposition(
t,
got,
tooLarge,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, tooLarge, waf.DispositionDeny)
wantLine(
t,
line,
@@ -318,12 +305,7 @@ func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
return
}
wantDisposition(
t,
got,
http.StatusOK,
wafprotocol.DispositionAllow,
)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
wantLine(t, line, http.StatusOK, requestlog.ActionForward)
if line.DurationCoreRuleSetMS == nil {
@@ -350,12 +332,7 @@ func TestHeaderLimitsHoldWhateverTheCoreRuleSetSettingsSay(t *testing.T) {
req := s.question(siteA, client, "/")
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(
t,
do(t, req),
tooLarge,
wafprotocol.DispositionDeny,
)
wantDisposition(t, do(t, req), tooLarge, waf.DispositionDeny)
})
}
}
@@ -371,12 +348,7 @@ func TestClientInTheAllowNetsSkipsTheHeaderLimits(t *testing.T) {
req := s.question(siteA, client, "/")
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(
t,
do(t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
)
wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow)
}
func TestObserveModeAllowsARequestOverTheHeaderLimitsUnread(t *testing.T) {
@@ -392,7 +364,7 @@ func TestObserveModeAllowsARequestOverTheHeaderLimitsUnread(t *testing.T) {
got := do(t, req)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
line := s.out.requestLine(t)
if line.WouldAction != requestlog.ActionRequestHeaderTooLarge ||
@@ -414,14 +386,14 @@ func TestTheRateLimitsDoNotCountRequestsOverTheHeaderLimits(t *testing.T) {
for range 2 {
req := s.question(siteA, client, "/")
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(t, do(t, req), tooLarge, wafprotocol.DispositionDeny)
wantDisposition(t, do(t, req), tooLarge, waf.DispositionDeny)
}
wantDisposition(
t,
s.ask(client, "/"),
http.StatusOK,
wafprotocol.DispositionAllow,
waf.DispositionAllow,
)
}
@@ -494,12 +466,7 @@ func TestTheHeaderLimitsComeBeforeEveryCheckThatRefuses(t *testing.T) {
req := s.question(siteA, client, tc.path)
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(
t,
do(t, req),
tooLarge,
wafprotocol.DispositionDeny,
)
wantDisposition(t, do(t, req), tooLarge, waf.DispositionDeny)
wantLine(
t,
s.out.requestLine(t),
@@ -536,16 +503,11 @@ func TestARequestOverTheHeaderLimitsRunsNoOtherCheck(t *testing.T) {
over := s.question(siteA, client, "/")
over.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(t, do(t, over), tooLarge, wafprotocol.DispositionDeny)
wantDisposition(t, do(t, over), tooLarge, waf.DispositionDeny)
within := s.question(siteA, client, "/")
within.Header.Set(cookieHeader, cookies(cookieLimit))
wantDisposition(
t,
do(t, within),
http.StatusOK,
wafprotocol.DispositionAllow,
)
wantDisposition(t, do(t, within), http.StatusOK, waf.DispositionAllow)
lines := s.out.requestLines(t, 2)
@@ -602,7 +564,7 @@ func TestNotWaitingForGeoJSNeverLowersAClientsLimit(t *testing.T) {
t,
s.ask(fromDE, "/"),
http.StatusForbidden,
wafprotocol.DispositionDeny,
waf.DispositionDeny,
)
line := s.out.requestLine(t)
@@ -639,7 +601,7 @@ func TestAClientGeoJSPlacedInNoCountryCountsAsUnknownWithoutWaiting(
t,
s.ask(client, "/"),
http.StatusForbidden,
wafprotocol.DispositionDeny,
waf.DispositionDeny,
)
line := s.out.requestLine(t)
@@ -668,7 +630,7 @@ func TestObserveModeAsksGeoJSNothingForARequestOverTheHeaderLimits(
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
got := do(t, req)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
if len(asked()) != 0 || got.header.Get("X-Client-Asn") != "" {
t.Errorf(
@@ -710,7 +672,7 @@ func askDirectly(
http.NoBody,
)
req.RemoteAddr = net.JoinHostPort(rproxyPeer, "1234")
req.Header.Set(wafprotocol.ClientHeader, client)
req.Header.Set(waf.ClientHeader, client)
req.Header.Set(requestIDHeader, "the-id")
change(req)
@@ -724,10 +686,10 @@ func askDirectly(
// banClient bans client at now.
func banClient(_ *testing.T, server *waf.Server, now time.Time) {
server.Ledger.Ban(
server.Ledger.BanForLimit(
netip.MustParsePrefix(client+"/32"),
now,
bans.Grounds{Cause: bans.CauseLimit},
bans.Notes{},
)
}
+100 -164
View File
@@ -1,7 +1,7 @@
package waf
// The heuristic active_ban, and banMaker, which makes the bans of several
// heuristics. active_ban reads SWW_WAF_MODE; the bans are the ban ledger's.
// The heuristic active_ban, and the code that makes bans, which several
// heuristics share. active_ban reads SWW_WAF_MODE; the bans are the ban ledger's.
import (
"context"
@@ -11,7 +11,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
@@ -26,22 +25,18 @@ func (rq *request) banResponse(action string) *refusal {
// in, in force at now. It notes for the log line when that ban ends. A request
// that makes the ban permanent, or in observe mode would have, raises the alert
// for it. Its figures are the ban's netblock and when it ends.
type activeBan struct {
banMaker banMaker
}
type activeBan struct{}
func (activeBan) name() string { return "active_ban" }
func (a activeBan) check(
func (activeBan) check(
_ context.Context,
rq *request,
now time.Time,
) *verdict {
ledger := a.banMaker.ledger
check := ledger.Check
if a.banMaker.observe {
check = ledger.Find // the ban refuses nothing, and stays as it is
check := rq.h.ledger.Check
if rq.h.config.Observe {
check = rq.h.ledger.Find // the ban refuses nothing, and stays as it is
}
ban, banned, madePermanent := check(rq.client, now)
@@ -51,7 +46,7 @@ func (a activeBan) check(
if madePermanent {
ban.Expires = time.Time{} // the ban made permanent, which Find leaves as it is
a.banMaker.alertBan(rq, ban)
rq.alertBan(ban)
}
if !banned {
@@ -86,55 +81,31 @@ func limitFigures(hit ratelimit.Hit) map[string]any {
// countedBytes returns the request's bytes, once it has ended, as the
// byte limits and the anomaly thresholds count them: the response's body
// bytes, the request's, or both, as bytesCounted, SWW_WAF_BYTES_COUNTED,
// says.
func (rq *request) countedBytes(bytesCounted config.BytesCounted) int64 {
// bytes, the request's, or both, as SWW_WAF_BYTES_COUNTED says.
func (rq *request) countedBytes() int64 {
response, request := rq.out.bytes, rq.requestBytes
switch bytesCounted {
case config.BytesCountedResponse:
switch rq.h.config.BytesCounted {
case "response":
return response
case config.BytesCountedRequest:
case "request":
return request
case config.BytesCountedBoth:
default: // both
return response + request
}
return response + request
}
// banMaker makes the bans of the heuristics that ban a client, in the ban
// ledger, and raises the alert for each. In observe mode it makes no ban, and
// raises the alert for the ban it would have made.
type banMaker struct {
ledger *bans.Ledger
// limiter is the table of clients the rate limits count. A ban for a limit
// sets the client's counters back to zero, and a ban's notes give how many
// requests its netblock has sent.
limiter *ratelimit.Limiter
metrics *metrics.Metrics
alerts *alerts.Queue
// observe is true in observe mode (SWW_WAF_MODE).
observe bool
// banResponse is SWW_WAF_BAN_RESPONSE, the status the notes of a ban for a
// clear sign of attack give.
banResponse int
// banScopeV4Prefix is SWW_WAF_BAN_SCOPE_V4_PREFIX and ipv6GroupPrefix
// SWW_WAF_IPV6_GROUP_PREFIX, which set the netblock a ban covers.
banScopeV4Prefix int
ipv6GroupPrefix int
}
// banForLimit bans the netblock of rq's client at now for a broken limit, the
// banForLimit bans the client's netblock at now for a broken limit, the
// one hit names, notes the offence for the log line and counts the hit in
// the metrics. status is what the client was sent, or is sent:
// SWW_WAF_BAN_RESPONSE for a request over a rate limit, what the report says for
// one whose bytes broke a byte limit, the refusal for one that broke the
// refusal limit. The ban's notes give the client's limit percentage for a
// rate limit or a byte limit; the refusal limit is not lowered. The ban sets
// the client's counters back to zero. In observe mode, banClient says what
// happens instead.
func (b banMaker) banForLimit(
rq *request,
// the client's counters back to zero. In observe mode it makes no ban and
// sets nothing back, and raises the alert for the ban it would have made,
// if that alert would be sent.
func (rq *request) banForLimit(
now time.Time,
hit ratelimit.Hit,
status int,
@@ -149,14 +120,25 @@ func (b banMaker) banForLimit(
}
rq.line.Offence = requestlog.OffenceLimit
rq.h.metrics.LimitHit(hit)
b.metrics.LimitHit(hit)
netblock := banNetblock(rq.h.config, rq.client)
if rq.h.config.Observe &&
!rq.wouldAlertBan(netblock, now, bans.CauseLimit) {
return
}
notes := bans.Notes{
Kind: hit.Kind,
Limit: hit.Limit,
Window: hit.Window,
Count: hit.Count,
ASN: rq.line.ASN,
ASName: rq.line.ASName,
Country: rq.line.Country,
Kind: hit.Kind,
Limit: hit.Limit,
Window: hit.Window,
Count: hit.Count,
Reputation: rq.reputation,
Request: rq.noted(now, status),
Requests: rq.netblockRequests(netblock),
}
switch hit.Kind {
@@ -166,130 +148,86 @@ func (b banMaker) banForLimit(
notes.LimitPercent, notes.LimitPercentSetting = rq.bytesLimitPercent.logged()
}
b.banClient(
rq,
now,
status,
bans.Grounds{Cause: bans.CauseLimit, Notes: notes},
)
}
// banForAttack bans the netblock of rq's client at now for a clear sign of
// attack, which notes name: the ban rule that matched, or the trap path
// asked for. It returns the action for that ban. In observe mode, banClient
// says what happens instead.
func (b banMaker) banForAttack(
rq *request, now time.Time, notes bans.Notes,
) string {
return b.banClient(
rq,
now,
b.banResponse,
bans.Grounds{Cause: bans.CauseAttack, Notes: notes},
)
}
// banClient bans the netblock of rq's client at now on grounds, and returns
// the action for that ban. It adds to the notes what the request tells of the
// client, with status, what the client was sent. A ban for a broken limit
// sets the client's counters back to zero.
//
// In observe mode it makes no ban and sets nothing back. It raises the
// alert for the ban it would have made, if that alert would be sent.
func (b banMaker) banClient(
rq *request,
now time.Time,
status int,
grounds bans.Grounds,
) string {
netblock := banNetblock(b.banScopeV4Prefix, b.ipv6GroupPrefix, rq.client)
if b.observe {
if !b.wouldAlertBan(netblock, now, grounds.Cause) {
return banAction(
b.ledger.WouldBePermanent(netblock, now, grounds.Cause),
)
if rq.h.config.Observe {
ban, wouldBan := rq.h.ledger.WouldBanForLimit(netblock, now, notes)
if wouldBan {
rq.alertBan(ban)
}
grounds.Notes = b.withClientNotes(
rq,
grounds.Notes,
now,
status,
netblock,
)
return
}
ban, wouldBan := b.ledger.WouldBan(netblock, now, grounds)
ban, made := rq.h.ledger.BanForLimit(netblock, now, notes)
rq.h.limiter.Reset(clientGroup(rq.h.config, rq.client))
rq.line.BanExpires = banExpires(ban)
if made {
rq.alertBan(ban)
}
}
// banForAttack bans the client's netblock at now for a clear sign of
// attack, which notes name: the ban rule that matched, or the trap path
// asked for. It fills in the rest of the notes. In observe mode it makes
// no ban, and raises the alert for the ban it would have made, if that
// alert would be sent. It returns the action for that ban.
func (rq *request) banForAttack(now time.Time, notes bans.Notes) string {
netblock := banNetblock(rq.h.config, rq.client)
if rq.h.config.Observe &&
!rq.wouldAlertBan(netblock, now, bans.CauseAttack) {
return banAction(
rq.h.ledger.WouldBePermanent(netblock, now, bans.CauseAttack),
)
}
notes.ASN = rq.line.ASN
notes.ASName = rq.line.ASName
notes.Country = rq.line.Country
notes.Reputation = rq.reputation
notes.Request = rq.noted(now, rq.h.config.BanResponse)
notes.Requests = rq.netblockRequests(netblock)
if rq.h.config.Observe {
ban, wouldBan := rq.h.ledger.WouldBanForAttack(netblock, now, notes)
if wouldBan {
b.alertBan(rq, ban)
rq.alertBan(ban)
}
return banAction(ban.Permanent())
}
grounds.Notes = b.withClientNotes(
rq,
grounds.Notes,
now,
status,
netblock,
)
ban, made := b.ledger.Ban(netblock, now, grounds)
if grounds.Cause == bans.CauseLimit {
b.limiter.Reset(clientGroup(b.ipv6GroupPrefix, rq.client))
}
ban, made := rq.h.ledger.BanForAttack(netblock, now, notes)
rq.line.BanExpires = banExpires(ban)
if made {
b.alertBan(rq, ban)
rq.alertBan(ban)
}
return banAction(ban.Permanent())
}
// withClientNotes is notes with what rq tells of its client added: its AS,
// country and reputation, the request at now with status, and how many
// requests netblock, the client's, has sent.
func (b banMaker) withClientNotes(
rq *request,
notes bans.Notes,
now time.Time,
status int,
netblock netip.Prefix,
) bans.Notes {
notes.ASN = rq.line.ASN
notes.ASName = rq.line.ASName
notes.Country = rq.line.Country
notes.Reputation = rq.reputation
notes.Request = rq.noted(now, status)
notes.Requests = b.netblockRequests(netblock)
return notes
}
// wouldAlertBan reports whether the alert for a ban on netblock for cause
// made at now would be sent. In observe mode the ban the request would
// have made is worked out only then, at most once per
// SWW_WAF_ALERT_COOLDOWN and never with no webhook set: its notes count the
// netblock's requests, which can mean going through every client.
func (b banMaker) wouldAlertBan(
func (rq *request) wouldAlertBan(
netblock netip.Prefix, now time.Time, cause string,
) bool {
event := alerts.EventBan
if b.ledger.WouldBePermanent(netblock, now, cause) {
if rq.h.ledger.WouldBePermanent(netblock, now, cause) {
event = alerts.EventPermanentBan
}
return b.alerts.WouldSend(event, netblock)
return rq.h.alerts.WouldSend(event, netblock)
}
// alertBan raises the alert for ban, which rq made, or made permanent:
// permanent_ban for a permanent ban, ban for another. Its detail gives the
// ban's cause, when it ends, and its notes, and in observe mode, where ban is
// the ban that would have been made, or made permanent, mode, observe.
func (b banMaker) alertBan(rq *request, ban bans.Ban) {
// alertBan raises the alert for ban, which the request made, or made
// permanent: permanent_ban for a permanent ban, ban for another. Its
// detail gives the ban's cause, when it ends, and its notes, and in
// observe mode, where ban is the ban that would have been made, or made
// permanent, mode, observe.
func (rq *request) alertBan(ban bans.Ban) {
event := alerts.EventBan
if ban.Permanent() {
event = alerts.EventPermanentBan
@@ -298,11 +236,11 @@ func (b banMaker) alertBan(rq *request, ban bans.Ban) {
detail := map[string]any{
"cause": ban.Cause, "ban_expires": banExpires(ban), "notes": ban.Notes,
}
if b.observe {
if rq.h.config.Observe {
detail["mode"] = "observe"
}
b.alerts.Raise(alerts.Alert{
rq.h.alerts.Raise(alerts.Alert{
Event: event,
Host: rq.site,
Client: rq.client,
@@ -315,13 +253,6 @@ func (b banMaker) alertBan(rq *request, ban bans.Ban) {
})
}
// netblockRequests is how many requests netblock has sent since it was
// first seen, the request being banned for included: the histories count it
// only once it has ended.
func (b banMaker) netblockRequests(netblock netip.Prefix) int64 {
return b.limiter.Requests(netblock) + 1
}
// noted is the request, at now, with status, what the client was sent, or
// in observe mode would have been, as the notes of the ban it makes keep
// it.
@@ -336,18 +267,23 @@ func (rq *request) noted(now time.Time, status int) bans.Request {
}
}
// netblockRequests is how many requests netblock has sent since it was
// first seen, this one included: the histories count it only once it has
// ended.
func (rq *request) netblockRequests(netblock netip.Prefix) int64 {
return rq.h.limiter.Requests(netblock) + 1
}
// banNetblock is the netblock a ban on client covers: its IPv4 address,
// widened to banScopeV4Prefix, SWW_WAF_BAN_SCOPE_V4_PREFIX, or the IPv6 group
// clientGroup counts it in, of length ipv6GroupPrefix.
func banNetblock(
banScopeV4Prefix, ipv6GroupPrefix int, client netip.Addr,
) netip.Prefix {
// widened to SWW_WAF_BAN_SCOPE_V4_PREFIX, or the IPv6 group clientGroup
// counts it in.
func banNetblock(cfg *config.WAF, client netip.Addr) netip.Prefix {
addr := client.Unmap()
if addr.Is4() {
return netip.PrefixFrom(addr, banScopeV4Prefix).Masked()
return netip.PrefixFrom(addr, cfg.BanScopeV4Prefix).Masked()
}
return clientGroup(ipv6GroupPrefix, addr)
return clientGroup(cfg, addr)
}
// banExpires is when ban ends, as the log line gives it: a time, or
+5 -4
View File
@@ -548,10 +548,6 @@ func (s *sender) requestWithHeader(
// body is sent as it is after the headers, header holding its
// Content-Length or Transfer-Encoding. header may hold several lines,
// separated by "\r\n". It returns the whole answer.
//
// It waits for the answer as long as it takes, since a busy host can slow
// the Core Rule Set past any shorter wait. go test's -timeout ends a test
// whose answer never comes.
func (s *sender) requestWithBody(
method, from, path, header, body string, status int, action string,
) (logLine, answer) {
@@ -570,6 +566,11 @@ func (s *sender) requestWithBody(
send(s.t, conn, method+" "+path+" HTTP/1.1\r\nHost: "+appHost+"\r\n"+
forwardedFor+": "+from+"\r\n"+header+"\r\n"+body)
err := conn.SetReadDeadline(time.Now().Add(waitLimit))
if err != nil {
s.t.Fatalf("set read deadline: %v", err)
}
var got answer
res, err := http.ReadResponse(bufio.NewReader(conn), nil)
+12 -12
View File
@@ -54,7 +54,7 @@ func (rq *request) runBesideLookup(
) *verdict {
var running atomic.Pointer[string]
running.Store(new(rq.h.trapPaths.name()))
running.Store(new(trapPaths{}.name()))
found := make(chan *besideLookupFindings, 1)
@@ -92,7 +92,7 @@ func (rq *request) runBesideLookup(
return v
}
return rq.runHeuristics(ctx, rq.h.afterBesideLookup, rq.h.now())
return rq.runHeuristics(ctx, afterBesideLookup(), rq.h.now())
}
// findBesideLookup does the work of the heuristics of besideLookup on the
@@ -105,39 +105,39 @@ func (rq *request) findBesideLookup(
) *besideLookupFindings {
findings := &besideLookupFindings{}
findings.trapPath = rq.h.trapPaths.trapPath(rq)
findings.trapPath = rq.trapPath()
if findings.trapPath != "" {
findings.refuses = true
return findings
}
running.Store(new(rq.h.ruleFiles.name()))
running.Store(new(ruleFiles{}.name()))
findings.rules = rq.h.ruleFiles.rules.Match(rq.in)
findings.rules = rq.h.rules.Match(rq.in)
if refusedByRule(findings.rules) {
findings.refuses = true
return findings
}
coreRuleSet := rq.h.coreRuleSet
running.Store(new(coreRuleSet.name()))
running.Store(new(coreRuleSet{}.name()))
if coreRuleSet.mode == config.CoreRuleSetModeOff ||
pathExempt(rq.in, coreRuleSet.exemptPaths) {
cfg := rq.h.config
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff ||
pathExempt(rq.in, cfg.CoreRuleSetExemptPaths) {
return findings
}
start := time.Now()
// A question's body has been read into memory, so that reading it here
// cannot fail.
findings.inspection, findings.read, _ = coreRuleSet.inspector.Inspect(
findings.inspection, findings.read, _ = rq.h.coreRuleSet.Inspect(
rq.in, rq.client, rq.in.Body,
)
findings.inspectionTime = time.Since(start)
findings.inspected = true
findings.refuses = coreRuleSet.refusedByCoreRuleSet(findings.inspection)
findings.refuses = rq.refusedByCoreRuleSet(findings.inspection)
return findings
}
@@ -151,7 +151,7 @@ func (rq *request) applyFindings(
) *verdict {
rq.findings = findings
return rq.runHeuristics(ctx, rq.h.besideLookup, rq.h.now())
return rq.runHeuristics(ctx, besideLookup(), rq.h.now())
}
// timedOut refuses the request for the heuristics of besideLookup not having
+22 -44
View File
@@ -2,7 +2,6 @@ package waf
import (
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/reputation"
)
// whole is the percentage of each limit a client gets when no biased
@@ -30,25 +29,6 @@ func biasedThresholdsSet(cfg *config.WAF) bool {
cfg.ASNLimitPercentURL != ""
}
// biasedThresholds are the settings and the lists by which limitPercentages
// lowers a client's limits.
type biasedThresholds struct {
// lists hold the file SWW_WAF_ASN_LIMIT_PERCENT_URL names.
lists *reputation.Lists
tagLimitPercent map[string]int64 // SWW_WAF_TAG_LIMIT_PERCENT
asnBytesLimitPercent map[string]int64 // SWW_WAF_ASN_BYTES_LIMIT_PERCENT
countryBytesLimitPercent map[string]int64 // SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT
unknownCountryLimitPercent int64 // SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT
// blocklistAction and blocklistLimitPercent are SWW_WAF_BLOCKLIST_ACTION
// and its percentage.
blocklistAction config.ReputationAction
blocklistLimitPercent int64
// dnsblAbuseIPDBAction and dnsblAbuseIPDBLimitPercent are
// SWW_WAF_DNSBL_ABUSEIPDB_ACTION and its percentage.
dnsblAbuseIPDBAction config.ReputationAction
dnsblAbuseIPDBLimitPercent int64
}
// limitPercentages returns the client's limit percentages, for the rate
// limits and for the byte limits, by its tags, its AS number and country
// as looked up, each "" when unknown, and the blocklists, DNSBL zones and
@@ -65,50 +45,48 @@ type biasedThresholds struct {
// the byte limits, SWW_WAF_ASN_BYTES_LIMIT_PERCENT and
// SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT take the place of the first three for
// an AS number or a country they list.
func (b biasedThresholds) limitPercentages(
rq *request,
) (percentage, percentage) {
func (rq *request) limitPercentages() (percentage, percentage) {
cfg := rq.h.config
asn, country := rq.line.ASN, rq.line.Country
unknown := percentage{percent: whole}
if country == "" && !rq.skippedLookupWait {
unknown = percentage{
b.unknownCountryLimitPercent,
config.UnknownCountryLimitPercentSetting,
cfg.UnknownCountryLimitPercent,
"SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT",
}
}
fetched := percentage{percent: whole}
if percent, listed := b.lists.ASNLimitPercent(asn); listed {
fetched = percentage{percent, config.ASNLimitPercentURLSetting}
if percent, listed := rq.h.lists.ASNLimitPercent(asn); listed {
fetched = percentage{percent, "SWW_WAF_ASN_LIMIT_PERCENT_URL"}
}
blocklisted := percentage{percent: whole}
if rq.blocklisted &&
b.blocklistAction == config.ReputationActionLimit {
if rq.blocklisted && cfg.BlocklistAction == "limit" {
blocklisted = percentage{
b.blocklistLimitPercent,
config.BlocklistActionSetting,
cfg.BlocklistLimitPercent,
"SWW_WAF_BLOCKLIST_ACTION",
}
}
dnsblAbuseIPDBListed := percentage{percent: whole}
if (rq.dnsblListed || rq.abuseIPDBHit) &&
b.dnsblAbuseIPDBAction == config.ReputationActionLimit {
cfg.DNSBLAbuseIPDBAction == "limit" {
dnsblAbuseIPDBListed = percentage{
b.dnsblAbuseIPDBLimitPercent,
config.DNSBLAbuseIPDBActionSetting,
cfg.DNSBLAbuseIPDBLimitPercent,
"SWW_WAF_DNSBL_ABUSEIPDB_ACTION",
}
}
const tagSetting = config.TagLimitPercentSetting
const tagSetting = "SWW_WAF_TAG_LIMIT_PERCENT"
asnRequests := lowest(
given(b.tagLimitPercent, asnTag(asn), tagSetting),
given(cfg.TagLimitPercent, asnTag(asn), tagSetting),
fetched,
)
countryRequests := given(
b.tagLimitPercent,
cfg.TagLimitPercent,
countryTag(country),
tagSetting,
)
@@ -119,25 +97,25 @@ func (b biasedThresholds) limitPercentages(
if tag != asnTag(asn) && tag != countryTag(country) {
otherTags = lowest(
otherTags,
given(b.tagLimitPercent, tag, tagSetting),
given(cfg.TagLimitPercent, tag, tagSetting),
)
}
}
asnBytes, countryBytes := asnRequests, countryRequests
if _, listed := b.asnBytesLimitPercent[asn]; listed {
if _, listed := cfg.ASNBytesLimitPercent[asn]; listed {
asnBytes = given(
b.asnBytesLimitPercent,
cfg.ASNBytesLimitPercent,
asn,
config.ASNBytesLimitPercentSetting,
"SWW_WAF_ASN_BYTES_LIMIT_PERCENT",
)
}
if _, listed := b.countryBytesLimitPercent[country]; listed {
if _, listed := cfg.CountryBytesLimitPercent[country]; listed {
countryBytes = given(
b.countryBytesLimitPercent,
cfg.CountryBytesLimitPercent,
country,
config.CountryBytesLimitPercentSetting,
"SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT",
)
}
+1 -2
View File
@@ -19,7 +19,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// The biased thresholds.
@@ -590,7 +589,7 @@ func serveFromDE(
t.Helper()
req := httptest.NewRequestWithContext(t.Context(), method, "/", body)
req.Header.Set(wafprotocol.ClientHeader, fromDE)
req.Header.Set(waf.ClientHeader, fromDE)
req.Header.Set(requestIDHeader, "the-id")
server.Handler.ServeHTTP(httptest.NewRecorder(), req)
+7 -13
View File
@@ -7,8 +7,6 @@ import (
"context"
"time"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
@@ -16,24 +14,20 @@ import (
// and refuses the request when SWW_WAF_BLOCKLIST_ACTION is deny. Being limit, it
// lowers the client's limits instead (see limitPercentages), and being log, it
// does nothing more. Its figure is the blocklists that list the client.
type blocklists struct {
lists *reputation.Lists
action config.ReputationAction // SWW_WAF_BLOCKLIST_ACTION
hitNoter reputationHitNoter
}
type blocklists struct{}
func (blocklists) name() string { return "blocklists" }
func (b blocklists) check(
func (blocklists) check(
_ context.Context,
rq *request,
_ time.Time,
) *verdict {
listedBy := b.lists.ListedBy(rq.client)
listedBy := rq.h.lists.ListedBy(rq.client)
rq.blocklisted = len(listedBy) > 0
b.hitNoter.noteListed(rq, listedBy, "listed by a blocklist")
rq.noteListed(listedBy, "listed by a blocklist")
if !b.refusedByBlocklists(listedBy) {
if !rq.refusedByBlocklists(listedBy) {
return nil
}
@@ -46,6 +40,6 @@ func (b blocklists) check(
// refusedByBlocklists reports whether the heuristic blocklists refuses a
// client, given listedBy, the blocklists that list it: it does when one lists
// it and SWW_WAF_BLOCKLIST_ACTION is deny.
func (b blocklists) refusedByBlocklists(listedBy []string) bool {
return len(listedBy) > 0 && b.action == config.ReputationActionDeny
func (rq *request) refusedByBlocklists(listedBy []string) bool {
return len(listedBy) > 0 && rq.h.config.BlocklistAction == deny
}
+6 -14
View File
@@ -6,9 +6,6 @@ package waf
import (
"context"
"time"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
)
// byteLimits counts the request's bytes, as countedBytes gives them, for the
@@ -19,16 +16,11 @@ import (
// the client over a byte limit, as its limit percentage for the byte limits
// lowers it, break it, which bans the client; the response was passed on whole.
// Its figures are the window, the count and the limit.
type byteLimits struct {
limiter *ratelimit.Limiter
bytesCounted config.BytesCounted // SWW_WAF_BYTES_COUNTED
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
banMaker banMaker
}
type byteLimits struct{}
func (byteLimits) name() string { return "byte_limits" }
func (b byteLimits) check(
func (byteLimits) check(
_ context.Context,
rq *request,
now time.Time,
@@ -37,10 +29,10 @@ func (b byteLimits) check(
return nil
}
counts, hit, over := b.limiter.CountBytes(
clientGroup(b.ipv6GroupPrefix, rq.client),
counts, hit, over := rq.h.limiter.CountBytes(
clientGroup(rq.h.config, rq.client),
now,
rq.countedBytes(b.bytesCounted),
rq.countedBytes(),
rq.bytesLimitPercent.percent,
)
rq.line.Counts.MinuteBytes = counts.MinuteBytes
@@ -51,7 +43,7 @@ func (b byteLimits) check(
return nil
}
b.banMaker.banForLimit(rq, now, hit, rq.out.status)
rq.banForLimit(now, hit, rq.out.status)
return &verdict{figures: limitFigures(hit)}
}
+9 -7
View File
@@ -6,7 +6,7 @@ import (
"net/netip"
"slices"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
"sneak.berlin/go/smallwebwaf/internal/config"
)
// peerAddress is the address of the request's TCP peer.
@@ -19,11 +19,14 @@ func peerAddress(r *http.Request) netip.Addr {
return addrPort.Addr().Unmap()
}
// requestIDHeader carries the request's id, from rproxy.
const requestIDHeader = "X-Request-ID"
// requestID is the request's id: the one a question gives, or for any other
// request a new random one.
func requestID(r *http.Request, isQuestion bool) string {
if isQuestion {
return r.Header.Get(wafprotocol.RequestIDHeader)
return r.Header.Get(requestIDHeader)
}
return rand.Text()
@@ -42,13 +45,12 @@ func scheme(r *http.Request, isQuestion bool) string {
// clientGroup is the client a request is counted toward: its IPv4
// address, or its IPv6 group, the netblock its IPv6 address is in of the
// length ipv6GroupPrefix, SWW_WAF_IPV6_GROUP_PREFIX, a /64 by default, since
// one abuser usually holds a whole /64. An IPv4 address in IPv6 form counts as
// IPv4.
func clientGroup(ipv6GroupPrefix int, addr netip.Addr) netip.Prefix {
// length SWW_WAF_IPV6_GROUP_PREFIX sets, a /64 by default, since one abuser
// usually holds a whole /64. An IPv4 address in IPv6 form counts as IPv4.
func clientGroup(cfg *config.WAF, addr netip.Addr) netip.Prefix {
addr = addr.Unmap()
if addr.Is6() {
return netip.PrefixFrom(addr, ipv6GroupPrefix).Masked()
return netip.PrefixFrom(addr, cfg.IPv6GroupPrefix).Masked()
}
return netip.PrefixFrom(addr, addr.BitLen())
+19 -31
View File
@@ -10,7 +10,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
@@ -23,22 +22,11 @@ import (
// exploitsignature, raises the core_rule_set_match alert, and in block mode
// refuses the request with 403, which is an offence its client's history
// counts. Its figures are the score and the threshold.
type coreRuleSet struct {
// inspector is the Core Rule Set, nil while SWW_WAF_CORE_RULE_SET_MODE is
// off.
inspector CoreRuleSetInspector
metrics *metrics.Metrics
alerts *alerts.Queue
mode config.CoreRuleSetMode // SWW_WAF_CORE_RULE_SET_MODE
exemptPaths []string // SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS
scoreThreshold int // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD
observe bool // SWW_WAF_MODE is observe
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
}
type coreRuleSet struct{}
func (coreRuleSet) name() string { return "core_rule_set" }
func (c coreRuleSet) check(
func (coreRuleSet) check(
_ context.Context,
rq *request,
_ time.Time,
@@ -48,6 +36,7 @@ func (c coreRuleSet) check(
return nil
}
cfg := rq.h.config
result := findings.inspection
rq.requestBytes = int64(len(findings.read))
@@ -58,18 +47,18 @@ func (c coreRuleSet) check(
rq.line.CoreRuleSetScore = &result.Score
for _, id := range result.RuleIDs {
c.metrics.CoreRuleSetMatched(string(c.mode), id)
rq.h.metrics.CoreRuleSetMatched(cfg.CoreRuleSetMode, id)
}
threshold := c.scoreThreshold
threshold := cfg.CoreRuleSetScoreThreshold
if threshold == 0 || result.Score < threshold {
return nil
}
rq.addTag(config.ExploitSignatureTag)
c.alertCoreRuleSetMatch(rq, result)
rq.alertCoreRuleSetMatch(result)
if !c.refusedByCoreRuleSet(result) {
if !rq.refusedByCoreRuleSet(result) {
return nil
}
@@ -87,21 +76,20 @@ func (c coreRuleSet) check(
// refusedByCoreRuleSet reports whether coreRuleSet refuses a request the Core
// Rule Set found result of: whether the score is at or over
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD in block mode.
func (c coreRuleSet) refusedByCoreRuleSet(result coreruleset.Result) bool {
threshold := c.scoreThreshold
func (rq *request) refusedByCoreRuleSet(result coreruleset.Result) bool {
cfg := rq.h.config
threshold := cfg.CoreRuleSetScoreThreshold
return c.mode == config.CoreRuleSetModeBlock &&
return cfg.CoreRuleSetMode == config.CoreRuleSetModeBlock &&
threshold != 0 && result.Score >= threshold
}
// alertCoreRuleSetMatch raises the core_rule_set_match alert for rq, which the
// Core Rule Set scored at result, at or over
// alertCoreRuleSetMatch raises the core_rule_set_match alert for the request,
// which the Core Rule Set scored at result, at or over
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD. Its detail gives the rule ids, the
// score, the method and the path with the query, and, for a request that is
// not refused for it, the mode: detect, or observe in observe mode.
func (c coreRuleSet) alertCoreRuleSetMatch(
rq *request, result coreruleset.Result,
) {
func (rq *request) alertCoreRuleSetMatch(result coreruleset.Result) {
detail := map[string]any{
"rule_ids": result.RuleIDs,
"score": result.Score,
@@ -110,17 +98,17 @@ func (c coreRuleSet) alertCoreRuleSetMatch(
}
switch {
case c.mode == config.CoreRuleSetModeDetect:
detail["mode"] = string(config.CoreRuleSetModeDetect)
case c.observe:
case rq.h.config.CoreRuleSetMode == config.CoreRuleSetModeDetect:
detail["mode"] = config.CoreRuleSetModeDetect
case rq.h.config.Observe:
detail["mode"] = "observe"
}
c.alerts.Raise(alerts.Alert{
rq.h.alerts.Raise(alerts.Alert{
Event: alerts.EventCoreRuleSetMatch,
Host: rq.site,
Client: rq.client,
Netblock: clientGroup(c.ipv6GroupPrefix, rq.client),
Netblock: clientGroup(rq.h.config, rq.client),
ASN: rq.line.ASN,
ASName: rq.line.ASName,
Country: rq.line.Country,
+3 -3
View File
@@ -12,7 +12,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// The Core Rule Set's settings the tests set, besides SWW_WAF_CORE_RULE_SET_MODE, and
@@ -395,9 +395,9 @@ func TestGiteasFixesApplyOnlyToAHostnameThatServesGitea(t *testing.T) {
})
wantDisposition(t, do(t, s.question(siteA, client, fileName)),
http.StatusOK, wafprotocol.DispositionAllow)
http.StatusOK, waf.DispositionAllow)
wantDisposition(t, do(t, s.question(siteB, client, fileName)),
http.StatusForbidden, wafprotocol.DispositionDeny)
http.StatusForbidden, waf.DispositionDeny)
lines := s.out.requestLines(t, 2)
wantLine(t, lines[0], http.StatusOK, requestlog.ActionForward)
+4 -8
View File
@@ -7,7 +7,6 @@ import (
"slices"
"time"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
@@ -15,18 +14,16 @@ import (
// does not hold the client's country as it was looked up. A client without a
// country, or whose country cannot be found, is refused. Its figures are the
// country and the setting that refused it.
type countries struct {
allowed []string // SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES
}
type countries struct{}
func (countries) name() string { return "countries" }
func (c countries) check(
func (countries) check(
_ context.Context,
rq *request,
_ time.Time,
) *verdict {
allowed := c.allowed
allowed := rq.h.config.ExclusivelyAllowedCountries
country := rq.line.Country
if len(allowed) == 0 || slices.Contains(allowed, country) {
@@ -36,8 +33,7 @@ func (c countries) check(
return &verdict{
action: requestlog.ActionCountryDenied,
figures: map[string]any{
"country": country,
"setting": config.ExclusivelyAllowedCountriesSetting,
"country": country, "setting": "SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES",
},
}
}
+3 -10
View File
@@ -4,7 +4,6 @@ package waf
import (
"context"
"net/netip"
"time"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
@@ -12,18 +11,12 @@ import (
// denyNets refuses a client in SWW_WAF_DENY_NETS. Its figure is the netblock of
// the setting the client is in.
type denyNets struct {
netblocks []netip.Prefix // SWW_WAF_DENY_NETS
}
type denyNets struct{}
func (denyNets) name() string { return "deny_nets" }
func (d denyNets) check(
_ context.Context,
rq *request,
_ time.Time,
) *verdict {
for _, netblock := range d.netblocks {
func (denyNets) check(_ context.Context, rq *request, _ time.Time) *verdict {
for _, netblock := range rq.h.config.DenyNets {
if netblock.Contains(rq.client) {
return &verdict{
action: requestlog.ActionClientDenied,
-47
View File
@@ -1,47 +0,0 @@
package waf
import (
"net/netip"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
func TestDenyNetsRefusesOnlyAClientInsideItsNetblocks(t *testing.T) {
t.Parallel()
heuristic := denyNets{
netblocks: []netip.Prefix{netip.MustParsePrefix("198.51.100.0/24")},
}
// The request has no handler: the heuristic reads only what it holds.
inside := &request{
decision: decision{client: netip.MustParseAddr("198.51.100.7")},
}
v := heuristic.check(t.Context(), inside, time.Time{})
if v == nil {
t.Fatal("a client inside the netblock was let through")
}
if v.action != requestlog.ActionClientDenied {
t.Errorf(
"action = %q, want %q",
v.action,
requestlog.ActionClientDenied,
)
}
if got := v.figures[figureNetblock]; got != "198.51.100.0/24" {
t.Errorf("netblock figure = %v, want 198.51.100.0/24", got)
}
outside := &request{
decision: decision{client: netip.MustParseAddr("203.0.113.7")},
}
if v := heuristic.check(t.Context(), outside, time.Time{}); v != nil {
t.Errorf("a client outside the netblock was refused: %+v", v)
}
}
+5 -14
View File
@@ -9,7 +9,6 @@ import (
"time"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
@@ -20,28 +19,20 @@ import (
// being log, it does nothing more. A zone without a verdict on the client is
// asked about it in the background, and the request does not wait for the
// answer. Its figure is the zones that list the client, their keys masked.
type dnsbl struct {
dnsbl *reputation.DNSBL
action config.ReputationAction // SWW_WAF_DNSBL_ABUSEIPDB_ACTION
hitNoter reputationHitNoter
}
type dnsbl struct{}
func (dnsbl) name() string { return "dnsbl" }
func (d dnsbl) check(
ctx context.Context,
rq *request,
_ time.Time,
) *verdict {
listedBy := d.dnsbl.ListedBy(ctx, rq.client)
func (dnsbl) check(ctx context.Context, rq *request, _ time.Time) *verdict {
listedBy := rq.h.dnsbl.ListedBy(ctx, rq.client)
rq.dnsblListed = len(listedBy) > 0
d.hitNoter.noteListed(rq, listedBy, "listed by a DNSBL zone")
rq.noteListed(listedBy, "listed by a DNSBL zone")
for _, zone := range listedBy {
rq.addTag(config.DNSBLTagPrefix + strings.ToLower(zone))
}
if !rq.dnsblListed || d.action != config.ReputationActionDeny {
if !rq.dnsblListed || rq.h.config.DNSBLAbuseIPDBAction != deny {
return nil
}
+3 -19
View File
@@ -5,7 +5,7 @@ import (
"testing"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
func TestHealthEndpointCountsTowardNoLimit(t *testing.T) {
@@ -25,13 +25,8 @@ func TestHealthEndpointCountsTowardNoLimit(t *testing.T) {
)
for range healthChecks {
got := s.ownEndpointRequest(
"",
http.MethodGet,
wafprotocol.HealthPath,
"",
http.StatusOK,
)
got := s.ownEndpointRequest("", http.MethodGet, waf.HealthPath, "",
http.StatusOK)
if string(got.body) != "ok\n" ||
got.header.Get("Content-Type") != contentType {
@@ -58,14 +53,3 @@ func TestHealthEndpointCountsTowardNoLimit(t *testing.T) {
}
}
}
func TestHealthEndpointIsFoundAtItsPathPercentEncoded(t *testing.T) {
t.Parallel()
// %68 is h. An endpoint is found by the decoded path.
s, _, _ := startWithClock(t, "", nil)
got := s.ownEndpointRequest("", http.MethodGet, "/_sww/%68ealthz", "",
http.StatusOK)
wantBody(t, got, "ok\n")
}
+77 -4
View File
@@ -5,10 +5,9 @@ import (
"time"
)
// To add a heuristic, write one file with a struct that implements heuristic
// and holds the settings and parts it reads. Add one line for it to one of the
// lists that newHandler builds, which hands it those settings and parts. Its
// settings go in config.WAF, and the comment at the top of its file names them.
// To add a heuristic, write one file with a type that implements heuristic,
// and add one line for it to one of the lists below. Its settings go in
// config.WAF, and the comment at the top of its file names them.
// heuristic is one of the checks that decide what waf does with a
// request.
@@ -37,6 +36,80 @@ const (
figureNetblock = "netblock"
)
// The lists of heuristics, each in the order its heuristics run. A client in
// SWW_WAF_ALLOW_NETS skips them all.
// beforeLookup are the heuristics that run before the client's AS number and
// country are looked up, so that a client they refuse is not looked up.
// header_limits comes first: it only measures and counts, and a request it
// refuses causes no other work.
func beforeLookup() []heuristic {
return []heuristic{
headerLimits{},
tagLists{},
routeWatchTags{},
denyNets{},
activeBan{},
}
}
// afterLookup are the heuristics that run once the lookup has been made.
func afterLookup() []heuristic {
return []heuristic{
countries{},
blocklists{},
dnsbl{},
abuseIPDB{},
tagDeny{},
rateLimits{},
}
}
// besideLookup are the heuristics that only read the request. Their work runs
// beside the lookup and afterLookup, in findBesideLookup, which changes nothing
// of the request. Their check then applies what that work found, and runs only
// once waf takes it.
func besideLookup() []heuristic {
return []heuristic{
trapPaths{},
ruleFiles{},
coreRuleSet{},
}
}
// afterBesideLookup are the heuristics that run once afterLookup and
// besideLookup have both let the request through. tag_deny runs a second time,
// on the tag rule_files and core_rule_set can add.
func afterBesideLookup() []heuristic {
return []heuristic{
tagDeny{},
}
}
// afterAnswer are the heuristics that run on a request once waf has
// answered it, before its decision line is written. They ban the client, and do
// not change what the request got: their verdict's action is not used.
func afterAnswer() []heuristic {
return []heuristic{
refusalLimit{},
}
}
// onReport are the heuristics that run on an allowed request when rproxy's
// report of it comes, before its report line is written. Like afterAnswer, they
// only ban.
func onReport() []heuristic {
return []heuristic{
byteLimits{},
}
}
// runAfterAnswer runs the heuristics of afterAnswer on the request, at the time
// it is called.
func (rq *request) runAfterAnswer(ctx context.Context) {
rq.runBanningHeuristics(ctx, afterAnswer())
}
// runHeuristics runs list, one of those checkClient runs, on the request at now,
// and returns the first verdict, or nil when none gives one. The log line's
// heuristic and heuristic_figures name the heuristic that gave it, and give its
+1 -2
View File
@@ -11,7 +11,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
func TestHistoryKeepsEachRequestOfTheClient(t *testing.T) {
@@ -119,7 +118,7 @@ func TestOwnEndpointsAreNotInTheHistory(t *testing.T) {
s.ownEndpointRequest(
"",
http.MethodGet,
wafprotocol.HealthPath,
waf.HealthPath,
"",
http.StatusOK,
)
+27 -8
View File
@@ -3,6 +3,7 @@ package waf
import (
"context"
"log/slog"
"net/http"
"net/netip"
"strings"
"time"
@@ -15,6 +16,15 @@ import (
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
)
// The headers in which an answer of allow gives rproxy the client's AS number
// and country, for the app, while SWW_WAF_ADD_LOOKUP_HEADERS is true. Go writes
// every header name in this form, as it sends it and as it receives it, so
// X-Client-ASN arrives as X-Client-Asn; header names are not case-sensitive.
const (
asnHeader = "X-Client-Asn"
countryHeader = "X-Client-Country"
)
// lookupResult is what the lookup gave for a request: whether its client was
// looked up, the answer, and whether the request skipped the wait for GeoJS.
type lookupResult struct {
@@ -39,16 +49,15 @@ func (rq *request) startLookUp(
) <-chan lookupResult {
result := make(chan lookupResult, 1)
if rq.h.config.LookupSource == config.LookupSourceOff ||
!canBePlaced(rq.client) {
if rq.h.config.LookupSource == "off" || !canBePlaced(rq.client) {
result <- lookupResult{}
return result
}
client := clientGroup(rq.h.config.IPv6GroupPrefix, rq.client)
client := clientGroup(rq.h.config, rq.client)
if rq.h.config.LookupSource == config.LookupSourceFile {
if rq.h.config.LookupSource == "file" {
result <- lookupResult{
lookedUp: true, answer: rq.h.lookupFile.LookUp(client),
}
@@ -133,10 +142,8 @@ func AddLookup(
) {
limiter.AddLookup(answer.Client, answer.Answered,
answer.ASN, answer.ASName, answer.Country)
netblock := banNetblock(
cfg.BanScopeV4Prefix, cfg.IPv6GroupPrefix, answer.Client.Addr(),
)
ledger.AddLookup(netblock, answer.ASN, answer.ASName, answer.Country)
ledger.AddLookup(banNetblock(cfg, answer.Client.Addr()),
answer.ASN, answer.ASName, answer.Country)
}
// GeoJSFailed logs a failure of GeoJS to processLog, raises a
@@ -159,6 +166,18 @@ func GeoJSFailed(
m.GeoJSFailures.Inc()
}
// setLookupHeaders sets the headers that give the client's AS number and
// country, leaving out one that is unknown.
func setLookupHeaders(header http.Header, asn, country string) {
if asn != "" {
header.Set(asnHeader, asn)
}
if country != "" {
header.Set(countryHeader, country)
}
}
// canBePlaced reports whether a lookup can place addr: private, loopback
// and link-local addresses have no AS number or country.
func canBePlaced(addr netip.Addr) bool {
+1 -2
View File
@@ -16,7 +16,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
const (
@@ -67,7 +66,7 @@ func TestMetricsNeedTheTokenAndOtherPathsAreNotFound(t *testing.T) {
{http.MethodPost, waf.MetricsPath, bearer, http.StatusNotFound},
{http.MethodGet, waf.MetricsPath + "/", bearer, http.StatusNotFound},
{http.MethodGet, "/_sww/bans", bearer, http.StatusNotFound},
{http.MethodPost, wafprotocol.HealthPath, "", http.StatusNotFound},
{http.MethodPost, waf.HealthPath, "", http.StatusNotFound},
} {
got := s.ownEndpointRequest(
tc.authorization,
+69 -168
View File
@@ -6,6 +6,7 @@ package waf
import (
"cmp"
"fmt"
"io"
"log/slog"
"net/http"
@@ -28,6 +29,14 @@ import (
"sneak.berlin/go/smallwebwaf/internal/rules"
)
// ownEndpointPrefix starts the path of each of waf's own endpoints. The app
// never sees a path under it.
const ownEndpointPrefix = "/_sww/"
// HealthPath is the health endpoint of waf and of rproxy, which the
// container's health check asks.
const HealthPath = "/_sww/healthz"
// MetricsPath is where the metrics are, for a request that carries
// SWW_WAF_METRICS_TOKEN.
const MetricsPath = "/_sww/metrics"
@@ -74,8 +83,8 @@ type Params struct {
// LookupFile is the lookup database they are looked up in while
// SWW_WAF_LOOKUP_SOURCE is file, and nil otherwise.
LookupFile *lookup.File
// CoreRuleSet is what inspects requests while
// SWW_WAF_CORE_RULE_SET_MODE is not off, and nil while it is.
// CoreRuleSet is what NewCoreRuleSet returns: what inspects requests
// while SWW_WAF_CORE_RULE_SET_MODE is not off, and nil while it is.
CoreRuleSet CoreRuleSetInspector
// Now tells the time by which requests are counted for the rate
// limits, bans are made and run out, and GeoJS's answers are kept,
@@ -125,19 +134,15 @@ type Server struct {
}
// newHandler returns the handler with what every request shares, the parts
// in params among it. It builds the heuristics, each with the settings and
// parts it reads, and the lists they run in.
//
//nolint:funlen // a few lines for each heuristic, a list that grows with them
// in params among it.
func newHandler(params Params) *handler {
awaitingReport, err := lru.New[string, *decision](maxAwaitingReport)
if err != nil {
panic(err) // New fails only for a size below one
}
cfg := params.Config
h := &handler{
config: cfg,
return &handler{
config: params.Config,
requestLog: params.RequestLog,
processLog: params.ProcessLog,
now: params.Now,
@@ -148,144 +153,81 @@ func newHandler(params Params) *handler {
geojs: params.GeoJS,
anomalies: params.Anomalies,
lookupFile: params.LookupFile,
lists: params.Lists,
dnsbl: params.DNSBL,
abuseIPDB: params.AbuseIPDB,
routeWatch: params.RouteWatch,
rules: params.Rules,
coreRuleSet: params.CoreRuleSet,
alerts: params.Alerts,
// The country list, the headers, the tag settings that name an asn=
// or country= tag and the biased thresholds act on the answer before
// the request goes on.
waitForGeoJS: len(cfg.ExclusivelyAllowedCountries) > 0 ||
cfg.AddLookupHeaders ||
cfg.TagsNeedLookup() ||
biasedThresholdsSet(cfg),
waitForGeoJS: len(params.Config.ExclusivelyAllowedCountries) > 0 ||
params.Config.AddLookupHeaders ||
params.Config.TagsNeedLookup() ||
biasedThresholdsSet(params.Config),
besideLookupLimit: cmp.Or(
params.BesideLookupTimeLimit,
besideLookupTimeLimit,
),
}
}
sharedBanMaker := banMaker{
ledger: params.Ledger,
limiter: params.Limiter,
metrics: params.Metrics,
alerts: params.Alerts,
observe: cfg.Observe,
banResponse: cfg.BanResponse,
banScopeV4Prefix: cfg.BanScopeV4Prefix,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
}
hitNoter := reputationHitNoter{
metrics: params.Metrics,
alerts: params.Alerts,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
// NewCoreRuleSet returns the Core Rule Set at SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL,
// without the rules SWW_WAF_CORE_RULE_SET_DISABLED_RULES switches off, reading bodies
// up to SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE, with the exclusions in
// SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE and the fixes of the apps in SWW_WAF_APPS,
// or nil while SWW_WAF_CORE_RULE_SET_MODE is off. An exclusions file that
// coreruleset.New refuses is an error in every mode. While the mode is off, a
// set file is loaded with the Core Rule Set only to check it.
func NewCoreRuleSet(cfg *config.WAF) (*coreruleset.CoreRuleSet, error) {
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff &&
cfg.CoreRuleSetExclusionsFile == "" {
return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error
}
h.blocklists = blocklists{
lists: params.Lists,
action: cfg.BlocklistAction,
hitNoter: hitNoter,
}
h.rateLimits = rateLimits{
limiter: params.Limiter,
exemptNets: cfg.RateLimitExemptNets,
exemptPaths: cfg.RateLimitExemptPaths,
banResponse: cfg.BanResponse,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
biasedThresholds: biasedThresholds{
lists: params.Lists,
tagLimitPercent: cfg.TagLimitPercent,
asnBytesLimitPercent: cfg.ASNBytesLimitPercent,
countryBytesLimitPercent: cfg.CountryBytesLimitPercent,
unknownCountryLimitPercent: cfg.UnknownCountryLimitPercent,
blocklistAction: cfg.BlocklistAction,
blocklistLimitPercent: cfg.BlocklistLimitPercent,
dnsblAbuseIPDBAction: cfg.DNSBLAbuseIPDBAction,
dnsblAbuseIPDBLimitPercent: cfg.DNSBLAbuseIPDBLimitPercent,
},
banMaker: sharedBanMaker,
}
h.trapPaths = trapPaths{paths: cfg.TrapPaths, banMaker: sharedBanMaker}
h.ruleFiles = ruleFiles{
rules: params.Rules,
metrics: params.Metrics,
banMaker: sharedBanMaker,
}
h.coreRuleSet = coreRuleSet{
inspector: params.CoreRuleSet,
metrics: params.Metrics,
alerts: params.Alerts,
mode: cfg.CoreRuleSetMode,
exemptPaths: cfg.CoreRuleSetExemptPaths,
scoreThreshold: cfg.CoreRuleSetScoreThreshold,
observe: cfg.Observe,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
coreRuleSet, err := coreruleset.New(coreruleset.Params{
ParanoiaLevel: cfg.CoreRuleSetParanoiaLevel,
DisabledRules: cfg.CoreRuleSetDisabledRules,
BodyInspectionSize: cfg.CoreRuleSetBodyInspectionSize,
ExclusionsFile: cfg.CoreRuleSetExclusionsFile,
Apps: cfg.Apps,
})
if err != nil {
return nil, fmt.Errorf(
"SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE: %w",
err,
)
}
h.beforeLookup = []heuristic{
headerLimits{},
tagLists{lists: params.Lists},
routeWatchTags{
routeWatch: params.RouteWatch,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
},
denyNets{netblocks: cfg.DenyNets},
activeBan{banMaker: sharedBanMaker},
}
h.afterLookup = []heuristic{
countries{allowed: cfg.ExclusivelyAllowedCountries},
h.blocklists,
dnsbl{
dnsbl: params.DNSBL,
action: cfg.DNSBLAbuseIPDBAction,
hitNoter: hitNoter,
},
abuseIPDB{
abuseIPDB: params.AbuseIPDB,
limiter: params.Limiter,
keySet: cfg.AbuseIPDBKey != "",
minScore: cfg.AbuseIPDBMinScore,
action: cfg.DNSBLAbuseIPDBAction,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
hitNoter: hitNoter,
},
tagDeny{tags: cfg.TagDeny},
h.rateLimits,
}
h.besideLookup = []heuristic{h.trapPaths, h.ruleFiles, h.coreRuleSet}
h.afterBesideLookup = []heuristic{tagDeny{tags: cfg.TagDeny}}
h.afterAnswer = []heuristic{
refusalLimit{
limiter: params.Limiter,
perMinute: cfg.RefusalsLimitPerMinute,
banResponse: cfg.BanResponse,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
banMaker: sharedBanMaker,
},
}
h.onReport = []heuristic{
byteLimits{
limiter: params.Limiter,
bytesCounted: cfg.BytesCounted,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
banMaker: sharedBanMaker,
},
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff {
return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error
}
h.ownEndpoints = h.routes()
return h
return coreRuleSet, nil
}
// handler is waf's handler. It holds what every request shares; what
// belongs to one request is in a request.
type handler struct {
config *config.WAF
requestLog io.Writer
processLog *slog.Logger
now func() time.Time
metrics *metrics.Metrics
limiter *ratelimit.Limiter
ledger *bans.Ledger
geojs *geojs.GeoJS
anomalies *anomaly.Counters
lookupFile *lookup.File
config *config.WAF
requestLog io.Writer
processLog *slog.Logger
now func() time.Time
metrics *metrics.Metrics
limiter *ratelimit.Limiter
ledger *bans.Ledger
geojs *geojs.GeoJS
anomalies *anomaly.Counters
lookupFile *lookup.File
lists *reputation.Lists
dnsbl *reputation.DNSBL
abuseIPDB *reputation.AbuseIPDB
routeWatch *routewatch.Client
rules *rules.Files
coreRuleSet CoreRuleSetInspector
alerts *alerts.Queue
// waitForGeoJS is true when a setting needs each request's answer from
// GeoJS before the request goes on.
waitForGeoJS bool
@@ -295,45 +237,4 @@ type handler struct {
// awaitingReport holds the decision on each answered question by its id
// until its report comes, and nil for a question still being handled.
awaitingReport *lru.Cache[string, *decision]
// ownEndpoints answers every request that is not a question, as routes
// lists the endpoints.
ownEndpoints http.Handler
// The lists of heuristics, each in the order its heuristics run. A client
// in SWW_WAF_ALLOW_NETS skips them all.
// beforeLookup are the heuristics that run before the client's AS number
// and country are looked up, so that a client they refuse is not looked
// up. header_limits comes first: it only measures and counts, and a
// request it refuses causes no other work.
beforeLookup []heuristic
// afterLookup are the heuristics that run once the lookup has been made.
afterLookup []heuristic
// besideLookup are the heuristics that only read the request. Their work
// runs beside the lookup and afterLookup, in findBesideLookup, which
// changes nothing of the request. Their check then applies what that work
// found, and runs only once waf takes it.
besideLookup []heuristic
// afterBesideLookup are the heuristics that run once afterLookup and
// besideLookup have both let the request through. tag_deny runs a second
// time, on the tag rule_files and core_rule_set can add.
afterBesideLookup []heuristic
// afterAnswer are the heuristics that run on a request once waf has
// answered it, before its decision line is written. They ban the client,
// and do not change what the request got: their verdict's action is not
// used.
afterAnswer []heuristic
// onReport are the heuristics that run on an allowed request when
// rproxy's report of it comes, before its report line is written. Like
// afterAnswer, they only ban.
onReport []heuristic
// The heuristics of the lists that other steps of a request use too:
// refusedWhateverTheAnswer asks blocklists and rateLimits, and
// findBesideLookup does the work of trapPaths, ruleFiles and coreRuleSet.
blocklists blocklists
rateLimits rateLimits
trapPaths trapPaths
ruleFiles ruleFiles
coreRuleSet coreRuleSet
}
+37 -58
View File
@@ -20,11 +20,9 @@ import (
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
"sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/processlog"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
@@ -194,11 +192,11 @@ func (o *output) requestLine(t *testing.T) logLine {
// endedLines waits for count requests that went through rproxy, or to waf's
// own endpoints, to have ended, and returns the last log line of each, in the
// order waf took them. A question's last line is the one its report writes.
// It waits as long as it takes, as requestWithBody does.
func (o *output) endedLines(t *testing.T, count int) []logLine {
t.Helper()
for {
deadline := time.Now().Add(waitLimit)
for time.Now().Before(deadline) {
found := o.lastLines(t)
ended := len(found) >= count
@@ -214,6 +212,10 @@ func (o *output) endedLines(t *testing.T, count int) []logLine {
time.Sleep(pollInterval)
}
t.Fatalf("fewer than %d requests have ended after %s", count, waitLimit)
return nil
}
// lastLines returns the last request log line written so far of each request,
@@ -395,11 +397,11 @@ func startRProxy(
server := rproxy.New(rproxy.Params{
Config: cfg,
RequestLog: io.Discard,
ProcessLog: processlog.New(processlog.Params{
Output: io.Discard,
InstanceName: "rproxy",
Level: slog.LevelInfo,
}),
ProcessLog: requestlog.NewProcessLogger(
io.Discard,
"rproxy",
slog.LevelInfo,
),
})
listener, err := (&net.ListenConfig{}).Listen(
@@ -468,11 +470,11 @@ func newWAFWith(
}
out := &output{}
processLog := processlog.New(processlog.Params{
Output: out,
InstanceName: cfg.InstanceName,
Level: cfg.LogLevel,
})
processLog := requestlog.NewProcessLogger(
out,
cfg.InstanceName,
cfg.LogLevel,
)
ruleFiles, err := rules.Load(rules.Params{
Dir: cfg.RuleFilesDir, Enabled: cfg.RuleFilesEnabled, ProcessLog: processLog,
@@ -502,20 +504,24 @@ func newWAFWith(
}
}
coreRuleSet, err := waf.NewCoreRuleSet(cfg)
if err != nil {
t.Fatalf("Core Rule Set: %v", err)
}
params := waf.Params{
Config: cfg,
RequestLog: out,
ProcessLog: processLog,
LookupFile: lookupFile,
Now: now,
Rules: ruleFiles,
Alerts: alertQueue,
Config: cfg,
RequestLog: out,
ProcessLog: processLog,
LookupFile: lookupFile,
CoreRuleSet: coreRuleSet,
Now: now,
Rules: ruleFiles,
Alerts: alertQueue,
// The tests of the limit set it back, and hold the Core Rule Set
// with a stand-in.
BesideLookupTimeLimit: time.Hour,
}
addCoreRuleSet(t, &params)
addCounters(&params)
addOutsideSources(&params, geojsURL, abuseIPDBCheckURL)
@@ -524,30 +530,6 @@ func newWAFWith(
return server, out, alertQueue
}
// addCoreRuleSet adds to params the Core Rule Set, with the settings for it,
// while SWW_WAF_CORE_RULE_SET_MODE is not off.
func addCoreRuleSet(t *testing.T, params *waf.Params) {
t.Helper()
cfg := params.Config
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff {
return
}
coreRuleSet, err := coreruleset.New(coreruleset.Params{
ParanoiaLevel: cfg.CoreRuleSetParanoiaLevel,
DisabledRules: cfg.CoreRuleSetDisabledRules,
BodyInspectionSize: cfg.CoreRuleSetBodyInspectionSize,
ExclusionsFile: cfg.CoreRuleSetExclusionsFile,
Apps: cfg.Apps,
})
if err != nil {
t.Fatalf("Core Rule Set: %v", err)
}
params.CoreRuleSet = coreRuleSet
}
// addCounters adds to params the metrics, the table of clients, the ledger
// and the anomaly counters, as sww builds them from params' settings. The
// metrics read the ledger, the table of clients and the rules.
@@ -558,17 +540,14 @@ func addCounters(params *waf.Params) {
cfg.MetricsBusiestCountriesAndASNs,
cfg.InstanceName,
)
params.Limiter = ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{
PerMinute: cfg.RateLimitPerMinute,
PerHour: cfg.RateLimitPerHour,
PerDay: cfg.RateLimitPerDay,
BytesPerMinute: cfg.BytesLimitPerMinute,
BytesPerHour: cfg.BytesLimitPerHour,
BytesPerDay: cfg.BytesLimitPerDay,
},
MaxClients: cfg.MaxTrackedClients,
})
params.Limiter = ratelimit.New(ratelimit.Limits{
PerMinute: cfg.RateLimitPerMinute,
PerHour: cfg.RateLimitPerHour,
PerDay: cfg.RateLimitPerDay,
BytesPerMinute: cfg.BytesLimitPerMinute,
BytesPerHour: cfg.BytesLimitPerHour,
BytesPerDay: cfg.BytesLimitPerDay,
}, cfg.MaxTrackedClients)
params.Ledger = bans.New(bans.Params{
LimitBanDuration: cfg.LimitBanDuration,
LimitBanRepeatWindow: cfg.LimitBanRepeatWindow,
+22 -34
View File
@@ -6,10 +6,8 @@ package waf
import (
"context"
"net/netip"
"time"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
@@ -20,37 +18,28 @@ import (
// refused, and bans the client. A request exempt from the rate limits is exempt
// from the byte limits too. Its figures are the window, the count and the
// limit.
type rateLimits struct {
limiter *ratelimit.Limiter
exemptNets []netip.Prefix // SWW_WAF_RATE_LIMIT_EXEMPT_NETS
exemptPaths []string // SWW_WAF_RATE_LIMIT_EXEMPT_PATHS
banResponse int // SWW_WAF_BAN_RESPONSE
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
biasedThresholds biasedThresholds
banMaker banMaker
}
type rateLimits struct{}
func (rateLimits) name() string { return "rate_limits" }
func (r rateLimits) check(
func (rateLimits) check(
_ context.Context,
rq *request,
now time.Time,
) *verdict {
rq.counted = r.countedByRateLimits(rq)
rq.counted = rq.countedByRateLimits()
if !rq.counted {
return nil
}
rq.rateLimitPercent, rq.bytesLimitPercent =
r.biasedThresholds.limitPercentages(rq)
rq.rateLimitPercent, rq.bytesLimitPercent = rq.limitPercentages()
rq.line.RateLimitPercent, rq.line.RateLimitPercentSetting =
rq.rateLimitPercent.logged()
rq.line.BytesLimitPercent, rq.line.BytesLimitPercentSetting =
rq.bytesLimitPercent.logged()
counts, hit, over := r.limiter.Count(
clientGroup(r.ipv6GroupPrefix, rq.client),
counts, hit, over := rq.h.limiter.Count(
clientGroup(rq.h.config, rq.client),
now,
rq.rateLimitPercent.percent,
)
@@ -60,29 +49,28 @@ func (r rateLimits) check(
return nil
}
r.banMaker.banForLimit(rq, now, hit, r.banResponse)
rq.banForLimit(now, hit, rq.h.config.BanResponse)
return &verdict{
action: requestlog.ActionClientIsRateLimited, figures: limitFigures(hit),
}
}
// overAWholeRateLimit reports whether the heuristic rate_limits will refuse rq
// at now, whatever its client's limit percentages: the rate limits count it,
// and one more request from its client takes it over a whole rate limit. It
// counts nothing.
func (r rateLimits) overAWholeRateLimit(rq *request, now time.Time) bool {
return r.countedByRateLimits(rq) &&
r.limiter.OverARateLimit(
clientGroup(r.ipv6GroupPrefix, rq.client),
now,
)
// overAWholeRateLimit reports whether the heuristic rate_limits will refuse the
// request at now, whatever its client's limit percentages: the rate limits
// count it, and one more request from its client takes it over a whole rate
// limit. It counts nothing.
func (rq *request) overAWholeRateLimit(now time.Time) bool {
return rq.countedByRateLimits() &&
rq.h.limiter.OverARateLimit(clientGroup(rq.h.config, rq.client), now)
}
// countedByRateLimits reports whether the rate limits count rq: its client is
// not in SWW_WAF_RATE_LIMIT_EXEMPT_NETS, and its path is not exempt under
// SWW_WAF_RATE_LIMIT_EXEMPT_PATHS.
func (r rateLimits) countedByRateLimits(rq *request) bool {
return !isInside(rq.client, r.exemptNets) &&
!pathExempt(rq.in, r.exemptPaths)
// countedByRateLimits reports whether the rate limits count the request: its
// client is not in SWW_WAF_RATE_LIMIT_EXEMPT_NETS, and its path is not exempt
// under SWW_WAF_RATE_LIMIT_EXEMPT_PATHS.
func (rq *request) countedByRateLimits() bool {
cfg := rq.h.config
return !isInside(rq.client, cfg.RateLimitExemptNets) &&
!pathExempt(rq.in, cfg.RateLimitExemptPaths)
}
+9 -15
View File
@@ -8,7 +8,6 @@ import (
"net/http"
"time"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
@@ -19,22 +18,17 @@ import (
// a limit, which bans the client. A client in SWW_WAF_ALLOW_NETS, which the
// heuristics skip, is not counted, and nothing is while the limit is off.
// Its figures are the window, the count and the limit.
type refusalLimit struct {
limiter *ratelimit.Limiter
perMinute int64 // SWW_WAF_REFUSALS_LIMIT_PER_MINUTE
banResponse int // SWW_WAF_BAN_RESPONSE
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
banMaker banMaker
}
type refusalLimit struct{}
func (refusalLimit) name() string { return "refusal_limit" }
func (r refusalLimit) check(
func (refusalLimit) check(
_ context.Context,
rq *request,
now time.Time,
) *verdict {
if r.perMinute == 0 {
cfg := rq.h.config
if cfg.RefusalsLimitPerMinute == 0 {
return nil
}
@@ -42,10 +36,10 @@ func (r refusalLimit) check(
return nil
}
hit, over := r.limiter.CountRefusal(
clientGroup(r.ipv6GroupPrefix, rq.client),
hit, over := rq.h.limiter.CountRefusal(
clientGroup(rq.h.config, rq.client),
now,
r.perMinute,
cfg.RefusalsLimitPerMinute,
)
if !over {
return nil
@@ -60,10 +54,10 @@ func (r refusalLimit) check(
status = http.StatusForbidden
case requestlog.ActionTemporarilyBanned,
requestlog.ActionPermanentlyBanned:
status = r.banResponse
status = cfg.BanResponse
}
r.banMaker.banForLimit(rq, now, hit, status)
rq.banForLimit(now, hit, status)
return &verdict{figures: limitFigures(hit)}
}
+11 -20
View File
@@ -5,35 +5,26 @@ package waf
import (
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/metrics"
)
// reputationHitNoter notes the reputation sources that list a request's
// client, for the reputation heuristics.
type reputationHitNoter struct {
metrics *metrics.Metrics
alerts *alerts.Queue
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
}
// deny is the SWW_WAF_BLOCKLIST_ACTION and the SWW_WAF_DNSBL_ABUSEIPDB_ACTION that
// refuses the requests of a client a source lists.
const deny = "deny"
// noteListed notes each of sources, the URLs of the blocklists or the
// DNSBL zones, their keys masked, that list rq's client, as noteHit does,
// DNSBL zones, their keys masked, that list the client, as noteHit does,
// with reason.
func (n reputationHitNoter) noteListed(
rq *request, sources []string, reason string,
) {
func (rq *request) noteListed(sources []string, reason string) {
for _, source := range sources {
n.noteHit(rq, bans.ReputationHit{Source: source}, reason)
rq.noteHit(bans.ReputationHit{Source: source}, reason)
}
}
// noteHit adds hit's source, which lists rq's client, to the log line's
// noteHit adds hit's source, which lists the client, to the log line's
// reputation, and hit to the notes of a ban the request makes, counts the
// source in the metrics, and raises a reputation_hit alert with reason,
// whose detail gives hit's source and score.
func (n reputationHitNoter) noteHit(
rq *request, hit bans.ReputationHit, reason string,
) {
func (rq *request) noteHit(hit bans.ReputationHit, reason string) {
detail := map[string]any{"source": hit.Source}
if hit.Score != nil {
detail["score"] = *hit.Score
@@ -41,12 +32,12 @@ func (n reputationHitNoter) noteHit(
rq.line.Reputation = append(rq.line.Reputation, hit.Source)
rq.reputation = append(rq.reputation, hit)
n.metrics.ReputationHit(hit.Source)
n.alerts.Raise(alerts.Alert{
rq.h.metrics.ReputationHit(hit.Source)
rq.h.alerts.Raise(alerts.Alert{
Event: alerts.EventReputationHit,
Host: rq.site,
Client: rq.client,
Netblock: clientGroup(n.ipv6GroupPrefix, rq.client),
Netblock: clientGroup(rq.h.config, rq.client),
ASN: rq.line.ASN,
ASName: rq.line.ASName,
Country: rq.line.Country,
+9 -24
View File
@@ -122,11 +122,8 @@ func (h *handler) startRequest(
RequestID: requestID(r, isQuestion),
PeerIP: peer.String(),
ForwardedFor: strings.Join(forwardedFor, ", "),
ClientGroup: clientGroup(
h.config.IPv6GroupPrefix,
client,
).String(),
ContentType: r.Header.Get("Content-Type"),
ClientGroup: clientGroup(h.config, client).String(),
ContentType: r.Header.Get("Content-Type"),
RequestHeaders: requestHeaders(
r,
h.config.LogRequestHeaders,
@@ -236,7 +233,7 @@ func (rq *request) checkClient(ctx context.Context) *verdict {
now := rq.h.now()
v := rq.runHeuristics(ctx, rq.h.beforeLookup, now)
v := rq.runHeuristics(ctx, beforeLookup(), now)
if v != nil {
return v
}
@@ -268,7 +265,7 @@ func (rq *request) runAfterLookup(
// again: a request counted at the time it arrived could be counted
// after a later one of its client's, which the rate limits would take
// for the clock set back.
return rq.runHeuristics(ctx, rq.h.afterLookup, rq.h.now())
return rq.runHeuristics(ctx, afterLookup(), rq.h.now())
}
// pathExempt reports whether r is exempt under prefixes,
@@ -348,16 +345,6 @@ func timing(start, end time.Time) *float64 {
return new(requestlog.Milliseconds(end.Sub(start)))
}
// banAndCount runs list, afterAnswer or onReport, whose heuristics only ban.
// It then counts the request for the anomaly thresholds and adds it to its
// client's history. list runs first, so that a refusal limit it finds broken
// is in that history. ctx is the request's own context.
func (rq *request) banAndCount(ctx context.Context, list []heuristic) {
rq.runBanningHeuristics(ctx, list)
rq.countAnomalies()
rq.addToHistory()
}
// addToHistory adds the request, which has ended, to its client's
// history, and counts its offences in the metrics, and then the lookup's
// answer about the client, as answerAtTheEnd gives it, to that history and
@@ -378,7 +365,7 @@ func (rq *request) addToHistory() {
}
rq.h.limiter.AddToHistory(
clientGroup(rq.h.config.IPv6GroupPrefix, rq.client),
clientGroup(rq.h.config, rq.client),
rq.h.now(),
request,
)
@@ -407,11 +394,11 @@ func (rq *request) countAnomalies() {
rq.h.anomalies.Count(rq.h.now(), anomaly.Request{
Host: rq.site,
Client: rq.client,
ClientGroup: clientGroup(rq.h.config.IPv6GroupPrefix, rq.client),
ClientGroup: clientGroup(rq.h.config, rq.client),
ASN: answer.ASN,
ASName: answer.ASName,
Country: answer.Country,
Bytes: rq.countedBytes(rq.h.config.BytesCounted),
Bytes: rq.countedBytes(),
})
}
@@ -434,13 +421,11 @@ func (rq *request) answerAtTheEnd() (geojs.Answer, bool) {
return geojs.Answer{}, false
}
if rq.h.config.LookupSource == config.LookupSourceFile {
if rq.h.config.LookupSource == "file" {
return rq.lookupAnswer, true
}
return rq.h.geojs.Kept(
clientGroup(rq.h.config.IPv6GroupPrefix, rq.client),
)
return rq.h.geojs.Kept(clientGroup(rq.h.config, rq.client))
}
// responseWriter is waf's answer to a request. It notes the status and
+2 -2
View File
@@ -13,7 +13,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
const (
@@ -248,7 +248,7 @@ func TestTimingsAreInOrder(t *testing.T) {
wantStatus(t, do(t, req), http.StatusForbidden)
refused := out.endedLines(t, 2)[1]
wantStatus(t, get(t, server.Addr, wafprotocol.HealthPath), http.StatusOK)
wantStatus(t, get(t, server.Addr, waf.HealthPath), http.StatusOK)
health := out.endedLines(t, 3)[2]
// A request passed to the app has every timing; one refused, none of
-142
View File
@@ -1,142 +0,0 @@
package waf
import (
"crypto/subtle"
"io"
"net/http"
"strings"
"github.com/go-chi/chi/v5"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// routes returns the router of waf's own endpoints. A question never reaches
// it, since ServeHTTP answers a question first. No check runs on a request
// here: waf's address is on the private network, where the peers are rproxy
// and the admin's tools, not visitors.
func (h *handler) routes() http.Handler {
r := chi.NewRouter()
r.Use(matchDecodedPath)
// The report writes no log line of its own: it finishes the line of the
// request it reports.
r.Post(wafprotocol.ReportPath, h.handleReport())
r.Group(func(r chi.Router) {
r.Use(h.logOwnEndpoint)
// The answer to a path with no endpoint, and to a method the path has
// no endpoint for, is set in this group, so that it is logged too.
r.NotFound(answerNotFound)
r.MethodNotAllowed(answerNotFound)
r.Get(wafprotocol.HealthPath, handleHealth())
r.Group(func(r chi.Router) {
r.Use(requireToken(h.config.MetricsToken))
r.Get(MetricsPath, h.metrics.ServeHTTP)
})
// The handlers under BansPath/ and ClientsPath read the address from
// the path themselves, so that one that is not an address gets 400
// after the token check.
r.Group(func(r chi.Router) {
r.Use(requireToken(h.config.AdminToken))
r.Get(BansPath, h.handleListBans())
r.Post(BansPath, h.handleAddBan())
r.Delete(BansPath+"/*", h.handleLiftBans())
r.Get(ClientsPath+"*", h.handleShowClient())
})
})
return r
}
// matchDecodedPath has the router match the decoded path, which the handlers
// read. chi matches the path as sent when it is encoded in an unusual way,
// which would make /_sww/%68ealthz a path with no endpoint.
func matchDecodedPath(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
chi.RouteContext(r.Context()).RoutePath = r.URL.Path
next.ServeHTTP(w, r)
})
}
// logOwnEndpoint writes one log line for each request, with the action
// own_endpoint. The line's request_bytes is how much of the body the handler
// read.
func (h *handler) logOwnEndpoint(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
peer := peerAddress(r)
rq := h.startRequest(w, r, peer, peer, false)
defer rq.finish()
rq.line.Action = requestlog.ActionOwnEndpoint
counted := r.Clone(r.Context())
counted.Body = &countedBody{
ReadCloser: r.Body,
read: &rq.requestBytes,
}
next.ServeHTTP(rq.out, counted)
})
}
// countedBody is a request's body that adds the bytes read of it to read.
type countedBody struct {
io.ReadCloser
read *int64
}
// Read reads part of the body.
func (b *countedBody) Read(p []byte) (int, error) {
n, err := b.ReadCloser.Read(p)
*b.read += int64(n)
return n, err
}
// requireToken returns middleware that lets a request through only when it
// carries token, as Authorization: Bearer <token>. It answers 401 to a request
// without it. While token is unset, it answers 404, as a path with no endpoint
// does.
func requireToken(token string) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
switch {
case token == "":
answerNotFound(w, r)
case !hasToken(r, token):
w.Header().Set("WWW-Authenticate", "Bearer")
http.Error(
w,
http.StatusText(http.StatusUnauthorized),
http.StatusUnauthorized,
)
default:
next.ServeHTTP(w, r)
}
},
)
}
}
// hasToken reports whether r carries token, as Authorization: Bearer
// <token>.
func hasToken(r *http.Request, token string) bool {
scheme, sent, _ := strings.Cut(r.Header.Get("Authorization"), " ")
return strings.EqualFold(scheme, "Bearer") &&
subtle.ConstantTimeCompare([]byte(sent), []byte(token)) == 1
}
// answerNotFound answers 404, for a path with no endpoint, and for a method
// the path has no endpoint for.
func answerNotFound(w http.ResponseWriter, _ *http.Request) {
http.Error(w, http.StatusText(http.StatusNotFound), http.StatusNotFound)
}
+5 -10
View File
@@ -8,7 +8,6 @@ import (
"time"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/routewatch"
)
// routeWatchTags adds to the request the tags asn=<number> and prefix=<netblock>
@@ -18,26 +17,22 @@ import (
// SWW_WAF_ROUTEWATCH_URL is unset it does nothing. Answers are kept by client and
// asked for by the request's address, and a route without an origin AS gives only
// the prefix= tag.
type routeWatchTags struct {
// routeWatch is nil while SWW_WAF_ROUTEWATCH_URL is unset.
routeWatch *routewatch.Client
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
}
type routeWatchTags struct{}
func (routeWatchTags) name() string { return "routewatch_tags" }
func (r routeWatchTags) check(
func (routeWatchTags) check(
ctx context.Context,
rq *request,
_ time.Time,
) *verdict {
if r.routeWatch == nil {
if rq.h.routeWatch == nil {
return nil
}
route, missing := r.routeWatch.Route(
route, missing := rq.h.routeWatch.Route(
ctx,
clientGroup(r.ipv6GroupPrefix, rq.client),
clientGroup(rq.h.config, rq.client),
rq.client,
)
if missing != "" {
+4 -10
View File
@@ -9,7 +9,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
)
@@ -23,15 +22,11 @@ import (
// mode raises the alert for the ban
// it would have made. Either rule's match is noted as an offence, for the
// client's history. Its figures are the rule's id and action.
type ruleFiles struct {
rules *rules.Files
metrics *metrics.Metrics
banMaker banMaker
}
type ruleFiles struct{}
func (ruleFiles) name() string { return "rule_files" }
func (r ruleFiles) check(
func (ruleFiles) check(
_ context.Context,
rq *request,
now time.Time,
@@ -40,7 +35,7 @@ func (r ruleFiles) check(
for _, rule := range matched {
rq.line.RuleIDs = append(rq.line.RuleIDs, rule.ID)
r.metrics.RuleMatched(rule.ID, rule.Action)
rq.h.metrics.RuleMatched(rule.ID, rule.Action)
}
if len(matched) == 0 {
@@ -65,8 +60,7 @@ func (r ruleFiles) check(
rq.attack = true
return &verdict{
action: r.banMaker.banForAttack(
rq,
action: rq.banForAttack(
now,
bans.Notes{RuleID: last.ID, Target: last.Target},
),
+3 -9
View File
@@ -12,19 +12,13 @@ import (
// tagDeny refuses a request that carries a tag in SWW_WAF_TAG_DENY. Its figure is
// the first such tag, in sorted order.
type tagDeny struct {
tags []string // SWW_WAF_TAG_DENY
}
type tagDeny struct{}
func (tagDeny) name() string { return "tag_deny" }
func (t tagDeny) check(
_ context.Context,
rq *request,
_ time.Time,
) *verdict {
func (tagDeny) check(_ context.Context, rq *request, _ time.Time) *verdict {
for _, tag := range rq.line.Tags {
if slices.Contains(t.tags, tag) {
if slices.Contains(rq.h.config.TagDeny, tag) {
return &verdict{
action: requestlog.ActionClientDenied,
figures: map[string]any{"tag": tag},
+3 -11
View File
@@ -6,25 +6,17 @@ package waf
import (
"context"
"time"
"sneak.berlin/go/smallwebwaf/internal/reputation"
)
// tagLists adds to the request one tag for each list of SWW_WAF_TAG_LISTS that
// holds the client, the tag being the list's name. It refuses nothing, and runs
// first so that every refusal's log line carries these tags.
type tagLists struct {
lists *reputation.Lists
}
type tagLists struct{}
func (tagLists) name() string { return "tag_lists" }
func (t tagLists) check(
_ context.Context,
rq *request,
_ time.Time,
) *verdict {
for _, name := range t.lists.TagsOf(rq.client) {
func (tagLists) check(_ context.Context, rq *request, _ time.Time) *verdict {
for _, name := range rq.h.lists.TagsOf(rq.client) {
rq.addTag(name)
}
+8 -15
View File
@@ -16,14 +16,11 @@ import (
// it bans the client's netblock, or in observe mode raises the alert for the
// ban it would have made. The ban's notes give the path as sent, and so does
// its figure.
type trapPaths struct {
paths []string // SWW_WAF_TRAP_PATHS
banMaker banMaker
}
type trapPaths struct{}
func (trapPaths) name() string { return "trap_paths" }
func (t trapPaths) check(
func (trapPaths) check(
_ context.Context,
rq *request,
now time.Time,
@@ -36,22 +33,18 @@ func (t trapPaths) check(
rq.attack = true
return &verdict{
action: t.banMaker.banForAttack(
rq,
now,
bans.Notes{TrapPath: path},
),
action: rq.banForAttack(now, bans.Notes{TrapPath: path}),
figures: map[string]any{"path": path},
}
}
// trapPath returns rq's path as sent, when its path as a path rule sees it,
// without the query, as sent or once percent-decoded, is in
// trapPath returns the request's path as sent, when its path as a path rule
// sees it, without the query, as sent or once percent-decoded, is in
// SWW_WAF_TRAP_PATHS, and "" otherwise.
func (t trapPaths) trapPath(rq *request) string {
func (rq *request) trapPath() string {
path := rules.Path(rq.in)
if !slices.Contains(t.paths, path) &&
!slices.Contains(t.paths, rules.DecodeOnce(path)) {
if !slices.Contains(rq.h.config.TrapPaths, path) &&
!slices.Contains(rq.h.config.TrapPaths, rules.DecodeOnce(path)) {
return ""
}
+298 -100
View File
@@ -1,21 +1,75 @@
package waf
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"net/http"
"net/netip"
"strconv"
"strings"
"time"
"unicode/utf8"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// The headers of rproxy's question and of waf's answer, as
// "waf" in SPEC.md gives them.
const (
// ClientHeader marks a request as rproxy's question, and gives the
// visitor's address.
ClientHeader = "X-Sww-Client"
// PeerHeader gives the address of rproxy's own TCP peer.
PeerHeader = "X-Sww-Peer"
// ContentLengthHeader gives the body length the visitor announced.
ContentLengthHeader = "X-Sww-Content-Length"
// DispositionHeader gives waf's answer.
DispositionHeader = "X-Sww-Disposition"
)
// The values of DispositionHeader.
const (
// DispositionAllow tells rproxy to pass the request to the app.
DispositionAllow = "allow"
// DispositionDeny tells rproxy to give the visitor the answer's status,
// headers and body.
DispositionDeny = "deny"
// DispositionClose tells rproxy to close the visitor's connection
// without an answer, which SWW_WAF_BAN_RESPONSE asks for when it is close.
// waf closing the question's connection instead would read as
// waf being down, and rproxy would then let the request through.
DispositionClose = "close"
)
// ReportPath is where rproxy sends the report of each request waf
// answered.
const ReportPath = "/_sww/report"
// QuestionBodyMaxBytes is the most of a visitor's body a question carries.
const QuestionBodyMaxBytes = 64 << 10
const (
// maxAwaitingReport is the most answered requests held until their
// reports come. Past it, the request answered longest ago is dropped.
maxAwaitingReport = 10000
// maxRequestIDBytes is the longest id a question may give.
maxRequestIDBytes = 128
// reportBodyMaxBytes is the most of a report's body that is read. It is
// far above any report's.
reportBodyMaxBytes = 1 << 20
// maxHTTPStatus is the highest status a report may give.
maxHTTPStatus = 599
// maxReportedBytes is the most bytes a report may give for each way, 1
// PiB, so that adding them up cannot overflow.
maxReportedBytes = 1 << 50
// maxReportedMilliseconds is the longest duration a report may give,
// about 31 years, so that it fits a time.Duration.
maxReportedMilliseconds = 1e12
// wafTimeout bounds how long a request to waf takes to
// arrive, and its answer to leave, and wafIdleTimeout how long a
// kept-open connection waits for its next request.
@@ -23,6 +77,26 @@ const (
wafIdleTimeout = 120 * time.Second
)
var (
errReportFieldMissing = errors.New("a field is missing")
errReportValue = errors.New("a value is out of range")
)
// Report is the JSON object rproxy posts to ReportPath once the visitor's
// answer has ended, its fields named as the request log's of the same
// meaning. RequestID is the question's id. Status is what the visitor got, 0
// if nothing. The durations are in milliseconds.
//
//nolint:tagliatelle // docs/request-log.md names the fields in snake_case
type Report struct {
RequestID string `json:"request_id"`
Status int `json:"status"`
RequestBytes int64 `json:"request_bytes"`
ResponseBytes int64 `json:"response_bytes"`
DurationTotalMS float64 `json:"duration_total_ms"`
DurationUpstreamTotalMS float64 `json:"duration_upstream_total_ms"`
}
// New returns the server `sww waf` runs. It answers
// rproxy's questions, takes rproxy's reports and answers its own endpoints. It
// passes nothing to an app. rproxy applies the size and time limits to the
@@ -49,25 +123,25 @@ func New(params Params) *Server {
GeoJS: h.geojs,
Anomalies: h.anomalies,
LookupFile: h.lookupFile,
Lists: params.Lists,
DNSBL: params.DNSBL,
AbuseIPDB: params.AbuseIPDB,
RouteWatch: params.RouteWatch,
Lists: h.lists,
DNSBL: h.dnsbl,
AbuseIPDB: h.abuseIPDB,
RouteWatch: h.routeWatch,
Metrics: h.metrics,
}
}
// ServeHTTP handles one request to waf: a request carrying
// wafprotocol.ClientHeader is rproxy's question, and any other is for one of
// waf's own endpoints.
// ClientHeader is rproxy's question, and any other is for waf
// itself.
func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
if len(r.Header.Values(wafprotocol.ClientHeader)) > 0 {
if len(r.Header.Values(ClientHeader)) > 0 {
h.answerQuestion(w, r)
return
}
h.ownEndpoints.ServeHTTP(w, r)
h.answerOwnEndpoint(w, r)
}
// refusedWhateverTheAnswer reports whether the request is one that does not wait
@@ -80,86 +154,112 @@ func (rq *request) refusedWhateverTheAnswer(now time.Time) bool {
return false
}
listedBy := rq.h.blocklists.lists.ListedBy(rq.client)
return rq.h.blocklists.refusedByBlocklists(listedBy) ||
rq.h.rateLimits.overAWholeRateLimit(rq, now)
return rq.refusedByBlocklists(rq.h.lists.ListedBy(rq.client)) ||
rq.overAWholeRateLimit(now)
}
// answerQuestion runs the checks on the visitor's request a question
// carries, and answers rproxy what to do with it. A denied request's counting
// is finished here, and an allowed one's by its report.
func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
question, ok := h.readQuestion(w, r)
client, body, ok := h.readQuestion(w, r)
if !ok {
return
}
// One that cannot be read leaves the log line's peer_ip invalid.
peer, _ := netip.ParseAddr(r.Header.Get(PeerHeader))
rq := h.startRequest(
w,
question.Visitor,
question.Client,
question.Peer,
asTheVisitorSentIt(r, body),
client.Unmap(),
peer.Unmap(),
true,
)
// These run even when a check panics. finish runs first, so that what is
// held has the finished log line. Go's server sends the answer only once
// this function has returned, so the report cannot come before it is held.
defer func() {
rq.finish()
rq.holdForReport()
}()
// The last deferred runs first, so that what is held has the finished log
// line. Go's server sends the answer only once this function has
// returned, so the report cannot come before it is held.
defer rq.holdForReport()
defer rq.finish()
rq.site = r.Host
rq.line.WAFLine = requestlog.WAFLineDecision
refused := rq.check(r.Context())
rq.checked = time.Now()
underOwnPrefix := strings.HasPrefix(
r.URL.Path,
wafprotocol.OwnEndpointPrefix,
)
underOwnPrefix := strings.HasPrefix(r.URL.Path, ownEndpointPrefix)
if refused == nil && !underOwnPrefix {
rq.allow()
// In observe mode, what enforce mode would have refused counts here.
rq.runBanningHeuristics(r.Context(), rq.h.afterAnswer)
defer rq.runAfterAnswer(r.Context())
rq.allow()
return
}
// The last deferred runs first: runAfterAnswer before addToHistory, so that
// a broken refusal limit is in the client's history.
defer rq.addToHistory()
defer rq.countAnomalies()
defer rq.runAfterAnswer(r.Context())
if refused != nil {
rq.deny(*refused)
} else {
// The app never sees this prefix. waf's own endpoints are at its
// own address, so this answers as a path with no endpoint does.
rq.line.Action = requestlog.ActionOwnEndpoint
wafprotocol.SetAnswerHeaders(
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionDeny},
)
http.Error(
rq.out,
http.StatusText(http.StatusNotFound),
http.StatusNotFound,
)
return
}
rq.banAndCount(r.Context(), rq.h.afterAnswer)
// The app never sees this prefix. waf's own endpoints are at its
// own address, so this answers as a path with no endpoint does.
rq.line.Action = requestlog.ActionOwnEndpoint
rq.out.Header().Set(DispositionHeader, DispositionDeny)
http.Error(
rq.out,
http.StatusText(http.StatusNotFound),
http.StatusNotFound,
)
}
// readQuestion reads a question, and reports whether waf takes it. It answers
// a question it does not take with 400 and no disposition. rproxy passes that
// 400 to the visitor, and SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION does
// not apply. The id of a question waf takes is held from here on, without a
// decision yet, so that another question with that id is refused while this
// one is being handled.
// readQuestion returns the visitor's address and the body a question carries,
// and whether waf takes the question. It answers a question it does not
// take with 400 and no disposition. rproxy passes that 400 to the visitor,
// and SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION does not apply. The id of
// a question waf takes is held from here on, without a decision yet, so
// that another question with that id is refused while this one is being
// handled.
//
// A question whose body stops arriving gets no answer, and is not looked at.
// The body is read before any check, since Go's server does not send an answer
// until it has read the body. A question whose body stops arriving gets no
// answer, and is not looked at.
func (h *handler) readQuestion(
w http.ResponseWriter, r *http.Request,
) (wafprotocol.Question, bool) {
question, err := wafprotocol.ReadQuestion(w, r)
if errors.Is(err, wafprotocol.ErrBodyStoppedArriving) {
) (netip.Addr, []byte, bool) {
client, err := netip.ParseAddr(r.Header.Get(ClientHeader))
if err != nil || client.Zone() != "" {
h.refuseQuestion(w, ClientHeader+" is not an address without a zone")
return netip.Addr{}, nil, false
}
id := r.Header.Get(requestIDHeader)
if id == "" || len(id) > maxRequestIDBytes || !utf8.ValidString(id) {
h.refuseQuestion(w, requestIDHeader+" is not a valid id")
return netip.Addr{}, nil, false
}
body, err := io.ReadAll(
http.MaxBytesReader(w, r.Body, QuestionBodyMaxBytes),
)
if _, tooLarge := errors.AsType[*http.MaxBytesError](err); tooLarge {
h.refuseQuestion(w, "the body is over 64 KiB")
return netip.Addr{}, nil, false
}
if err != nil {
h.processLog.Warn(
"a question's body stopped arriving",
"error",
@@ -170,23 +270,17 @@ func (h *handler) readQuestion(
panic(http.ErrAbortHandler)
}
if err != nil {
h.refuseQuestion(w, err.Error())
return question, false
}
held, _ := h.awaitingReport.ContainsOrAdd(question.RequestID, nil)
held, _ := h.awaitingReport.ContainsOrAdd(id, nil)
if held {
h.refuseQuestion(
w,
wafprotocol.RequestIDHeader+" is the id of a request waf holds",
requestIDHeader+" is the id of a request waf holds",
)
return question, false
return netip.Addr{}, nil, false
}
return question, true
return client, body, true
}
// refuseQuestion answers a question waf does not take, for reason.
@@ -195,18 +289,48 @@ func (h *handler) refuseQuestion(w http.ResponseWriter, reason string) {
http.Error(w, reason, http.StatusBadRequest)
}
// allow answers a question with wafprotocol.DispositionAllow and, while
// asTheVisitorSentIt returns a copy of r, a question, as the checks and the
// log line are to see it: without the headers rproxy adds for waf,
// with body, which has been read of r, and with the length the visitor
// announced, as -1 when it announced none. The question's own length covers
// only the part of the body it carries.
func asTheVisitorSentIt(r *http.Request, body []byte) *http.Request {
sent := r.Clone(r.Context())
announced := sent.Header.Get(ContentLengthHeader)
sent.Header.Del(ClientHeader)
sent.Header.Del(PeerHeader)
sent.Header.Del(ContentLengthHeader)
sent.Header.Del("Content-Length")
sent.ContentLength = -1
length, err := strconv.ParseInt(announced, 10, 64)
if err == nil && length >= 0 {
sent.ContentLength = length
sent.Header.Set("Content-Length", announced)
}
sent.Body = http.NoBody
if len(body) > 0 {
sent.Body = io.NopCloser(bytes.NewReader(body))
}
return sent
}
// allow answers a question with DispositionAllow and, while
// SWW_WAF_ADD_LOOKUP_HEADERS is true, the client's AS number and country.
func (rq *request) allow() {
rq.allowed = true
answer := wafprotocol.Answer{Disposition: wafprotocol.DispositionAllow}
header := rq.out.Header()
header.Set(DispositionHeader, DispositionAllow)
if rq.h.config.AddLookupHeaders {
answer.ASN = rq.line.ASN
answer.Country = rq.line.Country
setLookupHeaders(header, rq.line.ASN, rq.line.Country)
}
wafprotocol.SetAnswerHeaders(rq.out.Header(), answer)
rq.out.WriteHeader(http.StatusOK)
}
@@ -224,45 +348,117 @@ func (rq *request) deny(r refusal) {
if r.status == 0 {
// SWW_WAF_BAN_RESPONSE is close. Written past rq.out, since the log line's
// status stays 0: the visitor is sent nothing.
wafprotocol.SetAnswerHeaders(
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionClose},
)
rq.out.Header().Set(DispositionHeader, DispositionClose)
rq.out.ResponseWriter.WriteHeader(http.StatusOK)
return
}
wafprotocol.SetAnswerHeaders(
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionDeny},
)
rq.out.Header().Set(DispositionHeader, DispositionDeny)
http.Error(rq.out, http.StatusText(r.status), r.status)
}
// handleReport answers POST wafprotocol.ReportPath: it finishes the request
// the report names, if it holds that request's decision, and ignores the
// report otherwise.
func (h *handler) handleReport() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
report, err := wafprotocol.ReadReport(w, r)
if err != nil {
h.processLog.Warn("a report is refused", "error", err.Error())
http.Error(w, "the body is not a report", http.StatusBadRequest)
// answerOwnEndpoint answers a request to waf's own address. No check
// runs on it: the address is on the private network, where the peers are
// rproxy and the admin's tools, not visitors.
func (h *handler) answerOwnEndpoint(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodPost && r.URL.Path == ReportPath {
h.takeReport(w, r)
return
}
held, found := h.awaitingReport.Peek(report.RequestID)
// Remove reports true to one report only, of several that name the
// id. A question still being handled is held as nil.
if found && held != nil &&
h.awaitingReport.Remove(report.RequestID) {
h.finishWith(r.Context(), held, report)
}
w.WriteHeader(http.StatusNoContent)
return
}
peer := peerAddress(r)
rq := h.startRequest(w, r, peer, peer, false)
defer rq.finish()
if r.Method == http.MethodGet && r.URL.Path == HealthPath {
rq.answerHealth()
return
}
rq.answerAdmin()
}
// takeReport answers POST ReportPath: it finishes the request the report
// names, if it holds that request's decision, and ignores the report
// otherwise.
func (h *handler) takeReport(w http.ResponseWriter, r *http.Request) {
report, err := readReport(w, r)
if err != nil {
h.processLog.Warn("a report is refused", "error", err.Error())
http.Error(w, "the body is not a report", http.StatusBadRequest)
return
}
held, found := h.awaitingReport.Peek(report.RequestID)
// Remove reports true to one report only, of several that name the id. A
// question still being handled is held as nil.
if found && held != nil && h.awaitingReport.Remove(report.RequestID) {
h.finishWith(r.Context(), held, report)
}
w.WriteHeader(http.StatusNoContent)
}
// readReport reads a report from r's body, which must be one JSON object with
// all of Report's fields, in the ranges SPEC.md gives.
func readReport(w http.ResponseWriter, r *http.Request) (Report, error) {
var report Report
body, err := io.ReadAll(
http.MaxBytesReader(w, r.Body, reportBodyMaxBytes),
)
if err != nil {
return report, fmt.Errorf("read the body: %w", err)
}
var fields map[string]json.RawMessage
err = json.Unmarshal(body, &fields)
if err != nil {
return report, fmt.Errorf("decode the body: %w", err)
}
for _, name := range []string{
"request_id", "status", "request_bytes", "response_bytes",
"duration_total_ms", "duration_upstream_total_ms",
} {
value, there := fields[name]
if !there || string(value) == "null" {
return report, fmt.Errorf("%w: %s", errReportFieldMissing, name)
}
}
err = json.Unmarshal(body, &report)
if err != nil {
return report, fmt.Errorf("decode the body: %w", err)
}
return report, report.checkRanges()
}
// checkRanges returns an error for a value of the report that cannot be true.
func (report *Report) checkRanges() error {
switch {
case report.Status != 0 &&
(report.Status < http.StatusContinue || report.Status > maxHTTPStatus):
return fmt.Errorf("%w: status", errReportValue)
case report.RequestBytes < 0 || report.RequestBytes > maxReportedBytes:
return fmt.Errorf("%w: request_bytes", errReportValue)
case report.ResponseBytes < 0 || report.ResponseBytes > maxReportedBytes:
return fmt.Errorf("%w: response_bytes", errReportValue)
case report.DurationTotalMS < 0 || report.DurationTotalMS > maxReportedMilliseconds:
return fmt.Errorf("%w: duration_total_ms", errReportValue)
case report.DurationUpstreamTotalMS < 0 ||
report.DurationUpstreamTotalMS > maxReportedMilliseconds:
return fmt.Errorf("%w: duration_upstream_total_ms", errReportValue)
}
return nil
}
// finishWith finishes a request with what its report says, and writes the
@@ -274,7 +470,7 @@ func (h *handler) handleReport() http.HandlerFunc {
func (h *handler) finishWith(
ctx context.Context,
held *decision,
report wafprotocol.Report,
report Report,
) {
rq := &request{
decision: *held,
@@ -295,7 +491,9 @@ func (h *handler) finishWith(
if rq.allowed {
rq.line.DurationUpstreamTotalMS = &report.DurationUpstreamTotalMS
rq.banAndCount(ctx, rq.h.onReport)
rq.runBanningHeuristics(ctx, onReport())
rq.countAnomalies()
rq.addToHistory()
rq.h.metrics.RequestReported(
&rq.line,
+86 -195
View File
@@ -22,7 +22,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
const (
@@ -107,8 +106,8 @@ func (s *WAF) questionWith(
req := newRequest(s.t, method, s.addr, path, body)
req.Host = host
req.Header.Set(wafprotocol.ClientHeader, from)
req.Header.Set(wafprotocol.PeerHeader, rproxyPeer)
req.Header.Set(waf.ClientHeader, from)
req.Header.Set(waf.PeerHeader, rproxyPeer)
req.Header.Set(requestIDHeader, rand.Text())
return req
@@ -135,18 +134,13 @@ func (s *WAF) allowed(from string) string {
s.t.Helper()
req := s.question(siteA, from, "/")
wantDisposition(
s.t,
do(s.t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
)
wantDisposition(s.t, do(s.t, req), http.StatusOK, waf.DispositionAllow)
return req.Header.Get(requestIDHeader)
}
// report sends rproxy's report and returns the answer.
func (s *WAF) report(report wafprotocol.Report) answer {
func (s *WAF) report(report waf.Report) answer {
s.t.Helper()
body, err := json.Marshal(report)
@@ -161,11 +155,8 @@ func (s *WAF) report(report wafprotocol.Report) answer {
func (s *WAF) reportBody(body string) answer {
s.t.Helper()
return do(
s.t,
newRequest(s.t, http.MethodPost, s.addr, wafprotocol.ReportPath,
strings.NewReader(body)),
)
return do(s.t, newRequest(s.t, http.MethodPost, s.addr, waf.ReportPath,
strings.NewReader(body)))
}
// passedOn asks about a request from the visitor at from on siteA, and reports
@@ -173,7 +164,7 @@ func (s *WAF) reportBody(body string) answer {
func (s *WAF) passedOn(from string) {
s.t.Helper()
got := s.report(wafprotocol.Report{
got := s.report(waf.Report{
RequestID: s.allowed(from),
Status: http.StatusOK,
RequestBytes: bodyBytes,
@@ -219,7 +210,7 @@ func wantDisposition(
t.Helper()
wantStatus(t, got, status)
if answered := got.header.Get(wafprotocol.DispositionHeader); answered !=
if answered := got.header.Get(waf.DispositionHeader); answered !=
disposition {
t.Errorf(
"answered with disposition %q, want %q",
@@ -281,10 +272,10 @@ func TestQuestionAllowedIsAnsweredAndLogged(t *testing.T) {
req := s.question(siteA, client, "/page?q=1&token=visible")
req.Header.Set(requestIDHeader, "the-id")
req.Header.Set("X-Forwarded-Proto", "https")
req.Header.Set(wafprotocol.ContentLengthHeader, "1234")
req.Header.Set(waf.ContentLengthHeader, "1234")
got := do(t, req)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
wantBody(t, got, "")
if id := got.header.Get(requestIDHeader); id != "" {
@@ -317,12 +308,7 @@ func TestIPv6VisitorIsCountedInItsGroup(t *testing.T) {
s.allowed("2001:db8::1")
got := s.ask("2001:db8::2", "/")
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
}
func TestIPv4VisitorInIPv6FormIsTheIPv4Client(t *testing.T) {
@@ -331,14 +317,9 @@ func TestIPv4VisitorInIPv6FormIsTheIPv4Client(t *testing.T) {
s := startWAF(t, "", map[string]string{denyNets: client})
req := s.question(siteA, "::ffff:"+client, "/")
req.Header.Set(wafprotocol.PeerHeader, "::ffff:"+rproxyPeer)
req.Header.Set(waf.PeerHeader, "::ffff:"+rproxyPeer)
wantDisposition(
t,
do(t, req),
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
wantDisposition(t, do(t, req), http.StatusForbidden, waf.DispositionDeny)
line := s.out.requestLine(t)
if line.ClientIP != client || line.PeerIP != rproxyPeer {
@@ -358,13 +339,13 @@ func TestMalformedQuestionGets400AndAWarningAndNoLine(t *testing.T) {
change func(*http.Request)
}{
{"client not an address", func(req *http.Request) {
req.Header.Set(wafprotocol.ClientHeader, "not an address")
req.Header.Set(waf.ClientHeader, "not an address")
}},
{"client empty", func(req *http.Request) {
req.Header.Set(wafprotocol.ClientHeader, "")
req.Header.Set(waf.ClientHeader, "")
}},
{"client address with a zone", func(req *http.Request) {
req.Header.Set(wafprotocol.ClientHeader, "2001:db8::1%eth0")
req.Header.Set(waf.ClientHeader, "2001:db8::1%eth0")
}},
{"id absent", func(req *http.Request) {
req.Header.Del(requestIDHeader)
@@ -409,12 +390,7 @@ func TestLongestValidIDIsTaken(t *testing.T) {
req := s.question(siteA, client, "/")
req.Header.Set(requestIDHeader, strings.Repeat("x", 128))
wantDisposition(
t,
do(t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
)
wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow)
}
func TestQuestionWithABodyOver64KiBGets400(t *testing.T) {
@@ -423,18 +399,16 @@ func TestQuestionWithABodyOver64KiBGets400(t *testing.T) {
s := startWAF(t, "", nil)
over := strings.NewReader(
strings.Repeat("x", wafprotocol.QuestionBodyMaxBytes+1),
strings.Repeat("x", waf.QuestionBodyMaxBytes+1),
)
got := do(t, s.questionWith(http.MethodPost, siteA, client, "/", over))
wantDisposition(t, got, http.StatusBadRequest, "")
wantProcessMessages(t, s, "a question is refused", 1)
wantNoRequestLines(t, s)
whole := strings.NewReader(
strings.Repeat("x", wafprotocol.QuestionBodyMaxBytes),
)
whole := strings.NewReader(strings.Repeat("x", waf.QuestionBodyMaxBytes))
got = do(t, s.questionWith(http.MethodPost, siteA, client, "/", whole))
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
}
func TestQuestionWithAHeldIDIsRefused(t *testing.T) {
@@ -450,12 +424,12 @@ func TestQuestionWithAHeldIDIsRefused(t *testing.T) {
wantProcessMessages(t, s, "a question is refused", 1)
// A denied request's id is held as well.
denied := s.question(siteA, otherClient, wafprotocol.HealthPath)
denied := s.question(siteA, otherClient, waf.HealthPath)
wantDisposition(
t,
do(t, denied),
http.StatusNotFound,
wafprotocol.DispositionDeny,
waf.DispositionDeny,
)
again = s.question(siteA, client, "/")
@@ -463,9 +437,7 @@ func TestQuestionWithAHeldIDIsRefused(t *testing.T) {
wantDisposition(t, do(t, again), http.StatusBadRequest, "")
// The report ends the holding, and the refused questions changed nothing.
got := s.report(
wafprotocol.Report{RequestID: id, ResponseBytes: answerBytes},
)
got := s.report(waf.Report{RequestID: id, ResponseBytes: answerBytes})
wantStatus(t, got, http.StatusNoContent)
if history := s.history(); history.ResponseBytes != answerBytes {
@@ -474,12 +446,7 @@ func TestQuestionWithAHeldIDIsRefused(t *testing.T) {
again = s.question(siteA, client, "/")
again.Header.Set(requestIDHeader, id)
wantDisposition(
t,
do(t, again),
http.StatusOK,
wafprotocol.DispositionAllow,
)
wantDisposition(t, do(t, again), http.StatusOK, waf.DispositionAllow)
}
func TestReportNeverReachesAQuestionStillBeingHandled(t *testing.T) {
@@ -514,7 +481,7 @@ func TestReportNeverReachesAQuestionStillBeingHandled(t *testing.T) {
// A report for a question not yet answered is ignored, and that question's
// id is taken.
got := s.report(wafprotocol.Report{RequestID: id, ResponseBytes: 1000})
got := s.report(waf.Report{RequestID: id, ResponseBytes: 1000})
wantStatus(t, got, http.StatusNoContent)
// This would wait on the same lookup were the id not taken.
@@ -529,7 +496,7 @@ func TestReportNeverReachesAQuestionStillBeingHandled(t *testing.T) {
select {
case got := <-answered:
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
case <-time.After(waitLimit):
t.Fatal("the question got no answer")
}
@@ -548,9 +515,7 @@ func TestReportNeverReachesAQuestionStillBeingHandled(t *testing.T) {
}
// The true report comes after, and is the one that counts.
got = s.report(
wafprotocol.Report{RequestID: id, ResponseBytes: answerBytes},
)
got = s.report(waf.Report{RequestID: id, ResponseBytes: answerBytes})
wantStatus(t, got, http.StatusNoContent)
if history := s.history(); history.ResponseBytes != answerBytes {
@@ -568,17 +533,17 @@ func TestPeerAbsentOrUnreadableIsAnsweredAndLoggedInvalid(t *testing.T) {
s := startWAF(t, "", nil)
req := s.question(siteA, client, "/")
req.Header.Del(wafprotocol.PeerHeader)
req.Header.Del(waf.PeerHeader)
if peer != "" {
req.Header.Set(wafprotocol.PeerHeader, peer)
req.Header.Set(waf.PeerHeader, peer)
}
wantDisposition(
t,
do(t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
waf.DispositionAllow,
)
if line := s.out.requestLine(t); line.PeerIP != "invalid IP" {
@@ -606,12 +571,12 @@ func TestContentLengthUnreadableOrNegativeCountsAsAbsent(t *testing.T) {
})
req := s.question(siteA, client, "/")
req.Header.Set(wafprotocol.ContentLengthHeader, announced)
req.Header.Set(waf.ContentLengthHeader, announced)
wantDisposition(
t,
do(t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
waf.DispositionAllow,
)
if line := s.out.requestLine(t); line.ContentLength != 0 {
@@ -630,18 +595,13 @@ func TestHeaderSentTwiceUsesItsFirstValue(t *testing.T) {
s := startWAF(t, "", nil)
req := s.question(siteA, client, "/")
req.Header.Add(wafprotocol.ClientHeader, otherClient)
req.Header.Add(wafprotocol.PeerHeader, "172.18.0.3")
req.Header.Add(waf.ClientHeader, otherClient)
req.Header.Add(waf.PeerHeader, "172.18.0.3")
req.Header.Set(requestIDHeader, "first")
req.Header.Add(requestIDHeader, "second")
req.Header.Set(wafprotocol.ContentLengthHeader, "1234")
req.Header.Add(wafprotocol.ContentLengthHeader, "5678")
wantDisposition(
t,
do(t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
)
req.Header.Set(waf.ContentLengthHeader, "1234")
req.Header.Add(waf.ContentLengthHeader, "5678")
wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow)
line := s.out.requestLine(t)
if line.ClientIP != client || line.PeerIP != rproxyPeer ||
@@ -670,7 +630,7 @@ func sendQuestionWithPartOfItsBody(t *testing.T, conn net.Conn) {
_, err := fmt.Fprintf(conn,
"POST / HTTP/1.1\r\nHost: %s\r\n%s: %s\r\n%s: an-id\r\n"+
"Content-Length: 100\r\n\r\npart",
siteA, wafprotocol.ClientHeader, client, requestIDHeader)
siteA, waf.ClientHeader, client, requestIDHeader)
if err != nil {
t.Fatalf("send the question: %v", err)
}
@@ -739,7 +699,7 @@ func TestQuestionWhoseBodyStopsArrivingGetsNoAnswer(t *testing.T) {
t,
do(t, again),
http.StatusOK,
wafprotocol.DispositionAllow,
waf.DispositionAllow,
)
})
}
@@ -757,7 +717,7 @@ func TestQuestionDeniedGetsTheBanResponse(t *testing.T) {
})
got := s.ask(client, "/")
wantDisposition(t, got, status, wafprotocol.DispositionDeny)
wantDisposition(t, got, status, waf.DispositionDeny)
wantBody(t, got, http.StatusText(status)+"\n")
wantLine(
t,
@@ -777,7 +737,7 @@ func TestBanResponseCloseIsAnAnswerOfItsOwn(t *testing.T) {
})
got := s.ask(client, "/")
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionClose)
wantDisposition(t, got, http.StatusOK, waf.DispositionClose)
wantBody(t, got, "")
// The visitor is sent nothing.
wantLine(t, s.out.requestLine(t), 0, requestlog.ActionClientDenied)
@@ -801,22 +761,12 @@ func TestBlockRuleIsDeniedWith403AndABanRuleWithTheBanResponse(
})
got := s.ask(client, "/blocked")
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
wantLine(t, s.out.requestLine(t), http.StatusForbidden,
requestlog.ActionBlockedByBlockRule)
got = s.ask(client, "/.env")
wantDisposition(
t,
got,
http.StatusTooManyRequests,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, http.StatusTooManyRequests, waf.DispositionDeny)
wantLine(t, s.out.requestLines(t, 2)[1], http.StatusTooManyRequests,
requestlog.ActionTemporarilyBanned)
}
@@ -831,15 +781,10 @@ func TestQuestionUnderTheOwnPrefixIsDeniedWith404AfterTheChecks(
})
for _, path := range []string{
wafprotocol.HealthPath, waf.MetricsPath, wafprotocol.ReportPath,
waf.HealthPath, waf.MetricsPath, waf.ReportPath,
} {
got := s.ask(client, path)
wantDisposition(
t,
got,
http.StatusNotFound,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, http.StatusNotFound, waf.DispositionDeny)
}
wantLine(
@@ -850,12 +795,7 @@ func TestQuestionUnderTheOwnPrefixIsDeniedWith404AfterTheChecks(
)
got := s.ask(otherClient, waf.MetricsPath)
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
}
func TestObserveModeAllowsAndLogsWhatEnforceModeWouldHaveDone(t *testing.T) {
@@ -864,7 +804,7 @@ func TestObserveModeAllowsAndLogsWhatEnforceModeWouldHaveDone(t *testing.T) {
s := startWAF(t, "", map[string]string{mode: observe, denyNets: client})
got := s.ask(client, "/")
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
line := s.out.requestLine(t)
wantLine(t, line, http.StatusOK, requestlog.ActionForward)
@@ -882,7 +822,7 @@ func TestObserveModeRefusalAnsweredAllowCountsForTheRefusalLimit(
for range 2 {
got := s.ask(client, "/blocked")
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
}
lines := s.out.requestLines(t, 2)
@@ -906,12 +846,7 @@ func TestDeniedQuestionCountsForTheRefusalLimit(t *testing.T) {
for range 2 {
got := s.ask(client, "/blocked")
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
}
lines := s.out.requestLines(t, 2)
@@ -947,12 +882,7 @@ func TestAllowCarriesTheLookupHeadersWhileTheSettingIsSet(t *testing.T) {
})
got := s.ask(fromDE, "/")
wantDisposition(
t,
got,
http.StatusOK,
wafprotocol.DispositionAllow,
)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
asn, country := got.header.Get(
"X-Client-Asn",
@@ -986,7 +916,7 @@ func TestReportCountsTowardAByteLimitWhichThenBans(t *testing.T) {
}
// 100 bytes are over the limit of 99.
got := s.report(wafprotocol.Report{
got := s.report(waf.Report{
RequestID: id, Status: http.StatusBadGateway,
RequestBytes: bodyBytes, ResponseBytes: answerBytes,
})
@@ -1010,12 +940,7 @@ func TestReportCountsTowardAByteLimitWhichThenBans(t *testing.T) {
}
got = s.ask(client, "/")
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
}
func TestReportCountsAtThePercentageItsQuestionWasGiven(t *testing.T) {
@@ -1033,12 +958,12 @@ func TestReportCountsAtThePercentageItsQuestionWasGiven(t *testing.T) {
t,
s.ask(fromKP, "/"),
http.StatusOK,
wafprotocol.DispositionAllow,
waf.DispositionAllow,
)
s.passedOn(fromDE)
wantDisposition(t, s.ask(fromDE, "/"), http.StatusForbidden,
wafprotocol.DispositionDeny)
waf.DispositionDeny)
}
func TestReportAddsTheRequestToItsClientsHistoryAndTheAnomalyCounters(
@@ -1064,7 +989,7 @@ func TestReportAddsTheRequestToItsClientsHistoryAndTheAnomalyCounters(
)
}
got := s.report(wafprotocol.Report{
got := s.report(waf.Report{
RequestID: id, Status: http.StatusBadGateway,
RequestBytes: bodyBytes, ResponseBytes: answerBytes,
})
@@ -1096,7 +1021,7 @@ func TestDeniedRequestIsFinishedAtItsDecision(t *testing.T) {
t,
do(t, denied),
http.StatusForbidden,
wafprotocol.DispositionDeny,
waf.DispositionDeny,
)
counters := s.server.Anomalies.Snapshot()
@@ -1114,7 +1039,7 @@ func TestDeniedRequestIsFinishedAtItsDecision(t *testing.T) {
wantMetric(t, s.metrics(), series, 1)
got := s.report(wafprotocol.Report{
got := s.report(waf.Report{
RequestID: denied.Header.Get(
requestIDHeader,
), Status: http.StatusForbidden,
@@ -1157,7 +1082,7 @@ func TestMetricsCountAnAllowedRequestFromItsReport(t *testing.T) {
1,
) // the scrape itself
got := s.report(wafprotocol.Report{
got := s.report(waf.Report{
RequestID: id, Status: http.StatusBadGateway,
RequestBytes: bodyBytes, ResponseBytes: answerBytes,
DurationTotalMS: 5, DurationUpstreamTotalMS: 4,
@@ -1175,7 +1100,7 @@ func TestMetricsCountAnAllowedRequestFromItsReport(t *testing.T) {
// check needs a server that has not been scraped yet.
unscraped := startWAF(t, "", map[string]string{metricsToken: token})
got = unscraped.report(wafprotocol.Report{
got = unscraped.report(waf.Report{
RequestID: unscraped.allowed(client), Status: http.StatusBadGateway,
DurationTotalMS: 5,
})
@@ -1193,7 +1118,7 @@ func TestReportForARequestNotHeldIsIgnored(t *testing.T) {
s := startWAF(t, "", map[string]string{bytesLimitPerMinute: byteLimit})
got := s.report(wafprotocol.Report{
got := s.report(waf.Report{
RequestID: "no-such-request", ResponseBytes: 1000,
})
wantStatus(t, got, http.StatusNoContent)
@@ -1212,9 +1137,7 @@ func TestReportForARequestNotHeldIsIgnored(t *testing.T) {
// A second report of the same request does not count it twice.
id := s.allowed(client)
for range 2 {
got = s.report(
wafprotocol.Report{RequestID: id, ResponseBytes: answerBytes},
)
got = s.report(waf.Report{RequestID: id, ResponseBytes: answerBytes})
wantStatus(t, got, http.StatusNoContent)
}
@@ -1424,7 +1347,7 @@ func TestBytesAtTheirMostEachWayBanRatherThanOverflow(t *testing.T) {
s := startWAF(t, "", map[string]string{bytesLimitPerMinute: byteLimit})
got := s.report(wafprotocol.Report{
got := s.report(waf.Report{
RequestID: s.allowed(client), Status: http.StatusOK,
RequestBytes: reportedBytesLimit, ResponseBytes: reportedBytesLimit,
})
@@ -1518,14 +1441,9 @@ func TestReportWritesTheLastLineWithTheDecisionsAndTheReportsFields(
req.Header.Set("X-Forwarded-Proto", "https")
id := req.Header.Get(requestIDHeader)
wantDisposition(
t,
do(t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
)
wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow)
got := s.report(wafprotocol.Report{
got := s.report(waf.Report{
RequestID: id, Status: http.StatusCreated,
RequestBytes: bodyBytes, ResponseBytes: answerBytes,
DurationTotalMS: 12.5, DurationUpstreamTotalMS: 10.25,
@@ -1572,7 +1490,7 @@ func TestEveryDeniedRequestsReportGetsItsLastLine(t *testing.T) {
"close", map[string]string{denyNets: client, banResponse: banResponseClose}, "/",
0, requestlog.ActionClientDenied,
},
{"own prefix", nil, wafprotocol.HealthPath, http.StatusNotFound,
{"own prefix", nil, waf.HealthPath, http.StatusNotFound,
requestlog.ActionOwnEndpoint},
} {
t.Run(tc.name, func(t *testing.T) {
@@ -1584,7 +1502,7 @@ func TestEveryDeniedRequestsReportGetsItsLastLine(t *testing.T) {
id := req.Header.Get(requestIDHeader)
do(t, req)
got := s.report(wafprotocol.Report{
got := s.report(waf.Report{
RequestID: id, Status: tc.status, ResponseBytes: 8, DurationTotalMS: 3,
})
wantStatus(t, got, http.StatusNoContent)
@@ -1624,13 +1542,13 @@ func TestOldestAnsweredRequestIsDroppedPastTheBound(t *testing.T) {
t,
readAnswer(res),
http.StatusOK,
wafprotocol.DispositionAllow,
waf.DispositionAllow,
)
}
// The table is full again with next and those, without oldest.
got := s.report(
wafprotocol.Report{RequestID: oldest, ResponseBytes: answerBytes},
waf.Report{RequestID: oldest, ResponseBytes: answerBytes},
)
wantStatus(t, got, http.StatusNoContent)
@@ -1641,9 +1559,7 @@ func TestOldestAnsweredRequestIsDroppedPastTheBound(t *testing.T) {
)
}
got = s.report(
wafprotocol.Report{RequestID: next, ResponseBytes: answerBytes},
)
got = s.report(waf.Report{RequestID: next, ResponseBytes: answerBytes})
wantStatus(t, got, http.StatusNoContent)
if counted := s.history().Requests; counted != 1 {
@@ -1664,20 +1580,10 @@ func TestBanMadeThroughOneHostRefusesTheClientOnAnother(t *testing.T) {
)
got := do(t, s.question(siteA, client, "/.env"))
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
got = do(t, s.question(siteB, client, "/"))
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
lines := s.out.requestLines(t, 2)
if lines[0].Host != siteA || lines[1].Host != siteB {
@@ -1715,12 +1621,7 @@ func TestRequestsOnEverySiteCountTowardOneSetOfLimits(t *testing.T) {
s.allowed(client)
got := do(t, s.question(siteB, client, "/"))
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
}
func TestOwnEndpointsAnswerOnlyWithoutTheClientHeader(t *testing.T) {
@@ -1735,13 +1636,13 @@ func TestOwnEndpointsAnswerOnlyWithoutTheClientHeader(t *testing.T) {
path, authorization string
want int
}{
{wafprotocol.HealthPath, "", http.StatusOK},
{waf.HealthPath, "", http.StatusOK},
{waf.MetricsPath, bearer, http.StatusOK},
{waf.MetricsPath, "", http.StatusUnauthorized},
{waf.BansPath, adminBearer, http.StatusOK},
{waf.BansPath, "", http.StatusUnauthorized},
{waf.ClientsPath + client, adminBearer, http.StatusOK},
{wafprotocol.ReportPath, "", http.StatusNotFound},
{waf.ReportPath, "", http.StatusNotFound},
{"/_sww/other", "", http.StatusNotFound},
{"/", "", http.StatusNotFound},
} {
@@ -1758,10 +1659,10 @@ func TestOwnEndpointsAnswerOnlyWithoutTheClientHeader(t *testing.T) {
t,
do(t, req),
http.StatusNotFound,
wafprotocol.DispositionDeny,
waf.DispositionDeny,
)
} else {
wantDisposition(t, do(t, req), http.StatusOK, wafprotocol.DispositionAllow)
wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow)
}
}
}
@@ -1771,11 +1672,11 @@ func TestOwnEndpointsLinesDoNotSayWhichOfTheTwoTheyAre(t *testing.T) {
s := startWAF(t, "", nil)
wantStatus(t, get(t, s.addr, wafprotocol.HealthPath), http.StatusOK)
wantStatus(t, get(t, s.addr, waf.HealthPath), http.StatusOK)
line := s.out.requestLine(t)
if _, there := line.fields["waf_line"]; there ||
line.Path != wafprotocol.HealthPath {
line.Path != waf.HealthPath {
t.Errorf("the health check's log line is %+v", line.fields)
}
}
@@ -1787,7 +1688,7 @@ func TestALargeBodyAnnouncedAndLongHeadersAreNoReasonToRefuse(t *testing.T) {
req := s.questionWith(http.MethodPost, siteA, client, "/",
strings.NewReader("a body"))
req.Header.Set(wafprotocol.ContentLengthHeader, "1073741824")
req.Header.Set(waf.ContentLengthHeader, "1073741824")
req.Header.Set(forwardedFor, otherClient)
// 40,000 bytes of headers, over rproxy's 32 KiB, in lines within the
@@ -1797,7 +1698,7 @@ func TestALargeBodyAnnouncedAndLongHeadersAreNoReasonToRefuse(t *testing.T) {
}
got := do(t, req)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
line := s.out.requestLine(t)
if line.ClientIP != client || line.ContentLength != 1<<30 {
@@ -1875,7 +1776,7 @@ carried header:Content-Length block ^6$
strings.NewReader("a body"))
if tc.announced != "" {
req.Header.Set(wafprotocol.ContentLengthHeader, tc.announced)
req.Header.Set(waf.ContentLengthHeader, tc.announced)
}
wantStatus(t, do(t, req), tc.want)
@@ -1893,24 +1794,19 @@ func TestCoreRuleSetReadsTheBodyAQuestionCarries(t *testing.T) {
req := s.questionWith(http.MethodPost, siteA, client, "/",
strings.NewReader(body))
req.Header.Set("Content-Type", contentType)
req.Header.Set(wafprotocol.ContentLengthHeader, "100000")
req.Header.Set(waf.ContentLengthHeader, "100000")
return do(t, req)
}
got := ask(formData, "id=1'%20OR%20'1'='1")
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
// The first 8 KiB of a longer JSON body cannot be parsed, which is no
// attack, so the Core Rule Set does not read it.
cut := `{"text":"` + strings.Repeat("x", 8<<10)
got = ask("application/json", cut)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
}
func TestCoreRuleSetAlertNamesTheSite(t *testing.T) {
@@ -1919,12 +1815,7 @@ func TestCoreRuleSetAlertNamesTheSite(t *testing.T) {
s := startWAF(t, "", map[string]string{coreRuleSetMode: block})
got := s.ask(client, sqlInjection)
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
waiting := s.queue.Snapshot().Waiting[alerts.DestinationWebhook]
if len(waiting) != 1 ||
@@ -1945,7 +1836,7 @@ func TestReputationHitAlertNamesTheSite(t *testing.T) {
loadLists(t, s.server, map[string][]string{dropURL: {client}})
wantDisposition(t, s.ask(client, "/"), http.StatusForbidden,
wafprotocol.DispositionDeny)
waf.DispositionDeny)
waiting := s.queue.Snapshot().Waiting[alerts.DestinationWebhook]
if len(waiting) != 1 || waiting[0].Event != alerts.EventReputationHit ||
-46
View File
@@ -1,46 +0,0 @@
package wafprotocol
import "net/http"
// Answer is what waf's answer to a question tells rproxy in its headers.
// The status, the other headers and the body of an answer of deny are what
// the visitor gets.
type Answer struct {
// Disposition is what rproxy does with the request: DispositionAllow,
// DispositionDeny or DispositionClose.
Disposition string
// ASN and Country are the client's AS number and country, each empty
// when the answer does not give it.
ASN, Country string
}
// SetAnswerHeaders sets the headers of waf's answer that tell rproxy answer,
// leaving out an ASN or Country that is empty.
func SetAnswerHeaders(header http.Header, answer Answer) {
header.Set(DispositionHeader, answer.Disposition)
if answer.ASN != "" {
header.Set(ASNHeader, answer.ASN)
}
if answer.Country != "" {
header.Set(CountryHeader, answer.Country)
}
}
// ReadAnswer returns what the headers of waf's answer tell rproxy, and
// whether they give a valid disposition.
func ReadAnswer(header http.Header) (Answer, bool) {
disposition := header.Get(DispositionHeader)
switch disposition {
case DispositionAllow, DispositionDeny, DispositionClose:
return Answer{
Disposition: disposition,
ASN: header.Get(ASNHeader),
Country: header.Get(CountryHeader),
}, true
}
return Answer{}, false
}
-189
View File
@@ -1,189 +0,0 @@
package wafprotocol
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"net/http"
"net/netip"
"net/url"
"strconv"
"unicode/utf8"
)
// maxRequestIDBytes is the longest id a question may give.
const maxRequestIDBytes = 128
// ErrBodyStoppedArriving is the error ReadQuestion wraps when the question's
// body stops arriving. Every other error it returns is why waf refuses the
// question.
var ErrBodyStoppedArriving = errors.New("the body stopped arriving")
// Why waf refuses a question. Each is also the body of the refusal.
var (
errClientNotAnAddress = errors.New(
ClientHeader + " is not an address without a zone",
)
errRequestIDNotValid = errors.New(RequestIDHeader + " is not a valid id")
errBodyTooLarge = errors.New("the body is over 64 KiB")
)
// QuestionParams are what NewQuestion needs.
type QuestionParams struct {
// Visitor is the visitor's request.
Visitor *http.Request
// PathAsSent is Visitor's path as the visitor sent it, which the question
// sends unchanged.
PathAsSent string
// Body is the part of Visitor's body the question carries, at most
// QuestionBodyMaxBytes.
Body []byte
// WAFBackend is waf's address. The question takes its scheme and host.
WAFBackend *url.URL
// Client is the visitor's address, and Peer that of rproxy's own TCP
// peer.
Client, Peer netip.Addr
// RequestID is the request's id.
RequestID string
// Scheme is how the visitor reached rproxy, or the trusted proxy in front
// of it.
Scheme string
}
// Question is a question as waf reads it.
type Question struct {
// Client is the visitor's address.
Client netip.Addr
// Peer is the address of rproxy's own TCP peer. It is not valid when the
// question gives none that can be read.
Peer netip.Addr
// RequestID is the request's id.
RequestID string
// Visitor is the visitor's request, as asTheVisitorSentIt gives it.
Visitor *http.Request
}
// NewQuestion returns the question about a visitor's request, as "The
// question" in SPEC.md gives it: the request as the visitor sent it, with the
// part of the body it carries and rproxy's headers.
func NewQuestion(
ctx context.Context,
params QuestionParams,
) (*http.Request, error) {
in := params.Visitor
question, err := http.NewRequestWithContext(
ctx,
in.Method,
params.WAFBackend.String(),
bytes.NewReader(params.Body),
)
if err != nil {
return nil, fmt.Errorf("make the question: %w", err)
}
backend := question.URL
target := *in.URL
target.Scheme = backend.Scheme
target.Host = backend.Host
target.Opaque = params.PathAsSent
question.URL = &target
question.Host = in.Host
// The length NewRequestWithContext set is that of the body carried. This
// line makes a body whose length the visitor did not announce go chunked,
// as "The question" in SPEC.md requires.
question.TransferEncoding = in.TransferEncoding
question.Header = in.Header.Clone()
question.Header.Set(ClientHeader, params.Client.String())
question.Header.Set(PeerHeader, params.Peer.String())
question.Header.Del(ContentLengthHeader)
announced := in.Header.Get("Content-Length")
if announced != "" {
question.Header.Set(ContentLengthHeader, announced)
}
question.Header.Set(RequestIDHeader, params.RequestID)
question.Header.Set("X-Forwarded-Proto", params.Scheme)
// Go's client adds a User-Agent of its own to a request without the key.
_, sent := question.Header["User-Agent"]
if !sent {
question.Header["User-Agent"] = nil
}
return question, nil
}
// ReadQuestion reads the question r, with the body it carries. An error
// wrapping ErrBodyStoppedArriving means the body stopped arriving. Any other
// error is why waf refuses the question, and the question's body is then not
// read. The body is read whole, since Go's server does not send an answer
// until it has read the body.
func ReadQuestion(w http.ResponseWriter, r *http.Request) (Question, error) {
client, err := netip.ParseAddr(r.Header.Get(ClientHeader))
if err != nil || client.Zone() != "" {
return Question{}, errClientNotAnAddress
}
id := r.Header.Get(RequestIDHeader)
if id == "" || len(id) > maxRequestIDBytes || !utf8.ValidString(id) {
return Question{}, errRequestIDNotValid
}
body, err := io.ReadAll(
http.MaxBytesReader(w, r.Body, QuestionBodyMaxBytes),
)
if _, tooLarge := errors.AsType[*http.MaxBytesError](err); tooLarge {
return Question{}, errBodyTooLarge
}
if err != nil {
return Question{}, fmt.Errorf("%w: %w", ErrBodyStoppedArriving, err)
}
// One that cannot be read leaves the log line's peer_ip invalid.
peer, _ := netip.ParseAddr(r.Header.Get(PeerHeader))
return Question{
Client: client.Unmap(),
Peer: peer.Unmap(),
RequestID: id,
Visitor: asTheVisitorSentIt(r, body),
}, nil
}
// asTheVisitorSentIt returns a copy of r, a question, as the checks and the
// log line are to see it: without the headers rproxy adds for waf,
// with body, which has been read of r, and with the length the visitor
// announced, as -1 when it announced none. The question's own length covers
// only the part of the body it carries.
func asTheVisitorSentIt(r *http.Request, body []byte) *http.Request {
sent := r.Clone(r.Context())
announced := sent.Header.Get(ContentLengthHeader)
sent.Header.Del(ClientHeader)
sent.Header.Del(PeerHeader)
sent.Header.Del(ContentLengthHeader)
sent.Header.Del("Content-Length")
sent.ContentLength = -1
length, err := strconv.ParseInt(announced, 10, 64)
if err == nil && length >= 0 {
sent.ContentLength = length
sent.Header.Set("Content-Length", announced)
}
sent.Body = http.NoBody
if len(body) > 0 {
sent.Body = io.NopCloser(bytes.NewReader(body))
}
return sent
}
-123
View File
@@ -1,123 +0,0 @@
package wafprotocol
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
)
const (
// reportBodyMaxBytes is the most of a report's body that is read. It is
// far above any report's.
reportBodyMaxBytes = 1 << 20
// maxReportedBytes is the most bytes a report may give for each way, 1
// PiB, so that adding them up cannot overflow.
maxReportedBytes = 1 << 50
// maxReportedMilliseconds is the longest duration a report may give,
// about 31 years, so that it fits a time.Duration.
maxReportedMilliseconds = 1e12
)
var (
errReportFieldMissing = errors.New("a field is missing")
errReportValue = errors.New("a value is out of range")
)
// Report is the JSON object rproxy posts to ReportPath once the visitor's
// answer has ended, its fields named as the request log's of the same
// meaning. RequestID is the question's id. Status is what the visitor got, 0
// if nothing. The durations are in milliseconds.
//
//nolint:tagliatelle // docs/request-log.md names the fields in snake_case
type Report struct {
RequestID string `json:"request_id"`
Status int `json:"status"`
RequestBytes int64 `json:"request_bytes"`
ResponseBytes int64 `json:"response_bytes"`
DurationTotalMS float64 `json:"duration_total_ms"`
DurationUpstreamTotalMS float64 `json:"duration_upstream_total_ms"`
}
// NewReportRequest returns the request that posts report to reportURL, which
// is waf's ReportPath.
func NewReportRequest(
ctx context.Context,
reportURL string,
report *Report,
) (*http.Request, error) {
body, err := json.Marshal(report)
if err != nil {
return nil, fmt.Errorf("encode the report: %w", err)
}
req, err := http.NewRequestWithContext(
ctx, http.MethodPost, reportURL, bytes.NewReader(body))
if err != nil {
return nil, fmt.Errorf("make the request: %w", err)
}
req.Header.Set("Content-Type", "application/json")
return req, nil
}
// ReadReport reads a report from r's body, which must be one JSON object with
// all of Report's fields, in the ranges SPEC.md gives.
func ReadReport(w http.ResponseWriter, r *http.Request) (Report, error) {
var report Report
body, err := io.ReadAll(
http.MaxBytesReader(w, r.Body, reportBodyMaxBytes),
)
if err != nil {
return report, fmt.Errorf("read the body: %w", err)
}
var fields map[string]json.RawMessage
err = json.Unmarshal(body, &fields)
if err != nil {
return report, fmt.Errorf("decode the body: %w", err)
}
for _, name := range []string{
"request_id", "status", "request_bytes", "response_bytes",
"duration_total_ms", "duration_upstream_total_ms",
} {
value, there := fields[name]
if !there || string(value) == "null" {
return report, fmt.Errorf("%w: %s", errReportFieldMissing, name)
}
}
err = json.Unmarshal(body, &report)
if err != nil {
return report, fmt.Errorf("decode the body: %w", err)
}
return report, report.checkRanges()
}
// checkRanges returns an error for a value of the report that cannot be true.
func (report *Report) checkRanges() error {
switch {
case report.Status != 0 &&
(report.Status < http.StatusContinue || report.Status > ReportStatusMax):
return fmt.Errorf("%w: status", errReportValue)
case report.RequestBytes < 0 || report.RequestBytes > maxReportedBytes:
return fmt.Errorf("%w: request_bytes", errReportValue)
case report.ResponseBytes < 0 || report.ResponseBytes > maxReportedBytes:
return fmt.Errorf("%w: response_bytes", errReportValue)
case report.DurationTotalMS < 0 || report.DurationTotalMS > maxReportedMilliseconds:
return fmt.Errorf("%w: duration_total_ms", errReportValue)
case report.DurationUpstreamTotalMS < 0 ||
report.DurationUpstreamTotalMS > maxReportedMilliseconds:
return fmt.Errorf("%w: duration_upstream_total_ms", errReportValue)
}
return nil
}
-61
View File
@@ -1,61 +0,0 @@
// Package wafprotocol holds the three messages rproxy and waf send each
// other, as "`waf`" in SPEC.md gives them. rproxy asks waf a question about
// each visitor's request, and waf answers it. Once the visitor's answer has
// ended, rproxy sends waf a report. This package writes and reads each
// message, and declares every name and limit both sides must agree on.
package wafprotocol
// The headers of the question and of the answer. Go writes every header name
// in this form, as it sends it and as it receives it, so X-Client-ASN arrives
// as X-Client-Asn; header names are not case-sensitive.
const (
// ClientHeader marks a request as rproxy's question, and gives the
// visitor's address.
ClientHeader = "X-Sww-Client"
// PeerHeader gives the address of rproxy's own TCP peer.
PeerHeader = "X-Sww-Peer"
// ContentLengthHeader gives the body length the visitor announced.
ContentLengthHeader = "X-Sww-Content-Length"
// RequestIDHeader gives the request's id, which rproxy makes. rproxy
// also sets it on the request it passes to the app.
RequestIDHeader = "X-Request-ID"
// DispositionHeader gives waf's answer.
DispositionHeader = "X-Sww-Disposition"
// ASNHeader and CountryHeader give the client's AS number and country in
// an answer of allow. rproxy passes them to the app.
ASNHeader = "X-Client-Asn"
CountryHeader = "X-Client-Country"
)
// The values of DispositionHeader.
const (
// DispositionAllow tells rproxy to pass the request to the app.
DispositionAllow = "allow"
// DispositionDeny tells rproxy to give the visitor the answer's status,
// headers and body.
DispositionDeny = "deny"
// DispositionClose tells rproxy to close the visitor's connection
// without an answer, which SWW_WAF_BAN_RESPONSE asks for when it is close.
// waf closing the question's connection instead would read as
// waf being down, and rproxy would then let the request through.
DispositionClose = "close"
)
// OwnEndpointPrefix starts the path of every request for sww itself: waf's
// own endpoints and rproxy's health endpoint. The app never sees a path under
// it.
const OwnEndpointPrefix = "/_sww/"
// HealthPath is the health endpoint of waf and of rproxy, which the
// container's health check asks.
const HealthPath = "/_sww/healthz"
// ReportPath is where rproxy sends the report of each request waf
// answered.
const ReportPath = "/_sww/report"
// QuestionBodyMaxBytes is the most of a visitor's body a question carries.
const QuestionBodyMaxBytes = 64 << 10
// ReportStatusMax is the highest status a report may give.
const ReportStatusMax = 599
-204
View File
@@ -1,204 +0,0 @@
package wafprotocol_test
import (
"io"
"net/http"
"net/http/httptest"
"net/netip"
"net/url"
"strings"
"testing"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// requestID is the id the tests' questions and reports give.
const requestID = "an-id"
// questionRead is what the server startQuestionReader starts read of a
// question: the question, and the body it carries.
type questionRead struct {
question wafprotocol.Question
body string
}
// startQuestionReader starts a server that reads each question it gets with
// ReadQuestion. It returns the server's URL and the channel what it reads goes
// to.
func startQuestionReader(t *testing.T) (*url.URL, <-chan questionRead) {
t.Helper()
read := make(chan questionRead, 1)
server := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
question, err := wafprotocol.ReadQuestion(w, r)
if err != nil {
t.Errorf("read the question: %v", err)
}
var body []byte
if question.Visitor != nil {
body, _ = io.ReadAll(question.Visitor.Body)
}
read <- questionRead{question, string(body)}
},
))
t.Cleanup(server.Close)
serverURL, err := url.Parse(server.URL)
if err != nil {
t.Fatalf("parse the server's URL: %v", err)
}
return serverURL, read
}
// TestQuestionReadsAsWritten sends a question NewQuestion writes over HTTP,
// and checks that ReadQuestion gives back the visitor's request and what
// rproxy added.
func TestQuestionReadsAsWritten(t *testing.T) {
t.Parallel()
const (
target = "/a%2Fb?q=1"
body = "name=value"
held = "name"
)
client := netip.MustParseAddr("2001:db8::1")
peer := netip.MustParseAddr("192.0.2.1")
backend, read := startQuestionReader(t)
visitor := httptest.NewRequestWithContext(
t.Context(),
http.MethodPost,
"http://site.example.com"+target,
strings.NewReader(body),
)
visitor.Header.Set("Content-Length", "10")
visitor.Header.Set("User-Agent", "a visitor")
visitor.Header.Set(wafprotocol.PeerHeader, "the visitor's own")
question, err := wafprotocol.NewQuestion(
t.Context(),
wafprotocol.QuestionParams{
Visitor: visitor,
PathAsSent: "/a%2Fb",
Body: []byte(held),
WAFBackend: backend,
Client: client,
Peer: peer,
RequestID: requestID,
Scheme: "https",
},
)
if err != nil {
t.Fatalf("write the question: %v", err)
}
res, err := http.DefaultTransport.RoundTrip(question)
if err != nil {
t.Fatalf("send the question: %v", err)
}
_ = res.Body.Close()
got := <-read
if got.question.Visitor == nil {
t.FailNow()
}
if got.question.Client != client || got.question.Peer != peer ||
got.question.RequestID != requestID {
t.Errorf("read %+v", got.question)
}
sent := got.question.Visitor
if sent.Host != "site.example.com" || sent.RequestURI != target ||
sent.ContentLength != int64(len(body)) || got.body != held {
t.Errorf(
"read %s at %s, of length %d, carrying %q",
sent.RequestURI, sent.Host, sent.ContentLength, got.body,
)
}
for name, want := range map[string]string{
"Content-Length": "10",
"User-Agent": "a visitor",
"X-Forwarded-Proto": "https",
wafprotocol.RequestIDHeader: requestID,
wafprotocol.ClientHeader: "",
wafprotocol.PeerHeader: "",
wafprotocol.ContentLengthHeader: "",
} {
if sent.Header.Get(name) != want {
t.Errorf("%s is %q, want %q", name, sent.Header.Get(name), want)
}
}
}
// TestAnswerReadsAsWritten checks that ReadAnswer gives back what
// SetAnswerHeaders sets, and that headers without a valid disposition give
// nothing.
func TestAnswerReadsAsWritten(t *testing.T) {
t.Parallel()
for _, want := range []wafprotocol.Answer{
{Disposition: wafprotocol.DispositionAllow, ASN: "64496", Country: "DE"},
{Disposition: wafprotocol.DispositionAllow, Country: "DE"},
{Disposition: wafprotocol.DispositionDeny},
{Disposition: wafprotocol.DispositionClose},
} {
header := http.Header{}
wafprotocol.SetAnswerHeaders(header, want)
got, valid := wafprotocol.ReadAnswer(header)
if !valid || got != want {
t.Errorf("read %+v (valid: %t), want %+v", got, valid, want)
}
}
header := http.Header{}
header.Set(wafprotocol.DispositionHeader, "maybe")
header.Set(wafprotocol.ASNHeader, "64496")
got, valid := wafprotocol.ReadAnswer(header)
if valid || got != (wafprotocol.Answer{}) {
t.Errorf("read %+v (valid: %t), want nothing", got, valid)
}
}
// TestReportReadsAsWritten checks that ReadReport gives back the report
// NewReportRequest posts.
func TestReportReadsAsWritten(t *testing.T) {
t.Parallel()
want := wafprotocol.Report{
RequestID: requestID,
Status: http.StatusOK,
RequestBytes: 10,
ResponseBytes: 20,
DurationTotalMS: 1.5,
DurationUpstreamTotalMS: 0.5,
}
req, err := wafprotocol.NewReportRequest(
t.Context(),
"http://waf.example.com"+wafprotocol.ReportPath,
&want,
)
if err != nil {
t.Fatalf("write the report: %v", err)
}
if req.Method != http.MethodPost ||
req.URL.Path != wafprotocol.ReportPath {
t.Errorf("the report is %s %s", req.Method, req.URL.Path)
}
got, err := wafprotocol.ReadReport(httptest.NewRecorder(), req)
if err != nil || got != want {
t.Errorf("read %+v (%v), want %+v", got, err, want)
}
}