Compare commits

..
Author SHA1 Message Date
sneak 9d3f212b0d waf: a heuristic says itself that it refuses whatever the lookup answers (closes #446)
check / check (push) Waiting to run
refusedWhateverTheAnswer asks each heuristic of afterLookup that
implements the new refusesWithoutLookup interface, instead of naming
blocklists and rate_limits. The comment on adding a heuristic names the
places outside its file that may change with it.

Model: opus-5-5
2026-11-19 21:59:05 +00:00
clawbot 51989235d6 rproxy: the body wrappers hold only what they call (closes #431)
check / check (push) Waiting to run
requestBody and responseBody are read on goroutines other than the
handler's. They held the whole request; now they hold only the
functions they call and, for responseBody, the request's context.

Model: opus-5-5
2026-11-19 22:43:57 +01:00
clawbot e15edef583 Move what rproxy and waf send each other into wafprotocol (closes #425)
check / check (push) Waiting to run
internal/wafprotocol writes and reads the question, the answer and the
report, and declares once each header, path and limit both daemons use.
rproxy no longer imports internal/waf. Nothing on the wire changes.

Model: opus-5-5
2026-11-19 22:27:16 +01:00
clawbot 06faf0877c dirwatch: one loop that watches a directory, for rules and lookup (closes #429)
check / check (push) Waiting to run
rules and lookup each had the same loop: watch the directory, start a
timer at once, restart it at each change that counts, read again when it
fires. It now lives once in internal/dirwatch, with its quiet time and
its tests. Each package keeps its own log messages, its readAgain, and
its test for which changes count. internal/state keeps its own copy.

Model: opus-5-5
2026-11-19 22:13:55 +01:00
clawbot b2b1716907 sww: the three commands as cobra commands (closes #440)
check / check (push) Waiting to run
Main builds a cobra root command with rproxy, waf and healthcheck under
it, in place of the switch on os.Args and the hand-kept usage text.
Exit statuses stay: 2 with no command or an unknown one, otherwise what
the command returns. cobra's completion command is off. `sww help` and
`sww --help` now print the commands and exit 0.

Model: opus-5-5
2026-11-19 21:59:25 +01:00
clawbot 56a57125f1 waf: each heuristic holds its own settings and parts (closes #441)
check / check (push) Waiting to run
Every heuristic is a struct built once in newHandler with the settings
and collaborators it reads; no check reads the handler. The six lists
are handler fields built there. Code several heuristics share takes what
it needs as arguments or lives on a struct they hold: banMaker for bans
and their alerts, reputationHitNoter for reputation hits,
biasedThresholds for limit percentages. The handler keeps blocklists,
rateLimits, trapPaths, ruleFiles and coreRuleSet as fields too, since
refusedWhateverTheAnswer and findBesideLookup use them.

Model: opus-5-5
2026-11-19 21:43:55 +01:00
57 changed files with 1944 additions and 970 deletions

No files matched your search

+2
View File
@@ -902,6 +902,8 @@ stops the running process (see [`docs/state-files.md`](docs/state-files.md)).
- `internal/rproxy`: the reverse proxy in front of one app.
- `internal/waf`: `waf`'s server, which runs the checks and answers its own
endpoints.
- `internal/wafprotocol`: the question, the answer and the report that `rproxy`
and `waf` send each other.
- `internal/ratelimit`, `internal/bans`, `internal/rules`,
`internal/coreruleset`, `internal/geojs`, `internal/lookup`,
`internal/reputation`, `internal/routewatch`, `internal/anomaly` and
+10 -5
View File
@@ -69,17 +69,21 @@ never as a time.
that command's settings, and runs `internal/rproxy` until `SIGTERM` or
`SIGINT` (see "`rproxy`"). `sww healthcheck` is the image's health check.
With no command, or one it does not know, the binary prints the commands
and exits with status 2. `internal/sww` is the only package that imports
and exits with status 2. `sww help` and `sww --help` print them and exit
with status 0. `internal/sww` is the only package that imports
`go.uber.org/fx`. fx builds the parts of `waf` and `rproxy`, and starts
and stops their background work.
- `internal/config`: reads the settings, the one place they are read.
- `internal/waf`: `waf`'s server. It runs the checks in `check`, and holds
`waf`'s question, answer and report. It answers `waf`'s own endpoints in
`answerOwnEndpoint`. It counts bytes, refusals and anomalies, and writes
the log lines.
- `internal/waf`: `waf`'s server. It runs the checks in `check`, and answers
`waf`'s own endpoints in `answerOwnEndpoint`. It counts bytes, refusals
and anomalies, and writes the log lines.
- `internal/rproxy`: the reverse proxy in front of one app that asks `waf`
about each request. It passes a request to the app with the standard
library's `httputil.ReverseProxy`, within the time and size limits.
- `internal/wafprotocol`: the question, the answer and the report that
`rproxy` and `waf` send each other (see "`waf`"). It writes and reads each
one, and declares every header, path and limit both sides use.
`internal/rproxy` imports it, not `internal/waf`.
- `internal/ratelimit`: the table of clients. It counts requests, bytes and
refusals, and keeps each client's history.
- `internal/bans`: the bans. It holds each netblock's bans with their notes,
@@ -129,6 +133,7 @@ never as a time.
- `github.com/fsnotify/fsnotify` to notice edited files;
- `github.com/prometheus/client_golang` for the metrics;
- `github.com/spf13/viper` to read the settings from the environment;
- `github.com/spf13/cobra` for the commands and their help text;
- `github.com/maxmind/mmdbwriter`, in the tests, to write lookup databases;
- remote log sending written in this project, since the standard library's
`log/syslog` is frozen and writes only the older syslog format.
+2
View File
@@ -11,6 +11,7 @@ require (
github.com/maxmind/mmdbwriter v1.2.0
github.com/oschwald/maxminddb-golang/v2 v2.7.0
github.com/prometheus/client_golang v1.24.1
github.com/spf13/cobra v1.10.2
github.com/spf13/viper v1.21.0
go.uber.org/fx v1.24.0
golang.org/x/net v0.60.0
@@ -26,6 +27,7 @@ require (
github.com/goccy/go-yaml v1.19.2 // indirect
github.com/gotnospirit/makeplural v0.0.0-20180622080156-a5f48d94d976 // indirect
github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/kaptinlin/go-i18n v0.1.4 // indirect
github.com/kaptinlin/jsonschema v0.4.6 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
+9
View File
@@ -14,6 +14,7 @@ github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68 h1:KLEjvBBSw
github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68/go.mod h1:9XocblQEwK6EYylRth7PrPIifF2l0iSoYo4FAXhXU/8=
github.com/corazawaf/libinjection-go v0.3.4 h1:jX2Tit2xZxYJJyVryNXucYBZtQqS7473x9Nc6lgRRbk=
github.com/corazawaf/libinjection-go v0.3.4/go.mod h1:Ik/+w3UmTWH9yn366RgS9D95K3y7Atb5m/H/gXzzPCk=
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/foxcpp/go-mockdns v1.2.0 h1:omK3OrHRD1IWJz1FuFBCFquhXslXoF17OvBS6JPzZF0=
github.com/foxcpp/go-mockdns v1.2.0/go.mod h1:IhLeSFGed3mJIAXPH2aiRQB+kqz7oqu8ld2qVbOu7Wk=
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
@@ -34,6 +35,8 @@ github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 h1:c7gcN
github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092/go.mod h1:ZZAN4fkkful3l1lpJwF8JbW41ZiG9TwJ2ZlqzQovBNU=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/jcchavezs/mergefs v0.1.1 h1:D45R17m6dHnSVZefnhynoeZvcK2Uw0oTrRfoUOQ0S5Y=
github.com/jcchavezs/mergefs v0.1.1/go.mod h1:eRLTrsA+vFwQZ48hj8p8gki/5v9C2bFtHH5Mnn4bcGk=
github.com/kaptinlin/go-i18n v0.1.4 h1:wCiwAn1LOcvymvWIVAM4m5dUAMiHunTdEubLDk4hTGs=
@@ -72,6 +75,7 @@ github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+
github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY=
github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/sagikazarmark/locafero v0.11.0 h1:1iurJgmM9G3PA/I+wWYIOw/5SyBtxapeHDcg+AAIFXc=
github.com/sagikazarmark/locafero v0.11.0/go.mod h1:nVIGvgyzw595SUSUE6tvCp3YYTeHs15MvlmU87WwIik=
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 h1:+jumHNA0Wrelhe64i8F6HNlS8pkoyMv5sreGx2Ry5Rw=
@@ -80,6 +84,9 @@ github.com/spf13/afero v1.15.0 h1:b/YBCLWAJdFWJTN9cLhiXXcD7mzKn9Dm86dNnfyQw1I=
github.com/spf13/afero v1.15.0/go.mod h1:NC2ByUVxtQs4b3sIUphxK0NioZnmxgyCrfzeuq8lxMg=
github.com/spf13/cast v1.10.0 h1:h2x0u2shc1QuLHfxi+cTJvs30+ZAHOGRic8uyGTDWxY=
github.com/spf13/cast v1.10.0/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqeHo=
github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.21.0 h1:x5S+0EU27Lbphp4UKm1C+1oQO+rKx36vfCoaVebLFSU=
@@ -109,6 +116,7 @@ go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba h1:0b9z3AuHCjxk0x/opv64kcgZLBseWJUpBw5I82+2U4M=
@@ -127,6 +135,7 @@ golang.org/x/tools v0.50.0 h1:c2ifzfcuY7L90lZ2aKd8S4K2NpASF08SZx9ZuJkHmSU=
golang.org/x/tools v0.50.0/go.mod h1:7ulVMw3831Mwi5EZD6RomGyffr4VFjuNYXf2BbCEAV0=
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
rsc.io/binaryregexp v0.2.0 h1:HfqmD5MEmC0zvwBuF187nq9mdnXjXsSivRiXN7SmRkE=
+16 -7
View File
@@ -2,6 +2,7 @@ package rproxy
import (
"bufio"
"context"
"errors"
"io"
"net"
@@ -22,7 +23,9 @@ type requestBody struct {
// body is the visitor's body, ending in an *http.MaxBytesError past
// SWW_RPROXY_MAX_REQUEST_SIZE.
body io.ReadCloser
rq *request
// refuse refuses the request with a status, as the request's own refuse
// does.
refuse func(status int)
// held is the part read before the request goes to the app, which Read gives
// first.
held []byte
@@ -68,7 +71,7 @@ func (b *requestBody) Read(p []byte) (int, error) {
case errors.Is(err, io.EOF):
b.received.Store(true)
case errors.As(err, &tooLarge):
b.rq.refuse(http.StatusRequestEntityTooLarge)
b.refuse(http.StatusRequestEntityTooLarge)
}
return n, err
@@ -86,7 +89,13 @@ type responseBody struct {
// body is the app's body, ending in an *http.MaxBytesError past
// SWW_RPROXY_MAX_RESPONSE_SIZE.
body io.ReadCloser
rq *request
// refuse refuses the request with a status, as the request's own refuse
// does.
refuse func(status int)
// responseReceived is called once the app's whole answer has arrived.
responseReceived func()
// ctx is the visitor's request's context.
ctx context.Context //nolint:containedctx // the body ends with the request
}
// Read reads from the app's body.
@@ -100,15 +109,15 @@ func (b *responseBody) Read(p []byte) (int, error) {
switch {
case errors.Is(err, io.EOF):
b.rq.responseReceived()
b.responseReceived()
case errors.As(err, &tooLarge):
b.rq.refuse(http.StatusBadGateway)
b.refuse(http.StatusBadGateway)
return n, errResponseTooLarge
case b.rq.in.Context().Err() == nil:
case b.ctx.Err() == nil:
// The answer broke off, not because the visitor went away. If a time
// limit cut it, that refusal came first and is the one kept.
b.rq.refuse(http.StatusBadGateway)
b.refuse(http.StatusBadGateway)
}
return n, err
-10
View File
@@ -7,16 +7,6 @@ import (
"strings"
)
// The headers rproxy sets on the requests it passes to the app. Go writes every
// header name in this form, as it sends it and as it receives it, so
// X-Client-ASN arrives as X-Client-Asn, and Del removes a visitor's own whatever
// their case; header names are not case-sensitive.
const (
requestIDHeader = "X-Request-ID"
asnHeader = "X-Client-Asn"
countryHeader = "X-Client-Country"
)
// peerAddress is the address of the request's TCP peer, normally traefik, without
// any IPv6 zone, which waf refuses.
func peerAddress(r *http.Request) netip.Addr {
+3 -3
View File
@@ -20,7 +20,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/processlog"
"sneak.berlin/go/smallwebwaf/internal/rproxy"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// post is the start of a POST whose headers end with what follows it.
@@ -492,7 +492,7 @@ func TestASettingOfOffSetsNoLimit(t *testing.T) {
strings.NewReader("a body"))),
)
if got := get(t, addr, waf.HealthPath); got.status != http.StatusOK {
if got := get(t, addr, wafprotocol.HealthPath); got.status != http.StatusOK {
t.Errorf("answered %+v, want rproxy's own 200", got)
}
})
@@ -1480,7 +1480,7 @@ func TestAVisitorThatStopsALittleShortGetsAnAnswerRProxyGivesAtOnce(
{"a deny", post, http.StatusForbidden},
{
"the health check",
"GET " + waf.HealthPath + " HTTP/1.1\r\nHost: site.example\r\n",
"GET " + wafprotocol.HealthPath + " HTTP/1.1\r\nHost: site.example\r\n",
http.StatusOK,
},
} {
+13 -13
View File
@@ -12,11 +12,11 @@ import (
"strings"
"testing"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// questionBodyMaxBytes is the most of a body a question carries.
const questionBodyMaxBytes = waf.QuestionBodyMaxBytes
const questionBodyMaxBytes = wafprotocol.QuestionBodyMaxBytes
// chunked is the encoding of a body whose length the visitor did not announce.
const chunked = "chunked"
@@ -42,9 +42,9 @@ func TestQuestionIsTheVisitorsRequestWithRProxysHeaders(t *testing.T) {
req.Header.Set(forwardedForHeader, "198.51.100.1")
req.Header.Set(forwardedProtoHeader, "gopher")
req.Header.Set(requestIDHeader, "the-visitor's")
req.Header.Set(waf.ClientHeader, "198.51.100.2")
req.Header.Set(waf.PeerHeader, "198.51.100.3")
req.Header.Set(waf.ContentLengthHeader, "12345")
req.Header.Set(wafprotocol.ClientHeader, "198.51.100.2")
req.Header.Set(wafprotocol.PeerHeader, "198.51.100.3")
req.Header.Set(wafprotocol.ContentLengthHeader, "12345")
// Go's client then sends no User-Agent, so that one in the question would be
// Go's own.
req.Header["User-Agent"] = nil
@@ -73,8 +73,8 @@ func TestQuestionIsTheVisitorsRequestWithRProxysHeaders(t *testing.T) {
want.Set(forwardedForHeader, "198.51.100.1")
want.Set(forwardedProtoHeader, "http")
want.Set(requestIDHeader, asked.header.Get(requestIDHeader))
want.Set(waf.ClientHeader, localhost)
want.Set(waf.PeerHeader, localhost)
want.Set(wafprotocol.ClientHeader, localhost)
want.Set(wafprotocol.PeerHeader, localhost)
if !maps.EqualFunc(asked.header, want, slices.Equal) {
t.Errorf(
@@ -124,8 +124,8 @@ func TestQuestionNamesTheVisitorATrustedProxySpeaksFor(t *testing.T) {
wantGreeting(t, visit(t, req))
asked := standIn.question(t)
if asked.header.Get(waf.ClientHeader) != tc.client ||
asked.header.Get(waf.PeerHeader) != localhost ||
if asked.header.Get(wafprotocol.ClientHeader) != tc.client ||
asked.header.Get(wafprotocol.PeerHeader) != localhost ||
asked.header.Get(forwardedForHeader) != visitor ||
asked.header.Get(forwardedProtoHeader) != tc.proto {
t.Errorf("the question's headers are %v", asked.header)
@@ -191,7 +191,7 @@ func TestQuestionNamesTheClientAsTheForwardedAddressesSay(t *testing.T) {
req.Header[forwardedForHeader] = tc.header
wantGreeting(t, visit(t, req))
if got := standIn.question(t).header.Get(waf.ClientHeader); got !=
if got := standIn.question(t).header.Get(wafprotocol.ClientHeader); got !=
tc.client {
t.Errorf("the question names %s, want %s", got, tc.client)
}
@@ -282,7 +282,7 @@ func wantQuestionBody(
)
}
values := asked.header.Values(waf.ContentLengthHeader)
values := asked.header.Values(wafprotocol.ContentLengthHeader)
if announced {
if !slices.Equal(values, []string{strconv.Itoa(size)}) ||
@@ -586,8 +586,8 @@ func TestAQuestionNamesNoZone(t *testing.T) {
server.Handler.ServeHTTP(httptest.NewRecorder(), req)
asked := standIn.question(t)
if asked.header.Get(waf.ClientHeader) != tc.client ||
asked.header.Get(waf.PeerHeader) != tc.peer {
if asked.header.Get(wafprotocol.ClientHeader) != tc.client ||
asked.header.Get(wafprotocol.PeerHeader) != tc.peer {
t.Errorf("the question's headers are %v", asked.header)
}
+19 -16
View File
@@ -18,17 +18,13 @@ import (
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// flushAfterEachWrite has ReverseProxy pass on each part of the app's answer as
// soon as it arrives.
const flushAfterEachWrite time.Duration = -1
// reportStatusMax is the highest status waf takes in a report. An app can
// send a higher one, up to 999, and the visitor gets that as it is.
const reportStatusMax = 599
// plainText is the Content-Type of the answers rproxy gives itself.
const plainText = "text/plain; charset=utf-8"
@@ -126,8 +122,8 @@ func (h *handler) newRequest(
if r.Body != http.NoBody {
rq.body = &requestBody{
body: limitBody(r.Body, h.config.MaxRequestSize),
rq: rq,
body: limitBody(r.Body, h.config.MaxRequestSize),
refuse: rq.refuse,
}
}
@@ -154,7 +150,7 @@ func (rq *request) readFirstPartOfBody() bool {
_ = rq.rc.SetReadDeadline(rq.requestDeadline())
held, err := io.ReadAll(
io.LimitReader(rq.body, waf.QuestionBodyMaxBytes),
io.LimitReader(rq.body, wafprotocol.QuestionBodyMaxBytes),
)
// From here on, the request timer ends a read of the rest of the body, so that
// the answer can name the side rproxy was waiting on.
@@ -228,16 +224,17 @@ func (rq *request) rewrite(pr *httputil.ProxyRequest) {
setPathAsSent(pr.Out.URL, pr.In)
setForwardedHeaders(pr.In, pr.Out, rq.peer, rq.peerTrusted)
pr.Out.Header.Set(requestIDHeader, rq.id)
pr.Out.Header.Del(asnHeader)
pr.Out.Header.Del(countryHeader)
pr.Out.Header.Set(wafprotocol.RequestIDHeader, rq.id)
// Del removes a visitor's own, whatever their case.
pr.Out.Header.Del(wafprotocol.ASNHeader)
pr.Out.Header.Del(wafprotocol.CountryHeader)
if rq.asn != "" {
pr.Out.Header.Set(asnHeader, rq.asn)
pr.Out.Header.Set(wafprotocol.ASNHeader, rq.asn)
}
if rq.country != "" {
pr.Out.Header.Set(countryHeader, rq.country)
pr.Out.Header.Set(wafprotocol.CountryHeader, rq.country)
}
}
@@ -294,7 +291,12 @@ func (rq *request) modifyResponse(res *http.Response) error {
// the request, and SWW_RPROXY_MAX_RESPONSE_TIME_MS starts.
rq.startResponseTimer(requestEnd)
res.Body = &responseBody{body: limitBody(res.Body, maxBytes), rq: rq}
res.Body = &responseBody{
body: limitBody(res.Body, maxBytes),
refuse: rq.refuse,
responseReceived: rq.responseReceived,
ctx: rq.in.Context(),
}
rq.out.afterStatus = rq.setResponseDeadline
return nil
@@ -470,9 +472,10 @@ func (rq *request) finish(ctx context.Context) {
return
}
report := &waf.Report{
// An app can send a status up to 999, and the visitor gets it as it is.
report := &wafprotocol.Report{
RequestID: rq.id,
Status: min(rq.out.status, reportStatusMax),
Status: min(rq.out.status, wafprotocol.ReportStatusMax),
RequestBytes: requestBytes,
ResponseBytes: responseBytes,
DurationTotalMS: requestlog.Milliseconds(end.Sub(rq.start)),
+10 -12
View File
@@ -24,7 +24,7 @@ import (
"time"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// How rproxy keeps connections to the app and to waf open between
@@ -47,10 +47,6 @@ const (
idleTimeout = 120 * time.Second
)
// ownEndpointPrefix starts the path of every request for sww itself, which
// never reaches the app.
const ownEndpointPrefix = "/_sww/"
// allowedMethods lists the methods rproxy passes on, as an Allow header gives
// them. It answers any other method 405 itself.
// The WebDAV ones are for Nextcloud.
@@ -92,7 +88,9 @@ func New(params Params) *Server {
transport: newTransport(),
wafTransport: newWAFTransport(),
reportURL: (&url.URL{
Scheme: backend.Scheme, Host: backend.Host, Path: waf.ReportPath,
Scheme: backend.Scheme,
Host: backend.Host,
Path: wafprotocol.ReportPath,
}).String(),
breakerClock: time.Now,
}
@@ -179,7 +177,7 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
defer rq.readRestOfBody()
defer rq.finish(r.Context())
if r.Method == http.MethodGet && r.URL.Path == waf.HealthPath {
if r.Method == http.MethodGet && r.URL.Path == wafprotocol.HealthPath {
rq.answerHealth()
return
@@ -212,9 +210,9 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
rq.noteAnswer(answer)
switch rq.disposition {
case waf.DispositionAllow:
case wafprotocol.DispositionAllow:
rq.passToApp(r.Context())
case waf.DispositionClose:
case wafprotocol.DispositionClose:
// This panic has Go's server close the connection without an answer,
// and log nothing; the deferred finish still runs.
panic(http.ErrAbortHandler)
@@ -224,8 +222,8 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
}
// passToApp passes the request on to the app, unless a limit or its path keeps
// it from the app. waf answers a path under ownEndpointPrefix itself, so
// such a request gets here only when
// it from the app. waf answers a path under wafprotocol.OwnEndpointPrefix
// itself, so such a request gets here only when
// SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION allowed it.
// ctx is the request's own context.
func (rq *request) passToApp(ctx context.Context) {
@@ -234,7 +232,7 @@ func (rq *request) passToApp(ctx context.Context) {
switch {
case rq.refused.Load() != 0 || (maxBytes > 0 && rq.in.ContentLength > maxBytes):
rq.answer(http.StatusRequestEntityTooLarge)
case strings.HasPrefix(rq.in.URL.Path, ownEndpointPrefix):
case strings.HasPrefix(rq.in.URL.Path, wafprotocol.OwnEndpointPrefix):
rq.answer(http.StatusNotFound)
default:
rq.forward(ctx)
+18 -18
View File
@@ -23,7 +23,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/processlog"
"sneak.berlin/go/smallwebwaf/internal/rproxy"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
const (
@@ -56,7 +56,7 @@ const (
// The headers the tests set and look at.
const (
dispositionHeader = waf.DispositionHeader
dispositionHeader = wafprotocol.DispositionHeader
forwardedForHeader = "X-Forwarded-For"
forwardedProtoHeader = "X-Forwarded-Proto"
requestIDHeader = "X-Request-ID"
@@ -72,9 +72,9 @@ const (
// What the stand-in for waf answers.
const (
dispositionAllow = waf.DispositionAllow
dispositionDeny = waf.DispositionDeny
dispositionClose = waf.DispositionClose
dispositionAllow = wafprotocol.DispositionAllow
dispositionDeny = wafprotocol.DispositionDeny
dispositionClose = wafprotocol.DispositionClose
// deniedBody is the body of a deny.
deniedBody = "refused by waf"
)
@@ -175,7 +175,7 @@ type WAF struct {
*httptest.Server
questions chan question
reports chan waf.Report
reports chan wafprotocol.Report
}
// startWAF starts a stand-in for waf.
@@ -184,11 +184,11 @@ func startWAF(t *testing.T, answer http.HandlerFunc) *WAF {
s := &WAF{
questions: make(chan question, 10),
reports: make(chan waf.Report, 10),
reports: make(chan wafprotocol.Report, 10),
}
s.Server = httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(waf.ClientHeader) == "" {
if r.Header.Get(wafprotocol.ClientHeader) == "" {
s.takeReport(t, w, r)
return
@@ -217,11 +217,11 @@ func (s *WAF) takeReport(
) {
t.Helper()
var report waf.Report
var report wafprotocol.Report
err := json.NewDecoder(r.Body).Decode(&report)
if err != nil || r.Method != http.MethodPost ||
r.URL.Path != waf.ReportPath {
r.URL.Path != wafprotocol.ReportPath {
t.Errorf("%s %s is no report: %v", r.Method, r.URL.Path, err)
}
@@ -242,7 +242,7 @@ func (s *WAF) wantNoReport(t *testing.T) {
}
// report waits for a report.
func (s *WAF) report(t *testing.T) waf.Report {
func (s *WAF) report(t *testing.T) wafprotocol.Report {
t.Helper()
select {
@@ -251,7 +251,7 @@ func (s *WAF) report(t *testing.T) waf.Report {
case <-time.After(waitLimit):
t.Fatal("no report")
return waf.Report{}
return wafprotocol.Report{}
}
}
@@ -279,7 +279,7 @@ func startWAFAnsweringReports(
backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(waf.ClientHeader) != "" {
if r.Header.Get(wafprotocol.ClientHeader) != "" {
answer(w, r)
return
@@ -829,7 +829,7 @@ func TestAnAnswerRProxyHoldsWholeIsSentBeforeItsLogLine(t *testing.T) {
}{
{"a deny", "GET /" + host, deniedBody, http.StatusForbidden},
{"rproxy's own answer", "GET x:admin" + host, "Bad Request\n", http.StatusBadRequest},
{"the health check", "GET " + waf.HealthPath + host, "ok\n", http.StatusOK},
{"the health check", "GET " + wafprotocol.HealthPath + host, "ok\n", http.StatusOK},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
@@ -1602,19 +1602,19 @@ func TestHealthCheckIsAnsweredWhileWAFIsDown(t *testing.T) {
app := startApp(t, greet)
addr, out := startRProxy(t, nowhere, app.URL, nil)
got := get(t, addr, waf.HealthPath)
got := get(t, addr, wafprotocol.HealthPath)
if got.status != http.StatusOK || got.body != "ok\n" {
t.Errorf("answered %+v, want ok", got)
}
line := out.requestLine(t)
if line["path"] != waf.HealthPath || line["disposition"] != nil {
if line["path"] != wafprotocol.HealthPath || line["disposition"] != nil {
t.Errorf("line %v, want one with no disposition", line)
}
// With waf up too, it is not asked.
addr, _ = startRProxy(t, standIn.URL, app.URL, nil)
get(t, addr, waf.HealthPath)
get(t, addr, wafprotocol.HealthPath)
if len(standIn.questions) != 0 {
t.Error("waf was asked about the health check")
@@ -1775,7 +1775,7 @@ func TestWaitForReportsWaitsForAReportBeingSent(t *testing.T) {
release := make(chan struct{})
backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(waf.ClientHeader) != "" {
if r.Header.Get(wafprotocol.ClientHeader) != "" {
allow(w, r)
return
+34 -82
View File
@@ -1,9 +1,7 @@
package rproxy
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
@@ -11,7 +9,8 @@ import (
"net/http"
"strings"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/rules"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// answerBodyMaxBytes is the most of the body of waf's answer that is read.
@@ -40,15 +39,12 @@ type wafAnswer struct {
// notes nothing, and no report is due for it: waf holds no decision to report
// on.
func (rq *request) noteAnswer(answer *wafAnswer) {
disposition := answer.header.Get(waf.DispositionHeader)
switch disposition {
case waf.DispositionAllow, waf.DispositionDeny,
waf.DispositionClose:
rq.disposition = disposition
read, valid := wafprotocol.ReadAnswer(answer.header)
if valid {
rq.disposition = read.Disposition
rq.answered = true
rq.asn = answer.header.Get(asnHeader)
rq.country = answer.header.Get(countryHeader)
rq.asn = read.ASN
rq.country = read.Country
}
}
@@ -79,7 +75,7 @@ func (rq *request) handleNoUsableAnswer(ctx context.Context, err error) {
rq.disposition = rq.h.config.WAFBackendNoAnswerDisposition
rq.wafNoAnswer = true
if rq.disposition == waf.DispositionAllow {
if rq.disposition == wafprotocol.DispositionAllow {
rq.passToApp(ctx)
} else {
rq.answer(http.StatusServiceUnavailable)
@@ -119,7 +115,21 @@ func (rq *request) ask(ctx context.Context) (*wafAnswer, error) {
ctx, cancel := context.WithTimeout(ctx, rq.h.config.WAFBackendTimeout)
defer cancel()
question, err := rq.newQuestion(ctx)
var held []byte
if rq.body != nil {
held = rq.body.held
}
question, err := wafprotocol.NewQuestion(ctx, wafprotocol.QuestionParams{
Visitor: rq.in,
PathAsSent: rules.Path(rq.in),
Body: held,
WAFBackend: rq.h.config.WAFBackend,
Client: rq.client,
Peer: rq.peer,
RequestID: rq.id,
Scheme: scheme(rq.in, rq.peerTrusted),
})
if err != nil {
return nil, err
}
@@ -176,69 +186,13 @@ func canBeAsked(r *http.Request) bool {
len(r.Header.Values("User-Agent")) <= 1
}
// newQuestion returns the question about the request, as "The question" in
// SPEC.md gives it: the visitor's request as the visitor sent it, with the part
// of the body it holds and rproxy's headers.
func (rq *request) newQuestion(ctx context.Context) (*http.Request, error) {
in := rq.in
var held []byte
if rq.body != nil {
held = rq.body.held
}
question, err := http.NewRequestWithContext( //nolint:gosec // a setting's URL
ctx,
in.Method,
rq.h.config.WAFBackend.String(),
bytes.NewReader(held),
)
if err != nil {
return nil, fmt.Errorf("make the question: %w", err)
}
backend := question.URL
target := *in.URL
target.Scheme = backend.Scheme
target.Host = backend.Host
setPathAsSent(&target, in)
question.URL = &target
question.Host = in.Host
// The length NewRequestWithContext set is that of what is held. This line makes
// a body whose length the visitor did not announce go chunked, as "The question"
// in SPEC.md requires.
question.TransferEncoding = in.TransferEncoding
question.Header = in.Header.Clone()
question.Header.Set(waf.ClientHeader, rq.client.String())
question.Header.Set(waf.PeerHeader, rq.peer.String())
question.Header.Del(waf.ContentLengthHeader)
announced := in.Header.Get("Content-Length")
if announced != "" {
question.Header.Set(waf.ContentLengthHeader, announced)
}
question.Header.Set(requestIDHeader, rq.id)
question.Header.Set("X-Forwarded-Proto", scheme(in, rq.peerTrusted))
// Go's client adds a User-Agent of its own to a request without the key.
_, sent := question.Header["User-Agent"]
if !sent {
question.Header["User-Agent"] = nil
}
return question, nil
}
// answerWith gives the visitor waf's answer, less its disposition.
func (rq *request) answerWith(answer *wafAnswer) {
rq.prepareOwnAnswer(rq.requestDeadline())
header := rq.out.Header()
maps.Copy(header, answer.header)
header.Del(waf.DispositionHeader)
header.Del(wafprotocol.DispositionHeader)
// Go's server adds a Content-Type it guesses from the body to a response whose
// headers have no Content-Type key. The key with no value sends none.
@@ -252,7 +206,10 @@ func (rq *request) answerWith(answer *wafAnswer) {
// sendReport sends waf report on a goroutine of its own, so that the
// visitor does not wait for it. The report goes on after ctx, the request's own
// context, has ended.
func (h *handler) sendReport(ctx context.Context, report *waf.Report) {
func (h *handler) sendReport(
ctx context.Context,
report *wafprotocol.Report,
) {
h.reportsMu.Lock()
defer h.reportsMu.Unlock()
@@ -270,23 +227,18 @@ func (h *handler) sendReport(ctx context.Context, report *waf.Report) {
// postReport posts report to waf, and waits for its answer for up to
// SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS.
func (h *handler) postReport(ctx context.Context, report *waf.Report) error {
body, err := json.Marshal(report)
if err != nil {
return fmt.Errorf("encode the report: %w", err)
}
func (h *handler) postReport(
ctx context.Context,
report *wafprotocol.Report,
) error {
ctx, cancel := context.WithTimeout(ctx, h.config.WAFBackendTimeout)
defer cancel()
req, err := http.NewRequestWithContext( //nolint:gosec // a setting's URL
ctx, http.MethodPost, h.reportURL, bytes.NewReader(body))
req, err := wafprotocol.NewReportRequest(ctx, h.reportURL, report)
if err != nil {
return fmt.Errorf("make the request: %w", err)
return err
}
req.Header.Set("Content-Type", "application/json")
res, err := h.wafTransport.RoundTrip(req)
if err != nil {
return fmt.Errorf("send the report: %w", err)
+2 -2
View File
@@ -13,7 +13,7 @@ import (
"github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// healthCheckTimeout bounds the whole health check.
@@ -116,7 +116,7 @@ func askHealthEndpoint(ctx context.Context, listenAddr string) error {
health := "http://" + net.JoinHostPort(
"127.0.0.1",
port,
) + waf.HealthPath
) + wafprotocol.HealthPath
req, err := http.NewRequestWithContext(
ctx,
+17 -17
View File
@@ -21,7 +21,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/sww"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
const (
@@ -164,7 +164,7 @@ func TestABanThroughOneSiteRefusesTheClientOnTheOther(t *testing.T) {
// One request's three lines carry one id, rproxy's with waf's answer.
line := firstOut.line(t, "path", bannedProbe)
if line["disposition"] != waf.DispositionDeny ||
if line["disposition"] != wafprotocol.DispositionDeny ||
line["client_ip"] != placed {
t.Errorf("rproxy's line is %v", line)
}
@@ -249,7 +249,7 @@ func TestTheSiteAnswersWhileWAFIsStoppedAndStartedAgain(t *testing.T) {
wantStatus(t, site+"/after-stop", placed, http.StatusOK)
line := out.line(t, "path", "/after-stop")
if line["disposition"] != waf.DispositionAllow ||
if line["disposition"] != wafprotocol.DispositionAllow ||
line["waf_backend_no_answer"] != true {
t.Errorf("rproxy's line is %v", line)
}
@@ -437,7 +437,7 @@ func TestAChunkedBodyStaysABodyAtWAF(t *testing.T) {
const smuggledID = "smuggled-request"
smuggled := "GET " + bannedProbe + " HTTP/1.1\r\nHost: site.example\r\n" +
waf.ClientHeader + ": " + otherClient + "\r\n" +
wafprotocol.ClientHeader + ": " + otherClient + "\r\n" +
"X-Request-ID: " + smuggledID + "\r\n\r\n"
status := sendRaw(
@@ -605,10 +605,11 @@ func TestRProxyAsksAWAFThatSpeaksHTTP2OverHTTP1(t *testing.T) {
protocols := make(chan string, 10)
backend := httptest.NewUnstartedServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(waf.ClientHeader) != "" {
if r.Header.Get(wafprotocol.ClientHeader) != "" {
protocols <- r.Proto
w.Header().Set(waf.DispositionHeader, waf.DispositionDeny)
w.Header().
Set(wafprotocol.DispositionHeader, wafprotocol.DispositionDeny)
w.WriteHeader(http.StatusForbidden)
}
}))
@@ -690,8 +691,9 @@ func TestRProxyStoppingWaitsForAReportBeingSent(t *testing.T) {
release := make(chan struct{})
backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(waf.ClientHeader) != "" {
w.Header().Set(waf.DispositionHeader, waf.DispositionAllow)
if r.Header.Get(wafprotocol.ClientHeader) != "" {
w.Header().
Set(wafprotocol.DispositionHeader, wafprotocol.DispositionAllow)
return
}
@@ -779,10 +781,7 @@ func sendAndKeepOpen(t *testing.T, addr, request string) net.Conn {
func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
t.Parallel()
const (
cutOffAfter = 5 * time.Second
upstreamSetting = "SWW_RPROXY_UPSTREAM_URL"
)
const cutOffAfter = 5 * time.Second
asked := make(chan struct{})
release := make(chan struct{})
@@ -796,7 +795,8 @@ func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set(waf.DispositionHeader, waf.DispositionAllow)
w.Header().
Set(wafprotocol.DispositionHeader, wafprotocol.DispositionAllow)
}))
t.Cleanup(backend.Close)
@@ -804,9 +804,9 @@ func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
defer stop()
env := map[string]string{
rproxyListenAddr: localhost + ":0",
upstreamSetting: app.URL,
wafBackend: backend.URL,
rproxyListenAddr: localhost + ":0",
rproxyUpstreamURL: app.URL,
wafBackend: backend.URL,
}
out := &output{}
exited := make(chan int, 1)
@@ -932,7 +932,7 @@ func TestMainRunsRProxy(t *testing.T) {
addr, stop := startRProxyProcess(t, wafBackend+"=http://127.0.0.1:9")
req, err := http.NewRequestWithContext(t.Context(), http.MethodGet,
"http://"+addr+waf.HealthPath, http.NoBody)
"http://"+addr+wafprotocol.HealthPath, http.NoBody)
if err != nil {
t.Fatalf("new request: %v", err)
}
+52 -30
View File
@@ -10,13 +10,14 @@ package sww
import (
"context"
"fmt"
"errors"
"io"
"os"
"os/signal"
"syscall"
"time"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/config"
@@ -26,14 +27,7 @@ import (
// know.
const usageStatus = 2
// usage is what sww prints then.
const usage = `usage: sww <command>
commands:
rproxy the reverse proxy in front of one app
waf decides for rproxy, and keeps bans, counters and history
healthcheck the container's health check
`
var errNoCommand = errors.New("no command given")
// Params are what RunWAF and RunRProxy need from the process.
type Params struct {
@@ -51,14 +45,9 @@ type Params struct {
// Main runs the command os.Args names, and returns the process's exit status.
// `sww rproxy` and `sww waf` run until SIGTERM or SIGINT.
// With no command, or one it does not know, it prints the commands and returns
// usageStatus.
// With no command, or one it does not know, cobra prints the error and the
// commands to stderr, and Main returns usageStatus.
func Main(version string) int {
args := os.Args[1:]
if len(args) == 0 {
return printUsage(os.Stderr)
}
ctx, stop := signal.NotifyContext(context.Background(),
syscall.SIGTERM, os.Interrupt)
defer stop()
@@ -69,21 +58,54 @@ func Main(version string) int {
Stdout: os.Stdout,
}
switch args[0] {
case "rproxy":
return RunRProxy(ctx, params)
case "waf":
return RunWAF(ctx, params)
case "healthcheck":
return HealthCheck(ctx, args[1:], params.Viper, os.Stderr)
default:
return printUsage(os.Stderr)
// status is what the command that ran returns.
status := 0
root := &cobra.Command{
Use: "sww",
Short: "a web application firewall",
Args: cobra.NoArgs,
RunE: func(*cobra.Command, []string) error {
return errNoCommand
},
}
}
root.CompletionOptions.DisableDefaultCmd = true
// printUsage writes the commands to w, and returns usageStatus.
func printUsage(w io.Writer) int {
_, _ = fmt.Fprint(w, usage)
root.AddCommand(
&cobra.Command{
Use: "rproxy",
Short: "the reverse proxy in front of one app",
Run: func(cmd *cobra.Command, _ []string) {
status = RunRProxy(cmd.Context(), params)
},
},
&cobra.Command{
Use: "waf",
Short: "decides for rproxy, and keeps bans, counters and history",
Run: func(cmd *cobra.Command, _ []string) {
status = RunWAF(cmd.Context(), params)
},
},
&cobra.Command{
Use: "healthcheck",
Short: "the container's health check",
// HealthCheck refuses every argument itself, flags included.
DisableFlagParsing: true,
Run: func(cmd *cobra.Command, args []string) {
status = HealthCheck(
cmd.Context(),
args,
params.Viper,
cmd.ErrOrStderr(),
)
},
},
)
return usageStatus
err := root.ExecuteContext(ctx)
if err != nil {
return usageStatus
}
return status
}
+2 -2
View File
@@ -23,7 +23,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
"sneak.berlin/go/smallwebwaf/internal/sww"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
const (
@@ -1813,7 +1813,7 @@ func statusFrom(t *testing.T, url, from string) int {
}
req.Header.Set("X-Forwarded-For", from)
req.Header.Set(waf.ClientHeader, from)
req.Header.Set(wafprotocol.ClientHeader, from)
req.Header.Set("X-Request-ID", rand.Text())
transport := &http.Transport{}
+102 -37
View File
@@ -4,6 +4,7 @@ import (
"bytes"
"context"
"crypto/rand"
"errors"
"io"
"net/http"
"os"
@@ -20,7 +21,7 @@ import (
_ "time/tzdata"
"sneak.berlin/go/smallwebwaf/internal/sww"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// commandForMain, while set in the environment, has this test binary run Main
@@ -106,7 +107,7 @@ func TestMainRunsWAF(t *testing.T) {
starting := out.lineBeforeExit(t, "starting", exited)
addr, _ := starting["address"].(string)
ask(t, addr, "/", http.StatusOK, waf.DispositionAllow)
ask(t, addr, "/", http.StatusOK, wafprotocol.DispositionAllow)
// The files are written as sww stops.
err = cmd.Process.Signal(syscall.SIGTERM)
@@ -164,46 +165,104 @@ func wantUTC(t *testing.T, where string, value any) {
func TestMainWithoutACommandItKnowsPrintsTheCommands(t *testing.T) {
t.Parallel()
// completion is cobra's own command, which sww turns off.
for _, command := range []string{"", "serve", "completion"} {
t.Run("command "+command, func(t *testing.T) {
t.Parallel()
code, stdout, stderr := runMain(t, command)
if code != 2 {
t.Errorf("exit status %d, want 2", code)
}
wantTheCommands(t, "stderr", stderr)
if stdout != "" {
t.Errorf("stdout %q, want nothing", stdout)
}
})
}
}
func TestMainHelpPrintsTheCommands(t *testing.T) {
t.Parallel()
for _, command := range []string{"help", "--help"} {
t.Run("command "+command, func(t *testing.T) {
t.Parallel()
code, stdout, _ := runMain(t, command)
if code != 0 {
t.Errorf("exit status %d, want 0", code)
}
wantTheCommands(t, "stdout", stdout)
})
}
}
// wantTheCommands checks that text, what Main wrote to where, names each
// command.
func wantTheCommands(t *testing.T, where, text string) {
t.Helper()
for _, name := range []string{"rproxy", wafCommand, "healthcheck"} {
if !strings.Contains(text, name) {
t.Errorf("%s %q does not name %s", where, text, name)
}
}
}
func TestMainHealthCheckRefusesAnArgument(t *testing.T) {
t.Parallel()
for _, argument := range []string{"now", "--x", "-h"} {
t.Run("argument "+argument, func(t *testing.T) {
t.Parallel()
code, stdout, stderr := runMain(t, "healthcheck "+argument)
want := "sww healthcheck: unexpected argument \"" +
argument + "\"\n"
if code != 1 || stderr != want || stdout != "" {
t.Errorf("exit status %d, stdout %q and stderr %q, "+
"want 1, nothing and %q", code, stdout, stderr, want)
}
})
}
}
// runMain runs Main with command in a process of its own, and returns its
// exit status and what it wrote to stdout and stderr.
func runMain(t *testing.T, command string) (int, string, string) {
t.Helper()
binary, err := os.Executable()
if err != nil {
t.Fatalf("find the test binary: %v", err)
}
for _, command := range []string{"", "serve"} {
t.Run("command "+command, func(t *testing.T) {
t.Parallel()
ctx, cancel := context.WithTimeout(t.Context(), waitLimit)
defer cancel()
ctx, cancel := context.WithTimeout(t.Context(), waitLimit)
defer cancel()
var stdout, stderr bytes.Buffer
var stdout, stderr bytes.Buffer
//nolint:gosec // this test binary
cmd := exec.CommandContext(ctx, binary)
cmd.Stdout, cmd.Stderr = &stdout, &stderr
cmd := exec.CommandContext(ctx, binary)
cmd.Stdout, cmd.Stderr = &stdout, &stderr
cmd.Env = append(os.Environ(), commandForMain+"="+command)
cmd.Env = append(os.Environ(), commandForMain+"="+command)
err = cmd.Run()
err := cmd.Run()
if code := cmd.ProcessState.ExitCode(); code != 2 {
t.Errorf("exit status %d (%v), want 2", code, err)
}
for _, name := range []string{"rproxy", wafCommand, "healthcheck"} {
if !strings.Contains(stderr.String(), name) {
t.Errorf(
"stderr %q does not name %s",
stderr.String(),
name,
)
}
}
if stdout.Len() != 0 {
t.Errorf("stdout %q, want nothing", stdout.String())
}
})
var exitErr *exec.ExitError
if err != nil && !errors.As(err, &exitErr) {
t.Fatalf("run: %v", err)
}
return cmd.ProcessState.ExitCode(), stdout.String(), stderr.String()
}
// ask sends waf at addr rproxy's question about a request for path from
@@ -217,7 +276,7 @@ func ask(t *testing.T, addr, path string, status int, disposition string) {
t.Fatalf("new request: %v", err)
}
req.Header.Set(waf.ClientHeader, placed)
req.Header.Set(wafprotocol.ClientHeader, placed)
req.Header.Set("X-Request-ID", rand.Text())
transport := &http.Transport{}
@@ -232,11 +291,11 @@ func ask(t *testing.T, addr, path string, status int, disposition string) {
_ = res.Body.Close()
if res.StatusCode != status ||
res.Header.Get(waf.DispositionHeader) != disposition {
res.Header.Get(wafprotocol.DispositionHeader) != disposition {
t.Errorf(
"answered %d with disposition %q, want %d with %q",
res.StatusCode,
res.Header.Get(waf.DispositionHeader),
res.Header.Get(wafprotocol.DispositionHeader),
status,
disposition,
)
@@ -287,12 +346,18 @@ func TestWAFKeepsItsBansAcrossARestartAndNeedsNoApp(t *testing.T) {
}
runWAF(t, env, func(addr string) {
ask(t, addr, "/", http.StatusOK, waf.DispositionAllow)
ask(t, addr, "/", http.StatusOK, wafprotocol.DispositionAllow)
// The default rule file bans a probe for this.
ask(t, addr, "/.env", http.StatusForbidden, waf.DispositionDeny)
ask(
t,
addr,
"/.env",
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
})
runWAF(t, env, func(addr string) {
ask(t, addr, "/", http.StatusForbidden, waf.DispositionDeny)
ask(t, addr, "/", http.StatusForbidden, wafprotocol.DispositionDeny)
})
}
+20 -9
View File
@@ -22,31 +22,42 @@ import (
// an offence, and the request does not wait for the answer. The score is then
// used for each address of the client. Its figures are the score and
// SWW_WAF_ABUSEIPDB_MIN_SCORE.
type abuseIPDB struct{}
type abuseIPDB struct {
abuseIPDB *reputation.AbuseIPDB
// limiter is the table of clients the rate limits count, whose histories
// say which clients are offenders.
limiter *ratelimit.Limiter
keySet bool // SWW_WAF_ABUSEIPDB_KEY is set
minScore int64 // SWW_WAF_ABUSEIPDB_MIN_SCORE
action config.ReputationAction // SWW_WAF_DNSBL_ABUSEIPDB_ACTION
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
hitNoter reputationHitNoter
}
func (abuseIPDB) name() string { return "abuseipdb" }
func (abuseIPDB) check(
func (a abuseIPDB) check(
ctx context.Context,
rq *request,
_ time.Time,
) *verdict {
if rq.h.config.AbuseIPDBKey == "" {
if !a.keySet {
return nil
}
client := clientGroup(rq.h.config, rq.client)
held, _ := rq.h.limiter.Client(client)
client := clientGroup(a.ipv6GroupPrefix, rq.client)
held, _ := a.limiter.Client(client)
offender := held.History.Offences != ratelimit.Offences{}
score, hit := rq.h.abuseIPDB.Hit(ctx, client, rq.client, offender)
score, hit := a.abuseIPDB.Hit(ctx, client, rq.client, offender)
if !hit {
return nil
}
rq.abuseIPDBHit = true
rq.addTag(config.AbuseIPDBTag)
rq.noteHit(
a.hitNoter.noteHit(
rq,
bans.ReputationHit{
Source: reputation.AbuseIPDBSource,
Score: &score,
@@ -54,14 +65,14 @@ func (abuseIPDB) check(
"scored by AbuseIPDB at or over SWW_WAF_ABUSEIPDB_MIN_SCORE",
)
if rq.h.config.DNSBLAbuseIPDBAction != config.ReputationActionDeny {
if a.action != config.ReputationActionDeny {
return nil
}
return &verdict{
action: requestlog.ActionClientDenied,
figures: map[string]any{
figureScore: score, "min_score": rq.h.config.AbuseIPDBMinScore,
figureScore: score, "min_score": a.minScore,
},
}
}
+3 -1
View File
@@ -26,7 +26,9 @@ func BanAbuseIPDBBlacklist(
scores := make(map[netip.Prefix]int64, len(listed))
for _, client := range listed {
netblock := banNetblock(cfg, client.Addr)
netblock := banNetblock(
cfg.BanScopeV4Prefix, cfg.IPv6GroupPrefix, client.Addr,
)
scores[netblock] = max(scores[netblock], client.Score)
}
+8 -2
View File
@@ -224,7 +224,11 @@ func (h *handler) parseBanNetblock(value string) (netip.Prefix, error) {
return netip.Prefix{}, fmt.Errorf("netblock %q %w", value, errZone)
}
return banNetblock(h.config, addr), nil
return banNetblock(
h.config.BanScopeV4Prefix,
h.config.IPv6GroupPrefix,
addr,
), nil
}
// expiry returns when a ban made at now for duration ends: duration
@@ -295,7 +299,9 @@ func (rq *request) showClient() {
Bans: state.BanEntries(rq.h.ledger.Covering(addr)),
}
client, seen := rq.h.limiter.Client(clientGroup(rq.h.config, addr))
client, seen := rq.h.limiter.Client(
clientGroup(rq.h.config.IPv6GroupPrefix, addr),
)
if seen {
answer.Client = &client
}
+61 -23
View File
@@ -26,6 +26,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
const (
@@ -84,7 +85,7 @@ func TestSixteenThousandCookiesAreRefusedBeforeTheCoreRuleSetReadsThem(
got := do(t, req)
wantDisposition(t, got, tooLarge, waf.DispositionDeny)
wantDisposition(t, got, tooLarge, wafprotocol.DispositionDeny)
line := s.out.requestLine(t)
wantLine(t, line, tooLarge, requestlog.ActionRequestHeaderTooLarge)
@@ -164,9 +165,14 @@ func TestARequestLineOverTheLimitIsRefusedWith414BeforeAnyOtherCheck(
line := s.out.requestLine(t)
if tc.status == http.StatusOK {
wantDisposition(t, got, tc.status, waf.DispositionAllow)
wantDisposition(
t,
got,
tc.status,
wafprotocol.DispositionAllow,
)
} else {
wantDisposition(t, got, tc.status, waf.DispositionDeny)
wantDisposition(t, got, tc.status, wafprotocol.DispositionDeny)
}
wantLine(t, line, tc.status, tc.action)
@@ -219,9 +225,14 @@ func TestAHeaderLineOverTheLimitIsRefusedWith431(t *testing.T) {
got := do(t, req)
if tc.refused {
wantDisposition(t, got, tooLarge, waf.DispositionDeny)
wantDisposition(
t,
got,
tooLarge,
wafprotocol.DispositionDeny,
)
} else {
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
}
})
}
@@ -244,10 +255,7 @@ func TestRequestLineRefusalNamesItsHeuristicLengthAndLimit(t *testing.T) {
func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
t.Parallel()
repeated := make([]string, headerFieldLimit)
for i := range repeated {
repeated[i] = "1"
}
repeated := slices.Repeat([]string{"1"}, headerFieldLimit)
for _, tc := range []struct {
name string
@@ -294,7 +302,12 @@ func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
line := s.out.requestLine(t)
if tc.refused {
wantDisposition(t, got, tooLarge, waf.DispositionDeny)
wantDisposition(
t,
got,
tooLarge,
wafprotocol.DispositionDeny,
)
wantLine(
t,
line,
@@ -305,7 +318,12 @@ func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
return
}
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
wantDisposition(
t,
got,
http.StatusOK,
wafprotocol.DispositionAllow,
)
wantLine(t, line, http.StatusOK, requestlog.ActionForward)
if line.DurationCoreRuleSetMS == nil {
@@ -332,7 +350,12 @@ func TestHeaderLimitsHoldWhateverTheCoreRuleSetSettingsSay(t *testing.T) {
req := s.question(siteA, client, "/")
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(t, do(t, req), tooLarge, waf.DispositionDeny)
wantDisposition(
t,
do(t, req),
tooLarge,
wafprotocol.DispositionDeny,
)
})
}
}
@@ -348,7 +371,12 @@ func TestClientInTheAllowNetsSkipsTheHeaderLimits(t *testing.T) {
req := s.question(siteA, client, "/")
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow)
wantDisposition(
t,
do(t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
)
}
func TestObserveModeAllowsARequestOverTheHeaderLimitsUnread(t *testing.T) {
@@ -364,7 +392,7 @@ func TestObserveModeAllowsARequestOverTheHeaderLimitsUnread(t *testing.T) {
got := do(t, req)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
line := s.out.requestLine(t)
if line.WouldAction != requestlog.ActionRequestHeaderTooLarge ||
@@ -386,14 +414,14 @@ func TestTheRateLimitsDoNotCountRequestsOverTheHeaderLimits(t *testing.T) {
for range 2 {
req := s.question(siteA, client, "/")
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(t, do(t, req), tooLarge, waf.DispositionDeny)
wantDisposition(t, do(t, req), tooLarge, wafprotocol.DispositionDeny)
}
wantDisposition(
t,
s.ask(client, "/"),
http.StatusOK,
waf.DispositionAllow,
wafprotocol.DispositionAllow,
)
}
@@ -466,7 +494,12 @@ func TestTheHeaderLimitsComeBeforeEveryCheckThatRefuses(t *testing.T) {
req := s.question(siteA, client, tc.path)
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(t, do(t, req), tooLarge, waf.DispositionDeny)
wantDisposition(
t,
do(t, req),
tooLarge,
wafprotocol.DispositionDeny,
)
wantLine(
t,
s.out.requestLine(t),
@@ -503,11 +536,16 @@ func TestARequestOverTheHeaderLimitsRunsNoOtherCheck(t *testing.T) {
over := s.question(siteA, client, "/")
over.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(t, do(t, over), tooLarge, waf.DispositionDeny)
wantDisposition(t, do(t, over), tooLarge, wafprotocol.DispositionDeny)
within := s.question(siteA, client, "/")
within.Header.Set(cookieHeader, cookies(cookieLimit))
wantDisposition(t, do(t, within), http.StatusOK, waf.DispositionAllow)
wantDisposition(
t,
do(t, within),
http.StatusOK,
wafprotocol.DispositionAllow,
)
lines := s.out.requestLines(t, 2)
@@ -564,7 +602,7 @@ func TestNotWaitingForGeoJSNeverLowersAClientsLimit(t *testing.T) {
t,
s.ask(fromDE, "/"),
http.StatusForbidden,
waf.DispositionDeny,
wafprotocol.DispositionDeny,
)
line := s.out.requestLine(t)
@@ -601,7 +639,7 @@ func TestAClientGeoJSPlacedInNoCountryCountsAsUnknownWithoutWaiting(
t,
s.ask(client, "/"),
http.StatusForbidden,
waf.DispositionDeny,
wafprotocol.DispositionDeny,
)
line := s.out.requestLine(t)
@@ -630,7 +668,7 @@ func TestObserveModeAsksGeoJSNothingForARequestOverTheHeaderLimits(
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
got := do(t, req)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
if len(asked()) != 0 || got.header.Get("X-Client-Asn") != "" {
t.Errorf(
@@ -672,7 +710,7 @@ func askDirectly(
http.NoBody,
)
req.RemoteAddr = net.JoinHostPort(rproxyPeer, "1234")
req.Header.Set(waf.ClientHeader, client)
req.Header.Set(wafprotocol.ClientHeader, client)
req.Header.Set(requestIDHeader, "the-id")
change(req)
+105 -61
View File
@@ -1,7 +1,7 @@
package waf
// The heuristic active_ban, and the code that makes bans, which several
// heuristics share. active_ban reads SWW_WAF_MODE; the bans are the ban ledger's.
// The heuristic active_ban, and banMaker, which makes the bans of several
// heuristics. active_ban reads SWW_WAF_MODE; the bans are the ban ledger's.
import (
"context"
@@ -11,6 +11,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
@@ -25,18 +26,22 @@ func (rq *request) banResponse(action string) *refusal {
// in, in force at now. It notes for the log line when that ban ends. A request
// that makes the ban permanent, or in observe mode would have, raises the alert
// for it. Its figures are the ban's netblock and when it ends.
type activeBan struct{}
type activeBan struct {
banMaker banMaker
}
func (activeBan) name() string { return "active_ban" }
func (activeBan) check(
func (a activeBan) check(
_ context.Context,
rq *request,
now time.Time,
) *verdict {
check := rq.h.ledger.Check
if rq.h.config.Observe {
check = rq.h.ledger.Find // the ban refuses nothing, and stays as it is
ledger := a.banMaker.ledger
check := ledger.Check
if a.banMaker.observe {
check = ledger.Find // the ban refuses nothing, and stays as it is
}
ban, banned, madePermanent := check(rq.client, now)
@@ -46,7 +51,7 @@ func (activeBan) check(
if madePermanent {
ban.Expires = time.Time{} // the ban made permanent, which Find leaves as it is
rq.alertBan(ban)
a.banMaker.alertBan(rq, ban)
}
if !banned {
@@ -81,11 +86,12 @@ func limitFigures(hit ratelimit.Hit) map[string]any {
// countedBytes returns the request's bytes, once it has ended, as the
// byte limits and the anomaly thresholds count them: the response's body
// bytes, the request's, or both, as SWW_WAF_BYTES_COUNTED says.
func (rq *request) countedBytes() int64 {
// bytes, the request's, or both, as bytesCounted, SWW_WAF_BYTES_COUNTED,
// says.
func (rq *request) countedBytes(bytesCounted config.BytesCounted) int64 {
response, request := rq.out.bytes, rq.requestBytes
switch rq.h.config.BytesCounted {
switch bytesCounted {
case config.BytesCountedResponse:
return response
case config.BytesCountedRequest:
@@ -96,7 +102,29 @@ func (rq *request) countedBytes() int64 {
return response + request
}
// banForLimit bans the client's netblock at now for a broken limit, the
// banMaker makes the bans of the heuristics that ban a client, in the ban
// ledger, and raises the alert for each. In observe mode it makes no ban, and
// raises the alert for the ban it would have made.
type banMaker struct {
ledger *bans.Ledger
// limiter is the table of clients the rate limits count. A ban for a limit
// sets the client's counters back to zero, and a ban's notes give how many
// requests its netblock has sent.
limiter *ratelimit.Limiter
metrics *metrics.Metrics
alerts *alerts.Queue
// observe is true in observe mode (SWW_WAF_MODE).
observe bool
// banResponse is SWW_WAF_BAN_RESPONSE, the status the notes of a ban for a
// clear sign of attack give.
banResponse int
// banScopeV4Prefix is SWW_WAF_BAN_SCOPE_V4_PREFIX and ipv6GroupPrefix
// SWW_WAF_IPV6_GROUP_PREFIX, which set the netblock a ban covers.
banScopeV4Prefix int
ipv6GroupPrefix int
}
// banForLimit bans the netblock of rq's client at now for a broken limit, the
// one hit names, notes the offence for the log line and counts the hit in
// the metrics. status is what the client was sent, or is sent:
// SWW_WAF_BAN_RESPONSE for a request over a rate limit, what the report says for
@@ -105,7 +133,8 @@ func (rq *request) countedBytes() int64 {
// rate limit or a byte limit; the refusal limit is not lowered. The ban sets
// the client's counters back to zero. In observe mode, banClient says what
// happens instead.
func (rq *request) banForLimit(
func (b banMaker) banForLimit(
rq *request,
now time.Time,
hit ratelimit.Hit,
status int,
@@ -120,7 +149,8 @@ func (rq *request) banForLimit(
}
rq.line.Offence = requestlog.OffenceLimit
rq.h.metrics.LimitHit(hit)
b.metrics.LimitHit(hit)
notes := bans.Notes{
Kind: hit.Kind,
@@ -136,81 +166,94 @@ func (rq *request) banForLimit(
notes.LimitPercent, notes.LimitPercentSetting = rq.bytesLimitPercent.logged()
}
rq.banClient(
b.banClient(
rq,
now,
status,
bans.Grounds{Cause: bans.CauseLimit, Notes: notes},
)
}
// banForAttack bans the client's netblock at now for a clear sign of
// banForAttack bans the netblock of rq's client at now for a clear sign of
// attack, which notes name: the ban rule that matched, or the trap path
// asked for. It returns the action for that ban. In observe mode, banClient
// says what happens instead.
func (rq *request) banForAttack(now time.Time, notes bans.Notes) string {
return rq.banClient(
func (b banMaker) banForAttack(
rq *request, now time.Time, notes bans.Notes,
) string {
return b.banClient(
rq,
now,
rq.h.config.BanResponse,
b.banResponse,
bans.Grounds{Cause: bans.CauseAttack, Notes: notes},
)
}
// banClient bans the client's netblock at now on grounds, and returns the
// action for that ban. It adds to the notes what the request tells of the
// banClient bans the netblock of rq's client at now on grounds, and returns
// the action for that ban. It adds to the notes what the request tells of the
// client, with status, what the client was sent. A ban for a broken limit
// sets the client's counters back to zero.
//
// In observe mode it makes no ban and sets nothing back. It raises the
// alert for the ban it would have made, if that alert would be sent.
func (rq *request) banClient(
func (b banMaker) banClient(
rq *request,
now time.Time,
status int,
grounds bans.Grounds,
) string {
netblock := banNetblock(rq.h.config, rq.client)
netblock := banNetblock(b.banScopeV4Prefix, b.ipv6GroupPrefix, rq.client)
if rq.h.config.Observe {
if !rq.wouldAlertBan(netblock, now, grounds.Cause) {
if b.observe {
if !b.wouldAlertBan(netblock, now, grounds.Cause) {
return banAction(
rq.h.ledger.WouldBePermanent(netblock, now, grounds.Cause),
b.ledger.WouldBePermanent(netblock, now, grounds.Cause),
)
}
grounds.Notes = rq.withClientNotes(
grounds.Notes = b.withClientNotes(
rq,
grounds.Notes,
now,
status,
netblock,
)
ban, wouldBan := rq.h.ledger.WouldBan(netblock, now, grounds)
ban, wouldBan := b.ledger.WouldBan(netblock, now, grounds)
if wouldBan {
rq.alertBan(ban)
b.alertBan(rq, ban)
}
return banAction(ban.Permanent())
}
grounds.Notes = rq.withClientNotes(grounds.Notes, now, status, netblock)
grounds.Notes = b.withClientNotes(
rq,
grounds.Notes,
now,
status,
netblock,
)
ban, made := rq.h.ledger.Ban(netblock, now, grounds)
ban, made := b.ledger.Ban(netblock, now, grounds)
if grounds.Cause == bans.CauseLimit {
rq.h.limiter.Reset(clientGroup(rq.h.config, rq.client))
b.limiter.Reset(clientGroup(b.ipv6GroupPrefix, rq.client))
}
rq.line.BanExpires = banExpires(ban)
if made {
rq.alertBan(ban)
b.alertBan(rq, ban)
}
return banAction(ban.Permanent())
}
// withClientNotes is notes with what the request tells of the client
// added: its AS, country and reputation, the request at now with status,
// and how many requests netblock, the client's, has sent.
func (rq *request) withClientNotes(
// withClientNotes is notes with what rq tells of its client added: its AS,
// country and reputation, the request at now with status, and how many
// requests netblock, the client's, has sent.
func (b banMaker) withClientNotes(
rq *request,
notes bans.Notes,
now time.Time,
status int,
@@ -221,7 +264,7 @@ func (rq *request) withClientNotes(
notes.Country = rq.line.Country
notes.Reputation = rq.reputation
notes.Request = rq.noted(now, status)
notes.Requests = rq.netblockRequests(netblock)
notes.Requests = b.netblockRequests(netblock)
return notes
}
@@ -231,23 +274,22 @@ func (rq *request) withClientNotes(
// have made is worked out only then, at most once per
// SWW_WAF_ALERT_COOLDOWN and never with no webhook set: its notes count the
// netblock's requests, which can mean going through every client.
func (rq *request) wouldAlertBan(
func (b banMaker) wouldAlertBan(
netblock netip.Prefix, now time.Time, cause string,
) bool {
event := alerts.EventBan
if rq.h.ledger.WouldBePermanent(netblock, now, cause) {
if b.ledger.WouldBePermanent(netblock, now, cause) {
event = alerts.EventPermanentBan
}
return rq.h.alerts.WouldSend(event, netblock)
return b.alerts.WouldSend(event, netblock)
}
// alertBan raises the alert for ban, which the request made, or made
// permanent: permanent_ban for a permanent ban, ban for another. Its
// detail gives the ban's cause, when it ends, and its notes, and in
// observe mode, where ban is the ban that would have been made, or made
// permanent, mode, observe.
func (rq *request) alertBan(ban bans.Ban) {
// alertBan raises the alert for ban, which rq made, or made permanent:
// permanent_ban for a permanent ban, ban for another. Its detail gives the
// ban's cause, when it ends, and its notes, and in observe mode, where ban is
// the ban that would have been made, or made permanent, mode, observe.
func (b banMaker) alertBan(rq *request, ban bans.Ban) {
event := alerts.EventBan
if ban.Permanent() {
event = alerts.EventPermanentBan
@@ -256,11 +298,11 @@ func (rq *request) alertBan(ban bans.Ban) {
detail := map[string]any{
"cause": ban.Cause, "ban_expires": banExpires(ban), "notes": ban.Notes,
}
if rq.h.config.Observe {
if b.observe {
detail["mode"] = "observe"
}
rq.h.alerts.Raise(alerts.Alert{
b.alerts.Raise(alerts.Alert{
Event: event,
Host: rq.site,
Client: rq.client,
@@ -273,6 +315,13 @@ func (rq *request) alertBan(ban bans.Ban) {
})
}
// netblockRequests is how many requests netblock has sent since it was
// first seen, the request being banned for included: the histories count it
// only once it has ended.
func (b banMaker) netblockRequests(netblock netip.Prefix) int64 {
return b.limiter.Requests(netblock) + 1
}
// noted is the request, at now, with status, what the client was sent, or
// in observe mode would have been, as the notes of the ban it makes keep
// it.
@@ -287,23 +336,18 @@ func (rq *request) noted(now time.Time, status int) bans.Request {
}
}
// netblockRequests is how many requests netblock has sent since it was
// first seen, this one included: the histories count it only once it has
// ended.
func (rq *request) netblockRequests(netblock netip.Prefix) int64 {
return rq.h.limiter.Requests(netblock) + 1
}
// banNetblock is the netblock a ban on client covers: its IPv4 address,
// widened to SWW_WAF_BAN_SCOPE_V4_PREFIX, or the IPv6 group clientGroup
// counts it in.
func banNetblock(cfg *config.WAF, client netip.Addr) netip.Prefix {
// widened to banScopeV4Prefix, SWW_WAF_BAN_SCOPE_V4_PREFIX, or the IPv6 group
// clientGroup counts it in, of length ipv6GroupPrefix.
func banNetblock(
banScopeV4Prefix, ipv6GroupPrefix int, client netip.Addr,
) netip.Prefix {
addr := client.Unmap()
if addr.Is4() {
return netip.PrefixFrom(addr, cfg.BanScopeV4Prefix).Masked()
return netip.PrefixFrom(addr, banScopeV4Prefix).Masked()
}
return clientGroup(cfg, addr)
return clientGroup(ipv6GroupPrefix, addr)
}
// banExpires is when ban ends, as the log line gives it: a time, or
+12 -12
View File
@@ -54,7 +54,7 @@ func (rq *request) runBesideLookup(
) *verdict {
var running atomic.Pointer[string]
running.Store(new(trapPaths{}.name()))
running.Store(new(rq.h.trapPaths.name()))
found := make(chan *besideLookupFindings, 1)
@@ -92,7 +92,7 @@ func (rq *request) runBesideLookup(
return v
}
return rq.runHeuristics(ctx, afterBesideLookup(), rq.h.now())
return rq.runHeuristics(ctx, rq.h.afterBesideLookup, rq.h.now())
}
// findBesideLookup does the work of the heuristics of besideLookup on the
@@ -105,39 +105,39 @@ func (rq *request) findBesideLookup(
) *besideLookupFindings {
findings := &besideLookupFindings{}
findings.trapPath = rq.trapPath()
findings.trapPath = rq.h.trapPaths.trapPath(rq)
if findings.trapPath != "" {
findings.refuses = true
return findings
}
running.Store(new(ruleFiles{}.name()))
running.Store(new(rq.h.ruleFiles.name()))
findings.rules = rq.h.rules.Match(rq.in)
findings.rules = rq.h.ruleFiles.rules.Match(rq.in)
if refusedByRule(findings.rules) {
findings.refuses = true
return findings
}
running.Store(new(coreRuleSet{}.name()))
coreRuleSet := rq.h.coreRuleSet
running.Store(new(coreRuleSet.name()))
cfg := rq.h.config
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff ||
pathExempt(rq.in, cfg.CoreRuleSetExemptPaths) {
if coreRuleSet.mode == config.CoreRuleSetModeOff ||
pathExempt(rq.in, coreRuleSet.exemptPaths) {
return findings
}
start := time.Now()
// A question's body has been read into memory, so that reading it here
// cannot fail.
findings.inspection, findings.read, _ = rq.h.coreRuleSet.Inspect(
findings.inspection, findings.read, _ = coreRuleSet.inspector.Inspect(
rq.in, rq.client, rq.in.Body,
)
findings.inspectionTime = time.Since(start)
findings.inspected = true
findings.refuses = rq.refusedByCoreRuleSet(findings.inspection)
findings.refuses = coreRuleSet.refusedByCoreRuleSet(findings.inspection)
return findings
}
@@ -151,7 +151,7 @@ func (rq *request) applyFindings(
) *verdict {
rq.findings = findings
return rq.runHeuristics(ctx, besideLookup(), rq.h.now())
return rq.runHeuristics(ctx, rq.h.besideLookup, rq.h.now())
}
// timedOut refuses the request for the heuristics of besideLookup not having
+36 -15
View File
@@ -2,6 +2,7 @@ package waf
import (
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/reputation"
)
// whole is the percentage of each limit a client gets when no biased
@@ -29,6 +30,25 @@ func biasedThresholdsSet(cfg *config.WAF) bool {
cfg.ASNLimitPercentURL != ""
}
// biasedThresholds are the settings and the lists by which limitPercentages
// lowers a client's limits.
type biasedThresholds struct {
// lists hold the file SWW_WAF_ASN_LIMIT_PERCENT_URL names.
lists *reputation.Lists
tagLimitPercent map[string]int64 // SWW_WAF_TAG_LIMIT_PERCENT
asnBytesLimitPercent map[string]int64 // SWW_WAF_ASN_BYTES_LIMIT_PERCENT
countryBytesLimitPercent map[string]int64 // SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT
unknownCountryLimitPercent int64 // SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT
// blocklistAction and blocklistLimitPercent are SWW_WAF_BLOCKLIST_ACTION
// and its percentage.
blocklistAction config.ReputationAction
blocklistLimitPercent int64
// dnsblAbuseIPDBAction and dnsblAbuseIPDBLimitPercent are
// SWW_WAF_DNSBL_ABUSEIPDB_ACTION and its percentage.
dnsblAbuseIPDBAction config.ReputationAction
dnsblAbuseIPDBLimitPercent int64
}
// limitPercentages returns the client's limit percentages, for the rate
// limits and for the byte limits, by its tags, its AS number and country
// as looked up, each "" when unknown, and the blocklists, DNSBL zones and
@@ -45,37 +65,38 @@ func biasedThresholdsSet(cfg *config.WAF) bool {
// the byte limits, SWW_WAF_ASN_BYTES_LIMIT_PERCENT and
// SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT take the place of the first three for
// an AS number or a country they list.
func (rq *request) limitPercentages() (percentage, percentage) {
cfg := rq.h.config
func (b biasedThresholds) limitPercentages(
rq *request,
) (percentage, percentage) {
asn, country := rq.line.ASN, rq.line.Country
unknown := percentage{percent: whole}
if country == "" && !rq.skippedLookupWait {
unknown = percentage{
cfg.UnknownCountryLimitPercent,
b.unknownCountryLimitPercent,
config.UnknownCountryLimitPercentSetting,
}
}
fetched := percentage{percent: whole}
if percent, listed := rq.h.lists.ASNLimitPercent(asn); listed {
if percent, listed := b.lists.ASNLimitPercent(asn); listed {
fetched = percentage{percent, config.ASNLimitPercentURLSetting}
}
blocklisted := percentage{percent: whole}
if rq.blocklisted &&
cfg.BlocklistAction == config.ReputationActionLimit {
b.blocklistAction == config.ReputationActionLimit {
blocklisted = percentage{
cfg.BlocklistLimitPercent,
b.blocklistLimitPercent,
config.BlocklistActionSetting,
}
}
dnsblAbuseIPDBListed := percentage{percent: whole}
if (rq.dnsblListed || rq.abuseIPDBHit) &&
cfg.DNSBLAbuseIPDBAction == config.ReputationActionLimit {
b.dnsblAbuseIPDBAction == config.ReputationActionLimit {
dnsblAbuseIPDBListed = percentage{
cfg.DNSBLAbuseIPDBLimitPercent,
b.dnsblAbuseIPDBLimitPercent,
config.DNSBLAbuseIPDBActionSetting,
}
}
@@ -83,11 +104,11 @@ func (rq *request) limitPercentages() (percentage, percentage) {
const tagSetting = config.TagLimitPercentSetting
asnRequests := lowest(
given(cfg.TagLimitPercent, asnTag(asn), tagSetting),
given(b.tagLimitPercent, asnTag(asn), tagSetting),
fetched,
)
countryRequests := given(
cfg.TagLimitPercent,
b.tagLimitPercent,
countryTag(country),
tagSetting,
)
@@ -98,23 +119,23 @@ func (rq *request) limitPercentages() (percentage, percentage) {
if tag != asnTag(asn) && tag != countryTag(country) {
otherTags = lowest(
otherTags,
given(cfg.TagLimitPercent, tag, tagSetting),
given(b.tagLimitPercent, tag, tagSetting),
)
}
}
asnBytes, countryBytes := asnRequests, countryRequests
if _, listed := cfg.ASNBytesLimitPercent[asn]; listed {
if _, listed := b.asnBytesLimitPercent[asn]; listed {
asnBytes = given(
cfg.ASNBytesLimitPercent,
b.asnBytesLimitPercent,
asn,
config.ASNBytesLimitPercentSetting,
)
}
if _, listed := cfg.CountryBytesLimitPercent[country]; listed {
if _, listed := b.countryBytesLimitPercent[country]; listed {
countryBytes = given(
cfg.CountryBytesLimitPercent,
b.countryBytesLimitPercent,
country,
config.CountryBytesLimitPercentSetting,
)
+2 -1
View File
@@ -19,6 +19,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// The biased thresholds.
@@ -589,7 +590,7 @@ func serveFromDE(
t.Helper()
req := httptest.NewRequestWithContext(t.Context(), method, "/", body)
req.Header.Set(waf.ClientHeader, fromDE)
req.Header.Set(wafprotocol.ClientHeader, fromDE)
req.Header.Set(requestIDHeader, "the-id")
server.Handler.ServeHTTP(httptest.NewRecorder(), req)
+21 -8
View File
@@ -8,6 +8,7 @@ import (
"time"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
@@ -15,20 +16,24 @@ import (
// and refuses the request when SWW_WAF_BLOCKLIST_ACTION is deny. Being limit, it
// lowers the client's limits instead (see limitPercentages), and being log, it
// does nothing more. Its figure is the blocklists that list the client.
type blocklists struct{}
type blocklists struct {
lists *reputation.Lists
action config.ReputationAction // SWW_WAF_BLOCKLIST_ACTION
hitNoter reputationHitNoter
}
func (blocklists) name() string { return "blocklists" }
func (blocklists) check(
func (b blocklists) check(
_ context.Context,
rq *request,
_ time.Time,
) *verdict {
listedBy := rq.h.lists.ListedBy(rq.client)
listedBy := b.lists.ListedBy(rq.client)
rq.blocklisted = len(listedBy) > 0
rq.noteListed(listedBy, "listed by a blocklist")
b.hitNoter.noteListed(rq, listedBy, "listed by a blocklist")
if !rq.refusedByBlocklists(listedBy) {
if !b.refusedByBlocklists(listedBy) {
return nil
}
@@ -38,10 +43,18 @@ func (blocklists) check(
}
}
// refusesWhateverTheLookupAnswers reports whether blocklists refuses rq, as
// refusedByBlocklists says. The lookup's answer plays no part in it.
func (b blocklists) refusesWhateverTheLookupAnswers(
rq *request,
_ time.Time,
) bool {
return b.refusedByBlocklists(b.lists.ListedBy(rq.client))
}
// refusedByBlocklists reports whether the heuristic blocklists refuses a
// client, given listedBy, the blocklists that list it: it does when one lists
// it and SWW_WAF_BLOCKLIST_ACTION is deny.
func (rq *request) refusedByBlocklists(listedBy []string) bool {
return len(listedBy) > 0 && rq.h.config.BlocklistAction ==
config.ReputationActionDeny
func (b blocklists) refusedByBlocklists(listedBy []string) bool {
return len(listedBy) > 0 && b.action == config.ReputationActionDeny
}
+14 -6
View File
@@ -6,6 +6,9 @@ package waf
import (
"context"
"time"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
)
// byteLimits counts the request's bytes, as countedBytes gives them, for the
@@ -16,11 +19,16 @@ import (
// the client over a byte limit, as its limit percentage for the byte limits
// lowers it, break it, which bans the client; the response was passed on whole.
// Its figures are the window, the count and the limit.
type byteLimits struct{}
type byteLimits struct {
limiter *ratelimit.Limiter
bytesCounted config.BytesCounted // SWW_WAF_BYTES_COUNTED
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
banMaker banMaker
}
func (byteLimits) name() string { return "byte_limits" }
func (byteLimits) check(
func (b byteLimits) check(
_ context.Context,
rq *request,
now time.Time,
@@ -29,10 +37,10 @@ func (byteLimits) check(
return nil
}
counts, hit, over := rq.h.limiter.CountBytes(
clientGroup(rq.h.config, rq.client),
counts, hit, over := b.limiter.CountBytes(
clientGroup(b.ipv6GroupPrefix, rq.client),
now,
rq.countedBytes(),
rq.countedBytes(b.bytesCounted),
rq.bytesLimitPercent.percent,
)
rq.line.Counts.MinuteBytes = counts.MinuteBytes
@@ -43,7 +51,7 @@ func (byteLimits) check(
return nil
}
rq.banForLimit(now, hit, rq.out.status)
b.banMaker.banForLimit(rq, now, hit, rq.out.status)
return &verdict{figures: limitFigures(hit)}
}
+7 -9
View File
@@ -6,7 +6,7 @@ import (
"net/netip"
"slices"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// peerAddress is the address of the request's TCP peer.
@@ -19,14 +19,11 @@ func peerAddress(r *http.Request) netip.Addr {
return addrPort.Addr().Unmap()
}
// requestIDHeader carries the request's id, from rproxy.
const requestIDHeader = "X-Request-ID"
// requestID is the request's id: the one a question gives, or for any other
// request a new random one.
func requestID(r *http.Request, isQuestion bool) string {
if isQuestion {
return r.Header.Get(requestIDHeader)
return r.Header.Get(wafprotocol.RequestIDHeader)
}
return rand.Text()
@@ -45,12 +42,13 @@ func scheme(r *http.Request, isQuestion bool) string {
// clientGroup is the client a request is counted toward: its IPv4
// address, or its IPv6 group, the netblock its IPv6 address is in of the
// length SWW_WAF_IPV6_GROUP_PREFIX sets, a /64 by default, since one abuser
// usually holds a whole /64. An IPv4 address in IPv6 form counts as IPv4.
func clientGroup(cfg *config.WAF, addr netip.Addr) netip.Prefix {
// length ipv6GroupPrefix, SWW_WAF_IPV6_GROUP_PREFIX, a /64 by default, since
// one abuser usually holds a whole /64. An IPv4 address in IPv6 form counts as
// IPv4.
func clientGroup(ipv6GroupPrefix int, addr netip.Addr) netip.Prefix {
addr = addr.Unmap()
if addr.Is6() {
return netip.PrefixFrom(addr, cfg.IPv6GroupPrefix).Masked()
return netip.PrefixFrom(addr, ipv6GroupPrefix).Masked()
}
return netip.PrefixFrom(addr, addr.BitLen())
+30 -18
View File
@@ -10,6 +10,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
@@ -22,11 +23,22 @@ import (
// exploitsignature, raises the core_rule_set_match alert, and in block mode
// refuses the request with 403, which is an offence its client's history
// counts. Its figures are the score and the threshold.
type coreRuleSet struct{}
type coreRuleSet struct {
// inspector is the Core Rule Set, nil while SWW_WAF_CORE_RULE_SET_MODE is
// off.
inspector CoreRuleSetInspector
metrics *metrics.Metrics
alerts *alerts.Queue
mode config.CoreRuleSetMode // SWW_WAF_CORE_RULE_SET_MODE
exemptPaths []string // SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS
scoreThreshold int // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD
observe bool // SWW_WAF_MODE is observe
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
}
func (coreRuleSet) name() string { return "core_rule_set" }
func (coreRuleSet) check(
func (c coreRuleSet) check(
_ context.Context,
rq *request,
_ time.Time,
@@ -36,7 +48,6 @@ func (coreRuleSet) check(
return nil
}
cfg := rq.h.config
result := findings.inspection
rq.requestBytes = int64(len(findings.read))
@@ -47,18 +58,18 @@ func (coreRuleSet) check(
rq.line.CoreRuleSetScore = &result.Score
for _, id := range result.RuleIDs {
rq.h.metrics.CoreRuleSetMatched(string(cfg.CoreRuleSetMode), id)
c.metrics.CoreRuleSetMatched(string(c.mode), id)
}
threshold := cfg.CoreRuleSetScoreThreshold
threshold := c.scoreThreshold
if threshold == 0 || result.Score < threshold {
return nil
}
rq.addTag(config.ExploitSignatureTag)
rq.alertCoreRuleSetMatch(result)
c.alertCoreRuleSetMatch(rq, result)
if !rq.refusedByCoreRuleSet(result) {
if !c.refusedByCoreRuleSet(result) {
return nil
}
@@ -76,20 +87,21 @@ func (coreRuleSet) check(
// refusedByCoreRuleSet reports whether coreRuleSet refuses a request the Core
// Rule Set found result of: whether the score is at or over
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD in block mode.
func (rq *request) refusedByCoreRuleSet(result coreruleset.Result) bool {
cfg := rq.h.config
threshold := cfg.CoreRuleSetScoreThreshold
func (c coreRuleSet) refusedByCoreRuleSet(result coreruleset.Result) bool {
threshold := c.scoreThreshold
return cfg.CoreRuleSetMode == config.CoreRuleSetModeBlock &&
return c.mode == config.CoreRuleSetModeBlock &&
threshold != 0 && result.Score >= threshold
}
// alertCoreRuleSetMatch raises the core_rule_set_match alert for the request,
// which the Core Rule Set scored at result, at or over
// alertCoreRuleSetMatch raises the core_rule_set_match alert for rq, which the
// Core Rule Set scored at result, at or over
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD. Its detail gives the rule ids, the
// score, the method and the path with the query, and, for a request that is
// not refused for it, the mode: detect, or observe in observe mode.
func (rq *request) alertCoreRuleSetMatch(result coreruleset.Result) {
func (c coreRuleSet) alertCoreRuleSetMatch(
rq *request, result coreruleset.Result,
) {
detail := map[string]any{
"rule_ids": result.RuleIDs,
"score": result.Score,
@@ -98,17 +110,17 @@ func (rq *request) alertCoreRuleSetMatch(result coreruleset.Result) {
}
switch {
case rq.h.config.CoreRuleSetMode == config.CoreRuleSetModeDetect:
case c.mode == config.CoreRuleSetModeDetect:
detail["mode"] = string(config.CoreRuleSetModeDetect)
case rq.h.config.Observe:
case c.observe:
detail["mode"] = "observe"
}
rq.h.alerts.Raise(alerts.Alert{
c.alerts.Raise(alerts.Alert{
Event: alerts.EventCoreRuleSetMatch,
Host: rq.site,
Client: rq.client,
Netblock: clientGroup(rq.h.config, rq.client),
Netblock: clientGroup(c.ipv6GroupPrefix, rq.client),
ASN: rq.line.ASN,
ASName: rq.line.ASName,
Country: rq.line.Country,
+3 -3
View File
@@ -12,7 +12,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// The Core Rule Set's settings the tests set, besides SWW_WAF_CORE_RULE_SET_MODE, and
@@ -395,9 +395,9 @@ func TestGiteasFixesApplyOnlyToAHostnameThatServesGitea(t *testing.T) {
})
wantDisposition(t, do(t, s.question(siteA, client, fileName)),
http.StatusOK, waf.DispositionAllow)
http.StatusOK, wafprotocol.DispositionAllow)
wantDisposition(t, do(t, s.question(siteB, client, fileName)),
http.StatusForbidden, waf.DispositionDeny)
http.StatusForbidden, wafprotocol.DispositionDeny)
lines := s.out.requestLines(t, 2)
wantLine(t, lines[0], http.StatusOK, requestlog.ActionForward)
+5 -3
View File
@@ -15,16 +15,18 @@ import (
// does not hold the client's country as it was looked up. A client without a
// country, or whose country cannot be found, is refused. Its figures are the
// country and the setting that refused it.
type countries struct{}
type countries struct {
allowed []string // SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES
}
func (countries) name() string { return "countries" }
func (countries) check(
func (c countries) check(
_ context.Context,
rq *request,
_ time.Time,
) *verdict {
allowed := rq.h.config.ExclusivelyAllowedCountries
allowed := c.allowed
country := rq.line.Country
if len(allowed) == 0 || slices.Contains(allowed, country) {
+10 -3
View File
@@ -4,6 +4,7 @@ package waf
import (
"context"
"net/netip"
"time"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
@@ -11,12 +12,18 @@ import (
// denyNets refuses a client in SWW_WAF_DENY_NETS. Its figure is the netblock of
// the setting the client is in.
type denyNets struct{}
type denyNets struct {
netblocks []netip.Prefix // SWW_WAF_DENY_NETS
}
func (denyNets) name() string { return "deny_nets" }
func (denyNets) check(_ context.Context, rq *request, _ time.Time) *verdict {
for _, netblock := range rq.h.config.DenyNets {
func (d denyNets) check(
_ context.Context,
rq *request,
_ time.Time,
) *verdict {
for _, netblock := range d.netblocks {
if netblock.Contains(rq.client) {
return &verdict{
action: requestlog.ActionClientDenied,
+47
View File
@@ -0,0 +1,47 @@
package waf
import (
"net/netip"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
func TestDenyNetsRefusesOnlyAClientInsideItsNetblocks(t *testing.T) {
t.Parallel()
heuristic := denyNets{
netblocks: []netip.Prefix{netip.MustParsePrefix("198.51.100.0/24")},
}
// The request has no handler: the heuristic reads only what it holds.
inside := &request{
decision: decision{client: netip.MustParseAddr("198.51.100.7")},
}
v := heuristic.check(t.Context(), inside, time.Time{})
if v == nil {
t.Fatal("a client inside the netblock was let through")
}
if v.action != requestlog.ActionClientDenied {
t.Errorf(
"action = %q, want %q",
v.action,
requestlog.ActionClientDenied,
)
}
if got := v.figures[figureNetblock]; got != "198.51.100.0/24" {
t.Errorf("netblock figure = %v, want 198.51.100.0/24", got)
}
outside := &request{
decision: decision{client: netip.MustParseAddr("203.0.113.7")},
}
if v := heuristic.check(t.Context(), outside, time.Time{}); v != nil {
t.Errorf("a client outside the netblock was refused: %+v", v)
}
}
+14 -6
View File
@@ -9,6 +9,7 @@ import (
"time"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
@@ -19,21 +20,28 @@ import (
// being log, it does nothing more. A zone without a verdict on the client is
// asked about it in the background, and the request does not wait for the
// answer. Its figure is the zones that list the client, their keys masked.
type dnsbl struct{}
type dnsbl struct {
dnsbl *reputation.DNSBL
action config.ReputationAction // SWW_WAF_DNSBL_ABUSEIPDB_ACTION
hitNoter reputationHitNoter
}
func (dnsbl) name() string { return "dnsbl" }
func (dnsbl) check(ctx context.Context, rq *request, _ time.Time) *verdict {
listedBy := rq.h.dnsbl.ListedBy(ctx, rq.client)
func (d dnsbl) check(
ctx context.Context,
rq *request,
_ time.Time,
) *verdict {
listedBy := d.dnsbl.ListedBy(ctx, rq.client)
rq.dnsblListed = len(listedBy) > 0
rq.noteListed(listedBy, "listed by a DNSBL zone")
d.hitNoter.noteListed(rq, listedBy, "listed by a DNSBL zone")
for _, zone := range listedBy {
rq.addTag(config.DNSBLTagPrefix + strings.ToLower(zone))
}
if !rq.dnsblListed ||
rq.h.config.DNSBLAbuseIPDBAction != config.ReputationActionDeny {
if !rq.dnsblListed || d.action != config.ReputationActionDeny {
return nil
}
+8 -3
View File
@@ -5,7 +5,7 @@ import (
"testing"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
func TestHealthEndpointCountsTowardNoLimit(t *testing.T) {
@@ -25,8 +25,13 @@ func TestHealthEndpointCountsTowardNoLimit(t *testing.T) {
)
for range healthChecks {
got := s.ownEndpointRequest("", http.MethodGet, waf.HealthPath, "",
http.StatusOK)
got := s.ownEndpointRequest(
"",
http.MethodGet,
wafprotocol.HealthPath,
"",
http.StatusOK,
)
if string(got.body) != "ok\n" ||
got.header.Get("Content-Type") != contentType {
+23 -71
View File
@@ -5,9 +5,20 @@ import (
"time"
)
// To add a heuristic, write one file with a type that implements heuristic,
// and add one line for it to one of the lists below. Its settings go in
// config.WAF, and the comment at the top of its file names them.
// To add a heuristic, write one file with a struct that implements heuristic
// and holds the settings and parts it reads. Add one line for it to one of the
// lists that newHandler builds, which hands it those settings and parts. Its
// settings go in config.WAF, and the comment at the top of its file names them.
// A heuristic of afterLookup that can refuse a request whatever the lookup
// answers also implements refusesWithoutLookup.
//
// Three places outside its file may change with it:
// - waitForGeoJS in newHandler lists the settings that need the lookup's
// answer;
// - requestCanGetTag in internal/config/waf.go holds which setting can give a
// request which tag;
// - findBesideLookup does the work of each heuristic of besideLookup, by
// name.
// heuristic is one of the checks that decide what waf does with a
// request.
@@ -19,6 +30,15 @@ type heuristic interface {
check(ctx context.Context, rq *request, now time.Time) *verdict
}
// refusesWithoutLookup is a heuristic of afterLookup that can tell, before the
// lookup answers, that it will refuse a request whatever the answer.
type refusesWithoutLookup interface {
// refusesWhateverTheLookupAnswers reports whether the heuristic will refuse
// rq at now, whatever the lookup answers about its client. It changes
// nothing.
refusesWhateverTheLookupAnswers(rq *request, now time.Time) bool
}
// verdict is a heuristic refusing a request, or banning its client: the action
// the log line names, one of requestlog's Action constants, and the figures the
// heuristic acted on, by name, such as the count and the limit.
@@ -36,74 +56,6 @@ const (
figureNetblock = "netblock"
)
// The lists of heuristics, each in the order its heuristics run. A client in
// SWW_WAF_ALLOW_NETS skips them all.
// beforeLookup are the heuristics that run before the client's AS number and
// country are looked up, so that a client they refuse is not looked up.
// header_limits comes first: it only measures and counts, and a request it
// refuses causes no other work.
func beforeLookup() []heuristic {
return []heuristic{
headerLimits{},
tagLists{},
routeWatchTags{},
denyNets{},
activeBan{},
}
}
// afterLookup are the heuristics that run once the lookup has been made.
func afterLookup() []heuristic {
return []heuristic{
countries{},
blocklists{},
dnsbl{},
abuseIPDB{},
tagDeny{},
rateLimits{},
}
}
// besideLookup are the heuristics that only read the request. Their work runs
// beside the lookup and afterLookup, in findBesideLookup, which changes nothing
// of the request. Their check then applies what that work found, and runs only
// once waf takes it.
func besideLookup() []heuristic {
return []heuristic{
trapPaths{},
ruleFiles{},
coreRuleSet{},
}
}
// afterBesideLookup are the heuristics that run once afterLookup and
// besideLookup have both let the request through. tag_deny runs a second time,
// on the tag rule_files and core_rule_set can add.
func afterBesideLookup() []heuristic {
return []heuristic{
tagDeny{},
}
}
// afterAnswer are the heuristics that run on a request once waf has
// answered it, before its decision line is written. They ban the client, and do
// not change what the request got: their verdict's action is not used.
func afterAnswer() []heuristic {
return []heuristic{
refusalLimit{},
}
}
// onReport are the heuristics that run on an allowed request when rproxy's
// report of it comes, before its report line is written. Like afterAnswer, they
// only ban.
func onReport() []heuristic {
return []heuristic{
byteLimits{},
}
}
// runHeuristics runs list, one of those checkClient runs, on the request at now,
// and returns the first verdict, or nil when none gives one. The log line's
// heuristic and heuristic_figures name the heuristic that gave it, and give its
+2 -1
View File
@@ -11,6 +11,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
func TestHistoryKeepsEachRequestOfTheClient(t *testing.T) {
@@ -118,7 +119,7 @@ func TestOwnEndpointsAreNotInTheHistory(t *testing.T) {
s.ownEndpointRequest(
"",
http.MethodGet,
waf.HealthPath,
wafprotocol.HealthPath,
"",
http.StatusOK,
)
+5 -25
View File
@@ -3,7 +3,6 @@ package waf
import (
"context"
"log/slog"
"net/http"
"net/netip"
"strings"
"time"
@@ -16,15 +15,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
)
// The headers in which an answer of allow gives rproxy the client's AS number
// and country, for the app, while SWW_WAF_ADD_LOOKUP_HEADERS is true. Go writes
// every header name in this form, as it sends it and as it receives it, so
// X-Client-ASN arrives as X-Client-Asn; header names are not case-sensitive.
const (
asnHeader = "X-Client-Asn"
countryHeader = "X-Client-Country"
)
// lookupResult is what the lookup gave for a request: whether its client was
// looked up, the answer, and whether the request skipped the wait for GeoJS.
type lookupResult struct {
@@ -56,7 +46,7 @@ func (rq *request) startLookUp(
return result
}
client := clientGroup(rq.h.config, rq.client)
client := clientGroup(rq.h.config.IPv6GroupPrefix, rq.client)
if rq.h.config.LookupSource == config.LookupSourceFile {
result <- lookupResult{
@@ -143,8 +133,10 @@ func AddLookup(
) {
limiter.AddLookup(answer.Client, answer.Answered,
answer.ASN, answer.ASName, answer.Country)
ledger.AddLookup(banNetblock(cfg, answer.Client.Addr()),
answer.ASN, answer.ASName, answer.Country)
netblock := banNetblock(
cfg.BanScopeV4Prefix, cfg.IPv6GroupPrefix, answer.Client.Addr(),
)
ledger.AddLookup(netblock, answer.ASN, answer.ASName, answer.Country)
}
// GeoJSFailed logs a failure of GeoJS to processLog, raises a
@@ -167,18 +159,6 @@ func GeoJSFailed(
m.GeoJSFailures.Inc()
}
// setLookupHeaders sets the headers that give the client's AS number and
// country, leaving out one that is unknown.
func setLookupHeaders(header http.Header, asn, country string) {
if asn != "" {
header.Set(asnHeader, asn)
}
if country != "" {
header.Set(countryHeader, country)
}
}
// canBePlaced reports whether a lookup can place addr: private, loopback
// and link-local addresses have no AS number or country.
func canBePlaced(addr netip.Addr) bool {
+2 -1
View File
@@ -16,6 +16,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
const (
@@ -66,7 +67,7 @@ func TestMetricsNeedTheTokenAndOtherPathsAreNotFound(t *testing.T) {
{http.MethodPost, waf.MetricsPath, bearer, http.StatusNotFound},
{http.MethodGet, waf.MetricsPath + "/", bearer, http.StatusNotFound},
{http.MethodGet, "/_sww/bans", bearer, http.StatusNotFound},
{http.MethodPost, waf.HealthPath, "", http.StatusNotFound},
{http.MethodPost, wafprotocol.HealthPath, "", http.StatusNotFound},
} {
got := s.ownEndpointRequest(
tc.authorization,
+162 -39
View File
@@ -28,14 +28,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/rules"
)
// ownEndpointPrefix starts the path of each of waf's own endpoints. The app
// never sees a path under it.
const ownEndpointPrefix = "/_sww/"
// HealthPath is the health endpoint of waf and of rproxy, which the
// container's health check asks.
const HealthPath = "/_sww/healthz"
// MetricsPath is where the metrics are, for a request that carries
// SWW_WAF_METRICS_TOKEN.
const MetricsPath = "/_sww/metrics"
@@ -133,15 +125,19 @@ type Server struct {
}
// newHandler returns the handler with what every request shares, the parts
// in params among it.
// in params among it. It builds the heuristics, each with the settings and
// parts it reads, and the lists they run in.
//
//nolint:funlen // a few lines for each heuristic, a list that grows with them
func newHandler(params Params) *handler {
awaitingReport, err := lru.New[string, *decision](maxAwaitingReport)
if err != nil {
panic(err) // New fails only for a size below one
}
return &handler{
config: params.Config,
cfg := params.Config
h := &handler{
config: cfg,
requestLog: params.RequestLog,
processLog: params.ProcessLog,
now: params.Now,
@@ -152,47 +148,140 @@ func newHandler(params Params) *handler {
geojs: params.GeoJS,
anomalies: params.Anomalies,
lookupFile: params.LookupFile,
lists: params.Lists,
dnsbl: params.DNSBL,
abuseIPDB: params.AbuseIPDB,
routeWatch: params.RouteWatch,
rules: params.Rules,
coreRuleSet: params.CoreRuleSet,
alerts: params.Alerts,
// The country list, the headers, the tag settings that name an asn=
// or country= tag and the biased thresholds act on the answer before
// the request goes on.
waitForGeoJS: len(params.Config.ExclusivelyAllowedCountries) > 0 ||
params.Config.AddLookupHeaders ||
params.Config.TagsNeedLookup() ||
biasedThresholdsSet(params.Config),
waitForGeoJS: len(cfg.ExclusivelyAllowedCountries) > 0 ||
cfg.AddLookupHeaders ||
cfg.TagsNeedLookup() ||
biasedThresholdsSet(cfg),
besideLookupLimit: cmp.Or(
params.BesideLookupTimeLimit,
besideLookupTimeLimit,
),
}
sharedBanMaker := banMaker{
ledger: params.Ledger,
limiter: params.Limiter,
metrics: params.Metrics,
alerts: params.Alerts,
observe: cfg.Observe,
banResponse: cfg.BanResponse,
banScopeV4Prefix: cfg.BanScopeV4Prefix,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
}
hitNoter := reputationHitNoter{
metrics: params.Metrics,
alerts: params.Alerts,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
}
h.trapPaths = trapPaths{paths: cfg.TrapPaths, banMaker: sharedBanMaker}
h.ruleFiles = ruleFiles{
rules: params.Rules,
metrics: params.Metrics,
banMaker: sharedBanMaker,
}
h.coreRuleSet = coreRuleSet{
inspector: params.CoreRuleSet,
metrics: params.Metrics,
alerts: params.Alerts,
mode: cfg.CoreRuleSetMode,
exemptPaths: cfg.CoreRuleSetExemptPaths,
scoreThreshold: cfg.CoreRuleSetScoreThreshold,
observe: cfg.Observe,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
}
h.beforeLookup = []heuristic{
headerLimits{},
tagLists{lists: params.Lists},
routeWatchTags{
routeWatch: params.RouteWatch,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
},
denyNets{netblocks: cfg.DenyNets},
activeBan{banMaker: sharedBanMaker},
}
h.afterLookup = []heuristic{
countries{allowed: cfg.ExclusivelyAllowedCountries},
blocklists{
lists: params.Lists,
action: cfg.BlocklistAction,
hitNoter: hitNoter,
},
dnsbl{
dnsbl: params.DNSBL,
action: cfg.DNSBLAbuseIPDBAction,
hitNoter: hitNoter,
},
abuseIPDB{
abuseIPDB: params.AbuseIPDB,
limiter: params.Limiter,
keySet: cfg.AbuseIPDBKey != "",
minScore: cfg.AbuseIPDBMinScore,
action: cfg.DNSBLAbuseIPDBAction,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
hitNoter: hitNoter,
},
tagDeny{tags: cfg.TagDeny},
rateLimits{
limiter: params.Limiter,
exemptNets: cfg.RateLimitExemptNets,
exemptPaths: cfg.RateLimitExemptPaths,
banResponse: cfg.BanResponse,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
biasedThresholds: biasedThresholds{
lists: params.Lists,
tagLimitPercent: cfg.TagLimitPercent,
asnBytesLimitPercent: cfg.ASNBytesLimitPercent,
countryBytesLimitPercent: cfg.CountryBytesLimitPercent,
unknownCountryLimitPercent: cfg.UnknownCountryLimitPercent,
blocklistAction: cfg.BlocklistAction,
blocklistLimitPercent: cfg.BlocklistLimitPercent,
dnsblAbuseIPDBAction: cfg.DNSBLAbuseIPDBAction,
dnsblAbuseIPDBLimitPercent: cfg.DNSBLAbuseIPDBLimitPercent,
},
banMaker: sharedBanMaker,
},
}
h.besideLookup = []heuristic{h.trapPaths, h.ruleFiles, h.coreRuleSet}
h.afterBesideLookup = []heuristic{tagDeny{tags: cfg.TagDeny}}
h.afterAnswer = []heuristic{
refusalLimit{
limiter: params.Limiter,
perMinute: cfg.RefusalsLimitPerMinute,
banResponse: cfg.BanResponse,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
banMaker: sharedBanMaker,
},
}
h.onReport = []heuristic{
byteLimits{
limiter: params.Limiter,
bytesCounted: cfg.BytesCounted,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
banMaker: sharedBanMaker,
},
}
return h
}
// handler is waf's handler. It holds what every request shares; what
// belongs to one request is in a request.
type handler struct {
config *config.WAF
requestLog io.Writer
processLog *slog.Logger
now func() time.Time
metrics *metrics.Metrics
limiter *ratelimit.Limiter
ledger *bans.Ledger
geojs *geojs.GeoJS
anomalies *anomaly.Counters
lookupFile *lookup.File
lists *reputation.Lists
dnsbl *reputation.DNSBL
abuseIPDB *reputation.AbuseIPDB
routeWatch *routewatch.Client
rules *rules.Files
coreRuleSet CoreRuleSetInspector
alerts *alerts.Queue
config *config.WAF
requestLog io.Writer
processLog *slog.Logger
now func() time.Time
metrics *metrics.Metrics
limiter *ratelimit.Limiter
ledger *bans.Ledger
geojs *geojs.GeoJS
anomalies *anomaly.Counters
lookupFile *lookup.File
// waitForGeoJS is true when a setting needs each request's answer from
// GeoJS before the request goes on.
waitForGeoJS bool
@@ -202,4 +291,38 @@ type handler struct {
// awaitingReport holds the decision on each answered question by its id
// until its report comes, and nil for a question still being handled.
awaitingReport *lru.Cache[string, *decision]
// The lists of heuristics, each in the order its heuristics run. A client
// in SWW_WAF_ALLOW_NETS skips them all.
// beforeLookup are the heuristics that run before the client's AS number
// and country are looked up, so that a client they refuse is not looked
// up. header_limits comes first: it only measures and counts, and a
// request it refuses causes no other work.
beforeLookup []heuristic
// afterLookup are the heuristics that run once the lookup has been made.
afterLookup []heuristic
// besideLookup are the heuristics that only read the request. Their work
// runs beside the lookup and afterLookup, in findBesideLookup, which
// changes nothing of the request. Their check then applies what that work
// found, and runs only once waf takes it.
besideLookup []heuristic
// afterBesideLookup are the heuristics that run once afterLookup and
// besideLookup have both let the request through. tag_deny runs a second
// time, on the tag rule_files and core_rule_set can add.
afterBesideLookup []heuristic
// afterAnswer are the heuristics that run on a request once waf has
// answered it, before its decision line is written. They ban the client,
// and do not change what the request got: their verdict's action is not
// used.
afterAnswer []heuristic
// onReport are the heuristics that run on an allowed request when
// rproxy's report of it comes, before its report line is written. Like
// afterAnswer, they only ban.
onReport []heuristic
// The heuristics of besideLookup, whose work findBesideLookup does.
trapPaths trapPaths
ruleFiles ruleFiles
coreRuleSet coreRuleSet
}
+37 -22
View File
@@ -6,8 +6,10 @@ package waf
import (
"context"
"net/netip"
"time"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
@@ -18,28 +20,37 @@ import (
// refused, and bans the client. A request exempt from the rate limits is exempt
// from the byte limits too. Its figures are the window, the count and the
// limit.
type rateLimits struct{}
type rateLimits struct {
limiter *ratelimit.Limiter
exemptNets []netip.Prefix // SWW_WAF_RATE_LIMIT_EXEMPT_NETS
exemptPaths []string // SWW_WAF_RATE_LIMIT_EXEMPT_PATHS
banResponse int // SWW_WAF_BAN_RESPONSE
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
biasedThresholds biasedThresholds
banMaker banMaker
}
func (rateLimits) name() string { return "rate_limits" }
func (rateLimits) check(
func (r rateLimits) check(
_ context.Context,
rq *request,
now time.Time,
) *verdict {
rq.counted = rq.countedByRateLimits()
rq.counted = r.countedByRateLimits(rq)
if !rq.counted {
return nil
}
rq.rateLimitPercent, rq.bytesLimitPercent = rq.limitPercentages()
rq.rateLimitPercent, rq.bytesLimitPercent =
r.biasedThresholds.limitPercentages(rq)
rq.line.RateLimitPercent, rq.line.RateLimitPercentSetting =
rq.rateLimitPercent.logged()
rq.line.BytesLimitPercent, rq.line.BytesLimitPercentSetting =
rq.bytesLimitPercent.logged()
counts, hit, over := rq.h.limiter.Count(
clientGroup(rq.h.config, rq.client),
counts, hit, over := r.limiter.Count(
clientGroup(r.ipv6GroupPrefix, rq.client),
now,
rq.rateLimitPercent.percent,
)
@@ -49,28 +60,32 @@ func (rateLimits) check(
return nil
}
rq.banForLimit(now, hit, rq.h.config.BanResponse)
r.banMaker.banForLimit(rq, now, hit, r.banResponse)
return &verdict{
action: requestlog.ActionClientIsRateLimited, figures: limitFigures(hit),
}
}
// overAWholeRateLimit reports whether the heuristic rate_limits will refuse the
// request at now, whatever its client's limit percentages: the rate limits
// count it, and one more request from its client takes it over a whole rate
// limit. It counts nothing.
func (rq *request) overAWholeRateLimit(now time.Time) bool {
return rq.countedByRateLimits() &&
rq.h.limiter.OverARateLimit(clientGroup(rq.h.config, rq.client), now)
// refusesWhateverTheLookupAnswers reports whether the heuristic rate_limits
// will refuse rq at now, whatever its client's limit percentages, which the
// lookup's answer can lower: the rate limits count it, and one more request
// from its client takes it over a whole rate limit. It counts nothing.
func (r rateLimits) refusesWhateverTheLookupAnswers(
rq *request,
now time.Time,
) bool {
return r.countedByRateLimits(rq) &&
r.limiter.OverARateLimit(
clientGroup(r.ipv6GroupPrefix, rq.client),
now,
)
}
// countedByRateLimits reports whether the rate limits count the request: its
// client is not in SWW_WAF_RATE_LIMIT_EXEMPT_NETS, and its path is not exempt
// under SWW_WAF_RATE_LIMIT_EXEMPT_PATHS.
func (rq *request) countedByRateLimits() bool {
cfg := rq.h.config
return !isInside(rq.client, cfg.RateLimitExemptNets) &&
!pathExempt(rq.in, cfg.RateLimitExemptPaths)
// countedByRateLimits reports whether the rate limits count rq: its client is
// not in SWW_WAF_RATE_LIMIT_EXEMPT_NETS, and its path is not exempt under
// SWW_WAF_RATE_LIMIT_EXEMPT_PATHS.
func (r rateLimits) countedByRateLimits(rq *request) bool {
return !isInside(rq.client, r.exemptNets) &&
!pathExempt(rq.in, r.exemptPaths)
}
+15 -9
View File
@@ -8,6 +8,7 @@ import (
"net/http"
"time"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
@@ -18,17 +19,22 @@ import (
// a limit, which bans the client. A client in SWW_WAF_ALLOW_NETS, which the
// heuristics skip, is not counted, and nothing is while the limit is off.
// Its figures are the window, the count and the limit.
type refusalLimit struct{}
type refusalLimit struct {
limiter *ratelimit.Limiter
perMinute int64 // SWW_WAF_REFUSALS_LIMIT_PER_MINUTE
banResponse int // SWW_WAF_BAN_RESPONSE
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
banMaker banMaker
}
func (refusalLimit) name() string { return "refusal_limit" }
func (refusalLimit) check(
func (r refusalLimit) check(
_ context.Context,
rq *request,
now time.Time,
) *verdict {
cfg := rq.h.config
if cfg.RefusalsLimitPerMinute == 0 {
if r.perMinute == 0 {
return nil
}
@@ -36,10 +42,10 @@ func (refusalLimit) check(
return nil
}
hit, over := rq.h.limiter.CountRefusal(
clientGroup(rq.h.config, rq.client),
hit, over := r.limiter.CountRefusal(
clientGroup(r.ipv6GroupPrefix, rq.client),
now,
cfg.RefusalsLimitPerMinute,
r.perMinute,
)
if !over {
return nil
@@ -54,10 +60,10 @@ func (refusalLimit) check(
status = http.StatusForbidden
case requestlog.ActionTemporarilyBanned,
requestlog.ActionPermanentlyBanned:
status = cfg.BanResponse
status = r.banResponse
}
rq.banForLimit(now, hit, status)
r.banMaker.banForLimit(rq, now, hit, status)
return &verdict{figures: limitFigures(hit)}
}
+21 -8
View File
@@ -5,22 +5,35 @@ package waf
import (
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/metrics"
)
// reputationHitNoter notes the reputation sources that list a request's
// client, for the reputation heuristics.
type reputationHitNoter struct {
metrics *metrics.Metrics
alerts *alerts.Queue
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
}
// noteListed notes each of sources, the URLs of the blocklists or the
// DNSBL zones, their keys masked, that list the client, as noteHit does,
// DNSBL zones, their keys masked, that list rq's client, as noteHit does,
// with reason.
func (rq *request) noteListed(sources []string, reason string) {
func (n reputationHitNoter) noteListed(
rq *request, sources []string, reason string,
) {
for _, source := range sources {
rq.noteHit(bans.ReputationHit{Source: source}, reason)
n.noteHit(rq, bans.ReputationHit{Source: source}, reason)
}
}
// noteHit adds hit's source, which lists the client, to the log line's
// noteHit adds hit's source, which lists rq's client, to the log line's
// reputation, and hit to the notes of a ban the request makes, counts the
// source in the metrics, and raises a reputation_hit alert with reason,
// whose detail gives hit's source and score.
func (rq *request) noteHit(hit bans.ReputationHit, reason string) {
func (n reputationHitNoter) noteHit(
rq *request, hit bans.ReputationHit, reason string,
) {
detail := map[string]any{"source": hit.Source}
if hit.Score != nil {
detail["score"] = *hit.Score
@@ -28,12 +41,12 @@ func (rq *request) noteHit(hit bans.ReputationHit, reason string) {
rq.line.Reputation = append(rq.line.Reputation, hit.Source)
rq.reputation = append(rq.reputation, hit)
rq.h.metrics.ReputationHit(hit.Source)
rq.h.alerts.Raise(alerts.Alert{
n.metrics.ReputationHit(hit.Source)
n.alerts.Raise(alerts.Alert{
Event: alerts.EventReputationHit,
Host: rq.site,
Client: rq.client,
Netblock: clientGroup(rq.h.config, rq.client),
Netblock: clientGroup(n.ipv6GroupPrefix, rq.client),
ASN: rq.line.ASN,
ASName: rq.line.ASName,
Country: rq.line.Country,
+13 -8
View File
@@ -122,8 +122,11 @@ func (h *handler) startRequest(
RequestID: requestID(r, isQuestion),
PeerIP: peer.String(),
ForwardedFor: strings.Join(forwardedFor, ", "),
ClientGroup: clientGroup(h.config, client).String(),
ContentType: r.Header.Get("Content-Type"),
ClientGroup: clientGroup(
h.config.IPv6GroupPrefix,
client,
).String(),
ContentType: r.Header.Get("Content-Type"),
RequestHeaders: requestHeaders(
r,
h.config.LogRequestHeaders,
@@ -233,7 +236,7 @@ func (rq *request) checkClient(ctx context.Context) *verdict {
now := rq.h.now()
v := rq.runHeuristics(ctx, beforeLookup(), now)
v := rq.runHeuristics(ctx, rq.h.beforeLookup, now)
if v != nil {
return v
}
@@ -265,7 +268,7 @@ func (rq *request) runAfterLookup(
// again: a request counted at the time it arrived could be counted
// after a later one of its client's, which the rate limits would take
// for the clock set back.
return rq.runHeuristics(ctx, afterLookup(), rq.h.now())
return rq.runHeuristics(ctx, rq.h.afterLookup, rq.h.now())
}
// pathExempt reports whether r is exempt under prefixes,
@@ -375,7 +378,7 @@ func (rq *request) addToHistory() {
}
rq.h.limiter.AddToHistory(
clientGroup(rq.h.config, rq.client),
clientGroup(rq.h.config.IPv6GroupPrefix, rq.client),
rq.h.now(),
request,
)
@@ -404,11 +407,11 @@ func (rq *request) countAnomalies() {
rq.h.anomalies.Count(rq.h.now(), anomaly.Request{
Host: rq.site,
Client: rq.client,
ClientGroup: clientGroup(rq.h.config, rq.client),
ClientGroup: clientGroup(rq.h.config.IPv6GroupPrefix, rq.client),
ASN: answer.ASN,
ASName: answer.ASName,
Country: answer.Country,
Bytes: rq.countedBytes(),
Bytes: rq.countedBytes(rq.h.config.BytesCounted),
})
}
@@ -435,7 +438,9 @@ func (rq *request) answerAtTheEnd() (geojs.Answer, bool) {
return rq.lookupAnswer, true
}
return rq.h.geojs.Kept(clientGroup(rq.h.config, rq.client))
return rq.h.geojs.Kept(
clientGroup(rq.h.config.IPv6GroupPrefix, rq.client),
)
}
// responseWriter is waf's answer to a request. It notes the status and
+2 -2
View File
@@ -13,7 +13,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
const (
@@ -248,7 +248,7 @@ func TestTimingsAreInOrder(t *testing.T) {
wantStatus(t, do(t, req), http.StatusForbidden)
refused := out.endedLines(t, 2)[1]
wantStatus(t, get(t, server.Addr, waf.HealthPath), http.StatusOK)
wantStatus(t, get(t, server.Addr, wafprotocol.HealthPath), http.StatusOK)
health := out.endedLines(t, 3)[2]
// A request passed to the app has every timing; one refused, none of
+10 -5
View File
@@ -8,6 +8,7 @@ import (
"time"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/routewatch"
)
// routeWatchTags adds to the request the tags asn=<number> and prefix=<netblock>
@@ -17,22 +18,26 @@ import (
// SWW_WAF_ROUTEWATCH_URL is unset it does nothing. Answers are kept by client and
// asked for by the request's address, and a route without an origin AS gives only
// the prefix= tag.
type routeWatchTags struct{}
type routeWatchTags struct {
// routeWatch is nil while SWW_WAF_ROUTEWATCH_URL is unset.
routeWatch *routewatch.Client
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
}
func (routeWatchTags) name() string { return "routewatch_tags" }
func (routeWatchTags) check(
func (r routeWatchTags) check(
ctx context.Context,
rq *request,
_ time.Time,
) *verdict {
if rq.h.routeWatch == nil {
if r.routeWatch == nil {
return nil
}
route, missing := rq.h.routeWatch.Route(
route, missing := r.routeWatch.Route(
ctx,
clientGroup(rq.h.config, rq.client),
clientGroup(r.ipv6GroupPrefix, rq.client),
rq.client,
)
if missing != "" {
+10 -4
View File
@@ -9,6 +9,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
)
@@ -22,11 +23,15 @@ import (
// mode raises the alert for the ban
// it would have made. Either rule's match is noted as an offence, for the
// client's history. Its figures are the rule's id and action.
type ruleFiles struct{}
type ruleFiles struct {
rules *rules.Files
metrics *metrics.Metrics
banMaker banMaker
}
func (ruleFiles) name() string { return "rule_files" }
func (ruleFiles) check(
func (r ruleFiles) check(
_ context.Context,
rq *request,
now time.Time,
@@ -35,7 +40,7 @@ func (ruleFiles) check(
for _, rule := range matched {
rq.line.RuleIDs = append(rq.line.RuleIDs, rule.ID)
rq.h.metrics.RuleMatched(rule.ID, rule.Action)
r.metrics.RuleMatched(rule.ID, rule.Action)
}
if len(matched) == 0 {
@@ -60,7 +65,8 @@ func (ruleFiles) check(
rq.attack = true
return &verdict{
action: rq.banForAttack(
action: r.banMaker.banForAttack(
rq,
now,
bans.Notes{RuleID: last.ID, Target: last.Target},
),
+9 -3
View File
@@ -12,13 +12,19 @@ import (
// tagDeny refuses a request that carries a tag in SWW_WAF_TAG_DENY. Its figure is
// the first such tag, in sorted order.
type tagDeny struct{}
type tagDeny struct {
tags []string // SWW_WAF_TAG_DENY
}
func (tagDeny) name() string { return "tag_deny" }
func (tagDeny) check(_ context.Context, rq *request, _ time.Time) *verdict {
func (t tagDeny) check(
_ context.Context,
rq *request,
_ time.Time,
) *verdict {
for _, tag := range rq.line.Tags {
if slices.Contains(rq.h.config.TagDeny, tag) {
if slices.Contains(t.tags, tag) {
return &verdict{
action: requestlog.ActionClientDenied,
figures: map[string]any{"tag": tag},
+11 -3
View File
@@ -6,17 +6,25 @@ package waf
import (
"context"
"time"
"sneak.berlin/go/smallwebwaf/internal/reputation"
)
// tagLists adds to the request one tag for each list of SWW_WAF_TAG_LISTS that
// holds the client, the tag being the list's name. It refuses nothing, and runs
// first so that every refusal's log line carries these tags.
type tagLists struct{}
type tagLists struct {
lists *reputation.Lists
}
func (tagLists) name() string { return "tag_lists" }
func (tagLists) check(_ context.Context, rq *request, _ time.Time) *verdict {
for _, name := range rq.h.lists.TagsOf(rq.client) {
func (t tagLists) check(
_ context.Context,
rq *request,
_ time.Time,
) *verdict {
for _, name := range t.lists.TagsOf(rq.client) {
rq.addTag(name)
}
+15 -8
View File
@@ -16,11 +16,14 @@ import (
// it bans the client's netblock, or in observe mode raises the alert for the
// ban it would have made. The ban's notes give the path as sent, and so does
// its figure.
type trapPaths struct{}
type trapPaths struct {
paths []string // SWW_WAF_TRAP_PATHS
banMaker banMaker
}
func (trapPaths) name() string { return "trap_paths" }
func (trapPaths) check(
func (t trapPaths) check(
_ context.Context,
rq *request,
now time.Time,
@@ -33,18 +36,22 @@ func (trapPaths) check(
rq.attack = true
return &verdict{
action: rq.banForAttack(now, bans.Notes{TrapPath: path}),
action: t.banMaker.banForAttack(
rq,
now,
bans.Notes{TrapPath: path},
),
figures: map[string]any{"path": path},
}
}
// trapPath returns the request's path as sent, when its path as a path rule
// sees it, without the query, as sent or once percent-decoded, is in
// trapPath returns rq's path as sent, when its path as a path rule sees it,
// without the query, as sent or once percent-decoded, is in
// SWW_WAF_TRAP_PATHS, and "" otherwise.
func (rq *request) trapPath() string {
func (t trapPaths) trapPath(rq *request) string {
path := rules.Path(rq.in)
if !slices.Contains(rq.h.config.TrapPaths, path) &&
!slices.Contains(rq.h.config.TrapPaths, rules.DecodeOnce(path)) {
if !slices.Contains(t.paths, path) &&
!slices.Contains(t.paths, rules.DecodeOnce(path)) {
return ""
}
+72 -238
View File
@@ -1,75 +1,21 @@
package waf
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"net/http"
"net/netip"
"strconv"
"strings"
"time"
"unicode/utf8"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// The headers of rproxy's question and of waf's answer, as
// "waf" in SPEC.md gives them.
const (
// ClientHeader marks a request as rproxy's question, and gives the
// visitor's address.
ClientHeader = "X-Sww-Client"
// PeerHeader gives the address of rproxy's own TCP peer.
PeerHeader = "X-Sww-Peer"
// ContentLengthHeader gives the body length the visitor announced.
ContentLengthHeader = "X-Sww-Content-Length"
// DispositionHeader gives waf's answer.
DispositionHeader = "X-Sww-Disposition"
)
// The values of DispositionHeader.
const (
// DispositionAllow tells rproxy to pass the request to the app.
DispositionAllow = "allow"
// DispositionDeny tells rproxy to give the visitor the answer's status,
// headers and body.
DispositionDeny = "deny"
// DispositionClose tells rproxy to close the visitor's connection
// without an answer, which SWW_WAF_BAN_RESPONSE asks for when it is close.
// waf closing the question's connection instead would read as
// waf being down, and rproxy would then let the request through.
DispositionClose = "close"
)
// ReportPath is where rproxy sends the report of each request waf
// answered.
const ReportPath = "/_sww/report"
// QuestionBodyMaxBytes is the most of a visitor's body a question carries.
const QuestionBodyMaxBytes = 64 << 10
const (
// maxAwaitingReport is the most answered requests held until their
// reports come. Past it, the request answered longest ago is dropped.
maxAwaitingReport = 10000
// maxRequestIDBytes is the longest id a question may give.
maxRequestIDBytes = 128
// reportBodyMaxBytes is the most of a report's body that is read. It is
// far above any report's.
reportBodyMaxBytes = 1 << 20
// maxHTTPStatus is the highest status a report may give.
maxHTTPStatus = 599
// maxReportedBytes is the most bytes a report may give for each way, 1
// PiB, so that adding them up cannot overflow.
maxReportedBytes = 1 << 50
// maxReportedMilliseconds is the longest duration a report may give,
// about 31 years, so that it fits a time.Duration.
maxReportedMilliseconds = 1e12
// wafTimeout bounds how long a request to waf takes to
// arrive, and its answer to leave, and wafIdleTimeout how long a
// kept-open connection waits for its next request.
@@ -77,26 +23,6 @@ const (
wafIdleTimeout = 120 * time.Second
)
var (
errReportFieldMissing = errors.New("a field is missing")
errReportValue = errors.New("a value is out of range")
)
// Report is the JSON object rproxy posts to ReportPath once the visitor's
// answer has ended, its fields named as the request log's of the same
// meaning. RequestID is the question's id. Status is what the visitor got, 0
// if nothing. The durations are in milliseconds.
//
//nolint:tagliatelle // docs/request-log.md names the fields in snake_case
type Report struct {
RequestID string `json:"request_id"`
Status int `json:"status"`
RequestBytes int64 `json:"request_bytes"`
ResponseBytes int64 `json:"response_bytes"`
DurationTotalMS float64 `json:"duration_total_ms"`
DurationUpstreamTotalMS float64 `json:"duration_upstream_total_ms"`
}
// New returns the server `sww waf` runs. It answers
// rproxy's questions, takes rproxy's reports and answers its own endpoints. It
// passes nothing to an app. rproxy applies the size and time limits to the
@@ -123,19 +49,19 @@ func New(params Params) *Server {
GeoJS: h.geojs,
Anomalies: h.anomalies,
LookupFile: h.lookupFile,
Lists: h.lists,
DNSBL: h.dnsbl,
AbuseIPDB: h.abuseIPDB,
RouteWatch: h.routeWatch,
Lists: params.Lists,
DNSBL: params.DNSBL,
AbuseIPDB: params.AbuseIPDB,
RouteWatch: params.RouteWatch,
Metrics: h.metrics,
}
}
// ServeHTTP handles one request to waf: a request carrying
// ClientHeader is rproxy's question, and any other is for waf
// wafprotocol.ClientHeader is rproxy's question, and any other is for waf
// itself.
func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
if len(r.Header.Values(ClientHeader)) > 0 {
if len(r.Header.Values(wafprotocol.ClientHeader)) > 0 {
h.answerQuestion(w, r)
return
@@ -146,35 +72,38 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// refusedWhateverTheAnswer reports whether the request is one that does not wait
// for GeoJS, since the checks will refuse it at now whatever GeoJS answers about
// its client: the heuristic blocklists or rate_limits will refuse it, as a
// function in that heuristic's file says. In observe mode, which refuses
// nothing, none is. rproxy lets a request through when waf answers too late.
// its client: a heuristic of afterLookup that implements refusesWithoutLookup
// says it will refuse it. In observe mode, which refuses nothing, none is.
// rproxy lets a request through when waf answers too late.
func (rq *request) refusedWhateverTheAnswer(now time.Time) bool {
if rq.h.config.Observe {
return false
}
return rq.refusedByBlocklists(rq.h.lists.ListedBy(rq.client)) ||
rq.overAWholeRateLimit(now)
for _, each := range rq.h.afterLookup {
knows, ok := each.(refusesWithoutLookup)
if ok && knows.refusesWhateverTheLookupAnswers(rq, now) {
return true
}
}
return false
}
// answerQuestion runs the checks on the visitor's request a question
// carries, and answers rproxy what to do with it. A denied request's counting
// is finished here, and an allowed one's by its report.
func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
client, body, ok := h.readQuestion(w, r)
question, ok := h.readQuestion(w, r)
if !ok {
return
}
// One that cannot be read leaves the log line's peer_ip invalid.
peer, _ := netip.ParseAddr(r.Header.Get(PeerHeader))
rq := h.startRequest(
w,
asTheVisitorSentIt(r, body),
client.Unmap(),
peer.Unmap(),
question.Visitor,
question.Client,
question.Peer,
true,
)
// These run even when a check panics. finish runs first, so that what is
@@ -189,12 +118,15 @@ func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
rq.line.WAFLine = requestlog.WAFLineDecision
refused := rq.check(r.Context())
rq.checked = time.Now()
underOwnPrefix := strings.HasPrefix(r.URL.Path, ownEndpointPrefix)
underOwnPrefix := strings.HasPrefix(
r.URL.Path,
wafprotocol.OwnEndpointPrefix,
)
if refused == nil && !underOwnPrefix {
rq.allow()
// In observe mode, what enforce mode would have refused counts here.
rq.runBanningHeuristics(r.Context(), afterAnswer())
rq.runBanningHeuristics(r.Context(), rq.h.afterAnswer)
return
}
@@ -205,7 +137,10 @@ func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
// The app never sees this prefix. waf's own endpoints are at its
// own address, so this answers as a path with no endpoint does.
rq.line.Action = requestlog.ActionOwnEndpoint
rq.out.Header().Set(DispositionHeader, DispositionDeny)
wafprotocol.SetAnswerHeaders(
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionDeny},
)
http.Error(
rq.out,
http.StatusText(http.StatusNotFound),
@@ -213,48 +148,22 @@ func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
)
}
rq.banAndCount(r.Context(), afterAnswer())
rq.banAndCount(r.Context(), rq.h.afterAnswer)
}
// readQuestion returns the visitor's address and the body a question carries,
// and whether waf takes the question. It answers a question it does not
// take with 400 and no disposition. rproxy passes that 400 to the visitor,
// and SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION does not apply. The id of
// a question waf takes is held from here on, without a decision yet, so
// that another question with that id is refused while this one is being
// handled.
// readQuestion reads a question, and reports whether waf takes it. It answers
// a question it does not take with 400 and no disposition. rproxy passes that
// 400 to the visitor, and SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION does
// not apply. The id of a question waf takes is held from here on, without a
// decision yet, so that another question with that id is refused while this
// one is being handled.
//
// The body is read before any check, since Go's server does not send an answer
// until it has read the body. A question whose body stops arriving gets no
// answer, and is not looked at.
// A question whose body stops arriving gets no answer, and is not looked at.
func (h *handler) readQuestion(
w http.ResponseWriter, r *http.Request,
) (netip.Addr, []byte, bool) {
client, err := netip.ParseAddr(r.Header.Get(ClientHeader))
if err != nil || client.Zone() != "" {
h.refuseQuestion(w, ClientHeader+" is not an address without a zone")
return netip.Addr{}, nil, false
}
id := r.Header.Get(requestIDHeader)
if id == "" || len(id) > maxRequestIDBytes || !utf8.ValidString(id) {
h.refuseQuestion(w, requestIDHeader+" is not a valid id")
return netip.Addr{}, nil, false
}
body, err := io.ReadAll(
http.MaxBytesReader(w, r.Body, QuestionBodyMaxBytes),
)
if _, tooLarge := errors.AsType[*http.MaxBytesError](err); tooLarge {
h.refuseQuestion(w, "the body is over 64 KiB")
return netip.Addr{}, nil, false
}
if err != nil {
) (wafprotocol.Question, bool) {
question, err := wafprotocol.ReadQuestion(w, r)
if errors.Is(err, wafprotocol.ErrBodyStoppedArriving) {
h.processLog.Warn(
"a question's body stopped arriving",
"error",
@@ -265,17 +174,23 @@ func (h *handler) readQuestion(
panic(http.ErrAbortHandler)
}
held, _ := h.awaitingReport.ContainsOrAdd(id, nil)
if err != nil {
h.refuseQuestion(w, err.Error())
return question, false
}
held, _ := h.awaitingReport.ContainsOrAdd(question.RequestID, nil)
if held {
h.refuseQuestion(
w,
requestIDHeader+" is the id of a request waf holds",
wafprotocol.RequestIDHeader+" is the id of a request waf holds",
)
return netip.Addr{}, nil, false
return question, false
}
return client, body, true
return question, true
}
// refuseQuestion answers a question waf does not take, for reason.
@@ -284,48 +199,18 @@ func (h *handler) refuseQuestion(w http.ResponseWriter, reason string) {
http.Error(w, reason, http.StatusBadRequest)
}
// asTheVisitorSentIt returns a copy of r, a question, as the checks and the
// log line are to see it: without the headers rproxy adds for waf,
// with body, which has been read of r, and with the length the visitor
// announced, as -1 when it announced none. The question's own length covers
// only the part of the body it carries.
func asTheVisitorSentIt(r *http.Request, body []byte) *http.Request {
sent := r.Clone(r.Context())
announced := sent.Header.Get(ContentLengthHeader)
sent.Header.Del(ClientHeader)
sent.Header.Del(PeerHeader)
sent.Header.Del(ContentLengthHeader)
sent.Header.Del("Content-Length")
sent.ContentLength = -1
length, err := strconv.ParseInt(announced, 10, 64)
if err == nil && length >= 0 {
sent.ContentLength = length
sent.Header.Set("Content-Length", announced)
}
sent.Body = http.NoBody
if len(body) > 0 {
sent.Body = io.NopCloser(bytes.NewReader(body))
}
return sent
}
// allow answers a question with DispositionAllow and, while
// allow answers a question with wafprotocol.DispositionAllow and, while
// SWW_WAF_ADD_LOOKUP_HEADERS is true, the client's AS number and country.
func (rq *request) allow() {
rq.allowed = true
header := rq.out.Header()
header.Set(DispositionHeader, DispositionAllow)
answer := wafprotocol.Answer{Disposition: wafprotocol.DispositionAllow}
if rq.h.config.AddLookupHeaders {
setLookupHeaders(header, rq.line.ASN, rq.line.Country)
answer.ASN = rq.line.ASN
answer.Country = rq.line.Country
}
wafprotocol.SetAnswerHeaders(rq.out.Header(), answer)
rq.out.WriteHeader(http.StatusOK)
}
@@ -343,13 +228,19 @@ func (rq *request) deny(r refusal) {
if r.status == 0 {
// SWW_WAF_BAN_RESPONSE is close. Written past rq.out, since the log line's
// status stays 0: the visitor is sent nothing.
rq.out.Header().Set(DispositionHeader, DispositionClose)
wafprotocol.SetAnswerHeaders(
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionClose},
)
rq.out.ResponseWriter.WriteHeader(http.StatusOK)
return
}
rq.out.Header().Set(DispositionHeader, DispositionDeny)
wafprotocol.SetAnswerHeaders(
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionDeny},
)
http.Error(rq.out, http.StatusText(r.status), r.status)
}
@@ -357,7 +248,7 @@ func (rq *request) deny(r refusal) {
// runs on it: the address is on the private network, where the peers are
// rproxy and the admin's tools, not visitors.
func (h *handler) answerOwnEndpoint(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodPost && r.URL.Path == ReportPath {
if r.Method == http.MethodPost && r.URL.Path == wafprotocol.ReportPath {
h.takeReport(w, r)
return
@@ -368,7 +259,7 @@ func (h *handler) answerOwnEndpoint(w http.ResponseWriter, r *http.Request) {
rq := h.startRequest(w, r, peer, peer, false)
defer rq.finish()
if r.Method == http.MethodGet && r.URL.Path == HealthPath {
if r.Method == http.MethodGet && r.URL.Path == wafprotocol.HealthPath {
rq.answerHealth()
return
@@ -377,11 +268,11 @@ func (h *handler) answerOwnEndpoint(w http.ResponseWriter, r *http.Request) {
rq.answerAdmin()
}
// takeReport answers POST ReportPath: it finishes the request the report
// names, if it holds that request's decision, and ignores the report
// takeReport answers POST wafprotocol.ReportPath: it finishes the request the
// report names, if it holds that request's decision, and ignores the report
// otherwise.
func (h *handler) takeReport(w http.ResponseWriter, r *http.Request) {
report, err := readReport(w, r)
report, err := wafprotocol.ReadReport(w, r)
if err != nil {
h.processLog.Warn("a report is refused", "error", err.Error())
http.Error(w, "the body is not a report", http.StatusBadRequest)
@@ -399,63 +290,6 @@ func (h *handler) takeReport(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusNoContent)
}
// readReport reads a report from r's body, which must be one JSON object with
// all of Report's fields, in the ranges SPEC.md gives.
func readReport(w http.ResponseWriter, r *http.Request) (Report, error) {
var report Report
body, err := io.ReadAll(
http.MaxBytesReader(w, r.Body, reportBodyMaxBytes),
)
if err != nil {
return report, fmt.Errorf("read the body: %w", err)
}
var fields map[string]json.RawMessage
err = json.Unmarshal(body, &fields)
if err != nil {
return report, fmt.Errorf("decode the body: %w", err)
}
for _, name := range []string{
"request_id", "status", "request_bytes", "response_bytes",
"duration_total_ms", "duration_upstream_total_ms",
} {
value, there := fields[name]
if !there || string(value) == "null" {
return report, fmt.Errorf("%w: %s", errReportFieldMissing, name)
}
}
err = json.Unmarshal(body, &report)
if err != nil {
return report, fmt.Errorf("decode the body: %w", err)
}
return report, report.checkRanges()
}
// checkRanges returns an error for a value of the report that cannot be true.
func (report *Report) checkRanges() error {
switch {
case report.Status != 0 &&
(report.Status < http.StatusContinue || report.Status > maxHTTPStatus):
return fmt.Errorf("%w: status", errReportValue)
case report.RequestBytes < 0 || report.RequestBytes > maxReportedBytes:
return fmt.Errorf("%w: request_bytes", errReportValue)
case report.ResponseBytes < 0 || report.ResponseBytes > maxReportedBytes:
return fmt.Errorf("%w: response_bytes", errReportValue)
case report.DurationTotalMS < 0 || report.DurationTotalMS > maxReportedMilliseconds:
return fmt.Errorf("%w: duration_total_ms", errReportValue)
case report.DurationUpstreamTotalMS < 0 ||
report.DurationUpstreamTotalMS > maxReportedMilliseconds:
return fmt.Errorf("%w: duration_upstream_total_ms", errReportValue)
}
return nil
}
// finishWith finishes a request with what its report says, and writes the
// request's last log line, which holds its decision's fields and the report's.
// For an allowed request it first runs the heuristics of onReport, and adds the
@@ -465,7 +299,7 @@ func (report *Report) checkRanges() error {
func (h *handler) finishWith(
ctx context.Context,
held *decision,
report Report,
report wafprotocol.Report,
) {
rq := &request{
decision: *held,
@@ -486,7 +320,7 @@ func (h *handler) finishWith(
if rq.allowed {
rq.line.DurationUpstreamTotalMS = &report.DurationUpstreamTotalMS
rq.banAndCount(ctx, onReport())
rq.banAndCount(ctx, rq.h.onReport)
rq.h.metrics.RequestReported(
&rq.line,
+195 -86
View File
@@ -22,6 +22,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
const (
@@ -106,8 +107,8 @@ func (s *WAF) questionWith(
req := newRequest(s.t, method, s.addr, path, body)
req.Host = host
req.Header.Set(waf.ClientHeader, from)
req.Header.Set(waf.PeerHeader, rproxyPeer)
req.Header.Set(wafprotocol.ClientHeader, from)
req.Header.Set(wafprotocol.PeerHeader, rproxyPeer)
req.Header.Set(requestIDHeader, rand.Text())
return req
@@ -134,13 +135,18 @@ func (s *WAF) allowed(from string) string {
s.t.Helper()
req := s.question(siteA, from, "/")
wantDisposition(s.t, do(s.t, req), http.StatusOK, waf.DispositionAllow)
wantDisposition(
s.t,
do(s.t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
)
return req.Header.Get(requestIDHeader)
}
// report sends rproxy's report and returns the answer.
func (s *WAF) report(report waf.Report) answer {
func (s *WAF) report(report wafprotocol.Report) answer {
s.t.Helper()
body, err := json.Marshal(report)
@@ -155,8 +161,11 @@ func (s *WAF) report(report waf.Report) answer {
func (s *WAF) reportBody(body string) answer {
s.t.Helper()
return do(s.t, newRequest(s.t, http.MethodPost, s.addr, waf.ReportPath,
strings.NewReader(body)))
return do(
s.t,
newRequest(s.t, http.MethodPost, s.addr, wafprotocol.ReportPath,
strings.NewReader(body)),
)
}
// passedOn asks about a request from the visitor at from on siteA, and reports
@@ -164,7 +173,7 @@ func (s *WAF) reportBody(body string) answer {
func (s *WAF) passedOn(from string) {
s.t.Helper()
got := s.report(waf.Report{
got := s.report(wafprotocol.Report{
RequestID: s.allowed(from),
Status: http.StatusOK,
RequestBytes: bodyBytes,
@@ -210,7 +219,7 @@ func wantDisposition(
t.Helper()
wantStatus(t, got, status)
if answered := got.header.Get(waf.DispositionHeader); answered !=
if answered := got.header.Get(wafprotocol.DispositionHeader); answered !=
disposition {
t.Errorf(
"answered with disposition %q, want %q",
@@ -272,10 +281,10 @@ func TestQuestionAllowedIsAnsweredAndLogged(t *testing.T) {
req := s.question(siteA, client, "/page?q=1&token=visible")
req.Header.Set(requestIDHeader, "the-id")
req.Header.Set("X-Forwarded-Proto", "https")
req.Header.Set(waf.ContentLengthHeader, "1234")
req.Header.Set(wafprotocol.ContentLengthHeader, "1234")
got := do(t, req)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
wantBody(t, got, "")
if id := got.header.Get(requestIDHeader); id != "" {
@@ -308,7 +317,12 @@ func TestIPv6VisitorIsCountedInItsGroup(t *testing.T) {
s.allowed("2001:db8::1")
got := s.ask("2001:db8::2", "/")
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
}
func TestIPv4VisitorInIPv6FormIsTheIPv4Client(t *testing.T) {
@@ -317,9 +331,14 @@ func TestIPv4VisitorInIPv6FormIsTheIPv4Client(t *testing.T) {
s := startWAF(t, "", map[string]string{denyNets: client})
req := s.question(siteA, "::ffff:"+client, "/")
req.Header.Set(waf.PeerHeader, "::ffff:"+rproxyPeer)
req.Header.Set(wafprotocol.PeerHeader, "::ffff:"+rproxyPeer)
wantDisposition(t, do(t, req), http.StatusForbidden, waf.DispositionDeny)
wantDisposition(
t,
do(t, req),
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
line := s.out.requestLine(t)
if line.ClientIP != client || line.PeerIP != rproxyPeer {
@@ -339,13 +358,13 @@ func TestMalformedQuestionGets400AndAWarningAndNoLine(t *testing.T) {
change func(*http.Request)
}{
{"client not an address", func(req *http.Request) {
req.Header.Set(waf.ClientHeader, "not an address")
req.Header.Set(wafprotocol.ClientHeader, "not an address")
}},
{"client empty", func(req *http.Request) {
req.Header.Set(waf.ClientHeader, "")
req.Header.Set(wafprotocol.ClientHeader, "")
}},
{"client address with a zone", func(req *http.Request) {
req.Header.Set(waf.ClientHeader, "2001:db8::1%eth0")
req.Header.Set(wafprotocol.ClientHeader, "2001:db8::1%eth0")
}},
{"id absent", func(req *http.Request) {
req.Header.Del(requestIDHeader)
@@ -390,7 +409,12 @@ func TestLongestValidIDIsTaken(t *testing.T) {
req := s.question(siteA, client, "/")
req.Header.Set(requestIDHeader, strings.Repeat("x", 128))
wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow)
wantDisposition(
t,
do(t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
)
}
func TestQuestionWithABodyOver64KiBGets400(t *testing.T) {
@@ -399,16 +423,18 @@ func TestQuestionWithABodyOver64KiBGets400(t *testing.T) {
s := startWAF(t, "", nil)
over := strings.NewReader(
strings.Repeat("x", waf.QuestionBodyMaxBytes+1),
strings.Repeat("x", wafprotocol.QuestionBodyMaxBytes+1),
)
got := do(t, s.questionWith(http.MethodPost, siteA, client, "/", over))
wantDisposition(t, got, http.StatusBadRequest, "")
wantProcessMessages(t, s, "a question is refused", 1)
wantNoRequestLines(t, s)
whole := strings.NewReader(strings.Repeat("x", waf.QuestionBodyMaxBytes))
whole := strings.NewReader(
strings.Repeat("x", wafprotocol.QuestionBodyMaxBytes),
)
got = do(t, s.questionWith(http.MethodPost, siteA, client, "/", whole))
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
}
func TestQuestionWithAHeldIDIsRefused(t *testing.T) {
@@ -424,12 +450,12 @@ func TestQuestionWithAHeldIDIsRefused(t *testing.T) {
wantProcessMessages(t, s, "a question is refused", 1)
// A denied request's id is held as well.
denied := s.question(siteA, otherClient, waf.HealthPath)
denied := s.question(siteA, otherClient, wafprotocol.HealthPath)
wantDisposition(
t,
do(t, denied),
http.StatusNotFound,
waf.DispositionDeny,
wafprotocol.DispositionDeny,
)
again = s.question(siteA, client, "/")
@@ -437,7 +463,9 @@ func TestQuestionWithAHeldIDIsRefused(t *testing.T) {
wantDisposition(t, do(t, again), http.StatusBadRequest, "")
// The report ends the holding, and the refused questions changed nothing.
got := s.report(waf.Report{RequestID: id, ResponseBytes: answerBytes})
got := s.report(
wafprotocol.Report{RequestID: id, ResponseBytes: answerBytes},
)
wantStatus(t, got, http.StatusNoContent)
if history := s.history(); history.ResponseBytes != answerBytes {
@@ -446,7 +474,12 @@ func TestQuestionWithAHeldIDIsRefused(t *testing.T) {
again = s.question(siteA, client, "/")
again.Header.Set(requestIDHeader, id)
wantDisposition(t, do(t, again), http.StatusOK, waf.DispositionAllow)
wantDisposition(
t,
do(t, again),
http.StatusOK,
wafprotocol.DispositionAllow,
)
}
func TestReportNeverReachesAQuestionStillBeingHandled(t *testing.T) {
@@ -481,7 +514,7 @@ func TestReportNeverReachesAQuestionStillBeingHandled(t *testing.T) {
// A report for a question not yet answered is ignored, and that question's
// id is taken.
got := s.report(waf.Report{RequestID: id, ResponseBytes: 1000})
got := s.report(wafprotocol.Report{RequestID: id, ResponseBytes: 1000})
wantStatus(t, got, http.StatusNoContent)
// This would wait on the same lookup were the id not taken.
@@ -496,7 +529,7 @@ func TestReportNeverReachesAQuestionStillBeingHandled(t *testing.T) {
select {
case got := <-answered:
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
case <-time.After(waitLimit):
t.Fatal("the question got no answer")
}
@@ -515,7 +548,9 @@ func TestReportNeverReachesAQuestionStillBeingHandled(t *testing.T) {
}
// The true report comes after, and is the one that counts.
got = s.report(waf.Report{RequestID: id, ResponseBytes: answerBytes})
got = s.report(
wafprotocol.Report{RequestID: id, ResponseBytes: answerBytes},
)
wantStatus(t, got, http.StatusNoContent)
if history := s.history(); history.ResponseBytes != answerBytes {
@@ -533,17 +568,17 @@ func TestPeerAbsentOrUnreadableIsAnsweredAndLoggedInvalid(t *testing.T) {
s := startWAF(t, "", nil)
req := s.question(siteA, client, "/")
req.Header.Del(waf.PeerHeader)
req.Header.Del(wafprotocol.PeerHeader)
if peer != "" {
req.Header.Set(waf.PeerHeader, peer)
req.Header.Set(wafprotocol.PeerHeader, peer)
}
wantDisposition(
t,
do(t, req),
http.StatusOK,
waf.DispositionAllow,
wafprotocol.DispositionAllow,
)
if line := s.out.requestLine(t); line.PeerIP != "invalid IP" {
@@ -571,12 +606,12 @@ func TestContentLengthUnreadableOrNegativeCountsAsAbsent(t *testing.T) {
})
req := s.question(siteA, client, "/")
req.Header.Set(waf.ContentLengthHeader, announced)
req.Header.Set(wafprotocol.ContentLengthHeader, announced)
wantDisposition(
t,
do(t, req),
http.StatusOK,
waf.DispositionAllow,
wafprotocol.DispositionAllow,
)
if line := s.out.requestLine(t); line.ContentLength != 0 {
@@ -595,13 +630,18 @@ func TestHeaderSentTwiceUsesItsFirstValue(t *testing.T) {
s := startWAF(t, "", nil)
req := s.question(siteA, client, "/")
req.Header.Add(waf.ClientHeader, otherClient)
req.Header.Add(waf.PeerHeader, "172.18.0.3")
req.Header.Add(wafprotocol.ClientHeader, otherClient)
req.Header.Add(wafprotocol.PeerHeader, "172.18.0.3")
req.Header.Set(requestIDHeader, "first")
req.Header.Add(requestIDHeader, "second")
req.Header.Set(waf.ContentLengthHeader, "1234")
req.Header.Add(waf.ContentLengthHeader, "5678")
wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow)
req.Header.Set(wafprotocol.ContentLengthHeader, "1234")
req.Header.Add(wafprotocol.ContentLengthHeader, "5678")
wantDisposition(
t,
do(t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
)
line := s.out.requestLine(t)
if line.ClientIP != client || line.PeerIP != rproxyPeer ||
@@ -630,7 +670,7 @@ func sendQuestionWithPartOfItsBody(t *testing.T, conn net.Conn) {
_, err := fmt.Fprintf(conn,
"POST / HTTP/1.1\r\nHost: %s\r\n%s: %s\r\n%s: an-id\r\n"+
"Content-Length: 100\r\n\r\npart",
siteA, waf.ClientHeader, client, requestIDHeader)
siteA, wafprotocol.ClientHeader, client, requestIDHeader)
if err != nil {
t.Fatalf("send the question: %v", err)
}
@@ -699,7 +739,7 @@ func TestQuestionWhoseBodyStopsArrivingGetsNoAnswer(t *testing.T) {
t,
do(t, again),
http.StatusOK,
waf.DispositionAllow,
wafprotocol.DispositionAllow,
)
})
}
@@ -717,7 +757,7 @@ func TestQuestionDeniedGetsTheBanResponse(t *testing.T) {
})
got := s.ask(client, "/")
wantDisposition(t, got, status, waf.DispositionDeny)
wantDisposition(t, got, status, wafprotocol.DispositionDeny)
wantBody(t, got, http.StatusText(status)+"\n")
wantLine(
t,
@@ -737,7 +777,7 @@ func TestBanResponseCloseIsAnAnswerOfItsOwn(t *testing.T) {
})
got := s.ask(client, "/")
wantDisposition(t, got, http.StatusOK, waf.DispositionClose)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionClose)
wantBody(t, got, "")
// The visitor is sent nothing.
wantLine(t, s.out.requestLine(t), 0, requestlog.ActionClientDenied)
@@ -761,12 +801,22 @@ func TestBlockRuleIsDeniedWith403AndABanRuleWithTheBanResponse(
})
got := s.ask(client, "/blocked")
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
wantLine(t, s.out.requestLine(t), http.StatusForbidden,
requestlog.ActionBlockedByBlockRule)
got = s.ask(client, "/.env")
wantDisposition(t, got, http.StatusTooManyRequests, waf.DispositionDeny)
wantDisposition(
t,
got,
http.StatusTooManyRequests,
wafprotocol.DispositionDeny,
)
wantLine(t, s.out.requestLines(t, 2)[1], http.StatusTooManyRequests,
requestlog.ActionTemporarilyBanned)
}
@@ -781,10 +831,15 @@ func TestQuestionUnderTheOwnPrefixIsDeniedWith404AfterTheChecks(
})
for _, path := range []string{
waf.HealthPath, waf.MetricsPath, waf.ReportPath,
wafprotocol.HealthPath, waf.MetricsPath, wafprotocol.ReportPath,
} {
got := s.ask(client, path)
wantDisposition(t, got, http.StatusNotFound, waf.DispositionDeny)
wantDisposition(
t,
got,
http.StatusNotFound,
wafprotocol.DispositionDeny,
)
}
wantLine(
@@ -795,7 +850,12 @@ func TestQuestionUnderTheOwnPrefixIsDeniedWith404AfterTheChecks(
)
got := s.ask(otherClient, waf.MetricsPath)
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
}
func TestObserveModeAllowsAndLogsWhatEnforceModeWouldHaveDone(t *testing.T) {
@@ -804,7 +864,7 @@ func TestObserveModeAllowsAndLogsWhatEnforceModeWouldHaveDone(t *testing.T) {
s := startWAF(t, "", map[string]string{mode: observe, denyNets: client})
got := s.ask(client, "/")
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
line := s.out.requestLine(t)
wantLine(t, line, http.StatusOK, requestlog.ActionForward)
@@ -822,7 +882,7 @@ func TestObserveModeRefusalAnsweredAllowCountsForTheRefusalLimit(
for range 2 {
got := s.ask(client, "/blocked")
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
}
lines := s.out.requestLines(t, 2)
@@ -846,7 +906,12 @@ func TestDeniedQuestionCountsForTheRefusalLimit(t *testing.T) {
for range 2 {
got := s.ask(client, "/blocked")
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
}
lines := s.out.requestLines(t, 2)
@@ -882,7 +947,12 @@ func TestAllowCarriesTheLookupHeadersWhileTheSettingIsSet(t *testing.T) {
})
got := s.ask(fromDE, "/")
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
wantDisposition(
t,
got,
http.StatusOK,
wafprotocol.DispositionAllow,
)
asn, country := got.header.Get(
"X-Client-Asn",
@@ -916,7 +986,7 @@ func TestReportCountsTowardAByteLimitWhichThenBans(t *testing.T) {
}
// 100 bytes are over the limit of 99.
got := s.report(waf.Report{
got := s.report(wafprotocol.Report{
RequestID: id, Status: http.StatusBadGateway,
RequestBytes: bodyBytes, ResponseBytes: answerBytes,
})
@@ -940,7 +1010,12 @@ func TestReportCountsTowardAByteLimitWhichThenBans(t *testing.T) {
}
got = s.ask(client, "/")
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
}
func TestReportCountsAtThePercentageItsQuestionWasGiven(t *testing.T) {
@@ -958,12 +1033,12 @@ func TestReportCountsAtThePercentageItsQuestionWasGiven(t *testing.T) {
t,
s.ask(fromKP, "/"),
http.StatusOK,
waf.DispositionAllow,
wafprotocol.DispositionAllow,
)
s.passedOn(fromDE)
wantDisposition(t, s.ask(fromDE, "/"), http.StatusForbidden,
waf.DispositionDeny)
wafprotocol.DispositionDeny)
}
func TestReportAddsTheRequestToItsClientsHistoryAndTheAnomalyCounters(
@@ -989,7 +1064,7 @@ func TestReportAddsTheRequestToItsClientsHistoryAndTheAnomalyCounters(
)
}
got := s.report(waf.Report{
got := s.report(wafprotocol.Report{
RequestID: id, Status: http.StatusBadGateway,
RequestBytes: bodyBytes, ResponseBytes: answerBytes,
})
@@ -1021,7 +1096,7 @@ func TestDeniedRequestIsFinishedAtItsDecision(t *testing.T) {
t,
do(t, denied),
http.StatusForbidden,
waf.DispositionDeny,
wafprotocol.DispositionDeny,
)
counters := s.server.Anomalies.Snapshot()
@@ -1039,7 +1114,7 @@ func TestDeniedRequestIsFinishedAtItsDecision(t *testing.T) {
wantMetric(t, s.metrics(), series, 1)
got := s.report(waf.Report{
got := s.report(wafprotocol.Report{
RequestID: denied.Header.Get(
requestIDHeader,
), Status: http.StatusForbidden,
@@ -1082,7 +1157,7 @@ func TestMetricsCountAnAllowedRequestFromItsReport(t *testing.T) {
1,
) // the scrape itself
got := s.report(waf.Report{
got := s.report(wafprotocol.Report{
RequestID: id, Status: http.StatusBadGateway,
RequestBytes: bodyBytes, ResponseBytes: answerBytes,
DurationTotalMS: 5, DurationUpstreamTotalMS: 4,
@@ -1100,7 +1175,7 @@ func TestMetricsCountAnAllowedRequestFromItsReport(t *testing.T) {
// check needs a server that has not been scraped yet.
unscraped := startWAF(t, "", map[string]string{metricsToken: token})
got = unscraped.report(waf.Report{
got = unscraped.report(wafprotocol.Report{
RequestID: unscraped.allowed(client), Status: http.StatusBadGateway,
DurationTotalMS: 5,
})
@@ -1118,7 +1193,7 @@ func TestReportForARequestNotHeldIsIgnored(t *testing.T) {
s := startWAF(t, "", map[string]string{bytesLimitPerMinute: byteLimit})
got := s.report(waf.Report{
got := s.report(wafprotocol.Report{
RequestID: "no-such-request", ResponseBytes: 1000,
})
wantStatus(t, got, http.StatusNoContent)
@@ -1137,7 +1212,9 @@ func TestReportForARequestNotHeldIsIgnored(t *testing.T) {
// A second report of the same request does not count it twice.
id := s.allowed(client)
for range 2 {
got = s.report(waf.Report{RequestID: id, ResponseBytes: answerBytes})
got = s.report(
wafprotocol.Report{RequestID: id, ResponseBytes: answerBytes},
)
wantStatus(t, got, http.StatusNoContent)
}
@@ -1347,7 +1424,7 @@ func TestBytesAtTheirMostEachWayBanRatherThanOverflow(t *testing.T) {
s := startWAF(t, "", map[string]string{bytesLimitPerMinute: byteLimit})
got := s.report(waf.Report{
got := s.report(wafprotocol.Report{
RequestID: s.allowed(client), Status: http.StatusOK,
RequestBytes: reportedBytesLimit, ResponseBytes: reportedBytesLimit,
})
@@ -1441,9 +1518,14 @@ func TestReportWritesTheLastLineWithTheDecisionsAndTheReportsFields(
req.Header.Set("X-Forwarded-Proto", "https")
id := req.Header.Get(requestIDHeader)
wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow)
wantDisposition(
t,
do(t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
)
got := s.report(waf.Report{
got := s.report(wafprotocol.Report{
RequestID: id, Status: http.StatusCreated,
RequestBytes: bodyBytes, ResponseBytes: answerBytes,
DurationTotalMS: 12.5, DurationUpstreamTotalMS: 10.25,
@@ -1490,7 +1572,7 @@ func TestEveryDeniedRequestsReportGetsItsLastLine(t *testing.T) {
"close", map[string]string{denyNets: client, banResponse: banResponseClose}, "/",
0, requestlog.ActionClientDenied,
},
{"own prefix", nil, waf.HealthPath, http.StatusNotFound,
{"own prefix", nil, wafprotocol.HealthPath, http.StatusNotFound,
requestlog.ActionOwnEndpoint},
} {
t.Run(tc.name, func(t *testing.T) {
@@ -1502,7 +1584,7 @@ func TestEveryDeniedRequestsReportGetsItsLastLine(t *testing.T) {
id := req.Header.Get(requestIDHeader)
do(t, req)
got := s.report(waf.Report{
got := s.report(wafprotocol.Report{
RequestID: id, Status: tc.status, ResponseBytes: 8, DurationTotalMS: 3,
})
wantStatus(t, got, http.StatusNoContent)
@@ -1542,13 +1624,13 @@ func TestOldestAnsweredRequestIsDroppedPastTheBound(t *testing.T) {
t,
readAnswer(res),
http.StatusOK,
waf.DispositionAllow,
wafprotocol.DispositionAllow,
)
}
// The table is full again with next and those, without oldest.
got := s.report(
waf.Report{RequestID: oldest, ResponseBytes: answerBytes},
wafprotocol.Report{RequestID: oldest, ResponseBytes: answerBytes},
)
wantStatus(t, got, http.StatusNoContent)
@@ -1559,7 +1641,9 @@ func TestOldestAnsweredRequestIsDroppedPastTheBound(t *testing.T) {
)
}
got = s.report(waf.Report{RequestID: next, ResponseBytes: answerBytes})
got = s.report(
wafprotocol.Report{RequestID: next, ResponseBytes: answerBytes},
)
wantStatus(t, got, http.StatusNoContent)
if counted := s.history().Requests; counted != 1 {
@@ -1580,10 +1664,20 @@ func TestBanMadeThroughOneHostRefusesTheClientOnAnother(t *testing.T) {
)
got := do(t, s.question(siteA, client, "/.env"))
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
got = do(t, s.question(siteB, client, "/"))
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
lines := s.out.requestLines(t, 2)
if lines[0].Host != siteA || lines[1].Host != siteB {
@@ -1621,7 +1715,12 @@ func TestRequestsOnEverySiteCountTowardOneSetOfLimits(t *testing.T) {
s.allowed(client)
got := do(t, s.question(siteB, client, "/"))
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
}
func TestOwnEndpointsAnswerOnlyWithoutTheClientHeader(t *testing.T) {
@@ -1636,13 +1735,13 @@ func TestOwnEndpointsAnswerOnlyWithoutTheClientHeader(t *testing.T) {
path, authorization string
want int
}{
{waf.HealthPath, "", http.StatusOK},
{wafprotocol.HealthPath, "", http.StatusOK},
{waf.MetricsPath, bearer, http.StatusOK},
{waf.MetricsPath, "", http.StatusUnauthorized},
{waf.BansPath, adminBearer, http.StatusOK},
{waf.BansPath, "", http.StatusUnauthorized},
{waf.ClientsPath + client, adminBearer, http.StatusOK},
{waf.ReportPath, "", http.StatusNotFound},
{wafprotocol.ReportPath, "", http.StatusNotFound},
{"/_sww/other", "", http.StatusNotFound},
{"/", "", http.StatusNotFound},
} {
@@ -1659,10 +1758,10 @@ func TestOwnEndpointsAnswerOnlyWithoutTheClientHeader(t *testing.T) {
t,
do(t, req),
http.StatusNotFound,
waf.DispositionDeny,
wafprotocol.DispositionDeny,
)
} else {
wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow)
wantDisposition(t, do(t, req), http.StatusOK, wafprotocol.DispositionAllow)
}
}
}
@@ -1672,11 +1771,11 @@ func TestOwnEndpointsLinesDoNotSayWhichOfTheTwoTheyAre(t *testing.T) {
s := startWAF(t, "", nil)
wantStatus(t, get(t, s.addr, waf.HealthPath), http.StatusOK)
wantStatus(t, get(t, s.addr, wafprotocol.HealthPath), http.StatusOK)
line := s.out.requestLine(t)
if _, there := line.fields["waf_line"]; there ||
line.Path != waf.HealthPath {
line.Path != wafprotocol.HealthPath {
t.Errorf("the health check's log line is %+v", line.fields)
}
}
@@ -1688,7 +1787,7 @@ func TestALargeBodyAnnouncedAndLongHeadersAreNoReasonToRefuse(t *testing.T) {
req := s.questionWith(http.MethodPost, siteA, client, "/",
strings.NewReader("a body"))
req.Header.Set(waf.ContentLengthHeader, "1073741824")
req.Header.Set(wafprotocol.ContentLengthHeader, "1073741824")
req.Header.Set(forwardedFor, otherClient)
// 40,000 bytes of headers, over rproxy's 32 KiB, in lines within the
@@ -1698,7 +1797,7 @@ func TestALargeBodyAnnouncedAndLongHeadersAreNoReasonToRefuse(t *testing.T) {
}
got := do(t, req)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
line := s.out.requestLine(t)
if line.ClientIP != client || line.ContentLength != 1<<30 {
@@ -1776,7 +1875,7 @@ carried header:Content-Length block ^6$
strings.NewReader("a body"))
if tc.announced != "" {
req.Header.Set(waf.ContentLengthHeader, tc.announced)
req.Header.Set(wafprotocol.ContentLengthHeader, tc.announced)
}
wantStatus(t, do(t, req), tc.want)
@@ -1794,19 +1893,24 @@ func TestCoreRuleSetReadsTheBodyAQuestionCarries(t *testing.T) {
req := s.questionWith(http.MethodPost, siteA, client, "/",
strings.NewReader(body))
req.Header.Set("Content-Type", contentType)
req.Header.Set(waf.ContentLengthHeader, "100000")
req.Header.Set(wafprotocol.ContentLengthHeader, "100000")
return do(t, req)
}
got := ask(formData, "id=1'%20OR%20'1'='1")
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
// The first 8 KiB of a longer JSON body cannot be parsed, which is no
// attack, so the Core Rule Set does not read it.
cut := `{"text":"` + strings.Repeat("x", 8<<10)
got = ask("application/json", cut)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
}
func TestCoreRuleSetAlertNamesTheSite(t *testing.T) {
@@ -1815,7 +1919,12 @@ func TestCoreRuleSetAlertNamesTheSite(t *testing.T) {
s := startWAF(t, "", map[string]string{coreRuleSetMode: block})
got := s.ask(client, sqlInjection)
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
waiting := s.queue.Snapshot().Waiting[alerts.DestinationWebhook]
if len(waiting) != 1 ||
@@ -1836,7 +1945,7 @@ func TestReputationHitAlertNamesTheSite(t *testing.T) {
loadLists(t, s.server, map[string][]string{dropURL: {client}})
wantDisposition(t, s.ask(client, "/"), http.StatusForbidden,
waf.DispositionDeny)
wafprotocol.DispositionDeny)
waiting := s.queue.Snapshot().Waiting[alerts.DestinationWebhook]
if len(waiting) != 1 || waiting[0].Event != alerts.EventReputationHit ||
+46
View File
@@ -0,0 +1,46 @@
package wafprotocol
import "net/http"
// Answer is what waf's answer to a question tells rproxy in its headers.
// The status, the other headers and the body of an answer of deny are what
// the visitor gets.
type Answer struct {
// Disposition is what rproxy does with the request: DispositionAllow,
// DispositionDeny or DispositionClose.
Disposition string
// ASN and Country are the client's AS number and country, each empty
// when the answer does not give it.
ASN, Country string
}
// SetAnswerHeaders sets the headers of waf's answer that tell rproxy answer,
// leaving out an ASN or Country that is empty.
func SetAnswerHeaders(header http.Header, answer Answer) {
header.Set(DispositionHeader, answer.Disposition)
if answer.ASN != "" {
header.Set(ASNHeader, answer.ASN)
}
if answer.Country != "" {
header.Set(CountryHeader, answer.Country)
}
}
// ReadAnswer returns what the headers of waf's answer tell rproxy, and
// whether they give a valid disposition.
func ReadAnswer(header http.Header) (Answer, bool) {
disposition := header.Get(DispositionHeader)
switch disposition {
case DispositionAllow, DispositionDeny, DispositionClose:
return Answer{
Disposition: disposition,
ASN: header.Get(ASNHeader),
Country: header.Get(CountryHeader),
}, true
}
return Answer{}, false
}
+189
View File
@@ -0,0 +1,189 @@
package wafprotocol
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"net/http"
"net/netip"
"net/url"
"strconv"
"unicode/utf8"
)
// maxRequestIDBytes is the longest id a question may give.
const maxRequestIDBytes = 128
// ErrBodyStoppedArriving is the error ReadQuestion wraps when the question's
// body stops arriving. Every other error it returns is why waf refuses the
// question.
var ErrBodyStoppedArriving = errors.New("the body stopped arriving")
// Why waf refuses a question. Each is also the body of the refusal.
var (
errClientNotAnAddress = errors.New(
ClientHeader + " is not an address without a zone",
)
errRequestIDNotValid = errors.New(RequestIDHeader + " is not a valid id")
errBodyTooLarge = errors.New("the body is over 64 KiB")
)
// QuestionParams are what NewQuestion needs.
type QuestionParams struct {
// Visitor is the visitor's request.
Visitor *http.Request
// PathAsSent is Visitor's path as the visitor sent it, which the question
// sends unchanged.
PathAsSent string
// Body is the part of Visitor's body the question carries, at most
// QuestionBodyMaxBytes.
Body []byte
// WAFBackend is waf's address. The question takes its scheme and host.
WAFBackend *url.URL
// Client is the visitor's address, and Peer that of rproxy's own TCP
// peer.
Client, Peer netip.Addr
// RequestID is the request's id.
RequestID string
// Scheme is how the visitor reached rproxy, or the trusted proxy in front
// of it.
Scheme string
}
// Question is a question as waf reads it.
type Question struct {
// Client is the visitor's address.
Client netip.Addr
// Peer is the address of rproxy's own TCP peer. It is not valid when the
// question gives none that can be read.
Peer netip.Addr
// RequestID is the request's id.
RequestID string
// Visitor is the visitor's request, as asTheVisitorSentIt gives it.
Visitor *http.Request
}
// NewQuestion returns the question about a visitor's request, as "The
// question" in SPEC.md gives it: the request as the visitor sent it, with the
// part of the body it carries and rproxy's headers.
func NewQuestion(
ctx context.Context,
params QuestionParams,
) (*http.Request, error) {
in := params.Visitor
question, err := http.NewRequestWithContext(
ctx,
in.Method,
params.WAFBackend.String(),
bytes.NewReader(params.Body),
)
if err != nil {
return nil, fmt.Errorf("make the question: %w", err)
}
backend := question.URL
target := *in.URL
target.Scheme = backend.Scheme
target.Host = backend.Host
target.Opaque = params.PathAsSent
question.URL = &target
question.Host = in.Host
// The length NewRequestWithContext set is that of the body carried. This
// line makes a body whose length the visitor did not announce go chunked,
// as "The question" in SPEC.md requires.
question.TransferEncoding = in.TransferEncoding
question.Header = in.Header.Clone()
question.Header.Set(ClientHeader, params.Client.String())
question.Header.Set(PeerHeader, params.Peer.String())
question.Header.Del(ContentLengthHeader)
announced := in.Header.Get("Content-Length")
if announced != "" {
question.Header.Set(ContentLengthHeader, announced)
}
question.Header.Set(RequestIDHeader, params.RequestID)
question.Header.Set("X-Forwarded-Proto", params.Scheme)
// Go's client adds a User-Agent of its own to a request without the key.
_, sent := question.Header["User-Agent"]
if !sent {
question.Header["User-Agent"] = nil
}
return question, nil
}
// ReadQuestion reads the question r, with the body it carries. An error
// wrapping ErrBodyStoppedArriving means the body stopped arriving. Any other
// error is why waf refuses the question, and the question's body is then not
// read. The body is read whole, since Go's server does not send an answer
// until it has read the body.
func ReadQuestion(w http.ResponseWriter, r *http.Request) (Question, error) {
client, err := netip.ParseAddr(r.Header.Get(ClientHeader))
if err != nil || client.Zone() != "" {
return Question{}, errClientNotAnAddress
}
id := r.Header.Get(RequestIDHeader)
if id == "" || len(id) > maxRequestIDBytes || !utf8.ValidString(id) {
return Question{}, errRequestIDNotValid
}
body, err := io.ReadAll(
http.MaxBytesReader(w, r.Body, QuestionBodyMaxBytes),
)
if _, tooLarge := errors.AsType[*http.MaxBytesError](err); tooLarge {
return Question{}, errBodyTooLarge
}
if err != nil {
return Question{}, fmt.Errorf("%w: %w", ErrBodyStoppedArriving, err)
}
// One that cannot be read leaves the log line's peer_ip invalid.
peer, _ := netip.ParseAddr(r.Header.Get(PeerHeader))
return Question{
Client: client.Unmap(),
Peer: peer.Unmap(),
RequestID: id,
Visitor: asTheVisitorSentIt(r, body),
}, nil
}
// asTheVisitorSentIt returns a copy of r, a question, as the checks and the
// log line are to see it: without the headers rproxy adds for waf,
// with body, which has been read of r, and with the length the visitor
// announced, as -1 when it announced none. The question's own length covers
// only the part of the body it carries.
func asTheVisitorSentIt(r *http.Request, body []byte) *http.Request {
sent := r.Clone(r.Context())
announced := sent.Header.Get(ContentLengthHeader)
sent.Header.Del(ClientHeader)
sent.Header.Del(PeerHeader)
sent.Header.Del(ContentLengthHeader)
sent.Header.Del("Content-Length")
sent.ContentLength = -1
length, err := strconv.ParseInt(announced, 10, 64)
if err == nil && length >= 0 {
sent.ContentLength = length
sent.Header.Set("Content-Length", announced)
}
sent.Body = http.NoBody
if len(body) > 0 {
sent.Body = io.NopCloser(bytes.NewReader(body))
}
return sent
}
+123
View File
@@ -0,0 +1,123 @@
package wafprotocol
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
)
const (
// reportBodyMaxBytes is the most of a report's body that is read. It is
// far above any report's.
reportBodyMaxBytes = 1 << 20
// maxReportedBytes is the most bytes a report may give for each way, 1
// PiB, so that adding them up cannot overflow.
maxReportedBytes = 1 << 50
// maxReportedMilliseconds is the longest duration a report may give,
// about 31 years, so that it fits a time.Duration.
maxReportedMilliseconds = 1e12
)
var (
errReportFieldMissing = errors.New("a field is missing")
errReportValue = errors.New("a value is out of range")
)
// Report is the JSON object rproxy posts to ReportPath once the visitor's
// answer has ended, its fields named as the request log's of the same
// meaning. RequestID is the question's id. Status is what the visitor got, 0
// if nothing. The durations are in milliseconds.
//
//nolint:tagliatelle // docs/request-log.md names the fields in snake_case
type Report struct {
RequestID string `json:"request_id"`
Status int `json:"status"`
RequestBytes int64 `json:"request_bytes"`
ResponseBytes int64 `json:"response_bytes"`
DurationTotalMS float64 `json:"duration_total_ms"`
DurationUpstreamTotalMS float64 `json:"duration_upstream_total_ms"`
}
// NewReportRequest returns the request that posts report to reportURL, which
// is waf's ReportPath.
func NewReportRequest(
ctx context.Context,
reportURL string,
report *Report,
) (*http.Request, error) {
body, err := json.Marshal(report)
if err != nil {
return nil, fmt.Errorf("encode the report: %w", err)
}
req, err := http.NewRequestWithContext(
ctx, http.MethodPost, reportURL, bytes.NewReader(body))
if err != nil {
return nil, fmt.Errorf("make the request: %w", err)
}
req.Header.Set("Content-Type", "application/json")
return req, nil
}
// ReadReport reads a report from r's body, which must be one JSON object with
// all of Report's fields, in the ranges SPEC.md gives.
func ReadReport(w http.ResponseWriter, r *http.Request) (Report, error) {
var report Report
body, err := io.ReadAll(
http.MaxBytesReader(w, r.Body, reportBodyMaxBytes),
)
if err != nil {
return report, fmt.Errorf("read the body: %w", err)
}
var fields map[string]json.RawMessage
err = json.Unmarshal(body, &fields)
if err != nil {
return report, fmt.Errorf("decode the body: %w", err)
}
for _, name := range []string{
"request_id", "status", "request_bytes", "response_bytes",
"duration_total_ms", "duration_upstream_total_ms",
} {
value, there := fields[name]
if !there || string(value) == "null" {
return report, fmt.Errorf("%w: %s", errReportFieldMissing, name)
}
}
err = json.Unmarshal(body, &report)
if err != nil {
return report, fmt.Errorf("decode the body: %w", err)
}
return report, report.checkRanges()
}
// checkRanges returns an error for a value of the report that cannot be true.
func (report *Report) checkRanges() error {
switch {
case report.Status != 0 &&
(report.Status < http.StatusContinue || report.Status > ReportStatusMax):
return fmt.Errorf("%w: status", errReportValue)
case report.RequestBytes < 0 || report.RequestBytes > maxReportedBytes:
return fmt.Errorf("%w: request_bytes", errReportValue)
case report.ResponseBytes < 0 || report.ResponseBytes > maxReportedBytes:
return fmt.Errorf("%w: response_bytes", errReportValue)
case report.DurationTotalMS < 0 || report.DurationTotalMS > maxReportedMilliseconds:
return fmt.Errorf("%w: duration_total_ms", errReportValue)
case report.DurationUpstreamTotalMS < 0 ||
report.DurationUpstreamTotalMS > maxReportedMilliseconds:
return fmt.Errorf("%w: duration_upstream_total_ms", errReportValue)
}
return nil
}
+61
View File
@@ -0,0 +1,61 @@
// Package wafprotocol holds the three messages rproxy and waf send each
// other, as "`waf`" in SPEC.md gives them. rproxy asks waf a question about
// each visitor's request, and waf answers it. Once the visitor's answer has
// ended, rproxy sends waf a report. This package writes and reads each
// message, and declares every name and limit both sides must agree on.
package wafprotocol
// The headers of the question and of the answer. Go writes every header name
// in this form, as it sends it and as it receives it, so X-Client-ASN arrives
// as X-Client-Asn; header names are not case-sensitive.
const (
// ClientHeader marks a request as rproxy's question, and gives the
// visitor's address.
ClientHeader = "X-Sww-Client"
// PeerHeader gives the address of rproxy's own TCP peer.
PeerHeader = "X-Sww-Peer"
// ContentLengthHeader gives the body length the visitor announced.
ContentLengthHeader = "X-Sww-Content-Length"
// RequestIDHeader gives the request's id, which rproxy makes. rproxy
// also sets it on the request it passes to the app.
RequestIDHeader = "X-Request-ID"
// DispositionHeader gives waf's answer.
DispositionHeader = "X-Sww-Disposition"
// ASNHeader and CountryHeader give the client's AS number and country in
// an answer of allow. rproxy passes them to the app.
ASNHeader = "X-Client-Asn"
CountryHeader = "X-Client-Country"
)
// The values of DispositionHeader.
const (
// DispositionAllow tells rproxy to pass the request to the app.
DispositionAllow = "allow"
// DispositionDeny tells rproxy to give the visitor the answer's status,
// headers and body.
DispositionDeny = "deny"
// DispositionClose tells rproxy to close the visitor's connection
// without an answer, which SWW_WAF_BAN_RESPONSE asks for when it is close.
// waf closing the question's connection instead would read as
// waf being down, and rproxy would then let the request through.
DispositionClose = "close"
)
// OwnEndpointPrefix starts the path of every request for sww itself: waf's
// own endpoints and rproxy's health endpoint. The app never sees a path under
// it.
const OwnEndpointPrefix = "/_sww/"
// HealthPath is the health endpoint of waf and of rproxy, which the
// container's health check asks.
const HealthPath = "/_sww/healthz"
// ReportPath is where rproxy sends the report of each request waf
// answered.
const ReportPath = "/_sww/report"
// QuestionBodyMaxBytes is the most of a visitor's body a question carries.
const QuestionBodyMaxBytes = 64 << 10
// ReportStatusMax is the highest status a report may give.
const ReportStatusMax = 599
+204
View File
@@ -0,0 +1,204 @@
package wafprotocol_test
import (
"io"
"net/http"
"net/http/httptest"
"net/netip"
"net/url"
"strings"
"testing"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// requestID is the id the tests' questions and reports give.
const requestID = "an-id"
// questionRead is what the server startQuestionReader starts read of a
// question: the question, and the body it carries.
type questionRead struct {
question wafprotocol.Question
body string
}
// startQuestionReader starts a server that reads each question it gets with
// ReadQuestion. It returns the server's URL and the channel what it reads goes
// to.
func startQuestionReader(t *testing.T) (*url.URL, <-chan questionRead) {
t.Helper()
read := make(chan questionRead, 1)
server := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
question, err := wafprotocol.ReadQuestion(w, r)
if err != nil {
t.Errorf("read the question: %v", err)
}
var body []byte
if question.Visitor != nil {
body, _ = io.ReadAll(question.Visitor.Body)
}
read <- questionRead{question, string(body)}
},
))
t.Cleanup(server.Close)
serverURL, err := url.Parse(server.URL)
if err != nil {
t.Fatalf("parse the server's URL: %v", err)
}
return serverURL, read
}
// TestQuestionReadsAsWritten sends a question NewQuestion writes over HTTP,
// and checks that ReadQuestion gives back the visitor's request and what
// rproxy added.
func TestQuestionReadsAsWritten(t *testing.T) {
t.Parallel()
const (
target = "/a%2Fb?q=1"
body = "name=value"
held = "name"
)
client := netip.MustParseAddr("2001:db8::1")
peer := netip.MustParseAddr("192.0.2.1")
backend, read := startQuestionReader(t)
visitor := httptest.NewRequestWithContext(
t.Context(),
http.MethodPost,
"http://site.example.com"+target,
strings.NewReader(body),
)
visitor.Header.Set("Content-Length", "10")
visitor.Header.Set("User-Agent", "a visitor")
visitor.Header.Set(wafprotocol.PeerHeader, "the visitor's own")
question, err := wafprotocol.NewQuestion(
t.Context(),
wafprotocol.QuestionParams{
Visitor: visitor,
PathAsSent: "/a%2Fb",
Body: []byte(held),
WAFBackend: backend,
Client: client,
Peer: peer,
RequestID: requestID,
Scheme: "https",
},
)
if err != nil {
t.Fatalf("write the question: %v", err)
}
res, err := http.DefaultTransport.RoundTrip(question)
if err != nil {
t.Fatalf("send the question: %v", err)
}
_ = res.Body.Close()
got := <-read
if got.question.Visitor == nil {
t.FailNow()
}
if got.question.Client != client || got.question.Peer != peer ||
got.question.RequestID != requestID {
t.Errorf("read %+v", got.question)
}
sent := got.question.Visitor
if sent.Host != "site.example.com" || sent.RequestURI != target ||
sent.ContentLength != int64(len(body)) || got.body != held {
t.Errorf(
"read %s at %s, of length %d, carrying %q",
sent.RequestURI, sent.Host, sent.ContentLength, got.body,
)
}
for name, want := range map[string]string{
"Content-Length": "10",
"User-Agent": "a visitor",
"X-Forwarded-Proto": "https",
wafprotocol.RequestIDHeader: requestID,
wafprotocol.ClientHeader: "",
wafprotocol.PeerHeader: "",
wafprotocol.ContentLengthHeader: "",
} {
if sent.Header.Get(name) != want {
t.Errorf("%s is %q, want %q", name, sent.Header.Get(name), want)
}
}
}
// TestAnswerReadsAsWritten checks that ReadAnswer gives back what
// SetAnswerHeaders sets, and that headers without a valid disposition give
// nothing.
func TestAnswerReadsAsWritten(t *testing.T) {
t.Parallel()
for _, want := range []wafprotocol.Answer{
{Disposition: wafprotocol.DispositionAllow, ASN: "64496", Country: "DE"},
{Disposition: wafprotocol.DispositionAllow, Country: "DE"},
{Disposition: wafprotocol.DispositionDeny},
{Disposition: wafprotocol.DispositionClose},
} {
header := http.Header{}
wafprotocol.SetAnswerHeaders(header, want)
got, valid := wafprotocol.ReadAnswer(header)
if !valid || got != want {
t.Errorf("read %+v (valid: %t), want %+v", got, valid, want)
}
}
header := http.Header{}
header.Set(wafprotocol.DispositionHeader, "maybe")
header.Set(wafprotocol.ASNHeader, "64496")
got, valid := wafprotocol.ReadAnswer(header)
if valid || got != (wafprotocol.Answer{}) {
t.Errorf("read %+v (valid: %t), want nothing", got, valid)
}
}
// TestReportReadsAsWritten checks that ReadReport gives back the report
// NewReportRequest posts.
func TestReportReadsAsWritten(t *testing.T) {
t.Parallel()
want := wafprotocol.Report{
RequestID: requestID,
Status: http.StatusOK,
RequestBytes: 10,
ResponseBytes: 20,
DurationTotalMS: 1.5,
DurationUpstreamTotalMS: 0.5,
}
req, err := wafprotocol.NewReportRequest(
t.Context(),
"http://waf.example.com"+wafprotocol.ReportPath,
&want,
)
if err != nil {
t.Fatalf("write the report: %v", err)
}
if req.Method != http.MethodPost ||
req.URL.Path != wafprotocol.ReportPath {
t.Errorf("the report is %s %s", req.Method, req.URL.Path)
}
got, err := wafprotocol.ReadReport(httptest.NewRecorder(), req)
if err != nil || got != want {
t.Errorf("read %+v (%v), want %+v", got, err, want)
}
}