Compare commits
5
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 sneak
|
9c755ee358 | ||
|
sneak
|
f8676debda | ||
 clawbot
|
a41d61e9ad | ||
|
clawbot
|
84a9190d2b | ||
|
sneak
|
1bdf66cd21 |
@@ -123,9 +123,7 @@ runs steps 3 to 13, and does the counting of step 15. The first step that
|
||||
produces a final answer ends the checks. Step 15 runs for each request
|
||||
`servewaf` answers with `allow`, `deny` or `close`. `rproxy`'s size and time
|
||||
limits apply to the whole exchange. Each step names the section that holds its
|
||||
rules. Each check is a heuristic. Its name, in parentheses below, is the value
|
||||
of the request log's `heuristic` field. `refusal_limit` and `byte_limits` give
|
||||
theirs in `report_heuristic` instead.
|
||||
rules.
|
||||
|
||||
1. `GET /_smallwebwaf/healthz` is answered at once by `rproxy`, before any
|
||||
check, so a health checker never needs an exemption and is never refused.
|
||||
@@ -143,9 +141,9 @@ theirs in `report_heuristic` instead.
|
||||
around its address, a /64 by default, since one abuser usually controls a
|
||||
whole /64. A client is therefore one IPv4 address or one IPv6 group.
|
||||
3. The static lists. A client in `SWWAF_ALLOW_NETS` skips steps 4 to 12, the
|
||||
byte limits and the error burst. A client in `SWWAF_DENY_NETS` is refused
|
||||
(`deny_nets`); a client in both lists is let through.
|
||||
4. Bans. A client under a ban is refused (`active_ban`; see "Bans").
|
||||
byte limits and the error burst. A client in `SWWAF_DENY_NETS` is refused; a
|
||||
client in both lists is let through.
|
||||
4. Bans. A client under a ban is refused (see "Bans").
|
||||
5. The lookup, unless `SWWAF_LOOKUP_SOURCE` is `off` (see "Lookup of AS number
|
||||
and country"). The request goes on at once. It waits for the first GeoJS
|
||||
answer, up to `SWWAF_LOOKUP_TIMEOUT`, only when one of these needs it:
|
||||
@@ -155,26 +153,22 @@ theirs in `report_heuristic` instead.
|
||||
|
||||
A request that ends before the answer comes is logged without it.
|
||||
|
||||
6. The country lists (`countries`; see "Country lists").
|
||||
7. The reputation sources (see "Reputation"), in this order:
|
||||
- A client on a fetched blocklist gets `SWWAF_BLOCKLIST_ACTION`
|
||||
(`blocklists`).
|
||||
- A client on the CrowdSec decision list is banned (`crowdsec`).
|
||||
- A client listed by a DNS blocklist gets `SWWAF_REPUTATION_ACTION`
|
||||
(`dnsbl`).
|
||||
- A client scored at least `SWWAF_ABUSEIPDB_MIN_SCORE` by AbuseIPDB gets
|
||||
`SWWAF_REPUTATION_ACTION` (`abuseipdb`).
|
||||
6. The country lists (see "Country lists").
|
||||
7. The reputation sources (see "Reputation"). A client on a fetched blocklist
|
||||
gets `SWWAF_BLOCKLIST_ACTION`. A client on the CrowdSec decision list is
|
||||
banned. A client listed by a DNS blocklist, or scored at least
|
||||
`SWWAF_ABUSEIPDB_MIN_SCORE` by AbuseIPDB, gets `SWWAF_REPUTATION_ACTION`.
|
||||
8. The client's limit percentage (see "Biased thresholds").
|
||||
9. Rate limiting (`rate_limits`; see "Request rate limits, per client"). A
|
||||
request over a limit times the percentage is refused and bans the client.
|
||||
10. The trap paths (`trap_paths`; see "Attack detection"). A request for one is
|
||||
refused and bans the client.
|
||||
11. The rule files (`rule_files`; see "Rule files"). Matching stops at the first
|
||||
rule that refuses.
|
||||
12. The header limits (`header_limits`; see "Answering in time"), and then the
|
||||
Core Rule Set (`core_rule_set`; see "Attack detection"). A Core Rule Set
|
||||
match is refused with `403` while `SWWAF_WAF_MODE` is `block`, and only
|
||||
logged and alerted while it is `detect`.
|
||||
9. Rate limiting (see "Request rate limits, per client"). A request over a limit
|
||||
times the percentage is refused and bans the client.
|
||||
10. The trap paths (see "Attack detection"). A request for one is refused and
|
||||
bans the client.
|
||||
11. The rule files (see "Rule files"). Matching stops at the first rule that
|
||||
refuses.
|
||||
12. The header limits (see "Answering in time"), and then the Core Rule Set (see
|
||||
"Attack detection"). A Core Rule Set match is refused with `403` while
|
||||
`SWWAF_WAF_MODE` is `block`, and only logged and alerted while it is
|
||||
`detect`.
|
||||
13. A path under `/_smallwebwaf/` is answered `404`, so that the app never sees
|
||||
that prefix.
|
||||
14. `rproxy` passes the request to `SWRP_UPSTREAM_URL`, streaming (see "Passing
|
||||
@@ -188,14 +182,14 @@ theirs in `report_heuristic` instead.
|
||||
are added to the byte counters, as `SWWAF_BYTES_COUNT` says. For a
|
||||
WebSocket, what it carried each way counts once it closes. A client whose
|
||||
total passes a byte limit times the percentage has broken that limit and
|
||||
is banned (`byte_limits`). The response in progress is not cut off.
|
||||
is banned. The response in progress is not cut off.
|
||||
- For a request it allowed, the client's history and the anomaly counters
|
||||
are updated, and the alert thresholds are evaluated.
|
||||
|
||||
For a request it refused, `servewaf` adds nothing to the byte limits. It
|
||||
updates the history and the anomaly counters at its decision. It counts the
|
||||
refusals the error burst counts at the decision too (`refusal_limit`; see
|
||||
"Attack detection").
|
||||
refusals the error burst counts at the decision too (see "Attack
|
||||
detection").
|
||||
|
||||
In `SWWAF_MODE=observe`, every refusal above is logged as what would have
|
||||
happened, nothing is refused, and `servewaf` makes no ban of its own. A path
|
||||
|
||||
@@ -107,34 +107,6 @@ Every one of these lines has:
|
||||
- `would_action`: in `observe` mode, what `enforce` mode would have done with a
|
||||
request it would have refused. `action` then names what was done, normally
|
||||
`forward`.
|
||||
- `heuristic`: the heuristic that refused the request, in `observe` mode too. It
|
||||
is left out when none did.
|
||||
- `heuristic_figures`: an object of the figures that heuristic acted on. Each
|
||||
heuristic and its figures:
|
||||
- `deny_nets`: `netblock`, the entry of `SWWAF_DENY_NETS` the client is in;
|
||||
- `active_ban`: `netblock`, the ban's netblock, and `ban_expires`;
|
||||
- `countries`: `country`, and `setting`, the country list that refused it;
|
||||
- `blocklists`: `lists`, the blocklist URLs that list the client;
|
||||
- `crowdsec`: `scenario`, the decision's scenario, and `expires`, when the
|
||||
decision ends;
|
||||
- `dnsbl`: `zones`, the DNS blocklist zones that list the client, their keys
|
||||
masked;
|
||||
- `abuseipdb`: `score`, and `min_score`, `SWWAF_ABUSEIPDB_MIN_SCORE`;
|
||||
- `rate_limits`: `window`, `count` and `limit`;
|
||||
- `trap_paths`: `path`, the trap path asked for;
|
||||
- `rule_files`: `rule_id` and `rule_action`, of the rule that refused it;
|
||||
- `header_limits`: `fields`, `fields_limit`, `cookies` and `cookies_limit`;
|
||||
- `core_rule_set`: `score`, and `threshold`, `SWWAF_WAF_ANOMALY_THRESHOLD`;
|
||||
- `refusal_limit`: `window`, `count` and `limit`, of the error burst;
|
||||
- `byte_limits`: `window`, `count` and `limit`, the count in bytes.
|
||||
- `report_heuristic`: the heuristic that banned the client after `servewaf`
|
||||
answered the request, in `observe` mode too: `refusal_limit` or `byte_limits`.
|
||||
- `report_heuristic_figures`: an object of the figures that heuristic acted on,
|
||||
as listed above.
|
||||
|
||||
`byte_limits` bans when the report comes, so only the `report` line names
|
||||
it.
|
||||
|
||||
- `limit_percent` and `limit_percent_setting`: the client's percentage of the
|
||||
rate limits and the setting that lowered it, when one did. `bytes_percent` and
|
||||
`bytes_percent_setting` do the same for the byte limits.
|
||||
|
||||
@@ -10,12 +10,14 @@ require (
|
||||
github.com/maxmind/mmdbwriter v1.2.0
|
||||
github.com/oschwald/maxminddb-golang/v2 v2.7.0
|
||||
github.com/prometheus/client_golang v1.24.1
|
||||
github.com/spf13/viper v1.21.0
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/beorn7/perks v1.0.1 // indirect
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
github.com/corazawaf/libinjection-go v0.3.3 // indirect
|
||||
github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
|
||||
github.com/goccy/go-json v0.10.5 // indirect
|
||||
github.com/goccy/go-yaml v1.19.2 // indirect
|
||||
github.com/gotnospirit/makeplural v0.0.0-20180622080156-a5f48d94d976 // indirect
|
||||
@@ -25,14 +27,22 @@ require (
|
||||
github.com/kylelemons/godebug v1.1.0 // indirect
|
||||
github.com/magefile/mage v1.17.0 // indirect
|
||||
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
|
||||
github.com/pelletier/go-toml/v2 v2.2.4 // indirect
|
||||
github.com/petar-dambovaliev/aho-corasick v0.0.0-20250424160509-463d218d4745 // indirect
|
||||
github.com/prometheus/client_model v0.6.2 // indirect
|
||||
github.com/prometheus/common v0.70.1 // indirect
|
||||
github.com/prometheus/procfs v0.21.1 // indirect
|
||||
github.com/sagikazarmark/locafero v0.11.0 // indirect
|
||||
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 // indirect
|
||||
github.com/spf13/afero v1.15.0 // indirect
|
||||
github.com/spf13/cast v1.10.0 // indirect
|
||||
github.com/spf13/pflag v1.0.10 // indirect
|
||||
github.com/subosito/gotenv v1.6.0 // indirect
|
||||
github.com/tidwall/gjson v1.18.0 // indirect
|
||||
github.com/tidwall/match v1.1.1 // indirect
|
||||
github.com/tidwall/pretty v1.2.1 // indirect
|
||||
github.com/valllabh/ocsf-schema-golang v1.0.3 // indirect
|
||||
go.yaml.in/yaml/v3 v3.0.5 // indirect
|
||||
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba // indirect
|
||||
golang.org/x/net v0.58.0 // indirect
|
||||
golang.org/x/sync v0.23.0 // indirect
|
||||
|
||||
@@ -12,8 +12,12 @@ github.com/corazawaf/libinjection-go v0.3.3 h1:NhbXKRfRpqKzBMzv8zpCcnjyEw7BCVhBO
|
||||
github.com/corazawaf/libinjection-go v0.3.3/go.mod h1:Ik/+w3UmTWH9yn366RgS9D95K3y7Atb5m/H/gXzzPCk=
|
||||
github.com/foxcpp/go-mockdns v1.2.0 h1:omK3OrHRD1IWJz1FuFBCFquhXslXoF17OvBS6JPzZF0=
|
||||
github.com/foxcpp/go-mockdns v1.2.0/go.mod h1:IhLeSFGed3mJIAXPH2aiRQB+kqz7oqu8ld2qVbOu7Wk=
|
||||
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
|
||||
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
|
||||
github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho=
|
||||
github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo=
|
||||
github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
|
||||
github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
|
||||
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
|
||||
github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
|
||||
github.com/goccy/go-yaml v1.19.2 h1:PmFC1S6h8ljIz6gMRBopkjP1TVT7xuwrButHID66PoM=
|
||||
@@ -34,6 +38,10 @@ github.com/kaptinlin/jsonschema v0.4.6 h1:vOSFg5tjmfkOdKg+D6Oo4fVOM/pActWu/ntkPs
|
||||
github.com/kaptinlin/jsonschema v0.4.6/go.mod h1:1DUd7r5SdyB2ZnMtyB7uLv64dE3zTFTiYytDCd+AEL0=
|
||||
github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
|
||||
github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
|
||||
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
|
||||
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
|
||||
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
|
||||
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
|
||||
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
|
||||
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
|
||||
github.com/magefile/mage v1.17.0 h1:dS4tkq997Ism03akafC8509iqDjeE7TNTexI25Y7sXM=
|
||||
@@ -58,8 +66,24 @@ github.com/prometheus/common v0.70.1 h1:1HvjP4D5oL3t8RsPlwxA9onvvStjtIHYE5XuuwOi
|
||||
github.com/prometheus/common v0.70.1/go.mod h1:VdFUQDMZK3VLkurFUVhia6uys/0suUp86TJz5qbJRhc=
|
||||
github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+OJI=
|
||||
github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY=
|
||||
github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
|
||||
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
|
||||
github.com/sagikazarmark/locafero v0.11.0 h1:1iurJgmM9G3PA/I+wWYIOw/5SyBtxapeHDcg+AAIFXc=
|
||||
github.com/sagikazarmark/locafero v0.11.0/go.mod h1:nVIGvgyzw595SUSUE6tvCp3YYTeHs15MvlmU87WwIik=
|
||||
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 h1:+jumHNA0Wrelhe64i8F6HNlS8pkoyMv5sreGx2Ry5Rw=
|
||||
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8/go.mod h1:3n1Cwaq1E1/1lhQhtRK2ts/ZwZEhjcQeJQ1RuC6Q/8U=
|
||||
github.com/spf13/afero v1.15.0 h1:b/YBCLWAJdFWJTN9cLhiXXcD7mzKn9Dm86dNnfyQw1I=
|
||||
github.com/spf13/afero v1.15.0/go.mod h1:NC2ByUVxtQs4b3sIUphxK0NioZnmxgyCrfzeuq8lxMg=
|
||||
github.com/spf13/cast v1.10.0 h1:h2x0u2shc1QuLHfxi+cTJvs30+ZAHOGRic8uyGTDWxY=
|
||||
github.com/spf13/cast v1.10.0/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqeHo=
|
||||
github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
|
||||
github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
|
||||
github.com/spf13/viper v1.21.0 h1:x5S+0EU27Lbphp4UKm1C+1oQO+rKx36vfCoaVebLFSU=
|
||||
github.com/spf13/viper v1.21.0/go.mod h1:P0lhsswPGWD/1lZJ9ny3fYnVqxiegrlNrEmgLjbTCAY=
|
||||
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
|
||||
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
|
||||
github.com/subosito/gotenv v1.6.0 h1:9NlTDc1FTs4qu0DDq7AEtTPNw6SVm7uBMsUCUjABIf8=
|
||||
github.com/subosito/gotenv v1.6.0/go.mod h1:Dk4QP5c2W3ibzajGcXpNraDfq2IrhjMIvMSWPKKo0FU=
|
||||
github.com/tidwall/gjson v1.18.0 h1:FIDeeyB800efLX89e5a8Y0BNH+LOngJyGrIWxG2FKQY=
|
||||
github.com/tidwall/gjson v1.18.0/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
|
||||
github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=
|
||||
|
||||
+38
-18
@@ -26,6 +26,8 @@ import (
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/spf13/viper"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
||||
"sneak.berlin/go/smallwebwaf/internal/remotelog"
|
||||
@@ -450,15 +452,16 @@ var (
|
||||
"is in SWWAF_BLOCKLIST_URLS or is SWWAF_ASN_LIMIT_PERCENT_URL too")
|
||||
)
|
||||
|
||||
// FromEnvironment reads the settings with lookupEnv, normally
|
||||
// os.LookupEnv. A setting may instead be given as a file: the variable
|
||||
// named by the setting's name with _FILE added names the file, which is
|
||||
// read now (see lookup). A setting that is not set takes its default. A
|
||||
// setting that is set but invalid is an error that names it.
|
||||
// FromEnvironment reads the settings through v, normally the viper
|
||||
// NewViper returns, bound to the process's environment. A setting may
|
||||
// instead be given as a file: the variable named by the setting's name
|
||||
// with _FILE added names the file, which is read now (see lookup). A
|
||||
// setting that is not set takes its default. A setting that is set but
|
||||
// invalid is an error that names it.
|
||||
//
|
||||
//nolint:funlen // one line for each setting, a list that grows with them
|
||||
func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
|
||||
env := &environment{lookupEnv: lookupEnv}
|
||||
func FromEnvironment(v *viper.Viper) (*Config, error) {
|
||||
env := &environment{v: v, readFiles: true}
|
||||
cfg := &Config{
|
||||
ListenAddr: env.address("SWWAF_LISTEN_ADDR"),
|
||||
InstanceName: env.instanceName(),
|
||||
@@ -575,8 +578,8 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
|
||||
// Command reads only SMALLWEBWAF_COMMAND, the command the image runs and its
|
||||
// health check checks: rproxy when it is unset or empty, or servewaf. It cannot
|
||||
// be given as a file. Any other value is an error that names the setting.
|
||||
func Command(lookupEnv func(string) (string, bool)) (string, error) {
|
||||
command, _ := lookupEnv("SMALLWEBWAF_COMMAND")
|
||||
func Command(v *viper.Viper) (string, error) {
|
||||
command, _ := readSetting(v, "SMALLWEBWAF_COMMAND")
|
||||
|
||||
switch command {
|
||||
case "", "rproxy":
|
||||
@@ -591,8 +594,8 @@ func Command(lookupEnv func(string) (string, bool)) (string, error) {
|
||||
// ServeWAFListenAddr reads only SWWAF_LISTEN_ADDR, which may be given as a
|
||||
// file, as FromEnvironment does. The health check in servewaf's container needs
|
||||
// no other setting, so it reads no other, nor a file that another names.
|
||||
func ServeWAFListenAddr(lookupEnv func(string) (string, bool)) (string, error) {
|
||||
env := &environment{lookupEnv: lookupEnv}
|
||||
func ServeWAFListenAddr(v *viper.Viper) (string, error) {
|
||||
env := &environment{v: v, readFiles: true}
|
||||
listenAddr := env.address("SWWAF_LISTEN_ADDR")
|
||||
|
||||
return listenAddr, env.err
|
||||
@@ -602,8 +605,8 @@ func ServeWAFListenAddr(lookupEnv func(string) (string, bool)) (string, error) {
|
||||
// file, as FromEnvironment does, so that the line saying a setting is
|
||||
// invalid carries it too. A file that cannot be read gives the default
|
||||
// here, and FromEnvironment then stops the start over it.
|
||||
func InstanceName(lookupEnv func(string) (string, bool)) string {
|
||||
env := &environment{lookupEnv: lookupEnv}
|
||||
func InstanceName(v *viper.Viper) string {
|
||||
env := &environment{v: v, readFiles: true}
|
||||
|
||||
return env.instanceName()
|
||||
}
|
||||
@@ -619,13 +622,26 @@ func (c *Config) LogValue() slog.Value {
|
||||
}
|
||||
|
||||
// environment is where FromEnvironment reads the settings: it notes each
|
||||
// value for the log, and keeps the first error.
|
||||
// value for the log, and keeps the first error. It reads every setting
|
||||
// through v, the viper NewViper returns, and, unless readFiles is false,
|
||||
// honours each setting's _FILE form too.
|
||||
type environment struct {
|
||||
lookupEnv func(string) (string, bool)
|
||||
v *viper.Viper
|
||||
readFiles bool
|
||||
settings []slog.Attr
|
||||
err error
|
||||
}
|
||||
|
||||
// readSetting returns a setting's value and whether it is set, reading it
|
||||
// through v.
|
||||
func readSetting(v *viper.Viper, name string) (string, bool) {
|
||||
if !v.IsSet(name) {
|
||||
return "", false
|
||||
}
|
||||
|
||||
return v.GetString(name), true
|
||||
}
|
||||
|
||||
// value returns a setting's value, or its default when it is not set,
|
||||
// and notes it for the log.
|
||||
func (e *environment) value(name, defaultValue string) string {
|
||||
@@ -644,10 +660,14 @@ func (e *environment) value(name, defaultValue string) string {
|
||||
// names, less one newline at their end. It notes that file's path for the
|
||||
// log. Both variables set, or a file that cannot be read, is an error.
|
||||
func (e *environment) lookup(name string) (string, bool) {
|
||||
value, set := e.lookupEnv(name)
|
||||
value, set := readSetting(e.v, name)
|
||||
if !e.readFiles {
|
||||
return value, set
|
||||
}
|
||||
|
||||
fileName := name + "_FILE"
|
||||
|
||||
path, inFile := e.lookupEnv(fileName)
|
||||
path, inFile := readSetting(e.v, fileName)
|
||||
if !inFile {
|
||||
return value, set
|
||||
}
|
||||
@@ -1193,7 +1213,7 @@ func (e *environment) logRemoteURL(name string) *url.URL {
|
||||
// certificates. Unset or empty, it is nil. Its value names a file
|
||||
// already, so, unlike the other settings, it has no _FILE form.
|
||||
func (e *environment) certificates(name string) *x509.CertPool {
|
||||
path, _ := e.lookupEnv(name)
|
||||
path, _ := readSetting(e.v, name)
|
||||
e.settings = append(e.settings, slog.String(name, path))
|
||||
|
||||
if path == "" {
|
||||
|
||||
@@ -17,6 +17,8 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/viper"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
)
|
||||
@@ -217,18 +219,22 @@ const (
|
||||
// environment is a set of environment variables, for FromEnvironment.
|
||||
type environment map[string]string
|
||||
|
||||
// lookupEnv reads one of the variables, as os.LookupEnv does.
|
||||
func (e environment) lookupEnv(name string) (string, bool) {
|
||||
value, ok := e[name]
|
||||
// viper returns a viper Set with each of the variables, as the viper
|
||||
// NewViper returns one bound to a process's environment.
|
||||
func (e environment) viper() *viper.Viper {
|
||||
v := viper.New()
|
||||
for name, value := range e {
|
||||
v.Set(name, value)
|
||||
}
|
||||
|
||||
return value, ok
|
||||
return v
|
||||
}
|
||||
|
||||
// fromEnvironment reads the settings from env, which must be valid.
|
||||
func fromEnvironment(t *testing.T, env environment) *config.Config {
|
||||
t.Helper()
|
||||
|
||||
cfg, err := config.FromEnvironment(env.lookupEnv)
|
||||
cfg, err := config.FromEnvironment(env.viper())
|
||||
if err != nil {
|
||||
t.Fatalf("settings %v: %v", env, err)
|
||||
}
|
||||
@@ -455,7 +461,7 @@ func TestPathPrefixNotStartingWithSlashStopsTheStart(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := config.FromEnvironment(
|
||||
environment{rateLimitExemptPaths: "/favicon.ico,assets/"}.lookupEnv)
|
||||
environment{rateLimitExemptPaths: "/favicon.ico,assets/"}.viper())
|
||||
|
||||
want := rateLimitExemptPaths + `: "assets/" is not a path prefix ` +
|
||||
`starting with /, such as /assets/`
|
||||
@@ -510,7 +516,7 @@ func TestInvalidTrapPathOrErrorBurstThresholdStopsTheStart(t *testing.T) {
|
||||
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
|
||||
_, err := config.FromEnvironment(environment{tc.name: tc.value}.viper())
|
||||
|
||||
want := tc.name + ": " + tc.want
|
||||
if err == nil || err.Error() != want {
|
||||
@@ -636,7 +642,7 @@ func TestInvalidCoreRuleSetSettingStopsTheStart(t *testing.T) {
|
||||
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
|
||||
_, err := config.FromEnvironment(environment{tc.name: tc.value}.viper())
|
||||
|
||||
want := tc.name + ": " + tc.want
|
||||
if err == nil || err.Error() != want {
|
||||
@@ -761,7 +767,7 @@ func TestInvalidRemoteLogSettingStopsTheStart(t *testing.T) {
|
||||
{logRemoteAppName, ""}, {logRemoteAppName, "my app"},
|
||||
{logRemoteAppName, "gitéa"}, {logRemoteAppName, strings.Repeat("a", 49)},
|
||||
} {
|
||||
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
|
||||
_, err := config.FromEnvironment(environment{tc.name: tc.value}.viper())
|
||||
if err == nil || !strings.HasPrefix(err.Error(), tc.name+": ") {
|
||||
t.Errorf("%s=%q: error %v, want one naming it", tc.name, tc.value, err)
|
||||
}
|
||||
@@ -778,7 +784,7 @@ func TestRemoteLogCAFileWithoutCertificateStopsTheStart(t *testing.T) {
|
||||
t.Fatalf("write %s: %v", caFile, err)
|
||||
}
|
||||
|
||||
_, err = config.FromEnvironment(environment{logRemoteTLSCAFile: caFile}.lookupEnv)
|
||||
_, err = config.FromEnvironment(environment{logRemoteTLSCAFile: caFile}.viper())
|
||||
|
||||
want := logRemoteTLSCAFile + `: "` + caFile + `" holds no PEM certificate`
|
||||
if err == nil || err.Error() != want {
|
||||
@@ -793,7 +799,7 @@ func TestInstanceNameNotAnAppNameStopsTheStartOnlyWhileSending(t *testing.T) {
|
||||
|
||||
sending := environment{logRemoteURL: remoteURL, instanceName: spaced}
|
||||
|
||||
_, err := config.FromEnvironment(sending.lookupEnv)
|
||||
_, err := config.FromEnvironment(sending.viper())
|
||||
|
||||
want := logRemoteAppName + `: is unset, and ` + instanceName +
|
||||
` "fsn1 app1", its default, is not 1 to 48 printable ASCII characters ` +
|
||||
@@ -823,7 +829,7 @@ func TestAppNameSetStopsTheStartWhileSending(t *testing.T) {
|
||||
logRemoteURL: remoteURL,
|
||||
instanceName: instance,
|
||||
logRemoteAppName: "my app",
|
||||
}.lookupEnv)
|
||||
}.viper())
|
||||
|
||||
want := logRemoteAppName + `: "my app" is not 1 to 48 printable ASCII ` +
|
||||
`characters without a space, such as gitea`
|
||||
@@ -954,7 +960,7 @@ func TestWebhookHeadersAreLoggedMaskedAndNeverShown(t *testing.T) {
|
||||
// An item that is not a header is named by its place, not shown.
|
||||
_, err := config.FromEnvironment(environment{
|
||||
alertWebhookHeaders: "X-Team:ops," + secret,
|
||||
}.lookupEnv)
|
||||
}.viper())
|
||||
|
||||
want := alertWebhookHeaders + ": item 2 is not a header name followed by : " +
|
||||
"and the header's value, such as Authorization:Bearer <token>"
|
||||
@@ -991,7 +997,7 @@ func TestWebhookURLIsLoggedWithoutItsPathOrQueryAndNeverShown(t *testing.T) {
|
||||
"ftp://hooks.example/services/" + secret,
|
||||
"https://hooks.example/services/%zz" + secret,
|
||||
} {
|
||||
_, err := config.FromEnvironment(environment{alertWebhookURL: value}.lookupEnv)
|
||||
_, err := config.FromEnvironment(environment{alertWebhookURL: value}.viper())
|
||||
|
||||
want := alertWebhookURL + ": is not an http or https URL without a user or " +
|
||||
"a fragment, such as https://alerts.example/smallwebwaf"
|
||||
@@ -1042,7 +1048,7 @@ func TestNtfyTokenWithAControlCharacterStopsTheStartWithoutShowingIt(t *testing.
|
||||
"set": {alertNtfyToken: token + "\r"},
|
||||
"in a file": {alertNtfyToken + "_FILE": writeFile(t, token+"\r\n")},
|
||||
} {
|
||||
_, err := config.FromEnvironment(env.lookupEnv)
|
||||
_, err := config.FromEnvironment(env.viper())
|
||||
|
||||
want := alertNtfyToken + ": holds a control character, such as the " +
|
||||
"carriage return of a Windows line end"
|
||||
@@ -1059,7 +1065,7 @@ func TestInstanceNameWithAControlCharacterStopsTheStartOnlyWithNtfySet(t *testin
|
||||
|
||||
_, err := config.FromEnvironment(environment{
|
||||
instanceName: name, alertNtfyURL: "https://ntfy.example/smallwebwaf-alerts",
|
||||
}.lookupEnv)
|
||||
}.viper())
|
||||
|
||||
want := instanceName + `: "fsn1app1\r" holds a control character, such as the ` +
|
||||
`carriage return of a Windows line end, and is sent to ntfy in a header ` +
|
||||
@@ -1084,7 +1090,7 @@ func TestInstanceNameNotUTF8StopsTheStart(t *testing.T) {
|
||||
"set": {instanceName: latin1},
|
||||
"in a file": {instanceName + "_FILE": writeFile(t, latin1+"\n")},
|
||||
} {
|
||||
_, err := config.FromEnvironment(env.lookupEnv)
|
||||
_, err := config.FromEnvironment(env.viper())
|
||||
|
||||
want := instanceName + `: "caf\xe9" is not valid UTF-8`
|
||||
if err == nil || err.Error() != want {
|
||||
@@ -1099,7 +1105,7 @@ func TestCodeOnBothCountryListsStopsTheStart(t *testing.T) {
|
||||
_, err := config.FromEnvironment(environment{
|
||||
deniedCountries: "cn,ru",
|
||||
allowedCountries: "de,RU",
|
||||
}.lookupEnv)
|
||||
}.viper())
|
||||
if err == nil {
|
||||
t.Fatal("ru on both country lists was accepted")
|
||||
}
|
||||
@@ -1161,7 +1167,7 @@ func TestLookupDBPathGoesWithTheFileSourceAlone(t *testing.T) {
|
||||
lookupDBPath + ": is set while " + lookupSource + " is off; only file reads it",
|
||||
},
|
||||
} {
|
||||
_, err := config.FromEnvironment(tc.env.lookupEnv)
|
||||
_, err := config.FromEnvironment(tc.env.viper())
|
||||
if err == nil || err.Error() != tc.want {
|
||||
t.Errorf("settings %v: error %v, want %s", tc.env, err, tc.want)
|
||||
}
|
||||
@@ -1191,7 +1197,7 @@ func TestSettingNeedingLookupsStopsTheStartWhileTheyAreOff(t *testing.T) {
|
||||
|
||||
_, err := config.FromEnvironment(environment{
|
||||
lookupSource: off, name: value,
|
||||
}.lookupEnv)
|
||||
}.viper())
|
||||
if err == nil || !strings.HasPrefix(err.Error(), name+": ") ||
|
||||
!strings.Contains(err.Error(), lookupSource+" is off") {
|
||||
t.Errorf("error %v, want one naming %s and %s", err, name, lookupSource)
|
||||
@@ -1341,7 +1347,7 @@ func TestInvalidAnomalySettingStopsTheStartSayingWhatIsWrong(t *testing.T) {
|
||||
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
|
||||
_, err := config.FromEnvironment(environment{tc.name: tc.value}.viper())
|
||||
|
||||
want := tc.name + ": " + tc.want
|
||||
if err == nil || err.Error() != want {
|
||||
@@ -1420,7 +1426,7 @@ func TestInvalidBiasedThresholdStopsTheStartSayingWhatIsWrong(t *testing.T) {
|
||||
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
|
||||
_, err := config.FromEnvironment(environment{tc.name: tc.value}.viper())
|
||||
|
||||
want := tc.name + ": " + tc.want
|
||||
if err == nil || err.Error() != want {
|
||||
@@ -1517,7 +1523,7 @@ func TestInvalidReputationSettingStopsTheStartSayingWhatIsWrong(t *testing.T) {
|
||||
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
|
||||
_, err := config.FromEnvironment(environment{tc.name: tc.value}.viper())
|
||||
|
||||
want := tc.name + ": " + tc.want
|
||||
if err == nil || err.Error() != want {
|
||||
@@ -1532,7 +1538,7 @@ func TestASNLimitPercentURLThatIsABlocklistStopsTheStart(t *testing.T) {
|
||||
|
||||
_, err := config.FromEnvironment(environment{
|
||||
blocklistURLs: dropURL + "," + asnURL, asnLimitPercentURL: asnURL,
|
||||
}.lookupEnv)
|
||||
}.viper())
|
||||
|
||||
want := asnLimitPercentURL + `: "` + asnURL + `" is in SWWAF_BLOCKLIST_URLS too`
|
||||
if err == nil || err.Error() != want {
|
||||
@@ -1659,7 +1665,7 @@ func TestInvalidDNSBLSettingStopsTheStartSayingWhatIsWrong(t *testing.T) {
|
||||
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
|
||||
_, err := config.FromEnvironment(environment{tc.name: tc.value}.viper())
|
||||
|
||||
want := tc.name + ": " + tc.want
|
||||
if err == nil || err.Error() != want {
|
||||
@@ -1727,7 +1733,7 @@ func TestDNSBLZoneKeyIsLoggedMaskedAndNeverShown(t *testing.T) {
|
||||
{spamhaus + "," + otherZone, `"` + spamhausMasked + `" is listed twice`},
|
||||
{spamhaus + ",,", "has an empty item in its list"},
|
||||
} {
|
||||
_, err := config.FromEnvironment(environment{dnsblZones: tc.value}.lookupEnv)
|
||||
_, err := config.FromEnvironment(environment{dnsblZones: tc.value}.viper())
|
||||
|
||||
want := dnsblZones + ": " + tc.want
|
||||
if err == nil || err.Error() != want {
|
||||
@@ -1780,7 +1786,7 @@ func TestInvalidAbuseIPDBSettingStopsTheStartSayingWhatIsWrong(t *testing.T) {
|
||||
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
|
||||
_, err := config.FromEnvironment(environment{tc.name: tc.value}.viper())
|
||||
|
||||
want := tc.name + ": " + tc.want
|
||||
if err == nil || err.Error() != want {
|
||||
@@ -1894,7 +1900,7 @@ func TestInvalidCrowdSecSettingStopsTheStartSayingWhatIsWrong(t *testing.T) {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := config.FromEnvironment(tc.env.lookupEnv)
|
||||
_, err := config.FromEnvironment(tc.env.viper())
|
||||
if err == nil || err.Error() != tc.want {
|
||||
t.Errorf("error %v, want %s", err, tc.want)
|
||||
}
|
||||
@@ -2054,7 +2060,7 @@ func wantStartStopped(t *testing.T, invalid []struct{ name, value string }) {
|
||||
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
|
||||
_, err := config.FromEnvironment(environment{tc.name: tc.value}.viper())
|
||||
if err == nil {
|
||||
t.Fatalf("%s=%q was accepted", tc.name, tc.value)
|
||||
}
|
||||
@@ -2079,7 +2085,7 @@ func TestHostOrTransferEncodingStopsTheStart(t *testing.T) {
|
||||
t.Run(value, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := config.FromEnvironment(environment{logRequestHeaders: value}.lookupEnv)
|
||||
_, err := config.FromEnvironment(environment{logRequestHeaders: value}.viper())
|
||||
if err == nil || err.Error() != logRequestHeaders+": "+want {
|
||||
t.Errorf("error %v, want %s: %s", err, logRequestHeaders, want)
|
||||
}
|
||||
@@ -2096,7 +2102,7 @@ func TestShortTokenStopsTheStartWithoutShowingIt(t *testing.T) {
|
||||
t.Run(name+"="+value, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := config.FromEnvironment(environment{name: value}.lookupEnv)
|
||||
_, err := config.FromEnvironment(environment{name: value}.viper())
|
||||
|
||||
want := name + ": is shorter than 32 characters"
|
||||
if err == nil || err.Error() != want {
|
||||
@@ -2152,7 +2158,7 @@ func TestSettingFromFileIsCheckedAsTheSettingItself(t *testing.T) {
|
||||
|
||||
_, err := config.FromEnvironment(environment{
|
||||
bytesLimitPerMinute + "_FILE": writeFile(t, "lots\n"),
|
||||
}.lookupEnv)
|
||||
}.viper())
|
||||
|
||||
want := bytesLimitPerMinute + `: "lots" is not a size such as 512K, 100M or 5G, or off`
|
||||
if err == nil || err.Error() != want {
|
||||
@@ -2163,7 +2169,7 @@ func TestSettingFromFileIsCheckedAsTheSettingItself(t *testing.T) {
|
||||
logRemoteURL: remoteURL,
|
||||
instanceName: instance,
|
||||
logRemoteAppName + "_FILE": writeFile(t, "my app\n"),
|
||||
}.lookupEnv)
|
||||
}.viper())
|
||||
|
||||
want = logRemoteAppName + `: "my app" is not 1 to 48 printable ASCII ` +
|
||||
`characters without a space, such as gitea`
|
||||
@@ -2178,7 +2184,7 @@ func TestSettingAndItsFileBothSetStopsTheStart(t *testing.T) {
|
||||
_, err := config.FromEnvironment(environment{
|
||||
metricsToken: token,
|
||||
metricsToken + "_FILE": writeFile(t, token),
|
||||
}.lookupEnv)
|
||||
}.viper())
|
||||
|
||||
want := metricsToken + ": is set, and so is " + metricsToken +
|
||||
"_FILE; set only one of them"
|
||||
@@ -2193,7 +2199,7 @@ func TestUnreadableSettingFileStopsTheStart(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
|
||||
for _, path := range []string{filepath.Join(dir, "missing"), dir} {
|
||||
_, err := config.FromEnvironment(environment{metricsToken + "_FILE": path}.lookupEnv)
|
||||
_, err := config.FromEnvironment(environment{metricsToken + "_FILE": path}.viper())
|
||||
|
||||
want := metricsToken + "_FILE: cannot be read: "
|
||||
if err == nil || !strings.HasPrefix(err.Error(), want) {
|
||||
|
||||
+11
-24
@@ -8,8 +8,9 @@ import (
|
||||
"net/netip"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/viper"
|
||||
)
|
||||
|
||||
// RProxy is the settings of `smallwebwaf rproxy`. A size or time limit of zero
|
||||
@@ -58,13 +59,13 @@ var (
|
||||
"is not a whole number of milliseconds above zero, such as 60000, or off")
|
||||
)
|
||||
|
||||
// RProxyFromEnvironment reads rproxy's settings with lookupEnv, normally
|
||||
// os.LookupEnv. A setting that is not set takes its default, and
|
||||
// SWRP_WAF_BACKEND, which has none, must be set. A setting that is set but
|
||||
// invalid is an error that names it. Since rproxy never touches disk, its
|
||||
// settings have no _FILE form.
|
||||
func RProxyFromEnvironment(lookupEnv func(string) (string, bool)) (*RProxy, error) {
|
||||
env := &environment{lookupEnv: withoutFileForms(lookupEnv)}
|
||||
// RProxyFromEnvironment reads rproxy's settings through v, normally the viper
|
||||
// NewViper returns, bound to the process's environment. A setting that is not
|
||||
// set takes its default, and SWRP_WAF_BACKEND, which has none, must be set. A
|
||||
// setting that is set but invalid is an error that names it. Since rproxy
|
||||
// never touches disk, its settings have no _FILE form.
|
||||
func RProxyFromEnvironment(v *viper.Viper) (*RProxy, error) {
|
||||
env := &environment{v: v, readFiles: false}
|
||||
cfg := &RProxy{
|
||||
ListenAddr: env.address("SWRP_LISTEN_ADDR"),
|
||||
UpstreamURL: env.appURL("SWRP_UPSTREAM_URL", defaultUpstreamURL),
|
||||
@@ -91,9 +92,9 @@ func RProxyFromEnvironment(lookupEnv func(string) (string, bool)) (*RProxy, erro
|
||||
// SWRP_UPSTREAM_URL. The health check in rproxy's container needs no other
|
||||
// setting, so it reads no other.
|
||||
func RProxyListenAddrAndUpstreamURL(
|
||||
lookupEnv func(string) (string, bool),
|
||||
v *viper.Viper,
|
||||
) (string, *url.URL, error) {
|
||||
env := &environment{lookupEnv: withoutFileForms(lookupEnv)}
|
||||
env := &environment{v: v, readFiles: false}
|
||||
listenAddr := env.address("SWRP_LISTEN_ADDR")
|
||||
upstreamURL := env.appURL("SWRP_UPSTREAM_URL", defaultUpstreamURL)
|
||||
|
||||
@@ -104,20 +105,6 @@ func RProxyListenAddrAndUpstreamURL(
|
||||
return listenAddr, upstreamURL, nil
|
||||
}
|
||||
|
||||
// withoutFileForms returns lookupEnv finding no setting named with _FILE, since
|
||||
// rproxy never touches disk.
|
||||
func withoutFileForms(
|
||||
lookupEnv func(string) (string, bool),
|
||||
) func(string) (string, bool) {
|
||||
return func(name string) (string, bool) {
|
||||
if strings.HasSuffix(name, "_FILE") {
|
||||
return "", false
|
||||
}
|
||||
|
||||
return lookupEnv(name)
|
||||
}
|
||||
}
|
||||
|
||||
// LogValue makes an RProxy log as each setting's name with the value it was
|
||||
// given, or its default.
|
||||
func (c *RProxy) LogValue() slog.Value {
|
||||
|
||||
@@ -9,6 +9,8 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/viper"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
)
|
||||
|
||||
@@ -32,11 +34,12 @@ const (
|
||||
|
||||
// rproxyConfig reads rproxy's settings from env.
|
||||
func rproxyConfig(env map[string]string) (*config.RProxy, error) {
|
||||
return config.RProxyFromEnvironment(func(name string) (string, bool) {
|
||||
value, ok := env[name]
|
||||
v := viper.New()
|
||||
for name, value := range env {
|
||||
v.Set(name, value)
|
||||
}
|
||||
|
||||
return value, ok
|
||||
})
|
||||
return config.RProxyFromEnvironment(v)
|
||||
}
|
||||
|
||||
func TestRProxyDefaults(t *testing.T) {
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
package config
|
||||
|
||||
import "github.com/spf13/viper"
|
||||
|
||||
// NewViper returns the viper that reads the settings: each setting's
|
||||
// environment variable, bound to its own name. Every setting of servewaf and
|
||||
// rproxy is read through it, and none is read with os.Getenv or os.LookupEnv
|
||||
// outside it. An environment variable that is set but empty is set, as one
|
||||
// that holds a value is, which viper does not take for granted.
|
||||
func NewViper() *viper.Viper {
|
||||
v := viper.New()
|
||||
v.AutomaticEnv()
|
||||
v.AllowEmptyEnv(true)
|
||||
|
||||
return v
|
||||
}
|
||||
@@ -131,15 +131,6 @@ type Line struct {
|
||||
// ActionRuleBlocked, ActionWAFBlocked, or, in servewaf, ActionTooLarge
|
||||
// for the header limits.
|
||||
WouldAction string `json:"would_action,omitempty"`
|
||||
// Heuristic is the name of the heuristic that refused the request, in
|
||||
// observe mode too, and HeuristicFigures are the figures it acted on, by
|
||||
// name.
|
||||
Heuristic string `json:"heuristic,omitempty"`
|
||||
HeuristicFigures map[string]any `json:"heuristic_figures,omitempty"`
|
||||
// ReportHeuristic and ReportHeuristicFigures are the same for the
|
||||
// heuristic that banned the client after servewaf answered the request.
|
||||
ReportHeuristic string `json:"report_heuristic,omitempty"`
|
||||
ReportHeuristicFigures map[string]any `json:"report_heuristic_figures,omitempty"`
|
||||
// LimitPercent and LimitPercentSetting are, for a request the rate
|
||||
// limits counted whose client a biased threshold gives a percentage of
|
||||
// the rate limits below 100, that percentage and the setting that gave
|
||||
|
||||
@@ -33,13 +33,9 @@ func announcedLength(size int) string {
|
||||
func TestServerHasTheFixedLimits(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg, err := config.RProxyFromEnvironment(func(name string) (string, bool) {
|
||||
if name == "SWRP_WAF_BACKEND" {
|
||||
return nowhere, true
|
||||
}
|
||||
|
||||
return "", false
|
||||
})
|
||||
cfg, err := config.RProxyFromEnvironment(viperFrom(map[string]string{
|
||||
"SWRP_WAF_BACKEND": nowhere,
|
||||
}))
|
||||
if err != nil {
|
||||
t.Fatalf("settings: %v", err)
|
||||
}
|
||||
|
||||
@@ -18,6 +18,8 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/viper"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
"sneak.berlin/go/smallwebwaf/internal/rproxy"
|
||||
@@ -341,6 +343,17 @@ func startServer(
|
||||
return addr, out, server
|
||||
}
|
||||
|
||||
// viperFrom returns a viper Set with each of settings, as the viper
|
||||
// config.NewViper returns one bound to a process's environment.
|
||||
func viperFrom(settings map[string]string) *viper.Viper {
|
||||
v := viper.New()
|
||||
for name, value := range settings {
|
||||
v.Set(name, value)
|
||||
}
|
||||
|
||||
return v
|
||||
}
|
||||
|
||||
// startServerLogging is startServer with rproxy's request lines going to
|
||||
// requestLog and its process lines to processLog.
|
||||
func startServerLogging(
|
||||
@@ -354,11 +367,7 @@ func startServerLogging(
|
||||
}
|
||||
maps.Copy(settings, env)
|
||||
|
||||
cfg, err := config.RProxyFromEnvironment(func(name string) (string, bool) {
|
||||
value, ok := settings[name]
|
||||
|
||||
return value, ok
|
||||
})
|
||||
cfg, err := config.RProxyFromEnvironment(viperFrom(settings))
|
||||
if err != nil {
|
||||
t.Fatalf("settings %v: %v", settings, err)
|
||||
}
|
||||
|
||||
@@ -1,55 +0,0 @@
|
||||
package servewaf
|
||||
|
||||
// The heuristic abuseipdb. It reads SWWAF_ABUSEIPDB_KEY,
|
||||
// SWWAF_ABUSEIPDB_MIN_SCORE and SWWAF_REPUTATION_ACTION.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// abuseIPDB notes AbuseIPDB, as noteHit does, with the score, when its score
|
||||
// of the client is a hit, and refuses the request when SWWAF_REPUTATION_ACTION
|
||||
// is deny, as dnsbl does for a zone. While SWWAF_ABUSEIPDB_KEY is unset it does
|
||||
// nothing. A client without a score is checked in the background, by the
|
||||
// request's address, if its history counts an offence, and the request does
|
||||
// not wait for the answer. The score is then used for each address of the
|
||||
// client. Its figures are the score and SWWAF_ABUSEIPDB_MIN_SCORE.
|
||||
type abuseIPDB struct{}
|
||||
|
||||
func (abuseIPDB) name() string { return "abuseipdb" }
|
||||
|
||||
func (abuseIPDB) check(ctx context.Context, rq *request, _ time.Time) *verdict {
|
||||
if rq.h.config.AbuseIPDBKey == "" {
|
||||
return nil
|
||||
}
|
||||
|
||||
client := rq.h.clientGroup(rq.client)
|
||||
held, _ := rq.h.limiter.Client(client)
|
||||
offender := held.History.Offences != ratelimit.Offences{}
|
||||
|
||||
score, hit := rq.h.abuseIPDB.Hit(ctx, client, rq.client, offender)
|
||||
if !hit {
|
||||
return nil
|
||||
}
|
||||
|
||||
rq.abuseIPDBHit = true
|
||||
rq.noteHit(bans.ReputationHit{Source: reputation.AbuseIPDBSource, Score: &score},
|
||||
"scored by AbuseIPDB at or over SWWAF_ABUSEIPDB_MIN_SCORE")
|
||||
|
||||
if rq.h.config.ReputationAction != deny {
|
||||
return nil
|
||||
}
|
||||
|
||||
return &verdict{
|
||||
action: requestlog.ActionDenied,
|
||||
figures: map[string]any{
|
||||
figureScore: score, "min_score": rq.h.config.AbuseIPDBMinScore,
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -91,18 +91,6 @@ func TestSixteenThousandCookiesAreRefusedWellWithinTheTimeLimit(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestHeaderLimitRefusalNamesItsHeuristicCountsAndLimits(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := startServeWAF(t, "", nil)
|
||||
req := s.question(siteA, client, "/")
|
||||
fillHeaders(req, headerFieldLimit+1)
|
||||
do(t, req)
|
||||
|
||||
wantHeuristic(t, s.out.requestLine(t), "header_limits",
|
||||
`{"fields": 101, "fields_limit": 100, "cookies": 0, "cookies_limit": 50}`)
|
||||
}
|
||||
|
||||
func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
+79
-25
@@ -1,10 +1,7 @@
|
||||
package servewaf
|
||||
|
||||
// The heuristic active_ban, and the code that makes bans, which several
|
||||
// heuristics share. active_ban reads SWWAF_MODE; the bans are the ban ledger's.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"time"
|
||||
|
||||
@@ -21,15 +18,11 @@ func (rq *request) banResponse(action string) *refusal {
|
||||
return &refusal{status: rq.h.config.BanResponse, action: action}
|
||||
}
|
||||
|
||||
// activeBan refuses a client a ban covers: a ban on a netblock the client is
|
||||
// in, in force at now. It notes for the log line when that ban ends. A request
|
||||
// that makes the ban permanent, or in observe mode would have, raises the alert
|
||||
// for it. Its figures are the ban's netblock and when it ends.
|
||||
type activeBan struct{}
|
||||
|
||||
func (activeBan) name() string { return "active_ban" }
|
||||
|
||||
func (activeBan) check(_ context.Context, rq *request, now time.Time) *verdict {
|
||||
// banned reports whether a ban on a netblock the client is in covers the
|
||||
// request at now, and notes for the log line when that ban ends. A
|
||||
// request that makes the ban permanent, or in observe mode would have,
|
||||
// raises the alert for it.
|
||||
func (rq *request) banned(now time.Time) bool {
|
||||
check := rq.h.ledger.Check
|
||||
if rq.h.config.Observe {
|
||||
check = rq.h.ledger.Find // the ban refuses nothing, and stays as it is
|
||||
@@ -45,24 +38,85 @@ func (activeBan) check(_ context.Context, rq *request, now time.Time) *verdict {
|
||||
rq.alertBan(ban)
|
||||
}
|
||||
|
||||
if !banned {
|
||||
return nil
|
||||
return banned
|
||||
}
|
||||
|
||||
// limitBroken counts the request for the rate limits at now, notes the
|
||||
// client's counts for the log line, and reports whether the request takes
|
||||
// the client over a rate limit, as its limit percentage lowers it, which
|
||||
// breaks it.
|
||||
func (rq *request) limitBroken(now time.Time) bool {
|
||||
counts, hit, over := rq.h.limiter.Count(rq.h.clientGroup(rq.client), now,
|
||||
rq.limitPercent.percent)
|
||||
rq.line.Counts = counts
|
||||
|
||||
if over {
|
||||
rq.banForLimit(now, hit, rq.h.config.BanResponse)
|
||||
}
|
||||
|
||||
return &verdict{
|
||||
action: requestlog.ActionBanned,
|
||||
figures: map[string]any{
|
||||
figureNetblock: ban.Netblock.String(), "ban_expires": rq.line.BanExpires,
|
||||
},
|
||||
return over
|
||||
}
|
||||
|
||||
// countBytes counts the request's bytes, as countedBytes gives them, for
|
||||
// the byte limits, once its report has come, and notes the client's byte totals
|
||||
// for the log line; its requests stay there as the rate limits counted them.
|
||||
// Only an allowed request has them counted, and only one the rate limits
|
||||
// counted; in observe mode, not one that enforce mode would have refused. Bytes
|
||||
// that take the client over a byte limit, as its limit percentage for the byte
|
||||
// limits lowers it, break it; the response was passed on whole.
|
||||
func (rq *request) countBytes() {
|
||||
if !rq.counted || rq.line.WouldAction != "" {
|
||||
return
|
||||
}
|
||||
|
||||
now := rq.h.now()
|
||||
|
||||
counts, hit, over := rq.h.limiter.CountBytes(rq.h.clientGroup(rq.client), now,
|
||||
rq.countedBytes(), rq.bytesPercent.percent)
|
||||
rq.line.Counts.MinuteBytes = counts.MinuteBytes
|
||||
rq.line.Counts.HourBytes = counts.HourBytes
|
||||
rq.line.Counts.DayBytes = counts.DayBytes
|
||||
|
||||
if over {
|
||||
rq.banForLimit(now, hit, rq.out.status)
|
||||
}
|
||||
}
|
||||
|
||||
// limitFigures are the figures of a heuristic that broke hit, a limit: its
|
||||
// window, the count and the limit.
|
||||
func limitFigures(hit ratelimit.Hit) map[string]any {
|
||||
return map[string]any{
|
||||
figureWindow: hit.Window, figureCount: hit.Count, figureLimit: hit.Limit,
|
||||
// countRefusal counts the request for the error burst once it has been
|
||||
// answered, if servewaf refused it after a rule file match, a trap path or a
|
||||
// Core Rule Set match, and in observe mode if enforce mode would have: more than
|
||||
// SWWAF_ERROR_BURST_THRESHOLD such refusals of the client within a minute
|
||||
// break a limit. A client in SWWAF_ALLOW_NETS, which the checks skip, is
|
||||
// not counted, and nothing is while the threshold is off.
|
||||
func (rq *request) countRefusal() {
|
||||
cfg := rq.h.config
|
||||
if cfg.ErrorBurstThreshold == 0 {
|
||||
return
|
||||
}
|
||||
|
||||
if !rq.attack && !rq.ruleBlocked && !rq.wafBlocked {
|
||||
return
|
||||
}
|
||||
|
||||
now := rq.h.now()
|
||||
|
||||
hit, over := rq.h.limiter.CountRefusal(rq.h.clientGroup(rq.client), now,
|
||||
cfg.ErrorBurstThreshold)
|
||||
if !over {
|
||||
return
|
||||
}
|
||||
|
||||
// What the client was sent, or in observe mode would have been.
|
||||
status := rq.out.status
|
||||
|
||||
switch rq.line.WouldAction {
|
||||
case requestlog.ActionRuleBlocked, requestlog.ActionWAFBlocked:
|
||||
status = http.StatusForbidden
|
||||
case requestlog.ActionBanned:
|
||||
status = cfg.BanResponse
|
||||
}
|
||||
|
||||
rq.banForLimit(now, hit, status)
|
||||
}
|
||||
|
||||
// countedBytes returns the request's bytes, once it has ended, as the
|
||||
|
||||
@@ -228,22 +228,6 @@ func TestBannedClientIsRefusedBeforeItsCountryIsLookedUp(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRefusalUnderABanNamesItsHeuristicNetblockAndEnd(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, clk, _ := startWithClock(t, "", map[string]string{
|
||||
rateLimitPerMinute: "1", banScopeV4Prefix: "24",
|
||||
})
|
||||
expires := requestlog.FormatTime(clk.Now().Add(time.Hour))
|
||||
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
s.get(client, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
|
||||
line := s.get(otherClient, http.StatusForbidden, requestlog.ActionBanned)
|
||||
wantHeuristic(t, line, "active_ban",
|
||||
`{"netblock": "`+clientsNet+`", "ban_expires": "`+expires+`"}`)
|
||||
}
|
||||
|
||||
func TestBanResponseAnswersEveryRefusalButTheSizeLimits(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -1,41 +0,0 @@
|
||||
package servewaf
|
||||
|
||||
// The heuristic blocklists. It reads SWWAF_BLOCKLIST_ACTION, and the blocklists
|
||||
// through the reputation lists.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// blocklists notes the blocklists that list the client, as noteListed does,
|
||||
// and refuses the request when SWWAF_BLOCKLIST_ACTION is deny. Being limit, it
|
||||
// lowers the client's limits instead (see limitPercentages), and being log, it
|
||||
// does nothing more. Its figure is the blocklists that list the client.
|
||||
type blocklists struct{}
|
||||
|
||||
func (blocklists) name() string { return "blocklists" }
|
||||
|
||||
func (blocklists) check(_ context.Context, rq *request, _ time.Time) *verdict {
|
||||
listedBy := rq.h.lists.ListedBy(rq.client)
|
||||
rq.blocklisted = len(listedBy) > 0
|
||||
rq.noteListed(listedBy, "listed by a blocklist")
|
||||
|
||||
if !rq.refusedByBlocklists(listedBy) {
|
||||
return nil
|
||||
}
|
||||
|
||||
return &verdict{
|
||||
action: requestlog.ActionDenied,
|
||||
figures: map[string]any{"lists": listedBy},
|
||||
}
|
||||
}
|
||||
|
||||
// refusedByBlocklists reports whether the heuristic blocklists refuses a
|
||||
// client, given listedBy, the blocklists that list it: it does when one lists
|
||||
// it and SWWAF_BLOCKLIST_ACTION is deny.
|
||||
func (rq *request) refusedByBlocklists(listedBy []string) bool {
|
||||
return len(listedBy) > 0 && rq.h.config.BlocklistAction == deny
|
||||
}
|
||||
@@ -1,43 +0,0 @@
|
||||
package servewaf
|
||||
|
||||
// The heuristic byte_limits. It reads SWWAF_BYTES_COUNT, and the byte limits
|
||||
// and biased thresholds through the limiter and limitPercentages.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// byteLimits counts the request's bytes, as countedBytes gives them, for the
|
||||
// byte limits, once its report has come, and notes the client's byte totals for
|
||||
// the log line; its requests stay there as the rate limits counted them. Only
|
||||
// an allowed request has them counted, and only one the rate limits counted; in
|
||||
// observe mode, not one that enforce mode would have refused. Bytes that take
|
||||
// the client over a byte limit, as its limit percentage for the byte limits
|
||||
// lowers it, break it, which bans the client; the response was passed on whole.
|
||||
// Its figures are the window, the count and the limit.
|
||||
type byteLimits struct{}
|
||||
|
||||
func (byteLimits) name() string { return "byte_limits" }
|
||||
|
||||
func (byteLimits) check(_ context.Context, rq *request, now time.Time) *verdict {
|
||||
if !rq.counted || rq.line.WouldAction != "" {
|
||||
return nil
|
||||
}
|
||||
|
||||
counts, hit, over := rq.h.limiter.CountBytes(rq.h.clientGroup(rq.client), now,
|
||||
rq.countedBytes(), rq.bytesPercent.percent)
|
||||
rq.line.Counts.MinuteBytes = counts.MinuteBytes
|
||||
rq.line.Counts.HourBytes = counts.HourBytes
|
||||
rq.line.Counts.DayBytes = counts.DayBytes
|
||||
|
||||
if !over {
|
||||
return nil
|
||||
}
|
||||
|
||||
rq.banForLimit(now, hit, rq.out.status)
|
||||
|
||||
return &verdict{action: requestlog.ActionBanned, figures: limitFigures(hit)}
|
||||
}
|
||||
@@ -98,21 +98,6 @@ func TestEachByteLimitBansOnceTheResponseHasEnded(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestByteLimitBanNamesItsHeuristicWindowCountAndLimit(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _ := startWithAnswers(t, map[string]string{bytesLimitPerMinute: byteLimit})
|
||||
|
||||
line, _ := s.download()
|
||||
wantReportHeuristic(t, line, "", "null")
|
||||
|
||||
// Two answers of 70 bytes are over the limit of 99.
|
||||
line, _ = s.download()
|
||||
wantHeuristic(t, line, "", "null")
|
||||
wantReportHeuristic(t, line, "byte_limits",
|
||||
`{"window": "minute", "count": 140, "limit": 99}`)
|
||||
}
|
||||
|
||||
func TestResponseOverAByteLimitByItselfIsPassedOnWhole(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -1,10 +1,6 @@
|
||||
package servewaf
|
||||
|
||||
// The heuristic core_rule_set. It reads SWWAF_WAF_MODE, SWWAF_WAF_EXEMPT_PATHS
|
||||
// and SWWAF_WAF_ANOMALY_THRESHOLD.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
@@ -13,21 +9,18 @@ import (
|
||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||
)
|
||||
|
||||
// coreRuleSet inspects the request with the Core Rule Set, unless
|
||||
// checkCoreRuleSet inspects the request with the Core Rule Set, unless
|
||||
// SWWAF_WAF_MODE is off or SWWAF_WAF_EXEMPT_PATHS exempts its path, as
|
||||
// pathExempt decides, and notes the rules it matched and its score in the
|
||||
// log line, and the rules in the metrics. A score at or over
|
||||
// SWWAF_WAF_ANOMALY_THRESHOLD is a match: it raises the waf_block alert,
|
||||
// and in block mode refuses the request with 403, which is an offence its
|
||||
// client's history counts. Its figures are the score and the threshold.
|
||||
type coreRuleSet struct{}
|
||||
|
||||
func (coreRuleSet) name() string { return "core_rule_set" }
|
||||
|
||||
func (coreRuleSet) check(_ context.Context, rq *request, _ time.Time) *verdict {
|
||||
// and in block mode refuses the request, which is an offence its client's
|
||||
// history counts, and so returns ActionWAFBlocked. It returns "" for a
|
||||
// request it does not refuse.
|
||||
func (rq *request) checkCoreRuleSet() string {
|
||||
cfg := rq.h.config
|
||||
if cfg.WAFMode == config.WAFModeOff || pathExempt(rq.in, cfg.WAFExemptPaths) {
|
||||
return nil
|
||||
return ""
|
||||
}
|
||||
|
||||
start := time.Now()
|
||||
@@ -47,21 +40,18 @@ func (coreRuleSet) check(_ context.Context, rq *request, _ time.Time) *verdict {
|
||||
|
||||
threshold := cfg.WAFAnomalyThreshold
|
||||
if threshold == 0 || result.Score < threshold {
|
||||
return nil
|
||||
return ""
|
||||
}
|
||||
|
||||
rq.alertWAFBlock(result)
|
||||
|
||||
if cfg.WAFMode == config.WAFModeDetect {
|
||||
return nil
|
||||
return ""
|
||||
}
|
||||
|
||||
rq.wafBlocked = true
|
||||
|
||||
return &verdict{
|
||||
action: requestlog.ActionWAFBlocked,
|
||||
figures: map[string]any{figureScore: result.Score, "threshold": threshold},
|
||||
}
|
||||
return requestlog.ActionWAFBlocked
|
||||
}
|
||||
|
||||
// alertWAFBlock raises the waf_block alert for the request, which the Core
|
||||
|
||||
@@ -110,16 +110,6 @@ func TestCoreRuleSetRefusesAttacksInBlockModeAndOnlyLogsThemInDetectMode(t *test
|
||||
}
|
||||
}
|
||||
|
||||
func TestCoreRuleSetRefusalNamesItsHeuristicScoreAndThreshold(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{wafMode: block})
|
||||
|
||||
line := s.request(client, sqlInjection, http.StatusForbidden,
|
||||
requestlog.ActionWAFBlocked)
|
||||
wantHeuristic(t, line, "core_rule_set", `{"score": 5, "threshold": 5}`)
|
||||
}
|
||||
|
||||
func TestOrdinaryRequestIsInspectedAndPassed(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -1,42 +1,21 @@
|
||||
package servewaf
|
||||
|
||||
// The heuristic countries. It reads SWWAF_DENIED_COUNTRIES and
|
||||
// SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"slices"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// countries refuses a request by the client's country as it was looked up. A
|
||||
// client without a country, or whose country cannot be found, is refused only
|
||||
// by SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES. Its figures are the country and the
|
||||
// setting that refused it.
|
||||
type countries struct{}
|
||||
|
||||
func (countries) name() string { return "countries" }
|
||||
|
||||
func (countries) check(_ context.Context, rq *request, _ time.Time) *verdict {
|
||||
// countryDenied reports whether the country lists refuse the request, by
|
||||
// the client's country as it was looked up. A client without a country,
|
||||
// or whose country cannot be found, is refused only by
|
||||
// SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES.
|
||||
func (rq *request) countryDenied() bool {
|
||||
denied := rq.h.config.DeniedCountries
|
||||
allowed := rq.h.config.ExclusivelyAllowedCountries
|
||||
country := rq.line.Country
|
||||
|
||||
var setting string
|
||||
|
||||
switch {
|
||||
case slices.Contains(denied, country):
|
||||
setting = "SWWAF_DENIED_COUNTRIES"
|
||||
case len(allowed) > 0 && !slices.Contains(allowed, country):
|
||||
setting = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES"
|
||||
default:
|
||||
return nil
|
||||
if slices.Contains(denied, country) {
|
||||
return true
|
||||
}
|
||||
|
||||
return &verdict{
|
||||
action: requestlog.ActionCountryDenied,
|
||||
figures: map[string]any{"country": country, "setting": setting},
|
||||
}
|
||||
return len(allowed) > 0 && !slices.Contains(allowed, country)
|
||||
}
|
||||
|
||||
@@ -90,23 +90,6 @@ func TestCountryLists(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestCountryRefusalNamesItsHeuristicCountryAndSetting(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
geojsURL, _ := startGeoJS(t)
|
||||
s, _, _ := startWithClock(t, geojsURL, map[string]string{
|
||||
lookupTimeout: "1h", deniedCountries: "kp", allowedCountries: "de",
|
||||
})
|
||||
|
||||
line := s.get(fromKP, http.StatusForbidden, requestlog.ActionCountryDenied)
|
||||
wantHeuristic(t, line, "countries",
|
||||
`{"country": "KP", "setting": "SWWAF_DENIED_COUNTRIES"}`)
|
||||
|
||||
line = s.get(unplaced, http.StatusForbidden, requestlog.ActionCountryDenied)
|
||||
wantHeuristic(t, line, "countries",
|
||||
`{"country": "", "setting": "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES"}`)
|
||||
}
|
||||
|
||||
func TestCountryRefusalKeepsARequestFromTheApp(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -1,49 +0,0 @@
|
||||
package servewaf
|
||||
|
||||
// The heuristic crowdsec. It reads SWWAF_CROWDSEC_LAPI_URL, and the decisions
|
||||
// through the reputation lists.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// crowdSec refuses a request when a decision of the CrowdSec decision list on
|
||||
// the client is in force at now. It then notes the list, as noteHit does, and
|
||||
// bans the client until that decision ends. Its figures are the decision's
|
||||
// scenario and when it ends.
|
||||
type crowdSec struct{}
|
||||
|
||||
func (crowdSec) name() string { return "crowdsec" }
|
||||
|
||||
func (crowdSec) check(_ context.Context, rq *request, now time.Time) *verdict {
|
||||
decision, listed := rq.crowdSecDecisionInForce(now)
|
||||
if !listed {
|
||||
return nil
|
||||
}
|
||||
|
||||
rq.noteHit(bans.ReputationHit{Source: rq.h.config.CrowdSecDecisionsURL},
|
||||
"listed by the CrowdSec decision list")
|
||||
rq.banForCrowdSec(now, decision)
|
||||
|
||||
return &verdict{
|
||||
action: requestlog.ActionBanned,
|
||||
figures: map[string]any{
|
||||
"scenario": decision.Scenario,
|
||||
"expires": requestlog.FormatTime(decision.Expires),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// crowdSecDecisionInForce returns the decision of the CrowdSec decision list on
|
||||
// the client that is in force at now, and reports whether there is one, which
|
||||
// the heuristic crowdsec refuses the request for.
|
||||
func (rq *request) crowdSecDecisionInForce(
|
||||
now time.Time,
|
||||
) (reputation.Decision, bool) {
|
||||
return rq.h.lists.CrowdSecDecision(rq.client, now)
|
||||
}
|
||||
@@ -95,21 +95,6 @@ func TestClientTheCrowdSecDecisionListListsIsBannedUntilTheDecisionEnds(t *testi
|
||||
float64(fetched.Unix()))
|
||||
}
|
||||
|
||||
func TestCrowdSecBanNamesItsHeuristicScenarioAndEnd(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, clk, server, _ := startWithAlerts(t, map[string]string{
|
||||
crowdSecURL: lapi, crowdSecKey: bouncerKey,
|
||||
})
|
||||
loadDecisions(t, server, clk.Now(), `[{"duration": "4h0m0s", `+
|
||||
`"scenario": "crowdsecurity/ssh-bf", "scope": "Ip", "type": "ban", `+
|
||||
`"value": "`+client+`"}]`)
|
||||
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionBanned)
|
||||
wantHeuristic(t, line, "crowdsec", `{"scenario": "crowdsecurity/ssh-bf", `+
|
||||
`"expires": "`+requestlog.FormatTime(clk.Now().Add(4*time.Hour))+`"}`)
|
||||
}
|
||||
|
||||
func TestEndedCrowdSecDecisionNoLongerBansThoughTheCopyStillHoldsIt(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -1,29 +0,0 @@
|
||||
package servewaf
|
||||
|
||||
// The heuristic deny_nets. It reads SWWAF_DENY_NETS.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// denyNets refuses a client in SWWAF_DENY_NETS. Its figure is the netblock of
|
||||
// the setting the client is in.
|
||||
type denyNets struct{}
|
||||
|
||||
func (denyNets) name() string { return "deny_nets" }
|
||||
|
||||
func (denyNets) check(_ context.Context, rq *request, _ time.Time) *verdict {
|
||||
for _, netblock := range rq.h.config.DenyNets {
|
||||
if netblock.Contains(rq.client) {
|
||||
return &verdict{
|
||||
action: requestlog.ActionDenied,
|
||||
figures: map[string]any{figureNetblock: netblock.String()},
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -1,36 +0,0 @@
|
||||
package servewaf
|
||||
|
||||
// The heuristic dnsbl. It reads SWWAF_REPUTATION_ACTION, and the DNSBL zones'
|
||||
// verdicts through the DNSBL checker.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// dnsbl notes the DNSBL zones whose verdict lists the client, as noteListed
|
||||
// does, and refuses the request when SWWAF_REPUTATION_ACTION is deny. Being
|
||||
// limit, it lowers the client's limits instead (see limitPercentages), and
|
||||
// being log, it does nothing more. A zone without a verdict on the client is
|
||||
// asked about it in the background, and the request does not wait for the
|
||||
// answer. Its figure is the zones that list the client, their keys masked.
|
||||
type dnsbl struct{}
|
||||
|
||||
func (dnsbl) name() string { return "dnsbl" }
|
||||
|
||||
func (dnsbl) check(ctx context.Context, rq *request, _ time.Time) *verdict {
|
||||
listedBy := rq.h.dnsbl.ListedBy(ctx, rq.client)
|
||||
rq.dnsblListed = len(listedBy) > 0
|
||||
rq.noteListed(listedBy, "listed by a DNSBL zone")
|
||||
|
||||
if !rq.dnsblListed || rq.h.config.ReputationAction != deny {
|
||||
return nil
|
||||
}
|
||||
|
||||
return &verdict{
|
||||
action: requestlog.ActionDenied,
|
||||
figures: map[string]any{"zones": listedBy},
|
||||
}
|
||||
}
|
||||
@@ -106,25 +106,6 @@ func TestErrorBurstBreaksAtOneOverTheThreshold(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRefusalLimitBanNamesItsHeuristicWindowCountAndLimit(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startForErrorBurst(t, "1", nil)
|
||||
|
||||
const ruleFigures = `{"rule_id": "blocked", "rule_action": "block"}`
|
||||
|
||||
line := blockRule.send(s, client)
|
||||
wantHeuristic(t, line, "rule_files", ruleFigures)
|
||||
wantReportHeuristic(t, line, "", "null")
|
||||
|
||||
// The second refusal is the request refusal_limit bans for, and the line
|
||||
// names both.
|
||||
line = blockRule.send(s, client)
|
||||
wantHeuristic(t, line, "rule_files", ruleFigures)
|
||||
wantReportHeuristic(t, line, "refusal_limit",
|
||||
`{"window": "minute", "count": 2, "limit": 1}`)
|
||||
}
|
||||
|
||||
func TestErrorBurstBanNotesHistoryAndMetrics(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -1,73 +0,0 @@
|
||||
package servewaf
|
||||
|
||||
// The heuristic header_limits. It reads no setting: its limits are fixed.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// The header limits: the most cookies and header fields a request may have
|
||||
// when the checks reach the Core Rule Set. Real clients never reach them.
|
||||
const (
|
||||
maxCookies = 50
|
||||
maxHeaderFields = 100
|
||||
)
|
||||
|
||||
// headerLimits refuses a request over the header limits, with 431. Its figures
|
||||
// are the request's header fields and cookies, and the limit of each.
|
||||
type headerLimits struct{}
|
||||
|
||||
func (headerLimits) name() string { return "header_limits" }
|
||||
|
||||
func (headerLimits) check(_ context.Context, rq *request, _ time.Time) *verdict {
|
||||
fields, cookies := rq.headerCounts()
|
||||
if fields <= maxHeaderFields && cookies <= maxCookies {
|
||||
return nil
|
||||
}
|
||||
|
||||
return &verdict{
|
||||
action: requestlog.ActionTooLarge,
|
||||
figures: map[string]any{
|
||||
"fields": fields, "fields_limit": maxHeaderFields,
|
||||
"cookies": cookies, "cookies_limit": maxCookies,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// overHeaderLimits reports whether the request is over the header limits: it
|
||||
// has more than maxHeaderFields fields or more than maxCookies cookies, as
|
||||
// headerCounts counts them.
|
||||
func (rq *request) overHeaderLimits() bool {
|
||||
fields, cookies := rq.headerCounts()
|
||||
|
||||
return fields > maxHeaderFields || cookies > maxCookies
|
||||
}
|
||||
|
||||
// headerCounts returns the request's header fields, each value of a header sent
|
||||
// twice being one, and its cookies, each part of a Cookie header between
|
||||
// semicolons being one unless it is empty or only spaces and tabs. The Core
|
||||
// Rule Set runs each of its rules on each cookie, so that thousands of small
|
||||
// cookies keep it busy for over a second. It trims only spaces and control
|
||||
// characters, of which a header value can hold only tabs, so a part of Unicode
|
||||
// spaces is a cookie to it.
|
||||
func (rq *request) headerCounts() (int, int) {
|
||||
fields, cookies := 0, 0
|
||||
|
||||
for _, values := range rq.in.Header {
|
||||
fields += len(values)
|
||||
}
|
||||
|
||||
for _, value := range rq.in.Header.Values("Cookie") {
|
||||
for part := range strings.SplitSeq(value, ";") {
|
||||
if strings.Trim(part, " \t") != "" {
|
||||
cookies++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return fields, cookies
|
||||
}
|
||||
@@ -1,133 +0,0 @@
|
||||
package servewaf
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
)
|
||||
|
||||
// To add a heuristic, write one file with a type that implements heuristic,
|
||||
// and add one line for it to one of the lists below. Its settings go in
|
||||
// config.Config, and the comment at the top of its file names them.
|
||||
|
||||
// heuristic is one of the checks that decide what servewaf does with a request.
|
||||
type heuristic interface {
|
||||
// name is the heuristic's name, which the log line gives.
|
||||
name() string
|
||||
// check looks at the request at now, and returns nil to let it through.
|
||||
// ctx is the request's own context.
|
||||
check(ctx context.Context, rq *request, now time.Time) *verdict
|
||||
}
|
||||
|
||||
// verdict is a heuristic refusing a request, or banning its client: the action
|
||||
// the log line names, one of requestlog's Action constants, and the figures the
|
||||
// heuristic acted on, by name, such as the count and the limit.
|
||||
type verdict struct {
|
||||
action string
|
||||
figures map[string]any
|
||||
}
|
||||
|
||||
// The names of the figures that more than one heuristic gives.
|
||||
const (
|
||||
figureWindow = "window"
|
||||
figureCount = "count"
|
||||
figureLimit = "limit"
|
||||
figureScore = "score"
|
||||
figureNetblock = "netblock"
|
||||
)
|
||||
|
||||
// The lists of heuristics, each in the order its heuristics run. A client in
|
||||
// SWWAF_ALLOW_NETS skips them all.
|
||||
|
||||
// beforeLookup are the heuristics that run before the client's AS number and
|
||||
// country are looked up, so that a client they refuse is not looked up.
|
||||
func beforeLookup() []heuristic {
|
||||
return []heuristic{
|
||||
denyNets{},
|
||||
activeBan{},
|
||||
}
|
||||
}
|
||||
|
||||
// afterLookup are the heuristics that run once the lookup has been made.
|
||||
func afterLookup() []heuristic {
|
||||
return []heuristic{
|
||||
countries{},
|
||||
blocklists{},
|
||||
crowdSec{},
|
||||
dnsbl{},
|
||||
abuseIPDB{},
|
||||
rateLimits{},
|
||||
trapPaths{},
|
||||
ruleFiles{},
|
||||
headerLimits{},
|
||||
coreRuleSet{},
|
||||
}
|
||||
}
|
||||
|
||||
// afterAnswer are the heuristics that run on a request once servewaf has
|
||||
// answered it, before its decision line is written. They ban the client, and do
|
||||
// not change what the request got: their verdict's action is not used.
|
||||
func afterAnswer() []heuristic {
|
||||
return []heuristic{
|
||||
refusalLimit{},
|
||||
}
|
||||
}
|
||||
|
||||
// onReport are the heuristics that run on an allowed request when rproxy's
|
||||
// report of it comes, before its report line is written. Like afterAnswer, they
|
||||
// only ban.
|
||||
func onReport() []heuristic {
|
||||
return []heuristic{
|
||||
byteLimits{},
|
||||
}
|
||||
}
|
||||
|
||||
// runAfterAnswer runs the heuristics of afterAnswer on the request, at the time
|
||||
// it is called.
|
||||
func (rq *request) runAfterAnswer(ctx context.Context) {
|
||||
rq.runBanningHeuristics(ctx, afterAnswer())
|
||||
}
|
||||
|
||||
// runHeuristics runs list, beforeLookup or afterLookup, on the request at now,
|
||||
// and returns the first verdict, or nil when none gives one. The log line's
|
||||
// heuristic and heuristic_figures name the heuristic that gave it, and give its
|
||||
// figures. ctx is the request's own context.
|
||||
func (rq *request) runHeuristics(
|
||||
ctx context.Context, list []heuristic, now time.Time,
|
||||
) *verdict {
|
||||
name, v := rq.firstVerdict(ctx, list, now)
|
||||
if v != nil {
|
||||
rq.line.Heuristic = name
|
||||
rq.line.HeuristicFigures = v.figures
|
||||
}
|
||||
|
||||
return v
|
||||
}
|
||||
|
||||
// runBanningHeuristics runs list, afterAnswer or onReport, on the request at
|
||||
// the time it is called. The log line's report_heuristic and
|
||||
// report_heuristic_figures name the heuristic that gave a verdict, and give its
|
||||
// figures, so that heuristic and heuristic_figures still name the one that
|
||||
// refused the request. ctx is the request's own context.
|
||||
func (rq *request) runBanningHeuristics(ctx context.Context, list []heuristic) {
|
||||
name, v := rq.firstVerdict(ctx, list, rq.h.now())
|
||||
if v != nil {
|
||||
rq.line.ReportHeuristic = name
|
||||
rq.line.ReportHeuristicFigures = v.figures
|
||||
}
|
||||
}
|
||||
|
||||
// firstVerdict runs list on the request at now, in order, until a heuristic
|
||||
// gives a verdict, and returns that heuristic's name and its verdict, or "" and
|
||||
// nil when none gives one. ctx is the request's own context.
|
||||
func (rq *request) firstVerdict(
|
||||
ctx context.Context, list []heuristic, now time.Time,
|
||||
) (string, *verdict) {
|
||||
for _, each := range list {
|
||||
v := each.check(ctx, rq, now)
|
||||
if v != nil {
|
||||
return each.name(), v
|
||||
}
|
||||
}
|
||||
|
||||
return "", nil
|
||||
}
|
||||
@@ -14,18 +14,6 @@ import (
|
||||
// observe is the value of SWWAF_MODE for observe mode.
|
||||
const observe = "observe"
|
||||
|
||||
func TestObserveModeNamesTheHeuristicThatWouldHaveRefused(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{
|
||||
mode: observe, denyNets: client,
|
||||
})
|
||||
|
||||
line := s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionDenied)
|
||||
wantHeuristic(t, line, "deny_nets", `{"netblock": "`+client+`/32"}`)
|
||||
}
|
||||
|
||||
func TestObserveModeForwardsWhatEnforceModeRefuses(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -15,6 +15,8 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/viper"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||
@@ -347,6 +349,17 @@ func serve(t *testing.T, server *servewaf.Server) {
|
||||
// at serveWAFAddr, with the settings in env on top of the defaults, and returns
|
||||
// where it listens. What it writes is dropped. SWRP_TIMEOUT_MS is a minute unless
|
||||
// env sets it, so that on a busy host SWRP_DEFAULT_DISPOSITION does not decide.
|
||||
// viperFrom returns a viper Set with each of settings, as the viper
|
||||
// config.NewViper returns one bound to a process's environment.
|
||||
func viperFrom(settings map[string]string) *viper.Viper {
|
||||
v := viper.New()
|
||||
for name, value := range settings {
|
||||
v.Set(name, value)
|
||||
}
|
||||
|
||||
return v
|
||||
}
|
||||
|
||||
func startRProxy(
|
||||
t *testing.T, serveWAFAddr, appURL string, env map[string]string,
|
||||
) string {
|
||||
@@ -359,11 +372,7 @@ func startRProxy(
|
||||
}
|
||||
maps.Copy(settings, env)
|
||||
|
||||
cfg, err := config.RProxyFromEnvironment(func(name string) (string, bool) {
|
||||
value, ok := settings[name]
|
||||
|
||||
return value, ok
|
||||
})
|
||||
cfg, err := config.RProxyFromEnvironment(viperFrom(settings))
|
||||
if err != nil {
|
||||
t.Fatalf("settings %v: %v", settings, err)
|
||||
}
|
||||
@@ -423,11 +432,7 @@ func newServeWAFWith(
|
||||
|
||||
maps.Copy(settings, env)
|
||||
|
||||
cfg, err := config.FromEnvironment(func(name string) (string, bool) {
|
||||
value, ok := settings[name]
|
||||
|
||||
return value, ok
|
||||
})
|
||||
cfg, err := config.FromEnvironment(viperFrom(settings))
|
||||
if err != nil {
|
||||
t.Fatalf("settings %v: %v", settings, err)
|
||||
}
|
||||
|
||||
@@ -1,65 +0,0 @@
|
||||
package servewaf
|
||||
|
||||
// The heuristic rate_limits. It reads SWWAF_RATE_LIMIT_EXEMPT_NETS,
|
||||
// SWWAF_RATE_LIMIT_EXEMPT_PATHS and SWWAF_BAN_RESPONSE, and the rate limits and
|
||||
// biased thresholds through the limiter and limitPercentages.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// rateLimits counts the request for the rate limits, unless
|
||||
// countedByRateLimits exempts it, by the client's limit percentages, which it
|
||||
// notes for the log line with the client's counts. A request that takes the
|
||||
// client over a rate limit, as its limit percentage lowers it, breaks it: it is
|
||||
// refused, and bans the client. A request exempt from the rate limits is exempt
|
||||
// from the byte limits too. Its figures are the window, the count and the
|
||||
// limit.
|
||||
type rateLimits struct{}
|
||||
|
||||
func (rateLimits) name() string { return "rate_limits" }
|
||||
|
||||
func (rateLimits) check(_ context.Context, rq *request, now time.Time) *verdict {
|
||||
rq.counted = rq.countedByRateLimits()
|
||||
if !rq.counted {
|
||||
return nil
|
||||
}
|
||||
|
||||
rq.limitPercent, rq.bytesPercent = rq.limitPercentages()
|
||||
rq.line.LimitPercent, rq.line.LimitPercentSetting = rq.limitPercent.logged()
|
||||
rq.line.BytesPercent, rq.line.BytesPercentSetting = rq.bytesPercent.logged()
|
||||
|
||||
counts, hit, over := rq.h.limiter.Count(rq.h.clientGroup(rq.client), now,
|
||||
rq.limitPercent.percent)
|
||||
rq.line.Counts = counts
|
||||
|
||||
if !over {
|
||||
return nil
|
||||
}
|
||||
|
||||
rq.banForLimit(now, hit, rq.h.config.BanResponse)
|
||||
|
||||
return &verdict{action: requestlog.ActionRateLimited, figures: limitFigures(hit)}
|
||||
}
|
||||
|
||||
// overAWholeRateLimit reports whether the heuristic rate_limits will refuse the
|
||||
// request at now, whatever its client's limit percentages: the rate limits
|
||||
// count it, and one more request from its client takes it over a whole rate
|
||||
// limit. It counts nothing.
|
||||
func (rq *request) overAWholeRateLimit(now time.Time) bool {
|
||||
return rq.countedByRateLimits() &&
|
||||
rq.h.limiter.OverARateLimit(rq.h.clientGroup(rq.client), now)
|
||||
}
|
||||
|
||||
// countedByRateLimits reports whether the rate limits count the request: its
|
||||
// client is not in SWWAF_RATE_LIMIT_EXEMPT_NETS, and its path is not exempt
|
||||
// under SWWAF_RATE_LIMIT_EXEMPT_PATHS.
|
||||
func (rq *request) countedByRateLimits() bool {
|
||||
cfg := rq.h.config
|
||||
|
||||
return !isInside(rq.client, cfg.RateLimitExemptNets) &&
|
||||
!pathExempt(rq.in, cfg.RateLimitExemptPaths)
|
||||
}
|
||||
@@ -72,17 +72,6 @@ func TestRateLimitRefusesBeforeTheApp(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRateLimitRefusalNamesItsHeuristicWindowCountAndLimit(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{rateLimitPerMinute: "1"})
|
||||
|
||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionRateLimited)
|
||||
wantHeuristic(t, line, "rate_limits", `{"window": "minute", "count": 2, "limit": 1}`)
|
||||
}
|
||||
|
||||
func TestIPv6GroupPrefixSetsTheClientTheLimitsCount(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -1,54 +0,0 @@
|
||||
package servewaf
|
||||
|
||||
// The heuristic refusal_limit. It reads SWWAF_ERROR_BURST_THRESHOLD and
|
||||
// SWWAF_BAN_RESPONSE.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// refusalLimit counts the request once it has been answered, if servewaf
|
||||
// refused it after a rule file match, a trap path or a Core Rule Set match, and
|
||||
// in observe mode if enforce mode would have: more than
|
||||
// SWWAF_ERROR_BURST_THRESHOLD such refusals of the client within a minute break
|
||||
// a limit, which bans the client. A client in SWWAF_ALLOW_NETS, which the
|
||||
// heuristics skip, is not counted, and nothing is while the threshold is off.
|
||||
// Its figures are the window, the count and the limit.
|
||||
type refusalLimit struct{}
|
||||
|
||||
func (refusalLimit) name() string { return "refusal_limit" }
|
||||
|
||||
func (refusalLimit) check(_ context.Context, rq *request, now time.Time) *verdict {
|
||||
cfg := rq.h.config
|
||||
if cfg.ErrorBurstThreshold == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
if !rq.attack && !rq.ruleBlocked && !rq.wafBlocked {
|
||||
return nil
|
||||
}
|
||||
|
||||
hit, over := rq.h.limiter.CountRefusal(rq.h.clientGroup(rq.client), now,
|
||||
cfg.ErrorBurstThreshold)
|
||||
if !over {
|
||||
return nil
|
||||
}
|
||||
|
||||
// What the client was sent, or in observe mode would have been.
|
||||
status := rq.out.status
|
||||
|
||||
switch rq.line.WouldAction {
|
||||
case requestlog.ActionRuleBlocked, requestlog.ActionWAFBlocked:
|
||||
status = http.StatusForbidden
|
||||
case requestlog.ActionBanned:
|
||||
status = cfg.BanResponse
|
||||
}
|
||||
|
||||
rq.banForLimit(now, hit, status)
|
||||
|
||||
return &verdict{action: requestlog.ActionBanned, figures: limitFigures(hit)}
|
||||
}
|
||||
@@ -1,17 +1,91 @@
|
||||
package servewaf
|
||||
|
||||
// What the reputation heuristics share: blocklists, crowdsec, dnsbl and
|
||||
// abuseipdb.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||
)
|
||||
|
||||
// deny is the SWWAF_BLOCKLIST_ACTION and the SWWAF_REPUTATION_ACTION that
|
||||
// refuses the requests of a client a source lists.
|
||||
const deny = "deny"
|
||||
|
||||
// blocklistDenied notes the blocklists that list the client, as
|
||||
// noteListed does, and reports whether SWWAF_BLOCKLIST_ACTION, being deny,
|
||||
// refuses the request. Being limit, it lowers the client's limits instead
|
||||
// (see limitPercentages), and being log, it does nothing more.
|
||||
func (rq *request) blocklistDenied() bool {
|
||||
listedBy := rq.h.lists.ListedBy(rq.client)
|
||||
rq.blocklisted = len(listedBy) > 0
|
||||
rq.noteListed(listedBy, "listed by a blocklist")
|
||||
|
||||
return rq.blocklisted && rq.h.config.BlocklistAction == deny
|
||||
}
|
||||
|
||||
// crowdSecBanned reports whether a decision of the CrowdSec decision list
|
||||
// on the client is in force at now. If one is, it notes the list, as
|
||||
// noteHit does, and bans the client until that decision ends.
|
||||
func (rq *request) crowdSecBanned(now time.Time) bool {
|
||||
decision, listed := rq.h.lists.CrowdSecDecision(rq.client, now)
|
||||
if !listed {
|
||||
return false
|
||||
}
|
||||
|
||||
rq.noteHit(bans.ReputationHit{Source: rq.h.config.CrowdSecDecisionsURL},
|
||||
"listed by the CrowdSec decision list")
|
||||
rq.banForCrowdSec(now, decision)
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
// dnsblDenied notes the DNSBL zones whose verdict lists the client, as
|
||||
// noteListed does, and reports whether SWWAF_REPUTATION_ACTION, being
|
||||
// deny, refuses the request. Being limit, it lowers the client's limits
|
||||
// instead (see limitPercentages), and being log, it does nothing more. A
|
||||
// zone without a verdict on the client is asked about it in the
|
||||
// background, and the request does not wait for the answer. ctx is the
|
||||
// request's own context.
|
||||
func (rq *request) dnsblDenied(ctx context.Context) bool {
|
||||
listedBy := rq.h.dnsbl.ListedBy(ctx, rq.client)
|
||||
rq.dnsblListed = len(listedBy) > 0
|
||||
rq.noteListed(listedBy, "listed by a DNSBL zone")
|
||||
|
||||
return rq.dnsblListed && rq.h.config.ReputationAction == deny
|
||||
}
|
||||
|
||||
// abuseIPDBDenied notes AbuseIPDB, as noteHit does, with the score, when
|
||||
// its score of the client is a hit, and reports whether
|
||||
// SWWAF_REPUTATION_ACTION, being deny, refuses the request, as dnsblDenied
|
||||
// does for a zone. While SWWAF_ABUSEIPDB_KEY is unset it does nothing. A
|
||||
// client without a score is checked in the background, by the request's
|
||||
// address, if its history counts an offence, and the request does not
|
||||
// wait for the answer. The score is then used for each address of the
|
||||
// client. ctx is the request's own context.
|
||||
func (rq *request) abuseIPDBDenied(ctx context.Context) bool {
|
||||
if rq.h.config.AbuseIPDBKey == "" {
|
||||
return false
|
||||
}
|
||||
|
||||
client := rq.h.clientGroup(rq.client)
|
||||
held, _ := rq.h.limiter.Client(client)
|
||||
offender := held.History.Offences != ratelimit.Offences{}
|
||||
|
||||
score, hit := rq.h.abuseIPDB.Hit(ctx, client, rq.client, offender)
|
||||
if !hit {
|
||||
return false
|
||||
}
|
||||
|
||||
rq.abuseIPDBHit = true
|
||||
rq.noteHit(bans.ReputationHit{Source: reputation.AbuseIPDBSource, Score: &score},
|
||||
"scored by AbuseIPDB at or over SWWAF_ABUSEIPDB_MIN_SCORE")
|
||||
|
||||
return rq.h.config.ReputationAction == deny
|
||||
}
|
||||
|
||||
// noteListed notes each of sources, the URLs of the blocklists or the
|
||||
// DNSBL zones, their keys masked, that list the client, as noteHit does,
|
||||
// with reason.
|
||||
|
||||
@@ -118,56 +118,6 @@ func TestEachBlocklistActionForAListedAddressAndAListedNetblock(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestReputationRefusalNamesItsHeuristicAndWhatListedTheClient(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, tc := range []struct {
|
||||
heuristic string
|
||||
env map[string]string
|
||||
load func(*testing.T, *servewaf.Server)
|
||||
figures string
|
||||
}{
|
||||
{
|
||||
"blocklists",
|
||||
map[string]string{blocklistURLs: dropURL, blocklistAction: actionDeny},
|
||||
func(t *testing.T, server *servewaf.Server) {
|
||||
t.Helper()
|
||||
loadLists(t, server, map[string][]string{dropURL: {client}})
|
||||
},
|
||||
`{"lists": ["` + dropURL + `"]}`,
|
||||
},
|
||||
{
|
||||
"dnsbl",
|
||||
map[string]string{
|
||||
dnsblZones: dnsblZone, dnsblResolver: noResolver,
|
||||
reputationAction: actionDeny,
|
||||
},
|
||||
func(_ *testing.T, server *servewaf.Server) {
|
||||
loadVerdicts(server, map[string][]string{client: {dnsblZone}})
|
||||
},
|
||||
`{"zones": ["` + dnsblZone + `"]}`,
|
||||
},
|
||||
{
|
||||
"abuseipdb",
|
||||
map[string]string{abuseIPDBKey: accountKey, reputationAction: actionDeny},
|
||||
func(_ *testing.T, server *servewaf.Server) {
|
||||
loadScores(server, map[string]int64{client: 80})
|
||||
},
|
||||
`{"score": 80, "min_score": 75}`,
|
||||
},
|
||||
} {
|
||||
t.Run(tc.heuristic, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, server, _ := startWithLookups(t, tc.env)
|
||||
tc.load(t, server)
|
||||
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionDenied)
|
||||
wantHeuristic(t, line, tc.heuristic, tc.figures)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestBlocklistsComeAfterTheCountryListsAndSkipAllowNets(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -160,20 +160,16 @@ func requestHeaders(r *http.Request, names []string) map[string]string {
|
||||
}
|
||||
|
||||
// check is the one place where a request can be refused. It returns nil to let
|
||||
// the request through. The heuristics of checkClient decide: a request they
|
||||
// refuse is answered with SWWAF_BAN_RESPONSE, 403 for a block rule or the Core
|
||||
// Rule Set, or 431 for the header limits. In observe mode nothing is refused,
|
||||
// and the log line names what enforce mode would have done. ctx is the
|
||||
// request's own context.
|
||||
// the request through. The checks of checkClient decide: a request they refuse
|
||||
// is answered with SWWAF_BAN_RESPONSE, 403 for a block rule or the Core Rule Set,
|
||||
// or 431 for the header limits. In observe mode nothing is refused, and the log
|
||||
// line names what enforce mode would have done. ctx is the request's own context.
|
||||
func (rq *request) check(ctx context.Context) *refusal {
|
||||
v := rq.checkClient(ctx)
|
||||
if v == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
action := v.action
|
||||
action := rq.checkClient(ctx)
|
||||
|
||||
switch {
|
||||
case action == "":
|
||||
return nil
|
||||
case rq.h.config.Observe:
|
||||
rq.line.WouldAction = action
|
||||
|
||||
@@ -187,21 +183,35 @@ func (rq *request) check(ctx context.Context) *refusal {
|
||||
}
|
||||
}
|
||||
|
||||
// checkClient runs the heuristics of beforeLookup on the request, then looks
|
||||
// up the client's AS number and country, then runs those of afterLookup. It
|
||||
// returns the first verdict, or nil when none gives one. A client in
|
||||
// SWWAF_ALLOW_NETS skips them all, and is not looked up. ctx is the request's
|
||||
// own context.
|
||||
func (rq *request) checkClient(ctx context.Context) *verdict {
|
||||
if isInside(rq.client, rq.h.config.AllowNets) {
|
||||
return nil
|
||||
// checkClient runs the checks on the request's client, and returns the
|
||||
// action of the first that refuses the request, or "" when none does. A
|
||||
// client in SWWAF_ALLOW_NETS skips them, and is not looked up. For any
|
||||
// other client, SWWAF_DENY_NETS comes first, then a ban on its netblock,
|
||||
// so that a client either refuses is not looked up, then the lookup of
|
||||
// its AS number and country, then the country lists, then the blocklists,
|
||||
// then the CrowdSec decision list, which bans the client it lists, then
|
||||
// the DNSBL zones' verdicts, and then AbuseIPDB's score; a request any of
|
||||
// them refuses is not counted for the rate limits. Then come the
|
||||
// rate limits, unless the client is in SWWAF_RATE_LIMIT_EXEMPT_NETS or the
|
||||
// request's path is exempt under SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that
|
||||
// every other request is counted, each of them by the client's limit
|
||||
// percentages, then SWWAF_TRAP_PATHS, then the rule files, then the header
|
||||
// limits, and last the Core Rule Set. A request exempt from the rate limits
|
||||
// is exempt from the byte limits too. ctx is the request's own context.
|
||||
func (rq *request) checkClient(ctx context.Context) string {
|
||||
cfg := rq.h.config
|
||||
if isInside(rq.client, cfg.AllowNets) {
|
||||
return ""
|
||||
}
|
||||
|
||||
now := rq.h.now()
|
||||
|
||||
v := rq.runHeuristics(ctx, beforeLookup(), now)
|
||||
if v != nil {
|
||||
return v
|
||||
if isInside(rq.client, cfg.DenyNets) {
|
||||
return requestlog.ActionDenied
|
||||
}
|
||||
|
||||
if rq.banned(now) {
|
||||
return requestlog.ActionBanned
|
||||
}
|
||||
|
||||
rq.lookUp(ctx, now)
|
||||
@@ -210,7 +220,59 @@ func (rq *request) checkClient(ctx context.Context) *verdict {
|
||||
// again: a request counted at the time it arrived could be counted
|
||||
// after a later one of its client's, which the rate limits would take
|
||||
// for the clock set back.
|
||||
return rq.runHeuristics(ctx, afterLookup(), rq.h.now())
|
||||
now = rq.h.now()
|
||||
|
||||
if rq.countryDenied() {
|
||||
return requestlog.ActionCountryDenied
|
||||
}
|
||||
|
||||
if rq.blocklistDenied() {
|
||||
return requestlog.ActionDenied
|
||||
}
|
||||
|
||||
if rq.crowdSecBanned(now) {
|
||||
return requestlog.ActionBanned
|
||||
}
|
||||
|
||||
if rq.dnsblDenied(ctx) || rq.abuseIPDBDenied(ctx) {
|
||||
return requestlog.ActionDenied
|
||||
}
|
||||
|
||||
rq.counted = rq.countedByRateLimits()
|
||||
if rq.counted {
|
||||
rq.limitPercent, rq.bytesPercent = rq.limitPercentages()
|
||||
rq.line.LimitPercent, rq.line.LimitPercentSetting = rq.limitPercent.logged()
|
||||
rq.line.BytesPercent, rq.line.BytesPercentSetting = rq.bytesPercent.logged()
|
||||
}
|
||||
|
||||
if rq.counted && rq.limitBroken(now) {
|
||||
return requestlog.ActionRateLimited
|
||||
}
|
||||
|
||||
if rq.trapPath(now) {
|
||||
return requestlog.ActionBanned
|
||||
}
|
||||
|
||||
action := rq.checkRules(now)
|
||||
if action != "" {
|
||||
return action
|
||||
}
|
||||
|
||||
if rq.overHeaderLimits() {
|
||||
return requestlog.ActionTooLarge
|
||||
}
|
||||
|
||||
return rq.checkCoreRuleSet()
|
||||
}
|
||||
|
||||
// countedByRateLimits reports whether the rate limits count the request: its
|
||||
// client is not in SWWAF_RATE_LIMIT_EXEMPT_NETS, and its path is not exempt
|
||||
// under SWWAF_RATE_LIMIT_EXEMPT_PATHS.
|
||||
func (rq *request) countedByRateLimits() bool {
|
||||
cfg := rq.h.config
|
||||
|
||||
return !isInside(rq.client, cfg.RateLimitExemptNets) &&
|
||||
!pathExempt(rq.in, cfg.RateLimitExemptPaths)
|
||||
}
|
||||
|
||||
// pathExempt reports whether r is exempt under prefixes,
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
package servewaf_test
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"io"
|
||||
"math"
|
||||
"net/http"
|
||||
@@ -273,47 +272,6 @@ func wantTimings(t *testing.T, line logLine, want ...string) {
|
||||
}
|
||||
}
|
||||
|
||||
// wantHeuristic checks that line names heuristic as the one that refused the
|
||||
// request, with figures, a JSON object. A heuristic of "" with figures "null" is
|
||||
// a line that names none.
|
||||
func wantHeuristic(t *testing.T, line logLine, heuristic, figures string) {
|
||||
t.Helper()
|
||||
|
||||
if line.Heuristic != heuristic ||
|
||||
!reflect.DeepEqual(line.HeuristicFigures, decodeFigures(t, figures)) {
|
||||
t.Errorf("log line has heuristic %q with heuristic_figures %v, want %q with %s",
|
||||
line.Heuristic, line.HeuristicFigures, heuristic, figures)
|
||||
}
|
||||
}
|
||||
|
||||
// wantReportHeuristic checks report_heuristic and report_heuristic_figures as
|
||||
// wantHeuristic checks heuristic and heuristic_figures.
|
||||
func wantReportHeuristic(t *testing.T, line logLine, heuristic, figures string) {
|
||||
t.Helper()
|
||||
|
||||
if line.ReportHeuristic != heuristic ||
|
||||
!reflect.DeepEqual(line.ReportHeuristicFigures, decodeFigures(t, figures)) {
|
||||
t.Errorf("log line has report_heuristic %q with report_heuristic_figures %v, "+
|
||||
"want %q with %s",
|
||||
line.ReportHeuristic, line.ReportHeuristicFigures, heuristic, figures)
|
||||
}
|
||||
}
|
||||
|
||||
// decodeFigures decodes figures, a JSON object of a heuristic's figures, as
|
||||
// the log line's figures are decoded.
|
||||
func decodeFigures(t *testing.T, figures string) map[string]any {
|
||||
t.Helper()
|
||||
|
||||
var decoded map[string]any
|
||||
|
||||
err := json.Unmarshal([]byte(figures), &decoded)
|
||||
if err != nil {
|
||||
t.Fatalf("decode %s: %v", figures, err)
|
||||
}
|
||||
|
||||
return decoded
|
||||
}
|
||||
|
||||
// microseconds is a timing in whole microseconds.
|
||||
func microseconds(milliseconds float64) int64 {
|
||||
return int64(math.Round(milliseconds * 1000))
|
||||
|
||||
@@ -1,55 +0,0 @@
|
||||
package servewaf
|
||||
|
||||
// The heuristic rule_files. It reads the rules of the rule files, which
|
||||
// SWWAF_RULES_DIR and SWWAF_RULES_ENABLED name.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||
)
|
||||
|
||||
// ruleFiles checks the request against the rules of the rule files at now,
|
||||
// notes the ids of those it matches in the log line, and refuses the request
|
||||
// when the last rule it matches refuses it: with ActionRuleBlocked for a block
|
||||
// rule, and ActionBanned for a ban rule. A ban rule bans the client's netblock
|
||||
// for a clear sign of attack, or in observe mode raises the alert for the ban
|
||||
// it would have made. Either rule's match is noted as an offence, for the
|
||||
// client's history. Its figures are the rule's id and action.
|
||||
type ruleFiles struct{}
|
||||
|
||||
func (ruleFiles) name() string { return "rule_files" }
|
||||
|
||||
func (ruleFiles) check(_ context.Context, rq *request, now time.Time) *verdict {
|
||||
matched := rq.h.rules.Match(rq.in)
|
||||
|
||||
for _, rule := range matched {
|
||||
rq.line.RuleIDs = append(rq.line.RuleIDs, rule.ID)
|
||||
rq.h.metrics.RuleMatched(rule.ID, rule.Action)
|
||||
}
|
||||
|
||||
if len(matched) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Only the last rule matched can refuse the request.
|
||||
last := matched[len(matched)-1]
|
||||
figures := map[string]any{"rule_id": last.ID, "rule_action": last.Action}
|
||||
|
||||
switch last.Action {
|
||||
case rules.ActionBlock:
|
||||
rq.ruleBlocked = true
|
||||
|
||||
return &verdict{action: requestlog.ActionRuleBlocked, figures: figures}
|
||||
case rules.ActionBan:
|
||||
rq.attack = true
|
||||
rq.banForAttack(now, bans.Notes{RuleID: last.ID, Target: last.Target})
|
||||
|
||||
return &verdict{action: requestlog.ActionBanned, figures: figures}
|
||||
default:
|
||||
return nil
|
||||
}
|
||||
}
|
||||
@@ -93,21 +93,6 @@ func TestEachRuleAction(t *testing.T) {
|
||||
s.get(client, http.StatusTooManyRequests, requestlog.ActionBanned)
|
||||
}
|
||||
|
||||
func TestRuleRefusalNamesItsHeuristicAndTheRule(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{
|
||||
rulesDir: writeRules(t, "noted path log ^/\n"+testRules),
|
||||
})
|
||||
|
||||
line := s.request(client, "/blocked", http.StatusForbidden,
|
||||
requestlog.ActionRuleBlocked)
|
||||
wantHeuristic(t, line, "rule_files", `{"rule_id": "blocked", "rule_action": "block"}`)
|
||||
|
||||
line = s.request(client, "/.env", http.StatusForbidden, requestlog.ActionBanned)
|
||||
wantHeuristic(t, line, "rule_files", `{"rule_id": "probe", "rule_action": "ban"}`)
|
||||
}
|
||||
|
||||
func TestNextClearSignOfAttackAfterABanBansPermanently(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
package servewaf
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||
)
|
||||
|
||||
// trapPath reports whether the request asks for a path in
|
||||
// SWWAF_TRAP_PATHS: its path as a path rule sees it, without the query,
|
||||
// as sent or once percent-decoded, is one of them. Such a request is a
|
||||
// clear sign of attack, as a ban rule's match is: it bans the client's
|
||||
// netblock, or in observe mode raises the alert for the ban it would have
|
||||
// made. The ban's notes give the path as sent.
|
||||
func (rq *request) trapPath(now time.Time) bool {
|
||||
path := rules.Path(rq.in)
|
||||
if !slices.Contains(rq.h.config.TrapPaths, path) &&
|
||||
!slices.Contains(rq.h.config.TrapPaths, rules.DecodeOnce(path)) {
|
||||
return false
|
||||
}
|
||||
|
||||
rq.attack = true
|
||||
rq.banForAttack(now, bans.Notes{TrapPath: path})
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
// checkRules checks the request against the rules of the rule files at
|
||||
// now, notes the ids of those it matches in the log line, and returns the
|
||||
// action of the rule that refuses it, ActionRuleBlocked for a block rule
|
||||
// and ActionBanned for a ban rule, or "" when none does. A ban rule bans
|
||||
// the client's netblock for a clear sign of attack, or in observe mode
|
||||
// raises the alert for the ban it would have made. Either rule's match
|
||||
// is noted as an offence, for the client's history.
|
||||
func (rq *request) checkRules(now time.Time) string {
|
||||
matched := rq.h.rules.Match(rq.in)
|
||||
|
||||
for _, rule := range matched {
|
||||
rq.line.RuleIDs = append(rq.line.RuleIDs, rule.ID)
|
||||
rq.h.metrics.RuleMatched(rule.ID, rule.Action)
|
||||
}
|
||||
|
||||
if len(matched) == 0 {
|
||||
return ""
|
||||
}
|
||||
|
||||
// Only the last rule matched can refuse the request.
|
||||
switch last := matched[len(matched)-1]; last.Action {
|
||||
case rules.ActionBlock:
|
||||
rq.ruleBlocked = true
|
||||
|
||||
return requestlog.ActionRuleBlocked
|
||||
case rules.ActionBan:
|
||||
rq.attack = true
|
||||
rq.banForAttack(now, bans.Notes{RuleID: last.ID, Target: last.Target})
|
||||
|
||||
return requestlog.ActionBanned
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
@@ -2,7 +2,6 @@ package servewaf
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
@@ -53,6 +52,13 @@ const ReportPath = "/_smallwebwaf/report"
|
||||
// QuestionBodyMaxBytes is the most of a visitor's body a question carries.
|
||||
const QuestionBodyMaxBytes = 64 << 10
|
||||
|
||||
// The header limits: the most cookies and header fields a request may have
|
||||
// when the checks reach the Core Rule Set. Real clients never reach them.
|
||||
const (
|
||||
maxCookies = 50
|
||||
maxHeaderFields = 100
|
||||
)
|
||||
|
||||
const (
|
||||
// maxAwaitingReport is the most answered requests held until their
|
||||
// reports come. Past it, the request answered longest ago is dropped.
|
||||
@@ -141,20 +147,48 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
// refusedWhateverTheAnswer reports whether the request is one that does not wait
|
||||
// for GeoJS, since the checks will refuse it at now whatever GeoJS answers about
|
||||
// its client: the heuristic crowdsec, header_limits, blocklists or rate_limits
|
||||
// will refuse it, as a function in that heuristic's file says. In observe mode,
|
||||
// which refuses nothing, none is. rproxy lets a request through when servewaf
|
||||
// answers too late.
|
||||
// its client: it is over the header limits, a blocklist lists its client while
|
||||
// SWWAF_BLOCKLIST_ACTION is deny, the CrowdSec decision list lists it, or it is
|
||||
// over a whole rate limit. In observe mode, which refuses nothing, none is.
|
||||
// rproxy lets a request through when servewaf answers too late.
|
||||
func (rq *request) refusedWhateverTheAnswer(now time.Time) bool {
|
||||
if rq.h.config.Observe {
|
||||
return false
|
||||
}
|
||||
|
||||
_, decided := rq.crowdSecDecisionInForce(now)
|
||||
lists := rq.h.lists
|
||||
_, decided := lists.CrowdSecDecision(rq.client, now)
|
||||
|
||||
return decided || rq.overHeaderLimits() ||
|
||||
rq.refusedByBlocklists(rq.h.lists.ListedBy(rq.client)) ||
|
||||
rq.overAWholeRateLimit(now)
|
||||
(rq.h.config.BlocklistAction == deny && len(lists.ListedBy(rq.client)) > 0) ||
|
||||
(rq.countedByRateLimits() &&
|
||||
rq.h.limiter.OverARateLimit(rq.h.clientGroup(rq.client), now))
|
||||
}
|
||||
|
||||
// overHeaderLimits reports whether servewaf refuses the request for its
|
||||
// headers: it has more than maxHeaderFields fields, each value of a header sent
|
||||
// twice being one, or more than maxCookies cookies, each part of a Cookie header
|
||||
// between semicolons being one unless it is empty or only spaces and tabs. The
|
||||
// Core Rule Set runs each of its rules on each cookie, so that thousands of small
|
||||
// cookies keep it busy for over a second. It trims only spaces and control
|
||||
// characters, of which a header value can hold only tabs, so a part of Unicode
|
||||
// spaces is a cookie to it.
|
||||
func (rq *request) overHeaderLimits() bool {
|
||||
fields, cookies := 0, 0
|
||||
|
||||
for _, values := range rq.in.Header {
|
||||
fields += len(values)
|
||||
}
|
||||
|
||||
for _, value := range rq.in.Header.Values("Cookie") {
|
||||
for part := range strings.SplitSeq(value, ";") {
|
||||
if strings.Trim(part, " \t") != "" {
|
||||
cookies++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return fields > maxHeaderFields || cookies > maxCookies
|
||||
}
|
||||
|
||||
// answerQuestion runs the checks on the visitor's request a question
|
||||
@@ -185,18 +219,18 @@ func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
if refused == nil && !underOwnPrefix {
|
||||
// In observe mode, what enforce mode would have refused counts here.
|
||||
defer rq.runAfterAnswer(r.Context())
|
||||
defer rq.countRefusal()
|
||||
|
||||
rq.allow()
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
// The last deferred runs first: runAfterAnswer before addToHistory, so that
|
||||
// a broken refusal limit is in the client's history.
|
||||
// The last deferred runs first: countRefusal before addToHistory, so that a
|
||||
// broken error burst is in the client's history.
|
||||
defer rq.addToHistory()
|
||||
defer rq.countAnomalies()
|
||||
defer rq.runAfterAnswer(r.Context())
|
||||
defer rq.countRefusal()
|
||||
|
||||
if refused != nil {
|
||||
rq.deny(*refused)
|
||||
@@ -378,7 +412,7 @@ func (h *handler) takeReport(w http.ResponseWriter, r *http.Request) {
|
||||
// Remove reports true to one report only, of several that name the id. A
|
||||
// question still being handled is held as nil.
|
||||
if found && held != nil && h.awaitingReport.Remove(report.RequestID) {
|
||||
h.finishWith(r.Context(), held, report)
|
||||
h.finishWith(held, report)
|
||||
}
|
||||
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
@@ -441,11 +475,11 @@ func (report *Report) checkRanges() error {
|
||||
|
||||
// finishWith finishes a request with what its report says, and writes the
|
||||
// request's last log line, which holds its decision's fields and the report's.
|
||||
// For an allowed request it first runs the heuristics of onReport, and adds the
|
||||
// request to its client's history, the anomaly counters and the metrics, with
|
||||
// the status and bytes the report gives. A denied request's counting was
|
||||
// finished at its decision. ctx is the report's own context.
|
||||
func (h *handler) finishWith(ctx context.Context, held *decision, report Report) {
|
||||
// For an allowed request it first counts the request's bytes, and adds it to its
|
||||
// client's history, the anomaly counters and the metrics, with the status and
|
||||
// bytes the report gives. A denied request's counting was finished at its
|
||||
// decision.
|
||||
func (h *handler) finishWith(held *decision, report Report) {
|
||||
rq := &request{
|
||||
decision: *held,
|
||||
h: h,
|
||||
@@ -462,7 +496,7 @@ func (h *handler) finishWith(ctx context.Context, held *decision, report Report)
|
||||
if rq.allowed {
|
||||
rq.line.DurationUpstreamTotal = &report.DurationUpstreamTotal
|
||||
|
||||
rq.runBanningHeuristics(ctx, onReport())
|
||||
rq.countBytes()
|
||||
rq.countAnomalies()
|
||||
rq.addToHistory()
|
||||
|
||||
|
||||
@@ -104,15 +104,6 @@ func TestDenyNetsRefuseBeforeTheLookup(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestDenyNetsRefusalNamesItsHeuristicAndNetblock(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{denyNets: clientsNet})
|
||||
|
||||
line := s.get(client, http.StatusForbidden, requestlog.ActionDenied)
|
||||
wantHeuristic(t, line, "deny_nets", `{"netblock": "`+clientsNet+`"}`)
|
||||
}
|
||||
|
||||
func TestRequestRefusedByDenyNetsIsNotCounted(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -1,39 +0,0 @@
|
||||
package servewaf
|
||||
|
||||
// The heuristic trap_paths. It reads SWWAF_TRAP_PATHS.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"slices"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||
)
|
||||
|
||||
// trapPaths refuses a request for a path in SWWAF_TRAP_PATHS: its path as a
|
||||
// path rule sees it, without the query, as sent or once percent-decoded, is one
|
||||
// of them. Such a request is a clear sign of attack, as a ban rule's match is:
|
||||
// it bans the client's netblock, or in observe mode raises the alert for the
|
||||
// ban it would have made. The ban's notes give the path as sent, and so does
|
||||
// its figure.
|
||||
type trapPaths struct{}
|
||||
|
||||
func (trapPaths) name() string { return "trap_paths" }
|
||||
|
||||
func (trapPaths) check(_ context.Context, rq *request, now time.Time) *verdict {
|
||||
path := rules.Path(rq.in)
|
||||
if !slices.Contains(rq.h.config.TrapPaths, path) &&
|
||||
!slices.Contains(rq.h.config.TrapPaths, rules.DecodeOnce(path)) {
|
||||
return nil
|
||||
}
|
||||
|
||||
rq.attack = true
|
||||
rq.banForAttack(now, bans.Notes{TrapPath: path})
|
||||
|
||||
return &verdict{
|
||||
action: requestlog.ActionBanned,
|
||||
figures: map[string]any{"path": path},
|
||||
}
|
||||
}
|
||||
@@ -83,16 +83,6 @@ func TestTrapPathBansAsABanRuleDoes(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestTrapPathRefusalNamesItsHeuristicAndPath(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, _, _ := startWithClock(t, "", map[string]string{trapPaths: trapPathList})
|
||||
|
||||
line := s.request(client, "/wp-login.php?redirect_to=x", http.StatusForbidden,
|
||||
requestlog.ActionBanned)
|
||||
wantHeuristic(t, line, "trap_paths", `{"path": "/wp-login.php"}`)
|
||||
}
|
||||
|
||||
func TestTrapPathMatchesThePathDecodedOnce(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -10,6 +10,8 @@ import (
|
||||
"net/url"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/viper"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/servewaf"
|
||||
)
|
||||
@@ -31,7 +33,7 @@ var errHealthEndpoint = errors.New("the health endpoint answered")
|
||||
// args are the arguments after `healthcheck`; it takes none, and given
|
||||
// one it names it on stderr and returns 1 without checking anything.
|
||||
func HealthCheck(
|
||||
ctx context.Context, args []string, lookupEnv func(string) (string, bool),
|
||||
ctx context.Context, args []string, v *viper.Viper,
|
||||
stderr io.Writer,
|
||||
) int {
|
||||
if len(args) > 0 {
|
||||
@@ -44,15 +46,15 @@ func HealthCheck(
|
||||
ctx, cancel := context.WithTimeout(ctx, healthCheckTimeout)
|
||||
defer cancel()
|
||||
|
||||
command, err := config.Command(lookupEnv)
|
||||
command, err := config.Command(v)
|
||||
|
||||
switch {
|
||||
case err != nil:
|
||||
err = fmt.Errorf("invalid setting: %w", err)
|
||||
case command == "servewaf":
|
||||
err = checkServeWAF(ctx, lookupEnv)
|
||||
err = checkServeWAF(ctx, v)
|
||||
default:
|
||||
err = checkRProxyAndApp(ctx, lookupEnv)
|
||||
err = checkRProxyAndApp(ctx, v)
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
@@ -65,8 +67,8 @@ func HealthCheck(
|
||||
}
|
||||
|
||||
// checkServeWAF asks servewaf's health endpoint.
|
||||
func checkServeWAF(ctx context.Context, lookupEnv func(string) (string, bool)) error {
|
||||
listenAddr, err := config.ServeWAFListenAddr(lookupEnv)
|
||||
func checkServeWAF(ctx context.Context, v *viper.Viper) error {
|
||||
listenAddr, err := config.ServeWAFListenAddr(v)
|
||||
if err != nil {
|
||||
return fmt.Errorf("invalid setting: %w", err)
|
||||
}
|
||||
@@ -81,9 +83,9 @@ func checkServeWAF(ctx context.Context, lookupEnv func(string) (string, bool)) e
|
||||
|
||||
// checkRProxyAndApp asks rproxy's health endpoint, and connects to the app.
|
||||
func checkRProxyAndApp(
|
||||
ctx context.Context, lookupEnv func(string) (string, bool),
|
||||
ctx context.Context, v *viper.Viper,
|
||||
) error {
|
||||
listenAddr, upstreamURL, err := config.RProxyListenAddrAndUpstreamURL(lookupEnv)
|
||||
listenAddr, upstreamURL, err := config.RProxyListenAddrAndUpstreamURL(v)
|
||||
if err != nil {
|
||||
return fmt.Errorf("invalid setting: %w", err)
|
||||
}
|
||||
|
||||
@@ -145,11 +145,7 @@ func TestHealthCheckRefusesAnArgument(t *testing.T) {
|
||||
|
||||
var stderr bytes.Buffer
|
||||
|
||||
noSettings := func(string) (string, bool) {
|
||||
return "", false
|
||||
}
|
||||
|
||||
got := smallwebwaf.HealthCheck(t.Context(), []string{"now"}, noSettings, &stderr)
|
||||
got := smallwebwaf.HealthCheck(t.Context(), []string{"now"}, viperFrom(nil), &stderr)
|
||||
|
||||
want := "smallwebwaf healthcheck: unexpected argument \"now\"\n"
|
||||
if got != 1 || stderr.String() != want {
|
||||
@@ -166,11 +162,7 @@ func wantHealthCheck(t *testing.T, env map[string]string, status int, message st
|
||||
|
||||
var stderr bytes.Buffer
|
||||
|
||||
got := smallwebwaf.HealthCheck(t.Context(), nil, func(name string) (string, bool) {
|
||||
value, ok := env[name]
|
||||
|
||||
return value, ok
|
||||
}, &stderr)
|
||||
got := smallwebwaf.HealthCheck(t.Context(), nil, viperFrom(env), &stderr)
|
||||
|
||||
wrote := stderr.String()
|
||||
if got != status || !strings.HasPrefix(wrote, message) ||
|
||||
|
||||
@@ -23,7 +23,7 @@ func RunRProxy(ctx context.Context, params Params) int {
|
||||
hostname, _ := os.Hostname() // "" when the host has no name to give
|
||||
processLog := requestlog.NewProcessLogger(params.Stdout, hostname, slog.LevelInfo)
|
||||
|
||||
cfg, err := config.RProxyFromEnvironment(params.LookupEnv)
|
||||
cfg, err := config.RProxyFromEnvironment(params.Viper)
|
||||
if err != nil {
|
||||
processLog.Error("invalid setting", "error", err.Error())
|
||||
|
||||
|
||||
@@ -43,12 +43,8 @@ const (
|
||||
func runRProxy(ctx context.Context, env map[string]string, out *output) int {
|
||||
return smallwebwaf.RunRProxy(ctx, smallwebwaf.Params{
|
||||
Version: testVersion,
|
||||
LookupEnv: func(name string) (string, bool) {
|
||||
value, ok := env[name]
|
||||
|
||||
return value, ok
|
||||
},
|
||||
Stdout: out,
|
||||
Viper: viperFrom(env),
|
||||
Stdout: out,
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
@@ -20,6 +20,8 @@ import (
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/viper"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||
@@ -58,8 +60,9 @@ commands:
|
||||
type Params struct {
|
||||
// Version is the version of the binary, set when it is built.
|
||||
Version string
|
||||
// LookupEnv reads an environment variable, normally os.LookupEnv.
|
||||
LookupEnv func(string) (string, bool)
|
||||
// Viper reads the settings, normally the viper config.NewViper returns
|
||||
// for the process's environment.
|
||||
Viper *viper.Viper
|
||||
// Stdout receives the request log and the process's own messages.
|
||||
Stdout io.Writer
|
||||
}
|
||||
@@ -78,7 +81,7 @@ func Main(version string) int {
|
||||
syscall.SIGTERM, os.Interrupt)
|
||||
defer stop()
|
||||
|
||||
params := Params{Version: version, LookupEnv: os.LookupEnv, Stdout: os.Stdout}
|
||||
params := Params{Version: version, Viper: config.NewViper(), Stdout: os.Stdout}
|
||||
|
||||
switch args[0] {
|
||||
case "rproxy":
|
||||
@@ -86,7 +89,7 @@ func Main(version string) int {
|
||||
case "servewaf":
|
||||
return RunServeWAF(ctx, params)
|
||||
case "healthcheck":
|
||||
return HealthCheck(ctx, args[1:], os.LookupEnv, os.Stderr)
|
||||
return HealthCheck(ctx, args[1:], params.Viper, os.Stderr)
|
||||
default:
|
||||
return printUsage(os.Stderr)
|
||||
}
|
||||
@@ -106,9 +109,9 @@ func RunServeWAF(ctx context.Context, params Params) int {
|
||||
// Until the settings are read, the one message is an invalid setting's
|
||||
// error, which every SWWAF_LOG_LEVEL lets through.
|
||||
processLog := requestlog.NewProcessLogger(params.Stdout,
|
||||
config.InstanceName(params.LookupEnv), slog.LevelError)
|
||||
config.InstanceName(params.Viper), slog.LevelError)
|
||||
|
||||
cfg, err := config.FromEnvironment(params.LookupEnv)
|
||||
cfg, err := config.FromEnvironment(params.Viper)
|
||||
if err != nil {
|
||||
processLog.Error("invalid setting", "error", err.Error())
|
||||
|
||||
|
||||
@@ -19,6 +19,8 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/viper"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
|
||||
"sneak.berlin/go/smallwebwaf/internal/servewaf"
|
||||
"sneak.berlin/go/smallwebwaf/internal/smallwebwaf"
|
||||
@@ -122,21 +124,30 @@ func (o *output) text() string {
|
||||
return o.buf.String()
|
||||
}
|
||||
|
||||
// viperFrom returns a viper Set with each of settings, as the viper
|
||||
// config.NewViper returns one bound to a process's environment.
|
||||
func viperFrom(settings map[string]string) *viper.Viper {
|
||||
v := viper.New()
|
||||
for name, value := range settings {
|
||||
v.Set(name, value)
|
||||
}
|
||||
|
||||
return v
|
||||
}
|
||||
|
||||
// run runs `smallwebwaf servewaf` with the settings in env until ctx is done,
|
||||
// and returns its exit status. SWWAF_LOOKUP_SOURCE is off unless env sets it,
|
||||
// so that no test sends GeoJS its clients' addresses.
|
||||
func run(ctx context.Context, env map[string]string, out *output) int {
|
||||
v := viperFrom(env)
|
||||
if _, set := env["SWWAF_LOOKUP_SOURCE"]; !set {
|
||||
v.Set("SWWAF_LOOKUP_SOURCE", "off")
|
||||
}
|
||||
|
||||
return smallwebwaf.RunServeWAF(ctx, smallwebwaf.Params{
|
||||
Version: testVersion,
|
||||
LookupEnv: func(name string) (string, bool) {
|
||||
value, ok := env[name]
|
||||
if !ok && name == "SWWAF_LOOKUP_SOURCE" {
|
||||
return "off", true
|
||||
}
|
||||
|
||||
return value, ok
|
||||
},
|
||||
Stdout: out,
|
||||
Viper: v,
|
||||
Stdout: out,
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user