Compare commits

..
Author SHA1 Message Date
sneak 249d6a1142 sww: keep the stop cut-off test within funlen's 80 lines
check / check (push) Successful in 5m0s
On next, TestRProxyStoppingCutsOffARequestAfterFiveSeconds is at the
limit, and this PR's reformatting of the stand-in's answer adds a line,
so the linter fails on the merged tree. The test now uses the file's
rproxyUpstreamURL instead of declaring the same setting name again.

Model: opus-5-5
2026-11-19 20:54:11 +00:00
sneak 0a0850f797 Move what rproxy and waf send each other into wafprotocol (closes #425)
internal/wafprotocol writes and reads the question, the answer and the
report, and declares once each header, path and limit both daemons use.
rproxy no longer imports internal/waf. Nothing on the wire changes.

Model: opus-5-5
2026-11-19 20:54:11 +00:00
107 changed files with 3863 additions and 5277 deletions

No files matched your search

+9 -13
View File
@@ -288,9 +288,7 @@ account, a key, a token or an alert destination stays off until you give one.
- An anomaly is more requests or bytes than an `SWW_WAF_ANOMALY_` threshold
allows. Crossing a threshold sends one `anomaly` alert to the alert
destinations, and does nothing else: no client is banned, rate limited or
blocked. The thresholds per netblock and per AS number are on by default, set
high enough to alert only on obvious abuse. They are a starting point: tune
them to your own site's traffic, which the metrics show.
blocked.
- An invalid setting stops the process at start with one log line naming the
variable and what is wrong with the value. The settings in effect are logged
on one line at start, with tokens, keys and webhook paths shown as `********`.
@@ -398,16 +396,16 @@ Every setting the code reads, with its default:
| `SWW_WAF_ANOMALY_CLIENT_REQUESTS_PER_HOUR` | `off` | Sends one alert to the alert destinations when one client makes more requests than this in an hour, a count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_CLIENT_BYTES_PER_MINUTE` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for one client in a minute, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_CLIENT_BYTES_PER_HOUR` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for one client in an hour, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_NET_REQUESTS_PER_MINUTE` | `1200` | Sends one alert to the alert destinations when the netblock around a client makes more requests than this in a minute, a count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_NET_REQUESTS_PER_HOUR` | `15000` | Sends one alert to the alert destinations when the netblock around a client makes more requests than this in an hour, a count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_NET_BYTES_PER_MINUTE` | `5G` | Sends one alert to the alert destinations when more bytes than this are counted for the netblock around a client in a minute, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_NET_BYTES_PER_HOUR` | `50G` | Sends one alert to the alert destinations when more bytes than this are counted for the netblock around a client in an hour, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_NET_REQUESTS_PER_MINUTE` | `off` | Sends one alert to the alert destinations when the netblock around a client makes more requests than this in a minute, a count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_NET_REQUESTS_PER_HOUR` | `off` | Sends one alert to the alert destinations when the netblock around a client makes more requests than this in an hour, a count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_NET_BYTES_PER_MINUTE` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for the netblock around a client in a minute, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_NET_BYTES_PER_HOUR` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for the netblock around a client in an hour, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_NET_V4_PREFIX` | `24` | The prefix length of the netblock around an IPv4 client that the `SWW_WAF_ANOMALY_NET_` thresholds count, from 0 to 32. Those thresholds only send alerts. |
| `SWW_WAF_ANOMALY_NET_V6_PREFIX` | `48` | The prefix length of the netblock around an IPv6 client that the `SWW_WAF_ANOMALY_NET_` thresholds count, from 0 to 128. Those thresholds only send alerts. |
| `SWW_WAF_ANOMALY_ASN_REQUESTS_PER_MINUTE` | `6000`, or `off` while the lookup source is `off` | Sends one alert to the alert destinations when one AS number makes more requests than this in a minute, a count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_ASN_REQUESTS_PER_HOUR` | `75000`, or `off` while the lookup source is `off` | Sends one alert to the alert destinations when one AS number makes more requests than this in an hour, a count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_ASN_BYTES_PER_MINUTE` | `25G`, or `off` while the lookup source is `off` | Sends one alert to the alert destinations when more bytes than this are counted for one AS number in a minute, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_ASN_BYTES_PER_HOUR` | `250G`, or `off` while the lookup source is `off` | Sends one alert to the alert destinations when more bytes than this are counted for one AS number in an hour, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_ASN_REQUESTS_PER_MINUTE` | `off` | Sends one alert to the alert destinations when one AS number makes more requests than this in a minute, a count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_ASN_REQUESTS_PER_HOUR` | `off` | Sends one alert to the alert destinations when one AS number makes more requests than this in an hour, a count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_ASN_BYTES_PER_MINUTE` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for one AS number in a minute, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_ASN_BYTES_PER_HOUR` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for one AS number in an hour, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_TOTAL_REQUESTS_PER_MINUTE` | `off` | Sends one alert to the alert destinations when the whole service gets more requests than this in a minute, a count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_TOTAL_REQUESTS_PER_HOUR` | `off` | Sends one alert to the alert destinations when the whole service gets more requests than this in an hour, a count. No client is banned, rate limited or blocked. |
| `SWW_WAF_ANOMALY_TOTAL_BYTES_PER_MINUTE` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for the whole service in a minute, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
@@ -910,8 +908,6 @@ stops the running process (see [`docs/state-files.md`](docs/state-files.md)).
`internal/coreruleset`, `internal/geojs`, `internal/lookup`,
`internal/reputation`, `internal/routewatch`, `internal/anomaly` and
`internal/alerts`: one check or source each.
- `internal/dirwatch`: watches a directory, so that `internal/rules` and
`internal/lookup` read their files again when they change.
- `internal/state`: the state and cache files.
- `internal/requestlog`: the request log.
- `internal/processlog`: the process's own messages.
+40 -49
View File
@@ -69,14 +69,13 @@ never as a time.
that command's settings, and runs `internal/rproxy` until `SIGTERM` or
`SIGINT` (see "`rproxy`"). `sww healthcheck` is the image's health check.
With no command, or one it does not know, the binary prints the commands
and exits with status 2. `sww help` and `sww --help` print them and exit
with status 0. `internal/sww` is the only package that imports
and exits with status 2. `internal/sww` is the only package that imports
`go.uber.org/fx`. fx builds the parts of `waf` and `rproxy`, and starts
and stops their background work.
- `internal/config`: reads the settings, the one place they are read.
- `internal/waf`: `waf`'s server. It runs the checks in `check`. Its
`routes.go` lists `waf`'s own endpoints. It counts bytes, refusals and
anomalies, and writes the log lines.
- `internal/waf`: `waf`'s server. It runs the checks in `check`, and answers
`waf`'s own endpoints in `answerOwnEndpoint`. It counts bytes, refusals
and anomalies, and writes the log lines.
- `internal/rproxy`: the reverse proxy in front of one app that asks `waf`
about each request. It passes a request to the app with the standard
library's `httputil.ReverseProxy`, within the time and size limits.
@@ -97,9 +96,6 @@ never as a time.
imports no other package of this repository: `waf` gives it what it needs
to count and report failures.
- `internal/lookup`: looks clients up in the lookup database.
- `internal/dirwatch`: watches a directory, and calls a function once it has
gone a short time without a change. `internal/rules` and `internal/lookup`
read their files again this way.
- `internal/reputation`: fetches the lists, asks the DNS blocklists and
AbuseIPDB in the background, and keeps their answers. It also fetches
AbuseIPDB's blacklist, which `internal/waf` bans.
@@ -128,13 +124,11 @@ never as a time.
default `SWW_WAF_CORE_RULE_SET_DISABLED_RULES`, are built for that
version. They are checked again before the version changes, since rule
ids, and what each rule matches, change between versions;
- `github.com/go-chi/chi/v5` to route the requests to `waf`'s own endpoints;
- `github.com/hashicorp/golang-lru/v2` for the bounded tables;
- `github.com/oschwald/maxminddb-golang/v2` to read the lookup database;
- `github.com/fsnotify/fsnotify` to notice edited files;
- `github.com/prometheus/client_golang` for the metrics;
- `github.com/spf13/viper` to read the settings from the environment;
- `github.com/spf13/cobra` for the commands and their help text;
- `github.com/maxmind/mmdbwriter`, in the tests, to write lookup databases;
- remote log sending written in this project, since the standard library's
`log/syslog` is frozen and writes only the older syslog format.
@@ -828,8 +822,8 @@ The supported apps are:
- `vaultwarden`;
- `wordpress`.
Each app's fixes are in one file named for the app, such as
`internal/coreruleset/apps/gitea.conf`. gitea's fixes are:
Each app's fixes are in one file, such as `internal/coreruleset/gitea.conf`, and
one entry of `apps` in `internal/coreruleset/coreruleset.go`. gitea's fixes are:
1. Ten parameters are not checked against the Core Rule Set's lists of system
files (930120), shell paths (932160) and command names (932260), on any path:
@@ -862,8 +856,8 @@ Each app's fixes are in one file named for the app, such as
3. On gitea's paths, some fields skip the rules that refused ordinary gitea
requests there. They are the fields where gitea sends a search, a file's name
or text, a commit message, a branch name or a repository's name.
`internal/coreruleset/apps/gitea.conf` lists each rule and the request it
refused. The fields are:
`internal/coreruleset/gitea.conf` lists each rule and the request it refused.
The fields are:
- `q` on gitea's searches of code, of issues and pull requests, and of
commits. A code search holds code, such as `<script>`,
`SELECT * FROM users WHERE` or `../`. The searches of issues and pull
@@ -1048,7 +1042,7 @@ Each app's fixes are in one file named for the app, such as
- The `Content-Range` header of an artifact upload from
`actions/upload-artifact@v3` (920450).
- The passwords in a body, for each rule that random passwords set off. The
numbers are in `internal/coreruleset/apps/gitea.conf`.
numbers are in `internal/coreruleset/gitea.conf`.
- `password` on these paths:
- `/user/login`, `/user/sign_up`, `/user/activate`;
- `/user/link_account_signin` and `/user/link_account_signup`;
@@ -1080,8 +1074,8 @@ Each app's fixes are in one file named for the app, such as
such as `owner/python3-app` would have that push refused.
WordPress's fixes: on WordPress's paths, some fields skip the rules that refused
ordinary WordPress requests there. `internal/coreruleset/apps/wordpress.conf`
lists each rule and the request it refused. The REST API has four addresses:
ordinary WordPress requests there. `internal/coreruleset/wordpress.conf` lists
each rule and the request it refused. The REST API has four addresses:
- `/wp-json/...`
- `/index.php/wp-json/...`
@@ -1173,9 +1167,9 @@ Elsewhere WordPress's fixes do not apply. The paths are those of WordPress at
the site root.
Drupal's fixes: on Drupal's paths, some fields skip the rules that refused
ordinary Drupal requests there. `internal/coreruleset/apps/drupal.conf` lists
each rule and the request it refused. These query fields skip the lists of
system files and command names (930120, 932120, 932250, 932260):
ordinary Drupal requests there. `internal/coreruleset/drupal.conf` lists each
rule and the request it refused. These query fields skip the lists of system
files and command names (930120, 932120, 932250, 932260):
- On every page under `/admin`: `title`, `name`, `user`, `filename`, `filemime`,
`info`, `subject`, `author_name`, `search`, `q` and `destination`.
@@ -1265,10 +1259,10 @@ first path segment of two or three lowercase letters, optionally followed by a
hyphen and more letters, so `/api` counts too.
MediaWiki's fixes: on MediaWiki's paths, some fields skip the rules that refused
ordinary MediaWiki requests there. `internal/coreruleset/apps/mediawiki.conf`
lists each rule and the request it refused. These fields skip the lists of
system files and command names (930120, 932120, 932250, 932260), in the query or
the body:
ordinary MediaWiki requests there. `internal/coreruleset/mediawiki.conf` lists
each rule and the request it refused. These fields skip the lists of system
files and command names (930120, 932120, 932250, 932260), in the query or the
body:
- On MediaWiki's pages, which are `/index.php` and the paths under `/index.php/`
and `/wiki/`:
@@ -1374,9 +1368,9 @@ Elsewhere these fields keep every rule. The paths are those of MediaWiki at the
site root or under `/w/`.
Mastodon's fixes: on Mastodon's paths, some fields skip the rules that refused
ordinary Mastodon requests there. `internal/coreruleset/apps/mastodon.conf`
lists each rule and the request it refused. These fields skip the lists of
system files and command names (930120, 932120, 932250, 932260), in the query:
ordinary Mastodon requests there. `internal/coreruleset/mastodon.conf` lists
each rule and the request it refused. These fields skip the lists of system
files and command names (930120, 932120, 932250, 932260), in the query:
- On the API's search, accounts and admin accounts, under `/api/v1/` and
`/api/v2/`, the searches and names `q`, `acct`, `username`, `display_name`,
@@ -1537,7 +1531,7 @@ every rule.
Vaultwarden's fixes: on Vaultwarden's paths, some fields skip the rules that
refused ordinary Vaultwarden requests there.
`internal/coreruleset/apps/vaultwarden.conf` lists each rule and the request it
`internal/coreruleset/vaultwarden.conf` lists each rule and the request it
refused. The apps encrypt vault items, folders, collections and sends before
sending them, so those fields skip no rules. The fields below hold plain text.
@@ -1588,8 +1582,8 @@ has a parameter of the same name.
`token` in the body of `/admin`, the admin token typed on the admin page, skips
each rule that random passwords set off. The numbers are in
`internal/coreruleset/apps/vaultwarden.conf`. It applies only when the query has
no `token` parameter. `' OR '1'='1`, `../../etc/passwd`, `; cat /etc/passwd` and
`internal/coreruleset/vaultwarden.conf`. It applies only when the query has no
`token` parameter. `' OR '1'='1`, `../../etc/passwd`, `; cat /etc/passwd` and
`<img src=x onerror=x()>` sent as the token reach Vaultwarden;
`<script>alert(1)</script>` and `javascript:alert(1)` are still refused.
@@ -1611,7 +1605,7 @@ Elsewhere these fields keep every rule.
Nextcloud's fixes: on Nextcloud's paths, some fields skip the rules that refused
ordinary Nextcloud requests there, and its WebDAV methods are allowed.
`internal/coreruleset/apps/nextcloud.conf` lists each rule and the request it
`internal/coreruleset/nextcloud.conf` lists each rule and the request it
refused.
The sync client and the Files app use WebDAV on these paths:
@@ -1621,8 +1615,7 @@ The sync client and the Files app use WebDAV on these paths:
- `/remote.php/webdav/`;
- `/public.php/webdav/`.
These methods are allowed there, besides `PUT`, `PATCH` and `DELETE`, which
every app gets:
These methods are allowed there, besides those of the first change:
- `PROPFIND` and `PROPPATCH`;
- `MKCOL`;
@@ -1766,7 +1759,7 @@ gets gitea's fixes too. Forgejo's own fixes hold only what Forgejo sends and
gitea does not.
On Forgejo's paths, some fields skip the rules that refused ordinary Forgejo
requests there. `internal/coreruleset/apps/forgejo.conf` lists each rule and the
requests there. `internal/coreruleset/forgejo.conf` lists each rule and the
request it refused.
`q` skips rules on some of Forgejo's searches. The searches are:
@@ -1812,10 +1805,10 @@ The Core Rule Set reads the request without the `flash` cookie, in which Forgejo
puts what gitea puts in `gitea_flash` (gitea's second fix).
Ghost's fixes: on Ghost's paths, some fields skip the rules that refused
ordinary Ghost requests there. `internal/coreruleset/apps/ghost.conf` lists each
rule and the request it refused. Ghost's admin and integrations call the Admin
API under `/ghost/api/admin/`. Themes and Ghost's scripts call the Content API
under `/ghost/api/content/`, and members call `/members/api/`. The admin calls
ordinary Ghost requests there. `internal/coreruleset/ghost.conf` lists each rule
and the request it refused. Ghost's admin and integrations call the Admin API
under `/ghost/api/admin/`. Themes and Ghost's scripts call the Content API under
`/ghost/api/content/`, and members call `/members/api/`. The admin calls
ActivityPub, a service of its own, under `/.ghost/activitypub/`.
Eight rules refused words that name commands and files, such as `python3`,
@@ -1953,9 +1946,9 @@ That name holds `.profile`, a file on the list of system files, so the name
skips 930120 when a profile is saved.
The passwords skip each rule that random passwords set off. The numbers are in
`internal/coreruleset/apps/ghost.conf`. They are `password` when signing in, and
the passwords sent when creating the first staff user, accepting an invitation,
and resetting or changing a password. `' OR '1'='1`, `../../etc/passwd`,
`internal/coreruleset/ghost.conf`. They are `password` when signing in, and the
passwords sent when creating the first staff user, accepting an invitation, and
resetting or changing a password. `' OR '1'='1`, `../../etc/passwd`,
`; cat /etc/passwd` and `<img src=x onerror=x()>` sent as a password reach
Ghost; `<script>alert(1)</script>` and `javascript:alert(1)` are still refused.
@@ -2113,14 +2106,12 @@ host, sharing one address, would often fetch too close together.
### Anomaly thresholds
These only alert; nothing is blocked. The thresholds per netblock and per AS
number have defaults high enough to alert only on obvious abuse. The others are
off by default. What counts as unusual depends on each service's normal traffic,
which the metrics show, so the defaults are a starting point for the operator to
tune. An alert goes nowhere until the operator sets a destination. The
thresholds reveal a widely distributed scraper, thousands of addresses at low
rates each, which the per-client limits do not see. A low percentage for its
`asn=` tag in `SWW_WAF_TAG_LIMIT_PERCENT` is the response.
These only alert; nothing is blocked. None is set by default, and a threshold
left unset sends no alert. An alert needs a destination only the operator can
supply, and what counts as unusual depends on each service's normal traffic,
which the metrics show. They reveal a widely distributed scraper, thousands of
addresses at low rates each, which the per-client limits do not see. A low
percentage for its `asn=` tag in `SWW_WAF_TAG_LIMIT_PERCENT` is the response.
- Requests and body bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted
per minute and per hour, for each of:
-214
View File
@@ -1,214 +0,0 @@
services:
exampleapp:
# Your app, built from the Dockerfile in this directory, listening
# on port 8081. Publish no port of it: it is reached through rproxy.
build: .
rproxy:
image: git.eeqj.de/sneak/smallwebwaf:latest
environment:
SWW_RPROXY_WAF_BACKEND: http://waf:8080
SWW_RPROXY_UPSTREAM_URL: http://exampleapp:8081
# The netblocks whose X-Forwarded-For rproxy believes. By default
# the private ranges, which cover a TLS-terminating proxy on the
# Docker network. A proxy elsewhere needs its address added; a
# list you give replaces the default, so repeat the private
# ranges if they are still wanted. Set but empty, rproxy believes
# no forwarded header.
# SWW_RPROXY_TRUSTED_PROXIES: 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,203.0.113.5,2001:db8:1::/64
# By default rproxy passes a request to the app while waf gives
# no answer. Uncomment this to answer 503 instead.
# SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION: deny
# The largest request body (default 100M) and response body
# (default 1G): a size with K, M or G, or `off`. Raise them for
# uploads and downloads larger than that, such as git pushes and
# clones or image layers.
# SWW_RPROXY_MAX_REQUEST_SIZE: 2G
# SWW_RPROXY_MAX_RESPONSE_SIZE: 4G
# How long a request may take to reach the app (default 60000,
# one minute) and how long its answer may take to reach the
# visitor (default 1800000, 30 minutes), in milliseconds or
# `off`. Raise the first for slow uploads, the second for long
# downloads.
# SWW_RPROXY_MAX_REQUEST_TIME_MS: "600000"
# SWW_RPROXY_MAX_RESPONSE_TIME_MS: "7200000"
# traefik terminates TLS and forwards to rproxy:8080 over its own
# network, which the `networks` block at the end of this file names.
# No port is published.
networks:
- default
- traefik
labels:
traefik.enable: "true"
# rproxy is on two networks; traefik reaches it on this one.
traefik.docker.network: traefik
traefik.http.routers.exampleapp.rule: Host(`app.example.com`)
traefik.http.routers.exampleapp.entrypoints: websecure
traefik.http.routers.exampleapp.tls.certresolver: letsencrypt
traefik.http.services.exampleapp.loadbalancer.server.port: "8080"
waf:
image: git.eeqj.de/sneak/smallwebwaf:latest
environment:
SWW_COMMAND: waf
# `observe` refuses nothing and makes no ban; each log line says
# what `enforce` (the default) would have done. Start here to
# see what the defaults would refuse on your site.
# SWW_WAF_MODE: observe
# waf's own messages below `info` (the default), while finding
# out why something is refused.
# SWW_WAF_LOG_LEVEL: debug
# Netblocks whose clients skip every check, such as monitoring.
# SWW_WAF_ALLOW_NETS: 192.0.2.10,2001:db8:2::/48
# Netblocks whose clients are refused.
# SWW_WAF_DENY_NETS: 198.51.100.128/25,2001:db8:3::/48
# Netblocks exempt from rate limiting and the byte limits, such
# as the office and an automation host that calls an API. Their
# requests are still checked for attacks.
# SWW_WAF_RATE_LIMIT_EXEMPT_NETS: 203.0.113.0/24,198.51.100.7
# Refuse every request from a country, by its two-letter code.
# This refuses North Korea, Cuba, Russia and Iran. A client the
# lookup cannot place gets through; see SWW_WAF_LOOKUP_SOURCE.
# SWW_WAF_TAG_DENY: country=KP,country=CU,country=RU,country=IR
# Or from a network, by its AS number: here two hosting networks
# that send only scrapers, M247 (AS9009) and WorldStream
# (AS49981). Countries and networks go in the one list together.
# SWW_WAF_TAG_DENY: asn=9009,asn=49981,country=KP,country=RU
# Or let only these countries through; a client the lookup
# cannot place is refused too.
# SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES: us,de
# Address lists that each tag their clients' requests, as
# name=URL pairs; this one tags Tor exit nodes. A tag is refused
# by adding its name to SWW_WAF_TAG_DENY, such as
# `torexit,country=KP`, or given a share of the limits in
# SWW_WAF_TAG_LIMIT_PERCENT, such as `torexit:25`.
# SWW_WAF_TAG_LISTS: torexit=https://check.torproject.org/torbulkexitlist
# A percentage, from 1 to 99, of every limit for a tag: here
# half for DigitalOcean (AS14061), a quarter for OVH (AS16276)
# and a quarter for China. The lowest of a client's tags applies.
# SWW_WAF_TAG_LIMIT_PERCENT: asn=14061:50,asn=16276:25,country=CN:25
# The most requests a client may make in a minute, an hour and a
# day (defaults 120, 1500 and 6000), a count or `off`; the next
# one bans it. Lower them for a small site, raise them for a
# busy API.
# SWW_WAF_RATE_LIMIT_PER_MINUTE: "60"
# SWW_WAF_RATE_LIMIT_PER_HOUR: "600"
# SWW_WAF_RATE_LIMIT_PER_DAY: "3000"
# Path prefixes exempt from rate limiting and the byte limits,
# such as static assets that one page loads many of.
# SWW_WAF_RATE_LIMIT_EXEMPT_PATHS: /assets/,/static/
# Paths only scanners ask for; one request bans the client. Pick
# paths your app never serves.
# SWW_WAF_TRAP_PATHS: /wp-login.php,/xmlrpc.php,/wp-admin/
# How a refused client is answered: 403 (the default), 429 or
# `close`, which drops the connection.
# SWW_WAF_BAN_RESPONSE: "429"
# The ban for a first broken limit (default 1h) and for a first
# clear sign of attack (default 7d): a duration with m, h or d.
# A repeat within SWW_WAF_LIMIT_BAN_REPEAT_WINDOW (default 24h)
# gets a ban three times as long.
# SWW_WAF_LIMIT_BAN_DURATION: 30m
# SWW_WAF_ATTACK_BAN_DURATION: 30d
# The Core Rule Set, which finds attacks in requests. `block`
# (the default) refuses a match, `detect` only logs and alerts
# it, `off` runs no rule. Use `detect` while finding out what it
# would refuse of your app's ordinary requests.
# SWW_WAF_CORE_RULE_SET_MODE: detect
# Path prefixes the Core Rule Set does not inspect, for a part of
# the app that it keeps matching, such as an editor that posts
# HTML or code.
# SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS: /api/editor/,/admin/
# The apps each hostname serves, as hostname:app pairs separated
# by semicolons, so that each hostname gets its own apps' fixes
# to the attack detection and no other's. "Attack detection" in
# the README lists the apps.
# SWW_WAF_APPS: git.example.com:gitea;blog.example.com:wordpress
# Blocklists of addresses and netblocks, text files fetched once
# a day, whose clients are refused.
# SWW_WAF_BLOCKLIST_URLS: https://www.spamhaus.org/drop/drop.txt,https://www.spamhaus.org/drop/dropv6.txt
# With the key of an AbuseIPDB account, a client that has
# committed an offence is checked there in the background; one
# scored 75 or more (SWW_WAF_ABUSEIPDB_MIN_SCORE) gets what
# SWW_WAF_DNSBL_ABUSEIPDB_ACTION says: a quarter of every limit
# by default, or `deny`. At most 900 checks a day
# (SWW_WAF_ABUSEIPDB_DAILY_BUDGET), under the free plan's 1000.
# SWW_WAF_ABUSEIPDB_KEY: api-key-.....
# SWW_WAF_DNSBL_ABUSEIPDB_ACTION: deny
# By default waf looks each client's address up at GeoJS, a web
# service, for its country and AS number. Set `off`, it makes no
# external requests for lookups. Every setting at its default
# works without them; one that needs a lookup, such as a country
# or ASN setting above, then stops the start. Set `file`, it
# reads an IPinfo Lite database instead, mounted by its
# directory as under `volumes` below, so that a replaced file is
# seen.
# SWW_WAF_LOOKUP_SOURCE: "off"
# SWW_WAF_LOOKUP_SOURCE: file
# SWW_WAF_LOOKUP_DB_PATH: /var/lib/ipinfo/ipinfo_lite.mmdb
# Alerts about bans, attacks and failures, to an ntfy topic, a
# Slack incoming webhook or any webhook. The ntfy token is the
# access token ntfy issued for the topic.
# SWW_WAF_ALERT_NTFY_URL: https://ntfy.sh/my-alerts
# SWW_WAF_ALERT_NTFY_TOKEN: hunter2
# SWW_WAF_ALERT_SLACK_WEBHOOK_URL: https://hooks.slack.com/services/...
# Bearer tokens you make up, at least 32 characters each. The
# admin token switches on the ban and client endpoints, through
# which a ban is lifted, your own included, as
# docs/admin-endpoints.md describes. The metrics token switches
# on /_sww/metrics.
# SWW_WAF_ADMIN_TOKEN: hunter2
# SWW_WAF_METRICS_TOKEN: hunter2
# Any setting can be read from a file instead, by adding _FILE
# to its name: the file's contents are the value, less one
# newline at the end. That keeps a secret out of this file and
# out of `docker inspect`. Mount the files as under `volumes`
# below, and set each setting in one form only.
# SWW_WAF_ALERT_NTFY_TOKEN_FILE: /etc/sww/tokens/ntfy
# SWW_WAF_ADMIN_TOKEN_FILE: /etc/sww/tokens/admin
# SWW_WAF_METRICS_TOKEN_FILE: /etc/sww/tokens/metrics
# SWW_WAF_ABUSEIPDB_KEY_FILE: /etc/sww/tokens/abuseipdb
volumes:
# waf's state files: its clients and bans, kept across restarts.
- waf-state:/var/lib/sww
# Token files, each owned by uid 65532 (the sww user) with mode
# 0400, for the _FILE settings above.
# - /srv/sww/tokens:/etc/sww/tokens:ro
# The IPinfo Lite database, for SWW_WAF_LOOKUP_SOURCE=file.
# - /srv/ipinfo:/var/lib/ipinfo:ro
volumes:
waf-state:
# traefik's network, made outside this file.
networks:
traefik:
external: true
+2 -19
View File
@@ -378,9 +378,8 @@ Crossing a threshold sends one `anomaly` alert to the destinations under
Each scope has four thresholds: the most requests and the most body bytes
counted in a minute and in an hour. The bytes are those `SWW_WAF_BYTES_COUNTED`
names. Requests are counts, bytes are sizes. Each is `off` by default, except
those per netblock and per AS number listed below. The four thresholds per
client are:
names. Requests are counts, bytes are sizes. Each is `off` by default. The four
thresholds per client are:
- `SWW_WAF_ANOMALY_CLIENT_REQUESTS_PER_MINUTE`;
- `SWW_WAF_ANOMALY_CLIENT_REQUESTS_PER_HOUR`;
@@ -404,22 +403,6 @@ as `REQUESTS_PER_MINUTE`:
`office=203.0.113.0/24,scraper-x=198.51.100.0/22`. An item without `=` or a
name, or with a bad netblock, is refused. So is a name used twice.
The thresholds per netblock and per AS number have defaults set high enough to
alert only on obvious abuse:
- per netblock:
- `REQUESTS_PER_MINUTE` `1200` and `REQUESTS_PER_HOUR` `15000`, ten clients
at the default rate limits;
- `BYTES_PER_HOUR` `50G`, one client at the default byte limit per hour;
- `BYTES_PER_MINUTE` `5G`, a tenth of that.
- per AS number, five times those: `6000`, `75000`, `25G` and `250G`.
While `SWW_WAF_LOOKUP_SOURCE` is `off`, no request has an AS number, so the
thresholds per AS number are `off` by default. Setting one then stops the start.
These defaults are a starting point. Tune them to your own site's traffic, which
the metrics show.
## Settings given as files
To give a token as a file, as "Tokens" under "Deployment" in
-3
View File
@@ -7,12 +7,10 @@ require (
github.com/corazawaf/coraza-coreruleset/v4 v4.25.0
github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68
github.com/fsnotify/fsnotify v1.10.1
github.com/go-chi/chi/v5 v5.5.0
github.com/hashicorp/golang-lru/v2 v2.0.7
github.com/maxmind/mmdbwriter v1.2.0
github.com/oschwald/maxminddb-golang/v2 v2.7.0
github.com/prometheus/client_golang v1.24.1
github.com/spf13/cobra v1.10.2
github.com/spf13/viper v1.21.0
go.uber.org/fx v1.24.0
golang.org/x/net v0.60.0
@@ -28,7 +26,6 @@ require (
github.com/goccy/go-yaml v1.19.2 // indirect
github.com/gotnospirit/makeplural v0.0.0-20180622080156-a5f48d94d976 // indirect
github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/kaptinlin/go-i18n v0.1.4 // indirect
github.com/kaptinlin/jsonschema v0.4.6 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
-11
View File
@@ -14,15 +14,12 @@ github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68 h1:KLEjvBBSw
github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68/go.mod h1:9XocblQEwK6EYylRth7PrPIifF2l0iSoYo4FAXhXU/8=
github.com/corazawaf/libinjection-go v0.3.4 h1:jX2Tit2xZxYJJyVryNXucYBZtQqS7473x9Nc6lgRRbk=
github.com/corazawaf/libinjection-go v0.3.4/go.mod h1:Ik/+w3UmTWH9yn366RgS9D95K3y7Atb5m/H/gXzzPCk=
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/foxcpp/go-mockdns v1.2.0 h1:omK3OrHRD1IWJz1FuFBCFquhXslXoF17OvBS6JPzZF0=
github.com/foxcpp/go-mockdns v1.2.0/go.mod h1:IhLeSFGed3mJIAXPH2aiRQB+kqz7oqu8ld2qVbOu7Wk=
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho=
github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo=
github.com/go-chi/chi/v5 v5.5.0 h1:xdKQ1dJmVJsmgahrXSKhUx8NyVPzpwdlyrrQFpMPJgg=
github.com/go-chi/chi/v5 v5.5.0/go.mod h1:d1IG7GtKPSjPyBP365PH8sAGVHk3qfB/IhjFNRH1wWM=
github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
@@ -37,8 +34,6 @@ github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 h1:c7gcN
github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092/go.mod h1:ZZAN4fkkful3l1lpJwF8JbW41ZiG9TwJ2ZlqzQovBNU=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/jcchavezs/mergefs v0.1.1 h1:D45R17m6dHnSVZefnhynoeZvcK2Uw0oTrRfoUOQ0S5Y=
github.com/jcchavezs/mergefs v0.1.1/go.mod h1:eRLTrsA+vFwQZ48hj8p8gki/5v9C2bFtHH5Mnn4bcGk=
github.com/kaptinlin/go-i18n v0.1.4 h1:wCiwAn1LOcvymvWIVAM4m5dUAMiHunTdEubLDk4hTGs=
@@ -77,7 +72,6 @@ github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+
github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY=
github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/sagikazarmark/locafero v0.11.0 h1:1iurJgmM9G3PA/I+wWYIOw/5SyBtxapeHDcg+AAIFXc=
github.com/sagikazarmark/locafero v0.11.0/go.mod h1:nVIGvgyzw595SUSUE6tvCp3YYTeHs15MvlmU87WwIik=
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 h1:+jumHNA0Wrelhe64i8F6HNlS8pkoyMv5sreGx2Ry5Rw=
@@ -86,9 +80,6 @@ github.com/spf13/afero v1.15.0 h1:b/YBCLWAJdFWJTN9cLhiXXcD7mzKn9Dm86dNnfyQw1I=
github.com/spf13/afero v1.15.0/go.mod h1:NC2ByUVxtQs4b3sIUphxK0NioZnmxgyCrfzeuq8lxMg=
github.com/spf13/cast v1.10.0 h1:h2x0u2shc1QuLHfxi+cTJvs30+ZAHOGRic8uyGTDWxY=
github.com/spf13/cast v1.10.0/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqeHo=
github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.21.0 h1:x5S+0EU27Lbphp4UKm1C+1oQO+rKx36vfCoaVebLFSU=
@@ -118,7 +109,6 @@ go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba h1:0b9z3AuHCjxk0x/opv64kcgZLBseWJUpBw5I82+2U4M=
@@ -137,7 +127,6 @@ golang.org/x/tools v0.50.0 h1:c2ifzfcuY7L90lZ2aKd8S4K2NpASF08SZx9ZuJkHmSU=
golang.org/x/tools v0.50.0/go.mod h1:7ulVMw3831Mwi5EZD6RomGyffr4VFjuNYXf2BbCEAV0=
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
rsc.io/binaryregexp v0.2.0 h1:HfqmD5MEmC0zvwBuF187nq9mdnXjXsSivRiXN7SmRkE=
+42 -55
View File
@@ -34,76 +34,65 @@ import (
"time"
)
// Event is what an alert is for, such as EventBan.
type Event string
// The events an alert is for, as SWW_WAF_ALERT_EVENTS names them.
const (
// EventBan is a ban sww made.
EventBan Event = "ban"
EventBan = "ban"
// EventPermanentBan is a permanent ban sww made, or a ban it
// made permanent.
EventPermanentBan Event = "permanent_ban"
EventPermanentBan = "permanent_ban"
// EventAnomaly is a count of requests or bytes over an anomaly
// threshold.
EventAnomaly Event = "anomaly"
EventAnomaly = "anomaly"
// EventCoreRuleSetMatch is a request the Core Rule Set scored at or over
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD, refused in block mode, let through in
// detect mode.
EventCoreRuleSetMatch Event = "core_rule_set_match"
EventCoreRuleSetMatch = "core_rule_set_match"
// EventReputationHit is a request whose client a blocklist or a DNSBL
// zone lists, or whose AbuseIPDB score is a hit.
EventReputationHit Event = "reputation_hit"
EventReputationHit = "reputation_hit"
// EventSourceFailure is GeoJS failing or refusing sww, a fetch
// of a list failing, a query to a DNSBL zone or a check with AbuseIPDB
// failing or refused, or the day's AbuseIPDB checks used up.
EventSourceFailure Event = "source_failure"
EventSourceFailure = "source_failure"
// EventFileError is one of:
// - a rule file or state file edited while sww runs that does not parse;
// - a cache file that does not parse, at start or after an edit;
// - a state file or cache file that cannot be written;
// - a replacement of the lookup database that cannot be read.
EventFileError Event = "file_error"
EventFileError = "file_error"
// EventSummary is the summary sent as an hour ends: of the alerts held
// back in it past SWW_WAF_ALERT_MAX_PER_HOUR, and of the repeats held
// back by the cooldowns dropped as it ends, which no alert let through
// has given. It is sent with SWW_WAF_ALERT_MAX_PER_HOUR off too, for
// those repeats. SWW_WAF_ALERT_EVENTS does not name it.
EventSummary Event = "summary"
EventSummary = "summary"
)
// Events returns every event SWW_WAF_ALERT_EVENTS can name, which is its
// default.
func Events() []Event {
return []Event{
func Events() []string {
return []string{
EventBan, EventPermanentBan, EventCoreRuleSetMatch, EventAnomaly,
EventReputationHit, EventSourceFailure, EventFileError,
}
}
// DestinationName is the name of a destination alerts are sent to:
// DestinationWebhook, DestinationSlack or DestinationNtfy.
type DestinationName string
// The destinations alerts are sent to, as the metrics and alerts.json
// name them.
const (
// DestinationWebhook is the webhook SWW_WAF_ALERT_WEBHOOK_URL names.
DestinationWebhook DestinationName = "webhook"
DestinationWebhook = "webhook"
// DestinationSlack is the Slack incoming webhook
// SWW_WAF_ALERT_SLACK_WEBHOOK_URL names.
DestinationSlack DestinationName = "slack"
DestinationSlack = "slack"
// DestinationNtfy is the ntfy topic SWW_WAF_ALERT_NTFY_URL names.
DestinationNtfy DestinationName = "ntfy"
DestinationNtfy = "ntfy"
)
// Destinations returns every destination alerts can be sent to.
func Destinations() []DestinationName {
return []DestinationName{
DestinationWebhook,
DestinationSlack,
DestinationNtfy,
}
func Destinations() []string {
return []string{DestinationWebhook, DestinationSlack, DestinationNtfy}
}
const (
@@ -146,7 +135,7 @@ type Params struct {
NtfyURL *url.URL
NtfyToken string
// Events are the events alerts are sent for (SWW_WAF_ALERT_EVENTS).
Events []Event
Events []string
// Cooldown is how long a repeat of an alert is held back
// (SWW_WAF_ALERT_COOLDOWN), 0 for no time. MaxPerHour is the most alerts
// sent in an hour (SWW_WAF_ALERT_MAX_PER_HOUR), 0 for no limit.
@@ -170,7 +159,7 @@ type Params struct {
type Alert struct {
Instance string `json:"instance"`
Time time.Time `json:"time"`
Event Event `json:"event"`
Event string `json:"event"`
Host string `json:"host,omitempty"`
Client netip.Addr `json:"client"`
Netblock netip.Prefix `json:"netblock"`
@@ -197,7 +186,7 @@ type Alert struct {
//
//nolint:tagliatelle // the state files use snake_case, as the request log does
type Cooldown struct {
Event Event `json:"event"`
Event string `json:"event"`
Netblock netip.Prefix `json:"netblock"`
File string `json:"file,omitempty"`
Source string `json:"source,omitempty"`
@@ -214,18 +203,18 @@ type Cooldown struct {
//
//nolint:tagliatelle // the state files use snake_case, as the request log does
type Hour struct {
Start time.Time `json:"start"`
Sent int `json:"sent"`
HeldBack map[Event]int `json:"held_back"`
Start time.Time `json:"start"`
Sent int `json:"sent"`
HeldBack map[string]int `json:"held_back"`
}
// State is what alerts.json holds: the cooldowns, the hour under way, and
// for each destination set, the alerts waiting to be sent to it, oldest
// first.
type State struct {
Cooldowns []Cooldown `json:"cooldowns"`
Hour Hour `json:"hour"`
Waiting map[DestinationName][]Alert `json:"waiting"`
Cooldowns []Cooldown `json:"cooldowns"`
Hour Hour `json:"hour"`
Waiting map[string][]Alert `json:"waiting"`
}
// Counts are, for a destination, how many alerts it took, how many
@@ -259,7 +248,7 @@ type destination struct {
// name is how the metrics and alerts.json name the destination, and
// setting the setting that is its URL, which the log names in place
// of the URL.
name DestinationName
name string
setting string
url *url.URL
// message returns the body an alert is posted with, and the headers
@@ -284,7 +273,7 @@ type destination struct {
// scope with the same AS number or name, as its detail names them. Each
// is empty for an alert without one.
type cooldownKey struct {
event Event
event string
netblock netip.Prefix
file string
source string
@@ -317,7 +306,7 @@ func New(params Params) *Queue {
q := &Queue{
params: params,
cooldowns: map[cooldownKey]*Cooldown{},
hour: Hour{HeldBack: map[Event]int{}},
hour: Hour{HeldBack: map[string]int{}},
}
if params.WebhookURL != nil {
@@ -388,7 +377,7 @@ func (q *Queue) Raise(alert Alert) {
// event, no alert for event on netblock was let through less than
// Cooldown before, and fewer than MaxPerHour alerts have been let through
// in the hour under way. Unlike Raise, it counts nothing.
func (q *Queue) WouldSend(event Event, netblock netip.Prefix) bool {
func (q *Queue) WouldSend(event string, netblock netip.Prefix) bool {
if len(q.destinations) == 0 || !slices.Contains(q.params.Events, event) {
return false
}
@@ -444,7 +433,7 @@ func (q *Queue) Run(ctx context.Context) {
// Counts returns the counts of the destination name, all 0 for one not
// set.
func (q *Queue) Counts(name DestinationName) Counts {
func (q *Queue) Counts(name string) Counts {
for _, d := range q.destinations {
if d.name == name {
return Counts{
@@ -460,8 +449,8 @@ func (q *Queue) Counts(name DestinationName) Counts {
// DestinationsSet returns the destinations set, in the order of
// Destinations.
func (q *Queue) DestinationsSet() []DestinationName {
names := make([]DestinationName, 0, len(q.destinations))
func (q *Queue) DestinationsSet() []string {
names := make([]string, 0, len(q.destinations))
for _, d := range q.destinations {
names = append(names, d.name)
}
@@ -485,7 +474,7 @@ func (q *Queue) Snapshot() State {
state := State{
Cooldowns: make([]Cooldown, 0, len(q.cooldowns)),
Hour: q.hour,
Waiting: map[DestinationName][]Alert{},
Waiting: map[string][]Alert{},
}
state.Hour.HeldBack = maps.Clone(q.hour.HeldBack)
@@ -536,7 +525,7 @@ func (q *Queue) Load(state State) {
q.hour.HeldBack = maps.Clone(state.Hour.HeldBack)
if q.hour.HeldBack == nil {
q.hour.HeldBack = map[Event]int{}
q.hour.HeldBack = map[string]int{}
}
for _, d := range q.destinations {
@@ -548,7 +537,7 @@ func (q *Queue) Load(state State) {
// its URL, that URL, target, and message, which makes the messages sent
// to it.
func (q *Queue) addDestination(
name DestinationName, setting string, target *url.URL,
name, setting string, target *url.URL,
message func(alert *Alert) ([]byte, http.Header, error),
) {
q.destinations = append(q.destinations, &destination{
@@ -660,7 +649,7 @@ func (q *Queue) endHour(now time.Time) {
})
}
q.hour = Hour{Start: start, HeldBack: map[Event]int{}}
q.hour = Hour{Start: start, HeldBack: map[string]int{}}
}
// queue adds alert to the alerts waiting for each destination.
@@ -915,21 +904,20 @@ func (q *Queue) ntfyMessage(alert *Alert) ([]byte, http.Header, error) {
// ntfyPriority returns the priority an alert for event is published to
// ntfy with: high for an event the admin needs to look at.
func ntfyPriority(event Event) string {
func ntfyPriority(event string) string {
switch event {
case EventPermanentBan, EventAnomaly, EventSourceFailure, EventFileError:
return "high"
case EventReputationHit:
return "low"
case EventBan, EventCoreRuleSetMatch, EventSummary:
default: // ban, core_rule_set_match and summary
return "default"
}
return "default"
}
// ntfyTag returns the tag an alert for event is published to ntfy with,
// which ntfy shows as an emoji.
func ntfyTag(event Event) string {
func ntfyTag(event string) string {
switch event {
case EventBan, EventPermanentBan:
return "no_entry"
@@ -941,16 +929,15 @@ func ntfyTag(event Event) string {
return "label"
case EventSourceFailure, EventFileError:
return "warning"
case EventSummary:
default: // summary
return "bar_chart"
}
return "bar_chart"
}
// title returns the title of alert in Slack and ntfy: the instance and
// the event.
func title(alert *Alert) string {
return alert.Instance + ": " + string(alert.Event)
return alert.Instance + ": " + alert.Event
}
// text returns the text of alert in Slack and ntfy: its reason, then a
+20 -31
View File
@@ -126,7 +126,7 @@ func TestOnlyTheChosenEventsAreSent(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
params := newParams()
params.Events = []alerts.Event{
params.Events = []string{
alerts.EventSourceFailure,
alerts.EventFileError,
}
@@ -156,7 +156,7 @@ func TestWouldSendOnlyForTheChosenEvents(t *testing.T) {
t.Parallel()
params := newParams()
params.Events = []alerts.Event{
params.Events = []string{
alerts.EventSourceFailure,
alerts.EventFileError,
}
@@ -186,8 +186,8 @@ func TestRaiseDoesNothingWithoutADestination(t *testing.T) {
// No alert waits, no cooldown has started, and the hour counts none.
want := alerts.State{
Cooldowns: []alerts.Cooldown{},
Hour: alerts.Hour{HeldBack: map[alerts.Event]int{}},
Waiting: map[alerts.DestinationName][]alerts.Alert{},
Hour: alerts.Hour{HeldBack: map[string]int{}},
Waiting: map[string][]alerts.Alert{},
}
if got := q.Snapshot(); !reflect.DeepEqual(got, want) {
t.Errorf("state %+v, want %+v", got, want)
@@ -235,7 +235,7 @@ func TestRepeatWithinTheCooldownIsHeldBackAndCountedInTheNext(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
webhook, q := start(t, newParams())
raise := func(event alerts.Event, n int) {
raise := func(event string, n int) {
q.Raise(alerts.Alert{Event: event, Netblock: netblock(n)})
}
@@ -1014,7 +1014,7 @@ func TestNtfyTokenIsSentToNtfyAlone(t *testing.T) {
)
synctest.Wait()
for destination, want := range map[alerts.DestinationName][]string{
for destination, want := range map[string][]string{
alerts.DestinationWebhook: nil,
alerts.DestinationSlack: nil,
alerts.DestinationNtfy: tc.want,
@@ -1063,7 +1063,7 @@ func TestDestinationThatDoesNotAnswerHoldsUpNeitherOther(t *testing.T) {
// With no time passed, the webhook and ntfy have taken every alert,
// while Slack has not answered the first.
for destination, want := range map[alerts.DestinationName]alerts.Counts{
for destination, want := range map[string]alerts.Counts{
alerts.DestinationWebhook: {Sent: 3},
alerts.DestinationSlack: {},
alerts.DestinationNtfy: {Sent: 3},
@@ -1126,7 +1126,7 @@ func TestDropsAreCountedForEachDestination(t *testing.T) {
// the webhook takes every one.
const all = alerts.QueueSize + 1
for destination, want := range map[alerts.DestinationName]alerts.Counts{
for destination, want := range map[string]alerts.Counts{
alerts.DestinationWebhook: {Sent: all},
alerts.DestinationSlack: {Dropped: 1},
alerts.DestinationNtfy: {SendFailures: all, Dropped: all},
@@ -1152,21 +1152,11 @@ func TestEachDestinationIsSentOnlyTheAlertsWaitingForIt(t *testing.T) {
// As read from alerts.json. The alert waiting for the webhook, which
// is not set, is dropped.
q.Load(
roundTrip(
t,
alerts.State{
Waiting: map[alerts.DestinationName][]alerts.Alert{
alerts.DestinationWebhook: {waiting("first")},
alerts.DestinationSlack: {
waiting("second"),
waiting("third"),
},
alerts.DestinationNtfy: {waiting("fourth")},
},
},
),
)
q.Load(roundTrip(t, alerts.State{Waiting: map[string][]alerts.Alert{
alerts.DestinationWebhook: {waiting("first")},
alerts.DestinationSlack: {waiting("second"), waiting("third")},
alerts.DestinationNtfy: {waiting("fourth")},
}}))
synctest.Wait()
wantBodies(t, standIns[alerts.DestinationSlack],
@@ -1175,7 +1165,7 @@ func TestEachDestinationIsSentOnlyTheAlertsWaitingForIt(t *testing.T) {
wantBodies(t, standIns[alerts.DestinationNtfy], "fourth")
// alerts.json then lists Slack and ntfy alone, with no alert waiting.
want := map[alerts.DestinationName][]alerts.Alert{
want := map[string][]alerts.Alert{
alerts.DestinationSlack: {}, alerts.DestinationNtfy: {},
}
if got := q.Snapshot().Waiting; !reflect.DeepEqual(got, want) {
@@ -1344,11 +1334,11 @@ func start(t *testing.T, params alerts.Params) (*standIn, *alerts.Queue) {
func startAll(
t *testing.T,
params alerts.Params,
) (map[alerts.DestinationName]*standIn, *alerts.Queue) {
) (map[string]*standIn, *alerts.Queue) {
t.Helper()
q := alerts.New(params)
standIns := map[alerts.DestinationName]*standIn{}
standIns := map[string]*standIn{}
for _, destination := range q.DestinationsSet() {
standIns[destination] = &standIn{answers: answering}
@@ -1414,14 +1404,14 @@ func wantAlert(t *testing.T, got, want map[string]any) {
// wantEvents checks the events of the alerts the webhook was sent, in
// order.
func wantEvents(t *testing.T, webhook *standIn, want ...alerts.Event) {
func wantEvents(t *testing.T, webhook *standIn, want ...string) {
t.Helper()
got := make([]alerts.Event, 0, len(webhook.received()))
got := make([]string, 0, len(webhook.received()))
for _, request := range webhook.received() {
event, _ := request.alert["event"].(string)
got = append(got, alerts.Event(event))
got = append(got, event)
}
if !slices.Equal(got, want) {
@@ -1457,8 +1447,7 @@ func wantCounts(
// wantDestinationCounts checks q's counts for destination.
func wantDestinationCounts(
t *testing.T, q *alerts.Queue, destination alerts.DestinationName,
want alerts.Counts,
t *testing.T, q *alerts.Queue, destination string, want alerts.Counts,
) {
t.Helper()
+1 -3
View File
@@ -7,9 +7,7 @@ const QueueSize = queueSize
// SetTransport has q's requests to the destination name go through
// transport instead of the network.
func (q *Queue) SetTransport(
name DestinationName, transport http.RoundTripper,
) {
func (q *Queue) SetTransport(name string, transport http.RoundTripper) {
for _, d := range q.destinations {
if d.name == name {
d.httpClient.Transport = transport
+8 -10
View File
@@ -295,10 +295,9 @@ func (c *Counters) netAround(client netip.Addr) netip.Prefix {
// overThreshold is a count over its threshold: what it counts, requests or
// bytes, its window, the count and the threshold.
type overThreshold struct {
kind ratelimit.LimitKind
window string
count float64
threshold int64
kind, window string
count float64
threshold int64
}
// add counts a request of bytes at now in each of c's counts whose
@@ -347,10 +346,9 @@ func (c *Counter) add(
// bucketCount is one of a counter's four counts: requests or bytes, in a
// window of length, and the buckets they are counted in.
type bucketCount struct {
kind ratelimit.LimitKind
window string
length time.Duration
buckets *ratelimit.Buckets
kind, window string
length time.Duration
buckets *ratelimit.Buckets
}
// counts returns c's counts: requests and bytes in the minute, then in
@@ -406,8 +404,8 @@ func (k key) counter() *Counter {
// counted, the count and the threshold.
func alertFor(r Request, k key, o overThreshold) alerts.Alert {
detail := map[string]any{
"scope": k.scope, "window": o.window, "kind": string(o.kind),
"count": o.count, "threshold": o.threshold,
"scope": k.scope, "window": o.window, "kind": o.kind, "count": o.count,
"threshold": o.threshold,
}
var counted string
+17 -24
View File
@@ -16,26 +16,20 @@ import (
"time"
"github.com/hashicorp/golang-lru/v2/simplelru"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
)
// Cause is why a ban was made: CauseLimit, CauseAttack, CauseAdmin or
// CauseAbuseIPDB.
type Cause string
// The causes of bans.
const (
// CauseLimit is a ban sww made for a broken limit.
CauseLimit Cause = "limit"
CauseLimit = "limit"
// CauseAttack is a ban sww made for a clear sign of attack.
CauseAttack Cause = "attack"
CauseAttack = "attack"
// CauseAdmin is a ban an admin made, or one sww made that an
// admin keeps. It is never dropped.
CauseAdmin Cause = "admin"
CauseAdmin = "admin"
// CauseAbuseIPDB is a ban sww made for a client on AbuseIPDB's
// blacklist. It is forgotten once it has ended.
CauseAbuseIPDB Cause = "abuseipdb"
CauseAbuseIPDB = "abuseipdb"
)
// repeatFactor is how many times as long as the netblock's last ban a ban
@@ -74,7 +68,8 @@ type Ban struct {
Start time.Time
// Expires is when the ban ends, zero for a permanent ban.
Expires time.Time
Cause Cause
// Cause is CauseLimit, CauseAttack, CauseAdmin or CauseAbuseIPDB.
Cause string
// Reason is a short text: for a ban sww made, the limit broken, the
// rule that matched or AbuseIPDB's score; for an admin's, what the
// admin wrote.
@@ -116,10 +111,10 @@ type Notes struct {
// request that broke the limit included.
// These are what counted toward the ban, and the window is the time
// over which they came.
Kind ratelimit.LimitKind `json:"kind,omitempty"`
Limit int64 `json:"limit,omitempty"`
Window string `json:"window,omitempty"`
Count float64 `json:"count,omitempty"`
Kind string `json:"kind,omitempty"`
Limit int64 `json:"limit,omitempty"`
Window string `json:"window,omitempty"`
Count float64 `json:"count,omitempty"`
// LimitPercent and LimitPercentSetting are, for a ban for a limit a
// biased threshold lowered, the client's percentage of that kind of
// limit, of which Limit is the result, and the setting that gave it.
@@ -199,7 +194,7 @@ type Ledger struct {
// made is how many bans have been made since the start, by cause: by
// the ledger, and by an admin, through BanForAdmin or in an edit of
// bans.json.
made map[Cause]int
made map[string]int
// v4Lengths and v6Lengths are the lengths of the IPv4 and IPv6
// netblocks that have been banned. Check looks for a ban at each of
// them, so that a ban read from bans.json refuses every client in its
@@ -226,7 +221,7 @@ func New(params Params) *Ledger {
params: params,
changed: make(chan struct{}, 1),
netblocks: netblocks,
made: map[Cause]int{},
made: map[string]int{},
}
}
@@ -308,7 +303,7 @@ func activeBan(bans []Ban, now time.Time) *Ban {
// Grounds are why a request makes a ban: its cause, CauseLimit or
// CauseAttack, and its notes.
type Grounds struct {
Cause Cause
Cause string
Notes Notes
}
@@ -350,7 +345,7 @@ func (l *Ledger) WouldBan(
// or CauseAttack, made at now would be permanent, as Ban would make it. It
// works out nothing else of the ban.
func (l *Ledger) WouldBePermanent(
netblock netip.Prefix, now time.Time, cause Cause,
netblock netip.Prefix, now time.Time, cause string,
) bool {
l.mu.Lock()
defer l.mu.Unlock()
@@ -444,7 +439,7 @@ func (l *Ledger) BanForAbuseIPDB(
// The source is named as the request log names AbuseIPDB.
notes.Reputation = []ReputationHit{
{Source: string(CauseAbuseIPDB), Score: &score},
{Source: CauseAbuseIPDB, Score: &score},
}
l.add(Ban{
@@ -582,7 +577,7 @@ func (l *Ledger) AddLookup(
// for CauseLimit, CauseAttack and CauseAbuseIPDB, by the ledger; for
// CauseAdmin, by an admin, with BanForAdmin or in an edit of bans.json, as
// LoadEdit counts them. The bans read from bans.json at the start are not among them.
func (l *Ledger) Made(cause Cause) int {
func (l *Ledger) Made(cause string) int {
l.mu.Lock()
defer l.mu.Unlock()
@@ -775,10 +770,8 @@ func (l *Ledger) ban(
case CauseAttack:
ban.Reason = attackReason(notes)
ban.Expires = l.attackExpiry(held, now)
case CauseAdmin, CauseAbuseIPDB:
fallthrough
default:
panic("bans: a request's ban has the cause " + string(grounds.Cause))
panic("bans: a request's ban has the cause " + grounds.Cause)
}
if !keep {
+5 -10
View File
@@ -10,8 +10,6 @@ import (
"time"
"github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// RProxy is the settings of `sww rproxy`. A size or time limit of zero
@@ -34,7 +32,7 @@ type RProxy struct {
// cannot be connected to, does not answer at all, or does not answer
// within WAFBackendTimeout: allow or deny
// (SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION).
WAFBackendNoAnswerDisposition wafprotocol.Disposition
WAFBackendNoAnswerDisposition string
// MaxRequestSize is the largest request body (SWW_RPROXY_MAX_REQUEST_SIZE), and
// MaxResponseSize the largest response body (SWW_RPROXY_MAX_RESPONSE_SIZE).
MaxRequestSize int64
@@ -96,7 +94,7 @@ func RProxyFromEnvironment(v *viper.Viper) (*RProxy, error) {
),
WAFBackendNoAnswerDisposition: env.disposition(
"SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION",
wafprotocol.DispositionAllow,
"allow",
),
MaxRequestSize: env.size("SWW_RPROXY_MAX_REQUEST_SIZE", "100M"),
MaxResponseSize: env.size("SWW_RPROXY_MAX_RESPONSE_SIZE", "1G"),
@@ -168,12 +166,9 @@ func (e *environment) wafBackend(name string) *url.URL {
// disposition reads the setting that is what is done with a request when waf
// cannot be connected to, does not answer at all, or does not answer in time:
// allow or deny.
func (e *environment) disposition(
name string, defaultValue wafprotocol.Disposition,
) wafprotocol.Disposition {
value := wafprotocol.Disposition(e.value(name, string(defaultValue)))
if value != wafprotocol.DispositionAllow &&
value != wafprotocol.DispositionDeny {
func (e *environment) disposition(name, defaultValue string) string {
value := e.value(name, defaultValue)
if value != "allow" && value != "deny" {
e.check(name, fmt.Errorf("%q %w", value, errNotDisposition))
}
+1458 -1682
View File
File diff suppressed because it is too large. Load diff
+3 -56
View File
@@ -20,7 +20,6 @@ import (
"github.com/spf13/viper"
"golang.org/x/net/http/httpguts"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/anomaly"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
@@ -178,17 +177,6 @@ func loggedAnomalyDefaults() map[string]string {
logged[name] = off
}
maps.Copy(logged, map[string]string{
anomalyNet + requestsPerMinute: "1200",
anomalyNet + requestsPerHour: "15000",
anomalyNet + bytesPerMinute: "5G",
anomalyNet + bytesPerHour: "50G",
anomalyASN + requestsPerMinute: "6000",
anomalyASN + requestsPerHour: "75000",
anomalyASN + bytesPerMinute: "25G",
anomalyASN + bytesPerHour: "250G",
})
return logged
}
@@ -200,16 +188,6 @@ const (
defaultAlertCooldown = "15m"
)
// joinEvents returns events as SWW_WAF_ALERT_EVENTS lists them.
func joinEvents(events []alerts.Event) string {
names := make([]string, 0, len(events))
for _, event := range events {
names = append(names, string(event))
}
return strings.Join(names, ",")
}
// defaultIPReputationCacheTTL is the default of SWW_WAF_IP_REPUTATION_CACHE_TTL.
const defaultIPReputationCacheTTL = "24h"
@@ -1132,7 +1110,7 @@ func TestAlertSettingsDefaults(t *testing.T) {
if cfg.AlertWebhookURL != nil || len(cfg.AlertWebhookHeaders) != 0 ||
cfg.AlertSlackWebhookURL != nil || cfg.AlertNtfyURL != nil ||
cfg.AlertNtfyToken != "" ||
joinEvents(cfg.AlertEvents) != defaultAlertEvents ||
strings.Join(cfg.AlertEvents, ",") != defaultAlertEvents ||
cfg.AlertCooldown != 15*time.Minute || cfg.AlertMaxPerHour != 60 {
t.Errorf(
"alert settings %v, %v, %v, %v, %q, %v, %s and %d, want no URLs, "+
@@ -1179,10 +1157,7 @@ func TestAlertSettingsAsSet(t *testing.T) {
!reflect.DeepEqual(cfg.AlertWebhookHeaders, headers) ||
cfg.AlertSlackWebhookURL.String() != slack || cfg.AlertNtfyURL.String() != ntfy ||
cfg.AlertNtfyToken != token ||
!slices.Equal(
cfg.AlertEvents,
[]alerts.Event{"ban", "file_error"},
) ||
!slices.Equal(cfg.AlertEvents, []string{"ban", "file_error"}) ||
cfg.AlertCooldown != time.Hour || cfg.AlertMaxPerHour != 10 {
t.Errorf(
"alert settings %v, %v, %v, %v, %q, %v, %s and %d",
@@ -1576,35 +1551,7 @@ func TestAnomalySettingsDefaults(t *testing.T) {
t.Parallel()
cfg := wafFromEnvironment(t, environment{})
const gibibyte = int64(1) << 30
want := map[string]anomaly.Thresholds{
anomalyClient: {},
anomalyNet: {
RequestsPerMinute: 1200, RequestsPerHour: 15000,
BytesPerMinute: 5 * gibibyte, BytesPerHour: 50 * gibibyte,
},
anomalyASN: {
RequestsPerMinute: 6000, RequestsPerHour: 75000,
BytesPerMinute: 25 * gibibyte, BytesPerHour: 250 * gibibyte,
},
anomalyTotal: {}, anomalyWatchedNet: {},
}
if got := thresholdsByScope(cfg); !maps.Equal(got, want) {
t.Errorf("thresholds by scope\n%+v\nwant\n%+v", got, want)
}
// With no client looked up, the thresholds per AS number are off.
want[anomalyASN] = anomaly.Thresholds{}
got := thresholdsByScope(
wafFromEnvironment(t, environment{lookupSource: off}))
if !maps.Equal(got, want) {
t.Errorf("with %s off, thresholds by scope\n%+v\nwant\n%+v",
lookupSource, got, want)
}
wantAllOff(t, cfg)
if cfg.AnomalyNetV4Prefix != 24 || cfg.AnomalyNetV6Prefix != 48 ||
len(cfg.AnomalyWatchedNets) != 0 {
+65 -43
View File
@@ -12,10 +12,11 @@
package coreruleset
import (
"embed"
_ "embed"
"errors"
"fmt"
"io"
"maps"
"net"
"net/http"
"net/netip"
@@ -34,7 +35,7 @@ import (
// directives are the Core Rule Set as sww runs it, with the
// paranoia level for %d, bodyDirectives for the first %s while
// SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is set, the exclusions file for
// the second, and the fixes of every app in appFixes for the third.
// the second, and the directives of every app in apps for the third.
// Each rule sww adds has an id from 900000 to 900999, the ids the
// Core Rule Set keeps for the rules that set it up, which
// SWW_WAF_CORE_RULE_SET_DISABLED_RULES and the exclusions file refuse, so that no
@@ -53,12 +54,13 @@ Include @crs-setup.conf.example
SecAction "id:900000,phase:1,pass,nolog,\
setvar:tx.blocking_paranoia_level=%d"
# PUT, PATCH and DELETE are allowed besides GET, HEAD, POST and OPTIONS.
# The first change: PUT, PATCH and DELETE are allowed besides GET, HEAD,
# POST and OPTIONS.
SecAction "id:900200,phase:1,pass,nolog,\
setvar:'tx.allowed_methods=GET HEAD POST OPTIONS PUT PATCH DELETE'"
# Expect and Content-Encoding are taken off the Core Rule Set's list of the
# headers it refuses. Content-Encoding goes back on it for a
# The second: Expect and Content-Encoding are taken off the Core Rule Set's
# list of the headers it refuses. Content-Encoding goes back on it for a
# body the Core Rule Set reads (900260 in bodyDirectives).
SecAction "id:900250,phase:1,pass,nolog,\
setvar:'tx.restricted_headers_basic=/proxy/ /lock-token/ /content-range/ \
@@ -84,17 +86,18 @@ SecRule ARGUMENTS_LIMIT_REACHED "@eq 1" "id:900300,phase:2,pass,\
# The apps' fixes come before the rules for the same reason.
%s
# Only the rules for requests are loaded, and no response is inspected.
# The fifth: only the rules for requests are loaded, and no response is
# inspected.
Include @owasp_crs/REQUEST-*.conf
# redirect_uri is not checked for a URL naming an IP address or
# The third: redirect_uri is not checked for a URL naming an IP address or
# localhost. Coraza matches a parameter name here without regard to case.
# ARGS holds the fields of a form data or multipart body Coraza reads as
# well as the query parameters, so a field of this name is left out too.
SecRuleUpdateTargetById 931100 "!ARGS:redirect_uri"
SecRuleUpdateTargetById 934110 "!ARGS:redirect_uri"
# Referer is not checked for a Unix command without arguments
# The fourth: Referer is not checked for a Unix command without arguments
# or for Java starting a process.
SecRuleUpdateTargetById 932340 "!REQUEST_HEADERS:Referer"
SecRuleUpdateTargetById 944110 "!REQUEST_HEADERS:Referer"
@@ -119,9 +122,9 @@ SecRule REQUEST_HEADERS:Content-Type \
"@rx ^(?:application|text)/(?:[a-z0-9.-]+[+])?json" \
"id:900420,phase:1,pass,nolog,t:none,t:lowercase,ctl:requestBodyProcessor=JSON"
# Content-Encoding goes back on the list of the headers the Core Rule Set
# refuses (900250 in directives) for a body of a kind it reads, since a
# compressed body cannot be inspected.
# The rest of the second change: Content-Encoding is refused again on a
# body of a kind the Core Rule Set reads, since a compressed body cannot be
# inspected.
SecRule REQBODY_PROCESSOR "@rx ^(?:URLENCODED|MULTIPART|JSON|XML)$" \
"id:900260,phase:1,pass,nolog,\
setvar:'tx.restricted_headers_basic=%%{tx.restricted_headers_basic} \
@@ -142,19 +145,15 @@ SecRule MULTIPART_STRICT_ERROR "!@eq 0" "id:900450,phase:2,pass,\
severity:'CRITICAL',setvar:'tx.inbound_anomaly_score_pl1=+5'"
`
// appFixes holds the fixes of each app whose ordinary requests the Core
// Rule Set would refuse, in a file named for the app by the name
// SWW_WAF_APPS gives it, such as apps/gitea.conf. The fixes are Coraza
// exclusions, loaded before the Core Rule Set's rules. Their first rule
// skips to the marker they end with, unless Inspect sets the transaction
// variable named for the app, such as app_gitea. The skip reaches only
// phase 1, so a rule of a later phase must check that variable itself.
//
//go:embed apps/*.conf
var appFixes embed.FS
// app is what an app's file in appFixes cannot say about it.
// app is an app whose ordinary requests the Core Rule Set would refuse,
// with the fixes that let them pass.
type app struct {
// directives are Coraza exclusions, loaded before the Core Rule Set's
// rules. Their first rule skips to the marker they end with, unless
// Inspect sets the transaction variable named for the app, such as
// app_gitea. The skip reaches only phase 1, so a rule of a later phase
// must check that variable itself.
directives string
// cookies are the cookies the Core Rule Set reads a request without.
cookies []string
// forkOf names the app this one is a fork of, "" for none. A hostname
@@ -162,34 +161,60 @@ type app struct {
forkOf string
}
// apps are the apps that need more than their file in appFixes, by the
// name SWW_WAF_APPS gives each. flash is Forgejo's name for gitea_flash.
//go:embed gitea.conf
var giteaDirectives string
//go:embed wordpress.conf
var wordpressDirectives string
//go:embed drupal.conf
var drupalDirectives string
//go:embed mediawiki.conf
var mediawikiDirectives string
//go:embed mastodon.conf
var mastodonDirectives string
//go:embed vaultwarden.conf
var vaultwardenDirectives string
//go:embed nextcloud.conf
var nextcloudDirectives string
//go:embed forgejo.conf
var forgejoDirectives string
//go:embed ghost.conf
var ghostDirectives string
// apps are the apps whose fixes sww has, by the name SWW_WAF_APPS gives each.
// flash is Forgejo's name for gitea_flash.
//
//nolint:gochecknoglobals // a constant cannot be a map
var apps = map[string]app{
"drupal": {directives: drupalDirectives},
"forgejo": {
cookies: []string{"flash"},
forkOf: "gitea",
directives: forgejoDirectives,
cookies: []string{"flash"},
forkOf: "gitea",
},
"ghost": {directives: ghostDirectives},
"gitea": {
cookies: []string{"gitea_flash", "redirect_to"},
directives: giteaDirectives,
cookies: []string{"gitea_flash", "redirect_to"},
},
"mastodon": {directives: mastodonDirectives},
"mediawiki": {directives: mediawikiDirectives},
"nextcloud": {directives: nextcloudDirectives},
"vaultwarden": {directives: vaultwardenDirectives},
"wordpress": {directives: wordpressDirectives},
}
// AppNames returns the names of the apps whose fixes sww has, in
// alphabetical order.
func AppNames() []string {
// An embedded directory always reads.
files, _ := appFixes.ReadDir("apps")
names := make([]string, len(files))
for i, file := range files {
names[i] = strings.TrimSuffix(file.Name(), ".conf")
}
slices.Sort(names)
return names
return slices.Sorted(maps.Keys(apps))
}
// Params are what New needs.
@@ -348,11 +373,8 @@ func New(params Params) (*CoreRuleSet, error) {
}
var fixes strings.Builder
for _, name := range AppNames() {
// An embedded file always reads.
file, _ := appFixes.ReadFile("apps/" + name + ".conf")
fixes.WriteString(string(file) + "\n")
fixes.WriteString(apps[name].directives + "\n")
}
text := fmt.Sprintf(directives, params.ParanoiaLevel, body, exclusions,
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
+3 -3
View File
@@ -17,7 +17,7 @@ const (
)
// giteaSearches are searches of the pages gitea and Forgejo share whose q
// skips the lists of system files and command names, in gitea's fixes.
// skips the lists of system files and command names, in the eighth change.
// gitea serves its admin pages under /-/admin/ and Forgejo under /admin/.
//
//nolint:gochecknoglobals // a constant cannot be a list
@@ -222,8 +222,8 @@ func TestGiteaSearchExclusionsAreNarrow(t *testing.T) {
}
// Under /admin/, where Forgejo serves its admin pages, only the hard
// values are left out: Drupal's fixes let them through in q on every path
// there.
// values are left out: the tenth change lets them through in q on every
// path there, for Drupal.
for _, target := range []string{
"/admin/packages/delete?q=",
"/admin/packages?other=",
File renamed without changes.
File renamed without changes.
@@ -37,10 +37,9 @@
SecRule &TX:app_nextcloud "@eq 0" \
"id:900090,phase:1,pass,nolog,skipAfter:END_NEXTCLOUD"
# The sync client and the Files app use these WebDAV methods besides PUT,
# PATCH and DELETE, which sww allows for every app, on /remote.php/dav/ and
# /public.php/dav/ and on the older /remote.php/webdav/ and /public.php/webdav/.
# 911100 refused each:
# The sync client and the Files app use these WebDAV methods besides those
# the first change allows, on /remote.php/dav/ and /public.php/dav/ and on the
# older /remote.php/webdav/ and /public.php/webdav/. 911100 refused each:
# PROPFIND, PROPPATCH: listing a folder, marking a file a favourite
# MKCOL: making a folder, and starting a chunked upload
# COPY, MOVE: copying, moving and renaming, and ending a chunked upload
File renamed without changes.
-90
View File
@@ -1,90 +0,0 @@
// Package dirwatch watches a directory, and calls a function once the
// directory has gone a short time without a change after one. The rule
// files and the lookup database are read again this way.
package dirwatch
import (
"context"
"time"
"github.com/fsnotify/fsnotify"
)
// quietTime is how long the directory must go without a change before the
// function is called, so that a file still being written, such as one
// saved in place, appended to or copied in with scp, is read only once
// whole.
const quietTime = 2 * time.Second
// Watcher watches one directory. The directory is watched, rather than a
// file in it, so that a file replaced by a rename is seen.
type Watcher struct {
watcher *fsnotify.Watcher
}
// New starts watching dir, or returns why it cannot.
func New(dir string) (*Watcher, error) {
watcher, err := fsnotify.NewWatcher()
if err != nil {
return nil, err
}
err = watcher.Add(dir)
if err != nil {
_ = watcher.Close()
return nil, err
}
return &Watcher{watcher: watcher}, nil
}
// Close stops watching the directory.
func (w *Watcher) Close() error {
return w.watcher.Close()
}
// CallWhenQuiet calls changed each time the directory has gone quietTime
// without a change, after a change to a path for which counts is true, until
// ctx is done. It also calls failed with each error the watch reports. The
// wait starts at once, as if for a change, so that a change made before the
// watch began is taken in too.
func (w *Watcher) CallWhenQuiet(
ctx context.Context,
counts func(path string) bool,
changed func(),
failed func(error),
) {
callWhenQuiet(
ctx, w.watcher.Events, w.watcher.Errors, counts, changed, failed,
)
}
// callWhenQuiet is CallWhenQuiet, for the changes from events and the
// errors from errs.
func callWhenQuiet(
ctx context.Context,
events <-chan fsnotify.Event,
errs <-chan error,
counts func(path string) bool,
changed func(),
failed func(error),
) {
quiet := time.NewTimer(quietTime)
defer quiet.Stop()
for {
select {
case <-ctx.Done():
return
case event := <-events:
if counts(event.Name) {
quiet.Reset(quietTime)
}
case <-quiet.C:
changed()
case err := <-errs:
failed(err)
}
}
}
-167
View File
@@ -1,167 +0,0 @@
package dirwatch
import (
"context"
"errors"
"path/filepath"
"sync/atomic"
"testing"
"testing/synctest"
"time"
"github.com/fsnotify/fsnotify"
)
// errWatch is an error the watch reports.
var errWatch = errors.New("too many changes")
func TestDirectoryThatCannotBeWatchedIsAnError(t *testing.T) {
t.Parallel()
_, err := New(filepath.Join(t.TempDir(), "missing"))
if err == nil {
t.Error("a missing directory is watched, want an error")
}
}
// The tests below run callWhenQuiet in a synctest bubble, where time is a
// clock of the test's own: time.Sleep moves it on at once, and
// synctest.Wait returns once callWhenQuiet waits again, so that every call
// due by then is made. The test sends the changes itself, as the watch of
// a directory cannot run in a bubble.
func TestSecondChangeStartsTheWaitAgain(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
loop := start(t, everyPath)
loop.changes <- fsnotify.Event{Name: "50-app.rules", Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
loop.wantCalls(t, 0)
// The second part starts the wait again.
loop.changes <- fsnotify.Event{Name: "50-app.rules", Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
loop.wantCalls(t, 0)
time.Sleep(time.Nanosecond)
synctest.Wait()
loop.wantCalls(t, 1)
})
}
func TestCalledOnceQuietAfterTheStart(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
// An edit saved before the watch began sends no change, and is
// taken in by this call.
loop := start(t, everyPath)
time.Sleep(quietTime)
synctest.Wait()
loop.wantCalls(t, 1)
// Long after, it has been called only once.
time.Sleep(time.Hour)
synctest.Wait()
loop.wantCalls(t, 1)
})
}
func TestChangeThatDoesNotCountIsNoChange(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
loop := start(t, func(path string) bool {
return path == "ipinfo_lite.mmdb"
})
// The wait that starts with the watch ends with a call.
time.Sleep(quietTime)
synctest.Wait()
loop.changes <- fsnotify.Event{Name: "other.mmdb", Op: fsnotify.Create}
time.Sleep(quietTime)
synctest.Wait()
loop.wantCalls(t, 1)
loop.changes <- fsnotify.Event{Name: "ipinfo_lite.mmdb", Op: fsnotify.Write}
time.Sleep(quietTime)
synctest.Wait()
loop.wantCalls(t, 2)
})
}
func TestErrorsOfTheWatchHandedOn(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
loop := start(t, everyPath)
loop.errs <- errWatch
synctest.Wait()
if !errors.Is(loop.failure, errWatch) {
t.Errorf("failed with %v, want %v", loop.failure, errWatch)
}
})
}
// loop is a callWhenQuiet that a test runs.
type loop struct {
// changes and errs send it changes and errors.
changes chan fsnotify.Event
errs chan error
// calls counts its calls of changed, and failure is the last error it
// handed to failed.
calls atomic.Int32
failure error
}
// start runs callWhenQuiet, with counts, until the test ends.
func start(t *testing.T, counts func(path string) bool) *loop {
t.Helper()
l := &loop{changes: make(chan fsnotify.Event), errs: make(chan error)}
ctx, stop := context.WithCancel(t.Context())
stopped := make(chan struct{})
go func() {
callWhenQuiet(ctx, l.changes, l.errs, counts,
func() { l.calls.Add(1) },
func(err error) { l.failure = err },
)
close(stopped)
}()
t.Cleanup(func() {
stop()
<-stopped
})
return l
}
// everyPath counts a change to any path.
func everyPath(string) bool {
return true
}
// wantCalls checks how many times changed has been called.
func (l *loop) wantCalls(t *testing.T, want int32) {
t.Helper()
got := l.calls.Load()
if got != want {
t.Errorf("changed called %d times, want %d", got, want)
}
}
+49 -18
View File
@@ -13,13 +13,18 @@ import (
"sync"
"time"
"github.com/fsnotify/fsnotify"
"github.com/oschwald/maxminddb-golang/v2"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/dirwatch"
"sneak.berlin/go/smallwebwaf/internal/geojs"
)
// quietTime is how long the lookup database must go without a change
// before it is read again, so that a file still being copied in is read
// only once whole.
const quietTime = 2 * time.Second
// FileParams are what OpenFile needs.
type FileParams struct {
// Path is the lookup database, the IPinfo Lite file in its .mmdb form
@@ -117,13 +122,20 @@ func (f *File) ReadFailures() int {
}
// Watch watches the directory of the lookup database until ctx is done,
// and reads the file again once it has gone without a change for a short
// time, after it is replaced, written or removed, and after Watch starts
// watching, as dirwatch describes. A change to another file in its
// directory does not count. If the directory cannot be watched, that is
// logged, and the database read at start stays in use.
// and reads the file again once it has gone without a change for
// quietTime, after it is replaced, written or removed, and after Watch
// starts watching. If the directory cannot be watched, that is logged, and
// the database read at start stays in use.
func (f *File) Watch(ctx context.Context) {
watcher, err := dirwatch.New(filepath.Dir(f.params.Path))
watcher, err := fsnotify.NewWatcher()
if err == nil {
defer func() {
_ = watcher.Close()
}()
err = watcher.Add(filepath.Dir(f.params.Path))
}
if err != nil {
f.params.ProcessLog.Error(
"cannot watch the lookup database for replacements",
@@ -134,22 +146,41 @@ func (f *File) Watch(ctx context.Context) {
return
}
defer func() {
_ = watcher.Close()
}()
f.params.ProcessLog.Info("watching the lookup database for replacements",
"file", f.params.Path)
watcher.CallWhenQuiet(ctx, f.isDatabase, f.readAgain, func(err error) {
f.params.ProcessLog.Warn("watching the lookup database failed",
"error", err.Error())
})
f.readAfterChanges(ctx, watcher.Events, watcher.Errors)
}
// isDatabase reports whether path names the lookup database.
func (f *File) isDatabase(path string) bool {
return filepath.Clean(path) == filepath.Clean(f.params.Path)
// readAfterChanges reads the lookup database again once quietTime has
// passed without a change to it from events, until ctx is done, and logs
// the errors from errs. A change to another file in its directory does not
// count. The wait starts at once, as if for a change, so that a file
// replaced after OpenFile read it, and before its directory was watched,
// is read too.
func (f *File) readAfterChanges(
ctx context.Context, events <-chan fsnotify.Event, errs <-chan error,
) {
path := filepath.Clean(f.params.Path)
quiet := time.NewTimer(quietTime)
defer quiet.Stop()
for {
select {
case <-ctx.Done():
return
case event := <-events:
if filepath.Clean(event.Name) == path {
quiet.Reset(quietTime)
}
case <-quiet.C:
f.readAgain()
case err := <-errs:
f.params.ProcessLog.Warn("watching the lookup database failed",
"error", err.Error())
}
}
}
// readAgain reads the lookup database again, in place of the one in use,
+141 -35
View File
@@ -1,6 +1,7 @@
package lookup
import (
"context"
"log/slog"
"net/netip"
"net/url"
@@ -11,6 +12,7 @@ import (
"testing/synctest"
"time"
"github.com/fsnotify/fsnotify"
"github.com/maxmind/mmdbwriter/mmdbtype"
"sneak.berlin/go/smallwebwaf/internal/alerts"
@@ -131,48 +133,66 @@ func TestFileThatCannotBeReadIsAnError(t *testing.T) {
}
}
func TestChangeOfAnotherFileInTheDirectoryIsNoReplacement(t *testing.T) {
t.Parallel()
// The tests below run readAfterChanges in a synctest bubble, where time is
// a clock of the test's own: time.Sleep moves it on at once, and
// synctest.Wait returns once readAfterChanges waits again, so that every
// reading due by then is done. The test sends the changes itself, as the
// watch of a directory cannot run in a bubble.
dir := t.TempDir()
path := filepath.Join(dir, "ipinfo_lite.mmdb")
writeDatabase(t, path, "DE")
f := openFile(t, path, newQueue())
for changed, want := range map[string]bool{
path: true,
dir + "/./ipinfo_lite.mmdb": true,
filepath.Join(dir, "other.mmdb"): false,
} {
if f.isDatabase(changed) != want {
t.Errorf(
"a change to %s counts: %t, want %t",
changed,
!want,
want,
)
}
}
}
// The tests below run in a synctest bubble, where time is a clock of the
// test's own, so that the time a file is read can be checked. The wait for
// the directory to go quiet before a reading is tested in dirwatch.
func TestReplacementIsReadAgain(t *testing.T) {
func TestReplacementCopiedOverTheFileInTwoPartsIsReadOnlyWhole(
t *testing.T,
) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb")
dir := t.TempDir()
path := filepath.Join(dir, "ipinfo_lite.mmdb")
writeDatabase(t, path, "DE")
queue := newQueue()
f := openFile(t, path, queue)
changes := watch(t, f)
writeDatabase(t, path, "KP")
time.Sleep(time.Hour)
f.readAgain()
other := filepath.Join(dir, "replacement.mmdb")
writeDatabase(t, other, "KP")
//nolint:gosec // a file the test wrote
replacement, err := os.ReadFile(other)
if err != nil {
t.Fatalf("read %s: %v", other, err)
}
// The file in use is overwritten in place, and keeps giving what
// it gave. Its first part alone is not a .mmdb file.
file, err := os.Create(path) //nolint:gosec // a file the test wrote
if err != nil {
t.Fatalf("create %s: %v", path, err)
}
defer func() {
_ = file.Close()
}()
half := len(replacement) / 2
write(t, file, replacement[:half])
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantCountry(t, f, "DE")
// The second part starts the wait again.
write(t, file, replacement[half:])
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantCountry(t, f, "DE")
time.Sleep(time.Nanosecond)
synctest.Wait()
wantCountry(t, f, "KP")
if !f.LastRead().Equal(time.Now()) || f.ReadFailures() != 0 {
@@ -199,10 +219,15 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(
queue := newQueue()
f := openFile(t, path, queue)
read := f.LastRead()
changes := watch(t, f)
writeFile(t, path, "not a lookup database\n")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
// Long after, the replacement has been read once.
time.Sleep(time.Hour)
f.readAgain()
synctest.Wait()
wantCountry(t, f, "DE")
if !f.LastRead().Equal(read) || f.ReadFailures() != 1 {
@@ -215,7 +240,7 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(
}
wantAlerts(t, queue, alerts.Alert{
Time: read.Add(time.Hour),
Time: read.Add(quietTime),
Event: alerts.EventFileError,
Reason: "the lookup database cannot be read, and the one read before stays in use",
Detail: map[string]any{
@@ -227,6 +252,55 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(
})
}
func TestChangeOfAnotherFileInTheDirectoryIsNoReplacement(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "ipinfo_lite.mmdb")
writeDatabase(t, path, "DE")
f := openFile(t, path, newQueue())
changes := watch(t, f)
// The wait that starts with the watch ends with a reading.
time.Sleep(quietTime)
synctest.Wait()
writeDatabase(t, path, "KP")
changes <- fsnotify.Event{Name: filepath.Join(dir, "other.mmdb"), Op: fsnotify.Create}
time.Sleep(quietTime)
synctest.Wait()
wantCountry(t, f, "DE")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime)
synctest.Wait()
wantCountry(t, f, "KP")
})
}
func TestReplacementSavedBeforeTheWatchStartsIsRead(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb")
writeDatabase(t, path, "DE")
f := openFile(t, path, newQueue())
// Saved after OpenFile read the file, and before its directory was
// watched, so that no change is seen for it.
writeDatabase(t, path, "KP")
watch(t, f)
time.Sleep(quietTime)
synctest.Wait()
wantCountry(t, f, "KP")
})
}
// newQueue returns a queue of alerts for a webhook that is never sent
// them, so that they wait in it for the test to look at.
func newQueue() *alerts.Queue {
@@ -270,6 +344,28 @@ func openFile(t *testing.T, path string, queue *alerts.Queue) *File {
return f
}
// watch runs f's readAfterChanges until the test ends, and returns the
// channel that sends it changes.
func watch(t *testing.T, f *File) chan<- fsnotify.Event {
t.Helper()
changes := make(chan fsnotify.Event)
ctx, stop := context.WithCancel(t.Context())
stopped := make(chan struct{})
go func() {
f.readAfterChanges(ctx, changes, nil)
close(stopped)
}()
t.Cleanup(func() {
stop()
<-stopped
})
return changes
}
// wantCountry checks the country f gives testClient.
func wantCountry(t *testing.T, f *File, want string) {
t.Helper()
@@ -305,3 +401,13 @@ func writeFile(t *testing.T, path, content string) {
t.Fatalf("write %s: %v", path, err)
}
}
// write writes data to the end of file.
func write(t *testing.T, file *os.File, data []byte) {
t.Helper()
_, err := file.Write(data)
if err != nil {
t.Fatalf("write: %v", err)
}
}
+273 -400
View File
@@ -15,34 +15,12 @@ import (
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/remotelog"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
)
// Params are what New builds the metrics from. Each part is nil while it is
// off, and a part that is nil adds no metric of its own.
type Params struct {
// BusiestCount is how many countries and how many AS numbers get series
// of their own (SWW_WAF_METRICS_BUSIEST_COUNTRIES_AND_ASNS).
BusiestCount int
// InstanceName is the label instance every metric carries
// (SWW_WAF_INSTANCE_NAME).
InstanceName string
// Now returns the time at which Bans counts the bans active.
Now func() time.Time
Bans banLedger
Clients sized
Rules sized
Lists fetchedLists
DNSBL dnsblZones
AbuseIPDB abuseIPDBChecker
Alerts alertQueue
LookupFile lookupDatabase
RemoteLog remoteLogSender
}
// Metrics are sww's metrics. They are safe for concurrent use.
type Metrics struct {
// registry gives every metric registered with it the label instance.
@@ -58,10 +36,12 @@ type Metrics struct {
rateLimitHits *prometheus.CounterVec
offences *prometheus.CounterVec
coreRuleSetMatches *prometheus.CounterVec
ruleMatches *prometheus.CounterVec
reputationHits *prometheus.CounterVec
countries *busiest
asns *busiest
// ruleMatches are made by AddRules, and reputationHits by
// AddReputation.
ruleMatches *prometheus.CounterVec
reputationHits *prometheus.CounterVec
countries *busiest
asns *busiest
// GeoJSRequests are the requests to GeoJS, and GeoJSFailures those
// that failed. RequestsWithoutGeoJSAnswer are the requests that needed
@@ -79,18 +59,20 @@ type Metrics struct {
stateFileEditsSetAside *prometheus.CounterVec
}
// New returns the metrics, with the Go runtime's and the process's own, and
// those read from each part in params as the metrics are asked for.
// New returns the metrics, with the Go runtime's and the process's own.
// busiestCount is how many countries and how many AS numbers get series of
// their own (SWW_WAF_METRICS_BUSIEST_COUNTRIES_AND_ASNS). Every metric
// carries instanceName (SWW_WAF_INSTANCE_NAME) as its label instance.
//
//nolint:funlen // a few lines for each metric, a list that grows with them
func New(params Params) *Metrics {
func New(busiestCount int, instanceName string) *Metrics {
byStatus := []string{"status_class", "action"}
byFile := []string{"file"}
registry := prometheus.NewRegistry()
m := &Metrics{
registry: prometheus.WrapRegistererWith(
prometheus.Labels{"instance": params.InstanceName}, registry),
prometheus.Labels{"instance": instanceName}, registry),
handler: promhttp.HandlerFor(registry, promhttp.HandlerOpts{}),
inFlight: prometheus.NewGauge(prometheus.GaugeOpts{
Name: "sww_requests_in_flight",
@@ -131,19 +113,8 @@ func New(params Params) *Metrics {
"and the rule's id.",
[]string{"mode", "rule_id"},
),
ruleMatches: counterVec(
"sww_rule_matches_total",
"Requests that matched a rule of the rule files, by its id and action.",
[]string{"rule_id", "action"},
),
reputationHits: counterVec(
"sww_reputation_hits_total",
"Requests whose client a blocklist, a DNSBL zone or AbuseIPDB lists, "+
"by the list's URL, the zone, or abuseipdb.",
[]string{sourceLabel},
),
countries: newCountries(params.BusiestCount),
asns: newASNs(params.BusiestCount),
countries: newCountries(busiestCount),
asns: newASNs(busiestCount),
GeoJSRequests: prometheus.NewCounter(prometheus.CounterOpts{
Name: "sww_geojs_requests_total",
Help: "Requests to GeoJS.",
@@ -195,7 +166,6 @@ func New(params Params) *Metrics {
m.inFlight, m.requests, m.requestBytes, m.responseBytes,
m.requestDuration, m.upstreamDuration,
m.rateLimitHits, m.offences, m.coreRuleSetMatches,
m.ruleMatches, m.reputationHits,
m.countries, m.asns,
m.GeoJSRequests, m.GeoJSFailures, m.RequestsWithoutGeoJSAnswer,
m.stateFileWrites, m.stateFileWriteFailures,
@@ -203,45 +173,261 @@ func New(params Params) *Metrics {
m.stateFileEditsTakenIn, m.stateFileEditsSetAside,
)
if params.Bans != nil {
m.addBans(params.Bans, params.Now)
}
if params.Clients != nil {
m.addClients(params.Clients)
}
if params.Rules != nil {
m.addRules(params.Rules)
}
if params.Lists != nil {
m.addLists(params.Lists)
}
if params.DNSBL != nil {
m.addDNSBL(params.DNSBL)
}
if params.AbuseIPDB != nil {
m.addAbuseIPDB(params.AbuseIPDB)
}
if params.Alerts != nil {
m.addAlerts(params.Alerts)
}
if params.LookupFile != nil {
m.addLookupFile(params.LookupFile)
}
if params.RemoteLog != nil {
m.addRemoteLog(params.RemoteLog)
}
return m
}
// AddBansAndClients adds the metrics read from the ledger and the table
// of clients as the metrics are asked for: the bans made since the start,
// by cause, the bans active and permanent at now, and the clients in the
// table.
func (m *Metrics) AddBansAndClients(
ledger *bans.Ledger, limiter *ratelimit.Limiter, now func() time.Time,
) {
for _, cause := range []string{
bans.CauseLimit, bans.CauseAttack, bans.CauseAdmin, bans.CauseAbuseIPDB,
} {
m.registry.MustRegister(
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_bans_made_total",
Help: "Bans made, by cause.",
ConstLabels: prometheus.Labels{"cause": cause},
}, func() float64 {
return float64(ledger.Made(cause))
}),
)
}
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_active_bans",
Help: "Bans active now, the permanent ones included.",
}, func() float64 {
active, _ := ledger.Count(now())
return float64(active)
}),
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_permanent_bans",
Help: "Permanent bans not lifted.",
}, func() float64 {
_, permanent := ledger.Count(now())
return float64(permanent)
}),
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_tracked_clients",
Help: "Clients in the table of clients.",
}, func() float64 {
return float64(limiter.Len())
}),
)
}
// AddRules adds the metrics of the rule files: the requests that matched
// each rule, which RuleMatched counts, and the rules loaded from
// ruleFiles, read as the metrics are asked for. It is called once, before
// RuleMatched.
func (m *Metrics) AddRules(ruleFiles *rules.Files) {
m.ruleMatches = counterVec(
"sww_rule_matches_total",
"Requests that matched a rule of the rule files, by its id and action.",
[]string{"rule_id", "action"},
)
m.registry.MustRegister(m.ruleMatches,
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_rules_loaded",
Help: "Rules loaded from the rule files.",
}, func() float64 {
return float64(ruleFiles.Len())
}))
}
// AddRemoteLog adds the metrics of sending the log lines to
// SWW_WAF_LOG_REMOTE_URL, read from remote as the metrics are asked for: the
// lines sent, those dropped, and those waiting in the buffer.
func (m *Metrics) AddRemoteLog(remote *remotelog.Sender) {
m.registry.MustRegister(
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_remote_log_lines_sent_total",
Help: "Log lines sent to SWW_WAF_LOG_REMOTE_URL.",
}, func() float64 {
return float64(remote.Sent())
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_remote_log_lines_dropped_total",
Help: "Log lines dropped: the oldest in a full buffer, and those " +
"whose sending failed.",
}, func() float64 {
return float64(remote.Dropped())
}),
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_remote_log_lines_waiting",
Help: "Log lines in the buffer, waiting to be sent.",
}, func() float64 {
return float64(remote.Depth())
}),
)
}
// AddLookupFile adds the metrics of the lookup database, read as the
// metrics are asked for: when the file in use was read, which lastRead
// returns, and the replacements of it that could not be read, which
// readFailures returns.
func (m *Metrics) AddLookupFile(
lastRead func() time.Time,
readFailures func() int,
) {
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_lookup_database_last_read_timestamp_seconds",
Help: "When the lookup database in use was read, in seconds since 1970.",
}, func() float64 {
return float64(lastRead().Unix())
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_lookup_database_read_failures_total",
Help: "Replacements of the lookup database that could not be read.",
}, func() float64 {
return float64(readFailures())
}),
)
}
// sourceLabel is the label of the reputation metrics: a list's URL, a
// DNSBL zone, its key masked, or abuseipdb.
const sourceLabel = "source"
// AddReputation adds the metrics of the lists fetched from URLs and of the
// DNSBL zones, by source, each list's URL or each zone, its key masked as
// reputation.MaskZoneKey masks it: the requests whose client a blocklist, a
// zone's verdict or AbuseIPDB's score lists, which ReputationHit counts,
// and, read from lists and dnsbl as the metrics are asked for, for a list,
// the fetches that failed and when the copy in use was fetched, and for a
// zone, the queries made and those that failed. It is called once, before
// ReputationHit.
func (m *Metrics) AddReputation(
lists *reputation.Lists,
dnsbl *reputation.DNSBL,
) {
m.reputationHits = counterVec(
"sww_reputation_hits_total",
"Requests whose client a blocklist, a DNSBL zone or AbuseIPDB lists, "+
"by the list's URL, the zone, or abuseipdb.",
[]string{sourceLabel},
)
m.registry.MustRegister(m.reputationHits)
for _, zone := range dnsbl.Zones() {
source := prometheus.Labels{
sourceLabel: reputation.MaskZoneKey(zone),
}
m.addReputationQueries(
source,
func() int { return dnsbl.Queries(zone) },
)
m.addReputationFailures(
source,
func() int { return dnsbl.Failures(zone) },
)
}
for _, listURL := range lists.URLs() {
source := prometheus.Labels{sourceLabel: listURL}
m.addReputationFailures(
source,
func() int { return lists.Failures(listURL) },
)
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_reputation_last_fetch_timestamp_seconds",
Help: "When the copy of the list in use was fetched, in seconds since " +
"1970, or 0 while there is none.",
ConstLabels: source,
}, func() float64 {
fetched := lists.Fetched(listURL)
if fetched.IsZero() {
return 0
}
return float64(fetched.Unix())
}),
)
}
}
// AddAbuseIPDB adds the metrics of AbuseIPDB, with the source abuseipdb,
// read from abuseIPDB as the metrics are asked for: the checks made, those
// that failed, and how many checks the day's budget has left. It is
// called once, after AddReputation, while SWW_WAF_ABUSEIPDB_KEY is set.
func (m *Metrics) AddAbuseIPDB(abuseIPDB *reputation.AbuseIPDB) {
source := prometheus.Labels{sourceLabel: reputation.AbuseIPDBSource}
m.addReputationQueries(source, abuseIPDB.Checked)
m.addReputationFailures(source, abuseIPDB.Failures)
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_reputation_daily_budget_remaining",
Help: "Checks of the day's SWW_WAF_ABUSEIPDB_DAILY_BUDGET not yet spent.",
ConstLabels: source,
}, func() float64 {
return float64(abuseIPDB.BudgetLeft())
}),
)
}
// ReputationHit counts a request whose client source lists: a blocklist,
// by its URL, a DNSBL zone, its key masked, or AbuseIPDB, abuseipdb.
func (m *Metrics) ReputationHit(source string) {
m.reputationHits.WithLabelValues(source).Inc()
}
// AddAlerts adds the metrics of the alerts sent to each destination set,
// read from queue as the metrics are asked for, by destination: the
// alerts sent, the requests to the destination that failed, the alerts
// held back, which are the same for every destination, and those
// dropped. With no destination set, it adds none.
func (m *Metrics) AddAlerts(queue *alerts.Queue) {
for _, name := range queue.DestinationsSet() {
destination := prometheus.Labels{"destination": name}
m.registry.MustRegister(
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alerts_sent_total",
Help: "Alerts the destination took.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Counts(name).Sent)
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alert_send_failures_total",
Help: "Requests to the destination that failed.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Counts(name).SendFailures)
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alerts_suppressed_total",
Help: "Alerts held back: repeats within SWW_WAF_ALERT_COOLDOWN, and " +
"alerts past SWW_WAF_ALERT_MAX_PER_HOUR, for the hour's summary.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Suppressed())
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alerts_dropped_total",
Help: "Alerts dropped, the oldest first, from a full queue, and " +
"alerts given up as the destination refused them.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Counts(name).Dropped)
}),
)
}
}
// ServeHTTP answers with the metrics in the Prometheus text format.
func (m *Metrics) ServeHTTP(w http.ResponseWriter, r *http.Request) {
m.handler.ServeHTTP(w, r)
@@ -280,7 +466,7 @@ func (m *Metrics) RequestReported(
// LimitHit counts a request that broke a rate limit, a byte limit or the
// refusal limit, by the window and the kind of hit.
func (m *Metrics) LimitHit(hit ratelimit.Hit) {
m.rateLimitHits.WithLabelValues(hit.Window, string(hit.Kind)).Inc()
m.rateLimitHits.WithLabelValues(hit.Window, hit.Kind).Inc()
}
// Offences counts the offences of r, a request that has ended, as its
@@ -301,8 +487,8 @@ func (m *Metrics) Offences(r ratelimit.Request) {
// RuleMatched counts a request that matched the rule id, whose action is
// action.
func (m *Metrics) RuleMatched(id string, action rules.Action) {
m.ruleMatches.WithLabelValues(id, string(action)).Inc()
func (m *Metrics) RuleMatched(id, action string) {
m.ruleMatches.WithLabelValues(id, action).Inc()
}
// CoreRuleSetMatched counts a request that matched the Core Rule Set's rule id,
@@ -311,12 +497,6 @@ func (m *Metrics) CoreRuleSetMatched(mode string, id int) {
m.coreRuleSetMatches.WithLabelValues(mode, strconv.Itoa(id)).Inc()
}
// ReputationHit counts a request whose client source lists: a blocklist,
// by its URL, a DNSBL zone, its key masked, or AbuseIPDB, abuseipdb.
func (m *Metrics) ReputationHit(source string) {
m.reputationHits.WithLabelValues(source).Inc()
}
// StateFileWritten counts a write of the state or cache file name, of size
// bytes, that ended with err.
func (m *Metrics) StateFileWritten(name string, size int, err error) {
@@ -348,323 +528,16 @@ func (m *Metrics) StateFileEditSetAside(name string) {
m.stateFileEditsSetAside.WithLabelValues(name).Inc()
}
// banLedger is what the metrics read from the ledger of bans.
type banLedger interface {
// Made returns the bans made since the start with cause.
Made(cause bans.Cause) int
// Count returns the bans active at now, the permanent ones included,
// and the permanent ones.
Count(now time.Time) (active, permanent int)
}
// addBans adds the metrics read from ledger as the metrics are asked for:
// the bans made since the start, by cause, and the bans active and
// permanent at now.
func (m *Metrics) addBans(ledger banLedger, now func() time.Time) {
for _, cause := range []bans.Cause{
bans.CauseLimit, bans.CauseAttack, bans.CauseAdmin, bans.CauseAbuseIPDB,
} {
m.registry.MustRegister(
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_bans_made_total",
Help: "Bans made, by cause.",
ConstLabels: prometheus.Labels{"cause": string(cause)},
}, func() float64 {
return float64(ledger.Made(cause))
}),
)
}
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_active_bans",
Help: "Bans active now, the permanent ones included.",
}, func() float64 {
active, _ := ledger.Count(now())
return float64(active)
}),
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_permanent_bans",
Help: "Permanent bans not lifted.",
}, func() float64 {
_, permanent := ledger.Count(now())
return float64(permanent)
}),
)
}
// sized is what the metrics read from the table of clients and from the
// rule files.
type sized interface {
// Len returns how many clients the table holds, or how many rules were
// loaded from the rule files.
Len() int
}
// addClients adds the metric of the clients in table, read as the metrics
// are asked for.
func (m *Metrics) addClients(table sized) {
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_tracked_clients",
Help: "Clients in the table of clients.",
}, func() float64 {
return float64(table.Len())
}),
)
}
// addRules adds the metric of the rules loaded from files, read as the
// metrics are asked for.
func (m *Metrics) addRules(files sized) {
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_rules_loaded",
Help: "Rules loaded from the rule files.",
}, func() float64 {
return float64(files.Len())
}))
}
// remoteLogSender is what the metrics read from what sends the log lines to
// SWW_WAF_LOG_REMOTE_URL.
type remoteLogSender interface {
// Sent returns the lines sent.
Sent() int64
// Dropped returns the lines dropped.
Dropped() int64
// Depth returns the lines waiting in the buffer.
Depth() int
}
// addRemoteLog adds the metrics of sending the log lines to
// SWW_WAF_LOG_REMOTE_URL, read from remote as the metrics are asked for: the
// lines sent, those dropped, and those waiting in the buffer.
func (m *Metrics) addRemoteLog(remote remoteLogSender) {
m.registry.MustRegister(
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_remote_log_lines_sent_total",
Help: "Log lines sent to SWW_WAF_LOG_REMOTE_URL.",
}, func() float64 {
return float64(remote.Sent())
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_remote_log_lines_dropped_total",
Help: "Log lines dropped: the oldest in a full buffer, and those " +
"whose sending failed.",
}, func() float64 {
return float64(remote.Dropped())
}),
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_remote_log_lines_waiting",
Help: "Log lines in the buffer, waiting to be sent.",
}, func() float64 {
return float64(remote.Depth())
}),
)
}
// lookupDatabase is what the metrics read from the lookup database.
type lookupDatabase interface {
// LastRead returns when the file in use was read.
LastRead() time.Time
// ReadFailures returns the replacements of the file that could not be
// read.
ReadFailures() int
}
// addLookupFile adds the metrics of the lookup database, read from file as
// the metrics are asked for: when the file in use was read, and the
// replacements of it that could not be read.
func (m *Metrics) addLookupFile(file lookupDatabase) {
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_lookup_database_last_read_timestamp_seconds",
Help: "When the lookup database in use was read, in seconds since 1970.",
}, func() float64 {
return float64(file.LastRead().Unix())
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_lookup_database_read_failures_total",
Help: "Replacements of the lookup database that could not be read.",
}, func() float64 {
return float64(file.ReadFailures())
}),
)
}
// sourceLabel is the label of the reputation metrics: a list's URL, a
// DNSBL zone, its key masked, or abuseipdb.
const sourceLabel = "source"
// dnsblZones is what the metrics read from what asks the DNSBL zones.
type dnsblZones interface {
// Zones returns the zones asked, their keys unmasked.
Zones() []string
// Queries returns the queries made to zone.
Queries(zone string) int
// Failures returns the queries to zone that failed.
Failures(zone string) int
}
// addDNSBL adds the metrics of each zone dnsbl asks, by source, the zone,
// its key masked as reputation.MaskZoneKey masks it, read as the metrics are
// asked for: the queries made and those that failed.
func (m *Metrics) addDNSBL(dnsbl dnsblZones) {
for _, zone := range dnsbl.Zones() {
source := prometheus.Labels{
sourceLabel: reputation.MaskZoneKey(zone),
}
m.addReputationQueries(
source,
func() int { return dnsbl.Queries(zone) },
)
m.addReputationFailures(
source,
func() int { return dnsbl.Failures(zone) },
)
}
}
// fetchedLists is what the metrics read from the lists fetched from URLs.
type fetchedLists interface {
// URLs returns the lists' URLs.
URLs() []string
// Failures returns the fetches of the list at listURL that failed.
Failures(listURL string) int
// Fetched returns when the copy in use of the list at listURL was
// fetched, or the zero time while there is none.
Fetched(listURL string) time.Time
}
// addLists adds the metrics of each list fetched from a URL, by source, the
// list's URL, read from lists as the metrics are asked for: the fetches
// that failed and when the copy in use was fetched.
func (m *Metrics) addLists(lists fetchedLists) {
for _, listURL := range lists.URLs() {
source := prometheus.Labels{sourceLabel: listURL}
m.addReputationFailures(
source,
func() int { return lists.Failures(listURL) },
)
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_reputation_last_fetch_timestamp_seconds",
Help: "When the copy of the list in use was fetched, in seconds since " +
"1970, or 0 while there is none.",
ConstLabels: source,
}, func() float64 {
fetched := lists.Fetched(listURL)
if fetched.IsZero() {
return 0
}
return float64(fetched.Unix())
}),
)
}
}
// abuseIPDBChecker is what the metrics read from what checks clients with
// AbuseIPDB.
type abuseIPDBChecker interface {
// Checked returns the checks made.
Checked() int
// Failures returns the checks that failed.
Failures() int
// BudgetLeft returns how many checks the day's budget has left.
BudgetLeft() int
}
// addAbuseIPDB adds the metrics of AbuseIPDB, with the source abuseipdb,
// read from abuseIPDB as the metrics are asked for: the checks made, those
// that failed, and how many checks the day's budget has left.
func (m *Metrics) addAbuseIPDB(abuseIPDB abuseIPDBChecker) {
source := prometheus.Labels{sourceLabel: reputation.AbuseIPDBSource}
m.addReputationQueries(source, abuseIPDB.Checked)
m.addReputationFailures(source, abuseIPDB.Failures)
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_reputation_daily_budget_remaining",
Help: "Checks of the day's SWW_WAF_ABUSEIPDB_DAILY_BUDGET not yet spent.",
ConstLabels: source,
}, func() float64 {
return float64(abuseIPDB.BudgetLeft())
}),
)
}
// alertQueue is what the metrics read from the queue of alerts.
type alertQueue interface {
// DestinationsSet returns the names of the destinations set.
DestinationsSet() []alerts.DestinationName
// Counts returns the alerts sent to the destination name, the requests
// to it that failed, and the alerts dropped.
Counts(name alerts.DestinationName) alerts.Counts
// Suppressed returns the alerts held back.
Suppressed() int64
}
// addAlerts adds the metrics of the alerts sent to each destination set,
// read from queue as the metrics are asked for, by destination: the
// alerts sent, the requests to the destination that failed, the alerts
// held back, which are the same for every destination, and those
// dropped. With no destination set, it adds none.
func (m *Metrics) addAlerts(queue alertQueue) {
for _, name := range queue.DestinationsSet() {
destination := prometheus.Labels{"destination": string(name)}
m.registry.MustRegister(
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alerts_sent_total",
Help: "Alerts the destination took.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Counts(name).Sent)
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alert_send_failures_total",
Help: "Requests to the destination that failed.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Counts(name).SendFailures)
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alerts_suppressed_total",
Help: "Alerts held back: repeats within SWW_WAF_ALERT_COOLDOWN, and " +
"alerts past SWW_WAF_ALERT_MAX_PER_HOUR, for the hour's summary.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Suppressed())
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alerts_dropped_total",
Help: "Alerts dropped, the oldest first, from a full queue, and " +
"alerts given up as the destination refused them.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Counts(name).Dropped)
}),
)
}
}
// count counts a request that has ended, as RequestEnded and RequestReported
// describe.
func (m *Metrics) count(
line *requestlog.Line, duration, upstreamDuration time.Duration,
) {
class := statusClass(line.Status)
action := string(line.Action)
m.requests.WithLabelValues(class, action).Inc()
m.requestBytes.WithLabelValues(class, action).
m.requests.WithLabelValues(class, line.Action).Inc()
m.requestBytes.WithLabelValues(class, line.Action).
Add(float64(line.RequestBytes))
m.responseBytes.WithLabelValues(class, action).
m.responseBytes.WithLabelValues(class, line.Action).
Add(float64(line.ResponseBytes))
m.requestDuration.Observe(duration.Seconds())
-161
View File
@@ -1,161 +0,0 @@
package metrics_test
import (
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/metrics"
)
const (
listURL = "https://lists.example/block.txt"
zone = "dnsbl.example"
destination = "webhook"
)
// Each stand-in answers with a number of its own, so that each metric
// shows which part it was read from.
type ledgerStandIn struct{}
func (ledgerStandIn) Made(bans.Cause) int { return 1 }
func (ledgerStandIn) Count(time.Time) (int, int) { return 2, 1 }
type clientTableStandIn struct{}
func (clientTableStandIn) Len() int { return 3 }
type ruleFilesStandIn struct{}
func (ruleFilesStandIn) Len() int { return 4 }
type listsStandIn struct{}
func (listsStandIn) URLs() []string { return []string{listURL} }
func (listsStandIn) Failures(string) int { return 5 }
func (listsStandIn) Fetched(string) time.Time { return time.Time{} }
type dnsblStandIn struct{}
func (dnsblStandIn) Zones() []string { return []string{zone} }
func (dnsblStandIn) Queries(string) int { return 6 }
func (dnsblStandIn) Failures(string) int { return 0 }
type abuseIPDBStandIn struct{}
func (abuseIPDBStandIn) Checked() int { return 0 }
func (abuseIPDBStandIn) Failures() int { return 0 }
func (abuseIPDBStandIn) BudgetLeft() int { return 7 }
type alertQueueStandIn struct{}
func (alertQueueStandIn) DestinationsSet() []alerts.DestinationName {
return []alerts.DestinationName{destination}
}
func (alertQueueStandIn) Counts(alerts.DestinationName) alerts.Counts {
return alerts.Counts{Sent: 8}
}
func (alertQueueStandIn) Suppressed() int64 { return 0 }
type lookupDatabaseStandIn struct{}
func (lookupDatabaseStandIn) LastRead() time.Time { return time.Time{} }
func (lookupDatabaseStandIn) ReadFailures() int { return 9 }
type remoteLogStandIn struct{}
func (remoteLogStandIn) Sent() int64 { return 10 }
func (remoteLogStandIn) Dropped() int64 { return 0 }
func (remoteLogStandIn) Depth() int { return 0 }
func TestMetricsAreReadFromEachPart(t *testing.T) {
t.Parallel()
m := metrics.New(metrics.Params{
BusiestCount: 1,
InstanceName: "app",
Now: func() time.Time { return time.Time{} },
Bans: ledgerStandIn{},
Clients: clientTableStandIn{},
Rules: ruleFilesStandIn{},
Lists: listsStandIn{},
DNSBL: dnsblStandIn{},
AbuseIPDB: abuseIPDBStandIn{},
Alerts: alertQueueStandIn{},
LookupFile: lookupDatabaseStandIn{},
RemoteLog: remoteLogStandIn{},
})
scraped := scrape(t, m)
for _, line := range []string{
`sww_bans_made_total{cause="limit",instance="app"} 1`,
`sww_active_bans{instance="app"} 2`,
`sww_tracked_clients{instance="app"} 3`,
`sww_rules_loaded{instance="app"} 4`,
`sww_reputation_failures_total{instance="app",source="` + listURL + `"} 5`,
`sww_reputation_queries_total{instance="app",source="` + zone + `"} 6`,
`sww_reputation_daily_budget_remaining{instance="app",source="abuseipdb"} 7`,
`sww_alerts_sent_total{destination="` + destination + `",instance="app"} 8`,
`sww_lookup_database_read_failures_total{instance="app"} 9`,
`sww_remote_log_lines_sent_total{instance="app"} 10`,
} {
if !strings.Contains(scraped, "\n"+line+"\n") {
t.Errorf("metrics\n%s\nwant\n%s", scraped, line)
}
}
}
func TestPartThatIsOffAddsNoMetric(t *testing.T) {
t.Parallel()
scraped := scrape(t, metrics.New(metrics.Params{
BusiestCount: 1,
InstanceName: "app",
}))
for _, name := range []string{
"sww_bans_made_total",
"sww_tracked_clients",
"sww_rules_loaded",
"sww_reputation_",
"sww_alert",
"sww_lookup_database_",
"sww_remote_log_",
} {
if strings.Contains(scraped, name) {
t.Errorf("metrics\n%s\nshow %s", scraped, name)
}
}
}
// scrape returns the metrics m serves.
func scrape(t *testing.T, m *metrics.Metrics) string {
t.Helper()
scraped := httptest.NewRecorder()
m.ServeHTTP(
scraped,
httptest.NewRequestWithContext(
t.Context(), http.MethodGet, "/", http.NoBody,
),
)
return scraped.Body.String()
}
+6 -8
View File
@@ -19,19 +19,15 @@ import (
const day = 24 * time.Hour
// LimitKind is what a limit counts: KindRequests, KindBytes or
// KindRefusals.
type LimitKind string
// The kinds of limits, as the metrics name them.
const (
// KindRequests is a rate limit, on a client's requests.
KindRequests LimitKind = "requests"
KindRequests = "requests"
// KindBytes is a byte limit, on a client's bytes.
KindBytes LimitKind = "bytes"
KindBytes = "bytes"
// KindRefusals is the refusal limit, on a client's requests sww
// refused after a rule file match or for a missing or wrong token.
KindRefusals LimitKind = "refusals"
KindRefusals = "refusals"
)
// Limits are the most requests a client may make in a minute, an hour and
@@ -204,7 +200,9 @@ func New(params Params) *Limiter {
// Hit is a request that takes a client over a rate limit or the refusal
// limit, or whose bytes take it over a byte limit.
type Hit struct {
Kind LimitKind
// Kind is KindRequests for a rate limit, KindBytes for a byte limit,
// KindRefusals for the refusal limit.
Kind string
// Window is "minute", "hour" or "day".
Window string
// Limit is the window's limit, as the client's percentage of it.
+1 -1
View File
@@ -652,7 +652,7 @@ func wantBytesCount(
client netip.Prefix,
now time.Time,
bytes int64,
want ratelimit.LimitKind,
want string,
) {
t.Helper()
+12 -9
View File
@@ -445,9 +445,12 @@ func TestKeyIsSentInTheKeyHeaderAndNeverShown(t *testing.T) {
scores: map[string]int64{suspect: 100},
}
checker := newAbuseIPDB(abuseIPDB, p)
m := metrics.New(metrics.Params{
BusiestCount: 1, InstanceName: instanceName, AbuseIPDB: checker,
})
m := metrics.New(1, "app")
m.AddReputation(
reputation.New(params()),
reputation.NewDNSBL(dnsblParams()),
)
m.AddAbuseIPDB(checker)
// One check that AbuseIPDB answers, and one it refuses with an answer
// that names the key.
@@ -500,9 +503,12 @@ func TestMetricsCountTheChecksTheFailuresAndTheBudgetLeft(t *testing.T) {
p := abuseIPDBParams()
p.DailyBudget = 5
checker := newAbuseIPDB(abuseIPDB, p)
m := metrics.New(metrics.Params{
BusiestCount: 1, InstanceName: instanceName, AbuseIPDB: checker,
})
m := metrics.New(1, "app")
m.AddReputation(
reputation.New(params()),
reputation.NewDNSBL(dnsblParams()),
)
m.AddAbuseIPDB(checker)
// One check that AbuseIPDB answers, and one that fails.
hitFrom(t, checker, suspect, true)
@@ -805,9 +811,6 @@ func wantBudgetLeft(t *testing.T, checker *reputation.AbuseIPDB, want int) {
}
}
// instanceName is the label instance of the metrics the tests build.
const instanceName = "app"
// scrapeMetrics returns the metrics m serves.
func scrapeMetrics(t *testing.T, m *metrics.Metrics) string {
t.Helper()
+4 -6
View File
@@ -358,9 +358,8 @@ func TestMetricsCountEachZonesQueriesAndThoseThatFailed(t *testing.T) {
"99.2.0.192." + otherZone + ".": {rcode: serverFailure},
}}
dnsbl := newDNSBL(resolver, dnsblParams(zone, otherZone))
m := metrics.New(metrics.Params{
BusiestCount: 1, InstanceName: instanceName, DNSBL: dnsbl,
})
m := metrics.New(1, "app")
m.AddReputation(reputation.New(params()), dnsbl)
wantZones(t, dnsbl, listed)
synctest.Wait()
@@ -405,9 +404,8 @@ func TestZoneKeyIsMaskedInTheVerdictsTheFailuresAndTheMetrics(t *testing.T) {
"99.2.0.192." + keyed + ".": {addrs: []string{listing}},
"100.2.0.192." + keyed + ".": {rcode: serverFailure},
}}, p)
m := metrics.New(metrics.Params{
BusiestCount: 1, InstanceName: instanceName, DNSBL: dnsbl,
})
m := metrics.New(1, "app")
m.AddReputation(reputation.New(params()), dnsbl)
// Both clients are asked about before either answer comes, so that
// the failure does not keep the zone from the other query.
+25 -35
View File
@@ -12,94 +12,82 @@ import (
"time"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// Action is what waf did with a request, as its request line names it,
// such as ActionForward.
type Action string
// The action a request line names: what waf did with the request.
const (
// ActionForward is a request waf allowed, which rproxy passes to
// the app.
ActionForward Action = "forward"
ActionForward = "forward"
// ActionRequestLineTooLong is a request whose request line is too long for
// the header limits.
ActionRequestLineTooLong Action = "request_line_too_long"
ActionRequestLineTooLong = "request_line_too_long"
// ActionRequestHeaderTooLarge is a request whose headers are over the
// header limits.
ActionRequestHeaderTooLarge Action = "request_header_too_large"
ActionRequestHeaderTooLarge = "request_header_too_large"
// ActionClientIsRateLimited is a request refused because it took its
// client over a rate limit, which bans the client.
ActionClientIsRateLimited Action = "client_is_rate_limited"
ActionClientIsRateLimited = "client_is_rate_limited"
// ActionTemporarilyBanned is a request refused because a ban that runs
// out covers its client, or because it matched a ban rule or asked for a
// trap path, and so made such a ban.
ActionTemporarilyBanned Action = "temporarily_banned"
ActionTemporarilyBanned = "temporarily_banned"
// ActionPermanentlyBanned is a request refused because a ban that never
// runs out covers its client, or because the request made such a ban or
// made its client's ban permanent.
ActionPermanentlyBanned Action = "permanently_banned"
ActionPermanentlyBanned = "permanently_banned"
// ActionBlockedByBlockRule is a request refused because it matched a
// block rule.
ActionBlockedByBlockRule Action = "blocked_by_block_rule"
ActionBlockedByBlockRule = "blocked_by_block_rule"
// ActionBlockedByCoreRuleSet is a request refused because the Core Rule
// Set scored it at or over SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD.
ActionBlockedByCoreRuleSet Action = "blocked_by_core_rule_set"
ActionBlockedByCoreRuleSet = "blocked_by_core_rule_set"
// ActionClientDenied is a request refused because its client is in
// SWW_WAF_DENY_NETS, in a blocklist while SWW_WAF_BLOCKLIST_ACTION is deny,
// or listed by a DNSBL zone, or scored a hit by AbuseIPDB, while
// SWW_WAF_DNSBL_ABUSEIPDB_ACTION is deny, or that carries a tag in
// SWW_WAF_TAG_DENY.
ActionClientDenied Action = "client_denied"
ActionClientDenied = "client_denied"
// ActionCountryDenied is a request refused because its client's country
// is not in SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES.
ActionCountryDenied Action = "country_denied"
ActionCountryDenied = "country_denied"
// ActionChecksTimedOut is a request refused because the trap paths, the
// rule files and the Core Rule Set had not finished with it within 500
// milliseconds.
ActionChecksTimedOut Action = "checks_timed_out"
ActionChecksTimedOut = "checks_timed_out"
// ActionOwnEndpoint is a request waf answered at one of its own
// endpoints, or a question about a path under /_sww/.
ActionOwnEndpoint Action = "own_endpoint"
ActionOwnEndpoint = "own_endpoint"
)
// Offence is what a request was held as, as its request line names it.
type Offence string
// OffenceLimit is the offence a request line names for a request that
// broke a rate limit or the refusal limit, or whose bytes broke a byte
// limit.
const OffenceLimit Offence = "limit"
const OffenceLimit = "limit"
// LimitHitMinuteRefusals is the limit_hit a request line names for a request
// that broke the refusal limit.
const LimitHitMinuteRefusals = "minute_refusals"
// WAFLineKind is which of waf's two lines about a question a request line
// is: WAFLineDecision or WAFLineReport.
type WAFLineKind string
// The waf_line a request line names, which says which of
// waf's two lines about a question it is.
const (
// WAFLineDecision is waf's line at its decision.
WAFLineDecision WAFLineKind = "decision"
WAFLineDecision = "decision"
// WAFLineReport is waf's line when rproxy's report
// comes, with the decision's fields and the report's together.
WAFLineReport WAFLineKind = "report"
WAFLineReport = "report"
)
// timeLayout is RFC 3339 with milliseconds.
@@ -112,8 +100,9 @@ const timeLayout = "2006-01-02T15:04:05.000Z07:00"
//nolint:tagliatelle // docs/request-log.md names the fields in snake_case
type Line struct {
Type string `json:"type"`
// WAFLine is set on waf's two lines about a question.
WAFLine WAFLineKind `json:"waf_line,omitempty"`
// WAFLine is WAFLineDecision or WAFLineReport
// on waf's two lines about a question.
WAFLine string `json:"waf_line,omitempty"`
// The standard web log fields. Scheme is how the client reached rproxy, or
// the trusted proxy in front of it.
@@ -159,7 +148,7 @@ type Line struct {
Aborted bool `json:"aborted,omitempty"`
// The decision.
Action Action `json:"action"`
Action string `json:"action"`
// WouldAction is, in observe mode, the action enforce mode would have
// taken with a request it would have refused: ActionClientDenied,
// ActionTemporarilyBanned, ActionPermanentlyBanned, ActionCountryDenied,
@@ -167,7 +156,7 @@ type Line struct {
// ActionBlockedByCoreRuleSet, or, in waf, ActionRequestLineTooLong or
// ActionRequestHeaderTooLarge for the header limits, or
// ActionChecksTimedOut.
WouldAction Action `json:"would_action,omitempty"`
WouldAction string `json:"would_action,omitempty"`
// Heuristic is the name of the heuristic that refused the request, in
// observe mode too, and HeuristicFigures are the figures it acted on, by
// name.
@@ -213,8 +202,9 @@ type Line struct {
// RouteWatchTagsMissing, while SWW_WAF_ROUTEWATCH_URL is set, says why the
// request got no tags from routewatch: no_answer_yet, request_failed or
// no_route.
RouteWatchTagsMissing string `json:"routewatch_tags_missing,omitempty"`
Offence Offence `json:"offence,omitempty"`
RouteWatchTagsMissing string `json:"routewatch_tags_missing,omitempty"`
// Offence is the offence the request was held as, OffenceLimit.
Offence string `json:"offence,omitempty"`
// BanExpires is when the ban the request made, or was refused under,
// ends: a time, or "permanent".
BanExpires string `json:"ban_expires,omitempty"`
@@ -256,8 +246,8 @@ type RProxyLine struct {
// WAFBackendNoAnswer is true, the value of
// SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION. It is "" when neither
// gave one.
Disposition wafprotocol.Disposition `json:"disposition,omitempty"`
WAFBackendNoAnswer bool `json:"waf_backend_no_answer,omitempty"`
Disposition string `json:"disposition,omitempty"`
WAFBackendNoAnswer bool `json:"waf_backend_no_answer,omitempty"`
}
// Write writes line to w as one JSON line marked "type":"request".
+7 -16
View File
@@ -2,7 +2,6 @@ package rproxy
import (
"bufio"
"context"
"errors"
"io"
"net"
@@ -23,9 +22,7 @@ type requestBody struct {
// body is the visitor's body, ending in an *http.MaxBytesError past
// SWW_RPROXY_MAX_REQUEST_SIZE.
body io.ReadCloser
// refuse refuses the request with a status, as the request's own refuse
// does.
refuse func(status int)
rq *request
// held is the part read before the request goes to the app, which Read gives
// first.
held []byte
@@ -71,7 +68,7 @@ func (b *requestBody) Read(p []byte) (int, error) {
case errors.Is(err, io.EOF):
b.received.Store(true)
case errors.As(err, &tooLarge):
b.refuse(http.StatusRequestEntityTooLarge)
b.rq.refuse(http.StatusRequestEntityTooLarge)
}
return n, err
@@ -89,13 +86,7 @@ type responseBody struct {
// body is the app's body, ending in an *http.MaxBytesError past
// SWW_RPROXY_MAX_RESPONSE_SIZE.
body io.ReadCloser
// refuse refuses the request with a status, as the request's own refuse
// does.
refuse func(status int)
// responseReceived is called once the app's whole answer has arrived.
responseReceived func()
// ctx is the visitor's request's context.
ctx context.Context //nolint:containedctx // the body ends with the request
rq *request
}
// Read reads from the app's body.
@@ -109,15 +100,15 @@ func (b *responseBody) Read(p []byte) (int, error) {
switch {
case errors.Is(err, io.EOF):
b.responseReceived()
b.rq.responseReceived()
case errors.As(err, &tooLarge):
b.refuse(http.StatusBadGateway)
b.rq.refuse(http.StatusBadGateway)
return n, errResponseTooLarge
case b.ctx.Err() == nil:
case b.rq.in.Context().Err() == nil:
// The answer broke off, not because the visitor went away. If a time
// limit cut it, that refusal came first and is the one kept.
b.refuse(http.StatusBadGateway)
b.rq.refuse(http.StatusBadGateway)
}
return n, err
+1 -9
View File
@@ -1,14 +1,6 @@
package rproxy
import (
"net/http"
"time"
)
// HTTPServer is Go's server s holds, for tests that read its settings.
func (s *Server) HTTPServer() *http.Server {
return s.httpServer
}
import "time"
// StopBreakerClock has the circuit breaker of s read at as the time, so that how
// long requests take cannot matter to it. It is called before s serves.
+5 -5
View File
@@ -56,7 +56,7 @@ func TestServerHasTheFixedLimits(t *testing.T) {
InstanceName: "test",
Level: slog.LevelInfo,
}),
}).HTTPServer()
})
if server.Addr != ":8080" || server.MaxHeaderBytes != 28<<10 ||
server.IdleTimeout != 2*time.Minute || server.ReadHeaderTimeout != time.Minute {
@@ -709,7 +709,7 @@ func TestABodyLeftUnreadIsReadOnlyUntilTheRequestTimeLimit(t *testing.T) {
before := time.Now()
server.ServeHTTP(w, req)
server.Handler.ServeHTTP(w, req)
if w.Code != http.StatusNoContent {
t.Errorf("status %d, want the app's 204", w.Code)
@@ -762,7 +762,7 @@ func TestABodyLeftUnreadIsReadNoMoreOnceTheTimeToSendTheAnswerRunsOut(
before := time.Now()
server.ServeHTTP(w, req)
server.Handler.ServeHTTP(w, req)
if w.Code != http.StatusOK {
t.Errorf("status %d, want the app's 200", w.Code)
@@ -815,7 +815,7 @@ func TestAnAnswerRProxyGivesItselfHasTheResponseTimeToBeSent(t *testing.T) {
before := time.Now()
server.ServeHTTP(w, req)
server.Handler.ServeHTTP(w, req)
if w.Code != tc.status {
t.Errorf("status %d, want %d", w.Code, tc.status)
@@ -856,7 +856,7 @@ func TestAnAnswerRProxyGivesItselfHasTenSecondsToBeSentAtLeast(
before := time.Now()
server.ServeHTTP(w, req)
server.Handler.ServeHTTP(w, req)
if w.writeDeadline.Before(before.Add(least)) {
t.Errorf(
+1 -1
View File
@@ -583,7 +583,7 @@ func TestAQuestionNamesNoZone(t *testing.T) {
req.RemoteAddr = tc.remoteAddr
req.Header.Set(forwardedForHeader, tc.forwardedFor)
server.ServeHTTP(httptest.NewRecorder(), req)
server.Handler.ServeHTTP(httptest.NewRecorder(), req)
asked := standIn.question(t)
if asked.header.Get(wafprotocol.ClientHeader) != tc.client ||
+165 -38
View File
@@ -12,6 +12,7 @@ import (
"net/url"
"os"
"strconv"
"sync"
"sync/atomic"
"time"
@@ -62,7 +63,7 @@ type request struct {
// disposition is what is done with the request: waf's valid disposition,
// or SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION. It is "" when neither
// gave one.
disposition wafprotocol.Disposition
disposition string
// answered is true when waf gave disposition, which makes a report due,
// and wafNoAnswer when SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION did.
answered bool
@@ -81,9 +82,17 @@ type request struct {
// upgraded is the connection to the app once the app has switched protocols,
// as for a WebSocket, and nil otherwise.
upgraded *upgradedConn
// limits are the request's SWW_RPROXY_MAX_REQUEST_TIME_MS and
// SWW_RPROXY_MAX_RESPONSE_TIME_MS.
limits *timeLimits
// mu guards what follows. The time limits run on goroutines of their own,
// and the transport starts and stops them, and notes requestEnd, from its
// own; once timersStopped is set, neither time limit acts any more.
mu sync.Mutex
timersStopped bool
requestTimer *time.Timer
responseTimer *time.Timer
// requestEnd is when the app had been sent the whole request, or the answer
// started, if that was sooner.
requestEnd time.Time
}
// newRequest starts handling r: it notes the time, works out the visitor from
@@ -113,30 +122,17 @@ func (h *handler) newRequest(
if r.Body != http.NoBody {
rq.body = &requestBody{
body: limitBody(r.Body, h.config.MaxRequestSize),
refuse: rq.refuse,
body: limitBody(r.Body, h.config.MaxRequestSize),
rq: rq,
}
}
rq.limits = &timeLimits{
maxRequestTime: h.config.MaxRequestTime,
maxResponseTime: h.config.MaxResponseTime,
start: rq.start,
refuse: rq.refuse,
waitingForVisitor: func() bool {
return rq.body != nil && rq.body.waiting.Load()
},
endVisitorRead: func() {
_ = rq.rc.SetReadDeadline(time.Now())
},
}
return rq
}
// answerHealth answers the health check.
func (rq *request) answerHealth() {
rq.prepareOwnAnswer(rq.limits.requestDeadline())
rq.prepareOwnAnswer(rq.requestDeadline())
rq.out.Header().Set("Content-Type", plainText)
rq.send(http.StatusOK, []byte("ok\n"))
}
@@ -152,7 +148,7 @@ func (rq *request) readFirstPartOfBody() bool {
return true
}
_ = rq.rc.SetReadDeadline(rq.limits.requestDeadline())
_ = rq.rc.SetReadDeadline(rq.requestDeadline())
held, err := io.ReadAll(
io.LimitReader(rq.body, wafprotocol.QuestionBodyMaxBytes),
)
@@ -192,7 +188,7 @@ func (rq *request) forward(ctx context.Context) {
rq.cancel = cancel
ctx = httptrace.WithClientTrace(ctx, &httptrace.ClientTrace{
WroteRequest: rq.limits.wroteRequest,
WroteRequest: rq.wroteRequest,
})
out := rq.in.WithContext(ctx)
@@ -209,7 +205,7 @@ func (rq *request) forward(ctx context.Context) {
ErrorHandler: rq.answerError,
}
rq.limits.startRequestTimer()
rq.startRequestTimer()
rq.upstreamStart = time.Now()
reverseProxy.ServeHTTP(rq.out, out)
}
@@ -252,7 +248,7 @@ func (rq *request) modifyResponse(res *http.Response) error {
// switch, as when the app's Upgrade is not the visitor's, so the status
// is noted only once it has taken over the connection, by rq.out's
// Hijack.
rq.limits.stop()
rq.stopTimers()
conn, ok := res.Body.(io.ReadWriteCloser)
if ok {
@@ -263,7 +259,7 @@ func (rq *request) modifyResponse(res *http.Response) error {
return nil
}
requestEnd := rq.limits.endRequest()
requestEnd := rq.endRequest()
// An answer that arrives once SWW_RPROXY_MAX_RESPONSE_TIME_MS has run out is not
// passed on: the visitor is answered 504.
@@ -293,14 +289,9 @@ func (rq *request) modifyResponse(res *http.Response) error {
// An answer that arrives before the app has been sent the whole request ends
// the request, and SWW_RPROXY_MAX_RESPONSE_TIME_MS starts.
rq.limits.startResponseTimer(requestEnd)
rq.startResponseTimer(requestEnd)
res.Body = &responseBody{
body: limitBody(res.Body, maxBytes),
refuse: rq.refuse,
responseReceived: rq.limits.stop,
ctx: rq.in.Context(),
}
res.Body = &responseBody{body: limitBody(res.Body, maxBytes), rq: rq}
rq.out.afterStatus = rq.setResponseDeadline
return nil
@@ -351,7 +342,7 @@ func (rq *request) answer(status int) {
// A visitor found too slow is read no more; any other may go on sending until
// its time is up, so that the rest of the body is read and the request ends
// cleanly: by Go's server, or by readRestOfBody.
deadline := rq.limits.requestDeadline()
deadline := rq.requestDeadline()
if status == http.StatusRequestTimeout {
deadline = time.Now()
}
@@ -397,7 +388,7 @@ func (rq *request) closeAfterAnswer() {
// used that time up first, and at least minSendTime.
func (rq *request) prepareOwnAnswer(deadline time.Time) {
rq.stopReadingBody(deadline)
rq.limits.endRequest()
rq.endRequest()
limit := rq.h.config.MaxResponseTime
if limit > 0 {
@@ -418,7 +409,7 @@ func (rq *request) refuse(status int) {
// finish ends the request's time limits, writes its log line and, if waf
// answered the question, sends the report. ctx is the request's own context.
func (rq *request) finish(ctx context.Context) {
rq.limits.stop()
rq.stopTimers()
// After a cut-off answer, the reading of a body that has not arrived whole
// ends now. Go's server would otherwise keep the connection open while it
@@ -436,10 +427,10 @@ func (rq *request) finish(ctx context.Context) {
errors.Is(rq.rc.Flush(), os.ErrDeadlineExceeded):
rq.stopPassingOnBody(time.Now())
case rq.refused.Load() == 0:
rq.stopPassingOnBody(rq.limits.requestDeadline())
rq.stopPassingOnBody(rq.requestDeadline())
}
requestEnd := rq.limits.endRequest()
requestEnd := rq.endRequest()
end := time.Now()
requestBytes, responseBytes := rq.bytesEachWay()
@@ -513,6 +504,17 @@ func (rq *request) bytesEachWay() (int64, int64) {
return requestBytes, responseBytes
}
// requestDeadline is when the visitor must have sent its whole request, or zero
// when SWW_RPROXY_MAX_REQUEST_TIME_MS is off.
func (rq *request) requestDeadline() time.Time {
limit := rq.h.config.MaxRequestTime
if limit == 0 {
return time.Time{}
}
return rq.start.Add(limit)
}
// stopReadingBody ends, at deadline, the reading of a visitor's body that has not
// arrived whole: Go's server then reads no more of it, and closes the connection
// after the answer.
@@ -569,6 +571,100 @@ func (rq *request) stopPassingOnBody(deadline time.Time) {
rq.stopReadingBody(deadline)
}
// startRequestTimer starts SWW_RPROXY_MAX_REQUEST_TIME_MS, which runs from the
// request's headers being read until the app has been sent the whole request.
func (rq *request) startRequestTimer() {
deadline := rq.requestDeadline()
if deadline.IsZero() {
return
}
rq.mu.Lock()
defer rq.mu.Unlock()
rq.requestTimer = time.AfterFunc(
time.Until(deadline),
rq.requestTimedOut,
)
}
// requestTimedOut is called when SWW_RPROXY_MAX_REQUEST_TIME_MS runs out while the
// request is still on its way to the app. The answer names the side rproxy was
// waiting on at that moment: 408 when it was waiting for the visitor to send more
// of its body, 504 when it was waiting for the app to be reached or to take what
// it had.
func (rq *request) requestTimedOut() {
rq.mu.Lock()
defer rq.mu.Unlock()
if rq.timersStopped {
return
}
if rq.body == nil || !rq.body.waiting.Load() {
rq.refuse(http.StatusGatewayTimeout)
return
}
rq.refuse(http.StatusRequestTimeout)
// The transport gives up on the app only once its Read of the visitor's body
// returns, so that Read is ended now. The lock keeps this from reaching the
// connection after the request is handled.
_ = rq.rc.SetReadDeadline(time.Now())
}
// wroteRequest is called once the app has been sent the whole request:
// SWW_RPROXY_MAX_REQUEST_TIME_MS ends and SWW_RPROXY_MAX_RESPONSE_TIME_MS starts.
func (rq *request) wroteRequest(info httptrace.WroteRequestInfo) {
if info.Err != nil {
return // the transport gives up, or tries again
}
requestEnd := rq.endRequest()
rq.mu.Lock()
stopTimer(rq.requestTimer)
rq.mu.Unlock()
rq.startResponseTimer(requestEnd)
}
// startResponseTimer starts SWW_RPROXY_MAX_RESPONSE_TIME_MS, which runs from
// requestEnd, unless it has started already.
func (rq *request) startResponseTimer(requestEnd time.Time) {
limit := rq.h.config.MaxResponseTime
if limit == 0 {
return
}
rq.mu.Lock()
defer rq.mu.Unlock()
if rq.timersStopped || rq.responseTimer != nil {
return
}
rq.responseTimer = time.AfterFunc(time.Until(requestEnd.Add(limit)),
rq.responseTimedOut)
}
// responseTimedOut is called when SWW_RPROXY_MAX_RESPONSE_TIME_MS runs out before the
// app has sent its whole answer.
func (rq *request) responseTimedOut() {
rq.mu.Lock()
defer rq.mu.Unlock()
if !rq.timersStopped {
rq.refuse(http.StatusGatewayTimeout)
}
}
// responseReceived is called once the app has sent its whole answer.
func (rq *request) responseReceived() {
rq.stopTimers()
}
// setResponseDeadline sends the status and headers of the app's answer, with
// at least minSendTime, before ReverseProxy reads the answer's body, and then
// the rest has until SWW_RPROXY_MAX_RESPONSE_TIME_MS after the end of the request.
@@ -578,7 +674,7 @@ func (rq *request) setResponseDeadline() {
return
}
deadline := rq.limits.endRequest().Add(limit)
deadline := rq.endRequest().Add(limit)
_ = rq.rc.SetWriteDeadline(
time.Now().Add(max(time.Until(deadline), minSendTime)),
@@ -586,3 +682,34 @@ func (rq *request) setResponseDeadline() {
_ = rq.rc.Flush()
_ = rq.rc.SetWriteDeadline(deadline)
}
// endRequest notes now as the end of the request, unless an end is noted, and
// returns the end noted.
func (rq *request) endRequest() time.Time {
rq.mu.Lock()
defer rq.mu.Unlock()
if rq.requestEnd.IsZero() {
rq.requestEnd = time.Now()
}
return rq.requestEnd
}
// stopTimers stops the request's time limits and keeps any from starting later:
// the app's answer is complete, the connection upgraded, or the request handled.
func (rq *request) stopTimers() {
rq.mu.Lock()
defer rq.mu.Unlock()
rq.timersStopped = true
stopTimer(rq.requestTimer)
stopTimer(rq.responseTimer)
}
// stopTimer stops t, which is nil when its time limit is off.
func stopTimer(t *time.Timer) {
if t != nil {
t.Stop()
}
}
+25 -41
View File
@@ -16,11 +16,11 @@ import (
"io"
"log"
"log/slog"
"net"
"net/http"
"net/url"
"slices"
"strings"
"sync"
"time"
"sneak.berlin/go/smallwebwaf/internal/config"
@@ -64,8 +64,9 @@ type Params struct {
// Server is the server `sww rproxy` runs.
type Server struct {
httpServer *http.Server
h *handler
*http.Server
h *handler
}
// New returns the server `sww rproxy` runs. Go's server itself refuses
@@ -94,47 +95,28 @@ func New(params Params) *Server {
breakerClock: time.Now,
}
s := &Server{h: h}
s.httpServer = &http.Server{
Addr: params.Config.ListenAddr,
Handler: s,
ErrorLog: errorLog,
ReadHeaderTimeout: params.Config.MaxRequestTime,
IdleTimeout: idleTimeout,
MaxHeaderBytes: requestHeaderMaxBytes - goServerExtraHeaderBytes,
return &Server{
Server: &http.Server{
Addr: params.Config.ListenAddr,
Handler: h,
ErrorLog: errorLog,
ReadHeaderTimeout: params.Config.MaxRequestTime,
IdleTimeout: idleTimeout,
MaxHeaderBytes: requestHeaderMaxBytes - goServerExtraHeaderBytes,
},
h: h,
}
return s
}
// Serve serves requests on listener until Shutdown or Close, as Go's
// http.Server.Serve does.
func (s *Server) Serve(listener net.Listener) error {
return s.httpServer.Serve(listener)
}
// Shutdown stops serving once the requests in progress have finished, or
// when ctx is done, as Go's http.Server.Shutdown does.
func (s *Server) Shutdown(ctx context.Context) error {
return s.httpServer.Shutdown(ctx)
}
// Close stops serving at once and closes every connection, as Go's
// http.Server.Close does.
func (s *Server) Close() error {
return s.httpServer.Close()
}
// ServeHTTP handles one request from a visitor, as handler.ServeHTTP says.
func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
s.h.ServeHTTP(w, r)
}
// WaitForReports returns once the reports being sent have gone or failed,
// which SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS bounds. Any report after this call
// is dropped: that of a WebSocket that closes while rproxy stops, for one.
func (s *Server) WaitForReports() {
s.h.reports.stopAndWait()
s.h.reportsMu.Lock()
s.h.reportsStopped = true
s.h.reportsMu.Unlock()
s.h.reports.Wait()
}
// handler is the proxy. It holds what every request shares; what belongs to
@@ -154,8 +136,11 @@ type handler struct {
// the time.
breaker breaker
breakerClock func() time.Time
// reports are the goroutines sending reports to waf.
reports reportGoroutines
// reports waits for the reports being sent. reportsMu guards starting one
// against reportsStopped, which is set once nothing is to wait for another.
reports sync.WaitGroup
reportsMu sync.Mutex
reportsStopped bool
}
// newTransport returns what carries requests to the app. It never goes through
@@ -231,8 +216,7 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// This panic has Go's server close the connection without an answer,
// and log nothing; the deferred finish still runs.
panic(http.ErrAbortHandler)
case wafprotocol.DispositionDeny, "":
// "" is an error, or no valid disposition
default: // a deny, an error, or an answer without a valid disposition
rq.answerWith(answer)
}
}
+4 -4
View File
@@ -72,9 +72,9 @@ const (
// What the stand-in for waf answers.
const (
dispositionAllow = string(wafprotocol.DispositionAllow)
dispositionDeny = string(wafprotocol.DispositionDeny)
dispositionClose = string(wafprotocol.DispositionClose)
dispositionAllow = wafprotocol.DispositionAllow
dispositionDeny = wafprotocol.DispositionDeny
dispositionClose = wafprotocol.DispositionClose
// deniedBody is the body of a deny.
deniedBody = "refused by waf"
)
@@ -710,7 +710,7 @@ func TestAnAnswerWithoutAContentTypeGetsNone(t *testing.T) {
)
w := httptest.NewRecorder()
server.ServeHTTP(w, req)
server.Handler.ServeHTTP(w, req)
// Go's server guesses a type when the first part of the body goes before the
// headers are flushed, which is a matter of timing here. So this looks for what
-165
View File
@@ -1,165 +0,0 @@
package rproxy
import (
"net/http"
"net/http/httptrace"
"sync"
"time"
)
// timeLimits are one request's two time limits: SWW_RPROXY_MAX_REQUEST_TIME_MS,
// which runs from the request's headers being read until the app has been sent
// the whole request, and SWW_RPROXY_MAX_RESPONSE_TIME_MS, which runs from then
// until the app has sent its whole answer. A limit of zero is off. Each limit
// runs on a goroutine of its own, and the transport starts and stops them from
// its own.
type timeLimits struct {
maxRequestTime time.Duration
maxResponseTime time.Duration
// start is when the request's headers had been read.
start time.Time
// refuse refuses the request with 408 or 504 when a limit runs out.
refuse func(status int)
// waitingForVisitor reports whether a read of the visitor's body is waiting
// for the visitor to send more.
waitingForVisitor func() bool
// endVisitorRead ends at once a read of the visitor's body that is waiting.
endVisitorRead func()
// mu guards what follows. Once stopped is set, neither limit acts any more.
mu sync.Mutex
stopped bool
requestTimer *time.Timer
responseTimer *time.Timer
// requestEnd is when the app had been sent the whole request, or the answer
// started, if that was sooner.
requestEnd time.Time
}
// requestDeadline is when the visitor must have sent its whole request, or zero
// when SWW_RPROXY_MAX_REQUEST_TIME_MS is off.
func (l *timeLimits) requestDeadline() time.Time {
if l.maxRequestTime == 0 {
return time.Time{}
}
return l.start.Add(l.maxRequestTime)
}
// startRequestTimer starts SWW_RPROXY_MAX_REQUEST_TIME_MS.
func (l *timeLimits) startRequestTimer() {
deadline := l.requestDeadline()
if deadline.IsZero() {
return
}
l.mu.Lock()
defer l.mu.Unlock()
l.requestTimer = time.AfterFunc(time.Until(deadline), l.requestTimedOut)
}
// requestTimedOut is called when SWW_RPROXY_MAX_REQUEST_TIME_MS runs out while
// the request is still on its way to the app. The answer names the side rproxy
// was waiting on at that moment: 408 when it was waiting for the visitor to send
// more of its body, 504 when it was waiting for the app to be reached or to take
// what it had.
func (l *timeLimits) requestTimedOut() {
l.mu.Lock()
defer l.mu.Unlock()
if l.stopped {
return
}
if !l.waitingForVisitor() {
l.refuse(http.StatusGatewayTimeout)
return
}
l.refuse(http.StatusRequestTimeout)
// The transport gives up on the app only once its Read of the visitor's body
// returns, so that Read is ended now. The lock keeps this from reaching the
// connection after the request is handled.
l.endVisitorRead()
}
// wroteRequest is called once the app has been sent the whole request:
// SWW_RPROXY_MAX_REQUEST_TIME_MS ends and SWW_RPROXY_MAX_RESPONSE_TIME_MS
// starts.
func (l *timeLimits) wroteRequest(info httptrace.WroteRequestInfo) {
if info.Err != nil {
return // the transport gives up, or tries again
}
requestEnd := l.endRequest()
l.mu.Lock()
stopTimer(l.requestTimer)
l.mu.Unlock()
l.startResponseTimer(requestEnd)
}
// startResponseTimer starts SWW_RPROXY_MAX_RESPONSE_TIME_MS, which runs from
// requestEnd, unless it has started already.
func (l *timeLimits) startResponseTimer(requestEnd time.Time) {
if l.maxResponseTime == 0 {
return
}
l.mu.Lock()
defer l.mu.Unlock()
if l.stopped || l.responseTimer != nil {
return
}
l.responseTimer = time.AfterFunc(
time.Until(requestEnd.Add(l.maxResponseTime)),
l.responseTimedOut,
)
}
// responseTimedOut is called when SWW_RPROXY_MAX_RESPONSE_TIME_MS runs out
// before the app has sent its whole answer.
func (l *timeLimits) responseTimedOut() {
l.mu.Lock()
defer l.mu.Unlock()
if !l.stopped {
l.refuse(http.StatusGatewayTimeout)
}
}
// endRequest notes now as the end of the request, unless an end is noted, and
// returns the end noted.
func (l *timeLimits) endRequest() time.Time {
l.mu.Lock()
defer l.mu.Unlock()
if l.requestEnd.IsZero() {
l.requestEnd = time.Now()
}
return l.requestEnd
}
// stop stops both limits and keeps either from starting later: the app's
// answer is complete, the connection upgraded, or the request handled.
func (l *timeLimits) stop() {
l.mu.Lock()
defer l.mu.Unlock()
l.stopped = true
stopTimer(l.requestTimer)
stopTimer(l.responseTimer)
}
// stopTimer stops t, which is nil when its limit is off.
func stopTimer(t *time.Timer) {
if t != nil {
t.Stop()
}
}
-171
View File
@@ -1,171 +0,0 @@
package rproxy
import (
"net/http"
"net/http/httptrace"
"sync/atomic"
"testing"
"testing/synctest"
"time"
)
// The tests below run timeLimits in a synctest bubble, where time is a clock of
// the test's own: time.Sleep moves it on at once, and synctest.Wait returns once
// a limit that ran out has refused the request.
const (
testRequestTime = 3 * time.Second
testResponseTime = 5 * time.Second
)
// limitsSeen is what a timeLimits under test did to its request.
type limitsSeen struct {
// refused is the status of the last refusal, or zero.
refused atomic.Int64
// readEnded is true once the read of the visitor's body was ended.
readEnded atomic.Bool
// waiting is what waitingForVisitor reports.
waiting atomic.Bool
}
// newTestLimits returns time limits that started now, with what they do noted
// in seen.
func newTestLimits(seen *limitsSeen) *timeLimits {
return &timeLimits{
maxRequestTime: testRequestTime,
maxResponseTime: testResponseTime,
start: time.Now(),
refuse: func(status int) {
seen.refused.Store(int64(status))
},
waitingForVisitor: seen.waiting.Load,
endVisitorRead: func() {
seen.readEnded.Store(true)
},
}
}
// sleepThenWait moves the bubble's clock on by d and waits for what is then
// due.
func sleepThenWait(d time.Duration) {
time.Sleep(d)
synctest.Wait()
}
func TestRequestTimeRunningOutWhileTheVisitorIsWaitedOnRefuses408(
t *testing.T,
) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
seen := &limitsSeen{}
seen.waiting.Store(true)
limits := newTestLimits(seen)
limits.startRequestTimer()
sleepThenWait(testRequestTime - time.Millisecond)
if got := seen.refused.Load(); got != 0 {
t.Fatalf(
"refused %d before the limit ran out, want nothing",
got,
)
}
sleepThenWait(time.Millisecond)
if got := seen.refused.Load(); got != http.StatusRequestTimeout {
t.Errorf("refused %d, want 408", got)
}
if !seen.readEnded.Load() {
t.Error("the read of the visitor's body goes on, want it ended")
}
})
}
func TestRequestTimeRunningOutWhileTheAppIsWaitedOnRefuses504(
t *testing.T,
) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
seen := &limitsSeen{}
limits := newTestLimits(seen)
limits.startRequestTimer()
sleepThenWait(testRequestTime)
if got := seen.refused.Load(); got != http.StatusGatewayTimeout {
t.Errorf("refused %d, want 504", got)
}
if seen.readEnded.Load() {
t.Error("the read of the visitor's body was ended, want it not")
}
})
}
func TestResponseTimeRunsFromTheEndOfTheRequest(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
seen := &limitsSeen{}
limits := newTestLimits(seen)
limits.startRequestTimer()
sleepThenWait(time.Second)
end := time.Now()
limits.wroteRequest(httptrace.WroteRequestInfo{})
if got := limits.endRequest(); !got.Equal(end) {
t.Errorf("the request ended at %v, want %v", got, end)
}
// Past SWW_RPROXY_MAX_REQUEST_TIME_MS, which the end of the request
// stopped, and just short of SWW_RPROXY_MAX_RESPONSE_TIME_MS.
sleepThenWait(testResponseTime - time.Millisecond)
if got := seen.refused.Load(); got != 0 {
t.Fatalf(
"refused %d before the limit ran out, want nothing",
got,
)
}
sleepThenWait(time.Millisecond)
if got := seen.refused.Load(); got != http.StatusGatewayTimeout {
t.Errorf("refused %d, want 504", got)
}
})
}
func TestStoppedLimitsRefuseNothing(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
seen := &limitsSeen{}
seen.waiting.Store(true)
limits := newTestLimits(seen)
limits.startRequestTimer()
limits.stop()
limits.startResponseTimer(limits.endRequest())
sleepThenWait(testRequestTime + testResponseTime)
if got := seen.refused.Load(); got != 0 {
t.Errorf("refused %d, want nothing", got)
}
if seen.readEnded.Load() {
t.Error("the read of the visitor's body was ended, want it not")
}
})
}
+9 -34
View File
@@ -8,7 +8,6 @@ import (
"maps"
"net/http"
"strings"
"sync"
"sneak.berlin/go/smallwebwaf/internal/rules"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
@@ -189,7 +188,7 @@ func canBeAsked(r *http.Request) bool {
// answerWith gives the visitor waf's answer, less its disposition.
func (rq *request) answerWith(answer *wafAnswer) {
rq.prepareOwnAnswer(rq.limits.requestDeadline())
rq.prepareOwnAnswer(rq.requestDeadline())
header := rq.out.Header()
maps.Copy(header, answer.header)
@@ -211,7 +210,14 @@ func (h *handler) sendReport(
ctx context.Context,
report *wafprotocol.Report,
) {
h.reports.startUnlessStopped(func() {
h.reportsMu.Lock()
defer h.reportsMu.Unlock()
if h.reportsStopped {
return
}
h.reports.Go(func() {
err := h.postReport(context.WithoutCancel(ctx), report)
if err != nil {
h.processLog.Warn("a report is dropped", "error", err.Error())
@@ -219,37 +225,6 @@ func (h *handler) sendReport(
})
}
// reportGoroutines are the goroutines sending reports to waf. None starts once
// stopAndWait has been called.
type reportGoroutines struct {
mu sync.Mutex
stopped bool
running sync.WaitGroup
}
// startUnlessStopped runs send on a goroutine of its own, or drops it once
// stopAndWait has been called.
func (g *reportGoroutines) startUnlessStopped(send func()) {
g.mu.Lock()
defer g.mu.Unlock()
if g.stopped {
return
}
g.running.Go(send)
}
// stopAndWait stops any more goroutines from starting, and returns once those
// running have ended.
func (g *reportGoroutines) stopAndWait() {
g.mu.Lock()
g.stopped = true
g.mu.Unlock()
g.running.Wait()
}
// postReport posts report to waf, and waits for its answer for up to
// SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS.
func (h *handler) postReport(
+55 -29
View File
@@ -18,31 +18,34 @@ import (
"slices"
"strings"
"sync/atomic"
"time"
"github.com/fsnotify/fsnotify"
"golang.org/x/net/http/httpguts"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/dirwatch"
)
// Action is what a rule does when it matches: ActionLog, ActionBlock or
// ActionBan.
type Action string
// The actions a rule takes when it matches.
const (
// ActionLog notes the match in the request log, and does nothing else.
ActionLog Action = "log"
ActionLog = "log"
// ActionBlock refuses the request with 403.
ActionBlock Action = "block"
ActionBlock = "block"
// ActionBan refuses the request and bans the client's netblock: the
// request is a clear sign of attack.
ActionBan Action = "ban"
ActionBan = "ban"
)
// extension ends the name of every rule file.
const extension = ".rules"
// quietTime is how long SWW_WAF_RULE_FILES_DIR must go without a change before
// the rule files are read again, so that a file still being written, such
// as one saved in place, appended to or copied in with scp, is read only
// once whole.
const quietTime = 2 * time.Second
// headerTarget starts the target that is one request header,
// header:<Name>.
const headerTarget = "header:"
@@ -92,7 +95,8 @@ type Rule struct {
// Target is what the regex is matched against, such as path or
// header:Accept.
Target string
Action Action
// Action is ActionLog, ActionBlock or ActionBan.
Action string
regex *regexp.Regexp
}
@@ -172,18 +176,26 @@ func (f *Files) Len() int {
}
// Watch watches Dir until ctx is done, and reads the rule files again
// once Dir has had no change for a short time, after one is edited, added
// or removed, and after Watch starts watching, as dirwatch describes. If
// they then hold an error, the rules stay as they were, the error is
// logged with its file and line, and the files are read again after the
// next change. If Dir cannot be watched, that is logged, and the rules
// stay as they were loaded. While Enabled is false, Watch returns at once.
// once Dir has had no change for quietTime, after one is edited, added or
// removed, and after Watch starts watching. If they then hold an error,
// the rules stay as they were, the error is logged with its file and
// line, and the files are read again after the next change. If Dir cannot
// be watched, that is logged, and the rules stay as they were loaded.
// While Enabled is false, Watch returns at once.
func (f *Files) Watch(ctx context.Context) {
if !f.params.Enabled {
return
}
watcher, err := dirwatch.New(f.params.Dir)
watcher, err := fsnotify.NewWatcher()
if err == nil {
defer func() {
_ = watcher.Close()
}()
err = watcher.Add(f.params.Dir)
}
if err != nil {
f.params.ProcessLog.Error("cannot watch the rule files for edits",
"error", err.Error())
@@ -191,22 +203,36 @@ func (f *Files) Watch(ctx context.Context) {
return
}
defer func() {
_ = watcher.Close()
}()
f.params.ProcessLog.Info("watching the rule files for edits",
"directory", f.params.Dir)
// A change to any file in Dir counts.
watcher.CallWhenQuiet(ctx,
func(string) bool { return true },
f.readAgain,
func(err error) {
f.readAfterChanges(ctx, watcher.Events, watcher.Errors)
}
// readAfterChanges reads the rule files again once quietTime has passed
// without a change from events, until ctx is done, and logs the errors
// from errs. The wait starts at once, as if for a change, so that an edit
// saved after Load read the files, and before Dir was watched, is taken
// in too.
func (f *Files) readAfterChanges(
ctx context.Context, events <-chan fsnotify.Event, errs <-chan error,
) {
quiet := time.NewTimer(quietTime)
defer quiet.Stop()
for {
select {
case <-ctx.Done():
return
case <-events:
quiet.Reset(quietTime)
case <-quiet.C:
f.readAgain()
case err := <-errs:
f.params.ProcessLog.Warn("watching the rule files failed",
"error", err.Error())
},
)
}
}
}
// readAgain reads the rule files again, in place of the rules loaded, or
@@ -340,7 +366,7 @@ func parse(line string) (Rule, bool, error) {
return Rule{}, false, errNotRule
}
rule := Rule{ID: fields[1], Target: fields[2], Action: Action(fields[3])}
rule := Rule{ID: fields[1], Target: fields[2], Action: fields[3]}
headerName, isHeader := strings.CutPrefix(rule.Target, headerTarget)
switch {
@@ -368,7 +394,7 @@ func parse(line string) (Rule, bool, error) {
rule.Target,
errHeaderTakenOut,
)
case !slices.Contains([]Action{ActionLog, ActionBlock, ActionBan}, rule.Action):
case !slices.Contains([]string{ActionLog, ActionBlock, ActionBan}, rule.Action):
return Rule{}, false, fmt.Errorf(
"the action %q %w",
rule.Action,
+165
View File
@@ -0,0 +1,165 @@
package rules
import (
"context"
"log/slog"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"slices"
"testing"
"testing/synctest"
"time"
"github.com/fsnotify/fsnotify"
"sneak.berlin/go/smallwebwaf/internal/alerts"
)
// The tests below run readAfterChanges in a synctest bubble, where time is
// a clock of the test's own: time.Sleep moves it on at once, and
// synctest.Wait returns once readAfterChanges waits again, so that every
// reading due by then is done. The test sends the changes itself, as the
// watch of a directory cannot run in a bubble.
func TestFileWrittenInTwoPartsTakenInOnlyWhole(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "50-app.rules")
writeFile(t, path, "first path block ^/first\n")
files := load(t, dir)
changes := run(t, files)
file, err := os.Create(path) //nolint:gosec // a file the test wrote
if err != nil {
t.Fatalf("create: %v", err)
}
defer func() {
_ = file.Close()
}()
// The first part ends in the middle of a ban rule's regex, which,
// read then, would ban every request.
write(t, file, "first path block ^/first\nprobe path ban ^/")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantMatched(t, files, "/anything")
// The second part starts the wait again.
write(t, file, `\.env$`+"\n")
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
time.Sleep(quietTime - time.Nanosecond)
synctest.Wait()
wantMatched(t, files, "/.env")
time.Sleep(time.Nanosecond)
synctest.Wait()
wantMatched(t, files, "/.env", "probe")
wantMatched(t, files, "/anything")
})
}
func TestEditSavedBeforeTheWatchStartsTakenIn(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "50-app.rules")
writeFile(t, path, "first path block ^/first\n")
files := load(t, dir)
// Saved after Load read the files, and before the directory was
// watched, so that no change is seen for it.
writeFile(t, path, "first path block ^/edited\n")
run(t, files)
time.Sleep(quietTime)
synctest.Wait()
wantMatched(t, files, "/edited", "first")
})
}
// load loads the rules in dir.
func load(t *testing.T, dir string) *Files {
t.Helper()
files, err := Load(Params{
Dir: dir, Enabled: true, ProcessLog: slog.New(slog.DiscardHandler),
Alerts: alerts.New(alerts.Params{}),
})
if err != nil {
t.Fatalf("load: %v", err)
}
return files
}
// run runs files' readAfterChanges until the test ends, and returns the
// channel that sends it changes.
func run(t *testing.T, files *Files) chan<- fsnotify.Event {
t.Helper()
changes := make(chan fsnotify.Event)
ctx, stop := context.WithCancel(t.Context())
stopped := make(chan struct{})
go func() {
files.readAfterChanges(ctx, changes, nil)
close(stopped)
}()
t.Cleanup(func() {
stop()
<-stopped
})
return changes
}
// writeFile writes content to the file at path.
func writeFile(t *testing.T, path, content string) {
t.Helper()
err := os.WriteFile(path, []byte(content), 0o600)
if err != nil {
t.Fatalf("write %s: %v", path, err)
}
}
// write writes text to the end of file.
func write(t *testing.T, file *os.File, text string) {
t.Helper()
_, err := file.WriteString(text)
if err != nil {
t.Fatalf("write: %v", err)
}
}
// wantMatched checks the ids of the rules that a GET request for path
// matches, in order.
func wantMatched(t *testing.T, files *Files, path string, want ...string) {
t.Helper()
r := httptest.NewRequestWithContext(t.Context(), http.MethodGet,
"http://app.example"+path, nil)
matched := files.Match(r)
got := make([]string, 0, len(matched))
for _, rule := range matched {
got = append(got, rule.ID)
}
if !slices.Equal(got, want) {
t.Errorf("%s matched %v, want %v", path, got, want)
}
}
+6 -6
View File
@@ -181,7 +181,7 @@ type BanEntry struct {
Netblock netip.Prefix `json:"netblock"`
Start time.Time `json:"start"`
Expires *time.Time `json:"expires"`
Cause bans.Cause `json:"cause"`
Cause string `json:"cause"`
Reason string `json:"reason,omitempty"`
Lifted *time.Time `json:"lifted,omitempty"`
Notes bans.Notes `json:"notes"`
@@ -237,11 +237,11 @@ type routeWatchFile struct {
//
//nolint:tagliatelle // the state files use snake_case, as the request log does
type alertsFile struct {
Version int `json:"version"`
Cooldowns []alerts.Cooldown `json:"cooldowns"`
Hour alerts.Hour `json:"hour"`
Waiting map[alerts.DestinationName][]alerts.Alert `json:"waiting"`
AnomalyCounters []anomaly.Counter `json:"anomaly_counters"`
Version int `json:"version"`
Cooldowns []alerts.Cooldown `json:"cooldowns"`
Hour alerts.Hour `json:"hour"`
Waiting map[string][]alerts.Alert `json:"waiting"`
AnomalyCounters []anomaly.Counter `json:"anomaly_counters"`
}
// stateFile is the struct of a state file. Once the file is decoded, its
+5 -5
View File
@@ -1758,8 +1758,8 @@ func TestEditOfRouteWatchAndAlertsFilesTakenIn(t *testing.T) {
Event: alerts.EventBan, Netblock: netip.MustParsePrefix("198.51.100.0/24"),
Sent: midnight(),
}},
Hour: alerts.Hour{HeldBack: map[alerts.Event]int{}},
Waiting: map[alerts.DestinationName][]alerts.Alert{
Hour: alerts.Hour{HeldBack: map[string]int{}},
Waiting: map[string][]alerts.Alert{
alerts.DestinationWebhook: {
{Event: alerts.EventFileError, Time: midnight()},
},
@@ -2046,8 +2046,8 @@ func TestBanWithoutACauseTakenInAsAnAdmins(t *testing.T) {
var written struct {
Bans []struct {
Cause bans.Cause `json:"cause"`
Reason string `json:"reason"`
Cause string `json:"cause"`
Reason string `json:"reason"`
} `json:"bans"`
}
@@ -2430,7 +2430,7 @@ func midnight() time.Time {
// counts, with thresholds fill does not reach.
func newParams(dir string) state.Params {
discard := slog.New(slog.DiscardHandler)
m := metrics.New(metrics.Params{BusiestCount: 1, InstanceName: "app"})
m := metrics.New(1, "app")
queue := alerts.New(alerts.Params{
WebhookURL: &url.URL{Scheme: "https", Host: "alerts.example"},
Events: alerts.Events(),
+1 -9
View File
@@ -95,20 +95,12 @@ func background(loop func(context.Context)) fx.Hook {
)
}
// daemonServer is the server a daemon runs, waf.Server or rproxy.Server: the
// methods of Go's http.Server that serve starts and stops it with.
type daemonServer interface {
Serve(listener net.Listener) error
Shutdown(ctx context.Context) error
Close() error
}
// serve serves requests on listener from the start. At the stop it gives the
// requests in progress shutdownTimeout to finish, then closes their
// connections. When serving fails, it logs why and sends the error to
// failed.
func serve(
lc fx.Lifecycle, server daemonServer, listener net.Listener,
lc fx.Lifecycle, server *http.Server, listener net.Listener,
processLog *slog.Logger, failed servingFailed,
) {
served := make(chan error, 1)
+3 -7
View File
@@ -4,6 +4,7 @@ import (
"context"
"log/slog"
"net"
"net/http"
"os"
"go.uber.org/fx"
@@ -43,13 +44,8 @@ func RunRProxy(ctx context.Context, params Params) int {
app := fx.New(
fx.Supply(cfg, processLog, requestLog{params.Stdout}, failed),
fx.Provide(
// run waits for rproxy.Server's reports, and serve takes it as a
// daemonServer.
fx.Annotate(
newRProxyServer,
fx.As(fx.Self()),
fx.As(new(daemonServer)),
),
newRProxyServer,
func(server *rproxy.Server) *http.Server { return server.Server },
newRProxyListener,
),
fx.Invoke(serve),
+8 -8
View File
@@ -164,7 +164,7 @@ func TestABanThroughOneSiteRefusesTheClientOnTheOther(t *testing.T) {
// One request's three lines carry one id, rproxy's with waf's answer.
line := firstOut.line(t, "path", bannedProbe)
if line["disposition"] != string(wafprotocol.DispositionDeny) ||
if line["disposition"] != wafprotocol.DispositionDeny ||
line["client_ip"] != placed {
t.Errorf("rproxy's line is %v", line)
}
@@ -249,7 +249,7 @@ func TestTheSiteAnswersWhileWAFIsStoppedAndStartedAgain(t *testing.T) {
wantStatus(t, site+"/after-stop", placed, http.StatusOK)
line := out.line(t, "path", "/after-stop")
if line["disposition"] != string(wafprotocol.DispositionAllow) ||
if line["disposition"] != wafprotocol.DispositionAllow ||
line["waf_backend_no_answer"] != true {
t.Errorf("rproxy's line is %v", line)
}
@@ -608,8 +608,8 @@ func TestRProxyAsksAWAFThatSpeaksHTTP2OverHTTP1(t *testing.T) {
if r.Header.Get(wafprotocol.ClientHeader) != "" {
protocols <- r.Proto
w.Header().Set(wafprotocol.DispositionHeader,
string(wafprotocol.DispositionDeny))
w.Header().
Set(wafprotocol.DispositionHeader, wafprotocol.DispositionDeny)
w.WriteHeader(http.StatusForbidden)
}
}))
@@ -692,8 +692,8 @@ func TestRProxyStoppingWaitsForAReportBeingSent(t *testing.T) {
backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(wafprotocol.ClientHeader) != "" {
w.Header().Set(wafprotocol.DispositionHeader,
string(wafprotocol.DispositionAllow))
w.Header().
Set(wafprotocol.DispositionHeader, wafprotocol.DispositionAllow)
return
}
@@ -795,8 +795,8 @@ func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set(wafprotocol.DispositionHeader,
string(wafprotocol.DispositionAllow))
w.Header().
Set(wafprotocol.DispositionHeader, wafprotocol.DispositionAllow)
}))
t.Cleanup(backend.Close)
+30 -52
View File
@@ -10,14 +10,13 @@ package sww
import (
"context"
"errors"
"fmt"
"io"
"os"
"os/signal"
"syscall"
"time"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/config"
@@ -27,7 +26,14 @@ import (
// know.
const usageStatus = 2
var errNoCommand = errors.New("no command given")
// usage is what sww prints then.
const usage = `usage: sww <command>
commands:
rproxy the reverse proxy in front of one app
waf decides for rproxy, and keeps bans, counters and history
healthcheck the container's health check
`
// Params are what RunWAF and RunRProxy need from the process.
type Params struct {
@@ -45,9 +51,14 @@ type Params struct {
// Main runs the command os.Args names, and returns the process's exit status.
// `sww rproxy` and `sww waf` run until SIGTERM or SIGINT.
// With no command, or one it does not know, cobra prints the error and the
// commands to stderr, and Main returns usageStatus.
// With no command, or one it does not know, it prints the commands and returns
// usageStatus.
func Main(version string) int {
args := os.Args[1:]
if len(args) == 0 {
return printUsage(os.Stderr)
}
ctx, stop := signal.NotifyContext(context.Background(),
syscall.SIGTERM, os.Interrupt)
defer stop()
@@ -58,54 +69,21 @@ func Main(version string) int {
Stdout: os.Stdout,
}
// status is what the command that ran returns.
status := 0
root := &cobra.Command{
Use: "sww",
Short: "a web application firewall",
Args: cobra.NoArgs,
RunE: func(*cobra.Command, []string) error {
return errNoCommand
},
switch args[0] {
case "rproxy":
return RunRProxy(ctx, params)
case "waf":
return RunWAF(ctx, params)
case "healthcheck":
return HealthCheck(ctx, args[1:], params.Viper, os.Stderr)
default:
return printUsage(os.Stderr)
}
root.CompletionOptions.DisableDefaultCmd = true
}
root.AddCommand(
&cobra.Command{
Use: "rproxy",
Short: "the reverse proxy in front of one app",
Run: func(cmd *cobra.Command, _ []string) {
status = RunRProxy(cmd.Context(), params)
},
},
&cobra.Command{
Use: "waf",
Short: "decides for rproxy, and keeps bans, counters and history",
Run: func(cmd *cobra.Command, _ []string) {
status = RunWAF(cmd.Context(), params)
},
},
&cobra.Command{
Use: "healthcheck",
Short: "the container's health check",
// HealthCheck refuses every argument itself, flags included.
DisableFlagParsing: true,
Run: func(cmd *cobra.Command, args []string) {
status = HealthCheck(
cmd.Context(),
args,
params.Viper,
cmd.ErrOrStderr(),
)
},
},
)
// printUsage writes the commands to w, and returns usageStatus.
func printUsage(w io.Writer) int {
_, _ = fmt.Fprint(w, usage)
err := root.ExecuteContext(ctx)
if err != nil {
return usageStatus
}
return status
return usageStatus
}
+14 -20
View File
@@ -6,6 +6,7 @@ import (
"io"
"log/slog"
"net"
"net/http"
"time"
"go.uber.org/fx"
@@ -96,8 +97,8 @@ func RunWAF(ctx context.Context, params Params) int {
newAlertQueue, newRuleFiles, newCoreRuleSet, newLookupFile,
newLimiter, newLedger, newAnomalyCounters,
newLists, newDNSBL, newAbuseIPDB, newRouteWatch,
newMetrics, newGeoJS,
fx.Annotate(newWAFServer, fx.As(new(daemonServer))),
newMetrics, newGeoJS, newWAFServer,
func(server *waf.Server) *http.Server { return server.Server },
newWAFListener, newStateFiles,
),
// fx stops hooks in the reverse of the order they were added in. The
@@ -391,33 +392,26 @@ func newMetrics(
alertQueue *alerts.Queue, lookupFile *lookup.File,
remote *remotelog.Sender,
) *metrics.Metrics {
params := metrics.Params{
BusiestCount: cfg.MetricsBusiestCountriesAndASNs,
InstanceName: cfg.InstanceName,
Now: utcNow,
Bans: ledger,
Clients: limiter,
Rules: ruleFiles,
Lists: lists,
DNSBL: dnsbl,
Alerts: alertQueue,
m := metrics.New(cfg.MetricsBusiestCountriesAndASNs, cfg.InstanceName)
m.AddBansAndClients(ledger, limiter, utcNow)
m.AddRules(ruleFiles)
m.AddReputation(lists, dnsbl)
if cfg.AbuseIPDBKey != "" {
m.AddAbuseIPDB(abuseIPDB)
}
// A nil pointer in an interface field is not a nil interface, so a part
// that is off is left out rather than set to nil.
if cfg.AbuseIPDBKey != "" {
params.AbuseIPDB = abuseIPDB
}
m.AddAlerts(alertQueue)
if lookupFile != nil {
params.LookupFile = lookupFile
m.AddLookupFile(lookupFile.LastRead, lookupFile.ReadFailures)
}
if remote != nil {
params.RemoteLog = remote
m.AddRemoteLog(remote)
}
return metrics.New(params)
return m
}
// newGeoJS returns what looks clients up at GeoJS, none looked up yet. Each
+33 -97
View File
@@ -4,7 +4,6 @@ import (
"bytes"
"context"
"crypto/rand"
"errors"
"io"
"net/http"
"os"
@@ -165,114 +164,51 @@ func wantUTC(t *testing.T, where string, value any) {
func TestMainWithoutACommandItKnowsPrintsTheCommands(t *testing.T) {
t.Parallel()
// completion is cobra's own command, which sww turns off.
for _, command := range []string{"", "serve", "completion"} {
t.Run("command "+command, func(t *testing.T) {
t.Parallel()
code, stdout, stderr := runMain(t, command)
if code != 2 {
t.Errorf("exit status %d, want 2", code)
}
wantTheCommands(t, "stderr", stderr)
if stdout != "" {
t.Errorf("stdout %q, want nothing", stdout)
}
})
}
}
func TestMainHelpPrintsTheCommands(t *testing.T) {
t.Parallel()
for _, command := range []string{"help", "--help"} {
t.Run("command "+command, func(t *testing.T) {
t.Parallel()
code, stdout, _ := runMain(t, command)
if code != 0 {
t.Errorf("exit status %d, want 0", code)
}
wantTheCommands(t, "stdout", stdout)
})
}
}
// wantTheCommands checks that text, what Main wrote to where, names each
// command.
func wantTheCommands(t *testing.T, where, text string) {
t.Helper()
for _, name := range []string{"rproxy", wafCommand, "healthcheck"} {
if !strings.Contains(text, name) {
t.Errorf("%s %q does not name %s", where, text, name)
}
}
}
func TestMainHealthCheckRefusesAnArgument(t *testing.T) {
t.Parallel()
for _, argument := range []string{"now", "--x", "-h"} {
t.Run("argument "+argument, func(t *testing.T) {
t.Parallel()
code, stdout, stderr := runMain(t, "healthcheck "+argument)
want := "sww healthcheck: unexpected argument \"" +
argument + "\"\n"
if code != 1 || stderr != want || stdout != "" {
t.Errorf("exit status %d, stdout %q and stderr %q, "+
"want 1, nothing and %q", code, stdout, stderr, want)
}
})
}
}
// runMain runs Main with command in a process of its own, and returns its
// exit status and what it wrote to stdout and stderr.
func runMain(t *testing.T, command string) (int, string, string) {
t.Helper()
binary, err := os.Executable()
if err != nil {
t.Fatalf("find the test binary: %v", err)
}
ctx, cancel := context.WithTimeout(t.Context(), waitLimit)
defer cancel()
for _, command := range []string{"", "serve"} {
t.Run("command "+command, func(t *testing.T) {
t.Parallel()
var stdout, stderr bytes.Buffer
ctx, cancel := context.WithTimeout(t.Context(), waitLimit)
defer cancel()
//nolint:gosec // this test binary
cmd := exec.CommandContext(ctx, binary)
cmd.Stdout, cmd.Stderr = &stdout, &stderr
var stdout, stderr bytes.Buffer
cmd.Env = append(os.Environ(), commandForMain+"="+command)
cmd := exec.CommandContext(ctx, binary)
cmd.Stdout, cmd.Stderr = &stdout, &stderr
err = cmd.Run()
cmd.Env = append(os.Environ(), commandForMain+"="+command)
var exitErr *exec.ExitError
if err != nil && !errors.As(err, &exitErr) {
t.Fatalf("run: %v", err)
err := cmd.Run()
if code := cmd.ProcessState.ExitCode(); code != 2 {
t.Errorf("exit status %d (%v), want 2", code, err)
}
for _, name := range []string{"rproxy", wafCommand, "healthcheck"} {
if !strings.Contains(stderr.String(), name) {
t.Errorf(
"stderr %q does not name %s",
stderr.String(),
name,
)
}
}
if stdout.Len() != 0 {
t.Errorf("stdout %q, want nothing", stdout.String())
}
})
}
return cmd.ProcessState.ExitCode(), stdout.String(), stderr.String()
}
// ask sends waf at addr rproxy's question about a request for path from
// placed, checks the answer's status and disposition, and closes it.
func ask(
t *testing.T,
addr, path string,
status int,
disposition wafprotocol.Disposition,
) {
func ask(t *testing.T, addr, path string, status int, disposition string) {
t.Helper()
req, err := http.NewRequestWithContext(t.Context(), http.MethodGet,
@@ -295,12 +231,12 @@ func ask(
_, _ = io.Copy(io.Discard, res.Body)
_ = res.Body.Close()
answered := res.Header.Get(wafprotocol.DispositionHeader)
if res.StatusCode != status || answered != string(disposition) {
if res.StatusCode != status ||
res.Header.Get(wafprotocol.DispositionHeader) != disposition {
t.Errorf(
"answered %d with disposition %q, want %d with %q",
res.StatusCode,
answered,
res.Header.Get(wafprotocol.DispositionHeader),
status,
disposition,
)
+5 -9
View File
@@ -1,8 +1,7 @@
package waf
// The heuristic abuseipdb. It reads SWW_WAF_ABUSEIPDB_KEY,
// SWW_WAF_ABUSEIPDB_MIN_SCORE, SWW_WAF_DNSBL_ABUSEIPDB_ACTION and
// SWW_WAF_BAN_RESPONSE.
// SWW_WAF_ABUSEIPDB_MIN_SCORE and SWW_WAF_DNSBL_ABUSEIPDB_ACTION.
import (
"context"
@@ -17,11 +16,10 @@ import (
// abuseIPDB adds the tag abuseipdb and notes AbuseIPDB, as noteHit does, with
// the score, when its score of the client is a hit, and refuses the request
// with SWW_WAF_BAN_RESPONSE when SWW_WAF_DNSBL_ABUSEIPDB_ACTION is deny, as
// dnsbl does for a zone. While SWW_WAF_ABUSEIPDB_KEY is unset it does nothing.
// A client without a score is checked in the background, by the request's
// address, if its history counts an offence, and the request does not wait for
// the answer. The score is then
// when SWW_WAF_DNSBL_ABUSEIPDB_ACTION is deny, as dnsbl does for a zone. While
// SWW_WAF_ABUSEIPDB_KEY is unset it does nothing. A client without a score is
// checked in the background, by the request's address, if its history counts
// an offence, and the request does not wait for the answer. The score is then
// used for each address of the client. Its figures are the score and
// SWW_WAF_ABUSEIPDB_MIN_SCORE.
type abuseIPDB struct {
@@ -32,7 +30,6 @@ type abuseIPDB struct {
keySet bool // SWW_WAF_ABUSEIPDB_KEY is set
minScore int64 // SWW_WAF_ABUSEIPDB_MIN_SCORE
action config.ReputationAction // SWW_WAF_DNSBL_ABUSEIPDB_ACTION
banResponse int // SWW_WAF_BAN_RESPONSE
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
hitNoter reputationHitNoter
}
@@ -74,7 +71,6 @@ func (a abuseIPDB) check(
return &verdict{
action: requestlog.ActionClientDenied,
status: a.banResponse,
figures: map[string]any{
figureScore: score, "min_score": a.minScore,
},
+2 -2
View File
@@ -54,7 +54,7 @@ func TestClientsOnTheAbuseIPDBBlacklistAreBannedForTheBanDays(t *testing.T) {
})
// Two clients in one netblock make one ban, with the higher score.
server.banAbuseIPDBBlacklist([]reputation.BlacklistEntry{
server.BanAbuseIPDBBlacklist([]reputation.BlacklistEntry{
{Addr: netip.MustParseAddr(client), Score: 95},
{Addr: netip.MustParseAddr(otherClient), Score: 100},
{Addr: netip.MustParseAddr("2001:db8:5::1"), Score: 90},
@@ -88,7 +88,7 @@ func TestAbuseIPDBBlacklistMakesNoBanInObserveMode(t *testing.T) {
map[string]string{mode: "observe"},
)
server.banAbuseIPDBBlacklist([]reputation.BlacklistEntry{
server.BanAbuseIPDBBlacklist([]reputation.BlacklistEntry{
{Addr: netip.MustParseAddr(client), Score: 100},
})
-58
View File
@@ -1,58 +0,0 @@
package waf
// The heuristic active_ban. It reads SWW_WAF_MODE and SWW_WAF_BAN_RESPONSE; the
// bans are the ban ledger's.
import (
"context"
"time"
)
// activeBan refuses a client a ban covers, with SWW_WAF_BAN_RESPONSE: a ban on
// a netblock the client is in, in force at now. It notes for the log line when
// that ban ends. A request that makes the ban permanent, or in observe mode
// would have, raises the alert for it. Its figures are the ban's netblock and
// when it ends.
type activeBan struct {
banMaker banMaker
}
func (activeBan) name() string { return "active_ban" }
func (a activeBan) check(
_ context.Context,
rq *request,
now time.Time,
) *verdict {
ledger := a.banMaker.ledger
check := ledger.Check
if a.banMaker.observe {
check = ledger.Find // the ban refuses nothing, and stays as it is
}
ban, banned, madePermanent := check(rq.client, now)
var expires string
if banned {
expires = banExpires(ban)
rq.line.BanExpires = expires
}
if madePermanent {
ban.Expires = time.Time{} // the ban made permanent, which Find leaves as it is
a.banMaker.alertBan(rq, ban)
}
if !banned {
return nil
}
return &verdict{
action: banAction(ban.Permanent()),
status: a.banMaker.banResponse,
figures: map[string]any{
figureNetblock: ban.Netblock.String(), "ban_expires": expires,
},
}
}
+147 -92
View File
@@ -2,6 +2,7 @@ package waf
import (
"bytes"
"crypto/subtle"
"encoding/json"
"errors"
"fmt"
@@ -14,6 +15,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/state"
)
@@ -39,23 +41,80 @@ var (
errNotAddress = errors.New("is not an address, such as 203.0.113.9")
)
// handleHealth answers GET wafprotocol.HealthPath.
func handleHealth() http.HandlerFunc {
return func(w http.ResponseWriter, _ *http.Request) {
// Set here rather than left to Go's server, which would set it only
// after the log line has taken the response's headers.
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
_, _ = io.WriteString(w, "ok\n")
// answerHealth answers a request for the health endpoint.
func (rq *request) answerHealth() {
rq.line.Action = requestlog.ActionOwnEndpoint
// Set here rather than left to Go's server, which would set it only
// after the log line has taken the response's headers.
rq.out.Header().Set("Content-Type", "text/plain; charset=utf-8")
_, _ = io.WriteString(rq.out, "ok\n")
}
// answerAdmin answers a request for one of waf's own endpoints. Each
// needs a token, sent as Authorization: Bearer <token>: the metrics
// SWW_WAF_METRICS_TOKEN, the others SWW_WAF_ADMIN_TOKEN. A request without it is
// refused with 401. An endpoint whose token is unset answers 404, as any other
// path does.
func (rq *request) answerAdmin() {
rq.line.Action = requestlog.ActionOwnEndpoint
token, answer := rq.endpoint()
switch {
case token == "":
http.Error(
rq.out,
http.StatusText(http.StatusNotFound),
http.StatusNotFound,
)
case !hasToken(rq.in, token):
rq.out.Header().Set("WWW-Authenticate", "Bearer")
http.Error(
rq.out,
http.StatusText(http.StatusUnauthorized),
http.StatusUnauthorized,
)
default:
answer()
}
}
// handleListBans answers GET BansPath with every ban held.
func (h *handler) handleListBans() http.HandlerFunc {
return func(w http.ResponseWriter, _ *http.Request) {
h.answerBans(w, h.ledger.Snapshot())
// endpoint returns the token the request's endpoint needs, and what
// answers the request there; "" when there is no such endpoint.
func (rq *request) endpoint() (string, func()) {
cfg := rq.h.config
method, path := rq.in.Method, rq.in.URL.Path
switch {
case method == http.MethodGet && path == MetricsPath:
return cfg.MetricsToken, func() { rq.h.metrics.ServeHTTP(rq.out, rq.in) }
case method == http.MethodGet && path == BansPath:
return cfg.AdminToken, rq.listBans
case method == http.MethodPost && path == BansPath:
return cfg.AdminToken, rq.addBan
case method == http.MethodDelete && strings.HasPrefix(path, BansPath+"/"):
return cfg.AdminToken, rq.liftBans
case method == http.MethodGet && strings.HasPrefix(path, ClientsPath):
return cfg.AdminToken, rq.showClient
default:
return "", nil
}
}
// hasToken reports whether r carries token, as Authorization: Bearer
// <token>.
func hasToken(r *http.Request, token string) bool {
scheme, sent, _ := strings.Cut(r.Header.Get("Authorization"), " ")
return strings.EqualFold(scheme, "Bearer") &&
subtle.ConstantTimeCompare([]byte(sent), []byte(token)) == 1
}
// listBans answers GET BansPath with every ban held.
func (rq *request) listBans() {
rq.answerBans(rq.h.ledger.Snapshot())
}
// banToAdd is the body of POST BansPath.
type banToAdd struct {
// Netblock is a netblock, or a client's address, which stands for the
@@ -67,44 +126,44 @@ type banToAdd struct {
Reason string `json:"reason"`
}
// handleAddBan answers POST BansPath: it bans the netblock the body names,
// as an admin, from now for the duration the body gives, with its reason,
// addBan answers POST BansPath: it bans the netblock the body names, as
// an admin, from now for the duration the body gives, with its reason,
// and answers with that ban.
func (h *handler) handleAddBan() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
toAdd, err := readBanToAdd(r)
func (rq *request) addBan() {
toAdd, err := rq.readBanToAdd()
var (
netblock netip.Prefix
expires time.Time
now = h.now()
)
var (
netblock netip.Prefix
expires time.Time
now = rq.h.now()
)
if err == nil {
netblock, err = h.parseBanNetblock(toAdd.Netblock)
}
if err == nil {
expires, err = expiry(toAdd.Duration, now)
}
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
ban := h.ledger.BanForAdmin(netblock, now, expires, toAdd.Reason)
h.answerBans(w, []bans.Ban{ban})
if err == nil {
netblock, err = rq.h.parseBanNetblock(toAdd.Netblock)
}
if err == nil {
expires, err = expiry(toAdd.Duration, now)
}
if err != nil {
http.Error(rq.out, err.Error(), http.StatusBadRequest)
return
}
ban := rq.h.ledger.BanForAdmin(netblock, now, expires, toAdd.Reason)
rq.answerBans([]bans.Ban{ban})
}
// readBanToAdd reads the body of POST BansPath: a JSON object with
// nothing but whitespace after it, in at most banBodyMaxBytes.
func readBanToAdd(r *http.Request) (banToAdd, error) {
func (rq *request) readBanToAdd() (banToAdd, error) {
data, err := io.ReadAll(
http.MaxBytesReader(nil, r.Body, banBodyMaxBytes),
http.MaxBytesReader(nil, rq.in.Body, banBodyMaxBytes),
)
rq.requestBytes = int64(len(data))
if err != nil {
return banToAdd{}, fmt.Errorf("%w: %w", errNotBanToAdd, err)
}
@@ -191,31 +250,29 @@ func expiry(duration string, now time.Time) (time.Time, error) {
return now.Add(length), nil
}
// handleLiftBans answers DELETE BansPath/<client>: it lifts every ban
// active on a netblock the client's address is in, and answers with those
// bans, or with 404 when none is active.
func (h *handler) handleLiftBans() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
client, err := pathAddress(r.URL.Path, BansPath+"/")
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
// liftBans answers DELETE BansPath/<client>: it lifts every ban active on
// a netblock the client's address is in, and answers with those bans, or
// with 404 when none is active.
func (rq *request) liftBans() {
client, err := pathAddress(rq.in.URL.Path, BansPath+"/")
if err != nil {
http.Error(rq.out, err.Error(), http.StatusBadRequest)
return
}
lifted := h.ledger.Lift(client, h.now())
if len(lifted) == 0 {
http.Error(
w,
"no ban is active on "+client.String(),
http.StatusNotFound,
)
return
}
h.answerBans(w, lifted)
return
}
lifted := rq.h.ledger.Lift(client, rq.h.now())
if len(lifted) == 0 {
http.Error(
rq.out,
"no ban is active on "+client.String(),
http.StatusNotFound,
)
return
}
rq.answerBans(lifted)
}
// clientAnswer is the answer to GET ClientsPath<ip>: the client the
@@ -227,31 +284,29 @@ type clientAnswer struct {
Bans []state.BanEntry `json:"bans"`
}
// handleShowClient answers GET ClientsPath<ip> with what waf knows of the
// client: its counters, its history, which holds its country as last looked
// up and its offences, and its bans with their notes.
func (h *handler) handleShowClient() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
addr, err := pathAddress(r.URL.Path, ClientsPath)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
// showClient answers GET ClientsPath<ip> with what waf knows of the client:
// its counters, its history, which holds its country as last looked up and its
// offences, and its bans with their notes.
func (rq *request) showClient() {
addr, err := pathAddress(rq.in.URL.Path, ClientsPath)
if err != nil {
http.Error(rq.out, err.Error(), http.StatusBadRequest)
return
}
answer := clientAnswer{
Bans: state.BanEntries(h.ledger.Covering(addr)),
}
client, seen := h.limiter.Client(
clientGroup(h.config.IPv6GroupPrefix, addr),
)
if seen {
answer.Client = &client
}
h.answerJSON(w, answer)
return
}
answer := clientAnswer{
Bans: state.BanEntries(rq.h.ledger.Covering(addr)),
}
client, seen := rq.h.limiter.Client(
clientGroup(rq.h.config.IPv6GroupPrefix, addr),
)
if seen {
answer.Client = &client
}
rq.answerJSON(answer)
}
// pathAddress reads the client's address that follows prefix in path.
@@ -267,27 +322,27 @@ func pathAddress(path, prefix string) (netip.Addr, error) {
}
// answerBans answers with held under bans, as bans.json lists them.
func (h *handler) answerBans(w http.ResponseWriter, held []bans.Ban) {
h.answerJSON(w, struct {
func (rq *request) answerBans(held []bans.Ban) {
rq.answerJSON(struct {
Bans []state.BanEntry `json:"bans"`
}{state.BanEntries(held)})
}
// answerJSON answers with value as indented JSON.
func (h *handler) answerJSON(w http.ResponseWriter, value any) {
func (rq *request) answerJSON(value any) {
body, err := json.MarshalIndent(value, "", " ")
if err != nil {
h.processLog.Error(
rq.h.processLog.Error(
"encoding an answer failed",
"error",
err.Error(),
)
http.Error(w, http.StatusText(http.StatusInternalServerError),
http.Error(rq.out, http.StatusText(http.StatusInternalServerError),
http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write(append(body, '\n'))
rq.out.Header().Set("Content-Type", "application/json")
_, _ = rq.out.Write(append(body, '\n'))
}
-34
View File
@@ -14,7 +14,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/state"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
const (
@@ -334,39 +333,6 @@ func TestBanToAddThatCannotBeReadIsRefused(t *testing.T) {
}
}
func TestBanToAddIsLoggedWithTheBytesOfItsBody(t *testing.T) {
t.Parallel()
s, _, _ := startWithClock(
t,
"",
map[string]string{adminToken: adminSecret},
)
s.admin(http.MethodPost, waf.BansPath, banOtherClient, http.StatusOK)
line := s.out.endedLines(t, s.sent)[s.sent-1]
if line.RequestBytes != int64(len(banOtherClient)) {
t.Errorf(
"the ban to add was logged with request_bytes %d, want %d",
line.RequestBytes,
len(banOtherClient),
)
}
// The health endpoint reads none of a body sent to it.
s.ownEndpointRequest("", http.MethodGet, wafprotocol.HealthPath, "ok",
http.StatusOK)
line = s.out.endedLines(t, s.sent)[s.sent-1]
if line.RequestBytes != 0 {
t.Errorf(
"the health request was logged with request_bytes %d, want 0",
line.RequestBytes,
)
}
}
func TestClientEndpointShowsTheClientAndItsBans(t *testing.T) {
t.Parallel()
+6 -7
View File
@@ -11,6 +11,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
const (
@@ -249,7 +250,7 @@ func TestObserveModeWorksOutABanOnlyWhenItsAlertWouldBeSent(t *testing.T) {
// never sent them, and returns the queue they wait in as well.
func startWithAlerts(
t *testing.T, env map[string]string,
) (*sender, *clock, *wafAndParts, *alerts.Queue) {
) (*sender, *clock, *waf.Server, *alerts.Queue) {
t.Helper()
return startAppWithAlerts(
@@ -262,7 +263,7 @@ func startWithAlerts(
// startAppWithAlerts is startWithAlerts in front of the app handler.
func startAppWithAlerts(
t *testing.T, handler http.HandlerFunc, env map[string]string,
) (*sender, *clock, *wafAndParts, *alerts.Queue) {
) (*sender, *clock, *waf.Server, *alerts.Queue) {
t.Helper()
app := startApp(t, handler)
@@ -291,7 +292,7 @@ func startAppWithAlerts(
// the time raised, for ban, with its netblock, cause, reason and notes,
// which ends at expires, as the log line gives it.
func banAlert(
event alerts.Event,
event string,
raised time.Time,
client string,
ban bans.Ban,
@@ -306,9 +307,7 @@ func banAlert(
Netblock: ban.Netblock,
Reason: ban.Reason,
Detail: map[string]any{
"cause": string(
ban.Cause,
), "ban_expires": expires, "notes": ban.Notes,
"cause": ban.Cause, "ban_expires": expires, "notes": ban.Notes,
},
}
}
@@ -316,7 +315,7 @@ func banAlert(
// attackAlert is banAlert for a ban for the probe rule of testRules, with
// the netblock and the notes of ban.
func attackAlert(
event alerts.Event,
event string,
raised time.Time,
client string,
ban bans.Ban,
+7 -22
View File
@@ -15,6 +15,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/anomaly"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// The anomaly thresholds: the prefix of a scope followed by the end of a
@@ -32,22 +33,6 @@ const (
bytesPerHour = "BYTES_PER_HOUR"
)
// netAndASNThresholdsOff returns the anomaly thresholds per netblock and
// per AS number, the ones on by default, each set to off.
func netAndASNThresholdsOff() map[string]string {
settings := map[string]string{}
for _, scope := range []string{anomalyNet, anomalyASN} {
for _, end := range []string{
requestsPerMinute, requestsPerHour, bytesPerMinute, bytesPerHour,
} {
settings[scope+end] = off
}
}
return settings
}
// The other anomaly settings.
const (
anomalyNetV4Prefix = "SWW_WAF_ANOMALY_NET_V4_PREFIX"
@@ -101,10 +86,10 @@ func TestEachScopeAndWindowOverItsThresholdAlertsOncePerCooldown(
value int64
count float64
}{
{requestsPerMinute, string(ratelimit.KindRequests), minute, 2, 3},
{requestsPerHour, string(ratelimit.KindRequests), hour, 2, 3},
{bytesPerMinute, string(ratelimit.KindBytes), minute, 250, 300},
{bytesPerHour, string(ratelimit.KindBytes), hour, 250, 300},
{requestsPerMinute, ratelimit.KindRequests, minute, 2, 3},
{requestsPerHour, ratelimit.KindRequests, hour, 2, 3},
{bytesPerMinute, ratelimit.KindBytes, minute, 250, 300},
{bytesPerHour, ratelimit.KindBytes, hour, 250, 300},
} {
setting := scope.prefix + threshold.end
value := strconv.FormatInt(threshold.value, 10)
@@ -201,7 +186,7 @@ func TestEveryRequestIsCountedWhateverIsDoneWithIt(t *testing.T) {
for _, sent := range []struct {
from, path string
status int
action requestlog.Action
action string
}{
{allowed, "/", http.StatusOK, requestlog.ActionForward},
{exempt, "/", http.StatusOK, requestlog.ActionForward},
@@ -347,7 +332,7 @@ func TestRequestCountsForTheASNumberGeoJSGivesBeforeItEnds(t *testing.T) {
// kept.
geojsURL, _, release := startHeldGeoJS(t)
var server atomic.Pointer[wafAndParts]
var server atomic.Pointer[waf.Server]
app := startApp(t, func(http.ResponseWriter, *http.Request) {
release()
+17 -17
View File
@@ -135,7 +135,7 @@ func TestARequestLineOverTheLimitIsRefusedWith414BeforeAnyOtherCheck(
// cookies is how many cookies the request carries.
cookies int
status int
action requestlog.Action
action string
}{
{
"at the limit", lineLimit, 0, http.StatusOK,
@@ -439,7 +439,7 @@ func TestTheHeaderLimitsComeBeforeEveryCheckThatRefuses(t *testing.T) {
env map[string]string
path string
// prepare gets the server ready, at now, before the question.
prepare func(t *testing.T, server *wafAndParts, now time.Time)
prepare func(t *testing.T, server *waf.Server, now time.Time)
}{
{"SWW_WAF_DENY_NETS", map[string]string{denyNets: client}, "/", leaveAsItIs},
{"a ban", nil, "/", banClient},
@@ -454,21 +454,21 @@ func TestTheHeaderLimitsComeBeforeEveryCheckThatRefuses(t *testing.T) {
"a DNS blocklist", map[string]string{
dnsblZones: dnsblZone, dnsblResolver: noResolver, dnsblAbuseIPDBAction: actionDeny,
},
"/", func(_ *testing.T, server *wafAndParts, _ time.Time) {
"/", func(_ *testing.T, server *waf.Server, _ time.Time) {
loadVerdicts(server, map[string][]string{client: {dnsblZone}})
},
},
{
"AbuseIPDB",
map[string]string{abuseIPDBKey: accountKey, dnsblAbuseIPDBAction: actionDeny},
"/", func(_ *testing.T, server *wafAndParts, _ time.Time) {
"/", func(_ *testing.T, server *waf.Server, _ time.Time) {
// Over SWW_WAF_ABUSEIPDB_MIN_SCORE, 75 by default.
loadScores(server, map[string]int64{client: 80})
},
},
{
"SWW_WAF_TAG_DENY", map[string]string{tagLists: "vpn=" + vpnList, tagDeny: "vpn"},
"/", func(t *testing.T, server *wafAndParts, _ time.Time) {
"/", func(t *testing.T, server *waf.Server, _ time.Time) {
t.Helper()
loadLists(t, server, map[string][]string{vpnList: {client}})
},
@@ -685,7 +685,7 @@ func TestObserveModeAsksGeoJSNothingForARequestOverTheHeaderLimits(
// for env. No request to it may use the network.
func newWAFInBubble(
t *testing.T, env map[string]string,
) (*wafAndParts, *output) {
) (*waf.Server, *output) {
t.Helper()
settings := map[string]string{tagDeny: "country=KP"}
@@ -699,7 +699,7 @@ func newWAFInBubble(
// askDirectly hands a question about a request from client straight to server,
// as change leaves it, and returns the answer's status and how long it took.
func askDirectly(
t *testing.T, server *wafAndParts, change func(*http.Request),
t *testing.T, server *waf.Server, change func(*http.Request),
) (int, time.Duration) {
t.Helper()
@@ -717,13 +717,13 @@ func askDirectly(
recorder := httptest.NewRecorder()
began := time.Now()
server.ServeHTTP(recorder, req)
server.Handler.ServeHTTP(recorder, req)
return recorder.Code, time.Since(began)
}
// banClient bans client at now.
func banClient(_ *testing.T, server *wafAndParts, now time.Time) {
func banClient(_ *testing.T, server *waf.Server, now time.Time) {
server.Ledger.Ban(
netip.MustParsePrefix(client+"/32"),
now,
@@ -732,7 +732,7 @@ func banClient(_ *testing.T, server *wafAndParts, now time.Time) {
}
// countClient counts one request from client at now for the rate limits.
func countClient(_ *testing.T, server *wafAndParts, now time.Time) {
func countClient(_ *testing.T, server *waf.Server, now time.Time) {
server.Limiter.Count(
netip.MustParsePrefix(client+"/32"),
now,
@@ -741,7 +741,7 @@ func countClient(_ *testing.T, server *wafAndParts, now time.Time) {
}
// listClient puts client on the blocklist at dropURL.
func listClient(t *testing.T, server *wafAndParts, now time.Time) {
func listClient(t *testing.T, server *waf.Server, now time.Time) {
t.Helper()
err := server.Lists.Load([]reputation.List{{
@@ -753,7 +753,7 @@ func listClient(t *testing.T, server *wafAndParts, now time.Time) {
}
// leaveAsItIs prepares nothing.
func leaveAsItIs(*testing.T, *wafAndParts, time.Time) {}
func leaveAsItIs(*testing.T, *waf.Server, time.Time) {}
func TestARefusedRequestDoesNotWaitForASlowGeoJS(t *testing.T) {
t.Parallel()
@@ -764,11 +764,11 @@ func TestARefusedRequestDoesNotWaitForASlowGeoJS(t *testing.T) {
name string
env map[string]string
// prepare gets the server ready, at now, before the question.
prepare func(t *testing.T, server *wafAndParts, now time.Time)
prepare func(t *testing.T, server *waf.Server, now time.Time)
// cookie is the request's Cookie header.
cookie string
status int
action requestlog.Action
action string
}{
{
name: "SWW_WAF_DENY_NETS", env: map[string]string{denyNets: client},
@@ -849,9 +849,9 @@ func TestAQuestionWaitsForASlowGeoJSOnlyAsLongAsTheDefaultSays(
name string
env map[string]string
// prepare gets the server ready, at now, before the question.
prepare func(t *testing.T, server *wafAndParts, now time.Time)
prepare func(t *testing.T, server *waf.Server, now time.Time)
status int
action requestlog.Action
action string
}{
{"nothing refuses it", nil, leaveAsItIs, http.StatusOK, requestlog.ActionForward},
{
@@ -915,7 +915,7 @@ func TestWAFAnswersWithinTheTimeLimitWhileEveryOutsideSourceHangs(
held, _, _ := startHeldGeoJS(t)
resolver, releaseResolver := startHeldResolver(t)
server, out, _ := newWAFWith(t, func(*waf.Params) {},
server, out, _ := newWAFWith(t, waf.New,
silentGeoJSURL, silentAbuseIPDBURL, time.Now, map[string]string{
tagDeny: "country=KP", coreRuleSetMode: block,
dnsblZones: dnsblZone, dnsblResolver: resolver,
+67 -21
View File
@@ -1,9 +1,10 @@
package waf
// banMaker, which makes the bans of several heuristics. The bans are the ban
// ledger's.
// The heuristic active_ban, and banMaker, which makes the bans of several
// heuristics. active_ban reads SWW_WAF_MODE; the bans are the ban ledger's.
import (
"context"
"net/netip"
"time"
@@ -15,9 +16,59 @@ import (
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// banResponse is a refusal answered with SWW_WAF_BAN_RESPONSE, and logged
// with action.
func (rq *request) banResponse(action string) *refusal {
return &refusal{status: rq.h.config.BanResponse, action: action}
}
// activeBan refuses a client a ban covers: a ban on a netblock the client is
// in, in force at now. It notes for the log line when that ban ends. A request
// that makes the ban permanent, or in observe mode would have, raises the alert
// for it. Its figures are the ban's netblock and when it ends.
type activeBan struct {
banMaker banMaker
}
func (activeBan) name() string { return "active_ban" }
func (a activeBan) check(
_ context.Context,
rq *request,
now time.Time,
) *verdict {
ledger := a.banMaker.ledger
check := ledger.Check
if a.banMaker.observe {
check = ledger.Find // the ban refuses nothing, and stays as it is
}
ban, banned, madePermanent := check(rq.client, now)
if banned {
rq.line.BanExpires = banExpires(ban)
}
if madePermanent {
ban.Expires = time.Time{} // the ban made permanent, which Find leaves as it is
a.banMaker.alertBan(rq, ban)
}
if !banned {
return nil
}
return &verdict{
action: banAction(ban.Permanent()),
figures: map[string]any{
figureNetblock: ban.Netblock.String(), "ban_expires": rq.line.BanExpires,
},
}
}
// banAction is the action of a request refused under a ban or for making one:
// permanent says the ban never runs out.
func banAction(permanent bool) requestlog.Action {
func banAction(permanent bool) string {
if permanent {
return requestlog.ActionPermanentlyBanned
}
@@ -64,9 +115,8 @@ type banMaker struct {
alerts *alerts.Queue
// observe is true in observe mode (SWW_WAF_MODE).
observe bool
// banResponse is SWW_WAF_BAN_RESPONSE: the status of a request refused
// under a ban or for a clear sign of attack, which the notes of a ban for
// a clear sign of attack give.
// banResponse is SWW_WAF_BAN_RESPONSE, the status the notes of a ban for a
// clear sign of attack give.
banResponse int
// banScopeV4Prefix is SWW_WAF_BAN_SCOPE_V4_PREFIX and ipv6GroupPrefix
// SWW_WAF_IPV6_GROUP_PREFIX, which set the netblock a ban covers.
@@ -94,11 +144,10 @@ func (b banMaker) banForLimit(
rq.line.LimitHit = hit.Window + "_bytes" // as counts names the byte totals
case ratelimit.KindRefusals:
rq.line.LimitHit = requestlog.LimitHitMinuteRefusals
case ratelimit.KindRequests:
default:
rq.line.LimitHit = hit.Window
}
rq.brokeLimit = true
rq.line.Offence = requestlog.OffenceLimit
b.metrics.LimitHit(hit)
@@ -115,7 +164,6 @@ func (b banMaker) banForLimit(
notes.LimitPercent, notes.LimitPercentSetting = rq.rateLimitPercent.logged()
case ratelimit.KindBytes:
notes.LimitPercent, notes.LimitPercentSetting = rq.bytesLimitPercent.logged()
case ratelimit.KindRefusals: // the refusal limit is not lowered
}
b.banClient(
@@ -132,7 +180,7 @@ func (b banMaker) banForLimit(
// says what happens instead.
func (b banMaker) banForAttack(
rq *request, now time.Time, notes bans.Notes,
) requestlog.Action {
) string {
return b.banClient(
rq,
now,
@@ -153,7 +201,7 @@ func (b banMaker) banClient(
now time.Time,
status int,
grounds bans.Grounds,
) requestlog.Action {
) string {
netblock := banNetblock(b.banScopeV4Prefix, b.ipv6GroupPrefix, rq.client)
if b.observe {
@@ -211,9 +259,9 @@ func (b banMaker) withClientNotes(
status int,
netblock netip.Prefix,
) bans.Notes {
notes.ASN = rq.lookupAnswer.ASN
notes.ASName = rq.lookupAnswer.ASName
notes.Country = rq.lookupAnswer.Country
notes.ASN = rq.line.ASN
notes.ASName = rq.line.ASName
notes.Country = rq.line.Country
notes.Reputation = rq.reputation
notes.Request = rq.noted(now, status)
notes.Requests = b.netblockRequests(netblock)
@@ -227,7 +275,7 @@ func (b banMaker) withClientNotes(
// SWW_WAF_ALERT_COOLDOWN and never with no webhook set: its notes count the
// netblock's requests, which can mean going through every client.
func (b banMaker) wouldAlertBan(
netblock netip.Prefix, now time.Time, cause bans.Cause,
netblock netip.Prefix, now time.Time, cause string,
) bool {
event := alerts.EventBan
if b.ledger.WouldBePermanent(netblock, now, cause) {
@@ -248,9 +296,7 @@ func (b banMaker) alertBan(rq *request, ban bans.Ban) {
}
detail := map[string]any{
"cause": string(ban.Cause),
"ban_expires": banExpires(ban),
"notes": ban.Notes,
"cause": ban.Cause, "ban_expires": banExpires(ban), "notes": ban.Notes,
}
if b.observe {
detail["mode"] = "observe"
@@ -282,11 +328,11 @@ func (b banMaker) netblockRequests(netblock netip.Prefix) int64 {
func (rq *request) noted(now time.Time, status int) bans.Request {
return bans.Request{
Time: now,
Method: rq.method,
Host: rq.site,
Method: rq.line.Method,
Host: rq.line.Host,
Path: rq.requestURI,
Status: status,
UserAgent: rq.userAgent,
UserAgent: rq.line.UserAgent,
}
}
+6 -9
View File
@@ -16,6 +16,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
const (
@@ -465,7 +466,7 @@ func (c *clock) advance(d time.Duration) {
// window.
func startWithClock(
t *testing.T, geojsURL string, env map[string]string,
) (*sender, *clock, *wafAndParts) {
) (*sender, *clock, *waf.Server) {
t.Helper()
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
@@ -503,9 +504,7 @@ type sender struct {
}
// get sends a GET request for / from the client at from.
func (s *sender) get(
from string, status int, action requestlog.Action,
) logLine {
func (s *sender) get(from string, status int, action string) logLine {
s.t.Helper()
return s.request(from, "/", status, action)
@@ -518,7 +517,7 @@ func (s *sender) get(
func (s *sender) request(
from, path string,
status int,
action requestlog.Action,
action string,
) logLine {
s.t.Helper()
@@ -528,7 +527,7 @@ func (s *sender) request(
// requestWithHeader is request with header, such as "Authorization:
// Bearer x", added to the request unless it is "".
func (s *sender) requestWithHeader(
from, path, header string, status int, action requestlog.Action,
from, path, header string, status int, action string,
) logLine {
s.t.Helper()
@@ -554,9 +553,7 @@ func (s *sender) requestWithHeader(
// the Core Rule Set past any shorter wait. go test's -timeout ends a test
// whose answer never comes.
func (s *sender) requestWithBody(
method, from, path, header, body string,
status int,
action requestlog.Action,
method, from, path, header, body string, status int, action string,
) (logLine, answer) {
s.t.Helper()
-2
View File
@@ -4,7 +4,6 @@ package waf
import (
"context"
"net/http"
"sync/atomic"
"time"
@@ -162,7 +161,6 @@ func (rq *request) applyFindings(
func (rq *request) timedOut(running string) *verdict {
v := &verdict{
action: requestlog.ActionChecksTimedOut,
status: http.StatusServiceUnavailable,
figures: map[string]any{
"time_limit_ms": rq.h.besideLookupLimit.Milliseconds(),
},
+6 -4
View File
@@ -62,7 +62,7 @@ func coreRuleSetMatch() coreruleset.Result {
// at its default but for env. No request to it may use the network.
func newWAFHoldingTheCoreRuleSet(
t *testing.T, crs *heldCoreRuleSet, env map[string]string,
) (*wafAndParts, *output, *alerts.Queue) {
) (*waf.Server, *output, *alerts.Queue) {
t.Helper()
settings := map[string]string{
@@ -71,9 +71,11 @@ func newWAFHoldingTheCoreRuleSet(
}
maps.Copy(settings, env)
server, out, queue := newWAFWith(t, func(params *waf.Params) {
server, out, queue := newWAFWith(t, func(params waf.Params) *waf.Server {
params.CoreRuleSet = crs
params.BesideLookupTimeLimit = waf.BesideLookupTimeLimit
return waf.New(params)
}, silentGeoJSURL, abuseIPDBURL, time.Now, settings)
return server, out, queue
@@ -103,7 +105,7 @@ func wantAnswer(
// left no trace: line has none of its fields, and server has counted no match
// of it and raised no alert for one.
func wantNoCoreRuleSetTrace(
t *testing.T, server *wafAndParts, queue *alerts.Queue, line logLine,
t *testing.T, server *waf.Server, queue *alerts.Queue, line logLine,
) {
t.Helper()
@@ -196,7 +198,7 @@ type firstToRefuse struct {
env map[string]string
inspection time.Duration
took time.Duration
action requestlog.Action
action string
}
// firstToRefuseCases are the cases of
+2 -2
View File
@@ -68,7 +68,7 @@ type biasedThresholds struct {
func (b biasedThresholds) limitPercentages(
rq *request,
) (percentage, percentage) {
asn, country := rq.lookupAnswer.ASN, rq.lookupAnswer.Country
asn, country := rq.line.ASN, rq.line.Country
unknown := percentage{percent: whole}
if country == "" && !rq.skippedLookupWait {
@@ -115,7 +115,7 @@ func (b biasedThresholds) limitPercentages(
otherTags := percentage{percent: whole}
for _, tag := range rq.tags {
for _, tag := range rq.line.Tags {
if tag != asnTag(asn) && tag != countryTag(country) {
otherTags = lowest(
otherTags,
+7 -6
View File
@@ -18,6 +18,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
@@ -73,7 +74,7 @@ func TestEachBiasedThresholdLowersTheRateLimits(t *testing.T) {
// Half of 4 requests a minute: the third breaks the limit.
for _, sent := range []struct {
status int
action requestlog.Action
action string
}{
{http.StatusOK, requestlog.ActionForward},
{http.StatusOK, requestlog.ActionForward},
@@ -452,7 +453,7 @@ func TestBanForALoweredLimitGivesThePercentageInItsNotesAndItsAlert(
// limit, which is answered with status and logged with action.
before int
status int
action requestlog.Action
action string
// reason and want are the ban's reason, and its notes' limit
// percentage, as percentText gives it.
reason, want string
@@ -529,7 +530,7 @@ func TestBanForALoweredLimitGivesThePercentageInItsNotesAndItsAlert(
// clock.
func startWithLookups(
t *testing.T, env map[string]string,
) (*sender, *wafAndParts, *alerts.Queue) {
) (*sender, *waf.Server, *alerts.Queue) {
t.Helper()
s, _, server, queue := startWithLookupsAndClock(t, env)
@@ -544,7 +545,7 @@ func startWithLookups(
// country, and no other address.
func startWithLookupsAndClock(
t *testing.T, env map[string]string,
) (*sender, *clock, *wafAndParts, *alerts.Queue) {
) (*sender, *clock, *waf.Server, *alerts.Queue) {
t.Helper()
path := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb")
@@ -584,7 +585,7 @@ func (s *sender) uploadFrom(from string) logLine {
// straight to server's handler, without the network, and returns once it is
// answered.
func serveFromDE(
t *testing.T, server *wafAndParts, method string, body io.Reader,
t *testing.T, server *waf.Server, method string, body io.Reader,
) {
t.Helper()
@@ -592,7 +593,7 @@ func serveFromDE(
req.Header.Set(wafprotocol.ClientHeader, fromDE)
req.Header.Set(requestIDHeader, "the-id")
server.ServeHTTP(httptest.NewRecorder(), req)
server.Handler.ServeHTTP(httptest.NewRecorder(), req)
}
// wantPercent checks a limit percentage that a log line or a ban's notes
+8 -20
View File
@@ -1,7 +1,7 @@
package waf
// The heuristic blocklists. It reads SWW_WAF_BLOCKLIST_ACTION and
// SWW_WAF_BAN_RESPONSE, and the blocklists through the reputation lists.
// The heuristic blocklists. It reads SWW_WAF_BLOCKLIST_ACTION, and the blocklists
// through the reputation lists.
import (
"context"
@@ -13,15 +13,13 @@ import (
)
// blocklists notes the blocklists that list the client, as noteListed does,
// and refuses the request with SWW_WAF_BAN_RESPONSE when
// SWW_WAF_BLOCKLIST_ACTION is deny. Being limit, it lowers the client's limits
// instead (see limitPercentages), and being log, it does nothing more. Its
// figure is the blocklists that list the client.
// and refuses the request when SWW_WAF_BLOCKLIST_ACTION is deny. Being limit, it
// lowers the client's limits instead (see limitPercentages), and being log, it
// does nothing more. Its figure is the blocklists that list the client.
type blocklists struct {
lists *reputation.Lists
action config.ReputationAction // SWW_WAF_BLOCKLIST_ACTION
banResponse int // SWW_WAF_BAN_RESPONSE
hitNoter reputationHitNoter
lists *reputation.Lists
action config.ReputationAction // SWW_WAF_BLOCKLIST_ACTION
hitNoter reputationHitNoter
}
func (blocklists) name() string { return "blocklists" }
@@ -41,20 +39,10 @@ func (b blocklists) check(
return &verdict{
action: requestlog.ActionClientDenied,
status: b.banResponse,
figures: map[string]any{"lists": listedBy},
}
}
// refusesWhateverTheLookupAnswers reports whether blocklists refuses rq, as
// refusedByBlocklists says. The lookup's answer plays no part in it.
func (b blocklists) refusesWhateverTheLookupAnswers(
rq *request,
_ time.Time,
) bool {
return b.refusedByBlocklists(b.lists.ListedBy(rq.client))
}
// refusedByBlocklists reports whether the heuristic blocklists refuses a
// client, given listedBy, the blocklists that list it: it does when one lists
// it and SWW_WAF_BLOCKLIST_ACTION is deny.
+1 -1
View File
@@ -33,7 +33,7 @@ func (b byteLimits) check(
rq *request,
now time.Time,
) *verdict {
if !rq.counted || rq.wouldAction != "" {
if !rq.counted || rq.line.WouldAction != "" {
return nil
}
+3 -5
View File
@@ -5,7 +5,6 @@ package waf
import (
"context"
"net/http"
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
@@ -78,7 +77,6 @@ func (c coreRuleSet) check(
return &verdict{
action: requestlog.ActionBlockedByCoreRuleSet,
status: http.StatusForbidden,
figures: map[string]any{
figureScore: result.Score,
"threshold": threshold,
@@ -123,9 +121,9 @@ func (c coreRuleSet) alertCoreRuleSetMatch(
Host: rq.site,
Client: rq.client,
Netblock: clientGroup(c.ipv6GroupPrefix, rq.client),
ASN: rq.lookupAnswer.ASN,
ASName: rq.lookupAnswer.ASName,
Country: rq.lookupAnswer.Country,
ASN: rq.line.ASN,
ASName: rq.line.ASName,
Country: rq.line.Country,
Reason: "scored by the Core Rule Set at or over " +
"SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD",
Detail: detail,
+5 -7
View File
@@ -84,7 +84,7 @@ func TestCoreRuleSetMatchAddsTheExploitSignatureTag(t *testing.T) {
name string
mode string
status int
action requestlog.Action
action string
}{
{block, block, http.StatusForbidden, requestlog.ActionBlockedByCoreRuleSet},
{detect, detect, http.StatusOK, requestlog.ActionForward},
@@ -158,9 +158,8 @@ func TestCoreRuleSetRefusesAttacksInBlockModeAndOnlyLogsThemInDetectMode(
t.Parallel()
for _, tc := range []struct {
mode string
action requestlog.Action
status int
mode, action string
status int
}{
{block, requestlog.ActionBlockedByCoreRuleSet, http.StatusForbidden},
{detect, requestlog.ActionForward, http.StatusOK},
@@ -653,9 +652,8 @@ func TestCoreRuleSetMatchRaisesTheCoreRuleSetMatchAlert(t *testing.T) {
// status and action are what the request is answered and logged
// with, and alertMode what the alert's detail gives as mode, if
// anything.
status int
action requestlog.Action
alertMode string
status int
action, alertMode string
}{
{
"block", map[string]string{coreRuleSetMode: block},
+5 -9
View File
@@ -1,7 +1,6 @@
package waf
// The heuristic countries. It reads SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES and
// SWW_WAF_BAN_RESPONSE.
// The heuristic countries. It reads SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES.
import (
"context"
@@ -14,12 +13,10 @@ import (
// countries refuses a request when SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES is set and
// does not hold the client's country as it was looked up. A client without a
// country, or whose country cannot be found, is refused. It refuses with
// SWW_WAF_BAN_RESPONSE. Its figures are the country and the setting that
// refused it.
// country, or whose country cannot be found, is refused. Its figures are the
// country and the setting that refused it.
type countries struct {
allowed []string // SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES
banResponse int // SWW_WAF_BAN_RESPONSE
allowed []string // SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES
}
func (countries) name() string { return "countries" }
@@ -30,7 +27,7 @@ func (c countries) check(
_ time.Time,
) *verdict {
allowed := c.allowed
country := rq.lookupAnswer.Country
country := rq.line.Country
if len(allowed) == 0 || slices.Contains(allowed, country) {
return nil
@@ -38,7 +35,6 @@ func (c countries) check(
return &verdict{
action: requestlog.ActionCountryDenied,
status: c.banResponse,
figures: map[string]any{
"country": country,
"setting": config.ExclusivelyAllowedCountriesSetting,
+1 -1
View File
@@ -316,7 +316,7 @@ func TestExclusiveListRefusesAPrivateAddressUnlessAllowed(t *testing.T) {
name string
allowNets string
status int
action requestlog.Action
action string
}{
{
"not in SWW_WAF_ALLOW_NETS", "",
+4 -6
View File
@@ -1,6 +1,6 @@
package waf
// The heuristic deny_nets. It reads SWW_WAF_DENY_NETS and SWW_WAF_BAN_RESPONSE.
// The heuristic deny_nets. It reads SWW_WAF_DENY_NETS.
import (
"context"
@@ -10,11 +10,10 @@ import (
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// denyNets refuses a client in SWW_WAF_DENY_NETS, with SWW_WAF_BAN_RESPONSE.
// Its figure is the netblock of the setting the client is in.
// denyNets refuses a client in SWW_WAF_DENY_NETS. Its figure is the netblock of
// the setting the client is in.
type denyNets struct {
netblocks []netip.Prefix // SWW_WAF_DENY_NETS
banResponse int // SWW_WAF_BAN_RESPONSE
netblocks []netip.Prefix // SWW_WAF_DENY_NETS
}
func (denyNets) name() string { return "deny_nets" }
@@ -28,7 +27,6 @@ func (d denyNets) check(
if netblock.Contains(rq.client) {
return &verdict{
action: requestlog.ActionClientDenied,
status: d.banResponse,
figures: map[string]any{figureNetblock: netblock.String()},
}
}
+10 -14
View File
@@ -1,8 +1,7 @@
package waf
// The heuristic dnsbl. It reads SWW_WAF_DNSBL_ABUSEIPDB_ACTION and
// SWW_WAF_BAN_RESPONSE, and the DNSBL zones' verdicts through the DNSBL
// checker.
// The heuristic dnsbl. It reads SWW_WAF_DNSBL_ABUSEIPDB_ACTION, and the DNSBL zones'
// verdicts through the DNSBL checker.
import (
"context"
@@ -16,17 +15,15 @@ import (
// dnsbl notes the DNSBL zones whose verdict lists the client, as noteListed
// does, adds the tag dnsbl=<zone> for each, the zone's key masked, and refuses
// the request with SWW_WAF_BAN_RESPONSE when SWW_WAF_DNSBL_ABUSEIPDB_ACTION is
// deny. Being limit, it lowers the client's limits instead (see
// limitPercentages), and being log, it does nothing more. A zone without a
// verdict on the client is asked about it in the background, and the request
// does not wait for the answer. Its figure is the zones that list the client,
// their keys masked.
// the request when SWW_WAF_DNSBL_ABUSEIPDB_ACTION is deny. Being
// limit, it lowers the client's limits instead (see limitPercentages), and
// being log, it does nothing more. A zone without a verdict on the client is
// asked about it in the background, and the request does not wait for the
// answer. Its figure is the zones that list the client, their keys masked.
type dnsbl struct {
dnsbl *reputation.DNSBL
action config.ReputationAction // SWW_WAF_DNSBL_ABUSEIPDB_ACTION
banResponse int // SWW_WAF_BAN_RESPONSE
hitNoter reputationHitNoter
dnsbl *reputation.DNSBL
action config.ReputationAction // SWW_WAF_DNSBL_ABUSEIPDB_ACTION
hitNoter reputationHitNoter
}
func (dnsbl) name() string { return "dnsbl" }
@@ -50,7 +47,6 @@ func (d dnsbl) check(
return &verdict{
action: requestlog.ActionClientDenied,
status: d.banResponse,
figures: map[string]any{"zones": listedBy},
}
}
+6 -4
View File
@@ -1,11 +1,13 @@
package waf
import "net/http"
import "sneak.berlin/go/smallwebwaf/internal/reputation"
// BesideLookupTimeLimit is besideLookupTimeLimit.
const BesideLookupTimeLimit = besideLookupTimeLimit
// HTTPServer is Go's server s holds, for tests that read its settings.
func (s *Server) HTTPServer() *http.Server {
return s.httpServer
// BanAbuseIPDBBlacklist does on s what a fetch of AbuseIPDB's blacklist with
// the clients on listed does.
func (s *Server) BanAbuseIPDBBlacklist(listed []reputation.BlacklistEntry) {
h := s.Handler.(*handler) //nolint:forcetypeassert // New makes it one
BanAbuseIPDBBlacklist(h.config, h.ledger, h.processLog, h.now(), listed)
}
-3
View File
@@ -4,7 +4,6 @@ package waf
import (
"context"
"net/http"
"strings"
"time"
@@ -40,7 +39,6 @@ func (headerLimits) check(
if requestLine > maxLineBytes {
return &verdict{
action: requestlog.ActionRequestLineTooLong,
status: http.StatusRequestURITooLong,
figures: map[string]any{
"request_line_bytes": requestLine,
"request_line_bytes_limit": maxLineBytes,
@@ -58,7 +56,6 @@ func (headerLimits) check(
return &verdict{
action: requestlog.ActionRequestHeaderTooLarge,
status: http.StatusRequestHeaderFieldsTooLarge,
figures: map[string]any{
"longest_header_line_bytes": longest,
"header_line_bytes_limit": maxLineBytes,
-11
View File
@@ -58,14 +58,3 @@ func TestHealthEndpointCountsTowardNoLimit(t *testing.T) {
}
}
}
func TestHealthEndpointIsFoundAtItsPathPercentEncoded(t *testing.T) {
t.Parallel()
// %68 is h. An endpoint is found by the decoded path.
s, _, _ := startWithClock(t, "", nil)
got := s.ownEndpointRequest("", http.MethodGet, "/_sww/%68ealthz", "",
http.StatusOK)
wantBody(t, got, "ok\n")
}
+3 -27
View File
@@ -3,24 +3,12 @@ package waf
import (
"context"
"time"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// To add a heuristic, write one file with a struct that implements heuristic
// and holds the settings and parts it reads. Add one line for it to one of the
// lists that newHandler builds, which hands it those settings and parts. Its
// settings go in config.WAF, and the comment at the top of its file names them.
// A heuristic of afterLookup that can refuse a request whatever the lookup
// answers also implements refusesWithoutLookup.
//
// Three places outside its file may change with it:
// - waitForGeoJS in newHandler lists the settings that need the lookup's
// answer;
// - requestCanGetTag in internal/config/waf.go holds which setting can give a
// request which tag;
// - findBesideLookup does the work of each heuristic of besideLookup, by
// name.
// heuristic is one of the checks that decide what waf does with a
// request.
@@ -32,23 +20,11 @@ type heuristic interface {
check(ctx context.Context, rq *request, now time.Time) *verdict
}
// refusesWithoutLookup is a heuristic of afterLookup that can tell, before the
// lookup answers, that it will refuse a request whatever the answer.
type refusesWithoutLookup interface {
// refusesWhateverTheLookupAnswers reports whether the heuristic will refuse
// rq at now, whatever the lookup answers about its client. It changes
// nothing.
refusesWhateverTheLookupAnswers(rq *request, now time.Time) bool
}
// verdict is a heuristic refusing a request, or banning its client: the action
// the log line names, the status the visitor is answered with, and the figures
// the heuristic acted on, by name, such as the count and the limit. A status of
// 0 closes the visitor's connection without an answer. A verdict of afterAnswer
// or onReport refuses nothing, and has neither action nor status.
// the log line names, one of requestlog's Action constants, and the figures the
// heuristic acted on, by name, such as the count and the limit.
type verdict struct {
action requestlog.Action
status int
action string
figures map[string]any
}
+1 -1
View File
@@ -151,7 +151,7 @@ func TestPathUnderTheOwnPrefixIsNeitherForwardedNorRefused(t *testing.T) {
// historyOf returns the history of the client at addr.
func historyOf(
t *testing.T, server *wafAndParts, addr string,
t *testing.T, server *waf.Server, addr string,
) ratelimit.History {
t.Helper()
+2 -22
View File
@@ -80,26 +80,6 @@ func (rq *request) startLookUp(
return result
}
// refusedWhateverTheAnswer reports whether the request is one that does not wait
// for GeoJS, since the checks will refuse it at now whatever GeoJS answers about
// its client: a heuristic of afterLookup that implements refusesWithoutLookup
// says it will refuse it. In observe mode, which refuses nothing, none is.
// rproxy lets a request through when waf answers too late.
func (rq *request) refusedWhateverTheAnswer(now time.Time) bool {
if rq.h.config.Observe {
return false
}
for _, each := range rq.h.afterLookup {
knows, ok := each.(refusesWithoutLookup)
if ok && knows.refusesWhateverTheLookupAnswers(rq, now) {
return true
}
}
return false
}
// noteLookup notes result for the log line and as the request's asn= and
// country= tags.
func (rq *request) noteLookup(result lookupResult) {
@@ -114,11 +94,11 @@ func (rq *request) noteLookup(result lookupResult) {
rq.line.ASName = rq.lookupAnswer.ASName
rq.line.Country = rq.lookupAnswer.Country
if tag := asnTag(rq.lookupAnswer.ASN); tag != "" {
if tag := asnTag(rq.line.ASN); tag != "" {
rq.addTag(tag)
}
if tag := countryTag(rq.lookupAnswer.Country); tag != "" {
if tag := countryTag(rq.line.Country); tag != "" {
rq.addTag(tag)
}
}
+5 -8
View File
@@ -22,12 +22,12 @@ func TestBanActionSaysWhetherTheBanRunsOut(t *testing.T) {
for _, tc := range []struct {
name string
cause bans.Cause
cause string
// endsIn is how long after the request the ban ends: negative for a
// ban that has ended, and zero for a permanent ban.
endsIn time.Duration
path string
want requestlog.Action
want string
}{
{
"temporary ban", bans.CauseAdmin, time.Hour, "/",
@@ -170,10 +170,7 @@ func TestObserveModeForwardsWhatEnforceModeRefuses(t *testing.T) {
// and its second breaks it.
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
for _, sent := range []struct {
from string
refusal requestlog.Action
}{
for _, sent := range []struct{ from, refusal string }{
{denied, requestlog.ActionClientDenied},
{banned, requestlog.ActionTemporarilyBanned},
{fromKP, requestlog.ActionClientDenied},
@@ -256,7 +253,7 @@ func TestObserveModeMakesNoBanAndKeepsTheBansItHas(t *testing.T) {
// wantWouldAction checks the request log line's would_action, and that a
// line that should have none has no such field.
func wantWouldAction(t *testing.T, line logLine, want requestlog.Action) {
func wantWouldAction(t *testing.T, line logLine, want string) {
t.Helper()
got, present := line.fields["would_action"]
@@ -264,7 +261,7 @@ func wantWouldAction(t *testing.T, line logLine, want requestlog.Action) {
switch {
case want == "" && present:
t.Errorf("log line has would_action %v, want none", got)
case want != "" && got != string(want):
case want != "" && got != want:
t.Errorf("log line has would_action %v, want %s", got, want)
}
}
+334
View File
@@ -0,0 +1,334 @@
// Package waf is the server the waf command runs. It answers
// rproxy's questions, takes rproxy's reports and answers its own endpoints, and
// writes a log line for each. It also bans the clients on AbuseIPDB's
// blacklist each time the reputation package fetches it.
package waf
import (
"cmp"
"io"
"log/slog"
"net/http"
"net/netip"
"time"
lru "github.com/hashicorp/golang-lru/v2"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/anomaly"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
"sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/routewatch"
"sneak.berlin/go/smallwebwaf/internal/rules"
)
// MetricsPath is where the metrics are, for a request that carries
// SWW_WAF_METRICS_TOKEN.
const MetricsPath = "/_sww/metrics"
// BansPath is where an admin lists and adds bans, and, followed by / and
// a client's address, lifts them, with SWW_WAF_ADMIN_TOKEN.
const BansPath = "/_sww/bans"
// ClientsPath is where an admin asks what waf knows of a client, by
// the client's address after it, with SWW_WAF_ADMIN_TOKEN.
const ClientsPath = "/_sww/clients/"
// Params are what New needs.
type Params struct {
Config *config.WAF
// RequestLog receives two JSON lines per question, one at its decision and
// one when its report comes, and one per request to waf's own
// endpoints.
RequestLog io.Writer
// ProcessLog receives the process's own messages.
ProcessLog *slog.Logger
// Metrics count what waf does. Whoever builds them adds the metrics
// read from the other parts.
Metrics *metrics.Metrics
// Limiter is the table of clients the rate limits count, with each
// client's history.
Limiter *ratelimit.Limiter
// Ledger holds the bans.
Ledger *bans.Ledger
// Anomalies are the counters of the anomaly thresholds.
Anomalies *anomaly.Counters
// Lists are the lists fetched from URLs, DNSBL the DNSBL zones'
// verdicts, and AbuseIPDB AbuseIPDB's scores and the checks spent.
Lists *reputation.Lists
DNSBL *reputation.DNSBL
AbuseIPDB *reputation.AbuseIPDB
// RouteWatch asks routewatch about clients, and is nil while
// SWW_WAF_ROUTEWATCH_URL is unset.
RouteWatch *routewatch.Client
// GeoJS looks clients' AS numbers and countries up while
// SWW_WAF_LOOKUP_SOURCE is geojs. It is given AddLookup as its Answered
// and GeoJSFailed as its Failed.
GeoJS *geojs.GeoJS
// LookupFile is the lookup database they are looked up in while
// SWW_WAF_LOOKUP_SOURCE is file, and nil otherwise.
LookupFile *lookup.File
// CoreRuleSet is what inspects requests while
// SWW_WAF_CORE_RULE_SET_MODE is not off, and nil while it is.
CoreRuleSet CoreRuleSetInspector
// Now tells the time by which requests are counted for the rate
// limits, bans are made and run out, and GeoJS's answers are kept,
// normally time.Now in UTC, the time the state files give.
Now func() time.Time
// Rules are the rule files' rules, which each request is checked
// against.
Rules *rules.Files
// Alerts receive the alert for each ban waf makes or makes
// permanent, for each count over an anomaly threshold, for each request
// whose client a blocklist, a DNSBL zone or AbuseIPDB lists, for each
// request the Core Rule Set scores at or over
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD, and for GeoJS failing, a fetch of a list
// failing, a query to a DNSBL zone or a check with AbuseIPDB failing,
// or the day's AbuseIPDB checks used up.
Alerts *alerts.Queue
// BesideLookupTimeLimit is how long the heuristics of besideLookup have
// for a request. Zero means besideLookupTimeLimit. Tests that run the
// Core Rule Set lengthen it, since the race detector and a busy host can
// slow an inspection past besideLookupTimeLimit.
BesideLookupTimeLimit time.Duration
}
// CoreRuleSetInspector inspects a request with the Core Rule Set, as
// coreruleset.CoreRuleSet's Inspect says. Tests give waf a stand-in for it.
type CoreRuleSetInspector interface {
Inspect(
r *http.Request, client netip.Addr, body io.Reader,
) (coreruleset.Result, []byte, error)
}
// Server is the server waf runs, with the parts of its Params. The tests
// read them.
type Server struct {
*http.Server
Ledger *bans.Ledger
Limiter *ratelimit.Limiter
GeoJS *geojs.GeoJS
Anomalies *anomaly.Counters
LookupFile *lookup.File
Lists *reputation.Lists
DNSBL *reputation.DNSBL
AbuseIPDB *reputation.AbuseIPDB
RouteWatch *routewatch.Client
Metrics *metrics.Metrics
}
// newHandler returns the handler with what every request shares, the parts
// in params among it. It builds the heuristics, each with the settings and
// parts it reads, and the lists they run in.
//
//nolint:funlen // a few lines for each heuristic, a list that grows with them
func newHandler(params Params) *handler {
awaitingReport, err := lru.New[string, *decision](maxAwaitingReport)
if err != nil {
panic(err) // New fails only for a size below one
}
cfg := params.Config
h := &handler{
config: cfg,
requestLog: params.RequestLog,
processLog: params.ProcessLog,
now: params.Now,
metrics: params.Metrics,
awaitingReport: awaitingReport,
limiter: params.Limiter,
ledger: params.Ledger,
geojs: params.GeoJS,
anomalies: params.Anomalies,
lookupFile: params.LookupFile,
// The country list, the headers, the tag settings that name an asn=
// or country= tag and the biased thresholds act on the answer before
// the request goes on.
waitForGeoJS: len(cfg.ExclusivelyAllowedCountries) > 0 ||
cfg.AddLookupHeaders ||
cfg.TagsNeedLookup() ||
biasedThresholdsSet(cfg),
besideLookupLimit: cmp.Or(
params.BesideLookupTimeLimit,
besideLookupTimeLimit,
),
}
sharedBanMaker := banMaker{
ledger: params.Ledger,
limiter: params.Limiter,
metrics: params.Metrics,
alerts: params.Alerts,
observe: cfg.Observe,
banResponse: cfg.BanResponse,
banScopeV4Prefix: cfg.BanScopeV4Prefix,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
}
hitNoter := reputationHitNoter{
metrics: params.Metrics,
alerts: params.Alerts,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
}
h.blocklists = blocklists{
lists: params.Lists,
action: cfg.BlocklistAction,
hitNoter: hitNoter,
}
h.rateLimits = rateLimits{
limiter: params.Limiter,
exemptNets: cfg.RateLimitExemptNets,
exemptPaths: cfg.RateLimitExemptPaths,
banResponse: cfg.BanResponse,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
biasedThresholds: biasedThresholds{
lists: params.Lists,
tagLimitPercent: cfg.TagLimitPercent,
asnBytesLimitPercent: cfg.ASNBytesLimitPercent,
countryBytesLimitPercent: cfg.CountryBytesLimitPercent,
unknownCountryLimitPercent: cfg.UnknownCountryLimitPercent,
blocklistAction: cfg.BlocklistAction,
blocklistLimitPercent: cfg.BlocklistLimitPercent,
dnsblAbuseIPDBAction: cfg.DNSBLAbuseIPDBAction,
dnsblAbuseIPDBLimitPercent: cfg.DNSBLAbuseIPDBLimitPercent,
},
banMaker: sharedBanMaker,
}
h.trapPaths = trapPaths{paths: cfg.TrapPaths, banMaker: sharedBanMaker}
h.ruleFiles = ruleFiles{
rules: params.Rules,
metrics: params.Metrics,
banMaker: sharedBanMaker,
}
h.coreRuleSet = coreRuleSet{
inspector: params.CoreRuleSet,
metrics: params.Metrics,
alerts: params.Alerts,
mode: cfg.CoreRuleSetMode,
exemptPaths: cfg.CoreRuleSetExemptPaths,
scoreThreshold: cfg.CoreRuleSetScoreThreshold,
observe: cfg.Observe,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
}
h.beforeLookup = []heuristic{
headerLimits{},
tagLists{lists: params.Lists},
routeWatchTags{
routeWatch: params.RouteWatch,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
},
denyNets{netblocks: cfg.DenyNets},
activeBan{banMaker: sharedBanMaker},
}
h.afterLookup = []heuristic{
countries{allowed: cfg.ExclusivelyAllowedCountries},
h.blocklists,
dnsbl{
dnsbl: params.DNSBL,
action: cfg.DNSBLAbuseIPDBAction,
hitNoter: hitNoter,
},
abuseIPDB{
abuseIPDB: params.AbuseIPDB,
limiter: params.Limiter,
keySet: cfg.AbuseIPDBKey != "",
minScore: cfg.AbuseIPDBMinScore,
action: cfg.DNSBLAbuseIPDBAction,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
hitNoter: hitNoter,
},
tagDeny{tags: cfg.TagDeny},
h.rateLimits,
}
h.besideLookup = []heuristic{h.trapPaths, h.ruleFiles, h.coreRuleSet}
h.afterBesideLookup = []heuristic{tagDeny{tags: cfg.TagDeny}}
h.afterAnswer = []heuristic{
refusalLimit{
limiter: params.Limiter,
perMinute: cfg.RefusalsLimitPerMinute,
banResponse: cfg.BanResponse,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
banMaker: sharedBanMaker,
},
}
h.onReport = []heuristic{
byteLimits{
limiter: params.Limiter,
bytesCounted: cfg.BytesCounted,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
banMaker: sharedBanMaker,
},
}
return h
}
// handler is waf's handler. It holds what every request shares; what
// belongs to one request is in a request.
type handler struct {
config *config.WAF
requestLog io.Writer
processLog *slog.Logger
now func() time.Time
metrics *metrics.Metrics
limiter *ratelimit.Limiter
ledger *bans.Ledger
geojs *geojs.GeoJS
anomalies *anomaly.Counters
lookupFile *lookup.File
// waitForGeoJS is true when a setting needs each request's answer from
// GeoJS before the request goes on.
waitForGeoJS bool
// besideLookupLimit is how long the heuristics of besideLookup have for a
// request, as Params.BesideLookupTimeLimit says.
besideLookupLimit time.Duration
// awaitingReport holds the decision on each answered question by its id
// until its report comes, and nil for a question still being handled.
awaitingReport *lru.Cache[string, *decision]
// The lists of heuristics, each in the order its heuristics run. A client
// in SWW_WAF_ALLOW_NETS skips them all.
// beforeLookup are the heuristics that run before the client's AS number
// and country are looked up, so that a client they refuse is not looked
// up. header_limits comes first: it only measures and counts, and a
// request it refuses causes no other work.
beforeLookup []heuristic
// afterLookup are the heuristics that run once the lookup has been made.
afterLookup []heuristic
// besideLookup are the heuristics that only read the request. Their work
// runs beside the lookup and afterLookup, in findBesideLookup, which
// changes nothing of the request. Their check then applies what that work
// found, and runs only once waf takes it.
besideLookup []heuristic
// afterBesideLookup are the heuristics that run once afterLookup and
// besideLookup have both let the request through. tag_deny runs a second
// time, on the tag rule_files and core_rule_set can add.
afterBesideLookup []heuristic
// afterAnswer are the heuristics that run on a request once waf has
// answered it, before its decision line is written. They ban the client,
// and do not change what the request got: their verdict's action is not
// used.
afterAnswer []heuristic
// onReport are the heuristics that run on an allowed request when
// rproxy's report of it comes, before its report line is written. Like
// afterAnswer, they only ban.
onReport []heuristic
// The heuristics of the lists that other steps of a request use too:
// refusedWhateverTheAnswer asks blocklists and rateLimits, and
// findBesideLookup does the work of trapPaths, ruleFiles and coreRuleSet.
blocklists blocklists
rateLimits rateLimits
trapPaths trapPaths
ruleFiles ruleFiles
coreRuleSet coreRuleSet
}
+36 -75
View File
@@ -316,7 +316,7 @@ func startProxyWithGeoJS(
func startProxyWithClock(
t *testing.T, appURL, geojsURL string, now func() time.Time,
env map[string]string,
) (string, *output, *wafAndParts) {
) (string, *output, *waf.Server) {
t.Helper()
addr, out, server, _ := startProxyWithAlerts(
@@ -335,7 +335,7 @@ func startProxyWithClock(
func startProxyWithAlerts(
t *testing.T, appURL, geojsURL string, now func() time.Time,
env map[string]string,
) (string, *output, *wafAndParts, *alerts.Queue) {
) (string, *output, *waf.Server, *alerts.Queue) {
t.Helper()
server, out, alertQueue := newWAF(t, geojsURL, now, env)
@@ -344,9 +344,9 @@ func startProxyWithAlerts(
return startRProxy(t, server.Addr, appURL, env), out, server, alertQueue
}
// serve serves server on a port of its own until the test ends, and sets
// server.Addr to where it listens.
func serve(t *testing.T, server *wafAndParts) {
// serve serves server on a port of its own until the test ends, and sets its
// Addr to where it listens.
func serve(t *testing.T, server *waf.Server) {
t.Helper()
listener, err := (&net.ListenConfig{}).Listen(
@@ -422,32 +422,8 @@ func startRProxy(
return listener.Addr().String()
}
// wafAndParts is waf's server with the Params it was built from, so that a
// test can read the parts the server uses. Addr is where it listens once serve
// has it serving.
type wafAndParts struct {
*waf.Server
waf.Params
Addr string
}
// banAbuseIPDBBlacklist does on w what a fetch of AbuseIPDB's blacklist with
// the clients on listed does.
func (w *wafAndParts) banAbuseIPDBBlacklist(
listed []reputation.BlacklistEntry,
) {
waf.BanAbuseIPDBBlacklist(
w.Config,
w.Ledger,
w.ProcessLog,
w.Now(),
listed,
)
}
// newWAF makes waf's server, without starting it, and returns it with its
// parts, what it writes, and the queue of the alerts it raises, as the
// newWAF makes waf's server, without starting it, and
// returns it, what it writes, and the queue of the alerts it raises, as the
// settings in env make it. No alert is sent from the queue: they wait in it,
// for the test to look at.
// With no geojsURL, there is no stand-in for GeoJS to look clients up at, and
@@ -456,29 +432,25 @@ func (w *wafAndParts) banAbuseIPDBBlacklist(
// at abuseIPDBURL while env sets SWW_WAF_ABUSEIPDB_KEY, and its blacklist is
// fetched from abuseIPDBBlacklistURL.
// SWW_WAF_CORE_RULE_SET_MODE is off unless env sets it, so that only the tests of the
// Core Rule Set have their requests inspected by it. Likewise the anomaly thresholds
// per netblock and per AS number are off unless env sets them, so that only the tests
// of the anomaly thresholds count requests.
// Core Rule Set have their requests inspected by it.
func newWAF(
t *testing.T,
geojsURL string,
now func() time.Time,
env map[string]string,
) (*wafAndParts, *output, *alerts.Queue) {
) (*waf.Server, *output, *alerts.Queue) {
t.Helper()
return newWAFWith(
t, func(*waf.Params) {}, geojsURL, abuseIPDBURL, now, env,
)
return newWAFWith(t, waf.New, geojsURL, abuseIPDBURL, now, env)
}
// newWAFWith is newWAF with change applied to the Params before the server is
// built from them, and clients checked with AbuseIPDB at abuseIPDBCheckURL.
// newWAFWith is newWAF for the server build makes, with clients checked with
// AbuseIPDB at abuseIPDBCheckURL.
func newWAFWith(
t *testing.T, change func(*waf.Params),
t *testing.T, build func(waf.Params) *waf.Server,
geojsURL, abuseIPDBCheckURL string, now func() time.Time,
env map[string]string,
) (*wafAndParts, *output, *alerts.Queue) {
) (*waf.Server, *output, *alerts.Queue) {
t.Helper()
settings := map[string]string{
@@ -488,7 +460,6 @@ func newWAFWith(
settings[lookupSource] = off
}
maps.Copy(settings, netAndASNThresholdsOff())
maps.Copy(settings, env)
cfg, err := config.WAFFromEnvironment(configtest.Viper(settings))
@@ -547,9 +518,8 @@ func newWAFWith(
addCoreRuleSet(t, &params)
addCounters(&params)
addOutsideSources(&params, geojsURL, abuseIPDBCheckURL)
change(&params)
server := &wafAndParts{Server: waf.New(params), Params: params}
server := build(params)
return server, out, alertQueue
}
@@ -578,11 +548,16 @@ func addCoreRuleSet(t *testing.T, params *waf.Params) {
params.CoreRuleSet = coreRuleSet
}
// addCounters adds to params the table of clients, the ledger and the
// anomaly counters, as sww builds them from params' settings.
// addCounters adds to params the metrics, the table of clients, the ledger
// and the anomaly counters, as sww builds them from params' settings. The
// metrics read the ledger, the table of clients and the rules.
func addCounters(params *waf.Params) {
cfg := params.Config
params.Metrics = metrics.New(
cfg.MetricsBusiestCountriesAndASNs,
cfg.InstanceName,
)
params.Limiter = ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{
PerMinute: cfg.RateLimitPerMinute,
@@ -612,18 +587,26 @@ func addCounters(params *waf.Params) {
NamedNetblocks: cfg.AnomalyWatchedNets,
Alerts: params.Alerts,
})
params.Metrics.AddBansAndClients(
params.Ledger,
params.Limiter,
params.Now,
)
params.Metrics.AddRules(params.Rules)
}
// addOutsideSources adds to params the lists fetched from URLs, the DNSBL
// zones' verdicts, AbuseIPDB's scores from abuseIPDBCheckURL and its
// blacklist from abuseIPDBBlacklistURL, routewatch's answers while its
// setting is set, the metrics newMetrics returns, and GeoJS at geojsURL, as
// sww builds them from params' settings and the parts addCounters added.
// setting is set, and GeoJS at geojsURL, as sww builds
// them from params' settings and the parts addCounters added. The metrics
// read the lists, the zones' verdicts, and AbuseIPDB's while its key is set.
func addOutsideSources(
params *waf.Params, geojsURL, abuseIPDBCheckURL string,
) {
cfg := params.Config
limiter, ledger := params.Limiter, params.Ledger
m, limiter, ledger := params.Metrics, params.Limiter, params.Ledger
processLog, alertQueue := params.ProcessLog, params.Alerts
params.Lists = reputation.New(reputation.Params{
@@ -672,9 +655,6 @@ func addOutsideSources(
})
}
m := newMetrics(params)
params.Metrics = m
params.GeoJS = geojs.New(geojs.Params{
URL: geojsURL,
Answered: func(answer geojs.Answer) {
@@ -687,29 +667,12 @@ func addOutsideSources(
},
WentWithoutAnswer: m.RequestsWithoutGeoJSAnswer.Inc,
})
}
// newMetrics returns the metrics, as sww builds them from params' settings,
// read from the ledger, the table of clients, the rules, the lists, the
// zones' verdicts, and AbuseIPDB's while its key is set.
func newMetrics(params *waf.Params) *metrics.Metrics {
cfg := params.Config
metricsParams := metrics.Params{
BusiestCount: cfg.MetricsBusiestCountriesAndASNs,
InstanceName: cfg.InstanceName,
Now: params.Now,
Bans: params.Ledger,
Clients: params.Limiter,
Rules: params.Rules,
Lists: params.Lists,
DNSBL: params.DNSBL,
}
m.AddReputation(params.Lists, params.DNSBL)
if cfg.AbuseIPDBKey != "" {
metricsParams.AbuseIPDB = params.AbuseIPDB
m.AddAbuseIPDB(params.AbuseIPDB)
}
return metrics.New(metricsParams)
}
// newClient returns an HTTP client that sends requests as they are made,
@@ -831,9 +794,7 @@ func readResponse(t *testing.T, conn net.Conn) answer {
}
// wantLine checks the request log line's status and action.
func wantLine(
t *testing.T, line logLine, status int, action requestlog.Action,
) {
func wantLine(t *testing.T, line logLine, status int, action string) {
t.Helper()
if line.Status != status || line.Action != action {
-235
View File
@@ -1,235 +0,0 @@
package waf
import (
"context"
"errors"
"net/http"
"strings"
"time"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// answerQuestion runs the checks on the visitor's request a question
// carries, and answers rproxy what to do with it. A denied request's counting
// is finished here, and an allowed one's by its report.
func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
question, ok := h.readQuestion(w, r)
if !ok {
return
}
rq := h.startRequest(
w,
question.Visitor,
question.Client,
question.Peer,
true,
)
// These run even when a check panics. finish runs first, so that what is
// held has the finished log line. Go's server sends the answer only once
// this function has returned, so the report cannot come before it is held.
defer func() {
rq.finish()
rq.holdForReport()
}()
rq.site = r.Host
rq.line.WAFLine = requestlog.WAFLineDecision
refused := rq.check(r.Context())
rq.checked = time.Now()
underOwnPrefix := strings.HasPrefix(
r.URL.Path,
wafprotocol.OwnEndpointPrefix,
)
if refused == nil && !underOwnPrefix {
rq.allow()
// In observe mode, what enforce mode would have refused counts here.
rq.runBanningHeuristics(r.Context(), rq.h.afterAnswer)
return
}
if refused != nil {
rq.deny(*refused)
} else {
// The app never sees this prefix. waf's own endpoints are at its
// own address, so this answers as a path with no endpoint does.
rq.line.Action = requestlog.ActionOwnEndpoint
wafprotocol.SetAnswerHeaders(
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionDeny},
)
http.Error(
rq.out,
http.StatusText(http.StatusNotFound),
http.StatusNotFound,
)
}
rq.banAndCount(r.Context(), rq.h.afterAnswer)
}
// readQuestion reads a question, and reports whether waf takes it. It answers
// a question it does not take with 400 and no disposition. rproxy passes that
// 400 to the visitor, and SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION does
// not apply. The id of a question waf takes is held from here on, without a
// decision yet, so that another question with that id is refused while this
// one is being handled.
//
// A question whose body stops arriving gets no answer, and is not looked at.
func (h *handler) readQuestion(
w http.ResponseWriter, r *http.Request,
) (wafprotocol.Question, bool) {
question, err := wafprotocol.ReadQuestion(w, r)
if errors.Is(err, wafprotocol.ErrBodyStoppedArriving) {
h.processLog.Warn(
"a question's body stopped arriving",
"error",
err.Error(),
)
// This panic has Go's server close the connection without an answer,
// and log nothing.
panic(http.ErrAbortHandler)
}
if err != nil {
h.refuseQuestion(w, err.Error())
return question, false
}
held, _ := h.awaitingReport.ContainsOrAdd(question.RequestID, nil)
if held {
h.refuseQuestion(
w,
wafprotocol.RequestIDHeader+" is the id of a request waf holds",
)
return question, false
}
return question, true
}
// refuseQuestion answers a question waf does not take, for reason.
func (h *handler) refuseQuestion(w http.ResponseWriter, reason string) {
h.processLog.Warn("a question is refused", "reason", reason)
http.Error(w, reason, http.StatusBadRequest)
}
// allow answers a question with wafprotocol.DispositionAllow and, while
// SWW_WAF_ADD_LOOKUP_HEADERS is true, the client's AS number and country.
func (rq *request) allow() {
rq.allowed = true
answer := wafprotocol.Answer{Disposition: wafprotocol.DispositionAllow}
if rq.h.config.AddLookupHeaders {
answer.ASN = rq.lookupAnswer.ASN
answer.Country = rq.lookupAnswer.Country
}
wafprotocol.SetAnswerHeaders(rq.out.Header(), answer)
rq.out.WriteHeader(http.StatusOK)
}
// holdForReport holds a copy of what the report's steps need, by the request's
// id. A report only ever reaches that copy, never the request being handled.
func (rq *request) holdForReport() {
held := rq.decision
rq.h.awaitingReport.Add(rq.line.RequestID, &held)
}
// deny answers a question with what the visitor is to get for r.
func (rq *request) deny(r refusal) {
rq.refused = &r
if r.status == 0 {
// SWW_WAF_BAN_RESPONSE is close. Written past rq.out, since the log line's
// status stays 0: the visitor is sent nothing.
wafprotocol.SetAnswerHeaders(
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionClose},
)
rq.out.ResponseWriter.WriteHeader(http.StatusOK)
return
}
wafprotocol.SetAnswerHeaders(
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionDeny},
)
http.Error(rq.out, http.StatusText(r.status), r.status)
}
// handleReport answers POST wafprotocol.ReportPath: it finishes the request
// the report names, if it holds that request's decision, and ignores the
// report otherwise.
func (h *handler) handleReport() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
report, err := wafprotocol.ReadReport(w, r)
if err != nil {
h.processLog.Warn("a report is refused", "error", err.Error())
http.Error(w, "the body is not a report", http.StatusBadRequest)
return
}
held, found := h.awaitingReport.Peek(report.RequestID)
// Remove reports true to one report only, of several that name the
// id. A question still being handled is held as nil.
if found && held != nil &&
h.awaitingReport.Remove(report.RequestID) {
h.finishWith(r.Context(), held, report)
}
w.WriteHeader(http.StatusNoContent)
}
}
// finishWith finishes a request with what its report says, and writes the
// request's last log line, which holds its decision's fields and the report's.
// For an allowed request it first runs the heuristics of onReport, and adds the
// request to its client's history, the anomaly counters and the metrics, with
// the status and bytes the report gives. A denied request's counting was
// finished at its decision. ctx is the report's own context.
func (h *handler) finishWith(
ctx context.Context,
held *decision,
report wafprotocol.Report,
) {
rq := &request{
decision: *held,
h: h,
out: &responseWriter{
status: report.Status,
bytes: report.ResponseBytes,
},
requestBytes: report.RequestBytes,
}
rq.line.WAFLine = requestlog.WAFLineReport
rq.line.Status = report.Status
rq.line.RequestBytes = report.RequestBytes
rq.line.ResponseBytes = report.ResponseBytes
rq.line.DurationTotalMS = report.DurationTotalMS
if rq.allowed {
rq.line.DurationUpstreamTotalMS = &report.DurationUpstreamTotalMS
rq.banAndCount(ctx, rq.h.onReport)
rq.h.metrics.RequestReported(
&rq.line,
time.Duration(report.DurationTotalMS*float64(time.Millisecond)),
time.Duration(
report.DurationUpstreamTotalMS*float64(time.Millisecond),
),
)
}
rq.writeLine()
}
+9 -14
View File
@@ -17,9 +17,9 @@ import (
// countedByRateLimits exempts it, by the client's limit percentages, which it
// notes for the log line with the client's counts. A request that takes the
// client over a rate limit, as its limit percentage lowers it, breaks it: it is
// refused with SWW_WAF_BAN_RESPONSE, and bans the client. A request exempt
// from the rate limits is exempt from the byte limits too. Its figures are the
// window, the count and the limit.
// refused, and bans the client. A request exempt from the rate limits is exempt
// from the byte limits too. Its figures are the window, the count and the
// limit.
type rateLimits struct {
limiter *ratelimit.Limiter
exemptNets []netip.Prefix // SWW_WAF_RATE_LIMIT_EXEMPT_NETS
@@ -63,20 +63,15 @@ func (r rateLimits) check(
r.banMaker.banForLimit(rq, now, hit, r.banResponse)
return &verdict{
action: requestlog.ActionClientIsRateLimited,
status: r.banResponse,
figures: limitFigures(hit),
action: requestlog.ActionClientIsRateLimited, figures: limitFigures(hit),
}
}
// refusesWhateverTheLookupAnswers reports whether the heuristic rate_limits
// will refuse rq at now, whatever its client's limit percentages, which the
// lookup's answer can lower: the rate limits count it, and one more request
// from its client takes it over a whole rate limit. It counts nothing.
func (r rateLimits) refusesWhateverTheLookupAnswers(
rq *request,
now time.Time,
) bool {
// overAWholeRateLimit reports whether the heuristic rate_limits will refuse rq
// at now, whatever its client's limit percentages: the rate limits count it,
// and one more request from its client takes it over a whole rate limit. It
// counts nothing.
func (r rateLimits) overAWholeRateLimit(rq *request, now time.Time) bool {
return r.countedByRateLimits(rq) &&
r.limiter.OverARateLimit(
clientGroup(r.ipv6GroupPrefix, rq.client),
+1 -1
View File
@@ -121,7 +121,7 @@ func TestIPv6GroupPrefixSetsTheClientTheLimitsCount(t *testing.T) {
// status and action are those of the request that breaks the
// limit: a rate limit refuses it, a byte limit passes it on.
status int
action requestlog.Action
action string
}{
{rateLimitPerMinute, "1", http.StatusForbidden, requestlog.ActionClientIsRateLimited},
{bytesLimitPerMinute, byteLimit, http.StatusOK, requestlog.ActionForward},
+19 -4
View File
@@ -1,12 +1,15 @@
package waf
// The heuristic refusal_limit. It reads SWW_WAF_REFUSALS_LIMIT_PER_MINUTE.
// The heuristic refusal_limit. It reads SWW_WAF_REFUSALS_LIMIT_PER_MINUTE and
// SWW_WAF_BAN_RESPONSE.
import (
"context"
"net/http"
"time"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// refusalLimit counts the request once it has been answered, if waf
@@ -15,11 +18,11 @@ import (
// SWW_WAF_REFUSALS_LIMIT_PER_MINUTE such refusals of the client within a minute break
// a limit, which bans the client. A client in SWW_WAF_ALLOW_NETS, which the
// heuristics skip, is not counted, and nothing is while the limit is off.
// The ban's notes give the status the client was sent, or in observe mode
// would have been. Its figures are the window, the count and the limit.
// Its figures are the window, the count and the limit.
type refusalLimit struct {
limiter *ratelimit.Limiter
perMinute int64 // SWW_WAF_REFUSALS_LIMIT_PER_MINUTE
banResponse int // SWW_WAF_BAN_RESPONSE
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
banMaker banMaker
}
@@ -48,7 +51,19 @@ func (r refusalLimit) check(
return nil
}
r.banMaker.banForLimit(rq, now, hit, rq.refusalStatus)
// What the client was sent, or in observe mode would have been.
status := rq.out.status
switch rq.line.WouldAction {
case requestlog.ActionBlockedByBlockRule,
requestlog.ActionBlockedByCoreRuleSet:
status = http.StatusForbidden
case requestlog.ActionTemporarilyBanned,
requestlog.ActionPermanentlyBanned:
status = r.banResponse
}
r.banMaker.banForLimit(rq, now, hit, status)
return &verdict{figures: limitFigures(hit)}
}
+3 -2
View File
@@ -12,6 +12,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
const refusalsLimitPerMinute = "SWW_WAF_REFUSALS_LIMIT_PER_MINUTE"
@@ -51,7 +52,7 @@ func (r refused) send(s *sender, from string) logLine {
// SWW_WAF_REFUSALS_LIMIT_PER_MINUTE at limit, and the settings in env.
func startForRefusalLimit(
t *testing.T, limit string, env map[string]string,
) (*sender, *clock, *wafAndParts) {
) (*sender, *clock, *waf.Server) {
t.Helper()
settings := map[string]string{
@@ -73,7 +74,7 @@ func TestRefusalLimitBreaksAtOneOverTheLimit(t *testing.T) {
refusals []refused
// nextAction is the action of the client's next request: the next
// request after a ban for a ban rule makes it permanent.
nextAction requestlog.Action
nextAction string
}{
{
"block rule",
+3 -3
View File
@@ -47,9 +47,9 @@ func (n reputationHitNoter) noteHit(
Host: rq.site,
Client: rq.client,
Netblock: clientGroup(n.ipv6GroupPrefix, rq.client),
ASN: rq.lookupAnswer.ASN,
ASName: rq.lookupAnswer.ASName,
Country: rq.lookupAnswer.Country,
ASN: rq.line.ASN,
ASName: rq.line.ASName,
Country: rq.line.Country,
Reason: reason,
Detail: detail,
})
+25 -24
View File
@@ -18,6 +18,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// The reputation settings.
@@ -57,22 +58,22 @@ func TestEachBlocklistActionForAListedAddressAndAListedNetblock(
// statuses and actions are those of a listed client's three
// requests, and percent their rate_limit_percent, as percentText gives it.
statuses []int
actions []requestlog.Action
actions []string
percent string
}{
{
actionDeny, []int{http.StatusForbidden, http.StatusForbidden, http.StatusForbidden},
[]requestlog.Action{denied, denied, denied}, none,
[]string{denied, denied, denied}, none,
},
{
// Half of 4 requests a minute: the third breaks the limit.
limitHalf, []int{http.StatusOK, http.StatusOK, http.StatusForbidden},
[]requestlog.Action{forward, forward, requestlog.ActionClientIsRateLimited},
[]string{forward, forward, requestlog.ActionClientIsRateLimited},
"50 from " + blocklistAction,
},
{
actionLog, []int{http.StatusOK, http.StatusOK, http.StatusOK},
[]requestlog.Action{forward, forward, forward}, none,
[]string{forward, forward, forward}, none,
},
} {
t.Run(tc.action, func(t *testing.T) {
@@ -138,13 +139,13 @@ func TestReputationRefusalNamesItsHeuristicAndWhatListedTheClient(
for _, tc := range []struct {
heuristic string
env map[string]string
load func(*testing.T, *wafAndParts)
load func(*testing.T, *waf.Server)
figures string
}{
{
"blocklists",
map[string]string{blocklistURLs: dropURL, blocklistAction: actionDeny},
func(t *testing.T, server *wafAndParts) {
func(t *testing.T, server *waf.Server) {
t.Helper()
loadLists(t, server, map[string][]string{dropURL: {client}})
},
@@ -156,7 +157,7 @@ func TestReputationRefusalNamesItsHeuristicAndWhatListedTheClient(
dnsblZones: dnsblZone, dnsblResolver: noResolver,
dnsblAbuseIPDBAction: actionDeny,
},
func(_ *testing.T, server *wafAndParts) {
func(_ *testing.T, server *waf.Server) {
loadVerdicts(server, map[string][]string{client: {dnsblZone}})
},
`{"zones": ["` + dnsblZone + `"]}`,
@@ -164,7 +165,7 @@ func TestReputationRefusalNamesItsHeuristicAndWhatListedTheClient(
{
"abuseipdb",
map[string]string{abuseIPDBKey: accountKey, dnsblAbuseIPDBAction: actionDeny},
func(_ *testing.T, server *wafAndParts) {
func(_ *testing.T, server *waf.Server) {
loadScores(server, map[string]int64{client: 80})
},
`{"score": 80, "min_score": 75}`,
@@ -441,22 +442,22 @@ func TestEachDNSBLAbuseIPDBActionForAClientADNSBLZoneLists(t *testing.T) {
// statuses and actions are those of a listed client's three
// requests, and percent their rate_limit_percent, as percentText gives it.
statuses []int
actions []requestlog.Action
actions []string
percent string
}{
{
actionDeny, []int{http.StatusForbidden, http.StatusForbidden, http.StatusForbidden},
[]requestlog.Action{denied, denied, denied}, none,
[]string{denied, denied, denied}, none,
},
{
// Half of 4 requests a minute: the third breaks the limit.
limitHalf, []int{http.StatusOK, http.StatusOK, http.StatusForbidden},
[]requestlog.Action{forward, forward, requestlog.ActionClientIsRateLimited},
[]string{forward, forward, requestlog.ActionClientIsRateLimited},
"50 from " + dnsblAbuseIPDBAction,
},
{
actionLog, []int{http.StatusOK, http.StatusOK, http.StatusOK},
[]requestlog.Action{forward, forward, forward}, none,
[]string{forward, forward, forward}, none,
},
} {
t.Run(tc.action, func(t *testing.T) {
@@ -559,7 +560,7 @@ func TestReputationSourcesTagTheRequestWhateverTheirAction(t *testing.T) {
for _, tc := range []struct {
name string
env map[string]string
load func(*wafAndParts)
load func(*waf.Server)
want []string
}{
{
@@ -567,7 +568,7 @@ func TestReputationSourcesTagTheRequestWhateverTheirAction(t *testing.T) {
map[string]string{
dnsblZones: dnsblZone, dnsblResolver: noResolver, dnsblAbuseIPDBAction: actionLog,
},
func(server *wafAndParts) {
func(server *waf.Server) {
loadVerdicts(server, map[string][]string{fromDE: {dnsblZone}})
},
[]string{"asn=64496", "country=DE", "dnsbl=" + dnsblZone},
@@ -575,7 +576,7 @@ func TestReputationSourcesTagTheRequestWhateverTheirAction(t *testing.T) {
{
abuseipdb,
map[string]string{abuseIPDBKey: accountKey, dnsblAbuseIPDBAction: actionLog},
func(server *wafAndParts) {
func(server *waf.Server) {
loadScores(server, map[string]int64{fromDE: 80})
},
[]string{abuseipdb, "asn=64496", "country=DE"},
@@ -953,7 +954,7 @@ func TestClientRefusedForAnOffenceIsCheckedWithAbuseIPDBAtItsNextRequest(
// history then counts.
path string
status int
action requestlog.Action
action string
want ratelimit.Offences
}{
{
@@ -1017,22 +1018,22 @@ func TestEachDNSBLAbuseIPDBActionForAClientAbuseIPDBScoresAtOrOverTheMinimum(
// statuses and actions are those of fromDE's three requests, and
// percent their rate_limit_percent, as percentText gives it.
statuses []int
actions []requestlog.Action
actions []string
percent string
}{
{
actionDeny, []int{http.StatusForbidden, http.StatusForbidden, http.StatusForbidden},
[]requestlog.Action{denied, denied, denied}, none,
[]string{denied, denied, denied}, none,
},
{
// Half of 4 requests a minute: the third breaks the limit.
limitHalf, []int{http.StatusOK, http.StatusOK, http.StatusForbidden},
[]requestlog.Action{forward, forward, requestlog.ActionClientIsRateLimited},
[]string{forward, forward, requestlog.ActionClientIsRateLimited},
"50 from " + dnsblAbuseIPDBAction,
},
{
actionLog, []int{http.StatusOK, http.StatusOK, http.StatusOK},
[]requestlog.Action{forward, forward, forward}, none,
[]string{forward, forward, forward}, none,
},
} {
t.Run(tc.action, func(t *testing.T) {
@@ -1255,7 +1256,7 @@ func verdictsFetched() time.Time {
// verdictsFetched, as dnsbl.json would at start: one that lists the
// client from each zone listedBy gives for it, and one that does not from
// each other zone.
func loadVerdicts(server *wafAndParts, listedBy map[string][]string) {
func loadVerdicts(server *waf.Server, listedBy map[string][]string) {
verdicts := make(
[]reputation.Verdict,
0,
@@ -1280,7 +1281,7 @@ func loadVerdicts(server *wafAndParts, listedBy map[string][]string) {
// loadScores puts into server's AbuseIPDB the score scores gives each
// client, an IPv4 address, fetched at verdictsFetched, as
// abuseipdb-scores.json would at start.
func loadScores(server *wafAndParts, scores map[string]int64) {
func loadScores(server *waf.Server, scores map[string]int64) {
kept := make([]reputation.Score, 0, len(scores))
for client, score := range scores {
kept = append(kept, reputation.Score{
@@ -1294,7 +1295,7 @@ func loadScores(server *wafAndParts, scores map[string]int64) {
// wantAbuseIPDBChecks checks how many clients server has checked with
// AbuseIPDB.
func wantAbuseIPDBChecks(t *testing.T, server *wafAndParts, want int) {
func wantAbuseIPDBChecks(t *testing.T, server *waf.Server, want int) {
t.Helper()
if got := server.AbuseIPDB.Checked(); got != want {
@@ -1306,7 +1307,7 @@ func wantAbuseIPDBChecks(t *testing.T, server *wafAndParts, want int) {
// its lines, fetched at listsFetched, as lists.json would at start.
func loadLists(
t *testing.T,
server *wafAndParts,
server *waf.Server,
copies map[string][]string,
) {
t.Helper()
+45 -54
View File
@@ -21,71 +21,45 @@ import (
// to close its connection without an answer, and the action the log line names.
type refusal struct {
status int
action requestlog.Action
action string
}
// decision is what the checks note of a request: its log line, what the
// heuristics tell each other, and what counting the request needs once it has
// ended. Nothing reads the log line back to decide. waf keeps a copy of it
// until the request's report comes. It holds no credentials, since the log
// line never does, and neither the request's objects nor their bodies.
// decision is what the checks note of a request: its log line, and what
// counting the request needs once it has ended. waf keeps a copy of it
// until the request's report comes. It holds no credentials, since the log line
// never does, and neither the request's objects nor their bodies.
type decision struct {
line requestlog.Line
client netip.Addr
// method and userAgent are the request's method and User-Agent, and
// requestURI its path with the query, as a ban's notes keep them.
// startRequest sets them.
method, userAgent, requestURI string
// tags are the request's tags, sorted, each once. addTag adds them, for
// tagLists, routeWatchTags, noteLookup, dnsbl, abuseIPDB, ruleFiles and
// coreRuleSet.
tags []string
// requestURI is the path with the query, as a ban's notes keep it.
requestURI string
// site is, for a question, the host the visitor asked for, which the
// alerts and a ban's notes name, since one waf serves many sites.
// answerQuestion sets it.
// alerts name, since one waf serves many sites.
site string
// allowed is true for a question answered with DispositionAllow.
allowed bool
// lookedUp is true once the client's AS number and country have been
// looked up, whether or not an answer was there, and lookupAnswer is
// what the lookup gave then, the zero Answer while GeoJS had given none.
// noteLookup sets them.
lookedUp bool
lookupAnswer geojs.Answer
// skippedLookupWait is true for a request that did not wait for GeoJS and
// found no answer kept. SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT does not apply to it,
// so that not waiting never lowers its client's limits. noteLookup sets
// it.
// so that not waiting never lowers its client's limits.
skippedLookupWait bool
// counted is true for a request the rate limits counted, whose bytes
// the byte limits count once it has ended. rateLimitPercent and
// bytesLimitPercent are then its client's limit percentages for the rate
// limits and for the byte limits. rateLimits sets them.
// limits and for the byte limits.
counted bool
rateLimitPercent, bytesLimitPercent percentage
// attack is true for a request that matched a ban rule or asked for a
// trap path, blockedByBlockRule for one a block rule refused, and
// blockedByCoreRuleSet for one the Core Rule Set refused, each an offence
// its client's history counts. trapPaths and ruleFiles set attack,
// ruleFiles blockedByBlockRule, and coreRuleSet blockedByCoreRuleSet.
// its client's history counts.
attack, blockedByBlockRule, blockedByCoreRuleSet bool
// refusalStatus is the status of the verdict that refused the request, or
// in observe mode would have: what the visitor was answered with, or would
// have been. check sets it.
refusalStatus int
// wouldAction is, in observe mode, the action enforce mode would have
// refused the request with, one of requestlog's Action constants, and ""
// otherwise. check sets it.
wouldAction requestlog.Action
// brokeLimit is true for a request that broke a limit, an offence its
// client's history counts. banForLimit sets it.
brokeLimit bool
// blocklisted is true once blocklists finds a blocklist that lists the
// client, dnsblListed once dnsbl finds a DNSBL zone's verdict that does,
// and abuseIPDBHit once abuseIPDB finds AbuseIPDB's score of it a hit.
blocklisted, dnsblListed, abuseIPDBHit bool
// reputation is the reputation sources that list the client, for the
// notes of a ban the request makes. noteHit adds them.
// notes of a ban the request makes.
reputation []bans.ReputationHit
}
@@ -98,7 +72,11 @@ type request struct {
in *http.Request
out *responseWriter
start time.Time
// blocklisted is true once a blocklist is found to list the client,
// dnsblListed once a DNSBL zone's verdict is, and abuseIPDBHit once
// AbuseIPDB's score of it is a hit.
blocklisted, dnsblListed, abuseIPDBHit bool
start time.Time
// checked is when the checks were done.
checked time.Time
// refused is what the request was refused with, nil if it was not.
@@ -128,8 +106,6 @@ func (h *handler) startRequest(
rq := &request{
decision: decision{
client: client,
method: r.Method,
userAgent: r.UserAgent(),
requestURI: r.URL.RequestURI(),
line: requestlog.Line{
Time: requestlog.FormatTime(start),
@@ -175,12 +151,11 @@ func (h *handler) startRequest(
}
// addTag adds tag to the request's tags, which stay sorted and hold each tag
// once, and to its log line.
// once.
func (rq *request) addTag(tag string) {
at, found := slices.BinarySearch(rq.tags, tag)
at, found := slices.BinarySearch(rq.line.Tags, tag)
if !found {
rq.tags = slices.Insert(rq.tags, at, tag)
rq.line.Tags = rq.tags
rq.line.Tags = slices.Insert(rq.line.Tags, at, tag)
}
}
@@ -207,8 +182,9 @@ func requestHeaders(r *http.Request, names []string) map[string]string {
// check is the one place where a request can be refused. It returns nil to let
// the request through. The heuristics of checkClient decide: a request they
// refuse is answered with the status and logged with the action of the verdict
// that refused it. In observe mode nothing is refused, and the log line names
// refuse is answered with SWW_WAF_BAN_RESPONSE, 403 for a block rule or the Core
// Rule Set, 503 for heuristics that ran out of time, or 414 or 431 for the
// header limits. In observe mode nothing is refused, and the log line names
// what enforce mode would have done. ctx is the request's own context.
func (rq *request) check(ctx context.Context) *refusal {
v := rq.checkClient(ctx)
@@ -216,16 +192,31 @@ func (rq *request) check(ctx context.Context) *refusal {
return nil
}
rq.refusalStatus = v.status
action := v.action
if rq.h.config.Observe {
rq.wouldAction = v.action
rq.line.WouldAction = v.action
switch {
case rq.h.config.Observe:
rq.line.WouldAction = action
return nil
case action == requestlog.ActionBlockedByBlockRule ||
action == requestlog.ActionBlockedByCoreRuleSet:
return &refusal{status: http.StatusForbidden, action: action}
case action == requestlog.ActionChecksTimedOut:
return &refusal{
status: http.StatusServiceUnavailable,
action: action,
}
case action == requestlog.ActionRequestLineTooLong:
return &refusal{status: http.StatusRequestURITooLong, action: action}
case action == requestlog.ActionRequestHeaderTooLarge:
return &refusal{
status: http.StatusRequestHeaderFieldsTooLarge,
action: action,
}
default:
return rq.banResponse(action)
}
return &refusal{status: v.status, action: v.action}
}
// checkClient runs the heuristics on the request, and returns the first
@@ -380,7 +371,7 @@ func (rq *request) addToHistory() {
Status: rq.out.status,
RequestBytes: rq.requestBytes,
ResponseBytes: rq.out.bytes,
BrokeLimit: rq.brokeLimit,
BrokeLimit: rq.line.Offence == requestlog.OffenceLimit,
Attack: rq.attack,
BlockedByBlockRule: rq.blockedByBlockRule,
BlockedByCoreRuleSet: rq.blockedByCoreRuleSet,
-142
View File
@@ -1,142 +0,0 @@
package waf
import (
"crypto/subtle"
"io"
"net/http"
"strings"
"github.com/go-chi/chi/v5"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// routes returns the router of waf's own endpoints. A question never reaches
// it, since ServeHTTP answers a question first. No check runs on a request
// here: waf's address is on the private network, where the peers are rproxy
// and the admin's tools, not visitors.
func (h *handler) routes() http.Handler {
r := chi.NewRouter()
r.Use(matchDecodedPath)
// The report writes no log line of its own: it finishes the line of the
// request it reports.
r.Post(wafprotocol.ReportPath, h.handleReport())
r.Group(func(r chi.Router) {
r.Use(h.logOwnEndpoint)
// The answer to a path with no endpoint, and to a method the path has
// no endpoint for, is set in this group, so that it is logged too.
r.NotFound(answerNotFound)
r.MethodNotAllowed(answerNotFound)
r.Get(wafprotocol.HealthPath, handleHealth())
r.Group(func(r chi.Router) {
r.Use(requireToken(h.config.MetricsToken))
r.Get(MetricsPath, h.metrics.ServeHTTP)
})
// The handlers under BansPath/ and ClientsPath read the address from
// the path themselves, so that one that is not an address gets 400
// after the token check.
r.Group(func(r chi.Router) {
r.Use(requireToken(h.config.AdminToken))
r.Get(BansPath, h.handleListBans())
r.Post(BansPath, h.handleAddBan())
r.Delete(BansPath+"/*", h.handleLiftBans())
r.Get(ClientsPath+"*", h.handleShowClient())
})
})
return r
}
// matchDecodedPath has the router match the decoded path, which the handlers
// read. chi matches the path as sent when it is encoded in an unusual way,
// which would make /_sww/%68ealthz a path with no endpoint.
func matchDecodedPath(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
chi.RouteContext(r.Context()).RoutePath = r.URL.Path
next.ServeHTTP(w, r)
})
}
// logOwnEndpoint writes one log line for each request, with the action
// own_endpoint. The line's request_bytes is how much of the body the handler
// read.
func (h *handler) logOwnEndpoint(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
peer := peerAddress(r)
rq := h.startRequest(w, r, peer, peer, false)
defer rq.finish()
rq.line.Action = requestlog.ActionOwnEndpoint
counted := r.Clone(r.Context())
counted.Body = &countedBody{
ReadCloser: r.Body,
read: &rq.requestBytes,
}
next.ServeHTTP(rq.out, counted)
})
}
// countedBody is a request's body that adds the bytes read of it to read.
type countedBody struct {
io.ReadCloser
read *int64
}
// Read reads part of the body.
func (b *countedBody) Read(p []byte) (int, error) {
n, err := b.ReadCloser.Read(p)
*b.read += int64(n)
return n, err
}
// requireToken returns middleware that lets a request through only when it
// carries token, as Authorization: Bearer <token>. It answers 401 to a request
// without it. While token is unset, it answers 404, as a path with no endpoint
// does.
func requireToken(token string) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
switch {
case token == "":
answerNotFound(w, r)
case !hasToken(r, token):
w.Header().Set("WWW-Authenticate", "Bearer")
http.Error(
w,
http.StatusText(http.StatusUnauthorized),
http.StatusUnauthorized,
)
default:
next.ServeHTTP(w, r)
}
},
)
}
}
// hasToken reports whether r carries token, as Authorization: Bearer
// <token>.
func hasToken(r *http.Request, token string) bool {
scheme, sent, _ := strings.Cut(r.Header.Get("Authorization"), " ")
return strings.EqualFold(scheme, "Bearer") &&
subtle.ConstantTimeCompare([]byte(sent), []byte(token)) == 1
}
// answerNotFound answers 404, for a path with no endpoint, and for a method
// the path has no endpoint for.
func answerNotFound(w http.ResponseWriter, _ *http.Request) {
http.Error(w, http.StatusText(http.StatusNotFound), http.StatusNotFound)
}
+2 -1
View File
@@ -11,6 +11,7 @@ import (
"time"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// routeWatchURL is SWW_WAF_ROUTEWATCH_URL, and tagsMissing the log line's field
@@ -57,7 +58,7 @@ func startRouteWatch(t *testing.T, routes map[string]standInRoute) string {
// waitForRouteWatch waits until routewatch's answer about from, in the client
// group of the given length, is kept, so that the next request gets its tags.
func waitForRouteWatch(
t *testing.T, server *wafAndParts, from string, bits int,
t *testing.T, server *waf.Server, from string, bits int,
) {
t.Helper()
+5 -11
View File
@@ -1,12 +1,10 @@
package waf
// The heuristic rule_files. It reads the rules of the rule files, which
// SWW_WAF_RULE_FILES_DIR and SWW_WAF_RULE_FILES_ENABLED name, and
// SWW_WAF_BAN_RESPONSE.
// SWW_WAF_RULE_FILES_DIR and SWW_WAF_RULE_FILES_ENABLED name.
import (
"context"
"net/http"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
@@ -19,9 +17,8 @@ import (
// ruleFiles takes the rules of the rule files the request matched, which
// findBesideLookup found, notes their ids in the log line, adds the tag
// exploitsignature when it matches any, and refuses the request
// when the last rule it matches refuses it: with 403 and
// ActionBlockedByBlockRule for a block rule, and for a ban rule with
// SWW_WAF_BAN_RESPONSE and the action for the ban it makes. A ban
// when the last rule it matches refuses it: with ActionBlockedByBlockRule for a
// block rule, and for a ban rule with the action for the ban it makes. A ban
// rule bans the client's netblock for a clear sign of attack, or in observe
// mode raises the alert for the ban
// it would have made. Either rule's match is noted as an offence, for the
@@ -62,7 +59,6 @@ func (r ruleFiles) check(
return &verdict{
action: requestlog.ActionBlockedByBlockRule,
status: http.StatusForbidden,
figures: figures,
}
case rules.ActionBan:
@@ -74,13 +70,11 @@ func (r ruleFiles) check(
now,
bans.Notes{RuleID: last.ID, Target: last.Target},
),
status: r.banMaker.banResponse,
figures: figures,
}
case rules.ActionLog:
default:
return nil
}
return nil
}
// refusedByRule reports whether ruleFiles refuses a request that matched
+1 -1
View File
@@ -184,7 +184,7 @@ func TestRateLimitExemptNetsAreNeitherCountedNorRefused(t *testing.T) {
type sentRequest struct {
client string
status int
action requestlog.Action
action string
}
// wantAnswers sends requests to sww at addr one after another and
+5 -7
View File
@@ -1,6 +1,6 @@
package waf
// The heuristic tag_deny. It reads SWW_WAF_TAG_DENY and SWW_WAF_BAN_RESPONSE.
// The heuristic tag_deny. It reads SWW_WAF_TAG_DENY.
import (
"context"
@@ -10,11 +10,10 @@ import (
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// tagDeny refuses a request that carries a tag in SWW_WAF_TAG_DENY, with
// SWW_WAF_BAN_RESPONSE. Its figure is the first such tag, in sorted order.
// tagDeny refuses a request that carries a tag in SWW_WAF_TAG_DENY. Its figure is
// the first such tag, in sorted order.
type tagDeny struct {
tags []string // SWW_WAF_TAG_DENY
banResponse int // SWW_WAF_BAN_RESPONSE
tags []string // SWW_WAF_TAG_DENY
}
func (tagDeny) name() string { return "tag_deny" }
@@ -24,11 +23,10 @@ func (t tagDeny) check(
rq *request,
_ time.Time,
) *verdict {
for _, tag := range rq.tags {
for _, tag := range rq.line.Tags {
if slices.Contains(t.tags, tag) {
return &verdict{
action: requestlog.ActionClientDenied,
status: t.banResponse,
figures: map[string]any{"tag": tag},
}
}
Loaded 100 of 107 files, more files were not shown because too many files have changed in this diff. Show more