Compare commits
26
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b157359910 | ||
|
|
5dfa47c97b | ||
|
|
8219664ad1 | ||
|
|
58b9985b6d | ||
|
|
677df6a235 | ||
|
|
8188602a09 | ||
|
|
ee550be738 | ||
|
|
493aac02a1 | ||
|
|
4774518cb3 | ||
|
|
5d5e9542f8 | ||
|
|
1b9ca94774 | ||
|
|
9afbd1b683 | ||
|
|
cc77435f2a | ||
|
|
dcfdd12c51 | ||
|
|
2301a76029 | ||
|
|
abf6c8d517 | ||
|
|
17400f68b8 | ||
|
|
9d008ed31a | ||
|
|
86bb33bf07 | ||
|
|
91444a37b3 | ||
|
|
51989235d6 | ||
|
|
e15edef583 | ||
|
|
06faf0877c | ||
|
|
b2b1716907 | ||
|
|
56a57125f1 | ||
|
|
51245804d0 |
No files matched your search
@@ -308,6 +308,7 @@ Every setting the code reads, with its default:
|
|||||||
| `SWW_RPROXY_MAX_RESPONSE_SIZE` | `1G` | The largest response body, a size or `off`. The visitor gets `502` in place of a larger one, or it is cut off if it has already started. |
|
| `SWW_RPROXY_MAX_RESPONSE_SIZE` | `1G` | The largest response body, a size or `off`. The visitor gets `502` in place of a larger one, or it is cut off if it has already started. |
|
||||||
| `SWW_RPROXY_MAX_REQUEST_TIME_MS` | `60000` | Two time limits, each this long, in milliseconds or `off`: one for the visitor to send its request line and headers, then one for the whole request to reach the app. |
|
| `SWW_RPROXY_MAX_REQUEST_TIME_MS` | `60000` | Two time limits, each this long, in milliseconds or `off`: one for the visitor to send its request line and headers, then one for the whole request to reach the app. |
|
||||||
| `SWW_RPROXY_MAX_RESPONSE_TIME_MS` | `1800000` | How long the answer may take, from the end of the request to its last byte reaching the visitor, in milliseconds or `off`. |
|
| `SWW_RPROXY_MAX_RESPONSE_TIME_MS` | `1800000` | How long the answer may take, from the end of the request to its last byte reaching the visitor, in milliseconds or `off`. |
|
||||||
|
| `SWW_RPROXY_CONTACT_EMAIL` | unset | The email address the refusal pages link to. |
|
||||||
| `SWW_WAF_LISTEN_ADDR` | `:8080` | Where `waf` listens; leave the host part empty. |
|
| `SWW_WAF_LISTEN_ADDR` | `:8080` | Where `waf` listens; leave the host part empty. |
|
||||||
| `SWW_WAF_INSTANCE_NAME` | the container's host name | The name in every line `waf` logs, and in every alert and metric, such as `host1/waf`. |
|
| `SWW_WAF_INSTANCE_NAME` | the container's host name | The name in every line `waf` logs, and in every alert and metric, such as `host1/waf`. |
|
||||||
| `SWW_WAF_MODE` | `enforce` | `observe` refuses nothing and makes no ban; each log line says what `enforce` would have done. |
|
| `SWW_WAF_MODE` | `enforce` | `observe` refuses nothing and makes no ban; each log line says what `enforce` would have done. |
|
||||||
@@ -902,10 +903,14 @@ stops the running process (see [`docs/state-files.md`](docs/state-files.md)).
|
|||||||
- `internal/rproxy`: the reverse proxy in front of one app.
|
- `internal/rproxy`: the reverse proxy in front of one app.
|
||||||
- `internal/waf`: `waf`'s server, which runs the checks and answers its own
|
- `internal/waf`: `waf`'s server, which runs the checks and answers its own
|
||||||
endpoints.
|
endpoints.
|
||||||
|
- `internal/wafprotocol`: the question, the answer and the report that `rproxy`
|
||||||
|
and `waf` send each other.
|
||||||
- `internal/ratelimit`, `internal/bans`, `internal/rules`,
|
- `internal/ratelimit`, `internal/bans`, `internal/rules`,
|
||||||
`internal/coreruleset`, `internal/geojs`, `internal/lookup`,
|
`internal/coreruleset`, `internal/geojs`, `internal/lookup`,
|
||||||
`internal/reputation`, `internal/routewatch`, `internal/anomaly` and
|
`internal/reputation`, `internal/routewatch`, `internal/anomaly` and
|
||||||
`internal/alerts`: one check or source each.
|
`internal/alerts`: one check or source each.
|
||||||
|
- `internal/dirwatch`: watches a directory, so that `internal/rules` and
|
||||||
|
`internal/lookup` read their files again when they change.
|
||||||
- `internal/state`: the state and cache files.
|
- `internal/state`: the state and cache files.
|
||||||
- `internal/requestlog`: the request log.
|
- `internal/requestlog`: the request log.
|
||||||
- `internal/processlog`: the process's own messages.
|
- `internal/processlog`: the process's own messages.
|
||||||
|
|||||||
@@ -69,17 +69,21 @@ never as a time.
|
|||||||
that command's settings, and runs `internal/rproxy` until `SIGTERM` or
|
that command's settings, and runs `internal/rproxy` until `SIGTERM` or
|
||||||
`SIGINT` (see "`rproxy`"). `sww healthcheck` is the image's health check.
|
`SIGINT` (see "`rproxy`"). `sww healthcheck` is the image's health check.
|
||||||
With no command, or one it does not know, the binary prints the commands
|
With no command, or one it does not know, the binary prints the commands
|
||||||
and exits with status 2. `internal/sww` is the only package that imports
|
and exits with status 2. `sww help` and `sww --help` print them and exit
|
||||||
|
with status 0. `internal/sww` is the only package that imports
|
||||||
`go.uber.org/fx`. fx builds the parts of `waf` and `rproxy`, and starts
|
`go.uber.org/fx`. fx builds the parts of `waf` and `rproxy`, and starts
|
||||||
and stops their background work.
|
and stops their background work.
|
||||||
- `internal/config`: reads the settings, the one place they are read.
|
- `internal/config`: reads the settings, the one place they are read.
|
||||||
- `internal/waf`: `waf`'s server. It runs the checks in `check`, and holds
|
- `internal/waf`: `waf`'s server. It runs the checks in `check`. Its
|
||||||
`waf`'s question, answer and report. It answers `waf`'s own endpoints in
|
`routes.go` lists `waf`'s own endpoints. It counts bytes, refusals and
|
||||||
`answerOwnEndpoint`. It counts bytes, refusals and anomalies, and writes
|
anomalies, and writes the log lines.
|
||||||
the log lines.
|
|
||||||
- `internal/rproxy`: the reverse proxy in front of one app that asks `waf`
|
- `internal/rproxy`: the reverse proxy in front of one app that asks `waf`
|
||||||
about each request. It passes a request to the app with the standard
|
about each request. It passes a request to the app with the standard
|
||||||
library's `httputil.ReverseProxy`, within the time and size limits.
|
library's `httputil.ReverseProxy`, within the time and size limits.
|
||||||
|
- `internal/wafprotocol`: the question, the answer and the report that
|
||||||
|
`rproxy` and `waf` send each other (see "`waf`"). It writes and reads each
|
||||||
|
one, and declares every header, path and limit both sides use.
|
||||||
|
`internal/rproxy` imports it, not `internal/waf`.
|
||||||
- `internal/ratelimit`: the table of clients. It counts requests, bytes and
|
- `internal/ratelimit`: the table of clients. It counts requests, bytes and
|
||||||
refusals, and keeps each client's history.
|
refusals, and keeps each client's history.
|
||||||
- `internal/bans`: the bans. It holds each netblock's bans with their notes,
|
- `internal/bans`: the bans. It holds each netblock's bans with their notes,
|
||||||
@@ -93,6 +97,9 @@ never as a time.
|
|||||||
imports no other package of this repository: `waf` gives it what it needs
|
imports no other package of this repository: `waf` gives it what it needs
|
||||||
to count and report failures.
|
to count and report failures.
|
||||||
- `internal/lookup`: looks clients up in the lookup database.
|
- `internal/lookup`: looks clients up in the lookup database.
|
||||||
|
- `internal/dirwatch`: watches a directory, and calls a function once it has
|
||||||
|
gone a short time without a change. `internal/rules` and `internal/lookup`
|
||||||
|
read their files again this way.
|
||||||
- `internal/reputation`: fetches the lists, asks the DNS blocklists and
|
- `internal/reputation`: fetches the lists, asks the DNS blocklists and
|
||||||
AbuseIPDB in the background, and keeps their answers. It also fetches
|
AbuseIPDB in the background, and keeps their answers. It also fetches
|
||||||
AbuseIPDB's blacklist, which `internal/waf` bans.
|
AbuseIPDB's blacklist, which `internal/waf` bans.
|
||||||
@@ -121,11 +128,13 @@ never as a time.
|
|||||||
default `SWW_WAF_CORE_RULE_SET_DISABLED_RULES`, are built for that
|
default `SWW_WAF_CORE_RULE_SET_DISABLED_RULES`, are built for that
|
||||||
version. They are checked again before the version changes, since rule
|
version. They are checked again before the version changes, since rule
|
||||||
ids, and what each rule matches, change between versions;
|
ids, and what each rule matches, change between versions;
|
||||||
|
- `github.com/go-chi/chi/v5` to route the requests to `waf`'s own endpoints;
|
||||||
- `github.com/hashicorp/golang-lru/v2` for the bounded tables;
|
- `github.com/hashicorp/golang-lru/v2` for the bounded tables;
|
||||||
- `github.com/oschwald/maxminddb-golang/v2` to read the lookup database;
|
- `github.com/oschwald/maxminddb-golang/v2` to read the lookup database;
|
||||||
- `github.com/fsnotify/fsnotify` to notice edited files;
|
- `github.com/fsnotify/fsnotify` to notice edited files;
|
||||||
- `github.com/prometheus/client_golang` for the metrics;
|
- `github.com/prometheus/client_golang` for the metrics;
|
||||||
- `github.com/spf13/viper` to read the settings from the environment;
|
- `github.com/spf13/viper` to read the settings from the environment;
|
||||||
|
- `github.com/spf13/cobra` for the commands and their help text;
|
||||||
- `github.com/maxmind/mmdbwriter`, in the tests, to write lookup databases;
|
- `github.com/maxmind/mmdbwriter`, in the tests, to write lookup databases;
|
||||||
- remote log sending written in this project, since the standard library's
|
- remote log sending written in this project, since the standard library's
|
||||||
`log/syslog` is frozen and writes only the older syslog format.
|
`log/syslog` is frozen and writes only the older syslog format.
|
||||||
@@ -819,8 +828,8 @@ The supported apps are:
|
|||||||
- `vaultwarden`;
|
- `vaultwarden`;
|
||||||
- `wordpress`.
|
- `wordpress`.
|
||||||
|
|
||||||
Each app's fixes are in one file, such as `internal/coreruleset/gitea.conf`, and
|
Each app's fixes are in one file named for the app, such as
|
||||||
one entry of `apps` in `internal/coreruleset/coreruleset.go`. gitea's fixes are:
|
`internal/coreruleset/apps/gitea.conf`. gitea's fixes are:
|
||||||
|
|
||||||
1. Ten parameters are not checked against the Core Rule Set's lists of system
|
1. Ten parameters are not checked against the Core Rule Set's lists of system
|
||||||
files (930120), shell paths (932160) and command names (932260), on any path:
|
files (930120), shell paths (932160) and command names (932260), on any path:
|
||||||
@@ -853,8 +862,8 @@ one entry of `apps` in `internal/coreruleset/coreruleset.go`. gitea's fixes are:
|
|||||||
3. On gitea's paths, some fields skip the rules that refused ordinary gitea
|
3. On gitea's paths, some fields skip the rules that refused ordinary gitea
|
||||||
requests there. They are the fields where gitea sends a search, a file's name
|
requests there. They are the fields where gitea sends a search, a file's name
|
||||||
or text, a commit message, a branch name or a repository's name.
|
or text, a commit message, a branch name or a repository's name.
|
||||||
`internal/coreruleset/gitea.conf` lists each rule and the request it refused.
|
`internal/coreruleset/apps/gitea.conf` lists each rule and the request it
|
||||||
The fields are:
|
refused. The fields are:
|
||||||
- `q` on gitea's searches of code, of issues and pull requests, and of
|
- `q` on gitea's searches of code, of issues and pull requests, and of
|
||||||
commits. A code search holds code, such as `<script>`,
|
commits. A code search holds code, such as `<script>`,
|
||||||
`SELECT * FROM users WHERE` or `../`. The searches of issues and pull
|
`SELECT * FROM users WHERE` or `../`. The searches of issues and pull
|
||||||
@@ -1039,7 +1048,7 @@ one entry of `apps` in `internal/coreruleset/coreruleset.go`. gitea's fixes are:
|
|||||||
- The `Content-Range` header of an artifact upload from
|
- The `Content-Range` header of an artifact upload from
|
||||||
`actions/upload-artifact@v3` (920450).
|
`actions/upload-artifact@v3` (920450).
|
||||||
- The passwords in a body, for each rule that random passwords set off. The
|
- The passwords in a body, for each rule that random passwords set off. The
|
||||||
numbers are in `internal/coreruleset/gitea.conf`.
|
numbers are in `internal/coreruleset/apps/gitea.conf`.
|
||||||
- `password` on these paths:
|
- `password` on these paths:
|
||||||
- `/user/login`, `/user/sign_up`, `/user/activate`;
|
- `/user/login`, `/user/sign_up`, `/user/activate`;
|
||||||
- `/user/link_account_signin` and `/user/link_account_signup`;
|
- `/user/link_account_signin` and `/user/link_account_signup`;
|
||||||
@@ -1071,8 +1080,8 @@ one entry of `apps` in `internal/coreruleset/coreruleset.go`. gitea's fixes are:
|
|||||||
such as `owner/python3-app` would have that push refused.
|
such as `owner/python3-app` would have that push refused.
|
||||||
|
|
||||||
WordPress's fixes: on WordPress's paths, some fields skip the rules that refused
|
WordPress's fixes: on WordPress's paths, some fields skip the rules that refused
|
||||||
ordinary WordPress requests there. `internal/coreruleset/wordpress.conf` lists
|
ordinary WordPress requests there. `internal/coreruleset/apps/wordpress.conf`
|
||||||
each rule and the request it refused. The REST API has four addresses:
|
lists each rule and the request it refused. The REST API has four addresses:
|
||||||
|
|
||||||
- `/wp-json/...`
|
- `/wp-json/...`
|
||||||
- `/index.php/wp-json/...`
|
- `/index.php/wp-json/...`
|
||||||
@@ -1164,9 +1173,9 @@ Elsewhere WordPress's fixes do not apply. The paths are those of WordPress at
|
|||||||
the site root.
|
the site root.
|
||||||
|
|
||||||
Drupal's fixes: on Drupal's paths, some fields skip the rules that refused
|
Drupal's fixes: on Drupal's paths, some fields skip the rules that refused
|
||||||
ordinary Drupal requests there. `internal/coreruleset/drupal.conf` lists each
|
ordinary Drupal requests there. `internal/coreruleset/apps/drupal.conf` lists
|
||||||
rule and the request it refused. These query fields skip the lists of system
|
each rule and the request it refused. These query fields skip the lists of
|
||||||
files and command names (930120, 932120, 932250, 932260):
|
system files and command names (930120, 932120, 932250, 932260):
|
||||||
|
|
||||||
- On every page under `/admin`: `title`, `name`, `user`, `filename`, `filemime`,
|
- On every page under `/admin`: `title`, `name`, `user`, `filename`, `filemime`,
|
||||||
`info`, `subject`, `author_name`, `search`, `q` and `destination`.
|
`info`, `subject`, `author_name`, `search`, `q` and `destination`.
|
||||||
@@ -1256,10 +1265,10 @@ first path segment of two or three lowercase letters, optionally followed by a
|
|||||||
hyphen and more letters, so `/api` counts too.
|
hyphen and more letters, so `/api` counts too.
|
||||||
|
|
||||||
MediaWiki's fixes: on MediaWiki's paths, some fields skip the rules that refused
|
MediaWiki's fixes: on MediaWiki's paths, some fields skip the rules that refused
|
||||||
ordinary MediaWiki requests there. `internal/coreruleset/mediawiki.conf` lists
|
ordinary MediaWiki requests there. `internal/coreruleset/apps/mediawiki.conf`
|
||||||
each rule and the request it refused. These fields skip the lists of system
|
lists each rule and the request it refused. These fields skip the lists of
|
||||||
files and command names (930120, 932120, 932250, 932260), in the query or the
|
system files and command names (930120, 932120, 932250, 932260), in the query or
|
||||||
body:
|
the body:
|
||||||
|
|
||||||
- On MediaWiki's pages, which are `/index.php` and the paths under `/index.php/`
|
- On MediaWiki's pages, which are `/index.php` and the paths under `/index.php/`
|
||||||
and `/wiki/`:
|
and `/wiki/`:
|
||||||
@@ -1365,9 +1374,9 @@ Elsewhere these fields keep every rule. The paths are those of MediaWiki at the
|
|||||||
site root or under `/w/`.
|
site root or under `/w/`.
|
||||||
|
|
||||||
Mastodon's fixes: on Mastodon's paths, some fields skip the rules that refused
|
Mastodon's fixes: on Mastodon's paths, some fields skip the rules that refused
|
||||||
ordinary Mastodon requests there. `internal/coreruleset/mastodon.conf` lists
|
ordinary Mastodon requests there. `internal/coreruleset/apps/mastodon.conf`
|
||||||
each rule and the request it refused. These fields skip the lists of system
|
lists each rule and the request it refused. These fields skip the lists of
|
||||||
files and command names (930120, 932120, 932250, 932260), in the query:
|
system files and command names (930120, 932120, 932250, 932260), in the query:
|
||||||
|
|
||||||
- On the API's search, accounts and admin accounts, under `/api/v1/` and
|
- On the API's search, accounts and admin accounts, under `/api/v1/` and
|
||||||
`/api/v2/`, the searches and names `q`, `acct`, `username`, `display_name`,
|
`/api/v2/`, the searches and names `q`, `acct`, `username`, `display_name`,
|
||||||
@@ -1528,7 +1537,7 @@ every rule.
|
|||||||
|
|
||||||
Vaultwarden's fixes: on Vaultwarden's paths, some fields skip the rules that
|
Vaultwarden's fixes: on Vaultwarden's paths, some fields skip the rules that
|
||||||
refused ordinary Vaultwarden requests there.
|
refused ordinary Vaultwarden requests there.
|
||||||
`internal/coreruleset/vaultwarden.conf` lists each rule and the request it
|
`internal/coreruleset/apps/vaultwarden.conf` lists each rule and the request it
|
||||||
refused. The apps encrypt vault items, folders, collections and sends before
|
refused. The apps encrypt vault items, folders, collections and sends before
|
||||||
sending them, so those fields skip no rules. The fields below hold plain text.
|
sending them, so those fields skip no rules. The fields below hold plain text.
|
||||||
|
|
||||||
@@ -1579,8 +1588,8 @@ has a parameter of the same name.
|
|||||||
|
|
||||||
`token` in the body of `/admin`, the admin token typed on the admin page, skips
|
`token` in the body of `/admin`, the admin token typed on the admin page, skips
|
||||||
each rule that random passwords set off. The numbers are in
|
each rule that random passwords set off. The numbers are in
|
||||||
`internal/coreruleset/vaultwarden.conf`. It applies only when the query has no
|
`internal/coreruleset/apps/vaultwarden.conf`. It applies only when the query has
|
||||||
`token` parameter. `' OR '1'='1`, `../../etc/passwd`, `; cat /etc/passwd` and
|
no `token` parameter. `' OR '1'='1`, `../../etc/passwd`, `; cat /etc/passwd` and
|
||||||
`<img src=x onerror=x()>` sent as the token reach Vaultwarden;
|
`<img src=x onerror=x()>` sent as the token reach Vaultwarden;
|
||||||
`<script>alert(1)</script>` and `javascript:alert(1)` are still refused.
|
`<script>alert(1)</script>` and `javascript:alert(1)` are still refused.
|
||||||
|
|
||||||
@@ -1602,7 +1611,7 @@ Elsewhere these fields keep every rule.
|
|||||||
|
|
||||||
Nextcloud's fixes: on Nextcloud's paths, some fields skip the rules that refused
|
Nextcloud's fixes: on Nextcloud's paths, some fields skip the rules that refused
|
||||||
ordinary Nextcloud requests there, and its WebDAV methods are allowed.
|
ordinary Nextcloud requests there, and its WebDAV methods are allowed.
|
||||||
`internal/coreruleset/nextcloud.conf` lists each rule and the request it
|
`internal/coreruleset/apps/nextcloud.conf` lists each rule and the request it
|
||||||
refused.
|
refused.
|
||||||
|
|
||||||
The sync client and the Files app use WebDAV on these paths:
|
The sync client and the Files app use WebDAV on these paths:
|
||||||
@@ -1612,7 +1621,8 @@ The sync client and the Files app use WebDAV on these paths:
|
|||||||
- `/remote.php/webdav/`;
|
- `/remote.php/webdav/`;
|
||||||
- `/public.php/webdav/`.
|
- `/public.php/webdav/`.
|
||||||
|
|
||||||
These methods are allowed there, besides those of the first change:
|
These methods are allowed there, besides `PUT`, `PATCH` and `DELETE`, which
|
||||||
|
every app gets:
|
||||||
|
|
||||||
- `PROPFIND` and `PROPPATCH`;
|
- `PROPFIND` and `PROPPATCH`;
|
||||||
- `MKCOL`;
|
- `MKCOL`;
|
||||||
@@ -1756,7 +1766,7 @@ gets gitea's fixes too. Forgejo's own fixes hold only what Forgejo sends and
|
|||||||
gitea does not.
|
gitea does not.
|
||||||
|
|
||||||
On Forgejo's paths, some fields skip the rules that refused ordinary Forgejo
|
On Forgejo's paths, some fields skip the rules that refused ordinary Forgejo
|
||||||
requests there. `internal/coreruleset/forgejo.conf` lists each rule and the
|
requests there. `internal/coreruleset/apps/forgejo.conf` lists each rule and the
|
||||||
request it refused.
|
request it refused.
|
||||||
|
|
||||||
`q` skips rules on some of Forgejo's searches. The searches are:
|
`q` skips rules on some of Forgejo's searches. The searches are:
|
||||||
@@ -1802,10 +1812,10 @@ The Core Rule Set reads the request without the `flash` cookie, in which Forgejo
|
|||||||
puts what gitea puts in `gitea_flash` (gitea's second fix).
|
puts what gitea puts in `gitea_flash` (gitea's second fix).
|
||||||
|
|
||||||
Ghost's fixes: on Ghost's paths, some fields skip the rules that refused
|
Ghost's fixes: on Ghost's paths, some fields skip the rules that refused
|
||||||
ordinary Ghost requests there. `internal/coreruleset/ghost.conf` lists each rule
|
ordinary Ghost requests there. `internal/coreruleset/apps/ghost.conf` lists each
|
||||||
and the request it refused. Ghost's admin and integrations call the Admin API
|
rule and the request it refused. Ghost's admin and integrations call the Admin
|
||||||
under `/ghost/api/admin/`. Themes and Ghost's scripts call the Content API under
|
API under `/ghost/api/admin/`. Themes and Ghost's scripts call the Content API
|
||||||
`/ghost/api/content/`, and members call `/members/api/`. The admin calls
|
under `/ghost/api/content/`, and members call `/members/api/`. The admin calls
|
||||||
ActivityPub, a service of its own, under `/.ghost/activitypub/`.
|
ActivityPub, a service of its own, under `/.ghost/activitypub/`.
|
||||||
|
|
||||||
Eight rules refused words that name commands and files, such as `python3`,
|
Eight rules refused words that name commands and files, such as `python3`,
|
||||||
@@ -1943,9 +1953,9 @@ That name holds `.profile`, a file on the list of system files, so the name
|
|||||||
skips 930120 when a profile is saved.
|
skips 930120 when a profile is saved.
|
||||||
|
|
||||||
The passwords skip each rule that random passwords set off. The numbers are in
|
The passwords skip each rule that random passwords set off. The numbers are in
|
||||||
`internal/coreruleset/ghost.conf`. They are `password` when signing in, and the
|
`internal/coreruleset/apps/ghost.conf`. They are `password` when signing in, and
|
||||||
passwords sent when creating the first staff user, accepting an invitation, and
|
the passwords sent when creating the first staff user, accepting an invitation,
|
||||||
resetting or changing a password. `' OR '1'='1`, `../../etc/passwd`,
|
and resetting or changing a password. `' OR '1'='1`, `../../etc/passwd`,
|
||||||
`; cat /etc/passwd` and `<img src=x onerror=x()>` sent as a password reach
|
`; cat /etc/passwd` and `<img src=x onerror=x()>` sent as a password reach
|
||||||
Ghost; `<script>alert(1)</script>` and `javascript:alert(1)` are still refused.
|
Ghost; `<script>alert(1)</script>` and `javascript:alert(1)` are still refused.
|
||||||
|
|
||||||
@@ -2421,8 +2431,9 @@ but the last below carries `X-Sww-Disposition`.
|
|||||||
carries `X-Client-ASN` and `X-Client-Country`, which `rproxy` adds to the
|
carries `X-Client-ASN` and `X-Client-Country`, which `rproxy` adds to the
|
||||||
request it passes on. A header whose value is unknown is left out.
|
request it passes on. A header whose value is unknown is left out.
|
||||||
- `X-Sww-Disposition: deny`, with the status, headers and body the visitor is to
|
- `X-Sww-Disposition: deny`, with the status, headers and body the visitor is to
|
||||||
get. `rproxy` gives them to the visitor, less this header. The status is one
|
get. `rproxy` gives them to the visitor, less this header; a `403` or `429`
|
||||||
of:
|
gets `rproxy`'s own page as its body, linking `SWW_RPROXY_CONTACT_EMAIL` when
|
||||||
|
set. The status is one of:
|
||||||
- `SWW_WAF_BAN_RESPONSE`, for a refusal;
|
- `SWW_WAF_BAN_RESPONSE`, for a refusal;
|
||||||
- `403`, for a `block` rule or the Core Rule Set;
|
- `403`, for a `block` rule or the Core Rule Set;
|
||||||
- `414` or `431`, for a request over the header limits;
|
- `414` or `431`, for a request over the header limits;
|
||||||
|
|||||||
@@ -0,0 +1,217 @@
|
|||||||
|
services:
|
||||||
|
exampleapp:
|
||||||
|
# Your app, built from the Dockerfile in this directory, listening
|
||||||
|
# on port 8081. Publish no port of it: it is reached through rproxy.
|
||||||
|
build: .
|
||||||
|
|
||||||
|
rproxy:
|
||||||
|
image: git.eeqj.de/sneak/smallwebwaf:latest
|
||||||
|
environment:
|
||||||
|
SWW_RPROXY_WAF_BACKEND: http://waf:8080
|
||||||
|
SWW_RPROXY_UPSTREAM_URL: http://exampleapp:8081
|
||||||
|
|
||||||
|
# The address the refusal pages link to.
|
||||||
|
# SWW_RPROXY_CONTACT_EMAIL: ops@example.com
|
||||||
|
|
||||||
|
# The netblocks whose X-Forwarded-For rproxy believes. By default
|
||||||
|
# the private ranges, which cover a TLS-terminating proxy on the
|
||||||
|
# Docker network. A proxy elsewhere needs its address added; a
|
||||||
|
# list you give replaces the default, so repeat the private
|
||||||
|
# ranges if they are still wanted. Set but empty, rproxy believes
|
||||||
|
# no forwarded header.
|
||||||
|
# SWW_RPROXY_TRUSTED_PROXIES: 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,203.0.113.5,2001:db8:1::/64
|
||||||
|
|
||||||
|
# By default rproxy passes a request to the app while waf gives
|
||||||
|
# no answer. Uncomment this to answer 503 instead.
|
||||||
|
# SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION: deny
|
||||||
|
|
||||||
|
# The largest request body (default 100M) and response body
|
||||||
|
# (default 1G): a size with K, M or G, or `off`. Raise them for
|
||||||
|
# uploads and downloads larger than that, such as git pushes and
|
||||||
|
# clones or image layers.
|
||||||
|
# SWW_RPROXY_MAX_REQUEST_SIZE: 2G
|
||||||
|
# SWW_RPROXY_MAX_RESPONSE_SIZE: 4G
|
||||||
|
|
||||||
|
# How long a request may take to reach the app (default 60000,
|
||||||
|
# one minute) and how long its answer may take to reach the
|
||||||
|
# visitor (default 1800000, 30 minutes), in milliseconds or
|
||||||
|
# `off`. Raise the first for slow uploads, the second for long
|
||||||
|
# downloads.
|
||||||
|
# SWW_RPROXY_MAX_REQUEST_TIME_MS: "600000"
|
||||||
|
# SWW_RPROXY_MAX_RESPONSE_TIME_MS: "7200000"
|
||||||
|
|
||||||
|
# traefik terminates TLS and forwards to rproxy:8080 over its own
|
||||||
|
# network, which the `networks` block at the end of this file names.
|
||||||
|
# No port is published.
|
||||||
|
networks:
|
||||||
|
- default
|
||||||
|
- traefik
|
||||||
|
labels:
|
||||||
|
traefik.enable: "true"
|
||||||
|
# rproxy is on two networks; traefik reaches it on this one.
|
||||||
|
traefik.docker.network: traefik
|
||||||
|
traefik.http.routers.exampleapp.rule: Host(`app.example.com`)
|
||||||
|
traefik.http.routers.exampleapp.entrypoints: websecure
|
||||||
|
traefik.http.routers.exampleapp.tls.certresolver: letsencrypt
|
||||||
|
traefik.http.services.exampleapp.loadbalancer.server.port: "8080"
|
||||||
|
|
||||||
|
waf:
|
||||||
|
image: git.eeqj.de/sneak/smallwebwaf:latest
|
||||||
|
environment:
|
||||||
|
SWW_COMMAND: waf
|
||||||
|
|
||||||
|
# `observe` refuses nothing and makes no ban; each log line says
|
||||||
|
# what `enforce` (the default) would have done. Start here to
|
||||||
|
# see what the defaults would refuse on your site.
|
||||||
|
# SWW_WAF_MODE: observe
|
||||||
|
|
||||||
|
# waf's own messages below `info` (the default), while finding
|
||||||
|
# out why something is refused.
|
||||||
|
# SWW_WAF_LOG_LEVEL: debug
|
||||||
|
|
||||||
|
# Netblocks whose clients skip every check, such as monitoring.
|
||||||
|
# SWW_WAF_ALLOW_NETS: 192.0.2.10,2001:db8:2::/48
|
||||||
|
|
||||||
|
# Netblocks whose clients are refused.
|
||||||
|
# SWW_WAF_DENY_NETS: 198.51.100.128/25,2001:db8:3::/48
|
||||||
|
|
||||||
|
# Netblocks exempt from rate limiting and the byte limits, such
|
||||||
|
# as the office and an automation host that calls an API. Their
|
||||||
|
# requests are still checked for attacks.
|
||||||
|
# SWW_WAF_RATE_LIMIT_EXEMPT_NETS: 203.0.113.0/24,198.51.100.7
|
||||||
|
|
||||||
|
# Refuse every request from a country, by its two-letter code.
|
||||||
|
# This refuses North Korea, Cuba, Russia and Iran. A client the
|
||||||
|
# lookup cannot place gets through; see SWW_WAF_LOOKUP_SOURCE.
|
||||||
|
# SWW_WAF_TAG_DENY: country=KP,country=CU,country=RU,country=IR
|
||||||
|
|
||||||
|
# Or from a network, by its AS number: here two hosting networks
|
||||||
|
# that send only scrapers, M247 (AS9009) and WorldStream
|
||||||
|
# (AS49981). Countries and networks go in the one list together.
|
||||||
|
# SWW_WAF_TAG_DENY: asn=9009,asn=49981,country=KP,country=RU
|
||||||
|
|
||||||
|
# Or let only these countries through; a client the lookup
|
||||||
|
# cannot place is refused too.
|
||||||
|
# SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES: us,de
|
||||||
|
|
||||||
|
# Address lists that each tag their clients' requests, as
|
||||||
|
# name=URL pairs; this one tags Tor exit nodes. A tag is refused
|
||||||
|
# by adding its name to SWW_WAF_TAG_DENY, such as
|
||||||
|
# `torexit,country=KP`, or given a share of the limits in
|
||||||
|
# SWW_WAF_TAG_LIMIT_PERCENT, such as `torexit:25`.
|
||||||
|
# SWW_WAF_TAG_LISTS: torexit=https://check.torproject.org/torbulkexitlist
|
||||||
|
|
||||||
|
# A percentage, from 1 to 99, of every limit for a tag: here
|
||||||
|
# half for DigitalOcean (AS14061), a quarter for OVH (AS16276)
|
||||||
|
# and a quarter for China. The lowest of a client's tags applies.
|
||||||
|
# SWW_WAF_TAG_LIMIT_PERCENT: asn=14061:50,asn=16276:25,country=CN:25
|
||||||
|
|
||||||
|
# The most requests a client may make in a minute, an hour and a
|
||||||
|
# day (defaults 120, 1500 and 6000), a count or `off`; the next
|
||||||
|
# one bans it. Lower them for a small site, raise them for a
|
||||||
|
# busy API.
|
||||||
|
# SWW_WAF_RATE_LIMIT_PER_MINUTE: "60"
|
||||||
|
# SWW_WAF_RATE_LIMIT_PER_HOUR: "600"
|
||||||
|
# SWW_WAF_RATE_LIMIT_PER_DAY: "3000"
|
||||||
|
|
||||||
|
# Path prefixes exempt from rate limiting and the byte limits,
|
||||||
|
# such as static assets that one page loads many of.
|
||||||
|
# SWW_WAF_RATE_LIMIT_EXEMPT_PATHS: /assets/,/static/
|
||||||
|
|
||||||
|
# Paths only scanners ask for; one request bans the client. Pick
|
||||||
|
# paths your app never serves.
|
||||||
|
# SWW_WAF_TRAP_PATHS: /wp-login.php,/xmlrpc.php,/wp-admin/
|
||||||
|
|
||||||
|
# How a refused client is answered: 403 (the default), 429 or
|
||||||
|
# `close`, which drops the connection.
|
||||||
|
# SWW_WAF_BAN_RESPONSE: "429"
|
||||||
|
|
||||||
|
# The ban for a first broken limit (default 1h) and for a first
|
||||||
|
# clear sign of attack (default 7d): a duration with m, h or d.
|
||||||
|
# A repeat within SWW_WAF_LIMIT_BAN_REPEAT_WINDOW (default 24h)
|
||||||
|
# gets a ban three times as long.
|
||||||
|
# SWW_WAF_LIMIT_BAN_DURATION: 30m
|
||||||
|
# SWW_WAF_ATTACK_BAN_DURATION: 30d
|
||||||
|
|
||||||
|
# The Core Rule Set, which finds attacks in requests. `block`
|
||||||
|
# (the default) refuses a match, `detect` only logs and alerts
|
||||||
|
# it, `off` runs no rule. Use `detect` while finding out what it
|
||||||
|
# would refuse of your app's ordinary requests.
|
||||||
|
# SWW_WAF_CORE_RULE_SET_MODE: detect
|
||||||
|
|
||||||
|
# Path prefixes the Core Rule Set does not inspect, for a part of
|
||||||
|
# the app that it keeps matching, such as an editor that posts
|
||||||
|
# HTML or code.
|
||||||
|
# SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS: /api/editor/,/admin/
|
||||||
|
|
||||||
|
# The apps each hostname serves, as hostname:app pairs separated
|
||||||
|
# by semicolons, so that each hostname gets its own apps' fixes
|
||||||
|
# to the attack detection and no other's. "Attack detection" in
|
||||||
|
# the README lists the apps.
|
||||||
|
# SWW_WAF_APPS: git.example.com:gitea;blog.example.com:wordpress
|
||||||
|
|
||||||
|
# Blocklists of addresses and netblocks, text files fetched once
|
||||||
|
# a day, whose clients are refused.
|
||||||
|
# SWW_WAF_BLOCKLIST_URLS: https://www.spamhaus.org/drop/drop.txt,https://www.spamhaus.org/drop/dropv6.txt
|
||||||
|
|
||||||
|
# With the key of an AbuseIPDB account, a client that has
|
||||||
|
# committed an offence is checked there in the background; one
|
||||||
|
# scored 75 or more (SWW_WAF_ABUSEIPDB_MIN_SCORE) gets what
|
||||||
|
# SWW_WAF_DNSBL_ABUSEIPDB_ACTION says: a quarter of every limit
|
||||||
|
# by default, or `deny`. At most 900 checks a day
|
||||||
|
# (SWW_WAF_ABUSEIPDB_DAILY_BUDGET), under the free plan's 1000.
|
||||||
|
# SWW_WAF_ABUSEIPDB_KEY: api-key-.....
|
||||||
|
# SWW_WAF_DNSBL_ABUSEIPDB_ACTION: deny
|
||||||
|
|
||||||
|
# By default waf looks each client's address up at GeoJS, a web
|
||||||
|
# service, for its country and AS number. Set `off`, it makes no
|
||||||
|
# external requests for lookups. Every setting at its default
|
||||||
|
# works without them; one that needs a lookup, such as a country
|
||||||
|
# or ASN setting above, then stops the start. Set `file`, it
|
||||||
|
# reads an IPinfo Lite database instead, mounted by its
|
||||||
|
# directory as under `volumes` below, so that a replaced file is
|
||||||
|
# seen.
|
||||||
|
# SWW_WAF_LOOKUP_SOURCE: "off"
|
||||||
|
# SWW_WAF_LOOKUP_SOURCE: file
|
||||||
|
# SWW_WAF_LOOKUP_DB_PATH: /var/lib/ipinfo/ipinfo_lite.mmdb
|
||||||
|
|
||||||
|
# Alerts about bans, attacks and failures, to an ntfy topic, a
|
||||||
|
# Slack incoming webhook or any webhook. The ntfy token is the
|
||||||
|
# access token ntfy issued for the topic.
|
||||||
|
# SWW_WAF_ALERT_NTFY_URL: https://ntfy.sh/my-alerts
|
||||||
|
# SWW_WAF_ALERT_NTFY_TOKEN: hunter2
|
||||||
|
# SWW_WAF_ALERT_SLACK_WEBHOOK_URL: https://hooks.slack.com/services/...
|
||||||
|
|
||||||
|
# Bearer tokens you make up, at least 32 characters each. The
|
||||||
|
# admin token switches on the ban and client endpoints, through
|
||||||
|
# which a ban is lifted, your own included, as
|
||||||
|
# docs/admin-endpoints.md describes. The metrics token switches
|
||||||
|
# on /_sww/metrics.
|
||||||
|
# SWW_WAF_ADMIN_TOKEN: hunter2
|
||||||
|
# SWW_WAF_METRICS_TOKEN: hunter2
|
||||||
|
|
||||||
|
# Any setting can be read from a file instead, by adding _FILE
|
||||||
|
# to its name: the file's contents are the value, less one
|
||||||
|
# newline at the end. That keeps a secret out of this file and
|
||||||
|
# out of `docker inspect`. Mount the files as under `volumes`
|
||||||
|
# below, and set each setting in one form only.
|
||||||
|
# SWW_WAF_ALERT_NTFY_TOKEN_FILE: /etc/sww/tokens/ntfy
|
||||||
|
# SWW_WAF_ADMIN_TOKEN_FILE: /etc/sww/tokens/admin
|
||||||
|
# SWW_WAF_METRICS_TOKEN_FILE: /etc/sww/tokens/metrics
|
||||||
|
# SWW_WAF_ABUSEIPDB_KEY_FILE: /etc/sww/tokens/abuseipdb
|
||||||
|
volumes:
|
||||||
|
# waf's state files: its clients and bans, kept across restarts.
|
||||||
|
- waf-state:/var/lib/sww
|
||||||
|
# Token files, each owned by uid 65532 (the sww user) with mode
|
||||||
|
# 0400, for the _FILE settings above.
|
||||||
|
# - /srv/sww/tokens:/etc/sww/tokens:ro
|
||||||
|
# The IPinfo Lite database, for SWW_WAF_LOOKUP_SOURCE=file.
|
||||||
|
# - /srv/ipinfo:/var/lib/ipinfo:ro
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
waf-state:
|
||||||
|
|
||||||
|
# traefik's network, made outside this file.
|
||||||
|
networks:
|
||||||
|
traefik:
|
||||||
|
external: true
|
||||||
@@ -483,6 +483,8 @@ start, and an invalid value stops the start. Any can be given as a file, as a
|
|||||||
answer `rproxy` gives itself always has at least 10 seconds to be sent, and so
|
answer `rproxy` gives itself always has at least 10 seconds to be sent, and so
|
||||||
do the status and headers of the app's answer. A time in milliseconds or
|
do the status and headers of the app's answer. A time in milliseconds or
|
||||||
`off`.
|
`off`.
|
||||||
|
- `SWW_RPROXY_CONTACT_EMAIL` (default unset): the email address the refusal
|
||||||
|
pages link to.
|
||||||
- `SWW_RPROXY_LISTEN_ADDR` (default `:8080`): where `rproxy` listens, such as
|
- `SWW_RPROXY_LISTEN_ADDR` (default `:8080`): where `rproxy` listens, such as
|
||||||
`:9000`.
|
`:9000`.
|
||||||
- `SWW_RPROXY_UPSTREAM_URL` (default `http://127.0.0.1:8081`): the app. An
|
- `SWW_RPROXY_UPSTREAM_URL` (default `http://127.0.0.1:8081`): the app. An
|
||||||
|
|||||||
@@ -7,10 +7,12 @@ require (
|
|||||||
github.com/corazawaf/coraza-coreruleset/v4 v4.25.0
|
github.com/corazawaf/coraza-coreruleset/v4 v4.25.0
|
||||||
github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68
|
github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68
|
||||||
github.com/fsnotify/fsnotify v1.10.1
|
github.com/fsnotify/fsnotify v1.10.1
|
||||||
|
github.com/go-chi/chi/v5 v5.5.0
|
||||||
github.com/hashicorp/golang-lru/v2 v2.0.7
|
github.com/hashicorp/golang-lru/v2 v2.0.7
|
||||||
github.com/maxmind/mmdbwriter v1.2.0
|
github.com/maxmind/mmdbwriter v1.2.0
|
||||||
github.com/oschwald/maxminddb-golang/v2 v2.7.0
|
github.com/oschwald/maxminddb-golang/v2 v2.7.0
|
||||||
github.com/prometheus/client_golang v1.24.1
|
github.com/prometheus/client_golang v1.24.1
|
||||||
|
github.com/spf13/cobra v1.10.2
|
||||||
github.com/spf13/viper v1.21.0
|
github.com/spf13/viper v1.21.0
|
||||||
go.uber.org/fx v1.24.0
|
go.uber.org/fx v1.24.0
|
||||||
golang.org/x/net v0.60.0
|
golang.org/x/net v0.60.0
|
||||||
@@ -26,6 +28,7 @@ require (
|
|||||||
github.com/goccy/go-yaml v1.19.2 // indirect
|
github.com/goccy/go-yaml v1.19.2 // indirect
|
||||||
github.com/gotnospirit/makeplural v0.0.0-20180622080156-a5f48d94d976 // indirect
|
github.com/gotnospirit/makeplural v0.0.0-20180622080156-a5f48d94d976 // indirect
|
||||||
github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 // indirect
|
github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 // indirect
|
||||||
|
github.com/inconshreveable/mousetrap v1.1.0 // indirect
|
||||||
github.com/kaptinlin/go-i18n v0.1.4 // indirect
|
github.com/kaptinlin/go-i18n v0.1.4 // indirect
|
||||||
github.com/kaptinlin/jsonschema v0.4.6 // indirect
|
github.com/kaptinlin/jsonschema v0.4.6 // indirect
|
||||||
github.com/kylelemons/godebug v1.1.0 // indirect
|
github.com/kylelemons/godebug v1.1.0 // indirect
|
||||||
|
|||||||
@@ -14,12 +14,15 @@ github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68 h1:KLEjvBBSw
|
|||||||
github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68/go.mod h1:9XocblQEwK6EYylRth7PrPIifF2l0iSoYo4FAXhXU/8=
|
github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68/go.mod h1:9XocblQEwK6EYylRth7PrPIifF2l0iSoYo4FAXhXU/8=
|
||||||
github.com/corazawaf/libinjection-go v0.3.4 h1:jX2Tit2xZxYJJyVryNXucYBZtQqS7473x9Nc6lgRRbk=
|
github.com/corazawaf/libinjection-go v0.3.4 h1:jX2Tit2xZxYJJyVryNXucYBZtQqS7473x9Nc6lgRRbk=
|
||||||
github.com/corazawaf/libinjection-go v0.3.4/go.mod h1:Ik/+w3UmTWH9yn366RgS9D95K3y7Atb5m/H/gXzzPCk=
|
github.com/corazawaf/libinjection-go v0.3.4/go.mod h1:Ik/+w3UmTWH9yn366RgS9D95K3y7Atb5m/H/gXzzPCk=
|
||||||
|
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
|
||||||
github.com/foxcpp/go-mockdns v1.2.0 h1:omK3OrHRD1IWJz1FuFBCFquhXslXoF17OvBS6JPzZF0=
|
github.com/foxcpp/go-mockdns v1.2.0 h1:omK3OrHRD1IWJz1FuFBCFquhXslXoF17OvBS6JPzZF0=
|
||||||
github.com/foxcpp/go-mockdns v1.2.0/go.mod h1:IhLeSFGed3mJIAXPH2aiRQB+kqz7oqu8ld2qVbOu7Wk=
|
github.com/foxcpp/go-mockdns v1.2.0/go.mod h1:IhLeSFGed3mJIAXPH2aiRQB+kqz7oqu8ld2qVbOu7Wk=
|
||||||
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
|
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
|
||||||
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
|
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
|
||||||
github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho=
|
github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho=
|
||||||
github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo=
|
github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo=
|
||||||
|
github.com/go-chi/chi/v5 v5.5.0 h1:xdKQ1dJmVJsmgahrXSKhUx8NyVPzpwdlyrrQFpMPJgg=
|
||||||
|
github.com/go-chi/chi/v5 v5.5.0/go.mod h1:d1IG7GtKPSjPyBP365PH8sAGVHk3qfB/IhjFNRH1wWM=
|
||||||
github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
|
github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
|
||||||
github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
|
github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
|
||||||
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
|
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
|
||||||
@@ -34,6 +37,8 @@ github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092 h1:c7gcN
|
|||||||
github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092/go.mod h1:ZZAN4fkkful3l1lpJwF8JbW41ZiG9TwJ2ZlqzQovBNU=
|
github.com/gotnospirit/messageformat v0.0.0-20221001023931-dfe49f1eb092/go.mod h1:ZZAN4fkkful3l1lpJwF8JbW41ZiG9TwJ2ZlqzQovBNU=
|
||||||
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
|
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
|
||||||
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
|
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
|
||||||
|
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
|
||||||
|
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
|
||||||
github.com/jcchavezs/mergefs v0.1.1 h1:D45R17m6dHnSVZefnhynoeZvcK2Uw0oTrRfoUOQ0S5Y=
|
github.com/jcchavezs/mergefs v0.1.1 h1:D45R17m6dHnSVZefnhynoeZvcK2Uw0oTrRfoUOQ0S5Y=
|
||||||
github.com/jcchavezs/mergefs v0.1.1/go.mod h1:eRLTrsA+vFwQZ48hj8p8gki/5v9C2bFtHH5Mnn4bcGk=
|
github.com/jcchavezs/mergefs v0.1.1/go.mod h1:eRLTrsA+vFwQZ48hj8p8gki/5v9C2bFtHH5Mnn4bcGk=
|
||||||
github.com/kaptinlin/go-i18n v0.1.4 h1:wCiwAn1LOcvymvWIVAM4m5dUAMiHunTdEubLDk4hTGs=
|
github.com/kaptinlin/go-i18n v0.1.4 h1:wCiwAn1LOcvymvWIVAM4m5dUAMiHunTdEubLDk4hTGs=
|
||||||
@@ -72,6 +77,7 @@ github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+
|
|||||||
github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY=
|
github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY=
|
||||||
github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
|
github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
|
||||||
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
|
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
|
||||||
|
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
|
||||||
github.com/sagikazarmark/locafero v0.11.0 h1:1iurJgmM9G3PA/I+wWYIOw/5SyBtxapeHDcg+AAIFXc=
|
github.com/sagikazarmark/locafero v0.11.0 h1:1iurJgmM9G3PA/I+wWYIOw/5SyBtxapeHDcg+AAIFXc=
|
||||||
github.com/sagikazarmark/locafero v0.11.0/go.mod h1:nVIGvgyzw595SUSUE6tvCp3YYTeHs15MvlmU87WwIik=
|
github.com/sagikazarmark/locafero v0.11.0/go.mod h1:nVIGvgyzw595SUSUE6tvCp3YYTeHs15MvlmU87WwIik=
|
||||||
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 h1:+jumHNA0Wrelhe64i8F6HNlS8pkoyMv5sreGx2Ry5Rw=
|
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 h1:+jumHNA0Wrelhe64i8F6HNlS8pkoyMv5sreGx2Ry5Rw=
|
||||||
@@ -80,6 +86,9 @@ github.com/spf13/afero v1.15.0 h1:b/YBCLWAJdFWJTN9cLhiXXcD7mzKn9Dm86dNnfyQw1I=
|
|||||||
github.com/spf13/afero v1.15.0/go.mod h1:NC2ByUVxtQs4b3sIUphxK0NioZnmxgyCrfzeuq8lxMg=
|
github.com/spf13/afero v1.15.0/go.mod h1:NC2ByUVxtQs4b3sIUphxK0NioZnmxgyCrfzeuq8lxMg=
|
||||||
github.com/spf13/cast v1.10.0 h1:h2x0u2shc1QuLHfxi+cTJvs30+ZAHOGRic8uyGTDWxY=
|
github.com/spf13/cast v1.10.0 h1:h2x0u2shc1QuLHfxi+cTJvs30+ZAHOGRic8uyGTDWxY=
|
||||||
github.com/spf13/cast v1.10.0/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqeHo=
|
github.com/spf13/cast v1.10.0/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqeHo=
|
||||||
|
github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
|
||||||
|
github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
|
||||||
|
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
|
||||||
github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
|
github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
|
||||||
github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
|
github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
|
||||||
github.com/spf13/viper v1.21.0 h1:x5S+0EU27Lbphp4UKm1C+1oQO+rKx36vfCoaVebLFSU=
|
github.com/spf13/viper v1.21.0 h1:x5S+0EU27Lbphp4UKm1C+1oQO+rKx36vfCoaVebLFSU=
|
||||||
@@ -109,6 +118,7 @@ go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
|
|||||||
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
|
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
|
||||||
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
|
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
|
||||||
go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
|
go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
|
||||||
|
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
|
||||||
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
|
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
|
||||||
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
|
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
|
||||||
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba h1:0b9z3AuHCjxk0x/opv64kcgZLBseWJUpBw5I82+2U4M=
|
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba h1:0b9z3AuHCjxk0x/opv64kcgZLBseWJUpBw5I82+2U4M=
|
||||||
@@ -127,6 +137,7 @@ golang.org/x/tools v0.50.0 h1:c2ifzfcuY7L90lZ2aKd8S4K2NpASF08SZx9ZuJkHmSU=
|
|||||||
golang.org/x/tools v0.50.0/go.mod h1:7ulVMw3831Mwi5EZD6RomGyffr4VFjuNYXf2BbCEAV0=
|
golang.org/x/tools v0.50.0/go.mod h1:7ulVMw3831Mwi5EZD6RomGyffr4VFjuNYXf2BbCEAV0=
|
||||||
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
|
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
|
||||||
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
|
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
|
||||||
|
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||||
rsc.io/binaryregexp v0.2.0 h1:HfqmD5MEmC0zvwBuF187nq9mdnXjXsSivRiXN7SmRkE=
|
rsc.io/binaryregexp v0.2.0 h1:HfqmD5MEmC0zvwBuF187nq9mdnXjXsSivRiXN7SmRkE=
|
||||||
|
|||||||
+51
-38
@@ -34,65 +34,76 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// Event is what an alert is for, such as EventBan.
|
||||||
|
type Event string
|
||||||
|
|
||||||
// The events an alert is for, as SWW_WAF_ALERT_EVENTS names them.
|
// The events an alert is for, as SWW_WAF_ALERT_EVENTS names them.
|
||||||
const (
|
const (
|
||||||
// EventBan is a ban sww made.
|
// EventBan is a ban sww made.
|
||||||
EventBan = "ban"
|
EventBan Event = "ban"
|
||||||
// EventPermanentBan is a permanent ban sww made, or a ban it
|
// EventPermanentBan is a permanent ban sww made, or a ban it
|
||||||
// made permanent.
|
// made permanent.
|
||||||
EventPermanentBan = "permanent_ban"
|
EventPermanentBan Event = "permanent_ban"
|
||||||
// EventAnomaly is a count of requests or bytes over an anomaly
|
// EventAnomaly is a count of requests or bytes over an anomaly
|
||||||
// threshold.
|
// threshold.
|
||||||
EventAnomaly = "anomaly"
|
EventAnomaly Event = "anomaly"
|
||||||
// EventCoreRuleSetMatch is a request the Core Rule Set scored at or over
|
// EventCoreRuleSetMatch is a request the Core Rule Set scored at or over
|
||||||
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD, refused in block mode, let through in
|
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD, refused in block mode, let through in
|
||||||
// detect mode.
|
// detect mode.
|
||||||
EventCoreRuleSetMatch = "core_rule_set_match"
|
EventCoreRuleSetMatch Event = "core_rule_set_match"
|
||||||
// EventReputationHit is a request whose client a blocklist or a DNSBL
|
// EventReputationHit is a request whose client a blocklist or a DNSBL
|
||||||
// zone lists, or whose AbuseIPDB score is a hit.
|
// zone lists, or whose AbuseIPDB score is a hit.
|
||||||
EventReputationHit = "reputation_hit"
|
EventReputationHit Event = "reputation_hit"
|
||||||
// EventSourceFailure is GeoJS failing or refusing sww, a fetch
|
// EventSourceFailure is GeoJS failing or refusing sww, a fetch
|
||||||
// of a list failing, a query to a DNSBL zone or a check with AbuseIPDB
|
// of a list failing, a query to a DNSBL zone or a check with AbuseIPDB
|
||||||
// failing or refused, or the day's AbuseIPDB checks used up.
|
// failing or refused, or the day's AbuseIPDB checks used up.
|
||||||
EventSourceFailure = "source_failure"
|
EventSourceFailure Event = "source_failure"
|
||||||
// EventFileError is one of:
|
// EventFileError is one of:
|
||||||
// - a rule file or state file edited while sww runs that does not parse;
|
// - a rule file or state file edited while sww runs that does not parse;
|
||||||
// - a cache file that does not parse, at start or after an edit;
|
// - a cache file that does not parse, at start or after an edit;
|
||||||
// - a state file or cache file that cannot be written;
|
// - a state file or cache file that cannot be written;
|
||||||
// - a replacement of the lookup database that cannot be read.
|
// - a replacement of the lookup database that cannot be read.
|
||||||
EventFileError = "file_error"
|
EventFileError Event = "file_error"
|
||||||
// EventSummary is the summary sent as an hour ends: of the alerts held
|
// EventSummary is the summary sent as an hour ends: of the alerts held
|
||||||
// back in it past SWW_WAF_ALERT_MAX_PER_HOUR, and of the repeats held
|
// back in it past SWW_WAF_ALERT_MAX_PER_HOUR, and of the repeats held
|
||||||
// back by the cooldowns dropped as it ends, which no alert let through
|
// back by the cooldowns dropped as it ends, which no alert let through
|
||||||
// has given. It is sent with SWW_WAF_ALERT_MAX_PER_HOUR off too, for
|
// has given. It is sent with SWW_WAF_ALERT_MAX_PER_HOUR off too, for
|
||||||
// those repeats. SWW_WAF_ALERT_EVENTS does not name it.
|
// those repeats. SWW_WAF_ALERT_EVENTS does not name it.
|
||||||
EventSummary = "summary"
|
EventSummary Event = "summary"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Events returns every event SWW_WAF_ALERT_EVENTS can name, which is its
|
// Events returns every event SWW_WAF_ALERT_EVENTS can name, which is its
|
||||||
// default.
|
// default.
|
||||||
func Events() []string {
|
func Events() []Event {
|
||||||
return []string{
|
return []Event{
|
||||||
EventBan, EventPermanentBan, EventCoreRuleSetMatch, EventAnomaly,
|
EventBan, EventPermanentBan, EventCoreRuleSetMatch, EventAnomaly,
|
||||||
EventReputationHit, EventSourceFailure, EventFileError,
|
EventReputationHit, EventSourceFailure, EventFileError,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// DestinationName is the name of a destination alerts are sent to:
|
||||||
|
// DestinationWebhook, DestinationSlack or DestinationNtfy.
|
||||||
|
type DestinationName string
|
||||||
|
|
||||||
// The destinations alerts are sent to, as the metrics and alerts.json
|
// The destinations alerts are sent to, as the metrics and alerts.json
|
||||||
// name them.
|
// name them.
|
||||||
const (
|
const (
|
||||||
// DestinationWebhook is the webhook SWW_WAF_ALERT_WEBHOOK_URL names.
|
// DestinationWebhook is the webhook SWW_WAF_ALERT_WEBHOOK_URL names.
|
||||||
DestinationWebhook = "webhook"
|
DestinationWebhook DestinationName = "webhook"
|
||||||
// DestinationSlack is the Slack incoming webhook
|
// DestinationSlack is the Slack incoming webhook
|
||||||
// SWW_WAF_ALERT_SLACK_WEBHOOK_URL names.
|
// SWW_WAF_ALERT_SLACK_WEBHOOK_URL names.
|
||||||
DestinationSlack = "slack"
|
DestinationSlack DestinationName = "slack"
|
||||||
// DestinationNtfy is the ntfy topic SWW_WAF_ALERT_NTFY_URL names.
|
// DestinationNtfy is the ntfy topic SWW_WAF_ALERT_NTFY_URL names.
|
||||||
DestinationNtfy = "ntfy"
|
DestinationNtfy DestinationName = "ntfy"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Destinations returns every destination alerts can be sent to.
|
// Destinations returns every destination alerts can be sent to.
|
||||||
func Destinations() []string {
|
func Destinations() []DestinationName {
|
||||||
return []string{DestinationWebhook, DestinationSlack, DestinationNtfy}
|
return []DestinationName{
|
||||||
|
DestinationWebhook,
|
||||||
|
DestinationSlack,
|
||||||
|
DestinationNtfy,
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -135,7 +146,7 @@ type Params struct {
|
|||||||
NtfyURL *url.URL
|
NtfyURL *url.URL
|
||||||
NtfyToken string
|
NtfyToken string
|
||||||
// Events are the events alerts are sent for (SWW_WAF_ALERT_EVENTS).
|
// Events are the events alerts are sent for (SWW_WAF_ALERT_EVENTS).
|
||||||
Events []string
|
Events []Event
|
||||||
// Cooldown is how long a repeat of an alert is held back
|
// Cooldown is how long a repeat of an alert is held back
|
||||||
// (SWW_WAF_ALERT_COOLDOWN), 0 for no time. MaxPerHour is the most alerts
|
// (SWW_WAF_ALERT_COOLDOWN), 0 for no time. MaxPerHour is the most alerts
|
||||||
// sent in an hour (SWW_WAF_ALERT_MAX_PER_HOUR), 0 for no limit.
|
// sent in an hour (SWW_WAF_ALERT_MAX_PER_HOUR), 0 for no limit.
|
||||||
@@ -159,7 +170,7 @@ type Params struct {
|
|||||||
type Alert struct {
|
type Alert struct {
|
||||||
Instance string `json:"instance"`
|
Instance string `json:"instance"`
|
||||||
Time time.Time `json:"time"`
|
Time time.Time `json:"time"`
|
||||||
Event string `json:"event"`
|
Event Event `json:"event"`
|
||||||
Host string `json:"host,omitempty"`
|
Host string `json:"host,omitempty"`
|
||||||
Client netip.Addr `json:"client"`
|
Client netip.Addr `json:"client"`
|
||||||
Netblock netip.Prefix `json:"netblock"`
|
Netblock netip.Prefix `json:"netblock"`
|
||||||
@@ -186,7 +197,7 @@ type Alert struct {
|
|||||||
//
|
//
|
||||||
//nolint:tagliatelle // the state files use snake_case, as the request log does
|
//nolint:tagliatelle // the state files use snake_case, as the request log does
|
||||||
type Cooldown struct {
|
type Cooldown struct {
|
||||||
Event string `json:"event"`
|
Event Event `json:"event"`
|
||||||
Netblock netip.Prefix `json:"netblock"`
|
Netblock netip.Prefix `json:"netblock"`
|
||||||
File string `json:"file,omitempty"`
|
File string `json:"file,omitempty"`
|
||||||
Source string `json:"source,omitempty"`
|
Source string `json:"source,omitempty"`
|
||||||
@@ -205,7 +216,7 @@ type Cooldown struct {
|
|||||||
type Hour struct {
|
type Hour struct {
|
||||||
Start time.Time `json:"start"`
|
Start time.Time `json:"start"`
|
||||||
Sent int `json:"sent"`
|
Sent int `json:"sent"`
|
||||||
HeldBack map[string]int `json:"held_back"`
|
HeldBack map[Event]int `json:"held_back"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// State is what alerts.json holds: the cooldowns, the hour under way, and
|
// State is what alerts.json holds: the cooldowns, the hour under way, and
|
||||||
@@ -214,7 +225,7 @@ type Hour struct {
|
|||||||
type State struct {
|
type State struct {
|
||||||
Cooldowns []Cooldown `json:"cooldowns"`
|
Cooldowns []Cooldown `json:"cooldowns"`
|
||||||
Hour Hour `json:"hour"`
|
Hour Hour `json:"hour"`
|
||||||
Waiting map[string][]Alert `json:"waiting"`
|
Waiting map[DestinationName][]Alert `json:"waiting"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// Counts are, for a destination, how many alerts it took, how many
|
// Counts are, for a destination, how many alerts it took, how many
|
||||||
@@ -248,7 +259,7 @@ type destination struct {
|
|||||||
// name is how the metrics and alerts.json name the destination, and
|
// name is how the metrics and alerts.json name the destination, and
|
||||||
// setting the setting that is its URL, which the log names in place
|
// setting the setting that is its URL, which the log names in place
|
||||||
// of the URL.
|
// of the URL.
|
||||||
name string
|
name DestinationName
|
||||||
setting string
|
setting string
|
||||||
url *url.URL
|
url *url.URL
|
||||||
// message returns the body an alert is posted with, and the headers
|
// message returns the body an alert is posted with, and the headers
|
||||||
@@ -273,7 +284,7 @@ type destination struct {
|
|||||||
// scope with the same AS number or name, as its detail names them. Each
|
// scope with the same AS number or name, as its detail names them. Each
|
||||||
// is empty for an alert without one.
|
// is empty for an alert without one.
|
||||||
type cooldownKey struct {
|
type cooldownKey struct {
|
||||||
event string
|
event Event
|
||||||
netblock netip.Prefix
|
netblock netip.Prefix
|
||||||
file string
|
file string
|
||||||
source string
|
source string
|
||||||
@@ -306,7 +317,7 @@ func New(params Params) *Queue {
|
|||||||
q := &Queue{
|
q := &Queue{
|
||||||
params: params,
|
params: params,
|
||||||
cooldowns: map[cooldownKey]*Cooldown{},
|
cooldowns: map[cooldownKey]*Cooldown{},
|
||||||
hour: Hour{HeldBack: map[string]int{}},
|
hour: Hour{HeldBack: map[Event]int{}},
|
||||||
}
|
}
|
||||||
|
|
||||||
if params.WebhookURL != nil {
|
if params.WebhookURL != nil {
|
||||||
@@ -377,7 +388,7 @@ func (q *Queue) Raise(alert Alert) {
|
|||||||
// event, no alert for event on netblock was let through less than
|
// event, no alert for event on netblock was let through less than
|
||||||
// Cooldown before, and fewer than MaxPerHour alerts have been let through
|
// Cooldown before, and fewer than MaxPerHour alerts have been let through
|
||||||
// in the hour under way. Unlike Raise, it counts nothing.
|
// in the hour under way. Unlike Raise, it counts nothing.
|
||||||
func (q *Queue) WouldSend(event string, netblock netip.Prefix) bool {
|
func (q *Queue) WouldSend(event Event, netblock netip.Prefix) bool {
|
||||||
if len(q.destinations) == 0 || !slices.Contains(q.params.Events, event) {
|
if len(q.destinations) == 0 || !slices.Contains(q.params.Events, event) {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
@@ -433,7 +444,7 @@ func (q *Queue) Run(ctx context.Context) {
|
|||||||
|
|
||||||
// Counts returns the counts of the destination name, all 0 for one not
|
// Counts returns the counts of the destination name, all 0 for one not
|
||||||
// set.
|
// set.
|
||||||
func (q *Queue) Counts(name string) Counts {
|
func (q *Queue) Counts(name DestinationName) Counts {
|
||||||
for _, d := range q.destinations {
|
for _, d := range q.destinations {
|
||||||
if d.name == name {
|
if d.name == name {
|
||||||
return Counts{
|
return Counts{
|
||||||
@@ -449,8 +460,8 @@ func (q *Queue) Counts(name string) Counts {
|
|||||||
|
|
||||||
// DestinationsSet returns the destinations set, in the order of
|
// DestinationsSet returns the destinations set, in the order of
|
||||||
// Destinations.
|
// Destinations.
|
||||||
func (q *Queue) DestinationsSet() []string {
|
func (q *Queue) DestinationsSet() []DestinationName {
|
||||||
names := make([]string, 0, len(q.destinations))
|
names := make([]DestinationName, 0, len(q.destinations))
|
||||||
for _, d := range q.destinations {
|
for _, d := range q.destinations {
|
||||||
names = append(names, d.name)
|
names = append(names, d.name)
|
||||||
}
|
}
|
||||||
@@ -474,7 +485,7 @@ func (q *Queue) Snapshot() State {
|
|||||||
state := State{
|
state := State{
|
||||||
Cooldowns: make([]Cooldown, 0, len(q.cooldowns)),
|
Cooldowns: make([]Cooldown, 0, len(q.cooldowns)),
|
||||||
Hour: q.hour,
|
Hour: q.hour,
|
||||||
Waiting: map[string][]Alert{},
|
Waiting: map[DestinationName][]Alert{},
|
||||||
}
|
}
|
||||||
state.Hour.HeldBack = maps.Clone(q.hour.HeldBack)
|
state.Hour.HeldBack = maps.Clone(q.hour.HeldBack)
|
||||||
|
|
||||||
@@ -525,7 +536,7 @@ func (q *Queue) Load(state State) {
|
|||||||
q.hour.HeldBack = maps.Clone(state.Hour.HeldBack)
|
q.hour.HeldBack = maps.Clone(state.Hour.HeldBack)
|
||||||
|
|
||||||
if q.hour.HeldBack == nil {
|
if q.hour.HeldBack == nil {
|
||||||
q.hour.HeldBack = map[string]int{}
|
q.hour.HeldBack = map[Event]int{}
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, d := range q.destinations {
|
for _, d := range q.destinations {
|
||||||
@@ -537,7 +548,7 @@ func (q *Queue) Load(state State) {
|
|||||||
// its URL, that URL, target, and message, which makes the messages sent
|
// its URL, that URL, target, and message, which makes the messages sent
|
||||||
// to it.
|
// to it.
|
||||||
func (q *Queue) addDestination(
|
func (q *Queue) addDestination(
|
||||||
name, setting string, target *url.URL,
|
name DestinationName, setting string, target *url.URL,
|
||||||
message func(alert *Alert) ([]byte, http.Header, error),
|
message func(alert *Alert) ([]byte, http.Header, error),
|
||||||
) {
|
) {
|
||||||
q.destinations = append(q.destinations, &destination{
|
q.destinations = append(q.destinations, &destination{
|
||||||
@@ -649,7 +660,7 @@ func (q *Queue) endHour(now time.Time) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
q.hour = Hour{Start: start, HeldBack: map[string]int{}}
|
q.hour = Hour{Start: start, HeldBack: map[Event]int{}}
|
||||||
}
|
}
|
||||||
|
|
||||||
// queue adds alert to the alerts waiting for each destination.
|
// queue adds alert to the alerts waiting for each destination.
|
||||||
@@ -904,20 +915,21 @@ func (q *Queue) ntfyMessage(alert *Alert) ([]byte, http.Header, error) {
|
|||||||
|
|
||||||
// ntfyPriority returns the priority an alert for event is published to
|
// ntfyPriority returns the priority an alert for event is published to
|
||||||
// ntfy with: high for an event the admin needs to look at.
|
// ntfy with: high for an event the admin needs to look at.
|
||||||
func ntfyPriority(event string) string {
|
func ntfyPriority(event Event) string {
|
||||||
switch event {
|
switch event {
|
||||||
case EventPermanentBan, EventAnomaly, EventSourceFailure, EventFileError:
|
case EventPermanentBan, EventAnomaly, EventSourceFailure, EventFileError:
|
||||||
return "high"
|
return "high"
|
||||||
case EventReputationHit:
|
case EventReputationHit:
|
||||||
return "low"
|
return "low"
|
||||||
default: // ban, core_rule_set_match and summary
|
case EventBan, EventCoreRuleSetMatch, EventSummary:
|
||||||
return "default"
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
return "default"
|
||||||
}
|
}
|
||||||
|
|
||||||
// ntfyTag returns the tag an alert for event is published to ntfy with,
|
// ntfyTag returns the tag an alert for event is published to ntfy with,
|
||||||
// which ntfy shows as an emoji.
|
// which ntfy shows as an emoji.
|
||||||
func ntfyTag(event string) string {
|
func ntfyTag(event Event) string {
|
||||||
switch event {
|
switch event {
|
||||||
case EventBan, EventPermanentBan:
|
case EventBan, EventPermanentBan:
|
||||||
return "no_entry"
|
return "no_entry"
|
||||||
@@ -929,15 +941,16 @@ func ntfyTag(event string) string {
|
|||||||
return "label"
|
return "label"
|
||||||
case EventSourceFailure, EventFileError:
|
case EventSourceFailure, EventFileError:
|
||||||
return "warning"
|
return "warning"
|
||||||
default: // summary
|
case EventSummary:
|
||||||
return "bar_chart"
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
return "bar_chart"
|
||||||
}
|
}
|
||||||
|
|
||||||
// title returns the title of alert in Slack and ntfy: the instance and
|
// title returns the title of alert in Slack and ntfy: the instance and
|
||||||
// the event.
|
// the event.
|
||||||
func title(alert *Alert) string {
|
func title(alert *Alert) string {
|
||||||
return alert.Instance + ": " + alert.Event
|
return alert.Instance + ": " + string(alert.Event)
|
||||||
}
|
}
|
||||||
|
|
||||||
// text returns the text of alert in Slack and ntfy: its reason, then a
|
// text returns the text of alert in Slack and ntfy: its reason, then a
|
||||||
|
|||||||
@@ -126,7 +126,7 @@ func TestOnlyTheChosenEventsAreSent(t *testing.T) {
|
|||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
synctest.Test(t, func(t *testing.T) {
|
||||||
params := newParams()
|
params := newParams()
|
||||||
params.Events = []string{
|
params.Events = []alerts.Event{
|
||||||
alerts.EventSourceFailure,
|
alerts.EventSourceFailure,
|
||||||
alerts.EventFileError,
|
alerts.EventFileError,
|
||||||
}
|
}
|
||||||
@@ -156,7 +156,7 @@ func TestWouldSendOnlyForTheChosenEvents(t *testing.T) {
|
|||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
params := newParams()
|
params := newParams()
|
||||||
params.Events = []string{
|
params.Events = []alerts.Event{
|
||||||
alerts.EventSourceFailure,
|
alerts.EventSourceFailure,
|
||||||
alerts.EventFileError,
|
alerts.EventFileError,
|
||||||
}
|
}
|
||||||
@@ -186,8 +186,8 @@ func TestRaiseDoesNothingWithoutADestination(t *testing.T) {
|
|||||||
// No alert waits, no cooldown has started, and the hour counts none.
|
// No alert waits, no cooldown has started, and the hour counts none.
|
||||||
want := alerts.State{
|
want := alerts.State{
|
||||||
Cooldowns: []alerts.Cooldown{},
|
Cooldowns: []alerts.Cooldown{},
|
||||||
Hour: alerts.Hour{HeldBack: map[string]int{}},
|
Hour: alerts.Hour{HeldBack: map[alerts.Event]int{}},
|
||||||
Waiting: map[string][]alerts.Alert{},
|
Waiting: map[alerts.DestinationName][]alerts.Alert{},
|
||||||
}
|
}
|
||||||
if got := q.Snapshot(); !reflect.DeepEqual(got, want) {
|
if got := q.Snapshot(); !reflect.DeepEqual(got, want) {
|
||||||
t.Errorf("state %+v, want %+v", got, want)
|
t.Errorf("state %+v, want %+v", got, want)
|
||||||
@@ -235,7 +235,7 @@ func TestRepeatWithinTheCooldownIsHeldBackAndCountedInTheNext(t *testing.T) {
|
|||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
synctest.Test(t, func(t *testing.T) {
|
||||||
webhook, q := start(t, newParams())
|
webhook, q := start(t, newParams())
|
||||||
raise := func(event string, n int) {
|
raise := func(event alerts.Event, n int) {
|
||||||
q.Raise(alerts.Alert{Event: event, Netblock: netblock(n)})
|
q.Raise(alerts.Alert{Event: event, Netblock: netblock(n)})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1014,7 +1014,7 @@ func TestNtfyTokenIsSentToNtfyAlone(t *testing.T) {
|
|||||||
)
|
)
|
||||||
synctest.Wait()
|
synctest.Wait()
|
||||||
|
|
||||||
for destination, want := range map[string][]string{
|
for destination, want := range map[alerts.DestinationName][]string{
|
||||||
alerts.DestinationWebhook: nil,
|
alerts.DestinationWebhook: nil,
|
||||||
alerts.DestinationSlack: nil,
|
alerts.DestinationSlack: nil,
|
||||||
alerts.DestinationNtfy: tc.want,
|
alerts.DestinationNtfy: tc.want,
|
||||||
@@ -1063,7 +1063,7 @@ func TestDestinationThatDoesNotAnswerHoldsUpNeitherOther(t *testing.T) {
|
|||||||
|
|
||||||
// With no time passed, the webhook and ntfy have taken every alert,
|
// With no time passed, the webhook and ntfy have taken every alert,
|
||||||
// while Slack has not answered the first.
|
// while Slack has not answered the first.
|
||||||
for destination, want := range map[string]alerts.Counts{
|
for destination, want := range map[alerts.DestinationName]alerts.Counts{
|
||||||
alerts.DestinationWebhook: {Sent: 3},
|
alerts.DestinationWebhook: {Sent: 3},
|
||||||
alerts.DestinationSlack: {},
|
alerts.DestinationSlack: {},
|
||||||
alerts.DestinationNtfy: {Sent: 3},
|
alerts.DestinationNtfy: {Sent: 3},
|
||||||
@@ -1126,7 +1126,7 @@ func TestDropsAreCountedForEachDestination(t *testing.T) {
|
|||||||
// the webhook takes every one.
|
// the webhook takes every one.
|
||||||
const all = alerts.QueueSize + 1
|
const all = alerts.QueueSize + 1
|
||||||
|
|
||||||
for destination, want := range map[string]alerts.Counts{
|
for destination, want := range map[alerts.DestinationName]alerts.Counts{
|
||||||
alerts.DestinationWebhook: {Sent: all},
|
alerts.DestinationWebhook: {Sent: all},
|
||||||
alerts.DestinationSlack: {Dropped: 1},
|
alerts.DestinationSlack: {Dropped: 1},
|
||||||
alerts.DestinationNtfy: {SendFailures: all, Dropped: all},
|
alerts.DestinationNtfy: {SendFailures: all, Dropped: all},
|
||||||
@@ -1152,11 +1152,21 @@ func TestEachDestinationIsSentOnlyTheAlertsWaitingForIt(t *testing.T) {
|
|||||||
|
|
||||||
// As read from alerts.json. The alert waiting for the webhook, which
|
// As read from alerts.json. The alert waiting for the webhook, which
|
||||||
// is not set, is dropped.
|
// is not set, is dropped.
|
||||||
q.Load(roundTrip(t, alerts.State{Waiting: map[string][]alerts.Alert{
|
q.Load(
|
||||||
|
roundTrip(
|
||||||
|
t,
|
||||||
|
alerts.State{
|
||||||
|
Waiting: map[alerts.DestinationName][]alerts.Alert{
|
||||||
alerts.DestinationWebhook: {waiting("first")},
|
alerts.DestinationWebhook: {waiting("first")},
|
||||||
alerts.DestinationSlack: {waiting("second"), waiting("third")},
|
alerts.DestinationSlack: {
|
||||||
|
waiting("second"),
|
||||||
|
waiting("third"),
|
||||||
|
},
|
||||||
alerts.DestinationNtfy: {waiting("fourth")},
|
alerts.DestinationNtfy: {waiting("fourth")},
|
||||||
}}))
|
},
|
||||||
|
},
|
||||||
|
),
|
||||||
|
)
|
||||||
synctest.Wait()
|
synctest.Wait()
|
||||||
|
|
||||||
wantBodies(t, standIns[alerts.DestinationSlack],
|
wantBodies(t, standIns[alerts.DestinationSlack],
|
||||||
@@ -1165,7 +1175,7 @@ func TestEachDestinationIsSentOnlyTheAlertsWaitingForIt(t *testing.T) {
|
|||||||
wantBodies(t, standIns[alerts.DestinationNtfy], "fourth")
|
wantBodies(t, standIns[alerts.DestinationNtfy], "fourth")
|
||||||
|
|
||||||
// alerts.json then lists Slack and ntfy alone, with no alert waiting.
|
// alerts.json then lists Slack and ntfy alone, with no alert waiting.
|
||||||
want := map[string][]alerts.Alert{
|
want := map[alerts.DestinationName][]alerts.Alert{
|
||||||
alerts.DestinationSlack: {}, alerts.DestinationNtfy: {},
|
alerts.DestinationSlack: {}, alerts.DestinationNtfy: {},
|
||||||
}
|
}
|
||||||
if got := q.Snapshot().Waiting; !reflect.DeepEqual(got, want) {
|
if got := q.Snapshot().Waiting; !reflect.DeepEqual(got, want) {
|
||||||
@@ -1334,11 +1344,11 @@ func start(t *testing.T, params alerts.Params) (*standIn, *alerts.Queue) {
|
|||||||
func startAll(
|
func startAll(
|
||||||
t *testing.T,
|
t *testing.T,
|
||||||
params alerts.Params,
|
params alerts.Params,
|
||||||
) (map[string]*standIn, *alerts.Queue) {
|
) (map[alerts.DestinationName]*standIn, *alerts.Queue) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
q := alerts.New(params)
|
q := alerts.New(params)
|
||||||
standIns := map[string]*standIn{}
|
standIns := map[alerts.DestinationName]*standIn{}
|
||||||
|
|
||||||
for _, destination := range q.DestinationsSet() {
|
for _, destination := range q.DestinationsSet() {
|
||||||
standIns[destination] = &standIn{answers: answering}
|
standIns[destination] = &standIn{answers: answering}
|
||||||
@@ -1404,14 +1414,14 @@ func wantAlert(t *testing.T, got, want map[string]any) {
|
|||||||
|
|
||||||
// wantEvents checks the events of the alerts the webhook was sent, in
|
// wantEvents checks the events of the alerts the webhook was sent, in
|
||||||
// order.
|
// order.
|
||||||
func wantEvents(t *testing.T, webhook *standIn, want ...string) {
|
func wantEvents(t *testing.T, webhook *standIn, want ...alerts.Event) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
got := make([]string, 0, len(webhook.received()))
|
got := make([]alerts.Event, 0, len(webhook.received()))
|
||||||
|
|
||||||
for _, request := range webhook.received() {
|
for _, request := range webhook.received() {
|
||||||
event, _ := request.alert["event"].(string)
|
event, _ := request.alert["event"].(string)
|
||||||
got = append(got, event)
|
got = append(got, alerts.Event(event))
|
||||||
}
|
}
|
||||||
|
|
||||||
if !slices.Equal(got, want) {
|
if !slices.Equal(got, want) {
|
||||||
@@ -1447,7 +1457,8 @@ func wantCounts(
|
|||||||
|
|
||||||
// wantDestinationCounts checks q's counts for destination.
|
// wantDestinationCounts checks q's counts for destination.
|
||||||
func wantDestinationCounts(
|
func wantDestinationCounts(
|
||||||
t *testing.T, q *alerts.Queue, destination string, want alerts.Counts,
|
t *testing.T, q *alerts.Queue, destination alerts.DestinationName,
|
||||||
|
want alerts.Counts,
|
||||||
) {
|
) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
|
|||||||
@@ -7,7 +7,9 @@ const QueueSize = queueSize
|
|||||||
|
|
||||||
// SetTransport has q's requests to the destination name go through
|
// SetTransport has q's requests to the destination name go through
|
||||||
// transport instead of the network.
|
// transport instead of the network.
|
||||||
func (q *Queue) SetTransport(name string, transport http.RoundTripper) {
|
func (q *Queue) SetTransport(
|
||||||
|
name DestinationName, transport http.RoundTripper,
|
||||||
|
) {
|
||||||
for _, d := range q.destinations {
|
for _, d := range q.destinations {
|
||||||
if d.name == name {
|
if d.name == name {
|
||||||
d.httpClient.Transport = transport
|
d.httpClient.Transport = transport
|
||||||
|
|||||||
@@ -295,7 +295,8 @@ func (c *Counters) netAround(client netip.Addr) netip.Prefix {
|
|||||||
// overThreshold is a count over its threshold: what it counts, requests or
|
// overThreshold is a count over its threshold: what it counts, requests or
|
||||||
// bytes, its window, the count and the threshold.
|
// bytes, its window, the count and the threshold.
|
||||||
type overThreshold struct {
|
type overThreshold struct {
|
||||||
kind, window string
|
kind ratelimit.LimitKind
|
||||||
|
window string
|
||||||
count float64
|
count float64
|
||||||
threshold int64
|
threshold int64
|
||||||
}
|
}
|
||||||
@@ -346,7 +347,8 @@ func (c *Counter) add(
|
|||||||
// bucketCount is one of a counter's four counts: requests or bytes, in a
|
// bucketCount is one of a counter's four counts: requests or bytes, in a
|
||||||
// window of length, and the buckets they are counted in.
|
// window of length, and the buckets they are counted in.
|
||||||
type bucketCount struct {
|
type bucketCount struct {
|
||||||
kind, window string
|
kind ratelimit.LimitKind
|
||||||
|
window string
|
||||||
length time.Duration
|
length time.Duration
|
||||||
buckets *ratelimit.Buckets
|
buckets *ratelimit.Buckets
|
||||||
}
|
}
|
||||||
@@ -404,8 +406,8 @@ func (k key) counter() *Counter {
|
|||||||
// counted, the count and the threshold.
|
// counted, the count and the threshold.
|
||||||
func alertFor(r Request, k key, o overThreshold) alerts.Alert {
|
func alertFor(r Request, k key, o overThreshold) alerts.Alert {
|
||||||
detail := map[string]any{
|
detail := map[string]any{
|
||||||
"scope": k.scope, "window": o.window, "kind": o.kind, "count": o.count,
|
"scope": k.scope, "window": o.window, "kind": string(o.kind),
|
||||||
"threshold": o.threshold,
|
"count": o.count, "threshold": o.threshold,
|
||||||
}
|
}
|
||||||
|
|
||||||
var counted string
|
var counted string
|
||||||
|
|||||||
+21
-14
@@ -16,20 +16,26 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/hashicorp/golang-lru/v2/simplelru"
|
"github.com/hashicorp/golang-lru/v2/simplelru"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// Cause is why a ban was made: CauseLimit, CauseAttack, CauseAdmin or
|
||||||
|
// CauseAbuseIPDB.
|
||||||
|
type Cause string
|
||||||
|
|
||||||
// The causes of bans.
|
// The causes of bans.
|
||||||
const (
|
const (
|
||||||
// CauseLimit is a ban sww made for a broken limit.
|
// CauseLimit is a ban sww made for a broken limit.
|
||||||
CauseLimit = "limit"
|
CauseLimit Cause = "limit"
|
||||||
// CauseAttack is a ban sww made for a clear sign of attack.
|
// CauseAttack is a ban sww made for a clear sign of attack.
|
||||||
CauseAttack = "attack"
|
CauseAttack Cause = "attack"
|
||||||
// CauseAdmin is a ban an admin made, or one sww made that an
|
// CauseAdmin is a ban an admin made, or one sww made that an
|
||||||
// admin keeps. It is never dropped.
|
// admin keeps. It is never dropped.
|
||||||
CauseAdmin = "admin"
|
CauseAdmin Cause = "admin"
|
||||||
// CauseAbuseIPDB is a ban sww made for a client on AbuseIPDB's
|
// CauseAbuseIPDB is a ban sww made for a client on AbuseIPDB's
|
||||||
// blacklist. It is forgotten once it has ended.
|
// blacklist. It is forgotten once it has ended.
|
||||||
CauseAbuseIPDB = "abuseipdb"
|
CauseAbuseIPDB Cause = "abuseipdb"
|
||||||
)
|
)
|
||||||
|
|
||||||
// repeatFactor is how many times as long as the netblock's last ban a ban
|
// repeatFactor is how many times as long as the netblock's last ban a ban
|
||||||
@@ -68,8 +74,7 @@ type Ban struct {
|
|||||||
Start time.Time
|
Start time.Time
|
||||||
// Expires is when the ban ends, zero for a permanent ban.
|
// Expires is when the ban ends, zero for a permanent ban.
|
||||||
Expires time.Time
|
Expires time.Time
|
||||||
// Cause is CauseLimit, CauseAttack, CauseAdmin or CauseAbuseIPDB.
|
Cause Cause
|
||||||
Cause string
|
|
||||||
// Reason is a short text: for a ban sww made, the limit broken, the
|
// Reason is a short text: for a ban sww made, the limit broken, the
|
||||||
// rule that matched or AbuseIPDB's score; for an admin's, what the
|
// rule that matched or AbuseIPDB's score; for an admin's, what the
|
||||||
// admin wrote.
|
// admin wrote.
|
||||||
@@ -111,7 +116,7 @@ type Notes struct {
|
|||||||
// request that broke the limit included.
|
// request that broke the limit included.
|
||||||
// These are what counted toward the ban, and the window is the time
|
// These are what counted toward the ban, and the window is the time
|
||||||
// over which they came.
|
// over which they came.
|
||||||
Kind string `json:"kind,omitempty"`
|
Kind ratelimit.LimitKind `json:"kind,omitempty"`
|
||||||
Limit int64 `json:"limit,omitempty"`
|
Limit int64 `json:"limit,omitempty"`
|
||||||
Window string `json:"window,omitempty"`
|
Window string `json:"window,omitempty"`
|
||||||
Count float64 `json:"count,omitempty"`
|
Count float64 `json:"count,omitempty"`
|
||||||
@@ -194,7 +199,7 @@ type Ledger struct {
|
|||||||
// made is how many bans have been made since the start, by cause: by
|
// made is how many bans have been made since the start, by cause: by
|
||||||
// the ledger, and by an admin, through BanForAdmin or in an edit of
|
// the ledger, and by an admin, through BanForAdmin or in an edit of
|
||||||
// bans.json.
|
// bans.json.
|
||||||
made map[string]int
|
made map[Cause]int
|
||||||
// v4Lengths and v6Lengths are the lengths of the IPv4 and IPv6
|
// v4Lengths and v6Lengths are the lengths of the IPv4 and IPv6
|
||||||
// netblocks that have been banned. Check looks for a ban at each of
|
// netblocks that have been banned. Check looks for a ban at each of
|
||||||
// them, so that a ban read from bans.json refuses every client in its
|
// them, so that a ban read from bans.json refuses every client in its
|
||||||
@@ -221,7 +226,7 @@ func New(params Params) *Ledger {
|
|||||||
params: params,
|
params: params,
|
||||||
changed: make(chan struct{}, 1),
|
changed: make(chan struct{}, 1),
|
||||||
netblocks: netblocks,
|
netblocks: netblocks,
|
||||||
made: map[string]int{},
|
made: map[Cause]int{},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -303,7 +308,7 @@ func activeBan(bans []Ban, now time.Time) *Ban {
|
|||||||
// Grounds are why a request makes a ban: its cause, CauseLimit or
|
// Grounds are why a request makes a ban: its cause, CauseLimit or
|
||||||
// CauseAttack, and its notes.
|
// CauseAttack, and its notes.
|
||||||
type Grounds struct {
|
type Grounds struct {
|
||||||
Cause string
|
Cause Cause
|
||||||
Notes Notes
|
Notes Notes
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -345,7 +350,7 @@ func (l *Ledger) WouldBan(
|
|||||||
// or CauseAttack, made at now would be permanent, as Ban would make it. It
|
// or CauseAttack, made at now would be permanent, as Ban would make it. It
|
||||||
// works out nothing else of the ban.
|
// works out nothing else of the ban.
|
||||||
func (l *Ledger) WouldBePermanent(
|
func (l *Ledger) WouldBePermanent(
|
||||||
netblock netip.Prefix, now time.Time, cause string,
|
netblock netip.Prefix, now time.Time, cause Cause,
|
||||||
) bool {
|
) bool {
|
||||||
l.mu.Lock()
|
l.mu.Lock()
|
||||||
defer l.mu.Unlock()
|
defer l.mu.Unlock()
|
||||||
@@ -439,7 +444,7 @@ func (l *Ledger) BanForAbuseIPDB(
|
|||||||
|
|
||||||
// The source is named as the request log names AbuseIPDB.
|
// The source is named as the request log names AbuseIPDB.
|
||||||
notes.Reputation = []ReputationHit{
|
notes.Reputation = []ReputationHit{
|
||||||
{Source: CauseAbuseIPDB, Score: &score},
|
{Source: string(CauseAbuseIPDB), Score: &score},
|
||||||
}
|
}
|
||||||
|
|
||||||
l.add(Ban{
|
l.add(Ban{
|
||||||
@@ -577,7 +582,7 @@ func (l *Ledger) AddLookup(
|
|||||||
// for CauseLimit, CauseAttack and CauseAbuseIPDB, by the ledger; for
|
// for CauseLimit, CauseAttack and CauseAbuseIPDB, by the ledger; for
|
||||||
// CauseAdmin, by an admin, with BanForAdmin or in an edit of bans.json, as
|
// CauseAdmin, by an admin, with BanForAdmin or in an edit of bans.json, as
|
||||||
// LoadEdit counts them. The bans read from bans.json at the start are not among them.
|
// LoadEdit counts them. The bans read from bans.json at the start are not among them.
|
||||||
func (l *Ledger) Made(cause string) int {
|
func (l *Ledger) Made(cause Cause) int {
|
||||||
l.mu.Lock()
|
l.mu.Lock()
|
||||||
defer l.mu.Unlock()
|
defer l.mu.Unlock()
|
||||||
|
|
||||||
@@ -770,8 +775,10 @@ func (l *Ledger) ban(
|
|||||||
case CauseAttack:
|
case CauseAttack:
|
||||||
ban.Reason = attackReason(notes)
|
ban.Reason = attackReason(notes)
|
||||||
ban.Expires = l.attackExpiry(held, now)
|
ban.Expires = l.attackExpiry(held, now)
|
||||||
|
case CauseAdmin, CauseAbuseIPDB:
|
||||||
|
fallthrough
|
||||||
default:
|
default:
|
||||||
panic("bans: a request's ban has the cause " + grounds.Cause)
|
panic("bans: a request's ban has the cause " + string(grounds.Cause))
|
||||||
}
|
}
|
||||||
|
|
||||||
if !keep {
|
if !keep {
|
||||||
|
|||||||
@@ -10,6 +10,8 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/spf13/viper"
|
"github.com/spf13/viper"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
|
||||||
)
|
)
|
||||||
|
|
||||||
// RProxy is the settings of `sww rproxy`. A size or time limit of zero
|
// RProxy is the settings of `sww rproxy`. A size or time limit of zero
|
||||||
@@ -32,7 +34,7 @@ type RProxy struct {
|
|||||||
// cannot be connected to, does not answer at all, or does not answer
|
// cannot be connected to, does not answer at all, or does not answer
|
||||||
// within WAFBackendTimeout: allow or deny
|
// within WAFBackendTimeout: allow or deny
|
||||||
// (SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION).
|
// (SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION).
|
||||||
WAFBackendNoAnswerDisposition string
|
WAFBackendNoAnswerDisposition wafprotocol.Disposition
|
||||||
// MaxRequestSize is the largest request body (SWW_RPROXY_MAX_REQUEST_SIZE), and
|
// MaxRequestSize is the largest request body (SWW_RPROXY_MAX_REQUEST_SIZE), and
|
||||||
// MaxResponseSize the largest response body (SWW_RPROXY_MAX_RESPONSE_SIZE).
|
// MaxResponseSize the largest response body (SWW_RPROXY_MAX_RESPONSE_SIZE).
|
||||||
MaxRequestSize int64
|
MaxRequestSize int64
|
||||||
@@ -44,6 +46,9 @@ type RProxy struct {
|
|||||||
// (SWW_RPROXY_MAX_RESPONSE_TIME_MS).
|
// (SWW_RPROXY_MAX_RESPONSE_TIME_MS).
|
||||||
MaxRequestTime time.Duration
|
MaxRequestTime time.Duration
|
||||||
MaxResponseTime time.Duration
|
MaxResponseTime time.Duration
|
||||||
|
// ContactEmail is the address the pages a refused visitor gets link to,
|
||||||
|
// or "" for no link (SWW_RPROXY_CONTACT_EMAIL). It is taken as given.
|
||||||
|
ContactEmail string
|
||||||
|
|
||||||
// settings are the values read, as given or by default, for the log line
|
// settings are the values read, as given or by default, for the log line
|
||||||
// at start.
|
// at start.
|
||||||
@@ -94,7 +99,7 @@ func RProxyFromEnvironment(v *viper.Viper) (*RProxy, error) {
|
|||||||
),
|
),
|
||||||
WAFBackendNoAnswerDisposition: env.disposition(
|
WAFBackendNoAnswerDisposition: env.disposition(
|
||||||
"SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION",
|
"SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION",
|
||||||
"allow",
|
wafprotocol.DispositionAllow,
|
||||||
),
|
),
|
||||||
MaxRequestSize: env.size("SWW_RPROXY_MAX_REQUEST_SIZE", "100M"),
|
MaxRequestSize: env.size("SWW_RPROXY_MAX_REQUEST_SIZE", "100M"),
|
||||||
MaxResponseSize: env.size("SWW_RPROXY_MAX_RESPONSE_SIZE", "1G"),
|
MaxResponseSize: env.size("SWW_RPROXY_MAX_RESPONSE_SIZE", "1G"),
|
||||||
@@ -106,6 +111,7 @@ func RProxyFromEnvironment(v *viper.Viper) (*RProxy, error) {
|
|||||||
"SWW_RPROXY_MAX_RESPONSE_TIME_MS",
|
"SWW_RPROXY_MAX_RESPONSE_TIME_MS",
|
||||||
"1800000",
|
"1800000",
|
||||||
),
|
),
|
||||||
|
ContactEmail: env.value("SWW_RPROXY_CONTACT_EMAIL", ""),
|
||||||
}
|
}
|
||||||
|
|
||||||
if env.err != nil {
|
if env.err != nil {
|
||||||
@@ -166,9 +172,12 @@ func (e *environment) wafBackend(name string) *url.URL {
|
|||||||
// disposition reads the setting that is what is done with a request when waf
|
// disposition reads the setting that is what is done with a request when waf
|
||||||
// cannot be connected to, does not answer at all, or does not answer in time:
|
// cannot be connected to, does not answer at all, or does not answer in time:
|
||||||
// allow or deny.
|
// allow or deny.
|
||||||
func (e *environment) disposition(name, defaultValue string) string {
|
func (e *environment) disposition(
|
||||||
value := e.value(name, defaultValue)
|
name string, defaultValue wafprotocol.Disposition,
|
||||||
if value != "allow" && value != "deny" {
|
) wafprotocol.Disposition {
|
||||||
|
value := wafprotocol.Disposition(e.value(name, string(defaultValue)))
|
||||||
|
if value != wafprotocol.DispositionAllow &&
|
||||||
|
value != wafprotocol.DispositionDeny {
|
||||||
e.check(name, fmt.Errorf("%q %w", value, errNotDisposition))
|
e.check(name, fmt.Errorf("%q %w", value, errNotDisposition))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -25,6 +25,7 @@ const (
|
|||||||
rproxyMaxResponse = "SWW_RPROXY_MAX_RESPONSE_SIZE"
|
rproxyMaxResponse = "SWW_RPROXY_MAX_RESPONSE_SIZE"
|
||||||
rproxyRequestTime = "SWW_RPROXY_MAX_REQUEST_TIME_MS"
|
rproxyRequestTime = "SWW_RPROXY_MAX_REQUEST_TIME_MS"
|
||||||
rproxyAnswerTime = "SWW_RPROXY_MAX_RESPONSE_TIME_MS"
|
rproxyAnswerTime = "SWW_RPROXY_MAX_RESPONSE_TIME_MS"
|
||||||
|
rproxyContactEmail = "SWW_RPROXY_CONTACT_EMAIL"
|
||||||
backendURL = "http://waf:8080"
|
backendURL = "http://waf:8080"
|
||||||
rproxyNoAnswerAddr = ":8080"
|
rproxyNoAnswerAddr = ":8080"
|
||||||
rproxyNoAnswerApp = "http://127.0.0.1:8081"
|
rproxyNoAnswerApp = "http://127.0.0.1:8081"
|
||||||
@@ -250,6 +251,7 @@ func TestRProxySettingsAreLoggedByName(t *testing.T) {
|
|||||||
rproxyListen, rproxyUpstream, rproxyTrusted, wafBackend, wafBackendTimeout,
|
rproxyListen, rproxyUpstream, rproxyTrusted, wafBackend, wafBackendTimeout,
|
||||||
rproxyNoAnswer,
|
rproxyNoAnswer,
|
||||||
rproxyMaxRequest, rproxyMaxResponse, rproxyRequestTime, rproxyAnswerTime,
|
rproxyMaxRequest, rproxyMaxResponse, rproxyRequestTime, rproxyAnswerTime,
|
||||||
|
rproxyContactEmail,
|
||||||
} {
|
} {
|
||||||
if !slices.Contains(names, name) {
|
if !slices.Contains(names, name) {
|
||||||
t.Errorf("%s is not logged, only %v", name, names)
|
t.Errorf("%s is not logged, only %v", name, names)
|
||||||
|
|||||||
+1631
-1445
File diff suppressed because it is too large.
Load diff
@@ -20,6 +20,7 @@ import (
|
|||||||
"github.com/spf13/viper"
|
"github.com/spf13/viper"
|
||||||
"golang.org/x/net/http/httpguts"
|
"golang.org/x/net/http/httpguts"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
|
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
|
||||||
@@ -188,6 +189,16 @@ const (
|
|||||||
defaultAlertCooldown = "15m"
|
defaultAlertCooldown = "15m"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// joinEvents returns events as SWW_WAF_ALERT_EVENTS lists them.
|
||||||
|
func joinEvents(events []alerts.Event) string {
|
||||||
|
names := make([]string, 0, len(events))
|
||||||
|
for _, event := range events {
|
||||||
|
names = append(names, string(event))
|
||||||
|
}
|
||||||
|
|
||||||
|
return strings.Join(names, ",")
|
||||||
|
}
|
||||||
|
|
||||||
// defaultIPReputationCacheTTL is the default of SWW_WAF_IP_REPUTATION_CACHE_TTL.
|
// defaultIPReputationCacheTTL is the default of SWW_WAF_IP_REPUTATION_CACHE_TTL.
|
||||||
const defaultIPReputationCacheTTL = "24h"
|
const defaultIPReputationCacheTTL = "24h"
|
||||||
|
|
||||||
@@ -1110,7 +1121,7 @@ func TestAlertSettingsDefaults(t *testing.T) {
|
|||||||
if cfg.AlertWebhookURL != nil || len(cfg.AlertWebhookHeaders) != 0 ||
|
if cfg.AlertWebhookURL != nil || len(cfg.AlertWebhookHeaders) != 0 ||
|
||||||
cfg.AlertSlackWebhookURL != nil || cfg.AlertNtfyURL != nil ||
|
cfg.AlertSlackWebhookURL != nil || cfg.AlertNtfyURL != nil ||
|
||||||
cfg.AlertNtfyToken != "" ||
|
cfg.AlertNtfyToken != "" ||
|
||||||
strings.Join(cfg.AlertEvents, ",") != defaultAlertEvents ||
|
joinEvents(cfg.AlertEvents) != defaultAlertEvents ||
|
||||||
cfg.AlertCooldown != 15*time.Minute || cfg.AlertMaxPerHour != 60 {
|
cfg.AlertCooldown != 15*time.Minute || cfg.AlertMaxPerHour != 60 {
|
||||||
t.Errorf(
|
t.Errorf(
|
||||||
"alert settings %v, %v, %v, %v, %q, %v, %s and %d, want no URLs, "+
|
"alert settings %v, %v, %v, %v, %q, %v, %s and %d, want no URLs, "+
|
||||||
@@ -1157,7 +1168,10 @@ func TestAlertSettingsAsSet(t *testing.T) {
|
|||||||
!reflect.DeepEqual(cfg.AlertWebhookHeaders, headers) ||
|
!reflect.DeepEqual(cfg.AlertWebhookHeaders, headers) ||
|
||||||
cfg.AlertSlackWebhookURL.String() != slack || cfg.AlertNtfyURL.String() != ntfy ||
|
cfg.AlertSlackWebhookURL.String() != slack || cfg.AlertNtfyURL.String() != ntfy ||
|
||||||
cfg.AlertNtfyToken != token ||
|
cfg.AlertNtfyToken != token ||
|
||||||
!slices.Equal(cfg.AlertEvents, []string{"ban", "file_error"}) ||
|
!slices.Equal(
|
||||||
|
cfg.AlertEvents,
|
||||||
|
[]alerts.Event{"ban", "file_error"},
|
||||||
|
) ||
|
||||||
cfg.AlertCooldown != time.Hour || cfg.AlertMaxPerHour != 10 {
|
cfg.AlertCooldown != time.Hour || cfg.AlertMaxPerHour != 10 {
|
||||||
t.Errorf(
|
t.Errorf(
|
||||||
"alert settings %v, %v, %v, %v, %q, %v, %s and %d",
|
"alert settings %v, %v, %v, %v, %q, %v, %s and %d",
|
||||||
|
|||||||
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
@@ -37,9 +37,10 @@
|
|||||||
SecRule &TX:app_nextcloud "@eq 0" \
|
SecRule &TX:app_nextcloud "@eq 0" \
|
||||||
"id:900090,phase:1,pass,nolog,skipAfter:END_NEXTCLOUD"
|
"id:900090,phase:1,pass,nolog,skipAfter:END_NEXTCLOUD"
|
||||||
|
|
||||||
# The sync client and the Files app use these WebDAV methods besides those
|
# The sync client and the Files app use these WebDAV methods besides PUT,
|
||||||
# the first change allows, on /remote.php/dav/ and /public.php/dav/ and on the
|
# PATCH and DELETE, which sww allows for every app, on /remote.php/dav/ and
|
||||||
# older /remote.php/webdav/ and /public.php/webdav/. 911100 refused each:
|
# /public.php/dav/ and on the older /remote.php/webdav/ and /public.php/webdav/.
|
||||||
|
# 911100 refused each:
|
||||||
# PROPFIND, PROPPATCH: listing a folder, marking a file a favourite
|
# PROPFIND, PROPPATCH: listing a folder, marking a file a favourite
|
||||||
# MKCOL: making a folder, and starting a chunked upload
|
# MKCOL: making a folder, and starting a chunked upload
|
||||||
# COPY, MOVE: copying, moving and renaming, and ending a chunked upload
|
# COPY, MOVE: copying, moving and renaming, and ending a chunked upload
|
||||||
File renamed without changes.
File renamed without changes.
@@ -12,11 +12,10 @@
|
|||||||
package coreruleset
|
package coreruleset
|
||||||
|
|
||||||
import (
|
import (
|
||||||
_ "embed"
|
"embed"
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"maps"
|
|
||||||
"net"
|
"net"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
@@ -35,7 +34,7 @@ import (
|
|||||||
// directives are the Core Rule Set as sww runs it, with the
|
// directives are the Core Rule Set as sww runs it, with the
|
||||||
// paranoia level for %d, bodyDirectives for the first %s while
|
// paranoia level for %d, bodyDirectives for the first %s while
|
||||||
// SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is set, the exclusions file for
|
// SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is set, the exclusions file for
|
||||||
// the second, and the directives of every app in apps for the third.
|
// the second, and the fixes of every app in appFixes for the third.
|
||||||
// Each rule sww adds has an id from 900000 to 900999, the ids the
|
// Each rule sww adds has an id from 900000 to 900999, the ids the
|
||||||
// Core Rule Set keeps for the rules that set it up, which
|
// Core Rule Set keeps for the rules that set it up, which
|
||||||
// SWW_WAF_CORE_RULE_SET_DISABLED_RULES and the exclusions file refuse, so that no
|
// SWW_WAF_CORE_RULE_SET_DISABLED_RULES and the exclusions file refuse, so that no
|
||||||
@@ -54,13 +53,12 @@ Include @crs-setup.conf.example
|
|||||||
SecAction "id:900000,phase:1,pass,nolog,\
|
SecAction "id:900000,phase:1,pass,nolog,\
|
||||||
setvar:tx.blocking_paranoia_level=%d"
|
setvar:tx.blocking_paranoia_level=%d"
|
||||||
|
|
||||||
# The first change: PUT, PATCH and DELETE are allowed besides GET, HEAD,
|
# PUT, PATCH and DELETE are allowed besides GET, HEAD, POST and OPTIONS.
|
||||||
# POST and OPTIONS.
|
|
||||||
SecAction "id:900200,phase:1,pass,nolog,\
|
SecAction "id:900200,phase:1,pass,nolog,\
|
||||||
setvar:'tx.allowed_methods=GET HEAD POST OPTIONS PUT PATCH DELETE'"
|
setvar:'tx.allowed_methods=GET HEAD POST OPTIONS PUT PATCH DELETE'"
|
||||||
|
|
||||||
# The second: Expect and Content-Encoding are taken off the Core Rule Set's
|
# Expect and Content-Encoding are taken off the Core Rule Set's list of the
|
||||||
# list of the headers it refuses. Content-Encoding goes back on it for a
|
# headers it refuses. Content-Encoding goes back on it for a
|
||||||
# body the Core Rule Set reads (900260 in bodyDirectives).
|
# body the Core Rule Set reads (900260 in bodyDirectives).
|
||||||
SecAction "id:900250,phase:1,pass,nolog,\
|
SecAction "id:900250,phase:1,pass,nolog,\
|
||||||
setvar:'tx.restricted_headers_basic=/proxy/ /lock-token/ /content-range/ \
|
setvar:'tx.restricted_headers_basic=/proxy/ /lock-token/ /content-range/ \
|
||||||
@@ -86,18 +84,17 @@ SecRule ARGUMENTS_LIMIT_REACHED "@eq 1" "id:900300,phase:2,pass,\
|
|||||||
# The apps' fixes come before the rules for the same reason.
|
# The apps' fixes come before the rules for the same reason.
|
||||||
%s
|
%s
|
||||||
|
|
||||||
# The fifth: only the rules for requests are loaded, and no response is
|
# Only the rules for requests are loaded, and no response is inspected.
|
||||||
# inspected.
|
|
||||||
Include @owasp_crs/REQUEST-*.conf
|
Include @owasp_crs/REQUEST-*.conf
|
||||||
|
|
||||||
# The third: redirect_uri is not checked for a URL naming an IP address or
|
# redirect_uri is not checked for a URL naming an IP address or
|
||||||
# localhost. Coraza matches a parameter name here without regard to case.
|
# localhost. Coraza matches a parameter name here without regard to case.
|
||||||
# ARGS holds the fields of a form data or multipart body Coraza reads as
|
# ARGS holds the fields of a form data or multipart body Coraza reads as
|
||||||
# well as the query parameters, so a field of this name is left out too.
|
# well as the query parameters, so a field of this name is left out too.
|
||||||
SecRuleUpdateTargetById 931100 "!ARGS:redirect_uri"
|
SecRuleUpdateTargetById 931100 "!ARGS:redirect_uri"
|
||||||
SecRuleUpdateTargetById 934110 "!ARGS:redirect_uri"
|
SecRuleUpdateTargetById 934110 "!ARGS:redirect_uri"
|
||||||
|
|
||||||
# The fourth: Referer is not checked for a Unix command without arguments
|
# Referer is not checked for a Unix command without arguments
|
||||||
# or for Java starting a process.
|
# or for Java starting a process.
|
||||||
SecRuleUpdateTargetById 932340 "!REQUEST_HEADERS:Referer"
|
SecRuleUpdateTargetById 932340 "!REQUEST_HEADERS:Referer"
|
||||||
SecRuleUpdateTargetById 944110 "!REQUEST_HEADERS:Referer"
|
SecRuleUpdateTargetById 944110 "!REQUEST_HEADERS:Referer"
|
||||||
@@ -122,9 +119,9 @@ SecRule REQUEST_HEADERS:Content-Type \
|
|||||||
"@rx ^(?:application|text)/(?:[a-z0-9.-]+[+])?json" \
|
"@rx ^(?:application|text)/(?:[a-z0-9.-]+[+])?json" \
|
||||||
"id:900420,phase:1,pass,nolog,t:none,t:lowercase,ctl:requestBodyProcessor=JSON"
|
"id:900420,phase:1,pass,nolog,t:none,t:lowercase,ctl:requestBodyProcessor=JSON"
|
||||||
|
|
||||||
# The rest of the second change: Content-Encoding is refused again on a
|
# Content-Encoding goes back on the list of the headers the Core Rule Set
|
||||||
# body of a kind the Core Rule Set reads, since a compressed body cannot be
|
# refuses (900250 in directives) for a body of a kind it reads, since a
|
||||||
# inspected.
|
# compressed body cannot be inspected.
|
||||||
SecRule REQBODY_PROCESSOR "@rx ^(?:URLENCODED|MULTIPART|JSON|XML)$" \
|
SecRule REQBODY_PROCESSOR "@rx ^(?:URLENCODED|MULTIPART|JSON|XML)$" \
|
||||||
"id:900260,phase:1,pass,nolog,\
|
"id:900260,phase:1,pass,nolog,\
|
||||||
setvar:'tx.restricted_headers_basic=%%{tx.restricted_headers_basic} \
|
setvar:'tx.restricted_headers_basic=%%{tx.restricted_headers_basic} \
|
||||||
@@ -145,15 +142,19 @@ SecRule MULTIPART_STRICT_ERROR "!@eq 0" "id:900450,phase:2,pass,\
|
|||||||
severity:'CRITICAL',setvar:'tx.inbound_anomaly_score_pl1=+5'"
|
severity:'CRITICAL',setvar:'tx.inbound_anomaly_score_pl1=+5'"
|
||||||
`
|
`
|
||||||
|
|
||||||
// app is an app whose ordinary requests the Core Rule Set would refuse,
|
// appFixes holds the fixes of each app whose ordinary requests the Core
|
||||||
// with the fixes that let them pass.
|
// Rule Set would refuse, in a file named for the app by the name
|
||||||
|
// SWW_WAF_APPS gives it, such as apps/gitea.conf. The fixes are Coraza
|
||||||
|
// exclusions, loaded before the Core Rule Set's rules. Their first rule
|
||||||
|
// skips to the marker they end with, unless Inspect sets the transaction
|
||||||
|
// variable named for the app, such as app_gitea. The skip reaches only
|
||||||
|
// phase 1, so a rule of a later phase must check that variable itself.
|
||||||
|
//
|
||||||
|
//go:embed apps/*.conf
|
||||||
|
var appFixes embed.FS
|
||||||
|
|
||||||
|
// app is what an app's file in appFixes cannot say about it.
|
||||||
type app struct {
|
type app struct {
|
||||||
// directives are Coraza exclusions, loaded before the Core Rule Set's
|
|
||||||
// rules. Their first rule skips to the marker they end with, unless
|
|
||||||
// Inspect sets the transaction variable named for the app, such as
|
|
||||||
// app_gitea. The skip reaches only phase 1, so a rule of a later phase
|
|
||||||
// must check that variable itself.
|
|
||||||
directives string
|
|
||||||
// cookies are the cookies the Core Rule Set reads a request without.
|
// cookies are the cookies the Core Rule Set reads a request without.
|
||||||
cookies []string
|
cookies []string
|
||||||
// forkOf names the app this one is a fork of, "" for none. A hostname
|
// forkOf names the app this one is a fork of, "" for none. A hostname
|
||||||
@@ -161,60 +162,34 @@ type app struct {
|
|||||||
forkOf string
|
forkOf string
|
||||||
}
|
}
|
||||||
|
|
||||||
//go:embed gitea.conf
|
// apps are the apps that need more than their file in appFixes, by the
|
||||||
var giteaDirectives string
|
// name SWW_WAF_APPS gives each. flash is Forgejo's name for gitea_flash.
|
||||||
|
|
||||||
//go:embed wordpress.conf
|
|
||||||
var wordpressDirectives string
|
|
||||||
|
|
||||||
//go:embed drupal.conf
|
|
||||||
var drupalDirectives string
|
|
||||||
|
|
||||||
//go:embed mediawiki.conf
|
|
||||||
var mediawikiDirectives string
|
|
||||||
|
|
||||||
//go:embed mastodon.conf
|
|
||||||
var mastodonDirectives string
|
|
||||||
|
|
||||||
//go:embed vaultwarden.conf
|
|
||||||
var vaultwardenDirectives string
|
|
||||||
|
|
||||||
//go:embed nextcloud.conf
|
|
||||||
var nextcloudDirectives string
|
|
||||||
|
|
||||||
//go:embed forgejo.conf
|
|
||||||
var forgejoDirectives string
|
|
||||||
|
|
||||||
//go:embed ghost.conf
|
|
||||||
var ghostDirectives string
|
|
||||||
|
|
||||||
// apps are the apps whose fixes sww has, by the name SWW_WAF_APPS gives each.
|
|
||||||
// flash is Forgejo's name for gitea_flash.
|
|
||||||
//
|
//
|
||||||
//nolint:gochecknoglobals // a constant cannot be a map
|
//nolint:gochecknoglobals // a constant cannot be a map
|
||||||
var apps = map[string]app{
|
var apps = map[string]app{
|
||||||
"drupal": {directives: drupalDirectives},
|
|
||||||
"forgejo": {
|
"forgejo": {
|
||||||
directives: forgejoDirectives,
|
|
||||||
cookies: []string{"flash"},
|
cookies: []string{"flash"},
|
||||||
forkOf: "gitea",
|
forkOf: "gitea",
|
||||||
},
|
},
|
||||||
"ghost": {directives: ghostDirectives},
|
|
||||||
"gitea": {
|
"gitea": {
|
||||||
directives: giteaDirectives,
|
|
||||||
cookies: []string{"gitea_flash", "redirect_to"},
|
cookies: []string{"gitea_flash", "redirect_to"},
|
||||||
},
|
},
|
||||||
"mastodon": {directives: mastodonDirectives},
|
|
||||||
"mediawiki": {directives: mediawikiDirectives},
|
|
||||||
"nextcloud": {directives: nextcloudDirectives},
|
|
||||||
"vaultwarden": {directives: vaultwardenDirectives},
|
|
||||||
"wordpress": {directives: wordpressDirectives},
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// AppNames returns the names of the apps whose fixes sww has, in
|
// AppNames returns the names of the apps whose fixes sww has, in
|
||||||
// alphabetical order.
|
// alphabetical order.
|
||||||
func AppNames() []string {
|
func AppNames() []string {
|
||||||
return slices.Sorted(maps.Keys(apps))
|
// An embedded directory always reads.
|
||||||
|
files, _ := appFixes.ReadDir("apps")
|
||||||
|
|
||||||
|
names := make([]string, len(files))
|
||||||
|
for i, file := range files {
|
||||||
|
names[i] = strings.TrimSuffix(file.Name(), ".conf")
|
||||||
|
}
|
||||||
|
|
||||||
|
slices.Sort(names)
|
||||||
|
|
||||||
|
return names
|
||||||
}
|
}
|
||||||
|
|
||||||
// Params are what New needs.
|
// Params are what New needs.
|
||||||
@@ -373,8 +348,11 @@ func New(params Params) (*CoreRuleSet, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
var fixes strings.Builder
|
var fixes strings.Builder
|
||||||
|
|
||||||
for _, name := range AppNames() {
|
for _, name := range AppNames() {
|
||||||
fixes.WriteString(apps[name].directives + "\n")
|
// An embedded file always reads.
|
||||||
|
file, _ := appFixes.ReadFile("apps/" + name + ".conf")
|
||||||
|
fixes.WriteString(string(file) + "\n")
|
||||||
}
|
}
|
||||||
|
|
||||||
text := fmt.Sprintf(directives, params.ParanoiaLevel, body, exclusions,
|
text := fmt.Sprintf(directives, params.ParanoiaLevel, body, exclusions,
|
||||||
|
|||||||
@@ -17,7 +17,7 @@ const (
|
|||||||
)
|
)
|
||||||
|
|
||||||
// giteaSearches are searches of the pages gitea and Forgejo share whose q
|
// giteaSearches are searches of the pages gitea and Forgejo share whose q
|
||||||
// skips the lists of system files and command names, in the eighth change.
|
// skips the lists of system files and command names, in gitea's fixes.
|
||||||
// gitea serves its admin pages under /-/admin/ and Forgejo under /admin/.
|
// gitea serves its admin pages under /-/admin/ and Forgejo under /admin/.
|
||||||
//
|
//
|
||||||
//nolint:gochecknoglobals // a constant cannot be a list
|
//nolint:gochecknoglobals // a constant cannot be a list
|
||||||
@@ -222,8 +222,8 @@ func TestGiteaSearchExclusionsAreNarrow(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Under /admin/, where Forgejo serves its admin pages, only the hard
|
// Under /admin/, where Forgejo serves its admin pages, only the hard
|
||||||
// values are left out: the tenth change lets them through in q on every
|
// values are left out: Drupal's fixes let them through in q on every path
|
||||||
// path there, for Drupal.
|
// there.
|
||||||
for _, target := range []string{
|
for _, target := range []string{
|
||||||
"/admin/packages/delete?q=",
|
"/admin/packages/delete?q=",
|
||||||
"/admin/packages?other=",
|
"/admin/packages?other=",
|
||||||
|
|||||||
@@ -0,0 +1,90 @@
|
|||||||
|
// Package dirwatch watches a directory, and calls a function once the
|
||||||
|
// directory has gone a short time without a change after one. The rule
|
||||||
|
// files and the lookup database are read again this way.
|
||||||
|
package dirwatch
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/fsnotify/fsnotify"
|
||||||
|
)
|
||||||
|
|
||||||
|
// quietTime is how long the directory must go without a change before the
|
||||||
|
// function is called, so that a file still being written, such as one
|
||||||
|
// saved in place, appended to or copied in with scp, is read only once
|
||||||
|
// whole.
|
||||||
|
const quietTime = 2 * time.Second
|
||||||
|
|
||||||
|
// Watcher watches one directory. The directory is watched, rather than a
|
||||||
|
// file in it, so that a file replaced by a rename is seen.
|
||||||
|
type Watcher struct {
|
||||||
|
watcher *fsnotify.Watcher
|
||||||
|
}
|
||||||
|
|
||||||
|
// New starts watching dir, or returns why it cannot.
|
||||||
|
func New(dir string) (*Watcher, error) {
|
||||||
|
watcher, err := fsnotify.NewWatcher()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
err = watcher.Add(dir)
|
||||||
|
if err != nil {
|
||||||
|
_ = watcher.Close()
|
||||||
|
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
return &Watcher{watcher: watcher}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Close stops watching the directory.
|
||||||
|
func (w *Watcher) Close() error {
|
||||||
|
return w.watcher.Close()
|
||||||
|
}
|
||||||
|
|
||||||
|
// CallWhenQuiet calls changed each time the directory has gone quietTime
|
||||||
|
// without a change, after a change to a path for which counts is true, until
|
||||||
|
// ctx is done. It also calls failed with each error the watch reports. The
|
||||||
|
// wait starts at once, as if for a change, so that a change made before the
|
||||||
|
// watch began is taken in too.
|
||||||
|
func (w *Watcher) CallWhenQuiet(
|
||||||
|
ctx context.Context,
|
||||||
|
counts func(path string) bool,
|
||||||
|
changed func(),
|
||||||
|
failed func(error),
|
||||||
|
) {
|
||||||
|
callWhenQuiet(
|
||||||
|
ctx, w.watcher.Events, w.watcher.Errors, counts, changed, failed,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// callWhenQuiet is CallWhenQuiet, for the changes from events and the
|
||||||
|
// errors from errs.
|
||||||
|
func callWhenQuiet(
|
||||||
|
ctx context.Context,
|
||||||
|
events <-chan fsnotify.Event,
|
||||||
|
errs <-chan error,
|
||||||
|
counts func(path string) bool,
|
||||||
|
changed func(),
|
||||||
|
failed func(error),
|
||||||
|
) {
|
||||||
|
quiet := time.NewTimer(quietTime)
|
||||||
|
defer quiet.Stop()
|
||||||
|
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case event := <-events:
|
||||||
|
if counts(event.Name) {
|
||||||
|
quiet.Reset(quietTime)
|
||||||
|
}
|
||||||
|
case <-quiet.C:
|
||||||
|
changed()
|
||||||
|
case err := <-errs:
|
||||||
|
failed(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,167 @@
|
|||||||
|
package dirwatch
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"path/filepath"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"testing/synctest"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/fsnotify/fsnotify"
|
||||||
|
)
|
||||||
|
|
||||||
|
// errWatch is an error the watch reports.
|
||||||
|
var errWatch = errors.New("too many changes")
|
||||||
|
|
||||||
|
func TestDirectoryThatCannotBeWatchedIsAnError(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
_, err := New(filepath.Join(t.TempDir(), "missing"))
|
||||||
|
if err == nil {
|
||||||
|
t.Error("a missing directory is watched, want an error")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The tests below run callWhenQuiet in a synctest bubble, where time is a
|
||||||
|
// clock of the test's own: time.Sleep moves it on at once, and
|
||||||
|
// synctest.Wait returns once callWhenQuiet waits again, so that every call
|
||||||
|
// due by then is made. The test sends the changes itself, as the watch of
|
||||||
|
// a directory cannot run in a bubble.
|
||||||
|
|
||||||
|
func TestSecondChangeStartsTheWaitAgain(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
loop := start(t, everyPath)
|
||||||
|
|
||||||
|
loop.changes <- fsnotify.Event{Name: "50-app.rules", Op: fsnotify.Write}
|
||||||
|
|
||||||
|
time.Sleep(quietTime - time.Nanosecond)
|
||||||
|
synctest.Wait()
|
||||||
|
loop.wantCalls(t, 0)
|
||||||
|
|
||||||
|
// The second part starts the wait again.
|
||||||
|
loop.changes <- fsnotify.Event{Name: "50-app.rules", Op: fsnotify.Write}
|
||||||
|
|
||||||
|
time.Sleep(quietTime - time.Nanosecond)
|
||||||
|
synctest.Wait()
|
||||||
|
loop.wantCalls(t, 0)
|
||||||
|
|
||||||
|
time.Sleep(time.Nanosecond)
|
||||||
|
synctest.Wait()
|
||||||
|
loop.wantCalls(t, 1)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCalledOnceQuietAfterTheStart(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
// An edit saved before the watch began sends no change, and is
|
||||||
|
// taken in by this call.
|
||||||
|
loop := start(t, everyPath)
|
||||||
|
|
||||||
|
time.Sleep(quietTime)
|
||||||
|
synctest.Wait()
|
||||||
|
loop.wantCalls(t, 1)
|
||||||
|
|
||||||
|
// Long after, it has been called only once.
|
||||||
|
time.Sleep(time.Hour)
|
||||||
|
synctest.Wait()
|
||||||
|
loop.wantCalls(t, 1)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestChangeThatDoesNotCountIsNoChange(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
loop := start(t, func(path string) bool {
|
||||||
|
return path == "ipinfo_lite.mmdb"
|
||||||
|
})
|
||||||
|
|
||||||
|
// The wait that starts with the watch ends with a call.
|
||||||
|
time.Sleep(quietTime)
|
||||||
|
synctest.Wait()
|
||||||
|
|
||||||
|
loop.changes <- fsnotify.Event{Name: "other.mmdb", Op: fsnotify.Create}
|
||||||
|
|
||||||
|
time.Sleep(quietTime)
|
||||||
|
synctest.Wait()
|
||||||
|
loop.wantCalls(t, 1)
|
||||||
|
|
||||||
|
loop.changes <- fsnotify.Event{Name: "ipinfo_lite.mmdb", Op: fsnotify.Write}
|
||||||
|
|
||||||
|
time.Sleep(quietTime)
|
||||||
|
synctest.Wait()
|
||||||
|
loop.wantCalls(t, 2)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestErrorsOfTheWatchHandedOn(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
loop := start(t, everyPath)
|
||||||
|
|
||||||
|
loop.errs <- errWatch
|
||||||
|
|
||||||
|
synctest.Wait()
|
||||||
|
|
||||||
|
if !errors.Is(loop.failure, errWatch) {
|
||||||
|
t.Errorf("failed with %v, want %v", loop.failure, errWatch)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// loop is a callWhenQuiet that a test runs.
|
||||||
|
type loop struct {
|
||||||
|
// changes and errs send it changes and errors.
|
||||||
|
changes chan fsnotify.Event
|
||||||
|
errs chan error
|
||||||
|
// calls counts its calls of changed, and failure is the last error it
|
||||||
|
// handed to failed.
|
||||||
|
calls atomic.Int32
|
||||||
|
failure error
|
||||||
|
}
|
||||||
|
|
||||||
|
// start runs callWhenQuiet, with counts, until the test ends.
|
||||||
|
func start(t *testing.T, counts func(path string) bool) *loop {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
l := &loop{changes: make(chan fsnotify.Event), errs: make(chan error)}
|
||||||
|
ctx, stop := context.WithCancel(t.Context())
|
||||||
|
stopped := make(chan struct{})
|
||||||
|
|
||||||
|
go func() {
|
||||||
|
callWhenQuiet(ctx, l.changes, l.errs, counts,
|
||||||
|
func() { l.calls.Add(1) },
|
||||||
|
func(err error) { l.failure = err },
|
||||||
|
)
|
||||||
|
close(stopped)
|
||||||
|
}()
|
||||||
|
|
||||||
|
t.Cleanup(func() {
|
||||||
|
stop()
|
||||||
|
<-stopped
|
||||||
|
})
|
||||||
|
|
||||||
|
return l
|
||||||
|
}
|
||||||
|
|
||||||
|
// everyPath counts a change to any path.
|
||||||
|
func everyPath(string) bool {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// wantCalls checks how many times changed has been called.
|
||||||
|
func (l *loop) wantCalls(t *testing.T, want int32) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
got := l.calls.Load()
|
||||||
|
if got != want {
|
||||||
|
t.Errorf("changed called %d times, want %d", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
+18
-49
@@ -13,18 +13,13 @@ import (
|
|||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/fsnotify/fsnotify"
|
|
||||||
"github.com/oschwald/maxminddb-golang/v2"
|
"github.com/oschwald/maxminddb-golang/v2"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/dirwatch"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
)
|
)
|
||||||
|
|
||||||
// quietTime is how long the lookup database must go without a change
|
|
||||||
// before it is read again, so that a file still being copied in is read
|
|
||||||
// only once whole.
|
|
||||||
const quietTime = 2 * time.Second
|
|
||||||
|
|
||||||
// FileParams are what OpenFile needs.
|
// FileParams are what OpenFile needs.
|
||||||
type FileParams struct {
|
type FileParams struct {
|
||||||
// Path is the lookup database, the IPinfo Lite file in its .mmdb form
|
// Path is the lookup database, the IPinfo Lite file in its .mmdb form
|
||||||
@@ -122,20 +117,13 @@ func (f *File) ReadFailures() int {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Watch watches the directory of the lookup database until ctx is done,
|
// Watch watches the directory of the lookup database until ctx is done,
|
||||||
// and reads the file again once it has gone without a change for
|
// and reads the file again once it has gone without a change for a short
|
||||||
// quietTime, after it is replaced, written or removed, and after Watch
|
// time, after it is replaced, written or removed, and after Watch starts
|
||||||
// starts watching. If the directory cannot be watched, that is logged, and
|
// watching, as dirwatch describes. A change to another file in its
|
||||||
// the database read at start stays in use.
|
// directory does not count. If the directory cannot be watched, that is
|
||||||
|
// logged, and the database read at start stays in use.
|
||||||
func (f *File) Watch(ctx context.Context) {
|
func (f *File) Watch(ctx context.Context) {
|
||||||
watcher, err := fsnotify.NewWatcher()
|
watcher, err := dirwatch.New(filepath.Dir(f.params.Path))
|
||||||
if err == nil {
|
|
||||||
defer func() {
|
|
||||||
_ = watcher.Close()
|
|
||||||
}()
|
|
||||||
|
|
||||||
err = watcher.Add(filepath.Dir(f.params.Path))
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
f.params.ProcessLog.Error(
|
f.params.ProcessLog.Error(
|
||||||
"cannot watch the lookup database for replacements",
|
"cannot watch the lookup database for replacements",
|
||||||
@@ -146,41 +134,22 @@ func (f *File) Watch(ctx context.Context) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
defer func() {
|
||||||
|
_ = watcher.Close()
|
||||||
|
}()
|
||||||
|
|
||||||
f.params.ProcessLog.Info("watching the lookup database for replacements",
|
f.params.ProcessLog.Info("watching the lookup database for replacements",
|
||||||
"file", f.params.Path)
|
"file", f.params.Path)
|
||||||
|
|
||||||
f.readAfterChanges(ctx, watcher.Events, watcher.Errors)
|
watcher.CallWhenQuiet(ctx, f.isDatabase, f.readAgain, func(err error) {
|
||||||
}
|
|
||||||
|
|
||||||
// readAfterChanges reads the lookup database again once quietTime has
|
|
||||||
// passed without a change to it from events, until ctx is done, and logs
|
|
||||||
// the errors from errs. A change to another file in its directory does not
|
|
||||||
// count. The wait starts at once, as if for a change, so that a file
|
|
||||||
// replaced after OpenFile read it, and before its directory was watched,
|
|
||||||
// is read too.
|
|
||||||
func (f *File) readAfterChanges(
|
|
||||||
ctx context.Context, events <-chan fsnotify.Event, errs <-chan error,
|
|
||||||
) {
|
|
||||||
path := filepath.Clean(f.params.Path)
|
|
||||||
|
|
||||||
quiet := time.NewTimer(quietTime)
|
|
||||||
defer quiet.Stop()
|
|
||||||
|
|
||||||
for {
|
|
||||||
select {
|
|
||||||
case <-ctx.Done():
|
|
||||||
return
|
|
||||||
case event := <-events:
|
|
||||||
if filepath.Clean(event.Name) == path {
|
|
||||||
quiet.Reset(quietTime)
|
|
||||||
}
|
|
||||||
case <-quiet.C:
|
|
||||||
f.readAgain()
|
|
||||||
case err := <-errs:
|
|
||||||
f.params.ProcessLog.Warn("watching the lookup database failed",
|
f.params.ProcessLog.Warn("watching the lookup database failed",
|
||||||
"error", err.Error())
|
"error", err.Error())
|
||||||
}
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// isDatabase reports whether path names the lookup database.
|
||||||
|
func (f *File) isDatabase(path string) bool {
|
||||||
|
return filepath.Clean(path) == filepath.Clean(f.params.Path)
|
||||||
}
|
}
|
||||||
|
|
||||||
// readAgain reads the lookup database again, in place of the one in use,
|
// readAgain reads the lookup database again, in place of the one in use,
|
||||||
|
|||||||
@@ -1,7 +1,6 @@
|
|||||||
package lookup
|
package lookup
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"net/url"
|
"net/url"
|
||||||
@@ -12,7 +11,6 @@ import (
|
|||||||
"testing/synctest"
|
"testing/synctest"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/fsnotify/fsnotify"
|
|
||||||
"github.com/maxmind/mmdbwriter/mmdbtype"
|
"github.com/maxmind/mmdbwriter/mmdbtype"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
@@ -133,66 +131,48 @@ func TestFileThatCannotBeReadIsAnError(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// The tests below run readAfterChanges in a synctest bubble, where time is
|
func TestChangeOfAnotherFileInTheDirectoryIsNoReplacement(t *testing.T) {
|
||||||
// a clock of the test's own: time.Sleep moves it on at once, and
|
|
||||||
// synctest.Wait returns once readAfterChanges waits again, so that every
|
|
||||||
// reading due by then is done. The test sends the changes itself, as the
|
|
||||||
// watch of a directory cannot run in a bubble.
|
|
||||||
|
|
||||||
func TestReplacementCopiedOverTheFileInTwoPartsIsReadOnlyWhole(
|
|
||||||
t *testing.T,
|
|
||||||
) {
|
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
dir := t.TempDir()
|
dir := t.TempDir()
|
||||||
path := filepath.Join(dir, "ipinfo_lite.mmdb")
|
path := filepath.Join(dir, "ipinfo_lite.mmdb")
|
||||||
writeDatabase(t, path, "DE")
|
writeDatabase(t, path, "DE")
|
||||||
|
|
||||||
|
f := openFile(t, path, newQueue())
|
||||||
|
|
||||||
|
for changed, want := range map[string]bool{
|
||||||
|
path: true,
|
||||||
|
dir + "/./ipinfo_lite.mmdb": true,
|
||||||
|
filepath.Join(dir, "other.mmdb"): false,
|
||||||
|
} {
|
||||||
|
if f.isDatabase(changed) != want {
|
||||||
|
t.Errorf(
|
||||||
|
"a change to %s counts: %t, want %t",
|
||||||
|
changed,
|
||||||
|
!want,
|
||||||
|
want,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The tests below run in a synctest bubble, where time is a clock of the
|
||||||
|
// test's own, so that the time a file is read can be checked. The wait for
|
||||||
|
// the directory to go quiet before a reading is tested in dirwatch.
|
||||||
|
|
||||||
|
func TestReplacementIsReadAgain(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
path := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb")
|
||||||
|
writeDatabase(t, path, "DE")
|
||||||
|
|
||||||
queue := newQueue()
|
queue := newQueue()
|
||||||
f := openFile(t, path, queue)
|
f := openFile(t, path, queue)
|
||||||
changes := watch(t, f)
|
|
||||||
|
|
||||||
other := filepath.Join(dir, "replacement.mmdb")
|
writeDatabase(t, path, "KP")
|
||||||
writeDatabase(t, other, "KP")
|
time.Sleep(time.Hour)
|
||||||
|
f.readAgain()
|
||||||
//nolint:gosec // a file the test wrote
|
|
||||||
replacement, err := os.ReadFile(other)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("read %s: %v", other, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// The file in use is overwritten in place, and keeps giving what
|
|
||||||
// it gave. Its first part alone is not a .mmdb file.
|
|
||||||
file, err := os.Create(path) //nolint:gosec // a file the test wrote
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("create %s: %v", path, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() {
|
|
||||||
_ = file.Close()
|
|
||||||
}()
|
|
||||||
|
|
||||||
half := len(replacement) / 2
|
|
||||||
write(t, file, replacement[:half])
|
|
||||||
|
|
||||||
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
|
|
||||||
|
|
||||||
time.Sleep(quietTime - time.Nanosecond)
|
|
||||||
synctest.Wait()
|
|
||||||
wantCountry(t, f, "DE")
|
|
||||||
|
|
||||||
// The second part starts the wait again.
|
|
||||||
write(t, file, replacement[half:])
|
|
||||||
|
|
||||||
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
|
|
||||||
|
|
||||||
time.Sleep(quietTime - time.Nanosecond)
|
|
||||||
synctest.Wait()
|
|
||||||
wantCountry(t, f, "DE")
|
|
||||||
|
|
||||||
time.Sleep(time.Nanosecond)
|
|
||||||
synctest.Wait()
|
|
||||||
wantCountry(t, f, "KP")
|
wantCountry(t, f, "KP")
|
||||||
|
|
||||||
if !f.LastRead().Equal(time.Now()) || f.ReadFailures() != 0 {
|
if !f.LastRead().Equal(time.Now()) || f.ReadFailures() != 0 {
|
||||||
@@ -219,15 +199,10 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(
|
|||||||
queue := newQueue()
|
queue := newQueue()
|
||||||
f := openFile(t, path, queue)
|
f := openFile(t, path, queue)
|
||||||
read := f.LastRead()
|
read := f.LastRead()
|
||||||
changes := watch(t, f)
|
|
||||||
|
|
||||||
writeFile(t, path, "not a lookup database\n")
|
writeFile(t, path, "not a lookup database\n")
|
||||||
|
|
||||||
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
|
|
||||||
|
|
||||||
// Long after, the replacement has been read once.
|
|
||||||
time.Sleep(time.Hour)
|
time.Sleep(time.Hour)
|
||||||
synctest.Wait()
|
f.readAgain()
|
||||||
wantCountry(t, f, "DE")
|
wantCountry(t, f, "DE")
|
||||||
|
|
||||||
if !f.LastRead().Equal(read) || f.ReadFailures() != 1 {
|
if !f.LastRead().Equal(read) || f.ReadFailures() != 1 {
|
||||||
@@ -240,7 +215,7 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(
|
|||||||
}
|
}
|
||||||
|
|
||||||
wantAlerts(t, queue, alerts.Alert{
|
wantAlerts(t, queue, alerts.Alert{
|
||||||
Time: read.Add(quietTime),
|
Time: read.Add(time.Hour),
|
||||||
Event: alerts.EventFileError,
|
Event: alerts.EventFileError,
|
||||||
Reason: "the lookup database cannot be read, and the one read before stays in use",
|
Reason: "the lookup database cannot be read, and the one read before stays in use",
|
||||||
Detail: map[string]any{
|
Detail: map[string]any{
|
||||||
@@ -252,55 +227,6 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestChangeOfAnotherFileInTheDirectoryIsNoReplacement(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
dir := t.TempDir()
|
|
||||||
path := filepath.Join(dir, "ipinfo_lite.mmdb")
|
|
||||||
writeDatabase(t, path, "DE")
|
|
||||||
|
|
||||||
f := openFile(t, path, newQueue())
|
|
||||||
changes := watch(t, f)
|
|
||||||
|
|
||||||
// The wait that starts with the watch ends with a reading.
|
|
||||||
time.Sleep(quietTime)
|
|
||||||
synctest.Wait()
|
|
||||||
writeDatabase(t, path, "KP")
|
|
||||||
|
|
||||||
changes <- fsnotify.Event{Name: filepath.Join(dir, "other.mmdb"), Op: fsnotify.Create}
|
|
||||||
|
|
||||||
time.Sleep(quietTime)
|
|
||||||
synctest.Wait()
|
|
||||||
wantCountry(t, f, "DE")
|
|
||||||
|
|
||||||
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
|
|
||||||
|
|
||||||
time.Sleep(quietTime)
|
|
||||||
synctest.Wait()
|
|
||||||
wantCountry(t, f, "KP")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestReplacementSavedBeforeTheWatchStartsIsRead(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
path := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb")
|
|
||||||
writeDatabase(t, path, "DE")
|
|
||||||
|
|
||||||
f := openFile(t, path, newQueue())
|
|
||||||
|
|
||||||
// Saved after OpenFile read the file, and before its directory was
|
|
||||||
// watched, so that no change is seen for it.
|
|
||||||
writeDatabase(t, path, "KP")
|
|
||||||
watch(t, f)
|
|
||||||
time.Sleep(quietTime)
|
|
||||||
synctest.Wait()
|
|
||||||
wantCountry(t, f, "KP")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// newQueue returns a queue of alerts for a webhook that is never sent
|
// newQueue returns a queue of alerts for a webhook that is never sent
|
||||||
// them, so that they wait in it for the test to look at.
|
// them, so that they wait in it for the test to look at.
|
||||||
func newQueue() *alerts.Queue {
|
func newQueue() *alerts.Queue {
|
||||||
@@ -344,28 +270,6 @@ func openFile(t *testing.T, path string, queue *alerts.Queue) *File {
|
|||||||
return f
|
return f
|
||||||
}
|
}
|
||||||
|
|
||||||
// watch runs f's readAfterChanges until the test ends, and returns the
|
|
||||||
// channel that sends it changes.
|
|
||||||
func watch(t *testing.T, f *File) chan<- fsnotify.Event {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
changes := make(chan fsnotify.Event)
|
|
||||||
ctx, stop := context.WithCancel(t.Context())
|
|
||||||
stopped := make(chan struct{})
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
f.readAfterChanges(ctx, changes, nil)
|
|
||||||
close(stopped)
|
|
||||||
}()
|
|
||||||
|
|
||||||
t.Cleanup(func() {
|
|
||||||
stop()
|
|
||||||
<-stopped
|
|
||||||
})
|
|
||||||
|
|
||||||
return changes
|
|
||||||
}
|
|
||||||
|
|
||||||
// wantCountry checks the country f gives testClient.
|
// wantCountry checks the country f gives testClient.
|
||||||
func wantCountry(t *testing.T, f *File, want string) {
|
func wantCountry(t *testing.T, f *File, want string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
@@ -401,13 +305,3 @@ func writeFile(t *testing.T, path, content string) {
|
|||||||
t.Fatalf("write %s: %v", path, err)
|
t.Fatalf("write %s: %v", path, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// write writes data to the end of file.
|
|
||||||
func write(t *testing.T, file *os.File, data []byte) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
_, err := file.Write(data)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("write: %v", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
+396
-269
@@ -15,12 +15,34 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/remotelog"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// Params are what New builds the metrics from. Each part is nil while it is
|
||||||
|
// off, and a part that is nil adds no metric of its own.
|
||||||
|
type Params struct {
|
||||||
|
// BusiestCount is how many countries and how many AS numbers get series
|
||||||
|
// of their own (SWW_WAF_METRICS_BUSIEST_COUNTRIES_AND_ASNS).
|
||||||
|
BusiestCount int
|
||||||
|
// InstanceName is the label instance every metric carries
|
||||||
|
// (SWW_WAF_INSTANCE_NAME).
|
||||||
|
InstanceName string
|
||||||
|
// Now returns the time at which Bans counts the bans active.
|
||||||
|
Now func() time.Time
|
||||||
|
|
||||||
|
Bans banLedger
|
||||||
|
Clients sized
|
||||||
|
Rules sized
|
||||||
|
Lists fetchedLists
|
||||||
|
DNSBL dnsblZones
|
||||||
|
AbuseIPDB abuseIPDBChecker
|
||||||
|
Alerts alertQueue
|
||||||
|
LookupFile lookupDatabase
|
||||||
|
RemoteLog remoteLogSender
|
||||||
|
}
|
||||||
|
|
||||||
// Metrics are sww's metrics. They are safe for concurrent use.
|
// Metrics are sww's metrics. They are safe for concurrent use.
|
||||||
type Metrics struct {
|
type Metrics struct {
|
||||||
// registry gives every metric registered with it the label instance.
|
// registry gives every metric registered with it the label instance.
|
||||||
@@ -36,8 +58,6 @@ type Metrics struct {
|
|||||||
rateLimitHits *prometheus.CounterVec
|
rateLimitHits *prometheus.CounterVec
|
||||||
offences *prometheus.CounterVec
|
offences *prometheus.CounterVec
|
||||||
coreRuleSetMatches *prometheus.CounterVec
|
coreRuleSetMatches *prometheus.CounterVec
|
||||||
// ruleMatches are made by AddRules, and reputationHits by
|
|
||||||
// AddReputation.
|
|
||||||
ruleMatches *prometheus.CounterVec
|
ruleMatches *prometheus.CounterVec
|
||||||
reputationHits *prometheus.CounterVec
|
reputationHits *prometheus.CounterVec
|
||||||
countries *busiest
|
countries *busiest
|
||||||
@@ -59,20 +79,18 @@ type Metrics struct {
|
|||||||
stateFileEditsSetAside *prometheus.CounterVec
|
stateFileEditsSetAside *prometheus.CounterVec
|
||||||
}
|
}
|
||||||
|
|
||||||
// New returns the metrics, with the Go runtime's and the process's own.
|
// New returns the metrics, with the Go runtime's and the process's own, and
|
||||||
// busiestCount is how many countries and how many AS numbers get series of
|
// those read from each part in params as the metrics are asked for.
|
||||||
// their own (SWW_WAF_METRICS_BUSIEST_COUNTRIES_AND_ASNS). Every metric
|
|
||||||
// carries instanceName (SWW_WAF_INSTANCE_NAME) as its label instance.
|
|
||||||
//
|
//
|
||||||
//nolint:funlen // a few lines for each metric, a list that grows with them
|
//nolint:funlen // a few lines for each metric, a list that grows with them
|
||||||
func New(busiestCount int, instanceName string) *Metrics {
|
func New(params Params) *Metrics {
|
||||||
byStatus := []string{"status_class", "action"}
|
byStatus := []string{"status_class", "action"}
|
||||||
byFile := []string{"file"}
|
byFile := []string{"file"}
|
||||||
registry := prometheus.NewRegistry()
|
registry := prometheus.NewRegistry()
|
||||||
|
|
||||||
m := &Metrics{
|
m := &Metrics{
|
||||||
registry: prometheus.WrapRegistererWith(
|
registry: prometheus.WrapRegistererWith(
|
||||||
prometheus.Labels{"instance": instanceName}, registry),
|
prometheus.Labels{"instance": params.InstanceName}, registry),
|
||||||
handler: promhttp.HandlerFor(registry, promhttp.HandlerOpts{}),
|
handler: promhttp.HandlerFor(registry, promhttp.HandlerOpts{}),
|
||||||
inFlight: prometheus.NewGauge(prometheus.GaugeOpts{
|
inFlight: prometheus.NewGauge(prometheus.GaugeOpts{
|
||||||
Name: "sww_requests_in_flight",
|
Name: "sww_requests_in_flight",
|
||||||
@@ -113,8 +131,19 @@ func New(busiestCount int, instanceName string) *Metrics {
|
|||||||
"and the rule's id.",
|
"and the rule's id.",
|
||||||
[]string{"mode", "rule_id"},
|
[]string{"mode", "rule_id"},
|
||||||
),
|
),
|
||||||
countries: newCountries(busiestCount),
|
ruleMatches: counterVec(
|
||||||
asns: newASNs(busiestCount),
|
"sww_rule_matches_total",
|
||||||
|
"Requests that matched a rule of the rule files, by its id and action.",
|
||||||
|
[]string{"rule_id", "action"},
|
||||||
|
),
|
||||||
|
reputationHits: counterVec(
|
||||||
|
"sww_reputation_hits_total",
|
||||||
|
"Requests whose client a blocklist, a DNSBL zone or AbuseIPDB lists, "+
|
||||||
|
"by the list's URL, the zone, or abuseipdb.",
|
||||||
|
[]string{sourceLabel},
|
||||||
|
),
|
||||||
|
countries: newCountries(params.BusiestCount),
|
||||||
|
asns: newASNs(params.BusiestCount),
|
||||||
GeoJSRequests: prometheus.NewCounter(prometheus.CounterOpts{
|
GeoJSRequests: prometheus.NewCounter(prometheus.CounterOpts{
|
||||||
Name: "sww_geojs_requests_total",
|
Name: "sww_geojs_requests_total",
|
||||||
Help: "Requests to GeoJS.",
|
Help: "Requests to GeoJS.",
|
||||||
@@ -166,6 +195,7 @@ func New(busiestCount int, instanceName string) *Metrics {
|
|||||||
m.inFlight, m.requests, m.requestBytes, m.responseBytes,
|
m.inFlight, m.requests, m.requestBytes, m.responseBytes,
|
||||||
m.requestDuration, m.upstreamDuration,
|
m.requestDuration, m.upstreamDuration,
|
||||||
m.rateLimitHits, m.offences, m.coreRuleSetMatches,
|
m.rateLimitHits, m.offences, m.coreRuleSetMatches,
|
||||||
|
m.ruleMatches, m.reputationHits,
|
||||||
m.countries, m.asns,
|
m.countries, m.asns,
|
||||||
m.GeoJSRequests, m.GeoJSFailures, m.RequestsWithoutGeoJSAnswer,
|
m.GeoJSRequests, m.GeoJSFailures, m.RequestsWithoutGeoJSAnswer,
|
||||||
m.stateFileWrites, m.stateFileWriteFailures,
|
m.stateFileWrites, m.stateFileWriteFailures,
|
||||||
@@ -173,261 +203,45 @@ func New(busiestCount int, instanceName string) *Metrics {
|
|||||||
m.stateFileEditsTakenIn, m.stateFileEditsSetAside,
|
m.stateFileEditsTakenIn, m.stateFileEditsSetAside,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
if params.Bans != nil {
|
||||||
|
m.addBans(params.Bans, params.Now)
|
||||||
|
}
|
||||||
|
|
||||||
|
if params.Clients != nil {
|
||||||
|
m.addClients(params.Clients)
|
||||||
|
}
|
||||||
|
|
||||||
|
if params.Rules != nil {
|
||||||
|
m.addRules(params.Rules)
|
||||||
|
}
|
||||||
|
|
||||||
|
if params.Lists != nil {
|
||||||
|
m.addLists(params.Lists)
|
||||||
|
}
|
||||||
|
|
||||||
|
if params.DNSBL != nil {
|
||||||
|
m.addDNSBL(params.DNSBL)
|
||||||
|
}
|
||||||
|
|
||||||
|
if params.AbuseIPDB != nil {
|
||||||
|
m.addAbuseIPDB(params.AbuseIPDB)
|
||||||
|
}
|
||||||
|
|
||||||
|
if params.Alerts != nil {
|
||||||
|
m.addAlerts(params.Alerts)
|
||||||
|
}
|
||||||
|
|
||||||
|
if params.LookupFile != nil {
|
||||||
|
m.addLookupFile(params.LookupFile)
|
||||||
|
}
|
||||||
|
|
||||||
|
if params.RemoteLog != nil {
|
||||||
|
m.addRemoteLog(params.RemoteLog)
|
||||||
|
}
|
||||||
|
|
||||||
return m
|
return m
|
||||||
}
|
}
|
||||||
|
|
||||||
// AddBansAndClients adds the metrics read from the ledger and the table
|
|
||||||
// of clients as the metrics are asked for: the bans made since the start,
|
|
||||||
// by cause, the bans active and permanent at now, and the clients in the
|
|
||||||
// table.
|
|
||||||
func (m *Metrics) AddBansAndClients(
|
|
||||||
ledger *bans.Ledger, limiter *ratelimit.Limiter, now func() time.Time,
|
|
||||||
) {
|
|
||||||
for _, cause := range []string{
|
|
||||||
bans.CauseLimit, bans.CauseAttack, bans.CauseAdmin, bans.CauseAbuseIPDB,
|
|
||||||
} {
|
|
||||||
m.registry.MustRegister(
|
|
||||||
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
|
||||||
Name: "sww_bans_made_total",
|
|
||||||
Help: "Bans made, by cause.",
|
|
||||||
ConstLabels: prometheus.Labels{"cause": cause},
|
|
||||||
}, func() float64 {
|
|
||||||
return float64(ledger.Made(cause))
|
|
||||||
}),
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
m.registry.MustRegister(
|
|
||||||
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
|
|
||||||
Name: "sww_active_bans",
|
|
||||||
Help: "Bans active now, the permanent ones included.",
|
|
||||||
}, func() float64 {
|
|
||||||
active, _ := ledger.Count(now())
|
|
||||||
|
|
||||||
return float64(active)
|
|
||||||
}),
|
|
||||||
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
|
|
||||||
Name: "sww_permanent_bans",
|
|
||||||
Help: "Permanent bans not lifted.",
|
|
||||||
}, func() float64 {
|
|
||||||
_, permanent := ledger.Count(now())
|
|
||||||
|
|
||||||
return float64(permanent)
|
|
||||||
}),
|
|
||||||
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
|
|
||||||
Name: "sww_tracked_clients",
|
|
||||||
Help: "Clients in the table of clients.",
|
|
||||||
}, func() float64 {
|
|
||||||
return float64(limiter.Len())
|
|
||||||
}),
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
// AddRules adds the metrics of the rule files: the requests that matched
|
|
||||||
// each rule, which RuleMatched counts, and the rules loaded from
|
|
||||||
// ruleFiles, read as the metrics are asked for. It is called once, before
|
|
||||||
// RuleMatched.
|
|
||||||
func (m *Metrics) AddRules(ruleFiles *rules.Files) {
|
|
||||||
m.ruleMatches = counterVec(
|
|
||||||
"sww_rule_matches_total",
|
|
||||||
"Requests that matched a rule of the rule files, by its id and action.",
|
|
||||||
[]string{"rule_id", "action"},
|
|
||||||
)
|
|
||||||
|
|
||||||
m.registry.MustRegister(m.ruleMatches,
|
|
||||||
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
|
|
||||||
Name: "sww_rules_loaded",
|
|
||||||
Help: "Rules loaded from the rule files.",
|
|
||||||
}, func() float64 {
|
|
||||||
return float64(ruleFiles.Len())
|
|
||||||
}))
|
|
||||||
}
|
|
||||||
|
|
||||||
// AddRemoteLog adds the metrics of sending the log lines to
|
|
||||||
// SWW_WAF_LOG_REMOTE_URL, read from remote as the metrics are asked for: the
|
|
||||||
// lines sent, those dropped, and those waiting in the buffer.
|
|
||||||
func (m *Metrics) AddRemoteLog(remote *remotelog.Sender) {
|
|
||||||
m.registry.MustRegister(
|
|
||||||
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
|
||||||
Name: "sww_remote_log_lines_sent_total",
|
|
||||||
Help: "Log lines sent to SWW_WAF_LOG_REMOTE_URL.",
|
|
||||||
}, func() float64 {
|
|
||||||
return float64(remote.Sent())
|
|
||||||
}),
|
|
||||||
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
|
||||||
Name: "sww_remote_log_lines_dropped_total",
|
|
||||||
Help: "Log lines dropped: the oldest in a full buffer, and those " +
|
|
||||||
"whose sending failed.",
|
|
||||||
}, func() float64 {
|
|
||||||
return float64(remote.Dropped())
|
|
||||||
}),
|
|
||||||
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
|
|
||||||
Name: "sww_remote_log_lines_waiting",
|
|
||||||
Help: "Log lines in the buffer, waiting to be sent.",
|
|
||||||
}, func() float64 {
|
|
||||||
return float64(remote.Depth())
|
|
||||||
}),
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
// AddLookupFile adds the metrics of the lookup database, read as the
|
|
||||||
// metrics are asked for: when the file in use was read, which lastRead
|
|
||||||
// returns, and the replacements of it that could not be read, which
|
|
||||||
// readFailures returns.
|
|
||||||
func (m *Metrics) AddLookupFile(
|
|
||||||
lastRead func() time.Time,
|
|
||||||
readFailures func() int,
|
|
||||||
) {
|
|
||||||
m.registry.MustRegister(
|
|
||||||
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
|
|
||||||
Name: "sww_lookup_database_last_read_timestamp_seconds",
|
|
||||||
Help: "When the lookup database in use was read, in seconds since 1970.",
|
|
||||||
}, func() float64 {
|
|
||||||
return float64(lastRead().Unix())
|
|
||||||
}),
|
|
||||||
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
|
||||||
Name: "sww_lookup_database_read_failures_total",
|
|
||||||
Help: "Replacements of the lookup database that could not be read.",
|
|
||||||
}, func() float64 {
|
|
||||||
return float64(readFailures())
|
|
||||||
}),
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
// sourceLabel is the label of the reputation metrics: a list's URL, a
|
|
||||||
// DNSBL zone, its key masked, or abuseipdb.
|
|
||||||
const sourceLabel = "source"
|
|
||||||
|
|
||||||
// AddReputation adds the metrics of the lists fetched from URLs and of the
|
|
||||||
// DNSBL zones, by source, each list's URL or each zone, its key masked as
|
|
||||||
// reputation.MaskZoneKey masks it: the requests whose client a blocklist, a
|
|
||||||
// zone's verdict or AbuseIPDB's score lists, which ReputationHit counts,
|
|
||||||
// and, read from lists and dnsbl as the metrics are asked for, for a list,
|
|
||||||
// the fetches that failed and when the copy in use was fetched, and for a
|
|
||||||
// zone, the queries made and those that failed. It is called once, before
|
|
||||||
// ReputationHit.
|
|
||||||
func (m *Metrics) AddReputation(
|
|
||||||
lists *reputation.Lists,
|
|
||||||
dnsbl *reputation.DNSBL,
|
|
||||||
) {
|
|
||||||
m.reputationHits = counterVec(
|
|
||||||
"sww_reputation_hits_total",
|
|
||||||
"Requests whose client a blocklist, a DNSBL zone or AbuseIPDB lists, "+
|
|
||||||
"by the list's URL, the zone, or abuseipdb.",
|
|
||||||
[]string{sourceLabel},
|
|
||||||
)
|
|
||||||
m.registry.MustRegister(m.reputationHits)
|
|
||||||
|
|
||||||
for _, zone := range dnsbl.Zones() {
|
|
||||||
source := prometheus.Labels{
|
|
||||||
sourceLabel: reputation.MaskZoneKey(zone),
|
|
||||||
}
|
|
||||||
|
|
||||||
m.addReputationQueries(
|
|
||||||
source,
|
|
||||||
func() int { return dnsbl.Queries(zone) },
|
|
||||||
)
|
|
||||||
m.addReputationFailures(
|
|
||||||
source,
|
|
||||||
func() int { return dnsbl.Failures(zone) },
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, listURL := range lists.URLs() {
|
|
||||||
source := prometheus.Labels{sourceLabel: listURL}
|
|
||||||
|
|
||||||
m.addReputationFailures(
|
|
||||||
source,
|
|
||||||
func() int { return lists.Failures(listURL) },
|
|
||||||
)
|
|
||||||
m.registry.MustRegister(
|
|
||||||
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
|
|
||||||
Name: "sww_reputation_last_fetch_timestamp_seconds",
|
|
||||||
Help: "When the copy of the list in use was fetched, in seconds since " +
|
|
||||||
"1970, or 0 while there is none.",
|
|
||||||
ConstLabels: source,
|
|
||||||
}, func() float64 {
|
|
||||||
fetched := lists.Fetched(listURL)
|
|
||||||
if fetched.IsZero() {
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
||||||
return float64(fetched.Unix())
|
|
||||||
}),
|
|
||||||
)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// AddAbuseIPDB adds the metrics of AbuseIPDB, with the source abuseipdb,
|
|
||||||
// read from abuseIPDB as the metrics are asked for: the checks made, those
|
|
||||||
// that failed, and how many checks the day's budget has left. It is
|
|
||||||
// called once, after AddReputation, while SWW_WAF_ABUSEIPDB_KEY is set.
|
|
||||||
func (m *Metrics) AddAbuseIPDB(abuseIPDB *reputation.AbuseIPDB) {
|
|
||||||
source := prometheus.Labels{sourceLabel: reputation.AbuseIPDBSource}
|
|
||||||
|
|
||||||
m.addReputationQueries(source, abuseIPDB.Checked)
|
|
||||||
m.addReputationFailures(source, abuseIPDB.Failures)
|
|
||||||
m.registry.MustRegister(
|
|
||||||
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
|
|
||||||
Name: "sww_reputation_daily_budget_remaining",
|
|
||||||
Help: "Checks of the day's SWW_WAF_ABUSEIPDB_DAILY_BUDGET not yet spent.",
|
|
||||||
ConstLabels: source,
|
|
||||||
}, func() float64 {
|
|
||||||
return float64(abuseIPDB.BudgetLeft())
|
|
||||||
}),
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
// ReputationHit counts a request whose client source lists: a blocklist,
|
|
||||||
// by its URL, a DNSBL zone, its key masked, or AbuseIPDB, abuseipdb.
|
|
||||||
func (m *Metrics) ReputationHit(source string) {
|
|
||||||
m.reputationHits.WithLabelValues(source).Inc()
|
|
||||||
}
|
|
||||||
|
|
||||||
// AddAlerts adds the metrics of the alerts sent to each destination set,
|
|
||||||
// read from queue as the metrics are asked for, by destination: the
|
|
||||||
// alerts sent, the requests to the destination that failed, the alerts
|
|
||||||
// held back, which are the same for every destination, and those
|
|
||||||
// dropped. With no destination set, it adds none.
|
|
||||||
func (m *Metrics) AddAlerts(queue *alerts.Queue) {
|
|
||||||
for _, name := range queue.DestinationsSet() {
|
|
||||||
destination := prometheus.Labels{"destination": name}
|
|
||||||
|
|
||||||
m.registry.MustRegister(
|
|
||||||
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
|
||||||
Name: "sww_alerts_sent_total",
|
|
||||||
Help: "Alerts the destination took.",
|
|
||||||
ConstLabels: destination,
|
|
||||||
}, func() float64 {
|
|
||||||
return float64(queue.Counts(name).Sent)
|
|
||||||
}),
|
|
||||||
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
|
||||||
Name: "sww_alert_send_failures_total",
|
|
||||||
Help: "Requests to the destination that failed.",
|
|
||||||
ConstLabels: destination,
|
|
||||||
}, func() float64 {
|
|
||||||
return float64(queue.Counts(name).SendFailures)
|
|
||||||
}),
|
|
||||||
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
|
||||||
Name: "sww_alerts_suppressed_total",
|
|
||||||
Help: "Alerts held back: repeats within SWW_WAF_ALERT_COOLDOWN, and " +
|
|
||||||
"alerts past SWW_WAF_ALERT_MAX_PER_HOUR, for the hour's summary.",
|
|
||||||
ConstLabels: destination,
|
|
||||||
}, func() float64 {
|
|
||||||
return float64(queue.Suppressed())
|
|
||||||
}),
|
|
||||||
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
|
||||||
Name: "sww_alerts_dropped_total",
|
|
||||||
Help: "Alerts dropped, the oldest first, from a full queue, and " +
|
|
||||||
"alerts given up as the destination refused them.",
|
|
||||||
ConstLabels: destination,
|
|
||||||
}, func() float64 {
|
|
||||||
return float64(queue.Counts(name).Dropped)
|
|
||||||
}),
|
|
||||||
)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// ServeHTTP answers with the metrics in the Prometheus text format.
|
// ServeHTTP answers with the metrics in the Prometheus text format.
|
||||||
func (m *Metrics) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
func (m *Metrics) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||||
m.handler.ServeHTTP(w, r)
|
m.handler.ServeHTTP(w, r)
|
||||||
@@ -466,7 +280,7 @@ func (m *Metrics) RequestReported(
|
|||||||
// LimitHit counts a request that broke a rate limit, a byte limit or the
|
// LimitHit counts a request that broke a rate limit, a byte limit or the
|
||||||
// refusal limit, by the window and the kind of hit.
|
// refusal limit, by the window and the kind of hit.
|
||||||
func (m *Metrics) LimitHit(hit ratelimit.Hit) {
|
func (m *Metrics) LimitHit(hit ratelimit.Hit) {
|
||||||
m.rateLimitHits.WithLabelValues(hit.Window, hit.Kind).Inc()
|
m.rateLimitHits.WithLabelValues(hit.Window, string(hit.Kind)).Inc()
|
||||||
}
|
}
|
||||||
|
|
||||||
// Offences counts the offences of r, a request that has ended, as its
|
// Offences counts the offences of r, a request that has ended, as its
|
||||||
@@ -487,8 +301,8 @@ func (m *Metrics) Offences(r ratelimit.Request) {
|
|||||||
|
|
||||||
// RuleMatched counts a request that matched the rule id, whose action is
|
// RuleMatched counts a request that matched the rule id, whose action is
|
||||||
// action.
|
// action.
|
||||||
func (m *Metrics) RuleMatched(id, action string) {
|
func (m *Metrics) RuleMatched(id string, action rules.Action) {
|
||||||
m.ruleMatches.WithLabelValues(id, action).Inc()
|
m.ruleMatches.WithLabelValues(id, string(action)).Inc()
|
||||||
}
|
}
|
||||||
|
|
||||||
// CoreRuleSetMatched counts a request that matched the Core Rule Set's rule id,
|
// CoreRuleSetMatched counts a request that matched the Core Rule Set's rule id,
|
||||||
@@ -497,6 +311,12 @@ func (m *Metrics) CoreRuleSetMatched(mode string, id int) {
|
|||||||
m.coreRuleSetMatches.WithLabelValues(mode, strconv.Itoa(id)).Inc()
|
m.coreRuleSetMatches.WithLabelValues(mode, strconv.Itoa(id)).Inc()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ReputationHit counts a request whose client source lists: a blocklist,
|
||||||
|
// by its URL, a DNSBL zone, its key masked, or AbuseIPDB, abuseipdb.
|
||||||
|
func (m *Metrics) ReputationHit(source string) {
|
||||||
|
m.reputationHits.WithLabelValues(source).Inc()
|
||||||
|
}
|
||||||
|
|
||||||
// StateFileWritten counts a write of the state or cache file name, of size
|
// StateFileWritten counts a write of the state or cache file name, of size
|
||||||
// bytes, that ended with err.
|
// bytes, that ended with err.
|
||||||
func (m *Metrics) StateFileWritten(name string, size int, err error) {
|
func (m *Metrics) StateFileWritten(name string, size int, err error) {
|
||||||
@@ -528,16 +348,323 @@ func (m *Metrics) StateFileEditSetAside(name string) {
|
|||||||
m.stateFileEditsSetAside.WithLabelValues(name).Inc()
|
m.stateFileEditsSetAside.WithLabelValues(name).Inc()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// banLedger is what the metrics read from the ledger of bans.
|
||||||
|
type banLedger interface {
|
||||||
|
// Made returns the bans made since the start with cause.
|
||||||
|
Made(cause bans.Cause) int
|
||||||
|
// Count returns the bans active at now, the permanent ones included,
|
||||||
|
// and the permanent ones.
|
||||||
|
Count(now time.Time) (active, permanent int)
|
||||||
|
}
|
||||||
|
|
||||||
|
// addBans adds the metrics read from ledger as the metrics are asked for:
|
||||||
|
// the bans made since the start, by cause, and the bans active and
|
||||||
|
// permanent at now.
|
||||||
|
func (m *Metrics) addBans(ledger banLedger, now func() time.Time) {
|
||||||
|
for _, cause := range []bans.Cause{
|
||||||
|
bans.CauseLimit, bans.CauseAttack, bans.CauseAdmin, bans.CauseAbuseIPDB,
|
||||||
|
} {
|
||||||
|
m.registry.MustRegister(
|
||||||
|
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
||||||
|
Name: "sww_bans_made_total",
|
||||||
|
Help: "Bans made, by cause.",
|
||||||
|
ConstLabels: prometheus.Labels{"cause": string(cause)},
|
||||||
|
}, func() float64 {
|
||||||
|
return float64(ledger.Made(cause))
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
m.registry.MustRegister(
|
||||||
|
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
|
||||||
|
Name: "sww_active_bans",
|
||||||
|
Help: "Bans active now, the permanent ones included.",
|
||||||
|
}, func() float64 {
|
||||||
|
active, _ := ledger.Count(now())
|
||||||
|
|
||||||
|
return float64(active)
|
||||||
|
}),
|
||||||
|
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
|
||||||
|
Name: "sww_permanent_bans",
|
||||||
|
Help: "Permanent bans not lifted.",
|
||||||
|
}, func() float64 {
|
||||||
|
_, permanent := ledger.Count(now())
|
||||||
|
|
||||||
|
return float64(permanent)
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// sized is what the metrics read from the table of clients and from the
|
||||||
|
// rule files.
|
||||||
|
type sized interface {
|
||||||
|
// Len returns how many clients the table holds, or how many rules were
|
||||||
|
// loaded from the rule files.
|
||||||
|
Len() int
|
||||||
|
}
|
||||||
|
|
||||||
|
// addClients adds the metric of the clients in table, read as the metrics
|
||||||
|
// are asked for.
|
||||||
|
func (m *Metrics) addClients(table sized) {
|
||||||
|
m.registry.MustRegister(
|
||||||
|
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
|
||||||
|
Name: "sww_tracked_clients",
|
||||||
|
Help: "Clients in the table of clients.",
|
||||||
|
}, func() float64 {
|
||||||
|
return float64(table.Len())
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// addRules adds the metric of the rules loaded from files, read as the
|
||||||
|
// metrics are asked for.
|
||||||
|
func (m *Metrics) addRules(files sized) {
|
||||||
|
m.registry.MustRegister(
|
||||||
|
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
|
||||||
|
Name: "sww_rules_loaded",
|
||||||
|
Help: "Rules loaded from the rule files.",
|
||||||
|
}, func() float64 {
|
||||||
|
return float64(files.Len())
|
||||||
|
}))
|
||||||
|
}
|
||||||
|
|
||||||
|
// remoteLogSender is what the metrics read from what sends the log lines to
|
||||||
|
// SWW_WAF_LOG_REMOTE_URL.
|
||||||
|
type remoteLogSender interface {
|
||||||
|
// Sent returns the lines sent.
|
||||||
|
Sent() int64
|
||||||
|
// Dropped returns the lines dropped.
|
||||||
|
Dropped() int64
|
||||||
|
// Depth returns the lines waiting in the buffer.
|
||||||
|
Depth() int
|
||||||
|
}
|
||||||
|
|
||||||
|
// addRemoteLog adds the metrics of sending the log lines to
|
||||||
|
// SWW_WAF_LOG_REMOTE_URL, read from remote as the metrics are asked for: the
|
||||||
|
// lines sent, those dropped, and those waiting in the buffer.
|
||||||
|
func (m *Metrics) addRemoteLog(remote remoteLogSender) {
|
||||||
|
m.registry.MustRegister(
|
||||||
|
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
||||||
|
Name: "sww_remote_log_lines_sent_total",
|
||||||
|
Help: "Log lines sent to SWW_WAF_LOG_REMOTE_URL.",
|
||||||
|
}, func() float64 {
|
||||||
|
return float64(remote.Sent())
|
||||||
|
}),
|
||||||
|
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
||||||
|
Name: "sww_remote_log_lines_dropped_total",
|
||||||
|
Help: "Log lines dropped: the oldest in a full buffer, and those " +
|
||||||
|
"whose sending failed.",
|
||||||
|
}, func() float64 {
|
||||||
|
return float64(remote.Dropped())
|
||||||
|
}),
|
||||||
|
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
|
||||||
|
Name: "sww_remote_log_lines_waiting",
|
||||||
|
Help: "Log lines in the buffer, waiting to be sent.",
|
||||||
|
}, func() float64 {
|
||||||
|
return float64(remote.Depth())
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// lookupDatabase is what the metrics read from the lookup database.
|
||||||
|
type lookupDatabase interface {
|
||||||
|
// LastRead returns when the file in use was read.
|
||||||
|
LastRead() time.Time
|
||||||
|
// ReadFailures returns the replacements of the file that could not be
|
||||||
|
// read.
|
||||||
|
ReadFailures() int
|
||||||
|
}
|
||||||
|
|
||||||
|
// addLookupFile adds the metrics of the lookup database, read from file as
|
||||||
|
// the metrics are asked for: when the file in use was read, and the
|
||||||
|
// replacements of it that could not be read.
|
||||||
|
func (m *Metrics) addLookupFile(file lookupDatabase) {
|
||||||
|
m.registry.MustRegister(
|
||||||
|
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
|
||||||
|
Name: "sww_lookup_database_last_read_timestamp_seconds",
|
||||||
|
Help: "When the lookup database in use was read, in seconds since 1970.",
|
||||||
|
}, func() float64 {
|
||||||
|
return float64(file.LastRead().Unix())
|
||||||
|
}),
|
||||||
|
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
||||||
|
Name: "sww_lookup_database_read_failures_total",
|
||||||
|
Help: "Replacements of the lookup database that could not be read.",
|
||||||
|
}, func() float64 {
|
||||||
|
return float64(file.ReadFailures())
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// sourceLabel is the label of the reputation metrics: a list's URL, a
|
||||||
|
// DNSBL zone, its key masked, or abuseipdb.
|
||||||
|
const sourceLabel = "source"
|
||||||
|
|
||||||
|
// dnsblZones is what the metrics read from what asks the DNSBL zones.
|
||||||
|
type dnsblZones interface {
|
||||||
|
// Zones returns the zones asked, their keys unmasked.
|
||||||
|
Zones() []string
|
||||||
|
// Queries returns the queries made to zone.
|
||||||
|
Queries(zone string) int
|
||||||
|
// Failures returns the queries to zone that failed.
|
||||||
|
Failures(zone string) int
|
||||||
|
}
|
||||||
|
|
||||||
|
// addDNSBL adds the metrics of each zone dnsbl asks, by source, the zone,
|
||||||
|
// its key masked as reputation.MaskZoneKey masks it, read as the metrics are
|
||||||
|
// asked for: the queries made and those that failed.
|
||||||
|
func (m *Metrics) addDNSBL(dnsbl dnsblZones) {
|
||||||
|
for _, zone := range dnsbl.Zones() {
|
||||||
|
source := prometheus.Labels{
|
||||||
|
sourceLabel: reputation.MaskZoneKey(zone),
|
||||||
|
}
|
||||||
|
|
||||||
|
m.addReputationQueries(
|
||||||
|
source,
|
||||||
|
func() int { return dnsbl.Queries(zone) },
|
||||||
|
)
|
||||||
|
m.addReputationFailures(
|
||||||
|
source,
|
||||||
|
func() int { return dnsbl.Failures(zone) },
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// fetchedLists is what the metrics read from the lists fetched from URLs.
|
||||||
|
type fetchedLists interface {
|
||||||
|
// URLs returns the lists' URLs.
|
||||||
|
URLs() []string
|
||||||
|
// Failures returns the fetches of the list at listURL that failed.
|
||||||
|
Failures(listURL string) int
|
||||||
|
// Fetched returns when the copy in use of the list at listURL was
|
||||||
|
// fetched, or the zero time while there is none.
|
||||||
|
Fetched(listURL string) time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
// addLists adds the metrics of each list fetched from a URL, by source, the
|
||||||
|
// list's URL, read from lists as the metrics are asked for: the fetches
|
||||||
|
// that failed and when the copy in use was fetched.
|
||||||
|
func (m *Metrics) addLists(lists fetchedLists) {
|
||||||
|
for _, listURL := range lists.URLs() {
|
||||||
|
source := prometheus.Labels{sourceLabel: listURL}
|
||||||
|
|
||||||
|
m.addReputationFailures(
|
||||||
|
source,
|
||||||
|
func() int { return lists.Failures(listURL) },
|
||||||
|
)
|
||||||
|
m.registry.MustRegister(
|
||||||
|
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
|
||||||
|
Name: "sww_reputation_last_fetch_timestamp_seconds",
|
||||||
|
Help: "When the copy of the list in use was fetched, in seconds since " +
|
||||||
|
"1970, or 0 while there is none.",
|
||||||
|
ConstLabels: source,
|
||||||
|
}, func() float64 {
|
||||||
|
fetched := lists.Fetched(listURL)
|
||||||
|
if fetched.IsZero() {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
return float64(fetched.Unix())
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// abuseIPDBChecker is what the metrics read from what checks clients with
|
||||||
|
// AbuseIPDB.
|
||||||
|
type abuseIPDBChecker interface {
|
||||||
|
// Checked returns the checks made.
|
||||||
|
Checked() int
|
||||||
|
// Failures returns the checks that failed.
|
||||||
|
Failures() int
|
||||||
|
// BudgetLeft returns how many checks the day's budget has left.
|
||||||
|
BudgetLeft() int
|
||||||
|
}
|
||||||
|
|
||||||
|
// addAbuseIPDB adds the metrics of AbuseIPDB, with the source abuseipdb,
|
||||||
|
// read from abuseIPDB as the metrics are asked for: the checks made, those
|
||||||
|
// that failed, and how many checks the day's budget has left.
|
||||||
|
func (m *Metrics) addAbuseIPDB(abuseIPDB abuseIPDBChecker) {
|
||||||
|
source := prometheus.Labels{sourceLabel: reputation.AbuseIPDBSource}
|
||||||
|
|
||||||
|
m.addReputationQueries(source, abuseIPDB.Checked)
|
||||||
|
m.addReputationFailures(source, abuseIPDB.Failures)
|
||||||
|
m.registry.MustRegister(
|
||||||
|
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
|
||||||
|
Name: "sww_reputation_daily_budget_remaining",
|
||||||
|
Help: "Checks of the day's SWW_WAF_ABUSEIPDB_DAILY_BUDGET not yet spent.",
|
||||||
|
ConstLabels: source,
|
||||||
|
}, func() float64 {
|
||||||
|
return float64(abuseIPDB.BudgetLeft())
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// alertQueue is what the metrics read from the queue of alerts.
|
||||||
|
type alertQueue interface {
|
||||||
|
// DestinationsSet returns the names of the destinations set.
|
||||||
|
DestinationsSet() []alerts.DestinationName
|
||||||
|
// Counts returns the alerts sent to the destination name, the requests
|
||||||
|
// to it that failed, and the alerts dropped.
|
||||||
|
Counts(name alerts.DestinationName) alerts.Counts
|
||||||
|
// Suppressed returns the alerts held back.
|
||||||
|
Suppressed() int64
|
||||||
|
}
|
||||||
|
|
||||||
|
// addAlerts adds the metrics of the alerts sent to each destination set,
|
||||||
|
// read from queue as the metrics are asked for, by destination: the
|
||||||
|
// alerts sent, the requests to the destination that failed, the alerts
|
||||||
|
// held back, which are the same for every destination, and those
|
||||||
|
// dropped. With no destination set, it adds none.
|
||||||
|
func (m *Metrics) addAlerts(queue alertQueue) {
|
||||||
|
for _, name := range queue.DestinationsSet() {
|
||||||
|
destination := prometheus.Labels{"destination": string(name)}
|
||||||
|
|
||||||
|
m.registry.MustRegister(
|
||||||
|
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
||||||
|
Name: "sww_alerts_sent_total",
|
||||||
|
Help: "Alerts the destination took.",
|
||||||
|
ConstLabels: destination,
|
||||||
|
}, func() float64 {
|
||||||
|
return float64(queue.Counts(name).Sent)
|
||||||
|
}),
|
||||||
|
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
||||||
|
Name: "sww_alert_send_failures_total",
|
||||||
|
Help: "Requests to the destination that failed.",
|
||||||
|
ConstLabels: destination,
|
||||||
|
}, func() float64 {
|
||||||
|
return float64(queue.Counts(name).SendFailures)
|
||||||
|
}),
|
||||||
|
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
||||||
|
Name: "sww_alerts_suppressed_total",
|
||||||
|
Help: "Alerts held back: repeats within SWW_WAF_ALERT_COOLDOWN, and " +
|
||||||
|
"alerts past SWW_WAF_ALERT_MAX_PER_HOUR, for the hour's summary.",
|
||||||
|
ConstLabels: destination,
|
||||||
|
}, func() float64 {
|
||||||
|
return float64(queue.Suppressed())
|
||||||
|
}),
|
||||||
|
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
||||||
|
Name: "sww_alerts_dropped_total",
|
||||||
|
Help: "Alerts dropped, the oldest first, from a full queue, and " +
|
||||||
|
"alerts given up as the destination refused them.",
|
||||||
|
ConstLabels: destination,
|
||||||
|
}, func() float64 {
|
||||||
|
return float64(queue.Counts(name).Dropped)
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// count counts a request that has ended, as RequestEnded and RequestReported
|
// count counts a request that has ended, as RequestEnded and RequestReported
|
||||||
// describe.
|
// describe.
|
||||||
func (m *Metrics) count(
|
func (m *Metrics) count(
|
||||||
line *requestlog.Line, duration, upstreamDuration time.Duration,
|
line *requestlog.Line, duration, upstreamDuration time.Duration,
|
||||||
) {
|
) {
|
||||||
class := statusClass(line.Status)
|
class := statusClass(line.Status)
|
||||||
m.requests.WithLabelValues(class, line.Action).Inc()
|
action := string(line.Action)
|
||||||
m.requestBytes.WithLabelValues(class, line.Action).
|
m.requests.WithLabelValues(class, action).Inc()
|
||||||
|
m.requestBytes.WithLabelValues(class, action).
|
||||||
Add(float64(line.RequestBytes))
|
Add(float64(line.RequestBytes))
|
||||||
m.responseBytes.WithLabelValues(class, line.Action).
|
m.responseBytes.WithLabelValues(class, action).
|
||||||
Add(float64(line.ResponseBytes))
|
Add(float64(line.ResponseBytes))
|
||||||
m.requestDuration.Observe(duration.Seconds())
|
m.requestDuration.Observe(duration.Seconds())
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,161 @@
|
|||||||
|
package metrics_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
listURL = "https://lists.example/block.txt"
|
||||||
|
zone = "dnsbl.example"
|
||||||
|
destination = "webhook"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Each stand-in answers with a number of its own, so that each metric
|
||||||
|
// shows which part it was read from.
|
||||||
|
|
||||||
|
type ledgerStandIn struct{}
|
||||||
|
|
||||||
|
func (ledgerStandIn) Made(bans.Cause) int { return 1 }
|
||||||
|
|
||||||
|
func (ledgerStandIn) Count(time.Time) (int, int) { return 2, 1 }
|
||||||
|
|
||||||
|
type clientTableStandIn struct{}
|
||||||
|
|
||||||
|
func (clientTableStandIn) Len() int { return 3 }
|
||||||
|
|
||||||
|
type ruleFilesStandIn struct{}
|
||||||
|
|
||||||
|
func (ruleFilesStandIn) Len() int { return 4 }
|
||||||
|
|
||||||
|
type listsStandIn struct{}
|
||||||
|
|
||||||
|
func (listsStandIn) URLs() []string { return []string{listURL} }
|
||||||
|
|
||||||
|
func (listsStandIn) Failures(string) int { return 5 }
|
||||||
|
|
||||||
|
func (listsStandIn) Fetched(string) time.Time { return time.Time{} }
|
||||||
|
|
||||||
|
type dnsblStandIn struct{}
|
||||||
|
|
||||||
|
func (dnsblStandIn) Zones() []string { return []string{zone} }
|
||||||
|
|
||||||
|
func (dnsblStandIn) Queries(string) int { return 6 }
|
||||||
|
|
||||||
|
func (dnsblStandIn) Failures(string) int { return 0 }
|
||||||
|
|
||||||
|
type abuseIPDBStandIn struct{}
|
||||||
|
|
||||||
|
func (abuseIPDBStandIn) Checked() int { return 0 }
|
||||||
|
|
||||||
|
func (abuseIPDBStandIn) Failures() int { return 0 }
|
||||||
|
|
||||||
|
func (abuseIPDBStandIn) BudgetLeft() int { return 7 }
|
||||||
|
|
||||||
|
type alertQueueStandIn struct{}
|
||||||
|
|
||||||
|
func (alertQueueStandIn) DestinationsSet() []alerts.DestinationName {
|
||||||
|
return []alerts.DestinationName{destination}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (alertQueueStandIn) Counts(alerts.DestinationName) alerts.Counts {
|
||||||
|
return alerts.Counts{Sent: 8}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (alertQueueStandIn) Suppressed() int64 { return 0 }
|
||||||
|
|
||||||
|
type lookupDatabaseStandIn struct{}
|
||||||
|
|
||||||
|
func (lookupDatabaseStandIn) LastRead() time.Time { return time.Time{} }
|
||||||
|
|
||||||
|
func (lookupDatabaseStandIn) ReadFailures() int { return 9 }
|
||||||
|
|
||||||
|
type remoteLogStandIn struct{}
|
||||||
|
|
||||||
|
func (remoteLogStandIn) Sent() int64 { return 10 }
|
||||||
|
|
||||||
|
func (remoteLogStandIn) Dropped() int64 { return 0 }
|
||||||
|
|
||||||
|
func (remoteLogStandIn) Depth() int { return 0 }
|
||||||
|
|
||||||
|
func TestMetricsAreReadFromEachPart(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
m := metrics.New(metrics.Params{
|
||||||
|
BusiestCount: 1,
|
||||||
|
InstanceName: "app",
|
||||||
|
Now: func() time.Time { return time.Time{} },
|
||||||
|
Bans: ledgerStandIn{},
|
||||||
|
Clients: clientTableStandIn{},
|
||||||
|
Rules: ruleFilesStandIn{},
|
||||||
|
Lists: listsStandIn{},
|
||||||
|
DNSBL: dnsblStandIn{},
|
||||||
|
AbuseIPDB: abuseIPDBStandIn{},
|
||||||
|
Alerts: alertQueueStandIn{},
|
||||||
|
LookupFile: lookupDatabaseStandIn{},
|
||||||
|
RemoteLog: remoteLogStandIn{},
|
||||||
|
})
|
||||||
|
scraped := scrape(t, m)
|
||||||
|
|
||||||
|
for _, line := range []string{
|
||||||
|
`sww_bans_made_total{cause="limit",instance="app"} 1`,
|
||||||
|
`sww_active_bans{instance="app"} 2`,
|
||||||
|
`sww_tracked_clients{instance="app"} 3`,
|
||||||
|
`sww_rules_loaded{instance="app"} 4`,
|
||||||
|
`sww_reputation_failures_total{instance="app",source="` + listURL + `"} 5`,
|
||||||
|
`sww_reputation_queries_total{instance="app",source="` + zone + `"} 6`,
|
||||||
|
`sww_reputation_daily_budget_remaining{instance="app",source="abuseipdb"} 7`,
|
||||||
|
`sww_alerts_sent_total{destination="` + destination + `",instance="app"} 8`,
|
||||||
|
`sww_lookup_database_read_failures_total{instance="app"} 9`,
|
||||||
|
`sww_remote_log_lines_sent_total{instance="app"} 10`,
|
||||||
|
} {
|
||||||
|
if !strings.Contains(scraped, "\n"+line+"\n") {
|
||||||
|
t.Errorf("metrics\n%s\nwant\n%s", scraped, line)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPartThatIsOffAddsNoMetric(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
scraped := scrape(t, metrics.New(metrics.Params{
|
||||||
|
BusiestCount: 1,
|
||||||
|
InstanceName: "app",
|
||||||
|
}))
|
||||||
|
|
||||||
|
for _, name := range []string{
|
||||||
|
"sww_bans_made_total",
|
||||||
|
"sww_tracked_clients",
|
||||||
|
"sww_rules_loaded",
|
||||||
|
"sww_reputation_",
|
||||||
|
"sww_alert",
|
||||||
|
"sww_lookup_database_",
|
||||||
|
"sww_remote_log_",
|
||||||
|
} {
|
||||||
|
if strings.Contains(scraped, name) {
|
||||||
|
t.Errorf("metrics\n%s\nshow %s", scraped, name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// scrape returns the metrics m serves.
|
||||||
|
func scrape(t *testing.T, m *metrics.Metrics) string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
scraped := httptest.NewRecorder()
|
||||||
|
m.ServeHTTP(
|
||||||
|
scraped,
|
||||||
|
httptest.NewRequestWithContext(
|
||||||
|
t.Context(), http.MethodGet, "/", http.NoBody,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
return scraped.Body.String()
|
||||||
|
}
|
||||||
@@ -19,15 +19,19 @@ import (
|
|||||||
|
|
||||||
const day = 24 * time.Hour
|
const day = 24 * time.Hour
|
||||||
|
|
||||||
|
// LimitKind is what a limit counts: KindRequests, KindBytes or
|
||||||
|
// KindRefusals.
|
||||||
|
type LimitKind string
|
||||||
|
|
||||||
// The kinds of limits, as the metrics name them.
|
// The kinds of limits, as the metrics name them.
|
||||||
const (
|
const (
|
||||||
// KindRequests is a rate limit, on a client's requests.
|
// KindRequests is a rate limit, on a client's requests.
|
||||||
KindRequests = "requests"
|
KindRequests LimitKind = "requests"
|
||||||
// KindBytes is a byte limit, on a client's bytes.
|
// KindBytes is a byte limit, on a client's bytes.
|
||||||
KindBytes = "bytes"
|
KindBytes LimitKind = "bytes"
|
||||||
// KindRefusals is the refusal limit, on a client's requests sww
|
// KindRefusals is the refusal limit, on a client's requests sww
|
||||||
// refused after a rule file match or for a missing or wrong token.
|
// refused after a rule file match or for a missing or wrong token.
|
||||||
KindRefusals = "refusals"
|
KindRefusals LimitKind = "refusals"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Limits are the most requests a client may make in a minute, an hour and
|
// Limits are the most requests a client may make in a minute, an hour and
|
||||||
@@ -200,9 +204,7 @@ func New(params Params) *Limiter {
|
|||||||
// Hit is a request that takes a client over a rate limit or the refusal
|
// Hit is a request that takes a client over a rate limit or the refusal
|
||||||
// limit, or whose bytes take it over a byte limit.
|
// limit, or whose bytes take it over a byte limit.
|
||||||
type Hit struct {
|
type Hit struct {
|
||||||
// Kind is KindRequests for a rate limit, KindBytes for a byte limit,
|
Kind LimitKind
|
||||||
// KindRefusals for the refusal limit.
|
|
||||||
Kind string
|
|
||||||
// Window is "minute", "hour" or "day".
|
// Window is "minute", "hour" or "day".
|
||||||
Window string
|
Window string
|
||||||
// Limit is the window's limit, as the client's percentage of it.
|
// Limit is the window's limit, as the client's percentage of it.
|
||||||
|
|||||||
@@ -652,7 +652,7 @@ func wantBytesCount(
|
|||||||
client netip.Prefix,
|
client netip.Prefix,
|
||||||
now time.Time,
|
now time.Time,
|
||||||
bytes int64,
|
bytes int64,
|
||||||
want string,
|
want ratelimit.LimitKind,
|
||||||
) {
|
) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
|
|||||||
@@ -445,12 +445,9 @@ func TestKeyIsSentInTheKeyHeaderAndNeverShown(t *testing.T) {
|
|||||||
scores: map[string]int64{suspect: 100},
|
scores: map[string]int64{suspect: 100},
|
||||||
}
|
}
|
||||||
checker := newAbuseIPDB(abuseIPDB, p)
|
checker := newAbuseIPDB(abuseIPDB, p)
|
||||||
m := metrics.New(1, "app")
|
m := metrics.New(metrics.Params{
|
||||||
m.AddReputation(
|
BusiestCount: 1, InstanceName: instanceName, AbuseIPDB: checker,
|
||||||
reputation.New(params()),
|
})
|
||||||
reputation.NewDNSBL(dnsblParams()),
|
|
||||||
)
|
|
||||||
m.AddAbuseIPDB(checker)
|
|
||||||
|
|
||||||
// One check that AbuseIPDB answers, and one it refuses with an answer
|
// One check that AbuseIPDB answers, and one it refuses with an answer
|
||||||
// that names the key.
|
// that names the key.
|
||||||
@@ -503,12 +500,9 @@ func TestMetricsCountTheChecksTheFailuresAndTheBudgetLeft(t *testing.T) {
|
|||||||
p := abuseIPDBParams()
|
p := abuseIPDBParams()
|
||||||
p.DailyBudget = 5
|
p.DailyBudget = 5
|
||||||
checker := newAbuseIPDB(abuseIPDB, p)
|
checker := newAbuseIPDB(abuseIPDB, p)
|
||||||
m := metrics.New(1, "app")
|
m := metrics.New(metrics.Params{
|
||||||
m.AddReputation(
|
BusiestCount: 1, InstanceName: instanceName, AbuseIPDB: checker,
|
||||||
reputation.New(params()),
|
})
|
||||||
reputation.NewDNSBL(dnsblParams()),
|
|
||||||
)
|
|
||||||
m.AddAbuseIPDB(checker)
|
|
||||||
|
|
||||||
// One check that AbuseIPDB answers, and one that fails.
|
// One check that AbuseIPDB answers, and one that fails.
|
||||||
hitFrom(t, checker, suspect, true)
|
hitFrom(t, checker, suspect, true)
|
||||||
@@ -811,6 +805,9 @@ func wantBudgetLeft(t *testing.T, checker *reputation.AbuseIPDB, want int) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// instanceName is the label instance of the metrics the tests build.
|
||||||
|
const instanceName = "app"
|
||||||
|
|
||||||
// scrapeMetrics returns the metrics m serves.
|
// scrapeMetrics returns the metrics m serves.
|
||||||
func scrapeMetrics(t *testing.T, m *metrics.Metrics) string {
|
func scrapeMetrics(t *testing.T, m *metrics.Metrics) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|||||||
@@ -358,8 +358,9 @@ func TestMetricsCountEachZonesQueriesAndThoseThatFailed(t *testing.T) {
|
|||||||
"99.2.0.192." + otherZone + ".": {rcode: serverFailure},
|
"99.2.0.192." + otherZone + ".": {rcode: serverFailure},
|
||||||
}}
|
}}
|
||||||
dnsbl := newDNSBL(resolver, dnsblParams(zone, otherZone))
|
dnsbl := newDNSBL(resolver, dnsblParams(zone, otherZone))
|
||||||
m := metrics.New(1, "app")
|
m := metrics.New(metrics.Params{
|
||||||
m.AddReputation(reputation.New(params()), dnsbl)
|
BusiestCount: 1, InstanceName: instanceName, DNSBL: dnsbl,
|
||||||
|
})
|
||||||
|
|
||||||
wantZones(t, dnsbl, listed)
|
wantZones(t, dnsbl, listed)
|
||||||
synctest.Wait()
|
synctest.Wait()
|
||||||
@@ -404,8 +405,9 @@ func TestZoneKeyIsMaskedInTheVerdictsTheFailuresAndTheMetrics(t *testing.T) {
|
|||||||
"99.2.0.192." + keyed + ".": {addrs: []string{listing}},
|
"99.2.0.192." + keyed + ".": {addrs: []string{listing}},
|
||||||
"100.2.0.192." + keyed + ".": {rcode: serverFailure},
|
"100.2.0.192." + keyed + ".": {rcode: serverFailure},
|
||||||
}}, p)
|
}}, p)
|
||||||
m := metrics.New(1, "app")
|
m := metrics.New(metrics.Params{
|
||||||
m.AddReputation(reputation.New(params()), dnsbl)
|
BusiestCount: 1, InstanceName: instanceName, DNSBL: dnsbl,
|
||||||
|
})
|
||||||
|
|
||||||
// Both clients are asked about before either answer comes, so that
|
// Both clients are asked about before either answer comes, so that
|
||||||
// the failure does not keep the zone from the other query.
|
// the failure does not keep the zone from the other query.
|
||||||
|
|||||||
@@ -12,82 +12,94 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// Action is what waf did with a request, as its request line names it,
|
||||||
|
// such as ActionForward.
|
||||||
|
type Action string
|
||||||
|
|
||||||
// The action a request line names: what waf did with the request.
|
// The action a request line names: what waf did with the request.
|
||||||
const (
|
const (
|
||||||
// ActionForward is a request waf allowed, which rproxy passes to
|
// ActionForward is a request waf allowed, which rproxy passes to
|
||||||
// the app.
|
// the app.
|
||||||
ActionForward = "forward"
|
ActionForward Action = "forward"
|
||||||
|
|
||||||
// ActionRequestLineTooLong is a request whose request line is too long for
|
// ActionRequestLineTooLong is a request whose request line is too long for
|
||||||
// the header limits.
|
// the header limits.
|
||||||
ActionRequestLineTooLong = "request_line_too_long"
|
ActionRequestLineTooLong Action = "request_line_too_long"
|
||||||
|
|
||||||
// ActionRequestHeaderTooLarge is a request whose headers are over the
|
// ActionRequestHeaderTooLarge is a request whose headers are over the
|
||||||
// header limits.
|
// header limits.
|
||||||
ActionRequestHeaderTooLarge = "request_header_too_large"
|
ActionRequestHeaderTooLarge Action = "request_header_too_large"
|
||||||
|
|
||||||
// ActionClientIsRateLimited is a request refused because it took its
|
// ActionClientIsRateLimited is a request refused because it took its
|
||||||
// client over a rate limit, which bans the client.
|
// client over a rate limit, which bans the client.
|
||||||
ActionClientIsRateLimited = "client_is_rate_limited"
|
ActionClientIsRateLimited Action = "client_is_rate_limited"
|
||||||
|
|
||||||
// ActionTemporarilyBanned is a request refused because a ban that runs
|
// ActionTemporarilyBanned is a request refused because a ban that runs
|
||||||
// out covers its client, or because it matched a ban rule or asked for a
|
// out covers its client, or because it matched a ban rule or asked for a
|
||||||
// trap path, and so made such a ban.
|
// trap path, and so made such a ban.
|
||||||
ActionTemporarilyBanned = "temporarily_banned"
|
ActionTemporarilyBanned Action = "temporarily_banned"
|
||||||
|
|
||||||
// ActionPermanentlyBanned is a request refused because a ban that never
|
// ActionPermanentlyBanned is a request refused because a ban that never
|
||||||
// runs out covers its client, or because the request made such a ban or
|
// runs out covers its client, or because the request made such a ban or
|
||||||
// made its client's ban permanent.
|
// made its client's ban permanent.
|
||||||
ActionPermanentlyBanned = "permanently_banned"
|
ActionPermanentlyBanned Action = "permanently_banned"
|
||||||
|
|
||||||
// ActionBlockedByBlockRule is a request refused because it matched a
|
// ActionBlockedByBlockRule is a request refused because it matched a
|
||||||
// block rule.
|
// block rule.
|
||||||
ActionBlockedByBlockRule = "blocked_by_block_rule"
|
ActionBlockedByBlockRule Action = "blocked_by_block_rule"
|
||||||
|
|
||||||
// ActionBlockedByCoreRuleSet is a request refused because the Core Rule
|
// ActionBlockedByCoreRuleSet is a request refused because the Core Rule
|
||||||
// Set scored it at or over SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD.
|
// Set scored it at or over SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD.
|
||||||
ActionBlockedByCoreRuleSet = "blocked_by_core_rule_set"
|
ActionBlockedByCoreRuleSet Action = "blocked_by_core_rule_set"
|
||||||
|
|
||||||
// ActionClientDenied is a request refused because its client is in
|
// ActionClientDenied is a request refused because its client is in
|
||||||
// SWW_WAF_DENY_NETS, in a blocklist while SWW_WAF_BLOCKLIST_ACTION is deny,
|
// SWW_WAF_DENY_NETS, in a blocklist while SWW_WAF_BLOCKLIST_ACTION is deny,
|
||||||
// or listed by a DNSBL zone, or scored a hit by AbuseIPDB, while
|
// or listed by a DNSBL zone, or scored a hit by AbuseIPDB, while
|
||||||
// SWW_WAF_DNSBL_ABUSEIPDB_ACTION is deny, or that carries a tag in
|
// SWW_WAF_DNSBL_ABUSEIPDB_ACTION is deny, or that carries a tag in
|
||||||
// SWW_WAF_TAG_DENY.
|
// SWW_WAF_TAG_DENY.
|
||||||
ActionClientDenied = "client_denied"
|
ActionClientDenied Action = "client_denied"
|
||||||
|
|
||||||
// ActionCountryDenied is a request refused because its client's country
|
// ActionCountryDenied is a request refused because its client's country
|
||||||
// is not in SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES.
|
// is not in SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES.
|
||||||
ActionCountryDenied = "country_denied"
|
ActionCountryDenied Action = "country_denied"
|
||||||
|
|
||||||
// ActionChecksTimedOut is a request refused because the trap paths, the
|
// ActionChecksTimedOut is a request refused because the trap paths, the
|
||||||
// rule files and the Core Rule Set had not finished with it within 500
|
// rule files and the Core Rule Set had not finished with it within 500
|
||||||
// milliseconds.
|
// milliseconds.
|
||||||
ActionChecksTimedOut = "checks_timed_out"
|
ActionChecksTimedOut Action = "checks_timed_out"
|
||||||
|
|
||||||
// ActionOwnEndpoint is a request waf answered at one of its own
|
// ActionOwnEndpoint is a request waf answered at one of its own
|
||||||
// endpoints, or a question about a path under /_sww/.
|
// endpoints, or a question about a path under /_sww/.
|
||||||
ActionOwnEndpoint = "own_endpoint"
|
ActionOwnEndpoint Action = "own_endpoint"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// Offence is what a request was held as, as its request line names it.
|
||||||
|
type Offence string
|
||||||
|
|
||||||
// OffenceLimit is the offence a request line names for a request that
|
// OffenceLimit is the offence a request line names for a request that
|
||||||
// broke a rate limit or the refusal limit, or whose bytes broke a byte
|
// broke a rate limit or the refusal limit, or whose bytes broke a byte
|
||||||
// limit.
|
// limit.
|
||||||
const OffenceLimit = "limit"
|
const OffenceLimit Offence = "limit"
|
||||||
|
|
||||||
// LimitHitMinuteRefusals is the limit_hit a request line names for a request
|
// LimitHitMinuteRefusals is the limit_hit a request line names for a request
|
||||||
// that broke the refusal limit.
|
// that broke the refusal limit.
|
||||||
const LimitHitMinuteRefusals = "minute_refusals"
|
const LimitHitMinuteRefusals = "minute_refusals"
|
||||||
|
|
||||||
|
// WAFLineKind is which of waf's two lines about a question a request line
|
||||||
|
// is: WAFLineDecision or WAFLineReport.
|
||||||
|
type WAFLineKind string
|
||||||
|
|
||||||
// The waf_line a request line names, which says which of
|
// The waf_line a request line names, which says which of
|
||||||
// waf's two lines about a question it is.
|
// waf's two lines about a question it is.
|
||||||
const (
|
const (
|
||||||
// WAFLineDecision is waf's line at its decision.
|
// WAFLineDecision is waf's line at its decision.
|
||||||
WAFLineDecision = "decision"
|
WAFLineDecision WAFLineKind = "decision"
|
||||||
// WAFLineReport is waf's line when rproxy's report
|
// WAFLineReport is waf's line when rproxy's report
|
||||||
// comes, with the decision's fields and the report's together.
|
// comes, with the decision's fields and the report's together.
|
||||||
WAFLineReport = "report"
|
WAFLineReport WAFLineKind = "report"
|
||||||
)
|
)
|
||||||
|
|
||||||
// timeLayout is RFC 3339 with milliseconds.
|
// timeLayout is RFC 3339 with milliseconds.
|
||||||
@@ -100,9 +112,8 @@ const timeLayout = "2006-01-02T15:04:05.000Z07:00"
|
|||||||
//nolint:tagliatelle // docs/request-log.md names the fields in snake_case
|
//nolint:tagliatelle // docs/request-log.md names the fields in snake_case
|
||||||
type Line struct {
|
type Line struct {
|
||||||
Type string `json:"type"`
|
Type string `json:"type"`
|
||||||
// WAFLine is WAFLineDecision or WAFLineReport
|
// WAFLine is set on waf's two lines about a question.
|
||||||
// on waf's two lines about a question.
|
WAFLine WAFLineKind `json:"waf_line,omitempty"`
|
||||||
WAFLine string `json:"waf_line,omitempty"`
|
|
||||||
|
|
||||||
// The standard web log fields. Scheme is how the client reached rproxy, or
|
// The standard web log fields. Scheme is how the client reached rproxy, or
|
||||||
// the trusted proxy in front of it.
|
// the trusted proxy in front of it.
|
||||||
@@ -148,7 +159,7 @@ type Line struct {
|
|||||||
Aborted bool `json:"aborted,omitempty"`
|
Aborted bool `json:"aborted,omitempty"`
|
||||||
|
|
||||||
// The decision.
|
// The decision.
|
||||||
Action string `json:"action"`
|
Action Action `json:"action"`
|
||||||
// WouldAction is, in observe mode, the action enforce mode would have
|
// WouldAction is, in observe mode, the action enforce mode would have
|
||||||
// taken with a request it would have refused: ActionClientDenied,
|
// taken with a request it would have refused: ActionClientDenied,
|
||||||
// ActionTemporarilyBanned, ActionPermanentlyBanned, ActionCountryDenied,
|
// ActionTemporarilyBanned, ActionPermanentlyBanned, ActionCountryDenied,
|
||||||
@@ -156,7 +167,7 @@ type Line struct {
|
|||||||
// ActionBlockedByCoreRuleSet, or, in waf, ActionRequestLineTooLong or
|
// ActionBlockedByCoreRuleSet, or, in waf, ActionRequestLineTooLong or
|
||||||
// ActionRequestHeaderTooLarge for the header limits, or
|
// ActionRequestHeaderTooLarge for the header limits, or
|
||||||
// ActionChecksTimedOut.
|
// ActionChecksTimedOut.
|
||||||
WouldAction string `json:"would_action,omitempty"`
|
WouldAction Action `json:"would_action,omitempty"`
|
||||||
// Heuristic is the name of the heuristic that refused the request, in
|
// Heuristic is the name of the heuristic that refused the request, in
|
||||||
// observe mode too, and HeuristicFigures are the figures it acted on, by
|
// observe mode too, and HeuristicFigures are the figures it acted on, by
|
||||||
// name.
|
// name.
|
||||||
@@ -203,8 +214,7 @@ type Line struct {
|
|||||||
// request got no tags from routewatch: no_answer_yet, request_failed or
|
// request got no tags from routewatch: no_answer_yet, request_failed or
|
||||||
// no_route.
|
// no_route.
|
||||||
RouteWatchTagsMissing string `json:"routewatch_tags_missing,omitempty"`
|
RouteWatchTagsMissing string `json:"routewatch_tags_missing,omitempty"`
|
||||||
// Offence is the offence the request was held as, OffenceLimit.
|
Offence Offence `json:"offence,omitempty"`
|
||||||
Offence string `json:"offence,omitempty"`
|
|
||||||
// BanExpires is when the ban the request made, or was refused under,
|
// BanExpires is when the ban the request made, or was refused under,
|
||||||
// ends: a time, or "permanent".
|
// ends: a time, or "permanent".
|
||||||
BanExpires string `json:"ban_expires,omitempty"`
|
BanExpires string `json:"ban_expires,omitempty"`
|
||||||
@@ -246,7 +256,7 @@ type RProxyLine struct {
|
|||||||
// WAFBackendNoAnswer is true, the value of
|
// WAFBackendNoAnswer is true, the value of
|
||||||
// SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION. It is "" when neither
|
// SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION. It is "" when neither
|
||||||
// gave one.
|
// gave one.
|
||||||
Disposition string `json:"disposition,omitempty"`
|
Disposition wafprotocol.Disposition `json:"disposition,omitempty"`
|
||||||
WAFBackendNoAnswer bool `json:"waf_backend_no_answer,omitempty"`
|
WAFBackendNoAnswer bool `json:"waf_backend_no_answer,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,71 @@
|
|||||||
|
package rproxy
|
||||||
|
|
||||||
|
import (
|
||||||
|
"embed"
|
||||||
|
"html"
|
||||||
|
"net/http"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// pageFiles are the pages a refused visitor gets, as HTML with nothing
|
||||||
|
// external: refused.html for a 403, ratelimited.html for a 429.
|
||||||
|
//
|
||||||
|
//go:embed refused.html ratelimited.html
|
||||||
|
var pageFiles embed.FS
|
||||||
|
|
||||||
|
// contactSentence is the sentence of each page that becomes a link to
|
||||||
|
// SWW_RPROXY_CONTACT_EMAIL when that is set.
|
||||||
|
const contactSentence = "contact the site's operator"
|
||||||
|
|
||||||
|
// htmlContentType is the Content-Type of the pages.
|
||||||
|
const htmlContentType = "text/html; charset=utf-8"
|
||||||
|
|
||||||
|
// blockPages are the pages a refused visitor gets, ready to send, made once
|
||||||
|
// at start.
|
||||||
|
type blockPages struct {
|
||||||
|
refused []byte
|
||||||
|
rateLimited []byte
|
||||||
|
}
|
||||||
|
|
||||||
|
// newBlockPages makes the pages. With contactEmail set, each page's
|
||||||
|
// contactSentence is a mailto link to it.
|
||||||
|
func newBlockPages(contactEmail string) blockPages {
|
||||||
|
return blockPages{
|
||||||
|
refused: readPage("refused.html", contactEmail),
|
||||||
|
rateLimited: readPage("ratelimited.html", contactEmail),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// readPage reads the page in name from pageFiles, linking its
|
||||||
|
// contactSentence to contactEmail unless that is "". The files are part of
|
||||||
|
// the binary, so a missing one is a bug, and a panic.
|
||||||
|
func readPage(name, contactEmail string) []byte {
|
||||||
|
contents, err := pageFiles.ReadFile(name)
|
||||||
|
if err != nil {
|
||||||
|
panic(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if contactEmail == "" {
|
||||||
|
return contents
|
||||||
|
}
|
||||||
|
|
||||||
|
link := `<a href="mailto:` + html.EscapeString(contactEmail) + `">` +
|
||||||
|
contactSentence + `</a>`
|
||||||
|
|
||||||
|
return []byte(
|
||||||
|
strings.Replace(string(contents), contactSentence, link, 1),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// forStatus returns the page for a refusal with status, and whether there is
|
||||||
|
// one: the rate-limited page for a 429, the refused page for a 403.
|
||||||
|
func (p blockPages) forStatus(status int) ([]byte, bool) {
|
||||||
|
switch status {
|
||||||
|
case http.StatusForbidden:
|
||||||
|
return p.refused, true
|
||||||
|
case http.StatusTooManyRequests:
|
||||||
|
return p.rateLimited, true
|
||||||
|
default:
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -2,6 +2,7 @@ package rproxy
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"bufio"
|
"bufio"
|
||||||
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
"io"
|
"io"
|
||||||
"net"
|
"net"
|
||||||
@@ -22,7 +23,9 @@ type requestBody struct {
|
|||||||
// body is the visitor's body, ending in an *http.MaxBytesError past
|
// body is the visitor's body, ending in an *http.MaxBytesError past
|
||||||
// SWW_RPROXY_MAX_REQUEST_SIZE.
|
// SWW_RPROXY_MAX_REQUEST_SIZE.
|
||||||
body io.ReadCloser
|
body io.ReadCloser
|
||||||
rq *request
|
// refuse refuses the request with a status, as the request's own refuse
|
||||||
|
// does.
|
||||||
|
refuse func(status int)
|
||||||
// held is the part read before the request goes to the app, which Read gives
|
// held is the part read before the request goes to the app, which Read gives
|
||||||
// first.
|
// first.
|
||||||
held []byte
|
held []byte
|
||||||
@@ -68,7 +71,7 @@ func (b *requestBody) Read(p []byte) (int, error) {
|
|||||||
case errors.Is(err, io.EOF):
|
case errors.Is(err, io.EOF):
|
||||||
b.received.Store(true)
|
b.received.Store(true)
|
||||||
case errors.As(err, &tooLarge):
|
case errors.As(err, &tooLarge):
|
||||||
b.rq.refuse(http.StatusRequestEntityTooLarge)
|
b.refuse(http.StatusRequestEntityTooLarge)
|
||||||
}
|
}
|
||||||
|
|
||||||
return n, err
|
return n, err
|
||||||
@@ -86,7 +89,13 @@ type responseBody struct {
|
|||||||
// body is the app's body, ending in an *http.MaxBytesError past
|
// body is the app's body, ending in an *http.MaxBytesError past
|
||||||
// SWW_RPROXY_MAX_RESPONSE_SIZE.
|
// SWW_RPROXY_MAX_RESPONSE_SIZE.
|
||||||
body io.ReadCloser
|
body io.ReadCloser
|
||||||
rq *request
|
// refuse refuses the request with a status, as the request's own refuse
|
||||||
|
// does.
|
||||||
|
refuse func(status int)
|
||||||
|
// responseReceived is called once the app's whole answer has arrived.
|
||||||
|
responseReceived func()
|
||||||
|
// ctx is the visitor's request's context.
|
||||||
|
ctx context.Context //nolint:containedctx // the body ends with the request
|
||||||
}
|
}
|
||||||
|
|
||||||
// Read reads from the app's body.
|
// Read reads from the app's body.
|
||||||
@@ -100,15 +109,15 @@ func (b *responseBody) Read(p []byte) (int, error) {
|
|||||||
|
|
||||||
switch {
|
switch {
|
||||||
case errors.Is(err, io.EOF):
|
case errors.Is(err, io.EOF):
|
||||||
b.rq.responseReceived()
|
b.responseReceived()
|
||||||
case errors.As(err, &tooLarge):
|
case errors.As(err, &tooLarge):
|
||||||
b.rq.refuse(http.StatusBadGateway)
|
b.refuse(http.StatusBadGateway)
|
||||||
|
|
||||||
return n, errResponseTooLarge
|
return n, errResponseTooLarge
|
||||||
case b.rq.in.Context().Err() == nil:
|
case b.ctx.Err() == nil:
|
||||||
// The answer broke off, not because the visitor went away. If a time
|
// The answer broke off, not because the visitor went away. If a time
|
||||||
// limit cut it, that refusal came first and is the one kept.
|
// limit cut it, that refusal came first and is the one kept.
|
||||||
b.rq.refuse(http.StatusBadGateway)
|
b.refuse(http.StatusBadGateway)
|
||||||
}
|
}
|
||||||
|
|
||||||
return n, err
|
return n, err
|
||||||
|
|||||||
@@ -7,16 +7,6 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
)
|
)
|
||||||
|
|
||||||
// The headers rproxy sets on the requests it passes to the app. Go writes every
|
|
||||||
// header name in this form, as it sends it and as it receives it, so
|
|
||||||
// X-Client-ASN arrives as X-Client-Asn, and Del removes a visitor's own whatever
|
|
||||||
// their case; header names are not case-sensitive.
|
|
||||||
const (
|
|
||||||
requestIDHeader = "X-Request-ID"
|
|
||||||
asnHeader = "X-Client-Asn"
|
|
||||||
countryHeader = "X-Client-Country"
|
|
||||||
)
|
|
||||||
|
|
||||||
// peerAddress is the address of the request's TCP peer, normally traefik, without
|
// peerAddress is the address of the request's TCP peer, normally traefik, without
|
||||||
// any IPv6 zone, which waf refuses.
|
// any IPv6 zone, which waf refuses.
|
||||||
func peerAddress(r *http.Request) netip.Addr {
|
func peerAddress(r *http.Request) netip.Addr {
|
||||||
|
|||||||
@@ -1,6 +1,14 @@
|
|||||||
package rproxy
|
package rproxy
|
||||||
|
|
||||||
import "time"
|
import (
|
||||||
|
"net/http"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// HTTPServer is Go's server s holds, for tests that read its settings.
|
||||||
|
func (s *Server) HTTPServer() *http.Server {
|
||||||
|
return s.httpServer
|
||||||
|
}
|
||||||
|
|
||||||
// StopBreakerClock has the circuit breaker of s read at as the time, so that how
|
// StopBreakerClock has the circuit breaker of s read at as the time, so that how
|
||||||
// long requests take cannot matter to it. It is called before s serves.
|
// long requests take cannot matter to it. It is called before s serves.
|
||||||
|
|||||||
@@ -20,7 +20,7 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
|
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/processlog"
|
"sneak.berlin/go/smallwebwaf/internal/processlog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/rproxy"
|
"sneak.berlin/go/smallwebwaf/internal/rproxy"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
|
||||||
)
|
)
|
||||||
|
|
||||||
// post is the start of a POST whose headers end with what follows it.
|
// post is the start of a POST whose headers end with what follows it.
|
||||||
@@ -56,7 +56,7 @@ func TestServerHasTheFixedLimits(t *testing.T) {
|
|||||||
InstanceName: "test",
|
InstanceName: "test",
|
||||||
Level: slog.LevelInfo,
|
Level: slog.LevelInfo,
|
||||||
}),
|
}),
|
||||||
})
|
}).HTTPServer()
|
||||||
|
|
||||||
if server.Addr != ":8080" || server.MaxHeaderBytes != 28<<10 ||
|
if server.Addr != ":8080" || server.MaxHeaderBytes != 28<<10 ||
|
||||||
server.IdleTimeout != 2*time.Minute || server.ReadHeaderTimeout != time.Minute {
|
server.IdleTimeout != 2*time.Minute || server.ReadHeaderTimeout != time.Minute {
|
||||||
@@ -492,7 +492,7 @@ func TestASettingOfOffSetsNoLimit(t *testing.T) {
|
|||||||
strings.NewReader("a body"))),
|
strings.NewReader("a body"))),
|
||||||
)
|
)
|
||||||
|
|
||||||
if got := get(t, addr, waf.HealthPath); got.status != http.StatusOK {
|
if got := get(t, addr, wafprotocol.HealthPath); got.status != http.StatusOK {
|
||||||
t.Errorf("answered %+v, want rproxy's own 200", got)
|
t.Errorf("answered %+v, want rproxy's own 200", got)
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
@@ -709,7 +709,7 @@ func TestABodyLeftUnreadIsReadOnlyUntilTheRequestTimeLimit(t *testing.T) {
|
|||||||
|
|
||||||
before := time.Now()
|
before := time.Now()
|
||||||
|
|
||||||
server.Handler.ServeHTTP(w, req)
|
server.ServeHTTP(w, req)
|
||||||
|
|
||||||
if w.Code != http.StatusNoContent {
|
if w.Code != http.StatusNoContent {
|
||||||
t.Errorf("status %d, want the app's 204", w.Code)
|
t.Errorf("status %d, want the app's 204", w.Code)
|
||||||
@@ -762,7 +762,7 @@ func TestABodyLeftUnreadIsReadNoMoreOnceTheTimeToSendTheAnswerRunsOut(
|
|||||||
|
|
||||||
before := time.Now()
|
before := time.Now()
|
||||||
|
|
||||||
server.Handler.ServeHTTP(w, req)
|
server.ServeHTTP(w, req)
|
||||||
|
|
||||||
if w.Code != http.StatusOK {
|
if w.Code != http.StatusOK {
|
||||||
t.Errorf("status %d, want the app's 200", w.Code)
|
t.Errorf("status %d, want the app's 200", w.Code)
|
||||||
@@ -815,7 +815,7 @@ func TestAnAnswerRProxyGivesItselfHasTheResponseTimeToBeSent(t *testing.T) {
|
|||||||
|
|
||||||
before := time.Now()
|
before := time.Now()
|
||||||
|
|
||||||
server.Handler.ServeHTTP(w, req)
|
server.ServeHTTP(w, req)
|
||||||
|
|
||||||
if w.Code != tc.status {
|
if w.Code != tc.status {
|
||||||
t.Errorf("status %d, want %d", w.Code, tc.status)
|
t.Errorf("status %d, want %d", w.Code, tc.status)
|
||||||
@@ -856,7 +856,7 @@ func TestAnAnswerRProxyGivesItselfHasTenSecondsToBeSentAtLeast(
|
|||||||
|
|
||||||
before := time.Now()
|
before := time.Now()
|
||||||
|
|
||||||
server.Handler.ServeHTTP(w, req)
|
server.ServeHTTP(w, req)
|
||||||
|
|
||||||
if w.writeDeadline.Before(before.Add(least)) {
|
if w.writeDeadline.Before(before.Add(least)) {
|
||||||
t.Errorf(
|
t.Errorf(
|
||||||
@@ -1480,7 +1480,7 @@ func TestAVisitorThatStopsALittleShortGetsAnAnswerRProxyGivesAtOnce(
|
|||||||
{"a deny", post, http.StatusForbidden},
|
{"a deny", post, http.StatusForbidden},
|
||||||
{
|
{
|
||||||
"the health check",
|
"the health check",
|
||||||
"GET " + waf.HealthPath + " HTTP/1.1\r\nHost: site.example\r\n",
|
"GET " + wafprotocol.HealthPath + " HTTP/1.1\r\nHost: site.example\r\n",
|
||||||
http.StatusOK,
|
http.StatusOK,
|
||||||
},
|
},
|
||||||
} {
|
} {
|
||||||
|
|||||||
@@ -12,11 +12,11 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
|
||||||
)
|
)
|
||||||
|
|
||||||
// questionBodyMaxBytes is the most of a body a question carries.
|
// questionBodyMaxBytes is the most of a body a question carries.
|
||||||
const questionBodyMaxBytes = waf.QuestionBodyMaxBytes
|
const questionBodyMaxBytes = wafprotocol.QuestionBodyMaxBytes
|
||||||
|
|
||||||
// chunked is the encoding of a body whose length the visitor did not announce.
|
// chunked is the encoding of a body whose length the visitor did not announce.
|
||||||
const chunked = "chunked"
|
const chunked = "chunked"
|
||||||
@@ -42,9 +42,9 @@ func TestQuestionIsTheVisitorsRequestWithRProxysHeaders(t *testing.T) {
|
|||||||
req.Header.Set(forwardedForHeader, "198.51.100.1")
|
req.Header.Set(forwardedForHeader, "198.51.100.1")
|
||||||
req.Header.Set(forwardedProtoHeader, "gopher")
|
req.Header.Set(forwardedProtoHeader, "gopher")
|
||||||
req.Header.Set(requestIDHeader, "the-visitor's")
|
req.Header.Set(requestIDHeader, "the-visitor's")
|
||||||
req.Header.Set(waf.ClientHeader, "198.51.100.2")
|
req.Header.Set(wafprotocol.ClientHeader, "198.51.100.2")
|
||||||
req.Header.Set(waf.PeerHeader, "198.51.100.3")
|
req.Header.Set(wafprotocol.PeerHeader, "198.51.100.3")
|
||||||
req.Header.Set(waf.ContentLengthHeader, "12345")
|
req.Header.Set(wafprotocol.ContentLengthHeader, "12345")
|
||||||
// Go's client then sends no User-Agent, so that one in the question would be
|
// Go's client then sends no User-Agent, so that one in the question would be
|
||||||
// Go's own.
|
// Go's own.
|
||||||
req.Header["User-Agent"] = nil
|
req.Header["User-Agent"] = nil
|
||||||
@@ -73,8 +73,8 @@ func TestQuestionIsTheVisitorsRequestWithRProxysHeaders(t *testing.T) {
|
|||||||
want.Set(forwardedForHeader, "198.51.100.1")
|
want.Set(forwardedForHeader, "198.51.100.1")
|
||||||
want.Set(forwardedProtoHeader, "http")
|
want.Set(forwardedProtoHeader, "http")
|
||||||
want.Set(requestIDHeader, asked.header.Get(requestIDHeader))
|
want.Set(requestIDHeader, asked.header.Get(requestIDHeader))
|
||||||
want.Set(waf.ClientHeader, localhost)
|
want.Set(wafprotocol.ClientHeader, localhost)
|
||||||
want.Set(waf.PeerHeader, localhost)
|
want.Set(wafprotocol.PeerHeader, localhost)
|
||||||
|
|
||||||
if !maps.EqualFunc(asked.header, want, slices.Equal) {
|
if !maps.EqualFunc(asked.header, want, slices.Equal) {
|
||||||
t.Errorf(
|
t.Errorf(
|
||||||
@@ -124,8 +124,8 @@ func TestQuestionNamesTheVisitorATrustedProxySpeaksFor(t *testing.T) {
|
|||||||
wantGreeting(t, visit(t, req))
|
wantGreeting(t, visit(t, req))
|
||||||
|
|
||||||
asked := standIn.question(t)
|
asked := standIn.question(t)
|
||||||
if asked.header.Get(waf.ClientHeader) != tc.client ||
|
if asked.header.Get(wafprotocol.ClientHeader) != tc.client ||
|
||||||
asked.header.Get(waf.PeerHeader) != localhost ||
|
asked.header.Get(wafprotocol.PeerHeader) != localhost ||
|
||||||
asked.header.Get(forwardedForHeader) != visitor ||
|
asked.header.Get(forwardedForHeader) != visitor ||
|
||||||
asked.header.Get(forwardedProtoHeader) != tc.proto {
|
asked.header.Get(forwardedProtoHeader) != tc.proto {
|
||||||
t.Errorf("the question's headers are %v", asked.header)
|
t.Errorf("the question's headers are %v", asked.header)
|
||||||
@@ -191,7 +191,7 @@ func TestQuestionNamesTheClientAsTheForwardedAddressesSay(t *testing.T) {
|
|||||||
req.Header[forwardedForHeader] = tc.header
|
req.Header[forwardedForHeader] = tc.header
|
||||||
wantGreeting(t, visit(t, req))
|
wantGreeting(t, visit(t, req))
|
||||||
|
|
||||||
if got := standIn.question(t).header.Get(waf.ClientHeader); got !=
|
if got := standIn.question(t).header.Get(wafprotocol.ClientHeader); got !=
|
||||||
tc.client {
|
tc.client {
|
||||||
t.Errorf("the question names %s, want %s", got, tc.client)
|
t.Errorf("the question names %s, want %s", got, tc.client)
|
||||||
}
|
}
|
||||||
@@ -282,7 +282,7 @@ func wantQuestionBody(
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
values := asked.header.Values(waf.ContentLengthHeader)
|
values := asked.header.Values(wafprotocol.ContentLengthHeader)
|
||||||
|
|
||||||
if announced {
|
if announced {
|
||||||
if !slices.Equal(values, []string{strconv.Itoa(size)}) ||
|
if !slices.Equal(values, []string{strconv.Itoa(size)}) ||
|
||||||
@@ -583,11 +583,11 @@ func TestAQuestionNamesNoZone(t *testing.T) {
|
|||||||
req.RemoteAddr = tc.remoteAddr
|
req.RemoteAddr = tc.remoteAddr
|
||||||
req.Header.Set(forwardedForHeader, tc.forwardedFor)
|
req.Header.Set(forwardedForHeader, tc.forwardedFor)
|
||||||
|
|
||||||
server.Handler.ServeHTTP(httptest.NewRecorder(), req)
|
server.ServeHTTP(httptest.NewRecorder(), req)
|
||||||
|
|
||||||
asked := standIn.question(t)
|
asked := standIn.question(t)
|
||||||
if asked.header.Get(waf.ClientHeader) != tc.client ||
|
if asked.header.Get(wafprotocol.ClientHeader) != tc.client ||
|
||||||
asked.header.Get(waf.PeerHeader) != tc.peer {
|
asked.header.Get(wafprotocol.PeerHeader) != tc.peer {
|
||||||
t.Errorf("the question's headers are %v", asked.header)
|
t.Errorf("the question's headers are %v", asked.header)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,27 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||||
|
<meta name="robots" content="noindex">
|
||||||
|
<title>Too many requests</title>
|
||||||
|
<style>
|
||||||
|
:root{color-scheme:light dark;--fg:#1a1a1a;--muted:#5c5c5c;--bg:#f6f6f4}
|
||||||
|
@media(prefers-color-scheme:dark){:root{--fg:#ebebe8;--muted:#9a9a96;--bg:#161616}}
|
||||||
|
html{background:var(--bg);color:var(--fg);font:16px/1.5 system-ui,-apple-system,"Segoe UI",Roboto,sans-serif}
|
||||||
|
body{margin:0;min-height:100vh;display:flex;align-items:center;justify-content:center;padding:1.5rem}
|
||||||
|
main{max-width:34rem;width:100%}
|
||||||
|
p.code{margin:0 0 .5rem;color:var(--muted);font-size:.8rem;letter-spacing:.08em}
|
||||||
|
h1{margin:0 0 1rem;font-size:1.6rem;font-weight:600;line-height:1.25}
|
||||||
|
p{margin:0 0 1rem}
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<main>
|
||||||
|
<p class="code">429</p>
|
||||||
|
<h1>Too many requests</h1>
|
||||||
|
<p>Your address has sent more requests than this site allows in a short time. Wait a while, then try again.</p>
|
||||||
|
<p>If you think this is a mistake, contact the site's operator.</p>
|
||||||
|
</main>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||||
|
<meta name="robots" content="noindex">
|
||||||
|
<title>Request refused</title>
|
||||||
|
<style>
|
||||||
|
:root{color-scheme:light dark;--fg:#1a1a1a;--muted:#5c5c5c;--bg:#f6f6f4}
|
||||||
|
@media(prefers-color-scheme:dark){:root{--fg:#ebebe8;--muted:#9a9a96;--bg:#161616}}
|
||||||
|
html{background:var(--bg);color:var(--fg);font:16px/1.5 system-ui,-apple-system,"Segoe UI",Roboto,sans-serif}
|
||||||
|
body{margin:0;min-height:100vh;display:flex;align-items:center;justify-content:center;padding:1.5rem}
|
||||||
|
main{max-width:34rem;width:100%}
|
||||||
|
p.code{margin:0 0 .5rem;color:var(--muted);font-size:.8rem;letter-spacing:.08em}
|
||||||
|
h1{margin:0 0 1rem;font-size:1.6rem;font-weight:600;line-height:1.25}
|
||||||
|
p{margin:0 0 1rem}
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<main>
|
||||||
|
<p class="code">403</p>
|
||||||
|
<h1>Request refused</h1>
|
||||||
|
<p>This site did not accept your request.</p>
|
||||||
|
<p>If you think this is a mistake, contact the site's operator.</p>
|
||||||
|
</main>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
+48
-177
@@ -12,23 +12,18 @@ import (
|
|||||||
"net/url"
|
"net/url"
|
||||||
"os"
|
"os"
|
||||||
"strconv"
|
"strconv"
|
||||||
"sync"
|
|
||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
|
||||||
)
|
)
|
||||||
|
|
||||||
// flushAfterEachWrite has ReverseProxy pass on each part of the app's answer as
|
// flushAfterEachWrite has ReverseProxy pass on each part of the app's answer as
|
||||||
// soon as it arrives.
|
// soon as it arrives.
|
||||||
const flushAfterEachWrite time.Duration = -1
|
const flushAfterEachWrite time.Duration = -1
|
||||||
|
|
||||||
// reportStatusMax is the highest status waf takes in a report. An app can
|
|
||||||
// send a higher one, up to 999, and the visitor gets that as it is.
|
|
||||||
const reportStatusMax = 599
|
|
||||||
|
|
||||||
// plainText is the Content-Type of the answers rproxy gives itself.
|
// plainText is the Content-Type of the answers rproxy gives itself.
|
||||||
const plainText = "text/plain; charset=utf-8"
|
const plainText = "text/plain; charset=utf-8"
|
||||||
|
|
||||||
@@ -67,7 +62,7 @@ type request struct {
|
|||||||
// disposition is what is done with the request: waf's valid disposition,
|
// disposition is what is done with the request: waf's valid disposition,
|
||||||
// or SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION. It is "" when neither
|
// or SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION. It is "" when neither
|
||||||
// gave one.
|
// gave one.
|
||||||
disposition string
|
disposition wafprotocol.Disposition
|
||||||
// answered is true when waf gave disposition, which makes a report due,
|
// answered is true when waf gave disposition, which makes a report due,
|
||||||
// and wafNoAnswer when SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION did.
|
// and wafNoAnswer when SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION did.
|
||||||
answered bool
|
answered bool
|
||||||
@@ -86,17 +81,9 @@ type request struct {
|
|||||||
// upgraded is the connection to the app once the app has switched protocols,
|
// upgraded is the connection to the app once the app has switched protocols,
|
||||||
// as for a WebSocket, and nil otherwise.
|
// as for a WebSocket, and nil otherwise.
|
||||||
upgraded *upgradedConn
|
upgraded *upgradedConn
|
||||||
|
// limits are the request's SWW_RPROXY_MAX_REQUEST_TIME_MS and
|
||||||
// mu guards what follows. The time limits run on goroutines of their own,
|
// SWW_RPROXY_MAX_RESPONSE_TIME_MS.
|
||||||
// and the transport starts and stops them, and notes requestEnd, from its
|
limits *timeLimits
|
||||||
// own; once timersStopped is set, neither time limit acts any more.
|
|
||||||
mu sync.Mutex
|
|
||||||
timersStopped bool
|
|
||||||
requestTimer *time.Timer
|
|
||||||
responseTimer *time.Timer
|
|
||||||
// requestEnd is when the app had been sent the whole request, or the answer
|
|
||||||
// started, if that was sooner.
|
|
||||||
requestEnd time.Time
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// newRequest starts handling r: it notes the time, works out the visitor from
|
// newRequest starts handling r: it notes the time, works out the visitor from
|
||||||
@@ -127,16 +114,29 @@ func (h *handler) newRequest(
|
|||||||
if r.Body != http.NoBody {
|
if r.Body != http.NoBody {
|
||||||
rq.body = &requestBody{
|
rq.body = &requestBody{
|
||||||
body: limitBody(r.Body, h.config.MaxRequestSize),
|
body: limitBody(r.Body, h.config.MaxRequestSize),
|
||||||
rq: rq,
|
refuse: rq.refuse,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
rq.limits = &timeLimits{
|
||||||
|
maxRequestTime: h.config.MaxRequestTime,
|
||||||
|
maxResponseTime: h.config.MaxResponseTime,
|
||||||
|
start: rq.start,
|
||||||
|
refuse: rq.refuse,
|
||||||
|
waitingForVisitor: func() bool {
|
||||||
|
return rq.body != nil && rq.body.waiting.Load()
|
||||||
|
},
|
||||||
|
endVisitorRead: func() {
|
||||||
|
_ = rq.rc.SetReadDeadline(time.Now())
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
return rq
|
return rq
|
||||||
}
|
}
|
||||||
|
|
||||||
// answerHealth answers the health check.
|
// answerHealth answers the health check.
|
||||||
func (rq *request) answerHealth() {
|
func (rq *request) answerHealth() {
|
||||||
rq.prepareOwnAnswer(rq.requestDeadline())
|
rq.prepareOwnAnswer(rq.limits.requestDeadline())
|
||||||
rq.out.Header().Set("Content-Type", plainText)
|
rq.out.Header().Set("Content-Type", plainText)
|
||||||
rq.send(http.StatusOK, []byte("ok\n"))
|
rq.send(http.StatusOK, []byte("ok\n"))
|
||||||
}
|
}
|
||||||
@@ -152,9 +152,9 @@ func (rq *request) readFirstPartOfBody() bool {
|
|||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
|
|
||||||
_ = rq.rc.SetReadDeadline(rq.requestDeadline())
|
_ = rq.rc.SetReadDeadline(rq.limits.requestDeadline())
|
||||||
held, err := io.ReadAll(
|
held, err := io.ReadAll(
|
||||||
io.LimitReader(rq.body, waf.QuestionBodyMaxBytes),
|
io.LimitReader(rq.body, wafprotocol.QuestionBodyMaxBytes),
|
||||||
)
|
)
|
||||||
// From here on, the request timer ends a read of the rest of the body, so that
|
// From here on, the request timer ends a read of the rest of the body, so that
|
||||||
// the answer can name the side rproxy was waiting on.
|
// the answer can name the side rproxy was waiting on.
|
||||||
@@ -192,7 +192,7 @@ func (rq *request) forward(ctx context.Context) {
|
|||||||
|
|
||||||
rq.cancel = cancel
|
rq.cancel = cancel
|
||||||
ctx = httptrace.WithClientTrace(ctx, &httptrace.ClientTrace{
|
ctx = httptrace.WithClientTrace(ctx, &httptrace.ClientTrace{
|
||||||
WroteRequest: rq.wroteRequest,
|
WroteRequest: rq.limits.wroteRequest,
|
||||||
})
|
})
|
||||||
|
|
||||||
out := rq.in.WithContext(ctx)
|
out := rq.in.WithContext(ctx)
|
||||||
@@ -209,7 +209,7 @@ func (rq *request) forward(ctx context.Context) {
|
|||||||
ErrorHandler: rq.answerError,
|
ErrorHandler: rq.answerError,
|
||||||
}
|
}
|
||||||
|
|
||||||
rq.startRequestTimer()
|
rq.limits.startRequestTimer()
|
||||||
rq.upstreamStart = time.Now()
|
rq.upstreamStart = time.Now()
|
||||||
reverseProxy.ServeHTTP(rq.out, out)
|
reverseProxy.ServeHTTP(rq.out, out)
|
||||||
}
|
}
|
||||||
@@ -228,16 +228,17 @@ func (rq *request) rewrite(pr *httputil.ProxyRequest) {
|
|||||||
setPathAsSent(pr.Out.URL, pr.In)
|
setPathAsSent(pr.Out.URL, pr.In)
|
||||||
|
|
||||||
setForwardedHeaders(pr.In, pr.Out, rq.peer, rq.peerTrusted)
|
setForwardedHeaders(pr.In, pr.Out, rq.peer, rq.peerTrusted)
|
||||||
pr.Out.Header.Set(requestIDHeader, rq.id)
|
pr.Out.Header.Set(wafprotocol.RequestIDHeader, rq.id)
|
||||||
pr.Out.Header.Del(asnHeader)
|
// Del removes a visitor's own, whatever their case.
|
||||||
pr.Out.Header.Del(countryHeader)
|
pr.Out.Header.Del(wafprotocol.ASNHeader)
|
||||||
|
pr.Out.Header.Del(wafprotocol.CountryHeader)
|
||||||
|
|
||||||
if rq.asn != "" {
|
if rq.asn != "" {
|
||||||
pr.Out.Header.Set(asnHeader, rq.asn)
|
pr.Out.Header.Set(wafprotocol.ASNHeader, rq.asn)
|
||||||
}
|
}
|
||||||
|
|
||||||
if rq.country != "" {
|
if rq.country != "" {
|
||||||
pr.Out.Header.Set(countryHeader, rq.country)
|
pr.Out.Header.Set(wafprotocol.CountryHeader, rq.country)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -251,7 +252,7 @@ func (rq *request) modifyResponse(res *http.Response) error {
|
|||||||
// switch, as when the app's Upgrade is not the visitor's, so the status
|
// switch, as when the app's Upgrade is not the visitor's, so the status
|
||||||
// is noted only once it has taken over the connection, by rq.out's
|
// is noted only once it has taken over the connection, by rq.out's
|
||||||
// Hijack.
|
// Hijack.
|
||||||
rq.stopTimers()
|
rq.limits.stop()
|
||||||
|
|
||||||
conn, ok := res.Body.(io.ReadWriteCloser)
|
conn, ok := res.Body.(io.ReadWriteCloser)
|
||||||
if ok {
|
if ok {
|
||||||
@@ -262,7 +263,7 @@ func (rq *request) modifyResponse(res *http.Response) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
requestEnd := rq.endRequest()
|
requestEnd := rq.limits.endRequest()
|
||||||
|
|
||||||
// An answer that arrives once SWW_RPROXY_MAX_RESPONSE_TIME_MS has run out is not
|
// An answer that arrives once SWW_RPROXY_MAX_RESPONSE_TIME_MS has run out is not
|
||||||
// passed on: the visitor is answered 504.
|
// passed on: the visitor is answered 504.
|
||||||
@@ -292,9 +293,14 @@ func (rq *request) modifyResponse(res *http.Response) error {
|
|||||||
|
|
||||||
// An answer that arrives before the app has been sent the whole request ends
|
// An answer that arrives before the app has been sent the whole request ends
|
||||||
// the request, and SWW_RPROXY_MAX_RESPONSE_TIME_MS starts.
|
// the request, and SWW_RPROXY_MAX_RESPONSE_TIME_MS starts.
|
||||||
rq.startResponseTimer(requestEnd)
|
rq.limits.startResponseTimer(requestEnd)
|
||||||
|
|
||||||
res.Body = &responseBody{body: limitBody(res.Body, maxBytes), rq: rq}
|
res.Body = &responseBody{
|
||||||
|
body: limitBody(res.Body, maxBytes),
|
||||||
|
refuse: rq.refuse,
|
||||||
|
responseReceived: rq.limits.stop,
|
||||||
|
ctx: rq.in.Context(),
|
||||||
|
}
|
||||||
rq.out.afterStatus = rq.setResponseDeadline
|
rq.out.afterStatus = rq.setResponseDeadline
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
@@ -345,7 +351,7 @@ func (rq *request) answer(status int) {
|
|||||||
// A visitor found too slow is read no more; any other may go on sending until
|
// A visitor found too slow is read no more; any other may go on sending until
|
||||||
// its time is up, so that the rest of the body is read and the request ends
|
// its time is up, so that the rest of the body is read and the request ends
|
||||||
// cleanly: by Go's server, or by readRestOfBody.
|
// cleanly: by Go's server, or by readRestOfBody.
|
||||||
deadline := rq.requestDeadline()
|
deadline := rq.limits.requestDeadline()
|
||||||
if status == http.StatusRequestTimeout {
|
if status == http.StatusRequestTimeout {
|
||||||
deadline = time.Now()
|
deadline = time.Now()
|
||||||
}
|
}
|
||||||
@@ -391,7 +397,7 @@ func (rq *request) closeAfterAnswer() {
|
|||||||
// used that time up first, and at least minSendTime.
|
// used that time up first, and at least minSendTime.
|
||||||
func (rq *request) prepareOwnAnswer(deadline time.Time) {
|
func (rq *request) prepareOwnAnswer(deadline time.Time) {
|
||||||
rq.stopReadingBody(deadline)
|
rq.stopReadingBody(deadline)
|
||||||
rq.endRequest()
|
rq.limits.endRequest()
|
||||||
|
|
||||||
limit := rq.h.config.MaxResponseTime
|
limit := rq.h.config.MaxResponseTime
|
||||||
if limit > 0 {
|
if limit > 0 {
|
||||||
@@ -412,7 +418,7 @@ func (rq *request) refuse(status int) {
|
|||||||
// finish ends the request's time limits, writes its log line and, if waf
|
// finish ends the request's time limits, writes its log line and, if waf
|
||||||
// answered the question, sends the report. ctx is the request's own context.
|
// answered the question, sends the report. ctx is the request's own context.
|
||||||
func (rq *request) finish(ctx context.Context) {
|
func (rq *request) finish(ctx context.Context) {
|
||||||
rq.stopTimers()
|
rq.limits.stop()
|
||||||
|
|
||||||
// After a cut-off answer, the reading of a body that has not arrived whole
|
// After a cut-off answer, the reading of a body that has not arrived whole
|
||||||
// ends now. Go's server would otherwise keep the connection open while it
|
// ends now. Go's server would otherwise keep the connection open while it
|
||||||
@@ -430,10 +436,10 @@ func (rq *request) finish(ctx context.Context) {
|
|||||||
errors.Is(rq.rc.Flush(), os.ErrDeadlineExceeded):
|
errors.Is(rq.rc.Flush(), os.ErrDeadlineExceeded):
|
||||||
rq.stopPassingOnBody(time.Now())
|
rq.stopPassingOnBody(time.Now())
|
||||||
case rq.refused.Load() == 0:
|
case rq.refused.Load() == 0:
|
||||||
rq.stopPassingOnBody(rq.requestDeadline())
|
rq.stopPassingOnBody(rq.limits.requestDeadline())
|
||||||
}
|
}
|
||||||
|
|
||||||
requestEnd := rq.endRequest()
|
requestEnd := rq.limits.endRequest()
|
||||||
end := time.Now()
|
end := time.Now()
|
||||||
requestBytes, responseBytes := rq.bytesEachWay()
|
requestBytes, responseBytes := rq.bytesEachWay()
|
||||||
|
|
||||||
@@ -470,9 +476,10 @@ func (rq *request) finish(ctx context.Context) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
report := &waf.Report{
|
// An app can send a status up to 999, and the visitor gets it as it is.
|
||||||
|
report := &wafprotocol.Report{
|
||||||
RequestID: rq.id,
|
RequestID: rq.id,
|
||||||
Status: min(rq.out.status, reportStatusMax),
|
Status: min(rq.out.status, wafprotocol.ReportStatusMax),
|
||||||
RequestBytes: requestBytes,
|
RequestBytes: requestBytes,
|
||||||
ResponseBytes: responseBytes,
|
ResponseBytes: responseBytes,
|
||||||
DurationTotalMS: requestlog.Milliseconds(end.Sub(rq.start)),
|
DurationTotalMS: requestlog.Milliseconds(end.Sub(rq.start)),
|
||||||
@@ -506,17 +513,6 @@ func (rq *request) bytesEachWay() (int64, int64) {
|
|||||||
return requestBytes, responseBytes
|
return requestBytes, responseBytes
|
||||||
}
|
}
|
||||||
|
|
||||||
// requestDeadline is when the visitor must have sent its whole request, or zero
|
|
||||||
// when SWW_RPROXY_MAX_REQUEST_TIME_MS is off.
|
|
||||||
func (rq *request) requestDeadline() time.Time {
|
|
||||||
limit := rq.h.config.MaxRequestTime
|
|
||||||
if limit == 0 {
|
|
||||||
return time.Time{}
|
|
||||||
}
|
|
||||||
|
|
||||||
return rq.start.Add(limit)
|
|
||||||
}
|
|
||||||
|
|
||||||
// stopReadingBody ends, at deadline, the reading of a visitor's body that has not
|
// stopReadingBody ends, at deadline, the reading of a visitor's body that has not
|
||||||
// arrived whole: Go's server then reads no more of it, and closes the connection
|
// arrived whole: Go's server then reads no more of it, and closes the connection
|
||||||
// after the answer.
|
// after the answer.
|
||||||
@@ -573,100 +569,6 @@ func (rq *request) stopPassingOnBody(deadline time.Time) {
|
|||||||
rq.stopReadingBody(deadline)
|
rq.stopReadingBody(deadline)
|
||||||
}
|
}
|
||||||
|
|
||||||
// startRequestTimer starts SWW_RPROXY_MAX_REQUEST_TIME_MS, which runs from the
|
|
||||||
// request's headers being read until the app has been sent the whole request.
|
|
||||||
func (rq *request) startRequestTimer() {
|
|
||||||
deadline := rq.requestDeadline()
|
|
||||||
if deadline.IsZero() {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
rq.mu.Lock()
|
|
||||||
defer rq.mu.Unlock()
|
|
||||||
|
|
||||||
rq.requestTimer = time.AfterFunc(
|
|
||||||
time.Until(deadline),
|
|
||||||
rq.requestTimedOut,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
// requestTimedOut is called when SWW_RPROXY_MAX_REQUEST_TIME_MS runs out while the
|
|
||||||
// request is still on its way to the app. The answer names the side rproxy was
|
|
||||||
// waiting on at that moment: 408 when it was waiting for the visitor to send more
|
|
||||||
// of its body, 504 when it was waiting for the app to be reached or to take what
|
|
||||||
// it had.
|
|
||||||
func (rq *request) requestTimedOut() {
|
|
||||||
rq.mu.Lock()
|
|
||||||
defer rq.mu.Unlock()
|
|
||||||
|
|
||||||
if rq.timersStopped {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if rq.body == nil || !rq.body.waiting.Load() {
|
|
||||||
rq.refuse(http.StatusGatewayTimeout)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
rq.refuse(http.StatusRequestTimeout)
|
|
||||||
// The transport gives up on the app only once its Read of the visitor's body
|
|
||||||
// returns, so that Read is ended now. The lock keeps this from reaching the
|
|
||||||
// connection after the request is handled.
|
|
||||||
_ = rq.rc.SetReadDeadline(time.Now())
|
|
||||||
}
|
|
||||||
|
|
||||||
// wroteRequest is called once the app has been sent the whole request:
|
|
||||||
// SWW_RPROXY_MAX_REQUEST_TIME_MS ends and SWW_RPROXY_MAX_RESPONSE_TIME_MS starts.
|
|
||||||
func (rq *request) wroteRequest(info httptrace.WroteRequestInfo) {
|
|
||||||
if info.Err != nil {
|
|
||||||
return // the transport gives up, or tries again
|
|
||||||
}
|
|
||||||
|
|
||||||
requestEnd := rq.endRequest()
|
|
||||||
|
|
||||||
rq.mu.Lock()
|
|
||||||
stopTimer(rq.requestTimer)
|
|
||||||
rq.mu.Unlock()
|
|
||||||
|
|
||||||
rq.startResponseTimer(requestEnd)
|
|
||||||
}
|
|
||||||
|
|
||||||
// startResponseTimer starts SWW_RPROXY_MAX_RESPONSE_TIME_MS, which runs from
|
|
||||||
// requestEnd, unless it has started already.
|
|
||||||
func (rq *request) startResponseTimer(requestEnd time.Time) {
|
|
||||||
limit := rq.h.config.MaxResponseTime
|
|
||||||
if limit == 0 {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
rq.mu.Lock()
|
|
||||||
defer rq.mu.Unlock()
|
|
||||||
|
|
||||||
if rq.timersStopped || rq.responseTimer != nil {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
rq.responseTimer = time.AfterFunc(time.Until(requestEnd.Add(limit)),
|
|
||||||
rq.responseTimedOut)
|
|
||||||
}
|
|
||||||
|
|
||||||
// responseTimedOut is called when SWW_RPROXY_MAX_RESPONSE_TIME_MS runs out before the
|
|
||||||
// app has sent its whole answer.
|
|
||||||
func (rq *request) responseTimedOut() {
|
|
||||||
rq.mu.Lock()
|
|
||||||
defer rq.mu.Unlock()
|
|
||||||
|
|
||||||
if !rq.timersStopped {
|
|
||||||
rq.refuse(http.StatusGatewayTimeout)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// responseReceived is called once the app has sent its whole answer.
|
|
||||||
func (rq *request) responseReceived() {
|
|
||||||
rq.stopTimers()
|
|
||||||
}
|
|
||||||
|
|
||||||
// setResponseDeadline sends the status and headers of the app's answer, with
|
// setResponseDeadline sends the status and headers of the app's answer, with
|
||||||
// at least minSendTime, before ReverseProxy reads the answer's body, and then
|
// at least minSendTime, before ReverseProxy reads the answer's body, and then
|
||||||
// the rest has until SWW_RPROXY_MAX_RESPONSE_TIME_MS after the end of the request.
|
// the rest has until SWW_RPROXY_MAX_RESPONSE_TIME_MS after the end of the request.
|
||||||
@@ -676,7 +578,7 @@ func (rq *request) setResponseDeadline() {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
deadline := rq.endRequest().Add(limit)
|
deadline := rq.limits.endRequest().Add(limit)
|
||||||
|
|
||||||
_ = rq.rc.SetWriteDeadline(
|
_ = rq.rc.SetWriteDeadline(
|
||||||
time.Now().Add(max(time.Until(deadline), minSendTime)),
|
time.Now().Add(max(time.Until(deadline), minSendTime)),
|
||||||
@@ -684,34 +586,3 @@ func (rq *request) setResponseDeadline() {
|
|||||||
_ = rq.rc.Flush()
|
_ = rq.rc.Flush()
|
||||||
_ = rq.rc.SetWriteDeadline(deadline)
|
_ = rq.rc.SetWriteDeadline(deadline)
|
||||||
}
|
}
|
||||||
|
|
||||||
// endRequest notes now as the end of the request, unless an end is noted, and
|
|
||||||
// returns the end noted.
|
|
||||||
func (rq *request) endRequest() time.Time {
|
|
||||||
rq.mu.Lock()
|
|
||||||
defer rq.mu.Unlock()
|
|
||||||
|
|
||||||
if rq.requestEnd.IsZero() {
|
|
||||||
rq.requestEnd = time.Now()
|
|
||||||
}
|
|
||||||
|
|
||||||
return rq.requestEnd
|
|
||||||
}
|
|
||||||
|
|
||||||
// stopTimers stops the request's time limits and keeps any from starting later:
|
|
||||||
// the app's answer is complete, the connection upgraded, or the request handled.
|
|
||||||
func (rq *request) stopTimers() {
|
|
||||||
rq.mu.Lock()
|
|
||||||
defer rq.mu.Unlock()
|
|
||||||
|
|
||||||
rq.timersStopped = true
|
|
||||||
stopTimer(rq.requestTimer)
|
|
||||||
stopTimer(rq.responseTimer)
|
|
||||||
}
|
|
||||||
|
|
||||||
// stopTimer stops t, which is nil when its time limit is off.
|
|
||||||
func stopTimer(t *time.Timer) {
|
|
||||||
if t != nil {
|
|
||||||
t.Stop()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
+48
-31
@@ -16,15 +16,15 @@ import (
|
|||||||
"io"
|
"io"
|
||||||
"log"
|
"log"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
|
"net"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/url"
|
"net/url"
|
||||||
"slices"
|
"slices"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
|
||||||
)
|
)
|
||||||
|
|
||||||
// How rproxy keeps connections to the app and to waf open between
|
// How rproxy keeps connections to the app and to waf open between
|
||||||
@@ -47,10 +47,6 @@ const (
|
|||||||
idleTimeout = 120 * time.Second
|
idleTimeout = 120 * time.Second
|
||||||
)
|
)
|
||||||
|
|
||||||
// ownEndpointPrefix starts the path of every request for sww itself, which
|
|
||||||
// never reaches the app.
|
|
||||||
const ownEndpointPrefix = "/_sww/"
|
|
||||||
|
|
||||||
// allowedMethods lists the methods rproxy passes on, as an Allow header gives
|
// allowedMethods lists the methods rproxy passes on, as an Allow header gives
|
||||||
// them. It answers any other method 405 itself.
|
// them. It answers any other method 405 itself.
|
||||||
// The WebDAV ones are for Nextcloud.
|
// The WebDAV ones are for Nextcloud.
|
||||||
@@ -68,8 +64,7 @@ type Params struct {
|
|||||||
|
|
||||||
// Server is the server `sww rproxy` runs.
|
// Server is the server `sww rproxy` runs.
|
||||||
type Server struct {
|
type Server struct {
|
||||||
*http.Server
|
httpServer *http.Server
|
||||||
|
|
||||||
h *handler
|
h *handler
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -92,33 +87,55 @@ func New(params Params) *Server {
|
|||||||
transport: newTransport(),
|
transport: newTransport(),
|
||||||
wafTransport: newWAFTransport(),
|
wafTransport: newWAFTransport(),
|
||||||
reportURL: (&url.URL{
|
reportURL: (&url.URL{
|
||||||
Scheme: backend.Scheme, Host: backend.Host, Path: waf.ReportPath,
|
Scheme: backend.Scheme,
|
||||||
|
Host: backend.Host,
|
||||||
|
Path: wafprotocol.ReportPath,
|
||||||
}).String(),
|
}).String(),
|
||||||
breakerClock: time.Now,
|
breakerClock: time.Now,
|
||||||
|
pages: newBlockPages(params.Config.ContactEmail),
|
||||||
}
|
}
|
||||||
|
|
||||||
return &Server{
|
s := &Server{h: h}
|
||||||
Server: &http.Server{
|
s.httpServer = &http.Server{
|
||||||
Addr: params.Config.ListenAddr,
|
Addr: params.Config.ListenAddr,
|
||||||
Handler: h,
|
Handler: s,
|
||||||
ErrorLog: errorLog,
|
ErrorLog: errorLog,
|
||||||
ReadHeaderTimeout: params.Config.MaxRequestTime,
|
ReadHeaderTimeout: params.Config.MaxRequestTime,
|
||||||
IdleTimeout: idleTimeout,
|
IdleTimeout: idleTimeout,
|
||||||
MaxHeaderBytes: requestHeaderMaxBytes - goServerExtraHeaderBytes,
|
MaxHeaderBytes: requestHeaderMaxBytes - goServerExtraHeaderBytes,
|
||||||
},
|
|
||||||
h: h,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
// Serve serves requests on listener until Shutdown or Close, as Go's
|
||||||
|
// http.Server.Serve does.
|
||||||
|
func (s *Server) Serve(listener net.Listener) error {
|
||||||
|
return s.httpServer.Serve(listener)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Shutdown stops serving once the requests in progress have finished, or
|
||||||
|
// when ctx is done, as Go's http.Server.Shutdown does.
|
||||||
|
func (s *Server) Shutdown(ctx context.Context) error {
|
||||||
|
return s.httpServer.Shutdown(ctx)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Close stops serving at once and closes every connection, as Go's
|
||||||
|
// http.Server.Close does.
|
||||||
|
func (s *Server) Close() error {
|
||||||
|
return s.httpServer.Close()
|
||||||
|
}
|
||||||
|
|
||||||
|
// ServeHTTP handles one request from a visitor, as handler.ServeHTTP says.
|
||||||
|
func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||||
|
s.h.ServeHTTP(w, r)
|
||||||
}
|
}
|
||||||
|
|
||||||
// WaitForReports returns once the reports being sent have gone or failed,
|
// WaitForReports returns once the reports being sent have gone or failed,
|
||||||
// which SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS bounds. Any report after this call
|
// which SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS bounds. Any report after this call
|
||||||
// is dropped: that of a WebSocket that closes while rproxy stops, for one.
|
// is dropped: that of a WebSocket that closes while rproxy stops, for one.
|
||||||
func (s *Server) WaitForReports() {
|
func (s *Server) WaitForReports() {
|
||||||
s.h.reportsMu.Lock()
|
s.h.reports.stopAndWait()
|
||||||
s.h.reportsStopped = true
|
|
||||||
s.h.reportsMu.Unlock()
|
|
||||||
|
|
||||||
s.h.reports.Wait()
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// handler is the proxy. It holds what every request shares; what belongs to
|
// handler is the proxy. It holds what every request shares; what belongs to
|
||||||
@@ -138,11 +155,10 @@ type handler struct {
|
|||||||
// the time.
|
// the time.
|
||||||
breaker breaker
|
breaker breaker
|
||||||
breakerClock func() time.Time
|
breakerClock func() time.Time
|
||||||
// reports waits for the reports being sent. reportsMu guards starting one
|
// reports are the goroutines sending reports to waf.
|
||||||
// against reportsStopped, which is set once nothing is to wait for another.
|
reports reportGoroutines
|
||||||
reports sync.WaitGroup
|
// pages are the pages a refused visitor gets.
|
||||||
reportsMu sync.Mutex
|
pages blockPages
|
||||||
reportsStopped bool
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// newTransport returns what carries requests to the app. It never goes through
|
// newTransport returns what carries requests to the app. It never goes through
|
||||||
@@ -179,7 +195,7 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
|||||||
defer rq.readRestOfBody()
|
defer rq.readRestOfBody()
|
||||||
defer rq.finish(r.Context())
|
defer rq.finish(r.Context())
|
||||||
|
|
||||||
if r.Method == http.MethodGet && r.URL.Path == waf.HealthPath {
|
if r.Method == http.MethodGet && r.URL.Path == wafprotocol.HealthPath {
|
||||||
rq.answerHealth()
|
rq.answerHealth()
|
||||||
|
|
||||||
return
|
return
|
||||||
@@ -212,20 +228,21 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
|||||||
rq.noteAnswer(answer)
|
rq.noteAnswer(answer)
|
||||||
|
|
||||||
switch rq.disposition {
|
switch rq.disposition {
|
||||||
case waf.DispositionAllow:
|
case wafprotocol.DispositionAllow:
|
||||||
rq.passToApp(r.Context())
|
rq.passToApp(r.Context())
|
||||||
case waf.DispositionClose:
|
case wafprotocol.DispositionClose:
|
||||||
// This panic has Go's server close the connection without an answer,
|
// This panic has Go's server close the connection without an answer,
|
||||||
// and log nothing; the deferred finish still runs.
|
// and log nothing; the deferred finish still runs.
|
||||||
panic(http.ErrAbortHandler)
|
panic(http.ErrAbortHandler)
|
||||||
default: // a deny, an error, or an answer without a valid disposition
|
case wafprotocol.DispositionDeny, "":
|
||||||
|
// "" is an error, or no valid disposition
|
||||||
rq.answerWith(answer)
|
rq.answerWith(answer)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// passToApp passes the request on to the app, unless a limit or its path keeps
|
// passToApp passes the request on to the app, unless a limit or its path keeps
|
||||||
// it from the app. waf answers a path under ownEndpointPrefix itself, so
|
// it from the app. waf answers a path under wafprotocol.OwnEndpointPrefix
|
||||||
// such a request gets here only when
|
// itself, so such a request gets here only when
|
||||||
// SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION allowed it.
|
// SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION allowed it.
|
||||||
// ctx is the request's own context.
|
// ctx is the request's own context.
|
||||||
func (rq *request) passToApp(ctx context.Context) {
|
func (rq *request) passToApp(ctx context.Context) {
|
||||||
@@ -234,7 +251,7 @@ func (rq *request) passToApp(ctx context.Context) {
|
|||||||
switch {
|
switch {
|
||||||
case rq.refused.Load() != 0 || (maxBytes > 0 && rq.in.ContentLength > maxBytes):
|
case rq.refused.Load() != 0 || (maxBytes > 0 && rq.in.ContentLength > maxBytes):
|
||||||
rq.answer(http.StatusRequestEntityTooLarge)
|
rq.answer(http.StatusRequestEntityTooLarge)
|
||||||
case strings.HasPrefix(rq.in.URL.Path, ownEndpointPrefix):
|
case strings.HasPrefix(rq.in.URL.Path, wafprotocol.OwnEndpointPrefix):
|
||||||
rq.answer(http.StatusNotFound)
|
rq.answer(http.StatusNotFound)
|
||||||
default:
|
default:
|
||||||
rq.forward(ctx)
|
rq.forward(ctx)
|
||||||
|
|||||||
+199
-48
@@ -11,6 +11,7 @@ import (
|
|||||||
"net"
|
"net"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
|
"os"
|
||||||
"slices"
|
"slices"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -23,7 +24,7 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
|
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/processlog"
|
"sneak.berlin/go/smallwebwaf/internal/processlog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/rproxy"
|
"sneak.berlin/go/smallwebwaf/internal/rproxy"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -52,11 +53,59 @@ const (
|
|||||||
maxResponseSize = "SWW_RPROXY_MAX_RESPONSE_SIZE"
|
maxResponseSize = "SWW_RPROXY_MAX_RESPONSE_SIZE"
|
||||||
maxRequestTime = "SWW_RPROXY_MAX_REQUEST_TIME_MS"
|
maxRequestTime = "SWW_RPROXY_MAX_REQUEST_TIME_MS"
|
||||||
maxResponseTime = "SWW_RPROXY_MAX_RESPONSE_TIME_MS"
|
maxResponseTime = "SWW_RPROXY_MAX_RESPONSE_TIME_MS"
|
||||||
|
contactEmail = "SWW_RPROXY_CONTACT_EMAIL"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// The pages rproxy gives a refused visitor in place of waf's body.
|
||||||
|
const (
|
||||||
|
// refusedPageFile and rateLimitedPageFile are the pages' files, beside
|
||||||
|
// the code.
|
||||||
|
refusedPageFile = "refused.html"
|
||||||
|
rateLimitedPageFile = "ratelimited.html"
|
||||||
|
// contactSentence is the sentence of each page that becomes a link to
|
||||||
|
// SWW_RPROXY_CONTACT_EMAIL.
|
||||||
|
contactSentence = "contact the site's operator"
|
||||||
|
htmlContentType = "text/html; charset=utf-8"
|
||||||
|
)
|
||||||
|
|
||||||
|
// page is the page in file, as rproxy serves it without a contact email.
|
||||||
|
func page(t *testing.T, file string) string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
//nolint:gosec // a page beside the code
|
||||||
|
contents, err := os.ReadFile(file)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read the page: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return string(contents)
|
||||||
|
}
|
||||||
|
|
||||||
|
// wantPage checks that got is status with the page in file as its body, and
|
||||||
|
// the headers of a page.
|
||||||
|
func wantPage(t *testing.T, got answered, status int, file string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
if got.status != status || got.body != page(t, file) {
|
||||||
|
t.Errorf(
|
||||||
|
"answered %d %q, want %d and %s",
|
||||||
|
got.status,
|
||||||
|
got.body,
|
||||||
|
status,
|
||||||
|
file,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got.header.Get("Content-Type") != htmlContentType ||
|
||||||
|
got.header.Get("X-Content-Type-Options") != "nosniff" ||
|
||||||
|
got.header.Get("Cache-Control") != "no-store" {
|
||||||
|
t.Errorf("the page's headers are %v", got.header)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// The headers the tests set and look at.
|
// The headers the tests set and look at.
|
||||||
const (
|
const (
|
||||||
dispositionHeader = waf.DispositionHeader
|
dispositionHeader = wafprotocol.DispositionHeader
|
||||||
forwardedForHeader = "X-Forwarded-For"
|
forwardedForHeader = "X-Forwarded-For"
|
||||||
forwardedProtoHeader = "X-Forwarded-Proto"
|
forwardedProtoHeader = "X-Forwarded-Proto"
|
||||||
requestIDHeader = "X-Request-ID"
|
requestIDHeader = "X-Request-ID"
|
||||||
@@ -72,9 +121,9 @@ const (
|
|||||||
|
|
||||||
// What the stand-in for waf answers.
|
// What the stand-in for waf answers.
|
||||||
const (
|
const (
|
||||||
dispositionAllow = waf.DispositionAllow
|
dispositionAllow = string(wafprotocol.DispositionAllow)
|
||||||
dispositionDeny = waf.DispositionDeny
|
dispositionDeny = string(wafprotocol.DispositionDeny)
|
||||||
dispositionClose = waf.DispositionClose
|
dispositionClose = string(wafprotocol.DispositionClose)
|
||||||
// deniedBody is the body of a deny.
|
// deniedBody is the body of a deny.
|
||||||
deniedBody = "refused by waf"
|
deniedBody = "refused by waf"
|
||||||
)
|
)
|
||||||
@@ -175,7 +224,7 @@ type WAF struct {
|
|||||||
*httptest.Server
|
*httptest.Server
|
||||||
|
|
||||||
questions chan question
|
questions chan question
|
||||||
reports chan waf.Report
|
reports chan wafprotocol.Report
|
||||||
}
|
}
|
||||||
|
|
||||||
// startWAF starts a stand-in for waf.
|
// startWAF starts a stand-in for waf.
|
||||||
@@ -184,11 +233,11 @@ func startWAF(t *testing.T, answer http.HandlerFunc) *WAF {
|
|||||||
|
|
||||||
s := &WAF{
|
s := &WAF{
|
||||||
questions: make(chan question, 10),
|
questions: make(chan question, 10),
|
||||||
reports: make(chan waf.Report, 10),
|
reports: make(chan wafprotocol.Report, 10),
|
||||||
}
|
}
|
||||||
s.Server = httptest.NewServer(http.HandlerFunc(
|
s.Server = httptest.NewServer(http.HandlerFunc(
|
||||||
func(w http.ResponseWriter, r *http.Request) {
|
func(w http.ResponseWriter, r *http.Request) {
|
||||||
if r.Header.Get(waf.ClientHeader) == "" {
|
if r.Header.Get(wafprotocol.ClientHeader) == "" {
|
||||||
s.takeReport(t, w, r)
|
s.takeReport(t, w, r)
|
||||||
|
|
||||||
return
|
return
|
||||||
@@ -217,11 +266,11 @@ func (s *WAF) takeReport(
|
|||||||
) {
|
) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
var report waf.Report
|
var report wafprotocol.Report
|
||||||
|
|
||||||
err := json.NewDecoder(r.Body).Decode(&report)
|
err := json.NewDecoder(r.Body).Decode(&report)
|
||||||
if err != nil || r.Method != http.MethodPost ||
|
if err != nil || r.Method != http.MethodPost ||
|
||||||
r.URL.Path != waf.ReportPath {
|
r.URL.Path != wafprotocol.ReportPath {
|
||||||
t.Errorf("%s %s is no report: %v", r.Method, r.URL.Path, err)
|
t.Errorf("%s %s is no report: %v", r.Method, r.URL.Path, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -242,7 +291,7 @@ func (s *WAF) wantNoReport(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// report waits for a report.
|
// report waits for a report.
|
||||||
func (s *WAF) report(t *testing.T) waf.Report {
|
func (s *WAF) report(t *testing.T) wafprotocol.Report {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
select {
|
select {
|
||||||
@@ -251,7 +300,7 @@ func (s *WAF) report(t *testing.T) waf.Report {
|
|||||||
case <-time.After(waitLimit):
|
case <-time.After(waitLimit):
|
||||||
t.Fatal("no report")
|
t.Fatal("no report")
|
||||||
|
|
||||||
return waf.Report{}
|
return wafprotocol.Report{}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -279,7 +328,7 @@ func startWAFAnsweringReports(
|
|||||||
|
|
||||||
backend := httptest.NewServer(http.HandlerFunc(
|
backend := httptest.NewServer(http.HandlerFunc(
|
||||||
func(w http.ResponseWriter, r *http.Request) {
|
func(w http.ResponseWriter, r *http.Request) {
|
||||||
if r.Header.Get(waf.ClientHeader) != "" {
|
if r.Header.Get(wafprotocol.ClientHeader) != "" {
|
||||||
answer(w, r)
|
answer(w, r)
|
||||||
|
|
||||||
return
|
return
|
||||||
@@ -306,6 +355,20 @@ func deny(w http.ResponseWriter, _ *http.Request) {
|
|||||||
_, _ = io.WriteString(w, deniedBody)
|
_, _ = io.WriteString(w, deniedBody)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// answerAs answers a question with status and body, with disposition unless
|
||||||
|
// it is "", and a header of its own.
|
||||||
|
func answerAs(disposition string, status int, body string) http.HandlerFunc {
|
||||||
|
return func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
if disposition != "" {
|
||||||
|
w.Header().Set(dispositionHeader, disposition)
|
||||||
|
}
|
||||||
|
|
||||||
|
w.Header().Set(fromWAFHeader, addedHeaderValue)
|
||||||
|
w.WriteHeader(status)
|
||||||
|
_, _ = io.WriteString(w, body)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// startApp starts app as the app rproxy passes requests to.
|
// startApp starts app as the app rproxy passes requests to.
|
||||||
func startApp(t *testing.T, app http.HandlerFunc) *httptest.Server {
|
func startApp(t *testing.T, app http.HandlerFunc) *httptest.Server {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
@@ -710,7 +773,7 @@ func TestAnAnswerWithoutAContentTypeGetsNone(t *testing.T) {
|
|||||||
)
|
)
|
||||||
w := httptest.NewRecorder()
|
w := httptest.NewRecorder()
|
||||||
|
|
||||||
server.Handler.ServeHTTP(w, req)
|
server.ServeHTTP(w, req)
|
||||||
|
|
||||||
// Go's server guesses a type when the first part of the body goes before the
|
// Go's server guesses a type when the first part of the body goes before the
|
||||||
// headers are flushed, which is a matter of timing here. So this looks for what
|
// headers are flushed, which is a matter of timing here. So this looks for what
|
||||||
@@ -767,7 +830,7 @@ func TestRemovesTheVisitorsLookupHeadersWhenWAFGivesNone(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestDenyGivesTheVisitorWAFsAnswer(t *testing.T) {
|
func TestDenyGivesTheVisitorTheRefusedPageWithWAFsHeaders(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
standIn := startWAF(t, deny)
|
standIn := startWAF(t, deny)
|
||||||
@@ -777,9 +840,10 @@ func TestDenyGivesTheVisitorWAFsAnswer(t *testing.T) {
|
|||||||
addr, out := startRProxy(t, standIn.URL, app.URL, nil)
|
addr, out := startRProxy(t, standIn.URL, app.URL, nil)
|
||||||
|
|
||||||
got := get(t, addr, "/")
|
got := get(t, addr, "/")
|
||||||
if got.status != http.StatusForbidden || got.body != deniedBody ||
|
wantPage(t, got, http.StatusForbidden, refusedPageFile)
|
||||||
got.header.Get(fromWAFHeader) != addedHeaderValue {
|
|
||||||
t.Errorf("answered %+v, want waf's answer", got)
|
if got.header.Get(fromWAFHeader) != addedHeaderValue {
|
||||||
|
t.Errorf("answered %+v, want waf's headers", got)
|
||||||
}
|
}
|
||||||
|
|
||||||
if len(got.header.Values(dispositionHeader)) != 0 {
|
if len(got.header.Values(dispositionHeader)) != 0 {
|
||||||
@@ -788,7 +852,7 @@ func TestDenyGivesTheVisitorWAFsAnswer(t *testing.T) {
|
|||||||
|
|
||||||
report := standIn.report(t)
|
report := standIn.report(t)
|
||||||
if report.Status != http.StatusForbidden ||
|
if report.Status != http.StatusForbidden ||
|
||||||
report.ResponseBytes != int64(len(deniedBody)) ||
|
report.ResponseBytes != int64(len(page(t, refusedPageFile))) ||
|
||||||
report.DurationUpstreamTotalMS != 0 {
|
report.DurationUpstreamTotalMS != 0 {
|
||||||
t.Errorf("report %+v, of a request denied", report)
|
t.Errorf("report %+v, of a request denied", report)
|
||||||
}
|
}
|
||||||
@@ -800,7 +864,7 @@ func TestDenyGivesTheVisitorWAFsAnswer(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestADeniedHeadKeepsWAFsContentLength(t *testing.T) {
|
func TestADeniedHeadGivesThePagesContentLength(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
standIn := startWAF(t, deny)
|
standIn := startWAF(t, deny)
|
||||||
@@ -808,13 +872,13 @@ func TestADeniedHeadKeepsWAFsContentLength(t *testing.T) {
|
|||||||
addr, _ := startRProxy(t, standIn.URL, app.URL, nil)
|
addr, _ := startRProxy(t, standIn.URL, app.URL, nil)
|
||||||
|
|
||||||
got := visit(t, newRequest(t, http.MethodHead, addr, "/", http.NoBody))
|
got := visit(t, newRequest(t, http.MethodHead, addr, "/", http.NoBody))
|
||||||
if got.status != http.StatusForbidden ||
|
if got.status != http.StatusForbidden || got.body != "" ||
|
||||||
got.header.Get(
|
got.header.Get(
|
||||||
contentLengthHeader,
|
contentLengthHeader,
|
||||||
) != strconv.Itoa(
|
) != strconv.Itoa(
|
||||||
len(deniedBody),
|
len(page(t, refusedPageFile)),
|
||||||
) {
|
) {
|
||||||
t.Errorf("answered %+v, want waf's 403 and its length", got)
|
t.Errorf("answered %+v, want waf's 403 and the page's length", got)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -823,13 +887,15 @@ func TestAnAnswerRProxyHoldsWholeIsSentBeforeItsLogLine(t *testing.T) {
|
|||||||
|
|
||||||
const host = " HTTP/1.1\r\nHost: site.example\r\n\r\n"
|
const host = " HTTP/1.1\r\nHost: site.example\r\n\r\n"
|
||||||
|
|
||||||
|
refusedPage := page(t, refusedPageFile)
|
||||||
|
|
||||||
for _, tc := range []struct {
|
for _, tc := range []struct {
|
||||||
name, request, body string
|
name, request, body string
|
||||||
status int
|
status int
|
||||||
}{
|
}{
|
||||||
{"a deny", "GET /" + host, deniedBody, http.StatusForbidden},
|
{"a deny", "GET /" + host, refusedPage, http.StatusForbidden},
|
||||||
{"rproxy's own answer", "GET x:admin" + host, "Bad Request\n", http.StatusBadRequest},
|
{"rproxy's own answer", "GET x:admin" + host, "Bad Request\n", http.StatusBadRequest},
|
||||||
{"the health check", "GET " + waf.HealthPath + host, "ok\n", http.StatusOK},
|
{"the health check", "GET " + wafprotocol.HealthPath + host, "ok\n", http.StatusOK},
|
||||||
} {
|
} {
|
||||||
t.Run(tc.name, func(t *testing.T) {
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
@@ -899,6 +965,83 @@ func TestRProxysOwnAnswerIsPlainText(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestADenyOf429GivesTheVisitorTheRateLimitedPage(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
standIn := startWAF(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
w.Header().Set(dispositionHeader, dispositionDeny)
|
||||||
|
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
|
||||||
|
w.WriteHeader(http.StatusTooManyRequests)
|
||||||
|
_, _ = io.WriteString(w, deniedBody)
|
||||||
|
})
|
||||||
|
app := startApp(t, greet)
|
||||||
|
addr, _ := startRProxy(t, standIn.URL, app.URL, nil)
|
||||||
|
|
||||||
|
wantPage(
|
||||||
|
t,
|
||||||
|
get(t, addr, "/"),
|
||||||
|
http.StatusTooManyRequests,
|
||||||
|
rateLimitedPageFile,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestThePagesLinkTheContactEmail(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const email = "ops@example.com"
|
||||||
|
|
||||||
|
link := `<a href="mailto:` + email + `">` + contactSentence + `</a>`
|
||||||
|
|
||||||
|
for _, tc := range []struct {
|
||||||
|
file string
|
||||||
|
status int
|
||||||
|
}{
|
||||||
|
{refusedPageFile, http.StatusForbidden},
|
||||||
|
{rateLimitedPageFile, http.StatusTooManyRequests},
|
||||||
|
} {
|
||||||
|
t.Run(tc.file, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
standIn := startWAF(
|
||||||
|
t,
|
||||||
|
func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
w.Header().Set(dispositionHeader, dispositionDeny)
|
||||||
|
w.WriteHeader(tc.status)
|
||||||
|
},
|
||||||
|
)
|
||||||
|
app := startApp(t, greet)
|
||||||
|
addr, _ := startRProxy(
|
||||||
|
t,
|
||||||
|
standIn.URL,
|
||||||
|
app.URL,
|
||||||
|
map[string]string{contactEmail: email},
|
||||||
|
)
|
||||||
|
|
||||||
|
got := get(t, addr, "/")
|
||||||
|
want := strings.Replace(
|
||||||
|
page(t, tc.file),
|
||||||
|
contactSentence,
|
||||||
|
link,
|
||||||
|
1,
|
||||||
|
)
|
||||||
|
|
||||||
|
if got.status != tc.status || got.body != want {
|
||||||
|
t.Errorf(
|
||||||
|
"answered %d %q, want %d %q",
|
||||||
|
got.status,
|
||||||
|
got.body,
|
||||||
|
tc.status,
|
||||||
|
want,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !strings.Contains(page(t, tc.file), contactSentence) {
|
||||||
|
t.Errorf("%s has no %q to link", tc.file, contactSentence)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestCloseClosesTheConnectionWithoutAnAnswer(t *testing.T) {
|
func TestCloseClosesTheConnectionWithoutAnAnswer(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -1017,17 +1160,28 @@ func noAnswers() map[string]func(t *testing.T) string {
|
|||||||
func TestAnAnswerOtherThanAllowGoesToTheVisitorAsWAFGaveIt(t *testing.T) {
|
func TestAnAnswerOtherThanAllowGoesToTheVisitorAsWAFGaveIt(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
const unknownBody = "who knows\n"
|
||||||
|
|
||||||
|
refusedPage := page(t, refusedPageFile)
|
||||||
|
rateLimitedPage := page(t, rateLimitedPageFile)
|
||||||
|
|
||||||
|
// want is the body the visitor gets: waf's, but for a refusal, which gets
|
||||||
|
// rproxy's page for its status.
|
||||||
for _, tc := range []struct {
|
for _, tc := range []struct {
|
||||||
name, disposition, body string
|
name, disposition, body, want string
|
||||||
status int
|
status int
|
||||||
}{
|
}{
|
||||||
{"a refusal", dispositionDeny, deniedBody, http.StatusForbidden},
|
{"a refusal", dispositionDeny, deniedBody, refusedPage, http.StatusForbidden},
|
||||||
{"a refusal, 429", dispositionDeny, deniedBody, http.StatusTooManyRequests},
|
{
|
||||||
{"waf's 400", "", "refused\n", http.StatusBadRequest},
|
"a refusal, 429", dispositionDeny, deniedBody, rateLimitedPage,
|
||||||
{"a 500", "", "broken\n", http.StatusInternalServerError},
|
http.StatusTooManyRequests,
|
||||||
{"a 503 denied", dispositionDeny, "busy\n", http.StatusServiceUnavailable},
|
},
|
||||||
{"no header", "", "who knows\n", http.StatusOK},
|
{"waf's 400", "", "refused\n", "refused\n", http.StatusBadRequest},
|
||||||
{"an unknown disposition", "maybe", "who knows\n", http.StatusOK},
|
{"a 500", "", "broken\n", "broken\n", http.StatusInternalServerError},
|
||||||
|
{"a 503 denied", dispositionDeny, "busy\n", "busy\n", http.StatusServiceUnavailable},
|
||||||
|
{"a 403 without a disposition", "", "nope\n", "nope\n", http.StatusForbidden},
|
||||||
|
{"no header", "", unknownBody, unknownBody, http.StatusOK},
|
||||||
|
{"an unknown disposition", "maybe", unknownBody, unknownBody, http.StatusOK},
|
||||||
} {
|
} {
|
||||||
for _, setting := range []string{dispositionAllow, dispositionDeny} {
|
for _, setting := range []string{dispositionAllow, dispositionDeny} {
|
||||||
subtest := tc.name + " with " + noAnswerSetting + "=" + setting
|
subtest := tc.name + " with " + noAnswerSetting + "=" + setting
|
||||||
@@ -1036,15 +1190,7 @@ func TestAnAnswerOtherThanAllowGoesToTheVisitorAsWAFGaveIt(t *testing.T) {
|
|||||||
|
|
||||||
standIn := startWAF(
|
standIn := startWAF(
|
||||||
t,
|
t,
|
||||||
func(w http.ResponseWriter, _ *http.Request) {
|
answerAs(tc.disposition, tc.status, tc.body),
|
||||||
if tc.disposition != "" {
|
|
||||||
w.Header().Set(dispositionHeader, tc.disposition)
|
|
||||||
}
|
|
||||||
|
|
||||||
w.Header().Set(fromWAFHeader, addedHeaderValue)
|
|
||||||
w.WriteHeader(tc.status)
|
|
||||||
_, _ = io.WriteString(w, tc.body)
|
|
||||||
},
|
|
||||||
)
|
)
|
||||||
app := startApp(t, func(http.ResponseWriter, *http.Request) {
|
app := startApp(t, func(http.ResponseWriter, *http.Request) {
|
||||||
t.Error("the app was asked")
|
t.Error("the app was asked")
|
||||||
@@ -1059,9 +1205,14 @@ func TestAnAnswerOtherThanAllowGoesToTheVisitorAsWAFGaveIt(t *testing.T) {
|
|||||||
)
|
)
|
||||||
|
|
||||||
got := get(t, addr, "/")
|
got := get(t, addr, "/")
|
||||||
if got.status != tc.status || got.body != tc.body ||
|
if got.status != tc.status || got.body != tc.want ||
|
||||||
got.header.Get(fromWAFHeader) != addedHeaderValue {
|
got.header.Get(fromWAFHeader) != addedHeaderValue {
|
||||||
t.Errorf("answered %+v, want waf's answer", got)
|
t.Errorf(
|
||||||
|
"answered %+v, want %d %q",
|
||||||
|
got,
|
||||||
|
tc.status,
|
||||||
|
tc.want,
|
||||||
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
if len(got.header.Values(dispositionHeader)) != 0 {
|
if len(got.header.Values(dispositionHeader)) != 0 {
|
||||||
@@ -1602,19 +1753,19 @@ func TestHealthCheckIsAnsweredWhileWAFIsDown(t *testing.T) {
|
|||||||
app := startApp(t, greet)
|
app := startApp(t, greet)
|
||||||
addr, out := startRProxy(t, nowhere, app.URL, nil)
|
addr, out := startRProxy(t, nowhere, app.URL, nil)
|
||||||
|
|
||||||
got := get(t, addr, waf.HealthPath)
|
got := get(t, addr, wafprotocol.HealthPath)
|
||||||
if got.status != http.StatusOK || got.body != "ok\n" {
|
if got.status != http.StatusOK || got.body != "ok\n" {
|
||||||
t.Errorf("answered %+v, want ok", got)
|
t.Errorf("answered %+v, want ok", got)
|
||||||
}
|
}
|
||||||
|
|
||||||
line := out.requestLine(t)
|
line := out.requestLine(t)
|
||||||
if line["path"] != waf.HealthPath || line["disposition"] != nil {
|
if line["path"] != wafprotocol.HealthPath || line["disposition"] != nil {
|
||||||
t.Errorf("line %v, want one with no disposition", line)
|
t.Errorf("line %v, want one with no disposition", line)
|
||||||
}
|
}
|
||||||
|
|
||||||
// With waf up too, it is not asked.
|
// With waf up too, it is not asked.
|
||||||
addr, _ = startRProxy(t, standIn.URL, app.URL, nil)
|
addr, _ = startRProxy(t, standIn.URL, app.URL, nil)
|
||||||
get(t, addr, waf.HealthPath)
|
get(t, addr, wafprotocol.HealthPath)
|
||||||
|
|
||||||
if len(standIn.questions) != 0 {
|
if len(standIn.questions) != 0 {
|
||||||
t.Error("waf was asked about the health check")
|
t.Error("waf was asked about the health check")
|
||||||
@@ -1775,7 +1926,7 @@ func TestWaitForReportsWaitsForAReportBeingSent(t *testing.T) {
|
|||||||
release := make(chan struct{})
|
release := make(chan struct{})
|
||||||
backend := httptest.NewServer(http.HandlerFunc(
|
backend := httptest.NewServer(http.HandlerFunc(
|
||||||
func(w http.ResponseWriter, r *http.Request) {
|
func(w http.ResponseWriter, r *http.Request) {
|
||||||
if r.Header.Get(waf.ClientHeader) != "" {
|
if r.Header.Get(wafprotocol.ClientHeader) != "" {
|
||||||
allow(w, r)
|
allow(w, r)
|
||||||
|
|
||||||
return
|
return
|
||||||
|
|||||||
@@ -0,0 +1,165 @@
|
|||||||
|
package rproxy
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptrace"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// timeLimits are one request's two time limits: SWW_RPROXY_MAX_REQUEST_TIME_MS,
|
||||||
|
// which runs from the request's headers being read until the app has been sent
|
||||||
|
// the whole request, and SWW_RPROXY_MAX_RESPONSE_TIME_MS, which runs from then
|
||||||
|
// until the app has sent its whole answer. A limit of zero is off. Each limit
|
||||||
|
// runs on a goroutine of its own, and the transport starts and stops them from
|
||||||
|
// its own.
|
||||||
|
type timeLimits struct {
|
||||||
|
maxRequestTime time.Duration
|
||||||
|
maxResponseTime time.Duration
|
||||||
|
// start is when the request's headers had been read.
|
||||||
|
start time.Time
|
||||||
|
// refuse refuses the request with 408 or 504 when a limit runs out.
|
||||||
|
refuse func(status int)
|
||||||
|
// waitingForVisitor reports whether a read of the visitor's body is waiting
|
||||||
|
// for the visitor to send more.
|
||||||
|
waitingForVisitor func() bool
|
||||||
|
// endVisitorRead ends at once a read of the visitor's body that is waiting.
|
||||||
|
endVisitorRead func()
|
||||||
|
|
||||||
|
// mu guards what follows. Once stopped is set, neither limit acts any more.
|
||||||
|
mu sync.Mutex
|
||||||
|
stopped bool
|
||||||
|
requestTimer *time.Timer
|
||||||
|
responseTimer *time.Timer
|
||||||
|
// requestEnd is when the app had been sent the whole request, or the answer
|
||||||
|
// started, if that was sooner.
|
||||||
|
requestEnd time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
// requestDeadline is when the visitor must have sent its whole request, or zero
|
||||||
|
// when SWW_RPROXY_MAX_REQUEST_TIME_MS is off.
|
||||||
|
func (l *timeLimits) requestDeadline() time.Time {
|
||||||
|
if l.maxRequestTime == 0 {
|
||||||
|
return time.Time{}
|
||||||
|
}
|
||||||
|
|
||||||
|
return l.start.Add(l.maxRequestTime)
|
||||||
|
}
|
||||||
|
|
||||||
|
// startRequestTimer starts SWW_RPROXY_MAX_REQUEST_TIME_MS.
|
||||||
|
func (l *timeLimits) startRequestTimer() {
|
||||||
|
deadline := l.requestDeadline()
|
||||||
|
if deadline.IsZero() {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
l.mu.Lock()
|
||||||
|
defer l.mu.Unlock()
|
||||||
|
|
||||||
|
l.requestTimer = time.AfterFunc(time.Until(deadline), l.requestTimedOut)
|
||||||
|
}
|
||||||
|
|
||||||
|
// requestTimedOut is called when SWW_RPROXY_MAX_REQUEST_TIME_MS runs out while
|
||||||
|
// the request is still on its way to the app. The answer names the side rproxy
|
||||||
|
// was waiting on at that moment: 408 when it was waiting for the visitor to send
|
||||||
|
// more of its body, 504 when it was waiting for the app to be reached or to take
|
||||||
|
// what it had.
|
||||||
|
func (l *timeLimits) requestTimedOut() {
|
||||||
|
l.mu.Lock()
|
||||||
|
defer l.mu.Unlock()
|
||||||
|
|
||||||
|
if l.stopped {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if !l.waitingForVisitor() {
|
||||||
|
l.refuse(http.StatusGatewayTimeout)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
l.refuse(http.StatusRequestTimeout)
|
||||||
|
// The transport gives up on the app only once its Read of the visitor's body
|
||||||
|
// returns, so that Read is ended now. The lock keeps this from reaching the
|
||||||
|
// connection after the request is handled.
|
||||||
|
l.endVisitorRead()
|
||||||
|
}
|
||||||
|
|
||||||
|
// wroteRequest is called once the app has been sent the whole request:
|
||||||
|
// SWW_RPROXY_MAX_REQUEST_TIME_MS ends and SWW_RPROXY_MAX_RESPONSE_TIME_MS
|
||||||
|
// starts.
|
||||||
|
func (l *timeLimits) wroteRequest(info httptrace.WroteRequestInfo) {
|
||||||
|
if info.Err != nil {
|
||||||
|
return // the transport gives up, or tries again
|
||||||
|
}
|
||||||
|
|
||||||
|
requestEnd := l.endRequest()
|
||||||
|
|
||||||
|
l.mu.Lock()
|
||||||
|
stopTimer(l.requestTimer)
|
||||||
|
l.mu.Unlock()
|
||||||
|
|
||||||
|
l.startResponseTimer(requestEnd)
|
||||||
|
}
|
||||||
|
|
||||||
|
// startResponseTimer starts SWW_RPROXY_MAX_RESPONSE_TIME_MS, which runs from
|
||||||
|
// requestEnd, unless it has started already.
|
||||||
|
func (l *timeLimits) startResponseTimer(requestEnd time.Time) {
|
||||||
|
if l.maxResponseTime == 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
l.mu.Lock()
|
||||||
|
defer l.mu.Unlock()
|
||||||
|
|
||||||
|
if l.stopped || l.responseTimer != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
l.responseTimer = time.AfterFunc(
|
||||||
|
time.Until(requestEnd.Add(l.maxResponseTime)),
|
||||||
|
l.responseTimedOut,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// responseTimedOut is called when SWW_RPROXY_MAX_RESPONSE_TIME_MS runs out
|
||||||
|
// before the app has sent its whole answer.
|
||||||
|
func (l *timeLimits) responseTimedOut() {
|
||||||
|
l.mu.Lock()
|
||||||
|
defer l.mu.Unlock()
|
||||||
|
|
||||||
|
if !l.stopped {
|
||||||
|
l.refuse(http.StatusGatewayTimeout)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// endRequest notes now as the end of the request, unless an end is noted, and
|
||||||
|
// returns the end noted.
|
||||||
|
func (l *timeLimits) endRequest() time.Time {
|
||||||
|
l.mu.Lock()
|
||||||
|
defer l.mu.Unlock()
|
||||||
|
|
||||||
|
if l.requestEnd.IsZero() {
|
||||||
|
l.requestEnd = time.Now()
|
||||||
|
}
|
||||||
|
|
||||||
|
return l.requestEnd
|
||||||
|
}
|
||||||
|
|
||||||
|
// stop stops both limits and keeps either from starting later: the app's
|
||||||
|
// answer is complete, the connection upgraded, or the request handled.
|
||||||
|
func (l *timeLimits) stop() {
|
||||||
|
l.mu.Lock()
|
||||||
|
defer l.mu.Unlock()
|
||||||
|
|
||||||
|
l.stopped = true
|
||||||
|
stopTimer(l.requestTimer)
|
||||||
|
stopTimer(l.responseTimer)
|
||||||
|
}
|
||||||
|
|
||||||
|
// stopTimer stops t, which is nil when its limit is off.
|
||||||
|
func stopTimer(t *time.Timer) {
|
||||||
|
if t != nil {
|
||||||
|
t.Stop()
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,171 @@
|
|||||||
|
package rproxy
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptrace"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"testing/synctest"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The tests below run timeLimits in a synctest bubble, where time is a clock of
|
||||||
|
// the test's own: time.Sleep moves it on at once, and synctest.Wait returns once
|
||||||
|
// a limit that ran out has refused the request.
|
||||||
|
|
||||||
|
const (
|
||||||
|
testRequestTime = 3 * time.Second
|
||||||
|
testResponseTime = 5 * time.Second
|
||||||
|
)
|
||||||
|
|
||||||
|
// limitsSeen is what a timeLimits under test did to its request.
|
||||||
|
type limitsSeen struct {
|
||||||
|
// refused is the status of the last refusal, or zero.
|
||||||
|
refused atomic.Int64
|
||||||
|
// readEnded is true once the read of the visitor's body was ended.
|
||||||
|
readEnded atomic.Bool
|
||||||
|
// waiting is what waitingForVisitor reports.
|
||||||
|
waiting atomic.Bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// newTestLimits returns time limits that started now, with what they do noted
|
||||||
|
// in seen.
|
||||||
|
func newTestLimits(seen *limitsSeen) *timeLimits {
|
||||||
|
return &timeLimits{
|
||||||
|
maxRequestTime: testRequestTime,
|
||||||
|
maxResponseTime: testResponseTime,
|
||||||
|
start: time.Now(),
|
||||||
|
refuse: func(status int) {
|
||||||
|
seen.refused.Store(int64(status))
|
||||||
|
},
|
||||||
|
waitingForVisitor: seen.waiting.Load,
|
||||||
|
endVisitorRead: func() {
|
||||||
|
seen.readEnded.Store(true)
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// sleepThenWait moves the bubble's clock on by d and waits for what is then
|
||||||
|
// due.
|
||||||
|
func sleepThenWait(d time.Duration) {
|
||||||
|
time.Sleep(d)
|
||||||
|
synctest.Wait()
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRequestTimeRunningOutWhileTheVisitorIsWaitedOnRefuses408(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
seen := &limitsSeen{}
|
||||||
|
seen.waiting.Store(true)
|
||||||
|
|
||||||
|
limits := newTestLimits(seen)
|
||||||
|
limits.startRequestTimer()
|
||||||
|
|
||||||
|
sleepThenWait(testRequestTime - time.Millisecond)
|
||||||
|
|
||||||
|
if got := seen.refused.Load(); got != 0 {
|
||||||
|
t.Fatalf(
|
||||||
|
"refused %d before the limit ran out, want nothing",
|
||||||
|
got,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
sleepThenWait(time.Millisecond)
|
||||||
|
|
||||||
|
if got := seen.refused.Load(); got != http.StatusRequestTimeout {
|
||||||
|
t.Errorf("refused %d, want 408", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !seen.readEnded.Load() {
|
||||||
|
t.Error("the read of the visitor's body goes on, want it ended")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRequestTimeRunningOutWhileTheAppIsWaitedOnRefuses504(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
seen := &limitsSeen{}
|
||||||
|
|
||||||
|
limits := newTestLimits(seen)
|
||||||
|
limits.startRequestTimer()
|
||||||
|
|
||||||
|
sleepThenWait(testRequestTime)
|
||||||
|
|
||||||
|
if got := seen.refused.Load(); got != http.StatusGatewayTimeout {
|
||||||
|
t.Errorf("refused %d, want 504", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
if seen.readEnded.Load() {
|
||||||
|
t.Error("the read of the visitor's body was ended, want it not")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResponseTimeRunsFromTheEndOfTheRequest(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
seen := &limitsSeen{}
|
||||||
|
|
||||||
|
limits := newTestLimits(seen)
|
||||||
|
limits.startRequestTimer()
|
||||||
|
|
||||||
|
sleepThenWait(time.Second)
|
||||||
|
|
||||||
|
end := time.Now()
|
||||||
|
|
||||||
|
limits.wroteRequest(httptrace.WroteRequestInfo{})
|
||||||
|
|
||||||
|
if got := limits.endRequest(); !got.Equal(end) {
|
||||||
|
t.Errorf("the request ended at %v, want %v", got, end)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Past SWW_RPROXY_MAX_REQUEST_TIME_MS, which the end of the request
|
||||||
|
// stopped, and just short of SWW_RPROXY_MAX_RESPONSE_TIME_MS.
|
||||||
|
sleepThenWait(testResponseTime - time.Millisecond)
|
||||||
|
|
||||||
|
if got := seen.refused.Load(); got != 0 {
|
||||||
|
t.Fatalf(
|
||||||
|
"refused %d before the limit ran out, want nothing",
|
||||||
|
got,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
sleepThenWait(time.Millisecond)
|
||||||
|
|
||||||
|
if got := seen.refused.Load(); got != http.StatusGatewayTimeout {
|
||||||
|
t.Errorf("refused %d, want 504", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestStoppedLimitsRefuseNothing(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
seen := &limitsSeen{}
|
||||||
|
seen.waiting.Store(true)
|
||||||
|
|
||||||
|
limits := newTestLimits(seen)
|
||||||
|
limits.startRequestTimer()
|
||||||
|
limits.stop()
|
||||||
|
limits.startResponseTimer(limits.endRequest())
|
||||||
|
|
||||||
|
sleepThenWait(testRequestTime + testResponseTime)
|
||||||
|
|
||||||
|
if got := seen.refused.Load(); got != 0 {
|
||||||
|
t.Errorf("refused %d, want nothing", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
if seen.readEnded.Load() {
|
||||||
|
t.Error("the read of the visitor's body was ended, want it not")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
+92
-92
@@ -1,17 +1,18 @@
|
|||||||
package rproxy
|
package rproxy
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"bytes"
|
|
||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"maps"
|
"maps"
|
||||||
"net/http"
|
"net/http"
|
||||||
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
|
"sync"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
|
||||||
)
|
)
|
||||||
|
|
||||||
// answerBodyMaxBytes is the most of the body of waf's answer that is read.
|
// answerBodyMaxBytes is the most of the body of waf's answer that is read.
|
||||||
@@ -40,15 +41,12 @@ type wafAnswer struct {
|
|||||||
// notes nothing, and no report is due for it: waf holds no decision to report
|
// notes nothing, and no report is due for it: waf holds no decision to report
|
||||||
// on.
|
// on.
|
||||||
func (rq *request) noteAnswer(answer *wafAnswer) {
|
func (rq *request) noteAnswer(answer *wafAnswer) {
|
||||||
disposition := answer.header.Get(waf.DispositionHeader)
|
read, valid := wafprotocol.ReadAnswer(answer.header)
|
||||||
|
if valid {
|
||||||
switch disposition {
|
rq.disposition = read.Disposition
|
||||||
case waf.DispositionAllow, waf.DispositionDeny,
|
|
||||||
waf.DispositionClose:
|
|
||||||
rq.disposition = disposition
|
|
||||||
rq.answered = true
|
rq.answered = true
|
||||||
rq.asn = answer.header.Get(asnHeader)
|
rq.asn = read.ASN
|
||||||
rq.country = answer.header.Get(countryHeader)
|
rq.country = read.Country
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -79,7 +77,7 @@ func (rq *request) handleNoUsableAnswer(ctx context.Context, err error) {
|
|||||||
rq.disposition = rq.h.config.WAFBackendNoAnswerDisposition
|
rq.disposition = rq.h.config.WAFBackendNoAnswerDisposition
|
||||||
rq.wafNoAnswer = true
|
rq.wafNoAnswer = true
|
||||||
|
|
||||||
if rq.disposition == waf.DispositionAllow {
|
if rq.disposition == wafprotocol.DispositionAllow {
|
||||||
rq.passToApp(ctx)
|
rq.passToApp(ctx)
|
||||||
} else {
|
} else {
|
||||||
rq.answer(http.StatusServiceUnavailable)
|
rq.answer(http.StatusServiceUnavailable)
|
||||||
@@ -119,7 +117,21 @@ func (rq *request) ask(ctx context.Context) (*wafAnswer, error) {
|
|||||||
ctx, cancel := context.WithTimeout(ctx, rq.h.config.WAFBackendTimeout)
|
ctx, cancel := context.WithTimeout(ctx, rq.h.config.WAFBackendTimeout)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
|
|
||||||
question, err := rq.newQuestion(ctx)
|
var held []byte
|
||||||
|
if rq.body != nil {
|
||||||
|
held = rq.body.held
|
||||||
|
}
|
||||||
|
|
||||||
|
question, err := wafprotocol.NewQuestion(ctx, wafprotocol.QuestionParams{
|
||||||
|
Visitor: rq.in,
|
||||||
|
PathAsSent: rules.Path(rq.in),
|
||||||
|
Body: held,
|
||||||
|
WAFBackend: rq.h.config.WAFBackend,
|
||||||
|
Client: rq.client,
|
||||||
|
Peer: rq.peer,
|
||||||
|
RequestID: rq.id,
|
||||||
|
Scheme: scheme(rq.in, rq.peerTrusted),
|
||||||
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
@@ -176,69 +188,24 @@ func canBeAsked(r *http.Request) bool {
|
|||||||
len(r.Header.Values("User-Agent")) <= 1
|
len(r.Header.Values("User-Agent")) <= 1
|
||||||
}
|
}
|
||||||
|
|
||||||
// newQuestion returns the question about the request, as "The question" in
|
// answerWith gives the visitor waf's answer, less its disposition. A deny of
|
||||||
// SPEC.md gives it: the visitor's request as the visitor sent it, with the part
|
// 403 or 429 gets rproxy's page for that status as its body, in place of
|
||||||
// of the body it holds and rproxy's headers.
|
// waf's.
|
||||||
func (rq *request) newQuestion(ctx context.Context) (*http.Request, error) {
|
|
||||||
in := rq.in
|
|
||||||
|
|
||||||
var held []byte
|
|
||||||
if rq.body != nil {
|
|
||||||
held = rq.body.held
|
|
||||||
}
|
|
||||||
|
|
||||||
question, err := http.NewRequestWithContext( //nolint:gosec // a setting's URL
|
|
||||||
ctx,
|
|
||||||
in.Method,
|
|
||||||
rq.h.config.WAFBackend.String(),
|
|
||||||
bytes.NewReader(held),
|
|
||||||
)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("make the question: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
backend := question.URL
|
|
||||||
target := *in.URL
|
|
||||||
target.Scheme = backend.Scheme
|
|
||||||
target.Host = backend.Host
|
|
||||||
setPathAsSent(&target, in)
|
|
||||||
question.URL = &target
|
|
||||||
question.Host = in.Host
|
|
||||||
|
|
||||||
// The length NewRequestWithContext set is that of what is held. This line makes
|
|
||||||
// a body whose length the visitor did not announce go chunked, as "The question"
|
|
||||||
// in SPEC.md requires.
|
|
||||||
question.TransferEncoding = in.TransferEncoding
|
|
||||||
|
|
||||||
question.Header = in.Header.Clone()
|
|
||||||
question.Header.Set(waf.ClientHeader, rq.client.String())
|
|
||||||
question.Header.Set(waf.PeerHeader, rq.peer.String())
|
|
||||||
question.Header.Del(waf.ContentLengthHeader)
|
|
||||||
|
|
||||||
announced := in.Header.Get("Content-Length")
|
|
||||||
if announced != "" {
|
|
||||||
question.Header.Set(waf.ContentLengthHeader, announced)
|
|
||||||
}
|
|
||||||
|
|
||||||
question.Header.Set(requestIDHeader, rq.id)
|
|
||||||
question.Header.Set("X-Forwarded-Proto", scheme(in, rq.peerTrusted))
|
|
||||||
|
|
||||||
// Go's client adds a User-Agent of its own to a request without the key.
|
|
||||||
_, sent := question.Header["User-Agent"]
|
|
||||||
if !sent {
|
|
||||||
question.Header["User-Agent"] = nil
|
|
||||||
}
|
|
||||||
|
|
||||||
return question, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// answerWith gives the visitor waf's answer, less its disposition.
|
|
||||||
func (rq *request) answerWith(answer *wafAnswer) {
|
func (rq *request) answerWith(answer *wafAnswer) {
|
||||||
rq.prepareOwnAnswer(rq.requestDeadline())
|
rq.prepareOwnAnswer(rq.limits.requestDeadline())
|
||||||
|
|
||||||
header := rq.out.Header()
|
header := rq.out.Header()
|
||||||
maps.Copy(header, answer.header)
|
maps.Copy(header, answer.header)
|
||||||
header.Del(waf.DispositionHeader)
|
header.Del(wafprotocol.DispositionHeader)
|
||||||
|
|
||||||
|
if rq.disposition == wafprotocol.DispositionDeny {
|
||||||
|
page, found := rq.h.pages.forStatus(answer.status)
|
||||||
|
if found {
|
||||||
|
rq.sendPage(answer.status, page)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// Go's server adds a Content-Type it guesses from the body to a response whose
|
// Go's server adds a Content-Type it guesses from the body to a response whose
|
||||||
// headers have no Content-Type key. The key with no value sends none.
|
// headers have no Content-Type key. The key with no value sends none.
|
||||||
@@ -249,18 +216,25 @@ func (rq *request) answerWith(answer *wafAnswer) {
|
|||||||
rq.send(answer.status, answer.body)
|
rq.send(answer.status, answer.body)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// sendPage sends page as the body of a refusal with status. The length is set
|
||||||
|
// here too, since send leaves a HEAD's as waf gave it, which is of waf's body.
|
||||||
|
func (rq *request) sendPage(status int, page []byte) {
|
||||||
|
header := rq.out.Header()
|
||||||
|
header.Set("Content-Type", htmlContentType)
|
||||||
|
header.Set("X-Content-Type-Options", "nosniff")
|
||||||
|
header.Set("Cache-Control", "no-store")
|
||||||
|
header.Set("Content-Length", strconv.Itoa(len(page)))
|
||||||
|
rq.send(status, page)
|
||||||
|
}
|
||||||
|
|
||||||
// sendReport sends waf report on a goroutine of its own, so that the
|
// sendReport sends waf report on a goroutine of its own, so that the
|
||||||
// visitor does not wait for it. The report goes on after ctx, the request's own
|
// visitor does not wait for it. The report goes on after ctx, the request's own
|
||||||
// context, has ended.
|
// context, has ended.
|
||||||
func (h *handler) sendReport(ctx context.Context, report *waf.Report) {
|
func (h *handler) sendReport(
|
||||||
h.reportsMu.Lock()
|
ctx context.Context,
|
||||||
defer h.reportsMu.Unlock()
|
report *wafprotocol.Report,
|
||||||
|
) {
|
||||||
if h.reportsStopped {
|
h.reports.startUnlessStopped(func() {
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
h.reports.Go(func() {
|
|
||||||
err := h.postReport(context.WithoutCancel(ctx), report)
|
err := h.postReport(context.WithoutCancel(ctx), report)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
h.processLog.Warn("a report is dropped", "error", err.Error())
|
h.processLog.Warn("a report is dropped", "error", err.Error())
|
||||||
@@ -268,25 +242,51 @@ func (h *handler) sendReport(ctx context.Context, report *waf.Report) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
// postReport posts report to waf, and waits for its answer for up to
|
// reportGoroutines are the goroutines sending reports to waf. None starts once
|
||||||
// SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS.
|
// stopAndWait has been called.
|
||||||
func (h *handler) postReport(ctx context.Context, report *waf.Report) error {
|
type reportGoroutines struct {
|
||||||
body, err := json.Marshal(report)
|
mu sync.Mutex
|
||||||
if err != nil {
|
stopped bool
|
||||||
return fmt.Errorf("encode the report: %w", err)
|
running sync.WaitGroup
|
||||||
|
}
|
||||||
|
|
||||||
|
// startUnlessStopped runs send on a goroutine of its own, or drops it once
|
||||||
|
// stopAndWait has been called.
|
||||||
|
func (g *reportGoroutines) startUnlessStopped(send func()) {
|
||||||
|
g.mu.Lock()
|
||||||
|
defer g.mu.Unlock()
|
||||||
|
|
||||||
|
if g.stopped {
|
||||||
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
g.running.Go(send)
|
||||||
|
}
|
||||||
|
|
||||||
|
// stopAndWait stops any more goroutines from starting, and returns once those
|
||||||
|
// running have ended.
|
||||||
|
func (g *reportGoroutines) stopAndWait() {
|
||||||
|
g.mu.Lock()
|
||||||
|
g.stopped = true
|
||||||
|
g.mu.Unlock()
|
||||||
|
|
||||||
|
g.running.Wait()
|
||||||
|
}
|
||||||
|
|
||||||
|
// postReport posts report to waf, and waits for its answer for up to
|
||||||
|
// SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS.
|
||||||
|
func (h *handler) postReport(
|
||||||
|
ctx context.Context,
|
||||||
|
report *wafprotocol.Report,
|
||||||
|
) error {
|
||||||
ctx, cancel := context.WithTimeout(ctx, h.config.WAFBackendTimeout)
|
ctx, cancel := context.WithTimeout(ctx, h.config.WAFBackendTimeout)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
|
|
||||||
req, err := http.NewRequestWithContext( //nolint:gosec // a setting's URL
|
req, err := wafprotocol.NewReportRequest(ctx, h.reportURL, report)
|
||||||
ctx, http.MethodPost, h.reportURL, bytes.NewReader(body))
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("make the request: %w", err)
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
req.Header.Set("Content-Type", "application/json")
|
|
||||||
|
|
||||||
res, err := h.wafTransport.RoundTrip(req)
|
res, err := h.wafTransport.RoundTrip(req)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("send the report: %w", err)
|
return fmt.Errorf("send the report: %w", err)
|
||||||
|
|||||||
+29
-55
@@ -18,34 +18,31 @@ import (
|
|||||||
"slices"
|
"slices"
|
||||||
"strings"
|
"strings"
|
||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/fsnotify/fsnotify"
|
|
||||||
"golang.org/x/net/http/httpguts"
|
"golang.org/x/net/http/httpguts"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/dirwatch"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// Action is what a rule does when it matches: ActionLog, ActionBlock or
|
||||||
|
// ActionBan.
|
||||||
|
type Action string
|
||||||
|
|
||||||
// The actions a rule takes when it matches.
|
// The actions a rule takes when it matches.
|
||||||
const (
|
const (
|
||||||
// ActionLog notes the match in the request log, and does nothing else.
|
// ActionLog notes the match in the request log, and does nothing else.
|
||||||
ActionLog = "log"
|
ActionLog Action = "log"
|
||||||
// ActionBlock refuses the request with 403.
|
// ActionBlock refuses the request with 403.
|
||||||
ActionBlock = "block"
|
ActionBlock Action = "block"
|
||||||
// ActionBan refuses the request and bans the client's netblock: the
|
// ActionBan refuses the request and bans the client's netblock: the
|
||||||
// request is a clear sign of attack.
|
// request is a clear sign of attack.
|
||||||
ActionBan = "ban"
|
ActionBan Action = "ban"
|
||||||
)
|
)
|
||||||
|
|
||||||
// extension ends the name of every rule file.
|
// extension ends the name of every rule file.
|
||||||
const extension = ".rules"
|
const extension = ".rules"
|
||||||
|
|
||||||
// quietTime is how long SWW_WAF_RULE_FILES_DIR must go without a change before
|
|
||||||
// the rule files are read again, so that a file still being written, such
|
|
||||||
// as one saved in place, appended to or copied in with scp, is read only
|
|
||||||
// once whole.
|
|
||||||
const quietTime = 2 * time.Second
|
|
||||||
|
|
||||||
// headerTarget starts the target that is one request header,
|
// headerTarget starts the target that is one request header,
|
||||||
// header:<Name>.
|
// header:<Name>.
|
||||||
const headerTarget = "header:"
|
const headerTarget = "header:"
|
||||||
@@ -95,8 +92,7 @@ type Rule struct {
|
|||||||
// Target is what the regex is matched against, such as path or
|
// Target is what the regex is matched against, such as path or
|
||||||
// header:Accept.
|
// header:Accept.
|
||||||
Target string
|
Target string
|
||||||
// Action is ActionLog, ActionBlock or ActionBan.
|
Action Action
|
||||||
Action string
|
|
||||||
|
|
||||||
regex *regexp.Regexp
|
regex *regexp.Regexp
|
||||||
}
|
}
|
||||||
@@ -176,26 +172,18 @@ func (f *Files) Len() int {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Watch watches Dir until ctx is done, and reads the rule files again
|
// Watch watches Dir until ctx is done, and reads the rule files again
|
||||||
// once Dir has had no change for quietTime, after one is edited, added or
|
// once Dir has had no change for a short time, after one is edited, added
|
||||||
// removed, and after Watch starts watching. If they then hold an error,
|
// or removed, and after Watch starts watching, as dirwatch describes. If
|
||||||
// the rules stay as they were, the error is logged with its file and
|
// they then hold an error, the rules stay as they were, the error is
|
||||||
// line, and the files are read again after the next change. If Dir cannot
|
// logged with its file and line, and the files are read again after the
|
||||||
// be watched, that is logged, and the rules stay as they were loaded.
|
// next change. If Dir cannot be watched, that is logged, and the rules
|
||||||
// While Enabled is false, Watch returns at once.
|
// stay as they were loaded. While Enabled is false, Watch returns at once.
|
||||||
func (f *Files) Watch(ctx context.Context) {
|
func (f *Files) Watch(ctx context.Context) {
|
||||||
if !f.params.Enabled {
|
if !f.params.Enabled {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
watcher, err := fsnotify.NewWatcher()
|
watcher, err := dirwatch.New(f.params.Dir)
|
||||||
if err == nil {
|
|
||||||
defer func() {
|
|
||||||
_ = watcher.Close()
|
|
||||||
}()
|
|
||||||
|
|
||||||
err = watcher.Add(f.params.Dir)
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
f.params.ProcessLog.Error("cannot watch the rule files for edits",
|
f.params.ProcessLog.Error("cannot watch the rule files for edits",
|
||||||
"error", err.Error())
|
"error", err.Error())
|
||||||
@@ -203,36 +191,22 @@ func (f *Files) Watch(ctx context.Context) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
defer func() {
|
||||||
|
_ = watcher.Close()
|
||||||
|
}()
|
||||||
|
|
||||||
f.params.ProcessLog.Info("watching the rule files for edits",
|
f.params.ProcessLog.Info("watching the rule files for edits",
|
||||||
"directory", f.params.Dir)
|
"directory", f.params.Dir)
|
||||||
|
|
||||||
f.readAfterChanges(ctx, watcher.Events, watcher.Errors)
|
// A change to any file in Dir counts.
|
||||||
}
|
watcher.CallWhenQuiet(ctx,
|
||||||
|
func(string) bool { return true },
|
||||||
// readAfterChanges reads the rule files again once quietTime has passed
|
f.readAgain,
|
||||||
// without a change from events, until ctx is done, and logs the errors
|
func(err error) {
|
||||||
// from errs. The wait starts at once, as if for a change, so that an edit
|
|
||||||
// saved after Load read the files, and before Dir was watched, is taken
|
|
||||||
// in too.
|
|
||||||
func (f *Files) readAfterChanges(
|
|
||||||
ctx context.Context, events <-chan fsnotify.Event, errs <-chan error,
|
|
||||||
) {
|
|
||||||
quiet := time.NewTimer(quietTime)
|
|
||||||
defer quiet.Stop()
|
|
||||||
|
|
||||||
for {
|
|
||||||
select {
|
|
||||||
case <-ctx.Done():
|
|
||||||
return
|
|
||||||
case <-events:
|
|
||||||
quiet.Reset(quietTime)
|
|
||||||
case <-quiet.C:
|
|
||||||
f.readAgain()
|
|
||||||
case err := <-errs:
|
|
||||||
f.params.ProcessLog.Warn("watching the rule files failed",
|
f.params.ProcessLog.Warn("watching the rule files failed",
|
||||||
"error", err.Error())
|
"error", err.Error())
|
||||||
}
|
},
|
||||||
}
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
// readAgain reads the rule files again, in place of the rules loaded, or
|
// readAgain reads the rule files again, in place of the rules loaded, or
|
||||||
@@ -366,7 +340,7 @@ func parse(line string) (Rule, bool, error) {
|
|||||||
return Rule{}, false, errNotRule
|
return Rule{}, false, errNotRule
|
||||||
}
|
}
|
||||||
|
|
||||||
rule := Rule{ID: fields[1], Target: fields[2], Action: fields[3]}
|
rule := Rule{ID: fields[1], Target: fields[2], Action: Action(fields[3])}
|
||||||
headerName, isHeader := strings.CutPrefix(rule.Target, headerTarget)
|
headerName, isHeader := strings.CutPrefix(rule.Target, headerTarget)
|
||||||
|
|
||||||
switch {
|
switch {
|
||||||
@@ -394,7 +368,7 @@ func parse(line string) (Rule, bool, error) {
|
|||||||
rule.Target,
|
rule.Target,
|
||||||
errHeaderTakenOut,
|
errHeaderTakenOut,
|
||||||
)
|
)
|
||||||
case !slices.Contains([]string{ActionLog, ActionBlock, ActionBan}, rule.Action):
|
case !slices.Contains([]Action{ActionLog, ActionBlock, ActionBan}, rule.Action):
|
||||||
return Rule{}, false, fmt.Errorf(
|
return Rule{}, false, fmt.Errorf(
|
||||||
"the action %q %w",
|
"the action %q %w",
|
||||||
rule.Action,
|
rule.Action,
|
||||||
|
|||||||
@@ -1,165 +0,0 @@
|
|||||||
package rules
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"log/slog"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"slices"
|
|
||||||
"testing"
|
|
||||||
"testing/synctest"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/fsnotify/fsnotify"
|
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The tests below run readAfterChanges in a synctest bubble, where time is
|
|
||||||
// a clock of the test's own: time.Sleep moves it on at once, and
|
|
||||||
// synctest.Wait returns once readAfterChanges waits again, so that every
|
|
||||||
// reading due by then is done. The test sends the changes itself, as the
|
|
||||||
// watch of a directory cannot run in a bubble.
|
|
||||||
|
|
||||||
func TestFileWrittenInTwoPartsTakenInOnlyWhole(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
dir := t.TempDir()
|
|
||||||
path := filepath.Join(dir, "50-app.rules")
|
|
||||||
writeFile(t, path, "first path block ^/first\n")
|
|
||||||
files := load(t, dir)
|
|
||||||
changes := run(t, files)
|
|
||||||
|
|
||||||
file, err := os.Create(path) //nolint:gosec // a file the test wrote
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("create: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() {
|
|
||||||
_ = file.Close()
|
|
||||||
}()
|
|
||||||
|
|
||||||
// The first part ends in the middle of a ban rule's regex, which,
|
|
||||||
// read then, would ban every request.
|
|
||||||
write(t, file, "first path block ^/first\nprobe path ban ^/")
|
|
||||||
|
|
||||||
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
|
|
||||||
|
|
||||||
time.Sleep(quietTime - time.Nanosecond)
|
|
||||||
synctest.Wait()
|
|
||||||
wantMatched(t, files, "/anything")
|
|
||||||
|
|
||||||
// The second part starts the wait again.
|
|
||||||
write(t, file, `\.env$`+"\n")
|
|
||||||
|
|
||||||
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
|
|
||||||
|
|
||||||
time.Sleep(quietTime - time.Nanosecond)
|
|
||||||
synctest.Wait()
|
|
||||||
wantMatched(t, files, "/.env")
|
|
||||||
|
|
||||||
time.Sleep(time.Nanosecond)
|
|
||||||
synctest.Wait()
|
|
||||||
wantMatched(t, files, "/.env", "probe")
|
|
||||||
wantMatched(t, files, "/anything")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestEditSavedBeforeTheWatchStartsTakenIn(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
dir := t.TempDir()
|
|
||||||
path := filepath.Join(dir, "50-app.rules")
|
|
||||||
writeFile(t, path, "first path block ^/first\n")
|
|
||||||
files := load(t, dir)
|
|
||||||
|
|
||||||
// Saved after Load read the files, and before the directory was
|
|
||||||
// watched, so that no change is seen for it.
|
|
||||||
writeFile(t, path, "first path block ^/edited\n")
|
|
||||||
run(t, files)
|
|
||||||
time.Sleep(quietTime)
|
|
||||||
synctest.Wait()
|
|
||||||
wantMatched(t, files, "/edited", "first")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// load loads the rules in dir.
|
|
||||||
func load(t *testing.T, dir string) *Files {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
files, err := Load(Params{
|
|
||||||
Dir: dir, Enabled: true, ProcessLog: slog.New(slog.DiscardHandler),
|
|
||||||
Alerts: alerts.New(alerts.Params{}),
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("load: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return files
|
|
||||||
}
|
|
||||||
|
|
||||||
// run runs files' readAfterChanges until the test ends, and returns the
|
|
||||||
// channel that sends it changes.
|
|
||||||
func run(t *testing.T, files *Files) chan<- fsnotify.Event {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
changes := make(chan fsnotify.Event)
|
|
||||||
ctx, stop := context.WithCancel(t.Context())
|
|
||||||
stopped := make(chan struct{})
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
files.readAfterChanges(ctx, changes, nil)
|
|
||||||
close(stopped)
|
|
||||||
}()
|
|
||||||
|
|
||||||
t.Cleanup(func() {
|
|
||||||
stop()
|
|
||||||
<-stopped
|
|
||||||
})
|
|
||||||
|
|
||||||
return changes
|
|
||||||
}
|
|
||||||
|
|
||||||
// writeFile writes content to the file at path.
|
|
||||||
func writeFile(t *testing.T, path, content string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
err := os.WriteFile(path, []byte(content), 0o600)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("write %s: %v", path, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// write writes text to the end of file.
|
|
||||||
func write(t *testing.T, file *os.File, text string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
_, err := file.WriteString(text)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("write: %v", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// wantMatched checks the ids of the rules that a GET request for path
|
|
||||||
// matches, in order.
|
|
||||||
func wantMatched(t *testing.T, files *Files, path string, want ...string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
r := httptest.NewRequestWithContext(t.Context(), http.MethodGet,
|
|
||||||
"http://app.example"+path, nil)
|
|
||||||
|
|
||||||
matched := files.Match(r)
|
|
||||||
|
|
||||||
got := make([]string, 0, len(matched))
|
|
||||||
for _, rule := range matched {
|
|
||||||
got = append(got, rule.ID)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !slices.Equal(got, want) {
|
|
||||||
t.Errorf("%s matched %v, want %v", path, got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -181,7 +181,7 @@ type BanEntry struct {
|
|||||||
Netblock netip.Prefix `json:"netblock"`
|
Netblock netip.Prefix `json:"netblock"`
|
||||||
Start time.Time `json:"start"`
|
Start time.Time `json:"start"`
|
||||||
Expires *time.Time `json:"expires"`
|
Expires *time.Time `json:"expires"`
|
||||||
Cause string `json:"cause"`
|
Cause bans.Cause `json:"cause"`
|
||||||
Reason string `json:"reason,omitempty"`
|
Reason string `json:"reason,omitempty"`
|
||||||
Lifted *time.Time `json:"lifted,omitempty"`
|
Lifted *time.Time `json:"lifted,omitempty"`
|
||||||
Notes bans.Notes `json:"notes"`
|
Notes bans.Notes `json:"notes"`
|
||||||
@@ -240,7 +240,7 @@ type alertsFile struct {
|
|||||||
Version int `json:"version"`
|
Version int `json:"version"`
|
||||||
Cooldowns []alerts.Cooldown `json:"cooldowns"`
|
Cooldowns []alerts.Cooldown `json:"cooldowns"`
|
||||||
Hour alerts.Hour `json:"hour"`
|
Hour alerts.Hour `json:"hour"`
|
||||||
Waiting map[string][]alerts.Alert `json:"waiting"`
|
Waiting map[alerts.DestinationName][]alerts.Alert `json:"waiting"`
|
||||||
AnomalyCounters []anomaly.Counter `json:"anomaly_counters"`
|
AnomalyCounters []anomaly.Counter `json:"anomaly_counters"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -1758,8 +1758,8 @@ func TestEditOfRouteWatchAndAlertsFilesTakenIn(t *testing.T) {
|
|||||||
Event: alerts.EventBan, Netblock: netip.MustParsePrefix("198.51.100.0/24"),
|
Event: alerts.EventBan, Netblock: netip.MustParsePrefix("198.51.100.0/24"),
|
||||||
Sent: midnight(),
|
Sent: midnight(),
|
||||||
}},
|
}},
|
||||||
Hour: alerts.Hour{HeldBack: map[string]int{}},
|
Hour: alerts.Hour{HeldBack: map[alerts.Event]int{}},
|
||||||
Waiting: map[string][]alerts.Alert{
|
Waiting: map[alerts.DestinationName][]alerts.Alert{
|
||||||
alerts.DestinationWebhook: {
|
alerts.DestinationWebhook: {
|
||||||
{Event: alerts.EventFileError, Time: midnight()},
|
{Event: alerts.EventFileError, Time: midnight()},
|
||||||
},
|
},
|
||||||
@@ -2046,7 +2046,7 @@ func TestBanWithoutACauseTakenInAsAnAdmins(t *testing.T) {
|
|||||||
|
|
||||||
var written struct {
|
var written struct {
|
||||||
Bans []struct {
|
Bans []struct {
|
||||||
Cause string `json:"cause"`
|
Cause bans.Cause `json:"cause"`
|
||||||
Reason string `json:"reason"`
|
Reason string `json:"reason"`
|
||||||
} `json:"bans"`
|
} `json:"bans"`
|
||||||
}
|
}
|
||||||
@@ -2430,7 +2430,7 @@ func midnight() time.Time {
|
|||||||
// counts, with thresholds fill does not reach.
|
// counts, with thresholds fill does not reach.
|
||||||
func newParams(dir string) state.Params {
|
func newParams(dir string) state.Params {
|
||||||
discard := slog.New(slog.DiscardHandler)
|
discard := slog.New(slog.DiscardHandler)
|
||||||
m := metrics.New(1, "app")
|
m := metrics.New(metrics.Params{BusiestCount: 1, InstanceName: "app"})
|
||||||
queue := alerts.New(alerts.Params{
|
queue := alerts.New(alerts.Params{
|
||||||
WebhookURL: &url.URL{Scheme: "https", Host: "alerts.example"},
|
WebhookURL: &url.URL{Scheme: "https", Host: "alerts.example"},
|
||||||
Events: alerts.Events(),
|
Events: alerts.Events(),
|
||||||
|
|||||||
@@ -13,7 +13,7 @@ import (
|
|||||||
"github.com/spf13/viper"
|
"github.com/spf13/viper"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
|
||||||
)
|
)
|
||||||
|
|
||||||
// healthCheckTimeout bounds the whole health check.
|
// healthCheckTimeout bounds the whole health check.
|
||||||
@@ -116,7 +116,7 @@ func askHealthEndpoint(ctx context.Context, listenAddr string) error {
|
|||||||
health := "http://" + net.JoinHostPort(
|
health := "http://" + net.JoinHostPort(
|
||||||
"127.0.0.1",
|
"127.0.0.1",
|
||||||
port,
|
port,
|
||||||
) + waf.HealthPath
|
) + wafprotocol.HealthPath
|
||||||
|
|
||||||
req, err := http.NewRequestWithContext(
|
req, err := http.NewRequestWithContext(
|
||||||
ctx,
|
ctx,
|
||||||
|
|||||||
@@ -95,12 +95,20 @@ func background(loop func(context.Context)) fx.Hook {
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// daemonServer is the server a daemon runs, waf.Server or rproxy.Server: the
|
||||||
|
// methods of Go's http.Server that serve starts and stops it with.
|
||||||
|
type daemonServer interface {
|
||||||
|
Serve(listener net.Listener) error
|
||||||
|
Shutdown(ctx context.Context) error
|
||||||
|
Close() error
|
||||||
|
}
|
||||||
|
|
||||||
// serve serves requests on listener from the start. At the stop it gives the
|
// serve serves requests on listener from the start. At the stop it gives the
|
||||||
// requests in progress shutdownTimeout to finish, then closes their
|
// requests in progress shutdownTimeout to finish, then closes their
|
||||||
// connections. When serving fails, it logs why and sends the error to
|
// connections. When serving fails, it logs why and sends the error to
|
||||||
// failed.
|
// failed.
|
||||||
func serve(
|
func serve(
|
||||||
lc fx.Lifecycle, server *http.Server, listener net.Listener,
|
lc fx.Lifecycle, server daemonServer, listener net.Listener,
|
||||||
processLog *slog.Logger, failed servingFailed,
|
processLog *slog.Logger, failed servingFailed,
|
||||||
) {
|
) {
|
||||||
served := make(chan error, 1)
|
served := make(chan error, 1)
|
||||||
|
|||||||
@@ -4,7 +4,6 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"net"
|
"net"
|
||||||
"net/http"
|
|
||||||
"os"
|
"os"
|
||||||
|
|
||||||
"go.uber.org/fx"
|
"go.uber.org/fx"
|
||||||
@@ -44,8 +43,13 @@ func RunRProxy(ctx context.Context, params Params) int {
|
|||||||
app := fx.New(
|
app := fx.New(
|
||||||
fx.Supply(cfg, processLog, requestLog{params.Stdout}, failed),
|
fx.Supply(cfg, processLog, requestLog{params.Stdout}, failed),
|
||||||
fx.Provide(
|
fx.Provide(
|
||||||
|
// run waits for rproxy.Server's reports, and serve takes it as a
|
||||||
|
// daemonServer.
|
||||||
|
fx.Annotate(
|
||||||
newRProxyServer,
|
newRProxyServer,
|
||||||
func(server *rproxy.Server) *http.Server { return server.Server },
|
fx.As(fx.Self()),
|
||||||
|
fx.As(new(daemonServer)),
|
||||||
|
),
|
||||||
newRProxyListener,
|
newRProxyListener,
|
||||||
),
|
),
|
||||||
fx.Invoke(serve),
|
fx.Invoke(serve),
|
||||||
|
|||||||
+15
-15
@@ -21,7 +21,7 @@ import (
|
|||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
|
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/sww"
|
"sneak.berlin/go/smallwebwaf/internal/sww"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -164,7 +164,7 @@ func TestABanThroughOneSiteRefusesTheClientOnTheOther(t *testing.T) {
|
|||||||
|
|
||||||
// One request's three lines carry one id, rproxy's with waf's answer.
|
// One request's three lines carry one id, rproxy's with waf's answer.
|
||||||
line := firstOut.line(t, "path", bannedProbe)
|
line := firstOut.line(t, "path", bannedProbe)
|
||||||
if line["disposition"] != waf.DispositionDeny ||
|
if line["disposition"] != string(wafprotocol.DispositionDeny) ||
|
||||||
line["client_ip"] != placed {
|
line["client_ip"] != placed {
|
||||||
t.Errorf("rproxy's line is %v", line)
|
t.Errorf("rproxy's line is %v", line)
|
||||||
}
|
}
|
||||||
@@ -249,7 +249,7 @@ func TestTheSiteAnswersWhileWAFIsStoppedAndStartedAgain(t *testing.T) {
|
|||||||
wantStatus(t, site+"/after-stop", placed, http.StatusOK)
|
wantStatus(t, site+"/after-stop", placed, http.StatusOK)
|
||||||
|
|
||||||
line := out.line(t, "path", "/after-stop")
|
line := out.line(t, "path", "/after-stop")
|
||||||
if line["disposition"] != waf.DispositionAllow ||
|
if line["disposition"] != string(wafprotocol.DispositionAllow) ||
|
||||||
line["waf_backend_no_answer"] != true {
|
line["waf_backend_no_answer"] != true {
|
||||||
t.Errorf("rproxy's line is %v", line)
|
t.Errorf("rproxy's line is %v", line)
|
||||||
}
|
}
|
||||||
@@ -437,7 +437,7 @@ func TestAChunkedBodyStaysABodyAtWAF(t *testing.T) {
|
|||||||
const smuggledID = "smuggled-request"
|
const smuggledID = "smuggled-request"
|
||||||
|
|
||||||
smuggled := "GET " + bannedProbe + " HTTP/1.1\r\nHost: site.example\r\n" +
|
smuggled := "GET " + bannedProbe + " HTTP/1.1\r\nHost: site.example\r\n" +
|
||||||
waf.ClientHeader + ": " + otherClient + "\r\n" +
|
wafprotocol.ClientHeader + ": " + otherClient + "\r\n" +
|
||||||
"X-Request-ID: " + smuggledID + "\r\n\r\n"
|
"X-Request-ID: " + smuggledID + "\r\n\r\n"
|
||||||
|
|
||||||
status := sendRaw(
|
status := sendRaw(
|
||||||
@@ -605,10 +605,11 @@ func TestRProxyAsksAWAFThatSpeaksHTTP2OverHTTP1(t *testing.T) {
|
|||||||
protocols := make(chan string, 10)
|
protocols := make(chan string, 10)
|
||||||
backend := httptest.NewUnstartedServer(http.HandlerFunc(
|
backend := httptest.NewUnstartedServer(http.HandlerFunc(
|
||||||
func(w http.ResponseWriter, r *http.Request) {
|
func(w http.ResponseWriter, r *http.Request) {
|
||||||
if r.Header.Get(waf.ClientHeader) != "" {
|
if r.Header.Get(wafprotocol.ClientHeader) != "" {
|
||||||
protocols <- r.Proto
|
protocols <- r.Proto
|
||||||
|
|
||||||
w.Header().Set(waf.DispositionHeader, waf.DispositionDeny)
|
w.Header().Set(wafprotocol.DispositionHeader,
|
||||||
|
string(wafprotocol.DispositionDeny))
|
||||||
w.WriteHeader(http.StatusForbidden)
|
w.WriteHeader(http.StatusForbidden)
|
||||||
}
|
}
|
||||||
}))
|
}))
|
||||||
@@ -690,8 +691,9 @@ func TestRProxyStoppingWaitsForAReportBeingSent(t *testing.T) {
|
|||||||
release := make(chan struct{})
|
release := make(chan struct{})
|
||||||
backend := httptest.NewServer(http.HandlerFunc(
|
backend := httptest.NewServer(http.HandlerFunc(
|
||||||
func(w http.ResponseWriter, r *http.Request) {
|
func(w http.ResponseWriter, r *http.Request) {
|
||||||
if r.Header.Get(waf.ClientHeader) != "" {
|
if r.Header.Get(wafprotocol.ClientHeader) != "" {
|
||||||
w.Header().Set(waf.DispositionHeader, waf.DispositionAllow)
|
w.Header().Set(wafprotocol.DispositionHeader,
|
||||||
|
string(wafprotocol.DispositionAllow))
|
||||||
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -779,10 +781,7 @@ func sendAndKeepOpen(t *testing.T, addr, request string) net.Conn {
|
|||||||
func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
|
func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
const (
|
const cutOffAfter = 5 * time.Second
|
||||||
cutOffAfter = 5 * time.Second
|
|
||||||
upstreamSetting = "SWW_RPROXY_UPSTREAM_URL"
|
|
||||||
)
|
|
||||||
|
|
||||||
asked := make(chan struct{})
|
asked := make(chan struct{})
|
||||||
release := make(chan struct{})
|
release := make(chan struct{})
|
||||||
@@ -796,7 +795,8 @@ func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
|
|||||||
|
|
||||||
backend := httptest.NewServer(http.HandlerFunc(
|
backend := httptest.NewServer(http.HandlerFunc(
|
||||||
func(w http.ResponseWriter, _ *http.Request) {
|
func(w http.ResponseWriter, _ *http.Request) {
|
||||||
w.Header().Set(waf.DispositionHeader, waf.DispositionAllow)
|
w.Header().Set(wafprotocol.DispositionHeader,
|
||||||
|
string(wafprotocol.DispositionAllow))
|
||||||
}))
|
}))
|
||||||
t.Cleanup(backend.Close)
|
t.Cleanup(backend.Close)
|
||||||
|
|
||||||
@@ -805,7 +805,7 @@ func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
|
|||||||
|
|
||||||
env := map[string]string{
|
env := map[string]string{
|
||||||
rproxyListenAddr: localhost + ":0",
|
rproxyListenAddr: localhost + ":0",
|
||||||
upstreamSetting: app.URL,
|
rproxyUpstreamURL: app.URL,
|
||||||
wafBackend: backend.URL,
|
wafBackend: backend.URL,
|
||||||
}
|
}
|
||||||
out := &output{}
|
out := &output{}
|
||||||
@@ -932,7 +932,7 @@ func TestMainRunsRProxy(t *testing.T) {
|
|||||||
addr, stop := startRProxyProcess(t, wafBackend+"=http://127.0.0.1:9")
|
addr, stop := startRProxyProcess(t, wafBackend+"=http://127.0.0.1:9")
|
||||||
|
|
||||||
req, err := http.NewRequestWithContext(t.Context(), http.MethodGet,
|
req, err := http.NewRequestWithContext(t.Context(), http.MethodGet,
|
||||||
"http://"+addr+waf.HealthPath, http.NoBody)
|
"http://"+addr+wafprotocol.HealthPath, http.NoBody)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("new request: %v", err)
|
t.Fatalf("new request: %v", err)
|
||||||
}
|
}
|
||||||
|
|||||||
+51
-29
@@ -10,13 +10,14 @@ package sww
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"fmt"
|
"errors"
|
||||||
"io"
|
"io"
|
||||||
"os"
|
"os"
|
||||||
"os/signal"
|
"os/signal"
|
||||||
"syscall"
|
"syscall"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"github.com/spf13/cobra"
|
||||||
"github.com/spf13/viper"
|
"github.com/spf13/viper"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
@@ -26,14 +27,7 @@ import (
|
|||||||
// know.
|
// know.
|
||||||
const usageStatus = 2
|
const usageStatus = 2
|
||||||
|
|
||||||
// usage is what sww prints then.
|
var errNoCommand = errors.New("no command given")
|
||||||
const usage = `usage: sww <command>
|
|
||||||
|
|
||||||
commands:
|
|
||||||
rproxy the reverse proxy in front of one app
|
|
||||||
waf decides for rproxy, and keeps bans, counters and history
|
|
||||||
healthcheck the container's health check
|
|
||||||
`
|
|
||||||
|
|
||||||
// Params are what RunWAF and RunRProxy need from the process.
|
// Params are what RunWAF and RunRProxy need from the process.
|
||||||
type Params struct {
|
type Params struct {
|
||||||
@@ -51,14 +45,9 @@ type Params struct {
|
|||||||
|
|
||||||
// Main runs the command os.Args names, and returns the process's exit status.
|
// Main runs the command os.Args names, and returns the process's exit status.
|
||||||
// `sww rproxy` and `sww waf` run until SIGTERM or SIGINT.
|
// `sww rproxy` and `sww waf` run until SIGTERM or SIGINT.
|
||||||
// With no command, or one it does not know, it prints the commands and returns
|
// With no command, or one it does not know, cobra prints the error and the
|
||||||
// usageStatus.
|
// commands to stderr, and Main returns usageStatus.
|
||||||
func Main(version string) int {
|
func Main(version string) int {
|
||||||
args := os.Args[1:]
|
|
||||||
if len(args) == 0 {
|
|
||||||
return printUsage(os.Stderr)
|
|
||||||
}
|
|
||||||
|
|
||||||
ctx, stop := signal.NotifyContext(context.Background(),
|
ctx, stop := signal.NotifyContext(context.Background(),
|
||||||
syscall.SIGTERM, os.Interrupt)
|
syscall.SIGTERM, os.Interrupt)
|
||||||
defer stop()
|
defer stop()
|
||||||
@@ -69,21 +58,54 @@ func Main(version string) int {
|
|||||||
Stdout: os.Stdout,
|
Stdout: os.Stdout,
|
||||||
}
|
}
|
||||||
|
|
||||||
switch args[0] {
|
// status is what the command that ran returns.
|
||||||
case "rproxy":
|
status := 0
|
||||||
return RunRProxy(ctx, params)
|
|
||||||
case "waf":
|
root := &cobra.Command{
|
||||||
return RunWAF(ctx, params)
|
Use: "sww",
|
||||||
case "healthcheck":
|
Short: "a web application firewall",
|
||||||
return HealthCheck(ctx, args[1:], params.Viper, os.Stderr)
|
Args: cobra.NoArgs,
|
||||||
default:
|
RunE: func(*cobra.Command, []string) error {
|
||||||
return printUsage(os.Stderr)
|
return errNoCommand
|
||||||
|
},
|
||||||
}
|
}
|
||||||
}
|
root.CompletionOptions.DisableDefaultCmd = true
|
||||||
|
|
||||||
// printUsage writes the commands to w, and returns usageStatus.
|
root.AddCommand(
|
||||||
func printUsage(w io.Writer) int {
|
&cobra.Command{
|
||||||
_, _ = fmt.Fprint(w, usage)
|
Use: "rproxy",
|
||||||
|
Short: "the reverse proxy in front of one app",
|
||||||
|
Run: func(cmd *cobra.Command, _ []string) {
|
||||||
|
status = RunRProxy(cmd.Context(), params)
|
||||||
|
},
|
||||||
|
},
|
||||||
|
&cobra.Command{
|
||||||
|
Use: "waf",
|
||||||
|
Short: "decides for rproxy, and keeps bans, counters and history",
|
||||||
|
Run: func(cmd *cobra.Command, _ []string) {
|
||||||
|
status = RunWAF(cmd.Context(), params)
|
||||||
|
},
|
||||||
|
},
|
||||||
|
&cobra.Command{
|
||||||
|
Use: "healthcheck",
|
||||||
|
Short: "the container's health check",
|
||||||
|
// HealthCheck refuses every argument itself, flags included.
|
||||||
|
DisableFlagParsing: true,
|
||||||
|
Run: func(cmd *cobra.Command, args []string) {
|
||||||
|
status = HealthCheck(
|
||||||
|
cmd.Context(),
|
||||||
|
args,
|
||||||
|
params.Viper,
|
||||||
|
cmd.ErrOrStderr(),
|
||||||
|
)
|
||||||
|
},
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
err := root.ExecuteContext(ctx)
|
||||||
|
if err != nil {
|
||||||
return usageStatus
|
return usageStatus
|
||||||
|
}
|
||||||
|
|
||||||
|
return status
|
||||||
}
|
}
|
||||||
@@ -23,7 +23,7 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
|
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
|
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/sww"
|
"sneak.berlin/go/smallwebwaf/internal/sww"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -1813,7 +1813,7 @@ func statusFrom(t *testing.T, url, from string) int {
|
|||||||
}
|
}
|
||||||
|
|
||||||
req.Header.Set("X-Forwarded-For", from)
|
req.Header.Set("X-Forwarded-For", from)
|
||||||
req.Header.Set(waf.ClientHeader, from)
|
req.Header.Set(wafprotocol.ClientHeader, from)
|
||||||
req.Header.Set("X-Request-ID", rand.Text())
|
req.Header.Set("X-Request-ID", rand.Text())
|
||||||
|
|
||||||
transport := &http.Transport{}
|
transport := &http.Transport{}
|
||||||
|
|||||||
+20
-14
@@ -6,7 +6,6 @@ import (
|
|||||||
"io"
|
"io"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"net"
|
"net"
|
||||||
"net/http"
|
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"go.uber.org/fx"
|
"go.uber.org/fx"
|
||||||
@@ -97,8 +96,8 @@ func RunWAF(ctx context.Context, params Params) int {
|
|||||||
newAlertQueue, newRuleFiles, newCoreRuleSet, newLookupFile,
|
newAlertQueue, newRuleFiles, newCoreRuleSet, newLookupFile,
|
||||||
newLimiter, newLedger, newAnomalyCounters,
|
newLimiter, newLedger, newAnomalyCounters,
|
||||||
newLists, newDNSBL, newAbuseIPDB, newRouteWatch,
|
newLists, newDNSBL, newAbuseIPDB, newRouteWatch,
|
||||||
newMetrics, newGeoJS, newWAFServer,
|
newMetrics, newGeoJS,
|
||||||
func(server *waf.Server) *http.Server { return server.Server },
|
fx.Annotate(newWAFServer, fx.As(new(daemonServer))),
|
||||||
newWAFListener, newStateFiles,
|
newWAFListener, newStateFiles,
|
||||||
),
|
),
|
||||||
// fx stops hooks in the reverse of the order they were added in. The
|
// fx stops hooks in the reverse of the order they were added in. The
|
||||||
@@ -392,26 +391,33 @@ func newMetrics(
|
|||||||
alertQueue *alerts.Queue, lookupFile *lookup.File,
|
alertQueue *alerts.Queue, lookupFile *lookup.File,
|
||||||
remote *remotelog.Sender,
|
remote *remotelog.Sender,
|
||||||
) *metrics.Metrics {
|
) *metrics.Metrics {
|
||||||
m := metrics.New(cfg.MetricsBusiestCountriesAndASNs, cfg.InstanceName)
|
params := metrics.Params{
|
||||||
m.AddBansAndClients(ledger, limiter, utcNow)
|
BusiestCount: cfg.MetricsBusiestCountriesAndASNs,
|
||||||
m.AddRules(ruleFiles)
|
InstanceName: cfg.InstanceName,
|
||||||
m.AddReputation(lists, dnsbl)
|
Now: utcNow,
|
||||||
|
Bans: ledger,
|
||||||
if cfg.AbuseIPDBKey != "" {
|
Clients: limiter,
|
||||||
m.AddAbuseIPDB(abuseIPDB)
|
Rules: ruleFiles,
|
||||||
|
Lists: lists,
|
||||||
|
DNSBL: dnsbl,
|
||||||
|
Alerts: alertQueue,
|
||||||
}
|
}
|
||||||
|
|
||||||
m.AddAlerts(alertQueue)
|
// A nil pointer in an interface field is not a nil interface, so a part
|
||||||
|
// that is off is left out rather than set to nil.
|
||||||
|
if cfg.AbuseIPDBKey != "" {
|
||||||
|
params.AbuseIPDB = abuseIPDB
|
||||||
|
}
|
||||||
|
|
||||||
if lookupFile != nil {
|
if lookupFile != nil {
|
||||||
m.AddLookupFile(lookupFile.LastRead, lookupFile.ReadFailures)
|
params.LookupFile = lookupFile
|
||||||
}
|
}
|
||||||
|
|
||||||
if remote != nil {
|
if remote != nil {
|
||||||
m.AddRemoteLog(remote)
|
params.RemoteLog = remote
|
||||||
}
|
}
|
||||||
|
|
||||||
return m
|
return metrics.New(params)
|
||||||
}
|
}
|
||||||
|
|
||||||
// newGeoJS returns what looks clients up at GeoJS, none looked up yet. Each
|
// newGeoJS returns what looks clients up at GeoJS, none looked up yet. Each
|
||||||
|
|||||||
+102
-32
@@ -4,6 +4,7 @@ import (
|
|||||||
"bytes"
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
"crypto/rand"
|
"crypto/rand"
|
||||||
|
"errors"
|
||||||
"io"
|
"io"
|
||||||
"net/http"
|
"net/http"
|
||||||
"os"
|
"os"
|
||||||
@@ -20,7 +21,7 @@ import (
|
|||||||
_ "time/tzdata"
|
_ "time/tzdata"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/sww"
|
"sneak.berlin/go/smallwebwaf/internal/sww"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
|
||||||
)
|
)
|
||||||
|
|
||||||
// commandForMain, while set in the environment, has this test binary run Main
|
// commandForMain, while set in the environment, has this test binary run Main
|
||||||
@@ -106,7 +107,7 @@ func TestMainRunsWAF(t *testing.T) {
|
|||||||
|
|
||||||
starting := out.lineBeforeExit(t, "starting", exited)
|
starting := out.lineBeforeExit(t, "starting", exited)
|
||||||
addr, _ := starting["address"].(string)
|
addr, _ := starting["address"].(string)
|
||||||
ask(t, addr, "/", http.StatusOK, waf.DispositionAllow)
|
ask(t, addr, "/", http.StatusOK, wafprotocol.DispositionAllow)
|
||||||
|
|
||||||
// The files are written as sww stops.
|
// The files are written as sww stops.
|
||||||
err = cmd.Process.Signal(syscall.SIGTERM)
|
err = cmd.Process.Signal(syscall.SIGTERM)
|
||||||
@@ -164,51 +165,114 @@ func wantUTC(t *testing.T, where string, value any) {
|
|||||||
func TestMainWithoutACommandItKnowsPrintsTheCommands(t *testing.T) {
|
func TestMainWithoutACommandItKnowsPrintsTheCommands(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
// completion is cobra's own command, which sww turns off.
|
||||||
|
for _, command := range []string{"", "serve", "completion"} {
|
||||||
|
t.Run("command "+command, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
code, stdout, stderr := runMain(t, command)
|
||||||
|
|
||||||
|
if code != 2 {
|
||||||
|
t.Errorf("exit status %d, want 2", code)
|
||||||
|
}
|
||||||
|
|
||||||
|
wantTheCommands(t, "stderr", stderr)
|
||||||
|
|
||||||
|
if stdout != "" {
|
||||||
|
t.Errorf("stdout %q, want nothing", stdout)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMainHelpPrintsTheCommands(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
for _, command := range []string{"help", "--help"} {
|
||||||
|
t.Run("command "+command, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
code, stdout, _ := runMain(t, command)
|
||||||
|
|
||||||
|
if code != 0 {
|
||||||
|
t.Errorf("exit status %d, want 0", code)
|
||||||
|
}
|
||||||
|
|
||||||
|
wantTheCommands(t, "stdout", stdout)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// wantTheCommands checks that text, what Main wrote to where, names each
|
||||||
|
// command.
|
||||||
|
func wantTheCommands(t *testing.T, where, text string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
for _, name := range []string{"rproxy", wafCommand, "healthcheck"} {
|
||||||
|
if !strings.Contains(text, name) {
|
||||||
|
t.Errorf("%s %q does not name %s", where, text, name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMainHealthCheckRefusesAnArgument(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
for _, argument := range []string{"now", "--x", "-h"} {
|
||||||
|
t.Run("argument "+argument, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
code, stdout, stderr := runMain(t, "healthcheck "+argument)
|
||||||
|
|
||||||
|
want := "sww healthcheck: unexpected argument \"" +
|
||||||
|
argument + "\"\n"
|
||||||
|
if code != 1 || stderr != want || stdout != "" {
|
||||||
|
t.Errorf("exit status %d, stdout %q and stderr %q, "+
|
||||||
|
"want 1, nothing and %q", code, stdout, stderr, want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// runMain runs Main with command in a process of its own, and returns its
|
||||||
|
// exit status and what it wrote to stdout and stderr.
|
||||||
|
func runMain(t *testing.T, command string) (int, string, string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
binary, err := os.Executable()
|
binary, err := os.Executable()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("find the test binary: %v", err)
|
t.Fatalf("find the test binary: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, command := range []string{"", "serve"} {
|
|
||||||
t.Run("command "+command, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
ctx, cancel := context.WithTimeout(t.Context(), waitLimit)
|
ctx, cancel := context.WithTimeout(t.Context(), waitLimit)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
|
|
||||||
var stdout, stderr bytes.Buffer
|
var stdout, stderr bytes.Buffer
|
||||||
|
|
||||||
|
//nolint:gosec // this test binary
|
||||||
cmd := exec.CommandContext(ctx, binary)
|
cmd := exec.CommandContext(ctx, binary)
|
||||||
cmd.Stdout, cmd.Stderr = &stdout, &stderr
|
cmd.Stdout, cmd.Stderr = &stdout, &stderr
|
||||||
|
|
||||||
cmd.Env = append(os.Environ(), commandForMain+"="+command)
|
cmd.Env = append(os.Environ(), commandForMain+"="+command)
|
||||||
|
|
||||||
err := cmd.Run()
|
err = cmd.Run()
|
||||||
|
|
||||||
if code := cmd.ProcessState.ExitCode(); code != 2 {
|
var exitErr *exec.ExitError
|
||||||
t.Errorf("exit status %d (%v), want 2", code, err)
|
if err != nil && !errors.As(err, &exitErr) {
|
||||||
|
t.Fatalf("run: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, name := range []string{"rproxy", wafCommand, "healthcheck"} {
|
return cmd.ProcessState.ExitCode(), stdout.String(), stderr.String()
|
||||||
if !strings.Contains(stderr.String(), name) {
|
|
||||||
t.Errorf(
|
|
||||||
"stderr %q does not name %s",
|
|
||||||
stderr.String(),
|
|
||||||
name,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if stdout.Len() != 0 {
|
|
||||||
t.Errorf("stdout %q, want nothing", stdout.String())
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// ask sends waf at addr rproxy's question about a request for path from
|
// ask sends waf at addr rproxy's question about a request for path from
|
||||||
// placed, checks the answer's status and disposition, and closes it.
|
// placed, checks the answer's status and disposition, and closes it.
|
||||||
func ask(t *testing.T, addr, path string, status int, disposition string) {
|
func ask(
|
||||||
|
t *testing.T,
|
||||||
|
addr, path string,
|
||||||
|
status int,
|
||||||
|
disposition wafprotocol.Disposition,
|
||||||
|
) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
req, err := http.NewRequestWithContext(t.Context(), http.MethodGet,
|
req, err := http.NewRequestWithContext(t.Context(), http.MethodGet,
|
||||||
@@ -217,7 +281,7 @@ func ask(t *testing.T, addr, path string, status int, disposition string) {
|
|||||||
t.Fatalf("new request: %v", err)
|
t.Fatalf("new request: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
req.Header.Set(waf.ClientHeader, placed)
|
req.Header.Set(wafprotocol.ClientHeader, placed)
|
||||||
req.Header.Set("X-Request-ID", rand.Text())
|
req.Header.Set("X-Request-ID", rand.Text())
|
||||||
|
|
||||||
transport := &http.Transport{}
|
transport := &http.Transport{}
|
||||||
@@ -231,12 +295,12 @@ func ask(t *testing.T, addr, path string, status int, disposition string) {
|
|||||||
_, _ = io.Copy(io.Discard, res.Body)
|
_, _ = io.Copy(io.Discard, res.Body)
|
||||||
_ = res.Body.Close()
|
_ = res.Body.Close()
|
||||||
|
|
||||||
if res.StatusCode != status ||
|
answered := res.Header.Get(wafprotocol.DispositionHeader)
|
||||||
res.Header.Get(waf.DispositionHeader) != disposition {
|
if res.StatusCode != status || answered != string(disposition) {
|
||||||
t.Errorf(
|
t.Errorf(
|
||||||
"answered %d with disposition %q, want %d with %q",
|
"answered %d with disposition %q, want %d with %q",
|
||||||
res.StatusCode,
|
res.StatusCode,
|
||||||
res.Header.Get(waf.DispositionHeader),
|
answered,
|
||||||
status,
|
status,
|
||||||
disposition,
|
disposition,
|
||||||
)
|
)
|
||||||
@@ -287,12 +351,18 @@ func TestWAFKeepsItsBansAcrossARestartAndNeedsNoApp(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
runWAF(t, env, func(addr string) {
|
runWAF(t, env, func(addr string) {
|
||||||
ask(t, addr, "/", http.StatusOK, waf.DispositionAllow)
|
ask(t, addr, "/", http.StatusOK, wafprotocol.DispositionAllow)
|
||||||
// The default rule file bans a probe for this.
|
// The default rule file bans a probe for this.
|
||||||
ask(t, addr, "/.env", http.StatusForbidden, waf.DispositionDeny)
|
ask(
|
||||||
|
t,
|
||||||
|
addr,
|
||||||
|
"/.env",
|
||||||
|
http.StatusForbidden,
|
||||||
|
wafprotocol.DispositionDeny,
|
||||||
|
)
|
||||||
})
|
})
|
||||||
|
|
||||||
runWAF(t, env, func(addr string) {
|
runWAF(t, env, func(addr string) {
|
||||||
ask(t, addr, "/", http.StatusForbidden, waf.DispositionDeny)
|
ask(t, addr, "/", http.StatusForbidden, wafprotocol.DispositionDeny)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
+29
-14
@@ -1,7 +1,8 @@
|
|||||||
package waf
|
package waf
|
||||||
|
|
||||||
// The heuristic abuseipdb. It reads SWW_WAF_ABUSEIPDB_KEY,
|
// The heuristic abuseipdb. It reads SWW_WAF_ABUSEIPDB_KEY,
|
||||||
// SWW_WAF_ABUSEIPDB_MIN_SCORE and SWW_WAF_DNSBL_ABUSEIPDB_ACTION.
|
// SWW_WAF_ABUSEIPDB_MIN_SCORE, SWW_WAF_DNSBL_ABUSEIPDB_ACTION and
|
||||||
|
// SWW_WAF_BAN_RESPONSE.
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
@@ -16,37 +17,50 @@ import (
|
|||||||
|
|
||||||
// abuseIPDB adds the tag abuseipdb and notes AbuseIPDB, as noteHit does, with
|
// abuseIPDB adds the tag abuseipdb and notes AbuseIPDB, as noteHit does, with
|
||||||
// the score, when its score of the client is a hit, and refuses the request
|
// the score, when its score of the client is a hit, and refuses the request
|
||||||
// when SWW_WAF_DNSBL_ABUSEIPDB_ACTION is deny, as dnsbl does for a zone. While
|
// with SWW_WAF_BAN_RESPONSE when SWW_WAF_DNSBL_ABUSEIPDB_ACTION is deny, as
|
||||||
// SWW_WAF_ABUSEIPDB_KEY is unset it does nothing. A client without a score is
|
// dnsbl does for a zone. While SWW_WAF_ABUSEIPDB_KEY is unset it does nothing.
|
||||||
// checked in the background, by the request's address, if its history counts
|
// A client without a score is checked in the background, by the request's
|
||||||
// an offence, and the request does not wait for the answer. The score is then
|
// address, if its history counts an offence, and the request does not wait for
|
||||||
|
// the answer. The score is then
|
||||||
// used for each address of the client. Its figures are the score and
|
// used for each address of the client. Its figures are the score and
|
||||||
// SWW_WAF_ABUSEIPDB_MIN_SCORE.
|
// SWW_WAF_ABUSEIPDB_MIN_SCORE.
|
||||||
type abuseIPDB struct{}
|
type abuseIPDB struct {
|
||||||
|
abuseIPDB *reputation.AbuseIPDB
|
||||||
|
// limiter is the table of clients the rate limits count, whose histories
|
||||||
|
// say which clients are offenders.
|
||||||
|
limiter *ratelimit.Limiter
|
||||||
|
keySet bool // SWW_WAF_ABUSEIPDB_KEY is set
|
||||||
|
minScore int64 // SWW_WAF_ABUSEIPDB_MIN_SCORE
|
||||||
|
action config.ReputationAction // SWW_WAF_DNSBL_ABUSEIPDB_ACTION
|
||||||
|
banResponse int // SWW_WAF_BAN_RESPONSE
|
||||||
|
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
|
||||||
|
hitNoter reputationHitNoter
|
||||||
|
}
|
||||||
|
|
||||||
func (abuseIPDB) name() string { return "abuseipdb" }
|
func (abuseIPDB) name() string { return "abuseipdb" }
|
||||||
|
|
||||||
func (abuseIPDB) check(
|
func (a abuseIPDB) check(
|
||||||
ctx context.Context,
|
ctx context.Context,
|
||||||
rq *request,
|
rq *request,
|
||||||
_ time.Time,
|
_ time.Time,
|
||||||
) *verdict {
|
) *verdict {
|
||||||
if rq.h.config.AbuseIPDBKey == "" {
|
if !a.keySet {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
client := clientGroup(rq.h.config, rq.client)
|
client := clientGroup(a.ipv6GroupPrefix, rq.client)
|
||||||
held, _ := rq.h.limiter.Client(client)
|
held, _ := a.limiter.Client(client)
|
||||||
offender := held.History.Offences != ratelimit.Offences{}
|
offender := held.History.Offences != ratelimit.Offences{}
|
||||||
|
|
||||||
score, hit := rq.h.abuseIPDB.Hit(ctx, client, rq.client, offender)
|
score, hit := a.abuseIPDB.Hit(ctx, client, rq.client, offender)
|
||||||
if !hit {
|
if !hit {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
rq.abuseIPDBHit = true
|
rq.abuseIPDBHit = true
|
||||||
rq.addTag(config.AbuseIPDBTag)
|
rq.addTag(config.AbuseIPDBTag)
|
||||||
rq.noteHit(
|
a.hitNoter.noteHit(
|
||||||
|
rq,
|
||||||
bans.ReputationHit{
|
bans.ReputationHit{
|
||||||
Source: reputation.AbuseIPDBSource,
|
Source: reputation.AbuseIPDBSource,
|
||||||
Score: &score,
|
Score: &score,
|
||||||
@@ -54,14 +68,15 @@ func (abuseIPDB) check(
|
|||||||
"scored by AbuseIPDB at or over SWW_WAF_ABUSEIPDB_MIN_SCORE",
|
"scored by AbuseIPDB at or over SWW_WAF_ABUSEIPDB_MIN_SCORE",
|
||||||
)
|
)
|
||||||
|
|
||||||
if rq.h.config.DNSBLAbuseIPDBAction != config.ReputationActionDeny {
|
if a.action != config.ReputationActionDeny {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
return &verdict{
|
return &verdict{
|
||||||
action: requestlog.ActionClientDenied,
|
action: requestlog.ActionClientDenied,
|
||||||
|
status: a.banResponse,
|
||||||
figures: map[string]any{
|
figures: map[string]any{
|
||||||
figureScore: score, "min_score": rq.h.config.AbuseIPDBMinScore,
|
figureScore: score, "min_score": a.minScore,
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -26,7 +26,9 @@ func BanAbuseIPDBBlacklist(
|
|||||||
scores := make(map[netip.Prefix]int64, len(listed))
|
scores := make(map[netip.Prefix]int64, len(listed))
|
||||||
|
|
||||||
for _, client := range listed {
|
for _, client := range listed {
|
||||||
netblock := banNetblock(cfg, client.Addr)
|
netblock := banNetblock(
|
||||||
|
cfg.BanScopeV4Prefix, cfg.IPv6GroupPrefix, client.Addr,
|
||||||
|
)
|
||||||
scores[netblock] = max(scores[netblock], client.Score)
|
scores[netblock] = max(scores[netblock], client.Score)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -54,7 +54,7 @@ func TestClientsOnTheAbuseIPDBBlacklistAreBannedForTheBanDays(t *testing.T) {
|
|||||||
})
|
})
|
||||||
|
|
||||||
// Two clients in one netblock make one ban, with the higher score.
|
// Two clients in one netblock make one ban, with the higher score.
|
||||||
server.BanAbuseIPDBBlacklist([]reputation.BlacklistEntry{
|
server.banAbuseIPDBBlacklist([]reputation.BlacklistEntry{
|
||||||
{Addr: netip.MustParseAddr(client), Score: 95},
|
{Addr: netip.MustParseAddr(client), Score: 95},
|
||||||
{Addr: netip.MustParseAddr(otherClient), Score: 100},
|
{Addr: netip.MustParseAddr(otherClient), Score: 100},
|
||||||
{Addr: netip.MustParseAddr("2001:db8:5::1"), Score: 90},
|
{Addr: netip.MustParseAddr("2001:db8:5::1"), Score: 90},
|
||||||
@@ -88,7 +88,7 @@ func TestAbuseIPDBBlacklistMakesNoBanInObserveMode(t *testing.T) {
|
|||||||
map[string]string{mode: "observe"},
|
map[string]string{mode: "observe"},
|
||||||
)
|
)
|
||||||
|
|
||||||
server.BanAbuseIPDBBlacklist([]reputation.BlacklistEntry{
|
server.banAbuseIPDBBlacklist([]reputation.BlacklistEntry{
|
||||||
{Addr: netip.MustParseAddr(client), Score: 100},
|
{Addr: netip.MustParseAddr(client), Score: 100},
|
||||||
})
|
})
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,58 @@
|
|||||||
|
package waf
|
||||||
|
|
||||||
|
// The heuristic active_ban. It reads SWW_WAF_MODE and SWW_WAF_BAN_RESPONSE; the
|
||||||
|
// bans are the ban ledger's.
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// activeBan refuses a client a ban covers, with SWW_WAF_BAN_RESPONSE: a ban on
|
||||||
|
// a netblock the client is in, in force at now. It notes for the log line when
|
||||||
|
// that ban ends. A request that makes the ban permanent, or in observe mode
|
||||||
|
// would have, raises the alert for it. Its figures are the ban's netblock and
|
||||||
|
// when it ends.
|
||||||
|
type activeBan struct {
|
||||||
|
banMaker banMaker
|
||||||
|
}
|
||||||
|
|
||||||
|
func (activeBan) name() string { return "active_ban" }
|
||||||
|
|
||||||
|
func (a activeBan) check(
|
||||||
|
_ context.Context,
|
||||||
|
rq *request,
|
||||||
|
now time.Time,
|
||||||
|
) *verdict {
|
||||||
|
ledger := a.banMaker.ledger
|
||||||
|
|
||||||
|
check := ledger.Check
|
||||||
|
if a.banMaker.observe {
|
||||||
|
check = ledger.Find // the ban refuses nothing, and stays as it is
|
||||||
|
}
|
||||||
|
|
||||||
|
ban, banned, madePermanent := check(rq.client, now)
|
||||||
|
|
||||||
|
var expires string
|
||||||
|
if banned {
|
||||||
|
expires = banExpires(ban)
|
||||||
|
rq.line.BanExpires = expires
|
||||||
|
}
|
||||||
|
|
||||||
|
if madePermanent {
|
||||||
|
ban.Expires = time.Time{} // the ban made permanent, which Find leaves as it is
|
||||||
|
a.banMaker.alertBan(rq, ban)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !banned {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
return &verdict{
|
||||||
|
action: banAction(ban.Permanent()),
|
||||||
|
status: a.banMaker.banResponse,
|
||||||
|
figures: map[string]any{
|
||||||
|
figureNetblock: ban.Netblock.String(), "ban_expires": expires,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
+58
-107
@@ -2,7 +2,6 @@ package waf
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
"crypto/subtle"
|
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
@@ -15,7 +14,6 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/state"
|
"sneak.berlin/go/smallwebwaf/internal/state"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -41,80 +39,23 @@ var (
|
|||||||
errNotAddress = errors.New("is not an address, such as 203.0.113.9")
|
errNotAddress = errors.New("is not an address, such as 203.0.113.9")
|
||||||
)
|
)
|
||||||
|
|
||||||
// answerHealth answers a request for the health endpoint.
|
// handleHealth answers GET wafprotocol.HealthPath.
|
||||||
func (rq *request) answerHealth() {
|
func handleHealth() http.HandlerFunc {
|
||||||
rq.line.Action = requestlog.ActionOwnEndpoint
|
return func(w http.ResponseWriter, _ *http.Request) {
|
||||||
// Set here rather than left to Go's server, which would set it only
|
// Set here rather than left to Go's server, which would set it only
|
||||||
// after the log line has taken the response's headers.
|
// after the log line has taken the response's headers.
|
||||||
rq.out.Header().Set("Content-Type", "text/plain; charset=utf-8")
|
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
|
||||||
_, _ = io.WriteString(rq.out, "ok\n")
|
_, _ = io.WriteString(w, "ok\n")
|
||||||
}
|
|
||||||
|
|
||||||
// answerAdmin answers a request for one of waf's own endpoints. Each
|
|
||||||
// needs a token, sent as Authorization: Bearer <token>: the metrics
|
|
||||||
// SWW_WAF_METRICS_TOKEN, the others SWW_WAF_ADMIN_TOKEN. A request without it is
|
|
||||||
// refused with 401. An endpoint whose token is unset answers 404, as any other
|
|
||||||
// path does.
|
|
||||||
func (rq *request) answerAdmin() {
|
|
||||||
rq.line.Action = requestlog.ActionOwnEndpoint
|
|
||||||
|
|
||||||
token, answer := rq.endpoint()
|
|
||||||
|
|
||||||
switch {
|
|
||||||
case token == "":
|
|
||||||
http.Error(
|
|
||||||
rq.out,
|
|
||||||
http.StatusText(http.StatusNotFound),
|
|
||||||
http.StatusNotFound,
|
|
||||||
)
|
|
||||||
case !hasToken(rq.in, token):
|
|
||||||
rq.out.Header().Set("WWW-Authenticate", "Bearer")
|
|
||||||
http.Error(
|
|
||||||
rq.out,
|
|
||||||
http.StatusText(http.StatusUnauthorized),
|
|
||||||
http.StatusUnauthorized,
|
|
||||||
)
|
|
||||||
default:
|
|
||||||
answer()
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// endpoint returns the token the request's endpoint needs, and what
|
// handleListBans answers GET BansPath with every ban held.
|
||||||
// answers the request there; "" when there is no such endpoint.
|
func (h *handler) handleListBans() http.HandlerFunc {
|
||||||
func (rq *request) endpoint() (string, func()) {
|
return func(w http.ResponseWriter, _ *http.Request) {
|
||||||
cfg := rq.h.config
|
h.answerBans(w, h.ledger.Snapshot())
|
||||||
method, path := rq.in.Method, rq.in.URL.Path
|
|
||||||
|
|
||||||
switch {
|
|
||||||
case method == http.MethodGet && path == MetricsPath:
|
|
||||||
return cfg.MetricsToken, func() { rq.h.metrics.ServeHTTP(rq.out, rq.in) }
|
|
||||||
case method == http.MethodGet && path == BansPath:
|
|
||||||
return cfg.AdminToken, rq.listBans
|
|
||||||
case method == http.MethodPost && path == BansPath:
|
|
||||||
return cfg.AdminToken, rq.addBan
|
|
||||||
case method == http.MethodDelete && strings.HasPrefix(path, BansPath+"/"):
|
|
||||||
return cfg.AdminToken, rq.liftBans
|
|
||||||
case method == http.MethodGet && strings.HasPrefix(path, ClientsPath):
|
|
||||||
return cfg.AdminToken, rq.showClient
|
|
||||||
default:
|
|
||||||
return "", nil
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// hasToken reports whether r carries token, as Authorization: Bearer
|
|
||||||
// <token>.
|
|
||||||
func hasToken(r *http.Request, token string) bool {
|
|
||||||
scheme, sent, _ := strings.Cut(r.Header.Get("Authorization"), " ")
|
|
||||||
|
|
||||||
return strings.EqualFold(scheme, "Bearer") &&
|
|
||||||
subtle.ConstantTimeCompare([]byte(sent), []byte(token)) == 1
|
|
||||||
}
|
|
||||||
|
|
||||||
// listBans answers GET BansPath with every ban held.
|
|
||||||
func (rq *request) listBans() {
|
|
||||||
rq.answerBans(rq.h.ledger.Snapshot())
|
|
||||||
}
|
|
||||||
|
|
||||||
// banToAdd is the body of POST BansPath.
|
// banToAdd is the body of POST BansPath.
|
||||||
type banToAdd struct {
|
type banToAdd struct {
|
||||||
// Netblock is a netblock, or a client's address, which stands for the
|
// Netblock is a netblock, or a client's address, which stands for the
|
||||||
@@ -126,20 +67,21 @@ type banToAdd struct {
|
|||||||
Reason string `json:"reason"`
|
Reason string `json:"reason"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// addBan answers POST BansPath: it bans the netblock the body names, as
|
// handleAddBan answers POST BansPath: it bans the netblock the body names,
|
||||||
// an admin, from now for the duration the body gives, with its reason,
|
// as an admin, from now for the duration the body gives, with its reason,
|
||||||
// and answers with that ban.
|
// and answers with that ban.
|
||||||
func (rq *request) addBan() {
|
func (h *handler) handleAddBan() http.HandlerFunc {
|
||||||
toAdd, err := rq.readBanToAdd()
|
return func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
toAdd, err := readBanToAdd(r)
|
||||||
|
|
||||||
var (
|
var (
|
||||||
netblock netip.Prefix
|
netblock netip.Prefix
|
||||||
expires time.Time
|
expires time.Time
|
||||||
now = rq.h.now()
|
now = h.now()
|
||||||
)
|
)
|
||||||
|
|
||||||
if err == nil {
|
if err == nil {
|
||||||
netblock, err = rq.h.parseBanNetblock(toAdd.Netblock)
|
netblock, err = h.parseBanNetblock(toAdd.Netblock)
|
||||||
}
|
}
|
||||||
|
|
||||||
if err == nil {
|
if err == nil {
|
||||||
@@ -147,23 +89,22 @@ func (rq *request) addBan() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
http.Error(rq.out, err.Error(), http.StatusBadRequest)
|
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||||
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
ban := rq.h.ledger.BanForAdmin(netblock, now, expires, toAdd.Reason)
|
ban := h.ledger.BanForAdmin(netblock, now, expires, toAdd.Reason)
|
||||||
rq.answerBans([]bans.Ban{ban})
|
h.answerBans(w, []bans.Ban{ban})
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// readBanToAdd reads the body of POST BansPath: a JSON object with
|
// readBanToAdd reads the body of POST BansPath: a JSON object with
|
||||||
// nothing but whitespace after it, in at most banBodyMaxBytes.
|
// nothing but whitespace after it, in at most banBodyMaxBytes.
|
||||||
func (rq *request) readBanToAdd() (banToAdd, error) {
|
func readBanToAdd(r *http.Request) (banToAdd, error) {
|
||||||
data, err := io.ReadAll(
|
data, err := io.ReadAll(
|
||||||
http.MaxBytesReader(nil, rq.in.Body, banBodyMaxBytes),
|
http.MaxBytesReader(nil, r.Body, banBodyMaxBytes),
|
||||||
)
|
)
|
||||||
rq.requestBytes = int64(len(data))
|
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return banToAdd{}, fmt.Errorf("%w: %w", errNotBanToAdd, err)
|
return banToAdd{}, fmt.Errorf("%w: %w", errNotBanToAdd, err)
|
||||||
}
|
}
|
||||||
@@ -224,7 +165,11 @@ func (h *handler) parseBanNetblock(value string) (netip.Prefix, error) {
|
|||||||
return netip.Prefix{}, fmt.Errorf("netblock %q %w", value, errZone)
|
return netip.Prefix{}, fmt.Errorf("netblock %q %w", value, errZone)
|
||||||
}
|
}
|
||||||
|
|
||||||
return banNetblock(h.config, addr), nil
|
return banNetblock(
|
||||||
|
h.config.BanScopeV4Prefix,
|
||||||
|
h.config.IPv6GroupPrefix,
|
||||||
|
addr,
|
||||||
|
), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// expiry returns when a ban made at now for duration ends: duration
|
// expiry returns when a ban made at now for duration ends: duration
|
||||||
@@ -246,21 +191,22 @@ func expiry(duration string, now time.Time) (time.Time, error) {
|
|||||||
return now.Add(length), nil
|
return now.Add(length), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// liftBans answers DELETE BansPath/<client>: it lifts every ban active on
|
// handleLiftBans answers DELETE BansPath/<client>: it lifts every ban
|
||||||
// a netblock the client's address is in, and answers with those bans, or
|
// active on a netblock the client's address is in, and answers with those
|
||||||
// with 404 when none is active.
|
// bans, or with 404 when none is active.
|
||||||
func (rq *request) liftBans() {
|
func (h *handler) handleLiftBans() http.HandlerFunc {
|
||||||
client, err := pathAddress(rq.in.URL.Path, BansPath+"/")
|
return func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
client, err := pathAddress(r.URL.Path, BansPath+"/")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
http.Error(rq.out, err.Error(), http.StatusBadRequest)
|
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||||
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
lifted := rq.h.ledger.Lift(client, rq.h.now())
|
lifted := h.ledger.Lift(client, h.now())
|
||||||
if len(lifted) == 0 {
|
if len(lifted) == 0 {
|
||||||
http.Error(
|
http.Error(
|
||||||
rq.out,
|
w,
|
||||||
"no ban is active on "+client.String(),
|
"no ban is active on "+client.String(),
|
||||||
http.StatusNotFound,
|
http.StatusNotFound,
|
||||||
)
|
)
|
||||||
@@ -268,7 +214,8 @@ func (rq *request) liftBans() {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
rq.answerBans(lifted)
|
h.answerBans(w, lifted)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// clientAnswer is the answer to GET ClientsPath<ip>: the client the
|
// clientAnswer is the answer to GET ClientsPath<ip>: the client the
|
||||||
@@ -280,27 +227,31 @@ type clientAnswer struct {
|
|||||||
Bans []state.BanEntry `json:"bans"`
|
Bans []state.BanEntry `json:"bans"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// showClient answers GET ClientsPath<ip> with what waf knows of the client:
|
// handleShowClient answers GET ClientsPath<ip> with what waf knows of the
|
||||||
// its counters, its history, which holds its country as last looked up and its
|
// client: its counters, its history, which holds its country as last looked
|
||||||
// offences, and its bans with their notes.
|
// up and its offences, and its bans with their notes.
|
||||||
func (rq *request) showClient() {
|
func (h *handler) handleShowClient() http.HandlerFunc {
|
||||||
addr, err := pathAddress(rq.in.URL.Path, ClientsPath)
|
return func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
addr, err := pathAddress(r.URL.Path, ClientsPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
http.Error(rq.out, err.Error(), http.StatusBadRequest)
|
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||||
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
answer := clientAnswer{
|
answer := clientAnswer{
|
||||||
Bans: state.BanEntries(rq.h.ledger.Covering(addr)),
|
Bans: state.BanEntries(h.ledger.Covering(addr)),
|
||||||
}
|
}
|
||||||
|
|
||||||
client, seen := rq.h.limiter.Client(clientGroup(rq.h.config, addr))
|
client, seen := h.limiter.Client(
|
||||||
|
clientGroup(h.config.IPv6GroupPrefix, addr),
|
||||||
|
)
|
||||||
if seen {
|
if seen {
|
||||||
answer.Client = &client
|
answer.Client = &client
|
||||||
}
|
}
|
||||||
|
|
||||||
rq.answerJSON(answer)
|
h.answerJSON(w, answer)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// pathAddress reads the client's address that follows prefix in path.
|
// pathAddress reads the client's address that follows prefix in path.
|
||||||
@@ -316,27 +267,27 @@ func pathAddress(path, prefix string) (netip.Addr, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// answerBans answers with held under bans, as bans.json lists them.
|
// answerBans answers with held under bans, as bans.json lists them.
|
||||||
func (rq *request) answerBans(held []bans.Ban) {
|
func (h *handler) answerBans(w http.ResponseWriter, held []bans.Ban) {
|
||||||
rq.answerJSON(struct {
|
h.answerJSON(w, struct {
|
||||||
Bans []state.BanEntry `json:"bans"`
|
Bans []state.BanEntry `json:"bans"`
|
||||||
}{state.BanEntries(held)})
|
}{state.BanEntries(held)})
|
||||||
}
|
}
|
||||||
|
|
||||||
// answerJSON answers with value as indented JSON.
|
// answerJSON answers with value as indented JSON.
|
||||||
func (rq *request) answerJSON(value any) {
|
func (h *handler) answerJSON(w http.ResponseWriter, value any) {
|
||||||
body, err := json.MarshalIndent(value, "", " ")
|
body, err := json.MarshalIndent(value, "", " ")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
rq.h.processLog.Error(
|
h.processLog.Error(
|
||||||
"encoding an answer failed",
|
"encoding an answer failed",
|
||||||
"error",
|
"error",
|
||||||
err.Error(),
|
err.Error(),
|
||||||
)
|
)
|
||||||
http.Error(rq.out, http.StatusText(http.StatusInternalServerError),
|
http.Error(w, http.StatusText(http.StatusInternalServerError),
|
||||||
http.StatusInternalServerError)
|
http.StatusInternalServerError)
|
||||||
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
rq.out.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
_, _ = rq.out.Write(append(body, '\n'))
|
_, _ = w.Write(append(body, '\n'))
|
||||||
}
|
}
|
||||||
@@ -14,6 +14,7 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/state"
|
"sneak.berlin/go/smallwebwaf/internal/state"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -333,6 +334,39 @@ func TestBanToAddThatCannotBeReadIsRefused(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestBanToAddIsLoggedWithTheBytesOfItsBody(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
s, _, _ := startWithClock(
|
||||||
|
t,
|
||||||
|
"",
|
||||||
|
map[string]string{adminToken: adminSecret},
|
||||||
|
)
|
||||||
|
|
||||||
|
s.admin(http.MethodPost, waf.BansPath, banOtherClient, http.StatusOK)
|
||||||
|
|
||||||
|
line := s.out.endedLines(t, s.sent)[s.sent-1]
|
||||||
|
if line.RequestBytes != int64(len(banOtherClient)) {
|
||||||
|
t.Errorf(
|
||||||
|
"the ban to add was logged with request_bytes %d, want %d",
|
||||||
|
line.RequestBytes,
|
||||||
|
len(banOtherClient),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The health endpoint reads none of a body sent to it.
|
||||||
|
s.ownEndpointRequest("", http.MethodGet, wafprotocol.HealthPath, "ok",
|
||||||
|
http.StatusOK)
|
||||||
|
|
||||||
|
line = s.out.endedLines(t, s.sent)[s.sent-1]
|
||||||
|
if line.RequestBytes != 0 {
|
||||||
|
t.Errorf(
|
||||||
|
"the health request was logged with request_bytes %d, want 0",
|
||||||
|
line.RequestBytes,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestClientEndpointShowsTheClientAndItsBans(t *testing.T) {
|
func TestClientEndpointShowsTheClientAndItsBans(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -11,7 +11,6 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -250,7 +249,7 @@ func TestObserveModeWorksOutABanOnlyWhenItsAlertWouldBeSent(t *testing.T) {
|
|||||||
// never sent them, and returns the queue they wait in as well.
|
// never sent them, and returns the queue they wait in as well.
|
||||||
func startWithAlerts(
|
func startWithAlerts(
|
||||||
t *testing.T, env map[string]string,
|
t *testing.T, env map[string]string,
|
||||||
) (*sender, *clock, *waf.Server, *alerts.Queue) {
|
) (*sender, *clock, *wafAndParts, *alerts.Queue) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
return startAppWithAlerts(
|
return startAppWithAlerts(
|
||||||
@@ -263,7 +262,7 @@ func startWithAlerts(
|
|||||||
// startAppWithAlerts is startWithAlerts in front of the app handler.
|
// startAppWithAlerts is startWithAlerts in front of the app handler.
|
||||||
func startAppWithAlerts(
|
func startAppWithAlerts(
|
||||||
t *testing.T, handler http.HandlerFunc, env map[string]string,
|
t *testing.T, handler http.HandlerFunc, env map[string]string,
|
||||||
) (*sender, *clock, *waf.Server, *alerts.Queue) {
|
) (*sender, *clock, *wafAndParts, *alerts.Queue) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
app := startApp(t, handler)
|
app := startApp(t, handler)
|
||||||
@@ -292,7 +291,7 @@ func startAppWithAlerts(
|
|||||||
// the time raised, for ban, with its netblock, cause, reason and notes,
|
// the time raised, for ban, with its netblock, cause, reason and notes,
|
||||||
// which ends at expires, as the log line gives it.
|
// which ends at expires, as the log line gives it.
|
||||||
func banAlert(
|
func banAlert(
|
||||||
event string,
|
event alerts.Event,
|
||||||
raised time.Time,
|
raised time.Time,
|
||||||
client string,
|
client string,
|
||||||
ban bans.Ban,
|
ban bans.Ban,
|
||||||
@@ -307,7 +306,9 @@ func banAlert(
|
|||||||
Netblock: ban.Netblock,
|
Netblock: ban.Netblock,
|
||||||
Reason: ban.Reason,
|
Reason: ban.Reason,
|
||||||
Detail: map[string]any{
|
Detail: map[string]any{
|
||||||
"cause": ban.Cause, "ban_expires": expires, "notes": ban.Notes,
|
"cause": string(
|
||||||
|
ban.Cause,
|
||||||
|
), "ban_expires": expires, "notes": ban.Notes,
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -315,7 +316,7 @@ func banAlert(
|
|||||||
// attackAlert is banAlert for a ban for the probe rule of testRules, with
|
// attackAlert is banAlert for a ban for the probe rule of testRules, with
|
||||||
// the netblock and the notes of ban.
|
// the netblock and the notes of ban.
|
||||||
func attackAlert(
|
func attackAlert(
|
||||||
event string,
|
event alerts.Event,
|
||||||
raised time.Time,
|
raised time.Time,
|
||||||
client string,
|
client string,
|
||||||
ban bans.Ban,
|
ban bans.Ban,
|
||||||
|
|||||||
@@ -15,7 +15,6 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// The anomaly thresholds: the prefix of a scope followed by the end of a
|
// The anomaly thresholds: the prefix of a scope followed by the end of a
|
||||||
@@ -86,10 +85,10 @@ func TestEachScopeAndWindowOverItsThresholdAlertsOncePerCooldown(
|
|||||||
value int64
|
value int64
|
||||||
count float64
|
count float64
|
||||||
}{
|
}{
|
||||||
{requestsPerMinute, ratelimit.KindRequests, minute, 2, 3},
|
{requestsPerMinute, string(ratelimit.KindRequests), minute, 2, 3},
|
||||||
{requestsPerHour, ratelimit.KindRequests, hour, 2, 3},
|
{requestsPerHour, string(ratelimit.KindRequests), hour, 2, 3},
|
||||||
{bytesPerMinute, ratelimit.KindBytes, minute, 250, 300},
|
{bytesPerMinute, string(ratelimit.KindBytes), minute, 250, 300},
|
||||||
{bytesPerHour, ratelimit.KindBytes, hour, 250, 300},
|
{bytesPerHour, string(ratelimit.KindBytes), hour, 250, 300},
|
||||||
} {
|
} {
|
||||||
setting := scope.prefix + threshold.end
|
setting := scope.prefix + threshold.end
|
||||||
value := strconv.FormatInt(threshold.value, 10)
|
value := strconv.FormatInt(threshold.value, 10)
|
||||||
@@ -186,7 +185,7 @@ func TestEveryRequestIsCountedWhateverIsDoneWithIt(t *testing.T) {
|
|||||||
for _, sent := range []struct {
|
for _, sent := range []struct {
|
||||||
from, path string
|
from, path string
|
||||||
status int
|
status int
|
||||||
action string
|
action requestlog.Action
|
||||||
}{
|
}{
|
||||||
{allowed, "/", http.StatusOK, requestlog.ActionForward},
|
{allowed, "/", http.StatusOK, requestlog.ActionForward},
|
||||||
{exempt, "/", http.StatusOK, requestlog.ActionForward},
|
{exempt, "/", http.StatusOK, requestlog.ActionForward},
|
||||||
@@ -332,7 +331,7 @@ func TestRequestCountsForTheASNumberGeoJSGivesBeforeItEnds(t *testing.T) {
|
|||||||
// kept.
|
// kept.
|
||||||
geojsURL, _, release := startHeldGeoJS(t)
|
geojsURL, _, release := startHeldGeoJS(t)
|
||||||
|
|
||||||
var server atomic.Pointer[waf.Server]
|
var server atomic.Pointer[wafAndParts]
|
||||||
|
|
||||||
app := startApp(t, func(http.ResponseWriter, *http.Request) {
|
app := startApp(t, func(http.ResponseWriter, *http.Request) {
|
||||||
release()
|
release()
|
||||||
|
|||||||
@@ -26,6 +26,7 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -84,7 +85,7 @@ func TestSixteenThousandCookiesAreRefusedBeforeTheCoreRuleSetReadsThem(
|
|||||||
|
|
||||||
got := do(t, req)
|
got := do(t, req)
|
||||||
|
|
||||||
wantDisposition(t, got, tooLarge, waf.DispositionDeny)
|
wantDisposition(t, got, tooLarge, wafprotocol.DispositionDeny)
|
||||||
|
|
||||||
line := s.out.requestLine(t)
|
line := s.out.requestLine(t)
|
||||||
wantLine(t, line, tooLarge, requestlog.ActionRequestHeaderTooLarge)
|
wantLine(t, line, tooLarge, requestlog.ActionRequestHeaderTooLarge)
|
||||||
@@ -134,7 +135,7 @@ func TestARequestLineOverTheLimitIsRefusedWith414BeforeAnyOtherCheck(
|
|||||||
// cookies is how many cookies the request carries.
|
// cookies is how many cookies the request carries.
|
||||||
cookies int
|
cookies int
|
||||||
status int
|
status int
|
||||||
action string
|
action requestlog.Action
|
||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
"at the limit", lineLimit, 0, http.StatusOK,
|
"at the limit", lineLimit, 0, http.StatusOK,
|
||||||
@@ -164,9 +165,14 @@ func TestARequestLineOverTheLimitIsRefusedWith414BeforeAnyOtherCheck(
|
|||||||
line := s.out.requestLine(t)
|
line := s.out.requestLine(t)
|
||||||
|
|
||||||
if tc.status == http.StatusOK {
|
if tc.status == http.StatusOK {
|
||||||
wantDisposition(t, got, tc.status, waf.DispositionAllow)
|
wantDisposition(
|
||||||
|
t,
|
||||||
|
got,
|
||||||
|
tc.status,
|
||||||
|
wafprotocol.DispositionAllow,
|
||||||
|
)
|
||||||
} else {
|
} else {
|
||||||
wantDisposition(t, got, tc.status, waf.DispositionDeny)
|
wantDisposition(t, got, tc.status, wafprotocol.DispositionDeny)
|
||||||
}
|
}
|
||||||
|
|
||||||
wantLine(t, line, tc.status, tc.action)
|
wantLine(t, line, tc.status, tc.action)
|
||||||
@@ -219,9 +225,14 @@ func TestAHeaderLineOverTheLimitIsRefusedWith431(t *testing.T) {
|
|||||||
got := do(t, req)
|
got := do(t, req)
|
||||||
|
|
||||||
if tc.refused {
|
if tc.refused {
|
||||||
wantDisposition(t, got, tooLarge, waf.DispositionDeny)
|
wantDisposition(
|
||||||
|
t,
|
||||||
|
got,
|
||||||
|
tooLarge,
|
||||||
|
wafprotocol.DispositionDeny,
|
||||||
|
)
|
||||||
} else {
|
} else {
|
||||||
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
|
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
@@ -244,10 +255,7 @@ func TestRequestLineRefusalNamesItsHeuristicLengthAndLimit(t *testing.T) {
|
|||||||
func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
|
func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
repeated := make([]string, headerFieldLimit)
|
repeated := slices.Repeat([]string{"1"}, headerFieldLimit)
|
||||||
for i := range repeated {
|
|
||||||
repeated[i] = "1"
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tc := range []struct {
|
for _, tc := range []struct {
|
||||||
name string
|
name string
|
||||||
@@ -294,7 +302,12 @@ func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
|
|||||||
line := s.out.requestLine(t)
|
line := s.out.requestLine(t)
|
||||||
|
|
||||||
if tc.refused {
|
if tc.refused {
|
||||||
wantDisposition(t, got, tooLarge, waf.DispositionDeny)
|
wantDisposition(
|
||||||
|
t,
|
||||||
|
got,
|
||||||
|
tooLarge,
|
||||||
|
wafprotocol.DispositionDeny,
|
||||||
|
)
|
||||||
wantLine(
|
wantLine(
|
||||||
t,
|
t,
|
||||||
line,
|
line,
|
||||||
@@ -305,7 +318,12 @@ func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
|
wantDisposition(
|
||||||
|
t,
|
||||||
|
got,
|
||||||
|
http.StatusOK,
|
||||||
|
wafprotocol.DispositionAllow,
|
||||||
|
)
|
||||||
wantLine(t, line, http.StatusOK, requestlog.ActionForward)
|
wantLine(t, line, http.StatusOK, requestlog.ActionForward)
|
||||||
|
|
||||||
if line.DurationCoreRuleSetMS == nil {
|
if line.DurationCoreRuleSetMS == nil {
|
||||||
@@ -332,7 +350,12 @@ func TestHeaderLimitsHoldWhateverTheCoreRuleSetSettingsSay(t *testing.T) {
|
|||||||
req := s.question(siteA, client, "/")
|
req := s.question(siteA, client, "/")
|
||||||
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
|
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
|
||||||
|
|
||||||
wantDisposition(t, do(t, req), tooLarge, waf.DispositionDeny)
|
wantDisposition(
|
||||||
|
t,
|
||||||
|
do(t, req),
|
||||||
|
tooLarge,
|
||||||
|
wafprotocol.DispositionDeny,
|
||||||
|
)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -348,7 +371,12 @@ func TestClientInTheAllowNetsSkipsTheHeaderLimits(t *testing.T) {
|
|||||||
req := s.question(siteA, client, "/")
|
req := s.question(siteA, client, "/")
|
||||||
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
|
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
|
||||||
|
|
||||||
wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow)
|
wantDisposition(
|
||||||
|
t,
|
||||||
|
do(t, req),
|
||||||
|
http.StatusOK,
|
||||||
|
wafprotocol.DispositionAllow,
|
||||||
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestObserveModeAllowsARequestOverTheHeaderLimitsUnread(t *testing.T) {
|
func TestObserveModeAllowsARequestOverTheHeaderLimitsUnread(t *testing.T) {
|
||||||
@@ -364,7 +392,7 @@ func TestObserveModeAllowsARequestOverTheHeaderLimitsUnread(t *testing.T) {
|
|||||||
|
|
||||||
got := do(t, req)
|
got := do(t, req)
|
||||||
|
|
||||||
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
|
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
|
||||||
|
|
||||||
line := s.out.requestLine(t)
|
line := s.out.requestLine(t)
|
||||||
if line.WouldAction != requestlog.ActionRequestHeaderTooLarge ||
|
if line.WouldAction != requestlog.ActionRequestHeaderTooLarge ||
|
||||||
@@ -386,14 +414,14 @@ func TestTheRateLimitsDoNotCountRequestsOverTheHeaderLimits(t *testing.T) {
|
|||||||
for range 2 {
|
for range 2 {
|
||||||
req := s.question(siteA, client, "/")
|
req := s.question(siteA, client, "/")
|
||||||
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
|
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
|
||||||
wantDisposition(t, do(t, req), tooLarge, waf.DispositionDeny)
|
wantDisposition(t, do(t, req), tooLarge, wafprotocol.DispositionDeny)
|
||||||
}
|
}
|
||||||
|
|
||||||
wantDisposition(
|
wantDisposition(
|
||||||
t,
|
t,
|
||||||
s.ask(client, "/"),
|
s.ask(client, "/"),
|
||||||
http.StatusOK,
|
http.StatusOK,
|
||||||
waf.DispositionAllow,
|
wafprotocol.DispositionAllow,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -411,7 +439,7 @@ func TestTheHeaderLimitsComeBeforeEveryCheckThatRefuses(t *testing.T) {
|
|||||||
env map[string]string
|
env map[string]string
|
||||||
path string
|
path string
|
||||||
// prepare gets the server ready, at now, before the question.
|
// prepare gets the server ready, at now, before the question.
|
||||||
prepare func(t *testing.T, server *waf.Server, now time.Time)
|
prepare func(t *testing.T, server *wafAndParts, now time.Time)
|
||||||
}{
|
}{
|
||||||
{"SWW_WAF_DENY_NETS", map[string]string{denyNets: client}, "/", leaveAsItIs},
|
{"SWW_WAF_DENY_NETS", map[string]string{denyNets: client}, "/", leaveAsItIs},
|
||||||
{"a ban", nil, "/", banClient},
|
{"a ban", nil, "/", banClient},
|
||||||
@@ -426,21 +454,21 @@ func TestTheHeaderLimitsComeBeforeEveryCheckThatRefuses(t *testing.T) {
|
|||||||
"a DNS blocklist", map[string]string{
|
"a DNS blocklist", map[string]string{
|
||||||
dnsblZones: dnsblZone, dnsblResolver: noResolver, dnsblAbuseIPDBAction: actionDeny,
|
dnsblZones: dnsblZone, dnsblResolver: noResolver, dnsblAbuseIPDBAction: actionDeny,
|
||||||
},
|
},
|
||||||
"/", func(_ *testing.T, server *waf.Server, _ time.Time) {
|
"/", func(_ *testing.T, server *wafAndParts, _ time.Time) {
|
||||||
loadVerdicts(server, map[string][]string{client: {dnsblZone}})
|
loadVerdicts(server, map[string][]string{client: {dnsblZone}})
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"AbuseIPDB",
|
"AbuseIPDB",
|
||||||
map[string]string{abuseIPDBKey: accountKey, dnsblAbuseIPDBAction: actionDeny},
|
map[string]string{abuseIPDBKey: accountKey, dnsblAbuseIPDBAction: actionDeny},
|
||||||
"/", func(_ *testing.T, server *waf.Server, _ time.Time) {
|
"/", func(_ *testing.T, server *wafAndParts, _ time.Time) {
|
||||||
// Over SWW_WAF_ABUSEIPDB_MIN_SCORE, 75 by default.
|
// Over SWW_WAF_ABUSEIPDB_MIN_SCORE, 75 by default.
|
||||||
loadScores(server, map[string]int64{client: 80})
|
loadScores(server, map[string]int64{client: 80})
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"SWW_WAF_TAG_DENY", map[string]string{tagLists: "vpn=" + vpnList, tagDeny: "vpn"},
|
"SWW_WAF_TAG_DENY", map[string]string{tagLists: "vpn=" + vpnList, tagDeny: "vpn"},
|
||||||
"/", func(t *testing.T, server *waf.Server, _ time.Time) {
|
"/", func(t *testing.T, server *wafAndParts, _ time.Time) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
loadLists(t, server, map[string][]string{vpnList: {client}})
|
loadLists(t, server, map[string][]string{vpnList: {client}})
|
||||||
},
|
},
|
||||||
@@ -466,7 +494,12 @@ func TestTheHeaderLimitsComeBeforeEveryCheckThatRefuses(t *testing.T) {
|
|||||||
req := s.question(siteA, client, tc.path)
|
req := s.question(siteA, client, tc.path)
|
||||||
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
|
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
|
||||||
|
|
||||||
wantDisposition(t, do(t, req), tooLarge, waf.DispositionDeny)
|
wantDisposition(
|
||||||
|
t,
|
||||||
|
do(t, req),
|
||||||
|
tooLarge,
|
||||||
|
wafprotocol.DispositionDeny,
|
||||||
|
)
|
||||||
wantLine(
|
wantLine(
|
||||||
t,
|
t,
|
||||||
s.out.requestLine(t),
|
s.out.requestLine(t),
|
||||||
@@ -503,11 +536,16 @@ func TestARequestOverTheHeaderLimitsRunsNoOtherCheck(t *testing.T) {
|
|||||||
|
|
||||||
over := s.question(siteA, client, "/")
|
over := s.question(siteA, client, "/")
|
||||||
over.Header.Set(cookieHeader, cookies(cookieLimit+1))
|
over.Header.Set(cookieHeader, cookies(cookieLimit+1))
|
||||||
wantDisposition(t, do(t, over), tooLarge, waf.DispositionDeny)
|
wantDisposition(t, do(t, over), tooLarge, wafprotocol.DispositionDeny)
|
||||||
|
|
||||||
within := s.question(siteA, client, "/")
|
within := s.question(siteA, client, "/")
|
||||||
within.Header.Set(cookieHeader, cookies(cookieLimit))
|
within.Header.Set(cookieHeader, cookies(cookieLimit))
|
||||||
wantDisposition(t, do(t, within), http.StatusOK, waf.DispositionAllow)
|
wantDisposition(
|
||||||
|
t,
|
||||||
|
do(t, within),
|
||||||
|
http.StatusOK,
|
||||||
|
wafprotocol.DispositionAllow,
|
||||||
|
)
|
||||||
|
|
||||||
lines := s.out.requestLines(t, 2)
|
lines := s.out.requestLines(t, 2)
|
||||||
|
|
||||||
@@ -564,7 +602,7 @@ func TestNotWaitingForGeoJSNeverLowersAClientsLimit(t *testing.T) {
|
|||||||
t,
|
t,
|
||||||
s.ask(fromDE, "/"),
|
s.ask(fromDE, "/"),
|
||||||
http.StatusForbidden,
|
http.StatusForbidden,
|
||||||
waf.DispositionDeny,
|
wafprotocol.DispositionDeny,
|
||||||
)
|
)
|
||||||
|
|
||||||
line := s.out.requestLine(t)
|
line := s.out.requestLine(t)
|
||||||
@@ -601,7 +639,7 @@ func TestAClientGeoJSPlacedInNoCountryCountsAsUnknownWithoutWaiting(
|
|||||||
t,
|
t,
|
||||||
s.ask(client, "/"),
|
s.ask(client, "/"),
|
||||||
http.StatusForbidden,
|
http.StatusForbidden,
|
||||||
waf.DispositionDeny,
|
wafprotocol.DispositionDeny,
|
||||||
)
|
)
|
||||||
|
|
||||||
line := s.out.requestLine(t)
|
line := s.out.requestLine(t)
|
||||||
@@ -630,7 +668,7 @@ func TestObserveModeAsksGeoJSNothingForARequestOverTheHeaderLimits(
|
|||||||
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
|
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
|
||||||
|
|
||||||
got := do(t, req)
|
got := do(t, req)
|
||||||
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
|
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
|
||||||
|
|
||||||
if len(asked()) != 0 || got.header.Get("X-Client-Asn") != "" {
|
if len(asked()) != 0 || got.header.Get("X-Client-Asn") != "" {
|
||||||
t.Errorf(
|
t.Errorf(
|
||||||
@@ -647,7 +685,7 @@ func TestObserveModeAsksGeoJSNothingForARequestOverTheHeaderLimits(
|
|||||||
// for env. No request to it may use the network.
|
// for env. No request to it may use the network.
|
||||||
func newWAFInBubble(
|
func newWAFInBubble(
|
||||||
t *testing.T, env map[string]string,
|
t *testing.T, env map[string]string,
|
||||||
) (*waf.Server, *output) {
|
) (*wafAndParts, *output) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
settings := map[string]string{tagDeny: "country=KP"}
|
settings := map[string]string{tagDeny: "country=KP"}
|
||||||
@@ -661,7 +699,7 @@ func newWAFInBubble(
|
|||||||
// askDirectly hands a question about a request from client straight to server,
|
// askDirectly hands a question about a request from client straight to server,
|
||||||
// as change leaves it, and returns the answer's status and how long it took.
|
// as change leaves it, and returns the answer's status and how long it took.
|
||||||
func askDirectly(
|
func askDirectly(
|
||||||
t *testing.T, server *waf.Server, change func(*http.Request),
|
t *testing.T, server *wafAndParts, change func(*http.Request),
|
||||||
) (int, time.Duration) {
|
) (int, time.Duration) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
@@ -672,20 +710,20 @@ func askDirectly(
|
|||||||
http.NoBody,
|
http.NoBody,
|
||||||
)
|
)
|
||||||
req.RemoteAddr = net.JoinHostPort(rproxyPeer, "1234")
|
req.RemoteAddr = net.JoinHostPort(rproxyPeer, "1234")
|
||||||
req.Header.Set(waf.ClientHeader, client)
|
req.Header.Set(wafprotocol.ClientHeader, client)
|
||||||
req.Header.Set(requestIDHeader, "the-id")
|
req.Header.Set(requestIDHeader, "the-id")
|
||||||
change(req)
|
change(req)
|
||||||
|
|
||||||
recorder := httptest.NewRecorder()
|
recorder := httptest.NewRecorder()
|
||||||
began := time.Now()
|
began := time.Now()
|
||||||
|
|
||||||
server.Handler.ServeHTTP(recorder, req)
|
server.ServeHTTP(recorder, req)
|
||||||
|
|
||||||
return recorder.Code, time.Since(began)
|
return recorder.Code, time.Since(began)
|
||||||
}
|
}
|
||||||
|
|
||||||
// banClient bans client at now.
|
// banClient bans client at now.
|
||||||
func banClient(_ *testing.T, server *waf.Server, now time.Time) {
|
func banClient(_ *testing.T, server *wafAndParts, now time.Time) {
|
||||||
server.Ledger.Ban(
|
server.Ledger.Ban(
|
||||||
netip.MustParsePrefix(client+"/32"),
|
netip.MustParsePrefix(client+"/32"),
|
||||||
now,
|
now,
|
||||||
@@ -694,7 +732,7 @@ func banClient(_ *testing.T, server *waf.Server, now time.Time) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// countClient counts one request from client at now for the rate limits.
|
// countClient counts one request from client at now for the rate limits.
|
||||||
func countClient(_ *testing.T, server *waf.Server, now time.Time) {
|
func countClient(_ *testing.T, server *wafAndParts, now time.Time) {
|
||||||
server.Limiter.Count(
|
server.Limiter.Count(
|
||||||
netip.MustParsePrefix(client+"/32"),
|
netip.MustParsePrefix(client+"/32"),
|
||||||
now,
|
now,
|
||||||
@@ -703,7 +741,7 @@ func countClient(_ *testing.T, server *waf.Server, now time.Time) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// listClient puts client on the blocklist at dropURL.
|
// listClient puts client on the blocklist at dropURL.
|
||||||
func listClient(t *testing.T, server *waf.Server, now time.Time) {
|
func listClient(t *testing.T, server *wafAndParts, now time.Time) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
err := server.Lists.Load([]reputation.List{{
|
err := server.Lists.Load([]reputation.List{{
|
||||||
@@ -715,7 +753,7 @@ func listClient(t *testing.T, server *waf.Server, now time.Time) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// leaveAsItIs prepares nothing.
|
// leaveAsItIs prepares nothing.
|
||||||
func leaveAsItIs(*testing.T, *waf.Server, time.Time) {}
|
func leaveAsItIs(*testing.T, *wafAndParts, time.Time) {}
|
||||||
|
|
||||||
func TestARefusedRequestDoesNotWaitForASlowGeoJS(t *testing.T) {
|
func TestARefusedRequestDoesNotWaitForASlowGeoJS(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
@@ -726,11 +764,11 @@ func TestARefusedRequestDoesNotWaitForASlowGeoJS(t *testing.T) {
|
|||||||
name string
|
name string
|
||||||
env map[string]string
|
env map[string]string
|
||||||
// prepare gets the server ready, at now, before the question.
|
// prepare gets the server ready, at now, before the question.
|
||||||
prepare func(t *testing.T, server *waf.Server, now time.Time)
|
prepare func(t *testing.T, server *wafAndParts, now time.Time)
|
||||||
// cookie is the request's Cookie header.
|
// cookie is the request's Cookie header.
|
||||||
cookie string
|
cookie string
|
||||||
status int
|
status int
|
||||||
action string
|
action requestlog.Action
|
||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "SWW_WAF_DENY_NETS", env: map[string]string{denyNets: client},
|
name: "SWW_WAF_DENY_NETS", env: map[string]string{denyNets: client},
|
||||||
@@ -811,9 +849,9 @@ func TestAQuestionWaitsForASlowGeoJSOnlyAsLongAsTheDefaultSays(
|
|||||||
name string
|
name string
|
||||||
env map[string]string
|
env map[string]string
|
||||||
// prepare gets the server ready, at now, before the question.
|
// prepare gets the server ready, at now, before the question.
|
||||||
prepare func(t *testing.T, server *waf.Server, now time.Time)
|
prepare func(t *testing.T, server *wafAndParts, now time.Time)
|
||||||
status int
|
status int
|
||||||
action string
|
action requestlog.Action
|
||||||
}{
|
}{
|
||||||
{"nothing refuses it", nil, leaveAsItIs, http.StatusOK, requestlog.ActionForward},
|
{"nothing refuses it", nil, leaveAsItIs, http.StatusOK, requestlog.ActionForward},
|
||||||
{
|
{
|
||||||
@@ -877,7 +915,7 @@ func TestWAFAnswersWithinTheTimeLimitWhileEveryOutsideSourceHangs(
|
|||||||
held, _, _ := startHeldGeoJS(t)
|
held, _, _ := startHeldGeoJS(t)
|
||||||
resolver, releaseResolver := startHeldResolver(t)
|
resolver, releaseResolver := startHeldResolver(t)
|
||||||
|
|
||||||
server, out, _ := newWAFWith(t, waf.New,
|
server, out, _ := newWAFWith(t, func(*waf.Params) {},
|
||||||
silentGeoJSURL, silentAbuseIPDBURL, time.Now, map[string]string{
|
silentGeoJSURL, silentAbuseIPDBURL, time.Now, map[string]string{
|
||||||
tagDeny: "country=KP", coreRuleSetMode: block,
|
tagDeny: "country=KP", coreRuleSetMode: block,
|
||||||
dnsblZones: dnsblZone, dnsblResolver: resolver,
|
dnsblZones: dnsblZone, dnsblResolver: resolver,
|
||||||
|
|||||||
+111
-113
@@ -1,69 +1,23 @@
|
|||||||
package waf
|
package waf
|
||||||
|
|
||||||
// The heuristic active_ban, and the code that makes bans, which several
|
// banMaker, which makes the bans of several heuristics. The bans are the ban
|
||||||
// heuristics share. active_ban reads SWW_WAF_MODE; the bans are the ban ledger's.
|
// ledger's.
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
)
|
)
|
||||||
|
|
||||||
// banResponse is a refusal answered with SWW_WAF_BAN_RESPONSE, and logged
|
|
||||||
// with action.
|
|
||||||
func (rq *request) banResponse(action string) *refusal {
|
|
||||||
return &refusal{status: rq.h.config.BanResponse, action: action}
|
|
||||||
}
|
|
||||||
|
|
||||||
// activeBan refuses a client a ban covers: a ban on a netblock the client is
|
|
||||||
// in, in force at now. It notes for the log line when that ban ends. A request
|
|
||||||
// that makes the ban permanent, or in observe mode would have, raises the alert
|
|
||||||
// for it. Its figures are the ban's netblock and when it ends.
|
|
||||||
type activeBan struct{}
|
|
||||||
|
|
||||||
func (activeBan) name() string { return "active_ban" }
|
|
||||||
|
|
||||||
func (activeBan) check(
|
|
||||||
_ context.Context,
|
|
||||||
rq *request,
|
|
||||||
now time.Time,
|
|
||||||
) *verdict {
|
|
||||||
check := rq.h.ledger.Check
|
|
||||||
if rq.h.config.Observe {
|
|
||||||
check = rq.h.ledger.Find // the ban refuses nothing, and stays as it is
|
|
||||||
}
|
|
||||||
|
|
||||||
ban, banned, madePermanent := check(rq.client, now)
|
|
||||||
if banned {
|
|
||||||
rq.line.BanExpires = banExpires(ban)
|
|
||||||
}
|
|
||||||
|
|
||||||
if madePermanent {
|
|
||||||
ban.Expires = time.Time{} // the ban made permanent, which Find leaves as it is
|
|
||||||
rq.alertBan(ban)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !banned {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
return &verdict{
|
|
||||||
action: banAction(ban.Permanent()),
|
|
||||||
figures: map[string]any{
|
|
||||||
figureNetblock: ban.Netblock.String(), "ban_expires": rq.line.BanExpires,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// banAction is the action of a request refused under a ban or for making one:
|
// banAction is the action of a request refused under a ban or for making one:
|
||||||
// permanent says the ban never runs out.
|
// permanent says the ban never runs out.
|
||||||
func banAction(permanent bool) string {
|
func banAction(permanent bool) requestlog.Action {
|
||||||
if permanent {
|
if permanent {
|
||||||
return requestlog.ActionPermanentlyBanned
|
return requestlog.ActionPermanentlyBanned
|
||||||
}
|
}
|
||||||
@@ -81,11 +35,12 @@ func limitFigures(hit ratelimit.Hit) map[string]any {
|
|||||||
|
|
||||||
// countedBytes returns the request's bytes, once it has ended, as the
|
// countedBytes returns the request's bytes, once it has ended, as the
|
||||||
// byte limits and the anomaly thresholds count them: the response's body
|
// byte limits and the anomaly thresholds count them: the response's body
|
||||||
// bytes, the request's, or both, as SWW_WAF_BYTES_COUNTED says.
|
// bytes, the request's, or both, as bytesCounted, SWW_WAF_BYTES_COUNTED,
|
||||||
func (rq *request) countedBytes() int64 {
|
// says.
|
||||||
|
func (rq *request) countedBytes(bytesCounted config.BytesCounted) int64 {
|
||||||
response, request := rq.out.bytes, rq.requestBytes
|
response, request := rq.out.bytes, rq.requestBytes
|
||||||
|
|
||||||
switch rq.h.config.BytesCounted {
|
switch bytesCounted {
|
||||||
case config.BytesCountedResponse:
|
case config.BytesCountedResponse:
|
||||||
return response
|
return response
|
||||||
case config.BytesCountedRequest:
|
case config.BytesCountedRequest:
|
||||||
@@ -96,7 +51,30 @@ func (rq *request) countedBytes() int64 {
|
|||||||
return response + request
|
return response + request
|
||||||
}
|
}
|
||||||
|
|
||||||
// banForLimit bans the client's netblock at now for a broken limit, the
|
// banMaker makes the bans of the heuristics that ban a client, in the ban
|
||||||
|
// ledger, and raises the alert for each. In observe mode it makes no ban, and
|
||||||
|
// raises the alert for the ban it would have made.
|
||||||
|
type banMaker struct {
|
||||||
|
ledger *bans.Ledger
|
||||||
|
// limiter is the table of clients the rate limits count. A ban for a limit
|
||||||
|
// sets the client's counters back to zero, and a ban's notes give how many
|
||||||
|
// requests its netblock has sent.
|
||||||
|
limiter *ratelimit.Limiter
|
||||||
|
metrics *metrics.Metrics
|
||||||
|
alerts *alerts.Queue
|
||||||
|
// observe is true in observe mode (SWW_WAF_MODE).
|
||||||
|
observe bool
|
||||||
|
// banResponse is SWW_WAF_BAN_RESPONSE: the status of a request refused
|
||||||
|
// under a ban or for a clear sign of attack, which the notes of a ban for
|
||||||
|
// a clear sign of attack give.
|
||||||
|
banResponse int
|
||||||
|
// banScopeV4Prefix is SWW_WAF_BAN_SCOPE_V4_PREFIX and ipv6GroupPrefix
|
||||||
|
// SWW_WAF_IPV6_GROUP_PREFIX, which set the netblock a ban covers.
|
||||||
|
banScopeV4Prefix int
|
||||||
|
ipv6GroupPrefix int
|
||||||
|
}
|
||||||
|
|
||||||
|
// banForLimit bans the netblock of rq's client at now for a broken limit, the
|
||||||
// one hit names, notes the offence for the log line and counts the hit in
|
// one hit names, notes the offence for the log line and counts the hit in
|
||||||
// the metrics. status is what the client was sent, or is sent:
|
// the metrics. status is what the client was sent, or is sent:
|
||||||
// SWW_WAF_BAN_RESPONSE for a request over a rate limit, what the report says for
|
// SWW_WAF_BAN_RESPONSE for a request over a rate limit, what the report says for
|
||||||
@@ -105,7 +83,8 @@ func (rq *request) countedBytes() int64 {
|
|||||||
// rate limit or a byte limit; the refusal limit is not lowered. The ban sets
|
// rate limit or a byte limit; the refusal limit is not lowered. The ban sets
|
||||||
// the client's counters back to zero. In observe mode, banClient says what
|
// the client's counters back to zero. In observe mode, banClient says what
|
||||||
// happens instead.
|
// happens instead.
|
||||||
func (rq *request) banForLimit(
|
func (b banMaker) banForLimit(
|
||||||
|
rq *request,
|
||||||
now time.Time,
|
now time.Time,
|
||||||
hit ratelimit.Hit,
|
hit ratelimit.Hit,
|
||||||
status int,
|
status int,
|
||||||
@@ -115,12 +94,14 @@ func (rq *request) banForLimit(
|
|||||||
rq.line.LimitHit = hit.Window + "_bytes" // as counts names the byte totals
|
rq.line.LimitHit = hit.Window + "_bytes" // as counts names the byte totals
|
||||||
case ratelimit.KindRefusals:
|
case ratelimit.KindRefusals:
|
||||||
rq.line.LimitHit = requestlog.LimitHitMinuteRefusals
|
rq.line.LimitHit = requestlog.LimitHitMinuteRefusals
|
||||||
default:
|
case ratelimit.KindRequests:
|
||||||
rq.line.LimitHit = hit.Window
|
rq.line.LimitHit = hit.Window
|
||||||
}
|
}
|
||||||
|
|
||||||
|
rq.brokeLimit = true
|
||||||
rq.line.Offence = requestlog.OffenceLimit
|
rq.line.Offence = requestlog.OffenceLimit
|
||||||
rq.h.metrics.LimitHit(hit)
|
|
||||||
|
b.metrics.LimitHit(hit)
|
||||||
|
|
||||||
notes := bans.Notes{
|
notes := bans.Notes{
|
||||||
Kind: hit.Kind,
|
Kind: hit.Kind,
|
||||||
@@ -134,94 +115,108 @@ func (rq *request) banForLimit(
|
|||||||
notes.LimitPercent, notes.LimitPercentSetting = rq.rateLimitPercent.logged()
|
notes.LimitPercent, notes.LimitPercentSetting = rq.rateLimitPercent.logged()
|
||||||
case ratelimit.KindBytes:
|
case ratelimit.KindBytes:
|
||||||
notes.LimitPercent, notes.LimitPercentSetting = rq.bytesLimitPercent.logged()
|
notes.LimitPercent, notes.LimitPercentSetting = rq.bytesLimitPercent.logged()
|
||||||
|
case ratelimit.KindRefusals: // the refusal limit is not lowered
|
||||||
}
|
}
|
||||||
|
|
||||||
rq.banClient(
|
b.banClient(
|
||||||
|
rq,
|
||||||
now,
|
now,
|
||||||
status,
|
status,
|
||||||
bans.Grounds{Cause: bans.CauseLimit, Notes: notes},
|
bans.Grounds{Cause: bans.CauseLimit, Notes: notes},
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
// banForAttack bans the client's netblock at now for a clear sign of
|
// banForAttack bans the netblock of rq's client at now for a clear sign of
|
||||||
// attack, which notes name: the ban rule that matched, or the trap path
|
// attack, which notes name: the ban rule that matched, or the trap path
|
||||||
// asked for. It returns the action for that ban. In observe mode, banClient
|
// asked for. It returns the action for that ban. In observe mode, banClient
|
||||||
// says what happens instead.
|
// says what happens instead.
|
||||||
func (rq *request) banForAttack(now time.Time, notes bans.Notes) string {
|
func (b banMaker) banForAttack(
|
||||||
return rq.banClient(
|
rq *request, now time.Time, notes bans.Notes,
|
||||||
|
) requestlog.Action {
|
||||||
|
return b.banClient(
|
||||||
|
rq,
|
||||||
now,
|
now,
|
||||||
rq.h.config.BanResponse,
|
b.banResponse,
|
||||||
bans.Grounds{Cause: bans.CauseAttack, Notes: notes},
|
bans.Grounds{Cause: bans.CauseAttack, Notes: notes},
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
// banClient bans the client's netblock at now on grounds, and returns the
|
// banClient bans the netblock of rq's client at now on grounds, and returns
|
||||||
// action for that ban. It adds to the notes what the request tells of the
|
// the action for that ban. It adds to the notes what the request tells of the
|
||||||
// client, with status, what the client was sent. A ban for a broken limit
|
// client, with status, what the client was sent. A ban for a broken limit
|
||||||
// sets the client's counters back to zero.
|
// sets the client's counters back to zero.
|
||||||
//
|
//
|
||||||
// In observe mode it makes no ban and sets nothing back. It raises the
|
// In observe mode it makes no ban and sets nothing back. It raises the
|
||||||
// alert for the ban it would have made, if that alert would be sent.
|
// alert for the ban it would have made, if that alert would be sent.
|
||||||
func (rq *request) banClient(
|
func (b banMaker) banClient(
|
||||||
|
rq *request,
|
||||||
now time.Time,
|
now time.Time,
|
||||||
status int,
|
status int,
|
||||||
grounds bans.Grounds,
|
grounds bans.Grounds,
|
||||||
) string {
|
) requestlog.Action {
|
||||||
netblock := banNetblock(rq.h.config, rq.client)
|
netblock := banNetblock(b.banScopeV4Prefix, b.ipv6GroupPrefix, rq.client)
|
||||||
|
|
||||||
if rq.h.config.Observe {
|
if b.observe {
|
||||||
if !rq.wouldAlertBan(netblock, now, grounds.Cause) {
|
if !b.wouldAlertBan(netblock, now, grounds.Cause) {
|
||||||
return banAction(
|
return banAction(
|
||||||
rq.h.ledger.WouldBePermanent(netblock, now, grounds.Cause),
|
b.ledger.WouldBePermanent(netblock, now, grounds.Cause),
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
grounds.Notes = rq.withClientNotes(
|
grounds.Notes = b.withClientNotes(
|
||||||
|
rq,
|
||||||
grounds.Notes,
|
grounds.Notes,
|
||||||
now,
|
now,
|
||||||
status,
|
status,
|
||||||
netblock,
|
netblock,
|
||||||
)
|
)
|
||||||
|
|
||||||
ban, wouldBan := rq.h.ledger.WouldBan(netblock, now, grounds)
|
ban, wouldBan := b.ledger.WouldBan(netblock, now, grounds)
|
||||||
if wouldBan {
|
if wouldBan {
|
||||||
rq.alertBan(ban)
|
b.alertBan(rq, ban)
|
||||||
}
|
}
|
||||||
|
|
||||||
return banAction(ban.Permanent())
|
return banAction(ban.Permanent())
|
||||||
}
|
}
|
||||||
|
|
||||||
grounds.Notes = rq.withClientNotes(grounds.Notes, now, status, netblock)
|
grounds.Notes = b.withClientNotes(
|
||||||
|
rq,
|
||||||
|
grounds.Notes,
|
||||||
|
now,
|
||||||
|
status,
|
||||||
|
netblock,
|
||||||
|
)
|
||||||
|
|
||||||
ban, made := rq.h.ledger.Ban(netblock, now, grounds)
|
ban, made := b.ledger.Ban(netblock, now, grounds)
|
||||||
if grounds.Cause == bans.CauseLimit {
|
if grounds.Cause == bans.CauseLimit {
|
||||||
rq.h.limiter.Reset(clientGroup(rq.h.config, rq.client))
|
b.limiter.Reset(clientGroup(b.ipv6GroupPrefix, rq.client))
|
||||||
}
|
}
|
||||||
|
|
||||||
rq.line.BanExpires = banExpires(ban)
|
rq.line.BanExpires = banExpires(ban)
|
||||||
|
|
||||||
if made {
|
if made {
|
||||||
rq.alertBan(ban)
|
b.alertBan(rq, ban)
|
||||||
}
|
}
|
||||||
|
|
||||||
return banAction(ban.Permanent())
|
return banAction(ban.Permanent())
|
||||||
}
|
}
|
||||||
|
|
||||||
// withClientNotes is notes with what the request tells of the client
|
// withClientNotes is notes with what rq tells of its client added: its AS,
|
||||||
// added: its AS, country and reputation, the request at now with status,
|
// country and reputation, the request at now with status, and how many
|
||||||
// and how many requests netblock, the client's, has sent.
|
// requests netblock, the client's, has sent.
|
||||||
func (rq *request) withClientNotes(
|
func (b banMaker) withClientNotes(
|
||||||
|
rq *request,
|
||||||
notes bans.Notes,
|
notes bans.Notes,
|
||||||
now time.Time,
|
now time.Time,
|
||||||
status int,
|
status int,
|
||||||
netblock netip.Prefix,
|
netblock netip.Prefix,
|
||||||
) bans.Notes {
|
) bans.Notes {
|
||||||
notes.ASN = rq.line.ASN
|
notes.ASN = rq.lookupAnswer.ASN
|
||||||
notes.ASName = rq.line.ASName
|
notes.ASName = rq.lookupAnswer.ASName
|
||||||
notes.Country = rq.line.Country
|
notes.Country = rq.lookupAnswer.Country
|
||||||
notes.Reputation = rq.reputation
|
notes.Reputation = rq.reputation
|
||||||
notes.Request = rq.noted(now, status)
|
notes.Request = rq.noted(now, status)
|
||||||
notes.Requests = rq.netblockRequests(netblock)
|
notes.Requests = b.netblockRequests(netblock)
|
||||||
|
|
||||||
return notes
|
return notes
|
||||||
}
|
}
|
||||||
@@ -231,36 +226,37 @@ func (rq *request) withClientNotes(
|
|||||||
// have made is worked out only then, at most once per
|
// have made is worked out only then, at most once per
|
||||||
// SWW_WAF_ALERT_COOLDOWN and never with no webhook set: its notes count the
|
// SWW_WAF_ALERT_COOLDOWN and never with no webhook set: its notes count the
|
||||||
// netblock's requests, which can mean going through every client.
|
// netblock's requests, which can mean going through every client.
|
||||||
func (rq *request) wouldAlertBan(
|
func (b banMaker) wouldAlertBan(
|
||||||
netblock netip.Prefix, now time.Time, cause string,
|
netblock netip.Prefix, now time.Time, cause bans.Cause,
|
||||||
) bool {
|
) bool {
|
||||||
event := alerts.EventBan
|
event := alerts.EventBan
|
||||||
if rq.h.ledger.WouldBePermanent(netblock, now, cause) {
|
if b.ledger.WouldBePermanent(netblock, now, cause) {
|
||||||
event = alerts.EventPermanentBan
|
event = alerts.EventPermanentBan
|
||||||
}
|
}
|
||||||
|
|
||||||
return rq.h.alerts.WouldSend(event, netblock)
|
return b.alerts.WouldSend(event, netblock)
|
||||||
}
|
}
|
||||||
|
|
||||||
// alertBan raises the alert for ban, which the request made, or made
|
// alertBan raises the alert for ban, which rq made, or made permanent:
|
||||||
// permanent: permanent_ban for a permanent ban, ban for another. Its
|
// permanent_ban for a permanent ban, ban for another. Its detail gives the
|
||||||
// detail gives the ban's cause, when it ends, and its notes, and in
|
// ban's cause, when it ends, and its notes, and in observe mode, where ban is
|
||||||
// observe mode, where ban is the ban that would have been made, or made
|
// the ban that would have been made, or made permanent, mode, observe.
|
||||||
// permanent, mode, observe.
|
func (b banMaker) alertBan(rq *request, ban bans.Ban) {
|
||||||
func (rq *request) alertBan(ban bans.Ban) {
|
|
||||||
event := alerts.EventBan
|
event := alerts.EventBan
|
||||||
if ban.Permanent() {
|
if ban.Permanent() {
|
||||||
event = alerts.EventPermanentBan
|
event = alerts.EventPermanentBan
|
||||||
}
|
}
|
||||||
|
|
||||||
detail := map[string]any{
|
detail := map[string]any{
|
||||||
"cause": ban.Cause, "ban_expires": banExpires(ban), "notes": ban.Notes,
|
"cause": string(ban.Cause),
|
||||||
|
"ban_expires": banExpires(ban),
|
||||||
|
"notes": ban.Notes,
|
||||||
}
|
}
|
||||||
if rq.h.config.Observe {
|
if b.observe {
|
||||||
detail["mode"] = "observe"
|
detail["mode"] = "observe"
|
||||||
}
|
}
|
||||||
|
|
||||||
rq.h.alerts.Raise(alerts.Alert{
|
b.alerts.Raise(alerts.Alert{
|
||||||
Event: event,
|
Event: event,
|
||||||
Host: rq.site,
|
Host: rq.site,
|
||||||
Client: rq.client,
|
Client: rq.client,
|
||||||
@@ -273,37 +269,39 @@ func (rq *request) alertBan(ban bans.Ban) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// netblockRequests is how many requests netblock has sent since it was
|
||||||
|
// first seen, the request being banned for included: the histories count it
|
||||||
|
// only once it has ended.
|
||||||
|
func (b banMaker) netblockRequests(netblock netip.Prefix) int64 {
|
||||||
|
return b.limiter.Requests(netblock) + 1
|
||||||
|
}
|
||||||
|
|
||||||
// noted is the request, at now, with status, what the client was sent, or
|
// noted is the request, at now, with status, what the client was sent, or
|
||||||
// in observe mode would have been, as the notes of the ban it makes keep
|
// in observe mode would have been, as the notes of the ban it makes keep
|
||||||
// it.
|
// it.
|
||||||
func (rq *request) noted(now time.Time, status int) bans.Request {
|
func (rq *request) noted(now time.Time, status int) bans.Request {
|
||||||
return bans.Request{
|
return bans.Request{
|
||||||
Time: now,
|
Time: now,
|
||||||
Method: rq.line.Method,
|
Method: rq.method,
|
||||||
Host: rq.line.Host,
|
Host: rq.site,
|
||||||
Path: rq.requestURI,
|
Path: rq.requestURI,
|
||||||
Status: status,
|
Status: status,
|
||||||
UserAgent: rq.line.UserAgent,
|
UserAgent: rq.userAgent,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// netblockRequests is how many requests netblock has sent since it was
|
|
||||||
// first seen, this one included: the histories count it only once it has
|
|
||||||
// ended.
|
|
||||||
func (rq *request) netblockRequests(netblock netip.Prefix) int64 {
|
|
||||||
return rq.h.limiter.Requests(netblock) + 1
|
|
||||||
}
|
|
||||||
|
|
||||||
// banNetblock is the netblock a ban on client covers: its IPv4 address,
|
// banNetblock is the netblock a ban on client covers: its IPv4 address,
|
||||||
// widened to SWW_WAF_BAN_SCOPE_V4_PREFIX, or the IPv6 group clientGroup
|
// widened to banScopeV4Prefix, SWW_WAF_BAN_SCOPE_V4_PREFIX, or the IPv6 group
|
||||||
// counts it in.
|
// clientGroup counts it in, of length ipv6GroupPrefix.
|
||||||
func banNetblock(cfg *config.WAF, client netip.Addr) netip.Prefix {
|
func banNetblock(
|
||||||
|
banScopeV4Prefix, ipv6GroupPrefix int, client netip.Addr,
|
||||||
|
) netip.Prefix {
|
||||||
addr := client.Unmap()
|
addr := client.Unmap()
|
||||||
if addr.Is4() {
|
if addr.Is4() {
|
||||||
return netip.PrefixFrom(addr, cfg.BanScopeV4Prefix).Masked()
|
return netip.PrefixFrom(addr, banScopeV4Prefix).Masked()
|
||||||
}
|
}
|
||||||
|
|
||||||
return clientGroup(cfg, addr)
|
return clientGroup(ipv6GroupPrefix, addr)
|
||||||
}
|
}
|
||||||
|
|
||||||
// banExpires is when ban ends, as the log line gives it: a time, or
|
// banExpires is when ban ends, as the log line gives it: a time, or
|
||||||
|
|||||||
@@ -16,7 +16,6 @@ import (
|
|||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -466,7 +465,7 @@ func (c *clock) advance(d time.Duration) {
|
|||||||
// window.
|
// window.
|
||||||
func startWithClock(
|
func startWithClock(
|
||||||
t *testing.T, geojsURL string, env map[string]string,
|
t *testing.T, geojsURL string, env map[string]string,
|
||||||
) (*sender, *clock, *waf.Server) {
|
) (*sender, *clock, *wafAndParts) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
|
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
|
||||||
@@ -504,7 +503,9 @@ type sender struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// get sends a GET request for / from the client at from.
|
// get sends a GET request for / from the client at from.
|
||||||
func (s *sender) get(from string, status int, action string) logLine {
|
func (s *sender) get(
|
||||||
|
from string, status int, action requestlog.Action,
|
||||||
|
) logLine {
|
||||||
s.t.Helper()
|
s.t.Helper()
|
||||||
|
|
||||||
return s.request(from, "/", status, action)
|
return s.request(from, "/", status, action)
|
||||||
@@ -517,7 +518,7 @@ func (s *sender) get(from string, status int, action string) logLine {
|
|||||||
func (s *sender) request(
|
func (s *sender) request(
|
||||||
from, path string,
|
from, path string,
|
||||||
status int,
|
status int,
|
||||||
action string,
|
action requestlog.Action,
|
||||||
) logLine {
|
) logLine {
|
||||||
s.t.Helper()
|
s.t.Helper()
|
||||||
|
|
||||||
@@ -527,7 +528,7 @@ func (s *sender) request(
|
|||||||
// requestWithHeader is request with header, such as "Authorization:
|
// requestWithHeader is request with header, such as "Authorization:
|
||||||
// Bearer x", added to the request unless it is "".
|
// Bearer x", added to the request unless it is "".
|
||||||
func (s *sender) requestWithHeader(
|
func (s *sender) requestWithHeader(
|
||||||
from, path, header string, status int, action string,
|
from, path, header string, status int, action requestlog.Action,
|
||||||
) logLine {
|
) logLine {
|
||||||
s.t.Helper()
|
s.t.Helper()
|
||||||
|
|
||||||
@@ -553,7 +554,9 @@ func (s *sender) requestWithHeader(
|
|||||||
// the Core Rule Set past any shorter wait. go test's -timeout ends a test
|
// the Core Rule Set past any shorter wait. go test's -timeout ends a test
|
||||||
// whose answer never comes.
|
// whose answer never comes.
|
||||||
func (s *sender) requestWithBody(
|
func (s *sender) requestWithBody(
|
||||||
method, from, path, header, body string, status int, action string,
|
method, from, path, header, body string,
|
||||||
|
status int,
|
||||||
|
action requestlog.Action,
|
||||||
) (logLine, answer) {
|
) (logLine, answer) {
|
||||||
s.t.Helper()
|
s.t.Helper()
|
||||||
|
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ package waf
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"net/http"
|
||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -54,7 +55,7 @@ func (rq *request) runBesideLookup(
|
|||||||
) *verdict {
|
) *verdict {
|
||||||
var running atomic.Pointer[string]
|
var running atomic.Pointer[string]
|
||||||
|
|
||||||
running.Store(new(trapPaths{}.name()))
|
running.Store(new(rq.h.trapPaths.name()))
|
||||||
|
|
||||||
found := make(chan *besideLookupFindings, 1)
|
found := make(chan *besideLookupFindings, 1)
|
||||||
|
|
||||||
@@ -92,7 +93,7 @@ func (rq *request) runBesideLookup(
|
|||||||
return v
|
return v
|
||||||
}
|
}
|
||||||
|
|
||||||
return rq.runHeuristics(ctx, afterBesideLookup(), rq.h.now())
|
return rq.runHeuristics(ctx, rq.h.afterBesideLookup, rq.h.now())
|
||||||
}
|
}
|
||||||
|
|
||||||
// findBesideLookup does the work of the heuristics of besideLookup on the
|
// findBesideLookup does the work of the heuristics of besideLookup on the
|
||||||
@@ -105,39 +106,39 @@ func (rq *request) findBesideLookup(
|
|||||||
) *besideLookupFindings {
|
) *besideLookupFindings {
|
||||||
findings := &besideLookupFindings{}
|
findings := &besideLookupFindings{}
|
||||||
|
|
||||||
findings.trapPath = rq.trapPath()
|
findings.trapPath = rq.h.trapPaths.trapPath(rq)
|
||||||
if findings.trapPath != "" {
|
if findings.trapPath != "" {
|
||||||
findings.refuses = true
|
findings.refuses = true
|
||||||
|
|
||||||
return findings
|
return findings
|
||||||
}
|
}
|
||||||
|
|
||||||
running.Store(new(ruleFiles{}.name()))
|
running.Store(new(rq.h.ruleFiles.name()))
|
||||||
|
|
||||||
findings.rules = rq.h.rules.Match(rq.in)
|
findings.rules = rq.h.ruleFiles.rules.Match(rq.in)
|
||||||
if refusedByRule(findings.rules) {
|
if refusedByRule(findings.rules) {
|
||||||
findings.refuses = true
|
findings.refuses = true
|
||||||
|
|
||||||
return findings
|
return findings
|
||||||
}
|
}
|
||||||
|
|
||||||
running.Store(new(coreRuleSet{}.name()))
|
coreRuleSet := rq.h.coreRuleSet
|
||||||
|
running.Store(new(coreRuleSet.name()))
|
||||||
|
|
||||||
cfg := rq.h.config
|
if coreRuleSet.mode == config.CoreRuleSetModeOff ||
|
||||||
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff ||
|
pathExempt(rq.in, coreRuleSet.exemptPaths) {
|
||||||
pathExempt(rq.in, cfg.CoreRuleSetExemptPaths) {
|
|
||||||
return findings
|
return findings
|
||||||
}
|
}
|
||||||
|
|
||||||
start := time.Now()
|
start := time.Now()
|
||||||
// A question's body has been read into memory, so that reading it here
|
// A question's body has been read into memory, so that reading it here
|
||||||
// cannot fail.
|
// cannot fail.
|
||||||
findings.inspection, findings.read, _ = rq.h.coreRuleSet.Inspect(
|
findings.inspection, findings.read, _ = coreRuleSet.inspector.Inspect(
|
||||||
rq.in, rq.client, rq.in.Body,
|
rq.in, rq.client, rq.in.Body,
|
||||||
)
|
)
|
||||||
findings.inspectionTime = time.Since(start)
|
findings.inspectionTime = time.Since(start)
|
||||||
findings.inspected = true
|
findings.inspected = true
|
||||||
findings.refuses = rq.refusedByCoreRuleSet(findings.inspection)
|
findings.refuses = coreRuleSet.refusedByCoreRuleSet(findings.inspection)
|
||||||
|
|
||||||
return findings
|
return findings
|
||||||
}
|
}
|
||||||
@@ -151,7 +152,7 @@ func (rq *request) applyFindings(
|
|||||||
) *verdict {
|
) *verdict {
|
||||||
rq.findings = findings
|
rq.findings = findings
|
||||||
|
|
||||||
return rq.runHeuristics(ctx, besideLookup(), rq.h.now())
|
return rq.runHeuristics(ctx, rq.h.besideLookup, rq.h.now())
|
||||||
}
|
}
|
||||||
|
|
||||||
// timedOut refuses the request for the heuristics of besideLookup not having
|
// timedOut refuses the request for the heuristics of besideLookup not having
|
||||||
@@ -161,6 +162,7 @@ func (rq *request) applyFindings(
|
|||||||
func (rq *request) timedOut(running string) *verdict {
|
func (rq *request) timedOut(running string) *verdict {
|
||||||
v := &verdict{
|
v := &verdict{
|
||||||
action: requestlog.ActionChecksTimedOut,
|
action: requestlog.ActionChecksTimedOut,
|
||||||
|
status: http.StatusServiceUnavailable,
|
||||||
figures: map[string]any{
|
figures: map[string]any{
|
||||||
"time_limit_ms": rq.h.besideLookupLimit.Milliseconds(),
|
"time_limit_ms": rq.h.besideLookupLimit.Milliseconds(),
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -62,7 +62,7 @@ func coreRuleSetMatch() coreruleset.Result {
|
|||||||
// at its default but for env. No request to it may use the network.
|
// at its default but for env. No request to it may use the network.
|
||||||
func newWAFHoldingTheCoreRuleSet(
|
func newWAFHoldingTheCoreRuleSet(
|
||||||
t *testing.T, crs *heldCoreRuleSet, env map[string]string,
|
t *testing.T, crs *heldCoreRuleSet, env map[string]string,
|
||||||
) (*waf.Server, *output, *alerts.Queue) {
|
) (*wafAndParts, *output, *alerts.Queue) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
settings := map[string]string{
|
settings := map[string]string{
|
||||||
@@ -71,11 +71,9 @@ func newWAFHoldingTheCoreRuleSet(
|
|||||||
}
|
}
|
||||||
maps.Copy(settings, env)
|
maps.Copy(settings, env)
|
||||||
|
|
||||||
server, out, queue := newWAFWith(t, func(params waf.Params) *waf.Server {
|
server, out, queue := newWAFWith(t, func(params *waf.Params) {
|
||||||
params.CoreRuleSet = crs
|
params.CoreRuleSet = crs
|
||||||
params.BesideLookupTimeLimit = waf.BesideLookupTimeLimit
|
params.BesideLookupTimeLimit = waf.BesideLookupTimeLimit
|
||||||
|
|
||||||
return waf.New(params)
|
|
||||||
}, silentGeoJSURL, abuseIPDBURL, time.Now, settings)
|
}, silentGeoJSURL, abuseIPDBURL, time.Now, settings)
|
||||||
|
|
||||||
return server, out, queue
|
return server, out, queue
|
||||||
@@ -105,7 +103,7 @@ func wantAnswer(
|
|||||||
// left no trace: line has none of its fields, and server has counted no match
|
// left no trace: line has none of its fields, and server has counted no match
|
||||||
// of it and raised no alert for one.
|
// of it and raised no alert for one.
|
||||||
func wantNoCoreRuleSetTrace(
|
func wantNoCoreRuleSetTrace(
|
||||||
t *testing.T, server *waf.Server, queue *alerts.Queue, line logLine,
|
t *testing.T, server *wafAndParts, queue *alerts.Queue, line logLine,
|
||||||
) {
|
) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
@@ -198,7 +196,7 @@ type firstToRefuse struct {
|
|||||||
env map[string]string
|
env map[string]string
|
||||||
inspection time.Duration
|
inspection time.Duration
|
||||||
took time.Duration
|
took time.Duration
|
||||||
action string
|
action requestlog.Action
|
||||||
}
|
}
|
||||||
|
|
||||||
// firstToRefuseCases are the cases of
|
// firstToRefuseCases are the cases of
|
||||||
|
|||||||
+38
-17
@@ -2,6 +2,7 @@ package waf
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
)
|
)
|
||||||
|
|
||||||
// whole is the percentage of each limit a client gets when no biased
|
// whole is the percentage of each limit a client gets when no biased
|
||||||
@@ -29,6 +30,25 @@ func biasedThresholdsSet(cfg *config.WAF) bool {
|
|||||||
cfg.ASNLimitPercentURL != ""
|
cfg.ASNLimitPercentURL != ""
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// biasedThresholds are the settings and the lists by which limitPercentages
|
||||||
|
// lowers a client's limits.
|
||||||
|
type biasedThresholds struct {
|
||||||
|
// lists hold the file SWW_WAF_ASN_LIMIT_PERCENT_URL names.
|
||||||
|
lists *reputation.Lists
|
||||||
|
tagLimitPercent map[string]int64 // SWW_WAF_TAG_LIMIT_PERCENT
|
||||||
|
asnBytesLimitPercent map[string]int64 // SWW_WAF_ASN_BYTES_LIMIT_PERCENT
|
||||||
|
countryBytesLimitPercent map[string]int64 // SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT
|
||||||
|
unknownCountryLimitPercent int64 // SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT
|
||||||
|
// blocklistAction and blocklistLimitPercent are SWW_WAF_BLOCKLIST_ACTION
|
||||||
|
// and its percentage.
|
||||||
|
blocklistAction config.ReputationAction
|
||||||
|
blocklistLimitPercent int64
|
||||||
|
// dnsblAbuseIPDBAction and dnsblAbuseIPDBLimitPercent are
|
||||||
|
// SWW_WAF_DNSBL_ABUSEIPDB_ACTION and its percentage.
|
||||||
|
dnsblAbuseIPDBAction config.ReputationAction
|
||||||
|
dnsblAbuseIPDBLimitPercent int64
|
||||||
|
}
|
||||||
|
|
||||||
// limitPercentages returns the client's limit percentages, for the rate
|
// limitPercentages returns the client's limit percentages, for the rate
|
||||||
// limits and for the byte limits, by its tags, its AS number and country
|
// limits and for the byte limits, by its tags, its AS number and country
|
||||||
// as looked up, each "" when unknown, and the blocklists, DNSBL zones and
|
// as looked up, each "" when unknown, and the blocklists, DNSBL zones and
|
||||||
@@ -45,37 +65,38 @@ func biasedThresholdsSet(cfg *config.WAF) bool {
|
|||||||
// the byte limits, SWW_WAF_ASN_BYTES_LIMIT_PERCENT and
|
// the byte limits, SWW_WAF_ASN_BYTES_LIMIT_PERCENT and
|
||||||
// SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT take the place of the first three for
|
// SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT take the place of the first three for
|
||||||
// an AS number or a country they list.
|
// an AS number or a country they list.
|
||||||
func (rq *request) limitPercentages() (percentage, percentage) {
|
func (b biasedThresholds) limitPercentages(
|
||||||
cfg := rq.h.config
|
rq *request,
|
||||||
asn, country := rq.line.ASN, rq.line.Country
|
) (percentage, percentage) {
|
||||||
|
asn, country := rq.lookupAnswer.ASN, rq.lookupAnswer.Country
|
||||||
|
|
||||||
unknown := percentage{percent: whole}
|
unknown := percentage{percent: whole}
|
||||||
if country == "" && !rq.skippedLookupWait {
|
if country == "" && !rq.skippedLookupWait {
|
||||||
unknown = percentage{
|
unknown = percentage{
|
||||||
cfg.UnknownCountryLimitPercent,
|
b.unknownCountryLimitPercent,
|
||||||
config.UnknownCountryLimitPercentSetting,
|
config.UnknownCountryLimitPercentSetting,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fetched := percentage{percent: whole}
|
fetched := percentage{percent: whole}
|
||||||
if percent, listed := rq.h.lists.ASNLimitPercent(asn); listed {
|
if percent, listed := b.lists.ASNLimitPercent(asn); listed {
|
||||||
fetched = percentage{percent, config.ASNLimitPercentURLSetting}
|
fetched = percentage{percent, config.ASNLimitPercentURLSetting}
|
||||||
}
|
}
|
||||||
|
|
||||||
blocklisted := percentage{percent: whole}
|
blocklisted := percentage{percent: whole}
|
||||||
if rq.blocklisted &&
|
if rq.blocklisted &&
|
||||||
cfg.BlocklistAction == config.ReputationActionLimit {
|
b.blocklistAction == config.ReputationActionLimit {
|
||||||
blocklisted = percentage{
|
blocklisted = percentage{
|
||||||
cfg.BlocklistLimitPercent,
|
b.blocklistLimitPercent,
|
||||||
config.BlocklistActionSetting,
|
config.BlocklistActionSetting,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
dnsblAbuseIPDBListed := percentage{percent: whole}
|
dnsblAbuseIPDBListed := percentage{percent: whole}
|
||||||
if (rq.dnsblListed || rq.abuseIPDBHit) &&
|
if (rq.dnsblListed || rq.abuseIPDBHit) &&
|
||||||
cfg.DNSBLAbuseIPDBAction == config.ReputationActionLimit {
|
b.dnsblAbuseIPDBAction == config.ReputationActionLimit {
|
||||||
dnsblAbuseIPDBListed = percentage{
|
dnsblAbuseIPDBListed = percentage{
|
||||||
cfg.DNSBLAbuseIPDBLimitPercent,
|
b.dnsblAbuseIPDBLimitPercent,
|
||||||
config.DNSBLAbuseIPDBActionSetting,
|
config.DNSBLAbuseIPDBActionSetting,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -83,38 +104,38 @@ func (rq *request) limitPercentages() (percentage, percentage) {
|
|||||||
const tagSetting = config.TagLimitPercentSetting
|
const tagSetting = config.TagLimitPercentSetting
|
||||||
|
|
||||||
asnRequests := lowest(
|
asnRequests := lowest(
|
||||||
given(cfg.TagLimitPercent, asnTag(asn), tagSetting),
|
given(b.tagLimitPercent, asnTag(asn), tagSetting),
|
||||||
fetched,
|
fetched,
|
||||||
)
|
)
|
||||||
countryRequests := given(
|
countryRequests := given(
|
||||||
cfg.TagLimitPercent,
|
b.tagLimitPercent,
|
||||||
countryTag(country),
|
countryTag(country),
|
||||||
tagSetting,
|
tagSetting,
|
||||||
)
|
)
|
||||||
|
|
||||||
otherTags := percentage{percent: whole}
|
otherTags := percentage{percent: whole}
|
||||||
|
|
||||||
for _, tag := range rq.line.Tags {
|
for _, tag := range rq.tags {
|
||||||
if tag != asnTag(asn) && tag != countryTag(country) {
|
if tag != asnTag(asn) && tag != countryTag(country) {
|
||||||
otherTags = lowest(
|
otherTags = lowest(
|
||||||
otherTags,
|
otherTags,
|
||||||
given(cfg.TagLimitPercent, tag, tagSetting),
|
given(b.tagLimitPercent, tag, tagSetting),
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
asnBytes, countryBytes := asnRequests, countryRequests
|
asnBytes, countryBytes := asnRequests, countryRequests
|
||||||
if _, listed := cfg.ASNBytesLimitPercent[asn]; listed {
|
if _, listed := b.asnBytesLimitPercent[asn]; listed {
|
||||||
asnBytes = given(
|
asnBytes = given(
|
||||||
cfg.ASNBytesLimitPercent,
|
b.asnBytesLimitPercent,
|
||||||
asn,
|
asn,
|
||||||
config.ASNBytesLimitPercentSetting,
|
config.ASNBytesLimitPercentSetting,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
if _, listed := cfg.CountryBytesLimitPercent[country]; listed {
|
if _, listed := b.countryBytesLimitPercent[country]; listed {
|
||||||
countryBytes = given(
|
countryBytes = given(
|
||||||
cfg.CountryBytesLimitPercent,
|
b.countryBytesLimitPercent,
|
||||||
country,
|
country,
|
||||||
config.CountryBytesLimitPercentSetting,
|
config.CountryBytesLimitPercentSetting,
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -18,7 +18,7 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
|
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
|
||||||
)
|
)
|
||||||
|
|
||||||
// The biased thresholds.
|
// The biased thresholds.
|
||||||
@@ -73,7 +73,7 @@ func TestEachBiasedThresholdLowersTheRateLimits(t *testing.T) {
|
|||||||
// Half of 4 requests a minute: the third breaks the limit.
|
// Half of 4 requests a minute: the third breaks the limit.
|
||||||
for _, sent := range []struct {
|
for _, sent := range []struct {
|
||||||
status int
|
status int
|
||||||
action string
|
action requestlog.Action
|
||||||
}{
|
}{
|
||||||
{http.StatusOK, requestlog.ActionForward},
|
{http.StatusOK, requestlog.ActionForward},
|
||||||
{http.StatusOK, requestlog.ActionForward},
|
{http.StatusOK, requestlog.ActionForward},
|
||||||
@@ -452,7 +452,7 @@ func TestBanForALoweredLimitGivesThePercentageInItsNotesAndItsAlert(
|
|||||||
// limit, which is answered with status and logged with action.
|
// limit, which is answered with status and logged with action.
|
||||||
before int
|
before int
|
||||||
status int
|
status int
|
||||||
action string
|
action requestlog.Action
|
||||||
// reason and want are the ban's reason, and its notes' limit
|
// reason and want are the ban's reason, and its notes' limit
|
||||||
// percentage, as percentText gives it.
|
// percentage, as percentText gives it.
|
||||||
reason, want string
|
reason, want string
|
||||||
@@ -529,7 +529,7 @@ func TestBanForALoweredLimitGivesThePercentageInItsNotesAndItsAlert(
|
|||||||
// clock.
|
// clock.
|
||||||
func startWithLookups(
|
func startWithLookups(
|
||||||
t *testing.T, env map[string]string,
|
t *testing.T, env map[string]string,
|
||||||
) (*sender, *waf.Server, *alerts.Queue) {
|
) (*sender, *wafAndParts, *alerts.Queue) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
s, _, server, queue := startWithLookupsAndClock(t, env)
|
s, _, server, queue := startWithLookupsAndClock(t, env)
|
||||||
@@ -544,7 +544,7 @@ func startWithLookups(
|
|||||||
// country, and no other address.
|
// country, and no other address.
|
||||||
func startWithLookupsAndClock(
|
func startWithLookupsAndClock(
|
||||||
t *testing.T, env map[string]string,
|
t *testing.T, env map[string]string,
|
||||||
) (*sender, *clock, *waf.Server, *alerts.Queue) {
|
) (*sender, *clock, *wafAndParts, *alerts.Queue) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
path := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb")
|
path := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb")
|
||||||
@@ -584,15 +584,15 @@ func (s *sender) uploadFrom(from string) logLine {
|
|||||||
// straight to server's handler, without the network, and returns once it is
|
// straight to server's handler, without the network, and returns once it is
|
||||||
// answered.
|
// answered.
|
||||||
func serveFromDE(
|
func serveFromDE(
|
||||||
t *testing.T, server *waf.Server, method string, body io.Reader,
|
t *testing.T, server *wafAndParts, method string, body io.Reader,
|
||||||
) {
|
) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
req := httptest.NewRequestWithContext(t.Context(), method, "/", body)
|
req := httptest.NewRequestWithContext(t.Context(), method, "/", body)
|
||||||
req.Header.Set(waf.ClientHeader, fromDE)
|
req.Header.Set(wafprotocol.ClientHeader, fromDE)
|
||||||
req.Header.Set(requestIDHeader, "the-id")
|
req.Header.Set(requestIDHeader, "the-id")
|
||||||
|
|
||||||
server.Handler.ServeHTTP(httptest.NewRecorder(), req)
|
server.ServeHTTP(httptest.NewRecorder(), req)
|
||||||
}
|
}
|
||||||
|
|
||||||
// wantPercent checks a limit percentage that a log line or a ban's notes
|
// wantPercent checks a limit percentage that a log line or a ban's notes
|
||||||
|
|||||||
+29
-13
@@ -1,47 +1,63 @@
|
|||||||
package waf
|
package waf
|
||||||
|
|
||||||
// The heuristic blocklists. It reads SWW_WAF_BLOCKLIST_ACTION, and the blocklists
|
// The heuristic blocklists. It reads SWW_WAF_BLOCKLIST_ACTION and
|
||||||
// through the reputation lists.
|
// SWW_WAF_BAN_RESPONSE, and the blocklists through the reputation lists.
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
)
|
)
|
||||||
|
|
||||||
// blocklists notes the blocklists that list the client, as noteListed does,
|
// blocklists notes the blocklists that list the client, as noteListed does,
|
||||||
// and refuses the request when SWW_WAF_BLOCKLIST_ACTION is deny. Being limit, it
|
// and refuses the request with SWW_WAF_BAN_RESPONSE when
|
||||||
// lowers the client's limits instead (see limitPercentages), and being log, it
|
// SWW_WAF_BLOCKLIST_ACTION is deny. Being limit, it lowers the client's limits
|
||||||
// does nothing more. Its figure is the blocklists that list the client.
|
// instead (see limitPercentages), and being log, it does nothing more. Its
|
||||||
type blocklists struct{}
|
// figure is the blocklists that list the client.
|
||||||
|
type blocklists struct {
|
||||||
|
lists *reputation.Lists
|
||||||
|
action config.ReputationAction // SWW_WAF_BLOCKLIST_ACTION
|
||||||
|
banResponse int // SWW_WAF_BAN_RESPONSE
|
||||||
|
hitNoter reputationHitNoter
|
||||||
|
}
|
||||||
|
|
||||||
func (blocklists) name() string { return "blocklists" }
|
func (blocklists) name() string { return "blocklists" }
|
||||||
|
|
||||||
func (blocklists) check(
|
func (b blocklists) check(
|
||||||
_ context.Context,
|
_ context.Context,
|
||||||
rq *request,
|
rq *request,
|
||||||
_ time.Time,
|
_ time.Time,
|
||||||
) *verdict {
|
) *verdict {
|
||||||
listedBy := rq.h.lists.ListedBy(rq.client)
|
listedBy := b.lists.ListedBy(rq.client)
|
||||||
rq.blocklisted = len(listedBy) > 0
|
rq.blocklisted = len(listedBy) > 0
|
||||||
rq.noteListed(listedBy, "listed by a blocklist")
|
b.hitNoter.noteListed(rq, listedBy, "listed by a blocklist")
|
||||||
|
|
||||||
if !rq.refusedByBlocklists(listedBy) {
|
if !b.refusedByBlocklists(listedBy) {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
return &verdict{
|
return &verdict{
|
||||||
action: requestlog.ActionClientDenied,
|
action: requestlog.ActionClientDenied,
|
||||||
|
status: b.banResponse,
|
||||||
figures: map[string]any{"lists": listedBy},
|
figures: map[string]any{"lists": listedBy},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// refusesWhateverTheLookupAnswers reports whether blocklists refuses rq, as
|
||||||
|
// refusedByBlocklists says. The lookup's answer plays no part in it.
|
||||||
|
func (b blocklists) refusesWhateverTheLookupAnswers(
|
||||||
|
rq *request,
|
||||||
|
_ time.Time,
|
||||||
|
) bool {
|
||||||
|
return b.refusedByBlocklists(b.lists.ListedBy(rq.client))
|
||||||
|
}
|
||||||
|
|
||||||
// refusedByBlocklists reports whether the heuristic blocklists refuses a
|
// refusedByBlocklists reports whether the heuristic blocklists refuses a
|
||||||
// client, given listedBy, the blocklists that list it: it does when one lists
|
// client, given listedBy, the blocklists that list it: it does when one lists
|
||||||
// it and SWW_WAF_BLOCKLIST_ACTION is deny.
|
// it and SWW_WAF_BLOCKLIST_ACTION is deny.
|
||||||
func (rq *request) refusedByBlocklists(listedBy []string) bool {
|
func (b blocklists) refusedByBlocklists(listedBy []string) bool {
|
||||||
return len(listedBy) > 0 && rq.h.config.BlocklistAction ==
|
return len(listedBy) > 0 && b.action == config.ReputationActionDeny
|
||||||
config.ReputationActionDeny
|
|
||||||
}
|
}
|
||||||
@@ -6,6 +6,9 @@ package waf
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
)
|
)
|
||||||
|
|
||||||
// byteLimits counts the request's bytes, as countedBytes gives them, for the
|
// byteLimits counts the request's bytes, as countedBytes gives them, for the
|
||||||
@@ -16,23 +19,28 @@ import (
|
|||||||
// the client over a byte limit, as its limit percentage for the byte limits
|
// the client over a byte limit, as its limit percentage for the byte limits
|
||||||
// lowers it, break it, which bans the client; the response was passed on whole.
|
// lowers it, break it, which bans the client; the response was passed on whole.
|
||||||
// Its figures are the window, the count and the limit.
|
// Its figures are the window, the count and the limit.
|
||||||
type byteLimits struct{}
|
type byteLimits struct {
|
||||||
|
limiter *ratelimit.Limiter
|
||||||
|
bytesCounted config.BytesCounted // SWW_WAF_BYTES_COUNTED
|
||||||
|
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
|
||||||
|
banMaker banMaker
|
||||||
|
}
|
||||||
|
|
||||||
func (byteLimits) name() string { return "byte_limits" }
|
func (byteLimits) name() string { return "byte_limits" }
|
||||||
|
|
||||||
func (byteLimits) check(
|
func (b byteLimits) check(
|
||||||
_ context.Context,
|
_ context.Context,
|
||||||
rq *request,
|
rq *request,
|
||||||
now time.Time,
|
now time.Time,
|
||||||
) *verdict {
|
) *verdict {
|
||||||
if !rq.counted || rq.line.WouldAction != "" {
|
if !rq.counted || rq.wouldAction != "" {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
counts, hit, over := rq.h.limiter.CountBytes(
|
counts, hit, over := b.limiter.CountBytes(
|
||||||
clientGroup(rq.h.config, rq.client),
|
clientGroup(b.ipv6GroupPrefix, rq.client),
|
||||||
now,
|
now,
|
||||||
rq.countedBytes(),
|
rq.countedBytes(b.bytesCounted),
|
||||||
rq.bytesLimitPercent.percent,
|
rq.bytesLimitPercent.percent,
|
||||||
)
|
)
|
||||||
rq.line.Counts.MinuteBytes = counts.MinuteBytes
|
rq.line.Counts.MinuteBytes = counts.MinuteBytes
|
||||||
@@ -43,7 +51,7 @@ func (byteLimits) check(
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
rq.banForLimit(now, hit, rq.out.status)
|
b.banMaker.banForLimit(rq, now, hit, rq.out.status)
|
||||||
|
|
||||||
return &verdict{figures: limitFigures(hit)}
|
return &verdict{figures: limitFigures(hit)}
|
||||||
}
|
}
|
||||||
@@ -6,7 +6,7 @@ import (
|
|||||||
"net/netip"
|
"net/netip"
|
||||||
"slices"
|
"slices"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
|
||||||
)
|
)
|
||||||
|
|
||||||
// peerAddress is the address of the request's TCP peer.
|
// peerAddress is the address of the request's TCP peer.
|
||||||
@@ -19,14 +19,11 @@ func peerAddress(r *http.Request) netip.Addr {
|
|||||||
return addrPort.Addr().Unmap()
|
return addrPort.Addr().Unmap()
|
||||||
}
|
}
|
||||||
|
|
||||||
// requestIDHeader carries the request's id, from rproxy.
|
|
||||||
const requestIDHeader = "X-Request-ID"
|
|
||||||
|
|
||||||
// requestID is the request's id: the one a question gives, or for any other
|
// requestID is the request's id: the one a question gives, or for any other
|
||||||
// request a new random one.
|
// request a new random one.
|
||||||
func requestID(r *http.Request, isQuestion bool) string {
|
func requestID(r *http.Request, isQuestion bool) string {
|
||||||
if isQuestion {
|
if isQuestion {
|
||||||
return r.Header.Get(requestIDHeader)
|
return r.Header.Get(wafprotocol.RequestIDHeader)
|
||||||
}
|
}
|
||||||
|
|
||||||
return rand.Text()
|
return rand.Text()
|
||||||
@@ -45,12 +42,13 @@ func scheme(r *http.Request, isQuestion bool) string {
|
|||||||
|
|
||||||
// clientGroup is the client a request is counted toward: its IPv4
|
// clientGroup is the client a request is counted toward: its IPv4
|
||||||
// address, or its IPv6 group, the netblock its IPv6 address is in of the
|
// address, or its IPv6 group, the netblock its IPv6 address is in of the
|
||||||
// length SWW_WAF_IPV6_GROUP_PREFIX sets, a /64 by default, since one abuser
|
// length ipv6GroupPrefix, SWW_WAF_IPV6_GROUP_PREFIX, a /64 by default, since
|
||||||
// usually holds a whole /64. An IPv4 address in IPv6 form counts as IPv4.
|
// one abuser usually holds a whole /64. An IPv4 address in IPv6 form counts as
|
||||||
func clientGroup(cfg *config.WAF, addr netip.Addr) netip.Prefix {
|
// IPv4.
|
||||||
|
func clientGroup(ipv6GroupPrefix int, addr netip.Addr) netip.Prefix {
|
||||||
addr = addr.Unmap()
|
addr = addr.Unmap()
|
||||||
if addr.Is6() {
|
if addr.Is6() {
|
||||||
return netip.PrefixFrom(addr, cfg.IPv6GroupPrefix).Masked()
|
return netip.PrefixFrom(addr, ipv6GroupPrefix).Masked()
|
||||||
}
|
}
|
||||||
|
|
||||||
return netip.PrefixFrom(addr, addr.BitLen())
|
return netip.PrefixFrom(addr, addr.BitLen())
|
||||||
|
|||||||
+35
-21
@@ -5,11 +5,13 @@ package waf
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"net/http"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -22,11 +24,22 @@ import (
|
|||||||
// exploitsignature, raises the core_rule_set_match alert, and in block mode
|
// exploitsignature, raises the core_rule_set_match alert, and in block mode
|
||||||
// refuses the request with 403, which is an offence its client's history
|
// refuses the request with 403, which is an offence its client's history
|
||||||
// counts. Its figures are the score and the threshold.
|
// counts. Its figures are the score and the threshold.
|
||||||
type coreRuleSet struct{}
|
type coreRuleSet struct {
|
||||||
|
// inspector is the Core Rule Set, nil while SWW_WAF_CORE_RULE_SET_MODE is
|
||||||
|
// off.
|
||||||
|
inspector CoreRuleSetInspector
|
||||||
|
metrics *metrics.Metrics
|
||||||
|
alerts *alerts.Queue
|
||||||
|
mode config.CoreRuleSetMode // SWW_WAF_CORE_RULE_SET_MODE
|
||||||
|
exemptPaths []string // SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS
|
||||||
|
scoreThreshold int // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD
|
||||||
|
observe bool // SWW_WAF_MODE is observe
|
||||||
|
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
|
||||||
|
}
|
||||||
|
|
||||||
func (coreRuleSet) name() string { return "core_rule_set" }
|
func (coreRuleSet) name() string { return "core_rule_set" }
|
||||||
|
|
||||||
func (coreRuleSet) check(
|
func (c coreRuleSet) check(
|
||||||
_ context.Context,
|
_ context.Context,
|
||||||
rq *request,
|
rq *request,
|
||||||
_ time.Time,
|
_ time.Time,
|
||||||
@@ -36,7 +49,6 @@ func (coreRuleSet) check(
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
cfg := rq.h.config
|
|
||||||
result := findings.inspection
|
result := findings.inspection
|
||||||
rq.requestBytes = int64(len(findings.read))
|
rq.requestBytes = int64(len(findings.read))
|
||||||
|
|
||||||
@@ -47,18 +59,18 @@ func (coreRuleSet) check(
|
|||||||
rq.line.CoreRuleSetScore = &result.Score
|
rq.line.CoreRuleSetScore = &result.Score
|
||||||
|
|
||||||
for _, id := range result.RuleIDs {
|
for _, id := range result.RuleIDs {
|
||||||
rq.h.metrics.CoreRuleSetMatched(string(cfg.CoreRuleSetMode), id)
|
c.metrics.CoreRuleSetMatched(string(c.mode), id)
|
||||||
}
|
}
|
||||||
|
|
||||||
threshold := cfg.CoreRuleSetScoreThreshold
|
threshold := c.scoreThreshold
|
||||||
if threshold == 0 || result.Score < threshold {
|
if threshold == 0 || result.Score < threshold {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
rq.addTag(config.ExploitSignatureTag)
|
rq.addTag(config.ExploitSignatureTag)
|
||||||
rq.alertCoreRuleSetMatch(result)
|
c.alertCoreRuleSetMatch(rq, result)
|
||||||
|
|
||||||
if !rq.refusedByCoreRuleSet(result) {
|
if !c.refusedByCoreRuleSet(result) {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -66,6 +78,7 @@ func (coreRuleSet) check(
|
|||||||
|
|
||||||
return &verdict{
|
return &verdict{
|
||||||
action: requestlog.ActionBlockedByCoreRuleSet,
|
action: requestlog.ActionBlockedByCoreRuleSet,
|
||||||
|
status: http.StatusForbidden,
|
||||||
figures: map[string]any{
|
figures: map[string]any{
|
||||||
figureScore: result.Score,
|
figureScore: result.Score,
|
||||||
"threshold": threshold,
|
"threshold": threshold,
|
||||||
@@ -76,20 +89,21 @@ func (coreRuleSet) check(
|
|||||||
// refusedByCoreRuleSet reports whether coreRuleSet refuses a request the Core
|
// refusedByCoreRuleSet reports whether coreRuleSet refuses a request the Core
|
||||||
// Rule Set found result of: whether the score is at or over
|
// Rule Set found result of: whether the score is at or over
|
||||||
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD in block mode.
|
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD in block mode.
|
||||||
func (rq *request) refusedByCoreRuleSet(result coreruleset.Result) bool {
|
func (c coreRuleSet) refusedByCoreRuleSet(result coreruleset.Result) bool {
|
||||||
cfg := rq.h.config
|
threshold := c.scoreThreshold
|
||||||
threshold := cfg.CoreRuleSetScoreThreshold
|
|
||||||
|
|
||||||
return cfg.CoreRuleSetMode == config.CoreRuleSetModeBlock &&
|
return c.mode == config.CoreRuleSetModeBlock &&
|
||||||
threshold != 0 && result.Score >= threshold
|
threshold != 0 && result.Score >= threshold
|
||||||
}
|
}
|
||||||
|
|
||||||
// alertCoreRuleSetMatch raises the core_rule_set_match alert for the request,
|
// alertCoreRuleSetMatch raises the core_rule_set_match alert for rq, which the
|
||||||
// which the Core Rule Set scored at result, at or over
|
// Core Rule Set scored at result, at or over
|
||||||
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD. Its detail gives the rule ids, the
|
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD. Its detail gives the rule ids, the
|
||||||
// score, the method and the path with the query, and, for a request that is
|
// score, the method and the path with the query, and, for a request that is
|
||||||
// not refused for it, the mode: detect, or observe in observe mode.
|
// not refused for it, the mode: detect, or observe in observe mode.
|
||||||
func (rq *request) alertCoreRuleSetMatch(result coreruleset.Result) {
|
func (c coreRuleSet) alertCoreRuleSetMatch(
|
||||||
|
rq *request, result coreruleset.Result,
|
||||||
|
) {
|
||||||
detail := map[string]any{
|
detail := map[string]any{
|
||||||
"rule_ids": result.RuleIDs,
|
"rule_ids": result.RuleIDs,
|
||||||
"score": result.Score,
|
"score": result.Score,
|
||||||
@@ -98,20 +112,20 @@ func (rq *request) alertCoreRuleSetMatch(result coreruleset.Result) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
switch {
|
switch {
|
||||||
case rq.h.config.CoreRuleSetMode == config.CoreRuleSetModeDetect:
|
case c.mode == config.CoreRuleSetModeDetect:
|
||||||
detail["mode"] = string(config.CoreRuleSetModeDetect)
|
detail["mode"] = string(config.CoreRuleSetModeDetect)
|
||||||
case rq.h.config.Observe:
|
case c.observe:
|
||||||
detail["mode"] = "observe"
|
detail["mode"] = "observe"
|
||||||
}
|
}
|
||||||
|
|
||||||
rq.h.alerts.Raise(alerts.Alert{
|
c.alerts.Raise(alerts.Alert{
|
||||||
Event: alerts.EventCoreRuleSetMatch,
|
Event: alerts.EventCoreRuleSetMatch,
|
||||||
Host: rq.site,
|
Host: rq.site,
|
||||||
Client: rq.client,
|
Client: rq.client,
|
||||||
Netblock: clientGroup(rq.h.config, rq.client),
|
Netblock: clientGroup(c.ipv6GroupPrefix, rq.client),
|
||||||
ASN: rq.line.ASN,
|
ASN: rq.lookupAnswer.ASN,
|
||||||
ASName: rq.line.ASName,
|
ASName: rq.lookupAnswer.ASName,
|
||||||
Country: rq.line.Country,
|
Country: rq.lookupAnswer.Country,
|
||||||
Reason: "scored by the Core Rule Set at or over " +
|
Reason: "scored by the Core Rule Set at or over " +
|
||||||
"SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD",
|
"SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD",
|
||||||
Detail: detail,
|
Detail: detail,
|
||||||
|
|||||||
@@ -12,7 +12,7 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
|
||||||
)
|
)
|
||||||
|
|
||||||
// The Core Rule Set's settings the tests set, besides SWW_WAF_CORE_RULE_SET_MODE, and
|
// The Core Rule Set's settings the tests set, besides SWW_WAF_CORE_RULE_SET_MODE, and
|
||||||
@@ -84,7 +84,7 @@ func TestCoreRuleSetMatchAddsTheExploitSignatureTag(t *testing.T) {
|
|||||||
name string
|
name string
|
||||||
mode string
|
mode string
|
||||||
status int
|
status int
|
||||||
action string
|
action requestlog.Action
|
||||||
}{
|
}{
|
||||||
{block, block, http.StatusForbidden, requestlog.ActionBlockedByCoreRuleSet},
|
{block, block, http.StatusForbidden, requestlog.ActionBlockedByCoreRuleSet},
|
||||||
{detect, detect, http.StatusOK, requestlog.ActionForward},
|
{detect, detect, http.StatusOK, requestlog.ActionForward},
|
||||||
@@ -158,7 +158,8 @@ func TestCoreRuleSetRefusesAttacksInBlockModeAndOnlyLogsThemInDetectMode(
|
|||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
for _, tc := range []struct {
|
for _, tc := range []struct {
|
||||||
mode, action string
|
mode string
|
||||||
|
action requestlog.Action
|
||||||
status int
|
status int
|
||||||
}{
|
}{
|
||||||
{block, requestlog.ActionBlockedByCoreRuleSet, http.StatusForbidden},
|
{block, requestlog.ActionBlockedByCoreRuleSet, http.StatusForbidden},
|
||||||
@@ -395,9 +396,9 @@ func TestGiteasFixesApplyOnlyToAHostnameThatServesGitea(t *testing.T) {
|
|||||||
})
|
})
|
||||||
|
|
||||||
wantDisposition(t, do(t, s.question(siteA, client, fileName)),
|
wantDisposition(t, do(t, s.question(siteA, client, fileName)),
|
||||||
http.StatusOK, waf.DispositionAllow)
|
http.StatusOK, wafprotocol.DispositionAllow)
|
||||||
wantDisposition(t, do(t, s.question(siteB, client, fileName)),
|
wantDisposition(t, do(t, s.question(siteB, client, fileName)),
|
||||||
http.StatusForbidden, waf.DispositionDeny)
|
http.StatusForbidden, wafprotocol.DispositionDeny)
|
||||||
|
|
||||||
lines := s.out.requestLines(t, 2)
|
lines := s.out.requestLines(t, 2)
|
||||||
wantLine(t, lines[0], http.StatusOK, requestlog.ActionForward)
|
wantLine(t, lines[0], http.StatusOK, requestlog.ActionForward)
|
||||||
@@ -653,7 +654,8 @@ func TestCoreRuleSetMatchRaisesTheCoreRuleSetMatchAlert(t *testing.T) {
|
|||||||
// with, and alertMode what the alert's detail gives as mode, if
|
// with, and alertMode what the alert's detail gives as mode, if
|
||||||
// anything.
|
// anything.
|
||||||
status int
|
status int
|
||||||
action, alertMode string
|
action requestlog.Action
|
||||||
|
alertMode string
|
||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
"block", map[string]string{coreRuleSetMode: block},
|
"block", map[string]string{coreRuleSetMode: block},
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
package waf
|
package waf
|
||||||
|
|
||||||
// The heuristic countries. It reads SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES.
|
// The heuristic countries. It reads SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES and
|
||||||
|
// SWW_WAF_BAN_RESPONSE.
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
@@ -13,19 +14,23 @@ import (
|
|||||||
|
|
||||||
// countries refuses a request when SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES is set and
|
// countries refuses a request when SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES is set and
|
||||||
// does not hold the client's country as it was looked up. A client without a
|
// does not hold the client's country as it was looked up. A client without a
|
||||||
// country, or whose country cannot be found, is refused. Its figures are the
|
// country, or whose country cannot be found, is refused. It refuses with
|
||||||
// country and the setting that refused it.
|
// SWW_WAF_BAN_RESPONSE. Its figures are the country and the setting that
|
||||||
type countries struct{}
|
// refused it.
|
||||||
|
type countries struct {
|
||||||
|
allowed []string // SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES
|
||||||
|
banResponse int // SWW_WAF_BAN_RESPONSE
|
||||||
|
}
|
||||||
|
|
||||||
func (countries) name() string { return "countries" }
|
func (countries) name() string { return "countries" }
|
||||||
|
|
||||||
func (countries) check(
|
func (c countries) check(
|
||||||
_ context.Context,
|
_ context.Context,
|
||||||
rq *request,
|
rq *request,
|
||||||
_ time.Time,
|
_ time.Time,
|
||||||
) *verdict {
|
) *verdict {
|
||||||
allowed := rq.h.config.ExclusivelyAllowedCountries
|
allowed := c.allowed
|
||||||
country := rq.line.Country
|
country := rq.lookupAnswer.Country
|
||||||
|
|
||||||
if len(allowed) == 0 || slices.Contains(allowed, country) {
|
if len(allowed) == 0 || slices.Contains(allowed, country) {
|
||||||
return nil
|
return nil
|
||||||
@@ -33,6 +38,7 @@ func (countries) check(
|
|||||||
|
|
||||||
return &verdict{
|
return &verdict{
|
||||||
action: requestlog.ActionCountryDenied,
|
action: requestlog.ActionCountryDenied,
|
||||||
|
status: c.banResponse,
|
||||||
figures: map[string]any{
|
figures: map[string]any{
|
||||||
"country": country,
|
"country": country,
|
||||||
"setting": config.ExclusivelyAllowedCountriesSetting,
|
"setting": config.ExclusivelyAllowedCountriesSetting,
|
||||||
|
|||||||
@@ -316,7 +316,7 @@ func TestExclusiveListRefusesAPrivateAddressUnlessAllowed(t *testing.T) {
|
|||||||
name string
|
name string
|
||||||
allowNets string
|
allowNets string
|
||||||
status int
|
status int
|
||||||
action string
|
action requestlog.Action
|
||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
"not in SWW_WAF_ALLOW_NETS", "",
|
"not in SWW_WAF_ALLOW_NETS", "",
|
||||||
|
|||||||
@@ -1,25 +1,34 @@
|
|||||||
package waf
|
package waf
|
||||||
|
|
||||||
// The heuristic deny_nets. It reads SWW_WAF_DENY_NETS.
|
// The heuristic deny_nets. It reads SWW_WAF_DENY_NETS and SWW_WAF_BAN_RESPONSE.
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"net/netip"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
)
|
)
|
||||||
|
|
||||||
// denyNets refuses a client in SWW_WAF_DENY_NETS. Its figure is the netblock of
|
// denyNets refuses a client in SWW_WAF_DENY_NETS, with SWW_WAF_BAN_RESPONSE.
|
||||||
// the setting the client is in.
|
// Its figure is the netblock of the setting the client is in.
|
||||||
type denyNets struct{}
|
type denyNets struct {
|
||||||
|
netblocks []netip.Prefix // SWW_WAF_DENY_NETS
|
||||||
|
banResponse int // SWW_WAF_BAN_RESPONSE
|
||||||
|
}
|
||||||
|
|
||||||
func (denyNets) name() string { return "deny_nets" }
|
func (denyNets) name() string { return "deny_nets" }
|
||||||
|
|
||||||
func (denyNets) check(_ context.Context, rq *request, _ time.Time) *verdict {
|
func (d denyNets) check(
|
||||||
for _, netblock := range rq.h.config.DenyNets {
|
_ context.Context,
|
||||||
|
rq *request,
|
||||||
|
_ time.Time,
|
||||||
|
) *verdict {
|
||||||
|
for _, netblock := range d.netblocks {
|
||||||
if netblock.Contains(rq.client) {
|
if netblock.Contains(rq.client) {
|
||||||
return &verdict{
|
return &verdict{
|
||||||
action: requestlog.ActionClientDenied,
|
action: requestlog.ActionClientDenied,
|
||||||
|
status: d.banResponse,
|
||||||
figures: map[string]any{figureNetblock: netblock.String()},
|
figures: map[string]any{figureNetblock: netblock.String()},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,47 @@
|
|||||||
|
package waf
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/netip"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestDenyNetsRefusesOnlyAClientInsideItsNetblocks(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
heuristic := denyNets{
|
||||||
|
netblocks: []netip.Prefix{netip.MustParsePrefix("198.51.100.0/24")},
|
||||||
|
}
|
||||||
|
|
||||||
|
// The request has no handler: the heuristic reads only what it holds.
|
||||||
|
inside := &request{
|
||||||
|
decision: decision{client: netip.MustParseAddr("198.51.100.7")},
|
||||||
|
}
|
||||||
|
|
||||||
|
v := heuristic.check(t.Context(), inside, time.Time{})
|
||||||
|
if v == nil {
|
||||||
|
t.Fatal("a client inside the netblock was let through")
|
||||||
|
}
|
||||||
|
|
||||||
|
if v.action != requestlog.ActionClientDenied {
|
||||||
|
t.Errorf(
|
||||||
|
"action = %q, want %q",
|
||||||
|
v.action,
|
||||||
|
requestlog.ActionClientDenied,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got := v.figures[figureNetblock]; got != "198.51.100.0/24" {
|
||||||
|
t.Errorf("netblock figure = %v, want 198.51.100.0/24", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
outside := &request{
|
||||||
|
decision: decision{client: netip.MustParseAddr("203.0.113.7")},
|
||||||
|
}
|
||||||
|
|
||||||
|
if v := heuristic.check(t.Context(), outside, time.Time{}); v != nil {
|
||||||
|
t.Errorf("a client outside the netblock was refused: %+v", v)
|
||||||
|
}
|
||||||
|
}
|
||||||
+25
-13
@@ -1,7 +1,8 @@
|
|||||||
package waf
|
package waf
|
||||||
|
|
||||||
// The heuristic dnsbl. It reads SWW_WAF_DNSBL_ABUSEIPDB_ACTION, and the DNSBL zones'
|
// The heuristic dnsbl. It reads SWW_WAF_DNSBL_ABUSEIPDB_ACTION and
|
||||||
// verdicts through the DNSBL checker.
|
// SWW_WAF_BAN_RESPONSE, and the DNSBL zones' verdicts through the DNSBL
|
||||||
|
// checker.
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
@@ -9,36 +10,47 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
)
|
)
|
||||||
|
|
||||||
// dnsbl notes the DNSBL zones whose verdict lists the client, as noteListed
|
// dnsbl notes the DNSBL zones whose verdict lists the client, as noteListed
|
||||||
// does, adds the tag dnsbl=<zone> for each, the zone's key masked, and refuses
|
// does, adds the tag dnsbl=<zone> for each, the zone's key masked, and refuses
|
||||||
// the request when SWW_WAF_DNSBL_ABUSEIPDB_ACTION is deny. Being
|
// the request with SWW_WAF_BAN_RESPONSE when SWW_WAF_DNSBL_ABUSEIPDB_ACTION is
|
||||||
// limit, it lowers the client's limits instead (see limitPercentages), and
|
// deny. Being limit, it lowers the client's limits instead (see
|
||||||
// being log, it does nothing more. A zone without a verdict on the client is
|
// limitPercentages), and being log, it does nothing more. A zone without a
|
||||||
// asked about it in the background, and the request does not wait for the
|
// verdict on the client is asked about it in the background, and the request
|
||||||
// answer. Its figure is the zones that list the client, their keys masked.
|
// does not wait for the answer. Its figure is the zones that list the client,
|
||||||
type dnsbl struct{}
|
// their keys masked.
|
||||||
|
type dnsbl struct {
|
||||||
|
dnsbl *reputation.DNSBL
|
||||||
|
action config.ReputationAction // SWW_WAF_DNSBL_ABUSEIPDB_ACTION
|
||||||
|
banResponse int // SWW_WAF_BAN_RESPONSE
|
||||||
|
hitNoter reputationHitNoter
|
||||||
|
}
|
||||||
|
|
||||||
func (dnsbl) name() string { return "dnsbl" }
|
func (dnsbl) name() string { return "dnsbl" }
|
||||||
|
|
||||||
func (dnsbl) check(ctx context.Context, rq *request, _ time.Time) *verdict {
|
func (d dnsbl) check(
|
||||||
listedBy := rq.h.dnsbl.ListedBy(ctx, rq.client)
|
ctx context.Context,
|
||||||
|
rq *request,
|
||||||
|
_ time.Time,
|
||||||
|
) *verdict {
|
||||||
|
listedBy := d.dnsbl.ListedBy(ctx, rq.client)
|
||||||
rq.dnsblListed = len(listedBy) > 0
|
rq.dnsblListed = len(listedBy) > 0
|
||||||
rq.noteListed(listedBy, "listed by a DNSBL zone")
|
d.hitNoter.noteListed(rq, listedBy, "listed by a DNSBL zone")
|
||||||
|
|
||||||
for _, zone := range listedBy {
|
for _, zone := range listedBy {
|
||||||
rq.addTag(config.DNSBLTagPrefix + strings.ToLower(zone))
|
rq.addTag(config.DNSBLTagPrefix + strings.ToLower(zone))
|
||||||
}
|
}
|
||||||
|
|
||||||
if !rq.dnsblListed ||
|
if !rq.dnsblListed || d.action != config.ReputationActionDeny {
|
||||||
rq.h.config.DNSBLAbuseIPDBAction != config.ReputationActionDeny {
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
return &verdict{
|
return &verdict{
|
||||||
action: requestlog.ActionClientDenied,
|
action: requestlog.ActionClientDenied,
|
||||||
|
status: d.banResponse,
|
||||||
figures: map[string]any{"zones": listedBy},
|
figures: map[string]any{"zones": listedBy},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -1,13 +1,11 @@
|
|||||||
package waf
|
package waf
|
||||||
|
|
||||||
import "sneak.berlin/go/smallwebwaf/internal/reputation"
|
import "net/http"
|
||||||
|
|
||||||
// BesideLookupTimeLimit is besideLookupTimeLimit.
|
// BesideLookupTimeLimit is besideLookupTimeLimit.
|
||||||
const BesideLookupTimeLimit = besideLookupTimeLimit
|
const BesideLookupTimeLimit = besideLookupTimeLimit
|
||||||
|
|
||||||
// BanAbuseIPDBBlacklist does on s what a fetch of AbuseIPDB's blacklist with
|
// HTTPServer is Go's server s holds, for tests that read its settings.
|
||||||
// the clients on listed does.
|
func (s *Server) HTTPServer() *http.Server {
|
||||||
func (s *Server) BanAbuseIPDBBlacklist(listed []reputation.BlacklistEntry) {
|
return s.httpServer
|
||||||
h := s.Handler.(*handler) //nolint:forcetypeassert // New makes it one
|
|
||||||
BanAbuseIPDBBlacklist(h.config, h.ledger, h.processLog, h.now(), listed)
|
|
||||||
}
|
}
|
||||||
@@ -4,6 +4,7 @@ package waf
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"net/http"
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -39,6 +40,7 @@ func (headerLimits) check(
|
|||||||
if requestLine > maxLineBytes {
|
if requestLine > maxLineBytes {
|
||||||
return &verdict{
|
return &verdict{
|
||||||
action: requestlog.ActionRequestLineTooLong,
|
action: requestlog.ActionRequestLineTooLong,
|
||||||
|
status: http.StatusRequestURITooLong,
|
||||||
figures: map[string]any{
|
figures: map[string]any{
|
||||||
"request_line_bytes": requestLine,
|
"request_line_bytes": requestLine,
|
||||||
"request_line_bytes_limit": maxLineBytes,
|
"request_line_bytes_limit": maxLineBytes,
|
||||||
@@ -56,6 +58,7 @@ func (headerLimits) check(
|
|||||||
|
|
||||||
return &verdict{
|
return &verdict{
|
||||||
action: requestlog.ActionRequestHeaderTooLarge,
|
action: requestlog.ActionRequestHeaderTooLarge,
|
||||||
|
status: http.StatusRequestHeaderFieldsTooLarge,
|
||||||
figures: map[string]any{
|
figures: map[string]any{
|
||||||
"longest_header_line_bytes": longest,
|
"longest_header_line_bytes": longest,
|
||||||
"header_line_bytes_limit": maxLineBytes,
|
"header_line_bytes_limit": maxLineBytes,
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestHealthEndpointCountsTowardNoLimit(t *testing.T) {
|
func TestHealthEndpointCountsTowardNoLimit(t *testing.T) {
|
||||||
@@ -25,8 +25,13 @@ func TestHealthEndpointCountsTowardNoLimit(t *testing.T) {
|
|||||||
)
|
)
|
||||||
|
|
||||||
for range healthChecks {
|
for range healthChecks {
|
||||||
got := s.ownEndpointRequest("", http.MethodGet, waf.HealthPath, "",
|
got := s.ownEndpointRequest(
|
||||||
http.StatusOK)
|
"",
|
||||||
|
http.MethodGet,
|
||||||
|
wafprotocol.HealthPath,
|
||||||
|
"",
|
||||||
|
http.StatusOK,
|
||||||
|
)
|
||||||
|
|
||||||
if string(got.body) != "ok\n" ||
|
if string(got.body) != "ok\n" ||
|
||||||
got.header.Get("Content-Type") != contentType {
|
got.header.Get("Content-Type") != contentType {
|
||||||
@@ -53,3 +58,14 @@ func TestHealthEndpointCountsTowardNoLimit(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestHealthEndpointIsFoundAtItsPathPercentEncoded(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
// %68 is h. An endpoint is found by the decoded path.
|
||||||
|
s, _, _ := startWithClock(t, "", nil)
|
||||||
|
|
||||||
|
got := s.ownEndpointRequest("", http.MethodGet, "/_sww/%68ealthz", "",
|
||||||
|
http.StatusOK)
|
||||||
|
wantBody(t, got, "ok\n")
|
||||||
|
}
|
||||||
+31
-74
@@ -3,11 +3,24 @@ package waf
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
)
|
)
|
||||||
|
|
||||||
// To add a heuristic, write one file with a type that implements heuristic,
|
// To add a heuristic, write one file with a struct that implements heuristic
|
||||||
// and add one line for it to one of the lists below. Its settings go in
|
// and holds the settings and parts it reads. Add one line for it to one of the
|
||||||
// config.WAF, and the comment at the top of its file names them.
|
// lists that newHandler builds, which hands it those settings and parts. Its
|
||||||
|
// settings go in config.WAF, and the comment at the top of its file names them.
|
||||||
|
// A heuristic of afterLookup that can refuse a request whatever the lookup
|
||||||
|
// answers also implements refusesWithoutLookup.
|
||||||
|
//
|
||||||
|
// Three places outside its file may change with it:
|
||||||
|
// - waitForGeoJS in newHandler lists the settings that need the lookup's
|
||||||
|
// answer;
|
||||||
|
// - requestCanGetTag in internal/config/waf.go holds which setting can give a
|
||||||
|
// request which tag;
|
||||||
|
// - findBesideLookup does the work of each heuristic of besideLookup, by
|
||||||
|
// name.
|
||||||
|
|
||||||
// heuristic is one of the checks that decide what waf does with a
|
// heuristic is one of the checks that decide what waf does with a
|
||||||
// request.
|
// request.
|
||||||
@@ -19,11 +32,23 @@ type heuristic interface {
|
|||||||
check(ctx context.Context, rq *request, now time.Time) *verdict
|
check(ctx context.Context, rq *request, now time.Time) *verdict
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// refusesWithoutLookup is a heuristic of afterLookup that can tell, before the
|
||||||
|
// lookup answers, that it will refuse a request whatever the answer.
|
||||||
|
type refusesWithoutLookup interface {
|
||||||
|
// refusesWhateverTheLookupAnswers reports whether the heuristic will refuse
|
||||||
|
// rq at now, whatever the lookup answers about its client. It changes
|
||||||
|
// nothing.
|
||||||
|
refusesWhateverTheLookupAnswers(rq *request, now time.Time) bool
|
||||||
|
}
|
||||||
|
|
||||||
// verdict is a heuristic refusing a request, or banning its client: the action
|
// verdict is a heuristic refusing a request, or banning its client: the action
|
||||||
// the log line names, one of requestlog's Action constants, and the figures the
|
// the log line names, the status the visitor is answered with, and the figures
|
||||||
// heuristic acted on, by name, such as the count and the limit.
|
// the heuristic acted on, by name, such as the count and the limit. A status of
|
||||||
|
// 0 closes the visitor's connection without an answer. A verdict of afterAnswer
|
||||||
|
// or onReport refuses nothing, and has neither action nor status.
|
||||||
type verdict struct {
|
type verdict struct {
|
||||||
action string
|
action requestlog.Action
|
||||||
|
status int
|
||||||
figures map[string]any
|
figures map[string]any
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -36,74 +61,6 @@ const (
|
|||||||
figureNetblock = "netblock"
|
figureNetblock = "netblock"
|
||||||
)
|
)
|
||||||
|
|
||||||
// The lists of heuristics, each in the order its heuristics run. A client in
|
|
||||||
// SWW_WAF_ALLOW_NETS skips them all.
|
|
||||||
|
|
||||||
// beforeLookup are the heuristics that run before the client's AS number and
|
|
||||||
// country are looked up, so that a client they refuse is not looked up.
|
|
||||||
// header_limits comes first: it only measures and counts, and a request it
|
|
||||||
// refuses causes no other work.
|
|
||||||
func beforeLookup() []heuristic {
|
|
||||||
return []heuristic{
|
|
||||||
headerLimits{},
|
|
||||||
tagLists{},
|
|
||||||
routeWatchTags{},
|
|
||||||
denyNets{},
|
|
||||||
activeBan{},
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// afterLookup are the heuristics that run once the lookup has been made.
|
|
||||||
func afterLookup() []heuristic {
|
|
||||||
return []heuristic{
|
|
||||||
countries{},
|
|
||||||
blocklists{},
|
|
||||||
dnsbl{},
|
|
||||||
abuseIPDB{},
|
|
||||||
tagDeny{},
|
|
||||||
rateLimits{},
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// besideLookup are the heuristics that only read the request. Their work runs
|
|
||||||
// beside the lookup and afterLookup, in findBesideLookup, which changes nothing
|
|
||||||
// of the request. Their check then applies what that work found, and runs only
|
|
||||||
// once waf takes it.
|
|
||||||
func besideLookup() []heuristic {
|
|
||||||
return []heuristic{
|
|
||||||
trapPaths{},
|
|
||||||
ruleFiles{},
|
|
||||||
coreRuleSet{},
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// afterBesideLookup are the heuristics that run once afterLookup and
|
|
||||||
// besideLookup have both let the request through. tag_deny runs a second time,
|
|
||||||
// on the tag rule_files and core_rule_set can add.
|
|
||||||
func afterBesideLookup() []heuristic {
|
|
||||||
return []heuristic{
|
|
||||||
tagDeny{},
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// afterAnswer are the heuristics that run on a request once waf has
|
|
||||||
// answered it, before its decision line is written. They ban the client, and do
|
|
||||||
// not change what the request got: their verdict's action is not used.
|
|
||||||
func afterAnswer() []heuristic {
|
|
||||||
return []heuristic{
|
|
||||||
refusalLimit{},
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// onReport are the heuristics that run on an allowed request when rproxy's
|
|
||||||
// report of it comes, before its report line is written. Like afterAnswer, they
|
|
||||||
// only ban.
|
|
||||||
func onReport() []heuristic {
|
|
||||||
return []heuristic{
|
|
||||||
byteLimits{},
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// runHeuristics runs list, one of those checkClient runs, on the request at now,
|
// runHeuristics runs list, one of those checkClient runs, on the request at now,
|
||||||
// and returns the first verdict, or nil when none gives one. The log line's
|
// and returns the first verdict, or nil when none gives one. The log line's
|
||||||
// heuristic and heuristic_figures name the heuristic that gave it, and give its
|
// heuristic and heuristic_figures name the heuristic that gave it, and give its
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestHistoryKeepsEachRequestOfTheClient(t *testing.T) {
|
func TestHistoryKeepsEachRequestOfTheClient(t *testing.T) {
|
||||||
@@ -118,7 +119,7 @@ func TestOwnEndpointsAreNotInTheHistory(t *testing.T) {
|
|||||||
s.ownEndpointRequest(
|
s.ownEndpointRequest(
|
||||||
"",
|
"",
|
||||||
http.MethodGet,
|
http.MethodGet,
|
||||||
waf.HealthPath,
|
wafprotocol.HealthPath,
|
||||||
"",
|
"",
|
||||||
http.StatusOK,
|
http.StatusOK,
|
||||||
)
|
)
|
||||||
@@ -150,7 +151,7 @@ func TestPathUnderTheOwnPrefixIsNeitherForwardedNorRefused(t *testing.T) {
|
|||||||
|
|
||||||
// historyOf returns the history of the client at addr.
|
// historyOf returns the history of the client at addr.
|
||||||
func historyOf(
|
func historyOf(
|
||||||
t *testing.T, server *waf.Server, addr string,
|
t *testing.T, server *wafAndParts, addr string,
|
||||||
) ratelimit.History {
|
) ratelimit.History {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
|
|||||||
+27
-27
@@ -3,7 +3,6 @@ package waf
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"net/http"
|
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
@@ -16,15 +15,6 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
)
|
)
|
||||||
|
|
||||||
// The headers in which an answer of allow gives rproxy the client's AS number
|
|
||||||
// and country, for the app, while SWW_WAF_ADD_LOOKUP_HEADERS is true. Go writes
|
|
||||||
// every header name in this form, as it sends it and as it receives it, so
|
|
||||||
// X-Client-ASN arrives as X-Client-Asn; header names are not case-sensitive.
|
|
||||||
const (
|
|
||||||
asnHeader = "X-Client-Asn"
|
|
||||||
countryHeader = "X-Client-Country"
|
|
||||||
)
|
|
||||||
|
|
||||||
// lookupResult is what the lookup gave for a request: whether its client was
|
// lookupResult is what the lookup gave for a request: whether its client was
|
||||||
// looked up, the answer, and whether the request skipped the wait for GeoJS.
|
// looked up, the answer, and whether the request skipped the wait for GeoJS.
|
||||||
type lookupResult struct {
|
type lookupResult struct {
|
||||||
@@ -56,7 +46,7 @@ func (rq *request) startLookUp(
|
|||||||
return result
|
return result
|
||||||
}
|
}
|
||||||
|
|
||||||
client := clientGroup(rq.h.config, rq.client)
|
client := clientGroup(rq.h.config.IPv6GroupPrefix, rq.client)
|
||||||
|
|
||||||
if rq.h.config.LookupSource == config.LookupSourceFile {
|
if rq.h.config.LookupSource == config.LookupSourceFile {
|
||||||
result <- lookupResult{
|
result <- lookupResult{
|
||||||
@@ -90,6 +80,26 @@ func (rq *request) startLookUp(
|
|||||||
return result
|
return result
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// refusedWhateverTheAnswer reports whether the request is one that does not wait
|
||||||
|
// for GeoJS, since the checks will refuse it at now whatever GeoJS answers about
|
||||||
|
// its client: a heuristic of afterLookup that implements refusesWithoutLookup
|
||||||
|
// says it will refuse it. In observe mode, which refuses nothing, none is.
|
||||||
|
// rproxy lets a request through when waf answers too late.
|
||||||
|
func (rq *request) refusedWhateverTheAnswer(now time.Time) bool {
|
||||||
|
if rq.h.config.Observe {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, each := range rq.h.afterLookup {
|
||||||
|
knows, ok := each.(refusesWithoutLookup)
|
||||||
|
if ok && knows.refusesWhateverTheLookupAnswers(rq, now) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
// noteLookup notes result for the log line and as the request's asn= and
|
// noteLookup notes result for the log line and as the request's asn= and
|
||||||
// country= tags.
|
// country= tags.
|
||||||
func (rq *request) noteLookup(result lookupResult) {
|
func (rq *request) noteLookup(result lookupResult) {
|
||||||
@@ -104,11 +114,11 @@ func (rq *request) noteLookup(result lookupResult) {
|
|||||||
rq.line.ASName = rq.lookupAnswer.ASName
|
rq.line.ASName = rq.lookupAnswer.ASName
|
||||||
rq.line.Country = rq.lookupAnswer.Country
|
rq.line.Country = rq.lookupAnswer.Country
|
||||||
|
|
||||||
if tag := asnTag(rq.line.ASN); tag != "" {
|
if tag := asnTag(rq.lookupAnswer.ASN); tag != "" {
|
||||||
rq.addTag(tag)
|
rq.addTag(tag)
|
||||||
}
|
}
|
||||||
|
|
||||||
if tag := countryTag(rq.line.Country); tag != "" {
|
if tag := countryTag(rq.lookupAnswer.Country); tag != "" {
|
||||||
rq.addTag(tag)
|
rq.addTag(tag)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -143,8 +153,10 @@ func AddLookup(
|
|||||||
) {
|
) {
|
||||||
limiter.AddLookup(answer.Client, answer.Answered,
|
limiter.AddLookup(answer.Client, answer.Answered,
|
||||||
answer.ASN, answer.ASName, answer.Country)
|
answer.ASN, answer.ASName, answer.Country)
|
||||||
ledger.AddLookup(banNetblock(cfg, answer.Client.Addr()),
|
netblock := banNetblock(
|
||||||
answer.ASN, answer.ASName, answer.Country)
|
cfg.BanScopeV4Prefix, cfg.IPv6GroupPrefix, answer.Client.Addr(),
|
||||||
|
)
|
||||||
|
ledger.AddLookup(netblock, answer.ASN, answer.ASName, answer.Country)
|
||||||
}
|
}
|
||||||
|
|
||||||
// GeoJSFailed logs a failure of GeoJS to processLog, raises a
|
// GeoJSFailed logs a failure of GeoJS to processLog, raises a
|
||||||
@@ -167,18 +179,6 @@ func GeoJSFailed(
|
|||||||
m.GeoJSFailures.Inc()
|
m.GeoJSFailures.Inc()
|
||||||
}
|
}
|
||||||
|
|
||||||
// setLookupHeaders sets the headers that give the client's AS number and
|
|
||||||
// country, leaving out one that is unknown.
|
|
||||||
func setLookupHeaders(header http.Header, asn, country string) {
|
|
||||||
if asn != "" {
|
|
||||||
header.Set(asnHeader, asn)
|
|
||||||
}
|
|
||||||
|
|
||||||
if country != "" {
|
|
||||||
header.Set(countryHeader, country)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// canBePlaced reports whether a lookup can place addr: private, loopback
|
// canBePlaced reports whether a lookup can place addr: private, loopback
|
||||||
// and link-local addresses have no AS number or country.
|
// and link-local addresses have no AS number or country.
|
||||||
func canBePlaced(addr netip.Addr) bool {
|
func canBePlaced(addr netip.Addr) bool {
|
||||||
|
|||||||
@@ -16,6 +16,7 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -66,7 +67,7 @@ func TestMetricsNeedTheTokenAndOtherPathsAreNotFound(t *testing.T) {
|
|||||||
{http.MethodPost, waf.MetricsPath, bearer, http.StatusNotFound},
|
{http.MethodPost, waf.MetricsPath, bearer, http.StatusNotFound},
|
||||||
{http.MethodGet, waf.MetricsPath + "/", bearer, http.StatusNotFound},
|
{http.MethodGet, waf.MetricsPath + "/", bearer, http.StatusNotFound},
|
||||||
{http.MethodGet, "/_sww/bans", bearer, http.StatusNotFound},
|
{http.MethodGet, "/_sww/bans", bearer, http.StatusNotFound},
|
||||||
{http.MethodPost, waf.HealthPath, "", http.StatusNotFound},
|
{http.MethodPost, wafprotocol.HealthPath, "", http.StatusNotFound},
|
||||||
} {
|
} {
|
||||||
got := s.ownEndpointRequest(
|
got := s.ownEndpointRequest(
|
||||||
tc.authorization,
|
tc.authorization,
|
||||||
|
|||||||
@@ -22,12 +22,12 @@ func TestBanActionSaysWhetherTheBanRunsOut(t *testing.T) {
|
|||||||
|
|
||||||
for _, tc := range []struct {
|
for _, tc := range []struct {
|
||||||
name string
|
name string
|
||||||
cause string
|
cause bans.Cause
|
||||||
// endsIn is how long after the request the ban ends: negative for a
|
// endsIn is how long after the request the ban ends: negative for a
|
||||||
// ban that has ended, and zero for a permanent ban.
|
// ban that has ended, and zero for a permanent ban.
|
||||||
endsIn time.Duration
|
endsIn time.Duration
|
||||||
path string
|
path string
|
||||||
want string
|
want requestlog.Action
|
||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
"temporary ban", bans.CauseAdmin, time.Hour, "/",
|
"temporary ban", bans.CauseAdmin, time.Hour, "/",
|
||||||
@@ -170,7 +170,10 @@ func TestObserveModeForwardsWhatEnforceModeRefuses(t *testing.T) {
|
|||||||
// and its second breaks it.
|
// and its second breaks it.
|
||||||
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||||
|
|
||||||
for _, sent := range []struct{ from, refusal string }{
|
for _, sent := range []struct {
|
||||||
|
from string
|
||||||
|
refusal requestlog.Action
|
||||||
|
}{
|
||||||
{denied, requestlog.ActionClientDenied},
|
{denied, requestlog.ActionClientDenied},
|
||||||
{banned, requestlog.ActionTemporarilyBanned},
|
{banned, requestlog.ActionTemporarilyBanned},
|
||||||
{fromKP, requestlog.ActionClientDenied},
|
{fromKP, requestlog.ActionClientDenied},
|
||||||
@@ -253,7 +256,7 @@ func TestObserveModeMakesNoBanAndKeepsTheBansItHas(t *testing.T) {
|
|||||||
|
|
||||||
// wantWouldAction checks the request log line's would_action, and that a
|
// wantWouldAction checks the request log line's would_action, and that a
|
||||||
// line that should have none has no such field.
|
// line that should have none has no such field.
|
||||||
func wantWouldAction(t *testing.T, line logLine, want string) {
|
func wantWouldAction(t *testing.T, line logLine, want requestlog.Action) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
got, present := line.fields["would_action"]
|
got, present := line.fields["would_action"]
|
||||||
@@ -261,7 +264,7 @@ func wantWouldAction(t *testing.T, line logLine, want string) {
|
|||||||
switch {
|
switch {
|
||||||
case want == "" && present:
|
case want == "" && present:
|
||||||
t.Errorf("log line has would_action %v, want none", got)
|
t.Errorf("log line has would_action %v, want none", got)
|
||||||
case want != "" && got != want:
|
case want != "" && got != string(want):
|
||||||
t.Errorf("log line has would_action %v, want %s", got, want)
|
t.Errorf("log line has would_action %v, want %s", got, want)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -1,205 +0,0 @@
|
|||||||
// Package waf is the server the waf command runs. It answers
|
|
||||||
// rproxy's questions, takes rproxy's reports and answers its own endpoints, and
|
|
||||||
// writes a log line for each. It also bans the clients on AbuseIPDB's
|
|
||||||
// blacklist each time the reputation package fetches it.
|
|
||||||
package waf
|
|
||||||
|
|
||||||
import (
|
|
||||||
"cmp"
|
|
||||||
"io"
|
|
||||||
"log/slog"
|
|
||||||
"net/http"
|
|
||||||
"net/netip"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
lru "github.com/hashicorp/golang-lru/v2"
|
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/routewatch"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
|
||||||
)
|
|
||||||
|
|
||||||
// ownEndpointPrefix starts the path of each of waf's own endpoints. The app
|
|
||||||
// never sees a path under it.
|
|
||||||
const ownEndpointPrefix = "/_sww/"
|
|
||||||
|
|
||||||
// HealthPath is the health endpoint of waf and of rproxy, which the
|
|
||||||
// container's health check asks.
|
|
||||||
const HealthPath = "/_sww/healthz"
|
|
||||||
|
|
||||||
// MetricsPath is where the metrics are, for a request that carries
|
|
||||||
// SWW_WAF_METRICS_TOKEN.
|
|
||||||
const MetricsPath = "/_sww/metrics"
|
|
||||||
|
|
||||||
// BansPath is where an admin lists and adds bans, and, followed by / and
|
|
||||||
// a client's address, lifts them, with SWW_WAF_ADMIN_TOKEN.
|
|
||||||
const BansPath = "/_sww/bans"
|
|
||||||
|
|
||||||
// ClientsPath is where an admin asks what waf knows of a client, by
|
|
||||||
// the client's address after it, with SWW_WAF_ADMIN_TOKEN.
|
|
||||||
const ClientsPath = "/_sww/clients/"
|
|
||||||
|
|
||||||
// Params are what New needs.
|
|
||||||
type Params struct {
|
|
||||||
Config *config.WAF
|
|
||||||
// RequestLog receives two JSON lines per question, one at its decision and
|
|
||||||
// one when its report comes, and one per request to waf's own
|
|
||||||
// endpoints.
|
|
||||||
RequestLog io.Writer
|
|
||||||
// ProcessLog receives the process's own messages.
|
|
||||||
ProcessLog *slog.Logger
|
|
||||||
// Metrics count what waf does. Whoever builds them adds the metrics
|
|
||||||
// read from the other parts.
|
|
||||||
Metrics *metrics.Metrics
|
|
||||||
// Limiter is the table of clients the rate limits count, with each
|
|
||||||
// client's history.
|
|
||||||
Limiter *ratelimit.Limiter
|
|
||||||
// Ledger holds the bans.
|
|
||||||
Ledger *bans.Ledger
|
|
||||||
// Anomalies are the counters of the anomaly thresholds.
|
|
||||||
Anomalies *anomaly.Counters
|
|
||||||
// Lists are the lists fetched from URLs, DNSBL the DNSBL zones'
|
|
||||||
// verdicts, and AbuseIPDB AbuseIPDB's scores and the checks spent.
|
|
||||||
Lists *reputation.Lists
|
|
||||||
DNSBL *reputation.DNSBL
|
|
||||||
AbuseIPDB *reputation.AbuseIPDB
|
|
||||||
// RouteWatch asks routewatch about clients, and is nil while
|
|
||||||
// SWW_WAF_ROUTEWATCH_URL is unset.
|
|
||||||
RouteWatch *routewatch.Client
|
|
||||||
// GeoJS looks clients' AS numbers and countries up while
|
|
||||||
// SWW_WAF_LOOKUP_SOURCE is geojs. It is given AddLookup as its Answered
|
|
||||||
// and GeoJSFailed as its Failed.
|
|
||||||
GeoJS *geojs.GeoJS
|
|
||||||
// LookupFile is the lookup database they are looked up in while
|
|
||||||
// SWW_WAF_LOOKUP_SOURCE is file, and nil otherwise.
|
|
||||||
LookupFile *lookup.File
|
|
||||||
// CoreRuleSet is what inspects requests while
|
|
||||||
// SWW_WAF_CORE_RULE_SET_MODE is not off, and nil while it is.
|
|
||||||
CoreRuleSet CoreRuleSetInspector
|
|
||||||
// Now tells the time by which requests are counted for the rate
|
|
||||||
// limits, bans are made and run out, and GeoJS's answers are kept,
|
|
||||||
// normally time.Now in UTC, the time the state files give.
|
|
||||||
Now func() time.Time
|
|
||||||
// Rules are the rule files' rules, which each request is checked
|
|
||||||
// against.
|
|
||||||
Rules *rules.Files
|
|
||||||
// Alerts receive the alert for each ban waf makes or makes
|
|
||||||
// permanent, for each count over an anomaly threshold, for each request
|
|
||||||
// whose client a blocklist, a DNSBL zone or AbuseIPDB lists, for each
|
|
||||||
// request the Core Rule Set scores at or over
|
|
||||||
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD, and for GeoJS failing, a fetch of a list
|
|
||||||
// failing, a query to a DNSBL zone or a check with AbuseIPDB failing,
|
|
||||||
// or the day's AbuseIPDB checks used up.
|
|
||||||
Alerts *alerts.Queue
|
|
||||||
// BesideLookupTimeLimit is how long the heuristics of besideLookup have
|
|
||||||
// for a request. Zero means besideLookupTimeLimit. Tests that run the
|
|
||||||
// Core Rule Set lengthen it, since the race detector and a busy host can
|
|
||||||
// slow an inspection past besideLookupTimeLimit.
|
|
||||||
BesideLookupTimeLimit time.Duration
|
|
||||||
}
|
|
||||||
|
|
||||||
// CoreRuleSetInspector inspects a request with the Core Rule Set, as
|
|
||||||
// coreruleset.CoreRuleSet's Inspect says. Tests give waf a stand-in for it.
|
|
||||||
type CoreRuleSetInspector interface {
|
|
||||||
Inspect(
|
|
||||||
r *http.Request, client netip.Addr, body io.Reader,
|
|
||||||
) (coreruleset.Result, []byte, error)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Server is the server waf runs, with the parts of its Params. The tests
|
|
||||||
// read them.
|
|
||||||
type Server struct {
|
|
||||||
*http.Server
|
|
||||||
|
|
||||||
Ledger *bans.Ledger
|
|
||||||
Limiter *ratelimit.Limiter
|
|
||||||
GeoJS *geojs.GeoJS
|
|
||||||
Anomalies *anomaly.Counters
|
|
||||||
LookupFile *lookup.File
|
|
||||||
Lists *reputation.Lists
|
|
||||||
DNSBL *reputation.DNSBL
|
|
||||||
AbuseIPDB *reputation.AbuseIPDB
|
|
||||||
RouteWatch *routewatch.Client
|
|
||||||
Metrics *metrics.Metrics
|
|
||||||
}
|
|
||||||
|
|
||||||
// newHandler returns the handler with what every request shares, the parts
|
|
||||||
// in params among it.
|
|
||||||
func newHandler(params Params) *handler {
|
|
||||||
awaitingReport, err := lru.New[string, *decision](maxAwaitingReport)
|
|
||||||
if err != nil {
|
|
||||||
panic(err) // New fails only for a size below one
|
|
||||||
}
|
|
||||||
|
|
||||||
return &handler{
|
|
||||||
config: params.Config,
|
|
||||||
requestLog: params.RequestLog,
|
|
||||||
processLog: params.ProcessLog,
|
|
||||||
now: params.Now,
|
|
||||||
metrics: params.Metrics,
|
|
||||||
awaitingReport: awaitingReport,
|
|
||||||
limiter: params.Limiter,
|
|
||||||
ledger: params.Ledger,
|
|
||||||
geojs: params.GeoJS,
|
|
||||||
anomalies: params.Anomalies,
|
|
||||||
lookupFile: params.LookupFile,
|
|
||||||
lists: params.Lists,
|
|
||||||
dnsbl: params.DNSBL,
|
|
||||||
abuseIPDB: params.AbuseIPDB,
|
|
||||||
routeWatch: params.RouteWatch,
|
|
||||||
rules: params.Rules,
|
|
||||||
coreRuleSet: params.CoreRuleSet,
|
|
||||||
alerts: params.Alerts,
|
|
||||||
// The country list, the headers, the tag settings that name an asn=
|
|
||||||
// or country= tag and the biased thresholds act on the answer before
|
|
||||||
// the request goes on.
|
|
||||||
waitForGeoJS: len(params.Config.ExclusivelyAllowedCountries) > 0 ||
|
|
||||||
params.Config.AddLookupHeaders ||
|
|
||||||
params.Config.TagsNeedLookup() ||
|
|
||||||
biasedThresholdsSet(params.Config),
|
|
||||||
besideLookupLimit: cmp.Or(
|
|
||||||
params.BesideLookupTimeLimit,
|
|
||||||
besideLookupTimeLimit,
|
|
||||||
),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// handler is waf's handler. It holds what every request shares; what
|
|
||||||
// belongs to one request is in a request.
|
|
||||||
type handler struct {
|
|
||||||
config *config.WAF
|
|
||||||
requestLog io.Writer
|
|
||||||
processLog *slog.Logger
|
|
||||||
now func() time.Time
|
|
||||||
metrics *metrics.Metrics
|
|
||||||
limiter *ratelimit.Limiter
|
|
||||||
ledger *bans.Ledger
|
|
||||||
geojs *geojs.GeoJS
|
|
||||||
anomalies *anomaly.Counters
|
|
||||||
lookupFile *lookup.File
|
|
||||||
lists *reputation.Lists
|
|
||||||
dnsbl *reputation.DNSBL
|
|
||||||
abuseIPDB *reputation.AbuseIPDB
|
|
||||||
routeWatch *routewatch.Client
|
|
||||||
rules *rules.Files
|
|
||||||
coreRuleSet CoreRuleSetInspector
|
|
||||||
alerts *alerts.Queue
|
|
||||||
// waitForGeoJS is true when a setting needs each request's answer from
|
|
||||||
// GeoJS before the request goes on.
|
|
||||||
waitForGeoJS bool
|
|
||||||
// besideLookupLimit is how long the heuristics of besideLookup have for a
|
|
||||||
// request, as Params.BesideLookupTimeLimit says.
|
|
||||||
besideLookupLimit time.Duration
|
|
||||||
// awaitingReport holds the decision on each answered question by its id
|
|
||||||
// until its report comes, and nil for a question still being handled.
|
|
||||||
awaitingReport *lru.Cache[string, *decision]
|
|
||||||
}
|
|
||||||
+71
-35
@@ -316,7 +316,7 @@ func startProxyWithGeoJS(
|
|||||||
func startProxyWithClock(
|
func startProxyWithClock(
|
||||||
t *testing.T, appURL, geojsURL string, now func() time.Time,
|
t *testing.T, appURL, geojsURL string, now func() time.Time,
|
||||||
env map[string]string,
|
env map[string]string,
|
||||||
) (string, *output, *waf.Server) {
|
) (string, *output, *wafAndParts) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
addr, out, server, _ := startProxyWithAlerts(
|
addr, out, server, _ := startProxyWithAlerts(
|
||||||
@@ -335,7 +335,7 @@ func startProxyWithClock(
|
|||||||
func startProxyWithAlerts(
|
func startProxyWithAlerts(
|
||||||
t *testing.T, appURL, geojsURL string, now func() time.Time,
|
t *testing.T, appURL, geojsURL string, now func() time.Time,
|
||||||
env map[string]string,
|
env map[string]string,
|
||||||
) (string, *output, *waf.Server, *alerts.Queue) {
|
) (string, *output, *wafAndParts, *alerts.Queue) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
server, out, alertQueue := newWAF(t, geojsURL, now, env)
|
server, out, alertQueue := newWAF(t, geojsURL, now, env)
|
||||||
@@ -344,9 +344,9 @@ func startProxyWithAlerts(
|
|||||||
return startRProxy(t, server.Addr, appURL, env), out, server, alertQueue
|
return startRProxy(t, server.Addr, appURL, env), out, server, alertQueue
|
||||||
}
|
}
|
||||||
|
|
||||||
// serve serves server on a port of its own until the test ends, and sets its
|
// serve serves server on a port of its own until the test ends, and sets
|
||||||
// Addr to where it listens.
|
// server.Addr to where it listens.
|
||||||
func serve(t *testing.T, server *waf.Server) {
|
func serve(t *testing.T, server *wafAndParts) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
listener, err := (&net.ListenConfig{}).Listen(
|
listener, err := (&net.ListenConfig{}).Listen(
|
||||||
@@ -422,8 +422,32 @@ func startRProxy(
|
|||||||
return listener.Addr().String()
|
return listener.Addr().String()
|
||||||
}
|
}
|
||||||
|
|
||||||
// newWAF makes waf's server, without starting it, and
|
// wafAndParts is waf's server with the Params it was built from, so that a
|
||||||
// returns it, what it writes, and the queue of the alerts it raises, as the
|
// test can read the parts the server uses. Addr is where it listens once serve
|
||||||
|
// has it serving.
|
||||||
|
type wafAndParts struct {
|
||||||
|
*waf.Server
|
||||||
|
waf.Params
|
||||||
|
|
||||||
|
Addr string
|
||||||
|
}
|
||||||
|
|
||||||
|
// banAbuseIPDBBlacklist does on w what a fetch of AbuseIPDB's blacklist with
|
||||||
|
// the clients on listed does.
|
||||||
|
func (w *wafAndParts) banAbuseIPDBBlacklist(
|
||||||
|
listed []reputation.BlacklistEntry,
|
||||||
|
) {
|
||||||
|
waf.BanAbuseIPDBBlacklist(
|
||||||
|
w.Config,
|
||||||
|
w.Ledger,
|
||||||
|
w.ProcessLog,
|
||||||
|
w.Now(),
|
||||||
|
listed,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// newWAF makes waf's server, without starting it, and returns it with its
|
||||||
|
// parts, what it writes, and the queue of the alerts it raises, as the
|
||||||
// settings in env make it. No alert is sent from the queue: they wait in it,
|
// settings in env make it. No alert is sent from the queue: they wait in it,
|
||||||
// for the test to look at.
|
// for the test to look at.
|
||||||
// With no geojsURL, there is no stand-in for GeoJS to look clients up at, and
|
// With no geojsURL, there is no stand-in for GeoJS to look clients up at, and
|
||||||
@@ -438,19 +462,21 @@ func newWAF(
|
|||||||
geojsURL string,
|
geojsURL string,
|
||||||
now func() time.Time,
|
now func() time.Time,
|
||||||
env map[string]string,
|
env map[string]string,
|
||||||
) (*waf.Server, *output, *alerts.Queue) {
|
) (*wafAndParts, *output, *alerts.Queue) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
return newWAFWith(t, waf.New, geojsURL, abuseIPDBURL, now, env)
|
return newWAFWith(
|
||||||
|
t, func(*waf.Params) {}, geojsURL, abuseIPDBURL, now, env,
|
||||||
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
// newWAFWith is newWAF for the server build makes, with clients checked with
|
// newWAFWith is newWAF with change applied to the Params before the server is
|
||||||
// AbuseIPDB at abuseIPDBCheckURL.
|
// built from them, and clients checked with AbuseIPDB at abuseIPDBCheckURL.
|
||||||
func newWAFWith(
|
func newWAFWith(
|
||||||
t *testing.T, build func(waf.Params) *waf.Server,
|
t *testing.T, change func(*waf.Params),
|
||||||
geojsURL, abuseIPDBCheckURL string, now func() time.Time,
|
geojsURL, abuseIPDBCheckURL string, now func() time.Time,
|
||||||
env map[string]string,
|
env map[string]string,
|
||||||
) (*waf.Server, *output, *alerts.Queue) {
|
) (*wafAndParts, *output, *alerts.Queue) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
settings := map[string]string{
|
settings := map[string]string{
|
||||||
@@ -518,8 +544,9 @@ func newWAFWith(
|
|||||||
addCoreRuleSet(t, ¶ms)
|
addCoreRuleSet(t, ¶ms)
|
||||||
addCounters(¶ms)
|
addCounters(¶ms)
|
||||||
addOutsideSources(¶ms, geojsURL, abuseIPDBCheckURL)
|
addOutsideSources(¶ms, geojsURL, abuseIPDBCheckURL)
|
||||||
|
change(¶ms)
|
||||||
|
|
||||||
server := build(params)
|
server := &wafAndParts{Server: waf.New(params), Params: params}
|
||||||
|
|
||||||
return server, out, alertQueue
|
return server, out, alertQueue
|
||||||
}
|
}
|
||||||
@@ -548,16 +575,11 @@ func addCoreRuleSet(t *testing.T, params *waf.Params) {
|
|||||||
params.CoreRuleSet = coreRuleSet
|
params.CoreRuleSet = coreRuleSet
|
||||||
}
|
}
|
||||||
|
|
||||||
// addCounters adds to params the metrics, the table of clients, the ledger
|
// addCounters adds to params the table of clients, the ledger and the
|
||||||
// and the anomaly counters, as sww builds them from params' settings. The
|
// anomaly counters, as sww builds them from params' settings.
|
||||||
// metrics read the ledger, the table of clients and the rules.
|
|
||||||
func addCounters(params *waf.Params) {
|
func addCounters(params *waf.Params) {
|
||||||
cfg := params.Config
|
cfg := params.Config
|
||||||
|
|
||||||
params.Metrics = metrics.New(
|
|
||||||
cfg.MetricsBusiestCountriesAndASNs,
|
|
||||||
cfg.InstanceName,
|
|
||||||
)
|
|
||||||
params.Limiter = ratelimit.New(ratelimit.Params{
|
params.Limiter = ratelimit.New(ratelimit.Params{
|
||||||
Limits: ratelimit.Limits{
|
Limits: ratelimit.Limits{
|
||||||
PerMinute: cfg.RateLimitPerMinute,
|
PerMinute: cfg.RateLimitPerMinute,
|
||||||
@@ -587,26 +609,18 @@ func addCounters(params *waf.Params) {
|
|||||||
NamedNetblocks: cfg.AnomalyWatchedNets,
|
NamedNetblocks: cfg.AnomalyWatchedNets,
|
||||||
Alerts: params.Alerts,
|
Alerts: params.Alerts,
|
||||||
})
|
})
|
||||||
|
|
||||||
params.Metrics.AddBansAndClients(
|
|
||||||
params.Ledger,
|
|
||||||
params.Limiter,
|
|
||||||
params.Now,
|
|
||||||
)
|
|
||||||
params.Metrics.AddRules(params.Rules)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// addOutsideSources adds to params the lists fetched from URLs, the DNSBL
|
// addOutsideSources adds to params the lists fetched from URLs, the DNSBL
|
||||||
// zones' verdicts, AbuseIPDB's scores from abuseIPDBCheckURL and its
|
// zones' verdicts, AbuseIPDB's scores from abuseIPDBCheckURL and its
|
||||||
// blacklist from abuseIPDBBlacklistURL, routewatch's answers while its
|
// blacklist from abuseIPDBBlacklistURL, routewatch's answers while its
|
||||||
// setting is set, and GeoJS at geojsURL, as sww builds
|
// setting is set, the metrics newMetrics returns, and GeoJS at geojsURL, as
|
||||||
// them from params' settings and the parts addCounters added. The metrics
|
// sww builds them from params' settings and the parts addCounters added.
|
||||||
// read the lists, the zones' verdicts, and AbuseIPDB's while its key is set.
|
|
||||||
func addOutsideSources(
|
func addOutsideSources(
|
||||||
params *waf.Params, geojsURL, abuseIPDBCheckURL string,
|
params *waf.Params, geojsURL, abuseIPDBCheckURL string,
|
||||||
) {
|
) {
|
||||||
cfg := params.Config
|
cfg := params.Config
|
||||||
m, limiter, ledger := params.Metrics, params.Limiter, params.Ledger
|
limiter, ledger := params.Limiter, params.Ledger
|
||||||
processLog, alertQueue := params.ProcessLog, params.Alerts
|
processLog, alertQueue := params.ProcessLog, params.Alerts
|
||||||
|
|
||||||
params.Lists = reputation.New(reputation.Params{
|
params.Lists = reputation.New(reputation.Params{
|
||||||
@@ -655,6 +669,9 @@ func addOutsideSources(
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
m := newMetrics(params)
|
||||||
|
params.Metrics = m
|
||||||
|
|
||||||
params.GeoJS = geojs.New(geojs.Params{
|
params.GeoJS = geojs.New(geojs.Params{
|
||||||
URL: geojsURL,
|
URL: geojsURL,
|
||||||
Answered: func(answer geojs.Answer) {
|
Answered: func(answer geojs.Answer) {
|
||||||
@@ -667,12 +684,29 @@ func addOutsideSources(
|
|||||||
},
|
},
|
||||||
WentWithoutAnswer: m.RequestsWithoutGeoJSAnswer.Inc,
|
WentWithoutAnswer: m.RequestsWithoutGeoJSAnswer.Inc,
|
||||||
})
|
})
|
||||||
|
}
|
||||||
|
|
||||||
m.AddReputation(params.Lists, params.DNSBL)
|
// newMetrics returns the metrics, as sww builds them from params' settings,
|
||||||
|
// read from the ledger, the table of clients, the rules, the lists, the
|
||||||
|
// zones' verdicts, and AbuseIPDB's while its key is set.
|
||||||
|
func newMetrics(params *waf.Params) *metrics.Metrics {
|
||||||
|
cfg := params.Config
|
||||||
|
metricsParams := metrics.Params{
|
||||||
|
BusiestCount: cfg.MetricsBusiestCountriesAndASNs,
|
||||||
|
InstanceName: cfg.InstanceName,
|
||||||
|
Now: params.Now,
|
||||||
|
Bans: params.Ledger,
|
||||||
|
Clients: params.Limiter,
|
||||||
|
Rules: params.Rules,
|
||||||
|
Lists: params.Lists,
|
||||||
|
DNSBL: params.DNSBL,
|
||||||
|
}
|
||||||
|
|
||||||
if cfg.AbuseIPDBKey != "" {
|
if cfg.AbuseIPDBKey != "" {
|
||||||
m.AddAbuseIPDB(params.AbuseIPDB)
|
metricsParams.AbuseIPDB = params.AbuseIPDB
|
||||||
}
|
}
|
||||||
|
|
||||||
|
return metrics.New(metricsParams)
|
||||||
}
|
}
|
||||||
|
|
||||||
// newClient returns an HTTP client that sends requests as they are made,
|
// newClient returns an HTTP client that sends requests as they are made,
|
||||||
@@ -794,7 +828,9 @@ func readResponse(t *testing.T, conn net.Conn) answer {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// wantLine checks the request log line's status and action.
|
// wantLine checks the request log line's status and action.
|
||||||
func wantLine(t *testing.T, line logLine, status int, action string) {
|
func wantLine(
|
||||||
|
t *testing.T, line logLine, status int, action requestlog.Action,
|
||||||
|
) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
if line.Status != status || line.Action != action {
|
if line.Status != status || line.Action != action {
|
||||||
|
|||||||
@@ -0,0 +1,235 @@
|
|||||||
|
package waf
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"net/http"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
|
||||||
|
)
|
||||||
|
|
||||||
|
// answerQuestion runs the checks on the visitor's request a question
|
||||||
|
// carries, and answers rproxy what to do with it. A denied request's counting
|
||||||
|
// is finished here, and an allowed one's by its report.
|
||||||
|
func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
|
||||||
|
question, ok := h.readQuestion(w, r)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
rq := h.startRequest(
|
||||||
|
w,
|
||||||
|
question.Visitor,
|
||||||
|
question.Client,
|
||||||
|
question.Peer,
|
||||||
|
true,
|
||||||
|
)
|
||||||
|
// These run even when a check panics. finish runs first, so that what is
|
||||||
|
// held has the finished log line. Go's server sends the answer only once
|
||||||
|
// this function has returned, so the report cannot come before it is held.
|
||||||
|
defer func() {
|
||||||
|
rq.finish()
|
||||||
|
rq.holdForReport()
|
||||||
|
}()
|
||||||
|
|
||||||
|
rq.site = r.Host
|
||||||
|
rq.line.WAFLine = requestlog.WAFLineDecision
|
||||||
|
refused := rq.check(r.Context())
|
||||||
|
rq.checked = time.Now()
|
||||||
|
underOwnPrefix := strings.HasPrefix(
|
||||||
|
r.URL.Path,
|
||||||
|
wafprotocol.OwnEndpointPrefix,
|
||||||
|
)
|
||||||
|
|
||||||
|
if refused == nil && !underOwnPrefix {
|
||||||
|
rq.allow()
|
||||||
|
// In observe mode, what enforce mode would have refused counts here.
|
||||||
|
rq.runBanningHeuristics(r.Context(), rq.h.afterAnswer)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if refused != nil {
|
||||||
|
rq.deny(*refused)
|
||||||
|
} else {
|
||||||
|
// The app never sees this prefix. waf's own endpoints are at its
|
||||||
|
// own address, so this answers as a path with no endpoint does.
|
||||||
|
rq.line.Action = requestlog.ActionOwnEndpoint
|
||||||
|
wafprotocol.SetAnswerHeaders(
|
||||||
|
rq.out.Header(),
|
||||||
|
wafprotocol.Answer{Disposition: wafprotocol.DispositionDeny},
|
||||||
|
)
|
||||||
|
http.Error(
|
||||||
|
rq.out,
|
||||||
|
http.StatusText(http.StatusNotFound),
|
||||||
|
http.StatusNotFound,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
rq.banAndCount(r.Context(), rq.h.afterAnswer)
|
||||||
|
}
|
||||||
|
|
||||||
|
// readQuestion reads a question, and reports whether waf takes it. It answers
|
||||||
|
// a question it does not take with 400 and no disposition. rproxy passes that
|
||||||
|
// 400 to the visitor, and SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION does
|
||||||
|
// not apply. The id of a question waf takes is held from here on, without a
|
||||||
|
// decision yet, so that another question with that id is refused while this
|
||||||
|
// one is being handled.
|
||||||
|
//
|
||||||
|
// A question whose body stops arriving gets no answer, and is not looked at.
|
||||||
|
func (h *handler) readQuestion(
|
||||||
|
w http.ResponseWriter, r *http.Request,
|
||||||
|
) (wafprotocol.Question, bool) {
|
||||||
|
question, err := wafprotocol.ReadQuestion(w, r)
|
||||||
|
if errors.Is(err, wafprotocol.ErrBodyStoppedArriving) {
|
||||||
|
h.processLog.Warn(
|
||||||
|
"a question's body stopped arriving",
|
||||||
|
"error",
|
||||||
|
err.Error(),
|
||||||
|
)
|
||||||
|
// This panic has Go's server close the connection without an answer,
|
||||||
|
// and log nothing.
|
||||||
|
panic(http.ErrAbortHandler)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err != nil {
|
||||||
|
h.refuseQuestion(w, err.Error())
|
||||||
|
|
||||||
|
return question, false
|
||||||
|
}
|
||||||
|
|
||||||
|
held, _ := h.awaitingReport.ContainsOrAdd(question.RequestID, nil)
|
||||||
|
if held {
|
||||||
|
h.refuseQuestion(
|
||||||
|
w,
|
||||||
|
wafprotocol.RequestIDHeader+" is the id of a request waf holds",
|
||||||
|
)
|
||||||
|
|
||||||
|
return question, false
|
||||||
|
}
|
||||||
|
|
||||||
|
return question, true
|
||||||
|
}
|
||||||
|
|
||||||
|
// refuseQuestion answers a question waf does not take, for reason.
|
||||||
|
func (h *handler) refuseQuestion(w http.ResponseWriter, reason string) {
|
||||||
|
h.processLog.Warn("a question is refused", "reason", reason)
|
||||||
|
http.Error(w, reason, http.StatusBadRequest)
|
||||||
|
}
|
||||||
|
|
||||||
|
// allow answers a question with wafprotocol.DispositionAllow and, while
|
||||||
|
// SWW_WAF_ADD_LOOKUP_HEADERS is true, the client's AS number and country.
|
||||||
|
func (rq *request) allow() {
|
||||||
|
rq.allowed = true
|
||||||
|
|
||||||
|
answer := wafprotocol.Answer{Disposition: wafprotocol.DispositionAllow}
|
||||||
|
if rq.h.config.AddLookupHeaders {
|
||||||
|
answer.ASN = rq.lookupAnswer.ASN
|
||||||
|
answer.Country = rq.lookupAnswer.Country
|
||||||
|
}
|
||||||
|
|
||||||
|
wafprotocol.SetAnswerHeaders(rq.out.Header(), answer)
|
||||||
|
rq.out.WriteHeader(http.StatusOK)
|
||||||
|
}
|
||||||
|
|
||||||
|
// holdForReport holds a copy of what the report's steps need, by the request's
|
||||||
|
// id. A report only ever reaches that copy, never the request being handled.
|
||||||
|
func (rq *request) holdForReport() {
|
||||||
|
held := rq.decision
|
||||||
|
rq.h.awaitingReport.Add(rq.line.RequestID, &held)
|
||||||
|
}
|
||||||
|
|
||||||
|
// deny answers a question with what the visitor is to get for r.
|
||||||
|
func (rq *request) deny(r refusal) {
|
||||||
|
rq.refused = &r
|
||||||
|
|
||||||
|
if r.status == 0 {
|
||||||
|
// SWW_WAF_BAN_RESPONSE is close. Written past rq.out, since the log line's
|
||||||
|
// status stays 0: the visitor is sent nothing.
|
||||||
|
wafprotocol.SetAnswerHeaders(
|
||||||
|
rq.out.Header(),
|
||||||
|
wafprotocol.Answer{Disposition: wafprotocol.DispositionClose},
|
||||||
|
)
|
||||||
|
rq.out.ResponseWriter.WriteHeader(http.StatusOK)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
wafprotocol.SetAnswerHeaders(
|
||||||
|
rq.out.Header(),
|
||||||
|
wafprotocol.Answer{Disposition: wafprotocol.DispositionDeny},
|
||||||
|
)
|
||||||
|
http.Error(rq.out, http.StatusText(r.status), r.status)
|
||||||
|
}
|
||||||
|
|
||||||
|
// handleReport answers POST wafprotocol.ReportPath: it finishes the request
|
||||||
|
// the report names, if it holds that request's decision, and ignores the
|
||||||
|
// report otherwise.
|
||||||
|
func (h *handler) handleReport() http.HandlerFunc {
|
||||||
|
return func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
report, err := wafprotocol.ReadReport(w, r)
|
||||||
|
if err != nil {
|
||||||
|
h.processLog.Warn("a report is refused", "error", err.Error())
|
||||||
|
http.Error(w, "the body is not a report", http.StatusBadRequest)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
held, found := h.awaitingReport.Peek(report.RequestID)
|
||||||
|
// Remove reports true to one report only, of several that name the
|
||||||
|
// id. A question still being handled is held as nil.
|
||||||
|
if found && held != nil &&
|
||||||
|
h.awaitingReport.Remove(report.RequestID) {
|
||||||
|
h.finishWith(r.Context(), held, report)
|
||||||
|
}
|
||||||
|
|
||||||
|
w.WriteHeader(http.StatusNoContent)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// finishWith finishes a request with what its report says, and writes the
|
||||||
|
// request's last log line, which holds its decision's fields and the report's.
|
||||||
|
// For an allowed request it first runs the heuristics of onReport, and adds the
|
||||||
|
// request to its client's history, the anomaly counters and the metrics, with
|
||||||
|
// the status and bytes the report gives. A denied request's counting was
|
||||||
|
// finished at its decision. ctx is the report's own context.
|
||||||
|
func (h *handler) finishWith(
|
||||||
|
ctx context.Context,
|
||||||
|
held *decision,
|
||||||
|
report wafprotocol.Report,
|
||||||
|
) {
|
||||||
|
rq := &request{
|
||||||
|
decision: *held,
|
||||||
|
h: h,
|
||||||
|
out: &responseWriter{
|
||||||
|
status: report.Status,
|
||||||
|
bytes: report.ResponseBytes,
|
||||||
|
},
|
||||||
|
requestBytes: report.RequestBytes,
|
||||||
|
}
|
||||||
|
|
||||||
|
rq.line.WAFLine = requestlog.WAFLineReport
|
||||||
|
rq.line.Status = report.Status
|
||||||
|
rq.line.RequestBytes = report.RequestBytes
|
||||||
|
rq.line.ResponseBytes = report.ResponseBytes
|
||||||
|
rq.line.DurationTotalMS = report.DurationTotalMS
|
||||||
|
|
||||||
|
if rq.allowed {
|
||||||
|
rq.line.DurationUpstreamTotalMS = &report.DurationUpstreamTotalMS
|
||||||
|
|
||||||
|
rq.banAndCount(ctx, rq.h.onReport)
|
||||||
|
|
||||||
|
rq.h.metrics.RequestReported(
|
||||||
|
&rq.line,
|
||||||
|
time.Duration(report.DurationTotalMS*float64(time.Millisecond)),
|
||||||
|
time.Duration(
|
||||||
|
report.DurationUpstreamTotalMS*float64(time.Millisecond),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
rq.writeLine()
|
||||||
|
}
|
||||||
+43
-26
@@ -6,8 +6,10 @@ package waf
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"net/netip"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -15,31 +17,40 @@ import (
|
|||||||
// countedByRateLimits exempts it, by the client's limit percentages, which it
|
// countedByRateLimits exempts it, by the client's limit percentages, which it
|
||||||
// notes for the log line with the client's counts. A request that takes the
|
// notes for the log line with the client's counts. A request that takes the
|
||||||
// client over a rate limit, as its limit percentage lowers it, breaks it: it is
|
// client over a rate limit, as its limit percentage lowers it, breaks it: it is
|
||||||
// refused, and bans the client. A request exempt from the rate limits is exempt
|
// refused with SWW_WAF_BAN_RESPONSE, and bans the client. A request exempt
|
||||||
// from the byte limits too. Its figures are the window, the count and the
|
// from the rate limits is exempt from the byte limits too. Its figures are the
|
||||||
// limit.
|
// window, the count and the limit.
|
||||||
type rateLimits struct{}
|
type rateLimits struct {
|
||||||
|
limiter *ratelimit.Limiter
|
||||||
|
exemptNets []netip.Prefix // SWW_WAF_RATE_LIMIT_EXEMPT_NETS
|
||||||
|
exemptPaths []string // SWW_WAF_RATE_LIMIT_EXEMPT_PATHS
|
||||||
|
banResponse int // SWW_WAF_BAN_RESPONSE
|
||||||
|
ipv6GroupPrefix int // SWW_WAF_IPV6_GROUP_PREFIX
|
||||||
|
biasedThresholds biasedThresholds
|
||||||
|
banMaker banMaker
|
||||||
|
}
|
||||||
|
|
||||||
func (rateLimits) name() string { return "rate_limits" }
|
func (rateLimits) name() string { return "rate_limits" }
|
||||||
|
|
||||||
func (rateLimits) check(
|
func (r rateLimits) check(
|
||||||
_ context.Context,
|
_ context.Context,
|
||||||
rq *request,
|
rq *request,
|
||||||
now time.Time,
|
now time.Time,
|
||||||
) *verdict {
|
) *verdict {
|
||||||
rq.counted = rq.countedByRateLimits()
|
rq.counted = r.countedByRateLimits(rq)
|
||||||
if !rq.counted {
|
if !rq.counted {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
rq.rateLimitPercent, rq.bytesLimitPercent = rq.limitPercentages()
|
rq.rateLimitPercent, rq.bytesLimitPercent =
|
||||||
|
r.biasedThresholds.limitPercentages(rq)
|
||||||
rq.line.RateLimitPercent, rq.line.RateLimitPercentSetting =
|
rq.line.RateLimitPercent, rq.line.RateLimitPercentSetting =
|
||||||
rq.rateLimitPercent.logged()
|
rq.rateLimitPercent.logged()
|
||||||
rq.line.BytesLimitPercent, rq.line.BytesLimitPercentSetting =
|
rq.line.BytesLimitPercent, rq.line.BytesLimitPercentSetting =
|
||||||
rq.bytesLimitPercent.logged()
|
rq.bytesLimitPercent.logged()
|
||||||
|
|
||||||
counts, hit, over := rq.h.limiter.Count(
|
counts, hit, over := r.limiter.Count(
|
||||||
clientGroup(rq.h.config, rq.client),
|
clientGroup(r.ipv6GroupPrefix, rq.client),
|
||||||
now,
|
now,
|
||||||
rq.rateLimitPercent.percent,
|
rq.rateLimitPercent.percent,
|
||||||
)
|
)
|
||||||
@@ -49,28 +60,34 @@ func (rateLimits) check(
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
rq.banForLimit(now, hit, rq.h.config.BanResponse)
|
r.banMaker.banForLimit(rq, now, hit, r.banResponse)
|
||||||
|
|
||||||
return &verdict{
|
return &verdict{
|
||||||
action: requestlog.ActionClientIsRateLimited, figures: limitFigures(hit),
|
action: requestlog.ActionClientIsRateLimited,
|
||||||
|
status: r.banResponse,
|
||||||
|
figures: limitFigures(hit),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// overAWholeRateLimit reports whether the heuristic rate_limits will refuse the
|
// refusesWhateverTheLookupAnswers reports whether the heuristic rate_limits
|
||||||
// request at now, whatever its client's limit percentages: the rate limits
|
// will refuse rq at now, whatever its client's limit percentages, which the
|
||||||
// count it, and one more request from its client takes it over a whole rate
|
// lookup's answer can lower: the rate limits count it, and one more request
|
||||||
// limit. It counts nothing.
|
// from its client takes it over a whole rate limit. It counts nothing.
|
||||||
func (rq *request) overAWholeRateLimit(now time.Time) bool {
|
func (r rateLimits) refusesWhateverTheLookupAnswers(
|
||||||
return rq.countedByRateLimits() &&
|
rq *request,
|
||||||
rq.h.limiter.OverARateLimit(clientGroup(rq.h.config, rq.client), now)
|
now time.Time,
|
||||||
|
) bool {
|
||||||
|
return r.countedByRateLimits(rq) &&
|
||||||
|
r.limiter.OverARateLimit(
|
||||||
|
clientGroup(r.ipv6GroupPrefix, rq.client),
|
||||||
|
now,
|
||||||
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
// countedByRateLimits reports whether the rate limits count the request: its
|
// countedByRateLimits reports whether the rate limits count rq: its client is
|
||||||
// client is not in SWW_WAF_RATE_LIMIT_EXEMPT_NETS, and its path is not exempt
|
// not in SWW_WAF_RATE_LIMIT_EXEMPT_NETS, and its path is not exempt under
|
||||||
// under SWW_WAF_RATE_LIMIT_EXEMPT_PATHS.
|
// SWW_WAF_RATE_LIMIT_EXEMPT_PATHS.
|
||||||
func (rq *request) countedByRateLimits() bool {
|
func (r rateLimits) countedByRateLimits(rq *request) bool {
|
||||||
cfg := rq.h.config
|
return !isInside(rq.client, r.exemptNets) &&
|
||||||
|
!pathExempt(rq.in, r.exemptPaths)
|
||||||
return !isInside(rq.client, cfg.RateLimitExemptNets) &&
|
|
||||||
!pathExempt(rq.in, cfg.RateLimitExemptPaths)
|
|
||||||
}
|
}
|
||||||
Loaded 100 of 124 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user