Compare commits
41
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a0277879d7 | ||
|
|
2083584ee0 | ||
|
|
fe61d6b066 | ||
|
|
9a8ed745bd | ||
|
|
0a230c86ce | ||
|
|
40386dce6e | ||
|
|
433d5cc0a9 | ||
|
|
419ee1c441 | ||
|
|
aaa28f541b | ||
|
|
98d7432077 | ||
|
|
11ac8aee50 | ||
|
|
e7cc0adbdb | ||
|
|
805c64a256 | ||
|
|
967ccdf75e | ||
|
|
809d2664d7 | ||
|
|
01e6b5d47e | ||
|
|
5099448f79 | ||
|
|
eb48796f27 | ||
|
|
9f42460f70 | ||
|
|
9cb8f362bc | ||
|
|
48edab5dad | ||
|
|
dadd5768d5 | ||
|
|
e6a081134a | ||
|
|
4cb91a4870 | ||
|
|
b9972af6a0 | ||
|
|
c0d9d5787e | ||
|
|
8c9f20d346 | ||
|
|
caa3079847 | ||
|
|
a2fcc00ded | ||
|
|
3c6b0d519f | ||
|
|
f33d0a4fce | ||
|
|
0a2b2d809f | ||
|
|
23788920f7 | ||
|
|
8efd6339a6 | ||
|
|
a93243a294 | ||
|
|
89a60cf9fc | ||
|
|
a05a82b1af | ||
|
|
53d9bf6af6 | ||
|
|
3550956cbb | ||
|
|
2bea6516a8 | ||
|
|
f52b1f9ab5 |
No files matched your search
+14
@@ -161,6 +161,20 @@ RUN git checkout --quiet --detach b4b2c785308b1ce785b6155c7fe5f16879080193 \
|
|||||||
# ubuntu 26.04, 2026-09-27
|
# ubuntu 26.04, 2026-09-27
|
||||||
FROM ubuntu@sha256:f144425ff09be612d6d9ad965196e9cdc23dae1f42110a8a11a3e9a8198759f7
|
FROM ubuntu@sha256:f144425ff09be612d6d9ad965196e9cdc23dae1f42110a8a11a3e9a8198759f7
|
||||||
|
|
||||||
|
# The image's own labels, in place of the ones it would inherit from
|
||||||
|
# Ubuntu's. script/docker and script/cibuild pass the build arguments;
|
||||||
|
# a build that passes none labels the image unknown rather than leave
|
||||||
|
# Ubuntu's values.
|
||||||
|
ARG VERSION=unknown
|
||||||
|
ARG CREATED=unknown
|
||||||
|
ARG REVISION=unknown
|
||||||
|
LABEL org.opencontainers.image.title="smallwebwaf" \
|
||||||
|
org.opencontainers.image.description="A web application firewall for self-hosted web apps: sww rproxy in front of an app, and sww waf, which decides." \
|
||||||
|
org.opencontainers.image.version="${VERSION}" \
|
||||||
|
org.opencontainers.image.created="${CREATED}" \
|
||||||
|
org.opencontainers.image.revision="${REVISION}" \
|
||||||
|
org.opencontainers.image.source="https://git.eeqj.de/sneak/smallwebwaf"
|
||||||
|
|
||||||
# runit's install creates its _runit-log user with minsysusers, which
|
# runit's install creates its _runit-log user with minsysusers, which
|
||||||
# reads this file in place of runit's /usr/lib/sysusers.d/runit.conf.
|
# reads this file in place of runit's /usr/lib/sysusers.d/runit.conf.
|
||||||
# runit's line leaves out the shell, and minsysusers prints a Perl
|
# runit's line leaves out the shell, and minsysusers prints a Perl
|
||||||
|
|||||||
@@ -2,7 +2,20 @@
|
|||||||
|
|
||||||
`smallwebwaf` is an MIT-licensed web application firewall, written in Go by
|
`smallwebwaf` is an MIT-licensed web application firewall, written in Go by
|
||||||
[@sneak](https://sneak.berlin), for self-hosted web apps. It refuses the
|
[@sneak](https://sneak.berlin), for self-hosted web apps. It refuses the
|
||||||
requests of scrapers, scanners and other abusers before they reach an app.
|
requests of misbehaving scrapers, scanners and other abusers before they reach
|
||||||
|
an app.
|
||||||
|
|
||||||
|
With the default settings, a client is banned when it does any of these:
|
||||||
|
|
||||||
|
- makes more than 120 requests in a minute, 1500 in an hour or 6000 in a day;
|
||||||
|
- moves more than `50G` of request and response bodies in an hour, or `100G` in
|
||||||
|
a day;
|
||||||
|
- asks for a secret or backup file, such as `/.env` or `/.git/config`;
|
||||||
|
- sends a user agent that names a security scanner, such as `sqlmap` or `nikto`;
|
||||||
|
- has more than 30 requests in a minute refused as probes or attacks.
|
||||||
|
|
||||||
|
A scraper that does none of these is left alone. A scraper that waits 15 seconds
|
||||||
|
between requests stays under the rate limits.
|
||||||
|
|
||||||
## What it does
|
## What it does
|
||||||
|
|
||||||
@@ -10,10 +23,10 @@ A web application firewall reads each request on its way to a web app and
|
|||||||
refuses the ones that look like abuse. `smallwebwaf` is one program, `sww`,
|
refuses the ones that look like abuse. `smallwebwaf` is one program, `sww`,
|
||||||
shipped in one image, with two commands that do this together:
|
shipped in one image, with two commands that do this together:
|
||||||
|
|
||||||
- `sww rproxy` is a small reverse proxy in front of one app. traefik routes to
|
- `sww rproxy` is a small reverse proxy in front of one app. It sits behind a
|
||||||
it. It asks `waf` about each request, passes the allowed ones to the app, and
|
TLS-terminating proxy, such as traefik, nginx or Caddy. It asks `waf` about
|
||||||
passes the app's answers back. It keeps nothing between requests and writes
|
each request, passes the allowed ones to the app, and passes the app's answers
|
||||||
nothing to disk.
|
back. It writes nothing to disk.
|
||||||
- `sww waf` decides. It runs the checks and keeps the bans, the counters and
|
- `sww waf` decides. It runs the checks and keeps the bans, the counters and
|
||||||
each client's history. One `waf` serves many sites: a client banned through
|
each client's history. One `waf` serves many sites: a client banned through
|
||||||
one site is refused on all of them, and its requests on all of them count
|
one site is refused on all of them, and its requests on all of them count
|
||||||
@@ -22,7 +35,26 @@ shipped in one image, with two commands that do this together:
|
|||||||
The site keeps answering while `waf` is down, slow or restarting: without an
|
The site keeps answering while `waf` is down, slow or restarting: without an
|
||||||
answer in time, `rproxy` passes the request to the app.
|
answer in time, `rproxy` passes the request to the app.
|
||||||
|
|
||||||
It protects against:
|
`rproxy` also protects the app on its own, whether or not `waf` answers. With
|
||||||
|
the default settings, `rproxy`:
|
||||||
|
|
||||||
|
- answers `400` to a malformed request, such as a broken header line or a
|
||||||
|
missing `Host`;
|
||||||
|
- answers `405` to a method it does not pass on, such as `TRACE`;
|
||||||
|
- answers `431` to a request line and headers over 32 KiB together;
|
||||||
|
- gives the visitor 60 seconds to send the request line and headers, and then 60
|
||||||
|
seconds more for the whole request to reach the app;
|
||||||
|
- answers `413` to a request body over 100 MiB;
|
||||||
|
- gives the app's answer 30 minutes to reach the visitor, counted from the end
|
||||||
|
of the request;
|
||||||
|
- answers `502` in place of an app's answer whose body is over 1 GiB, or cuts it
|
||||||
|
off if it has started;
|
||||||
|
- closes a kept-open connection after 120 seconds idle.
|
||||||
|
|
||||||
|
`rproxy` does not limit the number of headers. `waf` limits the number of
|
||||||
|
headers and cookies, and the length of the request line and of each header line.
|
||||||
|
|
||||||
|
Together, `rproxy` and `waf` protect against:
|
||||||
|
|
||||||
- Scrapers that fetch every page of a site, all day. Each client gets a limit on
|
- Scrapers that fetch every page of a site, all day. Each client gets a limit on
|
||||||
requests and on body bytes per minute, per hour and per day. The defaults
|
requests and on body bytes per minute, per hour and per day. The defaults
|
||||||
@@ -43,9 +75,8 @@ It protects against:
|
|||||||
- give the clients of chosen networks and countries a fraction of every
|
- give the clients of chosen networks and countries a fraction of every
|
||||||
limit, so that they are banned sooner;
|
limit, so that they are banned sooner;
|
||||||
- refuse or limit clients on a blocklist, a DNS blocklist, or with a high
|
- refuse or limit clients on a blocklist, a DNS blocklist, or with a high
|
||||||
AbuseIPDB score.
|
AbuseIPDB score;
|
||||||
- Slow and oversized requests. Every request and every answer has a time limit
|
- ban the clients on AbuseIPDB's blacklist before they send a request.
|
||||||
and a size limit.
|
|
||||||
|
|
||||||
For the operator, it:
|
For the operator, it:
|
||||||
|
|
||||||
@@ -57,8 +88,7 @@ For the operator, it:
|
|||||||
|
|
||||||
Not built, and not planned:
|
Not built, and not planned:
|
||||||
|
|
||||||
- TLS and routing by host name. Both stay with traefik, the reverse proxy in
|
- TLS and routing by host name. Both stay with the TLS-terminating proxy.
|
||||||
front of `rproxy`.
|
|
||||||
- Browser challenges such as captchas.
|
- Browser challenges such as captchas.
|
||||||
- A web console, or a configuration file.
|
- A web console, or a configuration file.
|
||||||
- Defence against floods large enough to fill the host's network link.
|
- Defence against floods large enough to fill the host's network link.
|
||||||
@@ -90,20 +120,61 @@ accepts connections, whether or not `waf` answers; in `waf`'s container, once
|
|||||||
The usual setup is three containers on one private network:
|
The usual setup is three containers on one private network:
|
||||||
|
|
||||||
- the app;
|
- the app;
|
||||||
- `rproxy`, in front of the app, which traefik routes to:
|
- `rproxy`, in front of the app, which the TLS-terminating proxy forwards to:
|
||||||
`SWW_RPROXY_WAF_BACKEND` names `waf` and `SWW_RPROXY_UPSTREAM_URL` names the
|
`SWW_RPROXY_WAF_BACKEND` names `waf` and `SWW_RPROXY_UPSTREAM_URL` names the
|
||||||
app;
|
app;
|
||||||
- `waf`, with `SWW_COMMAND=waf` and a volume at `/var/lib/sww` for its state
|
- `waf`, with `SWW_COMMAND=waf` and a volume at `/var/lib/sww` for its state
|
||||||
files.
|
files.
|
||||||
|
|
||||||
Point traefik's labels at port 8080 of `rproxy`
|
Publish no port of the app or of `waf`. Anything on that network can reach the
|
||||||
(`traefik.http.services.<name>.loadbalancer.server.port=8080`). Publish no port
|
app around `rproxy`, and can ask `waf` questions, so the network holds only what
|
||||||
of the app or of `waf`. Anything on that network can reach the app around
|
you trust. Each further app adds its own container and its own `rproxy`, whose
|
||||||
`rproxy`, and can ask `waf` questions, so the network holds only what you trust.
|
|
||||||
Each further app adds its own container and its own `rproxy`, whose
|
|
||||||
`SWW_RPROXY_WAF_BACKEND` names the same `waf`. The "Smallest setup" example
|
`SWW_RPROXY_WAF_BACKEND` names the same `waf`. The "Smallest setup" example
|
||||||
below shows this shape.
|
below shows this shape.
|
||||||
|
|
||||||
|
### The TLS-terminating proxy
|
||||||
|
|
||||||
|
`rproxy` serves plain HTTP. It runs behind any TLS-terminating proxy, such as
|
||||||
|
traefik, nginx or Caddy. The TLS-terminating proxy must:
|
||||||
|
|
||||||
|
- forward each request to port 8080 of `rproxy` over plain HTTP;
|
||||||
|
- keep the visitor's `Host` header;
|
||||||
|
- add the visitor's address to `X-Forwarded-For`;
|
||||||
|
- set `X-Forwarded-Proto` to the scheme the visitor used;
|
||||||
|
- close an unused connection to `rproxy` in under 120 seconds;
|
||||||
|
- connect to `rproxy` from an address in `SWW_RPROXY_TRUSTED_PROXIES`, by
|
||||||
|
default `10.0.0.0/8`, `172.16.0.0/12` or `192.168.0.0/16`.
|
||||||
|
|
||||||
|
From any other address, `rproxy` ignores the forwarded headers and takes the
|
||||||
|
TLS-terminating proxy for the visitor. Every visitor then shares one address,
|
||||||
|
and one ban.
|
||||||
|
|
||||||
|
traefik and Caddy keep the `Host` header and set both forwarded headers by
|
||||||
|
default. traefik's Docker label
|
||||||
|
`traefik.http.services.<name>.loadbalancer.server.port=8080` names the port.
|
||||||
|
Caddy keeps an unused connection for 120 seconds by default, so its site
|
||||||
|
shortens that to 90 seconds:
|
||||||
|
|
||||||
|
```caddyfile
|
||||||
|
reverse_proxy rproxy:8080 {
|
||||||
|
transport http {
|
||||||
|
keepalive 90s
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
By default nginx sends the host in `proxy_pass` as `Host` and sets neither
|
||||||
|
forwarded header, so its site sets all three:
|
||||||
|
|
||||||
|
```nginx
|
||||||
|
location / {
|
||||||
|
proxy_pass http://rproxy:8080;
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
### `rproxy` in the app's container
|
### `rproxy` in the app's container
|
||||||
|
|
||||||
An app's image can instead be built on the `smallwebwaf` image, so that `rproxy`
|
An app's image can instead be built on the `smallwebwaf` image, so that `rproxy`
|
||||||
@@ -154,13 +225,13 @@ entrypoint starts `rproxy` and the app side by side, each as its own user. Run
|
|||||||
the app's image with `SWW_RPROXY_WAF_BACKEND` set. `deploy/example-app` is such
|
the app's image with `SWW_RPROXY_WAF_BACKEND` set. `deploy/example-app` is such
|
||||||
an app, and the "`rproxy` beside the app" example below runs one.
|
an app, and the "`rproxy` beside the app" example below runs one.
|
||||||
|
|
||||||
The app sees the headers traefik sent, with traefik's address added to
|
The app's TCP peer is `rproxy`. The app sees the headers the TLS-terminating
|
||||||
`X-Forwarded-For`, from a TCP peer that is `rproxy`. So the app must trust
|
proxy sent, with that proxy's address added to `X-Forwarded-For`. So the app
|
||||||
`rproxy`'s address for forwarded headers. In a container of its own, `rproxy` is
|
must trust `rproxy`'s address for forwarded headers. In a container of its own,
|
||||||
on the private network, which the usual default of a trusted-proxy setting
|
`rproxy` is on the private network, which the usual default of a trusted-proxy
|
||||||
covers. Beside the app, its address is `127.0.0.1` or `::1`, which the private
|
setting covers. Beside the app, its address is `127.0.0.1` or `::1`, which the
|
||||||
ranges leave out, so the app's own setting must name them too, as `app.run`
|
private ranges leave out, so the app's own setting must name them too, as
|
||||||
does.
|
`app.run` does.
|
||||||
|
|
||||||
### Checking that it works
|
### Checking that it works
|
||||||
|
|
||||||
@@ -168,7 +239,7 @@ does.
|
|||||||
`healthy` for each container of `smallwebwaf`.
|
`healthy` for each container of `smallwebwaf`.
|
||||||
- A log line with `"msg":"starting"` gives the version, the address and every
|
- A log line with `"msg":"starting"` gives the version, the address and every
|
||||||
setting in effect, with secrets masked.
|
setting in effect, with secrets masked.
|
||||||
- A request to the site through traefik gets the app's answer.
|
- A request to the site through the TLS-terminating proxy gets the app's answer.
|
||||||
`https://<site>/_sww/healthz` answers `ok`. `waf`'s log lines about the
|
`https://<site>/_sww/healthz` answers `ok`. `waf`'s log lines about the
|
||||||
request have `"action":"forward"`.
|
request have `"action":"forward"`.
|
||||||
- To see a refusal without banning yourself, run `waf` with
|
- To see a refusal without banning yourself, run `waf` with
|
||||||
@@ -214,129 +285,136 @@ account, a key, a token or an alert destination stays off until you give one.
|
|||||||
`SWW_WAF_BYTES_COUNTED` says. Headers are never counted. What the visitor sent
|
`SWW_WAF_BYTES_COUNTED` says. Headers are never counted. What the visitor sent
|
||||||
over a WebSocket counts as request bytes, and what the app sent as response
|
over a WebSocket counts as request bytes, and what the app sent as response
|
||||||
bytes.
|
bytes.
|
||||||
|
- An anomaly is more requests or bytes than an `SWW_WAF_ANOMALY_` threshold
|
||||||
|
allows. Crossing a threshold sends one `anomaly` alert to the alert
|
||||||
|
destinations, and does nothing else: no client is banned, rate limited or
|
||||||
|
blocked.
|
||||||
- An invalid setting stops the process at start with one log line naming the
|
- An invalid setting stops the process at start with one log line naming the
|
||||||
variable and what is wrong with the value. The settings in effect are logged
|
variable and what is wrong with the value. The settings in effect are logged
|
||||||
on one line at start, with tokens, keys and webhook paths shown as `********`.
|
on one line at start, with tokens, keys and webhook paths shown as `********`.
|
||||||
|
|
||||||
Every setting the code reads, with its default:
|
Every setting the code reads, with its default:
|
||||||
|
|
||||||
| Variable | Default | What it does |
|
| Variable | Default | What it does |
|
||||||
| ------------------------------------------------- | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
| --------------------------------------------------- | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||||
| `SWW_COMMAND` | `rproxy` | Which command the image runs, and its health check checks: `rproxy` or `waf`. |
|
| `SWW_COMMAND` | `rproxy` | Which command the image runs, and its health check checks: `rproxy` or `waf`. |
|
||||||
| `SWW_RPROXY_WAF_BACKEND` | required | `waf`'s URL, such as `http://waf:8080`. |
|
| `SWW_RPROXY_WAF_BACKEND` | required | `waf`'s URL, such as `http://waf:8080`. |
|
||||||
| `SWW_RPROXY_UPSTREAM_URL` | `http://127.0.0.1:8081` | The app's URL: a scheme, a host and an optional port, nothing more. |
|
| `SWW_RPROXY_UPSTREAM_URL` | `http://127.0.0.1:8081` | The app's URL: a scheme, a host and an optional port, nothing more. |
|
||||||
| `SWW_RPROXY_LISTEN_ADDR` | `:8080` | Where `rproxy` listens; leave the host part empty. |
|
| `SWW_RPROXY_LISTEN_ADDR` | `:8080` | Where `rproxy` listens; leave the host part empty. |
|
||||||
| `SWW_RPROXY_TRUSTED_PROXIES` | `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16` | The netblocks whose `X-Forwarded-For` `rproxy` believes; set but empty, it believes none. |
|
| `SWW_RPROXY_TRUSTED_PROXIES` | `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16` | The netblocks whose `X-Forwarded-For` `rproxy` believes; set but empty, it believes none. |
|
||||||
| `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS` | `2000` | How long `rproxy` waits for `waf`'s answer to a question and to a report, in milliseconds. |
|
| `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS` | `2000` | How long `rproxy` waits for `waf`'s answer to a question and to a report, in milliseconds. |
|
||||||
| `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` | `allow` | What `rproxy` does with a request when `waf` cannot be reached or does not answer in time: `allow` passes it to the app, `deny` answers `503`. |
|
| `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` | `allow` | What `rproxy` does with a request when `waf` gives no answer in time, and while the circuit breaker for asks to `waf` is open: `allow` passes it to the app, `deny` answers `503`. |
|
||||||
| `SWW_RPROXY_MAX_REQUEST_SIZE` | `100M` | The largest request body, a size or `off`; a larger one is answered `413`. |
|
| `SWW_RPROXY_MAX_REQUEST_SIZE` | `100M` | The largest request body, a size or `off`; a larger one is answered `413`. |
|
||||||
| `SWW_RPROXY_MAX_RESPONSE_SIZE` | `1G` | The largest response body, a size or `off`. The visitor gets `502` in place of a larger one, or it is cut off if it has already started. |
|
| `SWW_RPROXY_MAX_RESPONSE_SIZE` | `1G` | The largest response body, a size or `off`. The visitor gets `502` in place of a larger one, or it is cut off if it has already started. |
|
||||||
| `SWW_RPROXY_MAX_REQUEST_TIME_MS` | `60000` | Two time limits, each this long, in milliseconds or `off`: one for the visitor to send its request line and headers, then one for the whole request to reach the app. |
|
| `SWW_RPROXY_MAX_REQUEST_TIME_MS` | `60000` | Two time limits, each this long, in milliseconds or `off`: one for the visitor to send its request line and headers, then one for the whole request to reach the app. |
|
||||||
| `SWW_RPROXY_MAX_RESPONSE_TIME_MS` | `1800000` | How long the answer may take, from the end of the request to its last byte reaching the visitor, in milliseconds or `off`. |
|
| `SWW_RPROXY_MAX_RESPONSE_TIME_MS` | `1800000` | How long the answer may take, from the end of the request to its last byte reaching the visitor, in milliseconds or `off`. |
|
||||||
| `SWW_WAF_LISTEN_ADDR` | `:8080` | Where `waf` listens; leave the host part empty. |
|
| `SWW_WAF_LISTEN_ADDR` | `:8080` | Where `waf` listens; leave the host part empty. |
|
||||||
| `SWW_WAF_INSTANCE_NAME` | the container's host name | The name in every line `waf` logs, and in every alert and metric, such as `fsn1/waf`. |
|
| `SWW_WAF_INSTANCE_NAME` | the container's host name | The name in every line `waf` logs, and in every alert and metric, such as `host1/waf`. |
|
||||||
| `SWW_WAF_MODE` | `enforce` | `observe` refuses nothing and makes no ban; each log line says what `enforce` would have done. |
|
| `SWW_WAF_MODE` | `enforce` | `observe` refuses nothing and makes no ban; each log line says what `enforce` would have done. |
|
||||||
| `SWW_WAF_IPV6_GROUP_PREFIX` | `64` | The prefix length of the IPv6 netblock that counts as one client, from 32 to 128. |
|
| `SWW_WAF_IPV6_GROUP_PREFIX` | `64` | The prefix length of the IPv6 netblock that counts as one client, from 32 to 128. |
|
||||||
| `SWW_WAF_MAX_TRACKED_CLIENTS` | `20000` | The most clients kept, in memory and in `clients.json`; the least recently seen is dropped first. |
|
| `SWW_WAF_MAX_TRACKED_CLIENTS` | `20000` | The most clients kept, in memory and in `clients.json`; the least recently seen is dropped first. |
|
||||||
| `SWW_WAF_MAX_BANS` | `5000` | The most bans `waf` keeps of its own; bans an admin made do not count. |
|
| `SWW_WAF_MAX_BANS` | `5000` | The most bans `waf` keeps for broken limits and attacks; bans an admin made and bans for AbuseIPDB's blacklist do not count. |
|
||||||
| `SWW_WAF_ALLOW_NETS` | empty | Netblocks whose clients skip every check, such as your monitoring. |
|
| `SWW_WAF_ALLOW_NETS` | empty | Netblocks whose clients skip every check, such as your monitoring. |
|
||||||
| `SWW_WAF_RATE_LIMIT_EXEMPT_NETS` | empty | Netblocks whose clients are exempt from rate limiting and the byte limits. |
|
| `SWW_WAF_RATE_LIMIT_EXEMPT_NETS` | empty | Netblocks whose clients are exempt from rate limiting and the byte limits. |
|
||||||
| `SWW_WAF_DENY_NETS` | empty | Netblocks whose clients are refused. |
|
| `SWW_WAF_DENY_NETS` | empty | Netblocks whose clients are refused. |
|
||||||
| `SWW_WAF_RATE_LIMIT_PER_MINUTE` | `120` | The most requests a client may make in a minute, a count or `off`; the next request is refused and bans the client. |
|
| `SWW_WAF_RATE_LIMIT_PER_MINUTE` | `120` | The most requests a client may make in a minute, a count or `off`; the next request is refused and bans the client. |
|
||||||
| `SWW_WAF_RATE_LIMIT_PER_HOUR` | `1500` | The most requests a client may make in an hour, a count or `off`; the next request is refused and bans the client. |
|
| `SWW_WAF_RATE_LIMIT_PER_HOUR` | `1500` | The most requests a client may make in an hour, a count or `off`; the next request is refused and bans the client. |
|
||||||
| `SWW_WAF_RATE_LIMIT_PER_DAY` | `6000` | The most requests a client may make in a day, a count or `off`; the next request is refused and bans the client. |
|
| `SWW_WAF_RATE_LIMIT_PER_DAY` | `6000` | The most requests a client may make in a day, a count or `off`; the next request is refused and bans the client. |
|
||||||
| `SWW_WAF_RATE_LIMIT_EXEMPT_PATHS` | empty | Path prefixes exempt from rate limiting and the byte limits, such as `/assets/`. |
|
| `SWW_WAF_RATE_LIMIT_EXEMPT_PATHS` | empty | Path prefixes exempt from rate limiting and the byte limits, such as `/assets/`. |
|
||||||
| `SWW_WAF_BYTES_LIMIT_PER_MINUTE` | `off` | The most bytes counted for a client in a minute, a size or `off`. `SWW_WAF_BYTES_COUNTED` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. |
|
| `SWW_WAF_BYTES_LIMIT_PER_MINUTE` | `off` | The most bytes counted for a client in a minute, a size or `off`. `SWW_WAF_BYTES_COUNTED` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. |
|
||||||
| `SWW_WAF_BYTES_LIMIT_PER_HOUR` | `50G` | The most bytes counted for a client in an hour, a size or `off`. `SWW_WAF_BYTES_COUNTED` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. |
|
| `SWW_WAF_BYTES_LIMIT_PER_HOUR` | `50G` | The most bytes counted for a client in an hour, a size or `off`. `SWW_WAF_BYTES_COUNTED` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. |
|
||||||
| `SWW_WAF_BYTES_LIMIT_PER_DAY` | `100G` | The most bytes counted for a client in a day, a size or `off`. `SWW_WAF_BYTES_COUNTED` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. |
|
| `SWW_WAF_BYTES_LIMIT_PER_DAY` | `100G` | The most bytes counted for a client in a day, a size or `off`. `SWW_WAF_BYTES_COUNTED` says which bytes count: response bodies, request bodies or both. The request that passes the limit is not cut off, and the client is banned when its response ends. |
|
||||||
| `SWW_WAF_BYTES_COUNTED` | `both` | What the byte limits and the byte anomaly thresholds count: `response`, the app's response bodies; `request`, the visitors' request bodies; or `both`. What the visitor sent over a WebSocket counts as request bytes, and what the app sent as response bytes. |
|
| `SWW_WAF_BYTES_COUNTED` | `both` | What the byte limits and the byte anomaly thresholds count: `response`, the app's response bodies; `request`, the visitors' request bodies; or `both`. What the visitor sent over a WebSocket counts as request bytes, and what the app sent as response bytes. |
|
||||||
| `SWW_WAF_REFUSALS_LIMIT_PER_MINUTE` | `30` | The most refusals a client may get in a minute from a rule file, a trap path or the Core Rule Set, a count or `off`. The next one bans the client. |
|
| `SWW_WAF_REFUSALS_LIMIT_PER_MINUTE` | `30` | The most refusals a client may get in a minute from a rule file, a trap path or the Core Rule Set, a count or `off`. The next one bans the client. |
|
||||||
| `SWW_WAF_BAN_RESPONSE` | `403` | How a refused client is answered: `403`, `429` or `close`, which drops the connection. |
|
| `SWW_WAF_BAN_RESPONSE` | `403` | How a refused client is answered: `403`, `429` or `close`, which drops the connection. |
|
||||||
| `SWW_WAF_LIMIT_BAN_DURATION` | `1h` | The ban for a first broken limit. |
|
| `SWW_WAF_LIMIT_BAN_DURATION` | `1h` | The ban for a first broken limit. |
|
||||||
| `SWW_WAF_LIMIT_BAN_REPEAT_WINDOW` | `24h` | A client that breaks a limit again within this time after its last such ban ended gets a ban three times as long. |
|
| `SWW_WAF_LIMIT_BAN_REPEAT_WINDOW` | `24h` | A client that breaks a limit again within this time after its last such ban ended gets a ban three times as long. |
|
||||||
| `SWW_WAF_LIMIT_BAN_PERMANENT_ABOVE` | `7d` | A ban for a broken limit that would be longer than this is permanent instead. |
|
| `SWW_WAF_LIMIT_BAN_PERMANENT_ABOVE` | `7d` | A ban for a broken limit that would be longer than this is permanent instead. |
|
||||||
| `SWW_WAF_ATTACK_BAN_DURATION` | `7d` | The ban for a first clear sign of attack: a `ban` rule or a trap path. |
|
| `SWW_WAF_ATTACK_BAN_DURATION` | `7d` | The ban for a first clear sign of attack: a `ban` rule or a trap path. |
|
||||||
| `SWW_WAF_BAN_SCOPE_V4_PREFIX` | `32` | The prefix length of the netblock a ban on an IPv4 client covers, from 0 to 32. |
|
| `SWW_WAF_BAN_SCOPE_V4_PREFIX` | `32` | The prefix length of the netblock a ban on an IPv4 client covers, from 0 to 32. |
|
||||||
| `SWW_WAF_LOOKUP_SOURCE` | `geojs` | Where each client's AS number and country come from: `geojs` (a web service), `file` (a local database) or `off`. |
|
| `SWW_WAF_LOOKUP_SOURCE` | `geojs` | Where each client's AS number and country come from: `geojs` (a web service), `file` (a local database) or `off`. |
|
||||||
| `SWW_WAF_LOOKUP_DB_PATH` | empty | The IPinfo Lite database file, in its `.mmdb` form, read while the source is `file`. |
|
| `SWW_WAF_LOOKUP_DB_PATH` | empty | The IPinfo Lite database file, in its `.mmdb` form, read while the source is `file`. |
|
||||||
| `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` | `800` | How long a request waits for its client's first GeoJS answer when a setting needs it, and how long a request to GeoJS may take, in milliseconds. |
|
| `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` | `800` | How long a request waits for its client's first GeoJS answer when a setting needs it, in milliseconds. The request to GeoJS goes on, and a later answer is kept. |
|
||||||
| `SWW_WAF_ADD_LOOKUP_HEADERS` | `false` | `true` passes the app the client's AS number and country in `X-Client-ASN` and `X-Client-Country`. |
|
| `SWW_WAF_ADD_LOOKUP_HEADERS` | `true`, or `false` while the lookup source is `off` | `true` passes the app the client's AS number and country in `X-Client-ASN` and `X-Client-Country`. |
|
||||||
| `SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES` | empty | When set, the only countries whose clients get through; a client the lookup cannot place is refused too. |
|
| `SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES` | empty | When set, the only countries whose clients get through; a client the lookup cannot place is refused too. |
|
||||||
| `SWW_WAF_TAG_LISTS` | empty | Address lists that each add a tag to their clients' requests, as `name=URL` pairs, such as `torexit=https://example.com/tor.txt`. |
|
| `SWW_WAF_TAG_LISTS` | empty | Address lists that each add a tag to their clients' requests, as `name=URL` pairs, such as `torexit=https://example.com/tor.txt`. |
|
||||||
| `SWW_WAF_TAG_DENY` | empty | Tags whose requests are refused, such as `torexit,country=CN`. |
|
| `SWW_WAF_TAG_DENY` | empty | Tags whose requests are refused, such as `torexit,country=CN`. |
|
||||||
| `SWW_WAF_TAG_LIMIT_PERCENT` | empty | Tags whose clients get a percentage, from 1 to 99, of every request and byte limit, such as `vpn:50,asn=14061:25`; the lowest applies. |
|
| `SWW_WAF_TAG_LIMIT_PERCENT` | empty | Tags whose clients get a percentage, from 1 to 99, of every request and byte limit, such as `vpn:50,asn=14061:25`; the lowest applies. |
|
||||||
| `SWW_WAF_ROUTEWATCH_URL` | unset | The URL of a routewatch instance, which gives each client's `asn=` and `prefix=` tags. Unset, nothing asks routewatch. |
|
| `SWW_WAF_ROUTEWATCH_URL` | unset | The URL of a routewatch instance, which gives each client's `asn=` and `prefix=` tags. Unset, nothing asks routewatch. |
|
||||||
| `SWW_WAF_ASN_BYTES_LIMIT_PERCENT` | empty | A percentage of the byte limits alone for the AS numbers listed, in place of their `asn=` tag's percentage. |
|
| `SWW_WAF_ASN_BYTES_LIMIT_PERCENT` | empty | A percentage of the byte limits alone for the AS numbers listed, in place of their `asn=` tag's percentage. |
|
||||||
| `SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT` | empty | A percentage of the byte limits alone for the countries listed, in place of their `country=` tag's percentage. |
|
| `SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT` | empty | A percentage of the byte limits alone for the countries listed, in place of their `country=` tag's percentage. |
|
||||||
| `SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT` | `100` | The percentage of every limit a client without a country gets. |
|
| `SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT` | `100` | The percentage of every limit a client without a country gets. |
|
||||||
| `SWW_WAF_ASN_LIMIT_PERCENT_URL` | unset | The URL of a text file of `AS<number>:<percent>` lines, fetched like a blocklist; each line counts as an `asn=` tag in `SWW_WAF_TAG_LIMIT_PERCENT` (the lowest wins). |
|
| `SWW_WAF_ASN_LIMIT_PERCENT_URL` | unset | The URL of a text file of `AS<number>:<percent>` lines, fetched like a blocklist; each line counts as an `asn=` tag in `SWW_WAF_TAG_LIMIT_PERCENT` (the lowest wins). |
|
||||||
| `SWW_WAF_RULE_FILES_DIR` | `/etc/sww/rules.d` | The directory of the rule files, read at start and again when a file there changes. |
|
| `SWW_WAF_RULE_FILES_DIR` | `/etc/sww/rules.d` | The directory of the rule files, read at start and again when a file there changes. |
|
||||||
| `SWW_WAF_RULE_FILES_ENABLED` | `true` | `false` reads no rule file. |
|
| `SWW_WAF_RULE_FILES_ENABLED` | `true` | `false` reads no rule file. |
|
||||||
| `SWW_WAF_TRAP_PATHS` | empty | Paths only scanners ask for, such as `/wp-login.php,/xmlrpc.php`; a request for one bans the client. |
|
| `SWW_WAF_TRAP_PATHS` | empty | Paths only scanners ask for, such as `/wp-login.php,/xmlrpc.php`; a request for one bans the client. |
|
||||||
| `SWW_WAF_CORE_RULE_SET_MODE` | `block` | What the Core Rule Set does with a match: `block` refuses it with `403`, `detect` only logs and alerts it, `off` runs no rule. |
|
| `SWW_WAF_CORE_RULE_SET_MODE` | `block` | What the Core Rule Set does with a match: `block` refuses it with `403`, `detect` only logs and alerts it, `off` runs no rule. |
|
||||||
| `SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL` | `1` | The Core Rule Set's paranoia level, from 1 to 4; each level up finds more attacks and takes more ordinary requests for attacks. |
|
| `SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL` | `1` | The Core Rule Set's paranoia level, from 1 to 4; each level up finds more attacks and takes more ordinary requests for attacks. |
|
||||||
| `SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD` | `5` | The Core Rule Set score at which a request is a match, a whole number or `off`; one critical rule scores 5. |
|
| `SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD` | `5` | The Core Rule Set score at which a request is a match, a whole number or `off`; one critical rule scores 5. |
|
||||||
| `SWW_WAF_CORE_RULE_SET_DISABLED_RULES` | `920340,920420,920440,920640,930130,930140` | Core Rule Set rule ids switched off; a list you give replaces the default, so repeat these six to keep them off. |
|
| `SWW_WAF_CORE_RULE_SET_DISABLED_RULES` | `920340,920420,920440,920640,930130,930140` | Core Rule Set rule ids switched off; a list you give replaces the default, so repeat these six to keep them off. |
|
||||||
| `SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS` | empty | Path prefixes the Core Rule Set does not inspect. |
|
| `SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS` | empty | Path prefixes the Core Rule Set does not inspect. |
|
||||||
| `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` | `off` | How much of a request body the Core Rule Set reads, a size up to `64K`; `off` reads none. |
|
| `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` | `off` | How much of a request body the Core Rule Set reads, a size up to `8K`; `off` reads none. |
|
||||||
| `SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE` | unset | A file of Core Rule Set exclusions, as Coraza directives, that switch rules off for some fields or paths only. |
|
| `SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE` | unset | A file of Core Rule Set exclusions, as Coraza directives, that switch rules off for some fields or paths only. |
|
||||||
| `SWW_WAF_APPS` | unset | The apps each hostname serves, such as `git.example.com:gitea;blog.example.com:wordpress`; only those apps' fixes apply there. "Attack detection" lists the apps. Listing `forgejo` brings gitea's fixes too. Unset, every app's fixes apply to every hostname. |
|
| `SWW_WAF_APPS` | unset | The apps each hostname serves, such as `git.example.com:gitea;blog.example.com:wordpress`; only those apps' fixes apply there. "Attack detection" lists the apps. Listing `forgejo` brings gitea's fixes too. Unset, every app's fixes apply to every hostname. |
|
||||||
| `SWW_WAF_BLOCKLIST_URLS` | empty | Blocklists to fetch, text files of addresses and netblocks one to a line, such as `https://www.spamhaus.org/drop/drop.txt`. |
|
| `SWW_WAF_BLOCKLIST_URLS` | empty | Blocklists to fetch, text files of addresses and netblocks one to a line, such as `https://www.spamhaus.org/drop/drop.txt`. |
|
||||||
| `SWW_WAF_BLOCKLIST_REFRESH` | `24h` | How long after a list was last tried it is fetched again, `1h` or more. |
|
| `SWW_WAF_BLOCKLIST_REFRESH` | `24h` | How long after a list was last tried it is fetched again, `1h` or more. |
|
||||||
| `SWW_WAF_BLOCKLIST_ACTION` | `deny` | What a client on a blocklist gets: `deny`, `limit:<percent>` of every limit, or `log`. |
|
| `SWW_WAF_BLOCKLIST_ACTION` | `deny` | What a client on a blocklist gets: `deny`, `limit:<percent>` of every limit, or `log`. |
|
||||||
| `SWW_WAF_DNSBL_ZONES` | empty | DNS blocklists asked about each new client in the background, such as `dnsbl.dronebl.org`. |
|
| `SWW_WAF_DNSBL_ZONES` | empty | DNS blocklists asked about each new client in the background, such as `dnsbl.dronebl.org`. |
|
||||||
| `SWW_WAF_DNSBL_RESOLVER` | unset | The resolver the zones are asked through, an IP address with an optional port; unset, the host's resolver is used. |
|
| `SWW_WAF_DNSBL_RESOLVER` | unset | The resolver the zones are asked through, an IP address with an optional port; unset, the host's resolver is used. |
|
||||||
| `SWW_WAF_ABUSEIPDB_KEY` | unset | The key of your AbuseIPDB account; set, a client that has committed an offence is checked in the background. |
|
| `SWW_WAF_ABUSEIPDB_KEY` | unset | The key of your AbuseIPDB account. Set, a client that has committed an offence is checked in the background, and the clients on AbuseIPDB's blacklist are banned. |
|
||||||
| `SWW_WAF_ABUSEIPDB_MIN_SCORE` | `75` | The least AbuseIPDB abuse confidence score, from 0 to 100, that is a hit. |
|
| `SWW_WAF_ABUSEIPDB_MIN_SCORE` | `75` | The least AbuseIPDB abuse confidence score, from 0 to 100, that is a hit. |
|
||||||
| `SWW_WAF_ABUSEIPDB_DAILY_BUDGET` | `900` | The most AbuseIPDB checks made in a day. |
|
| `SWW_WAF_ABUSEIPDB_DAILY_BUDGET` | `900` | The most AbuseIPDB checks made in a day. |
|
||||||
| `SWW_WAF_DNSBL_ABUSEIPDB_ACTION` | `limit:25` | What a client listed by a DNS blocklist, or scored a hit by AbuseIPDB, gets: `deny`, `limit:<percent>` or `log`. |
|
| `SWW_WAF_DNSBL_ABUSEIPDB_ACTION` | `limit:25` | What a client listed by a DNS blocklist, or scored a hit by AbuseIPDB, gets: `deny`, `limit:<percent>` or `log`. |
|
||||||
| `SWW_WAF_IP_REPUTATION_CACHE_TTL` | `24h` | How long a DNS blocklist verdict or an AbuseIPDB score is used. |
|
| `SWW_WAF_IP_REPUTATION_CACHE_TTL` | `24h` | How long a DNS blocklist verdict or an AbuseIPDB score is used. |
|
||||||
| `SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS` | `2000` | How long a DNS blocklist query or an AbuseIPDB check may take, in milliseconds. |
|
| `SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS` | `2000` | How long a DNS blocklist query or an AbuseIPDB check may take, in milliseconds. |
|
||||||
| `SWW_WAF_STATE_DIR` | `/var/lib/sww` | The directory of the state files, an absolute path. |
|
| `SWW_WAF_ABUSEIPDB_THRESHOLD` | `90` | The least AbuseIPDB score, from 25 to 100, at which a client on its blacklist is banned. Do not set it below 50 to 75: lower scores list ordinary visitors. |
|
||||||
| `SWW_WAF_BANS_WRITE_DELAY` | `10s` | How long `bans.json` waits to be written after a ban is made, lifted or made permanent. |
|
| `SWW_WAF_ABUSEIPDB_BLACKLIST_REFRESH_INTERVAL_MINS` | `60` | How often AbuseIPDB's blacklist is fetched again, in minutes. |
|
||||||
| `SWW_WAF_STATE_WRITE_INTERVAL` | `15m` | How often every state and cache file is written. |
|
| `SWW_WAF_ABUSEIPDB_BAN_DAYS` | `30` | How long a client on AbuseIPDB's blacklist is banned, in days. |
|
||||||
| `SWW_WAF_CACHE_DIR` | `/var/cache/sww` | The directory of the cache files, an absolute path. |
|
| `SWW_WAF_STATE_DIR` | `/var/lib/sww` | The directory of the state files, an absolute path. |
|
||||||
| `SWW_WAF_LOG_LEVEL` | `info` | The least severe of `waf`'s own messages that are written: `debug`, `info`, `warn` or `error`; request lines are always written. |
|
| `SWW_WAF_BANS_WRITE_DELAY` | `10s` | How long `bans.json` waits to be written after a ban is made, lifted or made permanent. |
|
||||||
| `SWW_WAF_LOG_REQUEST_HEADERS` | `accept,accept-language,accept-encoding,content-type,origin,range` | The request headers whose values the request log records; `Authorization` and `Cookie` are never recorded. |
|
| `SWW_WAF_STATE_WRITE_INTERVAL` | `15m` | How often every state and cache file is written. |
|
||||||
| `SWW_WAF_LOG_REMOTE_URL` | unset | A syslog server that also gets every line `waf` writes: `syslog+udp://`, `syslog+tcp://` or `syslog+tls://` with a host and a port. |
|
| `SWW_WAF_CACHE_DIR` | `/var/cache/sww` | The directory of the cache files, an absolute path. |
|
||||||
| `SWW_WAF_LOG_REMOTE_TLS_CA_FILE` | unset | A file of PEM certificates a `syslog+tls` server's certificate must chain to, instead of the host's own. |
|
| `SWW_WAF_LOG_LEVEL` | `info` | The least severe of `waf`'s own messages that are written: `debug`, `info`, `warn` or `error`; request lines are always written. |
|
||||||
| `SWW_WAF_LOG_REMOTE_BUFFER_LINES` | `10000` | The most lines held while they wait to be sent; past it, the oldest is dropped. |
|
| `SWW_WAF_LOG_REQUEST_HEADERS` | `accept,accept-language,accept-encoding,content-type,origin,range` | The request headers whose values the request log records; `Authorization` and `Cookie` are never recorded. |
|
||||||
| `SWW_WAF_LOG_REMOTE_FACILITY` | `local0` | The syslog facility of each record sent. |
|
| `SWW_WAF_LOG_REMOTE_URL` | unset | A syslog server that also gets every line `waf` writes: `syslog+udp://`, `syslog+tcp://` or `syslog+tls://` with a host and a port. |
|
||||||
| `SWW_WAF_LOG_REMOTE_APP_NAME` | `SWW_WAF_INSTANCE_NAME` | The app name in each syslog record, 1 to 48 printable ASCII characters without a space. |
|
| `SWW_WAF_LOG_REMOTE_TLS_CA_FILE` | unset | A file of PEM certificates a `syslog+tls` server's certificate must chain to, instead of the host's own. |
|
||||||
| `SWW_WAF_ADMIN_TOKEN` | unset | The bearer token for the ban and client endpoints, at least 32 characters; unset, they answer `404`. |
|
| `SWW_WAF_LOG_REMOTE_BUFFER_LINES` | `10000` | The most lines held while they wait to be sent; past it, the oldest is dropped. |
|
||||||
| `SWW_WAF_METRICS_TOKEN` | unset | The bearer token for `/_sww/metrics`, at least 32 characters; unset, the metrics are off. |
|
| `SWW_WAF_LOG_REMOTE_FACILITY` | `local0` | The syslog facility of each record sent. |
|
||||||
| `SWW_WAF_METRICS_BUSIEST_COUNTRIES_AND_ASNS` | `50` | How many AS numbers and how many countries get metric series of their own; the rest are summed as `other`. |
|
| `SWW_WAF_LOG_REMOTE_APP_NAME` | `SWW_WAF_INSTANCE_NAME` | The app name in each syslog record, 1 to 48 printable ASCII characters without a space. |
|
||||||
| `SWW_WAF_ALERT_WEBHOOK_URL` | unset | A webhook each alert is posted to as a JSON object. |
|
| `SWW_WAF_ADMIN_TOKEN` | unset | The bearer token for the ban and client endpoints, at least 32 characters; unset, they answer `404`. |
|
||||||
| `SWW_WAF_ALERT_WEBHOOK_HEADERS` | empty | Headers sent with each post, a list of `Name:value`, such as `Authorization:Bearer <token>`. |
|
| `SWW_WAF_METRICS_TOKEN` | unset | The bearer token for `/_sww/metrics`, at least 32 characters; unset, the metrics are off. |
|
||||||
| `SWW_WAF_ALERT_SLACK_WEBHOOK_URL` | unset | A Slack incoming webhook each alert is posted to as a message. |
|
| `SWW_WAF_METRICS_BUSIEST_COUNTRIES_AND_ASNS` | `50` | How many AS numbers and how many countries get metric series of their own; the rest are summed as `other`. |
|
||||||
| `SWW_WAF_ALERT_NTFY_URL` | unset | An ntfy topic each alert is published to, as its full URL, such as `https://ntfy.sh/my-alerts`. |
|
| `SWW_WAF_ALERT_WEBHOOK_URL` | unset | A webhook each alert is posted to as a JSON object. |
|
||||||
| `SWW_WAF_ALERT_NTFY_TOKEN` | unset | An ntfy access token sent with each alert. |
|
| `SWW_WAF_ALERT_WEBHOOK_HEADERS` | empty | Headers sent with each post, a list of `Name:value`, such as `Authorization:Bearer <token>`. |
|
||||||
| `SWW_WAF_ALERT_EVENTS` | `ban,permanent_ban,core_rule_set_match,anomaly,reputation_hit,source_failure,file_error` | The events alerts are sent for; the hourly `summary` is sent whatever it says. |
|
| `SWW_WAF_ALERT_SLACK_WEBHOOK_URL` | unset | A Slack incoming webhook each alert is posted to as a message. |
|
||||||
| `SWW_WAF_ALERT_COOLDOWN` | `15m` | How long after an alert its repeats are held back and counted, a duration or `off`. |
|
| `SWW_WAF_ALERT_NTFY_URL` | unset | An ntfy topic each alert is published to, as its full URL, such as `https://ntfy.sh/my-alerts`. |
|
||||||
| `SWW_WAF_ALERT_MAX_PER_HOUR` | `60` | The most alerts sent in an hour, a number or `off`; the rest are counted in the hour's `summary`. |
|
| `SWW_WAF_ALERT_NTFY_TOKEN` | unset | An ntfy access token sent with each alert. |
|
||||||
| `SWW_WAF_ANOMALY_CLIENT_REQUESTS_PER_MINUTE` | `off` | Alert when one client makes more requests than this in a minute, a count. |
|
| `SWW_WAF_ALERT_EVENTS` | `ban,permanent_ban,core_rule_set_match,anomaly,reputation_hit,source_failure,file_error` | The events alerts are sent for; the hourly `summary` is sent whatever it says. |
|
||||||
| `SWW_WAF_ANOMALY_CLIENT_REQUESTS_PER_HOUR` | `off` | Alert when one client makes more requests than this in an hour, a count. |
|
| `SWW_WAF_ALERT_COOLDOWN` | `15m` | How long after an alert its repeats are held back and counted, a duration or `off`. |
|
||||||
| `SWW_WAF_ANOMALY_CLIENT_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for one client in a minute, a size. |
|
| `SWW_WAF_ALERT_MAX_PER_HOUR` | `60` | The most alerts sent in an hour, a number or `off`; the rest are counted in the hour's `summary`. |
|
||||||
| `SWW_WAF_ANOMALY_CLIENT_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for one client in an hour, a size. |
|
| `SWW_WAF_ANOMALY_CLIENT_REQUESTS_PER_MINUTE` | `off` | Sends one alert to the alert destinations when one client makes more requests than this in a minute, a count. No client is banned, rate limited or blocked. |
|
||||||
| `SWW_WAF_ANOMALY_NET_REQUESTS_PER_MINUTE` | `off` | Alert when the netblock around a client makes more requests than this in a minute, a count. |
|
| `SWW_WAF_ANOMALY_CLIENT_REQUESTS_PER_HOUR` | `off` | Sends one alert to the alert destinations when one client makes more requests than this in an hour, a count. No client is banned, rate limited or blocked. |
|
||||||
| `SWW_WAF_ANOMALY_NET_REQUESTS_PER_HOUR` | `off` | Alert when the netblock around a client makes more requests than this in an hour, a count. |
|
| `SWW_WAF_ANOMALY_CLIENT_BYTES_PER_MINUTE` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for one client in a minute, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
|
||||||
| `SWW_WAF_ANOMALY_NET_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for the netblock around a client in a minute, a size. |
|
| `SWW_WAF_ANOMALY_CLIENT_BYTES_PER_HOUR` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for one client in an hour, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
|
||||||
| `SWW_WAF_ANOMALY_NET_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for the netblock around a client in an hour, a size. |
|
| `SWW_WAF_ANOMALY_NET_REQUESTS_PER_MINUTE` | `off` | Sends one alert to the alert destinations when the netblock around a client makes more requests than this in a minute, a count. No client is banned, rate limited or blocked. |
|
||||||
| `SWW_WAF_ANOMALY_NET_V4_PREFIX` | `24` | The prefix length of the netblock around an IPv4 client that the `SWW_WAF_ANOMALY_NET_` thresholds count, from 0 to 32. |
|
| `SWW_WAF_ANOMALY_NET_REQUESTS_PER_HOUR` | `off` | Sends one alert to the alert destinations when the netblock around a client makes more requests than this in an hour, a count. No client is banned, rate limited or blocked. |
|
||||||
| `SWW_WAF_ANOMALY_NET_V6_PREFIX` | `48` | The prefix length of the netblock around an IPv6 client that the `SWW_WAF_ANOMALY_NET_` thresholds count, from 0 to 128. |
|
| `SWW_WAF_ANOMALY_NET_BYTES_PER_MINUTE` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for the netblock around a client in a minute, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
|
||||||
| `SWW_WAF_ANOMALY_ASN_REQUESTS_PER_MINUTE` | `off` | Alert when one AS number makes more requests than this in a minute, a count. |
|
| `SWW_WAF_ANOMALY_NET_BYTES_PER_HOUR` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for the netblock around a client in an hour, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
|
||||||
| `SWW_WAF_ANOMALY_ASN_REQUESTS_PER_HOUR` | `off` | Alert when one AS number makes more requests than this in an hour, a count. |
|
| `SWW_WAF_ANOMALY_NET_V4_PREFIX` | `24` | The prefix length of the netblock around an IPv4 client that the `SWW_WAF_ANOMALY_NET_` thresholds count, from 0 to 32. Those thresholds only send alerts. |
|
||||||
| `SWW_WAF_ANOMALY_ASN_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for one AS number in a minute, a size. |
|
| `SWW_WAF_ANOMALY_NET_V6_PREFIX` | `48` | The prefix length of the netblock around an IPv6 client that the `SWW_WAF_ANOMALY_NET_` thresholds count, from 0 to 128. Those thresholds only send alerts. |
|
||||||
| `SWW_WAF_ANOMALY_ASN_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for one AS number in an hour, a size. |
|
| `SWW_WAF_ANOMALY_ASN_REQUESTS_PER_MINUTE` | `off` | Sends one alert to the alert destinations when one AS number makes more requests than this in a minute, a count. No client is banned, rate limited or blocked. |
|
||||||
| `SWW_WAF_ANOMALY_TOTAL_REQUESTS_PER_MINUTE` | `off` | Alert when the whole service gets more requests than this in a minute, a count. |
|
| `SWW_WAF_ANOMALY_ASN_REQUESTS_PER_HOUR` | `off` | Sends one alert to the alert destinations when one AS number makes more requests than this in an hour, a count. No client is banned, rate limited or blocked. |
|
||||||
| `SWW_WAF_ANOMALY_TOTAL_REQUESTS_PER_HOUR` | `off` | Alert when the whole service gets more requests than this in an hour, a count. |
|
| `SWW_WAF_ANOMALY_ASN_BYTES_PER_MINUTE` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for one AS number in a minute, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
|
||||||
| `SWW_WAF_ANOMALY_TOTAL_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for the whole service in a minute, a size. |
|
| `SWW_WAF_ANOMALY_ASN_BYTES_PER_HOUR` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for one AS number in an hour, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
|
||||||
| `SWW_WAF_ANOMALY_TOTAL_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for the whole service in an hour, a size. |
|
| `SWW_WAF_ANOMALY_TOTAL_REQUESTS_PER_MINUTE` | `off` | Sends one alert to the alert destinations when the whole service gets more requests than this in a minute, a count. No client is banned, rate limited or blocked. |
|
||||||
| `SWW_WAF_ANOMALY_WATCHED_NETS` | empty | Named netblocks the `SWW_WAF_ANOMALY_WATCHED_NET_` thresholds count each as a whole, such as `office=203.0.113.0/24,scraper-x=198.51.100.0/22`. |
|
| `SWW_WAF_ANOMALY_TOTAL_REQUESTS_PER_HOUR` | `off` | Sends one alert to the alert destinations when the whole service gets more requests than this in an hour, a count. No client is banned, rate limited or blocked. |
|
||||||
| `SWW_WAF_ANOMALY_WATCHED_NET_REQUESTS_PER_MINUTE` | `off` | Alert when a watched netblock makes more requests than this in a minute, a count. |
|
| `SWW_WAF_ANOMALY_TOTAL_BYTES_PER_MINUTE` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for the whole service in a minute, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
|
||||||
| `SWW_WAF_ANOMALY_WATCHED_NET_REQUESTS_PER_HOUR` | `off` | Alert when a watched netblock makes more requests than this in an hour, a count. |
|
| `SWW_WAF_ANOMALY_TOTAL_BYTES_PER_HOUR` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for the whole service in an hour, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
|
||||||
| `SWW_WAF_ANOMALY_WATCHED_NET_BYTES_PER_MINUTE` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for a watched netblock in a minute, a size. |
|
| `SWW_WAF_ANOMALY_WATCHED_NETS` | empty | Named netblocks the `SWW_WAF_ANOMALY_WATCHED_NET_` thresholds count each as a whole, such as `office=203.0.113.0/24,scraper-x=198.51.100.0/22`. Those thresholds only send alerts. |
|
||||||
| `SWW_WAF_ANOMALY_WATCHED_NET_BYTES_PER_HOUR` | `off` | Alert when more than this many bytes, as `SWW_WAF_BYTES_COUNTED` counts them, are counted for a watched netblock in an hour, a size. |
|
| `SWW_WAF_ANOMALY_WATCHED_NET_REQUESTS_PER_MINUTE` | `off` | Sends one alert to the alert destinations when a watched netblock makes more requests than this in a minute, a count. No client is banned, rate limited or blocked. |
|
||||||
|
| `SWW_WAF_ANOMALY_WATCHED_NET_REQUESTS_PER_HOUR` | `off` | Sends one alert to the alert destinations when a watched netblock makes more requests than this in an hour, a count. No client is banned, rate limited or blocked. |
|
||||||
|
| `SWW_WAF_ANOMALY_WATCHED_NET_BYTES_PER_MINUTE` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for a watched netblock in a minute, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
|
||||||
|
| `SWW_WAF_ANOMALY_WATCHED_NET_BYTES_PER_HOUR` | `off` | Sends one alert to the alert destinations when more bytes than this are counted for a watched netblock in an hour, a size. `SWW_WAF_BYTES_COUNTED` says which bytes count. No client is banned, rate limited or blocked. |
|
||||||
|
|
||||||
[`docs/settings.md`](docs/settings.md) gives the values each setting takes in
|
[`docs/settings.md`](docs/settings.md) gives the values each setting takes in
|
||||||
full, and the limits that are fixed rather than settings.
|
full, and the limits that are fixed rather than settings.
|
||||||
@@ -346,8 +424,8 @@ full, and the limits that are fixed rather than settings.
|
|||||||
Each example is a complete `compose.yaml`. `smallwebwaf` is the tag
|
Each example is a complete `compose.yaml`. `smallwebwaf` is the tag
|
||||||
`make docker` gives the image; on another host, use the name you pushed it
|
`make docker` gives the image; on another host, use the name you pushed it
|
||||||
under, by digest. Docker Compose puts the services on a private network of its
|
under, by digest. Docker Compose puts the services on a private network of its
|
||||||
own, where each reaches the others by service name. Each example publishes
|
own, where each reaches the others by service name. Each example but "On the
|
||||||
`rproxy`'s port on the host, for trying it out.
|
open internet" publishes `rproxy`'s port on the host, for trying it out.
|
||||||
|
|
||||||
### Smallest setup
|
### Smallest setup
|
||||||
|
|
||||||
@@ -379,6 +457,142 @@ volumes:
|
|||||||
waf-state:
|
waf-state:
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### On the open internet
|
||||||
|
|
||||||
|
One gitea at `git.example.com`, with traefik in front of `rproxy`. traefik gets
|
||||||
|
the certificate from Let's Encrypt and is the only service with published ports.
|
||||||
|
`waf` sets what most sites set. More `waf` settings follow, commented out, each
|
||||||
|
with what it does and when to set it.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
services:
|
||||||
|
traefik:
|
||||||
|
# traefik v3.7.19, 2026-11-13
|
||||||
|
image: traefik@sha256:a6c0194dda77bcb804824580f36fa7edc56e315b1acfe933366c8b52844a78a4
|
||||||
|
restart: unless-stopped
|
||||||
|
command:
|
||||||
|
- --providers.docker=true
|
||||||
|
- --providers.docker.exposedByDefault=false
|
||||||
|
- --entryPoints.web.address=:80
|
||||||
|
- --entryPoints.websecure.address=:443
|
||||||
|
# Redirect plain HTTP to HTTPS.
|
||||||
|
- --entryPoints.web.http.redirections.entryPoint.to=websecure
|
||||||
|
- --entryPoints.web.http.redirections.entryPoint.scheme=https
|
||||||
|
# A Let's Encrypt certificate for each site, by the TLS-ALPN
|
||||||
|
# challenge on port 443.
|
||||||
|
- --entryPoints.websecure.http.tls.certResolver=le
|
||||||
|
- --certificatesResolvers.le.acme.email=hostmaster@example.com
|
||||||
|
- --certificatesResolvers.le.acme.storage=/letsencrypt/acme.json
|
||||||
|
- --certificatesResolvers.le.acme.tlsChallenge=true
|
||||||
|
# Give a request 10 minutes to arrive, rather than 1, for the
|
||||||
|
# large git pushes the rproxy lines below accept.
|
||||||
|
# - --entryPoints.websecure.transport.respondingTimeouts.readTimeout=600s
|
||||||
|
ports:
|
||||||
|
- "80:80"
|
||||||
|
- "443:443"
|
||||||
|
volumes:
|
||||||
|
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||||
|
- acme:/letsencrypt
|
||||||
|
|
||||||
|
app:
|
||||||
|
# Your app: here a gitea, which listens on port 3000.
|
||||||
|
image: registry.example/app@sha256:0000000000000000000000000000000000000000000000000000000000000000
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
rproxy:
|
||||||
|
image: smallwebwaf
|
||||||
|
restart: unless-stopped
|
||||||
|
environment:
|
||||||
|
SWW_RPROXY_WAF_BACKEND: http://waf:8080
|
||||||
|
SWW_RPROXY_UPSTREAM_URL: http://app:3000
|
||||||
|
# Accept git pushes up to 1 GiB, rather than 100 MiB, sent in
|
||||||
|
# up to 10 minutes, rather than 1, with traefik's readTimeout
|
||||||
|
# line above.
|
||||||
|
# SWW_RPROXY_MAX_REQUEST_SIZE: 1G
|
||||||
|
# SWW_RPROXY_MAX_REQUEST_TIME_MS: "600000"
|
||||||
|
labels:
|
||||||
|
- traefik.enable=true
|
||||||
|
- traefik.http.routers.git.rule=Host(`git.example.com`)
|
||||||
|
- traefik.http.routers.git.entryPoints=websecure
|
||||||
|
- traefik.http.services.git.loadBalancer.server.port=8080
|
||||||
|
|
||||||
|
waf:
|
||||||
|
image: smallwebwaf
|
||||||
|
restart: unless-stopped
|
||||||
|
environment:
|
||||||
|
SWW_COMMAND: waf
|
||||||
|
# The name in every log line, alert and metric. The default,
|
||||||
|
# the container's host name, is a random id under Compose.
|
||||||
|
SWW_WAF_INSTANCE_NAME: host1/waf
|
||||||
|
# The app each hostname serves. Only that app's fixes to the
|
||||||
|
# Core Rule Set apply there.
|
||||||
|
SWW_WAF_APPS: git.example.com:gitea
|
||||||
|
# Monitoring skips every check.
|
||||||
|
SWW_WAF_ALLOW_NETS: 192.0.2.10
|
||||||
|
# The office and the CI runner, which would exceed the rate
|
||||||
|
# limits, are never rate limited.
|
||||||
|
SWW_WAF_RATE_LIMIT_EXEMPT_NETS: 203.0.113.0/24,198.51.100.7
|
||||||
|
# WordPress paths, which only scanners ask a gitea for. One
|
||||||
|
# request for one bans the client.
|
||||||
|
SWW_WAF_TRAP_PATHS: /wp-login.php,/xmlrpc.php
|
||||||
|
# Spamhaus's list of netblocks hijacked or leased by
|
||||||
|
# criminals. Their clients are refused.
|
||||||
|
SWW_WAF_BLOCKLIST_URLS: https://www.spamhaus.org/drop/drop.txt
|
||||||
|
# The token for the ban endpoints, to lift a ban without
|
||||||
|
# editing bans.json.
|
||||||
|
SWW_WAF_ADMIN_TOKEN_FILE: /etc/sww/tokens/admin
|
||||||
|
# Alerts, such as one for each ban, to an ntfy topic.
|
||||||
|
SWW_WAF_ALERT_NTFY_URL: https://ntfy.example.com/smallwebwaf
|
||||||
|
|
||||||
|
# Refuse nothing and ban no one, for the first days. Each log
|
||||||
|
# line says what enforce mode would have done.
|
||||||
|
# SWW_WAF_MODE: observe
|
||||||
|
# The access token for an ntfy topic that is not public.
|
||||||
|
# SWW_WAF_ALERT_NTFY_TOKEN_FILE: /etc/sww/tokens/ntfy
|
||||||
|
# The token for /_sww/metrics, for Prometheus. Unset, the
|
||||||
|
# metrics are off.
|
||||||
|
# SWW_WAF_METRICS_TOKEN_FILE: /etc/sww/tokens/metrics
|
||||||
|
# 600 requests an hour rather than 1500, to ban slower
|
||||||
|
# scrapers. A fast human reader can reach it too.
|
||||||
|
# SWW_WAF_RATE_LIMIT_PER_HOUR: "600"
|
||||||
|
# A quarter of every limit for clients in two hosting
|
||||||
|
# networks, where scrapers run rather than people.
|
||||||
|
# SWW_WAF_TAG_LIMIT_PERCENT: asn=14061:25,asn=16276:25
|
||||||
|
# For a site only for people in a few countries. Clients
|
||||||
|
# elsewhere are refused, and so are any the lookup cannot place.
|
||||||
|
# SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES: de,at,ch
|
||||||
|
# Close a refused request's connection rather than answer
|
||||||
|
# 403. traefik then answers 502, so a scraper cannot tell a
|
||||||
|
# ban from a broken site.
|
||||||
|
# SWW_WAF_BAN_RESPONSE: close
|
||||||
|
# A copy of every log line to a syslog server.
|
||||||
|
# SWW_WAF_LOG_REMOTE_URL: syslog+tls://logs.example.com:6514
|
||||||
|
volumes:
|
||||||
|
- waf-state:/var/lib/sww
|
||||||
|
- /srv/sww/tokens:/etc/sww/tokens:ro
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
acme:
|
||||||
|
waf-state:
|
||||||
|
```
|
||||||
|
|
||||||
|
Change these before the first start:
|
||||||
|
|
||||||
|
- `git.example.com`, in the router rule and in `SWW_WAF_APPS`;
|
||||||
|
- the email address for Let's Encrypt;
|
||||||
|
- the ntfy topic's URL;
|
||||||
|
- the app's image and port;
|
||||||
|
- the netblocks, which are documentation addresses.
|
||||||
|
|
||||||
|
Make the admin token file first, as for any token file:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
mkdir -p /srv/sww/tokens
|
||||||
|
openssl rand -hex 32 > /srv/sww/tokens/admin
|
||||||
|
chown 65532:65532 /srv/sww/tokens/admin
|
||||||
|
chmod 0400 /srv/sww/tokens/admin
|
||||||
|
```
|
||||||
|
|
||||||
### Custom per-address limits
|
### Custom per-address limits
|
||||||
|
|
||||||
For a site with no API:
|
For a site with no API:
|
||||||
@@ -460,7 +674,7 @@ services:
|
|||||||
image: smallwebwaf
|
image: smallwebwaf
|
||||||
environment:
|
environment:
|
||||||
SWW_COMMAND: waf
|
SWW_COMMAND: waf
|
||||||
SWW_WAF_INSTANCE_NAME: fsn1/waf
|
SWW_WAF_INSTANCE_NAME: host1/waf
|
||||||
# Refuse nothing yet; log what enforce mode would have done.
|
# Refuse nothing yet; log what enforce mode would have done.
|
||||||
SWW_WAF_MODE: observe
|
SWW_WAF_MODE: observe
|
||||||
# Look addresses up in a local IPinfo Lite database, never GeoJS.
|
# Look addresses up in a local IPinfo Lite database, never GeoJS.
|
||||||
@@ -529,6 +743,11 @@ request reaches the app.
|
|||||||
16. A path under `/_sww/` is answered `404`, so that the app never sees that
|
16. A path under `/_sww/` is answered `404`, so that the app never sees that
|
||||||
prefix.
|
prefix.
|
||||||
|
|
||||||
|
Steps 12 to 14 only read the request. While the lookup waits for GeoJS, they run
|
||||||
|
at the same time as steps 6 to 11. The first of the two sides to refuse the
|
||||||
|
request ends the checks, and `waf` answers at once. Steps 12 to 14 have 500 ms
|
||||||
|
together.
|
||||||
|
|
||||||
`rproxy` takes the rest, with `waf` counting at the end:
|
`rproxy` takes the rest, with `waf` counting at the end:
|
||||||
|
|
||||||
17. The request is passed to the app, streaming.
|
17. The request is passed to the app, streaming.
|
||||||
@@ -540,17 +759,21 @@ request reaches the app.
|
|||||||
A refused visitor is answered with `SWW_WAF_BAN_RESPONSE`. A `block` rule or a
|
A refused visitor is answered with `SWW_WAF_BAN_RESPONSE`. A `block` rule or a
|
||||||
Core Rule Set match answers `403` and bans nobody, since the Core Rule Set has
|
Core Rule Set match answers `403` and bans nobody, since the Core Rule Set has
|
||||||
false positives; such refusals count toward `SWW_WAF_REFUSALS_LIMIT_PER_MINUTE`
|
false positives; such refusals count toward `SWW_WAF_REFUSALS_LIMIT_PER_MINUTE`
|
||||||
instead. A request over the header limits of step 3 is answered `431`, and bans
|
instead. A request over the header limits of step 3 is answered `414` for a
|
||||||
nobody either. `rproxy` itself answers `405` to a method it does not pass on,
|
request line over 8 KiB and `431` otherwise. It bans nobody either. Nor does a
|
||||||
and the status for a size or time limit, such as `413` for a body too large.
|
request that steps 12 to 14 have not finished within 500 ms, which is answered
|
||||||
|
`503`. `rproxy` itself answers `405` to a method it does not pass on, and the
|
||||||
|
status for a size or time limit, such as `413` for a body too large.
|
||||||
|
|
||||||
### Clients and bans
|
### Clients and bans
|
||||||
|
|
||||||
- A client is one IPv4 address or one IPv6 netblock of
|
- A client is one IPv4 address or one IPv6 netblock of
|
||||||
`SWW_WAF_IPV6_GROUP_PREFIX`, since one abuser usually controls a whole /64.
|
`SWW_WAF_IPV6_GROUP_PREFIX`, since one abuser usually controls a whole /64.
|
||||||
- Each limit is counted over a sliding window with a few integers per client.
|
- Each limit is counted over a sliding window with a few integers per client.
|
||||||
The table of clients and the table of bans each have a bound
|
The table of clients and the bans for limits and attacks each have a bound
|
||||||
(`SWW_WAF_MAX_TRACKED_CLIENTS`, `SWW_WAF_MAX_BANS`), so memory is bounded too.
|
(`SWW_WAF_MAX_TRACKED_CLIENTS`, `SWW_WAF_MAX_BANS`). The bans for AbuseIPDB's
|
||||||
|
blacklist are bounded by the clients it listed over the last
|
||||||
|
`SWW_WAF_ABUSEIPDB_BAN_DAYS` days. So memory is bounded too.
|
||||||
- A ban refuses every request from its netblock until it ends. Requests refused
|
- A ban refuses every request from its netblock until it ends. Requests refused
|
||||||
under a ban count toward no limit. A permanent ban ends only when an admin
|
under a ban count toward no limit. A permanent ban ends only when an admin
|
||||||
lifts it.
|
lifts it.
|
||||||
@@ -673,17 +896,23 @@ stops the running process (see [`docs/state-files.md`](docs/state-files.md)).
|
|||||||
|
|
||||||
- `cmd/sww`: the binary, which only calls `internal/sww`.
|
- `cmd/sww`: the binary, which only calls `internal/sww`.
|
||||||
- `internal/sww`: the process, as the command it is given, and the health check.
|
- `internal/sww`: the process, as the command it is given, and the health check.
|
||||||
|
It is the only package that imports `go.uber.org/fx`, which builds the parts
|
||||||
|
of `waf` and `rproxy`.
|
||||||
- `internal/config`: reads the settings, the one place they are read.
|
- `internal/config`: reads the settings, the one place they are read.
|
||||||
- `internal/rproxy`: the reverse proxy in front of one app.
|
- `internal/rproxy`: the reverse proxy in front of one app.
|
||||||
- `internal/waf`: `waf`'s server, which runs the checks and answers its own
|
- `internal/waf`: `waf`'s server, which runs the checks and answers its own
|
||||||
endpoints.
|
endpoints.
|
||||||
- `internal/ratelimit`, `internal/bans`, `internal/rules`,
|
- `internal/ratelimit`, `internal/bans`, `internal/rules`,
|
||||||
`internal/coreruleset`, `internal/lookup`, `internal/reputation`,
|
`internal/coreruleset`, `internal/geojs`, `internal/lookup`,
|
||||||
`internal/routewatch`, `internal/anomaly` and `internal/alerts`: one check or
|
`internal/reputation`, `internal/routewatch`, `internal/anomaly` and
|
||||||
source each.
|
`internal/alerts`: one check or source each.
|
||||||
- `internal/state`, `internal/requestlog`, `internal/remotelog` and
|
- `internal/dirwatch`: watches a directory, so that `internal/rules` and
|
||||||
`internal/metrics`: the state and cache files, the log lines, the syslog
|
`internal/lookup` read their files again when they change.
|
||||||
sender and the metrics.
|
- `internal/state`: the state and cache files.
|
||||||
|
- `internal/requestlog`: the request log.
|
||||||
|
- `internal/processlog`: the process's own messages.
|
||||||
|
- `internal/remotelog`: the syslog sender.
|
||||||
|
- `internal/metrics`: the metrics.
|
||||||
|
|
||||||
## Rationale
|
## Rationale
|
||||||
|
|
||||||
@@ -747,10 +976,9 @@ on the host.
|
|||||||
- `script/docker`: builds the image, whose build runs the tests and the linter
|
- `script/docker`: builds the image, whose build runs the tests and the linter
|
||||||
first. `make docker`.
|
first. `make docker`.
|
||||||
- `script/publish`: runs `script/docker`, whose build happens on the Docker host
|
- `script/publish`: runs `script/docker`, whose build happens on the Docker host
|
||||||
named by `DOCKER_HOST` (`ssh://root@fsn1app1` unless set). It then pushes the
|
named by `DOCKER_HOST`, then pushes the image to `PUBLISH_IMAGE`, such as
|
||||||
image to `PUBLISH_IMAGE` (`git.eeqj.de/sneak/smallwebwaf` unless set). It
|
`registry.example.com/smallwebwaf`. Both variables are required. It refuses a
|
||||||
refuses a working tree with uncommitted changes. `make publish`. It pushes
|
working tree with uncommitted changes. `make publish`. It pushes four tags:
|
||||||
four tags:
|
|
||||||
- the UTC date as `YYYY-MM-DD`;
|
- the UTC date as `YYYY-MM-DD`;
|
||||||
- the short commit hash;
|
- the short commit hash;
|
||||||
- the branch name;
|
- the branch name;
|
||||||
|
|||||||
@@ -3,7 +3,7 @@
|
|||||||
`smallwebwaf` is a web application firewall for self-hosted apps. It is one
|
`smallwebwaf` is a web application firewall for self-hosted apps. It is one
|
||||||
binary, `sww`, and two of its commands do the work:
|
binary, `sww`, and two of its commands do the work:
|
||||||
|
|
||||||
- `sww rproxy` is a reverse proxy between traefik and one app.
|
- `sww rproxy` is a reverse proxy between a TLS-terminating proxy and one app.
|
||||||
- `sww waf` decides what happens to each request, for many sites.
|
- `sww waf` decides what happens to each request, for many sites.
|
||||||
|
|
||||||
This document says what it must do and why, each requirement once. Everything in
|
This document says what it must do and why, each requirement once. Everything in
|
||||||
@@ -45,7 +45,8 @@ never as a time.
|
|||||||
|
|
||||||
## Non-goals
|
## Non-goals
|
||||||
|
|
||||||
- TLS, certificates and routing by host name. Those are traefik's job.
|
- TLS, certificates and routing by host name. Those are the job of the
|
||||||
|
TLS-terminating proxy, such as traefik, nginx or Caddy.
|
||||||
- More than one app behind one `rproxy`. Each app has its own.
|
- More than one app behind one `rproxy`. Each app has its own.
|
||||||
- Browser challenges such as captchas or proof of work. If wanted, chain Anubis
|
- Browser challenges such as captchas or proof of work. If wanted, chain Anubis
|
||||||
between `rproxy` and the app.
|
between `rproxy` and the app.
|
||||||
@@ -68,7 +69,9 @@ never as a time.
|
|||||||
that command's settings, and runs `internal/rproxy` until `SIGTERM` or
|
that command's settings, and runs `internal/rproxy` until `SIGTERM` or
|
||||||
`SIGINT` (see "`rproxy`"). `sww healthcheck` is the image's health check.
|
`SIGINT` (see "`rproxy`"). `sww healthcheck` is the image's health check.
|
||||||
With no command, or one it does not know, the binary prints the commands
|
With no command, or one it does not know, the binary prints the commands
|
||||||
and exits with status 2.
|
and exits with status 2. `internal/sww` is the only package that imports
|
||||||
|
`go.uber.org/fx`. fx builds the parts of `waf` and `rproxy`, and starts
|
||||||
|
and stops their background work.
|
||||||
- `internal/config`: reads the settings, the one place they are read.
|
- `internal/config`: reads the settings, the one place they are read.
|
||||||
- `internal/waf`: `waf`'s server. It runs the checks in `check`, and holds
|
- `internal/waf`: `waf`'s server. It runs the checks in `check`, and holds
|
||||||
`waf`'s question, answer and report. It answers `waf`'s own endpoints in
|
`waf`'s question, answer and report. It answers `waf`'s own endpoints in
|
||||||
@@ -86,10 +89,16 @@ never as a time.
|
|||||||
change, and tells which rules a request matches.
|
change, and tells which rules a request matches.
|
||||||
- `internal/coreruleset`: the Core Rule Set, with the five changes and the
|
- `internal/coreruleset`: the Core Rule Set, with the five changes and the
|
||||||
apps' fixes listed under "Attack detection", as Coraza directives.
|
apps' fixes listed under "Attack detection", as Coraza directives.
|
||||||
- `internal/lookup`: looks clients up through GeoJS or in the lookup
|
- `internal/geojs`: looks clients up through GeoJS and keeps the answers. It
|
||||||
database.
|
imports no other package of this repository: `waf` gives it what it needs
|
||||||
|
to count and report failures.
|
||||||
|
- `internal/lookup`: looks clients up in the lookup database.
|
||||||
|
- `internal/dirwatch`: watches a directory, and calls a function once it has
|
||||||
|
gone a short time without a change. `internal/rules` and `internal/lookup`
|
||||||
|
read their files again this way.
|
||||||
- `internal/reputation`: fetches the lists, asks the DNS blocklists and
|
- `internal/reputation`: fetches the lists, asks the DNS blocklists and
|
||||||
AbuseIPDB in the background, and keeps their answers.
|
AbuseIPDB in the background, and keeps their answers. It also fetches
|
||||||
|
AbuseIPDB's blacklist, which `internal/waf` bans.
|
||||||
- `internal/routewatch`: asks a routewatch instance about clients in the
|
- `internal/routewatch`: asks a routewatch instance about clients in the
|
||||||
background, and keeps its answers.
|
background, and keeps its answers.
|
||||||
- `internal/anomaly`: the anomaly counters and their alerts.
|
- `internal/anomaly`: the anomaly counters and their alerts.
|
||||||
@@ -98,8 +107,10 @@ never as a time.
|
|||||||
its own.
|
its own.
|
||||||
- `internal/state`: reads the state and cache files at start, takes in edits
|
- `internal/state`: reads the state and cache files at start, takes in edits
|
||||||
while running, and writes the files when due and at the stop.
|
while running, and writes the files when due and at the stop.
|
||||||
- `internal/requestlog` and `internal/remotelog`: the lines on stdout, and
|
- `internal/requestlog`: the request log, as JSON lines on stdout.
|
||||||
sending them to the syslog server.
|
- `internal/processlog`: the process lines on stdout, which hold the
|
||||||
|
process's own messages.
|
||||||
|
- `internal/remotelog`: sends the lines on stdout to the syslog server.
|
||||||
- `internal/metrics`: the metrics, in the Prometheus text format.
|
- `internal/metrics`: the metrics, in the Prometheus text format.
|
||||||
- `Dockerfile`: the lint and test phases, then the image.
|
- `Dockerfile`: the lint and test phases, then the image.
|
||||||
`deploy/example-app` is an app built on the image, with `rproxy` in its
|
`deploy/example-app` is an app built on the image, with `rproxy` in its
|
||||||
@@ -151,26 +162,27 @@ field gives that name. `refusal_limit` and `byte_limits` give theirs in
|
|||||||
around its address, a /64 by default, since one abuser usually controls a
|
around its address, a /64 by default, since one abuser usually controls a
|
||||||
whole /64. A client is therefore one IPv4 address or one IPv6 group.
|
whole /64. A client is therefore one IPv4 address or one IPv6 group.
|
||||||
3. The header limits (`header_limits`; see "Answering in time"). They come first
|
3. The header limits (`header_limits`; see "Answering in time"). They come first
|
||||||
after `SWW_WAF_ALLOW_NETS`, since they only count header fields and cookies.
|
after `SWW_WAF_ALLOW_NETS`, since they only measure lines and count header
|
||||||
Next, the tag lists (`tag_lists`; see "Request tags"). Each list that holds
|
fields and cookies. Next, the tag lists (`tag_lists`; see "Request tags").
|
||||||
the client adds its tag. Next, `routewatch_tags` adds the client's `asn=` and
|
Each list that holds the client adds its tag. Next, `routewatch_tags` adds
|
||||||
`prefix=` tags while `SWW_WAF_ROUTEWATCH_URL` is set. A client in
|
the client's `asn=` and `prefix=` tags while `SWW_WAF_ROUTEWATCH_URL` is set.
|
||||||
`SWW_WAF_ALLOW_NETS` skips the header limits and gets no tag.
|
A client in `SWW_WAF_ALLOW_NETS` skips the header limits and gets no tag.
|
||||||
4. The static lists. A client in `SWW_WAF_ALLOW_NETS` skips steps 5 to 15, the
|
4. The static lists. A client in `SWW_WAF_ALLOW_NETS` skips steps 5 to 15, the
|
||||||
byte limits and the refusal limit. A client in `SWW_WAF_DENY_NETS` is refused
|
byte limits and the refusal limit. A client in `SWW_WAF_DENY_NETS` is refused
|
||||||
(`deny_nets`); a client in both lists is let through.
|
(`deny_nets`); a client in both lists is let through.
|
||||||
5. Bans. A client under a ban is refused (`active_ban`; see "Bans").
|
5. Bans. A client under a ban is refused (`active_ban`; see "Bans").
|
||||||
6. The lookup, unless `SWW_WAF_LOOKUP_SOURCE` is `off` (see "Lookup of AS number
|
6. The lookup, unless `SWW_WAF_LOOKUP_SOURCE` is `off` (see "Lookup of AS number
|
||||||
and country"). The request goes on at once. It waits for the first GeoJS
|
and country"). The request waits for the first GeoJS answer, up to
|
||||||
answer, up to `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, only when one of
|
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, when one of these needs it:
|
||||||
these needs it:
|
- `SWW_WAF_ADD_LOOKUP_HEADERS`, which is `true` by default;
|
||||||
- the country list;
|
- the country list;
|
||||||
- a biased threshold;
|
- a biased threshold;
|
||||||
- `SWW_WAF_TAG_DENY` or `SWW_WAF_TAG_LIMIT_PERCENT` naming an `asn=` or
|
- `SWW_WAF_TAG_DENY` or `SWW_WAF_TAG_LIMIT_PERCENT` naming an `asn=` or
|
||||||
`country=` tag;
|
`country=` tag.
|
||||||
- `SWW_WAF_ADD_LOOKUP_HEADERS`.
|
|
||||||
|
|
||||||
A request that ends before the answer comes is logged without it.
|
Otherwise the request goes on at once. A request that ends before the answer
|
||||||
|
comes is logged without it. While the request waits, steps 12 to 14 run at
|
||||||
|
the same time as steps 6 to 11 (see "Answering in time").
|
||||||
|
|
||||||
7. The country list (`countries`; see "Country list").
|
7. The country list (`countries`; see "Country list").
|
||||||
8. The reputation sources (see "Reputation"), in this order:
|
8. The reputation sources (see "Reputation"), in this order:
|
||||||
@@ -197,10 +209,10 @@ field gives that name. `refusal_limit` and `byte_limits` give theirs in
|
|||||||
16. A path under `/_sww/` is answered `404`, so that the app never sees that
|
16. A path under `/_sww/` is answered `404`, so that the app never sees that
|
||||||
prefix.
|
prefix.
|
||||||
17. `rproxy` passes the request to `SWW_RPROXY_UPSTREAM_URL`, streaming (see
|
17. `rproxy` passes the request to `SWW_RPROXY_UPSTREAM_URL`, streaming (see
|
||||||
"Passing a request to the app"). With `SWW_WAF_ADD_LOOKUP_HEADERS`,
|
"Passing a request to the app"). While `SWW_WAF_ADD_LOOKUP_HEADERS` is
|
||||||
`X-Client-ASN` and `X-Client-Country` are added for the app's own logs; a
|
`true`, as by default, `X-Client-ASN` and `X-Client-Country` are added for
|
||||||
header whose value is unknown is left out. Whatever that setting says, any
|
the app's own logs; a header whose value is unknown is left out. Whatever
|
||||||
such header the client sent is removed.
|
that setting says, any such header the client sent is removed.
|
||||||
18. After the visitor's answer has ended, `rproxy` sends the report (see "The
|
18. After the visitor's answer has ended, `rproxy` sends the report (see "The
|
||||||
report"). `waf` then counts:
|
report"). `waf` then counts:
|
||||||
- For a request it allowed, the body bytes of the response, of the request
|
- For a request it allowed, the body bytes of the response, of the request
|
||||||
@@ -234,13 +246,15 @@ apply.
|
|||||||
load.
|
load.
|
||||||
- Memory is bounded. When the table of clients holds
|
- Memory is bounded. When the table of clients holds
|
||||||
`SWW_WAF_MAX_TRACKED_CLIENTS`, the least recently seen client is dropped
|
`SWW_WAF_MAX_TRACKED_CLIENTS`, the least recently seen client is dropped
|
||||||
first, with its history. The bans are bounded by `SWW_WAF_MAX_BANS` (see
|
first, with its history. The bans for broken limits and clear signs of attack
|
||||||
"Bans"). The GeoJS answers, the reputation verdicts and scores, the DNS
|
are bounded by `SWW_WAF_MAX_BANS` (see "Bans"). Those for AbuseIPDB's
|
||||||
queries under way and the anomaly counters have fixed bounds, listed under
|
blacklist are bounded by the clients it listed over the last
|
||||||
"Fixed limits" in `docs/settings.md`. Past its bound, the GeoJS answer or
|
`SWW_WAF_ABUSEIPDB_BAN_DAYS` days (see "Reputation"). The GeoJS answers, the
|
||||||
anomaly counter used longest ago goes first, and the verdict or score fetched
|
reputation verdicts and scores, the DNS queries under way and the anomaly
|
||||||
longest ago. Past the bound on DNS queries under way, a zone is asked at the
|
counters have fixed bounds, listed under "Fixed limits" in `docs/settings.md`.
|
||||||
client's next request.
|
Past its bound, the GeoJS answer or anomaly counter used longest ago goes
|
||||||
|
first, and the verdict or score fetched longest ago. Past the bound on DNS
|
||||||
|
queries under way, a zone is asked at the client's next request.
|
||||||
|
|
||||||
## Bans
|
## Bans
|
||||||
|
|
||||||
@@ -252,9 +266,9 @@ does not reach neighbours who did nothing. Requests refused under a ban are not
|
|||||||
counted against limits. They are counted in the client's history and in the
|
counted against limits. They are counted in the client's history and in the
|
||||||
ban's notes.
|
ban's notes.
|
||||||
|
|
||||||
A request over the header limits is refused with `431` before the bans are
|
A request over the header limits is refused with `414` or `431` before the bans
|
||||||
checked (see "Answering in time"). It makes no ban, makes no ban permanent and
|
are checked (see "Answering in time"). It makes no ban, makes no ban permanent
|
||||||
is not counted in a ban's notes.
|
and is not counted in a ban's notes.
|
||||||
|
|
||||||
An offence is a request `waf` holds against the client. It is one of:
|
An offence is a request `waf` holds against the client. It is one of:
|
||||||
|
|
||||||
@@ -295,6 +309,9 @@ to a ban, each by its own rule:
|
|||||||
Once it ends, only new requests can break a limit again. The client's
|
Once it ends, only new requests can break a limit again. The client's
|
||||||
history keeps its totals.
|
history keeps its totals.
|
||||||
|
|
||||||
|
A client on AbuseIPDB's blacklist is banned before it sends a request, with the
|
||||||
|
cause `abuseipdb` (see "Reputation").
|
||||||
|
|
||||||
A Core Rule Set match refuses only the request it matched, with `403` and
|
A Core Rule Set match refuses only the request it matched, with `403` and
|
||||||
without a ban, because the Core Rule Set has false positives. A `block` rule
|
without a ban, because the Core Rule Set has false positives. A `block` rule
|
||||||
does the same. Those refusals count toward the refusal limit, so a client that
|
does the same. Those refusals count toward the refusal limit, so a client that
|
||||||
@@ -321,11 +338,12 @@ does not count toward a longer ban; it refuses nothing. Deleting its entry from
|
|||||||
|
|
||||||
A permanent ban does not run out. It ends when an admin lifts it, or when `waf`
|
A permanent ban does not run out. It ends when an admin lifts it, or when `waf`
|
||||||
drops it from a full table, as below. When `waf` holds `SWW_WAF_MAX_BANS` bans
|
drops it from a full table, as below. When `waf` holds `SWW_WAF_MAX_BANS` bans
|
||||||
of its own, it drops the earliest ban of one netblock: the netblock that has
|
for broken limits and clear signs of attack, it drops the earliest such ban of
|
||||||
gone longest without a request. A past, active or permanent ban is dropped
|
one netblock: the netblock that has gone longest without a request. A past,
|
||||||
alike. A scanner whose ban was dropped that way is refused and banned again by
|
active or permanent ban is dropped alike. A scanner whose ban was dropped that
|
||||||
its next probe. A ban whose cause is `admin` is never dropped and does not count
|
way is refused and banned again by its next probe. Bans whose cause is `admin`
|
||||||
toward `SWW_WAF_MAX_BANS`, so such bans can never fill the table.
|
or `abuseipdb` do not count toward `SWW_WAF_MAX_BANS` and are never dropped to
|
||||||
|
make room, so they can never push out the others.
|
||||||
|
|
||||||
A request refused under a ban that runs out is logged with the action
|
A request refused under a ban that runs out is logged with the action
|
||||||
`temporarily_banned`. Under a permanent ban the action is `permanently_banned`.
|
`temporarily_banned`. Under a permanent ban the action is `permanently_banned`.
|
||||||
@@ -506,19 +524,25 @@ service that keeps its visitors' addresses on its own host.
|
|||||||
dropped from that table. The answer reaches the client's history and ban notes
|
dropped from that table. The answer reaches the client's history and ban notes
|
||||||
when it comes.
|
when it comes.
|
||||||
- `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` is how long a visitor's request
|
- `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` is how long a visitor's request
|
||||||
waits for its client's first answer, when a setting needs it (see step 6). It
|
waits for its client's first answer, when a setting needs it (see step 6). A
|
||||||
also bounds each request to GeoJS. A client whose answer has not come in time
|
client whose answer has not come in time counts as unknown until it comes, and
|
||||||
counts as unknown until it comes, and its later requests do not wait. Its
|
its later requests do not wait.
|
||||||
address is asked about again in the background.
|
- A request to GeoJS may take up to 10 seconds, however long visitors wait. It
|
||||||
|
goes on after the visitor's request ends, and its answer is kept. Say GeoJS
|
||||||
|
answers after 5 seconds. The visitor's request is decided after 800
|
||||||
|
milliseconds, without a country. A request from the same client 6 seconds
|
||||||
|
later gets the answer.
|
||||||
|
- Requests from one client that come together share one request to GeoJS.
|
||||||
- A client the lookup cannot place has an unknown country and AS number, which
|
- A client the lookup cannot place has an unknown country and AS number, which
|
||||||
the country list and the biased thresholds treat as described there. Nothing
|
the country list and the biased thresholds treat as described there. Nothing
|
||||||
else treats it differently.
|
else treats it differently.
|
||||||
- GeoJS publishes no rate limit, but its terms forbid "an excessive amount of
|
- GeoJS publishes no rate limit, but its terms forbid "an excessive amount of
|
||||||
API requests", judged by GeoJS alone, and let it block a caller. While GeoJS
|
API requests", judged by GeoJS alone, and let it block a caller. A request to
|
||||||
is slow, down or refusing `waf`, new clients count as unknown. GeoJS is then
|
GeoJS fails when it takes over 10 seconds or GeoJS answers with an error.
|
||||||
left alone for a second, and twice as long after each further failure, up to
|
While GeoJS is slow, down or refusing `waf`, new clients count as unknown.
|
||||||
five minutes. Each failure raises a `source_failure` alert (see "Alerting"). A
|
After a failure GeoJS is left alone for a second, and twice as long after each
|
||||||
service that cannot accept this uses the file.
|
further failure, up to five minutes. Each failure raises a `source_failure`
|
||||||
|
alert (see "Alerting"). A service that cannot accept this uses the file.
|
||||||
- The file is the free IPinfo Lite database (`ipinfo_lite.mmdb`), whose `asn`,
|
- The file is the free IPinfo Lite database (`ipinfo_lite.mmdb`), whose `asn`,
|
||||||
`as_name` and `country_code` fields are read. The operator downloads it with a
|
`as_name` and `country_code` fields are read. The operator downloads it with a
|
||||||
free IPinfo account and mounts it read-only; `waf` never fetches it and holds
|
free IPinfo account and mounts it read-only; `waf` never fetches it and holds
|
||||||
@@ -703,11 +727,12 @@ files, the limits and the bans still apply to the client that sends it.
|
|||||||
rules is loaded yet.
|
rules is loaded yet.
|
||||||
|
|
||||||
- `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` switches body inspection on, up
|
- `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` switches body inspection on, up
|
||||||
to that size. It takes at most `64K`, since `waf` gets no more of a body than
|
to that size. It takes at most `8K`, so that `waf` answers in time (see
|
||||||
that (see "The answer"). Form data and multipart bodies are read up to the
|
"Answering in time"). Form data and multipart bodies are read up to the size.
|
||||||
size. JSON and XML bodies are read only when no larger than the size, since
|
JSON and XML bodies are read only when no larger than the size, since those
|
||||||
those cannot be read in part. Any other body is not read: the Core Rule Set
|
cannot be read in part. A JSON body over 1 KiB is not read either, whatever
|
||||||
would read it as form data, where binary content such as a git push trips
|
the size (see "Answering in time"). Any other body is not read: the Core Rule
|
||||||
|
Set would read it as form data, where binary content such as a git push trips
|
||||||
rules written for text. A body the Core Rule Set cannot parse adds 5 to the
|
rules written for text. A body the Core Rule Set cannot parse adds 5 to the
|
||||||
score, and so does a multipart body cut by the limit in the middle of a part's
|
score, and so does a multipart body cut by the limit in the middle of a part's
|
||||||
header. Body inspection suits apps whose forms carry no code.
|
header. Body inspection suits apps whose forms carry no code.
|
||||||
@@ -1943,10 +1968,10 @@ leave out a cookie only when its name matches exactly, case included.
|
|||||||
### Ban lengths, response and scope
|
### Ban lengths, response and scope
|
||||||
|
|
||||||
`SWW_WAF_BAN_RESPONSE` is `403` by default, which makes a ban easy to recognise
|
`SWW_WAF_BAN_RESPONSE` is `403` by default, which makes a ban easy to recognise
|
||||||
when debugging; `429` and `close` are the alternatives. Behind traefik, `close`
|
when debugging; `429` and `close` are the alternatives. `close` does not leave
|
||||||
does not leave the client unanswered: traefik answers `502`, as it does whenever
|
the client unanswered: the TLS-terminating proxy answers `502`, as it does
|
||||||
its backend drops a connection. `SWW_WAF_BAN_SCOPE_V4_PREFIX` widens an IPv4 ban
|
whenever its backend drops a connection. `SWW_WAF_BAN_SCOPE_V4_PREFIX` widens an
|
||||||
to the surrounding netblock. The four ban lengths are under "Bans".
|
IPv4 ban to the surrounding netblock. The four ban lengths are under "Bans".
|
||||||
|
|
||||||
### Reputation
|
### Reputation
|
||||||
|
|
||||||
@@ -2025,6 +2050,26 @@ host, sharing one address, would often fetch too close together.
|
|||||||
- AbuseIPDB answers other than `200`;
|
- AbuseIPDB answers other than `200`;
|
||||||
- the answer gives no score;
|
- the answer gives no score;
|
||||||
- no answer comes within `SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS`.
|
- no answer comes within `SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS`.
|
||||||
|
- While `SWW_WAF_ABUSEIPDB_KEY` is set, AbuseIPDB's blacklist is fetched at the
|
||||||
|
start and every `SWW_WAF_ABUSEIPDB_BLACKLIST_REFRESH_INTERVAL_MINS` minutes
|
||||||
|
(default `60`). It lists the clients AbuseIPDB scores at
|
||||||
|
`SWW_WAF_ABUSEIPDB_THRESHOLD` (default `90`) or more. The blacklist itself is
|
||||||
|
not kept.
|
||||||
|
- Each client on it is banned for `SWW_WAF_ABUSEIPDB_BAN_DAYS` days (default
|
||||||
|
`30`), with the cause `abuseipdb`. The ban covers the client's netblock,
|
||||||
|
as any ban does.
|
||||||
|
- A netblock that already has an active ban is left as it is. So is one
|
||||||
|
whose `abuseipdb` ban an admin lifted, until that ban would have ended.
|
||||||
|
- An `abuseipdb` ban that has ended is forgotten at the next of these:
|
||||||
|
- a fetch of the blacklist in `enforce` mode;
|
||||||
|
- a read of `bans.json`, at the start or after an admin's edit.
|
||||||
|
- In `observe` mode the blacklist makes no ban.
|
||||||
|
- The threshold takes 25 to 100. It should not be set below 50 to 75, since
|
||||||
|
lower scores list ordinary visitors. AbuseIPDB serves free accounts the
|
||||||
|
blacklist at the threshold 100, whatever is asked for.
|
||||||
|
- AbuseIPDB limits the fetches of the blacklist a day: 5 for a free account.
|
||||||
|
A fetch over the limit fails, and is logged and alerted. The default
|
||||||
|
refresh fetches it 24 times a day.
|
||||||
- `SWW_WAF_DNSBL_ABUSEIPDB_ACTION`, for DNS blocklist and AbuseIPDB hits, is
|
- `SWW_WAF_DNSBL_ABUSEIPDB_ACTION`, for DNS blocklist and AbuseIPDB hits, is
|
||||||
`limit:25` by default. Such verdicts are less certain than a blocklist, so a
|
`limit:25` by default. Such verdicts are less certain than a blocklist, so a
|
||||||
listed client gets a quarter of every request and byte limit rather than a
|
listed client gets a quarter of every request and byte limit rather than a
|
||||||
@@ -2039,9 +2084,10 @@ host, sharing one address, would often fetch too close together.
|
|||||||
- Alerts go to a webhook as JSON, and to Slack and ntfy as formatted messages.
|
- Alerts go to a webhook as JSON, and to Slack and ntfy as formatted messages.
|
||||||
`docs/alerts.md` lists the events, the JSON object and the messages.
|
`docs/alerts.md` lists the events, the JSON object and the messages.
|
||||||
`SWW_WAF_ALERT_EVENTS` chooses which of the seven event types are sent; the
|
`SWW_WAF_ALERT_EVENTS` chooses which of the seven event types are sent; the
|
||||||
hourly `summary` is sent whatever it says. The bans an admin makes raise no
|
hourly `summary` is sent whatever it says. The bans an admin makes, and the
|
||||||
alert. In `observe` mode, a request that would have made a ban still raises
|
bans for AbuseIPDB's blacklist, raise no alert. In `observe` mode, a request
|
||||||
the ban's alert, with `mode` set to `observe` in its `detail`.
|
that would have made a ban still raises the ban's alert, with `mode` set to
|
||||||
|
`observe` in its `detail`.
|
||||||
- A repeat is an alert with the same event, and the same netblock, source, file
|
- A repeat is an alert with the same event, and the same netblock, source, file
|
||||||
or anomaly scope, as one sent less than `SWW_WAF_ALERT_COOLDOWN` before. An
|
or anomaly scope, as one sent less than `SWW_WAF_ALERT_COOLDOWN` before. An
|
||||||
anomaly scope is one of the things counted under "Anomaly thresholds". A
|
anomaly scope is one of the things counted under "Anomaly thresholds". A
|
||||||
@@ -2189,7 +2235,7 @@ lists the files, what each holds, and their sizes.
|
|||||||
these stop the process with a message naming the file and position:
|
these stop the process with a message naming the file and position:
|
||||||
- a file that does not parse, or has an unknown `version`;
|
- a file that does not parse, or has an unknown `version`;
|
||||||
- an entry without a field it needs;
|
- an entry without a field it needs;
|
||||||
- a ban whose cause is not one of the four;
|
- a ban whose cause is not `limit`, `attack`, `admin` or `abuseipdb`;
|
||||||
- a ban whose `expires` is not after its `start`.
|
- a ban whose `expires` is not after its `start`.
|
||||||
|
|
||||||
Starting with empty state would silently forgive every repeat offender.
|
Starting with empty state would silently forgive every repeat offender.
|
||||||
@@ -2369,19 +2415,22 @@ runs no check, writes no request log line and counts nothing in the metrics.
|
|||||||
|
|
||||||
### The answer
|
### The answer
|
||||||
|
|
||||||
`waf` runs steps 3 to 16 of "What happens to a request", in that order. Every
|
`waf` runs steps 3 to 16 of "What happens to a request", in that order, apart
|
||||||
answer but the last below carries `X-Sww-Disposition`.
|
from the steps that run at the same time (see "Answering in time"). Every answer
|
||||||
|
but the last below carries `X-Sww-Disposition`.
|
||||||
|
|
||||||
- `X-Sww-Disposition: allow`, status `200`, no body. `rproxy` passes the request
|
- `X-Sww-Disposition: allow`, status `200`, no body. `rproxy` passes the request
|
||||||
to the app. With `SWW_WAF_ADD_LOOKUP_HEADERS` the answer also carries
|
to the app. While `SWW_WAF_ADD_LOOKUP_HEADERS` is `true`, the answer also
|
||||||
`X-Client-ASN` and `X-Client-Country`, which `rproxy` adds to the request it
|
carries `X-Client-ASN` and `X-Client-Country`, which `rproxy` adds to the
|
||||||
passes on. A header whose value is unknown is left out.
|
request it passes on. A header whose value is unknown is left out.
|
||||||
- `X-Sww-Disposition: deny`, with the status, headers and body the visitor is to
|
- `X-Sww-Disposition: deny`, with the status, headers and body the visitor is to
|
||||||
get. `rproxy` gives them to the visitor, less this header. The status is one
|
get. `rproxy` gives them to the visitor, less this header. The status is one
|
||||||
of:
|
of:
|
||||||
- `SWW_WAF_BAN_RESPONSE`, for a refusal;
|
- `SWW_WAF_BAN_RESPONSE`, for a refusal;
|
||||||
- `403`, for a `block` rule or the Core Rule Set;
|
- `403`, for a `block` rule or the Core Rule Set;
|
||||||
- `431`, for a request over the header limits;
|
- `414` or `431`, for a request over the header limits;
|
||||||
|
- `503`, for a request that steps 12 to 14 have not finished within 500
|
||||||
|
milliseconds;
|
||||||
- `404`, for a path under `/_sww/` that passes the checks, so that the app
|
- `404`, for a path under `/_sww/` that passes the checks, so that the app
|
||||||
never sees that prefix.
|
never sees that prefix.
|
||||||
- `X-Sww-Disposition: close`, status `200`, no body. This takes the place of a
|
- `X-Sww-Disposition: close`, status `200`, no body. This takes the place of a
|
||||||
@@ -2396,23 +2445,28 @@ In `observe` mode a request that `enforce` mode would refuse is answered
|
|||||||
answered `404`.
|
answered `404`.
|
||||||
|
|
||||||
The Core Rule Set inspects the body a question carries only while
|
The Core Rule Set inspects the body a question carries only while
|
||||||
`SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` is set. It inspects at most 64 KiB
|
`SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` is set, and at most 8 KiB of it.
|
||||||
less one byte. So it does not inspect a JSON or XML body of 64 KiB or more,
|
|
||||||
which may have been cut short.
|
|
||||||
|
|
||||||
### Answering in time
|
### Answering in time
|
||||||
|
|
||||||
`rproxy` stops waiting for `waf` after `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, by
|
`rproxy` stops waiting for `waf` after `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, by
|
||||||
default 2 seconds, and `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` then
|
default 2 seconds, and `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` then
|
||||||
decides. The longest a question waits on an outside source is
|
decides. Two things in a decision take time:
|
||||||
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, by default 800 milliseconds, well
|
|
||||||
within that time.
|
- the wait for GeoJS, at most `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, by
|
||||||
|
default 800 milliseconds;
|
||||||
|
- the trap paths, the rule files and the Core Rule Set, which have 500
|
||||||
|
milliseconds.
|
||||||
|
|
||||||
|
They run at the same time, so `waf` answers within the longer of the two, well
|
||||||
|
within `rproxy`'s wait.
|
||||||
|
|
||||||
GeoJS is the only outside source a question can wait on. A question waits only
|
GeoJS is the only outside source a question can wait on. A question waits only
|
||||||
when step 6 says so, and for at most
|
when step 6 says so, and for at most
|
||||||
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, by default 800 milliseconds. A
|
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, by default 800 milliseconds. A
|
||||||
client without an answer by then counts as unknown. No question waits on these,
|
client without an answer by then counts as unknown. The setting bounds only the
|
||||||
which run in the background:
|
question's wait: the request to GeoJS goes on, and its answer is kept for later
|
||||||
|
questions. No question waits on these, which run in the background:
|
||||||
|
|
||||||
- the DNS blocklists and AbuseIPDB;
|
- the DNS blocklists and AbuseIPDB;
|
||||||
- fetching the blocklists, the tag lists and the file
|
- fetching the blocklists, the tag lists and the file
|
||||||
@@ -2437,38 +2491,92 @@ never wait. These can still wait for GeoJS:
|
|||||||
- a country list, which needs the answer;
|
- a country list, which needs the answer;
|
||||||
- a rate limit a percentage has lowered, which needs the answer;
|
- a rate limit a percentage has lowered, which needs the answer;
|
||||||
- a DNS blocklist or AbuseIPDB, while `SWW_WAF_DNSBL_ABUSEIPDB_ACTION` is
|
- a DNS blocklist or AbuseIPDB, while `SWW_WAF_DNSBL_ABUSEIPDB_ACTION` is
|
||||||
`deny`;
|
`deny`.
|
||||||
- a trap path;
|
|
||||||
- a rule file;
|
|
||||||
- the Core Rule Set.
|
|
||||||
|
|
||||||
The header limits are fixed, with no setting. `waf` refuses a request over
|
The trap paths, the rule files and the Core Rule Set, steps 12 to 14, run during
|
||||||
either one:
|
that wait. Once a request has passed the bans, the lookup starts. While it waits
|
||||||
|
for GeoJS, steps 12 to 14 run at the same time as steps 6 to 11. A lookup that
|
||||||
|
needs no wait has its answer at once, and steps 7 to 11 then run before steps 12
|
||||||
|
to 14 start.
|
||||||
|
|
||||||
|
- The first of the two sides to refuse the request ends the checks. `waf`
|
||||||
|
answers at once, and does not wait for the other side. The lookup goes on, and
|
||||||
|
its answer is kept.
|
||||||
|
- What the other side finds leaves no trace. It counts nothing, bans nothing,
|
||||||
|
raises no alert, counts no metric and adds nothing to the log line.
|
||||||
|
- When both sides would refuse, the one that finishes first is answered and
|
||||||
|
logged.
|
||||||
|
- When neither refuses, step 15 runs.
|
||||||
|
|
||||||
|
Steps 12 to 14 only read the request. They have 500 milliseconds together, from
|
||||||
|
their start. The limit is fixed, with no setting. A request they have not
|
||||||
|
finished by then is refused:
|
||||||
|
|
||||||
|
- the answer is `deny`, with `503`;
|
||||||
|
- the log line's action is `checks_timed_out`, and its `heuristic` names the
|
||||||
|
step that had not finished;
|
||||||
|
- the refusal makes no ban, is no offence and does not count toward the refusal
|
||||||
|
limit;
|
||||||
|
- it holds in every `SWW_WAF_CORE_RULE_SET_MODE`;
|
||||||
|
- in `observe` mode the request is answered `allow` with `would_action`, like
|
||||||
|
every refusal.
|
||||||
|
|
||||||
|
The limit is measured on the clock. While `waf`'s host is so busy that an
|
||||||
|
ordinary request takes 500 milliseconds, that request is refused too.
|
||||||
|
|
||||||
|
The header limits are fixed, with no setting. A line's length is in bytes and
|
||||||
|
does not count its line ending. `waf` refuses a request whose request line is
|
||||||
|
longer than 8 KiB with `414`. An example is `GET /search?q=` followed by 30,000
|
||||||
|
characters.
|
||||||
|
|
||||||
|
It refuses a request with `431` for any of:
|
||||||
|
|
||||||
|
- a header line longer than 8 KiB. Each value of a header sent twice is a line
|
||||||
|
of its own. `Host` counts as a line.
|
||||||
- more than 50 cookies. Each part of its `Cookie` headers between semicolons
|
- more than 50 cookies. Each part of its `Cookie` headers between semicolons
|
||||||
counts as one, unless it is empty or only spaces and tabs.
|
counts as one, unless it is empty or only spaces and tabs.
|
||||||
- more than 100 header fields. A header sent twice counts twice. `Host` and
|
- more than 100 header fields. A header sent twice counts twice. `Host` and
|
||||||
`Transfer-Encoding` do not count. `X-Request-ID` and `X-Forwarded-Proto` do.
|
`Transfer-Encoding` do not count. `X-Request-ID` and `X-Forwarded-Proto` do.
|
||||||
|
|
||||||
Real clients never reach these limits. The answer is `deny` with `431`, and the
|
Real clients rarely reach these limits. nginx refuses lines over 8 KiB by
|
||||||
log line's action is `request_header_too_large`. The refusal makes no ban and is
|
default, and Apache's default limit is about the same. The answer is `deny`. The
|
||||||
no offence.
|
log line's action is `request_line_too_long` for `414` and
|
||||||
|
`request_header_too_large` for `431`. The refusal makes no ban and is no
|
||||||
|
offence.
|
||||||
|
|
||||||
The header limits are the first check after `SWW_WAF_ALLOW_NETS`. A request over
|
The header limits are the first check after `SWW_WAF_ALLOW_NETS`. A request over
|
||||||
them is refused before any later check runs. So it causes no lookup, no query to
|
them is refused before any later check runs. So it causes no lookup, no query to
|
||||||
an outside source and no rule match, and the rate limits do not count it. A
|
an outside source and no rule match, and the rate limits do not count it. A
|
||||||
client under a ban or in `SWW_WAF_DENY_NETS` also gets `431` for such a request.
|
client under a ban or in `SWW_WAF_DENY_NETS` also gets `414` or `431` for such a
|
||||||
|
request.
|
||||||
|
|
||||||
The limits apply on every path and in every `SWW_WAF_CORE_RULE_SET_MODE`. A
|
The limits apply on every path and in every `SWW_WAF_CORE_RULE_SET_MODE`. A
|
||||||
client in `SWW_WAF_ALLOW_NETS` skips them, as it skips the Core Rule Set. In
|
client in `SWW_WAF_ALLOW_NETS` skips them, as it skips the Core Rule Set. In
|
||||||
`observe` mode such a request is answered `allow` with `would_action`, and no
|
`observe` mode such a request is answered `allow` with `would_action`, and no
|
||||||
other check runs on it.
|
other check runs on it.
|
||||||
|
|
||||||
The Core Rule Set runs each of its rules on each cookie, so thousands of small
|
The Core Rule Set is the one check a visitor can keep busy for long. It runs
|
||||||
cookies would keep it busy for over a second. Within the limits, one long cookie
|
each of its rules on each part of a request, so thousands of small cookies would
|
||||||
or query string can still keep it busy for a fraction of a second. So enough
|
keep it busy for over a second. These fixed limits bound what it reads:
|
||||||
such requests at once can overload `waf` past
|
|
||||||
`SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`.
|
- the header limits above;
|
||||||
|
- `rproxy`'s limit of 32 KiB on the request line and headers together (see
|
||||||
|
"`rproxy`'s limits");
|
||||||
|
- 8 KiB of a body, the most `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` takes,
|
||||||
|
or 1 KiB of a JSON body.
|
||||||
|
|
||||||
|
Coraza names each value in a JSON body by the whole path to it, such as
|
||||||
|
`json.users.0.name`. So the names and values the Core Rule Set reads in a JSON
|
||||||
|
body can add up to more than the body. From 1 KiB of JSON, Coraza makes at most
|
||||||
|
8 KiB of them, no more than the largest body of another kind.
|
||||||
|
|
||||||
|
A request can reach all of these at once. At paranoia level 4, with no rule
|
||||||
|
switched off, the Core Rule Set takes under a second on one core of a current
|
||||||
|
server to inspect it. `waf` refuses it if the inspection takes over 500
|
||||||
|
milliseconds. Coraza cannot stop an inspection part-way, so the inspection runs
|
||||||
|
to its end with nobody waiting for it. These limits keep one request from
|
||||||
|
holding a processor for more than about a second. Enough such requests at once
|
||||||
|
can still overload `waf` past `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`.
|
||||||
|
|
||||||
### The report
|
### The report
|
||||||
|
|
||||||
@@ -2572,11 +2680,11 @@ request and ban notes name the site by its host.
|
|||||||
## `rproxy`
|
## `rproxy`
|
||||||
|
|
||||||
`sww rproxy` is the reverse proxy in front of one app. It asks `waf` about each
|
`sww rproxy` is the reverse proxy in front of one app. It asks `waf` about each
|
||||||
request, and passes on what `waf` allows. It keeps nothing between requests and
|
request, and passes on what `waf` allows. It writes nothing to disk. The only
|
||||||
writes nothing to disk. When `waf` cannot be connected to, does not answer at
|
thing it keeps between requests is its circuit breaker for asks to `waf`.
|
||||||
all, or does not answer within `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`,
|
`SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` decides what is done with a
|
||||||
`SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` decides. Its default, `allow`,
|
request when `waf` gives no answer in time, and while the circuit breaker is
|
||||||
keeps the site answering.
|
open. Its default, `allow`, keeps the site answering.
|
||||||
|
|
||||||
`rproxy` reads only its `SWW_RPROXY_` settings, which `docs/settings.md` lists.
|
`rproxy` reads only its `SWW_RPROXY_` settings, which `docs/settings.md` lists.
|
||||||
One of them is required: `SWW_RPROXY_WAF_BACKEND`, `waf`'s URL. It has no
|
One of them is required: `SWW_RPROXY_WAF_BACKEND`, `waf`'s URL. It has no
|
||||||
@@ -2594,7 +2702,8 @@ only. The image runs it by default (see "Deployment").
|
|||||||
4. It refuses a request with a method it does not pass on, or one it cannot put
|
4. It refuses a request with a method it does not pass on, or one it cannot put
|
||||||
to `waf` as the visitor sent it (see "Requests `rproxy` refuses itself").
|
to `waf` as the visitor sent it (see "Requests `rproxy` refuses itself").
|
||||||
5. It reads up to the first 64 KiB of the body into memory.
|
5. It reads up to the first 64 KiB of the body into memory.
|
||||||
6. It sends `waf` the question (see "The question"). It speaks HTTP/1.1 to
|
6. It sends `waf` the question (see "The question"), unless the circuit breaker
|
||||||
|
keeps it from asking (see "The circuit breaker"). It speaks HTTP/1.1 to
|
||||||
`waf`, whatever `waf` offers. Go's HTTP/2 client would refuse to send a
|
`waf`, whatever `waf` offers. Go's HTTP/2 client would refuse to send a
|
||||||
visitor's `Upgrade` header, and most `Connection` headers. The question never
|
visitor's `Upgrade` header, and most `Connection` headers. The question never
|
||||||
carries the visitor's `Trailer`.
|
carries the visitor's `Trailer`.
|
||||||
@@ -2668,23 +2777,24 @@ log line has status `0`.
|
|||||||
### Without an answer from `waf`
|
### Without an answer from `waf`
|
||||||
|
|
||||||
`SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` decides what is done with the
|
`SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` decides what is done with the
|
||||||
request only when `waf`:
|
request when `waf`:
|
||||||
|
|
||||||
- cannot be connected to;
|
- cannot be connected to;
|
||||||
- does not answer at all;
|
- does not answer at all;
|
||||||
- does not answer within `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, which covers
|
- does not answer within `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, which covers
|
||||||
connecting, sending the question and reading the whole answer.
|
connecting, sending the question and reading the whole answer.
|
||||||
|
|
||||||
An answer whose body breaks off counts as no answer at all.
|
An answer whose body breaks off counts as no answer at all. The setting also
|
||||||
|
decides, without asking `waf`, while the circuit breaker is open.
|
||||||
|
|
||||||
`allow`, the default, passes the request to the app. `deny` answers `503`.
|
`allow`, the default, passes the request to the app. `deny` answers `503`.
|
||||||
Either way `rproxy` writes a process warning, its log line has
|
Either way `rproxy` writes a process warning, its log line has
|
||||||
`waf_backend_no_answer`, and it sends no report.
|
`waf_backend_no_answer`, and it sends no report.
|
||||||
|
|
||||||
A visitor can make `waf` slow. A request that keeps `waf` busy past
|
`waf` taking longer than `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS` to decide a request
|
||||||
`SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, like any request while `waf` is overloaded,
|
is a bug in `waf`. Such a request gets
|
||||||
gets `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION`. With `deny`, such a request
|
`SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION`, like any request while `waf` is
|
||||||
gets `503` instead of reaching the app.
|
overloaded. With `deny`, it gets `503` instead of reaching the app.
|
||||||
|
|
||||||
A path under `/_sww/` that `allow` would pass on is answered `404` instead, so
|
A path under `/_sww/` that `allow` would pass on is answered `404` instead, so
|
||||||
that the app never sees that prefix.
|
that the app never sees that prefix.
|
||||||
@@ -2699,6 +2809,28 @@ A visitor that goes away while `waf` is being asked is no failure of `waf`.
|
|||||||
`SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` does not apply. Its log line has
|
`SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` does not apply. Its log line has
|
||||||
status `0` and no `disposition`.
|
status `0` and no `disposition`.
|
||||||
|
|
||||||
|
### The circuit breaker
|
||||||
|
|
||||||
|
`rproxy` stops asking `waf` for a while when `waf` keeps failing to answer. An
|
||||||
|
ask fails when `waf` gives no answer in time (see "Without an answer from
|
||||||
|
`waf`"), or an answer `rproxy` cannot pass on. Any other answer in time
|
||||||
|
succeeds, whatever it decides. A visitor that goes away while `waf` is being
|
||||||
|
asked counts as neither.
|
||||||
|
|
||||||
|
The breaker works like this:
|
||||||
|
|
||||||
|
- It opens when 5 asks fail within 5 seconds. Fewer failures never open it.
|
||||||
|
- While it is open, a request gets
|
||||||
|
`SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` at once, without asking `waf`.
|
||||||
|
It gets the same process warning and `waf_backend_no_answer` as a request
|
||||||
|
whose ask failed.
|
||||||
|
- While it is open, one request a second still asks `waf`.
|
||||||
|
- Each ask that succeeds clears the failures counted so far, and closes the
|
||||||
|
breaker if it is open.
|
||||||
|
|
||||||
|
The breaker lives in memory, and a restart of `rproxy` closes it. It has no
|
||||||
|
settings.
|
||||||
|
|
||||||
### Passing a request to the app
|
### Passing a request to the app
|
||||||
|
|
||||||
`rproxy` sends the app at `SWW_RPROXY_UPSTREAM_URL` the visitor's request: the
|
`rproxy` sends the app at `SWW_RPROXY_UPSTREAM_URL` the visitor's request: the
|
||||||
@@ -2761,9 +2893,9 @@ its whole body. The visitor gets that answer at once, and the connection closes
|
|||||||
after it.
|
after it.
|
||||||
|
|
||||||
The largest request line and headers is 32 KiB. A kept-open connection is closed
|
The largest request line and headers is 32 KiB. A kept-open connection is closed
|
||||||
after 120 seconds idle. Neither has a setting. traefik, by default, closes an
|
after 120 seconds idle. Neither has a setting. The TLS-terminating proxy should
|
||||||
unused connection after 90 seconds. So traefik closes first, and never sends a
|
close an unused connection sooner, so that it never sends a request on a
|
||||||
request on a connection `rproxy` is closing.
|
connection `rproxy` is closing. traefik closes one after 90 seconds by default.
|
||||||
|
|
||||||
What the visitor gets when a limit is passed:
|
What the visitor gets when a limit is passed:
|
||||||
|
|
||||||
@@ -2814,14 +2946,16 @@ chooses: `waf` runs `waf`, and with it unset the image runs `rproxy`.
|
|||||||
The usual setup is three containers on one private network:
|
The usual setup is three containers on one private network:
|
||||||
|
|
||||||
- the app;
|
- the app;
|
||||||
- `rproxy`, which traefik routes to, with `SWW_RPROXY_UPSTREAM_URL` naming the
|
- `rproxy`, which the TLS-terminating proxy forwards to, with
|
||||||
app and `SWW_RPROXY_WAF_BACKEND` naming `waf`;
|
`SWW_RPROXY_UPSTREAM_URL` naming the app and `SWW_RPROXY_WAF_BACKEND` naming
|
||||||
|
`waf`;
|
||||||
- `waf`, with `SWW_COMMAND=waf` and a volume for its state files.
|
- `waf`, with `SWW_COMMAND=waf` and a volume for its state files.
|
||||||
|
|
||||||
Each further app adds its own container and its own `rproxy`, whose
|
Each further app adds its own container and its own `rproxy`, whose
|
||||||
`SWW_RPROXY_WAF_BACKEND` names the same `waf` (see "Sites"). Anything on that
|
`SWW_RPROXY_WAF_BACKEND` names the same `waf` (see "Sites"). Anything on that
|
||||||
network can reach an app around `rproxy`, and can ask `waf` questions or send it
|
network can reach an app around `rproxy`, and can ask `waf` questions or send it
|
||||||
reports. So the network holds only what you trust.
|
reports. So the network holds only what you trust. `README.md` lists what the
|
||||||
|
TLS-terminating proxy must do.
|
||||||
|
|
||||||
### `rproxy` in the app's container
|
### `rproxy` in the app's container
|
||||||
|
|
||||||
@@ -2893,10 +3027,10 @@ process that stops at start is started again every second, not in a tight loop.
|
|||||||
container's output, which `docker logs` shows.
|
container's output, which `docker logs` shows.
|
||||||
- When a service exits, runit runs its `run` script again, and the container
|
- When a service exits, runit runs its `run` script again, and the container
|
||||||
does not stop. Beside an app, the other process keeps running.
|
does not stop. Beside an app, the other process keeps running.
|
||||||
- While `rproxy` is down, traefik answers `502`. While the app is down, `rproxy`
|
- While `rproxy` is down, the TLS-terminating proxy answers `502`. While the app
|
||||||
answers `502`, which is not a client offence. While `waf` is down,
|
is down, `rproxy` answers `502`, which is not a client offence. While `waf` is
|
||||||
`SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` decides (see "Without an answer
|
down, `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` decides (see "Without an
|
||||||
from `waf`").
|
answer from `waf`").
|
||||||
- Only faults found at start stop `waf`:
|
- Only faults found at start stop `waf`:
|
||||||
- invalid configuration;
|
- invalid configuration;
|
||||||
- a lookup database that is missing or unreadable;
|
- a lookup database that is missing or unreadable;
|
||||||
@@ -2929,23 +3063,26 @@ checks:
|
|||||||
- With `SWW_COMMAND` unset, empty or `rproxy`, it passes while `rproxy` answers
|
- With `SWW_COMMAND` unset, empty or `rproxy`, it passes while `rproxy` answers
|
||||||
that request on `127.0.0.1`, at the port in `SWW_RPROXY_LISTEN_ADDR`, and the
|
that request on `127.0.0.1`, at the port in `SWW_RPROXY_LISTEN_ADDR`, and the
|
||||||
app accepts connections at `SWW_RPROXY_UPSTREAM_URL`. It does not ask `waf`,
|
app accepts connections at `SWW_RPROXY_UPSTREAM_URL`. It does not ask `waf`,
|
||||||
so that the site stays in traefik's routes while `waf` is down.
|
so that the site stays in the TLS-terminating proxy's routes while `waf` is
|
||||||
|
down.
|
||||||
- With any other value, it fails with a message naming `SWW_COMMAND` and its
|
- With any other value, it fails with a message naming `SWW_COMMAND` and its
|
||||||
value.
|
value.
|
||||||
|
|
||||||
traefik sends a container no requests until it shows as healthy. So the check
|
A TLS-terminating proxy that reads Docker's health status, as traefik does,
|
||||||
runs every second from the container's start until it first passes, for up to a
|
sends a container no requests until it shows as healthy. So the check runs every
|
||||||
minute, and every 30 seconds after that. An app with a health check of its own
|
second from the container's start until it first passes, for up to a minute, and
|
||||||
can replace the image's `HEALTHCHECK` with one that checks both.
|
every 30 seconds after that. An app with a health check of its own can replace
|
||||||
|
the image's `HEALTHCHECK` with one that checks both.
|
||||||
|
|
||||||
### Ports
|
### Ports
|
||||||
|
|
||||||
`rproxy` listens on no port but the one in `SWW_RPROXY_LISTEN_ADDR`, 8080 by
|
`rproxy` listens on no port but the one in `SWW_RPROXY_LISTEN_ADDR`, 8080 by
|
||||||
default, which traefik routes to. It passes requests to the app at
|
default, which the TLS-terminating proxy forwards to. It passes requests to the
|
||||||
`SWW_RPROXY_UPSTREAM_URL`, `http://127.0.0.1:8081` by default. Another port may
|
app at `SWW_RPROXY_UPSTREAM_URL`, `http://127.0.0.1:8081` by default. Another
|
||||||
be set: the health check follows the setting, and traefik's port label
|
port may be set: the health check follows the setting, and the TLS-terminating
|
||||||
(`traefik.http.services.<name>.loadbalancer.server.port`) must name it. An app
|
proxy must name it, as traefik's port label
|
||||||
beside `rproxy` must leave that port free.
|
(`traefik.http.services.<name>.loadbalancer.server.port`) does. An app beside
|
||||||
|
`rproxy` must leave that port free.
|
||||||
|
|
||||||
`waf` listens on no port but the one in `SWW_WAF_LISTEN_ADDR`, 8080 by default.
|
`waf` listens on no port but the one in `SWW_WAF_LISTEN_ADDR`, 8080 by default.
|
||||||
The questions, the reports and its own endpoints all come there.
|
The questions, the reports and its own endpoints all come there.
|
||||||
@@ -2967,8 +3104,8 @@ host directory, keeps them across deploys. A host directory mounted there needs
|
|||||||
no change of owner, and files an earlier owner left in it can be read and
|
no change of owner, and files an earlier owner left in it can be read and
|
||||||
replaced. The volume holds a few tens of MiB at most with the defaults, and
|
replaced. The volume holds a few tens of MiB at most with the defaults, and
|
||||||
needs no backup beyond the host's own. The image declares no volume, since every
|
needs no backup beyond the host's own. The image declares no volume, since every
|
||||||
app image built on it would inherit it. `rproxy` keeps nothing, and needs no
|
app image built on it would inherit it. `rproxy` writes nothing to disk, and
|
||||||
volume.
|
needs no volume.
|
||||||
|
|
||||||
`waf` keeps its cache files in `SWW_WAF_CACHE_DIR`, `/var/cache/sww` by default,
|
`waf` keeps its cache files in `SWW_WAF_CACHE_DIR`, `/var/cache/sww` by default,
|
||||||
which the `run` script makes. The cache directory needs no volume and can be
|
which the `run` script makes. The cache directory needs no volume and can be
|
||||||
@@ -2986,9 +3123,9 @@ in `docs/settings.md` shows the commands.
|
|||||||
### Forwarded headers
|
### Forwarded headers
|
||||||
|
|
||||||
The app's TCP peer is `rproxy`, and the `X-Forwarded-For` the app receives ends
|
The app's TCP peer is `rproxy`, and the `X-Forwarded-For` the app receives ends
|
||||||
with traefik's address, which `rproxy` adds after the visitor's. So the app must
|
with the TLS-terminating proxy's address, which `rproxy` adds after the
|
||||||
trust `rproxy`'s address for forwarded headers, besides the private address
|
visitor's. So the app must trust `rproxy`'s address for forwarded headers,
|
||||||
ranges traefik is on.
|
besides the private address ranges the TLS-terminating proxy is on.
|
||||||
|
|
||||||
- In a container of its own, `rproxy`'s address is on the private network. The
|
- In a container of its own, `rproxy`'s address is on the private network. The
|
||||||
usual default for a trusted-proxy setting, the private ranges, covers it.
|
usual default for a trusted-proxy setting, the private ranges, covers it.
|
||||||
|
|||||||
@@ -9,6 +9,10 @@
|
|||||||
ARG SWW_IMAGE=smallwebwaf
|
ARG SWW_IMAGE=smallwebwaf
|
||||||
FROM ${SWW_IMAGE}
|
FROM ${SWW_IMAGE}
|
||||||
|
|
||||||
|
# Its own title and description, in place of the smallwebwaf image's.
|
||||||
|
LABEL org.opencontainers.image.title="smallwebwaf-example-app" \
|
||||||
|
org.opencontainers.image.description="An example app built on the smallwebwaf image."
|
||||||
|
|
||||||
# Packages the app needs, from the nixpkgs in the image.
|
# Packages the app needs, from the nixpkgs in the image.
|
||||||
RUN nix-env -iA nixpkgs.busybox
|
RUN nix-env -iA nixpkgs.busybox
|
||||||
|
|
||||||
|
|||||||
+5
-4
@@ -6,7 +6,7 @@ back or dropped.
|
|||||||
|
|
||||||
## Events
|
## Events
|
||||||
|
|
||||||
- `ban`: a ban `waf` made.
|
- `ban`: a ban `waf` made for a broken limit or a clear sign of attack.
|
||||||
- `permanent_ban`: a permanent ban it made, or an existing ban made permanent by
|
- `permanent_ban`: a permanent ban it made, or an existing ban made permanent by
|
||||||
a request.
|
a request.
|
||||||
- `core_rule_set_match`: the Core Rule Set scored a request at or over the
|
- `core_rule_set_match`: the Core Rule Set scored a request at or over the
|
||||||
@@ -34,7 +34,7 @@ on one line:
|
|||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"instance": "fsn1/waf",
|
"instance": "host1/waf",
|
||||||
"time": "2026-10-06T12:00:50.123461Z",
|
"time": "2026-10-06T12:00:50.123461Z",
|
||||||
"event": "ban",
|
"event": "ban",
|
||||||
"host": "app.example",
|
"host": "app.example",
|
||||||
@@ -68,7 +68,8 @@ on one line:
|
|||||||
"earlier_bans": {
|
"earlier_bans": {
|
||||||
"limit": 0,
|
"limit": 0,
|
||||||
"attack": 0,
|
"attack": 0,
|
||||||
"admin": 0
|
"admin": 0,
|
||||||
|
"abuseipdb": 0
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
@@ -109,7 +110,7 @@ on one line:
|
|||||||
## Slack and ntfy
|
## Slack and ntfy
|
||||||
|
|
||||||
Slack and ntfy get a message with a title and a text. The title is the instance
|
Slack and ntfy get a message with a title and a text. The title is the instance
|
||||||
and the event, such as `fsn1/waf: ban`. The text is the `reason`, then one line
|
and the event, such as `host1/waf: ban`. The text is the `reason`, then one line
|
||||||
for each of these that the alert has:
|
for each of these that the alert has:
|
||||||
|
|
||||||
- the host, the client, the netblock and the country;
|
- the host, the client, the netblock and the country;
|
||||||
|
|||||||
+18
-5
@@ -26,7 +26,7 @@ These questions write no line:
|
|||||||
A `report` line:
|
A `report` line:
|
||||||
|
|
||||||
```
|
```
|
||||||
{"type":"request","waf_line":"report","time":"2026-10-03T12:00:00.123Z","instance":"fsn1/waf","client_ip":"203.0.113.9","method":"GET","scheme":"https","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","peer_ip":"172.18.0.2","forwarded_for":"203.0.113.9","client_group":"203.0.113.9/32","asn":"AS64496","as_name":"Example Net","country":"DE","request_headers":{"accept":"*/*"},"action":"forward","counts":{"minute":1,"hour":12,"day":40,"minute_bytes":5120,"hour_bytes":61440,"day_bytes":204800},"core_rule_set_score":0,"duration_total_ms":3.217,"duration_checks_ms":0.041,"duration_core_rule_set_ms":0.029,"duration_upstream_total_ms":3.104}
|
{"type":"request","waf_line":"report","time":"2026-10-03T12:00:00.123Z","instance":"host1/waf","client_ip":"203.0.113.9","method":"GET","scheme":"https","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","peer_ip":"172.18.0.2","forwarded_for":"203.0.113.9","client_group":"203.0.113.9/32","asn":"AS64496","as_name":"Example Net","country":"DE","request_headers":{"accept":"*/*"},"action":"forward","counts":{"minute":1,"hour":12,"day":40,"minute_bytes":5120,"hour_bytes":61440,"day_bytes":204800},"core_rule_set_score":0,"duration_total_ms":3.217,"duration_checks_ms":0.041,"duration_core_rule_set_ms":0.029,"duration_upstream_total_ms":3.104}
|
||||||
```
|
```
|
||||||
|
|
||||||
### Every line
|
### Every line
|
||||||
@@ -68,7 +68,8 @@ Every one of these lines has:
|
|||||||
### Request detail
|
### Request detail
|
||||||
|
|
||||||
- `request_id`: the id `rproxy` made for the request when it arrived.
|
- `request_id`: the id `rproxy` made for the request when it arrived.
|
||||||
- `peer_ip`: the address of `rproxy`'s TCP peer, normally traefik.
|
- `peer_ip`: the address of `rproxy`'s TCP peer, normally the TLS-terminating
|
||||||
|
proxy.
|
||||||
- `forwarded_for`: the `X-Forwarded-For` header the visitor's request came with.
|
- `forwarded_for`: the `X-Forwarded-For` header the visitor's request came with.
|
||||||
- `client_group`: the client as rate limiting counts it, an IPv4 address as a
|
- `client_group`: the client as rate limiting counts it, an IPv4 address as a
|
||||||
/32 or an IPv6 group.
|
/32 or an IPv6 group.
|
||||||
@@ -102,7 +103,10 @@ Every one of these lines has:
|
|||||||
ban this request made or made permanent;
|
ban this request made or made permanent;
|
||||||
- `country_denied`, `client_is_rate_limited`, `blocked_by_block_rule` and
|
- `country_denied`, `client_is_rate_limited`, `blocked_by_block_rule` and
|
||||||
`blocked_by_core_rule_set`: refused by that check;
|
`blocked_by_core_rule_set`: refused by that check;
|
||||||
- `request_header_too_large`: over a header limit;
|
- `request_line_too_long`: a request line over the header limits;
|
||||||
|
- `request_header_too_large`: headers over the header limits;
|
||||||
|
- `checks_timed_out`: refused with `503` because the trap paths, the rule
|
||||||
|
files and the Core Rule Set had not finished within 500 ms;
|
||||||
- `own_endpoint`: answered at one of `waf`'s own endpoints, or `404` for a
|
- `own_endpoint`: answered at one of `waf`'s own endpoints, or `404` for a
|
||||||
path under `/_sww/`.
|
path under `/_sww/`.
|
||||||
- `would_action`: in `observe` mode, what `enforce` mode would have done with a
|
- `would_action`: in `observe` mode, what `enforce` mode would have done with a
|
||||||
@@ -124,11 +128,19 @@ Every one of these lines has:
|
|||||||
- `rate_limits`: `window`, `count` and `limit`;
|
- `rate_limits`: `window`, `count` and `limit`;
|
||||||
- `trap_paths`: `path`, the trap path asked for;
|
- `trap_paths`: `path`, the trap path asked for;
|
||||||
- `rule_files`: `rule_id` and `rule_action`, of the rule that refused it;
|
- `rule_files`: `rule_id` and `rule_action`, of the rule that refused it;
|
||||||
- `header_limits`: `fields`, `fields_limit`, `cookies` and `cookies_limit`;
|
- `header_limits`: for a request line, `request_line_bytes` and
|
||||||
|
`request_line_bytes_limit`; for headers, `longest_header_line_bytes`,
|
||||||
|
`header_line_bytes_limit`, `fields`, `fields_limit`, `cookies` and
|
||||||
|
`cookies_limit`;
|
||||||
- `core_rule_set`: `score`, and `threshold`,
|
- `core_rule_set`: `score`, and `threshold`,
|
||||||
`SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD`;
|
`SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD`;
|
||||||
- `refusal_limit`: `window`, `count` and `limit`, of the refusal limit;
|
- `refusal_limit`: `window`, `count` and `limit`, of the refusal limit;
|
||||||
- `byte_limits`: `window`, `count` and `limit`, the count in bytes.
|
- `byte_limits`: `window`, `count` and `limit`, the count in bytes.
|
||||||
|
|
||||||
|
For a request refused as `checks_timed_out`, `heuristic` is the one of
|
||||||
|
`trap_paths`, `rule_files` and `core_rule_set` that had not finished, and
|
||||||
|
its figure is `time_limit_ms`, 500.
|
||||||
|
|
||||||
- `after_answer_heuristic`: the heuristic that banned the client after `waf`
|
- `after_answer_heuristic`: the heuristic that banned the client after `waf`
|
||||||
answered the request, in `observe` mode too: `refusal_limit` or `byte_limits`.
|
answered the request, in `observe` mode too: `refusal_limit` or `byte_limits`.
|
||||||
- `after_answer_heuristic_figures`: an object of the figures that heuristic
|
- `after_answer_heuristic_figures`: an object of the figures that heuristic
|
||||||
@@ -233,7 +245,8 @@ meanings where they differ:
|
|||||||
- `disposition`: what was done with the request: `allow`, `deny` or `close`. It
|
- `disposition`: what was done with the request: `allow`, `deny` or `close`. It
|
||||||
is `waf`'s answer, or the value of
|
is `waf`'s answer, or the value of
|
||||||
`SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION`. It is left out when:
|
`SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION`. It is left out when:
|
||||||
- `waf` was not asked, as for the health check;
|
- `waf` was not asked and `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` did
|
||||||
|
not apply, as for the health check;
|
||||||
- the visitor went away while `waf` was being asked;
|
- the visitor went away while `waf` was being asked;
|
||||||
- `waf`'s answer had no valid `X-Sww-Disposition`;
|
- `waf`'s answer had no valid `X-Sww-Disposition`;
|
||||||
- `rproxy` could not pass on `waf`'s answer, and answered `502`.
|
- `rproxy` could not pass on `waf`'s answer, and answered `502`.
|
||||||
|
|||||||
+38
-22
@@ -34,7 +34,7 @@ chooses between the two (see "The image's command" below).
|
|||||||
such as `:9000`. Leave the host out (see "Ports" in [`SPEC.md`](../SPEC.md)).
|
such as `:9000`. Leave the host out (see "Ports" in [`SPEC.md`](../SPEC.md)).
|
||||||
- `SWW_WAF_INSTANCE_NAME` (default: the container's host name): the `instance`
|
- `SWW_WAF_INSTANCE_NAME` (default: the container's host name): the `instance`
|
||||||
name in each line `waf` logs, and in each alert and metric, such as
|
name in each line `waf` logs, and in each alert and metric, such as
|
||||||
`fsn1/waf`. Valid UTF-8. No control character while `SWW_WAF_ALERT_NTFY_URL`
|
`host1/waf`. Valid UTF-8. No control character while `SWW_WAF_ALERT_NTFY_URL`
|
||||||
is set. 1 to 48 printable ASCII characters without a space while
|
is set. 1 to 48 printable ASCII characters without a space while
|
||||||
`SWW_WAF_LOG_REMOTE_URL` is set and `SWW_WAF_LOG_REMOTE_APP_NAME` is not.
|
`SWW_WAF_LOG_REMOTE_URL` is set and `SWW_WAF_LOG_REMOTE_APP_NAME` is not.
|
||||||
- `SWW_WAF_MODE` (default `enforce`): `enforce` or `observe`.
|
- `SWW_WAF_MODE` (default `enforce`): `enforce` or `observe`.
|
||||||
@@ -42,8 +42,8 @@ chooses between the two (see "The image's command" below).
|
|||||||
client's group, from 32 to 128.
|
client's group, from 32 to 128.
|
||||||
- `SWW_WAF_MAX_TRACKED_CLIENTS` (default `20000`): the most clients kept, a
|
- `SWW_WAF_MAX_TRACKED_CLIENTS` (default `20000`): the most clients kept, a
|
||||||
whole number above zero.
|
whole number above zero.
|
||||||
- `SWW_WAF_MAX_BANS` (default `5000`): how many of its own bans `waf` keeps, a
|
- `SWW_WAF_MAX_BANS` (default `5000`): the most bans `waf` keeps for broken
|
||||||
whole number above zero.
|
limits and attacks, a whole number above zero.
|
||||||
|
|
||||||
## Tokens
|
## Tokens
|
||||||
|
|
||||||
@@ -150,7 +150,7 @@ cut off; the client is banned.
|
|||||||
- `SWW_WAF_LOOKUP_SOURCE` (default `geojs`): `geojs`, `file` or `off`. `off`
|
- `SWW_WAF_LOOKUP_SOURCE` (default `geojs`): `geojs`, `file` or `off`. `off`
|
||||||
stops the start while a setting that needs a lookup is set:
|
stops the start while a setting that needs a lookup is set:
|
||||||
- `SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES`;
|
- `SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES`;
|
||||||
- `SWW_WAF_ADD_LOOKUP_HEADERS`;
|
- `SWW_WAF_ADD_LOOKUP_HEADERS` set to `true`;
|
||||||
- `SWW_WAF_TAG_DENY` or `SWW_WAF_TAG_LIMIT_PERCENT` naming a `country=` tag,
|
- `SWW_WAF_TAG_DENY` or `SWW_WAF_TAG_LIMIT_PERCENT` naming a `country=` tag,
|
||||||
or an `asn=` tag while `SWW_WAF_ROUTEWATCH_URL` is unset;
|
or an `asn=` tag while `SWW_WAF_ROUTEWATCH_URL` is unset;
|
||||||
- the two byte percentage lists and `SWW_WAF_ASN_LIMIT_PERCENT_URL`;
|
- the two byte percentage lists and `SWW_WAF_ASN_LIMIT_PERCENT_URL`;
|
||||||
@@ -159,11 +159,14 @@ cut off; the client is banned.
|
|||||||
- `SWW_WAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in
|
- `SWW_WAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in
|
||||||
its `.mmdb` form, for `SWW_WAF_LOOKUP_SOURCE=file`.
|
its `.mmdb` form, for `SWW_WAF_LOOKUP_SOURCE=file`.
|
||||||
- `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` (default `800`): how long a
|
- `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` (default `800`): how long a
|
||||||
request waits for its client's first answer from GeoJS, and how long a request
|
request waits for its client's first answer from GeoJS. A time in
|
||||||
to GeoJS may take. A time in milliseconds; never `off`.
|
milliseconds; never `off`. It does not cut off the request to GeoJS, whose
|
||||||
- `SWW_WAF_ADD_LOOKUP_HEADERS` (default `false`): `true` has `rproxy` pass the
|
answer is kept when it comes later.
|
||||||
|
- `SWW_WAF_ADD_LOOKUP_HEADERS` (default `true`): `true` has `rproxy` pass the
|
||||||
app the client's AS number in `X-Client-ASN` and its country in
|
app the client's AS number in `X-Client-ASN` and its country in
|
||||||
`X-Client-Country`.
|
`X-Client-Country`. The default is `false` while `SWW_WAF_LOOKUP_SOURCE` is
|
||||||
|
`off`. With GeoJS, a new client's first request waits for its answer, up to
|
||||||
|
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`.
|
||||||
|
|
||||||
## Country list
|
## Country list
|
||||||
|
|
||||||
@@ -249,10 +252,9 @@ tag that no request can get stops the start. A request can get only these:
|
|||||||
- `SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS` (default empty): path prefixes the Core
|
- `SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS` (default empty): path prefixes the Core
|
||||||
Rule Set does not inspect.
|
Rule Set does not inspect.
|
||||||
- `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` (default `off`): `off`, or how
|
- `SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE` (default `off`): `off`, or how
|
||||||
much of a request body the Core Rule Set reads, a size of at most `64K`. A
|
much of a request body the Core Rule Set reads, a size of at most `8K`. A
|
||||||
question carries no more of a body than 64 KiB, and the Core Rule Set reads at
|
larger size would let one request keep it busy past `rproxy`'s timeout (see
|
||||||
most 64 KiB less one byte of it, whatever the value (see "The answer" in
|
"Answering in time" in [`SPEC.md`](../SPEC.md)).
|
||||||
[`SPEC.md`](../SPEC.md)).
|
|
||||||
- `SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE` (default unset): a file of Core Rule
|
- `SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE` (default unset): a file of Core Rule
|
||||||
Set exclusions, written as Coraza directives. It is loaded before the Core
|
Set exclusions, written as Coraza directives. It is loaded before the Core
|
||||||
Rule Set's rules, so it holds only `SecAction` and `SecRule` lines that
|
Rule Set's rules, so it holds only `SecAction` and `SecRule` lines that
|
||||||
@@ -336,6 +338,13 @@ tag that no request can get stops the start. A request can get only these:
|
|||||||
or AbuseIPDB's score is used. A duration.
|
or AbuseIPDB's score is used. A duration.
|
||||||
- `SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS` (default `2000`): how long a query to a
|
- `SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS` (default `2000`): how long a query to a
|
||||||
zone or a check with AbuseIPDB may take. A time in milliseconds.
|
zone or a check with AbuseIPDB may take. A time in milliseconds.
|
||||||
|
- `SWW_WAF_ABUSEIPDB_THRESHOLD` (default `90`): the least abuse confidence score
|
||||||
|
of the clients on AbuseIPDB's blacklist, a whole number from 25 to 100. It
|
||||||
|
should not be set below 50 to 75, since lower scores list ordinary visitors.
|
||||||
|
- `SWW_WAF_ABUSEIPDB_BLACKLIST_REFRESH_INTERVAL_MINS` (default `60`): how often
|
||||||
|
the blacklist is fetched again, in minutes. A whole number above zero.
|
||||||
|
- `SWW_WAF_ABUSEIPDB_BAN_DAYS` (default `30`): how long the ban of a client on
|
||||||
|
the blacklist lasts, in days. A whole number above zero.
|
||||||
|
|
||||||
## Alerting
|
## Alerting
|
||||||
|
|
||||||
@@ -362,6 +371,11 @@ tag that no request can get stops the start. A request can get only these:
|
|||||||
|
|
||||||
## Anomaly thresholds
|
## Anomaly thresholds
|
||||||
|
|
||||||
|
An anomaly is more requests or bytes than one of these thresholds allows.
|
||||||
|
Crossing a threshold sends one `anomaly` alert to the destinations under
|
||||||
|
"Alerting", and does nothing else: no client is banned, rate limited or blocked.
|
||||||
|
`SWW_WAF_ALERT_COOLDOWN` holds back repeats of that alert.
|
||||||
|
|
||||||
Each scope has four thresholds: the most requests and the most body bytes
|
Each scope has four thresholds: the most requests and the most body bytes
|
||||||
counted in a minute and in an hour. The bytes are those `SWW_WAF_BYTES_COUNTED`
|
counted in a minute and in an hour. The bytes are those `SWW_WAF_BYTES_COUNTED`
|
||||||
names. Requests are counts, bytes are sizes. Each is `off` by default. The four
|
names. Requests are counts, bytes are sizes. Each is `off` by default. The four
|
||||||
@@ -421,8 +435,9 @@ A few limits are fixed rather than settings:
|
|||||||
- at most 20,000 anomaly counters are kept;
|
- at most 20,000 anomaly counters are kept;
|
||||||
- each alert destination's queue holds at most 1,000 alerts;
|
- each alert destination's queue holds at most 1,000 alerts;
|
||||||
- `waf` holds at most 10,000 answered requests awaiting their reports;
|
- `waf` holds at most 10,000 answered requests awaiting their reports;
|
||||||
- `waf` refuses a request with more than 50 cookies or more than 100 header
|
- `waf` refuses a request with a request line or header line over 8 KiB, more
|
||||||
fields (see "Answering in time" under "`waf`" in [`SPEC.md`](../SPEC.md));
|
than 50 cookies or more than 100 header fields (see "Answering in time" under
|
||||||
|
"`waf`" in [`SPEC.md`](../SPEC.md));
|
||||||
- `rproxy` takes a request line and headers of at most 32 KiB, and closes a
|
- `rproxy` takes a request line and headers of at most 32 KiB, and closes a
|
||||||
kept-open connection idle for 120 seconds;
|
kept-open connection idle for 120 seconds;
|
||||||
- `rproxy` reads at most 1 MiB of the body of `waf`'s answer, and answers `502`
|
- `rproxy` reads at most 1 MiB of the body of `waf`'s answer, and answers `502`
|
||||||
@@ -432,7 +447,9 @@ A few limits are fixed rather than settings:
|
|||||||
|
|
||||||
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` is the longest a question waits on
|
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` is the longest a question waits on
|
||||||
an outside source. Keep it well under `rproxy`'s
|
an outside source. Keep it well under `rproxy`'s
|
||||||
`SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, as the default is. No question waits for
|
`SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, as the default is. The trap paths, the rule
|
||||||
|
files and the Core Rule Set run during that wait and have 500 ms, so `waf`
|
||||||
|
answers within the longer of the two times. No question waits for
|
||||||
`SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS`, since the DNS blocklists and AbuseIPDB are
|
`SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS`, since the DNS blocklists and AbuseIPDB are
|
||||||
asked in the background (see "Answering in time" under "`waf`" in
|
asked in the background (see "Answering in time" under "`waf`" in
|
||||||
[`SPEC.md`](../SPEC.md)).
|
[`SPEC.md`](../SPEC.md)).
|
||||||
@@ -450,10 +467,9 @@ start, and an invalid value stops the start. Any can be given as a file, as a
|
|||||||
for `waf`'s answer, to a question and to a report. A time in milliseconds;
|
for `waf`'s answer, to a question and to a report. A time in milliseconds;
|
||||||
never `off`.
|
never `off`.
|
||||||
- `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` (default `allow`): what is done
|
- `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` (default `allow`): what is done
|
||||||
with a request when `waf` cannot be connected to, does not answer at all, or
|
with a request when `waf` gives no answer in time, and while the circuit
|
||||||
does not answer within `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`. `allow` passes it
|
breaker for asks to `waf` is open. `allow` passes it to the app, and `deny`
|
||||||
to the app, and `deny` answers `503`. It does not apply to an answer `waf`
|
answers `503`. It does not apply to an answer `waf` gives, such as an error.
|
||||||
gives, such as an error.
|
|
||||||
- `SWW_RPROXY_MAX_REQUEST_SIZE` (default `100M`): the largest request body. A
|
- `SWW_RPROXY_MAX_REQUEST_SIZE` (default `100M`): the largest request body. A
|
||||||
size or `off`.
|
size or `off`.
|
||||||
- `SWW_RPROXY_MAX_RESPONSE_SIZE` (default `1G`): the largest response body. A
|
- `SWW_RPROXY_MAX_RESPONSE_SIZE` (default `1G`): the largest response body. A
|
||||||
@@ -475,9 +491,9 @@ start, and an invalid value stops the start. Any can be given as a file, as a
|
|||||||
`10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`): the netblocks whose
|
`10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`): the netblocks whose
|
||||||
`X-Forwarded-For` is believed. Set but empty, it trusts nothing. A client
|
`X-Forwarded-For` is believed. Set but empty, it trusts nothing. A client
|
||||||
outside these netblocks cannot claim another address. Where other containers
|
outside these netblocks cannot claim another address. Where other containers
|
||||||
reach `rproxy` directly, or visitors on a private network reach traefik, they
|
reach `rproxy` directly, or visitors on a private network reach the
|
||||||
can name another address in the header. Narrowing the setting to traefik's own
|
TLS-terminating proxy, they can name another address in the header. Narrowing
|
||||||
network closes that gap.
|
the setting to the TLS-terminating proxy's own network closes that gap.
|
||||||
|
|
||||||
## The image's command
|
## The image's command
|
||||||
|
|
||||||
|
|||||||
+19
-5
@@ -21,11 +21,22 @@ top-level `version` of 1.
|
|||||||
|
|
||||||
## `bans.json`
|
## `bans.json`
|
||||||
|
|
||||||
Active and past bans: every ban an admin made, and up to `SWW_WAF_MAX_BANS` of
|
Active and past bans:
|
||||||
those `waf` made. Per entry:
|
|
||||||
|
- every ban an admin made;
|
||||||
|
- up to `SWW_WAF_MAX_BANS` of those `waf` made for a broken limit or an attack;
|
||||||
|
- the bans for AbuseIPDB's blacklist.
|
||||||
|
|
||||||
|
A ban for AbuseIPDB's blacklist that has ended is forgotten at the next of
|
||||||
|
these:
|
||||||
|
|
||||||
|
- a fetch of the blacklist in `enforce` mode;
|
||||||
|
- a read of `bans.json`, at the start or after an admin's edit.
|
||||||
|
|
||||||
|
Per entry:
|
||||||
|
|
||||||
- `netblock`, `start` and `expires` (`null` for a permanent ban);
|
- `netblock`, `start` and `expires` (`null` for a permanent ban);
|
||||||
- `cause`: `attack`, `limit` or `admin`;
|
- `cause`: `attack`, `limit`, `admin` or `abuseipdb`;
|
||||||
- `reason`: a short reason;
|
- `reason`: a short reason;
|
||||||
- `lifted`: when an admin lifted it, if one did;
|
- `lifted`: when an admin lifted it, if one did;
|
||||||
- `notes`: the notes below.
|
- `notes`: the notes below.
|
||||||
@@ -60,7 +71,8 @@ ban. The notes:
|
|||||||
- for a clear sign of attack, `rule_id` and `target`, or `trap_path`;
|
- for a clear sign of attack, `rule_id` and `target`, or `trap_path`;
|
||||||
- `reputation`: the sources the client was on, each with its `source` and, for
|
- `reputation`: the sources the client was on, each with its `source` and, for
|
||||||
AbuseIPDB, its `score`;
|
AbuseIPDB, its `score`;
|
||||||
- `request`: the request that caused the ban, each text cut to 256 bytes:
|
- `request`: the request that caused the ban, each text cut to 256 bytes, and
|
||||||
|
empty for a ban for AbuseIPDB's blacklist:
|
||||||
- `time`, `method` and `host`;
|
- `time`, `method` and `host`;
|
||||||
- `path`, with its query string;
|
- `path`, with its query string;
|
||||||
- `status` and `user_agent`;
|
- `status` and `user_agent`;
|
||||||
@@ -151,7 +163,9 @@ A cache file. routewatch's answers from the last hour, one per line:
|
|||||||
- `bans.json`: about 1.2 KiB per ban, at most about 2.5 KiB, so about 6 MiB at
|
- `bans.json`: about 1.2 KiB per ban, at most about 2.5 KiB, so about 6 MiB at
|
||||||
the default `SWW_WAF_MAX_BANS`. Written every 15 minutes, a full file comes to
|
the default `SWW_WAF_MAX_BANS`. Written every 15 minutes, a full file comes to
|
||||||
about 600 MiB of writes a day, and a hundred new bans a day add about 600 MiB
|
about 600 MiB of writes a day, and a hundred new bans a day add about 600 MiB
|
||||||
more.
|
more. A ban for AbuseIPDB's blacklist takes about 1 KiB. A free account's
|
||||||
|
blacklist lists 10,000 clients, so it adds at least 10 MiB, and 1 GiB of
|
||||||
|
writes a day.
|
||||||
- `lookups.json`: about 150 bytes per answer, under 1.5 GiB of writes a day when
|
- `lookups.json`: about 150 bytes per answer, under 1.5 GiB of writes a day when
|
||||||
full.
|
full.
|
||||||
- `routewatch.json`: about 120 bytes per answer, at most about 12 MiB.
|
- `routewatch.json`: about 120 bytes per answer, at most about 12 MiB.
|
||||||
|
|||||||
@@ -12,6 +12,9 @@ require (
|
|||||||
github.com/oschwald/maxminddb-golang/v2 v2.7.0
|
github.com/oschwald/maxminddb-golang/v2 v2.7.0
|
||||||
github.com/prometheus/client_golang v1.24.1
|
github.com/prometheus/client_golang v1.24.1
|
||||||
github.com/spf13/viper v1.21.0
|
github.com/spf13/viper v1.21.0
|
||||||
|
go.uber.org/fx v1.24.0
|
||||||
|
golang.org/x/net v0.60.0
|
||||||
|
golang.org/x/sys v0.48.0
|
||||||
)
|
)
|
||||||
|
|
||||||
require (
|
require (
|
||||||
@@ -43,11 +46,12 @@ require (
|
|||||||
github.com/tidwall/match v1.1.1 // indirect
|
github.com/tidwall/match v1.1.1 // indirect
|
||||||
github.com/tidwall/pretty v1.2.1 // indirect
|
github.com/tidwall/pretty v1.2.1 // indirect
|
||||||
github.com/valllabh/ocsf-schema-golang v1.0.3 // indirect
|
github.com/valllabh/ocsf-schema-golang v1.0.3 // indirect
|
||||||
|
go.uber.org/dig v1.19.0 // indirect
|
||||||
|
go.uber.org/multierr v1.10.0 // indirect
|
||||||
|
go.uber.org/zap v1.26.0 // indirect
|
||||||
go.yaml.in/yaml/v3 v3.0.5 // indirect
|
go.yaml.in/yaml/v3 v3.0.5 // indirect
|
||||||
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba // indirect
|
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba // indirect
|
||||||
golang.org/x/net v0.60.0 // indirect
|
|
||||||
golang.org/x/sync v0.23.0 // indirect
|
golang.org/x/sync v0.23.0 // indirect
|
||||||
golang.org/x/sys v0.48.0 // indirect
|
|
||||||
golang.org/x/text v0.42.0 // indirect
|
golang.org/x/text v0.42.0 // indirect
|
||||||
google.golang.org/protobuf v1.36.11 // indirect
|
google.golang.org/protobuf v1.36.11 // indirect
|
||||||
rsc.io/binaryregexp v0.2.0 // indirect
|
rsc.io/binaryregexp v0.2.0 // indirect
|
||||||
|
|||||||
@@ -97,8 +97,16 @@ github.com/tidwall/pretty v1.2.1 h1:qjsOFOWWQl+N3RsoF5/ssm1pHmJJwhjlSbZ51I6wMl4=
|
|||||||
github.com/tidwall/pretty v1.2.1/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
|
github.com/tidwall/pretty v1.2.1/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
|
||||||
github.com/valllabh/ocsf-schema-golang v1.0.3 h1:eR8k/3jP/OOqB8LRCtdJ4U+vlgd/gk5y3KMXoodrsrw=
|
github.com/valllabh/ocsf-schema-golang v1.0.3 h1:eR8k/3jP/OOqB8LRCtdJ4U+vlgd/gk5y3KMXoodrsrw=
|
||||||
github.com/valllabh/ocsf-schema-golang v1.0.3/go.mod h1:sZ3as9xqm1SSK5feFWIR2CuGeGRhsM7TR1MbpBctzPk=
|
github.com/valllabh/ocsf-schema-golang v1.0.3/go.mod h1:sZ3as9xqm1SSK5feFWIR2CuGeGRhsM7TR1MbpBctzPk=
|
||||||
|
go.uber.org/dig v1.19.0 h1:BACLhebsYdpQ7IROQ1AGPjrXcP5dF80U3gKoFzbaq/4=
|
||||||
|
go.uber.org/dig v1.19.0/go.mod h1:Us0rSJiThwCv2GteUN0Q7OKvU7n5J4dxZ9JKUXozFdE=
|
||||||
|
go.uber.org/fx v1.24.0 h1:wE8mruvpg2kiiL1Vqd0CC+tr0/24XIB10Iwp2lLWzkg=
|
||||||
|
go.uber.org/fx v1.24.0/go.mod h1:AmDeGyS+ZARGKM4tlH4FY2Jr63VjbEDJHtqXTGP5hbo=
|
||||||
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
|
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
|
||||||
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
|
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
|
||||||
|
go.uber.org/multierr v1.10.0 h1:S0h4aNzvfcFsC3dRF1jLoaov7oRaKqRGC/pUEJ2yvPQ=
|
||||||
|
go.uber.org/multierr v1.10.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
|
||||||
|
go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
|
||||||
|
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
|
||||||
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
|
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
|
||||||
go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
|
go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
|
||||||
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
|
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
|
||||||
|
|||||||
@@ -39,7 +39,7 @@ const (
|
|||||||
ntfyURL = "https://ntfy.example/sww-alerts"
|
ntfyURL = "https://ntfy.example/sww-alerts"
|
||||||
ntfyToken = "tk_0123456789abcdefghijklmnopq"
|
ntfyToken = "tk_0123456789abcdefghijklmnopq"
|
||||||
// instance is the instance name every alert gives.
|
// instance is the instance name every alert gives.
|
||||||
instance = "fsn1app1/blog"
|
instance = "host1/blog"
|
||||||
// started is when each test starts, as an alert gives it, and
|
// started is when each test starts, as an alert gives it, and
|
||||||
// anHourOn an hour later.
|
// anHourOn an hour later.
|
||||||
started = "2000-01-01T00:00:00Z"
|
started = "2000-01-01T00:00:00Z"
|
||||||
@@ -1160,8 +1160,8 @@ func TestEachDestinationIsSentOnlyTheAlertsWaitingForIt(t *testing.T) {
|
|||||||
synctest.Wait()
|
synctest.Wait()
|
||||||
|
|
||||||
wantBodies(t, standIns[alerts.DestinationSlack],
|
wantBodies(t, standIns[alerts.DestinationSlack],
|
||||||
`{"text":"*fsn1app1/blog: file_error*\nsecond"}`,
|
`{"text":"*host1/blog: file_error*\nsecond"}`,
|
||||||
`{"text":"*fsn1app1/blog: file_error*\nthird"}`)
|
`{"text":"*host1/blog: file_error*\nthird"}`)
|
||||||
wantBodies(t, standIns[alerts.DestinationNtfy], "fourth")
|
wantBodies(t, standIns[alerts.DestinationNtfy], "fourth")
|
||||||
|
|
||||||
// alerts.json then lists Slack and ntfy alone, with no alert waiting.
|
// alerts.json then lists Slack and ntfy alone, with no alert waiting.
|
||||||
@@ -1477,7 +1477,7 @@ func anAlertForEachEvent() []eventMessage {
|
|||||||
Reason: "requests per hour over the limit of 10000",
|
Reason: "requests per hour over the limit of 10000",
|
||||||
Detail: map[string]any{"mode": "observe"},
|
Detail: map[string]any{"mode": "observe"},
|
||||||
},
|
},
|
||||||
"fsn1app1/blog: ban", "default no_entry",
|
"host1/blog: ban", "default no_entry",
|
||||||
"requests per hour over the limit of 10000\nclient: 203.0.113.9\n" +
|
"requests per hour over the limit of 10000\nclient: 203.0.113.9\n" +
|
||||||
"netblock: 203.0.113.0/24\ncountry: DE\nmode: observe",
|
"netblock: 203.0.113.0/24\ncountry: DE\nmode: observe",
|
||||||
},
|
},
|
||||||
@@ -1489,7 +1489,7 @@ func anAlertForEachEvent() []eventMessage {
|
|||||||
), Country: "FR",
|
), Country: "FR",
|
||||||
Reason: "matched the rule env-file",
|
Reason: "matched the rule env-file",
|
||||||
},
|
},
|
||||||
"fsn1app1/blog: permanent_ban", "high no_entry",
|
"host1/blog: permanent_ban", "high no_entry",
|
||||||
"matched the rule env-file\nclient: 198.51.100.7\n" +
|
"matched the rule env-file\nclient: 198.51.100.7\n" +
|
||||||
"netblock: 198.51.100.7/32\ncountry: FR",
|
"netblock: 198.51.100.7/32\ncountry: FR",
|
||||||
},
|
},
|
||||||
@@ -1500,7 +1500,7 @@ func anAlertForEachEvent() []eventMessage {
|
|||||||
Reason: "refused by the Core Rule Set",
|
Reason: "refused by the Core Rule Set",
|
||||||
Host: "app.example",
|
Host: "app.example",
|
||||||
},
|
},
|
||||||
"fsn1app1/blog: core_rule_set_match", "default shield",
|
"host1/blog: core_rule_set_match", "default shield",
|
||||||
"refused by the Core Rule Set\nhost: app.example\nclient: 192.0.2.1\n" +
|
"refused by the Core Rule Set\nhost: app.example\nclient: 192.0.2.1\n" +
|
||||||
"netblock: 192.0.2.1/32",
|
"netblock: 192.0.2.1/32",
|
||||||
},
|
},
|
||||||
@@ -1509,7 +1509,7 @@ func anAlertForEachEvent() []eventMessage {
|
|||||||
Event: alerts.EventAnomaly, Netblock: netip.MustParsePrefix("192.0.2.0/24"),
|
Event: alerts.EventAnomaly, Netblock: netip.MustParsePrefix("192.0.2.0/24"),
|
||||||
Reason: "requests per minute over the threshold of 5000",
|
Reason: "requests per minute over the threshold of 5000",
|
||||||
},
|
},
|
||||||
"fsn1app1/blog: anomaly", "high chart_with_upwards_trend",
|
"host1/blog: anomaly", "high chart_with_upwards_trend",
|
||||||
"requests per minute over the threshold of 5000\nnetblock: 192.0.2.0/24",
|
"requests per minute over the threshold of 5000\nnetblock: 192.0.2.0/24",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
@@ -1518,7 +1518,7 @@ func anAlertForEachEvent() []eventMessage {
|
|||||||
Netblock: netip.MustParsePrefix("192.0.2.2/32"),
|
Netblock: netip.MustParsePrefix("192.0.2.2/32"),
|
||||||
Reason: "listed by a DNS blocklist",
|
Reason: "listed by a DNS blocklist",
|
||||||
},
|
},
|
||||||
"fsn1app1/blog: reputation_hit", "low label",
|
"host1/blog: reputation_hit", "low label",
|
||||||
"listed by a DNS blocklist\nclient: 192.0.2.2\nnetblock: 192.0.2.2/32",
|
"listed by a DNS blocklist\nclient: 192.0.2.2\nnetblock: 192.0.2.2/32",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
@@ -1526,7 +1526,7 @@ func anAlertForEachEvent() []eventMessage {
|
|||||||
Event: alerts.EventSourceFailure, Reason: "asking GeoJS failed",
|
Event: alerts.EventSourceFailure, Reason: "asking GeoJS failed",
|
||||||
Detail: map[string]any{"source": "geojs"},
|
Detail: map[string]any{"source": "geojs"},
|
||||||
},
|
},
|
||||||
"fsn1app1/blog: source_failure", "high warning",
|
"host1/blog: source_failure", "high warning",
|
||||||
"asking GeoJS failed\nsource: geojs",
|
"asking GeoJS failed\nsource: geojs",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
@@ -1537,7 +1537,7 @@ func anAlertForEachEvent() []eventMessage {
|
|||||||
"file": "/etc/sww/rules.d/50-app.rules", "error": "line 2: no action",
|
"file": "/etc/sww/rules.d/50-app.rules", "error": "line 2: no action",
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
"fsn1app1/blog: file_error", "high warning",
|
"host1/blog: file_error", "high warning",
|
||||||
"a rule file has an error, and the rules stay as they were\n" +
|
"a rule file has an error, and the rules stay as they were\n" +
|
||||||
"file: /etc/sww/rules.d/50-app.rules\nerror: line 2: no action",
|
"file: /etc/sww/rules.d/50-app.rules\nerror: line 2: no action",
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -0,0 +1,169 @@
|
|||||||
|
package bans_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/netip"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestBlacklistedNetblockIsBannedUntilTheGivenEnd(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
ledger := bans.New(defaultParams())
|
||||||
|
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
|
now := midnight()
|
||||||
|
expires := now.Add(30 * day)
|
||||||
|
|
||||||
|
made := ledger.BanForAbuseIPDB(
|
||||||
|
now, expires, map[netip.Prefix]int64{netblock: 95},
|
||||||
|
)
|
||||||
|
if made != 1 || ledger.Made(bans.CauseAbuseIPDB) != 1 {
|
||||||
|
t.Fatalf("made %d bans, %d counted, want 1 and 1",
|
||||||
|
made, ledger.Made(bans.CauseAbuseIPDB))
|
||||||
|
}
|
||||||
|
|
||||||
|
ban, banned, madePermanent := ledger.Check(netblock.Addr(), now)
|
||||||
|
if !banned || madePermanent || !ban.Expires.Equal(expires) ||
|
||||||
|
ban.Cause != bans.CauseAbuseIPDB ||
|
||||||
|
ban.Reason != "on AbuseIPDB's blacklist with a score of 95" ||
|
||||||
|
len(ban.Notes.Reputation) != 1 ||
|
||||||
|
ban.Notes.Reputation[0].Source != "abuseipdb" ||
|
||||||
|
*ban.Notes.Reputation[0].Score != 95 {
|
||||||
|
t.Errorf(
|
||||||
|
"ban %+v, banned %t, made permanent %t, want the blacklist's "+
|
||||||
|
"ban until %s",
|
||||||
|
ban,
|
||||||
|
banned,
|
||||||
|
madePermanent,
|
||||||
|
expires,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, banned, _ := ledger.Check(netblock.Addr(), expires); banned {
|
||||||
|
t.Error("the ban refuses a request at its end")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBlacklistLeavesANetblockWithAnActiveBanAsItIs(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
ledger := bans.New(defaultParams())
|
||||||
|
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
|
now := midnight()
|
||||||
|
scores := map[netip.Prefix]int64{netblock: 100}
|
||||||
|
|
||||||
|
limitBan, _ := ledger.Ban(
|
||||||
|
netblock,
|
||||||
|
now,
|
||||||
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
|
)
|
||||||
|
|
||||||
|
if made := ledger.BanForAbuseIPDB(now, now.Add(30*day), scores); made != 0 {
|
||||||
|
t.Errorf("made %d bans over a ban for a limit, want none", made)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Once that ban has ended the netblock is banned, and the blacklist
|
||||||
|
// fetched again makes no second ban while the first lasts.
|
||||||
|
later := limitBan.Expires
|
||||||
|
if made := ledger.BanForAbuseIPDB(later, later.Add(30*day), scores); made != 1 {
|
||||||
|
t.Errorf("made %d bans after the ban for a limit, want 1", made)
|
||||||
|
}
|
||||||
|
|
||||||
|
later = later.Add(time.Hour)
|
||||||
|
if made := ledger.BanForAbuseIPDB(later, later.Add(30*day), scores); made != 0 {
|
||||||
|
t.Errorf("made %d bans while the blacklist's lasts, want none", made)
|
||||||
|
}
|
||||||
|
|
||||||
|
wantBans(t, ledger, map[netip.Prefix]int{netblock: 2})
|
||||||
|
|
||||||
|
if got := ledger.Bans(netblock)[1].Notes.EarlierBans; got != (bans.EarlierBans{
|
||||||
|
Limit: 1,
|
||||||
|
}) {
|
||||||
|
t.Errorf("earlier bans %+v, want 1 for a limit", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLiftedBlacklistBanHoldsUntilItsEnd(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
ledger := bans.New(defaultParams())
|
||||||
|
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
|
now := midnight()
|
||||||
|
expires := now.Add(30 * day)
|
||||||
|
scores := map[netip.Prefix]int64{netblock: 100}
|
||||||
|
|
||||||
|
ledger.BanForAbuseIPDB(now, expires, scores)
|
||||||
|
ledger.Lift(netblock.Addr(), now)
|
||||||
|
|
||||||
|
// The admin lifted it: the blacklist fetched again until its end bans
|
||||||
|
// nothing.
|
||||||
|
later := expires.Add(-time.Hour)
|
||||||
|
if made := ledger.BanForAbuseIPDB(later, later.Add(30*day), scores); made != 0 {
|
||||||
|
t.Errorf("made %d bans before the lifted ban's end, want none", made)
|
||||||
|
}
|
||||||
|
|
||||||
|
// From its end on, the blacklist bans the netblock again, and the
|
||||||
|
// lifted ban, ended, is forgotten.
|
||||||
|
if made := ledger.BanForAbuseIPDB(expires, expires.Add(30*day), scores); made != 1 {
|
||||||
|
t.Errorf("made %d bans at the lifted ban's end, want 1", made)
|
||||||
|
}
|
||||||
|
|
||||||
|
wantBans(t, ledger, map[netip.Prefix]int{netblock: 1})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEndedBlacklistBansAreForgottenAtTheNextFetch(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
ledger := bans.New(defaultParams())
|
||||||
|
listed := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
|
limited := netip.MustParsePrefix("203.0.113.10/32")
|
||||||
|
now := midnight()
|
||||||
|
expires := now.Add(30 * day)
|
||||||
|
|
||||||
|
ledger.BanForAbuseIPDB(now, expires, map[netip.Prefix]int64{listed: 100})
|
||||||
|
ledger.Ban(limited, now, bans.Grounds{Cause: bans.CauseLimit})
|
||||||
|
|
||||||
|
// listed is off the blacklist when it is fetched after the ban's end.
|
||||||
|
if made := ledger.BanForAbuseIPDB(expires, expires.Add(30*day), nil); made != 0 {
|
||||||
|
t.Errorf("made %d bans for an empty blacklist, want none", made)
|
||||||
|
}
|
||||||
|
|
||||||
|
wantBans(t, ledger, map[netip.Prefix]int{listed: 0, limited: 1})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBlacklistBansDoNotCountTowardMaxBans(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
params := defaultParams()
|
||||||
|
params.MaxBans = 1
|
||||||
|
ledger := bans.New(params)
|
||||||
|
limited := netip.MustParsePrefix("203.0.113.1/32")
|
||||||
|
attacker := netip.MustParsePrefix("203.0.113.2/32")
|
||||||
|
now := midnight()
|
||||||
|
|
||||||
|
scores := map[netip.Prefix]int64{}
|
||||||
|
for i := range 3 {
|
||||||
|
scores[netip.PrefixFrom(netip.AddrFrom4([4]byte{198, 51, 100, byte(i)}), 32)] = 100
|
||||||
|
}
|
||||||
|
|
||||||
|
ledger.Ban(limited, now, bans.Grounds{Cause: bans.CauseLimit})
|
||||||
|
|
||||||
|
if made := ledger.BanForAbuseIPDB(now, now.Add(30*day), scores); made != 3 {
|
||||||
|
t.Errorf("made %d bans for 3 netblocks, want 3", made)
|
||||||
|
}
|
||||||
|
|
||||||
|
wantBans(t, ledger, map[netip.Prefix]int{limited: 1})
|
||||||
|
|
||||||
|
// The ban for an attack takes the place of the one for a limit, and no
|
||||||
|
// blacklist ban goes.
|
||||||
|
ledger.Ban(attacker, now, bans.Grounds{Cause: bans.CauseAttack})
|
||||||
|
|
||||||
|
want := map[netip.Prefix]int{limited: 0, attacker: 1}
|
||||||
|
for netblock := range scores {
|
||||||
|
want[netblock] = 1
|
||||||
|
}
|
||||||
|
|
||||||
|
wantBans(t, ledger, want)
|
||||||
|
}
|
||||||
+64
-23
@@ -61,7 +61,7 @@ func TestAdminsBansAreNeverDroppedAndDoNotCountTowardMaxBans(t *testing.T) {
|
|||||||
wantBans(t, ledger, map[netip.Prefix]int{adminsOnly: 2, both: 2})
|
wantBans(t, ledger, map[netip.Prefix]int{adminsOnly: 2, both: 2})
|
||||||
|
|
||||||
// A new ban drops the ban sww made, and only that one.
|
// A new ban drops the ban sww made, and only that one.
|
||||||
ledger.BanForLimit(second, midnight(), bans.Notes{})
|
ledger.Ban(second, midnight(), bans.Grounds{Cause: bans.CauseLimit})
|
||||||
wantBans(
|
wantBans(
|
||||||
t,
|
t,
|
||||||
ledger,
|
ledger,
|
||||||
@@ -77,7 +77,7 @@ func TestAdminsBansAreNeverDroppedAndDoNotCountTowardMaxBans(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// And the next drops that one.
|
// And the next drops that one.
|
||||||
ledger.BanForLimit(third, midnight(), bans.Notes{})
|
ledger.Ban(third, midnight(), bans.Grounds{Cause: bans.CauseLimit})
|
||||||
wantBans(
|
wantBans(
|
||||||
t,
|
t,
|
||||||
ledger,
|
ledger,
|
||||||
@@ -98,20 +98,28 @@ func TestEarlierBansCountABanDroppedAfterAnAdminsBan(t *testing.T) {
|
|||||||
// twice, each time after its last ban has ended.
|
// twice, each time after its last ban has ended.
|
||||||
ledger.BanForAdmin(netblock, midnight(), midnight().Add(time.Minute),
|
ledger.BanForAdmin(netblock, midnight(), midnight().Add(time.Minute),
|
||||||
"probes for logins")
|
"probes for logins")
|
||||||
first, _ := ledger.BanForLimit(
|
first, _ := ledger.Ban(
|
||||||
netblock,
|
netblock,
|
||||||
midnight().Add(time.Minute),
|
midnight().Add(time.Minute),
|
||||||
bans.Notes{},
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
|
)
|
||||||
|
second, _ := ledger.Ban(
|
||||||
|
netblock,
|
||||||
|
first.Expires,
|
||||||
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
)
|
)
|
||||||
second, _ := ledger.BanForLimit(netblock, first.Expires, bans.Notes{})
|
|
||||||
|
|
||||||
// A ban on another netblock fills the ledger, and the netblock's first
|
// A ban on another netblock fills the ledger, and the netblock's first
|
||||||
// ban for a limit, its earliest that is not an admin's, goes.
|
// ban for a limit, its earliest that is not an admin's, goes.
|
||||||
ledger.BanForLimit(other, second.Start, bans.Notes{})
|
ledger.Ban(other, second.Start, bans.Grounds{Cause: bans.CauseLimit})
|
||||||
wantBans(t, ledger, map[netip.Prefix]int{netblock: 2, other: 1})
|
wantBans(t, ledger, map[netip.Prefix]int{netblock: 2, other: 1})
|
||||||
|
|
||||||
// The netblock's next ban still counts it.
|
// The netblock's next ban still counts it.
|
||||||
third, _ := ledger.BanForLimit(netblock, second.Expires, bans.Notes{})
|
third, _ := ledger.Ban(
|
||||||
|
netblock,
|
||||||
|
second.Expires,
|
||||||
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
|
)
|
||||||
if third.Notes.EarlierBans != (bans.EarlierBans{Limit: 2, Admin: 1}) {
|
if third.Notes.EarlierBans != (bans.EarlierBans{Limit: 2, Admin: 1}) {
|
||||||
t.Errorf("the third ban for a limit has earlier bans %+v, "+
|
t.Errorf("the third ban for a limit has earlier bans %+v, "+
|
||||||
"want 2 for a limit and 1 an admin's", third.Notes.EarlierBans)
|
"want 2 for a limit and 1 an admin's", third.Notes.EarlierBans)
|
||||||
@@ -127,15 +135,27 @@ func TestEarlierBansCountABanAnAdminKeepsOnce(t *testing.T) {
|
|||||||
// The netblock breaks a limit twice, the second time after its first
|
// The netblock breaks a limit twice, the second time after its first
|
||||||
// ban has ended, and the second ban's notes count the first. An admin
|
// ban has ended, and the second ban's notes count the first. An admin
|
||||||
// then keeps the first ban.
|
// then keeps the first ban.
|
||||||
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
|
first, _ := ledger.Ban(
|
||||||
second, _ := ledger.BanForLimit(netblock, first.Expires, bans.Notes{})
|
netblock,
|
||||||
|
midnight(),
|
||||||
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
|
)
|
||||||
|
second, _ := ledger.Ban(
|
||||||
|
netblock,
|
||||||
|
first.Expires,
|
||||||
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
|
)
|
||||||
|
|
||||||
kept := first
|
kept := first
|
||||||
kept.Cause = bans.CauseAdmin
|
kept.Cause = bans.CauseAdmin
|
||||||
ledger.LoadEdit([]bans.Ban{kept, second})
|
ledger.LoadEdit([]bans.Ban{kept, second})
|
||||||
|
|
||||||
// The netblock's next ban counts the kept ban once, as an admin's.
|
// The netblock's next ban counts the kept ban once, as an admin's.
|
||||||
third, _ := ledger.BanForLimit(netblock, second.Expires, bans.Notes{})
|
third, _ := ledger.Ban(
|
||||||
|
netblock,
|
||||||
|
second.Expires,
|
||||||
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
|
)
|
||||||
if third.Notes.EarlierBans != (bans.EarlierBans{Limit: 1, Admin: 1}) {
|
if third.Notes.EarlierBans != (bans.EarlierBans{Limit: 1, Admin: 1}) {
|
||||||
t.Errorf("the third ban for a limit has earlier bans %+v, "+
|
t.Errorf("the third ban for a limit has earlier bans %+v, "+
|
||||||
"want 1 for a limit and 1 an admin's", third.Notes.EarlierBans)
|
"want 1 for a limit and 1 an admin's", third.Notes.EarlierBans)
|
||||||
@@ -147,20 +167,37 @@ func TestReasonOfTheBansSwwMakes(t *testing.T) {
|
|||||||
|
|
||||||
ledger := bans.New(defaultParams())
|
ledger := bans.New(defaultParams())
|
||||||
|
|
||||||
limit, _ := ledger.BanForLimit(
|
limit, _ := ledger.Ban(
|
||||||
netip.MustParsePrefix("203.0.113.1/32"),
|
netip.MustParsePrefix("203.0.113.1/32"),
|
||||||
midnight(),
|
midnight(),
|
||||||
bans.Notes{Kind: "requests", Limit: 1000, Window: "minute"},
|
bans.Grounds{
|
||||||
|
Cause: bans.CauseLimit,
|
||||||
|
Notes: bans.Notes{
|
||||||
|
Kind: "requests",
|
||||||
|
Limit: 1000,
|
||||||
|
Window: "minute",
|
||||||
|
},
|
||||||
|
},
|
||||||
)
|
)
|
||||||
byteLimit, _ := ledger.BanForLimit(
|
byteLimit, _ := ledger.Ban(
|
||||||
netip.MustParsePrefix("203.0.113.3/32"),
|
netip.MustParsePrefix("203.0.113.3/32"),
|
||||||
midnight(),
|
midnight(),
|
||||||
bans.Notes{Kind: "bytes", Limit: 10 << 30, Window: "hour"},
|
bans.Grounds{
|
||||||
|
Cause: bans.CauseLimit,
|
||||||
|
Notes: bans.Notes{
|
||||||
|
Kind: "bytes",
|
||||||
|
Limit: 10 << 30,
|
||||||
|
Window: "hour",
|
||||||
|
},
|
||||||
|
},
|
||||||
)
|
)
|
||||||
attack, _ := ledger.BanForAttack(
|
attack, _ := ledger.Ban(
|
||||||
netip.MustParsePrefix("203.0.113.2/32"),
|
netip.MustParsePrefix("203.0.113.2/32"),
|
||||||
midnight(),
|
midnight(),
|
||||||
bans.Notes{RuleID: "git-dir", Target: "path"},
|
bans.Grounds{
|
||||||
|
Cause: bans.CauseAttack,
|
||||||
|
Notes: bans.Notes{RuleID: "git-dir", Target: "path"},
|
||||||
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
for _, tc := range []struct{ got, want string }{
|
for _, tc := range []struct{ got, want string }{
|
||||||
@@ -200,7 +237,7 @@ func TestLiftedBanForALimitRefusesNothingAndMakesNoBanLonger(t *testing.T) {
|
|||||||
t.Error("the lifted ban refuses")
|
t.Error("the lifted ban refuses")
|
||||||
}
|
}
|
||||||
|
|
||||||
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
|
ban, _ := ledger.Ban(netblock, now, bans.Grounds{Cause: bans.CauseLimit})
|
||||||
if ban.Expires.Sub(ban.Start) != time.Hour ||
|
if ban.Expires.Sub(ban.Start) != time.Hour ||
|
||||||
ban.Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) {
|
ban.Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) {
|
||||||
t.Errorf(
|
t.Errorf(
|
||||||
@@ -251,7 +288,11 @@ func TestLiftedBanForAnAttackRefusesNothingAndMakesNoBanLonger(
|
|||||||
}
|
}
|
||||||
|
|
||||||
// The next clear sign of attack bans for seven days, as a first does.
|
// The next clear sign of attack bans for seven days, as a first does.
|
||||||
ban, _ := ledger.BanForAttack(netblock, now, bans.Notes{})
|
ban, _ := ledger.Ban(
|
||||||
|
netblock,
|
||||||
|
now,
|
||||||
|
bans.Grounds{Cause: bans.CauseAttack},
|
||||||
|
)
|
||||||
if ban.Expires.Sub(ban.Start) != 7*day {
|
if ban.Expires.Sub(ban.Start) != 7*day {
|
||||||
t.Errorf(
|
t.Errorf(
|
||||||
"the next ban for an attack ends at %s, want seven days on",
|
"the next ban for an attack ends at %s, want seven days on",
|
||||||
@@ -264,10 +305,10 @@ func TestLoadEditCountsTheBansAnAdminMade(t *testing.T) {
|
|||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
ledger := bans.New(defaultParams())
|
ledger := bans.New(defaultParams())
|
||||||
made, _ := ledger.BanForLimit(
|
made, _ := ledger.Ban(
|
||||||
netip.MustParsePrefix("203.0.113.1/32"),
|
netip.MustParsePrefix("203.0.113.1/32"),
|
||||||
midnight(),
|
midnight(),
|
||||||
bans.Notes{},
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
)
|
)
|
||||||
atStart := bans.Ban{
|
atStart := bans.Ban{
|
||||||
Netblock: netip.MustParsePrefix("203.0.113.2/32"),
|
Netblock: netip.MustParsePrefix("203.0.113.2/32"),
|
||||||
@@ -311,7 +352,7 @@ func TestAdminsBanIsMadeWhileAnotherLasts(t *testing.T) {
|
|||||||
ledger := bans.New(defaultParams())
|
ledger := bans.New(defaultParams())
|
||||||
|
|
||||||
// An hour's ban for a broken limit.
|
// An hour's ban for a broken limit.
|
||||||
ledger.BanForLimit(netblock, midnight(), bans.Notes{})
|
ledger.Ban(netblock, midnight(), bans.Grounds{Cause: bans.CauseLimit})
|
||||||
wantChanged(t, ledger, true)
|
wantChanged(t, ledger, true)
|
||||||
|
|
||||||
// A minute later an admin bans the netblock for good, named by an
|
// A minute later an admin bans the netblock for good, named by an
|
||||||
@@ -431,10 +472,10 @@ func TestPermanentBanOnAWiderNetblockIsReturnedOverATemporaryBan(
|
|||||||
ledger := bans.New(defaultParams())
|
ledger := bans.New(defaultParams())
|
||||||
|
|
||||||
// The temporary ban is made first, so that its netblock is looked at first.
|
// The temporary ban is made first, so that its netblock is looked at first.
|
||||||
ledger.BanForLimit(
|
ledger.Ban(
|
||||||
netip.MustParsePrefix("203.0.113.9/32"),
|
netip.MustParsePrefix("203.0.113.9/32"),
|
||||||
midnight(),
|
midnight(),
|
||||||
bans.Notes{},
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
)
|
)
|
||||||
ledger.BanForAdmin(
|
ledger.BanForAdmin(
|
||||||
netip.MustParsePrefix("203.0.113.0/24"),
|
netip.MustParsePrefix("203.0.113.0/24"),
|
||||||
|
|||||||
+215
-116
@@ -1,8 +1,9 @@
|
|||||||
// Package bans is the ban ledger: the bans sww makes on the
|
// Package bans is the ban ledger: the bans sww makes on the
|
||||||
// netblocks of clients that break a rate limit, a byte limit or the refusal
|
// netblocks of clients that break a rate limit, a byte limit or the refusal
|
||||||
// limit or show a clear sign of attack, and those an admin makes, with their
|
// limit or show a clear sign of attack, or are on AbuseIPDB's blacklist, and
|
||||||
// notes, as the "Bans" section of SPEC.md describes. The bans are kept in
|
// those an admin makes, with their notes, as the "Bans" section of SPEC.md
|
||||||
// memory, and written to bans.json and read from it by the state package.
|
// describes. The bans are kept in memory, and written to bans.json and read
|
||||||
|
// from it by the state package.
|
||||||
package bans
|
package bans
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -26,6 +27,9 @@ const (
|
|||||||
// CauseAdmin is a ban an admin made, or one sww made that an
|
// CauseAdmin is a ban an admin made, or one sww made that an
|
||||||
// admin keeps. It is never dropped.
|
// admin keeps. It is never dropped.
|
||||||
CauseAdmin = "admin"
|
CauseAdmin = "admin"
|
||||||
|
// CauseAbuseIPDB is a ban sww made for a client on AbuseIPDB's
|
||||||
|
// blacklist. It is forgotten once it has ended.
|
||||||
|
CauseAbuseIPDB = "abuseipdb"
|
||||||
)
|
)
|
||||||
|
|
||||||
// repeatFactor is how many times as long as the netblock's last ban a ban
|
// repeatFactor is how many times as long as the netblock's last ban a ban
|
||||||
@@ -50,10 +54,11 @@ type Params struct {
|
|||||||
// AttackBanDuration is how long a first ban for a clear sign of attack
|
// AttackBanDuration is how long a first ban for a clear sign of attack
|
||||||
// lasts.
|
// lasts.
|
||||||
AttackBanDuration time.Duration
|
AttackBanDuration time.Duration
|
||||||
// MaxBans is the most bans held whose cause is not CauseAdmin, at
|
// MaxBans is the most bans held whose cause is CauseLimit or
|
||||||
// least one. Past it, the earliest such ban of the netblock that has
|
// CauseAttack, at least one. Past it, the earliest such ban of the
|
||||||
// gone longest without a request is dropped. Bans whose cause is
|
// netblock that has gone longest without a request is dropped. Bans
|
||||||
// CauseAdmin are held besides, and never dropped.
|
// whose cause is CauseAdmin or CauseAbuseIPDB are held besides, and
|
||||||
|
// never dropped this way.
|
||||||
MaxBans int
|
MaxBans int
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -63,10 +68,11 @@ type Ban struct {
|
|||||||
Start time.Time
|
Start time.Time
|
||||||
// Expires is when the ban ends, zero for a permanent ban.
|
// Expires is when the ban ends, zero for a permanent ban.
|
||||||
Expires time.Time
|
Expires time.Time
|
||||||
// Cause is CauseLimit, CauseAttack or CauseAdmin.
|
// Cause is CauseLimit, CauseAttack, CauseAdmin or CauseAbuseIPDB.
|
||||||
Cause string
|
Cause string
|
||||||
// Reason is a short text: for a ban sww made, the limit broken or
|
// Reason is a short text: for a ban sww made, the limit broken, the
|
||||||
// the rule that matched; for an admin's, what the admin wrote.
|
// rule that matched or AbuseIPDB's score; for an admin's, what the
|
||||||
|
// admin wrote.
|
||||||
Reason string
|
Reason string
|
||||||
// Lifted is when an admin lifted the ban, zero while no admin has. A
|
// Lifted is when an admin lifted the ban, zero while no admin has. A
|
||||||
// lifted ban refuses nothing, and does not make the netblock's next
|
// lifted ban refuses nothing, and does not make the netblock's next
|
||||||
@@ -127,7 +133,8 @@ type Notes struct {
|
|||||||
// log's reputation names them. It is left out when none did.
|
// log's reputation names them. It is left out when none did.
|
||||||
Reputation []ReputationHit `json:"reputation,omitempty"`
|
Reputation []ReputationHit `json:"reputation,omitempty"`
|
||||||
// Request is the request that broke the limit, or whose bytes broke
|
// Request is the request that broke the limit, or whose bytes broke
|
||||||
// it, or that was the clear sign of attack.
|
// it, or that was the clear sign of attack. A ban for AbuseIPDB's
|
||||||
|
// blacklist has none, and leaves it zero.
|
||||||
Request Request `json:"request"`
|
Request Request `json:"request"`
|
||||||
// Requests is how many requests the netblock has sent since it was
|
// Requests is how many requests the netblock has sent since it was
|
||||||
// first seen, and Refused how many of them the ban has refused so
|
// first seen, and Refused how many of them the ban has refused so
|
||||||
@@ -150,9 +157,10 @@ type ReputationHit struct {
|
|||||||
|
|
||||||
// EarlierBans counts a netblock's bans before a ban, by cause.
|
// EarlierBans counts a netblock's bans before a ban, by cause.
|
||||||
type EarlierBans struct {
|
type EarlierBans struct {
|
||||||
Limit int `json:"limit"`
|
Limit int `json:"limit"`
|
||||||
Attack int `json:"attack"`
|
Attack int `json:"attack"`
|
||||||
Admin int `json:"admin"`
|
Admin int `json:"admin"`
|
||||||
|
AbuseIPDB int `json:"abuseipdb"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// Request is a request in a ban's notes. Each text is cut to 256 bytes.
|
// Request is a request in a ban's notes. Each text is cut to 256 bytes.
|
||||||
@@ -180,8 +188,8 @@ type Ledger struct {
|
|||||||
// netblocks holds each banned netblock's bans, oldest first. Check and
|
// netblocks holds each banned netblock's bans, oldest first. Check and
|
||||||
// Find make each netblock they find the most recently seen.
|
// Find make each netblock they find the most recently seen.
|
||||||
netblocks *simplelru.LRU[netip.Prefix, *[]Ban]
|
netblocks *simplelru.LRU[netip.Prefix, *[]Ban]
|
||||||
// held is how many bans netblocks holds whose cause is not CauseAdmin,
|
// held is how many bans netblocks holds whose cause is CauseLimit or
|
||||||
// at most params.MaxBans.
|
// CauseAttack, at most params.MaxBans.
|
||||||
held int
|
held int
|
||||||
// made is how many bans have been made since the start, by cause: by
|
// made is how many bans have been made since the start, by cause: by
|
||||||
// the ledger, and by an admin, through BanForAdmin or in an edit of
|
// the ledger, and by an admin, through BanForAdmin or in an edit of
|
||||||
@@ -198,8 +206,9 @@ type Ledger struct {
|
|||||||
// New returns a Ledger with no ban yet.
|
// New returns a Ledger with no ban yet.
|
||||||
func New(params Params) *Ledger {
|
func New(params Params) *Ledger {
|
||||||
// The ledger drops bans itself, and never those whose cause is
|
// The ledger drops bans itself, and never those whose cause is
|
||||||
// CauseAdmin, however many there are, so the LRU has no limit of its
|
// CauseAdmin or CauseAbuseIPDB to make room, however many there are, so
|
||||||
// own: it keeps the netblocks in the order they were last seen.
|
// the LRU has no limit of its own: it keeps the netblocks in the order
|
||||||
|
// they were last seen.
|
||||||
netblocks, err := simplelru.NewLRU[netip.Prefix, *[]Ban](
|
netblocks, err := simplelru.NewLRU[netip.Prefix, *[]Ban](
|
||||||
math.MaxInt,
|
math.MaxInt,
|
||||||
nil,
|
nil,
|
||||||
@@ -291,81 +300,50 @@ func activeBan(bans []Ban, now time.Time) *Ban {
|
|||||||
return newest
|
return newest
|
||||||
}
|
}
|
||||||
|
|
||||||
// BanForLimit bans netblock at now for a broken limit, with notes, and
|
// Grounds are why a request makes a ban: its cause, CauseLimit or
|
||||||
// returns the ban, and true. A first ban lasts LimitBanDuration. A ban
|
// CauseAttack, and its notes.
|
||||||
// made within LimitBanRepeatWindow after the netblock's ban that ended
|
type Grounds struct {
|
||||||
// last, other than one for a clear sign of attack or a lifted one, lasts
|
Cause string
|
||||||
// repeatFactor times as long as that one. A ban that would be longer
|
Notes Notes
|
||||||
// than LimitBanPermanentAbove is permanent instead. If a ban on netblock is still
|
|
||||||
// active, as when two of its requests break a limit at once, that ban is
|
|
||||||
// returned with false, and no other is made. The ledger fills in the
|
|
||||||
// notes' Refused and EarlierBans itself, and gives the ban the reason
|
|
||||||
// "<Kind> per <Window> over the limit of <Limit>", from the notes, such
|
|
||||||
// as "requests per minute over the limit of 1000".
|
|
||||||
func (l *Ledger) BanForLimit(
|
|
||||||
netblock netip.Prefix, now time.Time, notes Notes,
|
|
||||||
) (Ban, bool) {
|
|
||||||
return l.ban(
|
|
||||||
netblock,
|
|
||||||
now,
|
|
||||||
CauseLimit,
|
|
||||||
limitReason(notes),
|
|
||||||
notes,
|
|
||||||
true,
|
|
||||||
)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// WouldBanForLimit returns what BanForLimit would, without making the ban:
|
// Ban bans netblock at now on grounds, and returns the ban, and true. If
|
||||||
// what observe mode would have done.
|
// a ban on netblock is still active, as when two of its requests break a
|
||||||
func (l *Ledger) WouldBanForLimit(
|
// limit at once, that ban is returned with false, and no other is made.
|
||||||
netblock netip.Prefix, now time.Time, notes Notes,
|
// The ledger fills in the notes' Refused and EarlierBans itself.
|
||||||
|
//
|
||||||
|
// A ban for a broken limit:
|
||||||
|
// - lasts LimitBanDuration the first time;
|
||||||
|
// - lasts repeatFactor times as long as the netblock's ban that ended
|
||||||
|
// last, when made within LimitBanRepeatWindow after it, unless that
|
||||||
|
// ban was for a clear sign of attack or was lifted;
|
||||||
|
// - is permanent instead when it would be longer than
|
||||||
|
// LimitBanPermanentAbove;
|
||||||
|
// - has the reason "<Kind> per <Window> over the limit of <Limit>", from
|
||||||
|
// the notes, such as "requests per minute over the limit of 1000".
|
||||||
|
//
|
||||||
|
// A ban for a clear sign of attack:
|
||||||
|
// - lasts AttackBanDuration the first time;
|
||||||
|
// - is permanent once the netblock has had one that was not lifted;
|
||||||
|
// - has the reason "matched the rule <RuleID>", or "asked for the trap
|
||||||
|
// path <TrapPath>".
|
||||||
|
func (l *Ledger) Ban(
|
||||||
|
netblock netip.Prefix, now time.Time, grounds Grounds,
|
||||||
) (Ban, bool) {
|
) (Ban, bool) {
|
||||||
return l.ban(
|
return l.ban(netblock, now, grounds, true)
|
||||||
netblock,
|
|
||||||
now,
|
|
||||||
CauseLimit,
|
|
||||||
limitReason(notes),
|
|
||||||
notes,
|
|
||||||
false,
|
|
||||||
)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// BanForAttack bans netblock at now for a clear sign of attack, with
|
// WouldBan returns what Ban would, without making the ban: what observe
|
||||||
// notes, and returns the ban, and whether it made it, as BanForLimit
|
// mode would have done.
|
||||||
// does. A first ban lasts AttackBanDuration; once the netblock has had
|
func (l *Ledger) WouldBan(
|
||||||
// one that was not lifted, the next is permanent. Its reason is "matched
|
netblock netip.Prefix, now time.Time, grounds Grounds,
|
||||||
// the rule <RuleID>", or "asked for the trap path <TrapPath>".
|
|
||||||
func (l *Ledger) BanForAttack(
|
|
||||||
netblock netip.Prefix, now time.Time, notes Notes,
|
|
||||||
) (Ban, bool) {
|
) (Ban, bool) {
|
||||||
return l.ban(
|
return l.ban(netblock, now, grounds, false)
|
||||||
netblock,
|
|
||||||
now,
|
|
||||||
CauseAttack,
|
|
||||||
attackReason(notes),
|
|
||||||
notes,
|
|
||||||
true,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
// WouldBanForAttack returns what BanForAttack would, without making the
|
|
||||||
// ban: what observe mode would have done.
|
|
||||||
func (l *Ledger) WouldBanForAttack(
|
|
||||||
netblock netip.Prefix, now time.Time, notes Notes,
|
|
||||||
) (Ban, bool) {
|
|
||||||
return l.ban(
|
|
||||||
netblock,
|
|
||||||
now,
|
|
||||||
CauseAttack,
|
|
||||||
attackReason(notes),
|
|
||||||
notes,
|
|
||||||
false,
|
|
||||||
)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// WouldBePermanent reports whether a ban on netblock for cause, CauseLimit
|
// WouldBePermanent reports whether a ban on netblock for cause, CauseLimit
|
||||||
// or CauseAttack, made at now would be permanent, as BanForLimit or
|
// or CauseAttack, made at now would be permanent, as Ban would make it. It
|
||||||
// BanForAttack would make it. It works out nothing else of the ban.
|
// works out nothing else of the ban.
|
||||||
func (l *Ledger) WouldBePermanent(
|
func (l *Ledger) WouldBePermanent(
|
||||||
netblock netip.Prefix, now time.Time, cause string,
|
netblock netip.Prefix, now time.Time, cause string,
|
||||||
) bool {
|
) bool {
|
||||||
@@ -402,10 +380,9 @@ func attackReason(notes Notes) string {
|
|||||||
|
|
||||||
// BanForAdmin bans netblock at now for an admin, with reason, until
|
// BanForAdmin bans netblock at now for an admin, with reason, until
|
||||||
// expires, or for good when expires is zero, and returns the ban, whose
|
// expires, or for good when expires is zero, and returns the ban, whose
|
||||||
// cause is CauseAdmin. Unlike BanForLimit and BanForAttack, it makes the
|
// cause is CauseAdmin. Unlike Ban, it makes the ban even while another
|
||||||
// ban even while another on netblock is active, since the admin asked
|
// on netblock is active, since the admin asked for this one. The ledger
|
||||||
// for this one. The ledger fills in the notes' EarlierBans, and counts
|
// fills in the notes' EarlierBans, and counts the ban among those made.
|
||||||
// the ban among those made.
|
|
||||||
func (l *Ledger) BanForAdmin(
|
func (l *Ledger) BanForAdmin(
|
||||||
netblock netip.Prefix, now, expires time.Time, reason string,
|
netblock netip.Prefix, now, expires time.Time, reason string,
|
||||||
) Ban {
|
) Ban {
|
||||||
@@ -429,6 +406,88 @@ func (l *Ledger) BanForAdmin(
|
|||||||
return ban
|
return ban
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// BanForAbuseIPDB bans each netblock in scores at now until expires, and
|
||||||
|
// returns how many bans it made. scores are the netblocks of the clients
|
||||||
|
// on AbuseIPDB's blacklist, each with the highest score of its clients.
|
||||||
|
// A netblock is left as it is while a ban on it is active, or while a ban
|
||||||
|
// for the blacklist that an admin lifted has not reached its end: the
|
||||||
|
// admin's lift holds that long. Each ban whose cause is CauseAbuseIPDB
|
||||||
|
// and that has ended is forgotten first, so that the bans held for the
|
||||||
|
// blacklist are those of the clients listed within one ban's length.
|
||||||
|
func (l *Ledger) BanForAbuseIPDB(
|
||||||
|
now, expires time.Time, scores map[netip.Prefix]int64,
|
||||||
|
) int {
|
||||||
|
l.mu.Lock()
|
||||||
|
defer l.mu.Unlock()
|
||||||
|
|
||||||
|
changed := l.forgetEndedAbuseIPDBBans(now)
|
||||||
|
made := 0
|
||||||
|
|
||||||
|
for netblock, score := range scores {
|
||||||
|
netblock = netblock.Masked()
|
||||||
|
|
||||||
|
var notes Notes
|
||||||
|
|
||||||
|
if held, found := l.netblocks.Peek(netblock); found {
|
||||||
|
if activeBan(*held, now) != nil ||
|
||||||
|
liftedAbuseIPDBBan(*held, now) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
notes.EarlierBans = earlierBans(*held)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The source is named as the request log names AbuseIPDB.
|
||||||
|
notes.Reputation = []ReputationHit{
|
||||||
|
{Source: CauseAbuseIPDB, Score: &score},
|
||||||
|
}
|
||||||
|
|
||||||
|
l.add(Ban{
|
||||||
|
Netblock: netblock,
|
||||||
|
Start: now,
|
||||||
|
Expires: expires,
|
||||||
|
Cause: CauseAbuseIPDB,
|
||||||
|
Reason: fmt.Sprintf(
|
||||||
|
"on AbuseIPDB's blacklist with a score of %d", score,
|
||||||
|
),
|
||||||
|
Notes: notes,
|
||||||
|
})
|
||||||
|
|
||||||
|
made++
|
||||||
|
}
|
||||||
|
|
||||||
|
l.made[CauseAbuseIPDB] += made
|
||||||
|
|
||||||
|
if changed || made > 0 {
|
||||||
|
l.markChanged()
|
||||||
|
}
|
||||||
|
|
||||||
|
return made
|
||||||
|
}
|
||||||
|
|
||||||
|
// ForgetEndedAbuseIPDBBans forgets the bans whose cause is CauseAbuseIPDB
|
||||||
|
// and that have ended at now, as BanForAbuseIPDB does first. The state
|
||||||
|
// package calls it each time it reads bans.json, so that these bans are
|
||||||
|
// also forgotten while no fetch of the blacklist reaches the ledger.
|
||||||
|
func (l *Ledger) ForgetEndedAbuseIPDBBans(now time.Time) {
|
||||||
|
l.mu.Lock()
|
||||||
|
defer l.mu.Unlock()
|
||||||
|
|
||||||
|
if l.forgetEndedAbuseIPDBBans(now) {
|
||||||
|
l.markChanged()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// liftedAbuseIPDBBan reports whether bans, a netblock's, hold a ban whose
|
||||||
|
// cause is CauseAbuseIPDB that an admin lifted, and whose end is not
|
||||||
|
// reached at now.
|
||||||
|
func liftedAbuseIPDBBan(bans []Ban, now time.Time) bool {
|
||||||
|
return slices.ContainsFunc(bans, func(ban Ban) bool {
|
||||||
|
return ban.Cause == CauseAbuseIPDB && !ban.Lifted.IsZero() &&
|
||||||
|
(ban.Permanent() || now.Before(ban.Expires))
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
// Lift lifts, at now, every ban active then on a netblock client is in,
|
// Lift lifts, at now, every ban active then on a netblock client is in,
|
||||||
// as an admin does, and returns those bans. A lifted ban is kept, refuses
|
// as an admin does, and returns those bans. A lifted ban is kept, refuses
|
||||||
// nothing, and does not make the netblock's next ban longer.
|
// nothing, and does not make the netblock's next ban longer.
|
||||||
@@ -515,9 +574,9 @@ func (l *Ledger) AddLookup(
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Made returns how many bans for cause have been made since the start:
|
// Made returns how many bans for cause have been made since the start:
|
||||||
// for CauseLimit and CauseAttack, by the ledger; for CauseAdmin, by an
|
// for CauseLimit, CauseAttack and CauseAbuseIPDB, by the ledger; for
|
||||||
// admin, with BanForAdmin or in an edit of bans.json, as LoadEdit counts
|
// CauseAdmin, by an admin, with BanForAdmin or in an edit of bans.json, as
|
||||||
// them. The bans read from bans.json at the start are not among them.
|
// LoadEdit counts them. The bans read from bans.json at the start are not among them.
|
||||||
func (l *Ledger) Made(cause string) int {
|
func (l *Ledger) Made(cause string) int {
|
||||||
l.mu.Lock()
|
l.mu.Lock()
|
||||||
defer l.mu.Unlock()
|
defer l.mu.Unlock()
|
||||||
@@ -574,8 +633,8 @@ func (l *Ledger) Snapshot() []Ban {
|
|||||||
// seen. A ban without a cause is an admin's, and gets CauseAdmin. Each
|
// seen. A ban without a cause is an admin's, and gets CauseAdmin. Each
|
||||||
// netblock is masked to its length, so that 203.0.113.9/24 is
|
// netblock is masked to its length, so that 203.0.113.9/24 is
|
||||||
// 203.0.113.0/24, and each text in the notes is cut to 256 bytes. Past
|
// 203.0.113.0/24, and each text in the notes is cut to 256 bytes. Past
|
||||||
// MaxBans the earliest bans whose cause is not CauseAdmin are dropped, as
|
// MaxBans the earliest bans that count toward it are dropped, as when they
|
||||||
// when they are made.
|
// are made.
|
||||||
func (l *Ledger) Load(bans []Ban) {
|
func (l *Ledger) Load(bans []Ban) {
|
||||||
l.mu.Lock()
|
l.mu.Lock()
|
||||||
defer l.mu.Unlock()
|
defer l.mu.Unlock()
|
||||||
@@ -639,15 +698,42 @@ func (l *Ledger) holds(netblock netip.Prefix, start time.Time) bool {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
// ban bans netblock at now for cause, CauseLimit or CauseAttack, with
|
// forgetEndedAbuseIPDBBans forgets each ban whose cause is CauseAbuseIPDB
|
||||||
// reason and notes, as BanForLimit and BanForAttack describe, and returns
|
// and whose end is reached at now, lifted or not, and each netblock left
|
||||||
// the ban, and whether it made it. Unless keep is true, the ban is not
|
// without a ban. It reports whether it forgot one.
|
||||||
// made, only returned: it is the ban that would have been made.
|
func (l *Ledger) forgetEndedAbuseIPDBBans(now time.Time) bool {
|
||||||
|
forgot := false
|
||||||
|
|
||||||
|
for _, netblock := range l.netblocks.Keys() {
|
||||||
|
bans, _ := l.netblocks.Peek(netblock)
|
||||||
|
|
||||||
|
kept := slices.DeleteFunc(*bans, func(ban Ban) bool {
|
||||||
|
return ban.Cause == CauseAbuseIPDB && !ban.Permanent() &&
|
||||||
|
!now.Before(ban.Expires)
|
||||||
|
})
|
||||||
|
if len(kept) == len(*bans) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
forgot = true
|
||||||
|
|
||||||
|
if len(kept) == 0 {
|
||||||
|
l.netblocks.Remove(netblock)
|
||||||
|
} else {
|
||||||
|
*bans = kept
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return forgot
|
||||||
|
}
|
||||||
|
|
||||||
|
// ban bans netblock at now on grounds, as Ban describes, and returns the
|
||||||
|
// ban, and whether it made it. Unless keep is true, the ban is not made,
|
||||||
|
// only returned: it is the ban that would have been made.
|
||||||
func (l *Ledger) ban(
|
func (l *Ledger) ban(
|
||||||
netblock netip.Prefix,
|
netblock netip.Prefix,
|
||||||
now time.Time,
|
now time.Time,
|
||||||
cause, reason string,
|
grounds Grounds,
|
||||||
notes Notes,
|
|
||||||
keep bool,
|
keep bool,
|
||||||
) (Ban, bool) {
|
) (Ban, bool) {
|
||||||
l.mu.Lock()
|
l.mu.Lock()
|
||||||
@@ -656,6 +742,8 @@ func (l *Ledger) ban(
|
|||||||
// held are the netblock's bans, none of them active.
|
// held are the netblock's bans, none of them active.
|
||||||
var held []Ban
|
var held []Ban
|
||||||
|
|
||||||
|
notes := grounds.Notes
|
||||||
|
|
||||||
bans, found := l.netblocks.Get(netblock)
|
bans, found := l.netblocks.Get(netblock)
|
||||||
if found {
|
if found {
|
||||||
active := activeBan(*bans, now)
|
active := activeBan(*bans, now)
|
||||||
@@ -671,15 +759,19 @@ func (l *Ledger) ban(
|
|||||||
ban := Ban{
|
ban := Ban{
|
||||||
Netblock: netblock,
|
Netblock: netblock,
|
||||||
Start: now,
|
Start: now,
|
||||||
Cause: cause,
|
Cause: grounds.Cause,
|
||||||
Reason: reason,
|
|
||||||
Notes: notes,
|
Notes: notes,
|
||||||
}
|
}
|
||||||
|
|
||||||
if cause == CauseAttack {
|
switch grounds.Cause {
|
||||||
ban.Expires = l.attackExpiry(held, now)
|
case CauseLimit:
|
||||||
} else {
|
ban.Reason = limitReason(notes)
|
||||||
ban.Expires = l.limitExpiry(held, now)
|
ban.Expires = l.limitExpiry(held, now)
|
||||||
|
case CauseAttack:
|
||||||
|
ban.Reason = attackReason(notes)
|
||||||
|
ban.Expires = l.attackExpiry(held, now)
|
||||||
|
default:
|
||||||
|
panic("bans: a request's ban has the cause " + grounds.Cause)
|
||||||
}
|
}
|
||||||
|
|
||||||
if !keep {
|
if !keep {
|
||||||
@@ -687,7 +779,7 @@ func (l *Ledger) ban(
|
|||||||
}
|
}
|
||||||
|
|
||||||
l.add(ban)
|
l.add(ban)
|
||||||
l.made[cause]++
|
l.made[grounds.Cause]++
|
||||||
l.markChanged()
|
l.markChanged()
|
||||||
|
|
||||||
return ban, true
|
return ban, true
|
||||||
@@ -714,13 +806,16 @@ func earlierBans(held []Ban) EarlierBans {
|
|||||||
fromHere.Attack++
|
fromHere.Attack++
|
||||||
case CauseAdmin:
|
case CauseAdmin:
|
||||||
fromHere.Admin++
|
fromHere.Admin++
|
||||||
|
case CauseAbuseIPDB:
|
||||||
|
fromHere.AbuseIPDB++
|
||||||
}
|
}
|
||||||
|
|
||||||
notes := ban.Notes.EarlierBans
|
notes := ban.Notes.EarlierBans
|
||||||
count := EarlierBans{
|
count := EarlierBans{
|
||||||
Limit: notes.Limit + fromHere.Limit,
|
Limit: notes.Limit + fromHere.Limit,
|
||||||
Attack: notes.Attack + fromHere.Attack,
|
Attack: notes.Attack + fromHere.Attack,
|
||||||
Admin: notes.Admin + fromHere.Admin,
|
Admin: notes.Admin + fromHere.Admin,
|
||||||
|
AbuseIPDB: notes.AbuseIPDB + fromHere.AbuseIPDB,
|
||||||
}
|
}
|
||||||
|
|
||||||
// Of counts that name as many bans, the earlier ban's is taken: it
|
// Of counts that name as many bans, the earlier ban's is taken: it
|
||||||
@@ -735,7 +830,7 @@ func earlierBans(held []Ban) EarlierBans {
|
|||||||
|
|
||||||
// total returns how many bans e counts, of all causes.
|
// total returns how many bans e counts, of all causes.
|
||||||
func (e EarlierBans) total() int {
|
func (e EarlierBans) total() int {
|
||||||
return e.Limit + e.Attack + e.Admin
|
return e.Limit + e.Attack + e.Admin + e.AbuseIPDB
|
||||||
}
|
}
|
||||||
|
|
||||||
// markChanged has Changed receive a value, unless one is waiting already.
|
// markChanged has Changed receive a value, unless one is waiting already.
|
||||||
@@ -806,9 +901,9 @@ func (l *Ledger) covering(client netip.Addr) []*[]Ban {
|
|||||||
|
|
||||||
// add adds ban to its netblock's bans, after the last, and makes its
|
// add adds ban to its netblock's bans, after the last, and makes its
|
||||||
// netblock the most recently seen. With MaxBans held, it drops one first,
|
// netblock the most recently seen. With MaxBans held, it drops one first,
|
||||||
// unless ban's cause is CauseAdmin, which does not count toward MaxBans.
|
// unless ban does not count toward MaxBans.
|
||||||
func (l *Ledger) add(ban Ban) {
|
func (l *Ledger) add(ban Ban) {
|
||||||
counted := ban.Cause != CauseAdmin
|
counted := countsTowardMaxBans(ban)
|
||||||
if counted && l.held == l.params.MaxBans {
|
if counted && l.held == l.params.MaxBans {
|
||||||
l.dropOne()
|
l.dropOne()
|
||||||
}
|
}
|
||||||
@@ -889,7 +984,13 @@ func (l *Ledger) attackExpiry(held []Ban, now time.Time) time.Time {
|
|||||||
return now.Add(l.params.AttackBanDuration)
|
return now.Add(l.params.AttackBanDuration)
|
||||||
}
|
}
|
||||||
|
|
||||||
// dropOne drops the earliest ban whose cause is not CauseAdmin of the
|
// countsTowardMaxBans reports whether ban counts toward MaxBans: its cause
|
||||||
|
// is CauseLimit or CauseAttack.
|
||||||
|
func countsTowardMaxBans(ban Ban) bool {
|
||||||
|
return ban.Cause == CauseLimit || ban.Cause == CauseAttack
|
||||||
|
}
|
||||||
|
|
||||||
|
// dropOne drops the earliest ban that counts toward MaxBans of the
|
||||||
// netblock that has gone longest without a request, of those that hold
|
// netblock that has gone longest without a request, of those that hold
|
||||||
// such a ban, and the netblock with it if that was its only ban. It is
|
// such a ban, and the netblock with it if that was its only ban. It is
|
||||||
// called with at least one such ban held. It looks at each netblock once
|
// called with at least one such ban held. It looks at each netblock once
|
||||||
@@ -898,11 +999,9 @@ func (l *Ledger) dropOne() {
|
|||||||
for range l.netblocks.Len() {
|
for range l.netblocks.Len() {
|
||||||
netblock, bans, _ := l.netblocks.GetOldest()
|
netblock, bans, _ := l.netblocks.GetOldest()
|
||||||
|
|
||||||
i := slices.IndexFunc(*bans, func(ban Ban) bool {
|
i := slices.IndexFunc(*bans, countsTowardMaxBans)
|
||||||
return ban.Cause != CauseAdmin
|
|
||||||
})
|
|
||||||
if i < 0 {
|
if i < 0 {
|
||||||
// Its bans are all an admin's, and never dropped. Get makes
|
// None of its bans is dropped to make room. Get makes
|
||||||
// it the most recently seen, so that the next netblock is
|
// it the most recently seen, so that the next netblock is
|
||||||
// looked at; when it was seen matters only for dropping a
|
// looked at; when it was seen matters only for dropping a
|
||||||
// ban, and a ban added to it makes it the most recently seen
|
// ban, and a ban added to it makes it the most recently seen
|
||||||
|
|||||||
+121
-51
@@ -22,7 +22,11 @@ func TestRepeatsTripleUntilPermanent(t *testing.T) {
|
|||||||
// Each ban is followed by another as soon as it ends: 1, 3, 9, 27 and
|
// Each ban is followed by another as soon as it ends: 1, 3, 9, 27 and
|
||||||
// 81 hours.
|
// 81 hours.
|
||||||
for i, hours := range []int{1, 3, 9, 27, 81} {
|
for i, hours := range []int{1, 3, 9, 27, 81} {
|
||||||
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
|
ban, _ := ledger.Ban(
|
||||||
|
netblock,
|
||||||
|
now,
|
||||||
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
|
)
|
||||||
|
|
||||||
length := time.Duration(hours) * time.Hour
|
length := time.Duration(hours) * time.Hour
|
||||||
if !ban.Expires.Equal(now.Add(length)) ||
|
if !ban.Expires.Equal(now.Add(length)) ||
|
||||||
@@ -42,7 +46,7 @@ func TestRepeatsTripleUntilPermanent(t *testing.T) {
|
|||||||
|
|
||||||
// The sixth would last 243 hours, more than seven days: it is
|
// The sixth would last 243 hours, more than seven days: it is
|
||||||
// permanent, and never ends.
|
// permanent, and never ends.
|
||||||
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
|
ban, _ := ledger.Ban(netblock, now, bans.Grounds{Cause: bans.CauseLimit})
|
||||||
if !ban.Permanent() {
|
if !ban.Permanent() {
|
||||||
t.Fatalf("sixth ban ends at %s, want a permanent one", ban.Expires)
|
t.Fatalf("sixth ban ends at %s, want a permanent one", ban.Expires)
|
||||||
}
|
}
|
||||||
@@ -71,15 +75,15 @@ func TestRepeatWindowRunsOut(t *testing.T) {
|
|||||||
ledger := bans.New(defaultParams())
|
ledger := bans.New(defaultParams())
|
||||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
|
|
||||||
first, _ := ledger.BanForLimit(
|
first, _ := ledger.Ban(
|
||||||
netblock,
|
netblock,
|
||||||
midnight(),
|
midnight(),
|
||||||
bans.Notes{},
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
)
|
)
|
||||||
second, _ := ledger.BanForLimit(
|
second, _ := ledger.Ban(
|
||||||
netblock,
|
netblock,
|
||||||
first.Expires.Add(tc.gap),
|
first.Expires.Add(tc.gap),
|
||||||
bans.Notes{},
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
)
|
)
|
||||||
|
|
||||||
if second.Expires.Sub(second.Start) != tc.want ||
|
if second.Expires.Sub(second.Start) != tc.want ||
|
||||||
@@ -104,10 +108,10 @@ func TestFirstBanLongerThanLimitBanPermanentAboveIsPermanent(t *testing.T) {
|
|||||||
params.LimitBanDuration = params.LimitBanPermanentAbove + time.Hour
|
params.LimitBanDuration = params.LimitBanPermanentAbove + time.Hour
|
||||||
ledger := bans.New(params)
|
ledger := bans.New(params)
|
||||||
|
|
||||||
ban, _ := ledger.BanForLimit(
|
ban, _ := ledger.Ban(
|
||||||
netip.MustParsePrefix("203.0.113.9/32"),
|
netip.MustParsePrefix("203.0.113.9/32"),
|
||||||
midnight(),
|
midnight(),
|
||||||
bans.Notes{},
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
)
|
)
|
||||||
if !ban.Permanent() {
|
if !ban.Permanent() {
|
||||||
t.Errorf("first ban ends at %s, want a permanent one", ban.Expires)
|
t.Errorf("first ban ends at %s, want a permanent one", ban.Expires)
|
||||||
@@ -127,7 +131,11 @@ func TestLimitBanPermanentAboveSetFarOffDoesNotOverflow(t *testing.T) {
|
|||||||
now := midnight()
|
now := midnight()
|
||||||
|
|
||||||
for i := range 14 {
|
for i := range 14 {
|
||||||
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
|
ban, _ := ledger.Ban(
|
||||||
|
netblock,
|
||||||
|
now,
|
||||||
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
|
)
|
||||||
if !ban.Expires.After(ban.Start) {
|
if !ban.Expires.After(ban.Start) {
|
||||||
t.Fatalf(
|
t.Fatalf(
|
||||||
"ban %d starts at %s and ends at %s",
|
"ban %d starts at %s and ends at %s",
|
||||||
@@ -140,7 +148,7 @@ func TestLimitBanPermanentAboveSetFarOffDoesNotOverflow(t *testing.T) {
|
|||||||
now = ban.Expires
|
now = ban.Expires
|
||||||
}
|
}
|
||||||
|
|
||||||
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
|
ban, _ := ledger.Ban(netblock, now, bans.Grounds{Cause: bans.CauseLimit})
|
||||||
if !ban.Permanent() {
|
if !ban.Permanent() {
|
||||||
t.Errorf("15th ban ends at %s, want a permanent one", ban.Expires)
|
t.Errorf("15th ban ends at %s, want a permanent one", ban.Expires)
|
||||||
}
|
}
|
||||||
@@ -152,15 +160,19 @@ func TestBrokenLimitDuringABanMakesNoOther(t *testing.T) {
|
|||||||
ledger := bans.New(defaultParams())
|
ledger := bans.New(defaultParams())
|
||||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
|
|
||||||
first, made := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
|
first, made := ledger.Ban(
|
||||||
|
netblock,
|
||||||
|
midnight(),
|
||||||
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
|
)
|
||||||
if !made {
|
if !made {
|
||||||
t.Error("the first ban was not made")
|
t.Error("the first ban was not made")
|
||||||
}
|
}
|
||||||
|
|
||||||
again, made := ledger.BanForLimit(
|
again, made := ledger.Ban(
|
||||||
netblock,
|
netblock,
|
||||||
midnight().Add(time.Minute),
|
midnight().Add(time.Minute),
|
||||||
bans.Notes{},
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
)
|
)
|
||||||
|
|
||||||
if made || !reflect.DeepEqual(again, first) ||
|
if made || !reflect.DeepEqual(again, first) ||
|
||||||
@@ -175,10 +187,10 @@ func TestBrokenLimitDuringABanMakesNoOther(t *testing.T) {
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
again, made = ledger.BanForAttack(
|
again, made = ledger.Ban(
|
||||||
netblock,
|
netblock,
|
||||||
midnight().Add(time.Minute),
|
midnight().Add(time.Minute),
|
||||||
bans.Notes{},
|
bans.Grounds{Cause: bans.CauseAttack},
|
||||||
)
|
)
|
||||||
if made || !reflect.DeepEqual(again, first) {
|
if made || !reflect.DeepEqual(again, first) {
|
||||||
t.Errorf(
|
t.Errorf(
|
||||||
@@ -195,10 +207,10 @@ func TestCheckRefusesWhileTheBanLastsAndCountsTheRefusals(t *testing.T) {
|
|||||||
|
|
||||||
ledger := bans.New(defaultParams())
|
ledger := bans.New(defaultParams())
|
||||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
ban, _ := ledger.BanForLimit(
|
ban, _ := ledger.Ban(
|
||||||
netblock,
|
netblock,
|
||||||
midnight(),
|
midnight(),
|
||||||
bans.Notes{Requests: 5},
|
bans.Grounds{Cause: bans.CauseLimit, Notes: bans.Notes{Requests: 5}},
|
||||||
)
|
)
|
||||||
|
|
||||||
for range 3 {
|
for range 3 {
|
||||||
@@ -237,10 +249,10 @@ func TestFindCountsNothing(t *testing.T) {
|
|||||||
|
|
||||||
ledger := bans.New(defaultParams())
|
ledger := bans.New(defaultParams())
|
||||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
ban, _ := ledger.BanForLimit(
|
ban, _ := ledger.Ban(
|
||||||
netblock,
|
netblock,
|
||||||
midnight(),
|
midnight(),
|
||||||
bans.Notes{Requests: 5},
|
bans.Grounds{Cause: bans.CauseLimit, Notes: bans.Notes{Requests: 5}},
|
||||||
)
|
)
|
||||||
|
|
||||||
got, banned, _ := ledger.Find(
|
got, banned, _ := ledger.Find(
|
||||||
@@ -287,24 +299,24 @@ func TestMaxBansDropsTheEarliestBanOfTheNetblockSeenLongestAgo(
|
|||||||
d := netip.MustParsePrefix("2001:db8::/64")
|
d := netip.MustParsePrefix("2001:db8::/64")
|
||||||
now := midnight()
|
now := midnight()
|
||||||
|
|
||||||
first, _ := ledger.BanForLimit(a, now, bans.Notes{})
|
first, _ := ledger.Ban(a, now, bans.Grounds{Cause: bans.CauseLimit})
|
||||||
ledger.BanForLimit(b, now, bans.Notes{})
|
ledger.Ban(b, now, bans.Grounds{Cause: bans.CauseLimit})
|
||||||
ledger.BanForLimit(c, now, bans.Notes{})
|
ledger.Ban(c, now, bans.Grounds{Cause: bans.CauseLimit})
|
||||||
|
|
||||||
// A request from a makes b the netblock seen longest ago, and its ban
|
// A request from a makes b the netblock seen longest ago, and its ban
|
||||||
// goes to make room for d's.
|
// goes to make room for d's.
|
||||||
ledger.Check(a.Addr(), now)
|
ledger.Check(a.Addr(), now)
|
||||||
ledger.BanForLimit(d, now, bans.Notes{})
|
ledger.Ban(d, now, bans.Grounds{Cause: bans.CauseLimit})
|
||||||
wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 0, c: 1, d: 1})
|
wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 0, c: 1, d: 1})
|
||||||
|
|
||||||
// a is banned again once its ban has ended; c, seen longest ago, goes.
|
// a is banned again once its ban has ended; c, seen longest ago, goes.
|
||||||
ledger.BanForLimit(a, first.Expires, bans.Notes{})
|
ledger.Ban(a, first.Expires, bans.Grounds{Cause: bans.CauseLimit})
|
||||||
wantBans(t, ledger, map[netip.Prefix]int{a: 2, c: 0, d: 1})
|
wantBans(t, ledger, map[netip.Prefix]int{a: 2, c: 0, d: 1})
|
||||||
|
|
||||||
// With d seen since, a is seen longest ago, and its earlier ban goes
|
// With d seen since, a is seen longest ago, and its earlier ban goes
|
||||||
// first.
|
// first.
|
||||||
ledger.Check(d.Addr(), first.Expires)
|
ledger.Check(d.Addr(), first.Expires)
|
||||||
ledger.BanForLimit(b, first.Expires, bans.Notes{})
|
ledger.Ban(b, first.Expires, bans.Grounds{Cause: bans.CauseLimit})
|
||||||
wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 1, d: 1})
|
wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 1, d: 1})
|
||||||
|
|
||||||
if !ledger.Bans(a)[0].Start.Equal(first.Expires) {
|
if !ledger.Bans(a)[0].Start.Equal(first.Expires) {
|
||||||
@@ -325,8 +337,16 @@ func TestFullLedgerDropsTheEarlierBanOfTheNetblockBannedAgain(t *testing.T) {
|
|||||||
ledger := bans.New(params)
|
ledger := bans.New(params)
|
||||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
|
|
||||||
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
|
first, _ := ledger.Ban(
|
||||||
second, _ := ledger.BanForLimit(netblock, first.Expires, bans.Notes{})
|
netblock,
|
||||||
|
midnight(),
|
||||||
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
|
)
|
||||||
|
second, _ := ledger.Ban(
|
||||||
|
netblock,
|
||||||
|
first.Expires,
|
||||||
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
|
)
|
||||||
|
|
||||||
held := ledger.Bans(netblock)
|
held := ledger.Bans(netblock)
|
||||||
if len(held) != 1 || !reflect.DeepEqual(held[0], second) ||
|
if len(held) != 1 || !reflect.DeepEqual(held[0], second) ||
|
||||||
@@ -343,7 +363,11 @@ func TestRequestDuringAnAttackBanMakesItPermanent(t *testing.T) {
|
|||||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
notes := bans.Notes{RuleID: "env-file", Target: "path"}
|
notes := bans.Notes{RuleID: "env-file", Target: "path"}
|
||||||
|
|
||||||
ban, _ := ledger.BanForAttack(netblock, midnight(), notes)
|
ban, _ := ledger.Ban(
|
||||||
|
netblock,
|
||||||
|
midnight(),
|
||||||
|
bans.Grounds{Cause: bans.CauseAttack, Notes: notes},
|
||||||
|
)
|
||||||
if !ban.Expires.Equal(midnight().Add(7*day)) ||
|
if !ban.Expires.Equal(midnight().Add(7*day)) ||
|
||||||
ban.Cause != bans.CauseAttack ||
|
ban.Cause != bans.CauseAttack ||
|
||||||
ban.Notes.RuleID != "env-file" ||
|
ban.Notes.RuleID != "env-file" ||
|
||||||
@@ -418,8 +442,16 @@ func TestAttackAfterAnAttackBanHasEndedBansPermanently(t *testing.T) {
|
|||||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
|
|
||||||
// A ban for a broken limit before does not count.
|
// A ban for a broken limit before does not count.
|
||||||
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
|
first, _ := ledger.Ban(
|
||||||
second, _ := ledger.BanForAttack(netblock, first.Expires, bans.Notes{})
|
netblock,
|
||||||
|
midnight(),
|
||||||
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
|
)
|
||||||
|
second, _ := ledger.Ban(
|
||||||
|
netblock,
|
||||||
|
first.Expires,
|
||||||
|
bans.Grounds{Cause: bans.CauseAttack},
|
||||||
|
)
|
||||||
|
|
||||||
if second.Expires.Sub(second.Start) != 7*day {
|
if second.Expires.Sub(second.Start) != 7*day {
|
||||||
t.Fatalf("the first ban for an attack lasts %s, want 7 days",
|
t.Fatalf("the first ban for an attack lasts %s, want 7 days",
|
||||||
@@ -435,10 +467,10 @@ func TestAttackAfterAnAttackBanHasEndedBansPermanently(t *testing.T) {
|
|||||||
|
|
||||||
// Its notes show the earlier ban for an attack that makes it permanent,
|
// Its notes show the earlier ban for an attack that makes it permanent,
|
||||||
// beside the one for a limit.
|
// beside the one for a limit.
|
||||||
third, _ := ledger.BanForAttack(
|
third, _ := ledger.Ban(
|
||||||
netblock,
|
netblock,
|
||||||
second.Expires.Add(30*day),
|
second.Expires.Add(30*day),
|
||||||
bans.Notes{},
|
bans.Grounds{Cause: bans.CauseAttack},
|
||||||
)
|
)
|
||||||
if !third.Permanent() ||
|
if !third.Permanent() ||
|
||||||
third.Notes.EarlierBans != (bans.EarlierBans{Limit: 1, Attack: 1}) {
|
third.Notes.EarlierBans != (bans.EarlierBans{Limit: 1, Attack: 1}) {
|
||||||
@@ -452,14 +484,18 @@ func TestWouldBanGivesTheBanWithoutMakingIt(t *testing.T) {
|
|||||||
|
|
||||||
ledger := bans.New(defaultParams())
|
ledger := bans.New(defaultParams())
|
||||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
|
first, _ := ledger.Ban(
|
||||||
|
netblock,
|
||||||
|
midnight(),
|
||||||
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
|
)
|
||||||
wantChanged(t, ledger, true)
|
wantChanged(t, ledger, true)
|
||||||
|
|
||||||
// While the first ban lasts, none would be made.
|
// While the first ban lasts, none would be made.
|
||||||
during, would := ledger.WouldBanForAttack(
|
during, would := ledger.WouldBan(
|
||||||
netblock,
|
netblock,
|
||||||
midnight(),
|
midnight(),
|
||||||
bans.Notes{},
|
bans.Grounds{Cause: bans.CauseAttack},
|
||||||
)
|
)
|
||||||
if would || !reflect.DeepEqual(during, first) {
|
if would || !reflect.DeepEqual(during, first) {
|
||||||
t.Errorf(
|
t.Errorf(
|
||||||
@@ -472,13 +508,19 @@ func TestWouldBanGivesTheBanWithoutMakingIt(t *testing.T) {
|
|||||||
|
|
||||||
// As it ends, a clear sign of attack would ban for seven days, and a
|
// As it ends, a clear sign of attack would ban for seven days, and a
|
||||||
// limit broken again for three hours, but neither is made.
|
// limit broken again for three hours, but neither is made.
|
||||||
limitNotes := bans.Notes{Kind: "requests", Limit: 1, Window: "minute"}
|
limitGrounds := bans.Grounds{
|
||||||
attack, wouldAttack := ledger.WouldBanForAttack(netblock, first.Expires,
|
Cause: bans.CauseLimit,
|
||||||
bans.Notes{RuleID: "git-dir"})
|
Notes: bans.Notes{Kind: "requests", Limit: 1, Window: "minute"},
|
||||||
limit, wouldLimit := ledger.WouldBanForLimit(
|
}
|
||||||
|
attack, wouldAttack := ledger.WouldBan(netblock, first.Expires,
|
||||||
|
bans.Grounds{
|
||||||
|
Cause: bans.CauseAttack,
|
||||||
|
Notes: bans.Notes{RuleID: "git-dir"},
|
||||||
|
})
|
||||||
|
limit, wouldLimit := ledger.WouldBan(
|
||||||
netblock,
|
netblock,
|
||||||
first.Expires,
|
first.Expires,
|
||||||
limitNotes,
|
limitGrounds,
|
||||||
)
|
)
|
||||||
|
|
||||||
if !wouldAttack || !attack.Expires.Equal(first.Expires.Add(7*day)) ||
|
if !wouldAttack || !attack.Expires.Equal(first.Expires.Add(7*day)) ||
|
||||||
@@ -505,7 +547,7 @@ func TestWouldBanGivesTheBanWithoutMakingIt(t *testing.T) {
|
|||||||
wantChanged(t, ledger, false)
|
wantChanged(t, ledger, false)
|
||||||
|
|
||||||
// The ban made is the one that would have been.
|
// The ban made is the one that would have been.
|
||||||
made, _ := ledger.BanForLimit(netblock, first.Expires, limitNotes)
|
made, _ := ledger.Ban(netblock, first.Expires, limitGrounds)
|
||||||
if !reflect.DeepEqual(made, limit) {
|
if !reflect.DeepEqual(made, limit) {
|
||||||
t.Errorf("the ban made is %+v, want %+v", made, limit)
|
t.Errorf("the ban made is %+v, want %+v", made, limit)
|
||||||
}
|
}
|
||||||
@@ -526,7 +568,11 @@ func TestWouldBePermanentAnswersAsTheBanWouldBeMade(t *testing.T) {
|
|||||||
t.Fatalf("ban %d for a limit would be permanent", i+1)
|
t.Fatalf("ban %d for a limit would be permanent", i+1)
|
||||||
}
|
}
|
||||||
|
|
||||||
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
|
ban, _ := ledger.Ban(
|
||||||
|
netblock,
|
||||||
|
now,
|
||||||
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
|
)
|
||||||
now = ban.Expires
|
now = ban.Expires
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -539,7 +585,11 @@ func TestWouldBePermanentAnswersAsTheBanWouldBeMade(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Once a first ban for an attack has ended, the next would be permanent.
|
// Once a first ban for an attack has ended, the next would be permanent.
|
||||||
attack, _ := ledger.BanForAttack(netblock, now, bans.Notes{})
|
attack, _ := ledger.Ban(
|
||||||
|
netblock,
|
||||||
|
now,
|
||||||
|
bans.Grounds{Cause: bans.CauseAttack},
|
||||||
|
)
|
||||||
if !ledger.WouldBePermanent(netblock, attack.Expires, bans.CauseAttack) {
|
if !ledger.WouldBePermanent(netblock, attack.Expires, bans.CauseAttack) {
|
||||||
t.Error("a second ban for an attack would not be permanent")
|
t.Error("a second ban for an attack would not be permanent")
|
||||||
}
|
}
|
||||||
@@ -553,8 +603,16 @@ func TestAttackBanDoesNotLengthenTheNextBanForALimit(t *testing.T) {
|
|||||||
|
|
||||||
// Three times the seven days would be permanent; a limit broken as the
|
// Three times the seven days would be permanent; a limit broken as the
|
||||||
// ban for an attack ends bans for an hour, as a first broken limit does.
|
// ban for an attack ends bans for an hour, as a first broken limit does.
|
||||||
attack, _ := ledger.BanForAttack(netblock, midnight(), bans.Notes{})
|
attack, _ := ledger.Ban(
|
||||||
limit, _ := ledger.BanForLimit(netblock, attack.Expires, bans.Notes{})
|
netblock,
|
||||||
|
midnight(),
|
||||||
|
bans.Grounds{Cause: bans.CauseAttack},
|
||||||
|
)
|
||||||
|
limit, _ := ledger.Ban(
|
||||||
|
netblock,
|
||||||
|
attack.Expires,
|
||||||
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
|
)
|
||||||
|
|
||||||
if limit.Expires.Sub(limit.Start) != time.Hour ||
|
if limit.Expires.Sub(limit.Start) != time.Hour ||
|
||||||
limit.Cause != bans.CauseLimit {
|
limit.Cause != bans.CauseLimit {
|
||||||
@@ -578,10 +636,13 @@ func TestRequestTextsAreCutTo256Bytes(t *testing.T) {
|
|||||||
Time: midnight(), Method: long, Host: long, Path: long, Status: 403, UserAgent: long,
|
Time: midnight(), Method: long, Host: long, Path: long, Status: 403, UserAgent: long,
|
||||||
}
|
}
|
||||||
|
|
||||||
ban, _ := ledger.BanForLimit(
|
ban, _ := ledger.Ban(
|
||||||
netblock,
|
netblock,
|
||||||
midnight(),
|
midnight(),
|
||||||
bans.Notes{Request: request},
|
bans.Grounds{
|
||||||
|
Cause: bans.CauseLimit,
|
||||||
|
Notes: bans.Notes{Request: request},
|
||||||
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
cut := long[:256]
|
cut := long[:256]
|
||||||
@@ -607,11 +668,20 @@ func TestLookupFillsTheNotesOfTheNetblocksBansWithoutOne(t *testing.T) {
|
|||||||
|
|
||||||
// A ban made with the client's lookup, one made before it came, after
|
// A ban made with the client's lookup, one made before it came, after
|
||||||
// the first ended, and one on another netblock.
|
// the first ended, and one on another netblock.
|
||||||
ledger.BanForLimit(netblock, midnight(), bans.Notes{
|
ledger.Ban(netblock, midnight(), bans.Grounds{
|
||||||
ASN: "AS64497", ASName: "Other Net", Country: "FR",
|
Cause: bans.CauseLimit,
|
||||||
|
Notes: bans.Notes{
|
||||||
|
ASN: "AS64497",
|
||||||
|
ASName: "Other Net",
|
||||||
|
Country: "FR",
|
||||||
|
},
|
||||||
})
|
})
|
||||||
ledger.BanForLimit(netblock, midnight().Add(time.Hour), bans.Notes{})
|
ledger.Ban(
|
||||||
ledger.BanForLimit(other, midnight(), bans.Notes{})
|
netblock,
|
||||||
|
midnight().Add(time.Hour),
|
||||||
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
|
)
|
||||||
|
ledger.Ban(other, midnight(), bans.Grounds{Cause: bans.CauseLimit})
|
||||||
|
|
||||||
ledger.AddLookup(netblock, "AS64496", "Example Net", "DE")
|
ledger.AddLookup(netblock, "AS64496", "Example Net", "DE")
|
||||||
|
|
||||||
|
|||||||
@@ -19,25 +19,29 @@ func TestChangedAfterABanIsMade(t *testing.T) {
|
|||||||
|
|
||||||
wantChanged(t, ledger, false)
|
wantChanged(t, ledger, false)
|
||||||
|
|
||||||
ledger.BanForLimit(netblock, midnight(), bans.Notes{})
|
ledger.Ban(netblock, midnight(), bans.Grounds{Cause: bans.CauseLimit})
|
||||||
wantChanged(t, ledger, true)
|
wantChanged(t, ledger, true)
|
||||||
|
|
||||||
// A limit broken during the ban makes no other, and a refusal changes
|
// A limit broken during the ban makes no other, and a refusal changes
|
||||||
// only the counts in the notes, which wait for the interval's write.
|
// only the counts in the notes, which wait for the interval's write.
|
||||||
ledger.BanForLimit(netblock, midnight().Add(time.Minute), bans.Notes{})
|
ledger.Ban(
|
||||||
|
netblock,
|
||||||
|
midnight().Add(time.Minute),
|
||||||
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
|
)
|
||||||
ledger.Check(netblock.Addr(), midnight().Add(time.Minute))
|
ledger.Check(netblock.Addr(), midnight().Add(time.Minute))
|
||||||
wantChanged(t, ledger, false)
|
wantChanged(t, ledger, false)
|
||||||
|
|
||||||
// Two bans before the value is read leave one.
|
// Two bans before the value is read leave one.
|
||||||
ledger.BanForLimit(
|
ledger.Ban(
|
||||||
netip.MustParsePrefix("203.0.113.10/32"),
|
netip.MustParsePrefix("203.0.113.10/32"),
|
||||||
midnight(),
|
midnight(),
|
||||||
bans.Notes{},
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
)
|
)
|
||||||
ledger.BanForLimit(
|
ledger.Ban(
|
||||||
netip.MustParsePrefix("203.0.113.11/32"),
|
netip.MustParsePrefix("203.0.113.11/32"),
|
||||||
midnight(),
|
midnight(),
|
||||||
bans.Notes{},
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
)
|
)
|
||||||
wantChanged(t, ledger, true)
|
wantChanged(t, ledger, true)
|
||||||
wantChanged(t, ledger, false)
|
wantChanged(t, ledger, false)
|
||||||
@@ -51,10 +55,14 @@ func TestSnapshotListsEveryBanByNetblock(t *testing.T) {
|
|||||||
high := netip.MustParsePrefix("203.0.113.10/32")
|
high := netip.MustParsePrefix("203.0.113.10/32")
|
||||||
low := netip.MustParsePrefix("203.0.113.9/32")
|
low := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
|
|
||||||
first, _ := ledger.BanForLimit(v6, midnight(), bans.Notes{})
|
first, _ := ledger.Ban(
|
||||||
ledger.BanForLimit(high, midnight(), bans.Notes{})
|
v6,
|
||||||
ledger.BanForLimit(low, midnight(), bans.Notes{})
|
midnight(),
|
||||||
ledger.BanForLimit(v6, first.Expires, bans.Notes{})
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
|
)
|
||||||
|
ledger.Ban(high, midnight(), bans.Grounds{Cause: bans.CauseLimit})
|
||||||
|
ledger.Ban(low, midnight(), bans.Grounds{Cause: bans.CauseLimit})
|
||||||
|
ledger.Ban(v6, first.Expires, bans.Grounds{Cause: bans.CauseLimit})
|
||||||
|
|
||||||
snapshot := ledger.Snapshot()
|
snapshot := ledger.Snapshot()
|
||||||
|
|
||||||
@@ -80,7 +88,10 @@ func TestLoadedBansCarryOn(t *testing.T) {
|
|||||||
|
|
||||||
before := bans.New(defaultParams())
|
before := bans.New(defaultParams())
|
||||||
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
netblock := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
ban, _ := before.BanForLimit(netblock, midnight(), bans.Notes{Limit: 1})
|
ban, _ := before.Ban(netblock, midnight(), bans.Grounds{
|
||||||
|
Cause: bans.CauseLimit,
|
||||||
|
Notes: bans.Notes{Limit: 1},
|
||||||
|
})
|
||||||
|
|
||||||
// Loaded into a new ledger, as across a restart, the ban still refuses
|
// Loaded into a new ledger, as across a restart, the ban still refuses
|
||||||
// while it lasts, and once it has ended a broken limit bans for three
|
// while it lasts, and once it has ended a broken limit bans for three
|
||||||
@@ -96,7 +107,11 @@ func TestLoadedBansCarryOn(t *testing.T) {
|
|||||||
t.Error("the loaded ban does not refuse")
|
t.Error("the loaded ban does not refuse")
|
||||||
}
|
}
|
||||||
|
|
||||||
again, _ := after.BanForLimit(netblock, ban.Expires, bans.Notes{})
|
again, _ := after.Ban(
|
||||||
|
netblock,
|
||||||
|
ban.Expires,
|
||||||
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
|
)
|
||||||
if again.Expires.Sub(again.Start) != 3*time.Hour ||
|
if again.Expires.Sub(again.Start) != 3*time.Hour ||
|
||||||
again.Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) {
|
again.Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) {
|
||||||
t.Errorf(
|
t.Errorf(
|
||||||
@@ -124,10 +139,10 @@ func TestLoadedBanRefusesEveryClientInItsNetblock(t *testing.T) {
|
|||||||
Start: midnight(),
|
Start: midnight(),
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
ledger.BanForLimit(
|
ledger.Ban(
|
||||||
netip.MustParsePrefix("198.51.100.7/32"),
|
netip.MustParsePrefix("198.51.100.7/32"),
|
||||||
midnight(),
|
midnight(),
|
||||||
bans.Notes{},
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
)
|
)
|
||||||
|
|
||||||
for client, want := range map[string]bool{
|
for client, want := range map[string]bool{
|
||||||
@@ -200,7 +215,7 @@ func TestPermanentBanStartedBeforeAnEndedOneRefuses(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// A limit broken now makes no shorter ban over the permanent one.
|
// A limit broken now makes no shorter ban over the permanent one.
|
||||||
ban, _ = ledger.BanForLimit(netblock, now, bans.Notes{})
|
ban, _ = ledger.Ban(netblock, now, bans.Grounds{Cause: bans.CauseLimit})
|
||||||
if !ban.Permanent() || len(ledger.Bans(netblock)) != 2 {
|
if !ban.Permanent() || len(ledger.Bans(netblock)) != 2 {
|
||||||
t.Errorf(
|
t.Errorf(
|
||||||
"a broken limit returned %+v and left the netblock %d bans, "+
|
"a broken limit returned %+v and left the netblock %d bans, "+
|
||||||
@@ -236,10 +251,10 @@ func TestNextBanWorkedOutFromTheBanThatEndedLast(t *testing.T) {
|
|||||||
// Once both have ended, a limit broken within the repeat window bans
|
// Once both have ended, a limit broken within the repeat window bans
|
||||||
// for three times the 9 hours, and the notes count the two bans
|
// for three times the 9 hours, and the notes count the two bans
|
||||||
// before the 9-hour one and it, for a limit, and the admin's.
|
// before the 9-hour one and it, for a limit, and the admin's.
|
||||||
ban, _ := ledger.BanForLimit(
|
ban, _ := ledger.Ban(
|
||||||
netblock,
|
netblock,
|
||||||
nineHours.Expires.Add(time.Hour),
|
nineHours.Expires.Add(time.Hour),
|
||||||
bans.Notes{},
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
)
|
)
|
||||||
if ban.Expires.Sub(ban.Start) != 27*time.Hour ||
|
if ban.Expires.Sub(ban.Start) != 27*time.Hour ||
|
||||||
ban.Notes.EarlierBans != (bans.EarlierBans{Limit: 3, Admin: 1}) {
|
ban.Notes.EarlierBans != (bans.EarlierBans{Limit: 3, Admin: 1}) {
|
||||||
@@ -313,15 +328,15 @@ func TestLoadReplacesTheBansHeld(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// The ledger holds one ban, so it makes two more without dropping any.
|
// The ledger holds one ban, so it makes two more without dropping any.
|
||||||
first, _ := ledger.BanForLimit(
|
first, _ := ledger.Ban(
|
||||||
netip.MustParsePrefix("198.51.100.7/32"),
|
netip.MustParsePrefix("198.51.100.7/32"),
|
||||||
midnight(),
|
midnight(),
|
||||||
bans.Notes{},
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
)
|
)
|
||||||
second, _ := ledger.BanForLimit(
|
second, _ := ledger.Ban(
|
||||||
netip.MustParsePrefix("198.51.100.8/32"),
|
netip.MustParsePrefix("198.51.100.8/32"),
|
||||||
midnight(),
|
midnight(),
|
||||||
bans.Notes{},
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
)
|
)
|
||||||
|
|
||||||
want := []bans.Ban{first, second, kept}
|
want := []bans.Ban{first, second, kept}
|
||||||
|
|||||||
@@ -19,6 +19,8 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/spf13/viper"
|
"github.com/spf13/viper"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
)
|
)
|
||||||
|
|
||||||
// off is the value that switches a timeout, a size limit or a rate limit
|
// off is the value that switches a timeout, a size limit or a rate limit
|
||||||
@@ -40,9 +42,7 @@ var (
|
|||||||
errNotPositive = errors.New("must be more than zero, or off")
|
errNotPositive = errors.New("must be more than zero, or off")
|
||||||
errNotMilliseconds = errors.New(
|
errNotMilliseconds = errors.New(
|
||||||
"is not a whole number of milliseconds above zero, such as 2000")
|
"is not a whole number of milliseconds above zero, such as 2000")
|
||||||
errEmptyItem = errors.New("has an empty item in its list")
|
errEmptyItem = errors.New("has an empty item in its list")
|
||||||
errNotNetblock = errors.New(
|
|
||||||
"is not a netblock such as 10.0.0.0/8, or an address")
|
|
||||||
errNotListenAddr = errors.New(
|
errNotListenAddr = errors.New(
|
||||||
"is not an address to listen on, such as :8080")
|
"is not an address to listen on, such as :8080")
|
||||||
errNotCommand = errors.New("is not rproxy or waf")
|
errNotCommand = errors.New("is not rproxy or waf")
|
||||||
@@ -253,7 +253,7 @@ func parseNetblocks(value string) ([]netip.Prefix, error) {
|
|||||||
netblocks := make([]netip.Prefix, 0, len(items))
|
netblocks := make([]netip.Prefix, 0, len(items))
|
||||||
|
|
||||||
for _, item := range items {
|
for _, item := range items {
|
||||||
netblock, err := ParseNetblock(item)
|
netblock, err := reputation.ParseNetblock(item)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
@@ -264,27 +264,6 @@ func parseNetblocks(value string) ([]netip.Prefix, error) {
|
|||||||
return netblocks, nil
|
return netblocks, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// ParseNetblock reads a netblock in CIDR form, such as 10.0.0.0/8. A bare
|
|
||||||
// address is a netblock of that address alone, a /32 or a /128. A
|
|
||||||
// blocklist's lines are read with it too.
|
|
||||||
func ParseNetblock(value string) (netip.Prefix, error) {
|
|
||||||
if strings.Contains(value, "/") {
|
|
||||||
netblock, err := netip.ParsePrefix(value)
|
|
||||||
if err != nil {
|
|
||||||
return netip.Prefix{}, fmt.Errorf("%q %w", value, errNotNetblock)
|
|
||||||
}
|
|
||||||
|
|
||||||
return netblock.Masked(), nil
|
|
||||||
}
|
|
||||||
|
|
||||||
addr, err := netip.ParseAddr(value)
|
|
||||||
if err != nil || addr.Zone() != "" {
|
|
||||||
return netip.Prefix{}, fmt.Errorf("%q %w", value, errNotNetblock)
|
|
||||||
}
|
|
||||||
|
|
||||||
return netip.PrefixFrom(addr, addr.BitLen()), nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// parseListenAddr checks an address to listen on: an optional host and a
|
// parseListenAddr checks an address to listen on: an optional host and a
|
||||||
// port number.
|
// port number.
|
||||||
func parseListenAddr(value string) (string, error) {
|
func parseListenAddr(value string) (string, error) {
|
||||||
|
|||||||
+239
-158
@@ -20,11 +20,26 @@ import (
|
|||||||
"unicode/utf8"
|
"unicode/utf8"
|
||||||
|
|
||||||
"github.com/spf13/viper"
|
"github.com/spf13/viper"
|
||||||
|
"golang.org/x/net/http/httpguts"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/remotelog"
|
"sneak.berlin/go/smallwebwaf/internal/remotelog"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The names of the settings that waf writes into log lines and ban notes,
|
||||||
|
// as the setting that refused a request or lowered a client's limits.
|
||||||
|
const (
|
||||||
|
ExclusivelyAllowedCountriesSetting = "SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES"
|
||||||
|
TagLimitPercentSetting = "SWW_WAF_TAG_LIMIT_PERCENT"
|
||||||
|
ASNBytesLimitPercentSetting = "SWW_WAF_ASN_BYTES_LIMIT_PERCENT"
|
||||||
|
CountryBytesLimitPercentSetting = "SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT"
|
||||||
|
UnknownCountryLimitPercentSetting = "SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT"
|
||||||
|
ASNLimitPercentURLSetting = "SWW_WAF_ASN_LIMIT_PERCENT_URL"
|
||||||
|
BlocklistActionSetting = "SWW_WAF_BLOCKLIST_ACTION"
|
||||||
|
DNSBLAbuseIPDBActionSetting = "SWW_WAF_DNSBL_ABUSEIPDB_ACTION"
|
||||||
)
|
)
|
||||||
|
|
||||||
// WAF is the settings of `sww waf`. A time or size limit of zero is off.
|
// WAF is the settings of `sww waf`. A time or size limit of zero is off.
|
||||||
@@ -75,7 +90,7 @@ type WAF struct {
|
|||||||
BytesLimitPerMinute int64
|
BytesLimitPerMinute int64
|
||||||
BytesLimitPerHour int64
|
BytesLimitPerHour int64
|
||||||
BytesLimitPerDay int64
|
BytesLimitPerDay int64
|
||||||
BytesCounted string
|
BytesCounted BytesCounted
|
||||||
// LookupSource is where each client's AS number and country are looked
|
// LookupSource is where each client's AS number and country are looked
|
||||||
// up (SWW_WAF_LOOKUP_SOURCE): geojs, file, or off for nowhere.
|
// up (SWW_WAF_LOOKUP_SOURCE): geojs, file, or off for nowhere.
|
||||||
// LookupDBPath is the lookup database, the IPinfo Lite file looked up in
|
// LookupDBPath is the lookup database, the IPinfo Lite file looked up in
|
||||||
@@ -84,8 +99,9 @@ type WAF struct {
|
|||||||
// client's first answer from GeoJS while a setting needs it
|
// client's first answer from GeoJS while a setting needs it
|
||||||
// (SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS), which cannot be off.
|
// (SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS), which cannot be off.
|
||||||
// AddLookupHeaders is true when the app is passed the client's AS number
|
// AddLookupHeaders is true when the app is passed the client's AS number
|
||||||
// and country in headers (SWW_WAF_ADD_LOOKUP_HEADERS).
|
// and country in headers (SWW_WAF_ADD_LOOKUP_HEADERS), by default unless
|
||||||
LookupSource string
|
// LookupSource is off.
|
||||||
|
LookupSource LookupSource
|
||||||
LookupDBPath string
|
LookupDBPath string
|
||||||
ASNAndCountryLookupTimeout time.Duration
|
ASNAndCountryLookupTimeout time.Duration
|
||||||
AddLookupHeaders bool
|
AddLookupHeaders bool
|
||||||
@@ -99,7 +115,7 @@ type WAF struct {
|
|||||||
// with a tag that percentage of every rate limit and byte limit, from 1
|
// with a tag that percentage of every rate limit and byte limit, from 1
|
||||||
// to 99 (SWW_WAF_TAG_LIMIT_PERCENT). All hold tags as requests carry them: in
|
// to 99 (SWW_WAF_TAG_LIMIT_PERCENT). All hold tags as requests carry them: in
|
||||||
// lower case, a country code in capitals.
|
// lower case, a country code in capitals.
|
||||||
TagLists []TagList
|
TagLists []reputation.TagList
|
||||||
TagDeny []string
|
TagDeny []string
|
||||||
TagLimitPercent map[string]int64
|
TagLimitPercent map[string]int64
|
||||||
// RouteWatchURL is where the routewatch instance is that adds the asn=
|
// RouteWatchURL is where the routewatch instance is that adds the asn=
|
||||||
@@ -130,7 +146,7 @@ type WAF struct {
|
|||||||
// BlocklistLimitPercent is the percentage of every limit it gets.
|
// BlocklistLimitPercent is the percentage of every limit it gets.
|
||||||
BlocklistURLs []string
|
BlocklistURLs []string
|
||||||
BlocklistRefresh time.Duration
|
BlocklistRefresh time.Duration
|
||||||
BlocklistAction string
|
BlocklistAction ReputationAction
|
||||||
BlocklistLimitPercent int64
|
BlocklistLimitPercent int64
|
||||||
// DNSBLZones are the DNSBL zones clients are asked about
|
// DNSBLZones are the DNSBL zones clients are asked about
|
||||||
// (SWW_WAF_DNSBL_ZONES), through DNSBLResolver (SWW_WAF_DNSBL_RESOLVER), or
|
// (SWW_WAF_DNSBL_ZONES), through DNSBLResolver (SWW_WAF_DNSBL_RESOLVER), or
|
||||||
@@ -153,10 +169,18 @@ type WAF struct {
|
|||||||
AbuseIPDBKey string
|
AbuseIPDBKey string
|
||||||
AbuseIPDBMinScore int64
|
AbuseIPDBMinScore int64
|
||||||
AbuseIPDBDailyBudget int
|
AbuseIPDBDailyBudget int
|
||||||
DNSBLAbuseIPDBAction string
|
DNSBLAbuseIPDBAction ReputationAction
|
||||||
DNSBLAbuseIPDBLimitPercent int64
|
DNSBLAbuseIPDBLimitPercent int64
|
||||||
IPReputationCacheTTL time.Duration
|
IPReputationCacheTTL time.Duration
|
||||||
DNSBLAbuseIPDBTimeout time.Duration
|
DNSBLAbuseIPDBTimeout time.Duration
|
||||||
|
// While AbuseIPDBKey is set, AbuseIPDB's blacklist of the clients it
|
||||||
|
// scores AbuseIPDBThreshold or more (SWW_WAF_ABUSEIPDB_THRESHOLD), from
|
||||||
|
// 25 to 100, is fetched at the start and every AbuseIPDBBlacklistRefresh
|
||||||
|
// (SWW_WAF_ABUSEIPDB_BLACKLIST_REFRESH_INTERVAL_MINS). Each client on it
|
||||||
|
// is banned for AbuseIPDBBanDuration (SWW_WAF_ABUSEIPDB_BAN_DAYS).
|
||||||
|
AbuseIPDBThreshold int64
|
||||||
|
AbuseIPDBBlacklistRefresh time.Duration
|
||||||
|
AbuseIPDBBanDuration time.Duration
|
||||||
// BanResponse is the status a refused client is answered with, 403
|
// BanResponse is the status a refused client is answered with, 403
|
||||||
// or 429, or 0 to close the connection without an answer
|
// or 429, or 0 to close the connection without an answer
|
||||||
// (SWW_WAF_BAN_RESPONSE). It answers a banned client, a request that
|
// (SWW_WAF_BAN_RESPONSE). It answers a banned client, a request that
|
||||||
@@ -228,7 +252,7 @@ type WAF struct {
|
|||||||
// reads none.
|
// reads none.
|
||||||
// CoreRuleSetExclusionsFile is the path of the file of its exclusions
|
// CoreRuleSetExclusionsFile is the path of the file of its exclusions
|
||||||
// (SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE), "" while there is none.
|
// (SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE), "" while there is none.
|
||||||
CoreRuleSetMode string
|
CoreRuleSetMode CoreRuleSetMode
|
||||||
CoreRuleSetParanoiaLevel int
|
CoreRuleSetParanoiaLevel int
|
||||||
CoreRuleSetScoreThreshold int
|
CoreRuleSetScoreThreshold int
|
||||||
CoreRuleSetDisabledRules []int
|
CoreRuleSetDisabledRules []int
|
||||||
@@ -307,20 +331,66 @@ type WAF struct {
|
|||||||
settings []slog.Attr
|
settings []slog.Attr
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// BytesCounted is which body bytes count toward the byte limits and the
|
||||||
|
// byte anomaly thresholds, a value of SWW_WAF_BYTES_COUNTED.
|
||||||
|
type BytesCounted string
|
||||||
|
|
||||||
|
// The values of SWW_WAF_BYTES_COUNTED.
|
||||||
|
const (
|
||||||
|
// BytesCountedResponse counts the response's body bytes.
|
||||||
|
BytesCountedResponse BytesCounted = "response"
|
||||||
|
// BytesCountedRequest counts the request's body bytes.
|
||||||
|
BytesCountedRequest BytesCounted = "request"
|
||||||
|
// BytesCountedBoth counts the body bytes of both.
|
||||||
|
BytesCountedBoth BytesCounted = "both"
|
||||||
|
)
|
||||||
|
|
||||||
|
// LookupSource is where each client's AS number and country are looked up,
|
||||||
|
// a value of SWW_WAF_LOOKUP_SOURCE.
|
||||||
|
type LookupSource string
|
||||||
|
|
||||||
|
// The values of SWW_WAF_LOOKUP_SOURCE.
|
||||||
|
const (
|
||||||
|
// LookupSourceGeoJS asks GeoJS.
|
||||||
|
LookupSourceGeoJS LookupSource = "geojs"
|
||||||
|
// LookupSourceFile looks clients up in the lookup database, the file
|
||||||
|
// SWW_WAF_LOOKUP_DB_PATH names.
|
||||||
|
LookupSourceFile LookupSource = "file"
|
||||||
|
// LookupSourceOff looks no client up.
|
||||||
|
LookupSourceOff LookupSource = "off"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ReputationAction is what is done with a client a reputation source
|
||||||
|
// lists, a value of SWW_WAF_BLOCKLIST_ACTION and of
|
||||||
|
// SWW_WAF_DNSBL_ABUSEIPDB_ACTION.
|
||||||
|
type ReputationAction string
|
||||||
|
|
||||||
|
// The values of SWW_WAF_BLOCKLIST_ACTION and SWW_WAF_DNSBL_ABUSEIPDB_ACTION.
|
||||||
|
const (
|
||||||
|
// ReputationActionDeny refuses the client's requests.
|
||||||
|
ReputationActionDeny ReputationAction = "deny"
|
||||||
|
// ReputationActionLimit gives the client a percentage of every limit,
|
||||||
|
// written limit:<percent> in the setting.
|
||||||
|
ReputationActionLimit ReputationAction = "limit"
|
||||||
|
// ReputationActionLog only notes the hit in the log line and raises an
|
||||||
|
// alert.
|
||||||
|
ReputationActionLog ReputationAction = "log"
|
||||||
|
)
|
||||||
|
|
||||||
|
// CoreRuleSetMode is what the Core Rule Set does, a value of
|
||||||
|
// SWW_WAF_CORE_RULE_SET_MODE.
|
||||||
|
type CoreRuleSetMode string
|
||||||
|
|
||||||
// The values of SWW_WAF_CORE_RULE_SET_MODE.
|
// The values of SWW_WAF_CORE_RULE_SET_MODE.
|
||||||
const (
|
const (
|
||||||
// CoreRuleSetModeOff runs no request through the Core Rule Set.
|
// CoreRuleSetModeOff runs no request through the Core Rule Set.
|
||||||
CoreRuleSetModeOff = off
|
CoreRuleSetModeOff CoreRuleSetMode = "off"
|
||||||
// CoreRuleSetModeDetect logs and alerts a match, and refuses nothing.
|
// CoreRuleSetModeDetect logs and alerts a match, and refuses nothing.
|
||||||
CoreRuleSetModeDetect = "detect"
|
CoreRuleSetModeDetect CoreRuleSetMode = "detect"
|
||||||
// CoreRuleSetModeBlock refuses a match with 403.
|
// CoreRuleSetModeBlock refuses a match with 403.
|
||||||
CoreRuleSetModeBlock = "block"
|
CoreRuleSetModeBlock CoreRuleSetMode = "block"
|
||||||
)
|
)
|
||||||
|
|
||||||
// fileSource is the SWW_WAF_LOOKUP_SOURCE that looks clients up in the
|
|
||||||
// lookup database, the file SWW_WAF_LOOKUP_DB_PATH names.
|
|
||||||
const fileSource = "file"
|
|
||||||
|
|
||||||
const (
|
const (
|
||||||
day = 24 * time.Hour
|
day = 24 * time.Hour
|
||||||
ipv4Bits = 32
|
ipv4Bits = 32
|
||||||
@@ -329,6 +399,9 @@ const (
|
|||||||
// netblock a provider is usually given: a shorter one would make one
|
// netblock a provider is usually given: a shorter one would make one
|
||||||
// client of the customers of several providers.
|
// client of the customers of several providers.
|
||||||
minIPv6GroupPrefix = 32
|
minIPv6GroupPrefix = 32
|
||||||
|
// minAbuseIPDBThreshold is the lowest score AbuseIPDB's blacklist can
|
||||||
|
// be cut at.
|
||||||
|
minAbuseIPDBThreshold = 25
|
||||||
// minTokenLength is the fewest characters a token may have.
|
// minTokenLength is the fewest characters a token may have.
|
||||||
minTokenLength = 32
|
minTokenLength = 32
|
||||||
// maxParanoiaLevel is the Core Rule Set's highest paranoia level.
|
// maxParanoiaLevel is the Core Rule Set's highest paranoia level.
|
||||||
@@ -364,8 +437,8 @@ var (
|
|||||||
"it names the file to look clients up in",
|
"it names the file to look clients up in",
|
||||||
)
|
)
|
||||||
errDBPathUnused = errors.New("only file reads it")
|
errDBPathUnused = errors.New("only file reads it")
|
||||||
errOver64K = errors.New(
|
errOver8K = errors.New(
|
||||||
"is more than 64K, the most waf gets of a body")
|
"is more than 8K, the most the Core Rule Set can read in time")
|
||||||
errNotDurationAboveZero = errors.New(
|
errNotDurationAboveZero = errors.New(
|
||||||
"is not a duration above zero, such as 1h or 7d")
|
"is not a duration above zero, such as 1h or 7d")
|
||||||
errNotNumberAboveZero = errors.New(
|
errNotNumberAboveZero = errors.New(
|
||||||
@@ -439,11 +512,14 @@ var (
|
|||||||
"is not a whole number above zero, such as 60, or off",
|
"is not a whole number above zero, such as 60, or off",
|
||||||
)
|
)
|
||||||
errNotUTF8 = errors.New("is not valid UTF-8")
|
errNotUTF8 = errors.New("is not valid UTF-8")
|
||||||
errNotASN = errors.New("is not an AS number such as AS64496")
|
|
||||||
errNotPercentItem = errors.New(
|
errNotPercentItem = errors.New(
|
||||||
"is not a code, : and a percentage, such as AS64496:50 or cn:25")
|
"is not a code, : and a percentage, such as AS64496:50 or cn:25")
|
||||||
errNotPercent = errors.New(
|
errNotAbuseIPDBThreshold = errors.New(
|
||||||
"is not a percentage, a whole number from 0 to 100",
|
"is not a whole number from 25 to 100, the scores AbuseIPDB's " +
|
||||||
|
"blacklist can be cut at",
|
||||||
|
)
|
||||||
|
errNotWholeUnits = errors.New(
|
||||||
|
"is not a whole number above zero, such as 30, that fits a duration",
|
||||||
)
|
)
|
||||||
errListedTwice = errors.New("is listed twice")
|
errListedTwice = errors.New("is listed twice")
|
||||||
errNotListURL = errors.New(
|
errNotListURL = errors.New(
|
||||||
@@ -539,36 +615,38 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
|
|||||||
),
|
),
|
||||||
BytesLimitPerHour: env.size("SWW_WAF_BYTES_LIMIT_PER_HOUR", "50G"),
|
BytesLimitPerHour: env.size("SWW_WAF_BYTES_LIMIT_PER_HOUR", "50G"),
|
||||||
BytesLimitPerDay: env.size("SWW_WAF_BYTES_LIMIT_PER_DAY", "100G"),
|
BytesLimitPerDay: env.size("SWW_WAF_BYTES_LIMIT_PER_DAY", "100G"),
|
||||||
BytesCounted: env.bytesCounted("SWW_WAF_BYTES_COUNTED", "both"),
|
BytesCounted: env.bytesCounted(
|
||||||
|
"SWW_WAF_BYTES_COUNTED",
|
||||||
|
BytesCountedBoth,
|
||||||
|
),
|
||||||
LookupSource: env.lookupSource(
|
LookupSource: env.lookupSource(
|
||||||
"SWW_WAF_LOOKUP_SOURCE",
|
"SWW_WAF_LOOKUP_SOURCE",
|
||||||
"geojs",
|
LookupSourceGeoJS,
|
||||||
),
|
),
|
||||||
LookupDBPath: env.value("SWW_WAF_LOOKUP_DB_PATH", ""),
|
LookupDBPath: env.value("SWW_WAF_LOOKUP_DB_PATH", ""),
|
||||||
ASNAndCountryLookupTimeout: env.milliseconds(
|
ASNAndCountryLookupTimeout: env.milliseconds(
|
||||||
"SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS",
|
"SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS",
|
||||||
"800",
|
"800",
|
||||||
),
|
),
|
||||||
AddLookupHeaders: env.boolean("SWW_WAF_ADD_LOOKUP_HEADERS", "false"),
|
|
||||||
ExclusivelyAllowedCountries: env.countries(
|
ExclusivelyAllowedCountries: env.countries(
|
||||||
"SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES", ""),
|
ExclusivelyAllowedCountriesSetting, ""),
|
||||||
TagLists: env.tagLists("SWW_WAF_TAG_LISTS"),
|
TagLists: env.tagLists("SWW_WAF_TAG_LISTS"),
|
||||||
TagDeny: env.tags("SWW_WAF_TAG_DENY"),
|
TagDeny: env.tags("SWW_WAF_TAG_DENY"),
|
||||||
TagLimitPercent: env.tagPercents("SWW_WAF_TAG_LIMIT_PERCENT"),
|
TagLimitPercent: env.tagPercents(TagLimitPercentSetting),
|
||||||
RouteWatchURL: env.routeWatchURL("SWW_WAF_ROUTEWATCH_URL"),
|
RouteWatchURL: env.routeWatchURL("SWW_WAF_ROUTEWATCH_URL"),
|
||||||
ASNBytesLimitPercent: env.percents(
|
ASNBytesLimitPercent: env.percents(
|
||||||
"SWW_WAF_ASN_BYTES_LIMIT_PERCENT",
|
ASNBytesLimitPercentSetting,
|
||||||
ParseASN,
|
reputation.ParseASN,
|
||||||
),
|
),
|
||||||
CountryBytesLimitPercent: env.percents(
|
CountryBytesLimitPercent: env.percents(
|
||||||
"SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT",
|
CountryBytesLimitPercentSetting,
|
||||||
parseCountry,
|
parseCountry,
|
||||||
),
|
),
|
||||||
UnknownCountryLimitPercent: env.percent(
|
UnknownCountryLimitPercent: env.percent(
|
||||||
"SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT",
|
UnknownCountryLimitPercentSetting,
|
||||||
"100",
|
"100",
|
||||||
),
|
),
|
||||||
ASNLimitPercentURL: env.listURL("SWW_WAF_ASN_LIMIT_PERCENT_URL"),
|
ASNLimitPercentURL: env.listURL(ASNLimitPercentURLSetting),
|
||||||
BlocklistURLs: env.listURLs("SWW_WAF_BLOCKLIST_URLS"),
|
BlocklistURLs: env.listURLs("SWW_WAF_BLOCKLIST_URLS"),
|
||||||
BlocklistRefresh: env.refresh("SWW_WAF_BLOCKLIST_REFRESH", "24h"),
|
BlocklistRefresh: env.refresh("SWW_WAF_BLOCKLIST_REFRESH", "24h"),
|
||||||
DNSBLZones: env.zones("SWW_WAF_DNSBL_ZONES"),
|
DNSBLZones: env.zones("SWW_WAF_DNSBL_ZONES"),
|
||||||
@@ -587,6 +665,20 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
|
|||||||
"SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS",
|
"SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS",
|
||||||
"2000",
|
"2000",
|
||||||
),
|
),
|
||||||
|
AbuseIPDBThreshold: env.abuseIPDBThreshold(
|
||||||
|
"SWW_WAF_ABUSEIPDB_THRESHOLD",
|
||||||
|
"90",
|
||||||
|
),
|
||||||
|
AbuseIPDBBlacklistRefresh: env.wholeUnits(
|
||||||
|
"SWW_WAF_ABUSEIPDB_BLACKLIST_REFRESH_INTERVAL_MINS",
|
||||||
|
"60",
|
||||||
|
time.Minute,
|
||||||
|
),
|
||||||
|
AbuseIPDBBanDuration: env.wholeUnits(
|
||||||
|
"SWW_WAF_ABUSEIPDB_BAN_DAYS",
|
||||||
|
"30",
|
||||||
|
day,
|
||||||
|
),
|
||||||
BanResponse: env.banResponse("SWW_WAF_BAN_RESPONSE", "403"),
|
BanResponse: env.banResponse("SWW_WAF_BAN_RESPONSE", "403"),
|
||||||
LimitBanDuration: env.durationNotOff(
|
LimitBanDuration: env.durationNotOff(
|
||||||
"SWW_WAF_LIMIT_BAN_DURATION",
|
"SWW_WAF_LIMIT_BAN_DURATION",
|
||||||
@@ -724,10 +816,13 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
|
|||||||
|
|
||||||
cfg.LogRemoteAppName = env.appName("SWW_WAF_LOG_REMOTE_APP_NAME",
|
cfg.LogRemoteAppName = env.appName("SWW_WAF_LOG_REMOTE_APP_NAME",
|
||||||
cfg.InstanceName, cfg.LogRemoteURL != nil)
|
cfg.InstanceName, cfg.LogRemoteURL != nil)
|
||||||
|
// The app gets the headers by default, unless no client is looked up.
|
||||||
|
cfg.AddLookupHeaders = env.boolean("SWW_WAF_ADD_LOOKUP_HEADERS",
|
||||||
|
strconv.FormatBool(cfg.LookupSource != LookupSourceOff))
|
||||||
cfg.BlocklistAction, cfg.BlocklistLimitPercent = env.action(
|
cfg.BlocklistAction, cfg.BlocklistLimitPercent = env.action(
|
||||||
"SWW_WAF_BLOCKLIST_ACTION", "deny")
|
BlocklistActionSetting, string(ReputationActionDeny))
|
||||||
cfg.DNSBLAbuseIPDBAction, cfg.DNSBLAbuseIPDBLimitPercent = env.action(
|
cfg.DNSBLAbuseIPDBAction, cfg.DNSBLAbuseIPDBLimitPercent = env.action(
|
||||||
"SWW_WAF_DNSBL_ABUSEIPDB_ACTION", "limit:25")
|
DNSBLAbuseIPDBActionSetting, "limit:25")
|
||||||
|
|
||||||
env.checkInstanceNameForNtfy(cfg.InstanceName, cfg.AlertNtfyURL != nil)
|
env.checkInstanceNameForNtfy(cfg.InstanceName, cfg.AlertNtfyURL != nil)
|
||||||
env.checkLookupDBPath(cfg)
|
env.checkLookupDBPath(cfg)
|
||||||
@@ -823,9 +918,12 @@ func (e *environment) count(name, defaultValue string) int64 {
|
|||||||
|
|
||||||
// bytesCounted reads the setting that is which body bytes count toward the
|
// bytesCounted reads the setting that is which body bytes count toward the
|
||||||
// byte limits and the byte anomaly thresholds: response, request or both.
|
// byte limits and the byte anomaly thresholds: response, request or both.
|
||||||
func (e *environment) bytesCounted(name, defaultValue string) string {
|
func (e *environment) bytesCounted(
|
||||||
value := e.value(name, defaultValue)
|
name string, defaultValue BytesCounted,
|
||||||
if value != "response" && value != "request" && value != "both" {
|
) BytesCounted {
|
||||||
|
value := BytesCounted(e.value(name, string(defaultValue)))
|
||||||
|
if value != BytesCountedResponse && value != BytesCountedRequest &&
|
||||||
|
value != BytesCountedBoth {
|
||||||
e.check(name, fmt.Errorf("%q %w", value, errNotBytesCounted))
|
e.check(name, fmt.Errorf("%q %w", value, errNotBytesCounted))
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -870,8 +968,10 @@ func (e *environment) trapPaths(name string) []string {
|
|||||||
|
|
||||||
// coreRuleSetMode reads the setting that is what the Core Rule Set does: off,
|
// coreRuleSetMode reads the setting that is what the Core Rule Set does: off,
|
||||||
// detect or block.
|
// detect or block.
|
||||||
func (e *environment) coreRuleSetMode(name, defaultValue string) string {
|
func (e *environment) coreRuleSetMode(
|
||||||
mode := e.value(name, defaultValue)
|
name string, defaultValue CoreRuleSetMode,
|
||||||
|
) CoreRuleSetMode {
|
||||||
|
mode := CoreRuleSetMode(e.value(name, string(defaultValue)))
|
||||||
if mode != CoreRuleSetModeOff && mode != CoreRuleSetModeDetect &&
|
if mode != CoreRuleSetModeOff && mode != CoreRuleSetModeDetect &&
|
||||||
mode != CoreRuleSetModeBlock {
|
mode != CoreRuleSetModeBlock {
|
||||||
e.check(name, fmt.Errorf("%q %w", mode, errNotCoreRuleSetMode))
|
e.check(name, fmt.Errorf("%q %w", mode, errNotCoreRuleSetMode))
|
||||||
@@ -922,7 +1022,7 @@ func (e *environment) percents(
|
|||||||
}
|
}
|
||||||
|
|
||||||
// tagLists reads SWW_WAF_TAG_LISTS, which is empty by default.
|
// tagLists reads SWW_WAF_TAG_LISTS, which is empty by default.
|
||||||
func (e *environment) tagLists(name string) []TagList {
|
func (e *environment) tagLists(name string) []reputation.TagList {
|
||||||
lists, err := parseTagLists(e.value(name, ""))
|
lists, err := parseTagLists(e.value(name, ""))
|
||||||
e.check(name, err)
|
e.check(name, err)
|
||||||
|
|
||||||
@@ -948,12 +1048,41 @@ func (e *environment) tagPercents(name string) map[string]int64 {
|
|||||||
|
|
||||||
// percent reads a setting that is a percentage, from 0 to 100.
|
// percent reads a setting that is a percentage, from 0 to 100.
|
||||||
func (e *environment) percent(name, defaultValue string) int64 {
|
func (e *environment) percent(name, defaultValue string) int64 {
|
||||||
percent, err := ParsePercent(e.value(name, defaultValue))
|
percent, err := reputation.ParsePercent(e.value(name, defaultValue))
|
||||||
e.check(name, err)
|
e.check(name, err)
|
||||||
|
|
||||||
return percent
|
return percent
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// abuseIPDBThreshold reads the setting that is the least score of the
|
||||||
|
// clients on AbuseIPDB's blacklist: a whole number from 25 to 100, the
|
||||||
|
// range AbuseIPDB takes.
|
||||||
|
func (e *environment) abuseIPDBThreshold(name, defaultValue string) int64 {
|
||||||
|
value := e.value(name, defaultValue)
|
||||||
|
|
||||||
|
threshold, err := strconv.ParseInt(value, 10, 64)
|
||||||
|
if err != nil || threshold < minAbuseIPDBThreshold || threshold > 100 {
|
||||||
|
e.check(name, fmt.Errorf("%q %w", value, errNotAbuseIPDBThreshold))
|
||||||
|
}
|
||||||
|
|
||||||
|
return threshold
|
||||||
|
}
|
||||||
|
|
||||||
|
// wholeUnits reads a setting that is a whole number above zero of unit,
|
||||||
|
// such as minutes or days, and returns it as a duration.
|
||||||
|
func (e *environment) wholeUnits(
|
||||||
|
name, defaultValue string, unit time.Duration,
|
||||||
|
) time.Duration {
|
||||||
|
value := e.value(name, defaultValue)
|
||||||
|
|
||||||
|
n, err := strconv.ParseInt(value, 10, 64)
|
||||||
|
if err != nil || n <= 0 || n > math.MaxInt64/int64(unit) {
|
||||||
|
e.check(name, fmt.Errorf("%q %w", value, errNotWholeUnits))
|
||||||
|
}
|
||||||
|
|
||||||
|
return time.Duration(n) * unit
|
||||||
|
}
|
||||||
|
|
||||||
// listURL reads a setting that is the URL a list is fetched from, "" while
|
// listURL reads a setting that is the URL a list is fetched from, "" while
|
||||||
// it is unset or empty.
|
// it is unset or empty.
|
||||||
func (e *environment) listURL(name string) string {
|
func (e *environment) listURL(name string) string {
|
||||||
@@ -992,25 +1121,29 @@ func (e *environment) refresh(name, defaultValue string) time.Duration {
|
|||||||
// action reads a setting that is what is done with a client a blocklist
|
// action reads a setting that is what is done with a client a blocklist
|
||||||
// or a DNSBL zone lists: deny, log, or limit:<percent>, which it returns
|
// or a DNSBL zone lists: deny, log, or limit:<percent>, which it returns
|
||||||
// as limit and the percentage.
|
// as limit and the percentage.
|
||||||
func (e *environment) action(name, defaultValue string) (string, int64) {
|
func (e *environment) action(
|
||||||
value := e.value(name, defaultValue)
|
name, defaultValue string,
|
||||||
if value == "deny" || value == "log" {
|
) (ReputationAction, int64) {
|
||||||
|
value := ReputationAction(e.value(name, defaultValue))
|
||||||
|
if value == ReputationActionDeny || value == ReputationActionLog {
|
||||||
return value, 0
|
return value, 0
|
||||||
}
|
}
|
||||||
|
|
||||||
percentText, isLimit := strings.CutPrefix(value, "limit:")
|
percentText, isLimit := strings.CutPrefix(
|
||||||
|
string(value), string(ReputationActionLimit)+":",
|
||||||
|
)
|
||||||
|
|
||||||
percent, err := ParsePercent(percentText)
|
percent, err := reputation.ParsePercent(percentText)
|
||||||
if !isLimit || err != nil {
|
if !isLimit || err != nil {
|
||||||
e.check(name, fmt.Errorf("%q %w", value, errNotAction))
|
e.check(name, fmt.Errorf("%q %w", value, errNotAction))
|
||||||
}
|
}
|
||||||
|
|
||||||
return "limit", percent
|
return ReputationActionLimit, percent
|
||||||
}
|
}
|
||||||
|
|
||||||
// zones reads the setting that is the list of DNSBL zones. It is empty by
|
// zones reads the setting that is the list of DNSBL zones. It is empty by
|
||||||
// default. The log shows each zone with its key masked, as MaskZoneKey
|
// default. The log shows each zone with its key masked, as
|
||||||
// masks it.
|
// reputation.MaskZoneKey masks it.
|
||||||
func (e *environment) zones(name string) []string {
|
func (e *environment) zones(name string) []string {
|
||||||
value, _ := e.lookup(name)
|
value, _ := e.lookup(name)
|
||||||
zones, err := parseZones(value)
|
zones, err := parseZones(value)
|
||||||
@@ -1018,7 +1151,7 @@ func (e *environment) zones(name string) []string {
|
|||||||
|
|
||||||
logged := make([]string, len(zones))
|
logged := make([]string, len(zones))
|
||||||
for i, zone := range zones {
|
for i, zone := range zones {
|
||||||
logged[i] = MaskZoneKey(zone)
|
logged[i] = reputation.MaskZoneKey(zone)
|
||||||
}
|
}
|
||||||
|
|
||||||
e.settings = append(
|
e.settings = append(
|
||||||
@@ -1056,9 +1189,12 @@ func (e *environment) routeWatchURL(name string) string {
|
|||||||
|
|
||||||
// lookupSource reads the setting that is where clients are looked up:
|
// lookupSource reads the setting that is where clients are looked up:
|
||||||
// geojs, file, or off.
|
// geojs, file, or off.
|
||||||
func (e *environment) lookupSource(name, defaultValue string) string {
|
func (e *environment) lookupSource(
|
||||||
source := e.value(name, defaultValue)
|
name string, defaultValue LookupSource,
|
||||||
if source != "geojs" && source != fileSource && source != off {
|
) LookupSource {
|
||||||
|
source := LookupSource(e.value(name, string(defaultValue)))
|
||||||
|
if source != LookupSourceGeoJS && source != LookupSourceFile &&
|
||||||
|
source != LookupSourceOff {
|
||||||
e.check(name, fmt.Errorf("%q %w", source, errNotLookupSource))
|
e.check(name, fmt.Errorf("%q %w", source, errNotLookupSource))
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1070,12 +1206,12 @@ func (e *environment) lookupSource(name, defaultValue string) string {
|
|||||||
// one source at a time.
|
// one source at a time.
|
||||||
func (e *environment) checkLookupDBPath(cfg *WAF) {
|
func (e *environment) checkLookupDBPath(cfg *WAF) {
|
||||||
switch {
|
switch {
|
||||||
case cfg.LookupSource == fileSource && cfg.LookupDBPath == "":
|
case cfg.LookupSource == LookupSourceFile && cfg.LookupDBPath == "":
|
||||||
e.check("SWW_WAF_LOOKUP_SOURCE", fmt.Errorf(
|
e.check("SWW_WAF_LOOKUP_SOURCE", fmt.Errorf(
|
||||||
"is file while SWW_WAF_LOOKUP_DB_PATH is unset; %w",
|
"is file while SWW_WAF_LOOKUP_DB_PATH is unset; %w",
|
||||||
errNeedsDBPath,
|
errNeedsDBPath,
|
||||||
))
|
))
|
||||||
case cfg.LookupSource != fileSource && cfg.LookupDBPath != "":
|
case cfg.LookupSource != LookupSourceFile && cfg.LookupDBPath != "":
|
||||||
e.check("SWW_WAF_LOOKUP_DB_PATH", fmt.Errorf(
|
e.check("SWW_WAF_LOOKUP_DB_PATH", fmt.Errorf(
|
||||||
"is set while SWW_WAF_LOOKUP_SOURCE is %s; %w",
|
"is set while SWW_WAF_LOOKUP_SOURCE is %s; %w",
|
||||||
cfg.LookupSource,
|
cfg.LookupSource,
|
||||||
@@ -1095,12 +1231,12 @@ func (e *environment) checkLookupDBPath(cfg *WAF) {
|
|||||||
func (e *environment) checkCountriesAndLookups(cfg *WAF) {
|
func (e *environment) checkCountriesAndLookups(cfg *WAF) {
|
||||||
for _, country := range cfg.ExclusivelyAllowedCountries {
|
for _, country := range cfg.ExclusivelyAllowedCountries {
|
||||||
if slices.Contains(cfg.TagDeny, CountryTagPrefix+country) {
|
if slices.Contains(cfg.TagDeny, CountryTagPrefix+country) {
|
||||||
e.check("SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES",
|
e.check(ExclusivelyAllowedCountriesSetting,
|
||||||
fmt.Errorf("%q %w", country, errOnBothLists))
|
fmt.Errorf("%q %w", country, errOnBothLists))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if cfg.LookupSource != off {
|
if cfg.LookupSource != LookupSourceOff {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1114,15 +1250,15 @@ func (e *environment) checkCountriesAndLookups(cfg *WAF) {
|
|||||||
name string
|
name string
|
||||||
set bool
|
set bool
|
||||||
}{
|
}{
|
||||||
{"SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES", len(cfg.ExclusivelyAllowedCountries) > 0},
|
{ExclusivelyAllowedCountriesSetting, len(cfg.ExclusivelyAllowedCountries) > 0},
|
||||||
{"SWW_WAF_ADD_LOOKUP_HEADERS", cfg.AddLookupHeaders},
|
{"SWW_WAF_ADD_LOOKUP_HEADERS", cfg.AddLookupHeaders},
|
||||||
{"SWW_WAF_TAG_DENY", slices.ContainsFunc(cfg.TagDeny, onlyLookupGives)},
|
{"SWW_WAF_TAG_DENY", slices.ContainsFunc(cfg.TagDeny, onlyLookupGives)},
|
||||||
{"SWW_WAF_TAG_LIMIT_PERCENT", slices.ContainsFunc(
|
{TagLimitPercentSetting, slices.ContainsFunc(
|
||||||
slices.Collect(maps.Keys(cfg.TagLimitPercent)), onlyLookupGives)},
|
slices.Collect(maps.Keys(cfg.TagLimitPercent)), onlyLookupGives)},
|
||||||
{"SWW_WAF_ASN_BYTES_LIMIT_PERCENT", len(cfg.ASNBytesLimitPercent) > 0},
|
{ASNBytesLimitPercentSetting, len(cfg.ASNBytesLimitPercent) > 0},
|
||||||
{"SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT", len(cfg.CountryBytesLimitPercent) > 0},
|
{CountryBytesLimitPercentSetting, len(cfg.CountryBytesLimitPercent) > 0},
|
||||||
{"SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT", cfg.UnknownCountryLimitPercent < 100},
|
{UnknownCountryLimitPercentSetting, cfg.UnknownCountryLimitPercent < 100},
|
||||||
{"SWW_WAF_ASN_LIMIT_PERCENT_URL", cfg.ASNLimitPercentURL != ""},
|
{ASNLimitPercentURLSetting, cfg.ASNLimitPercentURL != ""},
|
||||||
{"SWW_WAF_ANOMALY_ASN_REQUESTS_PER_MINUTE", cfg.AnomalyASN.RequestsPerMinute > 0},
|
{"SWW_WAF_ANOMALY_ASN_REQUESTS_PER_MINUTE", cfg.AnomalyASN.RequestsPerMinute > 0},
|
||||||
{"SWW_WAF_ANOMALY_ASN_REQUESTS_PER_HOUR", cfg.AnomalyASN.RequestsPerHour > 0},
|
{"SWW_WAF_ANOMALY_ASN_REQUESTS_PER_HOUR", cfg.AnomalyASN.RequestsPerHour > 0},
|
||||||
{"SWW_WAF_ANOMALY_ASN_BYTES_PER_MINUTE", cfg.AnomalyASN.BytesPerMinute > 0},
|
{"SWW_WAF_ANOMALY_ASN_BYTES_PER_MINUTE", cfg.AnomalyASN.BytesPerMinute > 0},
|
||||||
@@ -1142,7 +1278,7 @@ func (e *environment) checkCountriesAndLookups(cfg *WAF) {
|
|||||||
// blocklist too: the file at a URL is fetched as one list or the other.
|
// blocklist too: the file at a URL is fetched as one list or the other.
|
||||||
func (e *environment) checkASNLimitPercentURL(cfg *WAF) {
|
func (e *environment) checkASNLimitPercentURL(cfg *WAF) {
|
||||||
if slices.Contains(cfg.BlocklistURLs, cfg.ASNLimitPercentURL) {
|
if slices.Contains(cfg.BlocklistURLs, cfg.ASNLimitPercentURL) {
|
||||||
e.check("SWW_WAF_ASN_LIMIT_PERCENT_URL",
|
e.check(ASNLimitPercentURLSetting,
|
||||||
fmt.Errorf("%q %w", cfg.ASNLimitPercentURL, errInBlocklistURLs))
|
fmt.Errorf("%q %w", cfg.ASNLimitPercentURL, errInBlocklistURLs))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -1169,14 +1305,18 @@ func (e *environment) checkTags(cfg *WAF) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
for tag := range cfg.TagLimitPercent {
|
for tag := range cfg.TagLimitPercent {
|
||||||
e.check("SWW_WAF_TAG_LIMIT_PERCENT", checkTag(cfg, tag))
|
e.check(TagLimitPercentSetting, checkTag(cfg, tag))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// checkTag returns an error when tag would never apply, as checkTags says.
|
// checkTag returns an error when tag would never apply, as checkTags says.
|
||||||
func checkTag(cfg *WAF, tag string) error {
|
func checkTag(cfg *WAF, tag string) error {
|
||||||
if !requestCanGetTag(cfg, tag) {
|
if !requestCanGetTag(cfg, tag) {
|
||||||
return fmt.Errorf("%q %w", MaskZoneKey(tag), errNoRequestGetsTag)
|
return fmt.Errorf(
|
||||||
|
"%q %w",
|
||||||
|
reputation.MaskZoneKey(tag),
|
||||||
|
errNoRequestGetsTag,
|
||||||
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
@@ -1184,7 +1324,7 @@ func checkTag(cfg *WAF, tag string) error {
|
|||||||
|
|
||||||
// requestCanGetTag reports whether anything cfg sets up can give a request tag.
|
// requestCanGetTag reports whether anything cfg sets up can give a request tag.
|
||||||
func requestCanGetTag(cfg *WAF, tag string) bool {
|
func requestCanGetTag(cfg *WAF, tag string) bool {
|
||||||
isListName := func(list TagList) bool { return list.Name == tag }
|
isListName := func(list reputation.TagList) bool { return list.Name == tag }
|
||||||
if isLookupTag(tag) || slices.ContainsFunc(cfg.TagLists, isListName) {
|
if isLookupTag(tag) || slices.ContainsFunc(cfg.TagLists, isListName) {
|
||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
@@ -1208,7 +1348,7 @@ func requestCanGetTag(cfg *WAF, tag string) bool {
|
|||||||
|
|
||||||
return isZoneTag &&
|
return isZoneTag &&
|
||||||
slices.ContainsFunc(cfg.DNSBLZones, func(each string) bool {
|
slices.ContainsFunc(cfg.DNSBLZones, func(each string) bool {
|
||||||
return strings.ToLower(MaskZoneKey(each)) == zone
|
return strings.ToLower(reputation.MaskZoneKey(each)) == zone
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1598,16 +1738,16 @@ func durationOrDays(value string) (time.Duration, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// parseCoreRuleSetBodyInspectionSize reads the most of a request body the Core Rule Set
|
// parseCoreRuleSetBodyInspectionSize reads the most of a request body the Core Rule Set
|
||||||
// reads: a size as parseSize reads it, or off, but at most 64K, since waf
|
// reads: a size as parseSize reads it, or off, but at most 8K, so that one
|
||||||
// gets no more of a body than that.
|
// request cannot keep the Core Rule Set busy past rproxy's timeout.
|
||||||
func parseCoreRuleSetBodyInspectionSize(value string) (int64, error) {
|
func parseCoreRuleSetBodyInspectionSize(value string) (int64, error) {
|
||||||
limit, err := parseSize(value)
|
limit, err := parseSize(value)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, err
|
return 0, err
|
||||||
}
|
}
|
||||||
|
|
||||||
if limit > 64*kibibyte {
|
if limit > 8*kibibyte {
|
||||||
return 0, fmt.Errorf("%q %w", value, errOver64K)
|
return 0, fmt.Errorf("%q %w", value, errOver8K)
|
||||||
}
|
}
|
||||||
|
|
||||||
return limit, nil
|
return limit, nil
|
||||||
@@ -1706,7 +1846,9 @@ func parseNamedNetblocks(value string) ([]anomaly.NamedNetblock, error) {
|
|||||||
return nil, fmt.Errorf("%q %w", item, errNotNamedNetblock)
|
return nil, fmt.Errorf("%q %w", item, errNotNamedNetblock)
|
||||||
}
|
}
|
||||||
|
|
||||||
netblock, err := ParseNetblock(strings.TrimSpace(netblockText))
|
netblock, err := reputation.ParseNetblock(
|
||||||
|
strings.TrimSpace(netblockText),
|
||||||
|
)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
@@ -1912,21 +2054,6 @@ func parseCountry(value string) (string, error) {
|
|||||||
return country, nil
|
return country, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// ParseASN reads an AS number such as AS64496, in either case, and
|
|
||||||
// returns it as the lookup gives it: AS and the number, in capitals and
|
|
||||||
// without leading zeros. The file SWW_WAF_ASN_LIMIT_PERCENT_URL names is
|
|
||||||
// read with it too.
|
|
||||||
func ParseASN(value string) (string, error) {
|
|
||||||
digits, hasAS := strings.CutPrefix(strings.ToUpper(value), "AS")
|
|
||||||
|
|
||||||
number, err := strconv.ParseUint(digits, 10, 32)
|
|
||||||
if !hasAS || err != nil {
|
|
||||||
return "", fmt.Errorf("%q %w", value, errNotASN)
|
|
||||||
}
|
|
||||||
|
|
||||||
return "AS" + strconv.FormatUint(number, 10), nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// The tags a request can get that the settings can name, as waf
|
// The tags a request can get that the settings can name, as waf
|
||||||
// writes them. The prefixes are followed by a value.
|
// writes them. The prefixes are followed by a value.
|
||||||
const (
|
const (
|
||||||
@@ -1940,24 +2067,17 @@ const (
|
|||||||
ExploitSignatureTag = "exploitsignature"
|
ExploitSignatureTag = "exploitsignature"
|
||||||
)
|
)
|
||||||
|
|
||||||
// TagList is an address list of SWW_WAF_TAG_LISTS: the tag it adds to the
|
|
||||||
// requests of the clients it holds, and the URL it is fetched from.
|
|
||||||
type TagList struct {
|
|
||||||
Name string
|
|
||||||
URL string
|
|
||||||
}
|
|
||||||
|
|
||||||
// parseTagLists reads a comma-separated list of name=URL pairs, such as
|
// parseTagLists reads a comma-separated list of name=URL pairs, such as
|
||||||
// torexit=https://example.com/tor.txt, and returns each list with its name
|
// torexit=https://example.com/tor.txt, and returns each list with its name
|
||||||
// in lower case. A name can hold a =, as vpn=mullvad does, and ends at the
|
// in lower case. A name can hold a =, as vpn=mullvad does, and ends at the
|
||||||
// = before http. A name or a URL listed twice is an error.
|
// = before http. A name or a URL listed twice is an error.
|
||||||
func parseTagLists(value string) ([]TagList, error) {
|
func parseTagLists(value string) ([]reputation.TagList, error) {
|
||||||
items, err := parseList(value)
|
items, err := parseList(value)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
lists := make([]TagList, 0, len(items))
|
lists := make([]reputation.TagList, 0, len(items))
|
||||||
|
|
||||||
for _, item := range items {
|
for _, item := range items {
|
||||||
end := strings.Index(item, "=http")
|
end := strings.Index(item, "=http")
|
||||||
@@ -1965,9 +2085,12 @@ func parseTagLists(value string) ([]TagList, error) {
|
|||||||
return nil, fmt.Errorf("%q %w", item, errNotTagList)
|
return nil, fmt.Errorf("%q %w", item, errNotTagList)
|
||||||
}
|
}
|
||||||
|
|
||||||
list := TagList{Name: strings.ToLower(item[:end]), URL: item[end+1:]}
|
list := reputation.TagList{
|
||||||
|
Name: strings.ToLower(item[:end]),
|
||||||
|
URL: item[end+1:],
|
||||||
|
}
|
||||||
|
|
||||||
if slices.ContainsFunc(lists, func(earlier TagList) bool {
|
if slices.ContainsFunc(lists, func(earlier reputation.TagList) bool {
|
||||||
return earlier.Name == list.Name || earlier.URL == list.URL
|
return earlier.Name == list.Name || earlier.URL == list.URL
|
||||||
}) {
|
}) {
|
||||||
return nil, fmt.Errorf("%q %w", item, errListedTwice)
|
return nil, fmt.Errorf("%q %w", item, errListedTwice)
|
||||||
@@ -1980,7 +2103,8 @@ func parseTagLists(value string) ([]TagList, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// parseTags reads a comma-separated list of tags, which parseTag reads. A
|
// parseTags reads a comma-separated list of tags, which parseTag reads. A
|
||||||
// tag listed twice is an error, which shows the tag as MaskZoneKey does.
|
// tag listed twice is an error, which shows the tag as
|
||||||
|
// reputation.MaskZoneKey does.
|
||||||
func parseTags(value string) ([]string, error) {
|
func parseTags(value string) ([]string, error) {
|
||||||
items, err := parseList(value)
|
items, err := parseList(value)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -2000,7 +2124,7 @@ func parseTags(value string) ([]string, error) {
|
|||||||
if slices.Contains(tags, tag) {
|
if slices.Contains(tags, tag) {
|
||||||
return nil, fmt.Errorf(
|
return nil, fmt.Errorf(
|
||||||
"%q %w",
|
"%q %w",
|
||||||
MaskZoneKey(item),
|
reputation.MaskZoneKey(item),
|
||||||
errListedTwice,
|
errListedTwice,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
@@ -2069,7 +2193,7 @@ func parseTagPercents(value string) (map[string]int64, error) {
|
|||||||
if colon <= 0 {
|
if colon <= 0 {
|
||||||
return nil, fmt.Errorf(
|
return nil, fmt.Errorf(
|
||||||
"%q %w",
|
"%q %w",
|
||||||
MaskZoneKey(item),
|
reputation.MaskZoneKey(item),
|
||||||
errNotTagPercentItem,
|
errNotTagPercentItem,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
@@ -2083,7 +2207,7 @@ func parseTagPercents(value string) (map[string]int64, error) {
|
|||||||
if err != nil || percent < 1 || percent > 99 {
|
if err != nil || percent < 1 || percent > 99 {
|
||||||
return nil, fmt.Errorf(
|
return nil, fmt.Errorf(
|
||||||
"%q %w",
|
"%q %w",
|
||||||
MaskZoneKey(item[colon+1:]),
|
reputation.MaskZoneKey(item[colon+1:]),
|
||||||
errNotTagPercent,
|
errNotTagPercent,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
@@ -2095,7 +2219,11 @@ func parseTagPercents(value string) (map[string]int64, error) {
|
|||||||
if _, listed := percents[tag]; listed {
|
if _, listed := percents[tag]; listed {
|
||||||
// tag, since item[:colon] may end in a space, which MaskZoneKey
|
// tag, since item[:colon] may end in a space, which MaskZoneKey
|
||||||
// would not mask.
|
// would not mask.
|
||||||
return nil, fmt.Errorf("%q %w", MaskZoneKey(tag), errListedTwice)
|
return nil, fmt.Errorf(
|
||||||
|
"%q %w",
|
||||||
|
reputation.MaskZoneKey(tag),
|
||||||
|
errListedTwice,
|
||||||
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
percents[tag] = percent
|
percents[tag] = percent
|
||||||
@@ -2129,7 +2257,7 @@ func parsePercents(
|
|||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
percent, err := ParsePercent(percentText)
|
percent, err := reputation.ParsePercent(percentText)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
@@ -2144,38 +2272,6 @@ func parsePercents(
|
|||||||
return percents, nil
|
return percents, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// ParsePercent reads a percentage, a whole number from 0 to 100. The file
|
|
||||||
// SWW_WAF_ASN_LIMIT_PERCENT_URL names is read with it too.
|
|
||||||
func ParsePercent(value string) (int64, error) {
|
|
||||||
percent, err := strconv.ParseInt(value, 10, 64)
|
|
||||||
if err != nil || percent < 0 || percent > 100 {
|
|
||||||
return 0, fmt.Errorf("%q %w", value, errNotPercent)
|
|
||||||
}
|
|
||||||
|
|
||||||
return percent, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// headerNameChars are the characters RFC 9110 allows in a header name:
|
|
||||||
// letters, digits and these marks.
|
|
||||||
const headerNameChars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz" +
|
|
||||||
"0123456789!#$%&'*+-.^_`|~"
|
|
||||||
|
|
||||||
// IsHeaderName reports whether name can be a header name: one or more of
|
|
||||||
// the characters RFC 9110 allows in one.
|
|
||||||
func IsHeaderName(name string) bool {
|
|
||||||
if name == "" {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, char := range name {
|
|
||||||
if !strings.ContainsRune(headerNameChars, char) {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
|
|
||||||
// parseHeaderNames reads a comma-separated list of header names in either
|
// parseHeaderNames reads a comma-separated list of header names in either
|
||||||
// case, and returns them in lower case. Host and Transfer-Encoding are
|
// case, and returns them in lower case. Host and Transfer-Encoding are
|
||||||
// refused: Go's HTTP server takes them out of the request's headers.
|
// refused: Go's HTTP server takes them out of the request's headers.
|
||||||
@@ -2188,7 +2284,7 @@ func parseHeaderNames(value string) ([]string, error) {
|
|||||||
headers := make([]string, 0, len(items))
|
headers := make([]string, 0, len(items))
|
||||||
|
|
||||||
for _, item := range items {
|
for _, item := range items {
|
||||||
if !IsHeaderName(item) {
|
if !httpguts.ValidHeaderFieldName(item) {
|
||||||
return nil, fmt.Errorf("%q %w", item, errNotHeaderName)
|
return nil, fmt.Errorf("%q %w", item, errNotHeaderName)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -2344,8 +2440,8 @@ const (
|
|||||||
// other name for one that does not exist, so that the zone would list no
|
// other name for one that does not exist, so that the zone would list no
|
||||||
// client. A zone listed twice is an error, whatever the case of its
|
// client. A zone listed twice is an error, whatever the case of its
|
||||||
// letters, which DNS names ignore, and whatever its key, since
|
// letters, which DNS names ignore, and whatever its key, since
|
||||||
// MaskZoneKey shows two keys of one zone alike. An error shows a zone as
|
// reputation.MaskZoneKey shows two keys of one zone alike. An error shows a
|
||||||
// MaskZoneKey does.
|
// zone as it does.
|
||||||
func parseZones(value string) ([]string, error) {
|
func parseZones(value string) ([]string, error) {
|
||||||
zones, err := parseList(value)
|
zones, err := parseList(value)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -2355,11 +2451,14 @@ func parseZones(value string) ([]string, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
for i, zone := range zones {
|
for i, zone := range zones {
|
||||||
shown := MaskZoneKey(zone)
|
shown := reputation.MaskZoneKey(zone)
|
||||||
listedBefore := slices.ContainsFunc(
|
listedBefore := slices.ContainsFunc(
|
||||||
zones[:i],
|
zones[:i],
|
||||||
func(earlier string) bool {
|
func(earlier string) bool {
|
||||||
return strings.EqualFold(MaskZoneKey(earlier), shown)
|
return strings.EqualFold(
|
||||||
|
reputation.MaskZoneKey(earlier),
|
||||||
|
shown,
|
||||||
|
)
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -2376,24 +2475,6 @@ func parseZones(value string) ([]string, error) {
|
|||||||
return zones, nil
|
return zones, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// MaskZoneKey returns zone with ******** in place of its key, if it is a
|
|
||||||
// zone of Spamhaus's keyed query service, a name under dq.spamhaus.net,
|
|
||||||
// such as <key>.xbl.dq.spamhaus.net, whose first label is the key. Any
|
|
||||||
// other zone it returns as it is. A zone is shown so wherever it leaves
|
|
||||||
// the process: in the log, the alerts and the metrics.
|
|
||||||
func MaskZoneKey(zone string) string {
|
|
||||||
// DNS names ignore case, and a name may be written with a dot at its
|
|
||||||
// end.
|
|
||||||
name := strings.TrimSuffix(strings.ToLower(zone), ".")
|
|
||||||
if !strings.HasSuffix(name, ".dq.spamhaus.net") {
|
|
||||||
return zone
|
|
||||||
}
|
|
||||||
|
|
||||||
_, rest, _ := strings.Cut(zone, ".")
|
|
||||||
|
|
||||||
return masked + "." + rest
|
|
||||||
}
|
|
||||||
|
|
||||||
// isZone reports whether each label of zone is as parseZones takes it. It
|
// isZone reports whether each label of zone is as parseZones takes it. It
|
||||||
// also checks the hostnames of SWW_WAF_APPS, which are DNS names too.
|
// also checks the hostnames of SWW_WAF_APPS, which are DNS names too.
|
||||||
func isZone(zone string) bool {
|
func isZone(zone string) bool {
|
||||||
@@ -2449,7 +2530,7 @@ func parseWebhookHeaders(value string) (http.Header, string, error) {
|
|||||||
name, headerValue, found := strings.Cut(item, ":")
|
name, headerValue, found := strings.Cut(item, ":")
|
||||||
name = strings.TrimSpace(name)
|
name = strings.TrimSpace(name)
|
||||||
|
|
||||||
if !found || !IsHeaderName(name) ||
|
if !found || !httpguts.ValidHeaderFieldName(name) ||
|
||||||
strings.ContainsAny(headerValue, "\r\n\x00") {
|
strings.ContainsAny(headerValue, "\r\n\x00") {
|
||||||
return nil, "", fmt.Errorf(
|
return nil, "", fmt.Errorf(
|
||||||
"item %d %w",
|
"item %d %w",
|
||||||
|
|||||||
+137
-45
@@ -18,10 +18,12 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/spf13/viper"
|
"github.com/spf13/viper"
|
||||||
|
"golang.org/x/net/http/httpguts"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
|
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
)
|
)
|
||||||
|
|
||||||
// The settings, by name.
|
// The settings, by name.
|
||||||
@@ -61,6 +63,9 @@ const (
|
|||||||
abuseIPDBKey = "SWW_WAF_ABUSEIPDB_KEY"
|
abuseIPDBKey = "SWW_WAF_ABUSEIPDB_KEY"
|
||||||
abuseIPDBMinScore = "SWW_WAF_ABUSEIPDB_MIN_SCORE"
|
abuseIPDBMinScore = "SWW_WAF_ABUSEIPDB_MIN_SCORE"
|
||||||
abuseIPDBDailyBudget = "SWW_WAF_ABUSEIPDB_DAILY_BUDGET"
|
abuseIPDBDailyBudget = "SWW_WAF_ABUSEIPDB_DAILY_BUDGET"
|
||||||
|
abuseIPDBThreshold = "SWW_WAF_ABUSEIPDB_THRESHOLD"
|
||||||
|
abuseIPDBBlacklistRefresh = "SWW_WAF_ABUSEIPDB_BLACKLIST_REFRESH_INTERVAL_MINS"
|
||||||
|
abuseIPDBBanDays = "SWW_WAF_ABUSEIPDB_BAN_DAYS"
|
||||||
dnsblAbuseIPDBAction = "SWW_WAF_DNSBL_ABUSEIPDB_ACTION"
|
dnsblAbuseIPDBAction = "SWW_WAF_DNSBL_ABUSEIPDB_ACTION"
|
||||||
ipReputationCacheTTL = "SWW_WAF_IP_REPUTATION_CACHE_TTL"
|
ipReputationCacheTTL = "SWW_WAF_IP_REPUTATION_CACHE_TTL"
|
||||||
dnsblAbuseIPDBTimeout = "SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS"
|
dnsblAbuseIPDBTimeout = "SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS"
|
||||||
@@ -218,7 +223,7 @@ const (
|
|||||||
// instance is an SWW_WAF_INSTANCE_NAME that is a valid app name too, and
|
// instance is an SWW_WAF_INSTANCE_NAME that is a valid app name too, and
|
||||||
// remoteURL an SWW_WAF_LOG_REMOTE_URL, for the tests that send the lines.
|
// remoteURL an SWW_WAF_LOG_REMOTE_URL, for the tests that send the lines.
|
||||||
const (
|
const (
|
||||||
instance = "fsn1app1/blog"
|
instance = "host1/blog"
|
||||||
remoteURL = "syslog+udp://192.0.2.1:514"
|
remoteURL = "syslog+udp://192.0.2.1:514"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -287,6 +292,7 @@ func TestDefaults(t *testing.T) {
|
|||||||
})
|
})
|
||||||
wantLookupSettings(t, cfg, config.WAF{
|
wantLookupSettings(t, cfg, config.WAF{
|
||||||
LookupSource: defaultLookupSource, ASNAndCountryLookupTimeout: 800 * time.Millisecond,
|
LookupSource: defaultLookupSource, ASNAndCountryLookupTimeout: 800 * time.Millisecond,
|
||||||
|
AddLookupHeaders: true,
|
||||||
})
|
})
|
||||||
wantByteLimitSettings(t, cfg, config.WAF{
|
wantByteLimitSettings(t, cfg, config.WAF{
|
||||||
BytesLimitPerMinute: 0, BytesLimitPerHour: 50 << 30,
|
BytesLimitPerMinute: 0, BytesLimitPerHour: 50 << 30,
|
||||||
@@ -610,12 +616,12 @@ func TestCoreRuleSetSettings(t *testing.T) {
|
|||||||
},
|
},
|
||||||
{
|
{
|
||||||
environment{
|
environment{
|
||||||
coreRuleSetMode: config.CoreRuleSetModeDetect,
|
coreRuleSetMode: string(config.CoreRuleSetModeDetect),
|
||||||
coreRuleSetParanoiaLevel: "4",
|
coreRuleSetParanoiaLevel: "4",
|
||||||
coreRuleSetScoreThreshold: "10",
|
coreRuleSetScoreThreshold: "10",
|
||||||
coreRuleSetDisabledRules: "942100, 920350",
|
coreRuleSetDisabledRules: "942100, 920350",
|
||||||
coreRuleSetExemptPaths: "/api/, /static/",
|
coreRuleSetExemptPaths: "/api/, /static/",
|
||||||
coreRuleSetBodyInspectionSize: "32K",
|
coreRuleSetBodyInspectionSize: "4K",
|
||||||
exclusionsFile: "/etc/sww/crs-exclusions.conf",
|
exclusionsFile: "/etc/sww/crs-exclusions.conf",
|
||||||
},
|
},
|
||||||
config.WAF{
|
config.WAF{
|
||||||
@@ -624,7 +630,7 @@ func TestCoreRuleSetSettings(t *testing.T) {
|
|||||||
CoreRuleSetScoreThreshold: 10,
|
CoreRuleSetScoreThreshold: 10,
|
||||||
CoreRuleSetDisabledRules: []int{942100, 920350},
|
CoreRuleSetDisabledRules: []int{942100, 920350},
|
||||||
CoreRuleSetExemptPaths: []string{"/api/", "/static/"},
|
CoreRuleSetExemptPaths: []string{"/api/", "/static/"},
|
||||||
CoreRuleSetBodyInspectionSize: 32 << 10,
|
CoreRuleSetBodyInspectionSize: 4 << 10,
|
||||||
CoreRuleSetExclusionsFile: "/etc/sww/crs-exclusions.conf",
|
CoreRuleSetExclusionsFile: "/etc/sww/crs-exclusions.conf",
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
@@ -761,15 +767,15 @@ func TestInvalidAppsStopsTheStart(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestCoreRuleSetBodyInspectionSizeOf64K(t *testing.T) {
|
func TestCoreRuleSetBodyInspectionSizeOf8K(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
cfg := wafFromEnvironment(
|
cfg := wafFromEnvironment(
|
||||||
t,
|
t,
|
||||||
environment{coreRuleSetBodyInspectionSize: "64K"},
|
environment{coreRuleSetBodyInspectionSize: "8K"},
|
||||||
)
|
)
|
||||||
if cfg.CoreRuleSetBodyInspectionSize != 64<<10 {
|
if cfg.CoreRuleSetBodyInspectionSize != 8<<10 {
|
||||||
t.Errorf("64K read as %d", cfg.CoreRuleSetBodyInspectionSize)
|
t.Errorf("8K read as %d", cfg.CoreRuleSetBodyInspectionSize)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -818,11 +824,13 @@ func TestInvalidCoreRuleSetSettingStopsTheStart(t *testing.T) {
|
|||||||
coreRuleSetBodyInspectionSize, "128KB",
|
coreRuleSetBodyInspectionSize, "128KB",
|
||||||
`"128KB" is not a size such as 512K, 100M or 5G, or off`,
|
`"128KB" is not a size such as 512K, 100M or 5G, or off`,
|
||||||
},
|
},
|
||||||
{coreRuleSetBodyInspectionSize, "65K", `"65K" is more than 64K, the most waf ` +
|
|
||||||
`gets of a body`},
|
|
||||||
{
|
{
|
||||||
coreRuleSetBodyInspectionSize, "65537",
|
coreRuleSetBodyInspectionSize, "9K",
|
||||||
`"65537" is more than 64K, the most waf gets of a body`,
|
`"9K" is more than 8K, the most the Core Rule Set can read in time`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
coreRuleSetBodyInspectionSize, "8193",
|
||||||
|
`"8193" is more than 8K, the most the Core Rule Set can read in time`,
|
||||||
},
|
},
|
||||||
} {
|
} {
|
||||||
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
|
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
|
||||||
@@ -844,11 +852,11 @@ func TestInstanceNameAndLoggedHeadersAsSet(t *testing.T) {
|
|||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
cfg := wafFromEnvironment(t, environment{
|
cfg := wafFromEnvironment(t, environment{
|
||||||
instanceName: "fsn1app1/blog",
|
instanceName: "host1/blog",
|
||||||
logRequestHeaders: " Accept , X-Custom",
|
logRequestHeaders: " Accept , X-Custom",
|
||||||
})
|
})
|
||||||
|
|
||||||
if cfg.InstanceName != "fsn1app1/blog" ||
|
if cfg.InstanceName != "host1/blog" ||
|
||||||
!slices.Equal(
|
!slices.Equal(
|
||||||
cfg.LogRequestHeaders,
|
cfg.LogRequestHeaders,
|
||||||
[]string{"accept", "x-custom"},
|
[]string{"accept", "x-custom"},
|
||||||
@@ -858,6 +866,32 @@ func TestInstanceNameAndLoggedHeadersAsSet(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// config and rules check header names with httpguts.ValidHeaderFieldName.
|
||||||
|
// It takes a name of one or more of the characters RFC 9110 allows in one,
|
||||||
|
// as the check it replaced did: letters, digits and these marks.
|
||||||
|
func TestHeaderNameCheckTakesTheRFC9110Characters(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const marks = "!#$%&'*+-.^_`|~"
|
||||||
|
|
||||||
|
if httpguts.ValidHeaderFieldName("") {
|
||||||
|
t.Error("the empty name is taken")
|
||||||
|
}
|
||||||
|
|
||||||
|
for b := range 256 {
|
||||||
|
char := byte(b)
|
||||||
|
want := 'a' <= char && char <= 'z' || 'A' <= char && char <= 'Z' ||
|
||||||
|
'0' <= char && char <= '9' || strings.IndexByte(marks, char) >= 0
|
||||||
|
|
||||||
|
// string(char) would be the UTF-8 of U+0080 to U+00FF, two bytes,
|
||||||
|
// for a byte above 0x7f.
|
||||||
|
name := string([]byte{char})
|
||||||
|
if got := httpguts.ValidHeaderFieldName(name); got != want {
|
||||||
|
t.Errorf("the name %q is taken: %t, want %t", name, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRemoteLogSettingsDefaults(t *testing.T) {
|
func TestRemoteLogSettingsDefaults(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -1021,14 +1055,14 @@ func TestInstanceNameNotAnAppNameStopsTheStartOnlyWhileSending(
|
|||||||
) {
|
) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
const spaced = "fsn1 app1"
|
const spaced = "host1 blog"
|
||||||
|
|
||||||
sending := environment{logRemoteURL: remoteURL, instanceName: spaced}
|
sending := environment{logRemoteURL: remoteURL, instanceName: spaced}
|
||||||
|
|
||||||
_, err := config.WAFFromEnvironment(sending.viper())
|
_, err := config.WAFFromEnvironment(sending.viper())
|
||||||
|
|
||||||
want := logRemoteAppName + `: is unset, and ` + instanceName +
|
want := logRemoteAppName + `: is unset, and ` + instanceName +
|
||||||
` "fsn1 app1", its default, is not 1 to 48 printable ASCII characters ` +
|
` "host1 blog", its default, is not 1 to 48 printable ASCII characters ` +
|
||||||
`without a space, such as blog`
|
`without a space, such as blog`
|
||||||
if err == nil || err.Error() != want {
|
if err == nil || err.Error() != want {
|
||||||
t.Errorf("error %v, want %s", err, want)
|
t.Errorf("error %v, want %s", err, want)
|
||||||
@@ -1330,13 +1364,13 @@ func TestInstanceNameWithAControlCharacterStopsTheStartOnlyWithNtfySet(
|
|||||||
) {
|
) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
const name = "fsn1app1\r"
|
const name = "host1\r"
|
||||||
|
|
||||||
_, err := config.WAFFromEnvironment(environment{
|
_, err := config.WAFFromEnvironment(environment{
|
||||||
instanceName: name, alertNtfyURL: "https://ntfy.example/sww-alerts",
|
instanceName: name, alertNtfyURL: "https://ntfy.example/sww-alerts",
|
||||||
}.viper())
|
}.viper())
|
||||||
|
|
||||||
want := instanceName + `: "fsn1app1\r" holds a control character, such as the ` +
|
want := instanceName + `: "host1\r" holds a control character, such as the ` +
|
||||||
`carriage return of a Windows line end, and is sent to ntfy in a header ` +
|
`carriage return of a Windows line end, and is sent to ntfy in a header ` +
|
||||||
`while ` + alertNtfyURL + ` is set`
|
`while ` + alertNtfyURL + ` is set`
|
||||||
if err == nil || err.Error() != want {
|
if err == nil || err.Error() != want {
|
||||||
@@ -1395,13 +1429,13 @@ func TestLookupSettingsAsSet(t *testing.T) {
|
|||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
cfg := wafFromEnvironment(t, environment{
|
cfg := wafFromEnvironment(t, environment{
|
||||||
asnAndCountryLookupTimeout: "500", addLookupHeaders: enabled,
|
asnAndCountryLookupTimeout: "500", addLookupHeaders: disabled,
|
||||||
})
|
})
|
||||||
wantLookupSettings(t, cfg, config.WAF{
|
wantLookupSettings(t, cfg, config.WAF{
|
||||||
LookupSource: defaultLookupSource, ASNAndCountryLookupTimeout: 500 * time.Millisecond,
|
LookupSource: defaultLookupSource, ASNAndCountryLookupTimeout: 500 * time.Millisecond,
|
||||||
AddLookupHeaders: true,
|
|
||||||
})
|
})
|
||||||
|
|
||||||
|
// With no client looked up, the headers are off unless set.
|
||||||
cfg = wafFromEnvironment(t, environment{lookupSource: off})
|
cfg = wafFromEnvironment(t, environment{lookupSource: off})
|
||||||
wantLookupSettings(t, cfg, config.WAF{
|
wantLookupSettings(t, cfg, config.WAF{
|
||||||
LookupSource: off, ASNAndCountryLookupTimeout: 800 * time.Millisecond,
|
LookupSource: off, ASNAndCountryLookupTimeout: 800 * time.Millisecond,
|
||||||
@@ -1422,6 +1456,10 @@ func TestLookupDBPathGoesWithTheFileSourceAlone(t *testing.T) {
|
|||||||
cfg.LookupSource, cfg.LookupDBPath, path)
|
cfg.LookupSource, cfg.LookupDBPath, path)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if !cfg.AddLookupHeaders {
|
||||||
|
t.Error("headers off by default with lookups from file, want on")
|
||||||
|
}
|
||||||
|
|
||||||
for _, tc := range []struct {
|
for _, tc := range []struct {
|
||||||
env environment
|
env environment
|
||||||
want string
|
want string
|
||||||
@@ -1684,7 +1722,7 @@ func TestTagSettingsAsSet(t *testing.T) {
|
|||||||
tagLimitPercent: "vpn=mullvad :50, asn=014061:25, COUNTRY=cn:1, dnsbl=zen.example:99",
|
tagLimitPercent: "vpn=mullvad :50, asn=014061:25, COUNTRY=cn:1, dnsbl=zen.example:99",
|
||||||
})
|
})
|
||||||
|
|
||||||
wantLists := []config.TagList{
|
wantLists := []reputation.TagList{
|
||||||
{Name: torExitTag, URL: torURL},
|
{Name: torExitTag, URL: torURL},
|
||||||
{Name: "vpn=mullvad", URL: "https://example.com/mullvad.txt"},
|
{Name: "vpn=mullvad", URL: "https://example.com/mullvad.txt"},
|
||||||
}
|
}
|
||||||
@@ -2023,8 +2061,9 @@ func TestReputationSettingsAsSet(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
for _, tc := range []struct {
|
for _, tc := range []struct {
|
||||||
value, action string
|
value string
|
||||||
percent int64
|
action config.ReputationAction
|
||||||
|
percent int64
|
||||||
}{
|
}{
|
||||||
{actionDeny, actionDeny, 0},
|
{actionDeny, actionDeny, 0},
|
||||||
{actionLog, actionLog, 0},
|
{actionLog, actionLog, 0},
|
||||||
@@ -2157,7 +2196,7 @@ func TestDNSBLSettingsAsSet(t *testing.T) {
|
|||||||
// want are the zones, resolver, action and percent config.WAF gives.
|
// want are the zones, resolver, action and percent config.WAF gives.
|
||||||
want []string
|
want []string
|
||||||
wantResolver string
|
wantResolver string
|
||||||
wantAction string
|
wantAction config.ReputationAction
|
||||||
wantPercent int64
|
wantPercent int64
|
||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
@@ -2277,25 +2316,6 @@ func TestInvalidDNSBLSettingStopsTheStartSayingWhatIsWrong(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestMaskZoneKeyMasksTheFirstLabelOfAZoneUnderDqSpamhausNet(
|
|
||||||
t *testing.T,
|
|
||||||
) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
for zone, want := range map[string]string{
|
|
||||||
spamhaus: spamhausMasked,
|
|
||||||
spamhaus + ".": spamhausMasked + ".",
|
|
||||||
"KEY.ZEN.DQ.SPAMHAUS.NET": "********.ZEN.DQ.SPAMHAUS.NET",
|
|
||||||
dronebl: dronebl,
|
|
||||||
"dq.spamhaus.net": "dq.spamhaus.net",
|
|
||||||
spamhaus + ".example": spamhaus + ".example",
|
|
||||||
} {
|
|
||||||
if got := config.MaskZoneKey(zone); got != want {
|
|
||||||
t.Errorf("MaskZoneKey(%q) is %q, want %q", zone, got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestDNSBLZoneKeyIsLoggedMaskedAndNeverShown(t *testing.T) {
|
func TestDNSBLZoneKeyIsLoggedMaskedAndNeverShown(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -2392,6 +2412,75 @@ func TestAbuseIPDBSettingsAsSet(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestAbuseIPDBBlacklistSettingsAsSet(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
cfg := wafFromEnvironment(t, environment{})
|
||||||
|
if cfg.AbuseIPDBThreshold != 90 ||
|
||||||
|
cfg.AbuseIPDBBlacklistRefresh != time.Hour ||
|
||||||
|
cfg.AbuseIPDBBanDuration != 30*24*time.Hour {
|
||||||
|
t.Errorf(
|
||||||
|
"by default, the threshold %d, the refresh %s and the ban %s, "+
|
||||||
|
"want 90, 1h and 720h",
|
||||||
|
cfg.AbuseIPDBThreshold,
|
||||||
|
cfg.AbuseIPDBBlacklistRefresh,
|
||||||
|
cfg.AbuseIPDBBanDuration,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg = wafFromEnvironment(t, environment{
|
||||||
|
abuseIPDBThreshold: "25", abuseIPDBBlacklistRefresh: "1440",
|
||||||
|
abuseIPDBBanDays: "1",
|
||||||
|
})
|
||||||
|
if cfg.AbuseIPDBThreshold != 25 ||
|
||||||
|
cfg.AbuseIPDBBlacklistRefresh != 24*time.Hour ||
|
||||||
|
cfg.AbuseIPDBBanDuration != 24*time.Hour {
|
||||||
|
t.Errorf(
|
||||||
|
"set, the threshold %d, the refresh %s and the ban %s, "+
|
||||||
|
"want 25, 24h and 24h",
|
||||||
|
cfg.AbuseIPDBThreshold,
|
||||||
|
cfg.AbuseIPDBBlacklistRefresh,
|
||||||
|
cfg.AbuseIPDBBanDuration,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestInvalidAbuseIPDBBlacklistSettingStopsTheStartSayingWhatIsWrong(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const (
|
||||||
|
notThreshold = " is not a whole number from 25 to 100, the scores " +
|
||||||
|
"AbuseIPDB's blacklist can be cut at"
|
||||||
|
notWhole = " is not a whole number above zero, such as 30, that fits " +
|
||||||
|
"a duration"
|
||||||
|
)
|
||||||
|
|
||||||
|
for _, tc := range []struct{ name, value, want string }{
|
||||||
|
{abuseIPDBThreshold, "24", `"24"` + notThreshold},
|
||||||
|
{abuseIPDBThreshold, "101", `"101"` + notThreshold},
|
||||||
|
{abuseIPDBThreshold, off, `"off"` + notThreshold},
|
||||||
|
{abuseIPDBBlacklistRefresh, "0", `"0"` + notWhole},
|
||||||
|
{abuseIPDBBlacklistRefresh, "1h", `"1h"` + notWhole},
|
||||||
|
{abuseIPDBBanDays, "-1", `"-1"` + notWhole},
|
||||||
|
{abuseIPDBBanDays, "200000", `"200000"` + notWhole},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
_, err := config.WAFFromEnvironment(
|
||||||
|
environment{tc.name: tc.value}.viper(),
|
||||||
|
)
|
||||||
|
|
||||||
|
want := tc.name + ": " + tc.want
|
||||||
|
if err == nil || err.Error() != want {
|
||||||
|
t.Errorf("error %v, want %s", err, want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestInvalidAbuseIPDBSettingStopsTheStartSayingWhatIsWrong(
|
func TestInvalidAbuseIPDBSettingStopsTheStartSayingWhatIsWrong(
|
||||||
t *testing.T,
|
t *testing.T,
|
||||||
) {
|
) {
|
||||||
@@ -2927,7 +3016,7 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
|||||||
lookupSource: defaultLookupSource,
|
lookupSource: defaultLookupSource,
|
||||||
lookupDBPath: "",
|
lookupDBPath: "",
|
||||||
asnAndCountryLookupTimeout: "800",
|
asnAndCountryLookupTimeout: "800",
|
||||||
addLookupHeaders: disabled,
|
addLookupHeaders: enabled,
|
||||||
allowedCountries: "",
|
allowedCountries: "",
|
||||||
tagLists: "",
|
tagLists: "",
|
||||||
tagDeny: "",
|
tagDeny: "",
|
||||||
@@ -2944,6 +3033,9 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
|||||||
abuseIPDBKey: "",
|
abuseIPDBKey: "",
|
||||||
abuseIPDBMinScore: "75",
|
abuseIPDBMinScore: "75",
|
||||||
abuseIPDBDailyBudget: "900",
|
abuseIPDBDailyBudget: "900",
|
||||||
|
abuseIPDBThreshold: "90",
|
||||||
|
abuseIPDBBlacklistRefresh: "60",
|
||||||
|
abuseIPDBBanDays: "30",
|
||||||
dnsblAbuseIPDBAction: "limit:25",
|
dnsblAbuseIPDBAction: "limit:25",
|
||||||
ipReputationCacheTTL: defaultIPReputationCacheTTL,
|
ipReputationCacheTTL: defaultIPReputationCacheTTL,
|
||||||
dnsblAbuseIPDBTimeout: "2000",
|
dnsblAbuseIPDBTimeout: "2000",
|
||||||
@@ -2967,7 +3059,7 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
|||||||
logLevel: "info",
|
logLevel: "info",
|
||||||
ruleFilesDir: "/etc/sww/rules.d",
|
ruleFilesDir: "/etc/sww/rules.d",
|
||||||
ruleFilesEnabled: "true",
|
ruleFilesEnabled: "true",
|
||||||
coreRuleSetMode: config.CoreRuleSetModeBlock,
|
coreRuleSetMode: string(config.CoreRuleSetModeBlock),
|
||||||
coreRuleSetParanoiaLevel: "1",
|
coreRuleSetParanoiaLevel: "1",
|
||||||
coreRuleSetScoreThreshold: "5",
|
coreRuleSetScoreThreshold: "5",
|
||||||
coreRuleSetDisabledRules: defaultCoreRuleSetDisabledRules,
|
coreRuleSetDisabledRules: defaultCoreRuleSetDisabledRules,
|
||||||
|
|||||||
@@ -481,12 +481,21 @@ func (c *CoreRuleSet) Inspect(
|
|||||||
return Result{RuleIDs: ids, Score: score(tx)}, read, err
|
return Result{RuleIDs: ids, Score: score(tx)}, read, err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// mostJSONRead is the largest JSON body the Core Rule Set reads, even while
|
||||||
|
// SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is larger. Coraza names each
|
||||||
|
// value in a JSON body by the whole path to it, so what it gives the Core
|
||||||
|
// Rule Set to read can outgrow the body: up to 8 times its size, and up to
|
||||||
|
// 4 KiB for a small body. 1 KiB of JSON thus gives at most 8 KiB, no more
|
||||||
|
// than the largest body of another kind.
|
||||||
|
const mostJSONRead = 1 << 10
|
||||||
|
|
||||||
// readBody reads body, the body of the request in tx, which has run on
|
// readBody reads body, the body of the request in tx, which has run on
|
||||||
// the request's headers, while SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is
|
// the request's headers, while SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is
|
||||||
// set and the body is of a kind the Core Rule Set reads: form data and
|
// set and the body is of a kind the Core Rule Set reads: form data and
|
||||||
// multipart, of which it reads the first c.bodyInspectionSize bytes, and
|
// multipart, of which it reads the first c.bodyInspectionSize bytes, and
|
||||||
// JSON and XML, which it reads only when they are no longer than that,
|
// JSON and XML, which it reads only when they are no longer than that,
|
||||||
// since they cannot be read in part.
|
// since they cannot be read in part. The Core Rule Set reads no JSON body
|
||||||
|
// longer than mostJSONRead either.
|
||||||
// readBody reads one byte past c.bodyInspectionSize, to tell which they
|
// readBody reads one byte past c.bodyInspectionSize, to tell which they
|
||||||
// are, gives the Core Rule Set what it reads, and returns what it read and
|
// are, gives the Core Rule Set what it reads, and returns what it read and
|
||||||
// the error that ended the reading early, if one did.
|
// the error that ended the reading early, if one did.
|
||||||
@@ -499,6 +508,8 @@ func (c *CoreRuleSet) readBody(
|
|||||||
}
|
}
|
||||||
|
|
||||||
inPart := false
|
inPart := false
|
||||||
|
// largest is the largest JSON or XML body the Core Rule Set reads.
|
||||||
|
largest := c.bodyInspectionSize
|
||||||
// How the body is read is a variable of the transaction, which only
|
// How the body is read is a variable of the transaction, which only
|
||||||
// Coraza's interface for plugins reads.
|
// Coraza's interface for plugins reads.
|
||||||
state := tx.(plugintypes.TransactionState) //nolint:forcetypeassert // every one is
|
state := tx.(plugintypes.TransactionState) //nolint:forcetypeassert // every one is
|
||||||
@@ -506,14 +517,16 @@ func (c *CoreRuleSet) readBody(
|
|||||||
switch state.Variables().RequestBodyProcessor().Get() {
|
switch state.Variables().RequestBodyProcessor().Get() {
|
||||||
case "URLENCODED", "MULTIPART":
|
case "URLENCODED", "MULTIPART":
|
||||||
inPart = true
|
inPart = true
|
||||||
case "JSON", "XML":
|
case "JSON":
|
||||||
|
largest = min(largest, mostJSONRead)
|
||||||
|
case "XML":
|
||||||
default:
|
default:
|
||||||
return nil, nil
|
return nil, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
read, err := io.ReadAll(io.LimitReader(body, c.bodyInspectionSize+1))
|
read, err := io.ReadAll(io.LimitReader(body, c.bodyInspectionSize+1))
|
||||||
|
|
||||||
if inPart || (err == nil && int64(len(read)) <= c.bodyInspectionSize) {
|
if inPart || (err == nil && int64(len(read)) <= largest) {
|
||||||
// Coraza holds what it reads of the body in memory, up to the
|
// Coraza holds what it reads of the body in memory, up to the
|
||||||
// limit, so this cannot fail.
|
// limit, so this cannot fail.
|
||||||
_, _, _ = tx.WriteRequestBody(read)
|
_, _, _ = tx.WriteRequestBody(read)
|
||||||
|
|||||||
@@ -390,22 +390,15 @@ func TestMoreParametersThanCorazaKeepsIsAMatch(t *testing.T) {
|
|||||||
matched(900300),
|
matched(900300),
|
||||||
)
|
)
|
||||||
|
|
||||||
// So it is with the fields of a form data or JSON body.
|
// So it is with the fields of a form data body. A JSON body the Core
|
||||||
|
// Rule Set reads, of at most mostJSONRead bytes, cannot hold that many.
|
||||||
crs = readingBodies(t)
|
crs = readingBodies(t)
|
||||||
|
|
||||||
for _, tc := range []struct{ header, body string }{
|
body := strings.Repeat("a=1&", 999) + attack
|
||||||
{formData, strings.Repeat("a=1&", 999) + attack},
|
wantBody(t, crs, post(formData), body, matched(942100), body)
|
||||||
{jsonBody, `{"a":[` + strings.Repeat("1,", 998) + `1],"id":"` + injection + `"}`},
|
|
||||||
} {
|
|
||||||
wantBody(t, crs, post(tc.header), tc.body, matched(942100), tc.body)
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tc := range []struct{ header, body string }{
|
body = strings.Repeat("a=1&", 1000) + attack
|
||||||
{formData, strings.Repeat("a=1&", 1000) + attack},
|
wantBody(t, crs, post(formData), body, matched(900300), body)
|
||||||
{jsonBody, `{"a":[` + strings.Repeat("1,", 999) + `1],"id":"` + injection + `"}`},
|
|
||||||
} {
|
|
||||||
wantBody(t, crs, post(tc.header), tc.body, matched(900300), tc.body)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRedirectURIMayNameALocalAddress(t *testing.T) {
|
func TestRedirectURIMayNameALocalAddress(t *testing.T) {
|
||||||
@@ -1512,13 +1505,15 @@ const (
|
|||||||
wpTags = "/wp-json/wp/v2/tags"
|
wpTags = "/wp-json/wp/v2/tags"
|
||||||
)
|
)
|
||||||
|
|
||||||
// readingBodiesUpTo64K returns the Core Rule Set as sww runs it by default,
|
// readingBodiesUpTo8K returns the Core Rule Set as sww runs it by default,
|
||||||
// but reading bodies up to the largest SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE.
|
// but reading bodies up to the largest SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE.
|
||||||
func readingBodiesUpTo64K(t *testing.T) *coreruleset.CoreRuleSet {
|
func readingBodiesUpTo8K(t *testing.T) *coreruleset.CoreRuleSet {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
return shared(t, coreruleset.Params{
|
return shared(t, coreruleset.Params{
|
||||||
ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, BodyInspectionSize: 64<<10 - 1,
|
ParanoiaLevel: 1,
|
||||||
|
DisabledRules: defaultDisabledRules,
|
||||||
|
BodyInspectionSize: mostBodyRead,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1582,7 +1577,7 @@ func TestWordPressAttacksAreRefused(t *testing.T) {
|
|||||||
|
|
||||||
const threshold = 5 // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default
|
const threshold = 5 // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD's default
|
||||||
|
|
||||||
defaults, bodies := atDefaults(t), readingBodiesUpTo64K(t)
|
defaults, bodies := atDefaults(t), readingBodiesUpTo8K(t)
|
||||||
|
|
||||||
for _, attack := range []string{
|
for _, attack := range []string{
|
||||||
sqlInjection,
|
sqlInjection,
|
||||||
@@ -1693,7 +1688,7 @@ func wordpressRESTAddresses(route string) []string {
|
|||||||
func TestWordPressExclusionsFollowEveryRESTAddress(t *testing.T) {
|
func TestWordPressExclusionsFollowEveryRESTAddress(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
crs := readingBodiesUpTo64K(t)
|
crs := readingBodiesUpTo8K(t)
|
||||||
content := `{"content":"` + wordpressBlock + `"}`
|
content := `{"content":"` + wordpressBlock + `"}`
|
||||||
|
|
||||||
for _, route := range []string{
|
for _, route := range []string{
|
||||||
@@ -1919,7 +1914,7 @@ func TestWordPressCommandAfterASeparatorIsRefusedOnSearchesAndNames(
|
|||||||
wantResult(t, crs, get(target+command), matched(932235))
|
wantResult(t, crs, get(target+command), matched(932235))
|
||||||
}
|
}
|
||||||
|
|
||||||
bodies := readingBodiesUpTo64K(t)
|
bodies := readingBodiesUpTo8K(t)
|
||||||
|
|
||||||
for _, p := range wordpressBodyFields(command) {
|
for _, p := range wordpressBodyFields(command) {
|
||||||
r := request{http.MethodPost, p.target, []string{p.contentType}}
|
r := request{http.MethodPost, p.target, []string{p.contentType}}
|
||||||
@@ -2143,7 +2138,7 @@ func TestWordPressBodyFieldsAreCheckedInTheQuery(t *testing.T) {
|
|||||||
func TestWordPressBodyNamesAndAddressesAreExcludedOnlyThere(t *testing.T) {
|
func TestWordPressBodyNamesAndAddressesAreExcludedOnlyThere(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
crs := readingBodiesUpTo64K(t)
|
crs := readingBodiesUpTo8K(t)
|
||||||
|
|
||||||
for _, v := range wordpressWords() {
|
for _, v := range wordpressWords() {
|
||||||
for _, field := range []string{"title", "excerpt", "name"} {
|
for _, field := range []string{"title", "excerpt", "name"} {
|
||||||
@@ -2262,7 +2257,7 @@ func TestWordPressExclusionsAreNarrow(t *testing.T) {
|
|||||||
length = "Content-Length: 2"
|
length = "Content-Length: 2"
|
||||||
)
|
)
|
||||||
|
|
||||||
crs := readingBodiesUpTo64K(t)
|
crs := readingBodiesUpTo8K(t)
|
||||||
saving := request{
|
saving := request{
|
||||||
http.MethodPost,
|
http.MethodPost,
|
||||||
wpPost,
|
wpPost,
|
||||||
@@ -2473,19 +2468,35 @@ func TestFormDataAndMultipartAreReadUpToTheLimit(t *testing.T) {
|
|||||||
func TestJSONAndXMLAreReadOnlyWhenNoLargerThanTheLimit(t *testing.T) {
|
func TestJSONAndXMLAreReadOnlyWhenNoLargerThanTheLimit(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
crs := readingBodies(t)
|
// The limit is the body inspection size, but at most mostJSONRead for
|
||||||
|
// JSON. smallSize is a body inspection size under mostJSONRead.
|
||||||
|
const smallSize = 512
|
||||||
|
|
||||||
for _, tc := range []struct{ header, head, tail string }{
|
bodies, smallBodies := readingBodies(t), shared(t, coreruleset.Params{
|
||||||
{jsonBody, `{"q":"` + injection + `","pad":"`, `"}`},
|
ParanoiaLevel: 1, DisabledRules: defaultDisabledRules,
|
||||||
{xmlBody, "<r><q>" + injection + "</q><pad>", "</pad></r>"},
|
BodyInspectionSize: smallSize,
|
||||||
|
})
|
||||||
|
jsonHead, jsonTail := `{"q":"`+injection+`","pad":"`, `"}`
|
||||||
|
|
||||||
|
for _, tc := range []struct {
|
||||||
|
crs *coreruleset.CoreRuleSet
|
||||||
|
header, head, tail string
|
||||||
|
limit int
|
||||||
|
}{
|
||||||
|
{bodies, jsonBody, jsonHead, jsonTail, mostJSONRead},
|
||||||
|
{smallBodies, jsonBody, jsonHead, jsonTail, smallSize},
|
||||||
|
{
|
||||||
|
bodies, xmlBody, "<r><q>" + injection + "</q><pad>", "</pad></r>",
|
||||||
|
bodyInspectionSize,
|
||||||
|
},
|
||||||
} {
|
} {
|
||||||
fits := padded(tc.head, tc.tail, bodyInspectionSize)
|
fits := padded(tc.head, tc.tail, tc.limit)
|
||||||
wantBody(t, crs, post(tc.header), fits, matched(942100), fits)
|
wantBody(t, tc.crs, post(tc.header), fits, matched(942100), fits)
|
||||||
|
|
||||||
larger := padded(tc.head, tc.tail, bodyInspectionSize+1)
|
larger := padded(tc.head, tc.tail, tc.limit+1)
|
||||||
wantBody(
|
wantBody(
|
||||||
t,
|
t,
|
||||||
crs,
|
tc.crs,
|
||||||
post(tc.header),
|
post(tc.header),
|
||||||
larger,
|
larger,
|
||||||
coreruleset.Result{},
|
coreruleset.Result{},
|
||||||
|
|||||||
@@ -0,0 +1,437 @@
|
|||||||
|
package coreruleset_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"maps"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"net/netip"
|
||||||
|
"regexp"
|
||||||
|
"runtime"
|
||||||
|
"slices"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"golang.org/x/sys/unix"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The bounds on what a visitor can send, as SPEC.md gives them.
|
||||||
|
const (
|
||||||
|
// lineLimit is the longest request line or header line.
|
||||||
|
lineLimit = 8 << 10
|
||||||
|
// headerLimit is the most bytes rproxy reads of a request line and
|
||||||
|
// headers together.
|
||||||
|
headerLimit = 32 << 10
|
||||||
|
// mostCookies and mostHeaderFields are waf's limits on cookies and header
|
||||||
|
// fields.
|
||||||
|
mostCookies = 50
|
||||||
|
mostHeaderFields = 100
|
||||||
|
// mostBodyRead is the largest SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE.
|
||||||
|
mostBodyRead = 8 << 10
|
||||||
|
// mostJSONRead is the largest JSON body the Core Rule Set reads.
|
||||||
|
mostJSONRead = 1 << 10
|
||||||
|
)
|
||||||
|
|
||||||
|
// waf refuses a request the Core Rule Set has not inspected within 500
|
||||||
|
// milliseconds, but Coraza cannot stop an inspection part-way, so it runs to
|
||||||
|
// its end with nobody waiting for it. The test checks that an inspection ends
|
||||||
|
// within about 1 second, so that a visitor cannot keep a processor busy for
|
||||||
|
// longer with one request.
|
||||||
|
//
|
||||||
|
// A busy host slows all work, so the test does not compare an inspection with
|
||||||
|
// 1 second. It compares it with referenceTime, timed on the same thread just
|
||||||
|
// before and after. inspectionTimeLimitMultiple is how many times
|
||||||
|
// referenceTime an inspection may take.
|
||||||
|
//
|
||||||
|
// On the host that runs our checks, referenceTime took at least 78
|
||||||
|
// milliseconds, so 13 times it is about 1 second. With that host fully busy,
|
||||||
|
// no try of the slowest request took more than 9.5 times referenceTime.
|
||||||
|
const inspectionTimeLimitMultiple = 13
|
||||||
|
|
||||||
|
// referencePattern is what referenceTime matches. Like the Core Rule Set's
|
||||||
|
// rules, it is a regular expression with alternatives and repeats.
|
||||||
|
var referencePattern = regexp.MustCompile(
|
||||||
|
`(?i)(union|select|insert|delete)\s+\w+\s+(from|into)\s+\w+[0-9]+;`,
|
||||||
|
)
|
||||||
|
|
||||||
|
// referenceTime returns the processor time the calling thread takes to do a
|
||||||
|
// fixed amount of work unrelated to the Core Rule Set, so that a change to the
|
||||||
|
// rules does not change it. referencePattern matches nowhere in the text, so
|
||||||
|
// each match reads all of it.
|
||||||
|
func referenceTime(t *testing.T) time.Duration {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
referenceText := text(4 << 10)
|
||||||
|
began := threadCPUTime(t)
|
||||||
|
|
||||||
|
for range 200 {
|
||||||
|
if referencePattern.MatchString(referenceText) {
|
||||||
|
t.Fatal("the reference pattern matched")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return threadCPUTime(t) - began
|
||||||
|
}
|
||||||
|
|
||||||
|
// text returns n bytes of letters, digits and spaces, which the Core Rule Set
|
||||||
|
// takes longer over than one letter repeated.
|
||||||
|
func text(n int) string {
|
||||||
|
const words = "select 0x2f from tbl_1 where name like abc_9 "
|
||||||
|
|
||||||
|
return strings.Repeat(words, n/len(words)+1)[:n]
|
||||||
|
}
|
||||||
|
|
||||||
|
// visitorRequest is a request a visitor can send, at the most of one or more
|
||||||
|
// of the bounds above.
|
||||||
|
type visitorRequest struct {
|
||||||
|
name string
|
||||||
|
target string
|
||||||
|
// header holds the headers besides Host.
|
||||||
|
header http.Header
|
||||||
|
// contentType is the body's type, "" for no body.
|
||||||
|
contentType string
|
||||||
|
body string
|
||||||
|
}
|
||||||
|
|
||||||
|
// longestTarget returns a target whose request line, "POST <target>
|
||||||
|
// HTTP/1.1", is lineLimit bytes long.
|
||||||
|
func longestTarget() string {
|
||||||
|
const around = len("POST /?q= HTTP/1.1")
|
||||||
|
|
||||||
|
return "/?q=" + strings.ReplaceAll(text(lineLimit-around), " ", "+")
|
||||||
|
}
|
||||||
|
|
||||||
|
// cookieLine returns a Cookie value whose line, "Cookie: <value>", is
|
||||||
|
// lineLimit bytes long, made of count cookies.
|
||||||
|
func cookieLine(count int) string {
|
||||||
|
const linePrefix = len("Cookie: ")
|
||||||
|
|
||||||
|
cookies := make([]string, count)
|
||||||
|
for i := range cookies {
|
||||||
|
cookies[i] = "c" + strconv.Itoa(i) + "="
|
||||||
|
}
|
||||||
|
|
||||||
|
// Each cookie is followed by "; ", but for the last.
|
||||||
|
room := lineLimit - linePrefix - len(strings.Join(cookies, "; "))
|
||||||
|
for i := range cookies {
|
||||||
|
share := room / count
|
||||||
|
if i == 0 {
|
||||||
|
share += room % count
|
||||||
|
}
|
||||||
|
|
||||||
|
cookies[i] += strings.ReplaceAll(text(share), " ", "_")
|
||||||
|
}
|
||||||
|
|
||||||
|
return strings.Join(cookies, "; ")
|
||||||
|
}
|
||||||
|
|
||||||
|
// headerValue returns a value whose line, "<name>: <value>", is n bytes long.
|
||||||
|
func headerValue(name string, n int) string {
|
||||||
|
return text(n - len(name) - len(": "))
|
||||||
|
}
|
||||||
|
|
||||||
|
// formFields returns a form body of mostBodyRead bytes holding count fields.
|
||||||
|
func formFields(count int) string {
|
||||||
|
fields := make([]string, count)
|
||||||
|
for i := range fields {
|
||||||
|
fields[i] = "f" + strconv.Itoa(i) + "="
|
||||||
|
}
|
||||||
|
|
||||||
|
room := mostBodyRead - len(strings.Join(fields, "&"))
|
||||||
|
for i := range fields {
|
||||||
|
share := room / count
|
||||||
|
if i == 0 {
|
||||||
|
share += room % count
|
||||||
|
}
|
||||||
|
|
||||||
|
fields[i] += strings.ReplaceAll(text(share), " ", "+")
|
||||||
|
}
|
||||||
|
|
||||||
|
return strings.Join(fields, "&")
|
||||||
|
}
|
||||||
|
|
||||||
|
// multipartBody returns a multipart body of mostBodyRead bytes, with the
|
||||||
|
// boundary "b", holding fields of about size bytes each.
|
||||||
|
func multipartBody(size int) string {
|
||||||
|
var body strings.Builder
|
||||||
|
|
||||||
|
const end = "--b--\r\n"
|
||||||
|
|
||||||
|
for i := 0; ; i++ {
|
||||||
|
head := "--b\r\nContent-Disposition: form-data; name=\"f" +
|
||||||
|
strconv.Itoa(i) + "\"\r\n\r\n"
|
||||||
|
|
||||||
|
room := mostBodyRead - body.Len() - len(
|
||||||
|
end,
|
||||||
|
) - len(
|
||||||
|
head,
|
||||||
|
) - len(
|
||||||
|
"\r\n",
|
||||||
|
)
|
||||||
|
if room <= 0 {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
body.WriteString(head + text(min(size, room)) + "\r\n")
|
||||||
|
}
|
||||||
|
|
||||||
|
body.WriteString(end)
|
||||||
|
|
||||||
|
return body.String()
|
||||||
|
}
|
||||||
|
|
||||||
|
// jsonArrayOfMostElements returns a JSON array of at most mostJSONRead bytes
|
||||||
|
// holding as many elements as fit, each the number 1.
|
||||||
|
func jsonArrayOfMostElements() string {
|
||||||
|
return "[1" + strings.Repeat(
|
||||||
|
",1",
|
||||||
|
(mostJSONRead-len("[1]"))/len(",1"),
|
||||||
|
) + "]"
|
||||||
|
}
|
||||||
|
|
||||||
|
// jsonOfMostElementsUnderAKey returns a JSON body of mostJSONRead bytes
|
||||||
|
// holding as many elements as fit, each the number 1, in an array under a
|
||||||
|
// key of 6 bytes. Coraza names each element "json.<key>.<index>", so a
|
||||||
|
// longer key makes longer names. 6 bytes is the longest key that keeps the
|
||||||
|
// names and values of this body within the 8 KiB Coraza allows.
|
||||||
|
func jsonOfMostElementsUnderAKey() string {
|
||||||
|
head := `{"` + text(6) + `":[1`
|
||||||
|
|
||||||
|
return head + strings.Repeat(
|
||||||
|
",1",
|
||||||
|
(mostJSONRead-len(head+"]}"))/len(",1"),
|
||||||
|
) + "]}"
|
||||||
|
}
|
||||||
|
|
||||||
|
// The kinds of body the Core Rule Set reads, and the header that names a
|
||||||
|
// request's cookies.
|
||||||
|
const (
|
||||||
|
formType = "application/x-www-form-urlencoded"
|
||||||
|
jsonType = "application/json"
|
||||||
|
multipartType = "multipart/form-data; boundary=b"
|
||||||
|
cookieHeader = "Cookie"
|
||||||
|
)
|
||||||
|
|
||||||
|
// fullLine is the length of each of the request line and the three or four
|
||||||
|
// header lines that together fill rproxy's limit, with room for the line
|
||||||
|
// endings and the other headers.
|
||||||
|
const fullLine = lineLimit - 64
|
||||||
|
|
||||||
|
// worstCaseRequests are, for each kind of thing a visitor sends, the requests
|
||||||
|
// that keep the Core Rule Set busiest, within the bounds above. The last
|
||||||
|
// holds the most of everything at once: Cookie lines keep it busier than
|
||||||
|
// other header lines, and the JSON body of the most elements under a key
|
||||||
|
// keeps it busier than the other bodies.
|
||||||
|
func worstCaseRequests() []visitorRequest {
|
||||||
|
fullCookieLine := cookieLine(1)[:fullLine-len("Cookie: ")]
|
||||||
|
|
||||||
|
return append(
|
||||||
|
append(worstCaseHeaders(), worstCaseBodies()...),
|
||||||
|
visitorRequest{
|
||||||
|
name: "the longest request line, three Cookie lines and body",
|
||||||
|
target: longestTarget()[:fullLine-len("POST HTTP/1.1")],
|
||||||
|
header: http.Header{cookieHeader: {
|
||||||
|
fullCookieLine, fullCookieLine, fullCookieLine,
|
||||||
|
}},
|
||||||
|
contentType: jsonType, body: jsonOfMostElementsUnderAKey(),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// worstCaseHeaders are the requests of worstCaseRequests with no body.
|
||||||
|
func worstCaseHeaders() []visitorRequest {
|
||||||
|
headerFields := http.Header{}
|
||||||
|
|
||||||
|
for i := range mostHeaderFields - 1 {
|
||||||
|
name := "X-Field-" + strconv.Itoa(i)
|
||||||
|
headerFields.Set(
|
||||||
|
name,
|
||||||
|
headerValue(name, headerLimit/mostHeaderFields),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
return []visitorRequest{
|
||||||
|
{name: "the longest request line", target: longestTarget()},
|
||||||
|
{
|
||||||
|
name: "the longest request line, as query parameters",
|
||||||
|
target: "/?" + strings.Repeat(
|
||||||
|
"a=1&",
|
||||||
|
(lineLimit-len("POST /? HTTP/1.1"))/len("a=1&"),
|
||||||
|
),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "the longest Cookie line, as one cookie",
|
||||||
|
target: "/",
|
||||||
|
header: http.Header{cookieHeader: {cookieLine(1)}},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "the longest Cookie line, as the most cookies",
|
||||||
|
target: "/",
|
||||||
|
header: http.Header{cookieHeader: {cookieLine(mostCookies)}},
|
||||||
|
},
|
||||||
|
{name: "the most header fields", target: "/", header: headerFields},
|
||||||
|
{
|
||||||
|
name: "the longest header lines",
|
||||||
|
target: "/",
|
||||||
|
header: http.Header{
|
||||||
|
"User-Agent": {headerValue("User-Agent", fullLine)},
|
||||||
|
"Referer": {headerValue("Referer", fullLine)},
|
||||||
|
"Accept": {headerValue("Accept", fullLine)},
|
||||||
|
"X-Long": {headerValue("X-Long", fullLine)},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// worstCaseBodies are the requests of worstCaseRequests whose body holds
|
||||||
|
// the most the Core Rule Set reads.
|
||||||
|
func worstCaseBodies() []visitorRequest {
|
||||||
|
const nested = (mostJSONRead - len(`{"a":1}`)) / len(`{"a":}`)
|
||||||
|
|
||||||
|
return []visitorRequest{
|
||||||
|
{
|
||||||
|
name: "a form body of one field", target: "/",
|
||||||
|
contentType: formType, body: formFields(1),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "a form body of the most fields Coraza keeps", target: "/",
|
||||||
|
contentType: formType, body: formFields(1000),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "a JSON body of one string", target: "/",
|
||||||
|
contentType: jsonType,
|
||||||
|
body: `{"a":"` + text(
|
||||||
|
mostJSONRead-len(`{"a":""}`),
|
||||||
|
) + `"}`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "a JSON body of the most elements", target: "/",
|
||||||
|
contentType: jsonType, body: jsonArrayOfMostElements(),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "a JSON body of the most elements under a key", target: "/",
|
||||||
|
contentType: jsonType, body: jsonOfMostElementsUnderAKey(),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "a JSON body nested deep", target: "/",
|
||||||
|
contentType: jsonType,
|
||||||
|
body: strings.Repeat(`{"a":`, nested) + "1" +
|
||||||
|
strings.Repeat("}", nested),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "an XML body of many elements", target: "/",
|
||||||
|
contentType: "application/xml",
|
||||||
|
body: "<a>" + strings.Repeat(
|
||||||
|
"<b>x</b>",
|
||||||
|
(mostBodyRead-len("<a></a>"))/len("<b>x</b>"),
|
||||||
|
) + "</a>",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "a multipart body of one field", target: "/",
|
||||||
|
contentType: multipartType, body: multipartBody(mostBodyRead),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "a multipart body of many fields", target: "/",
|
||||||
|
contentType: multipartType, body: multipartBody(1),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// threadCPUTime returns the processor time the calling thread has used.
|
||||||
|
func threadCPUTime(t *testing.T) time.Duration {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
var now unix.Timespec
|
||||||
|
|
||||||
|
err := unix.ClockGettime(unix.CLOCK_THREAD_CPUTIME_ID, &now)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read the thread's processor time: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return time.Duration(now.Nano())
|
||||||
|
}
|
||||||
|
|
||||||
|
// inspectionTime returns the processor time crs takes to inspect tc, and the
|
||||||
|
// mean of referenceTime just before and after. It fails t unless crs reads all
|
||||||
|
// of tc's body and Coraza parses it.
|
||||||
|
//
|
||||||
|
// All of it runs on this goroutine, kept on one thread, so the thread's
|
||||||
|
// processor time measures it. Time other work holds the processor does not
|
||||||
|
// count.
|
||||||
|
func inspectionTime(
|
||||||
|
t *testing.T, crs *coreruleset.CoreRuleSet, tc visitorRequest,
|
||||||
|
) (time.Duration, time.Duration) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
req := httptest.NewRequestWithContext(
|
||||||
|
t.Context(), http.MethodPost, tc.target, nil,
|
||||||
|
)
|
||||||
|
maps.Copy(req.Header, tc.header)
|
||||||
|
|
||||||
|
if tc.contentType != "" {
|
||||||
|
req.Header.Set("Content-Type", tc.contentType)
|
||||||
|
}
|
||||||
|
|
||||||
|
runtime.LockOSThread()
|
||||||
|
|
||||||
|
defer runtime.UnlockOSThread()
|
||||||
|
|
||||||
|
before := referenceTime(t)
|
||||||
|
|
||||||
|
began := threadCPUTime(t)
|
||||||
|
result, read, _ := crs.Inspect(
|
||||||
|
req, netip.MustParseAddr("192.0.2.1"), strings.NewReader(tc.body),
|
||||||
|
)
|
||||||
|
took := threadCPUTime(t) - began
|
||||||
|
|
||||||
|
reference := (before + referenceTime(t)) / 2
|
||||||
|
|
||||||
|
if string(read) != tc.body {
|
||||||
|
t.Errorf("read %d bytes of %d", len(read), len(tc.body))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Rule 900440 matches a body Coraza could not parse.
|
||||||
|
if slices.Contains(result.RuleIDs, 900440) {
|
||||||
|
t.Errorf("Coraza could not parse the body")
|
||||||
|
}
|
||||||
|
|
||||||
|
return took, reference
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestTheCoreRuleSetInspectsEachWorstCaseRequestInTime runs one at a time, so
|
||||||
|
// that the tests running in parallel do not slow it. Other work on the host
|
||||||
|
// can still slow one try more than the reference work around it, so a request
|
||||||
|
// fails only when three tries in a row take too long.
|
||||||
|
//
|
||||||
|
//nolint:paralleltest // one at a time, as above
|
||||||
|
func TestTheCoreRuleSetInspectsEachWorstCaseRequestInTime(t *testing.T) {
|
||||||
|
const tries = 3
|
||||||
|
|
||||||
|
// The settings that make the Core Rule Set slowest: the highest paranoia
|
||||||
|
// level, no rule switched off, and the most of a body read.
|
||||||
|
crs := shared(t, coreruleset.Params{
|
||||||
|
ParanoiaLevel: 4, BodyInspectionSize: mostBodyRead,
|
||||||
|
})
|
||||||
|
|
||||||
|
for _, tc := range worstCaseRequests() {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
var took, reference time.Duration
|
||||||
|
|
||||||
|
for range tries {
|
||||||
|
took, reference = inspectionTime(t, crs, tc)
|
||||||
|
if took <= inspectionTimeLimitMultiple*reference {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Errorf(
|
||||||
|
"took %s on the last of %d tries, over %d times the %s "+
|
||||||
|
"of the reference work",
|
||||||
|
took, tries, inspectionTimeLimitMultiple, reference,
|
||||||
|
)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,90 @@
|
|||||||
|
// Package dirwatch watches a directory, and calls a function once the
|
||||||
|
// directory has gone a short time without a change after one. The rule
|
||||||
|
// files and the lookup database are read again this way.
|
||||||
|
package dirwatch
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/fsnotify/fsnotify"
|
||||||
|
)
|
||||||
|
|
||||||
|
// quietTime is how long the directory must go without a change before the
|
||||||
|
// function is called, so that a file still being written, such as one
|
||||||
|
// saved in place, appended to or copied in with scp, is read only once
|
||||||
|
// whole.
|
||||||
|
const quietTime = 2 * time.Second
|
||||||
|
|
||||||
|
// Watcher watches one directory. The directory is watched, rather than a
|
||||||
|
// file in it, so that a file replaced by a rename is seen.
|
||||||
|
type Watcher struct {
|
||||||
|
watcher *fsnotify.Watcher
|
||||||
|
}
|
||||||
|
|
||||||
|
// New starts watching dir, or returns why it cannot.
|
||||||
|
func New(dir string) (*Watcher, error) {
|
||||||
|
watcher, err := fsnotify.NewWatcher()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
err = watcher.Add(dir)
|
||||||
|
if err != nil {
|
||||||
|
_ = watcher.Close()
|
||||||
|
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
return &Watcher{watcher: watcher}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Close stops watching the directory.
|
||||||
|
func (w *Watcher) Close() error {
|
||||||
|
return w.watcher.Close()
|
||||||
|
}
|
||||||
|
|
||||||
|
// CallWhenQuiet calls changed each time the directory has gone quietTime
|
||||||
|
// without a change, after a change to a path for which counts is true, until
|
||||||
|
// ctx is done. It also calls failed with each error the watch reports. The
|
||||||
|
// wait starts at once, as if for a change, so that a change made before the
|
||||||
|
// watch began is taken in too.
|
||||||
|
func (w *Watcher) CallWhenQuiet(
|
||||||
|
ctx context.Context,
|
||||||
|
counts func(path string) bool,
|
||||||
|
changed func(),
|
||||||
|
failed func(error),
|
||||||
|
) {
|
||||||
|
callWhenQuiet(
|
||||||
|
ctx, w.watcher.Events, w.watcher.Errors, counts, changed, failed,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// callWhenQuiet is CallWhenQuiet, for the changes from events and the
|
||||||
|
// errors from errs.
|
||||||
|
func callWhenQuiet(
|
||||||
|
ctx context.Context,
|
||||||
|
events <-chan fsnotify.Event,
|
||||||
|
errs <-chan error,
|
||||||
|
counts func(path string) bool,
|
||||||
|
changed func(),
|
||||||
|
failed func(error),
|
||||||
|
) {
|
||||||
|
quiet := time.NewTimer(quietTime)
|
||||||
|
defer quiet.Stop()
|
||||||
|
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case event := <-events:
|
||||||
|
if counts(event.Name) {
|
||||||
|
quiet.Reset(quietTime)
|
||||||
|
}
|
||||||
|
case <-quiet.C:
|
||||||
|
changed()
|
||||||
|
case err := <-errs:
|
||||||
|
failed(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,167 @@
|
|||||||
|
package dirwatch
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"path/filepath"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"testing/synctest"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/fsnotify/fsnotify"
|
||||||
|
)
|
||||||
|
|
||||||
|
// errWatch is an error the watch reports.
|
||||||
|
var errWatch = errors.New("too many changes")
|
||||||
|
|
||||||
|
func TestDirectoryThatCannotBeWatchedIsAnError(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
_, err := New(filepath.Join(t.TempDir(), "missing"))
|
||||||
|
if err == nil {
|
||||||
|
t.Error("a missing directory is watched, want an error")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The tests below run callWhenQuiet in a synctest bubble, where time is a
|
||||||
|
// clock of the test's own: time.Sleep moves it on at once, and
|
||||||
|
// synctest.Wait returns once callWhenQuiet waits again, so that every call
|
||||||
|
// due by then is made. The test sends the changes itself, as the watch of
|
||||||
|
// a directory cannot run in a bubble.
|
||||||
|
|
||||||
|
func TestSecondChangeStartsTheWaitAgain(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
loop := start(t, everyPath)
|
||||||
|
|
||||||
|
loop.changes <- fsnotify.Event{Name: "50-app.rules", Op: fsnotify.Write}
|
||||||
|
|
||||||
|
time.Sleep(quietTime - time.Nanosecond)
|
||||||
|
synctest.Wait()
|
||||||
|
loop.wantCalls(t, 0)
|
||||||
|
|
||||||
|
// The second part starts the wait again.
|
||||||
|
loop.changes <- fsnotify.Event{Name: "50-app.rules", Op: fsnotify.Write}
|
||||||
|
|
||||||
|
time.Sleep(quietTime - time.Nanosecond)
|
||||||
|
synctest.Wait()
|
||||||
|
loop.wantCalls(t, 0)
|
||||||
|
|
||||||
|
time.Sleep(time.Nanosecond)
|
||||||
|
synctest.Wait()
|
||||||
|
loop.wantCalls(t, 1)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCalledOnceQuietAfterTheStart(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
// An edit saved before the watch began sends no change, and is
|
||||||
|
// taken in by this call.
|
||||||
|
loop := start(t, everyPath)
|
||||||
|
|
||||||
|
time.Sleep(quietTime)
|
||||||
|
synctest.Wait()
|
||||||
|
loop.wantCalls(t, 1)
|
||||||
|
|
||||||
|
// Long after, it has been called only once.
|
||||||
|
time.Sleep(time.Hour)
|
||||||
|
synctest.Wait()
|
||||||
|
loop.wantCalls(t, 1)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestChangeThatDoesNotCountIsNoChange(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
loop := start(t, func(path string) bool {
|
||||||
|
return path == "ipinfo_lite.mmdb"
|
||||||
|
})
|
||||||
|
|
||||||
|
// The wait that starts with the watch ends with a call.
|
||||||
|
time.Sleep(quietTime)
|
||||||
|
synctest.Wait()
|
||||||
|
|
||||||
|
loop.changes <- fsnotify.Event{Name: "other.mmdb", Op: fsnotify.Create}
|
||||||
|
|
||||||
|
time.Sleep(quietTime)
|
||||||
|
synctest.Wait()
|
||||||
|
loop.wantCalls(t, 1)
|
||||||
|
|
||||||
|
loop.changes <- fsnotify.Event{Name: "ipinfo_lite.mmdb", Op: fsnotify.Write}
|
||||||
|
|
||||||
|
time.Sleep(quietTime)
|
||||||
|
synctest.Wait()
|
||||||
|
loop.wantCalls(t, 2)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestErrorsOfTheWatchHandedOn(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
loop := start(t, everyPath)
|
||||||
|
|
||||||
|
loop.errs <- errWatch
|
||||||
|
|
||||||
|
synctest.Wait()
|
||||||
|
|
||||||
|
if !errors.Is(loop.failure, errWatch) {
|
||||||
|
t.Errorf("failed with %v, want %v", loop.failure, errWatch)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// loop is a callWhenQuiet that a test runs.
|
||||||
|
type loop struct {
|
||||||
|
// changes and errs send it changes and errors.
|
||||||
|
changes chan fsnotify.Event
|
||||||
|
errs chan error
|
||||||
|
// calls counts its calls of changed, and failure is the last error it
|
||||||
|
// handed to failed.
|
||||||
|
calls atomic.Int32
|
||||||
|
failure error
|
||||||
|
}
|
||||||
|
|
||||||
|
// start runs callWhenQuiet, with counts, until the test ends.
|
||||||
|
func start(t *testing.T, counts func(path string) bool) *loop {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
l := &loop{changes: make(chan fsnotify.Event), errs: make(chan error)}
|
||||||
|
ctx, stop := context.WithCancel(t.Context())
|
||||||
|
stopped := make(chan struct{})
|
||||||
|
|
||||||
|
go func() {
|
||||||
|
callWhenQuiet(ctx, l.changes, l.errs, counts,
|
||||||
|
func() { l.calls.Add(1) },
|
||||||
|
func(err error) { l.failure = err },
|
||||||
|
)
|
||||||
|
close(stopped)
|
||||||
|
}()
|
||||||
|
|
||||||
|
t.Cleanup(func() {
|
||||||
|
stop()
|
||||||
|
<-stopped
|
||||||
|
})
|
||||||
|
|
||||||
|
return l
|
||||||
|
}
|
||||||
|
|
||||||
|
// everyPath counts a change to any path.
|
||||||
|
func everyPath(string) bool {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// wantCalls checks how many times changed has been called.
|
||||||
|
func (l *loop) wantCalls(t *testing.T, want int32) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
got := l.calls.Load()
|
||||||
|
if got != want {
|
||||||
|
t.Errorf("changed called %d times, want %d", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
package lookup
|
package geojs
|
||||||
|
|
||||||
import "net/http"
|
import "net/http"
|
||||||
|
|
||||||
@@ -1,9 +1,9 @@
|
|||||||
// Package lookup looks up each client's AS number and country, through
|
// Package geojs looks up clients' AS numbers and countries through the GeoJS
|
||||||
// the GeoJS web service or in the lookup database, the IPinfo Lite file
|
// web service. Its answers are kept in memory, for at most 100,000 clients and
|
||||||
// SWW_WAF_LOOKUP_DB_PATH names. GeoJS's answers are kept in memory, for at
|
// for 7 days each. A caller waits for an answer as long as it says. A request to
|
||||||
// most 100,000 clients and for 7 days each, and are written to
|
// GeoJS goes on after its callers stop waiting, and its answer is kept. The
|
||||||
// lookups.json and read from it by the state package.
|
// package imports no other package of this repository.
|
||||||
package lookup
|
package geojs
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
@@ -11,7 +11,6 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"log/slog"
|
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"slices"
|
"slices"
|
||||||
@@ -21,8 +20,6 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/hashicorp/golang-lru/v2/simplelru"
|
"github.com/hashicorp/golang-lru/v2/simplelru"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// URL is GeoJS's endpoint for an address's place and network. Asked about
|
// URL is GeoJS's endpoint for an address's place and network. Asked about
|
||||||
@@ -43,6 +40,9 @@ const (
|
|||||||
maxWaiting = 10000
|
maxWaiting = 10000
|
||||||
// maxPerRequest is how many addresses one request to GeoJS asks about.
|
// maxPerRequest is how many addresses one request to GeoJS asks about.
|
||||||
maxPerRequest = 200
|
maxPerRequest = 200
|
||||||
|
// requestTimeout is how long a request to GeoJS may take before it is
|
||||||
|
// abandoned, which is a failure. It is apart from how long callers wait.
|
||||||
|
requestTimeout = 10 * time.Second
|
||||||
// unknownASN is the AS number GeoJS gives when it knows none.
|
// unknownASN is the AS number GeoJS gives when it knows none.
|
||||||
unknownASN = 64512
|
unknownASN = 64512
|
||||||
// After a failure GeoJS is not asked again for a second, and for
|
// After a failure GeoJS is not asked again for a second, and for
|
||||||
@@ -64,39 +64,27 @@ var (
|
|||||||
type Params struct {
|
type Params struct {
|
||||||
// URL is where GeoJS is asked, normally URL.
|
// URL is where GeoJS is asked, normally URL.
|
||||||
URL string
|
URL string
|
||||||
// Timeout is how long a request waits for its client's first answer,
|
|
||||||
// and how long a request to GeoJS may take before it is abandoned
|
|
||||||
// (SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS).
|
|
||||||
Timeout time.Duration
|
|
||||||
// Wait is true when a setting needs each request's answer before the
|
|
||||||
// request goes on. Otherwise no request waits for one.
|
|
||||||
Wait bool
|
|
||||||
// Answered, unless nil, is given each answer GeoJS gives, once it is
|
// Answered, unless nil, is given each answer GeoJS gives, once it is
|
||||||
// kept.
|
// kept.
|
||||||
Answered func(Answer)
|
Answered func(Answer)
|
||||||
// Now tells the time, normally time.Now in UTC.
|
// Now tells the time, normally time.Now in UTC.
|
||||||
Now func() time.Time
|
Now func() time.Time
|
||||||
// ProcessLog receives GeoJS's failures.
|
// Asked is called for each request to GeoJS.
|
||||||
ProcessLog *slog.Logger
|
Asked func()
|
||||||
// Metrics count the requests to GeoJS, those that failed, and the
|
// Failed is called for each failure of GeoJS, with what went wrong and
|
||||||
// clients that go without an answer.
|
// how long GeoJS is left alone after it. The error never names the
|
||||||
Metrics *metrics.Metrics
|
// addresses asked about.
|
||||||
// Alerts receive a source_failure alert each time GeoJS fails.
|
Failed func(err error, askingAgainIn time.Duration)
|
||||||
Alerts *alerts.Queue
|
// WentWithoutAnswer is called each time a caller that waits for an
|
||||||
|
// answer goes on without one.
|
||||||
|
WentWithoutAnswer func()
|
||||||
}
|
}
|
||||||
|
|
||||||
// GeoJS looks up clients' AS numbers and countries through GeoJS. At most
|
// GeoJS looks up clients' AS numbers and countries through GeoJS. At most
|
||||||
// one request to GeoJS is under way at a time, and it asks about every
|
// one request to GeoJS is under way at a time, and it asks about every
|
||||||
// client waiting, up to maxPerRequest. It is safe for concurrent use.
|
// client waiting, up to maxPerRequest. It is safe for concurrent use.
|
||||||
type GeoJS struct {
|
type GeoJS struct {
|
||||||
url string
|
params Params
|
||||||
timeout time.Duration
|
|
||||||
wait bool
|
|
||||||
answered func(Answer)
|
|
||||||
now func() time.Time
|
|
||||||
processLog *slog.Logger
|
|
||||||
metrics *metrics.Metrics
|
|
||||||
alerts *alerts.Queue
|
|
||||||
// httpClient follows no redirect, so that visitors' addresses go to
|
// httpClient follows no redirect, so that visitors' addresses go to
|
||||||
// GeoJS alone: a redirect is a failure.
|
// GeoJS alone: a redirect is a failure.
|
||||||
httpClient *http.Client
|
httpClient *http.Client
|
||||||
@@ -106,20 +94,23 @@ type GeoJS struct {
|
|||||||
// waiting are the clients without an answer: those to ask GeoJS about,
|
// waiting are the clients without an answer: those to ask GeoJS about,
|
||||||
// and those it is being asked about.
|
// and those it is being asked about.
|
||||||
waiting map[netip.Prefix]*wait
|
waiting map[netip.Prefix]*wait
|
||||||
// asking is true while a request to GeoJS is under way.
|
// asking is true while a request to GeoJS is under way, and endAsking
|
||||||
asking bool
|
// ends it. stopped is true once StopAsking has been called, and
|
||||||
|
// askGoroutines waits for the goroutine that makes the requests.
|
||||||
|
asking bool
|
||||||
|
endAsking context.CancelFunc
|
||||||
|
stopped bool
|
||||||
|
askGoroutines sync.WaitGroup
|
||||||
// retryDelay is how long GeoJS is left alone after its last failure,
|
// retryDelay is how long GeoJS is left alone after its last failure,
|
||||||
// zero after an answer; retryAt is when it may be asked again.
|
// zero after an answer; retryAt is when it may be asked again.
|
||||||
retryDelay time.Duration
|
retryDelay time.Duration
|
||||||
retryAt time.Time
|
retryAt time.Time
|
||||||
}
|
}
|
||||||
|
|
||||||
// Answer is what GeoJS or the lookup database said about a client: its AS
|
// Answer is what GeoJS said about a client: its AS number, such as AS64496,
|
||||||
// number, such as AS64496, and the AS's name, both "" when the source knows
|
// and the AS's name, both "" when GeoJS knows no AS number for it; its
|
||||||
// no AS number for it; its country, "" when the source cannot place it;
|
// country, "" when GeoJS cannot place it; when GeoJS said so; and when the
|
||||||
// when the source said so; and, for GeoJS's answers, which lookups.json
|
// answer was last used. The zero Answer is that of a client with no answer.
|
||||||
// holds, when the answer was last used. The zero Answer is that of a
|
|
||||||
// client with no answer.
|
|
||||||
//
|
//
|
||||||
//nolint:tagliatelle // the state files use snake_case, as the request log does
|
//nolint:tagliatelle // the state files use snake_case, as the request log does
|
||||||
type Answer struct {
|
type Answer struct {
|
||||||
@@ -137,7 +128,7 @@ type wait struct {
|
|||||||
// replaced each time GeoJS fails before then.
|
// replaced each time GeoJS fails before then.
|
||||||
asked chan struct{}
|
asked chan struct{}
|
||||||
// late is true once the client has gone without an answer, for a
|
// late is true once the client has gone without an answer, for a
|
||||||
// whole timeout or because GeoJS failed: its requests no longer wait.
|
// whole wait or because GeoJS failed: its callers no longer wait.
|
||||||
late bool
|
late bool
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -149,14 +140,7 @@ func New(params Params) *GeoJS {
|
|||||||
}
|
}
|
||||||
|
|
||||||
return &GeoJS{
|
return &GeoJS{
|
||||||
url: params.URL,
|
params: params,
|
||||||
timeout: params.Timeout,
|
|
||||||
wait: params.Wait,
|
|
||||||
answered: params.Answered,
|
|
||||||
now: params.Now,
|
|
||||||
processLog: params.ProcessLog,
|
|
||||||
metrics: params.Metrics,
|
|
||||||
alerts: params.Alerts,
|
|
||||||
httpClient: &http.Client{
|
httpClient: &http.Client{
|
||||||
CheckRedirect: func(*http.Request, []*http.Request) error {
|
CheckRedirect: func(*http.Request, []*http.Request) error {
|
||||||
return http.ErrUseLastResponse
|
return http.ErrUseLastResponse
|
||||||
@@ -170,21 +154,22 @@ func New(params Params) *GeoJS {
|
|||||||
// LookUp returns the answer GeoJS gave about client, with its country as
|
// LookUp returns the answer GeoJS gave about client, with its country as
|
||||||
// a two-letter code in capitals, or the zero Answer when there is none
|
// a two-letter code in capitals, or the zero Answer when there is none
|
||||||
// yet. An answer is kept for 7 days. Without one, the client is asked
|
// yet. An answer is kept for 7 days. Without one, the client is asked
|
||||||
// about in the background, and, while Wait is set, the request waits up
|
// about in the background, and LookUp waits up to wait for the answer,
|
||||||
// to Timeout for the answer, unless the client has gone without one
|
// unless the client has gone without one before. The wait ends early when
|
||||||
// before. ctx is the context of the client's request, and ends the wait
|
// ctx ends. Neither ends the request to GeoJS.
|
||||||
// when it ends.
|
|
||||||
//
|
//
|
||||||
// GeoJS is asked about the client's first address, which is the client's
|
// GeoJS is asked about the client's first address, which is the client's
|
||||||
// own address for IPv4, and an address in the same place for an IPv6
|
// own address for IPv4, and an address in the same place for an IPv6
|
||||||
// netblock.
|
// netblock.
|
||||||
func (g *GeoJS) LookUp(ctx context.Context, client netip.Prefix) Answer {
|
func (g *GeoJS) LookUp(
|
||||||
answer, asked := g.answerOrWait(ctx, client, g.wait)
|
ctx context.Context, client netip.Prefix, wait time.Duration,
|
||||||
|
) Answer {
|
||||||
|
answer, asked := g.answerOrWait(ctx, client, true)
|
||||||
if asked == nil {
|
if asked == nil {
|
||||||
return answer
|
return answer
|
||||||
}
|
}
|
||||||
|
|
||||||
timer := time.NewTimer(g.timeout)
|
timer := time.NewTimer(wait)
|
||||||
defer timer.Stop()
|
defer timer.Stop()
|
||||||
|
|
||||||
select {
|
select {
|
||||||
@@ -198,7 +183,7 @@ func (g *GeoJS) LookUp(ctx context.Context, client netip.Prefix) Answer {
|
|||||||
|
|
||||||
answer, found := g.kept(client)
|
answer, found := g.kept(client)
|
||||||
if !found {
|
if !found {
|
||||||
g.metrics.RequestsWithoutGeoJSAnswer.Inc()
|
g.params.WentWithoutAnswer()
|
||||||
}
|
}
|
||||||
|
|
||||||
w, waiting := g.waiting[client]
|
w, waiting := g.waiting[client]
|
||||||
@@ -209,8 +194,8 @@ func (g *GeoJS) LookUp(ctx context.Context, client netip.Prefix) Answer {
|
|||||||
return answer
|
return answer
|
||||||
}
|
}
|
||||||
|
|
||||||
// LookUpWithoutWaiting is LookUp for a request that does not need the answer,
|
// LookUpWithoutWaiting is LookUp for a caller that does not need the answer:
|
||||||
// whatever Wait says: it returns the zero Answer at once when none is kept.
|
// it returns the zero Answer at once when none is kept.
|
||||||
func (g *GeoJS) LookUpWithoutWaiting(
|
func (g *GeoJS) LookUpWithoutWaiting(
|
||||||
ctx context.Context,
|
ctx context.Context,
|
||||||
client netip.Prefix,
|
client netip.Prefix,
|
||||||
@@ -220,6 +205,23 @@ func (g *GeoJS) LookUpWithoutWaiting(
|
|||||||
return answer
|
return answer
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// StopAsking ends the request to GeoJS under way, and returns once the
|
||||||
|
// goroutine that made it has. A request it ends is no failure: Failed is not
|
||||||
|
// called, and GeoJS is not left alone after it. After StopAsking, GeoJS is
|
||||||
|
// asked nothing.
|
||||||
|
func (g *GeoJS) StopAsking() {
|
||||||
|
g.mu.Lock()
|
||||||
|
|
||||||
|
g.stopped = true
|
||||||
|
if g.asking {
|
||||||
|
g.endAsking()
|
||||||
|
}
|
||||||
|
|
||||||
|
g.mu.Unlock()
|
||||||
|
|
||||||
|
g.askGoroutines.Wait()
|
||||||
|
}
|
||||||
|
|
||||||
// Kept returns client's answer, if one is kept, without asking GeoJS.
|
// Kept returns client's answer, if one is kept, without asking GeoJS.
|
||||||
func (g *GeoJS) Kept(client netip.Prefix) (Answer, bool) {
|
func (g *GeoJS) Kept(client netip.Prefix) (Answer, bool) {
|
||||||
g.mu.Lock()
|
g.mu.Lock()
|
||||||
@@ -228,8 +230,7 @@ func (g *GeoJS) Kept(client netip.Prefix) (Answer, bool) {
|
|||||||
return g.kept(client)
|
return g.kept(client)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Snapshot returns every answer kept, sorted by client, as lookups.json
|
// Snapshot returns every answer kept, sorted by client.
|
||||||
// lists them.
|
|
||||||
func (g *GeoJS) Snapshot() []Answer {
|
func (g *GeoJS) Snapshot() []Answer {
|
||||||
g.mu.Lock()
|
g.mu.Lock()
|
||||||
|
|
||||||
@@ -247,9 +248,9 @@ func (g *GeoJS) Snapshot() []Answer {
|
|||||||
return answers
|
return answers
|
||||||
}
|
}
|
||||||
|
|
||||||
// Load keeps answers read from lookups.json, in place of the answers it
|
// Load keeps answers, such as those of an earlier Snapshot, in place of the
|
||||||
// keeps, in the order they were last used, so that the one used longest
|
// answers it keeps, in the order they were last used, so that the one used
|
||||||
// ago is dropped first. Answers GeoJS gave keepFor ago or more are
|
// longest ago is dropped first. Answers GeoJS gave keepFor ago or more are
|
||||||
// dropped.
|
// dropped.
|
||||||
func (g *GeoJS) Load(answers []Answer) {
|
func (g *GeoJS) Load(answers []Answer) {
|
||||||
answers = slices.Clone(answers)
|
answers = slices.Clone(answers)
|
||||||
@@ -262,7 +263,7 @@ func (g *GeoJS) Load(answers []Answer) {
|
|||||||
|
|
||||||
g.answers.Purge()
|
g.answers.Purge()
|
||||||
|
|
||||||
now := g.now()
|
now := g.params.Now()
|
||||||
|
|
||||||
for _, answer := range answers {
|
for _, answer := range answers {
|
||||||
if now.Sub(answer.Answered) < keepFor {
|
if now.Sub(answer.Answered) < keepFor {
|
||||||
@@ -295,22 +296,23 @@ func (g *GeoJS) answerOrWait(
|
|||||||
g.ask(ctx)
|
g.ask(ctx)
|
||||||
|
|
||||||
if !mustWait {
|
if !mustWait {
|
||||||
return Answer{}, nil // the answer is not needed before the request goes on
|
return Answer{}, nil // the answer is not needed before the caller goes on
|
||||||
}
|
}
|
||||||
|
|
||||||
if w == nil {
|
if w == nil {
|
||||||
g.metrics.RequestsWithoutGeoJSAnswer.Inc()
|
g.params.WentWithoutAnswer()
|
||||||
|
|
||||||
return Answer{}, nil // too many clients wait already
|
return Answer{}, nil // too many clients wait already
|
||||||
}
|
}
|
||||||
|
|
||||||
if !g.asking {
|
if !g.asking {
|
||||||
// GeoJS is left alone after a failure, so no answer can come.
|
// GeoJS is left alone after a failure, or asked nothing after
|
||||||
|
// StopAsking, so no answer can come.
|
||||||
w.late = true
|
w.late = true
|
||||||
}
|
}
|
||||||
|
|
||||||
if w.late {
|
if w.late {
|
||||||
g.metrics.RequestsWithoutGeoJSAnswer.Inc()
|
g.params.WentWithoutAnswer()
|
||||||
|
|
||||||
return Answer{}, nil
|
return Answer{}, nil
|
||||||
}
|
}
|
||||||
@@ -321,7 +323,7 @@ func (g *GeoJS) answerOrWait(
|
|||||||
// kept returns client's answer, if GeoJS gave it less than keepFor ago,
|
// kept returns client's answer, if GeoJS gave it less than keepFor ago,
|
||||||
// and notes that it was used.
|
// and notes that it was used.
|
||||||
func (g *GeoJS) kept(client netip.Prefix) (Answer, bool) {
|
func (g *GeoJS) kept(client netip.Prefix) (Answer, bool) {
|
||||||
now := g.now()
|
now := g.params.Now()
|
||||||
|
|
||||||
kept, found := g.answers.Get(client)
|
kept, found := g.answers.Get(client)
|
||||||
if !found || now.Sub(kept.Answered) >= keepFor {
|
if !found || now.Sub(kept.Answered) >= keepFor {
|
||||||
@@ -334,22 +336,28 @@ func (g *GeoJS) kept(client netip.Prefix) (Answer, bool) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// ask starts asking GeoJS about the waiting clients, unless a request to
|
// ask starts asking GeoJS about the waiting clients, unless a request to
|
||||||
// it is under way or it is left alone after a failure. The requests to
|
// it is under way, it is left alone after a failure, or StopAsking has
|
||||||
// GeoJS are for every client waiting, so they go on when the client's
|
// been called. The requests to GeoJS are for every client waiting, so they
|
||||||
// request whose ctx is given ends.
|
// go on when the caller whose ctx is given stops waiting, until StopAsking.
|
||||||
func (g *GeoJS) ask(ctx context.Context) {
|
func (g *GeoJS) ask(ctx context.Context) {
|
||||||
if g.asking || g.now().Before(g.retryAt) {
|
if g.asking || g.stopped || g.params.Now().Before(g.retryAt) {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
g.asking = true
|
askCtx, end := context.WithCancel(context.WithoutCancel(ctx))
|
||||||
|
g.asking, g.endAsking = true, end
|
||||||
|
|
||||||
go g.askAboutWaiting(context.WithoutCancel(ctx))
|
g.askGoroutines.Go(func() {
|
||||||
|
defer end()
|
||||||
|
|
||||||
|
g.askAboutWaiting(askCtx)
|
||||||
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
// askAboutWaiting asks GeoJS about the waiting clients, one request at a
|
// askAboutWaiting asks GeoJS about the waiting clients, one request at a
|
||||||
// time, until none is left or GeoJS fails. Each answer kept is given to
|
// time, until none is left, GeoJS fails, or StopAsking ends ctx. Each
|
||||||
// Answered, outside the lock, since Answered takes locks of its own.
|
// answer kept is given to Answered, outside the lock, since Answered takes
|
||||||
|
// locks of its own.
|
||||||
func (g *GeoJS) askAboutWaiting(ctx context.Context) {
|
func (g *GeoJS) askAboutWaiting(ctx context.Context) {
|
||||||
for {
|
for {
|
||||||
clients := g.nextClients()
|
clients := g.nextClients()
|
||||||
@@ -358,11 +366,15 @@ func (g *GeoJS) askAboutWaiting(ctx context.Context) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
given, err := g.request(ctx, clients)
|
given, err := g.request(ctx, clients)
|
||||||
|
if err != nil && ctx.Err() != nil {
|
||||||
|
continue // ended by StopAsking, which is no failure
|
||||||
|
}
|
||||||
|
|
||||||
kept, answered := g.keep(clients, given, err)
|
kept, answered := g.keep(clients, given, err)
|
||||||
|
|
||||||
if g.answered != nil {
|
if g.params.Answered != nil {
|
||||||
for _, answer := range kept {
|
for _, answer := range kept {
|
||||||
g.answered(answer)
|
g.params.Answered(answer)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -373,13 +385,13 @@ func (g *GeoJS) askAboutWaiting(ctx context.Context) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// nextClients returns up to maxPerRequest of the waiting clients. When
|
// nextClients returns up to maxPerRequest of the waiting clients. When
|
||||||
// none is waiting, it returns none and notes that no request to GeoJS is
|
// none is waiting, or StopAsking has been called, it returns none and
|
||||||
// under way.
|
// notes that no request to GeoJS is under way.
|
||||||
func (g *GeoJS) nextClients() []netip.Prefix {
|
func (g *GeoJS) nextClients() []netip.Prefix {
|
||||||
g.mu.Lock()
|
g.mu.Lock()
|
||||||
defer g.mu.Unlock()
|
defer g.mu.Unlock()
|
||||||
|
|
||||||
if len(g.waiting) == 0 {
|
if len(g.waiting) == 0 || g.stopped {
|
||||||
g.asking = false
|
g.asking = false
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
@@ -411,7 +423,7 @@ func (g *GeoJS) keep(
|
|||||||
g.mu.Lock()
|
g.mu.Lock()
|
||||||
defer g.mu.Unlock()
|
defer g.mu.Unlock()
|
||||||
|
|
||||||
now := g.now()
|
now := g.params.Now()
|
||||||
kept := make([]Answer, 0, len(clients))
|
kept := make([]Answer, 0, len(clients))
|
||||||
leftOut := 0
|
leftOut := 0
|
||||||
|
|
||||||
@@ -441,8 +453,6 @@ func (g *GeoJS) keep(
|
|||||||
}
|
}
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
g.metrics.GeoJSFailures.Inc()
|
|
||||||
|
|
||||||
g.retryDelay = min(
|
g.retryDelay = min(
|
||||||
max(retryDelayFactor*g.retryDelay, firstRetryDelay),
|
max(retryDelayFactor*g.retryDelay, firstRetryDelay),
|
||||||
maxRetryDelay,
|
maxRetryDelay,
|
||||||
@@ -457,16 +467,7 @@ func (g *GeoJS) keep(
|
|||||||
w.late = true
|
w.late = true
|
||||||
}
|
}
|
||||||
|
|
||||||
g.processLog.Warn("asking GeoJS failed",
|
g.params.Failed(err, g.retryDelay)
|
||||||
"error", err.Error(), "asking_again_in", g.retryDelay.String())
|
|
||||||
g.alerts.Raise(alerts.Alert{
|
|
||||||
Event: alerts.EventSourceFailure,
|
|
||||||
Reason: "asking GeoJS failed",
|
|
||||||
Detail: map[string]any{
|
|
||||||
"source": "geojs", "error": err.Error(),
|
|
||||||
"asking_again_in": g.retryDelay.String(),
|
|
||||||
},
|
|
||||||
})
|
|
||||||
|
|
||||||
return kept, false
|
return kept, false
|
||||||
}
|
}
|
||||||
@@ -479,7 +480,8 @@ func (g *GeoJS) keep(
|
|||||||
// request asks GeoJS about clients in one request, and returns the answer
|
// request asks GeoJS about clients in one request, and returns the answer
|
||||||
// for each address GeoJS's answer names: its AS number and the AS's name,
|
// for each address GeoJS's answer names: its AS number and the AS's name,
|
||||||
// both "" for the AS number 64512, which GeoJS gives when it knows none,
|
// both "" for the AS number 64512, which GeoJS gives when it knows none,
|
||||||
// and its country, in capitals.
|
// and its country, in capitals. The request is abandoned after
|
||||||
|
// requestTimeout.
|
||||||
func (g *GeoJS) request(
|
func (g *GeoJS) request(
|
||||||
ctx context.Context, clients []netip.Prefix,
|
ctx context.Context, clients []netip.Prefix,
|
||||||
) (map[netip.Addr]Answer, error) {
|
) (map[netip.Addr]Answer, error) {
|
||||||
@@ -489,13 +491,13 @@ func (g *GeoJS) request(
|
|||||||
addrs = append(addrs, client.Addr().String())
|
addrs = append(addrs, client.Addr().String())
|
||||||
}
|
}
|
||||||
|
|
||||||
ctx, cancel := context.WithTimeout(ctx, g.timeout)
|
ctx, cancel := context.WithTimeout(ctx, requestTimeout)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
|
|
||||||
req, err := http.NewRequestWithContext(
|
req, err := http.NewRequestWithContext(
|
||||||
ctx,
|
ctx,
|
||||||
http.MethodGet,
|
http.MethodGet,
|
||||||
g.url,
|
g.params.URL,
|
||||||
http.NoBody,
|
http.NoBody,
|
||||||
)
|
)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -504,7 +506,7 @@ func (g *GeoJS) request(
|
|||||||
|
|
||||||
req.URL.RawQuery = "ip=" + strings.Join(addrs, ",")
|
req.URL.RawQuery = "ip=" + strings.Join(addrs, ",")
|
||||||
|
|
||||||
g.metrics.GeoJSRequests.Inc()
|
g.params.Asked()
|
||||||
|
|
||||||
res, err := g.httpClient.Do(req)
|
res, err := g.httpClient.Do(req)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
File diff suppressed because it is too large.
Load diff
@@ -1,4 +1,4 @@
|
|||||||
package lookup_test
|
package geojs_test
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"net/netip"
|
"net/netip"
|
||||||
@@ -7,7 +7,7 @@ import (
|
|||||||
"testing/synctest"
|
"testing/synctest"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestSnapshotHoldsEachAnswerAndWhenItWasLastUsed(t *testing.T) {
|
func TestSnapshotHoldsEachAnswerAndWhenItWasLastUsed(t *testing.T) {
|
||||||
@@ -25,7 +25,7 @@ func TestSnapshotHoldsEachAnswerAndWhenItWasLastUsed(t *testing.T) {
|
|||||||
clock.advance(time.Hour)
|
clock.advance(time.Hour)
|
||||||
wantCountry(t, g, placed, germany)
|
wantCountry(t, g, placed, germany)
|
||||||
|
|
||||||
want := []lookup.Answer{
|
want := []geojs.Answer{
|
||||||
{Client: notPlaced, Answered: asked, Used: asked},
|
{Client: notPlaced, Answered: asked, Used: asked},
|
||||||
{
|
{
|
||||||
Client: placed, ASN: asn, ASName: asName, Country: germany,
|
Client: placed, ASN: asn, ASName: asName, Country: germany,
|
||||||
@@ -41,24 +41,24 @@ func TestSnapshotHoldsEachAnswerAndWhenItWasLastUsed(t *testing.T) {
|
|||||||
func TestLoadedAnswersAreKeptFor7DaysFromWhenGeoJSGaveThem(t *testing.T) {
|
func TestLoadedAnswersAreKeptFor7DaysFromWhenGeoJSGaveThem(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
geojs, clock, g := start()
|
service, clock, g := start()
|
||||||
now := clock.Now()
|
now := clock.Now()
|
||||||
kept := lookup.Answer{
|
kept := geojs.Answer{
|
||||||
Client: netip.MustParsePrefix("203.0.113.9/32"),
|
Client: netip.MustParsePrefix("203.0.113.9/32"),
|
||||||
Country: "FR",
|
Country: "FR",
|
||||||
Answered: now.Add(-week + time.Second),
|
Answered: now.Add(-week + time.Second),
|
||||||
Used: now.Add(-time.Hour),
|
Used: now.Add(-time.Hour),
|
||||||
}
|
}
|
||||||
stale := lookup.Answer{
|
stale := geojs.Answer{
|
||||||
Client: netip.MustParsePrefix("203.0.113.10/32"),
|
Client: netip.MustParsePrefix("203.0.113.10/32"),
|
||||||
Country: "FR",
|
Country: "FR",
|
||||||
Answered: now.Add(-week),
|
Answered: now.Add(-week),
|
||||||
Used: now.Add(-time.Hour),
|
Used: now.Add(-time.Hour),
|
||||||
}
|
}
|
||||||
|
|
||||||
g.Load([]lookup.Answer{kept, stale})
|
g.Load([]geojs.Answer{kept, stale})
|
||||||
|
|
||||||
if got := g.Snapshot(); !slices.Equal(got, []lookup.Answer{kept}) {
|
if got := g.Snapshot(); !slices.Equal(got, []geojs.Answer{kept}) {
|
||||||
t.Errorf(
|
t.Errorf(
|
||||||
"kept %+v, want only the answer GeoJS gave less than 7 days ago",
|
"kept %+v, want only the answer GeoJS gave less than 7 days ago",
|
||||||
got,
|
got,
|
||||||
@@ -66,7 +66,7 @@ func TestLoadedAnswersAreKeptFor7DaysFromWhenGeoJSGaveThem(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
wantCountry(t, g, kept.Client, "FR")
|
wantCountry(t, g, kept.Client, "FR")
|
||||||
wantRequests(t, geojs, 0)
|
wantRequests(t, service, 0)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestLoadDropsTheAnswerUsedLongestAgoFirst(t *testing.T) {
|
func TestLoadDropsTheAnswerUsedLongestAgoFirst(t *testing.T) {
|
||||||
@@ -80,11 +80,11 @@ func TestLoadDropsTheAnswerUsedLongestAgoFirst(t *testing.T) {
|
|||||||
// lookups.json lists the answers by client. Here each was last used a
|
// lookups.json lists the answers by client. Here each was last used a
|
||||||
// second before the one listed before it, so the last listed is the
|
// second before the one listed before it, so the last listed is the
|
||||||
// one used longest ago, and the one dropped.
|
// one used longest ago, and the one dropped.
|
||||||
answers := make([]lookup.Answer, maxAnswers+1)
|
answers := make([]geojs.Answer, maxAnswers+1)
|
||||||
addr := netip.MustParseAddr("10.0.0.0")
|
addr := netip.MustParseAddr("10.0.0.0")
|
||||||
|
|
||||||
for i := range answers {
|
for i := range answers {
|
||||||
answers[i] = lookup.Answer{
|
answers[i] = geojs.Answer{
|
||||||
Client: netip.PrefixFrom(addr, addr.BitLen()),
|
Client: netip.PrefixFrom(addr, addr.BitLen()),
|
||||||
Country: germany,
|
Country: germany,
|
||||||
Answered: now,
|
Answered: now,
|
||||||
+24
-51
@@ -1,3 +1,6 @@
|
|||||||
|
// Package lookup looks up each client's AS number and country in the lookup
|
||||||
|
// database, the IPinfo Lite file SWW_WAF_LOOKUP_DB_PATH names. Its answers
|
||||||
|
// take the form of GeoJS's, so that waf handles both alike.
|
||||||
package lookup
|
package lookup
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -10,17 +13,13 @@ import (
|
|||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/fsnotify/fsnotify"
|
|
||||||
"github.com/oschwald/maxminddb-golang/v2"
|
"github.com/oschwald/maxminddb-golang/v2"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/dirwatch"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
)
|
)
|
||||||
|
|
||||||
// quietTime is how long the lookup database must go without a change
|
|
||||||
// before it is read again, so that a file still being copied in is read
|
|
||||||
// only once whole.
|
|
||||||
const quietTime = 2 * time.Second
|
|
||||||
|
|
||||||
// FileParams are what OpenFile needs.
|
// FileParams are what OpenFile needs.
|
||||||
type FileParams struct {
|
type FileParams struct {
|
||||||
// Path is the lookup database, the IPinfo Lite file in its .mmdb form
|
// Path is the lookup database, the IPinfo Lite file in its .mmdb form
|
||||||
@@ -77,7 +76,7 @@ func OpenFile(params FileParams) (*File, error) {
|
|||||||
// each "" when the database does not give it, as for an address missing
|
// each "" when the database does not give it, as for an address missing
|
||||||
// from it. The database is asked about the client's first address, as
|
// from it. The database is asked about the client's first address, as
|
||||||
// GeoJS is.
|
// GeoJS is.
|
||||||
func (f *File) LookUp(client netip.Prefix) Answer {
|
func (f *File) LookUp(client netip.Prefix) geojs.Answer {
|
||||||
f.mu.Lock()
|
f.mu.Lock()
|
||||||
reader := f.reader
|
reader := f.reader
|
||||||
f.mu.Unlock()
|
f.mu.Unlock()
|
||||||
@@ -91,7 +90,7 @@ func (f *File) LookUp(client netip.Prefix) Answer {
|
|||||||
found = record{}
|
found = record{}
|
||||||
}
|
}
|
||||||
|
|
||||||
return Answer{
|
return geojs.Answer{
|
||||||
Client: client,
|
Client: client,
|
||||||
ASN: found.ASN,
|
ASN: found.ASN,
|
||||||
ASName: found.ASName,
|
ASName: found.ASName,
|
||||||
@@ -118,20 +117,13 @@ func (f *File) ReadFailures() int {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Watch watches the directory of the lookup database until ctx is done,
|
// Watch watches the directory of the lookup database until ctx is done,
|
||||||
// and reads the file again once it has gone without a change for
|
// and reads the file again once it has gone without a change for a short
|
||||||
// quietTime, after it is replaced, written or removed, and after Watch
|
// time, after it is replaced, written or removed, and after Watch starts
|
||||||
// starts watching. If the directory cannot be watched, that is logged, and
|
// watching, as dirwatch describes. A change to another file in its
|
||||||
// the database read at start stays in use.
|
// directory does not count. If the directory cannot be watched, that is
|
||||||
|
// logged, and the database read at start stays in use.
|
||||||
func (f *File) Watch(ctx context.Context) {
|
func (f *File) Watch(ctx context.Context) {
|
||||||
watcher, err := fsnotify.NewWatcher()
|
watcher, err := dirwatch.New(filepath.Dir(f.params.Path))
|
||||||
if err == nil {
|
|
||||||
defer func() {
|
|
||||||
_ = watcher.Close()
|
|
||||||
}()
|
|
||||||
|
|
||||||
err = watcher.Add(filepath.Dir(f.params.Path))
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
f.params.ProcessLog.Error(
|
f.params.ProcessLog.Error(
|
||||||
"cannot watch the lookup database for replacements",
|
"cannot watch the lookup database for replacements",
|
||||||
@@ -142,41 +134,22 @@ func (f *File) Watch(ctx context.Context) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
defer func() {
|
||||||
|
_ = watcher.Close()
|
||||||
|
}()
|
||||||
|
|
||||||
f.params.ProcessLog.Info("watching the lookup database for replacements",
|
f.params.ProcessLog.Info("watching the lookup database for replacements",
|
||||||
"file", f.params.Path)
|
"file", f.params.Path)
|
||||||
|
|
||||||
f.readAfterChanges(ctx, watcher.Events, watcher.Errors)
|
watcher.CallWhenQuiet(ctx, f.isDatabase, f.readAgain, func(err error) {
|
||||||
|
f.params.ProcessLog.Warn("watching the lookup database failed",
|
||||||
|
"error", err.Error())
|
||||||
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
// readAfterChanges reads the lookup database again once quietTime has
|
// isDatabase reports whether path names the lookup database.
|
||||||
// passed without a change to it from events, until ctx is done, and logs
|
func (f *File) isDatabase(path string) bool {
|
||||||
// the errors from errs. A change to another file in its directory does not
|
return filepath.Clean(path) == filepath.Clean(f.params.Path)
|
||||||
// count. The wait starts at once, as if for a change, so that a file
|
|
||||||
// replaced after OpenFile read it, and before its directory was watched,
|
|
||||||
// is read too.
|
|
||||||
func (f *File) readAfterChanges(
|
|
||||||
ctx context.Context, events <-chan fsnotify.Event, errs <-chan error,
|
|
||||||
) {
|
|
||||||
path := filepath.Clean(f.params.Path)
|
|
||||||
|
|
||||||
quiet := time.NewTimer(quietTime)
|
|
||||||
defer quiet.Stop()
|
|
||||||
|
|
||||||
for {
|
|
||||||
select {
|
|
||||||
case <-ctx.Done():
|
|
||||||
return
|
|
||||||
case event := <-events:
|
|
||||||
if filepath.Clean(event.Name) == path {
|
|
||||||
quiet.Reset(quietTime)
|
|
||||||
}
|
|
||||||
case <-quiet.C:
|
|
||||||
f.readAgain()
|
|
||||||
case err := <-errs:
|
|
||||||
f.params.ProcessLog.Warn("watching the lookup database failed",
|
|
||||||
"error", err.Error())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// readAgain reads the lookup database again, in place of the one in use,
|
// readAgain reads the lookup database again, in place of the one in use,
|
||||||
|
|||||||
@@ -1,7 +1,6 @@
|
|||||||
package lookup
|
package lookup
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"net/url"
|
"net/url"
|
||||||
@@ -12,10 +11,10 @@ import (
|
|||||||
"testing/synctest"
|
"testing/synctest"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/fsnotify/fsnotify"
|
|
||||||
"github.com/maxmind/mmdbwriter/mmdbtype"
|
"github.com/maxmind/mmdbwriter/mmdbtype"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
|
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -68,7 +67,7 @@ func TestFilePlacesClientsAndCountsAnAddressMissingFromItAsUnknown(
|
|||||||
prefix := netip.MustParsePrefix(client)
|
prefix := netip.MustParsePrefix(client)
|
||||||
|
|
||||||
got := f.LookUp(prefix)
|
got := f.LookUp(prefix)
|
||||||
if got != (Answer{
|
if got != (geojs.Answer{
|
||||||
Client: prefix, ASN: want.ASN, ASName: want.ASName, Country: want.Country,
|
Client: prefix, ASN: want.ASN, ASName: want.ASName, Country: want.Country,
|
||||||
Answered: now,
|
Answered: now,
|
||||||
}) {
|
}) {
|
||||||
@@ -132,66 +131,48 @@ func TestFileThatCannotBeReadIsAnError(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// The tests below run readAfterChanges in a synctest bubble, where time is
|
func TestChangeOfAnotherFileInTheDirectoryIsNoReplacement(t *testing.T) {
|
||||||
// a clock of the test's own: time.Sleep moves it on at once, and
|
t.Parallel()
|
||||||
// synctest.Wait returns once readAfterChanges waits again, so that every
|
|
||||||
// reading due by then is done. The test sends the changes itself, as the
|
|
||||||
// watch of a directory cannot run in a bubble.
|
|
||||||
|
|
||||||
func TestReplacementCopiedOverTheFileInTwoPartsIsReadOnlyWhole(
|
dir := t.TempDir()
|
||||||
t *testing.T,
|
path := filepath.Join(dir, "ipinfo_lite.mmdb")
|
||||||
) {
|
writeDatabase(t, path, "DE")
|
||||||
|
|
||||||
|
f := openFile(t, path, newQueue())
|
||||||
|
|
||||||
|
for changed, want := range map[string]bool{
|
||||||
|
path: true,
|
||||||
|
dir + "/./ipinfo_lite.mmdb": true,
|
||||||
|
filepath.Join(dir, "other.mmdb"): false,
|
||||||
|
} {
|
||||||
|
if f.isDatabase(changed) != want {
|
||||||
|
t.Errorf(
|
||||||
|
"a change to %s counts: %t, want %t",
|
||||||
|
changed,
|
||||||
|
!want,
|
||||||
|
want,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The tests below run in a synctest bubble, where time is a clock of the
|
||||||
|
// test's own, so that the time a file is read can be checked. The wait for
|
||||||
|
// the directory to go quiet before a reading is tested in dirwatch.
|
||||||
|
|
||||||
|
func TestReplacementIsReadAgain(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
synctest.Test(t, func(t *testing.T) {
|
||||||
dir := t.TempDir()
|
path := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb")
|
||||||
path := filepath.Join(dir, "ipinfo_lite.mmdb")
|
|
||||||
writeDatabase(t, path, "DE")
|
writeDatabase(t, path, "DE")
|
||||||
|
|
||||||
queue := newQueue()
|
queue := newQueue()
|
||||||
f := openFile(t, path, queue)
|
f := openFile(t, path, queue)
|
||||||
changes := watch(t, f)
|
|
||||||
|
|
||||||
other := filepath.Join(dir, "replacement.mmdb")
|
writeDatabase(t, path, "KP")
|
||||||
writeDatabase(t, other, "KP")
|
time.Sleep(time.Hour)
|
||||||
|
f.readAgain()
|
||||||
//nolint:gosec // a file the test wrote
|
|
||||||
replacement, err := os.ReadFile(other)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("read %s: %v", other, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// The file in use is overwritten in place, and keeps giving what
|
|
||||||
// it gave. Its first part alone is not a .mmdb file.
|
|
||||||
file, err := os.Create(path) //nolint:gosec // a file the test wrote
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("create %s: %v", path, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() {
|
|
||||||
_ = file.Close()
|
|
||||||
}()
|
|
||||||
|
|
||||||
half := len(replacement) / 2
|
|
||||||
write(t, file, replacement[:half])
|
|
||||||
|
|
||||||
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
|
|
||||||
|
|
||||||
time.Sleep(quietTime - time.Nanosecond)
|
|
||||||
synctest.Wait()
|
|
||||||
wantCountry(t, f, "DE")
|
|
||||||
|
|
||||||
// The second part starts the wait again.
|
|
||||||
write(t, file, replacement[half:])
|
|
||||||
|
|
||||||
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
|
|
||||||
|
|
||||||
time.Sleep(quietTime - time.Nanosecond)
|
|
||||||
synctest.Wait()
|
|
||||||
wantCountry(t, f, "DE")
|
|
||||||
|
|
||||||
time.Sleep(time.Nanosecond)
|
|
||||||
synctest.Wait()
|
|
||||||
wantCountry(t, f, "KP")
|
wantCountry(t, f, "KP")
|
||||||
|
|
||||||
if !f.LastRead().Equal(time.Now()) || f.ReadFailures() != 0 {
|
if !f.LastRead().Equal(time.Now()) || f.ReadFailures() != 0 {
|
||||||
@@ -218,15 +199,10 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(
|
|||||||
queue := newQueue()
|
queue := newQueue()
|
||||||
f := openFile(t, path, queue)
|
f := openFile(t, path, queue)
|
||||||
read := f.LastRead()
|
read := f.LastRead()
|
||||||
changes := watch(t, f)
|
|
||||||
|
|
||||||
writeFile(t, path, "not a lookup database\n")
|
writeFile(t, path, "not a lookup database\n")
|
||||||
|
|
||||||
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
|
|
||||||
|
|
||||||
// Long after, the replacement has been read once.
|
|
||||||
time.Sleep(time.Hour)
|
time.Sleep(time.Hour)
|
||||||
synctest.Wait()
|
f.readAgain()
|
||||||
wantCountry(t, f, "DE")
|
wantCountry(t, f, "DE")
|
||||||
|
|
||||||
if !f.LastRead().Equal(read) || f.ReadFailures() != 1 {
|
if !f.LastRead().Equal(read) || f.ReadFailures() != 1 {
|
||||||
@@ -239,7 +215,7 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(
|
|||||||
}
|
}
|
||||||
|
|
||||||
wantAlerts(t, queue, alerts.Alert{
|
wantAlerts(t, queue, alerts.Alert{
|
||||||
Time: read.Add(quietTime),
|
Time: read.Add(time.Hour),
|
||||||
Event: alerts.EventFileError,
|
Event: alerts.EventFileError,
|
||||||
Reason: "the lookup database cannot be read, and the one read before stays in use",
|
Reason: "the lookup database cannot be read, and the one read before stays in use",
|
||||||
Detail: map[string]any{
|
Detail: map[string]any{
|
||||||
@@ -251,55 +227,6 @@ func TestReplacementThatCannotBeReadLeavesTheFileInUseWithOneAlert(
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestChangeOfAnotherFileInTheDirectoryIsNoReplacement(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
dir := t.TempDir()
|
|
||||||
path := filepath.Join(dir, "ipinfo_lite.mmdb")
|
|
||||||
writeDatabase(t, path, "DE")
|
|
||||||
|
|
||||||
f := openFile(t, path, newQueue())
|
|
||||||
changes := watch(t, f)
|
|
||||||
|
|
||||||
// The wait that starts with the watch ends with a reading.
|
|
||||||
time.Sleep(quietTime)
|
|
||||||
synctest.Wait()
|
|
||||||
writeDatabase(t, path, "KP")
|
|
||||||
|
|
||||||
changes <- fsnotify.Event{Name: filepath.Join(dir, "other.mmdb"), Op: fsnotify.Create}
|
|
||||||
|
|
||||||
time.Sleep(quietTime)
|
|
||||||
synctest.Wait()
|
|
||||||
wantCountry(t, f, "DE")
|
|
||||||
|
|
||||||
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
|
|
||||||
|
|
||||||
time.Sleep(quietTime)
|
|
||||||
synctest.Wait()
|
|
||||||
wantCountry(t, f, "KP")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestReplacementSavedBeforeTheWatchStartsIsRead(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
path := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb")
|
|
||||||
writeDatabase(t, path, "DE")
|
|
||||||
|
|
||||||
f := openFile(t, path, newQueue())
|
|
||||||
|
|
||||||
// Saved after OpenFile read the file, and before its directory was
|
|
||||||
// watched, so that no change is seen for it.
|
|
||||||
writeDatabase(t, path, "KP")
|
|
||||||
watch(t, f)
|
|
||||||
time.Sleep(quietTime)
|
|
||||||
synctest.Wait()
|
|
||||||
wantCountry(t, f, "KP")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// newQueue returns a queue of alerts for a webhook that is never sent
|
// newQueue returns a queue of alerts for a webhook that is never sent
|
||||||
// them, so that they wait in it for the test to look at.
|
// them, so that they wait in it for the test to look at.
|
||||||
func newQueue() *alerts.Queue {
|
func newQueue() *alerts.Queue {
|
||||||
@@ -343,28 +270,6 @@ func openFile(t *testing.T, path string, queue *alerts.Queue) *File {
|
|||||||
return f
|
return f
|
||||||
}
|
}
|
||||||
|
|
||||||
// watch runs f's readAfterChanges until the test ends, and returns the
|
|
||||||
// channel that sends it changes.
|
|
||||||
func watch(t *testing.T, f *File) chan<- fsnotify.Event {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
changes := make(chan fsnotify.Event)
|
|
||||||
ctx, stop := context.WithCancel(t.Context())
|
|
||||||
stopped := make(chan struct{})
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
f.readAfterChanges(ctx, changes, nil)
|
|
||||||
close(stopped)
|
|
||||||
}()
|
|
||||||
|
|
||||||
t.Cleanup(func() {
|
|
||||||
stop()
|
|
||||||
<-stopped
|
|
||||||
})
|
|
||||||
|
|
||||||
return changes
|
|
||||||
}
|
|
||||||
|
|
||||||
// wantCountry checks the country f gives testClient.
|
// wantCountry checks the country f gives testClient.
|
||||||
func wantCountry(t *testing.T, f *File, want string) {
|
func wantCountry(t *testing.T, f *File, want string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
@@ -400,13 +305,3 @@ func writeFile(t *testing.T, path, content string) {
|
|||||||
t.Fatalf("write %s: %v", path, err)
|
t.Fatalf("write %s: %v", path, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// write writes data to the end of file.
|
|
||||||
func write(t *testing.T, file *os.File, data []byte) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
_, err := file.Write(data)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("write: %v", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -14,7 +14,6 @@ import (
|
|||||||
"github.com/prometheus/client_golang/prometheus/promhttp"
|
"github.com/prometheus/client_golang/prometheus/promhttp"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/remotelog"
|
"sneak.berlin/go/smallwebwaf/internal/remotelog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
@@ -185,7 +184,7 @@ func (m *Metrics) AddBansAndClients(
|
|||||||
ledger *bans.Ledger, limiter *ratelimit.Limiter, now func() time.Time,
|
ledger *bans.Ledger, limiter *ratelimit.Limiter, now func() time.Time,
|
||||||
) {
|
) {
|
||||||
for _, cause := range []string{
|
for _, cause := range []string{
|
||||||
bans.CauseLimit, bans.CauseAttack, bans.CauseAdmin,
|
bans.CauseLimit, bans.CauseAttack, bans.CauseAdmin, bans.CauseAbuseIPDB,
|
||||||
} {
|
} {
|
||||||
m.registry.MustRegister(
|
m.registry.MustRegister(
|
||||||
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
prometheus.NewCounterFunc(prometheus.CounterOpts{
|
||||||
@@ -274,8 +273,7 @@ func (m *Metrics) AddRemoteLog(remote *remotelog.Sender) {
|
|||||||
// AddLookupFile adds the metrics of the lookup database, read as the
|
// AddLookupFile adds the metrics of the lookup database, read as the
|
||||||
// metrics are asked for: when the file in use was read, which lastRead
|
// metrics are asked for: when the file in use was read, which lastRead
|
||||||
// returns, and the replacements of it that could not be read, which
|
// returns, and the replacements of it that could not be read, which
|
||||||
// readFailures returns. The lookup package's File, which has both, cannot
|
// readFailures returns.
|
||||||
// be named here: that package counts GeoJS's requests in these metrics.
|
|
||||||
func (m *Metrics) AddLookupFile(
|
func (m *Metrics) AddLookupFile(
|
||||||
lastRead func() time.Time,
|
lastRead func() time.Time,
|
||||||
readFailures func() int,
|
readFailures func() int,
|
||||||
@@ -302,7 +300,7 @@ const sourceLabel = "source"
|
|||||||
|
|
||||||
// AddReputation adds the metrics of the lists fetched from URLs and of the
|
// AddReputation adds the metrics of the lists fetched from URLs and of the
|
||||||
// DNSBL zones, by source, each list's URL or each zone, its key masked as
|
// DNSBL zones, by source, each list's URL or each zone, its key masked as
|
||||||
// config.MaskZoneKey masks it: the requests whose client a blocklist, a
|
// reputation.MaskZoneKey masks it: the requests whose client a blocklist, a
|
||||||
// zone's verdict or AbuseIPDB's score lists, which ReputationHit counts,
|
// zone's verdict or AbuseIPDB's score lists, which ReputationHit counts,
|
||||||
// and, read from lists and dnsbl as the metrics are asked for, for a list,
|
// and, read from lists and dnsbl as the metrics are asked for, for a list,
|
||||||
// the fetches that failed and when the copy in use was fetched, and for a
|
// the fetches that failed and when the copy in use was fetched, and for a
|
||||||
@@ -321,7 +319,9 @@ func (m *Metrics) AddReputation(
|
|||||||
m.registry.MustRegister(m.reputationHits)
|
m.registry.MustRegister(m.reputationHits)
|
||||||
|
|
||||||
for _, zone := range dnsbl.Zones() {
|
for _, zone := range dnsbl.Zones() {
|
||||||
source := prometheus.Labels{sourceLabel: config.MaskZoneKey(zone)}
|
source := prometheus.Labels{
|
||||||
|
sourceLabel: reputation.MaskZoneKey(zone),
|
||||||
|
}
|
||||||
|
|
||||||
m.addReputationQueries(
|
m.addReputationQueries(
|
||||||
source,
|
source,
|
||||||
|
|||||||
@@ -0,0 +1,44 @@
|
|||||||
|
// Package processlog builds the logger for the process's own messages,
|
||||||
|
// which it writes as JSON lines marked "type":"process". The request log
|
||||||
|
// is package requestlog's.
|
||||||
|
package processlog
|
||||||
|
|
||||||
|
import (
|
||||||
|
"io"
|
||||||
|
"log/slog"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Params are what New needs.
|
||||||
|
type Params struct {
|
||||||
|
// Output is where the lines are written.
|
||||||
|
Output io.Writer
|
||||||
|
// InstanceName, SWW_WAF_INSTANCE_NAME, is each line's instance.
|
||||||
|
InstanceName string
|
||||||
|
// Level, SWW_WAF_LOG_LEVEL, is the least severe level written.
|
||||||
|
Level slog.Level
|
||||||
|
}
|
||||||
|
|
||||||
|
// New returns the logger for the process's own messages: JSON lines on
|
||||||
|
// params.Output, marked "type":"process", with the time in the same form
|
||||||
|
// as a request line's. It writes only the messages at params.Level or more
|
||||||
|
// severe; the request lines requestlog.Write writes are never held back.
|
||||||
|
func New(params Params) *slog.Logger {
|
||||||
|
handler := slog.NewJSONHandler(params.Output, &slog.HandlerOptions{
|
||||||
|
Level: params.Level,
|
||||||
|
ReplaceAttr: func(groups []string, attr slog.Attr) slog.Attr {
|
||||||
|
if attr.Key == slog.TimeKey && len(groups) == 0 {
|
||||||
|
return slog.String(
|
||||||
|
slog.TimeKey,
|
||||||
|
requestlog.FormatTime(attr.Value.Time()),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
return attr
|
||||||
|
},
|
||||||
|
})
|
||||||
|
|
||||||
|
return slog.New(handler).
|
||||||
|
With("type", "process", "instance", params.InstanceName)
|
||||||
|
}
|
||||||
@@ -0,0 +1,105 @@
|
|||||||
|
package processlog_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"encoding/json"
|
||||||
|
"log/slog"
|
||||||
|
"slices"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/processlog"
|
||||||
|
)
|
||||||
|
|
||||||
|
// instanceName is the instance the tests' loggers are built with.
|
||||||
|
const instanceName = "host1/blog"
|
||||||
|
|
||||||
|
func TestProcessLinesAreMarkedProcessAndGiveTheInstance(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
var out bytes.Buffer
|
||||||
|
|
||||||
|
processlog.New(processlog.Params{
|
||||||
|
Output: &out,
|
||||||
|
InstanceName: instanceName,
|
||||||
|
Level: slog.LevelInfo,
|
||||||
|
}).
|
||||||
|
Info("starting",
|
||||||
|
"version", "v1")
|
||||||
|
|
||||||
|
var fields map[string]any
|
||||||
|
|
||||||
|
err := json.Unmarshal(out.Bytes(), &fields)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("decode %q: %v", out.String(), err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if fields["type"] != "process" ||
|
||||||
|
fields["instance"] != "host1/blog" ||
|
||||||
|
fields["msg"] != "starting" ||
|
||||||
|
fields["level"] != "INFO" ||
|
||||||
|
fields["version"] != "v1" {
|
||||||
|
t.Errorf("process line %v", fields)
|
||||||
|
}
|
||||||
|
|
||||||
|
timeText, _ := fields["time"].(string)
|
||||||
|
|
||||||
|
logged, err := time.Parse(time.RFC3339, timeText)
|
||||||
|
if err != nil || !strings.HasSuffix(timeText, "Z") ||
|
||||||
|
len(timeText) != len("2006-01-02T15:04:05.000Z") ||
|
||||||
|
time.Since(logged) > time.Minute {
|
||||||
|
t.Errorf(
|
||||||
|
"process line time %q, want now in UTC with milliseconds",
|
||||||
|
timeText,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProcessLoggerWritesTheMessagesAtItsLevelOrMoreSevere(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
levels := []slog.Level{
|
||||||
|
slog.LevelDebug, slog.LevelInfo, slog.LevelWarn, slog.LevelError,
|
||||||
|
}
|
||||||
|
|
||||||
|
for i, level := range levels {
|
||||||
|
t.Run(level.String(), func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
var out bytes.Buffer
|
||||||
|
|
||||||
|
processLog := processlog.New(processlog.Params{
|
||||||
|
Output: &out,
|
||||||
|
InstanceName: instanceName,
|
||||||
|
Level: level,
|
||||||
|
})
|
||||||
|
for _, at := range levels {
|
||||||
|
processLog.Log(t.Context(), at, "message")
|
||||||
|
}
|
||||||
|
|
||||||
|
var got, want []string
|
||||||
|
|
||||||
|
for line := range strings.Lines(out.String()) {
|
||||||
|
var fields struct {
|
||||||
|
Level string `json:"level"`
|
||||||
|
}
|
||||||
|
|
||||||
|
err := json.Unmarshal([]byte(line), &fields)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("decode %q: %v", line, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
got = append(got, fields.Level)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, written := range levels[i:] {
|
||||||
|
want = append(want, written.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
if !slices.Equal(got, want) {
|
||||||
|
t.Errorf("lines at %v, want %v", got, want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -11,7 +11,7 @@ import (
|
|||||||
func TestHistoryKeepsEveryRequest(t *testing.T) {
|
func TestHistoryKeepsEveryRequest(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
|
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
|
||||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
start := midnight()
|
start := midnight()
|
||||||
|
|
||||||
@@ -57,7 +57,7 @@ func TestHistoryKeepsEveryRequest(t *testing.T) {
|
|||||||
func TestLookupReachesTheHistoryOfAClientInTheTable(t *testing.T) {
|
func TestLookupReachesTheHistoryOfAClientInTheTable(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
|
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
|
||||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
other := netip.MustParsePrefix("198.51.100.7/32")
|
other := netip.MustParsePrefix("198.51.100.7/32")
|
||||||
start := midnight()
|
start := midnight()
|
||||||
@@ -100,7 +100,10 @@ func TestLookupReachesTheHistoryOfAClientInTheTable(t *testing.T) {
|
|||||||
func TestResetKeepsTheHistory(t *testing.T) {
|
func TestResetKeepsTheHistory(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit}, tableSize)
|
limiter := ratelimit.New(ratelimit.Params{
|
||||||
|
Limits: ratelimit.Limits{PerMinute: limit},
|
||||||
|
MaxClients: tableSize,
|
||||||
|
})
|
||||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
start := midnight()
|
start := midnight()
|
||||||
|
|
||||||
@@ -123,7 +126,7 @@ func TestResetKeepsTheHistory(t *testing.T) {
|
|||||||
func TestRequestsAddsUpTheClientsInsideTheNetblock(t *testing.T) {
|
func TestRequestsAddsUpTheClientsInsideTheNetblock(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
|
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
|
||||||
|
|
||||||
for client, requests := range map[string]int{
|
for client, requests := range map[string]int{
|
||||||
"198.51.100.9/32": 2,
|
"198.51.100.9/32": 2,
|
||||||
|
|||||||
@@ -156,16 +156,28 @@ type Request struct {
|
|||||||
BlockedByCoreRuleSet bool
|
BlockedByCoreRuleSet bool
|
||||||
}
|
}
|
||||||
|
|
||||||
// New returns a Limiter for limits, with no client counted yet, whose
|
// Params are what New needs.
|
||||||
// table holds at most maxClients clients (SWW_WAF_MAX_TRACKED_CLIENTS). Past
|
type Params struct {
|
||||||
// it, the least recently seen client is dropped, with its history, and
|
// Limits are what each client's requests and bytes are counted against.
|
||||||
// starts afresh if it comes back.
|
Limits Limits
|
||||||
func New(limits Limits, maxClients int) *Limiter {
|
// MaxClients is the most clients the table holds,
|
||||||
clients, err := simplelru.NewLRU[netip.Prefix, *Client](maxClients, nil)
|
// SWW_WAF_MAX_TRACKED_CLIENTS. Past it, the least recently seen client
|
||||||
|
// is dropped, with its history, and starts afresh if it comes back.
|
||||||
|
MaxClients int
|
||||||
|
}
|
||||||
|
|
||||||
|
// New returns a Limiter with no client counted yet.
|
||||||
|
func New(params Params) *Limiter {
|
||||||
|
clients, err := simplelru.NewLRU[netip.Prefix, *Client](
|
||||||
|
params.MaxClients,
|
||||||
|
nil,
|
||||||
|
)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
panic(err) // NewLRU fails only for a size below one
|
panic(err) // NewLRU fails only for a size below one
|
||||||
}
|
}
|
||||||
|
|
||||||
|
limits := params.Limits
|
||||||
|
|
||||||
return &Limiter{
|
return &Limiter{
|
||||||
windows: [3]window{
|
windows: [3]window{
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -33,10 +33,10 @@ func TestOverARateLimitSaysWhatTheNextRequestWouldDoAndCountsNothing(
|
|||||||
|
|
||||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
now := midnight()
|
now := midnight()
|
||||||
limiter := ratelimit.New(
|
limiter := ratelimit.New(ratelimit.Params{
|
||||||
ratelimit.Limits{PerMinute: limit, PerDay: limit + 1},
|
Limits: ratelimit.Limits{PerMinute: limit, PerDay: limit + 1},
|
||||||
tableSize,
|
MaxClients: tableSize,
|
||||||
)
|
})
|
||||||
|
|
||||||
if limiter.OverARateLimit(client, now) {
|
if limiter.OverARateLimit(client, now) {
|
||||||
t.Error("a client never seen is over a rate limit")
|
t.Error("a client never seen is over a rate limit")
|
||||||
@@ -72,7 +72,7 @@ func TestOverARateLimitSaysWhatTheNextRequestWouldDoAndCountsNothing(
|
|||||||
wantCount(t, limiter, client, now, minute)
|
wantCount(t, limiter, client, now, minute)
|
||||||
|
|
||||||
// A limit that is off is never gone over.
|
// A limit that is off is never gone over.
|
||||||
off := ratelimit.New(ratelimit.Limits{}, tableSize)
|
off := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
|
||||||
off.Count(client, now, whole)
|
off.Count(client, now, whole)
|
||||||
|
|
||||||
if off.OverARateLimit(client, now) {
|
if off.OverARateLimit(client, now) {
|
||||||
@@ -95,7 +95,10 @@ func TestEachWindowRefusesAtItsLimitAndLetsTheClientBack(t *testing.T) {
|
|||||||
t.Run(tc.window, func(t *testing.T) {
|
t.Run(tc.window, func(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
limiter := ratelimit.New(tc.limits, tableSize)
|
limiter := ratelimit.New(ratelimit.Params{
|
||||||
|
Limits: tc.limits,
|
||||||
|
MaxClients: tableSize,
|
||||||
|
})
|
||||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
start := midnight()
|
start := midnight()
|
||||||
quarter := tc.length / 4
|
quarter := tc.length / 4
|
||||||
@@ -126,10 +129,10 @@ func TestEachWindowRefusesAtItsLimitAndLetsTheClientBack(t *testing.T) {
|
|||||||
func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
|
func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
limiter := ratelimit.New(
|
limiter := ratelimit.New(ratelimit.Params{
|
||||||
ratelimit.Limits{PerMinute: limit, PerHour: limit},
|
Limits: ratelimit.Limits{PerMinute: limit, PerHour: limit},
|
||||||
tableSize,
|
MaxClients: tableSize,
|
||||||
)
|
})
|
||||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
start := midnight()
|
start := midnight()
|
||||||
|
|
||||||
@@ -159,10 +162,13 @@ func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
|
|||||||
func TestClientGetsItsPercentageOfEachLimitRoundedDown(t *testing.T) {
|
func TestClientGetsItsPercentageOfEachLimitRoundedDown(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
limiter := ratelimit.New(
|
limiter := ratelimit.New(ratelimit.Params{
|
||||||
ratelimit.Limits{PerMinute: 5, BytesPerDay: math.MaxInt64},
|
Limits: ratelimit.Limits{
|
||||||
tableSize,
|
PerMinute: 5,
|
||||||
)
|
BytesPerDay: math.MaxInt64,
|
||||||
|
},
|
||||||
|
MaxClients: tableSize,
|
||||||
|
})
|
||||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
start := midnight()
|
start := midnight()
|
||||||
|
|
||||||
@@ -203,10 +209,10 @@ func TestZeroPercentIsAZeroAllowanceAndALimitOffStaysOff(t *testing.T) {
|
|||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
// Only the hour has limits: the minute's and the day's are off.
|
// Only the hour has limits: the minute's and the day's are off.
|
||||||
limiter := ratelimit.New(
|
limiter := ratelimit.New(ratelimit.Params{
|
||||||
ratelimit.Limits{PerHour: limit, BytesPerHour: 1000},
|
Limits: ratelimit.Limits{PerHour: limit, BytesPerHour: 1000},
|
||||||
tableSize,
|
MaxClients: tableSize,
|
||||||
)
|
})
|
||||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
start := midnight()
|
start := midnight()
|
||||||
|
|
||||||
@@ -253,7 +259,10 @@ func TestEachByteLimitIsBrokenByTheBytesCounted(t *testing.T) {
|
|||||||
t.Run(tc.window, func(t *testing.T) {
|
t.Run(tc.window, func(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
limiter := ratelimit.New(tc.limits, tableSize)
|
limiter := ratelimit.New(ratelimit.Params{
|
||||||
|
Limits: tc.limits,
|
||||||
|
MaxClients: tableSize,
|
||||||
|
})
|
||||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
|
|
||||||
// 600 bytes are within the limit, 600 more over it.
|
// 600 bytes are within the limit, 600 more over it.
|
||||||
@@ -287,10 +296,10 @@ func TestEachByteLimitIsBrokenByTheBytesCounted(t *testing.T) {
|
|||||||
func TestALimitIsBrokenOnlyByWhatIsAddedToIt(t *testing.T) {
|
func TestALimitIsBrokenOnlyByWhatIsAddedToIt(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
limiter := ratelimit.New(
|
limiter := ratelimit.New(ratelimit.Params{
|
||||||
ratelimit.Limits{PerMinute: 2, BytesPerMinute: 1000},
|
Limits: ratelimit.Limits{PerMinute: 2, BytesPerMinute: 1000},
|
||||||
tableSize,
|
MaxClients: tableSize,
|
||||||
)
|
})
|
||||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
other := netip.MustParsePrefix("203.0.113.10/32")
|
other := netip.MustParsePrefix("203.0.113.10/32")
|
||||||
start := midnight()
|
start := midnight()
|
||||||
@@ -314,7 +323,7 @@ func TestALimitIsBrokenOnlyByWhatIsAddedToIt(t *testing.T) {
|
|||||||
func TestCountGivesTheBytesInEachWindow(t *testing.T) {
|
func TestCountGivesTheBytesInEachWindow(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
|
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
|
||||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
start := midnight()
|
start := midnight()
|
||||||
|
|
||||||
@@ -340,7 +349,10 @@ func TestCountGivesTheBytesInEachWindow(t *testing.T) {
|
|||||||
func TestResetSetsTheBytesBackToZero(t *testing.T) {
|
func TestResetSetsTheBytesBackToZero(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
limiter := ratelimit.New(ratelimit.Limits{BytesPerDay: 1000}, tableSize)
|
limiter := ratelimit.New(ratelimit.Params{
|
||||||
|
Limits: ratelimit.Limits{BytesPerDay: 1000},
|
||||||
|
MaxClients: tableSize,
|
||||||
|
})
|
||||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
start := midnight()
|
start := midnight()
|
||||||
|
|
||||||
@@ -356,7 +368,7 @@ func TestRefusalsOverTheLimitInAMinuteBreakTheRefusalLimit(
|
|||||||
) {
|
) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
|
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
|
||||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
start := midnight()
|
start := midnight()
|
||||||
|
|
||||||
@@ -411,7 +423,7 @@ func TestRefusalsOverTheLimitInAMinuteBreakTheRefusalLimit(
|
|||||||
func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
|
func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
|
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
|
||||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
start := midnight()
|
start := midnight()
|
||||||
|
|
||||||
@@ -438,10 +450,10 @@ func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
|
|||||||
func TestResetSetsTheCountsBackToZero(t *testing.T) {
|
func TestResetSetsTheCountsBackToZero(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
limiter := ratelimit.New(
|
limiter := ratelimit.New(ratelimit.Params{
|
||||||
ratelimit.Limits{PerMinute: limit, PerDay: limit},
|
Limits: ratelimit.Limits{PerMinute: limit, PerDay: limit},
|
||||||
tableSize,
|
MaxClients: tableSize,
|
||||||
)
|
})
|
||||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
start := midnight()
|
start := midnight()
|
||||||
|
|
||||||
@@ -463,7 +475,10 @@ func TestResetSetsTheCountsBackToZero(t *testing.T) {
|
|||||||
func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) {
|
func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
limiter := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize)
|
limiter := ratelimit.New(ratelimit.Params{
|
||||||
|
Limits: ratelimit.Limits{PerHour: limit},
|
||||||
|
MaxClients: tableSize,
|
||||||
|
})
|
||||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
start := midnight()
|
start := midnight()
|
||||||
|
|
||||||
@@ -482,10 +497,10 @@ func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) {
|
|||||||
func TestRefusedRequestsCount(t *testing.T) {
|
func TestRefusedRequestsCount(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
limiter := ratelimit.New(
|
limiter := ratelimit.New(ratelimit.Params{
|
||||||
ratelimit.Limits{PerMinute: limit, PerHour: 2 * limit},
|
Limits: ratelimit.Limits{PerMinute: limit, PerHour: 2 * limit},
|
||||||
tableSize,
|
MaxClients: tableSize,
|
||||||
)
|
})
|
||||||
refused := netip.MustParsePrefix("203.0.113.9/32")
|
refused := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
within := netip.MustParsePrefix("203.0.113.10/32")
|
within := netip.MustParsePrefix("203.0.113.10/32")
|
||||||
start := midnight()
|
start := midnight()
|
||||||
@@ -518,7 +533,10 @@ func TestRefusedRequestsCount(t *testing.T) {
|
|||||||
func TestRequestCountedLateGoesInTheBucketUnderWay(t *testing.T) {
|
func TestRequestCountedLateGoesInTheBucketUnderWay(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit}, tableSize)
|
limiter := ratelimit.New(ratelimit.Params{
|
||||||
|
Limits: ratelimit.Limits{PerMinute: limit},
|
||||||
|
MaxClients: tableSize,
|
||||||
|
})
|
||||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
start := midnight()
|
start := midnight()
|
||||||
|
|
||||||
@@ -534,7 +552,10 @@ func TestRequestCountedLateGoesInTheBucketUnderWay(t *testing.T) {
|
|||||||
func TestClockSetBackStartsTheBucketsAfresh(t *testing.T) {
|
func TestClockSetBackStartsTheBucketsAfresh(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
limiter := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize)
|
limiter := ratelimit.New(ratelimit.Params{
|
||||||
|
Limits: ratelimit.Limits{PerHour: limit},
|
||||||
|
MaxClients: tableSize,
|
||||||
|
})
|
||||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
start := midnight()
|
start := midnight()
|
||||||
|
|
||||||
@@ -562,7 +583,10 @@ func TestKeepsAtMostMaxClientsDroppingTheLeastRecentlySeen(t *testing.T) {
|
|||||||
|
|
||||||
const maxClients = 3
|
const maxClients = 3
|
||||||
|
|
||||||
limiter := ratelimit.New(ratelimit.Limits{PerMinute: 1}, maxClients)
|
limiter := ratelimit.New(ratelimit.Params{
|
||||||
|
Limits: ratelimit.Limits{PerMinute: 1},
|
||||||
|
MaxClients: maxClients,
|
||||||
|
})
|
||||||
now := midnight()
|
now := midnight()
|
||||||
|
|
||||||
clients := make([]netip.Prefix, maxClients+1)
|
clients := make([]netip.Prefix, maxClients+1)
|
||||||
|
|||||||
@@ -19,7 +19,7 @@ func TestSnapshotListsTheClientsByAddress(t *testing.T) {
|
|||||||
"2001:db8::/64",
|
"2001:db8::/64",
|
||||||
}
|
}
|
||||||
|
|
||||||
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
|
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
|
||||||
for _, i := range []int{2, 3, 0, 1} {
|
for _, i := range []int{2, 3, 0, 1} {
|
||||||
limiter.Count(netip.MustParsePrefix(want[i]), midnight(), whole)
|
limiter.Count(netip.MustParsePrefix(want[i]), midnight(), whole)
|
||||||
}
|
}
|
||||||
@@ -52,7 +52,10 @@ func TestLoadedCountsCarryOn(t *testing.T) {
|
|||||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
start := midnight()
|
start := midnight()
|
||||||
|
|
||||||
before := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize)
|
before := ratelimit.New(ratelimit.Params{
|
||||||
|
Limits: ratelimit.Limits{PerHour: limit},
|
||||||
|
MaxClients: tableSize,
|
||||||
|
})
|
||||||
for range limit {
|
for range limit {
|
||||||
wantCount(t, before, client, start, "")
|
wantCount(t, before, client, start, "")
|
||||||
}
|
}
|
||||||
@@ -60,7 +63,10 @@ func TestLoadedCountsCarryOn(t *testing.T) {
|
|||||||
// Loaded into a new limiter, as across a restart, the client has no
|
// Loaded into a new limiter, as across a restart, the client has no
|
||||||
// fresh allowance.
|
// fresh allowance.
|
||||||
later := start.Add(time.Minute)
|
later := start.Add(time.Minute)
|
||||||
after := ratelimit.New(ratelimit.Limits{PerHour: limit}, tableSize)
|
after := ratelimit.New(ratelimit.Params{
|
||||||
|
Limits: ratelimit.Limits{PerHour: limit},
|
||||||
|
MaxClients: tableSize,
|
||||||
|
})
|
||||||
after.Load(before.Snapshot(), later)
|
after.Load(before.Snapshot(), later)
|
||||||
wantCount(t, after, client, later, hour)
|
wantCount(t, after, client, later, hour)
|
||||||
}
|
}
|
||||||
@@ -71,7 +77,7 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
|
|||||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
start := midnight()
|
start := midnight()
|
||||||
|
|
||||||
limiter := ratelimit.New(ratelimit.Limits{}, tableSize)
|
limiter := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
|
||||||
limiter.Count(client, start, whole)
|
limiter.Count(client, start, whole)
|
||||||
limiter.CountBytes(client, start, 5, whole)
|
limiter.CountBytes(client, start, 5, whole)
|
||||||
limiter.CountRefusal(client, start, limit)
|
limiter.CountRefusal(client, start, limit)
|
||||||
@@ -80,7 +86,7 @@ func TestLoadEmptiesBucketsWhoseTimeHasPassed(t *testing.T) {
|
|||||||
loaded := func(now time.Time) ratelimit.Client {
|
loaded := func(now time.Time) ratelimit.Client {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
after := ratelimit.New(ratelimit.Limits{}, tableSize)
|
after := ratelimit.New(ratelimit.Params{MaxClients: tableSize})
|
||||||
after.Load(limiter.Snapshot(), now)
|
after.Load(limiter.Snapshot(), now)
|
||||||
|
|
||||||
return after.Snapshot()[0]
|
return after.Snapshot()[0]
|
||||||
@@ -142,7 +148,7 @@ func TestLoadDropsTheLeastRecentlySeenFirst(t *testing.T) {
|
|||||||
addr = addr.Next()
|
addr = addr.Next()
|
||||||
}
|
}
|
||||||
|
|
||||||
limiter := ratelimit.New(ratelimit.Limits{}, maxClients)
|
limiter := ratelimit.New(ratelimit.Params{MaxClients: maxClients})
|
||||||
limiter.Load(clients, midnight())
|
limiter.Load(clients, midnight())
|
||||||
|
|
||||||
got := limiter.Snapshot()
|
got := limiter.Snapshot()
|
||||||
|
|||||||
@@ -44,7 +44,7 @@ import (
|
|||||||
const (
|
const (
|
||||||
// started is the time a record made as a test starts gives.
|
// started is the time a record made as a test starts gives.
|
||||||
started = "2000-01-01T00:00:00.000Z"
|
started = "2000-01-01T00:00:00.000Z"
|
||||||
appName = "fsn1app1/blog"
|
appName = "host1/blog"
|
||||||
// local0 is the number of the default facility, and local0Info the
|
// local0 is the number of the default facility, and local0Info the
|
||||||
// priority of its records.
|
// priority of its records.
|
||||||
local0 = 16
|
local0 = 16
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ import (
|
|||||||
"net/netip"
|
"net/netip"
|
||||||
"net/url"
|
"net/url"
|
||||||
"slices"
|
"slices"
|
||||||
|
"strconv"
|
||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -23,6 +24,9 @@ const (
|
|||||||
// AbuseIPDBURL is where clients are checked: the check endpoint of
|
// AbuseIPDBURL is where clients are checked: the check endpoint of
|
||||||
// AbuseIPDB's API.
|
// AbuseIPDB's API.
|
||||||
AbuseIPDBURL = "https://api.abuseipdb.com/api/v2/check"
|
AbuseIPDBURL = "https://api.abuseipdb.com/api/v2/check"
|
||||||
|
// AbuseIPDBBlacklistURL is where AbuseIPDB's blacklist is fetched from:
|
||||||
|
// the blacklist endpoint of AbuseIPDB's API.
|
||||||
|
AbuseIPDBBlacklistURL = "https://api.abuseipdb.com/api/v2/blacklist"
|
||||||
// AbuseIPDBSource is how the request log, the alerts and the metrics
|
// AbuseIPDBSource is how the request log, the alerts and the metrics
|
||||||
// name AbuseIPDB.
|
// name AbuseIPDB.
|
||||||
AbuseIPDBSource = "abuseipdb"
|
AbuseIPDBSource = "abuseipdb"
|
||||||
@@ -32,6 +36,8 @@ const (
|
|||||||
|
|
||||||
var (
|
var (
|
||||||
errNoScore = errors.New("the answer gives no abuseConfidenceScore")
|
errNoScore = errors.New("the answer gives no abuseConfidenceScore")
|
||||||
|
errNoBlacklist = errors.New("the answer gives no data")
|
||||||
|
errNotAddress = errors.New("is not an address")
|
||||||
errBudgetUsedUp = errors.New(
|
errBudgetUsedUp = errors.New(
|
||||||
"checks spent; none is made until the day ends at 00:00 UTC")
|
"checks spent; none is made until the day ends at 00:00 UTC")
|
||||||
)
|
)
|
||||||
@@ -53,6 +59,13 @@ type ChecksSpent struct {
|
|||||||
Spent int
|
Spent int
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// BlacklistEntry is a client on AbuseIPDB's blacklist: its address and its
|
||||||
|
// abuse confidence score.
|
||||||
|
type BlacklistEntry struct {
|
||||||
|
Addr netip.Addr
|
||||||
|
Score int64
|
||||||
|
}
|
||||||
|
|
||||||
// AbuseIPDBParams are what NewAbuseIPDB needs.
|
// AbuseIPDBParams are what NewAbuseIPDB needs.
|
||||||
type AbuseIPDBParams struct {
|
type AbuseIPDBParams struct {
|
||||||
// URL is where clients are checked, normally AbuseIPDBURL, with Key,
|
// URL is where clients are checked, normally AbuseIPDBURL, with Key,
|
||||||
@@ -69,10 +82,19 @@ type AbuseIPDBParams struct {
|
|||||||
// take (SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS).
|
// take (SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS).
|
||||||
CacheTTL time.Duration
|
CacheTTL time.Duration
|
||||||
Timeout time.Duration
|
Timeout time.Duration
|
||||||
|
// BlacklistURL is where the blacklist is fetched from, normally
|
||||||
|
// AbuseIPDBBlacklistURL, with the clients scored Threshold or more
|
||||||
|
// (SWW_WAF_ABUSEIPDB_THRESHOLD), every BlacklistRefresh
|
||||||
|
// (SWW_WAF_ABUSEIPDB_BLACKLIST_REFRESH_INTERVAL_MINS). Blacklisted
|
||||||
|
// receives the clients on each blacklist fetched.
|
||||||
|
BlacklistURL string
|
||||||
|
Threshold int64
|
||||||
|
BlacklistRefresh time.Duration
|
||||||
|
Blacklisted func([]BlacklistEntry)
|
||||||
// Now tells the time, normally time.Now in UTC.
|
// Now tells the time, normally time.Now in UTC.
|
||||||
Now func() time.Time
|
Now func() time.Time
|
||||||
// ProcessLog receives each check that fails, and why, and the day's
|
// ProcessLog receives each check or fetch of the blacklist that fails,
|
||||||
// budget used up.
|
// and why, and the day's budget used up.
|
||||||
ProcessLog *slog.Logger
|
ProcessLog *slog.Logger
|
||||||
// Alerts receive a source_failure alert for each.
|
// Alerts receive a source_failure alert for each.
|
||||||
Alerts *alerts.Queue
|
Alerts *alerts.Queue
|
||||||
@@ -88,8 +110,12 @@ type AbuseIPDB struct {
|
|||||||
// scores are by client. Each is added as it is fetched and never moved
|
// scores are by client. Each is added as it is fetched and never moved
|
||||||
// up, so that the one fetched longest ago is the first dropped.
|
// up, so that the one fetched longest ago is the first dropped.
|
||||||
scores *simplelru.LRU[netip.Prefix, Score]
|
scores *simplelru.LRU[netip.Prefix, Score]
|
||||||
// checking are the clients whose check is under way.
|
// checking are the clients whose check is under way, each with the
|
||||||
checking map[netip.Prefix]bool
|
// function that ends it. stopped is true once StopChecking has been
|
||||||
|
// called, and checkGoroutines waits for the goroutine of each check.
|
||||||
|
checking map[netip.Prefix]context.CancelFunc
|
||||||
|
stopped bool
|
||||||
|
checkGoroutines sync.WaitGroup
|
||||||
// day is the day, in UTC, of the checks spent counts.
|
// day is the day, in UTC, of the checks spent counts.
|
||||||
day civil.Date
|
day civil.Date
|
||||||
spent int
|
spent int
|
||||||
@@ -112,7 +138,7 @@ func NewAbuseIPDB(params AbuseIPDBParams) *AbuseIPDB {
|
|||||||
params: params,
|
params: params,
|
||||||
httpClient: &http.Client{},
|
httpClient: &http.Client{},
|
||||||
scores: scores,
|
scores: scores,
|
||||||
checking: map[netip.Prefix]bool{},
|
checking: map[netip.Prefix]context.CancelFunc{},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -122,11 +148,11 @@ func NewAbuseIPDB(params AbuseIPDBParams) *AbuseIPDB {
|
|||||||
// addresses its request comes from. A client without one is checked in
|
// addresses its request comes from. A client without one is checked in
|
||||||
// the background, by addr, the address its request came from, if
|
// the background, by addr, the address its request came from, if
|
||||||
// offender, if it has committed an offence, unless its check is under
|
// offender, if it has committed an offence, unless its check is under
|
||||||
// way, a check failed less than failureDelay ago, or the day's checks
|
// way, a check failed less than failureDelay ago, the day's checks have
|
||||||
// have used up DailyBudget; Hit never waits for a check. The check that
|
// used up DailyBudget, or StopChecking has been called; Hit never waits
|
||||||
// uses the budget up is logged and raised as a source_failure alert. ctx
|
// for a check. The check that uses the budget up is logged and raised as a
|
||||||
// is the context of the client's request, and a check goes on after the
|
// source_failure alert. ctx is the context of the client's request, and a
|
||||||
// request ends.
|
// check goes on after the request ends, until StopChecking.
|
||||||
func (a *AbuseIPDB) Hit(
|
func (a *AbuseIPDB) Hit(
|
||||||
ctx context.Context, client netip.Prefix, addr netip.Addr, offender bool,
|
ctx context.Context, client netip.Prefix, addr netip.Addr, offender bool,
|
||||||
) (int64, bool) {
|
) (int64, bool) {
|
||||||
@@ -145,14 +171,21 @@ func (a *AbuseIPDB) Hit(
|
|||||||
a.day, a.spent = today, 0
|
a.day, a.spent = today, 0
|
||||||
}
|
}
|
||||||
|
|
||||||
check := offender && !a.checking[client] && !now.Before(a.retryAt) &&
|
_, checking := a.checking[client]
|
||||||
a.spent < a.params.DailyBudget
|
|
||||||
|
check := offender && !checking && !a.stopped &&
|
||||||
|
!now.Before(a.retryAt) && a.spent < a.params.DailyBudget
|
||||||
if check {
|
if check {
|
||||||
a.checking[client] = true
|
checkCtx, end := context.WithCancel(context.WithoutCancel(ctx))
|
||||||
|
a.checking[client] = end
|
||||||
a.checks++
|
a.checks++
|
||||||
a.spent++
|
a.spent++
|
||||||
|
|
||||||
go a.check(context.WithoutCancel(ctx), client, addr)
|
a.checkGoroutines.Go(func() {
|
||||||
|
defer end()
|
||||||
|
|
||||||
|
a.check(checkCtx, client, addr)
|
||||||
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
usedUp := check && a.spent == a.params.DailyBudget
|
usedUp := check && a.spent == a.params.DailyBudget
|
||||||
@@ -167,6 +200,24 @@ func (a *AbuseIPDB) Hit(
|
|||||||
return 0, false
|
return 0, false
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// StopChecking ends the checks under way, and returns once their goroutines
|
||||||
|
// have. A check it ends gives no score and is no failure: it is neither
|
||||||
|
// counted, logged nor raised as an alert, and checks are not left alone
|
||||||
|
// after it. The ended check stays spent from the day's budget. After
|
||||||
|
// StopChecking, Hit checks no client.
|
||||||
|
func (a *AbuseIPDB) StopChecking() {
|
||||||
|
a.mu.Lock()
|
||||||
|
|
||||||
|
a.stopped = true
|
||||||
|
for _, end := range a.checking {
|
||||||
|
end()
|
||||||
|
}
|
||||||
|
|
||||||
|
a.mu.Unlock()
|
||||||
|
|
||||||
|
a.checkGoroutines.Wait()
|
||||||
|
}
|
||||||
|
|
||||||
// Checked returns how many checks were made.
|
// Checked returns how many checks were made.
|
||||||
func (a *AbuseIPDB) Checked() int {
|
func (a *AbuseIPDB) Checked() int {
|
||||||
a.mu.Lock()
|
a.mu.Lock()
|
||||||
@@ -256,10 +307,45 @@ func (a *AbuseIPDB) LoadChecksSpent(checks ChecksSpent) {
|
|||||||
a.day, a.spent = checks.Day, checks.Spent
|
a.day, a.spent = checks.Day, checks.Spent
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// RunBlacklist fetches AbuseIPDB's blacklist at once, and again every
|
||||||
|
// BlacklistRefresh, until ctx is done, and gives the clients on each copy
|
||||||
|
// fetched to Blacklisted. While Key is unset it does nothing. A fetch that
|
||||||
|
// fails is logged and raised as a source_failure alert, unless it was cut
|
||||||
|
// off as ctx ended.
|
||||||
|
func (a *AbuseIPDB) RunBlacklist(ctx context.Context) {
|
||||||
|
if a.params.Key == "" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
for {
|
||||||
|
listed, err := a.fetchBlacklist(ctx)
|
||||||
|
|
||||||
|
switch {
|
||||||
|
case ctx.Err() != nil:
|
||||||
|
return
|
||||||
|
case err != nil:
|
||||||
|
a.alert("fetching AbuseIPDB's blacklist failed", err)
|
||||||
|
default:
|
||||||
|
a.params.Blacklisted(listed)
|
||||||
|
}
|
||||||
|
|
||||||
|
timer := time.NewTimer(a.params.BlacklistRefresh)
|
||||||
|
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
timer.Stop()
|
||||||
|
|
||||||
|
return
|
||||||
|
case <-timer.C:
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// check checks client with AbuseIPDB by addr, one of its addresses, keeps
|
// check checks client with AbuseIPDB by addr, one of its addresses, keeps
|
||||||
// the score as client's, and notes the check as no longer under way. A
|
// the score as client's, and notes the check as no longer under way. A
|
||||||
// check that fails gives no score: it is counted, logged and raised as a
|
// check that fails gives no score: it is counted, logged and raised as a
|
||||||
// source_failure alert, and no client is checked for failureDelay.
|
// source_failure alert, and no client is checked for failureDelay. A check
|
||||||
|
// ended as ctx ends, by StopChecking, is no failure.
|
||||||
func (a *AbuseIPDB) check(
|
func (a *AbuseIPDB) check(
|
||||||
ctx context.Context,
|
ctx context.Context,
|
||||||
client netip.Prefix,
|
client netip.Prefix,
|
||||||
@@ -267,24 +353,26 @@ func (a *AbuseIPDB) check(
|
|||||||
) {
|
) {
|
||||||
score, err := a.ask(ctx, addr)
|
score, err := a.ask(ctx, addr)
|
||||||
now := a.params.Now()
|
now := a.params.Now()
|
||||||
|
isFailure := err != nil && ctx.Err() == nil
|
||||||
|
|
||||||
a.mu.Lock()
|
a.mu.Lock()
|
||||||
|
|
||||||
delete(a.checking, client)
|
delete(a.checking, client)
|
||||||
|
|
||||||
if err == nil {
|
switch {
|
||||||
|
case err == nil:
|
||||||
a.scores.Add(
|
a.scores.Add(
|
||||||
client,
|
client,
|
||||||
Score{Client: client, Score: score, Fetched: now},
|
Score{Client: client, Score: score, Fetched: now},
|
||||||
)
|
)
|
||||||
} else {
|
case isFailure:
|
||||||
a.failures++
|
a.failures++
|
||||||
a.retryAt = now.Add(failureDelay)
|
a.retryAt = now.Add(failureDelay)
|
||||||
}
|
}
|
||||||
|
|
||||||
a.mu.Unlock()
|
a.mu.Unlock()
|
||||||
|
|
||||||
if err != nil {
|
if isFailure {
|
||||||
a.alert("checking a client with AbuseIPDB failed", err)
|
a.alert("checking a client with AbuseIPDB failed", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -344,6 +432,94 @@ func (a *AbuseIPDB) ask(
|
|||||||
return *answer.Data.AbuseConfidenceScore, nil
|
return *answer.Data.AbuseConfidenceScore, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// fetchBlacklist fetches the clients on AbuseIPDB's blacklist scored
|
||||||
|
// Threshold or more, sending the key in the header Key. An answer other
|
||||||
|
// than 200, one longer than maxListBytes or without the list, an entry
|
||||||
|
// whose address does not read, and none within fetchTimeout, fail.
|
||||||
|
func (a *AbuseIPDB) fetchBlacklist(
|
||||||
|
ctx context.Context,
|
||||||
|
) ([]BlacklistEntry, error) {
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, fetchTimeout)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
query := url.Values{
|
||||||
|
"confidenceMinimum": {strconv.FormatInt(a.params.Threshold, 10)},
|
||||||
|
}
|
||||||
|
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodGet,
|
||||||
|
a.params.BlacklistURL+"?"+query.Encode(), http.NoBody)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("make the request: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
req.Header.Set("Key", a.params.Key)
|
||||||
|
req.Header.Set("Accept", "application/json")
|
||||||
|
|
||||||
|
res, err := a.httpClient.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("fetch the blacklist: %w", errors.Unwrap(err))
|
||||||
|
}
|
||||||
|
|
||||||
|
defer func() {
|
||||||
|
_ = res.Body.Close()
|
||||||
|
}()
|
||||||
|
|
||||||
|
if res.StatusCode != http.StatusOK {
|
||||||
|
return nil, fmt.Errorf("%w %s", errStatus, res.Status)
|
||||||
|
}
|
||||||
|
|
||||||
|
body, err := io.ReadAll(io.LimitReader(res.Body, maxListBytes+1))
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("read the blacklist: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(body) > maxListBytes {
|
||||||
|
return nil, errTooLong
|
||||||
|
}
|
||||||
|
|
||||||
|
return parseBlacklist(body)
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseBlacklist reads body, AbuseIPDB's answer with its blacklist.
|
||||||
|
func parseBlacklist(body []byte) ([]BlacklistEntry, error) {
|
||||||
|
var answer struct {
|
||||||
|
Data *[]struct {
|
||||||
|
IPAddress string `json:"ipAddress"`
|
||||||
|
AbuseConfidenceScore int64 `json:"abuseConfidenceScore"`
|
||||||
|
} `json:"data"`
|
||||||
|
}
|
||||||
|
|
||||||
|
err := json.Unmarshal(body, &answer)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("read the blacklist: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if answer.Data == nil {
|
||||||
|
return nil, errNoBlacklist
|
||||||
|
}
|
||||||
|
|
||||||
|
listed := make([]BlacklistEntry, 0, len(*answer.Data))
|
||||||
|
|
||||||
|
for i, entry := range *answer.Data {
|
||||||
|
addr, err := netip.ParseAddr(entry.IPAddress)
|
||||||
|
if err != nil || addr.Zone() != "" {
|
||||||
|
// The error leaves out what the entry holds, as it does a
|
||||||
|
// client's address.
|
||||||
|
return nil, fmt.Errorf(
|
||||||
|
"entry %d's ipAddress %w",
|
||||||
|
i+1,
|
||||||
|
errNotAddress,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
listed = append(listed, BlacklistEntry{
|
||||||
|
Addr: addr.Unmap(), Score: entry.AbuseConfidenceScore,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
return listed, nil
|
||||||
|
}
|
||||||
|
|
||||||
// alert raises a source_failure alert from AbuseIPDB with reason and err,
|
// alert raises a source_failure alert from AbuseIPDB with reason and err,
|
||||||
// and logs them.
|
// and logs them.
|
||||||
func (a *AbuseIPDB) alert(reason string, err error) {
|
func (a *AbuseIPDB) alert(reason string, err error) {
|
||||||
|
|||||||
@@ -394,6 +394,43 @@ func TestCheckNotAnsweredWithinTheTimeoutFailsAndHitNeverWaits(
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestStopCheckingEndsTheCheckUnderWayAsNoFailureAndChecksNoMore(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
queue := newQueue()
|
||||||
|
p := abuseIPDBParams()
|
||||||
|
p.Alerts = queue
|
||||||
|
abuseIPDB := &abuseIPDBStandIn{hanging: true}
|
||||||
|
checker := newAbuseIPDB(abuseIPDB, p)
|
||||||
|
|
||||||
|
wantScore(t, checker, suspect, true, 0, false)
|
||||||
|
synctest.Wait()
|
||||||
|
|
||||||
|
// The check is ended, not waited out, and is no failure.
|
||||||
|
began := time.Now()
|
||||||
|
|
||||||
|
checker.StopChecking()
|
||||||
|
|
||||||
|
if waited := time.Since(began); waited != 0 {
|
||||||
|
t.Errorf("StopChecking returned after %s, want at once", waited)
|
||||||
|
}
|
||||||
|
|
||||||
|
wantFailures(t, checker, 0)
|
||||||
|
|
||||||
|
if got := waiting(queue); len(got) != 0 {
|
||||||
|
t.Errorf("alerts waiting %+v, want none", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
// An offender without a score is then not checked.
|
||||||
|
wantScore(t, checker, other, true, 0, false)
|
||||||
|
synctest.Wait()
|
||||||
|
wantChecked(t, abuseIPDB, suspect)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
func TestKeyIsSentInTheKeyHeaderAndNeverShown(t *testing.T) {
|
func TestKeyIsSentInTheKeyHeaderAndNeverShown(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,252 @@
|
|||||||
|
package reputation_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"net/netip"
|
||||||
|
"slices"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"testing/synctest"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The tests of the blacklist run in synctest bubbles, as those of the
|
||||||
|
// checks do, with AbuseIPDB a stand-in reached without the network.
|
||||||
|
|
||||||
|
// blacklistURL is where the tests fetch the blacklist from.
|
||||||
|
const blacklistURL = "https://abuseipdb.example/api/v2/blacklist"
|
||||||
|
|
||||||
|
func TestBlacklistIsFetchedAtOnceAndEveryRefresh(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
abuseIPDB := &blacklistStandIn{body: `{"meta":{},"data":[` +
|
||||||
|
`{"ipAddress":"203.0.113.9","abuseConfidenceScore":100},` +
|
||||||
|
`{"ipAddress":"::ffff:198.51.100.7","abuseConfidenceScore":95},` +
|
||||||
|
`{"ipAddress":"2001:db8::9","abuseConfidenceScore":90}]}`}
|
||||||
|
got := runBlacklist(t, abuseIPDB, key, newQueue())
|
||||||
|
|
||||||
|
want := []reputation.BlacklistEntry{
|
||||||
|
{Addr: netip.MustParseAddr("203.0.113.9"), Score: 100},
|
||||||
|
{Addr: netip.MustParseAddr("198.51.100.7"), Score: 95},
|
||||||
|
{Addr: netip.MustParseAddr("2001:db8::9"), Score: 90},
|
||||||
|
}
|
||||||
|
|
||||||
|
synctest.Wait()
|
||||||
|
got.want(t, want)
|
||||||
|
|
||||||
|
time.Sleep(time.Hour - time.Nanosecond)
|
||||||
|
synctest.Wait()
|
||||||
|
got.want(t, want)
|
||||||
|
|
||||||
|
time.Sleep(time.Nanosecond)
|
||||||
|
synctest.Wait()
|
||||||
|
got.want(t, want, want)
|
||||||
|
|
||||||
|
// Each fetch asks for the clients scored 90 or more, with the key
|
||||||
|
// in the header Key alone.
|
||||||
|
abuseIPDB.mu.Lock()
|
||||||
|
defer abuseIPDB.mu.Unlock()
|
||||||
|
|
||||||
|
for _, req := range abuseIPDB.asked {
|
||||||
|
if req.URL.String() != blacklistURL+"?confidenceMinimum=90" ||
|
||||||
|
req.Header.Get("Key") != key {
|
||||||
|
t.Errorf(
|
||||||
|
"asked %s with the key %q, want %s?confidenceMinimum=90 "+
|
||||||
|
"with %q",
|
||||||
|
req.URL,
|
||||||
|
req.Header.Get("Key"),
|
||||||
|
blacklistURL,
|
||||||
|
key,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFailedBlacklistFetchIsAlertedAndTriedAgainAtTheNextRefresh(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
for _, tc := range []struct{ name, body, error string }{
|
||||||
|
{
|
||||||
|
"a refusal", "",
|
||||||
|
"the server answered 429 Too Many Requests",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"an answer without the list", `{"errors":[]}`,
|
||||||
|
"the answer gives no data",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"an entry that is not an address",
|
||||||
|
`{"data":[{"ipAddress":"203.0.113.9"},{"ipAddress":"example"}]}`,
|
||||||
|
"entry 2's ipAddress is not an address",
|
||||||
|
},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
abuseIPDB := &blacklistStandIn{body: tc.body}
|
||||||
|
if tc.body == "" {
|
||||||
|
abuseIPDB.status = http.StatusTooManyRequests
|
||||||
|
}
|
||||||
|
|
||||||
|
queue := newQueue()
|
||||||
|
got := runBlacklist(t, abuseIPDB, key, queue)
|
||||||
|
|
||||||
|
synctest.Wait()
|
||||||
|
|
||||||
|
got.want(t)
|
||||||
|
|
||||||
|
alerted := waiting(queue)
|
||||||
|
if len(alerted) != 1 ||
|
||||||
|
alerted[0].Event != alerts.EventSourceFailure ||
|
||||||
|
alerted[0].Reason != "fetching AbuseIPDB's blacklist failed" ||
|
||||||
|
alerted[0].Detail["source"] != reputation.AbuseIPDBSource ||
|
||||||
|
alerted[0].Detail["error"] != tc.error {
|
||||||
|
t.Errorf(
|
||||||
|
"alerts %+v, want the failure %q",
|
||||||
|
alerted,
|
||||||
|
tc.error,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
abuseIPDB.answer(`{"data":[]}`)
|
||||||
|
time.Sleep(time.Hour)
|
||||||
|
synctest.Wait()
|
||||||
|
got.want(t, []reputation.BlacklistEntry{})
|
||||||
|
})
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBlacklistIsNotFetchedWithoutAKey(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
abuseIPDB := &blacklistStandIn{body: `{"data":[]}`}
|
||||||
|
got := runBlacklist(t, abuseIPDB, "", newQueue())
|
||||||
|
|
||||||
|
time.Sleep(time.Hour)
|
||||||
|
synctest.Wait()
|
||||||
|
got.want(t)
|
||||||
|
|
||||||
|
if len(abuseIPDB.asked) != 0 {
|
||||||
|
t.Errorf("fetched the blacklist %d times, want none",
|
||||||
|
len(abuseIPDB.asked))
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// blacklistStandIn is a stand-in for AbuseIPDB that answers each fetch of
|
||||||
|
// the blacklist with status, 200 when it is zero, and body, and keeps the
|
||||||
|
// requests.
|
||||||
|
type blacklistStandIn struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
status int
|
||||||
|
body string
|
||||||
|
asked []*http.Request
|
||||||
|
}
|
||||||
|
|
||||||
|
// RoundTrip has the stand-in answer req, in place of the network.
|
||||||
|
func (s *blacklistStandIn) RoundTrip(
|
||||||
|
req *http.Request,
|
||||||
|
) (*http.Response, error) {
|
||||||
|
s.mu.Lock()
|
||||||
|
s.asked = append(s.asked, req)
|
||||||
|
status, body := s.status, s.body
|
||||||
|
s.mu.Unlock()
|
||||||
|
|
||||||
|
if status == 0 {
|
||||||
|
status = http.StatusOK
|
||||||
|
}
|
||||||
|
|
||||||
|
return &http.Response{
|
||||||
|
StatusCode: status,
|
||||||
|
Status: fmt.Sprintf("%d %s", status, http.StatusText(status)),
|
||||||
|
Header: http.Header{},
|
||||||
|
Body: io.NopCloser(strings.NewReader(body)),
|
||||||
|
Request: req,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// answer has the stand-in answer each fetch with 200 and body.
|
||||||
|
func (s *blacklistStandIn) answer(body string) {
|
||||||
|
s.mu.Lock()
|
||||||
|
defer s.mu.Unlock()
|
||||||
|
|
||||||
|
s.status, s.body = 0, body
|
||||||
|
}
|
||||||
|
|
||||||
|
// blacklisted holds the clients on each blacklist given to Blacklisted.
|
||||||
|
type blacklisted struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
fetched [][]reputation.BlacklistEntry
|
||||||
|
}
|
||||||
|
|
||||||
|
// want checks the clients on each blacklist given so far.
|
||||||
|
func (b *blacklisted) want(
|
||||||
|
t *testing.T,
|
||||||
|
want ...[]reputation.BlacklistEntry,
|
||||||
|
) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
b.mu.Lock()
|
||||||
|
defer b.mu.Unlock()
|
||||||
|
|
||||||
|
if !slices.EqualFunc(b.fetched, want, slices.Equal) {
|
||||||
|
t.Errorf("blacklists %v, want %v", b.fetched, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// runBlacklist runs RunBlacklist, until the test ends, with the key
|
||||||
|
// accountKey, the threshold 90 and a refresh of an hour, fetching from
|
||||||
|
// abuseIPDB and raising alerts into queue, and returns what it gives
|
||||||
|
// Blacklisted.
|
||||||
|
func runBlacklist(
|
||||||
|
t *testing.T, abuseIPDB *blacklistStandIn, accountKey string,
|
||||||
|
queue *alerts.Queue,
|
||||||
|
) *blacklisted {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
got := &blacklisted{}
|
||||||
|
p := abuseIPDBParams()
|
||||||
|
p.Key = accountKey
|
||||||
|
p.BlacklistURL = blacklistURL
|
||||||
|
p.Threshold = 90
|
||||||
|
p.BlacklistRefresh = time.Hour
|
||||||
|
p.Alerts = queue
|
||||||
|
p.Blacklisted = func(listed []reputation.BlacklistEntry) {
|
||||||
|
got.mu.Lock()
|
||||||
|
defer got.mu.Unlock()
|
||||||
|
|
||||||
|
got.fetched = append(got.fetched, listed)
|
||||||
|
}
|
||||||
|
|
||||||
|
checker := reputation.NewAbuseIPDB(p)
|
||||||
|
checker.SetTransport(abuseIPDB)
|
||||||
|
|
||||||
|
ctx, cancel := context.WithCancel(t.Context())
|
||||||
|
done := make(chan struct{})
|
||||||
|
|
||||||
|
go func() {
|
||||||
|
checker.RunBlacklist(ctx)
|
||||||
|
close(done)
|
||||||
|
}()
|
||||||
|
|
||||||
|
t.Cleanup(func() {
|
||||||
|
cancel()
|
||||||
|
<-done
|
||||||
|
})
|
||||||
|
|
||||||
|
return got
|
||||||
|
}
|
||||||
@@ -16,7 +16,6 @@ import (
|
|||||||
|
|
||||||
"github.com/hashicorp/golang-lru/v2/simplelru"
|
"github.com/hashicorp/golang-lru/v2/simplelru"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -32,6 +31,8 @@ const (
|
|||||||
// it fails, and no client is checked with AbuseIPDB after a check
|
// it fails, and no client is checked with AbuseIPDB after a check
|
||||||
// fails, so that a source refusing them is not asked on every request.
|
// fails, so that a source refusing them is not asked on every request.
|
||||||
failureDelay = time.Minute
|
failureDelay = time.Minute
|
||||||
|
// maskedKey is what MaskZoneKey shows in place of a zone's key.
|
||||||
|
maskedKey = "********"
|
||||||
)
|
)
|
||||||
|
|
||||||
var (
|
var (
|
||||||
@@ -82,8 +83,12 @@ type DNSBL struct {
|
|||||||
// verdicts are by query. Each is added as it is fetched and never moved
|
// verdicts are by query. Each is added as it is fetched and never moved
|
||||||
// up, so that the one fetched longest ago is the first dropped.
|
// up, so that the one fetched longest ago is the first dropped.
|
||||||
verdicts *simplelru.LRU[query, Verdict]
|
verdicts *simplelru.LRU[query, Verdict]
|
||||||
// asking are the queries under way.
|
// asking are the queries under way, each with the function that ends
|
||||||
asking map[query]bool
|
// it. stopped is true once StopAsking has been called, and
|
||||||
|
// queryGoroutines waits for the goroutine of each query.
|
||||||
|
asking map[query]context.CancelFunc
|
||||||
|
stopped bool
|
||||||
|
queryGoroutines sync.WaitGroup
|
||||||
// queries and failures count, by zone, the queries made and those that
|
// queries and failures count, by zone, the queries made and those that
|
||||||
// failed, and retryAt is when a zone whose last query failed may be
|
// failed, and retryAt is when a zone whose last query failed may be
|
||||||
// asked again.
|
// asked again.
|
||||||
@@ -120,13 +125,31 @@ func NewDNSBL(params DNSBLParams) *DNSBL {
|
|||||||
params: params,
|
params: params,
|
||||||
resolver: resolver,
|
resolver: resolver,
|
||||||
verdicts: verdicts,
|
verdicts: verdicts,
|
||||||
asking: map[query]bool{},
|
asking: map[query]context.CancelFunc{},
|
||||||
queries: map[string]int{},
|
queries: map[string]int{},
|
||||||
failures: map[string]int{},
|
failures: map[string]int{},
|
||||||
retryAt: map[string]time.Time{},
|
retryAt: map[string]time.Time{},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// MaskZoneKey returns zone with ******** in place of its key, if it is a
|
||||||
|
// zone of Spamhaus's keyed query service, a name under dq.spamhaus.net,
|
||||||
|
// such as <key>.xbl.dq.spamhaus.net, whose first label is the key. Any
|
||||||
|
// other zone it returns as it is. A zone is shown so wherever it leaves
|
||||||
|
// the process: in the log, the alerts and the metrics.
|
||||||
|
func MaskZoneKey(zone string) string {
|
||||||
|
// DNS names ignore case, and a name may be written with a dot at its
|
||||||
|
// end.
|
||||||
|
name := strings.TrimSuffix(strings.ToLower(zone), ".")
|
||||||
|
if !strings.HasSuffix(name, ".dq.spamhaus.net") {
|
||||||
|
return zone
|
||||||
|
}
|
||||||
|
|
||||||
|
_, rest, _ := strings.Cut(zone, ".")
|
||||||
|
|
||||||
|
return maskedKey + "." + rest
|
||||||
|
}
|
||||||
|
|
||||||
// Zones returns the zones, in the order SWW_WAF_DNSBL_ZONES names them.
|
// Zones returns the zones, in the order SWW_WAF_DNSBL_ZONES names them.
|
||||||
func (d *DNSBL) Zones() []string {
|
func (d *DNSBL) Zones() []string {
|
||||||
return slices.Clone(d.params.Zones)
|
return slices.Clone(d.params.Zones)
|
||||||
@@ -134,13 +157,14 @@ func (d *DNSBL) Zones() []string {
|
|||||||
|
|
||||||
// ListedBy returns the zones whose verdict on addr, a client's address,
|
// ListedBy returns the zones whose verdict on addr, a client's address,
|
||||||
// lists it, in the order SWW_WAF_DNSBL_ZONES names them, each with its key
|
// lists it, in the order SWW_WAF_DNSBL_ZONES names them, each with its key
|
||||||
// masked, as config.MaskZoneKey masks it, since they go to the request
|
// masked, as MaskZoneKey masks it, since they go to the request
|
||||||
// log, the alerts and the metrics. A verdict is used until CacheTTL has
|
// log, the alerts and the metrics. A verdict is used until CacheTTL has
|
||||||
// passed since it was fetched. Each zone without one is asked about addr
|
// passed since it was fetched. Each zone without one is asked about addr
|
||||||
// in the background, unless a query about addr to it is under way, the
|
// in the background, unless a query about addr to it is under way, the
|
||||||
// zone is left alone after a failure, or maxQueries are under way;
|
// zone is left alone after a failure, maxQueries are under way, or
|
||||||
// ListedBy never waits for a query. ctx is the context of the client's
|
// StopAsking has been called; ListedBy never waits for a query. ctx is the
|
||||||
// request, and a query goes on after the request ends.
|
// context of the client's request, and a query goes on after the request
|
||||||
|
// ends, until StopAsking.
|
||||||
func (d *DNSBL) ListedBy(ctx context.Context, addr netip.Addr) []string {
|
func (d *DNSBL) ListedBy(ctx context.Context, addr netip.Addr) []string {
|
||||||
d.mu.Lock()
|
d.mu.Lock()
|
||||||
defer d.mu.Unlock()
|
defer d.mu.Unlock()
|
||||||
@@ -153,23 +177,47 @@ func (d *DNSBL) ListedBy(ctx context.Context, addr netip.Addr) []string {
|
|||||||
q := query{zone: zone, client: addr}
|
q := query{zone: zone, client: addr}
|
||||||
|
|
||||||
kept, found := d.verdicts.Peek(q)
|
kept, found := d.verdicts.Peek(q)
|
||||||
|
_, asking := d.asking[q]
|
||||||
|
|
||||||
switch {
|
switch {
|
||||||
case found && now.Sub(kept.Fetched) < d.params.CacheTTL:
|
case found && now.Sub(kept.Fetched) < d.params.CacheTTL:
|
||||||
if kept.Listed {
|
if kept.Listed {
|
||||||
listedBy = append(listedBy, config.MaskZoneKey(zone))
|
listedBy = append(listedBy, MaskZoneKey(zone))
|
||||||
}
|
}
|
||||||
case !d.asking[q] && !now.Before(d.retryAt[zone]) && len(d.asking) < maxQueries:
|
case !asking && !d.stopped && !now.Before(d.retryAt[zone]) &&
|
||||||
d.asking[q] = true
|
len(d.asking) < maxQueries:
|
||||||
|
queryCtx, end := context.WithCancel(context.WithoutCancel(ctx))
|
||||||
|
d.asking[q] = end
|
||||||
d.queries[zone]++
|
d.queries[zone]++
|
||||||
|
|
||||||
go d.ask(context.WithoutCancel(ctx), q)
|
d.queryGoroutines.Go(func() {
|
||||||
|
defer end()
|
||||||
|
|
||||||
|
d.ask(queryCtx, q)
|
||||||
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return listedBy
|
return listedBy
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// StopAsking ends the queries under way, and returns once their
|
||||||
|
// goroutines have. A query it ends gives no verdict and is no failure: it
|
||||||
|
// is neither counted, logged nor raised as an alert, and the zone is not
|
||||||
|
// left alone after it. After StopAsking, ListedBy asks no zone.
|
||||||
|
func (d *DNSBL) StopAsking() {
|
||||||
|
d.mu.Lock()
|
||||||
|
|
||||||
|
d.stopped = true
|
||||||
|
for _, end := range d.asking {
|
||||||
|
end()
|
||||||
|
}
|
||||||
|
|
||||||
|
d.mu.Unlock()
|
||||||
|
|
||||||
|
d.queryGoroutines.Wait()
|
||||||
|
}
|
||||||
|
|
||||||
// Queries returns how many queries were made to zone.
|
// Queries returns how many queries were made to zone.
|
||||||
func (d *DNSBL) Queries(zone string) int {
|
func (d *DNSBL) Queries(zone string) int {
|
||||||
d.mu.Lock()
|
d.mu.Lock()
|
||||||
@@ -238,30 +286,32 @@ func (d *DNSBL) Load(verdicts []Verdict) {
|
|||||||
// query as no longer under way. A query that fails gives no verdict: it
|
// query as no longer under way. A query that fails gives no verdict: it
|
||||||
// is counted, logged and raised as a source_failure alert, which show the
|
// is counted, logged and raised as a source_failure alert, which show the
|
||||||
// zone with its key masked, and the zone is not asked again for
|
// zone with its key masked, and the zone is not asked again for
|
||||||
// failureDelay.
|
// failureDelay. A query ended as ctx ends, by StopAsking, is no failure.
|
||||||
func (d *DNSBL) ask(ctx context.Context, q query) {
|
func (d *DNSBL) ask(ctx context.Context, q query) {
|
||||||
listed, err := d.lookUp(ctx, q)
|
listed, err := d.lookUp(ctx, q)
|
||||||
now := d.params.Now()
|
now := d.params.Now()
|
||||||
|
isFailure := err != nil && ctx.Err() == nil
|
||||||
|
|
||||||
d.mu.Lock()
|
d.mu.Lock()
|
||||||
|
|
||||||
delete(d.asking, q)
|
delete(d.asking, q)
|
||||||
|
|
||||||
if err == nil {
|
switch {
|
||||||
|
case err == nil:
|
||||||
d.verdicts.Add(q, Verdict{
|
d.verdicts.Add(q, Verdict{
|
||||||
Zone: q.zone, Client: q.client, Listed: listed, Fetched: now,
|
Zone: q.zone, Client: q.client, Listed: listed, Fetched: now,
|
||||||
})
|
})
|
||||||
} else {
|
case isFailure:
|
||||||
d.failures[q.zone]++
|
d.failures[q.zone]++
|
||||||
d.retryAt[q.zone] = now.Add(failureDelay)
|
d.retryAt[q.zone] = now.Add(failureDelay)
|
||||||
}
|
}
|
||||||
|
|
||||||
d.mu.Unlock()
|
d.mu.Unlock()
|
||||||
|
|
||||||
if err != nil {
|
if isFailure {
|
||||||
const failed = "asking a DNSBL zone failed"
|
const failed = "asking a DNSBL zone failed"
|
||||||
|
|
||||||
shown := config.MaskZoneKey(q.zone)
|
shown := MaskZoneKey(q.zone)
|
||||||
|
|
||||||
// Raised before it is logged, so that the alert is there once the
|
// Raised before it is logged, so that the alert is there once the
|
||||||
// log line is.
|
// log line is.
|
||||||
|
|||||||
@@ -31,10 +31,12 @@ import (
|
|||||||
// this package: Go's resolver counts the queries under way in one
|
// this package: Go's resolver counts the queries under way in one
|
||||||
// sync.WaitGroup for the whole process, and the process fails when
|
// sync.WaitGroup for the whole process, and the process fails when
|
||||||
// queries from two bubbles, or from a bubble and from outside one, are
|
// queries from two bubbles, or from a bubble and from outside one, are
|
||||||
// under way at once. TestMain has the resolver make its configuration,
|
// under way at once. The one test outside a bubble is parallel, so it
|
||||||
// which it makes on its first query, outside every bubble, since the
|
// starts only once every test that is not parallel has ended; no other
|
||||||
// configuration holds a channel, which the bubble it was made in would
|
// parallel test of this package queries a resolver. TestMain has the
|
||||||
// keep to itself.
|
// resolver make its configuration, which it makes on its first query,
|
||||||
|
// outside every bubble, since the configuration holds a channel, which
|
||||||
|
// the bubble it was made in would keep to itself.
|
||||||
|
|
||||||
const (
|
const (
|
||||||
// zone and otherZone are the DNSBL zones the tests name.
|
// zone and otherZone are the DNSBL zones the tests name.
|
||||||
@@ -208,6 +210,45 @@ func TestQueryNotAnsweredWithinTheTimeoutFails(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
//
|
||||||
|
//nolint:paralleltest // one at a time, as the comment at the top of this file says
|
||||||
|
func TestStopAskingEndsTheQueryUnderWayAsNoFailureAndAsksNoMore(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
queue := newQueue()
|
||||||
|
p := dnsblParams(zone)
|
||||||
|
p.Alerts = queue
|
||||||
|
resolver := &resolverStandIn{hanging: true}
|
||||||
|
dnsbl := newDNSBL(resolver, p)
|
||||||
|
|
||||||
|
wantZones(t, dnsbl, listed)
|
||||||
|
synctest.Wait()
|
||||||
|
|
||||||
|
// The query is ended, not waited out, and is no failure.
|
||||||
|
began := time.Now()
|
||||||
|
|
||||||
|
dnsbl.StopAsking()
|
||||||
|
|
||||||
|
if waited := time.Since(began); waited != 0 {
|
||||||
|
t.Errorf("StopAsking returned after %s, want at once", waited)
|
||||||
|
}
|
||||||
|
|
||||||
|
wantQueries(t, dnsbl, 1, 0)
|
||||||
|
|
||||||
|
if got := waiting(queue); len(got) != 0 {
|
||||||
|
t.Errorf("alerts waiting %+v, want none", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A client without a verdict is then asked about in no zone.
|
||||||
|
wantZones(t, dnsbl, unlisted)
|
||||||
|
synctest.Wait()
|
||||||
|
wantQueries(t, dnsbl, 1, 0)
|
||||||
|
wantAsked(t, resolver, listedName)
|
||||||
|
waitForTheResolver()
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
//
|
//
|
||||||
//nolint:paralleltest // one at a time, as the comment at the top of this file says
|
//nolint:paralleltest // one at a time, as the comment at the top of this file says
|
||||||
func TestZoneThatFailsOrRefusesGivesNoVerdictAndIsLeftAloneForAMinute(
|
func TestZoneThatFailsOrRefusesGivesNoVerdictAndIsLeftAloneForAMinute(
|
||||||
@@ -525,8 +566,9 @@ func TestAtMost100000VerdictsKeptTheOneFetchedLongestAgoDroppedFirst(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
//nolint:paralleltest // one at a time, as the comment at the top of this file says
|
|
||||||
func TestQueriesGoToTheResolverSWWDNSBLResolverNames(t *testing.T) {
|
func TestQueriesGoToTheResolverSWWDNSBLResolverNames(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
resolver := &resolverStandIn{answers: map[string]answer{
|
resolver := &resolverStandIn{answers: map[string]answer{
|
||||||
listedName: {addrs: []string{listing}},
|
listedName: {addrs: []string{listing}},
|
||||||
}}
|
}}
|
||||||
@@ -818,3 +860,28 @@ func wantAsked(t *testing.T, resolver *resolverStandIn, want ...string) {
|
|||||||
t.Errorf("asked about %v, want %v", got, want)
|
t.Errorf("asked about %v, want %v", got, want)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestMaskZoneKeyMasksTheFirstLabelOfAZoneUnderDqSpamhausNet(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const (
|
||||||
|
dronebl = "dnsbl.dronebl.org"
|
||||||
|
spamhaus = "abcdefghijklmnopqrstuvwxyz.xbl.dq.spamhaus.net"
|
||||||
|
spamhausMasked = "********.xbl.dq.spamhaus.net"
|
||||||
|
)
|
||||||
|
|
||||||
|
for zone, want := range map[string]string{
|
||||||
|
spamhaus: spamhausMasked,
|
||||||
|
spamhaus + ".": spamhausMasked + ".",
|
||||||
|
"KEY.ZEN.DQ.SPAMHAUS.NET": "********.ZEN.DQ.SPAMHAUS.NET",
|
||||||
|
dronebl: dronebl,
|
||||||
|
"dq.spamhaus.net": "dq.spamhaus.net",
|
||||||
|
spamhaus + ".example": spamhaus + ".example",
|
||||||
|
} {
|
||||||
|
if got := reputation.MaskZoneKey(zone); got != want {
|
||||||
|
t.Errorf("MaskZoneKey(%q) is %q, want %q", zone, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -7,7 +7,10 @@
|
|||||||
// SWW_WAF_DNSBL_ZONES about clients, and keeps their verdicts, and checks
|
// SWW_WAF_DNSBL_ZONES about clients, and keeps their verdicts, and checks
|
||||||
// clients with AbuseIPDB, and keeps their scores and the checks spent
|
// clients with AbuseIPDB, and keeps their scores and the checks spent
|
||||||
// today. The state package writes all of these to files and reads them
|
// today. The state package writes all of these to files and reads them
|
||||||
// from there, so that a restart keeps them too.
|
// from there, so that a restart keeps them too. It fetches AbuseIPDB's
|
||||||
|
// blacklist, which it does not keep. Last, it reads the netblocks, AS
|
||||||
|
// numbers and percentages that both the settings and the lists hold, and
|
||||||
|
// masks the keys of DNSBL zones.
|
||||||
package reputation
|
package reputation
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -19,12 +22,12 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"slices"
|
"slices"
|
||||||
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -39,15 +42,28 @@ const (
|
|||||||
)
|
)
|
||||||
|
|
||||||
var (
|
var (
|
||||||
errStatus = errors.New("the server answered")
|
errStatus = errors.New("the server answered")
|
||||||
errTooLong = errors.New("the list is longer than 16 MiB")
|
errTooLong = errors.New("the list is longer than 16 MiB")
|
||||||
errNotNetblock = errors.New(
|
errNotNetblockLine = errors.New(
|
||||||
"is not an address or a netblock, such as 192.0.2.0/24",
|
"is not an address or a netblock, such as 192.0.2.0/24",
|
||||||
)
|
)
|
||||||
errNotASNPercent = errors.New(
|
errNotASNPercent = errors.New(
|
||||||
"is not an AS number, : and a percentage, such as AS64496:50")
|
"is not an AS number, : and a percentage, such as AS64496:50")
|
||||||
|
errNotNetblock = errors.New(
|
||||||
|
"is not a netblock such as 10.0.0.0/8, or an address")
|
||||||
|
errNotASN = errors.New("is not an AS number such as AS64496")
|
||||||
|
errNotPercent = errors.New(
|
||||||
|
"is not a percentage, a whole number from 0 to 100",
|
||||||
|
)
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// TagList is an address list of SWW_WAF_TAG_LISTS: the tag it adds to the
|
||||||
|
// requests of the clients it holds, and the URL it is fetched from.
|
||||||
|
type TagList struct {
|
||||||
|
Name string
|
||||||
|
URL string
|
||||||
|
}
|
||||||
|
|
||||||
// List is a list as lists.json holds it: the URL it is fetched from,
|
// List is a list as lists.json holds it: the URL it is fetched from,
|
||||||
// when it was last tried, the fetch failed or not, and its last good copy:
|
// when it was last tried, the fetch failed or not, and its last good copy:
|
||||||
// when that was fetched, and its lines, as fetched, comment lines
|
// when that was fetched, and its lines, as fetched, comment lines
|
||||||
@@ -67,7 +83,7 @@ type Params struct {
|
|||||||
BlocklistURLs []string
|
BlocklistURLs []string
|
||||||
ASNLimitPercentURL string
|
ASNLimitPercentURL string
|
||||||
// TagLists are the address lists that add a tag (SWW_WAF_TAG_LISTS).
|
// TagLists are the address lists that add a tag (SWW_WAF_TAG_LISTS).
|
||||||
TagLists []config.TagList
|
TagLists []TagList
|
||||||
// Refresh is how long after a list was last fetched or tried it is
|
// Refresh is how long after a list was last fetched or tried it is
|
||||||
// fetched again (SWW_WAF_BLOCKLIST_REFRESH).
|
// fetched again (SWW_WAF_BLOCKLIST_REFRESH).
|
||||||
Refresh time.Duration
|
Refresh time.Duration
|
||||||
@@ -464,7 +480,11 @@ func parseNetblocks(lines []string) (entries, error) {
|
|||||||
|
|
||||||
netblock, ok := parseNetblock(text)
|
netblock, ok := parseNetblock(text)
|
||||||
if !ok {
|
if !ok {
|
||||||
return entries{}, fmt.Errorf("line %d %w", i+1, errNotNetblock)
|
return entries{}, fmt.Errorf(
|
||||||
|
"line %d %w",
|
||||||
|
i+1,
|
||||||
|
errNotNetblockLine,
|
||||||
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
found.netblocks[netblock] = true
|
found.netblocks[netblock] = true
|
||||||
@@ -484,7 +504,7 @@ func parseNetblocks(lines []string) (entries, error) {
|
|||||||
// stands for, 192.0.2.0/24, and a mapped netblock shorter than /96, which
|
// stands for, 192.0.2.0/24, and a mapped netblock shorter than /96, which
|
||||||
// stands for none, is refused.
|
// stands for none, is refused.
|
||||||
func parseNetblock(text string) (netip.Prefix, bool) {
|
func parseNetblock(text string) (netip.Prefix, bool) {
|
||||||
netblock, err := config.ParseNetblock(text)
|
netblock, err := ParseNetblock(text)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return netip.Prefix{}, false
|
return netip.Prefix{}, false
|
||||||
}
|
}
|
||||||
@@ -519,9 +539,9 @@ func parsePercents(lines []string) (entries, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
asnText, percentText, _ := strings.Cut(text, ":")
|
asnText, percentText, _ := strings.Cut(text, ":")
|
||||||
asn, asnErr := config.ParseASN(asnText)
|
asn, asnErr := ParseASN(asnText)
|
||||||
|
|
||||||
percent, percentErr := config.ParsePercent(percentText)
|
percent, percentErr := ParsePercent(percentText)
|
||||||
if asnErr != nil || percentErr != nil {
|
if asnErr != nil || percentErr != nil {
|
||||||
return entries{}, fmt.Errorf("line %d %w", i+1, errNotASNPercent)
|
return entries{}, fmt.Errorf("line %d %w", i+1, errNotASNPercent)
|
||||||
}
|
}
|
||||||
@@ -535,6 +555,53 @@ func parsePercents(lines []string) (entries, error) {
|
|||||||
return found, nil
|
return found, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ParseNetblock reads a netblock in CIDR form, such as 10.0.0.0/8. A bare
|
||||||
|
// address is a netblock of that address alone, a /32 or a /128. A
|
||||||
|
// blocklist's lines are read with it too.
|
||||||
|
func ParseNetblock(value string) (netip.Prefix, error) {
|
||||||
|
if strings.Contains(value, "/") {
|
||||||
|
netblock, err := netip.ParsePrefix(value)
|
||||||
|
if err != nil {
|
||||||
|
return netip.Prefix{}, fmt.Errorf("%q %w", value, errNotNetblock)
|
||||||
|
}
|
||||||
|
|
||||||
|
return netblock.Masked(), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
addr, err := netip.ParseAddr(value)
|
||||||
|
if err != nil || addr.Zone() != "" {
|
||||||
|
return netip.Prefix{}, fmt.Errorf("%q %w", value, errNotNetblock)
|
||||||
|
}
|
||||||
|
|
||||||
|
return netip.PrefixFrom(addr, addr.BitLen()), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// ParseASN reads an AS number such as AS64496, in either case, and
|
||||||
|
// returns it as the lookup gives it: AS and the number, in capitals and
|
||||||
|
// without leading zeros. The file SWW_WAF_ASN_LIMIT_PERCENT_URL names is
|
||||||
|
// read with it too.
|
||||||
|
func ParseASN(value string) (string, error) {
|
||||||
|
digits, hasAS := strings.CutPrefix(strings.ToUpper(value), "AS")
|
||||||
|
|
||||||
|
number, err := strconv.ParseUint(digits, 10, 32)
|
||||||
|
if !hasAS || err != nil {
|
||||||
|
return "", fmt.Errorf("%q %w", value, errNotASN)
|
||||||
|
}
|
||||||
|
|
||||||
|
return "AS" + strconv.FormatUint(number, 10), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// ParsePercent reads a percentage, a whole number from 0 to 100. The file
|
||||||
|
// SWW_WAF_ASN_LIMIT_PERCENT_URL names is read with it too.
|
||||||
|
func ParsePercent(value string) (int64, error) {
|
||||||
|
percent, err := strconv.ParseInt(value, 10, 64)
|
||||||
|
if err != nil || percent < 0 || percent > 100 {
|
||||||
|
return 0, fmt.Errorf("%q %w", value, errNotPercent)
|
||||||
|
}
|
||||||
|
|
||||||
|
return percent, nil
|
||||||
|
}
|
||||||
|
|
||||||
// withoutComment returns line without anything after a ; or a #, and
|
// withoutComment returns line without anything after a ; or a #, and
|
||||||
// without the spaces around what is left.
|
// without the spaces around what is left.
|
||||||
func withoutComment(line string) string {
|
func withoutComment(line string) string {
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
// Package requestlog writes the lines sww prints on stdout: one
|
// Package requestlog writes the request log sww prints on stdout: JSON
|
||||||
// JSON object per request, two from waf and a short one from rproxy,
|
// lines marked "type":"request", at most two from waf and a short one
|
||||||
// marked "type":"request", and the process's own messages as JSON lines marked
|
// from rproxy for each request. Package processlog writes the process's
|
||||||
// "type":"process".
|
// own messages.
|
||||||
package requestlog
|
package requestlog
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -9,7 +9,6 @@ import (
|
|||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"log/slog"
|
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
@@ -21,7 +20,12 @@ const (
|
|||||||
// the app.
|
// the app.
|
||||||
ActionForward = "forward"
|
ActionForward = "forward"
|
||||||
|
|
||||||
// ActionRequestHeaderTooLarge is a request over the header limits.
|
// ActionRequestLineTooLong is a request whose request line is too long for
|
||||||
|
// the header limits.
|
||||||
|
ActionRequestLineTooLong = "request_line_too_long"
|
||||||
|
|
||||||
|
// ActionRequestHeaderTooLarge is a request whose headers are over the
|
||||||
|
// header limits.
|
||||||
ActionRequestHeaderTooLarge = "request_header_too_large"
|
ActionRequestHeaderTooLarge = "request_header_too_large"
|
||||||
|
|
||||||
// ActionClientIsRateLimited is a request refused because it took its
|
// ActionClientIsRateLimited is a request refused because it took its
|
||||||
@@ -56,6 +60,12 @@ const (
|
|||||||
// ActionCountryDenied is a request refused because its client's country
|
// ActionCountryDenied is a request refused because its client's country
|
||||||
// is not in SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES.
|
// is not in SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES.
|
||||||
ActionCountryDenied = "country_denied"
|
ActionCountryDenied = "country_denied"
|
||||||
|
|
||||||
|
// ActionChecksTimedOut is a request refused because the trap paths, the
|
||||||
|
// rule files and the Core Rule Set had not finished with it within 500
|
||||||
|
// milliseconds.
|
||||||
|
ActionChecksTimedOut = "checks_timed_out"
|
||||||
|
|
||||||
// ActionOwnEndpoint is a request waf answered at one of its own
|
// ActionOwnEndpoint is a request waf answered at one of its own
|
||||||
// endpoints, or a question about a path under /_sww/.
|
// endpoints, or a question about a path under /_sww/.
|
||||||
ActionOwnEndpoint = "own_endpoint"
|
ActionOwnEndpoint = "own_endpoint"
|
||||||
@@ -143,8 +153,9 @@ type Line struct {
|
|||||||
// taken with a request it would have refused: ActionClientDenied,
|
// taken with a request it would have refused: ActionClientDenied,
|
||||||
// ActionTemporarilyBanned, ActionPermanentlyBanned, ActionCountryDenied,
|
// ActionTemporarilyBanned, ActionPermanentlyBanned, ActionCountryDenied,
|
||||||
// ActionClientIsRateLimited, ActionBlockedByBlockRule,
|
// ActionClientIsRateLimited, ActionBlockedByBlockRule,
|
||||||
// ActionBlockedByCoreRuleSet, or, in waf,
|
// ActionBlockedByCoreRuleSet, or, in waf, ActionRequestLineTooLong or
|
||||||
// ActionRequestHeaderTooLarge for the header limits.
|
// ActionRequestHeaderTooLarge for the header limits, or
|
||||||
|
// ActionChecksTimedOut.
|
||||||
WouldAction string `json:"would_action,omitempty"`
|
WouldAction string `json:"would_action,omitempty"`
|
||||||
// Heuristic is the name of the heuristic that refused the request, in
|
// Heuristic is the name of the heuristic that refused the request, in
|
||||||
// observe mode too, and HeuristicFigures are the figures it acted on, by
|
// observe mode too, and HeuristicFigures are the figures it acted on, by
|
||||||
@@ -288,31 +299,3 @@ func Milliseconds(d time.Duration) float64 {
|
|||||||
time.Millisecond/time.Microsecond,
|
time.Millisecond/time.Microsecond,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
// NewProcessLogger returns the logger for the process's own messages:
|
|
||||||
// JSON lines on w, marked "type":"process", with the time in the same form
|
|
||||||
// as a request line's, and instanceName, SWW_WAF_INSTANCE_NAME, as instance.
|
|
||||||
// It writes only the messages at level, SWW_WAF_LOG_LEVEL, or more severe;
|
|
||||||
// the request lines Write writes are never held back.
|
|
||||||
func NewProcessLogger(
|
|
||||||
w io.Writer,
|
|
||||||
instanceName string,
|
|
||||||
level slog.Level,
|
|
||||||
) *slog.Logger {
|
|
||||||
handler := slog.NewJSONHandler(w, &slog.HandlerOptions{
|
|
||||||
Level: level,
|
|
||||||
ReplaceAttr: func(groups []string, attr slog.Attr) slog.Attr {
|
|
||||||
if attr.Key == slog.TimeKey && len(groups) == 0 {
|
|
||||||
return slog.String(
|
|
||||||
slog.TimeKey,
|
|
||||||
FormatTime(attr.Value.Time()),
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
return attr
|
|
||||||
},
|
|
||||||
})
|
|
||||||
|
|
||||||
return slog.New(handler).
|
|
||||||
With("type", "process", "instance", instanceName)
|
|
||||||
}
|
|
||||||
@@ -3,8 +3,6 @@ package requestlog_test
|
|||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"log/slog"
|
|
||||||
"slices"
|
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
@@ -134,88 +132,3 @@ func TestWriteRProxyLineWritesOneJSONLineWithOnlyItsFields(t *testing.T) {
|
|||||||
t.Errorf("wrote %s, want %s", out.String(), want)
|
t.Errorf("wrote %s, want %s", out.String(), want)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestProcessLinesAreMarkedProcessAndGiveTheInstance(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
var out bytes.Buffer
|
|
||||||
|
|
||||||
requestlog.NewProcessLogger(&out, "fsn1app1/blog", slog.LevelInfo).
|
|
||||||
Info("starting",
|
|
||||||
"version", "v1")
|
|
||||||
|
|
||||||
var fields map[string]any
|
|
||||||
|
|
||||||
err := json.Unmarshal(out.Bytes(), &fields)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("decode %q: %v", out.String(), err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if fields["type"] != "process" ||
|
|
||||||
fields["instance"] != "fsn1app1/blog" ||
|
|
||||||
fields["msg"] != "starting" ||
|
|
||||||
fields["level"] != "INFO" ||
|
|
||||||
fields["version"] != "v1" {
|
|
||||||
t.Errorf("process line %v", fields)
|
|
||||||
}
|
|
||||||
|
|
||||||
timeText, _ := fields["time"].(string)
|
|
||||||
|
|
||||||
logged, err := time.Parse(time.RFC3339, timeText)
|
|
||||||
if err != nil || !strings.HasSuffix(timeText, "Z") ||
|
|
||||||
len(timeText) != len("2006-01-02T15:04:05.000Z") ||
|
|
||||||
time.Since(logged) > time.Minute {
|
|
||||||
t.Errorf(
|
|
||||||
"process line time %q, want now in UTC with milliseconds",
|
|
||||||
timeText,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestProcessLoggerWritesTheMessagesAtItsLevelOrMoreSevere(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
levels := []slog.Level{
|
|
||||||
slog.LevelDebug, slog.LevelInfo, slog.LevelWarn, slog.LevelError,
|
|
||||||
}
|
|
||||||
|
|
||||||
for i, level := range levels {
|
|
||||||
t.Run(level.String(), func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
var out bytes.Buffer
|
|
||||||
|
|
||||||
processLog := requestlog.NewProcessLogger(
|
|
||||||
&out,
|
|
||||||
"fsn1app1/blog",
|
|
||||||
level,
|
|
||||||
)
|
|
||||||
for _, at := range levels {
|
|
||||||
processLog.Log(t.Context(), at, "message")
|
|
||||||
}
|
|
||||||
|
|
||||||
var got, want []string
|
|
||||||
|
|
||||||
for line := range strings.Lines(out.String()) {
|
|
||||||
var fields struct {
|
|
||||||
Level string `json:"level"`
|
|
||||||
}
|
|
||||||
|
|
||||||
err := json.Unmarshal([]byte(line), &fields)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("decode %q: %v", line, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
got = append(got, fields.Level)
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, written := range levels[i:] {
|
|
||||||
want = append(want, written.String())
|
|
||||||
}
|
|
||||||
|
|
||||||
if !slices.Equal(got, want) {
|
|
||||||
t.Errorf("lines at %v, want %v", got, want)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -99,8 +99,12 @@ type Client struct {
|
|||||||
// answers are by client. Each is added as it comes and never moved up,
|
// answers are by client. Each is added as it comes and never moved up,
|
||||||
// so that the one that came longest ago is the first dropped.
|
// so that the one that came longest ago is the first dropped.
|
||||||
answers *simplelru.LRU[netip.Prefix, Answer]
|
answers *simplelru.LRU[netip.Prefix, Answer]
|
||||||
// asking are the clients whose query is under way.
|
// asking are the clients whose query is under way, each with the
|
||||||
asking map[netip.Prefix]bool
|
// function that ends it. stopped is true once StopAsking has been
|
||||||
|
// called, and queryGoroutines waits for the goroutine of each query.
|
||||||
|
asking map[netip.Prefix]context.CancelFunc
|
||||||
|
stopped bool
|
||||||
|
queryGoroutines sync.WaitGroup
|
||||||
// retryAt is when routewatch may be asked again after a query failed.
|
// retryAt is when routewatch may be asked again after a query failed.
|
||||||
retryAt time.Time
|
retryAt time.Time
|
||||||
}
|
}
|
||||||
@@ -116,7 +120,7 @@ func New(params Params) *Client {
|
|||||||
params: params,
|
params: params,
|
||||||
httpClient: &http.Client{},
|
httpClient: &http.Client{},
|
||||||
answers: answers,
|
answers: answers,
|
||||||
asking: map[netip.Prefix]bool{},
|
asking: map[netip.Prefix]context.CancelFunc{},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -124,8 +128,9 @@ func New(params Params) *Client {
|
|||||||
// IPv6 group, or why it has no route. It never waits for a query. A client
|
// IPv6 group, or why it has no route. It never waits for a query. A client
|
||||||
// without an answer is asked about in the background, by addr, the address its
|
// without an answer is asked about in the background, by addr, the address its
|
||||||
// request came from, unless its query is under way, a query failed less than
|
// request came from, unless its query is under way, a query failed less than
|
||||||
// failureKept ago, or maxQueries are under way. ctx is the context of the
|
// failureKept ago, maxQueries are under way, or StopAsking has been called. ctx
|
||||||
// client's request, and a query goes on after the request ends.
|
// is the context of the client's request, and a query goes on after the request
|
||||||
|
// ends, until StopAsking.
|
||||||
func (c *Client) Route(
|
func (c *Client) Route(
|
||||||
ctx context.Context, client netip.Prefix, addr netip.Addr,
|
ctx context.Context, client netip.Prefix, addr netip.Addr,
|
||||||
) (Route, Missing) {
|
) (Route, Missing) {
|
||||||
@@ -147,15 +152,38 @@ func (c *Client) Route(
|
|||||||
return Route{}, RequestFailed
|
return Route{}, RequestFailed
|
||||||
}
|
}
|
||||||
|
|
||||||
if !c.asking[client] && len(c.asking) < maxQueries {
|
_, asking := c.asking[client]
|
||||||
c.asking[client] = true
|
if !asking && !c.stopped && len(c.asking) < maxQueries {
|
||||||
|
queryCtx, end := context.WithCancel(context.WithoutCancel(ctx))
|
||||||
|
c.asking[client] = end
|
||||||
|
|
||||||
go c.ask(context.WithoutCancel(ctx), client, addr)
|
c.queryGoroutines.Go(func() {
|
||||||
|
defer end()
|
||||||
|
|
||||||
|
c.ask(queryCtx, client, addr)
|
||||||
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
return Route{}, NoAnswerYet
|
return Route{}, NoAnswerYet
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// StopAsking ends the queries under way, and returns once their goroutines
|
||||||
|
// have. A query it ends gives no answer and is no failure: it is not logged,
|
||||||
|
// and routewatch is not left alone after it. After StopAsking, Route asks
|
||||||
|
// routewatch nothing.
|
||||||
|
func (c *Client) StopAsking() {
|
||||||
|
c.mu.Lock()
|
||||||
|
|
||||||
|
c.stopped = true
|
||||||
|
for _, end := range c.asking {
|
||||||
|
end()
|
||||||
|
}
|
||||||
|
|
||||||
|
c.mu.Unlock()
|
||||||
|
|
||||||
|
c.queryGoroutines.Wait()
|
||||||
|
}
|
||||||
|
|
||||||
// Snapshot returns every answer still in use, sorted by client, as
|
// Snapshot returns every answer still in use, sorted by client, as
|
||||||
// routewatch.json lists them.
|
// routewatch.json lists them.
|
||||||
func (c *Client) Snapshot() []Answer {
|
func (c *Client) Snapshot() []Answer {
|
||||||
@@ -205,7 +233,8 @@ func (c *Client) Load(answers []Answer) {
|
|||||||
// ask asks routewatch about addr, keeps the answer as client's, and notes the
|
// ask asks routewatch about addr, keeps the answer as client's, and notes the
|
||||||
// query as no longer under way. A query that fails gives no answer, and
|
// query as no longer under way. A query that fails gives no answer, and
|
||||||
// routewatch is not asked again for failureKept. Only the first failure of
|
// routewatch is not asked again for failureKept. Only the first failure of
|
||||||
// those under way is logged.
|
// those under way is logged. A query ended as ctx ends, by StopAsking, is no
|
||||||
|
// failure.
|
||||||
func (c *Client) ask(
|
func (c *Client) ask(
|
||||||
ctx context.Context,
|
ctx context.Context,
|
||||||
client netip.Prefix,
|
client netip.Prefix,
|
||||||
@@ -213,18 +242,20 @@ func (c *Client) ask(
|
|||||||
) {
|
) {
|
||||||
route, err := c.query(ctx, addr)
|
route, err := c.query(ctx, addr)
|
||||||
now := c.params.Now()
|
now := c.params.Now()
|
||||||
|
isFailure := err != nil && ctx.Err() == nil
|
||||||
|
|
||||||
c.mu.Lock()
|
c.mu.Lock()
|
||||||
|
|
||||||
delete(c.asking, client)
|
delete(c.asking, client)
|
||||||
|
|
||||||
firstFailure := err != nil && !now.Before(c.retryAt)
|
firstFailure := isFailure && !now.Before(c.retryAt)
|
||||||
|
|
||||||
if err == nil {
|
switch {
|
||||||
|
case err == nil:
|
||||||
c.answers.Add(client, Answer{
|
c.answers.Add(client, Answer{
|
||||||
Client: client, Netblock: route.Netblock, ASN: route.ASN, Answered: now,
|
Client: client, Netblock: route.Netblock, ASN: route.ASN, Answered: now,
|
||||||
})
|
})
|
||||||
} else {
|
case isFailure:
|
||||||
c.retryAt = now.Add(failureKept)
|
c.retryAt = now.Add(failureKept)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -361,6 +361,54 @@ func TestHangingRoutewatchNeverDelaysAndFailsTheQuery(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestStopAskingEndsTheQueryUnderWayAsNoFailureAndAsksNoMore(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
// In a synctest bubble, as TestHangingRoutewatchNeverDelaysAndFailsTheQuery
|
||||||
|
// says.
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
f := &fixture{t: t, log: &syncBuffer{}}
|
||||||
|
f.client = routewatch.New(routewatch.Params{
|
||||||
|
URL: silentURL, Now: time.Now,
|
||||||
|
ProcessLog: slog.New(slog.NewTextHandler(f.log, nil)),
|
||||||
|
})
|
||||||
|
|
||||||
|
f.route()
|
||||||
|
synctest.Wait()
|
||||||
|
|
||||||
|
// The query is ended, not waited out.
|
||||||
|
began := time.Now()
|
||||||
|
|
||||||
|
f.client.StopAsking()
|
||||||
|
|
||||||
|
if waited := time.Since(began); waited != 0 {
|
||||||
|
t.Errorf("StopAsking returned after %s, want at once", waited)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A query started now would fail within waitLimit. None is, and the
|
||||||
|
// one ended is no failure: nothing is logged, and routewatch is not
|
||||||
|
// left alone.
|
||||||
|
for range 2 {
|
||||||
|
if _, missing := f.route(); missing != routewatch.NoAnswerYet {
|
||||||
|
t.Errorf(
|
||||||
|
"Route gave %q, want %q",
|
||||||
|
missing,
|
||||||
|
routewatch.NoAnswerYet,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
time.Sleep(waitLimit)
|
||||||
|
synctest.Wait()
|
||||||
|
}
|
||||||
|
|
||||||
|
if f.log.String() != "" {
|
||||||
|
t.Errorf("process log %q, want nothing", f.log.String())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
func TestRepeatedRequestsMakeOneQueryAndTheAnswerExpires(t *testing.T) {
|
func TestRepeatedRequestsMakeOneQueryAndTheAnswerExpires(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,82 @@
|
|||||||
|
package rproxy
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// What opens the circuit breaker, and how often a request asks waf while it is
|
||||||
|
// open.
|
||||||
|
const (
|
||||||
|
breakerFailures = 5
|
||||||
|
breakerWindow = 5 * time.Second
|
||||||
|
breakerAskInterval = time.Second
|
||||||
|
)
|
||||||
|
|
||||||
|
// errBreakerOpen is why a request did not ask waf.
|
||||||
|
var errBreakerOpen = errors.New(
|
||||||
|
"waf is not asked while the circuit breaker is open",
|
||||||
|
)
|
||||||
|
|
||||||
|
// breaker is the circuit breaker for asks to waf. It opens when
|
||||||
|
// breakerFailures asks fail within breakerWindow. An ask fails when it gets no
|
||||||
|
// answer in time, or an answer rproxy cannot pass on. While it is open, at
|
||||||
|
// most one request each breakerAskInterval asks waf. Any usable answer closes
|
||||||
|
// it, whatever it decides, and forgets the failures before it.
|
||||||
|
type breaker struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
// failures holds the times of the latest failed asks, oldest first, at most
|
||||||
|
// breakerFailures of them.
|
||||||
|
failures []time.Time
|
||||||
|
open bool
|
||||||
|
// lastAsk is when a request last asked waf while the breaker was open.
|
||||||
|
lastAsk time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
// mayAsk reports whether a request may ask waf at now. While the breaker is
|
||||||
|
// open, it says yes once each breakerAskInterval.
|
||||||
|
func (b *breaker) mayAsk(now time.Time) bool {
|
||||||
|
b.mu.Lock()
|
||||||
|
defer b.mu.Unlock()
|
||||||
|
|
||||||
|
if !b.open {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
if now.Sub(b.lastAsk) < breakerAskInterval {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
b.lastAsk = now
|
||||||
|
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// noteAnswer notes that an ask got an answer rproxy can use. It closes the
|
||||||
|
// breaker and forgets the failures before it.
|
||||||
|
func (b *breaker) noteAnswer() {
|
||||||
|
b.mu.Lock()
|
||||||
|
defer b.mu.Unlock()
|
||||||
|
|
||||||
|
b.open = false
|
||||||
|
b.failures = nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// noteFailure notes that an ask that ended at now got no answer in time, and
|
||||||
|
// opens the breaker if breakerFailures asks have failed within breakerWindow.
|
||||||
|
func (b *breaker) noteFailure(now time.Time) {
|
||||||
|
b.mu.Lock()
|
||||||
|
defer b.mu.Unlock()
|
||||||
|
|
||||||
|
b.failures = append(b.failures, now)
|
||||||
|
if len(b.failures) > breakerFailures {
|
||||||
|
b.failures = b.failures[1:]
|
||||||
|
}
|
||||||
|
|
||||||
|
if !b.open && len(b.failures) == breakerFailures &&
|
||||||
|
now.Sub(b.failures[0]) <= breakerWindow {
|
||||||
|
b.open = true
|
||||||
|
b.lastAsk = now
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,142 @@
|
|||||||
|
package rproxy_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/rproxy"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The breaker tests give the breaker each time themselves, counted from
|
||||||
|
// time.Now(), and never wait.
|
||||||
|
|
||||||
|
// fail notes count failed asks on b, one each step from at.
|
||||||
|
func fail(b *rproxy.Breaker, at time.Time, count int, step time.Duration) {
|
||||||
|
for i := range count {
|
||||||
|
b.NoteFailure(at.Add(time.Duration(i) * step))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// openBreaker returns a breaker opened at now by five failed asks, one a
|
||||||
|
// second.
|
||||||
|
func openBreaker(now time.Time) *rproxy.Breaker {
|
||||||
|
b := &rproxy.Breaker{}
|
||||||
|
fail(b, now.Add(-4*time.Second), 5, time.Second)
|
||||||
|
|
||||||
|
return b
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFourFailedAsksWithinFiveSecondsLeaveTheBreakerClosed(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
now := time.Now()
|
||||||
|
b := &rproxy.Breaker{}
|
||||||
|
fail(b, now, 4, time.Second)
|
||||||
|
|
||||||
|
if !b.MayAsk(now.Add(3 * time.Second)) {
|
||||||
|
t.Error("a request may not ask, want it to")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFiveFailedAsksWithinFiveSecondsOpenTheBreaker(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
now := time.Now()
|
||||||
|
b := openBreaker(now)
|
||||||
|
|
||||||
|
if b.MayAsk(now) {
|
||||||
|
t.Error("a request may ask, want it not to")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFiveFailedAsksOverMoreThanFiveSecondsLeaveTheBreakerClosed(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
now := time.Now()
|
||||||
|
b := &rproxy.Breaker{}
|
||||||
|
fail(b, now, 5, 1300*time.Millisecond)
|
||||||
|
|
||||||
|
if !b.MayAsk(now.Add(6 * time.Second)) {
|
||||||
|
t.Error("a request may not ask, want it to")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAnAnswerBetweenFailedAsksKeepsTheBreakerClosed(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
now := time.Now()
|
||||||
|
b := &rproxy.Breaker{}
|
||||||
|
fail(b, now, 4, 100*time.Millisecond)
|
||||||
|
b.NoteAnswer()
|
||||||
|
fail(b, now.Add(500*time.Millisecond), 4, 100*time.Millisecond)
|
||||||
|
|
||||||
|
if !b.MayAsk(now.Add(900 * time.Millisecond)) {
|
||||||
|
t.Error("a request may not ask, want it to")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAnOpenBreakerLetsOneRequestAskEachSecond(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
now := time.Now()
|
||||||
|
b := openBreaker(now)
|
||||||
|
|
||||||
|
for _, tc := range []struct {
|
||||||
|
after time.Duration
|
||||||
|
want bool
|
||||||
|
}{
|
||||||
|
{999 * time.Millisecond, false},
|
||||||
|
{time.Second, true},
|
||||||
|
{time.Second, false},
|
||||||
|
{1500 * time.Millisecond, false},
|
||||||
|
{2 * time.Second, true},
|
||||||
|
} {
|
||||||
|
got := b.MayAsk(now.Add(tc.after))
|
||||||
|
if got != tc.want {
|
||||||
|
t.Errorf("may ask %v after it opened: %v, want %v",
|
||||||
|
tc.after, got, tc.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAFailedAskKeepsTheBreakerOpen(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
opened := time.Now()
|
||||||
|
b := openBreaker(opened)
|
||||||
|
now := opened.Add(time.Second)
|
||||||
|
b.MayAsk(now)
|
||||||
|
b.NoteFailure(now)
|
||||||
|
|
||||||
|
if b.MayAsk(now.Add(999 * time.Millisecond)) {
|
||||||
|
t.Error("a request may ask, want it not to")
|
||||||
|
}
|
||||||
|
|
||||||
|
if !b.MayAsk(now.Add(time.Second)) {
|
||||||
|
t.Error("a request may not ask a second later, want it to")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestOneAnswerClosesTheBreaker(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
opened := time.Now()
|
||||||
|
b := openBreaker(opened)
|
||||||
|
now := opened.Add(time.Second)
|
||||||
|
b.MayAsk(now)
|
||||||
|
b.NoteAnswer()
|
||||||
|
|
||||||
|
if !b.MayAsk(now) {
|
||||||
|
t.Error("a request may not ask, want it to")
|
||||||
|
}
|
||||||
|
|
||||||
|
// The failures before the answer no longer count, so one more does not open
|
||||||
|
// the breaker again.
|
||||||
|
b.NoteFailure(now)
|
||||||
|
|
||||||
|
if !b.MayAsk(now) {
|
||||||
|
t.Error("a request may not ask after one failure, want it to")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
package rproxy
|
||||||
|
|
||||||
|
import "time"
|
||||||
|
|
||||||
|
// StopBreakerClock has the circuit breaker of s read at as the time, so that how
|
||||||
|
// long requests take cannot matter to it. It is called before s serves.
|
||||||
|
func (s *Server) StopBreakerClock(at time.Time) {
|
||||||
|
s.h.breakerClock = func() time.Time { return at }
|
||||||
|
}
|
||||||
|
|
||||||
|
// Breaker is the circuit breaker, for tests that give it the time.
|
||||||
|
type Breaker struct {
|
||||||
|
b breaker
|
||||||
|
}
|
||||||
|
|
||||||
|
// MayAsk reports whether a request may ask waf at now.
|
||||||
|
func (b *Breaker) MayAsk(now time.Time) bool {
|
||||||
|
return b.b.mayAsk(now)
|
||||||
|
}
|
||||||
|
|
||||||
|
// NoteAnswer notes that an ask got an answer rproxy can use.
|
||||||
|
func (b *Breaker) NoteAnswer() {
|
||||||
|
b.b.noteAnswer()
|
||||||
|
}
|
||||||
|
|
||||||
|
// NoteFailure notes that an ask that ended at now got no answer in time.
|
||||||
|
func (b *Breaker) NoteFailure(now time.Time) {
|
||||||
|
b.b.noteFailure(now)
|
||||||
|
}
|
||||||
@@ -18,7 +18,7 @@ import (
|
|||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
|
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/processlog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/rproxy"
|
"sneak.berlin/go/smallwebwaf/internal/rproxy"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||||
)
|
)
|
||||||
@@ -51,11 +51,11 @@ func TestServerHasTheFixedLimits(t *testing.T) {
|
|||||||
server := rproxy.New(rproxy.Params{
|
server := rproxy.New(rproxy.Params{
|
||||||
Config: cfg,
|
Config: cfg,
|
||||||
RequestLog: io.Discard,
|
RequestLog: io.Discard,
|
||||||
ProcessLog: requestlog.NewProcessLogger(
|
ProcessLog: processlog.New(processlog.Params{
|
||||||
io.Discard,
|
Output: io.Discard,
|
||||||
"test",
|
InstanceName: "test",
|
||||||
slog.LevelInfo,
|
Level: slog.LevelInfo,
|
||||||
),
|
}),
|
||||||
})
|
})
|
||||||
|
|
||||||
if server.Addr != ":8080" || server.MaxHeaderBytes != 28<<10 ||
|
if server.Addr != ":8080" || server.MaxHeaderBytes != 28<<10 ||
|
||||||
@@ -71,6 +71,59 @@ func TestServerHasTheFixedLimits(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// getOfSize is a GET whose request line and headers, the blank line that ends
|
||||||
|
// them included, are size bytes long.
|
||||||
|
func getOfSize(size int) string {
|
||||||
|
const start, end = "GET / HTTP/1.1\r\nHost: site.example\r\nX-Pad: ", "\r\n\r\n"
|
||||||
|
|
||||||
|
return start + strings.Repeat("x", size-len(start)-len(end)) + end
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestARequestLineAndHeadersOf32KiBReachTheApp(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
standIn := startWAF(t, allow)
|
||||||
|
app := startApp(t, greet)
|
||||||
|
addr, _ := startRProxy(t, standIn.URL, app.URL, nil)
|
||||||
|
|
||||||
|
conn := dial(t, addr)
|
||||||
|
send(t, conn, getOfSize(32<<10))
|
||||||
|
|
||||||
|
if status := readStatus(t, conn); status != http.StatusOK {
|
||||||
|
t.Errorf("status %d, want the app's 200", status)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestARequestLineAndHeadersOver32KiBAreRefusedAndTheConnectionClosed(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
standIn := startWAF(t, allow)
|
||||||
|
app := startApp(t, func(http.ResponseWriter, *http.Request) {
|
||||||
|
t.Error("the app was asked")
|
||||||
|
})
|
||||||
|
addr, _ := startRProxy(t, standIn.URL, app.URL, nil)
|
||||||
|
|
||||||
|
conn := dial(t, addr)
|
||||||
|
send(t, conn, getOfSize(32<<10+1))
|
||||||
|
|
||||||
|
// io.ReadAll returns without an error only once rproxy has closed the
|
||||||
|
// connection.
|
||||||
|
text, err := io.ReadAll(conn)
|
||||||
|
if err != nil || !strings.HasPrefix(string(text), "HTTP/1.1 431 ") {
|
||||||
|
t.Errorf(
|
||||||
|
"read %q (%v), want 431 and the connection closed",
|
||||||
|
text,
|
||||||
|
err,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(standIn.questions) != 0 {
|
||||||
|
t.Error("waf was asked")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestABodyAnnouncedOverItsLimitIsRefusedOnceWAFAllowsIt(t *testing.T) {
|
func TestABodyAnnouncedOverItsLimitIsRefusedOnceWAFAllowsIt(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -817,10 +870,14 @@ func TestAnAnswerRProxyGivesItselfHasTenSecondsToBeSentAtLeast(
|
|||||||
func TestAVisitorTooSlowToSendTheRestIsAnswered408(t *testing.T) {
|
func TestAVisitorTooSlowToSendTheRestIsAnswered408(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
// rproxy is waiting for the rest of the body well within this limit, even on
|
||||||
|
// a busy host, and answers when it runs out, well before dial's deadline.
|
||||||
|
const limit = 5 * time.Second
|
||||||
|
|
||||||
standIn := startWAF(t, allow)
|
standIn := startWAF(t, allow)
|
||||||
app := startApp(t, greet)
|
app := startApp(t, greet)
|
||||||
addr, _ := startRProxy(t, standIn.URL, app.URL, map[string]string{
|
addr, _ := startRProxy(t, standIn.URL, app.URL, map[string]string{
|
||||||
maxRequestTime: "3000",
|
maxRequestTime: strconv.FormatInt(limit.Milliseconds(), 10),
|
||||||
})
|
})
|
||||||
|
|
||||||
conn := dial(t, addr)
|
conn := dial(t, addr)
|
||||||
@@ -1209,7 +1266,9 @@ func TestAVisitorThatStopsALittleShortIsReadUntilTheRequestTimeLimit(
|
|||||||
) {
|
) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
const limit = 2 * time.Second
|
// The app's answer arrives well within this limit, even on a busy host, and
|
||||||
|
// the connection closes when it runs out, well before dial's deadline.
|
||||||
|
const limit = 5 * time.Second
|
||||||
|
|
||||||
standIn := startWAF(t, allow)
|
standIn := startWAF(t, allow)
|
||||||
app := startApp(t, answerBeforeTheBody)
|
app := startApp(t, answerBeforeTheBody)
|
||||||
@@ -1374,17 +1433,20 @@ func TestTheRequestTimeLimitDoesNotCutAnAnswerOnceTheAppHasTheRequest(
|
|||||||
|
|
||||||
const answer = "the end of a long answer"
|
const answer = "the end of a long answer"
|
||||||
|
|
||||||
|
// The request reaches the app well within this limit, even on a busy host.
|
||||||
|
const limit = 5 * time.Second
|
||||||
|
|
||||||
standIn := startWAF(t, allow)
|
standIn := startWAF(t, allow)
|
||||||
app := startApp(t, func(w http.ResponseWriter, r *http.Request) {
|
app := startApp(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
_, _ = io.Copy(io.Discard, r.Body)
|
_, _ = io.Copy(io.Discard, r.Body)
|
||||||
_ = http.NewResponseController(w).Flush()
|
_ = http.NewResponseController(w).Flush()
|
||||||
|
|
||||||
time.Sleep(2100 * time.Millisecond)
|
time.Sleep(limit + 100*time.Millisecond)
|
||||||
|
|
||||||
_, _ = io.WriteString(w, answer)
|
_, _ = io.WriteString(w, answer)
|
||||||
})
|
})
|
||||||
addr, _ := startRProxy(t, standIn.URL, app.URL, map[string]string{
|
addr, _ := startRProxy(t, standIn.URL, app.URL, map[string]string{
|
||||||
maxRequestTime: "2000",
|
maxRequestTime: strconv.FormatInt(limit.Milliseconds(), 10),
|
||||||
})
|
})
|
||||||
|
|
||||||
got := visit(
|
got := visit(
|
||||||
|
|||||||
@@ -4,10 +4,11 @@
|
|||||||
// waf goes to the visitor as waf gave it.
|
// waf goes to the visitor as waf gave it.
|
||||||
// SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION decides only when waf cannot
|
// SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION decides only when waf cannot
|
||||||
// be connected to, does not answer at all, or does not answer within
|
// be connected to, does not answer at all, or does not answer within
|
||||||
// SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS. rproxy itself refuses a request with a
|
// SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS, or while the circuit breaker for asks to
|
||||||
// method it does not pass on, or one it cannot put to waf as the visitor
|
// waf is open. rproxy itself refuses a request with a method it does not pass
|
||||||
// sent it. It keeps nothing between requests and writes nothing to disk. "rproxy" in
|
// on, or one it cannot put to waf as the visitor sent it. It keeps nothing
|
||||||
// SPEC.md says what it does.
|
// between requests but the circuit breaker, and writes nothing to disk.
|
||||||
|
// "rproxy" in SPEC.md says what it does.
|
||||||
package rproxy
|
package rproxy
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -93,6 +94,7 @@ func New(params Params) *Server {
|
|||||||
reportURL: (&url.URL{
|
reportURL: (&url.URL{
|
||||||
Scheme: backend.Scheme, Host: backend.Host, Path: waf.ReportPath,
|
Scheme: backend.Scheme, Host: backend.Host, Path: waf.ReportPath,
|
||||||
}).String(),
|
}).String(),
|
||||||
|
breakerClock: time.Now,
|
||||||
}
|
}
|
||||||
|
|
||||||
return &Server{
|
return &Server{
|
||||||
@@ -132,6 +134,10 @@ type handler struct {
|
|||||||
wafTransport http.RoundTripper
|
wafTransport http.RoundTripper
|
||||||
// reportURL is where the reports go.
|
// reportURL is where the reports go.
|
||||||
reportURL string
|
reportURL string
|
||||||
|
// breaker is the circuit breaker for asks to waf, and breakerClock gives it
|
||||||
|
// the time.
|
||||||
|
breaker breaker
|
||||||
|
breakerClock func() time.Time
|
||||||
// reports waits for the reports being sent. reportsMu guards starting one
|
// reports waits for the reports being sent. reportsMu guards starting one
|
||||||
// against reportsStopped, which is set once nothing is to wait for another.
|
// against reportsStopped, which is set once nothing is to wait for another.
|
||||||
reports sync.WaitGroup
|
reports sync.WaitGroup
|
||||||
@@ -196,7 +202,7 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
answer, err := rq.ask(r.Context())
|
answer, err := rq.askUnlessBreakerOpen(r.Context())
|
||||||
if err != nil {
|
if err != nil {
|
||||||
rq.handleNoUsableAnswer(r.Context(), err)
|
rq.handleNoUsableAnswer(r.Context(), err)
|
||||||
|
|
||||||
|
|||||||
+205
-12
@@ -15,12 +15,13 @@ import (
|
|||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
|
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/processlog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/rproxy"
|
"sneak.berlin/go/smallwebwaf/internal/rproxy"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||||
)
|
)
|
||||||
@@ -323,7 +324,8 @@ func greet(w http.ResponseWriter, r *http.Request) {
|
|||||||
|
|
||||||
// startRProxy starts rproxy with waf at backend and the app at appURL, and
|
// startRProxy starts rproxy with waf at backend and the app at appURL, and
|
||||||
// the settings in env on top of the defaults. It returns where rproxy listens
|
// the settings in env on top of the defaults. It returns where rproxy listens
|
||||||
// and what it writes.
|
// and what it writes. SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS is a minute unless env
|
||||||
|
// sets it, so that a busy host cannot make rproxy drop a question or a report.
|
||||||
func startRProxy(
|
func startRProxy(
|
||||||
t *testing.T, backend, appURL string, env map[string]string,
|
t *testing.T, backend, appURL string, env map[string]string,
|
||||||
) (string, *output) {
|
) (string, *output) {
|
||||||
@@ -354,8 +356,22 @@ func startServerLogging(
|
|||||||
) (string, *rproxy.Server) {
|
) (string, *rproxy.Server) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
|
server := newServer(t, backend, appURL, env, requestLog, processLog)
|
||||||
|
|
||||||
|
return serve(t, server), server
|
||||||
|
}
|
||||||
|
|
||||||
|
// newServer returns rproxy, not yet serving, with its request lines going to
|
||||||
|
// requestLog and its process lines to processLog.
|
||||||
|
func newServer(
|
||||||
|
t *testing.T, backend, appURL string, env map[string]string,
|
||||||
|
requestLog, processLog io.Writer,
|
||||||
|
) *rproxy.Server {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
settings := map[string]string{
|
settings := map[string]string{
|
||||||
"SWW_RPROXY_WAF_BACKEND": backend, "SWW_RPROXY_UPSTREAM_URL": appURL,
|
"SWW_RPROXY_WAF_BACKEND": backend, "SWW_RPROXY_UPSTREAM_URL": appURL,
|
||||||
|
wafBackendTimeout: "60000",
|
||||||
}
|
}
|
||||||
maps.Copy(settings, env)
|
maps.Copy(settings, env)
|
||||||
|
|
||||||
@@ -364,15 +380,20 @@ func startServerLogging(
|
|||||||
t.Fatalf("settings %v: %v", settings, err)
|
t.Fatalf("settings %v: %v", settings, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
server := rproxy.New(rproxy.Params{
|
return rproxy.New(rproxy.Params{
|
||||||
Config: cfg,
|
Config: cfg,
|
||||||
RequestLog: requestLog,
|
RequestLog: requestLog,
|
||||||
ProcessLog: requestlog.NewProcessLogger(
|
ProcessLog: processlog.New(processlog.Params{
|
||||||
processLog,
|
Output: processLog,
|
||||||
"test",
|
InstanceName: "test",
|
||||||
slog.LevelInfo,
|
Level: slog.LevelInfo,
|
||||||
),
|
}),
|
||||||
})
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// serve has server serve until the test ends, and returns its address.
|
||||||
|
func serve(t *testing.T, server *rproxy.Server) string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
listener, err := (&net.ListenConfig{}).Listen(
|
listener, err := (&net.ListenConfig{}).Listen(
|
||||||
t.Context(),
|
t.Context(),
|
||||||
@@ -393,7 +414,7 @@ func startServerLogging(
|
|||||||
server.WaitForReports()
|
server.WaitForReports()
|
||||||
})
|
})
|
||||||
|
|
||||||
return listener.Addr().String(), server
|
return listener.Addr().String()
|
||||||
}
|
}
|
||||||
|
|
||||||
// answered is the visitor's answer.
|
// answered is the visitor's answer.
|
||||||
@@ -1395,6 +1416,171 @@ func TestSendsNoReportWithoutAnAnswer(t *testing.T) {
|
|||||||
standIn.wantNoReport(t)
|
standIn.wantNoReport(t)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestFiveFailedAsksKeepTheNextRequestFromAskingWAF(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
for name, failure := range map[string]http.HandlerFunc{
|
||||||
|
"no answer at all": func(http.ResponseWriter, *http.Request) {
|
||||||
|
panic(http.ErrAbortHandler)
|
||||||
|
},
|
||||||
|
"a slow answer": func(_ http.ResponseWriter, r *http.Request) {
|
||||||
|
<-r.Context().Done()
|
||||||
|
},
|
||||||
|
} {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
var answering atomic.Bool
|
||||||
|
|
||||||
|
standIn := startWAF(
|
||||||
|
t,
|
||||||
|
func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if answering.Load() {
|
||||||
|
allow(w, r)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
failure(w, r)
|
||||||
|
},
|
||||||
|
)
|
||||||
|
nextID := make(chan string, 1)
|
||||||
|
app := startApp(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if answering.Load() {
|
||||||
|
nextID <- r.Header.Get(requestIDHeader)
|
||||||
|
}
|
||||||
|
|
||||||
|
greet(w, r)
|
||||||
|
})
|
||||||
|
out := &output{}
|
||||||
|
server := newServer(
|
||||||
|
t, standIn.URL, app.URL,
|
||||||
|
map[string]string{wafBackendTimeout: "200"},
|
||||||
|
out, out,
|
||||||
|
)
|
||||||
|
// The breaker's clock stands still, as in
|
||||||
|
// TestAnswersThatCannotBePassedOnOpenTheBreaker: on a slow host, the five
|
||||||
|
// failures could take more than five seconds, and the next request could
|
||||||
|
// come more than a second later.
|
||||||
|
server.StopBreakerClock(time.Now())
|
||||||
|
addr := serve(t, server)
|
||||||
|
|
||||||
|
// An ask can time out before it reaches the stand-in, so these
|
||||||
|
// may leave fewer than five questions, or leave some late.
|
||||||
|
for range 5 {
|
||||||
|
wantGreeting(t, get(t, addr, "/"))
|
||||||
|
}
|
||||||
|
|
||||||
|
// rproxy writes this line before it answers a request it kept
|
||||||
|
// from asking waf, so no wait is needed.
|
||||||
|
if out.has("the circuit breaker is open") {
|
||||||
|
t.Error("the circuit breaker opened before five asks failed")
|
||||||
|
}
|
||||||
|
|
||||||
|
answering.Store(true)
|
||||||
|
wantGreeting(t, get(t, addr, "/"))
|
||||||
|
out.waitFor(t, "the circuit breaker is open")
|
||||||
|
|
||||||
|
// Had the next request asked waf, its question would have reached
|
||||||
|
// the stand-in before the request reached the app.
|
||||||
|
var id string
|
||||||
|
select {
|
||||||
|
case id = <-nextID:
|
||||||
|
default:
|
||||||
|
t.Fatal("the app did not get the next request")
|
||||||
|
}
|
||||||
|
|
||||||
|
for len(standIn.questions) > 0 {
|
||||||
|
asked := <-standIn.questions
|
||||||
|
if asked.header.Get(requestIDHeader) == id {
|
||||||
|
t.Error("waf was asked, want it not to be")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestVisitorsThatGoAwayWhileWAFIsAskedLeaveTheBreakerClosed(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
var answering atomic.Bool
|
||||||
|
|
||||||
|
standIn := startWAF(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if answering.Load() {
|
||||||
|
allow(w, r)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
<-r.Context().Done()
|
||||||
|
})
|
||||||
|
app := startApp(t, greet)
|
||||||
|
addr, out := startRProxy(t, standIn.URL, app.URL, map[string]string{
|
||||||
|
wafBackendTimeout: "60000",
|
||||||
|
})
|
||||||
|
|
||||||
|
// Five of these would open the breaker if they counted as failures.
|
||||||
|
for range 5 {
|
||||||
|
conn := dial(t, addr)
|
||||||
|
send(t, conn, "GET / HTTP/1.1\r\nHost: site.example\r\n\r\n")
|
||||||
|
id := standIn.question(t).header.Get(requestIDHeader)
|
||||||
|
|
||||||
|
_ = conn.Close()
|
||||||
|
|
||||||
|
// rproxy writes the request's log line once it is done with the ask.
|
||||||
|
out.waitFor(t, id)
|
||||||
|
}
|
||||||
|
|
||||||
|
answering.Store(true)
|
||||||
|
wantGreeting(t, get(t, addr, "/"))
|
||||||
|
standIn.question(t)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAnswersThatCannotBePassedOnOpenTheBreaker(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
var answering atomic.Bool
|
||||||
|
|
||||||
|
tooLarge := overOneMiB(dispositionDeny)
|
||||||
|
standIn := startWAF(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if answering.Load() {
|
||||||
|
allow(w, r)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
tooLarge(w, r)
|
||||||
|
})
|
||||||
|
app := startApp(t, greet)
|
||||||
|
out := &output{}
|
||||||
|
server := newServer(t, standIn.URL, app.URL, nil, out, out)
|
||||||
|
// The breaker's clock stands still. On a slow host, the five failures could
|
||||||
|
// otherwise take more than the five seconds that open the breaker. The next
|
||||||
|
// request could come more than a second later, when it may ask waf again.
|
||||||
|
server.StopBreakerClock(time.Now())
|
||||||
|
addr := serve(t, server)
|
||||||
|
|
||||||
|
for range 5 {
|
||||||
|
if got := get(t, addr, "/"); got.status != http.StatusBadGateway {
|
||||||
|
t.Errorf("answered %d, want 502", got.status)
|
||||||
|
}
|
||||||
|
|
||||||
|
standIn.question(t)
|
||||||
|
}
|
||||||
|
|
||||||
|
answering.Store(true)
|
||||||
|
wantGreeting(t, get(t, addr, "/"))
|
||||||
|
out.waitFor(t, "the circuit breaker is open")
|
||||||
|
|
||||||
|
select {
|
||||||
|
case <-standIn.questions:
|
||||||
|
t.Error("waf was asked, want it not to be")
|
||||||
|
case <-time.After(10 * pollInterval):
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestAPathUnderOwnPrefixNeverReachesTheApp(t *testing.T) {
|
func TestAPathUnderOwnPrefixNeverReachesTheApp(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -1649,11 +1835,15 @@ func TestWaitForReportsWaitsForAReportBeingSent(t *testing.T) {
|
|||||||
func TestReportOfAWebSocketCountsWhatItCarried(t *testing.T) {
|
func TestReportOfAWebSocketCountsWhatItCarried(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
// The app's answer to the upgrade arrives well within these limits, even on a
|
||||||
|
// busy host.
|
||||||
|
const limit = 5 * time.Second
|
||||||
|
|
||||||
standIn := startWAF(t, allow)
|
standIn := startWAF(t, allow)
|
||||||
app := startApp(t, echoAfterUpgrade)
|
app := startApp(t, echoAfterUpgrade)
|
||||||
addr, out := startRProxy(t, standIn.URL, app.URL, map[string]string{
|
addr, out := startRProxy(t, standIn.URL, app.URL, map[string]string{
|
||||||
maxRequestTime: "2000",
|
maxRequestTime: strconv.FormatInt(limit.Milliseconds(), 10),
|
||||||
maxResponseTime: "2000",
|
maxResponseTime: strconv.FormatInt(limit.Milliseconds(), 10),
|
||||||
})
|
})
|
||||||
|
|
||||||
conn := dial(t, addr)
|
conn := dial(t, addr)
|
||||||
@@ -1678,7 +1868,10 @@ func TestReportOfAWebSocketCountsWhatItCarried(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Past both time limits, the connection still carries what is sent.
|
// Past both time limits, the connection still carries what is sent.
|
||||||
time.Sleep(2100 * time.Millisecond)
|
time.Sleep(limit + 100*time.Millisecond)
|
||||||
|
|
||||||
|
// dial's deadline ran from before the upgrade and this wait.
|
||||||
|
_ = conn.SetDeadline(time.Now().Add(waitLimit))
|
||||||
|
|
||||||
const message = "still here\n"
|
const message = "still here\n"
|
||||||
|
|
||||||
|
|||||||
+27
-1
@@ -67,7 +67,8 @@ func (rq *request) handleNoUsableAnswer(ctx context.Context, err error) {
|
|||||||
rq.answer(http.StatusBadGateway)
|
rq.answer(http.StatusBadGateway)
|
||||||
default:
|
default:
|
||||||
// waf cannot be connected to, does not answer at all, or does not answer
|
// waf cannot be connected to, does not answer at all, or does not answer
|
||||||
// within SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS. A request that keeps waf
|
// within SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS; or the circuit breaker
|
||||||
|
// kept the request from asking. A request that keeps waf
|
||||||
// busy past SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS, like any request while
|
// busy past SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS, like any request while
|
||||||
// waf is overloaded, gets SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION
|
// waf is overloaded, gets SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION
|
||||||
// here. deny closes this, at the cost of answering 503 whenever waf is
|
// here. deny closes this, at the cost of answering 503 whenever waf is
|
||||||
@@ -86,6 +87,31 @@ func (rq *request) handleNoUsableAnswer(ctx context.Context, err error) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// askUnlessBreakerOpen asks waf, unless the circuit breaker is open and it is
|
||||||
|
// not yet time for a request to ask. It then tells the breaker whether waf
|
||||||
|
// answered in time. An answer rproxy cannot pass on counts as a failure. A
|
||||||
|
// visitor that goes away tells the breaker nothing. ctx is the request's own
|
||||||
|
// context.
|
||||||
|
func (rq *request) askUnlessBreakerOpen(
|
||||||
|
ctx context.Context,
|
||||||
|
) (*wafAnswer, error) {
|
||||||
|
b := &rq.h.breaker
|
||||||
|
if !b.mayAsk(rq.h.breakerClock()) {
|
||||||
|
return nil, errBreakerOpen
|
||||||
|
}
|
||||||
|
|
||||||
|
answer, err := rq.ask(ctx)
|
||||||
|
|
||||||
|
switch {
|
||||||
|
case err == nil:
|
||||||
|
b.noteAnswer()
|
||||||
|
case ctx.Err() == nil:
|
||||||
|
b.noteFailure(rq.h.breakerClock())
|
||||||
|
}
|
||||||
|
|
||||||
|
return answer, err
|
||||||
|
}
|
||||||
|
|
||||||
// ask sends waf the question, and returns its answer, which must have come
|
// ask sends waf the question, and returns its answer, which must have come
|
||||||
// whole within SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS. It ends when ctx does, as
|
// whole within SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS. It ends when ctx does, as
|
||||||
// when the visitor goes away.
|
// when the visitor goes away.
|
||||||
|
|||||||
+21
-50
@@ -18,12 +18,11 @@ import (
|
|||||||
"slices"
|
"slices"
|
||||||
"strings"
|
"strings"
|
||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/fsnotify/fsnotify"
|
"golang.org/x/net/http/httpguts"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/dirwatch"
|
||||||
)
|
)
|
||||||
|
|
||||||
// The actions a rule takes when it matches.
|
// The actions a rule takes when it matches.
|
||||||
@@ -40,12 +39,6 @@ const (
|
|||||||
// extension ends the name of every rule file.
|
// extension ends the name of every rule file.
|
||||||
const extension = ".rules"
|
const extension = ".rules"
|
||||||
|
|
||||||
// quietTime is how long SWW_WAF_RULE_FILES_DIR must go without a change before
|
|
||||||
// the rule files are read again, so that a file still being written, such
|
|
||||||
// as one saved in place, appended to or copied in with scp, is read only
|
|
||||||
// once whole.
|
|
||||||
const quietTime = 2 * time.Second
|
|
||||||
|
|
||||||
// headerTarget starts the target that is one request header,
|
// headerTarget starts the target that is one request header,
|
||||||
// header:<Name>.
|
// header:<Name>.
|
||||||
const headerTarget = "header:"
|
const headerTarget = "header:"
|
||||||
@@ -176,26 +169,18 @@ func (f *Files) Len() int {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Watch watches Dir until ctx is done, and reads the rule files again
|
// Watch watches Dir until ctx is done, and reads the rule files again
|
||||||
// once Dir has had no change for quietTime, after one is edited, added or
|
// once Dir has had no change for a short time, after one is edited, added
|
||||||
// removed, and after Watch starts watching. If they then hold an error,
|
// or removed, and after Watch starts watching, as dirwatch describes. If
|
||||||
// the rules stay as they were, the error is logged with its file and
|
// they then hold an error, the rules stay as they were, the error is
|
||||||
// line, and the files are read again after the next change. If Dir cannot
|
// logged with its file and line, and the files are read again after the
|
||||||
// be watched, that is logged, and the rules stay as they were loaded.
|
// next change. If Dir cannot be watched, that is logged, and the rules
|
||||||
// While Enabled is false, Watch returns at once.
|
// stay as they were loaded. While Enabled is false, Watch returns at once.
|
||||||
func (f *Files) Watch(ctx context.Context) {
|
func (f *Files) Watch(ctx context.Context) {
|
||||||
if !f.params.Enabled {
|
if !f.params.Enabled {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
watcher, err := fsnotify.NewWatcher()
|
watcher, err := dirwatch.New(f.params.Dir)
|
||||||
if err == nil {
|
|
||||||
defer func() {
|
|
||||||
_ = watcher.Close()
|
|
||||||
}()
|
|
||||||
|
|
||||||
err = watcher.Add(f.params.Dir)
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
f.params.ProcessLog.Error("cannot watch the rule files for edits",
|
f.params.ProcessLog.Error("cannot watch the rule files for edits",
|
||||||
"error", err.Error())
|
"error", err.Error())
|
||||||
@@ -203,36 +188,22 @@ func (f *Files) Watch(ctx context.Context) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
defer func() {
|
||||||
|
_ = watcher.Close()
|
||||||
|
}()
|
||||||
|
|
||||||
f.params.ProcessLog.Info("watching the rule files for edits",
|
f.params.ProcessLog.Info("watching the rule files for edits",
|
||||||
"directory", f.params.Dir)
|
"directory", f.params.Dir)
|
||||||
|
|
||||||
f.readAfterChanges(ctx, watcher.Events, watcher.Errors)
|
// A change to any file in Dir counts.
|
||||||
}
|
watcher.CallWhenQuiet(ctx,
|
||||||
|
func(string) bool { return true },
|
||||||
// readAfterChanges reads the rule files again once quietTime has passed
|
f.readAgain,
|
||||||
// without a change from events, until ctx is done, and logs the errors
|
func(err error) {
|
||||||
// from errs. The wait starts at once, as if for a change, so that an edit
|
|
||||||
// saved after Load read the files, and before Dir was watched, is taken
|
|
||||||
// in too.
|
|
||||||
func (f *Files) readAfterChanges(
|
|
||||||
ctx context.Context, events <-chan fsnotify.Event, errs <-chan error,
|
|
||||||
) {
|
|
||||||
quiet := time.NewTimer(quietTime)
|
|
||||||
defer quiet.Stop()
|
|
||||||
|
|
||||||
for {
|
|
||||||
select {
|
|
||||||
case <-ctx.Done():
|
|
||||||
return
|
|
||||||
case <-events:
|
|
||||||
quiet.Reset(quietTime)
|
|
||||||
case <-quiet.C:
|
|
||||||
f.readAgain()
|
|
||||||
case err := <-errs:
|
|
||||||
f.params.ProcessLog.Warn("watching the rule files failed",
|
f.params.ProcessLog.Warn("watching the rule files failed",
|
||||||
"error", err.Error())
|
"error", err.Error())
|
||||||
}
|
},
|
||||||
}
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
// readAgain reads the rule files again, in place of the rules loaded, or
|
// readAgain reads the rule files again, in place of the rules loaded, or
|
||||||
@@ -378,7 +349,7 @@ func parse(line string) (Rule, bool, error) {
|
|||||||
rule.Target,
|
rule.Target,
|
||||||
errNotTarget,
|
errNotTarget,
|
||||||
)
|
)
|
||||||
case isHeader && !config.IsHeaderName(headerName):
|
case isHeader && !httpguts.ValidHeaderFieldName(headerName):
|
||||||
return Rule{}, false, fmt.Errorf(
|
return Rule{}, false, fmt.Errorf(
|
||||||
"the target %q %w",
|
"the target %q %w",
|
||||||
rule.Target,
|
rule.Target,
|
||||||
|
|||||||
@@ -1,165 +0,0 @@
|
|||||||
package rules
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"log/slog"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"slices"
|
|
||||||
"testing"
|
|
||||||
"testing/synctest"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/fsnotify/fsnotify"
|
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The tests below run readAfterChanges in a synctest bubble, where time is
|
|
||||||
// a clock of the test's own: time.Sleep moves it on at once, and
|
|
||||||
// synctest.Wait returns once readAfterChanges waits again, so that every
|
|
||||||
// reading due by then is done. The test sends the changes itself, as the
|
|
||||||
// watch of a directory cannot run in a bubble.
|
|
||||||
|
|
||||||
func TestFileWrittenInTwoPartsTakenInOnlyWhole(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
dir := t.TempDir()
|
|
||||||
path := filepath.Join(dir, "50-app.rules")
|
|
||||||
writeFile(t, path, "first path block ^/first\n")
|
|
||||||
files := load(t, dir)
|
|
||||||
changes := run(t, files)
|
|
||||||
|
|
||||||
file, err := os.Create(path) //nolint:gosec // a file the test wrote
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("create: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() {
|
|
||||||
_ = file.Close()
|
|
||||||
}()
|
|
||||||
|
|
||||||
// The first part ends in the middle of a ban rule's regex, which,
|
|
||||||
// read then, would ban every request.
|
|
||||||
write(t, file, "first path block ^/first\nprobe path ban ^/")
|
|
||||||
|
|
||||||
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
|
|
||||||
|
|
||||||
time.Sleep(quietTime - time.Nanosecond)
|
|
||||||
synctest.Wait()
|
|
||||||
wantMatched(t, files, "/anything")
|
|
||||||
|
|
||||||
// The second part starts the wait again.
|
|
||||||
write(t, file, `\.env$`+"\n")
|
|
||||||
|
|
||||||
changes <- fsnotify.Event{Name: path, Op: fsnotify.Write}
|
|
||||||
|
|
||||||
time.Sleep(quietTime - time.Nanosecond)
|
|
||||||
synctest.Wait()
|
|
||||||
wantMatched(t, files, "/.env")
|
|
||||||
|
|
||||||
time.Sleep(time.Nanosecond)
|
|
||||||
synctest.Wait()
|
|
||||||
wantMatched(t, files, "/.env", "probe")
|
|
||||||
wantMatched(t, files, "/anything")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestEditSavedBeforeTheWatchStartsTakenIn(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
dir := t.TempDir()
|
|
||||||
path := filepath.Join(dir, "50-app.rules")
|
|
||||||
writeFile(t, path, "first path block ^/first\n")
|
|
||||||
files := load(t, dir)
|
|
||||||
|
|
||||||
// Saved after Load read the files, and before the directory was
|
|
||||||
// watched, so that no change is seen for it.
|
|
||||||
writeFile(t, path, "first path block ^/edited\n")
|
|
||||||
run(t, files)
|
|
||||||
time.Sleep(quietTime)
|
|
||||||
synctest.Wait()
|
|
||||||
wantMatched(t, files, "/edited", "first")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// load loads the rules in dir.
|
|
||||||
func load(t *testing.T, dir string) *Files {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
files, err := Load(Params{
|
|
||||||
Dir: dir, Enabled: true, ProcessLog: slog.New(slog.DiscardHandler),
|
|
||||||
Alerts: alerts.New(alerts.Params{}),
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("load: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return files
|
|
||||||
}
|
|
||||||
|
|
||||||
// run runs files' readAfterChanges until the test ends, and returns the
|
|
||||||
// channel that sends it changes.
|
|
||||||
func run(t *testing.T, files *Files) chan<- fsnotify.Event {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
changes := make(chan fsnotify.Event)
|
|
||||||
ctx, stop := context.WithCancel(t.Context())
|
|
||||||
stopped := make(chan struct{})
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
files.readAfterChanges(ctx, changes, nil)
|
|
||||||
close(stopped)
|
|
||||||
}()
|
|
||||||
|
|
||||||
t.Cleanup(func() {
|
|
||||||
stop()
|
|
||||||
<-stopped
|
|
||||||
})
|
|
||||||
|
|
||||||
return changes
|
|
||||||
}
|
|
||||||
|
|
||||||
// writeFile writes content to the file at path.
|
|
||||||
func writeFile(t *testing.T, path, content string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
err := os.WriteFile(path, []byte(content), 0o600)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("write %s: %v", path, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// write writes text to the end of file.
|
|
||||||
func write(t *testing.T, file *os.File, text string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
_, err := file.WriteString(text)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("write: %v", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// wantMatched checks the ids of the rules that a GET request for path
|
|
||||||
// matches, in order.
|
|
||||||
func wantMatched(t *testing.T, files *Files, path string, want ...string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
r := httptest.NewRequestWithContext(t.Context(), http.MethodGet,
|
|
||||||
"http://app.example"+path, nil)
|
|
||||||
|
|
||||||
matched := files.Match(r)
|
|
||||||
|
|
||||||
got := make([]string, 0, len(matched))
|
|
||||||
for _, rule := range matched {
|
|
||||||
got = append(got, rule.ID)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !slices.Equal(got, want) {
|
|
||||||
t.Errorf("%s matched %v, want %v", path, got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
+10
-7
@@ -46,7 +46,7 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
@@ -104,7 +104,7 @@ var (
|
|||||||
errVersion = errors.New("unknown version")
|
errVersion = errors.New("unknown version")
|
||||||
// errMissing is for an entry without a field it needs.
|
// errMissing is for an entry without a field it needs.
|
||||||
errMissing = errors.New("has no")
|
errMissing = errors.New("has no")
|
||||||
errCause = errors.New("is not limit, attack or admin")
|
errCause = errors.New("is not limit, attack, admin or abuseipdb")
|
||||||
errExpires = errors.New("is not after its start")
|
errExpires = errors.New("is not after its start")
|
||||||
errDestination = errors.New("is not webhook, slack or ntfy")
|
errDestination = errors.New("is not webhook, slack or ntfy")
|
||||||
errScope = errors.New(
|
errScope = errors.New(
|
||||||
@@ -135,7 +135,7 @@ type Params struct {
|
|||||||
// sww runs.
|
// sww runs.
|
||||||
Ledger *bans.Ledger
|
Ledger *bans.Ledger
|
||||||
Limiter *ratelimit.Limiter
|
Limiter *ratelimit.Limiter
|
||||||
GeoJS *lookup.GeoJS
|
GeoJS *geojs.GeoJS
|
||||||
Lists *reputation.Lists
|
Lists *reputation.Lists
|
||||||
DNSBL *reputation.DNSBL
|
DNSBL *reputation.DNSBL
|
||||||
AbuseIPDB *reputation.AbuseIPDB
|
AbuseIPDB *reputation.AbuseIPDB
|
||||||
@@ -195,8 +195,8 @@ type clientsFile struct {
|
|||||||
|
|
||||||
// lookupsFile is lookups.json, with each answer on a line of its own.
|
// lookupsFile is lookups.json, with each answer on a line of its own.
|
||||||
type lookupsFile struct {
|
type lookupsFile struct {
|
||||||
Version int `json:"version"`
|
Version int `json:"version"`
|
||||||
Lookups []lookup.Answer `json:"lookups"`
|
Lookups []geojs.Answer `json:"lookups"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// abuseIPDBChecksSpentFile is abuseipdb-checks-spent.json, indented for an
|
// abuseIPDBChecksSpentFile is abuseipdb-checks-spent.json, indented for an
|
||||||
@@ -638,6 +638,8 @@ func (f *Files) takeInState(
|
|||||||
f.params.Ledger.Load(held)
|
f.params.Ledger.Load(held)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
f.params.Ledger.ForgetEndedAbuseIPDBBans(f.params.Now())
|
||||||
|
|
||||||
entries = len(held)
|
entries = len(held)
|
||||||
case clientsJSON:
|
case clientsJSON:
|
||||||
var file clientsFile
|
var file clientsFile
|
||||||
@@ -967,7 +969,7 @@ func (e BanEntry) ban() bans.Ban {
|
|||||||
// each expires is read again as written. An expires not after the start
|
// each expires is read again as written. An expires not after the start
|
||||||
// is refused too, since the netblock's next ban for a broken limit, three
|
// is refused too, since the netblock's next ban for a broken limit, three
|
||||||
// times as long as this one, would then refuse nothing; and so is a cause
|
// times as long as this one, would then refuse nothing; and so is a cause
|
||||||
// other than limit, attack or admin, most likely misspelt.
|
// other than limit, attack, admin or abuseipdb, most likely misspelt.
|
||||||
func (f *bansFile) check(data []byte) error {
|
func (f *bansFile) check(data []byte) error {
|
||||||
var written struct {
|
var written struct {
|
||||||
Bans []struct {
|
Bans []struct {
|
||||||
@@ -991,7 +993,8 @@ func (f *bansFile) check(data []byte) error {
|
|||||||
case entry.Expires != nil && !entry.Expires.After(entry.Start):
|
case entry.Expires != nil && !entry.Expires.After(entry.Start):
|
||||||
return fmt.Errorf("entry %d's expires %w", i+1, errExpires)
|
return fmt.Errorf("entry %d's expires %w", i+1, errExpires)
|
||||||
case entry.Cause != "" && entry.Cause != bans.CauseLimit &&
|
case entry.Cause != "" && entry.Cause != bans.CauseLimit &&
|
||||||
entry.Cause != bans.CauseAttack && entry.Cause != bans.CauseAdmin:
|
entry.Cause != bans.CauseAttack && entry.Cause != bans.CauseAdmin &&
|
||||||
|
entry.Cause != bans.CauseAbuseIPDB:
|
||||||
return fmt.Errorf(
|
return fmt.Errorf(
|
||||||
"entry %d's cause %q %w",
|
"entry %d's cause %q %w",
|
||||||
i+1,
|
i+1,
|
||||||
|
|||||||
@@ -26,7 +26,7 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
@@ -114,7 +114,8 @@ const permanentBansJSON = `{
|
|||||||
"earlier_bans": {
|
"earlier_bans": {
|
||||||
"limit": 3,
|
"limit": 3,
|
||||||
"attack": 1,
|
"attack": 1,
|
||||||
"admin": 1
|
"admin": 1,
|
||||||
|
"abuseipdb": 0
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -159,7 +160,7 @@ const filledAlertsJSON = `{
|
|||||||
"waiting": {
|
"waiting": {
|
||||||
"webhook": [
|
"webhook": [
|
||||||
{
|
{
|
||||||
"instance": "fsn1app1/blog",
|
"instance": "host1/blog",
|
||||||
"time": "2026-10-06T00:00:00Z",
|
"time": "2026-10-06T00:00:00Z",
|
||||||
"event": "ban",
|
"event": "ban",
|
||||||
"client": "203.0.113.9",
|
"client": "203.0.113.9",
|
||||||
@@ -174,7 +175,7 @@ const filledAlertsJSON = `{
|
|||||||
"suppressed_repeats": 0
|
"suppressed_repeats": 0
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"instance": "fsn1app1/blog",
|
"instance": "host1/blog",
|
||||||
"time": "2026-10-06T00:00:00Z",
|
"time": "2026-10-06T00:00:00Z",
|
||||||
"event": "file_error",
|
"event": "file_error",
|
||||||
"client": "",
|
"client": "",
|
||||||
@@ -1190,7 +1191,7 @@ func TestBanWithAnotherCauseStopsTheStart(t *testing.T) {
|
|||||||
`"expires": "2026-10-06T04:00:00Z", "cause": "limit"}, `+
|
`"expires": "2026-10-06T04:00:00Z", "cause": "limit"}, `+
|
||||||
`{"netblock": "203.0.113.12/32", "start": "2026-10-06T00:00:00Z", `+
|
`{"netblock": "203.0.113.12/32", "start": "2026-10-06T00:00:00Z", `+
|
||||||
`"expires": null, "cause": "atack"}]}`,
|
`"expires": null, "cause": "atack"}]}`,
|
||||||
`: entry 4's cause "atack" is not limit, attack or admin`)
|
`: entry 4's cause "atack" is not limit, attack, admin or abuseipdb`)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestBanThatDoesNotEndAfterItsStartStopsTheStart(t *testing.T) {
|
func TestBanThatDoesNotEndAfterItsStartStopsTheStart(t *testing.T) {
|
||||||
@@ -1283,18 +1284,18 @@ func TestBansWrittenOnceBansWriteDelayAfterABan(t *testing.T) {
|
|||||||
|
|
||||||
// A second ban, made while the first waits to be written, puts the
|
// A second ban, made while the first waits to be written, puts the
|
||||||
// write off no further, and is written with it.
|
// write off no further, and is written with it.
|
||||||
first, _ := params.Ledger.BanForLimit(
|
first, _ := params.Ledger.Ban(
|
||||||
netip.MustParsePrefix("203.0.113.9/32"),
|
netip.MustParsePrefix("203.0.113.9/32"),
|
||||||
midnight(),
|
midnight(),
|
||||||
bans.Notes{},
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
)
|
)
|
||||||
|
|
||||||
time.Sleep(5 * time.Second)
|
time.Sleep(5 * time.Second)
|
||||||
|
|
||||||
second, _ := params.Ledger.BanForLimit(
|
second, _ := params.Ledger.Ban(
|
||||||
netip.MustParsePrefix("203.0.113.10/32"),
|
netip.MustParsePrefix("203.0.113.10/32"),
|
||||||
midnight(),
|
midnight(),
|
||||||
bans.Notes{},
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
)
|
)
|
||||||
|
|
||||||
time.Sleep(5*time.Second - time.Nanosecond)
|
time.Sleep(5*time.Second - time.Nanosecond)
|
||||||
@@ -1357,16 +1358,16 @@ func TestEditJustBeforeAScheduledWriteSurvivesIt(t *testing.T) {
|
|||||||
run(t, load(t, params).Run)
|
run(t, load(t, params).Run)
|
||||||
|
|
||||||
// A ban, and bans.json written with it.
|
// A ban, and bans.json written with it.
|
||||||
params.Ledger.BanForLimit(netip.MustParsePrefix("203.0.113.9/32"),
|
params.Ledger.Ban(netip.MustParsePrefix("203.0.113.9/32"),
|
||||||
midnight(), bans.Notes{})
|
midnight(), bans.Grounds{Cause: bans.CauseLimit})
|
||||||
time.Sleep(params.BansWriteDelay)
|
time.Sleep(params.BansWriteDelay)
|
||||||
synctest.Wait()
|
synctest.Wait()
|
||||||
|
|
||||||
// A second ban is to be written BansWriteDelay later. Just before
|
// A second ban is to be written BansWriteDelay later. Just before
|
||||||
// then, an admin saves bans.json with the first ban lifted and
|
// then, an admin saves bans.json with the first ban lifted and
|
||||||
// another added.
|
// another added.
|
||||||
params.Ledger.BanForLimit(netip.MustParsePrefix("203.0.113.10/32"),
|
params.Ledger.Ban(netip.MustParsePrefix("203.0.113.10/32"),
|
||||||
midnight(), bans.Notes{})
|
midnight(), bans.Grounds{Cause: bans.CauseLimit})
|
||||||
time.Sleep(params.BansWriteDelay - time.Nanosecond)
|
time.Sleep(params.BansWriteDelay - time.Nanosecond)
|
||||||
synctest.Wait()
|
synctest.Wait()
|
||||||
edit(t, dir, bansJSON, permanentBansJSON)
|
edit(t, dir, bansJSON, permanentBansJSON)
|
||||||
@@ -1482,10 +1483,10 @@ func TestFailedWriteLeavesTheFileAsItWas(t *testing.T) {
|
|||||||
t.Fatalf("mkdir: %v", err)
|
t.Fatalf("mkdir: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
params.Ledger.BanForLimit(
|
params.Ledger.Ban(
|
||||||
netip.MustParsePrefix("203.0.113.9/32"),
|
netip.MustParsePrefix("203.0.113.9/32"),
|
||||||
midnight(),
|
midnight(),
|
||||||
bans.Notes{},
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
)
|
)
|
||||||
params.Limiter.Count(
|
params.Limiter.Count(
|
||||||
netip.MustParsePrefix("203.0.113.9/32"),
|
netip.MustParsePrefix("203.0.113.9/32"),
|
||||||
@@ -1555,10 +1556,10 @@ func TestWritesAreCountedInTheMetrics(t *testing.T) {
|
|||||||
t.Fatalf("mkdir: %v", err)
|
t.Fatalf("mkdir: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
params.Ledger.BanForLimit(
|
params.Ledger.Ban(
|
||||||
netip.MustParsePrefix("203.0.113.9/32"),
|
netip.MustParsePrefix("203.0.113.9/32"),
|
||||||
midnight(),
|
midnight(),
|
||||||
bans.Notes{},
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
)
|
)
|
||||||
|
|
||||||
err = files.WriteAll()
|
err = files.WriteAll()
|
||||||
@@ -1700,7 +1701,7 @@ func TestEditOfEachFileTakenIn(t *testing.T) {
|
|||||||
t,
|
t,
|
||||||
lookupsJSON,
|
lookupsJSON,
|
||||||
params.GeoJS.Snapshot(),
|
params.GeoJS.Snapshot(),
|
||||||
[]lookup.Answer{
|
[]geojs.Answer{
|
||||||
{Client: client, Country: "FR", Answered: midnight()},
|
{Client: client, Country: "FR", Answered: midnight()},
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
@@ -2090,6 +2091,39 @@ func TestLiftedBanReadAtTheStart(t *testing.T) {
|
|||||||
wantLiftedBanKept(t, load(t, params), dir, params.Ledger)
|
wantLiftedBanKept(t, load(t, params), dir, params.Ledger)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestEndedAbuseIPDBBanForgottenWhenBansJSONIsReadAtTheStart(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
ended := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
|
lasting := netip.MustParsePrefix("203.0.113.10/32")
|
||||||
|
limited := netip.MustParsePrefix("203.0.113.11/32")
|
||||||
|
|
||||||
|
// At midnight, the start's time, the first ban has ended and the second
|
||||||
|
// lasts. The third has ended too, but is for a limit.
|
||||||
|
dir := t.TempDir()
|
||||||
|
edit(t, dir, bansJSON, `{"version": 1, "bans": [`+
|
||||||
|
`{"netblock": "203.0.113.9/32", "start": "2026-09-06T00:00:00Z", `+
|
||||||
|
`"expires": "2026-10-06T00:00:00Z", "cause": "abuseipdb"}, `+
|
||||||
|
`{"netblock": "203.0.113.10/32", "start": "2026-09-07T00:00:00Z", `+
|
||||||
|
`"expires": "2026-10-07T00:00:00Z", "cause": "abuseipdb"}, `+
|
||||||
|
`{"netblock": "203.0.113.11/32", "start": "2026-10-05T00:00:00Z", `+
|
||||||
|
`"expires": "2026-10-05T01:00:00Z", "cause": "limit"}]}`)
|
||||||
|
|
||||||
|
params := newParams(dir)
|
||||||
|
load(t, params)
|
||||||
|
|
||||||
|
for netblock, want := range map[netip.Prefix]int{
|
||||||
|
ended: 0, lasting: 1, limited: 1,
|
||||||
|
} {
|
||||||
|
if got := len(params.Ledger.Bans(netblock)); got != want {
|
||||||
|
t.Errorf("the ledger holds %d bans on %s, want %d",
|
||||||
|
got, netblock, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestBanLiftedByAnEditWhileRunning(t *testing.T) {
|
func TestBanLiftedByAnEditWhileRunning(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -2101,7 +2135,11 @@ func TestBanLiftedByAnEditWhileRunning(t *testing.T) {
|
|||||||
|
|
||||||
// The ban that liftedBansJSON lifts, before it is lifted.
|
// The ban that liftedBansJSON lifts, before it is lifted.
|
||||||
netblock := netip.MustParsePrefix(liftedClient + "/32")
|
netblock := netip.MustParsePrefix(liftedClient + "/32")
|
||||||
params.Ledger.BanForLimit(netblock, midnight(), bans.Notes{})
|
params.Ledger.Ban(
|
||||||
|
netblock,
|
||||||
|
midnight(),
|
||||||
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
|
)
|
||||||
|
|
||||||
_, banned, _ := params.Ledger.Find(netblock.Addr(), afterLifting())
|
_, banned, _ := params.Ledger.Find(netblock.Addr(), afterLifting())
|
||||||
if !banned {
|
if !banned {
|
||||||
@@ -2398,7 +2436,7 @@ func newParams(dir string) state.Params {
|
|||||||
Events: alerts.Events(),
|
Events: alerts.Events(),
|
||||||
Cooldown: 15 * time.Minute,
|
Cooldown: 15 * time.Minute,
|
||||||
MaxPerHour: 2,
|
MaxPerHour: 2,
|
||||||
Instance: "fsn1app1/blog",
|
Instance: "host1/blog",
|
||||||
Now: midnight,
|
Now: midnight,
|
||||||
})
|
})
|
||||||
|
|
||||||
@@ -2414,10 +2452,8 @@ func newParams(dir string) state.Params {
|
|||||||
AttackBanDuration: 7 * 24 * time.Hour,
|
AttackBanDuration: 7 * 24 * time.Hour,
|
||||||
MaxBans: 5000,
|
MaxBans: 5000,
|
||||||
}),
|
}),
|
||||||
Limiter: ratelimit.New(ratelimit.Limits{}, 20000),
|
Limiter: ratelimit.New(ratelimit.Params{MaxClients: 20000}),
|
||||||
GeoJS: lookup.New(lookup.Params{
|
GeoJS: geojs.New(geojs.Params{Now: midnight}),
|
||||||
Now: midnight, ProcessLog: discard, Metrics: m,
|
|
||||||
}),
|
|
||||||
Lists: reputation.New(reputation.Params{
|
Lists: reputation.New(reputation.Params{
|
||||||
BlocklistURLs: []string{
|
BlocklistURLs: []string{
|
||||||
blocklistURL,
|
blocklistURL,
|
||||||
@@ -2478,11 +2514,15 @@ func fill(params state.Params) {
|
|||||||
client := netip.MustParsePrefix("203.0.113.9/32")
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
|
|
||||||
params.Ledger.Load([]bans.Ban{permanentBan()})
|
params.Ledger.Load([]bans.Ban{permanentBan()})
|
||||||
params.Ledger.BanForLimit(client, now, bans.Notes{
|
params.Ledger.Ban(client, now, bans.Grounds{
|
||||||
ASN: asn, ASName: asName, Country: "DE", Limit: 1,
|
Cause: bans.CauseLimit,
|
||||||
|
Notes: bans.Notes{ASN: asn, ASName: asName, Country: "DE", Limit: 1},
|
||||||
})
|
})
|
||||||
params.Ledger.BanForAttack(netip.MustParsePrefix("192.0.2.1/32"), now,
|
params.Ledger.Ban(netip.MustParsePrefix("192.0.2.1/32"), now,
|
||||||
bans.Notes{RuleID: "env-file", Target: "path"})
|
bans.Grounds{
|
||||||
|
Cause: bans.CauseAttack,
|
||||||
|
Notes: bans.Notes{RuleID: "env-file", Target: "path"},
|
||||||
|
})
|
||||||
|
|
||||||
for _, c := range []string{"2001:db8::/64", "203.0.113.9/32", "192.0.2.1/32"} {
|
for _, c := range []string{"2001:db8::/64", "203.0.113.9/32", "192.0.2.1/32"} {
|
||||||
params.Limiter.Count(netip.MustParsePrefix(c), now, whole)
|
params.Limiter.Count(netip.MustParsePrefix(c), now, whole)
|
||||||
@@ -2494,7 +2534,7 @@ func fill(params state.Params) {
|
|||||||
})
|
})
|
||||||
params.Limiter.AddLookup(client, now.Add(-time.Hour), asn, asName, "DE")
|
params.Limiter.AddLookup(client, now.Add(-time.Hour), asn, asName, "DE")
|
||||||
|
|
||||||
params.GeoJS.Load([]lookup.Answer{
|
params.GeoJS.Load([]geojs.Answer{
|
||||||
{
|
{
|
||||||
Client: client, ASN: asn, ASName: asName, Country: "DE",
|
Client: client, ASN: asn, ASName: asName, Country: "DE",
|
||||||
Answered: now.Add(-time.Hour), Used: now,
|
Answered: now.Add(-time.Hour), Used: now,
|
||||||
@@ -2668,7 +2708,11 @@ func wantLiftedBanKept(
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Were the lifted ban counted, the next would last three hours.
|
// Were the lifted ban counted, the next would last three hours.
|
||||||
ban, _ := ledger.BanForLimit(netblock, afterLifting(), bans.Notes{})
|
ban, _ := ledger.Ban(
|
||||||
|
netblock,
|
||||||
|
afterLifting(),
|
||||||
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
|
)
|
||||||
if ban.Expires.Sub(ban.Start) != time.Hour {
|
if ban.Expires.Sub(ban.Start) != time.Hour {
|
||||||
t.Errorf(
|
t.Errorf(
|
||||||
"the next ban lasts %s, want 1h",
|
"the next ban lasts %s, want 1h",
|
||||||
|
|||||||
@@ -0,0 +1,198 @@
|
|||||||
|
package sww
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
"log/slog"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"go.uber.org/fx"
|
||||||
|
"go.uber.org/fx/fxevent"
|
||||||
|
)
|
||||||
|
|
||||||
|
// shutdownTimeout is how long requests in progress may take to finish
|
||||||
|
// once sww is told to stop, before their connections are closed.
|
||||||
|
// runit and docker wait a little longer before they kill the process.
|
||||||
|
const shutdownTimeout = 5 * time.Second
|
||||||
|
|
||||||
|
// requestLog is where a daemon's server writes the request log: stdout, and
|
||||||
|
// the remote log sender while there is one. fx tells parts apart by type, so
|
||||||
|
// it has a type of its own.
|
||||||
|
type requestLog struct {
|
||||||
|
io.Writer
|
||||||
|
}
|
||||||
|
|
||||||
|
// servingFailed gets why serving failed, for run, which then stops the
|
||||||
|
// daemon. fx tells parts apart by type, so it has a type of its own.
|
||||||
|
type servingFailed chan error
|
||||||
|
|
||||||
|
// utcNow is the time the state files, the alerts and the rate limits go by.
|
||||||
|
func utcNow() time.Time {
|
||||||
|
return time.Now().UTC()
|
||||||
|
}
|
||||||
|
|
||||||
|
// startError is why a daemon cannot start: the message its process log gives,
|
||||||
|
// and the error it logs with it.
|
||||||
|
type startError struct {
|
||||||
|
message string
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e startError) Error() string {
|
||||||
|
return e.message + ": " + e.err.Error()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e startError) Unwrap() error {
|
||||||
|
return e.err
|
||||||
|
}
|
||||||
|
|
||||||
|
// cannotStart logs why a daemon cannot start, with the message of the
|
||||||
|
// startError in err, and returns its exit status.
|
||||||
|
func cannotStart(processLog *slog.Logger, err error) int {
|
||||||
|
message, cause := "cannot start", err
|
||||||
|
|
||||||
|
failed, ok := errors.AsType[startError](err)
|
||||||
|
if ok {
|
||||||
|
message, cause = failed.message, failed.err
|
||||||
|
}
|
||||||
|
|
||||||
|
processLog.Error(message, "error", cause.Error())
|
||||||
|
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
// listen listens on address, which the setting names.
|
||||||
|
func listen(address, setting string) (net.Listener, error) {
|
||||||
|
listener, err := (&net.ListenConfig{}).Listen(
|
||||||
|
context.Background(), "tcp", address)
|
||||||
|
if err != nil {
|
||||||
|
return nil, startError{"cannot listen on " + setting, err}
|
||||||
|
}
|
||||||
|
|
||||||
|
return listener, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// background returns the hook that runs loop on a goroutine of its own from
|
||||||
|
// the start, and at the stop ends it and waits for it to return.
|
||||||
|
func background(loop func(context.Context)) fx.Hook {
|
||||||
|
running, stop := context.WithCancel(context.Background())
|
||||||
|
ended := make(chan struct{})
|
||||||
|
|
||||||
|
return fx.StartStopHook(
|
||||||
|
func() {
|
||||||
|
go func() {
|
||||||
|
loop(running)
|
||||||
|
close(ended)
|
||||||
|
}()
|
||||||
|
},
|
||||||
|
func() {
|
||||||
|
stop()
|
||||||
|
<-ended
|
||||||
|
},
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// serve serves requests on listener from the start. At the stop it gives the
|
||||||
|
// requests in progress shutdownTimeout to finish, then closes their
|
||||||
|
// connections. When serving fails, it logs why and sends the error to
|
||||||
|
// failed.
|
||||||
|
func serve(
|
||||||
|
lc fx.Lifecycle, server *http.Server, listener net.Listener,
|
||||||
|
processLog *slog.Logger, failed servingFailed,
|
||||||
|
) {
|
||||||
|
served := make(chan error, 1)
|
||||||
|
|
||||||
|
lc.Append(fx.Hook{
|
||||||
|
OnStart: func(context.Context) error {
|
||||||
|
go func() {
|
||||||
|
err := server.Serve(listener)
|
||||||
|
if !errors.Is(err, http.ErrServerClosed) {
|
||||||
|
processLog.Error("serving failed", "error", err.Error())
|
||||||
|
|
||||||
|
failed <- err
|
||||||
|
}
|
||||||
|
|
||||||
|
served <- err
|
||||||
|
}()
|
||||||
|
|
||||||
|
return nil
|
||||||
|
},
|
||||||
|
OnStop: func(ctx context.Context) error {
|
||||||
|
processLog.Info("stopping")
|
||||||
|
|
||||||
|
shutdownCtx, cancel := context.WithTimeout(ctx, shutdownTimeout)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
err := server.Shutdown(shutdownCtx)
|
||||||
|
if err != nil {
|
||||||
|
processLog.Warn("requests still in progress were cut off",
|
||||||
|
"error", err.Error())
|
||||||
|
|
||||||
|
_ = server.Close()
|
||||||
|
}
|
||||||
|
|
||||||
|
err = <-served
|
||||||
|
if !errors.Is(err, http.ErrServerClosed) {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
},
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// run starts app, which serves until ctx is done or serving fails, as
|
||||||
|
// failed tells. Then it stops app and takes the daemon's last step. It
|
||||||
|
// returns the process's exit status, 1 when serving failed, a part could
|
||||||
|
// not stop in order or the last step failed; each logs why.
|
||||||
|
func run(
|
||||||
|
ctx context.Context, app *fx.App, failed servingFailed,
|
||||||
|
processLog *slog.Logger, lastStep func() error,
|
||||||
|
) int {
|
||||||
|
// The hooks only start goroutines, and end them at the stop, so neither
|
||||||
|
// is cut short by a deadline or by ctx.
|
||||||
|
err := app.Start(context.WithoutCancel(ctx))
|
||||||
|
if err != nil {
|
||||||
|
return cannotStart(processLog, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
status := 0
|
||||||
|
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
case <-failed:
|
||||||
|
status = 1
|
||||||
|
}
|
||||||
|
|
||||||
|
err = app.Stop(context.WithoutCancel(ctx))
|
||||||
|
if err != nil {
|
||||||
|
status = 1
|
||||||
|
}
|
||||||
|
|
||||||
|
err = lastStep()
|
||||||
|
if err != nil {
|
||||||
|
status = 1
|
||||||
|
}
|
||||||
|
|
||||||
|
if status == 0 {
|
||||||
|
processLog.Info("stopped")
|
||||||
|
}
|
||||||
|
|
||||||
|
return status
|
||||||
|
}
|
||||||
|
|
||||||
|
// fxEventLogger returns what fx.WithLogger takes to write fx's own events to
|
||||||
|
// processLog at debug, which the daemons' log levels normally hold back. Its
|
||||||
|
// errors are at debug too: the daemon logs each one itself.
|
||||||
|
func fxEventLogger(processLog *slog.Logger) func() fxevent.Logger {
|
||||||
|
return func() fxevent.Logger {
|
||||||
|
logger := &fxevent.SlogLogger{Logger: processLog}
|
||||||
|
logger.UseLogLevel(slog.LevelDebug)
|
||||||
|
logger.UseErrorLevel(slog.LevelDebug)
|
||||||
|
|
||||||
|
return logger
|
||||||
|
}
|
||||||
|
}
|
||||||
+48
-69
@@ -2,14 +2,15 @@ package sww
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"errors"
|
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"net"
|
"net"
|
||||||
"net/http"
|
"net/http"
|
||||||
"os"
|
"os"
|
||||||
|
|
||||||
|
"go.uber.org/fx"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/processlog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/rproxy"
|
"sneak.berlin/go/smallwebwaf/internal/rproxy"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -22,34 +23,39 @@ import (
|
|||||||
// setting for either.
|
// setting for either.
|
||||||
func RunRProxy(ctx context.Context, params Params) int {
|
func RunRProxy(ctx context.Context, params Params) int {
|
||||||
hostname, _ := os.Hostname() // "" when the host has no name to give
|
hostname, _ := os.Hostname() // "" when the host has no name to give
|
||||||
processLog := requestlog.NewProcessLogger(
|
processLog := processlog.New(processlog.Params{
|
||||||
params.Stdout,
|
Output: params.Stdout,
|
||||||
hostname,
|
InstanceName: hostname,
|
||||||
slog.LevelInfo,
|
Level: slog.LevelInfo,
|
||||||
)
|
})
|
||||||
|
|
||||||
cfg, err := config.RProxyFromEnvironment(params.Viper)
|
cfg, err := config.RProxyFromEnvironment(params.Viper)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
processLog.Error("invalid setting", "error", err.Error())
|
return cannotStart(processLog, startError{"invalid setting", err})
|
||||||
|
|
||||||
return 1
|
|
||||||
}
|
}
|
||||||
|
|
||||||
server := rproxy.New(rproxy.Params{
|
var (
|
||||||
Config: cfg,
|
server *rproxy.Server
|
||||||
RequestLog: params.Stdout,
|
listener net.Listener
|
||||||
ProcessLog: processLog,
|
)
|
||||||
})
|
|
||||||
|
|
||||||
listener, err := (&net.ListenConfig{}).Listen(ctx, "tcp", cfg.ListenAddr)
|
failed := make(servingFailed, 1)
|
||||||
|
|
||||||
|
app := fx.New(
|
||||||
|
fx.Supply(cfg, processLog, requestLog{params.Stdout}, failed),
|
||||||
|
fx.Provide(
|
||||||
|
newRProxyServer,
|
||||||
|
func(server *rproxy.Server) *http.Server { return server.Server },
|
||||||
|
newRProxyListener,
|
||||||
|
),
|
||||||
|
fx.Invoke(serve),
|
||||||
|
fx.Populate(&server, &listener),
|
||||||
|
fx.WithLogger(fxEventLogger(processLog)),
|
||||||
|
)
|
||||||
|
|
||||||
|
err = app.Err()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
processLog.Error(
|
return cannotStart(processLog, err)
|
||||||
"cannot listen on SWW_RPROXY_LISTEN_ADDR",
|
|
||||||
"error",
|
|
||||||
err.Error(),
|
|
||||||
)
|
|
||||||
|
|
||||||
return 1
|
|
||||||
}
|
}
|
||||||
|
|
||||||
processLog.Info("starting",
|
processLog.Info("starting",
|
||||||
@@ -57,52 +63,25 @@ func RunRProxy(ctx context.Context, params Params) int {
|
|||||||
"address", listener.Addr().String(),
|
"address", listener.Addr().String(),
|
||||||
"settings", cfg)
|
"settings", cfg)
|
||||||
|
|
||||||
return serveRProxy(ctx, server, listener, processLog)
|
return run(ctx, app, failed, processLog, func() error {
|
||||||
|
server.WaitForReports()
|
||||||
|
|
||||||
|
return nil
|
||||||
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
// serveRProxy serves requests on listener until ctx is done, and then stops as
|
// newRProxyServer returns the server rproxy runs.
|
||||||
// RunRProxy says.
|
func newRProxyServer(
|
||||||
func serveRProxy(
|
cfg *config.RProxy, out requestLog, processLog *slog.Logger,
|
||||||
ctx context.Context, server *rproxy.Server, listener net.Listener,
|
) *rproxy.Server {
|
||||||
processLog *slog.Logger,
|
return rproxy.New(rproxy.Params{
|
||||||
) int {
|
Config: cfg,
|
||||||
served := make(chan error, 1)
|
RequestLog: out,
|
||||||
|
ProcessLog: processLog,
|
||||||
go func() {
|
})
|
||||||
served <- server.Serve(listener)
|
}
|
||||||
}()
|
|
||||||
|
// newRProxyListener listens on SWW_RPROXY_LISTEN_ADDR.
|
||||||
select {
|
func newRProxyListener(cfg *config.RProxy) (net.Listener, error) {
|
||||||
case err := <-served:
|
return listen(cfg.ListenAddr, "SWW_RPROXY_LISTEN_ADDR")
|
||||||
processLog.Error("serving failed", "error", err.Error())
|
|
||||||
|
|
||||||
return 1
|
|
||||||
case <-ctx.Done():
|
|
||||||
}
|
|
||||||
|
|
||||||
processLog.Info("stopping")
|
|
||||||
|
|
||||||
shutdownCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx),
|
|
||||||
shutdownTimeout)
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
err := server.Shutdown(shutdownCtx)
|
|
||||||
if err != nil {
|
|
||||||
processLog.Warn("requests still in progress were cut off",
|
|
||||||
"error", err.Error())
|
|
||||||
|
|
||||||
_ = server.Close()
|
|
||||||
}
|
|
||||||
|
|
||||||
err = <-served
|
|
||||||
if !errors.Is(err, http.ErrServerClosed) {
|
|
||||||
processLog.Error("serving failed", "error", err.Error())
|
|
||||||
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
|
|
||||||
server.WaitForReports()
|
|
||||||
processLog.Info("stopped")
|
|
||||||
|
|
||||||
return 0
|
|
||||||
}
|
}
|
||||||
+24
-11
@@ -51,7 +51,9 @@ func runRProxy(ctx context.Context, env map[string]string, out *output) int {
|
|||||||
|
|
||||||
// startRProxy runs rproxy with waf at backend and the app at appURL, and
|
// startRProxy runs rproxy with waf at backend and the app at appURL, and
|
||||||
// returns its URL and what it writes. It believes X-Forwarded-For from the
|
// returns its URL and what it writes. It believes X-Forwarded-For from the
|
||||||
// tests. It stops as the test ends, and that it stopped in order is checked.
|
// tests. It waits a minute for waf's answer, so that a slow answer on a busy
|
||||||
|
// host is not taken for no answer. It stops as the test ends, and that it
|
||||||
|
// stopped in order is checked.
|
||||||
func startRProxy(t *testing.T, backend, appURL string) (string, *output) {
|
func startRProxy(t *testing.T, backend, appURL string) (string, *output) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
@@ -65,6 +67,7 @@ func startRProxy(t *testing.T, backend, appURL string) (string, *output) {
|
|||||||
"SWW_RPROXY_UPSTREAM_URL": appURL,
|
"SWW_RPROXY_UPSTREAM_URL": appURL,
|
||||||
"SWW_RPROXY_TRUSTED_PROXIES": localhost + "/32",
|
"SWW_RPROXY_TRUSTED_PROXIES": localhost + "/32",
|
||||||
wafBackend: backend,
|
wafBackend: backend,
|
||||||
|
wafBackendTimeout: "60000",
|
||||||
}, out)
|
}, out)
|
||||||
}()
|
}()
|
||||||
|
|
||||||
@@ -260,7 +263,17 @@ func TestTheSiteAnswersWhileWAFIsStoppedAndStartedAgain(t *testing.T) {
|
|||||||
stop()
|
stop()
|
||||||
}()
|
}()
|
||||||
|
|
||||||
wantStatus(t, site+"/", placed, http.StatusForbidden)
|
// The visitor's asks have opened rproxy's circuit breaker, which lets one
|
||||||
|
// request a second ask until one is answered.
|
||||||
|
deadline := time.Now().Add(waitLimit)
|
||||||
|
for statusFrom(t, site+"/", placed) != http.StatusForbidden {
|
||||||
|
if time.Now().After(deadline) {
|
||||||
|
t.Fatalf("request from %s: never answered %d", placed,
|
||||||
|
http.StatusForbidden)
|
||||||
|
}
|
||||||
|
|
||||||
|
time.Sleep(pollInterval)
|
||||||
|
}
|
||||||
|
|
||||||
close(visitorStop)
|
close(visitorStop)
|
||||||
<-visitorDone
|
<-visitorDone
|
||||||
@@ -539,7 +552,9 @@ func TestAStatusWAFDoesNotTakeIsReportedAs599(t *testing.T) {
|
|||||||
// startRProxyProcess runs `sww rproxy` in a process of its own, with the
|
// startRProxyProcess runs `sww rproxy` in a process of its own, with the
|
||||||
// settings in env, each as NAME=value. It returns where rproxy listens and a
|
// settings in env, each as NAME=value. It returns where rproxy listens and a
|
||||||
// function that sends rproxy SIGTERM and fails the test unless it exits with
|
// function that sends rproxy SIGTERM and fails the test unless it exits with
|
||||||
// status 0 within waitLimit.
|
// status 0 within waitLimit. It waits a minute for waf's answers unless env
|
||||||
|
// sets SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS, so that a busy host cannot make it
|
||||||
|
// drop a question.
|
||||||
func startRProxyProcess(t *testing.T, env ...string) (string, func()) {
|
func startRProxyProcess(t *testing.T, env ...string) (string, func()) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
@@ -555,7 +570,8 @@ func startRProxyProcess(t *testing.T, env ...string) (string, func()) {
|
|||||||
|
|
||||||
cmd.Env = append(os.Environ(),
|
cmd.Env = append(os.Environ(),
|
||||||
commandForMain+"=rproxy",
|
commandForMain+"=rproxy",
|
||||||
"SWW_RPROXY_LISTEN_ADDR="+localhost+":0")
|
"SWW_RPROXY_LISTEN_ADDR="+localhost+":0",
|
||||||
|
wafBackendTimeout+"=60000")
|
||||||
cmd.Env = append(cmd.Env, env...)
|
cmd.Env = append(cmd.Env, env...)
|
||||||
|
|
||||||
exited := startProcess(t, cmd)
|
exited := startProcess(t, cmd)
|
||||||
@@ -739,8 +755,7 @@ func TestRProxyStoppingWaitsForAReportBeingSent(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// sendAndKeepOpen sends request to rproxy at addr, and returns the connection. A
|
// sendAndKeepOpen sends request to rproxy at addr, and returns the connection.
|
||||||
// read from it ends with an error once waitLimit is up.
|
|
||||||
func sendAndKeepOpen(t *testing.T, addr, request string) net.Conn {
|
func sendAndKeepOpen(t *testing.T, addr, request string) net.Conn {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
@@ -753,11 +768,6 @@ func sendAndKeepOpen(t *testing.T, addr, request string) net.Conn {
|
|||||||
_ = conn.Close()
|
_ = conn.Close()
|
||||||
})
|
})
|
||||||
|
|
||||||
err = conn.SetDeadline(time.Now().Add(waitLimit))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("set the deadline: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = io.WriteString(conn, request)
|
_, err = io.WriteString(conn, request)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("send: %v", err)
|
t.Fatalf("send: %v", err)
|
||||||
@@ -822,6 +832,9 @@ func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
|
|||||||
|
|
||||||
stop()
|
stop()
|
||||||
|
|
||||||
|
// Counted from the stop, so a slow wait for the app cannot shorten it.
|
||||||
|
_ = conn.SetReadDeadline(stopped.Add(cutOffAfter + waitLimit))
|
||||||
|
|
||||||
text, err := io.ReadAll(conn)
|
text, err := io.ReadAll(conn)
|
||||||
if err != nil || len(text) != 0 {
|
if err != nil || len(text) != 0 {
|
||||||
t.Errorf("read %q (%v), want the connection cut off", text, err)
|
t.Errorf("read %q (%v), want the connection cut off", text, err)
|
||||||
|
|||||||
+5
-393
@@ -4,17 +4,14 @@
|
|||||||
// stop, and then stops in an orderly way, writing the state and cache files.
|
// stop, and then stops in an orderly way, writing the state and cache files.
|
||||||
// `sww rproxy` reads rproxy's settings alone, and runs the reverse proxy in
|
// `sww rproxy` reads rproxy's settings alone, and runs the reverse proxy in
|
||||||
// front of one app that asks waf about each request. `sww healthcheck` is the
|
// front of one app that asks waf about each request. `sww healthcheck` is the
|
||||||
// container's health check.
|
// container's health check. fx builds the parts of each daemon, starts their
|
||||||
|
// background work and stops it.
|
||||||
package sww
|
package sww
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"errors"
|
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"log/slog"
|
|
||||||
"net"
|
|
||||||
"net/http"
|
|
||||||
"os"
|
"os"
|
||||||
"os/signal"
|
"os/signal"
|
||||||
"syscall"
|
"syscall"
|
||||||
@@ -22,28 +19,9 @@ import (
|
|||||||
|
|
||||||
"github.com/spf13/viper"
|
"github.com/spf13/viper"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/remotelog"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/state"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// shutdownTimeout is how long requests in progress may take to finish
|
|
||||||
// once sww is told to stop, before their connections are closed.
|
|
||||||
// runit and docker wait a little longer before they kill the process.
|
|
||||||
const shutdownTimeout = 5 * time.Second
|
|
||||||
|
|
||||||
// remoteLogStopTimeout is how long, as waf stops, the log lines still
|
|
||||||
// waiting are sent to SWW_WAF_LOG_REMOTE_URL before they are given up. stdout has
|
|
||||||
// carried them.
|
|
||||||
const remoteLogStopTimeout = 2 * time.Second
|
|
||||||
|
|
||||||
// usageStatus is the exit status with no command, or one sww does not
|
// usageStatus is the exit status with no command, or one sww does not
|
||||||
// know.
|
// know.
|
||||||
const usageStatus = 2
|
const usageStatus = 2
|
||||||
@@ -66,6 +44,9 @@ type Params struct {
|
|||||||
Viper *viper.Viper
|
Viper *viper.Viper
|
||||||
// Stdout receives the request log and the process's own messages.
|
// Stdout receives the request log and the process's own messages.
|
||||||
Stdout io.Writer
|
Stdout io.Writer
|
||||||
|
// BesideLookupTimeLimit is the waf.Params.BesideLookupTimeLimit RunWAF
|
||||||
|
// gives waf. Main leaves it zero, waf's own limit; tests lengthen it.
|
||||||
|
BesideLookupTimeLimit time.Duration
|
||||||
}
|
}
|
||||||
|
|
||||||
// Main runs the command os.Args names, and returns the process's exit status.
|
// Main runs the command os.Args names, and returns the process's exit status.
|
||||||
@@ -106,372 +87,3 @@ func printUsage(w io.Writer) int {
|
|||||||
|
|
||||||
return usageStatus
|
return usageStatus
|
||||||
}
|
}
|
||||||
|
|
||||||
// RunWAF is `sww waf`: it reads the settings, the rule files,
|
|
||||||
// the lookup database and the state and cache files, then serves requests until
|
|
||||||
// ctx is done. It returns the process's exit status, 1 when waf cannot start.
|
|
||||||
func RunWAF(ctx context.Context, params Params) int {
|
|
||||||
// Until the settings are read, the one message is an invalid setting's
|
|
||||||
// error, which every SWW_WAF_LOG_LEVEL lets through.
|
|
||||||
processLog := requestlog.NewProcessLogger(params.Stdout,
|
|
||||||
config.InstanceName(params.Viper), slog.LevelError)
|
|
||||||
|
|
||||||
cfg, err := config.WAFFromEnvironment(params.Viper)
|
|
||||||
if err != nil {
|
|
||||||
return cannotStart(processLog, "invalid setting", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// While SWW_WAF_LOG_REMOTE_URL is set, every line on stdout from here on
|
|
||||||
// is sent there too.
|
|
||||||
stdout := params.Stdout
|
|
||||||
|
|
||||||
var remote *remotelog.Sender
|
|
||||||
|
|
||||||
if cfg.LogRemoteURL != nil {
|
|
||||||
remote = newRemoteLogSender(cfg)
|
|
||||||
stdout = io.MultiWriter(params.Stdout, remote)
|
|
||||||
}
|
|
||||||
|
|
||||||
processLog = requestlog.NewProcessLogger(
|
|
||||||
stdout,
|
|
||||||
cfg.InstanceName,
|
|
||||||
cfg.LogLevel,
|
|
||||||
)
|
|
||||||
|
|
||||||
if remote != nil {
|
|
||||||
stopSending := startSending(ctx, remote, processLog)
|
|
||||||
defer stopSending()
|
|
||||||
}
|
|
||||||
|
|
||||||
return runWAFWithSettings(
|
|
||||||
ctx,
|
|
||||||
params.Version,
|
|
||||||
cfg,
|
|
||||||
stdout,
|
|
||||||
remote,
|
|
||||||
processLog,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
// runWAFWithSettings is the rest of RunWAF once the settings are read and
|
|
||||||
// the process log goes where they say. remote is nil unless
|
|
||||||
// SWW_WAF_LOG_REMOTE_URL is set.
|
|
||||||
func runWAFWithSettings(
|
|
||||||
ctx context.Context,
|
|
||||||
version string,
|
|
||||||
cfg *config.WAF,
|
|
||||||
stdout io.Writer,
|
|
||||||
remote *remotelog.Sender,
|
|
||||||
processLog *slog.Logger,
|
|
||||||
) int {
|
|
||||||
// The state files and the alerts give times in UTC.
|
|
||||||
now := func() time.Time { return time.Now().UTC() }
|
|
||||||
|
|
||||||
alertQueue := newAlertQueue(cfg, now, processLog)
|
|
||||||
|
|
||||||
ruleFiles, err := rules.Load(rules.Params{
|
|
||||||
Dir: cfg.RuleFilesDir,
|
|
||||||
Enabled: cfg.RuleFilesEnabled,
|
|
||||||
ProcessLog: processLog,
|
|
||||||
Alerts: alertQueue,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
return cannotStart(processLog, "cannot use the rule files", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
crs, err := waf.NewCoreRuleSet(cfg)
|
|
||||||
if err != nil {
|
|
||||||
return cannotStart(processLog, "cannot use the Core Rule Set", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
server, err := newServer(
|
|
||||||
cfg,
|
|
||||||
stdout,
|
|
||||||
processLog,
|
|
||||||
now,
|
|
||||||
ruleFiles,
|
|
||||||
alertQueue,
|
|
||||||
crs,
|
|
||||||
)
|
|
||||||
if err != nil {
|
|
||||||
return cannotStart(processLog, "cannot use the lookup database", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if remote != nil {
|
|
||||||
server.Metrics.AddRemoteLog(remote)
|
|
||||||
}
|
|
||||||
|
|
||||||
files, err := loadStateFiles(cfg, server, alertQueue, now, processLog)
|
|
||||||
if err != nil {
|
|
||||||
return cannotStart(
|
|
||||||
processLog,
|
|
||||||
"cannot use the state and cache files",
|
|
||||||
err,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
listener, err := (&net.ListenConfig{}).Listen(ctx, "tcp", cfg.ListenAddr)
|
|
||||||
if err != nil {
|
|
||||||
return cannotStart(
|
|
||||||
processLog,
|
|
||||||
"cannot listen on SWW_WAF_LISTEN_ADDR",
|
|
||||||
err,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
processLog.Info("starting",
|
|
||||||
"version", version,
|
|
||||||
"address", listener.Addr().String(),
|
|
||||||
"settings", cfg)
|
|
||||||
|
|
||||||
return serve(
|
|
||||||
ctx,
|
|
||||||
server,
|
|
||||||
listener,
|
|
||||||
files,
|
|
||||||
ruleFiles,
|
|
||||||
alertQueue,
|
|
||||||
processLog,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
// cannotStart logs why waf cannot start, and returns its exit status.
|
|
||||||
func cannotStart(processLog *slog.Logger, message string, err error) int {
|
|
||||||
processLog.Error(message, "error", err.Error())
|
|
||||||
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
|
|
||||||
// newServer returns the server waf runs, with the metrics of the alerts,
|
|
||||||
// after reading the lookup database while SWW_WAF_LOOKUP_SOURCE is file. A lookup
|
|
||||||
// database that cannot be read is an error.
|
|
||||||
func newServer(
|
|
||||||
cfg *config.WAF, stdout io.Writer, processLog *slog.Logger,
|
|
||||||
now func() time.Time, ruleFiles *rules.Files, alertQueue *alerts.Queue,
|
|
||||||
crs *coreruleset.CoreRuleSet,
|
|
||||||
) (*waf.Server, error) {
|
|
||||||
var lookupFile *lookup.File
|
|
||||||
|
|
||||||
if cfg.LookupSource == "file" {
|
|
||||||
var err error
|
|
||||||
|
|
||||||
lookupFile, err = lookup.OpenFile(lookup.FileParams{
|
|
||||||
Path: cfg.LookupDBPath,
|
|
||||||
Now: now,
|
|
||||||
ProcessLog: processLog,
|
|
||||||
Alerts: alertQueue,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
server := waf.New(waf.Params{
|
|
||||||
Config: cfg,
|
|
||||||
RequestLog: stdout,
|
|
||||||
ProcessLog: processLog,
|
|
||||||
GeoJSURL: lookup.URL,
|
|
||||||
AbuseIPDBURL: reputation.AbuseIPDBURL,
|
|
||||||
LookupFile: lookupFile,
|
|
||||||
CoreRuleSet: crs,
|
|
||||||
Now: now,
|
|
||||||
Rules: ruleFiles,
|
|
||||||
Alerts: alertQueue,
|
|
||||||
})
|
|
||||||
server.Metrics.AddAlerts(alertQueue)
|
|
||||||
|
|
||||||
if lookupFile != nil {
|
|
||||||
server.Metrics.AddLookupFile(
|
|
||||||
lookupFile.LastRead,
|
|
||||||
lookupFile.ReadFailures,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
return server, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// loadStateFiles reads the state and cache files into the parts of server and
|
|
||||||
// into alertQueue, as state.Load does.
|
|
||||||
func loadStateFiles(
|
|
||||||
cfg *config.WAF, server *waf.Server, alertQueue *alerts.Queue,
|
|
||||||
now func() time.Time, processLog *slog.Logger,
|
|
||||||
) (*state.Files, error) {
|
|
||||||
return state.Load(state.Params{
|
|
||||||
Dir: cfg.StateDir,
|
|
||||||
CacheDir: cfg.CacheDir,
|
|
||||||
BansWriteDelay: cfg.BansWriteDelay,
|
|
||||||
WriteInterval: cfg.StateWriteInterval,
|
|
||||||
Ledger: server.Ledger,
|
|
||||||
Limiter: server.Limiter,
|
|
||||||
GeoJS: server.GeoJS,
|
|
||||||
Lists: server.Lists,
|
|
||||||
DNSBL: server.DNSBL,
|
|
||||||
AbuseIPDB: server.AbuseIPDB,
|
|
||||||
RouteWatch: server.RouteWatch,
|
|
||||||
Alerts: alertQueue,
|
|
||||||
Anomalies: server.Anomalies,
|
|
||||||
Now: now,
|
|
||||||
ProcessLog: processLog,
|
|
||||||
Metrics: server.Metrics,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// newAlertQueue returns the queue of the alerts to the webhook, Slack and
|
|
||||||
// ntfy, with the settings for them.
|
|
||||||
func newAlertQueue(
|
|
||||||
cfg *config.WAF, now func() time.Time, processLog *slog.Logger,
|
|
||||||
) *alerts.Queue {
|
|
||||||
return alerts.New(alerts.Params{
|
|
||||||
WebhookURL: cfg.AlertWebhookURL,
|
|
||||||
WebhookHeaders: cfg.AlertWebhookHeaders,
|
|
||||||
SlackURL: cfg.AlertSlackWebhookURL,
|
|
||||||
NtfyURL: cfg.AlertNtfyURL,
|
|
||||||
NtfyToken: cfg.AlertNtfyToken,
|
|
||||||
Events: cfg.AlertEvents,
|
|
||||||
Cooldown: cfg.AlertCooldown,
|
|
||||||
MaxPerHour: cfg.AlertMaxPerHour,
|
|
||||||
Instance: cfg.InstanceName,
|
|
||||||
Now: now,
|
|
||||||
ProcessLog: processLog,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// newRemoteLogSender returns a sender of the log lines to
|
|
||||||
// SWW_WAF_LOG_REMOTE_URL, with the settings for it.
|
|
||||||
func newRemoteLogSender(cfg *config.WAF) *remotelog.Sender {
|
|
||||||
return remotelog.New(remotelog.Params{
|
|
||||||
URL: cfg.LogRemoteURL,
|
|
||||||
RootCAs: cfg.LogRemoteTLSCAs,
|
|
||||||
BufferLines: cfg.LogRemoteBufferLines,
|
|
||||||
Facility: cfg.LogRemoteFacility,
|
|
||||||
AppName: cfg.LogRemoteAppName,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// startSending runs remote until the function it returns is called, which
|
|
||||||
// then waits at most remoteLogStopTimeout for the lines still waiting to
|
|
||||||
// be sent. Sending goes on after ctx is done, so that the lines written
|
|
||||||
// while sww stops are sent too.
|
|
||||||
func startSending(
|
|
||||||
ctx context.Context, remote *remotelog.Sender, processLog *slog.Logger,
|
|
||||||
) func() {
|
|
||||||
sending, stop := context.WithCancel(context.WithoutCancel(ctx))
|
|
||||||
sent := make(chan struct{})
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
remote.Run(sending, processLog)
|
|
||||||
close(sent)
|
|
||||||
}()
|
|
||||||
|
|
||||||
return func() {
|
|
||||||
stop()
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-sent:
|
|
||||||
case <-time.After(remoteLogStopTimeout):
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// serve serves requests on listener, writes the state and cache files as they
|
|
||||||
// are due, takes in an admin's edits of them, reads the rule files again as
|
|
||||||
// they change, and the lookup database when it is replaced, fetches the
|
|
||||||
// lists the settings name by URL as they are due, and sends the alerts,
|
|
||||||
// until ctx is done. Then it gives the requests in progress
|
|
||||||
// shutdownTimeout to finish, and writes every state and cache file,
|
|
||||||
// alerts.json with the alerts still waiting.
|
|
||||||
func serve(
|
|
||||||
ctx context.Context, server *waf.Server, listener net.Listener,
|
|
||||||
files *state.Files, ruleFiles *rules.Files, alertQueue *alerts.Queue,
|
|
||||||
processLog *slog.Logger,
|
|
||||||
) int {
|
|
||||||
served := make(chan error, 1)
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
served <- server.Serve(listener)
|
|
||||||
}()
|
|
||||||
|
|
||||||
writing, stopWriting := context.WithCancel(ctx)
|
|
||||||
defer stopWriting()
|
|
||||||
|
|
||||||
written := inBackground(func() { files.Run(writing) })
|
|
||||||
watched := inBackground(func() { files.Watch(writing) })
|
|
||||||
rulesWatched := inBackground(func() { ruleFiles.Watch(writing) })
|
|
||||||
lookupFileWatched := inBackground(func() {
|
|
||||||
if server.LookupFile != nil {
|
|
||||||
server.LookupFile.Watch(writing)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
listsFetched := inBackground(func() { server.Lists.Run(writing) })
|
|
||||||
alertsSent := inBackground(func() { alertQueue.Run(writing) })
|
|
||||||
|
|
||||||
select {
|
|
||||||
case err := <-served:
|
|
||||||
processLog.Error("serving failed", "error", err.Error())
|
|
||||||
|
|
||||||
return 1
|
|
||||||
case <-ctx.Done():
|
|
||||||
}
|
|
||||||
|
|
||||||
processLog.Info("stopping")
|
|
||||||
|
|
||||||
shutdownCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx),
|
|
||||||
shutdownTimeout)
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
err := server.Shutdown(shutdownCtx)
|
|
||||||
if err != nil {
|
|
||||||
processLog.Warn("requests still in progress were cut off",
|
|
||||||
"error", err.Error())
|
|
||||||
|
|
||||||
_ = server.Close()
|
|
||||||
}
|
|
||||||
|
|
||||||
err = <-served
|
|
||||||
if !errors.Is(err, http.ErrServerClosed) {
|
|
||||||
processLog.Error("serving failed", "error", err.Error())
|
|
||||||
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
|
|
||||||
// Run and Watch have ended, so nothing else reads or writes the
|
|
||||||
// files, and no alert is being sent, so that alerts.json keeps every
|
|
||||||
// alert not yet sent. Every request has ended too, but for one cut off
|
|
||||||
// because Shutdown timed out, which adds to its client's history only as
|
|
||||||
// it ends. That can be after this write, and then that request is missing
|
|
||||||
// from clients.json.
|
|
||||||
<-written
|
|
||||||
<-watched
|
|
||||||
<-rulesWatched
|
|
||||||
<-lookupFileWatched
|
|
||||||
<-listsFetched
|
|
||||||
<-alertsSent
|
|
||||||
|
|
||||||
err = files.WriteAll()
|
|
||||||
if err != nil {
|
|
||||||
processLog.Error(
|
|
||||||
"writing the state and cache files failed",
|
|
||||||
"error",
|
|
||||||
err.Error(),
|
|
||||||
)
|
|
||||||
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
|
|
||||||
processLog.Info("stopped")
|
|
||||||
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
||||||
// inBackground runs task on a goroutine of its own, and returns a channel
|
|
||||||
// that is closed once task has returned.
|
|
||||||
func inBackground(task func()) <-chan struct{} {
|
|
||||||
done := make(chan struct{})
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
task()
|
|
||||||
close(done)
|
|
||||||
}()
|
|
||||||
|
|
||||||
return done
|
|
||||||
}
|
|
||||||
+20
-15
@@ -52,7 +52,7 @@ const (
|
|||||||
adminSecret = "fedcba9876543210fedcba9876543210"
|
adminSecret = "fedcba9876543210fedcba9876543210"
|
||||||
// instance is the SWW_WAF_INSTANCE_NAME the tests set where they look at
|
// instance is the SWW_WAF_INSTANCE_NAME the tests set where they look at
|
||||||
// it.
|
// it.
|
||||||
instance = "fsn1app1/blog"
|
instance = "host1/blog"
|
||||||
// greeting is what the tests' app answers.
|
// greeting is what the tests' app answers.
|
||||||
greeting = "hello from the app"
|
greeting = "hello from the app"
|
||||||
// placed is the client the tests' lookup databases place.
|
// placed is the client the tests' lookup databases place.
|
||||||
@@ -173,6 +173,9 @@ func run(ctx context.Context, env map[string]string, out *output) int {
|
|||||||
Version: testVersion,
|
Version: testVersion,
|
||||||
Viper: v,
|
Viper: v,
|
||||||
Stdout: out,
|
Stdout: out,
|
||||||
|
// The race detector and a busy host can slow the Core Rule Set's
|
||||||
|
// inspections past waf's own limit.
|
||||||
|
BesideLookupTimeLimit: time.Hour,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -386,8 +389,8 @@ func TestEveryLogLineAndMetricCarriesTheInstanceName(t *testing.T) {
|
|||||||
// the comments.
|
// the comments.
|
||||||
for line := range strings.Lines(metrics) {
|
for line := range strings.Lines(metrics) {
|
||||||
if !strings.HasPrefix(line, "#") &&
|
if !strings.HasPrefix(line, "#") &&
|
||||||
!strings.Contains(line, `instance="fsn1app1/blog"`) {
|
!strings.Contains(line, `instance="host1/blog"`) {
|
||||||
t.Errorf("series %q, without instance=\"fsn1app1/blog\"", line)
|
t.Errorf("series %q, without instance=\"host1/blog\"", line)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -657,7 +660,9 @@ func TestCoreRuleSetExclusionsFileThatCannotBeUsedStopsTheStartWhileTheCoreRuleS
|
|||||||
|
|
||||||
status := run(t.Context(), map[string]string{
|
status := run(t.Context(), map[string]string{
|
||||||
listenAddr: localhost + ":0", stateDir: t.TempDir(), ruleFilesDir: t.TempDir(),
|
listenAddr: localhost + ":0", stateDir: t.TempDir(), ruleFilesDir: t.TempDir(),
|
||||||
"SWW_WAF_CORE_RULE_SET_MODE": config.CoreRuleSetModeOff,
|
"SWW_WAF_CORE_RULE_SET_MODE": string(
|
||||||
|
config.CoreRuleSetModeOff,
|
||||||
|
),
|
||||||
"SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE": path,
|
"SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE": path,
|
||||||
}, out)
|
}, out)
|
||||||
if status != 1 {
|
if status != 1 {
|
||||||
@@ -734,12 +739,12 @@ func TestLookupDatabaseReplacedWhileRunningTakesEffect(t *testing.T) {
|
|||||||
t,
|
t,
|
||||||
url+"_sww/metrics",
|
url+"_sww/metrics",
|
||||||
token,
|
token,
|
||||||
`sww_lookup_database_read_failures_total{instance="fsn1app1/blog"} 1`,
|
`sww_lookup_database_read_failures_total{instance="host1/blog"} 1`,
|
||||||
)
|
)
|
||||||
lastRead = seriesValue(
|
lastRead = seriesValue(
|
||||||
t,
|
t,
|
||||||
metrics,
|
metrics,
|
||||||
`sww_lookup_database_last_read_timestamp_seconds{instance="fsn1app1/blog"}`,
|
`sww_lookup_database_last_read_timestamp_seconds{instance="host1/blog"}`,
|
||||||
)
|
)
|
||||||
|
|
||||||
wantStatus(t, url, placed, http.StatusOK)
|
wantStatus(t, url, placed, http.StatusOK)
|
||||||
@@ -801,7 +806,7 @@ func TestBlocklistTriesAndCopiesKeptInListsJSONAcrossRestarts(t *testing.T) {
|
|||||||
// too.
|
// too.
|
||||||
rateLimitExemptNets: placed + "," + localhost,
|
rateLimitExemptNets: placed + "," + localhost,
|
||||||
}
|
}
|
||||||
failures := `sww_reputation_failures_total{instance="fsn1app1/blog",` +
|
failures := `sww_reputation_failures_total{instance="host1/blog",` +
|
||||||
`source="` + torURL + `"} `
|
`source="` + torURL + `"} `
|
||||||
|
|
||||||
// tor.txt cannot be fetched at first, which is counted.
|
// tor.txt cannot be fetched at first, which is counted.
|
||||||
@@ -984,8 +989,8 @@ func TestStalledRemoteLogEndpointHoldsUpNoRequest(t *testing.T) {
|
|||||||
// last.
|
// last.
|
||||||
metrics := metricsText(t, url+"_sww/metrics", token)
|
metrics := metricsText(t, url+"_sww/metrics", token)
|
||||||
for _, series := range []string{
|
for _, series := range []string{
|
||||||
`sww_remote_log_lines_sent_total{instance="fsn1app1/blog"} 0`,
|
`sww_remote_log_lines_sent_total{instance="host1/blog"} 0`,
|
||||||
`sww_remote_log_lines_waiting{instance="fsn1app1/blog"} 1`,
|
`sww_remote_log_lines_waiting{instance="host1/blog"} 1`,
|
||||||
} {
|
} {
|
||||||
if !strings.Contains(metrics, "\n"+series+"\n") {
|
if !strings.Contains(metrics, "\n"+series+"\n") {
|
||||||
t.Errorf("no %q in the metrics:\n%s", series, metrics)
|
t.Errorf("no %q in the metrics:\n%s", series, metrics)
|
||||||
@@ -993,7 +998,7 @@ func TestStalledRemoteLogEndpointHoldsUpNoRequest(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
const dropped = "\nsww_remote_log_lines_dropped_total" +
|
const dropped = "\nsww_remote_log_lines_dropped_total" +
|
||||||
`{instance="fsn1app1/blog"} `
|
`{instance="host1/blog"} `
|
||||||
if strings.Contains(metrics, dropped+"0\n") ||
|
if strings.Contains(metrics, dropped+"0\n") ||
|
||||||
!strings.Contains(metrics, dropped) {
|
!strings.Contains(metrics, dropped) {
|
||||||
t.Errorf("no line dropped in the metrics:\n%s", metrics)
|
t.Errorf("no line dropped in the metrics:\n%s", metrics)
|
||||||
@@ -1089,7 +1094,7 @@ func TestBanIsAlertedAndAnAlertNotSentIsKeptAcrossARestart(t *testing.T) {
|
|||||||
runUntilStopped(t, env, func(url string) {
|
runUntilStopped(t, env, func(url string) {
|
||||||
webhook.waitFor(t, "permanent_ban", true)
|
webhook.waitFor(t, "permanent_ban", true)
|
||||||
|
|
||||||
const ofWebhook = `{destination="webhook",instance="fsn1app1/blog"}`
|
const ofWebhook = `{destination="webhook",instance="host1/blog"}`
|
||||||
|
|
||||||
metrics := metricsWith(t, url+"_sww/metrics", token,
|
metrics := metricsWith(t, url+"_sww/metrics", token,
|
||||||
"\nsww_alerts_sent_total"+ofWebhook+" 1\n")
|
"\nsww_alerts_sent_total"+ofWebhook+" 1\n")
|
||||||
@@ -1176,13 +1181,13 @@ func TestBanIsAlertedToSlackAndNtfy(t *testing.T) {
|
|||||||
err := json.Unmarshal([]byte(slackPost.body), &message)
|
err := json.Unmarshal([]byte(slackPost.body), &message)
|
||||||
|
|
||||||
if err != nil ||
|
if err != nil ||
|
||||||
!strings.HasPrefix(message.Text, "*fsn1app1/blog: ban*\n") ||
|
!strings.HasPrefix(message.Text, "*host1/blog: ban*\n") ||
|
||||||
!strings.Contains(message.Text, "\nclient: "+client+"\n") {
|
!strings.Contains(message.Text, "\nclient: "+client+"\n") {
|
||||||
t.Errorf("Slack was sent %s", slackPost.body)
|
t.Errorf("Slack was sent %s", slackPost.body)
|
||||||
}
|
}
|
||||||
|
|
||||||
ntfyPost := ntfy.firstPost(t)
|
ntfyPost := ntfy.firstPost(t)
|
||||||
if ntfyPost.header.Get("Title") != "fsn1app1/blog: ban" ||
|
if ntfyPost.header.Get("Title") != "host1/blog: ban" ||
|
||||||
ntfyPost.header.Get("Authorization") != "Bearer "+ntfyToken ||
|
ntfyPost.header.Get("Authorization") != "Bearer "+ntfyToken ||
|
||||||
!strings.Contains(ntfyPost.body, "\nclient: "+client+"\n") {
|
!strings.Contains(ntfyPost.body, "\nclient: "+client+"\n") {
|
||||||
t.Errorf("ntfy was sent %s, with the headers %v", ntfyPost.body,
|
t.Errorf("ntfy was sent %s, with the headers %v", ntfyPost.body,
|
||||||
@@ -1194,9 +1199,9 @@ func TestBanIsAlertedToSlackAndNtfy(t *testing.T) {
|
|||||||
// test process cannot fail the test.
|
// test process cannot fail the test.
|
||||||
sent := []string{
|
sent := []string{
|
||||||
"\nsww_alerts_sent_total{destination=\"slack\"," +
|
"\nsww_alerts_sent_total{destination=\"slack\"," +
|
||||||
"instance=\"fsn1app1/blog\"} 1\n",
|
"instance=\"host1/blog\"} 1\n",
|
||||||
"\nsww_alerts_sent_total{destination=\"ntfy\"," +
|
"\nsww_alerts_sent_total{destination=\"ntfy\"," +
|
||||||
"instance=\"fsn1app1/blog\"} 1\n",
|
"instance=\"host1/blog\"} 1\n",
|
||||||
}
|
}
|
||||||
|
|
||||||
metrics := metricsWith(t, url+"_sww/metrics", token, sent...)
|
metrics := metricsWith(t, url+"_sww/metrics", token, sent...)
|
||||||
|
|||||||
@@ -0,0 +1,560 @@
|
|||||||
|
package sww
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"log/slog"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"go.uber.org/fx"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/processlog"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/remotelog"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/routewatch"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/state"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||||
|
)
|
||||||
|
|
||||||
|
// remoteLogStopTimeout is how long, as waf stops, the log lines still
|
||||||
|
// waiting are sent to SWW_WAF_LOG_REMOTE_URL before they are given up. stdout has
|
||||||
|
// carried them.
|
||||||
|
const remoteLogStopTimeout = 2 * time.Second
|
||||||
|
|
||||||
|
// RunWAF is `sww waf`: it reads the settings, the rule files,
|
||||||
|
// the lookup database and the state and cache files, then serves requests until
|
||||||
|
// ctx is done. Then it gives the requests in progress shutdownTimeout to
|
||||||
|
// finish, ends its background work and writes every state and cache file,
|
||||||
|
// alerts.json with the alerts still waiting. It returns the process's exit
|
||||||
|
// status, 1 when waf cannot start or stop in order.
|
||||||
|
func RunWAF(ctx context.Context, params Params) int {
|
||||||
|
// Until the settings are read, the one message is an invalid setting's
|
||||||
|
// error, which every SWW_WAF_LOG_LEVEL lets through.
|
||||||
|
processLog := processlog.New(processlog.Params{
|
||||||
|
Output: params.Stdout,
|
||||||
|
InstanceName: config.InstanceName(params.Viper),
|
||||||
|
Level: slog.LevelError,
|
||||||
|
})
|
||||||
|
|
||||||
|
cfg, err := config.WAFFromEnvironment(params.Viper)
|
||||||
|
if err != nil {
|
||||||
|
return cannotStart(processLog, startError{"invalid setting", err})
|
||||||
|
}
|
||||||
|
|
||||||
|
// While SWW_WAF_LOG_REMOTE_URL is set, every line on stdout from here on
|
||||||
|
// is sent there too. The sender runs before any part is built, so that it
|
||||||
|
// sends why waf cannot start.
|
||||||
|
stdout := params.Stdout
|
||||||
|
|
||||||
|
var remote *remotelog.Sender
|
||||||
|
|
||||||
|
if cfg.LogRemoteURL != nil {
|
||||||
|
remote = newRemoteLogSender(cfg)
|
||||||
|
stdout = io.MultiWriter(params.Stdout, remote)
|
||||||
|
}
|
||||||
|
|
||||||
|
processLog = processlog.New(processlog.Params{
|
||||||
|
Output: stdout,
|
||||||
|
InstanceName: cfg.InstanceName,
|
||||||
|
Level: cfg.LogLevel,
|
||||||
|
})
|
||||||
|
|
||||||
|
if remote != nil {
|
||||||
|
stopSending := startSending(ctx, remote, processLog)
|
||||||
|
defer stopSending()
|
||||||
|
}
|
||||||
|
|
||||||
|
var (
|
||||||
|
files *state.Files
|
||||||
|
listener net.Listener
|
||||||
|
)
|
||||||
|
|
||||||
|
failed := make(servingFailed, 1)
|
||||||
|
|
||||||
|
app := fx.New(
|
||||||
|
fx.Supply(
|
||||||
|
params,
|
||||||
|
cfg,
|
||||||
|
processLog,
|
||||||
|
requestLog{stdout},
|
||||||
|
remote,
|
||||||
|
failed,
|
||||||
|
),
|
||||||
|
fx.Provide(
|
||||||
|
newAlertQueue, newRuleFiles, newCoreRuleSet, newLookupFile,
|
||||||
|
newLimiter, newLedger, newAnomalyCounters,
|
||||||
|
newLists, newDNSBL, newAbuseIPDB, newRouteWatch,
|
||||||
|
newMetrics, newGeoJS, newWAFServer,
|
||||||
|
func(server *waf.Server) *http.Server { return server.Server },
|
||||||
|
newWAFListener, newStateFiles,
|
||||||
|
),
|
||||||
|
// fx stops hooks in the reverse of the order they were added in. The
|
||||||
|
// state files are read before serving is set up, so that serving
|
||||||
|
// stops before their loops do.
|
||||||
|
fx.Populate(&files),
|
||||||
|
fx.Invoke(serve),
|
||||||
|
fx.Populate(&listener),
|
||||||
|
fx.WithLogger(fxEventLogger(processLog)),
|
||||||
|
)
|
||||||
|
|
||||||
|
err = app.Err()
|
||||||
|
if err != nil {
|
||||||
|
return cannotStart(processLog, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
processLog.Info("starting",
|
||||||
|
"version", params.Version,
|
||||||
|
"address", listener.Addr().String(),
|
||||||
|
"settings", cfg)
|
||||||
|
|
||||||
|
// The last write follows the stop of every hook, and so the end of every
|
||||||
|
// loop that changes the state or raises an alert, and of every ask of
|
||||||
|
// GeoJS, the DNSBL zones, AbuseIPDB and routewatch. A hook of the state
|
||||||
|
// files would stop before the loops of the parts built ahead of them.
|
||||||
|
return run(ctx, app, failed, processLog, func() error {
|
||||||
|
err := files.WriteAll()
|
||||||
|
if err != nil {
|
||||||
|
processLog.Error("writing the state and cache files failed",
|
||||||
|
"error", err.Error())
|
||||||
|
}
|
||||||
|
|
||||||
|
return err
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// newAlertQueue returns the queue of the alerts to the webhook, Slack and
|
||||||
|
// ntfy, with the settings for them. Until waf stops, it sends them.
|
||||||
|
func newAlertQueue(
|
||||||
|
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
|
||||||
|
) *alerts.Queue {
|
||||||
|
alertQueue := alerts.New(alerts.Params{
|
||||||
|
WebhookURL: cfg.AlertWebhookURL,
|
||||||
|
WebhookHeaders: cfg.AlertWebhookHeaders,
|
||||||
|
SlackURL: cfg.AlertSlackWebhookURL,
|
||||||
|
NtfyURL: cfg.AlertNtfyURL,
|
||||||
|
NtfyToken: cfg.AlertNtfyToken,
|
||||||
|
Events: cfg.AlertEvents,
|
||||||
|
Cooldown: cfg.AlertCooldown,
|
||||||
|
MaxPerHour: cfg.AlertMaxPerHour,
|
||||||
|
Instance: cfg.InstanceName,
|
||||||
|
Now: utcNow,
|
||||||
|
ProcessLog: processLog,
|
||||||
|
})
|
||||||
|
|
||||||
|
lc.Append(background(alertQueue.Run))
|
||||||
|
|
||||||
|
return alertQueue
|
||||||
|
}
|
||||||
|
|
||||||
|
// newRuleFiles reads the rule files, and reads them again as they change
|
||||||
|
// until waf stops.
|
||||||
|
func newRuleFiles(
|
||||||
|
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
|
||||||
|
alertQueue *alerts.Queue,
|
||||||
|
) (*rules.Files, error) {
|
||||||
|
ruleFiles, err := rules.Load(rules.Params{
|
||||||
|
Dir: cfg.RuleFilesDir,
|
||||||
|
Enabled: cfg.RuleFilesEnabled,
|
||||||
|
ProcessLog: processLog,
|
||||||
|
Alerts: alertQueue,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return nil, startError{"cannot use the rule files", err}
|
||||||
|
}
|
||||||
|
|
||||||
|
lc.Append(background(ruleFiles.Watch))
|
||||||
|
|
||||||
|
return ruleFiles, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// newCoreRuleSet returns the Core Rule Set at SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL,
|
||||||
|
// without the rules SWW_WAF_CORE_RULE_SET_DISABLED_RULES switches off, reading bodies
|
||||||
|
// up to SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE, with the exclusions in
|
||||||
|
// SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE and the fixes of the apps in SWW_WAF_APPS,
|
||||||
|
// or nil while SWW_WAF_CORE_RULE_SET_MODE is off. An exclusions file that
|
||||||
|
// coreruleset.New refuses is an error in every mode. While the mode is off, a
|
||||||
|
// set file is loaded with the Core Rule Set only to check it.
|
||||||
|
func newCoreRuleSet(cfg *config.WAF) (*coreruleset.CoreRuleSet, error) {
|
||||||
|
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff &&
|
||||||
|
cfg.CoreRuleSetExclusionsFile == "" {
|
||||||
|
return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error
|
||||||
|
}
|
||||||
|
|
||||||
|
crs, err := coreruleset.New(coreruleset.Params{
|
||||||
|
ParanoiaLevel: cfg.CoreRuleSetParanoiaLevel,
|
||||||
|
DisabledRules: cfg.CoreRuleSetDisabledRules,
|
||||||
|
BodyInspectionSize: cfg.CoreRuleSetBodyInspectionSize,
|
||||||
|
ExclusionsFile: cfg.CoreRuleSetExclusionsFile,
|
||||||
|
Apps: cfg.Apps,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return nil, startError{
|
||||||
|
"cannot use the Core Rule Set",
|
||||||
|
fmt.Errorf("SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE: %w", err),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff {
|
||||||
|
return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error
|
||||||
|
}
|
||||||
|
|
||||||
|
return crs, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// newLookupFile reads the lookup database while SWW_WAF_LOOKUP_SOURCE is
|
||||||
|
// file, and reads it again when it is replaced until waf stops. It returns
|
||||||
|
// nil while the source is anything else.
|
||||||
|
func newLookupFile(
|
||||||
|
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
|
||||||
|
alertQueue *alerts.Queue,
|
||||||
|
) (*lookup.File, error) {
|
||||||
|
if cfg.LookupSource != config.LookupSourceFile {
|
||||||
|
return nil, nil //nolint:nilnil // no lookup database, and no error
|
||||||
|
}
|
||||||
|
|
||||||
|
lookupFile, err := lookup.OpenFile(lookup.FileParams{
|
||||||
|
Path: cfg.LookupDBPath,
|
||||||
|
Now: utcNow,
|
||||||
|
ProcessLog: processLog,
|
||||||
|
Alerts: alertQueue,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return nil, startError{"cannot use the lookup database", err}
|
||||||
|
}
|
||||||
|
|
||||||
|
lc.Append(background(lookupFile.Watch))
|
||||||
|
|
||||||
|
return lookupFile, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// newLimiter returns the table of clients, with the rate and byte limits.
|
||||||
|
func newLimiter(cfg *config.WAF) *ratelimit.Limiter {
|
||||||
|
return ratelimit.New(ratelimit.Params{
|
||||||
|
Limits: ratelimit.Limits{
|
||||||
|
PerMinute: cfg.RateLimitPerMinute,
|
||||||
|
PerHour: cfg.RateLimitPerHour,
|
||||||
|
PerDay: cfg.RateLimitPerDay,
|
||||||
|
BytesPerMinute: cfg.BytesLimitPerMinute,
|
||||||
|
BytesPerHour: cfg.BytesLimitPerHour,
|
||||||
|
BytesPerDay: cfg.BytesLimitPerDay,
|
||||||
|
},
|
||||||
|
MaxClients: cfg.MaxTrackedClients,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// newLedger returns the ledger of the bans, with the settings for them.
|
||||||
|
func newLedger(cfg *config.WAF) *bans.Ledger {
|
||||||
|
return bans.New(bans.Params{
|
||||||
|
LimitBanDuration: cfg.LimitBanDuration,
|
||||||
|
LimitBanRepeatWindow: cfg.LimitBanRepeatWindow,
|
||||||
|
LimitBanPermanentAbove: cfg.LimitBanPermanentAbove,
|
||||||
|
AttackBanDuration: cfg.AttackBanDuration,
|
||||||
|
MaxBans: cfg.MaxBans,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// newAnomalyCounters returns the counters of the anomaly thresholds.
|
||||||
|
func newAnomalyCounters(
|
||||||
|
cfg *config.WAF, alertQueue *alerts.Queue,
|
||||||
|
) *anomaly.Counters {
|
||||||
|
return anomaly.New(anomaly.Params{
|
||||||
|
Client: cfg.AnomalyClient,
|
||||||
|
Net: cfg.AnomalyNet,
|
||||||
|
ASN: cfg.AnomalyASN,
|
||||||
|
Total: cfg.AnomalyTotal,
|
||||||
|
WatchedNet: cfg.AnomalyWatchedNet,
|
||||||
|
NetV4Prefix: cfg.AnomalyNetV4Prefix,
|
||||||
|
NetV6Prefix: cfg.AnomalyNetV6Prefix,
|
||||||
|
NamedNetblocks: cfg.AnomalyWatchedNets,
|
||||||
|
Alerts: alertQueue,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// newLists returns the lists the settings name by URL, none fetched yet.
|
||||||
|
// Until waf stops, it fetches them as they are due.
|
||||||
|
func newLists(
|
||||||
|
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
|
||||||
|
alertQueue *alerts.Queue,
|
||||||
|
) *reputation.Lists {
|
||||||
|
lists := reputation.New(reputation.Params{
|
||||||
|
BlocklistURLs: cfg.BlocklistURLs,
|
||||||
|
Refresh: cfg.BlocklistRefresh,
|
||||||
|
ASNLimitPercentURL: cfg.ASNLimitPercentURL,
|
||||||
|
TagLists: cfg.TagLists,
|
||||||
|
Now: utcNow,
|
||||||
|
ProcessLog: processLog,
|
||||||
|
Alerts: alertQueue,
|
||||||
|
})
|
||||||
|
|
||||||
|
lc.Append(background(lists.Run))
|
||||||
|
|
||||||
|
return lists
|
||||||
|
}
|
||||||
|
|
||||||
|
// newDNSBL returns what asks the DNSBL zones, none asked yet. At the stop it
|
||||||
|
// ends the queries under way, as reputation.DNSBL.StopAsking says.
|
||||||
|
func newDNSBL(
|
||||||
|
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
|
||||||
|
alertQueue *alerts.Queue,
|
||||||
|
) *reputation.DNSBL {
|
||||||
|
dnsbl := reputation.NewDNSBL(reputation.DNSBLParams{
|
||||||
|
Zones: cfg.DNSBLZones,
|
||||||
|
Resolver: cfg.DNSBLResolver,
|
||||||
|
CacheTTL: cfg.IPReputationCacheTTL,
|
||||||
|
Timeout: cfg.DNSBLAbuseIPDBTimeout,
|
||||||
|
Now: utcNow,
|
||||||
|
ProcessLog: processLog,
|
||||||
|
Alerts: alertQueue,
|
||||||
|
})
|
||||||
|
|
||||||
|
lc.Append(fx.StopHook(dnsbl.StopAsking))
|
||||||
|
|
||||||
|
return dnsbl
|
||||||
|
}
|
||||||
|
|
||||||
|
// newAbuseIPDB returns what checks clients with AbuseIPDB while
|
||||||
|
// SWW_WAF_ABUSEIPDB_KEY is set, none checked yet. Until waf stops, it
|
||||||
|
// fetches AbuseIPDB's blacklist as it is due, and bans the clients on it in
|
||||||
|
// ledger, as waf.BanAbuseIPDBBlacklist says. At the stop it ends the checks
|
||||||
|
// under way, as reputation.AbuseIPDB.StopChecking says.
|
||||||
|
func newAbuseIPDB(
|
||||||
|
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
|
||||||
|
alertQueue *alerts.Queue, ledger *bans.Ledger,
|
||||||
|
) *reputation.AbuseIPDB {
|
||||||
|
abuseIPDB := reputation.NewAbuseIPDB(reputation.AbuseIPDBParams{
|
||||||
|
URL: reputation.AbuseIPDBURL,
|
||||||
|
Key: cfg.AbuseIPDBKey,
|
||||||
|
MinScore: cfg.AbuseIPDBMinScore,
|
||||||
|
DailyBudget: cfg.AbuseIPDBDailyBudget,
|
||||||
|
CacheTTL: cfg.IPReputationCacheTTL,
|
||||||
|
Timeout: cfg.DNSBLAbuseIPDBTimeout,
|
||||||
|
BlacklistURL: reputation.AbuseIPDBBlacklistURL,
|
||||||
|
Threshold: cfg.AbuseIPDBThreshold,
|
||||||
|
BlacklistRefresh: cfg.AbuseIPDBBlacklistRefresh,
|
||||||
|
Blacklisted: func(listed []reputation.BlacklistEntry) {
|
||||||
|
waf.BanAbuseIPDBBlacklist(
|
||||||
|
cfg, ledger, processLog, utcNow(), listed,
|
||||||
|
)
|
||||||
|
},
|
||||||
|
Now: utcNow,
|
||||||
|
ProcessLog: processLog,
|
||||||
|
Alerts: alertQueue,
|
||||||
|
})
|
||||||
|
|
||||||
|
lc.Append(background(abuseIPDB.RunBlacklist))
|
||||||
|
lc.Append(fx.StopHook(abuseIPDB.StopChecking))
|
||||||
|
|
||||||
|
return abuseIPDB
|
||||||
|
}
|
||||||
|
|
||||||
|
// newRouteWatch returns what asks routewatch about clients, or nil while
|
||||||
|
// SWW_WAF_ROUTEWATCH_URL is unset. At the stop it ends the queries under
|
||||||
|
// way, as routewatch.Client.StopAsking says.
|
||||||
|
func newRouteWatch(
|
||||||
|
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
|
||||||
|
) *routewatch.Client {
|
||||||
|
if cfg.RouteWatchURL == "" {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
routeWatch := routewatch.New(routewatch.Params{
|
||||||
|
URL: cfg.RouteWatchURL,
|
||||||
|
Now: utcNow,
|
||||||
|
ProcessLog: processLog,
|
||||||
|
})
|
||||||
|
|
||||||
|
lc.Append(fx.StopHook(routeWatch.StopAsking))
|
||||||
|
|
||||||
|
return routeWatch
|
||||||
|
}
|
||||||
|
|
||||||
|
// newMetrics returns the metrics, with those read from the other parts as
|
||||||
|
// the metrics are asked for: AbuseIPDB's while SWW_WAF_ABUSEIPDB_KEY is set,
|
||||||
|
// the lookup database's while there is one, and the remote log sender's
|
||||||
|
// while there is one.
|
||||||
|
func newMetrics(
|
||||||
|
cfg *config.WAF, ledger *bans.Ledger, limiter *ratelimit.Limiter,
|
||||||
|
ruleFiles *rules.Files, lists *reputation.Lists,
|
||||||
|
dnsbl *reputation.DNSBL, abuseIPDB *reputation.AbuseIPDB,
|
||||||
|
alertQueue *alerts.Queue, lookupFile *lookup.File,
|
||||||
|
remote *remotelog.Sender,
|
||||||
|
) *metrics.Metrics {
|
||||||
|
m := metrics.New(cfg.MetricsBusiestCountriesAndASNs, cfg.InstanceName)
|
||||||
|
m.AddBansAndClients(ledger, limiter, utcNow)
|
||||||
|
m.AddRules(ruleFiles)
|
||||||
|
m.AddReputation(lists, dnsbl)
|
||||||
|
|
||||||
|
if cfg.AbuseIPDBKey != "" {
|
||||||
|
m.AddAbuseIPDB(abuseIPDB)
|
||||||
|
}
|
||||||
|
|
||||||
|
m.AddAlerts(alertQueue)
|
||||||
|
|
||||||
|
if lookupFile != nil {
|
||||||
|
m.AddLookupFile(lookupFile.LastRead, lookupFile.ReadFailures)
|
||||||
|
}
|
||||||
|
|
||||||
|
if remote != nil {
|
||||||
|
m.AddRemoteLog(remote)
|
||||||
|
}
|
||||||
|
|
||||||
|
return m
|
||||||
|
}
|
||||||
|
|
||||||
|
// newGeoJS returns what looks clients up at GeoJS, none looked up yet. Each
|
||||||
|
// answer goes to the limiter and the ledger, as waf.AddLookup says, and each
|
||||||
|
// failure is handled as waf.GeoJSFailed says. At the stop it ends the request
|
||||||
|
// under way, as geojs.GeoJS.StopAsking says.
|
||||||
|
func newGeoJS(
|
||||||
|
lc fx.Lifecycle,
|
||||||
|
cfg *config.WAF,
|
||||||
|
m *metrics.Metrics,
|
||||||
|
limiter *ratelimit.Limiter,
|
||||||
|
ledger *bans.Ledger,
|
||||||
|
alertQueue *alerts.Queue,
|
||||||
|
processLog *slog.Logger,
|
||||||
|
) *geojs.GeoJS {
|
||||||
|
geoJS := geojs.New(geojs.Params{
|
||||||
|
URL: geojs.URL,
|
||||||
|
Answered: func(answer geojs.Answer) {
|
||||||
|
waf.AddLookup(cfg, limiter, ledger, answer)
|
||||||
|
},
|
||||||
|
Now: utcNow,
|
||||||
|
Asked: m.GeoJSRequests.Inc,
|
||||||
|
Failed: func(err error, askingAgainIn time.Duration) {
|
||||||
|
waf.GeoJSFailed(processLog, alertQueue, m, err, askingAgainIn)
|
||||||
|
},
|
||||||
|
WentWithoutAnswer: m.RequestsWithoutGeoJSAnswer.Inc,
|
||||||
|
})
|
||||||
|
|
||||||
|
lc.Append(fx.StopHook(geoJS.StopAsking))
|
||||||
|
|
||||||
|
return geoJS
|
||||||
|
}
|
||||||
|
|
||||||
|
// newWAFServer returns the server waf runs, with its parts and the time
|
||||||
|
// limit params gives.
|
||||||
|
func newWAFServer(
|
||||||
|
params Params, cfg *config.WAF, out requestLog, processLog *slog.Logger,
|
||||||
|
m *metrics.Metrics, limiter *ratelimit.Limiter, ledger *bans.Ledger,
|
||||||
|
anomalies *anomaly.Counters, lists *reputation.Lists,
|
||||||
|
dnsbl *reputation.DNSBL, abuseIPDB *reputation.AbuseIPDB,
|
||||||
|
routeWatch *routewatch.Client, geoJS *geojs.GeoJS,
|
||||||
|
lookupFile *lookup.File, crs *coreruleset.CoreRuleSet,
|
||||||
|
ruleFiles *rules.Files, alertQueue *alerts.Queue,
|
||||||
|
) *waf.Server {
|
||||||
|
return waf.New(waf.Params{
|
||||||
|
Config: cfg,
|
||||||
|
RequestLog: out,
|
||||||
|
ProcessLog: processLog,
|
||||||
|
Metrics: m,
|
||||||
|
Limiter: limiter,
|
||||||
|
Ledger: ledger,
|
||||||
|
Anomalies: anomalies,
|
||||||
|
Lists: lists,
|
||||||
|
DNSBL: dnsbl,
|
||||||
|
AbuseIPDB: abuseIPDB,
|
||||||
|
RouteWatch: routeWatch,
|
||||||
|
GeoJS: geoJS,
|
||||||
|
LookupFile: lookupFile,
|
||||||
|
CoreRuleSet: crs,
|
||||||
|
Now: utcNow,
|
||||||
|
Rules: ruleFiles,
|
||||||
|
Alerts: alertQueue,
|
||||||
|
|
||||||
|
BesideLookupTimeLimit: params.BesideLookupTimeLimit,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// newWAFListener listens on SWW_WAF_LISTEN_ADDR.
|
||||||
|
func newWAFListener(cfg *config.WAF) (net.Listener, error) {
|
||||||
|
return listen(cfg.ListenAddr, "SWW_WAF_LISTEN_ADDR")
|
||||||
|
}
|
||||||
|
|
||||||
|
// newStateFiles reads the state and cache files into the parts and the
|
||||||
|
// alert queue, as state.Load does. Until waf stops, it writes them as they
|
||||||
|
// are due and takes in an admin's edits of them.
|
||||||
|
func newStateFiles(
|
||||||
|
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
|
||||||
|
m *metrics.Metrics, ledger *bans.Ledger, limiter *ratelimit.Limiter,
|
||||||
|
geoJS *geojs.GeoJS, lists *reputation.Lists, dnsbl *reputation.DNSBL,
|
||||||
|
abuseIPDB *reputation.AbuseIPDB, routeWatch *routewatch.Client,
|
||||||
|
alertQueue *alerts.Queue, anomalies *anomaly.Counters,
|
||||||
|
) (*state.Files, error) {
|
||||||
|
files, err := state.Load(state.Params{
|
||||||
|
Dir: cfg.StateDir,
|
||||||
|
CacheDir: cfg.CacheDir,
|
||||||
|
BansWriteDelay: cfg.BansWriteDelay,
|
||||||
|
WriteInterval: cfg.StateWriteInterval,
|
||||||
|
Ledger: ledger,
|
||||||
|
Limiter: limiter,
|
||||||
|
GeoJS: geoJS,
|
||||||
|
Lists: lists,
|
||||||
|
DNSBL: dnsbl,
|
||||||
|
AbuseIPDB: abuseIPDB,
|
||||||
|
RouteWatch: routeWatch,
|
||||||
|
Alerts: alertQueue,
|
||||||
|
Anomalies: anomalies,
|
||||||
|
Now: utcNow,
|
||||||
|
ProcessLog: processLog,
|
||||||
|
Metrics: m,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return nil, startError{"cannot use the state and cache files", err}
|
||||||
|
}
|
||||||
|
|
||||||
|
lc.Append(background(files.Run))
|
||||||
|
lc.Append(background(files.Watch))
|
||||||
|
|
||||||
|
return files, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// newRemoteLogSender returns a sender of the log lines to
|
||||||
|
// SWW_WAF_LOG_REMOTE_URL, with the settings for it.
|
||||||
|
func newRemoteLogSender(cfg *config.WAF) *remotelog.Sender {
|
||||||
|
return remotelog.New(remotelog.Params{
|
||||||
|
URL: cfg.LogRemoteURL,
|
||||||
|
RootCAs: cfg.LogRemoteTLSCAs,
|
||||||
|
BufferLines: cfg.LogRemoteBufferLines,
|
||||||
|
Facility: cfg.LogRemoteFacility,
|
||||||
|
AppName: cfg.LogRemoteAppName,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// startSending runs remote until the function it returns is called, which
|
||||||
|
// then waits at most remoteLogStopTimeout for the lines still waiting to
|
||||||
|
// be sent. Sending goes on after ctx is done, so that the lines written
|
||||||
|
// while sww stops are sent too.
|
||||||
|
func startSending(
|
||||||
|
ctx context.Context, remote *remotelog.Sender, processLog *slog.Logger,
|
||||||
|
) func() {
|
||||||
|
sending, stop := context.WithCancel(context.WithoutCancel(ctx))
|
||||||
|
sent := make(chan struct{})
|
||||||
|
|
||||||
|
go func() {
|
||||||
|
remote.Run(sending, processLog)
|
||||||
|
close(sent)
|
||||||
|
}()
|
||||||
|
|
||||||
|
return func() {
|
||||||
|
stop()
|
||||||
|
|
||||||
|
select {
|
||||||
|
case <-sent:
|
||||||
|
case <-time.After(remoteLogStopTimeout):
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -90,6 +90,8 @@ func TestMainRunsWAF(t *testing.T) {
|
|||||||
cmd := exec.CommandContext(t.Context(), binary)
|
cmd := exec.CommandContext(t.Context(), binary)
|
||||||
cmd.Stdout = out
|
cmd.Stdout = out
|
||||||
|
|
||||||
|
// The Core Rule Set is off. Main keeps waf's own limit for its checks,
|
||||||
|
// and on a busy host the Core Rule Set's inspection can pass it.
|
||||||
cmd.Env = append(os.Environ(),
|
cmd.Env = append(os.Environ(),
|
||||||
"TZ="+zone,
|
"TZ="+zone,
|
||||||
commandForMain+"="+wafCommand,
|
commandForMain+"="+wafCommand,
|
||||||
@@ -97,7 +99,8 @@ func TestMainRunsWAF(t *testing.T) {
|
|||||||
stateDir+"="+dir,
|
stateDir+"="+dir,
|
||||||
cacheDir+"="+dir,
|
cacheDir+"="+dir,
|
||||||
ruleFilesDir+"="+t.TempDir(),
|
ruleFilesDir+"="+t.TempDir(),
|
||||||
lookupSource+"=off")
|
lookupSource+"=off",
|
||||||
|
"SWW_WAF_CORE_RULE_SET_MODE=off")
|
||||||
|
|
||||||
exited := startProcess(t, cmd)
|
exited := startProcess(t, cmd)
|
||||||
|
|
||||||
|
|||||||
@@ -35,7 +35,7 @@ func (abuseIPDB) check(
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
client := rq.h.clientGroup(rq.client)
|
client := clientGroup(rq.h.config, rq.client)
|
||||||
held, _ := rq.h.limiter.Client(client)
|
held, _ := rq.h.limiter.Client(client)
|
||||||
offender := held.History.Offences != ratelimit.Offences{}
|
offender := held.History.Offences != ratelimit.Offences{}
|
||||||
|
|
||||||
@@ -54,7 +54,7 @@ func (abuseIPDB) check(
|
|||||||
"scored by AbuseIPDB at or over SWW_WAF_ABUSEIPDB_MIN_SCORE",
|
"scored by AbuseIPDB at or over SWW_WAF_ABUSEIPDB_MIN_SCORE",
|
||||||
)
|
)
|
||||||
|
|
||||||
if rq.h.config.DNSBLAbuseIPDBAction != deny {
|
if rq.h.config.DNSBLAbuseIPDBAction != config.ReputationActionDeny {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,46 @@
|
|||||||
|
package waf
|
||||||
|
|
||||||
|
// The bans for AbuseIPDB's blacklist. They read SWW_WAF_ABUSEIPDB_BAN_DAYS and
|
||||||
|
// SWW_WAF_MODE; the blacklist is fetched by the reputation package.
|
||||||
|
|
||||||
|
import (
|
||||||
|
"log/slog"
|
||||||
|
"net/netip"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
|
)
|
||||||
|
|
||||||
|
// BanAbuseIPDBBlacklist bans in ledger the netblock of each client on
|
||||||
|
// AbuseIPDB's blacklist, listed, for SWW_WAF_ABUSEIPDB_BAN_DAYS from now, as
|
||||||
|
// the ledger's BanForAbuseIPDB describes. It logs to processLog how many
|
||||||
|
// clients were listed and how many bans were made. Clients in one netblock
|
||||||
|
// make one ban, with the highest of their scores. In observe mode it makes
|
||||||
|
// no ban.
|
||||||
|
func BanAbuseIPDBBlacklist(
|
||||||
|
cfg *config.WAF, ledger *bans.Ledger, processLog *slog.Logger,
|
||||||
|
now time.Time, listed []reputation.BlacklistEntry,
|
||||||
|
) {
|
||||||
|
scores := make(map[netip.Prefix]int64, len(listed))
|
||||||
|
|
||||||
|
for _, client := range listed {
|
||||||
|
netblock := banNetblock(cfg, client.Addr)
|
||||||
|
scores[netblock] = max(scores[netblock], client.Score)
|
||||||
|
}
|
||||||
|
|
||||||
|
made := 0
|
||||||
|
|
||||||
|
if !cfg.Observe {
|
||||||
|
made = ledger.BanForAbuseIPDB(
|
||||||
|
now, now.Add(cfg.AbuseIPDBBanDuration), scores,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
processLog.Info(
|
||||||
|
"fetched AbuseIPDB's blacklist",
|
||||||
|
"clients", len(listed),
|
||||||
|
"bans_made", made,
|
||||||
|
)
|
||||||
|
}
|
||||||
@@ -0,0 +1,100 @@
|
|||||||
|
package waf_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"net/http"
|
||||||
|
"net/netip"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
|
)
|
||||||
|
|
||||||
|
// abuseIPDBBanDays is the setting of how long a ban for AbuseIPDB's
|
||||||
|
// blacklist lasts, in days.
|
||||||
|
const abuseIPDBBanDays = "SWW_WAF_ABUSEIPDB_BAN_DAYS"
|
||||||
|
|
||||||
|
func TestClientOnTheFetchedAbuseIPDBBlacklistIsRefused(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
s, clk, server := startWithClock(t, "", map[string]string{
|
||||||
|
abuseIPDBKey: accountKey,
|
||||||
|
})
|
||||||
|
|
||||||
|
ctx, cancel := context.WithCancel(t.Context())
|
||||||
|
stopped := make(chan struct{})
|
||||||
|
|
||||||
|
go func() {
|
||||||
|
server.AbuseIPDB.RunBlacklist(ctx)
|
||||||
|
close(stopped)
|
||||||
|
}()
|
||||||
|
|
||||||
|
t.Cleanup(func() {
|
||||||
|
cancel()
|
||||||
|
<-stopped
|
||||||
|
})
|
||||||
|
|
||||||
|
waitUntil(func() bool {
|
||||||
|
active, _ := server.Ledger.Count(clk.Now())
|
||||||
|
|
||||||
|
return active == 1
|
||||||
|
})
|
||||||
|
|
||||||
|
s.get(client, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||||
|
s.get(otherClient, http.StatusOK, requestlog.ActionForward)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestClientsOnTheAbuseIPDBBlacklistAreBannedForTheBanDays(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
s, clk, server := startWithClock(t, "", map[string]string{
|
||||||
|
abuseIPDBBanDays: "2", banScopeV4Prefix: "24",
|
||||||
|
})
|
||||||
|
|
||||||
|
// Two clients in one netblock make one ban, with the higher score.
|
||||||
|
server.BanAbuseIPDBBlacklist([]reputation.BlacklistEntry{
|
||||||
|
{Addr: netip.MustParseAddr(client), Score: 95},
|
||||||
|
{Addr: netip.MustParseAddr(otherClient), Score: 100},
|
||||||
|
{Addr: netip.MustParseAddr("2001:db8:5::1"), Score: 90},
|
||||||
|
})
|
||||||
|
|
||||||
|
held := server.Ledger.Bans(netip.MustParsePrefix(clientsNet))
|
||||||
|
if len(held) != 1 || held[0].Cause != bans.CauseAbuseIPDB ||
|
||||||
|
*held[0].Notes.Reputation[0].Score != 100 {
|
||||||
|
t.Errorf("bans on %s %+v, want one for the blacklist, with 100",
|
||||||
|
clientsNet, held)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Every client of a banned netblock is refused, the IPv6 address's /64
|
||||||
|
// included.
|
||||||
|
for _, from := range []string{"203.0.113.200", "2001:db8:5::ffff:1"} {
|
||||||
|
s.get(from, http.StatusForbidden, requestlog.ActionTemporarilyBanned)
|
||||||
|
}
|
||||||
|
|
||||||
|
s.get("2001:db8:5:1::1", http.StatusOK, requestlog.ActionForward)
|
||||||
|
|
||||||
|
clk.advance(2 * 24 * time.Hour)
|
||||||
|
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAbuseIPDBBlacklistMakesNoBanInObserveMode(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
s, clk, server := startWithClock(
|
||||||
|
t,
|
||||||
|
"",
|
||||||
|
map[string]string{mode: "observe"},
|
||||||
|
)
|
||||||
|
|
||||||
|
server.BanAbuseIPDBBlacklist([]reputation.BlacklistEntry{
|
||||||
|
{Addr: netip.MustParseAddr(client), Score: 100},
|
||||||
|
})
|
||||||
|
|
||||||
|
if active, _ := server.Ledger.Count(clk.Now()); active != 0 {
|
||||||
|
t.Errorf("%d bans active, want none", active)
|
||||||
|
}
|
||||||
|
|
||||||
|
s.get(client, http.StatusOK, requestlog.ActionForward)
|
||||||
|
}
|
||||||
@@ -139,7 +139,7 @@ func (rq *request) addBan() {
|
|||||||
)
|
)
|
||||||
|
|
||||||
if err == nil {
|
if err == nil {
|
||||||
netblock, err = rq.h.banNetblock(toAdd.Netblock)
|
netblock, err = rq.h.parseBanNetblock(toAdd.Netblock)
|
||||||
}
|
}
|
||||||
|
|
||||||
if err == nil {
|
if err == nil {
|
||||||
@@ -190,12 +190,12 @@ func (rq *request) readBanToAdd() (banToAdd, error) {
|
|||||||
return toAdd, nil
|
return toAdd, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// banNetblock reads value, a netblock such as 203.0.113.0/24, or a
|
// parseBanNetblock reads value, a netblock such as 203.0.113.0/24, or a
|
||||||
// client's address, which stands for the netblock a ban on that client
|
// client's address, which stands for the netblock a ban on that client
|
||||||
// covers. An IPv4-mapped netblock, such as ::ffff:203.0.113.0/120, is
|
// covers. An IPv4-mapped netblock, such as ::ffff:203.0.113.0/120, is
|
||||||
// refused, since a client's address is looked up as IPv4 and a ban on it
|
// refused, since a client's address is looked up as IPv4 and a ban on it
|
||||||
// would refuse nothing, and so is a value with a zone.
|
// would refuse nothing, and so is a value with a zone.
|
||||||
func (h *handler) banNetblock(value string) (netip.Prefix, error) {
|
func (h *handler) parseBanNetblock(value string) (netip.Prefix, error) {
|
||||||
netblock, err := netip.ParsePrefix(value)
|
netblock, err := netip.ParsePrefix(value)
|
||||||
if err == nil {
|
if err == nil {
|
||||||
if netblock.Addr().Is4In6() {
|
if netblock.Addr().Is4In6() {
|
||||||
@@ -224,7 +224,7 @@ func (h *handler) banNetblock(value string) (netip.Prefix, error) {
|
|||||||
return netip.Prefix{}, fmt.Errorf("netblock %q %w", value, errZone)
|
return netip.Prefix{}, fmt.Errorf("netblock %q %w", value, errZone)
|
||||||
}
|
}
|
||||||
|
|
||||||
return h.netblock(addr), nil
|
return banNetblock(h.config, addr), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// expiry returns when a ban made at now for duration ends: duration
|
// expiry returns when a ban made at now for duration ends: duration
|
||||||
@@ -295,7 +295,7 @@ func (rq *request) showClient() {
|
|||||||
Bans: state.BanEntries(rq.h.ledger.Covering(addr)),
|
Bans: state.BanEntries(rq.h.ledger.Covering(addr)),
|
||||||
}
|
}
|
||||||
|
|
||||||
client, seen := rq.h.limiter.Client(rq.h.clientGroup(addr))
|
client, seen := rq.h.limiter.Client(clientGroup(rq.h.config, addr))
|
||||||
if seen {
|
if seen {
|
||||||
answer.Client = &client
|
answer.Client = &client
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -37,10 +37,10 @@ func TestAdminEndpointsAreOffWhileTheTokenIsUnset(t *testing.T) {
|
|||||||
"",
|
"",
|
||||||
map[string]string{metricsToken: token},
|
map[string]string{metricsToken: token},
|
||||||
)
|
)
|
||||||
server.Ledger.BanForLimit(
|
server.Ledger.Ban(
|
||||||
netip.MustParsePrefix(otherClient+"/32"),
|
netip.MustParsePrefix(otherClient+"/32"),
|
||||||
clk.Now(),
|
clk.Now(),
|
||||||
bans.Notes{},
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
)
|
)
|
||||||
before := server.Ledger.Snapshot()
|
before := server.Ledger.Snapshot()
|
||||||
|
|
||||||
|
|||||||
@@ -18,7 +18,9 @@ const (
|
|||||||
alertWebhookURL = "SWW_WAF_ALERT_WEBHOOK_URL"
|
alertWebhookURL = "SWW_WAF_ALERT_WEBHOOK_URL"
|
||||||
alertMaxPerHour = "SWW_WAF_ALERT_MAX_PER_HOUR"
|
alertMaxPerHour = "SWW_WAF_ALERT_MAX_PER_HOUR"
|
||||||
// alertInstance is the instance every alert of these tests gives.
|
// alertInstance is the instance every alert of these tests gives.
|
||||||
alertInstance = "fsn1app1/blog"
|
alertInstance = "host1/blog"
|
||||||
|
// alertWebhook is where the tests send alerts.
|
||||||
|
alertWebhook = "https://alerts.example/sww"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestBanForABrokenLimitRaisesABanAlertWithItsNotes(t *testing.T) {
|
func TestBanForABrokenLimitRaisesABanAlertWithItsNotes(t *testing.T) {
|
||||||
@@ -128,11 +130,9 @@ func TestObserveModeRaisesTheBanAlertsItWouldHave(t *testing.T) {
|
|||||||
// A ban for a clear sign of attack, which a request under it would make
|
// A ban for a clear sign of attack, which a request under it would make
|
||||||
// permanent.
|
// permanent.
|
||||||
group := netip.MustParsePrefix(ipv6Group)
|
group := netip.MustParsePrefix(ipv6Group)
|
||||||
attackBan, _ := server.Ledger.BanForAttack(
|
attackBan, _ := server.Ledger.Ban(group, start, bans.Grounds{
|
||||||
group,
|
Cause: bans.CauseAttack, Notes: bans.Notes{RuleID: "probe"},
|
||||||
start,
|
})
|
||||||
bans.Notes{RuleID: "probe"},
|
|
||||||
)
|
|
||||||
|
|
||||||
// The third request breaks the limit, and so does the fourth, within the
|
// The third request breaks the limit, and so does the fourth, within the
|
||||||
// cooldown, which raises nothing. The probe is a clear sign of attack.
|
// cooldown, which raises nothing. The probe is a clear sign of attack.
|
||||||
@@ -270,7 +270,7 @@ func startAppWithAlerts(
|
|||||||
clk := &clock{now: time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)}
|
clk := &clock{now: time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)}
|
||||||
settings := map[string]string{
|
settings := map[string]string{
|
||||||
trustedProxies: trustLocalhost,
|
trustedProxies: trustLocalhost,
|
||||||
alertWebhookURL: "https://alerts.example/sww",
|
alertWebhookURL: alertWebhook,
|
||||||
instanceName: alertInstance,
|
instanceName: alertInstance,
|
||||||
}
|
}
|
||||||
maps.Copy(settings, env)
|
maps.Copy(settings, env)
|
||||||
|
|||||||
@@ -349,6 +349,7 @@ func TestRequestCountsForTheASNumberGeoJSGivesBeforeItEnds(t *testing.T) {
|
|||||||
map[string]string{
|
map[string]string{
|
||||||
trustedProxies: trustLocalhost,
|
trustedProxies: trustLocalhost,
|
||||||
asnAndCountryLookupTimeout: "3600000",
|
asnAndCountryLookupTimeout: "3600000",
|
||||||
|
addLookupHeaders: falseValue,
|
||||||
anomalyASN + requestsPerMinute: aLot,
|
anomalyASN + requestsPerMinute: aLot,
|
||||||
})
|
})
|
||||||
server.Store(started)
|
server.Store(started)
|
||||||
|
|||||||
+148
-19
@@ -20,7 +20,7 @@ import (
|
|||||||
"github.com/prometheus/client_golang/prometheus/testutil"
|
"github.com/prometheus/client_golang/prometheus/testutil"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
|
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
@@ -39,6 +39,7 @@ const (
|
|||||||
// The header limits SPEC.md gives.
|
// The header limits SPEC.md gives.
|
||||||
cookieLimit = 50
|
cookieLimit = 50
|
||||||
headerFieldLimit = 100
|
headerFieldLimit = 100
|
||||||
|
lineLimit = 8192
|
||||||
// cookieHeader is the header the cookie limit counts.
|
// cookieHeader is the header the cookie limit counts.
|
||||||
cookieHeader = "Cookie"
|
cookieHeader = "Cookie"
|
||||||
// tooLarge is how waf refuses a request over the header limits.
|
// tooLarge is how waf refuses a request over the header limits.
|
||||||
@@ -104,11 +105,139 @@ func TestHeaderLimitRefusalNamesItsHeuristicCountsAndLimits(t *testing.T) {
|
|||||||
fillHeaders(req, headerFieldLimit+1)
|
fillHeaders(req, headerFieldLimit+1)
|
||||||
do(t, req)
|
do(t, req)
|
||||||
|
|
||||||
|
// The longest header line is X-Request-ID's, with its 26-character id.
|
||||||
wantHeuristic(
|
wantHeuristic(
|
||||||
t,
|
t,
|
||||||
s.out.requestLine(t),
|
s.out.requestLine(t),
|
||||||
"header_limits",
|
"header_limits",
|
||||||
`{"fields": 101, "fields_limit": 100, "cookies": 0, "cookies_limit": 50}`,
|
`{"longest_header_line_bytes": 40, "header_line_bytes_limit": 8192,
|
||||||
|
"fields": 101, "fields_limit": 100, "cookies": 0, "cookies_limit": 50}`,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// longQuery returns a path whose request line, sent by Go's client as
|
||||||
|
// "GET <path> HTTP/1.1", is lineBytes long.
|
||||||
|
func longQuery(lineBytes int) string {
|
||||||
|
const around = len("GET /?q= HTTP/1.1")
|
||||||
|
|
||||||
|
return "/?q=" + strings.Repeat("a", lineBytes-around)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestARequestLineOverTheLimitIsRefusedWith414BeforeAnyOtherCheck(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
lineBytes int
|
||||||
|
// cookies is how many cookies the request carries.
|
||||||
|
cookies int
|
||||||
|
status int
|
||||||
|
action string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
"at the limit", lineLimit, 0, http.StatusOK,
|
||||||
|
requestlog.ActionForward,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"over the limit", lineLimit + 1, 0, http.StatusRequestURITooLong,
|
||||||
|
requestlog.ActionRequestLineTooLong,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"over the limit and the cookie limit", lineLimit + 1,
|
||||||
|
cookieLimit + 1, http.StatusRequestURITooLong,
|
||||||
|
requestlog.ActionRequestLineTooLong,
|
||||||
|
},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
s := startWAF(t, "", map[string]string{coreRuleSetMode: block})
|
||||||
|
req := s.question(siteA, client, longQuery(tc.lineBytes))
|
||||||
|
|
||||||
|
if tc.cookies > 0 {
|
||||||
|
req.Header.Set(cookieHeader, cookies(tc.cookies))
|
||||||
|
}
|
||||||
|
|
||||||
|
got := do(t, req)
|
||||||
|
line := s.out.requestLine(t)
|
||||||
|
|
||||||
|
if tc.status == http.StatusOK {
|
||||||
|
wantDisposition(t, got, tc.status, waf.DispositionAllow)
|
||||||
|
} else {
|
||||||
|
wantDisposition(t, got, tc.status, waf.DispositionDeny)
|
||||||
|
}
|
||||||
|
|
||||||
|
wantLine(t, line, tc.status, tc.action)
|
||||||
|
|
||||||
|
refused := tc.status != http.StatusOK
|
||||||
|
if refused != (line.DurationCoreRuleSetMS == nil) {
|
||||||
|
t.Errorf(
|
||||||
|
"Core Rule Set time %v for a request refused: %t",
|
||||||
|
line.DurationCoreRuleSetMS,
|
||||||
|
refused,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAHeaderLineOverTheLimitIsRefusedWith431(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
make func(*http.Request)
|
||||||
|
// refused is true for a request over the limit, and false for one at it.
|
||||||
|
refused bool
|
||||||
|
}{
|
||||||
|
{"a header line at the limit", func(req *http.Request) {
|
||||||
|
req.Header.Set("X-Long", strings.Repeat("a", lineLimit-len("X-Long: ")))
|
||||||
|
}, false},
|
||||||
|
{"a header line over the limit", func(req *http.Request) {
|
||||||
|
req.Header.Set("X-Long", strings.Repeat("a", lineLimit+1-len("X-Long: ")))
|
||||||
|
}, true},
|
||||||
|
{"one cookie over the limit", func(req *http.Request) {
|
||||||
|
req.Header.Set(cookieHeader, "a="+strings.Repeat("a", lineLimit))
|
||||||
|
}, true},
|
||||||
|
{"a Host line over the limit", func(req *http.Request) {
|
||||||
|
req.Host = strings.Repeat("a", lineLimit+1-len("Host: "))
|
||||||
|
}, true},
|
||||||
|
{"a header sent twice, each line within the limit", func(req *http.Request) {
|
||||||
|
req.Header.Add("X-Long", strings.Repeat("a", 8000))
|
||||||
|
req.Header.Add("X-Long", strings.Repeat("a", 8000))
|
||||||
|
}, false},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
s := startWAF(t, "", nil)
|
||||||
|
req := s.question(siteA, client, "/")
|
||||||
|
tc.make(req)
|
||||||
|
|
||||||
|
got := do(t, req)
|
||||||
|
|
||||||
|
if tc.refused {
|
||||||
|
wantDisposition(t, got, tooLarge, waf.DispositionDeny)
|
||||||
|
} else {
|
||||||
|
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRequestLineRefusalNamesItsHeuristicLengthAndLimit(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
s := startWAF(t, "", nil)
|
||||||
|
do(t, s.question(siteA, client, longQuery(lineLimit+1)))
|
||||||
|
|
||||||
|
wantHeuristic(
|
||||||
|
t,
|
||||||
|
s.out.requestLine(t),
|
||||||
|
"header_limits",
|
||||||
|
`{"request_line_bytes": 8193, "request_line_bytes_limit": 8192}`,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -462,7 +591,7 @@ func TestAClientGeoJSPlacedInNoCountryCountsAsUnknownWithoutWaiting(
|
|||||||
s := startWAF(t, geojsURL, map[string]string{
|
s := startWAF(t, geojsURL, map[string]string{
|
||||||
unknownCountryLimitPercent: "0", rateLimitPerMinute: "1",
|
unknownCountryLimitPercent: "0", rateLimitPerMinute: "1",
|
||||||
})
|
})
|
||||||
s.server.GeoJS.Load([]lookup.Answer{{
|
s.server.GeoJS.Load([]geojs.Answer{{
|
||||||
Client: netip.MustParsePrefix(client + "/32"), ASN: "AS64512",
|
Client: netip.MustParsePrefix(client + "/32"), ASN: "AS64512",
|
||||||
Answered: s.clock.Now(), Used: s.clock.Now(),
|
Answered: s.clock.Now(), Used: s.clock.Now(),
|
||||||
}})
|
}})
|
||||||
@@ -557,10 +686,10 @@ func askDirectly(
|
|||||||
|
|
||||||
// banClient bans client at now.
|
// banClient bans client at now.
|
||||||
func banClient(_ *testing.T, server *waf.Server, now time.Time) {
|
func banClient(_ *testing.T, server *waf.Server, now time.Time) {
|
||||||
server.Ledger.BanForLimit(
|
server.Ledger.Ban(
|
||||||
netip.MustParsePrefix(client+"/32"),
|
netip.MustParsePrefix(client+"/32"),
|
||||||
now,
|
now,
|
||||||
bans.Notes{},
|
bans.Grounds{Cause: bans.CauseLimit},
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -641,6 +770,8 @@ func TestARefusedRequestDoesNotWaitForASlowGeoJS(t *testing.T) {
|
|||||||
// TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays says,
|
// TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays says,
|
||||||
// so that a request that does not wait takes no time at all.
|
// so that a request that does not wait takes no time at all.
|
||||||
synctest.Test(t, func(t *testing.T) {
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
defer waitForGeoJSRequestsToBeAbandoned()
|
||||||
|
|
||||||
server, out := newWAFInBubble(t, tc.env)
|
server, out := newWAFInBubble(t, tc.env)
|
||||||
tc.prepare(t, server, time.Now())
|
tc.prepare(t, server, time.Now())
|
||||||
|
|
||||||
@@ -709,6 +840,8 @@ func TestAQuestionWaitsForASlowGeoJSOnlyAsLongAsTheDefaultSays(
|
|||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
defer waitForGeoJSRequestsToBeAbandoned()
|
||||||
|
|
||||||
server, out := newWAFInBubble(t, tc.env)
|
server, out := newWAFInBubble(t, tc.env)
|
||||||
tc.prepare(t, server, time.Now())
|
tc.prepare(t, server, time.Now())
|
||||||
|
|
||||||
@@ -744,18 +877,13 @@ func TestWAFAnswersWithinTheTimeLimitWhileEveryOutsideSourceHangs(
|
|||||||
held, _, _ := startHeldGeoJS(t)
|
held, _, _ := startHeldGeoJS(t)
|
||||||
resolver, releaseResolver := startHeldResolver(t)
|
resolver, releaseResolver := startHeldResolver(t)
|
||||||
|
|
||||||
server, out, _ := newWAFWith(t, func(
|
server, out, _ := newWAFWith(t, waf.New,
|
||||||
params waf.Params,
|
silentGeoJSURL, silentAbuseIPDBURL, time.Now, map[string]string{
|
||||||
) *waf.Server {
|
tagDeny: "country=KP", coreRuleSetMode: block,
|
||||||
params.AbuseIPDBURL = silentAbuseIPDBURL
|
dnsblZones: dnsblZone, dnsblResolver: resolver,
|
||||||
|
abuseIPDBKey: accountKey,
|
||||||
return waf.New(params)
|
blocklistURLs: held + "/drop.txt", asnLimitPercentURL: held + "/asn.txt",
|
||||||
}, silentGeoJSURL, time.Now, map[string]string{
|
})
|
||||||
tagDeny: "country=KP", coreRuleSetMode: block,
|
|
||||||
dnsblZones: dnsblZone, dnsblResolver: resolver,
|
|
||||||
abuseIPDBKey: accountKey,
|
|
||||||
blocklistURLs: held + "/drop.txt", asnLimitPercentURL: held + "/asn.txt",
|
|
||||||
})
|
|
||||||
|
|
||||||
// The lists are fetched outside the synctest bubble below: a wait on the
|
// The lists are fetched outside the synctest bubble below: a wait on the
|
||||||
// network inside it would keep its clock from moving on.
|
// network inside it would keep its clock from moving on.
|
||||||
@@ -821,10 +949,11 @@ func TestWAFAnswersWithinTheTimeLimitWhileEveryOutsideSourceHangs(
|
|||||||
t.Errorf("%d checks with AbuseIPDB, want 1", checked)
|
t.Errorf("%d checks with AbuseIPDB, want 1", checked)
|
||||||
}
|
}
|
||||||
|
|
||||||
// The query is answered, and the check with AbuseIPDB abandoned,
|
// The query is answered, and the check with AbuseIPDB and the request
|
||||||
// before the bubble ends.
|
// to GeoJS abandoned, before the bubble ends.
|
||||||
releaseResolver()
|
releaseResolver()
|
||||||
time.Sleep(defaultDNSBLAbuseIPDBWait)
|
time.Sleep(defaultDNSBLAbuseIPDBWait)
|
||||||
|
waitForGeoJSRequestsToBeAbandoned()
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+82
-61
@@ -10,6 +10,7 @@ import (
|
|||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
)
|
)
|
||||||
@@ -85,13 +86,14 @@ func (rq *request) countedBytes() int64 {
|
|||||||
response, request := rq.out.bytes, rq.requestBytes
|
response, request := rq.out.bytes, rq.requestBytes
|
||||||
|
|
||||||
switch rq.h.config.BytesCounted {
|
switch rq.h.config.BytesCounted {
|
||||||
case "response":
|
case config.BytesCountedResponse:
|
||||||
return response
|
return response
|
||||||
case "request":
|
case config.BytesCountedRequest:
|
||||||
return request
|
return request
|
||||||
default: // both
|
case config.BytesCountedBoth:
|
||||||
return response + request
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
return response + request
|
||||||
}
|
}
|
||||||
|
|
||||||
// banForLimit bans the client's netblock at now for a broken limit, the
|
// banForLimit bans the client's netblock at now for a broken limit, the
|
||||||
@@ -101,9 +103,8 @@ func (rq *request) countedBytes() int64 {
|
|||||||
// one whose bytes broke a byte limit, the refusal for one that broke the
|
// one whose bytes broke a byte limit, the refusal for one that broke the
|
||||||
// refusal limit. The ban's notes give the client's limit percentage for a
|
// refusal limit. The ban's notes give the client's limit percentage for a
|
||||||
// rate limit or a byte limit; the refusal limit is not lowered. The ban sets
|
// rate limit or a byte limit; the refusal limit is not lowered. The ban sets
|
||||||
// the client's counters back to zero. In observe mode it makes no ban and
|
// the client's counters back to zero. In observe mode, banClient says what
|
||||||
// sets nothing back, and raises the alert for the ban it would have made,
|
// happens instead.
|
||||||
// if that alert would be sent.
|
|
||||||
func (rq *request) banForLimit(
|
func (rq *request) banForLimit(
|
||||||
now time.Time,
|
now time.Time,
|
||||||
hit ratelimit.Hit,
|
hit ratelimit.Hit,
|
||||||
@@ -121,23 +122,11 @@ func (rq *request) banForLimit(
|
|||||||
rq.line.Offence = requestlog.OffenceLimit
|
rq.line.Offence = requestlog.OffenceLimit
|
||||||
rq.h.metrics.LimitHit(hit)
|
rq.h.metrics.LimitHit(hit)
|
||||||
|
|
||||||
netblock := rq.h.netblock(rq.client)
|
|
||||||
if rq.h.config.Observe &&
|
|
||||||
!rq.wouldAlertBan(netblock, now, bans.CauseLimit) {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
notes := bans.Notes{
|
notes := bans.Notes{
|
||||||
ASN: rq.line.ASN,
|
Kind: hit.Kind,
|
||||||
ASName: rq.line.ASName,
|
Limit: hit.Limit,
|
||||||
Country: rq.line.Country,
|
Window: hit.Window,
|
||||||
Kind: hit.Kind,
|
Count: hit.Count,
|
||||||
Limit: hit.Limit,
|
|
||||||
Window: hit.Window,
|
|
||||||
Count: hit.Count,
|
|
||||||
Reputation: rq.reputation,
|
|
||||||
Request: rq.noted(now, status),
|
|
||||||
Requests: rq.netblockRequests(netblock),
|
|
||||||
}
|
}
|
||||||
|
|
||||||
switch hit.Kind {
|
switch hit.Kind {
|
||||||
@@ -147,47 +136,54 @@ func (rq *request) banForLimit(
|
|||||||
notes.LimitPercent, notes.LimitPercentSetting = rq.bytesLimitPercent.logged()
|
notes.LimitPercent, notes.LimitPercentSetting = rq.bytesLimitPercent.logged()
|
||||||
}
|
}
|
||||||
|
|
||||||
if rq.h.config.Observe {
|
rq.banClient(
|
||||||
ban, wouldBan := rq.h.ledger.WouldBanForLimit(netblock, now, notes)
|
now,
|
||||||
if wouldBan {
|
status,
|
||||||
rq.alertBan(ban)
|
bans.Grounds{Cause: bans.CauseLimit, Notes: notes},
|
||||||
}
|
)
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
ban, made := rq.h.ledger.BanForLimit(netblock, now, notes)
|
|
||||||
rq.h.limiter.Reset(rq.h.clientGroup(rq.client))
|
|
||||||
rq.line.BanExpires = banExpires(ban)
|
|
||||||
|
|
||||||
if made {
|
|
||||||
rq.alertBan(ban)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// banForAttack bans the client's netblock at now for a clear sign of
|
// banForAttack bans the client's netblock at now for a clear sign of
|
||||||
// attack, which notes name: the ban rule that matched, or the trap path
|
// attack, which notes name: the ban rule that matched, or the trap path
|
||||||
// asked for. It fills in the rest of the notes. In observe mode it makes
|
// asked for. It returns the action for that ban. In observe mode, banClient
|
||||||
// no ban, and raises the alert for the ban it would have made, if that
|
// says what happens instead.
|
||||||
// alert would be sent. It returns the action for that ban.
|
|
||||||
func (rq *request) banForAttack(now time.Time, notes bans.Notes) string {
|
func (rq *request) banForAttack(now time.Time, notes bans.Notes) string {
|
||||||
netblock := rq.h.netblock(rq.client)
|
return rq.banClient(
|
||||||
if rq.h.config.Observe &&
|
now,
|
||||||
!rq.wouldAlertBan(netblock, now, bans.CauseAttack) {
|
rq.h.config.BanResponse,
|
||||||
return banAction(
|
bans.Grounds{Cause: bans.CauseAttack, Notes: notes},
|
||||||
rq.h.ledger.WouldBePermanent(netblock, now, bans.CauseAttack),
|
)
|
||||||
)
|
}
|
||||||
}
|
|
||||||
|
|
||||||
notes.ASN = rq.line.ASN
|
// banClient bans the client's netblock at now on grounds, and returns the
|
||||||
notes.ASName = rq.line.ASName
|
// action for that ban. It adds to the notes what the request tells of the
|
||||||
notes.Country = rq.line.Country
|
// client, with status, what the client was sent. A ban for a broken limit
|
||||||
notes.Reputation = rq.reputation
|
// sets the client's counters back to zero.
|
||||||
notes.Request = rq.noted(now, rq.h.config.BanResponse)
|
//
|
||||||
notes.Requests = rq.netblockRequests(netblock)
|
// In observe mode it makes no ban and sets nothing back. It raises the
|
||||||
|
// alert for the ban it would have made, if that alert would be sent.
|
||||||
|
func (rq *request) banClient(
|
||||||
|
now time.Time,
|
||||||
|
status int,
|
||||||
|
grounds bans.Grounds,
|
||||||
|
) string {
|
||||||
|
netblock := banNetblock(rq.h.config, rq.client)
|
||||||
|
|
||||||
if rq.h.config.Observe {
|
if rq.h.config.Observe {
|
||||||
ban, wouldBan := rq.h.ledger.WouldBanForAttack(netblock, now, notes)
|
if !rq.wouldAlertBan(netblock, now, grounds.Cause) {
|
||||||
|
return banAction(
|
||||||
|
rq.h.ledger.WouldBePermanent(netblock, now, grounds.Cause),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
grounds.Notes = rq.withClientNotes(
|
||||||
|
grounds.Notes,
|
||||||
|
now,
|
||||||
|
status,
|
||||||
|
netblock,
|
||||||
|
)
|
||||||
|
|
||||||
|
ban, wouldBan := rq.h.ledger.WouldBan(netblock, now, grounds)
|
||||||
if wouldBan {
|
if wouldBan {
|
||||||
rq.alertBan(ban)
|
rq.alertBan(ban)
|
||||||
}
|
}
|
||||||
@@ -195,7 +191,13 @@ func (rq *request) banForAttack(now time.Time, notes bans.Notes) string {
|
|||||||
return banAction(ban.Permanent())
|
return banAction(ban.Permanent())
|
||||||
}
|
}
|
||||||
|
|
||||||
ban, made := rq.h.ledger.BanForAttack(netblock, now, notes)
|
grounds.Notes = rq.withClientNotes(grounds.Notes, now, status, netblock)
|
||||||
|
|
||||||
|
ban, made := rq.h.ledger.Ban(netblock, now, grounds)
|
||||||
|
if grounds.Cause == bans.CauseLimit {
|
||||||
|
rq.h.limiter.Reset(clientGroup(rq.h.config, rq.client))
|
||||||
|
}
|
||||||
|
|
||||||
rq.line.BanExpires = banExpires(ban)
|
rq.line.BanExpires = banExpires(ban)
|
||||||
|
|
||||||
if made {
|
if made {
|
||||||
@@ -205,6 +207,25 @@ func (rq *request) banForAttack(now time.Time, notes bans.Notes) string {
|
|||||||
return banAction(ban.Permanent())
|
return banAction(ban.Permanent())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// withClientNotes is notes with what the request tells of the client
|
||||||
|
// added: its AS, country and reputation, the request at now with status,
|
||||||
|
// and how many requests netblock, the client's, has sent.
|
||||||
|
func (rq *request) withClientNotes(
|
||||||
|
notes bans.Notes,
|
||||||
|
now time.Time,
|
||||||
|
status int,
|
||||||
|
netblock netip.Prefix,
|
||||||
|
) bans.Notes {
|
||||||
|
notes.ASN = rq.line.ASN
|
||||||
|
notes.ASName = rq.line.ASName
|
||||||
|
notes.Country = rq.line.Country
|
||||||
|
notes.Reputation = rq.reputation
|
||||||
|
notes.Request = rq.noted(now, status)
|
||||||
|
notes.Requests = rq.netblockRequests(netblock)
|
||||||
|
|
||||||
|
return notes
|
||||||
|
}
|
||||||
|
|
||||||
// wouldAlertBan reports whether the alert for a ban on netblock for cause
|
// wouldAlertBan reports whether the alert for a ban on netblock for cause
|
||||||
// made at now would be sent. In observe mode the ban the request would
|
// made at now would be sent. In observe mode the ban the request would
|
||||||
// have made is worked out only then, at most once per
|
// have made is worked out only then, at most once per
|
||||||
@@ -273,16 +294,16 @@ func (rq *request) netblockRequests(netblock netip.Prefix) int64 {
|
|||||||
return rq.h.limiter.Requests(netblock) + 1
|
return rq.h.limiter.Requests(netblock) + 1
|
||||||
}
|
}
|
||||||
|
|
||||||
// netblock is the netblock a ban on client covers: its IPv4 address,
|
// banNetblock is the netblock a ban on client covers: its IPv4 address,
|
||||||
// widened to SWW_WAF_BAN_SCOPE_V4_PREFIX, or the IPv6 group clientGroup
|
// widened to SWW_WAF_BAN_SCOPE_V4_PREFIX, or the IPv6 group clientGroup
|
||||||
// counts it in.
|
// counts it in.
|
||||||
func (h *handler) netblock(client netip.Addr) netip.Prefix {
|
func banNetblock(cfg *config.WAF, client netip.Addr) netip.Prefix {
|
||||||
addr := client.Unmap()
|
addr := client.Unmap()
|
||||||
if addr.Is4() {
|
if addr.Is4() {
|
||||||
return netip.PrefixFrom(addr, h.config.BanScopeV4Prefix).Masked()
|
return netip.PrefixFrom(addr, cfg.BanScopeV4Prefix).Masked()
|
||||||
}
|
}
|
||||||
|
|
||||||
return h.clientGroup(addr)
|
return clientGroup(cfg, addr)
|
||||||
}
|
}
|
||||||
|
|
||||||
// banExpires is when ban ends, as the log line gives it: a time, or
|
// banExpires is when ban ends, as the log line gives it: a time, or
|
||||||
|
|||||||
@@ -548,6 +548,10 @@ func (s *sender) requestWithHeader(
|
|||||||
// body is sent as it is after the headers, header holding its
|
// body is sent as it is after the headers, header holding its
|
||||||
// Content-Length or Transfer-Encoding. header may hold several lines,
|
// Content-Length or Transfer-Encoding. header may hold several lines,
|
||||||
// separated by "\r\n". It returns the whole answer.
|
// separated by "\r\n". It returns the whole answer.
|
||||||
|
//
|
||||||
|
// It waits for the answer as long as it takes, since a busy host can slow
|
||||||
|
// the Core Rule Set past any shorter wait. go test's -timeout ends a test
|
||||||
|
// whose answer never comes.
|
||||||
func (s *sender) requestWithBody(
|
func (s *sender) requestWithBody(
|
||||||
method, from, path, header, body string, status int, action string,
|
method, from, path, header, body string, status int, action string,
|
||||||
) (logLine, answer) {
|
) (logLine, answer) {
|
||||||
@@ -566,11 +570,6 @@ func (s *sender) requestWithBody(
|
|||||||
send(s.t, conn, method+" "+path+" HTTP/1.1\r\nHost: "+appHost+"\r\n"+
|
send(s.t, conn, method+" "+path+" HTTP/1.1\r\nHost: "+appHost+"\r\n"+
|
||||||
forwardedFor+": "+from+"\r\n"+header+"\r\n"+body)
|
forwardedFor+": "+from+"\r\n"+header+"\r\n"+body)
|
||||||
|
|
||||||
err := conn.SetReadDeadline(time.Now().Add(waitLimit))
|
|
||||||
if err != nil {
|
|
||||||
s.t.Fatalf("set read deadline: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
var got answer
|
var got answer
|
||||||
|
|
||||||
res, err := http.ReadResponse(bufio.NewReader(conn), nil)
|
res, err := http.ReadResponse(bufio.NewReader(conn), nil)
|
||||||
|
|||||||
@@ -0,0 +1,172 @@
|
|||||||
|
package waf
|
||||||
|
|
||||||
|
// The heuristics of besideLookup, and how they run beside the lookup.
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"sync/atomic"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||||
|
)
|
||||||
|
|
||||||
|
// besideLookupTimeLimit is how long the heuristics of besideLookup have for a
|
||||||
|
// request, from when they start. They only read the request, and no request
|
||||||
|
// should take them that long. A request they have not finished by then is
|
||||||
|
// refused with 503, as ActionChecksTimedOut.
|
||||||
|
const besideLookupTimeLimit = 500 * time.Millisecond
|
||||||
|
|
||||||
|
// besideLookupFindings is what the heuristics of besideLookup found of a
|
||||||
|
// request, before any of it is applied to the request.
|
||||||
|
type besideLookupFindings struct {
|
||||||
|
// trapPath is the request's path when it is in SWW_WAF_TRAP_PATHS, and ""
|
||||||
|
// otherwise.
|
||||||
|
trapPath string
|
||||||
|
// rules are the rules of the rule files the request matched.
|
||||||
|
rules []rules.Rule
|
||||||
|
// inspected is true when the Core Rule Set inspected the request.
|
||||||
|
// inspection, read and inspectionTime are then what it found, what it read
|
||||||
|
// of the body, and how long it took.
|
||||||
|
inspected bool
|
||||||
|
inspection coreruleset.Result
|
||||||
|
read []byte
|
||||||
|
inspectionTime time.Duration
|
||||||
|
// refuses is true when one of the heuristics refuses the request, or would
|
||||||
|
// in observe mode. Those after it did no work.
|
||||||
|
refuses bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// runBesideLookup runs the heuristics of besideLookup while the lookup waits
|
||||||
|
// for GeoJS, and returns the first verdict, or nil when none gives one.
|
||||||
|
// lookedUp is the channel the lookup's result comes on, or nil once afterLookup
|
||||||
|
// has let the request through.
|
||||||
|
//
|
||||||
|
// The first of the two to refuse the request ends the checks: waf answers at
|
||||||
|
// once, and what the other found is dropped. besideLookup has
|
||||||
|
// besideLookupTimeLimit; past it, the request is refused as
|
||||||
|
// ActionChecksTimedOut. Once both have let the request through, the heuristics
|
||||||
|
// of afterBesideLookup run. ctx is the request's own context.
|
||||||
|
func (rq *request) runBesideLookup(
|
||||||
|
ctx context.Context, lookedUp <-chan lookupResult,
|
||||||
|
) *verdict {
|
||||||
|
var running atomic.Pointer[string]
|
||||||
|
|
||||||
|
running.Store(new(trapPaths{}.name()))
|
||||||
|
|
||||||
|
found := make(chan *besideLookupFindings, 1)
|
||||||
|
|
||||||
|
go func() { found <- rq.findBesideLookup(&running) }()
|
||||||
|
|
||||||
|
timeUp := time.NewTimer(rq.h.besideLookupLimit)
|
||||||
|
defer timeUp.Stop()
|
||||||
|
|
||||||
|
var findings *besideLookupFindings
|
||||||
|
|
||||||
|
for findings == nil || lookedUp != nil {
|
||||||
|
select {
|
||||||
|
case result := <-lookedUp:
|
||||||
|
lookedUp = nil
|
||||||
|
|
||||||
|
v := rq.runAfterLookup(ctx, result)
|
||||||
|
if v != nil {
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
case findings = <-found:
|
||||||
|
found = nil
|
||||||
|
|
||||||
|
timeUp.Stop()
|
||||||
|
|
||||||
|
if findings.refuses {
|
||||||
|
return rq.applyFindings(ctx, findings)
|
||||||
|
}
|
||||||
|
case <-timeUp.C:
|
||||||
|
return rq.timedOut(*running.Load())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
v := rq.applyFindings(ctx, findings)
|
||||||
|
if v != nil {
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
|
||||||
|
return rq.runHeuristics(ctx, afterBesideLookup(), rq.h.now())
|
||||||
|
}
|
||||||
|
|
||||||
|
// findBesideLookup does the work of the heuristics of besideLookup on the
|
||||||
|
// request, in their order, until one refuses it, and returns what they found.
|
||||||
|
// It changes nothing of the request or of what waf keeps, so that it can run
|
||||||
|
// while the lookup and afterLookup do. running holds the name of the heuristic
|
||||||
|
// at work, the first one's when findBesideLookup is called.
|
||||||
|
func (rq *request) findBesideLookup(
|
||||||
|
running *atomic.Pointer[string],
|
||||||
|
) *besideLookupFindings {
|
||||||
|
findings := &besideLookupFindings{}
|
||||||
|
|
||||||
|
findings.trapPath = rq.trapPath()
|
||||||
|
if findings.trapPath != "" {
|
||||||
|
findings.refuses = true
|
||||||
|
|
||||||
|
return findings
|
||||||
|
}
|
||||||
|
|
||||||
|
running.Store(new(ruleFiles{}.name()))
|
||||||
|
|
||||||
|
findings.rules = rq.h.rules.Match(rq.in)
|
||||||
|
if refusedByRule(findings.rules) {
|
||||||
|
findings.refuses = true
|
||||||
|
|
||||||
|
return findings
|
||||||
|
}
|
||||||
|
|
||||||
|
running.Store(new(coreRuleSet{}.name()))
|
||||||
|
|
||||||
|
cfg := rq.h.config
|
||||||
|
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff ||
|
||||||
|
pathExempt(rq.in, cfg.CoreRuleSetExemptPaths) {
|
||||||
|
return findings
|
||||||
|
}
|
||||||
|
|
||||||
|
start := time.Now()
|
||||||
|
// A question's body has been read into memory, so that reading it here
|
||||||
|
// cannot fail.
|
||||||
|
findings.inspection, findings.read, _ = rq.h.coreRuleSet.Inspect(
|
||||||
|
rq.in, rq.client, rq.in.Body,
|
||||||
|
)
|
||||||
|
findings.inspectionTime = time.Since(start)
|
||||||
|
findings.inspected = true
|
||||||
|
findings.refuses = rq.refusedByCoreRuleSet(findings.inspection)
|
||||||
|
|
||||||
|
return findings
|
||||||
|
}
|
||||||
|
|
||||||
|
// applyFindings applies findings, what findBesideLookup found, to the
|
||||||
|
// request: it runs the heuristics of besideLookup, which read them, at the
|
||||||
|
// time it is called, and returns the first verdict, or nil when none gives
|
||||||
|
// one.
|
||||||
|
func (rq *request) applyFindings(
|
||||||
|
ctx context.Context, findings *besideLookupFindings,
|
||||||
|
) *verdict {
|
||||||
|
rq.findings = findings
|
||||||
|
|
||||||
|
return rq.runHeuristics(ctx, besideLookup(), rq.h.now())
|
||||||
|
}
|
||||||
|
|
||||||
|
// timedOut refuses the request for the heuristics of besideLookup not having
|
||||||
|
// finished with it within besideLookupTimeLimit. running is the name of the
|
||||||
|
// one at work, which the log line's heuristic gives. The refusal makes no ban
|
||||||
|
// and is no offence.
|
||||||
|
func (rq *request) timedOut(running string) *verdict {
|
||||||
|
v := &verdict{
|
||||||
|
action: requestlog.ActionChecksTimedOut,
|
||||||
|
figures: map[string]any{
|
||||||
|
"time_limit_ms": rq.h.besideLookupLimit.Milliseconds(),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
rq.line.Heuristic = running
|
||||||
|
rq.line.HeuristicFigures = v.figures
|
||||||
|
|
||||||
|
return v
|
||||||
|
}
|
||||||
@@ -0,0 +1,426 @@
|
|||||||
|
package waf_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"io"
|
||||||
|
"maps"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"net/netip"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"testing/synctest"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/prometheus/client_golang/prometheus/testutil"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||||
|
)
|
||||||
|
|
||||||
|
// checksTimeLimit is how long the trap paths, the rule files and the Core Rule
|
||||||
|
// Set have for a request, as SPEC.md gives it, and checksTimedOut the status of
|
||||||
|
// a request they have not finished by then.
|
||||||
|
const (
|
||||||
|
checksTimeLimit = 500 * time.Millisecond
|
||||||
|
checksTimedOut = http.StatusServiceUnavailable
|
||||||
|
)
|
||||||
|
|
||||||
|
// heldCoreRuleSet is a stand-in for the Core Rule Set: each inspection takes
|
||||||
|
// took, then finds result. asked counts the inspections.
|
||||||
|
type heldCoreRuleSet struct {
|
||||||
|
took time.Duration
|
||||||
|
result coreruleset.Result
|
||||||
|
asked atomic.Int64
|
||||||
|
}
|
||||||
|
|
||||||
|
// Inspect inspects a request as the stand-in does, and reads all of body.
|
||||||
|
func (c *heldCoreRuleSet) Inspect(
|
||||||
|
_ *http.Request, _ netip.Addr, body io.Reader,
|
||||||
|
) (coreruleset.Result, []byte, error) {
|
||||||
|
c.asked.Add(1)
|
||||||
|
time.Sleep(c.took)
|
||||||
|
|
||||||
|
read, err := io.ReadAll(body)
|
||||||
|
|
||||||
|
return c.result, read, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// coreRuleSetMatch returns what the stand-in finds of an attack: rule 942100,
|
||||||
|
// with a score of 5, the default SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD.
|
||||||
|
func coreRuleSetMatch() coreruleset.Result {
|
||||||
|
return coreruleset.Result{RuleIDs: []int{942100}, Score: 5}
|
||||||
|
}
|
||||||
|
|
||||||
|
// newWAFHoldingTheCoreRuleSet makes a waf, to be asked in a synctest bubble,
|
||||||
|
// with crs in place of the Core Rule Set, in block mode. GeoJS never answers,
|
||||||
|
// a setting needs its answer, alerts go to a webhook, and everything else is
|
||||||
|
// at its default but for env. No request to it may use the network.
|
||||||
|
func newWAFHoldingTheCoreRuleSet(
|
||||||
|
t *testing.T, crs *heldCoreRuleSet, env map[string]string,
|
||||||
|
) (*waf.Server, *output, *alerts.Queue) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
settings := map[string]string{
|
||||||
|
tagDeny: denyKP, coreRuleSetMode: block,
|
||||||
|
alertWebhookURL: alertWebhook,
|
||||||
|
}
|
||||||
|
maps.Copy(settings, env)
|
||||||
|
|
||||||
|
server, out, queue := newWAFWith(t, func(params waf.Params) *waf.Server {
|
||||||
|
params.CoreRuleSet = crs
|
||||||
|
params.BesideLookupTimeLimit = waf.BesideLookupTimeLimit
|
||||||
|
|
||||||
|
return waf.New(params)
|
||||||
|
}, silentGeoJSURL, abuseIPDBURL, time.Now, settings)
|
||||||
|
|
||||||
|
return server, out, queue
|
||||||
|
}
|
||||||
|
|
||||||
|
// lookupWait is SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS for wait.
|
||||||
|
func lookupWait(wait time.Duration) string {
|
||||||
|
return strconv.FormatInt(wait.Milliseconds(), 10)
|
||||||
|
}
|
||||||
|
|
||||||
|
// wantAnswer checks the status of an answer and how long it took.
|
||||||
|
func wantAnswer(
|
||||||
|
t *testing.T, status int, took time.Duration,
|
||||||
|
wantStatus int, wantTook time.Duration, out *output,
|
||||||
|
) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
if status != wantStatus || took != wantTook {
|
||||||
|
t.Errorf(
|
||||||
|
"answered %d after %s, want %d after %s; output:\n%s",
|
||||||
|
status, took, wantStatus, wantTook, out.text(),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// wantNoCoreRuleSetTrace checks that what the Core Rule Set found of a request
|
||||||
|
// left no trace: line has none of its fields, and server has counted no match
|
||||||
|
// of it and raised no alert for one.
|
||||||
|
func wantNoCoreRuleSetTrace(
|
||||||
|
t *testing.T, server *waf.Server, queue *alerts.Queue, line logLine,
|
||||||
|
) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
if line.DurationCoreRuleSetMS != nil || line.CoreRuleSetScore != nil ||
|
||||||
|
line.CoreRuleSetRuleIDs != nil {
|
||||||
|
t.Errorf(
|
||||||
|
"log line has Core Rule Set time %v, score %v and rule ids %v, want none",
|
||||||
|
line.DurationCoreRuleSetMS,
|
||||||
|
line.CoreRuleSetScore,
|
||||||
|
line.CoreRuleSetRuleIDs,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
recorder := httptest.NewRecorder()
|
||||||
|
server.Metrics.ServeHTTP(
|
||||||
|
recorder,
|
||||||
|
httptest.NewRequestWithContext(
|
||||||
|
t.Context(),
|
||||||
|
http.MethodGet,
|
||||||
|
waf.MetricsPath,
|
||||||
|
http.NoBody,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
if strings.Contains(
|
||||||
|
recorder.Body.String(),
|
||||||
|
"sww_core_rule_set_matches_total{",
|
||||||
|
) {
|
||||||
|
t.Errorf("a Core Rule Set match is counted")
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, alert := range queue.Snapshot().Waiting[alerts.DestinationWebhook] {
|
||||||
|
if alert.Event == alerts.EventCoreRuleSetMatch {
|
||||||
|
t.Errorf("alert raised: %+v", alert)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTheLookupAndTheCoreRuleSetTakeTheLongerOfTheirTimesNotTheirSum(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
inspection := checksTimeLimit - time.Millisecond
|
||||||
|
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
// wait is how long the request waits for GeoJS.
|
||||||
|
wait time.Duration
|
||||||
|
}{
|
||||||
|
{"the lookup waits longer", defaultLookupWait},
|
||||||
|
{"the inspection takes longer", inspection / 2},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
crs := &heldCoreRuleSet{took: inspection}
|
||||||
|
server, out, _ := newWAFHoldingTheCoreRuleSet(
|
||||||
|
t,
|
||||||
|
crs,
|
||||||
|
map[string]string{
|
||||||
|
asnAndCountryLookupTimeout: lookupWait(tc.wait),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
defer waitForGeoJSRequestsToBeAbandoned()
|
||||||
|
|
||||||
|
status, took := askDirectly(
|
||||||
|
t,
|
||||||
|
server,
|
||||||
|
func(*http.Request) {},
|
||||||
|
)
|
||||||
|
|
||||||
|
wantAnswer(t, status, took, http.StatusOK,
|
||||||
|
max(tc.wait, inspection), out)
|
||||||
|
wantLine(t, out.requestLine(t), http.StatusOK,
|
||||||
|
requestlog.ActionForward)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// firstToRefuse is a case of
|
||||||
|
// TestTheFirstOfTheLookupAndTheRequestChecksToRefuseIsAnswered: a request
|
||||||
|
// under env, whose inspection by the Core Rule Set takes inspection, refused as
|
||||||
|
// action after took.
|
||||||
|
type firstToRefuse struct {
|
||||||
|
name string
|
||||||
|
env map[string]string
|
||||||
|
inspection time.Duration
|
||||||
|
took time.Duration
|
||||||
|
action string
|
||||||
|
}
|
||||||
|
|
||||||
|
// firstToRefuseCases are the cases of
|
||||||
|
// TestTheFirstOfTheLookupAndTheRequestChecksToRefuseIsAnswered.
|
||||||
|
func firstToRefuseCases() []firstToRefuse {
|
||||||
|
return []firstToRefuse{
|
||||||
|
{
|
||||||
|
name: "a trap path, before GeoJS answers",
|
||||||
|
env: map[string]string{trapPaths: "/", allowedCountries: "DE"},
|
||||||
|
took: 0, action: requestlog.ActionTemporarilyBanned,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "the Core Rule Set, before GeoJS answers",
|
||||||
|
env: map[string]string{allowedCountries: "DE"},
|
||||||
|
inspection: 100 * time.Millisecond,
|
||||||
|
took: 100 * time.Millisecond, action: requestlog.ActionBlockedByCoreRuleSet,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "the country list, while the Core Rule Set inspects",
|
||||||
|
env: map[string]string{
|
||||||
|
allowedCountries: "DE", asnAndCountryLookupTimeout: "100",
|
||||||
|
},
|
||||||
|
inspection: 400 * time.Millisecond,
|
||||||
|
took: 100 * time.Millisecond, action: requestlog.ActionCountryDenied,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "the rate limits, while the Core Rule Set inspects",
|
||||||
|
env: map[string]string{
|
||||||
|
rateLimitPerMinute: "2", unknownCountryLimitPercent: "50",
|
||||||
|
asnAndCountryLookupTimeout: "100",
|
||||||
|
},
|
||||||
|
inspection: 400 * time.Millisecond,
|
||||||
|
took: 100 * time.Millisecond, action: requestlog.ActionClientIsRateLimited,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTheFirstOfTheLookupAndTheRequestChecksToRefuseIsAnswered(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const forbidden = http.StatusForbidden
|
||||||
|
|
||||||
|
for _, tc := range firstToRefuseCases() {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
crs := &heldCoreRuleSet{
|
||||||
|
took: tc.inspection,
|
||||||
|
result: coreRuleSetMatch(),
|
||||||
|
}
|
||||||
|
server, out, queue := newWAFHoldingTheCoreRuleSet(t, crs, tc.env)
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
defer waitForGeoJSRequestsToBeAbandoned()
|
||||||
|
|
||||||
|
// Under the rate limits of the last case, the client's next
|
||||||
|
// request takes it over a limit only at the percentage of an
|
||||||
|
// unknown country, so it waits for GeoJS.
|
||||||
|
countClient(t, server, time.Now())
|
||||||
|
|
||||||
|
status, took := askDirectly(
|
||||||
|
t,
|
||||||
|
server,
|
||||||
|
func(*http.Request) {},
|
||||||
|
)
|
||||||
|
|
||||||
|
wantAnswer(t, status, took, forbidden, tc.took, out)
|
||||||
|
|
||||||
|
line := out.requestLine(t)
|
||||||
|
wantLine(t, line, forbidden, tc.action)
|
||||||
|
|
||||||
|
if tc.action == requestlog.ActionBlockedByCoreRuleSet {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if tc.action != requestlog.ActionClientIsRateLimited &&
|
||||||
|
line.Counts != (ratelimit.Counts{}) {
|
||||||
|
t.Errorf(
|
||||||
|
"the rate limits counted the request: %+v",
|
||||||
|
line.Counts,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// What the Core Rule Set finds once the request is answered is
|
||||||
|
// dropped.
|
||||||
|
time.Sleep(tc.inspection)
|
||||||
|
wantNoCoreRuleSetTrace(t, server, queue, line)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestARequestRefusedBeforeTheLookupStartsNoInspection(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
crs := &heldCoreRuleSet{took: time.Second}
|
||||||
|
server, out, _ := newWAFHoldingTheCoreRuleSet(t, crs, nil)
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
banClient(t, server, time.Now())
|
||||||
|
|
||||||
|
status, took := askDirectly(t, server, func(*http.Request) {})
|
||||||
|
|
||||||
|
wantAnswer(t, status, took, http.StatusForbidden, 0, out)
|
||||||
|
|
||||||
|
requests := testutil.ToFloat64(server.Metrics.GeoJSRequests)
|
||||||
|
if asked := crs.asked.Load(); asked != 0 || requests != 0 {
|
||||||
|
t.Errorf(
|
||||||
|
"%d inspections and %v requests to GeoJS, want none",
|
||||||
|
asked,
|
||||||
|
requests,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAClientInTheAllowNetsIsAllowedAtOnceAndNothingElseStarts(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
crs := &heldCoreRuleSet{took: time.Second, result: coreRuleSetMatch()}
|
||||||
|
server, out, _ := newWAFHoldingTheCoreRuleSet(t, crs,
|
||||||
|
map[string]string{allowNets: client, allowedCountries: "DE"})
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
status, took := askDirectly(t, server, func(*http.Request) {})
|
||||||
|
|
||||||
|
wantAnswer(t, status, took, http.StatusOK, 0, out)
|
||||||
|
|
||||||
|
requests := testutil.ToFloat64(server.Metrics.GeoJSRequests)
|
||||||
|
if asked := crs.asked.Load(); asked != 0 || requests != 0 {
|
||||||
|
t.Errorf(
|
||||||
|
"%d inspections and %v requests to GeoJS, want none",
|
||||||
|
asked,
|
||||||
|
requests,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestARequestTheCoreRuleSetHasNotFinishedWithIn500MillisecondsIsRefused(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
crs := &heldCoreRuleSet{took: checksTimeLimit + time.Second}
|
||||||
|
server, out, _ := newWAFHoldingTheCoreRuleSet(t, crs,
|
||||||
|
map[string]string{refusalsLimitPerMinute: "1"})
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
defer waitForGeoJSRequestsToBeAbandoned()
|
||||||
|
|
||||||
|
// Were it counted toward the refusal limit, the second refusal would
|
||||||
|
// ban the client, and the third request would be refused for the ban.
|
||||||
|
for i := range 3 {
|
||||||
|
status, took := askDirectly(t, server, func(req *http.Request) {
|
||||||
|
req.Header.Set(requestIDHeader, "id-"+strconv.Itoa(i))
|
||||||
|
})
|
||||||
|
|
||||||
|
wantAnswer(t, status, took, checksTimedOut, checksTimeLimit, out)
|
||||||
|
|
||||||
|
line := out.requestLines(t, i+1)[i]
|
||||||
|
wantLine(
|
||||||
|
t,
|
||||||
|
line,
|
||||||
|
checksTimedOut,
|
||||||
|
requestlog.ActionChecksTimedOut,
|
||||||
|
)
|
||||||
|
wantHeuristic(t, line, "core_rule_set", `{"time_limit_ms": 500}`)
|
||||||
|
|
||||||
|
if line.Offence != "" || line.BanExpires != "" {
|
||||||
|
t.Errorf(
|
||||||
|
"log line has offence %q and ban_expires %q, want neither",
|
||||||
|
line.Offence,
|
||||||
|
line.BanExpires,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if bans := server.Ledger.Snapshot(); len(bans) != 0 {
|
||||||
|
t.Errorf("bans %+v, want none", bans)
|
||||||
|
}
|
||||||
|
|
||||||
|
offences := historyOf(t, server, client).Offences
|
||||||
|
if offences != (ratelimit.Offences{}) {
|
||||||
|
t.Errorf("history counts the offences %+v, want none", offences)
|
||||||
|
}
|
||||||
|
|
||||||
|
time.Sleep(crs.took)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestObserveModeAllowsARequestTheCoreRuleSetHasNotFinishedWith(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
crs := &heldCoreRuleSet{took: checksTimeLimit + time.Second}
|
||||||
|
server, out, _ := newWAFHoldingTheCoreRuleSet(t, crs,
|
||||||
|
map[string]string{mode: observe})
|
||||||
|
|
||||||
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
defer waitForGeoJSRequestsToBeAbandoned()
|
||||||
|
|
||||||
|
status, took := askDirectly(t, server, func(*http.Request) {})
|
||||||
|
|
||||||
|
wantAnswer(t, status, took, http.StatusOK, checksTimeLimit, out)
|
||||||
|
|
||||||
|
line := out.requestLine(t)
|
||||||
|
wantLine(t, line, http.StatusOK, requestlog.ActionForward)
|
||||||
|
|
||||||
|
if line.WouldAction != requestlog.ActionChecksTimedOut {
|
||||||
|
t.Errorf(
|
||||||
|
"log line has would_action %q, want %q",
|
||||||
|
line.WouldAction,
|
||||||
|
requestlog.ActionChecksTimedOut,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
time.Sleep(crs.took)
|
||||||
|
})
|
||||||
|
}
|
||||||
+10
-9
@@ -53,33 +53,34 @@ func (rq *request) limitPercentages() (percentage, percentage) {
|
|||||||
if country == "" && !rq.skippedLookupWait {
|
if country == "" && !rq.skippedLookupWait {
|
||||||
unknown = percentage{
|
unknown = percentage{
|
||||||
cfg.UnknownCountryLimitPercent,
|
cfg.UnknownCountryLimitPercent,
|
||||||
"SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT",
|
config.UnknownCountryLimitPercentSetting,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fetched := percentage{percent: whole}
|
fetched := percentage{percent: whole}
|
||||||
if percent, listed := rq.h.lists.ASNLimitPercent(asn); listed {
|
if percent, listed := rq.h.lists.ASNLimitPercent(asn); listed {
|
||||||
fetched = percentage{percent, "SWW_WAF_ASN_LIMIT_PERCENT_URL"}
|
fetched = percentage{percent, config.ASNLimitPercentURLSetting}
|
||||||
}
|
}
|
||||||
|
|
||||||
blocklisted := percentage{percent: whole}
|
blocklisted := percentage{percent: whole}
|
||||||
if rq.blocklisted && cfg.BlocklistAction == "limit" {
|
if rq.blocklisted &&
|
||||||
|
cfg.BlocklistAction == config.ReputationActionLimit {
|
||||||
blocklisted = percentage{
|
blocklisted = percentage{
|
||||||
cfg.BlocklistLimitPercent,
|
cfg.BlocklistLimitPercent,
|
||||||
"SWW_WAF_BLOCKLIST_ACTION",
|
config.BlocklistActionSetting,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
dnsblAbuseIPDBListed := percentage{percent: whole}
|
dnsblAbuseIPDBListed := percentage{percent: whole}
|
||||||
if (rq.dnsblListed || rq.abuseIPDBHit) &&
|
if (rq.dnsblListed || rq.abuseIPDBHit) &&
|
||||||
cfg.DNSBLAbuseIPDBAction == "limit" {
|
cfg.DNSBLAbuseIPDBAction == config.ReputationActionLimit {
|
||||||
dnsblAbuseIPDBListed = percentage{
|
dnsblAbuseIPDBListed = percentage{
|
||||||
cfg.DNSBLAbuseIPDBLimitPercent,
|
cfg.DNSBLAbuseIPDBLimitPercent,
|
||||||
"SWW_WAF_DNSBL_ABUSEIPDB_ACTION",
|
config.DNSBLAbuseIPDBActionSetting,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
const tagSetting = "SWW_WAF_TAG_LIMIT_PERCENT"
|
const tagSetting = config.TagLimitPercentSetting
|
||||||
|
|
||||||
asnRequests := lowest(
|
asnRequests := lowest(
|
||||||
given(cfg.TagLimitPercent, asnTag(asn), tagSetting),
|
given(cfg.TagLimitPercent, asnTag(asn), tagSetting),
|
||||||
@@ -107,7 +108,7 @@ func (rq *request) limitPercentages() (percentage, percentage) {
|
|||||||
asnBytes = given(
|
asnBytes = given(
|
||||||
cfg.ASNBytesLimitPercent,
|
cfg.ASNBytesLimitPercent,
|
||||||
asn,
|
asn,
|
||||||
"SWW_WAF_ASN_BYTES_LIMIT_PERCENT",
|
config.ASNBytesLimitPercentSetting,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -115,7 +116,7 @@ func (rq *request) limitPercentages() (percentage, percentage) {
|
|||||||
countryBytes = given(
|
countryBytes = given(
|
||||||
cfg.CountryBytesLimitPercent,
|
cfg.CountryBytesLimitPercent,
|
||||||
country,
|
country,
|
||||||
"SWW_WAF_COUNTRY_BYTES_LIMIT_PERCENT",
|
config.CountryBytesLimitPercentSetting,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -339,6 +339,8 @@ func TestClientWithoutAnAnswerInTimeHasTheUnknownCountryLimitPercent(
|
|||||||
// TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays says, with
|
// TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays says, with
|
||||||
// a GeoJS that never answers.
|
// a GeoJS that never answers.
|
||||||
synctest.Test(t, func(t *testing.T) {
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
defer waitForGeoJSRequestsToBeAbandoned()
|
||||||
|
|
||||||
server, out, _ := newWAF(t, silentGeoJSURL, time.Now,
|
server, out, _ := newWAF(t, silentGeoJSURL, time.Now,
|
||||||
map[string]string{unknownCountryLimitPercent: "0"})
|
map[string]string{unknownCountryLimitPercent: "0"})
|
||||||
|
|
||||||
@@ -387,14 +389,18 @@ func TestRequestWaitsForItsLookupWhileABiasedThresholdIsSet(t *testing.T) {
|
|||||||
|
|
||||||
// In a synctest bubble, as
|
// In a synctest bubble, as
|
||||||
// TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays says,
|
// TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays says,
|
||||||
// with a GeoJS that never answers.
|
// with a GeoJS that never answers. The headers, which need the
|
||||||
|
// answer too, are off.
|
||||||
synctest.Test(t, func(t *testing.T) {
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
defer waitForGeoJSRequestsToBeAbandoned()
|
||||||
|
|
||||||
server, out, _ := newWAF(t, silentGeoJSURL, time.Now,
|
server, out, _ := newWAF(t, silentGeoJSURL, time.Now,
|
||||||
map[string]string{
|
map[string]string{
|
||||||
asnAndCountryLookupTimeout: strconv.FormatInt(
|
asnAndCountryLookupTimeout: strconv.FormatInt(
|
||||||
timeout.Milliseconds(), 10),
|
timeout.Milliseconds(), 10),
|
||||||
tagLists: "vpn=" + vpnList,
|
addLookupHeaders: falseValue,
|
||||||
tc.setting: tc.value,
|
tagLists: "vpn=" + vpnList,
|
||||||
|
tc.setting: tc.value,
|
||||||
})
|
})
|
||||||
began := time.Now()
|
began := time.Now()
|
||||||
|
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -41,5 +42,6 @@ func (blocklists) check(
|
|||||||
// client, given listedBy, the blocklists that list it: it does when one lists
|
// client, given listedBy, the blocklists that list it: it does when one lists
|
||||||
// it and SWW_WAF_BLOCKLIST_ACTION is deny.
|
// it and SWW_WAF_BLOCKLIST_ACTION is deny.
|
||||||
func (rq *request) refusedByBlocklists(listedBy []string) bool {
|
func (rq *request) refusedByBlocklists(listedBy []string) bool {
|
||||||
return len(listedBy) > 0 && rq.h.config.BlocklistAction == deny
|
return len(listedBy) > 0 && rq.h.config.BlocklistAction ==
|
||||||
|
config.ReputationActionDeny
|
||||||
}
|
}
|
||||||
@@ -30,7 +30,7 @@ func (byteLimits) check(
|
|||||||
}
|
}
|
||||||
|
|
||||||
counts, hit, over := rq.h.limiter.CountBytes(
|
counts, hit, over := rq.h.limiter.CountBytes(
|
||||||
rq.h.clientGroup(rq.client),
|
clientGroup(rq.h.config, rq.client),
|
||||||
now,
|
now,
|
||||||
rq.countedBytes(),
|
rq.countedBytes(),
|
||||||
rq.bytesLimitPercent.percent,
|
rq.bytesLimitPercent.percent,
|
||||||
|
|||||||
@@ -5,6 +5,8 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"slices"
|
"slices"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
)
|
)
|
||||||
|
|
||||||
// peerAddress is the address of the request's TCP peer.
|
// peerAddress is the address of the request's TCP peer.
|
||||||
@@ -45,10 +47,10 @@ func scheme(r *http.Request, isQuestion bool) string {
|
|||||||
// address, or its IPv6 group, the netblock its IPv6 address is in of the
|
// address, or its IPv6 group, the netblock its IPv6 address is in of the
|
||||||
// length SWW_WAF_IPV6_GROUP_PREFIX sets, a /64 by default, since one abuser
|
// length SWW_WAF_IPV6_GROUP_PREFIX sets, a /64 by default, since one abuser
|
||||||
// usually holds a whole /64. An IPv4 address in IPv6 form counts as IPv4.
|
// usually holds a whole /64. An IPv4 address in IPv6 form counts as IPv4.
|
||||||
func (h *handler) clientGroup(addr netip.Addr) netip.Prefix {
|
func clientGroup(cfg *config.WAF, addr netip.Addr) netip.Prefix {
|
||||||
addr = addr.Unmap()
|
addr = addr.Unmap()
|
||||||
if addr.Is6() {
|
if addr.Is6() {
|
||||||
return netip.PrefixFrom(addr, h.config.IPv6GroupPrefix).Masked()
|
return netip.PrefixFrom(addr, cfg.IPv6GroupPrefix).Masked()
|
||||||
}
|
}
|
||||||
|
|
||||||
return netip.PrefixFrom(addr, addr.BitLen())
|
return netip.PrefixFrom(addr, addr.BitLen())
|
||||||
|
|||||||
+26
-18
@@ -13,10 +13,11 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
)
|
)
|
||||||
|
|
||||||
// coreRuleSet inspects the request with the Core Rule Set, unless
|
// coreRuleSet takes what the Core Rule Set found of the request, which
|
||||||
// SWW_WAF_CORE_RULE_SET_MODE is off or SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS
|
// findBesideLookup inspected with it unless SWW_WAF_CORE_RULE_SET_MODE is off
|
||||||
// exempts its path, as pathExempt decides, and notes the rules it matched and
|
// or SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS exempts its path, as pathExempt
|
||||||
// its score in the log line, and the rules in the metrics. A score at or over
|
// decides. It notes the rules the request matched and its score in the log
|
||||||
|
// line, and the rules in the metrics. A score at or over
|
||||||
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD is a match: it adds the tag
|
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD is a match: it adds the tag
|
||||||
// exploitsignature, raises the core_rule_set_match alert, and in block mode
|
// exploitsignature, raises the core_rule_set_match alert, and in block mode
|
||||||
// refuses the request with 403, which is an offence its client's history
|
// refuses the request with 403, which is an offence its client's history
|
||||||
@@ -30,27 +31,23 @@ func (coreRuleSet) check(
|
|||||||
rq *request,
|
rq *request,
|
||||||
_ time.Time,
|
_ time.Time,
|
||||||
) *verdict {
|
) *verdict {
|
||||||
cfg := rq.h.config
|
findings := rq.findings
|
||||||
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff ||
|
if !findings.inspected {
|
||||||
pathExempt(rq.in, cfg.CoreRuleSetExemptPaths) {
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
start := time.Now()
|
cfg := rq.h.config
|
||||||
|
result := findings.inspection
|
||||||
// A question's body has been read into memory, so that reading it here
|
rq.requestBytes = int64(len(findings.read))
|
||||||
// cannot fail.
|
|
||||||
result, read, _ := rq.h.coreRuleSet.Inspect(rq.in, rq.client, rq.in.Body)
|
|
||||||
rq.requestBytes = int64(len(read))
|
|
||||||
|
|
||||||
rq.line.DurationCoreRuleSetMS = new(
|
rq.line.DurationCoreRuleSetMS = new(
|
||||||
requestlog.Milliseconds(time.Since(start)),
|
requestlog.Milliseconds(findings.inspectionTime),
|
||||||
)
|
)
|
||||||
rq.line.CoreRuleSetRuleIDs = result.RuleIDs
|
rq.line.CoreRuleSetRuleIDs = result.RuleIDs
|
||||||
rq.line.CoreRuleSetScore = &result.Score
|
rq.line.CoreRuleSetScore = &result.Score
|
||||||
|
|
||||||
for _, id := range result.RuleIDs {
|
for _, id := range result.RuleIDs {
|
||||||
rq.h.metrics.CoreRuleSetMatched(cfg.CoreRuleSetMode, id)
|
rq.h.metrics.CoreRuleSetMatched(string(cfg.CoreRuleSetMode), id)
|
||||||
}
|
}
|
||||||
|
|
||||||
threshold := cfg.CoreRuleSetScoreThreshold
|
threshold := cfg.CoreRuleSetScoreThreshold
|
||||||
@@ -61,7 +58,7 @@ func (coreRuleSet) check(
|
|||||||
rq.addTag(config.ExploitSignatureTag)
|
rq.addTag(config.ExploitSignatureTag)
|
||||||
rq.alertCoreRuleSetMatch(result)
|
rq.alertCoreRuleSetMatch(result)
|
||||||
|
|
||||||
if cfg.CoreRuleSetMode == config.CoreRuleSetModeDetect {
|
if !rq.refusedByCoreRuleSet(result) {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -76,6 +73,17 @@ func (coreRuleSet) check(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// refusedByCoreRuleSet reports whether coreRuleSet refuses a request the Core
|
||||||
|
// Rule Set found result of: whether the score is at or over
|
||||||
|
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD in block mode.
|
||||||
|
func (rq *request) refusedByCoreRuleSet(result coreruleset.Result) bool {
|
||||||
|
cfg := rq.h.config
|
||||||
|
threshold := cfg.CoreRuleSetScoreThreshold
|
||||||
|
|
||||||
|
return cfg.CoreRuleSetMode == config.CoreRuleSetModeBlock &&
|
||||||
|
threshold != 0 && result.Score >= threshold
|
||||||
|
}
|
||||||
|
|
||||||
// alertCoreRuleSetMatch raises the core_rule_set_match alert for the request,
|
// alertCoreRuleSetMatch raises the core_rule_set_match alert for the request,
|
||||||
// which the Core Rule Set scored at result, at or over
|
// which the Core Rule Set scored at result, at or over
|
||||||
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD. Its detail gives the rule ids, the
|
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD. Its detail gives the rule ids, the
|
||||||
@@ -91,7 +99,7 @@ func (rq *request) alertCoreRuleSetMatch(result coreruleset.Result) {
|
|||||||
|
|
||||||
switch {
|
switch {
|
||||||
case rq.h.config.CoreRuleSetMode == config.CoreRuleSetModeDetect:
|
case rq.h.config.CoreRuleSetMode == config.CoreRuleSetModeDetect:
|
||||||
detail["mode"] = config.CoreRuleSetModeDetect
|
detail["mode"] = string(config.CoreRuleSetModeDetect)
|
||||||
case rq.h.config.Observe:
|
case rq.h.config.Observe:
|
||||||
detail["mode"] = "observe"
|
detail["mode"] = "observe"
|
||||||
}
|
}
|
||||||
@@ -100,7 +108,7 @@ func (rq *request) alertCoreRuleSetMatch(result coreruleset.Result) {
|
|||||||
Event: alerts.EventCoreRuleSetMatch,
|
Event: alerts.EventCoreRuleSetMatch,
|
||||||
Host: rq.site,
|
Host: rq.site,
|
||||||
Client: rq.client,
|
Client: rq.client,
|
||||||
Netblock: rq.h.clientGroup(rq.client),
|
Netblock: clientGroup(rq.h.config, rq.client),
|
||||||
ASN: rq.line.ASN,
|
ASN: rq.line.ASN,
|
||||||
ASName: rq.line.ASName,
|
ASName: rq.line.ASName,
|
||||||
Country: rq.line.Country,
|
Country: rq.line.Country,
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ import (
|
|||||||
"slices"
|
"slices"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -33,7 +34,8 @@ func (countries) check(
|
|||||||
return &verdict{
|
return &verdict{
|
||||||
action: requestlog.ActionCountryDenied,
|
action: requestlog.ActionCountryDenied,
|
||||||
figures: map[string]any{
|
figures: map[string]any{
|
||||||
"country": country, "setting": "SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES",
|
"country": country,
|
||||||
|
"setting": config.ExclusivelyAllowedCountriesSetting,
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -108,8 +108,11 @@ func TestCountryRefusalNamesItsHeuristicCountryAndSetting(t *testing.T) {
|
|||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
geojsURL, _ := startGeoJS(t)
|
geojsURL, _ := startGeoJS(t)
|
||||||
|
// The headers are off, so that only the country list makes the request
|
||||||
|
// wait for its answer.
|
||||||
s, _, _ := startWithClock(t, geojsURL, map[string]string{
|
s, _, _ := startWithClock(t, geojsURL, map[string]string{
|
||||||
asnAndCountryLookupTimeout: "3600000", allowedCountries: "de",
|
asnAndCountryLookupTimeout: "3600000", addLookupHeaders: falseValue,
|
||||||
|
allowedCountries: "de",
|
||||||
})
|
})
|
||||||
|
|
||||||
line := s.get(
|
line := s.get(
|
||||||
@@ -239,7 +242,10 @@ func TestPrivateAddressIsNeverLookedUp(t *testing.T) {
|
|||||||
name string
|
name string
|
||||||
env map[string]string
|
env map[string]string
|
||||||
}{
|
}{
|
||||||
{"no setting needs the lookup", nil},
|
{
|
||||||
|
"no setting needs the lookup",
|
||||||
|
map[string]string{addLookupHeaders: falseValue},
|
||||||
|
},
|
||||||
{"a tag setting needs it", map[string]string{tagDeny: denyKP}},
|
{"a tag setting needs it", map[string]string{tagDeny: denyKP}},
|
||||||
} {
|
} {
|
||||||
t.Run(tc.name, func(t *testing.T) {
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
|||||||
@@ -32,7 +32,8 @@ func (dnsbl) check(ctx context.Context, rq *request, _ time.Time) *verdict {
|
|||||||
rq.addTag(config.DNSBLTagPrefix + strings.ToLower(zone))
|
rq.addTag(config.DNSBLTagPrefix + strings.ToLower(zone))
|
||||||
}
|
}
|
||||||
|
|
||||||
if !rq.dnsblListed || rq.h.config.DNSBLAbuseIPDBAction != deny {
|
if !rq.dnsblListed ||
|
||||||
|
rq.h.config.DNSBLAbuseIPDBAction != config.ReputationActionDeny {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,13 @@
|
|||||||
|
package waf
|
||||||
|
|
||||||
|
import "sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
|
|
||||||
|
// BesideLookupTimeLimit is besideLookupTimeLimit.
|
||||||
|
const BesideLookupTimeLimit = besideLookupTimeLimit
|
||||||
|
|
||||||
|
// BanAbuseIPDBBlacklist does on s what a fetch of AbuseIPDB's blacklist with
|
||||||
|
// the clients on listed does.
|
||||||
|
func (s *Server) BanAbuseIPDBBlacklist(listed []reputation.BlacklistEntry) {
|
||||||
|
h := s.Handler.(*handler) //nolint:forcetypeassert // New makes it one
|
||||||
|
BanAbuseIPDBBlacklist(h.config, h.ledger, h.processLog, h.now(), listed)
|
||||||
|
}
|
||||||
@@ -10,16 +10,22 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
)
|
)
|
||||||
|
|
||||||
// The header limits: the most cookies and header fields a request may have.
|
// The header limits: the most cookies and header fields a request may have,
|
||||||
// Real clients never reach them.
|
// and the longest its request line and each header line may be, in bytes, not
|
||||||
|
// counting the line ending. Real clients rarely reach them. maxLineBytes is
|
||||||
|
// what nginx allows by default.
|
||||||
const (
|
const (
|
||||||
maxCookies = 50
|
maxCookies = 50
|
||||||
maxHeaderFields = 100
|
maxHeaderFields = 100
|
||||||
|
maxLineBytes = 8 << 10
|
||||||
)
|
)
|
||||||
|
|
||||||
// headerLimits refuses a request with more than maxHeaderFields header fields
|
// headerLimits refuses a request with a request line longer than maxLineBytes,
|
||||||
// or more than maxCookies cookies, with 431. Its figures are the request's
|
// with 414. It refuses a request with a header line longer than maxLineBytes,
|
||||||
// header fields and cookies, and the limit of each.
|
// more than maxHeaderFields header fields or more than maxCookies cookies, with
|
||||||
|
// 431. Its figures are the request line's length and its limit for 414. For
|
||||||
|
// 431 they are the longest header line, the header fields and the cookies,
|
||||||
|
// each with its limit.
|
||||||
type headerLimits struct{}
|
type headerLimits struct{}
|
||||||
|
|
||||||
func (headerLimits) name() string { return "header_limits" }
|
func (headerLimits) name() string { return "header_limits" }
|
||||||
@@ -29,20 +35,59 @@ func (headerLimits) check(
|
|||||||
rq *request,
|
rq *request,
|
||||||
_ time.Time,
|
_ time.Time,
|
||||||
) *verdict {
|
) *verdict {
|
||||||
|
requestLine := rq.requestLineBytes()
|
||||||
|
if requestLine > maxLineBytes {
|
||||||
|
return &verdict{
|
||||||
|
action: requestlog.ActionRequestLineTooLong,
|
||||||
|
figures: map[string]any{
|
||||||
|
"request_line_bytes": requestLine,
|
||||||
|
"request_line_bytes_limit": maxLineBytes,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
longest := rq.longestHeaderLineBytes()
|
||||||
fields, cookies := rq.headerCounts()
|
fields, cookies := rq.headerCounts()
|
||||||
if fields <= maxHeaderFields && cookies <= maxCookies {
|
|
||||||
|
if longest <= maxLineBytes && fields <= maxHeaderFields &&
|
||||||
|
cookies <= maxCookies {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
return &verdict{
|
return &verdict{
|
||||||
action: requestlog.ActionRequestHeaderTooLarge,
|
action: requestlog.ActionRequestHeaderTooLarge,
|
||||||
figures: map[string]any{
|
figures: map[string]any{
|
||||||
"fields": fields, "fields_limit": maxHeaderFields,
|
"longest_header_line_bytes": longest,
|
||||||
|
"header_line_bytes_limit": maxLineBytes,
|
||||||
|
"fields": fields, "fields_limit": maxHeaderFields,
|
||||||
"cookies": cookies, "cookies_limit": maxCookies,
|
"cookies": cookies, "cookies_limit": maxCookies,
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// requestLineBytes returns the length of the request's request line, such as
|
||||||
|
// "GET /search?q=x HTTP/1.1", not counting its line ending. The target is the
|
||||||
|
// one the visitor sent, which rproxy's question carries unchanged.
|
||||||
|
func (rq *request) requestLineBytes() int {
|
||||||
|
return len(rq.in.Method) + len(" ") + len(rq.in.RequestURI) +
|
||||||
|
len(" ") + len(rq.in.Proto)
|
||||||
|
}
|
||||||
|
|
||||||
|
// longestHeaderLineBytes returns the length of the request's longest header
|
||||||
|
// line, such as "Cookie: a=1", not counting its line ending. Each value of a
|
||||||
|
// header sent twice is a line of its own, and Host is a line too.
|
||||||
|
func (rq *request) longestHeaderLineBytes() int {
|
||||||
|
longest := len("Host: ") + len(rq.in.Host)
|
||||||
|
|
||||||
|
for name, values := range rq.in.Header {
|
||||||
|
for _, value := range values {
|
||||||
|
longest = max(longest, len(name)+len(": ")+len(value))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return longest
|
||||||
|
}
|
||||||
|
|
||||||
// headerCounts returns the request's header fields, each value of a header sent
|
// headerCounts returns the request's header fields, each value of a header sent
|
||||||
// twice being one, and its cookies, each part of a Cookie header between
|
// twice being one, and its cookies, each part of a Cookie header between
|
||||||
// semicolons being one unless it is empty or only spaces and tabs. The Core
|
// semicolons being one unless it is empty or only spaces and tabs. The Core
|
||||||
|
|||||||
+20
-11
@@ -41,8 +41,8 @@ const (
|
|||||||
|
|
||||||
// beforeLookup are the heuristics that run before the client's AS number and
|
// beforeLookup are the heuristics that run before the client's AS number and
|
||||||
// country are looked up, so that a client they refuse is not looked up.
|
// country are looked up, so that a client they refuse is not looked up.
|
||||||
// header_limits comes first: it only counts, and a request it refuses causes no
|
// header_limits comes first: it only measures and counts, and a request it
|
||||||
// other work.
|
// refuses causes no other work.
|
||||||
func beforeLookup() []heuristic {
|
func beforeLookup() []heuristic {
|
||||||
return []heuristic{
|
return []heuristic{
|
||||||
headerLimits{},
|
headerLimits{},
|
||||||
@@ -54,8 +54,6 @@ func beforeLookup() []heuristic {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// afterLookup are the heuristics that run once the lookup has been made.
|
// afterLookup are the heuristics that run once the lookup has been made.
|
||||||
// tag_deny runs twice: before the rate limits, on the tags known by then, and
|
|
||||||
// last, on the tag rule_files and core_rule_set can add.
|
|
||||||
func afterLookup() []heuristic {
|
func afterLookup() []heuristic {
|
||||||
return []heuristic{
|
return []heuristic{
|
||||||
countries{},
|
countries{},
|
||||||
@@ -64,9 +62,26 @@ func afterLookup() []heuristic {
|
|||||||
abuseIPDB{},
|
abuseIPDB{},
|
||||||
tagDeny{},
|
tagDeny{},
|
||||||
rateLimits{},
|
rateLimits{},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// besideLookup are the heuristics that only read the request. Their work runs
|
||||||
|
// beside the lookup and afterLookup, in findBesideLookup, which changes nothing
|
||||||
|
// of the request. Their check then applies what that work found, and runs only
|
||||||
|
// once waf takes it.
|
||||||
|
func besideLookup() []heuristic {
|
||||||
|
return []heuristic{
|
||||||
trapPaths{},
|
trapPaths{},
|
||||||
ruleFiles{},
|
ruleFiles{},
|
||||||
coreRuleSet{},
|
coreRuleSet{},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// afterBesideLookup are the heuristics that run once afterLookup and
|
||||||
|
// besideLookup have both let the request through. tag_deny runs a second time,
|
||||||
|
// on the tag rule_files and core_rule_set can add.
|
||||||
|
func afterBesideLookup() []heuristic {
|
||||||
|
return []heuristic{
|
||||||
tagDeny{},
|
tagDeny{},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -89,13 +104,7 @@ func onReport() []heuristic {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// runAfterAnswer runs the heuristics of afterAnswer on the request, at the time
|
// runHeuristics runs list, one of those checkClient runs, on the request at now,
|
||||||
// it is called.
|
|
||||||
func (rq *request) runAfterAnswer(ctx context.Context) {
|
|
||||||
rq.runBanningHeuristics(ctx, afterAnswer())
|
|
||||||
}
|
|
||||||
|
|
||||||
// runHeuristics runs list, beforeLookup or afterLookup, on the request at now,
|
|
||||||
// and returns the first verdict, or nil when none gives one. The log line's
|
// and returns the first verdict, or nil when none gives one. The log line's
|
||||||
// heuristic and heuristic_figures name the heuristic that gave it, and give its
|
// heuristic and heuristic_figures name the heuristic that gave it, and give its
|
||||||
// figures. ctx is the request's own context.
|
// figures. ctx is the request's own context.
|
||||||
|
|||||||
+108
-31
@@ -2,17 +2,22 @@ package waf
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"log/slog"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
)
|
)
|
||||||
|
|
||||||
// The headers in which an answer of allow gives rproxy the client's AS number
|
// The headers in which an answer of allow gives rproxy the client's AS number
|
||||||
// and country, for the app, while SWW_WAF_ADD_LOOKUP_HEADERS is set. Go writes
|
// and country, for the app, while SWW_WAF_ADD_LOOKUP_HEADERS is true. Go writes
|
||||||
// every header name in this form, as it sends it and as it receives it, so
|
// every header name in this form, as it sends it and as it receives it, so
|
||||||
// X-Client-ASN arrives as X-Client-Asn; header names are not case-sensitive.
|
// X-Client-ASN arrives as X-Client-Asn; header names are not case-sensitive.
|
||||||
const (
|
const (
|
||||||
@@ -20,33 +25,81 @@ const (
|
|||||||
countryHeader = "X-Client-Country"
|
countryHeader = "X-Client-Country"
|
||||||
)
|
)
|
||||||
|
|
||||||
// lookUp looks up the client's AS number and country, in the lookup
|
// lookupResult is what the lookup gave for a request: whether its client was
|
||||||
// database or through GeoJS, and notes them for the log line and as the
|
// looked up, the answer, and whether the request skipped the wait for GeoJS.
|
||||||
// request's asn= and country= tags, unless
|
type lookupResult struct {
|
||||||
// SWW_WAF_LOOKUP_SOURCE is off or the client is on a private, loopback or
|
lookedUp, skippedWait bool
|
||||||
// link-local address, which no lookup can place. The lookup database
|
answer geojs.Answer
|
||||||
// answers at once. With GeoJS, while a setting needs the answer, such as a
|
}
|
||||||
// country list or a biased threshold, a new client's request waits for it,
|
|
||||||
// unless refusedWhateverTheAnswer reports true. ctx is the request's own
|
// startLookUp starts looking up the client's AS number and country, in the
|
||||||
// context.
|
// lookup database or through GeoJS, unless SWW_WAF_LOOKUP_SOURCE is off or the
|
||||||
func (rq *request) lookUp(ctx context.Context, now time.Time) {
|
// client is on a private, loopback or link-local address, which no lookup can
|
||||||
if rq.h.config.LookupSource == "off" || !canBePlaced(rq.client) {
|
// place. It returns the channel the result comes on. A result that needs no
|
||||||
|
// wait is on it when startLookUp returns: the lookup database's, an answer
|
||||||
|
// GeoJS gave earlier, or none. With GeoJS, while a setting needs the answer,
|
||||||
|
// such as a country list or a biased threshold, a new client's request waits
|
||||||
|
// for it up to SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS, unless
|
||||||
|
// refusedWhateverTheAnswer reports true. The wait goes on to its end even once
|
||||||
|
// the request has been answered, so that it ends as it would have for a
|
||||||
|
// request that waited. startLookUp changes nothing of the request. ctx is the
|
||||||
|
// request's own context.
|
||||||
|
func (rq *request) startLookUp(
|
||||||
|
ctx context.Context, now time.Time,
|
||||||
|
) <-chan lookupResult {
|
||||||
|
result := make(chan lookupResult, 1)
|
||||||
|
|
||||||
|
if rq.h.config.LookupSource == config.LookupSourceOff ||
|
||||||
|
!canBePlaced(rq.client) {
|
||||||
|
result <- lookupResult{}
|
||||||
|
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
client := clientGroup(rq.h.config, rq.client)
|
||||||
|
|
||||||
|
if rq.h.config.LookupSource == config.LookupSourceFile {
|
||||||
|
result <- lookupResult{
|
||||||
|
lookedUp: true, answer: rq.h.lookupFile.LookUp(client),
|
||||||
|
}
|
||||||
|
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
answer := rq.h.geojs.LookUpWithoutWaiting(ctx, client)
|
||||||
|
refused := rq.refusedWhateverTheAnswer(now)
|
||||||
|
|
||||||
|
if !answer.Answered.IsZero() || refused || !rq.h.waitForGeoJS {
|
||||||
|
result <- lookupResult{
|
||||||
|
lookedUp: true,
|
||||||
|
skippedWait: refused && answer.Answered.IsZero(),
|
||||||
|
answer: answer,
|
||||||
|
}
|
||||||
|
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
go func() {
|
||||||
|
result <- lookupResult{
|
||||||
|
lookedUp: true,
|
||||||
|
answer: rq.h.geojs.LookUp(context.WithoutCancel(ctx), client,
|
||||||
|
rq.h.config.ASNAndCountryLookupTimeout),
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
// noteLookup notes result for the log line and as the request's asn= and
|
||||||
|
// country= tags.
|
||||||
|
func (rq *request) noteLookup(result lookupResult) {
|
||||||
|
if !result.lookedUp {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
client := rq.h.clientGroup(rq.client)
|
|
||||||
|
|
||||||
switch {
|
|
||||||
case rq.h.config.LookupSource == "file":
|
|
||||||
rq.lookupAnswer = rq.h.lookupFile.LookUp(client)
|
|
||||||
case rq.refusedWhateverTheAnswer(now):
|
|
||||||
rq.lookupAnswer = rq.h.geojs.LookUpWithoutWaiting(ctx, client)
|
|
||||||
rq.skippedLookupWait = rq.lookupAnswer.Answered.IsZero()
|
|
||||||
default:
|
|
||||||
rq.lookupAnswer = rq.h.geojs.LookUp(ctx, client)
|
|
||||||
}
|
|
||||||
|
|
||||||
rq.lookedUp = true
|
rq.lookedUp = true
|
||||||
|
rq.lookupAnswer = result.answer
|
||||||
|
rq.skippedLookupWait = result.skippedWait
|
||||||
rq.line.ASN = rq.lookupAnswer.ASN
|
rq.line.ASN = rq.lookupAnswer.ASN
|
||||||
rq.line.ASName = rq.lookupAnswer.ASName
|
rq.line.ASName = rq.lookupAnswer.ASName
|
||||||
rq.line.Country = rq.lookupAnswer.Country
|
rq.line.Country = rq.lookupAnswer.Country
|
||||||
@@ -80,16 +133,40 @@ func countryTag(country string) string {
|
|||||||
return config.CountryTagPrefix + country
|
return config.CountryTagPrefix + country
|
||||||
}
|
}
|
||||||
|
|
||||||
// addLookup adds answer, an answer about a client from the lookup
|
// AddLookup adds answer, an answer about a client from the lookup database
|
||||||
// database or GeoJS, to the client's history, and to the notes of the bans
|
// or GeoJS, to the client's history in limiter, and to the notes of the bans
|
||||||
// on its netblock that have no AS number, AS name or country yet.
|
// in ledger on its netblock that have no AS number, AS name or country yet.
|
||||||
func (h *handler) addLookup(answer lookup.Answer) {
|
// cfg sets the netblock's size.
|
||||||
h.limiter.AddLookup(answer.Client, answer.Answered,
|
func AddLookup(
|
||||||
|
cfg *config.WAF, limiter *ratelimit.Limiter, ledger *bans.Ledger,
|
||||||
|
answer geojs.Answer,
|
||||||
|
) {
|
||||||
|
limiter.AddLookup(answer.Client, answer.Answered,
|
||||||
answer.ASN, answer.ASName, answer.Country)
|
answer.ASN, answer.ASName, answer.Country)
|
||||||
h.ledger.AddLookup(h.netblock(answer.Client.Addr()),
|
ledger.AddLookup(banNetblock(cfg, answer.Client.Addr()),
|
||||||
answer.ASN, answer.ASName, answer.Country)
|
answer.ASN, answer.ASName, answer.Country)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// GeoJSFailed logs a failure of GeoJS to processLog, raises a
|
||||||
|
// source_failure alert for it on alertQueue and counts it in m.
|
||||||
|
// askingAgainIn is how long GeoJS is left alone after it.
|
||||||
|
func GeoJSFailed(
|
||||||
|
processLog *slog.Logger, alertQueue *alerts.Queue, m *metrics.Metrics,
|
||||||
|
err error, askingAgainIn time.Duration,
|
||||||
|
) {
|
||||||
|
processLog.Warn("asking GeoJS failed",
|
||||||
|
"error", err.Error(), "asking_again_in", askingAgainIn.String())
|
||||||
|
alertQueue.Raise(alerts.Alert{
|
||||||
|
Event: alerts.EventSourceFailure,
|
||||||
|
Reason: "asking GeoJS failed",
|
||||||
|
Detail: map[string]any{
|
||||||
|
"source": "geojs", "error": err.Error(),
|
||||||
|
"asking_again_in": askingAgainIn.String(),
|
||||||
|
},
|
||||||
|
})
|
||||||
|
m.GeoJSFailures.Inc()
|
||||||
|
}
|
||||||
|
|
||||||
// setLookupHeaders sets the headers that give the client's AS number and
|
// setLookupHeaders sets the headers that give the client's AS number and
|
||||||
// country, leaving out one that is unknown.
|
// country, leaving out one that is unknown.
|
||||||
func setLookupHeaders(header http.Header, asn, country string) {
|
func setLookupHeaders(header http.Header, asn, country string) {
|
||||||
|
|||||||
@@ -12,7 +12,7 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
|
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
)
|
)
|
||||||
@@ -36,6 +36,7 @@ func TestEveryClientIsLookedUpWithoutWaitingWhileNoSettingNeedsIt(
|
|||||||
geojsURL, asked, release := startHeldGeoJS(t)
|
geojsURL, asked, release := startHeldGeoJS(t)
|
||||||
s, _, server := startWithClock(t, geojsURL, map[string]string{
|
s, _, server := startWithClock(t, geojsURL, map[string]string{
|
||||||
asnAndCountryLookupTimeout: "3600000",
|
asnAndCountryLookupTimeout: "3600000",
|
||||||
|
addLookupHeaders: falseValue,
|
||||||
rateLimitPerMinute: "1",
|
rateLimitPerMinute: "1",
|
||||||
})
|
})
|
||||||
|
|
||||||
@@ -137,7 +138,7 @@ func TestRequestThatWaitedForItsLookupIsCountedWhenItGoesOn(t *testing.T) {
|
|||||||
// of lookups.json would keep it, while the first request still waits
|
// of lookups.json would keep it, while the first request still waits
|
||||||
// for GeoJS. The second request goes on at once, and is counted.
|
// for GeoJS. The second request goes on at once, and is counted.
|
||||||
clk.advance(2500 * time.Millisecond)
|
clk.advance(2500 * time.Millisecond)
|
||||||
server.GeoJS.Load([]lookup.Answer{{
|
server.GeoJS.Load([]geojs.Answer{{
|
||||||
Client: netip.MustParsePrefix(
|
Client: netip.MustParsePrefix(
|
||||||
fromDE + "/32",
|
fromDE + "/32",
|
||||||
), ASN: asnDE, ASName: asNameDE,
|
), ASN: asnDE, ASName: asNameDE,
|
||||||
@@ -189,7 +190,7 @@ func TestASNumberAndNameInTheLogLineTheHistoryTheBanNotesAndTheAlert(
|
|||||||
clk.Now,
|
clk.Now,
|
||||||
map[string]string{
|
map[string]string{
|
||||||
trustedProxies: trustLocalhost,
|
trustedProxies: trustLocalhost,
|
||||||
alertWebhookURL: "https://alerts.example/sww",
|
alertWebhookURL: alertWebhook,
|
||||||
rateLimitPerMinute: "1",
|
rateLimitPerMinute: "1",
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
@@ -198,7 +199,7 @@ func TestASNumberAndNameInTheLogLineTheHistoryTheBanNotesAndTheAlert(
|
|||||||
// The answer is kept before the requests, so GeoJS is not asked, and
|
// The answer is kept before the requests, so GeoJS is not asked, and
|
||||||
// gives no answer of its own.
|
// gives no answer of its own.
|
||||||
netblock := netip.MustParsePrefix(fromDE + "/32")
|
netblock := netip.MustParsePrefix(fromDE + "/32")
|
||||||
server.GeoJS.Load([]lookup.Answer{{
|
server.GeoJS.Load([]geojs.Answer{{
|
||||||
Client: netblock, ASN: asnDE, ASName: asNameDE, Country: "DE",
|
Client: netblock, ASN: asnDE, ASName: asNameDE, Country: "DE",
|
||||||
Answered: clk.Now(), Used: clk.Now(),
|
Answered: clk.Now(), Used: clk.Now(),
|
||||||
}})
|
}})
|
||||||
@@ -248,7 +249,7 @@ func TestLookupSourceOffLooksNoClientUp(t *testing.T) {
|
|||||||
)
|
)
|
||||||
|
|
||||||
// Even an answer kept from before is not used.
|
// Even an answer kept from before is not used.
|
||||||
server.GeoJS.Load([]lookup.Answer{{
|
server.GeoJS.Load([]geojs.Answer{{
|
||||||
Client: netip.MustParsePrefix(
|
Client: netip.MustParsePrefix(
|
||||||
fromDE + "/32",
|
fromDE + "/32",
|
||||||
), ASN: asnDE, ASName: asNameDE,
|
), ASN: asnDE, ASName: asNameDE,
|
||||||
@@ -418,7 +419,7 @@ func TestClientsOwnLookupHeadersAreRemovedWhileTheSettingIsOff(
|
|||||||
geojsURL,
|
geojsURL,
|
||||||
time.Now,
|
time.Now,
|
||||||
map[string]string{
|
map[string]string{
|
||||||
trustedProxies: trustLocalhost,
|
trustedProxies: trustLocalhost, addLookupHeaders: falseValue,
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
s := &sender{t: t, addr: addr, out: out}
|
s := &sender{t: t, addr: addr, out: out}
|
||||||
@@ -449,6 +450,8 @@ func TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays(
|
|||||||
// network, which would keep that clock from moving on: the request is
|
// network, which would keep that clock from moving on: the request is
|
||||||
// handed to waf's handler, and GeoJS is one that never answers.
|
// handed to waf's handler, and GeoJS is one that never answers.
|
||||||
synctest.Test(t, func(t *testing.T) {
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
defer waitForGeoJSRequestsToBeAbandoned()
|
||||||
|
|
||||||
// Not the default. The exclusive list needs the answer, and the app
|
// Not the default. The exclusive list needs the answer, and the app
|
||||||
// is never reached.
|
// is never reached.
|
||||||
const timeout = 3 * time.Second
|
const timeout = 3 * time.Second
|
||||||
@@ -457,6 +460,8 @@ func TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays(
|
|||||||
map[string]string{
|
map[string]string{
|
||||||
asnAndCountryLookupTimeout: strconv.FormatInt(
|
asnAndCountryLookupTimeout: strconv.FormatInt(
|
||||||
timeout.Milliseconds(), 10),
|
timeout.Milliseconds(), 10),
|
||||||
|
// Off, so that only the country list makes the request wait.
|
||||||
|
addLookupHeaders: falseValue,
|
||||||
allowedCountries: "DE",
|
allowedCountries: "DE",
|
||||||
})
|
})
|
||||||
|
|
||||||
@@ -492,6 +497,17 @@ func TestMain(m *testing.M) {
|
|||||||
m.Run()
|
m.Run()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// geojsRequestTimeout is how long a request to GeoJS may take before it is
|
||||||
|
// abandoned.
|
||||||
|
const geojsRequestTimeout = 10 * time.Second
|
||||||
|
|
||||||
|
// waitForGeoJSRequestsToBeAbandoned sleeps until every request to the GeoJS at
|
||||||
|
// silentGeoJSURL made so far has been abandoned. A synctest bubble that asks
|
||||||
|
// it defers this: the bubble cannot end while a request waits on it.
|
||||||
|
func waitForGeoJSRequestsToBeAbandoned() {
|
||||||
|
time.Sleep(geojsRequestTimeout)
|
||||||
|
}
|
||||||
|
|
||||||
// silentGeoJS is the GeoJS at silentGeoJSURL.
|
// silentGeoJS is the GeoJS at silentGeoJSURL.
|
||||||
type silentGeoJS struct{}
|
type silentGeoJS struct{}
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
package waf_test
|
package waf_test
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
@@ -10,8 +11,9 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||||
)
|
)
|
||||||
@@ -248,7 +250,7 @@ func TestMetricsByCountryKeepTheBusiestAndCountTheRestAsOther(t *testing.T) {
|
|||||||
|
|
||||||
// The answers are kept before the requests, so that none waits for
|
// The answers are kept before the requests, so that none waits for
|
||||||
// GeoJS.
|
// GeoJS.
|
||||||
server.GeoJS.Load([]lookup.Answer{
|
server.GeoJS.Load([]geojs.Answer{
|
||||||
keptAnswer(fromKP, "KP"), keptAnswer(fromDE, "DE"),
|
keptAnswer(fromKP, "KP"), keptAnswer(fromDE, "DE"),
|
||||||
keptAnswer(fromFR, "FR"),
|
keptAnswer(fromFR, "FR"),
|
||||||
})
|
})
|
||||||
@@ -329,13 +331,13 @@ func TestMetricsByASNumberKeepTheBusiestAndCountTheRestAsOther(
|
|||||||
|
|
||||||
// The answers are kept before the requests, so that GeoJS gives none
|
// The answers are kept before the requests, so that GeoJS gives none
|
||||||
// of its own. Each client is in an AS of its own.
|
// of its own. Each client is in an AS of its own.
|
||||||
answer := func(addr, asn string) lookup.Answer {
|
answer := func(addr, asn string) geojs.Answer {
|
||||||
return lookup.Answer{
|
return geojs.Answer{
|
||||||
Client: netip.MustParsePrefix(addr + "/32"), ASN: asn,
|
Client: netip.MustParsePrefix(addr + "/32"), ASN: asn,
|
||||||
Answered: clk.Now(), Used: clk.Now(),
|
Answered: clk.Now(), Used: clk.Now(),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
server.GeoJS.Load([]lookup.Answer{
|
server.GeoJS.Load([]geojs.Answer{
|
||||||
answer(fromDE, "AS64501"), answer(fromKP, "AS64502"),
|
answer(fromDE, "AS64501"), answer(fromKP, "AS64502"),
|
||||||
})
|
})
|
||||||
|
|
||||||
@@ -357,21 +359,23 @@ func TestMetricsByASNumberKeepTheBusiestAndCountTheRestAsOther(
|
|||||||
`sww_asn_requests_total{asn="AS64502",instance="app"}`)
|
`sww_asn_requests_total{asn="AS64502",instance="app"}`)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestMetricsCountGeoJSRequestsAndFailures(t *testing.T) {
|
func TestGeoJSFailureIsCountedLoggedAndAlerted(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
geojs := httptest.NewServer(http.HandlerFunc(
|
failingGeoJS := httptest.NewServer(http.HandlerFunc(
|
||||||
func(w http.ResponseWriter, _ *http.Request) {
|
func(w http.ResponseWriter, _ *http.Request) {
|
||||||
w.WriteHeader(http.StatusServiceUnavailable)
|
w.WriteHeader(http.StatusServiceUnavailable)
|
||||||
}))
|
}))
|
||||||
t.Cleanup(geojs.Close)
|
t.Cleanup(failingGeoJS.Close)
|
||||||
|
|
||||||
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
|
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
|
||||||
addr, _, server := startProxyWithClock(t, app.URL, geojs.URL, time.Now,
|
addr, out, server, queue := startProxyWithAlerts(t, app.URL,
|
||||||
|
failingGeoJS.URL, time.Now,
|
||||||
map[string]string{
|
map[string]string{
|
||||||
trustedProxies: trustLocalhost,
|
trustedProxies: trustLocalhost,
|
||||||
metricsToken: token,
|
metricsToken: token,
|
||||||
tagDeny: denyKP,
|
tagDeny: denyKP,
|
||||||
|
alertWebhookURL: "https://alerts.example/waf",
|
||||||
})
|
})
|
||||||
|
|
||||||
// GeoJS fails, so the client has no country= tag, and SWW_WAF_TAG_DENY does
|
// GeoJS fails, so the client has no country= tag, and SWW_WAF_TAG_DENY does
|
||||||
@@ -401,14 +405,46 @@ func TestMetricsCountGeoJSRequestsAndFailures(t *testing.T) {
|
|||||||
`sww_requests_without_geojs_answer_total{instance="app"}`,
|
`sww_requests_without_geojs_answer_total{instance="app"}`,
|
||||||
1,
|
1,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// The process log's line names the failure but not the address asked
|
||||||
|
// about.
|
||||||
|
logged := ""
|
||||||
|
|
||||||
|
for line := range strings.Lines(out.text()) {
|
||||||
|
if strings.Contains(line, "asking GeoJS failed") {
|
||||||
|
logged = line
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if logged == "" || strings.Contains(logged, fromDE) {
|
||||||
|
t.Errorf(
|
||||||
|
"logged %q, want the failure without the address asked about",
|
||||||
|
logged,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// fmt prints a map sorted by its keys.
|
||||||
|
const want = "map[asking_again_in:1s " +
|
||||||
|
"error:GeoJS answered 503 Service Unavailable source:geojs]"
|
||||||
|
|
||||||
|
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
|
||||||
|
if len(waiting) != 1 || waiting[0].Event != alerts.EventSourceFailure ||
|
||||||
|
waiting[0].Reason != "asking GeoJS failed" ||
|
||||||
|
fmt.Sprint(waiting[0].Detail) != want {
|
||||||
|
t.Errorf(
|
||||||
|
"alerts waiting %+v, want one source_failure alert with %v",
|
||||||
|
waiting,
|
||||||
|
want,
|
||||||
|
)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// keptAnswer returns GeoJS's answer that the client at addr is in
|
// keptAnswer returns GeoJS's answer that the client at addr is in
|
||||||
// country, given now.
|
// country, given now.
|
||||||
func keptAnswer(addr, country string) lookup.Answer {
|
func keptAnswer(addr, country string) geojs.Answer {
|
||||||
now := time.Now()
|
now := time.Now()
|
||||||
|
|
||||||
return lookup.Answer{
|
return geojs.Answer{
|
||||||
Client: netip.MustParsePrefix(addr + "/32"), Country: country,
|
Client: netip.MustParsePrefix(addr + "/32"), Country: country,
|
||||||
Answered: now, Used: now,
|
Answered: now, Used: now,
|
||||||
}
|
}
|
||||||
|
|||||||
+76
-161
@@ -1,13 +1,15 @@
|
|||||||
// Package waf is the server the waf command runs. It answers
|
// Package waf is the server the waf command runs. It answers
|
||||||
// rproxy's questions, takes rproxy's reports and answers its own endpoints, and
|
// rproxy's questions, takes rproxy's reports and answers its own endpoints, and
|
||||||
// writes a log line for each.
|
// writes a log line for each. It also bans the clients on AbuseIPDB's
|
||||||
|
// blacklist each time the reputation package fetches it.
|
||||||
package waf
|
package waf
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"fmt"
|
"cmp"
|
||||||
"io"
|
"io"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"net/http"
|
"net/http"
|
||||||
|
"net/netip"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
lru "github.com/hashicorp/golang-lru/v2"
|
lru "github.com/hashicorp/golang-lru/v2"
|
||||||
@@ -17,6 +19,7 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
@@ -54,18 +57,34 @@ type Params struct {
|
|||||||
RequestLog io.Writer
|
RequestLog io.Writer
|
||||||
// ProcessLog receives the process's own messages.
|
// ProcessLog receives the process's own messages.
|
||||||
ProcessLog *slog.Logger
|
ProcessLog *slog.Logger
|
||||||
// GeoJSURL is where clients' AS numbers and countries are looked up
|
// Metrics count what waf does. Whoever builds them adds the metrics
|
||||||
// while SWW_WAF_LOOKUP_SOURCE is geojs, normally lookup.URL.
|
// read from the other parts.
|
||||||
GeoJSURL string
|
Metrics *metrics.Metrics
|
||||||
// AbuseIPDBURL is where clients are checked with AbuseIPDB while
|
// Limiter is the table of clients the rate limits count, with each
|
||||||
// SWW_WAF_ABUSEIPDB_KEY is set, normally reputation.AbuseIPDBURL.
|
// client's history.
|
||||||
AbuseIPDBURL string
|
Limiter *ratelimit.Limiter
|
||||||
|
// Ledger holds the bans.
|
||||||
|
Ledger *bans.Ledger
|
||||||
|
// Anomalies are the counters of the anomaly thresholds.
|
||||||
|
Anomalies *anomaly.Counters
|
||||||
|
// Lists are the lists fetched from URLs, DNSBL the DNSBL zones'
|
||||||
|
// verdicts, and AbuseIPDB AbuseIPDB's scores and the checks spent.
|
||||||
|
Lists *reputation.Lists
|
||||||
|
DNSBL *reputation.DNSBL
|
||||||
|
AbuseIPDB *reputation.AbuseIPDB
|
||||||
|
// RouteWatch asks routewatch about clients, and is nil while
|
||||||
|
// SWW_WAF_ROUTEWATCH_URL is unset.
|
||||||
|
RouteWatch *routewatch.Client
|
||||||
|
// GeoJS looks clients' AS numbers and countries up while
|
||||||
|
// SWW_WAF_LOOKUP_SOURCE is geojs. It is given AddLookup as its Answered
|
||||||
|
// and GeoJSFailed as its Failed.
|
||||||
|
GeoJS *geojs.GeoJS
|
||||||
// LookupFile is the lookup database they are looked up in while
|
// LookupFile is the lookup database they are looked up in while
|
||||||
// SWW_WAF_LOOKUP_SOURCE is file, and nil otherwise.
|
// SWW_WAF_LOOKUP_SOURCE is file, and nil otherwise.
|
||||||
LookupFile *lookup.File
|
LookupFile *lookup.File
|
||||||
// CoreRuleSet is what NewCoreRuleSet returns: what inspects requests
|
// CoreRuleSet is what inspects requests while
|
||||||
// while SWW_WAF_CORE_RULE_SET_MODE is not off, and nil while it is.
|
// SWW_WAF_CORE_RULE_SET_MODE is not off, and nil while it is.
|
||||||
CoreRuleSet *coreruleset.CoreRuleSet
|
CoreRuleSet CoreRuleSetInspector
|
||||||
// Now tells the time by which requests are counted for the rate
|
// Now tells the time by which requests are counted for the rate
|
||||||
// limits, bans are made and run out, and GeoJS's answers are kept,
|
// limits, bans are made and run out, and GeoJS's answers are kept,
|
||||||
// normally time.Now in UTC, the time the state files give.
|
// normally time.Now in UTC, the time the state files give.
|
||||||
@@ -81,19 +100,29 @@ type Params struct {
|
|||||||
// failing, a query to a DNSBL zone or a check with AbuseIPDB failing,
|
// failing, a query to a DNSBL zone or a check with AbuseIPDB failing,
|
||||||
// or the day's AbuseIPDB checks used up.
|
// or the day's AbuseIPDB checks used up.
|
||||||
Alerts *alerts.Queue
|
Alerts *alerts.Queue
|
||||||
|
// BesideLookupTimeLimit is how long the heuristics of besideLookup have
|
||||||
|
// for a request. Zero means besideLookupTimeLimit. Tests that run the
|
||||||
|
// Core Rule Set lengthen it, since the race detector and a busy host can
|
||||||
|
// slow an inspection past besideLookupTimeLimit.
|
||||||
|
BesideLookupTimeLimit time.Duration
|
||||||
}
|
}
|
||||||
|
|
||||||
// Server is the server waf runs, with the parts whose state the state
|
// CoreRuleSetInspector inspects a request with the Core Rule Set, as
|
||||||
// files keep, the lookup database, nil unless SWW_WAF_LOOKUP_SOURCE is file, the
|
// coreruleset.CoreRuleSet's Inspect says. Tests give waf a stand-in for it.
|
||||||
// lists fetched from URLs, which its Run fetches, the DNSBL zones' verdicts,
|
type CoreRuleSetInspector interface {
|
||||||
// AbuseIPDB's scores and checks spent, routewatch's answers, and the metrics.
|
Inspect(
|
||||||
// RouteWatch is nil while SWW_WAF_ROUTEWATCH_URL is unset.
|
r *http.Request, client netip.Addr, body io.Reader,
|
||||||
|
) (coreruleset.Result, []byte, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Server is the server waf runs, with the parts of its Params. The tests
|
||||||
|
// read them.
|
||||||
type Server struct {
|
type Server struct {
|
||||||
*http.Server
|
*http.Server
|
||||||
|
|
||||||
Ledger *bans.Ledger
|
Ledger *bans.Ledger
|
||||||
Limiter *ratelimit.Limiter
|
Limiter *ratelimit.Limiter
|
||||||
GeoJS *lookup.GeoJS
|
GeoJS *geojs.GeoJS
|
||||||
Anomalies *anomaly.Counters
|
Anomalies *anomaly.Counters
|
||||||
LookupFile *lookup.File
|
LookupFile *lookup.File
|
||||||
Lists *reputation.Lists
|
Lists *reputation.Lists
|
||||||
@@ -103,165 +132,45 @@ type Server struct {
|
|||||||
Metrics *metrics.Metrics
|
Metrics *metrics.Metrics
|
||||||
}
|
}
|
||||||
|
|
||||||
// newHandler returns the handler with what every request shares, as the
|
// newHandler returns the handler with what every request shares, the parts
|
||||||
// settings in params make it.
|
// in params among it.
|
||||||
func newHandler(params Params) *handler {
|
func newHandler(params Params) *handler {
|
||||||
m := metrics.New(
|
|
||||||
params.Config.MetricsBusiestCountriesAndASNs,
|
|
||||||
params.Config.InstanceName,
|
|
||||||
)
|
|
||||||
lists, dnsbl, abuseIPDB := newReputation(params, m)
|
|
||||||
|
|
||||||
awaitingReport, err := lru.New[string, *decision](maxAwaitingReport)
|
awaitingReport, err := lru.New[string, *decision](maxAwaitingReport)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
panic(err) // New fails only for a size below one
|
panic(err) // New fails only for a size below one
|
||||||
}
|
}
|
||||||
|
|
||||||
h := &handler{
|
return &handler{
|
||||||
config: params.Config,
|
config: params.Config,
|
||||||
requestLog: params.RequestLog,
|
requestLog: params.RequestLog,
|
||||||
processLog: params.ProcessLog,
|
processLog: params.ProcessLog,
|
||||||
now: params.Now,
|
now: params.Now,
|
||||||
metrics: m,
|
metrics: params.Metrics,
|
||||||
awaitingReport: awaitingReport,
|
awaitingReport: awaitingReport,
|
||||||
limiter: ratelimit.New(ratelimit.Limits{
|
limiter: params.Limiter,
|
||||||
PerMinute: params.Config.RateLimitPerMinute,
|
ledger: params.Ledger,
|
||||||
PerHour: params.Config.RateLimitPerHour,
|
geojs: params.GeoJS,
|
||||||
PerDay: params.Config.RateLimitPerDay,
|
anomalies: params.Anomalies,
|
||||||
BytesPerMinute: params.Config.BytesLimitPerMinute,
|
lookupFile: params.LookupFile,
|
||||||
BytesPerHour: params.Config.BytesLimitPerHour,
|
lists: params.Lists,
|
||||||
BytesPerDay: params.Config.BytesLimitPerDay,
|
dnsbl: params.DNSBL,
|
||||||
}, params.Config.MaxTrackedClients),
|
abuseIPDB: params.AbuseIPDB,
|
||||||
ledger: bans.New(bans.Params{
|
routeWatch: params.RouteWatch,
|
||||||
LimitBanDuration: params.Config.LimitBanDuration,
|
rules: params.Rules,
|
||||||
LimitBanRepeatWindow: params.Config.LimitBanRepeatWindow,
|
coreRuleSet: params.CoreRuleSet,
|
||||||
LimitBanPermanentAbove: params.Config.LimitBanPermanentAbove,
|
alerts: params.Alerts,
|
||||||
AttackBanDuration: params.Config.AttackBanDuration,
|
|
||||||
MaxBans: params.Config.MaxBans,
|
|
||||||
}),
|
|
||||||
anomalies: anomaly.New(anomaly.Params{
|
|
||||||
Client: params.Config.AnomalyClient,
|
|
||||||
Net: params.Config.AnomalyNet,
|
|
||||||
ASN: params.Config.AnomalyASN,
|
|
||||||
Total: params.Config.AnomalyTotal,
|
|
||||||
WatchedNet: params.Config.AnomalyWatchedNet,
|
|
||||||
NetV4Prefix: params.Config.AnomalyNetV4Prefix,
|
|
||||||
NetV6Prefix: params.Config.AnomalyNetV6Prefix,
|
|
||||||
NamedNetblocks: params.Config.AnomalyWatchedNets,
|
|
||||||
Alerts: params.Alerts,
|
|
||||||
}),
|
|
||||||
lookupFile: params.LookupFile,
|
|
||||||
lists: lists,
|
|
||||||
dnsbl: dnsbl,
|
|
||||||
abuseIPDB: abuseIPDB,
|
|
||||||
rules: params.Rules,
|
|
||||||
coreRuleSet: params.CoreRuleSet,
|
|
||||||
alerts: params.Alerts,
|
|
||||||
}
|
|
||||||
if params.Config.RouteWatchURL != "" {
|
|
||||||
h.routeWatch = routewatch.New(routewatch.Params{
|
|
||||||
URL: params.Config.RouteWatchURL, Now: params.Now, ProcessLog: params.ProcessLog,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
h.geojs = lookup.New(lookup.Params{
|
|
||||||
URL: params.GeoJSURL,
|
|
||||||
Timeout: params.Config.ASNAndCountryLookupTimeout,
|
|
||||||
// The country list, the headers, the tag settings that name an asn=
|
// The country list, the headers, the tag settings that name an asn=
|
||||||
// or country= tag and the biased thresholds act on the answer before
|
// or country= tag and the biased thresholds act on the answer before
|
||||||
// the request goes on.
|
// the request goes on.
|
||||||
Wait: len(params.Config.ExclusivelyAllowedCountries) > 0 ||
|
waitForGeoJS: len(params.Config.ExclusivelyAllowedCountries) > 0 ||
|
||||||
params.Config.AddLookupHeaders || params.Config.TagsNeedLookup() ||
|
params.Config.AddLookupHeaders ||
|
||||||
|
params.Config.TagsNeedLookup() ||
|
||||||
biasedThresholdsSet(params.Config),
|
biasedThresholdsSet(params.Config),
|
||||||
Answered: h.addLookup,
|
besideLookupLimit: cmp.Or(
|
||||||
Now: params.Now,
|
params.BesideLookupTimeLimit,
|
||||||
ProcessLog: params.ProcessLog,
|
besideLookupTimeLimit,
|
||||||
Metrics: m,
|
|
||||||
Alerts: params.Alerts,
|
|
||||||
})
|
|
||||||
m.AddBansAndClients(h.ledger, h.limiter, params.Now)
|
|
||||||
m.AddRules(params.Rules)
|
|
||||||
|
|
||||||
return h
|
|
||||||
}
|
|
||||||
|
|
||||||
// newReputation returns the lists fetched from URLs, the DNSBL zones'
|
|
||||||
// verdicts and AbuseIPDB's scores, as the settings in params name them,
|
|
||||||
// with none fetched, asked for or checked yet, and adds their metrics to
|
|
||||||
// m, AbuseIPDB's while SWW_WAF_ABUSEIPDB_KEY is set.
|
|
||||||
func newReputation(
|
|
||||||
params Params, m *metrics.Metrics,
|
|
||||||
) (*reputation.Lists, *reputation.DNSBL, *reputation.AbuseIPDB) {
|
|
||||||
cfg := params.Config
|
|
||||||
lists := reputation.New(reputation.Params{
|
|
||||||
BlocklistURLs: cfg.BlocklistURLs,
|
|
||||||
Refresh: cfg.BlocklistRefresh,
|
|
||||||
ASNLimitPercentURL: cfg.ASNLimitPercentURL,
|
|
||||||
TagLists: cfg.TagLists,
|
|
||||||
Now: params.Now,
|
|
||||||
ProcessLog: params.ProcessLog,
|
|
||||||
Alerts: params.Alerts,
|
|
||||||
})
|
|
||||||
dnsbl := reputation.NewDNSBL(reputation.DNSBLParams{
|
|
||||||
Zones: cfg.DNSBLZones, Resolver: cfg.DNSBLResolver,
|
|
||||||
CacheTTL: cfg.IPReputationCacheTTL,
|
|
||||||
Timeout: cfg.DNSBLAbuseIPDBTimeout, Now: params.Now, ProcessLog: params.ProcessLog,
|
|
||||||
Alerts: params.Alerts,
|
|
||||||
})
|
|
||||||
abuseIPDB := reputation.NewAbuseIPDB(reputation.AbuseIPDBParams{
|
|
||||||
URL: params.AbuseIPDBURL, Key: cfg.AbuseIPDBKey, MinScore: cfg.AbuseIPDBMinScore,
|
|
||||||
DailyBudget: cfg.AbuseIPDBDailyBudget, CacheTTL: cfg.IPReputationCacheTTL,
|
|
||||||
Timeout: cfg.DNSBLAbuseIPDBTimeout, Now: params.Now, ProcessLog: params.ProcessLog,
|
|
||||||
Alerts: params.Alerts,
|
|
||||||
})
|
|
||||||
|
|
||||||
m.AddReputation(lists, dnsbl)
|
|
||||||
|
|
||||||
if cfg.AbuseIPDBKey != "" {
|
|
||||||
m.AddAbuseIPDB(abuseIPDB)
|
|
||||||
}
|
|
||||||
|
|
||||||
return lists, dnsbl, abuseIPDB
|
|
||||||
}
|
|
||||||
|
|
||||||
// NewCoreRuleSet returns the Core Rule Set at SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL,
|
|
||||||
// without the rules SWW_WAF_CORE_RULE_SET_DISABLED_RULES switches off, reading bodies
|
|
||||||
// up to SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE, with the exclusions in
|
|
||||||
// SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE and the fixes of the apps in SWW_WAF_APPS,
|
|
||||||
// or nil while SWW_WAF_CORE_RULE_SET_MODE is off. It
|
|
||||||
// reads at most QuestionBodyMaxBytes less one byte: a question's body of
|
|
||||||
// QuestionBodyMaxBytes may be cut short, and the Core Rule Set does not read a
|
|
||||||
// JSON or XML body over its limit. An exclusions file that coreruleset.New refuses is
|
|
||||||
// an error in every mode. While the mode is off, a set file is loaded with the
|
|
||||||
// Core Rule Set only to check it.
|
|
||||||
func NewCoreRuleSet(cfg *config.WAF) (*coreruleset.CoreRuleSet, error) {
|
|
||||||
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff &&
|
|
||||||
cfg.CoreRuleSetExclusionsFile == "" {
|
|
||||||
return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error
|
|
||||||
}
|
|
||||||
|
|
||||||
coreRuleSet, err := coreruleset.New(coreruleset.Params{
|
|
||||||
ParanoiaLevel: cfg.CoreRuleSetParanoiaLevel,
|
|
||||||
DisabledRules: cfg.CoreRuleSetDisabledRules,
|
|
||||||
BodyInspectionSize: min(
|
|
||||||
cfg.CoreRuleSetBodyInspectionSize,
|
|
||||||
QuestionBodyMaxBytes-1,
|
|
||||||
),
|
),
|
||||||
ExclusionsFile: cfg.CoreRuleSetExclusionsFile,
|
|
||||||
Apps: cfg.Apps,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf(
|
|
||||||
"SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE: %w",
|
|
||||||
err,
|
|
||||||
)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff {
|
|
||||||
return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error
|
|
||||||
}
|
|
||||||
|
|
||||||
return coreRuleSet, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// handler is waf's handler. It holds what every request shares; what
|
// handler is waf's handler. It holds what every request shares; what
|
||||||
@@ -274,7 +183,7 @@ type handler struct {
|
|||||||
metrics *metrics.Metrics
|
metrics *metrics.Metrics
|
||||||
limiter *ratelimit.Limiter
|
limiter *ratelimit.Limiter
|
||||||
ledger *bans.Ledger
|
ledger *bans.Ledger
|
||||||
geojs *lookup.GeoJS
|
geojs *geojs.GeoJS
|
||||||
anomalies *anomaly.Counters
|
anomalies *anomaly.Counters
|
||||||
lookupFile *lookup.File
|
lookupFile *lookup.File
|
||||||
lists *reputation.Lists
|
lists *reputation.Lists
|
||||||
@@ -282,8 +191,14 @@ type handler struct {
|
|||||||
abuseIPDB *reputation.AbuseIPDB
|
abuseIPDB *reputation.AbuseIPDB
|
||||||
routeWatch *routewatch.Client
|
routeWatch *routewatch.Client
|
||||||
rules *rules.Files
|
rules *rules.Files
|
||||||
coreRuleSet *coreruleset.CoreRuleSet
|
coreRuleSet CoreRuleSetInspector
|
||||||
alerts *alerts.Queue
|
alerts *alerts.Queue
|
||||||
|
// waitForGeoJS is true when a setting needs each request's answer from
|
||||||
|
// GeoJS before the request goes on.
|
||||||
|
waitForGeoJS bool
|
||||||
|
// besideLookupLimit is how long the heuristics of besideLookup have for a
|
||||||
|
// request, as Params.BesideLookupTimeLimit says.
|
||||||
|
besideLookupLimit time.Duration
|
||||||
// awaitingReport holds the decision on each answered question by its id
|
// awaitingReport holds the decision on each answered question by its id
|
||||||
// until its report comes, and nil for a question still being handled.
|
// until its report comes, and nil for a question still being handled.
|
||||||
awaitingReport *lru.Cache[string, *decision]
|
awaitingReport *lru.Cache[string, *decision]
|
||||||
|
|||||||
+197
-33
@@ -16,10 +16,19 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/anomaly"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
|
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/metrics"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/processlog"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/routewatch"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/rproxy"
|
"sneak.berlin/go/smallwebwaf/internal/rproxy"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||||
@@ -93,6 +102,10 @@ const off = "off"
|
|||||||
// on is the value that switches a setting that takes true or false on.
|
// on is the value that switches a setting that takes true or false on.
|
||||||
const on = "true"
|
const on = "true"
|
||||||
|
|
||||||
|
// falseValue is the value that switches a setting that takes true or false
|
||||||
|
// off.
|
||||||
|
const falseValue = "false"
|
||||||
|
|
||||||
// output collects what sww writes on stdout.
|
// output collects what sww writes on stdout.
|
||||||
type output struct {
|
type output struct {
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
@@ -181,11 +194,11 @@ func (o *output) requestLine(t *testing.T) logLine {
|
|||||||
// endedLines waits for count requests that went through rproxy, or to waf's
|
// endedLines waits for count requests that went through rproxy, or to waf's
|
||||||
// own endpoints, to have ended, and returns the last log line of each, in the
|
// own endpoints, to have ended, and returns the last log line of each, in the
|
||||||
// order waf took them. A question's last line is the one its report writes.
|
// order waf took them. A question's last line is the one its report writes.
|
||||||
|
// It waits as long as it takes, as requestWithBody does.
|
||||||
func (o *output) endedLines(t *testing.T, count int) []logLine {
|
func (o *output) endedLines(t *testing.T, count int) []logLine {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
deadline := time.Now().Add(waitLimit)
|
for {
|
||||||
for time.Now().Before(deadline) {
|
|
||||||
found := o.lastLines(t)
|
found := o.lastLines(t)
|
||||||
|
|
||||||
ended := len(found) >= count
|
ended := len(found) >= count
|
||||||
@@ -201,10 +214,6 @@ func (o *output) endedLines(t *testing.T, count int) []logLine {
|
|||||||
|
|
||||||
time.Sleep(pollInterval)
|
time.Sleep(pollInterval)
|
||||||
}
|
}
|
||||||
|
|
||||||
t.Fatalf("fewer than %d requests have ended after %s", count, waitLimit)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// lastLines returns the last request log line written so far of each request,
|
// lastLines returns the last request log line written so far of each request,
|
||||||
@@ -386,11 +395,11 @@ func startRProxy(
|
|||||||
server := rproxy.New(rproxy.Params{
|
server := rproxy.New(rproxy.Params{
|
||||||
Config: cfg,
|
Config: cfg,
|
||||||
RequestLog: io.Discard,
|
RequestLog: io.Discard,
|
||||||
ProcessLog: requestlog.NewProcessLogger(
|
ProcessLog: processlog.New(processlog.Params{
|
||||||
io.Discard,
|
Output: io.Discard,
|
||||||
"rproxy",
|
InstanceName: "rproxy",
|
||||||
slog.LevelInfo,
|
Level: slog.LevelInfo,
|
||||||
),
|
}),
|
||||||
})
|
})
|
||||||
|
|
||||||
listener, err := (&net.ListenConfig{}).Listen(
|
listener, err := (&net.ListenConfig{}).Listen(
|
||||||
@@ -420,7 +429,8 @@ func startRProxy(
|
|||||||
// With no geojsURL, there is no stand-in for GeoJS to look clients up at, and
|
// With no geojsURL, there is no stand-in for GeoJS to look clients up at, and
|
||||||
// SWW_WAF_LOOKUP_SOURCE is off unless env sets it. While it is file, the lookup
|
// SWW_WAF_LOOKUP_SOURCE is off unless env sets it. While it is file, the lookup
|
||||||
// database SWW_WAF_LOOKUP_DB_PATH names is read. Clients are checked with AbuseIPDB
|
// database SWW_WAF_LOOKUP_DB_PATH names is read. Clients are checked with AbuseIPDB
|
||||||
// at abuseIPDBURL while env sets SWW_WAF_ABUSEIPDB_KEY.
|
// at abuseIPDBURL while env sets SWW_WAF_ABUSEIPDB_KEY, and its blacklist is
|
||||||
|
// fetched from abuseIPDBBlacklistURL.
|
||||||
// SWW_WAF_CORE_RULE_SET_MODE is off unless env sets it, so that only the tests of the
|
// SWW_WAF_CORE_RULE_SET_MODE is off unless env sets it, so that only the tests of the
|
||||||
// Core Rule Set have their requests inspected by it.
|
// Core Rule Set have their requests inspected by it.
|
||||||
func newWAF(
|
func newWAF(
|
||||||
@@ -431,13 +441,15 @@ func newWAF(
|
|||||||
) (*waf.Server, *output, *alerts.Queue) {
|
) (*waf.Server, *output, *alerts.Queue) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
return newWAFWith(t, waf.New, geojsURL, now, env)
|
return newWAFWith(t, waf.New, geojsURL, abuseIPDBURL, now, env)
|
||||||
}
|
}
|
||||||
|
|
||||||
// newWAFWith is newWAF for the server build makes.
|
// newWAFWith is newWAF for the server build makes, with clients checked with
|
||||||
|
// AbuseIPDB at abuseIPDBCheckURL.
|
||||||
func newWAFWith(
|
func newWAFWith(
|
||||||
t *testing.T, build func(waf.Params) *waf.Server,
|
t *testing.T, build func(waf.Params) *waf.Server,
|
||||||
geojsURL string, now func() time.Time, env map[string]string,
|
geojsURL, abuseIPDBCheckURL string, now func() time.Time,
|
||||||
|
env map[string]string,
|
||||||
) (*waf.Server, *output, *alerts.Queue) {
|
) (*waf.Server, *output, *alerts.Queue) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
@@ -456,11 +468,11 @@ func newWAFWith(
|
|||||||
}
|
}
|
||||||
|
|
||||||
out := &output{}
|
out := &output{}
|
||||||
processLog := requestlog.NewProcessLogger(
|
processLog := processlog.New(processlog.Params{
|
||||||
out,
|
Output: out,
|
||||||
cfg.InstanceName,
|
InstanceName: cfg.InstanceName,
|
||||||
cfg.LogLevel,
|
Level: cfg.LogLevel,
|
||||||
)
|
})
|
||||||
|
|
||||||
ruleFiles, err := rules.Load(rules.Params{
|
ruleFiles, err := rules.Load(rules.Params{
|
||||||
Dir: cfg.RuleFilesDir, Enabled: cfg.RuleFilesEnabled, ProcessLog: processLog,
|
Dir: cfg.RuleFilesDir, Enabled: cfg.RuleFilesEnabled, ProcessLog: processLog,
|
||||||
@@ -490,25 +502,177 @@ func newWAFWith(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
coreRuleSet, err := waf.NewCoreRuleSet(cfg)
|
params := waf.Params{
|
||||||
|
Config: cfg,
|
||||||
|
RequestLog: out,
|
||||||
|
ProcessLog: processLog,
|
||||||
|
LookupFile: lookupFile,
|
||||||
|
Now: now,
|
||||||
|
Rules: ruleFiles,
|
||||||
|
Alerts: alertQueue,
|
||||||
|
// The tests of the limit set it back, and hold the Core Rule Set
|
||||||
|
// with a stand-in.
|
||||||
|
BesideLookupTimeLimit: time.Hour,
|
||||||
|
}
|
||||||
|
|
||||||
|
addCoreRuleSet(t, ¶ms)
|
||||||
|
addCounters(¶ms)
|
||||||
|
addOutsideSources(¶ms, geojsURL, abuseIPDBCheckURL)
|
||||||
|
|
||||||
|
server := build(params)
|
||||||
|
|
||||||
|
return server, out, alertQueue
|
||||||
|
}
|
||||||
|
|
||||||
|
// addCoreRuleSet adds to params the Core Rule Set, with the settings for it,
|
||||||
|
// while SWW_WAF_CORE_RULE_SET_MODE is not off.
|
||||||
|
func addCoreRuleSet(t *testing.T, params *waf.Params) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
cfg := params.Config
|
||||||
|
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
coreRuleSet, err := coreruleset.New(coreruleset.Params{
|
||||||
|
ParanoiaLevel: cfg.CoreRuleSetParanoiaLevel,
|
||||||
|
DisabledRules: cfg.CoreRuleSetDisabledRules,
|
||||||
|
BodyInspectionSize: cfg.CoreRuleSetBodyInspectionSize,
|
||||||
|
ExclusionsFile: cfg.CoreRuleSetExclusionsFile,
|
||||||
|
Apps: cfg.Apps,
|
||||||
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Core Rule Set: %v", err)
|
t.Fatalf("Core Rule Set: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
server := build(waf.Params{
|
params.CoreRuleSet = coreRuleSet
|
||||||
Config: cfg,
|
}
|
||||||
RequestLog: out,
|
|
||||||
ProcessLog: processLog,
|
// addCounters adds to params the metrics, the table of clients, the ledger
|
||||||
GeoJSURL: geojsURL,
|
// and the anomaly counters, as sww builds them from params' settings. The
|
||||||
AbuseIPDBURL: abuseIPDBURL,
|
// metrics read the ledger, the table of clients and the rules.
|
||||||
LookupFile: lookupFile,
|
func addCounters(params *waf.Params) {
|
||||||
CoreRuleSet: coreRuleSet,
|
cfg := params.Config
|
||||||
Now: now,
|
|
||||||
Rules: ruleFiles,
|
params.Metrics = metrics.New(
|
||||||
Alerts: alertQueue,
|
cfg.MetricsBusiestCountriesAndASNs,
|
||||||
|
cfg.InstanceName,
|
||||||
|
)
|
||||||
|
params.Limiter = ratelimit.New(ratelimit.Params{
|
||||||
|
Limits: ratelimit.Limits{
|
||||||
|
PerMinute: cfg.RateLimitPerMinute,
|
||||||
|
PerHour: cfg.RateLimitPerHour,
|
||||||
|
PerDay: cfg.RateLimitPerDay,
|
||||||
|
BytesPerMinute: cfg.BytesLimitPerMinute,
|
||||||
|
BytesPerHour: cfg.BytesLimitPerHour,
|
||||||
|
BytesPerDay: cfg.BytesLimitPerDay,
|
||||||
|
},
|
||||||
|
MaxClients: cfg.MaxTrackedClients,
|
||||||
|
})
|
||||||
|
params.Ledger = bans.New(bans.Params{
|
||||||
|
LimitBanDuration: cfg.LimitBanDuration,
|
||||||
|
LimitBanRepeatWindow: cfg.LimitBanRepeatWindow,
|
||||||
|
LimitBanPermanentAbove: cfg.LimitBanPermanentAbove,
|
||||||
|
AttackBanDuration: cfg.AttackBanDuration,
|
||||||
|
MaxBans: cfg.MaxBans,
|
||||||
|
})
|
||||||
|
params.Anomalies = anomaly.New(anomaly.Params{
|
||||||
|
Client: cfg.AnomalyClient,
|
||||||
|
Net: cfg.AnomalyNet,
|
||||||
|
ASN: cfg.AnomalyASN,
|
||||||
|
Total: cfg.AnomalyTotal,
|
||||||
|
WatchedNet: cfg.AnomalyWatchedNet,
|
||||||
|
NetV4Prefix: cfg.AnomalyNetV4Prefix,
|
||||||
|
NetV6Prefix: cfg.AnomalyNetV6Prefix,
|
||||||
|
NamedNetblocks: cfg.AnomalyWatchedNets,
|
||||||
|
Alerts: params.Alerts,
|
||||||
})
|
})
|
||||||
|
|
||||||
return server, out, alertQueue
|
params.Metrics.AddBansAndClients(
|
||||||
|
params.Ledger,
|
||||||
|
params.Limiter,
|
||||||
|
params.Now,
|
||||||
|
)
|
||||||
|
params.Metrics.AddRules(params.Rules)
|
||||||
|
}
|
||||||
|
|
||||||
|
// addOutsideSources adds to params the lists fetched from URLs, the DNSBL
|
||||||
|
// zones' verdicts, AbuseIPDB's scores from abuseIPDBCheckURL and its
|
||||||
|
// blacklist from abuseIPDBBlacklistURL, routewatch's answers while its
|
||||||
|
// setting is set, and GeoJS at geojsURL, as sww builds
|
||||||
|
// them from params' settings and the parts addCounters added. The metrics
|
||||||
|
// read the lists, the zones' verdicts, and AbuseIPDB's while its key is set.
|
||||||
|
func addOutsideSources(
|
||||||
|
params *waf.Params, geojsURL, abuseIPDBCheckURL string,
|
||||||
|
) {
|
||||||
|
cfg := params.Config
|
||||||
|
m, limiter, ledger := params.Metrics, params.Limiter, params.Ledger
|
||||||
|
processLog, alertQueue := params.ProcessLog, params.Alerts
|
||||||
|
|
||||||
|
params.Lists = reputation.New(reputation.Params{
|
||||||
|
BlocklistURLs: cfg.BlocklistURLs,
|
||||||
|
Refresh: cfg.BlocklistRefresh,
|
||||||
|
ASNLimitPercentURL: cfg.ASNLimitPercentURL,
|
||||||
|
TagLists: cfg.TagLists,
|
||||||
|
Now: params.Now,
|
||||||
|
ProcessLog: processLog,
|
||||||
|
Alerts: alertQueue,
|
||||||
|
})
|
||||||
|
params.DNSBL = reputation.NewDNSBL(reputation.DNSBLParams{
|
||||||
|
Zones: cfg.DNSBLZones,
|
||||||
|
Resolver: cfg.DNSBLResolver,
|
||||||
|
CacheTTL: cfg.IPReputationCacheTTL,
|
||||||
|
Timeout: cfg.DNSBLAbuseIPDBTimeout,
|
||||||
|
Now: params.Now,
|
||||||
|
ProcessLog: processLog,
|
||||||
|
Alerts: alertQueue,
|
||||||
|
})
|
||||||
|
params.AbuseIPDB = reputation.NewAbuseIPDB(reputation.AbuseIPDBParams{
|
||||||
|
URL: abuseIPDBCheckURL,
|
||||||
|
Key: cfg.AbuseIPDBKey,
|
||||||
|
MinScore: cfg.AbuseIPDBMinScore,
|
||||||
|
DailyBudget: cfg.AbuseIPDBDailyBudget,
|
||||||
|
CacheTTL: cfg.IPReputationCacheTTL,
|
||||||
|
Timeout: cfg.DNSBLAbuseIPDBTimeout,
|
||||||
|
BlacklistURL: abuseIPDBBlacklistURL,
|
||||||
|
Threshold: cfg.AbuseIPDBThreshold,
|
||||||
|
BlacklistRefresh: cfg.AbuseIPDBBlacklistRefresh,
|
||||||
|
Blacklisted: func(listed []reputation.BlacklistEntry) {
|
||||||
|
waf.BanAbuseIPDBBlacklist(
|
||||||
|
cfg, ledger, processLog, params.Now(), listed,
|
||||||
|
)
|
||||||
|
},
|
||||||
|
Now: params.Now,
|
||||||
|
ProcessLog: processLog,
|
||||||
|
Alerts: alertQueue,
|
||||||
|
})
|
||||||
|
|
||||||
|
if cfg.RouteWatchURL != "" {
|
||||||
|
params.RouteWatch = routewatch.New(routewatch.Params{
|
||||||
|
URL: cfg.RouteWatchURL,
|
||||||
|
Now: params.Now,
|
||||||
|
ProcessLog: processLog,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
params.GeoJS = geojs.New(geojs.Params{
|
||||||
|
URL: geojsURL,
|
||||||
|
Answered: func(answer geojs.Answer) {
|
||||||
|
waf.AddLookup(cfg, limiter, ledger, answer)
|
||||||
|
},
|
||||||
|
Now: params.Now,
|
||||||
|
Asked: m.GeoJSRequests.Inc,
|
||||||
|
Failed: func(err error, askingAgainIn time.Duration) {
|
||||||
|
waf.GeoJSFailed(processLog, alertQueue, m, err, askingAgainIn)
|
||||||
|
},
|
||||||
|
WentWithoutAnswer: m.RequestsWithoutGeoJSAnswer.Inc,
|
||||||
|
})
|
||||||
|
|
||||||
|
m.AddReputation(params.Lists, params.DNSBL)
|
||||||
|
|
||||||
|
if cfg.AbuseIPDBKey != "" {
|
||||||
|
m.AddAbuseIPDB(params.AbuseIPDB)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// newClient returns an HTTP client that sends requests as they are made,
|
// newClient returns an HTTP client that sends requests as they are made,
|
||||||
|
|||||||
@@ -38,8 +38,11 @@ func (rateLimits) check(
|
|||||||
rq.line.BytesLimitPercent, rq.line.BytesLimitPercentSetting =
|
rq.line.BytesLimitPercent, rq.line.BytesLimitPercentSetting =
|
||||||
rq.bytesLimitPercent.logged()
|
rq.bytesLimitPercent.logged()
|
||||||
|
|
||||||
counts, hit, over := rq.h.limiter.Count(rq.h.clientGroup(rq.client), now,
|
counts, hit, over := rq.h.limiter.Count(
|
||||||
rq.rateLimitPercent.percent)
|
clientGroup(rq.h.config, rq.client),
|
||||||
|
now,
|
||||||
|
rq.rateLimitPercent.percent,
|
||||||
|
)
|
||||||
rq.line.Counts = counts
|
rq.line.Counts = counts
|
||||||
|
|
||||||
if !over {
|
if !over {
|
||||||
@@ -59,7 +62,7 @@ func (rateLimits) check(
|
|||||||
// limit. It counts nothing.
|
// limit. It counts nothing.
|
||||||
func (rq *request) overAWholeRateLimit(now time.Time) bool {
|
func (rq *request) overAWholeRateLimit(now time.Time) bool {
|
||||||
return rq.countedByRateLimits() &&
|
return rq.countedByRateLimits() &&
|
||||||
rq.h.limiter.OverARateLimit(rq.h.clientGroup(rq.client), now)
|
rq.h.limiter.OverARateLimit(clientGroup(rq.h.config, rq.client), now)
|
||||||
}
|
}
|
||||||
|
|
||||||
// countedByRateLimits reports whether the rate limits count the request: its
|
// countedByRateLimits reports whether the rate limits count the request: its
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ import (
|
|||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
"sneak.berlin/go/smallwebwaf/internal/geojs"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
)
|
)
|
||||||
@@ -162,7 +162,7 @@ func TestRateLimitExemptPathsAreNeitherCountedNorRefused(t *testing.T) {
|
|||||||
|
|
||||||
// The answers are kept before the requests, so that none waits for
|
// The answers are kept before the requests, so that none waits for
|
||||||
// GeoJS.
|
// GeoJS.
|
||||||
server.GeoJS.Load([]lookup.Answer{
|
server.GeoJS.Load([]geojs.Answer{
|
||||||
keptAnswer(client, "DE"), keptAnswer(fromKP, "KP"),
|
keptAnswer(client, "DE"), keptAnswer(fromKP, "KP"),
|
||||||
})
|
})
|
||||||
|
|
||||||
|
|||||||
@@ -36,8 +36,11 @@ func (refusalLimit) check(
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
hit, over := rq.h.limiter.CountRefusal(rq.h.clientGroup(rq.client), now,
|
hit, over := rq.h.limiter.CountRefusal(
|
||||||
cfg.RefusalsLimitPerMinute)
|
clientGroup(rq.h.config, rq.client),
|
||||||
|
now,
|
||||||
|
cfg.RefusalsLimitPerMinute,
|
||||||
|
)
|
||||||
if !over {
|
if !over {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -7,10 +7,6 @@ import (
|
|||||||
"sneak.berlin/go/smallwebwaf/internal/bans"
|
"sneak.berlin/go/smallwebwaf/internal/bans"
|
||||||
)
|
)
|
||||||
|
|
||||||
// deny is the SWW_WAF_BLOCKLIST_ACTION and the SWW_WAF_DNSBL_ABUSEIPDB_ACTION that
|
|
||||||
// refuses the requests of a client a source lists.
|
|
||||||
const deny = "deny"
|
|
||||||
|
|
||||||
// noteListed notes each of sources, the URLs of the blocklists or the
|
// noteListed notes each of sources, the URLs of the blocklists or the
|
||||||
// DNSBL zones, their keys masked, that list the client, as noteHit does,
|
// DNSBL zones, their keys masked, that list the client, as noteHit does,
|
||||||
// with reason.
|
// with reason.
|
||||||
@@ -37,7 +33,7 @@ func (rq *request) noteHit(hit bans.ReputationHit, reason string) {
|
|||||||
Event: alerts.EventReputationHit,
|
Event: alerts.EventReputationHit,
|
||||||
Host: rq.site,
|
Host: rq.site,
|
||||||
Client: rq.client,
|
Client: rq.client,
|
||||||
Netblock: rq.h.clientGroup(rq.client),
|
Netblock: clientGroup(rq.h.config, rq.client),
|
||||||
ASN: rq.line.ASN,
|
ASN: rq.line.ASN,
|
||||||
ASName: rq.line.ASName,
|
ASName: rq.line.ASName,
|
||||||
Country: rq.line.Country,
|
Country: rq.line.Country,
|
||||||
|
|||||||
@@ -822,22 +822,39 @@ const abuseipdb = reputation.AbuseIPDBSource
|
|||||||
// through which AbuseIPDB is asked.
|
// through which AbuseIPDB is asked.
|
||||||
const abuseIPDBURL = "abuseipdb://stand-in/api/v2/check"
|
const abuseIPDBURL = "abuseipdb://stand-in/api/v2/check"
|
||||||
|
|
||||||
|
// abuseIPDBBlacklistURL is where newWAF has AbuseIPDB's blacklist fetched
|
||||||
|
// from: at abuseIPDBStandIn too.
|
||||||
|
const abuseIPDBBlacklistURL = "abuseipdb://stand-in/api/v2/blacklist"
|
||||||
|
|
||||||
// abuseIPDBStandIn is a stand-in for AbuseIPDB that gives every client the
|
// abuseIPDBStandIn is a stand-in for AbuseIPDB that gives every client the
|
||||||
// score 100, at once and without the network.
|
// score 100, at once and without the network. Its blacklist lists client
|
||||||
|
// alone, with the score 100, to a fetch with accountKey that asks for the
|
||||||
|
// clients scored 90 or more, the default threshold; it refuses any other
|
||||||
|
// fetch of it with 400.
|
||||||
type abuseIPDBStandIn struct{}
|
type abuseIPDBStandIn struct{}
|
||||||
|
|
||||||
// RoundTrip answers req with the score 100.
|
// RoundTrip answers req with the score 100, or with the blacklist.
|
||||||
func (abuseIPDBStandIn) RoundTrip(
|
func (abuseIPDBStandIn) RoundTrip(
|
||||||
req *http.Request,
|
req *http.Request,
|
||||||
) (*http.Response, error) {
|
) (*http.Response, error) {
|
||||||
|
status, body := http.StatusOK, `{"data":{"abuseConfidenceScore":100}}`
|
||||||
|
|
||||||
|
if req.URL.Path == "/api/v2/blacklist" {
|
||||||
|
body = `{"data":[{"ipAddress":"` + client +
|
||||||
|
`","abuseConfidenceScore":100}]}`
|
||||||
|
|
||||||
|
if req.Header.Get("Key") != accountKey ||
|
||||||
|
req.URL.RawQuery != "confidenceMinimum=90" {
|
||||||
|
status, body = http.StatusBadRequest, `{"errors":[]}`
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
return &http.Response{
|
return &http.Response{
|
||||||
StatusCode: http.StatusOK,
|
StatusCode: status,
|
||||||
Status: "200 OK",
|
Status: fmt.Sprintf("%d %s", status, http.StatusText(status)),
|
||||||
Header: http.Header{},
|
Header: http.Header{},
|
||||||
Body: io.NopCloser(
|
Body: io.NopCloser(strings.NewReader(body)),
|
||||||
strings.NewReader(`{"data":{"abuseConfidenceScore":100}}`),
|
Request: req,
|
||||||
),
|
|
||||||
Request: req,
|
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
Loaded 100 of 110 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user